| *** rcernin has joined #openstack-swift | 02:27 | |
| *** rcernin has quit IRC | 03:31 | |
| *** psachin has joined #openstack-swift | 04:29 | |
| *** manuvakery has joined #openstack-swift | 04:39 | |
| *** jvisser has quit IRC | 04:55 | |
| *** viks___ has joined #openstack-swift | 04:57 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-swift | 05:34 | |
| *** zaitcev has quit IRC | 05:36 | |
| manuvakery | hi guys .. few days back i set the weight of a disk (DISK12) to zero but not removed from ring. All the partitions moved to other disks and everything seems fine . Now am getting the below error in the replicator | 06:02 |
|---|---|---|
| manuvakery | https://www.irccloud.com/pastebin/NtlOGa9S/replicator_error | 06:02 |
| manuvakery | Is it because the disk is still present in the ring ?? | 06:04 |
| seongsoocho | manuvakery: HI, Please check the owner of directory /srv/node/disk12 (swift:swift) | 06:21 |
| manuvakery | @seongsoocho> the owner is not swift for now . I was in the impression that if i set the weight to 0. swift wont do anything on that disk | 06:23 |
| seongsoocho | manuvakery: I think replicator make objects directory if not exists. check this code : https://github.com/openstack/swift/blob/2.23.1/swift/obj/replicator.py#L813 | 06:28 |
| manuvakery | seongsoocho: yeah correct. thanks | 06:31 |
| seongsoocho | :-) | 06:31 |
| manuvakery | seongsoocho: i can see one more error in the object-error.log , not related to the disk permission but connection with the hidden lock file swift uses to acquire the directory locking | 06:41 |
| manuvakery | https://www.irccloud.com/pastebin/zmOVNiC8/rep-error | 06:42 |
| manuvakery | the error is only on this specific folder | 06:42 |
| seongsoocho | manuvakery: what version do you use? | 06:43 |
| manuvakery | stein | 06:45 |
| manuvakery | if i do an ls on /srv/node/disk5/objects/210847/.lock i can see | 06:48 |
| manuvakery | https://usercontent.irccloud-cdn.com/file/aOmMUUNs/Selection_040.png | 06:48 |
| manuvakery | somehow there is an addition .lock directory | 06:49 |
| seongsoocho | manuvakery: oh.. The .lock should be a file not a directory. That's weird. | 06:50 |
| manuvakery | how about moving the contents from .lock dir to /srv/node/disk5/objects/210847 manually .. any harm? | 06:52 |
| seongsoocho | hmm .. What is the date of .lock/hashes.pkl ? If It is not a today, how about rename .lock directory to .lock_tmp ? I'm not sure the side effect to moving the contents manually. | 06:58 |
| manuvakery | seongsoocho: -rw------- 1 swift swift 1027 Feb 3 06:16 hashes.pkl its today | 07:01 |
| seongsoocho | oh ... Then we need from someone else. | 07:05 |
| manuvakery | seongsoocho: ok will wait then | 07:06 |
| *** tkajinam has quit IRC | 07:36 | |
| *** tkajinam has joined #openstack-swift | 07:38 | |
| *** pcaruana has joined #openstack-swift | 07:43 | |
| *** tkajinam_ has joined #openstack-swift | 08:01 | |
| openstackgerrit | David Sariel proposed openstack/swift master: DNM - dd probe tests https://review.opendev.org/705063 | 08:04 |
| *** tkajinam has quit IRC | 08:04 | |
| *** rpittau|afk is now known as rpittau | 08:05 | |
| *** rdejoux has joined #openstack-swift | 08:08 | |
| *** tesseract has joined #openstack-swift | 08:14 | |
| *** tkajinam_ has quit IRC | 08:38 | |
| *** mikecmpbll has joined #openstack-swift | 09:00 | |
| *** tkajinam has joined #openstack-swift | 11:28 | |
| *** rpittau is now known as rpittau|bbl | 11:42 | |
| *** psachin has quit IRC | 13:16 | |
| *** rpittau|bbl is now known as rpittau | 13:34 | |
| *** jvisser has joined #openstack-swift | 14:04 | |
| *** tkajinam has quit IRC | 14:16 | |
| clayg | manuvakery: seongsoocho: I've seen files unexpectedly be directories before - but I'm never quite sure if it was file system corruption or if an operator accidently mis-typed something on a node | 14:41 |
| clayg | manuvakery: seongsoocho: I don't think swift would automatically repair from this - ideally the code would quaratine the `.lock` directory | 14:41 |
| clayg | manuvakery: seongsoocho: there is some potential to create "dark data" if the information in those suffixes has been disconnected from the cluster for a long time | 14:42 |
| clayg | you should move it out of the way (i.e. manually quarantine, as seongsoocho suggested) and let replication finish | 14:42 |
| clayg | ... then inspect the contents of those quarantined suffixes with tools like swift-object-info and inspect the primary nodes and container listings for those objects | 14:43 |
| clayg | most likely they can be safely deleted 🤞 | 14:43 |
| manuvakery | clayg: u mean to say I can rename the .lock dir to .lock_tmp and let the replication take care the rest | 15:15 |
| clayg | yes, but I might even suggest moving it *out* of the data dir tree entirely - into the quarntine directory is always a safe place | 15:23 |
| clayg | but yes, move it out of the way for now and see if replication can get things back on track | 15:24 |
| clayg | I am curious if there's any .data files in those suffixes or if they're all empty? | 15:24 |
| manuvakery | Ok.. yes there is .data file inside the .lock dir | 15:28 |
| clayg | intrestig... so if you use use the swift-object-info tool after you've moved the data out of the way you may gain some additional insight into the nature of the data in the corrupt path | 15:37 |
| clayg | for example - it will tell you where all the primary homes for that object should be - if the primaries already have a copy of the same data then you can definately delete it! | 15:37 |
| *** rdejoux has quit IRC | 15:48 | |
| manuvakery | Ok . I will try that tomorrow. Thanks clayg | 15:48 |
| clayg | of course! do keep us posted 😁 | 15:49 |
| DHE | just want to verify something. I'm using ssync-based replication for my object servers and it seems I'm limited to 1 replication session per disk. so the maximum number of streams to a server is limited to the number of disks. I feel like that's not the intended result. | 15:51 |
| *** gyee has joined #openstack-swift | 16:04 | |
| *** rpittau is now known as rpittau|afk | 16:33 | |
| timburke | DHE, have a loot at replication_concurrency and replication_concurrency_per_device: https://github.com/openstack/swift/blob/2.24.0/etc/object-server.conf-sample#L167-L178 | 16:38 |
| DHE | ah, that's what I was looking for... I swear I looked... | 16:39 |
| DHE | thanks | 16:39 |
| timburke | FWIW, i'm pretty sure the default for replication_concurrency_per_device came out of a desire to match the pre https://github.com/openstack/swift/commit/e199192c behavior -- no idea about the default for replication_concurrency, but it seems to have come in with the original ssync code, so probably pulled out of thin air | 16:40 |
| DHE | I was thinking relative to rsync where I think the suggested defaults is 4 | 16:40 |
| timburke | we should consider updating those defaults to something more like what people actually tend to run with | 16:40 |
| DHE | stupid 10gig switch has a firmware bug that causes the port to negotiate at 1gig speeds, so this is capped anyway... | 16:42 |
| timburke | :-( | 16:45 |
| *** evrardjp has quit IRC | 17:33 | |
| *** evrardjp has joined #openstack-swift | 17:34 | |
| *** mikecmpbll has quit IRC | 17:44 | |
| *** mikecmpbll has joined #openstack-swift | 18:40 | |
| *** diablo_rojo has joined #openstack-swift | 19:02 | |
| *** mikecmpbll has quit IRC | 19:02 | |
| *** mikecmpbll has joined #openstack-swift | 19:03 | |
| *** zaitcev has joined #openstack-swift | 19:05 | |
| *** ChanServ sets mode: +v zaitcev | 19:05 | |
| *** mikecmpbll has quit IRC | 19:09 | |
| *** tesseract has quit IRC | 20:01 | |
| *** irclogbot_1 has quit IRC | 20:05 | |
| *** irclogbot_3 has joined #openstack-swift | 20:06 | |
| *** rdejoux has joined #openstack-swift | 21:06 | |
| *** rdejoux has quit IRC | 21:10 | |
| mattoliverau | Morning | 22:11 |
| openstackgerrit | Tim Burke proposed openstack/swift master: WIP: run probe tests on CentOS 8 https://review.opendev.org/690717 | 22:12 |
| *** tkajinam has joined #openstack-swift | 22:53 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: py3: Fix up probe tests https://review.opendev.org/705578 | 23:15 |
| openstackgerrit | Tim Burke proposed openstack/swift master: probe tests: Work when fronted by a TLS terminator https://review.opendev.org/705579 | 23:15 |
| openstackgerrit | Tim Burke proposed openstack/swift master: WIP: run probe tests on CentOS 8 https://review.opendev.org/690717 | 23:44 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
