| mattoliverau | morning | 00:01 |
|---|---|---|
| mattoliverau | belated obviously :P | 00:01 |
| *** mikecmpbll has quit IRC | 00:02 | |
| *** tkajinam has quit IRC | 00:07 | |
| *** tkajinam has joined #openstack-swift | 00:07 | |
| *** patchbot has quit IRC | 00:11 | |
| *** patchbot has joined #openstack-swift | 00:12 | |
| *** noonedeadpunk has quit IRC | 00:39 | |
| *** noonedeadpunk has joined #openstack-swift | 00:42 | |
| *** gyee has quit IRC | 00:47 | |
| openstackgerrit | Merged openstack/swift master: Forward-port stable-release changelog entries https://review.opendev.org/699514 | 01:20 |
| *** diablo_rojo has quit IRC | 01:55 | |
| openstackgerrit | Merged openstack/swift master: Fix type on py3 https://review.opendev.org/699513 | 02:15 |
| *** psachin has joined #openstack-swift | 03:39 | |
| *** StevenK_ is now known as StevenK | 03:44 | |
| tdasilva | https://blogs.oracle.com/linux/xfs-2019-development-retrospective | 04:05 |
| tdasilva | i wonder what these "Large Directory" changes are | 04:05 |
| mattoliverau | I wonder if Dave Chinner will be at LCA again this year. maybe with another XFS talk. I'll make sure to look out for it! | 04:30 |
| zaitcev_ | Doesn't give jack to Swift though | 04:40 |
| zaitcev_ | We use a rational size tree. Well, you can have 260000 partitions with a 22-bit prefix. | 04:40 |
| timburke | i wonder what "large" means in this context -- number of files/directories? size of FS metadata for all files/directories? i'm assuming not size of all file *data*... | 04:45 |
| zaitcev_ | I'm pretty sure he means the size of the image of the directory (in its backing file) as well as a typical number of files in it. Basically whatever affects the speeds on lookups and insertions. | 04:47 |
| *** pcaruana has joined #openstack-swift | 05:24 | |
| *** rcernin has quit IRC | 06:56 | |
| *** ccamacho has joined #openstack-swift | 07:38 | |
| *** rdejoux has quit IRC | 07:54 | |
| *** rdejoux has joined #openstack-swift | 08:26 | |
| *** tkajinam has quit IRC | 08:34 | |
| *** rpittau|afk is now known as rpittau | 08:59 | |
| *** mikecmpbll has joined #openstack-swift | 09:08 | |
| *** pawan-gupta has quit IRC | 09:47 | |
| *** csmart has quit IRC | 13:23 | |
| *** zaitcev__ has joined #openstack-swift | 14:18 | |
| *** ChanServ sets mode: +v zaitcev__ | 14:18 | |
| *** zaitcev_ has quit IRC | 14:22 | |
| *** mikecmpbll has quit IRC | 14:57 | |
| *** mikecmpbll has joined #openstack-swift | 14:59 | |
| *** rdejoux has quit IRC | 15:55 | |
| *** gyee has joined #openstack-swift | 16:03 | |
| *** btorch has joined #openstack-swift | 16:33 | |
| *** diablo_rojo has joined #openstack-swift | 17:18 | |
| *** rpittau is now known as rpittau|afk | 17:21 | |
| *** mikecmpbll has quit IRC | 17:39 | |
| *** psachin has quit IRC | 17:39 | |
| *** csmart has joined #openstack-swift | 17:42 | |
| *** baffle has joined #openstack-swift | 17:42 | |
| *** openstackgerrit has quit IRC | 18:43 | |
| *** btorch has quit IRC | 20:08 | |
| *** rdejoux has joined #openstack-swift | 20:52 | |
| timburke | meeting time! | 21:01 |
| timburke | tdasilva, clayg, mattoliverau, zaitcev__ ^^^ | 21:03 |
| *** zaitcev__ is now known as zaitcev | 21:12 | |
| *** szaher has quit IRC | 21:15 | |
| seongsoocho | oh I late for meeting. o/ | 21:16 |
| timburke | seongsoocho, that's ok! no worries :-) | 21:18 |
| alecuyer | hello, late as well, sorry | 21:22 |
| timburke | alecuyer, o/ | 21:44 |
| timburke | oh -- i maybe should have called out https://bugs.launchpad.net/swift/+bug/1856894 -- noticed it while working on a sharding/versioning probe test... | 21:45 |
| openstack | Launchpad bug 1856894 in OpenStack Object Storage (swift) "Can't shard containers with special characters in their names" [Undecided,New] | 21:45 |
| alecuyer | timburke: hello! sorry I was late and missed other meetings the weeks before, collided with family reunions but january should be more quiet :) | 21:45 |
| timburke | alecuyer, sounds good! and don't worry -- family comes first, for sure :D | 21:46 |
| seongsoocho | :-) Actually, I have a some trouble in my cluster (version: ocata) . The customer said that object are disapeared.. (The client got 404 for get object, after 1hours to uploaded object) | 21:48 |
| seongsoocho | I found that object in Quarantined directory and I don't know why the object is moved that directory. | 21:48 |
| seongsoocho | Is there any tips for figure out why the object quarantined? | 21:48 |
| rledisez | seongsoocho: you can try to run swift-object-info on the datafile. it gives some informations like invalid etag, missing header, … | 21:49 |
| seongsoocho | object-auditor is not running currently because of disk problem | 21:49 |
| rledisez | also, check your logs, there should be an info about which process quarantined the object (it could be replicator, server or auditor) | 21:49 |
| seongsoocho | oh.. swift-object-info.. ok | 21:49 |
| seongsoocho | I can't find any log about quarantined .... | 21:52 |
| seongsoocho | But I found a log for deleted object | 21:52 |
| seongsoocho | ``` | 21:52 |
| seongsoocho | object-server-replication: STDERR: ERROR:root:Quarantined /srv/node/sdd/objects/14/cfb/003abe738ee01be60fd3c69af5186cfb_1516935665.05830.ts to /srv/node/sdd/quarantined/objects/cfb because it is not a directory#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/swift/obj/diskfile.py", line 970, in _hash_suffix_dir#012 ondisk_info = self.cleanup_ondisk_files(hsh_path)#012 | 21:52 |
| seongsoocho | File "/usr/lib/python2.7/site-packages/swift/obj/diskfile.py", line 926, in cleanup_ondisk_files#012 files = listdir(hsh_path)#012 File "/usr/lib/python2.7/site-packages/swift/common/utils.py", line 3108, in listdir#012 return os.listdir(path)#012OSError: [Errno 20] Not a directory: '/srv/node/sdd/objects/14/cfb/003abe738ee01be60fd3c69af5186cfb_1516935665.05830.ts' | 21:52 |
| seongsoocho | ``` | 21:52 |
| alecuyer | hmm that _ looks funny ? | 21:53 |
| alecuyer | before the timestamp | 21:53 |
| rledisez | that don't look like a correct path | 21:53 |
| rledisez | did you had XFS corruption at some point? | 21:53 |
| *** pcaruana has quit IRC | 21:54 | |
| seongsoocho | nop. the filesystem works fine. Any error log in dmesg and system messages about XFS | 21:54 |
| rledisez | do you have ECC memory in your servers? it really looks like a bit-flip somewhere | 21:56 |
| rledisez | do you have a lot of this situation? | 21:57 |
| seongsoocho | A.. yes I have a memory error.. | 21:57 |
| timburke | did replication take care of replacing the quarantined copy? presumably, there should be a tombstone at /srv/node/sdd/objects/14/cfb/003abe738ee01be60fd3c69af5186cfb/1516935665.05830.ts | 21:57 |
| rledisez | ok, you might want to replace that memory first. and really think about getting ECC memory if it's acceptable in your budget | 21:58 |
| rledisez | I gotta go, good luck with your cluster seongsoocho! | 21:59 |
| seongsoocho | rledisez: Ok thank you for your help! | 21:59 |
| seongsoocho | I have a lot of this situation, but some servers have ECC error and some are not. | 22:02 |
| zaitcev | well... _ is 0x5F and / is 0x2F. That's more than just 1 bit flip. | 22:36 |
| zaitcev | cfb is also the name of the radix directory, that that's correct | 22:37 |
| zaitcev | And 1516935665 looks like a sensible timestamp | 22:37 |
| seongsoocho | yes.. The weird things is there are only one disk have quarantined directory in server. | 22:46 |
| *** tkajinam has joined #openstack-swift | 23:05 | |
| *** rcernin has joined #openstack-swift | 23:10 | |
| *** ccamacho has quit IRC | 23:23 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!