| openstackgerrit | Tim Burke proposed openstack/python-swiftclient master: Allow users to list containers/objects by size. https://review.opendev.org/708074 | 00:28 |
|---|---|---|
| *** gyee has quit IRC | 01:07 | |
| *** psachin has joined #openstack-swift | 03:28 | |
| *** evrardjp has quit IRC | 04:36 | |
| *** evrardjp has joined #openstack-swift | 04:37 | |
| openstackgerrit | Tim Burke proposed openstack/python-swiftclient master: Blacklist stestr 3.0.0 https://review.opendev.org/718309 | 05:45 |
| openstackgerrit | Tim Burke proposed openstack/swift stable/train: py3: stop barfing on message/rfc822 Content-Types https://review.opendev.org/718312 | 05:57 |
| *** rpittau|afk is now known as rpittau | 07:17 | |
| *** ccamacho has joined #openstack-swift | 07:39 | |
| *** rcernin has quit IRC | 08:27 | |
| *** tkajinam has quit IRC | 08:51 | |
| openstackgerrit | Romain de Joux proposed openstack/swift master: s3api: Propagate backend PATH_INFO in environ for other middleware https://review.opendev.org/718086 | 10:12 |
| *** rpittau is now known as rpittau|bbl | 10:24 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift master: Imported Translations from Zanata https://review.opendev.org/718385 | 10:48 |
| *** tkajinam has joined #openstack-swift | 12:14 | |
| *** rpittau|bbl is now known as rpittau | 12:23 | |
| openstackgerrit | Alexandre Lécuyer proposed openstack/swift feature/losf: Use hashes.pkl files for LOSF and remove list_partition_recursive() https://review.opendev.org/718445 | 13:22 |
| *** jv has quit IRC | 13:23 | |
| *** jv has joined #openstack-swift | 13:27 | |
| openstackgerrit | Merged openstack/swift feature/losf: Remove unused entries in go.sum https://review.opendev.org/718141 | 14:09 |
| viks____ | Hi, what could be causing the below error, anyone has any idea: | 14:45 |
| viks____ | ``` | 14:45 |
| viks____ | swift-storage0 container-server: ERROR __call__ error with DELETE /disk2/1552/5cc5358c50e34a3196eca65ea9039633/backup/Teest/Archive/test/c70c277b-4394-4a40-b33a-4ffde71c666c/92e7f53c-1564-c67a-5fc6-8033f8cba906/blocks/f095702de599e8c11a58396773784b28/1464.0cec0f83c3cb3e26894c23887b3a5099.5ae6b6933446440fb2ed522e2b757528.blk : LockTimeout (10s) | 14:45 |
| viks____ | /srv/node/disk2/containers/1552/e51/610717f4ac1d0c3ca1212deb49299e51/.lock (txn: txf5c6b201be2e4560ba359-005e8de1ed) | 14:45 |
| viks____ | Apr 8 13:31:26 swift-storage0 container-replicator: ERROR reading HTTP response from {'index': 0, u'replication_port': 6201, u'weight': 100.0, u'zone': 1, u'ip': u'10.0.0.88', u'region': 1, u'id': 0, u'replication_ip': u'172.16.0.88', u'meta': u'', u'device': u'disk2', u'port': 6201}: Timeout (10.0s) | 14:45 |
| viks____ | ``` | 14:45 |
| *** hoonetorg has quit IRC | 14:56 | |
| DHE | there's probably an existing replication job that's busy | 15:07 |
| *** gyee has joined #openstack-swift | 15:13 | |
| viks____ | DHE: thnx... but what is this `LockTimeout (10s)`... | 15:16 |
| DHE | replication locks various things, and especially replication may hold it locked for an extended period | 15:18 |
| viks____ | from where it is getting this 10s value from? i do not have any 10 seconds value set in my container-server.conf | 15:19 |
| DHE | that's just how long it will wait on a lock | 15:20 |
| viks____ | can i increase it ... i mean Locktimeout value? | 15:37 |
| *** hoonetorg has joined #openstack-swift | 16:01 | |
| DHE | I wouldn't bother most of the time. the lock is probably being held by a different cluster member trying to do the same sorts of things that this server wants to do | 16:20 |
| *** rpittau is now known as rpittau|afk | 16:23 | |
| *** psachin has quit IRC | 16:24 | |
| *** evrardjp has quit IRC | 16:37 | |
| *** evrardjp has joined #openstack-swift | 16:37 | |
| *** diablo_rojo has quit IRC | 18:02 | |
| *** diablo_rojo has joined #openstack-swift | 18:06 | |
| *** diablo_rojo has quit IRC | 18:53 | |
| openstackgerrit | Tim Burke proposed openstack/python-swiftclient master: Blacklist stestr 3.0.0 https://review.opendev.org/718309 | 19:50 |
| openstackgerrit | Romain LE DISEZ proposed openstack/swift master: proxy: stop sending frags to PyECLib with a Queue https://review.opendev.org/704892 | 19:58 |
| *** ccamacho has quit IRC | 20:08 | |
| openstackgerrit | Tim Burke proposed openstack/python-swiftclient master: object versioning features https://review.opendev.org/691877 | 20:39 |
| clayg | is there a swift meeting today? | 20:45 |
| timburke | yes please! | 20:45 |
| clayg | viks____: that 10s lock timeout means you have a lot of contention on your container disks 😬 | 20:46 |
| clayg | viks____: time for some sharding!!! or SSDs if you're not there already | 20:46 |
| clayg | of if you're using the same spindles for dbs and objs that's the worst - I'm not saying some swift deployments shouldn't be able to do that... | 20:47 |
| clayg | viks____: ... but with your access patterns that's not going to work for you for much longer - that's what that error is telling you | 20:47 |
| clayg | or rly, I guess that error wasn't really being helpful at all 😞 | 20:47 |
| clayg | software is terrible, but on the clusters where I've seen that error before the story is "you've run out of iops on your container disks" - so... you have to go from there! | 20:48 |
| clayg | I mean you could check me with iostat - look at your await in particular | 20:49 |
| clayg | those sqlite index updates can be bear 💪 | 20:49 |
| kota_ | morning | 20:57 |
| seongsoocho | kota_: morning | 20:59 |
| kota_ | seongsoocho: o/ | 20:59 |
| timburke | meeting time! | 21:00 |
| *** rcernin has joined #openstack-swift | 22:32 | |
| *** gyee has quit IRC | 22:49 | |
| *** threestrands has joined #openstack-swift | 23:10 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!