| openstackgerrit | Pete Zaitcev proposed openstack/swift master: Let developers/operators add watchers to object audit (simplified) https://review.opendev.org/706653 | 02:56 |
|---|---|---|
| *** psachin has joined #openstack-swift | 03:29 | |
| *** manuvakery has joined #openstack-swift | 03:34 | |
| openstackgerrit | Sam Morrison proposed openstack/swift master: Fix up curl commands when object name has unix dir characters https://review.opendev.org/738147 | 03:41 |
| *** camelCaser has quit IRC | 04:33 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** ccamel has joined #openstack-swift | 04:33 | |
| *** evrardjp has joined #openstack-swift | 04:33 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Identify deleted shards as shards https://review.opendev.org/738113 | 04:33 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Address a sharder/replicator race https://review.opendev.org/738114 | 04:33 |
| openstackgerrit | Tim Burke proposed openstack/swift master: WIP: sharding: Add some more shard-related output to swift-container-info https://review.opendev.org/721377 | 04:33 |
| openstackgerrit | Tim Burke proposed openstack/swift master: wip: Recover from sharding split-brain https://review.opendev.org/738149 | 04:33 |
| zaitcev | 2 more in a patch stack | 04:34 |
| sorrison | timburke: hey! | 04:42 |
| sorrison | thanks for the review, we had a massive crazy issue with our swift and we found this bug when auditing quarantined files | 04:43 |
| timburke | whee! | 04:43 |
| timburke | hope things are fairly recoverable | 04:43 |
| sorrison | yip | 04:44 |
| sorrison | we used eyaml to encrypt our swift_hash and it ended up changing our config from swift_hash_path_suffix = secret to swift_hash_path_suffix = 'secret' | 04:45 |
| sorrison | on 3 out of 4 of our regions.... | 04:45 |
| sorrison | pretty much all data on those hosts got put to quarantine until they ran out of space | 04:46 |
| timburke | 😭 | 04:46 |
| sorrison | we wrote some scripts to put them all back | 04:46 |
| sorrison | been a fun 24 hours | 04:47 |
| timburke | 🤔 maybe upstream could grow some sort of circuit-breaker options, like if there's more than N quarantines in an hour, shut everything down that's touching that disk... or that'd just lead to some *other* fire | 04:50 |
| timburke | anyway, i oughta go to bed. at least i *think* i've got the start of a way to recover from sharding split-brain! | 04:51 |
| sorrison | yeah or I wonder somehow the hash prefix/sufix couldn't be changed somehow | 04:52 |
| sorrison | anyway thanks for the speedy review! | 04:52 |
| *** manuvakery has quit IRC | 05:43 | |
| *** ormandj has quit IRC | 05:51 | |
| *** rpittau|afk is now known as rpittau | 06:29 | |
| openstackgerrit | Merged openstack/swift master: Fix up curl commands when object name has unix dir characters https://review.opendev.org/738147 | 07:46 |
| *** rpittau is now known as rpittau|bbl | 10:04 | |
| *** ccamacho has quit IRC | 10:42 | |
| *** rpittau|bbl is now known as rpittau | 12:12 | |
| *** ormandj has joined #openstack-swift | 12:50 | |
| zaitcev | What in the world are we doing invoking curl?! | 15:28 |
| zaitcev | The whole package is choke-full of HTTP clients and yet this? | 15:29 |
| zaitcev | Oh, wait. LOL okay | 15:31 |
| clayg | zaitcev: hahah ❤️ | 15:33 |
| clayg | that was weird, when I checked out p 735738 tests were passing for me - but I was able to duplicate the failure after a *rebase* | 15:49 |
| patchbot | https://review.opendev.org/#/c/735738/ - swift - s3api: Don't do naive HEAD request for auth - 3 patch sets | 15:49 |
| clayg | wild | 15:49 |
| *** rpittau is now known as rpittau|afk | 15:59 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: s3api: Don't do naive HEAD request for auth https://review.opendev.org/735738 | 16:06 |
| timburke | clayg, new test from p 733919? | 16:11 |
| patchbot | https://review.opendev.org/#/c/733919/ - swift - s3api: Allow CompleteMultipartUpload requests to b... (MERGED) - 5 patch sets | 16:11 |
| clayg | yessir 👍 | 16:11 |
| *** psachin has quit IRC | 16:18 | |
| *** gyee has joined #openstack-swift | 16:35 | |
| *** ndk_ has joined #openstack-swift | 18:03 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Move jobs from bionic to focal https://review.opendev.org/738269 | 19:45 |
| clayg | swift-get-nodes -Q is the greatest thing ever invented | 20:05 |
| clayg | 😍 | 20:05 |
| ormandj | clayg: fwiw, it seems the limitation on throughput is client-side - we can't get aws s3 (cli) to move more than about 120-130MB/s to amazon s3, wasabi, or anything else | 20:41 |
| ormandj | even with a concurrency of 20/20M chunks | 20:42 |
| ormandj | we've even tried from SSD, no dice, i'm wondering if it might be related to SSL/encryption | 20:42 |
| clayg | ormandj: that sounds frustrating 😞 | 21:01 |
| ormandj | very, have you seen better out of any clients? | 21:02 |
| ormandj | we're open to other client suggestions, we just haven't been able to find one that can push more. rclone was far worse | 21:02 |
| *** paladox has quit IRC | 21:33 | |
| *** paladox has joined #openstack-swift | 21:37 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Allow direct and internal clients to use the replication network https://review.opendev.org/735751 | 23:47 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!