| mattoliverau | oh nice tip :) | 00:02 |
|---|---|---|
| *** baojg has quit IRC | 00:09 | |
| *** baojg has joined #openstack-swift | 00:10 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: proxy: Put storage policy index in object responses https://review.opendev.org/749400 | 00:13 |
| openstackgerrit | Tim Burke proposed openstack/swift master: s3api: Ensure backend headers make it through s3api https://review.opendev.org/749401 | 00:13 |
| renich | mattoliverau: thanks ;D | 00:26 |
| renich | btw, in which branch should I commit my proposed doc changes? I'm editing the Apache Deployment Guide. | 00:26 |
| * renich couldn't find a documentation branch. | 00:26 | |
| mattoliverau | because your going to use gerrit, you can create your own branch for you changes, but just git review it up. It'll end up in master, which is fine.. but only once it's been approved. | 00:27 |
| mattoliverau | own branch for your own sanity. | 00:28 |
| renich | OK, thanks a lot mattoliverau | 00:38 |
| zaitcev | I do not use branches functionally. I only clone, clone, clone. It costs almost nothing. | 00:38 |
| zaitcev | I do checkout -b xxx, but only because it sets the topic in Gerrit. | 00:38 |
| zaitcev | It's a habit that is not vital in Swift, because we don't have anything compiled. But in other projects, such as Ceph, it can take several hours to recover from switching a branch. | 00:39 |
| zaitcev | If you clone, compiled artifacts match the checked out code. | 00:40 |
| zaitcev | Well, some people work around it with distcc | 00:40 |
| *** renich has quit IRC | 00:42 | |
| *** renich has joined #openstack-swift | 00:43 | |
| renich | zaitcev: oh, interesting! Maybe just have a workstation with 1 TiB of RAM and put everything on tmpfs? ;D | 00:44 |
| renich | ... With some ryzen EPIC II processors... | 00:45 |
| *** rcernin_ has joined #openstack-swift | 00:45 | |
| *** rcernin has quit IRC | 00:47 | |
| *** xiaolin has joined #openstack-swift | 01:09 | |
| *** renich has quit IRC | 01:38 | |
| *** xiaolin has quit IRC | 01:50 | |
| *** renich has joined #openstack-swift | 01:55 | |
| *** renich has quit IRC | 02:14 | |
| *** manuvakery has joined #openstack-swift | 02:19 | |
| *** renich has joined #openstack-swift | 02:30 | |
| *** baojg has quit IRC | 02:32 | |
| *** baojg has joined #openstack-swift | 02:33 | |
| *** josephillips has quit IRC | 02:37 | |
| openstackgerrit | Merged openstack/swift stable/train: py3: Work with proper native string paths in crypto meta https://review.opendev.org/748883 | 02:57 |
| *** renich has quit IRC | 03:03 | |
| *** rcernin_ has quit IRC | 03:19 | |
| *** rcernin_ has joined #openstack-swift | 03:34 | |
| *** psachin has joined #openstack-swift | 03:35 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: ec: Add an option to write fragments with legacy crc https://review.opendev.org/739164 | 03:59 |
| *** mahatic has joined #openstack-swift | 04:27 | |
| *** ChanServ sets mode: +v mahatic | 04:27 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-swift | 04:33 | |
| *** manuvakery has quit IRC | 04:39 | |
| *** m75abrams has joined #openstack-swift | 04:44 | |
| *** viks____ has joined #openstack-swift | 04:59 | |
| *** gyee has quit IRC | 05:07 | |
| *** baojg has quit IRC | 05:12 | |
| *** baojg has joined #openstack-swift | 05:13 | |
| mattoliverau | clayg, timburke: I'll start by saying haven't actaully tested this code yet, it's very rough.. but just brainstorm writing something that might handle gaps and over lapping ranges. And wanted to put it somewhere. Plan to test and debug and see how it goes, but running out of time today (need to do some payed work) but wanted something up before the meeting tomorrow, so it doesn't look like I've been ingoring the | 05:26 |
| mattoliverau | shrinking / automated sharding problems :) | 05:26 |
| mattoliverau | just a version 0.1. | 05:27 |
| mattoliverau | oh the code: https://gist.github.com/matthewoliver/bb52d5ea6f125064d26535223141b68a | 05:27 |
| mattoliverau | It scans a list of ranges, and puts them into either complete ranges or fragment ranges. The rest of the code, which is documented by brain dump comments, then deal with finding either the best complete path using a weighing or by building up a complete path from the fragments (if we have no complete path). | 05:29 |
| mattoliverau | Jsut wanted to get something iterate on and see if we can get something that get get us out of trouble, and start off as simple as possible | 05:30 |
| mattoliverau | the idea would be, the best path will return the best path and a list of left over ranges. In theory we should be able to insert the best path into all the bad ranges and they should cleave themselves to the correct locations and delete themselves. | 05:31 |
| mattoliverau | I should put this all in an etherpad. And once cleaned up in a WIP patch. I'll get that done before PTG ;) | 05:32 |
| *** viks____ has quit IRC | 06:01 | |
| *** baojg has quit IRC | 06:02 | |
| *** baojg has joined #openstack-swift | 06:03 | |
| *** viks____ has joined #openstack-swift | 06:04 | |
| openstackgerrit | Merged openstack/swift stable/ussuri: py3: Work with proper native string paths in crypto meta https://review.opendev.org/748882 | 06:29 |
| *** rcernin_ has quit IRC | 06:51 | |
| *** rcernin_ has joined #openstack-swift | 06:54 | |
| *** manuvakery has joined #openstack-swift | 06:57 | |
| *** rcernin_ has quit IRC | 07:35 | |
| zigo | timburke: On our production, we have 6 swift proxies with a round-robbin DNS. Would it be fine if we first remove 3 proxies from round-robbin, upgrade them to latest Ussuri + patch on py3, then turn off the 3 old py2 proxies and switch to the upgraded ones? | 07:42 |
| zigo | Would that be a procedure that would work? | 07:42 |
| zigo | Or just wait for the patch release, then it's ok to mix both? Do I understand correctly? | 07:44 |
| zigo | FYI, in Debian, I already uploaded Ussuri to Debian Unstable with the patch. | 07:45 |
| zigo | And also rebuilt the backport to Buster. | 07:45 |
| *** baojg has quit IRC | 07:53 | |
| *** baojg has joined #openstack-swift | 07:53 | |
| *** baojg has quit IRC | 07:59 | |
| *** rcernin_ has joined #openstack-swift | 08:40 | |
| *** rcernin_ has quit IRC | 08:46 | |
| *** baojg has joined #openstack-swift | 09:28 | |
| *** baojg has quit IRC | 09:54 | |
| kota_ | timburke: sorry to be late to respond to the liberasurecode. I added my current idea just before. | 10:02 |
| *** mikecmpbll has joined #openstack-swift | 10:12 | |
| *** m75abrams has quit IRC | 13:13 | |
| *** m75abrams has joined #openstack-swift | 13:15 | |
| clayg | zigo: if you wait for the patch release and upgrade py2 to the latest version before you upgrade to the new swift w/ py3 you can mix just fine - everyone will understand each others v2 formats (you could even switch over to v3 after to patched swift on py2 - then you're REALLY golden when upgrading to py3) | 13:57 |
| zigo | clayg: I cannot upgrade to the latest version using py2, because I don't have such packages available. | 13:57 |
| zigo | Starting with Stein, we moved all to Py3 if I'm not mistaking. | 13:58 |
| clayg | zigo: hopefully tim will do some backporting soon! will that help? | 13:58 |
| clayg | if you're already on py3 upgrading is smooth - perhaps I'm confused | 13:59 |
| zigo | clayg: If I understand correctly, I should add the patch to my Rocky deployment, and set v1 everywhere, then upgrade to Ussuri (directly) which also switches to Py3, then when done, I'm fine increasing to v2 (or v3?). | 13:59 |
| zigo | Right? | 13:59 |
| zigo | clayg: I'm just very much confused on what would be the workflow ... | 14:00 |
| clayg | after the upgrade you'll want to go to v3 | 14:00 |
| *** m75abrams has quit IRC | 14:01 | |
| zigo | clayg: Even with Ussuri? | 14:01 |
| clayg | old unpatched py2 code won't be able to read any encrypted data with non-ascii chars written by a py3 swift | 14:02 |
| clayg | zigo: i'd have to translate named openstack releases to swift commits or versions to answer that - sorry - tim's better at doing those in his head | 14:02 |
| clayg | i believe tim intends to packport the fix such that old py2 versions can update to new code first - then the update across python versions is smooth. perhaps you're suggesting we want to backport the fix further? | 14:04 |
| zigo | clayg: But isn't what the v1 / v2 / v3 thing is for? | 14:06 |
| zigo | Or is this unrelated to py2 not being able to read what py3 writes? | 14:07 |
| clayg | it's mostly unrelated - the ability to continue to write v2 even after you upgrade is what makes rolling upgrades smooth | 14:08 |
| zigo | Ok, got it. | 14:08 |
| zigo | So basically, the workflow would be: switch all proxies to py3 at once then... :) | 14:09 |
| clayg | sure, if you can switch all the proxies over at once (stop the world upgrade) you can dodge all of this mess | 14:10 |
| zigo | That's possible for us, as we're using haproxy we can do the trick and redirect to the newly upgrade proxies at once! :) | 14:11 |
| zigo | I just need to figure out what will be the smoothest way, but that's doable. | 14:11 |
| zigo | Thanks for the explanations. | 14:11 |
| zigo | BTW, I wrote something wrong: I believe I switched Swift to Python 3 starting with Train in Debian. | 14:12 |
| zigo | This matches the release of Debian Buster: I did the switch in the packages the summer after Buster was release, and that's IMO (without looking) Train. | 14:12 |
| zigo | Each of our 6 swiftproxy are behind haproxies, each on a roundrobbin DNS. So we would: | 14:17 |
| zigo | 1/ remove 3 proxies from DNS and wait | 14:17 |
| zigo | 2/ upgrade these 3 proxies | 14:17 |
| zigo | 3/ do the switch on the 3 not-upgrade server, so that haproxy points to the upgrade proxies | 14:17 |
| zigo | 4/ make the DNS point to the 3 upgraded proxies | 14:17 |
| zigo | 5/ upgrade the last 3 proxies | 14:17 |
| zigo | 6/ re-add the last 3 proxies to DNS | 14:17 |
| zigo | Just point 3 must be done "at once", there's a tiny small risk of race condition but that's ridiculous ... | 14:17 |
| ormandj | heya folks, is it still true that servers_per_port won't work with a separate replication network? : https://bugs.launchpad.net/swift/+bug/1669579 and corresponding: https://review.opendev.org/#/c/337861/ | 14:39 |
| openstack | Launchpad bug 1669579 in OpenStack Object Storage (swift) "servers_per_port will not bind to replication_port" [Medium,In progress] - Assigned to Romain LE DISEZ (rledisez) | 14:39 |
| patchbot | patch 337861 - swift - Permit to bind object-server on replication_port - 7 patch sets | 14:39 |
| timburke | good morning | 14:56 |
| timburke | zigo, i think that upgrade plan seems reasonable | 14:57 |
| ormandj | morning tim | 14:57 |
| timburke | train moving to py3 makes sense -- that was the first release with py3 support | 14:58 |
| timburke | the interest in rocky is good feedback -- i'll see how well backporting this that far goes | 14:58 |
| timburke | ormandj, servers_per_port and replication ports seem likely to still not play well together -- i don't know that i've tried it out though. i *did* get probe tests running with separate replication ports & servers (see p 741723) but i don't think i did any testing with servers_per_port at that time | 15:02 |
| patchbot | https://review.opendev.org/#/c/741723/ - swift - wip: Allow probe tests to run with separate replic... - 4 patch sets | 15:02 |
| timburke | i sould go look at how we (nvidia) deploy that... i'm fairly certain we're running servers_per_port, so i think if we've got separate replication/proxy networks, they must be split by ip rather than port | 15:07 |
| openstackgerrit | Tim Burke proposed openstack/swift stable/stein: py3: Work with proper native string paths in crypto meta https://review.opendev.org/749532 | 15:10 |
| clayg | timburke: we don't use servers_per_port in the 2.conf (replication server are different ip and just one replication port with N workers) | 15:23 |
| openstackgerrit | Tim Burke proposed openstack/swift stable/rocky: py3: Work with proper native string paths in crypto meta https://review.opendev.org/749536 | 15:24 |
| timburke | zigo: rocky wasn't too bad; conflicts were just because we hadn't ported encryption to py3 yet | 15:26 |
| ormandj | timburke: like ip per osd thing? | 15:35 |
| ormandj | because if this is a limitation, that's not great, does nobody use independent replication networks in production? since it seems like servers_per_port is a necessity | 15:36 |
| timburke | ormandj, i was thinking that we may well do servers_per_port for our replication servers, but with ports matching between replication and proxy networks and using bind_ip to differentiate them. that *may* still work ok? but clayg points out that, no, we do servers_per_port for proxy traffic but not replication | 15:43 |
| timburke | servers_per_port is less important for replication networks -- those slowing down shouldn't really impact end-users | 15:43 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Make stubs more realistic https://review.opendev.org/749541 | 15:48 |
| ormandj | timburke: ahh, ok. | 16:01 |
| clayg | timburke: quick glance at p 749382 - I think the refactor is totally worth it - even just branching out to per-resource _swift_*_error_handler is a big simplification | 16:22 |
| patchbot | https://review.opendev.org/#/c/749382/ - swift - s3api: Make quota-exceeded errors more obvious - 2 patch sets | 16:22 |
| clayg | I don't understand the KeyError interface - who is catching that that? | 16:23 |
| timburke | no body, as best i can tell! but i knew it couldn't be any *worse* than what we had before | 16:23 |
| clayg | oic | 16:24 |
| timburke | i was just about to ask for your and kota_'s opinions on that patch in here; you beat me to it ;-) | 16:24 |
| clayg | perhaps _swift_<resource>_<method>_error_handler? you could do a getattr or even `{'HEAD': self._swift_container_HEAD_error_handler, ... }[req.method]` kinda thing... | 16:26 |
| clayg | I'd be happy to play with after I finish a few other things, not sure what you have prioritized this morning | 16:26 |
| *** dsariel has quit IRC | 16:28 | |
| *** theintern_ has joined #openstack-swift | 16:29 | |
| *** theintern_ has quit IRC | 16:30 | |
| ormandj | clayg: THANK YOU :) re: quota | 16:33 |
| ormandj | that one has been a massive PITA for us | 16:33 |
| ormandj | so many clients have an enum for human-readable error messages, and they get back "ntp is out of sync" and various other things for a quota issue | 16:33 |
| ormandj | also, why are so many clients terrible? :) | 16:34 |
| clayg | ormandj: oh, that's all Tim - but ok, I'll try to get that looked at and merged asap | 16:36 |
| ormandj | clayg: well, i already tell tim thank you enough, so it's your turn :p | 16:37 |
| *** mikecmpbll has quit IRC | 16:50 | |
| *** psachin has quit IRC | 17:09 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Make all concurrent_get options per-policy https://review.opendev.org/737096 | 17:13 |
| *** josephillips has joined #openstack-swift | 17:15 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Client should retry when there's just one 404 and a bunch of errors https://review.opendev.org/744942 | 18:02 |
| openstackgerrit | Tim Burke proposed openstack/swift master: proxy: Put storage policy index in object responses https://review.opendev.org/749400 | 18:02 |
| openstackgerrit | Tim Burke proposed openstack/swift master: s3api: Ensure backend headers make it through s3api https://review.opendev.org/749401 | 18:02 |
| *** manuvakery has quit IRC | 18:05 | |
| *** renich has joined #openstack-swift | 18:09 | |
| *** mikecmpbll has joined #openstack-swift | 18:30 | |
| *** mikecmpbll has quit IRC | 18:31 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: proxy: Include thread_locals when spawning _fragment_GET_request https://review.opendev.org/749376 | 18:34 |
| *** renich has quit IRC | 18:45 | |
| timburke | clayg, i feel like *usually* if we're doing logging assertions we want to be looking at what's going out to syslog :-/ iirc that was at least half the reason i tried my hand at https://review.opendev.org/#/c/496535/ | 18:46 |
| patchbot | patch 496535 - swift - Simplify testing for logging at error vs exception - 2 patch sets | 18:46 |
| clayg | it's hard to say, there's a lot of BS going into those formatted log lines - I considered just asserting the txnid was in the captured record from the log_dict and thought the other was cuter - we could try to unify everything eventually | 18:48 |
| *** renich has joined #openstack-swift | 19:03 | |
| clayg | shoot - missed the other caller - thanks @timburke | 19:09 |
| timburke | clayg, should we go ahead and merge p 715492? how about p 702783? (was just reminded of them while pruning some old branches) | 19:10 |
| patchbot | https://review.opendev.org/#/c/715492/ - swift - s3api: Better logging for non-JSON when trying to ... - 2 patch sets | 19:10 |
| patchbot | https://review.opendev.org/#/c/702783/ - swift - container-sync: Stop continuing to update sync poi... - 2 patch sets | 19:10 |
| timburke | clayg, want me to take on fixing up the txn id patch? | 19:13 |
| clayg | yeah! thanks | 19:14 |
| clayg | you could probably find tests that already hit all the various spawn points with error conditions that should demonstrate the lack of the tnx-id | 19:15 |
| openstackgerrit | Tim Burke proposed openstack/swift master: proxy: Include thread_locals when spawning _fragment_GET_request https://review.opendev.org/749376 | 19:48 |
| openstackgerrit | Tim Burke proposed openstack/swift master: proxy: Include thread_locals when spawning _fragment_GET_request https://review.opendev.org/749376 | 19:57 |
| *** gyee has joined #openstack-swift | 20:17 | |
| timburke | almost meeting time! | 20:52 |
| clayg | can't wait!! 🎉 | 20:55 |
| * mattoliverau is also cooking pancakes at the same time as meeting (Zoe is 2 today! Time flies). | 21:02 | |
| timburke | \o/ happy birthday to her! and congrats to you, too -- they're a handful at this age ;-) | 21:03 |
| *** rcernin_ has joined #openstack-swift | 21:06 | |
| *** rcernin_ has quit IRC | 21:12 | |
| *** paladox has quit IRC | 21:59 | |
| *** timss has quit IRC | 21:59 | |
| *** paladox has joined #openstack-swift | 22:00 | |
| *** timss has joined #openstack-swift | 22:01 | |
| *** djhankb has quit IRC | 22:05 | |
| *** djhankb has joined #openstack-swift | 22:06 | |
| *** rcernin_ has joined #openstack-swift | 22:16 | |
| *** rcernin_ has quit IRC | 23:04 | |
| *** rcernin has joined #openstack-swift | 23:04 | |
| *** lifeless has quit IRC | 23:29 | |
| *** irclogbot_1 has quit IRC | 23:29 | |
| *** irclogbot_2 has joined #openstack-swift | 23:33 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!