| opendevreview | Alistair Coles proposed openstack/swift master: Use Namespaces when that's all we need https://review.opendev.org/c/openstack/swift/+/898244 | 14:52 |
|---|---|---|
| opendevreview | Alistair Coles proposed openstack/swift master: proxy: only use listing shards cache for 'auto' listings https://review.opendev.org/c/openstack/swift/+/901335 | 14:57 |
| timburke | on p 904548 -- what do we think about removing bin/swift from the swiftclient repo? anyone have objections? | 17:29 |
| patch-bot | https://review.opendev.org/c/openstack/python-swiftclient/+/904548 - python-swiftclient - Remove duplicate script entry leading to broken wh... - 1 patch set | 17:29 |
| timburke | #startmeeting swift | 21:00 |
| opendevmeet | Meeting started Wed Jan 10 21:00:14 2024 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| opendevmeet | The meeting name has been set to 'swift' | 21:00 |
| timburke | who's here for the swift team meeting? | 21:00 |
| clayg | this is where all the cool kids party right? | 21:00 |
| timburke | it's a clayg! been a bit :-) | 21:00 |
| mattoliver | o/ | 21:01 |
| jianjian | o/ | 21:01 |
| acoles | o/ | 21:02 |
| timburke | as usual, the agenda's at | 21:02 |
| timburke | #link https://wiki.openstack.org/wiki/Meetings/Swift | 21:02 |
| timburke | though i mostly haven't updated it since last week | 21:02 |
| timburke | #topic py312 support | 21:03 |
| timburke | i said i'd work on that a bit more last week, and i did! | 21:03 |
| timburke | there are a few patches up at https://review.opendev.org/q/project:openstack/swift+topic:py312 | 21:03 |
| timburke | with those, i've got unit tests passing on py312 (i forget whether i'd tried func tests; fairly certain i didn't try probe) | 21:04 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/904600 | 21:04 |
| patch-bot | patch 904600 - swift - Stop using deprecated datetime.utc* functions - 2 patch sets | 21:04 |
| mattoliver | nice work. And nice to see patchbot again :) | 21:05 |
| timburke | was a fun one; prior to that, we'd trip so many deprecations that a full pytest run would cause memory errors omm | 21:05 |
| timburke | i'ts got a pre-req that just refactors some s3api tests | 21:06 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/904599/1 | 21:06 |
| patch-bot | patch 904599 - swift - tests: Switch get_v4_amz_date_header to take timed... - 1 patch set | 21:06 |
| clayg | 🤣 | 21:06 |
| clayg | does the test runner like buffer them all? | 21:06 |
| timburke | apparently? something | 21:07 |
| clayg | or did you just run one file - see the warning - fix it - and then after no more OOM? | 21:07 |
| timburke | after the memory error, i switched to running one file (or at least, one subdir) at a time, then saw a boatload of deprecation warnings coming out of our Timestamp | 21:08 |
| timburke | there were also a handful of failures zigo cataloged at #2046352 | 21:09 |
| patch-bot | https://bugs.launchpad.net/swift/+bug/2046352 - Unit test failure under Python 3.12 (In Progress) | 21:09 |
| clayg | yuk; what a mess | 21:09 |
| timburke | i think i managed to track down the first two to an improvement in sum() | 21:09 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/904601 | 21:10 |
| patch-bot | patch 904601 - swift - tests: Fix float expectations for py312 - 2 patch sets | 21:10 |
| clayg | so, my vsaio doesn't have a py3.12 on it? Do we still have some deadsnakes or ppi plumbing that makes it easy to get py3.12 or did you just compile from source? | 21:10 |
| timburke | yeah, deadsnakes should be able to get you a 3.12 (on jammy, anyway) | 21:10 |
| timburke | i *have* been doing some compiling from source, but that tended to be more around the eventlet woes ;-) | 21:11 |
| clarkb | pyenv simplifies the from source problem a bit | 21:12 |
| indianwhocodes | o/ | 21:12 |
| clayg | `sudo apt-get install python3.12 -y` totally works - wtg deadsnakes | 21:12 |
| timburke | clarkb, thanks for the tip, but if i'm that far down the rabbit hole, i tend to be hopping around between shas ;-) | 21:13 |
| timburke | the last failure was to do with a failure to log about client disconnects, which seemed to ring some bells | 21:14 |
| clayg | https://github.com/pyenv/pyenv?tab=readme-ov-file#install-additional-python-versions | 21:14 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/904652 | 21:14 |
| patch-bot | patch 904652 - swift - Be more explicit about closes in string_along - 1 patch set | 21:14 |
| timburke | seemed to resolve it, though acoles had an idea for another approach in | 21:14 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/905011 | 21:14 |
| patch-bot | patch 905011 - swift - wip: ClosingIterator class - 1 patch set | 21:14 |
| timburke | all that's to say, i think most of those patches are good to go; the disconnects may require a little more thought, though | 21:16 |
| timburke | then we'll figure out what it takes to get a py312 gate job | 21:17 |
| acoles | it's about being more explicit about calling close() - which is how that GeneratorExit handler gets executed - either call close or rely on gc | 21:17 |
| timburke | i do like how the ClosingIterator removes the need for that sync point i added originally | 21:19 |
| timburke | i'll see about getting your follow-up to the point of passing tests -- i forget whether my suggestions would suffice | 21:19 |
| acoles | right. but looks like i broke tests, I'll take another look at it | 21:20 |
| timburke | pretty sure it was a failure to close once the iterator's exhausted | 21:20 |
| timburke | if we like that, i'm happy to use that instead (once tests pass) | 21:21 |
| timburke | next up | 21:21 |
| acoles | +1 your comment about other_iterables - that's the key thing with the string_along - zap the nested iterable too | 21:21 |
| timburke | #topic newest eventlet | 21:21 |
| timburke | i never did hear more about | 21:22 |
| clayg | acoles: timburke: both of the patches look way more complicated than i would hope things should have to be to write idomatic and correct wrapping of iterators that all get properly closed in python/wsgi/eventlet... but of the two i think the class-based impl may be a little easier for me to grok | 21:22 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/904459 | 21:22 |
| patch-bot | patch 904459 - swift - Get tests passing with latest eventlet - 2 patch sets | 21:22 |
| timburke | clayg, yeah, i was coming around on acoles's patch; i'm still said about how hard it is to correctly use generators for a response iter, though | 21:23 |
| timburke | sad, rather | 21:23 |
| clayg | 💯 😭 | 21:24 |
| timburke | on the patch to work with latest eventlet -- do we have any strong feelings about trying harder to make sure eventlet.monkey_patch isn't called in tests? | 21:25 |
| timburke | i could try moving the mock-out up into setup | 21:25 |
| timburke | or we could just land it as-is. i'd definitely prefer it if a fresh dev env wouldn't have segfaulting tests | 21:26 |
| timburke | ref #2047768 | 21:27 |
| patch-bot | https://bugs.launchpad.net/swift/+bug/2047768 - unittests failed with segmentation fault (In Progress) | 21:27 |
| timburke | continuing to wait for https://github.com/eventlet/eventlet/pull/866 to merge & be released is another strategy, i suppose... | 21:28 |
| timburke | all right, i'm'a just merge it | 21:30 |
| timburke | #topic s3api and part-number api | 21:30 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/894570 | 21:30 |
| patch-bot | patch 894570 - swift - slo: part-number=N query parameter support - 85 patch sets | 21:30 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/894580 | 21:30 |
| patch-bot | patch 894580 - swift - s3api: Support GET/HEAD request with ?partNumber - 93 patch sets | 21:30 |
| timburke | is there anything else that needs to be done on these, or are they just awaiting review? | 21:31 |
| mattoliver | Great question, I've been on vactaion, so would also love to know the answer :) | 21:32 |
| acoles | I believe they are ready | 21:32 |
| indianwhocodes | yes they are. | 21:33 |
| acoles | there's two patches: slo support and s3api support | 21:33 |
| acoles | of course, review tends to lead to more needing to be done, but you know what I mean by 'ready' | 21:34 |
| timburke | :-) | 21:34 |
| acoles | we've shipped them into prod | 21:34 |
| mattoliver | kk nice | 21:34 |
| timburke | all right -- who's planning on reviewing with an eye toward dropping a +2? | 21:35 |
| acoles | me | 21:35 |
| acoles | I've co-authored so another set of eyes would be good | 21:36 |
| timburke | sounds good. i'll plan on taking a look this week, too | 21:37 |
| timburke | #topic get_namespaces api | 21:37 |
| acoles | ditto!! | 21:37 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/890470 | 21:37 |
| patch-bot | patch 890470 - swift - Container-server: add container namespaces GET - 57 patch sets | 21:37 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/901335 | 21:38 |
| patch-bot | patch 901335 - swift - proxy: only use listing shards cache for 'auto' li... - 18 patch sets | 21:38 |
| acoles | ready to go, I've already voted | 21:38 |
| acoles | mattoliver: any chance you could swing by them again? think you're blamed for the proxy side :D | 21:38 |
| timburke | oh, there's one more in the middle! | 21:38 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/895602 | 21:38 |
| patch-bot | patch 895602 - swift - Proxy: Use namespaces when getting listing/updatin... - 45 patch sets | 21:38 |
| mattoliver | yes! That was todays plan | 21:38 |
| acoles | p901335 is the icing | 21:39 |
| acoles | patch 901335 is the icing | 21:39 |
| patch-bot | https://review.opendev.org/c/openstack/swift/+/901335 - swift - proxy: only use listing shards cache for 'auto' li... - 18 patch sets | 21:39 |
| mattoliver | I love that one :) | 21:39 |
| timburke | sorry, i think i've got it requiring the space in between... can work on that for next week | 21:39 |
| acoles | mattoliver: thanks | 21:40 |
| timburke | perfect, though. sounds like we've got lots of stuff nearing completion then :D | 21:40 |
| jianjian | I looked 901335 briefly before, will do that again later | 21:40 |
| timburke | one more last-minute topic | 21:41 |
| timburke | #topic swiftclient reviews | 21:42 |
| acoles | the first 2 patches add the new 'feature', the third (901335) tries to unravel the mess we discovered in the proxy container GET path | 21:42 |
| zaitcev | oh, snap. I forgot that swiftclient existed. | 21:42 |
| timburke | there are a few patches we probably ought to take a look at! | 21:42 |
| acoles | zaitcev: me too! | 21:42 |
| timburke | #link https://review.opendev.org/c/openstack/python-swiftclient/+/904548 | 21:42 |
| patch-bot | patch 904548 - python-swiftclient - Remove duplicate script entry leading to broken wh... - 1 patch set | 21:42 |
| timburke | seems to fix some wheel-building woes -- but i wonder if it goes far enough. is there any reason we *shouldn't* just get rid of the bin/swift shim? | 21:43 |
| timburke | #link https://review.opendev.org/c/openstack/python-swiftclient/+/904547 | 21:44 |
| patch-bot | patch 904547 - python-swiftclient - make setup dependencies discoverable - 1 patch set | 21:44 |
| timburke | jumps into the pyproject.toml world (though perhaps too aggressively -- i think various PTI things expect us to still have a setup.py) | 21:45 |
| acoles | timburke: help me grok this comment https://review.opendev.org/c/openstack/python-swiftclient/+/904548/comment/83bc61d9_b976a4d9/ - what could someone NOT do if it was removed? | 21:45 |
| patch-bot | patch 904548 - python-swiftclient - Remove duplicate script entry leading to broken wh... - 1 patch set | 21:45 |
| timburke | clone the repo then run ./bin/swift from the freshly-cloned repo | 21:46 |
| timburke | (at least, i think that's possible today?) | 21:46 |
| acoles | OIC, without installing first? | 21:47 |
| zaitcev | I do that once in a while. But at least for me it also requires PYTHONPATH set. | 21:47 |
| zaitcev | PYTHONPATH=/q/zaitcev/hail/python-swiftclient-tip $PYTHONPATH/bin/swift -A http://rhev-a24c-01.mpc.lab.eng.bos.redhat.com/auth/v1.0 -U test:tester -K testing upload testcont 2009_t3_beatty_v2.avi | 21:48 |
| acoles | whereas the entrypoint creates the same on install | 21:48 |
| zaitcev | no, wait, that's actually what NOT to do | 21:48 |
| zaitcev | PYTHONPATH=/q/zaitcev/hail/python-swiftclient-tip /q/zaitcev/hail/python-swiftclient-tip/bin/swift --os-cacert=/q/zaitcev/arc/CA/certs/cacert.pem -A https://rhev-a24c-01.mpc.lab.eng.bos.redhat.com/auth/v1.0 -U test:tester -K testing stat | 21:49 |
| zaitcev | better | 21:49 |
| zaitcev | Sorry for the noise. But to answer Tim's question: yes. | 21:49 |
| timburke | zaitcev, how do you feel about the `python -m swiftclient.shell` workaround? (though i'm pretty sure it also requires the PYTHONPATH changes) | 21:49 |
| zaitcev | I can adapt to anything, as long it's in doc/sources/*rst somewhere | 21:50 |
| zaitcev | Please don't just drop it in changelog. | 21:50 |
| timburke | all right; so leave it for now, maybe drop in a follow-up, but make sure there's some kind of docs. good feedback 👍 | 21:51 |
| clayg | Is the location of the files in the tree currently *documented* | 21:51 |
| timburke | there are a handful of other patches we could probably stand to review: https://review.opendev.org/q/project:openstack/python-swiftclient+status:open+-age:52w | 21:53 |
| timburke | all right, that's all i've got | 21:53 |
| timburke | #topic open discussion | 21:53 |
| timburke | anything else we ought to bring up in these last few minutes? | 21:54 |
| mattoliver | I've been scratching to do some dev, so been playing with shard replication sync points. p 905064 | 21:54 |
| patch-bot | https://review.opendev.org/c/openstack/swift/+/905064 - swift - wip: shard replication sync points - 1 patch set | 21:54 |
| mattoliver | no where near ready, just thinking about how it'll work.. that might work. need to write tests. | 21:55 |
| mattoliver | Also, I went a head and landed dark data watcher, finally! p 787656 | 21:55 |
| patch-bot | https://review.opendev.org/c/openstack/swift/+/787656 - swift - Make the dark data watcher work with sharded conta... (MERGED) - 14 patch sets | 21:55 |
| timburke | 🎉 | 21:56 |
| mattoliver | Sorry it took so long zaitcev | 21:56 |
| jianjian | nice! | 21:56 |
| timburke | and sorry i didn't do more to help my own patch... | 21:56 |
| zaitcev | np, I actually joined the meeting primarily to thank you | 21:56 |
| jianjian | shard replication sync point is also a good idea | 21:56 |
| mattoliver | lol | 21:56 |
| zaitcev | Still, to think that we have a patch outstanding with 93 revsions! Dunno what I was unhappy about :-) | 21:57 |
| acoles | mattoliver: 🙇 | 21:57 |
| mattoliver | lol, not patchsets time in that case I think :) | 21:57 |
| mattoliver | Currently when we replicate a root container, we replicate ALL shardranges both ways everytime. The patch bacially reuses the incoming/outgoing syncs table to keep track of whats been sent and only send shards that are new. | 21:58 |
| mattoliver | There might be a little jitter as we still send both ways.. but should be better then before.. but like I said, I haven't really tested it enough, that first patchset was getting a first version off my computer :) | 21:59 |
| mattoliver | that's all I got, I think we're almost at time | 22:00 |
| timburke | it'd be interesting to see how often shard ranges are actually changing... maybe we could get some stats out of merge_shard_ranges... | 22:00 |
| timburke | but you're right -- we're at time | 22:01 |
| timburke | thank you all for coming, and thank you for working on swift! | 22:01 |
| timburke | #endmeeting | 22:01 |
| opendevmeet | Meeting ended Wed Jan 10 22:01:20 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 22:01 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/swift/2024/swift.2024-01-10-21.00.html | 22:01 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/swift/2024/swift.2024-01-10-21.00.txt | 22:01 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/swift/2024/swift.2024-01-10-21.00.log.html | 22:01 |
| clayg | I thought there was some fundamental problem with the dark data watcher at scale? Was it just the loaf of HEADS and listings or something with the watcher call back back pressure on the auditor runtime? Or should everyone turn the t on? | 22:01 |
| timburke | pretty sure it was the listing load -- but if you tune down your auditors, it ought to be manageable | 22:02 |
| mattoliver | I added a warning to the doc talking about a possible misplaced edgecase and log should always be used first. But yes is also says it can be expensive. | 22:02 |
| mattoliver | Point was, we already had the dark data watcher landed.. but now its better as it at least supports looking into sharded containers.. which before it didn't. | 22:03 |
| clayg | Oh! Yeah that’s better then 😁 | 22:04 |
| zaitcev | I tune my auditors by looking at how invasive they are, so I didn't have problems as such, but it may be a concern, yes. Fortunately, watchers are completely off by default, and anyone having extra load can switch them off. | 22:08 |
| zaitcev | Nobody uses them for some kind of vital function, do they? | 22:08 |
| opendevreview | Merged openstack/swift master: Get tests passing with latest eventlet https://review.opendev.org/c/openstack/swift/+/904459 | 22:49 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!