21:00:14 <timburke> #startmeeting swift
21:00:14 <opendevmeet> Meeting started Wed Jan 10 21:00:14 2024 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:14 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:14 <opendevmeet> The meeting name has been set to 'swift'
21:00:22 <timburke> who's here for the swift team meeting?
21:00:31 <clayg> this is where all the cool kids party right?
21:00:51 <timburke> it's a clayg! been a bit :-)
21:01:01 <mattoliver> o/
21:01:39 <jianjian> o/
21:02:10 <acoles> o/
21:02:19 <timburke> as usual, the agenda's at
21:02:21 <timburke> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:02:34 <timburke> though i mostly haven't updated it since last week
21:03:12 <timburke> #topic py312 support
21:03:26 <timburke> i said i'd work on that a bit more last week, and i did!
21:03:43 <timburke> there are a few patches up at https://review.opendev.org/q/project:openstack/swift+topic:py312
21:04:40 <timburke> with those, i've got unit tests passing on py312 (i forget whether i'd tried func tests; fairly certain i didn't try probe)
21:04:57 <timburke> #link https://review.opendev.org/c/openstack/swift/+/904600
21:04:58 <patch-bot> patch 904600 - swift - Stop using deprecated datetime.utc* functions - 2 patch sets
21:05:20 <mattoliver> nice work. And nice to see patchbot again :)
21:05:38 <timburke> was a fun one; prior to that, we'd trip so many deprecations that a full pytest run would cause memory errors omm
21:06:07 <timburke> i'ts got a pre-req that just refactors some s3api tests
21:06:09 <timburke> #link https://review.opendev.org/c/openstack/swift/+/904599/1
21:06:10 <patch-bot> patch 904599 - swift - tests: Switch get_v4_amz_date_header to take timed... - 1 patch set
21:06:19 <clayg> 🤣
21:06:38 <clayg> does the test runner like buffer them all?
21:07:09 <timburke> apparently? something
21:07:13 <clayg> or did you just run one file - see the warning - fix it - and then after no more OOM?
21:08:37 <timburke> after the memory error, i switched to running one file (or at least, one subdir) at a time, then saw a boatload of deprecation warnings coming out of our Timestamp
21:09:11 <timburke> there were also a handful of failures zigo cataloged at #2046352
21:09:12 <patch-bot> https://bugs.launchpad.net/swift/+bug/2046352 - Unit test failure under Python 3.12 (In Progress)
21:09:37 <clayg> yuk; what a mess
21:09:55 <timburke> i think i managed to track down the first two to an improvement in sum()
21:10:08 <timburke> #link https://review.opendev.org/c/openstack/swift/+/904601
21:10:08 <patch-bot> patch 904601 - swift - tests: Fix float expectations for py312 - 2 patch sets
21:10:24 <clayg> so, my vsaio doesn't have a py3.12 on it?  Do we still have some deadsnakes or ppi plumbing that makes it easy to get py3.12 or did you just compile from source?
21:10:59 <timburke> yeah, deadsnakes should be able to get you a 3.12 (on jammy, anyway)
21:11:44 <timburke> i *have* been doing some compiling from source, but that tended to be more around the eventlet woes ;-)
21:12:08 <clarkb> pyenv simplifies the from source problem a bit
21:12:32 <indianwhocodes> o/
21:12:38 <clayg> `sudo apt-get install python3.12 -y` totally works - wtg deadsnakes
21:13:15 <timburke> clarkb, thanks for the tip, but if i'm that far down the rabbit hole, i tend to be hopping around between shas ;-)
21:14:10 <timburke> the last failure was to do with a failure to log about client disconnects, which seemed to ring some bells
21:14:10 <clayg> https://github.com/pyenv/pyenv?tab=readme-ov-file#install-additional-python-versions
21:14:30 <timburke> #link https://review.opendev.org/c/openstack/swift/+/904652
21:14:31 <patch-bot> patch 904652 - swift - Be more explicit about closes in string_along - 1 patch set
21:14:51 <timburke> seemed to resolve it, though acoles had an idea for another approach in
21:14:58 <timburke> #link https://review.opendev.org/c/openstack/swift/+/905011
21:14:58 <patch-bot> patch 905011 - swift - wip: ClosingIterator class - 1 patch set
21:16:38 <timburke> all that's to say, i think most of those patches are good to go; the disconnects may require a little more thought, though
21:17:04 <timburke> then we'll figure out what it takes to get a py312 gate job
21:17:22 <acoles> it's about being more explicit about calling close() - which is how that GeneratorExit handler gets executed - either call close or rely on gc
21:19:08 <timburke> i do like how the ClosingIterator removes the need for that sync point i added originally
21:19:54 <timburke> i'll see about getting your follow-up to the point of passing tests -- i forget whether my suggestions would suffice
21:20:03 <acoles> right. but looks like i broke tests, I'll take another look at it
21:20:34 <timburke> pretty sure it was a failure to close once the iterator's exhausted
21:21:11 <timburke> if we like that, i'm happy to use that instead (once tests pass)
21:21:36 <timburke> next up
21:21:37 <acoles> +1 your comment about other_iterables - that's the key thing with the string_along - zap the nested iterable too
21:21:53 <timburke> #topic newest eventlet
21:22:08 <timburke> i never did hear more about
21:22:08 <clayg> acoles: timburke: both of the patches look way more complicated than i would hope things should have to be to write idomatic and correct wrapping of iterators that all get properly closed in python/wsgi/eventlet... but of the two i think the class-based impl may be a little easier for me to grok
21:22:18 <timburke> #link https://review.opendev.org/c/openstack/swift/+/904459
21:22:18 <patch-bot> patch 904459 - swift - Get tests passing with latest eventlet - 2 patch sets
21:23:32 <timburke> clayg, yeah, i was coming around on acoles's patch; i'm still said about how hard it is to correctly use generators for a response iter, though
21:23:40 <timburke> sad, rather
21:24:04 <clayg> 💯 😭
21:25:28 <timburke> on the patch to work with latest eventlet -- do we have any strong feelings about trying harder to make sure eventlet.monkey_patch isn't called in tests?
21:25:58 <timburke> i could try moving the mock-out up into setup
21:26:49 <timburke> or we could just land it as-is. i'd definitely prefer it if a fresh dev env wouldn't have segfaulting tests
21:27:01 <timburke> ref #2047768
21:27:02 <patch-bot> https://bugs.launchpad.net/swift/+bug/2047768 - unittests failed with segmentation fault (In Progress)
21:28:29 <timburke> continuing to wait for https://github.com/eventlet/eventlet/pull/866 to merge & be released is another strategy, i suppose...
21:30:01 <timburke> all right, i'm'a just merge it
21:30:13 <timburke> #topic s3api and part-number api
21:30:21 <timburke> #link https://review.opendev.org/c/openstack/swift/+/894570
21:30:22 <patch-bot> patch 894570 - swift - slo: part-number=N query parameter support - 85 patch sets
21:30:28 <timburke> #link https://review.opendev.org/c/openstack/swift/+/894580
21:30:29 <patch-bot> patch 894580 - swift - s3api: Support GET/HEAD request with ?partNumber - 93 patch sets
21:31:00 <timburke> is there anything else that needs to be done on these, or are they just awaiting review?
21:32:53 <mattoliver> Great question, I've been on vactaion, so would also love to know the answer :)
21:32:54 <acoles> I believe they are ready
21:33:11 <indianwhocodes> yes they are.
21:33:19 <acoles> there's two patches: slo support and s3api support
21:34:07 <acoles> of course, review tends to lead to more needing to be done, but you know what I mean by 'ready'
21:34:15 <timburke> :-)
21:34:28 <acoles> we've shipped them into prod
21:34:55 <mattoliver> kk nice
21:35:23 <timburke> all right -- who's planning on reviewing with an eye toward dropping a +2?
21:35:40 <acoles> me
21:36:34 <acoles> I've co-authored so another set of eyes would be good
21:37:00 <timburke> sounds good. i'll plan on taking a look this week, too
21:37:26 <timburke> #topic get_namespaces api
21:37:42 <acoles> ditto!!
21:37:54 <timburke> #link https://review.opendev.org/c/openstack/swift/+/890470
21:37:55 <patch-bot> patch 890470 - swift - Container-server: add container namespaces GET - 57 patch sets
21:38:04 <timburke> #link https://review.opendev.org/c/openstack/swift/+/901335
21:38:04 <patch-bot> patch 901335 - swift - proxy: only use listing shards cache for 'auto' li... - 18 patch sets
21:38:09 <acoles> ready to go, I've already voted
21:38:30 <acoles> mattoliver: any chance you could swing by them again? think you're blamed for the proxy side :D
21:38:31 <timburke> oh, there's one more in the middle!
21:38:34 <timburke> #link https://review.opendev.org/c/openstack/swift/+/895602
21:38:35 <patch-bot> patch 895602 - swift - Proxy: Use namespaces when getting listing/updatin... - 45 patch sets
21:38:45 <mattoliver> yes! That was todays plan
21:39:09 <acoles> p901335 is the icing
21:39:18 <acoles> patch 901335 is the icing
21:39:18 <patch-bot> https://review.opendev.org/c/openstack/swift/+/901335 - swift - proxy: only use listing shards cache for 'auto' li... - 18 patch sets
21:39:38 <mattoliver> I love that one :)
21:39:48 <timburke> sorry, i think i've got it requiring the space in between... can work on that for next week
21:40:05 <acoles> mattoliver: thanks
21:40:35 <timburke> perfect, though. sounds like we've got lots of stuff nearing completion then :D
21:40:44 <jianjian> I looked 901335 briefly before, will do that again later
21:41:52 <timburke> one more last-minute topic
21:42:00 <timburke> #topic swiftclient reviews
21:42:02 <acoles> the first 2 patches add the new 'feature', the third (901335) tries to unravel the mess we discovered in the proxy container GET path
21:42:16 <zaitcev> oh, snap. I forgot that swiftclient existed.
21:42:19 <timburke> there are a few patches we probably ought to take a look at!
21:42:32 <acoles> zaitcev: me too!
21:42:48 <timburke> #link https://review.opendev.org/c/openstack/python-swiftclient/+/904548
21:42:48 <patch-bot> patch 904548 - python-swiftclient - Remove duplicate script entry leading to broken wh... - 1 patch set
21:43:36 <timburke> seems to fix some wheel-building woes -- but i wonder if it goes far enough. is there any reason we *shouldn't* just get rid of the bin/swift shim?
21:44:19 <timburke> #link https://review.opendev.org/c/openstack/python-swiftclient/+/904547
21:44:19 <patch-bot> patch 904547 - python-swiftclient - make setup dependencies discoverable - 1 patch set
21:45:03 <timburke> jumps into the pyproject.toml world (though perhaps too aggressively -- i think various PTI things expect us to still have a setup.py)
21:45:18 <acoles> timburke: help me grok this comment https://review.opendev.org/c/openstack/python-swiftclient/+/904548/comment/83bc61d9_b976a4d9/ - what could someone NOT do if it was removed?
21:45:18 <patch-bot> patch 904548 - python-swiftclient - Remove duplicate script entry leading to broken wh... - 1 patch set
21:46:06 <timburke> clone the repo then run ./bin/swift from the freshly-cloned repo
21:46:24 <timburke> (at least, i think that's possible today?)
21:47:15 <acoles> OIC, without installing first?
21:47:18 <zaitcev> I do that once in a while. But at least for me it also requires PYTHONPATH set.
21:48:06 <zaitcev> PYTHONPATH=/q/zaitcev/hail/python-swiftclient-tip $PYTHONPATH/bin/swift -A http://rhev-a24c-01.mpc.lab.eng.bos.redhat.com/auth/v1.0 -U test:tester -K testing upload testcont 2009_t3_beatty_v2.avi
21:48:16 <acoles> whereas the entrypoint creates the same on install
21:48:49 <zaitcev> no, wait, that's actually what NOT to do
21:49:16 <zaitcev> PYTHONPATH=/q/zaitcev/hail/python-swiftclient-tip /q/zaitcev/hail/python-swiftclient-tip/bin/swift --os-cacert=/q/zaitcev/arc/CA/certs/cacert.pem -A https://rhev-a24c-01.mpc.lab.eng.bos.redhat.com/auth/v1.0 -U test:tester -K testing stat
21:49:22 <zaitcev> better
21:49:36 <zaitcev> Sorry for the noise. But to answer Tim's question: yes.
21:49:37 <timburke> zaitcev, how do you feel about the `python -m swiftclient.shell` workaround? (though i'm pretty sure it also requires the PYTHONPATH changes)
21:50:14 <zaitcev> I can adapt to anything, as long it's in doc/sources/*rst somewhere
21:50:30 <zaitcev> Please don't just drop it in changelog.
21:51:13 <timburke> all right; so leave it for now, maybe drop in a follow-up, but make sure there's some kind of docs. good feedback 👍
21:51:14 <clayg> Is the location of the files in the tree currently *documented*
21:53:04 <timburke> there are a handful of other patches we could probably stand to review: https://review.opendev.org/q/project:openstack/python-swiftclient+status:open+-age:52w
21:53:48 <timburke> all right, that's all i've got
21:53:53 <timburke> #topic open discussion
21:54:06 <timburke> anything else we ought to bring up in these last few minutes?
21:54:45 <mattoliver> I've been scratching to do some dev, so been playing with shard replication sync points. p 905064
21:54:46 <patch-bot> https://review.opendev.org/c/openstack/swift/+/905064 - swift - wip: shard replication sync points - 1 patch set
21:55:03 <mattoliver> no where near ready, just thinking about how it'll work.. that might work. need to write tests.
21:55:53 <mattoliver> Also, I went a head and landed dark data watcher, finally! p 787656
21:55:53 <patch-bot> https://review.opendev.org/c/openstack/swift/+/787656 - swift - Make the dark data watcher work with sharded conta... (MERGED) - 14 patch sets
21:56:08 <timburke> 🎉
21:56:10 <mattoliver> Sorry it took so long zaitcev
21:56:18 <jianjian> nice!
21:56:24 <timburke> and sorry i didn't do more to help my own patch...
21:56:29 <zaitcev> np, I actually joined the meeting primarily to thank you
21:56:30 <jianjian> shard replication sync point is also a good idea
21:56:32 <mattoliver> lol
21:57:13 <zaitcev> Still, to think that we have a patch outstanding with 93 revsions! Dunno what I was unhappy about :-)
21:57:14 <acoles> mattoliver:  🙇
21:57:41 <mattoliver> lol, not patchsets time in that case I think :)
21:58:41 <mattoliver> Currently when we replicate a root container, we replicate ALL shardranges both ways everytime. The patch bacially reuses the incoming/outgoing syncs table to keep track of whats been sent and only send shards that are new.
21:59:34 <mattoliver> There might be a little jitter as we still send both ways.. but should be better then before.. but like I said, I haven't really tested it enough, that first patchset was getting a first version off my computer :)
22:00:09 <mattoliver> that's all I got, I think we're almost at time
22:00:58 <timburke> it'd be interesting to see how often shard ranges are actually changing... maybe we could get some stats out of merge_shard_ranges...
22:01:07 <timburke> but you're right -- we're at time
22:01:17 <timburke> thank you all for coming, and thank you for working on swift!
22:01:20 <timburke> #endmeeting