21:00:10 <timburke> #startmeeting swift
21:00:10 <opendevmeet> Meeting started Wed May 17 21:00:10 2023 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:10 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:10 <opendevmeet> The meeting name has been set to 'swift'
21:00:17 <timburke> who's here for the swift meeting?
21:00:33 <mattoliver> o/
21:01:12 <acoles> o/
21:02:23 <timburke> i negelcted to update the agenda again, but i think there was only one main new thing i wanted to draw attention to
21:02:54 <timburke> real quick on old business:
21:03:20 <timburke> #topic GeneratorExits in EC GET path
21:03:27 <timburke> #link https://review.opendev.org/c/openstack/swift/+/880356
21:04:04 <timburke> still hasn't merged -- i was trying to give clayg some time to take another look, but i might just go ahead and merge it since it's been a couple weeks
21:04:16 <timburke> good work, indianwhocodes!
21:04:48 <timburke> #topic requests-mock fixture and recursion errors
21:04:55 <timburke> #link https://review.opendev.org/c/openstack/swift/+/882105
21:05:44 <timburke> has merged, so we're unblocked on adding some s3 testing to probe tests
21:05:52 <acoles> https://review.opendev.org/c/openstack/swift/+/880356/comment/4a113b26_d6fd1c9d/
21:05:52 <mattoliver> Nice
21:06:08 <acoles> was there an answer to this question https://review.opendev.org/c/openstack/swift/+/880356/comment/4a113b26_d6fd1c9d/
21:08:01 <timburke> no -- good point. i *think* the answer will be "nothing material"; either way, the finally block's executing, and there isn't much of anything else in that generator -- it might have some implications up a stack level, though
21:08:37 <timburke> (also, i suppose i'm -1 on the unnecessary `pass`)
21:09:54 <timburke> thanks for keeping me honest, acoles ;-)
21:10:12 <timburke> #topic non-ascii py2 metadata on py3
21:10:17 <timburke> #link https://review.opendev.org/c/openstack/swift/+/878558
21:10:24 <acoles> I guss something used to get an exception and now doesn't, maybe that was the bug (there's no bug report, but there's mention of a traceback??)
21:11:44 <timburke> metadata fix has merged, but the user reported there's some new issue -- will be anxious to see the new bug report when it comes through
21:12:19 <mattoliver> oh really :(
21:12:44 <mattoliver> I did confirm every 2.6 and 2.7 version for the magic
21:12:54 <mattoliver> or lack of magic
21:15:04 <timburke> seems to be something with ssync in a py3-only cluster -- so i'm trying running probe tests like that, to start (though i'm not entirely sure we've got probe tests to exercise replicating non-ascii meta...)
21:16:13 <mattoliver> kk
21:16:37 <timburke> i think that about covers it for old items
21:17:03 <timburke> the main new thing i wanted to bring up was something acoles has been running down this week
21:17:13 <timburke> #topic non-cooperative EC PUTs
21:17:32 <timburke> #link https://bugs.launchpad.net/swift/+bug/2019955
21:18:03 <timburke> we've seen some strange things, like timeouts popping well after they were scheduled
21:19:10 <timburke> turned out, our network was fast enough to keep up with us reading in client chunks, encoding them to frags, and writing them out to backend buffers such that no calls ever actually blocked
21:19:17 <acoles> oh yeah, patch here https://review.opendev.org/c/openstack/swift/+/883367?usp=email
21:19:33 <mattoliver> oh, interesting!
21:20:18 <timburke> we already had some protection from that in the GET path, such that every 5 chunks we'd sleep(0) just to give other greenthreads a chance, but nothing like it in the PUT path
21:20:20 <acoles> we got alerted to this by some timeouts popping but taking much longer to pop that the timeout value!
21:21:08 <mattoliver> Nice find, and I guess it's also thanks to all the timing logs.
21:21:13 <acoles> turns out that neither the thread being timed and the thread managing the timeout would run while a PUT thread was chugging away on a GB or so of EC data
21:21:49 <acoles> once the PUT complete, the other threads ran and boom, the timeout finally popped
21:22:22 <acoles> With the fix, we hopefully won't see those timeouts at all
21:22:33 <timburke> i take it as another reminder that we should probably look into using something other than eventlet for our per-client parallelism
21:24:08 <acoles> I got some feedback on the patch, I'll give it another spin and hopefully come up with something a little more elegant
21:24:21 <timburke> 👍
21:25:00 <timburke> oh! one other new thing i wanted to bring up
21:25:05 <timburke> #topic netifaces
21:25:44 <timburke> it was pointed out on the ML that the project has been archived
21:25:47 <timburke> #link https://lists.openstack.org/pipermail/openstack-discuss/2023-April/033261.html
21:26:13 <timburke> which doesn't seem like a great place for us to have one of our dependencies
21:26:22 <mattoliver> yeah :(
21:26:58 <timburke> we only use it for getting a node's own IP addresses, so i took a stab at making the C calls ourselves and dropping the dep
21:27:10 <timburke> #link https://review.opendev.org/c/openstack/swift/+/873222
21:27:44 <timburke> there was also a bit of utils-breakup to it, since i was adding a fair few new classes
21:28:32 <timburke> anybody feel strongly about having me respin that as two patches? first pull things apart, then make my changes?
21:29:34 <zaitcev> wait, why two
21:29:37 <mattoliver> I'll take a look at it. But from an initial glance, I think its fine as one.
21:29:57 <zaitcev> I remember that patch, it looked just fine to me.
21:30:23 <zaitcev> Oh, you mean moving things in utils.
21:30:27 <mattoliver> we want to split utils up where is makes sense.. and this is just that, and it has enough to warrant it. Nice work Tim.
21:30:41 <mattoliver> I'll review it today
21:31:02 <zaitcev> I ended opening it in another window too and then compared line by line.
21:31:59 <timburke> thanks mattoliver -- but it's not "just that", which was why i'm debating about respinning it as two stacked patches ;-)
21:32:30 <zaitcev> Not worth debating in this case IMHO. Just do it next time.
21:33:09 <timburke> also note that it uses https://www.man7.org/linux/man-pages/man3/getifaddrs.3.html -- and i'm not entirely sure what platforms support it
21:34:21 <timburke> so, for now, it can still fall back to netifaces (which looks to have some other ways it can get interface data sometimes), but log a warning about it
21:34:48 <timburke> assuming we haven't gotten any reports from users, we can fully remove it in a couple years or something
21:35:13 <timburke> all right, i think that's all i've got
21:35:17 <timburke> #topic open discussion
21:35:25 <timburke> anything else we should bring up this week?
21:35:58 <zaitcev> Naah. Well, aside from the perennial poking Alistair about the watcher vs sharding.
21:35:59 <acoles> I get 3 test failures in test_ipaddrs, so maybe macos doesn't support it
21:36:18 <acoles> 🏃‍♂️
21:36:42 <acoles> zaitcev: it made it as far as my todo list :)
21:37:37 <mattoliver> I have a slightly crazy patch: https://review.opendev.org/c/openstack/swift/+/883050 which is attempting to no-op py2 tracing on the tracing chain... tests in py2 now seem to pass. Allowing the code to run in py2 (but you don't get tracing) because optel is py3 only.
21:37:57 <timburke> acoles, thanks -- i meant to try it on OS X...
21:38:15 <mattoliver> although, just removing py2 support to swift would also work :P
21:38:29 <timburke> mattoliver, 🎉
21:38:33 <timburke> one of these days...
21:38:41 <mattoliver> lol
21:40:03 <zaitcev> Did you guys know that nVIDIA has a massive OpenStack installation?
21:40:27 <timburke> i may have heard something about a pretty big swift cluster there ;-)
21:41:06 <timburke> but yeah, i've also heard about us using some of the other projects -- though i'm siloed enough to not actually have much interaction with them
21:42:54 <mattoliver> Did we know who they are, I might reach out.. though, who and what not might need to be a discussion taken offline.
21:43:21 <mattoliver> If your not referring to use of swift, then cool :)
21:45:46 <acoles> n2d o/
21:46:26 <timburke> all right, i think i'll call it anyway
21:46:37 <timburke> thank you all for coming, and thank you for working on swift!
21:46:40 <timburke> #endmeeting