| opendevreview | Merged openstack/swift master: docs: Update versions in Getting Started doc https://review.opendev.org/c/openstack/swift/+/883314 | 00:17 |
|---|---|---|
| opendevreview | Merged openstack/swift master: docs: Remove references to out-dated install guides https://review.opendev.org/c/openstack/swift/+/883313 | 00:20 |
| opendevreview | OpenStack Proposal Bot proposed openstack/swift master: Imported Translations from Zanata https://review.opendev.org/c/openstack/swift/+/882635 | 03:04 |
| edausq | hello and thank you for the work on "Properly read py2 object metadata on py3" | 07:23 |
| edausq | we have been testing it but we are now encountering a metadata issue, on a full py3 cluster, after ssync replication | 07:25 |
| edausq | I don't have much logs for now to open a ticket | 07:29 |
| kota | for the meeting tomorrow, sorry, I got a cold and cough so i'll be absent on that one. | 09:30 |
| opendevreview | Alistair Coles proposed openstack/swift master: proxy: add periodic zero-time sleep during object PUT https://review.opendev.org/c/openstack/swift/+/883367 | 12:31 |
| opendevreview | Alistair Coles proposed openstack/swift master: Support swift.proxy_logging_status in request env https://review.opendev.org/c/openstack/swift/+/784808 | 16:04 |
| opendevreview | Alistair Coles proposed openstack/swift master: s3api: add option to log '503 Slow Down' as 429 https://review.opendev.org/c/openstack/swift/+/883415 | 16:04 |
| timburke | no worries, kota! rest up | 17:00 |
| timburke | edausq, well shoot. i'll be interested in seeing some logs for it; maybe even better, find an affected object and grab the output of `xattr -l <diskfile>` | 17:02 |
| timburke | #startmeeting swift | 21:00 |
| opendevmeet | Meeting started Wed May 17 21:00:10 2023 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| opendevmeet | The meeting name has been set to 'swift' | 21:00 |
| timburke | who's here for the swift meeting? | 21:00 |
| mattoliver | o/ | 21:00 |
| acoles | o/ | 21:01 |
| timburke | i negelcted to update the agenda again, but i think there was only one main new thing i wanted to draw attention to | 21:02 |
| timburke | real quick on old business: | 21:02 |
| timburke | #topic GeneratorExits in EC GET path | 21:03 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/880356 | 21:03 |
| timburke | still hasn't merged -- i was trying to give clayg some time to take another look, but i might just go ahead and merge it since it's been a couple weeks | 21:04 |
| timburke | good work, indianwhocodes! | 21:04 |
| timburke | #topic requests-mock fixture and recursion errors | 21:04 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/882105 | 21:04 |
| timburke | has merged, so we're unblocked on adding some s3 testing to probe tests | 21:05 |
| acoles | https://review.opendev.org/c/openstack/swift/+/880356/comment/4a113b26_d6fd1c9d/ | 21:05 |
| mattoliver | Nice | 21:05 |
| acoles | was there an answer to this question https://review.opendev.org/c/openstack/swift/+/880356/comment/4a113b26_d6fd1c9d/ | 21:06 |
| timburke | no -- good point. i *think* the answer will be "nothing material"; either way, the finally block's executing, and there isn't much of anything else in that generator -- it might have some implications up a stack level, though | 21:08 |
| timburke | (also, i suppose i'm -1 on the unnecessary `pass`) | 21:08 |
| timburke | thanks for keeping me honest, acoles ;-) | 21:09 |
| timburke | #topic non-ascii py2 metadata on py3 | 21:10 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/878558 | 21:10 |
| acoles | I guss something used to get an exception and now doesn't, maybe that was the bug (there's no bug report, but there's mention of a traceback??) | 21:10 |
| timburke | metadata fix has merged, but the user reported there's some new issue -- will be anxious to see the new bug report when it comes through | 21:11 |
| mattoliver | oh really :( | 21:12 |
| mattoliver | I did confirm every 2.6 and 2.7 version for the magic | 21:12 |
| mattoliver | or lack of magic | 21:12 |
| timburke | seems to be something with ssync in a py3-only cluster -- so i'm trying running probe tests like that, to start (though i'm not entirely sure we've got probe tests to exercise replicating non-ascii meta...) | 21:15 |
| mattoliver | kk | 21:16 |
| timburke | i think that about covers it for old items | 21:16 |
| timburke | the main new thing i wanted to bring up was something acoles has been running down this week | 21:17 |
| timburke | #topic non-cooperative EC PUTs | 21:17 |
| timburke | #link https://bugs.launchpad.net/swift/+bug/2019955 | 21:17 |
| timburke | we've seen some strange things, like timeouts popping well after they were scheduled | 21:18 |
| timburke | turned out, our network was fast enough to keep up with us reading in client chunks, encoding them to frags, and writing them out to backend buffers such that no calls ever actually blocked | 21:19 |
| acoles | oh yeah, patch here https://review.opendev.org/c/openstack/swift/+/883367?usp=email | 21:19 |
| mattoliver | oh, interesting! | 21:19 |
| timburke | we already had some protection from that in the GET path, such that every 5 chunks we'd sleep(0) just to give other greenthreads a chance, but nothing like it in the PUT path | 21:20 |
| acoles | we got alerted to this by some timeouts popping but taking much longer to pop that the timeout value! | 21:20 |
| mattoliver | Nice find, and I guess it's also thanks to all the timing logs. | 21:21 |
| acoles | turns out that neither the thread being timed and the thread managing the timeout would run while a PUT thread was chugging away on a GB or so of EC data | 21:21 |
| acoles | once the PUT complete, the other threads ran and boom, the timeout finally popped | 21:21 |
| acoles | With the fix, we hopefully won't see those timeouts at all | 21:22 |
| timburke | i take it as another reminder that we should probably look into using something other than eventlet for our per-client parallelism | 21:22 |
| acoles | I got some feedback on the patch, I'll give it another spin and hopefully come up with something a little more elegant | 21:24 |
| timburke | 👍 | 21:24 |
| timburke | oh! one other new thing i wanted to bring up | 21:25 |
| timburke | #topic netifaces | 21:25 |
| timburke | it was pointed out on the ML that the project has been archived | 21:25 |
| timburke | #link https://lists.openstack.org/pipermail/openstack-discuss/2023-April/033261.html | 21:25 |
| timburke | which doesn't seem like a great place for us to have one of our dependencies | 21:26 |
| mattoliver | yeah :( | 21:26 |
| timburke | we only use it for getting a node's own IP addresses, so i took a stab at making the C calls ourselves and dropping the dep | 21:26 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/873222 | 21:27 |
| timburke | there was also a bit of utils-breakup to it, since i was adding a fair few new classes | 21:27 |
| timburke | anybody feel strongly about having me respin that as two patches? first pull things apart, then make my changes? | 21:28 |
| zaitcev | wait, why two | 21:29 |
| mattoliver | I'll take a look at it. But from an initial glance, I think its fine as one. | 21:29 |
| zaitcev | I remember that patch, it looked just fine to me. | 21:29 |
| zaitcev | Oh, you mean moving things in utils. | 21:30 |
| mattoliver | we want to split utils up where is makes sense.. and this is just that, and it has enough to warrant it. Nice work Tim. | 21:30 |
| mattoliver | I'll review it today | 21:30 |
| zaitcev | I ended opening it in another window too and then compared line by line. | 21:31 |
| timburke | thanks mattoliver -- but it's not "just that", which was why i'm debating about respinning it as two stacked patches ;-) | 21:31 |
| zaitcev | Not worth debating in this case IMHO. Just do it next time. | 21:32 |
| timburke | also note that it uses https://www.man7.org/linux/man-pages/man3/getifaddrs.3.html -- and i'm not entirely sure what platforms support it | 21:33 |
| timburke | so, for now, it can still fall back to netifaces (which looks to have some other ways it can get interface data sometimes), but log a warning about it | 21:34 |
| timburke | assuming we haven't gotten any reports from users, we can fully remove it in a couple years or something | 21:34 |
| timburke | all right, i think that's all i've got | 21:35 |
| timburke | #topic open discussion | 21:35 |
| timburke | anything else we should bring up this week? | 21:35 |
| zaitcev | Naah. Well, aside from the perennial poking Alistair about the watcher vs sharding. | 21:35 |
| acoles | I get 3 test failures in test_ipaddrs, so maybe macos doesn't support it | 21:35 |
| acoles | 🏃♂️ | 21:36 |
| acoles | zaitcev: it made it as far as my todo list :) | 21:36 |
| mattoliver | I have a slightly crazy patch: https://review.opendev.org/c/openstack/swift/+/883050 which is attempting to no-op py2 tracing on the tracing chain... tests in py2 now seem to pass. Allowing the code to run in py2 (but you don't get tracing) because optel is py3 only. | 21:37 |
| timburke | acoles, thanks -- i meant to try it on OS X... | 21:37 |
| mattoliver | although, just removing py2 support to swift would also work :P | 21:38 |
| timburke | mattoliver, 🎉 | 21:38 |
| timburke | one of these days... | 21:38 |
| mattoliver | lol | 21:38 |
| zaitcev | Did you guys know that nVIDIA has a massive OpenStack installation? | 21:40 |
| timburke | i may have heard something about a pretty big swift cluster there ;-) | 21:40 |
| timburke | but yeah, i've also heard about us using some of the other projects -- though i'm siloed enough to not actually have much interaction with them | 21:41 |
| mattoliver | Did we know who they are, I might reach out.. though, who and what not might need to be a discussion taken offline. | 21:42 |
| mattoliver | If your not referring to use of swift, then cool :) | 21:43 |
| acoles | n2d o/ | 21:45 |
| timburke | all right, i think i'll call it anyway | 21:46 |
| timburke | thank you all for coming, and thank you for working on swift! | 21:46 |
| timburke | #endmeeting | 21:46 |
| opendevmeet | Meeting ended Wed May 17 21:46:40 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:46 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/swift/2023/swift.2023-05-17-21.00.html | 21:46 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/swift/2023/swift.2023-05-17-21.00.txt | 21:46 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/swift/2023/swift.2023-05-17-21.00.log.html | 21:46 |
| opendevreview | Tim Burke proposed openstack/swift master: tests: Make dark data probe tests pass with sync_method = ssync https://review.opendev.org/c/openstack/swift/+/883452 | 22:26 |
| opendevreview | Tim Burke proposed openstack/swift master: probe tests: Skip versioning tests if not enabled https://review.opendev.org/c/openstack/swift/+/883455 | 23:03 |
| opendevreview | Merged openstack/swift master: Unit test for keepalive timeout https://review.opendev.org/c/openstack/swift/+/881517 | 23:20 |
| opendevreview | Merged openstack/swift master: memcached: log user provided keys in exception error logging. https://review.opendev.org/c/openstack/swift/+/880903 | 23:20 |
| opendevreview | Merged openstack/swift master: Add cap_length helper https://review.opendev.org/c/openstack/swift/+/831020 | 23:20 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!