| timburke__ | zaitcev, should i plan on doing a pyeclib release soon-ish to close out https://bugs.launchpad.net/pyeclib/+bug/1933533? or does it not matter that much for you? | 00:05 |
|---|---|---|
| zaitcev | Like you said, if you fix it in time, I'll never notice it. | 00:09 |
| timburke__ | well, the fix should be on master -- the question is whether it'd be good/useful for you to have a new tag that includes it | 00:11 |
| timburke__ | (i just went ahead and merged it a few weeks ago: https://review.opendev.org/c/openstack/pyeclib/+/815407 ) | 00:11 |
| zaitcev | Oh, right. Yes, I'll need a new release but it is not urgent. | 00:11 |
| timburke__ | 👍 there are a couple things i'm thinking i might want for a new tag: get some py310 testing (so i can justify adding the trove classifier, too), and deprecate jerasure (https://review.opendev.org/c/openstack/pyeclib/+/815410) | 00:19 |
| timburke__ | being able to build manylinux wheels would be cool (https://review.opendev.org/c/openstack/pyeclib/+/817498) but it'd also depend on https://review.opendev.org/c/openstack/liberasurecode/+/817497 and getting a new libec release out, and i still need to sort out how to actually *publish them*, so... *shrug* | 00:20 |
| opendevreview | Tim Burke proposed openstack/swift master: proxy: Config options to add a chance of skipping memcache https://review.opendev.org/c/openstack/swift/+/736802 | 00:48 |
| timburke__ | huh. the stable/train patch has a job where pip errors out trying to fetch constraints: SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)'),)': /constraints/upper/train | 00:52 |
| timburke__ | looks similar to errors i saw running experimental jobs recently on https://review.opendev.org/c/openstack/swift/+/821478/ | 00:53 |
| timburke__ | ussuri patch, too: https://zuul.opendev.org/t/openstack/build/16b6636eb6ac4359a7c7d39a5308ffe6 | 00:58 |
| opendevreview | Matthew Oliver proposed openstack/swift master: Updater: pop oldest and bucket heapable follow up https://review.opendev.org/c/openstack/swift/+/821790 | 01:57 |
| opendevreview | Matthew Oliver proposed openstack/swift master: Updater: pop oldest and bucket heapable follow up https://review.opendev.org/c/openstack/swift/+/821790 | 02:01 |
| opendevreview | Merged openstack/swift stable/xena: Fix cname_lookup test https://review.opendev.org/c/openstack/swift/+/821758 | 06:57 |
| *** redrobot6 is now known as redrobot | 10:34 | |
| *** tkajinam is now known as Guest8517 | 11:58 | |
| *** clayg_ is now known as clayg | 15:00 | |
| *** viks___ is now known as viks__ | 15:00 | |
| *** erbarr_ is now known as erbarr | 15:01 | |
| *** clarkb is now known as Guest8536 | 15:11 | |
| *** Guest8536 is now known as clarkb | 16:18 | |
| opendevreview | Alistair Coles proposed openstack/swift master: updater: per bucket deferral queues https://review.opendev.org/c/openstack/swift/+/821736 | 16:22 |
| opendevreview | Alistair Coles proposed openstack/swift master: updater: per bucket deferral queues https://review.opendev.org/c/openstack/swift/+/821736 | 16:34 |
| opendevreview | Alistair Coles proposed openstack/swift master: updater: per bucket deferral queues https://review.opendev.org/c/openstack/swift/+/821736 | 18:21 |
| opendevreview | Merged openstack/swift stable/wallaby: Fix cname_lookup test https://review.opendev.org/c/openstack/swift/+/821759 | 19:24 |
| opendevreview | Merged openstack/swift master: Move CI from CentOS 8 to CentOS 8 Stream https://review.opendev.org/c/openstack/swift/+/820964 | 20:51 |
| seongsoocho | good morning | 20:56 |
| timburke | #startmeeting swift | 21:00 |
| opendevmeet | Meeting started Wed Dec 15 21:00:07 2021 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| opendevmeet | The meeting name has been set to 'swift' | 21:00 |
| timburke | who's here for the swift meeting? | 21:00 |
| seongsoocho | o/ | 21:01 |
| acoles | o/ | 21:01 |
| mattoliver | o/ | 21:01 |
| timburke | as usual, the agenda's at | 21:02 |
| timburke | #link https://wiki.openstack.org/wiki/Meetings/Swift | 21:02 |
| timburke | first up | 21:02 |
| timburke | #topic object-updater fairness | 21:02 |
| timburke | you may have noticed a buynch of patches lately fiddling with how the object-updater handles ratelimiting | 21:03 |
| kota | mornin | 21:03 |
| opendevreview | Alistair Coles proposed openstack/swift master: updater: per bucket deferral queues https://review.opendev.org/c/openstack/swift/+/821736 | 21:04 |
| timburke | there's enough of them floating around now that it can be a little tricky to see what's the main thrust of the effort ;-) | 21:04 |
| acoles | there's one right there ^^ :D | 21:05 |
| mattoliver | lol | 21:05 |
| timburke | my impression is that that's the end of the main chain, is that right? and we'll likely squash/abandon most of the others? | 21:06 |
| timburke | there are three main ideas: | 21:06 |
| acoles | timburke: correct, that's the end and if we settle on the idea will be squashed with parent and grandparent | 21:07 |
| timburke | https://review.opendev.org/c/openstack/swift/+/820962/ sets a per-container ratelimit on updates -- and when the limit is exceeded, we drop updates to be picked up on the next cycle | 21:07 |
| acoles | mattoliver: this could probably be abandoned (but was very useful!) | 21:07 |
| acoles | https://review.opendev.org/c/openstack/swift/+/821790/2 | 21:08 |
| mattoliver | yeah, kk no problem | 21:08 |
| mattoliver | I'll abandon that and the other priority queue chain seeing as yours incorparates the best of all the worlds :) | 21:08 |
| timburke | https://review.opendev.org/c/openstack/swift/+/821326/ and https://review.opendev.org/c/openstack/swift/+/821480/ should probably get squashed together -- they let us keep some of that deferred work in our head so we can make effective use of the time that we'd otherwise be sleeping for the sake of the configured interval | 21:09 |
| timburke | and https://review.opendev.org/c/openstack/swift/+/821736/ better prioritizes that deferred work so we don't burn a bunch of CPU shuffling a deque | 21:10 |
| timburke | overall, the expectation is that we'll have two ratelimits for updaters: one that protects us from using up all the iops searching for work to do, and another that protects the container servers from getting overloaded | 21:12 |
| timburke | and you can adjust the interval as something of a slider to decide how much time you want to devote to overloaded containers vs responsive containers | 21:13 |
| timburke | i don't have much to add to all that -- i think we've got plenty of attention on it and it's moving in a good direction. just wanted to keep everyone else in the loop, too :-) | 21:14 |
| timburke | #topic keeping things in memcache | 21:15 |
| timburke | we've seen some bad thundering-herd behaviors when items fall out of memcached -- but just increasing cache time is risky since it increases the likelihood that the cache will be stale | 21:16 |
| timburke | a while back i started playing with an idea to randomly skip going to memcache sometimes, so we'd get a refresh before things fall out | 21:17 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/736802 | 21:17 |
| mattoliver | makes a bunch of sense, under high load when something expires out of cache the server can get hit quite hard. keeping it hot seems like a win. | 21:19 |
| mattoliver | I'll review the patch today | 21:19 |
| timburke | incorporating some of the early feedback got a little hairy, though -- needed to start digging through app.app.app... to find config values for every time a middleware wants to call get_container_info | 21:19 |
| timburke | i think i've got a plan, though, and it involves getting rid of pipeline_property :-D | 21:19 |
| acoles | do it! | 21:20 |
| mattoliver | lol, should we revisit your old less brittle pipeline_property | 21:20 |
| timburke | so mattoliver, i wouldn't worry about reviewing it until i push up a new chain ;-) | 21:20 |
| mattoliver | kk | 21:20 |
| mattoliver | on the same vein as this, I've been working on optionally putting error limited into memcache so things can globally share this info | 21:21 |
| mattoliver | And even playing with the updater sharing the same error limiting info, so if a container server is overloaded and say a proxy knows it, so does others rather then having to each figure it out themselves (by failing to hit it 11 times by default). | 21:22 |
| mattoliver | So if anyone gets bored: https://review.opendev.org/c/openstack/swift/+/820486 | 21:22 |
| timburke | definitely seems interesting | 21:23 |
| timburke | next up | 21:23 |
| acoles | as an FYI, I worked on a different approach to keeping shard ranges in cache here https://review.opendev.org/c/openstack/swift/+/817294, but we had some concerns about the bleeding of concerns from proxy to container server so I am currently hoping timburke 's randomised approach works out | 21:23 |
| timburke | #topic tempurl signature logging | 21:23 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/817476 | 21:23 |
| timburke | has anyone had a chance to revisit this patch? | 21:23 |
| mattoliver | I reviewed it at some point, but got distracted. I'll loop back and see where it's at today if you like | 21:25 |
| timburke | that'd be great, thanks mattoliver! | 21:25 |
| timburke | #topic recon and servers_per_port | 21:26 |
| timburke | i still need to follow up on https://review.opendev.org/c/openstack/swift/+/818881 | 21:26 |
| timburke | but does anyone have concerns about the approach of only querying once per ip? | 21:28 |
| timburke | i could imagine deploying with not all disks available to all ports, but i'm not sure that anyone actually *does* | 21:29 |
| timburke | i guess i won't worry about it then | 21:30 |
| timburke | #topic request tracing | 21:31 |
| timburke | mattoliver, when's that talk again? | 21:31 |
| mattoliver | Let me find the link :) | 21:31 |
| timburke | #link https://lca2022.linux.org.au/schedule/presentation/32/ | 21:32 |
| timburke | next month! | 21:32 |
| mattoliver | oh thanks, yeah 15th Jan | 21:32 |
| mattoliver | Its cause I tweeted about the tracing at the PTG so now I'm turning it into a talk. | 21:33 |
| timburke | need anything to help prep for it? | 21:33 |
| mattoliver | Plus side is, it's getting me to clean up and fix a bunch. | 21:33 |
| mattoliver | not at this point. I don't plan on it merging anytime soon. Trying to debug why bulk and slo seem to loose the known spans. | 21:34 |
| mattoliver | debugging they're there at the start of the middleware, but gone in a function call in the same middleware.. so need to dig into how that can happen. | 21:35 |
| timburke | my best guess would be something with the greenthreadpools -- i seem to remember that being tricky when poking at similar ideas with zipkin | 21:35 |
| timburke | should i keep it on the agenda for next week? | 21:35 |
| mattoliver | yeah, I've got some interesting scafalding to get around some green pool things. | 21:36 |
| mattoliver | this seems to be different (but probably isn't once I dig in some more). | 21:36 |
| timburke | all right, that's all i've got | 21:37 |
| timburke | #topic open discussion | 21:37 |
| mattoliver | was wanting to do a bulk extract as a interesting example in the talk.. but need to get it working first :P | 21:37 |
| mattoliver | yeah, keep it in the agenda and it'll motivate me to fix it :P | 21:37 |
| timburke | what else should we bring up this week? | 21:37 |
| zaitcev | Nothing for me. | 21:38 |
| mattoliver | I'm good | 21:38 |
| timburke | oh -- speaking of next week, should we bother meeting? or should we next meet next year? | 21:39 |
| mattoliver | oh yeah, next week I'll be driving to my parents (10 hours away). so probably not around | 21:39 |
| mattoliver | So I'm ok for skipping | 21:39 |
| timburke | i'm voting for next year :-) | 21:41 |
| acoles | +1 or skipping | 21:41 |
| timburke | enjoy the end of the year everyone! | 21:42 |
| kota | I'm ok | 21:42 |
| seongsoocho | me too :-) | 21:42 |
| timburke | all right -- thank you all for coming, thank you for working on swift, and talk to you next year! | 21:42 |
| timburke | #endmeeting | 21:42 |
| opendevmeet | Meeting ended Wed Dec 15 21:42:41 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:42 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/swift/2021/swift.2021-12-15-21.00.html | 21:42 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/swift/2021/swift.2021-12-15-21.00.txt | 21:42 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/swift/2021/swift.2021-12-15-21.00.log.html | 21:42 |
| opendevreview | Tim Burke proposed openstack/swift master: proxy: Add a chance to skip memcache when looking for shard ranges https://review.opendev.org/c/openstack/swift/+/736802 | 23:21 |
| opendevreview | Tim Burke proposed openstack/swift master: Get rid of pipeline_property https://review.opendev.org/c/openstack/swift/+/821920 | 23:21 |
| opendevreview | Tim Burke proposed openstack/swift master: proxy: Add a chance to skip memcache for get_*_info calls https://review.opendev.org/c/openstack/swift/+/821921 | 23:21 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!