| opendevreview | Jianjian Huo proposed openstack/swift master: Proxy: add metrics related to error limiter. https://review.opendev.org/c/openstack/swift/+/863446 | 04:39 |
|---|---|---|
| opendevreview | Jianjian Huo proposed openstack/swift master: Proxy: add metrics related to error limiter. https://review.opendev.org/c/openstack/swift/+/863446 | 05:34 |
| opendevreview | Jianjian Huo proposed openstack/swift master: sharding: avoid transient overlaps while auditing. https://review.opendev.org/c/openstack/swift/+/864088 | 07:14 |
| timburke | #startmeeting swift | 21:00 |
| opendevmeet | Meeting started Wed Nov 9 21:00:08 2022 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| opendevmeet | The meeting name has been set to 'swift' | 21:00 |
| timburke | who's here for the swift team meeting? | 21:00 |
| mattoliver | o/ | 21:00 |
| mattoliver | (looks like I just made it in time) | 21:00 |
| timburke | sorry to have skipped out on so many meetings -- but i did at least update the agenda today! | 21:01 |
| timburke | #link https://wiki.openstack.org/wiki/Meetings/Swift | 21:01 |
| timburke | i only have a couple topics, so let's get started | 21:02 |
| timburke | #topic broken func tests | 21:02 |
| timburke | recently there was a fix for a CPython CVE | 21:03 |
| timburke | #link https://github.com/python/cpython/issues/87389 | 21:03 |
| kota | sorry, late a bit | 21:03 |
| timburke | that's been working its way through backports | 21:03 |
| timburke | #link https://python-security.readthedocs.io/vuln/http-server-redirection.html | 21:03 |
| mattoliver | ahh that explains the // patch I saw pop up | 21:04 |
| timburke | the fix made it so urls like http://swift.example.com//v1/AUTH_test (which used to 400) would get translated to http://swift.example.com/v1/AUTH_test before they get to us | 21:05 |
| timburke | which doesn't seem so bad -- but when you have something like domain_remap, it can change the object name for a url like http://container.account.swift.example.com//object | 21:06 |
| timburke | i've written up a bug for them | 21:07 |
| timburke | #link https://github.com/python/cpython/issues/99220 | 21:07 |
| timburke | and a patch for us | 21:08 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/863441 | 21:08 |
| timburke | we first saw the issue with some non-voting centos9/fips func tests, but since they were non-voting the investigation wasn't really prioritized | 21:09 |
| timburke | recently, though, it started affecting our rolling upgrade jobs -- and since there's no fixing old proxy code, i had to just make them non-voting for now | 21:10 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/863929 | 21:10 |
| timburke | if anyone has time to review the fix, that'd be much appreciated! once we have a fix landed, i'll propose backports, too | 21:11 |
| timburke | any questions or comments? | 21:12 |
| mattoliver | I'll put it on my list. Although I've been really distracted these last few weeks. But will try to get to them in the next day or 2. | 21:12 |
| timburke | thanks | 21:12 |
| timburke | not that i really *want* to maintain a new library, but eventually i feel like we'll kind of have to just write our own request parser :-( | 21:14 |
| timburke | speaking of cpython... | 21:14 |
| timburke | #topic testing under py310 | 21:14 |
| timburke | we're getting closer and closer to testing on py310! | 21:14 |
| timburke | the pytest is nearly passing | 21:15 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/851099 | 21:15 |
| timburke | "the pytest patch" i mean | 21:15 |
| mattoliver | lol, yeah, it's been harder then expected to just work. | 21:16 |
| timburke | the only remaining issue is the requirements-check | 21:16 |
| mattoliver | mostly due to requirement versions etc | 21:16 |
| timburke | so i proposed a patch to add pytest-cov to requirements | 21:16 |
| timburke | #link https://review.opendev.org/c/openstack/requirements/+/863581/ | 21:16 |
| timburke | and fortunately, once those land, it looks like py310 unit tests already pass | 21:18 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/850947 | 21:18 |
| mattoliver | oh nice! | 21:18 |
| timburke | so we're getting there! next, i might try to switch our func (or even probe) tests to jammy | 21:19 |
| mattoliver | oh good idea. | 21:20 |
| timburke | that's all i've got | 21:20 |
| timburke | #topic open discussion | 21:20 |
| timburke | what else should we discuss this week? | 21:20 |
| mattoliver | I've been looking at Al's shard ranges from root final patch. Looking good. Think I'm almost happy it wont break things. But will solve some brittleness | 21:21 |
| mattoliver | I also talked to clay briefly about ring v2. And we wants me to revisit the builder into ring stuff. | 21:22 |
| timburke | nice -- am i right to think that acoles's patch you're talking about is https://review.opendev.org/c/openstack/swift/+/852905 ? | 21:23 |
| mattoliver | I said probably needs some disucssions esp with timburke regarding my serialzation classes stuff. So I think I have him onboard with landing the first few patches. | 21:23 |
| mattoliver | yup that's it | 21:23 |
| mattoliver | then looping back to get ringbulder classes after | 21:24 |
| timburke | i definitely like the idea of having some movement on that chain :-) | 21:25 |
| mattoliver | the default will be v1 rings, so if need be we can wait a bit if we want to include more bits into it (before reworking our downstream controller). | 21:25 |
| mattoliver | yeah +1 | 21:25 |
| mattoliver | So I plan, if I'm not pulled into any more fires, to prioritize re-reviewing those first 2 and getting them landed | 21:26 |
| timburke | 👍 | 21:26 |
| mattoliver | I also reworked the Otel tracing chain to support an Otel Collector. | 21:27 |
| mattoliver | so not just jaeger | 21:27 |
| mattoliver | As we have movement on wanting to test that at work. | 21:27 |
| timburke | nice | 21:27 |
| mattoliver | #link https://review.opendev.org/c/openstack/swift/+/857559 | 21:28 |
| timburke | oh! i also have a patch to improve wsgi server start-up time when you configure a bunch of workers | 21:28 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/862843 | 21:28 |
| mattoliver | And I sometimes do wonder if I should convince someone in testing out the sharded pending files patch. | 21:28 |
| mattoliver | oh nice one! | 21:28 |
| mattoliver | great, I have notices a "reload" can take a while. I assume this would speed that up too? | 21:29 |
| mattoliver | though that might be caused by finishing old requests. | 21:29 |
| timburke | yep -- i saw a reload that took ~70s come down to ~15s | 21:30 |
| mattoliver | \o/ | 21:30 |
| mattoliver | that's awesome! | 21:30 |
| timburke | reload should just wait for the new workers to come up; old workers can continue running | 21:30 |
| mattoliver | The reload stuff is awesome btw! | 21:31 |
| mattoliver | we used it a bunch in a new mcrouter rollout. And it's freakin cool! | 21:31 |
| timburke | nice! | 21:32 |
| timburke | oh! speaking of old workers -- i added a configurable timeout following a reload to start stopping workers more forcefully | 21:33 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/789035 | 21:33 |
| timburke | that patch started out just being about quieting some log messages, but i really like having that extra state on hand | 21:33 |
| mattoliver | nice | 21:34 |
| mattoliver | I mean maybe they're dealing with a long running request.. but nice to have a hammer | 21:34 |
| timburke | yeah... 2hr might be a little short for the default, i suppose... | 21:35 |
| mattoliver | who knows, that's one of those hard things to pick | 21:37 |
| mattoliver | how long is a peice of string. | 21:37 |
| timburke | as much as anything, i just want to give ops a way that they can be *sure* no old workers are running with stale configs | 21:37 |
| mattoliver | 2hrs seems like a ok starting point. and we can extend from there | 21:37 |
| mattoliver | +1 | 21:38 |
| mattoliver | That's all I have for the moment | 21:39 |
| timburke | all right, i think i'll call it | 21:39 |
| timburke | thank you for coming, and thank you for working on swift! | 21:39 |
| timburke | #endmeeting | 21:39 |
| opendevmeet | Meeting ended Wed Nov 9 21:39:40 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:39 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/swift/2022/swift.2022-11-09-21.00.html | 21:39 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/swift/2022/swift.2022-11-09-21.00.txt | 21:39 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/swift/2022/swift.2022-11-09-21.00.log.html | 21:39 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!