| opendevreview | Merged openstack/swift stable/zed: Fix docs build https://review.opendev.org/c/openstack/swift/+/870780 | 01:13 |
|---|---|---|
| opendevreview | Merged openstack/swift stable/yoga: Fix docs build https://review.opendev.org/c/openstack/swift/+/870781 | 01:18 |
| opendevreview | Tim Burke proposed openstack/swift stable/victoria: Fix docs build https://review.opendev.org/c/openstack/swift/+/870784 | 01:50 |
| opendevreview | Merged openstack/swift master: Skip coverage reports when running pytest directly https://review.opendev.org/c/openstack/swift/+/870865 | 02:23 |
| opendevreview | Merged openstack/swift stable/xena: Fix docs build https://review.opendev.org/c/openstack/swift/+/870782 | 03:22 |
| opendevreview | Merged openstack/swift master: Clean up project URLs for PyPI https://review.opendev.org/c/openstack/swift/+/870862 | 03:42 |
| opendevreview | Merged openstack/swift master: Don't run reno as part of building an sdist https://review.opendev.org/c/openstack/swift/+/870863 | 04:17 |
| opendevreview | Merged openstack/swift stable/wallaby: Fix docs build https://review.opendev.org/c/openstack/swift/+/870783 | 07:00 |
| opendevreview | Merged openstack/swift master: s3api: Prevent XXE injections https://review.opendev.org/c/openstack/swift/+/870823 | 07:01 |
| opendevreview | Jianjian Huo proposed openstack/swift master: Proxy: restructure cached updating shard ranges https://review.opendev.org/c/openstack/swift/+/870886 | 07:32 |
| mcape | still struggling with our rocky->yoga upgrade, since our nodes are on centos7... and there are no packages for yoga. | 10:26 |
| mcape | currently i'm testing upgrade using code repository : | 10:27 |
| mcape | install modules from requirements.txt via pip | 10:27 |
| mcape | and do "python setup.py install" after that | 10:27 |
| mcape | functional tests are coming through, all daemons are running fine | 10:27 |
| mcape | the main downside to this approach -- is that it is difficult to do a downgrade (since part of the modules are from pip, and part are previously installed by packet manager), | 10:27 |
| mcape | downgraded code runs, but functional tests are not starting due to various dependency problems. | 10:28 |
| mcape | only managed to do upgrade-downgrade-run tests succefully once, but since that I can't reproduce that success | 10:28 |
| mcape | so the question is maybe there is better path? | 10:28 |
| mcape | another concern is a lot of warnings from sharder, which reports missed shard ranges, while the objects are thankfully reachable | 11:55 |
| mcape | logs look like this https://pastebin.com/khgVcsvE | 11:55 |
| mcape | not sure if upgrade will fix that or make things worse :-/ | 11:56 |
| opendevreview | Olivier Chaze proposed openstack/swift master: Emptying buffer when quota is exceeded https://bugs.launchpad.net/swift/+bug/2002985 https://review.opendev.org/c/openstack/swift/+/870541 | 12:29 |
| *** gmann is now known as gmann_afk | 17:29 | |
| *** gmann_afk is now known as gmann | 17:41 | |
| *** gmann is now known as gmann_afk | 18:06 | |
| *** gmann_afk is now known as gmann | 18:56 | |
| kota | good morning | 20:56 |
| indianwhocodes | good afternoon | 21:00 |
| mattoliver | Morning | 21:00 |
| opendevreview | Merged openstack/swift master: Clean up a bunch of deprecation warnings https://review.opendev.org/c/openstack/swift/+/851100 | 21:01 |
| timburke | #startmeeting swift | 21:01 |
| opendevmeet | Meeting started Wed Jan 18 21:01:28 2023 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:01 |
| opendevmeet | The meeting name has been set to 'swift' | 21:01 |
| timburke | who's here for the swift meeting? | 21:01 |
| mattoliver | o/ | 21:02 |
| zaitcev | o7 | 21:02 |
| kota | o/ | 21:02 |
| indianwhocodes | o/ | 21:03 |
| timburke | as usual, the agenda's at | 21:03 |
| timburke | #link https://wiki.openstack.org/wiki/Meetings/Swift | 21:03 |
| timburke | first up | 21:03 |
| timburke | #topic CVE-2022-47950 | 21:03 |
| timburke | there's a critical CVE for swift that was made public this week | 21:04 |
| timburke | security researchers at OVH discovered a way for authenticated clients to read arbitrary files from proxy-servers | 21:05 |
| timburke | #link https://bugs.launchpad.net/swift/+bug/1998625 | 21:06 |
| timburke | this included things like tempauth credentials, keymaster root secrets, swift.conf swift_hash_path_prefix/suffix values, and auth_token service user credentials | 21:07 |
| timburke | the good news is, this only affects clusters with S3 access enabled | 21:08 |
| timburke | the bad news is, it's not limited to s3api; swift3 is also affected (for clusters that aren't on rocky yet) | 21:09 |
| timburke | the fix is already merged to master | 21:10 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/870823 | 21:10 |
| mattoliver | do we still have a swift3 repo? I haven't looked at that for years | 21:10 |
| timburke | and backports are up for wallaby through zed -- though i'm realizing that the functional test may need a small update | 21:10 |
| timburke | https://opendev.org/x/swift3/ is still a thing, though inactive | 21:11 |
| mattoliver | kk | 21:12 |
| timburke | maybe also worth noting: the github mirroring went away a while ago: https://github.com/openstack-archive/swift3/ | 21:12 |
| timburke | i don't think any patch is expected for swift3 -- i just want to be clear about the scope of affected clusters | 21:13 |
| mattoliver | ok, and the readme does say it's frozen and all new patches goto the s3api middleware, so maybe its ok. | 21:14 |
| mattoliver | but a good reason for anyone to finally upgrade from queens or before | 21:14 |
| mattoliver | zaitcev: did you guys have anyone still on pre-rocky? or is that something to ask cschwede ? | 21:15 |
| timburke | for sure! thankfully, even if operators can't upgrade directly to a more-recent swift, the code change is literally one line | 21:15 |
| timburke | https://review.opendev.org/c/openstack/swift/+/870823/1/swift/common/middleware/s3api/etree.py | 21:15 |
| zaitcev | mattoliver: Yes. We still have customers on Queens. | 21:16 |
| timburke | unfortunately, the best mitigation i see for anyone that can't do *any* sort of code change is to disable S3 access | 21:17 |
| mattoliver | the backport is going to what, as low as wallaby you said, so its probably more of a case to anything before there. | 21:17 |
| mattoliver | Like you say tho isn't a 1 liner | 21:17 |
| mattoliver | if redhat still package swift3 for queens maybe they can just at a patch to it. | 21:18 |
| timburke | if i can get stable gates happy again, i'm happy to do more backports -- more on that later, tho | 21:18 |
| mattoliver | kk | 21:18 |
| timburke | any other questions or comments on the CVE? i want to give everyone a chance to digest it a bit | 21:20 |
| timburke | all right, we'll move on -- if anyone needs more info about it, feel free to reach out to me, either in channel or privately | 21:23 |
| timburke | and i'd like to give special thanks to OVH for discovering the issue, mattoliver acoles and clayg for reviewing the patch, and fungi for getting the patches into gerrit! | 21:23 |
| timburke | #topic vPTG | 21:23 |
| fungi | my pleasure! | 21:24 |
| timburke | just a reminder that there's a vPTG scheduled for the end of march | 21:24 |
| timburke | mattoliver and i still need to get doodle poll and etherpad up | 21:24 |
| timburke | but everyone should go ahead and register if they haven't already! | 21:25 |
| timburke | #link https://www.eventbrite.com/e/project-teams-gathering-march-2023-tickets-483971570997 | 21:25 |
| mattoliver | oh yeah, I'll get on that etherpad, so we can start gathering ideas | 21:25 |
| timburke | thanks mattoliver | 21:25 |
| timburke | #topic gate health | 21:26 |
| timburke | another week, another gate breakage | 21:26 |
| mattoliver | sigh | 21:26 |
| timburke | dnspython 2.3.0 was recently released, which caused eventlet's monkey-patching of it to break | 21:27 |
| mattoliver | oh great :( | 21:27 |
| timburke | this would happen as soon as you went to import eventlet | 21:27 |
| timburke | #link https://github.com/eventlet/eventlet/issues/781 | 21:27 |
| timburke | good news is that a fix has merged to eventlet, and temoto tagged a new 0.33.3 release that includes it | 21:28 |
| mattoliver | So we need to pin the package until there is an upstream fix? | 21:28 |
| mattoliver | oh wow, I like a quick turn around! | 21:28 |
| timburke | ...yes and no. so, we *should* have been safe from this because of the general openstack upper-constraints policy | 21:29 |
| timburke | https://github.com/openstack/requirements/blob/master/upper-constraints.txt still lists dnspython===2.2.1 | 21:29 |
| timburke | and in fact, *most* of our jobs were still fine | 21:30 |
| timburke | buuuut... our docs build previously only specified the extra doc-building requirements | 21:30 |
| mattoliver | hey, it looks like you fixed it in upstream eventlet timburke, nice work! | 21:30 |
| timburke | :D | 21:30 |
| timburke | i also went ahead and fixed our docs job to properly use constraints | 21:31 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/870853 | 21:31 |
| fungi | if it makes anyone feel any better (it probably doesn't), you're not alone. mistral spotted the exact same problem today, so i was able to point amorin at timburke's fix. worked like a charm there as well | 21:31 |
| timburke | 👍 as long as the pain saves someone else some trouble, right? | 21:32 |
| mattoliver | nice | 21:33 |
| timburke | really, i don't think i would've minded *nearly* as much if we weren't also trying to get the CVE fix merged | 21:33 |
| timburke | the doc-building fix also got backported -- wallaby through zed all have it now | 21:34 |
| mattoliver | lol, true | 21:34 |
| timburke | victoria started running into other issues, though | 21:35 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/870784 | 21:35 |
| timburke | this is unfortunate and annoying -- not two weeks ago that gate was working and we merged the fix for the cpython '//' bug | 21:36 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/868143 | 21:36 |
| timburke | i'll work on trying to get it functional again, but wanted to float a couple ideas | 21:38 |
| timburke | 1. declare more stable branches end-of-life | 21:39 |
| mattoliver | failures seem to be caused by the smmap package verson mismatch, maybe needs a specific pin? | 21:40 |
| timburke | 2. remove (or mark non-voting) some/many jobs from stable branches | 21:40 |
| mattoliver | yeah true, is there an openstack policy on how many stable branches to maintain? | 21:41 |
| timburke | mattoliver, yeah -- i think it's probably related to py2 support. thinking longer term, though, i'm willing to bet we'll see similar issues cropping up for py36 (which is no longer maintained by upstream cpython) | 21:41 |
| mattoliver | yeah, definitely looks py2, as obvioulsy the new version of smmap looks only py3+ | 21:42 |
| mattoliver | so yeah, we're just going to get more and more of these | 21:42 |
| timburke | there's https://docs.openstack.org/project-team-guide/stable-branches.html -- "OpenStack development typically has 3 branches active at any point of time, master (the current development release), stable (the most recent release) and oldstable (previous release)." | 21:43 |
| mattoliver | we either need to lock all pre py3 branches down, pinning everything. Or maybe mark them as non-voting knowing that py2 support has bacially gone | 21:43 |
| timburke | with the introduction of "extended maintenance" i'm not sure that quote is entirely true | 21:44 |
| mattoliver | Well based on n-2, we could remove a bunch of stable | 21:44 |
| mattoliver | although I know redhat do support for longer. So do they need more stable branches upstream? | 21:44 |
| mattoliver | or do we just need to support n-2 + some point in time LTS, but the latter can be non-voting for a bunch of tests? | 21:45 |
| timburke | my impression is that we have a lot of discretion about both how many stable branches to support and what level of testing and support "extended maintenance" should mean | 21:47 |
| mattoliver | extended maintence does say reduced CI commitment which makes me think, non-voting or even removed gate checks | 21:48 |
| timburke | fwiw, i've left so many stable branches open so far for exactly the sort of situation in which we find ourselves: a pretty big security issue for which it'd be nice to offer downstream packagers an official release for old versions | 21:49 |
| mattoliver | yeah | 21:49 |
| mattoliver | which is good | 21:49 |
| timburke | but if it turns into me fighting with CI off and on for a month or two to be able to land *anything*, i'm not sure it's worth it | 21:50 |
| mattoliver | go n-2 of them should be supported (cI etc) the rest we should be able to reduce CI committment. maybe reduce it to pep8 and unit tests? | 21:51 |
| mattoliver | I guess functional and probe too, but we seem to be having package issues breaking those | 21:52 |
| mattoliver | OR, do we just force in this security bug to the old old stables and then EOF everything below wallaby and be done with it for now? | 21:53 |
| mattoliver | *security patch | 21:53 |
| timburke | not on victoria, at least. could keep in-process func tests but drop dsvm, for example | 21:53 |
| mattoliver | true | 21:53 |
| timburke | idk -- i'll poke at it for another day or two, see what i can come up with. seems like a good topic for the PTG tho :-) | 21:54 |
| timburke | #topic open discussion | 21:54 |
| timburke | anything else we should bring up this week? | 21:54 |
| mattoliver | Well I can't speak for anyone, but for now, I think reduce CI to make them work and get the current security fix in, then we can discuss ditching them at the PTG | 21:55 |
| mattoliver | or at least having our own definition of reduced CI committment that doesn't take all your time | 21:55 |
| mattoliver | also sorry about it always falling to you timburke | 21:55 |
| timburke | eh, no worries -- there are worse things. just feels like i'm spinning my wheels sometimes, tho | 21:56 |
| mattoliver | OK open discussion, I'm playing with a sharding statemachine change to just freakin update the state_timestamp everytime a state changes (why we don't already is historic), but I feel it'll make life easier. So having a play with how much churn it causes | 21:58 |
| mattoliver | alot it seems in tests. | 21:58 |
| mattoliver | haven't pushed it up yet, and it's currently only an experiment, but if it works I think it'll make the early cleave and active stuff much more trivial to deal with. | 21:58 |
| timburke | nice! looking forward to it | 21:59 |
| mattoliver | trying to tread carefully because the state_timestamp is used to cycle round the statemachine, and statemachine changes can be fickle, so hoping I haven't missed something past al or I knew that currement me doesn't. | 21:59 |
| timburke | i do wonder why it isn't like that already... | 21:59 |
| mattoliver | yeah, I agree, and it's because state_timestamp was added later to allow us to do back to active in roots, ie restart the statemachine | 22:00 |
| timburke | all right, we're at time so i'll call it | 22:00 |
| mattoliver | kk | 22:01 |
| timburke | thank you all for coming, and thank you for working on swift! | 22:01 |
| timburke | #endmeeting | 22:01 |
| opendevmeet | Meeting ended Wed Jan 18 22:01:15 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 22:01 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/swift/2023/swift.2023-01-18-21.01.html | 22:01 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/swift/2023/swift.2023-01-18-21.01.txt | 22:01 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/swift/2023/swift.2023-01-18-21.01.log.html | 22:01 |
| mattoliver | thanks for all the hardwork on the gate timburke | 22:01 |
| opendevreview | Merged openstack/swift stable/zed: s3api: Prevent XXE injections https://review.opendev.org/c/openstack/swift/+/870825 | 22:03 |
| timburke | 🎉 i just realized! swift had it's 10,000th commit recently! 🎉 | 22:59 |
| timburke | of course, nearly half of those are merge commits -- but still, it's fun to celebrate round numbers :-) | 23:00 |
| opendevreview | Tim Burke proposed openstack/swift master: tests: Ensure XXE injection tests have config loaded https://review.opendev.org/c/openstack/swift/+/871005 | 23:16 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!