18:00:39 #startmeeting tc 18:00:39 Meeting started Tue Apr 25 18:00:39 2023 UTC and is due to finish in 60 minutes. The chair is knikolla. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:39 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:00:39 The meeting name has been set to 'tc' 18:00:47 o/ 18:00:49 Hi all, welcome to the weekly meeting of the OpenStack Technical Committee 18:00:52 o/ 18:00:53 o/ 18:00:58 #topic Roll call 18:01:00 o/ 18:01:01 o/ 18:01:06 A reminder that this meeting is held under the OpenInfra Code of Conduct available at https://openinfra.dev/legal/code-of-conduct 18:01:10 o/ 18:01:12 o/ 18:01:12 Today's meeting agenda can be found at https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee 18:01:12 o/ 18:01:20 o/ 18:02:25 #topic Follow up on past action items 18:02:28 I don't see any action items noted down from the previous meeting 18:02:30 o/ 18:02:52 #topic Gate health check 18:03:01 Any updates to report for this week on the state of the gate? 18:03:16 lol 18:03:22 lot of updates :) 18:03:24 * bauzas rolls eyes 18:03:29 the gate is closed and locked ;) 18:03:43 the gate has been deathly ill this week, but things may be better after many reverts 18:03:45 yeah, it's in bad shape recently 18:03:55 still waiting for initial results 18:03:59 probably worth deferring discussion of the python 3.8 drops until the dedicated topic later in the agenda 18:04:32 also probably worth mentioning the ceph thing here 18:04:39 even though it's acutely related to 38, 18:04:48 Well, my lunch overlapped with the discussion of the last hour, so hold your eye rolls for now :) 18:04:52 I'm starting to become concerned about our ability to get ceph tested on jammy in short order 18:04:54 and the volume detach ssh findings ? 18:06:12 dansmith: if we bring back py38then bringing back focal make sense and ceph jobs will get time at least for this cycle 18:06:24 I also noticed a lot of failures due to issues with mirrors, especially last week but I don't know if that's somehow fixed already 18:06:25 knikolla: just read the scroll while the next person types:) 18:06:26 but yes, we need to sort out it for jammy at some point 18:06:29 bauzas: every time I look at those I'm seeing the guest's boot seems to have been sufficiently late, so not sure really, but yeah that too 18:06:41 I'm going to start an etherpad to track all these issues in a parallel manner, as (un)fortunately IRC is a linear medium. 18:06:46 #link https://etherpad.opendev.org/p/gate-health-2023 18:06:56 gmann: not really, IMHO, because if the PTI doesn't include focal, then we're not testing ceph on our supported platforms 18:07:13 gmann: the reverts have taken the pressure off, but we still need to figure out what we're doing here 18:07:19 slaweq: i'm happy to follow up on the mirror issues after the meeting, though the opendev sysadmins meeting is immediately after this one 18:07:56 dansmith: yes. going more closure to release make it more concern things 18:08:01 fungi I will go afk just after this meeting, sorry 18:08:04 I can talk about it tomorrow morning 18:08:11 slaweq: tomorrow is great, thanks! 18:08:17 I'm not familiar with the specific issues we're seeing on jammy/with ceph but I'm happy to help if there is anything that I can do from the downstream distro side as well 18:08:54 jamespage: no indication that it's distro related at least at this moment 18:09:20 well feel free to pull me in if needed :) 18:09:26 this is change which try to mvoe it to jammy #link https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/865315 18:09:26 ack, thanks 18:11:17 Please jot down things in the above linked etherpad, as we have a packed agenda I want to move on to the next item and we can focus on the gate outside of the meeting. https://etherpad.opendev.org/p/gate-health-2023 18:11:59 #topic Broken docs due to inconsistent release naming 18:12:19 I proposed a patch that renames the docs directory to 2023.1 instead of 2023.1.antelope 18:12:20 https://review.opendev.org/c/openstack/openstack-manuals/+/881290?usp=dashboard 18:12:39 looking 18:12:45 thanks, we need the redirect also there 18:12:47 There's still issues with releases.openstack.org being named antelope, so the links to there don't work 18:13:13 knikolla: I'll read through and +w if good after the meeting 18:13:18 I think we should make it /2023.1 as other project specific release notes are with relerase version not name 18:13:19 And yes, the redirects. 18:13:32 spotz_: there is comment there on redirect 18:13:32 That used to be a Jimmy thing but might be a Josh thing now? 18:13:37 gmann: ++ 18:13:51 gmann: Will look, saving reading until we're done:) 18:13:54 example #link https://docs.openstack.org/releasenotes/aodh/2023.1.html 18:14:09 which redirects specifically? 18:14:25 redirects from docs.openstack.org/2023.1.antelope to 2023.1 18:14:31 that isn't a foundation issue it is an openstack-manuals issue 18:14:32 basically this https://releases.openstack.org/antelope/index.html -> https://releases.openstack.org/2023.1/index.html 18:14:44 the foundation webdev folks have nothing to do with docs.o.o, never have 18:14:50 that's why i was asking 18:14:52 oh yes it is not foundation at all 18:15:07 it is in openstack-manuals we need to add redirect 18:15:17 it's an htaccess file in openstack-manuals, right 18:15:21 yes 18:15:23 yep 18:15:38 there is the separate issue where the index itself is built with broken links 18:15:48 this is also an openstack-manuals issue and I haven't seen resolution for this issue either 18:16:04 if there's anything you need changed on the openstack.org site pertaining to links to documentation or something, i'm happy to loop in the right foundation webdev folks 18:16:09 knikolla: is your patch change this too? https://releases.openstack.org/antelope/index.html -> https://releases.openstack.org/2023.1/index.html 18:16:20 or this page is from other place? 18:16:30 gmann, i don't believe it does. no. 18:16:40 i have to hunt down where that is. 18:16:54 ohk 18:17:08 that'll be the openstack/releases repo 18:17:09 I'm taking this work on and tracking the issues I find and patches here 18:17:10 #link https://etherpad.opendev.org/p/docs-issues-2023 18:17:17 Ok sounds like hold off on the review until we have all the pieces ready? 18:17:25 I'll add the index issue and release name in the therpad as well. 18:17:30 knikolla: may be this? #link https://github.com/openstack/releases/blob/master/doc/source/antelope/index.rst 18:17:40 there will be multiple changes to fix the various problems, because some are in different repositories 18:18:02 spotz: and it is not foundation things, it is in openstack-manual which used to be maintained by doc sig and now TC 18:18:18 gmann: Yeah I got that already 18:18:39 fungi: ++, there's a lot of interrelated things 18:18:45 we need to make sure consistency. 18:18:46 anyhow, feel free to reach out to me with new issues you find and checking the etherpad. 18:19:03 thanks knikolla for fixing it. will add in etherpad if I find any other 18:19:10 thank you gmann 18:19:12 #topic Following new release naming convention by packagers (UCA/RDO) 18:19:28 I'm not familiar with the topic, anyone care to elaborate? 18:19:45 I think it was added during last week meeting. noonedeadpunk ? 18:19:56 if I remember correctly 18:20:17 yup. so the case here is that uca/rdo does name repos as antelope instead of 2023.1 18:20:37 well, those are distros 18:20:39 bearing in mind that uca and rdo are not governed by the openstack tc 18:20:45 yeah that 18:20:45 yeah :) 18:20:58 we can't really dictate their names 18:21:01 When I talked to RDO folks they were reffering our docs that both names are valid 18:21:06 i think here proposal is to have consistency which can be more better 18:21:09 but they did not mind changing that 18:21:32 tc does not need to force them but giving recommendation for consistency is good 18:21:32 So we could give some recommendations or ask them 18:21:34 noonedeadpunk: well, all the stable branches now proof themselves on how a release shall be named 18:21:38 noonedeadpunk: ++ 18:21:38 if distros could and want change it, that would be great IMO 18:21:49 yeah 18:22:33 so spotz and jamespage are here and I guess they might know some insides and if it's possible to be consistent here from their side 18:22:38 it may make sense to have some guidance on a recommended way to label our releases, and on how to combine the release number and the marketing moniker for the release in the least confusing way possible 18:22:40 makes sense. it's good to be consistent ourselves moving forward so that provides clear guidelines for others as well. 18:23:14 I can check with coreycb on the UCA side 18:23:21 noonedeadpunk: We should be able to rename directories and such for RDO 18:23:42 since the codename gets a lot of press and ends up in news articles about each release, i can understand downstream distributions wanting to include it for clarity 18:23:42 ++ thanks jamespage spotz 18:24:14 Yes, so where I agree is that codenames are closer to ppl then release numbers 18:24:14 ++ 18:24:38 But double naming has potential of raising huge confusion 18:24:53 in Ubuntu codenames and release versions are used fairly interchangeable 18:24:59 noonedeadpunk: well, fwiw, even Nova will switch to release numbers in Launchpad by 2024.1 18:25:13 for examples, we can look at how distros like debian and ubuntu combine their release codenames and release numbers for public-facing communications 18:25:14 supporting both is creating more problems 18:25:15 agree fully, noonedeadpunk 18:25:31 most of the internal technical plumbing is codename based by the release uses the version 18:25:34 As indeed I hardly imagine in regular talk of 2 operators see how they will use release number when codename is present 18:25:34 bauzas: true. 18:25:52 fungi: once we have fixed docs, I want to look into building the infrastructure for that. The current docs tooling didn't support the name being different than the branch. 18:26:03 I just feel that the nickname can continue to exist, provided we don't have it as a *reference* 18:26:11 knikolla: yeah, that makes sense 18:26:14 But it makes a lot of sense to create the possibility to decouple the displayed name and the branch name. 18:27:04 bauzas: I think it might still show up on like the releases page and referenced on the index page as a project. Kind of how you type out a fullname then abbreviate after it 18:27:28 in ubuntu we stuck with the codename antelope for the cloud archive since it aligns with use of the ubuntu codename in /etc/apt/sources.list 18:27:59 at least the TC reference is clear : release numbers are prioritary and should always take precedence over codenames 18:28:04 using the release numbers in urls, suite names, package mirrors seems to be the tc's recommendation, but shouldn't exclude the possibility to also mention the release name in more "cosmetic" parts of documentation and the like 18:28:21 ++, i think we already have a resolutions pushing for numbers. 18:28:26 yes 18:28:30 fungi: +1 and fwiw I feel the TC reference to clearly stating this already 18:28:40 coreycb: yes, but would it be possible in future releases to switch to release version? 18:28:53 so it's mainly a matter of turning that into clearer guidance for downstream distributions and communicating it to them 18:28:53 @link https://governance.openstack.org/tc/resolutions/20220524-release-identification-process.html 18:28:58 As we're trying to align on using that instead of codenames across all of the docs 18:28:58 #link https://governance.openstack.org/tc/resolutions/20220524-release-identification-process.html 18:28:59 knikolla: even a TC reference page, not only a TC resolution (which is sometimes hard to find) 18:29:12 thanks gmann 18:29:16 #link https://governance.openstack.org/tc/reference/release-naming.html 18:29:20 bauzas: ^^ 18:29:35 btw we don't inlcude there any guidance for downstream maintainers 18:29:43 noonedeadpunk: it's certainly possible if there's a good argument for it 18:29:44 yes, that's what i'm saying 18:29:49 anything else on the topic? 18:30:39 So should we add some guidance to release-naming? 18:31:06 I'd be in favor of such guidance. 18:31:18 it would make sense to add a section about downstream redistribution recommending use of the release number for things like mirror directories, suite identifiers, and so on 18:31:27 that will be good if we can mention it explicitly about what we recommend for downstream 18:31:27 ++ 18:31:32 things that matter to distributions 18:31:41 right now it's focused on things that matter upstream 18:31:46 yeah 18:31:50 coreycb: would that be a good argument? 18:31:52 most importantly it reduces confusion for the users of those distributions 18:32:13 noonedeadpunk: sorry, which point? 18:32:41 With the current docs inability to reference both 2023.1 and antelope on the same line it's not immediately obvious to new users who might be using something downstream or from another vendor. 18:32:46 If TC will add suggestions for maintainers of downstream packages on naming of OpenStack releases to https://governance.openstack.org/tc/reference/release-naming.html 18:33:08 coreycb: about adding a section to the release-naming document providing guidance to downstream package maintainers and other redistributors 18:33:47 at least that will help for future release if antelope cannot be fixed at this stage 18:34:04 anyone volunteers to take the action to propose the addition to the reference? 18:34:08 Yeah, for Antelope it's too late at this point 18:34:12 I can 18:34:22 fungi: sure but I wouldn't mind getting a chance to review the recommendations 18:34:53 #action noonedeadpunk to propose a patch to reference that makes the recommendation for downstream packagers to use the version name rather than codename. 18:35:08 thanks noonedeadpunk 18:35:17 coreycb: they'll be in gerrit! ;) 18:35:27 We can continue the discussion on Gerrit and fine tune the specifics there. 18:35:27 I'll add you to reviewers explicitly 18:35:30 coreycb: we will add/ping you 18:35:33 +1 18:35:51 Moving on to the next topic 18:35:54 #topic Schedule of removing support for Python versions by libraries - how it should align with coordinated releases (tooz case) 18:35:57 and rdo folks as well :) 18:36:08 (and zigo) 18:36:08 noonedeadpunk: thanks for noticing it 18:37:24 some of us have already discussed this to death at this point 18:37:31 I know you talked extensively about this before the meeting and I haven't been able yet to catch up on the conversation. 18:37:33 yeah 18:37:33 yes, so this is the topic that was heavily discussed even before the meeting 18:37:39 I wonder if maybe it would be better to have one person put up a proposal and we can discuss from there? 18:37:51 dansmith: ++ 18:37:58 you wanna do that dansmith? 18:37:59 yeah, i started the discussion two hours early in hopes we could get to some shared consensus and not consume the whole meeting with it 18:38:00 can I try ? 18:38:01 +1 dansmith you want ? 18:38:05 noonedeadpunk: I was going to suggest you :D 18:38:11 ah, ok, I can 18:38:22 I'm pretty slammed with the fallout from this still 18:38:33 if you're willing, that would be cool, but I can if you don't want to 18:38:57 at least I'm more than happy to contribute to it 18:39:19 I think noonedeadpunk is typing the proposal :) 18:39:21 would a quick recap be useful for the meeting logs? 18:39:23 I'm good but bauzas seem to be eager to :) 18:39:52 I would also encourage sending a message to the mailing list with a link to the proposal on Gerrit to invite wider awareness 18:40:13 I think one thing that should be done right now is get the release team tostop making releases 18:40:18 fungi: seems appropriate 18:40:20 knikolla: ++ 18:40:32 to stop the bleeding and allow the proposal(s) to be worked through 18:40:32 so who is putting proposal here? 18:40:36 fungi: ++ to a quick recap, i still have like 50 lines left to read in the scrollback 18:40:47 clarkb: that that's probably a good idea 18:41:01 I can summarize where I think we landed 18:41:32 please do summarize as I didn't read previous discussion at all :) 18:41:37 quick recap then: we have a pti that does not mention us testing/supporting the 2023.2 release on python 3.8, and that has resulted in some projects eagerly removing their 3.8 testing and marking themselves as not installable on 3.8 18:42:00 #link https://governance.openstack.org/tc/reference/runtimes/2023.2.html 18:42:05 I think (a) recommend that we not aggressively bump the python version minimum to meet the PTI (b) keep some minimal like unit jobs on older python versions to ensure language compatibility even beyond supported platforms, 18:42:07 this makes them no longer coinstallable with other projects who are in the process of trying to remove 3.8 jobs but haven't finished doing so yet 18:42:08 plus we need probably to clarify what PTI is 18:42:41 some projects have very legitimate reasons for still running 3.8 jobs, including issues getting ceph working on 3.9 18:42:49 (c) recommend wider support amongst "library packages" (for whatever that means) and (d) perhaps better define how the scheduling of bumps like this should be digested in a cycle where they're changing 18:43:02 ++, mentioning explicitly it in PTI and we also be less aggressive to drop older py from testing runtime 18:43:05 also nested virt issues with ubuntu jammy in one of our test node donors 18:43:07 dansmith: I think you captured the 4 actions we discussed 18:43:12 yes, ++ dansmith 18:43:20 * dansmith takes a bow 18:43:23 dansmith: ++, all four 18:43:46 so anyway, yes, some coordination would be useful in order to stop projects from breaking each other's testing while removing python 3.8 testing of their own 18:44:19 in the near term, we need to stop the bleeding, longer term we would benefit from having a clear process and schedule for such things over the course of a development cycle 18:44:20 the #3 is more a recommendation to say "if you think you're imported, consider to bump your minimums probably at the latest rather than at the earliest" 18:44:30 and do we want to bring back py38 in testing runtime as it is not very mandatory to drop at least in this cycle ? 18:44:50 gmann: thought it was captured by action #2 18:44:59 gmann: yes, I do 18:45:06 bauzas: right 18:45:11 I hope people haven't already started taking advantage of 3.9 features 18:45:21 knikolla: hahaha 18:45:25 ok, so we will add it in general template not just recommend to test 18:45:37 by 3 weeks ? man, this is not eager, this is avid ! 18:45:54 not being* eager 18:46:21 :) 18:46:21 gmann: I think those 4 actions have to be documented in 2023.2 PTI 18:46:41 bauzas: yes and update testing runtime for 2023.2 18:46:52 isn't that what 2023.2 PTI means? 18:46:56 and how about focal? do we want to add it as best effort testing in 2023.2 testint runtime? 18:47:04 gmann: noonedeadpunk is opposed to that, 18:47:15 and I'm okay with that, modulo my concerns about ceph 18:47:17 Yeas, don't like that idea at all 18:47:18 we could also do a better job of communicating that the pti is how we describe the end result, it's not the process for getting there 18:47:21 dansmith: I mean few of the things you mentiond can go in general PTI documentation also to follow in future also 18:47:33 gmann: ack 18:47:37 dansmith: noonedeadpunk: yeah because of ceph 18:47:38 gmann giving the fact that we have those issues with nested virt on jammy I think we should add Focal back as best effort 18:47:50 slaweq: ah that too 18:47:55 because still some projects are doing that actually :) 18:48:02 gmann: no, because it won't work for nova and neutron at very least 18:48:06 slaweq: the problem there is that nova was going to drop support for focal's libvirt like weeks ago if allowed 18:48:09 slaweq: so it becomes sticky 18:48:14 gmann: oh, you'd prefer to enforce those recommendations in the global PTI doc ? 18:48:16 https://governance.openstack.org/tc/reference/project-testing-interface.html 18:48:31 as both of them require more modern versions of qemu/libvirt/ovs/ovn/etc then is provided 18:48:59 bauzas yeah we should do that so that we avoid this situation in near future, especially less aggressive on drooping older python from release pti 18:49:11 yeah, neutron wanted to start requiring ovs built from downloaded source code last cycle, right? 18:49:12 gmann: ah yes, I confuse the current and global PTI documents.. so I agree, both 18:49:26 noonedeadpunk: dansmith ohk. in that case I agree it is not easy to add focal then 18:49:27 gmann: cool with that, this makes them generic 18:49:36 gmann: also, if we add focal back, we will need to carry it till 2024.1, just in case 18:49:52 yup, because B is non-SLURP 18:50:16 I dunno that I agree with that, 18:50:17 humm, true 18:50:26 I think we haven't discussing the upgrade scenarios in a skip-level release but we're consistent 18:50:33 but if we're not adding it back, no need to argue about it :) 18:50:35 that might also ease the fips testing situation, since there's no ubuntu fips support for jammy yet? 18:50:45 fungi: yeah tht's true... 18:50:56 but again, nova would have to not do its min version bump on libvirt 18:50:56 yeah, let's not do for focal. 18:51:07 it won't offend me, but sean will be quite unhappy with you people :D 18:51:13 ok, makes sense 18:51:32 I think let's go with py38 things and dansmith proposals of those 4 points 18:51:40 and I think this might hold on virtiofs support or make it tricky... anyway 18:51:47 noonedeadpunk's proposal of my four points right? 18:51:55 ++ 18:52:06 feel free to refer to them as "The Smith Plan(tm)" 18:52:12 :) 18:52:21 I will not charge royalties for the use of my four points until at least 2030 18:52:23 :) 18:52:31 noonedeadpunk: not sure I see the implications about virtiofs support but meh, not the right chan and time to discuss this :) 18:52:40 dansmith or noonedeadpunk, either of you wants to write the required changes? 18:52:47 knikolla: noonedeadpunk is writing them 18:52:59 ++ 18:53:05 ++ 18:53:16 i can do general job template changes once governance things are merged 18:53:25 sweet 18:53:42 #action noonedeadpunk write the words for "The Smith Plan(tm)" (the script of the movie about changing PTI and saving the world from the dangers of getting rid of py38) 18:53:44 and olso (possibly others) are going to need to revert changes in repos 18:54:06 knikolla: I did not release the screenplay rights, to be clear 18:54:07 I reached hberaud earlier today 18:54:10 and the release team needs to put a hold on release requests for things that merged changes to mark themselves uninstallable on 3.8 18:54:10 yeah. and if any other projects has done that 18:54:29 he knows the situation so I hope he's aware of the implications of a new release 18:54:41 I think we'd need to come up with ML as well quite ASAP 18:54:46 bringing back the check-uc reqs job for py38 asap would be good too 18:54:47 but I can loop back tomorrow and discuss with elod and hervé 18:54:55 fungi: that's already merged IIUC 18:54:57 let's push the governance changes, merge then and ML 18:55:01 But won't be able to do that until tomorrow afternoon just to have that said 18:55:03 as in, like minutes ago 18:55:12 dansmith: just while we were discussing? awesome! 18:55:19 Seems like there's a lot of moving parts, noonedeadpunk, do you want to also start an etherpad for dealing with the fallout? 18:55:26 fungi: https://review.opendev.org/c/openstack/requirements/+/881433 18:55:41 perfect 18:56:00 we flap our gums and frickler gets sh*t done 18:56:13 hehe 18:56:37 #link https://etherpad.opendev.org/p/the-smith-plan 18:56:45 brilliant! 18:56:55 haha 18:56:56 we should not forget to clean the link up until 2030 18:57:01 100% rotten tomatoes 18:57:15 s/until/before 18:57:15 I will consider renewal plans six months ahead of the deadline 18:57:16 I can push it on ML asking project/release team to hold dropping the py38 until we get the pti change merged 18:57:38 noonedeadpunk: straight copy the etherpad content into a gerrit patch and then I'll happily +1 it :) 18:57:46 on* 18:57:47 haha 18:57:48 alright, we're almost out of time. 18:57:57 #action gmann send an email on ML asking project/release team to hold dropping the py38 until we get the pti change merged 18:58:04 I'm glad.. I'm so sick of py38 at this point :) 18:58:17 I think we have a good plan forward. 18:58:20 :) it seems more than py27 18:58:28 hah.. too soon. 18:59:20 Thanks all for a productive meeting. We're out of time. 18:59:21 move to adjourn? 18:59:25 #endmeeting