18:00:04 #startmeeting tc 18:00:04 Meeting started Tue Jul 18 18:00:04 2023 UTC and is due to finish in 60 minutes. The chair is knikolla. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:04 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:00:04 The meeting name has been set to 'tc' 18:00:10 Hi all, welcome to the weekly meeting of the OpenStack Technical Committee 18:00:15 A reminder that this meeting is held under the OpenInfra Code of Conduct available at https://openinfra.dev/legal/code-of-conduct 18:00:19 Today's meeting agenda can be found at https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee 18:00:22 #topic Roll Call 18:00:23 o/ 18:00:23 o/ 18:00:27 We have one noted absence: slaweq. 18:00:28 o/ 18:00:30 o/ 18:00:32 o/ 18:00:33 o/ 18:00:42 o/ 18:02:12 #topic Follow up on past action items 18:02:19 knikolla will amend the proposal for Unmaintained branches. 18:02:31 I have done so, and we can circle back on this item in the next topic. 18:02:50 #topic Extended Maintenance / Unmaintained 18:02:56 #link https://etherpad.opendev.org/p/openstack-exteneded-maintenance 18:03:01 I have updated the proposal from last week 18:03:06 #link https://review.opendev.org/c/openstack/governance/+/888771 18:03:15 I have renamed the status to Unmaintained. That’s what we already call branches that are EOL and this sends a strong signal about the differences between unmaintained and maintained, rather than introducing new words. 18:03:33 There’s a few things that we need to iron out. 18:03:42 I do like that symmetry, but someone else suggested obsolete, which I also like, fwiw 18:04:04 The transition from Maintained to Unmaintained: opt-in process, etc. 18:04:04 The transition from Unmaintained to branch deletion: criteria, timelines, possible grace periods, etc. 18:04:04 And the transition from branch delete to Unmaintained branch: tagging, etc. 18:04:40 I think unmaintained compare to obsolete especially with our 'Maintained' phase naming 18:04:53 * like unmaintained 18:04:58 I think the symmetry helps a lot. 18:05:10 yes, let's keep it simple 18:05:19 +1 18:05:21 And it reduces the perceived difference between Unmaintained with a branch, and unmaintained with no branch. 18:05:32 Namely, only the existence of the branch. 18:05:40 we have some suggestion and discussion on those open item in gerrit too 18:05:45 as long as we can handle the unmaintained-maint group name problem :P 18:05:58 unmaintainers 18:06:11 We can bikeshed on the group name in Gerrit :) 18:06:29 knikolla: can you give a quick description of the workflow from master->stable->unmaintained->EOL ... i am unclear on the branch renaming and deletion 18:06:30 we can rename that to unmaintained-core group 18:06:49 master->maintained->unmaintained->EOL 18:07:24 Stable to unmaintained: stable/ is deleted, tagged as -eol, new branch is created in unmaintained/ 18:07:55 ok, so EOL happens before unmaintained ? 18:08:00 how about all open changes ? 18:08:04 o/ 18:08:18 open changes have to be abandoned or merged first 18:08:30 gmann: good point, those would have to be recreated. 18:08:30 rosmaita: not EOL as such but a new branch for that to maintained as unmaintained 18:08:43 But we need this kind of friction as it sends a clear signal of lack of continuity. 18:08:54 This is now a new thing, in a difference status. 18:08:56 yeah, we need to cleanup open changes also, hope we do not have many 18:08:57 different* 18:09:41 just keep in mind we used to do something similar at the transition from pre-release to release, and stopped doing that because of the challenges it created, but hopefully the degree of churn around end of maintenance is not as great as around end of release candidate period 18:09:58 Then we're in territory of timelines, as changes could be valid backports, that needs to be merged but lack reviews 18:10:00 and it explicitly call out that existing EM model does not work so this is new model and who need these branches need to step up for maintenance 18:10:36 noonedeadpunk: that would send a strong signal to get those merged now, because this is the last opportunity to cut a release with those fixes. 18:10:39 noonedeadpunk: that valid backport can be re-proposed right 18:10:57 cherry-picked to another branch... but yeah 18:11:02 as whatever is still open at the time of transition will not be able to receive releases anymore. 18:11:23 if nobody's reviewing proposed backports on a branch, that's a good indicator that either there needs to be a different group reviewing them or the branch isn't serving a purpose 18:11:32 exactly 18:11:48 But then there's no sense to rename the branch kind of? 18:12:10 I am not sure I catched if moving to unmaintained is default behaviour or on-request? 18:12:20 renaming the branch is part of the marketing to ensure that users know if it breaks they get the pieces 18:12:23 or per-demand 18:12:33 noonedeadpunk: that's why i suggested just closing branches at that point, and recreating them later if a group of interested reviewers comes forward 18:12:34 since one of the big problems with EM right now is operator misunderstanding about what is supported in what ways 18:12:37 noonedeadpunk: renaming is marketing, but also to ensure we can use different group with different permissions. 18:13:23 the "renaming" happens in two basic steps already: delete old branch, create new branch from the previous branch state. you can stop after the first step and then do the second step later if there's interest 18:13:26 noonedeadpunk: as per automatic or manual, I'm open to feedback. So far I think we're leaning on manual. 18:13:28 on-request create more inconsistency and confusion. i think default behaviour make sense 18:13:47 The biggest difficulty I've had nailing down about on-demand based branching 18:13:59 is what do we do about tagging and communicating state of things 18:14:03 and it is one time transition we have to do which is not smooth or best for everyone but solve the long term problem 18:14:06 moving to unmaintained is dictated by the schedule, just as it is today right? 18:14:16 which I guess we have as a lesser problem when it comes time to retire the unmaintained/ branches 18:14:32 if you mean "unmaintained vs straight to eol" I think we need to do the former to allow for time for a group to materialize 18:14:48 gmann: ++ 18:14:54 a group which can later renew the branch every cycle 18:14:58 Also - are there technical reasons we don't wanna close ability to create tags on this new branch (do releases)? As that kinda would make sense, if we allow to have like post releases or smth like that? 18:15:04 because on-demand creation would have to fall on the project team. 18:15:17 or like development releases 18:15:17 So release [18 months pass] -> unmaintained/ automatically -> must be renewed every 6 months 18:15:22 I like that, because it builds in a grace period 18:15:27 ++ ^^ 18:15:27 yes 18:15:35 but still gives up opt-in style semantics 18:15:47 so after that first 6 months, nobody comes, unmaintained/ branch is retired -- two questions: 18:15:51 renewed every 6 month ? 18:15:53 1) What do we tag that final commit? 18:15:57 alternately, you get a little longer in unmaintained before you need to renew, but I'm not super opinionated about that I think 18:16:02 as long as it's one cycle for "free" 18:16:08 2) What do we do if someone doesn't renew it after 6 months but someone shows up a year later and wants to resurrect it 18:16:20 IMO we can go with the 4 or 5 years policy and after that move unmaintained -> EOL 18:16:23 noonedeadpunk: if teams are already uneasy about people getting the impression that these branches are still maintained, i expect they'd be even more uneasy about someone tagging more releases from unmaintained branches 18:16:30 yeah, I think the first free cycle makes sense. Was leaning towards that today while rewriting the patch. 18:16:32 JayF: right, that's why I think maybe a little longer default free period might help avoid needing to answer the resurrection question as much 18:16:41 I wrote here how it looks like if time based moving to EOL https://review.opendev.org/c/openstack/governance/+/888771/1/resolutions/20230707-extended-maintenance-new-requirements.rst#74 18:16:53 dansmith: the period of time matters less to me right now than the answer to those two questions, regardless of how long "grace period" is 18:17:01 JayF: ack 18:17:06 dansmith: I chose a cycle just b/c it's simpler to think in those terms since it's our automatic stopping point now 18:17:16 ack^2 18:17:17 main idea if to keep them open as much as we can 18:17:20 fungi: well, but if we're talking about oslo-ish, not being able to release some fixes is really a bummer 18:17:39 As then you can't have this in your constraints/requirements 18:17:56 or well.. you can, by SHA, but well... 18:18:15 noonedeadpunk: we already experience a version of that problem today as well, fwiw 18:18:29 on-demand basis have its own challenges on how fast we can communicate it operators to come and ask for unimantaiend branche 18:18:33 noonedeadpunk: given how some bugs in Ironic can be related to sushy (and I think os-brick/cinder are similarly tied) 18:19:11 Well, yes, so I was wondering why to prohibit some post/dev releases on unmaintained branches 18:19:38 As that would make more easy to consume, and hopefully to get some involvement 18:19:42 As an Ironic core and PTL of Ironic; I don't want software released that's called "ironic" that installs via `pip install ironic`, that may have been reviewed/maintained to a lesser standard 18:19:44 if teams are still tagging releases from branches, it seems like those branches are still actually maintained 18:20:04 which I thought was also the original impetus behind 'no releases from EM branches' as well 18:20:09 As what point of spending time on keeping alive unmaintained version if there is no straight way to consume there 18:20:12 *these 18:20:14 ++, if a specific project needs longer maintenance schedules, that's a different problem to solve than this one. 18:20:30 noonedeadpunk: there is a large number of larger deployers who are already setup to build and use their own downstream git repos 18:20:47 it sounds more like there's interest in some libraries staying in maintained phase longer than other projects 18:20:50 Unless it's standalone I can see them being potentially around longer 18:20:58 JayF: these would never contribute back to unmaintained branches 18:21:08 noonedeadpunk: that is decidedly untrue 18:21:08 Not even sure about maintained ones 18:21:24 noonedeadpunk: I've personally backported changes of that nature, at almost every place I've worked 18:21:32 if we do release it add testing things also 18:21:40 but what's the point of spending time when you still have to maintain the same thing downstream? 18:21:48 ppl like to do the work twice? 18:21:50 we need to test them well before release so I agree it move towards maintained phase 18:21:51 noonedeadpunk: the argument for having no-longer maintained branches is that downstream distributions tell us they want to collaborate on backports of fixes in those branches without burdening the project teams 18:22:22 noonedeadpunk: two reasons: 1) Changes that are not acceptable to backport upstream (features; object changes) and 2) downstream moves much, much faster than upstream -- you can solve your own problem easier than solving the problem for all cases generically 18:22:39 noonedeadpunk: I'm happy to talk about these use cases in depth in another venue; it seems off topic a bit for this discussion 18:23:10 noonedeadpunk: but I do assure you the process as I laid out is how it's worked literally everywhere I've worked that ran openstack :) 18:23:14 fungi: yes, I get that. The point is that the only deliverable openstack does release on their own, without unclear (or hidden) development in downstream, are releases in PIP 18:23:46 noonedeadpunk: even if those branches do not get the idealized level of contribution that we were hoping, there was a significant number of people who really wanted them still around, even without releases, and if we can do that in a way that clearly says "we as upstream are not maintaining this" and not much burden to our teams, I think we should. 18:23:55 So by doing effort in having such scheme with maintained/unmaintained, we disable ourselves to produce deliveravbles 18:24:15 Yes; but that's also the status quo with EM projects 18:24:19 we release tags in git, and source tarballs. the installable artifacts built from those (wheels, container images, et cetera) are not the releases 18:24:32 for fast discussion, I think we need another call to short out the open items one by one. I am getting mixed up here from what item we started to discuss and which one we are in :) 18:24:41 And I kinda find EM pretty much useless because of that 18:24:49 ANd we're trying to fix EMs 18:24:55 gmann++ 18:24:56 EM is useless to most of us, but not all of us :) 18:25:04 :) 18:25:23 I think we shuold all, individually, take an action to make sure our concerns are represented and asked in gerrit 18:25:31 so we can assess them in a threaded format where things won't get lost 18:25:40 or let;s have a another call 18:25:42 I get we're enabling downstreams, but I also want openstack to have some profit from that as well 18:25:43 then maybe from there go to another call 18:25:55 and out things in gerrit like we did for initial discussion 18:25:57 If I don't see progress towards a consensus in Gerrit by end of week I'll think about organizing another call for next week. 18:25:59 gmann: I know at least for me, I need that first step to get thoughts organized so we can have an agenda and avoid talking about everything all at once :D 18:26:04 or on the mailing list if the concerns aren't specific to a particular proposal 18:26:24 ++ sounds good knikolla 18:26:52 knikolla: but we need to add unmaintained->EOL in that proposal right which is open 18:26:57 In the meanwhile, to be more inclusive of the whole community, Gerrit and ML are great venues of collaboration. 18:27:23 gmann: yes. That part needs to be ironed out. 18:27:47 but do you want to merge the proposal without that and do that as follow up ? 18:27:59 Currently the various ideas are: 1) renew every cycle after the first X cycles, or 2) a set timed Y number of cycles unmaintained across all repos. 18:28:45 gmann: I would really prefer to merge something that clearly defines the Unmaintained -> branch deletion step. 18:28:55 As that will answer the Cinder question. 18:29:14 also, we need to address the SLURP issue 18:29:21 Yes, I like idea of coordinated approach... 18:29:28 knikolla: as this is resolution, I will suggest to do it in one shot instead of keeping ->EOL moving open which is important part to understand proposal as whole 18:29:34 But downstream maintainers obviously don't like that 18:29:46 i think seanmooney had a good point that we should *not* allow skipping branches 18:29:53 ++ 18:30:01 yeah 18:30:16 I se no reason we need to keep non-slurp branches between slurps, 18:30:18 that is why I am suggesting to have another call this week and figure out those things 18:30:28 I compromised on not skipping SLURP branches when revising the proposal. 18:30:32 if we're saying it's allowed to upgrade normally and skip the intermediate ones 18:30:47 otherwise the burden goes *up* in unmaintained 18:30:50 My only concern is that it's consistent 18:31:03 i am ok with the burden going UP in unmaintained 18:31:12 it makes people moving to SLURP model more 18:31:15 it'll get confusing if we have some SLURP unmaintained/ and some mid-slurp having/not having them inconsistently 18:31:35 I totally don't understand why it's okay to skip non-slurps during the maintained phase and not after 18:31:38 in fact, 18:31:48 let's skip them during development too 18:31:53 I'd go so far as to say we could just make it so that non-slurps aren't even eligible for unmaintained status 18:32:09 that will reduce the number of branches we could have in this status, and eliminate some process 18:32:34 to be fair, skipping non-slurps during development is a good topic to discuss... Another time:) 18:32:44 i.e. non-slurps go straight to EOL, no renewal allowed 18:32:59 I like that idea actually 18:33:01 I think it does not harm much. 18:33:03 dansmith: I like that almost as much as I like rosmaita's suggestion :D 18:33:11 the only reason we disallowed skipping branches previously was because we didn't support branch-skipping upgrades, so now that we do... i agree 18:33:14 I do think that's osmething we should explicitly check w/consituency for 18:33:21 This will also align downstreams more on versions 18:33:22 (gentle ping that I want to timebox this conversation at 40 mins past the hour to allow for the next item in the agenda) 18:33:22 fungi: right 18:33:25 keeping SLURP open for long term maintaining make sense 18:33:36 noonedeadpunk: yep 18:34:15 Ok, I will revise resolution to make non-SLURP branches ineligible for Unmaintained. 18:34:24 sounds good to me 18:34:28 -2 from me right now 18:34:39 can we schedule a another call tomorrow same time as our meeting? 18.00 UTC 18:34:46 i think everyone should re read sean's comment on the patch 18:34:54 I do not think I'll be ready for a call on this topic by tomorrow this time 18:35:10 otherwise it is hard to progress on gerrit with those items open 18:35:13 gmann: I don't think there is enough reason to justify a call on such short notice. 18:35:18 -2 because of the non-SLURP or that's where you're at currently rosmaita? 18:35:19 rosmaita: sean is talking about during the maintained phase no? 18:35:31 knikolla: JayF may be day after tomorrow ? 18:35:51 without that we are not going to progress much and will hang around whole cycle on this 18:35:53 I have to read the full proposal again and make a review pass, I'd rather have enough time for a little async figuring before going sync again 18:35:55 spotz[m]: becasuse of non-slurp 18:36:04 rosmaita: also sean describes "or closing the branch" which I think means straight to eol, which is what we're describing here 18:36:12 gmann: maybe monday before next meeting, perhaps, if you are champing at the bit to get one scheduled :D 18:36:14 dansmith: o dpmt 18:36:23 try to translate that! 18:36:30 ? 18:36:37 JayF: sure, I was thinking this week but Monday is ok also 18:36:37 gmann: I do want to see progress stall first before I call a video-meeting as it's not super inclusive to the community as a whole. 18:36:57 knikolla: I think we're talking all over ourselves right now, and a call would help clear that up a lot, FWIW 18:36:58 sure, let's have on Monday then 18:37:07 i thought he was talkinga aobut unmaintained, because the maintained branches are the normal stable branches that this proposal doesn't touch 18:37:08 this sort of thing gets pretty thorny with all the parallel threads here 18:37:13 yeah 18:37:31 Hence why there is a gerrit proposal which will be updated by EOD today to reflect these threads. 18:37:32 "we cannot skip non-slurp release either while the non-slurp release is still maintianed." 18:37:35 we are not going to get any outcome with IRC discussion, call is helpful in such situation 18:37:42 "we likely either shoudl not allow that or close the branch" 18:38:17 I think the first sentence means we should not change current stable policy, which is true, 18:38:25 and that we should not have an *open* unmaintained branch that gets skipped 18:38:42 and instead we should either require the backport to hit that branch too *or* close it so it's clear 18:38:59 and this would achieve the second thing there, which is no non-slurp would ever be in unmaintained state, so there's nothing to skip 18:39:59 So current open items: 1. unmaintained -> EOL process 2. SLURP/non-SLURP in unmaintained phase 3. release of unmaintained branches 4 ? 18:40:52 That's all the time we have for this specific topic. Please participate in the discussion in Gerrit. 18:41:00 gmann: that's a good summary I think. 18:41:09 #topic Release notes guidelines for SLURP/NON-SLURP cadence 18:41:14 Speaking of SLURP... 18:41:29 gmann: do you want to provide an introduction for this topic? 18:41:56 I was reviewing one of the nova change of AZ sch filter removal and then realized we have not figure out the release guidelines for SLURP/non-SLURP 18:42:12 this was one of the open items since SLURP model and is in c tracker also 18:42:15 TC tracker 18:42:38 #link https://etherpad.opendev.org/p/tc-2023.2-tracker#L47 18:42:48 iirc it was boiling down to tooling 18:43:01 and how to copy renos nicely from non-SLURP to SLURP kinda 18:43:10 yeah but I do not have much status on that 18:43:11 (and render these) 18:43:26 but at least we should finalize one approach and provide guidelines to community 18:43:33 i think we need to agree on the strategy before doing the tooling 18:43:44 this has been open for over a year: https://review.opendev.org/c/openstack/project-team-guide/+/843457 18:44:36 rosmaita: added that to my list, will read and comment soon 18:44:50 ++, added to my todo list for review. 18:45:14 ok so let's review that and figure out what is closed way to move 18:45:36 #action tc-members to review https://review.opendev.org/c/openstack/project-team-guide/+/843457 18:45:39 ++ 18:45:39 rosmaita: I remember you have some example patch also from cinder to show how it will looks like ? 18:45:40 homework for everyone :) 18:46:18 gmann: i dont' know if those still exist, i will look 18:46:23 knikolla: we're just cramming because the exam is coming soon ;) 18:46:41 rosmaita: no issue, just checking. 18:48:07 anything else on the topic? 18:48:37 Moving on 18:48:39 #topic Gate health check 18:48:44 Any updates on the state of the gate? 18:49:00 not good 18:49:16 Ironic has continued to focus on correcting some CI issues; things aren't great but they are better than they were 3 weeks ago at this time so... progress? I guess? 18:49:21 a number of things have been creeping up, guest kernel crashes, another volume issue it seems 18:49:35 Ironic actually ID'd a major issue: dnsmasq is failing in newer ubuntu 18:49:44 mkfs occasionally times us out formatting a volume and there are iscsi errors in the syslog and io errors on the guest kernel console 18:49:49 We are having to use the package from older ubuntu to avoid the *segfaults* killing the daemon 18:49:54 I've been rechecking one patch for days now 18:50:09 so if any of you are relying on dnsmasq (I think that's only Ironic?) beware the SIGSEGV 18:51:01 JayF: thanks for letting know 18:51:20 also, we have seen many timeout in last couple of weeks, I am refactoring the test run 18:51:24 like 1. running slow tests in parallel which helped to reduce tempest-slow job execution time to 60% 2. increase the concurrency of test runner #link https://review.opendev.org/q/topic:job-timeout+project:openstack/tempest 18:51:38 2nd one is failing on other failure like mkfs dansmith mentioned 18:52:31 but if that work at least it will help running tests on more worker. though it does not solve the worker test balance issue 18:53:34 thanks for doing that work. 18:54:37 # topic Reviews and Open Discussion 18:54:41 #topic Reviews and Open Discussion 18:54:49 #link https://review.opendev.org/q/project:openstack/governance+status:open 18:54:55 i found the review gmann was referring to (sample SLURP release notes) 18:54:55 https://review.opendev.org/c/openstack/cinder/+/840996 18:54:55 i just hit recheck to regenerate the release notes, but who knows if the job will pass 18:54:55 also, it's old enough so that it still uses the "tick-tock" vocabulary 18:54:55 "tick" == SLURP 18:54:57 "tock" == non-SLURP 18:55:08 rosmaita: great, thanks 18:55:23 awesome, thanks for finding that. 18:57:03 Unless there's anything else. I propose closing the meeting. 18:57:45 #endmeeting