17:59:04 #startmeeting tc 17:59:04 Meeting started Tue Jul 25 17:59:04 2023 UTC and is due to finish in 60 minutes. The chair is knikolla. Information about MeetBot at http://wiki.debian.org/MeetBot. 17:59:04 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 17:59:04 The meeting name has been set to 'tc' 17:59:21 Hi all, welcome to the weekly meeting of the OpenStack Technical Committee 17:59:25 A reminder that this meeting is held under the OpenInfra Code of Conduct available at https://openinfra.dev/legal/code-of-conduct 17:59:29 Today's meeting agenda can be found at https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee 17:59:33 #topic Roll Call 17:59:35 o/ 17:59:36 knikolla: you are in a hurry to get started! 17:59:37 o/ 17:59:37 o/ 17:59:43 o/ 18:00:03 o/ 18:00:07 o/ 18:00:08 o/ 18:00:33 rosmaita: haha, a bit, yes. 18:00:42 o/ 18:00:58 No noted absences for today. 18:01:08 o/ 18:01:31 and everyone's already here, nice. 18:01:43 #topic Follow up on past action items 18:01:51 knikolla will amend proposal for Unmaintained branches. 18:02:09 I have, we can talk more about it during its respective topic. 18:02:14 tc-members to review https://review.opendev.org/c/openstack/project-team-guide/+/843457 18:02:35 I haven't had the time, I hope others had a chance to take a look. 18:03:01 got some good comments from gmann 18:03:15 the idea lgtm but we need to see the automation of it otherwise it requires extra manual work which might endup project going in just link/copy the previous release notes 18:03:16 Awesome. We can go in more detail during its respective item as well. 18:03:17 Moving on. 18:03:28 #topic Extended Maintenance 18:03:34 yeah ++ for some automation concept at least on paper 18:03:34 #link https://review.opendev.org/c/openstack/governance/+/888771 18:04:00 Got some really good discussion on the Unmaintained governance patch and I made revisions to bring it aligned to our previous discussions and suggestions. 18:04:18 Please take a look and let me know if you have any questions or things you'd like to bring up related to the topic. 18:04:23 I haven't checked since yesterday update :( 18:04:35 thanks, latest version lgtm 18:04:52 there are still some things to clarify there 18:05:04 but those aside I'm good with what's there now 18:05:37 i'm still a bit unclear about whether the unmaintained branch is created automatically, or is opt-in 18:05:44 Especially related to the processes of opt-in and opt-out. We just mention that there'll be a document to refer to. 18:06:05 for current EM, it is 3 automatic opn-in 18:06:11 Creation is automatic. But a team can opt-out after it's created similar to the EOL process right now. 18:06:22 but that seems not alligned with what rosmaita mentioning of cinder team plan 18:06:28 (at least that's what I imagine, obviously open to feedback) 18:06:31 I really liked the proposal, as written, where the branch gets created by default the first time then retired if not renewed. 18:06:51 For teams like Cinder, can't they just ignore the unmaintained/ branch, just as we were hoping folks would for EM if nobody came around to help? 18:07:09 so no automatic creation for cinder ? 18:07:10 well, why create it in the first place if no one is going to use it? 18:07:17 seems like the end-of-month deadline for getting a recommendation back to the cinder team is rapidly approaching (august starts one week from today), just as a time-check 18:07:18 I also think that with the rename to Unmaintained, there should be significantly less reasons for a team to opt-out, but more likely just to not to renew. 18:07:38 If we haven't solved that basic ill: a project team feels too strong of ownership to leave a branch unmaintained, then we still have a problem, yeah? 18:07:41 So I would hope that Cinder reconsiders. 18:07:48 rosmaita: let me tell you why I like it: because we demonstrate that those people don't exist 18:07:55 knikolla: ++, that is what I understood. it will a good signal on hey we had unmaintiand brnahces created and no one show up to shutting down 18:08:04 rosmaita: there is value in having an actual year-old stale branch to point at when people complain about you shutting things down 18:08:09 rosmaita how can we know from the beginning that nobody is going to use it? Maybe people will step in during e.g. first cycle when branch is unmaintained? 18:08:19 IMO we should give them a chance for that 18:08:27 slaweq: yeah, that also possible 18:08:32 well, i am judging by the silence on the ML about the current cinder EOL proposal 18:08:34 conversely though, why create it until someone expresses an interest in having it? 18:08:44 yeah, and also it helps us avoid the situation where someone wants to re-open an EOL'd branch 18:08:51 and when will see there are unmaintained brahcnes for nova and they need cinder too so they might show up to maintain 18:09:23 fungi: i think it lowers the barrier for someone that has an interest. 18:09:37 but doesn't put undue burden on the team itself. 18:09:53 rosmaita: bluntly, I mostly agree with you here and would rather a dead branch not be created in the first place; but I think we have to demontrate the dire-ness of the circumstances around those old branches to get the mindshare to change further 18:09:55 rosmaita: and it will not ask project team to maintain it, it is just similar to what cinder team asking in ML. a better communication of asking maintianence who need it 18:10:22 JayF: we do not know if that will be dead or not right? by creating those we will get to know fr sure 18:10:36 gmann: If someone wanted that branch to be alive, they've had months to step up at this point 18:10:50 JayF: we're talking about future branches though right? 18:10:52 gmann: I am convinced those contributors don't exist; but I'm also convinced that words in IRC aren't likely to change minds here 18:10:58 JayF: ML is not always perfct place to communicate what we want. it is just a one of the way 18:11:04 dansmith: I am using the past to predict future events :) 18:11:10 ok, so how is this going to work for cinder right now? we are proposing EOL of train, ussuri, victoria, wallaby, and xena 18:11:27 i guess train and ussuri are fine to go EOL & be deleted 18:11:34 my point is Cinder communicated it on ML and let's do the same via new process of unmaintined branch creation 18:11:45 I actually +1 to Seans proposal and consider Y applicable for unmaintained 18:11:48 rosmaita: yes, only Victoria, wallaby, xena for now 18:12:11 To be same "testbed" as for SLURP 18:12:14 And Yoga? 18:12:14 so, v, w, and x will go to 'unmaintained' along with everyone else? 18:12:42 If Cinder EOL their branches before we cut the unmaintained branches, no branches will be created for them, based on current language of the proposal. 18:12:57 rosmaita: only v,w,x will be unmaintined automatically and rest all not 18:13:03 JayF: I see 7 pending backports from people not associated with the core team against nova's stable/wallaby right now.. so let's not say they completely don't exist 18:13:19 based on current language only SLURP will have unmaintained, which starts from Antelope 18:13:28 dansmith: I'm speaking for the Cinder case 18:13:45 noonedeadpunk: I added a new section in the proposal in the bottom about current branches. 18:13:52 rosmaita: asper current proposal we will create unmaintained/victoria , unmaintained/wllaby, and unmaintained/xena as automatic opt-in and test all are not not automatic opt-in 18:13:52 dansmith: Part of why there's value in treating them separate in policy; Cinder has fewer resources than Nova generally, so supporting branches is a larger burden 18:14:06 s/test/rest 18:14:07 JayF: I don't think they do lately, fwiw 18:14:22 JayF: cinder team do nto need to support 18:14:32 cinder looks to me to have a number of similar backports proposed to wallaby 18:14:51 just saying, we're trying to clear the air here and open up some space to make it easier for more of those people to be here, 18:14:57 it is just opening unmaintained branches and project team will not do any work there. if no one shows up then unmaintianed branches move to EOL 18:15:08 so we shouldn't say "well it was super hard and obscure how to do this in the past, so those people must not exist" 18:15:45 dansmith: that's why I have been saying, as a demo, I'm onboard for opening the first year universally and seeing what happens :) I don't think anything will happen but I've been wrong in the past will be again :D 18:16:22 mkay, seems like you're saying the opposite, but ... okay 18:16:23 knikolla: ah. sorry, haven't finished yet... 18:16:58 that is what we are tryting with this new model, instead of shutdown EM concept, let's open up the things one more time but this time with people gets what they signup to maintain it 18:17:16 dansmith: I'm saying I empathize a lot with rosmaita's position and could easily hold it if the majority was there :) I do hold that if a bunch of people disagree it's a sign that maybe more caution is required :D 18:18:17 rosmaita: what extra work you think on cinder team with these 3 unmaitained branches ? 18:18:36 after initial setup 18:19:02 well, the PTL has to assign an unmaintained liaison. that would be on the Cinder team. 18:19:08 i'm just trying to understand the proposal and how these branches will be killed off eventually 18:19:25 knikolla: just liaison but nto maintenance right? 18:19:28 knikolla: so lets say, in Cinder's case, there is nobody vvolunteering to be a liason 18:19:44 knikolla: what happens then? Because essentially Cinder saying "we only want to care about supported branches" is implying that's extremely likely 18:19:59 it is just a liason to keep track of unmaintained branch not to MAINTAINED them 18:20:01 JayF: in that case the PTL has the Unmaintained liaison role, but nobody is interested in Unmaintained so no action item for the PTL at all. 18:20:21 Branches get created and killed off automatically with no intervention 18:20:23 so PTL at least has to be traffic cop for unmaintained/ 18:20:33 for a year, ushering anyone interested into the core group 18:20:43 if that's the case, I'd want extremely clear guidelines saying "give access" 18:21:02 to be frank, populating unmaintained group manually sounds like a pita for small projects 18:21:17 I do not want to be in a situation, as Ironic PTL, where the community wants something retired but Shade E. Character comes along and says "I'll take over this unmaintained branch" 18:21:20 PTL assigns Unmaintained Liaison. Unmaintained Liaison manages the ACLs for the unmaintained-core group. 18:21:27 noonedeadpunk: it is just 1 branch with SLURP (3 for now pre-SLURP) 18:21:29 I would add some default there and then anyone who wants on top 18:21:44 gmann: we have plenty of projects just with 2-3 maintainers 18:21:53 (at best) 18:22:07 we have plenty of projects where only 2-3 maintainers would do that kind of maintanance :) 18:22:10 Would it be helpful to have a TC managed unmaintained-core across all project Unmaintained branches, for known good actor operators? 18:22:16 knikolla: yes 18:22:18 so saying them that to unmaintained they need to populate the group each time is kinda meh... 18:22:18 noonedeadpunk: yes but there is zero maintained work on unmaintained branches from project team. 18:22:34 knikolla: As PTL of $project, giving me decision power over unmaintained branches means that I'm still responsible for them 18:22:35 why we think adding someone in core group is lot of work 18:22:44 gmann: it's not work-work, it's responsibility-work 18:22:45 but why to prohibit maint team by default to vote if the want to? 18:22:53 *if they want to 18:23:00 JayF: which is 30 sec in a 6 month 18:23:02 gmann: if I give access to ironic-unmaintained-core to someone who abuses it, that's going to feel like I screwed up 18:23:09 noonedeadpunk: because if they can they will, through a sense of responsibility. 18:23:14 like reviewing some backport might be easy enough if they have permissions, but for PTL allowing that is pita 18:23:21 gmann: that's what I mean, not that it's work to click the buttons; it's explicitly giving responsibility that PTLs are trying to disavow 18:23:26 JayF: that can heppen in current core also what we do in that case? 18:23:38 gmann: PTLs signed up to maintain core lists for maintained branches :) 18:23:46 knikolla: sense of responsibility somehow is not applicable to PTLs? 18:23:51 if the PTL wants, they can include the entire stable-maint group. 18:23:56 yeah, so what is extra respo in that 18:24:04 this is all about trying to ensure the existing maintenance group can disavow themselves of a unmaintained/ branch 18:24:16 if that group is the keyholder to that unmaintained/ branch, then they are responsible 18:24:21 Some projects already include -core in -stable-maint. 18:24:25 PTL do this task for any branch core group in that project 18:24:50 let's do like this 18:24:51 I really don't get that point of repsonsibility 18:25:24 JayF IMO we are going a bit too much into what can possibly go wrong, currently for many smaller projects we, as TC are assigning PTL after election just because someone raise hand and said that will do it 18:25:25 noonedeadpunk: I'm having a hard time explaining it because it's something that's really core to me personally, so I've never had to tease it out. If you give me the keys to something, and I open it for them, I'm responsible for what they do with that access. 18:25:30 1. assign stable-maint group (core stable group nit project stable group) to unmaintained branches as default 18:25:30 2. they can add new people if there is any 18:25:35 3. n work on PTL 18:25:38 no work 18:25:44 in such case that person can not only screw unmaintained branches but also master 18:26:47 distilling the burden of responsibility into the tasks which go along with that responsibility is missing a large chunk of what responsibility means. even being responsible for something which has no accompanying tasks can still represent a burden on someone (being responsible means you're held accountable for all related decisions and indecisions). or maybe people mean something other 18:26:48 than accountability when they say "responsible" 18:26:54 JayF: well, all projects are community driven. Having no personal responsibility is basically header of each file in each repo under apache 2.0 18:27:03 gmann: i want to avoid having -stable-maint be included by default because there are already people who look at the group of who has approval powers on a branch and send an email to all of them. 18:27:05 slaweq: everything you said I agree with; but I come to a different conclusion (mainly I wish we appointed less PTLs, but I can't fix that :D) 18:27:16 IMO, PTL/liaison keep track of unmaintained branch and add people is just a very small resp and work. 18:27:21 fungi: you nailed my concern, on the button 18:27:42 gmann++ 18:27:45 knikolla: yeah, that is true 18:27:54 fungi: do you know if PTL can jsut add group to group rather then list everyone explicitly? 18:27:59 fungi: and almost definitionally, if these folks are outside of the existing community, $PTL has little/no context to know if they are good or bad actor 18:28:07 As I can't recall that being possible 18:28:12 i'm pretty sure they can, cause we have already done that in keystone. 18:28:24 or well, at least without pushing a patch to project-config 18:28:30 I'm 99% sure that's possible in the web ui 18:28:32 let's do this and if we see PTLcannot do this then we can discuss. but this is very small part of the proposal solving many thjings 18:28:40 ok, gotcha 18:28:46 I think this is the nexus of the change though, right? 18:28:49 https://review.opendev.org/admin/groups/ed7e98cd14f0ec4dfd09d10bbb64e1bd3ac1fa15,members 18:28:55 noonedeadpunk: yes, groups can include other groups in gerrit. an owner of a group can set another group as included (i'd need to test whether they can set a group as included without being a member of that group though, if that's important to the question) 18:29:01 The core problem: project teams feel ownership and responsibility over branches they shouldn't have ownership and responsibility for 18:29:14 we can't punt the question of responsibility when that's one of the big fulcrums this whole problem rests upon 18:29:23 noonedeadpunk: ah, yes, I see that now 18:29:24 so as far as branches go, is this what we are agreeing to? https://etherpad.opendev.org/p/24gf87QcmV6xF4StbLQx 18:29:50 noonedeadpunk: but also gerrit acls can inherit from other acls, so things for unmaintained branches could be delegated to a central acl through inheritance 18:30:15 honestly, a centralized, instead of per-project, UM core group could make sense 18:30:19 have TC or a TC liason manage it 18:30:21 we already have openstack project acls inheriting from a general openstack acl that adds release manager perms 18:30:31 rosmaita: and I guess we delete Stein/Train now? 18:30:32 you can safely assume operators would be running >1 openstack service, and might have interest in >1 18:30:50 noonedeadpunk: well, i'm just talking about cinder 18:30:56 aha, ok 18:31:08 rosmaita: nice, all good except when 2023.1 is unmaintined https://etherpad.opendev.org/p/24gf87QcmV6xF4StbLQx#L19 18:31:16 rosmaita: pretty much correct with the exception of, when 2023.1 becomes unmaintained, all other branches are removed and need to be opted-in. 18:31:18 But we need to have plan for other projects as well:) 18:31:44 knikolla: ok, please go ahead and correct the etherpad 18:32:08 updated. 18:32:20 thanks, i somehow missed that in the proposal 18:32:23 rosmaita: once we have 2023.1 goes to unmainited it is super easy to have only 1 unmaintained 18:33:09 ++, updated the etherpad to reflect that. 18:33:18 thanks 18:34:54 so, we agreed to EOL S/T/U in 2 weeks from now I assume? 18:35:10 just want to verify that I'm correct that keeping stein/train/ussuri are optional, but x/y/z is required? 18:35:14 rosmaita: do you feel that will make the cinder team willing to keep xwv around? 18:35:24 So for LN 23-25, are those opt-in too? or do they have to be forced to eol those? 18:35:47 rosmaita: i removed language that said keeping anything around is required. 18:35:58 manual opt-in is always option, we will not force to delete them if anyone interested to maintain he, 18:36:00 them 18:36:09 manual opt-out* 18:36:10 ack 18:36:12 ty gmann 18:37:07 well, now that we are more clear about what's expected in the 'unmaintained' branches, i don't think there will be a problem keeping v/w/x around 18:37:15 (as far as cinder is concerned) 18:37:22 rosmaita: cool 18:38:03 ok, i think i am more clear 18:38:18 rosmaita: awesome :) 18:38:55 next question is do we need to tag for example stable/victoria before renaming the branch to unmaintained/victoria? 18:38:57 and we're representatives of the community. If this process doesn't seem to work for the community either, we can go back to the drawing board. 18:39:51 =1 18:39:53 rosmaita: yes, that process we need to document in here https://docs.openstack.org/project-team-guide/stable-branches.html 18:40:12 or have a pointer there to a new document. 18:40:17 as that still says stable branches. 18:40:46 actually, forget that, we already have victoria-em tag 18:40:56 I think new document will be clear and explicit about what is 'unmaintinaed' and it process 18:40:58 i guess we can create unmaintained/victoria from that? 18:41:12 yes, makes sense 18:41:18 everyone have these as EM 18:41:26 *tagged as EM 18:41:30 rosmaita: if there are backport after -em then we need to take that in unmantinaed right ? 18:41:47 right 18:42:04 I think creating new tag and from there create unmaintianed will be good 18:42:05 wait. 18:42:18 why not to create unmaintained from top of current stable? 18:42:20 and close all the open backport and ask to do the same in new unmaintained brahc 18:42:25 rosmaita: After the meeting can you respond to the ML thread about Cinder EOL with regards to the outcome of this conversation? 18:42:34 noonedeadpunk: yes, that is what we need to do 18:42:51 knikolla: i will do that 18:42:59 we can do with latest hash or create new tag -eom like dansmith mentioned 18:43:10 I was just going to ask the same - why You want to ask people to do backports from stable/victoria to u/victoria now is they were merged into s/victoria after em tag? 18:43:12 that was not my idea :) 18:43:24 it's too late to reject that :D 18:43:25 ohk :) I just saw your comment 18:43:57 I am ok with tag or not tag but we should create unmaintianed from top of that we have currently on stable/vicrotia 18:44:01 I would like to move on to the next topic item 18:44:04 ++ 18:44:19 this is an implementation detail that doesn't require taking time from the meeting :) 18:44:28 #topic Release notes guidelines for SLURP/NON-SLURP cadence 18:44:32 what we do with TripleO? :D 18:44:40 #link https://review.opendev.org/c/openstack/project-team-guide/+/843457 18:45:09 noonedeadpunk: ugh... 18:45:11 noonedeadpunk: same. wallaby is only the one they want and that goes to unmaintained 18:45:22 yup 18:45:23 there they can maintain it 18:46:07 I'm pretty much fine with that. I expect RedHat not being that fine 18:46:35 As they exlicitly say it's "maintained" 18:46:47 knikolla: I have not re-reviewed that proposal since last year because I've been busy this week putting out gate fires, which I hope we're going to get to on the agenda 18:46:53 since it's much more pressing than any of this, IMHO 18:47:01 well, it was not "maintained" according to openstack's stable branch terminology for quite some time 18:47:09 we can move to the gate then. 18:47:16 #topic Gate Health Check 18:47:17 for releasenote 18:47:26 dansmith: floor is all yours 18:47:30 so the gate is very unhealthy at the moment. 18:47:40 gmann and I have been working on timeout issue mitigation all week, 18:47:50 but have been unable to merge most of those patches due to other failures 18:48:18 it seems like we've got a number of neutron-related issues cropping up.. I see a bunch of unreachable network issues and some failed tests about overlapping subnets and things 18:48:32 so we could really use some neutron attention to those things I think 18:48:54 I am going to check for more neutron tests if we can speed up those too especially scenario tests 18:48:57 there are also still some cinder-related issues that are causing plenty of troubles, but I'm worried that they're related to IO performance on the workers 18:49:03 dansmith please send me links to those issues, I will try to check them 18:49:09 but if anything has changed recently in cinder that could explain some of that, that'd be good to know 18:49:31 slaweq: ack, I meant to have some links ready for you today, but failed to collect them for $reasons 18:50:05 i dont' think there have been any major changes to cinder or os-brick that would account for a change like that 18:50:08 slaweq: here's one: https://73af546ea11737b343c2-b256b951e18c7d63775be8a7f99fd99d.ssl.cf5.rackcdn.com/889196/2/gate/tempest-full-py3/9c0d56d/testr_results.html 18:50:09 slaweq: this is one of them test_port_security_macspoofing_port , i will send link in neutron channel, yatin was looking into that initially 18:50:10 sure, I will try to take a look this week 18:50:42 slaweq: that link is a volume test failing because of a router issue, so unrelated to actual volumes 18:50:58 and another actual network test that failed, which could be a knock-on issue 18:51:08 ack, will check tomorrow morning 18:51:36 thanks 18:52:08 slaweq: pinged about port test in neutron channel 18:52:33 rosmaita: I think the one big right now for cinder stuff is that we create a server with a volume, ssh in and mkfs it, and it times out after 30 minutes without completion or some such 18:52:34 we are close to release and if we can improve the timeout issue it will be very helpful 18:52:54 gmann: yeah, the timeout stuff will really help overall I think, but we have to be able to actually merge those 18:53:09 the server actions split has made a big difference for in the nova set I've been rechecking, fwiw 18:53:16 haven't seen any timeouts in the last 36 hours or so 18:53:21 true 18:53:39 and I am hopeful this test run concurrency increase will make job timeout improve https://review.opendev.org/c/openstack/tempest/+/887220/9 18:53:47 * dansmith nods 18:53:59 tempest-slow job is now half of the time now by running them serially and concurrency increase 18:54:14 gmann ack, I added this also to my todo list for tomorrow 18:54:17 *running them paralley 18:54:22 slaweq: thanks 18:54:23 yeah, and that's where a lot of the fails are in that series I think 18:55:30 anything else on the topic? 18:55:35 anyway, in two weeks of rechecking a series of easy nova patches, I've gotten one to merge -- that's how bad it is 18:55:57 oh, sounds really bad... 18:56:07 knikolla: that's all my actual things 18:56:10 In a bit of a brighter note; Ironic is through the valley of our CI for now... I'm sad to hear that's not the case across everyone 18:56:11 We had jsut couple of timeouts, which are gone with next recheck 18:56:27 yeah and challenge is to merge the fixes where other issue stop them to merger 18:56:29 dansmith yeap, I saw it also in the rechecks stats https://etherpad.opendev.org/p/recheck-weekly-summary - we need a lot of rechecks to get patches merged 18:56:35 we have merged multiple fixes, including fixing more locking/retrying issues when testing under sqlite 18:56:38 slaweq: yeah :( 18:57:15 we're almost at time 18:57:28 thanks all, and thanks for all the hard work keeping the gate running 18:58:03 Thanks everyone! 18:58:22 #endmeeting