| *** pojadhav- is now known as pojadhav | 04:55 | |
| *** diablo_rojo is now known as Guest519 | 05:41 | |
| -opendevstatus- NOTICE: zuul isn't executing check jobs at the moment, investigation is ongoing, please be patient | 07:19 | |
| opendevreview | Chris MacNaughton proposed openstack/governance master: Add Ganesha based Ceph NFS Charm https://review.opendev.org/c/openstack/governance/+/835429 | 10:16 |
|---|---|---|
| opendevreview | Chris MacNaughton proposed openstack/governance master: Add Ganesha based Ceph NFS Charm https://review.opendev.org/c/openstack/governance/+/835429 | 10:17 |
| *** pojadhav is now known as pojadhav|afk | 11:59 | |
| *** pojadhav|afk is now known as pojadhav | 14:02 | |
| ade_lee | gmann, is there an update as to when tc will discuss fips stuff at ptg? | 15:00 |
| dansmith | ade_lee: sometime next week | 15:27 |
| * dansmith ducks | 15:27 | |
| gmann | ade_lee: have not schedule the exact time yet, please add your preference in etherpad and I will schedule accordingly. | 16:38 |
| elodilles | hi tc team! we are having a discussion over yoga release #openstack-release , as it seems a library release (oslo.context) broke a couple of projects (mistral, murano, trove, sahara, zaqar, magnum) that we haven't discover until now | 17:28 |
| elodilles | we discussed through the possible options | 17:29 |
| fungi | tc-members: ^ | 17:29 |
| mnaser | is it the removal of 'tenant'? | 17:30 |
| mnaser | https://review.opendev.org/c/openstack/magnum/+/834195 magnum seems to be resolved if that's the cfase | 17:30 |
| elodilles | mnaser: yes | 17:30 |
| mnaser | with a backport - https://review.opendev.org/c/openstack/magnum/+/834296 | 17:31 |
| mnaser | i guess mistral is missing a backport https://review.opendev.org/c/openstack/mistral/+/831960 | 17:32 |
| elodilles | so the question is whether it is OK to TC to force-release them before Wednesday, Yoga release? | 17:32 |
| fungi | some of the affected projects are going to need new release candidates. some may need tc approval to bypass lack of reviewer attention and merge the lingering fixes | 17:32 |
| * mnaser isn't TC but sees that as pretty reasonable =) | 17:33 | |
| fungi | problem is we're down to the wire, the proverbial "release button" is due to be pushed in about 42 hours from now | 17:34 |
| mnaser | ..yeah | 17:36 |
| ttx | yeah we need "enough" TC approval to force those patches in today-ish. | 17:37 |
| mnaser | i guess its approval for both changes + release patches, right? | 17:38 |
| ttx | pinging gmann jungleboyj dansmith diablo_rojo__ arne_wiebalck slaweq knikolla rosmaita and Amy | 17:38 |
| mnaser | maybe it might be good to generate a list of the changes to be approved and if we get enough tc's to asynchronusly +1 them, then we can just move forward having someone from infra escalate privileges and +W | 17:38 |
| ttx | I'd go simple and just have a TC blanket approval that it's OK for fungi to force patches in to solve that specific issue | 17:39 |
| dansmith | force release which, the projects? | 17:39 |
| dansmith | or force patches in to remove the tenant usage before the projects get actually released? | 17:40 |
| mnaser | i think its both | 17:40 |
| ttx | Force tenant -> project_id changes in some projects master+stable/yoga that were broken from oslo.context deprecation two months ago and never realized it | 17:40 |
| dansmith | because we just released oslo.context and now they're broken? | 17:41 |
| fungi | it looks like it's mostly just affecting tests which were using "tenant" | 17:41 |
| mnaser | dansmith: seems like it.. | 17:41 |
| ttx | well, oslo.context was released two months ago | 17:41 |
| fungi | dansmith: "just released" months ago | 17:41 |
| ttx | they just did not land anything since | 17:41 |
| dansmith | okay I'm not sure I get it then | 17:41 |
| fungi | those projects simply had no activity to notice it was broken | 17:41 |
| dansmith | ahhh | 17:41 |
| ttx | until zigo... tested them | 17:42 |
| ttx | so not we release them broken, or we forcefully fix them | 17:42 |
| elodilles | dansmith: the list of the projects is this so far: mistral, murano, trove, sahara, zaqar, magnum | 17:42 |
| ttx | now* | 17:42 |
| fungi | for the most part these are very "low activity" projects (apply whatever euphemism you prefer) which are impacted | 17:42 |
| dansmith | I guess I'm not sure why we need to force patches in.. isn't it breaking now and passing with the new patches to remove the legacy? | 17:42 |
| elodilles | as fungi says ^^^ | 17:42 |
| fungi | dansmith: not "force" merge them, but "force" approve them | 17:43 |
| ttx | not forcing... just allowing fungi to +2a them | 17:43 |
| fungi | some of these projects effectively have no reviewers | 17:43 |
| dansmith | because there aren't cores around? | 17:43 |
| ttx | yes | 17:43 |
| dansmith | ah okay I thought we were talking about force merging things (i.e. things that won't pass tests | 17:43 |
| dansmith | for sure + 1 from me on that | 17:43 |
| mnaser | yeah its force approve rather than force merge | 17:44 |
| dansmith | ++ | 17:44 |
| ttx | sorry should have been clearer | 17:44 |
| gmann | but how project with broken code released/ | 17:44 |
| fungi | gmann: because we don't test the projects when we tag them | 17:44 |
| mnaser | gmann: i think it's because release jobs dont test the actual project i guess | 17:44 |
| elodilles | ("force" here means we are too close to release deadline and beyond freeze time) | 17:44 |
| ttx | the whole process assumes a minimal amount of activity really | 17:44 |
| gmann | but on project side those might have been tested | 17:44 |
| gmann | i saw many projects fixed those | 17:45 |
| fungi | gmann: they were merging changes at some point, then oslo.context released with a breaking change, those projects had no new activity after that, and were eventually tagged in that state | 17:45 |
| ttx | We'll fix it to re-test things ahead of time, but we need to fix the mess now/today | 17:45 |
| mnaser | gmann: some projects do seem t ohave landed fixes, but not all i guess | 17:45 |
| gmann | k | 17:45 |
| mnaser | i tracked that magnum was fixed but yeah i guess it's trying to emergency fix the ones that werent fixed and didnt know they were broken for 2+ months | 17:45 |
| ttx | most projects are ok, only a handful of half-abandoned projects are the issue | 17:45 |
| mnaser | and fixing the release process today to test before release is probably not the right time to do now :p | 17:45 |
| ttx | right | 17:46 |
| gmann | yeah, this example shows those projects are lexx active | 17:46 |
| gmann | less | 17:46 |
| fungi | right, which is why the release team is needing to ask the tc to step in or grant someone emergency permission | 17:46 |
| gmann | yeah, it should be on projects side test. that is why I am always on favor of let project push the release patch and they check the state of project at least | 17:46 |
| mnaser | i suspect we might have never seen a release patch pushed if that was to happen heh | 17:47 |
| mnaser | (and i think that's how it was before) | 17:47 |
| fungi | i'll wager that most of these did not propose their own releases for the same reason they didn't know they were broken | 17:47 |
| gmann | yeah | 17:47 |
| jungleboyj | I am fine with blanket support for getting patches merged to fix things. | 17:48 |
| jungleboyj | Why would we support a broken release? :-) | 17:48 |
| ttx | jungleboyj: :) | 17:48 |
| gmann | ttx: elodilles so proposal here to merge those projects fixes forcefully and release them? | 17:48 |
| ttx | The proposal is to allow fungi to +2a a bunch of patches in those projects | 17:49 |
| mnaser | i think being technical here, not force merge but force approve (i.e. tests will pass and gate :p) | 17:49 |
| ttx | so that we can re-release them in time for Yoga inclusion | 17:49 |
| fungi | gmann: yeah, not merge them forcefully (they're passing tests), just have someone (i.e. me) use admin access to approve them instead of the project's core reviewers | 17:49 |
| fungi | and also not require the release team to get approval from the ptls to tag new release candidates | 17:49 |
| gmann | humm, honestly saying, I am not sure if this will help much. and with this, projects not going to become more active or so. | 17:50 |
| ttx | (we don;t really need that, we routinely do releases that are not PTL-approved) | 17:50 |
| gmann | may be revert the oslo.context change and release it? | 17:50 |
| ttx | gmann: that's way more impactful umho | 17:50 |
| ttx | rolling back 6 months of work of getting it into oslo | 17:50 |
| mnaser | i know i don't have a say in this, but i rather we fix the broken projects rather than accomodate the semi-dead projects | 17:50 |
| fungi | new oslo.context will also need new requirements bump | 17:51 |
| ttx | mnaser: exactly | 17:51 |
| elodilles | mnaser: ++ | 17:51 |
| fungi | and projects have been testing with the current oslo.requirements version, so we risk introducing a regression we won't catch | 17:51 |
| mnaser | and this wasn't a last minute change by oslo.context | 17:51 |
| mnaser | it's a 2 month old change afaik | 17:51 |
| ttx | Release team discussed various scenarii and this is the least release-impactful | 17:51 |
| fungi | er, oslo.context i mean | 17:51 |
| elodilles | mnaser: exactly, oslo.context was released @ Feb 3rd | 17:52 |
| ttx | We just need approval to proceed and bypass normal review | 17:52 |
| gmann | I see, agree on oslo.context things | 17:52 |
| jungleboyj | I agree that reverting is higher risk. | 17:52 |
| gmann | tenant/project is becoming very costly for us :) | 17:52 |
| fungi | also the problem was called out on openstack-discuss weeks ago by a distro package maintainer, and those projects never apparently saw the discussion | 17:52 |
| ttx | eh yes | 17:52 |
| dansmith | gmann: :) | 17:53 |
| mnaser | the day openstack client drops OS_TENANT_NAME, now that'll be a fun one ;) | 17:53 |
| gmann | even in nova, there is BP going cince more than years | 17:53 |
| gmann | since | 17:53 |
| dansmith | this is how I feel about "cleanup" activities that don't actually bring gain other than perceived developer purity | 17:53 |
| dansmith | like renames in APIs... | 17:53 |
| gmann | mnaser: I will say it should not. :) | 17:53 |
| gmann | dansmith: indeed | 17:53 |
| fungi | anyway, it seems like we've probably got some consensus from a subset of the tc and the release team, with no lingering dissent. if other tc-members want to weigh in then please do, but in the absence of objections (we don't have time for a tc meeting and passing a resolution at this point), i'll plan to start approving the remaining fixes in the next couple of hours | 17:55 |
| jungleboyj | fungi: ++ Thank you! | 17:56 |
| gmann | main question is also on PTLs, we recently just 2 week before appointed PTL for all those projects with their asking and they even do not know their project code is broken or not? | 17:56 |
| fungi | that seems to be the case, yes | 17:56 |
| dansmith | ttx didn't get spotz mentioned in the ping, so pinging spotz in case spotz is around | 17:56 |
| fungi | i can't say i cross-checked the entire list against ptl appointments, but it sounds about right | 17:57 |
| gmann | fungi: we need to record the decision before we start approving them | 17:57 |
| dansmith | also maybe spotz[m] | 17:57 |
| spotz | reading | 17:57 |
| fungi | gmann: yes, do you want elodilles or me to send something to openstack-discuss? | 17:57 |
| gmann | yes please, and I can start a quick meeting here as many TC members are here and record the vote also. | 17:58 |
| gmann | in case any project start question on governance and force merge patches in their project | 17:59 |
| jungleboyj | Okie dokie. | 17:59 |
| spotz | Something to the list would be good, I think if we need to approve them to make the release we shold | 17:59 |
| gmann | elodilles: ttx what deadline you need them to release? I think max by tomorrow right? | 17:59 |
| fungi | sooner, because we need to tag release candidates too | 18:00 |
| fungi | today, ideally | 18:00 |
| gmann | yeah, its already 28 today | 18:00 |
| mnaser | i think sending an ML notification and if someone disagrees with it so much they can propose a revert | 18:00 |
| spotz | I'll be around all day just ping here or on any reviews you need me. Shouldn't be afk for more then like 45 minutes | 18:00 |
| gmann | ok, please send it on ML and I will start a quick meeting for vote record. | 18:01 |
| mnaser | i'd say its better to help those teams release non-broken projects than worry about them being upset that 'we merged something that helped their software not be broken' but i'm just talking from the sidelines now :) | 18:01 |
| gmann | mnaser: more then revert, it can raise other concerns also as we are discussing and merging the change within today | 18:01 |
| mnaser | we're only having this conversation because these projects didn't look after and tests their stuff for 2 months | 18:02 |
| gmann | ideally that is ok but we never know what project can react how | 18:02 |
| fungi | i'd have been more in favor of just leaving those services out of the coordinated release until they're fixed, but at this point we've basically committed to users that they'll be included | 18:03 |
| gmann | IMO, we should never release such project where code is broken and no maintainer. but we are at the last stage of yoga release so might not be good timing for it | 18:03 |
| gmann | fungi: +100 | 18:03 |
| jungleboyj | ++ | 18:03 |
| spotz | +100 | 18:03 |
| gmann | but something we can add in process that such cases we will just leave the release instead of emergency approval. but we can do that for future. | 18:04 |
| fungi | after talking with elodilles, i'll send a proposal to openstack-discuss on behalf of the release team and we can get some official result recorded there in a reply from one of you | 18:04 |
| fungi | working on that now, i'll let you know once it's posted | 18:05 |
| gmann | fungi: thanks, I will send op top of that. | 18:05 |
| gmann | tc-members: starting a quick meeting to record this exception. | 18:07 |
| gmann | #startmeeting tc | 18:07 |
| opendevmeet | Meeting started Mon Mar 28 18:07:09 2022 UTC and is due to finish in 60 minutes. The chair is gmann. Information about MeetBot at http://wiki.debian.org/MeetBot. | 18:07 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 18:07 |
| opendevmeet | The meeting name has been set to 'tc' | 18:07 |
| gmann | tc-members: this quick meeting is to get agreement on few broken projects Yoga release and best solution | 18:08 |
| jungleboyj | o/ | 18:08 |
| dansmith | o/ | 18:08 |
| gmann | as we discussed on IRC, there are few active projects that has not merged the fixes of oslo.context tenant_id removal change and we are at | 18:09 |
| gmann | few less active* | 18:09 |
| gmann | we are at yoga release time and need to get agreement on release team proposal. | 18:09 |
| jungleboyj | Ok. | 18:10 |
| gmann | few options to handle this is 1. do not release those projects 2. TC permit fungi to approve those fixes on projects side and release them 3. ? | 18:10 |
| fungi | elodilles started this pad to try to accumulate the list of what's needed (whether it's approving stable branch changes or proposing new release candidates): https://etherpad.opendev.org/p/tenant-projectid-last-minute-fixes | 18:10 |
| gmann | 1st solution is tool late as per yoga relaese time | 18:11 |
| gmann | 2 option seems more feasible at this stage. | 18:11 |
| jungleboyj | Agreed. | 18:12 |
| gmann | if you agree on 2nd I can start the vote on this to record this exception. or any other 3rd option ? | 18:12 |
| jungleboyj | Based on the discussions we have had it sounds like option 2 is the only viable one. | 18:12 |
| dansmith | vote on #2 | 18:12 |
| gmann | sure, let me start | 18:12 |
| gmann | #startvote Should TC permit fungi to merge the fixes in required projects? Yes, No | 18:13 |
| opendevmeet | Begin voting on: Should TC permit fungi to merge the fixes in required projects? Valid vote options are Yes, No. | 18:13 |
| opendevmeet | Vote using '#vote OPTION'. Only your last vote counts. | 18:13 |
| gmann | project list mentioned in #link https://etherpad.opendev.org/p/tenant-projectid-last-minute-fixes | 18:14 |
| gmann | Yes | 18:14 |
| jungleboyj | #vote Yes | 18:14 |
| dansmith | #vote Yes | 18:14 |
| gmann | #vote Yes | 18:14 |
| spotz | #vote yes | 18:14 |
| jungleboyj | All opposed ? | 18:14 |
| gmann | diablo_rojo__ arne_wiebalck slaweq kristi rosmaita ? | 18:16 |
| gmann | we have 4 TCs voted out of 9. | 18:16 |
| gmann | knikolla | 18:17 |
| gmann | ok, I think we have the agreement here with 4 votes. | 18:17 |
| gmann | #endvote | 18:17 |
| opendevmeet | Voted on "Should TC permit fungi to merge the fixes in required projects?" Results are | 18:17 |
| opendevmeet | Yes (4): dansmith, gmann, jungleboyj, spotz | 18:17 |
| gmann | thanks all for joining. that is all for this meeting. | 18:18 |
| gmann | I will reply on fungi email on ML with the TC agreement on allowing him to merge the fixes. | 18:19 |
| gmann | #endmeeting | 18:19 |
| opendevmeet | Meeting ended Mon Mar 28 18:19:02 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 18:19 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/tc/2022/tc.2022-03-28-18.07.html | 18:19 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/tc/2022/tc.2022-03-28-18.07.txt | 18:19 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/tc/2022/tc.2022-03-28-18.07.log.html | 18:19 |
| fungi | yep, just making sure i have all my references in order | 18:19 |
| fungi | will hit send in a few | 18:19 |
| jungleboyj | Thank you! | 18:22 |
| fungi | gmann: okay, it's sent... https://lists.openstack.org/pipermail/openstack-discuss/2022-March/027864.html | 18:29 |
| fungi | had to wrestle my mail client into submission for some reason, so took a few minutes longer than hoped | 18:30 |
| gmann | I have added this topic again in Zed PTG, this is something we discussed in Yoga PTG and thought of introducing the tech-preview framework to catch such project early before final release and avoid these situation | 18:30 |
| gmann | fungi: replied with the TC agreement link. | 18:43 |
| fungi | thanks! i'll get started approving things shortly | 18:43 |
| gmann | +1, thanks | 18:44 |
| jungleboyj | +1 ... Thanks for all you do fungi! | 18:44 |
| fungi | it's my pleasure | 18:44 |
| fungi | the release team members are the real heroes of the release, i'm just pitching in to make sure things go as smoothly as they can | 18:45 |
| jungleboyj | :-) | 18:45 |
| spotz | Thanks release team and fungi! | 19:23 |
| dansmith | well, this is going well :) | 19:55 |
| fungi | yeah, i've proposed a naive fix to murano myself, no idea if it will pass testing but it's a start | 20:04 |
| fungi | backports for trove are in progress still | 20:04 |
| opendevreview | Ghanshyam proposed openstack/governance master: Add goal readiness checklist https://review.opendev.org/c/openstack/governance/+/835102 | 21:05 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!