| *** tetsuro has joined #openstack-release | 00:10 | |
| *** tetsuro has quit IRC | 00:12 | |
| *** tetsuro has joined #openstack-release | 00:13 | |
| *** tetsuro has quit IRC | 00:19 | |
| *** tetsuro has joined #openstack-release | 00:20 | |
| *** tetsuro_ has joined #openstack-release | 00:30 | |
| *** tetsuro has quit IRC | 00:33 | |
| *** mlavalle has quit IRC | 00:40 | |
| *** tosky has quit IRC | 00:50 | |
| *** ianychoi_ has joined #openstack-release | 01:38 | |
| *** ianychoi has quit IRC | 01:40 | |
| *** mgoddard has quit IRC | 01:49 | |
| *** mgoddard has joined #openstack-release | 01:57 | |
| *** tetsuro_ has quit IRC | 03:51 | |
| *** tetsuro has joined #openstack-release | 04:16 | |
| *** udesale has joined #openstack-release | 04:32 | |
| *** tetsuro_ has joined #openstack-release | 04:49 | |
| *** tetsuro has quit IRC | 04:52 | |
| *** tetsuro has joined #openstack-release | 05:03 | |
| openstackgerrit | Abhishek Kekane proposed openstack/releases master: Release Glance 17.0.1 https://review.opendev.org/713779 | 05:04 |
|---|---|---|
| *** tetsuro_ has quit IRC | 05:06 | |
| *** dave-mccowan has quit IRC | 05:10 | |
| *** tetsuro_ has joined #openstack-release | 05:15 | |
| *** tetsuro has quit IRC | 05:18 | |
| openstackgerrit | Abhishek Kekane proposed openstack/releases master: Release glance_store 0.26.2 https://review.opendev.org/713782 | 05:18 |
| openstackgerrit | Abhishek Kekane proposed openstack/releases master: Release python-glanceclient 2.13.2 https://review.opendev.org/713783 | 05:34 |
| *** evrardjp has quit IRC | 05:36 | |
| *** evrardjp has joined #openstack-release | 05:36 | |
| *** zxiiro has quit IRC | 06:24 | |
| *** tetsuro_ has quit IRC | 07:49 | |
| *** slaweq has joined #openstack-release | 08:06 | |
| *** portdirect has quit IRC | 08:08 | |
| *** guilhermesp has quit IRC | 08:09 | |
| *** mwhahaha has quit IRC | 08:09 | |
| *** portdirect has joined #openstack-release | 08:10 | |
| *** mnaser has quit IRC | 08:10 | |
| *** mwhahaha has joined #openstack-release | 08:10 | |
| *** mnaser has joined #openstack-release | 08:11 | |
| *** rpittau|afk is now known as rpittau | 08:11 | |
| *** guilhermesp has joined #openstack-release | 08:11 | |
| *** e0ne has joined #openstack-release | 08:16 | |
| *** tosky has joined #openstack-release | 08:21 | |
| *** jbadiapa has joined #openstack-release | 08:33 | |
| *** tetsuro has joined #openstack-release | 08:34 | |
| *** tetsuro_ has joined #openstack-release | 08:40 | |
| *** tetsuro has quit IRC | 08:43 | |
| ttx | smcginnis: we got two recent false negative on check-release-approval | 09:35 |
| ttx | http://zuul.openstack.org/build/911208ee679d4466a0477a596d72f267 | 09:35 |
| ttx | http://zuul.openstack.org/build/7d07b54ae3f14668b1fe16ca7c255c94 | 09:35 |
| *** e0ne has quit IRC | 09:36 | |
| ttx | Those happened on two different executors, so i don't think the issue is pinned to one | 09:36 |
| *** e0ne has joined #openstack-release | 09:36 | |
| *** dtantsur|afk is now known as dtantsur | 09:39 | |
| *** tetsuro has joined #openstack-release | 09:40 | |
| *** tetsuro_ has quit IRC | 09:43 | |
| *** tetsuro has quit IRC | 09:56 | |
| openstackgerrit | Merged openstack/releases master: Release keystoneauth1 3.13.2 for stable/stein https://review.opendev.org/713636 | 10:05 |
| openstackgerrit | Merged openstack/releases master: Oslo releases for 2020-03-18 https://review.opendev.org/713749 | 10:06 |
| *** hberaud has quit IRC | 11:19 | |
| *** dtantsur is now known as dtantsur|afk | 11:20 | |
| *** rpittau is now known as rpittau|bbl | 11:30 | |
| *** dave-mccowan has joined #openstack-release | 11:53 | |
| smcginnis | ttx: Well, at least it's more data. And one hypothesis proven wrong. | 11:57 |
| openstackgerrit | Merged openstack/releases master: Release Glance 17.0.1 https://review.opendev.org/713779 | 12:18 |
| openstackgerrit | Merged openstack/releases master: Release glance_store 0.26.2 https://review.opendev.org/713782 | 12:18 |
| openstackgerrit | Merged openstack/releases master: Release python-glanceclient 2.13.2 https://review.opendev.org/713783 | 12:18 |
| *** udesale_ has joined #openstack-release | 12:19 | |
| *** udesale has quit IRC | 12:20 | |
| *** e0ne_ has joined #openstack-release | 12:20 | |
| *** e0ne has quit IRC | 12:21 | |
| ttx | my current hypothesis is that the extended query (o=DETAILED_LABELS) somehow times out and gerrit hides it by returning the data it successfully retrieved | 12:31 |
| ttx | hence my hope that using /detail instead would use a different code path | 12:32 |
| *** hberaud has joined #openstack-release | 12:41 | |
| *** rpittau|bbl is now known as rpittau | 12:59 | |
| smcginnis | We should find out soon enough now. | 13:30 |
| *** udesale_ has quit IRC | 14:00 | |
| openstackgerrit | Rabi Mishra proposed openstack/releases master: Release tripleo-common 12.2.0 https://review.opendev.org/713891 | 14:27 |
| *** amoralej has joined #openstack-release | 14:47 | |
| amoralej | hi can we consider rocky in EM already? my understanding was so but https://releases.openstack.org/ says Maintained | 14:48 |
| amoralej | smcginnis, ^ | 14:49 |
| smcginnis | amoralej: Projects are in the process of transitioning to EM. Only a few left. We can probably update that page now to reflect that Rocky is no longer Maintained. | 14:51 |
| amoralej | ack, thx | 14:52 |
| *** armstrong has joined #openstack-release | 14:57 | |
| *** ricolin has quit IRC | 15:08 | |
| *** ricolin has joined #openstack-release | 15:13 | |
| openstackgerrit | Sean McGinnis proposed openstack/releases master: [glance] Transition Rocky to EM https://review.opendev.org/709888 | 15:22 |
| evrardjp | sorry I have been under fire the whole week :/ | 15:58 |
| evrardjp | I will still attend the meeting if there is one today. | 15:58 |
| smcginnis | Yep! | 15:59 |
| smcginnis | #startmeeting releaseteam | 16:00 |
| openstack | Meeting started Thu Mar 19 16:00:01 2020 UTC and is due to finish in 60 minutes. The chair is smcginnis. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| *** openstack changes topic to " (Meeting topic: releaseteam)" | 16:00 | |
| openstack | The meeting name has been set to 'releaseteam' | 16:00 |
| ttx | o/ | 16:00 |
| smcginnis | Ping list: ttx armstrong diablo_rojo, diablo_rojo_phon | 16:00 |
| hberaud | o/ | 16:00 |
| evrardjp | o/ | 16:00 |
| smcginnis | #link https://etherpad.openstack.org/p/ussuri-relmgt-tracking Agenda | 16:00 |
| elod | o/ | 16:00 |
| smcginnis | Good turnout. | 16:00 |
| smcginnis | #topic Outstanding rocky-em patches | 16:00 |
| *** openstack changes topic to "Outstanding rocky-em patches (Meeting topic: releaseteam)" | 16:00 | |
| smcginnis | #link https://review.opendev.org/#/q/status:open+project:openstack/releases+branch:master+topic:rocky-em EM Patches | 16:01 |
| smcginnis | Glance should be good to go now - https://review.opendev.org/709888 | 16:01 |
| smcginnis | Great if someone else can take a look and approve that. | 16:01 |
| diablo_rojo | o/ | 16:01 |
| smcginnis | Cloudkitty - https://review.opendev.org/#/c/709920/ | 16:01 |
| evrardjp | I will have a look after the meeting. | 16:01 |
| smcginnis | No response from PTL and it looks like there are unreleased commits. | 16:01 |
| smcginnis | So... would be nice to get a release out for those, but also an indication that this may actually be unmaintained. | 16:02 |
| smcginnis | Thoughts on that? | 16:02 |
| evrardjp | Maybe the new PTL didn't get the memo. Will ping in their channel | 16:02 |
| *** chandankumar is now known as raukadah | 16:02 | |
| evrardjp | they changed PTL during the cycle | 16:02 |
| smcginnis | Ah, OK. We can wait on that one a bit then. | 16:03 |
| evrardjp | I expect some quirks on comms | 16:03 |
| smcginnis | Telemetry - https://review.opendev.org/#/c/709880/ | 16:03 |
| smcginnis | PTL approved, but unreleased commits. | 16:03 |
| evrardjp | :/ | 16:03 |
| smcginnis | No response to question regarding doing another release first. | 16:03 |
| evrardjp | let's go ahead then ? | 16:03 |
| evrardjp | I mean if it was approved by PTL.... | 16:04 |
| smcginnis | Yeah, kind of what I was thinking. | 16:04 |
| smcginnis | Other thoughts? | 16:04 |
| evrardjp | let's see if there are complaints later ;) | 16:04 |
| evrardjp | haha | 16:04 |
| elod | I think the same, if there are no answers then I guess those can be EM'd :/ | 16:04 |
| ttx | nope go for it | 16:04 |
| smcginnis | At least looking at ceilometer, a few fixes, but nothing critical. | 16:05 |
| diablo_rojo | I would hope that the PTL actually looked at the patches.. | 16:06 |
| smcginnis | And of course, distros and end users can always apply unreleased commits if actually needed. | 16:06 |
| elod | exactly | 16:06 |
| smcginnis | If someone could approve that telemetry one then... | 16:06 |
| smcginnis | OSA - https://review.opendev.org/#/c/709875/ | 16:06 |
| smcginnis | These cycle trailing ones wanted to wait until other projects had finished going to EM. | 16:07 |
| smcginnis | So I think this one can wait a little more. | 16:07 |
| evrardjp | we should define a limit | 16:07 |
| evrardjp | how long to wait ? | 16:07 |
| ttx | telemetry one approved | 16:07 |
| smcginnis | Next week maybe? I think glance and telemetry are the only big ones that OSA would need to wait for? | 16:08 |
| smcginnis | Heat - https://review.opendev.org/#/c/709889/ | 16:08 |
| smcginnis | ricolin: There are some questions on the heat rocky em patch. | 16:08 |
| smcginnis | That one at least needs the sha's updated. | 16:09 |
| smcginnis | Validation has since been added to make sure that doesn't pass anymore. | 16:09 |
| smcginnis | I guess go ahead and update that and transition if no response by next week? Any other ideas? | 16:10 |
| smcginnis | We can go back to that one later. | 16:11 |
| smcginnis | TripleO - https://review.opendev.org/#/c/709912/ | 16:11 |
| diablo_rojo | I think that is plenty of time that we've waited.. | 16:11 |
| smcginnis | That one actually looks it's OK, just never got a PTL response. | 16:12 |
| smcginnis | So I think we can approve that. | 16:12 |
| ttx | ok on it | 16:12 |
| smcginnis | Last one, Kolla - https://review.opendev.org/#/c/709893/ | 16:12 |
| smcginnis | Commented on their that they wanted to wait for others to transition. | 16:13 |
| smcginnis | So that one we can wait until glance and telemetry (and maybe Heat) make it through. | 16:13 |
| smcginnis | I can comment on their now to ask Mark to reassess their readiness. | 16:13 |
| evrardjp | sounds good | 16:14 |
| smcginnis | Well, some progress on those at least. | 16:14 |
| smcginnis | I will also put up a patch to reflect the stable phase on releases.o.o after the meeting. | 16:14 |
| smcginnis | #topic Outstanding ussuri-cwi patches | 16:14 |
| *** openstack changes topic to "Outstanding ussuri-cwi patches (Meeting topic: releaseteam)" | 16:14 | |
| smcginnis | #link https://review.opendev.org/#/q/status:open+project:openstack/releases+branch:master+topic:ussuri-cwi | 16:15 |
| smcginnis | Two outstanding ones. | 16:15 |
| smcginnis | Freezer - https://review.opendev.org/#/c/709844/ | 16:15 |
| smcginnis | They have not done a release. A little confusing comment on there, but I think we are OK to go ahead and switch them to -rc. | 16:15 |
| evrardjp | Isn't freezer already in the freezer? | 16:15 |
| smcginnis | Hah | 16:16 |
| smcginnis | Not sure on that projects state, but it has been iffy for awhile now I think. | 16:16 |
| evrardjp | I thought only SUSE was working on it recently, but I might be wrong, I have to double check. It might be worth checking if we shouldn't retire it instead. | 16:16 |
| smcginnis | So probably safe enough to do. Unless it needs to be completely dropped for Ussuri. | 16:16 |
| smcginnis | Anyone know anything about its state? | 16:16 |
| ttx | on a call, brb | 16:16 |
| hberaud | nope... | 16:16 |
| evrardjp | the safe bet would be to move it to cycle-with-rc and deprecate it next cycle | 16:17 |
| smcginnis | Is the TC still doing health checks? | 16:17 |
| evrardjp | but that increase the lifespan | 16:17 |
| evrardjp | not actively as we used to have | 16:17 |
| diablo_rojo | smcginnis, not really | 16:17 |
| evrardjp | we are trying a different approach to see the healthiness of the project, using "golden signals" | 16:17 |
| evrardjp | basically this is a signal :D | 16:18 |
| diablo_rojo | PTL is from ZTE | 16:18 |
| evrardjp | so I am thinking -- how should we interact on this | 16:18 |
| diablo_rojo | I think we move it to cycle with rc | 16:18 |
| diablo_rojo | and then we will see if they elect a PTL.. | 16:18 |
| smcginnis | I think it's the release teams place to decide if it's part of the coordinated release or not. But TCs role to see if it should even exist anymore. | 16:19 |
| diablo_rojo | I would agree | 16:19 |
| smcginnis | I guess since we got at least some activity from the PTL (even if it was confusing) we can do the switch to rc, then see how things go. | 16:19 |
| evrardjp | agreed on both diablo_rojo and smcginnis comment | 16:19 |
| smcginnis | So if someone wants to merge that patch, I think that's the right path at this point. | 16:19 |
| evrardjp | I just want to make sure the TC is aware of this :) | 16:19 |
| smcginnis | I suppose PTL election will be another "signal" on that. | 16:20 |
| evrardjp | correct. | 16:20 |
| evrardjp | at least that's my understanding. ttx can tell us more about the signals. | 16:20 |
| smcginnis | The other one is similar - https://review.opendev.org/#/c/709845/ | 16:20 |
| smcginnis | Karbor. Some comment, but kept thiss time, on that one. | 16:21 |
| smcginnis | But no follow up release. | 16:21 |
| smcginnis | So I think we go ahead with the switch. Otherwise drop since technically they have not met the inclusion criteria. | 16:21 |
| evrardjp | the problem I have is that the more we wait, the more it will take to deprecate :) but I think it's too late to decide for this cycle, and let's not take harsh decisions lightly | 16:21 |
| evrardjp | (that's the tc hat speaking, sorry for that) | 16:21 |
| diablo_rojo | +2 Sounds good to me. We can only give so much time. | 16:22 |
| evrardjp | smcginnis: agreed | 16:22 |
| smcginnis | "Projects must participate in at least two milestones in order to be considered part of the release." | 16:22 |
| smcginnis | https://releases.openstack.org/ussuri/schedule.html#u-mf | 16:22 |
| smcginnis | We do clearly state what is expected, and they have not met that expectation. | 16:22 |
| evrardjp | agreed | 16:22 |
| smcginnis | Would like ttx's input on this too. I don't think we've ever had to fully drop a whole project before. | 16:22 |
| evrardjp | it's never too late to start those kind of things. | 16:23 |
| diablo_rojo | Really? Never? | 16:23 |
| diablo_rojo | I'm surprised. | 16:23 |
| evrardjp | smcginnis: just a question: are you afraid of the impact on the tooling for the release? | 16:23 |
| smcginnis | At least since I've been involved. The threat of the stick has usually worked to get them to do a release. | 16:24 |
| smcginnis | evrardjp: No, I think we're fine. | 16:24 |
| smcginnis | I will propose dropping the files. | 16:24 |
| smcginnis | We can get comments on the review for that. | 16:24 |
| evrardjp | yeah or release-management none | 16:24 |
| evrardjp | or whatever | 16:24 |
| evrardjp | :) | 16:24 |
| smcginnis | But I wonder if maybe we should do the same for freezer. | 16:24 |
| smcginnis | Something cycle based like this, I don't think we would want to say release-management none. | 16:24 |
| evrardjp | we should probably drop an email on the ML | 16:25 |
| evrardjp | smcginnis: I agree it was more: if there is a real urgency, maybe we can do something | 16:25 |
| smcginnis | What do folks think about including freezer along with karbor? | 16:25 |
| evrardjp | not that governance is the fastest for merging things :p haha | 16:25 |
| smcginnis | ;) | 16:25 |
| evrardjp | how long can this wait? | 16:26 |
| diablo_rojo | smcginnis, freezer had one release but karbor has had none right? | 16:26 |
| smcginnis | We're already way past the deadline. Not sure we want to wait much longer in the cycle to communicate something like this. | 16:26 |
| ttx | back | 16:26 |
| evrardjp | ok I guess we should go ahead then | 16:26 |
| ttx | catching up | 16:26 |
| smcginnis | evrardjp: No, neither. | 16:26 |
| smcginnis | evrardjp: https://review.opendev.org/#/c/709844/1/deliverables/ussuri/freezer.yaml | 16:27 |
| smcginnis | ttx: tldr; should we drop freezer and karbor from being included in the ussuri release since they have not done releases by the deadline. | 16:27 |
| ttx | technically we are past membership freeze, so what's in we are committed to release | 16:28 |
| evrardjp | I have the impression freezer is a repeating offender, but maybe it's my wrong impression | 16:28 |
| evrardjp | oh that's true too | 16:28 |
| evrardjp | so that's the solution | 16:28 |
| smcginnis | But with no releases at all, just a placeholder file, are those actually "in"? | 16:28 |
| ttx | smcginnis: the idea behind membership freeze was to communicate early what will be in release | 16:29 |
| ttx | for packagers and all | 16:29 |
| ttx | not sure who actually pays attention, but I like the concept | 16:29 |
| smcginnis | True | 16:29 |
| evrardjp | smcginnis: technically -- why not? assuming the project is very stable it might not require new patches and new releases, but be part of the coordinated release | 16:29 |
| openstackgerrit | Merged openstack/releases master: [Telemetry] Transition Rocky to EM https://review.opendev.org/709880 | 16:29 |
| smcginnis | OK, then just force both of these to -rc, even though at least one the PTL has said no. | 16:30 |
| openstackgerrit | Elod Illes proposed openstack/releases master: Set Rocky status to Extended Maintenance https://review.opendev.org/713926 | 16:30 |
| evrardjp | ttx: I have doubts about the usefulness, but that's a conversation for another time :p | 16:30 |
| ttx | smcginnis: yes... that or proposing an intermediary release immediately | 16:30 |
| smcginnis | I'm concerned about the lack of visibility into zombie projects that we just end up taking care of. | 16:30 |
| ttx | I would switch karbor, and propose a release for freezer on HEAD | 16:30 |
| smcginnis | We did add the "forced" flag awhile back. | 16:30 |
| smcginnis | Maybe we should think about doing that as a way to track the ones that do not get any response from the teams. | 16:31 |
| ttx | Feels like a ML thread about it could trigger some attention | 16:31 |
| ttx | one for each | 16:31 |
| smcginnis | ttx: I'm concerned that we asked them to get an intermediary release out to stay cwi, then no response at all. | 16:31 |
| ttx | that way we (1) track it and (2) use the shame | 16:31 |
| evrardjp | would you mind tackling this? :) | 16:31 |
| evrardjp | I am afraid of my wording. | 16:32 |
| ttx | smcginnis: then we force-approve it | 16:32 |
| evrardjp | I agree with ttx plan. currently I have voted on those, and if they propose a patch, I can vote again. | 16:32 |
| smcginnis | OK, switch karbor and force freezer. That works for me. | 16:33 |
| ttx | then I'll create a thread for each | 16:33 |
| smcginnis | ttx: Thanks for taking that. | 16:33 |
| smcginnis | I can get the release proposed. | 16:33 |
| ttx | mind you, the thread might conclude membership freeze should not prevent us from removing stuff at the last minute | 16:33 |
| smcginnis | I like that too. | 16:33 |
| ttx | hmm | 16:34 |
| smcginnis | Oh good, the additional validations caught something with the tripleo patch. I'll get that fixed up too. | 16:34 |
| ttx | I'm confused | 16:34 |
| evrardjp | ttx: ? | 16:35 |
| ttx | so freezer is the one where jiaopengju said "please ignore this comment." | 16:35 |
| ttx | basically not -1 it | 16:35 |
| ttx | so that would be the one we'd swtch to -rc | 16:35 |
| * hberaud need to eject, sorry, see you tomorrow | 16:36 | |
| ttx | karbor is the one with a standing -1 | 16:36 |
| ttx | so the one for which we should propose a release | 16:36 |
| smcginnis | Yep. I think that's waht we said, release karbor, transition freezer. | 16:36 |
| ttx | ok, you had said OK, switch karbor and force freezer | 16:37 |
| smcginnis | Oh, nope, I stated that backwards earlier. | 16:37 |
| smcginnis | Sorry. | 16:37 |
| ttx | as long as we are on the same boat | 16:37 |
| smcginnis | Do what I meant, not what I said. :) | 16:37 |
| evrardjp | oh interesting the three of us understood it correctly, but the words were wrong :) | 16:37 |
| smcginnis | ttx: As long as it's not a cruise ship. | 16:37 |
| evrardjp | smcginnis: hahaha | 16:37 |
| evrardjp | #nevertoosoon | 16:37 |
| smcginnis | #alwaystoosoon | 16:38 |
| smcginnis | OK, I think that horse is well beaten. | 16:38 |
| diablo_rojo | #neveracruiseship | 16:38 |
| smcginnis | Hah~ | 16:38 |
| smcginnis | #topic Validate countdown email | 16:38 |
| *** openstack changes topic to "Validate countdown email (Meeting topic: releaseteam)" | 16:38 | |
| smcginnis | #link https://etherpad.openstack.org/p/relmgmt-weekly-emails | 16:38 |
| smcginnis | Line 210 | 16:39 |
| smcginnis | diablo_rojo: Should we expand on the cycle highlights in there to point out appropriate content for that? | 16:41 |
| smcginnis | We at least have the links. | 16:41 |
| diablo_rojo | smcginnis, I was just debating that. | 16:41 |
| smcginnis | I added one line to address a common confusion at least. | 16:42 |
| diablo_rojo | I think there is enough detail there since you are linking to the email etc. | 16:42 |
| diablo_rojo | Yeah that is a good addition | 16:42 |
| diablo_rojo | Maybe add it to the dates list at the bottom | 16:42 |
| smcginnis | What is the deadline for these? | 16:42 |
| smcginnis | OK, perfect. | 16:43 |
| smcginnis | Thanks ttx as always for getting these prepped. | 16:43 |
| smcginnis | I will send out tomorrow, my morning. | 16:43 |
| diablo_rojo | Oh interesting | 16:44 |
| diablo_rojo | We dont have the deadline on the release schedule for ussuri | 16:44 |
| diablo_rojo | I will get that patch out today | 16:44 |
| smcginnis | Can't hurt. | 16:44 |
| smcginnis | #topic Unmaintained for older branches | 16:44 |
| *** openstack changes topic to "Unmaintained for older branches (Meeting topic: releaseteam)" | 16:44 | |
| smcginnis | Just wanted to check with the team. | 16:44 |
| smcginnis | We have some really older stable branches. | 16:44 |
| smcginnis | With some things being still "maintained" and some not. | 16:45 |
| smcginnis | It was never really clear to me what the signal would be to know if and when something would transition to unmaintained. | 16:45 |
| smcginnis | Other than the team expressing declaring it. | 16:45 |
| smcginnis | But I'm fairly sure not a lot of teams are still paying much attention to Ocata. | 16:46 |
| elod | if the branch of a project does not get patches about ~6 months. maybe? | 16:46 |
| smcginnis | I guess that's one sign. | 16:46 |
| smcginnis | Or proposed patches that are not getting merged. | 16:46 |
| ttx | yeah I like objective signs | 16:46 |
| smcginnis | Or broken CI. | 16:46 |
| ttx | since if we ask around there will always be someone saving it | 16:47 |
| elod | yes, those, too | 16:47 |
| smcginnis | Secondary question, do we need to wait for all teams before we show these as unmaintained on releases.o.o? | 16:47 |
| fungi | i suppose "broken ci" is only a signal if it's broken for a while and nobody is fixing it | 16:48 |
| fungi | that may be hard to gauge in a simple spot-check | 16:48 |
| smcginnis | Do we run stable periodic jobs on ocata yet? | 16:48 |
| evrardjp | fungi: I am afraid of a move to a pseudo noop job to make things green | 16:48 |
| smcginnis | Yeah, I've seen some of that. | 16:48 |
| elod | yes, there are periodic jobs still against ocata | 16:48 |
| smcginnis | elod: OK, great. | 16:48 |
| fungi | plenty of stable branch jobs are perpetually broken because nobody looks at them, but they're also usually not the same jobs being run against proposed changes | 16:49 |
| elod | and there are some constantly failing :) | 16:49 |
| smcginnis | It does seem like there's a usual set of failures every day. I haven't been paying attention to which branches those were coming from. | 16:49 |
| fungi | also one of the options spelled out in the extended maintenance section of the stable branch chapter in the project teams guide is to drop broken ci jobs | 16:49 |
| smcginnis | So I guess as far as this team is concerned, we just leave these as Unmaintained until we are told otherwise? | 16:50 |
| fungi | because the extended maintenance proponents felt having an untested (or nominally tested) branch still receiving patches was better than straight eol | 16:50 |
| elod | heat, nova-powervm and nowadays octavia are failing in ocata | 16:51 |
| johnsom | Ocata is long EOL for Octavia..... | 16:51 |
| smcginnis | OK, not something this team needs to solve today. Just wanted to raise it to get folks thinking. | 16:52 |
| smcginnis | johnsom: Was that announced on the mailing list? | 16:52 |
| smcginnis | But makes me think, we need some central place to track that too. | 16:52 |
| johnsom | Yeah, I would have to dig way back in the way-back machine | 16:52 |
| elod | hmmm. then i guess the jobs should be dropped in such cases | 16:52 |
| smcginnis | johnsom: Doesn't look like it every got an ocata-eol tag proposed. | 16:53 |
| johnsom | Pike was the first 1.0 release, so we announced that it was as far back as we go | 16:53 |
| smcginnis | That would reflect that state on the site. | 16:53 |
| fungi | maybe we're missing a clear process for removing periodic jobs when branches transition to unmaintained | 16:53 |
| smcginnis | We should document what needs to be done. Do the -eol tagging, remove periodic stable jobs, etc. | 16:53 |
| johnsom | We were just having a conversation about this on our channel. It's really not clear how individual teams can EOL or "declare" Unmaintained. beyond the e-mail list which gets lost to history. | 16:54 |
| fungi | the "new" extended maintenance process got rid of the eol tags, i thought | 16:54 |
| smcginnis | IIRC, part of the process was a team had to announce it and wait 6 months in case someone else came along and volunteered to maintain the older branch than the official team would. | 16:54 |
| fungi | or maybe it only got rid of deleting eol branches | 16:54 |
| smcginnis | fungi: It didn't get rid of branches being EOL though. Just how it got to that phase. | 16:54 |
| johnsom | Right, there was talk that the new process would no longer remove the branches like the old EOL did | 16:55 |
| smcginnis | Yeah, maybe the deletion. That sounds right. | 16:55 |
| elod | I can prepare something to the stable-branch page and we can refine there the general TODOs, is that OK? | 16:55 |
| smcginnis | elod: That would be great! | 16:55 |
| fungi | ahh, rereading there is an eol phase, which comes 6 months after unmaintained | 16:55 |
| fungi | https://docs.openstack.org/project-team-guide/stable-branches.html#end-of-life | 16:55 |
| smcginnis | It's that process of getting between maintained and eol that isn't as clear as I would like. | 16:55 |
| fungi | current process does still tag and delete branches at eol time | 16:55 |
| smcginnis | Anyway, not something we need to solve in this meeting. I'm going to move on to our last agenda topic. | 16:56 |
| smcginnis | #topic | 16:56 |
| *** openstack changes topic to " (Meeting topic: releaseteam)" | 16:56 | |
| johnsom | Frankly, we are talking about EOL up to Rocky. | 16:56 |
| smcginnis | #topic Check-release-approval false negatives | 16:56 |
| evrardjp | that's quite the topic | 16:56 |
| *** openstack changes topic to "Check-release-approval false negatives (Meeting topic: releaseteam)" | 16:56 | |
| smcginnis | johnsom: I think that makes sense. | 16:56 |
| ttx | yeah so the latest change did not really fix anything | 16:56 |
| smcginnis | #link http://zuul.openstack.org/build/d385f0d89418420baaf4b8a1d05d345f | 16:56 |
| fungi | looks like the transition from unmaintained to eol is pretty clear, it's the transition from extended maintenance to unmaintained which is muddy | 16:56 |
| ttx | we already have a false negative | 16:56 |
| smcginnis | Looks like the same symptoms. | 16:56 |
| smcginnis | So something we need help on the gerrit side to figure out. | 16:56 |
| ttx | Currently picking it apart to check if anything else is missing | 16:57 |
| fungi | ooh, i like this game. i'll play too | 16:57 |
| ttx | The code does not seem to have crazy code paths that would explain this behavior | 16:57 |
| ttx | It's not limited to a specific executor | 16:57 |
| smcginnis | Really seems like it would have to be a bug in gerrit. | 16:58 |
| ttx | smcginnis: yeah, one which I can't reproduce locally | 16:58 |
| ttx | I ran a lot of tests and never hit it | 16:58 |
| smcginnis | fungi: Do you know if there are any debug logs on the gerrit side that could help? | 16:58 |
| ttx | While it seems to be between 5% and 20% hit rate in zuul | 16:58 |
| fungi | gerrit's only useful logging is for the ssh api, really | 16:59 |
| ttx | interesting... | 16:59 |
| smcginnis | ttx: And that's not an option from the executor? | 16:59 |
| ttx | "permitted_labels": {} | 17:00 |
| ttx | The failed JSON has ^ | 17:00 |
| ttx | Successful JSON has the data in that array | 17:00 |
| smcginnis | We're over time, so I'm going to end the meeting. But we should keep on this. | 17:01 |
| smcginnis | #endmeeting | 17:01 |
| *** openstack changes topic to "OpenStack Release Managers office - Come here to discuss how to release OpenStack components - Logged at http://eavesdrop.openstack.org/irclogs/%23openstack-release/" | 17:01 | |
| openstack | Meeting ended Thu Mar 19 17:01:46 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 17:01 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/releaseteam/2020/releaseteam.2020-03-19-16.00.html | 17:01 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/releaseteam/2020/releaseteam.2020-03-19-16.00.txt | 17:01 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/releaseteam/2020/releaseteam.2020-03-19-16.00.log.html | 17:01 |
| fungi | my ipv6 route is busted so i'm having trouble pulling current state of the releases repo to make sure the liaison e-mail address matches what gerrit's using for that account | 17:02 |
| ttx | fungi: what's the easiest way to access the code that is actually running on our gerrit? | 17:02 |
| fungi | will be a few minutes | 17:02 |
| fungi | ttx: as in the source code for the gerrit service? | 17:03 |
| diablo_rojo | smcginnis, thank you! | 17:03 |
| ttx | I'd like to dig into what could make "permitted_labels" empty | 17:03 |
| ttx | fungi: yes... | 17:03 |
| evrardjp | thanks smcginnis and others ! | 17:03 |
| fungi | as of tomorrow the answer will be download our gerrit docker image | 17:03 |
| fungi | you picked an interesting day to ask ;) | 17:03 |
| clarkb | ttx: opendev/gerrit on openstack/2.13 branch iirc | 17:03 |
| ttx | ok | 17:03 |
| fungi | yeah, technically that will get you the gerrit we'll be running on review.opendev.org tomorrow | 17:04 |
| fungi | but for your purposes that's probably sufficient | 17:04 |
| ttx | yeah, I want to track down what fills permitted_labels | 17:05 |
| fungi | i've confirmed the cause isn't anything as simple as an e-mail address mismatch | 17:05 |
| ttx | because i can see how that would affect how "labels" gets filled | 17:05 |
| smcginnis | fungi: The interesting thing is the same account can comment twice, and the first time if fails, the second time it works correctly. | 17:05 |
| fungi | the preferred_email field in gerrit for that account matches the email entry for tripleo in data/release_liaisons.yaml | 17:05 |
| fungi | yeah, sounds like this could be zuul missing gerrit events? | 17:07 |
| fungi | https://zuul.openstack.org/builds?job_name=check-release-approval&project=openstack%2Freleases&change=713891 | 17:07 |
| ttx | fungi: how so? | 17:08 |
| ttx | We correctly make a Gerrit query and that query returns imcomplete data | 17:09 |
| ttx | That's totally a Gerrit bug | 17:09 |
| ttx | Not a missed event | 17:09 |
| fungi | that build corresponds with when rabi uploaded the change, the build timestamp is prior to alex's comment | 17:09 |
| fungi | unless i'm misunderstanding the problem | 17:10 |
| ttx | I'll restate the problem :) | 17:10 |
| fungi | gerrit says alex left a +1 at 14:29z | 17:10 |
| ttx | The script makes a query asking for detailed label information (o=DETAILED_LABELS, or more recently /detail) | 17:11 |
| ttx | and gerrit answers with erroneous information that does not match its JSON answer format | 17:11 |
| ttx | If you run it again, it will answer correctly | 17:11 |
| openstackgerrit | Javier Peña proposed openstack/releases master: Release python-keystoneclient 3.19.1 for stable/stein https://review.opendev.org/713938 | 17:11 |
| ttx | http://zuul.openstack.org/build/d385f0d89418420baaf4b8a1d05d345f/console shows the partial JSON response | 17:11 |
| ttx | In particular, missing labels['Code-Review']['all'] | 17:12 |
| ttx | but also missing other labels[*]['all'] | 17:12 |
| ttx | and permitted_labels | 17:12 |
| fungi | gerrit will omit empty fields | 17:13 |
| fungi | for some fields | 17:13 |
| ttx | in that case it does reply "permitted_labels": {} | 17:13 |
| ttx | http://paste.ubuntu.com/p/v8GrT3qFWD/ shows the complete answer | 17:14 |
| ttx | http://zuul.openstack.org/build/d385f0d89418420baaf4b8a1d05d345f/console shows the one we got | 17:14 |
| ttx | The only difference I could spot is in the structure of "labels" and the fact that permitted_labels is empty | 17:15 |
| ttx | I suspect those are related. Gerrit thinks tehre is no permitted label and switches the label data to a simpler content | 17:16 |
| ttx | more specifically, missing labels[*]['all'] and labels[*]['recommended'] | 17:16 |
| ttx | (while still providing labels[*]['values'] and labels[*]['default_value'] | 17:18 |
| fungi | though that task occurred at 14:29:36 which is prior to the 14:29:43 Code-Review +1 from alex | 17:18 |
| fungi | so i'm confused as to what it's trying to check there | 17:18 |
| openstackgerrit | Kendall Nelson proposed openstack/releases master: Add Cycle Highlights to U Release Schedule https://review.opendev.org/713940 | 17:18 |
| ttx | fungi: the originasl patchset sunmission | 17:18 |
| fungi | at the time that task ran, there were no labels with votes | 17:19 |
| ttx | fungi: if the patch is provided by PTL, it's PTL-approved | 17:19 |
| ttx | ah.... | 17:19 |
| ttx | I see what you mean | 17:19 |
| * ttx thinks | 17:19 | |
| fungi | so the job may need to treat missing labels as an indicator to look at the change owner id | 17:20 |
| ttx | so 'all' would be skipped | 17:20 |
| ttx | in case there is no single review yet | 17:20 |
| fungi | unless i completely misunderstand how it's implemented | 17:20 |
| ttx | still weird that permitted_labels would be empty | 17:20 |
| ttx | but let me check | 17:20 |
| fungi | wouldn't be the weirdest thing i've seen in gerrit's api ;) | 17:21 |
| fungi | also may entirely change when we upgrade | 17:21 |
| fungi | (though no eta on that, we need to switch to our containerized deployment tomorrow and then look at upgrade timelines) | 17:21 |
| ttx | ha | 17:21 |
| ttx | Yeah, we load the list of approvers from ['labels']['Code-Review']['all'] then add ['owner']['email'] | 17:22 |
| ttx | Let me see if all recent issues match that scenario | 17:22 |
| *** mlavalle has joined #openstack-release | 17:23 | |
| ttx | yep, that could explain it | 17:24 |
| openstackgerrit | Sean McGinnis proposed openstack/releases master: [tripleo] Transition Rocky to EM https://review.opendev.org/709912 | 17:24 |
| openstackgerrit | Sean McGinnis proposed openstack/releases master: Create stable/rocky branch for tripleo-ipsec https://review.opendev.org/713941 | 17:24 |
| ttx | will post fix tomorrow | 17:25 |
| ttx | fungi: thanks for your analysis! I did not take into account the fact that Gerrit would answer different formats | 17:26 |
| ttx | Still don't get why it would answer "permitted_labels: {}" | 17:27 |
| ttx | but will happily ignore that if my fix fixes the issue | 17:27 |
| openstackgerrit | Sean McGinnis proposed openstack/releases master: Release freezer ussuri deliverables https://review.opendev.org/713942 | 17:31 |
| smcginnis | Doh, got it backwards again!! | 17:33 |
| openstackgerrit | Sean McGinnis proposed openstack/releases master: Release karbor ussuri deliverables https://review.opendev.org/713946 | 17:35 |
| *** evrardjp has quit IRC | 17:36 | |
| *** evrardjp has joined #openstack-release | 17:36 | |
| openstackgerrit | Merged openstack/releases master: Switch freezer to cycle-with-rc https://review.opendev.org/709844 | 17:48 |
| openstackgerrit | Colleen Murphy proposed openstack/releases master: Release keystonemiddleware 6.0.1 https://review.opendev.org/713954 | 18:05 |
| *** rpittau is now known as rpittau|afk | 18:06 | |
| *** armstrong has quit IRC | 19:06 | |
| *** mlavalle has quit IRC | 19:49 | |
| *** mlavalle has joined #openstack-release | 19:58 | |
| *** e0ne_ has quit IRC | 21:04 | |
| *** e0ne has joined #openstack-release | 21:08 | |
| *** e0ne has quit IRC | 21:22 | |
| openstackgerrit | Merged openstack/releases master: Release tripleo-common 12.2.0 https://review.opendev.org/713891 | 21:39 |
| openstackgerrit | Merged openstack/releases master: Create stable/rocky branch for tripleo-ipsec https://review.opendev.org/713941 | 21:39 |
| openstackgerrit | Merged openstack/releases master: [glance] Transition Rocky to EM https://review.opendev.org/709888 | 21:39 |
| *** dhellmann_ has joined #openstack-release | 21:53 | |
| *** dhellmann has quit IRC | 21:54 | |
| *** dhellmann_ is now known as dhellmann | 21:54 | |
| openstackgerrit | Merged openstack/releases master: [tripleo] Transition Rocky to EM https://review.opendev.org/709912 | 22:20 |
| *** zigo has quit IRC | 22:22 | |
| *** zigo has joined #openstack-release | 22:32 | |
| *** slaweq has quit IRC | 22:45 | |
| *** jbadiapa has quit IRC | 22:50 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!