16:00:01 <smcginnis> #startmeeting releaseteam
16:00:02 <openstack> Meeting started Thu Apr  4 16:00:01 2019 UTC and is due to finish in 60 minutes.  The chair is smcginnis. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:04 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:06 <openstack> The meeting name has been set to 'releaseteam'
16:00:08 <smcginnis> ttx: You're jumping the gun. :)
16:00:14 * ttx might have to urgently pause in middle to drive kid somewhere
16:00:15 <smcginnis> #link https://etherpad.openstack.org/p/stein-relmgt-tracking
16:00:24 <ttx> that's why I'm starting early
16:00:24 <smcginnis> They can walk.
16:00:30 <smcginnis> :)
16:00:53 <smcginnis> We're on line 518.
16:01:09 <smcginnis> Ping list: smcginnis ttx dhellmann diablo_rojo hberaud evrardjp fungi armstrong
16:01:15 <dhellmann> o/
16:01:41 <evrardjp> o/
16:01:44 <smcginnis> #topic Stein release
16:02:03 <smcginnis> We had one unreleased that I took care of this morning, so we can skip that.
16:02:12 <smcginnis> Missing stable branches: There's a few.
16:02:16 <hberaud> o/
16:02:24 <smcginnis> Looks like the proposed patches did not include any branching.
16:02:46 <smcginnis> I figure we can branch those early tomorrow. Or should we do it today?
16:03:48 <evrardjp> mmm
16:04:00 <smcginnis> Or no one cares. :D
16:04:04 <evrardjp> you mean for L523?
16:04:25 <ttx> So it's normal to have missing stable
16:04:26 <dhellmann> do we have anything other than tempest plugins that don't have branches?
16:04:33 <smcginnis> Yeah, basically.
16:04:35 <ttx> we are supposed to cut them this week
16:04:43 <smcginnis> dhellmann: Yeah, there's a list of them in the etherpad in the meeting agenda.
16:04:45 <ttx> A couple are very old releases though
16:04:56 <ttx> so we might want to doublecheck recent changes
16:04:57 <smcginnis> ec2-api is very low activity.
16:04:58 <evrardjp> my question is what do you really want to branch :)
16:05:05 <smcginnis> Not sure about manila-ui.
16:05:11 <dhellmann> oh, sorry, I was looking too high up in that section of the pad
16:05:13 <evrardjp> and shouldn't that be at least done by the PTLs?
16:05:26 <ttx> It's all fine since they all have done a release during the cycle
16:05:37 <smcginnis> evrardjp: Well, a lot should be done by the PTLs, but they don't. We need to make sure a stable branch is created.
16:05:45 <ttx> and we said we would cut stable/stein from most recent release
16:06:08 <smcginnis> Yep, we did announce we would do them if they were not proposed by now.
16:06:09 <openstackgerrit> Ivan Kolodyazhny proposed openstack/releases master: Release Horizon Stein RC2  https://review.openstack.org/650055
16:06:14 <evrardjp> smcginnis: agreed on general releases, tagging. But here it's to create a branch -- I am not so sure. Sorry to get cold feet
16:06:17 <ttx> manila-ui has only test changes
16:06:28 <ttx> so 2.17.0 probably ok
16:06:48 <ttx> and 8.0.0 is current for ec2-api
16:06:49 <dhellmann> evrardjp : part of the move we've made over the last couple of cycles is to do more of this stuff for the teams, instead of trying to chase down everyone to ensure they do it themselves
16:06:52 <ttx> so all good
16:06:58 <smcginnis> evrardjp: It's a good point that a stable branch can be created from the tag as needed. But good to have them created before they are needed to make sure.
16:07:17 <evrardjp> dhellmann: agreed I like this, but I thought branching was a big thing due to the backport burden
16:07:29 <ttx> no branching is cheap
16:07:31 <evrardjp> but line 534 clarifies it's not a big thing
16:07:42 <smcginnis> I can propose the branches today and push them through. Or we can approve them early tomorrow. Any preferences?
16:07:42 <evrardjp> ttx: true for us :)
16:08:01 <ttx> smcginnis: approve early tomorrow
16:08:04 <gouthamr> smcginnis: to clarify, we did release stein with https://review.openstack.org/#/c/631776/ in manila-ui, no new changes were made, except gate stuff like ttx mentioned
16:08:18 <smcginnis> gouthamr: Thanks!
16:08:19 <evrardjp> smcginnis: no preference
16:08:21 <dhellmann> gouthamr : if we branch without those changes, your new stable branch will be broken
16:08:21 <tbarron> gouthamr: +1
16:08:27 <dhellmann> s/will/may
16:08:41 <dhellmann> do you need to tag a more current point release so we can branch from that?
16:09:14 <tbarron> dhellmann: gouthamr: we can certainly do that
16:09:19 <gouthamr> dhellmann: sure, yes.. tbarron is here, so we will do momentarily
16:09:31 <dhellmann> sounds good, thanks
16:09:44 <smcginnis> Great, thanks. That should help with the concern evrardjp raised of then making teams need to backport things.
16:10:30 <smcginnis> OK, translations...
16:10:47 <smcginnis> Horizon was the only one called out in the etherpad, but I see Ivan just proposed an RC2.
16:10:51 <smcginnis> So we should be good there.
16:11:00 <ttx> ++
16:11:06 <smcginnis> Late RCs
16:11:24 <smcginnis> I saw a new nova RC proposed.
16:11:30 * smcginnis checks if it includes that bug.
16:11:38 <ttx> there is a thread about that bug on the list
16:11:45 <ttx> so I think we need to keep an eye on that one
16:11:48 <smcginnis> Ah, missed it.
16:12:09 <ttx> it's the only panic-potential thing I know of
16:12:24 <smcginnis> Does look like something that's pretty critical.
16:12:34 <ttx> But then I have been traveling a bit last week, did you all hear of anything else?
16:12:43 <dhellmann> is that the oslo.service thing?
16:12:48 <ttx> smcginnis: it's critical if it is a regression
16:12:48 <smcginnis> Though Matt's last comment on the bug says it was there in rocky.
16:13:21 <ttx> right, so now it's almost a feature
16:13:23 <smcginnis> So not a regression.
16:13:30 * diablo_rojo sneaks in late becuase she didn't notice her IRCclient was disconnected
16:13:37 <smcginnis> Undocumented feature. ;)
16:14:19 <ttx> More generally, there is general flakiness in tests that point to a potential loss of quality
16:14:20 <smcginnis> So I think we can continue with plans to do final release proposal. If that comes through in time, great. If not, looks like it will need to be a stable backport at least for rocky.
16:14:31 <ttx> so we need to keep our ears on the ground
16:14:59 <smcginnis> ++
16:15:10 <dhellmann> we could propose the nova final release on top of that rc2 and let eric block it until they're ready
16:15:20 * dhellmann doesn't see efried in channel
16:15:35 <dhellmann> maybe melwitt is managing that instead?
16:15:48 <smcginnis> It is still stein.
16:15:54 <dhellmann> yeah
16:16:20 <smcginnis> We can follow up and watch for that.
16:16:38 <smcginnis> Let's move on to release week organization.
16:17:07 <smcginnis> So after the final RC deadline today, we can propose the final release patch.
16:17:21 <smcginnis> I can do that in the morning. We don't need to wait on the branching for that.
16:17:28 <dhellmann> ++
16:17:51 <smcginnis> For the newer folks, we add all PTLs to that and request that they ack the patch.
16:18:07 <smcginnis> Not strictly needed. We won't hold it if we don't get an ack. But gives us the warm fuzzies.
16:18:18 <smcginnis> And ends up in the metadata that the PTLs signed off on it.
16:18:52 <ttx> I don't think we did: "The week before final release test the release process using the openstack/release-test repository."
16:19:08 <dhellmann> I saw one of those earlier today? or late yesterday?
16:19:12 <dhellmann> and we've done releases this week
16:19:13 <smcginnis> We did do a release-test release yesterday.
16:19:32 <smcginnis> I used dhellmann's test patch to make sure things were working after the sphinx 2 issues.
16:19:50 <smcginnis> s/yesterday/earlier this week/
16:19:56 <smcginnis> I've lost track of all time.
16:19:58 <dhellmann> that's in the process because we don't expect to be doing releases this week, and once we had an unexpected failure in the process
16:20:05 <ttx> cool
16:20:12 * ttx strikes out
16:20:14 <smcginnis> We've had a LOT of releases this week this time around.
16:20:16 <dhellmann> do we have a step to ask infra to hold off changing zuul config that might effect releases?
16:20:29 <smcginnis> I don't think we call that out.
16:20:30 <ttx> "On the morning of the deadline for final release candidates, check the list of unreleased changes for milestone projects and verify with the PTLs and liaisons that they are planning a release or that they do not need one." ??
16:20:42 <ttx> I haven't checked that ^
16:21:38 <dhellmann> I can run the script while we continue the meeting
16:21:43 <dhellmann> tools/list_rc_updates.sh
16:21:44 <ttx> dhellmann: thx!
16:21:44 <smcginnis> OK, thanks dhellmann
16:22:14 <smcginnis> Next week we should run the missing-releases script to look for any missing tarballs, pypi packages, etc.
16:22:27 <smcginnis> Anyone want to volunteer for that?
16:22:30 <ttx> ack, can do
16:22:36 <fungi> yeah, exercising the automation lightly before pushing the big red button remains a good idea. the infra team is trying to limit changes which could impact release processes, but you never know what could slip through
16:22:44 <smcginnis> Another easy "run script, report issues" task evrardjp and diablo_rojo_phon ;)
16:22:54 <smcginnis> Thanks fungi
16:23:31 <evrardjp> will have a look smcginnis
16:23:43 <evrardjp> I suppose the ttx "ack" was for that, so I can do that with him
16:23:52 <ttx> evrardjp: sure
16:23:53 <diablo_rojo> I can look too
16:24:23 <smcginnis> Thanks!
16:24:53 <smcginnis> Next task - after release, we need to update series_status so releases.o.o shows it as released.
16:25:05 <smcginnis> Pretty simple update with an example of previous times.
16:25:11 <dhellmann> here's the unreleased rc report: http://paste.openstack.org/show/748878/
16:25:12 <dhellmann> I can look through that report for anything important
16:25:59 <smcginnis> Good, mostly normal branching updates for most of them.
16:26:40 <smcginnis> Any volunteers to update the series_status. It's an easy one.
16:26:51 <smcginnis> I can take it if no volunteers.
16:26:58 <smcginnis> But trying to "delegate".
16:27:31 <smcginnis> OK, I'll put my name on that one. Just let me know if someone else wants to grab it though.
16:27:32 <ttx> Confirming PR is officially going out at 10am CT on the 10th, so we should target finishing the release process around that
16:27:45 <evrardjp> take it,and I will check if I can help you tomorrow
16:27:49 <evrardjp> smcginnis: ^
16:27:49 <diablo_rojo> smcginnis, I would say go ahead and do it, but I will look at it too.
16:27:56 <ttx> (it's also ok if we take more tiem, it's just a nice target to have)
16:27:59 <evrardjp> diablo_rojo: haha :)
16:28:01 <smcginnis> evrardjp: OK, thanks.
16:28:07 <smcginnis> ttx: Thanks, was wondering about tat.
16:28:11 <smcginnis> that
16:28:52 <smcginnis> I'll get the announcement ready to send out Wednesday.
16:29:03 <smcginnis> Do we usually wait for the PR to go out first, or doesn't it matter.
16:29:10 <smcginnis> I can't remember now. Just around the same time?
16:29:26 <ttx> We try to finish before
16:29:30 <openstackgerrit> Doug Hellmann proposed openstack/releases master: heat 12.0.0.0rc2  https://review.openstack.org/650066
16:29:47 <ttx> like 1-30min before the PR release
16:30:04 <smcginnis> OK, then we should approve that final patch early and I will have the announcement queued up to go.
16:30:09 <ttx> (the PR usually leaks as press articles want to be the first out)
16:30:36 <ttx> final patch can be approved a long time before
16:30:46 <fungi> in don't have anything going on that morning, so will make sure i'm on hand to keep tabs on the automation/ci bits
16:31:14 <smcginnis> fungi: Thanks, that will be great to make sure nothing gets tripped up.
16:32:01 <smcginnis> ttx: Do you want to hit the approval on the final release patch earlier in the day so it's done in time?
16:32:14 <ttx> smcginnis: that is generally what we do
16:32:22 <smcginnis> We can all +2 it by Tuesday, so it should be good to go.
16:32:41 <ttx> re: unreleased changes, looks like we should ping a few teams for late RCs before doing teh final-release patch
16:32:44 <openstackgerrit> Doug Hellmann proposed openstack/releases master: magnum 8.0.0.0rc2  https://review.openstack.org/650067
16:33:05 <smcginnis> Hah, or just propose them I suppose. Thanks dhellmann
16:33:20 <dhellmann> yeah, there are only a few, I'll just do them now
16:33:31 <ttx> ++
16:33:45 <evrardjp> thanks dhellmann
16:33:45 <openstackgerrit> Doug Hellmann proposed openstack/releases master: mistral 8.0.0.0rc2  https://review.openstack.org/650069
16:33:57 <ttx> OK I think that is all
16:33:57 <smcginnis> Let's get those through as soon as they pass today. Add branches and propose final release tomorrow morning.
16:34:18 <smcginnis> Collect PTL +1's, add out own +2's, then ttx approves early on the 10th.
16:34:19 <openstackgerrit> Doug Hellmann proposed openstack/releases master: murano 7.0.0.0rc2  https://review.openstack.org/650070
16:34:24 <smcginnis> Love it when a plan comes together.
16:34:39 <evrardjp> :)
16:34:58 <openstackgerrit> Doug Hellmann proposed openstack/releases master: octavia 4.0.0.0rc2  https://review.openstack.org/650071
16:35:02 <smcginnis> I think (knock on wood) we are in good shape.
16:35:12 <dhellmann> ok, that's all of them
16:35:28 <smcginnis> #topic Open floor
16:35:29 <johnsom> dhellmann I already have a release patch up
16:35:45 <dhellmann> johnsom : ah, thanks, which project?
16:35:51 <fungi> so... what time on wednesday should i be keeping an eye on things? 14:00z? earlier?
16:35:53 <johnsom> dhellmann https://review.openstack.org/649765
16:36:09 <dhellmann> ah, your wip dropped it off of my dashboard
16:36:20 <smcginnis> fungi: It sounds like earlier since we want it complete by 15:00
16:36:31 <johnsom> Yeah, just waiting for a patch or two to get through zuul
16:36:48 <fungi> smcginnis: thanks, i mathed wrong ;)
16:37:06 <smcginnis> ttx: You want to coordinate with fungi on timing there to have the fire brigade ready?
16:37:10 <ttx> All set. Just need to find a stein to drink on release day
16:37:37 <fungi> i have a wooden stein on hand
16:37:41 <fungi> not lidded though
16:37:53 <ttx> fungi: how much time do you expect the patch to run?
16:38:11 <evrardjp> I have two of them not sure I can post one of the two in time. ttx :(
16:38:22 <ttx> i was thinking on hitting W+1 around 9:00utc
16:38:25 <fungi> judging from recent past releases, seems like it was something like 30 minutes from push to finish
16:38:42 * smcginnis is really disappointed no vendors used the opportunity to give out stein swag
16:38:44 <ttx> ah, then yes, 13z?
16:38:54 <evrardjp> smcginnis: :)
16:39:00 <evrardjp> ttx: isn't that short on time?
16:39:04 <fungi> i can certainly be around at 09:00z but might be a bit groggy
16:39:21 <ttx> fungi: how about 12:00z
16:39:21 <smcginnis> Hopefully you can sleep through things.
16:39:34 <fungi> we'll probably have ianw and frickler more wide-eyed and bushy-tailed even if i'm not
16:39:38 <ttx> that leaves us 3 hours to fix things on a 30-min run script
16:39:42 <fungi> but sure, 12:00z is easy for me
16:40:07 <melwitt> hi, I just wanted to mention that the bug we're holding rc2 for is new in stein and is a security problem (leaks cell database and message queue credentials to the notifications message queue)
16:40:12 <dhellmann> this is for wednesday, right?
16:40:16 <ttx> also, not the end of the world if things are not ready by 15z
16:40:31 <fungi> melwitt: i saw, thanks for prioritizing that one!
16:40:38 <smcginnis> melwitt: Ah, I remember Matt mentioning that. How's that coming along?
16:41:01 <evrardjp> ttx: I suppose it's not great to have PR on something not yet released? Or did I misunderstand the times?
16:41:09 <melwitt> smcginnis: master change is approved, we just had to recheck it a few min ago. so just have to babysit through the gate and then approve the stein backport after
16:41:37 <smcginnis> OK, good. Then barring gate shenanigans, we should be good on timing there.
16:41:47 <ttx> evrardjp: PR release time is 15z. If we push the patch around 12z that will be done by then
16:42:05 <smcginnis> fungi: Maybe we should bump up that nova patch?
16:42:11 <evrardjp> yeah but it was for your comment "not the end of the world if things are not ready by 15z"
16:42:16 <ttx> Also PR is not really resulting in thousands of downloads.
16:42:25 <evrardjp> ttx: fiar
16:42:27 <evrardjp> fair*
16:42:29 <fungi> i've blocked out 1200-1500z in my schedule on the 10th now
16:43:25 <ttx> What we need to have up by 15z is the links that the PR points to (the landing page for stein release)
16:43:28 <dhellmann> I'll be around early, but have to leave for a while at 1400
16:44:06 <ttx> that in turn points to releases.o.o/stein which may still have RCs and nobody would notice :)
16:44:22 <fungi> evrardjp: it's a double-edged sword, because it we release too far ahead the outlets who are syndicating the press release feel undercut because details will have already had time to circulate and steal their thunder
16:44:28 <evrardjp> ttx: I see
16:44:38 <evrardjp> fungi: indeed
16:45:00 <smcginnis> I will plan on running the meeting next Thursday so we can debrief. But then tonyb will likely take over from there. Let's watch for updates from him on meeting details.
16:45:09 <ttx> also, we usually get press articles BEFORE the release is out anyway
16:45:29 <fungi> smcginnis: melwitt: is there urgency in getting the master security fix in before proposing/approving the backport for stein?
16:46:09 <dhellmann> I hope the backport patch is already up?
16:46:12 <smcginnis> Normally we want the later one to merge before cherry-picking, but probably safe to go ahead in this case.
16:46:14 <melwitt> we have the stein backport proposed
16:46:19 <smcginnis> OK, good.
16:46:25 <evrardjp> fungi: I guess if +w  and merged while master is not merged, then it could be a pain .
16:46:28 <melwitt> it's just we don't +W until the master change merges
16:46:50 <melwitt> yeah, exactly evrardjp
16:46:53 <fungi> fair enough. how long do we expect that to set back the last rc?
16:47:00 <melwitt> if the master change fails in the gate or something
16:47:01 <evrardjp> then it's fine if it's just waiting for +w
16:47:22 <evrardjp> I guess babysitting will matter :)
16:47:30 <evrardjp> it always is!
16:47:35 <melwitt> yes, we are babysitting it
16:47:48 <melwitt> just had to recheck it for an ssh timeout a little while ago
16:48:14 <smcginnis> Just wondering if it needs an infra magic kick to the top to make sure it makes it through in time to not impact final release activities.
16:48:16 <evrardjp> hoping the gates gods are stable
16:48:17 <johnsom> Same here, we have two in the gates, but keep getting nova timeouts. 5 minutes to schedule?? ugh
16:48:38 <melwitt> ?? that doesn't sound normal
16:48:49 <johnsom> Agreed. That was limestone
16:48:59 <melwitt> oh, yeah ok
16:49:04 <johnsom> Vexxhost just had a cirros stuck in build
16:50:01 <evrardjp> that's probably a discussion for infra :)
16:50:06 <fungi> yeah, looks like there's a bit of a run on the gate for a variety of failures
16:50:35 <fungi> per ttx's earlier comment about various sorts of failures on the rise suggesting openstack code quality slipping
16:51:19 <johnsom> I think what we are seeing is mostly high load on limited resources. These have not been happening over the course of the release.
16:52:00 <smcginnis> I guess we'll watch and see what happens.
16:52:06 <smcginnis> Anything else for the meeting today?
16:52:10 <ttx> nope
16:52:11 <evrardjp> none
16:52:13 <fungi> yep, but remember these tests are running on openstack... so problems seen in the infrastructure we're using are still quite oftenn openstack problems
16:52:44 <cdent> fungi++
16:52:49 <smcginnis> Thanks everyone!
16:52:51 <smcginnis> #endmeeting