15:01:33 <smcginnis> #startmeeting releaseteam
15:01:34 <openstack> Meeting started Fri Mar 16 15:01:33 2018 UTC and is due to finish in 60 minutes.  The chair is smcginnis. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:36 <mlavalle> Thanks for attending
15:01:36 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:38 <openstack> The meeting name has been set to 'releaseteam'
15:01:49 <smcginnis> ping  dhellmann, dims, fungi, tonyb, lbragstad, ttx, armstrong
15:01:54 <knikolla> o/
15:02:04 <ttx> o/
15:02:06 <mlavalle> sorry for the delay smcginnis :-(
15:02:09 <lbragstad> o/
15:02:10 <annabelleB> o/
15:02:14 <smcginnis> mlavalle: No problem!
15:02:32 <smcginnis> Agenda can be found at https://etherpad.openstack.org/p/rocky-relmgt-tracking
15:02:36 <dhellmann> o/
15:02:36 <lbragstad> just fyi - knikolla is taking over the release liaison role for keystone
15:02:37 <smcginnis> Currently line 38
15:02:45 <smcginnis> lbragstad: ack
15:02:48 <smcginnis> Welcome knikolla
15:02:56 <lbragstad> so - expect to see him more frequently in the release meetings :)
15:03:09 <smcginnis> Excellent
15:03:15 <knikolla> smcginnis: thanks o/ :)
15:03:23 <smcginnis> #topic Cycle trailing status
15:03:43 <smcginnis> Long live queens I guess. :)
15:04:03 <ttx> indeed
15:04:10 <smcginnis> #link http://lists.openstack.org/pipermail/openstack-dev/2018-March/128215.html
15:04:32 <smcginnis> Normally we would be done, but now we have some time for those projects to wrap things up.
15:04:39 <smcginnis> ttx: Anything you want to say about this?
15:04:54 <ttx> not really
15:05:12 <smcginnis> Did we get updates to PROCESS to reflect these changes?
15:05:22 <ttx> just wanted to review the tasks above to see what's still needed
15:05:55 <smcginnis> The tasks in the etherpad? Or the end of cycle ones in the process doc?
15:06:26 <fungi> is there some idea as to when the stragglers might release?
15:06:44 <fungi> wondering if it makes sense for me to delay merging the signing key rotation a few days or something
15:07:04 <smcginnis> The ML post says up to 3 months after the coordinated release, so I don't think we should wait.
15:07:39 <smcginnis> Which does mean some Queens targeted things will be signed with the rocky key, but I think we end up like that anyway with stable releases, right?
15:08:46 <fungi> yeah, it's not a big deal (as discussed at last week's meeting)
15:09:06 <fungi> if we thought it was going to be sometime in the coming week then i would push off my monday plan
15:09:19 <fungi> but i'll just stick to the schedule
15:09:36 <smcginnis> It's possible there may be some, but no one that I'm aware of at the moment.
15:09:45 <smcginnis> So probably better not to hold things up for it.
15:09:47 <fungi> no sweat
15:09:56 <smcginnis> fungi: Thanks for checking.
15:10:19 <ttx> yes tasks on the etherpad
15:10:20 <smcginnis> Which also reminds me, infra had planned on some gerrit downtime after cycle-trailing release was complete.
15:10:38 <smcginnis> But that can probably still happen. It was/is planned for tomorrow.
15:10:59 <smcginnis> fungi: I forget the details. Do you know the nature of the dowtime?
15:11:26 <fungi> i think we decided to push it out a week due to one of the interested parties just coming back from vacation today
15:11:34 <fungi> it was just going to be a project rename
15:11:39 <fungi> well, a couple of project renames
15:11:50 <smcginnis> Ah, OK. I'll note that in our etherpad just to make sure we're all aware.
15:11:58 <smcginnis> Shouldn't make a difference over the weekend, but just in case.
15:12:09 <smcginnis> fungi: So current plan is one week from tomorrow?
15:12:15 <fungi> we were originally also going to roll in the nova-specs repository repair but then ended up having to step up the timeline and perform that in an emergency maintenance last week instead
15:12:26 <fungi> er, lemme double-check the meeting details
15:12:42 <smcginnis> No worries, we can follow up on that later.
15:13:05 <smcginnis> Anyone have anything else to discuss about cycle-trailing projects?
15:13:10 <fungi> sure, i'll let you know at the end of the meeting
15:13:39 <smcginnis> #topic Contribution ladder
15:13:48 <smcginnis> ttx: Did you want to take this?
15:13:53 <ttx> yes
15:14:28 <ttx> so I want to make sure we caputure the newcomers that came to us in Dublin
15:14:32 <ttx> capture
15:14:40 <ttx> annabelleB and armstrong
15:14:48 <annabelleB> newcomer reporting for duty!
15:14:49 <ttx> by giving them a way to get started
15:14:55 <smcginnis> #link https://docs.google.com/drawings/d/1B9u2VpLeuzj-1T1VJrpgaE7_uM9T7cRlEXLFf56ggKE/edit
15:15:10 <ttx> I feel like we need to describe levels of engagement
15:15:13 <smcginnis> #link https://etherpad.openstack.org/p/relmgmt-onboarding
15:15:16 <ttx> First level being
15:15:33 <ttx> a basic understanding of the process and infra machinery behind it
15:15:49 <ttx> being able to review 90% of the release requests and take a review day
15:16:16 <ttx> (including creating issues for job failures, not necessarily debug then)
15:16:20 <ttx> them)
15:16:37 <ttx> are we on the same page on what that first level would look like ?
15:16:51 <smcginnis> I like that description.
15:16:56 <ttx> or more ? or less ?
15:17:26 <dhellmann> that seems like a lot
15:17:30 <fungi> also, nice diagram
15:17:39 <ttx> basically it's about approving stuff that you feel comfortable approving
15:17:44 <dhellmann> maybe step 1 is just reviewing the release requests themselvs?
15:17:46 <ttx> and having a peek behind the scenes
15:17:53 <dhellmann> and step 2 is understanding more of what happens after approval
15:18:08 <ttx> dhellmann: you mean, in case of failure, just call Doug?
15:18:13 <ttx> I mean $Doug
15:18:23 <smcginnis> ttx: Is the idea that would be a spectrum where after all of step 1 is covered and understood, that would then be the point at which they would be added as core?
15:18:23 * dhellmann remembers to change his phone number again for this cycle
15:18:26 <fungi> doug-for-a-day
15:18:37 <smcginnis> :)
15:18:52 <smcginnis> WWDD - What would Doug do?
15:19:02 <dhellmann> I like this diagram a lot, btw
15:19:12 <ttx> smcginnis: I would add them as Core at stage 1, with understandig that they should only approve stuff they are comfortable approving
15:19:31 <ttx> I want then to be useful from day 0
15:19:35 <dhellmann> what is step 2?
15:19:37 <smcginnis> But not at the start of stage 1 right? Or we need a stage 0 for new folks to get to that point?
15:19:39 <dhellmann> or stage 2, I guess
15:20:29 <ttx> good question, I described it earlier, let me fetch that
15:20:30 * dhellmann opens the etherpad
15:21:08 <smcginnis> I started jotting down details in that etherpad.
15:21:16 <smcginnis> It definitely needs more work.
15:21:22 <ttx> stage 2 is knowing the cycle process by heart
15:21:33 <dhellmann> it sounds like stage 1 includes things like understanding the various rules for the different release models?
15:21:35 <ttx> stage 3 is knowing the whole infrastructure behind it
15:21:36 <smcginnis> But wanted to start capturing things to work towards some more official documentation.
15:21:41 <ttx> (also known as the Doug level
15:21:43 <ttx> )
15:22:11 <dhellmann> heh
15:22:14 <ttx> etherpad?
15:22:18 <dhellmann> https://etherpad.openstack.org/p/relmgmt-onboarding
15:22:25 * ttx creates etherpads for breakfast
15:23:04 <ttx> dhellmann: so regarding your stage 0.9 question above
15:23:24 <ttx> like not requiring basic understanding of the process / infra behind it
15:23:56 <ttx> I would say we can have a stage before my stage 1 where people just do non-core reviews
15:24:24 <ttx> But I feel like to be trusted to do release core reviews you need that basic understanding of what happens when you press W+1
15:24:38 <smcginnis> I do think someone needs to have some understanding and experience before we give them the stage 1 details and core rights.
15:24:47 <ttx> So stage 0.9 is actually a stage 0
15:24:49 <dhellmann> ok, I can see that as fair
15:24:51 <ttx> since anyone can do it
15:25:11 <ttx> So Stage 0 - release +1 reviews
15:25:29 <ttx> Stage 1 - minimal necessary to be trusted with W+1
15:26:06 <ttx> We could/should document the review rules though, even for stage 0
15:26:13 <dhellmann> I guess the stage definitions are "you must be able to do..." descriptions, rather than "you will do..."
15:26:23 <smcginnis> OK, I think we have the same thought, just conveying it a little differently.
15:26:24 <ttx> since I feel like people are not participating in release reviews ebcause they have no idea what we look for
15:26:39 <smcginnis> ++
15:26:51 <smcginnis> Regardless, we need better documentation of what is involved.
15:26:52 <dhellmann> yeah, I've tried to put as many of the rules into the code as I could, but the ones that are left are pretty subtle so docs would help
15:27:02 <ttx> heck, sometimes I wonder how to describe my process
15:27:24 <smcginnis> It might help us too, since we probably each approach the reviews slightly differently.
15:27:29 <ttx> It's almost supernatural sense of the yaml feeling right
15:27:48 <ttx> a combination of trusting the submitter to actually know semver
15:27:57 <ttx> and diving and check
15:28:24 <smcginnis> I think the validation testing has gotten good enough that the semi-subjective part of doing reviews is a lot more nuanced.
15:28:24 <fungi> i appreciate the level of detail there. i'm confident i could do those things today if i had the bandwidth for them
15:28:32 <ttx> which means that .Y bumps on current release series are mostly a no-brainer, just check PTL/liaison approval
15:28:37 <dhellmann> at this point a lot of the time it comes down to "you can't do this at this point in the cycle"
15:29:16 <smcginnis> Time in cycle, stable status, and semver adherance are probably the big gotchas.
15:29:30 <ttx> right, so we should document this
15:29:34 <dhellmann> ++
15:29:43 <ttx> Maybe make that part of stage 0
15:29:52 <ttx> "what to look for in a release review"
15:30:08 <smcginnis> There's probably a lot I put under stage 1 in the etherpad that can/should move to a stage 0.
15:30:20 <ttx> yes
15:30:46 <ttx> A lot of stage 1 is knowing when not to approve (like when there is some doubt) and leaving it to someone else
15:31:16 <ttx> frankly if we have people in stage 1 taking care of 90% of the requests, we'd be in a good place
15:31:29 <dhellmann> true
15:31:41 <smcginnis> I've been looking at these as transitional stages where you start at stage 1 without it and as you absorb the details I currently have in stage 1, you "complete" that level and move on to stage 2. Just a different way of looking at it.
15:32:11 <dhellmann> smcginnis : yeah, we should agree on whether we're using pre-versioning or post-versioning for stages ;-)
15:32:35 <dhellmann> after that it should be clear where to put each part of the description
15:32:39 <smcginnis> But it may be better to have the stage 0 and have these details a declarative statement of what should be known to be at that stage.
15:33:00 <dhellmann> this feels like another thing (like the process) that it would be good for us to *publish* as documentation, but we don't really have a place to do that.
15:33:24 <smcginnis> Should we have a doc section that gets published like other repos?
15:33:36 <dhellmann> how do you feel about extending some of the releases.o.o reference section with this stuff?
15:33:54 <smcginnis> Although, "publishing" as rst files when viewing the repo works well too.
15:33:59 <dhellmann> or yeah, maybe publish to docs.o.o/releases or something?
15:34:10 <smcginnis> I was thinking the latter.
15:34:33 <dhellmann> that makes sense
15:34:42 <smcginnis> Just conceptually I think of releases.o.o as a reference point for what has been released, not how to get it released. But that could be changed too.
15:34:59 <dhellmann> we've hesitated to add too much how-to material there for just that reason
15:35:00 <ttx> should we create a CONTRIBUTION_LADDER.rst file ?
15:35:19 <smcginnis> I think that would be a good next step after we churn through this etherpad.
15:35:22 <dhellmann> sure
15:35:29 <dhellmann> and we can figure out publishing separately
15:35:36 <dhellmann> maybe docs.o.o/releases/contrib like the other teams
15:35:40 <smcginnis> I figured it would take a lot of reworking and back and forth that would be easier done in an etherpad before publishing in a more static form.
15:35:47 <dhellmann> ++
15:35:52 <smcginnis> dhellmann: I like that idea.
15:35:52 <ttx> OK, I raised it as a topic for the meeting because I don't want our padawans to get bored to death waiting for us to get our act together
15:36:00 <smcginnis> :)
15:36:34 <dhellmann> would it be useful to set up a buddy system for training?
15:36:38 <smcginnis> Yeah, probably don't need to spend too much time in the meeting on this, but good start. We can keep working through this.
15:36:50 <dhellmann> or are folks comfortable just asking questions in channel and getting an answer from whoever is around?
15:36:52 <smcginnis> Pair releasing, like pair programming?
15:36:53 <fungi> probably fine to stick it in the usual CONTRIBUTING.rst and then use a sphinx include directive to stitch that into the docs tree
15:37:01 <annabelleB> :: would like a buddy ::
15:37:14 <dhellmann> annabelleB : sign me up
15:37:27 <fungi> since github (and presumably other systems) consider CONTRIBUTING.rst as a standard place to find that sort of info
15:37:28 <smcginnis> I can always be a backup too.
15:37:40 <dhellmann> fungi : good point
15:37:59 <smcginnis> Depending on the length, not sure if we want to include it in, or link to it. But we can see where things end up and what looks best.
15:38:24 * smcginnis looks at our CONTRIBUTING and realizes it's pretty bare
15:38:36 <smcginnis> Yeah, that may be a good place actually.
15:38:38 <ttx> one of the trickiest things to determine when reviewing release requests is whether it's a good time to release wrt. infra status
15:38:44 <dhellmann> we can at least start there
15:39:02 <ttx> so i still want to work on some grafana dashboard that would answer that question near-objectively
15:39:33 <ttx> If graphs look too wrong, probably not a great time
15:40:15 <smcginnis> That could be very useful.
15:40:16 <ttx> One of the things that is hard to extract from our metrics is time to merge
15:40:31 <dhellmann> there's a way to ask zuul about recent job failures, but if no release jobs have been run in a while the data may not be good
15:40:39 <ttx> Like look at top of gate and see how much time since it was enqueued amd how much time left to run
15:40:54 <ttx> add those two
15:40:59 <dhellmann> #link http://zuul.openstack.org/builds.html?pipeline=release
15:41:38 <ttx> which is why I fear that Grafana.o.o won't cut it
15:41:57 <ttx> the data is a bit hard to compile
15:42:08 <dhellmann> that report shows "duration" but I suspect that's only how long the job actually took to run, and doesn't account for any waiting period
15:42:09 <smcginnis> That duration shows how long it took the test to run, but not how long it had to wait before it started running.
15:42:15 <smcginnis> jinx
15:42:37 <ttx> It's still good input. Rate of recent fails on release jobs is prety useful
15:43:06 <ttx> Ideally we'd condense all those inputs into a single traffic light style indicator
15:43:18 <dhellmann> I wonder if that zuul data is available via an api
15:43:22 <ttx> and a number of dashboards to explain the area having problems
15:43:52 <dhellmann> this is drifting pretty far from an onboarding ladder, though
15:44:06 <fungi> the data is stored in a relational database, but i don't think there's (yet) an api exposed to query that beyond what the dashboard presents
15:44:11 <ttx> right... but I feel like it would be extremely useful for stage 1 people
15:44:34 <smcginnis> Yeah, we can probably move on for now and come back to this as we work through those details.
15:44:47 <fungi> though basic success/failure for specific pipeline+job combinations should also be queryable from graphite
15:44:49 <smcginnis> ttx: Want to add an action to our planning etherpad so it's not lost?
15:45:10 <ttx> on it
15:45:26 <smcginnis> OK, I'll move on then.
15:45:37 <smcginnis> #topic Sending periodic "unreleased" reports
15:46:16 <smcginnis> I do have some cinder related reports that I run periodically and publish to a web site.
15:46:28 <smcginnis> I've actually been meaning to add this as one of the things that gets run.
15:46:40 <smcginnis> It would make it easier for us to easily check on that.
15:46:50 <smcginnis> But good question as to whether we should send that data out.
15:47:16 <fungi> what kind of reports?
15:47:41 <dhellmann> I was thinking a periodic job that sent the output of list_unreleased_changes.sh for each team
15:47:50 <dhellmann> though only sending if there was actually something to send
15:47:57 <dhellmann> and a very long period, like a month
15:48:25 <dhellmann> so we'd have an email with a subject like "[cinder](pike) unreleased changes" and it would include all of the pike deliverables for that team
15:48:29 <smcginnis> I kind of like that idea.
15:48:48 <ttx> who would we send that to ?
15:48:52 <dhellmann> openstack-dev
15:49:31 <dhellmann> this came up because someone asked for a neutron pike release this week and when I looked there were dozens of patches
15:49:32 <ttx> that would be... pretty noisy
15:49:54 <dhellmann> well, if the email is only 1 per month and only if there are unreleased patches, it shouldn't be that many for teams on top of their releases
15:50:07 <dhellmann> and if they aren't doing backports it wouldn't generate anything
15:50:14 <dhellmann> and we would leave out master, of course
15:50:26 <smcginnis> Yeah, I think once a month, and only if significant patches found, wouldn't be too noisy.
15:50:27 <ttx> dhellmann: I fear the mailman day effect. So maybe that should send emails one month after last release
15:50:39 <dhellmann> ah, hmm
15:50:43 <dhellmann> yeah, that's an interesting idea
15:50:48 <ttx> i.e the job would run daily but consider the date of last release
15:51:00 <dhellmann> we could run the job daily, but if it hasn't been 30 days since the last release we don't announce anything either?
15:51:00 <ttx> that way you don't train people to ignore email every 1st of the month
15:51:10 <fungi> that seems less likely to fall through the cracks at least
15:51:12 <ttx> right
15:51:21 <dhellmann> ok, sure, that makes sense
15:51:25 <smcginnis> So if last_release > one_month and len(merged_patches) > signficant_amount: send email?
15:51:51 <dhellmann> I'll make some notes in the planning doc. I'm not sure I'm going to get to it this cycle myself, but I could advise someone else who wanted to work on it.
15:51:59 <ttx> smcginnis: well, you would not want to send one every day after the 30th day either
15:52:06 <fungi> or do you do it on a modulus of ~30 days since the last release?
15:52:10 <smcginnis> ttx: True
15:52:12 <ttx> fungi: what fungi said
15:52:24 <ttx> you send on every last release monthversary
15:52:35 <dhellmann> maybe significant_amount == 5?
15:52:42 <dhellmann> or higher?
15:52:58 <smcginnis> Bikeshed mode - engaged.
15:53:02 <dhellmann> hah
15:53:02 <ttx> 5 sounds fine to me if no release for a month
15:53:10 <persia> Maybe only send the monthversary notices when the number of patches has changed since the last notice
15:53:10 <ttx> and blue
15:53:14 <dhellmann> ok, I'll add a note
15:53:24 <ttx> stateful!
15:53:40 <persia> ttx: No, just check if a change landed in the last 30 days.
15:53:55 <smcginnis> It hasn't been uncommon to see 5 commits all for zuul job changes.
15:53:57 <persia> commit_date of master HEAD should be enough.
15:54:06 <ttx> ok, ok... if one no longer can troll
15:54:54 <ttx> ok 5 minutes left
15:55:28 <smcginnis> Probably should add this to the planning etherpad too?
15:55:50 <smcginnis> But one more topic, so I'll move on.
15:55:52 <dhellmann> line 177
15:55:58 <smcginnis> #topic Stopping requirements sync
15:56:06 <smcginnis> #link
15:56:08 <smcginnis> Oops
15:56:10 <smcginnis> #link http://lists.openstack.org/pipermail/openstack-dev/2018-March/128352.html
15:56:18 <dhellmann> this was a late addition, I just wanted to make sure folks saw the email and provided feedback
15:56:37 <smcginnis> I need to reread it to understand the issue with how we do things now.
15:56:45 <smcginnis> But responses so far seem favorable.
15:56:50 <dhellmann> I'm happy to talk through it if folks have detailed questions about the background
15:57:19 <smcginnis> Is there a one or two sentence summary of what problems we have now that this is avoiding?
15:57:34 <dhellmann> we end up releasing oslo libraries just because of requirements bumps
15:58:03 <fungi> projects (i could name examples) would like to be able to more accurately reflect their individual backwards compatibility with their own dependencies
15:58:13 <ttx> yes yjay is silly
15:58:22 <ttx> smcginnis: i think it removes unneeded complexity
15:58:24 <dhellmann> also some of the teams that want their tools to be usable on their own don't always want to use the latest version of a dependency that some other project needs
15:58:37 <smcginnis> But still enforces compatiblity for coinstallability?
15:58:47 <fungi> right
15:58:49 <dhellmann> yes, the upper-constraints.txt list provides that
15:59:04 <smcginnis> OK, sounds good to me then.
15:59:32 <dhellmann> my proposed change is a simplified version of what I remember the original plan being, where I don't ask teams to run integration tests using the lower bounds
15:59:51 <dhellmann> it doesn't prevent that, but as I have no real interest in it I'm leaving that to others
16:00:20 <dhellmann> it seems appropriate to try to have this done by the time we get to vancouver, since the plan was drawn up the last time we were there
16:00:34 <smcginnis> Anyone have last minute comments?
16:00:35 <dhellmann> the original plan, that is, not my proposal
16:00:36 <fungi> other benefits are 1. less back-and-forth between reqs repo changes and per-project changes when they want to adjust a minimum, and 2. no more reqs sync changes
16:00:36 <smcginnis> Out of time.
16:00:48 <fungi> reviewing the infra meeting log ( http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-03-13-19.01.log.html#l-60 ) and conferring with clarkb, the original plan was to do project rename maintenance later today but the revised plan is to reevaluate on tuesday and come up with a new scheduled date/time (so might be a week from today, might be longer)
16:00:49 <dhellmann> ack, use the ML thread for questions please
16:01:02 <smcginnis> fungi: Ack, thanks.
16:01:10 <smcginnis> Thanks everyone.
16:01:18 <smcginnis> #endmeeting