15:00:03 <smcginnis> #startmeeting releaseteam
15:00:04 <openstack> Meeting started Fri Jun  1 15:00:03 2018 UTC and is due to finish in 60 minutes.  The chair is smcginnis. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:05 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:07 <openstack> The meeting name has been set to 'releaseteam'
15:00:10 <smcginnis> Ping list: dhellmann, dims, fungi, tonyb, lbragstad, ttx, armstrong, annabelleB
15:00:16 <annabelleB> o/
15:00:17 <ttx> Hi!
15:00:23 <smcginnis> #link https://etherpad.openstack.org/p/rocky-relmgt-tracking Agenda
15:00:23 <fungi> howdy
15:00:24 <dhellmann> o/
15:01:12 <fungi> looks like we're around line 216 today?
15:01:31 <smcginnis> Yep, looks right.
15:01:37 <smcginnis> #topic Rocky-2 tasks
15:01:44 <smcginnis> #link https://github.com/openstack/releases/blob/master/PROCESS.rst
15:02:10 <smcginnis> So other than tagging the milestone releases, I guess I should have included a mention of the unrelease changes.
15:02:25 <smcginnis> I actually did run the report, but didn't include that.
15:02:28 <ttx> I'll be traveling Tue-Thu, but shoudl be around to take my share on teh Wednesday
15:02:33 <smcginnis> I'll get that next tune,
15:02:36 <smcginnis> *time
15:02:47 <smcginnis> ttx: Great, thanks.
15:03:01 <smcginnis> Any other issues/concerns for next week's milestone activities?
15:03:27 <smcginnis> OK, I don't think I have anything going on, so should be good on my end.
15:03:36 <smcginnis> #topic Tagging right fixes progress
15:03:37 <dhellmann> no concerns here
15:03:52 <fungi> no planned infrastructure disruption either
15:04:02 <smcginnis> fungi: OK, great!
15:04:10 <smcginnis> ttx: Was this your topic?
15:05:42 <ttx> yes
15:06:01 <ttx> so... ec2api was done already
15:06:23 <ttx> For dragonflow, I looked it up and they did not do a queens release
15:06:42 <dhellmann> that's unfortunate
15:06:44 <smcginnis> Surprised we missed that until now.
15:06:48 <ttx> which makes me question what to do there
15:06:51 <fungi> i thnik they may be one of those projects which trail substantially
15:06:54 <fungi> er, think
15:07:18 <fungi> as in they develop against the latest neutron release rather than against master?
15:07:27 <ttx> given where it sits on the map (neutron plugin) I would be inclined to consider it for unofficalnessity
15:07:41 <smcginnis> At least I don't see them listed as part of the coordinated release: https://releases.openstack.org/queens/index.html
15:07:49 <ttx> probably one to prioritize in the "health" TC activity
15:08:09 <ttx> last release was 9 months ago
15:08:10 <dhellmann> I'll make a note of that now
15:08:32 <ttx> They have a bunch of commits though, so it's not "dead"
15:08:35 <smcginnis> Ah, I seem to recall some internal mention of this.
15:09:11 <fungi> 2017-09-01 for their 4.0 release
15:09:12 <dhellmann> they have 393 commits since that last release
15:09:25 <dhellmann> that seems pretty active, are they just not tagging?
15:09:35 <ttx> It's hard to consider it a part of openstack if it's not released basically
15:09:40 <dhellmann> yeah
15:09:56 <fungi> so they basically released 4.0 immediately after the coordinated pile release
15:10:05 <ttx> I'm not saying it should die... I just question the value of it being seen as a part of the product rather than a product add-on
15:10:07 <fungi> er, pike release
15:10:16 <smcginnis> So do we propose to move them out of governance?
15:10:29 <ttx> smcginnis: I'd wait for someone from teh TC to dive into it
15:10:35 <dhellmann> fungi : quite the freudian slip that was :-)
15:10:39 <fungi> hah
15:10:42 <smcginnis> :D
15:10:43 <ttx> We just need to prioritize it up
15:10:46 <fungi> and get some feedback from the dragonflow devs too, of course
15:11:09 <dhellmann> I've added a note to the "actively monitoring teams" section of the tc tracker in the wiki
15:11:09 <ttx> next up:
15:11:16 <fungi> looks like oanson is probably the most active contributor there (and has been pushing their tags)
15:11:16 <smcginnis> I can start a ML thread after the meeting raising the question.
15:11:24 <dhellmann> I intended to raise that in office hours next week when most everyone is back online
15:11:30 <smcginnis> dhellmann: ++
15:11:33 <ttx> Rally -- I overlooked their response to the last ML thread about it
15:11:41 <ttx> http://lists.openstack.org/pipermail/openstack-dev/2018-April/129405.html
15:11:41 <dhellmann> smcginnis : a ML thread from the release PTL would be a good start, too
15:11:57 <ttx> basically arguing that non-cycle-based things should retain tagging rights
15:11:58 <dhellmann> I don't find either of those arguments particularly convincing
15:12:04 <smcginnis> #action smcginnis to start ML thread asking for Dragonflow status update
15:12:21 <dhellmann> but yeah, if they don't want to be listed on the releases.o.o site I guess I don't want to argue over it
15:12:37 <smcginnis> We do state now in the documentation that coming under governance means the release team must be the ones handling tagging.
15:12:37 <ttx> dhellmann: yeah, so my initial "ACL compliance" target was the openstack / openstack-operations buckets from the map[tm]
15:12:58 <fungi> so rally wants to keep their release notes in the tag message rather than in their repository content?
15:13:15 <ttx> they want to keep not depending on us, I think
15:13:43 <dhellmann> that seems like the real reason, yes
15:14:11 <dhellmann> we have some old data about releases. we could just remove that so the site doesn't appear to be out of date.
15:14:18 <ttx> I'm fine with the "I should be able to do things how I damn well please" mentality, but I also think things can live outside of the "openstack" official space
15:14:25 <fungi> right, the message indicates a dislike for the rules about when releases are appropriate, and a concern that there might not be release team members around to approve (urgent?) release requests
15:14:30 <smcginnis> ttx: Agree
15:14:39 <ttx> wanting both is what I disagree with
15:14:52 <dhellmann> ++
15:14:53 <smcginnis> Sounds like this one should be "OpenStack hosted" and not officially governed if that's what they want.
15:15:03 <ttx> we hold "openstack deliverables" to a given standard, and that standard now includes managed released
15:15:32 <ttx> if only so that you can not make non-peer-reviewed release blunders that would stain the openstyck name
15:15:52 <fungi> infra being the main outlier, and as noted in the most recent ml thread we want to move most of the infra repos out of openstack officially too
15:16:16 <fungi> so i think it's a fine precedent
15:16:38 <dhellmann> so, does someone need to reply to that ML thread proposing the 2 options? either letting us manage the releases or removing rally from governance?
15:16:59 <smcginnis> Sounds like something the TC chair should address. :)
15:17:04 <ttx> I'd say that's a TC-level discussion
15:17:08 <fungi> agreed
15:17:16 <ttx> enforcing that rule that is now in governance or not
15:17:17 <dhellmann> ok, maybe our release team PTL will raise that then? :-)
15:17:36 <smcginnis> Hehe, OK.
15:18:11 <ttx> I think the key is what are the minimal stquality andards we want to enforce for things to be associated with the "openstack"
15:18:25 <ttx> damn it weechat+mosh stop eating my characters
15:18:49 <dhellmann> smcginnis : if you lay out the reasons why we have the rule, I will offer the option of rally leaving governance
15:19:25 <ttx> I think those quality standards now include peer-reviewed releases basically
15:20:05 <ttx> and that is what that resolution said
15:20:18 <smcginnis> dhellmann: Are you thinking as a reply on that thread? Or should I bring this up in the next office hours?
15:21:06 <ttx> reply to thread would be good
15:21:10 <dhellmann> smcginnis : let's start by reviving the thread. I'll make sure the tc-members see it.
15:21:16 <fungi> i think if it's brought up on the thread as a discussion, that works
15:21:26 <dhellmann> maybe add [tc] to the subject if it isn't already there
15:21:27 <ttx> Andrey felt like his answer went into deaf ears
15:21:27 <smcginnis> OK, sounds like a plan.
15:21:55 <ttx> I for one missed it
15:22:05 <ttx> ok, next topic!
15:22:10 <smcginnis> I saw it, but didn't plan on responding right away.
15:22:10 <fungi> yeah, i honestly don't remember seeing his reply
15:22:23 <fungi> i probably did read it and immediately forgot
15:23:04 <dhellmann> yeah, I'm not sure why I glossed over that one
15:23:17 <smcginnis> #action smcginnis to respond to http://lists.openstack.org/pipermail/openstack-dev/2018-April/129405.html with reasons why governed projects need to use the release team
15:26:02 <smcginnis> ttx: Want to discuss the aclfixer patch?
15:26:15 <fungi> so what's the deal with the puppet-pacemaker addition then? just needs to have the acl updated to conform or is that an indefinite blacklist?
15:26:15 <ttx> should be straightforward
15:26:27 <ttx> just needs to be fixed, I think
15:26:30 <dhellmann> yeah, I wanted to ask about that, too
15:26:45 <dhellmann> should we just fix their acls?
15:26:47 <ttx> I could post the fix and update that aclfixer patch
15:27:10 <ttx> just wanted to have time to dive deeper into it see where that came from
15:27:18 <ttx> to prevent it from happening again
15:27:33 <dhellmann> ok
15:27:42 <ttx> I'll post the patch and update the aclfixer list
15:27:51 <ttx> so don't approve just now
15:28:03 <smcginnis> OK, sounds good.
15:28:23 <smcginnis> #topic Stable library maintenance
15:28:26 <ttx> OK, so I realized this morning that I signed up for that one, and wanted to discuss it for a bit
15:28:50 <ttx> what would be your definition of a stable library ? Wouldn't we expect req bumps on those ?
15:29:05 <dhellmann> since we no longer sync them, we might not have any
15:29:09 <dhellmann> think oslo.i18n
15:29:33 <dhellmann> that's basically never going to see new features, and will only get bug fixes if there are any, but it's a thin wrapper around gettext, so there aren't likely to be many
15:29:39 <smcginnis> I think we would need them. Aren't the older ones still just using g-r and u-c?
15:30:04 <dhellmann> we don't change g-r on older branches
15:30:10 <dhellmann> they do use u-c, but those aren't synced
15:30:26 <ttx> ok
15:30:33 <ttx> so in this case...
15:31:00 <ttx> the less-disruptive way is probably to allow multiple stable branches to be created from a given master release
15:31:14 <dhellmann> we'll have a lot of somewhat-but-less stable libs, but oslo.i18n seems likely to stay as it is for ages
15:31:31 <ttx> We would just branch it from the last master release if nothing was pushed there since last stable
15:31:59 <ttx> In case new things land in master, we just need to jump n stable branches in that versioning
15:32:05 <ttx> x.y+n.0
15:32:19 <ttx> Any obvious issue with that strawman ?
15:32:33 <ttx> I feel like not creating branches until they are needed is actually more painful
15:32:37 <dhellmann> it sounds like extra backporting work
15:32:48 <ttx> backporting?
15:33:03 <dhellmann> if we have a queens version that ends up being used for 3 cycles, then we have a fix, would we backport it to all of those other empty branches?
15:33:13 <dhellmann> why not just backport from master to queens and do 2 releases?
15:33:20 <fungi> seems like it might be challenging to predict version numbers for those branches?
15:34:09 <ttx> hmm let me think... we rely on constraints rather than branch names in tests, right
15:34:35 <dhellmann> in unit tests, yes
15:34:36 <dhellmann> the branch names are used when testing changes to the libraries in the integration tests
15:34:44 <dhellmann> ideally we'd be able to use version numbers instead of series names, but that's a lot of tooling updates
15:34:53 <dhellmann> stable/2.0 instead of stable/queens
15:34:56 <ttx> I fear that tracking which branch is used for where might be a bit challenging
15:35:03 <ttx> What we really need is branch aliases :)
15:35:05 <dhellmann> the telemetry team tried that and it turned into a lot of work
15:35:13 <fungi> say we branch stable/rocky and then later branch stable/stein from the same commit... later we discover a bug (security vulnerability fix?) which needs to be committed in master and backported... what do we tag for the stable branch release versions?
15:35:51 <ttx> so.. master has 1.18.0
15:35:57 <dhellmann> if the version before stable/rocky was 1.0.0 then the version on stable/rocky is 1.1.0 and the version on stable/stein is 1.2.0
15:35:59 <ttx> you create stable/rocky from that
15:36:04 <dhellmann> the release tools already require that
15:36:17 <ttx> then create stable/stein from that
15:36:27 <smcginnis> I have to admit, I'm not following all this. Why do we need to do something different than our current stable library releases?
15:36:32 <ttx> then you fix the thing in master and release 1.20.0
15:36:38 <fungi> so as long as we pick the stable/rocky version number before we pick the stable/steni version number, it could work out
15:36:41 <ttx> then 1.18.1 and 1.19.0
15:36:44 <dhellmann> fungi : yes
15:37:08 <fungi> but we need to make sure we don't tag something in stable/stein before stable/rocky
15:37:14 <ttx> what dhellmann says is that it's preferable to do 1.19.0 on master and 1.18.1 for rocky AND stein
15:37:19 <fungi> or we could paint ourselves into a corner with version numbering
15:37:28 <dhellmann> smcginnis : I guess the question is, do we need to create empty branches when there were no changes for several series?
15:37:35 <ttx> fungi: that's why I suggest 1.19.0
15:37:51 <dhellmann> ttx: right, we would just use the rocky version in the stein series, too
15:38:02 <smcginnis> dhellmann: Just speaking from experience with os-brick, we always create a stable branch.
15:38:04 <dhellmann> I don't know if that's going to be confusing for downstream folks, but that's how we treat 3rd party tools
15:38:24 <smcginnis> Of course, it always does have changes, so I guess that's the key here.
15:38:32 <fungi> ttx: right, my example could be extended to include stable/t and we need to make sure that we pick version numbers across r, s and t which are in order then
15:38:39 <dhellmann> right
15:38:45 <smcginnis> But I think we should still create branches each release for any "active" libs.
15:38:50 <ttx> I'd say the 3 options are
15:38:55 <fungi> so can't just rely on incrementing y in x.y.z
15:39:05 <dhellmann> fungi : yeah, we'll have to be careful with the reviews. or we could make the tool smart enough to detect those errors
15:39:12 <ttx> 1/ force "artificial" releases even if there are no changes and just reuse same process
15:39:30 <dhellmann> if we create a branch, there will always be at least 1 patch containing the .gitreview update
15:39:35 <ttx> 2/ do not force releases, but still create branches from latest releases before they are even needed
15:39:51 <ttx> 3/ do not force releases, and reuse stable branches from cycle to cycle
15:40:34 <ttx> 3 is conceptually the best, but I have a hard time to wrap my head around potential consequences
15:40:35 <smcginnis> 1 seems the safest to me.
15:40:48 <ttx> 1 is the safest but weirdest when there are no more changes
15:40:56 <ttx> 2 is a trade-off
15:41:29 <dhellmann> 2 is the case where we have to be careful with versions, as fungi pointed out
15:41:29 <ttx> do we agree on the 3 options ?
15:41:37 <fungi> yeah, #1 while safe deos also seem like a treadmill of process-imposed busywork
15:41:46 <dhellmann> I suppose option 4 is to not worry about stable branches at all for those things
15:41:47 <fungi> s/deos/does/
15:42:01 <ttx> switch them to independent ?
15:42:10 <dhellmann> and option 5 is to not do anything and create the stable branches when we need them
15:42:39 <dhellmann> (option 5 is like 2, except for the timing)
15:42:46 <ttx> dhellmann: you mean create all the missing branches once you need them ?
15:42:50 <dhellmann> right
15:43:04 <dhellmann> I'm not sure it's a good idea, but for completeness sake I thought I'd mention it
15:43:04 <ttx> It's a 2bis
15:43:07 <fungi> i want to say there were some drawbacks to #4 and #5 (we did them in the past at various times) but trying to recall what the problems were now
15:43:37 <ttx> is #4 like switching things to be independent ?
15:43:45 * ttx documents in etherpad
15:43:50 <fungi> certainly not creating stable branches leads to problems when you have a new release you don't want used by stable branches of other projects
15:43:56 <dhellmann> well, we have stable branches for independent projects sometimes
15:44:19 <dhellmann> fungi : with the constraints system in place, new releases can be controlled
15:44:24 <fungi> though i suppose constraints on those stable branches help
15:44:29 <fungi> yeah, that
15:44:54 <smcginnis> Yeah, we're probably in better shape now to handle these.
15:44:54 <fungi> we didn't really have constraints widely used back when we were encountering the issues which caused us to force stable branches for all libs
15:44:54 <dhellmann> in fact 3 would require extra manual work to update u-c in stable branches
15:45:06 <dhellmann> right
15:45:24 <fungi> so... 4/5 actually seem sort of okay and would certainly be less work
15:45:46 <smcginnis> Not to be lazy, but less work is probably better at this point.
15:45:52 <dhellmann> option 5 feels appealing, because it lets us do the work we need to fix something at the time it's broken without having to do extra work up front
15:46:00 <fungi> i think it would end up being #5 because there will be critical fixes you need backported to an older version used by stable-branch projects
15:46:12 <dhellmann> otoh, doing it up front means it's done as part of a batch, so we're ready when things are broken
15:46:42 <dhellmann> so I'm sort of torn between 5 and 2
15:46:43 <ttx> yeah, it's self-imposed work, but part of a routine
15:46:46 <fungi> well, release requests can come with branching requests, right? so it's not really that much extra convenience to have the branches precreated
15:46:53 <ttx> I renamed 5 to 2bis in the etherpad
15:47:02 <dhellmann> fungi : they need the branch before they can backport a fix
15:47:14 <dhellmann> ok
15:47:17 <ttx> since it's close to 2 and helps estimating disruption potential
15:47:21 <fungi> oh, good point. so it's two release requests
15:47:35 <fungi> create the branch, backport the fix, tag the release
15:48:05 <dhellmann> they're likely going to want a release from master, and the branch request could come in that patch, but at that point it's not really much extra effort to make it separately
15:48:21 <dhellmann> maybe we try 2bis and if we run into problems we switch to 2?
15:48:21 <fungi> i still feel like #5 is preferable to #2 _if_ the frequency of backports is very low
15:48:26 <ttx> We don't have to decide now, just wanted to lay down the options to give it some extra thought
15:48:39 <smcginnis> Was just going to say, we don't need to decide today.
15:48:41 <dhellmann> I'm expecting a very small number of things in this category, and for those things to be very stable.
15:48:54 <smcginnis> Let's give it a bit and let things sink in and see if we think of any other concerns.
15:48:58 <ttx> One issue is how we "mark" those
15:49:05 <dhellmann> ++ to contemplation
15:49:16 <dhellmann> ttx: yeah, maybe a new release model?
15:49:31 <fungi> it's also possible to switch from #2 to #2bis in the future if we discover that #2 isn't buying us much for the additional effort
15:49:38 <dhellmann> fungi : good point
15:49:53 <dhellmann> it seems like we can switch between any of the options, really
15:50:06 <fungi> sure
15:50:08 <smcginnis> Yeah, 2bis is really just lazy loaded 2.
15:50:17 <ttx> ok... let's it simmer a bit. Next topic ?
15:50:19 <dhellmann> ++ for being lazy
15:50:36 <smcginnis> Any aspect of extended maintenance lib releases we should consider with this?
15:50:46 <smcginnis> I thought that was actually the topic at first glance.
15:51:01 <smcginnis> Probably the same applies?
15:51:20 <ttx> brain cells level: 0
15:51:22 <dhellmann> I think we decided to change the devstack job to install everything from source by default for EM branches, didn't we?
15:51:47 <smcginnis> Was that the decision in the session. I couldn't remember if we came to a decision.
15:52:01 <ttx> that was the consensus there yes
15:52:07 <dhellmann> I felt like it was the preferred option, although I'm not sure if someone stepped up to write that change.
15:52:20 <dhellmann> it may have been another case of dealing with it when it actually comes up
15:52:34 <smcginnis> OK, that works then. We can move on.
15:52:45 <smcginnis> #topic Planned time off
15:53:11 <ttx> I realized my summer vacation is on FF week, again
15:53:20 <smcginnis> I know I will be gone R-7, but that should be a quiet week.
15:53:21 <ttx> also in Europython week
15:53:30 <smcginnis> I shouldn't have any conflicts on FF week  (R-5).
15:53:48 <dhellmann> I'll be traipsing all over scotland that week
15:53:51 <ttx> I refreshed my travel schedule for the rest of the cycle
15:53:56 <smcginnis> dhellmann: Nice!
15:54:00 <d0ugal> EuroPython \o/
15:54:16 <apetrich> \o/
15:54:18 <dhellmann> well,not so much "all over" as "Edinburgh and Glasgow"
15:54:49 <smcginnis> So I guess I'll be holding down the fort that week with whatever help I can get.
15:55:08 <smcginnis> And will be definitely leaning on fungi if any release/infra issues come up. :)
15:55:48 <fungi> i'm gone for the second half of july... trying to see what that works out to on the release schedule
15:55:50 <smcginnis> This cycle has been much more reliable with those things, so hopefully this is no issue.
15:55:53 * smcginnis knocks on wood
15:56:11 <ttx> it's a good test at least
15:56:22 <smcginnis> OK, then I can just cry for help in -infra if something happens. :D
15:56:32 <smcginnis> But not feeling too worried.
15:56:36 <fungi> yeah, i disappear in the middle of r5 and return in the middle of r3
15:56:40 <dhellmann> on conference days (the end of the week) I'll check in on irc periodically
15:56:49 <fungi> er, i mean r6 and r4
15:57:12 <smcginnis> dhellmann: That would be great. Don't want to distract you, but might be good if any questions come up.
15:57:33 <dhellmann> I'm on twitter and signal, too
15:57:51 <fungi> so yeah, my travel is july 18-31 according to my notes, though i'll have internet access during some of that
15:57:53 <smcginnis> I think I can hunt most of you down if I really need to.
15:58:13 * dhellmann turns on his cloaking device
15:58:19 <smcginnis> Hah ;)
15:58:29 <smcginnis> Anything else in the last minute+?
15:59:11 <dhellmann> nothing from me
15:59:13 <smcginnis> We need to talk succession at some point, but we have time yet. But time today.
15:59:23 <smcginnis> But not time today I mean.
15:59:34 <smcginnis> OK, thanks everyone!
15:59:45 <smcginnis> #endmeeting