15:01:25 <mriedem> #startmeeting stable
15:01:26 <openstack> Meeting started Tue Jan 12 15:01:25 2016 UTC and is due to finish in 60 minutes.  The chair is mriedem. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:27 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:29 <ttx> o/
15:01:29 <openstack> The meeting name has been set to 'stable'
15:01:42 <mriedem> hi, who is all here (besides ttx)?
15:01:50 <ihrachys> o/
15:02:04 <mriedem> mtreinish: tonyb?
15:02:13 * ihrachys attends two meetings, can lag
15:02:28 <mriedem> i'll give it another minute or so
15:03:00 <anteaya> I can lurk if that is helpful to numbers
15:03:17 <mriedem> it does
15:03:20 <anteaya> yay
15:03:25 <mriedem> well let's just get started
15:03:26 * anteaya is useful today
15:03:33 <mriedem> #link agenda https://wiki.openstack.org/wiki/Meetings/StableTeam
15:03:40 <mriedem> #topic status
15:03:52 <mriedem> periodic-stable failures http://goo.gl/5qiw2U
15:03:59 <mriedem> there are really just 2 issues with those
15:04:15 <mriedem> 1. neutron-*aas jobs aren't getting the branch set in the periodic-stable jobs properly
15:04:32 <mriedem> ihrachys: proposed https://review.openstack.org/#/c/245525/ and tonyb is picking it up
15:04:59 <ihrachys> tonyb++
15:05:01 <mriedem> 2. ceilomer dsvm job is having issues setting the branch for gnocchi https://review.openstack.org/#/c/264769/
15:05:12 <mriedem> gordc is playing d-g whack a mole with that one
15:05:32 <mriedem> basically, d-g is trying to clone stable/liberty gnocchi which doesn't exist, so he has to override the branch that d-g pulls for that job
15:05:39 <mriedem> i'm hoping sdague and mtreinish can review that one
15:05:58 <mriedem> that's it for the periodic-stable jobs
15:06:00 <ttx> the dashboard shows horizon issues as well ?
15:06:12 <mriedem> the openstack-health one?
15:06:26 <mriedem> oh, that was last week
15:06:27 <ttx> no the link you pointed us to
15:06:30 <ttx> oh ok
15:06:36 <mriedem> yeah, last week was the kilo + liberty fubar
15:06:43 <mriedem> where horizon wouldn't install in dsvm on kilo
15:06:50 <mriedem> which broke all of kilo dsvm and liberty grenade jobs
15:06:55 <mriedem> that was fixed last week
15:07:31 <mriedem> i should probably adjust that logstash link to 2 days rather than 7 maybe
15:07:37 <sdague> mriedem: I just approved it, I do think that the gnocchi story here tells a more important release concept that mixing independent release components with integrated release components just makes for pain
15:07:55 <anteaya> sing it
15:08:03 <mriedem> well
15:08:22 <mriedem> big tent everyone releases on their own schedule, so theoretically this should be fine right?
15:08:33 <sdague> as long as they don't want to have linked jobs
15:08:44 <mriedem> gnocchi is a library isn't it?
15:08:45 <sdague> it's the linked jobs that's the issue
15:08:47 <mriedem> yeah
15:09:03 <mriedem> i guess if ceilometer just pulled down a pip of gnocchi from pypi then it wouldn't be a problem
15:09:11 <ttx> sdague: maybe we should make that more obvious in the doc where we explain both choices
15:09:14 <sdague> so you can work like a library and install from pypi
15:09:19 <sdague> and that's fine
15:09:21 <ttx> like in the project team guide
15:09:21 <anteaya> I think this may be a gap where the theory of the big tent and the reality of what people with with it causes issues
15:09:24 <sdague> but co-gating is not
15:09:44 <mriedem> anyone want an action for a doc update?
15:09:49 <sdague> and even our libraries have stable/* branches
15:09:50 * mriedem is not sure where that is
15:09:51 <ttx> anteaya: the consequences of choice are not always well explained
15:09:57 <ttx> (yet)
15:10:04 <anteaya> or understood as well make up the rules
15:10:11 <sdague> so I think anything out of openstack that has branches needs a stable/liberty branch
15:10:20 <anteaya> s/well/we
15:10:29 <sdague> I think that's the thing that we missed before
15:10:55 <mriedem> sdague: or has to be installed from pypi
15:11:06 <anteaya> I don't mind taking an action item for a patch to the project team guide but it will need heavy review as I don't know what it should say
15:11:21 <anteaya> but I'll kick it off
15:11:21 <ttx> sdague: interesting... would you mind raising a thread about that once you have it formalized ? I'd like us to discuss that a bit more
15:11:45 <mriedem> i will gladly assign an action to someone here
15:11:51 <ttx> anteaya: we need to get to the bottom of what that means first
15:11:59 <anteaya> understood
15:12:09 <anteaya> I agree clarity is helpful
15:12:30 <sdague> ttx: I've got a few other hard problems on my plate right now, this is just something I've been mulling as I've watched this gnocchi review hit various brick walls
15:12:43 <mriedem> #action mriedem to talk to ppl about co-gating and what that means for big tent and projects with branches and w/o, like ceilometer + gnocchi
15:13:28 <mriedem> i also wonder how that has implications for lifeless' spec that is part of a longer term goal to remove stable branches
15:13:31 <mriedem> or i thought it was
15:13:35 <mriedem> but we can get into that later
15:13:41 <ttx> I mean, I'm open for limiting the choices there, I just want us to think a bit more about it before we announce anything
15:13:57 <sdague> yeh, that's fair. I think there are 2 modes we know are good
15:14:05 <anteaya> the choices already are limited, we just need to inform folks
15:14:15 <anteaya> people can't test anything with everything else
15:14:20 <sdague> you don't have branches, you release from master, and support all supported stable branches with your master
15:14:24 <sdague> see: tempest
15:14:31 <anteaya> neutron drivers on independant release have run into this
15:14:51 <mriedem> nothing depends on tempest though,
15:14:51 <sdague> you release with branches, and have a branch that's appropriate for the official release branches
15:14:57 <mriedem> so in this example, gnocchi is tempest
15:15:01 <ttx> sdague: sounds fair
15:15:32 <sdague> mriedem: lets say python-novaclient then
15:15:39 <mriedem> which has branches :)
15:15:45 <sdague> it does now
15:16:13 <sdague> part of the problem is we used to actually have this model, then capping forced a lot of new branches, then people thought they could do anything they wanted with branches
15:16:47 <mriedem> if ceilometer depends on gnocchi as a library, i'm not really sure why it's not just installing it via pypi
15:16:56 <mriedem> at whatever version it needs
15:17:07 <mriedem> gnocchi can have master and support all stable if it wants
15:17:15 <sdague> yeh, I don't know
15:17:16 <ttx> feels like a discussion for the release team anyway
15:17:27 <mriedem> i'll also follow up with gordc and jd i suppose
15:17:44 <apevec> mriedem, it cannot b/c of deps on master might be too new for older branches
15:18:26 <ttx> if it has too many common deps it should probably follow the cycle basically
15:18:29 <mriedem> apevec: i'm saying ceilometer wouldn't depend on master gnochi
15:18:51 <mriedem> ceilometer needs to depend on a version of gnocchi that is constrained by upper-constraints
15:18:56 <mriedem> in each stable branch of ceilometer
15:19:29 <mriedem> and if gnocchi does a backwards incompatible thing, then it's breaking backwards compat and we either have to cap it or revert that breaking change
15:19:39 <mriedem> which gets into lifeless' spec https://review.openstack.org/#/c/226157/
15:20:10 <mriedem> that would also mean that the requirements repo changes would have to be gated on ceilometer, which they aren't today, but that gets into more details than we want here
15:20:29 <mriedem> shall we move off this topic?
15:20:34 <ttx> yes
15:20:45 <mriedem> ok
15:20:49 <mriedem> ttx: Remaining stable/kilo (coordinated) point releases
15:20:56 <ttx> So... Starting with liberty we do separated, on-demand stable point releases
15:21:03 <ttx> But we still have a few coordinated point releases to do for kilo
15:21:05 <apevec> gnocchi is actually a service, ceilo depends only on gnocchiclient
15:21:11 <ttx> #link https://wiki.openstack.org/wiki/StableBranchRelease#Rinse.2C_Repeat
15:21:19 <ttx> 2015.1.3 January, 2016
15:21:23 <ttx> 2015.1.4 (eol) 2016-05-02
15:21:29 <ttx> Daviey (in a private thread with apevec and me) proposed to handle both
15:21:41 <mriedem> way to be not open :)
15:21:49 <mriedem> sheesh
15:22:08 <mriedem> i'm fine with that
15:22:16 <ttx> indeed, but since he volunteered I forgave him
15:22:24 <ttx> It would be great to go through the release process and check what still applies
15:22:29 <ttx> Also check if the needed tooling is still available, since we did remove some tools from release-tools recently
15:22:48 <ttx> So I'll contact him and check we still have all ducks in row
15:22:58 <ttx> #action ttx to go through the stable release process with Daviey to doublecheck tooling
15:23:21 <mriedem> cool
15:23:23 <apevec> mriedem, my fault, I started that offlist thread from juno EOL announcement
15:23:47 <mriedem> ttx: anything else on that item?
15:23:50 <apevec> ttx, couldn't we use part of new tooling, just for pushing tags?
15:23:52 <mriedem> kilo is pretty stable atm
15:23:52 <ttx> It's good to have a volunteer for the last 2, so that nobody else has to dive into old process
15:24:07 <ttx> apevec: that is what I want to figure out
15:24:10 <anteaya> yay kilo is ~stable
15:24:32 <ttx> mriedem: I'm done with this topic
15:24:40 <mriedem> ttx: Stable team tags: moving from has-stable-branches to follows-stable-policy ?
15:24:40 <ttx> unless there are questions
15:24:48 <ttx> OK; so... We currently have one tag to describe /some/ stable branch support:
15:24:55 <ttx> #link http://governance.openstack.org/reference/tags/release_has-stable-branches.html
15:25:07 <ttx> The problem is... this tag is a bit overloaded and doesn't really mean anything useful anymore
15:25:17 <ttx> Any project can set up stable branches now, they can even name them stable/$series
15:25:25 <ttx> they don't have to follow the policy or even add us to their $project-stable-maint teams
15:25:35 <ttx> So I'd like to propose we retire this now-useless tag and replace it by something more useful
15:25:45 <ttx> The way to think about it is in terms of answering a question that downstream users have
15:25:55 <ttx> In our domain I think what they want to know is which projects have "stable branches that they can rely on"
15:26:06 <ttx> i.e. which projects follow the stable branch policy, as defined and checked by the stable team
15:26:17 <anteaya> do we have a tag that identifies if a project interacts with the release team, in a follows their guidance kind of way?
15:26:25 <ttx> So I'd like us to propose a stable:follows-policy tag, which would be granted by the stable team:
15:26:54 <ttx> anteaya: we have release:managed, but that means somethign different. It's also non-optional to release so basically all projects need to follow
15:27:08 <anteaya> okay
15:27:11 <ttx> - as long as we are confident they are following the policy
15:27:15 <ttx> - once we are added to their stable +2 team
15:27:30 <ttx> then we would apply stable:follows-policy
15:27:34 <ttx> basically using the tag as the carrot for following the common stable policy
15:27:44 <ttx> and using tag removal as the stick to encourage outliers to better comply
15:27:56 <ttx> how does that sound ?
15:27:56 <anteaya> what is the criteria for removal?
15:28:08 <mriedem> backporting and merging things that are against the stable policy
15:28:10 <mriedem> like features or api changes
15:28:12 <anteaya> it is the removal part we seem to have the hardest part with
15:28:13 <ttx> anteaya: "project does not follow stable branch policy" ?
15:28:38 <ttx> If that sounds like a good idea I can draft a tag definition
15:28:39 <mriedem> i have no problem with removal if someone continually gets caught breaking policy (like more than once as a mistake)
15:28:54 <mriedem> it's hard to police given there are so many
15:29:02 <anteaya> mriedem: that's the issue
15:29:02 <ttx> that we would review as a team before proposing it to the tc
15:29:25 <mriedem> so really someone has to speak up and say project x backported and merged a thing which broke my production cloud - which rarely happens
15:29:31 <ttx> Just wanted a temperature reading from the team before I get started
15:29:33 <mriedem> or some projects CI would have to start failing
15:29:34 <anteaya> mriedem: right
15:29:41 <mriedem> ttx: i'm all for this
15:29:47 <ttx> NB: I would limit the tag to stable branches that track common stable series (like stable/liberty)
15:29:50 <mriedem> it's just the oversight that's hard
15:29:57 <anteaya> mriedem: yes
15:29:57 <mrunge> wow, that quite hard to decide
15:30:00 <ttx> rather than also apply it to project-specific branches (like stable/1.0)
15:30:21 <mrunge> how often did we saw some bad backports, breaking compatibility?
15:30:41 <mriedem> mrunge: that's the thing, you don't really at a global level
15:30:49 <mriedem> especially if no one speaks up in irc or the ML if it happens
15:30:53 <mrunge> But yes, I agree, we should encourage folks to look at that
15:30:56 <ttx> mrunge: some projects just don't backport bugfixes
15:31:14 <mrunge> Ideally, we would have more test coverage
15:31:16 <ttx> it's fine, they just won't have the tag
15:31:30 <ttx> since downstream users should know their stable branches are mostly worthless
15:31:35 <mriedem> i guess i wasn't thinking about whether or not they are backporting bug fixes
15:31:50 <ihrachys> mrunge: I often catch those in neutron
15:32:00 <ihrachys> I mean, I block them :)
15:32:02 <mriedem> since if you consider the number of bugs fixed in nova per master release vs how many get backported (which are applicable), it's a drop in the bucket
15:32:11 <mrunge> ihrachys, yes, that's greatly appreciated
15:32:17 <ttx> I mean, it's a significant piece of information to communicate downstream, and currently it is quite hard to determine
15:32:28 <mriedem> yeah
15:32:36 <ttx> and the current tag hides it completely
15:32:39 <mriedem> rather than bugs, it's probably just easier now to see if they are doing stable point releases
15:32:40 <ihrachys> mriedem: in neutron, we backport everything applicable to N-1
15:32:43 <ttx> One remaining question though is whether we would allow "late" stable branches to have the tag
15:32:53 <ttx> things that are release:independent because they can't make the release date, but still want to have a branch called stable/liberty
15:33:17 <ttx> like fuel
15:33:22 <anteaya> or can they make a stable/liberty branch in the middle of mitaka
15:33:28 <ttx> or some neutron stadium things
15:33:31 <anteaya> which someone wanted to do the other day
15:33:49 <mriedem> we've created stable/juno branches on things after the fact, when we needed to backport something
15:33:51 <sdague> why is it an issue to post date releases?
15:33:55 <ttx> my gut feeling is  that those should be outside policy but meh
15:34:00 <mriedem> chef cookbooks cut stable branches long after master is released too
15:34:15 <anteaya> cut or create?
15:34:28 <mriedem> they cut their mitaka release a few months after mitaka is released in the code projects
15:34:29 <anteaya> or does the difference matter?
15:34:30 <ttx> sdague: we are basically communicating: "those projects have sane stable branches"
15:34:36 <mriedem> b/c the cookbooks are stabilizing the release
15:34:42 <ttx> I think part of being sane is to have them created around the same time
15:34:44 <sdague> yeh, I think we need to realize that stable/liberty is a convenience concept to users of stuff that is going to all work in liberty
15:34:49 <ttx> so that you can follow through them as a user
15:34:56 <anteaya> mriedem: that makes sense due to their work
15:35:12 <anteaya> ttx: I agree, created dates need to be in the release in question
15:35:13 <ttx> but as I said I don't feel strongly about this
15:35:14 <mriedem> i agree with sdague, the branch is there to indicate it works with the liberty releaes of the code
15:35:18 <sdague> ttx: I guess, all the orgs I know that are deploying from stable don't do so until + 4 months or so
15:35:40 <mriedem> as long as they are following the stable branch policy, i don't really care when they create the branch
15:35:43 <ttx> sdague: ok, I can leave that out of the definition
15:35:55 <mriedem> there are some projects that i think still have active stable/icehouse branches, fwiw :)
15:36:01 <ttx> I think I have enough to start drafting
15:36:02 <anteaya> but it may affect their ability to test that branch
15:36:11 <mriedem> anteaya: sure, like stable/icehouse
15:36:15 <anteaya> right
15:36:20 <mriedem> but yeah, let's review in the review when that's up
15:36:23 <anteaya> I think that needs to be identified
15:36:31 <mriedem> moving on
15:36:33 <sdague> it might be worth tagging projects that get a stable branch out at release date, those that hit it within 2 months, and those which show up later
15:36:37 <mriedem> ttx: Liberty stable point releases status: do we need to push for some releases ?
15:36:43 <ttx> #action ttx to draft a stable:follows-policy tag definition for the team to discuss
15:36:48 <ttx> So for liberty we said that projects would just request stable point releases themselves
15:36:55 <ttx> Most did, but some may need some encouragement/reminder
15:37:02 <ttx> I think regular releases are important to make our stable branches look better
15:37:03 <mrunge> yes!
15:37:09 <ttx> so I think it's still the stable team responsibility to track that releases are regularly made
15:37:21 <mriedem> yes, i've been pushing on the nova one
15:37:23 <ttx> For liberty the following projects did not make a release since liberty release:
15:37:27 <mriedem> we have a security fix that has to get in first
15:37:31 <ttx> barbican, congress, designate, heat, horizon, mistral, nova, sahara, trove, zaqar
15:37:39 <ttx> swift and aodh haven't released either but since they are doing intermediary releases that is not as needed imho
15:37:49 <ttx> Some of them just didn't have that much stable activity, like barbican, congress, trove or zaqar
15:37:59 <ttx> (might be worth checking if they really are following stable branch policy and backporting "enough" changes, but that is another topic)
15:38:10 <ttx> Some of them have 34+ changes and should really be doing a point release: Heat, Mistral, Nova, Sahara
15:38:19 <ttx> Some have 17+ and should consider one: Designate, Horizon
15:38:33 <ttx> So in summary I think we should regularly review this and have someone assigned to the nagging
15:38:41 <ttx> I'm happy to take the first round
15:38:45 <mrunge> horizon has probably more, including i18n updates, which are quite big
15:39:13 <mriedem> ttx: thanks for pulling the numbers, i agree we have to stay on top of it, i've been busy trying to get the nova one ready
15:39:23 <ttx> probably some automated way of figuring out how much they need a release would be nice too
15:39:25 <mriedem> so i'd appreciate it if you want to own that first round of nagging
15:39:43 <ttx> Yeah I'll own it, figure out what metrics we should track there
15:39:47 <mriedem> well i think we know after you've backported security fixes, you should release
15:40:03 <ttx> then we can review that regularly, say once every ~ 6 weeks or so
15:40:08 <mriedem> in the case of nova, we really didn't start talking about it until some regression fixes were backported
15:40:33 <mriedem> from my pov, it's going to be hard to just look at what's been backported to e.g. zaqar and know if they need something
15:40:44 <mriedem> there could be 20 piddly backports or 2 critical backports
15:40:47 <ttx> yeah that should not prevent releasing more aggressively
15:41:08 <mriedem> sure
15:41:22 <mriedem> we could probably have some tooling that scrapes the bug priority too
15:41:27 <ttx> it's just that before liberty we had the safety net of the catch-all coordinated release
15:41:30 <mriedem> and flag if there are critical bugs backported that aren't yet released
15:41:33 <ihrachys> ttx: I think oslo team should have some scripts they use to track unreleased changes
15:41:53 <ttx> yeah, I was thinking going from that
15:42:01 <ttx> anyway another action for me
15:42:12 <dims> ./list_unreleased_changes.sh in release-tools repo
15:42:20 <ttx> #action ttx to do the first stable release nagging round
15:42:20 <apevec> should CVE fix trigger point release?
15:42:26 <ttx> apevec: yes
15:42:26 <mriedem> apevec: yeah
15:42:28 <apevec> Nova had one right?
15:42:38 <mriedem> apevec: yes, the backports are being reviewed and then we'll do 12.0.1
15:42:45 <ttx> (for stable/liberty only)
15:43:01 <ttx> alright that is all I had
15:43:28 <mriedem> ttx: to get the list that haven't released yet, where did you come up with that? the releases repo?
15:43:54 <apevec> ttx, there was a thread from zigo about stable horizon supporint latest django ?
15:44:03 <ttx> mriedem: and some hand count on git logs
15:44:15 <zigo> apevec: Hi!
15:44:35 <apevec> zigo, hi - why do you want django 1.9 ?
15:44:38 <apevec> 1.8 is their LTS
15:44:40 <zigo> I've wrote lots of patches for Django 1.9 in Horizon already.
15:44:41 <ttx> apevec: not related to my topic, but yes
15:44:43 <mriedem> apevec: we had to cap django_compressor in stable/kilo horizon last week b/c the latest release drops support for django<1.8
15:44:48 <zigo> apevec: Debian Sid already has Django 1.9.
15:44:56 <zigo> So there's an RC bug filled.
15:44:59 <zigo> I need to have it fixed.
15:45:23 <apevec> mrunge said 1.8 is better  choice
15:45:25 <zigo> Also, Horizon in Mitaka will need to support Django 1.9, as most likely, it's going to be the version in the next LTS.
15:45:28 <apevec> 1.9 will be out of support soon
15:45:43 <zigo> Better or not, we need to move forward and support the latest version.
15:45:52 <mrunge> django-1.9 won't be the next LTS version
15:45:57 <mrunge> it's 1.8
15:46:45 <ttx> feels like debian bet on the wrong horse then
15:47:02 <zigo> Well, anyway, I need to fix https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=809575
15:47:03 <openstack> Debian bug 809575 in src:horizon "horizon: FTBFS: ImportError: cannot import name importlib" [Serious,Open]
15:47:19 <zigo> ttx: Debian Sid is always on the edge.
15:47:37 <ttx> zigo: someone will have some fun backporting 1.8 sec fixes to 1.9 in 2017
15:47:47 <zigo> Also, remember that the packages for Django are maintained in Debian, and just sync to Ubuntu.
15:47:59 <mrunge> zigo has a point, as we don't even support Django-1.9 right now in mitaka
15:48:20 <ttx> but yeah, I think that's orthogonal to the issue at hand
15:48:26 <zigo> mrunge: I really hope that my patches will be accepted when I send them (of course, without destroying 1.8 compat).
15:48:37 <zigo> As much as I know, no problem with that so far.
15:48:59 <mrunge> zigo, I would prefer, if you'd file bugs in launchpad about that
15:49:00 <zigo> I did couples of django.getversion() calls when needed.
15:49:00 <ttx> we'll need to support >=1.8 anyway
15:49:22 <mrunge> I would like see fixes backported to liberty as well
15:49:36 <mrunge> just making horizon work in more environments
15:49:38 <zigo> As I wrote to you in #openstack-horizon, I prefer to finish the work first, and make sure nothing is broken, before attempting to upstream my work.
15:49:51 <zigo> Timur S. wrote to me he will try to help tomorrow as well.
15:50:07 <mrunge> I was just asking to document issues, just to be able to track them
15:50:25 <zigo> Well, the issue is that when you fix one issue, it uncovers another one.
15:50:32 <mrunge> and I'm really glad you're trying to fix those issues
15:50:39 <zigo> So it's impossible to document all until all is kind of fixed.
15:50:49 <mrunge> yes, sure. but still we can track it then
15:51:04 <mrunge> we should have a tracker for django-1.9 support anyways
15:51:10 <mrunge> makes sense?
15:51:14 <mriedem> can we move this to -horizon?
15:51:19 <zigo> I'll do things properly and file bugs.
15:51:25 <mrunge> we already had the same discussion
15:51:26 <zigo> It's just too early for that right now, IMO.
15:51:34 <mrunge> and agreed to disagree
15:51:38 <mrunge> ;-)
15:51:40 <zigo> :)
15:51:47 <mriedem> #topic open discussion
15:51:53 <mriedem> there was nothing on the agenda
15:51:58 <mriedem> does anyone have anything else for today?
15:52:16 <mriedem> ok, well let's end the meeting, thanks everyone
15:52:19 <mriedem> #endmeeting