14:04:37 <dprince> #startmeeting tripleo
14:04:39 <d0ugal> derekh++
14:04:39 <openstack> Meeting started Tue Mar  1 14:04:37 2016 UTC and is due to finish in 60 minutes.  The chair is dprince. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:04:40 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:04:43 <openstack> The meeting name has been set to 'tripleo'
14:04:44 <derekh> \o/
14:04:54 <dprince> hello :)
14:04:55 <akrivoka> hiya \o
14:05:02 <shardy> hi!
14:05:02 <rbrady> :)
14:05:02 <gfidente> derekh++
14:05:06 <gfidente> I am speechless
14:05:20 <adarazs> l33t hack!
14:05:23 <derekh> ;-) say nothing
14:05:33 <leanderthal> o/
14:05:43 <d0ugal> o/
14:05:56 <trown> o/
14:06:15 <hrybacki> o/
14:06:32 <leanderthal> derekh++
14:07:04 <dprince> #topic agenda
14:07:04 <dprince> * bugs
14:07:04 <dprince> * Projects releases or stable backports
14:07:04 <dprince> * CI
14:07:04 <dprince> * Specs
14:07:06 <dprince> * one off agenda items
14:07:09 <dprince> * open discussion
14:07:23 <dprince> there on no on off agenda items this week I think
14:07:41 <dprince> anything else to add to the agenda before we start?
14:07:51 <shardy> dprince: I'd like to discuss when we branch for mitaka RC, but that can be in the releases/stable section
14:08:56 <dprince> shardy: cool
14:09:00 <dprince> lets go
14:09:04 <dprince> #topic bugs
14:09:40 <saneax> Hello \o
14:09:41 <dprince> There were a couple IPv4 network isolation bugs fixed this week
14:10:24 <dprince> #link https://bugs.launchpad.net/tripleo/+bug/1551048
14:10:24 <openstack> Launchpad bug 1551048 in tripleo "network validation tests fail: No module named ipaddr" [Critical,Fix released] - Assigned to Dan Prince (dan-prince)
14:10:29 <dprince> #link https://bugs.launchpad.net/tripleo/+bug/1550380
14:10:29 <openstack> Launchpad bug 1550380 in tripleo "network isolation broken when SoftwareConfigTransport == POLL_TEMP_URL" [Critical,Fix released] - Assigned to Dan Prince (dan-prince)
14:10:58 <dprince> both of those should be fixed. I was surprised to see that IPv4 network isolation has probably been broken for some time
14:11:25 <marios> o/ joining late cos other call
14:11:27 <dprince> locally IPv4 netiso is working for me again
14:11:41 * bandini waves
14:11:48 <shardy> dprince: the second one looks worrying from an upgrade perspective, e.g 20-os-net-config still isn't owned by any package is it?
14:12:06 <shardy> so if (as I think stevebaker wanted) we switch to tempurl on upgrade, things may break?
14:12:19 <dprince> shardy: I don't think it effects upgrades unless you also switch to Heat TEMP URL's
14:12:33 <dprince> shardy: and then you could push a new 20-os-net-config with the artifacts patch
14:13:18 <shardy> dprince: Ok, just something to bear in mind, looks like the patch switching transports for liberty stalled:
14:13:21 <shardy> https://review.openstack.org/#/c/257657/
14:13:22 <dprince> shardy: basically you'd create a tarball and use t-h-t to extract that tarball
14:14:06 <shardy> dprince: ack, I guess provided we document it in the upgrade/release notes for mitaka all should be OK then
14:14:22 <ndipanov> derekh, ugh sorry guys my bad
14:14:37 <dprince> on a related note I'd really like to see us get any sort of IPv4 network isolation CI job in place ASAP https://review.openstack.org/#/c/273424/
14:14:48 <derekh> ndipanov: no prob, we got there in the end
14:14:55 <shardy> switching away from 20-os-net-config ref https://review.openstack.org/#/c/271450/, or at least getting it packaged seems like a better plan long term to me tho
14:15:33 <dprince> shardy: yep
14:15:50 <dprince> okay, any other bugs to talk about this week?
14:16:16 <dprince> the puppet-keystone fix blocking master has landed (keystone-manage bootstrap, etc.)
14:16:52 <dprince> derekh mentioned to me earlier that we actually had 2 of the master periodic jobs pass for the first time I think last night
14:17:04 <shardy> \o/
14:17:15 <derekh> dprince: yup, http://tripleo.org/cistatus-periodic.html
14:17:21 <slagle> dprince: it did, but we had to also patch tht to not use it
14:17:27 <slagle> or, EmilienM did rather
14:17:45 <slagle> dprince: https://bugs.launchpad.net/tripleo/+bug/1551501
14:17:45 <openstack> Launchpad bug 1551501 in tripleo "CI: HA jobs failing with Error: /Stage[main]/Keystone/Exec[keystone-manage bootstrap]: Failed to call refresh: keystone-manage bootstrap --bootstrap-password <password> returned 1 instead of one of [0]" [Critical,Fix released] - Assigned to Emilien Macchi (emilienm)
14:17:47 <dprince> right, and now that https://bugs.launchpad.net/tripleo/+bug/1551501 is fixed the HA job should pass again.
14:18:03 <EmilienM> but you're going to have boostrap issues when you'll update keystone
14:18:40 <dprince> EmilienM: I see. SO the master job will still be failing
14:18:55 <dprince> EmilienM: for the HA job we just need to run the bootstrap manually then?
14:19:05 <dprince> EmilienM: or later in the steps?
14:19:31 <EmilienM> dprince: I don't know, we need testing
14:19:58 <dprince> okay, so it sounds like the HA job for master is still broken  to me. I guess we'll see what happens
14:20:34 <dprince> #topic CI
14:21:04 <dprince> It is my understanding we are hitting memory issues with a couple new patches
14:21:40 <derekh> dprince: yup, the patch for aodh and gnocchi
14:21:43 <dprince> slagle, derekh: and we think that using a bit of swap may help get things passing? until we can get more memory added to our nodes
14:21:48 <shardy> We also need to decide if we're going to disable the containers job ref https://review.openstack.org/#/c/285325/
14:22:16 <slagle> dprince: it's a theory anyway
14:22:30 <derekh> dprince: we can bump it a little, but would like to find out what exactly needs the RAM first, also if swap works then great ;-)
14:22:36 <slagle> dprince: we'd need to get the swap patch fixed up, base the aodh/gnocchi patches on that, and then see how performance is affected
14:23:02 <derekh> yup, what slagle said
14:23:09 <dprince> shardy: +A for disabling contains (for now)
14:23:27 <dprince> slagle: agree, lets go with this plan
14:23:51 <derekh> I've been mostly concentrating on CIv3.0 stuff, but at a quick glance, things seem to be ticking along at the moment, although the HA job seems to be failing too much...
14:23:55 <slagle> ok, will push up another revision to the swap patch
14:24:24 <slagle> i'm thinking i will leave it not used by default, since it's not really a production configuration, but then we can create a custom env in CI to use it
14:24:39 <rhallisey> dprince, oh by the way
14:24:44 <rhallisey> I almost have the gate fixed
14:25:09 <dprince> rhallisey: nice, a simple revert gets contains back in the mix...
14:25:19 <rhallisey> yup just letting you know
14:25:47 <derekh> next I want to get the periodic job to push artifacts to a mirror-server, would be good if somebody could look over the patch series to enable the mirror server https://review.openstack.org/#/q/status:open+project:openstack-infra/tripleo-ci+branch:master+topic:mirror-server
14:26:00 <derekh> hit a merge conflict earlier today /me will ifx
14:26:24 <dprince> derekh: nice
14:26:53 <dprince> I will try and review it
14:27:09 <derekh> dprince: thanks
14:27:14 <dprince> #topic Projects releases or stable backports
14:27:30 <dprince> shardy: you wanted to mention Mitaka branches?
14:28:24 <slagle> #info slagle plans to do stable liberty/releases later today
14:28:51 <dprince> slagle: ack, thanks
14:30:01 <shardy> dprince: yup, it's nearly RC time for other projects:
14:30:01 <shardy> #link http://releases.openstack.org/mitaka/schedule.html
14:30:01 <shardy> and although we've not done milestone releases, I'd like to propose we do an RC release, which will also enable us to branch for N in ~2weeks
14:30:01 <shardy> which may also unblock some features expected to progress in early N
14:30:01 <shardy> #link https://github.com/openstack/releases
14:30:01 <shardy> what do folks think?
14:30:02 <shardy> e.g can we land all the remaining features in the next 7-10 days then declare a RC1?
14:31:06 <trown> what are the remaining features?
14:31:37 <jistr> the main ones are probably IPv6, SSL, and upgrades?
14:31:54 <slagle> and satellite 5 support, although that is only one patch
14:31:58 <marios> jistr: any outstanding integrations?
14:32:00 <bnemec> SSL is already in master.
14:32:02 <michchap> I'd like to get opendaylight merged if possible
14:32:11 <derekh> In order to be able to test it, we'll need to coordinate with #rdo so we have a repository to use
14:32:25 <marios> jistr: (sorry wasn't really a question, i meant 'these too')
14:32:30 <dprince> shardy: given that we don't actually have CI to cover any of those (yet) I'm questioning if we can do this in 2 weeks
14:32:39 <jistr> marios: yeah. there are some patches from bigswitch to os-net-config pending, and stable/liberty backport of opencontrail i think
14:32:47 <bnemec> The other problem with this timing is that we're going to be simultaneously pushing backports for the downstream release that is due at about the same time.
14:32:47 <jistr> marios: might be more than that
14:32:59 <marios> shardy: how stable the ci is over those days is crucial
14:33:48 <bnemec> Hmm, lost shardy.
14:34:28 <derekh> should I change my nick again?
14:34:44 <dprince> derekh: please do so we can move on
14:34:54 <derekh> ;-)
14:35:01 <dprince> no, maybe he'll re-join in a minute
14:35:16 <dprince> sounds like there is concern 7-10 days is to soon for a Mitaka branch though
14:35:30 <trown> what is the alternative though?
14:35:47 <shardy> gah, poorly timed network outage :(
14:36:13 <dprince> trown: alternative would be to actually create CI jobs (upstream) which test these features before we claim them as features on a stable Mitaka branch
14:36:15 <trown> if we do not put out a RC with the rest of openstack, then it makes it pretty impossible to release tripleo in RDO with the rest of openstack
14:36:19 <derekh> shardy: http://paste.openstack.org/show/488759/
14:36:42 <shardy> http://paste.openstack.org/show/488760/ is what I wrote fwiw
14:36:46 <dprince> trown: or we cut a release and don't claim any of the new features we aren't testing
14:36:50 <shardy> derekh: thanks :)
14:36:56 <trown> dprince: I would be more into that
14:37:09 <trown> the release schedule is not arbitrary
14:37:58 <slagle> trown: do you know when the rdo mitaka repos will be up?
14:38:05 <slagle> will that happen at rc1 as well?
14:38:31 <dprince> shardy: I like the motivation for this in that it opens up master again for architectural work
14:38:42 <trown> slagle: that is when we could start working on it, but RDO aims to release within 1 week of upstream
14:38:45 <dprince> shardy: i.e. composable service roles
14:38:49 <dprince> and split stack
14:39:00 <shardy> dprince: yup, and it also makes it much better for those consuming tripleo IMO, as trown has indicated
14:39:35 <dprince> shardy: my concern is that I don't want to see everyone slam down a bunch of code that probably isn't working (upstream anyways)
14:40:02 <dprince> shardy: so if the plan is to cut the branch without the code-slamming then I'm probably fine
14:40:18 <shardy> it probably implies a big push on reviews over the next week tho I guess
14:40:26 <shardy> re the CI thing, we can announce features as experimental if they don't have CI coverage?
14:41:04 <dprince> shardy: yes, but my concern w/ things like IPv6 network isolation is that by landing those patches we'll just break IPv4 (again)
14:41:20 <dprince> shardy: because we still don't have the CI job on that...
14:42:08 <bnemec> I guess I'm fine with cutting the Mitaka branches whenever, but I'll note again that over the next couple of weeks my focus is likely to be on downstream OSP 8.
14:42:10 <adarazs> I think gfidente is working on ipv4 netiso, that should happen in the near future I think.
14:42:26 <shardy> dprince: yup, I guess the question is just is it feasable to get to a reasonable branch point in ~2 weeks
14:42:26 <shardy> I know we discussed having a lagging release on the ML, but I personally am not in favor of that
14:42:26 <shardy> I think it'll be much clearer if we just start adhering to the exact same process/schedule as the rest of OpenStack
14:42:30 <trown> dprince: but how much time will it take to get that stuff... is an extra week likely to make any difference?
14:42:52 <trown> I am -1 on lagging release
14:43:00 <gfidente> yeah short update on that, we're in a sort of debugging session and trying to figure what is wrong
14:43:26 <slagle> i'd like to see the needed features at least landed in master before cutting mitaka
14:43:27 <adarazs> gfidente: let me know if I can help.
14:43:34 <bnemec> gfidente: You saw my comment on the latest patch set?
14:43:37 <dprince> okay, so if not laggin the release is the priority then fine. I just don't like slamming in features that haven't been tested on upstream
14:43:37 <jistr> slagle: +1
14:43:39 <slagle> if they're not, we'd have to violate our new policy of not backporting features
14:43:57 <bnemec> Can we come up with a list of must-haves for Mitaka?
14:43:58 <shardy> bnemec: Yeah, I guess that is the subtext to my question, e.g if we need a big push on reviews, we need folks with bandwith to do them, and probably a list of the backlog we expect to land for stable/mitaka
14:44:10 <dprince> it appears to me that many of the patches are being testing against stable/liberty (OSP8) and then thrown upstream... where they may or may not work
14:44:11 <shardy> if we're not going to allow feature backports, getting that right is a lot more important
14:44:22 <dprince> this isn't the way it was supposed to work
14:44:25 <gfidente> bnemec, I dropped those yes, the overcloud external network is a flat network bridged into br-ex with 192.0.2 subnetting
14:44:48 <gfidente> bnemec, this is why pingtest works without netiso
14:45:20 <gfidente> adarazs, ack, pvt
14:45:24 <shardy> dprince: agreed, but getting this release cut at the right time is IMHO the first step to fixing our process
14:45:36 <shardy> then the deadlines for things landing upstream become very clear from now onwards
14:45:40 <bnemec> gfidente: Umm, no it's not.  We should talk after the meeting.
14:45:53 <dprince> what I'm suggesting is it would be fine to land a feature upstream. Say IPv6, so long as whoever is reviewing it is manually verifying that it actually works and doesn't break previous features for each patch. Because we simply don't have enough CI coverage on much of this to verify we aren't breaking things
14:46:55 <dprince> shardy: I'm fine with that release angle, just saying that there seems to be disagreement about landing or not landing all these features in 2 weeks time
14:46:58 <shardy> dprince: requiring some manual testing is probably fine, provided folks actually have time to do that as well as the reviews (bnemec is giving me the feeling some folks don't have bandwidth for either, which is clearly problematic)
14:47:07 <jistr> i think we do have the IPv6 + SSL + upgrades support very close though. If we can land those while making sure we don't break anything else, i'd be even ok with landing the majority of those features even without knowing that the new features work 100% great (experimental features, as shardy said), and then we can fix the new features later. Main concern is breaking what we already had but we don't have CI coverage on (e.g. IPv4
14:47:07 <jistr> net iso).
14:47:37 <trown> it is problematic to work on the previous release for a whole cycle and then try to land all the features in the last 2 weeks
14:47:43 <jistr> yea :(
14:47:57 <shardy> lol @ "fix the new features later"
14:48:07 <bnemec> This is why I'd be in favor of lagging a bit this cycle and then cutting N in sync with everyone else.
14:48:12 <jistr> experimental feature == will fix later
14:48:13 <bnemec> N'Sync even. ;-)
14:48:14 <jistr> :))
14:48:38 <trown> if we lag release, tripleo will not be able to participate in RDO Mitaka release
14:48:53 <dprince> jistr: sure, I agree on the experimental features
14:48:59 <trown> which means even more packstack users
14:49:15 <shardy> and if we lag, we'll keep all N work blocked, which will eat into the time for N feature development
14:50:10 <shardy> for me, not being able to participate in the RDO mitaka release is a deal breaker, and it means we have to branch at the right time
14:50:17 <dprince> jistr: but network isolation IPv4 has been broken for weeks upstream so we clearly need to do a better job of having reviewers (manually) test upstream until some of these CI jobs are in place.
14:50:18 <bnemec> I mean, that's fine, but if the features aren't landed then we're screwed.
14:50:19 <trown> same
14:50:33 <bnemec> We either violate the new backport policy, or we carry downstream patches for stuff.
14:50:42 <dprince> getting some of these automated in CI is really the only scalable solution
14:50:46 <bnemec> Neither of which are good either.
14:51:08 <trown> I would prefer downstream patches over twisting the upstream process for downstream needs
14:51:11 <jistr> dprince: +1 on prioritizing automated CI
14:51:28 <trown> that is a really non-starter argument for any other openstack project
14:51:45 <bnemec> Can we take this discussion offline and come up with the list of features that must go into Mitaka, and make a decision from there?
14:51:53 <bnemec> trown: It's also reality in this project right now.
14:52:04 <bnemec> Most of the developers have downstream responsibility, whether we like it or not.
14:52:19 <shardy> trown: I agree, IMHO we can't block the entire upstream release due to downstream requirements, worst case some patches will have to be carried, but it'd be far better if we could have a coordinated push and just test/land what remains
14:52:29 <trown> ya, but it is still possible to seperate downstream arguments from upstream process and needs
14:52:29 <slagle> downstream patches are also a non starter
14:52:42 <slagle> it's not even a worst case scenario
14:52:45 <slagle> it's no scenario
14:52:55 <bnemec> Downstream patches are what got us into this situation in the first place, FWIW.
14:53:01 <trown> ok, so if tripleo is now a downstream project, I do not have much to add
14:53:12 <bnemec> It's why we have features in Kilo that don't exist in Mitaka yet.
14:53:18 <shardy> working on the previous release when we should have branched is also part of our current problem tho
14:53:53 <bnemec> Totally agree, but until we get everything to the same level feature-wise, we're going to be working on the previous release.
14:53:55 <trown> imagine trying to make this argument in nova
14:53:59 <jistr> bnemec: +1
14:54:03 <shardy> Ok, looks like we don't have consensus - how about this:
14:54:06 <bnemec> trown: We are not Nova.
14:54:18 <shardy> 1. We create an etherpad with blocker for mitaka patches, like today
14:54:21 <slagle> my opinion is that if we want to branch on time, we have to allow for some *possible* feature backports to stable/mitaka *if* they have not all landed
14:54:29 <shardy> 2. revisit next week, and make a call re the branch for RC1 date
14:54:42 <bnemec> shardy: +1
14:55:01 <jistr> +1
14:55:04 <slagle> wfm
14:55:05 <dprince> sounds like a fine plan to me shardy
14:55:07 <shardy> slagle: Ok, I guess we can discuss that if it becomes clear certain patches won't land
14:55:37 <shardy> #link https://etherpad.openstack.org/p/tripleo-mitaka-rc-blockers
14:55:43 <slagle> yea, i just dont want to shoot ourselves in the foot
14:55:57 <slagle> i think we can pull this off, branch on time, avoid downstream patches
14:56:04 <shardy> Please add patches there, and we can review/discuss in #tripleo to ensure they are actually blockers
14:56:05 <slagle> we just might have to allow ourselves some initial leeway
14:56:32 <dprince> okay, very little time left in the meeting
14:56:40 <dprince> #topic open discussion
14:56:55 <dprince> anything else someone needs to bring up?
14:57:14 <michchap> I just came along to mention I put up a spec for moving the puppet manifests into a module and reducing the code duplication
14:57:59 <michchap> I wanted to check I'm not missing some really obvious reason not to do that
14:59:09 <dprince> michchap: we already planned to do this as part of composable services
14:59:22 <dprince> michchap: so ++ for the idea
14:59:28 <michchap> dprince: is there an implementation or should I put one up?
14:59:46 <dprince> michchap: lets talk on #tripleo afterwards
15:00:09 <michchap> dprince: sure
15:00:13 <dprince> thanks everyone
15:00:17 <dprince> #endmeeting