14:00:29 <shardy> #startmeeting tripleo
14:00:32 <dtantsur> o/
14:00:34 <d0ugal> Hey!
14:00:35 <openstack> Meeting started Tue Nov 15 14:00:29 2016 UTC and is due to finish in 60 minutes.  The chair is shardy. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:37 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:37 <sshnaidm> o/
14:00:39 <slagle> hi
14:00:40 <openstack> The meeting name has been set to 'tripleo'
14:00:41 <shardy> Hi all, who's around?
14:00:42 <adarazs> o/
14:00:43 <ccamacho> hi folks! o/
14:00:44 <jpich> o/
14:00:46 <marios> \o
14:00:51 <beagles> o/
14:00:56 <owalsh> o/
14:01:06 <arxcruz> o/
14:01:15 <cdearborn> o/
14:01:28 <shardy> #link https://wiki.openstack.org/wiki/Meetings/TripleO
14:01:51 <shardy> #link https://etherpad.openstack.org/p/tripleo-meeting-items
14:02:00 <shardy> Please add any one-off items there ^^
14:02:03 <shadower> hey
14:02:16 <matbu> o/
14:02:19 <shardy> Ok, hi all, lets get started! :)
14:02:23 <shardy> #topic agenda
14:02:23 <shardy> * one off agenda items
14:02:23 <shardy> * bugs
14:02:23 <shardy> * Projects releases or stable backports
14:02:23 <shardy> * CI
14:02:25 <weshay> o/
14:02:25 <shardy> * Specs
14:02:28 <shardy> * open discussion
14:02:31 <trown> o/
14:02:46 <mwhahaha> o/
14:02:53 <panda> o/
14:03:10 <pradk> o/
14:04:02 <shardy> #topic one off agenda items
14:04:18 <shardy> Ok so we have a few this week, panda you want to go first?
14:04:38 <panda> shardy: sure, my items really comes down to a single question
14:05:33 <panda> where do we want to test IPv6 ? there are two proposals, using updates job -OR- moving ll HA jobs IPv6, move network isolation testing for IPv4 on non-ha job, and test non-netiso environment on multinode.
14:06:56 <panda> moving all seems complicated, but  don't know if testing only updates on master is enough coverage
14:07:37 <shardy> This is kind of related to a recent question from bandini ref can we just always enable pacemaker, to reduce the differences between the nonha and ha jobs
14:08:02 <shardy> as a developer I'm not wildly excited about that, but it does potentially reduce the test matrix
14:08:44 <shardy> panda: so an updates test will do a deploy, then an update, so it seems like pretty reasonable coverage to me
14:09:05 <shardy> the question I have is how we ensure per-service coverage now that optional services are covered via the test scenarios
14:09:18 <shardy> IME most ipv6 breakage happens in the per-service templates or profiles
14:09:39 <shardy> any other thoughts on this folks?
14:09:42 <slagle> i think we'd have to work towards ipv6 support in multinode jobs
14:09:49 <panda> shardy: and it's ok to test it only on master
14:11:19 <shardy> panda: surely we'll want to test updates and upgrades on stable branches too?
14:11:43 <shardy> maintaining that functionality on stable branches seems even more important than on master to me
14:12:25 <panda> ok
14:12:46 <shardy> panda: Do you want to follow up with a ML thread on this, and hopefully folks can respond when they've had time to consider the options?
14:13:09 <panda> shardy: sure, please follow up on this post http://lists.openstack.org/pipermail/openstack-dev/2016-October/105934.html
14:13:17 <panda> anyone ^
14:13:32 <shardy> panda: aha, apologies I missed that one
14:13:33 <shardy> thanks!
14:13:43 <shardy> sshnaidm: care to go next?
14:13:50 <sshnaidm> my topic is very short, please review and merge the quickstart experimental job patch: https://review.openstack.org/#/c/381094/ that's it, thanks
14:14:11 <shardy> #action all to review https://review.openstack.org/#/c/381094 quickstart CI patch
14:14:23 <shardy> Ok my item is related to the above discussion re updates/upgrades
14:14:37 <shardy> I'm prototyping composable upgrades here https://review.openstack.org/#/c/393448/
14:14:58 <shardy> I'd really like some way to validate the eventual implementation against our current monolithic upgrade implementation
14:15:10 <shardy> matbu: what's the status of your upgrades CI patch?
14:15:35 <shardy> weshay: and/or do we have third-party ci jobs which can provide this coverage?
14:15:37 <marios> shardy: thanks i also updated the spec and point at that now too (thanks for comments)
14:15:45 <matbu> shardy: i'm (re)-working in this review since monday
14:16:14 <matbu> shardy: i think we are really close to see it passing
14:16:35 <shardy> #link https://review.openstack.org/#/c/323750/ upgrades CI patch
14:16:43 <matbu> shardy: i have made a little tool with ansible that allow me to reproduce tripleo-ci failure: https://github.com/matbu/ansible-role-tripleo-ci
14:16:47 <shardy> matbu: Ok, what are the blockers to getting this landed?
14:16:58 <matbu> it could be useful for anyone who wants to debug a tripleo-ci deployment
14:17:13 <matbu> shardy: just having the experimental job green
14:17:27 <shardy> ideally I'd like CI coverage of upgrades, then post a series of patches doing a major rework of the upgrades implementation, and prove the job is still green afterwards
14:17:52 <shardy> matbu: Ok, I was unsure if we were just having issues keeping within the timeout or if there were other blockers
14:18:22 <matbu> shardy: ack, i really want to ends this review asap, in order to have this job for ocata work
14:19:04 <shardy> matbu: ack, OK thanks for the update, lets sync up later and see if we can land it $soon
14:19:21 <shardy> I'll also rebase my patches such that we can try them against the job
14:19:33 <matbu> shardy: ack
14:20:01 <matbu> shardy: i think we would have to add two major upgrade, one for M->N and N->O
14:20:07 * shardy looks around for fultonj
14:20:46 <shardy> Ok looks like he's not here, I'll follow-up offline
14:21:10 <shardy> I'd appreciate folks input on that spec tho, as atm I don't see how we can make progress until containers happen
14:21:24 <shardy> #topic bugs
14:22:02 <shardy> #link https://bugs.launchpad.net/tripleo/
14:22:12 <marios> shardy:  https://bugs.launchpad.net/tripleo/+bug/1640812 is a pretty big one for updates testing
14:22:12 <openstack> Launchpad bug 1640812 in tripleo "Network connectivity lost on node reboot" [Critical,In progress] - Assigned to Brent Eagles (beagles)
14:22:28 <marios> shardy: but beagles is working on it, mostly request for reviews
14:23:01 <shardy> marios: thanks, yeah I checked out those patches yesterday and I don't think they were ready to land yet
14:23:05 <shardy> will revisit today
14:23:06 <marios> shardy: the initial approach was changed, added a comment with pointers on the bug to current approach
14:23:31 * beagles nods thanks to marios on that
14:23:45 <marios> thanks beagles for the quick responses and fixes
14:24:02 <shardy> Ok well I'm in the process of deferring $everything to ocata-2, as I'd like to propose the release tomorrow
14:24:11 <shardy> so please add any critical patches to https://review.openstack.org/#/q/status:open+topic:tripleo/ocata-1
14:24:15 <beagles> the downside I think right now is without a patch in tht that depends on that series it isn't being exercised by our current tht ci
14:24:22 <shardy> any bugs not tracked there will be deferred tomorrow if they've not landed
14:25:00 <shardy> Any other important bugs to raise?
14:25:20 <shardy> mwhahaha: I'd appreciate your feedback on https://review.openstack.org/#/c/397644/
14:25:35 <mwhahaha> sure
14:25:39 <shardy> that's a pretty major bug as heat is basically broken for all usage where metadata polling or signalling is used
14:25:46 <shardy> (overcloud heat)
14:25:52 <shardy> mwhahaha: thanks!
14:26:33 <shardy> Ok any other bugs to discuss this week?
14:27:28 <shardy> #topic Projects releases or stable backports
14:27:39 <shardy> So, slagle did the stable release happen in the end last week?
14:27:48 <slagle> no, it happened last night :)
14:27:55 <shardy> slagle: aha :)
14:28:01 * shardy should have checked before asking :)
14:28:12 <shardy> Ok, well good that it's done, thanks for sorting it
14:28:15 <slagle> np!
14:28:26 <shardy> anything else to mention re stable releases or backports?
14:28:33 <slagle> we can still do another newton release as needed, maybe in 2-3 weeks
14:28:39 <shardy> +1 sounds good
14:28:43 <bandini> +1
14:28:59 <shardy> Ok so I already mentioned ocata-1 - that is due this week
14:29:15 <shardy> #link https://launchpad.net/tripleo/+milestone/ocata-1
14:29:45 <shardy> Quite a few in-progress bugs remaining, so please please help with reviews so we can land as many as possible
14:30:03 <shardy> I've already deferred most of the lower priority and/or unassigned things
14:30:21 <shardy> anything remaining will be deferred tomorrow unless super-critical
14:31:07 <shardy> https://releases.openstack.org/ocata/schedule.html
14:31:35 <shardy> ocata-2 is going to be only about a month later, so we'll have to push hard on landing some blueprints to avoid such a last-minute rush
14:31:52 <shardy> Any other questions or comments related to releases?
14:32:42 <shardy> #topic CI
14:32:59 <shardy> Been a fairly good week for CI I think - anyone have anything to mention here?
14:33:23 <shardy> flaper87: Any update on reinstating the containers CI job?
14:33:46 <panda> one promotion blocker this week, but it's already being addressed.
14:35:18 <shardy> Ookay then..
14:35:25 <shardy> #topic Specs
14:35:44 <shardy> So IIRC EmilienM was keen for a spec freeze to happen yesterday
14:35:58 <shardy> but AFAICS the review rate has been low, so I'm not sure we can enforce that
14:36:01 <trown> https://review.openstack.org/#/c/386250/ looks mergable, but I wanted to bring it here one more time first
14:36:15 <shardy> perhaps we can say anything not already at least proposed will not be considered
14:36:25 <shardy> for ocata
14:36:28 <shadower> +1 to that
14:36:36 * beagles looks at his shuffling feet, trying to hide his guilt...
14:37:29 <bandini> +1 yeah
14:37:33 <bandini> seems reasonable
14:37:34 <shardy> beagles: heh, we're all in the same position I think, spec reviews are time consuming and everyone is really busy
14:37:46 <trown> seems fair
14:38:01 <weshay> shardy, I'm working on container ci atm, it's wip and running w/ quickstart
14:38:04 <beagles> +1 to proposal
14:38:22 <shardy> weshay: Ok, that sounds good, thanks for the update! :)
14:38:33 <dtantsur> does it apply to blueprints?
14:38:41 <beagles> weshay: cool news!
14:38:50 <shardy> #action all to have a final push on spec reviews so we can declare feature proposal freeze
14:39:14 <dtantsur> shardy, question above ^^^
14:39:19 <shardy> dtantsur: I think there will be some flexibility around blueprints, but all major features really should be defined by now, or we'll simply run out of time
14:39:39 <dtantsur> do we have any kind of blueprint review process?
14:39:51 <marios> shardy: updated the composable upgrades spec... i think it may be mergable soon https://review.openstack.org/#/c/392116/
14:40:35 <marios> shardy: it points to your wip and i think describes the approach at a high level.
14:40:40 <shardy> dtantsur: we reviewed all of them in Barcelona, but in general it's something the PTL does, grooming the backlog and prioritizing
14:40:41 <dtantsur> of blueprint I wrote nobody apparently have reviewed https://blueprints.launchpad.net/tripleo/+spec/raid-workflow
14:40:45 <shardy> I've also been helping with it
14:40:59 <dtantsur> (we can chat about it outside of the meeting ofc)
14:41:18 <shardy> dtantsur: ack, I'd missed that one, will review, thanks
14:41:22 <dtantsur> thanks
14:41:46 <shardy> marios: ack, thanks
14:42:07 <shardy> marios: I'd appreciate your input on the WIP patch, it's basically working locally but I've not yet completed a full upgrade with it
14:42:26 <shardy> marios: the hard part I think will be unravelling the pacemaker mangling scripts
14:42:36 <shardy> probably just have ansible call those for the first iteration
14:42:53 <bandini> yeah
14:43:12 <shardy> Anyone have any other specs or features/blueprints they want to mention?
14:43:27 <marios> shardy: yes it isn't clear to me how we will fit those into the overarching workflow either.
14:43:51 <marios> shardy: honestly thanks for working on the protoype we are starting to calm a little now on blockers so hopefully lifecycle will engage more there
14:43:55 <beagles> I filed one yesterday related to a change in how neutron does security groups...
14:44:04 <shardy> marios: I think it will be step6 -> run pcmk scripts for the first iteration
14:44:25 <shardy> marios: but yeah, for sure we'll need more flexibility in future when we get to composable ha etc
14:44:29 <shardy> feedback welcome
14:44:30 <shardy> :)
14:44:35 <owalsh> I should have an update for https://review.openstack.org/#/c/388162/ today, reviews much appreciated
14:44:53 <shardy> beagles: got a link?
14:44:56 <beagles> it was brought to my attention last week and while I get the desire to do this... it's going to need some discussion and clarification "across the board" to see if it is something that can be reasonably tackled
14:45:15 <shardy> owalsh: ack, thanks will do
14:45:19 <beagles> shardy: https://blueprints.launchpad.net/tripleo/+spec/neutron-firewall-driver-upgrade
14:45:53 <beagles> tbh I'm not sure that anything like this has been attempted as part of a tripleo upgrade - it affects guest VMs in the overcloud
14:46:18 <beagles> pretty scary really
14:46:33 <shardy> yeah sounds terrifying, what could possibly go wrong :D
14:46:41 <shardy> but thanks, will check it out
14:46:58 <shardy> sounds like the migration mechanism may be sufficiently complex that we need a spec?
14:47:25 <shardy> defining the migration steps (and proving them outside the context of TripleO) sounds like the first step?
14:48:08 <shardy> I'd also seek to clarify if users will be willing to accept the risk of this kind of change on an already deployed cloud
14:48:19 <beagles> yup.. that's currently a WIP.. been in a few discussions regarding. My gut says that at best we might try for something that is optionally enabled on update
14:48:22 <shardy> vs making it a deploy time option which is immutable on upgrade
14:48:37 <beagles> right
14:48:59 <shardy> beagles: I assume we've got feedback there are folks who want to attempt this, right?
14:49:55 <beagles> shardy: that's a great question - I'll ask for clarification on what the driving forces are here.
14:50:14 <shardy> beagles: ack, sounds good, thanks!
14:50:20 <shardy> #topic Open Discussion
14:50:26 <shardy> Anyone have anything else to raise?
14:51:04 <shardy> dprince, flaper87: I'd appreciate your feedback on https://review.openstack.org/#/c/393448/
14:51:25 <shardy> particularly how we'd handle this workflow for heat orchestrated containers
14:51:26 <dprince> shardy: ack
14:51:50 <shardy> I suspect the logic will be totally different, but it'd be nice to sketch out how this will look for containers, so we've got upgrade support from day1
14:51:58 <dprince> shardy: tentatively positive. I like that it is integrated with the existing composable services I think
14:52:21 <shardy> or, at least not to wire in anything for puppet/ansible that will totally conflict with interfaces needed for containers
14:53:12 <shardy> Ok anything else folks?
14:53:40 * shardy waits 1 minute
14:54:15 <shardy> Ok thanks all!
14:54:18 <shardy> #endmeeting