14:00:29 <shardy> #startmeeting tripleo 14:00:32 <dtantsur> o/ 14:00:34 <d0ugal> Hey! 14:00:35 <openstack> Meeting started Tue Nov 15 14:00:29 2016 UTC and is due to finish in 60 minutes. The chair is shardy. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:37 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:37 <sshnaidm> o/ 14:00:39 <slagle> hi 14:00:40 <openstack> The meeting name has been set to 'tripleo' 14:00:41 <shardy> Hi all, who's around? 14:00:42 <adarazs> o/ 14:00:43 <ccamacho> hi folks! o/ 14:00:44 <jpich> o/ 14:00:46 <marios> \o 14:00:51 <beagles> o/ 14:00:56 <owalsh> o/ 14:01:06 <arxcruz> o/ 14:01:15 <cdearborn> o/ 14:01:28 <shardy> #link https://wiki.openstack.org/wiki/Meetings/TripleO 14:01:51 <shardy> #link https://etherpad.openstack.org/p/tripleo-meeting-items 14:02:00 <shardy> Please add any one-off items there ^^ 14:02:03 <shadower> hey 14:02:16 <matbu> o/ 14:02:19 <shardy> Ok, hi all, lets get started! :) 14:02:23 <shardy> #topic agenda 14:02:23 <shardy> * one off agenda items 14:02:23 <shardy> * bugs 14:02:23 <shardy> * Projects releases or stable backports 14:02:23 <shardy> * CI 14:02:25 <weshay> o/ 14:02:25 <shardy> * Specs 14:02:28 <shardy> * open discussion 14:02:31 <trown> o/ 14:02:46 <mwhahaha> o/ 14:02:53 <panda> o/ 14:03:10 <pradk> o/ 14:04:02 <shardy> #topic one off agenda items 14:04:18 <shardy> Ok so we have a few this week, panda you want to go first? 14:04:38 <panda> shardy: sure, my items really comes down to a single question 14:05:33 <panda> where do we want to test IPv6 ? there are two proposals, using updates job -OR- moving ll HA jobs IPv6, move network isolation testing for IPv4 on non-ha job, and test non-netiso environment on multinode. 14:06:56 <panda> moving all seems complicated, but don't know if testing only updates on master is enough coverage 14:07:37 <shardy> This is kind of related to a recent question from bandini ref can we just always enable pacemaker, to reduce the differences between the nonha and ha jobs 14:08:02 <shardy> as a developer I'm not wildly excited about that, but it does potentially reduce the test matrix 14:08:44 <shardy> panda: so an updates test will do a deploy, then an update, so it seems like pretty reasonable coverage to me 14:09:05 <shardy> the question I have is how we ensure per-service coverage now that optional services are covered via the test scenarios 14:09:18 <shardy> IME most ipv6 breakage happens in the per-service templates or profiles 14:09:39 <shardy> any other thoughts on this folks? 14:09:42 <slagle> i think we'd have to work towards ipv6 support in multinode jobs 14:09:49 <panda> shardy: and it's ok to test it only on master 14:11:19 <shardy> panda: surely we'll want to test updates and upgrades on stable branches too? 14:11:43 <shardy> maintaining that functionality on stable branches seems even more important than on master to me 14:12:25 <panda> ok 14:12:46 <shardy> panda: Do you want to follow up with a ML thread on this, and hopefully folks can respond when they've had time to consider the options? 14:13:09 <panda> shardy: sure, please follow up on this post http://lists.openstack.org/pipermail/openstack-dev/2016-October/105934.html 14:13:17 <panda> anyone ^ 14:13:32 <shardy> panda: aha, apologies I missed that one 14:13:33 <shardy> thanks! 14:13:43 <shardy> sshnaidm: care to go next? 14:13:50 <sshnaidm> my topic is very short, please review and merge the quickstart experimental job patch: https://review.openstack.org/#/c/381094/ that's it, thanks 14:14:11 <shardy> #action all to review https://review.openstack.org/#/c/381094 quickstart CI patch 14:14:23 <shardy> Ok my item is related to the above discussion re updates/upgrades 14:14:37 <shardy> I'm prototyping composable upgrades here https://review.openstack.org/#/c/393448/ 14:14:58 <shardy> I'd really like some way to validate the eventual implementation against our current monolithic upgrade implementation 14:15:10 <shardy> matbu: what's the status of your upgrades CI patch? 14:15:35 <shardy> weshay: and/or do we have third-party ci jobs which can provide this coverage? 14:15:37 <marios> shardy: thanks i also updated the spec and point at that now too (thanks for comments) 14:15:45 <matbu> shardy: i'm (re)-working in this review since monday 14:16:14 <matbu> shardy: i think we are really close to see it passing 14:16:35 <shardy> #link https://review.openstack.org/#/c/323750/ upgrades CI patch 14:16:43 <matbu> shardy: i have made a little tool with ansible that allow me to reproduce tripleo-ci failure: https://github.com/matbu/ansible-role-tripleo-ci 14:16:47 <shardy> matbu: Ok, what are the blockers to getting this landed? 14:16:58 <matbu> it could be useful for anyone who wants to debug a tripleo-ci deployment 14:17:13 <matbu> shardy: just having the experimental job green 14:17:27 <shardy> ideally I'd like CI coverage of upgrades, then post a series of patches doing a major rework of the upgrades implementation, and prove the job is still green afterwards 14:17:52 <shardy> matbu: Ok, I was unsure if we were just having issues keeping within the timeout or if there were other blockers 14:18:22 <matbu> shardy: ack, i really want to ends this review asap, in order to have this job for ocata work 14:19:04 <shardy> matbu: ack, OK thanks for the update, lets sync up later and see if we can land it $soon 14:19:21 <shardy> I'll also rebase my patches such that we can try them against the job 14:19:33 <matbu> shardy: ack 14:20:01 <matbu> shardy: i think we would have to add two major upgrade, one for M->N and N->O 14:20:07 * shardy looks around for fultonj 14:20:46 <shardy> Ok looks like he's not here, I'll follow-up offline 14:21:10 <shardy> I'd appreciate folks input on that spec tho, as atm I don't see how we can make progress until containers happen 14:21:24 <shardy> #topic bugs 14:22:02 <shardy> #link https://bugs.launchpad.net/tripleo/ 14:22:12 <marios> shardy: https://bugs.launchpad.net/tripleo/+bug/1640812 is a pretty big one for updates testing 14:22:12 <openstack> Launchpad bug 1640812 in tripleo "Network connectivity lost on node reboot" [Critical,In progress] - Assigned to Brent Eagles (beagles) 14:22:28 <marios> shardy: but beagles is working on it, mostly request for reviews 14:23:01 <shardy> marios: thanks, yeah I checked out those patches yesterday and I don't think they were ready to land yet 14:23:05 <shardy> will revisit today 14:23:06 <marios> shardy: the initial approach was changed, added a comment with pointers on the bug to current approach 14:23:31 * beagles nods thanks to marios on that 14:23:45 <marios> thanks beagles for the quick responses and fixes 14:24:02 <shardy> Ok well I'm in the process of deferring $everything to ocata-2, as I'd like to propose the release tomorrow 14:24:11 <shardy> so please add any critical patches to https://review.openstack.org/#/q/status:open+topic:tripleo/ocata-1 14:24:15 <beagles> the downside I think right now is without a patch in tht that depends on that series it isn't being exercised by our current tht ci 14:24:22 <shardy> any bugs not tracked there will be deferred tomorrow if they've not landed 14:25:00 <shardy> Any other important bugs to raise? 14:25:20 <shardy> mwhahaha: I'd appreciate your feedback on https://review.openstack.org/#/c/397644/ 14:25:35 <mwhahaha> sure 14:25:39 <shardy> that's a pretty major bug as heat is basically broken for all usage where metadata polling or signalling is used 14:25:46 <shardy> (overcloud heat) 14:25:52 <shardy> mwhahaha: thanks! 14:26:33 <shardy> Ok any other bugs to discuss this week? 14:27:28 <shardy> #topic Projects releases or stable backports 14:27:39 <shardy> So, slagle did the stable release happen in the end last week? 14:27:48 <slagle> no, it happened last night :) 14:27:55 <shardy> slagle: aha :) 14:28:01 * shardy should have checked before asking :) 14:28:12 <shardy> Ok, well good that it's done, thanks for sorting it 14:28:15 <slagle> np! 14:28:26 <shardy> anything else to mention re stable releases or backports? 14:28:33 <slagle> we can still do another newton release as needed, maybe in 2-3 weeks 14:28:39 <shardy> +1 sounds good 14:28:43 <bandini> +1 14:28:59 <shardy> Ok so I already mentioned ocata-1 - that is due this week 14:29:15 <shardy> #link https://launchpad.net/tripleo/+milestone/ocata-1 14:29:45 <shardy> Quite a few in-progress bugs remaining, so please please help with reviews so we can land as many as possible 14:30:03 <shardy> I've already deferred most of the lower priority and/or unassigned things 14:30:21 <shardy> anything remaining will be deferred tomorrow unless super-critical 14:31:07 <shardy> https://releases.openstack.org/ocata/schedule.html 14:31:35 <shardy> ocata-2 is going to be only about a month later, so we'll have to push hard on landing some blueprints to avoid such a last-minute rush 14:31:52 <shardy> Any other questions or comments related to releases? 14:32:42 <shardy> #topic CI 14:32:59 <shardy> Been a fairly good week for CI I think - anyone have anything to mention here? 14:33:23 <shardy> flaper87: Any update on reinstating the containers CI job? 14:33:46 <panda> one promotion blocker this week, but it's already being addressed. 14:35:18 <shardy> Ookay then.. 14:35:25 <shardy> #topic Specs 14:35:44 <shardy> So IIRC EmilienM was keen for a spec freeze to happen yesterday 14:35:58 <shardy> but AFAICS the review rate has been low, so I'm not sure we can enforce that 14:36:01 <trown> https://review.openstack.org/#/c/386250/ looks mergable, but I wanted to bring it here one more time first 14:36:15 <shardy> perhaps we can say anything not already at least proposed will not be considered 14:36:25 <shardy> for ocata 14:36:28 <shadower> +1 to that 14:36:36 * beagles looks at his shuffling feet, trying to hide his guilt... 14:37:29 <bandini> +1 yeah 14:37:33 <bandini> seems reasonable 14:37:34 <shardy> beagles: heh, we're all in the same position I think, spec reviews are time consuming and everyone is really busy 14:37:46 <trown> seems fair 14:38:01 <weshay> shardy, I'm working on container ci atm, it's wip and running w/ quickstart 14:38:04 <beagles> +1 to proposal 14:38:22 <shardy> weshay: Ok, that sounds good, thanks for the update! :) 14:38:33 <dtantsur> does it apply to blueprints? 14:38:41 <beagles> weshay: cool news! 14:38:50 <shardy> #action all to have a final push on spec reviews so we can declare feature proposal freeze 14:39:14 <dtantsur> shardy, question above ^^^ 14:39:19 <shardy> dtantsur: I think there will be some flexibility around blueprints, but all major features really should be defined by now, or we'll simply run out of time 14:39:39 <dtantsur> do we have any kind of blueprint review process? 14:39:51 <marios> shardy: updated the composable upgrades spec... i think it may be mergable soon https://review.openstack.org/#/c/392116/ 14:40:35 <marios> shardy: it points to your wip and i think describes the approach at a high level. 14:40:40 <shardy> dtantsur: we reviewed all of them in Barcelona, but in general it's something the PTL does, grooming the backlog and prioritizing 14:40:41 <dtantsur> of blueprint I wrote nobody apparently have reviewed https://blueprints.launchpad.net/tripleo/+spec/raid-workflow 14:40:45 <shardy> I've also been helping with it 14:40:59 <dtantsur> (we can chat about it outside of the meeting ofc) 14:41:18 <shardy> dtantsur: ack, I'd missed that one, will review, thanks 14:41:22 <dtantsur> thanks 14:41:46 <shardy> marios: ack, thanks 14:42:07 <shardy> marios: I'd appreciate your input on the WIP patch, it's basically working locally but I've not yet completed a full upgrade with it 14:42:26 <shardy> marios: the hard part I think will be unravelling the pacemaker mangling scripts 14:42:36 <shardy> probably just have ansible call those for the first iteration 14:42:53 <bandini> yeah 14:43:12 <shardy> Anyone have any other specs or features/blueprints they want to mention? 14:43:27 <marios> shardy: yes it isn't clear to me how we will fit those into the overarching workflow either. 14:43:51 <marios> shardy: honestly thanks for working on the protoype we are starting to calm a little now on blockers so hopefully lifecycle will engage more there 14:43:55 <beagles> I filed one yesterday related to a change in how neutron does security groups... 14:44:04 <shardy> marios: I think it will be step6 -> run pcmk scripts for the first iteration 14:44:25 <shardy> marios: but yeah, for sure we'll need more flexibility in future when we get to composable ha etc 14:44:29 <shardy> feedback welcome 14:44:30 <shardy> :) 14:44:35 <owalsh> I should have an update for https://review.openstack.org/#/c/388162/ today, reviews much appreciated 14:44:53 <shardy> beagles: got a link? 14:44:56 <beagles> it was brought to my attention last week and while I get the desire to do this... it's going to need some discussion and clarification "across the board" to see if it is something that can be reasonably tackled 14:45:15 <shardy> owalsh: ack, thanks will do 14:45:19 <beagles> shardy: https://blueprints.launchpad.net/tripleo/+spec/neutron-firewall-driver-upgrade 14:45:53 <beagles> tbh I'm not sure that anything like this has been attempted as part of a tripleo upgrade - it affects guest VMs in the overcloud 14:46:18 <beagles> pretty scary really 14:46:33 <shardy> yeah sounds terrifying, what could possibly go wrong :D 14:46:41 <shardy> but thanks, will check it out 14:46:58 <shardy> sounds like the migration mechanism may be sufficiently complex that we need a spec? 14:47:25 <shardy> defining the migration steps (and proving them outside the context of TripleO) sounds like the first step? 14:48:08 <shardy> I'd also seek to clarify if users will be willing to accept the risk of this kind of change on an already deployed cloud 14:48:19 <beagles> yup.. that's currently a WIP.. been in a few discussions regarding. My gut says that at best we might try for something that is optionally enabled on update 14:48:22 <shardy> vs making it a deploy time option which is immutable on upgrade 14:48:37 <beagles> right 14:48:59 <shardy> beagles: I assume we've got feedback there are folks who want to attempt this, right? 14:49:55 <beagles> shardy: that's a great question - I'll ask for clarification on what the driving forces are here. 14:50:14 <shardy> beagles: ack, sounds good, thanks! 14:50:20 <shardy> #topic Open Discussion 14:50:26 <shardy> Anyone have anything else to raise? 14:51:04 <shardy> dprince, flaper87: I'd appreciate your feedback on https://review.openstack.org/#/c/393448/ 14:51:25 <shardy> particularly how we'd handle this workflow for heat orchestrated containers 14:51:26 <dprince> shardy: ack 14:51:50 <shardy> I suspect the logic will be totally different, but it'd be nice to sketch out how this will look for containers, so we've got upgrade support from day1 14:51:58 <dprince> shardy: tentatively positive. I like that it is integrated with the existing composable services I think 14:52:21 <shardy> or, at least not to wire in anything for puppet/ansible that will totally conflict with interfaces needed for containers 14:53:12 <shardy> Ok anything else folks? 14:53:40 * shardy waits 1 minute 14:54:15 <shardy> Ok thanks all! 14:54:18 <shardy> #endmeeting