14:00:07 <shardy> #startmeeting tripleo
14:00:08 <openstack> Meeting started Tue Jun 14 14:00:07 2016 UTC and is due to finish in 60 minutes.  The chair is shardy. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:09 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:12 <openstack> The meeting name has been set to 'tripleo'
14:00:14 <shardy> #topic rollcall
14:00:19 <EmilienM> hello !
14:00:21 <coolsvap> o/
14:00:25 <marios> o/
14:00:26 <sshnaidm> o/
14:00:27 <rhallisey> hi
14:00:27 <ccamacho> Hello guys!
14:00:27 <shardy> Hi all!
14:00:30 <[1]cdearborn> o/
14:00:33 <mandre> hi
14:00:42 <saneax> Hi folks
14:00:49 <skramaja> hi
14:00:58 <beagles> o/
14:01:02 <qasims> o/
14:01:06 <karthiks> Hello
14:01:19 <shadower> hey
14:01:30 <akrivoka> \o
14:01:40 <trown> o/
14:01:42 <adarazs> \o
14:01:53 <derekh> o/
14:02:21 <d0ugal> \o/
14:02:33 <shardy> #topic agenda
14:02:33 <shardy> * one off agenda items
14:02:33 <shardy> * bugs
14:02:33 <shardy> * Projects releases or stable backports
14:02:33 <shardy> * CI
14:02:35 <shardy> * Specs
14:02:38 <shardy> * open discussion
14:02:42 <shardy> #link https://wiki.openstack.org/wiki/Meetings/TripleO
14:02:54 <shardy> Does anyone have any one-off items to add this week?
14:03:17 <jtomasek> o/
14:03:28 <marios> shardy: did you want to bring up composable upgrades a bit if we have time
14:03:42 <marios> i guess open discussion
14:04:05 <shardy> marios: yup, good idea - let's leave that till open discussion, then it can continue on the ML
14:04:15 <marios> s/you/i/we yeah sure
14:04:19 <shardy> I've been meaning to start a thread about it, but working on some examples first
14:04:29 <marios> ok thanks
14:04:56 <shardy> Ok, if there's nothing else, let's get through the recurring items
14:05:03 <shardy> #topic bugs
14:05:09 <weshay> o/
14:05:26 <shardy> #link https://bugs.launchpad.net/tripleo/
14:06:03 <jrist> o/
14:06:03 <shardy> Does anyone have any bugs to mention this week in particular?
14:06:58 <beagles> yes
14:07:05 <shardy> CI looks a bit happier today, but the HA job still looks fairly broken
14:07:15 <beagles> there are a few mtu related bugs that, while we are dealing with on master
14:07:20 <beagles> are going to be a pain in mitaka
14:07:36 <beagles> e.g. https://bugs.launchpad.net/tripleo/+bug/1590101
14:07:36 <openstack> Launchpad bug 1590101 in tripleo "global_physnet_mtu is set to 1400 instead of ethernet standard 1500" [High,Triaged]
14:07:59 <EmilienM> shardy: HA job looks stable to me
14:08:24 <EmilienM> shardy: do you have logs handy or a LP?
14:08:24 * derekh hit that mtu bug while deploying rh2
14:08:42 <sshnaidm> not critical, but worth to see: https://bugs.launchpad.net/tripleo/+bug/1592284  I think there is a misconfiguration again.
14:08:42 <openstack> Launchpad bug 1592284 in tripleo "Sahara tests fail" [Undecided,New]
14:09:24 <beagles> I'm uncertain what the best way of dealing with this as a fix to mitaka - i.e. is it acceptable to alter the semantics of that configuration in a patch
14:09:24 <shardy> EmilienM: ah, sorry my results were stale :)
14:09:34 <EmilienM> yeah, our CI is currently quite stable
14:10:21 <bnemec> beagles: Given that the current semantics are wrong, I think it would be.
14:10:32 <beagles> fwiw, for mitaka we might be better without mtu configuration altogether, but I haven't tested in yet
14:10:49 <beagles> s/in/it/
14:11:11 <beagles> bnemec: yeah, that's my thinking
14:11:23 <beagles> I'll submit patches asap and we can discuss more :)
14:11:54 <shardy> beagles: +1, given the current behavior a backport seems justified
14:12:05 <derekh> beagles: I'm setting it to 1500, but leaving unset by default I think would work too
14:12:20 <shardy> Any other bugs to mention before we move on?
14:12:56 <derekh> not sure how common this is
14:13:06 <derekh> but another bug I hit deploying rh2 is https://bugs.launchpad.net/tripleo/+bug/1587656
14:13:06 <openstack> Launchpad bug 1587656 in tripleo "undercloud install fails if devices exist with no dhcp" [High,Triaged]
14:13:30 <derekh> the undercloud node has 6 nics, all six are wired up but don't have dhcp
14:13:50 <derekh> so undercloud install tries to get dhcp for all interfaces and fails
14:13:55 <bnemec> Really?  I deploy underclouds all the time with nics that can't DHCP.
14:14:33 <derekh> bnemec: yup, well that at least the conclusing I came to when deploying rh2
14:14:51 <derekh> bnemec: are your nics connected?
14:15:02 <derekh> bnemec: to a network
14:15:18 <slagle> dhcp_all_interfaces.yaml should only get written out if the actual os-net-config comamnd fails
14:15:21 <slagle> it's a fallback
14:15:44 <slagle> so probably need to trace that back to why it failed in the first place
14:15:55 <bnemec> Oh, yeah.  ERROR: configuration of safe defaults failed.
14:16:03 <derekh> slagle: ahh, so my os-net-config failed for some reason? will see if I can find out why
14:16:23 <bnemec> Incidentally, this is the second time I've seen the safe defaults fail when a config was invalid.  We should look into that too.
14:16:26 <slagle> yea see if you can find the actual root error earlier up
14:16:40 <slagle> bnemec: yea, the safe defaults i've never seen work tbh
14:16:45 <derekh> slagle: will do
14:17:24 <slagle> bnemec: i dont think it makes sense for the undercloud either
14:17:29 <slagle> for the overcloud, maybe it does
14:17:41 <shardy> Ok, sounds good, lets move on to the next agenda item and follow up in the bug/#tripleo
14:17:55 <shardy> #topic Projects releases or stable backports
14:18:07 <shardy> So, it's only one month until n-2:
14:18:16 <shardy> #link http://releases.openstack.org/newton/schedule.html
14:18:29 <shardy> #link https://launchpad.net/tripleo/+milestone/newton-2
14:18:35 <EmilienM> I hope we can land all composable services by then
14:18:49 <shardy> we need to push hard on landing all the things that are only started on in progress
14:19:03 <shardy> if anyone thinks they will miss the n-2 deadline, please move it to n3
14:19:26 <shardy> also, any "Unknown" delivery status, please change (or I'll bump them to n-3)
14:20:30 <shardy> Does anyone have any questions/comments re the release schedule, or other release related things to mention?
14:21:47 <coolsvap> from the last meeting we needed volunteers with who can help with project releases, any update on that?
14:22:26 <shardy> coolsvap: EmilienM has offered to be the release liason, and anyone can propose patches to openstack/releases too for interim releases that address specific issues
14:22:40 <shardy> coolsvap: IIRC you were interested in keeping track of the stable branch releases?
14:23:02 <coolsvap> shardy, yeah i can help
14:24:15 <shardy> coolsvap: ack, sounds good, well basically it just requires figuring out when a release is required, then pushing the patch to openstack/releases
14:24:40 <EmilienM> coolsvap: we can do next release together
14:24:41 <shardy> then ping me and/or EmilienM and we can review it together
14:24:55 <shardy> sounds good
14:24:56 <coolsvap> shardy, yup EmilienM thx sure
14:25:36 <shardy> I think we agreed to keep stable branch releases as "on demand" and master branch aligned with the milestones (possibly with additional interim releases where needed)
14:25:56 <shardy> the main thing I think is some what more regular releases from the stable branches than we have been doing recently
14:26:11 <shardy> Anything else re releases?
14:26:47 <shardy> #topic CI
14:26:54 <shardy> derekh: care to give us an update?
14:27:45 <derekh> we had a testenv problem from friday evening that we resolvved on monday morning, so CI was out of the weekend
14:28:21 <derekh> I havn't been keeping track of regression breaking trunk deploys, I think EmilienM fixed one yesterday
14:28:52 <EmilienM> yeah we landed one patch to fix HA job
14:29:02 <EmilienM> all CI jobs from master & stable/mitaka are gree
14:29:04 <EmilienM> green*
14:29:11 <slagle> in other ci news, i'm still trying to push the tripleo nodepool multinode work forward
14:29:21 <derekh> on the rh2 front, I'll be redeploying it later(with a few other team members) this evening and should be pushing a patch to add it to infra tomorrow
14:29:30 <EmilienM> slagle: do you have blockers / or/and need review?
14:29:32 <derekh> once that done, I'll be adding an experimenta OVB based job
14:29:44 <slagle> reworking things a bit presently to use the same networking setup as devstack networking (ovs vxlan tunnels), as i think that will be more well received with infra
14:30:04 <slagle> and we can collaborate on that
14:30:31 <slagle> there was also some feedback that if we could start with just 2 nodes instead of 3, it'd be better to help prove things
14:30:40 <slagle> that would require us to deploy an aio overcloud
14:30:56 <slagle> which may be possible when composable nova-compute lands
14:31:11 <shardy> the upgrades job on stable/liberty has been broken since march - is anyone interested in looking into that?
14:31:15 <EmilienM> slagle: we're on it and it should happen soon, we started to land nova composable things today
14:31:59 <shardy> it would be good to get that resolved, as it's stalled a lot of stable/liberty reviews I think
14:32:04 <EmilienM> slagle: I'll work on compute role as a priority tomorrow
14:32:16 <slagle> EmilienM: thx
14:32:21 <EmilienM> shardy: yes, I can look at it
14:32:46 <reduxio> Hello. Can anyone point me to where i need to go to register as a third party driver developer?
14:32:51 <shardy> EmilienM: if you're busy with composable services don't spend too much time on it, perhaps some other folks can help out
14:33:20 <weshay> matbu, ^
14:33:31 <shardy> reduxio: Hi! There's no registration process as such, you can just show up with patches to our repos
14:33:56 <shardy> reduxio: FYI we're in the process of some major internal refactoring, so you may want to drop into #tripleo to discuss before starting
14:34:02 <EmilienM> shardy: ok I won't spend too much time, just a quick look to see where does it come from
14:34:36 <shardy> reduxio: there are several integration points for vendor drivers, so if you give us more details, we can help you get started
14:34:49 <shardy> (in #tripleo after the meeting is probably best)
14:35:15 <shardy> EmilienM: ack, thanks!
14:35:21 <EmilienM> reduxio: if it's a neutron or cinder driver, please ping me I can also help
14:35:40 <shardy> #topic Specs
14:35:40 <reduxio> shardy: Thanks, i will join tripleo.
14:35:55 <shardy> So, there's been some discussion on the topic of composable upgrades
14:36:08 <shardy> https://etherpad.openstack.org/p/tripleo-composable-upgrades-discussion
14:36:27 <shardy> Basically there's some interest in using ansible to drive the upgrade process (vs the scripts currently used)
14:36:44 <shardy> I'm also looking at enabling rolling updates via SoftwareDeploymentGroup as an alternative
14:37:24 <marios> to clarify, the ansible poc for upgrades is to enable upgrade of specific services.
14:37:36 <shardy> it'd be good to have wider review of the patches referenced, so we can start to gain consensus on whether the proposed approaches are viable, and if they're a good fit for TripleO
14:37:56 <shardy> I'm planning to start a ML thread with more details when I get the alternative heat based approach working
14:38:00 <marios> compared to the current workflow which is to upgrade per node (docs at https://review.openstack.org/#/c/308985/ for current workflow)
14:38:52 <jokke_> quick question of that. I see a lot of "rolling upgrades" been talked around at the moment. Does this mean tripleO components or the services in this specific case?
14:39:12 <shardy> jokke_: it means the services currently running on the controller
14:39:15 <reduxio> EmilienM: It's a cinder driver, and i will appreciate any help since i am new to this and kind of stumbling...
14:39:19 <shardy> and/or upgrading the controllers one by one
14:39:58 <jistr> though one-by-one might not be often possible due to db migrations, at least during major upgrades
14:40:00 <jokke_> shardy: ok, then it might be worth of waiting until the service teams figures out how to do rolling upgrades before widly trying to figure out how we do it in the deployment end ;)
14:40:14 <rhallisey> I don't know if a one by one will work
14:40:15 <jistr> (we already do minor updates in a rolling fashion)
14:40:30 <rhallisey> I'm also skeptical if rolling service by service will work
14:40:33 <marios> jokke_: agree, though this is more about the mechanism at this stage
14:40:49 <marios> jokke_: once we have per version upgrade 'hook' we can do the thing the service teams tell us is the right thing
14:41:11 <marios> s/per version/per service
14:41:14 <jistr> rhallisey: yea the PoC i submitted should allow rolling the service-by-service into one big step if we discover that service-by-service is undoable with RPM dependencies
14:41:27 <jokke_> marios: gotcha
14:41:47 <jistr> rhallisey: s/rolling/conflating/
14:41:50 <shardy> Yeah, it's about wiring things in to enable per-service definition of what's required to do an upgrade
14:41:51 <jistr> to be more precise
14:42:38 <shardy> perhaps the rolling upgrade thing is a tangent, but it's one of the reasons cited for wanting to use ansible
14:43:03 <jokke_> just thought that it might be worth of raising that seems like most of the services does not know how/if their individual components can be upgraded on the fly or not ;)
14:43:12 <shardy> I'm making SoftwareDeploymentGroup able to do it, so we can compare the approaches
14:43:28 <jistr> shardy: ah ok i see what you mean. Well it depends on the particular service. E.g. for swift storage or probably nova computes, we indeed want a rolling upgrade even when it's a major upgrade.
14:43:47 <jistr> but for others, like many controller services, rolling upgrade isn't possible
14:44:06 <jistr> so flexibility in this is key i think
14:44:11 <shardy> jistr: ack, but we still want/need some method to perform rolling updates where the role and services support it
14:44:45 <shardy> Ok, I guess we can follow up on the ML, but in the meantime pls comment on the patches
14:45:01 <jistr> yea those approaches (rolling vs. non-rolling) could be mixed perhaps, even on a single node
14:45:04 <rhallisey> shardy, right, for that reason I'd argue we use ansible
14:45:06 <shardy> we may have some side-meetings on this topic over the next couple of weeks to gain consensus
14:45:09 <jistr> depends again if we hit issues with RPM deps
14:45:12 <rhallisey> because technially services could be anywhere
14:45:14 <shardy> (location/time tbc)
14:46:13 <rhallisey> there is a bluejeans call tmr for anyone interested
14:46:28 <rhallisey> figured I'd throw out and open invite
14:47:34 <shardy> Anything else on specs to raise this week?
14:47:40 <shadower> The validations spec could use some eyballs. It's stuck waiting for more reviews/feedback
14:47:53 <shardy> shadower: ack, thanks for the reminder
14:48:00 <shadower> I understand there's a ton other work so people don't have time
14:48:05 <shadower> but it's a small spec
14:48:12 <shadower> #link https://review.openstack.org/#/c/255792/
14:48:33 <shardy> the new validations repo is setup now, and I didn't see any objections to adding shadower and mandre to tripleo-core so they can approve patches for validations
14:48:47 <shardy> I'll do that later today
14:48:53 <shadower> shardy: thanks!
14:49:09 <shardy> #topic Open Discussion
14:49:21 <shardy> One thing from me
14:49:36 <shardy> I'm going to be travelling next week, so can someone volunteer to run this meeting?
14:50:00 <shardy> I may make it, but likely to be in the airport with uncertain wifi connectivity
14:50:07 <trown> I can run the meeting next week
14:50:19 <shardy> #info trown to run next weeks meeting
14:50:23 <derekh> quick one from me, If people have time, could ye take a look at updating the development environment census http://lists.openstack.org/pipermail/openstack-dev/2016-June/097284.html
14:50:23 <shardy> trown: thanks! :)
14:50:26 <trown> np
14:51:45 <shardy> Ok, anything else to raise before we wrap up the meeting?
14:52:33 <shardy> Alright then, thanks everyone!
14:52:36 <shardy> #endmeeting