14:00:16 <shardy> #startmeeting tripleo
14:00:22 <openstack> Meeting started Tue May 24 14:00:16 2016 UTC and is due to finish in 60 minutes.  The chair is shardy. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:23 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:26 <openstack> The meeting name has been set to 'tripleo'
14:00:29 <EmilienM> o/
14:00:30 <shardy> #topic rollcall
14:00:30 <derekh> o/
14:00:35 <shardy> Hey all, who's around?
14:00:37 <rbrady> \o
14:00:37 <jdob> [T]/
14:00:47 <pradk> o/
14:00:58 <trown> o/
14:01:50 <shardy> #topic agenda
14:01:55 * derekh wonders if the left handed people are using different slashes to the right handed people
14:01:59 <d0ugal> o/
14:02:03 <shardy> #link https://wiki.openstack.org/wiki/Meetings/TripleO
14:02:16 <shardy> * one off agenda items
14:02:16 <shardy> * bugs
14:02:16 <shardy> * Projects releases or stable backports
14:02:16 <shardy> * CI
14:02:16 <shardy> * Specs
14:02:18 <shardy> * open discussion
14:02:24 <shardy> Anyone have any one-off items to add today?
14:03:10 <shardy> #topic one off agenda items
14:03:22 <shardy> #info Request for mentoring : http://lists.openstack.org/pipermail/openstack-dev/2016-May/095667.html
14:03:35 <shardy> So this was just to draw attention to a request for mentoring on the ML
14:03:47 * beagles joins a bit late
14:03:58 <thrash> o/
14:04:05 <slagle> shardy: i saw that and was wondering if it meant that there has been a specific request for tripleo mentoring?
14:04:07 <shardy> TripleO is listed as a project with potential need for mentors, so feel free to step up if you're interested in doing that
14:04:18 <trown> oh neat
14:04:45 <shardy> slagle: tbh I'm not sure - I saw TripleO on the list and thought it'd be good if at least a couple of folks stepped up to find out ;)
14:05:05 <shardy> it may be that it's just missing coverage from the pool of mentors I guess
14:05:42 <trown> I will volunteer for that, and can report back :)
14:05:51 <shardy> anyways, let me know if anyone does so and needs help e.g finding low-hanging-fruit bugs or introductory materials
14:05:56 <shardy> trown: thanks!
14:06:08 <shardy> #topic bugs
14:06:24 <shardy> https://bugs.launchpad.net/tripleo/
14:06:30 <shardy> #link https://bugs.launchpad.net/tripleo/
14:06:52 <shardy> So we got bitten by an issue with hiera yesterday, which is now fixed via RDO packaging
14:07:09 <shardy> and there's still an outstanding issue re pip-and-virtualenv with a patch due to land
14:07:23 <shardy> anything else to raise, in particular impacting CI?
14:07:35 <slagle> i think we can land the pip and venv fix now
14:07:39 <shardy> trown: have you looked lately at the reasons why the periodic job is failing?
14:07:45 <slagle> it has passed all jobs cumulatively
14:07:53 <slagle> https://review.openstack.org/#/c/320019/
14:07:55 <shardy> slagle: +1 - I just saw that all jobs passed over $rechecks
14:08:21 <slagle> i just merged it :)
14:08:28 <shardy> thanks :)
14:08:29 <trown> shardy: not for a couple days no... a few other fires in RDO land
14:09:04 <shardy> trown: Ok, the reason is related to releases - we're due to do an n-1 milestone release soon and we need a clean promote to get the good sha's from
14:09:18 <shardy> One more thing related to bugs:
14:09:21 <shardy> " 326 Open bugs "
14:09:33 <shardy> I noticed there's a lot of Fix Commited bugs making that number larger
14:09:58 <shardy> when we cut releases for all-the-things for n-1, I'd like to run a script over the tripleo bugs and flip all Fix Committed to Fix Released
14:10:05 <trown> shardy: ok, we have a promote in RDO from a few days ago, but I will look into what is wrong on master now that we got hiera sorted
14:10:05 <shardy> does that sound OK to people?
14:10:16 <shardy> trown: that would be awesome, thanks!
14:10:18 <beagles> +1 shardy
14:10:32 <slagle> +1
14:10:44 <shardy> it'd be great to get the open bugs back to a somewhat manageable level, and I think that should help a lot
14:11:31 <shardy> Ok, any other bugs to highlight?
14:12:10 <shardy> #topic Projects releases or stable backports
14:12:38 <shardy> So, as I mentioned, n-1 is due on or around next Friday
14:12:45 <shardy> #link http://releases.openstack.org/newton/schedule.html
14:13:25 <shardy> I know historically we've not done milestone releases, but I wanted to propose a patch to openstack/releases next time we get a clean periodic promote so we've got a coordinated milestone release
14:13:46 <shardy> I guess folks can consume it either via the package versions, or via the delorean hash from the periodic promote?
14:14:19 <shardy> obviously we might automate that via quickstart in future, but any objections to just getting things started by cutting some releases around the milestone?
14:14:30 <trown> +1 to milestone releases
14:14:59 <slagle> sounds good to me
14:15:00 <jistr> +1 would be nice to get more towards continuous releasing if possible
14:15:13 <shardy> Ok, sounds good then
14:15:18 <slagle> maybe we could even create a symlink to the repo on the delorean server
14:15:18 <trown> I think we could only consume them via DLRN for now (for packages)
14:15:44 <shardy> I'll see if I can write a simple script that scrapes the periodic job hashes and formats an openstack/releases patch
14:15:56 <shardy> slagle: Yeah, I was wondering if we could do that
14:16:01 <trown> we could definitely create a symlink on DLRN server
14:16:17 <trown> I have been wanting that for RDO test days for a while
14:16:24 <shardy> So, first job is get periodic job passing, trown kindly offered to help there
14:16:36 <trown> quickstart images made it less of an issue, but would still be nice to have
14:17:13 <derekh> shardy: your script to look for the hash, can just look in this file for the hash of the last passing periodic jobs http://66.187.229.99/builds/current-tripleo/metadata.txt
14:17:33 <derekh> shardy: don't ask me about the duplicate entries, I thought I fixed that, will look again
14:17:38 <shardy> one nit is I'm actually on PTO next week (bad timing!), so I may need a volunteer to drive the release script or I'll do it when I return on 6th June
14:17:54 <EmilienM> I can help on that
14:18:02 <EmilienM> I'm doing the same thing for our Puppet modules
14:18:04 <slagle> shardy: yea, ping me if you want help
14:18:08 <jistr> how long are the hashes kept? If we symlink it, would the target expire at some point?
14:18:17 <shardy> derekh: yup, thanks - then we just need to wire that in to generate the releases
14:18:26 <shardy> slagle, EmilienM: thanks, will do!
14:19:06 <shardy> One thing related to stable backports
14:19:17 <shardy> the liberty upgrades job - anyone looked into why pingtest is broken?
14:19:36 <trown> jistr: I will bring it up on RDO meeting tomorrow, I think we can get permanent milestone repos on DLRN
14:21:13 <shardy> https://bugs.launchpad.net/tripleo/+bug/1545318 is the bug I think
14:21:14 <openstack> Launchpad bug 1545318 in tripleo "stable/liberty CI failing on HA job pingtest with 503 Service Unavailable" [Critical,Triaged] - Assigned to James Slagle (james-slagle)
14:21:29 <jistr> shardy: we never made it pass IIRC. I've tried deploying liberty but it failed much earlier than that for me, haven't had bandwidth to pursue that further (got XFS inode malformed errors when ironic tried to provision the machines)
14:21:33 <shardy> although I thought there was another one related to the upgrades job
14:22:50 <shardy> Ok, lets take the discussion to #tripleo after I guess
14:23:00 <shardy> it'd be good to understand what's needed to fix it
14:23:30 <shardy> Anything else related to backports or releases to cover?
14:23:46 <shardy> I saw some discussion re the aodh upgrade plan, sounds like the non-backport approach is workable?
14:23:50 <shardy> pradk: ^^
14:24:25 <pradk> shardy, yea so far it seems workable
14:24:44 <shardy> Ok, that's good news :)
14:24:46 <pradk> shardy, i have a patch for review https://review.openstack.org/#/c/320150/ just need to wire the aodh upgrade template into upgrade flow
14:24:52 <pradk> and see how it all works out
14:25:22 <shardy> ack, sounds good, thanks for the update
14:25:49 <shardy> perhaps jistr can take a look and provide feedback on how best to integrate it into the current upgrade flow
14:25:54 <pradk> yea i'll work with jistr in getting that wired
14:26:19 <shardy> #topic CI
14:26:21 <jistr> yea. Current upgrade flow is docs. So we need one more step in the docs :)
14:26:46 <shardy> jistr: noted :)
14:27:06 <shardy> So, modulo the CI breaking bugs, things have been going pretty well with the upgraded CI?
14:27:25 <shardy> derekh: do you have anything to raise re CI?
14:27:50 <derekh> Nothing on CI from me except to say I started yesterday deploying on a new rack to switch our CI(a subset of it) onto, in 6 weeks while the other rack is down
14:28:20 <shardy> derekh: So is the plan to switch one job to the new rack, so we maintain some coverage while the old rack is moved?
14:28:58 <derekh> shardy: yes, we had been talking about moveing everything to rdo cloud but it looks like it wont have the capacity available for us
14:29:07 <derekh> shardy: so we're back to the origional plan
14:29:41 <shardy> derekh: Ok - I guess we could still maintain some confidence with e.g the upgrades job and some manual testing
14:29:56 <derekh> shardy: if anybody had cycle we could also look into making more use of the infra nodes
14:30:02 <derekh> shardy: yup
14:30:06 <derekh> *cycles
14:30:36 <coolsvap> derekh, shardy i am looking forward to work on tripleo ci if someone could mentor initially
14:30:48 <shardy> I noticed some nonha jobs are getting close to ~1hr - did anyone look into the comments by pabelanger re devstack/zuul-cloner?
14:30:58 <shardy> sounds like that may buy use a few more minutes
14:31:04 <shardy> s/use/us
14:31:59 <shardy> coolsvap: good news!
14:32:11 <derekh> shardy: also around the same time we did the HW upgrade we started using the overcloud cached images, so I think both things together have helped a lot
14:33:13 <derekh> shardy: ya, looks like using zuul-cloner directly might save us more  time, not sure if pabelanger has started looking at it yet
14:33:14 <shardy> derekh: Yeah, it's definitely improved things a quite a bit :)
14:33:22 <ccamacho> coolsvap, managed to have your CI working?
14:33:51 <coolsvap> ccamacho, its in progress after i applied the fix for pip-virtualenv***
14:34:03 <shardy> Ok, anything else to raise re CI this week?
14:34:49 <shardy> #topic Specs
14:34:58 <shardy> #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open
14:36:00 <shardy> https://review.openstack.org/#/c/319264/2/specs/newton/overcloud-upgrades-per-service.rst needs reviewers
14:36:36 <shardy> I'm particularly interested to explore the requirements around Mistral *and* Ansible there
14:36:56 <shardy> e.g if we do that how do we ensure a clean split
14:37:21 <shardy> the sriov and dpdk ones also need further feedback
14:37:30 <shardy> anyone have anything else to raise re specs?
14:37:34 <rhallisey> shardy, I was thinking of proposing a kubernetes-controller spec to compliment the already containerized compute node
14:38:00 <rhallisey> would maybe be a poc at first, but I could get it in writing in case people were interested in reading about it
14:38:09 <shardy> rhallisey: Ok, sounds like something that would probably benefit from some initial prototyping?
14:38:45 <rhallisey> ya it still needs some more code laid down, but I figured I would mention it
14:38:46 <shardy> rhallisey: +1 on a poc, I think that will be hard to discuss meaningfully purely in the spec, but it would provide useful context I'm sure
14:39:22 <rhallisey> shardy, cool sounds good
14:39:40 <ccamacho> coolsvap, ack
14:39:51 <shardy> Ok, well we got through the regular agenda fairly fast today - did I miss anything? :)
14:40:10 <shardy> #topic Open Discussion
14:40:16 <beagles> I started drafting a spec on neutron DVR late last week and should be putting up for initial review this week. So kind of premature mentione really ;)
14:40:54 <beagles> I figured it would be get to get out in the open soonest so everyone can have an opportunity to see what kind of thing will be required there
14:41:07 <shardy> beagles: ack, thanks - and thanks for the reviews of sriov/dpdk
14:42:25 <shardy> Ok, anyone have anything else to raise?
14:42:28 <beagles> shardy: np... need to re-review :) good to get some other eyes on them to see what other kinds of things are required there.
14:42:33 * shardy waits 2 minutes
14:44:12 <shardy> Ok, I guess we can end the meeting early today then, thanks everyone!
14:44:25 <jistr> o/
14:44:29 <shardy> #endmeeting