14:00:16 #startmeeting tripleo 14:00:22 Meeting started Tue May 24 14:00:16 2016 UTC and is due to finish in 60 minutes. The chair is shardy. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:23 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:26 The meeting name has been set to 'tripleo' 14:00:29 o/ 14:00:30 #topic rollcall 14:00:30 o/ 14:00:35 Hey all, who's around? 14:00:37 \o 14:00:37 [T]/ 14:00:47 o/ 14:00:58 o/ 14:01:50 #topic agenda 14:01:55 * derekh wonders if the left handed people are using different slashes to the right handed people 14:01:59 o/ 14:02:03 #link https://wiki.openstack.org/wiki/Meetings/TripleO 14:02:16 * one off agenda items 14:02:16 * bugs 14:02:16 * Projects releases or stable backports 14:02:16 * CI 14:02:16 * Specs 14:02:18 * open discussion 14:02:24 Anyone have any one-off items to add today? 14:03:10 #topic one off agenda items 14:03:22 #info Request for mentoring : http://lists.openstack.org/pipermail/openstack-dev/2016-May/095667.html 14:03:35 So this was just to draw attention to a request for mentoring on the ML 14:03:47 * beagles joins a bit late 14:03:58 o/ 14:04:05 shardy: i saw that and was wondering if it meant that there has been a specific request for tripleo mentoring? 14:04:07 TripleO is listed as a project with potential need for mentors, so feel free to step up if you're interested in doing that 14:04:18 oh neat 14:04:45 slagle: tbh I'm not sure - I saw TripleO on the list and thought it'd be good if at least a couple of folks stepped up to find out ;) 14:05:05 it may be that it's just missing coverage from the pool of mentors I guess 14:05:42 I will volunteer for that, and can report back :) 14:05:51 anyways, let me know if anyone does so and needs help e.g finding low-hanging-fruit bugs or introductory materials 14:05:56 trown: thanks! 14:06:08 #topic bugs 14:06:24 https://bugs.launchpad.net/tripleo/ 14:06:30 #link https://bugs.launchpad.net/tripleo/ 14:06:52 So we got bitten by an issue with hiera yesterday, which is now fixed via RDO packaging 14:07:09 and there's still an outstanding issue re pip-and-virtualenv with a patch due to land 14:07:23 anything else to raise, in particular impacting CI? 14:07:35 i think we can land the pip and venv fix now 14:07:39 trown: have you looked lately at the reasons why the periodic job is failing? 14:07:45 it has passed all jobs cumulatively 14:07:53 https://review.openstack.org/#/c/320019/ 14:07:55 slagle: +1 - I just saw that all jobs passed over $rechecks 14:08:21 i just merged it :) 14:08:28 thanks :) 14:08:29 shardy: not for a couple days no... a few other fires in RDO land 14:09:04 trown: Ok, the reason is related to releases - we're due to do an n-1 milestone release soon and we need a clean promote to get the good sha's from 14:09:18 One more thing related to bugs: 14:09:21 " 326 Open bugs " 14:09:33 I noticed there's a lot of Fix Commited bugs making that number larger 14:09:58 when we cut releases for all-the-things for n-1, I'd like to run a script over the tripleo bugs and flip all Fix Committed to Fix Released 14:10:05 shardy: ok, we have a promote in RDO from a few days ago, but I will look into what is wrong on master now that we got hiera sorted 14:10:05 does that sound OK to people? 14:10:16 trown: that would be awesome, thanks! 14:10:18 +1 shardy 14:10:32 +1 14:10:44 it'd be great to get the open bugs back to a somewhat manageable level, and I think that should help a lot 14:11:31 Ok, any other bugs to highlight? 14:12:10 #topic Projects releases or stable backports 14:12:38 So, as I mentioned, n-1 is due on or around next Friday 14:12:45 #link http://releases.openstack.org/newton/schedule.html 14:13:25 I know historically we've not done milestone releases, but I wanted to propose a patch to openstack/releases next time we get a clean periodic promote so we've got a coordinated milestone release 14:13:46 I guess folks can consume it either via the package versions, or via the delorean hash from the periodic promote? 14:14:19 obviously we might automate that via quickstart in future, but any objections to just getting things started by cutting some releases around the milestone? 14:14:30 +1 to milestone releases 14:14:59 sounds good to me 14:15:00 +1 would be nice to get more towards continuous releasing if possible 14:15:13 Ok, sounds good then 14:15:18 maybe we could even create a symlink to the repo on the delorean server 14:15:18 I think we could only consume them via DLRN for now (for packages) 14:15:44 I'll see if I can write a simple script that scrapes the periodic job hashes and formats an openstack/releases patch 14:15:56 slagle: Yeah, I was wondering if we could do that 14:16:01 we could definitely create a symlink on DLRN server 14:16:17 I have been wanting that for RDO test days for a while 14:16:24 So, first job is get periodic job passing, trown kindly offered to help there 14:16:36 quickstart images made it less of an issue, but would still be nice to have 14:17:13 shardy: your script to look for the hash, can just look in this file for the hash of the last passing periodic jobs http://66.187.229.99/builds/current-tripleo/metadata.txt 14:17:33 shardy: don't ask me about the duplicate entries, I thought I fixed that, will look again 14:17:38 one nit is I'm actually on PTO next week (bad timing!), so I may need a volunteer to drive the release script or I'll do it when I return on 6th June 14:17:54 I can help on that 14:18:02 I'm doing the same thing for our Puppet modules 14:18:04 shardy: yea, ping me if you want help 14:18:08 how long are the hashes kept? If we symlink it, would the target expire at some point? 14:18:17 derekh: yup, thanks - then we just need to wire that in to generate the releases 14:18:26 slagle, EmilienM: thanks, will do! 14:19:06 One thing related to stable backports 14:19:17 the liberty upgrades job - anyone looked into why pingtest is broken? 14:19:36 jistr: I will bring it up on RDO meeting tomorrow, I think we can get permanent milestone repos on DLRN 14:21:13 https://bugs.launchpad.net/tripleo/+bug/1545318 is the bug I think 14:21:14 Launchpad bug 1545318 in tripleo "stable/liberty CI failing on HA job pingtest with 503 Service Unavailable" [Critical,Triaged] - Assigned to James Slagle (james-slagle) 14:21:29 shardy: we never made it pass IIRC. I've tried deploying liberty but it failed much earlier than that for me, haven't had bandwidth to pursue that further (got XFS inode malformed errors when ironic tried to provision the machines) 14:21:33 although I thought there was another one related to the upgrades job 14:22:50 Ok, lets take the discussion to #tripleo after I guess 14:23:00 it'd be good to understand what's needed to fix it 14:23:30 Anything else related to backports or releases to cover? 14:23:46 I saw some discussion re the aodh upgrade plan, sounds like the non-backport approach is workable? 14:23:50 pradk: ^^ 14:24:25 shardy, yea so far it seems workable 14:24:44 Ok, that's good news :) 14:24:46 shardy, i have a patch for review https://review.openstack.org/#/c/320150/ just need to wire the aodh upgrade template into upgrade flow 14:24:52 and see how it all works out 14:25:22 ack, sounds good, thanks for the update 14:25:49 perhaps jistr can take a look and provide feedback on how best to integrate it into the current upgrade flow 14:25:54 yea i'll work with jistr in getting that wired 14:26:19 #topic CI 14:26:21 yea. Current upgrade flow is docs. So we need one more step in the docs :) 14:26:46 jistr: noted :) 14:27:06 So, modulo the CI breaking bugs, things have been going pretty well with the upgraded CI? 14:27:25 derekh: do you have anything to raise re CI? 14:27:50 Nothing on CI from me except to say I started yesterday deploying on a new rack to switch our CI(a subset of it) onto, in 6 weeks while the other rack is down 14:28:20 derekh: So is the plan to switch one job to the new rack, so we maintain some coverage while the old rack is moved? 14:28:58 shardy: yes, we had been talking about moveing everything to rdo cloud but it looks like it wont have the capacity available for us 14:29:07 shardy: so we're back to the origional plan 14:29:41 derekh: Ok - I guess we could still maintain some confidence with e.g the upgrades job and some manual testing 14:29:56 shardy: if anybody had cycle we could also look into making more use of the infra nodes 14:30:02 shardy: yup 14:30:06 *cycles 14:30:36 derekh, shardy i am looking forward to work on tripleo ci if someone could mentor initially 14:30:48 I noticed some nonha jobs are getting close to ~1hr - did anyone look into the comments by pabelanger re devstack/zuul-cloner? 14:30:58 sounds like that may buy use a few more minutes 14:31:04 s/use/us 14:31:59 coolsvap: good news! 14:32:11 shardy: also around the same time we did the HW upgrade we started using the overcloud cached images, so I think both things together have helped a lot 14:33:13 shardy: ya, looks like using zuul-cloner directly might save us more time, not sure if pabelanger has started looking at it yet 14:33:14 derekh: Yeah, it's definitely improved things a quite a bit :) 14:33:22 coolsvap, managed to have your CI working? 14:33:51 ccamacho, its in progress after i applied the fix for pip-virtualenv*** 14:34:03 Ok, anything else to raise re CI this week? 14:34:49 #topic Specs 14:34:58 #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open 14:36:00 https://review.openstack.org/#/c/319264/2/specs/newton/overcloud-upgrades-per-service.rst needs reviewers 14:36:36 I'm particularly interested to explore the requirements around Mistral *and* Ansible there 14:36:56 e.g if we do that how do we ensure a clean split 14:37:21 the sriov and dpdk ones also need further feedback 14:37:30 anyone have anything else to raise re specs? 14:37:34 shardy, I was thinking of proposing a kubernetes-controller spec to compliment the already containerized compute node 14:38:00 would maybe be a poc at first, but I could get it in writing in case people were interested in reading about it 14:38:09 rhallisey: Ok, sounds like something that would probably benefit from some initial prototyping? 14:38:45 ya it still needs some more code laid down, but I figured I would mention it 14:38:46 rhallisey: +1 on a poc, I think that will be hard to discuss meaningfully purely in the spec, but it would provide useful context I'm sure 14:39:22 shardy, cool sounds good 14:39:40 coolsvap, ack 14:39:51 Ok, well we got through the regular agenda fairly fast today - did I miss anything? :) 14:40:10 #topic Open Discussion 14:40:16 I started drafting a spec on neutron DVR late last week and should be putting up for initial review this week. So kind of premature mentione really ;) 14:40:54 I figured it would be get to get out in the open soonest so everyone can have an opportunity to see what kind of thing will be required there 14:41:07 beagles: ack, thanks - and thanks for the reviews of sriov/dpdk 14:42:25 Ok, anyone have anything else to raise? 14:42:28 shardy: np... need to re-review :) good to get some other eyes on them to see what other kinds of things are required there. 14:42:33 * shardy waits 2 minutes 14:44:12 Ok, I guess we can end the meeting early today then, thanks everyone! 14:44:25 o/ 14:44:29 #endmeeting