19:00:49 <slagle> #startmeeting tripleo 19:00:49 <openstack> Meeting started Tue Sep 8 19:00:49 2015 UTC and is due to finish in 60 minutes. The chair is slagle. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:50 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:00:53 <openstack> The meeting name has been set to 'tripleo' 19:00:55 <slagle> i'm sure someone will hate me for that 19:01:04 <derekh> phew I thought I was late 19:01:16 <akrivoka> hiya 19:01:17 <jdob> derekh: you were, you missed the entire first meeting 19:01:30 <slagle> yea, we're on to the 2nd meeting already 19:01:48 <slagle> #topic agenda 19:02:07 <slagle> I've added several one off items to the agenda 19:02:10 <derekh> jdob: slagle: caught up on the minutes 19:02:20 <slagle> lol 19:02:33 <jdob> which, oddly enough, describe a meeting that wasn't even a minute long 19:03:07 <slagle> let's fly through the regular agenda, then hit the one off stuff at the end, if there are no objections 19:03:22 <slagle> * bugs 19:03:22 <slagle> * reviews 19:03:22 <slagle> * Projects needing releases 19:03:22 <slagle> * CI 19:03:22 <slagle> * Specs 19:03:24 <slagle> * open discussion 19:03:32 <slagle> * one off items 19:03:53 <slagle> #topic bugs 19:04:14 <slagle> there are lots of untriaged bugs that i'll plan to go through this week 19:04:16 <dprince> slagle: we've been hitting a bug or two a day recently 19:04:40 <dprince> slagle: related to CI failures, a very bad time... feature freeze in OpenStack is 19:05:02 <slagle> dprince: which are the ones that people should be focusing on? 19:05:24 <derekh> Yup, we hit at least 6 or 7 bugs/regressions last week that have killed the ci jobs 19:05:51 <dprince> so someone needs to look at the recent Horizon issue: https://bugs.launchpad.net/tripleo/+bug/1492416 19:05:52 <openstack> Launchpad bug 1492416 in tripleo "Apache fails to start, .HorizonScssFilter: command not found" [High,Triaged] 19:06:18 <dprince> the top issue IMO is this one though https://bugs.launchpad.net/tripleo/+bug/1492028 19:06:19 <openstack> Launchpad bug 1492028 in tripleo "puppet overcloud deployment timeouts" [High,In progress] - Assigned to Dan Prince (dan-prince) 19:06:45 <dprince> We might need Steve bakers help this one 19:06:56 <derekh> The ceph jobs is unreliable and the HA jobs doesn't pass, part of me wants to just ignore those and move to the instack based jobs first 19:06:59 <dprince> lots of CI failures over the last few weeks related to this 19:07:04 <derekh> then see if anything needs to be dealt with 19:07:28 <dprince> derekh: do you have a cherry pick for this yet https://bugs.launchpad.net/tripleo/+bug/1493442 19:07:29 <openstack> Launchpad bug 1493442 in puppet-glance "puppet-glance: No such file or directory - /etc/glance/glance-api.conf" [High,In progress] - Assigned to Dan Prince (dan-prince) 19:07:32 <slagle> for https://bugs.launchpad.net/tripleo/+bug/1492028, has anyone been able to repro locally? 19:07:33 <openstack> Launchpad bug 1492028 in tripleo "puppet overcloud deployment timeouts" [High,In progress] - Assigned to Dan Prince (dan-prince) 19:07:52 <dprince> slagle: I did reproduce it once 19:08:02 <dprince> slagle: but I didn't have DEBUG enabled so it was a fail 19:08:18 <dprince> slagle: We do however have DEBUG CI logs so that should help somewhat 19:08:26 <derekh> dprince: nope, my test failed and I was going to retry but I've just noticed tripleo ci jobs aren't running for some reason, 19:08:39 <dprince> slagle: I have a feeling it is race related... perhaps a DB race condition in heat or something 19:08:56 <derekh> slagle: I've tried about 8 or 9 runs in the CI rack and havn't reproduced 19:09:07 <dprince> derekh: yeah, a Delorean failure now :) 19:09:17 <derekh> dprince: ya ? 19:09:33 <dprince> slagle: to bring this up to a higher level, we probably need more help on getting CI stable 19:09:52 <dprince> slagle: because issues are almost coming in faster than we fix them 19:10:08 <slagle> yea, agreed 19:10:41 <slagle> all i can do is ask around 19:10:44 <slagle> by ask, i mean beg 19:11:15 <dprince> derekh: yes, I saw one weird delorean failure. I'll post you a link 19:11:20 <derekh> dprince: this ? 2015-09-08 18:16:34.347 | FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel 19:11:33 <slagle> derekh: is the instack job testing with master of the puppet modules? 19:11:44 <dprince> derekh: yeah, that was it 19:11:47 <slagle> just wondering if we need the devtest puppet jobs at all anymore. dprince? 19:11:54 <derekh> dprince: that happens when jenkins looses connection to the instance 19:12:09 <derekh> slagle: yes it is 19:12:12 <slagle> that would help give folks one thing to focus on, vs. switching between the 2 to try and help 19:12:17 <derekh> slagle: the plan is to remove those jobs 19:12:31 <derekh> slagle: https://review.openstack.org/#/c/220088/ 19:12:33 <derekh> dprince: ^ 19:12:35 <dprince> yeah, can probably consolidate 19:12:41 <derekh> that moves the ceph job 19:13:08 <derekh> and passed earlier today, I was working on the ha job next but got hit by failures 19:13:26 <slagle> ok 19:13:55 <dprince> derekh: I'm going to push a DIB_REPOREF_puppet_glance fix for the one issue 19:13:58 <slagle> #info need more people to help out with CI failures 19:14:00 <derekh> dprince: I *think* those errors you mention with delorean are actually a problem with jenkins or our cloud 19:14:18 <slagle> #link https://bugs.launchpad.net/tripleo/+bug/1493442 19:14:19 <openstack> Launchpad bug 1493442 in puppet-glance "puppet-glance: No such file or directory - /etc/glance/glance-api.conf" [High,In progress] - Assigned to Dan Prince (dan-prince) 19:14:28 <slagle> #link https://bugs.launchpad.net/tripleo/+bug/1492416 19:14:28 <openstack> Launchpad bug 1492416 in tripleo "Apache fails to start, .HorizonScssFilter: command not found" [High,Triaged] 19:14:35 <slagle> #link https://bugs.launchpad.net/tripleo/+bug/1492028 19:14:37 <openstack> Launchpad bug 1492028 in tripleo "puppet overcloud deployment timeouts" [High,In progress] - Assigned to Dan Prince (dan-prince) 19:14:49 <dprince> one more bug I just hit 19:14:52 <derekh> dprince: ok, we need to talk to infra aswell to see why we currently only processing one job at a time 19:14:54 <dprince> #link https://bugs.launchpad.net/tripleo/+bug/1493502 19:14:55 <openstack> Launchpad bug 1493502 in tripleo "Failed to validate: resources.StorageMgmtNetwork: Value must be a string" [High,In progress] - Assigned to Dan Prince (dan-prince) 19:14:58 <derekh> dprince: look at the check-tripleo queue 19:15:16 <slagle> did the storagemgt patch merge? 19:15:28 <dprince> slagle: appoligies if you wanted a quick meeting, we've got plenty to talk about :) 19:16:05 <dprince> derekh: ack on the queue 19:16:21 <slagle> dprince: i'll look at https://bugs.launchpad.net/tripleo/+bug/1492416 19:16:22 <openstack> Launchpad bug 1492416 in tripleo "Apache fails to start, .HorizonScssFilter: command not found" [High,Triaged] - Assigned to James Slagle (james-slagle) 19:17:42 <slagle> alright, so are the 4 highest priority bugs? 19:17:47 <slagle> *those 19:18:25 <dprince> slagle: yeah, and without a Horizon fix we may well have to just disable Horizon if we need to bump the DELOREAN_URL 19:19:44 <bnemec> Looks like the Horizon backport of the fix is already proposed with 1 +2 though. 19:19:58 <bnemec> Might just be a question of poking the right person to get that merged. 19:20:57 <dprince> bnemec: nice, can you link that into the bug? 19:21:06 * dprince doesn't follow Horizon to closely 19:21:18 <bnemec> dprince: Yeah, it's in the related bz. One second. 19:21:47 <slagle> dprince: yea if we need to disable in the interim, fine by me 19:21:54 <dprince> slagle: did we talk about releases yet? 19:22:11 <david-lyle> dprince, bnemec: I can try to track down another +2, but any stable core should be able to approve, you have the Horizon +2 19:22:28 * david-lyle just gets pinged on horizon 19:22:33 <slagle> dprince: no. it's on the agenda 19:22:37 <dprince> david-lyle: great, thanks 19:23:06 <dprince> slagle: so briefly might I request a release of os-cloud-config, and the new tripleo client? 19:23:21 <dprince> slagle: only because using those in a venv is occasionally useful 19:23:42 <derekh> dprince: do we need to go through a renaming exercies of the tripleo client first ? 19:24:04 <slagle> dprince: let's talk about that when we get to releases 19:24:22 <dprince> derekh: http://git.openstack.org/cgit/openstack/python-tripleoclient/tree/ 19:24:33 <dprince> derekh: we do need to finish some work, yes 19:24:37 <dprince> slagle: sure, I'll be quiet 19:24:38 <slagle> is there any other bug business? 19:25:00 <dprince> not from me 19:25:02 <slagle> #topic reviews 19:25:20 <slagle> dprince: i'm sure you want to ask for reviews :) 19:25:48 <dprince> slagle: always :), but seriously I probably have rebasing to do first 19:26:16 <slagle> there are also going to be (i suspect) a small flood of reviews to get things fixed up in the new repos that got moved to git.openstack.org 19:26:27 <dprince> slagle: seriously there is some os-net-config stuff that could land though 19:26:39 <dprince> slagle: that would help the static ctlplane patch series along... 19:26:41 <slagle> so if folks could pay attention to that as well, it will help unblock stuff 19:26:51 <slagle> dprince: can you link them here? 19:27:04 <derekh> also we abandon a bunch of reviews for the stuff we no longer ci test ? mainly bash element stuff but that can come once things settle down a bit 19:27:10 <dprince> #link https://review.openstack.org/#/c/213351/ 19:27:12 <derekh> *we can 19:27:27 <dprince> slagle: we should try to fast track the stuff for new projects I think 19:27:48 <dprince> slagle: meaning nothing other than review them as top priority 19:28:00 <slagle> dprince: yea, that's what i was thinking, there's no CI running on them currently 19:28:22 <slagle> speaking of which, derekh, can you have a look at https://review.openstack.org/221457 19:28:37 <slagle> was trying to add the CI jobs to instack*/python-tripleoclient 19:29:01 <derekh> slagle: looking 19:29:23 <derekh> slagle: gate-tripleo-ironic-overcloud-f21-nonha is currently running on them 19:29:34 <bnemec> slagle: https://review.openstack.org/#/c/220167/ 19:29:37 <derekh> slagle: but ya, we should add the others 19:29:49 <slagle> oh is it? maybe it's just b/c the queue was stuck or something 19:30:03 <bnemec> Oh, that's a different one. 19:30:42 <derekh> slagle: yup, queue is stuck, dunno why only noticed befor this meeting 19:31:19 <derekh> slagle: we should merge you partch anyways, will review this evening 19:31:30 <slagle> derekh: i think it's fine to abandon the old reviews in combination with the deprecation of the bash elements 19:31:44 <derekh> slagle: ok 19:31:53 <slagle> which we should be able to do shortly 19:32:23 <slagle> anything else on reviews? 19:32:47 <slagle> #topic releases 19:33:16 <slagle> we can release all the things as needed 19:33:30 <slagle> i'm happy to do it 19:33:31 <dprince> sure, os-cloud-config would be helpful to release 19:33:54 <dprince> the "capabilities" stuff is helpful in a venv (to me at least) 19:33:57 <slagle> #info release python-tripleoclient after merge of https://review.openstack.org/#/c/221320/ 19:34:23 <slagle> #action slagle to do releases 19:34:48 <slagle> #topic CI 19:35:18 <dprince> nice work derekh on the instack jobs 19:35:20 <slagle> #info CI now running an instack based job 19:35:30 <slagle> yes, thanks to derekh :) 19:35:46 <derekh> thanks 19:35:59 <derekh> we should stop pulling the old version of instack and friends 19:36:00 <derekh> https://review.openstack.org/#/c/221474/ 19:36:11 <derekh> that pulls them from the openstack donain 19:36:33 <derekh> we can remove these altogether once this merges https://review.openstack.org/#/c/220551/ 19:36:44 <dprince> derekh: when CI passes you can send it :) 19:36:52 <derekh> dprince: ack 19:37:01 <dprince> derekh: or I will if I think to check it later 19:37:14 <derekh> Adding them to the projects list means we don't have to manually do it 19:37:46 <slagle> and you posted a patch for the HA and ceph jobs as well right? 19:39:32 <dprince> slagle: here is the patch series for the HA and ceph jobs https://review.openstack.org/#/c/220088/ 19:39:40 <slagle> cool 19:39:42 <derekh> slagle: yup, the ceph job passed earlier, the HA then failed on the glance regression 19:40:07 <slagle> #help CI needs more volunteers to help out 19:40:18 <derekh> slagle: those gets us off devtest altogether, we can then drop one of the nonha jobs 19:40:24 <bnemec> #link https://review.openstack.org/#/c/220088/ 19:40:28 <derekh> slagle: and remove a bunch of code from toci 19:41:12 <slagle> ok 19:41:16 <slagle> any other CI business? 19:41:33 <slagle> #topic specs 19:41:35 <derekh> nothing from me 19:41:40 <akrivoka> we have a spec up for a tripleo overcloud deployment library 19:41:55 * bnemec needs to look at that 19:41:56 <akrivoka> the idea is to formalize the deployment workflow and build a library to support it 19:42:05 <akrivoka> I encourage everyone to take a look and review 19:42:10 <akrivoka> #link https://review.openstack.org/#/c/219754 19:42:13 <slagle> #link https://review.openstack.org/219754 spec for overcloud deployment library 19:42:17 <slagle> :) 19:42:19 <akrivoka> :) 19:43:06 <dprince> akrivoka: nice, is this simliar to tripleoclient... but in a different format? 19:43:30 <dprince> akrivoka: I will just read the spec and then leave comments there 19:43:46 <slagle> it's building on tripleo-common 19:43:49 <akrivoka> dprince: the idea is to pull the business logic out of the tripleo client and put it in this library so that it can be reused by different consumer (CLI, GUI) 19:43:59 <bnemec> It's all the stuff in tripleoclient that's not supposed to be in tripleoclient. 19:44:10 <dprince> akrivoka: perfect 19:44:12 <akrivoka> and more! :) 19:44:35 <slagle> akrivoka: thanks for bringing it up, i'll have another look this week 19:44:46 <akrivoka> awesome, thanks 19:44:49 <slagle> the spec is not that long, so everyone please have a look 19:45:34 <slagle> #topic open discussion / one off items 19:45:55 <slagle> #link https://etherpad.openstack.org/p/tripleo-mitaka-proposed-sessions etherpad for design summit planning 19:46:09 <slagle> please add your ideas/proposals there 19:46:46 <slagle> if we get more than proposals than sessions (that would be a good thing), we can ask folks to rank them 19:46:51 <dprince> slagle: but you've already added mine ideas 19:47:13 <slagle> dprince: i did 19:47:20 <slagle> i'm sure you have more 19:47:36 <slagle> i added some other things too :) 19:47:49 <dprince> cool, looks like a good start 19:48:07 <dprince> this docker one could be fun 19:48:27 <slagle> also note that the kolla folks have offered to spend some time with tripleo during their contributor meetup 19:48:53 <slagle> so, we should plan on that, maybe some cross pollination after lunch on friday or something 19:49:04 <slagle> i think their meetup will be the 2nd half of friday 19:49:13 <dprince> slagle: yeah, apparently their PTL is buying all the kolla devs drinks 19:49:26 <sdake_> apparently ;) 19:49:29 <derekh> dprince: ya, PTL's always do that 19:49:30 <dprince> we should do that! 19:49:30 <bnemec> *hint, hint* 19:50:04 <slagle> i'll buy any devs a drink 19:50:08 <slagle> there i said it 19:50:18 <slagle> i hope water is cheap in tokyo 19:50:30 <bnemec> lol 19:50:56 <derekh> Water (0.33 liter bottle) 113.15 ¥ 19:51:21 <derekh> 113 Japanese Yen equals 19:51:21 <derekh> 0.9424 US Dollar 19:51:25 <derekh> you should be ok 19:51:32 <slagle> everything from "liter" and after is gibberish to me 19:51:58 <derekh> slagle: don't worry, give me your credit card and I'll figure out the hard stuff 19:52:27 <derekh> on another note, I'd like to start using trello again 19:52:40 <derekh> #link http://lists.openstack.org/pipermail/openstack-dev/2015-September/073861.html using trello 19:53:13 <derekh> read that ^^ I'm gonna move things around tomorrow on it and populate it and home others come join the party 19:53:38 <slagle> :), wfm 19:53:45 <slagle> i like the overall picture it gives 19:53:58 <dprince> derekh: I won't stop you 19:54:43 <dprince> derekh: that is as much encouragement as I could muster about this topic. Sorry :) 19:54:54 <derekh> I'm not saying we all need to keeps fine grained details up there but I think there is a happy medium beween that and nobody knowing what we are doing 19:55:05 <derekh> dprince: Thats for the effort 19:55:08 <derekh> *thanks 19:55:41 <slagle> yea, i think we can do better about letting folks know what we're working on 19:55:45 <slagle> well i know i can anyway 19:55:53 <slagle> even if it's just amongst oursevles 19:56:11 <slagle> but it's not a hold anyone to the fire "your work must be tracked in trello" or anything 19:56:31 <derekh> slagle: yup 19:57:17 <slagle> alright everyone, i appreciate the participation today :) 19:57:38 <dprince> rock on OoO 19:57:40 <slagle> if there's nothing else, i'll see you all in #tripleo 19:57:46 <akrivoka> \o 19:57:48 <jdob> \o 19:57:53 <derekh> o/ 19:57:55 <slagle> #endmeeting