19:04:04 <lifeless> #startmeeting tripleo 19:04:05 <openstack> Meeting started Tue Jul 1 19:04:04 2014 UTC and is due to finish in 60 minutes. The chair is lifeless. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:04:06 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:04:09 <openstack> The meeting name has been set to 'tripleo' 19:04:12 <derekh_> hi 19:04:51 <bnemec> o/ 19:05:15 <gfidente> o/ 19:05:20 <pblaho> o/ 19:05:39 <greghaynes> O/ 19:06:16 <tchaypo> morning 19:06:31 <lifeless> #topic agenda 19:06:39 <lifeless> bugs 19:06:40 <lifeless> reviews 19:06:40 <lifeless> Projects needing releases 19:06:40 <lifeless> CD Cloud status 19:06:40 <lifeless> CI 19:06:42 <lifeless> Tuskar 19:06:44 <lifeless> Specs 19:06:47 <lifeless> Insert one-off agenda items here 19:06:49 <lifeless> open discussion 19:06:54 <lifeless> #topic bugs 19:07:02 <lifeless> #link https://bugs.launchpad.net/tripleo/ 19:07:02 <lifeless> #link https://bugs.launchpad.net/diskimage-builder/ 19:07:02 <lifeless> #link https://bugs.launchpad.net/os-refresh-config 19:07:02 <lifeless> #link https://bugs.launchpad.net/os-apply-config 19:07:02 <lifeless> #link https://bugs.launchpad.net/os-collect-config 19:07:04 <lifeless> #link https://bugs.launchpad.net/os-cloud-config 19:07:07 <lifeless> #link https://bugs.launchpad.net/tuskar 19:07:09 <lifeless> #link https://bugs.launchpad.net/python-tuskarclient 19:07:16 <tchaypo> This is my weekly reminder that anyone can use the link and info commands, not just lifeless - if you have something worth noting in the meeting minutes feel freeo to tag it 19:07:33 <lifeless> tchaypo: you could put that in the stuff I copy-paste :) 19:07:57 <tchaypo> I approve of this lazinesss 19:08:23 * jdob resists temptation to make a note about our ability to make notes 19:09:13 <lifeless> so we have some untriaged I see 19:09:26 <lifeless> and still crits 19:09:41 <lifeless> is michael kerrin online for the meeting? bug 1263294 19:09:42 <uvirtbot> Launchpad bug 1263294 in tripleo "ephemeral0 of /dev/sda1 triggers 'did not find entry for sda1 in /sys/block'" [Critical,In progress] https://launchpad.net/bugs/1263294 19:11:33 <lifeless> derekh_: how is 1308407 looking ? 19:12:17 <lifeless> derekh_: I believe your patch has been superceded? Does the new review reference the bug ? 19:12:33 <lifeless> GheRivero_afk: how is bug 1316985 coming ? 19:12:34 <uvirtbot> Launchpad bug 1316985 in tripleo "set -eu may spuriously break dkms module" [Critical,In progress] https://launchpad.net/bugs/1316985 19:13:12 <lifeless> no sign of rpod online - anyone know the status of bug 1317056 ? 19:13:17 <uvirtbot> Launchpad bug 1317056 in tripleo "Guest VM FS corruption after compute host reboot" [Critical,Triaged] https://launchpad.net/bugs/1317056 19:13:24 <lifeless> and lastly bug 1319979 also GheRivero_afk 19:13:25 <uvirtbot> Launchpad bug 1319979 in tripleo "rabbitmq-server fails to restart in check-tripleo-overcloud-precise" [Critical,Triaged] https://launchpad.net/bugs/1319979 19:13:58 <lifeless> I think the last one is invalid, since that job has happily passed many times since 19:14:02 <derekh_> lifeless: I added a reference to the review a few minutes ago, going to review it after meeting 19:14:05 <lifeless> we used it to debug neutron the other day 19:14:49 <lifeless> what else 19:14:58 <lifeless> bug 1329068 - slicknick was on that 19:14:58 <uvirtbot> Launchpad bug 1329068 in diskimage-builder "The trove guest-image is unable to publish the guest-config via cloud-init" [Critical,In progress] https://launchpad.net/bugs/1329068 19:15:00 <lifeless> but its still open 19:15:34 <tchaypo> I've added Closes-Bug: #1308407 to the new patchset 19:16:13 <tchaypo> via the slick new in-browser commit message editor 19:16:34 <lifeless> in 1329068 there is a fix merged to dib 19:16:46 <lifeless> I think we can close it now? SpamapS - do you know ? 19:17:00 <SpamapS> lifeless: checking 19:17:14 <SpamapS> lifeless: yeah I thought that was closed already. 19:17:42 <clarkb> tchaypo: boo 19:17:44 <bnemec> It should have - the second patch is tagged Closes-Bug. 19:17:47 <clarkb> tchaypo: you reset all the votes 19:17:48 <SpamapS> ahh yeah it's committed 19:17:53 <SpamapS> lifeless: they used partial intead of closes 19:18:17 <bnemec> Oh, the second one is trove-integration. 19:18:20 <tchaypo> clarkb: it wasn't smart enough to see that as a minor "only commit message was changed" commit? 19:18:33 <clarkb> tchaypo: no beacuse we actually test commit messages 19:18:37 <clarkb> tchaypo: for better or worse 19:19:11 <tchaypo> clarkb: to make sure they have periods in the right places and so on? 19:19:17 <clarkb> yes 19:19:21 <lifeless> stabbity stab stab 19:19:24 <tchaypo> my apologies 19:19:59 <lifeless> stabbity at the 'or worse', not you. 19:20:11 <SpamapS> lifeless: bug 1329068 was released in 0.1.21 :) 19:20:12 <uvirtbot> Launchpad bug 1329068 in diskimage-builder "The trove guest-image is unable to publish the guest-config via cloud-init" [Critical,In progress] https://launchpad.net/bugs/1329068 19:20:15 * SpamapS marks as such 19:20:24 <lifeless> ok thats all the bugs 19:20:57 <lifeless> though looking at the untriaged reminded me to collar ironic folk about removing the flavor integration :( 19:21:26 <lifeless> #topic reviews 19:21:38 <lifeless> #info There's a new dashboard linked from https://wiki.openstack.org/wiki/TripleO#Review_team - look for "TripleO Inbox Dashboard" 19:21:40 <lifeless> #link http://russellbryant.net/openstack-stats/tripleo-openreviews.html 19:21:43 <lifeless> #link http://russellbryant.net/openstack-stats/tripleo-reviewers-30.txt 19:21:46 <lifeless> #link http://russellbryant.net/openstack-stats/tripleo-reviewers-90.txt 19:22:00 <lifeless> Stats since the last revision without -1 or -2 : 19:22:00 <lifeless> Average wait time: 9 days, 10 hours, 55 minutes 19:22:00 <lifeless> 1st quartile wait time: 1 days, 6 hours, 51 minutes 19:22:00 <lifeless> Median wait time: 5 days, 18 hours, 17 minutes 19:22:00 <lifeless> 3rd quartile wait time: 13 days, 11 hours, 45 minutes 19:22:05 <lifeless> we've crept back up again 19:22:31 <lifeless> Looking at the baseline stats 19:22:40 <lifeless> *lots* of our cores are not getting 3/day. 19:22:44 <gfidente> just in case, sources for the gerrit dashboard linked from the wiki are here: https://github.com/stackforge/gerrit-dash-creator/blob/master/dashboards/tripleo.dash 19:23:09 <lifeless> #info REMINDER: Core reviewer commitment was 3 a day, rain or shine. 19:23:18 <lifeless> #link http://russellbryant.net/openstack-stats/tripleo-reviewers-30.txt 19:24:03 <tchaypo> I was looking at the stats and I noticed we seem to have lots of reviews that have just one +1 19:24:29 <tchaypo> ego https://review.openstack.org/#/c/86316/ which is 55 days since last revision, has had no negative feedback, but needs some +2s to proceed 19:24:41 <greghaynes> tchaypo: that one was just recently restored 19:24:46 <greghaynes> seems stats dont take that into account 19:25:02 <SpamapS> lifeless: Been pulled off on Heat work and also there are vacations happening around this time of year... so that throws it off. Anyway, for me I hope to slam the queue tomorrow. 19:25:38 <tchaypo> That was the case with some others that I saw too - they'd been auto-abandoned, then restored, then #crickets 19:25:56 <lifeless> SpamapS: 64 is what you get over a 30 calendar day period with 5 day work week 19:25:59 <gfidente> the dashboard actually has a nice section titled "No Negative Feedback" which should be handy for changes needing only +2 19:26:11 * gfidente encourages using the dashboard :) 19:26:18 <lifeless> SpamapS: you're over that - so thanks for the apology, but don't stress. We have folk below that by a significant marging. 19:26:44 <lifeless> margin. 19:26:51 <tchaypo> gfidente: I reckon that's worth a #info 19:26:56 <lifeless> Does anyone have specific reviews they want to call out for help or attention on ? 19:27:09 <gfidente> tchaypo, lifeless did it already :P 19:27:33 <SpamapS> lifeless: oh I aim for 90 :) weekends? what are weekends? ;-) 19:27:47 <SpamapS> actually I have family commitments I need to attend to ATM ... 19:27:54 <SpamapS> anybody need anything from me before I disappear? 19:28:07 <greghaynes> SpamapS: input on https://review.openstack.org/#/c/98709/ at some point 19:28:08 <tchaypo> oh yeah - because I put it into the paste stuff last week. 19:28:14 <tchaypo> total win for laziness! 19:28:25 <lifeless> SpamapS: not I, though -specs review please :) 19:28:30 <pblaho> can someone point me to place where I can discover diiference http://www.stackalytics.com/report/contribution/tripleo-group/90 vs. http://russellbryant.net/openstack-stats/tripleo-reviewers-90.txt 19:28:41 <tchaypo> lifeless: Are you planning to do a stats run to look for new cores soon? 19:29:11 <lifeless> tchaypo: every day this last week, but sob its hard. Also its not a stats run - the stats is like 30 seonds. Its a systematic metareview 19:30:03 <tchaypo> lifeless: not even that, the stats are on russellbryant.net before you even think about wanting them.. 19:30:06 <lifeless> tchaypo: there's been lots of knee jerk reaction to the very idea of stats being involved, so I'm trying to avoid any suggestion that the stats dominates; they never did... 19:30:19 <tchaypo> <3 19:30:35 <lifeless> pblaho: well, other than them being different? DIfferent code bases. 19:31:04 <pblaho> lifeless: :-) 19:31:04 <tchaypo> pblaho: apparently not 19:31:22 <tchaypo> gfidente: btw, how's the plan to get a hosted .html of those links going? 19:31:32 <SpamapS> ACK, o/ 19:31:40 <gfidente> gfidente, sorry but didn't look into that :( 19:31:41 <lifeless> ok, next topic time 19:31:48 <gfidente> tchaypo, ^^ .( 19:31:53 <lifeless> #projects needing releases 19:32:00 <lifeless> do we have a victim^Wvolunteer 19:32:01 <jdob> i will do it. i will bring the ring to Mordor 19:32:06 <lifeless> \o/ 19:32:13 <lifeless> #action jdob to release the world 19:32:21 <jdob> i'm not gonna lie, part of me volunteering was just so I could say that 19:32:26 <bnemec> One does not simply release into Mordor... 19:32:39 <tchaypo> gfidente: ah, cool 19:32:40 <lifeless> lol, grin 19:32:44 <lifeless> thats made my day 19:32:48 <lifeless> #topic CD cloud status 19:32:57 <lifeless> RH seems up - any specifics to add dprince / derekh_ ? 19:33:28 <greghaynes> Seed memory issues 19:33:29 <derekh_> lifeless: one of the TE's was failing all overcloud jobs 19:33:33 <dprince> lifeless; we got 10 more nodes today 19:33:43 <lifeless> dprince: \o/ fantastic 19:34:03 <dprince> lifeless: had a minor issues to fix because there was an IP address collision (the new DRAC's were in our undercloud range). 19:34:10 <greghaynes> I keep seeing fails like https://launchpad.net/bugs/1336479 19:34:12 <dprince> lifeless: but we've resolved that 19:34:12 <lifeless> greghaynes: righto - we still need to rebuild them; we need to make the seed size honour the node size request - I noted my local seed vm was 2G when my nodes were 3G just before 19:34:13 <uvirtbot> Launchpad bug 1336479 in tripleo "CI build failing due to insufficient memory on seed" [High,Confirmed] 19:34:16 <derekh_> lifeless: I've killed it so should bring stats up a little https://bugs.launchpad.net/tripleo/+bug/1335926 19:34:17 <uvirtbot> Launchpad bug 1335926 in tripleo "Can't destroy libvirt domain on TE host" [High,Confirmed] 19:35:02 <derekh_> Wi'll rebuild the TE host later to get bac to full capacity 19:35:21 <lifeless> dprince: are they online now or pending deploy ? 19:35:32 <derekh_> then we gotta decide when to being the new hosts into the overcloud 19:35:47 <derekh_> lifeless: their sitting there waiting for us to use them 19:35:58 <dprince> lifeless: ^^ yep. don't rush us. Just saying we got access to them. We still need to do some burn in on them I think 19:36:07 <lifeless> cool 19:36:29 <lifeless> we need to get stevenk's fixes to toci's environment creation in too 19:36:34 <lifeless> to unlock the better Ironic API 19:36:35 <dprince> lifeless: our last round of hardware was burned-in for us. This time they pretty much just racked them I think 19:36:51 <lifeless> and we need to fix the seed sizing thing one way or another 19:36:52 <lifeless> dprince: cool 19:37:00 <derekh_> lifeless: link to the fix ? 19:37:01 <lifeless> ok so hp1 and hp2 regions 19:37:39 <lifeless> derekh_: looks to me like he analysed it and hasn't pushed it 19:37:47 <derekh_> lifeless: ok 19:37:50 <lifeless> derekh_: basically the ssh key isn't included in the per-node metadata. 19:38:09 <lifeless> derekh_: so we can't go from per-node metadata to ironic credentials field and drop the file injection 19:38:26 <lifeless> derekh_: which would mean we have /no/ difference between bm and vm for ironic, so one less template variation. 19:38:26 <derekh_> lifeless: ok will take a look once its up 19:38:42 <lifeless> #action stevenk to push up toci fix for ssh creds in nodes list 19:39:04 <lifeless> hp1 and hp2 are still down; we have the bastion on both and a uc on hp1; Ng - did you get a uc up on hp2 ? 19:39:34 <lifeless> lifeless, tchaypo and derekh_ are working on the vlan functionality re-fit to let us bring them up. hp1's machines are all vetted for config - 46 of em 19:39:54 <lifeless> hp2 we haven't commenced burnin / validation yet. 19:39:55 <derekh_> lifeless: I push a minor fix for your ensure-bridge patches today and verified the MAC thing is working, didn't get to the vlan part but its next on the list 19:40:02 <Ng> lifeless: that's my focus for the rest of the day :) 19:40:14 <lifeless> Ng: cool 19:40:28 <lifeless> derekh_: thanks. I'll pull that down, then be on the vlan thing myself 19:40:43 <lifeless> #topic CI 19:40:49 <lifeless> derekh_: da floor is yours 19:41:31 <derekh_> dprince: mentioned a error happening on some jobs pinging the user instance , I could do with somebody looking into that 19:41:39 <derekh_> if not I'll take a look 19:42:21 <derekh_> about 5 to 10% of overcloud jobs were failing because of https://bugs.launchpad.net/tripleo/+bug/1335926 but that should be ok now 19:42:23 <uvirtbot> Launchpad bug 1335926 in tripleo "Can't destroy libvirt domain on TE host" [High,Confirmed] 19:42:31 <slagle> is there a bug or a link to a failed job? 19:42:37 <slagle> (for the user instance one) 19:42:58 <derekh_> not much on the ci spec has moved recently hopfully I'll get more time on it once we get the HP cloud up and running 19:43:14 <derekh_> slagle: I'll find one and paste it 19:44:15 <derekh_> slagle: this looks like it http://logs.openstack.org/09/98709/4/check-tripleo/check-tripleo-overcloud-f20/1d5fe2c/console.html 19:44:22 <slagle> k 19:45:06 <derekh_> although there is no garentee that not related to the patch 19:46:02 <derekh_> lifeless: thats all from me at the moment 19:46:42 <derekh_> slagle: opp, you said bug, I don't think there is one yet, will do 19:47:23 <lifeless> ok 19:47:24 <slagle> oh that's np, i just wanted to see a failed log 19:47:29 <lifeless> #topic tuskar 19:47:30 <slagle> but file away if you want :) 19:47:57 <jdob> nothing specific from me this week 19:48:06 <lifeless> #topic specs 19:48:48 <lifeless> so for HA 19:48:54 <lifeless> we need 103692 reviewed 19:49:00 <lifeless> turns out the PKI stuff is actually a little complex 19:49:24 <lifeless> there's a few with +2 and no cmoments for a bit that I'm going to +A this week 19:49:26 <lifeless> also 19:49:34 <lifeless> I'd like to call everyones attention to 19:49:42 <lifeless> #info backwards compat spec - please review 19:49:54 <tchaypo> lifeless: link? 19:49:58 <lifeless> #link https://review.openstack.org/102081 19:50:05 <lifeless> tchaypo: patience 19:50:29 <tchaypo> lifeless: well that's embarrassing 19:50:48 <lifeless> whoever logged into hp1 - use screen -xr so you don't detach everyone else :) 19:51:13 <lifeless> any other specs issues to raise? 19:52:55 <lifeless> #topic open discussion 19:53:01 <lifeless> I have a thing 19:53:05 <lifeless> the mid cycle is coming up 19:53:13 <lifeless> last cycle we did a mix of talking and doing 19:53:29 <lifeless> and one bit of feedback we received from the talkers was that having a broad agenda is useful 19:53:40 <lifeless> so if you have specific bits of face to face talking you want done,please put them in the etherpad 19:54:09 <lifeless> #info if you have agenda items for the mid-cycle please add to the agenda, a schedule for discussion-times will be forthcoming from those inputs 19:54:13 <derekh_> +1, I think we would have a lot to gain if we scheduled a few design summit like session 19:55:03 <slagle> good idea 19:56:19 <gfidente> #link https://etherpad.openstack.org/p/juno-midcycle-meetup 19:59:30 <lifeless> ok, seems like we're done here! thanks everyone. 20:00:04 <lifeless> #endmeeting