19:05:11 <lifeless> #startmeeting tripleo 19:05:12 <openstack> Meeting started Tue Aug 26 19:05:11 2014 UTC and is due to finish in 60 minutes. The chair is lifeless. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:05:13 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:05:15 <openstack> The meeting name has been set to 'tripleo' 19:05:41 <bnemec> \o 19:07:00 <lifeless> #topic agenda 19:07:00 <lifeless> * bugs 19:07:00 <lifeless> * reviews 19:07:00 <lifeless> * Projects needing releases 19:07:00 <lifeless> * CD Cloud status 19:07:02 <lifeless> * CI 19:07:05 <lifeless> * Tuskar 19:07:07 <lifeless> * Specs 19:07:10 <lifeless> * open discussion 19:07:22 <lifeless> Remember that anyone can use the link and info commands, not just the moderator - if you have something worth noting in the meeting minutes feel free to tag it 19:07:26 <lifeless> #topic bugs 19:07:39 <lifeless> #link https://bugs.launchpad.net/tripleo/ 19:07:39 <lifeless> #link https://bugs.launchpad.net/diskimage-builder/ 19:07:39 <lifeless> #link https://bugs.launchpad.net/os-refresh-config 19:07:39 <lifeless> #link https://bugs.launchpad.net/os-apply-config 19:07:39 <lifeless> #link https://bugs.launchpad.net/os-collect-config 19:07:41 <lifeless> #link https://bugs.launchpad.net/os-cloud-config 19:07:44 <lifeless> #link https://bugs.launchpad.net/tuskar 19:07:46 <lifeless> #link https://bugs.launchpad.net/python-tuskarclient 19:08:14 <slagle> i don't think we have any untriaged bugs 19:08:31 <lifeless> we have a bunch of criticals still though 19:08:35 <lifeless> bug 1263294 19:08:36 <uvirtbot> Launchpad bug 1263294 in tripleo "ephemeral0 of /dev/sda1 triggers 'did not find entry for sda1 in /sys/block'" [Critical,In progress] https://launchpad.net/bugs/1263294 19:08:43 <lifeless> bug 1316985 19:08:43 <uvirtbot> Launchpad bug 1316985 in tripleo "set -eu may spuriously break dkms module" [Critical,In progress] https://launchpad.net/bugs/1316985 19:08:49 <lifeless> bug 1317056 19:08:50 <uvirtbot> Launchpad bug 1317056 in tripleo "Guest VM FS corruption after compute host reboot" [Critical,Triaged] https://launchpad.net/bugs/1317056 19:08:57 <lifeless> bug 1361235 19:08:59 <uvirtbot> Launchpad bug 1361235 in tripleo "visit horizon failure because of import module failure" [Critical,In progress] https://launchpad.net/bugs/1361235 19:09:24 <tchaypo> GheRivero: Do you have an update on https://launchpad.net/bugs/1316985 ? 19:09:25 <uvirtbot> Launchpad bug 1316985 in tripleo "set -eu may spuriously break dkms module" [Critical,In progress] 19:09:58 <GheRivero> I just need some love" https://review.openstack.org/#/c/104115/ 19:10:01 <slagle> i've put a patch for a workaround for 1361235 19:10:30 <slagle> needs some reviews, https://review.openstack.org/#/c/116721/ 19:10:46 <slagle> we could drop the bug to High...i was on the fence 19:11:12 <lifeless> broken builds are crit IMO here - if we were tempesting this it would have failed 19:11:34 <lifeless> boris-42: have you seen https://bugs.launchpad.net/tripleo/+bug/1361235 ? 19:11:35 <uvirtbot> Launchpad bug 1361235 in tripleo "visit horizon failure because of import module failure" [Critical,In progress] 19:12:00 <lifeless> #info please review https://review.openstack.org/#/c/116721/ (critical bugfix) 19:12:31 <lifeless> no sign of rpodolyka - hope he's ok 19:13:11 <tchaypo> #info GheRivero needs some reviews on https://review.openstack.org/#/c/104115/ (critical bugfix) 19:13:41 <lifeless> tchaypo: could you perhaps volunteer to ping michael and roman for the other two bugs ? 19:14:20 <tchaypo> lifeless: consider me volunteered 19:15:04 <lifeless> #info tchaypo will follow up on bug 1263294 and bug 1317056 with assignees 19:15:05 <uvirtbot> Launchpad bug 1263294 in tripleo "ephemeral0 of /dev/sda1 triggers 'did not find entry for sda1 in /sys/block'" [Critical,In progress] https://launchpad.net/bugs/1263294 19:15:07 <uvirtbot> Launchpad bug 1317056 in tripleo "Guest VM FS corruption after compute host reboot" [Critical,Triaged] https://launchpad.net/bugs/1317056 19:15:20 <lifeless> any other bug stuff to discuss? 19:16:09 <lifeless> #topic reviews 19:16:10 <lifeless> #info There's a new dashboard linked from https://wiki.openstack.org/wiki/TripleO#Review_team - look for "TripleO Inbox Dashboard" 19:16:13 <lifeless> #link http://russellbryant.net/openstack-stats/tripleo-openreviews.html 19:16:16 <lifeless> #link http://russellbryant.net/openstack-stats/tripleo-reviewers-30.txt 19:16:19 <lifeless> #link http://russellbryant.net/openstack-stats/tripleo-reviewers-90.txt 19:16:46 <lifeless> 19:16:46 <lifeless> Stats since the last revision without -1 or -2 : 19:16:46 <lifeless> Average wait time: 12 days, 9 hours, 42 minutes 19:16:46 <lifeless> 1st quartile wait time: 4 days, 23 hours, 21 minutes 19:16:46 <lifeless> Median wait time: 11 days, 4 hours, 30 minutes 19:16:49 <lifeless> 3rd quartile wait time: 18 days, 11 hours, 26 minutes 19:17:11 <tchaypo> Same trend as recently, and I don't think there's been much discussion on the list about what we're measuring being wrong 19:17:20 <tchaypo> s/wrong/not the thing we want to measure/ 19:17:31 <lifeless> You started a thread - thanks. I have it tagged to reply to 19:17:53 <lifeless> any pet reviews folk want to ask for help on? 19:19:00 <lifeless> I have three I'm going to beg for reviews on 19:19:22 <lifeless> https://review.openstack.org/#/c/102081/ - backwards compat spec 19:20:05 <lifeless> https://review.openstack.org/#/c/103454/ vlan support for the public virtual control IP 19:20:20 <tchaypo> linkify them? 19:20:33 <lifeless> https://review.openstack.org/#/c/103757/ - incubator support for 103454 19:20:41 <lifeless> tchaypo: sure 19:21:00 <lifeless> #info reviews sought on https://review.openstack.org/#/c/102081/ - backwards compat spec 19:21:01 <slagle> i'll re-reivew the backwards compat spec. been meaning to circle back to it 19:21:13 <lifeless> #info reviews sought on https://review.openstack.org/#/c/103454/ vlan support for the public virtual control IP 19:21:18 <alexislee> Not super important but it'd be nice to have: 19:21:24 <alexislee> #link https://review.openstack.org/#/c/82038/ 19:21:24 <lifeless> #info reviews sought on https://review.openstack.org/#/c/103757/ - incubator support for 103454 19:21:26 <alexislee> #link https://review.openstack.org/#/c/110366/ 19:22:03 <lifeless> #info reviews sought on https://review.openstack.org/#/c/82038/ - o-a-c meta support 19:22:56 <lifeless> #info reviews sought on https://review.openstack.org/#/c/110366/ - declarative static files 19:23:39 <lifeless> ok, next topic 19:23:52 <lifeless> #topic Projects needing releases 19:24:01 <lifeless> #info marios_ to release the world 19:24:11 <lifeless> (He volunteered in #tripleo :)) 19:24:17 <lifeless> #topic CD Cloud status 19:24:54 <lifeless> #info HP1 is still in burnin, no specific new to report 19:25:01 <lifeless> I believe RH1 is still happy 19:25:04 <lifeless> dprince: ^ ? 19:25:12 <lifeless> tchaypo: status of HP2 ? 19:25:13 <dprince> lifeless: :) 19:25:28 <tchaypo> HP2 now has the full 100 machines available to us. 4 are reserved for special use; 4 need some love and attention; the 14 have been register-noded and the other 82 will get that treatment today 19:25:31 <lifeless> #info RH1 is copacetic 19:26:08 <lifeless> #info RH2 is still in bring-up, capacity of 100 machines at this stage. 19:26:11 <lifeless> bah 19:26:12 <lifeless> #undo 19:26:13 <openstack> Removing item from minutes: <ircmeeting.items.Info object at 0x2927650> 19:26:19 <lifeless> #info HP2 is still in bring-up, capacity of 100 machines at this stage. 19:26:23 <tchaypo> There are some reviews up for the tripleocd element that I'll be testing out - in theory they should make the scripts able to handle any region with a few config files 19:26:33 <lifeless> #topic CI 19:26:56 <tchaypo> https://review.openstack.org/116534 is the top of the tree. Probably should have mentioned during reviews. 19:27:15 <lifeless> http://goodsquishy.com/downloads/tripleo-jobs.html looks ok 19:27:23 <lifeless> any other CI news? 19:28:09 <lifeless> #topic tuskar 19:28:18 <lifeless> Any Tuskar items ? 19:28:20 <jdob> nothing from me 19:29:03 <lifeless> #topic specs 19:30:51 <lifeless> ok 19:31:05 <dprince> lifeless: review my spec! 19:31:07 <lifeless> gfidente is not here 19:31:12 <slagle> please respond to lifeless's email about specs if you have an opinion on that 19:31:29 <lifeless> #info please respond to lifeless's email about specs if you have an opinion on that 19:31:33 <lifeless> #topic open discussion 19:31:49 <lifeless> gfidente had a thing about deedicated jobs for block storage and object storage 19:32:18 <slagle> #link http://lists.openstack.org/pipermail/openstack-dev/2014-August/043455.html 19:32:38 <slagle> the spec email ^ 19:32:38 <lifeless> dprince: I'd like to turn on the vlan job as well before we merge the os-net-config stuff 19:32:41 <boris-42> lifeless yes working on it 19:32:45 <tchaypo> that was discussed last week, consensus at the time was to go ahead with them 19:32:45 <lifeless> dprince: we are -very- close to that 19:32:46 <boris-42> lifeless sorry for long reply 19:33:33 <lifeless> dprince: last week you were concerned about adding more jobs 19:34:22 <dprince> lifeless: until we have more capicity I'd like to hold still 19:34:54 <lifeless> dprince: I think we have enough - we're not seeing any significant queuing ATM AFAICT 19:35:10 <lifeless> dprince: I'm not talking about adding the full spread 19:36:04 <lifeless> dprince: what about we add it (when the deps have landed) and review after one week, pull it out if we've cross the safe threshold 19:36:57 <dprince> lifeless: just the vlan job is probably okay but I'd like to consider combining it with one of the existing jobs 19:37:40 <lifeless> dprince: if we move the ironic HA job to voting 19:37:55 <lifeless> dprince: we would then have two overcloud jobs, and we could make one vlan 19:38:20 <lifeless> dprince: but! I think we need to run the vlan stuff non-voting as a separate job initially, because we don't want to wedge ourselves 19:38:35 <dprince> lifeless: seperate initially is fine, then combine them :) 19:38:39 <lifeless> yah 19:38:55 <lifeless> anyone have thoughts on making the ha job voting? 19:39:07 * bnemec checks goodsquishy 19:39:28 <slagle> i still see failures on changes that are unrelated to ha 19:39:56 <lifeless> ok, so some reliability improvements still needed 19:40:52 <greghaynes> ohai 19:40:53 <lifeless> TheJulia: / greghaynes: are you tracking the ha job reliability? It needs more to be turned on as voting 19:41:07 <lifeless> jprovazn: ^ 19:41:08 <greghaynes> yes, I have been trying to debug CI fails 19:41:29 <lifeless> ok, do you need help ? 19:42:08 <greghaynes> Actually, yes! There was one scary one where br-ex interface was never created, and this seemed beyond my network script knowledge 19:42:38 <greghaynes> currently am debugging one where a node also ended up as error, deleting state in nova.... which I dont get how that is even possible 19:42:57 <lifeless> greghaynes: we've seen that quite a lot in the HP1 burnin 19:42:59 <greghaynes> anywho, Ill just keep rambling about them in #tripleo, please to help if possible :) 19:43:22 <lifeless> greghaynes: might want to turn on debug for the ha jobs to help track this down (sadface, logging again, sadface) 19:43:50 <greghaynes> aha. I think ill do the secondary logging sink to store debug in a separate file 19:44:05 <greghaynes> good idea 19:44:18 <lifeless> greghaynes: doing normal debug via the heat parameter would get it into logstash 19:44:54 <lifeless> greghaynes: https://review.openstack.org/#/c/112778/ 19:45:25 <lifeless> any other stuff to chat about ? 19:45:28 <greghaynes> hrm ok 19:45:48 <dprince> lifeless: also, would be nice to have more fedora jobs in CI 19:45:53 <dprince> lifeless: just saying :) 19:46:27 <lifeless> dprince: hah sure 19:46:51 <greghaynes> Was the mariadb element being unusable brought up? 19:46:54 <lifeless> dprince: I have no objection - we do want a spread. but should we change jobs or put the effort into the new jobs 19:46:57 <lifeless> greghaynes: nope 19:47:06 <bnemec> greghaynes: Not yet, but I was considering it. :-) 19:47:07 <greghaynes> bnemec: mentioned that the mariadb element doesnt actually work ATM 19:47:22 <bnemec> It looks that way to me at least. 19:47:22 <dprince> lifeless: either, I'd like to see more of a spread too 19:47:31 <slagle> didn't we talk about deprecating then deleting that? 19:47:38 <slagle> mariadb-rpm is the one that should be used 19:47:51 <slagle> iirc, no one spoke up about wanting to use mariadb on ubuntu 19:48:06 <bnemec> Yeah, I'm not sure we even need to deprecate it. I'm pretty sure it never worked, so it's literally impossible that anyone is using it. 19:48:06 <greghaynes> ah, either way, I would *really* like to see the maridb(-rpm) be used in our CI 19:48:06 <slagle> and on !ubuntu, they'd use mariadb-rpm 19:48:16 <greghaynes> since mysql is kind of a moving target for us 19:48:18 <slagle> mariadb-rpm is being used by the fedora job 19:48:24 <slagle> afaik 19:48:24 <bnemec> ^ 19:48:41 <bnemec> It's the incubator default for fedora. 19:48:49 <slagle> right, so CI will pick that up 19:48:54 <greghaynes> oh. so it is CI'd? 19:48:56 <greghaynes> well, all is well 19:49:25 <dprince> slagle: I'm seeing DB connection issues today w/ mariadb 19:50:18 <slagle> dprince: ok... 19:50:39 <slagle> we've got some passing CI jobs, so i don't think it's across the board 19:51:25 <bnemec> I haven't had any db issues in devtest today. 19:51:35 <dprince> slagle: :) haven't chased it further myself. Just thought I'd mention it 19:51:57 <slagle> lifeless: i've pushed up all the backports needed to fix the stable branches 19:52:10 <slagle> and did a fedora and ubuntu successful run locally 19:52:34 <slagle> just fyi on that front 19:52:34 <ccrouch1> greghaynes: any quick comments on the operators meetup while everyone is here? I know you've been doing some play-by-play commentary on #tripleo 19:52:58 <greghaynes> hah, yes, really people want some kind of reasonable arch. docs! 19:53:13 <greghaynes> there is a lot of "I get what youre trying to do, but I know your complicated...." 19:53:17 <lifeless> slagle: awesome 19:53:24 <lifeless> slagle: sounds like multiple things 19:53:30 <lifeless> slagle: how will we land it ? 19:54:01 <slagle> lifeless: this needs to land first: https://review.openstack.org/#/c/115755/ 19:54:14 <slagle> then let's see how the jobs on the backports fair 19:54:30 <slagle> i think i pushed them up in the right order 19:54:50 <bnemec> https://review.openstack.org/#/c/115755/ passed CI, BTW 19:54:59 <lifeless> slagle: I think your +A on it is all thats needed :P 19:55:06 <slagle> ah, so it is :) 19:55:10 <greghaynes> +A'd 19:55:17 <lifeless> well +2 + +A 19:55:34 <bnemec> ninj+A'd 19:55:53 <rbrady> greghaynes: I have an arch doc I worked on a while back: https://github.com/rbrady/tripleo/blob/master/docs/architecture_overview.rst 19:56:11 <greghaynes> So, one thing that would be *amazing* is at least a graphic of our seed, uc, oc deal 19:56:43 <greghaynes> rbrady: nice! 19:57:22 <lifeless> 3m warning 19:58:03 <ccrouch1> i've read through rbrady's doc and i think it helps with the big picture. Its not specific to devtest. Its more focussed on Undercloud and Overcloud being the big things 19:58:27 <ccrouch1> might be useful on its own or as a starting point for something else 19:58:33 <greghaynes> Getting that or something similar onto our docs page would be awesome :) 19:58:36 <greghaynes> yes 19:59:21 <ccrouch1> i'm sure rbrady would be open to pull requests :-) and moving it somewhere else, if people wanted that 19:59:35 <rbrady> ccrouch1: +1 20:00:02 <lifeless> ok folk thats all that we can write 20:00:06 <lifeless> Thanks! 20:00:09 <lifeless> #endmeeting