19:05:11 <lifeless> #startmeeting tripleo
19:05:12 <openstack> Meeting started Tue Aug 26 19:05:11 2014 UTC and is due to finish in 60 minutes.  The chair is lifeless. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:05:13 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:05:15 <openstack> The meeting name has been set to 'tripleo'
19:05:41 <bnemec> \o
19:07:00 <lifeless> #topic agenda
19:07:00 <lifeless> * bugs
19:07:00 <lifeless> * reviews
19:07:00 <lifeless> * Projects needing releases
19:07:00 <lifeless> * CD Cloud status
19:07:02 <lifeless> * CI
19:07:05 <lifeless> * Tuskar
19:07:07 <lifeless> * Specs
19:07:10 <lifeless> * open discussion
19:07:22 <lifeless> Remember that anyone can use the link and info commands, not just the moderator - if you have something worth noting in the meeting minutes feel free to tag it
19:07:26 <lifeless> #topic bugs
19:07:39 <lifeless> #link https://bugs.launchpad.net/tripleo/
19:07:39 <lifeless> #link https://bugs.launchpad.net/diskimage-builder/
19:07:39 <lifeless> #link https://bugs.launchpad.net/os-refresh-config
19:07:39 <lifeless> #link https://bugs.launchpad.net/os-apply-config
19:07:39 <lifeless> #link https://bugs.launchpad.net/os-collect-config
19:07:41 <lifeless> #link https://bugs.launchpad.net/os-cloud-config
19:07:44 <lifeless> #link https://bugs.launchpad.net/tuskar
19:07:46 <lifeless> #link https://bugs.launchpad.net/python-tuskarclient
19:08:14 <slagle> i don't think we have any untriaged bugs
19:08:31 <lifeless> we have a bunch of criticals still though
19:08:35 <lifeless> bug 1263294
19:08:36 <uvirtbot> Launchpad bug 1263294 in tripleo "ephemeral0 of /dev/sda1 triggers 'did not find entry for sda1 in /sys/block'" [Critical,In progress] https://launchpad.net/bugs/1263294
19:08:43 <lifeless> bug 1316985
19:08:43 <uvirtbot> Launchpad bug 1316985 in tripleo "set -eu may spuriously break dkms module" [Critical,In progress] https://launchpad.net/bugs/1316985
19:08:49 <lifeless> bug 1317056
19:08:50 <uvirtbot> Launchpad bug 1317056 in tripleo "Guest VM FS corruption after compute host reboot" [Critical,Triaged] https://launchpad.net/bugs/1317056
19:08:57 <lifeless> bug 1361235
19:08:59 <uvirtbot> Launchpad bug 1361235 in tripleo "visit horizon failure because of import module failure" [Critical,In progress] https://launchpad.net/bugs/1361235
19:09:24 <tchaypo> GheRivero: Do you have an update on https://launchpad.net/bugs/1316985 ?
19:09:25 <uvirtbot> Launchpad bug 1316985 in tripleo "set -eu may spuriously break dkms module" [Critical,In progress]
19:09:58 <GheRivero> I just need some love" https://review.openstack.org/#/c/104115/
19:10:01 <slagle> i've put a patch for a workaround for 1361235
19:10:30 <slagle> needs some reviews, https://review.openstack.org/#/c/116721/
19:10:46 <slagle> we could drop the bug to High...i was on the fence
19:11:12 <lifeless> broken builds are crit IMO here - if we were tempesting this it would have failed
19:11:34 <lifeless> boris-42: have you seen https://bugs.launchpad.net/tripleo/+bug/1361235 ?
19:11:35 <uvirtbot> Launchpad bug 1361235 in tripleo "visit horizon failure because of import module failure" [Critical,In progress]
19:12:00 <lifeless> #info please review https://review.openstack.org/#/c/116721/ (critical bugfix)
19:12:31 <lifeless> no sign of rpodolyka - hope he's ok
19:13:11 <tchaypo> #info GheRivero needs some reviews on https://review.openstack.org/#/c/104115/ (critical bugfix)
19:13:41 <lifeless> tchaypo: could you perhaps volunteer to ping michael and roman for the other two bugs ?
19:14:20 <tchaypo> lifeless: consider me volunteered
19:15:04 <lifeless> #info tchaypo will follow up on bug 1263294 and bug 1317056 with assignees
19:15:05 <uvirtbot> Launchpad bug 1263294 in tripleo "ephemeral0 of /dev/sda1 triggers 'did not find entry for sda1 in /sys/block'" [Critical,In progress] https://launchpad.net/bugs/1263294
19:15:07 <uvirtbot> Launchpad bug 1317056 in tripleo "Guest VM FS corruption after compute host reboot" [Critical,Triaged] https://launchpad.net/bugs/1317056
19:15:20 <lifeless> any other bug stuff to discuss?
19:16:09 <lifeless> #topic reviews
19:16:10 <lifeless> #info There's a new dashboard linked from https://wiki.openstack.org/wiki/TripleO#Review_team - look for "TripleO Inbox Dashboard"
19:16:13 <lifeless> #link http://russellbryant.net/openstack-stats/tripleo-openreviews.html
19:16:16 <lifeless> #link http://russellbryant.net/openstack-stats/tripleo-reviewers-30.txt
19:16:19 <lifeless> #link http://russellbryant.net/openstack-stats/tripleo-reviewers-90.txt
19:16:46 <lifeless> 
19:16:46 <lifeless> Stats since the last revision without -1 or -2 :
19:16:46 <lifeless> Average wait time: 12 days, 9 hours, 42 minutes
19:16:46 <lifeless> 1st quartile wait time: 4 days, 23 hours, 21 minutes
19:16:46 <lifeless> Median wait time: 11 days, 4 hours, 30 minutes
19:16:49 <lifeless> 3rd quartile wait time: 18 days, 11 hours, 26 minutes
19:17:11 <tchaypo> Same trend as recently, and I don't think there's been much discussion on the list about what we're measuring being wrong
19:17:20 <tchaypo> s/wrong/not the thing we want to measure/
19:17:31 <lifeless> You started a thread - thanks. I have it tagged to reply to
19:17:53 <lifeless> any pet reviews folk want to ask for help on?
19:19:00 <lifeless> I have three I'm going to beg for reviews on
19:19:22 <lifeless> https://review.openstack.org/#/c/102081/ - backwards compat spec
19:20:05 <lifeless> https://review.openstack.org/#/c/103454/ vlan support for the public virtual control IP
19:20:20 <tchaypo> linkify them?
19:20:33 <lifeless> https://review.openstack.org/#/c/103757/ - incubator support for 103454
19:20:41 <lifeless> tchaypo: sure
19:21:00 <lifeless> #info reviews sought on https://review.openstack.org/#/c/102081/ - backwards compat spec
19:21:01 <slagle> i'll re-reivew the backwards compat spec. been meaning to circle back to it
19:21:13 <lifeless> #info reviews sought on https://review.openstack.org/#/c/103454/ vlan support for the public virtual control IP
19:21:18 <alexislee> Not super important but it'd be nice to have:
19:21:24 <alexislee> #link  https://review.openstack.org/#/c/82038/
19:21:24 <lifeless> #info reviews sought on https://review.openstack.org/#/c/103757/ - incubator support for 103454
19:21:26 <alexislee> #link https://review.openstack.org/#/c/110366/
19:22:03 <lifeless> #info reviews sought on https://review.openstack.org/#/c/82038/ - o-a-c meta support
19:22:56 <lifeless> #info reviews sought on https://review.openstack.org/#/c/110366/ - declarative static files
19:23:39 <lifeless> ok, next topic
19:23:52 <lifeless> #topic Projects needing releases
19:24:01 <lifeless> #info marios_ to release the world
19:24:11 <lifeless> (He volunteered in #tripleo :))
19:24:17 <lifeless> #topic CD Cloud status
19:24:54 <lifeless> #info HP1 is still in burnin, no specific new  to report
19:25:01 <lifeless> I believe RH1 is still happy
19:25:04 <lifeless> dprince: ^ ?
19:25:12 <lifeless> tchaypo: status of HP2 ?
19:25:13 <dprince> lifeless: :)
19:25:28 <tchaypo> HP2 now has the full 100 machines available to us. 4 are reserved for special use; 4 need some love and attention; the 14 have been register-noded and the other 82 will get that treatment today
19:25:31 <lifeless> #info RH1 is copacetic
19:26:08 <lifeless> #info RH2 is still in bring-up, capacity of 100 machines at this stage.
19:26:11 <lifeless> bah
19:26:12 <lifeless> #undo
19:26:13 <openstack> Removing item from minutes: <ircmeeting.items.Info object at 0x2927650>
19:26:19 <lifeless> #info HP2 is still in bring-up, capacity of 100 machines at this stage.
19:26:23 <tchaypo> There are some reviews up for the tripleocd element that I'll be testing out - in theory they should make the scripts able to handle any region with a few config files
19:26:33 <lifeless> #topic CI
19:26:56 <tchaypo> https://review.openstack.org/116534 is the top of the tree. Probably should have mentioned during reviews.
19:27:15 <lifeless> http://goodsquishy.com/downloads/tripleo-jobs.html looks ok
19:27:23 <lifeless> any other CI news?
19:28:09 <lifeless> #topic tuskar
19:28:18 <lifeless> Any Tuskar items ?
19:28:20 <jdob> nothing from me
19:29:03 <lifeless> #topic specs
19:30:51 <lifeless> ok
19:31:05 <dprince> lifeless: review my spec!
19:31:07 <lifeless> gfidente is not here
19:31:12 <slagle> please respond to lifeless's email about specs if you have an opinion on that
19:31:29 <lifeless> #info please respond to lifeless's email about specs if you have an opinion on that
19:31:33 <lifeless> #topic open discussion
19:31:49 <lifeless> gfidente had a thing about deedicated jobs for block storage and object storage
19:32:18 <slagle> #link http://lists.openstack.org/pipermail/openstack-dev/2014-August/043455.html
19:32:38 <slagle> the spec email ^
19:32:38 <lifeless> dprince: I'd like to turn on the vlan job as well before we merge the os-net-config stuff
19:32:41 <boris-42> lifeless yes working on it
19:32:45 <tchaypo> that was discussed last week, consensus at the time was to go ahead with them
19:32:45 <lifeless> dprince: we are -very- close to that
19:32:46 <boris-42> lifeless sorry for long reply
19:33:33 <lifeless> dprince: last week you were concerned about adding more jobs
19:34:22 <dprince> lifeless: until we have more capicity I'd like to hold still
19:34:54 <lifeless> dprince: I think we have enough - we're not seeing any significant queuing ATM AFAICT
19:35:10 <lifeless> dprince: I'm not talking about adding the full spread
19:36:04 <lifeless> dprince: what about we add it (when the deps have landed) and review after one week, pull it out if we've cross the safe threshold
19:36:57 <dprince> lifeless: just the vlan job is probably okay but I'd like to consider combining it with one of the existing jobs
19:37:40 <lifeless> dprince: if we move the ironic HA job to voting
19:37:55 <lifeless> dprince: we would then have two overcloud jobs, and we could make one vlan
19:38:20 <lifeless> dprince: but! I think we need to run the vlan stuff non-voting as a separate job initially, because we don't want to wedge ourselves
19:38:35 <dprince> lifeless: seperate initially is fine, then combine them :)
19:38:39 <lifeless> yah
19:38:55 <lifeless> anyone have thoughts on making the ha job voting?
19:39:07 * bnemec checks goodsquishy
19:39:28 <slagle> i still see failures on changes that are unrelated to ha
19:39:56 <lifeless> ok, so some reliability improvements still needed
19:40:52 <greghaynes> ohai
19:40:53 <lifeless> TheJulia: / greghaynes: are you tracking the ha job reliability? It needs more to be turned on as voting
19:41:07 <lifeless> jprovazn: ^
19:41:08 <greghaynes> yes, I have been trying to debug CI fails
19:41:29 <lifeless> ok, do you need help ?
19:42:08 <greghaynes> Actually, yes! There was one scary one where br-ex interface was never created, and this seemed beyond my network script knowledge
19:42:38 <greghaynes> currently am debugging one where a node also ended up as error, deleting state in nova.... which I dont get how that is even possible
19:42:57 <lifeless> greghaynes: we've seen that quite a lot in the HP1 burnin
19:42:59 <greghaynes> anywho, Ill just keep rambling about them in #tripleo, please to help if possible :)
19:43:22 <lifeless> greghaynes: might want to turn on debug for the ha jobs to help track this down (sadface, logging again, sadface)
19:43:50 <greghaynes> aha. I think ill do the secondary logging sink to store debug in a separate file
19:44:05 <greghaynes> good idea
19:44:18 <lifeless> greghaynes: doing normal debug via the heat parameter would get it into logstash
19:44:54 <lifeless> greghaynes: https://review.openstack.org/#/c/112778/
19:45:25 <lifeless> any other stuff to chat about ?
19:45:28 <greghaynes> hrm ok
19:45:48 <dprince> lifeless: also, would be nice to have more fedora jobs in CI
19:45:53 <dprince> lifeless: just saying :)
19:46:27 <lifeless> dprince: hah sure
19:46:51 <greghaynes> Was the mariadb element being unusable brought up?
19:46:54 <lifeless> dprince: I have no objection - we do want a spread. but should we change jobs or put the effort into the new jobs
19:46:57 <lifeless> greghaynes: nope
19:47:06 <bnemec> greghaynes: Not yet, but I was considering it. :-)
19:47:07 <greghaynes> bnemec: mentioned that the mariadb element doesnt actually work ATM
19:47:22 <bnemec> It looks that way to me at least.
19:47:22 <dprince> lifeless: either, I'd like to see more of a spread too
19:47:31 <slagle> didn't we talk about deprecating then deleting that?
19:47:38 <slagle> mariadb-rpm is the one that should be used
19:47:51 <slagle> iirc, no one spoke up about wanting to use mariadb on ubuntu
19:48:06 <bnemec> Yeah, I'm not sure we even need to deprecate it.  I'm pretty sure it never worked, so it's literally impossible that anyone is using it.
19:48:06 <greghaynes> ah, either way, I would *really* like to see the maridb(-rpm) be used in our CI
19:48:06 <slagle> and on !ubuntu, they'd use mariadb-rpm
19:48:16 <greghaynes> since mysql is kind of a moving target for us
19:48:18 <slagle> mariadb-rpm is being used by the fedora job
19:48:24 <slagle> afaik
19:48:24 <bnemec> ^
19:48:41 <bnemec> It's the incubator default for fedora.
19:48:49 <slagle> right, so CI will pick that up
19:48:54 <greghaynes> oh. so it is CI'd?
19:48:56 <greghaynes> well, all is well
19:49:25 <dprince> slagle: I'm seeing DB connection issues today w/ mariadb
19:50:18 <slagle> dprince: ok...
19:50:39 <slagle> we've got some passing CI jobs, so i don't think it's across the board
19:51:25 <bnemec> I haven't had any db issues in devtest today.
19:51:35 <dprince> slagle: :) haven't chased it further myself. Just thought I'd mention it
19:51:57 <slagle> lifeless: i've pushed up all the backports needed to fix the stable branches
19:52:10 <slagle> and did a fedora and ubuntu successful run locally
19:52:34 <slagle> just fyi on that front
19:52:34 <ccrouch1> greghaynes: any quick comments on the operators meetup while everyone is here? I know you've been doing some play-by-play commentary on #tripleo
19:52:58 <greghaynes> hah, yes, really people want some kind of reasonable arch. docs!
19:53:13 <greghaynes> there is a lot of "I get what youre trying to do, but I know your complicated...."
19:53:17 <lifeless> slagle: awesome
19:53:24 <lifeless> slagle: sounds like multiple things
19:53:30 <lifeless> slagle: how will we land it ?
19:54:01 <slagle> lifeless: this needs to land first: https://review.openstack.org/#/c/115755/
19:54:14 <slagle> then let's see how the jobs on the backports fair
19:54:30 <slagle> i think i pushed them up in the right order
19:54:50 <bnemec> https://review.openstack.org/#/c/115755/ passed CI, BTW
19:54:59 <lifeless> slagle: I think your +A on it is all thats needed :P
19:55:06 <slagle> ah, so it is :)
19:55:10 <greghaynes> +A'd
19:55:17 <lifeless> well +2 + +A
19:55:34 <bnemec> ninj+A'd
19:55:53 <rbrady> greghaynes: I have an arch doc I worked on a while back: https://github.com/rbrady/tripleo/blob/master/docs/architecture_overview.rst
19:56:11 <greghaynes> So, one thing that would be *amazing* is at least a graphic of our seed, uc, oc deal
19:56:43 <greghaynes> rbrady: nice!
19:57:22 <lifeless> 3m warning
19:58:03 <ccrouch1> i've read through rbrady's doc and i think it helps with the big picture. Its not specific to devtest. Its more focussed on Undercloud and Overcloud being the big things
19:58:27 <ccrouch1> might be useful on its own or as a starting point for something else
19:58:33 <greghaynes> Getting that or something similar onto our docs page would be awesome :)
19:58:36 <greghaynes> yes
19:59:21 <ccrouch1> i'm sure rbrady would be open to pull requests :-) and moving it somewhere else, if people wanted that
19:59:35 <rbrady> ccrouch1: +1
20:00:02 <lifeless> ok folk thats all that we can write
20:00:06 <lifeless> Thanks!
20:00:09 <lifeless> #endmeeting