19:00:55 <lifeless> #startmeeting tripleo
19:00:55 <jistr> hi
19:00:56 <openstack> Meeting started Tue Jan 14 19:00:55 2014 UTC and is due to finish in 60 minutes.  The chair is lifeless. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:57 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:00:59 <ifarkas> o/
19:01:00 <openstack> The meeting name has been set to 'tripleo'
19:01:01 <slagle> hello
19:01:06 <jtomasek> hey
19:01:07 <lifeless> #topic agenda
19:01:10 <lifeless> bugs
19:01:10 <lifeless> reviews
19:01:11 <lifeless> Projects needing releases
19:01:11 <lifeless> CD Cloud status
19:01:11 <lifeless> CI virtualized testing progress
19:01:12 <lifeless> Insert one-off agenda items here
19:01:15 <lifeless> open discussion
19:01:18 <lifeless> #topic bugs
19:01:28 <lifeless> #link https://bugs.launchpad.net/tripleo/
19:01:29 <lifeless> #link https://bugs.launchpad.net/diskimage-builder/
19:01:29 <lifeless> #link https://bugs.launchpad.net/os-refresh-config
19:01:29 <lifeless> #link https://bugs.launchpad.net/os-apply-config
19:01:29 <lifeless> #link https://bugs.launchpad.net/os-collect-config
19:01:31 <lifeless> #link https://bugs.launchpad.net/tuskar
19:01:33 <lifeless> #link https://bugs.launchpad.net/tuskar-ui
19:01:36 <lifeless> #link https://bugs.launchpad.net/python-tuskarclient
19:02:25 <SpamapS> o/
19:02:29 <lifeless> we have some untriaged bugs in tripleo
19:02:34 <lifeless> and a bunch of criticals
19:02:44 <lifeless> dib has many untriaged bugs
19:03:18 <slagle> i'll handle https://bugs.launchpad.net/diskimage-builder/+bug/1263902 as i already fixed it
19:03:26 <SpamapS> lifeless: I wonder if we could get reviewday to show untriaged bugs mixed in with reviews.
19:03:26 <slagle> didn't know there was a bug for it :/
19:03:52 <lifeless> so we had the commitment from everyone to do a little triage one day a week
19:04:05 <pleia2> SpamapS: there is also bugday, which might be worth a look
19:04:16 <pleia2> SpamapS: right now it's just graphs, but can probably be modified
19:04:16 <SpamapS> pleia2: moar placesss
19:04:20 <lifeless> looking at the dates, clearly that isn't happening (holidays notwithstanding)
19:04:35 <SpamapS> I just discovered I was not subscribed to diskimage-builder
19:04:40 <SpamapS> I tend to use my email for bug triage
19:05:43 <lifeless> I'll do a pass this evening, make sure we'll all triaged; shall we recommit to each do triage no less than once a week ?
19:05:58 <lifeless> with 20 odd folk thats /more/ than sufficient
19:06:10 <SpamapS> lifeless: is https://bugs.launchpad.net/tripleo/+bug/1266800 still an issue (boot-seed-vm failing) ?
19:06:15 <slagle> process question: can we triage our own bugs?
19:06:20 <SpamapS> slagle: yes
19:06:22 <lifeless> slagle: yes
19:06:32 <slagle> then i'll triage the 2 i filed :)
19:06:40 <lifeless> slagle: you should never file a bug untriaged if you have access to set the right bits
19:06:52 <slagle> ack
19:08:44 <lifeless> SpamapS: I fstrimmed and the issue hasn't reoccured
19:10:10 <SpamapS> 1265807 is a duplicate of 1266513
19:10:21 <SpamapS> can somebody confirm that?
19:11:15 <rpodolyaka1> seems so
19:11:24 <jprovazn> SpamapS, yes
19:11:42 <SpamapS> ok will mark as such
19:11:43 <lifeless> yep
19:12:59 <lifeless> SpamapS: I don't tink 1261253 is critical; there is a workaround
19:13:07 <lifeless> SpamapS: and the default setup doesn't use a mirror
19:13:48 <SpamapS> lifeless: agreed, High is more appropriate. Changing.
19:14:12 <lifeless> https://bugs.launchpad.net/tripleo/+bug/1254555 is fixreleased from out perspective I think
19:14:21 <lifeless> isn't it?
19:14:53 <SpamapS> lifeless: IIRC we are just working around it with lots of retries
19:15:14 <lifeless> hmm, see comment 7
19:15:19 <SpamapS> lifeless: we can maybe lower it to High as well since it is worked around.
19:15:38 <lifeless> neutron reckon it shouldn't be presenting symptoms any more
19:15:48 <SpamapS> lifeless: I think that is sort of "the problem" with race conditions.. work around them and people think they go away. :(
19:16:29 <lifeless> so, we should take the bandaid off, if the problem is gone, great, if not we should ask neutron to re-prioritise it
19:16:45 <rpodolyaka1> +1
19:17:17 <lifeless> https://bugs.launchpad.net/tripleo/+bug/1254246 looks like it's stuck in review
19:17:37 <rpodolyaka1> so it's actually kind of fixed for us
19:17:43 <rpodolyaka1> due to how we do db-sync in neutron
19:17:57 <rpodolyaka1> but to be 'fully fixed' that patch must be merged
19:18:15 <lifeless> rpodolyaka1: ok. devananda hit it yesterday, but I don't know how old his seed was
19:18:34 <rpodolyaka1> lifeless: hmm, should be fixed for a while now
19:18:45 <lifeless> rpodolyaka1: his seed might be quite old :)
19:18:53 <rpodolyaka1> I think so
19:19:03 <lifeless> ok, so lets FR it
19:19:07 <SpamapS> sounds like it needs somebody to reconfirm and adjust status appropriately
19:19:07 <lifeless> for tripleo
19:19:27 <lifeless> I certainly haven't seen it
19:19:34 <lifeless> and I've been doing lots of fresh builds recently
19:20:15 <lifeless> ok so
19:20:21 <lifeless> any more bug discussion?
19:21:39 <dprince> lifeless: real quick
19:21:49 <dprince> https://bugs.launchpad.net/diskimage-builder/+bug/1239480
19:22:01 <dprince> lifeless: is truncating /etc/network/interfaces an option there?
19:22:27 <dprince> lifeless: I just messed w/ dhcp-all-interfaces and am happy to clean triage on its bugs... but need clarification on that one issue
19:22:50 <lifeless> dprince: good question
19:23:11 <dprince> lifeless: you can look at it later too, just thought it worth a mention.
19:23:13 <lifeless> dprince: I think truncating is probably a bad idea, because unlike fedora its all munged into one file
19:23:47 <lifeless> dprince: editing 'auto eth0' out would be more surgical
19:23:56 <SpamapS> lifeless: I just made a debian cloud image and I just didn't even bother to make /etc/network/interfaces .. I just used dhcp-interfaces-all
19:23:57 <lifeless> I'm thinking about folk with bridges or vpns defined
19:23:59 <dprince> lifeless: yeah. well then its going to have to be sliced out then :(
19:24:04 <SpamapS> or dhcp-all-interfaces.. :)
19:24:25 <lifeless> #topic reviews
19:24:44 <lifeless> http://russellbryant.net/openstack-stats/tripleo-openreviews.html
19:24:51 <marios> down for me all day ^^
19:25:08 <lifeless> ok, guess we're skipping it :)
19:25:46 <lifeless> #topic  Projects needing releases
19:25:55 <lifeless> Do we have a volunteer ?
19:26:03 <SpamapS> http://webcache.googleusercontent.com/search?q=cache:7IE9twHe6I8J:russellbryant.net/openstack-stats/tripleo-openreviews.html+&cd=1&hl=en&ct=clnk&gl=us&client=ubuntu
19:26:06 <SpamapS> cached version
19:26:13 <SpamapS> doh
19:26:14 <SpamapS> Jan 8
19:26:23 <SpamapS> come on google.. that thing changes every hour at least :)
19:26:28 <lifeless> he might have moved it to an infra host
19:26:32 <lifeless> will follow up out of band
19:26:42 <lifeless> russellb: ^
19:26:53 <lifeless> rpodolyaka1: are you up for doing releases?
19:26:59 <rpodolyaka1> sure:)
19:27:19 <lifeless> cool
19:27:26 <lifeless> #topic CD Cloud status
19:27:34 <lifeless> dprince: hows the RH region coming along?
19:28:27 <dprince> lifeless: still pushing to get a set of IPv4 IPs.
19:28:36 <dprince> lifeless: and to get everyone public access
19:28:46 <lifeless> dprince: ok. No IPv6 either?
19:29:02 <dprince> lifeless: internally I have access now and I've got my seed, undercloud, overcloud running now.
19:29:13 <russellb> it hasn't moved to infra, it's just down
19:29:16 <dprince> lifeless: mostly just testing the water, etc to work the kinks out ahead of time.
19:29:26 <russellb> it's hosted by "random friend hosting inc"
19:29:47 <dprince> lifeless: IPv6 is going to be harder than getting IPv4 I think. At least in our current datacenter.
19:30:01 <dprince> lifeless: so progress, and I'm pushing on it as much as I can....
19:30:13 <lifeless> dprince: so I had an idea on IPv6
19:30:21 <lifeless> dprince: if we get one ipv4 address
19:30:29 <SpamapS> For our CD cloud, we're experiencing some weirdness on the undercloud box regarding networking.
19:30:30 <lifeless> dprince: we can get a IPv6 subnet from HE
19:30:44 <dprince> lifeless: what is HE?
19:30:57 <lifeless> hurricane electric
19:31:02 <SpamapS> tunnel ftw
19:31:15 <dprince> ah, that. Sure. We could go that route I suppose.
19:31:43 <dprince> If that works, and allows us to hook it up to infra I'm fine w/ it.
19:31:59 <lifeless> ok, so HP region
19:32:07 <lifeless> SpamapS: thoroughly weirdness
19:32:23 <dprince> I'm still going for a set of IPv4 address, if it ends up being a small set then perhaps the tunneling is a good option.
19:32:40 <dprince> SpamapS: is this another mellanox driver thing?
19:34:26 <SpamapS> dprince: no
19:34:35 <SpamapS> dprince: I bumped it earlier today and it had no effect
19:35:25 <SpamapS> one thing.. perhaps the box is being DoS'd or used as a reflector. It has basically no firewall.
19:36:57 <lifeless> SpamapS: nope
19:37:12 <lifeless> SpamapS: 1.5kb of traffic on the public itnerface
19:37:28 <SpamapS> lifeless: its also healthy right now
19:37:38 <lifeless> SpamapS: not really
19:37:41 <SpamapS> lifeless: we should probably graph traffic
19:37:58 <lifeless> we should implement the monitoring we've been mubmling about
19:38:07 <SpamapS> lifeless: it is screaming fast for me now.. much better than it has been the last 4 hours
19:38:11 <SpamapS> yeah
19:38:21 <lifeless> ok so
19:38:35 <lifeless> was the bastion ill too ?
19:38:37 <lifeless> or other ips?
19:39:57 <lifeless> so, lets look at tht out of the meeting
19:40:04 <lifeless> I've spun up a new cloud
19:40:08 <lifeless> ci-overcloud.tripleo.org
19:40:19 <lifeless> the intent is to use this for jenkins slaves
19:40:22 <lifeless> and the broker
19:40:45 <lifeless> it has a new baremetal network 192.168.1.0 for the test traffic to be on
19:40:52 <lifeless> and ssl courtesy of ng
19:41:16 <SpamapS> woo
19:41:21 <lifeless> like the poc, this will be static - as soon as we have nodowntime deploys of the cd-overcloud we can ditch it
19:41:28 <lifeless> it has one compute node
19:41:41 <lifeless> so the intent (if it's not obvious) is that it won't have a lot of load on it ever
19:43:12 <lifeless> with the state everythings in I hope we can be in the silent queue next week
19:44:13 <SpamapS> wooot
19:44:53 <lifeless> #topic CI virtualized testing progress
19:45:05 <lifeless> pleia2: / dprince: ^
19:45:28 <pleia2> I don't have updates myself, but I would like to do a sync up call with dprince and derekh soon (tomorrow?)
19:45:59 <dprince> Yes. It is time for a sync up. I'll send out an invite here shortly for tommorow
19:46:04 <pleia2> dprince: great, thanks
19:46:29 <lifeless> #topic open discussion
19:46:56 <SpamapS> I just wanted to bring up that I could use help working on Heat for TripleO purposes.
19:47:14 <dprince> SpamapS: what specifically needs helping with?
19:47:25 <SpamapS> We're going to want to start using resource/server groups, and rolling updates.. and retry updates.. and...
19:47:53 <lifeless> we finished migrating stuff to /mnt/state as far as we know right?
19:47:55 <SpamapS> dprince: hot-software-config is a blueprint that we need to clean up the templates and make them not rely on merge.py
19:47:56 <dprince> lifeless: BTW. I reviewed most of the use_ephemeral Nova branches today. I think those are all very close to being able to land. The last branch in the series needs a little fixup though...
19:48:00 <SpamapS> lifeless: not /etc
19:48:14 <SpamapS> lifeless: we're at a state where we survive rebuild
19:48:21 <SpamapS> lifeless: but readonly / is still a lot of patches away
19:48:22 <dprince> lifeless: I haven't tested a read only image on Fedora yet
19:48:24 <lifeless> SpamapS: ok
19:48:35 <lifeless> dprince: see SpamapS ^ we're not expecting ro to work [yet]
19:48:38 <dprince> lifeless: I'd say we have a good bit of work on our side yet.
19:48:43 <dprince> lifeless: right.
19:48:44 <lifeless> dprince: for anyone ;)
19:48:47 <SpamapS> lifeless: it works because os-collect-config re-asserts all the state again.
19:48:58 <lifeless> which is ok, the first goal was /mnt state preservation
19:49:05 <lifeless> readonly / isn't immediate
19:49:40 <SpamapS> also rabbit does not perserve state across rebuild
19:50:04 <SpamapS> preserve
19:50:15 <dprince> SpamapS: So if I work on the hot-software-config does that mean I get to say I'm a hot developer?
19:50:29 * dprince jokes
19:50:39 <slagle> i may have some cycles soon'ish to look at Heat related stuff
19:50:58 <SpamapS> dprince: you already say that, but now we might not roll our eyes :)
19:51:08 <lifeless> boom-tish
19:51:19 <dprince> I'll say boom to that
19:51:20 <slagle> where's a good place to start? just getting familiar with hot-software-config?
19:51:22 * dprince BOOM
19:51:24 <SpamapS> anyway, just suggesting that I think our Heat usage has suffered and is the ugliest part of TripleO IMO.
19:51:48 <SpamapS> slagle: getting familiar with running heat beyond 'stack-create' will help.
19:52:11 <slagle> i stack-delete a lot too
19:52:26 <SpamapS> stack-update is where the trouble starts
19:52:39 <slagle> ok :)
19:54:02 <SpamapS> I will start tagging Heat bugs with tripleo if I see things that would be good for all of us to track
19:54:24 <SpamapS> that is all.
19:54:53 <ccrouch> slagle: SpamapS: i was talking to sdake about  https://wiki.openstack.org/wiki/Heat/Blueprints/hot-software-config yesterday
19:55:06 <ccrouch> he expects it to land by milestone 3
19:55:24 <lsmola> SpamapS, great
19:55:46 <SpamapS> Yeah if we all got out and pushed on it, we could, in parallel, switch TripleO to using it and thus improve Heat at the same time we improve TripleO.
19:55:54 <ccrouch> so i dont know if we can help move that along, or just make sure that what will land will be sufficient for us to get rid of merge.py
19:56:14 <SpamapS> ccrouch: last I checked stevebaker had made a call for volunteers on a few pieces
19:56:21 <ccrouch> ok cool
19:56:22 <dprince> SpamapS: when we do this I would also like to refactor things to support an all-in-one overcloud as well.
19:56:30 <lifeless> is everyone tracking trello still ? Sounds like hot should be in there
19:56:44 <slagle> dprince: +1, that would be nice
19:56:52 <SpamapS> http://lists.openstack.org/pipermail/openstack-dev/2013-December/022269.html
19:57:31 <SpamapS> lifeless: I don't know if that is a trello thing or not. It will pay off a bunch of technical debt in merge.py and the resulting painful-to-maintain templates...
19:57:33 <dprince> slagle: thanks, I postponed my previous effort there because HOT is so close
19:58:00 <lifeless> SpamapS: so the basic ideas is that we're moving as a group tackling things together rather than siloing off
19:58:11 <lifeless> SpamapS: you're saying this is the next thing we need, no?
19:58:24 <lifeless> SpamapS: heat work, to get update working properly?
19:58:27 <marios> SpamapS: do you guys think its close enough that i shouldn't waste any more time with https://review.openstack.org/#/c/52045/ and related?
19:58:29 <SpamapS> dprince: yeah the whole point of hot-software-config is to make it easier to write config separate from server placement.
19:59:00 <SpamapS> lifeless: no. It is not the next thing.
19:59:19 <SpamapS> we can keep going deeper in debt to merge.py for a long time to come.
19:59:51 <SpamapS> lifeless: the next thing we need in heat is updatable wait conditions and retry-on-update
19:59:59 <lsmola> SpamapS, so we are keepig merge.py in I ?
20:00:03 <lifeless> #endmeeting