20:01:08 <lifeless> #startmeeting tripleo
20:01:09 <openstack> Meeting started Mon Jul 22 20:01:08 2013 UTC.  The chair is lifeless. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:01:10 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
20:01:13 <openstack> The meeting name has been set to 'tripleo'
20:01:13 <lifeless> #topic agenda
20:01:20 <lifeless> bugs
20:01:21 <lifeless> Grizzly test rack status
20:01:21 <lifeless> CI virtualized testing progress
20:01:21 <lifeless> open discussion
20:01:56 <lifeless> #topic bugs
20:02:06 <lifeless> https://bugs.launchpad.net/tripleo/
20:02:06 <lifeless> https://bugs.launchpad.net/diskimage-builder/
20:02:06 <lifeless> https://bugs.launchpad.net/os-refresh-config
20:02:06 <lifeless> https://bugs.launchpad.net/os-apply-config
20:02:08 <lifeless> https://bugs.launchpad.net/os-collect-config
20:03:41 <jog0> o/
20:04:00 <SpamapS> o/
20:04:03 <lifeless> https://bugs.launchpad.net/tripleo/+bug/1182249
20:04:35 <SpamapS> lifeless: _almost_ ready to tackle that
20:04:42 <lifeless> https://bugs.launchpad.net/tripleo/+bug/1183223
20:04:47 <lifeless> https://bugs.launchpad.net/tripleo/+bug/1184484
20:04:49 <SpamapS> lifeless: once we swap in os-collect-config , should be able to realistically address it.
20:04:54 <lifeless> https://bugs.launchpad.net/tripleo/+bug/1189385
20:04:59 <lifeless> https://bugs.launchpad.net/tripleo/+bug/1200201
20:05:03 <lifeless> https://bugs.launchpad.net/tripleo/+bug/1201580
20:05:09 <lifeless> https://bugs.launchpad.net/tripleo/+bug/1201581
20:05:14 <lifeless> https://bugs.launchpad.net/tripleo/+bug/1201584
20:05:19 <lifeless> https://bugs.launchpad.net/tripleo/+bug/1202322
20:05:29 <lifeless> https://bugs.launchpad.net/diskimage-builder/+bug/1202612
20:05:32 <jog0> will be working on 1184484 this week
20:05:40 <lifeless> wheee we have a bunch of crits ;)
20:06:25 <SpamapS> did we lose the bug bot?
20:06:35 <lifeless> bug 1202322
20:06:39 <jog0> been cought up in rootwrap nova-network entrypoints land
20:06:41 <lifeless> appears so
20:07:54 <lifeless> ok so
20:08:04 <lifeless> the dib one is kinda worrying
20:08:13 <lifeless> since it's -really- harsh when it happens
20:08:37 <SpamapS> lifeless: https://bugs.launchpad.net/tripleo/+bug/1202322 just need to land the 2 reviews on os-collect-config and then try devtest with os-collect-config instead of heat-cfntools
20:08:40 <derekh> lifeless: this happened me quite a few times last week, but not once today ....
20:09:01 <lifeless> SpamapS: Well, lets get that done :>
20:09:13 <derekh> lifeless: If it happens again will try and track it down
20:09:25 <SpamapS> lifeless: yeah, its what I was working on, when the meeting started :)
20:09:34 <lifeless> SpamapS: 1201581 - do we need new tenant instances ?
20:09:38 <lifeless> SpamapS: or just server side ?
20:09:52 <SpamapS> lifeless: also for the di-b bug.. we can fix it by using rm --one-file-system
20:10:22 <lifeless> SpamapS: you think the bind mounted dev is still in place when we rm ?
20:10:27 <lifeless> SpamapS: say so in the bug :)
20:10:36 <SpamapS> lifeless: I think we can just deploy a newer keystoneclient in keystone's venv.
20:11:36 <lifeless> bug 1201580 is going to be dependent on the precious fs movement stuff
20:11:45 <lifeless> plus in-instance upgrade hacks
20:12:19 <lifeless> jog0: we will need your nova expertise at some point, we have this crazy idea about updating the boot ramdisk + kernel for ari+aki using flavors
20:12:30 <lifeless> jog0: (and making that work and push out to existing instances)
20:12:41 <lifeless> s/flavors/images/
20:12:43 <jog0> lifeless: is this the rsync based thing?
20:12:54 <lifeless> jog0: not directly, but tied into it.
20:13:04 <SpamapS> lifeless: I was just thinking that we could push out a git-tree-puller and a 'pip install -U''er and an 'apt-get upgrade''er
20:13:16 <lifeless> SpamapS: ah, so a hack :)
20:13:19 <SpamapS> lifeless: as ghetto and stinky as possible
20:13:24 <SpamapS> right
20:13:26 <jog0> lifeless: have a link to this crazy idea?
20:14:03 <lifeless> jog0: no, I put a bug and etherpad up about the issue
20:14:06 <lifeless> *will*
20:14:31 <jog0> lifeless: cool
20:15:20 <lifeless> #action lifeless to ensure we have bugs surrounding the in-instance upgrade path and new ramdisks/kernels
20:15:51 <lifeless> jog0: actually what we'd like to do is to rebase an instance onto a new kernel/ramdisk and disk image - but not reboot it - trust it will redo the image contents itself.
20:16:00 <lifeless> but yeah, will write that up
20:16:38 <jog0> interesting, thanks don't fully grok how that works so a writeup would be great
20:16:44 <lifeless> SpamapS: - https://bugs.launchpad.net/tripleo/+bug/1200201 - still exists ?
20:17:29 <SpamapS> lifeless: I haven't verified it is closed yet.
20:17:35 <SpamapS> lifeless: forgot to tag it in ORC-REFACTOR
20:17:38 <lifeless> kk
20:18:10 <lifeless> https://bugs.launchpad.net/tripleo/+bug/1189385 is still pending something; we haven't seen reproduction in a while now.
20:18:13 * SpamapS assigns self
20:18:24 <lifeless> and I still owe https://bugs.launchpad.net/tripleo/+bug/1184484 some config extraction
20:18:33 <lifeless> ok
20:18:45 <lifeless> any pet bugs folk want to chat about ?
20:19:35 <lifeless> ok
20:19:46 <lifeless> #topic grizzly test rack status
20:19:53 <lifeless> so this is ticking along
20:20:06 <lifeless> I found the network node services had no upstart jobs yesterday
20:20:09 <lifeless> this had everything down
20:20:13 <lifeless> and I have NFI how/why..
20:20:35 <lifeless> I added them using os-svc-install
20:20:37 <lifeless> but sheese.
20:20:45 <SpamapS> I think the thing is a little rickety and concerning. We realy do need at least a tiny subset rack to be able to CD to so we don't have a dead duck.
20:22:16 <SpamapS> A huge portion of what was done in the POC has been rewritten and refactored a lot since then.. no idea if it would apply to that rack now. :-P
20:22:55 <lifeless> so there are spare machines
20:23:24 <lifeless> someone needs to grab the hw list and examine the machines that are faulty and try recovery
20:23:31 <lifeless> I can offer some offline hints about that
20:23:43 <lifeless> also I have a list of 8 or so other machines that were earmarked for monty and are idle
20:23:51 <lifeless> again, someone with time needs to JFDI
20:24:19 <SpamapS> time + access ;)
20:24:29 <lifeless> so, access - good point.
20:25:17 <lifeless> This is HP hardware in a production datacentre; I don't have the authority to give control plane access to the cloud to non-HP staff, *but* any HP staff involved in tripleo should be totally fine.
20:25:51 <lifeless> # action HP tripleoers If you don't have access to the POC rack control plane. ping me/ng/spamaps - all of us should be able to add you.
20:25:55 <lifeless> #action HP tripleoers If you don't have access to the POC rack control plane. ping me/ng/spamaps - all of us should be able to add you.
20:26:17 <lifeless> huh, failbot ?
20:26:25 <lifeless> #action HP-tripleoers If you don't have access to the POC rack control plane. ping me/ng/spamaps - all of us should be able to add you.
20:26:30 <lifeless> NFI....
20:26:57 <SpamapS> hey the bug bot is back :)
20:27:05 <lifeless> heh
20:27:14 <lifeless> so - we need to action the criticals around the rack
20:27:19 <lifeless> but we talked about that
20:27:25 <lifeless> so - next topic time ?
20:27:35 <SpamapS> are all of the criticals assigned?
20:27:45 <SpamapS> or at least, the blocking criticals?
20:27:51 <SpamapS> (may be ordering issues..)
20:28:41 <lifeless> no
20:28:43 <lifeless> they are not
20:29:27 <SpamapS> Ok well I think we can address them as criticals and just attack them one by one.
20:29:39 <lifeless> yup
20:29:41 <lifeless> #topic CI virtualized testing progress
20:29:48 <lifeless> ok, that works. da fuq
20:29:55 <lifeless> pleia2: oh hai.
20:30:03 <lifeless> pleia2: I suspect you're going to say 'nochange' :>
20:30:26 <pleia2> yeah, at oscon this week
20:31:08 <lifeless> #topic open discussion
20:32:30 <dkehn> once again all reviews have been addressed on the neutron  and neutronclient, going to push for merge in next meeting
20:32:37 <lifeless> coool!
20:32:43 <SpamapS> It's worth stating here, I am overhauling os-refresh-config and replacing cfn-hup with os-collect-config .. so please do report any weirdness you see there.
20:32:53 <lifeless> wooo
20:33:09 <lifeless> I'm going to be AWL from thursday through wednesday
20:33:44 <SpamapS> Oh and
20:33:47 <SpamapS> we're like, official and stuff
20:33:55 <lifeless> I have some leave thursday/fri then tuesday doing tech @ work day in Sydney. Mon and wednesday are a combination of being not-at-home connectivity spottiness and travel.
20:34:04 <lifeless> so I need someone to run this meeting next week.
20:34:32 * SpamapS checks schedule to be sure
20:35:07 <SpamapS> lifeless: I will run it
20:35:18 <lifeless> thanks!
20:35:26 <lifeless> #action SpamapS run da meeting next week.
20:35:33 <lifeless> #help
20:36:29 <lifeless> ok
20:36:35 <lifeless> so something I think we should try and sync on
20:36:37 <lifeless> is the roadmapish
20:36:46 <lifeless> we're now at the opencloud - woo!
20:37:08 <lifeless> in my head, it's now time to take our narrow feature set and start expanding sideways
20:37:12 <lifeless> - updates
20:37:14 <lifeless> - HA
20:37:34 <SpamapS> I need to spend some time on bringing Heat up to our expectations.
20:37:40 <SpamapS> we need more failures to be retryable
20:37:56 <SpamapS> and rolling/canary updates will not make h3 if I don't start on it by next week.
20:37:57 <lifeless> - bare metal improvements (like including vendor firmware flashes in the deploy ramdisk)
20:38:06 <lifeless> - heat ^
20:38:44 <lifeless> SpamapS: so H3 doesn't worry me too much, as long as we're not entirely blocked for 2 months - can you land it with an option to enable it or something
20:40:21 <SpamapS> lifeless: it can be marked experimental for sure.
20:40:31 <SpamapS> lifeless: it won't interfere with anything until you say "please update using canaries/rolling"
20:40:43 <lifeless> SpamapS: great
20:40:45 <SpamapS> so yeah, the manual can say "This is experimental use at your own risk"
20:40:59 <lifeless> SpamapS: well more I mean do you *need* to stress about H3
20:41:03 <lifeless> we deploy trunk today
20:41:11 <lifeless> so as long as we can get the support into trunk...
20:41:38 <SpamapS> I have a collaborator from outside tripleo who will be helping, who might care about H3 :)
20:41:42 <lifeless> kk
20:42:03 <lifeless> what else can we bifurcate onto
20:42:05 <lifeless> - performance
20:42:16 <lifeless> - monitoring [as NobodyCam is already!]
20:42:20 <SpamapS> CI/CD for di-b
20:42:22 <lifeless> - reporting
20:42:28 <lifeless> - CICD yes yes yes!
20:42:31 <SpamapS> I know a lot has been done already
20:42:40 <lifeless> basically - we've automated the stuff the POC taught us.
20:42:45 <SpamapS> seems like we're close with the offline features to being able to test it in isolation
20:42:51 <lifeless> Perhaps we need a new stretch goal to consolidate around ?
20:42:59 <SpamapS> well there is this sprint..
20:43:40 <lifeless> its ages off, we should be finished by then :P
20:43:47 <SpamapS> true
20:44:42 <lifeless> baseline - if anyone is aimless, we have tonnes to do, ping me [or anyone else on the team] and we'll help you find a useful thing that is within reach
20:45:05 <lifeless> last call on discussion ...
20:47:15 <lifeless> ok, thanks for playing!
20:47:18 <lifeless> #endmeeting