22:02:40 <jeblair> #startmeeting zuul
22:02:41 <openstack> Meeting started Mon Feb 13 22:02:40 2017 UTC and is due to finish in 60 minutes.  The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot.
22:02:42 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
22:02:44 <openstack> The meeting name has been set to 'zuul'
22:02:52 <jeblair> #link agenda https://wiki.openstack.org/wiki/Meetings/Zuul
22:03:02 <jeblair> #link previous meeting http://eavesdrop.openstack.org/meetings/zuul/2017/zuul.2017-02-06-22.03.html
22:03:28 <jeblair> i conveniently did not alter the agenda from last week
22:03:55 <jeblair> seeing as how it still seems relevant :)
22:04:25 <jeblair> #topic Status updates: Nodepool Zookeeper work
22:04:56 <jeblair> Shrews: iiuc, we're at "we can probably launch nodes at the ptg" yeah?
22:05:10 <SpamapS> o/
22:05:21 <Shrews> jeblair: yes
22:05:41 <Shrews> we seem to be launching them in test scenarios, at least
22:06:31 * fungi takes that as a sign
22:06:41 <Shrews> and in the gate-dsvm-nodepool job (when i temporarily enabled it)
22:07:26 <Shrews> http://logs.openstack.org/49/431649/6/check/gate-dsvm-nodepool/c796757/console.html#_2017-02-10_16_38_09_806118
22:07:58 <Shrews> we can't re-enable that job until I get deletion working
22:08:21 <pabelanger> o/
22:09:11 <pabelanger> Shrews: what about moving the exit below our nodepool delete syntax for now?
22:09:29 <Shrews> pabelanger: we could do that
22:09:39 <jeblair> we might have delete soon anyway... :)
22:09:44 <fungi> that risks sticking around after we fix deletions
22:09:53 <jesusaur> o/
22:09:53 <Shrews> fungi: that was my worry  :)
22:09:59 <jeblair> at least, if it's a choice, i'd rather we just forge on with deleting and then enable the job
22:10:00 <fungi> unless we track undoing that workaround in the job
22:10:25 <pabelanger> ++
22:10:39 <jeblair> pabelanger: how go the cli tools?
22:11:08 <pabelanger> good so far, I'm re-enabling skipped tests at the moment.  I need to look into alien-list commands
22:11:27 <pabelanger> I'll have more patches up for tomorrow
22:12:11 <pabelanger> will need some help with the job-create / job-delete tests, I don't think we have that in zookeeper yet
22:12:35 <jeblair> pabelanger: no; maybe we should defer those for a little longer?
22:12:46 <pabelanger> ya, if people are okay with that
22:13:03 <jeblair> it's a fairly stand-alone feature that i don't think we need for the ptg, so maybe engergy better spent elsewhere for now
22:13:15 <pabelanger> ack
22:14:03 <jeblair> fungi, mordred, Shrews: ^?
22:14:43 <Shrews> what are the job-create/-delete tests?
22:14:52 <jeblair> Shrews: it's the auto-hold feature
22:15:02 <jeblair> we should probably see about renaming the commands to auto-hold too
22:15:12 <fungi> yeah, i don't feel like we need that implemented before next week
22:15:24 <jeblair> auto-hold a node when a specific job fails
22:15:33 <fungi> it shouldn't prevent the planned exercise
22:15:34 <mordred> I agree
22:15:45 <jeblair> also, erm, i don't think we can implement it in nodepool :)
22:15:56 <jeblair> so even better, let's talk about where to put that at the ptg :)
22:16:02 <Shrews> yeah, was just wondering how that would work
22:16:35 <jeblair> (maybe it could become a zuul feature (to instruct nodepool to hold nodes))
22:16:50 <jeblair> #agreed defer job-create/job-delete (auto-hold) related work until after the ptg
22:17:07 <fungi> right, the context for that is primarily in zuul
22:17:35 <jeblair> yeah, i'm happy that nodepool knows less and less about 'jobs'.  :)
22:17:58 <pabelanger> +1
22:18:14 <jeblair> pabelanger, Shrews: anything else on this topic?
22:18:30 <Shrews> Nope
22:18:36 <pabelanger> nothing here
22:18:49 <jeblair> #topic Status updates: Devstack-gate roles refactoring
22:19:04 <jeblair> rcarrillocruz sends his regrets as he can't make this meeting
22:19:34 <jeblair> so unless anyone else wants to jump in here, i'll move on?
22:20:09 <jeblair> #topic Status updates: Zuul test enablement
22:20:53 <jeblair> SpamapS and adam_g have been pushing on this recently
22:21:15 <SpamapS> Not much to report
22:21:32 <SpamapS> I have been distracted away from the main problem I hit Friday but hope to get back to it tomorrow.
22:21:37 <jeblair> i'm about halfway through reviewing the current stack; if anyone else wants to review that would be appreciated
22:21:40 <adam_g> i should have some more cycles to pick away at this starting soon
22:21:59 <jeblair> adam_g: \o/
22:22:12 <SpamapS> I believe there's a problem with the behind_dequeue test that manifests as a timeout waiting for settle (possibly a bug that makes it never settle)
22:22:13 <jhesketh> Yep, I need to catch up on reviews and can focus on that
22:22:49 <jeblair> SpamapS: indeed; i think if someone can sit down with that test and a full pot of coffee, we'd all be the better for it
22:23:08 <jeblair> jhesketh: thanks!
22:23:28 <SpamapS> jeblair: I will have about 3 hours to stare at it tomorrow. :)
22:24:01 * jeblair guesses: SpamapS enjoy your flight!
22:25:04 <jeblair> #Status updates: Zuul Ansible running
22:25:11 <jeblair> #topic Status updates: Zuul Ansible running
22:25:31 <SpamapS> jeblair: almost.. waiting for a shuttle from Reno -> Tahoe ;)
22:25:48 <jeblair> oy
22:26:10 <jeblair> i think my crazy playbook change finally merged
22:26:24 <jeblair> so we're running playbooks and there's some semblance of rules around inheritance, etc now
22:26:40 <jeblair> mordred: it looks like https://review.openstack.org/428798 is about ready
22:26:47 <jeblair> jhesketh: ^ you might be interested in that one too
22:27:06 <jhesketh> cool :-)
22:27:24 <jeblair> SpamapS has reservations about the approach there....
22:27:37 <jeblair> i understand the hesitation there and agree with it
22:27:55 <mordred> I do too
22:27:58 <pabelanger> Yay
22:28:13 <jeblair> the only thing i'd add is that we really do need to get that right, because without it, there's pretty much nothing standing between our army of contributors and compromising the credentials we protect our systems and openstack development with
22:28:38 <jeblair> i also think we should wear belts and suspenders
22:28:42 <SpamapS> I think it's 100% worthwhile to pursue a playbook-friendly approach
22:28:46 <SpamapS> rather
22:28:46 <mordred> yah. I am of the opinion that neither containment nor ansible hacks are good enough by themselves - and we need to do as good a job as we can on both
22:28:48 <SpamapS> playbook-author-friendly
22:29:04 <jeblair> but i want to convey that if we fail at our 'best effort' there, it will be a *significant* failure.
22:29:12 <mordred> yup
22:29:18 <SpamapS> However, the zuul-operator-friendly approach is likely to provide a launcher containment mechanism.
22:29:32 <pabelanger> Ya, going to be a little scary the first time it happens :(
22:29:38 <SpamapS> I wonder if we should work out a spec for what's required.
22:29:44 <jeblair> it needs to not happen :)
22:29:52 <SpamapS> yeah it shouldn't happen. We can think through this. :)
22:30:14 <SpamapS> I mean, other than SSH'ing to boxes and reading local files, the launcher probably doesn't need to do much else.
22:30:20 <jeblair> part of why we stopped using jenkins is security related.  if we replace it with something worse, zomg.
22:31:02 <jeblair> anyway, i don't think anyone disagrees with any of the facts on the ground.  i mostly wanted to emphasize that as a matter of approach.  :)
22:31:08 <mordred> ++
22:31:35 <SpamapS> so feels like we could spec something pretty narrow out. Like, dib up a tarball with ansible in it, bind mount in the relevant data for the job, and execute ansible-playbook in a significantly hobbled container.
22:31:56 <SpamapS> but.. needs a spec and some time spent on it
22:32:26 <mordred> agree. adding the container exceution shouldn't be terribly difficult, it just needs all of our skepticism pointed at it :)
22:32:29 <SpamapS> also somehow
22:32:32 <SpamapS> we have to write some piece
22:32:34 <SpamapS> in Rust
22:32:36 <SpamapS> ;)
22:32:51 * mordred tucks SpamapS back into his rust-hole
22:32:54 * SpamapS sees a setuid Rust binary in our future
22:32:54 <jeblair> anyway.... :)
22:33:10 <jeblair> i still have roles and repo set up on my back-burner
22:33:14 <SpamapS> actually I wonder if we could leverage privsep
22:33:45 <jeblair> but i'm switching gears to 'actually get daemons running' this week in prep for ptg, so not sure if they'll be there by then
22:34:08 <jeblair> mordred: oh, the other thing is, what's the state (or what needs to be done) for telnet console log streaming of shell commands?
22:34:48 <jesusaur> jeblair: what needs to be done for repo set up? that sounds like it might be some long-hanging fruit i can pick
22:34:50 <mordred> jeblair: thank you. I was tyring to remember what the next thing I needed to work on was
22:36:14 <jeblair> jesusaur: implement the parts that pass through the "repos:" part of jobs to the launcher
22:36:55 <jeblair> jesusaur: and yeah, that change probably isn't too extensive
22:37:41 <jeblair> basically, add it to the parser and model, pass it through json to the launcher, and then make sure the launcher adds it to the merge stack (very similar to what i just did for playbook repo setup)
22:37:54 <jesusaur> jeblair: cool, i'll take a stab at that this week
22:38:16 <jeblair> jesusaur: sounds good.  i don't know if it has its own story yet
22:38:47 <jeblair> #topic Progress summary
22:38:53 <jeblair> which brings us to.... :)
22:39:27 <jeblair> SpamapS: i think we may have a couple of new stories we could add
22:39:48 <jeblair> mordred: telnet shell console log streaming
22:39:50 <SpamapS> #link https://storyboard.openstack.org/#!/board/41
22:39:59 <jeblair> jesusaur: repo setup in launcher
22:40:09 <jeblair> jeblair: roles setup in launcher
22:40:17 <SpamapS> mordred: jesusaur I'll add those and assign the main task to you.
22:40:27 <mordred> \o/
22:40:27 <jesusaur> SpamapS: thanks
22:40:37 <jeblair> at least, i don't think they already exist.  i'm working from memory
22:41:48 <SpamapS> Does anyone have any issue with the way storyboard is working out?
22:42:10 <SpamapS> I feel like people are doing a good job of picking up tasks and stories for the bigger things, and I haven't seen much duplication other than accidental test dupes
22:42:23 <jeblair> my main issue is lack of time to work on boartty
22:42:24 <SpamapS> jeblair: I'm searching first.. not finding them. :)
22:42:38 <Shrews> SpamapS: I still avoid it, but I'm probably the oddball
22:42:44 <SpamapS> jeblair: I've been using boartty and I quite enjoy it, when it doesn't crash :)
22:42:58 <SpamapS> Shrews: we all seem to know your turf and don't step on it either :)
22:44:46 <jeblair> let's use the last 15m to finish checking in on ptg prep
22:44:51 <jeblair> #topic PTG prep
22:44:57 <jeblair> #link https://etherpad.openstack.org/p/pike-ptg-zuul
22:45:09 <jeblair> fungi: did you get a chance to poke at the nodepool metadata question?
22:45:23 <fungi> nope, was hoping to look at it today
22:45:31 <jeblair> cool
22:45:38 <fungi> i have a strong recollection it's greedy right now
22:45:49 <fungi> and so may pose issues
22:45:57 <jeblair> okay, it might be nice to fix that before the ptg if we can
22:45:59 <fungi> but need to dig into the source code to confirm
22:46:33 <jeblair> if anyone wants to volunteer to pitch in on a nodepool v2+v3 thing, see fungi
22:46:48 <fungi> i want to say we punted on making it possible to have two nodepools coexist in a single tenant until we needed it to do so, which is i guess now :/
22:46:57 <jeblair> (or nodepool v0+v3 i dunno)
22:47:05 <pabelanger> I'm pretty much ready to bring nl01.o.o online starting tomorrow.
22:47:19 <fungi> pabelanger: awesome!
22:47:22 <jeblair> pabelanger: cool, did all those patches get reviewed/merged?
22:47:39 <pabelanger> not merged, still looking for some more eyes
22:48:20 <jeblair> i think i reviewed them, but if i missed some pls ping me
22:48:22 <pabelanger> zuulv3-dev.o.o is a little more complex, trying to get all the puppet things in place, but going to be a slow process.  I think for that, I'm going to launch a bare server, and slowly start moving things into puppet.
22:48:45 <jeblair> pabelanger: that sounds good
22:49:17 <clarkb> fungi: jeblair just really quickly reading the nodepool v2 code. if you use a different provider name then it will be ignored
22:49:32 <clarkb> so call the new one rax-iad-v3-test or something and it should avoid being deleted by prod nodepool
22:49:41 <jeblair> oh huh, interesting.
22:49:43 <clarkb> (you'll need to check v3 also has that behavior)
22:50:23 <clarkb> provider_name is the metadata we attach to instances
22:50:26 <fungi> clarkb: oh, if it's requiring the provider name to match then we can indeed work around that at least
22:50:34 <clarkb> and if the value there isn't in the current processes config it passes it over
22:50:37 <fungi> v3 doesn
22:50:51 <fungi> 't have leak cleanup yet afaik
22:51:15 <jeblair> indeed, it doesn't have cleanup yet.
22:51:19 <Shrews> fungi: it has NO cleanup yet, but will only delete nodes that are registered with ZK anyway
22:51:30 <Shrews> when ready
22:51:48 <jeblair> we will probably want to add leak cleanup back in as well later on
22:52:03 <Shrews> yeah, but that won't be this week, i don't believe
22:52:11 <fungi> right
22:52:20 <jeblair> yeah, more of a 'before release' thing
22:52:50 <jeblair> any other ptg-prep things we should be talking/thinking/doing?
22:53:07 <jeblair> i sent out my 'zuulv3 primer' email
22:53:12 <jeblair> hopefully that helps.  sorry it's long.
22:53:22 <jeblair> i know some people read it at least :)
22:53:46 <clarkb> I skimmed will have to read properly soon
22:54:15 <jesusaur> will there be a way for those of us not at the ptg in person to follow along from home?
22:54:48 <jeblair> jesusaur: i don't know that we have firm plans, but we have more than one person asking that
22:54:50 <jhesketh> jeblair: it was a good email, thanks for putting that together
22:55:01 <jeblair> knowing us, i'm guessing most of us will have an irc window open
22:55:24 <fungi> yeah, worst case, someone in the room hanging in irc can pass along things to hack on
22:55:27 <jeblair> maybe we can have someone connect up to pbx.o.o
22:56:14 <jeblair> (i'm partly being vague since this is our first one and i don't know what to expect; maybe others have more thoughts)
22:56:53 <jesusaur> i've never tried to use pbx.o.o, but imo irc is a fine plan
22:57:24 <clarkb> pdx worked well when I listened into vancouver sessions
22:57:34 <pabelanger> same for me for Japan
22:57:56 <jeblair> jesusaur: i think you can help by making sure we know you're around and want to participate; we'll need to make sure we communicate scheduling to you and other remote folks so you know to do that :)
22:58:04 <clarkb> wow nice typo on pbx clarkb
22:58:23 <jeblair> jesusaur: hopefully the ethercalc thing will help with that; i don't know if we know what that will look like yet though...
22:58:28 <fungi> someone has airports on the brain
22:58:54 <jeblair> clarkb: p[bd]x is awesome.  and weird.
23:00:04 <jeblair> thanks everyone for all the ptg prep!  and all the other work!
23:00:12 <jeblair> see you next week!
23:00:17 <fungi> thanks, jeblair!
23:00:18 <pabelanger> ++
23:00:19 <jeblair> (possibly virtually)
23:00:21 <jesusaur> ++
23:00:23 <jeblair> #endmeeting