22:04:18 <jeblair> #startmeeting zuul
22:04:19 <openstack> Meeting started Mon May 22 22:04:18 2017 UTC and is due to finish in 60 minutes.  The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot.
22:04:20 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
22:04:22 * mordred waves
22:04:23 <openstack> The meeting name has been set to 'zuul'
22:04:36 * jeblair searches pockets for agenda
22:04:43 <jeblair> ah here we are
22:04:50 <jeblair> #link agenda https://wiki.openstack.org/wiki/Meetings/Zuul
22:05:06 <SpamapS> o/
22:05:06 <jeblair> #link previous meeting http://eavesdrop.openstack.org/meetings/zuul/2017/zuul.2017-05-01-22.05.html
22:05:15 <jeblair> #topic  Announcements
22:05:38 <jeblair> I have been reliably informed that
22:05:42 <jeblair> #info SpamapS has been added to Zuul and Nodepool core teams
22:05:47 <mordred> \o/
22:05:47 <fungi> thanks for agreeing to pitch in on that!
22:05:57 * mordred hands SpamapS a fish he found laying on the ground
22:05:59 * jlk claps
22:06:02 <tristanC> nice, congrats SpamapS!
22:06:05 <jeblair> SpamapS: thanks for the reviews!  and stuff!
22:06:12 <SpamapS> Mmmm fish
22:06:22 <SpamapS> Thanks for the code!
22:06:47 <fungi> condolences seem more appropriate for core duties ;)
22:06:58 <jeblair> fungi: you used your outside voice again
22:07:04 <fungi> oops!
22:07:16 * rbergeron sends a nice package of stay-puft marshmallows as a honorarium
22:07:35 <jeblair> #topic Actions from last meeting
22:07:45 <SpamapS> I did roast marshmallows this weekend in zuul's honor
22:07:51 <jeblair> pabelanger create new ppa under openstack-ci-core and add bubblewrap to it; copy existing source package from ansible bubblewrap ppa.
22:07:53 <rbergeron> you were camping :)
22:08:18 <jeblair> pabelanger: if you've shaken the summit off... what's the status of the bubblewrap ppa thingie?
22:08:36 <pabelanger> jeblair: it is done
22:08:39 <fungi> i think i ended up creating the ppa in the end due to silly lp permissions
22:08:46 <pabelanger> https://review.openstack.org/#/c/461849/
22:08:59 <jeblair> cool.  one point for pabelanger and one for fungi then.
22:09:15 <jeblair> two points from slytherin just cause.
22:09:25 <fungi> minus one point for lp's permissions model
22:09:33 <clarkb> ya project owners can only create them
22:09:39 <clarkb> and the openstack-admins group owns it
22:09:44 <pabelanger> Ya, I wasn't part of that
22:10:22 <jeblair> pabelanger: is that held up on the since-fixed sphinx thing?
22:10:37 <jeblair> yeah, looks like pabelanger just rechecked it
22:10:43 <pabelanger> ya, it should be green now
22:10:58 <jeblair> #info bubblewrap testing can commence once  https://review.openstack.org/#/c/461849/  lands
22:11:14 <jeblair> pabelanger, fungi: thanks!
22:11:27 <jeblair> #topic Status updates: Zuul test enablement
22:11:42 <SpamapS> I took last week off from that to focus on py3k :)
22:11:51 <jeblair> SpamapS: good choice, as it turns out
22:12:12 <SpamapS> indeed, we're now py35 compatible :)
22:12:22 <mordred> tobiash even has a v3 running in python3 now!
22:12:31 <SpamapS> *nice*
22:12:55 <SpamapS> I tried to do the same for Bonny but we have some delta still for github
22:13:01 <jesusaur> w00t, new pythons
22:13:03 <jeblair> one of the next things i want to do is finish moving the cloner branch logic into the executor (along with adding some bits to the job language).  that will move/obsolete most of the cloner tests (and the cloner itself, as it exists today).
22:13:15 <mordred> ++
22:13:34 <SpamapS> on that topic, the tests left are a few hairy ones
22:13:44 <jeblair> i'm not sure how many tests the cloner accounts for in our deficit (is that 15 or just one? anyway...)
22:13:53 <jesusaur> jeblair: is that part of the effort to make the executors push git repos to the workers?
22:14:05 <SpamapS> well we have skips for py3 now
22:14:15 <SpamapS> so we really have two re-enablement efforts
22:14:17 <jeblair> jesusaur: yep
22:14:44 <mordred> SpamapS: hopefully the py3 skips are easy to pick off
22:14:48 <SpamapS> "Fails on py3" tests in test_scheduler should be prioritized as first.
22:14:55 <jeblair> jesusaur: (to be fair, we're doing that part now; so this is more "doing it right" :)
22:15:09 <SpamapS> They were really confusing fails, not obvious at all.
22:15:28 <mordred> SpamapS: ++ - there was a thing tobiash thought was a bug earlier, but was a config bug ... but it was of hte order of "confusing bug"
22:15:44 <jeblair> also worth noting -- it *seems* something about the GC/lifecycle of git repo objects we were counting on in py2 has changed in py3, causing sporadic failures
22:16:07 <jeblair> #link debugging python3 git repo gc issues: https://review.openstack.org/466810
22:16:31 <SpamapS> indeed, so we're not going to backslide on py3 but we still have the last mile to go
22:17:00 <SpamapS> I'd very much appreciate some fresh eyes on those skipped tests.
22:17:32 <pabelanger> Nodepool should also been python3 compat now too, just need a few more reviews
22:17:35 <jeblair> Shrews brought up the suggestion of making the py3 tests nonvoting if we can't catch that one soon.  i'd like to spend some more time trying to figure it out before we do that though.
22:17:47 <SpamapS> pabelanger: I'll take a look :)
22:17:48 <pabelanger> also have experimental dsvm jobs in progress
22:18:20 <SpamapS> I'd say take a day or two to debug before we demote the tests.
22:18:23 <mordred> SpamapS: my stack is still stupid long, but I just added looking with fresh eyes to my list
22:18:25 <SpamapS> Since it's racey... we need data
22:18:51 <SpamapS> but I guess non-voting would still provide data
22:19:13 <clarkb> also worth noting that python3 garbage colelctor has had at least one nasty bug in the past
22:19:41 * Shrews cannot mordred.stack.pop() faster than all the mordred.stack.push()'s
22:19:45 <jeblair> (another option is that we disable the git repo gc check; however, it has always shown legit bugs in the past)
22:20:11 <rbergeron> shrews: lol
22:20:13 <jeblair> anything else?
22:20:48 <jeblair> #topic Status updates: Zuul sample jobs
22:21:15 <jeblair> pabelanger has been doing a lot of work here (some of which just merged)
22:21:46 <mordred> \o/
22:22:33 <jeblair> i think it's getting tricky to keep track of the bits that we expect to share and the bits that we don't with all of the jobs in the same repo
22:23:01 <jeblair> i think we should go ahead and create the zuul-base-jobs and zuul-jobs repos, and move things we expect to share in there
22:23:10 <mordred> ++
22:23:21 <mordred> I think having them be future theoretical things is hard
22:23:40 <jeblair> yep, i keep tripping over myself saying "well, this job when it's in the other repo will be like..."
22:24:05 <fungi> wfm
22:24:05 <jlk> more repos, more chances to test cross-repo testing!
22:24:28 <jeblair> pabelanger: is pushing up the repo creation requests on your plate?
22:24:52 <pabelanger> currently no, but I can add it
22:25:09 <jeblair> pabelanger: thanks!
22:25:21 <jeblair> #action pabelanger create zuul-base-jobs and zuul-jobs repos
22:25:56 <jeblair> considering we're not sure we like the separate openstack-zuul-jobs and openstack-zuul-roles repos, let's just start with "-jobs" for now, and not create any more "-roles" atm.
22:27:08 <jeblair> i think once that's done, and we have some basic structure set up, we should dust off the devstack ansiblification soon.  we're going to be ready for that soon.
22:27:43 <jeblair> (probably shortly after i finish this "check out the right branch" stuff :)
22:28:17 <jeblair> anything else on jobs?
22:28:34 <SpamapS> I don't remember why we split roles and jobs. Is that written down somewhere?
22:28:49 <pabelanger> nothing on myside
22:29:46 <jeblair> SpamapS: experimenting with how best to set up these sorts of repos?
22:30:52 <rbergeron> because roles should hopefully be reusable in different jobs?
22:30:57 <rbergeron> (maybe?)
22:31:22 <jeblair> rbergeron: yeah... maybe you want the tox roles but no tox jobs?
22:31:43 <mordred> I have a hunch there will be many repo-organization learning experiences over the next 6 months
22:31:49 <jeblair> (though ponting at a zuul repo to get roles without jobs is also a thing we need to do for third-party ci)
22:32:04 <jeblair> mordred: yep.
22:32:13 <clarkb> the jobs are explicitly enabled per project though right? so just including the repo will get you the roles but not necessarily the jobs?
22:32:40 <jeblair> and even at this point, i don't think that we've learned much more than "it's annoying having them in two repos when you're starting out and keep re-orging the entire system".  :)
22:32:49 * jlk would suggest not pre-emptively exploding the repo count
22:33:04 <jeblair> clarkb: yeah, but they are still *defined* as soon as zuul sees a .zuul.yaml
22:33:44 <jeblair> so fundamentally the question addressed by two repos is "i want to use the zuul stdlib tox role, but not the zuul stdlib tox job definition.  how do i do that?"
22:33:47 <pabelanger> ya, right now our base job points to openstack-zuul-roles, so today all jobs get the roles
22:34:01 <rbergeron> i think the "experimenting" to figure otu what makes sense... makes sense. imo
22:34:04 <rbergeron> :)
22:34:22 <jeblair> (where the actor in my question is a zuul operator)
22:34:39 <jeblair> right now, the only way is "split them into two repos".  in the future, we will have that option and also "zuul operators can specify which configuration objects to load from which projects"
22:35:32 <jeblair> jlk: yeah, it's easier to add later than delete them
22:35:34 <jlk> can jobs in one repo mask jobs in another? Is there a priority stack?
22:35:49 <jeblair> jlk: they can not; job definitions with the same name in multiple repos is a configuration error
22:35:57 <jlk> OIC
22:36:00 <clarkb> which caused trouble when moving jobs around
22:36:03 * rbergeron would almost wonder if having a openstack-galaxy might be interesting but it may be early for that -- but i also think that if we want to leverage (or even enhance) the ansible ecosystem... following those patterns might be good / wise / sane
22:36:09 <jeblair> (thus, this is sort of important :)
22:36:54 <rbergeron> (or even if role -- even if it looks like a role -- needs a better type of terminology here, might be a thing)
22:37:26 <jeblair> rbergeron: indeed, a lot of these are pretty specialized roles, and we occasionally run into confusion around that
22:38:06 <jeblair> let's move on so we don't starve other topics
22:38:13 <jeblair> #topic Status updates: Github
22:38:13 * mordred is hungry
22:38:21 <jeblair> (or our developers!)
22:38:36 <mordred> oh - I reviewed your rework of the plugin interface - did I vote on it?
22:38:44 <jeblair> mordred: yes!
22:38:52 <mordred> yay!
22:39:00 <mordred> I forget to click the button sometimes
22:39:01 <jlk> I think we have enough votes for W+
22:39:26 <jeblair> #link https://review.openstack.org/466105 is the change in question
22:39:34 <jlk> which will allow me to more easily rebase the rest of the stack on it.
22:39:47 <jeblair> we paused the github branch review/merging process to do that
22:40:01 <jlk> I have few obligations this week so I should be able to get the rebase done quickly and up for review again.
22:40:29 <jeblair> it cleans up one of the larger areas which we knew was pretty gnarly.  it's a smallish user-facing change as well as an api cleanup, so i asked a few other folks to give it a quick once-over
22:40:30 <mordred> I have the gh follow up patch open in the browser for review
22:41:05 <jeblair> things are starting to feel actually extensible like we have a real api now.  almost.  :)
22:41:13 <rbergeron> jlk: huge kudos on that, btw -- /me sends you a candy bar or a bicycle or something exciting. virtual bicycle. that is. :)
22:41:14 <mordred> ikr?
22:41:22 <rbergeron> all the hugs
22:41:36 <jlk> I'll settle for a round in London :D
22:41:54 <rbergeron> jlk: i can do that. i will be expensing it but the love is all the same
22:42:00 <jeblair> #topic Status updates: Bubblewrap
22:42:17 <jeblair> SpamapS: once pabelanger's change merges, are we completely unblocked on this?
22:42:29 <jlk> bubble wrap to bindep just merged
22:42:30 <jeblair> ha!
22:42:30 <jeblair> 22:42 < openstackgerrit> Merged openstack-infra/zuul feature/zuulv3: Add bubblewrap to bindep / test-setup.sh  https://review.openstack.org/461849
22:43:01 <jeblair> openstackgerrit has comic timing
22:43:12 <rbergeron> lol
22:43:27 <fungi> nice
22:44:04 <pabelanger> did we figure out the zuul.paths issue for bubblewrap?
22:44:07 * SpamapS +A'd
22:44:22 <SpamapS> pabelanger: what issue was that?
22:44:30 * SpamapS has been fighting ssh-agent for a bit
22:44:42 <jeblair> ah, ssh-agent, that's the next blocker i guess
22:44:58 <pabelanger> pip install uses symlinks, which are outside bubblewrap for testing
22:45:01 <jeblair> #link next step in bubblewrap https://review.openstack.org/462712
22:45:03 <pabelanger> https://review.openstack.org/#/c/461881/
22:45:07 <SpamapS> pabelanger: oh that, no we didn't figure that out
22:45:11 * jlk read that as zuul.pants
22:45:22 <mordred> jlk: you NEVER want to get zuul.pants wrong
22:45:26 <jeblair> #link other next step in bubblewrap https://review.openstack.org/461881
22:45:35 <jlk> mordred: that's what the belt and suspenders are for.
22:45:38 <SpamapS> I think the two thoughts were 1) library/pypi for the shared bits, 2) just copy pasta into the actions
22:45:39 <clarkb> pabelanger: thats due to pip install -e right?
22:45:44 <clarkb> pabelanger: could just stop doing that
22:45:52 <pabelanger> ya
22:46:34 <jeblair> i like copy pasta into actions
22:46:46 <jeblair> there's no reason the action plugins actually need to *import zuul itself*
22:46:57 <SpamapS> jeblair: +1
22:49:19 <jeblair> #topic  Progress summary
22:49:51 <SpamapS> #link https://storyboard.openstack.org/#!/board/41
22:50:04 <SpamapS> feels a little stalled
22:50:41 <rbergeron> we did just come out of summit and all that crap.
22:50:48 <SpamapS> True
22:50:49 * rbergeron uses her backspace key wisely with a more appropriate word
22:50:53 <SpamapS> https://storyboard.openstack.org/#!/story/2000792 is almost done
22:51:15 <jeblair> it would be great if folks looking for a way to help would check that board
22:51:17 <SpamapS> I forgot to mention it in the patches
22:51:43 <jeblair> and if you identify an issue that needs to be worked on before we can start minimally using v3 in production, please add it to that list
22:51:49 * SpamapS moves it to in progress
22:52:16 <jeblair> we're pretty close, but there's still a good chunk to be done, and we need to stay focused
22:53:25 <jeblair> #topic Open Discussion
22:53:36 <Shrews> mordred: pretty sure i know the answer, but has there been any movement on the gear-zk shim?
22:53:43 <rbergeron> I haz a thing also
22:54:17 <mordred> Shrews: nope - sorry
22:54:36 <mordred> jeblair: so - 2 questions came up earlier today that it seemed we should have an answer for
22:55:09 <tristanC> regarding static node support, is there a spec/change already?
22:55:09 <jeblair> Shrews, mordred: if mordred finishes the shim, it'll still be useful for the transition, however, i'm fairly happy with the load testing nodepool is getting, and we will be able to ramp it up slowly.
22:55:21 <mordred> jeblair: one is where to put tobiash's Dockerfile -- it seemed the thing people liked the most so far as "in another repo, such as zuul-docker or docker-zuul or something"
22:55:35 <jeblair> tristanC: yes, see https://review.openstack.org/461509 (which i should add to tomorrows infra team meeting to get approved)
22:55:46 <jeblair> tristanC: (spec yes, change no)
22:56:05 <clarkb> fwiw, I think that adding a ton of extra repos makes it harder for people jumping in to discover/use tools like that which seem largely aimed at make quickstart quick and easy
22:56:08 <tristanC> perfect, thanks!
22:56:14 <jeblair> tristanC: tobiash was interested in potentially doing some work related to that; we should sync with him if you're also interested
22:56:24 <clarkb> (can document, but no one reads the docs :P )
22:57:00 <jeblair> mordred: yeah, i feel like we're all probably on the same page about not wanting to mislead people in thinking that "docker", "vagrant", etc are the way to run zuul
22:57:17 <jlk> clarkb: I agree, but it may be too early to dump a definitive docker file in the zuul repo. There are a lot of things to sort out around deploy/run zuul/nodepool
22:57:23 <jeblair> so either in-repo in contrib/$file, or in another repo.
22:57:34 <jeblair> see, jlk just said "definitive" :)
22:57:45 <jeblair> which i think illustrates the problem
22:57:53 <clarkb> I'm not sure why it would be definitive... especially if its in eg contrib/docker
22:57:55 <tristanC> and regarding a zuul dashboard, are people interested to define the rest api?
22:57:57 <mordred> yah. I can see a day in the future where we may want to provide a docker image for people - but I don't think we're there yet
22:58:06 <jlk> contrib/ makes that somewhat clear
22:58:23 <mordred> contrib does not bother me at all
22:58:28 <jeblair> mordred: yeah if/when that happens, easy to promote it up.
22:58:31 <mordred> ++
22:58:32 <jlk> I _do_ hope we get to the point (soon) where we can document Docker/k8s as a way to deploy and run Zuul in production.
22:58:48 <jeblair> i can live with contrib/ but i'd like to know if we have some folks willing to approve patches to that
22:58:52 <jeblair> because i will not be
22:58:57 <rbergeron> :)
22:59:01 <jlk> I'll be reviewing them
22:59:08 * rbergeron will throw in before the hour is up --
22:59:08 <Shrews> i feel any deploy thing should be separate from the thing being deployed
22:59:12 <rbergeron> so: I have been utter crap on the zuul-updates mail lately -- between travel and boss having emergency surgery crap and filling on for everything on both fronts, life sucked :D -- but i would like to get on that train again, so I will chat with ppl tomorrow in channel about coallating the last... 1.5 months of that, or so. so heads up that i will beep tomorrow. :)
22:59:12 <jlk> I left numerous commentary in the existing changes.
22:59:13 <jeblair> (on account of i can't review them)
22:59:15 <pabelanger> Shrews: +1
22:59:22 * rbergeron apologizes for sucking
22:59:40 <jeblair> rbergeron: a new update would be great, thank you!
22:59:46 <mordred> the other thing is tristanC's dashboard patch - but I think we're out of meeting time to dig in to it ... I can try to send an email to the mailing-list with thoughts I had in my brain for folks
23:00:04 <jlk> shoot I really need to get on that list
23:00:10 <jeblair> mordred, tristanC: can you add items to the agenda next time?
23:00:16 <jeblair> i didn't know we had things queued up for the meeting
23:00:22 <mordred> jeblair: yah - I totaly forgot to pre-add this time
23:00:25 <mordred> that's my bad
23:00:27 <jeblair> there's a place " General topics (add agenda items here)" where you can add them :)
23:00:34 <fungi> thanks for starting that back up rbergeron!
23:00:39 <jeblair> i will happily push things around to accomodate stuff there
23:00:44 <mordred> ++
23:00:51 <tristanC> thanks!
23:00:54 <jeblair> unfortunately, i don't know that we have come to a consensus on the docker thing
23:00:57 <mordred> well - I'll start a thread - and maybe we won't even need a topic next time
23:01:10 <jeblair> so let's put docker back on next week's agenda
23:01:13 <mordred> ++
23:01:15 <rbergeron> fungi: np. it's one of the things i can actually do to help atm, it's just .. worklife has been a juggling game lately :)
23:01:34 <rbergeron> but it's useful to have folks know what's coming (vital, kinda, really ;D)
23:01:35 <jeblair> thanks all!
23:01:38 <jeblair> #endmeeting