22:03:40 <jeblair> #startmeeting zuul
22:03:41 <openstack> Meeting started Mon Mar  6 22:03:40 2017 UTC and is due to finish in 60 minutes.  The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot.
22:03:42 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
22:03:44 <openstack> The meeting name has been set to 'zuul'
22:04:26 <jeblair> #link agenda https://wiki.openstack.org/wiki/Meetings/Zuul
22:04:39 <jeblair> #link previous meeting http://eavesdrop.openstack.org/meetings/zuul/2017/zuul.2017-02-27-22.03.html
22:04:49 <jeblair> #topic Actions from last meeting
22:05:05 <jeblair> please review lists in storyboard; make sure things are relevant, add in what we're learned. some items may have been reprioritized, though likely not significantly
22:05:10 <jeblair> i guess that was addressed to everyone
22:05:35 <jeblair> i did quite a bit of that.  we'll talk about it later in the progress summary topic
22:05:41 <jeblair> mordred we think is working on blog post covering things that happened at PTG, also laying out info / vague roadmap for community
22:05:47 <jeblair> we think that happened :)
22:06:05 <jeblair> there were even followup questions
22:06:09 <jeblair> rbergeron to draft up zuul v3 status mail for ppl to poke at, then send. and then just be shippin' in weeks afterwards.
22:06:16 <SpamapS> That happened
22:06:23 <jeblair> indeed
22:06:49 <fungi> and an excellent read it was
22:06:57 <jeblair> we should try to remember to sprinkle lots of #info, etc, in the meeting to help with that
22:07:26 <jeblair> though hopefully rbergeron will show up because she's really good at that
22:07:45 <jeblair> #topic Status updates: Nodepool
22:07:59 <jeblair> Shrews: what's the latest?
22:08:02 <Shrews> I fixed a thing
22:08:18 <jeblair> the 'out of quota' thing?
22:08:22 <Shrews> yeah
22:08:35 <jeblair> is that running in infra 'production'?
22:08:35 <Shrews> https://storyboard.openstack.org/#!/story/2000886
22:08:54 <Shrews> not that i'm aware of. just merged today
22:09:17 <Shrews> and we now group nodes for a request within the same AZ
22:09:21 <pabelanger> I haven't restarted nl01.o.o in some time
22:09:27 <jeblair> #action Shrews restart nodepoolv3 launcher to pick up quota fix
22:09:34 <jeblair> Shrews: mind if i action you on that? :)
22:09:59 <Shrews> jeblair: better to ask forgiveness than permission, eh?
22:10:01 <Shrews> sure
22:10:13 * jeblair keeps the #undo card in his pocket
22:10:42 <Shrews> just need to add back stats and do leaked node cleanup, then i think nodepool will be complete
22:10:48 <Shrews> if folks stop finding bugs, that is
22:10:54 <jeblair> no promises
22:11:02 <pabelanger> Shrews: ssh-keyscan too :)
22:11:05 <jeblair> #info nodepool allocation algorithm improved to handle at-quota case
22:11:33 <jeblair> #info remaining tasks for nodepool feature parity: leaked node cleanup and statsd support
22:11:34 <Shrews> pabelanger: is that a requirement or a nice-to-have?
22:11:59 <pabelanger> Shrews: good question. I'd like to have it but will defer to jeblair
22:12:37 <jeblair> i think we should get it in early... most of the code is already written (just in the wrong program), and it's a (minor) change to the nodepool-zuul protocol, so easier to do now
22:13:13 <Shrews> jeblair: which program does it reside in now?
22:13:25 <jeblair> but i'd rank it after the other things
22:13:42 <jeblair> Shrews: it's in zuul
22:13:56 <jeblair> Shrews: now that i say that, i don't actually know whether we want to run ssh-keyscan, or use paramiko.
22:14:12 <jeblair> so er, the code may not mostly be written.  but either way, it should not be much.
22:14:53 <jeblair> https://storyboard.openstack.org/#!/story/2000897
22:15:17 <Shrews> If no one else jumps on it, I'll do it at the end of the other tasks.
22:15:21 <jeblair> that's tagged low-hanging-fruit, btw.
22:15:29 <jeblair> so a good opportunity for someone else to jump on it
22:16:09 <jeblair> continuing on the nodepool topic, i was hoping that mordred would be here to clue us in on the status of the shim
22:16:28 <jeblair> because i think once we're at feature parity, we should be in a position to consider using it and getting scale testing
22:16:48 <jeblair> does anyone else know the status of that?
22:17:14 <Shrews> I haven't seen anything other than the 2 current reviews he has up
22:17:20 <Shrews> and they've been idle a while
22:17:21 <jeblair> i see changes up, but they fail tests, so i'm not sure if they're ready for review or not...
22:17:38 <Shrews> jeblair: they will always fail tests. i think we'll have to disable them
22:17:54 <Shrews> that was the plan, I believe
22:18:00 <pabelanger> not sure either
22:18:15 <jeblair> ah, well, that still needs to be done then, since we can't merge them until then :)
22:18:34 <jeblair> the changes still have lots of todos
22:19:01 <jeblair> moving on...
22:19:30 <jeblair> is rcarrillocruz, clarkb,  or anyone else able to talk about devstack-gate roles refactoring?
22:20:14 <rcarrillocruz> would be good to get reviews on https://review.openstack.org/#/c/435933/
22:20:26 <jeblair> #topic Status updates: Devstack-gate roles refactoring
22:20:35 <jeblair> #link need reviews on  https://review.openstack.org/#/c/435933/
22:20:40 <rcarrillocruz> thx!
22:21:14 <jeblair> rcarrillocruz, maybe we need clarkb to review that?
22:21:25 <rcarrillocruz> yeah, for sure
22:22:00 <jeblair> #action clarkb review https://review.openstack.org/#/c/435933/
22:22:31 <jeblair> anyone else with knowledge of ansible and/or networking please take a look at that :)
22:23:12 <Shrews> jeblair: fwiw, my action item is complete. nodepool-launcher restarted on nl01
22:23:22 <jeblair> rcarrillocruz: thanks
22:23:34 <jeblair> Shrews: gee, i think we have to remember that until next week.  ;)
22:23:45 <jeblair> #topic Status updates: Zuul test enablement
22:24:01 <Shrews> i'm guaranteed to forget
22:24:02 <SpamapS> I saw another batch land the last few days
22:24:56 <jeblair> I've been working with cullen taylor (nick?) on test_tags
22:25:04 <SpamapS> I believe we have 25 outstanding skips, with something like 10 in review/todo
22:25:10 <SpamapS> Cullen == eggshell
22:25:20 <jeblair> cool
22:25:42 <jeblair> i think i'm getting to the point where i'm ready to say we should drop all the layoutvalidator tests
22:25:47 <SpamapS> The swift test is also still blocked on auth being implemented
22:26:19 <jeblair> when i added the 'report config errors' changes, i also added (minimal) tests for those, and i think they cover the same ideas in spirit
22:26:58 <jeblair> so let's say i'll give the layoutvalidator a once-over and see if it's testing some class of error that would still apply but is not covered in the new config error reporting, and otherwise remove it
22:27:13 <SpamapS> Yeah that should chip a few away
22:27:29 <jeblair> #action jeblair consider removing layoutvalidator and tests
22:27:44 <SpamapS> note that I think 2000773 is so big it crashes firefox
22:27:53 <SpamapS> I can only really interact with it with a fresh Firefox
22:29:03 <SpamapS> jeblair: I wonder if we should just start a second one and maybe report bugs in storyboard
22:29:08 <jeblair> SpamapS: jhesketh and i also spoke at the ptg about the swift support, and i think we came to the conclusion that it's a good candidate for removal in its current form, but we should look into adding some 'standard library' type support for it (since it's very much a potentially useful artifact publishing mechanism)
22:29:25 <SpamapS> OH
22:29:35 <SpamapS> yeah that would be nice to do in ansible
22:29:51 <jeblair> jhesketh: hopefully i characterized that correctly?
22:30:09 <fungi> agreed, anything that can be safely extracted into ansible playbooks simplifies zuul itself
22:30:34 <jhesketh> Yep. There isn't any obvious reason to special case it
22:30:39 <fungi> even if the playbooks end up in a sort of stdlib
22:30:41 <SpamapS> btw the best count I can come up with is 25 @skips left, 7 in review.
22:30:54 <jhesketh> It's the same class citizen as any other uploader this way
22:31:31 <jeblair> so those two things should knock the total down a bit
22:31:32 <jhesketh> But does need the auth stuff for the signing key
22:31:50 <jeblair> jhesketh: yeah, i think we can implement that with the auth mechanism we have planned
22:32:14 <jhesketh> Exactly :-)
22:32:20 <SpamapS> cool, so the focus should be on auth
22:32:23 <SpamapS> wasn't rcarrillocruz working on that?
22:33:19 <jeblair> yes, there's a wip for part of it here: https://review.openstack.org/406382
22:34:11 <SpamapS> k
22:34:21 <jeblair> but i know he's busy and has mentioned that if we need someone to take it over for him, we can reassign it [speaking for rcarrillocruz since i know it's late there...]
22:34:43 <SpamapS> probably another week or two before we _have_ to have it to make progress
22:34:46 * rbergeron waves from airport, sorry i missed things thus far
22:35:00 <jeblair> SpamapS: sounds about right
22:35:24 <jeblair> SpamapS: i also promised to triage the remaining skips.  i haven't done that yet
22:35:35 <jeblair> #action jeblair remove swift and layoutvalidator as appropriate
22:35:45 <jeblair> #action jeblair triage remaining skipped tests
22:35:52 <SpamapS> mmmmmm less skips
22:36:59 <jeblair> anything else on tests?
22:37:26 <jeblair> rbergeron: hopefully i #'d enough things to help :)
22:37:45 <jeblair> #topic Status updates: Zuul sample jobs
22:37:51 <jeblair> oh i should have mentioned this earlier
22:37:57 <jeblair> i swapped out the 'zuul ansible running' status updates for this
22:38:12 <jeblair> since i think we're more or less running ansible now....
22:38:31 <jeblair> pabelanger has started an effort to create some base jobs
22:38:39 <jeblair> this is an extension of the work at the ptg
22:39:06 <jeblair> and i think that eventually it will serve two things: the foundation for our jobs in openstack, as well as the basis of the thing we keep calling the 'standard library'
22:39:18 <pabelanger> yup, see +topic:zuulv3-ansible
22:39:25 <jeblair> so i'd like to get lots of folks looking at this from a number of different perspectives:
22:39:49 <jeblair> will this work for openstack-infra?  will this work for other zuul users?
22:40:15 <jeblair> does it make sense to an ansible user?  does it make sense to a random developer who isn't particularly interested in using ansible per se?
22:41:00 <jeblair> some of these are contradictory -- not all aspects have to be all things to all people.  but knowing what audience each job or component is aiming for will help evaluate it.
22:41:07 <SpamapS> jeblair: is the idea that a one-time automatic conversion can be seeded by these base jobs?
22:41:20 <pabelanger> Right, it is heavily geared towards openstack-infra right now. We are still depending on bash scripts for most things
22:41:56 <jeblair> SpamapS: partly yes -- for instance, one class of jobs "gate-.*-python27" will be an automatic transition to a hand-crafted "tox-py27" job
22:42:22 <jeblair> SpamapS: other automatic transitions will be more ugly -- a random job that runs a shell script will probably automatically generate an ugly ansible "shell" task
22:42:29 <SpamapS> I like that very much. Seems like those most interested in maintaining those job definitions would be good to coopt here.
22:42:47 <jeblair> SpamapS: but when we talk about the tox python job, i think we should be looking at making the best, simple, zuul+ansible way of running tox
22:43:00 <SpamapS> yeah cover 80/20
22:43:11 <SpamapS> maybe even better
22:43:23 <jeblair> something that any openstack dev who, today, makes a patch that cargo-cult's some stuff in project-config/jjb will find just as comprehensible, if not more so
22:43:24 <pabelanger> jeblair: does that mean we want to refactor our run-tox.sh script now too?
22:44:21 <jeblair> pabelanger: it's worth thinking about -- but if something like that gets gnarley, we can always defer it until later
22:44:24 <SpamapS> jeblair: will there be room in zuul for other language std jobs post-infra-migration do you think?
22:44:50 <SpamapS> like, a 'rust-cargo-test' job?
22:45:22 <SpamapS> it's a bit too far down the road to think deeply about
22:45:42 <pabelanger> jeblair: I could go ether way, I'm aiming for easy more right now in our base jobs. But, if the time is right, we can start removing our dependency on jenkins/scripts folder too
22:45:43 <SpamapS> but it's something that's coming to mind for me, as we start to poke at the in-repo job definition language and compare it to .travis.yml
22:45:47 <jeblair> SpamapS: i think one of the choices we're going to get to make in the medium-term is how expansive we want our stdlib to be (also, technically, how do we implement it and where does it live?  in zuul, or in a new repo?).  my personal wish is that we can be somewhat expansive in that, and take the approach we did with jjb and be radically accepting of whatever people offer :)
22:45:57 <fungi> also want to make sure we don't end up with a kitchen sink stdlib like the one that has congealed inside dib (no offense to dib devs/reviewers, but i gather they too are eager to split that up soon)
22:46:13 <fungi> python's "batteries included" approach seems like a sane enough goal though
22:46:37 <jeblair> fungi: i agree with you too, and i don't know if that means i hold two contradictory opinions.  :)
22:46:46 <jeblair> fungi: 'batteries included' is nice :)
22:46:50 <SpamapS> Ansible shares this problem.
22:47:04 <SpamapS> and is radically accepting of things people maintain, IIRC
22:47:22 <SpamapS> anyway, just food for thought
22:47:25 <jeblair> SpamapS: i think i can safely say that if it's very expansive, it won't be in the zuul repo.  :)
22:47:31 <fungi> we're already making it easy to use multiple repos for playbooks, so having a standard library that ships with zuul and another one for the more openstacky/ancillary resuable things could be a good compromise?
22:47:48 <jeblair> and yeah, the other thing is: the actual way we want this to be expansive is ansible roles...
22:48:03 <jeblair> so the more we can "just use galaxy" the better
22:48:29 * SpamapS raises eyebrows at galaxy quality... :-/
22:48:30 <jeblair> (and maybe the stdlib is a role repo in galaxy?)
22:48:33 <Shrews> would importable stdlibs be a thing we'd want to consider? or is that unsafe?
22:48:52 <SpamapS> would be quite nice to just have them in a galaxy-like collection though. Just not sure I love galaxy much.
22:49:35 <SpamapS> Shrews: I think they have to be vetted by us to be included as batteries... because they will run in the secure context.
22:50:00 <jeblair> anyway -- these are all things we should keep in mind -- but we should know that they aren't answered questions yet.  at the moment, pabelanger has some changes up that we should review with these things in mind.  but also, we will land them (at least temporarily) in the zuul repo so we can continue to work through these things and shape them.
22:50:58 <jeblair> let's move on before we run out of time
22:51:07 <jeblair> #topic Progress summary
22:51:13 <SpamapS> oo I get to link
22:51:19 <SpamapS> #link https://storyboard.openstack.org/#!/board/41
22:51:32 <SpamapS> The board is _deep_
22:51:32 <jeblair> i filed a bunch of new stories and tidied up some things
22:51:37 <SpamapS> Yeah it looks great
22:51:44 <SpamapS> thanks for grooming
22:51:51 <jeblair> i also tagged a lot of things as low-hanging-fruit
22:52:48 <Shrews> SpamapS: feel free to mark https://storyboard.openstack.org/#!/story/2000886 done
22:53:09 <jeblair> it's possible that my threshold for LHF is slightly higher than it was previously -- i think at this point i'd describe it as "able to be done without deep knowledge of the whole system, but will require some familiarity with at least one component or subsystem in order to complete (but can be learned in progress)"
22:53:26 <jeblair> but that's just where we are right now :)
22:53:47 <SpamapS> Shrews: got a change ID / review link I can mention to close the loop?
22:54:09 <SpamapS> jeblair: I think that's a good bar for the contributors we have and are ramping up
22:54:27 <Shrews> SpamapS: https://review.openstack.org/440780
22:54:30 <SpamapS> Shrews: thanks
22:54:34 <jeblair> i feel like a lot of the major subsystem refactors that were blocking are resolved now, and we're sort of at the top of the hill where more folks should be able to dive in more easily
22:55:02 <pabelanger> ++
22:55:47 <SpamapS> so if everyone can just make sure their assigned things are up to date or if you see something you're doing that isn't assigned to you, let me know or assign yourself, that helps a lot thanks
22:56:02 <SpamapS> jeblair: yes it's definitely more approachable now
22:57:26 <jeblair> SpamapS: i don't know how to search for zuulv3 and low-hanging-fruit
22:57:47 <jeblair> SpamapS: we may need to make sure that people who want to find lhf have a good way of doing that
22:58:16 <SpamapS> jeblair: yeah I can't do the intersection either. Seems like a bug in storyboard
22:58:16 <jeblair> (i assumed i could search for both, but when i search for one tag, then another, it replaced the first)
22:58:33 <SpamapS> jeblair: I'll take an action to figure that out
22:58:43 <jeblair> also, i spelled it "low-hanging-fruit"
22:58:52 <jeblair> there's another lhf tag in there without the second hyphen
22:59:06 <SpamapS> we could re-tag them zuulv3-lowfruit ro something
22:59:10 <SpamapS> or
22:59:16 <SpamapS> lame, but a workaround if storyboard is broken
22:59:31 <jeblair> SpamapS: would need to update the board if we do that
22:59:40 <SpamapS> jeblair: yeah
23:00:06 <SpamapS> we're about out of time
23:00:08 <jeblair> SpamapS: oh, we can at least search for low-hanging-fruit and openstack-infra/zuul
23:00:19 <jeblair> thanks everyone!
23:00:22 <jeblair> #endmeeting