22:01:57 <jeblair> #startmeeting zuul
22:01:58 <openstack> Meeting started Mon Mar 27 22:01:57 2017 UTC and is due to finish in 60 minutes.  The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot.
22:01:59 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
22:02:01 <openstack> The meeting name has been set to 'zuul'
22:02:05 <jeblair> #topic  Actions from last meeting
22:02:15 <jeblair> oh whoops
22:02:21 <jeblair> #link https://wiki.openstack.org/wiki/Meetings/Zuul#Agenda_for_next_meeting agenda
22:02:27 <jeblair> #link http://eavesdrop.openstack.org/meetings/zuul/2017/zuul.2017-03-20-22.02.html previous meeting
22:02:33 <jeblair> mordred to let ppl know when to review changes he's working on that the details are long to share while metal tubing
22:02:59 <jeblair> mordred: i think that was you were expecting to hack on the nodepool-zk shim and ping us when ready
22:03:26 <jeblair> i don't remember being pinged, so i guess that didn't happen
22:03:32 <mordred> yes - this is correct
22:03:56 <mordred> I was goign to work on that, and worked on other things instead - I now expect to hack on the shim _this_ week
22:04:30 <jeblair> okay
22:04:40 <jeblair> i'm not sure it's useful to re-action that
22:04:59 <jeblair> if a shim shows up one day, that's great.
22:05:34 <jeblair> #topic Status updates (nodepool)
22:05:46 <jeblair> that's the status of the shim i guess :)
22:06:04 * rbergeron arrives late
22:06:06 <mordred> the shim loves you all
22:06:27 <jeblair> i put a bunch of things on the agenda which i thought we should make sure to make progress on this week
22:06:29 <jeblair> they all merged
22:06:37 <jeblair> so, neat trick.  i'll have to try that again.
22:06:38 <mordred> \o/
22:06:39 <Shrews> not all
22:06:45 <Shrews> docs changes are still up
22:06:57 <Shrews> but some of those merged
22:07:00 <mordred> Shrews: ssssshhhhhh
22:07:03 <pabelanger> I can review again
22:07:39 <jeblair> #link https://review.openstack.org/447647 legacy openstack setting removed
22:07:40 <Shrews> At SpamapS's's'ss urging, I went through the current nodepool docs and made several updates. Those are the result
22:07:50 <SpamapS> lovely
22:07:59 <jeblair> #link https://review.openstack.org/448814 update nodepool config syntax
22:08:01 <SpamapS> I'll try urging more too
22:08:04 <jeblair> those are the ones that merged
22:08:21 <jeblair> (those are the tips of stacks of related changes -- so those were significant efforts)
22:08:27 <jeblair> just like the docs changes
22:08:32 <pabelanger> jeblair: Shrews: trivial, but did we want make the zookeeper host settings the same between zuul and nodepool? because today are a little different
22:08:35 <jeblair> which we should be able to merge now:
22:08:48 <jeblair> #link https://review.openstack.org/449140 nodepool docs changes
22:09:20 <jeblair> pabelanger: yeah, it would be nice if we can normalize that.
22:09:35 <Shrews> pabelanger: what's different?
22:09:50 <jeblair> there was some technical obstacle to using nodepool's way in zuul which i don't recall
22:10:29 <jeblair> Shrews: nodepool has a data structure in yaml, zuul has a connect string. iirc
22:10:30 <clarkb> if I haven't flipped things around zuul's way was chosen ebcause that way we didn't have to construct the string for kazoo for it
22:10:46 <jeblair> oh!
22:10:49 <clarkb> and we did that because zuul's config is ini
22:10:52 <jeblair> it's because zuul uses ini conf  files
22:10:52 <jeblair> ya
22:11:00 <clarkb> we could do the same for nodepool just store string in yaml
22:11:01 <pabelanger> zuul: zookeeper_hosts=127.0.0.1:2181,example.org:2181
22:11:14 <pabelanger> nodepool is list
22:11:39 <jeblair> also, when we consider that we may need to put ZK credentials in the nodepool secret file, that may affect what we want to do too.
22:11:42 <clarkb> ya its a yaml list of all the things that get concatenated together tiwh commas to pass to kazoo
22:12:43 <Shrews> if we do the secrets thing for nodepool, the structure still seems best IMO
22:13:04 <Shrews> but i'm not strongly tied to i
22:13:06 <Shrews> t
22:13:55 <jeblair> so maybe we should poke at doing zk authentication while keeping this in mind.
22:14:13 <Shrews> speaking of which, the nodepool secrets file is now optional
22:14:22 <pabelanger> I can poke at zk auth things
22:14:22 <Shrews> b/c there is nothing read from it  :)
22:14:41 <Shrews> pabelanger: awesome. thx
22:15:00 <clarkb> do we want to keep it optional and allow unauthenticated zk going forward?
22:15:03 <clarkb> (I don't actually knwo)
22:15:16 <mordred> unauth zk for people running a local nodepool might be nice
22:15:24 <pabelanger> ya
22:15:26 <mordred> like, if you wante to run a nodepool in your house for some reason
22:15:34 <rbergeron> (who wouldn't?)
22:15:45 <SpamapS> my house is a node pool
22:15:48 <mordred> (this is probably more important for zuul all-in-one on a laptop)
22:16:24 <jeblair> i think optional authentication is probably okay.  i think we should at least use ssl in openstack-infra.  possibly auth.
22:17:23 <jeblair> i also am somewhat more inclined than previously to have things like zk connection info in a separate config file from the image/provider config.
22:17:58 <pabelanger> sure
22:18:16 <jeblair> i was hopeful we could drop the 'secrets' file entirely, but i doubt we will be able to do that.
22:18:44 <fungi> it's nice to be able to rely on unix file permissions to protect things which absolutely need it without protecting lots of other things that don
22:18:46 <fungi> 't
22:19:02 <jeblair> anyway, we can bikeshed on that later, anything else we should cover on nodepool?
22:19:16 <jeblair> #action pabelanger look at zk authentication for nodepool
22:19:41 <jeblair> #info consider reconciling nodepool and zuul zookeeper configuration syntaxes
22:19:57 <jeblair> #topic Status updates (Devstack-gate roles refactoring)
22:20:23 <jeblair> did anyone want to push on this or should we skip to next topic?
22:20:36 <clarkb> one thing
22:20:47 <clarkb> related to this we broke third party CIs
22:20:57 <clarkb> I haven't looked to see if the old code would've broken them too (likely)
22:21:10 <clarkb> but just something to keep in mind as review happens for these
22:21:13 <jeblair> i think it's premature to say this broke third party cis
22:21:34 <jeblair> but i agree that is worth keeping in mind
22:21:54 <clarkb> ya I think it likely the old code would've broken on them too
22:22:00 <clarkb> I just haven't had a chance to check
22:22:43 <jeblair> third-party ci operators are welcome to participate in this and review code
22:23:54 <jeblair> (i think we've been pretty clear that we're happy to accomodate the use case in devstack-gate but we will not take operational responsibility for 100 ci systems on our own)
22:24:09 <fungi> clarkb: actually, if this is the change i think you're talking about, it was part of a series to try to fix the pbr integration tests
22:24:27 <clarkb> fungi: yes, but only the first chagne was needed to fix pbr
22:24:41 <clarkb> fungi: the code that broke ci was our network ping test which now failed after we checked return codes
22:24:47 <jeblair> but back to the topic -- last week we agreed this was important, but i'm not sure we signed anyone up to drive it.
22:25:03 <clarkb> the current change to do network overlays is really close
22:25:05 <jeblair> should we do that now, or should we revisit how important we think it is?
22:25:10 <clarkb> If someone wants to pick it up I can help with it
22:25:49 <clarkb> I can also push patches if someone else wants to review the cahnges
22:25:59 <clarkb> but so far I haven't seen a ton of interest in reviewin them (by cores at least)
22:27:18 <jeblair> would anyone like to volunteer for either of those things?
22:27:27 <pabelanger> are we at the step where we want to run devstack (current layout) under zuulv3?
22:27:36 <clarkb> old code would've broken too
22:27:38 <jeblair> i don't think so
22:27:41 <jeblair> pabelanger: ^
22:27:55 <clarkb> it relied on region and cloud being set in the /etc/nodepool/provider file which doesn't necessarily imply mirror exists
22:28:28 <jeblair> this is still trying to get it organized into roles so that we can have a first-rate example of how we think a complex job can be set up
22:28:35 <jeblair> this is something we've all said we want and is important
22:28:43 <jeblair> so i'd love it if someone would volunteer for one of those things
22:29:38 <jeblair> but i'm not going to pretend it's a priority if no one is going to do it, so if no one steps up now, i'm dropping it from the agenda and moving it to backlog on the board.
22:30:22 <jhesketh> I'd like to help but I don't think I understand the parts well enough sorry
22:30:34 <fungi> it's worth noting that the early changes serve as good examples for how to split more of it up
22:31:08 <pabelanger> if we are not going to do more tox jobs or other jobs for openstack-infra, I can start working on devstack
22:31:16 <fungi> and we have loads of testing (granted we react retroactively to third-party breakage)
22:31:27 <pabelanger> I would have like to get another project going and iterate on our stdlib for tox
22:31:34 <pabelanger> but if devstack is important, I can shift
22:31:36 <fungi> so you can mostly know if what you're splitting out will change overall behavior
22:33:09 <pabelanger> if not devstack, I was planning on working on our roles for nodepool / zuul / etc all in one
22:33:21 <jeblair> that's pretty important too
22:33:39 <jeblair> i thought we had more folks interested in writing ansible
22:33:40 <pabelanger> I would prefer to focus on that
22:34:14 <jeblair> we've been on this topic for 14 minutes and no one has said "i volunteer to drive this"
22:35:10 * rbergeron would think more folks might be interested in writing ansible as well -- maybe we are not being explicit enough?
22:35:18 <rbergeron> (not that i'm biased or anything)
22:35:29 <jeblair> rbergeron: maybe they aren't showing up to this meeting or otherwise participating
22:35:47 <jeblair> we've been pretty clear about this topic when it's come up in prior meetings and other areas
22:35:48 <Shrews> an email to the ML for volunteers?
22:35:52 <clarkb> fwiw I'm happy to help, but so far have done so as a reviewer and actually think careful independent review has been good on this task so far
22:36:04 <jeblair> it's strictly ansible work; almost nothing to do with zuul at all
22:36:07 <clarkb> so don't want to transitition to writing the patches unless someone lese is willing to also do the review
22:36:17 <clarkb> but happy to do either task if someone else will do the other :)
22:36:27 <pabelanger> I can help review
22:36:36 <jeblair> Shrews: yeah, let's send an email to the ML
22:36:39 <rbergeron> i just wonder if ... people know that's the possibility at hand and/or understand that perhaps other perceptions about "the bar is xyz high" ... is not so high for this? i would think maybe asking at least hte openstack-ansible folks, kolla folks, bifrost folks, etc. might yield some humans :)
22:37:03 <clarkb> I think part of it for the osa/kolla/et al folks is devstack is such a dirty word
22:37:08 <fungi> we have a big, complex shell script (well, okay it;s a couple of shell scripts) that need ansibling. sounds pretty compelling anyway ;)
22:37:09 <clarkb> even though this really doesn't have much to do with devstack
22:37:13 <rbergeron> i mean there's a pretty decently sized # of ansible users (who would probably also be great as far as sanity checking anyway)
22:37:44 <rbergeron> clarkb: yeah, but i think they all <3 ansible enough that... yeah. i think they all know how this benefits a lot of folks.
22:37:57 <jeblair> rbergeron: do you want to see if you can wrangle anyone via ML, etc?
22:38:13 <rbergeron> jeblair: i can do that -- is ML really the best place for that?
22:38:27 <rocky_g> cc the operators list on the email.  They may be willing...
22:38:36 <jeblair> rbergeron: i have no idea :|
22:38:43 * rbergeron hesitates to be like "hey ansible this stuff we need halp" because .. i mean, all projects always need help
22:39:04 <rbergeron> but I am sure i can figure out the appropriate wording for such things
22:39:19 <jeblair> rbergeron: if you want to wrangle by asking people behind the scenes, that's fine too
22:39:20 <rbergeron> and am always unafraid of looking silly anyway, mostly
22:39:36 <rbergeron> jeblair: i may do that first -- i do have the mental list of humans
22:39:47 <jeblair> i'm just at a place where there's this think we pretend is a priority every week during this meeting and nothing's moving.  so i'm going to stop pretending.
22:39:52 * rbergeron nods
22:39:54 <jeblair> s/think/thing/
22:40:25 <rbergeron> okay, i will do the recruitment :) (I mean, this is one thing i'm decent at :D)
22:40:30 <jeblair> #action rbergeron try to find someone to work on ansiblification of devstack-gate
22:40:44 <jeblair> rbergeron: thank you :)
22:40:56 <jeblair> #topic Status updates (Zuul test enablement)
22:41:16 <SpamapS> We landed some more!
22:41:22 <jeblair> yay!
22:41:24 <SpamapS> #link https://etherpad.openstack.org/p/zuulv3skips
22:41:27 <SpamapS> I updated that
22:41:34 * rbergeron apologizes for not being more meeting-note-taking tonigght but schedule is all weird being on east coast (meeting at 6pm instead of 3pm; dear lord, no idea how shrews does this :D)
22:41:38 <rbergeron> yassss :)
22:41:40 <SpamapS> covered landed tests with strikethroughs
22:41:56 <SpamapS> and linked a few of the in-flight ones
22:42:04 <Shrews> rbergeron: i suffer with alcohol in hand
22:42:05 <jeblair> jesusaur: are you still hacking on 446275 -- the merge check thing?
22:42:11 <SpamapS> we're down to about 18 that aren't being worked
22:42:18 <SpamapS> and most are in the "Straightforward" category.
22:42:38 <SpamapS> feels like progress. :)
22:42:38 <jeblair> i think some of those might inadvertently depend on 446275, though i haven't checked on how many
22:42:58 <SpamapS> that makes sense
22:43:04 <jesusaur> jeblair: yeah, but it's turning out to be quite a game of whack-a-mole
22:43:10 <Shrews> SpamapS: how many people do we have actively working zuul tests now?
22:43:28 <jeblair> jesusaur: okay, let me know if i can help or if you need me to clean up my own mess :)
22:43:40 <jesusaur> if anyone would like to help debug failures on 446275 I would appreciate it, otherwise I will slowly work through them
22:43:40 <SpamapS> Shrews: me, jesusaur, pabelanger, adam_g, eggshell
22:43:44 <SpamapS> maybe more
22:44:16 <jeblair> jesusaur: okay, i'll take a look at what's failing
22:44:23 <pabelanger> I haven't touched a zuul test is some time, apologies for that
22:44:25 <jesusaur> awesome
22:44:34 <SpamapS> I don't know how I'll find tasks to do once they're all re-enabled. ;)
22:44:43 <Shrews> SpamapS: ah, ok. about to envelope myself in the warm arms of zuul, so was wondering if it would be a good place to step into
22:44:46 <SpamapS> pabelanger: you are listed as owning a couple
22:45:03 <pabelanger> SpamapS: yes, I keep saying I will fix them.  I will do this tomorrow
22:45:16 <SpamapS> Shrews: I think it's a good place yeah, though there's a bit of a learning curve on the test harness that might be easier climbed by writing new tests.
22:45:32 <jeblair> we only have 15 mins left, so i'm going to perform some emergency agenda surgery
22:45:47 <jeblair> #Status updates (Zuul secrets)
22:45:53 <jeblair> #topic Status updates (Zuul secrets)
22:45:56 <Shrews> jeblair: we can skip my topic
22:45:59 <Shrews> if it helps
22:46:05 <jeblair> Shrews: i want to make sure we get yours :)
22:46:18 <jeblair> #link http://lists.openstack.org/pipermail/openstack-infra/2017-March/005252.html
22:46:57 <jeblair> i think the results of that thread were basically "do the rsa thing for now, maybe do something else later also"
22:47:34 <jeblair> so i think that's our plan for the moment -- continue with pkcs1-oaep, and defer hybrid or other approaches for later
22:47:39 <jeblair> sound right?
22:48:19 <pabelanger> no objections from me
22:48:34 <jeblair> clarkb, fungi, SpamapS: ^ ?
22:48:45 <SpamapS> concur
22:48:49 <mordred> ++
22:48:49 <fungi> yeah, that's what i got from it
22:48:54 <clarkb> I don't recall the thread reaching a consensus but I think thats probably fine place to start
22:49:03 <clarkb> also maybe use a 8096 bit key?
22:49:13 <jesusaur> jeblair: if there are still concerns about payload size, could we potentially use an 8192 bit rsa key?
22:49:17 <jesusaur> clarkb: yep
22:49:19 <clarkb> er
22:49:22 <clarkb> jesusaur: can math better than me
22:49:22 <fungi> the discussion basically went the same direction i'd already supported in irc, so just sort of sat on the sidelines
22:49:23 <SpamapS> 5120 bit would work fine
22:49:38 <jeblair> clarkb: yeah, it was more like SpamapS advocated that and no one else objected or advocated anything else :)
22:49:40 <SpamapS> if we really really really want to be able to store 4096 bit privkeys
22:50:00 <clarkb> SpamapS: there is more than just ssh keys though so I think being generous is probably a good idea
22:50:07 <jeblair> #agreed proceed with pkcs1-oaep, consider other forms of encryption later
22:50:31 <jeblair> that makes all the secrets twice as large
22:51:05 <jeblair> #info consider changing the rsa keysize
22:51:08 <jeblair> let's talk about that more later
22:51:11 <fungi> yeah, you're basically trading field capacity for extra overhead of all crypted data
22:51:11 <pabelanger> because I don't know, what is the downside to that? more CPU?
22:51:21 <SpamapS> 8192 just evokes the image of a hobbit wielding a claymore in my mind
22:51:21 <clarkb> pabelanger: ya thats probably the biggest one
22:51:38 <jeblair> so the changes to implement this are in this stack:
22:51:43 <jeblair> #link https://review.openstack.org/406382 encryption stack
22:51:46 <fungi> pabelanger: even harder to read yaml files, but they'll already be pretty bad even with 4096-bit
22:51:47 <jesusaur> pabelanger: yeah, main reason it's not used is increased computation time with negligible increase in security
22:51:57 <jeblair> #link https://review.openstack.org/447087 crypto review
22:52:03 <jeblair> #link https://review.openstack.org/447088 crypto review
22:52:23 <jeblair> i think it would be great if we could find someone who understands what some of those magic numbers mean to review those 2 changes
22:52:41 <jeblair> those really nicely put all the crypto decisions front and center
22:52:49 <jeblair> (the rest are about integrating into zuul)
22:53:05 <fungi> jesusaur: but also it's more often used for encrypting common-length (short) bits of data like hashes. if we want to encrypt variable length stuff the key size determines how long that can be
22:53:25 <jeblair> so if anyone here feels they can take a look and say "yes, this is how you should invoke the rsa algorithm", please look at 447087 and 447088
22:53:46 * fungi stars those
22:54:07 <jeblair> and if we feel that we should reach out to anyone outside our community to review these, those are the two i think would be most profitable
22:54:48 <jeblair> SpamapS, fungi, rbergeron: ^ maybe you have ideas about that?
22:54:58 <jeblair> anyone else too.  just trying to put some likely folks on the spot.  :)
22:55:24 <jeblair> #topic ZuulV3 @ Boston Summit (Shrews)
22:55:25 <SpamapS> jeblair: I'm not sure who I'd reach out to
22:55:33 <Shrews> So, we touched on this very briefly last week, but I wanted to get some hard confirmation on whether or not we plan to do any zuulv3 team dev gathering at the summit like we did at the ptg.
22:55:53 <Shrews> Sounded like we were leaning "no"
22:55:53 <clarkb> alex gaynor and dstufft maybe?
22:56:21 <mordred> Shrews: I also got that sense
22:56:50 <jeblair> i also got that sense
22:57:00 <fungi> there will be some flexible hacking space in boston if people want to take advantage of it, but the tone of the summit/forum is to reach outside existing per groups and get into discussions with other segments of the community
22:57:10 <fungi> s/per/peer/
22:57:21 <Shrews> ok, i think that answers the question definitively enough for me
22:57:30 <jeblair> i plan on attending wearing my general openstack-infra hat
22:57:49 <fungi> so there's not much emphasis on structured team-oriented activities
22:58:35 <mordred> I plan on wearing my general openstack-infra hat - and also my "large consumer of OpenStack APIs" hat
22:58:38 <fungi> and yeah, i'll be there with infra ptl, tc, vmt, foundation staff, et cetera hats on. not sure how to wear them at the same time
22:58:41 <Shrews> fwiw, i'm glad that i'm not important enough to have to wear other hats that divert my attention from fun stuff
22:58:51 <mordred> (which are the same hat, to be fair)
22:59:10 <mordred> fungi: oh, yeah - I suppose I'll also have my tc hat on
22:59:17 <fungi> maybe i'll just wear a sombrero
22:59:23 <pabelanger> Same, still waiting on travel funding from manager
22:59:27 <jeblair> mordred: it's the one right next to your "loud consumer of openstakc apis" hat?
22:59:41 <mordred> jeblair: I'M ALWAYS LOUD IT'S A GIVEN
22:59:56 <fungi> pretty sure that's a trucker hat
23:00:08 <SpamapS> I'll be there to speak on Tuesday about Zuul things
23:00:16 <SpamapS> and Forum as a user
23:00:23 <jeblair> thanks everyone!
23:00:26 <fungi> i have a firehose infra talk with mtreinish and then am on a security panel
23:00:36 <jeblair> we'll talk about neglected agenda topics next week
23:00:39 <SpamapS> cheerio
23:00:41 <jeblair> #endmeeting