22:02:27 <jeblair> #startmeeting zuul
22:02:28 <openstack> Meeting started Mon Jul 17 22:02:27 2017 UTC and is due to finish in 60 minutes.  The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot.
22:02:29 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
22:02:31 <openstack> The meeting name has been set to 'zuul'
22:02:43 <jeblair> #link agenda https://wiki.openstack.org/wiki/Meetings/Zuul
22:02:51 <jeblair> #link previous meeting http://eavesdrop.openstack.org/meetings/zuul/2017/zuul.2017-07-10-22.02.html
22:03:00 <jeblair> #topic Announcements
22:04:00 <jeblair> i reckon since the "roll out near/at PTG" plan garnered good feedback at the infra meeting last week, we should probably suggest that interested zuul folks plan on being at the ptg
22:04:21 <fungi> i'll be there!
22:04:34 <jeblair> mordred, fungi: that sound about right to you?
22:04:40 <fungi> totally
22:04:41 <Shrews> Have we decided on a day? I saw suggestions of doing it the weekend before, which might necessitate arriving earlier
22:04:43 <jeblair> there's a page here
22:04:55 <jeblair> #link openstack ptg https://www.openstack.org/ptg/
22:05:04 <pabelanger> working on manager approval now
22:05:17 <clarkb> I think there are other meetings that people will be distracted by on the weekend before
22:05:22 <clarkb> board/tc things
22:05:30 <jeblair> Shrews: i don't think so
22:05:43 <fungi> yeah, i'm flying in saturday and sequestered with the tc/uc/bod all sunday
22:05:47 <jeblair> clarkb: heh, is that an argument for or against doing it on the weekend?  ;)
22:06:03 <fungi> but don't let my absence those days stop a good plan if others are available
22:06:16 <clarkb> fungi: as would be mordred I expect
22:06:29 <jeblair> Shrews: yeah, sounds like we should seek clarity asap.
22:06:43 <jeblair> mordred: thoughts?
22:07:33 <fungi> in all honesty, i'm likely one of the least useful people to a zuul cut-over since i've been spending my time trying to make sure the rest of you are free to focus on zuul development... but i'm happy to help when i'm around/undistracted
22:08:09 <mordred> jeblair: I will have the board/tc thing ... but I doubt I'll be very distracted by it
22:08:32 <pabelanger> Ya, I'm sure I could fly in a day or so early too
22:08:35 <mordred> which is to say - I'll be available whenever I need to be available other than the specific board/tc overlap hours
22:08:36 <pabelanger> if we are going that route
22:09:21 <jeblair> mordred: can you work on a proposal for timing for us to discuss this week?
22:09:32 <fungi> mordred: keep in mind the board/tc overlap hours are like 9am-5pm sunday or something like that. it's not our usual half-day joint thing we do at the summit
22:09:45 <mordred> fwiw - the cutover ITSELF should be take less time than a gerrit upgrade - prepping to make sure we're comfortable with the process and then fielding human interaction and fixing things we missed after is, I think, going to be the bulk of the work
22:09:47 <mordred> jeblair: yes, I can
22:09:53 <mordred> fungi: oh - good point
22:10:10 <jeblair> that way we're not under pressure to make a snap decision right now, but we should make a decision soon since it affects travel plans
22:10:15 <mordred> ++
22:10:35 <jeblair> #action mordred propose ptg-related cutover timeline
22:10:49 <jeblair> mordred: thanks!
22:10:54 <mordred> \o/
22:10:54 <jeblair> #topic  Actions from last meeting
22:11:09 <jeblair> jlk add meat to github docs re app setup and depends-on
22:11:56 <jeblair> i don't think that happened yet?
22:12:03 <jeblair> but jlk is on vacation, so he's off the hook
22:12:10 <jeblair> #action jlk add meat to github docs re app setup and depends-on
22:12:16 <jeblair> #action jeblair make anchors for keywords
22:12:22 <jeblair> that ^ also hasn't happened
22:12:26 <jeblair> mordred add v3 doc link to readme
22:12:33 <mordred> I remember doing that?
22:12:39 <jeblair> yes!  that happened!
22:12:41 <mordred> \o/
22:12:45 * mordred getting stuff done
22:12:47 <jeblair> mordred make infra init script for zuul-web in puppet-zuul
22:13:27 <pabelanger> think I seen that
22:13:32 <fungi> sounds like a useful thing to have
22:13:43 <jeblair> #link https://review.openstack.org/482889
22:14:01 <pabelanger> I did https://review.openstack.org/#/c/471456/ a while back too
22:14:12 <jeblair> cool, let's see if we infra-core folks can get that landed
22:14:17 <jeblair> i'd really like to see the web console streaming :)
22:14:21 <fungi> starred
22:14:24 <mordred> I need to fix cmurphy's feedback on the followup too
22:14:42 <jeblair> #topic Status Updates: Standard job library
22:14:48 <clarkb> re init scripts we should be thinking about moving away from kill -9
22:14:56 <mordred> clarkb: yah - that's on the list
22:15:12 <jeblair> i'll go first here:
22:15:34 <jeblair> i've updated the openstack base job to be slightly less insane
22:15:46 <jeblair> we now pass the log url back to zuul from ansible
22:16:05 <jeblair> which is really cool as it means we don't have to do double-entry bookeeping in both the job definition and the job itself
22:16:31 <mordred> yay!
22:16:39 <jeblair> *eventually* that will let us have a base job definition with log publishing in zuul-jobs, but it doesn't quite make sense yet
22:16:50 <jeblair> i think in order for it to make sense there, we also have to have site-local variables
22:17:06 <jeblair> so that we can parameterize things like "the log host's name", etc, in the base job
22:17:29 <jeblair> (also, we may want to add other methods of log publishing to the base job, and parameterize the use of those)
22:17:52 <jeblair> however, i always expect sites to be able to override the base job with their own implementation
22:17:55 <jeblair> which is what we're doing now
22:18:08 <jeblair> so i've but a simplified base job in zuul-jobs, which just does the git repo setup, but not log copying
22:18:19 <jeblair> and openstack overrides that in project-config with a version which adds log copying
22:18:27 <jeblair> so we're demonstrating that facility at least
22:19:09 <jeblair> i think i'm done touching that for a while, at lest until site-local vars lands
22:19:31 <fungi> very cool!
22:19:46 <pabelanger> indeed
22:19:48 <mordred> good. I'm glad that the number of places to look for content is much smaller with that shift landed too
22:20:01 <jeblair> yeah, openstack-zuul-roles is about to be emptied out
22:20:10 <mordred> and the times when it's somewhere else are exptremely specific
22:20:11 <jeblair> so we're basically just project-config, zuul-jobs, and openstack-zuul-jobs now
22:20:27 <fungi> 3 is better than 4
22:20:46 <jeblair> and all the roles used by our base job in project-config are actually in zuul-jobs
22:20:48 <fungi> there's a zuul-base-jobs too right?
22:20:50 <mordred> yup. and the majority of jobs people will be most likely to look at more frequently will be in openstack-zuul-jobs
22:20:52 <pabelanger> for sure, I was finding myself grepping alot of git repos
22:20:57 <mordred> fungi: that's zuul-jobs
22:20:58 <mordred> pabelanger: ++
22:20:59 <jeblair> fungi: yes, but we're not using it either at the moment
22:21:04 <mordred> ah
22:21:10 <fungi> just making sure
22:21:10 * mordred shuts up - misunderstood question
22:21:14 <jeblair> it *was* going to hold the base job before we invented config shadowing
22:21:23 <fungi> we gave ourselves... lots of options for places to put this stuff
22:21:32 <pabelanger> fungi: ya
22:21:35 <mordred> yup. turns out spreading across tons of repos is not advantageous :)
22:21:44 <jeblair> yeah, we should go through and leave closed signs on some of these repos soon
22:21:48 <mordred> ++
22:21:55 <jeblair> spreading out over the right number is :)
22:21:58 <mordred> yup
22:22:15 <fungi> i have an infra meeting topic about retiring some unused repos, i can add those to the list
22:22:16 <jeblair> mordred: how goes the unittest, etc work?
22:22:16 <mordred> I think we can explain the three places now - (four if you count 'in each project' as a place)
22:22:53 <mordred> jeblair: pabelanger has some patches up I owe him reviews on ... I've dived down the logging hole because debugging some of the jobs was getting annoying
22:22:56 <pabelanger> I have https://review.openstack.org/#/c/483936/ up for review too, which moves some openstack specific things into openstack-zuul-jobs
22:24:33 <jeblair> mordred: *nod*.  i think 2001105 and 2001106 (report executor errors back to users) is the next high priority punchlist item
22:25:18 <jeblair> mordred: the build log has gotten a lot better.  thanks pabelanger too.  :)
22:25:30 <jeblair> i think once we get the json return out of it, it'll be shiny.
22:25:40 <pabelanger> Ya, much eaiser to read now
22:26:11 <jeblair> anyone have anything else to say on this?
22:26:23 <pabelanger> I'll be pusing more tox changes up tomorrow
22:26:28 <mordred> nope.
22:26:28 <pabelanger> so reviews will be helpful
22:27:05 <jeblair> pabelanger, mordred: thanks!
22:27:17 <jeblair> #topic Status Updates: (web) console streaming
22:27:25 <jeblair> i would have segued to this earlier if i were a better meeting chair
22:27:39 <jeblair> anyway, we have an init script in progress
22:28:01 <jeblair> we also need a client page
22:28:24 <jeblair> did tobiash's sample page land, or did that get turned into the serve static html change?
22:29:20 <mordred> it's still in work
22:29:31 <mordred> tobiash and I each have versions up
22:29:31 <Shrews> https://review.openstack.org/483503
22:29:38 <mordred> the next step is to combine elements of each
22:29:57 <mordred> which tobias has said he'll do when he gets back in on wednesday if I don't get to it before then
22:30:24 <jeblair> Shrews: ah thanks
22:30:25 <mordred> https://review.openstack.org/#/c/483947/ is the other one
22:30:42 <mordred> so I tihnk we're close on that one
22:30:44 <jeblair> and https://review.openstack.org/481599 was the other other one i guess
22:30:57 <jeblair> that's the one i was thinking about
22:31:10 <mordred> yes - the final result will be a synthesis of those three patches
22:31:37 <Shrews> jeblair: 599 should probably be abandoned. wrong direction on that one
22:31:50 <jeblair> Shrews: it is already abandoned!
22:31:55 <jeblair> though it is the parent of 947
22:32:06 <mordred> we need to squash
22:32:10 <Shrews> jeblair: such efficiency (/me forgot)
22:32:48 <jeblair> okay, so the way forward is for mordred and/or tobias to combine all that stuff, hopefully around the middle of the week
22:33:00 <mordred> one of the new wrinkles we learned is that tobias has to run his websocket server on a different host
22:33:16 <mordred> or, through a different proxy/domain more specifically
22:33:35 <jeblair> the main question i wanted to raise was whether there was something simple we could do in the interim so we could at least try it, but it sounds like that's not the best path forward.
22:33:37 <mordred> I've got a document started with that captured, as well as other things from this, that I'll try to get out by mid-week too
22:34:14 <mordred> jeblair: we're really close - and once the websocket server is runing, one should be able to host the static file locally for a hacky version
22:34:27 <mordred> it seems websockets do not have the CORS restrictions
22:35:02 <mordred> so once the zuul-web is running, it shoudl be possible, if annoying, to play with it
22:35:10 <jeblair> mordred: can you clarify?  should we aim to get the websocket server running then host a static file locally?  or should we push ahead with you and tobiash's change?
22:35:25 <mordred> jeblair: we should push forward with the current changes
22:35:28 <mordred> they're very close
22:36:09 <mordred> I'm just saying that with the zuul-web puppet already in flight, there will be a thing running that one could physically poke at and stream content from should one choose between the time that starts existing and the time we get the patch finished
22:36:44 <jeblair> mordred: okay, my original question was whether we should attempt to do that or not, which is why i asked whether there was a simple static file we could put in place to do that
22:36:52 <jeblair> but i'm still getting mixed messages on whether that's worth doing
22:36:58 <mordred> gotcha. no. it is not
22:37:03 <jeblair> mordred: how about we aim to get the websocket server running by wednesday, so when the new thing lands, we'll be ready to use it?
22:37:09 <mordred> yes
22:37:21 <jeblair> yay!  a plan is planned!
22:37:52 <jeblair> anything else on console streaming?  web or otherwise?
22:38:01 * fungi loves it when a plan comes together
22:38:35 <jeblair> #topic Status Updates:  Zuul test enablement
22:38:44 <jeblair> i haven't done anything on this
22:39:08 <pabelanger> same
22:39:27 <jeblair> but maybe i should switch to this and get it off the agenda, if other folks have lost interest in it.
22:40:09 <jeblair> #topic Should we continue allowing use of hostname for unique builder ID, or simply force use of UUID? (Shrews)
22:40:22 <jeblair> #link https://review.openstack.org/484414
22:40:47 <Shrews> did a thing in 414 that got that off my "should-be-done-at-some-point" list
22:40:58 <Shrews> based on suggestions from jeblair and SpamapS
22:41:12 <fungi> uuid seems more consistent with the rest of what we have for ids of other components
22:41:14 <jeblair> w00t
22:41:18 <Shrews> pabelanger suggested that maybe we shouldn't support using hostname at all. i can find no argument against this
22:41:36 <Shrews> but wanted concensus on it before i made it so
22:41:44 <jeblair> i vote: force the use of uuid and make it automatic and (nearly) invisible.  :)
22:41:51 <jeblair> (except i like your transition plan)
22:42:04 <mordred> ++
22:42:08 <jeblair> (so keep the hostname comparison in there for a release)
22:42:12 <fungi> as long as there is a means to map them to one another for translating between nova and zuul/nodepool i have no problem with not having hostnames all over the place
22:42:16 * SpamapS ponders
22:42:21 <Shrews> jeblair: with your suggestion to use images-dir, we don't even need to add a new config option, so should be very transparent
22:43:05 <SpamapS> So, if we want to be really careful about data models and such...
22:43:10 <SpamapS> there should be a hostname and an id field
22:43:18 <clarkb> ++
22:43:24 <SpamapS> and the hostname should only mean something if there's no ID.
22:43:27 <clarkb> re SpamapS suggestion becuse humans want a hostname
22:43:30 <SpamapS> (basically versioning the schema)
22:43:31 <Shrews> SpamapS: yep, that was suggested by jeblair and i have a PS that adds that
22:43:36 <SpamapS> and yeah, keep it there so we can see it.
22:43:40 <mordred> ++
22:43:41 <clarkb> SpamapS: I'd keep using it but just for humans
22:44:01 <jeblair> fungi: 414 addresses identifying image builders, so not changing anything about nodes
22:44:07 <mordred> yah - unless someone has a bunch of builder servers all named darkstar
22:44:08 <Shrews> the new hostname field will basically be decorative
22:44:19 <SpamapS> That's up to them. :)
22:44:21 <mordred> hostnames are much better for reporting :)
22:44:38 <Shrews> let me rephrase, the builder will not use the new hostname field  :)
22:44:51 <jeblair> and since we're just adding a field, we get to kick the "zookeeper data model versioning and upgrade" can down the road some more!
22:44:55 <SpamapS> oh, I"m suggesting that the new field not be the same one we used to put the hostname in.
22:45:28 <SpamapS> so that may be slightly different
22:45:35 <SpamapS> I don't like changing the meaning of data fields.
22:45:38 <Shrews> SpamapS: node.builder will still be used for matching. node.hostname will be just the hostname
22:45:39 <fungi> oh, i should have looked closer. so uuids of builders instead of hostnames of builders. still seems reasonable as long as there's some means of reporting or otherwise looking up the hostname
22:46:00 <clarkb> fungi: I think we just keep the hostname field and populate it as we do today
22:46:05 <Shrews> and node.builder will contain the UUID
22:46:07 <SpamapS> Shrews: yeah, I'm suggesting you leave .builder alone, and add .builder_id
22:46:08 <clarkb> fungi: we just don't use it for the comparison to check ownership of an image
22:46:13 <clarkb> SpamapS: ya that
22:46:19 <jeblair> SpamapS: ++
22:46:35 <fungi> wfm
22:46:49 <Shrews> SpamapS: ah, i see. ya, we can do that
22:46:59 <SpamapS> and then if there's no .builder_id, we fall back to old semantics in new code. Old code keeps working the same.
22:47:14 <SpamapS> and as jeblair says... can goes down the road
22:47:23 <SpamapS> (nobody likes that can anyway)
22:47:52 <jeblair> i'm really excited about writing the equivalent of alembic for zookeeper.
22:48:03 <fungi> yeesh
22:48:09 <Shrews> ok, we have a decision on this topic. thx all
22:48:25 <Shrews> jeblair: i'm excited about you writing that too!  :)
22:48:57 <jeblair> #topic Open Discussion
22:49:01 <jeblair> free for all!
22:49:18 <Shrews> o
22:49:22 <Shrews> oops
22:49:33 <jeblair> earlier today in infra, we got a question from an openstack developer about secrets
22:49:41 <clarkb> I'm hoping to be back to the real world next week where I will be setting up a zuul v2.5 and v3 on review-dev to test gerrit 2.13.9. I will run both with tobiash's fix which seems sane at first glance
22:49:58 <Shrews> i'll give you an open topic for discussion: zuulv3 stickers for ptg.... discuss
22:50:01 <jeblair> and the answer was "yes, we have implemented that, and it works like this and should do what you need on day 1 of our rollout"
22:50:26 <jeblair> and i think that marks a nice transition from "oh yeah we're totally going to do that later" which is what our answers have been.
22:50:26 <fungi> it was awesome
22:50:45 <pabelanger> Shrews: I'm working to find a zuul t-shirt
22:50:53 <pabelanger> Shrews: also, yes
22:51:01 <clarkb> Shrews: pabelanger zuul is a dinosaur now btw
22:51:07 <clarkb> which may make for good mascot
22:51:10 <pabelanger> indeed
22:51:12 <fungi> destroyer of shins!
22:51:28 <jeblair> mascot *and* motto all in one
22:51:36 <fungi> also the nickname of our youngest cat, who loves to shred my shins for inexplicable reasons
22:51:47 <SpamapS> speaking of day 1
22:51:52 <SpamapS> There's this PTG coming up
22:51:58 <jeblair> clarkb: cool, thanks -- should we work on merging tobiash's change or you want to poke some more first?
22:52:10 <SpamapS> last PTG was like, a turbo boost 5-day-energy shot for Zuul dev
22:52:34 <jeblair> clarkb: i believe it has both master and zuulv3 versions now
22:52:40 <clarkb> jeblair: it does
22:52:45 <fungi> this one will be turbo boost 5-day-energy shot for zuul configs? ;)
22:52:53 <mordred> that's what I'm imagining
22:53:02 <clarkb> jeblair: I think we can work on merging both cahnges, at first glance it seems a direct way to solve the problem with little ambiguity
22:53:23 <clarkb> zuul and gerrit must agree case sensitvely
22:53:28 <jeblair> clarkb: cool
22:53:40 <fungi> clarkb: oh, so you were able to  confirm you see the issue on 2.13?
22:54:13 <clarkb> fungi: no I haven't yet, but I trust tobiash
22:54:31 <clarkb> actually we could ask wmf
22:54:35 <clarkb> if we want more input
22:55:01 <SpamapS> Well I'm just wondering if there's something we want to make sure we have ready before PTG.. so we can maybe broaden the effort and have lots of infra-eyes on zuulv3
22:55:22 <SpamapS> The current pace of role/job def looks good, so maybe we're already on track.
22:55:44 <mordred> SpamapS: I feel like maybe you may have missed the top of the meeting?
22:56:08 <SpamapS> Oh yes I did
22:56:22 <SpamapS> overlap and such. I'll scroll back and you can pretend I said this then.
22:56:24 <mordred> SpamapS: tl;dr tenantive plan is to go live with full transition of opentsack to zuul v3 at teh beginning of the PTG or immediately before
22:56:39 <mordred> SpamapS: so - I verymuch agree with you :)
22:56:45 <SpamapS> oh and now I need to read infra logs too.. yeah. :)
22:56:48 <mordred> \o/
22:56:59 <jeblair> SpamapS: https://etherpad.openstack.org/p/zuulv3-migration
22:57:09 <fungi> yeah, we talked about it last week in the infra team meeting
22:57:13 <SpamapS> been a bit distracted of late, so hopefully I can contribute as we get closer
22:57:16 <pabelanger> pre-PTG FTW
22:57:23 <jeblair> to be fair, we do need to send that out as an email :)
22:57:31 <mordred> oh yeah. I'll put that on mylist
22:57:46 <fungi> yes! ml or it didn't happen ;)
22:57:53 <mordred> fungi, jeblair: infra list for now, dev list once we've more firm on the date?
22:57:57 <jeblair> probably a good thing to do after we've thought through the exact logistics of a date a bit more (re: earlier discussion in this meeting)
22:57:57 <fungi> i'll do a general ptg planning thread too
22:58:05 <mordred> jeblair: jinx
22:58:08 <fungi> mordred: sounds good
22:59:28 <jeblair> thanks everyone, and enjoy your near-minute of extra time!
22:59:31 <jeblair> #endmeeting