22:02:37 <jeblair> #startmeeting zuul
22:02:38 <openstack> Meeting started Mon Jul 24 22:02:37 2017 UTC and is due to finish in 60 minutes.  The chair is jeblair. Information about MeetBot at http://wiki.debian.org/MeetBot.
22:02:39 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
22:02:42 <openstack> The meeting name has been set to 'zuul'
22:02:51 <jeblair> #topic Actions from last meeting
22:03:06 <jeblair> mordred propose ptg-related cutover timeline
22:03:16 <jeblair> i haven't seen any activity regarding this
22:03:47 <jeblair> and i haven't seen mordred around today
22:03:56 <fungi> nor have i. i wonder if he left it in an airport lounge
22:04:01 <jeblair> ya
22:04:24 <jeblair> but this came up because last week we said "folks should go to the ptg cause we want to do the cutover there"
22:04:30 <jeblair> and then other folks said "when?"
22:04:46 <jeblair> and we said "... erm... around the ptg?"
22:05:27 <jeblair> so anyway, storytime over, i think we need to figure this out asap
22:05:43 <jeblair> fungi: you mentioned you and mordred are in an all-day meeting on sunday, yeah?
22:05:47 <Shrews> yep. gotta book rsn
22:06:03 <fungi> jeblair: ~8am-5pm i think, yes
22:06:30 <fungi> and i probably won't be checked into my hotel until early saturday evening
22:06:49 <fungi> though i could try to help over in-flight wifi or something
22:07:14 <jeblair> so let's assume that doing the cutover during the meeting is out.  :)
22:08:15 <SpamapS> o/
22:08:17 <jeblair> and yeah, saturday is similarly problematic for both of you, and potentially others.
22:09:01 <jeblair> so it's looking a lot like either we just stay up all sunday night with pizza and beer, or monday morning.
22:09:05 <SpamapS> as yet I'm not scheduled to attend the PTG
22:09:08 <fungi> well, i am limited on availability of flights, and have tickets to a concert right before which christine won't let me skip since it's for her birthday
22:09:18 <SpamapS> but I can request and go if folks feel I should be there
22:09:27 <fungi> but yeah, i can meet people saturday night and sunday night
22:10:07 <clarkb> saturday is hard beacuse it makes a 5 day trip a week long trip
22:10:13 <clarkb> but not impossible
22:10:29 <fungi> SpamapS: i wouldn't want to pressure anyone into going, but i do love seeing you and you're always a huge help
22:10:43 <Shrews> how much time are we anticipating this would take up? according to what mordred said last, we slowly do some migrations up to the point of the ptg, then just switch the rest.
22:11:33 <jeblair> Shrews: i would say physically making the switch: not much time.  dealing with the fallout?  a lot.
22:11:52 <jeblair> which is why i'm leaning toward monday morning
22:12:02 <SpamapS> fungi: ty :)
22:12:05 <Shrews> SpamapS: i need a karoake/bar/spa guide... if you aren't there, i will be lost
22:12:22 <jeblair> if we switch saturday night, then we're out fungi and mordred on sunday to help fix things.
22:12:45 <SpamapS> Shrews: the reasons for going are stacking up! :)
22:13:01 <fungi> i suppose we can always do a trial cut over saturday night with a plan to rollback regardless of the result, and then repeat sunday night?
22:13:24 <Shrews> jeblair: so, sounds like switch sunday night, deal with fallout monday before everybody gets all ramped up on their group things sounds reasonable?
22:13:40 <Shrews> or fungi's thing
22:13:43 <fungi> i also hate people scheduling around my availability, but mordred does seem a little more critical to the effort
22:13:46 <SpamapS> Are we avoiding doing it any earlier because we don't think project team members will be available to help fix their jobs due to travel?
22:14:26 <SpamapS> To me, our usual distributed working environment will be good for all but the hardest fails.
22:14:36 <jeblair> SpamapS: mordred and i (at least) are at ansible SF the week before
22:14:50 <SpamapS> Ah ok there are conflicts .. that makes sense.
22:15:24 <fungi> and my travelling starts wednesday before that weekend due to aforementioned concertgoing
22:15:40 <jeblair> fungi: yeah, i like the idea of trial rollouts in the evenings.
22:15:56 <SpamapS> Yeah sounds like Saturday might be the way to go. Could cut peoples week short maybe, so they're not there for 7 days.
22:16:07 <jeblair> i'm not sure i actually want to make the switch on sunday night and leave it running since i'm pretty sure that will actually mean no sleep :)
22:16:20 <SpamapS> like, have some come for saturday, some sunday, some monday... so you get a rolling team and fresh brains on Monday
22:16:50 <jeblair> but i could do a trial saturday night, then accumulate things to work on on the flight over on sunday, repeat sunday night, then go for real on monday.
22:17:15 <pabelanger> I like the idea of running it for a bit on saturday / sunday. Collect some data (failures) then work on playbooks / roles
22:17:35 <pabelanger> jeblair: +1
22:18:25 <SpamapS> Yeah, the more we can do to handle the obvious problems, the less critical it will be to have a huge mass of people working on it after the real cutover.
22:18:32 <clarkb> maybe run it as shadow zuul starting saturday, fix issues as they come up then make shadow zuul real zuul on monday
22:18:52 <SpamapS> Also should we maybe put a few open session times on the cross-project schedule for team reps to come debug with us?
22:18:53 <clarkb> that way you don't have to go back and forth each day
22:19:00 <jeblair> clarkb: i think mordreds plan is to have it shadowing some significant portion before the ptg even
22:19:17 <jeblair> SpamapS: yes!  i think that's on our brainstorming etherpad already even
22:19:20 <pabelanger> Ya, we are pretty close to shadowing tox jobs right now
22:19:52 <SpamapS> ah good
22:19:53 <fungi> i'm also in the middle of procrastinating a brief announcement to the infra ml pointing people at that etherpad
22:21:15 <jeblair> how's this for something to agree on:  we plan to briefly switch to zuulv3 on saturday and sunday evenings before the ptg, and aim for a permanent cutover (if there are no blockers) suday evening or monday morning.
22:21:38 <fungi> wfm
22:21:48 <jeblair> so that accomodates folks traveling and/or meeting on saturday or sunday
22:22:04 <fungi> monday morning plan to officially go live seems better the more i think about it
22:22:09 <jamielennox> you have venue acess for that time?
22:22:14 * jamielennox catching up
22:22:20 <clarkb> jamielennox: bar access you mean?
22:22:23 <fungi> so that nobody feels compelled to stay up all sunday night
22:22:24 <jamielennox> or just doing that virtually
22:22:33 <jeblair> jamielennox: good question.  yeah, bar+virtual i think
22:22:41 <fungi> jamielennox: the conference room with the really long one-sided table
22:22:56 <SpamapS> Perhaps the foundation can make a room available on Sunday?
22:22:56 <pabelanger> wfm
22:22:58 <jamielennox> clarkb: bars are venues, just wondering if you were meaning physical presence for people or not
22:22:58 <jeblair> personally, if we go with this, i'll be at home sat night, then bar/hotel/whatever sun night.
22:23:39 <jeblair> SpamapS: my guess is that a request for evening venue use would not be straightforward
22:23:44 <jamielennox> bars are possibly my favourite venues
22:23:47 <SpamapS> jeblair: oh good point
22:23:47 <clarkb> jeblair: that is likely to be my situation as well
22:24:10 <fungi> SpamapS: i can ask the organizers for a room sunday, but i don't think they actually have the venue booked (the bod/uc/tc meeting is somewhere else if memory serves)
22:24:16 <clarkb> I'd virtual saturday, fly in sunday, bar sunday evening
22:24:29 <jeblair> #agreed we plan to briefly switch to zuulv3 on saturday and sunday evenings before the ptg, and aim for a permanent cutover (if there are no blockers) monday morning
22:24:42 <jeblair> fungi: i dropped the "sunday evening cutover" option :)
22:25:00 <fungi> i'm cool with that. rest up, cut over monday morning
22:25:19 <jeblair> #agreed we might find a bar sunday evening to help facilitate a smooth transition
22:25:45 <fungi> yeah, hacking in hotel bar/lounge/lobby seems fine to me
22:26:36 <jeblair> everyone have what they need to book now?
22:26:51 <fungi> jawol!
22:26:58 <jeblair> #link https://www.openstack.org/ptg/
22:27:19 <Shrews> sounds good to me
22:27:39 <jeblair> fungi: officially we have a room for how many days?
22:27:45 <fungi> five
22:27:49 <pabelanger> yup, will talk with manager in the morning
22:27:55 <jeblair> oh nice :)
22:28:06 <fungi> though i'm reiterating in my e-mail to the list which should go out shortly after this meeting and my dinner wrap up
22:28:12 <jeblair> so we don't even need to find a bar mid-week
22:28:38 <Shrews> jeblair: that is an incorrect statement
22:28:40 <fungi> monday/tuesday we should try to make some space in our room for people from other teams to come ask us questions or for help with their problems
22:28:47 <Shrews> b/c i will *so* need to find a bar
22:28:49 <Shrews> :)
22:28:57 <clarkb> fungi: and possibly even do embedded debugging/outreach
22:29:02 <clarkb> rather than forcing people to find us
22:29:06 <fungi> yes, that too
22:29:20 <jeblair> Shrews, fungi, clarkb: ++
22:29:43 <fungi> but specifically one of the things monday/tuesday are slotted for this time around is horizontal teams helping other teams
22:30:03 <fungi> so we should plan to have guests, i think
22:30:10 <jeblair> fungi: including helping them with problems that we're causing for them!
22:30:18 <pabelanger> :)
22:30:21 <fungi> *yes*
22:30:52 <jeblair> i like this format.  i think it will work well.
22:31:09 <jeblair> all right!  that's one action item taken care of.
22:31:35 <jeblair> i'm going to drop the other two because those are just turning into jlk/jeblair todolist items.  i don't think we've forgotten.
22:31:44 <jeblair> jlk add meat to github docs re app setup and depends-on
22:31:45 <jeblair> jeblair make anchors for keywords
22:31:47 <jeblair> for reference
22:32:03 <clarkb> we should probably send mail to the dev list with that rough plan as well
22:32:18 <clarkb> just so that it isn't a complete surprise when people show up on monday
22:32:26 <jeblair> clarkb: yeah.  i'll try to hound mordred about that this week.
22:32:59 <jeblair> and if that fails, i will write an email saying "mordred is planning on...."  :)
22:33:28 <jeblair> #action mordred send email to dev list about ptg cutover
22:33:29 <fungi> right, i assumed that fair warning was a prominent aspect of mordred's thing
22:33:49 <fungi> and would incorporate our tentative schedule
22:33:56 <jeblair> yeah, and we still have time to produce at least a 1 month warning.
22:34:38 <jeblair> #topic Status updates: standard job library
22:34:53 <jeblair> pabelanger: how's the tox job coming?
22:34:58 <pabelanger> slowly :)
22:35:18 <pabelanger> I think we're almost done getting shell scripts out of tox playbooks / roles
22:35:31 <pabelanger> only a few things left. Hope to finished that up this week
22:36:12 <jeblair> pabelanger: w00t
22:36:22 <pabelanger> https://review.openstack.org/#/q/topic:tox_environment_defaults is ready for review / bikeshed
22:36:28 <pabelanger> if people are interested
22:36:44 <pabelanger> that brings online openstack-py35 job with upper-constraints
22:36:51 <jeblair> i landed a bunch of changes to how post/tag/periodic jobs are run
22:37:14 <jeblair> which should make it pretty easy to use the same check jobs in those pipelines
22:37:19 <pabelanger> also started testing multinode jobs, found a few issues and patches uploaded. Mostly around logging
22:37:50 <pabelanger> https://review.openstack.org/#/c/486228/
22:38:37 <jeblair> i just pushed up a revision of jamielennox's site variables change: https://review.openstack.org/447734
22:38:50 <jeblair> which i broke.  i'll fix it.
22:39:08 <jeblair> hopefully jamielennox doesn't hate it  :)
22:39:14 <jamielennox> jeblair: i was going to chat with you on that in #zuul afterwards
22:39:26 <jeblair> (uhoh)
22:39:38 <jamielennox> oh, i only looked at the basics and it's fine
22:39:54 <jamielennox> but mostly the reason we wanted it was to put different variables in for like a log server
22:40:02 <jeblair> at any rate, the relevance here is that if we get something like that in, then we can continue to put more configurable things in zuul-jobs
22:40:08 <jeblair> like that ^ :)
22:40:12 <jamielennox> i haven't looked at this, but could the same thing be accomplished with job shadowing for different environments?
22:40:13 <fungi> should we be planning to slush-freeze project-config:jenkins/*/* soon to avoid regressions?
22:40:28 <jamielennox> but i still think it's a useful idea
22:40:32 <jeblair> jamielennox: yes it can.  i think we may want all of these tools.
22:41:18 <jeblair> jamielennox: i think the hope is to try to make the jobs in zuul-jobs (including "base") as generally useful as possible, with variables to help with that up to a point.  then when you cross that point and you really need different local behavior, shadow jobs.
22:41:58 <fungi> like, for anything which isn't slated as part of the scripted config transformation, i'm worried people working on playbook versions may miss subsequent alterations on the "legacy" production copies between now and cut-over
22:42:34 <jeblair> fungi: i think we're going to need to keep jenkins/jobs moving until close to the cutover
22:42:43 <jeblair> fungi: but maybe we can freeze jenkins/scripts now?
22:43:10 <fungi> well, _most_ changes under jenkins/jobs/ are to job payloads, not nearly so much to macros
22:43:42 <jeblair> fungi: right, so i'm hoping we can keep caught up with changes to them with subsequent runs of the migration script
22:43:50 <jeblair> fungi: freezing macros earlier might be a good idea
22:44:03 <jeblair> fungi: that should help avoid getting an entirely new class of job we have to write new handlers for
22:44:08 <pabelanger> I would say we already have some non backward compatible changes in our zuulv3 jobs now.  We should get them documented some place too
22:44:45 <jeblair> pabelanger: how so?  aren't we running them on repos that are running the current versions as well?
22:44:55 <fungi> yeah, my concern was mostly macros.yaml, maybe python-jobs.yaml and some others along those lines
22:45:56 <jeblair> pabelanger: i mean we're making lots of *changes*, but i don't think we're breaking compatability per-se.  we are still using the CTI.
22:46:01 <pabelanger> jeblair: I was thinking about subunit 50MB file size and no longer pip freeze output in logging
22:46:23 <clarkb> pabelanger: why are we no longer keeping pip freeze output in the logs?
22:46:36 <pabelanger> clarkb: we are, but tox does it by default
22:46:42 <pabelanger> rather then us calling pip freeze
22:47:10 <jeblair> pabelanger: yeah, those are worth mentioning so people are aware.  i would not characterize them as breaking backwards compat.
22:47:31 <jeblair> i expect us to write a "zuulv3 for openstack devs" document before the ptg.  that would be a good place to put those.
22:47:36 <pabelanger> another biggest change is removal of fallback support in bindep. That hasn;t merged but patches are up
22:48:13 <clarkb> pabelanger: that will break a lot of stuff if it went in today, iirc most projecst have not switched to using bindep
22:48:39 <jeblair> that should really be a post-v3-cutover task, no?
22:48:51 <pabelanger> clarkb: agree, but we might not want that in zuul-jobs. So we might need to carry something for openstack-zuul-jobs?
22:49:04 <jeblair> pabelanger: no, we granted ourselves a waiver on that.  :)
22:49:23 <jeblair> pabelanger: it's a thing we decided we can put in zuul-jobs temporarily, until we remove the fallback.
22:49:46 <jeblair> pabelanger: (maybe we should enable it with a site-local variable so it doesn't bother other people)
22:50:37 <jeblair> and hopefully soon after zuulv3, we can remove it
22:50:37 <pabelanger> possible, if we still want fallback.txt, I can rework 482650 to try and support it, while keeping it out of zuul-jobs too
22:50:56 <SpamapS> the less changes we have to juggle in transition, the better
22:51:10 <jeblair> ++
22:51:34 <SpamapS> would rather have a few rolls of duct tape worth of fixes on the final zuul-jobs than unusable perfection
22:51:34 <jeblair> this one has a simple path forward regardless of the transition, so no need to tie the two together
22:52:01 <SpamapS> yeah
22:52:17 <SpamapS> ANd anybody who is bothered.. can help fix it. :)
22:52:36 <jeblair> SpamapS: by editing a text file!
22:52:47 <SpamapS> hopefully they can exit that editor after
22:53:02 <jeblair> anything else on jobs?
22:53:39 <jeblair> #topic Status updates: (web) console streaming
22:54:01 <jeblair> thanks to Shrews, mordred, tobiash, we haz web console streaming
22:54:12 <jeblair> if you haven't seen it, go look, it's awesome
22:54:21 <fungi> i never got to witness the poc in action. is the apache config in place on zuulv3.o.o now?
22:54:25 <jeblair> yep
22:54:38 * fungi goes to the bigger computer
22:54:51 * jeblair rechecks a change
22:55:09 <jeblair> http://zuulv3.openstack.org/static/stream.html?uuid=0f3f5a18029b426f96a1596028576282&logfile=console.log
22:55:13 <fungi> nice!
22:55:23 <clarkb> I like its hosted under /static :)
22:55:24 <jeblair> that's linked to from the status page
22:55:32 <jeblair> http://zuulv3.openstack.org/
22:55:33 <fungi> yeah, that's the link i followed
22:55:42 <SpamapS> I expect the Jenkins folks to sue us any day now ;)
22:55:50 <fungi> and i just saw it update spontaneously, so really streaming!
22:56:05 <jeblair> clarkb: yep, apache ftw
22:56:17 <jeblair> SpamapS: ours is white on black.  there's is black on white.
22:56:20 <jeblair> completely different.
22:56:37 <jeblair> also...
22:56:43 <jeblair> ours has *hostnames* in the log
22:56:46 * fungi goes on to prove that black is white and gets himself run over at the next zebra crossing
22:56:50 <jeblair> because it's a streaming multi-host console log
22:57:02 <clarkb> feature idea, make redirects to the logs servers once the ephemeral stream is gone, also support timestamp based linking
22:57:09 <jeblair> which is.. i mean, have you seen one of those before?  :)
22:57:26 <jeblair> clarkb: yeah, that was an inadvertant bug.  i think there's a fix in flight?
22:57:31 <jeblair> (the link switch)
22:57:36 <Shrews> jeblair: in place now
22:57:38 <clarkb> ah cool
22:57:40 <fungi> jeblair: technically we didn't implement streaming web console logs, we invented job logging over finger protocol and a streaming web proxy for finger ;)
22:58:02 <SpamapS> obviously
22:58:03 <SpamapS> finger
22:58:06 <jeblair> fungi: ayup
22:58:11 <fungi> we just happened to then give it the finger
22:58:38 <SpamapS> so glad we could finally give telnet users the finger.
22:58:39 <jeblair> we still need a finger multiplexer, so we can "finger uuid@zuul.openstack.org"
22:58:52 <jeblair> but even i will admit, that's not a pre-ptg blocker.  :)
22:59:18 <fungi> does travis have streaming logs of jobs as they run?
22:59:23 <SpamapS> at least that's already built into the protocol :)
22:59:29 <SpamapS> fungi: yes
22:59:31 <fungi> i will admit to having never really looked at travis at all
22:59:48 <SpamapS> with terminal colors
22:59:59 <fungi> in that case i like ours better
23:00:02 <fungi> ;)
23:00:18 <jamielennox> yea, for future the part i like about travis's streaming is that you can collapse different bits
23:00:35 <fungi> 2017-07-24 23:00:01.677235 | ubuntu-xenial | 1mwriting output... ;49;00m[  3%] madmin/client;49;00m
23:00:40 <jeblair> jamielennox: we are annotating the different phases, so we can totally do that
23:00:45 <fungi> you're saying that would have been colorful in travis i guess
23:00:48 <jamielennox> like it will collapse the machine setup (that i probably don't care about) when it's done
23:00:53 <SpamapS> they have the benefit of strong coupling between test runner and frontend...
23:01:02 <SpamapS> but we kind of do too since we get to have ansible action plugins
23:01:13 <jeblair> SpamapS: yeah, there's a lot of stuff we can do
23:01:15 <jamielennox> jeblair: yep, this is impressive already, not trying to get too far ahead
23:01:18 <Shrews> time's up folks
23:01:21 <jeblair> see also mordred's json output
23:01:24 <jeblair> Shrews: so it is
23:01:28 <clarkb> jamielennox: as a noob to $project I hate that. makes it really hard to find anything with grepping
23:01:35 <jeblair> thanks everyone!
23:01:37 <jeblair> #endmeeting