19:03:33 <fungi> #startmeeting infra
19:03:34 <openstack> Meeting started Tue Dec  6 19:03:33 2016 UTC and is due to finish in 60 minutes.  The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:03:35 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:03:37 <openstack> The meeting name has been set to 'infra'
19:03:45 <fungi> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:03:50 <fungi> #topic Announcements
19:03:55 <jlvillal> o/
19:03:59 <fungi> #info REMINDER: If you want to come hack on Infra things at the PTG a couple months from now in Atlanta, don't forget to sign up!
19:04:05 <fungi> #link https://pikeptg.eventbrite.com/
19:04:09 <fungi> #link http://www.openstack.org/ptg
19:04:15 <fungi> i'm told there is plenty of travel assistance available too--if you need it don't be embarassed to ask for it
19:04:22 <fungi> #link http://www.openstack.org/ptg#tab_travel
19:04:34 <fungi> as always, feel free to hit me up with announcements you want included in future meetings
19:04:48 <fungi> #topic Actions from last meeting
19:04:54 <fungi> #link http://eavesdrop.openstack.org/meetings/infra/2016/infra.2016-11-29-19.03.html
19:04:58 <fungi> fungi send summit session summary to infra ml
19:05:02 <fungi> #link http://lists.openstack.org/pipermail/openstack-dev/2016-December/108536.html Ocata Summit Infra Sessions Recap
19:05:06 <fungi> FINALLY!
19:05:11 <zara_the_lemur__> \o/
19:05:34 <fungi> and reply to that with any corrections/additions, of course
19:05:42 <fungi> pabelanger add openstackci::zuul_launcher puppet class
19:05:56 <fungi> do we have a change for that yet?
19:06:02 <pabelanger> sadly no
19:06:08 <pabelanger> on my plate to do
19:06:18 <fungi> okay, cool--thanks!
19:06:22 <pabelanger> I'll make time this week to finish it up
19:06:25 <fungi> #action pabelanger add openstackci::zuul_launcher puppet class
19:06:52 <fungi> and that's it for the action items from last week, i think
19:07:36 <fungi> #topic Specs approval: PROPOSED Zuul v3: use Zookeeper for Nodepool-Zuul protocol (jeblair)
19:07:57 <jeblair> i think this is ready
19:08:00 <fungi> #link https://review.openstack.org/305506 Zuul v3: use Zookeeper for Nodepool-Zuul protocol
19:08:17 <jeblair> it's gone through some revision and should be fairly well vetted now
19:08:35 <fungi> #info Council voting is open on the "Zuul v3: use Zookeeper for Nodepool-Zuul protocol" change until 19:00 UTC on Thursday, December 8.
19:08:41 <jeblair> jhesketh caught some typos, i suggest we vote on this version and i'll fix in a followup
19:08:49 <fungi> sounds good to me
19:08:59 <jhesketh> +1
19:09:45 <jeblair> and incidentally, i think we'll be about ready to start on this shortly after it lands
19:09:51 <fungi> great timing
19:10:11 <jhesketh> Awesome :-)
19:11:51 <jeblair> [eot from me]
19:11:58 <fungi> thanks jeblair
19:12:06 <fungi> anybody else have any concerns before we move on?
19:12:50 <fungi> #topic Priority Efforts: Nodepool: Use Zookeeper for Workers (jeblair)
19:13:07 <fungi> talked about this in the zuul meeting yesterday, right?
19:13:35 <jeblair> yes --
19:13:46 <jeblair> #link zk blockers from zuul meeting http://eavesdrop.openstack.org/meetings/zuul/2016/zuul.2016-12-05-22.02.html
19:14:17 <jeblair> if you look at the action items there, we identified some things that we want to see happen before we think we're ready to start using the new zk nodepool builder in production
19:14:25 <jeblair> it's a pretty short and simple list
19:14:39 <jeblair> so there's a good chance we'll have it done late this week, early next week
19:15:09 <pabelanger> nb02.o.o is already online, thanks to Shrews help
19:15:18 <jeblair> when that's accomplished, i'd like to switch nodepool.o.o to use the zuulv3 branch (which will cause the nodepoold there to use the builds from the zk builder)
19:15:40 <jeblair> and if that goes well, merge feature/zuulv3 into master shortly thereafter
19:15:50 <jeblair> and switch all the nodepool hosts back to running master
19:16:10 <fungi> review focus on project:openstack-infra/nodepool branch:feature/zuulv3 is appreciated to get this in place in a timely fashion?
19:16:14 <jeblair> does that sound good to folks?  any concerns?  additional blockers we should add to the punch list?
19:16:28 <clarkb> are there no changes on master that we need in v3 first? eg should the merge come first?
19:16:34 <jeblair> fungi: yep
19:16:57 <jeblair> clarkb: jhesketh merged master into v3 recently
19:17:03 <jeblair> so i think we're set
19:17:08 <clarkb> ah ok
19:17:39 <jeblair> we could merge first, but this lets us quickly revert without making a mess of the git tree :)
19:17:48 <clarkb> ya
19:17:48 <fungi> looks like maybe there's been one change on master since then?
19:17:55 <jhesketh> If it's shifted I can merge into v3 again
19:18:11 <fungi> Merge "Have an ending line-feed on the generated id_rsa.pub file" is a day newer
19:18:43 * jlvillal spots his commit
19:18:44 <jeblair> jhesketh: if you could do another merge, that would be swell.  should be easy.  :)
19:18:54 <clarkb> fungi: don't think that one affects us
19:19:01 <clarkb> but getting up to syncage would be good
19:19:05 <fungi> yeah, it was https://review.openstack.org/383496
19:19:22 <jhesketh> Yep. Is it worth waiting until we're close to deploy v3 though
19:19:30 <fungi> agreed, that's more just let's make sure it doesn't get lost/conflict
19:19:51 <mordred> o/
19:20:13 <jeblair> jhesketh: i think we're pretty close; i say we do it now, and hopefully master won't move too much in the next week or two.
19:20:25 <fungi> i think that's a fine plan
19:20:31 <jhesketh> Okay
19:21:43 <fungi> #agreed We're on track to switch to ZK-based nodepool builders in roughly a week's time.
19:22:32 <fungi> it's a sweet spot between holidays, so there's hopefully a fair number of people around to work out any kinks but not such high volume of activity that the impact will be dreadful should something go sideways
19:22:53 <mordred> ++
19:22:55 <jeblair> yeah, and there should only be a momentary outage when we restart nodepoold
19:23:18 <dhellmann> that's all coming before the 2nd milestone?
19:23:50 <Shrews> why not create a stable-2.5 branch, just in case?
19:24:15 <fungi> dhellmann: yeah, we should shoot to not have this impact your... thursday activities next week?
19:24:33 <dhellmann> the milestone is on the 15th, so next week should be fine
19:24:36 <fungi> dhellmann: any specific days we need to blacklist for disruptive changes?
19:24:42 <dhellmann> wait, what's today?
19:24:49 <dhellmann> yeah, as long as it's before thursday it should be ok
19:24:55 <fungi> today's the 6th
19:25:04 <dhellmann> yeah, my calendar was on the wrong page :-)
19:25:10 <jeblair> Shrews: well, it would be a stable 0.0 branch if we did.  and i don't believe we intend to support it.  we made a tag for the last version of nodepool people should use if they want to avoid zk.
19:25:23 <dhellmann> so avoiding thursday and friday would be good, but it sounds like that's the plan
19:25:59 <jeblair> dhellmann: yeah, i'm thinking *this* thursday or friday, or if not, early next week.
19:26:05 <dhellmann> sounds good
19:26:06 <fungi> #info Coordinate any potential disruptions late next week with the Release team.
19:26:21 <clarkb> this week si fine right?
19:26:33 <fungi> yeah, it's a dead week on https://releases.openstack.org/ocata/schedule.html
19:26:34 <clarkb> pretty sure when we decided to do the xenial stuff this week it was to avoid release things
19:26:37 <clarkb> kk
19:26:42 <jeblair> once we merge v3 into master, i think we should issue another release and suggest early adopters may want to try using it along with us.
19:26:50 <mordred> jeblair: ++
19:26:55 <dhellmann> yeah, mainly just trying to avoid changes *on* deadline days, so it sounds like your plans are fine
19:27:21 <fungi> thanks for jumping in dhellmann!
19:27:28 * dhellmann goes back to lurking
19:27:41 <fungi> your constant lurkiness is always appreciated
19:27:52 <jeblair> everyone should lurk so well
19:28:08 <fungi> i definitely want to make sure we keep release activity disruptions to a minimum, in particular
19:28:33 <fungi> there are even fewer of them than there are of us
19:29:04 <fungi> okay, so additional release. minor bump, or major prerelease?
19:29:32 <clarkb> it would have to be major prerelase I think if we wanted to semver right?
19:29:58 <clarkb> these changes aren't entirely backward compatible
19:30:07 <fungi> well, we're at a 0.x release still
19:30:09 <clarkb> (though for the most part your old config will work with new stuff with small changes
19:30:12 <mordred> clarkb: yes - but we've never released a supported version of nodepool
19:30:14 <clarkb> ah
19:30:42 <jeblair> well, depending on how you read this with semver, it might actually could be a micro, because it is backwards compatibleish
19:30:47 <fungi> so 0.4.0 would be reasonable, but i could also get behind 1.0.0.0b1
19:31:03 <fungi> i guess depends on how ish the ish part is
19:31:17 <clarkb> the ish part is you need new config for zk and need to have a zk server
19:31:31 <clarkb> everything else is compat I think
19:31:34 <fungi> i'm assuming nodepool 1.0.0 is targeted roughly coincident with zuul 3.0.0
19:31:38 <mordred> yah.
19:32:45 <jeblair> at any rate, i lean toward 0.4.0 with a slightly generous interpretation of semver
19:33:06 <fungi> i think we agreed to pre-announce any nodepool backward incompatibilities on the infra ml in advance too? or was nibalizer satisfied if we just told people to pin to something a while back?
19:33:27 <fungi> i mean, i know we pretty thoroughly announced the zk addition was in the pipeline
19:33:28 <clarkb> ya I think we told people to pin. maybe double check we didn't say pin to <1.0
19:33:38 <clarkb> because that could influence this version picking
19:33:55 <jeblair> i'm still happy to pre-announce both the merge and the release.
19:34:17 <jeblair> good opportunity to let people know what's happening, and in the case of the release, let people know how to start using it
19:35:33 <jeblair> (but both of those will happen after the v3 branch is in prod, so we have some time)
19:35:55 <fungi> awesome
19:36:43 <fungi> okay, anything else to discuss right now on the road toward nodepool 1.x?
19:37:42 <jeblair> nope
19:37:52 <fungi> thanks jeblair!
19:37:54 <fungi> #topic Priority Efforts: Newton testing on Xenial (clarkb)
19:38:13 <clarkb> So this is mostly a heads up that we are moving ahead with the day we picked in Barcelona (today)
19:38:22 <fungi> #link https://etherpad.openstack.org/p/xenial-work-remaining Work remaining for Newton testing on Xenial
19:38:26 <AJaeger> please all help with reviewing: https://review.openstack.org/#/q/topic:st-nicholas-xenial+status:open
19:38:35 <clarkb> my rough process has been to go through jenkins/jobs/*.yaml alphabetically and crank out changes
19:38:41 <clarkb> AJaeger is going through the list in reverse sort order
19:38:44 <AJaeger> we've frozen project-config and will only merge these xenial changes for now.
19:38:54 <fungi> #info Today is the Xenial cut-over flag day; conversion changes are in flight.
19:39:01 <AJaeger> anybody that wants to help? Reviewing and doing changes, both is needed!
19:39:17 <clarkb> anyone that cna help review is much appreciated. If you want to help write changes too we can carve you out a chunk of yaml files in the middle of the alphabet
19:39:53 <clarkb> I have a feeling this work will carry over into tomorrow. Just based on how much progress we have been making
19:39:57 * AJaeger is not sure whether we finish today - I'm at s and will finish that but not sure I can take on much more.
19:39:59 <clarkb> its not really slow just lots to do
19:40:03 <fungi> #link https://review.openstack.org/#/q/is:open+topic:st-nicholas-xenial please review remaining job changes for the switch to xenial
19:40:40 <fungi> i don't think it's crucial that we _finish_ today. we told people we'd start force-converting them today
19:41:09 <fungi> and progress has been great so far (thanks especially to AJaeger and clarkb who have been writing most of them)
19:41:11 <AJaeger> clarkb: I won't be able to help much tomorrow
19:41:25 <clarkb> AJaeger: ok I am sure there will be others around. Thanks for all the work you have done its been a big help
19:41:39 <clarkb> fungi: I do think we want to get done this week to avoid release team conflicts but yes
19:41:41 <AJaeger> clarkb: thanks for driving this!
19:41:51 <fungi> clarkb: yep, completely agree
19:42:11 <fungi> also once it's done, i think we can put the last nails in the coffin for this priority effort/spec?
19:42:29 <clarkb> yup
19:42:41 <fungi> will be nice to scratch one more off the list
19:43:08 <clarkb> though I do think it is showing us we have a lot of cleanup that we should push on. Basically there is a lot of cruft in our jobs. Experimental and non voting jobs that in theory don't have anyone caring for them since they haven't been updated
19:43:24 <clarkb> we should probably think about clearing that out before we do any zuulv3 transition
19:43:27 <fungi> yeah, looks like the last work item will be covered:
19:43:27 <clarkb> reduces the problem set
19:43:46 <fungi> #link http://specs.openstack.org/openstack-infra/infra-specs/specs/newton-on-xenial.html#work-items Newton testing on Xenial: Implementation Work Items
19:44:26 <AJaeger> yes, we need some more cleanups - I'm doing minor local ones while changing but we need some more later
19:44:27 <clarkb> (I don't think this cleanup should be tracked under this priority effort, its just something I am noticing as I do the work and it could make zuulv3 easier to get it done)
19:44:45 <fungi> SpamapS: ^ that might be a good task to get on one of the planning boards?
19:45:18 <fungi> basically before we convert our corpus of job configs, try to filter out any unused cruft
19:45:26 <pabelanger> AJaeger: I'll review again shortly
19:45:40 <jeblair> fungi: we can, but i don't want to set it up as a blocker
19:45:56 <jeblair> fungi: we're going to end up with tools to convert our jobs mostly automatically
19:45:59 <clarkb> ya not sure its a blocker
19:46:20 <jeblair> so they should not be a large cost to the conversion -- cleanup can happen both before and afterwords
19:46:21 <fungi> okay, so more of a "it will be a nice low-hanging fruit task"
19:47:05 <clarkb> more of a "we should really do this beacuse its getting gross in there a bit"
19:47:21 <fungi> anyway, in summary: review review review
19:48:03 <fungi> lots of very similar but subtly different changes in flight in project-config, and we've mostly frozen job config changes for any other efforts until we worth these through to inimize merge conflicting
19:48:08 <AJaeger> we also need some mroe reviewers - yolanda and myself review a lot on project-config, we could use a third reviewer ;) Especially for our own changes I have to ping quite often ;(
19:48:37 <AJaeger> we emptied the open queue before the freeze to an all-time low of 89 open reviews ;)
19:48:56 <fungi> totally. infra needs more reviewers in general, but the project-config reviews are extremely high-volume. i also hope that zuul
19:49:04 <fungi> v3 minimizes that some
19:49:25 <jeblair> yes, i'm focusing less on project-config reviews so i can make zuulv3 so we have fewer project-config reviews
19:49:40 <jeblair> i'm sorry that makes things worse right now, but i hope it will make things better in the future
19:49:52 <ianw> i can make an effort to get back to more project-config reviews
19:49:54 <AJaeger> jeblair: appreciated
19:50:14 <AJaeger> thanks, ianw
19:50:36 <pabelanger> I can chip in more too
19:51:20 <fungi> doesn't look like we have any general topics on the agenda today, so...
19:51:23 <fungi> #topic Open discussion
19:51:57 <jlvillal> I guilty ask for a review request: devstack-gate https://review.openstack.org/#/c/396717/
19:52:03 <ianw> i don't really know what happened with the pypi mirror volume the other day.  if anyone with more detailed afs knowledge can glean anything from the logs, that would be cool
19:52:05 <jlvillal> And a meetbot one: https://review.openstack.org/404407
19:52:16 <clarkb> oh good I have alredy reviewed that one
19:52:22 <jlvillal> heh
19:52:38 <fungi> and yes, thanks to everyone who finds time to do some reviewing in general. we've lost some key contributors partially or completely in recent months, but our change volume hasn't dropped appreciably
19:53:48 <fungi> jlvillal: thanks, that also reminds me...
19:53:53 <fungi> #link http://lists.openstack.org/pipermail/openstack-infra/2016-December/004951.html the fedora community wants to collaborate with us on our meetbot fork, and possibly on the errbot-based rewrite
19:54:32 <mordred> \o/
19:54:42 <fungi> i'm always happy when we get opportunities like this to collaborate on community infrastructure projects with others outside openstack
19:55:13 <jeblair> ++
19:55:45 <flaper87> fungi: collaboration on meetbot sounds awesome indeed
19:55:49 <fungi> bkero: harlowja: ^ i think both of you had maybe looked into some of our ircbot stuff, so that thread might interest you?
19:56:54 <fungi> flaper87: yeah, it's dead upstream, and our fork seems to be the only active one around. so i guess we're upstream now :/
19:57:02 <clarkb> fungi: funny how that happens
19:57:14 <clarkb> "you are crazy enough to be using that software I wrote? tag you're it!"
19:57:19 <jlvillal> Yeah. Last commit upstream seems to be 2010 or something like that.
19:58:07 <SpamapS> fungi: regarding cleaning up jobs, I don't think we've defined the set of jobs that have to run reliably to get to operational zuulv3. The capabilities have been the focus. But it's a good point that as we get closer, we'll want to cut the fat.
19:58:27 <bkero> fungi: Sounds good to me
19:58:32 <bkero> Thanks for the mention
19:59:10 <fungi> aaaaand... we're just about out of time. thanks everyone!
20:00:00 <fungi> see you next week, same bat time, same bat channel
20:00:02 <fungi> #endmeeting