19:03:33 #startmeeting infra 19:03:34 Meeting started Tue Dec 6 19:03:33 2016 UTC and is due to finish in 60 minutes. The chair is fungi. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:03:35 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:03:37 The meeting name has been set to 'infra' 19:03:45 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:03:50 #topic Announcements 19:03:55 o/ 19:03:59 #info REMINDER: If you want to come hack on Infra things at the PTG a couple months from now in Atlanta, don't forget to sign up! 19:04:05 #link https://pikeptg.eventbrite.com/ 19:04:09 #link http://www.openstack.org/ptg 19:04:15 i'm told there is plenty of travel assistance available too--if you need it don't be embarassed to ask for it 19:04:22 #link http://www.openstack.org/ptg#tab_travel 19:04:34 as always, feel free to hit me up with announcements you want included in future meetings 19:04:48 #topic Actions from last meeting 19:04:54 #link http://eavesdrop.openstack.org/meetings/infra/2016/infra.2016-11-29-19.03.html 19:04:58 fungi send summit session summary to infra ml 19:05:02 #link http://lists.openstack.org/pipermail/openstack-dev/2016-December/108536.html Ocata Summit Infra Sessions Recap 19:05:06 FINALLY! 19:05:11 \o/ 19:05:34 and reply to that with any corrections/additions, of course 19:05:42 pabelanger add openstackci::zuul_launcher puppet class 19:05:56 do we have a change for that yet? 19:06:02 sadly no 19:06:08 on my plate to do 19:06:18 okay, cool--thanks! 19:06:22 I'll make time this week to finish it up 19:06:25 #action pabelanger add openstackci::zuul_launcher puppet class 19:06:52 and that's it for the action items from last week, i think 19:07:36 #topic Specs approval: PROPOSED Zuul v3: use Zookeeper for Nodepool-Zuul protocol (jeblair) 19:07:57 i think this is ready 19:08:00 #link https://review.openstack.org/305506 Zuul v3: use Zookeeper for Nodepool-Zuul protocol 19:08:17 it's gone through some revision and should be fairly well vetted now 19:08:35 #info Council voting is open on the "Zuul v3: use Zookeeper for Nodepool-Zuul protocol" change until 19:00 UTC on Thursday, December 8. 19:08:41 jhesketh caught some typos, i suggest we vote on this version and i'll fix in a followup 19:08:49 sounds good to me 19:08:59 +1 19:09:45 and incidentally, i think we'll be about ready to start on this shortly after it lands 19:09:51 great timing 19:10:11 Awesome :-) 19:11:51 [eot from me] 19:11:58 thanks jeblair 19:12:06 anybody else have any concerns before we move on? 19:12:50 #topic Priority Efforts: Nodepool: Use Zookeeper for Workers (jeblair) 19:13:07 talked about this in the zuul meeting yesterday, right? 19:13:35 yes -- 19:13:46 #link zk blockers from zuul meeting http://eavesdrop.openstack.org/meetings/zuul/2016/zuul.2016-12-05-22.02.html 19:14:17 if you look at the action items there, we identified some things that we want to see happen before we think we're ready to start using the new zk nodepool builder in production 19:14:25 it's a pretty short and simple list 19:14:39 so there's a good chance we'll have it done late this week, early next week 19:15:09 nb02.o.o is already online, thanks to Shrews help 19:15:18 when that's accomplished, i'd like to switch nodepool.o.o to use the zuulv3 branch (which will cause the nodepoold there to use the builds from the zk builder) 19:15:40 and if that goes well, merge feature/zuulv3 into master shortly thereafter 19:15:50 and switch all the nodepool hosts back to running master 19:16:10 review focus on project:openstack-infra/nodepool branch:feature/zuulv3 is appreciated to get this in place in a timely fashion? 19:16:14 does that sound good to folks? any concerns? additional blockers we should add to the punch list? 19:16:28 are there no changes on master that we need in v3 first? eg should the merge come first? 19:16:34 fungi: yep 19:16:57 clarkb: jhesketh merged master into v3 recently 19:17:03 so i think we're set 19:17:08 ah ok 19:17:39 we could merge first, but this lets us quickly revert without making a mess of the git tree :) 19:17:48 ya 19:17:48 looks like maybe there's been one change on master since then? 19:17:55 If it's shifted I can merge into v3 again 19:18:11 Merge "Have an ending line-feed on the generated id_rsa.pub file" is a day newer 19:18:43 * jlvillal spots his commit 19:18:44 jhesketh: if you could do another merge, that would be swell. should be easy. :) 19:18:54 fungi: don't think that one affects us 19:19:01 but getting up to syncage would be good 19:19:05 yeah, it was https://review.openstack.org/383496 19:19:22 Yep. Is it worth waiting until we're close to deploy v3 though 19:19:30 agreed, that's more just let's make sure it doesn't get lost/conflict 19:19:51 o/ 19:20:13 jhesketh: i think we're pretty close; i say we do it now, and hopefully master won't move too much in the next week or two. 19:20:25 i think that's a fine plan 19:20:31 Okay 19:21:43 #agreed We're on track to switch to ZK-based nodepool builders in roughly a week's time. 19:22:32 it's a sweet spot between holidays, so there's hopefully a fair number of people around to work out any kinks but not such high volume of activity that the impact will be dreadful should something go sideways 19:22:53 ++ 19:22:55 yeah, and there should only be a momentary outage when we restart nodepoold 19:23:18 that's all coming before the 2nd milestone? 19:23:50 why not create a stable-2.5 branch, just in case? 19:24:15 dhellmann: yeah, we should shoot to not have this impact your... thursday activities next week? 19:24:33 the milestone is on the 15th, so next week should be fine 19:24:36 dhellmann: any specific days we need to blacklist for disruptive changes? 19:24:42 wait, what's today? 19:24:49 yeah, as long as it's before thursday it should be ok 19:24:55 today's the 6th 19:25:04 yeah, my calendar was on the wrong page :-) 19:25:10 Shrews: well, it would be a stable 0.0 branch if we did. and i don't believe we intend to support it. we made a tag for the last version of nodepool people should use if they want to avoid zk. 19:25:23 so avoiding thursday and friday would be good, but it sounds like that's the plan 19:25:59 dhellmann: yeah, i'm thinking *this* thursday or friday, or if not, early next week. 19:26:05 sounds good 19:26:06 #info Coordinate any potential disruptions late next week with the Release team. 19:26:21 this week si fine right? 19:26:33 yeah, it's a dead week on https://releases.openstack.org/ocata/schedule.html 19:26:34 pretty sure when we decided to do the xenial stuff this week it was to avoid release things 19:26:37 kk 19:26:42 once we merge v3 into master, i think we should issue another release and suggest early adopters may want to try using it along with us. 19:26:50 jeblair: ++ 19:26:55 yeah, mainly just trying to avoid changes *on* deadline days, so it sounds like your plans are fine 19:27:21 thanks for jumping in dhellmann! 19:27:28 * dhellmann goes back to lurking 19:27:41 your constant lurkiness is always appreciated 19:27:52 everyone should lurk so well 19:28:08 i definitely want to make sure we keep release activity disruptions to a minimum, in particular 19:28:33 there are even fewer of them than there are of us 19:29:04 okay, so additional release. minor bump, or major prerelease? 19:29:32 it would have to be major prerelase I think if we wanted to semver right? 19:29:58 these changes aren't entirely backward compatible 19:30:07 well, we're at a 0.x release still 19:30:09 (though for the most part your old config will work with new stuff with small changes 19:30:12 clarkb: yes - but we've never released a supported version of nodepool 19:30:14 ah 19:30:42 well, depending on how you read this with semver, it might actually could be a micro, because it is backwards compatibleish 19:30:47 so 0.4.0 would be reasonable, but i could also get behind 1.0.0.0b1 19:31:03 i guess depends on how ish the ish part is 19:31:17 the ish part is you need new config for zk and need to have a zk server 19:31:31 everything else is compat I think 19:31:34 i'm assuming nodepool 1.0.0 is targeted roughly coincident with zuul 3.0.0 19:31:38 yah. 19:32:45 at any rate, i lean toward 0.4.0 with a slightly generous interpretation of semver 19:33:06 i think we agreed to pre-announce any nodepool backward incompatibilities on the infra ml in advance too? or was nibalizer satisfied if we just told people to pin to something a while back? 19:33:27 i mean, i know we pretty thoroughly announced the zk addition was in the pipeline 19:33:28 ya I think we told people to pin. maybe double check we didn't say pin to <1.0 19:33:38 because that could influence this version picking 19:33:55 i'm still happy to pre-announce both the merge and the release. 19:34:17 good opportunity to let people know what's happening, and in the case of the release, let people know how to start using it 19:35:33 (but both of those will happen after the v3 branch is in prod, so we have some time) 19:35:55 awesome 19:36:43 okay, anything else to discuss right now on the road toward nodepool 1.x? 19:37:42 nope 19:37:52 thanks jeblair! 19:37:54 #topic Priority Efforts: Newton testing on Xenial (clarkb) 19:38:13 So this is mostly a heads up that we are moving ahead with the day we picked in Barcelona (today) 19:38:22 #link https://etherpad.openstack.org/p/xenial-work-remaining Work remaining for Newton testing on Xenial 19:38:26 please all help with reviewing: https://review.openstack.org/#/q/topic:st-nicholas-xenial+status:open 19:38:35 my rough process has been to go through jenkins/jobs/*.yaml alphabetically and crank out changes 19:38:41 AJaeger is going through the list in reverse sort order 19:38:44 we've frozen project-config and will only merge these xenial changes for now. 19:38:54 #info Today is the Xenial cut-over flag day; conversion changes are in flight. 19:39:01 anybody that wants to help? Reviewing and doing changes, both is needed! 19:39:17 anyone that cna help review is much appreciated. If you want to help write changes too we can carve you out a chunk of yaml files in the middle of the alphabet 19:39:53 I have a feeling this work will carry over into tomorrow. Just based on how much progress we have been making 19:39:57 * AJaeger is not sure whether we finish today - I'm at s and will finish that but not sure I can take on much more. 19:39:59 its not really slow just lots to do 19:40:03 #link https://review.openstack.org/#/q/is:open+topic:st-nicholas-xenial please review remaining job changes for the switch to xenial 19:40:40 i don't think it's crucial that we _finish_ today. we told people we'd start force-converting them today 19:41:09 and progress has been great so far (thanks especially to AJaeger and clarkb who have been writing most of them) 19:41:11 clarkb: I won't be able to help much tomorrow 19:41:25 AJaeger: ok I am sure there will be others around. Thanks for all the work you have done its been a big help 19:41:39 fungi: I do think we want to get done this week to avoid release team conflicts but yes 19:41:41 clarkb: thanks for driving this! 19:41:51 clarkb: yep, completely agree 19:42:11 also once it's done, i think we can put the last nails in the coffin for this priority effort/spec? 19:42:29 yup 19:42:41 will be nice to scratch one more off the list 19:43:08 though I do think it is showing us we have a lot of cleanup that we should push on. Basically there is a lot of cruft in our jobs. Experimental and non voting jobs that in theory don't have anyone caring for them since they haven't been updated 19:43:24 we should probably think about clearing that out before we do any zuulv3 transition 19:43:27 yeah, looks like the last work item will be covered: 19:43:27 reduces the problem set 19:43:46 #link http://specs.openstack.org/openstack-infra/infra-specs/specs/newton-on-xenial.html#work-items Newton testing on Xenial: Implementation Work Items 19:44:26 yes, we need some more cleanups - I'm doing minor local ones while changing but we need some more later 19:44:27 (I don't think this cleanup should be tracked under this priority effort, its just something I am noticing as I do the work and it could make zuulv3 easier to get it done) 19:44:45 SpamapS: ^ that might be a good task to get on one of the planning boards? 19:45:18 basically before we convert our corpus of job configs, try to filter out any unused cruft 19:45:26 AJaeger: I'll review again shortly 19:45:40 fungi: we can, but i don't want to set it up as a blocker 19:45:56 fungi: we're going to end up with tools to convert our jobs mostly automatically 19:45:59 ya not sure its a blocker 19:46:20 so they should not be a large cost to the conversion -- cleanup can happen both before and afterwords 19:46:21 okay, so more of a "it will be a nice low-hanging fruit task" 19:47:05 more of a "we should really do this beacuse its getting gross in there a bit" 19:47:21 anyway, in summary: review review review 19:48:03 lots of very similar but subtly different changes in flight in project-config, and we've mostly frozen job config changes for any other efforts until we worth these through to inimize merge conflicting 19:48:08 we also need some mroe reviewers - yolanda and myself review a lot on project-config, we could use a third reviewer ;) Especially for our own changes I have to ping quite often ;( 19:48:37 we emptied the open queue before the freeze to an all-time low of 89 open reviews ;) 19:48:56 totally. infra needs more reviewers in general, but the project-config reviews are extremely high-volume. i also hope that zuul 19:49:04 v3 minimizes that some 19:49:25 yes, i'm focusing less on project-config reviews so i can make zuulv3 so we have fewer project-config reviews 19:49:40 i'm sorry that makes things worse right now, but i hope it will make things better in the future 19:49:52 i can make an effort to get back to more project-config reviews 19:49:54 jeblair: appreciated 19:50:14 thanks, ianw 19:50:36 I can chip in more too 19:51:20 doesn't look like we have any general topics on the agenda today, so... 19:51:23 #topic Open discussion 19:51:57 I guilty ask for a review request: devstack-gate https://review.openstack.org/#/c/396717/ 19:52:03 i don't really know what happened with the pypi mirror volume the other day. if anyone with more detailed afs knowledge can glean anything from the logs, that would be cool 19:52:05 And a meetbot one: https://review.openstack.org/404407 19:52:16 oh good I have alredy reviewed that one 19:52:22 heh 19:52:38 and yes, thanks to everyone who finds time to do some reviewing in general. we've lost some key contributors partially or completely in recent months, but our change volume hasn't dropped appreciably 19:53:48 jlvillal: thanks, that also reminds me... 19:53:53 #link http://lists.openstack.org/pipermail/openstack-infra/2016-December/004951.html the fedora community wants to collaborate with us on our meetbot fork, and possibly on the errbot-based rewrite 19:54:32 \o/ 19:54:42 i'm always happy when we get opportunities like this to collaborate on community infrastructure projects with others outside openstack 19:55:13 ++ 19:55:45 fungi: collaboration on meetbot sounds awesome indeed 19:55:49 bkero: harlowja: ^ i think both of you had maybe looked into some of our ircbot stuff, so that thread might interest you? 19:56:54 flaper87: yeah, it's dead upstream, and our fork seems to be the only active one around. so i guess we're upstream now :/ 19:57:02 fungi: funny how that happens 19:57:14 "you are crazy enough to be using that software I wrote? tag you're it!" 19:57:19 Yeah. Last commit upstream seems to be 2010 or something like that. 19:58:07 fungi: regarding cleaning up jobs, I don't think we've defined the set of jobs that have to run reliably to get to operational zuulv3. The capabilities have been the focus. But it's a good point that as we get closer, we'll want to cut the fat. 19:58:27 fungi: Sounds good to me 19:58:32 Thanks for the mention 19:59:10 aaaaand... we're just about out of time. thanks everyone! 20:00:00 see you next week, same bat time, same bat channel 20:00:02 #endmeeting