19:01:35 #startmeeting infra 19:01:36 Meeting started Tue Apr 24 19:01:35 2018 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:37 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:40 The meeting name has been set to 'infra' 19:01:56 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:03:03 ianw has an apologies entry at the top of the agenda, I don't think ianw is going to be attending the meeting due to holiday in australia but I think that is related to the dib changes realted to initramfs generation? 19:03:12 o/ 19:03:20 I'll assume that is what it is and no apology necessary, we got it sorted out if not fully debugged and are functional again 19:03:27 #topic Announcements 19:03:46 i read that as 'apologies for absence' 19:03:59 oh that may be what it is 19:04:05 in which case no apology necessary either :P 19:05:05 * corvus apologizes for absinthe 19:05:16 As for announcements, a TC election has just started. If youa are eligible to vote please don't forget to do so 19:05:42 you should have an email in your inbox with a unique voting url. If you don't have one and expected one you can reach out to the election officials to help debug that 19:06:25 #topic Actions from last meeting 19:06:32 #link http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-04-17-19.01.txt minutes from last meeting 19:06:34 hello 19:07:09 no actions recorded 19:07:42 #topics Specs approval 19:07:59 #topic Specs approval 19:08:20 #link https://review.openstack.org/563849 Direction for third party CI tooling 19:08:51 this is a spec that ianw pushed up recently that is the output of a previous meeting. I don't think its ready for approval yet but wanted to point it out since we were talking about thsi in previous zuul and infra meetings 19:09:00 if you get a moment review of that would be helpful 19:09:21 #topic Priority Efforts 19:09:34 fungi: Anything new from the land of storyboard? 19:10:00 aside from prepping for barbican migration scheduled for friday, nothing new on my end yet 19:10:15 #link https://review.openstack.org/564047 Add Barbican to StoryBoard 19:11:05 #topic Modernizing Control Plane Management 19:11:29 cmurphy has done some great work to revitalize the puppet stuff necessary to update. There is a new spec too 19:11:41 or rather an updated spec 19:11:49 #link https://review.openstack.org/#/c/449933/ Upgrade to puppet 4 19:12:09 I haven't seen an updated ansiblification spec nor mordred's container spec yet 19:12:29 pabelanger: mordred ^ it would be helpful to get those up soon as I think at laest on the puppet side we are ready to give it a proper consideration 19:12:56 also proposed moving mordred's old puppet 4 spec out of the help wanted section https://review.openstack.org/#/c/563592/ 19:13:12 clarkb: ack, will make time tomorrow 19:14:00 if we get those specs all up this week I'll send email to the infra list and point them all out so that we can work on reviewing and reading them prior to next weeks meeting 19:14:14 (we can also use that thread for communicating about this as well) 19:14:35 I'll have to see where mordred is at too to see if that is still planned 19:14:41 (I think he is traveling in italy or something for a pycon) 19:15:06 cmurphy: considering you've been pushing on this anything else worth adding? 19:15:25 i don't think so 19:15:46 will probably be able to close out https://review.openstack.org/#/c/563592/ soonish 19:15:52 cool, lets aim for next week having concrete comparison then. Thank you for helping on this 19:15:57 np 19:16:11 #topic General Topics 19:16:44 Quick status update on pip10 situation. It happened once, then sort of happened again once our images updated late last week 19:17:05 people seem to be aware of the general problem though and have been addressing it in projects as necessary 19:17:38 fallout was less serious than i anticipated 19:17:39 This reminds me, that we should probably do a glance over of infra's deployment logs to ensure we aren't hitting it anywhere in our deployments (I can work on doing that) 19:17:47 so, other than devstack, is everything using pip10 now? 19:17:54 in jobs 19:18:03 afaik, yes 19:18:11 its definitely what you get by default on our images 19:18:27 I'm not sure if other deployment testing tooling has had similar problems and if they have pinned 19:18:31 if some teams are overriding it in their in-repo job configs, we likely won't find out 19:18:45 yah, a few hiccups with centos-7, but think they've been all fixed by ianw 19:18:51 #link https://review.openstack.org/#/c/561426 Devstack psutil fix 19:19:03 #link https://review.openstack.org/#/c/562884 devstack workarounds 19:19:13 #link https://review.openstack.org/#/c/562937 Devstack don't install numpy 19:19:22 #link https://review.openstack.org/#/c/561597/9 Devstack use pip10 19:19:38 and it seems like ianw is advocating for fixing the 3 things pip10 breaks in devstack and leaving it as-is, rather than clarkb's put-it-all-in-a-venv idea? 19:19:51 corvus: I think for the short term that is the suggestion. 19:20:10 corvus: on my put it all in a venv idea ianw has asked that we get broader consensus since it will affect all the things 19:20:21 clarkb: like plugins, etc? 19:20:27 also brought up the idea of running it by the TC so I've added it to the TC's office hours discussion topics list 19:20:28 corvus: exactly 19:20:30 longer-term, the venv solution seems like nice future proofind 19:20:32 proofing 19:20:52 since we don't know when the next dep will get added that conflicts with something distro-installed 19:21:17 nor whether we can make the same compromises on other platforms the devstack team might want to extend support for 19:21:25 fungi: or if pip11 will be more aggressive in this upstream position 19:21:30 yep 19:21:51 clarkb: didn't you have a ml post? 19:21:59 there was indeed 19:22:06 with basically nobody replynig 19:22:17 ya that came up when I talked to doug briefly about it 19:22:38 well, clarkb knows i support it; i didn't think it needed a +1 from me, but i'd be happy to reply with that if it would help 19:23:02 corvus: ya it might be good to have someone other than me explain why they think it would be a good idea too :) 19:23:34 what i can't really weigh in on is the impact to plugins; i understand that less. 19:23:34 I think the fear is that we know it will make people's lives more difficult near term and no one wants to have the crwods wrath sent their way 19:23:43 so better to at least make sure we make the right decsions when we do that and point to the reasoning 19:24:23 +1 19:24:44 maybe making the venv codepath optional via a toggle which we warn teams we'll be throwing at the start of the stein cycle? 19:25:00 fungi: ya I have to make it option for grenade support anyways 19:25:03 fewer torches and pitchforks maybe 19:25:08 fungi: so maybe thats a good first step, get the support in but leave it off by default 19:25:20 turn it on in some jobs, warn people to test it and flip going forward 19:25:21 clarkb: is there a path to support grenade eventually? 19:25:36 clarkb: (like, after the toggle is in both sides of a grenade test, grenade can use it?) 19:25:38 clarkb: we should atleast have some jobs running it, since we did have USE_VENV but was completely broken 19:26:06 corvus: yes, one option is to backport it to all the old devstack and support grenade the same way in all envs. We would merge the feature to all devstack branches first, but not enabled it. Merge usage and enabling of feature to all grenade branches. Then enable it in devstack by default 19:26:12 (this is necessary to avoid force merging) 19:26:18 corvus: exactly 19:26:26 clarkb: cool 19:26:42 the other is we roll forward and have feature in master (rocky) today then only use it in rocky -> stein upgrades and so on 19:27:09 I think ^^ leaves us open to greande jobs and older devstack jobs running on stable branches breaking due to reasons fungi pointed out 19:27:20 and its probably best to just get it addressed across the board 19:27:38 yeah, though less likely on stable branches 19:27:53 either way, the plans are compatible -- it's mostly a matter of how fast we do it :) 19:27:54 yeah, pip is one of the few deps we shouldn't really be trying to pin, even on stable branches 19:27:58 corvus: yup 19:28:19 but we should pick our battles 19:28:35 getting things working safely for master can be first priority 19:28:47 I also think that now that dust is settling and people have had a few days to understand the underlying problem rather than just make jobs work as fast as possible we can better discuss long term direction 19:29:14 (I think the pre release discussions suffered from this problem, we said it was an issue and would break us but until it did it was low priority) 19:30:08 The other two items on the agenda are the Third party CI spec which I broght up earlier. Ianw asking for early feedback on it. 19:30:16 And DIB, Xenial, and rackspace problems 19:30:21 i think the warnings we provided ahead of time, even if partly ignored, helped 19:30:43 One of the problems with the pip10 rollout was we had paused xenial image builds because they stopped booting on rax 19:31:11 we traced this back to a change made in dib to rebuild the initramfs at the end of the job which broke udev and glean and system cooperating to produce working networking at least on rax 19:31:18 yah, still haven't really figured out why glean stopped working 19:31:23 that was mostly unfortunate coincidence, but it did end up masking/delaying some issues 19:31:39 That change was reverted in dib but we should likely work to figure that out because it was made to fixing an lvm based image problem 19:31:55 if you have knowledge of the initramfs and udev and glean and systemd your help would be much appreciated 19:33:15 I think I've had to declare bankruptcy on getting normally scheduled work operations done ~twice in the last week or so 19:33:18 its been fun :) 19:33:27 unfortunately the wrong kind of fun 19:34:17 I think that is about it. 19:34:25 #topic open discussion 19:34:34 What did I miss? 19:35:01 I could use a few more reviews on https://review.openstack.org/#/q/topic:debian-stable+status:open to help finalize the removal of debian-jessie from nodepool. I'll then start the process to clean up AFS mirrors 19:36:09 well, you didn't miss this -- but clarkb, jlvillal, and i had a chat earlier and jlvillal is interested in helping on the gerritbot portion of the limnoria spec -- so i've created a feature/limnoria branch of gerritbot, jlvillal will start making a gerritbot limnoria plugin there, and later this week, i'll start on the puppet to install/run limnoria on eavesdrop. 19:36:28 and mostly a FYI: been pushing on the removal of old jenkins based scripts from bindep: https://review.openstack.org/#/q/topic:bindep.txt+status:open all in the effort to help move bindep.txt in tree for projects. That allows us to then remove bindep-fallback.txt in 2 cycles 19:36:59 there's *plenty* of opportunity for parallel work on that spec -- each of our bots should be converted at a minimum, and there's some new plugins to write. 19:37:24 right since the bots work as plugins to the central framework we don't need to produce them serially 19:37:24 lastly, ubuntu-bionic release is this week. So we should keep an eye out for projects wanting to gate on it. 19:37:37 pabelanger: thursday is official day right? 19:37:48 clarkb: yah 19:37:52 thankfully bionic is a lot less dramatic of an update for server users I expect 19:37:58 the big chagne is all on the desktop side of the world 19:38:06 yup, we are in really good shape this time around 19:38:21 pabelanger: we're carrying around 'legacy' stuff a bit longer than i'd like, but since devstack/tempest are still evolving a little, i suppose it's good to maintain it for a bit longer. ++ 19:38:29 #link https://review.openstack.org/557979 work is also progressing on the limesurvey implementation 19:38:45 i'm hoping to have time to take the latest patchset for a spin tomorrow 19:38:47 maybe we can set a cycle goal for stein or something to remove all legacy jobs 19:39:21 corvus: yah, I think stein might work. 19:39:35 yes, elimination of legacy imported job definitions might make for a nice cross-project goal 19:40:16 the bindep changes didn't get much push back, and our bindep role was a nice noop, which made it nice and easy! 19:41:40 do we want to have 19 minutes back to eat lunch or have an afternoon snack or find breakfast or eat dinner? 19:41:59 or all of the above? 19:42:07 i need to heat up the wok for some shrimp fried rice, so would welcome a few spare minutes ;) 19:42:14 mmm breakfast for dinner during afternoon snack time 19:42:33 alright, if there is anything else feel free to find us on the infra mailing list or in #openstack-infra 19:42:34 not sure whether it's lunch or dinner, but it'll be tasty either way 19:42:36 thank you everyone 19:42:41 thanks clarkb! 19:42:43 #endmeeting