19:01:29 <clarkb> #startmeeting infra
19:01:29 <openstack> Meeting started Tue Dec 12 19:01:29 2017 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:31 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:33 <openstack> The meeting name has been set to 'infra'
19:01:41 <clarkb> #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting
19:02:01 <clarkb> #topic Announcements
19:02:03 <pabelanger> o/
19:02:15 <ianw> o/
19:02:26 <clarkb> It is now official dmsimard and frickler are infra root
19:02:32 <dmsimard> \o
19:02:48 <pabelanger> welcome to the jungle!
19:03:04 <jeblair> yay!  try not to delete the backup server!
19:03:11 <fungi> the assistance is most heartily welcomed
19:03:11 <pabelanger> lolo
19:03:17 <frickler> o/
19:03:24 <jeblair> at least, not by accident
19:03:53 <clarkb> Other than that it is the time of year where those of us in the northern hemisphere end up sick and everyone is randomly on vacation or otherwise not working
19:04:09 <clarkb> mordred and I apparently have the plague (though I am getting over it finally)
19:04:17 <clarkb> so ya stay healthy
19:04:22 <pabelanger> +1
19:04:27 <clarkb> #topic Actions from last meeting
19:04:30 <fungi> plenty of limes in my grog
19:04:38 <fungi> #action fungi delete jenkins account -1 verify votes from open changes in gerrit and announce it
19:04:42 <clarkb> #link http://eavesdrop.openstack.org/meetings/infra/2017/infra.2017-12-05-19.01.txt Minutes from last meeting
19:04:42 <fungi> i haven't done that yet. wanted to find a quiet time due to the number of code review events that's going to generate, and ended up busy with other things over the weekend
19:04:56 <clarkb> fungi: it also hasn't been very quiet when not weekend
19:05:10 <fungi> now, for example, would not be a great time ;)
19:05:38 <clarkb> #topic Specs approval
19:05:46 <clarkb> #link https://review.openstack.org/#/c/524024/ Top level project hosting
19:06:14 <clarkb> I've now reviewed this spec a few times. I think it is ready to be put up for council voting. I think others have managed to take  a look as well
19:06:22 <clarkb> What do we think? ready to push forward?
19:06:33 <fungi> i think so
19:06:37 <jeblair> as do i
19:06:59 <frickler> +1
19:07:09 <pabelanger> I haven't look at the latest, but think there is momentum around it
19:07:15 <pabelanger> plan to look again after meeting
19:07:53 <jeblair> fyi, i just put the mailman multi-site patches up for review
19:07:54 <fungi> the diff from ps1 is pretty small
19:08:52 <jeblair> there is opportunity for other folks to jump in as well (there are 4 services this addresses)
19:09:02 <pabelanger> jeblair: we are aiming to get it done before PTG right?
19:09:09 <dmsimard> I'm ok with the proposal, I can add my +1
19:09:37 <jeblair> yes, i would like to use this for zuul before we release 3.0 which i want to do before the ptg
19:09:38 <clarkb> ok lets put it up for council vote then. I'll look over reviews probably thursday afternoon
19:09:47 <pabelanger> wfm
19:09:59 <dmsimard> I just put up another spec if we want to share it for this topic
19:10:02 <jeblair> so that's driving my enthusiasm for quick implementation :)
19:10:16 <fungi> jeblair: i suppose goal being that we can have the updated urls/addresses for docs, git repo and mailing list?
19:10:41 <jeblair> fungi: yep, so we're not wasting a marketing opportunity
19:10:44 <fungi> included in the zuul 3.0.0 release announcement i mean
19:10:52 <clarkb> ya I think you don't want to tell everyone to look somewhere new shortly after a major release
19:10:53 <fungi> (and in its docs and whatnot)
19:10:58 <clarkb> best to get that all lined up along with release
19:11:24 <jeblair> also, it will help convince folks we're not solely openstack-focused
19:11:34 <jeblair> which is a big line item on the 3.0 release
19:11:55 <pabelanger> +1
19:12:05 <fungi> "zuul: it's not just for openstack any more"
19:12:13 <jeblair> also, since lists.katacontainers.io exists, we can fold that back into the one true listserver after this lands
19:12:14 <dmsimard> there is no openstack, only zuul
19:12:14 <fungi> (not that it particularly was before)
19:12:44 <pabelanger> Yah, ptg is coming up fast, so agree we should vote
19:13:15 <clarkb> #agreed https://review.openstack.org/#/c/524024/ up for council vote this week. Please vote by Thursday afternoon Pacific Time
19:13:35 <fungi> dmsimard: did you intend to rollcall +1 that spec?
19:13:44 <dmsimard> can I do that ?
19:13:49 <dmsimard> or is it just TC ?
19:13:54 <fungi> infra council
19:14:05 <dmsimard> I mean, I see I have the permissions to do that, but I didn't think I would -- okay, I will.
19:14:10 <fungi> if you see the option to use it, then yes you can use it ;)
19:14:20 <clarkb> your membership in infraroot is what gives you that and yes you can use it
19:14:30 * dmsimard still learning
19:14:37 <jeblair> dmsimard, frickler: https://docs.openstack.org/infra/system-config/project.html#teams
19:15:12 <dmsimard> I didn't even know infrastructure council was a thing, TIL.
19:15:47 <jeblair> it's essentially our group decision making process
19:16:10 <jeblair> hopefully just the right amount of structure without being too much bureaucracy
19:16:19 <dmsimard> wfm
19:16:21 <clarkb> are there any other details/concerns about this spec and/or effort that we want to talk about here?
19:16:32 <clarkb> (otherwise I think we should keep moving because there are things to talk about!)
19:16:45 <jeblair> clarkb: eot from me
19:16:57 <clarkb> #topic Priority Efforts
19:17:05 <clarkb> #topic Zuul v3
19:17:26 <clarkb> Couple of things on zuulv3 front off the top of my head, the new dashboard has been deployed. It is shiny
19:17:34 <clarkb> But also we are currently suffering from a memory leak
19:17:58 <clarkb> (probably dont' want to debug the memory leak here, but worth communicating it is a problem we know about)
19:18:20 <fungi> debugging is happening in #openstack-infra-incident
19:19:11 <clarkb> GitPython got a release
19:19:19 * fungi cheers
19:19:22 <clarkb> Command socket changes are happening as is the finger server work
19:19:35 <clarkb> Seems like a steady amount of progress
19:19:54 <jeblair> yep, chugging along
19:20:46 <fungi> in yesterday's meeting we also confirmed some initial direction for having a usable but minimal base job in the stdlib
19:20:48 <AJaeger> sorry for beeing late
19:21:27 <pabelanger> also confirmed with leifmadsen zuul quickstart docs in progress. going to sync up and help with testing them
19:21:36 <clarkb> pabelanger: are there patches yet?
19:22:05 <pabelanger> clarkb: just an etherpad, let me get url
19:22:17 <pabelanger> https://etherpad.openstack.org/p/zuulv3-quickstart
19:22:28 <pabelanger> I was going to see about first pass and moving that in to RST format
19:24:03 <clarkb> ok, one question. As a non user of the red hat distros, would it make more sense to provide a distro agnostic guide? install zk from tarball, run zuul from virtualenv sort of thing?
19:24:11 <clarkb> I guess we don't have to sort that out here and now
19:24:27 <pabelanger> clarkb: yah, I think we could
19:24:38 <fungi> that way lies the trap of the openstack install guides ;)
19:24:55 <jeblair> i don't think the distro part of it is that big
19:25:01 <AJaeger> yeah ;( Been there and had the fun ;(
19:25:27 <fungi> granted, i don't have a good answer for the balance between lots of distro-specific install guides and trying to make distro-less inslall instructions
19:25:38 <jeblair> the zuul/nodepool bits will all be from source
19:25:52 <pabelanger> yah
19:25:58 <jeblair> that doesn't leave much distro-specific stuff
19:26:06 <dmsimard> the distro part is mostly around if you want to installed packaged version of things
19:26:08 <pabelanger> fedora has bwrap by default, xenial needs external PPA atm
19:26:09 <dmsimard> like zookeeper, etc
19:26:16 <AJaeger> we can have distro specific initial paragraphs - if somebody provides them. Like install package A, B,C
19:26:17 <clarkb> jeblair: ya I think the thing that made it clunky to me is that for eg centos you already have to break out of the distro to zk
19:26:24 <clarkb> so maybe just do that in the base docs
19:26:32 <pabelanger> or create bindep.txt file :D
19:27:21 <clarkb> maybe that specific section could be install from tarball if you don't have distro package for it then it applies, but details like that aren't important for now :)
19:28:46 <pabelanger> sure
19:29:07 <fungi> sure, i suppose we don't need the guide to cover "how to install <random dependency>" we can just link to their documentation
19:29:20 <pabelanger> fungi: yah, good point
19:29:38 <AJaeger> yes - we're targetting knowledgable admins
19:30:16 <fungi> no doubt the zk docs have ample detail on how to get up and running on a variety of distros anyway
19:30:21 <ianw> ... who might be more interested in reading puppet/ansible anyway?
19:30:49 <dmsimard> who knows, maybe this'll fork out to windmill and/or the nodepool/zuul roles anyway
19:30:51 <dmsimard> :D
19:31:19 <clarkb> ya I think the thing I would try to avoid is having so much detail that we end up with 10 ways to zookeeper
19:31:25 <clarkb> (currently we have 2)
19:31:57 <clarkb> any other zuul related topics people want to discuss?
19:32:30 <frickler> yes
19:32:44 <frickler> some folks are confused by the change to zuul.projects
19:33:02 <frickler> and I failed to find an announcement for that, do we have one or is that still pending?
19:33:02 <clarkb> frickler: from a list to a dict?
19:33:06 <frickler> clarkb: yep
19:33:40 <ianw> it's already out there, what did i miss?
19:33:48 <frickler> some jobs are currently failing since it was activated last week
19:33:48 <ianw> i thought we updated users
19:34:16 <frickler> there was a case earlier today in -infra
19:34:31 <clarkb> sounds like we thought we updated all the uses of it but may have missed some?
19:34:46 <jeblair> why don't we just use codesearch to find the remaining cases and fix them?
19:34:55 <dmsimard> ianw: possible we missed a few cases.. after all, we reverted like twice and during that time span people may have used it
19:34:58 <frickler> http://logs.openstack.org/59/527059/1/check/ansible-role-k8s-glance-kubernetes-centos/657411d/job-output.txt.gz#_2017-12-12_06_16_17_847993
19:35:18 <AJaeger> flaper87 wanted to fix that ^
19:35:27 <ianw> yep ... i can take an action item to double check things
19:36:41 <jeblair> hrm, that project doesn't show up in codesearch
19:36:43 <clarkb> ianw: frickler and maybe send a note to the dev list saying its due to a change and we are working to correct occurences of it let us know if you find them?
19:37:07 <ianw> ok, if we thing there's more than one?
19:37:16 <fungi> there's likely to continue to be people who have pending changes which reintroduce the old structure, so it will take time for them to catch on and fix their changes too
19:37:19 <dmsimard> jeblair: possible codesearch is stuck, last time that happened puppet hadn't ran or wasn't updating the list of projects properly
19:37:20 <ianw> i'm not seeing that in codesearch, which is why i'm guessing i didn't fix it :/
19:37:34 <dmsimard> pabelanger fixed it iirc
19:37:55 <pabelanger> we need to stop /start codesearch to pick up new projects
19:37:58 <pabelanger> it is a manual process
19:38:13 <clarkb> I guess step one is restart the service then
19:38:14 <dmsimard> pabelanger: that's it? just restart? We could set up a cron to run daily or something ..
19:38:20 <clarkb> then requery and see what we missed
19:38:20 <ianw> ?  that's the first i've heard of that ... that seems like something we should puppet
19:38:23 <fungi> yes, hound doesn't have a mechanism for adding or removing repos to index on the fly
19:38:36 <clarkb> ianw: it was puppeted in the past but that resulted in the service always being down iirc
19:38:52 <pabelanger> dmsimard: right, we've never picked a day for an outage. Takes about 5 / 10 mins to reindex
19:38:57 <fungi> yeah, because it takes so long to reindex everything when starting
19:39:06 <fungi> probably more than that these days
19:39:24 <clarkb> lets time it this time around then we'll have info on what might work for automating it
19:39:27 <fungi> well, also the puppeting was broken and it kept restarting the service whether or not there were new projects to add
19:39:30 <clarkb> (like pick a time for it once a day or something)
19:39:40 <clarkb> I guess that is what dmsimard said
19:40:00 <ianw> if it just replaced the webpage with "indexing ... give me a minute" i think that would be enough
19:40:04 <ianw> rather than just disappearing
19:40:43 <ianw> i will volunteer to 1) reindexing 2) looking into .projects failures 3) proposing some approach to automated reindexing if we like
19:40:52 <clarkb> cool, sounds like a plan
19:40:55 <frickler> that node is to be upgraded to xenial anyway, isn't it?
19:41:01 <clarkb> frickler: yes
19:41:32 <clarkb> lets go with that plan and the upgrade will happen somewhere in there but one doesn't really block the other
19:41:42 <ianw> good point, i can try brining it up as xenial first ... if it's going ok, then maybe switching is easy
19:42:04 <ianw> but if it gets stuck we can use the existing
19:42:31 <clarkb> this sets up the next topic nicely though which we should get into before we run out of time
19:42:34 <clarkb> #topic General topics
19:42:58 <clarkb> We are upgrading control plane servers in a virtual sprint this week
19:43:15 <AJaeger> if you go over xenial updates, please check also whether we have dead repos or files in system-config. Let's retire and cleanup.
19:43:25 <clarkb> I think things have gone reasonably well but we've also hit a few speedbumps.
19:43:30 <AJaeger> My small cleanup :  https://review.openstack.org/#/q/topic:retire-puppet-apps - retire puppet-apps_site
19:43:56 <clarkb> There is a bit of boilerplate necessary to to add digits to server fqdns for example
19:44:04 <dmsimard> I'm trying to figure out a generic playbook (which might end up living in a role) to re-install things, especially considering 18.04 is coming soon: https://review.openstack.org/#/c/527301/ I'll work on it some more and let you know when I feel it's ready for review but feel free to leave comments etc
19:44:33 <ianw> AJaeger: maybe put that under infra-xenial topic if you like, i'm checking that regularly
19:44:37 <clarkb> dmsimard: I'm worried that we'll need service specific upgrade roles/playbooks just to cover all the bases
19:44:48 <fungi> i'd be wary of automating reinstallation given we have services relying on remote databases, with data on cinder volumes, et cetera
19:44:53 <dmsimard> clarkb: yeah, that's why it might end up forking out to different roles
19:44:56 <clarkb> dmsimard: because we run such a diverse set of services there isn't always consistency in ya what fungi said
19:45:06 <pabelanger> it is possible we could do this in post pipeline now, and only trigger when we add new projects
19:45:16 <pabelanger> ianw: ^
19:45:17 <jeblair> re-install what?
19:45:19 <AJaeger> clarkb: shall I follow ianw 's suggestion to change the topic?
19:45:29 <dmsimard> fungi: the goal is mostly to target the "easy" ones, like logstash-workers.
19:45:43 <dmsimard> There's a lot of them and it's not very hard to automate.
19:45:56 <clarkb> AJaeger: ya that works for me especially since it is related
19:46:09 <dmsimard> I understand there's maybe 50% of "pets" we might not be able to automate but if we can automate the 50% of cattles, it'll save a good amount of time
19:46:22 <pabelanger> http://git.openstack.org/cgit/openstack/ansible-role-cloud-launcher also has logic to control server creation, we just never finished off running puppet after server was launched
19:46:39 <AJaeger> clarkb, ianw will do
19:46:45 <fungi> well, also a lot of the work this time around, like clarkb noted, is switching from our old model of replacing servers with more of teh same name to having numbered servers
19:46:57 <fungi> that won't have to be redone in the future (in theory)
19:47:41 <pabelanger> AJaeger: do you know the status of install_tox.sh removal in projects?
19:47:44 <ianw> i think anything's automatable ... the question for me is if you spend more time automating something than actually using the automation.  i do think there's diminishing returns
19:48:29 <AJaeger> pabelanger: all the ones that can be done, are done ;) Meaning: Repos that need to install other repos like horizon or neutron are not done yet. We don't have a solution for them so far.
19:48:39 <dmsimard> ianw: cue xkcd :D
19:48:45 <clarkb> Given what we've gone through the last couple days or so are we finding this time is valuable (new roots finding it useful)?
19:48:52 <fungi> dmsimard: https://xkcd.com/1205/
19:48:58 <clarkb> I've sort of been all over the place playing support so want to make sure that I'm not missing anything important
19:49:05 <dmsimard> fungi: oh I had another one in mind
19:49:17 <fungi> ahh
19:49:27 <pabelanger> clarkb: I think so, we've had new people creating servers
19:49:34 <dmsimard> fungi: https://xkcd.com/1319/
19:49:53 <fungi> dmsimard: yes, also appropriate
19:50:02 <AJaeger> pabelanger: https://review.openstack.org/#/q/status:open++topic:rm-tox_install show open ones - and we currently have 84 tox_install.sh files still in tree (including the open ones)
19:50:07 <dmsimard> clarkb: I'm learning a bunch so it's been a great experience for me
19:50:14 <dmsimard> definitely a good way to ramp up new roots imo
19:50:23 <AJaeger> pabelanger: so, those need some more work - mordred wanted to find a solution, help is welcome.
19:50:41 <ianw> heh, i mean you should see the makefile for my thesis.  it's a work of make art ... built on a solid foundation of procrastination
19:50:45 <pabelanger> AJaeger: thanks, will look
19:51:31 <dmsimard> anything else for general topics ?
19:51:42 <clarkb> ok I think there was some concern the boilerplate stuff was a lot of effort for minimal return and that we might consider a different approach. I am glad we decided to do this early and not wait until next month as I think we are all learning things :)
19:52:16 <pabelanger> +1
19:52:48 <clarkb> The last general topic I wanted to bring up was project renames and possibly trying ianw's fix for nova-specs
19:53:16 <clarkb> maybe what I'll do is send out an email and poll for when is a good day to try and do that. Mordred had at least one additional project rename that isn't listed on the meeting agenda yet iirc
19:53:55 <fungi> sounds good
19:54:11 <ianw> do we have a procedure worked out now?
19:54:32 <clarkb> ianw: not a complete on I don't think
19:54:44 <clarkb> that means step 0 is write up a doc first I guess
19:54:49 <clarkb> then propose days
19:54:53 <ianw> ok, we can reuse the old doc @ https://etherpad.openstack.org/p/rename-2017-10-20
19:54:55 <ianw> it was close
19:55:16 <ianw> i will be happy to translate that into actual docs after we're done
19:55:21 <clarkb> ok I'll see if I have time to poke at that
19:55:28 <ianw> i'm fairly sure we're going to hit unexpected issues :)
19:55:32 <clarkb> indeed
19:55:39 <clarkb> #topic Open Discussion
19:55:56 <clarkb> we have ~5 minutes left by my local clock. Anything else?
19:55:58 <dmsimard> Better late than never, but I put up the spec for continuous deployment dashboard: https://review.openstack.org/#/c/527500/
19:56:16 <dmsimard> It's a first draft but ready for reviews
19:58:08 <clarkb> Holidays are fast approaching. I will be around next week and can host a meeting then (probably good for recapping sprint and zuul things) but week after I won't be here and then week after that I am not sure
19:58:29 <dmsimard> good point
19:58:30 <clarkb> I guess 1st is monday not tuesday so I will probably be here first week of the year
19:58:51 <dmsimard> I guess the infrastructure is not solicited as much during the holidays, I guess it's infra-root best effort during that period ?
19:59:04 <fungi> dmsimard: it's best effort any time
19:59:13 <pabelanger> yah, I some some PTO I have to burn before Dec.31, I might be getting an early start in coming days. Will keep people updated
19:59:16 <clarkb> and it definitely gets very quiet around here starting around now
19:59:17 <dmsimard> fungi: you're correct :D
19:59:23 <fungi> but yes, there tend to be fewer people around to fix the fewer things that break this time of year
19:59:52 <fungi> and teh community has been generally accustomed to that idea
19:59:53 <jeblair> i'll also be away the last 2 weeks
20:00:00 <ianw> i am out from 12/21 - 01/07 fyi
20:00:16 <fungi> i won't be away, but i'll be dealing with visiting family so won't be at the computer as much
20:00:45 <clarkb> so ya just be aware that it is quiet and we should probably be a bit more slushy/frozen as people may not be able to fix problems as quickly
20:00:51 <clarkb> but I think that should naturally happen with people being afk
20:00:55 <clarkb> and we are at time
20:01:01 <clarkb> Thank you everyone
20:01:03 <clarkb> #endmeeting