19:01:13 #startmeeting infra 19:01:14 Meeting started Tue May 8 19:01:13 2018 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:15 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:17 The meeting name has been set to 'infra' 19:01:18 o/ 19:01:22 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:01:35 #topic Announcements 19:01:48 Summit/OpenDev is in just under two weeks 19:02:08 If you are attending don't forget to do whatever prep you need :) 19:02:14 * corvus frets 19:02:35 * fungi is so behind 19:02:42 we need a support group 19:03:15 I was silly and decided to take a mini vacation next week for my birthday. Which further cuts into my available time (but I'm excited to go fishing and sit on the (cold) oregon beach) 19:03:28 sounds marvellous 19:03:29 Due to ^ we'll need an infra meeting chair volunteer 19:03:29 happy birthday 19:03:51 and I figure we'll cancel the May 22 meeting as many of us will be at the summit 19:04:00 Let me know if interested in chairing on the 15th 19:04:17 o/ 19:04:34 * fungi hopes that's cmurphy volunteering to chair next week's meeting ;) 19:04:41 lolno 19:04:43 fungi: I totally read it that way too :P 19:04:43 that's how I read it 19:04:49 * cmurphy runs 19:05:04 anything else need to be announced? 19:05:14 i don't seem to have any conflicts for the 15th so _can_ chair, but totally wouldn't want to take that pleasure away from anyone who wants to volunteer. i've chaired plenty of these already 19:05:33 I think cmurphy should take a turn 19:05:54 I think cmurphy might also be semi afk too so won't push too hard on that :) 19:06:01 ah okay 19:06:06 another time then 19:06:11 yeah i am in and out for the next couple weeks 19:06:25 #topic Actions from last meeting 19:06:32 #link http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-05-01-19.01.txt minutes from last meeting 19:06:57 There wasn't a specific #action recorded but more of a collective please go review our three config management future specs. Seems like there were comments on the whole set so I think people were reading them 19:07:13 Before we dig into that subject 19:07:16 #topic Specs approval 19:07:23 #link https://review.openstack.org/349831 Survey tool spec 19:07:24 patch 349831 - openstack-infra/infra-specs - Add survey spec 19:07:31 o/ 19:07:51 anteaya: has indicated that ^ should be ready for review. tl;dr is lets run a limesurvey to host the surveys that various groups want to run around openstack 19:08:06 fungi: I think you've been helping out there are you happy for that to be up for approval this week as well? 19:08:39 yeah, i think i've already registered a rollcall +1 on it 19:08:44 yup I see that now 19:08:55 and yes he has been helping out a lot 19:08:59 the biggest unknown was the self-service signup to create surveys 19:09:04 for which I'm very grateful 19:09:25 if the infra council voters could look over that spec this week it would be appreciated. I'll make a not to look at it probably late on thursday as far as approving it goes 19:09:33 *make a note 19:09:40 but digging into configuration now that we have a demo/prototype up, it looks like it should be able to handle webserver auth so we have a good chance of getting apache mod_auth_openid working 19:10:06 cool 19:10:13 i'm hoping to give that a shot tomorrow now that i've found all the commented configuration defaults for limesurvey 19:10:24 thank you 19:10:30 also it does support pluggable authentication, so we could roll our own if someone gets ambitious 19:10:36 * anteaya ladles on more gratitude 19:11:04 anteaya: I read that as gravy 19:11:14 mordred, when did you last eat? 19:11:23 might you be hungry? 19:11:37 yeah, now i want to go find food after this meeting 19:11:41 anteaya: about 15 minutes ago ... but gravy is always good 19:11:42 sorry 19:11:48 good gravy 19:11:50 #topic Priority efforts 19:11:58 * clarkb takes gravy as the cue we are ready for next topic :) 19:12:11 clarkb, good pickup 19:12:11 all topics are now gravy 19:12:17 we'll start with storyboard. Looks like migrations continue. Heat is done and loci is in the queue? 19:12:27 yeah, heat was a massive import 19:12:41 >5k bugs 19:12:46 er, >4k bugs 19:12:49 still a lot 19:13:05 took in the neighborhood of 12-14 hours to import, i think 19:13:22 wow 19:13:23 loci should be quick by comparison 19:13:39 Poked at the next tripleo squad again too- trying to get them on the schedule of migrations 19:13:40 many things will be quick by comparison 19:14:05 my flight to vancouver will be quick by comparison 19:14:21 mordred, nothing compared to what neutron and nova will be (I tried to do a test of the neutron one and it ran for a week before I stopped it) 19:14:54 the model there is sort of weird though. i'm not entirely convinced it will make sense for them to import all tripleo bugs into tripleo-ci but start by only importing the ones tagged as security and then import the others into the same project later 19:15:14 er, into tripleo-common 19:15:17 not tripleo-ci 19:15:20 fungi, agreed 19:15:28 I think making a repo for them would make more sense 19:15:54 any bugs which are already in lp and then after the import they add the security tag... do those need to get closed and reopened in storyboard? no idea 19:16:41 Its definitely messy. 19:16:46 the tripleo leadership have expressed concerns that having storyboard tasks tied to repositories isn't compatible with their workflow and they'd rather just have all tasks be for their team 19:17:13 Then they should all go at once rather than squad by squad 19:17:26 probably should at least 19:17:34 fungi: that is how they operate in lp I guess? 19:18:11 yeah, there is a "tripleo" project in lp and then they add bug tags to indicate which subteams are working on what bugs, seems like 19:18:52 gotcha 19:19:17 so anyway, the closest we have under our current model (there is no "tripleo" repo) is to migrate their bugs to have tasks for the tripleo-common repo 19:19:42 but as diablo_rojo notes, their desire to migrate subteam at a time makes this extra weird 19:20:27 that seems weird to me as well - and something that seems pretty rife for confusion and stress while it's happening 19:20:57 is that something we should try to have a more direct conversation with the tripleo team about? 19:21:05 (thinking with summit coming up that may be an opportunity there) 19:21:45 while that would probably be useful, i think i'm booked solid by this point 19:22:01 We can add it to the agenda for the storyboard forum session monday 19:22:01 ya I feel that way too 19:22:02 unless we can somehow have that conversation while i'm asleep 19:22:04 if they can make it 19:22:16 fungi: or corner mwhahaha and EmilienM_PTO in a bar on tuesday 19:22:17 i predict i'll be having many conversations while asleep by the last day 19:22:55 * mwhahaha disagrees with the confusion part but whatever 19:23:04 fungi: we're allowed to sleep at the summit? 19:23:08 mordred: you aren't 19:23:23 i won't be at summit, but i'd be happy to have a chat about it sooner, as we need to figure out the migration to storyboard anyway 19:23:26 mwhahaha, would you be able to make it to the Storyboard session Monday? 19:23:29 Dang 19:23:31 nvm 19:23:59 tl;dr, we probably should just mass cutover sooner but i need to sent a note to th eml this week 19:24:07 this subteam move probably isn't working out 19:24:48 cool sounds like we should be able to sort something out then 19:25:02 (both parties are motivated etc) 19:25:51 mwhahaha, it worked for a few of the squads, but yeah its probably time if we can swing it with your conributors 19:26:29 it's not really working cause it's confusing everyone on where to create and there's no fix except duplicating work 19:27:27 that's the sort of confusion i was concerned partial migration would cause 19:29:07 alright are we ready to move on to the next topic? 19:29:30 #topic Modern Config Management 19:29:35 #link https://review.openstack.org/449933 Puppet 4 Infra 19:29:40 #link https://review.openstack.org/469983 Ansible Infra 19:29:45 #link https://review.openstack.org/565550 Containerized infra 19:30:04 Now that we've had a chance to read these through and consider them, what are our thoughts/impressiosn? 19:30:14 pabelanger pointed out that ansible and container potentially share a common base 19:30:51 +1 19:31:06 that is the setup email and users and install docker portion right? 19:31:08 so if we like either of the two of them, it might be worth splitting out an ansible-base spec to describe that portion 19:31:11 clarkb: yah 19:31:12 yeah, and to some extent, we may end up with some hosts where we have some ansible and no containers (eg, afs servers). it may really be a matter of degree? 19:31:18 and how to deal with hiera/secrets 19:31:24 corvus: ++ 19:31:59 I am however, liking puppet-4 (I guess 5 because of bionic) as it might be less work 19:32:26 i'm leaning toward the container approach, mostly because it completely decouples us from operating system dependencies 19:32:35 my initial impression was that puppet 4 and then 5 feel like less upfront work. Like we can probably get to 4 in a week or two. But then persistent issues like bionic not having puppet 5 packages and the weird renaming of silly things and distros undoing that seems like pain I'd like to avoid 19:32:40 I would prefer either ansible or container specs (or some combo of the two) - but obvs sinceI wrote it I prefer the container one 19:32:51 i'm trying to verify whether puppet 4 will be supported on bionic, i think it will be 19:33:19 corvus: ++ - that's my main motivation from it - that and it shifts some burden to the ci system 19:33:26 which, it turns out, we hae a good one of 19:33:52 and ya I think containerizing applications gives us some new features that will be useful over time that both puppet and ansible largely lack as we'd use them otherwise 19:34:35 If I replace containers with the word packaging, I think I'm okay with them :) mordred did a good job outlining some use cases I was having trouble seeing 19:34:41 (whereas puppet and ansible are roughly equivalent in "power"/utility, puppet even has thair own remote ssh tool now (its literally just ansible written in ruby for puppet)) 19:34:43 I'd just like to see whatever the decision is well publicized so that in 2 months when someone says, hey you should all do [thing] someone can point to a url 19:34:47 it's still not clear to me that ditching puppet for $next_thing necessarily fixes the need to periodically update our configuration nor the odds of backward-incompatible changes to file paths over time 19:34:53 pabelanger: \o/ (and yea, that's how I've been thinking of them) 19:34:54 with puppet, we have to have a working version of puppet on every host, and if that version is different (it often is) things get complicated. with ansible, we only need a working version of ansible on one host (the controller). and with containers, because so many of our services are things we run from source, we can drop a lot of the work we're doing to deploy those via a config management 19:34:54 system. 19:35:25 okay, that i find compelling 19:35:32 corvus: aiui you can do that with puppet now, though that might require first getting to puppet 5? 19:35:45 by 19:35:51 corvus: they literally implemented ansible remote execution in ruby for puppet resources 19:36:00 er, by "that" i mean the ability to not need to install a client/agent on all our various systems 19:36:13 so does actually solve the consistency issue 19:36:15 clarkb: it's a good model 19:36:29 corvus: yup, I think ansible sort of proved that point to puppet :) 19:36:36 as i mentioned in my comments, just because the containers are there, still most of the configuration is/can be in puppet? i mean it's not a thrilling idea to translate huge blobs of erb config templates into jinja just because 19:36:37 the containers idea further minimizes the ansible we will end up with. 19:37:05 ianw: the containers plan ends up with puppet going completely away 19:37:13 ianw: and using ansible to template out config files 19:37:42 ends up with ... but it's not like we need to have a flag day switch? 19:37:44 yah, as I understand it, we'd write the container as Dockerfile, then use ansible to template and bind-mount a config into it 19:37:45 (I agre that translating erb to jinja sounds like unfun) 19:37:47 and to make determinations on what package renames happen between distro versions? 19:37:51 ianw: right. we don' tneed to flag day 19:38:03 mordred, ianw: one server at a time, i'd reckon 19:38:07 ++ 19:38:12 ya I think any transition to using more ansible would invole puppet_run_all running less "all" 19:38:21 and things being converted being more ansibly 19:38:27 eventuall all will be an empty set 19:38:30 (i... actually think translating erb to jinja2 sounds fun) 19:38:49 * mordred assigns corvus the erb to jinja2 tasks 19:38:53 but i can at least agree it's busy work, and it is a cost that we should allocate to the ansible options :) 19:39:35 I also think in that scenario we'd likely want to continue to keep pupept on enough life support that it is viable until we are off it. Which likely means puppet 4 is still something we'd do? maybe just with less rigorous testing of each application deployment? 19:39:38 the part I didn't fully understand, are we moving away from puppetmaster.o.o to say zuul post pipelines? Or new bastion host? 19:40:33 pabelanger: I think beacuse puppet remains until we are off of it the initial transition would require a bastion host remains 19:40:48 then a future spec could have us switch to zuul directly driving the deployment 19:40:52 pabelanger: I was thinking we keep puppetmaster for now - but we could certainly start triggering some playbooks on it as a static host in a post pipeline 19:40:54 (basically thats out of scope I think) 19:40:58 clarkb: ++ 19:41:01 okay, keep in mind, we do need to rebuild that server (since it is trusty) 19:41:28 i think moving to puppet 4 is little enough work that we can keep going with it without rejecting future plans 19:41:28 (but maybe just inplace upgrade) 19:41:59 pabelanger: yeah, maybe we just punt for 8 months and if it's still important, do an in-place upgrade 19:42:14 (we have a year of trusty remaining) 19:42:20 wfm 19:42:33 cmurphy: yah - and puppet needs to stick around for a while anyway for transition purposes 19:42:40 cmurphy: mordred exactly 19:43:31 there's a little handwavey in the container spec about hiera/secrets and inventory that I think we should likely do some POCs of options 19:44:28 so it's feeling like we have a little of all three specs we could consider in tandem? 19:44:49 cmurphy: ++ i still see having puppet4 in the mix as useful over the medium term 19:44:59 continue with puppet 4 uplift, get ansiblification working, possibly tack on containerization? 19:45:23 fungi: ya I get the sense what we'll acutally end up with is life support puppet4 short/medium term, transition to ansible base + container application "packaging" longer term, eventually having zuul do deployments (but this last bit should be its own spec) 19:45:30 fungi: yah, that order seems right. We'll need ansible in place before containers 19:45:37 fungi: yes, though i'd like us to disambiguate that to remove the 'possibly' :) 19:45:39 clarkb: ++ 19:45:43 having typed that does anyone here think that is a terrible idea just as their initial reaction? 19:45:54 sure, having a roadmap is good 19:46:09 nope. I think it's good ... a few pending details notwithstanding 19:46:28 yah, I'm willing to try 19:46:33 i'm not opposed to just sticking with puppet (i know it, even if it's changing), though i need to get more familiar with ansible/jinja anyway given zuul's use of it... 19:46:56 should we make a new spec which lays out all 3 things in a single document, or revise the specs to build on each other? 19:47:15 corvus: I kinda think one doc will be best for people's understadning if not directly invovled day to day 19:47:35 maybe as a next step I should through my summary out to the infra lst today, and if no one objects we can start on making that the future spec? 19:47:36 * mordred can take a stab at that ... programming is harder than text writing right now 19:47:57 I guess nothing prevents starting to work on that now either if mordred wants to do word typing instead of programming typing 19:47:58 reaching for the special keys is the thing that blows the most 19:48:06 yeah, i can git behind the short-term puppet 4 and medium-term migrate to ansible (if that's what the others who have time to spend on this are interested in doing), but i'm not yet sold on the longer-term containers piece (no offense, mordred) 19:48:17 er, get behind 19:48:21 * fungi sighs at finger memory 19:48:30 fungi: i do that all the time 19:48:48 +1 to migration idea 19:48:56 fungi: none taken! having been skeptical myself for a long time I both appreciate and likely agree with at least some if not most of your concerns 19:49:01 ok sounds like we have a plan to make a plan :) 19:49:05 woot 19:49:06 I'll send that out after lunch then 19:49:13 yay 19:49:19 before we run out of time there is a related topic from pabelanger we should at least touch on 19:49:27 i liked planning the plan to make a plan 19:49:31 #topic General Topics 19:49:42 pabelanger brings up a second round of xenial upgrades 19:49:59 given the above soft decision do we think we should continue to push towards xenial? 19:50:24 smarcet was asking just today about upgrading openstackid-dev and openstackid.org to xenial so that openstackid can take advantage of newer laravel which needs newer php 19:50:27 fungi: let me know what i can do to convince you -- i'm keen on it not only because it will reduce our deploy-in-config-management footprint, but i think it's coming at an opportune time when we're also at risk of fallout from pip10 and the like, where we might have to be looking at changing our deployment-from-source strategy anyway 19:50:32 [sorry for the topic leak] 19:50:40 yah, we have 1 year left on trusty and some long lived servers still to do. At this point, I think it is mostly about scheduling outages 19:50:49 my initial thought is since short/medium term we'd be puppet 4ing anyways (likely) that getting to xenial helps us avoid having another factor in the migration process. Basically get it out of the way then migrate applications based on the application not hosting platform 19:51:18 [also, wow, turns out that was actually still relevant to this topic] 19:51:25 corvus: :) 19:51:45 also getting to systemd across the baord just makes toehr stuff easier 19:51:56 not because its systemd but because its consistent 19:51:59 yeah, i wouldn't want the fact that we decide to do do something different in general which we're not quite ready to start implementing yet hold back server upgrades we need 19:52:02 (don't panic systemd haters :) ) 19:52:10 clarkb: agree 19:52:31 i imagine we don't want to touch openstackid for the next 3 weeks anyway? 19:52:38 corvus: ++ 19:52:51 corvus: that would be a great question for smarcet &co 19:52:55 yah, https://ethercalc.openstack.org/osiuhjzjq336 is the propose week for sprints 19:52:59 which is after summit 19:53:13 r-11, r-10, r-09 19:53:16 i sure hope they're not wanting to switch platforms for it right before the summit, right 19:53:25 #link https://ethercalc.openstack.org/osiuhjzjq336 help select a week for a xenial upgrade sprint 19:53:39 and https://etherpad.openstack.org/p/infra-sprint-xenial-upgrades-part2 has a list of servers we still need to migrate 19:53:53 #link https://etherpad.openstack.org/p/infra-sprint-xenial-upgrades-part2 servers that still need to migrate to xenial 19:54:40 if interesting in helping out please indicate your availability 19:55:00 thanks! 19:55:03 I do think it would be wortwhile and helpful 19:55:17 it was also a great learning experience for new infra-root 19:55:27 indeed 19:55:31 ++ 19:55:40 alright we have 5 minutes left I'll open the floor to anything else 19:55:45 #topic Open Discussion 19:56:19 I heard a rumor of zuul stickers so keep space for those :P 19:56:30 I also still have more infra ant stickers if you want them 19:57:04 clarkb: sounds like team dinner is monday night? 19:57:28 oh right I proposed monday night at a brewpub near to the convention center. I think we can swing that without going through too much official paper pushing 19:57:46 (I don't want to pay a deposit on a room then sort out paying with a single card) 19:57:54 clarkb, good call 19:58:01 sounds good to me 19:58:23 the proposed location seems quite large, has pool tables and brew their own beer. Food wise they have vegetarian options and gluten free looks like but not vegan or at least nothign screams vegan 19:58:26 great 19:59:30 if for some reason you don't like this option I'm totally open to alternatives I'll just be asking that you do some of the leg work too :) 19:59:32 let me know 19:59:53 i'm up for losing a few rounds of pool 20:00:06 and we are at time. Thanks everyone 20:00:08 #endmeeting