19:01:46 #startmeeting infra 19:01:46 Meeting started Tue Aug 14 19:01:46 2018 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:47 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:49 The meeting name has been set to 'infra' 19:02:04 #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting#Agenda_for_next_meeting 19:02:15 #topic Announcements 19:02:30 I'm going to be on vacation next week. 19:02:52 fungi: we don't have a stein release key yet do we? 19:03:09 nope, but i'm happy to generate one unless someone else wants to take a try at it 19:03:24 the process documentation should be very easy to follow 19:03:41 and i'm happy to provide support too 19:03:44 fungi: when have you tried to have those generated by relative to the cycle? 19:04:13 ideally before release day, but any time in the prior cycle is fine 19:04:24 does it happen on puppetmaster or bridge now? 19:04:29 as long as the expiration date is set for a sufficient duration 19:04:41 it should be doable on bridge 19:04:59 https://releases.openstack.org/rocky/schedule.html says Ideally it would be done next week 19:05:08 or between now and then 19:05:12 really it just needs to happen on a secure system to which we have access so we can all double-check the fingerprint of the symmetrically-encrypted master key on disk 19:05:34 anyway it occurred to me that we normally announce the need to go sign the key in the announcement portion of this meeting and realized we hadn't done that here so brought it up 19:05:58 makes sense--thanks for the reminder! 19:06:11 we also put it into the release schedule so we get a reminder from the release team 19:06:33 * diablo_rojo sneaks in a little late 19:06:38 fungi: ah ok so they should reach out as well? I probably won't commit to it as I'll likely be afk when they want it 19:06:58 looks like diablo_rojo just volunteered! ;) 19:07:18 Anything else to announce? 19:07:21 (kidding, it needs to be an infra-root volunteer specifically) 19:07:25 Umm..uh.. 19:07:28 whew 19:07:48 but yeah, i'll plan to do it later this week. if anyone else wants to instead, let me know 19:08:18 clarkb: I'm also going to be out most ofnext week 19:08:42 clarkb: I leave wednesday for vietnam - so will be traveling for $hours - then the week and a half after that I'll be on vacation 19:09:12 fun! I'm headed to the oregon coast with family while parents are here. Going to try our luck at crabbing 19:09:13 sounds like we may be short staffed enough that we should all just take a break :) 19:09:58 clarkb: I'll wave across the pacific 19:09:59 #topic Actions from last meeting 19:10:12 #link http://eavesdrop.openstack.org/meetings/infra/2018/infra.2018-08-07-19.01.txt Minutes from last meeting 19:10:24 I don't see any actions 19:10:37 #topic Specs approval 19:11:04 ianw: Should we be looking at the letsencrypt spec at this point? You mentioned in IRC earlier that you didn't feel like you had other things to change? 19:11:14 maybe reviews this week and look at putting up for approvla next week? 19:11:48 yes please, i've incorporated all the current review comments at this point, so ready for review 19:12:03 #link https://review.openstack.org/#/c/587283/ Letsencrypt spec 19:12:24 Please review and we can try to put it up for approval next week 19:12:31 i'm about 2/3 through reading the current version (i think). a slew of minor comments pending in gertty for it, but am generally in favor so far 19:12:40 fungi: thanks 19:12:56 #topic Priority Efforts 19:13:03 #topic Storyboard 19:14:14 Haven't seen anything major here, but projects continue to plan migrations 19:14:19 diablo_rojo: any updates? 19:14:25 I just saw on the ML puppet is interested 19:14:35 yup, that was encouraging 19:14:39 so I will help the new PTL and the project through stuff. 19:14:44 looking to migrate next cycle sounds like 19:14:48 We have a lot of open reviews that need some love. 19:14:49 https://review.openstack.org/#/q/project:openstack-infra/storyboard-webclient+status:open 19:15:04 fungi, next cycle or just whenever they feel comfortable 19:15:26 the puppet-openstack ptl appears to have taken the initiative to deploy a copy of storyboard locally and perform an import himself even 19:15:38 #link https://review.openstack.org/#/q/project:openstack-infra/storyboard-webclient+status:open Storyboard reviews 19:15:40 Yeah that was awesome. 19:16:02 Thanks clarkb :) I can never remember to do that unless I am leading the meeting. 19:16:22 * fungi notes diablo_rojo has volunteered to chair future infra team meetings ;) 19:16:32 We do need a volunteer for next week :P 19:16:32 *sigh* 19:17:11 * diablo_rojo averts eyes like she did in highschool to avoid having to answer a question she didn't know the answer to 19:17:23 seems like that's it for storyboard updates this week 19:17:32 #topic Config Managment Updates 19:17:32 Just pleading for reviews 19:17:52 o/ 19:17:58 Things are moving here, mordred has bridge.o.o up and running and cmurphy's parade of puppet 4 changes are moving along 19:18:01 I thought we were just about ready to do the puppetmaster to bridge cutover ... but then realized there were a couple of special-cases with exim config, and the base playbook has exim stuff in it 19:18:16 topic:puppet-4 and topic:update-cfg-mgmt 19:18:36 so hopefully we'll get the special cases handled - once we do - I'll want to do the cutover - at which point we should be able to just do small patches replacing a puppet bit with an ansible bit 19:18:49 mordred: i guess we should all strive to help you achieve that before you go on vacation? :) 19:18:51 mostly just have new tests and then some handpicked nodes to turn the future parser on 19:19:04 corvus: yes. it should be TOTALLY achievable this week 19:19:09 also this xenial fix for askbot https://review.openstack.org/585196 19:19:11 (i'd love to avoid having the puppetmaster split across a long period). cool. 19:19:24 it's honestly just those exim patches and making sure we like them 19:19:51 corvus: so maybe let's sync up on those after the meeting? 19:19:52 and for those following along the new server will continue to run puppet against most things, we just don't need puppet on the control node to do that 19:20:04 then we'll migrate puppetized things one or a handful at a time 19:20:21 (while concurrently making them use puppet that isn't ancient (because we expect the off puppet migration to take time) 19:20:30 yes 19:20:31 cmurphy: looks like that falls back to being stuck on https://review.openstack.org/#/c/559178/ (minor fork of puppet-solr?) 19:20:49 additionally, this initial migration also handles a chunk of the things in openstack_project::server in ansible 19:21:00 so users, initial packages, etc - are all migrated into ansible already 19:21:27 so each "run" of puppet will run the base.yaml playbook first, then will run remote_puppet* 19:21:31 mordred: have you run it against bridge.o.o itself? i noticed my key was wrong there 19:21:47 ianw: i think askbot in general is stuck on that patch but the patch i linked shouldn't be stuck on anything 19:21:56 ianw: yes - I think we should have landed the patch to fix your key - but I haven't re-run it since that landed 19:22:56 cmurphy: i think it's on top of the current outstanding stack (585196)? 19:22:57 Mostly just need reviews and coordinating hte larger action of deleting puppetmaster.o.o then? 19:23:37 ianw: oh yes you're right 19:23:53 i was looking at owner:self derp 19:23:55 clarkb: yah 19:24:29 mordred: so does puppet currently redo some of the stuff that the new base.yaml does? 19:24:49 ianw: it would if we ran the stack as it is landed right now 19:24:56 ianw: by the end of the current proposed stack, no 19:25:21 ianw: although - with the exception of exim config on 4 hosts - the puppet and ansible are currently producing the same results 19:25:32 so the puppet run is a no-op on the things that are the same 19:25:41 I did run the base.yaml playbook against review-dev.openstack.org 19:25:57 ianw: your ssh key should be fixed now 19:26:23 thanks :) 19:26:33 one other thing, just so people are aware - bridge is running ansible 2.6 using python3 - and is also using the new openstack inventory _plugin_ instead of the old openstack inventory script 19:26:43 plus some other new fancy ansible inventory plugins 19:26:55 so if you haven't seen that stuff yet, it's worth looking at it - hopefully it's understandable 19:26:56 mordred: any concern with doing that and wanting to use "random" roles for services? 19:27:14 mordred: maybe just so it's in the minutes, and i think related -- what's the thinking on moving kerberos/afs roles into system-config for -> https://review.openstack.org/#/c/590591/ 19:27:24 (and i presume similar roles) 19:27:24 we add python3 support to them I guess if that becoes a problem 19:27:31 clarkb: yah 19:27:38 clarkb: we're gonna have to add it at some point :) 19:27:42 http://git.openstack.org/cgit/openstack-infra/system-config/tree/playbooks/roles/install-ansible/files/groups.yaml 19:27:58 is the replacement for how we add hosts into ansible groups from the hacked-together groups.txt thing 19:28:50 ianw: I think we should move them into system-config - I also think roles wants to move up into the root of system-config - but we need to do a dance with role paths for that, and I'm worried about the puppetmaster/bridge split and complexity 19:29:33 so I think we should shift roles after we cutover - and add your afs/krb roles as the first thing 19:30:12 yeah, i'm kind of missing the why they're better there bit :) 19:30:33 ianw: rather than being in their own repos you mean? 19:30:54 or zuul-jobs? won't that get dragged in? 19:31:10 no - we're not running these in zuul - we run ansible on bridge in its own context 19:31:28 mordred: well we also use them in zuul jobs (or could use them) for wheel build jobs to set up afs 19:31:32 so if we used them from zuul-jobs we'd wind up adding zuul-jobs roles path to our production role path on system-config - which seems backwards to me 19:31:44 but yes - if we put them in system-config, we can add system-config to the roles path for the jobs 19:32:01 so that way it's "use the production roles in testing" rather than "use the testing roles in production" 19:32:28 mordred: ya I like that also makes it clear if you change this then productio nchanges 19:32:32 yah 19:32:34 which probably won't be so clear if in zuul-jobs 19:33:09 eventually having an ansible-role-openafs-client repo would probably be aces - but for now keeping them in system-config while we sort it out seems like less churn 19:33:42 fwiw I kind of like the aggregate roles situation in eg zuul-jobs or ozj better than puppet-.* everything we ended up with 19:33:46 ok > zuul-jobs roles path to our production role path on system-config 19:33:59 i guess that was my misunderstanding, i thought that would be fine 19:34:17 maybe even kind of the idea of zuul-jobs as generic collection of things 19:34:44 well, i see it as a collection of jobs that are generally useful 19:35:20 so maybe an infra roles stdlib repo would make sense rather than a repo per role (but not sure how happy ansible is with that construct generally) 19:35:34 i don't know how generally applicable our afs mirror wheel building jobs are. maybe? 19:35:55 corvus: this is just generic "setup as a kerberos client" and "install afs client" roles 19:35:58 and I think system-config is an easy place to start so no objections to use it 19:36:04 clarkb: ansible is growing the ability for grok a repo of roles - but its not there yet 19:36:25 there is a new tool called "mazer" that is being developed to do fancier things with role management 19:36:44 ianw: right, but with no job to use them defined there, it's questionable whether that's the right place. 19:36:46 so if we did it, it would be fine, but it would involve us putting the git repo on disk and adding it to roles path 19:37:11 (we have *some* roles in zuul-jobs without jobs using them, but generally those are ones which are designed to be used in a config repo) 19:37:24 clarkb: which, incidentally, is what https://review.openstack.org/#/c/590752/3 does 19:38:16 anything else before we continue on our way down the agenda? 19:38:25 ok, in terms of testing ... i have https://review.openstack.org/#/c/589335/ ... i don't think that pulls in system-config roles, i guess that will be ok to do in o-z-j? 19:38:48 ianw: yes, I believe so 19:39:14 ok, sorry, if i have more questions i'll follow up, but thanks 19:39:40 ianw: awesome - and sorry, I keep meaning to try to overlap days with you more to chat about it 19:40:37 #topic General Topics 19:41:13 Really quickly. its been pointed out that we need to continue upgrading off of trusty because it has abou 8 months of life left 19:41:47 I'm trying to push on ethercalc and etherpad as they had a bit of work done around their puppet to support it. I think we are to the point where a new ethercalc can be deployed so will do that today after the meeting 19:41:57 #link https://review.openstack.org/#/c/591017/ Support for digitized etherpad servers 19:42:12 that change is necesary for etherpad. mordred I think it runs into group change stuff that we need to coordinate on 19:42:24 mordred: ^ maybe we wait for bridge cutover before booting new etherpad* nodes 19:42:33 yah - that'll probably be easiest 19:42:42 ok I'll WIP it then 19:42:54 that way we won't be trying to juggle both boxes - but we'll also push hard on the cutover 19:43:25 There are a number of services that need to be updated to newer distro. I think next month is a great time to work on that as the release will be behind us 19:43:44 When I get back from vacation I'll try to put together some planning for a sprint as I think those have worked well in the past 19:43:54 (likely after ptg given that we'll be juggling that soon enough) 19:44:40 ianw: Want to talk about third party ci spec? 19:44:47 #link https://review.openstack.org/563849 Direction setting for 3rd Party CI 19:44:54 ++ it's much easier if we collaborate, as you get stuck with a bunch of puppet patches on repos that nobody really watches 19:45:04 ianw: indeed 19:45:05 and then get distracted :) at least i do :) 19:45:34 re 3rd party; just a quick note -- prior reviews out of all the options windmill was pulling ahead 19:45:57 i think as stage 2, we should really define what we mean by 3rd party ci -- i.e. what you start with, and where you get to 19:46:06 so i've written that out in the spec 19:46:16 so i'd like reviews on that 19:46:47 i think, once we have those two bits, we have a complete thing we can start actually working on, with criteria for it being "done" 19:46:54 ++ 19:47:22 that's all :) 19:48:30 #topic Open Discussion 19:48:47 It occurs to me I didn't explicitly ask for a volunteer to run for the meeting next weke but we will need one 19:49:08 Let me knowif interested 19:49:27 I'd like to push on our base job rework if possible, http://lists.openstack.org/pipermail/openstack-infra/2018-August/006032.html for more info 19:49:39 I likey need to rebase conflicts 19:50:50 pabelanger: why is base-ozj in project-config? 19:51:17 pabelanger: i did look, some of the reviews i got a bit scared because they were moving big bits. a bit more description will help in the changes i think 19:51:20 might just be me but that seems confusing organizationally 19:51:25 ianw: ++ 19:51:39 okay, will update 19:51:48 but base-ozj is needed for integeation testing of zuul-jobs 19:52:10 which is currently base-minimal 19:52:13 pabelanger: mostly its weird that base-ozj isn't in ozj 19:52:25 clarkb: open to suggestions for naming 19:52:36 jobs in ozj will use it 19:53:38 mmm, so when i said i was in favor of the le spec so far, i hadn't gotten to the actual implementation proposal yet. worried it's going to be overengineered, fragile and less secure than the current situation we have (no offense, ianw, i really thought the earlier discussions about "proxying" acme renewals and having a site proxy to bootstrap new systems was meant to be parody) 19:54:12 pabelanger: can followup after the meeting 19:54:30 pabelanger: but I think it would be helpful if we could have ascii art or similar of where we are today and where we want to be 19:54:48 pabelanger: at least for me understanding these organizational changes are aided by diagrams 19:55:19 okay 19:57:26 I'll let the last 3 minutes run out if there is anything else, but that is all I had 19:57:31 thank you everyone 20:00:14 #endmeeting