19:01:38 #startmeeting infra 19:01:39 Meeting started Tue Mar 24 19:01:38 2020 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:40 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:42 The meeting name has been set to 'infra' 19:01:46 Anyone else here for the meeting? 19:01:52 #link http://lists.openstack.org/pipermail/openstack-infra/2020-March/006612.html Our Agenda 19:01:57 o/ 19:02:10 o/ 19:02:11 #topic Announcements 19:02:14 o/ 19:02:21 #link http://lists.openstack.org/pipermail/foundation/2020-March/002854.html PTG going virtual 19:02:38 I'm sure most or all of us have seen this news already, but wanted to call it out if you hadn't 19:02:49 thanks for the heads up! 19:03:09 The big impact for us is that we'll likely try to support the virtual event though tools 19:03:33 o/ 19:03:39 corvus has begun work on a jitsi deployment incorporating our etherpad which looks promising but I'm sure tehre will be other things as well 19:03:58 clarkb: any idea what other things? 19:04:11 corvus: right now? no :) 19:04:48 though collecting artifacts might be something that becomes more important 19:05:12 especially if conversations span timezones and we need to forward more concrete thoughts/items 19:05:22 like scrolls and treasure chests and magical amulets? 19:05:44 dibs on the +1 sword 19:05:54 * mordred weilds the +2 mace of bludgeoning 19:06:05 i thought that was what the etherpad was for... 19:06:30 that probably is what the etherpad is for 19:06:48 just trying to figure out if i'm barking up the wrong tree 19:06:55 corvus: I don't think you are 19:06:58 I think your tree is great! 19:07:00 i'm guessing "other things" might be improvements to ptgbot 19:07:11 scheduling tools? dunno 19:07:15 but there are a lot of moving parts and we've been involved in the technology side a bit in the past 19:07:17 I'm guessing clarkb is just trying to pessimistically anticipate a general need for "other things" 19:07:20 (so I expect more moving parts) 19:07:31 yeah, lots of other opportunities 19:07:38 * mordred thinks meatpad is amazing 19:07:49 heh 19:07:54 * mordred would like to propose that we cname it so that both spellings are acceptable 19:08:09 veggiepad is leaner and better for you 19:08:17 ok - maybe 2 cnames :) 19:08:47 we should continue and then we can talk about meetpad in more detail 19:08:50 #topic Actions from last meeting 19:09:03 ianw had an action to cleanup files02 and old static resources 19:09:09 I think I saw a statusbot log action for that? 19:09:15 #link http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-03-17-19.01.txt minutes from last meeting 19:09:26 yes, the servers are gone. there was some fun when i tried to look at the volumes :) 19:09:58 i think that is sorted now, need to catch up, but i'll leave the old static volume and have a note to clear it out in a few months, just in case 19:10:12 sounds good, thanks 19:10:23 ... so ... i think the transition is done. i can move it in specs to the done pile later 19:10:28 ++ 19:10:45 #topic Specs approval 19:10:51 * mordred hands a +2 chicken of awesome to ianw 19:10:56 really quickly before we talk meetpad is the xwiki spec 19:11:00 #link https://review.opendev.org/#/c/710057/ xwiki for wikis 19:11:28 fungi: ttx ^ I think we probably can put this up for approval this week given the discussion we had on the topic in last weeks meeting 19:11:44 I think we are largely on the same page that experimentation around the wiki problem is worthwhile 19:11:56 is there any disagreement with that? 19:12:16 i'm fine with putting it up for a vote but also don't expect it's too urgent 19:12:39 good either way 19:13:08 ya I'm not sure we'll get too much additional feedback if we drag it out, and the topic is fresh in minds now given the discussion last week 19:14:02 ok I'll look to landing that EOD thursday local time. 19:14:06 #link https://review.opendev.org/714189 meetpad 19:14:44 For anyone not familiar the meetpad idea is to combine jitsi meet and etherpad into a remote collaboration tool for events like a virtual PTG (though sure to have uses elsewhere too) 19:14:52 corvus: ^ probably ahs a lot more to say about it 19:15:50 i think that about covers it 19:16:12 i'm optimistic that having a tool focused on the design session workflow will be a passable substitute 19:16:22 i've tested it small-scale, but not large scale 19:16:46 since this is a low-commitment standalone service, we might want to think about going aheand and spinning it up to see how it goes 19:16:52 ++ 19:16:53 ya I'm hoping we can use it at medium scale a bit before the main event 19:16:58 we can always shut it down later; it has no persistent storage or anything 19:17:00 i can imagine ways in which it could even make some remote participation in otherwise in-person events more viable 19:17:02 especially since you've got the ansbiels already 19:17:13 and yes, the complete implementation is ready to go 19:17:26 On the deployment side my one concern is that the shape of the deployment is very different than our existing docker stuff 19:17:37 well, not *very* different 19:17:44 i managed to get it using host networking 19:17:58 ya, but instead of process per container its lots of processes per container (including cron) 19:18:11 it will just look different from the management side which is probably ok if people are aware of it 19:18:36 fwiw, i'm not really aware of it :) 19:18:45 i think the cron stuff is only for certbot? which we also won't use 19:18:52 because i hooked it into our existing acme stuff 19:18:54 ah ok so we can trim out the bits we don't need 19:19:04 yeah - I thnk the images have more stuff in them - but we're not necessarily using those images in that way 19:19:05 well, i wouldn't trim it out 19:19:15 which is fine by me 19:19:45 i'd like to use upstream images if possible 19:20:01 (right now, we need a customized local image because i have a patch outstanding) 19:20:03 yeah. we're really only building these atm because we have to carry a patch right? 19:20:05 yeah 19:20:06 ya I'm not opposed, more I wanted to call out how they seemed to be different than how we've tried to construct things 19:20:19 if we run the upstream images it will be even more clear 19:20:30 well, i'm not sure they're that different 19:20:58 they're using our acme letsencrypt stuff, and there are 4 containers for the 4 processes 19:21:23 ok 19:21:27 I've misread the cron stuff then 19:21:38 it's there, we just won't use it so we can ignore it 19:22:08 i think it fits in pretty well with our model; our docker-compose file is really close to the upstream one 19:22:08 so the images have more in them that we'll be leveraging is the main concern? 19:22:18 so i think it's worked out pretty nice actually 19:22:27 er, more in then THAN we'll be leveraging i meant 19:22:33 * fungi can't type 19:22:34 it's close enough to our model that i was able to complete the entire implementation stack in less than a day 19:22:53 fungi: on my initial read it appeared that we would be using cron to run certbot 19:23:13 but I guess thats all there to avoid divergence from the upstream images and not actually to be consumed in our setup 19:23:13 and we're not, got it 19:23:35 we may yet be 19:23:37 but it won't do anything 19:24:11 if we really want to pull it out, we can, but i don't think it's a big deal 19:24:37 if the plan is to use upstream images anyway may as well keep it (since we'll get it back again once we move to their images 19:24:49 i'd like to do that, yes 19:24:52 ++ 19:25:19 seems fine to me 19:25:27 so what's the timing here? 19:25:35 please review the spec if you haven't already. Then plan to land it next week? Though at the rate implementation is going maybe we should try to get people to review it this week and land the spec sooner than later 19:26:07 to be clear, implementation is done :) 19:26:15 the next thing is to create the server after the implementation patches land 19:26:36 i'd be cool landing it this week 19:26:39 * mordred too 19:26:58 some of us have already reviewed and voted in favor of the spec 19:27:06 of active roots its missing ianw and myself (and I've actually read it just not finalized my vote (which is positive I'm excited) 19:27:07 +1, looking forward to tsting this 19:27:09 doesn't seem like it needs adjustments 19:27:18 ianw: ^ you ok with landing that this week? 19:27:39 yes 19:27:41 also, it's a timely effort, so the less we unnecessarily delay progress on it, the better 19:27:52 cool I'll give people until thursday to review it then plan to approve it 19:28:04 should we ping anyone from other communities about this? 19:28:16 or stand up the server and then do that? 19:28:20 corvus: I half expect it would be easier to explain once the service is up 19:28:24 me too 19:28:27 i think asking them to help test it out makes more sense 19:28:28 "go look at this" 19:28:43 yeah, i tend to think so too. there's so little cost in standing it up 19:29:10 i expect we'll want to conduct at least one "load test" where we try to get lots of folks using it at the same time 19:29:32 so we can work out sizing before it gets slammed and falls over at ptg time 19:29:45 as a time check we are about halfway through our allotted time and have more to discuss. Do you think we've gotten the important bits for this spec sorted (until thursday at least)? 19:29:47 ya 19:29:57 clarkb: yep, thanks! 19:30:05 #topic Priority Efforts 19:30:12 #topic Update Config Management 19:30:30 mordred: (or fungi or corvus ?) can you fill us in on the gerrit on containers status? 19:30:50 well ... 19:30:57 we ALMOST updated on friday during renames 19:31:07 but then there was some container content issues - mostly the fact that jeepyb was missing 19:31:08 the maintenance went really well, right up until.. yeah. so close! 19:31:25 we've been unwinding an fixing some issues in the job 19:31:32 hook script entrypoints were missing 19:31:42 was the main setback 19:31:49 as soon as I get another set of images built with the latest jeepyb changes (so close last time) - we'll be in a good position to do another quick restart 19:32:00 that should be no longer than a stop/start 19:32:12 in the process - we learned a gerrit thing that's worth sharing ... 19:32:19 this has also spurred greater testing for jeepyb, so it's not all bad 19:32:37 gerrit init does three things: it downloads the database driver, it expands the plugins from the war, and it runs db migrations 19:32:55 we updated the image build script to do the first two - as they make no sense to be runtime activities in a container image 19:33:11 so now seriously we only have to run init for db migrations when we upgrade 19:33:18 so - just a note for everyone to be aware of 19:34:17 and until we get restarted on that new image project creation is not expected to work 19:34:21 (so we are holding new projects for now) 19:34:39 mordred: and the testing will incorporate testing of manage-projects? 19:34:59 it just occured to me that we may want to bind mount in the jeepyb cache for that? somethign to sort out side of the meeting 19:35:09 we already do! 19:35:16 excellent 19:35:18 although that's worth a mention 19:35:31 new project creation and anything else on the gerrit server is blocked, right (this includes gerritbot config changes) 19:35:34 I added helper scripts for manage-projects and track-upstream that run docker run with the appropriate bind-mounts 19:35:38 fungi: yah 19:35:51 do we need to hold new projects right now? 19:35:57 so if you go to review-dev and just run manage-projects - it should work 19:36:07 it's still in emergency, right? 19:36:09 yah - I think AJaeger has been holding them 19:36:10 that's right 19:36:12 so i guess nothing will run automatically 19:36:19 ok. just wanted to double check my understanding on that 19:36:20 corvus: seems like it's safe to keep approving them, they just won't take effect 19:36:36 yeah, though that's potentiall confusing 19:37:15 right 19:37:41 * fungi was confused that the infra-manual rename hadn't taken effect for gerritbot, for example) 19:38:18 ok sounds like that may be it. Lets move on as we have even more to talk about :) 19:38:21 #topic OpenDev 19:38:49 We've updated the way Gitea restarts happen. This should allow gerrit's replication plugin to properly detect the remote service is offline which will result in retries 19:39:17 https://review.opendev.org/714691 is small change that will run this in production (we do test it in checka nd gate too) just to be sure it doesn't cause problems unexpectedly later 19:39:43 Also the OpenStack TC governance change to excise OpenDev has landed (mentioend this last week) and we've landed our documentation updates 19:40:10 woot 19:40:10 Two big todos that come out of that is I think we need to formally elect a project lead as well as start forming the Advisory Board 19:40:28 clarkb: my understanding is that you are now lifetime project lead, right? 19:40:36 mordred: I was going to nominate you 19:40:38 :P 19:40:45 hot potato 19:41:04 not it 19:41:12 more seriously though I was going to start noodling on how the process for both of those. We've got lots of hsitory to work with can liekly come up with something sane 19:41:26 if you've got ideas or thoughts let me know or feel free to start a mailing list thead 19:41:40 and that takes us to the last set of opendev related topics I had on the agenda (which I'm going to reorder sligthly) 19:41:56 location of future meetings as well as need for opendev-discuss mailing list 19:42:24 initially I had thought we could do our weekly meetings in #opendev but frickler has indicated that the overlap in communications would cause problems 19:43:01 I think that concern is valid, so maybe we should create an #opendev-auxillary (I can't spell) where we can have meetings as well as use it for quiet space like the old infra-incident? 19:43:01 yes, I've experienced that e.g. in the qa team and I would not like it 19:43:31 then anyone looking to catch up can check that channel for any super important updates and if empty skim #opendev at their leisure 19:43:47 ++ to separate 19:43:55 a related question is whether we need something similar to or replacing #openstack-infra-incident 19:43:56 corvus: any concerns with that from a freenode perspective? I suppose its another channel for bots but we are trimming channels elsewhere for the bots 19:44:11 fungi: ya I kinda expect we can simply use a single side channel for both 19:44:23 because if there is an important incident we won't be meeting anyway 19:44:44 wfm 19:44:48 (I'm trying to be friendly to bots and not add too many extra channels) 19:44:48 or we make an incident-themed meeting 19:44:52 frickler: ya or that 19:45:15 so bikeshed some name which covers meetings and incidents 19:45:16 i think we own the #opendev namespace 19:45:20 then the channel could in fact be named #opendev-meeting, I'd think that would make it's purpose more easily visible 19:45:27 #opendev-bikeshed isn't taken ;) 19:45:41 frickler: thats a good point. I'm happy to go with that 19:46:31 anyone want to take an action to push up changes to make that official? 19:46:38 we can bikeshed further in review if necessary 19:46:45 i can do that unless someone else really wants it 19:47:09 #action fungi Add new opendev meeting (and incident) channel 19:47:26 the other related piece of this is mailing lists 19:47:47 I thinkwe can just go ahead aand create an opendev-discuss list on lists.opendev.org and start transitioning more of our email communication there 19:47:58 Then we can possibly fold openstack-infra into openstack-discuss as well 19:48:00 service-discuss@ would make sense given we already have service-announce@ 19:48:10 fungi: ++ 19:48:27 i can do the change for that as well 19:48:44 #action fungi Add service-discuss@lists.opendev.org mailing list 19:48:47 thank you! 19:48:54 ++ 19:49:19 and again, names can be debated in the change 19:49:49 yup 19:50:05 #topic General Topics 19:50:15 The last agenda item was a catchup on mediwiki updates/cahnges 19:50:26 I don't think there has been any movement on the puppet retrofit 19:50:51 but did want to have this as an opportunity to talk about the possibility of dockerizing the deployment 19:51:07 mnaser: corvus ^ I think you've both been busy so don't expect there is anything new to discuss, but if there is feel free to share 19:51:26 i don't have anything to report there 19:52:07 if i find time i'll still try to make progress on the puppet module because it's the closest thing we have to documentation for the target configuration 19:52:46 but mostly i suspect it needs troubleshooting done with extensions and possibly extension conflicts 19:54:04 ++ 19:54:17 and that takes us to the free for all portion of the meeting 19:54:21 #topic Open Discussion 19:56:04 we lucked out and our monthly delivery of TP arrived (a few days late) from amazon. But now I'm distributing TP to family 19:56:27 we have plenty for ourselves 19:58:27 Also oreilly has apparently cancelled all in person events forever. If you were set up to speak at one of their events that info may be useful to you 19:58:31 people here tend to stockpile it to fling during the st patrick's day parade (which wound up cancelled). this produced an odd dynamic with people unexpectedly sitting on stockpiles of the stuff and stores already running low before the panic hit 19:59:11 (or attend I suppose) 19:59:43 And that is time. Thank you everyone! 19:59:46 #endmeeting