19:01:17 #startmeeting infra 19:01:18 Meeting started Tue Nov 17 19:01:17 2020 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:19 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:21 The meeting name has been set to 'infra' 19:01:25 #link http://lists.opendev.org/pipermail/service-discuss/2020-November/000135.html Our Agenda 19:02:01 #topic Announcements 19:02:09 Wallaby cycle signing key has been activated https://review.opendev.org/760364 19:02:13 Please sign if you haven't yet https://docs.opendev.org/opendev/system-config/latest/signing.html 19:02:21 * clarkb makes a note to sit down after lunch and get this done 19:02:33 i think i did 19:04:55 #topic Actions from last meeting 19:05:01 #link http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-11-10-19.01.txt minutes from last meeting 19:05:28 No actions recorded. Moving on 19:05:34 #topic Priority Efforts 19:05:41 #topic Update Config Management 19:05:52 ianw: looks like you've started investigating a docker deployed hound? 19:06:22 yes the project has had a bit of a second life and moved in github, breaking the puppet 19:06:43 it's going to be a pretty straight-forward container + apache, so i think it's best to just modernise it 19:06:53 what's still lacking for elasticsearch backend in gitea? 19:07:22 fungi: a release for 1.13.0 19:07:31 then testing to see how to deploy it and all that 19:07:42 oh, got it. for some reason i thought that already got released 19:08:03 they were down to like two issues remaining for the release on the issue tracker but it got a new rc and not a release 19:08:08 maybe that means it is close? 19:08:31 yeah, i think it's probably best to have this in our pocket so that can be done ad-hoc, given it will be a fair bit more deployment work 19:09:12 ya lot of unknowns (like sizing of elasticsearch deplioyments) 19:09:59 any other config management items to bring up? 19:11:17 #topic OpenDev 19:11:33 We are just a few days from the planned Gerrit upgrade to 3.2 19:11:45 #link http://lists.opendev.org/pipermail/service-announce/2020-October/000012.html 19:12:02 Need to land the non WIP changes at https://review.opendev.org/#/q/status:open+topic:gerrit-upgrade-prep 19:12:07 * yoctozepto is here for the meeting too 19:12:31 Looks like those have received recent reviews so after the meeting I think we can start landing them. fungi did you want to review those and maybe approve them if they look good? I will be around to help if things get sad 19:13:02 https://review.opendev.org/#/c/762712/ in particular is important because it conflicts with the WIP portion of my stack so I need to rebase once this one lands 19:13:21 #link https://etherpad.opendev.org/p/opendev-gerrit-3.2-upgrade-plan the plan is here. A few items still need a little work. Please take a look. 19:13:52 For the items that need work the first is deciding how we want to backup the 2.13 state as part of the upgrade. After some thought I think I prefer mounting a new volume on the side, then rsyncing into it 19:14:00 if you have better ideas please leave them on the etherpad 19:14:37 the alternative would probably be to umount the volume and then snapshot it with openstack api calls 19:14:59 fungi: yup, my concern with that is that we're relying on cinder to do all that successfully for us, which it has done previously if we prefer that appraoch 19:15:00 down-side to that is there are extra steps involved in turning that into a mountable replacement 19:15:54 The other big todo item was thinking about how to get config management into sync with the post upgrade state. I decided that squashing the 4 changes that reflect new state into one change and landing it before zuul starts is the best option. I have done that squashing and kept the old changes with distinct changeids to ease reviewing 19:16:04 i'm in favor of the rsync plan. i wonder if pre-syncing anything into it would speed up the final sync during the maintenance, or if all the repack/gc would still require replacing most of it 19:16:23 fungi: I think we backup prior to gc'ing so maybe we should run that early 19:16:59 I like that idea 19:17:14 re config management syncup I need to update the etherpad to reflect that I did the squash 19:17:28 If you can think of anything else to consider please reach out and update the etherpad too 19:18:52 One thing to note is I tried to put time stamps/estimates in the etherpad to give people a sense for how this process should progress over time 19:19:08 this info is based on what we learned doing the review-test upgrades and should be fairly accurate particularly for the gerrit upgrade steps 19:19:38 i agree an rsync copy seems the simplest way to keep things separate 19:19:59 can always flip it r/o mount to avoid any mistakes too 19:20:35 borg is running too :) 19:20:48 yup Ifigure we've always got borg/bup to fall back on if we need it 19:20:57 this is more of a convenience for quicker recovery if necessary 19:23:03 cool. I'll work on updating the etherpad to better reflect those choices today 19:23:25 then hopeflly the next cuple of days we can do some last minute checks as well as prep the "backup" volume 19:25:18 Anything else on this topic? 19:26:14 just a big thanks to you and fungi for working through it! 19:26:39 #topic General topics 19:26:45 #topic Bup and Borg Backups 19:27:07 This was just mentioned and it seems like keeping up to date on progress here is a good thing too 19:27:28 ianw: I know you had talked about starting the backups over now that you've trimmed them a bit. Has that happened or have we decided that isn't necessary? 19:27:53 yes, i did that, particularly for review and zuul 19:28:04 so everything has been running in production mode now for ~ a week 19:28:20 i think let's get through the gerrit upgrade, then we can audit the borg logs look good and retire bup 19:28:28 s/logs/backups 19:28:45 great, figuring out when to stop using bup was my next question. Doing audits after the upgrade sounds good to me 19:29:05 also we're backup up to two separte providers as a result of this as well? 19:29:11 that is a good improvement regardless of the tool in use 19:29:25 yep, vexxhost & rax.ord with borg 19:29:52 the vexxhost only has a 1tb volume, as we ran out of quota. we can re-assign some of the bup quota to that i imagine 19:30:16 but we can also prune now, which we couldn't before with bup 19:31:46 neat 19:31:54 Any thing else on bup/borg/backups? 19:32:07 nope 19:32:23 #topic General Topics 19:32:47 There were a number of other topics including, PTG followups, meetpad access in china, openstackid.org, and splitting up the puppet else job 19:33:02 I've been super head down on the gerrit stuff and not had time for any of ^ myself 19:33:15 is there anything new on any of these topics others would like to bring up? 19:35:14 Sounds like no 19:35:16 none for me -- still trying to reduce puppet :) 19:35:22 ++ thank you for working on that 19:35:27 #topic Open Discussion 19:35:42 Anything else? 19:37:41 Next week is a US holiday Thursday and Friday. I plan to be around for the first half of the week and will host a meeting at our regular time 19:38:03 I'll likely be looking for followups to the gerrit upgrade more than anything else and try and enjoy some quiet time after a busy few months 19:38:50 i have visited the USA outlet store and acquired frozen cranberries for sauce, as that's not really a thing here 19:39:28 all you need is a bog and some cool weather... oh 19:41:28 ianw: the best cranberries come in a can and look like jello 19:41:47 we can probably ship some to you though I have no idea what that would cost or how long it would take 19:42:17 clarkb: i think that's one of those things you had to grow up with to like :) i couldn't get into the beans with mushroom soup either. but yams and marshmallows, that was a revelation 19:42:28 ha 19:43:34 i have to disagree on the congealed canned berries. but then again i grew up with stewed fresh ones 19:43:56 we only got the canned stuff because we lived on the island. Fresh and dried are relatively new to me. 19:44:04 anything which comes in a can should not be served still in can shape 19:44:09 fungi you might like to look in on https://review.opendev.org/#/c/761621/ and the follow-on https://review.opendev.org/762588 as that updates the build table to not be unselectable in zuul ui. i think i remember you having some troubles there 19:44:57 ianw: ooh, thanks! yes that limitation has continued to plague me 19:45:11 alright sounds like that may be it. Feel free to follow up with us in #opendev or at service-discuss@lists.opendev.org 19:45:24 I'll call the meeting now and we can all have 15 minutes of breakfast/lunch/dinner back :) 19:45:29 thank you everyone! 19:45:32 #endmeeting