19:01:33 #startmeeting infra 19:01:34 Meeting started Tue Jul 21 19:01:33 2020 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:35 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:38 The meeting name has been set to 'infra' 19:01:48 o/ 19:02:16 o/ 19:02:26 #link http://lists.opendev.org/pipermail/service-discuss/2020-July/000058.html Our Agenda 19:02:34 #topic Announcements 19:03:32 OpenDev virtual event #2 happening July 20-22 19:04:00 This event is happening. The upgraded version of etherpad seems to be holding up just fine 19:04:13 #topic Actions from last meeting 19:04:41 #link http://eavesdrop.openstack.org/meetings/infra/2020/infra.2020-07-14-19.01.txt minutes from last meeting 19:05:16 No actions recorded 19:05:21 #topic Specs approval 19:05:38 #link https://review.opendev.org/#/c/731838/ Authentication broker service 19:05:48 again not quite ready for approval, but definitely available for feedback 19:06:46 fungi: ^ anything else to add on this? You saw my feedback? 19:09:24 yep, thanks for reviewing that, will incorporate into the next revision 19:10:28 #topic Priority Efforts 19:10:37 #topic Update Config Management 19:11:47 Zuul and Nodepool are running on containers now with the exception of nb03 19:12:21 nb03 runs on an arm64 host so needs an arm64 image 19:12:56 I've collected a number of arm64 container image build fixes for nodepool in this change https://review.opendev.org/#/c/741942/ 19:13:27 latest problem is that cryptography did a release between the time we had working wheel cache setup on cryptography 2.9.2 and vhd-util ppa being fixed so now there is an uncached cryptography 3.0.0 and we try to build that on buildx and timeout 19:13:32 ping me if you need me here; i'm working on zuul emergency restart 19:13:57 ianw: ^ we expect that mirror to update periodically right? any idea how soon we should have a new wheel for cryptography? 19:14:48 but I think ocne that happens we can land that change then we will have an image and can start looking at updating nb03 19:15:28 yes, wheels *should* update daily, however, i had a look and there were network issues the other day 19:16:24 kevinz fiddled something with routers in the cloud, and i haven't looked since. so i'll get back to it and see -- but the jobs couldn't get a reliable ipv4 connection and so afs was just not happy 19:16:50 got it, in any case that seems to be the current issue and so getting the wheel cache updated would be helpful 19:17:11 Related to services running in containers we've just discovered that docker-compose up -d doesn't pull new images if it already has images but they are out of date 19:17:25 something to be aware of and maybe something we'll address in our playbooks 19:17:29 one thing is that the release job only releases if *all* the wheel builds pass ... i'll look at that. it might have just been done like that before we could do something like a semaphore 19:17:59 ianw: if we're able to safely copy out successful builds doing partial updates seems fine? 19:18:25 I mean if we don't have new A or B package we'll try to build both in the downstream jobs. If we have only A or B, we'll try to build the other and that is still a net win time wise 19:18:42 if we want different versions that needs to be controlled by requirements or constraints 19:18:52 yeah, there's no comments that suggest it was done for particular consistency reasons 19:20:08 http://lists.openstack.org/pipermail/openstack-discuss/2020-July/016022.html is also related to updating config management 19:20:45 I'ev sent a response and approved on of the changes. But more input on the general goal there would be good. The mailing list would probably be best for that so that Javier sees it too though 19:21:43 #topic OpenDev 19:21:50 #link http://lists.opendev.org/pipermail/service-discuss/2020-July/000057.html Advisory Board welcomed. 19:22:00 I think that mostly makes the advisory board and its membership official 19:22:12 I've encouraged our volunteers to get involved and we'll see where it takes us 19:22:14 ianw: possibly related to network issues, but sslcheck is also saying for the past couple of days that the mirror cert there is is expiring unexpectedly soon 19:22:41 fungi: ok, will look 19:22:47 #link http://lists.opendev.org/pipermail/service-announce/2020-July/000007.html Gerrit /p/ mirror deprecation. 19:23:08 This is another email I wrote. With plan to get ahead of gerrit repurposing /p/ for project dashboards instead of git mirrors 19:23:19 this simplifies the number of locations we need to replicate and makes branch management simpler 19:23:33 and since it will happen when we update gerrit anyway we may as well do it now and get those other wins :) 19:24:48 I'll propose a change to make those urls 403 when I get a momement, but I have about a week and a half 19:25:05 and I'll go ping the cinder channel too as they seem to be the bulk of the users of that old path 19:25:20 Anything else OpenDev related to bring up? 19:26:21 #topic General Topics 19:26:25 the openstack-infra ml is now closed 19:26:32 * fungi wasn't quick enough 19:26:36 fungi: oh ya, thank you for doing that 19:26:42 np 19:26:52 #topic Bup and Borg Backups 19:27:14 ianw: for Bup I was curious if you had success doing a restore from the hosts that had their local indexes cleaned up. 19:27:28 and for Borg I think it would probably be good to have a short intro to your thoughts/plan around it 19:28:02 ahhh, yes, i didn't get to that restore. i will loop back on that 19:28:35 mostly because i started looking at a new backup server, and it seems, as we've discussed before, bup is effectively dead 19:29:03 there's no python3 support, and it doesn't seem to be coming unfortunately, and it's dropped from focal 19:29:47 so i had a serious look at rdiff-backup and borg; both are very close to bup in terms of the hosts backing themsevles up to a central location and deduping on the server side 19:30:18 borg seemed to be more active, had clarkb's approval, and had a lot of other features 19:30:32 the clarkb seal of approval 19:30:43 #link https://review.opendev.org/741366 19:30:52 (ya I use borg locally, I've not had to use it in an emergency yet but seems to have a number of good features like append only support, encryption at rest if we want it, fuse mounts for restoration) 19:30:52 so that is the roles to do borg backups 19:31:49 i'd suggest, if we're happy to move on with it, we bring up a borg backup server and switch in a ... lesser value host (maybe not start with review i mean) ... and start from there to get a bit of experience 19:32:21 I like that idea 19:32:25 seems we're backing up review-dev 19:32:31 also we potentially get the ability to prune append only backups? 19:32:32 or we were anyway 19:32:42 not sure how effective that will be but that could be a great feature 19:33:08 append-only is implemented in above -- it does seem that we could run jobs on the backup server to then prune repositories 19:33:38 so clients can't delete their history (restricted ssh command), but we can on the server side 19:35:20 and in reviewing the change this is implemented completely to the side of bup so we could switch a host over with little pain I expect? 19:35:34 basically add it to borg backups, let it backup with borg, check things, then disable bup ? 19:35:57 yes, i deliberately kept them totally separate, with separate group names etc. so they can run completely in parallel 19:36:17 thanks, wise plan 19:37:42 that's more or less it; if we like it, i'm happy to move on with starting the server and initial backup runs (as mentioend eview-dev seems a good place to start) 19:38:53 ++ 19:39:04 I think I'm ok with it, but corvus would be a good source of feedback too having built the bup stuff 19:39:14 I'll review the latest ps looks like some of my feedback was addressed 19:39:49 #topic Project Renames 19:40:13 Last week we pencilled in Friday for project renames. I think I'm going to be in a reasonable spot to do that still. fungi are you still able to help? 19:40:22 i am indeed 19:40:35 still wide open for friday 19:40:50 great, should we say ~15:00 UTC on friday for that? I've been having ealry days with opendev event happening so keeping to that schedule seems fine 19:41:06 and thursday ish we can curate an etherpad and the chagnes and make sure we are ready 19:41:34 i'm good with 1500z 19:42:17 great. I can sent an email about that to service-announce today after lunch 19:43:38 #topic Server Upgrades 19:43:53 fungi: anything new on wiki side of things? I don't expect so but didn't want to leave it out if there is 19:44:03 zilch 19:44:21 #topic Open Discussion 19:44:31 That was it for our scheduled agenda. Anything else to bring up? 19:45:23 I guess I should mention that https://review.opendev.org/#/c/741277/ is the next thing in my series of branch management changes. Land that should be safe and it won't be used until we tag the library. https://review.opendev.org/741279 tests it 19:49:17 Sounds like that may be it? Thank you everyone. I know it was a somewhat distracted meeting. See you next week. 19:49:24 #endmeeting