19:00:09 #startmeeting infra 19:00:09 Meeting started Tue Apr 23 19:00:09 2024 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:09 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:00:09 The meeting name has been set to 'infra' 19:00:16 #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/WBD5BLMI6ZPKUB6FOSZ65P7P5RR7CHFS/ Our Agenda 19:00:31 #topic Announcements 19:00:46 I have no announcements. Did I miss anything announcement worthy? 19:01:24 i don't think so 19:01:43 #topic Upgrading Old Servers 19:02:13 haven't seen anything on this subject over the last week. I did have a mini panic when I thought Focal was EOL this month but it has another 12 months before that happens 19:02:57 it's been brought up that wiki.o.o is still very far behind and its ua/esm subscription expired a while ago 19:03:18 i'm trying to work out where that discussion with canonical stalled 19:03:39 thanks. I think the best case short term there is to renew the esm sub if possible and then contineu to look at a replacement 19:03:58 but rebuilding it on a newer system, if anyone has time, would address both that and the impending ssl cert expiration for it 19:04:25 ya though I'm happy to deal with the cert in a few weeks (which is on the agenda for later) 19:05:47 #topic MariaDB Upgrades 19:06:10 Etherpad and mailman have new shiny upgraded mariadbs now. Gerrit and Gitea are the remaining services that need similar treatment 19:06:29 After thinking about it a bit due to the ease of upgrades for everything else I'm kidna thinking we may get away with just sending it for gitea 19:07:05 CI should check general compatibility with newer mariadb and if that checks out then the upgrade process itself has been pretty reliable so far and we can probably expect it to continue to be reliable 19:07:36 I'll push up a change for gitea and if anyone is concerned about that we can post those concerns in review 19:07:57 On the gerrit side of the house we will have to upgrade the db manually since the current ansible is very hands off with container management (and we want it to be) 19:08:18 But I do think we can just pick a time for gerrit and get it done since gerrit's db isn't very critical and the actual downtime should be quite short 19:08:55 i agree 19:09:18 in that case I guess I should also push up a change to reflect the update that will ultimately be somewhat manual in gerrit as well 19:09:55 #topic AFS Mirror Cleanups 19:10:24 I haven't made any new progress on Xenial cleanup. I think the scope of it is daunting enough that I'm happy to keep kicking it down the road if I can :) but I should just dive in and see what I learn 19:11:26 #topic Building Ubuntu Noble Nodes 19:11:42 https://review.opendev.org/c/opendev/glean/+/915907 merged which adds python3.12 support to glean 19:11:56 official release day for noble is thursday this week, right? 19:11:57 I believe that our image builds use glean from releases by default though 19:12:05 fungi: yes 19:12:23 This means we should plan to make a glean release. I can do that probably on Thursday ish as well 19:12:39 once we have a release we then need to monitor our next image builds but we have decent coverage of glean in CI so not too worried 19:12:55 #link https://review.opendev.org/c/zuul/nodepool/+/916053 Update Nodepool debootstrap to support Noble 19:13:08 separately we need to update Nodepool images to add support to debootstrap for noble 19:13:25 corvus: ^ not sure if you have any input on that, but I think it does have the votes it needs so maybe we just go ahead and approve it now 19:14:06 sgtm; i'll add a general +2 (i haven't reviewed the details) 19:14:12 ack thanks 19:14:35 Good news is there is solid progress here. Thank you frickler for testing it out and finding where the initial issues are 19:14:41 3rd patch needed will be https://review.opendev.org/c/openstack/project-config/+/916050 19:16:02 once we've got these three items sorted out we should be able to start building images and/or adding mirroring 19:16:30 looks like those changes are approved now 19:16:54 so just need the glean tag (assuming things merge/promote successfully) 19:17:09 yup and I've got that on my todo list now 19:17:18 awesome, thanks 19:17:44 I'm not committing to doing it immediately because I'm not sure what i need to do to get my gpg key out of cold storage and usable again 19:17:56 there is also a testing patch in dib fwiw https://review.opendev.org/c/openstack/diskimage-builder/+/915915 19:18:25 I'll update that once the glean release is done 19:18:28 sounds good 19:18:39 #topic Etherpad 2.0.x Upgrade 19:19:26 I can't remember the state of this during our last meeting, but I got the unreleased state of Etherpad to work with new api auth methods and added a bunch of testing to ensure that our admin tasks don't regress with the new setup 19:19:57 Everything seems to be working which means that as soon as etherpad makes a release we should be read to upgrade. I don't want to upgrade until a release is done just in case anything else changes 19:20:29 #link https://review.opendev.org/c/opendev/system-config/+/914119 Etherpad updates if interested 19:20:40 that chagne has all the new testing and docs and so on if you are interested in taking an early look at it 19:20:51 #topic Add reference to the project-team-guide in the fail ci msg 19:21:00 #link https://review.opendev.org/914189 19:21:07 fungi: you added this topic want to fill us in? 19:21:21 this was more of a concern with blurring lines between opendev and openstack 19:21:31 see my comments on the linked review 19:22:15 gotcha, I think I'm ok with adding that to the openstack tenant and having those of us still in the openstack tenant that probably shouldn't deal with it 19:22:15 basically, are we (collectively) okay having failure results in the openstack zuul tenant linking to openstack-specific guidance/documentation? 19:22:39 and if anyone has questions or concerns we can point them in the direction of using a different tenant 19:23:04 (just to be clear that I don't think we should add all the possible debug doc links to the openstack tenant, more that we're in the openstack tenant so openstack things are probably ok) 19:23:16 normally i'd say yes, but the openstack tenant was our original/default tenant and is still used by a lot of non-openstack projects as a result 19:23:25 I agree that this may be a good opportunity to progress with tenant separation if needed 19:23:57 its also something the vast majority of people are unlikely to read unfornately 19:24:04 i agree with clarkb and frickler 19:24:10 I know I click the zuul summary button myself :) 19:24:15 i'm okay approving it, but wanted to make sure there weren't major objections 19:24:24 the openstack guidelines don't sound too unreasonable for other projects I'd say 19:24:37 right, they're pretty general and not bad 19:24:59 i'm just wary of setting new precedent without further consultation 19:25:04 I'll +2 it after the meeting to make my stance clear in review (and to record it better) 19:25:07 thx for mentioning the patch, though, I had missed it somehow 19:25:12 sounds good, thanks 19:25:41 #topic Gerrit 3.9 Upgrade Planning 19:26:04 We're in a position now where I think we could upgrade Gerrit tomorrow if we wanted to 19:26:22 Gerrit 3.9 reindexing should work, our images are updated to pull that fix in, etc 19:26:30 #link https://www.gerritcodereview.com/3.9.html Release Notes 19:26:54 there are some intentional breaking changes though as noted in the release notes. None of them really impact us from what I can tell. 19:27:06 There is also a very straightforward downgrade path back to 3.8 if necessary 19:27:47 i'm up for a tomorrow upgrade ;) 19:27:51 All that to say our next steps here should be to write up a document to track things like downgrade testing and breaking change review. Then we can schedule a day for the upgrade and get it done. I'll volunteer to get that document started and we can coordinate from there 19:28:02 to be clear I don't think it will happen tomorrow :) just that the major known issues have been addressed :) 19:28:10 yeah, got it 19:28:53 I also want us to keep in mind that we should start thinking about a gerrit server upgrade. Historically we've tried to give people some notice of these in order to update their firewall rules. THough honestly with the number of old clients attempting to authenticate and failing it may be a good idea to make it a surprise :) 19:28:56 also my availability for the first half of next month is pretty limited, so if i'm going to help it will either be next week or several weeks out 19:29:05 several weeks out is probably fine 19:29:15 since there is prep work to get done and plenty of other stuff to do as well 19:30:12 Talking out loud here I think our longer term planning should be something like upgrade to 3.9 in Mayish, upgrade the underlying server to noble during the summer sometime, then upgrade to 3.10 in November/December/January time frame 19:30:46 One other thing to note that doesn't really affect us since we've already done the work is JGit just proposed that they will drop java 11 support 19:30:56 We're already on java 17 so that is fine for us 19:31:05 would we consider moving away from vexxhost with the server? like we still don't have feedback about the unplanned shutdown yet? also recurring network issues 19:31:56 Considering the alternative options I'm not sure there is a clear cut winner in hosting. However, that is a good question and somethign we should keep in mind as we plan for server replacement 19:32:20 vexxhost in particular lets us run a large server which the jvm running gerrit seems to appreciate 19:33:24 I would be seeing ovh as possible option, maybe we can the the flavor question with them if there is interest in that 19:33:27 Let's bring that back up after we've upgraded to 3.9 and start planning the server replacement. Ideally we'd be able to work with vexxhost to address the concerns 19:33:44 s/the/address/ 19:35:08 #topic Wiki Cert Renewal 19:35:24 In the past I've renewed this cert when there has been a week or less of validity 19:35:56 the current cert expires May 18, 2024. Which means renewing it sometime after May 11 19:36:12 I'm happy to do that, but didn't want others worried it was ignored as we get the daily warning email 19:36:26 ftr i'll be out of town, though it's a straight-up file replacement and apache2 restart 19:37:02 just noticing that iirc I still have no login for that server, so could not help right away if needed 19:37:39 frickler: thats a good call out. I suspect tonyb doesn't either. I can probably manually address that too when dealing with the cert 19:38:01 I do. fwiw 19:38:11 ah ok 19:38:12 i'm happy to add/update ssh keys in a few minutes 19:38:16 fungi: thanks! 19:38:34 i can just copy/merge from a properly config-managed server, i expect 19:38:40 I'm wondering if we can build a new server in that opendev domain to get LE so this becomes less of a problem 19:38:49 note there's a review with a new key up 19:38:58 ..... as part of the server upgrades away from bionic 19:39:05 tonyb: the domain isn't the problem, just need ansible working 19:39:20 ahh okay. nevermind 19:39:31 but yes, if we can get it onto a newer ubuntu version, easily solved 19:39:33 we'll get that "soon" 19:39:50 In any case I wanted to make sure that others know the deadline here is known and that I intend on addressing it manually 19:40:01 +1 19:40:13 Normally I would ask for volunteers but it involves paying money and filign an expense report and I don't expect others to do that here 19:40:46 side note: The cost of a cert and the cost of a beer have converged and in the wrong direction 19:41:00 fingers crossed this is the last year you expense a cert on domaincheap 19:41:20 #topic Open Discussion 19:41:21 so you can focus on expensing beer 19:41:24 Anything else? 19:41:38 if we can get a new machine before may 18 donee need to do the expense dance? 19:42:11 as mentioned above, i'll be travelling may 2-13 and not available to help with things 19:42:32 regarding the ceph reef mirror change, do we want to/can we move all the repos onto a single volume, so we don't need a new one for each release? 19:42:48 tonyb: probably not, but I wouldn't worry about it 19:42:50 i could do the cert replacement on may 14 but that's cutting it close 19:42:51 fwiw I'll be in the US from Sunday 28/April to mid June (maybe the 17th) 19:43:07 also I'll be away this Fri and Wed-Fri next week 19:43:12 tonyb: basically renewing the cert is cheap enough we shouldn't consider it a hard deadlien thing 19:43:35 clarkb: okay. 19:44:00 oh, in other news, i've started in on a dns record audit for openstack.org as mentioned in last week's meeting 19:44:04 I hope you all manage to enjoy your time off 19:44:48 fungi: manage to delete any records yet? 19:45:09 it's complicated by the domain being shared jurisdiction with foundation staff and webdev contractors, so all three parties err on the side of caution in not deleting records we've collectively forgotten why we added 19:45:35 i see quite a few i'm pretty sure can be deleted, but i'll float a list first 19:46:33 sounds good, thank you for looking at that 19:46:50 I'll give it a few more minutes if anyone else has somethign to share otherwise we can probably end about 10 minutes early today 19:46:54 ideally, we'll clean up what we know can go, then provide the other two parties with a list of records we don't have any interest in and let them separately filter the remainder 19:48:08 current record count is close to 300, so it's nontrivial to untangle 19:48:36 wow that's much more than I would have guessed. 19:49:15 there are still records in there pointing to sites that contained openstack project policy board (tc predecessor) election results 19:49:46 easily irrelevant for over a decade 19:50:08 fair enough 19:50:15 sounds like that was everything. Thank you for your time today and help running OpenDev. We'll be back here same time and location next week. 19:50:19 #endmeeting