19:01:09 #startmeeting infra 19:01:09 Meeting started Tue Jul 20 19:01:09 2021 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:09 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:09 The meeting name has been set to 'infra' 19:01:12 #link http://lists.opendev.org/pipermail/service-discuss/2021-July/000269.html Our Agenda 19:01:33 #topic Announcements 19:01:38 I didn't have any announcements 19:01:53 #topic Actions from last meeting 19:02:02 #link http://eavesdrop.openstack.org/meetings/infra/2021/infra.2021-07-13-19.01.txt minutes from last meeting 19:02:11 #action someone write spec to replace Cacti with Prometheus 19:02:26 O/ 19:02:29 o/ 19:02:31 As far as I know that action hasn't been done. But good news is I think we're getting throug some of the stuff that had been distracting us and so that might get done soon :) 19:02:45 #topic Specs Approval 19:02:53 #link https://review.opendev.org/796156 Supporting communications on our very own Matrix homeserver 19:03:14 Last week we said we'd put this up for approval this week. Big part of that was people weren't around last week that might want to review the change 19:03:36 * mordred does dance 19:03:43 fungi: ianw ^ if you can give that a read before Thursday EOD clarkb time that would be great 19:03:53 yeah 19:03:57 clarkb: you're your own timezone now 19:04:06 tristanC has a change up to run a matrix gerritbot too which might be interesting to others 19:04:51 #topic Topics 19:04:57 #topic Gerrit Server Upgrade 19:05:08 This happened. Thank you ianw for doing the bulk of the work on this one. 19:05:53 We've had a few minor hiccups with things like java sql conncetor libraries and getting ssh host keys sorted last minute and making the change to reflect this in config management pass actually pass 19:06:01 all told those are small issues and this went really well. 19:06:30 I wanted to call out for followups we now have the ability to do some gerrit cache and memory usage tuning that we didn't really have room for before 19:06:36 the end result is we now have a bigger gerrit with a local db for the reviewed file checkmarks? 19:06:43 Maybe we let the server run for a week or two then look at data? 19:06:45 fungi: yup 19:07:23 The othe rthing that ianw has mentioned is we should all check our homedirs and the gerrit2 homedir for files we want to live on the new server too 19:07:47 I've already moved the ones I know I wanted to move, but still need to go look at the rest to see if any of it is worth carrying over 19:08:12 #link https://etherpad.opendev.org/p/gerrit-upgrade-2021 Look here for changes to review in followup to the server move 19:08:29 the etherpad also has a few changes that ianw has pushed in followup. I think I've reviewed them all, and other reviews would be nice too 19:08:50 ianw: anything else to say about this? 19:09:07 Do we want to keep it on the agenda for next week or can we consider this done? 19:09:25 nope, it seems like everything is going well. i'll wait for frickler_pto to become frickler and then i think we can retire the old server 19:09:42 oh and i think i have some log archives i can move across too 19:09:44 how is the mariadb connector fix applied to the image build? sed -i? 19:10:05 fungi: currently we reverted to the mysql connector, which is still compatible 19:10:06 fungi: we landed it via normal gerrit changes after manually editing the config file 19:10:30 we have both libs present in the image to support the old and new server db setups so we just had to change the connection line to specify the other lib 19:10:30 oh, got it, so all in configuration 19:10:57 yup, once mariadb connector works properly we can update the config as well as update our image to remove the old mysql lib 19:10:58 we're not actually running with the mariadb connector patched 19:11:05 correct 19:11:08 that makes sense, thanks 19:11:09 yep; no rush to get back. i was just using the mariadb connector because, well, we were connecting to mariadb :) 19:11:29 i wonder why they added a special one for that 19:12:15 just the way the subclasses work i think 19:12:31 basically string matching the db uri and then choosing classes 19:12:50 i do wonder why they added it with at least two bugs that made it not work at all, and nobody noticed, however 19:13:31 just seems odd that with as little as gerrit uses an rdbms for now, they'd incur the maintenance overhead of slight variations of a generic sql connector 19:13:42 historical baggage i suppose 19:13:51 anyway, not necessary to dig into now 19:13:52 it may have been added back in 2.16 or prior for the actual db when that existed 19:13:54 ya 19:14:15 #topic Gerrit Account Cleanup 19:14:33 I've been thinking about scheduling for this and the project renaming that we said we'd all do "week after the upgrade" 19:15:16 In my head it makes sense to do the external id pruning Say on Monday. That is 3 weeks after I disabled the related accounts. Then do the project renames later in the week, say Friday. 19:15:28 that sounds reasonable to me 19:15:36 The reason for this is if we need to do account surgery after the external id cleanup happens due to the cleanups we can take advantage of the outage to do it 19:15:58 we did surgery on two accounts during the move for unrelated reasons and it went well but does require a downtime 19:16:15 I don't expect we'll end up doing that as part of the project rename downtime, but it is nice to have the option 19:16:37 Ok I'll plan to run the external id cleanup script on Monday then. 19:16:37 yep 19:17:10 #topic Gerrit Project Renames 19:17:35 As just mentioned I'm suggesting Friday July 30 for this and probably early that day. Say 15:00UTC ? 19:17:58 that gives us time to do more testing if we can find time to do that testing but also the buffer to account cleanups which we can use to fix accounts if necessary 19:18:00 i'm around and happy to do it then 19:18:14 ok lets do it 19:18:30 I can send out email about that as well 19:19:29 #topic Gitea01 backup issues 19:19:38 I don't expect anything new has come up about this but wanted to double check 19:19:53 no i haven't looked into it sorry 19:20:01 no worries you were busy :) 19:20:02 i haven't heard any more on the ipv6 issues 19:20:38 i could take it out of the lb and reboot it i guess 19:20:55 and does look like it's still going on 19:20:58 the switch it off and on again method probably has a high likelyhood of fixing things 19:21:16 hah ok 19:21:26 yeah, neutron likes to get a kick in the interface every now and then 19:22:01 we can sync back after that happens one way or the other 19:22:09 sounds good 19:22:13 #topic Gitea 1.14.4 upgrade 19:22:41 I started working on this when all the other gerrit and zuul work was happening and didn't want to impact those as they seemed more important 19:23:11 but now that is largely behind us so I'd like to go ahead and land this upgrade when I have time to monitor it. Currently I think I have time tomorrow morning any objections to landing the change to upgrade gitea then? 19:23:20 #link https://review.opendev.org/c/opendev/system-config/+/800274 gitea upgrade 19:23:38 Its a fairly large gitea upgrade as far as the change delta goes, but I did hold a node and it looked good 19:23:50 and our testing caught one issue with a changing api so that seems to be working well for us 19:24:46 i'm good with it, finally reviewed and checked it out today 19:24:49 Sounds like no objections. If things are quiet when I start my day tomorrow I'll approve it 19:25:06 ++ 19:25:14 #topic High AFS utilization 19:25:15 i'll be more around tomorrow anyway, more than today 19:25:20 fungi: cool 19:25:47 Yesterday it was pointed out that the opensuse mirror was sad. It had a stale lock which I removed and then manually did updates for after and is happy now 19:26:00 In the process of looking into that I noticed htat our afs vicepa utilization is high 19:26:08 we have less than 10% remaining disk space 19:26:34 The debian volume looks like it may need to grow as well as we are right up against quota there 19:26:38 were we able to reclaim much with the centos cleanup? 19:26:49 fungi: yup, but then we backfilled with openeuler mirroring :) 19:26:59 we can push to try and drop debian-stretch images 19:27:20 yeah, although bullseye is fully frozen now, so we can probably drop some debian soon 19:27:24 ya I was going to ask about that particularly since debian is near its quota 19:27:27 or yeah, just push to drop it now 19:27:42 I think that would be a good next step if we think it isn't too disruptive to users for some reason 19:27:55 The other one that is large and I'm not sure still used is mirror.yum-puppetlabs 19:28:00 but yes, stretch is already "oldstable" (the release before the current stable) and is about to become "oldoldstable" 19:28:48 I think tripleo and/or puppet-openstack may be the users of the yum puppetlabs mirror. I'm wondering if they continue to use it at all and if so if they only use a small subset 19:28:52 we should probably ask the puppet-openstack team if they're still using that, and if so whether they're using all or just a small part 19:29:00 currently that mirror seems to have packages for old fedora and sles and so on 19:29:05 fungi: ++ exactly 19:29:29 I suspect we can trim that done one way or another. 19:29:46 I can send email to openstack-discuss asking puppet-openstack and tripleo to weigh in on it 19:29:56 fungi: ianw: any idea what the first step for debian trimming would be? 19:30:21 probably check for zuul configs referencing debian-stretch, though searching stable branches will be hard 19:30:36 probably better would be to start a discussion proposing we drop it, and then see if we get pushback 19:31:04 fungi: that seems reasonable. Is that something you might want to send to service-discuss? 19:31:10 yeah, i can have a look at it 19:31:18 ianw: that would be great thanks 19:31:30 i think both discussion, but also directly proposing changes might help 19:31:35 stretch has been obsolete for ~2 years since buster came out, and most projects aren't supporting their branches past that anyway 19:31:59 anything we can do to get dib's testing more under control is good too :) 19:32:07 The last note I put related to this on the agenda is I've noticed discussion about adding Alma linux to our node list. 19:32:09 i can bring up dropping stretch on the openstack-discuss ml 19:32:54 I think starting with clenaup is the right option but we should probably be prepared to add a third 1TB device to our vicepa setups if the linux distros keep multiplying 19:33:15 There is probably another disucssion there about whether we can delete centos entirely if we do alma or similar but I think it is a bit early for that 19:33:51 i mean we can build images before putting the mirror in 19:33:54 aha, alma linux is in the same vein as rocky linux and (non-stream) centos. i knew the name sounded familiaer 19:34:21 ianw: yup that is true, though I suspect ne wdistros like that may want us to offloda for them as much as possible 19:34:31 fungi: yes, alma seems to be the one iwth all the momentum right now 19:34:35 yes, mirrors are more of a general speedup, and matter more for our more heavily-used distros 19:34:58 so we might just consider mirrors to be out of scope for less-used images we offer 19:35:53 if a particular label only sees use in a handful of builds a week, the overhead in maintaining a mirror for the software is clearly overkill 19:36:34 taking that into account, we could always just consider dropping some of our mirrors of significantly under-utilized stuff even if we don't drop the images 19:36:38 fungi: yup, that is why I also mentioned the possibility of mv centos alma && sed -i -e s/centos/alma/ *zuul.yaml :) 19:37:08 (this was a bit of my concern with openeuler as it seems like that has a long path to being used) 19:37:12 But all of that currnetly depends on imgaes and people being interested in the work and so 19:38:05 ianw: thats a good point. I guess we can take a high level view once the afs pruning happens and we know where we stand 19:38:18 then start thinking a bit more critically about the various distros and whether we need full mirrors for all of them 19:38:44 Not something we need to solve today, I just saw the demand seems to be creeping up and wanted to call it out 19:39:24 #topic PTG Participation 19:39:38 I discovered yesterday that the deadline to sign up for PTG participation is tomorrow 19:40:03 The PTG is running October 18-22, 2021. If you are curious yes this sign up deadline is earlier than in the past 19:40:18 it feels sudden, but i guess that's because i've been on vacation 19:40:34 The office hours we did last time around seemed to work well for 2 hours in the EU friendly timezone. 19:40:40 s/timezone/time block/ 19:40:54 I'm happy to sign up for similar this time around as a result. 19:41:01 Does anyone think we should have more time than that? 19:41:19 Or perhaps think we should skip entirely? 19:42:24 what we did last time worked for me 19:43:09 fungi: ok, I'm proposing we trim that back to a single 2 hour block during the EU afternoon NA morning since that is where we got participation last time 19:43:19 ianw: ^ will you feel excluded if we do that ? 19:43:39 no 19:44:23 alright I'll sign us up for that. If anyone wants mor etime let me know and I'll adjust to try and include that 19:44:49 #topic Open Discussion 19:45:08 That was all I had. Except I remember thinking of something I should bring up during the meeting earlier today and I've completely forgotten what it was 19:45:27 Anything else? 19:45:42 i got nuthin' 19:46:49 nothing more from me 19:47:31 Alright then, thank you everyone for your time and now you can go find $meal :) 19:47:40 thanks clarkb! 19:47:54 As always feel free to find us in #opendev or at service-discuss@lists.opendev.org and we can discuss anything there that was missed here 19:47:57 #endmeeting