19:01:10 <clarkb> #startmeeting infra
19:01:10 <opendevmeet> Meeting started Tue Jun 27 19:01:10 2023 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:01:10 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:01:10 <opendevmeet> The meeting name has been set to 'infra'
19:01:17 <clarkb> #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/77NQ27ARPLEVBLT7ZSUVDWER6YYACCXN/ Our Agenda
19:01:20 <clarkb> #topic Announcements
19:02:22 <clarkb> Next tuesday, July 4, is a US holiday. I won't be able to attend this meeting on that day as I'll either be traveling (you might have noticed my timezone hasn't shifted and that is because I caught covid and shift all my travel back by two weeks), or celebrating the holiday wondering how much longer it will take to get over covid
19:03:41 <clarkb> I suspect that others will also be afk so we can probably cancel that meeting
19:04:39 <clarkb> #topic Bastion Host Changes
19:04:52 <clarkb> #link https://review.opendev.org/q/topic:bridge-backups This set of changes still needs additonal reviewers
19:05:02 <clarkb> #topic Mailman 3
19:05:22 <clarkb> I suspect fungi isn't able to join today and I also doubt much new has happened on this one due to the summit and travel etc
19:05:33 <clarkb> fungi: please let us know if these assumptions are bad
19:06:35 <clarkb> #topic Gerrit Updates
19:06:56 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/885317 Update Gerrit 3.8 image to final 3.8 release
19:07:16 <clarkb> This is a bit of a bookkeeping change to bump up gerrit versions. We were previously testing with a hacked up gerrit 3.8 rc build
19:07:23 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/884779 Revert bind mounts for Gerrit plugin data
19:07:40 <clarkb> And this undoes the bind mounting of plugin data so that we clear out the leaked replication task files when we down/up the container
19:07:57 <clarkb> I continue to think this isn't urgent but the longer we put it off the more files there will be to clean up in the end
19:08:26 <clarkb> #topic Upgrading Servers
19:08:46 <clarkb> tonyb's efforts to make the ci registry more testable have resulted in better testing and a new host replacing the old one. THank you for that !
19:09:23 <clarkb> The old host still needs to be removed https://review.opendev.org/c/opendev/system-config/+/887001 but we can queue that up behind the zuul executor replacements
19:09:36 <clarkb> since they all update the inventory file it the prod job queues can get fairly long
19:09:56 <clarkb> once that chagne lands we can delete the old server too. The new server is in production though due to dns updates and the rest of this is lower priority
19:10:08 <clarkb> Thanks again for the help and happy to help work through more of these with interested volunteers
19:10:16 <clarkb> #topic Fedora Cleanup
19:10:29 <clarkb> I haven't seen any new movement on this and don't think we have tonyb here today
19:10:56 <clarkb> However, it is probably worth noting that frickler expanding afs volumes to move the logjam out of the way for bookworm
19:11:04 <clarkb> thank you for pushing that forward frickler
19:11:49 <clarkb> #topic Quo vadis Storyboard
19:12:16 <clarkb> There was discussion in #openstack-infra ealier today about making projects inactive possibling removing those projects from search lists making it difficult to find the project IDs to perform cleanup activities
19:12:23 <fungi> around for a moment, assumptions about mm3 are spot on, unfortunately :(
19:12:37 <clarkb> I don't think we understand all the details here, but probably worth clarifying with projects as we mark them inactive if this is the case
19:13:02 <clarkb> so that they are aware any potential cleanup actions they want ot take may be more complicated (though it doesn't seem like any of the actions were prevented just more difficult to find identifiers)
19:13:17 <fungi> we can also look up sb project ids with a db query if the api isn't sufficient in that case
19:14:38 <fungi> (but does require a root sysadmin)
19:15:02 <clarkb> ack, considering this is he first time it has come up that may be good enough
19:15:09 <clarkb> doesn't seem to be a common occurrence in any case
19:15:18 <clarkb> #topic Gitea 1.20
19:15:27 <clarkb> Gitea 1.20 exists in RC form now (up to rc2)
19:15:46 <clarkb> I've pushed a change to start testing things, but we don't have a proper change log yet so this is unlikely to be comprehensive
19:15:51 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/886993
19:16:24 <clarkb> The "good" news is that this has exposed a problem in the oauth2 implementation where despite us disabling it it insists a JWT_SECRET value be set then tries to write it back to disk which fails because we don't allow gitea to write to its config file
19:16:29 <clarkb> #link https://github.com/go-gitea/gitea/issues/25377
19:16:52 <clarkb> looks like upstream may treat this as a bug, but they say the undocumented behavior of gitea writing back new config to its own config file is intended and won't be changed
19:17:05 <clarkb> people really need to stop designing software this way. Makes it impossible to use ocnfiguration management sanely
19:17:36 <clarkb> Poking around at it I think the only real issue there is the JWT_SECRET though and the other normalization I did doesn't appera to be required. It doesn't result in failed writes back to disk at least
19:18:10 <corvus> also super unsafe
19:18:35 <fungi> doesn't allow privilege separation between write access to configs and processes for the service
19:19:26 <clarkb> ya I think we should continue to set up perms on that file os that gitea can't write back to it then rely on our ci system to ensure we're configuring things properly
19:19:31 <clarkb> aka make it fail if it wants to write back
19:19:42 <clarkb> #topic Etherpad 1.9.1
19:19:49 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/887006
19:20:21 <clarkb> There is a new etherpad release. However I'm not sure we want to upgrade at this point... Number lists don't increment the number values. They are always just 1. 1. 1. etc apparently (though I've not tested this myself)
19:20:45 <clarkb> Also the tagged commit for 1.9.1 doesn't actually build. So 887006 checks out a commit a few commits later after they fixed the lock file contents so that it can build
19:20:54 <fungi> i guess this is per a bug report somewhere?
19:21:01 <clarkb> I'm hoping we get a 1.9.2 that fixes the build issue at least and then we can consider building that
19:21:05 <corvus> 🎶...one is the onliest number that you'll ever know...🎶
19:21:11 <clarkb> fungi: yes there is an issue for the numbered list thing
19:23:40 <clarkb> I've got github release event alerts enabled for both gitea and etherpad so hopefully we see new releases with fixes for things
19:23:55 <clarkb> unitl then be aware of the change and feel free to review them. But we probably won't upgrade until a few things get fixed
19:24:00 <clarkb> #topic Open Discussion
19:24:02 <clarkb> Anything else?
19:24:10 <corvus> re server upgrades: i've resumed the attempt to update the zuul-executors to jammy
19:24:58 <corvus> (after fungi's fix to the apt config which is what caused the previous rollback)
19:25:18 <corvus> er, to be clear, the incorrect apt config caused the rollback; fungi fixed it.  :)
19:25:18 <corvus> not caused it :)
19:25:50 <fungi> just a reminder that i'm not around much this week, popping in from time to time as i get a moment (like now obviously). i'll be my usual self again next week
19:26:23 <corvus> be whoever you want to be, i say
19:27:42 <fungi> sage advice
19:28:32 <clarkb> I'll be around but also trying to take it easy. I'm still testing positive for covid as of today
19:28:53 <fungi> hope you and the fam feel better soon
19:29:12 <corvus> ++
19:29:21 <clarkb> and in theory I'm traveling a week from now. That should be long enough to shake this off
19:29:34 <clarkb> and then timezones will be weird. But for this week I'm around
19:29:48 <fungi> i'll be home and caught up by the time you're travelling, so that should work out
19:30:35 <clarkb> sounds good
19:30:58 <clarkb> I don't need to take up any more of our time today. Thank you everyone! We'll likely skip next week (I won't be here anyway) and then try to pick this back up July 11
19:31:01 <clarkb> #endmeeting