19:00:12 <clarkb> #startmeeting infra
19:00:12 <opendevmeet> Meeting started Tue Mar 19 19:00:12 2024 UTC and is due to finish in 60 minutes.  The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:12 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:00:12 <opendevmeet> The meeting name has been set to 'infra'
19:00:25 <clarkb> #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/2KGXT47R66KRZPHD6I2HXB23SLIKCYOZ/ Our Agenda
19:00:35 <clarkb> #topic Announcements
19:00:50 <clarkb> fungi: made a git-review 2.4.0 release yesterday. Now is a good time to update if you haven't already
19:01:02 <clarkb> having us use the latest version helps find any issues that might be present through our day to day use
19:01:29 <clarkb> And the PTG is fast approaching
19:01:31 <clarkb> #link https://etherpad.opendev.org/p/apr2024-ptg-opendev Get your PTG agenda items on the agenda doc
19:01:32 <fungi> though also installing the branch tip should always be safe, since we do test it
19:01:51 <fungi> and helps us find problems prior to tagging a new release
19:02:00 <clarkb> ++
19:02:25 <clarkb> for the PTG I've gone ahead and added soem stuff. But as previously mentioend I expect ti to be fairly low key on our side and more of an outreach to our users sort of thing
19:02:34 <clarkb> help them with their items rather than push any specific thigns from our side
19:02:43 <clarkb> that said feel free to add content to that etherpad
19:03:43 <clarkb> #topic Server Upgrades
19:04:05 <clarkb> I haven't seen any new movement on this and I've been plenty distracted by old distro cleanups and rax auth and zuul reviews
19:04:30 <clarkb> I suspect that the rest of us may similarly be distracted and not have anything new here. I'll give it a few minuets for others to chime in if I've missed anything before we move on
19:07:28 <clarkb> #topic MariaDB Upgrades
19:07:36 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/910999 Upgrade refstack mariadb to 10.11
19:07:40 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/911000 Upgrade etherpad mariadb to 10.11
19:07:54 <clarkb> if I can get reviews on these changes I'm happy to approve and watch the rollout when it is convenient
19:08:21 <clarkb> I think thats mostlywhere this is at. Just sanity checks that we're ok with the process applied to these services and then finding time to do it
19:09:31 <clarkb> #topic AFS Mirror cleanups
19:09:36 <clarkb> CentOS 7 is gone
19:09:43 <fungi> and there was much rejoicing
19:09:43 <clarkb> thank you everyone for the help getting things into shape to make that possible
19:10:20 <fungi> if that experience has taught me anything, it's that i'm not looking forward to untangling ubuntu-xenial removal
19:10:23 <clarkb> I did notice that I made some mistakes in cleaning up the opensuse obs centos 7 stuff whcih led to even more cleanups to the opensuse volume
19:10:29 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/913608 Final OpenSUSE mirror cleanup
19:10:48 <clarkb> this is opensuse cleanup but it is related to centos 7 :) and I think is the last step in finally clearing up both opensuse and centos 7
19:11:12 <clarkb> and ya after that is done it is time to start looking at ubuntu xenial removal which is going to be "fun"
19:11:30 <clarkb> I've already got some changes up under topic:drop-ubuntu-xenial but that is going to be just the tip of the iceberg I expect
19:11:55 <fungi> more like a continental ice sheet
19:12:12 <fungi> or at least a modest glacier
19:12:14 <clarkb> The upside to all of this is our afs volume capacity is looking far moer healthy. Which means we should have plenty of space for things like ubuntu 24.04 and maybe others
19:13:12 <clarkb> so ya reviews welcome on that last cleanup change then be on the lookout for xenial removal stuff probably more serisouly after the PTG? I think between now and then I may have too much other stuff to work on
19:13:23 <clarkb> #topic Rebuilding Gerrit Images
19:13:27 <clarkb> Of which this is one :)
19:13:59 <clarkb> I'd like to get our gerrit 3.9 image updated to 3.9.2. One big reason for this is it will make it easier to test 3.9's broken change reindexing to better understand if and how that affects us before we upgrade
19:14:19 <fungi> sounds like a great idea
19:14:24 <clarkb> but that will also upate our gerrit 3.8 image which means we should couple landing that change with a production restart of gerrit. Might be a good thing for late this week
19:15:06 <clarkb> similar to the db upgrade changes I'm happy if others review the changes then I can approve and babysit as necessary (in this case do a docker-compose pull and down and up -d)
19:15:32 <clarkb> re the 3.9 reindexing issue I also put it on the April 4th gerrit community meeting agenda to discuss the impact and better understand the issue
19:15:47 <clarkb> it seems like a fairly serious problem but they aren't rushing to fix it so I may just lack understanding
19:16:47 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/912470 Update our 3.9 image to 3.9.2
19:16:56 <clarkb> I forgot to link the change. That chagne is the one that needs reviews :)
19:18:22 <clarkb> #topic Rackspace MFA
19:18:58 <clarkb> We will be required to setup MFA in rax March 26th but last week we decided to opt into it early to find any expected problems in a more controlled manner
19:19:12 <clarkb> as noted yesterday fungi and I have meetings basically all day today so we said we'd do this tomorrow
19:19:22 <clarkb> fungi: I'll just catch up with you tomorrow moring on that and we can work through it?
19:19:41 <fungi> yep, sounds good to me
19:20:36 <clarkb> #topic Project Renames
19:20:53 <clarkb> preparing for this also in the list of things I need to look at before worrying too much about ubuntu xenial removal
19:21:08 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/911622 Move gerrit replication queue aside during project renames.
19:21:15 <clarkb> this is a prep change for the playbook itself
19:23:10 <clarkb> Once we're a bit closer we'll need to records changes too but don't need to do that too early
19:24:32 <clarkb> #topic Linaro Cloud Cert Renewal
19:24:47 <clarkb> Our helpful certchecker tool is warning us that this cert will expire on April 15
19:25:07 <clarkb> I've written down the process for renewing the cert on the node itself which si reachable from bridge
19:25:28 <clarkb> its fairly straightforward we run acme.sh to get a new LE cert. Then copy its contents where kolla can find it and run kolla
19:26:21 <frickler> where is that writeup? I could take a look at this
19:26:31 <clarkb> frickler: its in the root homedir on the server itself
19:27:12 <fungi> should we port that process document into system-config's docs for posterity, or is it too full of sensitive info?
19:27:22 <clarkb> fungi: we probably could
19:27:35 <clarkb> I don't think it has any sensitive info in the process.
19:28:15 <clarkb> the server is access via bridge (and the server is the last one in our ansible inventory). You ssh in via bridge and then the doc is there. Happy to help port to our proper docs now that we've used thep rocess more than once and didn't need to make big changes ot it
19:28:24 <clarkb> the first time I wrote it down I was distilling from a couple of emails.
19:28:37 <clarkb> but I think tonyb did the last one without any issue off the local doc so ya proper hosting of that is fine
19:28:41 <fungi> if discoverability is a concern, replacing it with a one-line file linking to the appropriate section of our docs would likely work
19:29:06 <clarkb> ++
19:29:09 <frickler> I'll take a look at it later this week, seems it isn't urgent yet
19:29:17 <clarkb> frickler: ya not urgent. Let me know if you have any questions
19:29:30 <frickler> except for the daily warning mail, which is a good reminder ;)
19:29:45 <clarkb> #topic Nodepool Image Delete After Upload
19:30:01 <clarkb> corvus wanted to point out that this new feature has landed in nodepool and we can make use of it if we like
19:30:30 <clarkb> basically you can update the configs to opt into having nodepool builder clean up on disk image files after they have been uploaded to clouds. Additionally you can opt to keep certain image formats and clear out others.
19:30:57 <clarkb> In our case I think we could choose to delete all but the qcow2 format of the image since that file is the smallest and we can generate the other two (raw and vhd) from the qcow2 if necessary
19:31:03 <corvus> might be worth asking what we want out of local images; recovery in case of cloud issues? user access for reproducibility?
19:31:32 <clarkb> corvus: currently we do host files for users to do local reproduction but we only serve the qcow2 files to force people into the lower bandwidth option
19:31:52 <clarkb> for this reason I don't think we want to delete all of the image files after upload, but I do think we can get away with keeping only the qcow2 file
19:31:52 <corvus> so qcow2 for user access would be great then; and i think we're probably not too worried about cloud issues?
19:31:57 <fungi> we did make them downloadable for user reproducibility reasons, i point people to those with some regularity
19:32:23 <clarkb> corvus: ya I think for cloud issues we have enough clouds that we would fetch from one of them to push into another if absolutely necessary
19:32:33 <clarkb> or just rebuild and let other clouds deal with that particular image type in the interim
19:32:36 <corvus> like, if we lose a cloud we could probably live with either "let the next rebuild fix it" or "convert it ourselves" if we're in a dire situation.
19:32:43 <clarkb> exactly
19:33:11 <fungi> or remove that cloud from out configuration temporarily
19:33:24 <fungi> s/out/our/
19:34:44 <clarkb> maybe go ahead and push up a change to keep only the qcow2s post upload and if there are any further concerns they can be posted in review?
19:35:17 <frickler> with a similar reasoning we might be ok with only keeping the latest local image and not multiple iterations?
19:35:44 <clarkb> yes, though i'm not sure if nodepool can express that yet
19:36:13 <frickler> yes, this was more of like another feature idea
19:37:09 <clarkb> #topic Open Discussion
19:37:27 <clarkb> That took us to the end of the listed agenda but I forgot to add a couple things before sending it
19:37:30 <clarkb> #link https://review.opendev.org/c/opendev/system-config/+/913686 Update Gitea to 1.21.8
19:38:00 <clarkb> This fixes a number of bugs in gitea supposedly. Would be good to get that in to keep up to date
19:38:13 <clarkb> And then separately etherpad released a 2.0 (and then very quickly a 2.0.1)
19:38:37 <clarkb> The changes to etehrpad appear to largely be in how etherpad is installed and managed (they use `pnpm` now I don't know what that means really yet)
19:39:01 <clarkb> It appears that this means we'll have to completely redo our docker image builds but I think the upgrade itself is straightforward as the db doesn't change
19:39:24 <fungi> pnpm seems to be a package manager for npm
19:39:32 <clarkb> if anyone else wants to look into that please feel free. Our current image is basically a fork of their image because they weren't reliably rebuilding their images for nodejs security issues
19:39:47 <clarkb> So we need to port our fork over to the new stuff and then test the result
19:39:58 <fungi> https://pnpm.io/
19:40:15 <clarkb> you'll note that npm is a package manager itsefl
19:40:19 <clarkb> so now we have package manager managers
19:40:33 <fungi> looks like a meta package manager that tries to use some sort of caching file tree
19:40:52 <fungi> throws around terms like "content-addressable storage"
19:41:27 <clarkb> if I end up looking at this it will almost certainly be a post PTG task for me
19:41:38 <clarkb> and post PTG is probably fine since big update to etherpad pre PTG are scary
19:41:41 <fungi> also looks like pnpm positions itself as a more efficient replacement for npm
19:42:49 <clarkb> Anything else?
19:43:17 <clarkb> I feel like there is a lot but we're moving slowly due to the openstack release. I bring that up to reassure people its ok
19:43:33 <clarkb> priorities and all that
19:43:57 <fungi> pnpm seems to also try t replace yarn
19:44:16 <clarkb> fungi: the js community never found a tool that was too good to replace
19:44:42 <fungi> i secretly replaced the js community with folgers crystals
19:46:04 <clarkb> sounds like that may be it
19:46:08 <clarkb> Thanks everyone!
19:46:15 <clarkb> #endmeeting