| clarkb | meeting time | 19:00 |
|---|---|---|
| * fungi waves | 19:00 | |
| clarkb | I think I'm awake enough I can run through this agenda | 19:00 |
| fungi | tc call is running a few minutes over but i can chair if you want just may need to delay a few minutes | 19:00 |
| clarkb | nah I can do it | 19:00 |
| fungi | thanks! | 19:01 |
| fungi | and good morning | 19:01 |
| clarkb | I've got the cacophony of birds outside reminding me it is time to wake up anyway | 19:01 |
| clarkb | #startmeeting infra | 19:01 |
| opendevmeet | Meeting started Tue Jul 11 19:01:25 2023 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| opendevmeet | The meeting name has been set to 'infra' | 19:01 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/FV2S3YE62K34SWSZRQNISEERZU3IR5A7/ Our Agenda | 19:01 |
| clarkb | #topic Announcements | 19:02 |
| clarkb | I did make it to UTC+11 | 19:02 |
| clarkb | I'm finding that the best time to sit at a computer is something like 01:00/02:00 UTC and later simply due to weather. But we'll see as I get more settled this is only day 5 or something there | 19:03 |
| clarkb | #topic Topics | 19:04 |
| clarkb | #topic Bastion Host Updates | 19:04 |
| clarkb | #link https://review.opendev.org/q/topic:bridge-backups | 19:04 |
| clarkb | Looks like this set of changes from ianw could still use some infra root review | 19:05 |
| clarkb | if we can get that review done we can plan the sharing of the individual key portions | 19:05 |
| clarkb | #topic Mailman 3 | 19:06 |
| clarkb | fungi: any updates on the vhosting? then we can talk about the http 429 error emails | 19:06 |
| fungi | no new progress, though a couple of things to bring up yeah | 19:06 |
| clarkb | go for it on the new things | 19:06 |
| fungi | the first you mentioned, i'm looking to see if there's a way to create fallback error page templates for django | 19:07 |
| fungi | but perhaps someone more familiar with django knows? | 19:07 |
| fungi | i know we can create specific error page templates for each status | 19:07 |
| fungi | so we could create a 429 error page template, but what i'm unsure about is if there's a way to have an error page template that applies to any error response which doesn't have its own separate template | 19:08 |
| fungi | i think i recall tonyb mentioning some familiarity with django so i might pick his brain later if so | 19:08 |
| clarkb | I'm unsure myself | 19:09 |
| fungi | assuming my web searches and documentation digging turn up little of value | 19:09 |
| clarkb | a default would be nice if possible but I suspect adding a 429 file would be a big improement alone | 19:09 |
| tonyb | I don't think it was me | 19:09 |
| fungi | oh too bad | 19:10 |
| tonyb | sorry I'll do better :P | 19:10 |
| fungi | the other item is we've had a couple of (mild) spam incidents on the rust-vmm ml, similar to what hit zuul-discuss a few months back. for now it's just been one address i initially unsubscribed and then they resubscribed and sent more, so after the second time i switched the default moderation policy for their address to discard instead of unsubscribing them | 19:10 |
| fungi | but still might consider switching the default moderation policy for all users on that list to moderate and then individually updating them to accept after they send good messages | 19:11 |
| fungi | that is if the problem continues | 19:12 |
| clarkb | I'm good with that but ideally if we can find a moderator in that community to do the filtering | 19:12 |
| clarkb | I'm not sure we should be filtering for random lists like that. | 19:12 |
| fungi | well, yes i stepped in as a moderator since i was already subscribed and the only current community moderator had gone on sabbatical, but we found another volunteer to take over now | 19:13 |
| clarkb | great | 19:14 |
| fungi | my concern is it seems like the killer feature of mm3, the ability for people to post via http, increases the spam risk as well | 19:14 |
| fungi | which is going to mean a potentially increased amount of work for list moderators | 19:14 |
| clarkb | Though two? incidents in ~6 months ins't too bad | 19:15 |
| fungi | yeah, basically | 19:16 |
| fungi | but these are also very low-volume and fairly low-profile lists | 19:16 |
| fungi | so i don't know how that may translate to some of the more established lists once they get migrated | 19:16 |
| fungi | something to keep an eye out for | 19:16 |
| clarkb | there is probably only one way to find out unfortunately | 19:16 |
| fungi | agreed | 19:17 |
| fungi | anyway, that's all i had on this topic | 19:17 |
| clarkb | #topic Gerrit Updates | 19:17 |
| clarkb | We are still building a Gerrit 3.8 RC image. This is only used for testing the 3.7 to 3.8 upgrade as well as genereal gerrit tests on the 3.8 version but it would be good to fix that | 19:17 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/885317?usp=dashboard Build final 3.8.0 release images | 19:17 |
| clarkb | Additionally the Gerrit replication tasks stuff is still ongoing | 19:18 |
| clarkb | I think my recommendation at this point is that we revert the bind mount for the task data so that when we periodically update our gerrit image and replace the gerrit contianer those files get automatically cleaned up | 19:18 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/884779?usp=dashboard Stop bind mounting replication task file location | 19:18 |
| clarkb | If we can get reviews on one or both of those then we can coordinate the moves on the server itself to ensure we're using the latest image and also cleaning up the leaked files etc | 19:19 |
| fungi | what's the impact to container restarts? | 19:19 |
| fungi | if we down/up the gerrit container, do we lose queued replication events? | 19:19 |
| clarkb | fungi: yes. This was the case until very recently when I swapped out the giteas though so we were living with that for a while already | 19:20 |
| clarkb | The tradeoff here is that having many leaked files on disk is potentially problematic when that number gets large neough. Also these bad replication tasks produce errors on gerrit startup that flood the logs | 19:21 |
| clarkb | we'd be trading better replication resiliency for better service resiliency I think | 19:21 |
| clarkb | fungi: that said having anothe rset of eyes look over the situation may produce additional ideas. The alternative I've got is the gerrit container startup script updates that try to clean up the leaked file sfor us. I don't think the script will clear all the file scurrently but having a smaller set to look at will help identify the additional ones | 19:22 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/880672 Clear leaked replication tasks at gerrit startup using a script | 19:23 |
| clarkb | I'm happy to continue down that path as well, its jus thte most risky and effort needed option | 19:24 |
| fungi | thanks, makes sense | 19:24 |
| clarkb | risky because we are automating file deletions | 19:24 |
| clarkb | for a todo here maybe fungi can take a look this week and next week we can pick an option and proceed from there? | 19:25 |
| clarkb | The other Gerrit item is disallowing implicit merges across branches in our All-Projects ACL | 19:25 |
| clarkb | I can't think of any reason to not do this and I don't recall any objections to this in prior meetings where this was discussed | 19:26 |
| fungi | yeah, i should be able to | 19:26 |
| clarkb | receive.rejectImplicitMerges is the config option to reject those when set to true | 19:26 |
| fungi | did i propose a change for that? i can't even remember now | 19:26 |
| clarkb | fungi: I don't think so since it has to be done directly in All-Project sthen simply recorded in our docs? | 19:26 |
| clarkb | there may be a chngae to do the recording bit /me looks | 19:26 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/885318 | 19:27 |
| clarkb | so ya if you have time to push that All-Projects update I think you can +A the change to record it in our docs | 19:27 |
| fungi | oh, cool | 19:28 |
| fungi | i guess someone did propose that | 19:28 |
| clarkb | that was all I had for gerrit. Anything else befor ewe move on? | 19:29 |
| clarkb | #topic Server Upgrades | 19:31 |
| clarkb | I'm not aware of any changes here since we last met | 19:31 |
| clarkb | tonyb helped push the insecure ci registry upgrade through. I may still need to delete the old server I can't recal lif I did that right now | 19:31 |
| corvus | ze04-ze06 upgraded to jammy today | 19:32 |
| clarkb | I think tonyb is looking at other things now in order to diversify the bootstrapping process as an OpenDev contributor so I'll try to look at some of the remaining stragglers myself as I have time | 19:32 |
| clarkb | corvus: excellent | 19:32 |
| tonyb | the cleanup is removing it (01) from the inventory and then infra-root deleting the 01 vm? | 19:32 |
| clarkb | tonyb: correct | 19:32 |
| tonyb | Okay | 19:33 |
| clarkb | #topic Fedora Cleanup | 19:34 |
| clarkb | tonyb: I've lost track o fwhere we were in the mirror configuration stuff. Are there changes you need reivewing on or input on diretion? | 19:34 |
| tonyb | I need to update the mirror setup witrh the new mirrorinfo variable | 19:35 |
| clarkb | tonyb: is that something where some dedicated time to work through it would be helpful? if so we can probably sort that out with newly overlapping timezones | 19:35 |
| tonyb | Yeah, that's a good idea. I understand the concept of what needs to happen but I'm in danger or overthiking it | 19:36 |
| clarkb | ok lets sync up when it isn't first thing in the morning for both of us and take it from there | 19:37 |
| tonyb | great | 19:37 |
| clarkb | #topic Quo vadis Storyboard | 19:37 |
| fungi | i think i switched a neutron deliverable repo over to inactive and updated its description to point to lp last week? openstack/networking-odl | 19:38 |
| clarkb | One thing I noticed the other day is that some projects like starlingx are still createing subprojects in storyboard. We haven't told them to stop and I'm not sure we should, but they were confused that it seems to take some time to do that creation. I think we are only creating new storyboard projects once a day | 19:38 |
| clarkb | At this point I'm not sure there is much benefit in having project-config updates trigger the storyboard job more quickly | 19:38 |
| clarkb | But it was a thing people noticed so I'm mentioning it here | 19:39 |
| fungi | there was also some discussion about sb in the #openstack-sdks channel, in particular a user was surprised to discover that an unescaped <script> tag in a story description resulted in truncating the text once displayed. fairly easy to avoid, but turned into some conversation about why getting a fix for such things implemented would be tough with out current deployment | 19:39 |
| tonyb | Can we run the create, I assume via cron, twice (#gasp) a day? | 19:39 |
| clarkb | tonyb: I believe the job that does it is infra-prod-remote-puppet-else | 19:40 |
| clarkb | which at this point should mostly just be storyboard? | 19:40 |
| clarkb | we have removed the vast majority of any remaining puppet | 19:40 |
| tonyb | Ahhh okay | 19:40 |
| clarkb | so ya we basically run that job more often or when necessary to decrease the wait time | 19:41 |
| clarkb | anything else storyboard related? | 19:42 |
| fungi | i got nothin' | 19:43 |
| clarkb | #topic Gitea Upgrades | 19:43 |
| clarkb | Gitea 1.19.4 exists and fungi has pushed an update to upgrade us | 19:43 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/887734?usp=dashboard Upgrade Gitea to 1.19.4 | 19:43 |
| fungi | seems to be fairly minor for our purposes, fixes mostly to stuff we disable anyway | 19:44 |
| clarkb | These bugfix point upgrades tend to be pretty safe and straightforward though this one has a small template update | 19:44 |
| fungi | so also probably not urgent | 19:44 |
| clarkb | The other gitea upgrade to think about is the 1.20 update. They only have RC releases so far and no changelog so also not urgent | 19:44 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/886993?usp=dashboard Begin process to upgrade gitea to 1.20 | 19:45 |
| clarkb | The 1.20 upgrade should happen after we upgrade to 1.19.latest | 19:45 |
| clarkb | fungi: I think I may be able to be around with overlap in your timezone early tomorrow morning for me later afternoon for you if we want to land that 1.19.4 chnage an dmonitor | 19:46 |
| clarkb | I'll ping you tomorrow if I manage that and we can take it from there? | 19:46 |
| fungi | yeah, sure that works | 19:46 |
| fungi | i should be at the keyboard by 1200z | 19:47 |
| clarkb | I don't expect trouble but good to have people around if necessary | 19:47 |
| clarkb | 1900 is probbaly about as eraly as I can manage :) | 19:47 |
| clarkb | though maybe my evening overlaps with 1200 I need to math that out | 19:47 |
| fungi | oh, you said early in your timezone not the other way around | 19:47 |
| clarkb | #topic Etherpad Upgrade | 19:47 |
| clarkb | fungi: ya | 19:47 |
| clarkb | After a long release drought Etherpad made a 1.9.1 release | 19:48 |
| fungi | i assume the commit we've been running on is included in that release | 19:48 |
| clarkb | At first the tagged sha didn't actually build nad I was forced to use a commit that fixed the build issues after the reelase. But I think they updated/replaced the tag and now it seems to work | 19:48 |
| fungi | aha, cool | 19:48 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/887006?usp=dashboard Etherpad 1.9.1 | 19:48 |
| clarkb | I updated that chnage to use the tag again and it did not fial | 19:49 |
| clarkb | Now there is an issue where numbered lists don't properly increment the list number values so every entry is 1. basically making it a weird bulleted list | 19:49 |
| tonyb | Going back to gitea (sorry), is there any merrit to landing the bullseye -> bookworm update in the same window? | 19:49 |
| clarkb | tonyb: no I think we should decouple those if we can. Basically swap gitea to the new debian with a fixed gitea version | 19:50 |
| tonyb | okay | 19:50 |
| clarkb | tonyb: the gitea upgrades are very low impact (we roll through them one by one and shouldn't lose any replication events and the haproxy should handle http requests too) | 19:50 |
| tonyb | Okay | 19:50 |
| clarkb | if the gitea upgrades were a bit more impactful then we should cnsider combining but they ar esuper transparent to users | 19:50 |
| clarkb | Going back to etherpad I think that this is laos not very urgent given the known list bug | 19:51 |
| clarkb | I also haven't held a node yet to interact with it which is probbaly a good idea to double check that we don't hvae any plugin interactions that will create problems for us | 19:51 |
| clarkb | Reviews welcome and I'll try to get a held node up soon | 19:52 |
| clarkb | #topic Open Discussion | 19:53 |
| clarkb | Anything else? | 19:53 |
| tonyb | Just to note that we started the bullseye to bookworm updates | 19:53 |
| fungi | yay! | 19:53 |
| tonyb | the first set of services I tried failed due to, what I think is, missing requires to get the speculative images in the buildset registry | 19:54 |
| clarkb | ya I looked at that briefly and wasn't able to understand what was missing. It looks like we have what we need | 19:54 |
| tonyb | Hopefully with some push and TZ overlap we can make solid progress | 19:55 |
| clarkb | I feel like this comes up semi regularly though and Ineed to be better about writing down what the issue was/improving my understanding | 19:55 |
| clarkb | corvus: any chance yo umight have a few minutes to look at that? | 19:55 |
| clarkb | https://zuul.opendev.org/t/openstack/build/ab79e98cdd0242649cbc50593e87dae1/log/job-output.txt#723 is the failure | 19:55 |
| corvus | yeah i'll take a look and followup in #opendev | 19:56 |
| clarkb | thank you | 19:56 |
| tonyb | Thanks | 19:57 |
| clarkb | sounds like that is everything for now. Thank you everyone! | 19:58 |
| clarkb | #endmeeting | 19:58 |
| opendevmeet | Meeting ended Tue Jul 11 19:58:51 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 19:58 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2023/infra.2023-07-11-19.01.html | 19:58 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2023/infra.2023-07-11-19.01.txt | 19:58 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2023/infra.2023-07-11-19.01.log.html | 19:58 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!