19:01:09 #startmeeting infra 19:01:10 Meeting started Tue Jan 10 19:01:09 2023 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:10 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:10 The meeting name has been set to 'infra' 19:01:12 #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/5WW6CBUKQUWVNW7C5RRLXW7PWUXDPY75/ Our Agenda 19:01:49 #topic Announcements 19:02:03 The Open Infra Foundation individual board member election is happening right now 19:02:20 if you are a member of the foundation you should've received a ballot via email. 19:03:03 Please vote if you are able to. 19:03:17 #topic Bastion Host Updates 19:03:43 ianw: I know prior to holidays you were querying people to see if anything else needed migrating from the old bastion before it was shutdown. 19:04:10 Looks like the old bastion is still up, but maybe we can shut it down soon? I didn't see anyone indicate anything needed to be moved at last poll 19:04:21 yeah i think i got a response from everyone active 19:04:55 i can shut it down, and we can consider removal after a little 19:05:00 sounds good 19:05:00 more time 19:05:07 Anything else bastion related? 19:05:30 i don't think so ... the backup bits i think might need a rebase now 19:06:07 #topic Mailman 3 19:06:34 I think we may still need to restart services to pick up the site owner update. ALso frickler discovered that local root alias is not working anyway. 19:06:54 do we want to use a different site owner value or fix the root@lists01 alias in exim? I assume we prefer fixing the alias? 19:07:01 and i have some more related changes up for review 19:07:27 including an upgrade to the latest mm3 versions from last week 19:07:31 these changes are related to fixing the vhosting of lists themselves? 19:07:33 yes 19:08:14 https://review.opendev.org/c/opendev/system-config/+/867986/ and children? Looks like they are all workinprogress right now 19:08:35 https://review.opendev.org/q/topic:mailman3+status:open is a better link. Finds a couple extra changes 19:09:01 fungi: what do you think the plan there should be? 19:09:17 just a sec, need to page a lot of this back into my head ;) 19:09:55 i think i still need to review the failures on 867987 19:10:40 i have a feeling it's probably just going to need test adjustments, but don't think i've confirmed that yet 19:11:04 the upgrade is up for discussion. the change passes at least 19:11:25 fungi: was that generated to be in sync with how the upstream docker images are handling the version upgrade too? 19:11:26 and the upgrade notes for the new versions don't indicate any extra steps needed 19:11:39 my main concern is with the dep pinning as that seems to be a major headache for everyone doing mm3 19:11:55 but once you have a good install it sounds like they handle upgrades pretty transparently 19:11:56 yes, much of that was ported from the docker image repo 19:12:14 awesome. In that case I think we can land that either before or after fixing the other issues, Probably your preference really 19:12:50 i think i'd like to dig into fixing the domain separation first 19:13:18 after we get that ironed out though, and upgrade mailman to the latest releases, i should be ready to start planning for the next site migrations 19:13:31 works for me. Maybe ping us when you're in a state where extra eyeballs could be helpful? Whether just code review or helping with debugging on held nodes 19:13:38 yep 19:13:58 migrating lists.katacontainers.io should probably be prioritized since it frees up an entire server 19:14:25 makes sense 19:14:28 but also i've been fighting oom kills of random processes on the old lists.openstack.org server so the more we move off of it the better 19:14:48 (i just moments ago killed and restarted all relevant processes on that server yet again) 19:15:04 Anything else on this topic? 19:15:14 fungi: i'm assuming you shut down/disabled the migrated processes already? 19:15:21 yes 19:15:30 figured :) 19:15:48 yeah, if i hadn't, that would be an easy way to relieve some pressure on it 19:15:52 too bad 19:17:00 #topic Quo vadis Storyboard 19:17:36 I swear I sent that followup email I said I would send but I'm not finding it now... 19:19:49 In any case there hasn't been much movement on this topic. As mentioned before I think we need to make a proposal (probably towards turning it off in a gracefaul manner if possible) based on the feedback we've gotten 19:20:11 I feel like I've got enough distractions this week that I won't get to that this week whcih means we can let it simmer for a bit longer :/ 19:20:24 Still happy to receive any and all additional feedback others may have though 19:21:10 #topic Gerrit 3.6 19:21:17 The upgrade happened and we haven't reverted 19:21:37 ianw sent email to the repo discuss list to bring up the label listing behavior that is new in 3.6 (I have't seent an responses yet) 19:22:07 I think we are at a point where we should remove 3.5 image builds, add 3.7 image builds, and update our upgrade testing to perform a 3.6 to 3.7 upgrade 19:22:20 ++ i don't see us going back to 3.5 19:22:34 sgtm 19:22:40 ianw: I'm happy to help with the wrangling of those changes if you'd like. 19:23:08 sure :) 19:23:35 i also started on 19:23:46 #link https://review.opendev.org/c/openstack/project-config/+/867931 19:24:06 to update copyConditions in our configs 19:24:25 this is one of potentially several similar types of changes we'll need to make prior to upgrading to 3.7 19:24:47 we have some in our All-Projects which I think will need manual intervention, but i'd like to put some of it into the gate testing first 19:25:05 ++ there is less blast radius outside of all-projects 19:25:11 some work started on that @ 19:25:13 #link https://review.opendev.org/c/opendev/system-config/+/868054 19:26:28 anything else gerrit upgrade related before we jump to the next thing? 19:26:48 nope, more to discuss in the future as we work on submit requirements etc :) 19:27:06 #topic Nox 19:27:18 This is the evolution of the Tox v4 topic we had previously 19:27:38 The way the tox v4 situation has panned out has led to us needing to deal with new issues with every tox v4 release 19:27:57 Eventually some of us got tired of that and started looking at alternatives: in particular nox. 19:28:25 Nox is a tox alternative that uses a simple python file to specify its targets (it calls them sessions) and relies on standard tools and processes for installing things 19:28:49 This to me is the main feature with nox that has me wanting to switch. Tox has reimplemented libraries for certain peps and it led to weird behavior 19:29:01 Sticking to standard tools ensures we don't need todebug oddness like that 19:29:16 Long story short Zuul is already largely moved to nox and I think OpenDev should do the same 19:29:22 #link https://review.opendev.org/c/opendev/bindep/+/868004 An example conversion 19:29:33 I've got this change up for bindep that serves as an example too 19:29:57 All of the zuul-jobs tooling for artifact retrieval and siblings support should be there now too 19:30:28 the only counterargument against nox i've heard so far is that some folks prefer the declarative nature of tox.ini files over turing-complete python scripts for configuration, but tox.ini is pretty far down the road to being a programming language of its own at this point too 19:30:32 And as a user I've been relly happy using nox. I don't think I would suggest the move if I wasn't. The syntax for command is a little different but its not so different and it is quick to pick up 19:30:56 fungi: ya and I would argue that this is tox's greatest flw right now because they keep changing the semantic meaning of that declarativeconfig 19:31:00 making it effectively useless 19:32:16 i'm good with switching bindep over to see how it goes (and zuul/nodepool are already basically there) 19:32:30 Anyway, I intend on pushing changes to move active opendev repos to nox. But I don't want to get too far down that path until we've got general consensus this isn't a problem 19:33:03 So please do take a look at the bindep change, fetch that change locally and run nox yourselves and see what you think. THen let me know :) 19:33:42 convincing large and complex projects like openstack to switch away from tox is probably not a windmill i feel like tilting at (especially since they're stuck using tox on older branches for years to come), but i think for opendev's projects it makes sense and shouldn't be too big of a lift 19:34:15 if it were me, i'd pin those older branches to tox v3 then nox on master 19:34:43 yeah, they're pinning the older branches anyway, as it turns out 19:34:53 but stuff like release tooling will have to support both 19:35:21 depending on where I get with zuul and opedev I might take a look at the release tooling, but I agree its a much stronger preexisting force you run up against there 19:35:33 i'm very happy with nox and i like the idea of us using it for opendev projects 19:36:18 woot my first couple of bits of feedback :) 19:36:24 I think we can move on to the next thing 19:36:31 #topic Service Coordinator Election 19:36:41 It has been about 5 months since we last elected our service coordinator 19:37:12 6 months after our previous nomination period is January 31, 2023 - February 14, 2023 19:37:35 I'd like to propose we open that two week period for nominations for the next service coordinator and take it from there 19:38:18 If there are no objections to this timeframe I'll send email to service-discuss making it official soon 19:38:50 as I've said before I'm happy for someone else to do the work too :) 19:39:38 #topic Linaro Cloud Move and Cleanup 19:39:54 ianw: want to fill us in on the recent updates for the linaro cloud move 19:40:50 yep sure, we have credentials 19:41:18 there are changes to enable the cloud for CI @ 19:41:20 #link https://review.opendev.org/q/topic:linaro-2022 19:42:08 one issue is that it doesn't have much disk. so much so that our 800gb volume for the builder /opt would leave nothing else 19:42:27 so i asked osuosl if they could give us compute quota to run a nodepool builder there, which they kindly provided 19:43:16 i have the host up, storage attached, etc, and some changes out there to add it to system-config & dns 19:43:44 so i think i'd like to add that, and get it working and building things 19:43:53 then disable nb03 19:44:04 then cut testing over to new cloud 19:44:06 sounds great 19:44:11 then disable old cloud 19:44:16 oh I just approved the first two chagnes that shift over to the new cloud 19:44:21 should I WIP them? 19:44:38 I guess its only the second that is a problem. I'll WIP it 19:44:56 it's ok, just we will want to monitor to make sure it is actually working 19:45:12 heh ok switched back to +A 19:45:48 it can all really happen in any order, depending on our attention span to monitor the different bits :) 19:45:50 thank you for picking this up. I think it will make ed happy and in theory these newer test nodes should be very quick 19:46:46 Anything else related to the cloud moves? 19:46:50 yep, good to try and work with the providers to keep everyone happy :) 19:47:33 no more on that, will keep working at it 19:47:53 #topic Upgrading Bionic Servers 19:47:58 #link https://etherpad.opendev.org/p/opendev-bionic-server-upgrades 19:48:09 This is going to take on a bit more urgency due to bionic EOL in april 19:48:28 I'm personally hoping to be able to really start picking things off of that list starting next week. 19:48:54 If anyone else ends up helping out please stick a note in the etherpad so that we don't step on each other. Also, help is appreciated :) 19:50:06 #topic Open Discussion 19:50:17 #link https://review.opendev.org/c/opendev/system-config/+/866781 Retire Mordred as infra root 19:50:45 one thing frickler pointed out was the increasing storage numbers 19:50:46 Can other infra-root review this change? mordred pushed it himself so no concerns there. Unfortunately I think the time has come for this little bit of cleanup 19:50:55 #link https://grafana.opendev.org/d/9871b26303/afs?orgId=1&from=now-6M&to=now 19:51:28 yes, not critical yet, but worth watching 19:51:28 looks like ubuntu and ubuntu.ports have grown a bit 19:51:30 I wonder why 19:51:40 since they should be pretty fixed after the jammy release last april 19:51:49 which, apropos the prior topic, one win we could have there is purging everything xenial related 19:52:11 the open euler mirror also doubled in size 19:52:23 so ya we probably needto make another pruning pass 19:52:33 i'd like to land the nodepool openstack statemachine change soon. there is a non-zero chance of real-world performance problems. 19:52:40 once we get all the nodepool tox/nox/k8s changes merged, how about i tag a 8.0.1 release, then if there is a problem we can revert opendev to it quickly? 19:52:50 corvus: ++ to tagging first 19:53:01 clarkb: yeah, i thought we got a new openeuler release but purged the old one 19:53:55 debian security almost doubled too 19:54:10 definitely a good idea to dig into that. Thank you for calling it out ianw and frickler 19:54:39 cool, i'll make sure the tag happens and will keep folks updated. let me know if there's any other concerns, preparation, or safeguards we'd like for opendev. i figured i would manually upgrade launchers slowly to keep an eye on things. 19:55:05 corvus: you'll need to put them in the emergency file to do them manually as our hourly jobs are really good at updating them otherwise 19:55:19 (remember this is on bridge01.opendev.org now) 19:55:40 clarkb: ack. and i've already retrained to use bridge.opendev.org :) 19:56:00 and thanks, i forgot that could happen hourly and not nightly 19:56:24 so: tag, emergency file, manual upgrade. 19:56:24 #link https://review.opendev.org/c/opendev/system-config/+/848796/3 19:56:48 ^ is the openeuler mirror drop -- i knew i'd seen something about it. we should probably do that at least 19:57:00 ++ I'll review it now 19:58:11 (incidentally, tagging 8.0.1 will be a nice double check of any release job nox updates) 19:58:26 if we have to do an 8.0.2 it's no big deal 19:58:53 numbers are free 19:58:59 we won't run out 19:59:09 And we are just about at time 19:59:39 thank you everyone. Apologies for not communicating better last week. I ened up sick and wasn't in a place to computer. I'm still sort of on the tail end of that too but much more with it now :) 19:59:53 We'll be back next week. Happy new year! 19:59:57 we didn't have anything worth covering last week anyway 20:00:00 thanks clarkb! 20:00:01 #endmeeting