19:01:08 #startmeeting infra 19:01:08 Meeting started Tue Jan 17 19:01:08 2023 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:08 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:08 The meeting name has been set to 'infra' 19:01:15 #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/MB3XWSVFXEROBXFXU5V3DX3D3QVMYLQR/ Our Agenda 19:01:58 #topic Announcements 19:02:14 I didn't have anything to announce. Did anyone else? 19:02:46 nothing particularly exciting, no 19:03:16 #topic Topics 19:03:22 #topic Bastion Host Updates 19:03:47 I have a small update for ansible installation https://review.opendev.org/c/opendev/system-config/+/843330 19:04:23 Old bridge is still up. But I expect we'll shut it down real soon now 19:05:26 Other than that I think new bridge has been happy. Was there anything else to mention on this topic? Probably drop it from the weekly agenda once old bridge is shutdown 19:05:59 (sorry bouncer issues) 19:06:22 ianw: anything to add before we move onto the next item? 19:06:41 not really, there's still some things with some backup changes, and possibly some work on the parallel running 19:06:56 i haven't looked at either lately though 19:06:58 oh right. I've been bad and mean to do reviews of those changes but keep getting distracted 19:07:06 * clarkb makes a note on the todo list and maybe that will help 19:07:53 yeah, i had some down time over the holiday weekend and tried to dig into older system-config changes which have been growing stale 19:08:27 #topic Mailman 3 19:08:28 it would be good to get the parallel stuff finished 19:08:33 ++ 19:08:57 no progress on the mm3 front since last week's meeting 19:08:59 fungi: lsat week I said feel free to reach out if we can help with the mailman 3 stuff. You've got changes but they are WIP'd. Anything new? 19:09:03 ah that answers that. 19:09:11 Anything we can do to help move it along? 19:09:27 but year end/beginning paperwork is mostly wrapped up now, so i should have more time to get back to it 19:10:08 sound good. 19:10:26 not mailman3, but on mailman2 ... 19:10:37 it's currently in emergency with some custom apache rules 19:10:57 oh, right, that's one thing that sucked up a bit of our collective time last week 19:11:13 #link https://review.opendev.org/c/opendev/system-config/+/869779 19:11:28 I'll review that change after the meeting 19:11:34 thanks! 19:11:40 but I guess we can remove the host from emergency once that lands? 19:11:47 should be able to, yes 19:11:48 would allow us to codify those rules 19:12:01 we'd have to make sure we define the current rules in the secrets for the host 19:12:04 excellent, ya I'll get on that as soon as the meeting is done 19:12:29 although, i'm not 100% sure if that issue is ongoing 19:13:25 anyway, can double check and we can discuss outside meeting 19:14:22 #topic Quo vadis Storyboard 19:14:40 This is something i Just haven't had time to get to with all of the other paperwork and distractions. I'm hoping to be able to get to it at some point though 19:15:22 #topic Gerrit 3.6 19:15:29 #link https://review.opendev.org/c/opendev/system-config/+/870118 Fix Gerrit 3.6 image build 19:15:58 Gerrit updated the bazel version they use which is stricter about checking some flogger logging code. This has resulted in one of the plugins we build erroring on the bazel build 19:16:19 The change above has us consume the latest version of that plugin which fixes the error. Looking at the difference between 3.6.3 and latest of that plugin I expect this to be safe 19:16:42 Then separately I've got a stack of changes to drop gerrit 3.5 image builds and add 3.7 image builds and test all that 19:16:49 #link https://review.opendev.org/c/opendev/system-config/+/869763 beginning of stack to remove 3.5 images and add 3.7 and associated testing. 19:17:19 ++ will review 19:17:50 probably the most interesting bit of all that is the modifications to the upgrade testing to make the upgrade functional 19:18:26 converts Verified to a submit requirement dropping the function specification (the docs actually say we should set function to noop but they also say function is deprecated so I decided to try just not setting it and that appeas to work) 19:18:48 but ya its straightforward to start, the real fun will be as we test 3.7 more and figure out what all in our acls needs updating 19:19:02 ++ 19:19:04 ... on that 19:19:32 as we found out late last week, we may have committed some bad ACL's and not noticed 19:19:35 full details are at 19:19:37 #link https://etherpad.opendev.org/p/760YNeM5OEFS1hlr7bE5 19:20:11 i think maybe if fungi could review that, i can take on doing an audit of where we're at, following the checklist in there, at some quiet time 19:21:10 ianw: my only other piece of feedback is that afte rremoving the cache file maybe we want to manually run the tool against a specific project or three just to be sure we don't have unexpected behavior with the cache file out of the way before we do it against everything 19:21:12 otherwise lgtm 19:21:59 i'll take a look, but may not have time until tomorrow 19:22:52 good idea, actually it's probably better to do something like loop the project list and keep each in a separate log file. i can make an update to the procedure 19:23:31 i think manage-projects works by getting the list of all projects, but then filtering it by any command-line parameters, iirc from code inspection last week 19:23:40 yes that sounds right 19:25:14 Sounds like that is all for Gerrit 19:25:18 #topic Linaro Cloud Move 19:25:39 The new nb04 arm64 builder is up and running in osuosl and I think ianw mentioned the new linaro cloud is running jobs? 19:25:55 Ed emailed earlier today asking for an update. YOu can update us then copy paste into an email :) 19:26:17 i'm not sure on that -- a node did get allocated, but the wrong flavor :) so there is some communication happening 19:26:50 but yeah, nb04 has been my main focus and that seems to be working. so i think we can get rid of nb03 to have one less thing in linaro 19:27:05 but now i need to debug the node launches in the new cloud 19:27:26 ianw: one easy thing might be to rename the flavor if we can 19:27:48 the other idea I had was maybe we can set max-servers in the old linaro cloud to zero at this point? 19:28:09 oh that should be fixed -- by dropping the min-ram in the config, it makes it an exact match on the flavor name 19:28:17 ah 19:28:31 but something is up with launching the nodes i'm not sure about yet, it seems, there's nothing running 19:28:35 I see 19:29:10 probably a good idea to give ed that update. Basically we're close and shutting down resources in the old cloud just have to figure out new boots in the new cloud? 19:29:15 i'll have to dig through the logs. i got a bit sidetracked because there's a ton of extra messages in there from a leaking node issue, that i had a fix for but now is in merge conflict with the new statemachine stuff :) 19:29:32 yep, i will respond, sorry haven't even opened emails yet :) 19:29:42 thanks! and thank you for pushing this along 19:30:04 no worries, getting closer! always interesting finding the things we didn't expect 19:30:12 #topic Upgrading Old Servers 19:30:44 As mentioned last week I didn't have any news and didn't expect to have news today. I don't. That said I'm hopeful if I can burn through some of my backlog this week I might actually get to booting a new server before our next meeting 19:30:57 #link https://etherpad.opendev.org/p/opendev-bionic-server-upgrades Notes 19:31:08 Please add your name next to any hosts you plan on dealing with 19:31:16 to avoid stepping on toes as we work through this 19:31:35 #topic Fips job updates 19:31:48 ade_lee added this to the agenda after I sent it it out but we've got plenty of time so I think we can dig in 19:32:13 there are two outstanding changes to make this work. The one ot zuul-jobs is the one I'm concerned about since it makes a backward incompatible change. Does anyone know if that has been communicated like fungi requested? 19:32:26 https://review.opendev.org/c/zuul/zuul-jobs/+/866881 that change 19:32:51 then the other question I had is if the second change can land at any time or if it needs to wait for ^ 19:32:55 https://review.opendev.org/c/openstack/project-config/+/867112 19:33:17 clarkb, it hasn't yet - but I can send it out -- I wanted to get some +2's first 19:33:27 in case there were any other changes needed 19:33:49 ade_lee: gotcha, in that case I can try to review the zuul-jobs change today and then you can send email based on that I Guess 19:34:01 clarkb, as far as I know only one team is using these jobs (and that is tripleo which is setting the boolean true in any case) 19:34:04 ade_lee: is the project-config chang eeffecitvely a noop until the zuul-jobs change lands? 19:34:06 i think the project-config change can probably land at any time since it's just supplying additional context to the zuul-jobs parent? 19:34:26 but i didn't review it with that possibility in mind 19:34:35 yeah I think so 19:34:45 ade_lee: zuul-jobs is a bit of a global zuul library. What this means is within opendev there may be only one team but we have to be mindful of other potential users 19:34:52 I already did talk with the tripleo folks 19:35:01 ack understood 19:35:37 clarkb, if you're good with the change, I'll send it out once I get a +2 19:35:55 ok I'll try to take a look today (it won't happen immediately after the meeting as I need to review themailman change then find lunch) 19:36:04 #topic Gitea Upgrades 19:36:07 cool thanks 19:36:12 Another new topic since we have time. 19:36:14 fungi: i we don't really care if the extant parent job is passed the secret 19:36:25 i.e. if we merge that project config change now? 19:36:28 ianw: right, it doesn't currently do anything with it 19:36:45 #link https://review.opendev.org/c/opendev/system-config/+/870850 Upgrade Gitea to 1.17.4 and its child upgrades us to 1.18.1 19:37:23 Over the holidays there were a couple of gitea updates that I meant to upgrade us to. I think the 1.17.4 upgrade is very straightforward and quite safe. The 1.18.1 upgrade form 1.17.4 is mroe involved. THere are template updates and golang is bumped to 1.19 and there are some config changes 19:37:49 anyway reviews welcome and I think we can hold a 1.18.1 host for manual interaction too just to be extra sure (I haven't gotten that far yet) 19:38:36 I wanted zuul to tell me if we got anything catastrophically wrong before I worried about holding stuff 19:39:26 ++ will review 19:39:26 #topic Open Discussion 19:39:31 That was all I had on the agenda 19:39:34 Anything else? 19:39:54 i got nuthin' 19:40:07 apropos nothing, but i was wondering if maybe we should do something like apache redirect opendev.org / to some sort of fancier static page 19:40:25 we can also have gitea serve it via the template file pretty easily 19:40:35 the upside to that is it helps people find repos and orgs in gitea more easily 19:40:44 I want ot say fungi has been noolding on that sort of change 19:40:58 i still feel like that kind of goes from 0 - 100mph on changes via gerrit and gating really quickly 19:41:25 "that" being the current page we have there 19:41:43 ya I think the content can certainly use editing. But I think having gitea be navigable from the content is a good thing (and using the gitea templates gets us that integration for free ish) 19:41:59 yeah, for now the change i have up to add donor logos takes the current assets directory approach 19:42:19 what became of the afs disk usage topic? do we want to add it to the agenda to at least watch it regularly? 19:42:39 fungi: oh, i feel like i've missed that, is it just one change? 19:42:43 the last time i talked about shadowing the main page in apache, one of the objections was that we need to be careful any sub-paths it incorporates don't shadow git namespaces 19:43:06 frickler: we did purge some openeuler stuff, but it didn't make a big difference 19:43:09 frickler: sure, can you add it? 19:43:19 the openeuler cleanup would've been about 100GB iirc 19:43:24 #link https://review.opendev.org/869091 Feature our cloud donors on opendev.org 19:43:26 ianw: ^ 19:43:31 will look, thanks 19:44:17 ianw: and i too am interested in some reworking of the page content at the very least 19:44:32 oh, I missed the openeuler cleanup. with that we're at least well below 90% again 19:44:38 ianw: like some of the faq we have up there is probably no longer necessary now that we've been around for a while 19:45:02 frickler: I do think more cleanup would be good especially to understand the growth in some of the volumes 19:45:16 but openeuler cleanup was a quick bandaid since the change was already written 19:45:35 i think i mentioned it last week, but i feel like purging xenial nodes, and everything related to it in various mirror volumes would be a help 19:45:43 ya that would make a good dent 19:45:48 absolutely 19:46:02 but also doesn't explain why ubuntu has grown so much over the last 6 monmths 19:46:16 (which may also expose somethign inefficient we are doing) 19:47:27 we could make the mirror scripts dump some sort of du log 19:47:46 although i'm not sure how long that would take as it stats everything 19:48:03 ok, I'll add the topic to the agenda with those ideas listed 19:50:23 Sounds like that may be it for today? Thank you for your time and your help running OpenDev 19:50:38 Oh! I've just remembered one other item 19:51:03 the service coordinator election. I plan to send email to service-discuss making january 31 - february 14 the official time frame for nominations since no one objected 19:51:19 I'll try yo get that out after lunch today 19:51:27 thanks! 19:51:38 #endmeeting