19:01:13 #startmeeting infra 19:01:13 Meeting started Tue Mar 21 19:01:13 2023 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:13 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:01:13 The meeting name has been set to 'infra' 19:01:20 #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/QH5SYKQ7JY6DF65R2NQQ7RTLONFQQIXR/ Our Agenda 19:01:28 #topic Announcements 19:01:48 OpenStack is doing a release this week (should happen early tomorrow relative to me) 19:02:00 Please refrain from updating base jobs or restarting zuul if we can avoid it 19:02:32 though we do have a release job configuration change to merge later today in support of that 19:02:39 is that the gpg key update? 19:02:57 #link https://review.opendev.org/877552 Temporarily remove release docs semaphores 19:03:03 oh that one 19:03:12 i'm un-wipping it now 19:03:16 we don't need the gpg key update to be clear? that happens after the release? 19:03:37 after, yes. i've set that up for monday 19:03:43 cool 19:04:35 And next week is the ptg. Similar story around avoiding making changes but to a different set of tools. Etherpad and meetpad in particular for the PTG 19:05:18 I suspect at least some of us will be participating in he PTG as well so we may want to skip next week's meeting. But I'll wait for more of the schedule to congeal as I'm not sure yet if it poses a problem 19:06:04 our usual meeting time is during the break 19:06:11 just to be clear 19:06:16 yes, but if you've spent 4 hours in meetings ou may need a break 19:06:23 I know I would :) 19:06:24 yes, i likely will do ;) 19:06:53 #topic Topics 19:07:05 #topic Docker Free Team Organization Shutdown 19:07:24 last week docker announced that the free team organization setup we rely on for our docker images is going away on April 14 19:07:50 We had a meeting on thursday to discuss options and create a plan to move forward. Where we ended up was to move our container images to quay.io 19:07:58 #link https://etherpad.opendev.org/p/MJTzrNTDMFyEUxi1ReSo Notes on the situation and planning 19:08:11 We've taken some decent notes through the whole thing which can be found there 19:08:18 #link https://quay.io/organization/opendevorg Will become the new home for opendev images 19:08:40 I've created a new org on quay and I think all infra-root but frickler have been added? Happy to add frickler when notified of an account name 19:08:55 There are also some job/role updates that we need to make this move happen 19:09:00 #link https://review.opendev.org/c/zuul/zuul-jobs/+/877834 Role to precreate public images on quay.io 19:09:31 If you simply push to quay.io you end up with a private repo by default. The way around this is to precreate the repo which this role does (or you can switch it to public after pushing with the private default) 19:09:40 #link https://review.opendev.org/c/zuul/zuul-jobs/+/838919/ Updates to container roles to make them replace docker roles for quay.io 19:09:53 and this stack of changes updates the generic container roles to add functionality that was in he docker roles that we need 19:10:09 corvus: and to be clear the reason we can't use the docker roles is they have hardcoded docker hub as the only destination? 19:10:25 corvus: after reviewing this stack I wondered why we couldn't just tell the docker roles to talk somewhere else 19:10:29 yes, they are designed specifically for docker hub 19:10:37 thanks, that helps 19:10:47 mostly the promote role uses docker specific apis 19:11:07 on the whole I think we've got a reasonable plan and there is progress. Reviews and help are welcome 19:11:27 On top of those changes we'll need to update our jobs themselves to make use of the updated roles 19:11:34 the other roles are supposed to be pretty similar, just with a few minor assumptions removed and defaults changed 19:11:35 however 19:11:37 the build-container-image role does not appear to have any support for multi-arch builds 19:12:10 is there anyone interested in multi-arch support who is willing to look into bring build-container-image up to par? 19:12:36 corvus: the main use of that today in our world is nodepool-builder right? 19:12:46 I think opendev and zuul don't multiarch anywhere else. 19:13:05 i think so; but is there base image support to facilitate that? 19:13:07 which role are we building our multi-arch containers with currently? 19:13:18 fungi: build-docker-image 19:13:32 corvus: oh yes, the base images are also built for arm64 and x86_64 19:13:59 I should be able to take a look at that if no one else is interested. I've poked at our base images and the multi arch stuff in the past 19:14:02 i'll have to context switch it all back in, reviewing those changes should help with that, but i can probably take a look at it 19:14:18 ianw: oh cool. and ya I would have to context swithc it in. buildx and manifests and all that 19:14:21 okay, so we'd need to port the multi-arch bits from build-docker-image to build-container-image sounds like 19:14:23 ++ i'm sure we can figure something out 19:14:38 thanks; i'm willing to pitch in to help, but it's too much for me alone right now. 19:15:02 thanks for calling it out, i wouldn't have thought to check 19:15:03 fwiw I think some of the reason this was missing is buildah didn't support multiarch buidls until more recently 19:15:23 however, you can run docker with the container roles too (and in fact I think that is what opendev at least should do for max compatibility for now) 19:15:34 I think changing to buildah is a possibility but something we shouldn't worry about now 19:15:35 buildah is a non-docker-specific buildx replacement? 19:15:38 agreed re using "docker" with the "container" roles 19:15:57 fungi: buildah is the podman image builder. docker does both the container runtime and image building via one interface but podman and buildah split it out 19:16:32 they both use buildkit as the underlying implementation for the new stuff though, so convergence is hopefully possible 19:17:19 ok cool ianw I'll probably try to look at this late today/early tomorrow for me 19:17:36 anything else related to the container registry move? 19:17:56 i've completed work on a redirection system for zuul 19:18:12 that was an awesome idea, btw 19:18:17 so i will propose that zuul use "registry.zuul-ci.org/..." as the canonical locations for images 19:18:59 it's a pretty easy option for any other domains that want to do something similar 19:19:08 ya we should probably do similar for opendev.org. 19:19:51 it's probably worth advertising it somehow? 19:20:03 ianw: advertising that redirects work? 19:20:09 ianw: can you elaborate? 19:20:17 oh that it's a thing that other projects can do? 19:20:31 like registry.kolla could be a thing if they wanted? 19:20:32 like a blog post about the design or something? 19:20:34 yeah, particularly like openstack projects might want to consolidate, etc. 19:21:01 okay i can write something up 19:21:13 maybe send it to the service-discuss list... 19:21:15 yes, i already saw one similar blog post rise to the top of HN and i think they got it wrong too (might not have worked with podman) 19:21:21 and then copy or link other lists to that? 19:21:34 i suppose i could write an actual blog post 19:21:34 for kolla specifically, it sounded like they were already deploying to multiple registries and were just planning to drop dockerhub, but there are potentally projects even outside opendev that could benefit from seeing that the idea is simple and works 19:22:25 s/deploying/publishing/ 19:22:40 ya its not something people may realize is possible so getting the word out could be useful 19:22:49 yep, just think it would to get the word out that it's a thing you can do, and here's an example 19:23:27 i think there's 2 audiences: how to do this in general; and how to do this in opendev 19:23:33 for the latter, a pointer to code/changes might be in order 19:23:58 #action corvus write up blog post/email/something to share registry redirect idea/configuration 19:24:18 ++ 19:24:25 probably solveable by just linking the opendev examples in the general writeup, but yeah i agree it may be two audiences for one post even 19:24:35 if we're happy with the idea for opendev, i can try to make a clean series of changes for that 19:24:50 (since the git history for zuul is not clear due to experimentation) 19:24:51 corvus: I think I like the idea since it should reduce some over the necessary changes if we do this again 19:25:13 k. expect that later this week :) 19:25:18 yes, also if we're going that route we should do it before we switch registries, in order to save ourselves additional patches to review 19:25:47 exactly this is the best opportunity for doing it 19:26:36 * fungi prepares for docker inc to break the official docker client's ability to follow redirects 19:26:55 ok lets move on as we're almost half out of time and only one topic in. THank you everyone who has helped push this along. Its going to be a bit of work and won't happen overnight but we're making progress 19:27:05 yes, thanks! 19:27:05 #topic Bastion Host Updates 19:27:25 I mostly wanted to call out that fungi had to boot the old bastion briefly to grab some gpg data off of it. Now is a great time to brainstorm if anything else needs moving 19:27:45 I suspect that given how long its been and we've only run into this one issue that there isn't. But the sooner we address anything else missing the better 19:28:08 yeah, it's not still booted, was only up for a few minutes, but we should be very sure before we delete the server so that we don't need to restore from backups later 19:28:30 ++ the other thing is, if we do want to go down the path of distributed backups of this (reviews welcome :) then we need a list of things to backup as well. so thinking about what to grab can be codified into that list 19:28:41 ianw: that is a great idea 19:28:45 /root ;) 19:28:55 and /etc/ansible 19:29:15 fungi: did you say it was ~gb though? 19:29:33 i don't think it needs to be. there was a metric ton of ansible tempfiles in there 19:29:50 as long as nobody puts anything in /root that isn't critical to backup, that can work, but i also thing huge backups will be hard to deal with 19:29:57 i just copied it all because i didn't want to have to wade through it while i was in a hurry to do key generation stuff 19:30:39 hrm I bet the new server has tempfiles for ansible too though 19:30:44 because ansible runnign as root 19:30:49 something to look at I guess 19:30:49 yeah. i was being glib. specifically /root/signing.gnupg 19:30:56 and /root/certs 19:31:08 and /root/passwords 19:31:10 fair enough, i think i didn't copy the whole thing as sort of a counter-point to that, trying to just keep the important bits 19:31:57 the idea with the distributed backups is all that will go in a ansible variable; it could be either checked in, or in /etc/ansible 19:32:11 a copy of the /root from old bridge can be found in /root/old-bridge-root-homedir.tar.xz for now, until we decide to toss it 19:32:36 (i was thinking /etc/ansible, to make it less inspectable, but open to ideas) 19:32:46 ianw: as far as landing the backups goes we're mostly waiting on reviews? 19:33:16 i remember saying i was going to review those changes and then not doing so 19:34:22 anything else bridge related? 19:34:36 not this week 19:34:50 #topic Mailman 3 19:35:01 fungi: anything new with mailman 3 domain management stuff? 19:35:33 negatory, sorry 19:35:39 #topic Gerrit Updates 19:36:03 ianw: looks like the last submit-requirements and copyConditions change (the one for testing it in system-config-run-review-*) landed 19:36:20 ianw: the conversion process should be largely done at this point? 19:36:33 yes; i've spent some time updating 19:36:36 #link https://etherpad.opendev.org/p/gerrit-upgrade-3.7 19:36:44 for that, and other issues 19:37:13 we just merged a change to rebuild images from 3.7-stable, so i plan to bring up a test host with that 19:37:13 ya the other thing was the 3.7.1 bug 19:37:44 that could likely be our production image. if there's another release before our upgrade, we can rebuild at that 19:37:49 sounds good 19:38:05 in particular the bug we rebuilt to grab a fix for has to do with navigating related chnges in the ui 19:38:15 something I know I do so good to have fixed 19:38:27 it would be worth going through the "DONE" tags on that page just to double check people agree with my conclusions about various issues 19:38:49 I'll try to review that today 19:39:14 this upgrade is very similar to prior upgrades except that we must take a logner downtime for offline reindexing 19:39:24 for whatever reason they apparently didn't support online reindexing for this one 19:40:41 #topic Project renames 19:40:49 related to the gerrit upgrade is the plan for project renames 19:41:00 we've decided to do both one after the other on April 6 starting at 22:00 UTC 19:41:43 fungi: other than better understanding our infra-project-manage-projects job sequencing when we've got old state in the earlier chagnes the other open question I've got is if we need to have any more care around openstack gettingthe xstatic library 19:41:44 fwiw, i did an online reindex recently trying to run down a problem that turned out to be unrelated to indexing, but it only took a few hours to complete 19:42:25 fungi: should we maybe bring that up in the tc meeting tomorrow? 19:42:36 xstatic-angular-fileupload? 19:42:42 "are you ok with this and does it create any conflict with moin moin?" 19:42:48 fungi: yes 19:44:16 looks like it's got some shared authorship, yeah 19:44:23 worth bringing up 19:44:39 homepage link for it on pypi goes to https://github.com/danialfarid/ng-file-upload 19:44:57 ok I'll try to be there tomorrow morning (shouldn't be an issue other than being a few minutes late for a school run) 19:45:05 last touched a little over 4 years ago there 19:46:01 #topic Updating old servers 19:46:38 The docker situation has distracted me from this. Which is mostly fine since I was going to hold off on gitea and etherpad work until later (can finalize gitea situation end of this week early next week and do etherpad after the ptg) 19:46:52 ianw: was there anything new with the nameserver planning? 19:47:34 no, i've also been distracted. there are some preparatory changes to merge, then i need to finish the checklist for switching and we can review. parallel i can start new servers 19:48:01 ok ping me if I can help. I think I've reviewed changes though 19:48:17 #topic AFS disk utilization 19:48:40 Nothing really new here. I've left it on the agenda as a reminder it is something we need to keep an eye on. We do seem to have stablized near 10% free on the main server 19:48:59 and eventually fedora 36 should get cleaned up relieving some pressure just in time for new debian :) 19:49:23 yep i'll try to get 37 builds going soon 19:50:09 #topic Gitea 1.19 19:50:33 Over the weekend gitea released 1.19.0. I had already pushed a chagne to sanity check 1.19.0-rc1 and there were minimal updates necessary to get 1.19.0 going 19:50:43 that said I think we should hold off until we've stabilized the gitea replacements 19:50:55 which is also planned for after the openstack release 19:51:01 #link https://review.opendev.org/c/opendev/system-config/+/877541 Upgrade opendev.org to 1.19.0 19:51:28 this release adds an actions runner system which is disabled by default. I have explicitly disabled it in this change to avoid that default changing on us later since we've got zuul 19:52:06 Otherwise the update seems pretty straightforward. THe commit message in that change tries to highlight all of the breaking changes from the release notes with my own notes on how/why things do or don't affect us 19:52:34 interestingly there were html template updates between 1.19.0-rc1 and 1.19.0. I'm not sure that has happened before 19:52:43 a good reminder for us to double check that between releases though 19:53:24 #topic Quo vadis Storyboard 19:53:57 last week frickler pointed out that there may be storyboard discussions at the ptg next week. In #opendev I mentioned that I think we should continue to encourage any projects migrating off of storyboard to collaborate to avoid duplicate effort on tooling/process 19:54:12 so i guess keep an ear to the ground for projects bringing this up at the ptg and encourage them to weigh in on the opendev discussion? 19:54:32 fungi: yup and/or maybe push them towards each other and have shared discussion at the ptg too? 19:54:41 sounds good 19:54:42 I'm also happy to jump into those sessions if people notify me 19:55:17 yeah, one up-side to vptg, easy to drop from one call and jump into another (faster than running across the hall, much less to an adjacent building) 19:55:33 I also mentioend that I think longer term we (opendev) should think about sunsetting storyboard due to our lack of maintenance for the server and service 19:55:56 one good idea that came out of me mentioning that is we could try to make some sort of read only copy of the content and host that hopefully with much simpler hosting demands 19:56:22 but I also poitned out that I'm not the lone decision maker and welcomed feedback on that idea. Consider this an official reuqest for feedback from this group 19:57:04 #topic Open Discussion 19:57:10 We've got ~3 minutes for anything else 19:57:26 having a search engine friendly view of the content instead of just browser-side rendering was, ironically, one of the wishlist items for further sb development 19:57:43 #link https://review.opendev.org/c/opendev/system-config/+/877735 19:57:50 is one to split out the ssh server setup 19:58:10 from base. i'd like to try running that against the linaro cloud to make sure the root login auth, etc. are setup right 19:58:37 we can also potentially extend that to inmotion 19:58:54 but I Thin kstarting with linaro is great 19:58:56 the prod job we added is working; but i did have to manually allow the zuul login 19:59:12 (which is ok, that's setup by the launch node usually, which obviously didn't run) 19:59:26 ianw: its not zuul but bridge right? 19:59:31 I mean zuul via bridge 19:59:33 not direct 20:00:17 yeah, basically allowing root logins from bridge. but that config will be deployed with the ssh role, so if we change bridge again it will be ok 20:00:56 speaking of linaro, i think we had a good justification to add them to the donor logos on https://opendev.org/ 20:01:05 anybody know who we should reach out to about that? 20:01:09 kevinz? 20:01:16 fungi: yes kevinz is who I would start with 20:01:19 k 20:01:35 and we are over time 20:01:40 thanks clarkb 20:01:42 ! 20:01:44 or perhaps ... sorry i'm blanking on the name ... but the person we sent the usage blurb to 20:01:51 ianw: good call 20:01:56 thank you everyone! feel free to continue discussion in #opendev or on the mailing list. But I don't need to keep you in here any longer 20:01:56 works on arm person 20:02:11 #endmeeting