| clarkb | meeting time | 19:00 |
|---|---|---|
| clarkb | #startmeeting infra | 19:01 |
| opendevmeet | Meeting started Tue Mar 21 19:01:13 2023 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| opendevmeet | The meeting name has been set to 'infra' | 19:01 |
| clarkb | #link https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/QH5SYKQ7JY6DF65R2NQQ7RTLONFQQIXR/ Our Agenda | 19:01 |
| clarkb | #topic Announcements | 19:01 |
| clarkb | OpenStack is doing a release this week (should happen early tomorrow relative to me) | 19:01 |
| clarkb | Please refrain from updating base jobs or restarting zuul if we can avoid it | 19:02 |
| fungi | though we do have a release job configuration change to merge later today in support of that | 19:02 |
| clarkb | is that the gpg key update? | 19:02 |
| fungi | #link https://review.opendev.org/877552 Temporarily remove release docs semaphores | 19:02 |
| clarkb | oh that one | 19:03 |
| fungi | i'm un-wipping it now | 19:03 |
| clarkb | we don't need the gpg key update to be clear? that happens after the release? | 19:03 |
| fungi | after, yes. i've set that up for monday | 19:03 |
| clarkb | cool | 19:03 |
| clarkb | And next week is the ptg. Similar story around avoiding making changes but to a different set of tools. Etherpad and meetpad in particular for the PTG | 19:04 |
| clarkb | I suspect at least some of us will be participating in he PTG as well so we may want to skip next week's meeting. But I'll wait for more of the schedule to congeal as I'm not sure yet if it poses a problem | 19:05 |
| fungi | our usual meeting time is during the break | 19:06 |
| fungi | just to be clear | 19:06 |
| clarkb | yes, but if you've spent 4 hours in meetings ou may need a break | 19:06 |
| clarkb | I know I would :) | 19:06 |
| fungi | yes, i likely will do ;) | 19:06 |
| clarkb | #topic Topics | 19:06 |
| clarkb | #topic Docker Free Team Organization Shutdown | 19:07 |
| clarkb | last week docker announced that the free team organization setup we rely on for our docker images is going away on April 14 | 19:07 |
| clarkb | We had a meeting on thursday to discuss options and create a plan to move forward. Where we ended up was to move our container images to quay.io | 19:07 |
| clarkb | #link https://etherpad.opendev.org/p/MJTzrNTDMFyEUxi1ReSo Notes on the situation and planning | 19:07 |
| clarkb | We've taken some decent notes through the whole thing which can be found there | 19:08 |
| clarkb | #link https://quay.io/organization/opendevorg Will become the new home for opendev images | 19:08 |
| clarkb | I've created a new org on quay and I think all infra-root but frickler have been added? Happy to add frickler when notified of an account name | 19:08 |
| clarkb | There are also some job/role updates that we need to make this move happen | 19:08 |
| clarkb | #link https://review.opendev.org/c/zuul/zuul-jobs/+/877834 Role to precreate public images on quay.io | 19:09 |
| clarkb | If you simply push to quay.io you end up with a private repo by default. The way around this is to precreate the repo which this role does (or you can switch it to public after pushing with the private default) | 19:09 |
| clarkb | #link https://review.opendev.org/c/zuul/zuul-jobs/+/838919/ Updates to container roles to make them replace docker roles for quay.io | 19:09 |
| clarkb | and this stack of changes updates the generic container roles to add functionality that was in he docker roles that we need | 19:09 |
| clarkb | corvus: and to be clear the reason we can't use the docker roles is they have hardcoded docker hub as the only destination? | 19:10 |
| clarkb | corvus: after reviewing this stack I wondered why we couldn't just tell the docker roles to talk somewhere else | 19:10 |
| corvus | yes, they are designed specifically for docker hub | 19:10 |
| clarkb | thanks, that helps | 19:10 |
| corvus | mostly the promote role uses docker specific apis | 19:10 |
| clarkb | on the whole I think we've got a reasonable plan and there is progress. Reviews and help are welcome | 19:11 |
| clarkb | On top of those changes we'll need to update our jobs themselves to make use of the updated roles | 19:11 |
| corvus | the other roles are supposed to be pretty similar, just with a few minor assumptions removed and defaults changed | 19:11 |
| corvus | however | 19:11 |
| corvus | the build-container-image role does not appear to have any support for multi-arch builds | 19:11 |
| corvus | is there anyone interested in multi-arch support who is willing to look into bring build-container-image up to par? | 19:12 |
| clarkb | corvus: the main use of that today in our world is nodepool-builder right? | 19:12 |
| clarkb | I think opendev and zuul don't multiarch anywhere else. | 19:12 |
| corvus | i think so; but is there base image support to facilitate that? | 19:13 |
| fungi | which role are we building our multi-arch containers with currently? | 19:13 |
| corvus | fungi: build-docker-image | 19:13 |
| clarkb | corvus: oh yes, the base images are also built for arm64 and x86_64 | 19:13 |
| clarkb | I should be able to take a look at that if no one else is interested. I've poked at our base images and the multi arch stuff in the past | 19:13 |
| ianw | i'll have to context switch it all back in, reviewing those changes should help with that, but i can probably take a look at it | 19:14 |
| clarkb | ianw: oh cool. and ya I would have to context swithc it in. buildx and manifests and all that | 19:14 |
| fungi | okay, so we'd need to port the multi-arch bits from build-docker-image to build-container-image sounds like | 19:14 |
| ianw | ++ i'm sure we can figure something out | 19:14 |
| corvus | thanks; i'm willing to pitch in to help, but it's too much for me alone right now. | 19:14 |
| fungi | thanks for calling it out, i wouldn't have thought to check | 19:15 |
| clarkb | fwiw I think some of the reason this was missing is buildah didn't support multiarch buidls until more recently | 19:15 |
| clarkb | however, you can run docker with the container roles too (and in fact I think that is what opendev at least should do for max compatibility for now) | 19:15 |
| clarkb | I think changing to buildah is a possibility but something we shouldn't worry about now | 19:15 |
| fungi | buildah is a non-docker-specific buildx replacement? | 19:15 |
| corvus | agreed re using "docker" with the "container" roles | 19:15 |
| clarkb | fungi: buildah is the podman image builder. docker does both the container runtime and image building via one interface but podman and buildah split it out | 19:15 |
| corvus | they both use buildkit as the underlying implementation for the new stuff though, so convergence is hopefully possible | 19:16 |
| clarkb | ok cool ianw I'll probably try to look at this late today/early tomorrow for me | 19:17 |
| clarkb | anything else related to the container registry move? | 19:17 |
| corvus | i've completed work on a redirection system for zuul | 19:17 |
| fungi | that was an awesome idea, btw | 19:18 |
| corvus | so i will propose that zuul use "registry.zuul-ci.org/..." as the canonical locations for images | 19:18 |
| corvus | it's a pretty easy option for any other domains that want to do something similar | 19:18 |
| clarkb | ya we should probably do similar for opendev.org. | 19:19 |
| ianw | it's probably worth advertising it somehow? | 19:19 |
| clarkb | ianw: advertising that redirects work? | 19:20 |
| corvus | ianw: can you elaborate? | 19:20 |
| corvus | oh that it's a thing that other projects can do? | 19:20 |
| corvus | like registry.kolla could be a thing if they wanted? | 19:20 |
| fungi | like a blog post about the design or something? | 19:20 |
| ianw | yeah, particularly like openstack projects might want to consolidate, etc. | 19:20 |
| corvus | okay i can write something up | 19:21 |
| corvus | maybe send it to the service-discuss list... | 19:21 |
| ianw | yes, i already saw one similar blog post rise to the top of HN and i think they got it wrong too (might not have worked with podman) | 19:21 |
| corvus | and then copy or link other lists to that? | 19:21 |
| corvus | i suppose i could write an actual blog post | 19:21 |
| fungi | for kolla specifically, it sounded like they were already deploying to multiple registries and were just planning to drop dockerhub, but there are potentally projects even outside opendev that could benefit from seeing that the idea is simple and works | 19:21 |
| fungi | s/deploying/publishing/ | 19:22 |
| clarkb | ya its not something people may realize is possible so getting the word out could be useful | 19:22 |
| ianw | yep, just think it would to get the word out that it's a thing you can do, and here's an example | 19:22 |
| corvus | i think there's 2 audiences: how to do this in general; and how to do this in opendev | 19:23 |
| corvus | for the latter, a pointer to code/changes might be in order | 19:23 |
| corvus | #action corvus write up blog post/email/something to share registry redirect idea/configuration | 19:23 |
| clarkb | ++ | 19:24 |
| fungi | probably solveable by just linking the opendev examples in the general writeup, but yeah i agree it may be two audiences for one post even | 19:24 |
| corvus | if we're happy with the idea for opendev, i can try to make a clean series of changes for that | 19:24 |
| corvus | (since the git history for zuul is not clear due to experimentation) | 19:24 |
| clarkb | corvus: I think I like the idea since it should reduce some over the necessary changes if we do this again | 19:24 |
| corvus | k. expect that later this week :) | 19:25 |
| fungi | yes, also if we're going that route we should do it before we switch registries, in order to save ourselves additional patches to review | 19:25 |
| clarkb | exactly this is the best opportunity for doing it | 19:25 |
| * fungi prepares for docker inc to break the official docker client's ability to follow redirects | 19:26 | |
| clarkb | ok lets move on as we're almost half out of time and only one topic in. THank you everyone who has helped push this along. Its going to be a bit of work and won't happen overnight but we're making progress | 19:26 |
| fungi | yes, thanks! | 19:27 |
| clarkb | #topic Bastion Host Updates | 19:27 |
| clarkb | I mostly wanted to call out that fungi had to boot the old bastion briefly to grab some gpg data off of it. Now is a great time to brainstorm if anything else needs moving | 19:27 |
| clarkb | I suspect that given how long its been and we've only run into this one issue that there isn't. But the sooner we address anything else missing the better | 19:27 |
| fungi | yeah, it's not still booted, was only up for a few minutes, but we should be very sure before we delete the server so that we don't need to restore from backups later | 19:28 |
| ianw | ++ the other thing is, if we do want to go down the path of distributed backups of this (reviews welcome :) then we need a list of things to backup as well. so thinking about what to grab can be codified into that list | 19:28 |
| clarkb | ianw: that is a great idea | 19:28 |
| fungi | /root ;) | 19:28 |
| clarkb | and /etc/ansible | 19:28 |
| ianw | fungi: did you say it was ~gb though? | 19:29 |
| fungi | i don't think it needs to be. there was a metric ton of ansible tempfiles in there | 19:29 |
| ianw | as long as nobody puts anything in /root that isn't critical to backup, that can work, but i also thing huge backups will be hard to deal with | 19:29 |
| fungi | i just copied it all because i didn't want to have to wade through it while i was in a hurry to do key generation stuff | 19:29 |
| clarkb | hrm I bet the new server has tempfiles for ansible too though | 19:30 |
| clarkb | because ansible runnign as root | 19:30 |
| clarkb | something to look at I guess | 19:30 |
| fungi | yeah. i was being glib. specifically /root/signing.gnupg | 19:30 |
| fungi | and /root/certs | 19:30 |
| fungi | and /root/passwords | 19:31 |
| ianw | fair enough, i think i didn't copy the whole thing as sort of a counter-point to that, trying to just keep the important bits | 19:31 |
| ianw | the idea with the distributed backups is all that will go in a ansible variable; it could be either checked in, or in /etc/ansible | 19:31 |
| fungi | a copy of the /root from old bridge can be found in /root/old-bridge-root-homedir.tar.xz for now, until we decide to toss it | 19:32 |
| ianw | (i was thinking /etc/ansible, to make it less inspectable, but open to ideas) | 19:32 |
| clarkb | ianw: as far as landing the backups goes we're mostly waiting on reviews? | 19:32 |
| fungi | i remember saying i was going to review those changes and then not doing so | 19:33 |
| clarkb | anything else bridge related? | 19:34 |
| ianw | not this week | 19:34 |
| clarkb | #topic Mailman 3 | 19:34 |
| clarkb | fungi: anything new with mailman 3 domain management stuff? | 19:35 |
| fungi | negatory, sorry | 19:35 |
| clarkb | #topic Gerrit Updates | 19:35 |
| clarkb | ianw: looks like the last submit-requirements and copyConditions change (the one for testing it in system-config-run-review-*) landed | 19:36 |
| clarkb | ianw: the conversion process should be largely done at this point? | 19:36 |
| ianw | yes; i've spent some time updating | 19:36 |
| ianw | #link https://etherpad.opendev.org/p/gerrit-upgrade-3.7 | 19:36 |
| ianw | for that, and other issues | 19:36 |
| ianw | we just merged a change to rebuild images from 3.7-stable, so i plan to bring up a test host with that | 19:37 |
| clarkb | ya the other thing was the 3.7.1 bug | 19:37 |
| ianw | that could likely be our production image. if there's another release before our upgrade, we can rebuild at that | 19:37 |
| clarkb | sounds good | 19:37 |
| clarkb | in particular the bug we rebuilt to grab a fix for has to do with navigating related chnges in the ui | 19:38 |
| clarkb | something I know I do so good to have fixed | 19:38 |
| ianw | it would be worth going through the "DONE" tags on that page just to double check people agree with my conclusions about various issues | 19:38 |
| clarkb | I'll try to review that today | 19:38 |
| clarkb | this upgrade is very similar to prior upgrades except that we must take a logner downtime for offline reindexing | 19:39 |
| clarkb | for whatever reason they apparently didn't support online reindexing for this one | 19:39 |
| clarkb | #topic Project renames | 19:40 |
| clarkb | related to the gerrit upgrade is the plan for project renames | 19:40 |
| clarkb | we've decided to do both one after the other on April 6 starting at 22:00 UTC | 19:41 |
| clarkb | fungi: other than better understanding our infra-project-manage-projects job sequencing when we've got old state in the earlier chagnes the other open question I've got is if we need to have any more care around openstack gettingthe xstatic library | 19:41 |
| fungi | fwiw, i did an online reindex recently trying to run down a problem that turned out to be unrelated to indexing, but it only took a few hours to complete | 19:41 |
| clarkb | fungi: should we maybe bring that up in the tc meeting tomorrow? | 19:42 |
| fungi | xstatic-angular-fileupload? | 19:42 |
| clarkb | "are you ok with this and does it create any conflict with moin moin?" | 19:42 |
| clarkb | fungi: yes | 19:42 |
| fungi | looks like it's got some shared authorship, yeah | 19:44 |
| fungi | worth bringing up | 19:44 |
| fungi | homepage link for it on pypi goes to https://github.com/danialfarid/ng-file-upload | 19:44 |
| clarkb | ok I'll try to be there tomorrow morning (shouldn't be an issue other than being a few minutes late for a school run) | 19:44 |
| fungi | last touched a little over 4 years ago there | 19:45 |
| clarkb | #topic Updating old servers | 19:46 |
| clarkb | The docker situation has distracted me from this. Which is mostly fine since I was going to hold off on gitea and etherpad work until later (can finalize gitea situation end of this week early next week and do etherpad after the ptg) | 19:46 |
| clarkb | ianw: was there anything new with the nameserver planning? | 19:46 |
| ianw | no, i've also been distracted. there are some preparatory changes to merge, then i need to finish the checklist for switching and we can review. parallel i can start new servers | 19:47 |
| clarkb | ok ping me if I can help. I think I've reviewed changes though | 19:48 |
| clarkb | #topic AFS disk utilization | 19:48 |
| clarkb | Nothing really new here. I've left it on the agenda as a reminder it is something we need to keep an eye on. We do seem to have stablized near 10% free on the main server | 19:48 |
| clarkb | and eventually fedora 36 should get cleaned up relieving some pressure just in time for new debian :) | 19:48 |
| ianw | yep i'll try to get 37 builds going soon | 19:49 |
| clarkb | #topic Gitea 1.19 | 19:50 |
| clarkb | Over the weekend gitea released 1.19.0. I had already pushed a chagne to sanity check 1.19.0-rc1 and there were minimal updates necessary to get 1.19.0 going | 19:50 |
| clarkb | that said I think we should hold off until we've stabilized the gitea replacements | 19:50 |
| clarkb | which is also planned for after the openstack release | 19:50 |
| clarkb | #link https://review.opendev.org/c/opendev/system-config/+/877541 Upgrade opendev.org to 1.19.0 | 19:51 |
| clarkb | this release adds an actions runner system which is disabled by default. I have explicitly disabled it in this change to avoid that default changing on us later since we've got zuul | 19:51 |
| clarkb | Otherwise the update seems pretty straightforward. THe commit message in that change tries to highlight all of the breaking changes from the release notes with my own notes on how/why things do or don't affect us | 19:52 |
| clarkb | interestingly there were html template updates between 1.19.0-rc1 and 1.19.0. I'm not sure that has happened before | 19:52 |
| clarkb | a good reminder for us to double check that between releases though | 19:52 |
| clarkb | #topic Quo vadis Storyboard | 19:53 |
| clarkb | last week frickler pointed out that there may be storyboard discussions at the ptg next week. In #opendev I mentioned that I think we should continue to encourage any projects migrating off of storyboard to collaborate to avoid duplicate effort on tooling/process | 19:53 |
| fungi | so i guess keep an ear to the ground for projects bringing this up at the ptg and encourage them to weigh in on the opendev discussion? | 19:54 |
| clarkb | fungi: yup and/or maybe push them towards each other and have shared discussion at the ptg too? | 19:54 |
| fungi | sounds good | 19:54 |
| clarkb | I'm also happy to jump into those sessions if people notify me | 19:54 |
| fungi | yeah, one up-side to vptg, easy to drop from one call and jump into another (faster than running across the hall, much less to an adjacent building) | 19:55 |
| clarkb | I also mentioend that I think longer term we (opendev) should think about sunsetting storyboard due to our lack of maintenance for the server and service | 19:55 |
| clarkb | one good idea that came out of me mentioning that is we could try to make some sort of read only copy of the content and host that hopefully with much simpler hosting demands | 19:55 |
| clarkb | but I also poitned out that I'm not the lone decision maker and welcomed feedback on that idea. Consider this an official reuqest for feedback from this group | 19:56 |
| clarkb | #topic Open Discussion | 19:57 |
| clarkb | We've got ~3 minutes for anything else | 19:57 |
| fungi | having a search engine friendly view of the content instead of just browser-side rendering was, ironically, one of the wishlist items for further sb development | 19:57 |
| ianw | #link https://review.opendev.org/c/opendev/system-config/+/877735 | 19:57 |
| ianw | is one to split out the ssh server setup | 19:57 |
| ianw | from base. i'd like to try running that against the linaro cloud to make sure the root login auth, etc. are setup right | 19:58 |
| clarkb | we can also potentially extend that to inmotion | 19:58 |
| clarkb | but I Thin kstarting with linaro is great | 19:58 |
| ianw | the prod job we added is working; but i did have to manually allow the zuul login | 19:58 |
| ianw | (which is ok, that's setup by the launch node usually, which obviously didn't run) | 19:59 |
| clarkb | ianw: its not zuul but bridge right? | 19:59 |
| clarkb | I mean zuul via bridge | 19:59 |
| clarkb | not direct | 19:59 |
| ianw | yeah, basically allowing root logins from bridge. but that config will be deployed with the ssh role, so if we change bridge again it will be ok | 20:00 |
| fungi | speaking of linaro, i think we had a good justification to add them to the donor logos on https://opendev.org/ | 20:00 |
| fungi | anybody know who we should reach out to about that? | 20:01 |
| fungi | kevinz? | 20:01 |
| clarkb | fungi: yes kevinz is who I would start with | 20:01 |
| fungi | k | 20:01 |
| clarkb | and we are over time | 20:01 |
| fungi | thanks clarkb | 20:01 |
| fungi | ! | 20:01 |
| ianw | or perhaps ... sorry i'm blanking on the name ... but the person we sent the usage blurb to | 20:01 |
| fungi | ianw: good call | 20:01 |
| clarkb | thank you everyone! feel free to continue discussion in #opendev or on the mailing list. But I don't need to keep you in here any longer | 20:01 |
| fungi | works on arm person | 20:01 |
| clarkb | #endmeeting | 20:02 |
| opendevmeet | Meeting ended Tue Mar 21 20:02:11 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 20:02 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/infra/2023/infra.2023-03-21-19.01.html | 20:02 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/infra/2023/infra.2023-03-21-19.01.txt | 20:02 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/infra/2023/infra.2023-03-21-19.01.log.html | 20:02 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!