| *** dhill is now known as Guest8283 | 00:15 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: wip: test idea for registry pull https://review.opendev.org/c/opendev/system-config/+/877884 | 00:21 |
|---|---|---|
| ianw | https://imgur.com/a/2Sx426k or https://104.130.253.50/c/x/test-project/+/3 are some markdown comments | 00:41 |
| ianw | i think it's a release note bug that it's listed as an experiment, it's enabled by default | 00:49 |
| ianw | the @user tagging in comments is not enabled afaict, but is an experiment. i don't think we need to turn that on explicitly, let's just let upstream work out the kinks in the main instance | 00:50 |
| opendevreview | Merged openstack/diskimage-builder master: Removal focal pins for testing https://review.opendev.org/c/openstack/diskimage-builder/+/877443 | 00:52 |
| opendevreview | Merged openstack/diskimage-builder master: Update Fedora to 37 https://review.opendev.org/c/openstack/diskimage-builder/+/876482 | 00:52 |
| ianw | clarkb: https://review.opendev.org/c/opendev/system-config/+/868054 is one that adds some s-r and copyconditions labels to the gerrit testing, just to finalise the testing there | 01:00 |
| opendevreview | Ian Wienand proposed opendev/system-config master: wip: test idea for registry pull https://review.opendev.org/c/opendev/system-config/+/877884 | 02:09 |
| opendevreview | Ian Wienand proposed opendev/system-config master: wip: test idea for registry pull https://review.opendev.org/c/opendev/system-config/+/877884 | 03:51 |
| opendevreview | Ian Wienand proposed opendev/system-config master: wip: test idea for registry pull https://review.opendev.org/c/opendev/system-config/+/877884 | 04:31 |
| opendevreview | Ian Wienand proposed opendev/system-config master: wip: test idea for registry pull https://review.opendev.org/c/opendev/system-config/+/877884 | 06:25 |
| opendevreview | Ian Wienand proposed opendev/system-config master: wip: test idea for registry pull https://review.opendev.org/c/opendev/system-config/+/877884 | 07:18 |
| opendevreview | Ian Wienand proposed opendev/system-config master: wip: test idea for registry pull https://review.opendev.org/c/opendev/system-config/+/877884 | 07:19 |
| opendevreview | Ian Wienand proposed opendev/system-config master: wip: test idea for registry pull https://review.opendev.org/c/opendev/system-config/+/877884 | 07:53 |
| opendevreview | Ian Wienand proposed opendev/system-config master: wip: test idea for registry pull https://review.opendev.org/c/opendev/system-config/+/877884 | 08:31 |
| opendevreview | Ian Wienand proposed opendev/system-config master: wip: test idea for registry pull https://review.opendev.org/c/opendev/system-config/+/877884 | 09:19 |
| opendevreview | daniel.pawlik proposed zuul/zuul-jobs master: DNM - Provide ensure-microshift role https://review.opendev.org/c/zuul/zuul-jobs/+/876081 | 10:39 |
| *** arxcruz is now known as arxcruz|rover | 10:57 | |
| clarkb | Gitea 1.19 released over the weekend. I'll look into the release notes and update my rc change (I don't think we should land that this week with the openstack release, but good to get it out of the way) | 15:13 |
| clarkb | Also I need to followup on updating jitsi meet's docker iamge locations | 15:14 |
| clarkb | but first kernel updates! | 15:15 |
| clarkb | jitsi meet have not updated their docker image location yet | 15:28 |
| fungi | mmm | 15:28 |
| *** dhill is now known as Guest8338 | 15:33 | |
| opendevreview | James E. Blair proposed opendev/system-config master: wip: test idea for registry pull https://review.opendev.org/c/opendev/system-config/+/877884 | 15:37 |
| clarkb | "Return 404 instead of 403 if user can not access the repo (#23155) (#23158)" <- from gitea 1.19 release notes. Similar to what I was grumping about the quay API on friday | 15:43 |
| clarkb | "Repositories: by default disable all units except code and pulls on forks (#22541)" is another fun one that won't affect us | 15:44 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update gitea to 1.19.x https://review.opendev.org/c/opendev/system-config/+/877541 | 16:04 |
| clarkb | there was a template update between rc and final release | 16:04 |
| clarkb | good think I double checked | 16:04 |
| clarkb | fungi: ianw: more people are complaining about Gerrit 3.7.1's related changes behavior | 16:26 |
| clarkb | I'm hopeful that will get a fix landed before we upgrade | 16:26 |
| fungi | seems likely | 16:29 |
| fungi | even if there's no 3.7.2 yet, we're deploying images autobuilt from the tip of their 3.7 stable branch anyway right? | 16:29 |
| clarkb | correct | 16:29 |
| clarkb | corvus: https://review.opendev.org/c/zuul/zuul-jobs/+/877834 is where I ended up with the precreation of quay repos. I need to catch up on your aliasing of registries work. Is that basically just look at https://review.opendev.org/c/opendev/system-config/+/877884 ? | 16:32 |
| fungi | ianw spotted a misbehavior in testing it, per his last comment | 16:37 |
| clarkb | ianw: the markdown thing? | 16:38 |
| clarkb | er sorry fungi ^? | 16:39 |
| fungi | clarkb: incorrect pattern match with the redirect in 877859 | 16:40 |
| clarkb | ah thanks | 16:41 |
| fungi | suggested going from * to + | 16:41 |
| corvus | yeah and i agree with that (i even meant to type that, but pasted the wrong thing) | 16:47 |
| corvus | i think ianw's change supercedes the testinfra change, since it does the full docker/podman pull, so i rebased it away from the testinfra change. it's possible that ianw also wanted a testinfra that does a curl... if so, we can add that. it basically would have nothing in common with the testinfra change anyway, so i'd probably just stick the new one on the end... | 16:52 |
| corvus | i think 877884 looks good now after that rebase away from testinfra -- https://zuul.opendev.org/t/openstack/build/5eedf23dc6274ba893f6342aba3e2545/console is showing the actual pull | 16:54 |
| corvus | ianw did make one change i wasn't planning on doing -- he made /v1/ work. i was only planning on proxying v2 since i'm not sure how to test that v1 is actually working, but i guess we can try proxying v1 and see how it goes :) | 16:55 |
| corvus | we will need to check the logs though to make sure that we're not falling back to v1 | 16:55 |
| clarkb | I think just about everything is v2 only at this point? | 16:55 |
| corvus | yeah, that's why that was my plan | 16:55 |
| corvus | if we don't proxy v1, then we know if we've broken v2. | 16:56 |
| corvus | with the v1 proxy in there, i'm not actually sure i would have noticed that we needed the empty index file for v2 to work. | 16:56 |
| corvus | my preference still would probably be to only support v2 | 16:56 |
| corvus | so if we want to support v1, thin i think my change + ianw's are ready to merge (except ianw's needs a new commit msg). if we want to disable v1 we need to tweak it. | 16:58 |
| clarkb | I can go either way on it. I'm not actually sure modern clients fall back to v1 it would just be for old cliens? | 16:59 |
| corvus | podman tries to | 16:59 |
| clarkb | huh til | 16:59 |
| opendevreview | James E. Blair proposed opendev/system-config master: Add a functional test for registry.zuul-ci.org https://review.opendev.org/c/opendev/system-config/+/877884 | 17:00 |
| corvus | okay that's ianw's change with the commit msg updated | 17:00 |
| clarkb | oh I guess that is what ianw's -1 comment was getting at. Podman is falling back | 17:00 |
| clarkb | corvus: one reason to proxy v1 would be ifwe end up sitting in front of a registry that is v1 only. But we can add that support when it becomes necessary and it should be less and less necessary over time | 17:02 |
| corvus | https://8464fd24502a6c238377-7f41bd38e61f614f484bdc6903cb8f38.ssl.cf2.rackcdn.com/877884/11/check/system-config-run-static/5eedf23/static99.opendev.org/apache2/registry.zuul-ci.org_access.log confirms that we are supporting v2 correctly | 17:02 |
| corvus | (and that the v1 proxy isn't masking a v2 failure) | 17:02 |
| corvus | that's actually a pretty good argument for the standalone testinfra check -- if we proxy v1, then we definitely should have a testinfra that does a "GET /v2/" to make sure that the v2 empty index page works and is not redirected | 17:03 |
| opendevreview | James E. Blair proposed opendev/system-config master: Remove v1 proxy from registry.zuul-ci.org https://review.opendev.org/c/opendev/system-config/+/877989 | 17:05 |
| corvus | okay that's a followup to remove v1. i don't feel strongly about that, but that puts the system in the state that i originally imagined. let's see if ianw wants to keep v1. if so, i have no objection. | 17:05 |
| clarkb | https://gerrit-review.googlesource.com/c/gerrit/+/364214 is the fix for the gerrit thing whcih has landed. This means we should be able to hold a 3.7.1-foo build and double check it on our end | 17:11 |
| opendevreview | James E. Blair proposed opendev/system-config master: Add testinfra for registry.zuul-ci.org https://review.opendev.org/c/opendev/system-config/+/877860 | 17:15 |
| opendevreview | James E. Blair proposed opendev/system-config master: Add testinfra for registry.zuul-ci.org https://review.opendev.org/c/opendev/system-config/+/877860 | 17:17 |
| corvus | clarkb fungi ianw : okay i think 877859 877884 877860 are ready for immediate merging; i think we agree on everything in them. then 877989 is the open question about whether or not we support v1. | 17:18 |
| corvus | topic:reg-proxy | 17:18 |
| clarkb | ok I'll review now | 17:18 |
| fungi | oh, i see, ianw's -1 on 877859 is actually about the testing change not the change he technically voted against | 17:20 |
| fungi | no, nevermind it's about the RewriteRule in 877859 | 17:20 |
| corvus | yep | 17:21 |
| clarkb | corvus: two questions. The old registry.zuul-ci.org wasn't used for anything? Or do we need to ensure what we are proxying to is valid first? Second do you intend on proxying to corvus/ long term or should we update that now? | 17:21 |
| corvus | fungi: and i'm proposing we land my "broken" change and let ianw's testing change fix it. that's okay because it's not in production. | 17:21 |
| fungi | so is that rewriterule being fixed in a followup change? | 17:21 |
| fungi | aha, yes it is. thanks! wfm | 17:21 |
| fungi | i should have just reviewed the others first | 17:22 |
| corvus | clarkb: registry.zuul-ci.org didn't exist before a few days ago, so no worries there. not long term, will proxy to some as-yet-undetermined org. | 17:22 |
| corvus | i'd like to merge this and double check everything is working (since this is a functional change from my ".htaccess" approach i started last week) | 17:23 |
| clarkb | corvus: the index file we create isn't accessible due to the redirect is it? | 17:23 |
| corvus | then: pick a quay org; copy the repos, then update this to the real production values | 17:23 |
| fungi | clarkb: the followup change fixes that | 17:23 |
| corvus | clarkb: it is not accessible in my first change, it is with ianw's fix | 17:23 |
| clarkb | fungi: right, but the followup is the one we aren't sure we will merge? | 17:23 |
| clarkb | wondering if we should create the index file in ianw's fixup too? | 17:24 |
| corvus | er no | 17:24 |
| corvus | i'm so sorry | 17:24 |
| corvus | i feel like i've burned 200 hours of developer time because i typed a "*" instead of a "+" on a saturday... | 17:24 |
| fungi | i'm not following. the rewriterule in 877859 shadows the index.html file, but then in 877884 it changes to no longer shadow it | 17:25 |
| corvus | give me a minute to prepare an explanation | 17:25 |
| corvus | okay, first; i updated the hashtags, so there are 4 changes here under hashtag:reg-proxy | 17:26 |
| fungi | clarkb: i didn't realize we were still unsure about merging 877884, i'd approved it | 17:26 |
| corvus | wait why isn't that hashtagging working | 17:26 |
| clarkb | corvus: one of them has hashtag:hashtag | 17:27 |
| clarkb | the other three seem to have set properly | 17:27 |
| corvus | yeah ianw set that... that's the one i'm trying to change... | 17:27 |
| corvus | i guess i can't because i don't own it or something? | 17:28 |
| corvus | oy | 17:28 |
| corvus | i thought we made hashtags editable? | 17:28 |
| fungi | we still haven't officially decided to allow non-owners to edit hashtags | 17:28 |
| fungi | which is gerrit's default | 17:28 |
| fungi | i'm in favor of doing so | 17:28 |
| corvus | it makes zero sense i can't edit the hashtag but i can edit the topic | 17:28 |
| fungi | yep | 17:28 |
| JayF | That can be changed per-project; Ironic already has fwiw | 17:29 |
| corvus | zuul has too i think | 17:29 |
| fungi | some openstack teams played with granting core reviewers exclusive rights to set hashtags and were worried about random users adding/removing them, but i don't know that the concern was warranted nor even still relevant | 17:29 |
| clarkb | I think the main concern is that its an unbounded value unlike topic? | 17:30 |
| corvus | okay, let me explain, i mean sum up | 17:30 |
| corvus | https://review.opendev.org/q/topic:reg-proxy | 17:30 |
| clarkb | but ya we can probably just go ahead and update All-Projects | 17:30 |
| corvus | all of those changes expect "remove v1 proxy" are ready to merge now, and when all 3 are merged, they will produce a working system and there are no unresolved concerns with those 3. | 17:31 |
| corvus | the only remaining unresolved concern is with whether or not we proxy v1 requests, so let's at least wait for ianw on that one. | 17:31 |
| clarkb | gotcha I misinterpreted the earlier statement about maybe merging things to mean the one that adds v1 not removes it. | 17:31 |
| clarkb | I do think the most correct thing is to add the index file when it becomes accessible, but it isn't worth restacking again to make that change | 17:32 |
| clarkb | and we are using 302s since the resources may move again | 17:34 |
| corvus | i don't disagree | 17:34 |
| corvus | yep | 17:34 |
| clarkb | oh and it will change very soon to address corvus/ but even then we still want to use 302s | 17:36 |
| corvus | yep | 17:36 |
| clarkb | fungi: do you want to review https://review.opendev.org/c/opendev/system-config/+/877859 ? I +2'd it but you approved the child so wanted to make sure you had a chance to look at it first | 17:37 |
| fungi | i thought i already had? | 17:37 |
| corvus | oh, re-reading ianw's comment on podman and v1 -- i think he's saying that even podman won't talk to v1 even though it hits the v1 ping url. that seems to suggest it's really six-of-one whether we proxy v1 or not. | 17:38 |
| fungi | gertty still shows me +2 on it | 17:38 |
| clarkb | fungi: oh it shows now. I don't know how I missed it | 17:38 |
| clarkb | thats probabl pebkac on my end | 17:38 |
| fungi | yeah, i +2'd that one yesterday, it hasn't changed since | 17:38 |
| corvus | i'm going to go ahead and +w that because i believe all of ianw's concerns have been addressed | 17:39 |
| clarkb | wfm | 17:39 |
| corvus | clarkb: and you have a +2 from me on 877834 | 17:39 |
| clarkb | corvus: cool. I think we can adjust that one too if we find the api isn't quite in line with what we expect out of the updated jobs | 17:40 |
| clarkb | corvus: you were still going to work on the push side of the jobs right? | 17:40 |
| corvus | clarkb: yes -- the registry proxy did bump that a bit though; hopefully within a day or two? | 17:46 |
| Clark[m] | corvus: that last message didn't make it to IRC :( but sounds good to me | 17:46 |
| Clark[m] | and it just went through. A lag of a minute or two | 17:46 |
| opendevreview | Merged opendev/system-config master: Use a static Apache config for registries https://review.opendev.org/c/opendev/system-config/+/877859 | 18:16 |
| opendevreview | Merged opendev/system-config master: Add a functional test for registry.zuul-ci.org https://review.opendev.org/c/opendev/system-config/+/877884 | 18:16 |
| opendevreview | Merged opendev/system-config master: Add testinfra for registry.zuul-ci.org https://review.opendev.org/c/opendev/system-config/+/877860 | 18:28 |
| noonedeadpunk | Hey folks. I'm having issue with rocky-container element and wanted to raise a concern on approach of taking upstream containerfiles. As it's suuuper anoying to do that in environments with no internet connectivity. Now just to build couple of images we kind of need to mirror dockerhub which is really meh. | 19:21 |
| noonedeadpunk | But you're still maintianing mirrors for DNF as they're required regardless | 19:21 |
| noonedeadpunk | But dockerhub or smth simmilar is quite a pite to have as a requirement for such simple thing. And building rocky container image kind of jeopardize whole idea of having dib | 19:22 |
| Clark[m] | I'm not sure I fully agree. You can cache a single rocky image and that's all you need | 19:23 |
| noonedeadpunk | It's still yet another system to maintain | 19:23 |
| noonedeadpunk | And ensure it's state and monitor, etc | 19:23 |
| Clark[m] | But also you can't build rocky/fedora/centos-stream using dnf on a number of platforms because it needs to be new enough. We decided that trying to sort that out for all platforms is basically impossible hence relying on the upstream minimal container images to bootstrap the container | 19:24 |
| Clark[m] | The alternative is you can't build the image at all which really does negate the purpose of div | 19:24 |
| Clark[m] | noonedeadpunk: it doesn't need to be a complicated system. You can just import the image into your builder and be done with it | 19:24 |
| Clark[m] | A full mirror isn't required | 19:25 |
| noonedeadpunk | Or maintain https://review.opendev.org/c/openstack/diskimage-builder/+/805800 as internal element... | 19:25 |
| Clark[m] | Sure can always have other elements. The main issue is you can't get a new enough dnf on debuntu to build newer rpm based distros iirc | 19:26 |
| noonedeadpunk | Well, I think c9s is now built using dnf? | 19:27 |
| noonedeadpunk | So r9 kind of should not have any issue to be built as well | 19:27 |
| Clark[m] | Using container images as a base addresses that as well as in the opposite direction (debootstrap on rpm distros) | 19:27 |
| Clark[m] | The problem is you have to install dnf on Ubuntu/debian | 19:27 |
| Clark[m] | And that isn't new enough so it just doesn't work | 19:27 |
| noonedeadpunk | Yeah, or build on Centos | 19:28 |
| noonedeadpunk | But to use container you need to install podman | 19:28 |
| Clark[m] | Rather than try and solve this dependency problem (which exists elsewhere too) the decision was to rely on container images to bootstrap the base chroot | 19:28 |
| noonedeadpunk | And then to ensure that image I'm using is in sync and always latest | 19:28 |
| Clark[m] | And that solves the issue basically universally | 19:28 |
| fungi | noonedeadpunk: not sure if it was already mentioned, but NeilHanlon is also talking about having rocky container images somewhere other than dockerhub in the future | 19:28 |
| Clark[m] | You don't need to ensure the image is in sync and the latest | 19:28 |
| Clark[m] | The dib build will update the image. | 19:29 |
| Clark[m] | This is why I'm saying it really shouldn't be a big deal you import the image locally once and you are done | 19:29 |
| noonedeadpunk | Thant kind of reminds me approaches when instead of regulary update images some cloud providers just inject `apt update` to cloud-init... | 19:29 |
| NeilHanlon | fwiw, and this doesn't exactly solve the problem but I guess it could... Rocky also publishes our container rootfs as .tar.xz | 19:30 |
| fungi | noonedeadpunk: building on centos gets to be problematic when you want to generate a very new debian or ubuntu and centos only has old versions of apt available | 19:30 |
| NeilHanlon | e.g. https://dl.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-Container-Base.latest.x86_64.tar.xz | 19:30 |
| Clark[m] | Yes the problem exists in both directions hence the containerfile approach to avoid it in the first place | 19:30 |
| NeilHanlon | time to unite and make sure we all have up to date versions of one another's tools 👀 | 19:31 |
| noonedeadpunk | ^++ | 19:31 |
| Clark[m] | noonedeadpunk this is different than having cloud init update at boot because it happens pre boot | 19:31 |
| fungi | the idea was to find a universal bootstrapping solution, so if centos, fedora and other distros also provide a rootfs tarball then that could be an alternative to containers i guess | 19:32 |
| Clark[m] | It's just a bootstrap build env. You build should then update itself | 19:32 |
| fungi | since all the distros seem to provide container images capable of running their bootstrapping tools, containers seemed like the solution | 19:32 |
| fungi | rather than maintaining a different bootstrapping solution for each distro | 19:33 |
| fungi | or, rather, different solution to setting up the pre-bootstrapping environment for each distro | 19:33 |
| noonedeadpunk | Probably I'm jsut not convinced about idea that 2yo container image can be jsut updated with dnf and work nicely... | 19:33 |
| noonedeadpunk | And won't be messed up with any new element | 19:34 |
| noonedeadpunk | especially when package names are changed as well as repo names | 19:34 |
| fungi | dib isn't building a container image though, it's using an existing container image to get the necessary tools to bootstrap a working rootfs chroot | 19:34 |
| noonedeadpunk | Yeah, exactly my point, that I just was percieving dib as autonomous image building tool | 19:35 |
| Clark[m] | noonedeadpunk it has to be or the distro is broken as I may turn on a two year old install and update it. The only distro this has ever broken for me is Arch | 19:35 |
| noonedeadpunk | And it appears to be not anymore | 19:35 |
| Clark[m] | I think you are mischaracterizing things | 19:35 |
| fungi | severely | 19:35 |
| noonedeadpunk | For me CentOS used to be pita | 19:35 |
| Clark[m] | Dib has chosen to shift the tools it uses for bootstrapping | 19:35 |
| noonedeadpunk | It added an external requirement for some images | 19:36 |
| fungi | using the containerfile method, we don't have to worry about whether ubuntu lts-1 has new enough dnf or apt or whatever to create a chroot of another distro | 19:36 |
| noonedeadpunk | Yeah, I got what problem you was solving | 19:36 |
| noonedeadpunk | I'm jsut saying that it requires now external connection to other sources, it uses more traffic and I would be surprised if bootstrap time went down | 19:38 |
| fungi | for some images maybe, but overall it replaced the need to have an cross-distro package of an external package manager compatible with the distro you wanted to bootstrap with needing a container image of the distro you want to bootstrap | 19:38 |
| Clark[m] | So I think there are two important things to keep in mind. Dib has made a decision to solve a specific problem that has long plagued it. Yes this added a new dependency, but not with any malicious intent. And you can address your issue with the new tool in a straightforward manner. Second dib is flexible and allows you to build images however you like using it's run phases design. We aren't stopping you from doing something different | 19:39 |
| fungi | so for cross-distro building (what dib is primarily designed for) it replaced one dependency with another | 19:39 |
| fungi | instead of needing to find an installable and working version of the package manager for the distro you're building, you need a container image for the distro you're building. it's not really an additional dependency, if anything it can reduce dependencies when you're building lots of different distro images | 19:40 |
| NeilHanlon | question: i see some elements seem to have logic which downloads qcow2 files and extracts a rootfs from them (centos, e.g.) - is that still in use? | 19:41 |
| noonedeadpunk | Well, in rocky-container element I saw great comment, like `Distributions vary in their podman version and various bugs related to this.` | 19:41 |
| fungi | NeilHanlon: likely still in use somewhere, yes. opendev doesn't and hasn't used those but they were added by people who did | 19:42 |
| Clark[m] | NeilHanlon: yes but they aren't minimal builds. Which is what the container file element provides | 19:42 |
| noonedeadpunk | Also my point was mainly in external dependency. so instead of need for mirrors you also now need dockerfiles | 19:42 |
| noonedeadpunk | *docker images | 19:42 |
| Clark[m] | noonedeadpunk yes, but that isn't a fatal flaw and I don't think you should characterize it that way. It is a compromise | 19:43 |
| noonedeadpunk | Is anything except rocky use container approach as of today? | 19:44 |
| clarkb | noonedeadpunk: fedora | 19:44 |
| noonedeadpunk | ah,ok | 19:44 |
| fungi | also i don't think anyone's said dib is going to start deleting other ways of building images, but this was a new solution which specifically solves a problem opendev has building lots of different distros, including very new releases of distros, from a single stable server distro which can be years old | 19:45 |
| noonedeadpunk | btw, question then - do you happen to know if fcos is supported by dib? | 19:45 |
| fungi | fedora was the one which prompted adding the containerfile method, because we couldn't install new enough dnf on ubuntu lts | 19:45 |
| clarkb | I have no idea if coreos can be built by the fedora elemnts | 19:46 |
| noonedeadpunk | hm... Ok, so basically if we work on https://review.opendev.org/c/openstack/diskimage-builder/+/805800 to pass it CI it has chances to be merged? | 19:46 |
| noonedeadpunk | Sorry, I'm jsut trying to understand all options | 19:46 |
| clarkb | NeilHanlon: historically dib has had two major styles of image build. The first is one that starts with the upstream published images and modifies them. This has a couple downsides mostly in that its hard t oremove things from the base image so you end up stuck with stuff ou ma not want (for example cloud-init). The other style is to start with yum/dnf/debootstrap/apt and create a | 19:47 |
| clarkb | new image from scratch in the chroot. The containerfile stuff is addressing this second style of image | 19:47 |
| clarkb | noonedeadpunk: I'm not sure that ianw wants to commit to maintaining that in dib directly. You can definitely have out of tree elements though | 19:47 |
| noonedeadpunk | And I think for rocky/fedora there's also currently no option for the first style (of upstream qcow being modified)? | 19:48 |
| fungi | yeah, there are multiple ways of building the same distros already included in dib though, so it might be accepted. would be up to the dib maintainers to decide | 19:48 |
| clarkb | noonedeadpunk: There might be for fedora. But rocky is new enough that I don't think anyone added it | 19:48 |
| clarkb | we (opendev) are not fans of the first style of image and if we had our way we'd probably delete all those elements but enough people use them that that would be problematic | 19:49 |
| clarkb | its funny because for a long time red hatters kept saying we didn't support that method... | 19:50 |
| clarkb | but we definitely did despite our interest in not doing so | 19:50 |
| clarkb | that always came up with that alternative tool was discussed. libguestfs based thing | 19:50 |
| clarkb | eventually I stopped trying to argue and piont out the exact use case they had is already supported | 19:50 |
| clarkb | we just don't actively use it in opendev because we don't want cloud init in our images | 19:51 |
| fungi | also the opendev sysadmins as a group aren't the ones deciding what goes into dib, it's maintained by its own core reviewers within an openstack sig | 19:51 |
| clarkb | yes its like 50% opendev and 50% ironic though | 19:51 |
| clarkb | so lots of overlap | 19:51 |
| noonedeadpunk | well, I'm considering dib for building public images, so it's they'd be kind of close to the upstreamed qcow | 19:52 |
| NeilHanlon | I also have personal interest in building the Rocky images with DIB, so.. there are some "synergies" here | 19:52 |
| noonedeadpunk | But I was also trying to build minimal from scratch rather then full qcow | 19:52 |
| NeilHanlon | Right now our images are built using anaconda and it's a ... mess :) | 19:53 |
| noonedeadpunk | But do need cloud-init for sure, though there's an element for it) | 19:53 |
| noonedeadpunk | oh, ok, good point not to use them then :D | 19:53 |
| NeilHanlon | :P | 19:53 |
| clarkb | NeilHanlon: we also had centos-stream images available before centos did aiui | 19:53 |
| clarkb | because with dib adding support was pretty quick but whatever upstream uses was not | 19:54 |
| fungi | yeah, our avoidance of cloud-init is somewhat situational. we test a lot of python software, including projects whose jobs like to install python libs into the system python context, so the slew of python libraries cloud-init depends on can create conflicts with testing | 19:54 |
| clarkb | fungi: that and cloud init didn't support clouds like rackspace for a long time (this was fixed though) | 19:54 |
| noonedeadpunk | and cloud-init is useless with fedora now... which is super annoying to be frank... | 19:55 |
| fungi | yeah, i mean, we could have worked with the cloud-init maintainers to fix that problem, "fixing" their reliance on third-party libraries far less likely to be accepted though | 19:55 |
| clarkb | but ya you should be able to write out of tree elements at the very least. And can have conversations in #openstack-dib with the dib maintainers about whether or not any of the proposed elements make sense in tree | 19:56 |
| NeilHanlon | more than happy to collaborate on that | 19:56 |
| fungi | even we have out-of-tree dib elements which contain specific things for our environment | 19:56 |
| clarkb | mostly I think its a problem of having 3 different ways to express the same thing creating confusion and maintenance overhead. But if there is user interest and people willing to help then thats probably addressable | 19:56 |
| noonedeadpunk | anyway thanks, I will try to think about options... Maybe will attempt to revive patch with building rocky from dnf... as it should work jsut nice as well as c9s does atm... But given fedora... doh... Probably we will end up having local copy of dockerhub or smth, which is annoying, but seems not much options | 19:57 |
| clarkb | noonedeadpunk: again I really don't think you need a proper mirror | 19:57 |
| clarkb | you just need to seed your local image repo with the base images | 19:57 |
| clarkb | skopeo can do this iirc | 19:57 |
| fungi | i have a feeling a lot of distros are going to start putting container images in other/additional registries soon anyway | 19:58 |
| clarkb | fungi: ya I don't think the issue is docker hub as much as the assumption that they will need torun a docker registry cache/mirror | 19:58 |
| noonedeadpunk | I can already feel how messy that could be | 19:58 |
| clarkb | instead you just do skopeo something something this image upstream to my local disk please | 19:58 |
| NeilHanlon | noonedeadpunk, btw since you mentioned FCOS... http://dl.rockylinux.org/pub/sig/8/ostree/x86_64/isos/ | 19:58 |
| noonedeadpunk | clarkb: I will read about skopeo for sure | 19:59 |
| fungi | we do maintain a caching dockerhub proxy, though it's mildly convoluted mainly because of how braindead their api is | 19:59 |
| noonedeadpunk | NeilHanlon: I was mentioning it mainly because of magnum being dependant on it | 20:00 |
| fungi | and also because we're trying to get by with apache mod_proxy instead of something more full-featured like squid | 20:00 |
| noonedeadpunk | I'm not sure rocky would be compatible with it? | 20:00 |
| clarkb | noonedeadpunk: oh magnum should stop relying on fedora but no one listens to me on that one :) | 20:00 |
| noonedeadpunk | I can't agree more on that.... | 20:00 |
| clarkb | noonedeadpunk: the problem there is a cloud gets deployed with magnum and at that point the fedora release magnum relies on has at best 13 months of support. But then you still have users running k8s clusters more than 13 months later | 20:01 |
| * NeilHanlon adds it to the list of things to put rocky in | 20:01 | |
| fungi | magnum's going to need to find a way to make old magnum versions support new fedora versions if their plan is to stick with fedora | 20:01 |
| fungi | i tried to point that out to them on the ml recently | 20:01 |
| clarkb | this is one of the major reasons we (opendev) do not use magnum | 20:01 |
| NeilHanlon | i know centos stream is doing releases of ostree/coreos stuff now | 20:01 |
| NeilHanlon | that might be their end game | 20:01 |
| ianw | corvus: yeah, sorry, i left that stack in a bit of a state as it was getting late here too. sorry i mentioned in https://review.opendev.org/c/opendev/system-config/+/877859 the /v1 thing was a red-herring | 20:03 |
| ianw | when i was looking at the apache logs, i at first didn't realise the first "ping" podman does is a GET to just /v2 | 20:03 |
| ianw | when that fails, it seems to fall back to /v1/_ping | 20:04 |
| ianw | so, before i realised that, i thought maybe we had to proxy /v1 to get the first ping to work or something. but agree we should drop it | 20:04 |
| ianw | oh, and i set the hashtag just as a test of what zuul does to hashtag events actually, which is a separate thing ... | 20:11 |
| clarkb | I think it just ignores them currently? | 20:12 |
| ianw | noonedeadpunk: i do get that building in an isolated environment the containerfile's dependency on pulling from the registry makes things more difficult. tbh i don't think we really considered that case | 20:22 |
| ianw | i think cross-distro minimal builds are probably the biggest non-starter | 20:26 |
| ianw | i will say that it has not been easy running podman inside docker in the build environment, it fairly constantly breaks | 20:28 |
| ianw | ultimately we just make a tarball of that. so i wonder if perhaps something like having the ability to take a pre-cached tarball might work better | 20:29 |
| ianw | i've vaguely thought about this as an escape hatch for opendev infra too, should we get to the point we can't get podman running inside the nodepool-builder containers | 20:30 |
| ianw | essentially pre-cache the root image and then use that | 20:30 |
| ianw | ... which in the end is, i guess, basically what a .qcow2 is ... | 20:31 |
| noonedeadpunk | Well, we're running dib with zuul, so we technically can select image where dib will run | 20:34 |
| noonedeadpunk | It creates some chicken-egg though... | 20:34 |
| clarkb | ianw: fwiw when containerfile stuff was first envisioned I thought that is what it was going to be. fetch the image and extract it then execute dib steps. But we modified that to fetch the image using Dockfile which can/does run a docker image build before the extraction step | 20:35 |
| noonedeadpunk | And root images usually are way more minimal then any qcow is | 20:35 |
| clarkb | ianw: if we stopped doing Dockerfile evaluation we could still use the container image as the "tarball" and extract it to the chroot without needing a container runtime | 20:35 |
| clarkb | I think skopeo can do this out of the box? | 20:36 |
| opendevreview | Merged opendev/system-config master: Remove v1 proxy from registry.zuul-ci.org https://review.opendev.org/c/opendev/system-config/+/877989 | 20:41 |
| ianw | right, i don't think we *need* to do anything in the dockerfile, it's just convenience | 20:42 |
| ianw | does skopeo want to setup any sort of cgroups or namespaces etc. to do that? it's the nesting of all that which causes problems. if it's just pulling a manifest via http and dumping it to a .tar.gz that seems much more nested-container friendly | 20:43 |
| ianw | and yeah, apart from just being huge, .qcow2's have traditional been a source of all sorts of problems too. files moving around, packages changing, bootloader layouts changing, etc. | 20:46 |
| clarkb | ianw: I'm not 100% sure, but it definitely can write an image out to a filesystem prefix iirc | 20:46 |
| ianw | so the smallest target with the least changing parts is the goal (as has been discussed above, i know that's not new info :) | 20:46 |
| * NeilHanlon has a master plan which involves a digraph of image dependencies and a single command that builds every variant of the images Rocky makes | 20:47 | |
| clarkb | https://github.com/containers/skopeo/blob/main/docs/skopeo-copy.1.md#examples | 20:47 |
| corvus | ianw: no worries about the state of the stack -- thank you for the test! and sorry for the regex error | 20:48 |
| clarkb | looks like you end up with each layer in a different tarball then presumably you can extract them in order to get a flattened filesystem result? | 20:48 |
| clarkb | infra-root I've just updated the meeting agenda with the updates I had. If you've got something to add now is a good time :) I'll send it out later today | 20:50 |
| corvus | ianw: clarkb fwiw, "use dockerfile syntax as an alternative to writing elements" was definitely a goal for the containerfile stuff. i think that's worth keeping, so if you want to mod it to do something else, maybe a new element for that would be in order? | 20:52 |
| clarkb | oh yup I think the Dockerfile interpretation has been useful. I can just see how the runtime requirements might make an alternative a good choice for the main distro elements | 20:53 |
| corvus | ++ | 20:53 |
| clarkb | it is a much more succint syntax for editing an image and if you as a user want/need that it is a useful tool | 20:53 |
| ianw | $ skopeo copy docker://busybox:latest docker-archive:archive-file.tar:busybox:latest | 20:54 |
| ianw | FATA[0000] initializing destination docker-archive:archive-file.tar:docker.io/library/busybox:latest: docker-archive doesn't support modifying existing images | 20:54 |
| corvus | yes, i have used it to build like 5 line dockerfile images | 20:54 |
| corvus | ianw clarkb : the nodepool-in-zuul work will also eventually address the cross-platform build case by running dib in standard zuul/nodepool vms | 20:54 |
| corvus | so that's another escape valve we should have | 20:54 |
| ianw | that's true, but also i don't know if we have the resources to keep dib working as a build *host* on many different platforms | 20:55 |
| clarkb | corvus: that assumes that everyone is running dib in zuul though | 20:56 |
| corvus | (i'm expecting to start in earnest on the zuul work after the openstack state machine driver finishes stabilizing) | 20:56 |
| clarkb | I think for zuul users its an outlet, but dib has users that run it outside of zuul and that is a use case we shouldn't drop | 20:56 |
| corvus | oh sure, i was responding to ianw's opendev escape valve | 20:56 |
| clarkb | ah | 20:56 |
| corvus | (so, in the future, opendev won't need to run nb03 or whatever because it can just run dib on a real arm vm) | 20:57 |
| corvus | ianw: point taken; i think all options are open to us this way. we can build fedora-x86-via-containerfile on a debian-x86 zuul vm; and we can build ubuntu-arm-via-debootstrap on a debian-arm zuul vm. | 21:01 |
| corvus | (iow we can cross-build with dib where it makes sense -- keep the supported host systems small; and native-build where it's near-impossible to avoid, like arm) | 21:02 |
| fungi | though jobs which need dib's target platform to run dib to build new images does pose a bit of a catch-22 | 21:04 |
| fungi | albeit we solve similar problems manually now | 21:04 |
| corvus | fungi: yeah, i anticipate we would use cloud images for the image build jobs -- either in all cases, or perhaps only when bootstrapping. depending on whichever we think is easier. | 21:08 |
| clarkb | last call for meeting agenda updates | 23:39 |
| ianw | fungi: if you have a sec, would you mind checking https://review.opendev.org/c/openstack/project-config/+/875997/3 which is a linter update for submit-requirements. fairly straight forward i think, but a double check would be good incase i missed a corner case or something | 23:41 |
| fungi | ianw: the function keyword got moved out of alpha order in the list, was that intentional? | 23:45 |
| clarkb | because I'm looking at the agenda. fungi do you know if it is appropriate for openstack to adopt xstatic libs given their apparent deal with the moinmoin folks? | 23:47 |
| clarkb | it seems like maybe xstatic libs should not be in openstack if they are comanaged by people who hvae zero interest in gerrit? | 23:47 |
| ianw | fungi: there is no reason for that. i think it just got mixed up when i shuffled the changes around | 23:47 |
| fungi | i think it depends on the lib | 23:47 |
| fungi | some are maintained exclusively by the horizon team | 23:48 |
| clarkb | oh I see it wasn't a universal shared setup? | 23:49 |
| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : check for function/s-r in normalize https://review.opendev.org/c/openstack/project-config/+/875997 | 23:49 |
| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acl : check for capital booleans in normalize https://review.opendev.org/c/openstack/project-config/+/877571 | 23:49 |
| opendevreview | Ian Wienand proposed openstack/project-config master: openstack/release : return to non-blocking submit rule https://review.opendev.org/c/openstack/project-config/+/877721 | 23:50 |
| clarkb | any idea if this one that is being renamed is shared? I guess that is what we need to sort out | 23:50 |
| clarkb | ok the agenda is official and will be in your inbox momentarily | 23:53 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!