| amorin | hey team, fungi I received your mail, will do the necessary to extend | 08:47 |
|---|---|---|
| frickler | amorin: cool, thx for the update, so we can cancel switching off things | 08:49 |
| frickler | infra-root: due to $things I won't be around next week. I'll still try to make it to the "early" (UTC afternoon) preptg sessions | 09:02 |
| amorin | frickler, fungi, clarkb all done on ovh billing side, should be good, let me know | 10:49 |
| fungi | thanks amorin! | 13:06 |
| GhostLyr_ | Hello! Excuse me, is there any way to contact one of the contributors to openstack/etcd3gw? I was wondering if this is open for community contributions. There are several issues in the associated launchpad project but it seems nobody from the project ever interacted with reported bugs in any way, despite even patches being available for some of them. | 13:06 |
| GhostLyr_ | Meanwhile there have been commits to the git repo, so I'm assuming it's not dead or abandoned. | 13:07 |
| fungi | GhostLyr_: openstack's oslo team is responsible for that repository: https://governance.openstack.org/tc/reference/projects/oslo.html | 13:11 |
| GhostLyr_ | thank you, will move the discussion to their channel. <bows out> | 13:15 |
| corvus | clarkb: i'm thinking of letting zuul restart on the current master state (which will be 9.5.0 when i push the proposed tag); then tomorrow merge the refactor, do the zk state reset, and restart again. how's that sound? | 16:00 |
| corvus | ("letting zuul restart" == the normal scheduled auto restart) | 16:01 |
| clarkb | that seems reasonable to me. I do plan to review the rest of the refactor stack this morning. The big one was an easy rereview before breakfast but I'm going to sort out food before doing the others | 16:02 |
| fungi | i haven't been reviewing the refactor changes, but seems reasonable to me too | 16:02 |
| fungi | based on what i know if the situation anyway | 16:02 |
| clarkb | its a fairly bit change internally, but the test cases should have us well covered on behavior changes | 16:03 |
| clarkb | and most of the behavior changes should be limited to circular dependencies which we don't use | 16:03 |
| clarkb | amorin: Thank you! | 16:07 |
| clarkb | corvus: the main thing would be ensuring that the refactor doesn't merge until the entire restart is complete. Otherwise we could automatically pull in too new zuul halfway through | 16:14 |
| corvus | yep | 16:15 |
| opendevreview | Lajos Katona proposed openstack/project-config master: unmaintained: gerritbot notification for Networking https://review.opendev.org/c/openstack/project-config/+/908606 | 16:46 |
| opendevreview | Jeremy Stanley proposed opendev/zone-opendev.org master: Switch the keycloak CNAME to the new server https://review.opendev.org/c/opendev/zone-opendev.org/+/908357 | 17:27 |
| opendevreview | Jeremy Stanley proposed opendev/zone-opendev.org master: Add DNS entries for another new Keycloak server https://review.opendev.org/c/opendev/zone-opendev.org/+/908608 | 17:27 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Update Zuul auth config for new Keycloak images https://review.opendev.org/c/opendev/system-config/+/908353 | 17:35 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Add backups for the new Keycloak server https://review.opendev.org/c/opendev/system-config/+/908359 | 17:35 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Clarify testinfra socket name for keycloak rdbms https://review.opendev.org/c/opendev/system-config/+/908364 | 17:35 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Inventory entry for another new Keycloak server https://review.opendev.org/c/opendev/system-config/+/908609 | 17:35 |
| fungi | infra-root: the new new keycloack replacement's changes are incorporated into the stack now, and i've updated https://etherpad.opendev.org/p/keycloak-refresh-2024 with references to the updated changes | 17:40 |
| clarkb | ack I'll rereview when I'm through with this zuul refactor stack | 17:40 |
| fungi | thanks! getting 908608 and 908609 merged asap will help me work out if there's anything else wrong that wasn't caught in testing | 17:40 |
| fungi | basically once 908609 deploys i should be able to start following the instructions from the zuul docs for configuring the realm | 17:41 |
| clarkb | oh I can review those now then while I'm between zuul changes | 17:46 |
| fungi | yeah, it's just dns and inventory | 17:47 |
| fungi | should be very quick to check, hopefully | 17:47 |
| fungi | everything else was just rebased onto those (with a couple of edits to switch from the previous replacement to the newer one) | 17:48 |
| clarkb | you can probably self approve those if frickler and corvus don't have time this morning. Its just a like for like replacement in existing changes to swap the new unused hosts out | 17:48 |
| fungi | right, and for a server which never saw production (never even successfully deployed) | 17:48 |
| fungi | also has probably been breaking other deploy jobs | 17:48 |
| fungi | now that i think about it | 17:49 |
| corvus | i noticed that the centos-9 stream image builds are failing in nodepool and diskimage functional test jobs. i don't plan on digging into that, but thought i would mention it here in case other folks were interested | 17:49 |
| corvus | https://zuul.opendev.org/t/openstack/build/dffdfd61d92e4e5d8f72abb67d4bcc95/log/nodepool/builds/test-image-09a333eaa8194b198524b2f0ac3cab05.log#369 | 17:50 |
| corvus | sample failure ^ | 17:50 |
| corvus | seems likely to affect prod too | 17:50 |
| clarkb | at first glance that looks like a bug in dnf | 17:50 |
| corvus | agreed | 17:51 |
| fungi | thanks for the heads up. amoralej and apevec had worked out an updated mirror host for us, so maybe this is from a deferred package update that has now landed with the updated mirror content | 17:51 |
| clarkb | fungi: I'm not sure we use our mirrors in those jobs but I'm checking | 17:51 |
| clarkb | oh hrm centos 8 fails with the same error. This might be a dnf bug in bookworm | 17:52 |
| fungi | i have a feeling it's a version mismatch, yeah | 17:53 |
| clarkb | this is the debootstrap like step where we jumpstart things | 17:53 |
| clarkb | so ya I don't think this is related to changes in our centos mirroring as this is a bookworm packge I think | 17:53 |
| fungi | makes sense, i'm looking to see what updated and when | 17:54 |
| apevec | ah nodepool builder uses dnf from Debian | 17:54 |
| fungi | right, it's a debian container running diskimage-builder which then builds images for centos, ubuntu, suse, gentoo, whatever | 17:55 |
| clarkb | apevec: sort of. Nodepool is executing diskimage builder. But they are all installed on debian bookworm in the images we use so using dnf from there | 17:55 |
| clarkb | you can install disk image builder elsewhere and it will use different dnfs in that case. But our installation is set up this way | 17:55 |
| clarkb | digging through debian package changelogs I don't see any recent updates. But maybe I haven't found the correct package yet | 17:58 |
| apevec | so this is in debian, not centos root: | 17:59 |
| apevec | 2024-02-08 14:32:50.604 | self.command.configure()... (full message at <https://matrix.org/_matrix/media/v3/download/matrix.org/xniBpeTriqDsiaPKVEruUvzC>) | 17:59 |
| fungi | https://tracker.debian.org/pkg/dnf | 17:59 |
| fungi | nothing recent | 17:59 |
| clarkb | apevec: yes I think so. Because that command is running the download fetch of the initial chroot environment. Once that is done it switches to the chroot and uses the fedora/centos provided dnf | 18:00 |
| clarkb | fetching docker://insecure-ci-registry.opendev.org:5000/quay.io/zuul-ci/nodepool-builder:a485bf250dbc4cf287e6009f6f1c5348_siblings should allow us to inspect the bookworm installation directly | 18:01 |
| fungi | the tip of the backtrace is from a file in this package, but also hasn't changed in debian recently: https://tracker.debian.org/pkg/dnf-plugins-core | 18:01 |
| apevec | yeah, on RHEL/CS path would be /usr/lib/python3.*/site-packages/... | 18:02 |
| fungi | no relevant bugs reported (yet) in debian against either the dnf nor dnf-plugins-core source or binary packages | 18:04 |
| fungi | nor the python3-dnf binary package (which is what supplies dnf.util) | 18:08 |
| clarkb | is it possible that dnf is patching itself at runtime via the upstream? (that seems crazy but may be possible) | 18:11 |
| apevec | yeah, crazy it would be, at least in RPM it would break rpm --verify ... | 18:13 |
| apevec | to test the theory, what does dpkg --verify say? | 18:14 |
| fungi | just as a check, i can confirm that the current packaged python3-dnf in debian does not have a _is_file_pattern_present in its dnf.util, just playing around with it in the repl | 18:18 |
| clarkb | I'm struggling to download that container image with docker locally | 18:19 |
| clarkb | says there is a 500 error somewhere but it isn't clear if dockerd or the remote registry is returning that | 18:19 |
| fungi | interestingly, the /usr/lib/python3/dist-packages/dnf-plugins/download.py from dnf-plugins-core doesn't appear to have any calls to dnf.util._is_file_pattern_present() in my version | 18:20 |
| clarkb | https://zuul.opendev.org/t/openstack/build/a485bf250dbc4cf287e6009f6f1c5348/log/job-output.txt you can see it installing dnf in this log | 18:21 |
| clarkb | corvus: fyi I wasn't able to `docker run -it --rm thaturlabove bash` to fetch the artifact out of the registry. Not sure why yet but that may interest you. POssibly related to the version negotiation that skopeo is broke on too? | 18:23 |
| clarkb | fungi: apevec https://zuul.opendev.org/t/openstack/build/a485bf250dbc4cf287e6009f6f1c5348/log/job-output.txt#18277 | 18:23 |
| fungi | it looks like the DownloadCommand.configure() method in my copy doesn't attempt to check for file presence | 18:23 |
| clarkb | we aren't using the package for dnf-plugins-core. I think we can clean that up. I'll work on a patch | 18:24 |
| fungi | d'oh! | 18:24 |
| fungi | version mismatch it is | 18:24 |
| fungi | good find | 18:25 |
| clarkb | now if debian package search would stop 503'ing I can confirm I have the correct package | 18:27 |
| fungi | yes, unfortunately packages.debian.org is down, i was noticing that earlier | 18:27 |
| fungi | the database backend anyway | 18:28 |
| fungi | i was going to ask about it in #debian-devel but figure people know and it will be up when it's up | 18:28 |
| apevec | ahh so I guess dnf-plugins was git cloned b/c it was not packaged in Debian ? | 18:29 |
| apevec | good catch clarkb++ | 18:29 |
| clarkb | correct | 18:29 |
| fungi | it's packaged in debian, but maybe once upon a time it wasn't or something we needed wasn't included | 18:29 |
| apevec | if still not packages, we'd need to clone matching tag at least | 18:30 |
| clarkb | fungi: what is the new package name? debian is 503ing for me | 18:30 |
| fungi | clarkb: dnf-plugins-core | 18:30 |
| clarkb | thanks | 18:30 |
| fungi | i linked to the tracker page for it a ways up in scrollback too | 18:30 |
| fungi | which was my fallback since pdo wasn't working at the moment | 18:31 |
| clarkb | remote: https://review.opendev.org/c/zuul/nodepool/+/908613 Use Debian packaged dnf-plugins-core | 18:31 |
| fungi | +1 line, -11 lines. that's my kinda change | 18:31 |
| fungi | aha, so that was indeed a temporary hack 3 years ago while we were waiting for ftp-master to approve the dnf-plugins-core upload | 18:33 |
| fungi | probably could have cleaned that up when we upgraded the images from bullseye to bookworm | 18:34 |
| fungi | oh! actually it's not in bullseye, but it's in bullseye-backports | 18:36 |
| fungi | so we could have also used it with bullseye (not relevant now) | 18:38 |
| fungi | it was available in bullseye-backports as of 2023-01-17, roughly 5 months before bookworm was released | 18:40 |
| fungi | but still a couple of years after we initially needed it | 18:40 |
| opendevreview | Merged opendev/zone-opendev.org master: Add DNS entries for another new Keycloak server https://review.opendev.org/c/opendev/zone-opendev.org/+/908608 | 18:41 |
| clarkb | ya and this was missed when we bumped to bookworm | 18:43 |
| clarkb | might be good to annotate things like that with a "should be fixed in bookworm" or similar to help our eyeballs catch them | 18:43 |
| fungi | agreed, if we came up with a standard we could even make stuff like that greppable so it's easier to spot when we're doing updates | 18:44 |
| opendevreview | Jay Faulkner proposed openstack/project-config master: ironic-unmaintained-core access to all Ironic proj https://review.opendev.org/c/openstack/project-config/+/908647 | 18:48 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Remove command.warn usage https://review.opendev.org/c/zuul/zuul-jobs/+/908671 | 19:26 |
| clarkb | fungi: does https://zuul.opendev.org/t/zuul/build/41936f28d7f04c47ac575f63394d2256/log/job-output.txt#9492-9494 imply zypper and dnf-plugins-core are not coinstallable? | 19:38 |
| clarkb | wow it does thats I don't understand | 19:40 |
| clarkb | `apt show dnf-plugins-core` says Breaks: zypper Replaces: zypper | 19:40 |
| clarkb | they should be completely independent tools | 19:41 |
| corvus | clarkb: i took one step in debugging registry: https://paste.opendev.org/show/bmAmz72srOWV71sUThCQ/ | 19:41 |
| clarkb | dnf and zypper use the same dep solver iirc | 19:41 |
| corvus | clarkb: (first part of the paste is the traceback; the second part is a python repr of the contents of that object in storage) | 19:42 |
| corvus | (obtained out of band) | 19:42 |
| clarkb | corvus: using a new datastructure? I wonder if that is related to the version changes that skopeo hit then | 19:42 |
| opendevreview | Merged opendev/system-config master: Inventory entry for another new Keycloak server https://review.opendev.org/c/opendev/system-config/+/908609 | 19:42 |
| clarkb | I think there may be a way to force docker to use the older version and we can use that to see if it resolves if so | 19:42 |
| corvus | clarkb: could be; i'm unsure and have not paged everything back in | 19:42 |
| corvus | clarkb: though i have a hunch this may relate to multi-arch image formats, and zuul-registry may need an update to handle them | 19:45 |
| clarkb | oh i thought it already did? | 19:46 |
| clarkb | maybe not in the way that docker wants it to though | 19:46 |
| corvus | yes that | 19:47 |
| clarkb | fungi: `apt-file list zypper` and `apt-file list dnf-plugins-core` don't show any overlap in files. I'm not sure how dnf-plugins-core can break zypper. Seems like a bug in the package for dnf-plugins-core? | 19:47 |
| corvus | clarkb: podman run that image works | 19:47 |
| corvus | podman run --rm -it insecure-ci-registry.opendev.org:5000/quay.io/zuul-ci/nodepool-builder:a485bf250dbc4cf287e6009f6f1c5348_siblings | 19:48 |
| clarkb | got it so protocol level issues likely rather than any true problem | 19:48 |
| clarkb | remote: https://review.opendev.org/c/zuul/nodepool/+/908613 Fetch compatibile dnf download command in container image | 19:57 |
| clarkb | this makes me sad. I really wonder why the package would replace and break zypper. There is no zypper plugin in the plugins list that I can see. | 19:57 |
| clarkb | the changelog for dnf-plugins-core says the breaks zypper was added to fix 1028085 | 19:59 |
| clarkb | https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1028085 | 20:00 |
| clarkb | ok they conflict for the needs restarting manpage | 20:01 |
| clarkb | but dnf-plugins-core doesn't seem to install that command? | 20:01 |
| clarkb | oh it is a dnf plugin | 20:02 |
| clarkb | I feel like there has to be a better way to address this problem. Maybe the manpage should be dnf-needs-restarting.gz.1 ? | 20:03 |
| clarkb | as far as I can tell you can't execute it without the dnf prefix but I'm installing things now to check | 20:04 |
| corvus | the containerfile element was designed to avoid bootstrapping issues; maybe one or more of these images should switch to using that? | 20:05 |
| clarkb | ya `man needs-restarting` works but `needs-restarting` is container not found. | 20:05 |
| corvus | (just brainstorming/reminding; i haven't thought about the particulars) | 20:06 |
| clarkb | corvus: ya I think that is a longer term posibility. I think the debian package is broken too imo but I don't now what the rules are for man 1 presence for commands that don't actually exist | 20:06 |
| corvus | clarkb: container not found? | 20:06 |
| clarkb | sorry command not found | 20:06 |
| clarkb | bash: needs-restarting: command not found | 20:06 |
| corvus | clarkb: oh you have to docker your docker before you docker | 20:07 |
| clarkb | zypper acutally installs a /usr/bin/needs-restarting and installs a needs-restarting.gz.1 file. dnf installs a plugin you can run as `dnf needs-restarting` then installs a manpage for needs-restarting.gz.1 | 20:07 |
| clarkb | my "I don't actually know debian rules for this stuff" opinion is that the dnf-plugins-core package is buggy and it should not install all these manpages for dnf subcommands at the root manpage space | 20:08 |
| corvus | based on only that sentence, i agree that sounds like the dnf package is not doing something right. | 20:08 |
| clarkb | using git as an example beacuse it has many subcommands all of those commands are git-foo.1.gz | 20:10 |
| clarkb | fungi: I suspect you know the rules and conventions and I'll defer to you on whether or not we should complain and file a bug | 20:11 |
| fungi | clarkb: i have a feeling it's that red hat's and suse's rpm tools can't coexist | 21:06 |
| clarkb | fungi: the only conflict is a manpage though | 21:06 |
| clarkb | sure they can't exist if the manpages have to be in their current locations. But nothing about the tools themselves conflcit aiui | 21:06 |
| clarkb | you can even install them together on tumbleweed | 21:07 |
| clarkb | dnf and dnf-plugins-core are valid pacakges and I already have zypper installed | 21:07 |
| clarkb | and zypper info --conflicts for both don't show zypper conflicting. | 21:08 |
| fungi | oh that's funny | 21:09 |
| fungi | going to see if i can tell when/why the conflict was added. normally package conflicts shouldn't be used for unrelated tooling | 21:10 |
| clarkb | fungi: the changelog has it | 21:10 |
| clarkb | https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1028085 is related though it seems like they were already making the update before that issue was filed? | 21:10 |
| clarkb | the tumbleweed package has a deconflicts clause in the specifle https://build.opensuse.org/package/view_file/openSUSE:Factory/dnf-plugins-core/dnf-plugins-core.spec?expand=1 | 21:12 |
| clarkb | mv %{buildroot}%{_mandir}/man1/needs-restarting.1 %{buildroot}%{_mandir}/man1/dnfutils-needs-restarting.1 | 21:12 |
| clarkb | so ya tumbleweed is doing what I suggest debian do above | 21:13 |
| fungi | it's in section 1, so in theory is the manpage for a command called dnfutils-needs-restarting | 21:17 |
| fungi | but if both packages ship that manpage, it implies they both also ship that executable? | 21:17 |
| fungi | except it sounds like they don't, so one or the other is probably including that manpage in error | 21:18 |
| clarkb | the manpage that conflicts is for `needs-restarting`. On debian only zypper installs this in /usr/bin/ to use this command with dnf on debian you have to run `dnf needs-restarting` | 21:20 |
| clarkb | on tumbleweed it looks like with conflicts they both install a binary for needs-restarting and when deconflicted it gets moved to dnfutils-needs-restarting | 21:20 |
| clarkb | the debian package includes a number of dnf plugin subcommands as top level manpages that don't actualyl appear to get installed as top level binaries | 21:21 |
| fungi | oh weird | 21:21 |
| fungi | yeah that's an interesting corner case | 21:21 |
| clarkb | my "never built a distro" naive take is that none of those manpages in the dnf-plugins-core package should lack dnf- prefixes | 21:22 |
| clarkb | because none of them are installed as actual commands and instead they are `dnf foobar` commands | 21:22 |
| fungi | git sets the same precedent for its subcommands though | 21:23 |
| clarkb | ah yup tumbleweed will install a shim that checks $0 and acts accordingy for subcommands if you run them directly | 21:23 |
| clarkb | fungi: git does the thing that would fix this | 21:23 |
| clarkb | they are all git-foo not foo | 21:23 |
| fungi | i mean git sets the same precedent for manpages on its subcommands | 21:24 |
| fungi | man git-commit | 21:24 |
| fungi | et cetera | 21:24 |
| clarkb | there is a /usr/libexec/dnf-utils-3 on debian that I wonder if it does something weird | 21:24 |
| clarkb | liek make needs-restart work somehow but `which` couldn't find it so its not using PATH | 21:24 |
| clarkb | als /usr/share/man/man8/dnf-needs-restarting.8.gz exists | 21:25 |
| clarkb | in any case there shouldn't be anything that actually prevents these two packages from installing next to each other as tumbleweed packaging shows. But it may require moving a manpage out of the way | 21:26 |
| clarkb | testing again I can't run `needs-restarting` on bookworm directly. So no magic happening as far as I can tell | 21:27 |
| clarkb | you'd need something like suse's dnf-utils to interpret the command and fork out to the correct subcommand | 21:28 |
| clarkb | fungi: I think I'm caught up on keycloak reviews. Let me know if I missed one | 21:29 |
| clarkb | `mv %{buildroot}%{_libexecdir}/dnf-utils-3 %{buildroot}%{_libexecdir}/dnf-utils` from the tumbleweed spec. So that /usr/libexec/dnf-utils-3 tool is the glue but it isn't installed to do the gluing as far as I can tell | 21:33 |
| clarkb | (I actually installed the packages on the container in case some packaging scripts did it but doesn't appear to) | 21:33 |
| clarkb | this is likely to be a very lightly used set of packages so probably not urgent upstream if willing to address ti at all. And we've got a reasonable workaround for now I think | 21:40 |
| fungi | hot take on #debian-devel is that it's a policy violation to add a conflicts to a package rather than rename or omit the conflicting file | 21:40 |
| clarkb | oh itneresting | 21:41 |
| fungi | conflicts is intended to be reserved for packages that absolutely cannot be used on the same system | 21:41 |
| fungi | which was also my vague recollection, so seems corroborated at least | 21:50 |
| fungi | clarkb: there's also the cname change, not at all urgent: https://review.opendev.org/908357 | 23:46 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!