| fungi | oh, great point! | 00:07 |
|---|---|---|
| opendevreview | Merged openstack/project-config master: Use https with apt in our ubuntu image builds https://review.opendev.org/c/openstack/project-config/+/840392 | 00:19 |
| ianw | fungi: so i think the idea is to use "debuild --no-sign" then the promote/upload job should call debsign on the artifacts | 00:21 |
| fungi | yeah, that's what i would expect | 00:23 |
| fungi | we could sign in the build and then check and replace the signature in upload, but that's pretty pointless as it either means ephemeral first keys which aren't really conveying anything or more than one key we have to manage | 00:24 |
| fungi | the channels between the artifact upload and download are already reasonably well-secured | 00:25 |
| fungi | ianw: i would specifically expect `debuild -S -us` | 00:27 |
| fungi | assuming you only want to build a source package and not sign the result | 00:28 |
| ianw | i think you need -sa to upload the source tarball which ppa wants | 00:30 |
| fungi | isn't that what debsign will do? | 00:32 |
| *** rlandy is now known as rlandy|out | 00:33 | |
| ianw | ... maybe? | 00:41 |
| fungi | i thought -sa was "sign all" but the debuild manpage is not all that helpful | 00:42 |
| ianw | yeah that ends up debuild -> dpkg-buildpackage -> dpkg-genchanges | 00:42 |
| fungi | oh, because it's a wrapper for those, yep | 00:42 |
| ianw | a wrapper wrapper :) | 00:42 |
| fungi | and dpkg-buildpackage manpage says -sa is passed along to dpkg-genchanges | 00:43 |
| ianw | is "pass the parcel" an american kids game? | 00:43 |
| fungi | so `debuild -S -sa -us` i guess? | 00:44 |
| fungi | build only source package, include source package in changes file, don't sign source package | 00:47 |
| fungi | ahh, need -uc too (don't sign changes file) | 00:47 |
| ianw | yeah i got "debuild -S -sa --no-sign" | 00:48 |
| ianw | https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/840383 is building the packages, i just need to figure out copying something useful back | 00:48 |
| fungi | and --no-sign is equivalent to -uc -us, got it | 00:48 |
| *** mazzy5098812929580859495 is now known as mazzy509881292958085949 | 00:59 | |
| ianw | viola -> https://zuul.opendev.org/t/openstack/build/7065fd1000bf4872b8b7cabc5debda25/logs | 01:02 |
| fungi | christine's growing a bunch of violas on our deck | 01:04 |
| fungi | also i had a friend in high school who played the viola | 01:06 |
| fungi | huh, foldoc says there was also an early hypercard-like interpreted text language out of berkeley called viola | 01:08 |
| ianw | music to my ears :) | 01:22 |
| diablo_rojo_phone | Looks good to me @clarkb | 01:51 |
| *** ysandeep|sick is now known as ysandeep | 05:02 | |
| *** marios is now known as marios|ruck | 05:05 | |
| *** prometheanfire is now known as Guest0 | 06:23 | |
| opendevreview | Merged opendev/system-config master: etherpad: remove session key https://review.opendev.org/c/opendev/system-config/+/804466 | 06:27 |
| opendevreview | Merged opendev/system-config master: Remove open-vm-tools from servers https://review.opendev.org/c/opendev/system-config/+/596254 | 06:40 |
| *** jpena|off is now known as jpena | 06:59 | |
| *** pojadhav is now known as pojadhav|lunch | 07:38 | |
| frickler | wheel builds seem to have been going well, all wheel volumes released 3h ago | 07:43 |
| *** pojadhav|lunch is now known as pojadhav | 07:58 | |
| *** ysandeep is now known as ysandeep|lunch | 08:27 | |
| frickler | oh, that explains so many matrix things flapping https://matrix.org/blog/2022/05/04/0-34-0-security-release-for-matrix-appservice-irc-high-severity/ | 08:43 |
| *** ysandeep|lunch is now known as ysandeep | 09:42 | |
| *** rlandy|out is now known as rlandy | 10:15 | |
| *** dviroel|out is now known as dviroel | 11:20 | |
| *** pojadhav is now known as pojadhav|brb | 12:04 | |
| *** ysandeep is now known as ysandeep|coffee | 12:22 | |
| fungi | #status log Retired 11 OpenStack mailing lists which were unused for 3 years or longer: http://lists.openstack.org/pipermail/openstack-discuss/2022-May/028406.html | 13:00 |
| opendevstatus | fungi: finished logging | 13:00 |
| *** ysandeep|coffee is now known as ysandeep | 13:01 | |
| *** pojadhav|brb is now known as pojadhav | 13:03 | |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: elastic-recheck: allow releasers to merge/delete https://review.opendev.org/c/openstack/project-config/+/840455 | 13:20 |
| *** ysandeep is now known as ysandeep|brb | 13:42 | |
| jrosser | just fyi i have ubuntu-jammy jobs failing with `libsystemd-dev : Depends: libsystemd0 (= 249.11-0ubuntu3) but 249.11-0ubuntu3.1 is to be installed` | 13:54 |
| Clark[m] | jrosser: is that a bug in the packages? If so we're probably waiting on upstream to fix and for our mirror to sync that | 14:03 |
| frickler | saw that in devstack tests, too | 14:03 |
| jrosser | i'm not sure actually, i was just trying to reproduce it in a VM | 14:03 |
| frickler | the pkgs were updated yesterday, from the mirror logs everything looks fine | 14:03 |
| jrosser | and i'm having a TIL moment with https://discourse.ubuntu.com/t/phased-updates-in-apt-in-21-04/20345 | 14:04 |
| jrosser | re. 249.11-0ubuntu3.1 1 (phased 30%) | 14:04 |
| frickler | https://zuul.opendev.org/t/openstack/build/a869fae1f87e44c1867cb098182f532b as example | 14:04 |
| jrosser | yes thats exactly whats happing in my OSA jobs too | 14:05 |
| frickler | jrosser: where did you see that 30%? fun | 14:05 |
| jrosser | https://paste.opendev.org/show/bf3C82BDF9sGjrFxBT4E/ | 14:05 |
| Clark[m] | We should probably disable phased updates on our VMs so that they produce consistent results from one test to another. However, we may already do that due to the machine-id file? I don't recall if our VMs set that | 14:08 |
| jrosser | i just set this APT::Get::Always-Include-Phased-Updates "true"; | 14:10 |
| jrosser | and i can locally update to 249.11-0ubuntu3.1 without trouble | 14:10 |
| Clark[m] | jrosser: if I'm reading correctly the systemd-dev package is lagging the phased libsystemd update? That definitely seems like an Ubuntu packaging bug. But I think for CI we don't want phasing and it should be disabled in our image builds | 14:10 |
| frickler | ykarel: ^^ fyi | 14:10 |
| jrosser | there is something odd going on, as with the package held back in my VM by the phasing i didnt get an error | 14:11 |
| jrosser | and disabling the phasing made the install work as expected | 14:11 |
| Clark[m] | Ya because the -dev package wants the older version of libsystemd only | 14:11 |
| jrosser | right | 14:11 |
| Clark[m] | Holding back that update makes the -dev package happy | 14:11 |
| jrosser | agreed that disabling the phasing is a good idea | 14:12 |
| ykarel | frickler, ack Thanks | 14:12 |
| Clark[m] | When you allow phased updates it pulls in the phased .1 update and breaks -dev. This is almost certainly a bug in phased updates. But we don't want them anyway in CI IMO | 14:12 |
| Clark[m] | It is probably worth reporting the issue to Ubuntu then also updating our images. I can take a look at that after school runs if no one else has done it yet | 14:13 |
| *** ysandeep|brb is now known as ysandeep | 14:20 | |
| frickler | I think I'm going to hold a node for the devstack job to see the exact situation on our images | 14:29 |
| jrosser | frickler: i think comparing apt policy for libsystemd-dev and libsystemd0 would be interesting | 14:31 |
| frickler | jrosser: yep. plus I can live test the different Phased-Updates settings | 14:32 |
| Clark[m] | ++ | 14:38 |
| fungi | thanks for running that down | 14:39 |
| fungi | in debian you see that all the time in sid (which is the main reason it's referred to as "unstable") but migration of packages to the testing suite requires those interdependencies to be satisfied and lock-step transitions like that happen all at once | 14:40 |
| fungi | i'm surprised this is something ubuntu doesn't catch before packages get into the archive | 14:41 |
| jrosser | in this case i think they come from the same source package, which makes it stranger still | 14:41 |
| frickler | fungi: I think the issue happens when our images install the updated lib, but phased apt decides it doesn't want the updated dev lib yet | 14:41 |
| fungi | yeah, but i would definitely believe that there are corner cases where they've introduced such bugs in their phased update scheme | 14:41 |
| frickler | https://paste.opendev.org/show/bdeolr7cbwqYmdnJ4HQk/ | 14:41 |
| frickler | asked in #ubuntu-server now | 14:42 |
| fungi | that makes sense. we've also seen this before when our mirrors lag behind our image builds, though we mostly addressed it by building our images from our mirrors | 14:42 |
| frickler | fungi: the thing is that this issue happens even then | 14:43 |
| fungi | yeah, implying it's selecting older packages than we have available on the mirror | 14:43 |
| frickler | because the decision about the phase is different during image build than when the image is executed | 14:44 |
| Clark[m] | That post says no phasing in a chroot though and our builds are chroots | 14:44 |
| fungi | and i agree, "phased" updates is designed to result in inconsistencies, while we want consistency | 14:44 |
| frickler | o.k., going to test with APT::Get::Always-Include-Phased-Updates "true"; now | 14:45 |
| fungi | Clark[m]: right, that's probably part of this. we always pull the most recent in the image build, but the vm booted from it has a 90% chance of wanting something older | 14:45 |
| Clark[m] | But newer should only happen if phased is enabled? Without phasing you are always older? Or is it the opposite? If opposite that seems completely broken | 14:47 |
| fungi | reading more | 14:49 |
| Clark[m] | I wonder if bullseye debootstrap is clueless so we get newer rather than older without phasing | 14:52 |
| fungi | "otherwise phasing will be disabled, and phased updates always included." | 14:53 |
| fungi | that implies to me that the behavior in a chroot is to always include the new package as well | 14:54 |
| fungi | disabling phasing seems to imply always installing the new package, while phasing enabled adds the delay | 14:54 |
| Clark[m] | Wow that seems broken to me but that would explain it | 14:54 |
| Clark[m] | Seems like you'd rather phase opt ins in over time then flip a flag that says everyone gets the package after $time | 14:56 |
| Clark[m] | But that would confirm disabling phasing in our images should fix it. Initially I thought our elements should do it since consistency is a CI feature but with that behavior I think DIB should do it | 14:57 |
| opendevreview | Dr. Jens Harbott proposed openstack/project-config master: Always include phased updates for Jammy https://review.opendev.org/c/openstack/project-config/+/840502 | 14:58 |
| frickler | Clark[m]: fungi: tested manually on 198.72.124.32 if you want to have a look there ^^ | 14:58 |
| Clark[m] | frickler: ^ maybe also push an update to dib then we can remove it from our side when dib releases with that update? | 14:58 |
| frickler | ah, yes, I can do that later, I was going for the quick win first ;) | 14:59 |
| * frickler will have a break, feel free to update if needed | 14:59 | |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Always include phased updates for Jammy https://review.opendev.org/c/openstack/project-config/+/840502 | 15:16 |
| *** dviroel is now known as dviroel|lunch | 15:18 | |
| *** ysandeep is now known as ysandeep|out | 15:19 | |
| clarkb | fungi: frickler: can we set that flag if on ubuntu and debian regardless of release? I assume older apt will ignore the config and newer apt will do the right thing? | 15:19 |
| clarkb | mostly concerned that with the next debuntu release we'll have to go through this again if we pin it to jammy | 15:20 |
| fungi | i have doubts debian will make use of phased updates, but per my review comment i agree we should at least enable it for all future ubuntu versions starting with the one where they added it, but i agree it's probably safe to just set it everywhere since it should be ignored by old releases anyway | 15:21 |
| clarkb | do we want to update that now or land this and do that ina followup? | 15:22 |
| clarkb | I guess we can land this and then fix it in dib properly | 15:22 |
| fungi | i would land this now and do a more thorough fix in dib instead | 15:22 |
| clarkb | since dib really wants this anyway | 15:22 |
| clarkb | +A and then ya we can followup in dib to do more future proofing | 15:24 |
| opendevreview | Merged openstack/project-config master: Always include phased updates for Jammy https://review.opendev.org/c/openstack/project-config/+/840502 | 15:32 |
| frickler | clarkb: fungi: I wasn't sure whether old apt like in xenial would complain, will test before proposing it for dib | 15:32 |
| fungi | i suppose we can force new jammy images once that's deployed? | 15:34 |
| opendevreview | Clark Boylan proposed openstack/diskimage-builder master: Add Jammy functesting to dib https://review.opendev.org/c/openstack/diskimage-builder/+/840391 | 15:35 |
| clarkb | fungi: ya triggering new builds is probably a good idea | 15:35 |
| clarkb | fungi: frickler any objections to sending https://etherpad.opendev.org/p/rdrjMFJuUe-uCjRgEjkr once my next meeting is over to announce ethercalc shutdown? | 15:35 |
| frickler | clarkb: go ahead with that | 15:36 |
| frickler | first feedback from #ubuntu-devel is that image builds should not be using phased updates | 15:37 |
| fungi | the implication seems to be that the newer (phased) versions always get installed if running apt in a chroot. is that a misinterpretation? | 15:38 |
| clarkb | and dib builds in a chroot. I suppose maybe we can undo the behavior explicitly? | 15:39 |
| fungi | also if running apt when no machine-id is yet present in /etc (which might be the case) | 15:39 |
| fungi | regardless of whether the newer phased versions are a good idea to include in virtual machine base images, they're probably a good idea for ci images because we'd rather catch behavior changes from the newer phased versions early | 15:40 |
| clarkb | but also we aren't using phased updates because we are in a chroot so we install the latest stuff without phasing and then break later outside of the chroot | 15:41 |
| clarkb | I think the ideal would be for image builds to do the most conservative thing in the general case then users can opt into newer stuff but it appears this system does not work in that manner? | 15:41 |
| clarkb | anyway we build in a chroot so shouldn't be phasing anything and that seems to be the root of the problem | 15:41 |
| clarkb | it might also make sense for the system to automatically fast forward in a phase if it already has packages from a newer phase. But I assume the mechanism here is too simple to solve for that | 15:44 |
| fungi | but also i wouldn't be surprised if we're initially bootstrapping packages with the older focal apt from outside the chroot, and could be installing newer packages at that time | 15:45 |
| *** marios|ruck is now known as marios|out | 15:45 | |
| clarkb | fungi: we run in a bullseye environment. But ya that could be too | 15:45 |
| clarkb | In any case I agree using the latest packages consistently without variance is ideal for a CI system | 15:46 |
| fungi | oh, good point, i forgot it's getting called from inside a container | 15:46 |
| fungi | and the deploy has succeeded | 15:47 |
| fungi | is the best way of forcing a new jammy image to nodepool dib-image-delete the oldest of the two ready ones? | 15:48 |
| fungi | actually we have three ready, one just came ready 16 minutes ago | 15:49 |
| clarkb | no that will just use the previous build instead while it builds a new image. You want the image build command instead | 15:49 |
| fungi | no, 16 hours ago. i can't read | 15:49 |
| clarkb | it will replace the image just as quickly but without reverting git cache state by using an older image | 15:49 |
| fungi | thanks! it's been submitted | 15:50 |
| fungi | i don't see it building yet, but i guess it has to wait for an available builder | 15:51 |
| clarkb | correct | 15:51 |
| fungi | might be nice if dib-image-list also reported pending builds | 15:51 |
| clarkb | it si building on nb01 now | 15:51 |
| clarkb | it does | 15:51 |
| fungi | ahh | 15:51 |
| clarkb | oh wait I see what you mean | 15:51 |
| clarkb | a queued state ++ | 15:51 |
| fungi | right, pending as in not yet building but needed | 15:52 |
| fungi | anyway, i guess the delay was just the round-trip time through zk and waiting for the available builder to notice | 15:52 |
| frickler | seems I messed up https://nb01.opendev.org/ubuntu-jammy-0000000011.log | 15:54 |
| frickler | /opt/dib_tmp/dib_build.emPIJ8tW/hooks/root.d/60-apt-phased-updates: line 30: syntax error near unexpected token `fi' | 15:55 |
| fungi | gah | 15:56 |
| clarkb | there is a missing then on line 26 | 15:56 |
| fungi | i didn't spot that either :/ | 15:56 |
| clarkb | unfortunate the linter can catch tab width problems but no if then else syntax issues | 15:56 |
| opendevreview | Dr. Jens Harbott proposed openstack/project-config master: Fix apt-phased-updates https://review.opendev.org/c/openstack/project-config/+/840507 | 15:57 |
| clarkb | aha "I hope we can remove that ischroot code path in 21.10, that should give people time to change their build chroots." | 15:57 |
| clarkb | man I wish this was better documented (as requested in that thread and got shut down) | 15:57 |
| clarkb | I guess ultimately we should control this explicitly in dib elements somewhere. The changes above are still good short term workarounds | 15:58 |
| fungi | but also it could be that the apt we're bootstrapping with is too old to notice the phased metadata, or that we're bootstrapping before the machine-id file is present | 15:58 |
| fungi | either of which could also result in the observed behavior | 15:59 |
| *** jpena is now known as jpena|off | 16:00 | |
| frickler | there shouldn't be a machine-id file in the image IMO, because then all VMs created from it would have the same | 16:01 |
| clarkb | ++ generally that is generated by cloud-init or similar. I'm not sure if glean does that but maybe it should | 16:02 |
| frickler | from #ubuntu-devel I gather that all of this works kind of like expected and our override seems to be a valid solution for our CI environment | 16:02 |
| jrosser | i've just run into the same thing with the way OSA builds container rootfs using debootstrap | 16:02 |
| jrosser | building a jammy rootfs on a jammy host got into exactly the same libsystemd trap | 16:03 |
| frickler | it would be interesting to see how ubuntu cloud images look like, but they haven't been built for jammy since the release | 16:03 |
| opendevreview | Merged openstack/project-config master: Fix apt-phased-updates https://review.opendev.org/c/openstack/project-config/+/840507 | 16:14 |
| *** dviroel|lunch is now known as dviroel | 16:15 | |
| frickler | this is looking better now, waiting for all the git updates to finish https://nb02.opendev.org/ubuntu-jammy-0000000035.log | 16:46 |
| opendevreview | Clark Boylan proposed openstack/diskimage-builder master: Add Jammy functesting to dib https://review.opendev.org/c/openstack/diskimage-builder/+/840391 | 17:09 |
| clarkb | ok I'm going to send that ethercalc email to service-announce@lists.opendev.org and openstack-discuss@lists.openstack.org now | 17:11 |
| fungi | thanks! | 17:13 |
| clarkb | done | 17:15 |
| opendevreview | Clark Boylan proposed openstack/diskimage-builder master: Add Jammy functesting to dib https://review.opendev.org/c/openstack/diskimage-builder/+/840391 | 17:33 |
| clarkb | fungi: in ^ I'm trying to install debootstrap/unstable with ansible's package module. It says no package debootstrap/unstable available. Do you think that is because I'm missing an apt-get update after adding the deb repo or is that likely the package module not understanding my request to install the unstable package? | 17:43 |
| clarkb | I guess I can just run a shell task and do it | 17:44 |
| clarkb | oh I need a .list extension in sources.list.d maybe that is it | 17:44 |
| opendevreview | Clark Boylan proposed openstack/diskimage-builder master: Add Jammy functesting to dib https://review.opendev.org/c/openstack/diskimage-builder/+/840391 | 17:45 |
| clarkb | I'll use the apt module if ^ fails as it can force a cache update | 17:46 |
| opendevreview | Clark Boylan proposed openstack/diskimage-builder master: Add Jammy functesting to dib https://review.opendev.org/c/openstack/diskimage-builder/+/840391 | 17:52 |
| fungi | looking | 17:57 |
| clarkb | thee is a default_release flag to the apt module I'll try next if this fails | 17:58 |
| fungi | in theory you should see the unstable url appear in the log when apt updates its indices | 17:59 |
| clarkb | ya but this is all behind ansible modules. I guess ultimately I can fallback to using a shell script | 17:59 |
| fungi | the modules are hiding the output from apt? | 18:00 |
| clarkb | I think so | 18:01 |
| clarkb | or at least summarizing it. I'm not sure if the update_cache flag will give you cache update logs | 18:01 |
| opendevreview | Clark Boylan proposed openstack/diskimage-builder master: Add Jammy functesting to dib https://review.opendev.org/c/openstack/diskimage-builder/+/840391 | 18:02 |
| clarkb | if ^ doesn't work I may have to try the shell script option | 18:02 |
| fungi | yeah, i mean it should be straightforward... add unstable to the list of package repositories, update the cache of indices, specify unstable when upgrading the package. done | 18:04 |
| clarkb | thats what I think I've done at least | 18:05 |
| fungi | that said, setting default_release in the apt module may cause it to pull all debootstrap's dependencies from unstable as well, rather than just debootstrap | 18:06 |
| fungi | you can tell apt "i want to install from unstable, install these things" or you can tell apt "i want to install these things and some of them are from unstable" | 18:06 |
| clarkb | ya I wonder if my configuration of the repo isn't right somehow | 18:07 |
| fungi | `apt install debootstrap/unstable` is an example of the latter | 18:07 |
| fungi | `apt install -t unstable debootstrap` is the former | 18:07 |
| clarkb | the second task in https://review.opendev.org/c/openstack/diskimage-builder/+/840391/8/roles/dib-ensure-debootstrap/tasks/main.yaml shoudl add the repo config. Then the update_cache: yes flag should pull its indexes | 18:08 |
| clarkb | fungi: anything obviously wrong in that second task? | 18:08 |
| clarkb | apt_pkg.Error: E:The value 'unstable' is invalid for APT::Default-Release as such a release is not available in the sources <- that would imply I'm doing something wrong with that setup | 18:09 |
| fungi | yeah, that looks right to me though | 18:10 |
| fungi | an example line from my workstation's /etc/apt/sources.list: | 18:10 |
| fungi | deb http://deb.debian.org/debian/ unstable main | 18:10 |
| fungi | the trailing / ideally shouldn't matter | 18:10 |
| fungi | but i have a feeling the update step isn't happening correctly | 18:11 |
| clarkb | did I use /etc/apt/sources.list.d wrong? permissions or something? | 18:11 |
| clarkb | I can have apt do an explicit update first | 18:11 |
| clarkb | without trying to install anything | 18:11 |
| fungi | yeah, i was going to ask if you're sure update_cache does an apt update before the apt install | 18:12 |
| clarkb | well the documentation implies it does but it may check the release options first | 18:13 |
| fungi | a custom package repository i add in the .d looks similar permissions and ownership wise: | 18:13 |
| clarkb | https://docs.ansible.com/ansible/latest/collections/ansible/builtin/apt_module.html#parameter-update_cache | 18:13 |
| fungi | -rw-r--r-- 1 root root 48 Jul 14 2019 /etc/apt/sources.list.d/weather.list | 18:13 |
| opendevreview | Clark Boylan proposed openstack/diskimage-builder master: Add Jammy functesting to dib https://review.opendev.org/c/openstack/diskimage-builder/+/840391 | 18:14 |
| clarkb | now with a standalone apt cache update | 18:14 |
| fungi | as a test that unstable is a recognised repository, i did `sudo apt install debootstrap/unstable` on my workstation and got: | 18:15 |
| fungi | Selected version '1.0.126+nmu1' (Debian:unstable [all]) for 'debootstrap' | 18:15 |
| clarkb | ya I took this from the nodepool image builds which run shell commands | 18:16 |
| clarkb | so it should work, this is purely an ansible problem at this point I think. I'll probably just run shell with ansible :/ | 18:17 |
| fungi | yes, package management is one of those things i'm not sure ansible's abstraction layer is needed for | 18:20 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Pull keycloak from quay.io https://review.opendev.org/c/opendev/system-config/+/840529 | 18:23 |
| clarkb | corvus: ^ fyi | 18:23 |
| clarkb | corvus: I'm not actually sure that is correct as we may have to run keycloak:legacy to keep using wildfly instad of whatever the nwe thing is. I'm not sure if you can transition from old thing to new thing like this trivially. but testing should at least give us some info | 18:24 |
| clarkb | also https://www.keycloak.org/server/containers has a lot more info about this as we dig in | 18:24 |
| clarkb | fungi: ok manual refresh seems to have worked but then it failed later trying to install podman I think beacuse we may have pulled in too much stuff from unstable | 18:26 |
| fungi | yeah, back to shell again, just install debootstrap/unstable instead of install -t unstable debootstrap | 18:26 |
| clarkb | I may try to do debootstrap/unstable after the explicit cache update but then ya time to do shell after if that breaks | 18:27 |
| opendevreview | Clark Boylan proposed openstack/diskimage-builder master: Add Jammy functesting to dib https://review.opendev.org/c/openstack/diskimage-builder/+/840391 | 18:27 |
| clarkb | and of course that fails wow | 18:35 |
| * clarkb breaks out the shell | 18:36 | |
| opendevreview | Clark Boylan proposed openstack/diskimage-builder master: Add Jammy functesting to dib https://review.opendev.org/c/openstack/diskimage-builder/+/840391 | 18:38 |
| clarkb | hrm that worked but the podman install failure from when we used -t unstable remains | 18:44 |
| clarkb | fungi: if you get a chance can you see if https://zuul.opendev.org/t/openstack/build/f4b39bab647640cfab64575656379989 makes sense to you? I'm wondering if somehow unstable is being used in that podman install so we're leaking | 18:44 |
| clarkb | or maybe libsemanage updated in the distro and podman packaging hasn't caught up yet? | 18:46 |
| clarkb | hrm no, version 3.3-1 is from unstable so it certainly appears to be a leak somehow | 18:46 |
| clarkb | oh weird libsemanage-common is explicitly installed from unstable in the nodepool docker image | 18:47 |
| fungi | yeah. looks like that's from https://review.opendev.org/827388 "Dockerfile: explicitly install uidmap package" | 18:51 |
| opendevreview | Clark Boylan proposed openstack/diskimage-builder master: Add Jammy functesting to dib https://review.opendev.org/c/openstack/diskimage-builder/+/840391 | 18:51 |
| fungi | looks like we got new ubuntu-jammy built 1.5 hours ago | 19:29 |
| fungi | seems to be uploaded everywhere now, so hopefully the package version mismatches are resolved at this point | 19:30 |
| *** Guest0 is now known as prometheanfire | 19:47 | |
| clarkb | https://zuul.opendev.org/t/openstack/build/6468ee78ff7e4c6b86d281a4b3e53f1f/log/logs/ubuntu-minimal_jammy-arm64-build-succeeds.FAIL.log#480 now it tries to build jammy but fails on the zstd problem extracting a package? How does this work in our nodepool iamges? | 20:01 |
| clarkb | oh we install zstd | 20:01 |
| opendevreview | Clark Boylan proposed openstack/diskimage-builder master: Add Jammy functesting to dib https://review.opendev.org/c/openstack/diskimage-builder/+/840391 | 20:03 |
| fungi | to make an apple pie from scratch, you must first invent the universe | 20:04 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Pull keycloak from quay.io https://review.opendev.org/c/opendev/system-config/+/840529 | 20:08 |
| clarkb | Testing shows the non wildfly modern container is not compatibile with the wildfly stuff :/ that ps updates to use the legacy tag which is wildfly whihc should hopefully work then we can think about transitioning to not wildlfy as a followup | 20:09 |
| clarkb | fungi: https://review.opendev.org/c/opendev/system-config/+/838948 is a simple review if you have time for it | 20:15 |
| fungi | clarkb: graphite isn't retired, was that file just unused? | 20:22 |
| Clark[m] | fungi: we should be using graphite02.opendev.org or similar now so the .openstack.org host vars are not valid | 20:27 |
| Clark[m] | double check that though | 20:27 |
| fungi | the one you're deleting is inventory/service/group_vars/graphite_opendev.org | 20:27 |
| fungi | but yeah there's also a inventory/service/group_vars/graphite.yaml | 20:28 |
| Clark[m] | oh right, that was an ansible specific group that ended up getting renamed to just graphite | 20:28 |
| Clark[m] | https://opendev.org/opendev/system-config/src/branch/master/inventory/service/groups.yaml#L63-L64 there is no graphite_opendev.org group anymore | 20:28 |
| fungi | they're identical, except that the latter adds an entry for zuul-lb | 20:28 |
| fungi | so i aree, this one is unused and prime for deletion | 20:29 |
| opendevreview | Clark Boylan proposed opendev/base-jobs master: Add ubuntu jammy x86 nodeset https://review.opendev.org/c/opendev/base-jobs/+/840355 | 21:04 |
| opendevreview | Clark Boylan proposed opendev/base-jobs master: Add Jammy Arm64 nodeset https://review.opendev.org/c/opendev/base-jobs/+/840538 | 21:04 |
| clarkb | I went ahead and split those two nodesets up into two changes because 840355 is ready to go now if I can get reviewers. But 840538 should wait on the arm64 image and node change sto land | 21:04 |
| clarkb | ianw: if you can review the arm64 image and node changes I'll happily monitor them tomorrow after approving them too | 21:05 |
| ianw | clarkb: oh yeah sorry, will do | 21:05 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Switch py3.10 testing to Ubuntu Jammy https://review.opendev.org/c/zuul/zuul-jobs/+/840539 | 21:11 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Make note of python_version being a string value https://review.opendev.org/c/zuul/zuul-jobs/+/840540 | 21:11 |
| clarkb | hrm did the zuul v6 work break our zuul_return in testing? | 21:14 |
| opendevreview | Clark Boylan proposed opendev/base-jobs master: Add ubuntu jammy x86 nodeset https://review.opendev.org/c/opendev/base-jobs/+/840355 | 21:17 |
| opendevreview | Clark Boylan proposed opendev/base-jobs master: Add Jammy Arm64 nodeset https://review.opendev.org/c/opendev/base-jobs/+/840538 | 21:17 |
| opendevreview | Clark Boylan proposed opendev/base-jobs master: Fix zuul_return in ansible linter https://review.opendev.org/c/opendev/base-jobs/+/840541 | 21:17 |
| clarkb | I think that may fix it | 21:17 |
| clarkb | ianw: also not sure if you saw but Jammy does phased package installs and by default we seem to bypass phasing in the dib image builds due to being in a chroot but then our images boot and get phased and things break. The solution appears to be to just disable phasing entirely. We've done so for infra images but I think we should probably move that into dib them remove the workaround | 21:28 |
| clarkb | in openstack/project/config/nodepool/elements | 21:28 |
| ianw | huh, i had not heard of this | 21:30 |
| fungi | yeah, it was new to me as well | 21:31 |
| ianw | i will refrain from commenting on my first impressions... | 21:31 |
| fungi | seems to be an ubuntu idea which they got implemented in apt upstream, but i don't expect debian will actually use the feature | 21:31 |
| clarkb | also to make things confusing disabling phasing is the same as enabling it 100% of the time rather than doing the phased rollouts | 21:32 |
| clarkb | probably more accurate to say "bypass phasing and always install latest available packages" | 21:32 |
| fungi | the phased updates feature would have been better termed "delay updates" | 21:32 |
| fungi | if you disable phased updates, you're disabling the delayed rollout basically, and everything updates as it becomes available | 21:33 |
| ianw | are we likely to see less stable updates because people are thinking "eh, if it breaks, i'll roll it back quickly"? | 21:35 |
| fungi | that seems to be the idea, yes | 21:35 |
| fungi | move fast and roll back what you break | 21:36 |
| ianw | APT::Get::Never-Include-Phased-Updates would mean that you only get the updates *after* phasing? | 21:36 |
| fungi | i think so? | 21:36 |
| clarkb | but I thnk we ewrne't sure if bulseye debootstrap etc would honor it | 21:36 |
| fungi | per further discussion, they're really sore about people pointing out the lack of documentation there | 21:37 |
| clarkb | for opendev's purposes installing latest available all the time consistently made sense for CI, but it is possible that dib needs to be more careful | 21:37 |
| ianw | sigh, i mean from a generic dib pov i guess it wants a flag so people can go either way with it? | 21:38 |
| clarkb | that also seems reasonable | 21:38 |
| ianw | maybe you want your images with "staged" updates? | 21:38 |
| clarkb | I think for opendev our users already expect some delta to lates tsince we build images daily | 21:39 |
| clarkb | which means if you really want latest you should upate anyway? But I'm not sure how many jobs do a system update first | 21:39 |
| clarkb | all that to say I think opendev would be fine if we did only oldest available packages too. The key is to not mix and match whic his what pbroke stuff | 21:39 |
| ianw | not many i would say, devstack probably does but generally i don't think things are running "apt-get update" before they start | 21:40 |
| fungi | they are because of the bindep role | 21:42 |
| ianw | good point | 21:43 |
| clarkb | but only for the packages in bindep right? That is probably close neough for most things | 21:44 |
| fungi | right, they're not doing an apt upgrade, which is maybe what ianw actually meant | 21:44 |
| fungi | (update pulls new indices, upgrade pulls new packages) | 21:45 |
| clarkb | unrelated to all this the gerrit 3.5 memory use problem has gotte nmore interesting. Uptsream thinks it may be related to a plugin in the downstream install? We may be ok after all | 21:46 |
| *** dviroel is now known as dviroel|out | 21:50 | |
| fungi | interesting indeed. so this was observed in gerrithub then? | 21:53 |
| fungi | which plugin? | 21:53 |
| clarkb | gerrithub noticed it and had problems in january. Someone else is debugging it more recently in their different gerrit. They havne't narrowed it down to a specific plugin yet, but the memory leak appears to happen due to ddnyamic key values in a metrics table and gerrit core reportedly only uses static keys | 21:54 |
| clarkb | the next debuggin gsteps there are to dump the keys to try and trace them back to the source | 21:54 |
| fungi | neat | 21:55 |
| fungi | so no smoking gun yet | 21:56 |
| clarkb | infra-root https://review.opendev.org/c/opendev/base-jobs/+/840541 and child will add the jammy nodeset so that people can use that shorthand in their jobs. The grandchild should wait on arm4 nodes being available. That should all work now that I Fixed the zuul_return thing with zuulv6 | 21:57 |
| ianw | i just have to do some morning things, will come back to consider dib updates for staging flags, openafs builds and monitoring arm64 builds | 22:00 |
| clarkb | thanks! | 22:05 |
| opendevreview | Merged openstack/project-config master: Add Jammy arm64 images https://review.opendev.org/c/openstack/project-config/+/840393 | 22:07 |
| opendevreview | Merged opendev/base-jobs master: Fix zuul_return in ansible linter https://review.opendev.org/c/opendev/base-jobs/+/840541 | 22:08 |
| clarkb | I'm going to restart my irc client. Hopefully I come back in one piece :) | 22:13 |
| clarkb | I seem to have made it back | 22:22 |
| clarkb | but now all my channels are in the wrong place I should reorgniaze | 22:23 |
| *** rlandy is now known as rlandy|bbl | 22:25 | |
| clarkb | fungi: any concern with approving https://review.opendev.org/c/opendev/base-jobs/+/840355 now to add an ubuntu-jammy nodeset? | 22:48 |
| fungi | i've approved it now, thought i'd already reviewed that | 22:49 |
| fungi | sorry! | 22:50 |
| clarkb | It hit the parent issue and then I also split it as arm64 instances weren't ready yet | 22:50 |
| opendevreview | Merged opendev/base-jobs master: Add ubuntu jammy x86 nodeset https://review.opendev.org/c/opendev/base-jobs/+/840355 | 23:08 |
| clarkb | I just edited openstack-zuul-jobs-core to remove frickler as an explicit member (frickler is part of infra-core which is also included) and jlk as jlk is no longer active | 23:23 |
| fungi | thanks | 23:24 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Switch py3.10 testing to Ubuntu Jammy https://review.opendev.org/c/zuul/zuul-jobs/+/840539 | 23:48 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Make note of python_version being a string value https://review.opendev.org/c/zuul/zuul-jobs/+/840540 | 23:48 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Switch py3.10 testing to Ubuntu Jammy https://review.opendev.org/c/zuul/zuul-jobs/+/840539 | 23:56 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Make note of python_version being a string value https://review.opendev.org/c/zuul/zuul-jobs/+/840540 | 23:56 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!