| *** rlandy is now known as rlandy|out | 00:19 | |
| ianw | sigh, a gate failure on that | 02:23 |
|---|---|---|
| ianw | error: Status code: 503 for https://mirrors.centos.org/metalin ... blah -- upstream failure | 02:24 |
| wanght26 | test | 03:21 |
| *** NeilHanlon_ is now known as NeilHanlon | 03:57 | |
| *** NeilHanlon is now known as neil | 03:59 | |
| *** neil is now known as NeilHanlon | 03:59 | |
| *** diablo_rojo is now known as Guest6861 | 04:32 | |
| *** ysandeep|out is now known as ysandeep | 05:18 | |
| opendevreview | Annu Mishra proposed openstack/diskimage-builder master: Add subscription-manager repo names for RHEL-9 https://review.opendev.org/c/openstack/diskimage-builder/+/851734 | 06:28 |
| opendevreview | Merged openstack/diskimage-builder master: Disable the opensuse functest https://review.opendev.org/c/openstack/diskimage-builder/+/851759 | 06:46 |
| *** jpena|off is now known as jpena | 07:37 | |
| *** ysandeep is now known as ysandeep|lunch | 09:05 | |
| opendevreview | Merged openstack/diskimage-builder master: rockylinux : create machine-id early https://review.opendev.org/c/openstack/diskimage-builder/+/851687 | 09:27 |
| *** ysandeep|lunch is now known as ysandeep | 10:21 | |
| *** tosky_ is now known as tosky | 10:24 | |
| *** rlandy|out is now known as rlandy | 10:30 | |
| *** dasm|off is now known as dasm | 11:06 | |
| opendevreview | Merged openstack/diskimage-builder master: Upgrade openEuler to 22.03 LTS https://review.opendev.org/c/openstack/diskimage-builder/+/848680 | 11:22 |
| *** dviroel|out is now known as dviroel | 11:25 | |
| fungi | python 3.10.6 is out! | 12:05 |
| priteau_ | THanks for merging the rocky fix in DIB. Do you know when a new RL8 image would be available on opendev? | 12:31 |
| *** priteau_ is now known as priteau | 12:38 | |
| fungi | priteau_: usually ianw pushes a new tag for dib, then we need to bump the minimum in the nodepool repo and merge that so new service container images are built, and then we upgrade nodepool to run from those new containers | 12:39 |
| priteau | OK | 12:40 |
| fungi | we can probably get through all that in the next 24 hours | 12:40 |
| priteau | That would be great :) | 12:41 |
| outbrito | Hey folks, g'day! Am I crazy or gerrit is showing all WIP changes as they also were all in merge conflict (without being actually) on the dashboard? | 13:08 |
| fungi | outbrito: we think that started with the upgrade to gerrit 3.5 a month or so ago. clarkb has supplied a fix upstream, i think? but in short, gerrit considers wip changes unmergeable, and so treats that the same as a merge conflict | 13:24 |
| fungi | the downside is that it masks any actual merge conflict with wip changes, so yes we'd like to see that solved | 13:25 |
| *** frenzyfriday|rover is now known as frenzyfriday|rover|lunch | 13:28 | |
| opendevreview | Gage Hugo proposed openstack/project-config master: End project gating for openstack-helm-addons https://review.opendev.org/c/openstack/project-config/+/851857 | 13:37 |
| opendevreview | Gage Hugo proposed openstack/project-config master: Retire openstack-helm-addons repository https://review.opendev.org/c/openstack/project-config/+/851858 | 13:37 |
| opendevreview | Gage Hugo proposed openstack/project-config master: Retire openstack-helm-addons repository https://review.opendev.org/c/openstack/project-config/+/851858 | 13:43 |
| *** frenzyfriday|rover|lunch is now known as frenzyfriday|rover | 14:05 | |
| Clark[m] | fungi: I haven't pushed a fix yet because the flag is binary for "is mergable" but is used to determine if there are merge conflicts and if the change is a WIP and I'm not sure what the best fix is. Probably to have multiple flags and completely redo the index and dashboard and that is beyond my Gerrit contribution ability. I did file a bug though | 14:07 |
| *** ysandeep is now known as ysandeep|dinner | 14:47 | |
| fungi | ahh, okay, i misremembered that. thanks for clarifying! | 14:55 |
| *** dviroel is now known as dviroel|lunch | 15:14 | |
| *** ysandeep|dinner is now known as ysandeep | 15:29 | |
| *** ysandeep is now known as ysandeep|out | 15:43 | |
| *** dviroel|lunch is now known as dviroel| | 16:15 | |
| *** dviroel| is now known as dviroel | 16:15 | |
| *** jpena is now known as jpena|off | 16:34 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server https://review.opendev.org/c/opendev/system-config/+/851248 | 17:18 |
| opendevreview | Clark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server https://review.opendev.org/c/opendev/system-config/+/851248 | 17:59 |
| fungi | ianw: once you're around and settled, are you good with tagging the rocky fix in dib and pushing a nodepool requirements minimum increase for that? if so, i'm on hand to expedite review and help with builder restarts | 18:04 |
| clarkb | note the builders will automatically restart when the hourly infra-prod-service-nodepool job runs (so it will happen relatively quickly on its own) | 18:05 |
| opendevreview | Neil Hanlon proposed openstack/diskimage-builder master: Add Rockylinux 9 build configuration and update jobs for 8 and 9 https://review.opendev.org/c/openstack/diskimage-builder/+/848901 | 18:07 |
| opendevreview | Clark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server https://review.opendev.org/c/opendev/system-config/+/851248 | 18:36 |
| outbrito | fungi, ack | 18:48 |
| fungi | clarkb: oh, good point, so we just need the tag and nodepool change merged | 18:52 |
| ianw | there was a request to have 848901 in the release | 19:06 |
| fungi | k | 19:06 |
| fungi | i'll check that out too | 19:07 |
| *** kopecmartin_ is now known as kopecmartin | 19:09 | |
| NeilHanlon | ty fungi | 19:09 |
| fungi | np, just trying to grease the wheels where i can | 19:10 |
| NeilHanlon | and ianw! | 19:10 |
| ianw | # ls /boot | 19:22 |
| ianw | initramfs-5.14.0-70.17.1.el9_0.x86_64.img | 19:22 |
| ianw | NeilHanlon: in testing, just in a container, I don't see kernel-core installing anything to /boot with 9 images either with or without an /etc/machine-id :/ | 19:22 |
| fungi | kernels are for suckers | 19:23 |
| NeilHanlon | hrm.. I ran dib manually and see this | 19:23 |
| NeilHanlon | https://paste.opendev.org/show/b8cpIzFWPB9B8LyaLpvc/ | 19:23 |
| ianw | yeah, https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d81/848901/3/check/dib-nodepool-functional-openstack-rockylinux-9-containerfile-src/d818ba9/nodepool/builds/test-image-0000000001.log | 19:24 |
| ianw | does show that there's a kernel in there. but how it got there is a bit of a mystery! | 19:24 |
| ianw | and that job passed, so the image booted | 19:24 |
| NeilHanlon | https://drop1.neilhanlon.me/irc/uploads/46ae407b3ae31e5c/image.png | 19:26 |
| NeilHanlon | except, as a gif.. | 19:26 |
| fungi | everything's better as a gif | 19:50 |
| fungi | unless you pronounce it wrong | 19:50 |
| fungi | it's "gif", not "gif" | 19:50 |
| NeilHanlon | Agreed | 19:53 |
| opendevreview | Merged openstack/project-config master: Revert "Temporarily turn on console logs for rocky in iweb" https://review.opendev.org/c/openstack/project-config/+/851520 | 19:55 |
| *** dviroel is now known as dviroel|biab | 19:56 | |
| ianw | well this seems a different path | 20:11 |
| ianw | /usr/lib/kernel/install.d/90-loaderentry.install fails out with | 20:11 |
| ianw | [ "$KERNEL_INSTALL_LAYOUT" = "bls" ] || exit 0 | 20:12 |
| ianw | + '[' other = bls ']' | 20:12 |
| ianw | "Defaults to bls if $BOOT/ENTRY-TOKEN exists, or other otherwise" | 20:12 |
| ianw | well, if i "mkdir `cat /etc/machine-id` and run kernel-install ... it installs a kernel | 20:15 |
| fungi | this is on rocky 9? | 20:16 |
| ianw | yeah in a rocky 9 container at leats | 20:17 |
| ianw | https://github.com/systemd/systemd/blob/main/src/boot/bootctl.c#L179 | 20:23 |
| ianw | now i'm more confused than ever | 20:23 |
| ianw | /etc/machine-id, /etc/machine-info, /etc/kernel/entry-token, and the layout field in /etc/kernel/install.conf | 20:23 |
| ianw | ... i mean seriously ... | 20:23 |
| NeilHanlon | systemd strikes again... | 20:27 |
| NeilHanlon | injecting `GRUB_ENABLE_BLSCFG=true` into the grubcfg __may__ help | 20:28 |
| NeilHanlon | idk. this is all a mess :/ | 20:29 |
| fungi | lennart is my copilot | 20:30 |
| *** dviroel|biab is now known as dviroel | 20:31 | |
| ianw | my main concern is that we're basically in the same situation, with something installing the kernel we don't quite understand, and just waiting for that to decide to stop installing the kernel | 20:33 |
| NeilHanlon | yeah, I agree with you there. I'm taking this back to the rocky releng team too to see if we can unwrap it some more | 20:33 |
| ianw | perhaps that is just the ground state of existence | 20:33 |
| ianw | ok, so if "os-prober" is installed *before* kernel-core, it seems to install the kernel | 20:37 |
| ianw | interestingly, *without* /etc/machine-id being set | 20:37 |
| ianw | i think actually it's /usr/lib/kernel/install.d/20-grub.install | 20:39 |
| NeilHanlon | i think it may be grub2-tools-minimal | 20:41 |
| NeilHanlon | which is a dep of osprober | 20:42 |
| NeilHanlon | yep, i think that's it.. | 20:42 |
| ianw | + cp -aT /lib/modules/5.14.0-70.17.1.el9_0.x86_64/vmlinuz /boot/vmlinuz-5.14.0-70.17.1.el9_0.x86_64 | 20:43 |
| ianw | the thing that is actually copping the kernel into /boot is 20-grub.install ... which is grub2-common-2.06-27.el9_0.7.noarch | 20:43 |
| NeilHanlon | seems these tools are required by the bits of that script which matter https://rpa.st/25HQ | 20:44 |
| ianw | if you rm -rf /boot/* then edit 20-grub.install to run with "bash -x" | 20:44 |
| ianw | then run | 20:44 |
| ianw | /bin/kernel-install --verbose add 5.14.0-70.17.1.el9_0.x86_64 /lib/modules/5.14.0-70.17.1.el9_0.x86_64/vmlinuz | 20:44 |
| NeilHanlon | eval "$(grub2-get-kernel-settings)" || true -- line 104 of 20-grub.install | 20:45 |
| ianw | yeah, that bit is actually inside | 20:46 |
| ianw | if [[ "x${GRUB_ENABLE_BLSCFG}" = "xtrue" ]] || [[ ! -f /sbin/new-kernel-pkg ]]; then | 20:46 |
| NeilHanlon | fwiw, that variable is set in the default grub config in rocky | 20:47 |
| ianw | ... except i don't think the container has /etc/default/grub at this point | 20:48 |
| ianw | what owns/generates that file? | 20:48 |
| NeilHanlon | grub2-tools | 20:49 |
| ianw | interesting, i have that installed but no /etc/default/grub | 20:49 |
| ianw | so basically it seems like we're *not* using BLS generating these images. that matches with what we see in the build logs | 20:50 |
| ianw | 11MiB /opt/dib/tmp/dib_build.OSgwAG5e/built/boot/vmlinuz-5.14.0-70.17.1.el9_0.x86_64 | 20:50 |
| ianw | the kernel isn't under a machine-id directory | 20:50 |
| ianw | i guess my question/problem is that we are not really doing this intentionally, it seems to work just by chance | 20:51 |
| *** dviroel is now known as dviroel|afk | 20:58 | |
| NeilHanlon | Okay.. so I built the container as in the containerfile for 9, and entered inside it. I installed grub2-tools-minimal, and then kernel-c | 20:59 |
| NeilHanlon | kernel-core* | 20:59 |
| NeilHanlon | I got a couple errors during the kernel-core scriptlets, but the resultant bootdir: https://paste.opendev.org/show/bI6RxokGxsoNIz11G9Db/ | 21:00 |
| ianw | that does look like bls layout | 21:01 |
| ianw | where did 0bfb6ce5470443588d374015a6eaa5ff come from? | 21:01 |
| ianw | i'm assuming not /etc/machine-id | 21:02 |
| NeilHanlon | correct, not from there as it's empty | 21:05 |
| NeilHanlon | you ask a good question, however I know not yet the answer :) | 21:05 |
| opendevreview | Merged openstack/diskimage-builder master: Add Rockylinux 9 build configuration and update jobs for 8 and 9 https://review.opendev.org/c/openstack/diskimage-builder/+/848901 | 21:08 |
| *** dasm is now known as dasm|off | 21:09 | |
| NeilHanlon | okay. so it's actually (I think for real this time) - systemd-udevd which is letting this work. It's a dependency of the grub2-minimal-tools script. It provides other files above and beyond the 20-grub.install you found.. including 20-grubby.install and 90-loaderentry.install | 21:14 |
| NeilHanlon | id comes from here ianw: bin/kernel-install:[ -z "$MACHINE_ID" ] && MACHINE_ID="$(systemd-id128 new)" | 21:30 |
| ianw | huh, ok | 21:35 |
| ianw | i don't want to waste too much of everyones time on this, but it's probably worth distilling this into the dib source somewhere so when it breaks we have some breadcrumbs | 21:38 |
| opendevreview | Merged openstack/project-config master: trigger-rtd: fix role syntax https://review.opendev.org/c/openstack/project-config/+/851274 | 21:38 |
| opendevreview | Merged openstack/project-config master: linter: update some YAML whitespace issues https://review.opendev.org/c/openstack/project-config/+/851275 | 21:39 |
| clarkb | BLS is just a special format for entries in /boot/ but it side effects how kernels are installed in rocky? (trying to keep up) | 21:40 |
| clarkb | I'm going to recheck my mm3 change after putting a hold in place. I don't understand the current test failure | 21:49 |
| clarkb | I've noticed I still have a hold of a gerrit server that I used to debug cache behaviors. I don't need that anymore so will delete the hold | 21:50 |
| ianw | clarkb: BLS is the format yep that puts things in /boot and i guess the theory is uses unique identifiers to stop things overwriting each other, which is/was/sometimes based on the machine-id, but is sometimes not | 21:55 |
| opendevreview | Clark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server https://review.opendev.org/c/opendev/system-config/+/851248 | 22:57 |
| clarkb | fungi: ^ I think that is getting really close to having services running | 22:57 |
| *** Guest6861 is now known as diablo_rojo | 23:05 | |
| fungi | oh, yay! | 23:05 |
| clarkb | ok yup got a +1 from zuul finally which means all the services are listening on the expected ports and their configs didn't explode | 23:21 |
| clarkb | That means the next steps will be an apache front end proxy and on the backend working to configure lists and domains | 23:21 |
| *** rlandy is now known as rlandy|out | 23:22 | |
| ianw | is there a mm2->mm3 migration tool? istr it was difficult to get the archives in there | 23:31 |
| clarkb | there is a system for doing it. YOu copy files around and run a tool iirc | 23:35 |
| clarkb | I think once we get this far enough along in the CI system one of the steps will be to test that migration with some of our exisitng mm2 lists | 23:35 |
| ianw | ok cool. definitely cart before horse situation optimizing that now :) | 23:44 |
| fungi | yeah, i think i linked/mentioned the migration tooling in the spec | 23:50 |
| fungi | a bunch of very large communities have migrated their lists at this point so it should be reasonably battle-tested | 23:52 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
