| ianw | my second thought is "that's the type of thing i could find out looking at the build history" :) | 00:53 |
|---|---|---|
| ianw | https://zuul.opendev.org/t/openstack/builds?job_name=dib-nodepool-functional-openstack-rockylinux-8-containerfile-src&project=openstack/diskimage-builder | 00:53 |
| ianw | which doens't look promising | 00:53 |
| ianw | it's varied between timed out and failed | 00:54 |
| ianw | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c20/850882/1/check/dib-nodepool-functional-openstack-rockylinux-8-containerfile-src/c20bf27/instances/120e2c99-7990-42d3-916b-49e5e364559a/console.log | 00:56 |
| ianw | it does end up a grub prompt | 00:56 |
| ianw | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c20/850882/1/check/dib-nodepool-functional-openstack-rockylinux-8-containerfile-src/c20bf27/nodepool/builds/test-image-0000000001.log | 00:58 |
| ianw | thankfully we have a working one https://zuul.opendev.org/t/openstack/build/67982e8612574c02b465e3c71058d21d/logs | 01:00 |
| fungi | oh! i forgot we had jobs exercising that | 01:13 |
| fungi | so it would be possible to compare them after all | 01:14 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: [wip] Revert "Fix BLS entries for /boot partitions" https://review.opendev.org/c/openstack/diskimage-builder/+/851683 | 01:15 |
| ianw | for clarity : | 01:17 |
| ianw | working -> https://b505db8eeb4a19717aca-6e6a98e42a02be15877850d4946fa43f.ssl.cf1.rackcdn.com/848680/2/check/dib-nodepool-functional-openstack-rockylinux-8-containerfile-src/67982e8/nodepool/builds/test-image-0000000001.log | 01:17 |
| ianw | not working -> https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c20/850882/1/check/dib-nodepool-functional-openstack-rockylinux-8-containerfile-src/c20bf27/nodepool/builds/test-image-0000000001.log | 01:17 |
| ianw | from that i suspect https://review.opendev.org/c/openstack/diskimage-builder/+/846838 | 01:17 |
| ianw | i don't know why, yet, but that seems to be the difference | 01:18 |
| ianw | i don't really feel like it could be that change, a priori | 01:58 |
| *** ysandeep|out is now known as ysandeep | 02:08 | |
| ianw | it's like /boot just disappeared. despite nothing seeming to change in terms of installed packages, etc. | 02:29 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: [wip] rockylinux : make /boot directory https://review.opendev.org/c/openstack/diskimage-builder/+/851687 | 02:56 |
| *** ysandeep is now known as ysandeep|afk | 03:40 | |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: [wip] rockylinux : make /boot directory https://review.opendev.org/c/openstack/diskimage-builder/+/851687 | 04:47 |
| *** ysandeep|afk is now known as ysandeep | 05:20 | |
| frickler_pto | oh, fun, flake8 2.9.0 added E275, insisting on whitespace around keywords. very annoying for all the folks that have been treating things like "assert" and "raise" as functions. the correct form now would be "assert x" or maybe "assert (x)" instead of "assert(x)" | 05:35 |
| *** frickler_pto is now known as frickler | 05:35 | |
| *** akahat_ is now known as akahat | 05:39 | |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: rockylinux : create machine-id early https://review.opendev.org/c/openstack/diskimage-builder/+/851687 | 05:58 |
| ianw | clarkb/NeilHanlon: ^ so I think that fixes it. I'm open to other suggestions | 05:59 |
| *** tobias-urdin_pto is now known as tobias-urdin | 06:06 | |
| *** ykarel_ is now known as ykarel | 06:41 | |
| *** ysandeep is now known as ysandeep|afk | 06:51 | |
| *** gibi_pto is now known as gibi | 07:05 | |
| *** ysandeep|afk is now known as ysandeep | 07:18 | |
| *** jpena|off is now known as jpena | 07:32 | |
| opendevreview | wangxiyuan proposed openstack/diskimage-builder master: Upgrade openEuler to 22.03 LTS https://review.opendev.org/c/openstack/diskimage-builder/+/848680 | 07:34 |
| *** ysandeep is now known as ysandeep|lunch | 07:50 | |
| *** soniya29 is now known as soniya29|afk | 08:13 | |
| *** ysandeep|lunch is now known as ysandeep | 09:40 | |
| *** lbragstad9 is now known as lbragstad | 10:30 | |
| *** rlandy_ is now known as rlandy | 10:33 | |
| opendevreview | Annu Mishra proposed openstack/diskimage-builder master: Adds required RHEL9 changes https://review.opendev.org/c/openstack/diskimage-builder/+/851734 | 11:14 |
| fungi | frickler: oh, i missed the new check because the codebase i tested the new version on was one where i went on a tear not long ago to fix those exact cases, del(x) was another i had mistakenly inserted a lot of | 11:20 |
| opendevreview | wangxiyuan proposed openstack/diskimage-builder master: Upgrade openEuler to 22.03 LTS https://review.opendev.org/c/openstack/diskimage-builder/+/848680 | 11:23 |
| *** pojadhav is now known as pojadhav|afk | 11:33 | |
| *** dviroel_ is now known as dviroel | 11:45 | |
| *** ysandeep is now known as ysandeep|afk | 12:23 | |
| *** elodilles is now known as elodilles_afk | 12:26 | |
| *** mnasiadka_ is now known as mnasiadka | 12:26 | |
| *** pojadhav|afk is now known as pojadhav | 13:15 | |
| NeilHanlon | ianw: that's a great catch.. i'm confused about the behavior and I'm gonna look into that. but if it works there, let's just call it good, IMO | 13:19 |
| NeilHanlon | seems the gate check failed for a transient reason | 13:22 |
| NeilHanlon | oh, i know what this is... iputils pulls in systemd during the container build 🤦 | 13:23 |
| frickler | NeilHanlon: the gate failure seems related to https://review.opendev.org/c/openstack/diskimage-builder/+/849445 | 13:27 |
| *** ysandeep|afk is now known as ysandeep | 13:33 | |
| *** Guest6564 is now known as dasm | 14:00 | |
| *** elodilles_afk is now known as elodilles | 14:02 | |
| opendevreview | Merged openstack/diskimage-builder master: opensuse: better report checksum errors https://review.opendev.org/c/openstack/diskimage-builder/+/849445 | 14:44 |
| fungi | since regular attendees to the diversity and inclusion working group meeting have had frequent problems with browsers blocking audio for jitsi-meet, i've manually switched on the "prejoin page" in advance of the meeting in order to confirm whether that does help matters | 14:46 |
| fungi | specifically, switching config.prejoinPageEnabled from false to true in /var/jitsi-meet/web/config.js on meetpad01 | 14:47 |
| Clark[m] | I'm not sure if you need to restart containers to see that change. But probably not | 14:49 |
| fungi | i did not need to, i already tested that it shows up in the browser | 14:50 |
| fungi | the config confusingly says that the prejoinConfig block replaces prejoinPageEnabled, but turning that on instead seemed to cause the interface not to load at all | 14:52 |
| fungi | it's possible i needed to keep it in the "var config = {" block instead of just prepending "config." to the name though | 14:53 |
| fungi | but for now prejoinPageEnabled does seem to still work | 14:53 |
| fungi | ahh, yeah that was it | 14:55 |
| fungi | switched over to that for now | 14:55 |
| *** frenzyfriday|rover is now known as frenzyfriday|rover|lunch | 15:01 | |
| fungi | Clark[m]: spotz says she was unable to get the jitsi mobile app to unmute after joining | 15:06 |
| fungi | something to test later | 15:06 |
| *** melwitt_ is now known as melwitt | 15:11 | |
| Guest6563 | yup I can help test that when you have a moment to be on the other isde of the call | 15:14 |
| *** Guest6563 is now known as clarkb | 15:16 | |
| clarkb | And now I am me again | 15:17 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update to Gitea 1.17 https://review.opendev.org/c/opendev/system-config/+/847204 | 15:32 |
| clarkb | Gitea 1.17.0 exists now. I'm not in rush to upgrade. Seems likethey often do a .1 and/or .2 bugfix release pretty quickly to address issues people find. But Since th erelease properly exists now I've updated the chagne and removed the WIP designation as it is technically mergeable now if reviewers are happy | 15:32 |
| *** rlandy is now known as rlandy|brb | 15:40 | |
| *** frenzyfriday|rover|lunch is now known as frenzyfriday|rover | 15:51 | |
| *** rlandy|brb is now known as rlandy | 15:52 | |
| *** tweining is now known as tweining|off | 15:52 | |
| clarkb | NeilHanlon: frickler fungi ianw I've manually downlaoded that opensuse image and checked the sha256sum. They are indeed different :/ | 15:59 |
| clarkb | this has happened before and they updated the sha256sum file after the fact. Considering this has happened again it wouldn't surprise me if their build system is producing buggy results | 15:59 |
| clarkb | dirk: ^ fyi https://download.opensuse.org/repositories/Cloud:/Images:/Leap_15.3/images/openSUSE-Leap-15.3.x86_64-NoCloud.qcow2 doesn't match its corresponding sha256sum file | 16:00 |
| NeilHanlon | what in god's green earth is that url pattern lol | 16:03 |
| clarkb | I forget what the : denotes but if you drop the : you get different content along those path entries | 16:08 |
| NeilHanlon | huh, interesting. I've never really looked at opensuse much | 16:08 |
| *** dviroel is now known as dviroel|lunch | 16:22 | |
| *** marios is now known as marios|out | 16:24 | |
| *** ysandeep is now known as ysandeep|out | 16:26 | |
| clarkb | what i don't understand is how https://review.opendev.org/c/openstack/diskimage-builder/+/849445 passed but the rocky fix didn't | 16:30 |
| clarkb | are we getting different versions of files, maybe depending on locality or load balancer luck? | 16:30 |
| clarkb | ah yup these may ultimately get served by mirrorsand maybe they aren't in sync for the different files? | 16:33 |
| opendevreview | Clark Boylan proposed openstack/diskimage-builder master: Disable the opensuse functest https://review.opendev.org/c/openstack/diskimage-builder/+/851759 | 16:36 |
| clarkb | thats sort of the nuclear option | 16:36 |
| NeilHanlon | ah, yeah that could be it, especially if it's new. we have a similar problem with rocky due to the staggered way in which mirrors can grab content | 16:39 |
| fungi | usually when dealing with checksummed or signed things is to add the new artifact without removing the old one, then replace the signatures/checksums to point to the new artifact, then clean up the old artifact (usually after a bit of a delay) | 16:41 |
| NeilHanlon | agreed | 16:43 |
| fungi | that way if a mirror grabs the files at any random time, it should get a coherent artifact+sig combo | 16:43 |
| opendevreview | Clark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server https://review.opendev.org/c/opendev/system-config/+/851248 | 16:44 |
| *** jpena is now known as jpena|off | 17:14 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server https://review.opendev.org/c/opendev/system-config/+/851248 | 17:20 |
| *** dviroel|lunch is now known as dviroel | 17:24 | |
| clarkb | fungi: ^ current issue is both mailman core and mailman web say mysql isn't available but I've got a test case indicating it is listening on the specified port. Any idea what might be going on there? | 18:00 |
| clarkb | I did change the mariadb version and use host networking compared to the original docker compose file. I wonder if it is related to one or both of those changes | 18:01 |
| clarkb | (I'm specifying connection to localhost:3306 in the mysql urls so the host nteworking update shouldn't matter | 18:02 |
| clarkb | https://github.com/maxking/docker-mailman/blob/main/web/docker-entrypoint.sh#L17-L25 appears to be the code that is looping | 18:03 |
| clarkb | based on that I don't think the version mismatch is a likely issue since mysqladmin ping is likely to have compat across versions? | 18:04 |
| fungi | ssl vs plain? | 18:09 |
| fungi | authentication? | 18:09 |
| clarkb | looking at the little check there it seems to not do auth its just checking if the port is listening? which the testinfra test says it is | 18:10 |
| clarkb | but it is doing it fro mthe context of the docker container itself not the host so maybe something with how host networking is setup? | 18:10 |
| clarkb | I might have to hold a node to debug that. | 18:10 |
| fungi | yeah, i would confirm the containers can really reach each other before assuming there's something deeper wrong | 18:11 |
| clarkb | cross checking our other docker compose files we don't have to specify ports when using network_mode:host (which makes sense as we aren't NATing at all) | 18:13 |
| clarkb | hold is created and recheck requested | 18:21 |
| clarkb | tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN | 19:08 |
| clarkb | bah I think it may be that localhost == local socket and that isn't present in the container | 19:10 |
| * clarkb tries 127.0.0.1 | 19:10 | |
| clarkb | that doesn't work because urlparse with 127.0.0.1 returns None for the hostname | 19:16 |
| clarkb | huh it works on my local python3 | 19:18 |
| clarkb | hrm no something else must be broken | 19:20 |
| clarkb | ok I htink https://github.com/maxking/docker-mailman/blob/66dc40f9ae73960be2febd8f668983d91b4bca47/postorius/mailman-web/settings.py#L55 is our first case of "the upstream images are problematic" | 19:30 |
| clarkb | I'll push up a change that should illustrate how this fails | 19:30 |
| clarkb | the previous issue had to do with quoting :/ | 19:30 |
| opendevreview | Clark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server https://review.opendev.org/c/opendev/system-config/+/851248 | 19:31 |
| clarkb | I'm overriding the allowed hosts list but it is an append not a replacement | 19:32 |
| clarkb | and they lookup a thing that isn't in dns because we arne't using docker networking. We can either add records to /etc/hosts for those names to point at localhost or not use host networking or override that file maybe or do our own images | 19:33 |
| opendevreview | Clark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server https://review.opendev.org/c/opendev/system-config/+/851248 | 20:32 |
| clarkb | I don't expect ^ to help but if it does work the nmaybe that is an easy workaround | 20:33 |
| *** dviroel is now known as dviroel|out | 21:26 | |
| *** dasm is now known as dasm|off | 21:41 | |
| clarkb | yup the hostname appears to be ignored | 21:53 |
| ianw | clarkb/NeilHanlon: yeah, i played with this a while ago and got different things from the mirrors at different points, even signed up to the bug tracker but then it went away | 22:04 |
| ianw | apropos opensuse images | 22:04 |
| ianw | https://meetings.opendev.org/irclogs/%23openstack-dib/%23openstack-dib.2022-07-12.log.html#t2022-07-12T01:11:20 to be exact | 22:06 |
| clarkb | ya Itried a recheck but it seems to have failed. I also pushed a change to rmeove that test entirely | 22:08 |
| clarkb | another approach may be to pull the opensuse image from a specific mirror. Do you think that would be a whorthwile change? | 22:08 |
| clarkb | I can write that if so | 22:08 |
| ianw | if we have some data maybe we should disable but file a bug report upstream. at least we can report it, even if nobody fixes it | 22:10 |
| fungi | thing is, even if they ignore the bug report it'll likely fix itself at their next mirror push or something | 22:12 |
| clarkb | the data I have is earlier today I downlodaed the file and got sha256sum b77c0c48f970c00ab9c99a574114f41d54c12a51c4d29e31e2945bfdf28dc9c8 for the actual file but the sha256sum file on the webserver says 864667deb442174c05d5fb8cd2dd9a8165ded0fb13f31ea7e96139ab911a7531 | 22:12 |
| ianw | yeah so we have it out of sync today and on 2022-07-11 @ https://paste.opendev.org/show/b1r9oMWXQNHzqmsTGcx9/ | 22:13 |
| ianw | did you end up at provo-mirror.opensuse.org? | 22:14 |
| fungi | another option might be to add some strategic checksum validation to out rsync script and skip the vos release when it's no good | 22:15 |
| fungi | s/out/our/ | 22:15 |
| ianw | for this test, i think we're pulling form upstream directly, i don't think we mirror these bits | 22:15 |
| fungi | ahh, okay | 22:15 |
| clarkb | correct this is straight from upstream | 22:18 |
| clarkb | I think it might be more stable if we just pick a specific mirror | 22:18 |
| clarkb | unless that mirror is out of sync for similar reasons or goes down entirely | 22:18 |
| ianw | in that paste above i was pulling from the provo- mirror from a us-based vm. but locally i got sent to some very unreliable .au mirror that cut out most of the time | 22:20 |
| ianw | will file something in a bit and we can disable for now | 22:24 |
| clarkb | sound sgood | 22:26 |
| clarkb | I've updated the meeting agenda for tomorrow. Anything else to add before I send it out? | 22:41 |
| * fungi checks his open reviews | 22:42 | |
| fungi | some reviews have come up for git-review | 22:43 |
| clarkb | Feel free to add an agenda item if git review needs meeting coverage. | 22:44 |
| fungi | i've got a pair of changes about making its test rebase process clearer to users, and there's another change to add support for a newer gerrit feature we can't test without increasing the gerrit version we test against | 22:44 |
| fungi | not sure if those rise to the level of an agenda topic | 22:44 |
| clarkb | the reabse thing is a fun one since people seem extremely confused about it. I really thing git review -x leads to that | 22:45 |
| clarkb | hopefully your chagnes can make that more clear | 22:45 |
| fungi | leave some comments on 850054 or its child if they don't | 22:45 |
| fungi | increasing the gerrit version we test with is always complicated though, since the git-review testsuite does a bespoke gerrit deployment that invariably needs care and feeding with each new version | 22:46 |
| fungi | i suspect the proper solution is to remake the test setup for git-review | 22:47 |
| clarkb | once it gets to 3.2 and newer it should be pretty consistent fro mthen on | 22:47 |
| clarkb | fungi: the second change related to rebases is workinprogress. were you looking for feedback ro still editing it? | 22:55 |
| fungi | yeah, that needs some consensus building, i guess. i think that's why i did wip initially | 22:57 |
| fungi | because it's a behavior change | 22:57 |
| clarkb | gotcha I can leave a note about me being ok with the behavior change. I almost always git rebase --abort when that happens | 22:58 |
| clarkb | the one thing that maybe would make it a good idea to keep as is is if people are less comfortable knowing how to start a rebase but once started can finish it | 22:58 |
| ianw | clarkb: filed https://bugzilla.suse.com/show_bug.cgi?id=1202041 for the images | 23:48 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Disable the opensuse functest https://review.opendev.org/c/openstack/diskimage-builder/+/851759 | 23:51 |
| ianw | ^ just a respin with the bug link | 23:51 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!