| ianw | fungi: ahh, ok, sorry yeah it made a bit more sense when i kept reading on :) | 00:00 |
|---|---|---|
| ianw | yeah that next change error is weird | 00:00 |
| ianw | No section: 'testenv:linters' | 00:01 |
| fungi | right, i tink it's getting a tox_extra_args which is being set in the previous playbook. maybe it's possible to clear it or scope it to that playbook? | 00:02 |
| ianw | it would probably help to debug dump the entire arguments passed, i don't think we can get that any other way | 00:05 |
| fungi | oh, good point | 00:08 |
| mordred | mnaser: yikes. Travelling is hard! | 00:13 |
| mordred | mnaser: Tldr buster to bullseye upgrade ... And also an update to fix arm | 00:14 |
| mnaser | mordred: I thought wearing a mask for a whole day is hard. Wearing one in a flight for 6 hours gives me a whole another level of respect for folks who have to do that (and grateful for us who don’t need to) | 00:14 |
| mordred | There's buster and bullseye tags now to facilitate upgrade | 00:14 |
| mordred | mnaser: ++ yah | 00:14 |
| mnaser | mordred: oh cool, I haven’t dabbled much in building images, I will dig into it a little bit later and see how that works | 00:14 |
| mordred | I've done one flight. It's bonkers | 00:14 |
| mnaser | Hope you’re staying safe too btw :) | 00:14 |
| mnaser | Yeah, 6 hours here (I’m in sfo) | 00:15 |
| mordred | Heck yes ... Evacuated to Panama City Beach Florida :) | 00:15 |
| * mordred returns to walking on the beach with a drink | 00:16 | |
| corvus | fungi, ianw: fungi asked for help in #zuul and i responded.... did you see that response? | 00:27 |
| fungi | corvus: not yet, i'm at a different computer at the moment but i'll check the one i've got matrix on in a few, thanks! | 00:34 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: [dnm] testing ubuntu builds https://review.opendev.org/c/openstack/diskimage-builder/+/806660 | 01:56 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: [dnm] testing ubuntu builds https://review.opendev.org/c/openstack/diskimage-builder/+/806660 | 03:16 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: [dnm] testing ubuntu builds https://review.opendev.org/c/openstack/diskimage-builder/+/806660 | 06:04 |
| opendevreview | Merged zuul/zuul-jobs master: ensure-podman: enable testing of centos-8-stream https://review.opendev.org/c/zuul/zuul-jobs/+/805433 | 06:05 |
| opendevreview | Merged zuul/zuul-jobs master: ensure-docker: enable centos-8-stream testing https://review.opendev.org/c/zuul/zuul-jobs/+/805432 | 06:05 |
| *** ysandeep|away is now known as ysandeep | 06:12 | |
| *** jpena|off is now known as jpena | 07:51 | |
| *** ysandeep is now known as ysandeep|lunch | 08:10 | |
| *** ysandeep|lunch is now known as ysandeep | 09:03 | |
| *** bhagyashris_ is now known as bhagyashris | 10:07 | |
| *** ysandeep is now known as ysandeep|afk | 11:01 | |
| *** jpena is now known as jpena|lunch | 11:38 | |
| *** ysandeep|afk is now known as ysandeep | 11:58 | |
| opendevreview | Merged openstack/project-config master: Add os-vif-core to stable maintainers https://review.opendev.org/c/openstack/project-config/+/798071 | 12:10 |
| mazzy | fungi, mordred: so guys i was able to make an element with flatcar but not yet in fully. with root.d i've designed a script to fetch flatcar sdk. the thing is that from what I've understood dib create an image from the content of chroot. however in my case the image that has to be created is just in a folder inside the chroot but it is not the | 12:40 |
| mazzy | entire chroot. take a look at here https://kinvolk.io/docs/flatcar-container-linux/latest/reference/developer-guides/sdk-modifying-flatcar/#create-the-flatcar-container-linux-os-image | 12:40 |
| *** jpena|lunch is now known as jpena | 12:41 | |
| opendevreview | Arx Cruz proposed opendev/elastic-recheck rdo: Updates for docker and frontpage of elastic-recheck https://review.opendev.org/c/opendev/elastic-recheck/+/806739 | 12:57 |
| *** ysandeep is now known as ysandeep|afk | 14:31 | |
| clarkb | fungi: do you know if my test kata list server got delete? if that hasn't been done I can do that this morning | 14:47 |
| fungi | i don't think it's been deleted yet, i didn't want to clean it up until i could confirm with you | 14:47 |
| clarkb | fungi: I think the upgrade went well from what I have seen and have no other reason to keep the test server around unless you can think of something to check for lists.o.o there | 14:48 |
| fungi | yeah, all clear in my opinion | 14:49 |
| clarkb | ok I've put that on my todo list for this morning after meetings and getting the opendev team meeting agenda out | 14:49 |
| clarkb | infra-root it looks like the zuul changes to update how zk secrets are managed have landed. https://review.opendev.org/c/opendev/system-config/+/803992 is our change to accomodate that in opendev. I suspect zuul hasn't been restarted with those changes yet which means we probably don't want to land that chagne yet but reviews would be good | 15:42 |
| corvus | clarkb: i think we're safe to land it now; presumably the new image has deployed and the commands are available. nothing in ZK changes, we just stop reading stuff in from disk (and that will take effect when we do restart). | 15:44 |
| *** jpena is now known as jpena|off | 16:02 | |
| *** ysandeep|afk is now known as ysandeep | 16:05 | |
| clarkb | corvus: I think because I'm using docker exec things will fail until we update the running images, but that is probably ok as well | 16:13 |
| clarkb | agreed that we should be able to land that now and it will start functioning when the zuul installation is updated | 16:14 |
| corvus | clarkb: oh right sorry. agreed. | 16:15 |
| opendevreview | Jeremy Stanley proposed zuul/zuul-jobs master: Add tox_config_file rolevar to tox https://review.opendev.org/c/zuul/zuul-jobs/+/806613 | 16:15 |
| opendevreview | Jeremy Stanley proposed zuul/zuul-jobs master: Support verbose showconfig in tox siblings https://review.opendev.org/c/zuul/zuul-jobs/+/806621 | 16:15 |
| opendevreview | Jeremy Stanley proposed zuul/zuul-jobs master: Include tox_extra_args in tox siblings tasks https://review.opendev.org/c/zuul/zuul-jobs/+/806612 | 16:15 |
| clarkb | fungi: if you get a chance can you confirm clarkb-test-lists.kc.io b36e34f3-8c8c-4808-8320-c58ddd4ccd80 is the instance to delete? | 16:19 |
| fungi | b36e34f3-8c8c-4808-8320-c58ddd4ccd80 is what i see for clarkb-test-lists.kc.io | 16:21 |
| clarkb | alright deleting it now | 16:21 |
| fungi | some autoheld nodes for lists upgrade testing | 16:22 |
| fungi | you've also got, i mean | 16:22 |
| fungi | a 3-node group for 794036 | 16:22 |
| clarkb | fungi: if you don't think there is anymore testing we can/should do with those then we can delete them as well (the autohold removal will do that for us) | 16:23 |
| clarkb | fungi: I can do that if you confirm there isn't any other testing/checking you want to do with them | 16:23 |
| fungi | i don't know of anything else we were going to try there, but i was mainly just making sure you knew they were still held | 16:23 |
| clarkb | yup I'm aware. I think I have a gitea held as well taht can be cleaned up as waiting for 1.15.1 seems like a good idea (the mysql fix did appear to land) | 16:24 |
| clarkb | fungi: I'll remove those two autoholds now | 16:25 |
| clarkb | mnaser: I'll also remove your autohold for loci keystone as it never hit anything | 16:26 |
| clarkb | I assume that means it got sorted out separtely. If not we can add a new hold | 16:26 |
| mnaser | clarkb: yes, nothing merged since then but it works now.. who knows | 16:51 |
| mnaser | i will Ask for another one if I am troubleshooting this later | 16:51 |
| clarkb | mnaser: sounds good | 16:51 |
| *** dviroel|out is now known as dviroel|ruck | 16:54 | |
| opendevreview | Jeremy Stanley proposed zuul/zuul-jobs master: Support verbose showconfig in tox siblings https://review.opendev.org/c/zuul/zuul-jobs/+/806621 | 17:15 |
| opendevreview | Jeremy Stanley proposed zuul/zuul-jobs master: Include tox_extra_args in tox siblings tasks https://review.opendev.org/c/zuul/zuul-jobs/+/806612 | 17:15 |
| *** ysandeep is now known as ysandeep|away | 17:54 | |
| clarkb | infra-root service-nodepool failed in the hourly deploy jobs because nb03 is unreachable. I suspect we need to force restart it or similar, but before I dig in would it be better to fix up the nodepool image stuff or are we happy with that situation now? | 18:23 |
| clarkb | connection reset by peer is the error I get trying to ssh | 18:24 |
| fungi | nb03 is the arm64 builder? | 18:25 |
| clarkb | INFO: task jbd2/dm-3-8:480 blocked for more than 120 seconds. <- seems like it has sad disks? | 18:25 |
| clarkb | fungi: yes | 18:25 |
| clarkb | the console log also shows dracut is sad | 18:25 |
| fungi | yeah, that does look like it may have lost contact with the storage somehow | 18:25 |
| fungi | if it's iscsi then could be a network blip | 18:25 |
| fungi | hard to know since we don't have insight into how the storage is attached | 18:26 |
| clarkb | I'm not sure what backs it | 18:26 |
| clarkb | right | 18:26 |
| fungi | though it could also be cpu, the jbd2 kernel threads wake up a lot and would be more likely to show up as blocked when there's cpu starvation on the host | 18:27 |
| fungi | if it's really lost contact with storage, you'll usually also see scsi timeouts or similar logged for the corresponding block devices | 18:28 |
| fungi | and messages about switching the filesystems on them to read-only | 18:28 |
| opendevreview | Mathieu Gagné proposed openstack/project-config master: INAP mtl01 region is now owned by iWeb https://review.opendev.org/c/openstack/project-config/+/806788 | 18:30 |
| clarkb | Should I go ahead and ask openstack to reboot it? or wait on nodepool images or perhaps bring it up with kevinz ? | 18:30 |
| fungi | presmably kevinz has logs on the host side anyway. if you reboot it, we can check syslog to see roughly when it seems to have gotten stuck | 18:31 |
| fungi | what nodepool images are you talking about waiting on? | 18:32 |
| clarkb | mgagne: left a note on ^ about how I think we need to add the new name in, then wind down the old name. | 18:34 |
| clarkb | otherwise we might have a mess to cleanup with orphaned resources in nodepool. Let us know if you'd like one of us to push newer patchests or if you'd prefer to do it | 18:34 |
| mgagne | I agree with your comments, I'll look into it. | 18:35 |
| clarkb | thanks! | 18:35 |
| fungi | yeah, ignore my review comment, it's basically the same thing clarkb said | 18:35 |
| clarkb | fungi: the docker images which have given us trouble recently | 18:35 |
| fungi | clarkb: oh, wait on new nodepool software containers to be published to dockerhub so we can deploy and boot onto those? | 18:36 |
| fungi | s/boot/docker-compose up/ | 18:36 |
| fungi | we'd need to reboot the server anyway before we can deploy a new container image onto it, right? | 18:37 |
| clarkb | fungi: ya as I'm not sure if this may have been caused by sad docker images. I don't know if we've updated them since we half reverted | 18:37 |
| fungi | ahh | 18:37 |
| clarkb | fungi: we do but that will start the docker container back up again aiui | 18:37 |
| fungi | right but that's going to happen either way, right? unless we delete the server entirely and build a new one, or rescue boot it so we can disable services on the filesystem | 18:38 |
| clarkb | ya thats true I guess. We'd have to rescue boot it and stop the container from starting on boot | 18:38 |
| clarkb | if that was the issue | 18:38 |
| clarkb | I'll reboot it now | 18:38 |
| fungi | sounds as good as anything | 18:41 |
| fungi | i'll hop on and look at logs once it's up | 18:41 |
| clarkb | its up. Looks like the nodepool-builder image is older and should be fine | 18:42 |
| clarkb | I guess we probably didn't succeed in testing with the sad images? I didn't know if that was the case but seems to be | 18:42 |
| fungi | there's a gap in the syslog from Aug 29 17:33:20 to Aug 31 03:42:24 | 18:43 |
| mgagne | clarkb: should I take care of the system-config update or will one of you do it? | 18:43 |
| opendevreview | Mathieu Gagné proposed openstack/project-config master: INAP mtl01 region is now owned by iWeb https://review.opendev.org/c/openstack/project-config/+/806791 | 18:44 |
| opendevreview | Mathieu Gagné proposed openstack/project-config master: INAP mtl01 region is now owned by iWeb https://review.opendev.org/c/openstack/project-config/+/806788 | 18:44 |
| clarkb | mgagne: I can do it if you prefer | 18:45 |
| fungi | most reccent log entry prior to the reboot messages was at Aug 31 15:31:08 | 18:45 |
| mgagne | I can do it, I was wondering if it requires informations/details I didn't have access to. | 18:45 |
| clarkb | mgagne: it shouldn't all of the secret bits are templated there. If we want new variables you can use the new names in system-config and one of the infra-root will update the secrets properly before landing the change | 18:46 |
| mgagne | guess that could also be done in 2 steps, introduce the new one with existing names and later remove the old one and rename? | 18:47 |
| clarkb | yup | 18:47 |
| fungi | right, it's mostly about being able to rely on the nodepool launchers and builders to wind down and clean up any remaining server instances and images | 18:48 |
| fungi | because we can't easily reassociate them to a different provider name | 18:48 |
| mazzy | does anyone has any clue about this issue https://pastebin.com/quAtmkfp? | 18:54 |
| fungi | clarkb: how should we handle the mirror server? will just adding records in dns be enough to get servers booting in the "new" provider talking to it? i expect not, we'll need to add the new name to the acme config for a new https cert too at a minimum, and i'll need to check the apache config to see if the vhost cares what name you request | 18:54 |
| clarkb | fungi: oh good point, yes I think we need to update DNS and then have acme rissue the cert | 18:56 |
| fungi | mazzy: that looks like diskimage_builder hasn't been installed into python's import path | 18:56 |
| clarkb | fungi: I don't think that apache cares about the name as I can hit https://mirror01.dfw.rax.opendev.org without trouble | 18:56 |
| fungi | clarkb: yeah, so dns and acme | 18:57 |
| fungi | which will let us transition without having to replace the mirror server immediately | 18:57 |
| clarkb | yup | 18:58 |
| mazzy | fungi: but that is done installing dib via virtualenv | 18:58 |
| mazzy | and i have done that. | 18:58 |
| clarkb | are you executing diskimage-builder from the same virtualenv? It seems to be looking at a source path and not an install path | 18:59 |
| fungi | mazzy: i agree installing it in a venv/virtualenv should be enough as long as the env is activated or you're invoking an entrypoint within it rather than, say, whatever python is on your default system path | 18:59 |
| mazzy | fungi: the env is activated | 19:02 |
| mazzy | (env) ubuntu@ubuntu:~/dib/diskimage-builder$ which python3 | 19:02 |
| mazzy | /home/ubuntu/dib/env/bin/python3 | 19:02 |
| mazzy | (env) ubuntu@ubuntu:~/dib/diskimage-builder$ which disk-image-create | 19:02 |
| mazzy | /home/ubuntu/dib/env/bin/disk-image-create | 19:02 |
| fungi | it shouldn't be failing to import from diskimage_builder in that case | 19:03 |
| opendevreview | Mathieu Gagné proposed opendev/system-config master: Introduce iweb cloud configs https://review.opendev.org/c/opendev/system-config/+/806795 | 19:04 |
| fungi | mazzy: i see a few lines before the exception it also complains "/usr/bin/env: 'python3': No such file or directory" | 19:04 |
| fungi | that may be a clue | 19:04 |
| mazzy | yeah this is what i really do not understand - why it's failing on that | 19:05 |
| opendevreview | Mathieu Gagné proposed openstack/project-config master: INAP mtl01 region is now owned by iWeb https://review.opendev.org/c/openstack/project-config/+/806788 | 19:23 |
| fungi | mazzy: so anyway, it looks like pre-install.d/10-package-installs gets started, tries to run some script which has a shebang like "#!/usr/bin/env python3" (could even be the 10-package-installs executable itself?), and finds no python3 it its executable path. the traceback later for the ModuleNotFoundError looks like it was raised during error handling/cleanup so may not be directly related to | 19:34 |
| fungi | the underlying issue | 19:34 |
| ianw | fungi: yeah sorry we went on a bit of a parallel track in #openstack-dib. python3 is required *in* the chroot, it's basically an assumption it is there | 19:41 |
| fungi | neat | 19:41 |
| fungi | good to know | 19:41 |
| mazzy | fungi, ianw: alos another thing is there a way to disable package-installs to run? i would add a distro that does not has a package manager so there is no sense to run package-installs | 19:49 |
| ianw | mazzy: there is not currently. i guess the package-installs element could have a flag that you define in an environment.d file to not run. | 19:50 |
| ianw | mazzy: perhaps let's discuss in #openstack-dib | 19:52 |
| mazzy | yeah it makes sense. | 19:52 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Stop managing OpenStackID servers https://review.opendev.org/c/opendev/system-config/+/806798 | 19:54 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Add assets and a related docker image/bundle https://review.opendev.org/c/opendev/system-config/+/805932 | 20:17 |
| opendevreview | Ian Wienand proposed opendev/system-config master: gitea: use assets bundle https://review.opendev.org/c/opendev/system-config/+/805933 | 20:17 |
| opendevreview | Clark Boylan proposed openstack/project-config master: INAP mtl01 region is now owned by iWeb https://review.opendev.org/c/openstack/project-config/+/806788 | 20:30 |
| clarkb | mgagne: ^ that just sets a depends on for the system-config change. | 20:30 |
| fungi | clarkb: should i go ahead and push the dns record additions too? | 20:34 |
| fungi | or are you already writing it? | 20:34 |
| clarkb | fungi: I hadn't started if you want to do it. but I can as well | 20:35 |
| fungi | no worries, on it now | 20:35 |
| clarkb | I'm just getting through the set of changes I said I would review this afternoon :) | 20:35 |
| clarkb | fungi: ianw https://review.opendev.org/c/opendev/system-config/+/803992 would be good to review. Note that docker exec will fail until we update our zuul installation but corvus and I are both ok with that | 20:35 |
| clarkb | fungi: I suppose we also need a system-config update to add the name to the cert name list | 20:37 |
| clarkb | fungi: we can put that on top of https://review.opendev.org/c/opendev/system-config/+/806795 or squash it in. | 20:37 |
| fungi | yeah, i thought you were including it when you said system-config change | 20:38 |
| fungi | but could be separate too | 20:38 |
| mgagne | clarkb: thanks | 20:38 |
| clarkb | fungi: oh no the system-config change already existed and is safe to alnd as is, but the project-config one isn't until we land the existing system-config change and update the cert and dns | 20:40 |
| clarkb | I'll update system-config | 20:40 |
| opendevreview | Jeremy Stanley proposed opendev/zone-opendev.org master: Add iweb mtl01 mirror records https://review.opendev.org/c/opendev/zone-opendev.org/+/806805 | 20:41 |
| fungi | also i guess when we clean up the old dns records we'll also need to update the hostname in the inventory? | 20:42 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Introduce iweb cloud configs https://review.opendev.org/c/opendev/system-config/+/806795 | 20:43 |
| clarkb | fungi: we shouldn't update the old dns records as long as the hostname is the old hostname imo | 20:43 |
| fungi | yeah, i didn't touch the old records just added a complete new copy for the new subdomain and bumped the serial | 20:43 |
| clarkb | we can drop the CNAME for the old name but the fqdn for server isn't changing | 20:43 |
| clarkb | I think https://review.opendev.org/c/opendev/system-config/+/806795 should be mergable as soon as that dns update lands. I'll review dns now | 20:44 |
| fungi | got it, so we wouldn't alter the hostname in the inventory unless we actually rebuilt/replaced the server instance | 20:44 |
| clarkb | fungi: yes, that way everything lines up with nova | 20:44 |
| fungi | wfm | 20:44 |
| ianw | clarkb: lgtm, that file should match backups | 20:45 |
| clarkb | fungi: I've approved the dns update | 20:46 |
| clarkb | fungi: and i think my system-config update matches the names you added acme challenge records for | 20:46 |
| fungi | yep, just checked and it matches | 20:50 |
| clarkb | Looks like we may need to publish the assets image afterall? https://zuul.opendev.org/t/openstack/build/3e448801258f40f29a688162ef899309 | 20:51 |
| clarkb | I'm ok with that and have +2'd the assets image change | 20:51 |
| clarkb | hrm maybe not reviewing the change I think possibly there is a small thing to change | 20:52 |
| clarkb | ianw: ^ left a note on that change with what I think should make it happier? | 20:53 |
| opendevreview | Ian Wienand proposed opendev/system-config master: gitea: use assets bundle https://review.opendev.org/c/opendev/system-config/+/805933 | 20:54 |
| ianw | clarkb: sorry, yeah just looking :) i left the "requires" off the image build job | 20:54 |
| clarkb | ianw: and you don't need the new requires on the system-config-run-gitea job | 20:54 |
| ianw | ah, yeah | 20:55 |
| clarkb | ianw: one sec one more thing | 20:55 |
| clarkb | comment posted | 20:55 |
| opendevreview | Ian Wienand proposed opendev/system-config master: gitea: use assets bundle https://review.opendev.org/c/opendev/system-config/+/805933 | 20:55 |
| clarkb | I was too slow :) one more small thing on ps4 to address | 20:56 |
| ianw | clarkb: hrm, i feel like we've had this discussion before. if it should pull that image from the registry or use released images | 20:57 |
| clarkb | ianw: right I think if you only use released images then gating won't use the speculative state | 20:58 |
| clarkb | with the requires on the upload you cna have an update to both the assets and gitea images enter the gate and the end result is coherent | 20:58 |
| opendevreview | Ian Wienand proposed opendev/system-config master: gitea: use assets bundle https://review.opendev.org/c/opendev/system-config/+/805933 | 20:59 |
| ianw | ok that adds it | 20:59 |
| clarkb | I've +2'd that as I expect it should work now :) | 21:00 |
| ianw | if that works, i think we can also add the asset to the gerrit container and serve the image file directly from review too | 21:00 |
| clarkb | ianw: yup, though I'm not sure how that works for gerrit? I Guess we serve other assets directly like the theme file? | 21:01 |
| opendevreview | Merged opendev/zone-opendev.org master: Add iweb mtl01 mirror records https://review.opendev.org/c/opendev/zone-opendev.org/+/806805 | 21:03 |
| ianw | clarkb: doesn't it serve up things under "/static"? | 21:05 |
| ianw | i was thinking put the image in there, then update the --header-icon variable in css | 21:06 |
| clarkb | ya looks like if you put things in review_site/static it serves them at /gerrit/static ? | 21:07 |
| clarkb | ianw: currently we bind mount that and populate it via ansible on the host | 21:08 |
| clarkb | but we could move all that content into the docker image and then add in the image? | 21:08 |
| clarkb | or we can write the file to the static dir with ansible and not use the "package" docker image in this instance | 21:09 |
| fungi | zuul seems to think 806798 is safe to go in now, i expect we're going to have some cert refresh job failures until it merges | 21:10 |
| ianw | yes that works too; ansible doesn't have the context issues the gitea container | 21:10 |
| clarkb | fungi: 806798 lgtm but might be good to have ianw check it for any missing bits | 21:13 |
| ianw | lgtm, handlers all gone | 21:15 |
| clarkb | fungi: ^ do you want to approve that change or should I? | 21:31 |
| ianw | clarkb: https://zuul.opendev.org/t/openstack/build/00870d67f01c4605a4ea0bd2cb148365/log/job-output.txt#835 is not the error i was expecting from the gitea build | 21:33 |
| ianw | #8 ERROR: failed commit on ref "config-sha256:c250addbd759142b5feb692703a422348faecf3ddfa2cc8191ab43d265c7bc07": unexpected commit digest sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855, expected sha256:c250addbd759142b5feb692703a422348faecf3ddfa2cc8191ab43d265c7bc07: failed precondition | 21:33 |
| clarkb | I'll approve https://review.opendev.org/c/opendev/system-config/+/803992 tomorrow when I can double check the results of that cron entry | 21:34 |
| clarkb | I'll be afk at that time today so not a good option today | 21:34 |
| clarkb | ianw: hrm I wonder if that is due to various rechecks? Is it possible we're seeing an old sha from the registry or old sha in the zuul db? | 21:35 |
| clarkb | ianw: sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 is the hash for an empty string | 21:35 |
| ianw | clarkb: heh, i'm going to assume you don't just have that committed to memory :) | 22:08 |
| ianw | https://zuul.opendev.org/t/openstack/build/068e9f0bdd69484d9520875a17d96230/log/docker/buildset_registry.txt#94 | 22:12 |
| ianw | Head blob _local opendevorg/assets sha256:c250addbd759142b5feb692703a422348faecf3ddfa2cc8191ab43d265c7bc07 | 22:13 |
| clarkb | ianw: I don't, but I remember the registry serving empty blobs before so cheked it. I thought that was before zuul got its own registry though | 22:20 |
| ianw | looks like the job ran on 158.69.65.114 | 22:21 |
| ianw | "docker logs --since 2h b5f0acdcb0d5 2>&1 | grep --color 158.69.65.114" doesn't have any hits | 22:22 |
| ianw | is it possible it got ...7bc07 via zuul artifacts, but then didn't actually hit the intermediate registry? | 22:24 |
| clarkb | ianw: oh that would be interesting, and docker hub served an empty blob? | 22:29 |
| ianw | first it says | 22:34 |
| ianw | #13 [auth] opendevorg/assets:pull token for zuul-jobs.buildset-registry:5000 | 22:34 |
| ianw | #13 sha256:28d5d2508429d320ed30c9b68d3a33016c268a91ab51841f606dab1fc692834d | 22:34 |
| ianw | mordred's change @ https://opendev.org/zuul/zuul-jobs/commit/70db82198c4bd0a6a0dd78221c2fae4bc7e9682f to write the buildkitd.toml feels *maybe* related | 22:54 |
| clarkb | maybe we aren't instructing buildkit to find the right image? | 22:55 |
| ianw | yeah, but it does seem the registry is seeing queries https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f53/805933/6/check/opendev-buildset-registry/f5336df/docker/buildset_registry.txt | 22:56 |
| ianw | if i put a hold on system-config-build-image-gitea , that's not going to hold the parent registry as well, is it? | 22:58 |
| clarkb | ianw: I don't think so | 22:59 |
| clarkb | ianw: you would need to hold both jobs | 22:59 |
| ianw | this is a bit annoying because zuul-jobs can't depend-on from system-config | 23:00 |
| fungi | or make a dnm change to cause the job in question to busywait/pause while you investigate on the node | 23:01 |
| fungi | rather than using an autohold | 23:01 |
| ianw | that only gives me 90 minutes :) | 23:01 |
| ianw | i feel like this is configured correctly and we may be seeing something odd with buildkit pulling from the buildset registry | 23:02 |
| ianw | either part of that statement may be wrong | 23:02 |
| opendevreview | Ian Wienand proposed opendev/base-jobs master: buildset-registry: add flag to make job fail https://review.opendev.org/c/opendev/base-jobs/+/806818 | 23:14 |
| opendevreview | Shnaidman Sagi (Sergey) proposed openstack/diskimage-builder master: Correct path for CentOS 9 stream base image https://review.opendev.org/c/openstack/diskimage-builder/+/806819 | 23:24 |
| opendevreview | Ian Wienand proposed opendev/system-config master: gitea: use assets bundle https://review.opendev.org/c/opendev/system-config/+/805933 | 23:26 |
| sshnaidm | ianw, please review https://review.opendev.org/c/openstack/diskimage-builder/+/806819 | 23:27 |
| ianw | sshnaidm: what is still using this element? | 23:28 |
| sshnaidm | ianw, for example: https://99957bd7ffedb79bb17e-02cf1f4ef0de29ab49209009be295d1d.ssl.cf2.rackcdn.com/806812/2/check/ironic-python-agent-check-image-dib-centos9/4fbd3bf/job-output.txt | 23:29 |
| ianw | is there a reason this can't use centos-minimal builds? | 23:32 |
| Clark[m] | Would it work if users set the version as 9-stream instead? Seems weird to alias 9 to 9-stream | 23:32 |
| sshnaidm | ianw, idk, but it looks for cloud image in the wrong path.. | 23:32 |
| *** sshnaidm is now known as sshnaidm|afk | 23:34 | |
| ianw | i'm open to suggestions on how to test 806818 | 23:37 |
| opendevreview | Merged opendev/system-config master: Stop managing OpenStackID servers https://review.opendev.org/c/opendev/system-config/+/806798 | 23:38 |
| opendevreview | Merged opendev/system-config master: Introduce iweb cloud configs https://review.opendev.org/c/opendev/system-config/+/806795 | 23:38 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!