| clarkb | then it should magically work if that is all there | 00:00 |
|---|---|---|
| ianw | yeah, it is there | 00:00 |
| ianw | (the buildset registry) | 00:00 |
| ianw | i guess i need it in the project jobs: | 00:01 |
| clarkb | hrm in that case I would expect it to work. You do need to set up the job ordering within the buildset too if building the image and running the job in separate jobs. But otherwise I think that would do it? | 00:01 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] paste service https://review.opendev.org/c/opendev/system-config/+/798400 | 00:03 |
| corvus | ianw, clarkb: shouldn't that be run on changes to lodgeit, not system-config? | 00:04 |
| corvus | basically, build the image in the lodgeit repo, then rely on the buildset registry to bring over the previously built image | 00:05 |
| ianw | i was hoping that depends-on: <lodgeit change> would test in system-config | 00:06 |
| corvus | it should | 00:06 |
| corvus | i believe PS15 is correct | 00:07 |
| corvus | but it's gone now, so i can't see what was wrong with it | 00:07 |
| ianw | i can restore and put that back in the queue, but that wasn't running the lodgeit image build job | 00:07 |
| corvus | it's not supposed to | 00:07 |
| corvus | 798419 runs that job | 00:08 |
| corvus | (or actually, already ran it) | 00:08 |
| corvus | so when run-lodgeit runs, it will fetch the previously built artifact due to the depends on | 00:08 |
| ianw | oh! yeah i forgot about persistence and that it had already run | 00:09 |
| corvus | actually, i guess ps 15 wasn't correct because it had a dependency on the lodgeit build job | 00:09 |
| corvus | so basically, remove all mention of lodgeit-build-opendev-image from 798400 | 00:10 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] paste service https://review.opendev.org/c/opendev/system-config/+/798400 | 00:11 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] paste service https://review.opendev.org/c/opendev/system-config/+/798400 | 00:13 |
| corvus | ianw: couple more quick comments that aren't important now; but check 'em when the dust settles | 00:13 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] paste service https://review.opendev.org/c/opendev/system-config/+/798400 | 00:14 |
| ianw | corvus: thanks. yeah, so the "requires: lodgeit-container-image" should keep things in order if they end up in the queue at the same time, right | 00:15 |
| corvus | yep | 00:25 |
| ianw | thanks | 00:31 |
| ianw | here's another mystery | 00:31 |
| ianw | https://zuul.opendev.org/t/openstack/builds?project=openstack%2Frequirements&pipeline=periodic | 00:31 |
| ianw | i feel like we should have a periodic build ~2021-06-28 06:<something> | 00:31 |
| ianw | every run of the periodic scheduler seems to put into the logs | 00:33 |
| ianw | 2021-06-30 00:31:23,007 DEBUG zuul.Pipeline.openstack.periodic: Starting queue processor: periodic | 00:33 |
| ianw | 2021-06-30 00:31:23,007 DEBUG zuul.Pipeline.openstack.periodic: [e: afdb7039494649c09e8bb2b64158a385] Checking for changes needed by <Branch 0x7fe343803e50 openstack/requirements refs/heads/master updated None..None>: | 00:33 |
| ianw | 2021-06-30 00:31:23,007 DEBUG zuul.Pipeline.openstack.periodic: [e: afdb7039494649c09e8bb2b64158a385] <class 'zuul.model.Branch'> does not support dependencies | 00:33 |
| fungi | ianw: has openstack/requirements fixed its config errors yet? | 00:33 |
| fungi | yeah, i don't see it in the errors list any longer, so it's probably not that | 00:35 |
| ianw | ahh, it probably is that causing it to be skipped | 00:36 |
| ianw | https://review.opendev.org/c/openstack/requirements/+/798361 | 00:36 |
| ianw | yeah, i'd expect todays run | 00:37 |
| fungi | yeah, not sure then | 00:37 |
| ianw | it's been a few months(!) since that release job worked :/ | 00:37 |
| ianw | hrm, lodgeit still pulled from upstream | 01:10 |
| ianw | 2021-06-30 00:42:24.687160 | bridge.openstack.org | Pulling lodgeit ... digest: sha256:c0ff85021d4572123e... | 01:10 |
| ianw | that matches https://registry.hub.docker.com/layers/opendevorg/lodgeit/latest/images/sha256-c0ff85021d4572123ee0b6835331d088b3d37d53296b9cdaffd5cf20df90a647?context=explore | 01:10 |
| ianw | ohh, i guess lodgeit doesn't use the intermediate registry | 01:12 |
| opendevreview | Ian Wienand proposed opendev/lodgeit master: Use opendev-buildset registry https://review.opendev.org/c/opendev/lodgeit/+/798780 | 01:18 |
| opendevreview | Merged opendev/lodgeit master: Use opendev-buildset registry https://review.opendev.org/c/opendev/lodgeit/+/798780 | 02:00 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [wip] paste service https://review.opendev.org/c/opendev/system-config/+/798400 | 03:20 |
| *** ykarel|away is now known as ykarel | 04:24 | |
| *** sshnaidm is now known as sshnaidm|afk | 04:39 | |
| *** marios is now known as marios|ruck | 05:08 | |
| *** ysandeep|away is now known as ysandeep | 05:10 | |
| opendevreview | Ian Wienand proposed opendev/base-jobs master: Add opendev-intermediate-registry-base https://review.opendev.org/c/opendev/base-jobs/+/798789 | 05:28 |
| ianw | corvus / clarkb: ^ re lodgeit. i may be off-base here, but the system-config-run-paste job still isn't consuming the results of the lodgeit change. i can't see how our current system-config jobs are supposed to pull from the intermediate registry, hence ^ | 05:30 |
| opendevreview | Ian Wienand proposed opendev/base-jobs master: Add opendev-intermediate-registry-base https://review.opendev.org/c/opendev/base-jobs/+/798789 | 05:51 |
| *** amoralej|off is now known as amoralej | 06:13 | |
| opendevreview | Dmitriy Rabotyagov proposed opendev/lodgeit master: Add mariadb requirements https://review.opendev.org/c/opendev/lodgeit/+/798411 | 06:27 |
| opendevreview | Merged opendev/lodgeit master: Allow for overriding title https://review.opendev.org/c/opendev/lodgeit/+/798419 | 06:54 |
| *** marios is now known as marios|ruck | 07:16 | |
| *** jpena|off is now known as jpena | 07:39 | |
| *** marios is now known as marios|ruck | 08:09 | |
| opendevreview | Benjamin Schanzel proposed zuul/zuul-jobs master: Fix a bug in s3 log uploader with .gz files https://review.opendev.org/c/zuul/zuul-jobs/+/798802 | 08:15 |
| *** ykarel is now known as ykarel|lunch | 09:24 | |
| *** ykarel|lunch is now known as ykarel | 10:33 | |
| *** sshnaidm|afk is now known as sshnaidm | 10:53 | |
| opendevreview | Ian Wienand proposed opendev/lodgeit master: Add mariadb requirements https://review.opendev.org/c/opendev/lodgeit/+/798411 | 10:54 |
| *** jpena is now known as jpena|lunch | 11:27 | |
| *** dviroel|out is now known as dviroel | 11:29 | |
| roman_g | Good morning, team. Could you have a look at docs publishing job results for me, please? According to the logs, 'promote' job 'promote-airship-project-docs' has copied updated documentation to the AFS share. But the website still hosts old documentation structure. | 11:57 |
| roman_g | Logs: https://zuul.opendev.org/t/openstack/build/18c25e9c1dd2449e98abebe5c9f90ace/console#1/0/27/localhost | 11:57 |
| roman_g | Actual website: https://docs.airshipit.org/airshipctl/cli/airshipctl.html | 11:59 |
| roman_g | Expected: https://98f17a7a28d5343bcff7-cb207f5accf30a643a07da583fd48b42.ssl.cf1.rackcdn.com/789250/21/gate/openstack-tox-docs/7cb1f0a/docs/cli/index.html | 11:59 |
| *** marios|ruck is now known as marios|ruck|call | 12:01 | |
| roman_g | For example, new <<CHANGED>>>f+++++++++ _sources/cli/baremetal/airshipctl_baremetal.rst.txt gets copied to AFS (a bit strange that it's a .txt file), it's visible on a preview site https://98f17a7a28d5343bcff7-cb207f5accf30a643a07da583fd48b42.ssl.cf1.rackcdn.com/789250/21/gate/openstack-tox-docs/7cb1f0a/docs/cli/baremetal/airshipctl_baremetal.html, bit not available on actual website https://docs.airshipit.org/airshipctl/cli/baremetal/airshipctl_baremetal | 12:02 |
| roman_g | .html | 12:02 |
| roman_g | https://docs.airshipit.org/airshipctl/cli/baremetal/airshipctl_baremetal.html | 12:02 |
| *** amoralej is now known as amoralej|lunch | 12:07 | |
| *** jpena|lunch is now known as jpena | 12:27 | |
| *** gthiemon1e is now known as gthiemonge | 12:49 | |
| *** marios|ruck|call is now known as marios|ruck | 12:57 | |
| *** amoralej|lunch is now known as amoralej | 13:16 | |
| rosmaita | when someone has a few minutes, it looks like merged cinder-specs are no longer being published to https://specs.openstack.org/openstack/cinder-specs/ | 13:25 |
| rosmaita | this one is from last week: https://review.opendev.org/c/openstack/cinder-specs/+/797655 | 13:25 |
| rosmaita | this one if from earlier today: https://review.opendev.org/c/openstack/cinder-specs/+/796166 | 13:25 |
| opendevreview | Hitesh Kumar proposed openstack/diskimage-builder master: Migrate from testr to stestr https://review.opendev.org/c/openstack/diskimage-builder/+/789246 | 13:30 |
| roman_g | rosmaita looks like we have similar issue. Your logs are here: https://zuul.opendev.org/t/openstack/build/5d2ad1e705264313b7d86d6091ff95bd/console#1/0/27/localhost | 13:33 |
| fungi | roman_g: rosmaita: the similarities in what you're reporting sounds like we may have problems with static site publication. i have to run some errands this morning but should be able to take a look in a couple of hours if nobody beats me to it | 13:38 |
| roman_g | fungi thank you! | 13:38 |
| rosmaita | fungi: no rush, thanks! | 13:38 |
| *** ysandeep is now known as ysandeep|afk | 13:52 | |
| frickler | infra-root: rosmaita: roman_g: that looks like some brokenness with the changed jobs after the zuul update | 13:55 |
| frickler | drwxr-xr-x 7 10012 10001 2048 Jun 28 15:07 '/afs/.openstack.org/project/airshipit.org/docs/{zuul[project][short_name]}/' | 13:55 |
| frickler | drwxr-xr-x 5 10012 10001 2048 Jun 30 13:08 '/afs/.openstack.org/project/specs.openstack.org/{zuul[project][name]}/' | 13:55 |
| frickler | but I'll leave it to others to fix it, busy day here today | 13:56 |
| frickler | actually you can see that path also in the log that roman_g posted above (https://zuul.opendev.org/t/openstack/build/18c25e9c1dd2449e98abebe5c9f90ace/console#1/0/27/localhost) | 13:57 |
| rosmaita | ok, thanks for troubleshooting | 13:57 |
| roman_g | frickler I thought it is being substituted :) | 13:58 |
| roman_g | Replaced with actual value. | 13:58 |
| roman_g | frickler then I would expect this https://docs.airshipit.org/{zuul[project][short_name]/airshipctl/cli/baremetal/airshipctl_baremetal.html or this https://docs.airshipit.org/{zuul[project][short_name]/cli/baremetal/airshipctl_baremetal.html URLs to possibly work. But it doesn't. | 14:01 |
| clarkb | infra-root (particularly corvus and mordred since you indicated interest) looking like 17:00 UTC tomorrow or Friday for EMS discussion. I've suggested tomorrow so that we don't cut into the weekend, but am waiting for confirmation on that | 14:33 |
| clarkb | mordred: corvus: confirmed that 17:00 tomorrow (Thursday) is the time. They seem interested in using matrix for the call and I've passed along your matrix user links to help coordinate that | 14:38 |
| *** ysandeep|afk is now known as ysandeep | 14:43 | |
| *** ysandeep is now known as ysandeep|away | 15:02 | |
| *** ykarel is now known as ykarel|away | 15:08 | |
| opendevreview | Roman Gorshunov proposed opendev/base-jobs master: Fix mistypes in variables in secrets https://review.opendev.org/c/opendev/base-jobs/+/798915 | 15:15 |
| roman_g | frickler thank you for hints. | 15:17 |
| roman_g | fungi, hopefully this ^^ will fix issue, but some manual cleanup would still be needed on afs share, plus a refresh of HTTP redirects configs. | 15:18 |
| fungi | roman_g: no, since zuul 4.6.0 jinja templating won't work in secrets, it was an exploitable security hole. that's why corvus switched those to use variable substitution in https://review.opendev.org/797910 but maybe there's still something not quite right with them, i'm back now and looking over the logs | 15:25 |
| roman_g | fungi thank you for the explanation. | 15:25 |
| corvus | clarkb: 1700utc tomorrow wfm thanks! | 15:27 |
| fungi | roman_g: my guess is we're missing a lookup() function in the consuming job but checking to be sure | 15:27 |
| corvus | fungi: maybe a playbook missing a .format() call? | 15:27 |
| fungi | er, .format() right | 15:27 |
| fungi | the "Set target path" task is setting target_dir to the literal here: https://zuul.opendev.org/t/openstack/build/18c25e9c1dd2449e98abebe5c9f90ace/console#1/0/14/localhost but it's supposed to be using .format() already according to https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/docs/promote.yaml#L57 | 15:32 |
| fungi | corvus: i don't suppose you can spot the cause there? it looks consistent with what we did elsewhere... are the unset extra variables maybe causing the .format() function to silently fail? | 15:34 |
| *** amoralej is now known as amoralej|off | 15:34 | |
| corvus | oh there's a publish_site job variable | 15:38 |
| corvus | https://zuul.opendev.org/t/openstack/job/promote-airship-project-docs | 15:39 |
| fungi | we do put publish_site=publish_site in the .format() there | 15:40 |
| corvus | fungi: i think the promote.yaml playbook assumes publish_site is a plain string, but that's a template | 15:41 |
| corvus | so it would need its own format() call | 15:41 |
| fungi | ooh | 15:41 |
| corvus | fungi: tbh, that's not safe | 15:41 |
| corvus | well, i take that back | 15:42 |
| corvus | it is safe because it's set in project-config and final:true | 15:42 |
| corvus | so should be fine to add an extra .format(zuul=zuul) to publish_site | 15:43 |
| mordred | clarkb: awesome. I'm free then - I've put it on my calendar | 15:43 |
| corvus | (re safety, we just can't let any individual projects override that; but setting final:true fixes that in place in project-config, so it's good) | 15:43 |
| fungi | corvus: yep, good find. in almost all cases we set publish_site to a plain string constant, promote-openstack-specs and promote-airship-project-docs are the two places in project-config we have variable substitution within the string | 15:46 |
| fungi | and they're the two reported problem cases | 15:46 |
| clarkb | fungi: there is a report in #openstack-infra about a similar issue for another job | 15:47 |
| fungi | target_dir: "{{ target_dict.path.format(zuul=zuul, publish_site=publish_site.format(zuul=zuul), special_publish_directory=special_publish_directory) }}" | 15:47 |
| fungi | nested .format() calls like that? | 15:47 |
| clarkb | promote-deploy-guide for openstack/charm-deployment-guide | 15:48 |
| corvus | fungi: nested should be fine i think | 15:48 |
| corvus | fungi: hrm, that may be a problem if publish_site isn't defined | 15:48 |
| corvus | oh, i guess publish_site is always defined? | 15:50 |
| corvus | yeah, i don't think publishsite=publishsite would work otherwise. so should be fine | 15:50 |
| fungi | mm, i thought previously you had said it looked like unset variables caused them to be ignored in the .format() call, but perhaps it is always set | 15:56 |
| fungi | corvus: also entirely unrelated, but extremely curious... what keeps eating your underscores? | 15:56 |
| corvus | fungi: probably the weechat matrix plugin | 15:57 |
| fungi | neat, i haven't tried it yet | 15:57 |
| fungi | just such an odd sort of bug | 15:58 |
| corvus | i think it's treating them as formatting characters | 16:01 |
| fungi | ahh, like it thinks you're doing _underline_ markup? | 16:03 |
| corvus | ya | 16:03 |
| fungi | corvus: so, in order to test the nested .format() calls with base-test, are we pretty much limited to making alternative versions of some of the jobs in project-config which descend from base-test and then switching some project to using those, since this is promote pipeline stuff? | 16:06 |
| corvus | fungi: yeah; i'd probably just yolo it in and watch the first promote run; at least it won't break everything | 16:11 |
| *** marios|ruck is now known as marios|out | 16:12 | |
| fungi | yeah, it's a pragmatic compromise in this case, otherwise it'll take upwards of a day to get fixed and possibly cooperation of additional teams to merge some things | 16:13 |
| fungi | i'll get the presumed fix pushed up now | 16:13 |
| opendevreview | Jeremy Stanley proposed opendev/base-jobs master: Format publish_site in docs promotion https://review.opendev.org/c/opendev/base-jobs/+/798915 | 16:25 |
| fungi | roman_g: rosmaita: pmatulis: frickler: clarkb: corvus: ^ that's the proposed solution | 16:25 |
| clarkb | fungi: great I'll take a look shortly | 16:26 |
| clarkb | I've +2'd the change. I've held off approving it as I think pmatulis' promote-deploy-guide job doesn't set publish_site and I'm not sure what effect that will have with fungi's fix | 16:37 |
| clarkb | I guess it will just fail the job bceause the var isn't set? | 16:37 |
| fungi | i'll take a closer look at the promote-deploy-guide failures, since if it's not setting publish_site the problem with it may be separate anyway | 16:38 |
| opendevreview | Martin Kopec proposed opendev/system-config master: refstack: trigger image upload https://review.opendev.org/c/opendev/system-config/+/798931 | 16:38 |
| clarkb | I guess we can land the change and fix everything but promote-deploy-guide then they can psuh and update to project-config to set the publish_site | 16:38 |
| fungi | is that the only consumer which needs it set but doesn't have it? | 16:39 |
| clarkb | of that I am not complete sure. It does seem like a number of those jobs do set the value though | 16:39 |
| fungi | yeah, the failure on https://zuul.opendev.org/t/openstack/build/c0c942192d454799b4262395755ea5af is separate i think | 16:41 |
| fungi | it's dying when the "Set target directory if master" task tries to expand {{ afs.targets.master }} | 16:42 |
| corvus | yeah, if the var isn't set, it should already fail | 16:43 |
| clarkb | promote-deploy-guide has a different parent too | 16:43 |
| corvus | adding a format to it won't change that | 16:43 |
| clarkb | the publish_site using jobs all have promote-tox-docs-site-base as a parent but promote-deploy-guide's parent is opendev-promote-docs-base | 16:44 |
| clarkb | Is publish_site specific to promote-tox-docs-site-base ? if so romote-deploy-guide has a similar target_dir dict problem but for another reason? | 16:44 |
| fungi | it looks to me like it's saying the afs dict doesn't have a targets key at all | 16:45 |
| fungi | it should be getting set in the secret | 16:46 |
| clarkb | afsdocs_secret-deploy-guide specifically | 16:47 |
| clarkb | if I search that name I don't get any results | 16:47 |
| clarkb | is the secret just not set at all? | 16:47 |
| clarkb | I suppose it could be on a stable branch and hound isn't seeing it | 16:48 |
| fungi | it's in opendev/base-jobs as afsdocs_secret-deploy-guide | 16:48 |
| fungi | and does have a target with a master path | 16:49 |
| *** jpena is now known as jpena|off | 16:49 | |
| clarkb | the secret needs to be in the repo defining the job though? | 16:50 |
| fungi | er, i mean in openstack/project-9config | 16:50 |
| fungi | https://opendev.org/openstack/project-config/src/branch/master/zuul.d/secrets.yaml#L110-L133 | 16:51 |
| clarkb | The error was: 'dict object' has no attribute 'targets' | 16:51 |
| clarkb | we have target but not targets ? | 16:51 |
| fungi | aha! | 16:51 |
| fungi | great eye | 16:51 |
| fungi | yep, typo. that is probably my error too | 16:51 |
| fungi | i think i fixed all these up post-upgrade | 16:51 |
| fungi | i'll get a fix pushed for that after i double-check it's the only one like that | 16:52 |
| clarkb | sounds good | 16:52 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Correct targets for afsdocs_secret-deploy-guide https://review.opendev.org/c/openstack/project-config/+/798932 | 16:54 |
| fungi | pmatulis: clarkb: ^ that ought to solve it | 16:54 |
| fungi | it was indeed a completely separate problem | 16:54 |
| clarkb | fungi: should I approve the base-jobs change now? | 16:55 |
| clarkb | I had only +2'd it previously but corvus also +2'd which means if you are happy with it we can land it | 16:56 |
| fungi | yeah, just want to watch subsequent promote jobs as soon as it lands | 16:57 |
| fungi | if we have something we can approve which would trigger one of the known problem jobs that would help too | 16:57 |
| clarkb | roman_g: rosmaita ^ may have something? | 16:57 |
| fungi | we'll of course also be equally watching for any other unrelated docs promotion failures this fix unexpectedly might result in | 16:59 |
| roman_g | fungi clarkb Thank you! | 17:01 |
| roman_g | Yes, we would be merging today. | 17:01 |
| roman_g | I've submitted small PS for merge in airship/airshipctl repo. It should trigger docs update. | 17:04 |
| roman_g | https://review.opendev.org/c/airship/airshipctl/+/797732 | 17:04 |
| *** anbanerj|rover is now known as frenzy_friday | 17:07 | |
| fungi | roman_g: the fix hasn't merged yet, but looks like you already approved that change | 17:07 |
| roman_g | fungi Oops. I will find something else. | 17:07 |
| fungi | or you could reset your workflow vote if you catch it before it merges, and then approve again after we get the fix in | 17:08 |
| fungi | if the workflow +1 isn't there when the jobs complete, gerrit won't let zuul merge it and the review will stay open until it gets approved again | 17:08 |
| roman_g | Have reset it. | 17:09 |
| roman_g | Gate jobs are just starting. | 17:09 |
| fungi | they'll still run and report success or (in this case) failure, but the change won't merge without the workflow +1 present when they do | 17:10 |
| roman_g | I would be offline, will submit the change later today. Will read channel logs. | 17:10 |
| fungi | i've self-approved the publish_site fix now | 17:11 |
| roman_g | Write here when it gets merged. | 17:13 |
| roman_g | Will be back later today. | 17:13 |
| fungi | roman_g: i'll let you know, definitely, thanks for the help! | 17:13 |
| roman_g | fungi Thank you! | 17:13 |
| opendevreview | Merged opendev/base-jobs master: Format publish_site in docs promotion https://review.opendev.org/c/opendev/base-jobs/+/798915 | 17:24 |
| fungi | roman_g: you should be clear to approve 797732 as a test any time now, thanks again! | 17:35 |
| fungi | watching https://zuul.opendev.org/t/openstack/builds?pipeline=promote we don't have any newer results yet, but there have been a lot of promote-openstack-releasenotes failures, looking into those next | 17:37 |
| fungi | the field 'args' has an invalid value ({'target_dir': '{{ target_dict.path.format(zuul=zuul, publish_site=publish_site, special_publish_directory=special_publish_directory) }}'}), and could not be converted to an dict.The error was: Single '}' encountered in format string | 17:38 |
| fungi | looks like we missed switching double {{ }} somewhere for that | 17:38 |
| fungi | oh, or maybe that's not it. delving deeper | 17:40 |
| fungi | aha, looks like maybe another typo | 17:41 |
| fungi | "path":"/afs/.openstack.org/docs/releasenotes/{{ zuul[project][short_name]}/" | 17:41 |
| fungi | yep, found it in project-config, another typo of mine! | 17:44 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Correct branch path in afsdocs_secret-releasenotes https://review.opendev.org/c/openstack/project-config/+/798938 | 17:45 |
| fungi | that ought to address it | 17:45 |
| pmatulis | fungi, awesome. i'll test it out | 17:49 |
| fungi | pmatulis: well, the deploy guide fix hasn't merged yet, but it's got a core review on it so i'll self-approve it now | 17:53 |
| fungi | it's a trivially obvious patch anyway | 17:54 |
| fungi | as is 798938 above if any config-core reviewer is available to take a look | 17:55 |
| opendevreview | Merged openstack/project-config master: Correct targets for afsdocs_secret-deploy-guide https://review.opendev.org/c/openstack/project-config/+/798932 | 18:01 |
| fungi | pmatulis: ^ you should be clear to try again now | 18:01 |
| pmatulis | perfect thanks | 18:02 |
| fungi | ugh, many promote-openstack-tox-docs failures now | 18:11 |
| fungi | The task includes an option with an undefined variable. The error was: 'publish_site' is undefined | 18:11 |
| fungi | half-expected | 18:12 |
| fungi | that's using afsdocs_secret-tox-docs | 18:13 |
| fungi | which indeed does not set publish_site | 18:14 |
| fungi | corvus: is there a trivial way to default that to an empty value in the playbook? or do i need to set empty strings for it in every consuming job? | 18:17 |
| fungi | (every consuming job which doesn't already set it) | 18:17 |
| fungi | i guess that would be every job parented to opendev-promote-docs-base, or maybe we can set a fallback value there | 18:18 |
| fungi | i guess an earlier task in the playbook can check whether publish_site is undefined and then set it to some empty value | 18:20 |
| fungi | er, that probably runs afoul of the var freezing though | 18:20 |
| fungi | could a task locally set a variable based on the value of publish_site if defined with a default of some sort, and then we use that in building the path rather than directly consuming publish_site? | 18:23 |
| fungi | finding documentation on the .format() method is not an easy task. in part because i'm not sure if that's handled by directly jinja, ansible, or python | 18:30 |
| fungi | help from anyone with more than my limited, hamfisted ansible knowledge would be most welcome | 18:31 |
| fungi | the word "format" turns up so frequently in things not related at all to that method, i'm having trouble finding out what's responsible for handling it | 18:32 |
| fungi | if ansible were handling it, looks like it would be via a filter construction such as |filter() rather than as a . attribute of the var | 18:34 |
| corvus | fungi: huh, i thought it would have failed before adding the format, was i wrong about that? | 18:36 |
| pmatulis | fungi, failure: https://zuul.opendev.org/t/openstack/build/e160f37b355f4bd5a896711a09887800 | 18:37 |
| fungi | i don't know that you were wrong, all i know is that the error looks like this: https://zuul.opendev.org/t/openstack/build/7c4a1ff4a3694137a389753e5be1fe6a | 18:37 |
| corvus | oh weird | 18:37 |
| fungi | pmatulis: yes, that's the new regression i'm trying to understand now | 18:37 |
| corvus | so "{foo}".format(foo=foo, bar=bar) fails if foo is not defined, but does not fail if bar is not defined | 18:38 |
| corvus | there's definitely some not-quite-python magic going on there | 18:38 |
| fungi | oh yeah i agree that's odd | 18:39 |
| corvus | fungi: i think you'll either need to make sure its always defined, or add a separate task | 18:39 |
| fungi | corvus: see above, will forcing a default when undefined be possible with the var freezing introduced in 4.6.0? | 18:40 |
| fungi | or will we need to use it to set a value for some local proxy variable instead? | 18:40 |
| corvus | the second was what i was thinking. | 18:42 |
| corvus | ``` - set_fact: | 18:44 |
| corvus | publish_site: "{{ '{x}'.format(x=publish_site) }}" | 18:44 |
| corvus | when: publish_site is defined | 18:44 |
| * corvus < https://matrix.org/_matrix/media/r0/download/matrix.org/AjJzRgpMPhYVRBqtOOPNGDRN/message.txt > | 18:45 | |
| corvus | fungi: ^ i think that would work | 18:45 |
| fungi | lookin' | 18:45 |
| corvus | er not quite... | 18:45 |
| corvus | - set_fact: | 18:46 |
| corvus | publish_site: "{{ publish_site.format(zuul=zuul) }}" | 18:46 |
| corvus | when: publish_site is defined | 18:46 |
| fungi | corvus: aha! so setting a fact will let us substitute for the original var, i guess? | 18:46 |
| fungi | oh, shadowing it when it _is_ defined | 18:46 |
| corvus | fungi: there -- i think that's what's needed. yep, so that should do the formatting if the var exists, and if it doesn't, we'll rely on the existing magic behavior that doesn't cause an error if it's not defined. | 18:46 |
| opendevreview | Jeremy Stanley proposed opendev/base-jobs master: Separately set publish_site for docs promotion https://review.opendev.org/c/opendev/base-jobs/+/798945 | 18:50 |
| fungi | corvus: so like that ^ | 18:50 |
| corvus | fungi: lgtm! | 18:52 |
| fungi | thanks | 18:53 |
| fungi | W! | 18:53 |
| fungi | pmatulis: roman_g: 798945 will hopefully take care of the newest regression and then we can check again | 18:55 |
| pmatulis | will wait to test again | 18:57 |
| opendevreview | Merged opendev/base-jobs master: Separately set publish_site for docs promotion https://review.opendev.org/c/opendev/base-jobs/+/798945 | 19:02 |
| roman_g | fungi shall I wait? | 19:07 |
| fungi | roman_g: pmatulis: it *may* be fixed now that 798945 has merged, so worth trying | 19:13 |
| roman_g | ok | 19:13 |
| fungi | i'm back to checking promote jobs to see if that's solved the larger regression | 19:13 |
| fungi | so far no new promote pipeline jobs have reported since that merged, aside from a promote-openstack-releasenotes which is separately broken (fix for that is up for review as 798938) | 19:15 |
| roman_g | Running Post jobs | 19:18 |
| roman_g | Promote will start in about 10min | 19:18 |
| fungi | roman_g: there's this which just succeeded: https://zuul.opendev.org/t/openstack/build/5b161c59b226496d8c0e05a7b63f8c0c | 19:19 |
| roman_g | fungi problem solved. Works perfect. | 19:21 |
| fungi | looks like it wrote into /afs/.openstack.org/project/airshipit.org/docs/airshipctl so that seems correct | 19:21 |
| fungi | roman_g: thanks for confirming! | 19:21 |
| fungi | corvus: ^ looks like that was the solution, thanks! | 19:21 |
| roman_g | fungi thank you! AFS probably needs some cleanup. | 19:21 |
| corvus | \o/ | 19:21 |
| fungi | roman_g: hopefully not since we publish with rsync --delete which in theory should remove the old trees, but worth double-checking | 19:22 |
| roman_g | Good. | 19:22 |
| roman_g | Have a great rest of the day and thank you all for your support. | 19:22 |
| fungi | possible the erroneous paths were entirely outside the normal sync path though, so i'll double-check and clean up any stray cruft | 19:23 |
| fungi | roman_g: you too, and thanks for bringing the issue to our attention1 | 19:23 |
| fungi | s/1/!/ | 19:23 |
| fungi | we also had a successful run of promote-openstack-tox-docs for openstack/os-brick so the most recent regression is definitely addressed now | 19:25 |
| opendevreview | Merged openstack/project-config master: Correct branch path in afsdocs_secret-releasenotes https://review.opendev.org/c/openstack/project-config/+/798938 | 19:42 |
| *** dviroel is now known as dviroel|out | 20:41 | |
| clarkb | infra-root (particularly ianw and corvus I think) upstrema gerrit is discussing Depends-On and how to support it in plugins. Apparently the zuul plugin already does support it and qualcomm has another plugin that wants to support it. | 21:20 |
| clarkb | I don't really have super stron opinions on it. I think its probably ok to have multiple plugins support the same behaviors as long as the behaviors are similar | 21:21 |
| clarkb | but others may have better ideas than "ya go for it" and want ot catch up on the upstrema thread | 21:21 |
| corvus | clarkb: thanks for the heads up; i don't think i have much to add to that | 21:42 |
| fungi | other than "great idea, wish i'd thought of it! (oh, wait...)" | 21:51 |
| clarkb | I did end up responding because zuul support for circular deps came up and I noted zuul does support them now | 21:52 |
| fungi | if memory serves, depends-on footers came about as a result of discussions we had at our mid-cycle in manhattan... is that right? | 21:55 |
| mordred | I know we discussed circular deps in manhattan | 21:57 |
| mordred | but I have a non-substantiated feeling-ish memory that depends-on itself pre-dates that? | 21:57 |
| clarkb | my memory doesn't go back that far | 21:58 |
| fungi | yeah, though i want to say circular deps came up because we hadn't nailed down the implementation for cross-project change dependencies yet. i'm likely misremembering | 21:58 |
| mordred | I feel like people discussed things, and some of them wanted things that were silly, but some of them wanted things that were reasonable | 21:58 |
| fungi | yes, there were opinions | 22:02 |
| fungi | not all of them compatible | 22:02 |
| corvus | opinions were had | 22:03 |
| corvus | mordred: depends-on originally came from gerrit itself and i believe that did pre-date manhattan | 22:03 |
| persia | While I wasn't part of any of the discussions in manhattan, my memory of when that occured suggests that depends-on predates that, but extending depends-on to generic cross-project (or cross-hosting solution) does not. | 22:03 |
| fungi | oh, gerrit added depends-on tracking in commit footers? | 22:03 |
| corvus | fungi: no, but there was a patch for it | 22:04 |
| fungi | persia: definitely cross-hosting syntax came much later in zuul v3 | 22:04 |
| corvus | that everyone thought would be merged; but then it wasn't. it may have gotten caught up in the same CLA snafu as single-page-diff | 22:04 |
| fungi | aha | 22:04 |
| fungi | so we thought zuul's depends-on would be compliant with a feature which ended up not being carried in gerrit. this seems like a recurring theme ;) | 22:05 |
| corvus | yep, and so i think our picking that up and extending it to support multiple sources is appropriate since gerrit itself abdicated it | 22:06 |
| fungi | https://review.openstack.org/144555 was the change to implement it in zuul, first patchset uploaded new year's eve of 2014 | 22:07 |
| fungi | i seem to have approved it on february 1 | 22:08 |
| corvus | i know how to par-tay | 22:09 |
| fungi | indeed | 22:10 |
| ianw | clarkb / corvus : https://review.opendev.org/c/opendev/base-jobs/+/798789 came out of me trying to use speculative images yesterday. i might be wrong, but i don't see where system-config pulls from the intermediate registry? could you take a look at that change? | 22:10 |
| clarkb | neat we fixed my concern with zuulv3 :) | 22:10 |
| clarkb | ianw: I think it relies on the parent jobs configuring docker to use the intermediate registry first then fallback to dockerhub | 22:11 |
| clarkb | ianw: I want to say what your change is trying to do there is supposed to already happen automatically via the job hierarchy | 22:11 |
| ianw | clarkb: it's very possible i've got something wrong, but i couldn't seem to make that happen | 22:12 |
| corvus | ianw: i think you just need to run opendev-buildset-registry | 22:12 |
| ianw | this is what i had assumed too, but https://review.opendev.org/c/opendev/system-config/+/798400 is running opendev-buildset-registry and didn't pull in the depends-on change | 22:14 |
| ianw | (i have a few comments with links @ 798789) | 22:14 |
| corvus | ianw: does the run job run use-buildset-registry? | 22:15 |
| ianw | yes, here it is running on paste which is where we want to pull the image https://zuul.opendev.org/t/openstack/build/c878c86b207d4cbf971c87abd5f70748/console#1/0/40/paste01.opendev.org | 22:18 |
| corvus | ianw: gotcha; then i think your change may be correct | 22:19 |
| corvus | ianw: i think that probably is the missing piece; i don't know off the top of my head if there's something else we expected to run the pull (i'm assuming you've looked and didn't find anything). | 22:20 |
| ianw | yeah, i could not find anywhere other than build-docker-image @ https://opendev.org/zuul/zuul-jobs/src/branch/master/zuul.d/docker-jobs.yaml#L9 | 22:21 |
| ianw | i considered maybe just putting it in the opendev base job toott | 22:22 |
| ianw | it seemed like it should do nothing if no buildset registry defined | 22:22 |
| clarkb | so we add the pre playbook for image builds which populates the buildset registry in jobs that are't building imgaes? | 22:24 |
| ianw | clarkb: it's perhaps misnamed @ https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/docker-image/pre.yaml | 22:25 |
| ianw | it only runs the one task which is pull-from-intermediate-registry | 22:26 |
| clarkb | ianw: ya it was added for the imgae builds so it got the name. I don't know that we need to rename it just making sure I'm follwoing along | 22:26 |
| clarkb | basically we do the right thing when building images but not when consuming them? | 22:26 |
| ianw | yes, since that role runs as part of build-docker-image, whenever we're in the context of that running against our buildset registry things work | 22:27 |
| clarkb | opendev-build-docker-image-base seems to be equivalent to your chang but it adds the secret in too | 22:27 |
| clarkb | do we need the secret? if yes then maybe just reuse opendev-build-docker-image-base if not then maybe we should note that in the new base job | 22:28 |
| ianw | i would think we only need the secret to push, which in this case (consuming lodgeit container) we don't | 22:29 |
| clarkb | got it | 22:30 |
| ianw | however, perhaps something like that which also starts the opendev-buildset-registry would work? | 22:30 |
| clarkb | and ya I agree the intermediate registry allows anonymous reads | 22:30 |
| ianw | that might mean jobs with this theoretical "opendev-use-docker-image-base" parent don't have to also run opendev-builset-registry? | 22:31 |
| ianw | what happens if we have two buildset registries, though? | 22:32 |
| clarkb | do the registry all in one? I think we colocate into a central paused job buildest registry to avoid extra IO to the various registries but I think that would work | 22:32 |
| clarkb | really the balance is limiting bw usage to the outside world aiui | 22:32 |
| ianw | hrm, there might be room for both then. one like proposed for something like system-config that runs a lot of container jobs, and a self-contained one if you have just one job that speculatively consumes another project's containers | 22:34 |
| ianw | i'm guessing nobody actually has a project like the latter atm | 22:35 |
| clarkb | ya it seems to be common to grow multiple jobs. For example Zuul uses the same images all over but then runs a ton of different jobs on them | 22:36 |
| clarkb | I +2'd the base jobs change. There rae no secrets involved and it adds a new job without modifying existing jobs That should mean its safe to land without any additional testing or exposure concerns | 22:39 |
| ianw | i think statusbot from system-config would have the same problem; but i probably never tried a depends-on for any changes | 22:40 |
| clarkb | ianw: I'll let you approve it if you want to proceed with that method of using the buildset registry rathe rthan all in one | 22:40 |
| corvus | that's still using the all-in-one | 22:40 |
| clarkb | oh is it? | 22:40 |
| corvus | iiuc the idea would be to reparent the run job on that so that it pulls in the dependency | 22:41 |
| corvus | but it'll still pull it in to the buildset registry | 22:41 |
| clarkb | ya rereading the docstring on the job that seems to be the case | 22:41 |
| clarkb | will it pull ot a central buildest registry too? | 22:41 |
| clarkb | supporting both cases? | 22:41 |
| corvus | (btw, i don't think we should have more than one job run a buildset registry) | 22:42 |
| corvus | (it's called a buildset registry for a reason :) | 22:42 |
| ianw | corvus: yeah, i think we're saying in contrast to https://opendev.org/opendev/base-jobs/src/branch/master/zuul.d/jobs.yaml#L167 there is no dependency on opendev-buildset-registry in the job | 22:43 |
| corvus | the natural next step to evolve this would be to collect the artifacts for the whole buildset so that the buildset registry job can do the pulling for all the jobs, then this change wouldn't be needed. | 22:43 |
| ianw | so you have to arrange for your project to be running that | 22:43 |
| corvus | yep, and system-config does | 22:44 |
| ianw | whereas we could *also* add a job "opendev-use-docker-image-base" that has a " parent: opendev-buildset-registry" for projects to use if they just want to consume someone else's container | 22:44 |
| corvus | please don't; the buildset registry is not designed for that | 22:45 |
| clarkb | now I'm confused again. The change that has been +2'd is designed to work with a central buildest registry run in a paused job right? | 22:46 |
| corvus | it's designed to work with a buildset registry; it doesn't care where it runs | 22:47 |
| corvus | (which is the right way to deal with a buildset registry :) | 22:47 |
| clarkb | well if you inherit from that job and then run the buildest registry in pre-run that will be too late due to the nesting behavior of jobs | 22:47 |
| clarkb | which means you're probably looking for a buildest registry elsewhere and not an all in one setup | 22:47 |
| corvus | true | 22:48 |
| ianw | corvus: i'm not sure i follow why https://opendev.org/opendev/base-jobs/src/branch/master/zuul.d/jobs.yaml#L167 has a parent of opendev-buildset-registry but a sibling "opendev-use-docker-image" would be a wrong pattern | 22:50 |
| corvus | that's fine, it does the right thing; i'm just saying please don't run more than one buildset registry | 22:52 |
| corvus | i think there are too many ideas being discussed at once, sorry for the confusion | 22:52 |
| ianw | ahh, right, yes we all agree that one buildset registry per ... buildset ... is the way to go (it's in the name :) | 22:54 |
| ianw | now i look at https://zuul-ci.org/docs/zuul-jobs/docker-image.html i feel like i could add some relevant info on consuming images that would be helpful | 22:55 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Add a growvols utility for growing LVM volumes https://review.opendev.org/c/openstack/diskimage-builder/+/791083 | 23:17 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!