| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Add container build jobs https://review.opendev.org/c/zuul/zuul-jobs/+/878291 | 01:14 |
|---|---|---|
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [dnm] test direct push https://review.opendev.org/c/zuul/zuul-jobs/+/878296 | 01:42 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [dnm] test direct push https://review.opendev.org/c/zuul/zuul-jobs/+/878296 | 01:54 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [dnm] test direct push https://review.opendev.org/c/zuul/zuul-jobs/+/878296 | 02:01 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [dnm] test direct push https://review.opendev.org/c/zuul/zuul-jobs/+/878296 | 02:07 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: buildx: remove experimental flags https://review.opendev.org/c/zuul/zuul-jobs/+/878293 | 02:37 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: buildx: remove experimental flags https://review.opendev.org/c/zuul/zuul-jobs/+/878293 | 03:06 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [dnm] test direct push https://review.opendev.org/c/zuul/zuul-jobs/+/878296 | 03:06 |
| ianw | there is a https://gerrit.googlesource.com/gerrit/+/v3.7.2 now | 03:22 |
| ianw | https://23.253.56.187/ is a held host | 03:23 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [dnm] test direct push https://review.opendev.org/c/zuul/zuul-jobs/+/878296 | 03:26 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [dnm] test direct push https://review.opendev.org/c/zuul/zuul-jobs/+/878296 | 03:32 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [dnm] test direct push https://review.opendev.org/c/zuul/zuul-jobs/+/878296 | 03:40 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [dnm] build-container-image : refactor buildx a bit https://review.opendev.org/c/zuul/zuul-jobs/+/878296 | 05:18 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [dnm] testing manifest create https://review.opendev.org/c/zuul/zuul-jobs/+/878304 | 06:06 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [dnm] testing manifest create https://review.opendev.org/c/zuul/zuul-jobs/+/878304 | 06:34 |
| ianw | i think review is not happy :/ | 06:53 |
| tkajinam | seems we have some trouble with gerrit ? | 07:02 |
| tkajinam | 502 proxy error | 07:05 |
| tkajinam | hm seems it's back | 07:05 |
| ianw | i've banned and ip and restarted it ... we'll see | 07:05 |
| ianw | tkajinam: that was just the restart, try again should work | 07:05 |
| tkajinam | ah, ok now gerrit dashboard loads and git review passes. | 07:06 |
| tkajinam | ianw, thanks ! | 07:06 |
| ianw | heh don't thank me yet, i'm still not sure what exactly is happening :) | 07:06 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [dnm] testing manifest create https://review.opendev.org/c/zuul/zuul-jobs/+/878304 | 07:07 |
| tkajinam | wired. it seems gerrit created two review urls for same change id | 07:07 |
| tkajinam | https://review.opendev.org/c/openstack/tripleo-ansible/+/878310 | 07:07 |
| tkajinam | https://review.opendev.org/c/openstack/tripleo-ansible/+/878311 | 07:08 |
| tkajinam | I'll abandon these as I remember some problems with a similar situation and will push it with a different change id. I guess we can just abandon these but I'll share this for records | 07:08 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [dnm] testing manifest create https://review.opendev.org/c/zuul/zuul-jobs/+/878304 | 07:50 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [dnm] testing manifest create https://review.opendev.org/c/zuul/zuul-jobs/+/878304 | 08:04 |
| *** jpena|off is now known as jpena | 08:23 | |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [dnm] testing manifest create https://review.opendev.org/c/zuul/zuul-jobs/+/878304 | 08:51 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [dnm] testing manifest create https://review.opendev.org/c/zuul/zuul-jobs/+/878304 | 09:56 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [dnm] testing manifest create https://review.opendev.org/c/zuul/zuul-jobs/+/878304 | 10:23 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [dnm] testing manifest create https://review.opendev.org/c/zuul/zuul-jobs/+/878304 | 10:31 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [dnm] testing manifest create https://review.opendev.org/c/zuul/zuul-jobs/+/878304 | 10:47 |
| *** JasonF is now known as JayF | 13:56 | |
| fungi | we're back to higher activity levels in openstack now that the release is past, and the rax-ord graph looks pretty healthy alongside the other two regions at ~full use: https://grafana.opendev.org/d/a8667d6647/nodepool-rackspace?orgId=1&from=now-6h&to=now | 14:50 |
| fungi | we're seeing ~75% in use and 25% building/deleting in rax-ord, whereas previously it would have been almost 0% in use and 100% building/deleting | 14:52 |
| fungi | so i think we can close out that chapter | 14:52 |
| clarkb | that is good news. I'm sorting out tea and breakfast but I need to rereview gitea server load/demand and decide if I'm proceeding with old server permanent removal today | 15:12 |
| clarkb | I should considering replacing python3.8 and 3.10 with just 3.11. >2600 packge updates due to python getting all the updates | 15:36 |
| fungi | that's certainly one up side to distros that pick one version of python at a time | 15:37 |
| clarkb | I finished my tea before local updates today. heh | 15:39 |
| dtantsur | clarkb: hi, has https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032773.html resolved your concerns re image deletions? I somehow missed your question, but I think Jay addressed it | 16:05 |
| clarkb | dtantsur: I think maybe half addressed. It wasn't clear to me how that mapped onto the specific request and the deletion for specific files. I also think there continues to be a bug in how ironic is publishing images based on that feedback. It seems you expect people to never use old images but then don't publish a "just use this image" image and instead publish very version and | 16:06 |
| clarkb | disto specific things that need periodic cleanup? | 16:06 |
| clarkb | What I take away form that is that ironic should probably publish a single golden image and expect people to use that. Then update that image over time | 16:06 |
| dtantsur | clarkb: we stopped publishing the images that are part of this clean up. We no longer publish images for bugfix branches (even up-to-date ones) because we cannot offer a reasonable support for them. | 16:07 |
| clarkb | aha that is the bit I was missing | 16:07 |
| dtantsur | we do publish images per stable branched (and don't delete them) | 16:07 |
| dtantsur | we mark the images with distros for clarity, but we have only one distro per each branch | 16:07 |
| dtantsur | that does lead to situations like when we start publishing "master" images on centos8 and then have to switch to centos9 and remove the incorrect images | 16:08 |
| dtantsur | (keeping the centos8 image is misleading because it's not actually "master", it's "master at the point before the switch") | 16:09 |
| clarkb | ya I think that should be avoided | 16:09 |
| clarkb | your users don't actually care what the distro is right? this is a utility that gets booted to execute things on the host prior to/after user images run | 16:10 |
| dtantsur | well.. they do. distro implies shipped drivers. | 16:10 |
| clarkb | but that centos 8 version is dead and unsupported now and will never receive updates? | 16:10 |
| dtantsur | we had to keep support for CentOS 7 exceptionally long because CS8 removed some drivers for hardware still in use | 16:10 |
| dtantsur | centos8-master is dead, yes | 16:11 |
| clarkb | right but that may not be clear to anyone who is using it | 16:11 |
| clarkb | whereas if you just have the latest image the would automatically convert over | 16:11 |
| dtantsur | it can be a breakage either way. a new distro may (and does in practice) stop supporting certain hardware. | 16:11 |
| dtantsur | we'll consider going versionless once we reach CS10 :) | 16:12 |
| dtantsur | (if we drop the version or the whole distro prefix now, we'll once again end up with images to delete) | 16:12 |
| clarkb | right but that can be a one time thing instead of an every 6 months thing | 16:13 |
| dtantsur | yeah, but why do it now instead of at the next update? | 16:13 |
| dtantsur | (I hope CentOS Stream does not get a new major version every 6 months, that's the reason we picked it over e.g. ubuntu or fedora) | 16:13 |
| JayF | Yeah, we can't drop the distro out of the IPA image names because that's somethign an operator needs to know | 16:14 |
| JayF | we often see folks break during those transitions because of upstream hardware support changes | 16:14 |
| JayF | I think this seems like a bigger problem because dtantsur is cleaning up literally almost a decade of bad IPA images | 16:15 |
| JayF | whereas in the normal case now I believe we'd be removing 1/2 at a time if we catch up and stay caught up | 16:15 |
| fungi | i remember deleting some a few years ago, but maybe it was a more selective cleanup and not a purge | 16:15 |
| JayF | we did a big move, and you cleaned up the old location | 16:16 |
| dtantsur | fungi: it was because of a mistake in the build jobs. unfortunately, they're not trivial to test in advance (if they are, please tell me how) | 16:16 |
| dtantsur | and maybe that as well | 16:16 |
| fungi | in theory we could do something with them like we do for testing container images | 16:16 |
| clarkb | dtantsur: mostly I brought it up because requests like that are a major "smell" for automation/CI/release managment/packaging | 16:16 |
| JayF | Are there other openstack projects that ship, as a deliverable, a full OS image/ramdisk? | 16:17 |
| clarkb | I wanted to take it as an opportunity for the ironic project to think about how they organize and publish artifacts that end users rely on. So that we can avoid needing to potentially break them again later | 16:17 |
| JayF | If so, how do they manage this problem? | 16:17 |
| JayF | I guess there's even another angle on it: a full OS image/ramdisk targetted at hardware, not VMs | 16:17 |
| clarkb | JayF: I think octavia and trove and sahara also publish images, but their official process is for people to build them locally | 16:17 |
| clarkb | they are os images not ramdisks | 16:18 |
| fungi | virtual machine images basically, yes | 16:18 |
| JayF | I think this is why we have a bigger problem; in our case hardware support matters so we can't handwave the distro or version | 16:18 |
| johnsom | Octavia builds an image, but it is marked for testing only and is primarily used by the deployment tool test jobs. | 16:18 |
| JayF | making it ugly when we have to change those b/c we don't delete the old (is there a way we could automate this piece? I doubt it?) | 16:18 |
| clarkb | JayF: but I'm hearing that you kinda do by silently dropping support for centos 8 | 16:18 |
| clarkb | I get that the centos 8 image might be necessary but in that case you should probably continue updating it? | 16:19 |
| opendevreview | Elod Illes proposed openstack/project-config master: Unpin nodeset for tag-releases https://review.opendev.org/c/openstack/project-config/+/878472 | 16:19 |
| clarkb | johnsom: ok cool my memory was correct then | 16:19 |
| clarkb | JayF: dtantsur to be clear I don't know what the answers are. I just wanted people to think about them before we go and delete things | 16:19 |
| JayF | clarkb: CentOS Stream 8 EOL'd in 2021 | 16:19 |
| clarkb | JayF: right but stream still exists and presumably with os driver overalp | 16:20 |
| JayF | We support CentOS 9 Stream now, which is next | 16:20 |
| clarkb | right but dtantsur is saying that 8 doesn't have the same os drivers as 9 so some people are likely still intentionally using a now not updated 8 image? | 16:20 |
| JayF | clarkb: I don't love the stream model, and would personally prefer we test against a more stable Rocky or Alma linux; but that's not just my choice so I go with what the community likes | 16:21 |
| clarkb | But like I said I'm not here to prescribe anything. More just "this is a good opportunity to look at why we are deleting things and determine if that is appropriate or necessary and shift if appropriate" | 16:21 |
| JayF | clarkb: or in those cases, we have modified images to make them have better support | 16:21 |
| JayF | clarkb: we actually are going to add a package to our centos 9 images to fix an issue with some stuff that was moved into extra-hardware | 16:21 |
| JayF | My answer is: I agree this all seems like it could be better. I'm not sure what concrete things we could do given current limitations to improve it. | 16:22 |
| clarkb | from what little I know it soundslike maybe you should publish a single up to date image for each base distro that you support | 16:22 |
| clarkb | that gets you driver coverage and ensures that the tooling is up to date and doesn't get stale (staleness being the concenr leading to deletion aiui) | 16:23 |
| JayF | The problem with that comes when we change distros, from Version X to X+1. We publish master iamges so in those cases; we have to cleanup the old distro-named releases | 16:24 |
| clarkb | just me talking out loud here: maybe master images should be testing only and not be distro specific. Just have an image that meets the need of development. Then you have centos-7-ipa-latest centos-8-stream-ipa-latest centos-9-stream-ipa-latest (or similar) for that latest release of ipa | 16:27 |
| clarkb | Then the only time you delete anything is when a centos version is 10 years old and no longer getting updates | 16:28 |
| clarkb | and it a single file with a clear end of life plan | 16:28 |
| JayF | CentOS doesn't follow that model anymore. There is no LTS for CentOS Stream. | 16:28 |
| JayF | At least AIUI | 16:28 |
| clarkb | there is some planned sunset though | 16:28 |
| clarkb | whatever it is | 16:28 |
| JayF | for the CentOS Stream 8, it was 2021. and 9 is the current version | 16:28 |
| JayF | so there's only one version currently supported | 16:28 |
| clarkb | centos 8 stream is currently supported | 16:28 |
| clarkb | looks like may 31 2024 is the centos 8 stream eol | 16:29 |
| * JayF finding conflicting info on the internet and now looking for a more authoritative source | 16:29 | |
| dtantsur | Are you suggesting we keep building C7/CS8 versions of master IPA? these releases are not in PTI and their Python 3 versions make everything complicated | 16:29 |
| clarkb | so thats the date you delete those images | 16:29 |
| clarkb | dtantsur: what I'm hearing is that some users need centos 7 for old hardware. If I'm a ironic user I would expect ipa to continue to work with the old hardware I have | 16:30 |
| dtantsur | yeah, and we kept it supported for longer than we wanted. but we cannot any more. | 16:30 |
| clarkb | and not require me to use old unsupported buggy insecure releases | 16:30 |
| clarkb | if ironic isn't able to do that (which I a totally valid stance from a development perspective) then I think that should be more clear and we can probably help do that via the publication practices | 16:31 |
| clarkb | JayF: https://www.centos.org/centos-stream/ states May 31, 2024 fwiw | 16:33 |
| JayF | clarkb: yeah I see that, but is CentOS Stream 8 still in the PTI? | 16:33 |
| dtantsur | yep, CS8 is still alive and kicking. but its Python 3.6 makes it harder for us to build on it. | 16:33 |
| dtantsur | in any case, our image choice is to a large extent driven by the PTI | 16:33 |
| dtantsur | s/image/distro/ | 16:34 |
| dtantsur | Our policy is essentially "pick a distro from the PTI and publish images with it until it changes" | 16:34 |
| dtantsur | which may not be great, but it's a nice compromise between putting a reasonable load on us and putting up any images at all | 16:35 |
| JayF | we need to *notify* operators if that changes so they know what's up, and we can try to get their stuff working on newer releases | 16:35 |
| JayF | hence the version id being in the name | 16:35 |
| dtantsur | do you think IPA-builder release notes are not enough? we may pick another venue | 16:35 |
| JayF | honestly this whole chat makes me wonder if we need to publish a readme alongside our tarballs outlining this policy along with the cleanup | 16:37 |
| JayF | or something similar | 16:37 |
| JayF | this is one of those things where our behavior is documented, but in like 3 different places spread across openstack docs :D | 16:37 |
| clarkb | JayF: that is not a bad idea. Even if it serves as a pointer to those other locations. Since people are going to grab the images in that spot they will likely notice a readme | 16:38 |
| clarkb | I know i read the readmes for things like the openshift client releases that way | 16:38 |
| dtantsur | is it possible to somehow render the readme in the listing? | 16:38 |
| dtantsur | I know some projects do it | 16:38 |
| clarkb | but again I was just trying to make sure we'd done some due diligence before deleting things to make sure we weren't going to surprise people and to avoid getting in a spot where we repeat potential mistakes | 16:39 |
| clarkb | dtantsur: if you supply the html/js/css | 16:39 |
| clarkb | but no by default its just apache mod autoindex iirc | 16:39 |
| JayF | clarkb: yeah, from our perspective this is about trying to make sure we don't publish images that just won't work if someone downloads them | 16:39 |
| JayF | clarkb: so if I have a vote; I'd say dtantsur and I will take the action to more clearly document this alongside, but can we execute the changes requested to make sure in the interim, users don't get years-old images? | 16:40 |
| clarkb | JayF: when you say won't work thats with latest ironic right? | 16:40 |
| dtantsur | yeah, particularly coreos-image, which sounds cool until you understand it's even not the coreos that exists now | 16:40 |
| clarkb | one of my questions was whether or not it would be necessary for old installs out there | 16:40 |
| JayF | We publish images versioned to our stable versions | 16:40 |
| clarkb | (it sounds like no generally an old install can use a newer ipa as long as the base ipa image supports their hardware) | 16:41 |
| JayF | so if you were installing Train (weeps inside), you could download our train-published image | 16:41 |
| dtantsur | we keep branched images up to liberty actually: https://tarballs.opendev.org/openstack/ironic-python-agent/coreos/ | 16:41 |
| JayF | clarkb: it's actually pretty flexible. We had one Ironic<>IPA break (agent token) but if you're on the right side of that, you're mostly OK | 16:41 |
| dtantsur | although these use the ancient coreos, I don't request deleting them | 16:41 |
| JayF | and I bet they work with the proper ironic version | 16:41 |
| JayF | no reason they wouldn't | 16:42 |
| dtantsur | but the master one is dangerous since it may imply that the image works with ironic master. and it may or may not, or may break at some weird point. nobody knows nowadays. | 16:42 |
| JayF | btu it's slightly horrifying to think about someone actually running a published ramdisk image from 7 years ago lol | 16:42 |
| dtantsur | yeah, I hope they understand the risks :) | 16:42 |
| clarkb | I agree its scary, but what I've found as a cloud user is that often times we don't make those decisions | 16:45 |
| clarkb | granted the ipa image is more of a deployment decision | 16:45 |
| * JayF goes back to his roost | 16:46 | |
| JayF | *roots | 16:46 |
| JayF | if someone is running Liberty Ironic, bless their heart | 16:46 |
| JayF | the amount of pain they are feeling for a vastly inferior cloud project is immense :| | 16:47 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Add container build jobs https://review.opendev.org/c/zuul/zuul-jobs/+/878291 | 16:47 |
| clarkb | talking out loud here: you mentioned you don't use ubuntu because they have a major release every 6 months. This isn't really true anymore. The LTS releases occur every 2 years. And one thing Ubuntu is actually really good at ime is hardware compatibility. Paritcularly with their hardware enablement kernels. Just throwing that out there | 16:56 |
| dtantsur | clarkb: yeah, there was also a trademark concern.. maybe JayF remembers it better, something about removing ubuntu branding if you modify it enough. | 17:09 |
| opendevreview | Merged zuul/zuul-jobs master: Add container build jobs https://review.opendev.org/c/zuul/zuul-jobs/+/878291 | 17:09 |
| JayF | Way back in the day there was an issue publishing images labelled "ubuntu" without their explicit consent | 17:09 |
| clarkb | ya I remember that. I thought it was only if you were rebuilding things but I'm not a lawer | 17:10 |
| JayF | It was not, at least in context of Rackspace Cloud at the time. | 17:10 |
| JayF | I mean, that's not a good reason to not do it unless we actually investigated that space | 17:10 |
| JayF | but traditionally that's why Ironic published non-ubuntu images | 17:11 |
| JayF | debian and then flipped to centos at some point while I was mostly-gone from the project iic | 17:11 |
| JayF | *iirc | 17:11 |
| clarkb | infra-root I've scanned the cacti graphs for gitea09-14 and I don't see anything that concerns me since the openstack release. I've removed my WIP vote on https://review.opendev.org/c/opendev/system-config/+/877298 and if I don't hear any objections I'll proceed with cleaning gitea01-04 up today | 17:26 |
| clarkb | that first change stops gerrit replication to the 4 servers. I'll trigger full gerrit replication after that lands to ensure we didn't miss anything | 17:26 |
| *** jpena is now known as jpena|off | 17:26 | |
| clarkb | Once that is done we can land https://review.opendev.org/c/opendev/system-config/+/877301/ to pull the servers out of config management. THen we delete them and then clean up dns | 17:27 |
| opendevreview | James E. Blair proposed opendev/base-jobs master: Add container-image jobs https://review.opendev.org/c/opendev/base-jobs/+/878288 | 17:47 |
| corvus | clarkb: ^ updated that change and replied | 17:48 |
| clarkb | corvus: ack so avoid documenting it in that section entirely since the parent jobs capture it | 17:49 |
| corvus | clarkb: well, more like it's just one possible way to use the jobs. so if we want to have some narrative structure around the different possible ways, let's put that in something like https://opendev.org/opendev/base-jobs/src/branch/master/doc/source/docker-image.rst | 17:50 |
| clarkb | got it | 17:50 |
| corvus | (the removal of that section is an unrelated change -- it's a duplicate of what's in credentials.rst) | 17:50 |
| corvus | (basically the job docs split that part up, so each job's docs only shows the relevant jobvars) | 17:51 |
| opendevreview | Merged opendev/base-jobs master: Add container-image jobs https://review.opendev.org/c/opendev/base-jobs/+/878288 | 18:10 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Add docker buildx multiarch support to container roleset https://review.opendev.org/c/zuul/zuul-jobs/+/878246 | 18:14 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: buildx: remove experimental flags https://review.opendev.org/c/zuul/zuul-jobs/+/878293 | 18:14 |
| opendevreview | James E. Blair proposed opendev/base-jobs master: Add ensure-* roles to container image jobs https://review.opendev.org/c/opendev/base-jobs/+/878478 | 18:52 |
| corvus | clarkb: ^ i missed a thing | 18:52 |
| clarkb | loking | 18:52 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [dnm] testing manifest create https://review.opendev.org/c/zuul/zuul-jobs/+/878304 | 19:25 |
| ianw | Do you really want to submit the above commits? | 19:34 |
| ianw | Type 'yes' to confirm, other to cancel: yes | 19:34 |
| ianw | error: remote unpack failed: error Missing blob 098d484a6143c07ea9aeaa261da1785d42e53ee9 | 19:34 |
| ianw | fatal: Unpack error, check server log | 19:34 |
| ianw | that's a new one ... | 19:34 |
| ianw | this is in zuul-jobs | 19:36 |
| opendevreview | Merged opendev/base-jobs master: Add ensure-* roles to container image jobs https://review.opendev.org/c/opendev/base-jobs/+/878478 | 19:40 |
| ianw | i can replicate it but i don't know why | 19:42 |
| ianw | git clone https://opendev.org/zuul/zuul-jobs | 19:42 |
| ianw | git fetch https://review.opendev.org/zuul/zuul-jobs refs/changes/46/878246/7 && git checkout FETCH_HEAD | 19:42 |
| ianw | add a change | 19:42 |
| ianw | git review | 19:42 |
| corvus | difference in packing on those 2 repos? (and possibly a difference on a specific gitea backend?) | 19:43 |
| Clark[m] | I think that's the --no-thin thing | 19:43 |
| Clark[m] | Git review has a flag to set that now and it has to do with some jgit thing iirc | 19:43 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [dnm] testing manifest create https://review.opendev.org/c/zuul/zuul-jobs/+/878304 | 19:45 |
| ianw | Clark[m]: ahh, good call. first time i've seen that! | 19:45 |
| ianw | i guess cloning from gitea but then pushing to gerrit is probably at the heart of it somewhere | 19:48 |
| opendevreview | lotorev vitaly proposed opendev/system-config master: docs: Fix link to https://testinfra.readthedocs.io/ https://review.opendev.org/c/opendev/system-config/+/878435 | 19:54 |
| opendevreview | Merged zuul/zuul-jobs master: Add support for passing env vars to the container build env https://review.opendev.org/c/zuul/zuul-jobs/+/878239 | 19:55 |
| opendevreview | James E. Blair proposed opendev/base-jobs master: Fix container-image pre playbook container_command default https://review.opendev.org/c/opendev/base-jobs/+/878482 | 19:56 |
| opendevreview | lotorev vitaly proposed opendev/system-config master: docs: Fix link to https://testinfra.readthedocs.io/ https://review.opendev.org/c/opendev/system-config/+/878435 | 19:56 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Fix container-image pre playbook container_command default https://review.opendev.org/c/zuul/zuul-jobs/+/878483 | 19:57 |
| opendevreview | lotorev vitaly proposed opendev/system-config master: docs: Fix link to https://testinfra.readthedocs.io/ https://review.opendev.org/c/opendev/system-config/+/878435 | 20:02 |
| opendevreview | lotorev vitaly proposed opendev/system-config master: Fix typos in doc/source/open-infrastructure.rst https://review.opendev.org/c/opendev/system-config/+/878435 | 20:14 |
| opendevreview | James E. Blair proposed opendev/base-jobs master: Move pull-from-intermediate-registry to localhost https://review.opendev.org/c/opendev/base-jobs/+/878484 | 20:19 |
| corvus | Clark: ianw ^ another oops | 20:19 |
| opendevreview | Merged opendev/base-jobs master: Fix container-image pre playbook container_command default https://review.opendev.org/c/opendev/base-jobs/+/878482 | 20:40 |
| opendevreview | Merged opendev/base-jobs master: Move pull-from-intermediate-registry to localhost https://review.opendev.org/c/opendev/base-jobs/+/878484 | 20:42 |
| clarkb | ok I havne't heard any opposition to removing the old giteas. I'm going to approve the change to stop replication now | 21:01 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/877298 is on its way now. I'll trigger full rereplication after that takes effect to mitigate any lost events | 21:04 |
| * fungi loudly unopposes | 21:12 | |
| opendevreview | Merged opendev/system-config master: Remove gitea01-04 from Gerrit replication https://review.opendev.org/c/opendev/system-config/+/877298 | 21:13 |
| clarkb | infra-root I've finally gotten around to looking into infra-project-manage-projects to see if I can determine why it is ok to have multiple projects.yaml updates in a row after renaming projects and I simply don't get it. | 21:45 |
| clarkb | The job has openstack/project-config as a required project which causes the merged change to be checked out https://opendev.org/opendev/system-config/src/branch/master/zuul.d/infra-prod.yaml#L89-L104 | 21:46 |
| fungi | the checkout ends up having the equivalent of the branch tip, right? so subsequent runs are a no-op | 21:46 |
| fungi | since this is happening post-merge | 21:46 |
| clarkb | I don't think so? | 21:47 |
| fungi | hmm... | 21:47 |
| clarkb | https://opendev.org/opendev/system-config/src/branch/master/playbooks/manage-projects.yaml#L29-L31 is run by the job to copy project-config from zuul homedir (as setup by the required projects) to review | 21:47 |
| fungi | i thought a zuul checkout always merged the change in question to the branch tip (which post-merge is an empty diff) | 21:47 |
| clarkb | fungi: oh I see what you are saying | 21:48 |
| clarkb | that I'm not sure of. But I guess the recent cherry pick support fixes in zuul would agree with that | 21:49 |
| fungi | i was pretty sure using the zuul checkout there was just an easy way to make sure the repo was refreshed | 21:49 |
| clarkb | that role run by the job above does explicitly check out master for system-config but only for periodic/hourly jobs https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/infra-prod/setup-src.yaml#L37-L53 | 21:49 |
| clarkb | fungi: but I think are are saying that if we merge A, B, and C in that order each with a different project rename in them. That they will all effectively run with the content of C because when zuul merges A to tip of master its already got A B C in it and it isn't running without B and C? | 21:51 |
| clarkb | *but I think what you are saying | 21:52 |
| clarkb | for some reason I really thought the order was preserved | 21:53 |
| fungi | yes | 21:53 |
| fungi | that's what i was assuming happened, at least, but we probably need to inspect the history of the checkout for the A build to see if B and C appear | 21:54 |
| fungi | in order to progress from assumption to fact | 21:54 |
| fungi | or just look at A's manage projects log to see if it took the expected actions from B and C | 21:55 |
| fungi | and whether the builds for B and C were ultimately no-op | 21:55 |
| clarkb | oh sorry the base-jobs role I linked does updates project-config form master too but only in the hourly pipelines | 21:57 |
| clarkb | s/hourly/periodic and hourly/ | 21:57 |
| clarkb | This code did change since the lsat time we did renames but it should be a pretty direct port of what we had before | 21:58 |
| clarkb | said another way it was always just periodic and hourly pipelines that got the master update explicitly | 21:58 |
| clarkb | fungi: but I think you may be on to something. Deploy is triggered by change-merged not ref-updated. change-merged merges the change again and it would noop? | 22:00 |
| clarkb | corvus: ^ is that your understanding? I'll try to confirm this (maybe via zuul tests?) before we need to decide if we squash all the project rename chnages into a single change | 22:00 |
| clarkb | worst case I think that is our safety valve and it isn't a difficult one | 22:01 |
| clarkb | I just like to understand what is going on | 22:01 |
| corvus | got a build for me to look at? | 22:02 |
| clarkb | corvus: not off the top of my head, but let me dig around | 22:03 |
| corvus | clarkb: i think i have enough context clues from what you and fungi wrote, so don't worry, i'll try to answer | 22:06 |
| clarkb | corvus: https://zuul.opendev.org/t/openstack/build/44aa9576778840919d321e10ce258cde then https://zuul.opendev.org/t/openstack/build/3da70eb3e1294a4fa11b356f3a54c606 look like they ran back to back for subseuqent project creations | 22:06 |
| corvus | ack ok | 22:06 |
| corvus | i think the answer is yes, that's all correct. and the rule of thumb is use deploy for things where the change matters and not the repo content; and use post when the repo content matters but not the change. | 22:08 |
| opendevreview | Merged opendev/system-config master: Fix typos in doc/source/open-infrastructure.rst https://review.opendev.org/c/opendev/system-config/+/878435 | 22:08 |
| corvus | er i should say "change-merged" rather than "deploy" and "ref-updated" rather than "post" but you get the idea | 22:08 |
| clarkb | corvus: ok so we don't actually need to worry about change A recreating the project that B renames becuase the jobs effectively run as if A abd B are both present in the repo in a change-merged pipeline | 22:09 |
| clarkb | we do need to merge them very quickly together though? | 22:10 |
| clarkb | replication config has updated. I'll trigger global replication now | 22:10 |
| corvus | i think the behavior is probably worth some serious thought | 22:10 |
| clarkb | corvus: I agree and it is why I wanted to figure it out before we do the actual renames | 22:11 |
| clarkb | I'll continue to noodle on it and read into what zuul does and read those logs. Thought probably more tomorrow now that I'm understanding this will take a bit of digging | 22:13 |
| corvus | clarkb: i think the deploy pipeline/job structure assumes idempotency | 22:13 |
| corvus | so if the rename playbook doesn't, could be tricky | 22:13 |
| clarkb | corvus: right we typically rename everything in one go outside of zuul to shorten the outage fo rgerrit. Then we merge the changes that update the names of the repos after gerrit back up again to reflect the new reality with the expectation that things noop | 22:14 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Check and grow the GPT structure to the end of the volume https://review.opendev.org/c/openstack/diskimage-builder/+/827558 | 22:14 |
| clarkb | My understanding is that this could be problematic for the seuqenced merges but then we've done them in the past (unfrotauntely years ago) long enough ago that I don't recall exactly what happened but that we didn't think we ended up with new empty repos with the wrong names | 22:15 |
| clarkb | fuzzy memory says that maybe the jobs for A failed due to the inconsistent state before running manage-projects so that it only actually runs the job on the last change which is consistent | 22:16 |
| clarkb | but I don't want to rely on that as an assumption. I'll continue to look into this | 22:16 |
| clarkb | now that I've written that out I want to say that is precisely what happened | 22:17 |
| clarkb | I seem to recall the jobs were aborted due to an earlier failure of some sort due to the inconsistency | 22:17 |
| clarkb | heh ya those examples I linked had only manage-projects in their buildset | 22:18 |
| clarkb | as far as workarounds we can squash the changes or we can disable zuul infra-prod interactions until after those changes are processed. Then hourly will do the correct thing checking out master | 22:21 |
| opendevreview | Merged zuul/zuul-jobs master: Add docker buildx multiarch support to container roleset https://review.opendev.org/c/zuul/zuul-jobs/+/878246 | 22:23 |
| opendevreview | Merged zuul/zuul-jobs master: buildx: remove experimental flags https://review.opendev.org/c/zuul/zuul-jobs/+/878293 | 22:23 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: build-container-image: directly push with buildx https://review.opendev.org/c/zuul/zuul-jobs/+/878487 | 22:26 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Handle zk_image.repository not being defined in container roles https://review.opendev.org/c/zuul/zuul-jobs/+/878488 | 22:29 |
| clarkb | replication is complete | 22:29 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Handle zk_image.repository not being defined in container roles https://review.opendev.org/c/zuul/zuul-jobs/+/878488 | 22:32 |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Handle credential repository not being defined in container roles https://review.opendev.org/c/zuul/zuul-jobs/+/878488 | 22:33 |
| *** promethe- is now known as prometheanfire | 23:08 | |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: push-to-intermediate-registry: look for container_images variable https://review.opendev.org/c/zuul/zuul-jobs/+/878492 | 23:33 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: push-to-intermediate-registry: look for container_images variable https://review.opendev.org/c/zuul/zuul-jobs/+/878492 | 23:35 |
| opendevreview | Merged zuul/zuul-jobs master: Fix container-image pre playbook container_command default https://review.opendev.org/c/zuul/zuul-jobs/+/878483 | 23:37 |
| opendevreview | Merged zuul/zuul-jobs master: Handle credential repository not being defined in container roles https://review.opendev.org/c/zuul/zuul-jobs/+/878488 | 23:38 |
| corvus | infra-root: i updated emilienm's script: https://paste.opendev.org/show/bYuiVPu62mWDqjaAw93m/ | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!