| ianw | ok, so that did fix the build @ https://nb04.opendev.org/rockylinux-9-arm64-0000000426.log | 00:02 |
|---|---|---|
| ianw | it's copying and making images now. that will be good as the new linaro cloud didn't have rocky images because we shutdown nb03 | 00:02 |
| opendevreview | Merged zuul/zuul-jobs master: linters-requirements : update Ansible to 2.12, ansible-lint <6.12.0 https://review.opendev.org/c/zuul/zuul-jobs/+/872371 | 00:34 |
| clarkb | fungi: fwiw root alias on new lists server seems to work now as i'm getting some emails about it | 00:38 |
| fungi | coolies | 00:38 |
| opendevreview | Merged zuul/zuul-jobs master: ansible-lint: Ignore some command-instead-of-module warnings https://review.opendev.org/c/zuul/zuul-jobs/+/872489 | 00:44 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: zuul-jobs-test-registry-docker-* : update to jammy nodes https://review.opendev.org/c/zuul/zuul-jobs/+/872365 | 00:53 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: fix a bunch of command-instead-of-shell errors https://review.opendev.org/c/zuul/zuul-jobs/+/872490 | 00:53 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: add names to blocks/includes, etc. https://review.opendev.org/c/zuul/zuul-jobs/+/872491 | 00:53 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: ignore use of mkdir https://review.opendev.org/c/zuul/zuul-jobs/+/872492 | 00:53 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: use pipefail https://review.opendev.org/c/zuul/zuul-jobs/+/872493 | 00:53 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: ignore latest git pull https://review.opendev.org/c/zuul/zuul-jobs/+/872494 | 00:53 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: uncap https://review.opendev.org/c/zuul/zuul-jobs/+/872495 | 00:54 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: build-docker-image: fix change prefix https://review.opendev.org/c/zuul/zuul-jobs/+/872258 | 00:54 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: container-roles-jobs: Update tests to jammy nodes https://review.opendev.org/c/zuul/zuul-jobs/+/872375 | 00:54 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: zuul-jobs-test-registry-docker-* : update to jammy nodes https://review.opendev.org/c/zuul/zuul-jobs/+/872365 | 03:25 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ensure-skopeo: add install from upstream option https://review.opendev.org/c/zuul/zuul-jobs/+/872617 | 03:25 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ensure-skopeo: add install from upstream option https://review.opendev.org/c/zuul/zuul-jobs/+/872617 | 03:37 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: zuul-jobs-test-registry-docker-* : update to jammy nodes https://review.opendev.org/c/zuul/zuul-jobs/+/872365 | 03:37 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ensure-skopeo: add install from upstream option https://review.opendev.org/c/zuul/zuul-jobs/+/872617 | 03:41 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: zuul-jobs-test-registry-docker-* : update to jammy nodes https://review.opendev.org/c/zuul/zuul-jobs/+/872365 | 03:41 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ensure-skopeo: add install from upstream option https://review.opendev.org/c/zuul/zuul-jobs/+/872617 | 03:44 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: zuul-jobs-test-registry-docker-* : update to jammy nodes https://review.opendev.org/c/zuul/zuul-jobs/+/872365 | 03:44 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ensure-skopeo: add install from upstream option https://review.opendev.org/c/zuul/zuul-jobs/+/872617 | 04:06 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: zuul-jobs-test-registry-docker-* : update to jammy nodes https://review.opendev.org/c/zuul/zuul-jobs/+/872365 | 04:06 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: fix a bunch of command-instead-of-shell errors https://review.opendev.org/c/zuul/zuul-jobs/+/872490 | 04:06 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: add names to blocks/includes, etc. https://review.opendev.org/c/zuul/zuul-jobs/+/872491 | 04:06 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: ignore use of mkdir https://review.opendev.org/c/zuul/zuul-jobs/+/872492 | 04:06 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: use pipefail https://review.opendev.org/c/zuul/zuul-jobs/+/872493 | 04:06 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: ignore latest git pull https://review.opendev.org/c/zuul/zuul-jobs/+/872494 | 04:06 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: uncap https://review.opendev.org/c/zuul/zuul-jobs/+/872495 | 04:06 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: build-docker-image: fix change prefix https://review.opendev.org/c/zuul/zuul-jobs/+/872258 | 04:06 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: container-roles-jobs: Update tests to jammy nodes https://review.opendev.org/c/zuul/zuul-jobs/+/872375 | 04:06 |
| *** yadnesh|away is now known as yadnesh | 04:12 | |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: use pipefail https://review.opendev.org/c/zuul/zuul-jobs/+/872493 | 04:37 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: ignore latest git pull https://review.opendev.org/c/zuul/zuul-jobs/+/872494 | 04:37 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: uncap https://review.opendev.org/c/zuul/zuul-jobs/+/872495 | 04:37 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: build-docker-image: fix change prefix https://review.opendev.org/c/zuul/zuul-jobs/+/872258 | 04:37 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: container-roles-jobs: Update tests to jammy nodes https://review.opendev.org/c/zuul/zuul-jobs/+/872375 | 04:37 |
| *** ysandeep is now known as ysandeep|ruck | 06:08 | |
| *** jpena|off is now known as jpena | 08:44 | |
| *** ysandeep|ruck is now known as ysandeep|ruck|afk | 10:52 | |
| *** rlandy|out is now known as rlandy | 11:06 | |
| *** dviroel|rout is now known as dviroel | 11:19 | |
| *** ysandeep|ruck|afk is now known as ysandeep|ruck | 11:40 | |
| ysandeep|ruck | folks o/ we are observing jobs failure with retry retry_limit(no logs): https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-content-provider | 12:15 |
| ysandeep|ruck | could someone please take a look on failures ^^ | 12:15 |
| ysandeep|ruck | more example: https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-content-provider-zed | 12:23 |
| fungi | ysandeep|ruck: does the job use docker, by any chance? docker-ce 23.0.0 was released yesterday and broke a few things | 12:59 |
| fungi | though looks like that job started failing today between 09:17:29 and 10:47:22 utc | 13:00 |
| ysandeep|ruck | fungi, no we don't use docker, as this is c9 stream node we have podman instead: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b4c/872631/1/check/tripleo-ci-centos-9-content-provider/b4c6e09/logs/undercloud/var/log/extra/package-list-installed.txt | 13:02 |
| ysandeep|ruck | podman.x86_64 2:4.3.1-3.el9 @quickstart-centos-appstreams | 13:02 |
| ysandeep|ruck | we do create a docker_registry container in job but that's at very late stage: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b4c/872631/1/check/tripleo-ci-centos-9-content-provider/b4c6e09/logs/undercloud/var/log/extra/podman/podman_allinfo.log | 13:05 |
| ysandeep|ruck | In the failing job, job failed within 15s | 13:06 |
| ysandeep|ruck | so doesn't seems related to docker to me | 13:06 |
| fungi | ysandeep|ruck: https://paste.opendev.org/show/bYWaSrxHSlZzbUKke95y/ | 13:08 |
| fungi | looks like somebody broke ansible-collections/community.general | 13:08 |
| fungi | the patchback/backports/stable-6/a64df658c5eb09621feffb2f6e4c6b0b14501c59/pr-5928 branch specifically, i think | 13:13 |
| *** yadnesh is now known as yadnesh|away | 13:15 | |
| fungi | seems maybe that got deleted but was referenced, could still be in the executor's cache, checking... | 13:17 |
| ysandeep|ruck | yes, trying to understand how that patch break us, the changes looks unrelated :D | 13:18 |
| ysandeep|ruck | thanks for digging in | 13:18 |
| fungi | https://github.com/ansible-collections/community.general/pull/5928 seems to have merged around the time the errors started, but the error is specifically that the ref for that branch can't be found any more. it may be the ansible community's workflow creating pull request branches inside the repo that's causing zuul to choke | 13:19 |
| fungi | because zuul wants to be able to provide every branch of a repository it's told to prepare | 13:20 |
| fungi | for some reason it thinks that branch is supposed to exist | 13:20 |
| fungi | but cloning the repo from github myself, it's definitely not a branch any longer, the executor just doesn't seem to realize that yet | 13:21 |
| fungi | the clone of that repo on the executor where that error was logged doesn't seem to have any pr branches now though | 13:25 |
| fungi | maybe the problem has resolved itself and there was just a protracted inconsistency between what github was reporting for branches and what was actually backed by refs? | 13:25 |
| ysandeep|ruck | fungi: thanks for explanation, I will keep an eye on new jobs to see if all clear now | 13:26 |
| fungi | ysandeep|ruck: the last failure for the job you mentioned was a few hours ago, so yeah it could have been transient | 13:27 |
| ysandeep|ruck | fungi++ thanks! | 13:28 |
| ysandeep|ruck | marios, ^^ fyi.. | 13:28 |
| marios | thank you ysandeep|ruck fungi | 13:47 |
| *** dasm|off is now known as dasm | 13:52 | |
| *** dasm is now known as dasm|rover | 13:52 | |
| Clark[m] | fungi: ysandeep|ruck that is a known issue with zuul, GitHub and ansibles branch deletion workflow. | 14:22 |
| ysandeep|ruck | Clark[m], ack thanks! | 14:23 |
| Clark[m] | In the time between listing branches and trying to configure branches the branch is deleted. Typically this isn't an issue with GitHub because you make the branch in a fork. But Ansible creates the temporary branches on the local repo instead sometimes | 14:23 |
| *** ysandeep|ruck is now known as ysandeep | 15:14 | |
| *** dviroel is now known as dviroel|lunch | 15:14 | |
| *** ysandeep is now known as ysandeep|out | 15:15 | |
| fungi | the writing is on the wall for `sudo pip install` in debian and ubuntu: https://salsa.debian.org/cpython-team/python3/-/blob/master/debian/README.venv | 15:35 |
| fungi | https://discuss.python.org/t/10302/69 | 15:36 |
| clarkb | fungi: is that going to be a bookworm change? | 15:37 |
| fungi | "This will carry into Ubuntu 23.04 (“lunar”) too." | 15:37 |
| fungi | yes | 15:37 |
| fungi | so we'll hit it in debian sooner than ubuntu | 15:38 |
| clarkb | that will affect our docker image system but only minimally as we should be able to shift it to a /venv/ venv or whatever | 15:38 |
| clarkb | really the bigger issue is going to be devstack | 15:38 |
| clarkb | you should give them a heads up | 15:38 |
| fungi | it'll be a much bigger hit to devstack circa ubuntu 22.04 lts next year, yes | 15:39 |
| fungi | er, 24.04 lts i mean | 15:39 |
| fungi | i planned to | 15:39 |
| clarkb | for opendev our legacy stuff running on older LTSs may sudo pip install but over time more and more stuff is moving into our python base image setup | 15:39 |
| clarkb | so if we can generically make a venv at say /venv and then add /venv/bin to $PATH we'll be largely sorted there I suspect | 15:39 |
| clarkb | for things like devstack well I tried to move everything into venvs when pip first started refusing to uninstall things that were not recorded properly and the response was a pretty loud meh | 15:40 |
| clarkb | for our dib builds we already use venvs, but dib itself may eed changes to accomodate its own bootstrapping maybe? | 15:41 |
| clarkb | glean for example is in a venv in the image | 15:42 |
| fungi | yeah, i think we've seen this coming from miles away, so our base stuff is already in pretty good shape | 15:43 |
| clarkb | and having the python base images means we should only need to solve this once generically for everything using that which is a huge bonus | 15:44 |
| fungi | yes | 15:46 |
| fungi | also as noted in those discussions, --system-site-packages should still work for venvs that need to consume python libs from distro packages (if we still have any of those to worry about) | 15:47 |
| clarkb | I don't think we do. The recent update to jeepyb to list the launchpad lib as a pypi package was the last case I knew of where we might need that | 15:48 |
| clarkb | so we literally fixed that this week? :) | 15:48 |
| clarkb | fungi: we should add a note to the meeting agenda about it too. That way we can double check we aren't forgetting anything that could be a problem. And maybe start the conversion process early | 15:52 |
| elodilles | hi, i'm planning to run the EOL branch cleanup script to delete some EOL'd kolla branches | 16:06 |
| clarkb | sounds good | 16:06 |
| *** dviroel|lunch is now known as dviroel | 16:14 | |
| opendevreview | Merged openstack/project-config master: Step 1: End Project Gating, Retire xstatic-font-awesome https://review.opendev.org/c/openstack/project-config/+/871703 | 16:21 |
| elodilles | done, these were deleted: https://paste.opendev.org/show/bviw6l2VW0j6IBIQ6Y2X/ | 16:22 |
| fungi | clarkb: in case it isn't obvious, the reason 871703 doesn't remove the template with the github replication job is that we want the retirement change that gets merged in the repo to also get mirrored to gh | 16:51 |
| clarkb | fungi: yup, its more that the template is poorly named (imo) and makes it seem like it is likely to do more | 16:52 |
| clarkb | I was worried it would run check and gate jobs preventing the retirement change from landing | 16:52 |
| fungi | the retirement instructions for openstack were updated to cover that after we discovered the chicken-and-egg problem with removing jobs so that the retirement change could merge | 16:52 |
| fungi | and yes, the idea with that template is that it might eventually include other jobs which are only allowed/required for official openstack projects, but i agree the name is currently misleading with only one job in it | 16:53 |
| mnaser | i'm having a hard time pushing a review | 17:07 |
| mnaser | looks like it's failing to scp the commit hook, getting: "subsystem request failed on channel 0", "scp: Connection closed" | 17:07 |
| mnaser | ssh mnaser@review.opendev.org -p29418 welcomes me just fine | 17:08 |
| clarkb | mnaser: I think you need a newer git-review? iirc scp is being deprecated by ssh and you need to send extra flags or something to opt into making it work on newer sstems and newer git-review addresses that | 17:08 |
| mnaser | ahhhh ok | 17:08 |
| clarkb | fungi: ^ that sound familiar? | 17:09 |
| clarkb | mnaser: I think what new ssh does is uses sftp in some compat mode but that doesn't work with the java scp implementation so you have to force actual scp | 17:09 |
| mnaser | yeah you're right, bumping git review worked | 17:09 |
| mnaser | i was under the impression it was possible an infra thing since the manual scp didnt work | 17:09 |
| clarkb | we should probably look into switching ot an http GET of the hook | 17:11 |
| clarkb | I think the thing preventing that is if we have an ssh remote we don't necessarily know where the https server is listening | 17:11 |
| fungi | mnaser: clarkb: yes, the current git-review release (and maybe the one prior to it) include the switch from the scp subsystem to sftp | 17:13 |
| fungi | or, rather, include the workaround for it | 17:14 |
| fungi | basically gerrit's mina-sshd still implements the old scp protocol and openssh (via git) wants it to support sftp, so git-review passes an ssh option to tell it to use the old scp protocol anyway | 17:15 |
| fungi | we will probably eventually need mina-sshd to support sftp or, as you note, switch to https for hook retrieval (or serve the hook in some other way) if openssh eventually drops support for the scp protocol entirely, but i don't think they've announced plans to do that | 17:16 |
| fungi | another option we discussed was to ship a copy of the commit hook in git-review itself | 17:17 |
| *** jpena is now known as jpena|off | 17:35 | |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: container-roles-jobs: Update tests to jammy nodes https://review.opendev.org/c/zuul/zuul-jobs/+/872375 | 18:54 |
| *** dviroel is now known as dviroel|pto | 21:03 | |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Fix ansible-lint no-jinja-when occurence https://review.opendev.org/c/zuul/zuul-jobs/+/872698 | 21:05 |
| *** dasm|rover is now known as dasm|afk | 21:36 | |
| clarkb | fungi: it occured to me that we don't use the debian python and pip on those docker images so we may not need to do anything for our docker images | 22:53 |
| fungi | good point | 22:57 |
| clarkb | fungi: also comment son https://review.opendev.org/c/opendev/system-config/+/869091 | 22:59 |
| fungi | thanks! | 23:05 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!