| clarkb | I guess it isn't an error from apt-key to add an expired key | 00:00 |
|---|---|---|
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Avoid tox-bindep in test-tox job https://review.opendev.org/c/zuul/zuul-jobs/+/866945 | 00:00 |
| corvus | more allowlist fallout | 00:00 |
| ianw | https://kubic.opensuse.org/blog/2022-06-10-kubic-retired/ <- i don't know if i knew this | 00:02 |
| fungi | clarkb: i suppose one workaround to the --showconfig --verbose problem is to run once without --verbose and parse that, then run it again with --verbose and log it | 00:06 |
| clarkb | fungi: in this case we had been trying to consistently run all the tox commands using the same args so that the right envs are selected and so on. I think for showconfig it is ok for us to drop the verbose flag which is what I've done beacuse we're not interested in anything but the config | 00:07 |
| clarkb | its possible that we'll find scenarios where this isn't true and in that case we can possibly filter out the -vv from extra_tox_args and pass the rest in | 00:07 |
| clarkb | but from a UX perspective I think a command whose purpose is to emit a config "file" should do that in a valid manner | 00:07 |
| clarkb | but I'm not going to argue it further since apparently I'm complaining (I was trying to help and offered ideas to solutions and may have even tried to write a patch if one or another was selected) | 00:08 |
| fungi | i have a feeling the maintainer may be more short than usual due to handling an influx of new v4-related issues after asking people to test prereleases | 00:10 |
| fungi | but yes, not worth arguing that one further, i agree | 00:11 |
| clarkb | this might be a situation where a new tool would've been better | 00:12 |
| fungi | yes, a separate tox4 package or something | 00:13 |
| clarkb | if you've decided that you need a complete rewrite and backward compatibility is not feasible then tox2 is a better choice than breaking everyone | 00:13 |
| clarkb | then tell everyone to switch to that or take on maintenance of the old code base | 00:13 |
| fungi | xot | 00:13 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: drop kubic repos https://review.opendev.org/c/zuul/zuul-jobs/+/866947 | 00:14 |
| clarkb | see also nose | 00:14 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: drop kubic repos on 22.04 https://review.opendev.org/c/zuul/zuul-jobs/+/866947 | 00:15 |
| fungi | ROOT: 177 E HandledError| could not recognize config file test-playbooks/python [tox/run.py:22] | 00:16 |
| clarkb | fungi: what is the context of that? | 00:17 |
| fungi | that's what 866945 is doing now | 00:17 |
| fungi | https://zuul.opendev.org/t/zuul/build/7fd51570fe2e47caba067af38ed4e7f7 | 00:17 |
| clarkb | https://opendev.org/zuul/zuul-jobs/src/branch/master/test-playbooks/python/tox.yaml#L36 we set that flag to chnge the context of whee we are running and that dir has a tox.ini in it | 00:18 |
| clarkb | we might need to set tox_config_file too? | 00:19 |
| fungi | oh, maybe it no longer automatically looks in there for the config with v4 | 00:21 |
| clarkb | I see it | 00:21 |
| clarkb | what does tox_envlist: '%s' mean? | 00:22 |
| clarkb | tox -e%s ? | 00:22 |
| clarkb | oh its an actual env why would it have that name... | 00:24 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Flip ensure-tox to tox v4 by default https://review.opendev.org/c/zuul/zuul-jobs/+/866943 | 00:25 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Avoid tox-bindep in test-tox job https://review.opendev.org/c/zuul/zuul-jobs/+/866945 | 00:25 |
| clarkb | I think that will fix it | 00:25 |
| ianw | i guess i was mistaken, in that 22.04 has podman, but not cri-o. it looks like ubuntu are on the microk8s train, with containerd or kata as the runtime, presumably why there's nothing about non-external cri-o packages | 00:27 |
| clarkb | ianw: this is using minikube right? | 00:27 |
| clarkb | surely they document how to run it on ubuntu? maybe the answer is "run this vm on your ubuntu" | 00:28 |
| ianw | yes, basically the way to install it on ubuntu is to use the suse distribution of cri-o | 00:28 |
| ianw | "it" == minikube | 00:28 |
| ianw | probably the more "native" thing to do for ensure-kubernetes on ubuntu would be to install their mirok8s snap | 00:29 |
| clarkb | but minikube is an upstream k8s project I thought. Surely they would make it work somehow? or maybe you're thinking they tie into microk8s? | 00:30 |
| ianw | aiui (which may be wrong) -- minikube =~ microk8s. they both talk to container runtimes underneath -- microk8s to containerd, minikube to docker/cri-o/containerd | 00:32 |
| clarkb | right but they come from different places. I'm just surprised if the upstream project stopped making minikube possible on ubuntu is all | 00:32 |
| clarkb | like openstack dropping devstack because canonical has their deployment system for openstack | 00:33 |
| ianw | well they haven't -- but you have to get cri-o from the external suse repositories | 00:33 |
| ianw | (the ones where the signing key has now expired) | 00:33 |
| clarkb | I see | 00:34 |
| ianw | this is where i got a bit confused -- https://minikube.sigs.k8s.io/docs/drivers/podman/ | 00:36 |
| ianw | you can use podman (included in distro now) -- but *that* then talks to cri-o | 00:37 |
| ianw | we already test this with the docker backend -- zuul-jobs-test-registry-buildset-registry-k8s-docker | 00:39 |
| clarkb | can we use docker instead? we have to fix the shim right? | 00:41 |
| ianw | clarkb: heh, yes my thought too :) -> https://review.opendev.org/c/zuul/nodepool/+/866948 | 00:48 |
| clarkb | fungi: the issue appears related to running out of the wrong dir? | 00:51 |
| corvus | clarkb: adding tox.ini didn't fix it, and the error message makes me think it perhaps isn't necessary ("is neither file or directory") | 00:51 |
| clarkb | corvus: ya at first I thought maybe it is breaking on %s but that isn't it. If I cd into that dir and run against tox.ini renamed to foo.ini it does work | 00:51 |
| clarkb | so I'm making that change now | 00:51 |
| clarkb | we should continue to test the use of -c. I suspect the issue is that the top level dir of that repo also has a tox.ini and it gets mad? | 00:52 |
| clarkb | that might also be a bug | 00:52 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Flip ensure-tox to tox v4 by default https://review.opendev.org/c/zuul/zuul-jobs/+/866943 | 00:53 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Avoid tox-bindep in test-tox job https://review.opendev.org/c/zuul/zuul-jobs/+/866945 | 00:53 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: don't install cri-o with docker backend https://review.opendev.org/c/zuul/zuul-jobs/+/866950 | 01:39 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Flip ensure-tox to tox v4 by default https://review.opendev.org/c/zuul/zuul-jobs/+/866943 | 01:49 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Avoid tox-bindep in test-tox job https://review.opendev.org/c/zuul/zuul-jobs/+/866945 | 01:49 |
| clarkb | I think what is going on is that if you have a tox.ini in the current dir then tox breaks when you use -c | 01:50 |
| clarkb | doesn't matter where the -c points to | 01:50 |
| clarkb | I think the above change works around that and also allows tox to find the tox.ini by default in the test dir for the earlier checks | 01:50 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: don't install cri-o with docker backend https://review.opendev.org/c/zuul/zuul-jobs/+/866950 | 01:55 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: don't install cri-o with docker backend https://review.opendev.org/c/zuul/zuul-jobs/+/866950 | 02:05 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: don't install cri-o with docker backend https://review.opendev.org/c/zuul/zuul-jobs/+/866950 | 02:25 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: don't install cri-o with docker backend https://review.opendev.org/c/zuul/zuul-jobs/+/866950 | 02:51 |
| *** yadnesh|away is now known as yadnesh | 03:41 | |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure-kubernetes-microk8s https://review.opendev.org/c/zuul/zuul-jobs/+/866953 | 03:54 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure-kubernetes-microk8s https://review.opendev.org/c/zuul/zuul-jobs/+/866953 | 03:56 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure-kubernetes-microk8s https://review.opendev.org/c/zuul/zuul-jobs/+/866953 | 03:58 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure-kubernetes-microk8s https://review.opendev.org/c/zuul/zuul-jobs/+/866953 | 04:08 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure-kubernetes-microk8s https://review.opendev.org/c/zuul/zuul-jobs/+/866953 | 04:13 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure-kubernetes-microk8s https://review.opendev.org/c/zuul/zuul-jobs/+/866953 | 04:19 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure-kubernetes-microk8s https://review.opendev.org/c/zuul/zuul-jobs/+/866953 | 05:07 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure-kubernetes-microk8s https://review.opendev.org/c/zuul/zuul-jobs/+/866953 | 05:20 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure-kubernetes-microk8s https://review.opendev.org/c/zuul/zuul-jobs/+/866953 | 05:33 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure-kubernetes-microk8s https://review.opendev.org/c/zuul/zuul-jobs/+/866953 | 06:02 |
| *** marios is now known as marios|ruck | 06:05 | |
| *** ysandeep is now known as ysandeep|afk | 06:20 | |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure-kubernetes-microk8s https://review.opendev.org/c/zuul/zuul-jobs/+/866953 | 07:36 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: [wip] ensure-kubernetes-microk8s https://review.opendev.org/c/zuul/zuul-jobs/+/866953 | 07:48 |
| *** ysandeep is now known as ysandeep|lunch | 09:03 | |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: add microk8s support https://review.opendev.org/c/zuul/zuul-jobs/+/866953 | 09:38 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: add microk8s support https://review.opendev.org/c/zuul/zuul-jobs/+/866953 | 09:44 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: add microk8s support https://review.opendev.org/c/zuul/zuul-jobs/+/866953 | 09:52 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ensure-kubernetes: add microk8s support https://review.opendev.org/c/zuul/zuul-jobs/+/866953 | 10:13 |
| *** ysandeep|lunch is now known as ysandeep | 10:31 | |
| *** dviroel|afk is now known as dviroel | 10:51 | |
| *** chandankumar is now known as chkumar|afk | 11:00 | |
| *** rlandy|out is now known as rlandy|rover | 11:18 | |
| gksk | Hi Team | 11:52 |
| gksk | I am facing one issue while connecting to nodepool vms not sure zuul user is being created or not and i am unable to ssh with ZUUL_USER_SSH_PUBLIC_KEY to nodepool vms | 11:55 |
| gksk | can any one help me to resolve this issue | 11:55 |
| gksk | I have added public key to ZUUL_USER_SSH_PUBLIC_KEY attribute in nodepool.yaml | 11:55 |
| *** pojadhav is now known as pojadhav|afk | 12:00 | |
| *** chkumar|afk is now known as chandankumar | 12:03 | |
| fungi | gk you need to create the user and add the executor's ssh key as authorized when building your vm images. we do it with a diskimage-builder element like https://opendev.org/openstack/project-config/src/branch/master/nodepool/elements/zuul-worker | 12:48 |
| *** dasm|off is now known as dasm | 13:00 | |
| *** ysandeep is now known as ysandeep|afk | 13:11 | |
| *** pojadhav|afk is now known as pojadhav | 13:23 | |
| gksk | @fungi I got it Thank you | 13:45 |
| *** ysandeep|afk is now known as ysandeep | 14:03 | |
| *** pojadhav is now known as pojadhav|dinner | 15:06 | |
| *** ysandeep is now known as ysandeep|out | 15:25 | |
| *** ykarel_ is now known as ykarel|away | 15:47 | |
| fungi | clarkb: frickler: i guess the resolution to the # problem was to deem it notabug: "The ``#`` character now always acts as comment within ``tox.ini`` or ``setup.cfg`` tox configuration file. Where you need to pass on a ``#`` character you will need to escape it in form of ``\#`` so tox does not handle everything right of the ``#`` character as a comment." https://github.com/tox-dev/tox/pull/2621 | 16:07 |
| *** pojadhav|dinner is now known as pojadhav | 16:10 | |
| *** dviroel is now known as dviroel|lunch | 16:10 | |
| fungi | seems like any backward-incompatible behavior changes not caught and called out in prerelease testing are just going to be deemed "working as intended" | 16:15 |
| clarkb | fungi: yup in the original issue frickler filed they basically said not a bug so I updated my zuul-jobs stack to add \s | 16:18 |
| *** marios|ruck is now known as marios|out | 16:39 | |
| *** dviroel|lunch is now known as dviroel | 17:06 | |
| opendevreview | Merged zuul/zuul-jobs master: Move tox logs into env specific dirs https://review.opendev.org/c/zuul/zuul-jobs/+/866939 | 17:07 |
| *** yadnesh is now known as yadnesh|away | 17:16 | |
| opendevreview | Merged zuul/zuul-jobs master: Update zuul-jobs to handle tox3 and tox4 https://review.opendev.org/c/zuul/zuul-jobs/+/866935 | 17:23 |
| opendevreview | Sorin Sbârnea proposed zuul/zuul-jobs master: Fixed schema errors reported by ansible-lint https://review.opendev.org/c/zuul/zuul-jobs/+/867037 | 17:50 |
| *** rlandy|rover is now known as rlandy|dr_appt | 18:39 | |
| frickler | well in my test the \# does work as well with older tox, so I think that that's ok-ish | 18:47 |
| clarkb | frickler: ya it seems to work in the chagnes I made too | 18:47 |
| fungi | oh, good. i was worried it had to change at the same time rather than in advance | 18:55 |
| clarkb | just as a heads up I'm not sure I'll have time to get to manual zuul restarts today (I didn't yesterday either). corvus indicated he thought this wasn't necessary as we have decent test coverage | 19:00 |
| clarkb | I've run into the tox 4 update and nodepool testing and openstacksdk stuff wall | 19:01 |
| fungi | on pending mm3 work, we still have a few patches at topic:mailman3 if any infra-root are up for additional reviews | 19:02 |
| fungi | also has anyone else noticed yet that the lists.zuul-ci.org hyperkitty pages say lists.opendev.org in the top-left corner? | 19:02 |
| clarkb | fungi: neat its a hyperlink to the proper url though | 19:03 |
| clarkb | I'm guessing thats a bug if it can render the link properly but not the text | 19:03 |
| fungi | possible we simply missed a config option somewhere | 19:04 |
| fungi | oh, the mm3 changes have enough votes to approve. i'll just self-approve them | 19:04 |
| opendevreview | Merged opendev/zone-zuul-ci.org master: Restore the default TTL to lists https://review.opendev.org/c/opendev/zone-zuul-ci.org/+/866615 | 19:07 |
| opendevreview | Merged opendev/zone-opendev.org master: Restore the default TTL to lists https://review.opendev.org/c/opendev/zone-opendev.org/+/866607 | 19:07 |
| *** dviroel_ is now known as dviroel | 19:17 | |
| opendevreview | Merged opendev/system-config master: Remove opendev and zuul sites from old mm2 server https://review.opendev.org/c/opendev/system-config/+/866630 | 19:30 |
| *** rlandy|dr_appt is now known as rlandy|rover | 19:55 | |
| *** dviroel is now known as dviroel|afk | 20:20 | |
| *** anbanerj is now known as frenzy_friday | 20:44 | |
| fungi | infra-prod-base failure deploying the lists.openstack.org config cleanup: https://zuul.opendev.org/t/openstack/build/9fb5e8eaa5f449afb3e1c43d26341f9c | 21:01 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Fix nodepool API ops Grafana graphs https://review.opendev.org/c/openstack/project-config/+/867054 | 21:02 |
| clarkb | infra-root ^ I believe that will fix the nodepool api oerations graphs | 21:02 |
| clarkb | fungi: you need to look at the file on bridge or decrypt the one in swift | 21:03 |
| clarkb | the job log itself is unfortunately less useful for infra-prod jobs | 21:03 |
| fungi | TASK [base/server : Install distro specific packages] fatal: [afs02.dfw.openstack.org]: FAILED! => no available installation candidate for yaml-mode | 21:03 |
| fungi | clarkb: yeah, that's where i was headed | 21:03 |
| clarkb | that pacakge seems to exist on many ubuntus maybe apt is sad for some reason/ | 21:04 |
| fungi | i guess that's coming from here: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/base/server/vars/Debian.yaml#L4 | 21:04 |
| clarkb | ya I think we install things like that to make corvus and ianw lives easier (they are emacs users iirc) | 21:05 |
| fungi | `apt search yaml-mode` on afs02.dfw.openstack.org says there's no installation candidate for it | 21:06 |
| clarkb | might need to manually try to install the package heh way ahead of me | 21:06 |
| clarkb | thats weird | 21:06 |
| clarkb | its focal or jammy right? | 21:06 |
| clarkb | both have the package but it is a virtual package | 21:06 |
| fungi | focal | 21:06 |
| fungi | Note, selecting 'elpa-yaml-mode' instead of 'yaml-mode' | 21:06 |
| fungi | elpa-yaml-mode is already the newest version (0.0.14-1). | 21:07 |
| clarkb | has newer ansible's package management broken handling of virtual packages? | 21:07 |
| clarkb | that could be a side effect of updating ansible on rbdige | 21:07 |
| fungi | that seems likely, since `apt install yaml-mode` does nothing (noting that "elpa-yaml-mode is already the newest version") and exits 0 | 21:08 |
| fungi | when did we update ansible on bridge? | 21:09 |
| fungi | infra-prod-base ran successfully as recently as 2022-12-06 15:37:02 | 21:09 |
| fungi | periodic at 2022-12-07 02:54:42 failed | 21:10 |
| fungi | so that's a ~11 hour window | 21:10 |
| clarkb | fungi: it was yesterday I think once ianw's stack of changes to address that landed | 21:10 |
| fungi | looking at https://zuul.opendev.org/t/openstack/builds?job_name=infra-prod-base&project=opendev/system-config | 21:10 |
| clarkb | ianw would have a better idea of timeframe since i was busy looking at tox | 21:11 |
| fungi | yeah, i agree timing seems like it could line up in that case | 21:11 |
| clarkb | its possible we'll have a number of these small issues to sort out with new ansible :( | 21:11 |
| fungi | where "sort out" means stop using ansible modules and replace them with shell commands ansible-lint will no doubt complain about until we stop running it | 21:11 |
| clarkb | or use the actual package name instead? looks like bionic and older have proper yaml-mode and focal and newer use the virtual package | 21:12 |
| clarkb | fungi: its possible this is a bug in the package module and not in the apt module? we could try swapping those out for one another. It is curious that this doesn't error in our own ci though. Maybe when it installs it the first time it works fine? | 21:13 |
| fungi | yeah, possible. for now i'm trying to think how we actually test this | 21:13 |
| clarkb | we do have jobs that run the base playbook. Maybe update the job to run the playbook twice back to back? | 21:13 |
| clarkb | we do that with a few other playbooks (gitea off he top of my head) | 21:14 |
| clarkb | I think you just list the playbook twice in the list of playbooks to have the test harness run | 21:14 |
| fungi | oh, thinking that installing virtual packages is working but upgrading them is broken? | 21:15 |
| fungi | i guess that's a possibility | 21:16 |
| clarkb | yes since the ansible upgrade passed testing that implies it works on a node that doesn't have yaml-mode preinstalled on it and what we are seeing here is a node failing when it is preinstalled | 21:16 |
| clarkb | so I wonder if this is an idempotency bug | 21:16 |
| fungi | reasonable assumption | 21:17 |
| clarkb | I think running the base playbook twice in ci is a good idempotency check either way | 21:19 |
| clarkb | so would be worthwhile writing that change even if it ends up being a dead end for this issue | 21:19 |
| fungi | looks like all the jobs inheriting from system-config-run use playbooks/zuul/run-base.yaml in the run phase | 21:19 |
| fungi | i suppose the system-config-run-base job is a good place to do that? | 21:20 |
| clarkb | fungi: system-config-run-base is the job yup | 21:20 |
| clarkb | it has a variety of nodes to match the various distro releases | 21:20 |
| fungi | oh, though other jobs also inherit from system-config-run-base | 21:20 |
| fungi | but i guess they replace the run phase when doing so | 21:20 |
| clarkb | thats fine they all override the nodeset and playbooks | 21:21 |
| clarkb | yup | 21:21 |
| fungi | system-config-run-base doesn't actually have its own run playbook because that's provided by system-config-run | 21:21 |
| fungi | so should i double it up in system-config-run-base or in system-config-run? | 21:21 |
| fungi | i guess if i do it in system-config-run-base then there's no point to system-config-run having a run phase itself | 21:22 |
| clarkb | I suspect the parent one has it because it needs a run defined? Or will zuul handle that case? | 21:22 |
| fungi | yeah, that's what i expect is happening | 21:22 |
| clarkb | I'd do it in the child (run-base) since that is the actual thing doing it and the parent is just bookkeeping | 21:22 |
| fungi | any preference whether i add two copies of the base playbook to the run phase of system-config-run-base and then clean up the one in system-config-run which will be unused, or add a second one to system-config-run | 21:23 |
| fungi | okay, so the first | 21:23 |
| fungi | system-config-run is already abstract so nothing can actually run it | 21:24 |
| fungi | oh, though system-config-run-containers is parented to system-config-run and doesn't add anything at all so i guess the run phase in system-config-run won't actually be unused | 21:25 |
| ianw | hrm, i feel like this elpa yaml mode is a virtual package has been a bug before ... | 21:25 |
| fungi | also, other jobs inherit from system-config-run-base instead of from system-config-run without setting a run playbook (system-config-run-base-ansible-devel at least) so would inherit the doubling | 21:26 |
| fungi | actually that's the only one, so maybe it's fine | 21:26 |
| ianw | https://github.com/ansible/ansible/issues/76779 | 21:27 |
| clarkb | ianw: do you know if `package` uses `apt` or if it is a separate codebase? | 21:28 |
| clarkb | in this case we appare to use package but maybe we should switch to apt? | 21:28 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Run the base playbook twice to test idempotency https://review.opendev.org/c/opendev/system-config/+/867058 | 21:28 |
| ianw | clarkb: well that fix should be in 2.14 ... | 21:29 |
| ianw | i'm not sure i'm understanding how we missed it if it's broken, because we found it with our -devel job ... | 21:30 |
| fungi | unless we're testing with a different version of ansible than we've got on bridge? | 21:31 |
| ianw | we shouldn't be ... but something feels amiss here. i just have to do school run and can poke more | 21:32 |
| *** dasm is now known as dasm|off | 21:50 | |
| ianw | "no available installation candidate for yaml-mode" does seem different to "E: Version '0.0.14-1' for 'yaml-mode' was not found" | 22:08 |
| ianw | but in the same general region i guess | 22:08 |
| ianw | afs01, afs02, afsdb02, lists.openstack.org & lists.katacontainers.io are the listed failures | 22:09 |
| ianw | they are all focal hosts | 22:10 |
| ianw | /usr/ansible-venv/bin/ansible -vvv afs01.dfw.openstack.org -m package -a 'package=yaml-mode' works ...? | 22:15 |
| jrosser | apt history.log can be useful to find exactly what ansible attempted to do | 22:15 |
| jrosser | the command never ends up being displayed | 22:15 |
| ianw | jrosser: yeah, good idea, unfortunately nothing referring to yaml-mode in there :/ | 22:19 |
| ianw | also a playbook using package: doesn't trigger it against afs01 | 22:20 |
| ianw | i guess i'll try a manual run of the base playbook limited to the host | 22:21 |
| jrosser | as far as i know package: is just a proxy type thing that calls the platform specifit apt:/yum: modules | 22:22 |
| ianw | ... which passes too ... | 22:26 |
| fungi | the idempotency change for the base playbook seems to also work and not hit that error, though it failed during log collection with a connection failure | 22:30 |
| ianw | it's definitely on bridge01 ... there's only one version of ansible installed ... https://zuul.opendev.org/t/openstack/build/08b5591ed79b4edcb1a68759c4fad5ef/console | 22:32 |
| clarkb | catching up. yubioath-desktop stopped working for generating totp keys after updates... ykman oath account code foo still works so it isn't completely broken but I tried the backup device and different usb controllers and different computers... ugh | 22:36 |
| ianw | https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/apt.py#L726 is what we hit | 22:38 |
| clarkb | I am able to u2f with the devices too. Must be an application layer bug (now to stop distracting people with key issues) | 22:39 |
| ianw | haha distractions welcome | 22:39 |
| ianw | this is from the last good run of base -> https://paste.opendev.org/show/bvaYbHtjeKs4cnRa8xI3/ | 22:45 |
| ianw | it seems like the package: install is reporting changed on these hosts. maybe these are our only focal hosts | 22:46 |
| clarkb | ianw: I think others are focal too like all of zuul | 22:46 |
| ianw | yeah ... | 22:47 |
| ianw | # less base.yaml.log.2022-12-06T15:37:55 | grep elpa-yaml-mode | wc -l | 22:47 |
| ianw | 5 | 22:47 |
| *** rlandy|rover is now known as rlandy|out | 22:47 | |
| clarkb | the afs hosts were upgraded in place iirc maybe related to that? | 22:47 |
| ianw | ahhh --- in place upgrades could very definitely be the difference with these 5 | 22:47 |
| ianw | did we do that for lists.opensatck.org? | 22:47 |
| clarkb | ianw: yes lists was inpace too in order to preserve the ip reputation | 22:48 |
| clarkb | we're giving that up with lists01.opendev.org because hte inplace has run out of steam due to kernel problems | 22:48 |
| ianw | ok ... well i think we have a partial smoking gun then | 22:49 |
| ianw | https://paste.opendev.org/show/b3EAvluXhg5tOrHNNMHD/ | 22:50 |
| ianw | it's really like it's running with a old/different version of ansible | 22:59 |
| clarkb | did it maybe cache something on the client side to execute the installation? | 22:59 |
| clarkb | ansible does copy that content over and maybe that is stale somehow? | 22:59 |
| ianw | this is the bootstrap job from the same buildset, where we print the ansible ver -> https://zuul.opendev.org/t/openstack/build/f154fd86f9384c8cb273c7dc8b361ea9/console#2/1/16/bridge01.opendev.org | 22:59 |
| ianw | there's no files in .ansible on afs02 | 23:03 |
| ianw | hrm, i did get it just then with a standalone run -- i put "-f 50" on it like the main run does ... | 23:07 |
| ianw | nope, that didn't make afs01 hit it | 23:10 |
| ianw | alright, i don't know why, but afs01 doesn't hit it on manual testing, but afs02 does | 23:12 |
| ianw | https://paste.opendev.org/show/briE6350LK2YxBAKMFxj/ is the difference | 23:14 |
| ianw | we could just purge and re-install this package and forget about it. but it does annoy me that there must be an ansible issue here ... | 23:14 |
| clarkb | I'm ok with easy workarounds for things like this | 23:15 |
| clarkb | :) | 23:15 |
| ianw | # apt-cache show yaml-mode | 23:17 |
| ianw | N: Can't select versions from package 'yaml-mode' as it is purely virtual | 23:17 |
| ianw | on afs01 | 23:17 |
| ianw | $ apt-cache show yaml-mode | 23:17 |
| ianw | Package: yaml-mode | 23:17 |
| ianw | on afs02 | 23:17 |
| ianw | HOW ARE THEY DIFFERENT!?!?! | 23:17 |
| ianw | rc yaml-mode 0.0.9-2 all emacs mode for YAML files | 23:22 |
| ianw | "removed but it's configuration files are still there" | 23:22 |
| JayF | apt-get purge {package} # gets you to uninstalled | 23:28 |
| JayF | but will remove all the config/data dirs too, so only do that if you're sure that's OK | 23:28 |
| ianw | what's weird is that this doesn't have any conf files | 23:31 |
| ianw | $ dpkg -S yaml-mode | grep '^yaml-mode' | 23:32 |
| ianw | yaml-mode: /etc/emacs/site-start.d/50yaml-mode.el | 23:32 |
| JayF | That doesn't count as one? | 23:32 |
| ianw | so we've removed the package, but it hasn't removed this file ... which is part of the package | 23:32 |
| ianw | https://packages.ubuntu.com/bionic/all/yaml-mode/filelist | 23:32 |
| JayF | yeah, purge will get rid of that | 23:32 |
| JayF | because it's in etc | 23:32 |
| ianw | well apt should have got rid of it, it's one of the package files | 23:33 |
| * JayF is fuzzy on modern ubuntu packaging but that's what he remembers from decades-old debian | 23:33 | |
| ianw | i wonder if this leads us back to some sort of apt bug | 23:33 |
| ianw | when we've done the in-place dist-upgrade, it's done something like remove epel-mode and installed elpa-epel-mode. but it hasn't actually removed one of the epel-mode files. so now it appears as a zombie in dpkg, which is messing up ansible's apt-cache parsing | 23:35 |
| ianw | i think we went trusty->bionic->focal on these | 23:39 |
| ianw | ok, in a trusty container, if i install then remove yaml-mode, it leaves behind | 23:42 |
| ianw | # dpkg -S yaml-mode | 23:42 |
| ianw | yaml-mode: /etc/emacs/site-start.d/50yaml-mode.el | 23:42 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Fix nodepool API ops Grafana graphs https://review.opendev.org/c/openstack/project-config/+/867054 | 23:44 |
| clarkb | I love the screenshots cc ianw found some bugs | 23:44 |
| ianw | clarkb: :) there's instructions there to setup a container which makes fro pretty quick iteration too | 23:44 |
| ianw | https://docs.opendev.org/opendev/system-config/latest/grafana.html#local-development | 23:45 |
| clarkb | ya in this case I juts have to ifddle resource paths | 23:46 |
| ianw | i could chase this all day. i'm 99.99% certain it's to do with yaml-mode installing it's files into /etc and then them hanging around a conffiles | 23:56 |
| ianw | but it's rather pointless ... | 23:56 |
| ianw | i'm just going to log into the 5 hosts and purge the package | 23:56 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!