| corvus | infra-root: i have issued the commands to spawn a full complement of zuul executors on jammy; i'm going to shut down half of them (1-6) gracefully now and attempt to replace them all half at a time this weekend. | 00:24 |
|---|---|---|
| Clark[m] | Ack | 00:27 |
| opendevreview | James E. Blair proposed opendev/zone-opendev.org master: Convert spaces to tabs https://review.opendev.org/c/opendev/zone-opendev.org/+/885166 | 00:36 |
| corvus | i aim to do better than last time ^ :) | 00:36 |
| opendevreview | James E. Blair proposed opendev/zone-opendev.org master: Replace ze01-ze06 https://review.opendev.org/c/opendev/zone-opendev.org/+/885167 | 00:39 |
| opendevreview | James E. Blair proposed opendev/zone-opendev.org master: Replace ze07-ze12 https://review.opendev.org/c/opendev/zone-opendev.org/+/885168 | 00:39 |
| opendevreview | James E. Blair proposed opendev/system-config master: Replace ze01-ze06 https://review.opendev.org/c/opendev/system-config/+/885169 | 00:44 |
| opendevreview | James E. Blair proposed opendev/system-config master: Replace ze07-ze12 https://review.opendev.org/c/opendev/system-config/+/885170 | 00:44 |
| frickler | this looks unhealthy https://zuul.opendev.org/t/openstack/builds?job_name=infra-prod-service-zuul&project=opendev/system-config | 07:37 |
| corvus | the openafs-amd64-hwe ppa is gone? | 13:52 |
| corvus | E: Failed to fetch http://ppa.launchpad.net/openstack-ci-core/openafs-amd64-hwe/ubuntu/dists/xenial/InRelease 403 Forbidden [IP: 2620:2d:4000:1::3e 80] | 13:52 |
| corvus | there's just an "openafs" ppa | 13:53 |
| corvus | infra-root: ^ i have no visibility into how or why that may have happened | 13:56 |
| corvus | (oh if it's not clear, this is in response to frickler 's observation -- that error is why the zuul service jobs are failing) | 13:57 |
| corvus | oh hrm, since those are focal nodes, it should be using the "openafs" package... | 13:59 |
| corvus | s/package/ppa/ | 13:59 |
| corvus | that must mean that somehow the new hosts are affecting the fact cache, despite not being added to the inventory yet. | 14:01 |
| corvus | do we somehow have the openstack inventory plugin operating and providing ip addresses for hosts? | 14:03 |
| corvus | oh! or does the launch script do that for us? | 14:03 |
| corvus | that must be it. so this is a somewhat unanticipated result of same-name replacement. | 14:04 |
| corvus | i think the best way to proceed is add hosts to emergency and roll forward with jammy replacement | 14:05 |
| opendevreview | James E. Blair proposed opendev/system-config master: Replace ze01-ze06 https://review.opendev.org/c/opendev/system-config/+/885169 | 14:07 |
| opendevreview | James E. Blair proposed opendev/system-config master: Replace ze07-ze12 https://review.opendev.org/c/opendev/system-config/+/885170 | 14:07 |
| opendevreview | James E. Blair proposed opendev/system-config master: Run zuul-executor on jammy https://review.opendev.org/c/opendev/system-config/+/885171 | 14:07 |
| corvus | infra-root: i'm self-approving the first set of changes; feel free to review or retro-review them | 14:09 |
| opendevreview | Merged opendev/zone-opendev.org master: Convert spaces to tabs https://review.opendev.org/c/opendev/zone-opendev.org/+/885166 | 14:15 |
| opendevreview | Merged opendev/zone-opendev.org master: Replace ze01-ze06 https://review.opendev.org/c/opendev/zone-opendev.org/+/885167 | 14:15 |
| opendevreview | Merged opendev/system-config master: Run zuul-executor on jammy https://review.opendev.org/c/opendev/system-config/+/885171 | 16:14 |
| opendevreview | Merged opendev/system-config master: Replace ze01-ze06 https://review.opendev.org/c/opendev/system-config/+/885169 | 16:14 |
| corvus | I've removed ze01-ze06 from emergency and removed cached facts for all executors, so the next hourly run (which is, unfortunately, about 50 minutes away) should deploy them. | 16:21 |
| corvus | oh it looks like there's also a full system-config deploy buildset running now; that includes a zuul deploy as well, so i guess that's in the race. we'll see who wins. | 16:22 |
| opendevreview | Radosław Piliszek proposed openstack/project-config master: Add nebulous/zuul-jobs https://review.opendev.org/c/openstack/project-config/+/885177 | 17:10 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!