| ianw | i can get the python3.11 to work by editing it's rpath in the builder container | 01:48 |
|---|---|---|
| ianw | i imagine a bug report along the lines of "i enable unstable on the buster container then pull in packages from there" will not be considered helpful | 01:48 |
| ianw | bullseye i mean | 01:49 |
| *** ysandeep|out is now known as ysandeep | 02:17 | |
| Clark[m] | Though that comment indicates this should be fine so maybe they would accept it? | 02:46 |
| ianw | yeah i'm trying to replicate it in a less obscure way | 02:47 |
| ianw | ok, i believe https://github.com/docker-library/python/issues/784 is actionable | 03:44 |
| *** ysandeep is now known as ysandeep|afk | 05:08 | |
| *** pojadhav- is now known as pojadhav | 06:12 | |
| *** arxcruz is now known as arxcruz|ruck | 07:54 | |
| opendevreview | Cedric Jeanneret proposed opendev/system-config master: Correct (again) how ansible-galaxy proxy is configured https://review.opendev.org/c/opendev/system-config/+/869819 | 07:58 |
| *** jpena|off is now known as jpena | 08:39 | |
| *** soniya29|rover is now known as soniya29|rover|lunch | 09:18 | |
| *** elodilles is now known as elodilles_pto | 09:58 | |
| *** soniya is now known as soniya29|rover | 10:20 | |
| *** rlandy|out is now known as rlandy | 11:14 | |
| *** soniya29|rover is now known as soniya29|rover|brb | 11:26 | |
| *** soniya29|rover|brb is now known as soniya29|rover | 11:54 | |
| mnasiadka | Has something changed in zuul/nodepool? Some builds of Kolla jobs have nodepool.private_ipv4 as an empty string? (And we're using it for our multi-node-vxlan-overlay role) | 12:18 |
| mnasiadka | example: https://f8764396ab2585080976-ab9cdbb0aa9d61bf3167c27c9fa17003.ssl.cf2.rackcdn.com/867771/10/gate/kolla-ansible-debian-source-upgrade/fca1bd0/zuul-info/inventory.yaml | 12:18 |
| fungi | mnasiadka: we do constantly upgrade zuul and nodepool from the tip of their master branches, so many somethings are always changing in zuul and nodepool. as to whether that's the reason for the behavior change you've observed, no idea. would need to go digging in recent commits | 12:41 |
| mnasiadka | in theory docs say private_ipv4 is optional | 12:42 |
| mnasiadka | but jobs retrying in pre and failing are not something that is fun, and we haven't seen that surely in the previous days | 12:43 |
| fungi | mnasiadka: my first guess would be https://review.opendev.org/c/zuul/nodepool/+/862522 "Convert openstack driver to statemachine" which merged yesterday | 12:44 |
| fungi | but yes, if private_ipv4 isn't guaranteed to exist, you may want to use a public address fallback | 12:45 |
| mnasiadka | seems so | 12:45 |
| fungi | certainly not all of our providers supply a separate private ipv4 address, i wonder if the former behavior in the openstack driver was to fill it with the public address | 12:45 |
| mnasiadka | probably, let me fix that on our side, thanks for help | 12:46 |
| fungi | yep, the old driver had this... | 12:46 |
| fungi | if not self.node.private_ipv4: self.node.private_ipv4 = server.public_v4 | 12:46 |
| fungi | skimming the new driver code, i'm not seeing it do that anywhere | 12:47 |
| fungi | given the documentation says there may not be a value for private_ipv4 i don't know whether this would be treated as a regression, but i'll bring it up in matrix | 12:48 |
| *** ysandeep|afk is now known as ysandeep | 13:20 | |
| *** dasm|off is now known as dasm | 14:34 | |
| frickler | don't have time to look deeper today, but devstack multinode might be seeing a similar issue https://zuul.opendev.org/t/openstack/build/edafb380f8714fc4b0db27a527c6e692 | 14:38 |
| *** ysandeep is now known as ysandeep|out | 14:42 | |
| *** tosky_ is now known as tosky | 14:47 | |
| fungi | it's up for discussion in the zuul matrix channel at the moment whether nodepool should make the old openstack behavior consistent across all the drivers, or add a release note | 14:49 |
| fungi | but regardless, we should probably just make our job definitions more robust, since we were relying on something that the documentation said wasn't reliable | 14:50 |
| opendevreview | Cedric Jeanneret proposed opendev/system-config master: Correct (again) how ansible-galaxy proxy is configured https://review.opendev.org/c/opendev/system-config/+/869819 | 14:50 |
| fungi | frickler: interestingly, it looks like the multi-node-bridge role in zuul-jobs does apply a fallback to public_ipv4 when private_ipv4 is not defined | 14:55 |
| corvus | i think we're going to just match the old behavior for now, then figure out what we want to do in the future and announce a transition period. we should be able to restart the launchers in about an hour if we don't hit any job failures. | 15:00 |
| fungi | codesearch indicates that openstack/devstack and openstack/grenade do indeed use private_ipv4 exclusively for setting envvars in localrc, also openstack/openstack-zuul-jobs and openstack/tacker and openstack/kolla-ansible and openstack/skyline-console and openstack/octavia-tempest-plugin have some custom jobs that may assume it's always set, as well as the ovs_vxlan_bridge_peers role in | 15:01 |
| fungi | openstack/devstack-gate, also a couple of playbooks in vexxhost/openstack-operator | 15:01 |
| fungi | so yeah, i have a feeling this is more widespread than we first realized | 15:02 |
| fungi | mnasiadka: ^ just an update in case you're not glued to this channel, we've approved a change to nodepool to make the new driver backwards compatible with that behavior for now, and will come up with a transition plan to allow projects time to update their configurations without further disruption | 15:05 |
| fungi | once 870100 merges we can deploy updated container images with the fix and restart the launchers, so new nodes after that point will all have a private_ipv4 | 15:08 |
| opendevreview | Cedric Jeanneret proposed opendev/system-config master: Correct (again) how ansible-galaxy proxy is configured https://review.opendev.org/c/opendev/system-config/+/869819 | 15:43 |
| *** soniya29|rover is now known as soniya29|rover|out | 15:43 | |
| opendevreview | Clark Boylan proposed opendev/jeepyb master: Raise an error if acl pushes fail https://review.opendev.org/c/opendev/jeepyb/+/869873 | 16:23 |
| clarkb | thats a typo fix in the commit message but should also retrigger failing jobs | 16:23 |
| *** marios is now known as marios|out | 16:39 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add Gerrit 3.7 images https://review.opendev.org/c/opendev/system-config/+/869765 | 16:40 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Convert Verified MaxWithBlock to submit-requirement in testing https://review.opendev.org/c/opendev/system-config/+/870109 | 16:40 |
| clarkb | That change illustrates the next sort of mass gerrit refs/meta/config update we'll need to do after the copycondition update | 16:44 |
| clarkb | I've also just asked on Gerrit discord about some confusing submit-requirement config documentation | 16:53 |
| opendevreview | Merged opendev/system-config master: Override default branch name https://review.opendev.org/c/opendev/system-config/+/869975 | 17:00 |
| clarkb | oh heh I messaged the slack room | 17:26 |
| mnasiadka | fungi: good to know, but I'll backport the already merged change to stable branches to be ready ;-) | 17:34 |
| *** jpena is now known as jpena|off | 17:36 | |
| corvus | i'm going to force-merge the nodepool change; it has hit its own bug 3 times in different jobs based on random provider allocation | 17:37 |
| clarkb | infra-root the stack at https://review.opendev.org/c/opendev/system-config/+/869763 is enough to bootsrap 3.7 image builds | 17:42 |
| clarkb | I haven't started on the upgrade job as I half expect that to be mpre involved. But I guess I should start that next | 17:43 |
| corvus | i will pull and restart launchers now | 17:47 |
| corvus | restarted nl01 | 17:49 |
| corvus | #status log restarted all nodepool launchers to pick up private_ipv4 bugfix | 17:51 |
| opendevstatus | corvus: finished logging | 17:51 |
| corvus | for safety, let's say as a rule of thumb that jobs started after 1800 utc should not expect to see the issue any more. | 17:52 |
| clarkb | corvus: sounds good | 17:53 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add Gerrit 3.6 -> 3.7 upgrade test job https://review.opendev.org/c/opendev/system-config/+/870114 | 17:56 |
| fungi | thanks corvus | 17:59 |
| fungi | i stepped away for lunch and missed all the excitement | 17:59 |
| clarkb | it looks like our zuul resuts plugin continues to work with the new js library. I'm slightly surprised by this. Our theming does too | 18:13 |
| clarkb | Two fewer things to worry about pre upgrade though I suspect that both will need updates eventually due to the js framework update | 18:13 |
| clarkb | arg looks like compiling protobuf for gerrit 3.6 builds is broken right now | 18:19 |
| clarkb | and this impacts my jeepyb change | 18:19 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Fix Gerrit 3.6 image build https://review.opendev.org/c/opendev/system-config/+/870118 | 18:33 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add Gerrit 3.6 -> 3.7 upgrade test job https://review.opendev.org/c/opendev/system-config/+/870114 | 18:55 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add Gerrit 3.6 -> 3.7 upgrade test job https://review.opendev.org/c/opendev/system-config/+/870114 | 20:04 |
| *** dasm is now known as dasm|off | 23:37 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!