| JayF | I'm not willing to try via git this time; it's my EOD and I don't think it's wise to push those changes when I'm going to bail afterwards | 00:00 |
|---|---|---|
| JayF | if there's another way to check outside of web or JFDI, I'm willing to try that, otherwise I'll find out if it works the hard way first thing tomorrow :D | 00:00 |
| fungi | i'll be around | 00:00 |
| fungi | just give me a heads up | 00:00 |
| JayF | ack; will-do. Will probably try to get on it early, like 7a/8a PST | 00:01 |
| ianw | yeah, just to confirm i pulled the refs/meta/config on ironic-inspector and it's udpated | 00:01 |
| ianw | updated even | 00:01 |
| fungi | i'm on the opposite coast, so no problem | 00:02 |
| ianw | 2023-01-11 23:55:19,365: manage_projects - INFO - Processing project: openstack/ironic-inspector | 00:02 |
| ianw | 2023-01-11 23:55:22,594: manage_projects - INFO - Processing project: openstack/ironic-lib | 00:02 |
| fungi | perfect | 00:03 |
| ianw | so, let's say 5 seconds/project | 00:03 |
| fungi | for ~2.2k projects if memory serves | 00:03 |
| clarkb | we could also do them in batches | 00:03 |
| clarkb | by removing subsets of the cache file | 00:03 |
| ianw | $ cat manage-projects.yaml.log | grep 'manage_projects - INFO - Processing' | wc -l | 00:04 |
| ianw | 2329 | 00:04 |
| fungi | yeah, ls-projects says 2331 but we don't update all of those | 00:04 |
| ianw | so, 4800 is going to be tight | 00:05 |
| fungi | i think it includes the All-Projects and All-Users repos in ls-projects | 00:05 |
| fungi | not so much "tight" as ~2.5x that long | 00:06 |
| fungi | we could blow away slices of the checksums | 00:06 |
| fungi | let it reapply in smaller batches | 00:06 |
| ianw | or run manually | 00:07 |
| Clark[m] | The upside to only removing hashes is that it will reconcile the refs meta config without trying to do the other stuff too | 00:09 |
| ianw | i feel a bit vested in this because removing the deprecated copy conditions is going to be almost the same thing | 00:09 |
| Clark[m] | And I think only the refs meta config has this issue in jeepyb though before we rely on that reviewing it is a good idea | 00:09 |
| Clark[m] | Mostly I get scared making changes to jeepyb because the potential for pain is high but if we aren't changing jeepyb code we're probably fine | 00:10 |
| fungi | manage-projects is the only code we're still relying on in jeepyb right? | 00:10 |
| Clark[m] | And the lp sync stuff | 00:11 |
| ianw | i will write some thoughts at https://etherpad.opendev.org/p/760YNeM5OEFS1hlr7bE5 and we can come up with a checklist plan | 00:12 |
| fungi | oh, yeah the bug updating scripts never did get replaced by zuul jobs | 00:28 |
| fungi | though it seems like they could be trivially converted | 00:28 |
| ianw | clarkb/fungi: https://etherpad.opendev.org/p/760YNeM5OEFS1hlr7bE5 is my summary of the situation. no urgent but we can follow up on all the todos | 01:17 |
| ianw | now ... ipv6 issues :/ | 01:17 |
| Clark[m] | ianw: left a couple of notes on the ehterpad but I think that captures it well | 01:29 |
| ianw | thx, will review | 01:47 |
| opendevreview | Ian Wienand proposed opendev/system-config master: iptables: handle hosts in allowed groups not having an ipv6 address https://review.opendev.org/c/opendev/system-config/+/869886 | 02:49 |
| opendevreview | Ian Wienand proposed opendev/system-config master: nodepool-base: use ipv4 ZK addresses if we don't have an ipv6 address https://review.opendev.org/c/opendev/system-config/+/869888 | 03:12 |
| opendevreview | Ian Wienand proposed opendev/system-config master: iptables: handle hosts in allowed groups not having an ipv6 address https://review.opendev.org/c/opendev/system-config/+/869886 | 04:28 |
| opendevreview | Ian Wienand proposed opendev/system-config master: nodepool-base: use ipv4 ZK addresses if we don't have an ipv6 address https://review.opendev.org/c/opendev/system-config/+/869888 | 04:28 |
| opendevreview | Ian Wienand proposed opendev/system-config master: infra-prod: save run logs even if being published https://review.opendev.org/c/opendev/system-config/+/869891 | 04:59 |
| opendevreview | Ian Wienand proposed opendev/system-config master: infra-prod: don't encrypt logs if they are being published https://review.opendev.org/c/opendev/system-config/+/869892 | 04:59 |
| opendevreview | Ian Wienand proposed opendev/system-config master: iptables: handle hosts in allowed groups not having an ipv6 address https://review.opendev.org/c/opendev/system-config/+/869886 | 05:24 |
| opendevreview | Ian Wienand proposed opendev/system-config master: nodepool-base: use ipv4 ZK addresses if we don't have an ipv6 address https://review.opendev.org/c/opendev/system-config/+/869888 | 05:24 |
| *** ysandeep is now known as ysandeep|ruck | 05:32 | |
| *** marios is now known as marios|rover | 06:03 | |
| *** ysandeep|ruck is now known as ysandeep|ruck|brb | 06:54 | |
| *** soniya29 is now known as soniya29|afk | 07:16 | |
| *** ysandeep|ruck|brb is now known as ysandeep|ruck | 07:21 | |
| *** ysandeep|ruck is now known as ysandeep|lunch | 07:22 | |
| opendevreview | Cedric Jeanneret proposed opendev/system-config master: Correct (again) how ansible-galaxy proxy is configured https://review.opendev.org/c/opendev/system-config/+/869819 | 07:33 |
| *** ysandeep|lunch is now known as ysandeep|ruck | 08:12 | |
| *** jpena|off is now known as jpena | 08:37 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: iptables: handle hosts in allowed groups not having an ipv6 address https://review.opendev.org/c/opendev/system-config/+/869886 | 10:50 |
| opendevreview | Ian Wienand proposed opendev/system-config master: nodepool-base: use ipv4 ZK addresses if we don't have an ipv6 address https://review.opendev.org/c/opendev/system-config/+/869888 | 10:50 |
| *** rlandy|out is now known as rlandy | 11:11 | |
| opendevreview | Cedric Jeanneret proposed opendev/system-config master: Correct (again) how ansible-galaxy proxy is configured https://review.opendev.org/c/opendev/system-config/+/869819 | 11:20 |
| *** gthiemon1e is now known as gthiemonge | 11:36 | |
| opendevreview | Cedric Jeanneret proposed opendev/system-config master: Correct (again) how ansible-galaxy proxy is configured https://review.opendev.org/c/opendev/system-config/+/869819 | 12:04 |
| *** ysandeep|ruck is now known as ysandeep|ruck|afk | 12:27 | |
| opendevreview | Cedric Jeanneret proposed opendev/system-config master: Correct (again) how ansible-galaxy proxy is configured https://review.opendev.org/c/opendev/system-config/+/869819 | 12:54 |
| opendevreview | Cedric Jeanneret proposed opendev/system-config master: Override default branch name https://review.opendev.org/c/opendev/system-config/+/869975 | 13:02 |
| *** dasm|off is now known as dasm | 13:10 | |
| opendevreview | Cedric Jeanneret proposed opendev/system-config master: Correct (again) how ansible-galaxy proxy is configured https://review.opendev.org/c/opendev/system-config/+/869819 | 13:42 |
| opendevreview | Riccardo Pittau proposed openstack/project-config master: Reduce virtualpdu tests https://review.opendev.org/c/openstack/project-config/+/869979 | 13:45 |
| opendevreview | Cedric Jeanneret proposed opendev/system-config master: Override default branch name https://review.opendev.org/c/opendev/system-config/+/869975 | 13:54 |
| *** ysandeep|ruck|afk is now known as ysandeep|ruck | 14:02 | |
| fungi | not that i'm interested in changing directions with our keycloak work, but https://github.com/zitadel/zitadel just came to my attention as a recent alternative | 14:24 |
| opendevreview | Cedric Jeanneret proposed opendev/system-config master: Correct (again) how ansible-galaxy proxy is configured https://review.opendev.org/c/opendev/system-config/+/869819 | 14:24 |
| opendevreview | Merged openstack/project-config master: ARM64 nodes : rename larger types to be more descriptive https://review.opendev.org/c/openstack/project-config/+/868439 | 14:25 |
| opendevreview | Merged openstack/project-config master: Add new linaro cloud https://review.opendev.org/c/openstack/project-config/+/868442 | 14:25 |
| opendevreview | Thierry Carrez proposed opendev/irc-meetings master: Rotate Large Scale SIG meeting schedule https://review.opendev.org/c/opendev/irc-meetings/+/869989 | 14:29 |
| opendevreview | Jeremy Stanley proposed opendev/irc-meetings master: Fix tox.ini for Tox v4 https://review.opendev.org/c/opendev/irc-meetings/+/870011 | 14:44 |
| opendevreview | Merged opendev/irc-meetings master: Fix tox.ini for Tox v4 https://review.opendev.org/c/opendev/irc-meetings/+/870011 | 15:05 |
| opendevreview | Cedric Jeanneret proposed opendev/system-config master: Correct (again) how ansible-galaxy proxy is configured https://review.opendev.org/c/opendev/system-config/+/869819 | 15:25 |
| *** ysandeep|ruck is now known as ysandeep|dinner | 16:06 | |
| clarkb | as a heads up I'm way behind on paperwork I need to get done by the end of theweek so I'm going to try and focus on that today | 16:16 |
| clarkb | I might even go do literal paperwork to help my avoid distractions on the computer | 16:16 |
| fungi | yeah, i'm in a similar situation | 16:20 |
| opendevreview | Cedric Jeanneret proposed opendev/system-config master: Correct (again) how ansible-galaxy proxy is configured https://review.opendev.org/c/opendev/system-config/+/869819 | 16:21 |
| *** ysandeep|dinner is now known as ysandeep | 16:23 | |
| opendevreview | Merged opendev/irc-meetings master: Rotate Large Scale SIG meeting schedule https://review.opendev.org/c/opendev/irc-meetings/+/869989 | 16:32 |
| *** marios|rover is now known as marios|out | 16:35 | |
| JayF | running ironic-inspector bugfix branch cleanups now | 16:44 |
| JayF | complete | 16:46 |
| fungi | yay! | 16:46 |
| JayF | hopefully ironic moves from a starring role in zuul config errors | 16:48 |
| JayF | and goes more into being an extra lol | 16:48 |
| fungi | they're called "background actors" these days ;) | 16:49 |
| *** ysandeep is now known as ysandeep|out | 16:55 | |
| *** jpena is now known as jpena|off | 17:28 | |
| corvus | i have approved the nodepool openstack statemachine driver change, so i'll put the launchers in the emergency file now | 17:59 |
| corvus | also the builders | 17:59 |
| corvus | can we put a group name in there? (like "nodepool")? | 18:01 |
| corvus | nope | 18:04 |
| fungi | yeah, not sure how that would work since it's already a group. i guess ansible doesn't do nested groups? | 18:37 |
| corvus | at least not that the simple and obvious way :) | 18:44 |
| corvus | okay change merged, i'm going to kick nl01 | 19:10 |
| corvus | nl01 is running the new code | 19:15 |
| corvus | i note it's deleting some leaked servers, and they appear to be actually leaked (in that they were failed deletes from hours ago) | 19:16 |
| corvus | i see both create and delete statemachines completing | 19:18 |
| fungi | that sounds promising | 19:33 |
| corvus | got a bunch of timeouts deleting servers, but considering the leaked deletes from earlier, i guess that's expected. | 19:35 |
| fungi | yes, it's unfortunately not all that uncommon in our providers | 19:37 |
| corvus | i'll grab some food then do some more rolling | 19:57 |
| corvus | https://paste.opendev.org/show/b26RcjZ5PSO2aLfNOLxr/ is an interesting error from keystone, but appears to be a one-off | 20:05 |
| ianw | infra-root: https://review.opendev.org/q/topic:linaro-ipv6-add are two fixes for the osuosl builder not having ipv4 addresses that i think are ready for review. would be good to unbreak the base job. | 20:12 |
| corvus | kicked nl02 which appears to be limestone/inmotion | 20:41 |
| corvus | infra-root: before i restart nl03 -- i'm concerned with this exception (which is happening on the old code): https://paste.opendev.org/show/b3BVSjjwnTJIlY0AaTBU/ | 20:56 |
| corvus | yeah, that's a syntax error in the config file | 20:57 |
| corvus | is nodepool going to actually start up with that? | 20:57 |
| corvus | i'm guessing so since it was last restarted 35 hours ago | 20:57 |
| corvus | so unless we messed up the syntax since then it must have worked | 20:57 |
| ianw | hrm, since linaro ins implicated we may have messed up the config since then, i think some additions may have merged | 20:57 |
| ianw | https://review.opendev.org/c/openstack/project-config/+/868442 | 20:58 |
| corvus | oh the error is in clouds.yaml sorry i should have said | 20:58 |
| ianw | ahh, i guess then 868442 might have started using the wrong clouds.yaml | 20:59 |
| ianw | what have i messed up :/? | 20:59 |
| corvus | i'll fix | 20:59 |
| opendevreview | James E. Blair proposed opendev/system-config master: Fix clouds.yaml syntax error for linaro https://review.opendev.org/c/opendev/system-config/+/870031 | 21:00 |
| corvus | ianw: ^ | 21:00 |
| ianw | it shoul dhave a value before metrics i guess | 21:00 |
| corvus | but that was added in dec, so i'm guessing the launcher can still start | 21:00 |
| corvus | i think i'd like to go ahead and restart the launcher, and if it barfs, i'll manually fix up clouds.yaml on nl03 | 21:01 |
| ianw | true, probably only if it references that, which it just started doing. sgtm | 21:01 |
| corvus | restarting now | 21:02 |
| corvus | it's in a restart loop. it's a mystery to me why this didn't happen earlier | 21:03 |
| corvus | will apply manual fix now | 21:03 |
| ianw | probably last time it started it didn't have linaro in the config file, so i was ignoring the invalid entry | 21:03 |
| corvus | ah gotcha | 21:04 |
| corvus | it's up now | 21:04 |
| corvus | ftr, that change got a failed node request, but that was from me shutting down nl03 and not having any fallback regions; not related to running on the new code. | 21:06 |
| ianw | ok; i'm not sure if images are uploaded and what is working or not in the new linaro cloud yet | 21:09 |
| corvus | restarting nl04 | 21:19 |
| ianw | well nb03, which would upload the images, has the same bad config, so that's not happening | 21:21 |
| corvus | ianw: i'm ready to restart builders -- should i fix nb03 manually and restart it first? | 21:24 |
| corvus | looks like you beat me to the fix | 21:25 |
| ianw | yeah, but please restart, i'll watch the linaro side of things | 21:25 |
| corvus | it's currently in a restart loop with a pretty weird error: ModuleNotFoundError: No module named 'nodepool' | 21:26 |
| corvus | i'm just going to go ahead and pull the new image and restart it | 21:26 |
| corvus | that error seems to still be present | 21:27 |
| corvus | can be seen with docker logs | 21:27 |
| clarkb | that makes me think of the tox issues with skipsdist. Though no tox should be involved here | 21:29 |
| ianw | very weird, hard to think a container that does that could make it through publishing, so must be something quite odd | 21:30 |
| corvus | yeah, a simple docker run --rm -it zuul/nodepool-builder gets past that point | 21:31 |
| corvus | nb03 is an arm host? | 21:31 |
| ianw | yes | 21:31 |
| corvus | okay that explains the behavior difference at least | 21:31 |
| clarkb | probably need to exec bash on that image and see what the python installation situation looks like? | 21:32 |
| ianw | yeah i am in it now ... | 21:32 |
| corvus | if i run that command on my desktop it works, but not if i run it on nb03 | 21:32 |
| ianw | # python3 -m nodepool | 21:32 |
| ianw | /usr/local/bin/python3: No module named nodepool | 21:32 |
| corvus | so it seems like we have nodepool image building problems that are at least a few days if not weeks old | 21:33 |
| corvus | there are a bunch of nodepool files in the image | 21:33 |
| ianw | no __init__ or nodepool.py though | 21:34 |
| ianw | no that's not true, __init__ is there | 21:34 |
| corvus | '/usr/local/lib/python3.11/site-packages' is not in sys.path on arm, but it is on x86 | 21:34 |
| clarkb | https://zuul.opendev.org/t/zuul/build/1331b0fcc39147799528b6c731cbea7e/log/job-output.txt#7090 | 21:35 |
| clarkb | thats the build for the recent release note update (picked that one as a minimal change) and it seemsto clearly record nodepool being installed on arm64 | 21:35 |
| corvus | arm: ['', '/usr/local/lib/python311.zip', '/usr/local/lib/python3.11', '/usr/local/lib/python3.11/lib-dynload', '/usr/local/lib/python3.11/dist-packages'] | 21:35 |
| clarkb | (if you scorll up a bit that #24 step has aarch stuff in it) | 21:35 |
| corvus | x86: ['', '/usr/local/lib/python311.zip', '/usr/local/lib/python3.11', '/usr/local/lib/python3.11/lib-dynload', '/usr/local/lib/python3.11/site-packages'] | 21:35 |
| clarkb | wow thats weird | 21:35 |
| corvus | checking 8.0.0 | 21:36 |
| corvus | it had site-packages, but was also python 3.10.8 | 21:37 |
| corvus | not 3.11.1 | 21:37 |
| corvus | so that could have happened with the python version bump? | 21:37 |
| clarkb | seems possible | 21:37 |
| ianw | our python builder is based on the upstream one, so that implies the bug there? | 21:38 |
| corvus | ianw: that's my initial thought | 21:38 |
| corvus | what do folks think about a temporary PYTHONPATH env setting? | 21:39 |
| corvus | in the docker file | 21:39 |
| clarkb | corvus: I'm good with that | 21:39 |
| clarkb | https://github.com/docker-library/python/issues is their bug tracker I'm trying to pullthat up and see if anyone has complained | 21:39 |
| clarkb | I'm not seeing anything related to that quickly scanning their issues list | 21:40 |
| corvus | docker.io/opendevorg/python-base:3.11-bullseye on arm produces site-packages -- so it may either be our fault, or was very recently fixed | 21:41 |
| ianw | nothing obviously merged about it i can see | 21:41 |
| clarkb | looking at their dockerfiles I'm not seeing anything obvious that they do to manage python path. I guess this could be a bug in whatever is building python for those images | 21:42 |
| clarkb | but ya we can force a rebuild on the opendev base images | 21:42 |
| corvus | to be clear, opendev base image looks correct | 21:42 |
| clarkb | oh | 21:42 |
| clarkb | sorry I read that as the upstream image being correct not opendev. Opendev hasn't updated them in a little bit | 21:42 |
| clarkb | thats something that gets baked into the python install at a pretty base level right? I'm surprised if the opendev base image is fine that the nodepool build on top of that would somehow break it | 21:43 |
| corvus | i agree | 21:44 |
| ianw | root@155a26dfa547:/usr/local/lib/python3.11/site-packages/nodepool# python3 -m site | 21:45 |
| ianw | sys.path = [ | 21:45 |
| ianw | '/usr/local/lib/python3.11/site-packages/nodepool', | 21:45 |
| corvus | okay with a locally applied dockerfile fix, it is up and spewing errors about deleting things | 21:46 |
| ianw | that doesn't tally with print(sys.path) from the python interpreter | 21:46 |
| corvus | is that picking up the cwd? | 21:47 |
| ianw | ahh, yes, good point | 21:47 |
| corvus | also -- this didn't happen for the launcher | 21:49 |
| opendevreview | James E. Blair proposed opendev/system-config master: Override pythonpath in nodepool builder https://review.opendev.org/c/opendev/system-config/+/870034 | 21:49 |
| ianw | although we don't have an arm64 launcher | 21:50 |
| corvus | oh ok that explains that :) | 21:50 |
| corvus | i think i need color coded prompts or something to help keep track of what and where each of these hosts are :) | 21:51 |
| ianw | this may be some interaction with -> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=962665 | 21:51 |
| clarkb | in this case we use a compiled python on debian which seems to be part of the considering of that bug report | 21:52 |
| corvus | i'm going to restart the rest of the builders | 21:53 |
| corvus | ianw: do you want to +w https://review.opendev.org/870034 ? | 21:54 |
| corvus | (i want to make sure to get back to a safe place to drop them from emergency) | 21:54 |
| ianw | ok | 21:55 |
| ianw | so to summarise opendevorg/python-<base|builder>:3.11-bullseye both have site-packages in their sys.path by default | 21:58 |
| corvus | what's the status of nb04? | 21:58 |
| corvus | ianw: yes that's what i saw too | 21:59 |
| ianw | nb04 isn't deployed yet | 21:59 |
| corvus | ah ok :) | 21:59 |
| ianw | that is an arm64 builder intended to replace nb03 | 21:59 |
| corvus | i'll docker-compose down it | 22:00 |
| ianw | it has no ipv6, which tripped up some deployment bits, so it is waiting on https://review.opendev.org/q/topic:linaro-ipv6-add | 22:00 |
| corvus | gotcha | 22:00 |
| opendevreview | Merged opendev/system-config master: Fix clouds.yaml syntax error for linaro https://review.opendev.org/c/opendev/system-config/+/870031 | 22:00 |
| clarkb | ianw: fwiw I +2'd those changes for nb04 but didn't approve them as I'm forcing myself to do paperwork today :) | 22:03 |
| clarkb | (finally making progress on that portion of my todo list so yay) | 22:03 |
| ianw | yep firewall changes are ... special | 22:06 |
| clarkb | in this case if it breaks it won't add rules so its proably safe since ipv4 will stay up... | 22:08 |
| corvus | restarted nb01 and nb02 | 22:21 |
| ianw | # dpkg -S usr/lib/python3.11/site.py | 22:27 |
| ianw | libpython3.11-minimal:arm64: /usr/lib/python3.11/site.py | 22:27 |
| ianw | i think that the "docker" python is picking up the site.py from the system python | 22:27 |
| clarkb | ianw: oh is something installing system python? | 22:28 |
| clarkb | maybe it is just listed in bindep? I wonder if we can avoid that by marking python as already installed in the base image | 22:28 |
| ianw | hrm, i dunno, that file is there on x86 too | 22:28 |
| ianw | removing libpython3.11-minimal does seem to fix it -- but purges things like "dnf" that we need in the builder image | 22:30 |
| *** dasm is now known as dasm|off | 22:31 | |
| clarkb | ah so we actually do need the sytem python for some system python apcakges | 22:31 |
| ianw | now i'm wondering if that PYTHONPATH override will actually break arm64 builds :( | 22:48 |
| opendevreview | Merged opendev/system-config master: Override pythonpath in nodepool builder https://review.opendev.org/c/opendev/system-config/+/870034 | 22:57 |
| ianw | root@891a09c22653:/# ls -lh /usr/local/bin/python3.11 | 22:57 |
| ianw | -rwxr-xr-x 1 root root 15K Nov 15 20:16 /usr/local/bin/python3.11 | 22:57 |
| ianw | root@891a09c22653:/# ls -lh /usr/bin/python3.11 | 22:57 |
| ianw | -rwxr-xr-x 1 root root 6.6M Dec 31 10:23 /usr/bin/python3.11 | 22:57 |
| ianw | i have no idea what they're doing differently ... one seems to include a lot more than the other | 22:58 |
| clarkb | ianw: I believe the intent is to keep them separated sufficiently that both could coexist (there were old issues about this on their issue tracker) | 23:16 |
| ianw | well it's not working :) | 23:19 |
| ianw | ldd reports they both use the same libpython.so | 23:19 |
| clarkb | wut | 23:19 |
| clarkb | https://github.com/docker-library/python/issues/777#issuecomment-1358366197 for the intent | 23:20 |
| ianw | 579 execve("/usr/local/bin/python3", ["/usr/local/bin/python3", "-m", "site"], 0xffffec9764f8 /* 14 vars */) = 0 | 23:20 |
| ianw | 579 openat(AT_FDCWD, "/usr/lib/aarch64-linux-gnu/libpython3.11.so.1.0", O_RDONLY|O_CLOEXEC) = 3 | 23:20 |
| corvus | ianw: are you concerned with the PYTHONPATH being propogated into the dib execution? | 23:21 |
| ianw | corvus: i am wondering if it will mean that tools that we isntall int he builder environment, like dnf for the minimal builds, will now not work | 23:23 |
| ianw | # strings /usr/lib/aarch64-linux-gnu/libpython3.11.so.1.0 | grep dist-package | 23:28 |
| ianw | dist-packages | 23:28 |
| ianw | lib/python3/dist-packages. | 23:28 |
| ianw | i think that the dist-packages default paths must be built into the .so -- it doesn't appear to even reference site.py or anything similar on startup | 23:29 |
| ianw | hrm, i wonder if ld library path override changes it ... | 23:29 |
| ianw | # LD_LIBRARY_PATH=/usr/local/lib/ /usr/local/bin/python3 | 23:30 |
| ianw | '/usr/local/lib/python3.11/site-packages', | 23:30 |
| ianw | ... it does ... | 23:30 |
| ianw | so it seems probably to be that /usr/local/bin/python3 is ultimately using the system .so | 23:31 |
| *** rlandy is now known as rlandy|out | 23:31 | |
| ianw | but why only in the final image, not the builder?!? | 23:31 |
| clarkb | ianw: except the system .so is 3.11 which isn't available on bullseye | 23:31 |
| clarkb | so that has to be the built python 3.11 not the distro python 3 .so | 23:31 |
| clarkb | bullseye has 3.9 on it | 23:31 |
| ianw | well it's coming from ii libpython3.11-minimal:arm64 3.11.1-2 arm64 Minimal subset of the Python language (version 3.11) | 23:32 |
| clarkb | ya that doesn't exist in debian | 23:33 |
| clarkb | so they are already shimming it in so that dpkg is aware | 23:33 |
| clarkb | maybe we do that? | 23:33 |
| ianw | OOHHH ... we might have unstable as part of the Dockerfile to pull in podman etc ... | 23:33 |
| clarkb | hrm I thought we only installed specific packages from unstable though. But ya that could explain it | 23:34 |
| corvus | ianw: okay, you mean that the same issue that we fixed for the builder process with PYTHONPATH may also affect other tools and all we've done is advance to the point that the build will fail because of that. if so, i agree that's a risk, but also that it's probably no worse than just not attempting the builds at all, so we don't need to revert the pythonpath change. | 23:34 |
| ianw | 2023-01-11 09:09:39.911744 | ubuntu-jammy | #29 55.47 Get:21 http://deb.debian.org/debian unstable/main arm64 libpython3.11-stdlib arm64 3.11.1-2 [1672 kB] | 23:35 |
| corvus | (or am i misunderstanding?) | 23:35 |
| ianw | corvus: honestly, i don't know. my intuition is that it will make centos-minimal builds on arm64 fail, but i'm just guessing. i don't really understand | 23:35 |
| ianw | ... well, i'm understanding a bit more ... but it still is all a big mess :) | 23:36 |
| ianw | it seems that we're installing the same python as in the builder, and the built python isn't self-contained enough to avoid using the system packages | 23:36 |
| corvus | ianw: okay, but they will fail to build, so we won't be trying to boot them or anything, we just won't get new updates. and the alternative is that we won't even attempt the build, so basically the same. right? | 23:37 |
| corvus | (my primary concern here is trying to figure out if we're okay to remove the hosts from emergency) | 23:38 |
| ianw | i guess i'd agree with that, yes. it's very unlikely to manage to create and upload a broken image | 23:38 |
| corvus | we could also just shut down nb03 and leave it in emergency | 23:39 |
| ianw | i think maybe leave nb04 in emergency, but just re-enable nb03 | 23:41 |
| corvus | okay, i will make it so | 23:41 |
| ianw | we'll have to figure this out, one way or the other | 23:41 |
| ianw | i wonder if we can block python3.11 from installing | 23:42 |
| Clark[m] | We should be able to stop installing things from unstable and be specific about that? Or is dnf from unstable and that requires unstable python? | 23:42 |
| ianw | i don't know exactly why it's being pulled in | 23:42 |
| corvus | nodepool hosts other than nb04 are enabled again | 23:42 |
| ianw | it's pulled in by "binutils curl dnf debian-keyring dosfstools gdisk git kp | 23:45 |
| ianw | artx qemu-utils vhd-util procps xz-utils zypper zstd debootstrap/unstable" | 23:45 |
| Clark[m] | We do specify the default release is stable. Maybe that isn't working? Debootstrap doesn't seem to dep on python | 23:51 |
| Clark[m] | Dnf does depend on python3 which is supposed to find the release specific version I think | 23:53 |
| ianw | i feel like the python in the docker build should be using rpath to isolate itself | 23:53 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!