| fungi | i am often least convinced by the things i write | 00:00 |
|---|---|---|
| opendevreview | Tony Breeds proposed opendev/system-config master: [dnm] checking testing for the existing registry https://review.opendev.org/c/opendev/system-config/+/885524 | 00:02 |
| opendevreview | Tony Breeds proposed opendev/system-config master: [DNM] Test insecure-ci-registry.opendev.org on jammy https://review.opendev.org/c/opendev/system-config/+/885421 | 00:17 |
| ianw | it is quite suspicious that arm64 builds https://zuul.opendev.org/t/openstack/build/249b0f5ce2384516933eb1e91c8af52e/log/dkms-make-logs/var/lib/dkms/openafs/1.8.9-1.el9/5.14.0-319.el9.aarch64/aarch64/log/make.log | 00:29 |
| frickler | ianw: I would support option 3), which kind of matches the current plan for devstack | 05:33 |
| *** amoralej|off is now known as amoralej | 06:02 | |
| *** amoralej is now known as amoralej|lunch | 12:27 | |
| *** amoralej|lunch is now known as amoralej | 13:03 | |
| opendevreview | Tony Breeds proposed opendev/system-config master: [DNM] Test insecure-ci-registry.opendev.org on jammy https://review.opendev.org/c/opendev/system-config/+/885421 | 15:07 |
| *** amoralej is now known as amoralej|off | 15:19 | |
| opendevreview | Tony Breeds proposed opendev/system-config master: [DNM] Test insecure-ci-registry.opendev.org on jammy https://review.opendev.org/c/opendev/system-config/+/885421 | 15:49 |
| opendevreview | Tony Breeds proposed opendev/system-config master: [DNM] Test insecure-ci-registry.opendev.org on jammy https://review.opendev.org/c/opendev/system-config/+/885421 | 16:16 |
| opendevreview | Tony Breeds proposed opendev/system-config master: [DNM] Test insecure-ci-registry.opendev.org on jammy https://review.opendev.org/c/opendev/system-config/+/885421 | 16:40 |
| clarkb | tonyb: did comparing to the existing setup help at all? | 16:41 |
| tonyb | Not really, it "just works" on the current version. | 16:41 |
| tonyb | if this build doesn't make progress I'll put all the debugging stuff into a patch do the jammy update on top of that. | 16:42 |
| clarkb | tonyb: did it have the swift auth exception too? | 16:42 |
| tonyb | Yeah same error in the registry log but the registry started. | 16:43 |
| clarkb | weird | 16:43 |
| tonyb | but current *guess* is that the docker verions in bionic and jammy differ in how host networking exposes ports | 16:43 |
| tonyb | on jammy the conatiner *is* running (well not in my last couple of patches becuse I broke it) but the port (:5000) isn't visible/bound to the hosts networking stack | 16:44 |
| clarkb | tonyb: even with host networking set? | 16:45 |
| tonyb | Yup | 16:45 |
| clarkb | weird | 16:45 |
| clarkb | we can hold a node to inspect that | 16:45 |
| clarkb | (though I need to load my ssh keys to do that first) | 16:46 |
| fungi | i can set an autohold if you have the details | 16:47 |
| fungi | change/job | 16:47 |
| tonyb | 885241/system-config-run-docker-registry | 16:48 |
| fungi | sudo zuul-client autohold --tenant=openstack --project=opendev.org/opendev/system-config --job=system-config-run-docker-registry --ref='refs/changes/41/885241/.*' --reason='tonyb investigating docker issues on jammy' | 16:51 |
| fungi | the trap is set | 16:51 |
| fungi | on the next failure, let me know what ssh key you want added | 16:51 |
| tonyb | So it's tha same docker version: https://paste.opendev.org/show/bMPMlV5o0rBTrYjdwq8t/ | 16:52 |
| tonyb | fungi: Thanks | 16:55 |
| opendevreview | Tony Breeds proposed opendev/system-config master: [DNM] Test insecure-ci-registry.opendev.org on jammy https://review.opendev.org/c/opendev/system-config/+/885421 | 17:07 |
| fungi | tonyb: ssh root158.69.69.182 | 17:53 |
| fungi | er, root@158.69.69.182 | 17:53 |
| tonyb | root@158.69.69.182: Permission denied (publickey). | 18:00 |
| fungi | er, checking | 18:01 |
| tonyb | Thanks | 18:01 |
| tonyb | I tried zuul@ also | 18:01 |
| fungi | tonyb: try root again? i think the way i appended the authorized_keys file caused it to not end with a newline, so opensshd may not have liked t hat | 18:02 |
| tonyb | Same result | 18:02 |
| fungi | this may be due to special configuration in our deployment orchestration limiting root user access, i'll pivot | 18:03 |
| tonyb | okay | 18:03 |
| fungi | tonyb: try zuul@ | 18:04 |
| tonyb | bingo | 18:04 |
| fungi | cool, hopefully the zuul user can sudo whatever but let me know if you need someone with root perms to make it possible | 18:06 |
| tonyb | Nope sudo is working | 18:06 |
| fungi | cool | 18:07 |
| clarkb | ya we restrict the source of root logins iirc | 18:07 |
| clarkb | which is fine for the rest of us because it splats our users down | 18:07 |
| fungi | right, i just didn't realize that would impact appending a key to ~root/.ssh/authorized_keys on a test deploy | 18:08 |
| tonyb | It looks like the zuul-registry process spawned by dumbinit is crashing sometime after it lots the authexception | 18:20 |
| clarkb | firstthought was a python version difference but that is fixed by the container. That is really odd considering that the appljication itself should be basically the same between the two systems due to the container | 18:22 |
| clarkb | tonyb: did it provision a self signed cert using the fake LE stuff (when I read the logs it looked like it was doing so but maybe that is failing somewhere preventing it from listening on the socket?) | 18:23 |
| tonyb | It looks like it has /etc/letsencrypt-certs/insecure-ci-registry99.opendev.org/ has a bunch of files | 18:24 |
| clarkb | cool it probably worked then. There should be a .cert iirc | 18:24 |
| clarkb | if there is a .cert then the fake negotiation worked | 18:24 |
| clarkb | maybe it is .crt | 18:24 |
| tonyb | .cer, .csr, .key | 18:24 |
| clarkb | probably not the cert then. Maybe need to strace the container process (probably no strace in the container though) or add extra debugging somehow | 18:26 |
| tonyb | Yeah, I'm puzzling how to do some of that. | 18:27 |
| tonyb | I have a meeting I need to go to | 18:28 |
| tonyb | Okay so my current theory is that a) the underlying problem *is* the keystone auth issue killing the container and docker[1] restarting it ; and b) somehow the 'bionic' node keeps the service "alive" long enough for the registry to appear to be up. | 19:39 |
| tonyb | [1] despite what I said earlier about the dumbinit restarting things I think it's docker | 19:40 |
| tonyb | I'm just not sure how to really test it | 19:40 |
| tonyb | fungi: I think you can remove the autohold while I ponder | 19:40 |
| tonyb | Okay /me has a some ideas | 20:00 |
| fungi | will do, thanks | 20:25 |
| fungi | cleaned up | 20:27 |
| tonyb | Thanks | 20:28 |
| opendevreview | Tony Breeds proposed opendev/system-config master: [DNM] Test insecure-ci-registry.opendev.org on jammy https://review.opendev.org/c/opendev/system-config/+/885421 | 20:42 |
| opendevreview | Tony Breeds proposed opendev/system-config master: [DNM] Add debbugging to docker config https://review.opendev.org/c/opendev/system-config/+/885659 | 20:42 |
| clarkb | tonyb: we may just want to check that docker is starting the container rather than check for the port being open | 21:09 |
| tonyb | Well that'd work too, but feels kinda scary without confirmation that my hunch is in the right ballpark | 21:10 |
| tonyb | I can confirm that both bionic and jammy *DO NOT* show port 5000 being open and in listen right after docker-compose up completes and that the conatiner is restarted by dockerd at least once while the job is running | 21:12 |
| opendevreview | Tony Breeds proposed opendev/system-config master: [DNM] Add debbugging to docker config https://review.opendev.org/c/opendev/system-config/+/885659 | 21:19 |
| opendevreview | Tony Breeds proposed opendev/system-config master: [DNM] Test insecure-ci-registry.opendev.org on jammy https://review.opendev.org/c/opendev/system-config/+/885421 | 21:19 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!