| *** dviroel|rover is now known as dviroel|out | 00:11 | |
| opendevreview | Merged opendev/system-config master: Build Gerrit on top of our python-base images https://review.opendev.org/c/opendev/system-config/+/870874 | 00:18 |
|---|---|---|
| clarkb | exciting | 00:19 |
| clarkb | Not sure what the plan for restarting gerrit is to pick that up. But I should be able to get to it tomorrow | 00:19 |
| opendevreview | Ian Wienand proposed openstack/project-config master: nodepool: infra-package-needs; cleanup python https://review.opendev.org/c/openstack/project-config/+/872476 | 00:24 |
| opendevreview | Ian Wienand proposed openstack/project-config master: nodepool: infra-package-needs; remove lvm2 https://review.opendev.org/c/openstack/project-config/+/872477 | 00:24 |
| opendevreview | Ian Wienand proposed openstack/project-config master: nodepool: infra-package-needs; cleanup tox installs https://review.opendev.org/c/openstack/project-config/+/872478 | 00:24 |
| ianw | clarkb: i can do it in a couple of hours when it is quiet if you like | 00:29 |
| clarkb | ianw: up to you | 00:29 |
| clarkb | I'm happ for it to be done for me :) but also just wanted to let others know I can get to it tomorrow | 00:30 |
| ianw | i can restart it after my lunch | 00:38 |
| opendevreview | Ian Wienand proposed openstack/project-config master: nodepool: infra-package-needs; cleanup tox installs https://review.opendev.org/c/openstack/project-config/+/872478 | 00:46 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: zuul-jobs-test-registry-docker-* : update to jammy nodes https://review.opendev.org/c/zuul/zuul-jobs/+/872365 | 00:54 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: build-docker-image: fix change prefix https://review.opendev.org/c/zuul/zuul-jobs/+/872258 | 00:54 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: container-roles-jobs: Update tests to jammy nodes https://review.opendev.org/c/zuul/zuul-jobs/+/872375 | 00:54 |
| tonyb | Is said restart why I have *slightly* out of sync repos: https://paste.opendev.org/show/bUVHRCFuMihTuuVVnAR0/ | 01:44 |
| tonyb | my gerrit remote is ahead of origin (https) | 01:45 |
| tonyb | Also fetching from origin is very slow | 01:45 |
| Clark[m] | No the restart hasn't happened yet | 01:50 |
| Clark[m] | There is a delay in syncing from Gerrit to gitea but it should be small | 01:51 |
| Clark[m] | Is OpenDev.org synced now? If not can you check the SSL cert names for https://OpenDev.org to identify the backend? We may need to take it out of rotation and figure out what is going on with it | 01:52 |
| tonyb | Clark[m]: It still isn't in sync and I get: subject=CN = gitea08.opendev.org | 01:54 |
| Clark[m] | Looking at cacti it appears to be experiencing a large spike in traffic | 01:59 |
| Clark[m] | I half expect it we pull it that load will transfer to another node due to how haproxy balances | 01:59 |
| Clark[m] | I'm not in a good spot to debug the DoS | 01:59 |
| tonyb | Okay. At least there is somewhat of a reason. | 02:08 |
| * tonyb will wait | 02:08 | |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: linters-requirements : update Ansible to 2.12 https://review.opendev.org/c/zuul/zuul-jobs/+/872371 | 04:08 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: zuul-jobs-test-registry-docker-* : update to jammy nodes https://review.opendev.org/c/zuul/zuul-jobs/+/872365 | 04:08 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: build-docker-image: fix change prefix https://review.opendev.org/c/zuul/zuul-jobs/+/872258 | 04:08 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: container-roles-jobs: Update tests to jammy nodes https://review.opendev.org/c/zuul/zuul-jobs/+/872375 | 04:08 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: Ignore some command-instead-of-module warnings https://review.opendev.org/c/zuul/zuul-jobs/+/872489 | 04:08 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: fix a bunch of command-instead-of-shell errors https://review.opendev.org/c/zuul/zuul-jobs/+/872490 | 04:08 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: add names to blocks/includes, etc. https://review.opendev.org/c/zuul/zuul-jobs/+/872491 | 04:08 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: ignore use of mkdir https://review.opendev.org/c/zuul/zuul-jobs/+/872492 | 04:08 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: use pipefail https://review.opendev.org/c/zuul/zuul-jobs/+/872493 | 04:08 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: ignore latest git pull https://review.opendev.org/c/zuul/zuul-jobs/+/872494 | 04:08 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: uncap https://review.opendev.org/c/zuul/zuul-jobs/+/872495 | 04:08 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: linters-requirements : update Ansible to 2.12, ansible-lint <6.12.0 https://review.opendev.org/c/zuul/zuul-jobs/+/872371 | 04:09 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: Ignore some command-instead-of-module warnings https://review.opendev.org/c/zuul/zuul-jobs/+/872489 | 04:09 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: fix a bunch of command-instead-of-shell errors https://review.opendev.org/c/zuul/zuul-jobs/+/872490 | 04:09 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: add names to blocks/includes, etc. https://review.opendev.org/c/zuul/zuul-jobs/+/872491 | 04:09 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: ignore use of mkdir https://review.opendev.org/c/zuul/zuul-jobs/+/872492 | 04:09 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: use pipefail https://review.opendev.org/c/zuul/zuul-jobs/+/872493 | 04:09 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: ignore latest git pull https://review.opendev.org/c/zuul/zuul-jobs/+/872494 | 04:09 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: uncap https://review.opendev.org/c/zuul/zuul-jobs/+/872495 | 04:09 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: zuul-jobs-test-registry-docker-* : update to jammy nodes https://review.opendev.org/c/zuul/zuul-jobs/+/872365 | 04:09 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: build-docker-image: fix change prefix https://review.opendev.org/c/zuul/zuul-jobs/+/872258 | 04:09 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: container-roles-jobs: Update tests to jammy nodes https://review.opendev.org/c/zuul/zuul-jobs/+/872375 | 04:09 |
| *** thuvh1 is now known as thuvh | 04:19 | |
| ianw | #status log restarted gerrit @ https://review.opendev.org/c/opendev/system-config/+/870874 | 05:29 |
| opendevstatus | ianw: finished logging | 05:29 |
| *** yadnesh|away is now known as yadnesh | 05:51 | |
| *** pojadhav is now known as pojadhav|ruck | 07:58 | |
| *** jpena|off is now known as jpena | 08:22 | |
| *** rlandy|out is now known as rlandy | 11:17 | |
| *** dviroel|out is now known as dviroel|rover | 11:37 | |
| *** ysandeep is now known as ysandeep|afk | 12:36 | |
| mnasiadka | has anything changed in regards to apparmor in debian/ubuntu opendev images? We (Kolla) started seeing some failures with missing apparmor_parser binary | 12:37 |
| fungi | mnasiadka: i don't really know much about apparmor, but poking around on some random debian and ubuntu machines i find there's a /usr/sbin/apparmor_parser executable present which isn't owned by any installed package, so i have no clue where it comes from | 12:55 |
| fungi | and so no idea how to install or uninstall it, therefore no idea what might change to result in it disappearing | 12:56 |
| fungi | this will need a bit more research | 12:56 |
| fungi | mnasiadka: is it possible the file is actually there but for some reason /usr/sbin isn't in the $PATH for the user you're running some command as? | 12:58 |
| mnasiadka | It's docker that is complaining it can't find the profile / apparmor_parser | 12:59 |
| mnasiadka | https://6429f102495b6dfc66e8-e0b3841327e693df7529796034dab315.ssl.cf5.rackcdn.com/872526/1/check/kolla-build-debian/58e8a3e/kolla/build/000_FAILED_base.log | 12:59 |
| mnasiadka | so yes, it's missing in $PATH | 12:59 |
| fungi | okay, so maybe docker's execution path has changed | 12:59 |
| mnasiadka | Feb 02 11:10:17 np0032948303 dockerd[2614]: time="2023-02-02T11:10:17.759641169Z" level=error msg="AppArmor enabled on system but the docker-default profile could not be loaded: running `apparmor_parser apparmor_parser --version` failed with output: \nerror: exec: \"apparmor_parser\": executable file not found in $PATH" | 13:00 |
| mnasiadka | hmm, docker 23.0.0 | 13:01 |
| fungi | is that new? | 13:01 |
| mnasiadka | probably is | 13:02 |
| fungi | maybe it used to run as root but now starts as a non-root user which lacks the sbin dirs in PATH? | 13:02 |
| mnasiadka | "Unpacking docker-ce (5:23.0.0-1~debian.11~bullseye) ...", | 13:05 |
| mnasiadka | I'm pretty sure that's some new version - I'll pin for now | 13:05 |
| mnasiadka | thanks fungi :) | 13:13 |
| fungi | mnasiadka: i guess you're getting that directly from moby? https://packages.debian.org/docker-ce says it's not part of debian | 13:14 |
| fungi | i suppose they're a lot more lax about introducing backwards incompatibilities than debian is | 13:14 |
| mnasiadka | fungi: directly from download.docker.com - what is funny 23.0 is for deb/ubuntu, but not for centos ;) | 13:19 |
| fungi | according to their github, there's no official 23.0.0 release yet, just release candidates (rc4 yesterday) | 13:19 |
| fungi | so i guess they decided to guinea pig the debian/ubuntu users | 13:20 |
| fungi | centos seems to carry 23.0.0 packages directly in a "test" tree | 13:21 |
| fungi | ahh, no, that's docker.com | 13:21 |
| fungi | mnasiadka: https://download.docker.com/linux/centos/9/x86_64/test/Packages/ | 13:21 |
| fungi | docker-ce-23.0.0-1.el9.x86_64.rpm | 13:22 |
| fungi | 2022-12-07 00:33:24 | 13:23 |
| fungi | er, copied from the wrong line, 2023-02-02 10:27:04 | 13:23 |
| fungi | so yes, that does seem very new, like only a few hours ago new | 13:23 |
| fungi | infra-root: heads up, docker 23.0.0 is out as of the past few hours, and seems to possibly be breaking stuff | 13:24 |
| Clark[m] | (I have no idea why I'm awake this early) fungi: it looks like the apparmor package normally provides /sbin/apparmor_parser. Maybe the /usr/sbin path is from debootstrap because the kernel hass apparmor enabled? mnasiadka I half expect that if you install the apparmor package this would work. We should also investigate if we should install that package on our base images | 13:49 |
| Clark[m] | fungi: also we install docker-ce from docker in our control plane but don't auto update and should have apparmor installed iirc. Something to be aware of but I don't expect 23.0.0 is causing problems for us there yet | 13:50 |
| mnasiadka | looking at the installation it's adding apparmor as a dependency, I'll have a look on a clean VM to see if it fails the same way | 13:51 |
| Clark[m] | The dib build logs show apparmor is suggested due to something else we install but we don't install suggested to keep image size down. But we can explicitly add it to the list if it works around a half baked apparmor setup | 13:52 |
| *** dasm|off is now known as dasm | 13:53 | |
| mnasiadka | I think we can sort it out in Kolla CI, no need to bloat the image for everybody ;) | 13:56 |
| Clark[m] | Well it might be a real bug if we have apparmor enabled kernels but no user space controls. Not completely sure of this yet though | 13:58 |
| fungi | Clark[m]: you're right, /sbin/apparmor_parser is coming from the apparmor package, but /usr/sbin was earlier in my path and i was being confused by the new "merged-usr" in debian (which has been typical on ubuntu for a while) | 14:09 |
| fungi | lrwxrwxrwx 1 root root 8 Oct 12 14:19 /sbin -> usr/sbin | 14:09 |
| fungi | that only just changed in debian/unstable officially in the past month or two | 14:10 |
| fungi | for existing systems i mean (automated migration to merged-usr) | 14:11 |
| fungi | newly installed systems were getting it for a while before that | 14:11 |
| fungi | anyway, i suspect the problem is docker-ce either started depending on apparmor poorly, or has depended on apparmor for a while but no longer has sbin in path for some reason | 14:12 |
| Clark[m] | They don't appear to have updated release notes yet unfortunately https://docs.docker.com/engine/release-notes/ | 14:21 |
| mnasiadka | yeah, dump a new version on users and don't update release notes ;-) | 14:27 |
| fungi | well, it's the "community edition" so i'm sure they see it as an incentive for users to pay for enterprise if they want exclusive features like not-be-completely-broken and actual-release-notes | 14:36 |
| mnasiadka | fungi, Clark[m]: installing apparmor fixes the problem indeed, it's not installed (although I would expect docker depends on it on Debian/Ubuntu...) | 15:04 |
| fungi | how interesting | 15:05 |
| Clark[m] | Ya we should probably investigate if we should preinstall it since the kernel appears to have it enabled | 15:14 |
| fungi | it's probably better for jobs to install it if they need it, for thoroughness, but i can see it both ways | 15:18 |
| Clark[m] | Ya I could see it both ways too. Seems like we caught an error in the docker package if it doesn't pull in apparmor for example | 15:21 |
| *** ysandeep|afk is now known as ysandeep|out | 15:27 | |
| *** yadnesh is now known as yadnesh|away | 15:51 | |
| clarkb | mnasiadka: would be curious to hear if docker works otherwise with the update. Just beause we run it too and are likely to run into similar problems | 16:16 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Install apparmor when installing upstream docker-ce https://review.opendev.org/c/zuul/zuul-jobs/+/872569 | 16:48 |
| opendevreview | Rodolfo Alonso proposed openstack/project-config master: Set "main" as default branch of "sqlalchemy/sqlalchemy" https://review.opendev.org/c/openstack/project-config/+/872591 | 17:08 |
| clarkb | zuul is discovering the docker 23 issues now too. | 17:17 |
| clarkb | fungi: I think it may be a half decent idea for us (opendev) to put a pin in system-config | 17:17 |
| clarkb | fungi: since there appear to be a number of incompatibilities. | 17:17 |
| fungi | i don't object to that | 17:17 |
| clarkb | mostly thinking sort out the issues in the CI system (what I'm digging into now) but pin our prod hosts in the meantime. THat said risk is low to us since auto updates don't autoupdate docker | 17:21 |
| clarkb | also the incompatibilities with apparmor shouldn't hit us as I believe we have apparmor installed and buildx isn't something we run in prod | 17:21 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Fix ensure-docker for docker-ce 23.0.0 upstream packaging https://review.opendev.org/c/zuul/zuul-jobs/+/872569 | 17:27 |
| clarkb | fungi: ya looking at system-config/playbooks/roles/install-docker/tasks/upstream.yaml we merely ensure present. And checking on some servers with dpkg -l we seem to be on 20 still | 17:30 |
| clarkb | the reason for this is upgrading docker restarts containers and we don't want that happening automatically | 17:30 |
| clarkb | so I think we're good until we deploy a new server and can focus on the CI side? But help with a pin wouldn't be the worst thing | 17:31 |
| *** jpena is now known as jpena|off | 17:34 | |
| opendevreview | Merged zuul/zuul-jobs master: Fix ensure-docker for docker-ce 23.0.0 upstream packaging https://review.opendev.org/c/zuul/zuul-jobs/+/872569 | 18:00 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: DNM double checking docker 23 fixes https://review.opendev.org/c/zuul/zuul-jobs/+/872594 | 18:02 |
| clarkb | mnasiadka: the apparmor thing is known: https://github.com/moby/moby/issues/44900 also buildx was split into a separate package that you need to install if you rely on buildx. zuul-jobs' ensure-docker role has just been updated to address both items | 18:14 |
| opendevreview | Maksim Malchuk proposed openstack/diskimage-builder master: Add swap support https://review.opendev.org/c/openstack/diskimage-builder/+/869270 | 19:10 |
| ianw | clarkb/fungi: if you have a sec could you look over https://review.opendev.org/q/topic:package-needs-drop-curl which is a small series to clean up some installs on images. the main motivator is removing the curl install tot get rocky working, but once i started looking the others seemed worthwhile cleaning up | 20:12 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: fix a bunch of command-instead-of-shell errors https://review.opendev.org/c/zuul/zuul-jobs/+/872490 | 20:31 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: add names to blocks/includes, etc. https://review.opendev.org/c/zuul/zuul-jobs/+/872491 | 20:31 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: ignore use of mkdir https://review.opendev.org/c/zuul/zuul-jobs/+/872492 | 20:31 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: use pipefail https://review.opendev.org/c/zuul/zuul-jobs/+/872493 | 20:31 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: ignore latest git pull https://review.opendev.org/c/zuul/zuul-jobs/+/872494 | 20:31 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: ansible-lint: uncap https://review.opendev.org/c/zuul/zuul-jobs/+/872495 | 20:31 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: zuul-jobs-test-registry-docker-* : update to jammy nodes https://review.opendev.org/c/zuul/zuul-jobs/+/872365 | 20:31 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: build-docker-image: fix change prefix https://review.opendev.org/c/zuul/zuul-jobs/+/872258 | 20:31 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: container-roles-jobs: Update tests to jammy nodes https://review.opendev.org/c/zuul/zuul-jobs/+/872375 | 20:31 |
| clarkb | ianw: I've reviewed the stack. The python cleanup one is not safe I don't think (left comments on the change) | 20:57 |
| clarkb | also some comments on the other changes but +2 for the others | 20:57 |
| ianw | clarkb: thanks. i think the difference now is that dib has ensured that python3 is installed on all images | 20:59 |
| clarkb | ianw: oh I see no matter what we do we end up with python3 but not the dev package | 21:01 |
| ianw | although what actually installs the dev package from the pkg-map ... | 21:03 |
| ianw | fedora, openeuler and suse | 21:04 |
| ianw | now i look at that, i think we probably should just drop it all together | 21:06 |
| clarkb | ubuntu is installing it | 21:06 |
| clarkb | and debian too. They just use the default name | 21:07 |
| clarkb | I think we need to be careful in particular ansible relies on python being there and we can't easily bootstrap that | 21:08 |
| clarkb | that said it doesn't rely on the dev package just the runtime | 21:08 |
| ianw | yeah, python will be there | 21:09 |
| ianw | i think we can merge these slowly, over days, so one goes in at a time | 21:10 |
| ianw | the curl one first to get rocky working | 21:10 |
| clarkb | ya that seems reasonable | 21:12 |
| ianw | afaics centos-9 doesn't install python3-devel, and i guess nobody has complained | 21:15 |
| ianw | https://nb01.opendev.org/centos-9-stream-0000010704.log | 21:15 |
| ianw | it's in bindep fallback ... although what's using that these days | 21:17 |
| fungi | i think we stopped actually falling back on that? if we didn't, probably should plan to sunset it | 21:34 |
| clarkb | ozj has a couple of legacy things that still refer to it at least | 21:35 |
| fungi | also things we can hopefully sunset | 21:38 |
| ianw | yeah i see "xenial" in there at a brief look | 21:48 |
| ianw | and "rocky" | 21:48 |
| opendevreview | Merged openstack/project-config master: nodepool: infra-package-needs; drop curl https://review.opendev.org/c/openstack/project-config/+/872473 | 22:08 |
| ianw | i'll monitor rocky (the other rocky, the distro :) | 22:19 |
| fungi | rocky xxiii: rocky vs rocky | 22:32 |
| *** dviroel|rover is now known as dviroel|rout | 22:38 | |
| *** dasm is now known as dasm|off | 22:42 | |
| *** rlandy is now known as rlandy|out | 22:57 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!