| * dpawlik reading back | 05:13 | |
| *** amoralej|off is now known as amoralej | 07:19 | |
| *** amoralej is now known as amoralej|lunch | 10:59 | |
| slittle1 | Please add me as first core in starlingx-app-node-feature-discovery-core. Thanks! | 12:20 |
|---|---|---|
| fungi | slittle1: done | 12:23 |
| *** amoralej|lunch is now known as amoralej | 12:24 | |
| *** gboutry[m] is now known as gboutry | 13:12 | |
| *** d34dh0r5| is now known as d34dh0r53 | 13:35 | |
| fungi | apparently this is today's security clickbait: https://www.interruptlabs.co.uk//articles/linux-ipv6-route-of-death | 14:33 |
| clarkb | https://20a1c0106742d2f14911-911ceff0bc5fb361e3ec036d1e694e86.ssl.cf1.rackcdn.com/883312/3/check/system-config-run-gitea/19cd88c/bridge99.opendev.org/ara-report/results/480.html this shows another podman and docker difference. podman-compose won't let you down containers that are not running | 15:18 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM testing podman as container manager for services https://review.opendev.org/c/opendev/system-config/+/883312 | 15:21 |
| clarkb | I'm going to put holds on the jobs for ^ to start working through the behavior differences | 15:21 |
| clarkb | after cleaning up the unused fedora mirror content we are below 90% AFS capacity again \o/ | 15:38 |
| fungi | excellent! | 15:39 |
| clarkb | oh maybe the errors in the podman compose output are ignored? This is all very confusing. It is writing errors to stderr but then returning 0 | 15:43 |
| clarkb | I think this is the actual issue "Error: error running container create option: invalid log driver: invalid argument", | 15:45 |
| clarkb | which is a difference but not the one I thought I was seeing | 15:45 |
| clarkb | https://github.com/containers/podman/issues/12537 this seems to confirm it is a difference and one that is not implemented | 15:47 |
| fungi | i suppose that can be worked around, at least | 15:54 |
| fungi | though i'll miss being able to just look in syslog | 15:54 |
| fungi | not the end of the world | 15:54 |
| clarkb | I'm going to push a patchset that swaps it to journald to start. Theoretically we can have journald write out to disk like syslog was. But waiting for nodes to be held first as I think that will be helpful for additional debugging | 15:57 |
| clarkb | running `podman-compose pull` again with no image updtes doesn't seem to produce output that is different in a meaningful way compared to the first pull. So you can't really tell if you are pulling new images or not :( this is currently used by the gitea role to determine if it needs to do a graceful service restart | 16:06 |
| clarkb | we may need to do an image list, pull, then image list again and compare | 16:08 |
| fungi | that's probably more thorough anyway | 16:20 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Modernize install-podman https://review.opendev.org/c/opendev/system-config/+/883311 | 16:22 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM testing podman as container manager for services https://review.opendev.org/c/opendev/system-config/+/883312 | 16:22 |
| *** amoralej is now known as amoralej|off | 16:41 | |
| clarkb | https://github.com/containers/podman/issues/13200 is going to be an issue with using journald. Not a catastrophic one just more difficult to use journald this way | 17:00 |
| clarkb | I think this is the first real issue that we can't just easily workaround but we can also probably live with it | 17:01 |
| corvus | how are we using journald with docker? | 17:08 |
| clarkb | corvus: we are using syslog with docker which podman does not support. The podman suggested alternative is journald, but tagging does not work with journal whihc makes it more difficult to use (either via journalctl to see logs from a specific container or via journald + syslog integration to write out specific log files to disk) | 17:09 |
| clarkb | corvus: the syslog + docker integration is so that we get log files for each container which is nice for persisting and rotating logs as default docker logging is one giant buffer and it gets deleted when you replace a container | 17:10 |
| clarkb | in thise case I think we will just have to accept the deficiencies with the podman journald support and learn to filter other ways | 17:10 |
| corvus | ugh. i guess "conmon" is the thing to grep for to get logs from (any) container | 17:14 |
| corvus | since most of our systems are 1vm:1container, that's probably barely workable | 17:14 |
| clarkb | ya and it appears to have the pid there too | 17:15 |
| clarkb | so you can filter by conmod[$PID] or something | 17:15 |
| fungi | not great but i think we can work with that | 17:16 |
| fungi | maybe it will improve in time | 17:16 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Switch from deprecated require-approval to require https://review.opendev.org/c/openstack/project-config/+/883431 | 17:40 |
| fungi | we have a number of other tenants in the same situation as ^ | 17:41 |
| NeilHanlon | heya folks, wondering what's involved in adding rocky to the infra mirrors. we have had some test failures due to outdated mirrors that I think would be good to try and squash, or at least control | 17:47 |
| NeilHanlon | s/rocky/rocky linux/ | 17:48 |
| fungi | NeilHanlon: aside from space requirements (we've been talking about finding things to purge to keep from adding more backing volumes and making it more fragile) we'd need a public rsync server to pull from | 17:49 |
| fungi | also note that our mirrors are not immune to the sorts of issues you describe, we see them all the time because whatever mirror we're pulling updates from has gone stale | 17:50 |
| NeilHanlon | sure, sure, that makes sense. at least for the first part, I can provide access to our mirrors via rsync and make sure there's a slot available. we've been meaning to setup a better tiering system for a while, anyways.. | 17:52 |
| NeilHanlon | i can get some estimations of how much space it'd take up, though it's probably similar to CentOS | 17:52 |
| fungi | usually we don't pull from the primary sources of other distros because they require that anyone doing so become an advertised part of their public mirror network, and we can't make appropriate guarantees for accessibility (nor have we okayed the likely increases in bandwidth use with our infrastructure donors) | 17:53 |
| fungi | so we tend to pull from tier-2 mirrors, which as you've also observed have a tendency to suffer updating issues from time to time | 17:54 |
| fungi | also any sort of client filtering by address would likely cause issues if we need to rebuild our mirror-update server, since we'd need to get the new addresses added to the primary's filter | 17:55 |
| NeilHanlon | i understand, that makes sense | 17:57 |
| NeilHanlon | for what it's worth, we don't have any such policy at this time. we're not 100% sure whether we will or not, though, in the future | 17:58 |
| corvus | i'm going to experimentally restart one of the executors; it may restart some builds. | 18:57 |
| corvus | i love how docker-compose pull just keeps counting up from 0% to 100% as many times as needed :) | 18:59 |
| corvus | #status log hard restarted ze01 | 19:00 |
| opendevstatus | corvus: finished logging | 19:00 |
| fungi | thanks! | 19:00 |
| fungi | and yes, dockermath is probably a thing unto itself | 19:00 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Switch from deprecated require-approval to require https://review.opendev.org/c/openstack/project-config/+/883431 | 19:15 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM testing podman as container manager for services https://review.opendev.org/c/opendev/system-config/+/883312 | 19:31 |
| clarkb | I think this is getting close. nodepool-builder doesn't actually talk to a zookeeper so its all unhappy at runtime. We'll need to hold the host and then probably spin all that up using the nodepool test tools | 19:31 |
| clarkb | I'm going to work on gerrit before doing that though since it should be more straightforward | 19:36 |
| *** elodilles is now known as elodilles_ooo | 19:42 | |
| corvus | I'm doing a hard restart of the rest of the zuul executors; this will restart jobs | 19:53 |
| corvus | #status log restarted remaining zuul executors | 19:53 |
| fungi | thanks! | 19:53 |
| opendevstatus | corvus: finished logging | 19:53 |
| corvus | looks like we have like 130 nodes perpetually deleting :/ | 19:57 |
| clarkb | are they all in rax-iad? | 19:58 |
| clarkb | I think it struggles | 19:58 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM testing podman as container manager for services https://review.opendev.org/c/opendev/system-config/+/883312 | 21:27 |
| clarkb | minor podman difference: it doesn't respect docker.io as the unqualified domain. Not a bad difference but ^ addresses that in one case | 21:37 |
| clarkb | I'm realizing too that we will need to sort out a transition from docker to podman somehow | 22:12 |
| clarkb | the way the change is written above it jus assumes docker was never there which won't be the case in production. I'll have to think on that | 22:13 |
| fungi | as in cleanly uninstalling docker before installing podman? | 22:16 |
| fungi | we're installing from deb packages, so hopefully a purge will be thorough | 22:16 |
| fungi | if disruptive | 22:16 |
| fungi | but... also those debs were probably not up to debian (or even ubuntu) standards, so are quite likely to leave trash behind | 22:17 |
| clarkb | fungi: more stopping services under docker and restarting under podman | 22:18 |
| clarkb | we can cleanup docker separately but services will conflict with each other (ports, bind mounts, etc) otherwise | 22:18 |
| fungi | got it | 22:24 |
| *** dmellado95 is now known as dmellado9 | 23:04 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!