| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: [dnm] trying fedora 35 https://review.opendev.org/c/openstack/diskimage-builder/+/815574 | 01:03 |
|---|---|---|
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: [dnm] trying fedora 35 https://review.opendev.org/c/openstack/diskimage-builder/+/815574 | 02:02 |
| *** pojadhav|out is now known as pojadhav|ruck | 02:46 | |
| *** ysandeep|out is now known as ysandeep | 05:06 | |
| *** ykarel|away is now known as ykarel | 05:46 | |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: fedora-container: update to Fedora 35 https://review.opendev.org/c/openstack/diskimage-builder/+/815574 | 05:48 |
| opendevreview | Merged openstack/diskimage-builder master: dracut-regenerate: drop Python 2 packages https://review.opendev.org/c/openstack/diskimage-builder/+/815409 | 05:55 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: fedora-container: regenerate initramfs for F34 https://review.opendev.org/c/openstack/diskimage-builder/+/815385 | 05:55 |
| *** jpena|off is now known as jpena | 06:54 | |
| *** ysandeep is now known as ysandeep|lunch | 07:48 | |
| *** pojadhav is now known as pojadhav|ruck | 07:51 | |
| *** ykarel is now known as ykarel|lunch | 08:34 | |
| esimone | infra-root: Can I ask what are the next steps to evaluate the insertion of a new project in the X namespace? https://review.opendev.org/c/openstack/project-config/+/795118 | 08:40 |
| *** ysandeep|lunch is now known as ysandeep | 08:45 | |
| frickler | esimone: that project was created in airship namespace, do you want to move it to x/? otherwise it may just be missing a follow-up patch to set up job definitions | 09:24 |
| esimone | frickler: yes, if it possible I prefer the x/ namespace. It is not part of nor use airship philosophy. | 09:35 |
| frickler | esimone: hmm, we have https://docs.opendev.org/opendev/infra-manual/latest/creators.html#project-renames but that might be openstack-specific, I'll need to check with other admins later | 09:39 |
| esimone | frickler: thank you. You can consider siss related to OpenStack as Kayobe do, but simpler and linked to Ceph. | 09:44 |
| *** ysandeep is now known as ysandeep|afk | 09:46 | |
| frickler | esimone: oh, you were talking about https://review.opendev.org/c/openstack/project-config/+/814890 ? the link you posted was for airship/image-builder | 09:57 |
| *** ykarel|lunch is now known as ykarel | 10:04 | |
| esimone | Please, frickler: Yes... please, forgive my big mistake. A wrong cut&paste. I'm so sorry. | 10:05 |
| frickler | esimone: no problem, that patch looks good to me, should hopefully get another review and approval soon | 10:09 |
| esimone | frickler: thanks | 10:13 |
| *** ysandeep|afk is now known as ysandeep | 10:37 | |
| *** dviroel|rover|afk is now known as dviroel|rover | 10:50 | |
| *** ykarel is now known as ykarel|afk | 11:15 | |
| *** jpena is now known as jpena|lunch | 11:23 | |
| *** ysandeep is now known as ysandeep|mtg | 11:35 | |
| *** ysandeep|mtg is now known as ysandeep | 11:51 | |
| *** jpena|lunch is now known as jpena | 12:22 | |
| *** ykarel|afk is now known as ykarel | 13:19 | |
| opendevreview | Merged openstack/project-config master: Add siss under x namespace https://review.opendev.org/c/openstack/project-config/+/814890 | 13:32 |
| clarkb | Not really here yet, but latest update on the gerrit + mina updates is that MINA 2.7.1 isn't actually ever going to happen and is just a placeholder for 2.8.0. Not sure why they don't just refer to 2.8.0 in that case. Also "using 2.8.0 in Gerrit is not a piece of cake because this Apache MINA sshd project unfortunately breaks API between minor releases regularly." and they have to do | 14:15 |
| clarkb | extensive testing with jgit apparently | 14:15 |
| clarkb | I'm staying out of it but can't help but wonder if this would be easier if some backward compat was maintained | 14:16 |
| fungi | or maybe there's a use for your backported implementation after all | 14:17 |
| clarkb | ya that was also mentioend in the email. I could resurrect that and base it off of the actual java updates rather than my poor attempt at a kotlin translation | 14:18 |
| *** ricolin_ is now known as ricolin | 14:40 | |
| *** ykarel is now known as ykarel|away | 14:51 | |
| *** ysandeep is now known as ysandeep|mtg | 15:03 | |
| *** pojadhav|ruck is now known as pojadhav|dinner | 15:08 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: Upgrade to gitea 1.15.6 https://review.opendev.org/c/opendev/system-config/+/815860 | 15:40 |
| clarkb | infra-root fyi ^ new gitea release today | 15:40 |
| fungi | they sure are churning those out at a steady clip lately | 15:41 |
| clarkb | debian is killing which. Shell scripts everywhere panic | 15:48 |
| *** ysandeep|mtg is now known as ysandeep | 15:49 | |
| fungi | not killing it, but you'll need to explicitly choose which implementation of which you want installed if you intend to keep using it, the one which was installed by default is going away | 15:50 |
| clarkb | right but the million and one packages that expect which to exist won't know anything about that and fail when which isn't explicitly installed | 15:50 |
| clarkb | s/packages/packages and scripts/ | 15:50 |
| fungi | the discussion highlights that there is no "standard" implementation of the `which` utility, every unix derivative rolled their own slightly different versions, and the maintainers of the one debian had been installing are tired of having to maintain their own | 15:51 |
| clarkb | debians own package maintainer documentation apparently said use which until 2018 | 15:51 |
| clarkb | I'm not saying its wrong to try and use a better standard (command -v) but if you said "use this tool" for years nad years then ripping it out is going to be a problem | 15:52 |
| fungi | oh, for sure | 15:52 |
| fungi | i didn't mean to imply it's not going to be a painful transition | 15:52 |
| fungi | it's just also not a decision which was taken lightly | 15:53 |
| fungi | discussion around it has been ongoing for months on the debian-devel ml | 15:53 |
| *** ysandeep is now known as ysandeep|out | 15:53 | |
| clarkb | ya it just surprises me when changes like that are made. I much prefer the kernel's approach | 15:53 |
| clarkb | that was the standard interface according to the docs. I'm sorry but that mens you own it now | 15:54 |
| clarkb | I get it isn't ideal but... | 15:54 |
| clarkb | we've got a bunch of which use that will need to be updated | 15:55 |
| clarkb | it also makes what the kernel manages to do that much more impressive | 15:57 |
| fungi | i have a feeling bookworm will release with some implementation of which as part of the standard set, if not essential | 16:03 |
| *** jpena is now known as jpena|off | 16:05 | |
| *** marios is now known as marios|out | 16:14 | |
| esimone | infra-root: sorry, who to ask to be added to the siss-core and siss-release Gerrit groups (x/siss has just been created)? | 16:20 |
| fungi | esimone: sure, i'll add you now, give me a moment | 16:22 |
| fungi | esimone: done | 16:25 |
| esimone | fungi: many thanks | 16:32 |
| *** pojadhav|dinner is now known as pojadhav|out | 16:43 | |
| opendevreview | Gonéri Le Bouder proposed zuul/zuul-jobs master: use-buildset-registry: remove useless import https://review.opendev.org/c/zuul/zuul-jobs/+/815866 | 16:51 |
| yuriys | clarkb; fungi; ive installed the self signed on the s3/swift service, so you should be all set to use that (may need to copy the ca crt) to your s3 client, etc. | 17:01 |
| clarkb | yuriys: cool, thanks | 17:02 |
| Goneri | The problem above prevents use from using the role on Fedora 34 (py39). The line does not make a lot of sense and it looks like a mistake was introduce during the import of the module. | 17:03 |
| clarkb | Goneri: note that zuul-jobs is primarily discussed on matrix in the #zuul:opendev.org room | 17:03 |
| Goneri | oh, sorry. | 17:03 |
| opendevreview | Merged zuul/zuul-jobs master: use-buildset-registry: remove useless import https://review.opendev.org/c/zuul/zuul-jobs/+/815866 | 17:36 |
| *** sshnaidm is now known as sshnaidm|afk | 17:50 | |
| clarkb | fungi: do we want to wait for ianw to start the day before approving https://review.opendev.org/c/opendev/system-config/+/815860 or just go for it? I can help keep an eye on it for much of today at least | 18:11 |
| clarkb | I think the ovh bhs1 mirror is sad | 18:11 |
| clarkb | it is returning errors for pypi proxying and npm proxying | 18:12 |
| fungi | you looking into it or shall i? | 18:12 |
| clarkb | but testing it from here eg https://mirror.bhs1.ovh.opendev.org/pypi/simple works | 18:12 |
| clarkb | fungi: I'm starting to look at it but extra eyeballs would be good since the evidence I'm seeing shows it is fine | 18:12 |
| clarkb | but https://zuul.opendev.org/t/zuul/build/c4e8e957cfc246adb6807d73c20249f6 and https://zuul.opendev.org/t/zuul/build/aca338f263f847ebbf7cbda8e07181eb disagree with me | 18:13 |
| clarkb | ok I was able to reproduce with a download of the npmjs package in that second link. retrying succeeded | 18:14 |
| clarkb | apache's error log complains of segmentation faults in children | 18:15 |
| fungi | /dev/mapper/main-apache 98G 98G 64K 100% /var/cache/apache2 | 18:15 |
| clarkb | we are using almost no memory there, no OOMs, and the most recent afs sad was almost a week ago | 18:15 |
| clarkb | aha that would explain it | 18:16 |
| fungi | check df | 18:16 |
| clarkb | an htcacheclean has been running since 18:00 | 18:16 |
| fungi | this has happened there before, i think because of poor i/o performance causing htcacheclean to take forever to complete, and then we end up with multiple runs fighting one another | 18:16 |
| clarkb | I only see one run currently fwiw | 18:17 |
| fungi | www-data 19795 0.0 0.0 19912 424 ? Ss Oct08 0:30 /usr/bin/htcacheclean -d 120 -p /var/cache/apache2/mod_cache_disk -l 300M -n | 18:17 |
| clarkb | thats a different unused cache | 18:17 |
| fungi | ahh, okay | 18:17 |
| clarkb | (the default that apache ships with) | 18:17 |
| fungi | /var/cache/apache2/proxy is the one we're using i guess | 18:17 |
| clarkb | yes | 18:18 |
| clarkb | do we disable the provider and give htcacheclean an opportunity to catch up? | 18:18 |
| fungi | looks like htcacheclean runs out of /etc/cron.daily/apache2 | 18:18 |
| clarkb | I suppose the more forceful option is to stop services, delete the cache and start over | 18:19 |
| clarkb | fungi: it runs out of roots normal crontab | 18:19 |
| clarkb | at the start of ever hour | 18:19 |
| fungi | aha, so it does, hourly there | 18:19 |
| fungi | yeah, so /etc/cron.daily/apache2 is probably what's responsible for the run with /var/cache/apache2/mod_cache_disk | 18:20 |
| clarkb | yes I think so | 18:20 |
| corvus | buffer 19 | 18:20 |
| fungi | looks like we don't log our htcacheclean so hard to know how long it's taking | 18:20 |
| clarkb | fungi: I don't think it really logs anything either | 18:21 |
| fungi | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=68218&rra_id=all suggests it filled up around 17:20 | 18:22 |
| clarkb | I wonder if this coincides with kolla or loci or tripleo publishin a new set of docker images | 18:22 |
| clarkb | I'm working on a change to disable bhs1 in nodepool but I think we need to manually apply it to reliably land anything right now | 18:23 |
| corvus | what about rm -fr /var/cache/apache2/modcachedisk ? | 18:23 |
| fungi | looks like it normally hovers at the htcacheclean limit, we may want to consider lowering that given it can easily eat that much cache in less than the period between htcacheclean runs | 18:24 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Disable OVH BHS1 due to a sad mirror node https://review.opendev.org/c/openstack/project-config/+/815896 | 18:24 |
| corvus | it's just a cache right? restarting from a cold cache seems like a fine thing to do | 18:24 |
| fungi | corvus: i think we'll want to temporarily disable builds there either way. a recursive delete is likely to take a long time to complete and we wouldn't want to do that with apache actively serving from there | 18:24 |
| clarkb | corvus: yes I think that is safe from the mirror's perspective but unknown how long it will take | 18:25 |
| corvus | stop apache; mv directory; rm directory in background; wait a minute; start apache | 18:26 |
| fungi | ahh, yeah i guess the mv of /var/cache/apache2/proxy would work since /var/cache/apache2 is what we mount | 18:26 |
| clarkb | I've manually edited max-servers for the region on nl04 | 18:26 |
| fungi | i'll do corvus's suggestion | 18:27 |
| clarkb | fungi: thanks. Let me know when you think we're good to reset max-servers and I can do that | 18:27 |
| corvus | that should make the removal atomic from apache's perspective, and as long as we give it a few minutes head start, apache should have headroom while the background rm runs | 18:27 |
| clarkb | (and we can avoid landing that change I guess) | 18:27 |
| corvus | cool | 18:27 |
| corvus | fungi: oh, don't forget to throw in a mkdir/chmod/chown of the proxy dir to match the moved one | 18:27 |
| fungi | yeah, i'm killing the htcacheclean processes too since i'm moving the directory out from under them | 18:28 |
| corvus | ++ | 18:28 |
| fungi | okay, it's back up and running with the old cachedir in the process of deleting in a root screen session | 18:30 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Reduce htcachclean limit on our mirrors https://review.opendev.org/c/opendev/system-config/+/815897 | 18:30 |
| fungi | please test | 18:30 |
| clarkb | Something like ^ for lowering the limit | 18:30 |
| clarkb | https://mirror.bhs1.ovh.opendev.org/pypi/simple/ loads for me | 18:31 |
| corvus | ++ | 18:31 |
| fungi | i seem to be able to pull cached contents through it, yeah | 18:31 |
| clarkb | it made my firefox sad though | 18:31 |
| fungi | er, proxied contents i mean | 18:31 |
| clarkb | https://mirror.bhs1.ovh.opendev.org:4443/registry.npmjs/@patternfly/react-core/-/react-core-4.152.4.tgz also seems to be working from here | 18:32 |
| fungi | deletion seems to be progressing quickly | 18:32 |
| clarkb | I'll abandon my project-config change since it appears unnecessary | 18:32 |
| clarkb | fungi: is the rm done? | 18:36 |
| clarkb | and if so should I reset max-servers on nl04? | 18:36 |
| fungi | yeah, seems like it just finished moments ago | 18:36 |
| clarkb | I'll go ahead and restore max-servers now then | 18:37 |
| fungi | which tends to suggest we're caching fewer, larger files rather than tons of teensy ones | 18:37 |
| fungi | i wonder if we could get by running htcacheclean more frequently | 18:37 |
| clarkb | fungi: I think we implemented the lock file around it because it sometimes takes longer tha nthe hour already | 18:38 |
| fungi | would need to watch how fast it's able to traverse the cache though | 18:38 |
| fungi | yeah | 18:38 |
| clarkb | and yes fewer larger files is what I would expect from kolla/tripleo/loci types doing a massive update to their docker images | 18:38 |
| clarkb | I think all told their images come out in the 10-20GB range | 18:38 |
| clarkb | and if we're carrying the old version and the new version we can quickly bump up usage there | 18:39 |
| clarkb | (I wish that apache's hashing method was decipherable to make it easier to figure this stuff out) | 18:39 |
| clarkb | I think if we want to debug in more depth in the future we need to use du on directories then pick the largest 5% or so and look for strings in their contents | 18:39 |
| clarkb | should we do something like #status notice mirror.bhs1.ovh.opendev.org filled its disk. We have corrected this issue and jobs that failed due to this mirror can now be rechecked. | 18:41 |
| fungi | yeah, that looks good | 18:41 |
| fungi | maybe include the times? | 18:41 |
| fungi | looks like errors would have started around or shortly after 17:25 and stopped around 18:25 | 18:42 |
| clarkb | updated to #status notice mirror.bhs1.ovh.opendev.org filled its disk around 17:25 UTC. We have corrected this issue around 18:25 UTC and jobs that failed due to this mirror can now be rechecked. | 18:42 |
| fungi | lgtm! | 18:42 |
| clarkb | #status notice mirror.bhs1.ovh.opendev.org filled its disk around 17:25 UTC. We have corrected this issue around 18:25 UTC and jobs that failed due to this mirror can be rechecked. | 18:43 |
| opendevstatus | clarkb: sending notice | 18:43 |
| -opendevstatus- NOTICE: mirror.bhs1.ovh.opendev.org filled its disk around 17:25 UTC. We have corrected this issue around 18:25 UTC and jobs that failed due to this mirror can be rechecked. | 18:43 | |
| clarkb | now back to thinking about gitea. Do we want to go ahead and proceed with that or wait for ianw's day to start and give a proper second review? | 18:46 |
| clarkb | None of the bugs in the changelog look super important for us. I mostly want to keep up to avoid falling behind. Keep the delta minimal and all that | 18:47 |
| fungi | yes, i'm good going ahead with the gitea upgrade | 18:50 |
| fungi | happy to approve it now | 18:50 |
| clarkb | cool I'll be grabbing lunch soon but that is a good way to spend time while we wait on zuul to test it :) | 18:51 |
| fungi | indeed | 18:56 |
| opendevreview | Gonéri Le Bouder proposed zuul/zuul-jobs master: enable-fips: Fedora also support FIPS https://review.opendev.org/c/zuul/zuul-jobs/+/815901 | 19:04 |
| opendevreview | Gonéri Le Bouder proposed zuul/zuul-jobs master: enable-fips: Fedora also support FIPS https://review.opendev.org/c/zuul/zuul-jobs/+/815901 | 19:21 |
| opendevreview | Merged opendev/system-config master: Upgrade to gitea 1.15.6 https://review.opendev.org/c/opendev/system-config/+/815860 | 20:11 |
| fungi | now the wait for deploy | 20:13 |
| clarkb | nb03 is unable to run docker-compose up right now and that is causing service-nodepool to fail. Disks seem fine but the server has fairly high system load and the service isn't actually running | 20:31 |
| clarkb | I think this may be a problem with docker itself. I'm trying to stop the running container which isn't actually running anyp rocesses from what i can see | 20:31 |
| clarkb | ok same error trying to stop as we get from up | 20:34 |
| clarkb | I'm trying to stop and start docker itself now | 20:34 |
| corvus | load average is stuck at 4 while idle; that's not a great sign | 20:35 |
| clarkb | ya | 20:36 |
| clarkb | docker did eventually stop. I'm trying to start it now so I can cleanly stop any docker things that might have hung around | 20:37 |
| clarkb | but I agree I think there is a bigger issue and not sure what it is yet | 20:37 |
| clarkb | the gitea job just started and should get through the giteas in the next 20 minutes or so | 20:38 |
| fungi | i'll be eating right about then, but still plenty available to test/troubleshoot that | 20:42 |
| clarkb | gitea01 has upgraded. it looks happy to me | 20:42 |
| clarkb | I'll keep an eye on them but usually we expecttrouble on gitea01 as the first one to go and if it is happy the rest end up happy too | 20:43 |
| clarkb | docker still has not started on nb03 | 20:43 |
| clarkb | I have no idea where that load is coming from | 20:43 |
| corvus | i'm guessing stuck kernal io threads or something? | 20:44 |
| clarkb | there are a number of kernel tasks | 20:44 |
| fungi | there's one cpu stuck at 100% iowait | 20:45 |
| clarkb | we've got 8 processors and 25% iowait. Would that actually be two cpus? But ya lots of kworker threads and those are responsible for IO and the like | 20:46 |
| clarkb | I'm suspecting this might be a reboot | 20:47 |
| clarkb | and hand wave around "its arm and there are bugs" | 20:47 |
| fungi | cacti couldn't reach it for several hours starting just after utc midnight | 20:47 |
| fungi | and when it started responding again, 100% of its cpus were in use by "system" | 20:48 |
| fungi | though the stuck iowait has been happening for ~4 days according to http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=68716&rra_id=all | 20:50 |
| fungi | like maybe something got "stuck" on monday | 20:50 |
| clarkb | any objections to attempting `sudo reboot`? Not sure if there is more people want to look at | 20:52 |
| clarkb | I suppose linaro might be interested in it? | 20:52 |
| clarkb | assuming this is some issue with arm64 + linux + docker + something that might be of interest to them | 20:52 |
| fungi | oh, right, this is the arm builder? | 20:52 |
| clarkb | yes | 20:52 |
| fungi | yeah, couldn't hurt to see if they want to do some digging there | 20:52 |
| fungi | maybe it's something pathological with the hypervisor | 20:53 |
| clarkb | kevinz: ^ hey if you are around today our nb03.opendev.org instance in the linaro cloud is exhibiting some sort of problem with high idle system load. We think our next step is to reboot the server but thought you might be interested in details potentially if this represents an issue with arm64 on linux with docker and whatever else we might trip over | 20:59 |
| clarkb | kevinz: is there any information you would like us to collect from the VM and share? | 20:59 |
| clarkb | gitea01-08 appear to have all upgraded. I don't notice any immediate problems | 21:00 |
| fungi | yeah, all seems fine to me on the gitea servers/site | 21:28 |
| *** dviroel|rover is now known as dviroel|rover|afk | 21:33 | |
| ianw | clarkb: if you have time for https://review.opendev.org/c/openstack/diskimage-builder/+/811392 which adds 9-stream to centos-minimal, i'm sure it would be appreciated | 21:59 |
| ianw | afaict it builds, it boots, so we're more or less ready to move on with it | 21:59 |
| clarkb | ya I'm not sure how much review I can do beyond general structure stuff. Not a centos expert, but if it builds and boots that is a very good sign :) | 22:08 |
| ianw | i think that's about it. for production purposes, a mirror is about 200gb when i dry-run rsync'd it recently | 22:09 |
| ianw | there's still a few debian stable bits hanging on i need to clear, i hope to free that soon | 22:10 |
| ianw | also i guess we'll need a f35 mirror as well. we only need f34+f35 for a short period as we cut over jobs | 22:10 |
| ianw | afs01 is at 3.8/4TB | 22:11 |
| ianw | afs02 at 3.4/4TB | 22:12 |
| clarkb | soon we'll be able to clean out centos-8 as well | 22:14 |
| ianw | if we can't find too many other things to optimise we might be at the point we need a bit more disk on them | 22:14 |
| ianw | yeah | 22:14 |
| ianw | more disk == more chance of a disk screwing up == more chance of afs recovery hell | 22:14 |
| clarkb | https://mirror.bhs1.ovh.opendev.org/ubuntu-ports/dists/ I think we can remove xenial from that too | 22:14 |
| clarkb | debian stretch + ubuntu ports xenial is probably a good chunk of disk. Then when we get to to centos-8 eol another chunk | 22:15 |
| ianw | yes, indeed! i'll get a change up for that, we stopped xenial arm64 | 22:15 |
| opendevreview | Ian Wienand proposed opendev/system-config master: reprepro: stop mirroring ubuntu-ports Xenial https://review.opendev.org/c/opendev/system-config/+/815914 | 22:17 |
| ianw | we had a user pop up in #openstack-dib asking about using our mirrors and the gpg signing, on query they said "however the idea behind using the same mirror as the community is to avoid problems like oslo.vmware, which ended up affecting other CIs but not the community's CI." | 22:18 |
| clarkb | we explicitly do not want people using our mirrors. For reasons like the bhs1 problem too | 22:18 |
| clarkb | *bhs1 problem today | 22:18 |
| ianw | i'm not aware of what that issue was. but i did make it clear we operate the mirrors for the benefit of the CI we run, so at any time, like above, we might kill/move/drop cloud etc. | 22:19 |
| clarkb | we were able to address that within our systems but we wouldn't necessarily be able to update them if we turn it off. | 22:19 |
| clarkb | basically the disk filled. We set max-servers to 0 then started surgery | 22:19 |
| clarkb | if the issue is prolonged they would be broken but we would be fine | 22:19 |
| clarkb | and ya clouds come and go too | 22:19 |
| fungi | and we also may reorganize our mirrors without warning | 22:20 |
| clarkb | ianw: I think the solution to the wheel problem is to have requirements actually test that it can install stuff from pypi without our mirrors (and frickler was workong on that) | 22:20 |
| ianw | we don't explicitly list reprepro cleanup | 22:22 |
| ianw | iirc, after applying the removal, if we take the lock and run "|reprepro --delete clearvanished| that should do it? | 22:23 |
| ianw | (applying the removal to the config files) | 22:23 |
| clarkb | oh I never remember but ya I think there is a reprepro command for it | 22:24 |
| ianw | clarkb: yes that sound reasonable. i feel like there's already some sort of giant co-isntall test? | 22:24 |
| clarkb | ianw: ya there is, but it uses our wheels. I think frickler's update was to run a second pass without our mirrors set | 22:24 |
| clarkb | ianw: any idea why https://review.opendev.org/c/openstack/project-config/+/811442/2/nodepool/elements/nodepool-base/package-installs.yaml is needed? | 22:24 |
| ianw | clarkb: oh, i'm assuming similar to what fungi was talking about, everyone is dropping iptables lately | 22:25 |
| clarkb | I see. I wonder if anyone has replaced iptables-persistent without needing to do firewalld or something else silly | 22:26 |
| ianw | that should be in pkg-map anyway | 22:27 |
| clarkb | I think we should also wait for haveged to haev a proper package or not install it on centos-9 (and jobs there can be slow I guess) | 22:28 |
| clarkb | historically the rdo package hosting has not been very reliable and I don't want that to be a weak link in our builds | 22:28 |
| ianw | has it been replaced with some systemd component, etc? | 22:29 |
| clarkb | I'm not sure they just indicated on that change that rdo is packaging it until centos 9 has a package for it | 22:31 |
| clarkb | I'm not going to -1 over this but https://review.opendev.org/c/openstack/diskimage-builder/+/811392/20/diskimage_builder/elements/yum-minimal/package-installs.yaml has no newline at the end of the file now so it shows up in the diff :P | 22:31 |
| ianw | it's probably worth evaluating if this should be using rng-tools or something else | 22:32 |
| opendevreview | Ian Wienand proposed openstack/project-config master: dstat graph: update to version with fixes https://review.opendev.org/c/openstack/project-config/+/815915 | 22:39 |
| ianw | clarkb / corvus : ^ that should fix actual opendev jobs using the roles getting the layout fixes | 22:40 |
| clarkb | ianw: any idea why https://review.opendev.org/c/openstack/diskimage-builder/+/811392/20/diskimage_builder/elements/yum-minimal/root.d/08-yum-chroot#146 line 146 changes? | 22:45 |
| ianw | i was thinking that was actually a bug fix. i'm not sure why we'd strip the -stream off it | 22:47 |
| clarkb | ya I guess it would've been nice to split out the various stream-8 related work into a couple of parent changes but if we have to fix things we can sort it out then | 22:51 |
| clarkb | the biggest impact is if we have to revert | 22:51 |
| clarkb | I've approved it and also left a note about how it is weird to need to install glibc explicitly | 22:51 |
| mordred | clarkb: installing glibc explicitly is, indeed, weird | 22:52 |
| clarkb | ya I'm not sure what a distro is without a libc | 22:52 |
| clarkb | without a kernel ok you're a container image probably | 22:52 |
| clarkb | but with a libc you're some sort of all in one thing and the distro doesn't make sense anymore :0 | 22:53 |
| fungi | you're not a container image, you're a statically compiled executable | 22:55 |
| corvus | ianw: thanks approved! | 22:58 |
| ianw | fungi: do you feel like there's any reason why "reprepro --delete clearvanished" couldn't be part of the regular update script steps? | 23:04 |
| fungi | will that remove packages which we only just updated? | 23:04 |
| ianw | i think it should only remove things we've dropped from the config files | 23:05 |
| fungi | just making sure it won't undo our intentional one-pulse delay on deleting packages which are no longer listed in indices | 23:06 |
| ianw | i'm just writing instructions on how to apply changes like https://review.opendev.org/c/opendev/system-config/+/815914 | 23:06 |
| opendevreview | Merged openstack/project-config master: dstat graph: update to version with fixes https://review.opendev.org/c/openstack/project-config/+/815915 | 23:07 |
| ianw | i guess that comes down to "does clearvanished imply deleteifunreferenced" | 23:08 |
| ianw | it seems even wikimedia do it by hand ... https://wikitech.wikimedia.org/wiki/Reprepro#Removing_a_component | 23:28 |
| opendevreview | Ian Wienand proposed opendev/system-config master: reprepro: add note on removing components https://review.opendev.org/c/opendev/system-config/+/815920 | 23:47 |
| clarkb | ianw: should ^ do a vos release too before dropping the lock | 23:48 |
| opendevreview | Ian Wienand proposed opendev/system-config master: reprepro: add note on removing components https://review.opendev.org/c/opendev/system-config/+/815920 | 23:50 |
| ianw | clarkb: good point, done :) | 23:50 |
| clarkb | +2 | 23:50 |
| ianw | clarkb: i assume you're ok with removing xenial ubuntu-ports? https://review.opendev.org/c/opendev/system-config/+/815914 | 23:52 |
| ianw | if so i'll have a go at it this afternoon and validate the instructions with removing that | 23:53 |
| clarkb | ya I'm good with it. Just looking over the role now to make sure we don't have to clean anything else up | 23:55 |
| clarkb | seems like when it was puppet there were a lot more bits to clean up | 23:55 |
| clarkb | aha the updates file doesn't know about specific releases | 23:56 |
| clarkb | so ya this lgtm | 23:56 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!