| *** rlandy|bbl is now known as rlandy|out | 00:18 | |
| *** Guest0 is now known as prometheanfire | 00:29 | |
| ianw | not sure if we discussed it here wrt. the fedora mirror; the kernel.org admins did get back to us and told us for rsync not to use the "us.kernel.org" and that kernel.org is always in the US anyway | 00:48 |
|---|---|---|
| ianw | anyway the fb mirror seems stable enough, so i guess we'll just stay with that, but good info just in case | 00:50 |
| fungi | wfm | 00:55 |
| *** dviroel|afk is now known as dviroel|out | 01:26 | |
| opendevreview | Merged opendev/system-config master: gitea-set-org-logos: openstack logo centered https://review.opendev.org/c/opendev/system-config/+/834582 | 01:26 |
| ianw | i think all the 9-stream images should have unbound log fix no | 03:20 |
| ianw | now | 03:20 |
| ianw | [root@centos-9-stream-ovh-bhs1-0029635673 log]# ls -lh unbound.log | 03:28 |
| ianw | -rw-r--r--. 1 unbound root 541K May 13 03:28 unbound.log | 03:28 |
| johnsom | ack, I will fire off my test | 03:38 |
| johnsom | ianw wahoo. It seems to work: https://2e1e65592f03a6343ed5-0195d60ad5bfbf7ab35d976149e4aebe.ssl.cf2.rackcdn.com/841493/5/check/designate-bind9-centos9stream-unbound-debug/f54b790/controller/logs/unbound_log.txt | 04:29 |
| ianw | yay | 04:30 |
| *** ysandeep|rover|out is now known as ysandeep|rover | 04:50 | |
| opendevreview | David Ostrovsky proposed zuul/zuul-jobs master: Bump bazelisk version to 1.11.0 https://review.opendev.org/c/zuul/zuul-jobs/+/841487 | 05:04 |
| opendevreview | David Ostrovsky proposed zuul/zuul-jobs master: Bump bazelisk version to 1.11.0 https://review.opendev.org/c/zuul/zuul-jobs/+/841487 | 05:05 |
| opendevreview | Sandeep Yadav proposed openstack/diskimage-builder master: [DNM] test patch https://review.opendev.org/c/openstack/diskimage-builder/+/841065 | 05:34 |
| opendevreview | Merged openstack/diskimage-builder master: Adopted dkms element to work on Ubuntu Jammy and nvidia drivers https://review.opendev.org/c/openstack/diskimage-builder/+/841465 | 06:17 |
| opendevreview | Merged openstack/diskimage-builder master: Add Jammy functesting to dib https://review.opendev.org/c/openstack/diskimage-builder/+/840391 | 06:31 |
| *** jpena|off is now known as jpena | 07:02 | |
| opendevreview | Albin Vass proposed zuul/zuul-jobs master: Re-enable tests for prepare-workspace-git https://review.opendev.org/c/zuul/zuul-jobs/+/841376 | 07:09 |
| opendevreview | Albin Vass proposed zuul/zuul-jobs master: mirror-workspace-git: urlencode src_dir https://review.opendev.org/c/zuul/zuul-jobs/+/839225 | 07:09 |
| opendevreview | Albin Vass proposed zuul/zuul-jobs master: Add jobs to test base roles with unique workspace scheme https://review.opendev.org/c/zuul/zuul-jobs/+/841683 | 07:09 |
| *** jpodivin_ is now known as jpodivin | 07:15 | |
| *** ysandeep|rover is now known as ysandeep|rover|lunch | 07:40 | |
| *** lajoskatona_ is now known as lajoskatona | 07:42 | |
| *** ysandeep|rover|lunch is now known as ysandeep|rover | 08:49 | |
| *** rlandy|out is now known as rlandy | 10:09 | |
| *** marios is now known as marios|afk | 10:10 | |
| *** marios|afk is now known as marios | 10:45 | |
| *** dviroel|out is now known as dviroel | 11:00 | |
| opendevreview | Albin Vass proposed zuul/zuul-jobs master: mirror-workspace-git: urlencode src_dir https://review.opendev.org/c/zuul/zuul-jobs/+/839225 | 11:21 |
| opendevreview | Albin Vass proposed zuul/zuul-jobs master: mirror-workspace-git: urlencode src_dir https://review.opendev.org/c/zuul/zuul-jobs/+/839225 | 11:41 |
| opendevreview | Albin Vass proposed zuul/zuul-jobs master: mirror-workspace-git: urlencode src_dir https://review.opendev.org/c/zuul/zuul-jobs/+/839225 | 11:50 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: DNM: Testing image builds for bazelisk 1.11.0 https://review.opendev.org/c/opendev/system-config/+/841721 | 11:54 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: DNM: Testing image builds for bazelisk 1.11.0 https://review.opendev.org/c/opendev/system-config/+/841721 | 12:42 |
| dtantsur | Hey folks! If a patch modifies the set of post jobs, will the new or the old jobs be triggered by its merge? | 13:32 |
| dtantsur | It seems that "the old" is the answer, which I find a bit surprising. | 13:33 |
| fungi | dtantsur: it should be using the just-merged behavior | 14:27 |
| fungi | have an example where it didn't? | 14:28 |
| fungi | could be a bug, or maybe there's some other cause | 14:28 |
| dtantsur | I had an example, but I guess its gone already | 14:39 |
| fungi | was it from more than a month ago? | 14:39 |
| dtantsur | fungi: now, today. when https://opendev.org/openstack/ironic-python-agent-builder/commit/d543aa8bf4eb09de6ee9174e35588f4c422b87fd was merged. | 14:40 |
| fungi | then we should have logs from the post pipeline jobs which that triggered | 14:40 |
| fungi | i can look closer once the openstack release meeting is over | 14:41 |
| dtantsur | thanks! it's not urgent, I just was surprised and wanted to check if it's a bug | 14:42 |
| frickler | I can confirm that the centos-8 was still triggered instead of centos-9. will check zuul logs now https://zuul.opendev.org/t/openstack/builds?project=openstack%2Fironic-python-agent-builder&pipeline=post&skip=0 | 14:51 |
| *** ykarel_ is now known as ykarel | 15:05 | |
| frickler | fungi: corvus: not sure I've collected all the relevant logs, but it looks like a race between the reconfiguration handling on zuul02 and the freezing of the post jobs on zuul01 https://paste.opendev.org/show/b04RrC7X2MUxscdoEjqw/ | 15:09 |
| fungi | oh! so the reconfig was handled by one scheduler but the post pipeline was being managed by the other? | 15:11 |
| *** dviroel is now known as dviroel|lunch | 15:12 | |
| clarkb | The pipeline is supposed to know if the cached config is stale based on the zk monotonically increasing version numbers | 15:25 |
| clarkb | it checks that number to see if it is old/stale and updates if not. But maybe we're processing things out of order so that check ends up valid? | 15:26 |
| clarkb | fungi: I'm doing local updates and will reboot then I'm going to go grab the lock for debian reprepro so we can land https://review.opendev.org/c/opendev/system-config/+/839623 | 15:27 |
| clarkb | the number of packages I have to pull and install is quite large. This might take longer than initially expected | 15:40 |
| clarkb | its like they updated gcc and recompiled everything | 15:40 |
| fungi | sounds like my experience yesterday. i'd put off updates for far too long because i didn't want to have to find time to rebuild my terminal emulator presets since there was a major update to crt which switched around the font handling and geometry | 15:43 |
| fungi | because i'd procrastinated, apt wanted to download >1.5gb of packages and needed a further ~1gb free on top of that to unpack them all | 15:44 |
| fungi | moral of this story: don't put off package updates, but especially not if you're using debian/sid (because the package churn for it is insane) | 15:45 |
| *** ysandeep|rover is now known as ysandeep|out | 15:55 | |
| *** dviroel|lunch is now known as dviroel | 16:00 | |
| *** marios is now known as marios|out | 16:06 | |
| clarkb | I've grabbed the debian repo mirror lock on mirror-update in a screen. Tired of waiting on local package updates | 16:32 |
| clarkb | fungi: any reason to not approve that change now? | 16:32 |
| fungi | nope, approving now | 16:32 |
| *** gibi is now known as gibi_pto | 16:39 | |
| clarkb | One thing we should do when these cleanups are complete (and debian is the last one?) is reset our afs volume quotas to be more inline with current utilization | 16:44 |
| fungi | agreed | 16:44 |
| fungi | also i tested gerrit image builds with a depends-on to davido's ensure-bazelisk update for zuul-jobs, and i see build failures for 3.5 and master (but 3.4 works with it, looks like) | 16:45 |
| fungi | i need to dig into the actual errors though, as they may be unrelated | 16:45 |
| clarkb | fungi: looks like the jvm crashed | 16:49 |
| clarkb | thats exciting | 16:49 |
| fungi | indeed! | 16:53 |
| clarkb | PerfMemory::alloc(unsigned long)+0x59 I wonder if it ran out of memory | 16:53 |
| *** jpena is now known as jpena|off | 16:59 | |
| fungi | the log for the 3.4 build does indicate it installed and used the new bazelisk, so it's a valid test at least | 17:03 |
| fungi | Bazelisk version: v1.11.0 | 17:03 |
| fungi | in both the working 3.4 and failing 3.5 builds | 17:03 |
| fungi | i'm going to recheck it to see if the 3.5 failure is persistent | 17:03 |
| opendevreview | Merged opendev/system-config master: Stop mirroring source packages for debian https://review.opendev.org/c/opendev/system-config/+/839623 | 17:04 |
| mordred | when I was doing those builds originally - weird memory failure errors were the biggest struggle I had | 17:05 |
| fungi | yeah, that's why i'm going to recheck a few times | 17:05 |
| mordred | it was one of the reasons I eventually gave up on having a multi-stage dockerfile and just built it on the vm and copied the war into the container | 17:06 |
| mordred | ++ | 17:06 |
| fungi | entirely possible we'll see 3.5 succeed and 3.4 crash next | 17:06 |
| mordred | won't that be exciting? | 17:06 |
| fungi | it'll certainly be... something | 17:06 |
| fungi | it might be a clue that i should knock off early for some friday beverages | 17:07 |
| fungi | only time will tell | 17:07 |
| clarkb | I'm going to pause midday and go on a bike ride before the big rain storms arrive | 17:21 |
| clarkb | the weather here continues to think it is march and not may | 17:24 |
| clarkb | it is getting old, but have to take advantage of the lack of rain as much as possible | 17:24 |
| clarkb | I'm going to run clearvanished, deleteunreferenced, then a normal reprepro run against hte debian mirrors now (the deploy is just about done) | 17:27 |
| clarkb | fungi: hrm we may still be setting the deb-src on debian nodes. I don't know why | 17:44 |
| clarkb | fungi: should I ^C the reprepro update that would likely break all the debian jobs as a result? | 17:44 |
| clarkb | because I'm a derp | 17:48 |
| clarkb | I think we can just quickly alnd a fix for this sintead | 17:48 |
| fungi | oh! | 17:48 |
| fungi | where are we still setting it? dib? | 17:49 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Fix the disabling of deb src repos in configure-mirrors https://review.opendev.org/c/zuul/zuul-jobs/+/841785 | 17:50 |
| clarkb | fungi: ^ no I hopped on a running job and checked and it was there I think that is the fix I was a derop | 17:50 |
| clarkb | anyway I did a ^C whcih stopped the script after it checked reprepro mirror integrity so no publish has been made yet | 17:51 |
| clarkb | we shouldn't be broken yet, but landing ^ and then checking another host doesn't add deb-src then running reprepro would be good | 17:51 |
| clarkb | once that change lands we can trigger a job that runs against debian then hop on the server and check what /etc/apt/sources.list.d/defaults.list looks like | 17:53 |
| fungi | yep | 17:54 |
| clarkb | and then we can rerun the reprepro command in the screen. Good to know that a single ^C seems to be safe. Reprepro even reorts that giving it a second ^C will kill it but at the risk of db corruption. Otherwise it seems to wait until it can exit cleanly (but with non zero exit code) | 17:57 |
| fungi | yeah, love robust programs which actually trap sigint with useful behaviors | 17:58 |
| fungi | true to form, random jvm allocation crash did not occur on the 3.5 build after rechecking (nor on 3.4) | 18:02 |
| fungi | master build still failed, but i haven't looked into why | 18:02 |
| fungi | the master builds seem to be unrelated persistent breakage | 18:07 |
| opendevreview | Merged zuul/zuul-jobs master: Fix the disabling of deb src repos in configure-mirrors https://review.opendev.org/c/zuul/zuul-jobs/+/841785 | 18:07 |
| fungi | plugins/its-base/src/main/java/com/googlesource/gerrit/plugins/its/base/util/PropertyAttributeExtractor.java:28: error: package org.apache.commons.lang does not exist | 18:07 |
| fungi | i'll recheck again to see if the jvm crash is frequent | 18:08 |
| clarkb | fungi: I rechecked https://review.opendev.org/c/zuul/zuul-jobs/+/839594 which should get us bullseye and buster nodes to check live for deb-src | 18:08 |
| fungi | thanks! | 18:08 |
| clarkb | fungi: ya looks good on bullseye | 18:09 |
| fungi | perfect | 18:09 |
| clarkb | only deb http://mirror.mtl01.iweb.opendev.org/debian bullseye main is in there and the mlt01 mirror indicates this isn't our default from the image build | 18:10 |
| fungi | however, we should probably wait a bit before proceeding with the mirror cleanup in case there are jobs started from an earlier config state which might try to pull indices | 18:10 |
| clarkb | good point | 18:10 |
| fungi | though the mirror cleanup also won't take effect globally until we vos release the volume | 18:10 |
| clarkb | ya though thats what the next step is on the mirror (running our normal reprepro update script which does a vos release) | 18:11 |
| clarkb | I think we can wait on that a bit if we want. THough I was planning on stepping out in about an hour for that bike ride. I can update the mirror afterwards | 18:11 |
| clarkb | buster looks good too | 18:12 |
| clarkb | I've run a nodepool list and filtered for debian nodes so that I can run it again later and see if any of the old nodes still exist | 18:13 |
| clarkb | if they do we can wait, if they don't we can proceed | 18:13 |
| clarkb | ok all of the debian nodes from before the fix appear to have rotated out. I'm going to run reprepro now | 20:58 |
| clarkb | it has completed and I have released the flock | 21:16 |
| clarkb | that saved us another 121GB of disk | 21:18 |
| fungi | pretty significant! | 21:45 |
| fungi | that was including the security mirror? | 21:46 |
| *** dviroel is now known as dviroel|out | 21:57 | |
| clarkb | no I think security is separate and never had source | 22:14 |
| clarkb | also sorry I got totally nerd sniped by my xmonad config. Turns out I had a bunch of deprecated stuff that I could replace with some simpler config | 22:14 |
| clarkb | turns out I can still read just enough haskell to be dangerous :) | 22:21 |
| clarkb | also confirmed that tumbleweed updated its compiler to gcc 12.1 and recompiled a bunch of stuff whcih explains my giant update today | 22:30 |
| opendevreview | Merged zuul/zuul-jobs master: Bump bazelisk version to 1.11.0 https://review.opendev.org/c/zuul/zuul-jobs/+/841487 | 23:33 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!