| ianw | facebook is a lot of things, but "under-resourced infrastructure" probably isn't one of those things :) | 00:02 |
|---|---|---|
| *** rlandy|bbl is now known as rlandy|out | 00:03 | |
| clarkb | sometimes they break bgp and can't open the datacenter doors though | 00:04 |
| fungi | or they break the data center doors and then can't fix bgp | 00:10 |
| fungi | and when it's both... that's called a thursday | 00:11 |
| *** artom_ is now known as artom | 00:47 | |
| Clark[m] | https://github.com/acmesh-official/acme.sh/runs/6380327180?check_suite_focus=true "The log was not found. It may have been deleted based on retention settings." Guess I'm not debugging it then. Also every other job was cancelled so I don't get any logs from them either? | 01:10 |
| Clark[m] | It's like we need to help GitHub with their CI stuff | 01:11 |
| Clark[m] | ianw: the change was merged anyway. If you have time can you check the run we get in a few hours? Not sure if that is past the end of your day | 01:12 |
| ianw | Clark[m]: oh, excellent, yes i'll check on it | 01:19 |
| ianw | i'm pretty sure we've got better ci of many upstreams than they do of themselves :) | 01:19 |
| *** ysandeep|out is now known as ysandeep|rover | 01:21 | |
| ianw | oh although i'm not sure we run dev | 01:28 |
| ianw | oh, we do ... | 01:30 |
| opendevreview | Merged opendev/system-config master: mirror-update: switch Fedora mirror https://review.opendev.org/c/opendev/system-config/+/841349 | 02:01 |
| *** ysandeep|rover is now known as ysandeep|rover|bbl | 02:08 | |
| ianw | we could probably cull i686 packages too, fedora seems to pull them in | 03:09 |
| *** bhagyashris|out is now known as bhagyashris | 03:58 | |
| *** ysandeep|rover|bbl is now known as ysandeep|rover | 04:23 | |
| ianw | i can see that ~ 2022-05-11T03:22:24Z acme.sh updated everywhere to bee5cb55a133905c49794a4962fdb1b16b9c92f9 ; which is clarkb's change + something trivial | 04:43 |
| ianw | we then reissued codesearch, vexxhost mirror, limestone mirror, zuul01, zuul02 | 04:47 |
| ianw | $ openssl s_client -servername codesearch.opendev.org -connect codesearch01.opendev.org:443 2>/dev/null | openssl x509 -noout -dates | 04:50 |
| ianw | notBefore=May 11 02:31:33 2022 GMT | 04:50 |
| ianw | notAfter=Aug 9 02:31:32 2022 GMT | 04:50 |
| ianw | which seemed weird, but then | 04:50 |
| ianw | https://community.letsencrypt.org/t/why-my-le-cert-notbefore-date-one-hour-before-i-apply-to-issue/60848/2 | 04:50 |
| *** soniya29 is now known as soniya29|ruck | 04:51 | |
| ianw | le certs are backdated by an hour to avoid clock skew issues. which is logical, but TIL. so the timestamps line up | 04:51 |
| ianw | ergo, clarkb fixed it :) | 04:51 |
| frickler | I tested running reprepro to mirror 22.04 on a 22.04 host and I get the same errors as on 20.04, so currently that won't help us. updated https://bugs.launchpad.net/ubuntu/+source/reprepro/+bug/1968198 accordingly | 05:26 |
| frickler | I also checked that our mirrors are still stuck at 10% phased updates, while upstream has long finished all phasing for current 22.04 updates | 05:27 |
| opendevreview | Merged opendev/system-config master: mirror-update: clean up Fedora ignores https://review.opendev.org/c/opendev/system-config/+/837637 | 05:29 |
| ianw | frickler: thanks, that's interesting and a bit annoying. is that error different to the compresssion issue though? | 06:33 |
| *** ysandeep|rover is now known as ysandeep|rover|brb | 07:29 | |
| *** soniya is now known as soniya|ruck | 07:40 | |
| *** ysandeep|rover|brb is now known as ysandeep|rover | 07:40 | |
| frickler | ianw: the error on 22.04 looks exactly the same as on 20.04: zstd: error 70 : Write error : cannot write decoded block : Broken pipe | 07:49 |
| *** soniya|ruck is now known as soniya|ruck|dinner | 07:51 | |
| *** soniya|ruck|dinner is now known as soniya|ruck|lunch | 07:53 | |
| opendevreview | Albin Vass proposed zuul/zuul-jobs master: mirror-workspace-git: urlencode src_dir https://review.opendev.org/c/zuul/zuul-jobs/+/839225 | 07:55 |
| opendevreview | Albin Vass proposed zuul/zuul-jobs master: Re-enable tests for prepare-workspace-git https://review.opendev.org/c/zuul/zuul-jobs/+/841376 | 07:55 |
| *** ysandeep|rover is now known as ysandeep|rover|lunch | 08:01 | |
| *** soniya|ruck|lunch is now known as soniya|ruck | 08:56 | |
| *** ysandeep|rover|lunch is now known as ysandeep|rover | 09:02 | |
| *** jpena|off is now known as jpena | 09:39 | |
| *** bhagyashris_ is now known as bhagyashris | 11:03 | |
| *** rlandy|out is now known as rlandy | 11:21 | |
| *** dviroel|out is now known as dviroel | 11:31 | |
| *** ysandeep|rover is now known as ysandeep|rover|brb | 11:32 | |
| fungi | ianw: frickler: https://github.com/ionos-cloud/reprepro/issues/20 seems to indicate that the error is benign, but also suggests that the solution may be as simple as installing zstd? | 11:32 |
| fungi | though i think people discussing there may be rerunning reprepro, not seeing the error (because no new affected packages were synced that time) and assuming it was fixed | 11:34 |
| fungi | hard to know | 11:34 |
| *** ysandeep|rover|brb is now known as ysandeep|rover | 11:37 | |
| *** artom_ is now known as artom | 11:42 | |
| *** ysandeep|rover is now known as ysandeep|rover|mtg | 11:55 | |
| fungi | or that they're conflating multiple issues | 11:55 |
| frickler | zstd is installed on my jammy vm, seems it is preinstalled in the ubuntu cloud image, also reprepro depends on it. but interesting to see that ionos seems to have done their own fork. but understandable seeing the timeline | 11:55 |
| * frickler fights the urge to dig into reprepro sources | 11:57 | |
| *** ysandeep|rover|mtg is now known as ysandeep|rover | 12:37 | |
| *** ysandeep|rover is now known as ysandeep|rover|brb | 13:21 | |
| *** ysandeep|rover|brb is now known as ysandeep|rover | 13:31 | |
| *** bhagyashris_ is now known as bhagyashris | 13:33 | |
| clarkb | yay for having certs refreshed | 14:53 |
| clarkb | I have a dentist visit in a couple of hours so will be popping out after my morning meeting | 14:53 |
| clarkb | fungi: frickler: ya I agree zstd seemed to be installed | 14:53 |
| *** soniya|ruck is now known as soniya|out | 15:05 | |
| *** dviroel is now known as dviroel|lunch | 15:09 | |
| johnsom | Hi neighbors, I am seeing "Could not resolve host: mirror-int.dfw.rax.opendev.org" on at least one job. I think I saw some recent chatter on this issue. Does it still need research or is it a known issue? The unbound log is empty. | 15:21 |
| *** ysandeep|rover is now known as ysandeep|rover|out | 15:26 | |
| fungi | johnsom: i saw someone point out a similar name resolution error on a build earlier, though it was in a different provider so i guess it's not provider-specific networking issues (or it's impacting multiple providers). normally the nodes should have unbound installed and running as recursive resolvers pointing at a couple of different dns resolver ip addresses on the internet (opendns and | 15:30 |
| fungi | google dns maybe?), and then resolv.conf configured to use the loopback address for lookups | 15:30 |
| fungi | this is in order to perform lookup caching locally and reduce the number of dns queries each node sends | 15:31 |
| johnsom | Yeah, unbound is there, but the log is empty and I don't see the conf for it. Basically I will dig a bit deeper on this if it's not a "known" issue. I just didn't want to duplicate effort. | 15:31 |
| fungi | right. i also compared to a working build earlier and the unbound log was empty then too | 15:32 |
| johnsom | Ok, let me dig into this and see if I can find why the log is empty in these runs. That at least is a starting point. | 15:33 |
| fungi | agreed, i don't know if that means we've broken unbound logging/log collection, or that we're not going through unbound for queries so it has nothing to log | 15:35 |
| johnsom | It normally has some startup noise. It looks like it's just centos jobs that the log is empty (checked centos 9 stream) so far. The ubuntu jobs are populated. | 15:36 |
| clarkb | you should always always link to specific job logs | 15:36 |
| johnsom | https://zuul.opendev.org/t/openstack/build/4293bd9af704494388894f1069c48579/logs | 15:37 |
| clarkb | there are so many things that can affect dns resolution like running in a conatiner | 15:37 |
| clarkb | that seems to fail because dnf didn't find a rabbitmq package? Where is the dns lookup failure? | 15:39 |
| johnsom | https://zuul.opendev.org/t/openstack/build/4293bd9af704494388894f1069c48579/log/job-output.txt#2131 | 15:40 |
| clarkb | ok so that is happening on centos-9 on our base image install without cnotainers being involved | 15:41 |
| clarkb | interesting that that error wasn't fatal | 15:42 |
| clarkb | both ns1 and ns2 currently resolve that name so we aren't stuck in some sort of bad authoritative server problem | 15:42 |
| clarkb | the other hunch is that we should check if centos-9 is honoring our unbound resolver or if it is doing its own thing because centos 9 is new | 15:43 |
| clarkb | the empty unbound log that you point out is suspicous | 15:44 |
| *** marios is now known as marios|out | 15:44 | |
| fungi | systemd has its own bus-based dns resolver implementation, right? maybe it's going through that instead and ignoring resolv.conf completely | 15:44 |
| johnsom | So far, ubuntu and debian are all collecting the logs, but c9s and the fedora jobs have empty logs. I am poking around for a c8s job that still has logs to see if it's empty too. I suspect there is just a collection issue on the RH tribe | 15:45 |
| clarkb | fungi: it is supposed to figure that out I thought | 15:46 |
| clarkb | but ya its been a semi frequent fight to convince various distros to actually use our resolver and not the one the cloud attempts to use | 15:46 |
| frickler | these are fips jobs, maybe the reboot interferes with our unbound setup | 15:47 |
| clarkb | https://zuul.opendev.org/t/openstack/build/4293bd9af704494388894f1069c48579/log/controller/logs/worlddump-latest.txt#172 shows unbound is running at least | 15:48 |
| johnsom | Yeah, c8s non-fips also has an empty unbound log. So, we have a log collection issue on that side. | 15:49 |
| johnsom | I don't mean to distract you all, I am fine digging into this | 15:50 |
| clarkb | its fine I'm going to disappear to the dentist soon :P | 15:50 |
| clarkb | then I'll be very distracted | 15:50 |
| johnsom | lol, I will certainly take unbound over that | 15:51 |
| fungi | "better than a trip to the dentist" | 15:54 |
| *** dviroel|lunch is now known as dviroel | 16:03 | |
| *** jpena is now known as jpena|off | 17:01 | |
| johnsom | This is probably why: https://opendev.org/openstack/project-config/src/branch/master/nodepool/elements/nodepool-base/finalise.d/89-boot-settings#L71 | 17:26 |
| johnsom | Giving this a go: https://review.opendev.org/c/openstack/devstack/+/841460 | 17:36 |
| fungi | oh, makes sense | 18:26 |
| fungi | good catch! | 18:26 |
| opendevreview | Denys Mishchenko proposed openstack/diskimage-builder master: Adopted dkms element to work on Ubuntu Jammy and nvidia drivers https://review.opendev.org/c/openstack/diskimage-builder/+/841465 | 19:13 |
| opendevreview | Denys Mishchenko proposed openstack/diskimage-builder master: Adopted dkms element to work on Ubuntu Jammy and nvidia drivers https://review.opendev.org/c/openstack/diskimage-builder/+/841465 | 19:17 |
| clarkb | on zuul02 I see Le_Keylength='2048' which confirms my expectation that it would write that back out again and this is only a problem at the migration point | 19:53 |
| clarkb | now that we are close to being able to remove debian src packages from our mirrors I'm realizing I should've written down the repo cleanup steps someplace other than eavesdrop's logs | 20:10 |
| fungi | i thought ianw added them to the system-config docs... double-checking, i might have imagined reviewing that change | 20:11 |
| fungi | clarkb: https://docs.opendev.org/opendev/system-config/latest/reprepro.html#removing-components | 20:12 |
| fungi | is that what you were looking for? | 20:13 |
| clarkb | ah yup there they are | 20:14 |
| clarkb | ianw: if you get a chance today https://review.opendev.org/c/opendev/system-config/+/838948 is a straightforward system-config cleanup but good to have eyebalsl on that in case we end up unconfiguring stuff that is still alive | 20:52 |
| *** dviroel is now known as dviroel|afk | 20:53 | |
| *** timburke__ is now known as timburke | 21:00 | |
| fungi | pip 22.1 was just tagged. https://pip.pypa.io/en/stable/news/#v22-1 | 21:21 |
| fungi | (most of the changes appear under the 22.1b1 entry though because apparently towncrier isn't as smart as reno) | 21:22 |
| *** rlandy is now known as rlandy|bbl | 22:08 | |
| ianw | clarkb: will do | 22:13 |
| clarkb | fungi: I'm not seeing anything problematic in there but "Explains why specified version cannot be retrieved when Requires-Python is not satisfied." should be a nice addition | 22:15 |
| fungi | yep! | 22:16 |
| *** mazzy5098812929580859494 is now known as mazzy509881292958085949 | 22:22 | |
| ianw | https://launchpad.net/~openstack-ci-core/+archive/ubuntu/python-pip is another one of these in use ppa's i guess | 23:19 |
| ianw | iirc we use it for trusty ensure-pip? | 23:19 |
| clarkb | oh xenial? Its beacuse the distro pip is pip 8 and and doesn't understand python-requires so we do a pip 9 so that it can upgrade to the latest version that undersatnds python 3.5 iirc | 23:20 |
| ianw | i'm guessing we can remove https://launchpad.net/~openstack-ci-core/+archive/ubuntu/python-bpo-27945-backport ? | 23:20 |
| clarkb | ianw: I believe that is correct. We hit that with zuul iirc | 23:20 |
| clarkb | and it did end up getting an SRU eventually | 23:21 |
| ianw | and afaics we don't use https://launchpad.net/~openstack-ci-core/+archive/ubuntu/bubblewrap any more? | 23:24 |
| ianw | https://launchpad.net/~openstack-ci-core/+archive/ubuntu/debootstrap we dropped from the nodepool image, but now use the unstable version of debootstrap | 23:25 |
| ianw | that was fairly recent though so i think leave that for the time being | 23:25 |
| opendevreview | Merged opendev/system-config master: Remove group vars for removed services https://review.opendev.org/c/opendev/system-config/+/838948 | 23:27 |
| clarkb | ya bwrap we switched to the distro version as soon as it supported user namespacing in the distro version iirc and we've been on distro versions for quite some time | 23:28 |
| ianw | i'll get rid of that to avoid confusion too | 23:28 |
| opendevreview | Ian Wienand proposed opendev/infra-vhd-util-deb master: Add README.rst about empty master branch https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/841502 | 23:31 |
| opendevreview | Merged opendev/infra-vhd-util-deb master: Add README.rst about empty master branch https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/841502 | 23:32 |
| ianw | openstack-zuul-jobs 840572 merged; so i'm going to push some changes to the distro branches now that will build and push openafs/vhd-util packages to the production repos | 23:33 |
| opendevreview | Ian Wienand proposed opendev/infra-vhd-util-deb focal: Trigger upstream build https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/841504 | 23:36 |
| clarkb | corvus: this isn't urgent but I'm selfishly wanting to update zuul(-web) to the version that will scroll to the linked log line. Is that something that would be reaosnable to do tomorrow? | 23:37 |
| opendevreview | Ian Wienand proposed opendev/infra-openafs-deb trusty: Reset changelog version https://review.opendev.org/c/opendev/infra-openafs-deb/+/841505 | 23:37 |
| opendevreview | Ian Wienand proposed opendev/infra-openafs-deb xenial: Add build stamp to push to production PPA https://review.opendev.org/c/opendev/infra-openafs-deb/+/841506 | 23:39 |
| opendevreview | Ian Wienand proposed opendev/infra-openafs-deb bionic: Add build stamp to push to production PPA https://review.opendev.org/c/opendev/infra-openafs-deb/+/841507 | 23:40 |
| opendevreview | Ian Wienand proposed opendev/infra-openafs-deb focal: Add build stamp to push to production PPA https://review.opendev.org/c/opendev/infra-openafs-deb/+/841508 | 23:40 |
| opendevreview | Ian Wienand proposed opendev/infra-openafs-deb jammy: Add build stamp to push to production PPA https://review.opendev.org/c/opendev/infra-openafs-deb/+/841509 | 23:41 |
| corvus | clarkb: i see no reason not too; should be possible to do standalone | 23:45 |
| clarkb | great, I'll give that a go tomorrow morning then | 23:45 |
| corvus | fyi, i'm giving a talk to the gerrit hackathon tomorrow morning, so won't be available until later. but don't wait for me. :) | 23:47 |
| opendevreview | Merged opendev/infra-openafs-deb trusty: Reset changelog version https://review.opendev.org/c/opendev/infra-openafs-deb/+/841505 | 23:48 |
| opendevreview | Merged opendev/infra-vhd-util-deb focal: Trigger upstream build https://review.opendev.org/c/opendev/infra-vhd-util-deb/+/841504 | 23:50 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!