| opendevreview | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/897273 | 02:18 |
|---|---|---|
| opendevreview | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/c/openstack/project-config/+/897273 | 03:00 |
| opendevreview | daniel.pawlik proposed zuul/zuul-jobs master: Add feature to set --vm-driver name for minikube https://review.opendev.org/c/zuul/zuul-jobs/+/894755 | 07:42 |
| priteau | Is there an issue with opendev rockylinux mirrors? Kayobe CI keeps failing with: Depsolve Error occurred: \n Problem: cannot install the best candidate for the job\n - nothing provides python3 = 3.9.16-1.el9_2.2 needed by python3-devel-3.9.16-1.el9_2.2.x86_64\n - nothing provides python3-libs(x86-64) = 3.9.16-1.el9_2.2 needed by python3-devel-3.9.16-1.el9_2.2.x86_64 | 07:59 |
| frickler | priteau: I don't thing we mirror rocky, that should be just a caching proxy. can you check the status of upstream mirrors? I seem to remember that there were some more issues in the recent past already. NeilHanlon might know more | 08:06 |
| priteau | Maybe we're just unlucky and are hitting an unsynced mirror. python3-devel is in appstream but python3 and python3-libs are in baseos. | 08:10 |
| frickler | infra-root: wheel builds for debian seem to be failing for 1 month according to the release timers on https://grafana.opendev.org/d/9871b26303/afs?orgId=1 , maybe we should just stop doing wheel builds in general with nobody really around to maintain them any longer | 08:13 |
| fungi | frickler: interesting, looks like the wheel architecture changed from debian-11-x86_64 to debian-11.7-x86_64 | 11:49 |
| fungi | wheel builds seem to be succeeding, it's writing them into afs that breaks because the directory name is now different | 11:50 |
| fungi | https://zuul.opendev.org/t/openstack/build/e20bf2e2d60b439f8eae93ce90fd1a7a | 11:50 |
| fungi | i think this is an ansible change | 11:53 |
| fungi | we're populating that from ansible_distribution_version, which for bullseye changed from 11 to 11.7 around the time we made the default ansible version update | 11:55 |
| fungi | i guess we need to adjust playbooks/publish/wheel-mirror.yaml to treat debian the same way as centos now and just use ansible_distribution_major_version | 11:55 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Use Debian major version in Debian wheel filenames https://review.opendev.org/c/openstack/project-config/+/897545 | 12:04 |
| fungi | frickler: that ^ ought to address it | 12:04 |
| frickler | fungi: oh, interesting, iirc osa had a similar issue with the name changing recentish, one month ago might fit | 12:06 |
| fungi | well, i narrowed down the start of the post_failure results to be the next run after the ansible default change merged for that tenant | 12:07 |
| fungi | so hard to imagine anything else might have changed on that same day to break it | 12:07 |
| frickler | oh, so it was the zuul ansible version change. noonedeadpunk jrosser ^^ not sure if your issue has gotten resolved, but it may still be interesting for you | 12:10 |
| noonedeadpunk | I do recall that, but can't recall detail already :( | 12:12 |
| noonedeadpunk | though... we could forget to address that... | 12:12 |
| noonedeadpunk | so it's good you reminded about that:) | 12:13 |
| jrosser | I think we saw different versions reported from the host and from an lxc built with debootstrap | 12:17 |
| jrosser | for reasons which were completely unclear | 12:17 |
| NeilHanlon | frickler, priteau: we had (another) outage yesterday evening--but I've put in place the hopefully final mitigation against the issues we're having that will, at the very least, bring back stability to dl.rockylinux.org and our mirrorlist server | 12:32 |
| NeilHanlon | (cc noonedeadpunk, jrosser -- and thanks for putting up with the noise these last few weeks) | 12:33 |
| frickler | NeilHanlon: thx for the update | 12:41 |
| noonedeadpunk | ++ | 12:42 |
| *** blarnath is now known as d34dh0r53 | 13:11 | |
| opendevreview | daniel.pawlik proposed zuul/zuul-jobs master: Add feature to set --vm-driver name for minikube https://review.opendev.org/c/zuul/zuul-jobs/+/894755 | 13:39 |
| opendevreview | daniel.pawlik proposed zuul/zuul-jobs master: DNM Add workaround for unqualified-search in Minikube https://review.opendev.org/c/zuul/zuul-jobs/+/897337 | 13:40 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Use Debian major version in Debian wheel filenames https://review.opendev.org/c/openstack/project-config/+/897545 | 13:45 |
| opendevreview | daniel.pawlik proposed zuul/zuul-jobs master: DNM Add workaround for unqualified-search in Minikube https://review.opendev.org/c/zuul/zuul-jobs/+/897337 | 13:53 |
| clarkb | fungi: any opinion on whether we should land the ua update first or the gitea upgrade? | 15:16 |
| clarkb | I'm good to approve either or both right now | 15:16 |
| fungi | i guess as long as we keep in mind that a problem with requests getting through could be due to either, i'm fine merging both at once | 15:17 |
| fungi | what do you think about merging 897086 ? the spf one is seeming less urgent based on the current finger-pointing going on between zdi/exim/libspf2 | 15:19 |
| fungi | i approved both 897244 and 897500 | 15:20 |
| Clark[m] | 897086 seems fine. We already do that for other names. | 15:22 |
| Clark[m] | And ya test coverage for UA filters is actually decent | 15:23 |
| fungi | cool, approved 897086 too | 15:24 |
| fungi | Clark[m]: do you have an opinion on 897545? seems like a behavior change in ansible 8 we hadn't previously spotted | 15:26 |
| clarkb | fungi: do you have a link to example build logs? | 15:44 |
| clarkb | at least for current debian builds would be nice | 15:45 |
| fungi | clarkb: in scrollback, but i added the link in a review comment just now | 15:46 |
| clarkb | thanks https://zuul.opendev.org/t/openstack/build/e20bf2e2d60b439f8eae93ce90fd1a7a/log/zuul-info/host-info.wheel-cache-debian-bullseye-python3.yaml#152-155 does appear to show the difference | 15:48 |
| opendevreview | Merged openstack/project-config master: Use Debian major version in Debian wheel filenames https://review.opendev.org/c/openstack/project-config/+/897545 | 15:56 |
| mnaser | fungi: can i know what is the ips that you see as unreachable via ipv6? i see a /47 from that range being announced but nothing else which is 2600:6c00:2::/47 | 15:58 |
| fungi | mnaser: 2600:6c60:5300:375:96de:80ff:feec:f9e7 and 240d:1a:6af:1b00:6ee4:daff:fe51:3c95 are two in different parts of the world that don't seem to be routable in ca-ymq-1 but are routable in sjc1 | 15:59 |
| mnaser | interesting i dont even see an announcement for that | 15:59 |
| fungi | i see them in looking glasses for various backbones | 15:59 |
| mnaser | ah ok i see it now 1 sec | 16:00 |
| fungi | mnaser: i'm getting responses--thanks!!! | 16:02 |
| mnaser | one sec, i just wanna make sure it doesnt break again | 16:02 |
| fungi | sure, just confirming that there do seem to be routes now, so you found it | 16:02 |
| mnaser | are they both good? | 16:03 |
| mnaser | i hate to tell you what i did | 16:03 |
| fungi | i don't control the other one, i'll have to give that user a heads up to test | 16:03 |
| mnaser | 'clear bgp ipv6 *' on a friday | 16:03 |
| fungi | oof | 16:03 |
| mnaser | but i dont see a routing loop in traceroute liek i did before | 16:03 |
| fungi | any idea why they were being filtered? ran out of table space? | 16:04 |
| mnaser | no there's plenty, it's actually frr/quagga that does the bgp bits but who knows if something got stuck so kernel/frr was not in sync | 16:04 |
| fungi | oh fun | 16:04 |
| fungi | i'll let you know if it crops up again, but i'm guessing we just got "lucky" that it was an announcement for my isp that started to get dropped on the floor | 16:05 |
| mnaser | sounds like it might be one of those | 16:05 |
| fungi | so if it does happen again, it may be different users impacted next time | 16:06 |
| fungi | thanks again mnaser! very happy to be able to drop my forced-v4 workarounds for talking to gerrit | 16:07 |
| mnaser | no problem, sorry it took so long | 16:07 |
| fungi | no apologies needed | 16:08 |
| fungi | we're all quite busy, i understand completely | 16:08 |
| guilhermesp | Thanks mnaser and appreciate the patience fungi ! | 16:17 |
| fungi | no problem, i'm just happy i could help | 16:18 |
| fungi | i know how hard it can be to deal with vague reports from users | 16:19 |
| fungi | my new netbook picked last night to go toes-up, so i've scrambled to switch my workstation over from pulseaudio to pipewire/wireplumber to get it working reliably with my bt headset and wireless mute button in time for today's openmetal call. seems i've got everything working, but apologies in advance if my audio is terrible, and if i run into technical difficulties please proceed without me | 16:23 |
| fungi | travelling for ato a week from tomorrow will be even more fun. at least my new phone is just a compact debian-based computer, and i had the foresight to buy a laptop-like "dock" for it, so i can use it as a normal laptop too | 16:33 |
| fungi | but i haven't copied my important stuff onto it yet, so i guess that's yet one more thing i need to do over the coming week | 16:34 |
| fungi | the entire drive is luks-encrypted (have to supply a keyfile on removeable media or enter a passphrase when powering it on), so i feel pretty safe using it for opendev stuff | 16:36 |
| clarkb | my laptop is also sad and I really need to call lenovo but its low on my priority list due to lack of planned travel | 16:43 |
| opendevreview | Radosław Piliszek proposed openstack/project-config master: Add nebulous/resource-manager https://review.opendev.org/c/openstack/project-config/+/897576 | 16:45 |
| opendevreview | Merged opendev/system-config master: Upgrade to gitea 1.20.5 https://review.opendev.org/c/opendev/system-config/+/897244 | 16:54 |
| opendevreview | Merged opendev/system-config master: Update the apache ua filter set https://review.opendev.org/c/opendev/system-config/+/897500 | 16:56 |
| opendevreview | Merged opendev/system-config master: Blackhole deliveries for Mailman v3 local user https://review.opendev.org/c/opendev/system-config/+/897086 | 16:56 |
| clarkb | fungi: thank you for taking notes | 18:04 |
| clarkb | It looks like gitea is generally happy but I still need to check all 6 backends updated | 18:04 |
| fungi | for anyone who is curious about the openmetal redeployemnt discussion, raw notes can be found at https://etherpad.opendev.org/p/redeploying-opendevs-openmetal-cloud | 18:05 |
| fungi | gitea is still working for me after both those changes deployed | 18:08 |
| clarkb | I liek stepping through all 6 urls like https://giteaXY.opendev.org:3081/opendev/system-config to confirm the page loads and the version reported is as expected. I've done so and this lgtm | 18:11 |
| fungi | exim deferral queue on lists01 is down to 43 messages after the blackhole alias for the mailman@ address deployed | 18:11 |
| fungi | i need to run to the hardware store and pick up some supplies for more weekend renovation projects, but should be back in about an hour | 18:12 |
| opendevreview | Merged openstack/project-config master: Add nebulous/resource-manager https://review.opendev.org/c/openstack/project-config/+/897576 | 18:15 |
| fungi | exim mainlog is no longer filling up with errors for that address either | 18:16 |
| fungi | speaking of logs, i guess we should see if there are still nefarious crawler agent strings in the gitea server apache logs worth blocking | 18:17 |
| clarkb | ++ | 18:26 |
| fungi | okay, back for now | 19:25 |
| clarkb | I'm going to stop banging my head against the wall called java and find lunch then go do some school stuff | 19:40 |
| fungi | wall o' java | 19:40 |
| fungi | enjoy! | 19:40 |
| fungi | i may be grabbing early dinner shortly myself. nice, quiet friday in opendev | 19:47 |
| fungi | yeah, heading out again for a bit | 19:51 |
| fungi | and still nothing's broken | 22:08 |
| clarkb | I ended up pushing what I've got in response to their comments. I believe I have it filtering refs properly or at least under my test case. But Now I've made the laeked file problem workse :) | 22:08 |
| fungi | greatest friday ever | 22:08 |
| clarkb | I've asked for feedback on the approach though. I don't really want to keep banging my head agains the wall if this isn't what they had in mind | 22:08 |
| fungi | yeah, it doesn't sound great | 22:09 |
| clarkb | the upside is I think it is progress? For a while there it was just replicating everything including what it shouldn't | 22:09 |
| clarkb | so getting back to proper filtration is at least something I guess | 22:09 |
| clarkb | side note: I can't figure out how to see the replication log when running tests which makes it really difficult/annoying to see what is going on | 22:10 |
| clarkb | I've asked for infos on how to do that as well | 22:10 |
| clarkb | I think if I can get that info I can trace things and make sense of it all. I could also theoretically attach jdb but this is all multithreaded and I worry that would be even more confusing | 22:10 |
| clarkb | fungi: for next week you still think Monday is a good day to take the Gerrit bookworm plunge? | 22:11 |
| clarkb | s/still// | 22:11 |
| clarkb | not sure if you ever thought that was a good idea. Asking now :) | 22:12 |
| fungi | yep, i have nothing major planned for that day other than openstack artifact signing key rotation | 22:12 |
| fungi | sounds like a good plan | 22:12 |
| clarkb | cool. I think I verified what I could previously but I'll try to do a quick recheck Monday of things and then we can go for it I guess | 22:12 |
| clarkb | I was hoping we could bundle in this replication file leak fix but that seems very unlikely | 22:13 |
| fungi | another time | 22:13 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!