| ianw | ^ anyone have an issue with adding the dns/inventory entries for bridge01.opendev.org? | 00:01 |
|---|---|---|
| Clark[m] | ianw: I've switched to dinner mode, but that inventory addition doesn't modify the groups so should be low impact if I understand your change stack. I'm good with that as a first step | 00:05 |
| ianw | no probs -- yes intended to be no-op. i'll be watching closely as i figure out the final switching steps | 00:08 |
| opendevreview | Merged opendev/zone-opendev.org master: Add bridge01.opendev.org https://review.opendev.org/c/opendev/zone-opendev.org/+/862543 | 00:14 |
| ianw | hrm, it looks like the prod playbooks aren't as happy i would have thought :/ | 00:27 |
| ianw | The error appears to be in '/var/lib/zuul/builds/f118475c70fc47f78f1422c3365ed2d5/untrusted/project_0/opendev.org/opendev/system-config/playbooks/zuul/run-production-playbook-post.yaml': line 3, column 7, but may | 00:28 |
| ianw | the role 'add-bastion-host' was not found | 00:28 |
| opendevreview | Ian Wienand proposed opendev/system-config master: Move add-bastion-host to playbooks/zuul/roles https://review.opendev.org/c/opendev/system-config/+/862545 | 00:44 |
| opendevreview | Merged openstack/diskimage-builder master: Added example configuration https://review.opendev.org/c/openstack/diskimage-builder/+/861582 | 01:16 |
| opendevreview | Merged opendev/system-config master: Move add-bastion-host to playbooks/zuul/roles https://review.opendev.org/c/opendev/system-config/+/862545 | 01:35 |
| opendevreview | Ian Wienand proposed opendev/system-config master: add-bastion-host: use hostname directly https://review.opendev.org/c/opendev/system-config/+/862546 | 01:58 |
| opendevreview | Ian Wienand proposed opendev/zone-opendev.org master: Bump serial number https://review.opendev.org/c/opendev/zone-opendev.org/+/862548 | 02:36 |
| opendevreview | Merged opendev/system-config master: add-bastion-host: use hostname directly https://review.opendev.org/c/opendev/system-config/+/862546 | 02:54 |
| opendevreview | Merged opendev/zone-opendev.org master: Bump serial number https://review.opendev.org/c/opendev/zone-opendev.org/+/862548 | 03:04 |
| ianw | ok, sorry about that, prod jobs should be fixed now ... https://zuul.opendev.org/t/openstack/builds?job_name=infra-prod-service-nodepool&project=opendev/system-config | 03:25 |
| ianw | what i totally missed what that monitoring the bootstrap-bridge job is not enough, because that didn't actually use the production playbooks ... doh | 03:25 |
| opendevreview | Merged opendev/system-config master: Add bridge01.opendev.org to inventory https://review.opendev.org/c/opendev/system-config/+/862544 | 04:04 |
| opendevreview | Ian Wienand proposed opendev/base-jobs master: Switch to bridge01.opendev.org https://review.opendev.org/c/opendev/base-jobs/+/862551 | 05:08 |
| *** marios is now known as marios|ruck | 05:08 | |
| opendevreview | Ian Wienand proposed opendev/system-config master: Switch bridge to bridge01.opendev.org https://review.opendev.org/c/opendev/system-config/+/861112 | 05:12 |
| frickler | I'm seeing very slow cloning happening from opendev.org right now for the nova repo. according to openssl I'm landing on gitea02, but cacti looks normal though | 10:05 |
| *** rlandy|out is now known as rlandy | 10:33 | |
| *** dviroel|out is now known as dviroel | 11:29 | |
| fungi | frickler: fwiw, it seems slow for me too | 11:54 |
| fungi | a little over 6 minutes for me to clone it | 11:57 |
| fungi | and i seem to be getting balanced to gitea06 | 12:02 |
| *** rlandy is now known as rlandy|mtg | 12:02 | |
| fungi | and it may be more that twice as slow for me over ipv4 as compared to ipv6 | 12:23 |
| fungi | git clone -4 just now took over 14 minutes | 12:23 |
| fungi | cloning directly from gitea06 in a shell on gitea-lb02 goes quickly though (~1 minute) | 12:37 |
| frickler | if I override opendev.org in /etc/hosts to point to lb01, I get full speed. so seems to be something about lb02 | 12:45 |
| frickler | hmm, or maybe the issue has stopped somehow, still fast after reverting that override | 12:47 |
| fungi | yes, seems fast from here now too. based on observed behaviors, i expect it was a network issue upstream from our virtual servers (whether inside vexxhost sjc1, at the border, or within a backbone provider, hard to know which from our vantage point) | 13:17 |
| fungi | well, now it's slowing down again for me (but not as slow as it was earlier) | 13:24 |
| *** dasm|off is now known as dasm|rover | 13:24 | |
| fungi | around 3.5 minutes | 13:25 |
| fungi | so whatever the network problem it is may be ongoing, just having a variable impact | 13:28 |
| *** rlandy|mtg is now known as rlandy | 13:33 | |
| Clark[m] | Can you try going through lb01 when it is slow just to confirm that it isn't load balancer specific? | 13:56 |
| fungi | sure | 13:56 |
| Clark[m] | But also I'm fairly certain I've seen 5 minute nova clones in the past. So I'm not sure this is a new problem either | 13:57 |
| Clark[m] | Iirc our zuul clone timeout is ~10 minutes due to nova. But that clones from Gerrit. | 13:57 |
| fungi | also it was slow bypassing lb02 and going directly to a gitea backend | 13:58 |
| fungi | so i doubt lb01 would (even could) make that any better | 13:58 |
| Clark[m] | ++ | 13:58 |
| fungi | i just got a 12 minute duration cloning nova | 13:59 |
| fungi | (through lb02 that time) | 13:59 |
| fungi | trying through lb01 now | 13:59 |
| fungi | but it doesn't look appreciably faster | 13:59 |
| Clark[m] | I guess another thing complicating this is lb02 may balance you to a slow backend and lb01 to a fast one. But looks like both produce similar behavior implying it isn't a fault of the load balancer upgrade | 14:00 |
| fungi | yeah, also similarly slow going directly to different backends | 14:00 |
| fungi | watching the data transfer rates more closely, the speed is dramatically impacted by lengthy periods where basically nothing is getting through | 14:26 |
| fungi | so it seems bursty, not steadily slow | 14:27 |
| fungi | but i'm not seeing obvious signs of packet loss either | 14:34 |
| *** rlandy is now known as rlandy|dr_appt | 14:37 | |
| Clark[m] | That could be git being slow to construct pack files? | 14:52 |
| fungi | possible, i suppose, but it's not during the "Enumerating/Counting/Compressing objects" phases | 15:09 |
| fungi | i was seeing it in the "Receiving objects" phase | 15:10 |
| clarkb | huh | 15:10 |
| clarkb | as a heads up I've got people working on my house today which I think will at some point include shutting off power to my home. I've got my network stack on UPS and if that fails I can tether off of my phone. But I'll likely only bother if that occurs during our meeting. Otherwise I'll take it as an opportunity to go for a walk or something :) | 15:12 |
| slittle | Please add me as first core of starlingx-app-security-profiles-operator-core. All add the others. | 15:14 |
| fungi | slittle: done! | 15:15 |
| clarkb | fungi: frickler: considering my possible network outage and the gitea stuff you've been looking at already do we want ot proceed with https://review.opendev.org/c/opendev/system-config/+/862374 today or should we hold off? | 15:16 |
| slittle | thanks | 15:16 |
| fungi | any time | 15:16 |
| fungi | clarkb: i think it's fine to move forward with that. i don't see anything to indicate that the new lb is at fault | 15:16 |
| fungi | and i expect to be around all day if we need to address something | 15:17 |
| clarkb | fungi: did you want to approve it or should I? | 15:18 |
| clarkb | anotehr thing to consider is that will trigger jobs for all the things since it updates the inventory | 15:18 |
| clarkb | we might want to check with ianw to see if the bridge work is stable enough for that | 15:19 |
| fungi | i can approve it momentarily | 15:21 |
| opendevreview | Merged opendev/system-config master: Remove gitea-lb01 and jvb02 from our inventory https://review.opendev.org/c/opendev/system-config/+/862374 | 15:30 |
| clarkb | thanks! | 15:30 |
| clarkb | fungi: https://review.opendev.org/c/opendev/gerritbot/+/861474 is the last change before we can drop python 3.8 image builds | 15:40 |
| clarkb | I'm hoping to be able to tackle the 3.9 -> 3.10 updates soon too. But those are much more involved (we had a lot more stuff on 3.9 after the 3.7 drop) | 15:41 |
| clarkb | infra-root last call on feedback for the storyboard email draft https://etherpad.opendev.org/p/fAVaSBNXEzwsMpfcrKz3 | 15:44 |
| *** jpena is now known as jpena|off | 15:48 | |
| clarkb | also today is the day we said we would switch the default nodeset to jammy | 15:50 |
| *** marios|ruck is now known as marios|out | 15:51 | |
| *** dviroel is now known as dviroel|lunch | 15:56 | |
| clarkb | https://review.opendev.org/c/zuul/zuul/+/862622 that might be the first python 3.11 job run on opendev? | 15:59 |
| fungi | funny, i was just trying out 3.12.0a1 on my workstation now that it exists | 16:03 |
| clarkb | apparently 3.12 will add some limited jit functionality to the interpreter | 16:04 |
| fungi | that's the hope, though it's way too early to know for sure all that will land | 16:05 |
| fungi | it looks like we don't have a base-jobs change for the default nodeset up. i'll propose that momentarily | 16:07 |
| opendevreview | Jeremy Stanley proposed opendev/base-jobs master: Switch default nodeset to ubuntu-jammy https://review.opendev.org/c/opendev/base-jobs/+/862624 | 16:12 |
| fungi | infra-root: ^ | 16:13 |
| opendevreview | Merged opendev/gerritbot master: Switch the docker image over to python 3.10 https://review.opendev.org/c/opendev/gerritbot/+/861474 | 16:14 |
| clarkb | https://zuul.opendev.org/t/zuul/build/3bad42f7277f414b934d80e025333b05 note the error on that one. Not sure I've ever seen that before | 16:16 |
| clarkb | oh I interpreted it as a debian install error due to a hash mismatch | 16:17 |
| clarkb | but I think this is a separate thing. Far less interesting :) | 16:18 |
| clarkb | if I manually pull the tarball the sha checks out. Looking at the log I suspect we got a short read so the file was incomplete? | 16:23 |
| clarkb | network problems abound? | 16:24 |
| fungi | perhaps | 16:25 |
| fungi | bad day to be on the internet maybe | 16:26 |
| fungi | i guess we can include the announcement of 862624 in today's meeting and then merge it once the meeting concludes | 16:27 |
| clarkb | works for me | 16:27 |
| corvus | in case anyone finds it interesting, across all of opendev's cloud providers, it takes a pretty consistent 3-4 seconds for the initial nova api create http request to return. | 16:41 |
| *** rlandy|dr_appt is now known as rlandy | 16:55 | |
| *** dviroel|lunch is now known as dviroel| | 17:04 | |
| *** dviroel| is now known as dviroel | 17:04 | |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Pin py38 jobs to focal https://review.opendev.org/c/zuul/zuul-jobs/+/862628 | 17:09 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Add tox-py311 job https://review.opendev.org/c/zuul/zuul-jobs/+/862629 | 17:09 |
| clarkb | infra-root heads up docker hub appears to be having trouble | 17:12 |
| clarkb | probably good to double check image promotions occur as expected when merging changes (the gerritbot promotion appears to have succeeded) | 17:12 |
| fungi | thanks for spotting | 17:13 |
| * corvus keeps refreshing https://status.docker.com/ expecting to see something other than green | 17:14 | |
| clarkb | I'm going to send that storyboard email now as I haven't heard objections | 17:15 |
| corvus | https://forums.docker.com/t/unexpected-http-status-530/130583/4 has a comment suggesting cloudflare issues | 17:15 |
| corvus | https://www.cloudflarestatus.com/incidents/kdpqngcbbn25 | 17:15 |
| clarkb | that could explain why github release downloads (for the openshift client tarball) also had problems | 17:15 |
| clarkb | and email sent | 17:21 |
| *** dviroel is now known as dviroel|appt | 17:27 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add python 3.11 docker images https://review.opendev.org/c/opendev/system-config/+/862631 | 17:44 |
| corvus | cloudflare claims issue is resolved | 19:00 |
| fungi | i've approved 862624 to switch our default nodeset to ubuntu-jammy now | 20:01 |
| clarkb | thanks! | 20:01 |
| opendevreview | Merged opendev/base-jobs master: Switch default nodeset to ubuntu-jammy https://review.opendev.org/c/opendev/base-jobs/+/862624 | 20:06 |
| fungi | so took effect as of 20:06 utc today | 20:14 |
| clarkb | I've pushed a PR to fix pip's testsuite, but github doesn't allow you to stack PRs and there is no depends on for github actions so I can't get the other fix to run on this fix | 20:37 |
| *** dviroel|appt is now known as dviroel | 20:39 | |
| ianw | clarkb: it's barbaric isn't it :) | 20:41 |
| ianw | i'm going to keep some notes at https://etherpad.opendev.org/p/bastion-upgrade-nodes-2022-10 as i figure out upgrade. i'll condense into a more checklist thing when it's all working | 20:42 |
| clarkb | turns out my PR to fix pip won't fix things because they actually do a `git submodule update` on the package install side of things outside of the test suite | 20:56 |
| clarkb | rip them. | 20:56 |
| clarkb | (its possible they may have to stop supporting this feature unless users explicitly toggle a flag?) | 20:57 |
| *** rlandy is now known as rlandy|bbl | 21:22 | |
| clarkb | since docker hub is apparently happier now I'm going to approve the change that removes the 3.8 images. I don't believe anything is running on them anymore | 21:55 |
| *** dviroel is now known as dviroel|afk | 21:57 | |
| fungi | sounds great, thanks | 22:06 |
| opendevreview | Merged opendev/system-config master: Drop python 3.8 base image builds https://review.opendev.org/c/opendev/system-config/+/861480 | 22:06 |
| opendevreview | Merged opendev/base-jobs master: Switch to bridge01.opendev.org https://review.opendev.org/c/opendev/base-jobs/+/862551 | 22:38 |
| opendevreview | Clark Boylan proposed opendev/system-config master: WIP Upgrade to Gitea 1.18 https://review.opendev.org/c/opendev/system-config/+/862661 | 23:49 |
| clarkb | I don't necessarily expect that to work yet. They didn't publish a list of change sfor 1.18.0-rc0 so I've just compared the dockerfile (new golang version) and the templates | 23:50 |
| clarkb | there is likely more that needs dealing with. | 23:50 |
| clarkb | If it does work we should be able to confirm my vendor file classification fix makes our repos look less weird | 23:52 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!