| mnasiadka | Hello | 11:22 |
|---|---|---|
| mnasiadka | Can somebody check if Ubuntu/aarch64 mirror is being synced properly? I've been seeing this for the last week or so: https://d4834ede78e2cbe3c327-9f5a5e2736e26bdd8715596753fafe10.ssl.cf1.rackcdn.com/909915/10/check-arm64/kolla-build-ubuntu-aarch64/a31a841/kolla/build/000_FAILED_openstack-base.log | 11:22 |
| frickler | mnasiadka: there's some "WRONG SIZE" messages in https://mirror.iad3.inmotion.opendev.org/logs/reprepro/ubuntu-ports.log , not sure if that's a sign of some broken upstream mirror or some other issue, will try to dig further later today | 12:02 |
| mnasiadka | frickler: thanks | 12:02 |
| *** d34dh0r5- is now known as d34dh0r53 | 13:39 | |
| frickler | Warning parsing /etc/reprepro/ubuntu/updates, line 21: unknown architecture 'arm64' will be ignored! | 14:19 |
| frickler | that's from https://mirror.regionone.linaro.opendev.org/logs/reprepro/ubuntu.log and is the location for the "basic" ubuntu-ports mirror, together with line 29 for which this message also appears | 14:20 |
| frickler | infra-root: ^^ did something change about our mirror setup recently? | 14:21 |
| fungi | we mirror ubuntu-ports separately | 14:23 |
| fungi | are you sure those warnings are new? | 14:24 |
| frickler | fungi: hmm, no I'm not and likely our logs aren't going far enough back | 14:25 |
| fungi | https://static.opendev.org/mirror/logs/reprepro/ubuntu-ports.log has the arm64 mirroring | 14:25 |
| fungi | and yeah, i see the "wrong size of" errors you mentioned there | 14:25 |
| fungi | oh, never mind, i see you also found the same log | 14:26 |
| fungi | lots of zero-byte files | 14:26 |
| fungi | though also at least one shorter-than-expected file | 14:27 |
| fungi | maybe we ran out of quota in that volume? | 14:27 |
| fungi | mmm, nope, i upped it (last week? the week before?) so it's only at 93% used now | 14:28 |
| fungi | i need to go run some errands (should be gone less than an hour), but can also try to look more closely when i get back | 14:29 |
| frickler | oh, but last release is 11 days ago, so that situation may be from right before that quota bump | 14:29 |
| fungi | aha, maybe we need to force it to redownload those files | 14:29 |
| fungi | good eye | 14:29 |
| frickler | can I just delete them manually or how would you force that? | 14:30 |
| fungi | it's possible they were truncated when the volume exceeded quota and reprepro doesn't redownload files it finds in the local copy | 14:30 |
| fungi | yeah, i'd just delete them from the writeable volume and rerun the reprepro script | 14:30 |
| frickler | ok. I'll look into that | 14:31 |
| clarkb | for the record I haven't touched ubuntu mirroring (either x86 or arm64) yet as part of the cleanups I've done | 15:07 |
| clarkb | only debian and suse | 15:07 |
| fungi | #status log Pruned backup volume on backup02.ca-ymq-1.vexxhost.opendev.org reducing it from 92% to 69% utilization | 15:23 |
| opendevstatus | fungi: finished logging | 15:23 |
| fungi | i'm actually going to disappear again in a few minutes to grab an early lunch since there don't seem to be any new emergencies | 15:24 |
| clarkb | enjoy | 15:25 |
| fungi | thanks! | 15:25 |
| frickler | hmm, now reprepro says: Missing file '/afs/.openstack.org/mirror/ubuntu-ports/pool/main/m/mesa/libd3dadapter9-mesa_23.2.1-1ubuntu3.1~22.04.2_arm64.deb'! | 15:26 |
| frickler | although it is still running, maybe it will try to refetch later? | 15:27 |
| clarkb | frickler: ya it wouldn't surprise me if it is just making note of that so that it knows to fetch it later | 15:31 |
| frickler | no, it finished. will try to read reprepro docs before I try to download these manually | 15:31 |
| frickler | looks like we want "$REPREPRO _forget $file" in addition to the rm | 15:33 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update Gerrit 3.9 images to 3.9.2 https://review.opendev.org/c/opendev/system-config/+/912470 | 15:40 |
| frickler | seems that that worked, rerunning in a tmux session on mirror-update | 15:40 |
| opendevreview | Merged openstack/diskimage-builder master: feat: add almalinux-container element https://review.opendev.org/c/openstack/diskimage-builder/+/883855 | 16:03 |
| frickler | looks like I missed to forget one file, will need to wait for the cron run to finish now | 16:18 |
| opendevreview | Merged openstack/diskimage-builder master: Call grub2-mkconfig with --update-bls-cmdline https://review.opendev.org/c/openstack/diskimage-builder/+/910977 | 16:24 |
| fungi | frickler: thanks, i'm around now and can help/take over if you need | 16:51 |
| frickler | fungi: running the next attempt right now, hoping that will result in a new error at least | 16:53 |
| fungi | cool, sounds like you're close at least, and good to know that reprepro needs to be told to forget specific files even if they're deleted (or is this a workaround to not need to manually delete those files?) | 16:56 |
| frickler | fungi: it's actually in our docs to do both for corrupted .deb files, somewhere deep down in https://docs.opendev.org/opendev/system-config/latest/reprepro.html#advanced-recovery-techniques | 16:58 |
| fungi | oh wow, i missed we have that | 17:05 |
| clarkb | infra-root I learned about the bcc eBPF tooling over the weekend. I think it will be useful for profiling devstack and maybe zuul ci jobs. I think there is also a possibility that these tools could be useful when debugging our own systems. Wanted to throw the idea of possibly installing the tools on our control plane servers | 17:52 |
| fungi | profiling general syscall activity, or network-specific? | 17:54 |
| fungi | bpf always makes me think network since that's what it was primarily intended for, but i gather its utility has expanded | 17:55 |
| clarkb | fungi: eBPF doesn't stand for extended berkley packet filter anymore because it is generic. And ya bcc can do a lot of stuff out of the box https://github.com/iovisor/bcc/tree/master/tools | 17:55 |
| clarkb | fungi: not just expanded, completely outgrown | 17:56 |
| clarkb | there is even a tool in there to report slow mysql queries | 17:56 |
| fungi | got it. my first experiences with bpf were firewalling hooks | 17:56 |
| clarkb | for devstack the interesting stuff is disk/block/file io related | 17:56 |
| clarkb | the change I'm using to prove it out is starting with swapin counts for processes and an oomkiller detector | 17:57 |
| fungi | oh awesome | 17:57 |
| clarkb | there is also language vm (python, java, ruby, perl, php, etc) monitoring that I think will be useful for zuul to make sense of why its jobs sometimes just timeout | 17:57 |
| clarkb | since zuul is going to largely be bound by python and java | 17:58 |
| fungi | i guess it still needs some sort of bucketing/analysis tools | 17:58 |
| clarkb | ya its mostly recording info with some tools optionally recording in csv and json | 17:59 |
| clarkb | but some of them also report histograms using ascii graphs | 17:59 |
| clarkb | the oomkill one just reports a short summary on oomkiller occurences which I think could be helpful for devsatck because its a single log file that will tell you what triggered oomkiller and what was chosen to be killed | 18:00 |
| clarkb | you won't have to dig through syslog for that info, makes it much more clear | 18:00 |
| clarkb | for our production nodes its more likely that we'd run the java vm monitor when things are slow in gerrit or zookeeper and have it reports GCs | 18:01 |
| clarkb | or look at io health when haproxy stops responding | 18:01 |
| clarkb | etc | 18:01 |
| fungi | and the python analyzer for devstack i suppose | 18:04 |
| clarkb | I'm worried that may produce too much data | 18:04 |
| clarkb | but I could try it and see how it goes | 18:04 |
| fungi | might make more sense in per-project unit tests, yeah | 18:04 |
| clarkb | oddly newer ubuntu and debian packaging seems to not include the swapin tool | 18:44 |
| clarkb | the man page and example docs are in the package but not hte tool itself | 18:44 |
| clarkb | and older distros don't seem to want to have linux headers installed by default (or as a package dep) so bullseye fails entirely. Weird | 18:44 |
| fungi | usually you only install the kernel headers if you're building out-of-tree modules or rebuilding kernels | 18:47 |
| clarkb | ya in this case you need them because eBPF needs to "link" to them | 18:48 |
| clarkb | it seems to just work on the nweer distros but not the older ones. So I guess they streamlined that stuff | 18:48 |
| fungi | er, i guess you don't technically even need them for kernel rebuilds since you rebuild new headers in that case. it's really usually just for modules | 18:49 |
| clarkb | I may just ignore the older releases and keep it simple. But also I'm having no luck finding out why swapin is not installed in newer noble or bookworm packaging | 18:50 |
| frickler | ok I give up on the ports mirror for today. I removed all the files repropro complained about, told it to forget about them, but it still complains about two of the files not being found. happy for someone else to pick this up | 18:56 |
| fungi | i'll take a shot at it, thanks again frickler! | 19:00 |
| clarkb | fungi: if you look at the bpfcc-tools package on debian bookworm does anything stand out to you as for why tools/swapin isn't being installed? | 20:05 |
| clarkb | noble doesn't have it either but jammy does so I don't think this is a newer tool that was added and just hasn't been packaged yet | 20:06 |
| fungi | looks like it comes with the bpftrace package maybe? | 20:07 |
| clarkb | oh weird | 20:08 |
| fungi | https://packages.debian.org/bookworm/amd64/bpftrace/filelist | 20:09 |
| fungi | shows /usr/sbin/swapin.bt if that's what you're looking for | 20:10 |
| clarkb | sort of. Its a different implementation but it provides the same info | 20:10 |
| clarkb | I can just use that implemtnation instead | 20:11 |
| fungi | oh, it's possible newer bpfcc has different options | 20:12 |
| clarkb | fungi: well the package in bookworm packages the manpage and examples for swapin but not the command itself | 20:13 |
| clarkb | I can just fallback to bpftrace swapin.bt though. Reading the source I definitely prefer bcc. Its actually a bit lower level but relies on python and C both things I understand whereas the bt stuff is magic | 20:13 |
| fungi | yeah, also they have an example txt file i saw | 20:13 |
| clarkb | I wonder if this is another case of inappropriate handling for conflicts | 20:14 |
| fungi | very well could be | 20:15 |
| clarkb | rather than install under different names we just can't install the things together | 20:15 |
| fungi | you might find a bug report reflecting that | 20:15 |
| clarkb | fungi: I couldn't find any bugs fwiw | 21:06 |
| clarkb | not a big deal I think the latest patchset should work around it | 21:06 |
| clarkb | fungi: https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/912469 did pass ci so all we needed was the new kernel images | 21:06 |
| clarkb | er new kernels in rebuilt images | 21:06 |
| fungi | oh cool | 21:16 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Add setuptools for python3.12 support in venvs https://review.opendev.org/c/openstack/diskimage-builder/+/902497 | 22:10 |
| clarkb | I've updated the meeting agenda with the stuff I'm aware of | 22:37 |
| clarkb | please add other items or let me know what edits I should add then I'll send that out as my day ends | 22:37 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!