| Unit193 | I guess if that's linked, https://almalinux.org/blog/impact-of-rhel-changes/ might be useful too? | 00:45 |
|---|---|---|
| NeilHanlon | yep, thanks Unit193! | 01:00 |
| frickler | mnasiadka: ubuntu-ports should have been up2date again since about 0UTC | 06:19 |
| frickler | starting afs volume additions now | 06:19 |
| frickler | humm, seems our OSC version doesn't like to create volumes on rackspace: VolumeManager.create() got an unexpected keyword argument 'backup_id' | 06:23 |
| frickler | gotta check some old venvs | 06:23 |
| frickler | o.k., what I have in ~/venv does a better job | 06:24 |
| frickler | there's some inconsistency in our setup, the volumes are called afs0?.dfw.opendev.org/main?? but the servers are still afs0?.dfw.openstack.org | 06:30 |
| frickler | not sure if renaming the volumes would be a good idea. I did rename one of the new volumes which I incidentally created with a wrong name first, so it should work fine technically | 06:31 |
| frickler | "pvs" on afs02 says for all pv: WARNING: PV /dev/xvdh1 in VG main is using an old PV header, modify the VG to update. | 06:33 |
| frickler | guess that should clean up once I actually update the vg, we'll see. I'll still start with afs01 | 06:34 |
| mnasiadka | frickler: thanks | 06:38 |
| frickler | /dev/mapper/main-vicepa 5.0T 3.6T 1.4T 72% /vicepa | 06:47 |
| frickler | I'll wait a bit just in case before I move to afs02 | 06:48 |
| frickler | after doing the vgextend on afs02, the warning about old headers is gone as expected | 07:12 |
| frickler | o.k., done, everything lgtm, just grafana seem to need some time to pick up the new used% data | 07:19 |
| frickler | is anyone else seeing gerrit slowness? might be my local connection only | 07:25 |
| frickler | #status log added another 1T volume for /vicepa on afs01/02.dfw to increase AFS capacity | 07:38 |
| opendevstatus | frickler: finished logging | 07:38 |
| frickler | I've prepared a list of quota adjustments at https://etherpad.opendev.org/p/afs-mirror-quota, those would in preparation of mirroring bookworm use up 590 of the 1024 newly added Gs | 08:13 |
| opendevreview | Caio Cesar Ferreira proposed openstack/project-config master: Add Power Metrics app to StarlingX https://review.opendev.org/c/openstack/project-config/+/886814 | 11:36 |
| frickler | clarkb: fungi: please have a look at the quota adjustments and then the bookwork mirror patch https://review.opendev.org/c/opendev/system-config/+/881952 if we merge that soonish I'd start the initial run in a screen with log held and watch it over the weekend | 14:08 |
| frickler | bookworm even | 14:08 |
| frickler | https://review.opendev.org/c/opendev/system-config/+/886259 would also be nice, although likely we have a couple of months now again | 14:12 |
| frickler | also it seems we've lost track of https://review.opendev.org/c/openstack/project-config/+/879722, was there any reason not to merge it? guess I'll just approve it ... | 14:16 |
| opendevreview | Merged openstack/project-config master: wheel builds : move to individual releases https://review.opendev.org/c/openstack/project-config/+/879722 | 14:35 |
| fungi | frickler: proposed adjustments lgtm, though any idea why the centos volume needs so much more space than centos-stream? makes me suspect we missed cleaning something up | 14:37 |
| fungi | aha, i guess we still have 7 mirrored | 14:38 |
| fungi | and we still have centos-7 nodes | 14:39 |
| frickler | so more cleanup potential ahead, yay | 14:39 |
| frickler | I'll go ahead with the quota adjustsments, then | 14:40 |
| fungi | thanks for working on all that! | 14:41 |
| fungi | we can also reduce quotas in some of those volumes later if we clean up enough content | 14:44 |
| fungi | the quotas are mainly serving as a backstop so that something going haywire on some particular mirror can't fill up the disk and impact other unrelated volumes | 14:44 |
| frickler | yes, will recheck in particular once we know how much space the bookwork mirrors really need | 15:05 |
| frickler | #status log updated quota for multiple AFS mirror.* volumes to adapt to current demand | 15:06 |
| opendevstatus | frickler: finished logging | 15:06 |
| clarkb | frickler: thank you for taking care of that | 16:22 |
| clarkb | tonyb: I approved the insecure ci registry test fix. Thank you for digging into that and apologies it took me so long to review it | 16:31 |
| clarkb | fungi: frickler: next up for insecure ci registry is launching a new jammy node and updating https://review.opendev.org/c/opendev/system-config/+/885421 to add it into the inventory then also pushing dns updates. Any interest in that? I think corvus, ianw and I have launched nodes recently and want to make sure this opportunity doesn't slip by if you want ot make sure you're still | 16:33 |
| clarkb | familiar with it | 16:33 |
| opendevreview | Merged opendev/system-config master: Update testing for insecure-ci-registry https://review.opendev.org/c/opendev/system-config/+/885804 | 16:56 |
| fungi | i have to run some pre-travel errands in about 30 minutes, but should be able to knock that out when i get back | 17:00 |
| fungi | from the errand-running | 17:00 |
| frickler | yeah, pretty sure there'll be other opportunities for node launches. also still waiting for reviews on https://review.opendev.org/c/opendev/system-config/+/881952 after I almost decided to self-approve this morning | 17:18 |
| fungi | i approved it just now after double checking the official mirrors, but we'll want to check the reprepro logs from the next pass after that deploys | 17:28 |
| fungi | actually, we'll probably want to hold the lock for the debian cronjob on mirror-update.o.o and run the first pass manually in screen with timeouts turned off | 17:34 |
| fungi | i'll go ahead and grab the lock in a screen session | 17:35 |
| fungi | infra-root: i did `flock -n /var/run/reprepro/debian.lock bash` in a root screen session on mirror-update.o.o just now | 17:36 |
| clarkb | did you make sure you grabbed it? I think it doesn't give much indication if it fails since you are already typicalyl running bash | 17:36 |
| fungi | yeah, there was no session running at the time | 17:36 |
| fungi | no cronjob running for it, i mean | 17:37 |
| fungi | anyway, heading out to run errands now, but can help with the initial mirror run for that in an hour or so | 17:37 |
| clarkb | thanks! | 17:37 |
| opendevreview | Merged opendev/system-config master: reprepro: add Debian Bookworm config https://review.opendev.org/c/opendev/system-config/+/881952 | 17:56 |
| fungi | i'm back, starting the mirror run since the deploy succeeded a few minutes ago | 19:11 |
| fungi | and it's going now | 19:12 |
| fungi | it's on http://ftp.us.debian.org/debian/pool/main/c/ so going to be quite a while, i think | 19:48 |
| clarkb | not very zoom zoom | 20:03 |
| clarkb | I'm launching a new ci registry momentarily | 20:07 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Deploy insecure-ci-registry.opendev.org on jammy https://review.opendev.org/c/opendev/system-config/+/885421 | 20:18 |
| opendevreview | Clark Boylan proposed opendev/zone-opendev.org master: Add DNS records for insecure-ci-registry02 https://review.opendev.org/c/opendev/zone-opendev.org/+/886874 | 20:21 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Deploy insecure-ci-registry.opendev.org on jammy https://review.opendev.org/c/opendev/system-config/+/885421 | 20:22 |
| opendevreview | Clark Boylan proposed opendev/zone-opendev.org master: Remove insecure-ci-registry01 and update CNAME to 02 https://review.opendev.org/c/opendev/zone-opendev.org/+/886875 | 20:23 |
| clarkb | tonyb: ^ fyi something like that to finish up this process | 20:23 |
| opendevreview | Merged opendev/zone-opendev.org master: Add DNS records for insecure-ci-registry02 https://review.opendev.org/c/opendev/zone-opendev.org/+/886874 | 20:42 |
| fungi | debian main mirroring is finally past l/lib... | 22:07 |
| corvus | i've been seeing multiple post failures on nodepool jobs during the cleanup phase | 22:09 |
| corvus | er, sorry, zuul and nodepool upload-image jobs | 22:10 |
| corvus | https://zuul.opendev.org/t/zuul/builds | 22:11 |
| corvus | looks like every "-image" job is hitting it | 22:11 |
| Clark[m] | In cleanup? I didn't think that was supposed to be able to break jobs | 22:12 |
| corvus | oh yep that's a red herring | 22:14 |
| corvus | it's a timeout on push to intermediate registry | 22:14 |
| Clark[m] | Looking at the log it ya that | 22:14 |
| corvus | i saw some intermediate registry chat earlier -- anything happening with thatL | 22:15 |
| Clark[m] | I want to say when we've seen this before we have had to restart the daemon for the registry because cherry py gets sad | 22:15 |
| corvus | ? | 22:15 |
| Clark[m] | No just DNS for a new server but the new server hasn't been added to inventory yet | 22:15 |
| Clark[m] | It shouldn't affect anything yet | 22:15 |
| fungi | we haven't replaced the registry server yet, no | 22:15 |
| fungi | new dns records were added, the cname is still untouched | 22:15 |
| corvus | cool, i'll check logs and (probably) restart | 22:16 |
| fungi | the inventory addition is in the gate | 22:16 |
| fungi | cname change will be ready to approve once that deploys | 22:16 |
| corvus | ssl.SSLError: [SSL: NO_SUITABLE_KEY_SHARE] no suitable key share (_ssl.c:1007) | 22:16 |
| corvus | 2023-06-23 17:15:06,189 ERROR cherrypy.error: [23/Jun/2023:17:15:06] ENGINE Error in HTTPServer.serve | 22:16 |
| fungi | ew | 22:17 |
| corvus | yeah i think that's one of the "the ssl subsystem in cherrypy is hung" errors | 22:17 |
| corvus | connection attempts were hanging, so i kicked it. better now | 22:18 |
| fungi | thanks! | 22:18 |
| opendevreview | Merged opendev/system-config master: Deploy insecure-ci-registry.opendev.org on jammy https://review.opendev.org/c/opendev/system-config/+/885421 | 22:42 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!