| fungi | looks like i get nsec/rrsig records back from it too | 00:21 |
|---|---|---|
| ianw | i'm not seeing anything completely obvious like "fix centos" in https://gerrit.openafs.org/#/q/status:merged | 00:51 |
| ianw | though visiting is a bit of a blast to the past | 00:52 |
| opendevreview | Ian Wienand proposed opendev/system-config master: [dnm] trying to get some logs for openafs on centos builds https://review.opendev.org/c/opendev/system-config/+/881528 | 00:55 |
| ianw | it is a build error "/var/lib/dkms/openafs/1.8.9-0.pre1.el9/build/src/libafs/MODLOAD-5.14.0-302.el9.x86_64-SP/osi_vnodeops.c:2272:20: error: implicit declaration of function ‘add_to_page_cache’; did you mean ‘add_to_page_cache_lru’" | 01:10 |
| ianw | i guess https://gerrit.openafs.org/#/c/15148/ | 01:11 |
| genekuo | ianw: It seems like this patch is waiting for a workflow+1, is there anything missing that I should do? | 02:55 |
| genekuo | https://review.opendev.org/c/opendev/system-config/+/881163 | 02:55 |
| ianw | genekuo: sorry, nope that's fine. i had held off just because i thought it might be something we'd use to test pushing our images to quay.io, clarkb is working on it. but I think it's fine to go ahead as i know he's got local testing setup etc. | 03:13 |
| genekuo | got it, thanks | 03:14 |
| opendevreview | Ian Wienand proposed opendev/system-config master: nodepool: switch images to quay.io https://review.opendev.org/c/opendev/system-config/+/881534 | 03:25 |
| ianw | sigh, i guess the openafs rpm build jobs build the rpms, but don't test the install :/ | 03:45 |
| ianw | so https://zuul.opendev.org/t/openstack/build/6e1f71776c4c4b22be8ccd1eee3d2383/console passes, but https://zuul.opendev.org/t/openstack/build/83011db0dbef443fa8a3371dce3acfb8 still fails | 03:46 |
| opendevreview | Merged opendev/system-config master: Build houndd Directly https://review.opendev.org/c/opendev/system-config/+/881163 | 04:09 |
| ianw | i may have got the naming wrong https://static.opendev.org/project/tarballs.opendev.org/openstack/openstack-zuul-jobs/openafs/centos9-stream/RPMS/x86_64/ | 04:29 |
| ianw | hrm, no the 1.8.9 rpm's arent' there | 04:33 |
| ianw | but https://zuul.opendev.org/t/openstack/build/022c65f5fdd741229ef00bf362848cd9/console copied them | 04:33 |
| ianw | ... and they are in the afs R/W volume | 04:33 |
| ianw | this implies either vos release hasn't happened, or static is showing old content | 04:34 |
| ianw | 2023-04-26 04:35:02,529 release DEBUG Running: ssh -T -i /root/.ssh/id_vos_release vos_release@afs01.dfw.openstack.org -- vos release project.tarballs | 04:35 |
| ianw | 2023-04-26 04:35:03,225 release DEBUG | 04:35 |
| ianw | 2023-04-26 04:35:03,225 release ERROR Release of project.tarballs failed | 04:35 |
| ianw | great | 04:35 |
| ianw | i wonder how related this is to the afsdb failure this morning ... | 04:36 |
| ianw | vos listvldb -locked shows only tarballs | 04:39 |
| ianw | afs-release.log.3.gz:2023-04-23 23:55:03,259 release ERROR Release of project.tarballs failed | 04:40 |
| ianw | appears to be the first error | 04:40 |
| ianw | i've unlocked it, and now it needs to do a full release :/ | 04:44 |
| ianw | it's doing it in a screen on mirror-update, but it might take a while | 04:45 |
| ianw | #status log mirror-update02 in emergency as it runs a full release of project.tarballs after the volume became locked during a prior operation | 04:45 |
| opendevstatus | ianw: finished logging | 04:45 |
| *** travissoto is now known as travisholton | 05:33 | |
| frickler | need to fix afs in order to be able to fix afs, nice ;) | 07:16 |
| frickler | seems to be still running strong, no idea how to measure progress | 07:20 |
| frickler | grafana shows afs01.ord has dropped from 262GB to 42GB two hours ago and is now slightly increasing again. if the target is the previous level, we haven't even done 10% yet | 07:25 |
| *** amoralej|off is now known as amoralej | 07:25 | |
| ianw | it's basically running at 10mbit | 08:16 |
| ianw | i think that's more or less the limit of the way it queues packets. so yeah, it probably tracks for a long time | 08:18 |
| frickler | yes, certainly not the storage solution made for high performance. did we ever consider other solutions? like maybe zfs with snapshots and replication? | 08:57 |
| opendevreview | Ching Kuo proposed opendev/system-config master: Update Hound to Use Python 3.11 Base Images https://review.opendev.org/c/opendev/system-config/+/880908 | 10:53 |
| *** amoralej is now known as amoralej|lunch | 11:02 | |
| opendevreview | Maksim Malchuk proposed openstack/diskimage-builder master: Extend the checksum files generation procedure https://review.opendev.org/c/openstack/diskimage-builder/+/881298 | 11:38 |
| *** amoralej|lunch is now known as amoralej | 12:10 | |
| clarkb | frickler: I don't think zfs was ever considered due to it not having a good linux story when this was built and it still has licensing concerns (though none any worse than openafs aiui) | 15:15 |
| clarkb | but also I'm not sure zfs send is really a good substitute. We would need to be able to have 3TB of disk in every location | 15:16 |
| frickler | yeah, likely not enough pain with afs yet to really dig deeper into this | 15:23 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Update ensure-quay-repo to run opportunistically https://review.opendev.org/c/zuul/zuul-jobs/+/881521 | 16:04 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Use consistent registry_type var name across roles https://review.opendev.org/c/zuul/zuul-jobs/+/881524 | 16:04 |
| opendevreview | Clark Boylan proposed opendev/system-config master: WIP Base jobs for quay.io image publishing https://review.opendev.org/c/opendev/system-config/+/881285 | 16:07 |
| clarkb | ianw: fungi: are we to the point where we should ask the foundation to update NS and glue records for opendev.org and zuulci.org? | 16:18 |
| fungi | was the plan to do that before changing the ns records and soa in our zone files? | 16:28 |
| fungi | because if not, we haven't done that yet | 16:29 |
| clarkb | looking at the etherpad the plan was to update the zones first on our end | 16:33 |
| clarkb | Doesn't look like that step has been updated on the etherpad for as far as it got yesterday | 16:34 |
| clarkb | so ya I think we need to land those changes first then talk to csc/foundation | 16:34 |
| fungi | agreed | 16:43 |
| fungi | otherwise the registrars will likely balk at making the change since it's not set in our zones | 16:43 |
| fungi | i can take a look in a few and approve/rebase/propose those as needed | 16:44 |
| clarkb | fungi: k. I'm happy to wait a few hours if we want ot sync up with ianw too first. I was mostly trying to figureo ut where things were and to ensure I wasn't holding anything up between opendev and the registrar | 16:45 |
| *** amoralej is now known as amoralej|off | 16:49 | |
| fungi | sure, i can hold off approving and just check for presence and state of those changes | 16:52 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Switch to nodepool images on quay.io https://review.opendev.org/c/opendev/system-config/+/881591 | 17:41 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Switch zuul container images to quay.io https://review.opendev.org/c/opendev/system-config/+/881592 | 17:41 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Switch the zuul-registry image location to quay.io https://review.opendev.org/c/opendev/system-config/+/881593 | 17:42 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Cleanup unused nodepool-base-legacy role https://review.opendev.org/c/opendev/system-config/+/881594 | 17:43 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Switch jeepyb over to Gerrit 3.7 image builds https://review.opendev.org/c/openstack/project-config/+/881595 | 17:49 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Remove Gerrit 3.6 image builds and test jobs https://review.opendev.org/c/opendev/system-config/+/881596 | 17:54 |
| opendevreview | Ghanshyam proposed openstack/project-config master: Correct the patrole repo gerrit acl to openstack/retired.config https://review.opendev.org/c/openstack/project-config/+/881598 | 18:01 |
| gmann | fungi: frickler ^^ I updated patrole repo acl wrongly. fixing that so that i can close the open reviews in that repo | 18:02 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add Gerrit 3.8 image builds and test jobs https://review.opendev.org/c/opendev/system-config/+/881599 | 18:03 |
| clarkb | I'm going to pause here on the gerrit stuff. Want to make sure 3.8 builds at all and can be deployed from scratch before we start worrying about the upgrade job | 18:06 |
| clarkb | oh and I need the project-config update to jeepyb to land before the system-config changes are even testable | 18:08 |
| clarkb | I'll plan to clean up etherpad01 after lunch today so in about an hour and a half or so | 18:22 |
| JayF | Are things cached on specs.openstack.org? | 18:30 |
| JayF | https://review.opendev.org/c/openstack/ironic-specs/+/879381 landed, post jobs are posted there all successful, but https://specs.openstack.org/openstack/ironic-specs/ the link is not here, and if I manually construct the URL, the workstreams spec is not there | 18:30 |
| clarkb | JayF: the content is likely hosted in afs and gets written to the RW volume. Then periodically the RW volume gets promoted to the RO volumes and the RO volumes are what we serve | 18:31 |
| JayF | ack; so just need to wait for someone to release the vos (? it's been a while since I admin'd AFS :D) | 18:31 |
| clarkb | I believe this happens roughly every 5 minutes, but there was something with the tarballs volume in scrollback needing to be resynced from scratch and that taking time whihc may have backed things up | 18:31 |
| clarkb | JayF: ya its automated vos releases but with locking to avoid stepping on each other. I think the resync of tarballs has the others backed up behind it | 18:32 |
| JayF | sure, it's not a rush to get this published was just confused when I was gonna go reference it :D | 18:32 |
| frickler | releases are blocked by the resync for maybe another day or so | 18:32 |
| clarkb | we could do a manual vos release out of band maybe since it is for a different volume | 18:33 |
| JayF | If you wanna do it just to do it; feel free. It causes me no pain, frustration, or delay to wait a day or two :) | 18:33 |
| fungi | fwiw, waiting on the tarballs volume resync before updating the releases site is best anyway, since otherwise the releases site is going to mention and link to new releases on the tarballs site that aren't there yet | 18:42 |
| opendevreview | Merged openstack/project-config master: Correct the patrole repo gerrit acl to openstack/retired.config https://review.opendev.org/c/openstack/project-config/+/881598 | 18:54 |
| fungi | clarkb: ianw: looks like the current state is that gating.dev is already listing ns03/ns04 as additional ns records, while 880577 for opendev.org and 880909 for zuul-?ci.org have 2x+2 and are ready to merge | 19:28 |
| clarkb | fungi: can you review https://review.opendev.org/c/openstack/project-config/+/881595 that will unstick the followup changes for updating gerrit image stuff | 19:49 |
| fungi | yeopp | 19:50 |
| opendevreview | Merged openstack/project-config master: Switch jeepyb over to Gerrit 3.7 image builds https://review.opendev.org/c/openstack/project-config/+/881595 | 19:57 |
| * clarkb rechecks things | 20:00 | |
| clarkb | I'm looking at etherpad01 cleanup and have noticed that openstack server show is buggy anddoes not list attached volumes | 20:07 |
| clarkb | infra-root ^ a warning as that could be potentially dangerous | 20:07 |
| clarkb | infra-root last call to say don't delete etherpad01 and its volume. I'll get to that in about 10 minutes | 20:07 |
| clarkb | I hear no objections. Proceeding now | 20:17 |
| clarkb | #status log Deleted etherpad01.opendev.org (648795e3-a523-4998-8256-8e40c6e6f222) and its volume (020a2963-1d11-4665-bfdf-1fefb74c8a9f) to complete the etherpad server replacement and cleanup | 20:21 |
| opendevstatus | clarkb: finished logging | 20:21 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add Gerrit 3.8 image builds and test jobs https://review.opendev.org/c/opendev/system-config/+/881599 | 20:33 |
| fungi | thanks! | 20:33 |
| clarkb | Looking at the 3.8 release notes I think the upgrade process itself is going to be simpler than 3.7s but I haven't digested what all the changes we need to accomodate pre upgrade are yet | 20:39 |
| clarkb | cool I think the gerrit 3.8 war is building now | 21:00 |
| ianw | tarballs release still running | 21:09 |
| ianw | since things went ok with gating.dev i'll approve the other NS addition changes now. then the registrars can be updated at leisure | 21:10 |
| clarkb | ianw: re tarballs what precipitated that? a stale lock? | 21:11 |
| opendevreview | Merged opendev/zone-opendev.org master: Add Jammy refresh NS records https://review.opendev.org/c/opendev/zone-opendev.org/+/880577 | 21:13 |
| ianw | clarkb: yeah the volume was locked. the first afs-release that failed was "afs-release.log.3.gz:2023-04-23 23:55:03,259 release ERROR Release of project.tarballs failed" | 21:13 |
| ianw | so something must have happened to the release before that? | 21:13 |
| clarkb | gotcha | 21:14 |
| opendevreview | Merged opendev/zone-zuul-ci.org master: Add Jammy refresh NS records https://review.opendev.org/c/opendev/zone-zuul-ci.org/+/880909 | 21:17 |
| ianw | fungi/clarkb: opendev.org and zuulci.org have the 03/04 NS records now, so they are safe to be switched at the registrar when ready | 21:24 |
| clarkb | ianw: thanks fungi mentioned he could coordinate as I have a school function in a little bit | 21:26 |
| ianw | no rush | 21:26 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add Gerrit 3.7 -> 3.8 upgrade job https://review.opendev.org/c/opendev/system-config/+/881647 | 21:32 |
| clarkb | ianw: to be clear we need to delete opendev.org NS ns1.opendev.org and opendev.org NS ns2.opendev.org records and add opendev.org NS ns03.opendev.org and opendev.org NS ns04.opendev.org records? | 21:33 |
| clarkb | then swap out opendev.org with zuulci.org at the beginning of the records for that domain? | 21:34 |
| clarkb | I want to make sure that we request everything we need clearly for them | 21:34 |
| fungi | yeah, i think that's what we expect, but will give foundation folks a heads up as soon as ianw confirms | 21:41 |
| clarkb | our gerrit theme plugin js will need to be updated: https://463c9e079fe7143c6d37-383420cf0a891de3c4e4fd7735a83cf2.ssl.cf5.rackcdn.com/881599/2/check/system-config-run-review-3.8/f242dd0/bridge99.opendev.org/screenshots/gerrit-main-page.png | 21:56 |
| *** dmitriis is now known as Guest12271 | 22:01 | |
| clarkb | https://zuul.opendev.org/t/openstack/build/2b01997d0f794f8c9cc75db93b18631e but we can upgrade from 3.7 to 3.8 apparently | 22:01 |
| clarkb | fungi: ^ I think you can slap a DNM change on top of that and hold the 3.8 job and check the gitea links | 22:02 |
| ianw | clarkb: yep, we need to replace ns1/ns2 with ns03/ns04, and update glue records for both. | 22:03 |
| clarkb | fungi: ^ theres the confirmation | 22:04 |
| ianw | i don't know if we want to bring it up, zuulci.org probably doesn't need glue records, it would be one less thing to manage next time | 22:04 |
| fungi | thanks ianw, i'll let them know now (no idea how long the change will take to complete) | 22:04 |
| clarkb | and i guess make note that ns1/ns2 don't have the 0 prefix in the digits but ns03 and ns04 do | 22:05 |
| ianw | there's no rush, we just don't want to turn off the old servers until done :) | 22:05 |
| fungi | clarkb: yep | 22:05 |
| fungi | clarkb: i had included that in the message i was hovering over the send button for | 22:05 |
| fungi | ianw: you mean cease to serve zuulci.org? otherwise we should still maintain its domain registration, and so specify nameservers for it | 22:06 |
| clarkb | fungi: cease serving the NS records for zuulci.org out of the .org domain | 22:07 |
| clarkb | and let them get served out of zuulci.org instead I think | 22:07 |
| fungi | usually the registrar handles figuring that part out | 22:08 |
| clarkb | fwiw I can never remember exactly when those records are necessary in the parent domain | 22:08 |
| clarkb | chickens and eggs are confusing | 22:08 |
| fungi | at least i've never used a registrar where you independently altered your whois and requested glue record injection into the tld zone | 22:09 |
| fungi | usually you just say "these are my new nameservers" and then they update the whois and also inject glue records if they're needed | 22:09 |
| ianw | oh that may be the case. porkbun we've been talking about does let you set glue records, if you want | 22:10 |
| ianw | so yeah, they may be set automatically | 22:10 |
| ianw | dig +noall +authority +additional +norecurse @a0.org.afilias-nst.info. NS zuulci.org | 22:10 |
| fungi | typically, glue records are only injected into the tld zone if they're within the same domain or a subdomain | 22:10 |
| fungi | but some registrars may just do it for all domains regardless | 22:11 |
| ianw | right, they make sense for opendev.org, but zuulci.org has them too | 22:11 |
| clarkb | infra-root I think the gerrit image cleanup and addition of 3.8 stuff is good to go now: https://review.opendev.org/c/opendev/system-config/+/881596/ and children. Fixing up the issues in 3.8 should happen separately | 22:11 |
| clarkb | isn't there still a chicken and egg when you query NS zuulci.org. .org has to point you at ns03.opendev.org otherwise ou won't know what to talk to? | 22:12 |
| ianw | thanks! will look | 22:12 |
| clarkb | anyway we don't need to get into all those details | 22:12 |
| ianw | yeah, but it can look up ns03.opendev.org | 22:13 |
| fungi | clarkb: the ns records need to be in the tld zone, but not any glue records (a/aaaa) | 22:13 |
| clarkb | aha | 22:13 |
| ianw | whereas when opendev.org says "ask ns03.opendev.org" there's a loop | 22:13 |
| fungi | yep | 22:14 |
| fungi | all of the registrars i've ever worked with don't give you a separate choice to inject glue records, presumably because dns and domain registration are already confusing enough and they just want customers to give them (recurring) revenue without incurring additional support questions from people who don't grok the distinction | 22:16 |
| opendevreview | Clark Boylan proposed zuul/zuul-jobs master: Use consistent registry type var name across roles https://review.opendev.org/c/zuul/zuul-jobs/+/881524 | 22:34 |
| clarkb | I don't actually know if that will make the linter happy. We will find out | 22:34 |
| ianw | hrm, i hope that some function has just changed name and it's not a whole rewrite of the zuul plugin | 23:28 |
| ianw | Remove registerStyleModule() plugin API -- Use plugin.styleApi().insertCSSRule() instead. | 23:33 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
