| *** jpena|off is now known as jpena | 08:01 | |
| *** ykarel is now known as ykarel|lunch | 09:01 | |
| *** ykarel|lunch is now known as ykarel | 10:27 | |
| *** d34dh0r5- is now known as d34dh0r53 | 13:45 | |
| opendevreview | Julia Kreger proposed openstack/diskimage-builder master: Correct boot path to cover FIPS usage cases https://review.opendev.org/c/openstack/diskimage-builder/+/876192 | 14:42 |
|---|---|---|
| clarkb | I think the auto update of our haproxy deployment may have reenabled the gitea01-04 backends (though cacti show them as fairly idle the container restarted). I'll push up a change to get those pulled out of config for haproxy as soon as local updates and reboots complete | 15:20 |
| fungi | great catch, i hadn't thought to check back behind that | 15:28 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Remove gitea01-04 from our haproxy config https://review.opendev.org/c/opendev/system-config/+/877296 | 15:33 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Remove gitea01-04 from Gerrit replication https://review.opendev.org/c/opendev/system-config/+/877298 | 15:35 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Remove gitea01-04 from configuration management https://review.opendev.org/c/opendev/system-config/+/877301 | 15:38 |
| clarkb | Note the depends on in ^ is something we can probably land today? | 15:38 |
| fungi | yep, approved it | 15:41 |
| clarkb | thanks. | 15:42 |
| clarkb | https://www.phoronix.com/news/ipmitool-GitHub-Suspended cc JayF | 16:43 |
| fungi | oof, that's no good | 16:51 |
| opendevreview | Merged opendev/system-config master: Switch borg backup from gitea01 to gitea09 https://review.opendev.org/c/opendev/system-config/+/876471 | 16:52 |
| fungi | system-config-run-base failing again. looking into it | 16:55 |
| fungi | oh, failed in check, rogue vm in rax-iad from the looks of it | 16:58 |
| clarkb | oh is jayf out this week? I should probably share that in the ironic channel | 17:05 |
| fungi | right, he did post to the ml saying he'd be gone | 17:12 |
| clarkb | that tox + pytest situation is really interesting. More reason to use nox I guess <_< | 17:17 |
| opendevreview | Clark Boylan proposed opendev/git-review master: Switch from tox to nox https://review.opendev.org/c/opendev/git-review/+/871652 | 17:19 |
| clarkb | thats a rebase to address a conflict introduced by the gerrit 3.4.4 swap | 17:19 |
| *** jpena is now known as jpena|off | 17:20 | |
| opendevreview | Clark Boylan proposed opendev/git-review master: Switch to Gerrit 3.7 in testing https://review.opendev.org/c/opendev/git-review/+/877313 | 17:21 |
| clarkb | and ^ issomething I meant to do previously just to see if it owrks at all | 17:21 |
| fungi | will be interesting to see the results of that | 17:21 |
| clarkb | fungi: the git-review + gerrit 3.7 change passes testing | 18:01 |
| clarkb | personally I think I lean towards ensuring git-review works with newer gerrit since we're not making many changes to git-review at this point but gerrit does make changes. It also helps ensure we don't fall behind again. But there is some value in ensuring you're compatible with old gerrit too particularly when you do make functional chagnes to git-review | 18:02 |
| fungi | we could also parameterize it i suppose, and test both? | 18:03 |
| fungi | maybe drop the intermediate python versions, which would keep the job count the same | 18:03 |
| fungi | so we test oldest/newest python with oldest/newest gerrit | 18:04 |
| fungi | probably worth bumping the upper python version too, at this point | 18:04 |
| clarkb | yup that is an option | 18:05 |
| fungi | should probably decide on the switch to nox first though, in order to not create more churn for tox config | 18:05 |
| clarkb | something like python3.6 + Gerrit 3.4 and python3.6 + Gerrit 3.7 and python3.11 + Gerrit 3.4 and python3.11 + Gerrit 3.7 | 18:06 |
| fungi | yes, exactly | 18:06 |
| clarkb | yup. I'm personally comfortable running nox locally. I also like that nox seems to be simpler but with more expressive tooling when necessary | 18:06 |
| fungi | probably no need to override the golden site number, just make it so we can pass in a warfile url override? | 18:06 |
| clarkb | ya I think you define a default and only bump the golden site number when that default changes. | 18:07 |
| clarkb | but allow overrides | 18:07 |
| fungi | right, just have to decide whether we want that default to be old gerrit or new gerrit | 18:07 |
| fungi | i guess it's a question of which would we prefer people test with locally | 18:07 |
| clarkb | no real preference there if CI checks the bounds | 18:09 |
| fungi | i guess keeping the older one as default helps catch situations where people are introducing features which aren't supported until later | 18:11 |
| fungi | which was what prompted all of this in the first place | 18:11 |
| clarkb | ++ | 18:11 |
| fungi | that's as close to a "good reason" as i can come up with, and it's kinda flimsy | 18:11 |
| fungi | but better than nothing i guess | 18:12 |
| opendevreview | Merged opendev/system-config master: Remove gitea01-04 from our haproxy config https://review.opendev.org/c/opendev/system-config/+/877296 | 18:19 |
| clarkb | I was thinking about a bit about when to remove gitea01-04. Usually we've got a big rush around the openstack release as people pull the release. Maybe clean up the old servers at the end of next week after that rush? | 18:20 |
| clarkb | Not sure how conservative we should be as its still a bit unknown to me if the cpu steal problem and/or iowait is an actual problem | 18:20 |
| corvus | yeah, i think there may be a backlog in the bridge -- i'm seeing oftc -> matrix messages a bit later | 18:22 |
| opendevreview | Clark Boylan proposed opendev/git-review master: Test old and new Gerrit https://review.opendev.org/c/opendev/git-review/+/877313 | 20:13 |
| opendevreview | Clark Boylan proposed opendev/git-review master: Test Python bounds only https://review.opendev.org/c/opendev/git-review/+/877321 | 20:13 |
| clarkb | something like that maybe for git-review | 20:13 |
| opendevreview | Clark Boylan proposed opendev/git-review master: Test old and new Gerrit https://review.opendev.org/c/opendev/git-review/+/877313 | 20:15 |
| opendevreview | Clark Boylan proposed opendev/git-review master: Test old and new Gerrit https://review.opendev.org/c/opendev/git-review/+/877313 | 20:17 |
| ianw | clarkb: if you have some time to loop over https://review.opendev.org/q/topic:jammy-dns that would be good | 20:22 |
| ianw | i haven't fully finished https://etherpad.opendev.org/p/2023-opendev-dns because some of it, like variable names etc. depend on the stack ^ for an actual checklist | 20:23 |
| ianw | but i could probably work on starting replacement servers asynchronously anyway to get them ready | 20:23 |
| ianw | i was thinking it would be good to have one in rax and one in vexxhost. only need to be very small instances | 20:24 |
| opendevreview | Clark Boylan proposed opendev/git-review master: Test old and new Gerrit https://review.opendev.org/c/opendev/git-review/+/877313 | 20:24 |
| clarkb | ianw: I think we already do the split clouds for them so ++ to continuing that | 20:24 |
| ianw | yeah, it's split like that now | 20:25 |
| clarkb | the swap NS records step happens via our registrar to update opendev.org at the .org level right? | 20:26 |
| fungi | yeah, spreading our authoritative resolvers between multiple providers makes sense for resilience, which we already do (ns1 in rax, ns2 in vex) | 20:26 |
| fungi | seems you agree with the approach we originally used ;) | 20:27 |
| ianw | haha yes | 20:29 |
| ianw | clarkb: yeah, but they should match what's in the opendev.org zone file too | 20:30 |
| clarkb | ianw: right I'm just trying to map the sequence of events in my head. Do we add the new NS servers to the zone first then update registrar for top level zone then remove our NS servers? | 20:32 |
| clarkb | or does it not matter so much and we can just be out of sync for a bit as long as both sets of servers have accurate enough data and listen on port 53? | 20:33 |
| ianw | i think we can change the registrar, and then change the zone file, as long as we don't turn off ns1,ns2 in the mean time | 20:36 |
| clarkb | makes sense | 20:36 |
| fungi | yeah, either order works | 20:36 |
| fungi | what we supply to the registrar ends up injected as glue records into the .org tld zone | 20:36 |
| clarkb | ianw: for https://review.opendev.org/c/opendev/system-config/+/876936/ I think we should see if corvus has a minute to look at it. corvus wrote the initial implementation and may be able to fill in why those names were chosen | 20:36 |
| clarkb | fungi: right and those are ultimately what and NS query would lookup right? | 20:37 |
| clarkb | beacuse of the whole chicken and egg problem | 20:37 |
| fungi | that and the addition of ns records to our zones can happen in parallel, but the registrar may balk if we aren't already publishing ns records so i'd update the zonefiles first | 20:37 |
| ianw | ++ that is an optional change, it just made a bit more sense to me layed out like that | 20:37 |
| fungi | i'll note that i haven't looked at any dns server related changes yet but would like to | 20:38 |
| ianw | fungi: please do :) https://review.opendev.org/q/topic:jammy-dns is the stack that is all just mechanical stuff. the checklist @ https://etherpad.opendev.org/p/2023-opendev-dns isn't complete but i will update that and ask for review "soon" | 20:39 |
| clarkb | fungi: I think jammy ssh stuff is broken against gerrit 3.4.4 :) (this is the ssh-rsa problem so we can fix it with a different key type) | 20:40 |
| fungi | better than the bionic ssh problem with gerrit 3.4.5 and later, which doesn't have a great workaround | 20:41 |
| fungi | though interesting the py36 job would have passed in that case | 20:41 |
| fungi | maybe i haven't fully grokked the problem there | 20:42 |
| opendevreview | Clark Boylan proposed opendev/git-review master: Test old and new Gerrit https://review.opendev.org/c/opendev/git-review/+/877313 | 20:43 |
| clarkb | fungi: MINA 2.7 or 2.8 fixed the issue you have with 3.4.5. But as far as Ican tell the MINA version that fix thing swas never backported to 3.4 newer than 3.4.5 | 20:44 |
| ianw | (we could also upgrade these in place and just move on ... but i think it has some value rotating these, and having a checklist for how we did it, so that if we ever actually *need* to replace these things we have a template that worked at least once) | 20:44 |
| fungi | clarkb: it was supposedly fixed in mina-sshd 2.8 yeah | 20:44 |
| clarkb | ianw: I agree I think this is better hygiene particularly for DNS servers its good to do that | 20:45 |
| clarkb | fungi: 3.4 is EOL now so won't ever get those updates either | 20:46 |
| clarkb | 3.5 should go eol in a couple of months (when 3.8 is released) | 20:47 |
| fungi | wish we had a good idea of which gerrit versions are in use in the wild | 20:47 |
| ianw | fungi: there was a message in the chat the other day something along the lines of "we are still on our 2.6 port" ... openafs is one i look at occasionally that's 2.13 | 20:49 |
| clarkb | ianw: I think 2.6 is broadcom? its one of the well involved companies so I worry less aout them. They know they are behind and are working to upgrade | 20:49 |
| ianw | "that's a version of the replication plugin that we backported to work with 2.7 Gerrit because we're not yet off our fork..." ... that was it | 20:50 |
| fungi | i guess similar to the openstack testing and pyca/cryptography version discussion, we (git-review maintainers) can accept bug reports for regressions with old gerrit versions even if they're too old for us to easily test in an automated fashion | 20:50 |
| clarkb | ++ | 20:51 |
| ianw | that sounds like a very reasonable position | 20:51 |
| clarkb | they can also use old versions of git review which were long battle tested against old gerrit | 20:51 |
| fungi | sure, but i'd like for them to still be able to take advantage of newer features/fixes too (at least those that don't depend on newer gerrit) | 20:59 |
| fungi | particularly since we may need to make future adjustments to work with newer python interpreters or git on the client side, which is independent from what gerrit version the servers they interact with might run | 21:00 |
| opendevreview | Merged openstack/project-config master: gerrit/acl : Convert Backport-Candidate to submit-requirements https://review.opendev.org/c/openstack/project-config/+/875993 | 21:32 |
| clarkb | ianw: did you want me to followup to arm with an etherpad where we can start drafting something? iwas going to send an email last week then totally spaced on it and see the todo item on the list now | 22:01 |
| ianw | heh i was just thinking of that too | 22:01 |
| ianw | maybe we can work @ https://etherpad.opendev.org/p/arm-marketing-statement | 22:02 |
| corvus | clarkb: ianw i think ianw's names are fine, and really the server names could be adjusted too. i +1d the change based on that feedback, but i have not done detailed analysis of ansible, etc, so refrained from +2 | 22:06 |
| clarkb | corvus: thanks! I think double checking the intent was mostly what we needed. We can go over the ansible | 22:08 |
| ianw | indeed, thanks | 22:19 |
| clarkb | ianw: if you're starting to draft something maybe you want to followup to the email? | 22:22 |
| ianw | clarbk: i think https://etherpad.opendev.org/p/arm-marketing-statement is a first attempt? | 22:22 |
| ianw | yeah, i don't know if that should say openinfra somewhere too maybe? | 22:24 |
| ianw | the link they have https://git.airshipit.org/cgit doesn't work, on https://openinfra.dev/projects | 22:25 |
| clarkb | ianw: I tried to make an edit that incorporates something about the open infra foundation | 22:26 |
| ianw | ++ i think that's probably pretty good now | 22:29 |
| ianw | i can reply to the mail | 22:30 |
| clarkb | ianw: let me see if foundation is ok with their name in there | 22:31 |
| clarkb | (I'm sure its fine but I can ask really quickly) | 22:31 |
| ianw | ++ good idea | 22:31 |
| ianw | or have someone with a marketing degree write something even better, haha | 22:31 |
| ianw | ok, the Backport-Candidate labels are now all using s-r too; has deployed https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_53d/875993/6/deploy/infra-prod-manage-projects/53d1f0b/manage-projects.yaml.log | 22:41 |
| clarkb | ianw: I've been told someone will take a look but it might not be until tomorrow morning relative to North american time zones | 22:43 |
| ianw | no worries, let's reply after that to avoid confusion over what's final or not | 22:44 |
| clarkb | ++ | 22:44 |
| opendevreview | Merged openstack/project-config master: gerrit/acl : handle key / values with multiple = https://review.opendev.org/c/openstack/project-config/+/875994 | 22:48 |
| clarkb | I've just made updates to our meeting agenda. Please let me know if I've forgotten anything | 22:59 |
| clarkb | I'm following up on the gitea01 to gitea09 borg backup move and I think the reason we haven't updated gitea09 to do backups yet is we need the service-borg-backup playbook to run and my change ti update the inventory and host vars didn't trigger it. It should run at like 0400 UTC in about 5 hours I think? | 23:07 |
| clarkb | I'll look at things tomorrow in that case | 23:07 |
| ianw | huh, that might be an oversight | 23:17 |
| clarkb | gitea load averages look much more stable now. gitea11 is busy too just without the load fun | 23:39 |
| opendevreview | Merged opendev/system-config master: dns: remove old openstack.org nameservers from iptables list https://review.opendev.org/c/opendev/system-config/+/876908 | 23:44 |
| opendevreview | Merged opendev/system-config master: Remove unused adns1/ns* host_vars files https://review.opendev.org/c/opendev/system-config/+/876909 | 23:44 |
| opendevreview | Merged openstack/project-config master: gerrit/acl : Convert Review-Priority to submit-requirements https://review.opendev.org/c/openstack/project-config/+/875995 | 23:54 |
| ianw | ^ since -2 blocks on that one it's a bit more "active" than the other s-r rules. no booleans, but just keep any eye out if any complaints (hasn't deployed yet) | 23:55 |
| Clark[m] | It's too bad we didn't coordinate the AND thing with April 1st :) | 23:58 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!