| opendevreview | Brian Rosmaita proposed openstack/project-config master: Implement openstack-unmaintained-core group https://review.opendev.org/c/openstack/project-config/+/902110 | 02:50 |
|---|---|---|
| frickler | clarkb: johnsom: ah, sorry for blocking that patch, I just wanted to make sure that it reaches the V+1 state so it can get merged faster after the gerrit update. but then it looks like I gave a good practicing opportunity to tonyb, so not too bad in the end ;) | 07:55 |
| *** maxh1 is now known as maxh | 08:01 | |
| *** tobias-urdin is now known as tobias-urdin-pto | 09:34 | |
| tonyb | frickler: yup it worked out pretty well really | 12:57 |
| ykarel | Hi need help in checking what's wrong with job neutron-ovs-tempest-plugin-scenario-iptables_hybrid-nftables, it fails in RETRY_LIMIT, and looking at console last task is Preparing Job Workspace | 14:19 |
| ykarel | can someone check zuul executor log, or any other way to troubleshoot it? | 14:21 |
| fungi | ykarel: looks like it's hitting that on stable/xena but working on stable/yoga, correct? | 14:22 |
| fungi | https://zuul.opendev.org/t/openstack/builds?job_name=neutron-ovs-tempest-plugin-scenario-iptables_hybrid-nftables&skip=0 | 14:23 |
| fungi | ykarel: this is what i'm finding on the executor. i think it's having trouble with the checkout override for neutron-tempest-plugin: https://paste.opendev.org/show/bFYWQAjMw8rZquUTVBR7/ | 14:32 |
| opendevreview | Merged openstack/project-config master: Implement openstack-unmaintained-core group https://review.opendev.org/c/openstack/project-config/+/902110 | 14:33 |
| fungi | ykarel: looks like 2023-11-18 is when it started doing this, but the checkout override has been in place since 2023-04-21, so i suspect this is a zuul behavior change | 14:36 |
| frickler | that override is in place for half a year, can this be a regression in zuul's handling of it? | 14:37 |
| fungi | corvus: if you have a moment, does that look like it could be related to ref handling changes a couple of weeks ago? seems like it started happening coincident with our automated zuul upgrade two weeks ago | 14:37 |
| corvus | fungi: can do | 14:39 |
| fungi | skimming codesearch, we don't have a lot of jobs specifying override-checkout to a tag, but there are more beyond just neutron | 14:42 |
| frickler | https://review.opendev.org/c/zuul/zuul/+/900489 looks suspicous to me, it only talks about branch refs, not about tags | 14:42 |
| ykarel | fungi, yes only xena impacted, for yoga it works fine. last success of job was on 17th Nov | 14:47 |
| frickler | nice, found another javascript error in the zuul UI while checking for other examples, reported in #zuul | 14:51 |
| clarkb | I've got a meeting in 20 ish minutes. After that I'll generate a new ed25519 ssh key for gerrit -> gitea replication and push a change to rotate it into gitea as well as add it to hostvars for gerrit | 16:33 |
| opendevreview | Brian Rosmaita proposed openstack/project-config master: Address TODO in acl normalization script https://review.opendev.org/c/openstack/project-config/+/902314 | 16:46 |
| *** tkajinam is now known as Guest8666 | 17:02 | |
| tonyb | clarkb: Did that key get made? | 17:23 |
| TheJulia | fungi: you can reclaim that node, thanks. looks like it was all stupid human tricks | 17:27 |
| fungi | TheJulia: isn't it always? ;) | 17:28 |
| TheJulia | often, yes | 17:28 |
| clarkb | tonyb: not yet, I'm about to do that | 17:28 |
| fungi | hold has been deleted, thanks TheJulia! | 17:28 |
| clarkb | tonyb: is that something you want to do together? | 17:29 |
| clarkb | tonyb: I'm currently doing system updates then will reboot and then load ssh keys so I can ssh into bridge and do that | 17:30 |
| clarkb | system updates locally to be clear | 17:30 |
| tonyb | fungi: zuul-client autohold-list #to get the request_id; zuul-client autohold-delete request_id ? | 17:30 |
| clarkb | tonyb: yup you need to pass the --tenant openstack (or other tenant name) flag as well but those are the commands | 17:30 |
| fungi | tonyb: exactly, on one of the schedulers, with sudo or as root, and need to tell it the tenant | 17:31 |
| tonyb | cool beans | 17:31 |
| fungi | though to parrot corvus's comment from yesterday, it could be done through the webui too | 17:31 |
| tonyb | clarkb: I figured it was local updates ;P and sure I gues creating all new credentials doesn't happen often so it'd be good to see if there is anything unexpected | 17:32 |
| tonyb | fungi, corvus: noted | 17:32 |
| clarkb | tonyb: ok I've rebooted and added ssh keys. We can use taht same meetpad we've been using? | 17:41 |
| clarkb | and Iv'e started a root screen on bridge | 17:42 |
| tonyb | okay. gimme 2 mins | 17:43 |
| tonyb | okay I'm attached | 17:44 |
| fungi | proposed command lgtm | 17:45 |
| tonyb | +1 | 17:45 |
| fungi | looks like it worked | 17:46 |
| clarkb | yup I'll edit hostvars in a sec | 17:46 |
| clarkb | just trying to find the old one to make the adjacent | 17:48 |
| clarkb | have to refer to the code to find the old names | 17:48 |
| fungi | hopefully gerrit handles new-style private keys | 17:50 |
| clarkb | hrm that is a good question | 17:51 |
| clarkb | I feel like it didn't before and we had to manually convert all of our keys at one point but I'm not sure if MINA updates have solved that | 17:51 |
| fungi | they've been "new" for enough years that i expect it's not an issue | 17:51 |
| clarkb | https://github.com/apache/mina-sshd/blob/master/docs/client-setup.md#openssh-file-format-support | 17:53 |
| clarkb | now to figure out which version of mina gerrit 3.8 has | 17:53 |
| clarkb | I see the eddsa artifact in a bazel BUILD file for mina so I'm pretty sure we've got those deps for ed25519 but still not finding the release version | 17:55 |
| tonyb | That doc says: "(although for reading ed25519 keys one needs to add the EdDSA support artifacts)", how would I verify we have inclueded that artifact? | 17:55 |
| tonyb | Thanks, you're a mind reader ;P | 17:56 |
| fungi | my bigger concern was over the "-----BEGIN OPENSSH PRIVATE KEY-----" vs "-----BEGIN <algorithm> PRIVATE KEY-----" | 17:56 |
| clarkb | fungi: yup and we are running this version: https://github.com/apache/mina-sshd/blob/sshd-2.9.2/docs/client-setup.md#openssh-file-format-support | 17:57 |
| clarkb | inside gerrit/tools/nongoogle.bzl there is an SSHD_VERS var set that sets all the mina lib versions | 17:57 |
| clarkb | inside that same file eddsa is listed as a dep | 17:57 |
| clarkb | tonyb: ^ | 17:57 |
| clarkb | so I think we're good. I'll proceed | 17:57 |
| clarkb | should I rm the original files that were used to generate things? | 17:58 |
| fungi | i would shred and then rm them | 17:59 |
| fungi | at least the privkey | 17:59 |
| tonyb | I agree | 18:00 |
| clarkb | ok done | 18:01 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Rotate the new Gitea replication key into Gitea config https://review.opendev.org/c/opendev/system-config/+/902315 | 18:02 |
| fungi | specifically, https://man.openbsd.org/ssh-keygen#m is what i was wondering about | 18:03 |
| fungi | since mina-ssh may want pem format instead of the new openssh-specific format | 18:03 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Switch Gerrit replication to using an ed25519 key https://review.opendev.org/c/opendev/system-config/+/902169 | 18:03 |
| clarkb | fungi: yes the link I gave above seems to say it accepts openssh formated files? | 18:04 |
| clarkb | if you click the hyperlinked OpenSSH in the link it seems to point at the new format so I think that is correct | 18:05 |
| fungi | can you quote the passage you're looking at? it's a big document and what i found was "Reading key files in PEM format (including encrypted ones) is supported by default for the standard keys and formats. Using additional non-standard special features requires that the Bouncy Castle supporting artifacts be available in the code's classpath." | 18:05 |
| tonyb | "OpenSSH file format support | 18:05 |
| tonyb | The code supports OpenSSH formatted files without any specific extra artifacts (although for reading ed25519 keys one needs to add the EdDSA support artifacts). " | 18:05 |
| fungi | openssh isn't creating them in pem format lately, but its own RFC4716 format | 18:05 |
| clarkb | fungi: it deep links to the specific section but here is teh quote: "The code supports OpenSSH formatted files without any specific extra artifacts (although for reading ed25519 keys one needs to add the EdDSA support artifacts)." The OpenSSH string there is a hyperlink to the openssh new private key format | 18:05 |
| fungi | aha, perfect. thanks | 18:06 |
| clarkb | http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/ssh/PROTOCOL.key?rev=1.1&content-type=text/x-cvsweb-markup is what that links to | 18:06 |
| corvus | fungi: remote: https://review.opendev.org/c/zuul/zuul/+/902316 Fix repo state restore for zuul role tag override | 18:06 |
| fungi | thanks corvus! frickler ykarel ^ | 18:08 |
| clarkb | note we don't want to approve the gerrit change until after the gitea change has applied just to make sure that applies cleanly first | 18:09 |
| clarkb | also reviewers please double check the host vars I added when you review the gerrit change | 18:09 |
| clarkb | I'm now going to look at gerrit 3.2 mina which I'malmost positive did not support the new formatted keys and we can hopefully use that toconform mina added support in the interim | 18:09 |
| tonyb | clarkb: Should we do the playbooks/roles/gerrit/templates/gerrit_ssh_config.j2 part of 902169 in a seperate patch or perhaps use the RSA key and have a seperate patch to switch in the new key | 18:10 |
| clarkb | hrm https://github.com/apache/mina-sshd/blob/sshd-2.4.0/docs/client-setup.md#openssh-file-format-support is the gerrit 3.2 version and it also reports support there. So less confident now | 18:10 |
| tonyb | that'd give us a change to double check ssh with both keys before we "go live" | 18:10 |
| clarkb | tonyb: ya maybe first change switches over to using the rsa key (though note we have to restart gerrit to pick that up each time we change it) then a followup to do ed25519 | 18:11 |
| clarkb | the need for a restart is why I didn't both splitting it up because I was thinking we should minimize those, but might give us more confidence | 18:11 |
| clarkb | fungi: also I can convert to PEM If we think we need to | 18:11 |
| fungi | clarkb: nah, it looks like they do intentionally support openssh key format, i just wasn't seeing the keywords i expected in their document | 18:12 |
| clarkb | ok I'll shutdown the screen now then | 18:12 |
| clarkb | fungi: tonyb: let me know if you think we should split up the change to explicitly choose the rsa key first, do a restart, then udpate to use the ed25519 key and do another restart | 18:14 |
| corvus | i'm going to restart zuul-web to pick up the recent js changes | 18:14 |
| tonyb | corvus: okay | 18:16 |
| clarkb | I've just noticed that the gerrit replication plugin docs do say you should use the PEM format | 18:17 |
| clarkb | https://gerrit.googlesource.com/plugins/replication/+/refs/heads/master/src/main/resources/Documentation/config.md | 18:17 |
| clarkb | it is possible that is no longer necessary and we can risk it or I can just go ahead and convert the key now | 18:18 |
| tonyb | clarkb: Yeah I think converting to PEM is probably safest | 18:19 |
| tonyb | clarkb: WRT gerrit_ssh_config.j2 I'm happy to go with your risk assessment. | 18:20 |
| clarkb | tonyb: I think our "it doesn't work" fallback is to (re)move the .ssh/config file and restart gerrit again | 18:20 |
| clarkb | I'm somewhat inclined to try a single restart given that is straightforward | 18:21 |
| tonyb | Okay | 18:21 |
| clarkb | heh now I wish I ahdn't shredded and rm'ed the files | 18:21 |
| clarkb | will the pubkey format change? | 18:22 |
| clarkb | I'm half inclined to genreate new keys if so since I'll have to update the gitea change | 18:22 |
| * clarkb does this the key isn't used anywhere anyway so a new one is file | 18:22 | |
| clarkb | *fine | 18:22 |
| tonyb | okay | 18:22 |
| tonyb | curse fungi and his detail focused questions! | 18:23 |
| corvus | #status log restarted zuul-web to pick up js changes | 18:23 |
| opendevstatus | corvus: finished logging | 18:23 |
| clarkb | heh I prefer getting this stuff right before we put it in production | 18:24 |
| tonyb | I'm sure | 18:24 |
| fungi | clarkb: yeah, generating a new key is easier than converting an existing one | 18:24 |
| fungi | i'd recommend that where possible | 18:24 |
| clarkb | hrm I did -m PEM but the output file didn't change the header and footer text | 18:27 |
| clarkb | fungi: does the header and footer not change? | 18:30 |
| * clarkb is testing locally | 18:30 | |
| clarkb | with rsa keys the headers change. With ed25519 they don't. Not sure if those headers actually matter for parsing within the client | 18:31 |
| clarkb | `file` distinguishes the pem vs openssh but only for rsa as well | 18:32 |
| clarkb | I wonder if ed25519 can only exist as new format key? | 18:33 |
| fungi | oh, maybe... | 18:34 |
| clarkb | the man page hints at this " Setting a format of “PEM” when generating or updating a supported private key type will cause the key to be stored in the legacy PEM private key format." | 18:35 |
| clarkb | I think our next step may be to test this with a held node | 18:35 |
| clarkb | this == ed25519 and replication | 18:36 |
| clarkb | which is annoying but better than breaking in production when it doesn't need to | 18:36 |
| clarkb | in that case I won't bother replacing this with the new key. Instaed I'll try to setup a test situation that can verify the ed25519 key works with MINA in modern gerrit | 18:37 |
| clarkb | or we can just use a large rsa key | 18:37 |
| clarkb | here I thought using rsa would be more work :) | 18:37 |
| clarkb | opinions? | 18:37 |
| tonyb | I think we should use a larger RSA key. ed25519 is nice for other reasons but not essential for what we're trying to do | 18:38 |
| tonyb | we can use the new key rotation tasks later to switch to ed25519 after we've tested it | 18:39 |
| fungi | https://unix.stackexchange.com/questions/746465/how-do-i-convert-an-openssh-generated-ed25519-key-to-the-pem-format | 18:39 |
| fungi | https://security.stackexchange.com/questions/267711/how-can-i-convert-an-ed25519-key-in-pkcs8-to-openssh-private-key-format/267767#267767 | 18:39 |
| fungi | "OpenSSH doesn't currently support reading or writing Ed25519 keys in any format other than the OpenSSH native key format." | 18:40 |
| fungi | https://bugzilla.mindrot.org/show_bug.cgi?id=3195 | 18:41 |
| fungi | fixed in openssh 9.6 | 18:41 |
| clarkb | fungi: that confirms my suspicion then. I'm happy to do a 4096 or 8192 bit rsa key instead and update the existing changes | 18:41 |
| clarkb | and then we can still use .ssh/config to select the key we want I think | 18:41 |
| clarkb | and have two rsa keys side by side | 18:42 |
| clarkb | that might make it more difficult to determine if we're authenticating with the new key but I think we needed to sort that out with ed25519 anyway | 18:42 |
| clarkb | I'm thinking we should do 8192 to avoid needing to rotate again in the near future when germany decides 4096 isn't enough | 18:43 |
| tonyb | clarkb: sounds good to me | 18:46 |
| fungi | conventional cryptanalytic wisdom is that any advances which make 4096-bit rsa easy enough to factor are likely to be advances which have simply broken rsa entirely, in which case 8192-bit rsa will be just as broken. 4096 is already larger than the latest openssh's default of 3072 | 18:47 |
| clarkb | 3072 is also the minimum that gitea will accept | 18:48 |
| clarkb | I guess 4096 is fine as that is above the minimum | 18:48 |
| clarkb | `ssh-keygen -m PEM -f ./replication_id_rsa_B -t rsa -b 4096 -C 'gerrit@gitea.opendev.org 20231130'` is what I intend to use | 18:49 |
| clarkb | and then I'll update the chagnes I just pushed to use that instead with appropriately renamed ansible vars | 18:49 |
| fungi | there's a law of diminishing returns at play, since the work factor involved in brute-forcing an rsa key is exponentially proportional to its length, so 4096 is already so much stronger than 3092 that it's unlikely to be brute-forced before the heat death of the universe | 18:49 |
| clarkb | but I guess the difference between 2048 and 3072 is a problem | 18:50 |
| fungi | 1024-bit is known to be unsafe, 2048 is probably fine in most cases but just to be safe openssh raised its default to 3072 recently just to be sure | 18:51 |
| fungi | because there's a risk that in the distant future 2048-bit might be brute-forcible (3072 really shouldn't) | 18:51 |
| fungi | recommendations have generally been to replace 2048 keys by/not use them after ~2030 just to be safe | 18:52 |
| tonyb | fungi: interesting timeline. | 18:53 |
| fungi | speaking of timelines, https://crypto.stackexchange.com/questions/1978/how-big-an-rsa-key-is-considered-secure-today#1982 has a neat timeline of what's actually been publicly acknowledged as factored | 18:54 |
| tonyb | clarkb: that command looks good to me, although I tend to avoid spaces in the comment | 18:54 |
| fungi | nobody has actually broken a 1024-bit rsa key and proven it publicly yet, though they're getting sort of close | 18:55 |
| clarkb | tonyb: ya I just noticed that the existing comment doesn't have spaces. I can remove the space in hostvars since it shouldn't actually affect the key material to do so | 18:56 |
| clarkb | though I guess it is encoded into the private key itself | 18:57 |
| * clarkb generates a new key | 18:57 | |
| clarkb | I'm getting really good at generating keys | 18:57 |
| tonyb | clarkb: #silverlining | 18:58 |
| clarkb | fungi: the reason gitea set their limit to 3072 is germany is now saying you should use at least that many bits apparently | 18:58 |
| clarkb | today rather than in 2030 but that may be a "start now so its done in time" thing | 18:59 |
| fungi | yes, based on voodoo science, but then again politicians have no idea what any of this means | 18:59 |
| opendevreview | Merged opendev/system-config master: Add inventory/LE records for mirror02.bhs1.ovh and mirror03.gra1.ovh https://review.opendev.org/c/opendev/system-config/+/901628 | 19:00 |
| opendevreview | Merged opendev/system-config master: Add inventory/LE records for mirror02.dfw.rax https://review.opendev.org/c/opendev/system-config/+/902008 | 19:00 |
| fungi | mathematicians say 1024-bit rsa is still working but it could be factored in coming years, so using larger keys means that stuff you encrypt now won't be accessible to attackers years in the future once breaking 1024-bit rsa becomes relatively easy. security folks see the warning from mathematicians and say, well if the mathematicians say we should be using at least 2048 bit now to be on the | 19:03 |
| fungi | safe side, then let's go ahead and recommend something even stronger just in case. politicians then see what the security folks said, and increase it even more so they can look like they're doing something useful | 19:03 |
| fungi | it becomes more absurd when, instead of talking about long-term/archival encryption or the ability to trust old signatures, you look at it from the perspective of point-in-time authentication and encrypted communication streams | 19:06 |
| fungi | basically, the risk in this case is that someone records a packet capture of our ssh session and we transmit sensitive data over it, then many years in the future the surreptitious packet capture they've held onto all that time can finally be decrypted and they'll be able to see what we transmitted over the connection | 19:07 |
| fungi | none of which is even sensitive now, much less in the distant future | 19:08 |
| tonyb | True | 19:10 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Rotate the new Gitea replication key into Gitea config https://review.opendev.org/c/opendev/system-config/+/902315 | 19:13 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Switch Gerrit replication to a larger RSA key https://review.opendev.org/c/opendev/system-config/+/902169 | 19:13 |
| clarkb | ok I think that should be mergeable now. Reviewers: please review both changes before approving the first and double check the contents in ansible host vars | 19:16 |
| clarkb | also maybe wait for CI results to come back just to make sure I didn't do anyting silly | 19:16 |
| clarkb | I am going to help my brother install a new drop in stove (it amazes me that drop in stoves were ever considered a good idea) at 2100 UTC so I'll be afk for a bit otherwise I'm around today to address feedback and/or monitor changes as they land | 19:24 |
| clarkb | maybe goal should be to get gitea updated today then plan for a gerrit restart to pick up new config tomorrow? | 19:24 |
| clarkb | the gitea test job logs should also show us successfully adding both keys without removing them (the remove task should be skipped) | 19:25 |
| clarkb | nevermind delivery says thirty minutes away. That is probably better as I can hopefully get that done while waiting for zuul | 19:30 |
| corvus | they're nice if you keep your ovens in the wall! | 19:35 |
| fungi | i keep my ovens in the ceiling so i can elevate my cooking | 19:37 |
| corvus | fungi: above your drop-in-floor-mounted range? | 19:39 |
| fungi | of course, it also doubles as a footwarmer in the winter | 19:40 |
| clarkb | I'm noting that we do sometimes run the gitea and gerrit jobs at the same time. I wonder if there is a way to make test gerrit replicate to test gitea without too much time delays. The naive way to do it would be to have gitea run first which is not short then run gerrit which is also no short | 19:46 |
| clarkb | anyway not something we need to solve right now. Just an idea for something that would be neat to test | 19:46 |
| clarkb | I'm going to pop out now but will try to keep an eye on any feedback from zuul or reviewers while I avoid electrocution | 19:46 |
| fungi | rubberize yourself | 19:47 |
| tonyb | https://zeldatearsofthekingdom.wiki.fextralife.com/file/Zelda-Tears-of-the-Kingdom/rubber-set-zelda-totk-wiki-guide.png | 19:49 |
| fungi | yep, that looks like it ought to do the trick | 19:49 |
| corvus | clarkb: since gerrit would need the gitea address, i think gerrit pausing after starting then running gitea is the most straightforward way but slow. another idea would be to combine them into one multi-node job. that might make sense for this particular pairing. | 20:14 |
| Clark[m] | corvus: ya maybe combining makes sense. Also before electrocution we must do limb removal carpentry work | 20:29 |
| tonyb | Change 901628 failed keycloak and nodepool in the deploy pipeline, both with gateway timeouts doing the docker-compose pull. | 20:35 |
| tonyb | I'm assuming that this is "okay", and wont block 902008 from running. | 20:37 |
| Clark[m] | tonyb the second change should run regardless and apply both | 20:55 |
| Clark[m] | Since they touch the same code iirc | 20:55 |
| tonyb | okay. I'll keep an eye on that | 20:56 |
| fungi | meeting up with some friends to do holiday things, so disappearing nowish for the evening, but will be around all tomorrow as usual | 21:32 |
| corvus | i'm continuing to see dockerhub errors locally too | 21:39 |
| corvus | third attempt succeeded; so seems like partial outage | 21:41 |
| tonyb | I don't understand the state of things after the 2 runs of the deploy pipeline after 901628 and 902008 merged. 901628 ran all the jobs (keycloak and nodepool failed because docker.io partial outage), importantly letsencrypt and service-mirror ran and included all 3 of the new mirror nodes (https://paste.opendev.org/show/bQjWBV4oe6YKpydX6HpK/) Then 902008 ran and letsencrypt failed (possibly because there was no work to be done?) | 23:10 |
| tonyb | so none of the dpenedent jobs (including service-{mirror,keycloak,nodepool}) ran. | 23:11 |
| tonyb | I think overall it's fine as across the 2 runs the work we really needed letsencrypt and service-mirror ran and the 3 nodes look okay after looking at SSL, process table and mounted filesystems | 23:16 |
| Clark[m] | tonyb: without being able to check that seems possible. The only thing is the LE job should succeed even when no work is to be done (we run it daily in periodic jobs for example) | 23:24 |
| tonyb | okay. I'll look some more when I get home. the fact the LE job failed was what worried me. | 23:30 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
