| ianw | yeah i was thinking abstract out the steps into roles, and then call the same roles from both places. but it looks to fiddly to bother | 00:02 |
|---|---|---|
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Depend on dib-python to set DIB_PYTHON_VERSION https://review.opendev.org/c/openstack/diskimage-builder/+/802408 | 00:03 |
| ianw | still i feel like import_playbook might work ... or adding it as another playbook in the test | 00:03 |
| Clark[m] | ianw: the older code added it as another playbook in the test but these require specific vars to be passed in which is awkward for the current test setup. I'll think on making it less clunky | 00:17 |
| ianw | yeah i get that. would just be good if the logs weren't jumbled with another nested ansible run. could possibly just use ANSIBLE_LOG_PATH and collect that log file as something easier to examine | 00:20 |
| opendevreview | Tristan Cacqueray proposed opendev/system-config master: Run matrix-gerritbot on eavesdrop https://review.opendev.org/c/opendev/system-config/+/800506 | 02:20 |
| *** marios is now known as marios|ruck | 05:04 | |
| *** amoralej|off is now known as amoralej | 06:46 | |
| *** marios|ruck is now known as marios | 07:01 | |
| *** marios is now known as marios|ruck | 07:03 | |
| *** ykarel|away is now known as ykarel | 07:28 | |
| *** rpittau|afk is now known as rpittau | 07:29 | |
| *** ykarel is now known as ykarel|lunch | 08:47 | |
| yoctozepto | hmm, got something new from gerrit today | 09:55 |
| yoctozepto | error: remote unpack failed: error Missing tree d700505440e0092db1a7831758dc10b6ef46a56f | 09:55 |
| yoctozepto | fatal: Unpack error, check server log | 09:55 |
| yoctozepto | the next try succeeded | 09:56 |
| *** ykarel|lunch is now known as ykarel | 10:07 | |
| fungi | yoctozepto: we've seen similar errors in the past we think may be related to thin pushes from cgit to jgit, and clarkb added a --no-thin command line option to git-review to serve as a potential workaround | 11:50 |
| yoctozepto | ack, thanks fungi | 11:54 |
| opendevreview | Ananya proposed opendev/elastic-recheck master: Run elastic-recheck container https://review.opendev.org/c/opendev/elastic-recheck/+/729623 | 11:55 |
| opendevreview | Ananya proposed opendev/elastic-recheck master: Run elastic-recheck container https://review.opendev.org/c/opendev/elastic-recheck/+/729623 | 12:03 |
| fungi | yoctozepto: though previously we'd seen errors like that be persistent for a particular commit history (otherwise requiring something like a rebase to solve) | 12:07 |
| yoctozepto | interesting, I have only retried | 12:39 |
| *** amoralej is now known as amoralej|lunch | 13:02 | |
| *** amoralej|lunch is now known as amoralej | 13:43 | |
| opendevreview | Ananya proposed opendev/elastic-recheck master: Run elastic-recheck container https://review.opendev.org/c/opendev/elastic-recheck/+/729623 | 13:44 |
| fungi | yoctozepto: was it a single change or a series of dependent changes? | 13:45 |
| fungi | there is a bug report about it at https://storyboard.openstack.org/#!/story/1332549 | 13:49 |
| fungi | i'll update that with info about the --no-thin workaround in 2.1.0 | 13:49 |
| yoctozepto | fungi: single, just one commit on top of current master in the releases repo | 13:58 |
| yoctozepto | very simple | 13:58 |
| fungi | interesting | 13:58 |
| yoctozepto | indeed | 13:58 |
| yoctozepto | https://review.opendev.org/c/openstack/releases/+/802447 | 13:59 |
| *** ykarel is now known as ykarel|away | 14:03 | |
| clarkb | I think yoctozepto was one of the people that hit this a few months back which caused us to dig into it? | 14:11 |
| clarkb | with openstack-ansible repo iirc | 14:11 |
| yoctozepto | I would bet it was kolla-ansible instead | 14:11 |
| clarkb | Anyway, current suspicion is that a particular client interaction between C git and jgit causes it to happen. And using the --no-thin flag tells git to not use the optimized negotiation and use the more brute force version which seems to work reliably | 14:11 |
| yoctozepto | but then there at least was a stack of changes and it did not give up | 14:11 |
| clarkb | I wouldn't use --no-thin for every push, just if you hit this | 14:11 |
| yoctozepto | ahh | 14:12 |
| yoctozepto | ok | 14:12 |
| yoctozepto | well, second try just worked (TM) | 14:12 |
| fungi | right, that's why it's not set up as a config option | 14:12 |
| fungi | thin pushes should be much more efficient when they work, but in rare cases jgit and cgit don't seem to agree on what should be included | 14:13 |
| clarkb | yup | 14:13 |
| yoctozepto | odd, perhaps the jgit workers get clumsy | 14:13 |
| clarkb | dpawlik: got back to me with the details I need to clean up that conflict. I'll do that one today rather than batching it up since I have user info confirming the correct fix | 14:16 |
| fungi | awesome | 14:16 |
| dpawlik | clarkb, fungi: thank you folks :) | 14:17 |
| fungi | i've updated story 1332549 with details of the --no-thin option in git-review 2.1.0 but i left the task in a todo state because it's not clear to me whether we should be treating this as a git-review bug (it's most likely a bug in jgit), but also if we close it then it's less likely future users who run into the error will find that report | 14:51 |
| clarkb | and the issue may go away on its own as people update git? I dunno I've never seen the issue locally myself and run a fairly up to date git | 14:52 |
| fungi | i haven't either, but on the other hand i likely don't push changes to gerrit nearly as often as some users | 14:52 |
| fungi | so statistically speaking, my sample size may not be significant enough to expect to have encountered it | 14:53 |
| clarkb | yoctozepto: out of curiosity are you using bionic's git? another group that hit this was using bionic iirc | 15:03 |
| yoctozepto | clarkb: no, focal's | 15:04 |
| yoctozepto | git version 2.25.1 | 15:04 |
| clarkb | thanks | 15:05 |
| opendevreview | Tristan Cacqueray proposed opendev/system-config master: Run matrix-gerritbot on eavesdrop https://review.opendev.org/c/opendev/system-config/+/800506 | 15:27 |
| *** marios|ruck is now known as marios | 15:37 | |
| *** marios is now known as marios|ruck | 15:39 | |
| sshnaidm | hi, how do I get "public key" of project ansible-collections-openstack ? | 15:51 |
| opendevreview | Abhishek Kekane proposed openstack/project-config master: Remove 'glance' group from project entry https://review.opendev.org/c/openstack/project-config/+/802548 | 15:52 |
| *** marios|ruck is now known as marios | 15:56 | |
| *** marios is now known as marios|ruck | 15:57 | |
| fungi | sshnaidm: see https://docs.opendev.org/opendev/infra-manual/latest/drivers.html#using-secrets for an example | 16:02 |
| corvus | sshnaidm: also https://zuul-ci.org/docs/zuul-client/commands.html#encrypt is an option | 16:03 |
| *** marios|ruck is now known as marios|out | 16:13 | |
| sshnaidm | fungi, can you please take a look? is it the right pipeline? https://review.opendev.org/c/openstack/ansible-collections-openstack/+/802379 | 16:21 |
| fungi | corvus: currently the opendev infrastructure manual's section on zuul secrets links to https://docs.opendev.org/opendev/infra-manual/latest/drivers.html#using-secrets which has been updated with a (broken) link to zuul-client, do you think we should link directly to the encrypt command doc for it instead? (also i'll push a fix momentarily for that link markup error in the zuul docs) | 16:26 |
| fungi | can also update the example in infra-manual to use zuul-client | 16:27 |
| corvus | fungi: i don't see the broken link to zuul-client | 16:29 |
| fungi | corvus: https://zuul-ci.org/docs/zuul/discussion/encryption.html last paragraph just before the usage output | 16:30 |
| fungi | fixup proposed in 802554 | 16:31 |
| corvus | fungi: and yeah, i lean toward updating the manual to link to zuul-client; i don't think we're in a rush to remove that script, but it's probably the better long-term option. i've been trying to use it myself. | 16:31 |
| corvus | fungi: got it (re link) | 16:31 |
| fungi | cool, i'll freshen up the infra-manual section in that case. thanks! | 16:32 |
| *** sshnaidm is now known as sshnaidm|afk | 16:34 | |
| *** rpittau is now known as rpittau|afk | 16:40 | |
| *** amoralej is now known as amoralej|off | 17:11 | |
| clarkb | dpawlik: ok your extra account should be cleaned up now. If you want to double check the active used account is working as expected that would be great (though I anticipate it is fine) | 17:39 |
| clarkb | I've got a new consistency check running to output a new set of input conflicts to the audit script. Then I'll get the audit results and the next batch or proposed cleanups onto review02 for review | 17:41 |
| dmsimard | I haven't sent a patch to gerrit in a while and it's not accepting my ssh key anymore, is there a gotcha or new crypto requirement ? I tried removing my key and adding it back, didn't work either | 17:42 |
| clarkb | dmsimard: are you on fedora and using an rsa key? | 17:45 |
| dmsimard | yes and yes | 17:46 |
| * fungi loves this one | 17:46 | |
| dmsimard | what did I miss ? :D | 17:46 |
| fungi | fedora decided to be proactive abd opaquely break ssh-rsa which is merely deprecated in openssh | 17:47 |
| opendevreview | Jay Faulkner proposed openstack/diskimage-builder master: Permit specification of extra bootstrap packages https://review.opendev.org/c/openstack/diskimage-builder/+/802592 | 17:47 |
| fungi | your easiest solution is to switch to an ecdsa key, because the fedora maintainers blindly broke rsa2 support for ssh in situations where the server doesn't support dynamic negotiation options in the protocol | 17:48 |
| clarkb | yup and on top of that when breaking ssh-rsa in the way they broke it they were supposed to update client defaults to fallback to the rsa version that is allowed | 17:48 |
| clarkb | but they didn't do that. They only half made the change and now you get this behavior | 17:48 |
| dmsimard | I see, searching for that does yield some results which I'll dig into -- I'm due for a key rotation soon anyway | 17:49 |
| clarkb | What Fedora should do is also update their openssh client to fallback to rsa-sha2 variants rather than the upstream default of sha1 | 17:49 |
| clarkb | then their change would likely just work in 99% of cases. As is it is far more likely to break | 17:49 |
| dmsimard | I've found that adding "PubkeyAcceptedKeyTypes +ssh-rsa" to ~/.ssh/config for review.opendev.org fixes it for now so I'll run with that today but will renew my ssh key soon, thanks for the help <3 | 17:50 |
| clarkb | dmsimard: be careful doing that because I think people said that made a global change | 17:51 |
| dmsimard | oh ? despite being under a specific host ? | 17:51 |
| clarkb | dmsimard: basically you can ssh rsa to all the hosts now or something which is why we stopped recommending people do that and instead suggest using a non rsa key for fedora users | 17:51 |
| clarkb | dmsimard: ya it has to do with how ssh overrides those defaults | 17:51 |
| dmsimard | interesting | 17:51 |
| clarkb | (it is confusing and I may be misremembering) | 17:51 |
| fungi | it's possible that constraining it to a specific host entry only downgrades security for connections to that specific remote host | 17:52 |
| clarkb | anyway, I'm not a fedora user nor have I ever been. But if fedora users want to suggest to fedora that they supplement the disabling of rsa sha1 variants with a fallback in openssh client to rsa sha2 variants instead of sha1 that would be great | 17:53 |
| clarkb | The ssh rfc says this should be done when the sha1 variants are disabled | 17:53 |
| fungi | but you're choosing a platform which has decided it's in your best interests to not use sha1 in the ssh protocol key exchanges, so we avoid recommending to users that they go against the security guidance of their chosen platfirms | 17:53 |
| fungi | platforms | 17:53 |
| clarkb | its just that no one but fedora has done that yet so the clients have gotten those updates | 17:53 |
| clarkb | infra-root I've got everything on review02 except for the latest audit results as the audit is still running | 17:54 |
| clarkb | and now the audit results are in place. infra-root you can now review my proposed cleanups with the latest audit results yaml file to cross check against (or do your own cross checking) | 17:57 |
| clarkb | infra-root while I had extra perms I took the chance to look at melody. It reports we maxed out at 84GB of memory in gerrit | 18:00 |
| clarkb | basically double what we could provide the server before | 18:00 |
| fungi | that's awesome | 18:01 |
| fungi | and awe-inspiring | 18:02 |
| fungi | mnaser might like that tidbit too | 18:02 |
| * melwitt loves faster gerrit | 18:22 | |
| clarkb | yuriys: fyi we can talk about the cloud fixup here if you're comfortable with that. | 18:32 |
| mordred | that's amazing - also - WOW that's a lot of RAM | 18:38 |
| yuriys | Yep that's fine. The two instances still appear to be stuck, currently in a meeting, but after will continue to see how to clean those two up. Just seems that I'll have to manually purge from mariadb, which is my least favorite resolution method. | 18:45 |
| clarkb | ok. The other thing I was going to try was a yum update and reboots for kernel patches | 18:45 |
| clarkb | We've got our team meeting in about 15 minutes, then I need lunch then I'll look at that | 18:46 |
| clarkb | Then I'm reviewing a stack of zuul changes | 18:46 |
| yuriys | Sounds like a plan. | 18:46 |
| clarkb | hopefully in the shade under a tree outside :) | 18:46 |
| yuriys | I'm curious whether or not it's also a good idea to pull the latest victoria container images. | 18:46 |
| yuriys | On top of the DNF updates. | 18:46 |
| fungi | i have no objection to that suggestion | 18:50 |
| fungi | it's non-impacting for us and we can always do a fresh redeploy there worst case | 18:51 |
| yuriys | Sounds good. | 18:51 |
| fungi | rebuilding the mirror vm is likely to be the most involved step if we redeploy | 18:52 |
| fungi | but most of that is automated as well | 18:52 |
| yuriys | I want to say I'd like to avoid that... that really shouldn't be normal for cloud 'updates' lol. | 18:52 |
| fungi | sure, but this is openstack ;) | 18:52 |
| yuriys | lol! | 18:52 |
| fungi | in openstack, our motto is "anything's possible" *wink* | 18:52 |
| fungi | (not officially our motto, but should be, right after "if it's not tested it's broken") | 18:53 |
| clarkb | ya I'm good with experimenting here. If we can help you all learn stuff I'm game :) | 18:54 |
| yuriys | Yeah for sure. I'm down to jump in a meets and screenshare all the funsies as usual. My typical workflow. Been a while since I've talked to you guys. | 18:55 |
| yuriys | But if not, it's cool, I SEE HOW IT IS | 18:55 |
| clarkb | I'm open to that though that might be better for tomorrow (I'm wide open tomorrow if that works for you) | 18:55 |
| clarkb | and we can do the kernel updates and reboots and all that fun then too | 18:55 |
| yuriys | oo yeah, same, wednesday way nicer for me. | 18:56 |
| clarkb | yuriys: I can be around as early as 8am pacific if you want to pick a time | 18:56 |
| fungi | yeah, i'm multi-tasking irc and backlogged yardwork, so teleconferencing isn't super compatible with my afternoon (not that my presence is strictly necessary either) | 18:56 |
| fungi | my tomorrow is also mostly open | 18:57 |
| yuriys | Lets just figure it out tomorrow lol. | 18:57 |
| clarkb | works for me :) | 18:57 |
| fungi | samesies | 18:57 |
| mordred | corvus: I +2'd the matrix changes but didn't +A anything | 20:15 |
| corvus | mordred: thanks! | 20:45 |
| opendevreview | Jay Faulkner proposed openstack/diskimage-builder master: Permit specification of extra bootstrap packages https://review.opendev.org/c/openstack/diskimage-builder/+/802592 | 20:47 |
| clarkb | service coordinator election email has been sent. You should see it momentarily | 20:53 |
| opendevreview | Ian Wienand proposed openstack/project-config master: Revert "nodepool: pause centos-8-stream builds" https://review.opendev.org/c/openstack/project-config/+/802619 | 20:56 |
| ianw | i'll check in on the builders a bit later and monitor ^ ... those images are getting very old now; but we should be gtg with new dib release deployed | 20:58 |
| ianw | also noticed the arm64 job timed out, e.g. https://zuul.opendev.org/t/openstack/build/1e6031f1b07f43488f04acb2a736e653/logs | 21:01 |
| ianw | i'll look into that; it always suggests to me an issue where we're building too many wheels | 21:01 |
| clarkb | ianw: on the nodepool side? | 21:02 |
| clarkb | /t/openstack shouldn't be nodepool though | 21:02 |
| ianw | no that was runtime system-config job | 21:02 |
| clarkb | ah | 21:03 |
| clarkb | the review-test cleanup change lgtm and I approved the docker-compose restart flags change | 21:05 |
| clarkb | I'm going to go scope out some shade in a moment and dig into the zuul changes | 21:06 |
| clarkb | infra-root does anyone know if we prefer to split https://review.opendev.org/c/openstack/project-config/+/790093 into to changes so that we can land them normally or do we force merge to get around the zuul error? | 22:53 |
| clarkb | that change includes the zuul/main.yaml tenant update to include the new project as well as the layout change and zuul won't ever +1 that as is but we can force merge after we rename in gerrit to address that | 22:54 |
| fungi | in past renames since the advent of zuul v3 we've bypassed gating to merge the changes whole | 23:07 |
| opendevreview | Merged openstack/project-config master: Revert "nodepool: pause centos-8-stream builds" https://review.opendev.org/c/openstack/project-config/+/802619 | 23:22 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Test the rename_repos playbook https://review.opendev.org/c/opendev/system-config/+/802112 | 23:54 |
| clarkb | ianw: fungi: that attempts to address the concern with nested ansible using import_playbook (I half expect that will break in unexpected ways but if it works it should solve this well) and it adds an explicit test for the CI-tools-updated group rename | 23:55 |
| ianw | cool | 23:55 |
| ianw | i feel like that ansible should already be looking at playbooks etc from the zuul checkout, so i think it has a > 0 chance of working :) | 23:56 |
| clarkb | ya I'm mostly concerend there is some gotcha with import_playbook that makes it unusable here as imports and includes always seem clunky | 23:56 |
| clarkb | but if it does work then it should work well | 23:56 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!