| opendevreview | Merged opendev/system-config master: [testinfra] Update Ansible canary. https://review.opendev.org/c/opendev/system-config/+/902098 | 01:11 |
|---|---|---|
| opendevreview | Brian Haley proposed openstack/project-config master: [Neutron-lib] Update Grafana Dashboard https://review.opendev.org/c/openstack/project-config/+/902119 | 01:11 |
| johnsom | Ok, here is an odd one: https://review.opendev.org/c/openstack/designate/+/901871 | 02:27 |
| johnsom | Workflowed, gate jobs passed at 9:33 am, but the patch isn't marked as merged | 02:28 |
| johnsom | no cross repo dependencies, no parent patches | 02:28 |
| johnsom | (no un-merged parent patches I should say) | 02:29 |
| johnsom | I'm going to leave it in case you want to look into it. I guess I can try a recheck tomorrow | 02:30 |
| Clark[m] | It's the rebased on behalf of thing tripping a Gerrit bug. Should be fixed after we land the 3.8.3 update change and restart Gerrit | 03:00 |
| frickler | Clark[m]: what's the time frame for that? the current case seems no longer related to a missing email address, so I don't see a workaround like before. and we are talking about an urgent bug fix here, so having it blocked by a gerrit issue is kind of a critical situation | 04:45 |
| frickler | ah, the fix already made it into a new gerrit release and we only need to deploy https://review.opendev.org/c/opendev/system-config/+/901992 ? | 04:51 |
| tkajinam | (I assume "the current case" is https://review.opendev.org/c/openstack/designate/+/901871 | 04:52 |
| frickler | tkajinam: yes | 04:52 |
| tkajinam | ah, it was already raised by johnsom. sorry I overlooked it | 04:52 |
| frickler | seeing this issue and also the revert happening for 3.9.0, I'm wondering whether we may have moved too far to the edge with keeping gerrit close to the latest releases | 04:54 |
| opendevreview | Merged openstack/project-config master: [check-release-approval] Fix distributed leadership handling https://review.opendev.org/c/openstack/project-config/+/896785 | 08:31 |
| telepathynoise | **Worldpeace not expected until 6+8+3+6+8+3** | 09:06 |
| chenz_work | got a failed build here due to a key rotation issue on a node: https://zuul.opendev.org/t/openstack/build/fc8f826f54af49c49d29017dec40533e | 09:39 |
| chenz_work | what can be done to fix this? | 09:39 |
| frickler | chenz_work: this sometimes happens due to issues in one of our cloud providers which are out of our control. however, this should happen seldom enough that when you do a recheck, the error is unlikely to repeat | 09:49 |
| chenz_work | frickler: thanks, that seems to have worked :-) | 10:06 |
| Clark[m] | frickler: the day the fix that needed backporting was identified I reached out to Gerrit and they confirmed. I pushed a backport and it was approved shortly after. I also have a change up to update our gerrit installation which has been in review but needs a good time for us to restart Gerrit after. Landing this change and restarting gerrit will pull on the identified fix | 14:24 |
| Clark[m] | frickler: I don't think we are flying too close to the sun here. Our testing caught the bug with 3.9.0 immediately after it was released (if you ignore the holiday days I didn't bother to work on). The other issue was latent in Gerrit 3.8 since it released 6 months ago. I doubt waiting another 6 months would help much | 14:25 |
| Clark[m] | As for 901871 there may be another issue I was unable to check logs when it was brought up last night but the reading seemed to mimic the other identified problem so I associated it with that | 14:26 |
| frickler | Clark[m]: I checked that the gerrit error for 901871 looks the same as earlier, I just don't see where we get the null account from in that case | 14:34 |
| Clark[m] | I'll update the commit message on the 3.8.3 change after the school run this morning and we can plan a time to merge and restart. Seems possible to do that today | 14:47 |
| fungi | yes, i'll be out from ~16:00-17:00 utc but am otherwise available to help with a gerrit restart | 14:47 |
| Clark[m] | I have an appointment at 1900 that will probably last an hour as well | 14:49 |
| *** blarnath is now known as d34dh0r53 | 15:00 | |
| opendevreview | Will Szumski proposed openstack/diskimage-builder master: Adds support for setting build-args when using containerfile https://review.opendev.org/c/openstack/diskimage-builder/+/902160 | 15:05 |
| opendevreview | Will Szumski proposed openstack/diskimage-builder master: Adds support for setting build-args when using containerfile https://review.opendev.org/c/openstack/diskimage-builder/+/902160 | 15:13 |
| fungi | okay, i'm heading out, should be back in roughly an hour | 15:58 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update Gerrit 3.8 images to 3.8.3 https://review.opendev.org/c/opendev/system-config/+/901992 | 16:11 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add gerrit 3.9 image builds https://review.opendev.org/c/opendev/system-config/+/901468 | 16:11 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Add gerrit 3.8 to 3.9 upgrade testing https://review.opendev.org/c/opendev/system-config/+/901469 | 16:11 |
| clarkb | that updates the commit message for the 3.8.3 update. frickler does that look better to you? | 16:12 |
| clarkb | going back to why I think our current gerrit upgrade cadence is a good thing: We know what it is like to get left behind and stuck on old gerrit releases. We're basically stuck with no bugfixes and no features and an upstream that has no interest in helping us. With us staying about 1 release behind upstream we remain on a supported release, and we can easily have testing set up for | 16:13 |
| clarkb | the latest release (as we do now). This means we can both contribute our own fixes/needs upstream and get things reviewed by the experts as well as work with them to influence the future direction | 16:13 |
| clarkb | Essentially we're in a good balance point between testing all the new stuff and being too stale to move forward effectively | 16:14 |
| opendevreview | Merged zuul/zuul-jobs master: tox: Do not concat stdout and stderr in getting siblings https://review.opendev.org/c/zuul/zuul-jobs/+/901072 | 16:35 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Switch Gerrit replication to using an ed25519 key https://review.opendev.org/c/opendev/system-config/+/902169 | 16:42 |
| clarkb | infra-root ^ that is the gerrit side change for a gitea replication key rotation | 16:42 |
| clarkb | I think it would be good for people to be happy with both changes we can then land the gitea one and ensure it noops. Generate a new key and add it to key then add the key material to host var secrets and land the above change to have gerrit use it | 16:43 |
| opendevreview | Will Szumski proposed openstack/diskimage-builder master: Adds support for setting build-args when using containerfile https://review.opendev.org/c/openstack/diskimage-builder/+/902160 | 17:00 |
| fungi | okay, i'm back and not going anywhere else for the day, so available when people want to do gerrit work | 17:13 |
| clarkb | fungi: I think we should go ahead and approve the change as soon as we're happy with it. Then we can do a restart during my afternoon? I don't want to do it before my appointment will feel too rushed | 17:16 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/901992 that change to avoid confusion | 17:16 |
| fungi | wfm, yes | 17:16 |
| fungi | i had already +2'd the commit message update, but approving now | 17:17 |
| opendevreview | Merged opendev/system-config master: Update Gerrit 3.8 images to 3.8.3 https://review.opendev.org/c/opendev/system-config/+/901992 | 17:47 |
| clarkb | I don't think that actually promoted the image arg | 17:52 |
| clarkb | we can push a Dockerfile noop update and/or have the file matchers match the zuul.d/docker/gerrit.yaml file | 17:54 |
| clarkb | do we have a preference? I kinda like the idea of updating the file matchers because I think this isn't the first time this has happened to us | 17:54 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Add OpenInfra Asia mailing lists https://review.opendev.org/c/opendev/system-config/+/902176 | 17:55 |
| clarkb | fungi: ^ thoughts above since you approved teh change | 17:57 |
| fungi | clarkb: yes, let's fix the matchers | 17:57 |
| fungi | then we don't have to worry about it again | 17:58 |
| clarkb | ++ I'll leave the master image build out of that though since master is something that is less important | 17:59 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Trigger gerrit image promotion when the gerrit image jobs update https://review.opendev.org/c/opendev/system-config/+/902177 | 18:01 |
| clarkb | I think ^ should do it | 18:01 |
| fungi | isn't zuul supposed to detect changes to job definitions and run them anyway? | 18:04 |
| fungi | i guess it's that we're wanting a different job to run than what is changed? | 18:04 |
| clarkb | yes, but as noted in the commit message I think the issue here is the type of pipeline | 18:04 |
| fungi | yeah, okay | 18:04 |
| clarkb | we're triggering on change update or ref updated and I forget which can't actualyl do the thing | 18:04 |
| clarkb | one can iirc but the other can't and we're using the one that can't | 18:05 |
| clarkb | but if that isn't the case merging that change should also result in no promotion | 18:05 |
| clarkb | and we can revert in that case and I guess go back to doing the noop dockerfile update | 18:05 |
| fungi | yep, that was my thinking as well | 18:06 |
| fungi | we'll see either way | 18:06 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Trigger gerrit image promotion when the gerrit image jobs update https://review.opendev.org/c/opendev/system-config/+/902177 | 18:19 |
| clarkb | fungi: ^ I realized we weren't actually testing the new gerrit only building it | 18:20 |
| clarkb | I think that will fix this problem | 18:20 |
| fungi | aha | 18:21 |
| clarkb | I have to pop out in about 10-15 minutes to get to my appointment. Should be back around 20:00 UTC I hope | 18:32 |
| fungi | sounds good, i'll be here | 18:47 |
| opendevreview | Merged opendev/system-config master: Add OpenInfra Asia mailing lists https://review.opendev.org/c/opendev/system-config/+/902176 | 19:01 |
| fungi | deploy failed for 902176 and it looks like it's because infra-prod-base failed TASK [base/server : Ensure required build packages for non-wheel architectures] due to an apt update failure on mirror01.regionone.osuosl (unexpected file size for http://ddebs.ubuntu.com/dists/focal-updates/universe/binary-arm64/Packages.xz) | 19:48 |
| fungi | i'm able to apt update on that server cleanly now, so it must have been a transitive issue | 19:49 |
| fungi | transient | 19:49 |
| tonyb | fungi: so we'd reenque that job? | 19:50 |
| fungi | tonyb: yep, i just ran this on zuul01: `sudo zuul-client enqueue --tenant=openstack --pipeline=deploy --project=opendev/system-config --change=902176,1` | 19:51 |
| fungi | mainly because i'm not sure if the hourly deploy will run infra-prod-service-lists3 | 19:52 |
| fungi | the daily will, but would rather not wait for that | 19:52 |
| tonyb | Okay. For my own education I'll try to find answer to that question. | 19:53 |
| fungi | also, if i'd waited, then reenqueuing that would have to wait for the hourly buildset to finish, since they share a semaphore | 19:54 |
| fungi | tonyb: https://zuul.opendev.org/t/openstack/builds?pipeline=opendev-prod-hourly | 19:55 |
| fungi | pretty easy to check | 19:55 |
| TheJulia | Any chance you guys can hold the next failure of job "ironic-tempest-ipa-partition-uefi-pxe-grub2" for changeset 901182? Specifically we're seeing boot loader file sizes be setupidly off from what they should be when we try to download them, which is preventing us from validating if httpboot works or not. | 19:55 |
| tonyb | Ahh that's much easier than my process, but in my own defence I was starting from fundamentals ;P | 19:56 |
| tonyb | fungi: So it looks infra-prod-service-lists3 isn't on that pipeline. Should it be? | 19:57 |
| fungi | TheJulia: done | 19:57 |
| TheJulia | fungi: thanks! | 19:57 |
| fungi | for the record, i ran this on zuul01: | 19:57 |
| fungi | sudo zuul-client autohold --tenant=openstack --project=opendev.org/openstack/ironic --job=ironic-tempest-ipa-partition-uefi-pxe-grub2 --ref='refs/changes/82/901182/.*' --reason='TheJulia investigating a discrepancy in boot loader file sizes' | 19:57 |
| tonyb | fungi: perfect. You are forgiven for doing it faster than I could figure it out ;P | 19:58 |
| TheJulia | <3 | 19:58 |
| fungi | tonyb: no, i think we mainly keep the hourly buildset to infra-prod-base and a handful of other quick jobs. what i meant to say is i was pretty sure the listserv deploy job wasn't included there | 19:59 |
| fungi | so didn't want to wait for the daily deploy | 20:00 |
| tonyb | Ah okay got it | 20:00 |
| * tonyb has started a 'cheatsheet' doc with those last 2 scenarios | 20:04 | |
| corvus | clarkb: the promote pipeline does support auto job change detection, however the tenant reconfiguration with the change completes before adding the item to the pipeline, so there is no change from the running config (and that strict sequencing is deliberate -- we can now rely on that). | 20:04 |
| tonyb | I should turn it into an etherpad or something else shareable | 20:04 |
| corvus | tonyb: you can also use the web ui. | 20:04 |
| opendevreview | Brian Haley proposed openstack/project-config master: [Neutron-tempest-plugin] Update Grafana Dashboard https://review.opendev.org/c/openstack/project-config/+/902184 | 20:05 |
| corvus | also, if someone finishes hooking up keycloak to openstackid or whatever then we could start handing that out to other folks. | 20:05 |
| tonyb | corvus: I could that is true | 20:05 |
| Clark[m] | corvus we promote in the deploy pipeline | 20:06 |
| Clark[m] | Not sure if that changes anything | 20:06 |
| tonyb | corvus: I can do that, next year, at this stage. | 20:06 |
| *** cloudnull0 is now known as cloudnull | 20:07 | |
| corvus | tonyb: that's awesome. that would only be a 25% extension in the project timeline to date. :) | 20:08 |
| corvus | i mean, it depends on where you start counting. the server is like 4 years old... the idea is like 8 years old. | 20:09 |
| corvus | idea/spec | 20:09 |
| tonyb | LOL | 20:09 |
| tonyb | You can't rush a good thing | 20:10 |
| corvus | Clark: that makes my head explode but also it's the same. :) | 20:10 |
| fungi | tonyb: https://docs.opendev.org/opendev/infra-specs/latest/specs/central-auth.html if you want to review... not set in stone, just where we were hopefully heading | 20:10 |
| fungi | i think i wrote some of that spec while sitting in a teahouse in the middle of the garden at the tokyo summit venue | 20:11 |
| tonyb | How wonderful and terrible at the same time | 20:12 |
| fungi | fueled by suntori highball in a can from the 7-eleven | 20:12 |
| tonyb | LOL | 20:12 |
| corvus | i remember significant discussion at sydney when the monorail existed | 20:12 |
| fungi | what's that word? monorail! | 20:13 |
| tonyb | There is a monorail car inside the google office in Sydney | 20:14 |
| fungi | i've sold monorails to brockway, ogdenville and north haverbrook | 20:14 |
| corvus | fungi: turns out it was more of a shelbyville thing after all | 20:15 |
| fungi | every visit to seattle gives me flashbacks of that episode | 20:16 |
| Clark[m] | corvus: so if I understand correctly the latest change to explicitly trigger on that file updating won't promote this time either but will for the next time? | 20:17 |
| corvus | Clark: i think it should because the actual trigger will take effect by then | 20:17 |
| corvus | er actual files trigger | 20:17 |
| Clark[m] | Ah | 20:17 |
| corvus | (it's running now because of auto-config change, and when it merges, the tenant reconfig will be strictly sequenced before anything else relying on that config change, so the files matcher will be in effect before it considers adding that change to any pipelines) | 20:18 |
| * clarkb is finally back. Took a detour to eat some lunch | 20:33 | |
| clarkb | looks like I'm still ahead of the image promotion though so I didn't miss anything | 20:33 |
| opendevreview | Merged opendev/system-config master: Trigger gerrit image promotion when the gerrit image jobs update https://review.opendev.org/c/opendev/system-config/+/902177 | 20:38 |
| clarkb | cool promote did run | 20:42 |
| clarkb | so ya I think we can plan to do a gerrit restart whenever we like at this point. Did someone else want to drive it? THe main thing is moving aside the replication waiting queue between the docker-compose down and docker-compose up -d | 20:43 |
| fungi | i'm happy to do it unless tonyb wants a turn | 20:52 |
| tonyb | I think I'd rather watch one more to verify I grok the process. | 20:56 |
| fungi | no sweat, i got this | 20:57 |
| tonyb | Thanks | 20:57 |
| clarkb | in that case fungi: I think the process is basically docker-compose pull, docker-compose down, mv /whatever/the/path/is /home/gerrit/tmp/someplace/else, docker-compose up -d | 20:57 |
| fungi | step 0: check the image list on the server, but yeah | 20:57 |
| tonyb | let me know when you have a screen session | 20:57 |
| clarkb | Iv'e been moving things aside os that I have data to debug further though at this point that is probably less critical | 20:58 |
| tonyb | okay cool | 20:58 |
| fungi | i've initiated a root screen session on review.o.o | 20:58 |
| clarkb | I'm attached | 20:58 |
| fungi | we do have a lot of old images we should probably clean up | 20:58 |
| * tonyb to | 20:59 | |
| fungi | opendevorg/gerrit 3.2 780c9efed97e 2 years ago 793MB | 20:59 |
| fungi | is there a good way to prune these? should i care? | 20:59 |
| clarkb | fungi: there is a way. Most of our services autoprune images but we don't do that for gerrit iirc. But even then autoprune won't prune tagged images so the last tag for each gerrit version lives on | 21:00 |
| clarkb | fungi: what I think we should do is worry less about the 3.2/3.3 etc images and instead manually run a prune command to clear out everything older than say 6 months and get the intermediate images that are left behind | 21:00 |
| clarkb | and then later we can come through and manually delete the 3.2/3.3/ etc images | 21:01 |
| clarkb | fungi: its docker image prune iirc | 21:01 |
| clarkb | we have examples in system-config | 21:01 |
| fungi | cool, finding | 21:02 |
| clarkb | `docker image prune -f --filter "until=72h"` is what we do automatically on gitea | 21:02 |
| fungi | yeah, just found that in playbooks/roles/gitea/tasks/main.yaml | 21:02 |
| fungi | we want longer i guess? 180d? | 21:02 |
| clarkb | yes though the comment in gitea says it doesn't take day filters | 21:03 |
| clarkb | in my command history on review02 I ran `sudo docker image prune --filter "until=2022-05-01T00:00:00" -f` once | 21:03 |
| clarkb | I ran that on 2022-10-13T22:52:51 | 21:04 |
| clarkb | so I think I was doing 6 months ish then too | 21:04 |
| fungi | docker image prune -f --filter "until=2023-05-29T00:00:00" | 21:04 |
| clarkb | that looks about right | 21:05 |
| tonyb | +1 | 21:05 |
| fungi | running | 21:05 |
| fungi | status notice The Gerrit service on review.opendev.org will be restarting momentarily for a patch update to address a recently observed regression preventing some changes from merging | 21:06 |
| fungi | that look like an okay announcement? | 21:07 |
| clarkb | lgtm | 21:07 |
| tonyb | Yup good to me | 21:07 |
| clarkb | /home/gerrit2/review_site/data/replication/ref-updates/waiting <- this is the path to move aside | 21:07 |
| fungi | thanks, i almost forgot | 21:07 |
| fungi | pulling images now | 21:07 |
| fungi | and done | 21:08 |
| fungi | #status notice The Gerrit service on review.opendev.org will be restarting momentarily for a patch update to address a recently observed regression preventing some changes from merging | 21:08 |
| opendevstatus | fungi: sending notice | 21:08 |
| clarkb | you can mv it to /home/gerrit2/tmp/fungi/waiting_20231129 to follow what I've done when I do it | 21:08 |
| -opendevstatus- NOTICE: The Gerrit service on review.opendev.org will be restarting momentarily for a patch update to address a recently observed regression preventing some changes from merging | 21:08 | |
| clarkb | note it doesn't matter for gerrit but it is better to run the docker-compose commands in the docker-compose yaml dir in case there is an env file | 21:08 |
| fungi | yeah, i normally add an env option | 21:09 |
| fungi | haven't we usually moved that dir somewhere in the gerrit homedir to make it atomic? | 21:09 |
| clarkb | fungi: yes /home/gerrit/2/tmp/fungi is in the same disk | 21:09 |
| clarkb | and that tmp path is excluded from backups | 21:09 |
| fungi | d'oh | 21:10 |
| fungi | done and containers coming back up | 21:10 |
| fungi | i initially misread your suggestion as /home/fungi | 21:11 |
| opendevstatus | fungi: finished sending notice | 21:11 |
| clarkb | web ui loads for me | 21:11 |
| fungi | Powered by Gerrit Code Review (3.8.3-2-gb446549261-dirty) | 21:11 |
| clarkb | https://gerrit.googlesource.com/gerrit/+/refs/heads/stable-3.8 shows b44... is the cherry pick of the fix I pushed | 21:12 |
| clarkb | johnsom: frickler tkajinam I think you can try to merge those chagnes again | 21:12 |
| fungi | i was mainly going off the 3.8.3 since that's newer than what we had before | 21:12 |
| clarkb | ++ I also wanted to confirm the other bugfix we want was included too | 21:13 |
| clarkb | and it appears to be | 21:13 |
| johnsom | Ok, the patch I know of is staceyatorres@gmail.com | 21:13 |
| johnsom | Sorry, bad paste, https://review.opendev.org/c/openstack/designate/+/901871 | 21:13 |
| fungi | oh, we have a test case? | 21:14 |
| fungi | excellent | 21:14 |
| clarkb | hrm looks like frickler -W'd it so we need to remove that first | 21:14 |
| clarkb | but then you can reapprove and see if zuul can merge it | 21:14 |
| clarkb | johnsom: are you able to remove the -W as a core reviewer or do we need to do that as gerrit admin? | 21:14 |
| johnsom | It doesn't look like I can.... | 21:15 |
| johnsom | I thought we put ACLs in to allow that. hmmm | 21:15 |
| fungi | https://opendev.org/openstack/project-config/src/branch/master/gerrit/acls/openstack/designate.config grants toggleWipState to designate-core but doesn't allow workflow vote deletion | 21:17 |
| clarkb | I think we need to promote our regular account to admin to do it through the web ui. The only option through ssh is to remove frickler as a reviewer entirely | 21:18 |
| fungi | probably only admins can delete votes | 21:18 |
| fungi | yeah | 21:18 |
| johnsom | Yeah, that must be what I was thinking of | 21:18 |
| clarkb | tonyb: do you want to do that? | 21:18 |
| fungi | in time, we should probably scale back the label-Workflow to only 0..+1 and start relying exclusively on wip state | 21:19 |
| clarkb | fungi: I actually prefer the label myself still. It resets on new patchsets is the main reason which maybe we can make a toggle for the other thing | 21:19 |
| clarkb | I'm also no sure it was helpful to set wip on this cahnge. WIP is when you don't want something to merge but we want the opposite here :) | 21:21 |
| fungi | alternatively, a core reviewer could rebase the change and then reapprove it, but that will take longer courtesy of the clean-check policy for the openstack tenant | 21:22 |
| clarkb | and it may dirty our test case | 21:22 |
| clarkb | we know it can merge in the current state but a rebase by one of us may make it mergeable udner the old code | 21:22 |
| clarkb | anyway fungi tonyb or me who wants to become admin to remove the -W? | 21:23 |
| * tonyb has been looking up how to do that | 21:23 | |
| clarkb | tonyb: you need to use your ssh admin account to add your regular account to the Administrators group so that you can use the web ui to click the (x) next to the vote | 21:24 |
| clarkb | tonyb: so basically the process is `ssh -p29418 foo.admin@review.opendev.org gerrit set-members Administrators --add yourreguarly account` refresh the gerrit web page for that change, click the button to remove the vote then run set-members --remove instead of --add | 21:25 |
| tonyb | Ah okay. | 21:26 |
| clarkb | you can also do the gerrit ssh command `gerrit ls-members Administrators` to see current membership | 21:26 |
| opendevreview | Brian Rosmaita proposed openstack/project-config master: Implement openstack-unmaintained-core group https://review.opendev.org/c/openstack/project-config/+/902110 | 21:27 |
| tonyb | Okay frickler's -W has been removed | 21:29 |
| clarkb | johnsom: if you remove your +W and then add it back again that should trigger zuul to load it straight into the gate | 21:29 |
| clarkb | and then we awit for jobs to complete and hopefully succeed and see if zuul can merge the change | 21:30 |
| tonyb | and my account is no longer an Admin | 21:30 |
| clarkb | tonyb: the web ui has updated a couple times since I had to do that myself was the (x) fairly obvious and easy to find? | 21:30 |
| johnsom | Done | 21:31 |
| clarkb | johnsom: thanks! I see it in the gate now as expected | 21:31 |
| tonyb | clarkb: There was a little 'transcan' icon next to the vote. I had to do a little clicking to get it to show up but generally it wasn't too puzzling. | 21:32 |
| clarkb | cool good to know it is a trashcan now and not an 'x' | 21:34 |
| clarkb | tonyb: there is probably a way to do it via the rest api as well. But figuring out the values to plugin to get the right vote will probably take much longer than simply using the ui | 21:38 |
| tonyb | clarkb: It looks like we could do: ssh -p 29418 myname.admin@review.opendev.org "gerrit set-reviewers --project foo/bar --remove $USER_WITH_VOTE 123456" based on our docs? | 21:41 |
| clarkb | tonyb: I think that removes all of that users's votes which is fine for removing a -2 Verified because zuul only votes on verified. But for humans its more impactful? | 21:41 |
| clarkb | iirc the docs are talking specifically about remove a zuul -2 to make something mergeable | 21:41 |
| clarkb | that would've worked here but then we would lose frickler's +2 code review | 21:42 |
| tonyb | Ahhh got it | 21:43 |
| clarkb | https://review.opendev.org/c/openstack/neutron/+/901074 this change merged after the update. I have no reason to think it is a test case for the issue though. More just "merges generally work" | 21:48 |
| tonyb | Any ideas on https://zuul.opendev.org/t/openstack/build/d4a8d5cba63b4a2594a8a2db15355e00 (Change 901468,6) failure? | 21:52 |
| fungi | still better than "more arbitrary changes are now refusing to merge" i guess | 21:52 |
| clarkb | tonyb: thats weird beacuse we just build gerrit 3.8 like 4 times (we only wanted it to be built once) | 21:53 |
| clarkb | tonyb: it looks like the jvm itself crashed | 21:55 |
| fungi | shenanigans | 21:55 |
| clarkb | https://zuul.opendev.org/t/openstack/build/d4a8d5cba63b4a2594a8a2db15355e00/log/job-output.txt#9603-9615 | 21:56 |
| clarkb | in libc | 21:56 |
| clarkb | I do notice that we seem to be building with java 11 but should build with 17 now | 21:56 |
| clarkb | hrm the job does do ensure-java against version 17 | 21:57 |
| clarkb | https://zuul.opendev.org/t/openstack/build/d4a8d5cba63b4a2594a8a2db15355e00/log/job-output.txt#9354 and java 17 is our default java. Maybe ensure-bazelisk is running under java 11 anyway for some reason | 21:59 |
| clarkb | hrm no as far as I can tell bazelisk uses the default java by default. Maybe the build target needs a specific java version and it switches over (this joys of build tools that do whatever they want) | 22:00 |
| clarkb | ah yup the command has external/remotejdk11_linux/bin/java in it so I think that is what has happened there. | 22:00 |
| clarkb | I think we might have to treat this as "cosmic rays" type of situation | 22:01 |
| clarkb | in reality it was maybe some sort of race in libc that created a fault? iirc those are becoming popular these days | 22:01 |
| clarkb | https://rachelbythebay.com/w/2017/01/30/env/ | 22:01 |
| fungi | wow, nice | 22:05 |
| fungi | "Modifications of environment variables are not allowed in multi-threaded programs. -- the glibc manual" | 22:06 |
| clarkb | I'm trying to think what sorts of stuff the jvm will go through libc for. It can't be much | 22:08 |
| fungi | https://rachelbythebay.com/w/2023/10/16/env/ | 22:08 |
| fungi | "If you're on Linux, and you're using glibc, you're probably a passenger on this boat. Try not to drill any more holes." | 22:19 |
| johnsom | The patch merged | 22:24 |
| tonyb | \o/ | 22:25 |
| fungi | esselent | 22:35 |
| clarkb | excellent | 22:35 |
| clarkb | fungi: next on my ops todo list is https://review.opendev.org/c/opendev/system-config/+/901082 and https://review.opendev.org/c/opendev/system-config/+/902169 that first one I think is safe to merge whenever (it should noop because the key already exists in gitea) | 22:37 |
| clarkb | once that first one is in I'll generate a new key and do a followup change to set the new pubkey in gitea and add the key to hostvars and we can merge the second if we are happy with it | 22:37 |
| clarkb | and that should be all we need to do to rotate keys between gerrit and gitea other than cleaning up the old key | 22:38 |
| fungi | TheJulia: ssh root@217.182.140.93 | 22:49 |
| tonyb | For creating replacement infra servers it's 0) test new config; 1) add server; 2) add DNS; 3) add to inventory etc . I assume cleanup is the reverse remove from inventory, remove from DNS, remove old server #profit | 22:58 |
| clarkb | tonyb: pretty much. The important thing is to remove from inventory first to avoid errosr when we try to renew LE stuff on the server and DNS is gone | 22:59 |
| clarkb | but I think you can remove dns after you delete the server if that lines up better | 22:59 |
| fungi | order on cleanup is somewhat loose, but basically remove from dns and inventory (order irrelevant), possibly create an archival image of the server (if there's a chance we might need to recreate it), then delete it in the provider | 23:00 |
| tonyb | Got it. | 23:43 |
| tonyb | I'll try spinning up new jvb and meetpad servers tomorrow | 23:45 |
| opendevreview | Merged opendev/system-config master: Add ssh key rotation to gitea ssh key management https://review.opendev.org/c/opendev/system-config/+/901082 | 23:53 |
| opendevreview | Brian Rosmaita proposed openstack/project-config master: Implement openstack-unmaintained-core group https://review.opendev.org/c/openstack/project-config/+/902110 | 23:58 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!