| johnsom | clarkb Yep. So, along that line.... openstackdocstheme , grep -r comes up empty | 00:00 |
|---|---|---|
| ianw | clarkb: lgtm; i think just approve them since you investigated | 00:01 |
| ianw | i think i may have noticed the effects of that, but never dug into it, thanks | 00:01 |
| clarkb | ianw: can do, I'll +A and monitor tomorrow then | 00:02 |
| clarkb | johnsom: if you look in project-config/gerrit/projects.yaml it shows which acl file is used for every project (by default the file has the same name of the project if not specified) | 00:03 |
| clarkb | johnsom: looks like that repo uses acls/openstack/oslo-independent.config | 00:03 |
| johnsom | Ah, got it. I guess I am super rusty with working in project-config. lol Thanks! | 00:04 |
| clarkb | no problem. It isn't really self documented | 00:04 |
| ianw | https://review.opendev.org/c/opendev/grafyaml/+/825990 is a boring one but needed to run against the latest grafanas it seems. pairs with https://review.opendev.org/c/opendev/system-config/+/825410 which proposes just using the latest tag for grafana, so we break on things like this early instead of having them sit latent | 00:06 |
| clarkb | ianw: ok. Does that take into account that the db is apparently ephemeral and if we update the containers we'll need to regenerate dashboard? I avoided rebooting grafana on friday for this reason | 00:07 |
| clarkb | I wasn't sure how to force it to regenerate afterwards | 00:07 |
| clarkb | I guess we probably have a service grafana job that can be manually triggered | 00:07 |
| ianw | yes i feel like we trigger updates when db's change or we restart the container | 00:08 |
| ianw | yeah it's in the deploy pipeline of project-config | 00:09 |
| clarkb | got it, so worst case land a nop change to the grafyaml files | 00:09 |
| ianw | https://opendev.org/openstack/project-config/src/branch/master/zuul.d/projects.yaml#L4531 | 00:10 |
| ianw | yeah, or run the playbook by hand i guess | 00:10 |
| ianw | i'm not sure the db being ephemeral is a bug or a feature | 00:11 |
| clarkb | if we load it back in again on a restart that is fine | 00:11 |
| clarkb | My biggest concern is not being able to drop the container safely | 00:12 |
| clarkb | and that would only be a problem if we need to do some crazy intervention afterwards I guess | 00:12 |
| ianw | we should perhaps just have a script on disk in /usr/local/bin/ to quickly reload the dashboards | 00:12 |
| clarkb | that would work too | 00:13 |
| ianw | ok, on the todo list. also need to update the documentation to account for not having grafyaml in the container as well | 00:13 |
| opendevreview | Merged opendev/system-config master: Upgrade Gerrit to 3.4 https://review.opendev.org/c/opendev/system-config/+/826148 | 00:15 |
| opendevreview | Merged opendev/system-config master: Add docs for restoring an etherpad https://review.opendev.org/c/opendev/system-config/+/826017 | 00:15 |
| clarkb | I've made some updated to the meeting agenda. Is there anything else to add? | 00:18 |
| clarkb | if so please get it on there in the next little bit before I send it out | 00:18 |
| clarkb | https://opendev.org/opendev/system-config/commits/branch/master replication is still looking good | 00:19 |
| clarkb | ok agenda sent | 00:44 |
| clarkb | the job to set 3.4 in the docker compose file seems to have nooped that file as expected | 00:45 |
| ianw | yep agree lgtm | 01:17 |
| opendevreview | Ian Wienand proposed opendev/system-config master: grafana: update docs and make an import script https://review.opendev.org/c/opendev/system-config/+/826241 | 03:19 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: yum-minimal: don't strip -* from releasever https://review.opendev.org/c/openstack/diskimage-builder/+/826244 | 04:51 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: Add debian-bullseye-arm64 build test https://review.opendev.org/c/openstack/diskimage-builder/+/821652 | 04:54 |
| opendevreview | Ian Wienand proposed openstack/diskimage-builder master: debian-minimal: remove old testing targets https://review.opendev.org/c/openstack/diskimage-builder/+/821654 | 04:54 |
| *** ysandeep|out is now known as ysandeep | 05:41 | |
| *** amoralej|off is now known as amoralej | 08:04 | |
| *** jpena|off is now known as jpena | 08:35 | |
| *** ysandeep is now known as ysandeep|lunch | 09:27 | |
| *** ysandeep|lunch is now known as ysandeep | 10:28 | |
| *** rlandy|out is now known as rlandy|ruck | 11:14 | |
| *** dviroel|out is now known as dviroel | 11:20 | |
| *** sshnaidm|afk is now known as sshnaidm | 12:28 | |
| *** ysandeep is now known as ysandeep|mtg | 12:59 | |
| gthiemonge | Hey Folks, with the latest gerrit update, I can no longer use the "is:mergeable" search term, it fails with an error (400 Bad Request). It also seems that changes in "Merge Conflict" are no longer flagged when they are in a list view. | 13:11 |
| *** mrunge_ is now known as mrunge | 13:11 | |
| *** amoralej is now known as amoralej|lunch | 13:31 | |
| Clark[m] | gthiemonge: that is a known change https://www.gerritcodereview.com/3.4.html#ismergeable-predicate-is-disabled-per-default | 13:54 |
| gthiemonge | Clark[m]: Ok, I can leave without is:mergeable, but the missing "merge conflict" status in a list is more annoying :-( | 13:59 |
| Clark[m] | They are essentially the same thing. | 14:00 |
| Clark[m] | Well from the Gerrit cost perspective as they are both calculated by the same process | 14:01 |
| Clark[m] | Th good news is zuul reports merge failures so we don't completely lose this info. However, zuul doesn't constantly recalculate it (which is where a lot of the cost is) | 14:01 |
| *** amoralej|lunch is now known as amoralej | 14:02 | |
| opendevreview | Merged openstack/project-config master: Use same ACL for all OpenStack-Ansible Projects https://review.opendev.org/c/openstack/project-config/+/824230 | 14:06 |
| opendevreview | Merged openstack/project-config master: Add Backport-Candidate label to openstack-ansible ACL https://review.opendev.org/c/openstack/project-config/+/824229 | 14:06 |
| Clark[m] | fungi: I won't be able to look for a bit yet but the releases change in behavior may be related to the MINA update between 3.3 and 3.4. This update doesn't get us far enough ahead to fix the ssh-rsa sha1 issue but they did move MINA ahead a bit. | 14:11 |
| *** ysandeep|mtg is now known as ysandeep | 14:17 | |
| fungi | Clark[m]: yeah, looking deeper, i think that may have been a benign warning and the error is something to do with tag pushing permissions | 14:19 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Update Gerrit IP addresses in SSH key secrets https://review.opendev.org/c/openstack/project-config/+/826294 | 14:25 |
| fungi | elodilles: did something change about how tags are being created? this doesn't look like it's making a signed tag: https://zuul.opendev.org/t/openstack/build/a490703c83514eb7ba9d0ff6307f7371/log/job-output.txt#626 | 14:27 |
| fungi | oh, nevermind, there's multiple lines | 14:27 |
| fungi | the final line has the -s | 14:27 |
| Clark[m] | fungi: maybe check the Gerrit side logs if you haven't already? | 14:36 |
| fungi | yeah, i'm about to | 14:36 |
| fungi | it doesn't log any errors in the sshd_log | 14:42 |
| fungi | nothing related in error_log | 14:43 |
| Clark[m] | What about the error_log? | 14:43 |
| Clark[m] | Ah ok | 14:43 |
| Clark[m] | I wonder if trying it manually will produce stdout that is helpful like when we push changes and it gives you the change link or an error saying you can't push an identical patchset | 14:44 |
| fungi | i'm pondering whether it's time to tag one of our own repos | 14:47 |
| Clark[m] | Or sandbox | 14:48 |
| fungi | well, if git-review or bindep or similar are due for a release anyway, it would provide a direct means of troubleshooting | 14:49 |
| fungi | we can also of course add a testinfra test where we push a tag, though there will be a bit of key generation to worry about | 14:49 |
| Clark[m] | I think bindep has a couple changes that would be ok to tag | 14:50 |
| fungi | yeah, looking | 14:50 |
| fungi | and trying to get settled in to listen to the board meeting in 10 minutes | 14:50 |
| Clark[m] | Ya that is why I'm up early today | 14:51 |
| fungi | the only non-test-related bindep change to land since 2.10.1 is a fix to support a missing newline on the last line of an input file | 14:53 |
| fungi | so i'll make it 2.10.2 | 14:54 |
| clarkb | ++ | 14:54 |
| clarkb | fungi: once we have a bit more info from ^ I can send email to the gerrit mailing list with what we have found if it indicates a regression. | 14:55 |
| fungi | remote: You need 'Create Tag' rights to push a normal tag. | 14:58 |
| fungi | ! [remote rejected] 2.10.2 -> 2.10.2 (prohibited by Gerrit: not permitted: create tag) | 14:58 |
| fungi | so yes, i can replicate this | 14:59 |
| clarkb | fungi: and your tag was signed? I can send email about that in a bit. But in the meantime I guess we add that permission to the openstack meta perms and notify others? | 14:59 |
| clarkb | (and they can update perms as they go?) | 14:59 |
| fungi | right, pgp-signed tag | 14:59 |
| fungi | git tag -s | 14:59 |
| clarkb | fungi: or maybe we update bindep first and confirm that fixes it for signed tags | 14:59 |
| clarkb | and if that works update openstack meta config | 14:59 |
| fungi | yep | 14:59 |
| clarkb | sounds like a plan, thank you for looking at this | 15:00 |
| fungi | working on it once i get the board meeting up | 15:00 |
| *** sshnaidm is now known as sshnaidm|afk | 15:01 | |
| opendevreview | James E. Blair proposed zuul/zuul-jobs master: Add upload-logs-ibm role https://review.opendev.org/c/zuul/zuul-jobs/+/826158 | 15:03 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Grant Create Annotated Tag perms on bindep https://review.opendev.org/c/openstack/project-config/+/826305 | 15:10 |
| fungi | clarkb: ^ | 15:10 |
| clarkb | fungi: side note looks like pushSignedTag and pushTag are legacy values that are translated to createTag and createSignedTag | 15:12 |
| clarkb | we can sort that out later. I'll approve ^ | 15:12 |
| fungi | oh neat, i wonder if that's related | 15:12 |
| clarkb | I dno't think so looking at the code, but it is possible I suppose | 15:13 |
| clarkb | fungi: I guess if you want we could change to createSignedTag first | 15:13 |
| clarkb | maybe that is a better thing to check first | 15:13 |
| clarkb | I'll remove my +A | 15:13 |
| fungi | yeah, i'll give that a shot, thanks | 15:13 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Update bindep ACL to use new createSignedTag perm https://review.opendev.org/c/openstack/project-config/+/826309 | 15:17 |
| fungi | clarkb: i pushed it as a separate change and will wip the old one for now | 15:18 |
| clarkb | sounds good | 15:18 |
| clarkb | approved | 15:19 |
| fungi | if that one works, i'll just update all the acls and our docs asap | 15:20 |
| clarkb | and I can send email to the gerrit list asking about it. That wasn't in the breaking changes list :) | 15:20 |
| *** ysandeep is now known as ysandeep|out | 15:23 | |
| fungi | indeed | 15:26 |
| clarkb | I asked quickly on their slack in case anyone recognizes this and can provide advice | 15:26 |
| clarkb | fungi: you may also want to fetch refs/meta/config for bindep as admin after that update applies to be extra sure it updated as we expect | 15:29 |
| fungi | yeah | 15:30 |
| fungi | i plan to make sure it actually applies | 15:30 |
| clarkb | ++ | 15:30 |
| fungi | i went ahead and fetched it now for a baseline, and it's still pushSignedTag, so it's not being silently translated by gerrit on push i don't think | 15:33 |
| clarkb | fungi: ya looking at the code any translation that may be happening seems to happen internally. And I'm wondering if they stopped doing that somewhere | 15:33 |
| opendevreview | Merged openstack/project-config master: Update Gerrit IP addresses in SSH key secrets https://review.opendev.org/c/openstack/project-config/+/826294 | 15:34 |
| *** dviroel is now known as dviroel|lunch | 15:37 | |
| clarkb | I think merging ^ caused a reconfiguration slowing us down :) | 15:39 |
| opendevreview | Merged openstack/project-config master: Update bindep ACL to use new createSignedTag perm https://review.opendev.org/c/openstack/project-config/+/826309 | 15:39 |
| yoctozepto | hi! long time no see; would it be possible to add this button to opendev's gerrit? https://gerrit-review.googlesource.com/Documentation/images/user-review-ui-change-screen-quick-approve.png | 15:42 |
| yoctozepto | (cc priteau) | 15:43 |
| clarkb | yoctozepto: I think it is already there. Sort of | 15:43 |
| clarkb | I get a workflow+1 button | 15:44 |
| yoctozepto | clarkb: only W+1 | 15:44 |
| yoctozepto | yeah | 15:44 |
| clarkb | ya I suspect it is related to what is necessary to make things submittable | 15:44 |
| yoctozepto | ah | 15:44 |
| clarkb | so gerrit is being "smart" and saying the last vote needed is W+1 and shows you that button | 15:44 |
| yoctozepto | makes sense | 15:44 |
| clarkb | on upstream gerrit they only have CR+2 | 15:44 |
| clarkb | so they are equivalent behaviors due to gerrit's smartness | 15:44 |
| yoctozepto | ok, I get it, thanks | 15:45 |
| priteau | I see, thanks | 15:46 |
| clarkb | yoctozepto: I think you can test this by adding a W+1 and not a CR+2 | 15:46 |
| clarkb | then see if CR+2 shows up | 15:46 |
| yoctozepto | clarkb: yeah, I have independently thought about that and it shows CR+2 indeed | 15:46 |
| yoctozepto | it somehow cannot show both :D | 15:47 |
| clarkb | ya its a short cut for showing you specifically what is necessary to submit | 15:47 |
| yoctozepto | mhm | 15:47 |
| clarkb | so it only shows up when all other votes are done | 15:47 |
| fungi | V+2 is also required though, right? | 15:48 |
| clarkb | fungi: gerrit knows Verified is special | 15:48 |
| fungi | (and only zuul has permission to add that) | 15:48 |
| clarkb | but yes | 15:48 |
| clarkb | fungi: the deploy job completed. Checking if zuul says it was a success | 15:48 |
| clarkb | yup was a success according to zuul. I think you can refetch the acl now and check it | 15:49 |
| clarkb | (I don't have keys in place yet) | 15:49 |
| fungi | refs/meta/config has createSignedTag = group bindep-release now | 15:49 |
| fungi | and yeah, i saw the e-mail from gerrit about the successful deploy so was already pulling that | 15:50 |
| fungi | "remote: You need 'Create Tag' rights to push a normal tag." | 15:52 |
| fungi | well, it was worth a try. updating the other workaorund now | 15:52 |
| clarkb | ++ | 15:52 |
| *** sshnaidm|afk is now known as sshnaidm | 15:55 | |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Grant Create Annotated Tag perms on bindep https://review.opendev.org/c/openstack/project-config/+/826305 | 15:56 |
| clarkb | can you remove the WIP on that? I'll apprive it | 15:56 |
| fungi | oh, yep. i want to make one update to it anyway | 15:57 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Grant Create Annotated Tag perms on bindep https://review.opendev.org/c/openstack/project-config/+/826305 | 15:57 |
| clarkb | ah yup removing the pushTag option | 15:58 |
| fungi | it's readied now | 15:58 |
| clarkb | approved | 15:59 |
| fungi | thanks | 16:00 |
| clarkb | Should we send a note to service-announce so that more than openstack is aware of this? | 16:00 |
| clarkb | I can do that if you think that is worthwhile. | 16:00 |
| fungi | once we see if this works, yes i was planning to | 16:01 |
| clarkb | ah ok sounds good | 16:01 |
| fungi | another thought, has gerrit possibly added a feature for tracking signing keys (for authorizing pushing signed commits as well as tags)? | 16:01 |
| fungi | maybe if it doesn't recognize the key that made the signature as authorized it falls back to treating it as unsigned? | 16:01 |
| clarkb | they do have features for maintaining a keychain to show if signed objects are verified but pretty sure none of that impacts your ability to push | 16:01 |
| clarkb | just whether or not it is validated by the keychain after | 16:02 |
| fungi | ahh | 16:02 |
| clarkb | https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#receive.enableSignedPush | 16:03 |
| clarkb | we set enableSignedPush explicitly to false | 16:03 |
| clarkb | it really shouldn't be that. If it is that then this is a pretty big gerrit bug | 16:04 |
| fungi | we can certainly test that next | 16:05 |
| fungi | also why are we setting enableSignedPush to false? that seems like something we wouldn't want to reject | 16:05 |
| fungi | oh, also signed pushes != signed commits | 16:05 |
| clarkb | fungi: because of this problem that you describe | 16:05 |
| clarkb | enabled signed pushes means every signed object needs to be validated by the gerrit keychain | 16:05 |
| fungi | oh, got it | 16:06 |
| clarkb | its a validation thing. You can still push signed with it set to false | 16:06 |
| fungi | well, presumably that the push operations need to be signed, not the git objects | 16:06 |
| fungi | signing push operations is separate from signing commits (you can do either or both) | 16:06 |
| clarkb | well hwoever git does it. If you set that to true gerrit will validate against its keychain in all-users | 16:06 |
| clarkb | we don't want that | 16:06 |
| clarkb | ah | 16:06 |
| fungi | i think we do allow signed commits to be pushed | 16:07 |
| fungi | but i guess by unsigned push operations | 16:07 |
| clarkb | yes people like mtreinish have pushed signed commits for a long time iirc | 16:08 |
| fungi | the reason signed push was added to git was that just because a commit is signed, that doesn't mean pushing it to the remote was authorized by the signer | 16:08 |
| clarkb | looking at the code I don't think it is a key validation problem. | 16:08 |
| clarkb | There is a case statement with createSignedTag handled first then createTag | 16:09 |
| clarkb | I suspect a regression where it doesn't see it as a signed tag for whatever reason so falls into the regular tag perm checking | 16:09 |
| fungi | as do i | 16:09 |
| fungi | and most people just allow both so haven't noticed | 16:09 |
| fungi | and wmf hasn't upgraded to 3.4 yet | 16:10 |
| clarkb | ya | 16:10 |
| clarkb | https://gerrit.googlesource.com/gerrit/+/refs/heads/stable-3.4/java/com/google/gerrit/server/permissions/RefControl.java#547 is what we're hitting and not the block above | 16:13 |
| fungi | and that changed since 3.3? or how it's being called into changhed? | 16:15 |
| clarkb | the check method doesn't seem different. I expect a delta in the caller somewhere | 16:15 |
| clarkb | 826305 doesn't seem to eb gating despite getting a +1? | 16:20 |
| clarkb | maybe I need to be more patient with zuul | 16:21 |
| fungi | note that the status page displays per-pipeline event/result queues now | 16:23 |
| fungi | separate from the general queues | 16:23 |
| clarkb | https://gerrit.googlesource.com/gerrit/+/refs/heads/stable-3.3/java/com/google/gerrit/server/project/CreateRefControl.java#105 and https://gerrit.googlesource.com/gerrit/+/refs/heads/stable-3.4/java/com/google/gerrit/server/project/CreateRefControl.java#105 are the same | 16:23 |
| clarkb | but maybe getFullMessage() isn't returnign the same data now? | 16:24 |
| fungi | it's gating now | 16:25 |
| clarkb | looking at that code I half expect your fix to work since we're falling through and failing the check. We're just failing the wrong unexpected check. Explicitly setting that check to pass as your change should do should hopefully make it work. Then we can send mail to the gerrit mailing list and take it from there | 16:27 |
| fungi | assuming it works, should we push a temporary mass acl update? | 16:30 |
| clarkb | I guess? | 16:31 |
| clarkb | I dunno | 16:31 |
| fungi | if so, i'll get it underway before sending to the announce list | 16:31 |
| clarkb | The concern would be that people could push insigned commits. I guess I'm ok with that | 16:32 |
| clarkb | we can warn them to be sure they continue to sign as the platform won't prevent them from forgetting to | 16:32 |
| fungi | right, i'd basically plan to revert it asap | 16:33 |
| opendevreview | Merged openstack/project-config master: Grant Create Annotated Tag perms on bindep https://review.opendev.org/c/openstack/project-config/+/826305 | 16:33 |
| clarkb | I think I found it | 16:36 |
| clarkb | let me see if I can get a link to the commit in jgit | 16:36 |
| clarkb | https://gerrit.googlesource.com/jgit/+/dd3846513bbc682b9c51b09d369687ab7a036a49%5E%21/ | 16:37 |
| *** dviroel|lunch is now known as dviroel | 16:38 | |
| clarkb | the tests in there pretty clearly show the signature isn't in the results of getFullMessage() anymore | 16:39 |
| clarkb | I'm trying to figure out which version of jgit is used now | 16:39 |
| fungi | whoops | 16:40 |
| clarkb | ya that commit is in the 3.4 jgit submodule checkout and not in 3.3's | 16:40 |
| clarkb | now I'm basically 99.9% sure :) | 16:40 |
| clarkb | I might even be able to fix this if I can figure out how to get the signature from jgit | 16:41 |
| fungi | you have to get the full message | 16:41 |
| fungi | it's literally embedded in the end of the tag message | 16:41 |
| fungi | the annotation | 16:41 |
| clarkb | fungi: ya but jgit getFullMessage() doesn't include it | 16:41 |
| clarkb | thats what the commit I linked above does | 16:41 |
| clarkb | anyway I think we can temporarily add the extra permission then work to fix this upstream in gerrit since I'm prety sure I identified the issue now | 16:42 |
| clarkb | ya I see hwo to do it | 16:44 |
| *** marios is now known as marios|out | 16:50 | |
| clarkb | fungi: looks like deploy for bindep is done I guess you can test that now? | 16:53 |
| corvus | clarkb: fungi any outstanding zuul questions from yesterday? sorry i'm declaring backscroll bankruptcy... | 16:53 |
| clarkb | corvus: nope. Only issue is a gerrit issue which we think we just ran down | 16:54 |
| clarkb | I'm working on an issue and a fix for upstream | 16:54 |
| clarkb | and fungi is testing a workaround | 16:54 |
| corvus | ++ | 16:54 |
| clarkb | corvus: there is an impact to zuul though. Until we know the workaround works and apply it to zuul you won't be able to push zuul tags | 16:54 |
| corvus | yeah, was just reading back; makes sense | 16:55 |
| fungi | createTag = group bindep-release is now definitely in there from gerrit's perspective, so testing again | 16:56 |
| fungi | * [new tag] 2.10.2 -> 2.10.2 | 16:56 |
| fungi | that seems to have solved it | 16:56 |
| clarkb | woot | 16:56 |
| fungi | so next question, do we push this workaround to everyone's acls and plan to revert it once we can update to a fixed gerrit? | 16:57 |
| clarkb | I'll get the issue filed momentarily. push the fix upstream, then we can also do a local patch with our ci builds and test there | 16:57 |
| corvus | tldr workaround is allow pushing unsigned tags because signed tags appear to be unsigned, and tell ppl to be real careful? | 16:57 |
| clarkb | corvus: yes | 16:57 |
| fungi | corvus: precisely | 16:57 |
| clarkb | fungi: I think we can | 16:57 |
| clarkb | since I'd prefer we not deploy a forked gerrit. I'm ok with using local patched gerrit to test thigns but best to not fork imo | 16:57 |
| fungi | i'll start putting together the bulk change in that case. we should take this opportunity to also switch to the non-deprecated tag pushing access names | 16:58 |
| corvus | fwiw, no current plans for zuul releases this week; but yes next week. | 16:58 |
| fungi | i know starlingx is on the eve of a major release as well | 16:58 |
| clarkb | https://bugs.chromium.org/p/gerrit/issues/detail?id=15616 issue filed | 16:58 |
| fungi | thanks! | 16:58 |
| clarkb | working on a aptch for upstream next. Then will also push a change to test our thing locally | 16:59 |
| fungi | i'm putting together the sed syntax this requires. will need backrefs | 16:59 |
| clarkb | https://gerrit-review.googlesource.com/c/gerrit/+/328839 | 17:04 |
| clarkb | the new bindep made it to pypi a couple minutes ago | 17:07 |
| clarkb | I'm going to find some breakfast now and will get our local build for testing pushed up afterwards | 17:08 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Work around signed tag regression from Gerrit 3.4 https://review.opendev.org/c/openstack/project-config/+/826334 | 17:15 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Remove unsigned tagging permission from projects https://review.opendev.org/c/openstack/project-config/+/826335 | 17:15 |
| fungi | clarkb: corvus: ^ | 17:16 |
| fungi | i'll wip 826335 but it's there so we don't forget to unwind this at the earliest opportunity | 17:16 |
| clarkb | hrm I wonder if our job will timeout trying to apply all those updates | 17:16 |
| clarkb | do we want to find out the hard way or update in blocks? | 17:16 |
| clarkb | also shouldn't openstack configs not include it at all? | 17:17 |
| fungi | manage-projects is idempotent, so if applying fails we can just rerun manually | 17:17 |
| clarkb | ok | 17:17 |
| fungi | clarkb: not all openstack namespace repos are released by the release team | 17:17 |
| fungi | if you look at which acls are updating in that namespace, you'll see it's mainly repos managed by sigs | 17:19 |
| fungi | i'll start working on an announcement next | 17:20 |
| fungi | but need to grab a cup of tea and switch computers now that the board meeting has wrapped | 17:20 |
| clarkb | I've reviewed the first 10% of the change and it looks good so far. But I really need to eat so will come back to this | 17:20 |
| clarkb | ya same here was early morning and i need to go take some feel human steps :) | 17:20 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM Testing fix for pushing signed gerrit tags https://review.opendev.org/c/opendev/system-config/+/826341 | 17:34 |
| *** jpena is now known as jpena|off | 17:34 | |
| fungi | clarkb: frickler: corvus: proposed announcement is here: https://etherpad.opendev.org/p/qIWue_aVGRXzGEaZCxkD | 17:35 |
| fungi | feel free to edit | 17:35 |
| clarkb | fungi: I made a couple of changes | 17:36 |
| clarkb | feel free to revert them. I think the message loosk good | 17:36 |
| fungi | looks like the bulk acl edit needs some fixups for alpha reordering in acls which had additional stuff in the same blocks | 17:37 |
| frickler | I'm fine with the announcement text, too, I'd just suggest to also send it to openstack-discuss | 17:40 |
| fungi | i was planning to separately follow up to openstack-discuss on the ongoing thread we've been using there | 17:41 |
| fungi | as well as alert the denizens of #openstack-release | 17:42 |
| frickler | ah, that's fine, too. /me didn't check mails for two hours. also great that you could work this all out already | 17:42 |
| corvus | lgtm | 17:43 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Work around signed tag regression from Gerrit 3.4 https://review.opendev.org/c/openstack/project-config/+/826334 | 17:43 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Remove unsigned tagging permission from projects https://review.opendev.org/c/openstack/project-config/+/826335 | 17:43 |
| fungi | thanks | 17:43 |
| fungi | those should hopefully pass our normalization checks this time | 17:43 |
| clarkb | I've got an autohold in plce for the system-config-run-review3.4 job that 826341 has triggered. We should be able to locally push a signed git tag there confirming the fix | 17:44 |
| clarkb | Might take some time as we'll have to update the acls and create a gpg key. Maybe I should've tried to do more of this directly in the CI system. Too many things today though | 17:44 |
| *** amoralej is now known as amoralej|off | 17:44 | |
| fungi | btw, if ever there was a time i appreciated our forced normalization for acl files, it's now. rather than being a total mess, these bulk updates were barely more than a simple sed -i | 17:44 |
| clarkb | nice | 17:45 |
| fungi | i'll work on the documentation updates for the new createSignedTag syntax while i'm waiting for those to gate | 17:47 |
| clarkb | fungi: I'm super deep into the first change but does the revert keep the new create directives over push? | 17:58 |
| clarkb | I assume so since it says remove unsiged tag permission not revert | 17:58 |
| clarkb | I've gotten through to the first x/ repo and no issues jump out at me. I'll keep reviewing | 18:03 |
| fungi | yeah, the revert isn't a git revert, just a sed -i to remove the createTag lines | 18:04 |
| fungi | and go back to disallowing them in our linter | 18:04 |
| clarkb | ++ | 18:04 |
| fungi | looks like they're passing tests now | 18:09 |
| fungi | once we confirm successful deployment and/or manually run manage-projects, i'll send the announcement | 18:09 |
| clarkb | I've +2'd the first change after looking at all the files (quickly but I opened them and tried to check the group stayed the same and the new directitves were the ones we want) | 18:13 |
| clarkb | not sure if frickler intends on reviewing but we can probably approve now? | 18:13 |
| clarkb | corvus: ^ you might be interested since zuul is affected too | 18:13 |
| mtreinish | fungi, clarkb: yeah I've been pushing signed commits for a long time. | 18:19 |
| fungi | thanks for confirming! | 18:19 |
| mtreinish | for example: https://opendev.org/opendev/subunit2sql/commit/35793871f3c7ac21f078218f86f9a5f89c4fa56f | 18:20 |
| fungi | yeah, we've observed it does interesting things to the gitea webui | 18:20 |
| corvus | fungi: clarkb both lgtm | 18:21 |
| fungi | specifically the "No known key found for this signature in database" message displayed under the committer there | 18:21 |
| fungi | nice that it does display the gpg key id though | 18:21 |
| fungi | so someone can check it themselves | 18:21 |
| mtreinish | yeah, I just assumed that was because gitea had it's own key database and I hadn't submitted to it. But I think we talked about that a while ago and it was for something else | 18:21 |
| mtreinish | on github it behaves a bit differently. You upload your own public key and then it puts a little 'verified' tag on commits. Like: https://github.com/mtreinish/stestr/commit/4c219d02fd857a8a1645940d5a792d1e9a5224bf | 18:23 |
| mtreinish | and if you haven't uploaded your pubkey it says 'unverified' | 18:23 |
| fungi | i think current gitea basically does the same, but yeah because our gitea doesn't have accounts it's not possible to map those | 18:24 |
| clarkb | 158.69.73.149 should be our held gerrit with my patch applied | 18:40 |
| clarkb | Will work on testing with that after meetings and lunch | 18:40 |
| fungi | i theory, pushing a tag is something we could add to our testinfra too | 18:53 |
| clarkb | yup and I think we should | 18:53 |
| clarkb | but I looked at the code for it and tehre is a bit that would need to happen. I think quicker for now to confirm it works, then work on adding that after | 18:53 |
| clarkb | Unless you think I should just go ahead and start iteratign on it with CI? | 18:54 |
| fungi | nah, testing a held node for now is a good quick check that we're on teh right track | 18:54 |
| fungi | i have a feeling most of the work on the testinfra would be key generation | 18:54 |
| clarkb | The rtt for those jobs is a bit long so working through it first should hopefully limit the number of cycles too | 18:55 |
| opendevreview | Jeremy Stanley proposed opendev/infra-manual master: Update recommended ACL for createSignedTag keyword https://review.opendev.org/c/opendev/infra-manual/+/826358 | 18:57 |
| opendevreview | Merged openstack/project-config master: Work around signed tag regression from Gerrit 3.4 https://review.opendev.org/c/openstack/project-config/+/826334 | 18:58 |
| clarkb | any idea why zuul didn't gate https://review.opendev.org/c/opendev/system-config/+/826156 ? | 18:59 |
| clarkb | oh I bet it reported back when gerrit was down because i pushed that before the downtime | 19:00 |
| fungi | i'll keep tabs on the 826334 deployment | 19:00 |
| clarkb | ya that makes sense | 19:00 |
| clarkb | fungi: thanks. I need to go run the meeting now | 19:00 |
| fungi | Build succeeded (deploy pipeline). | 19:24 |
| fungi | hooray! | 19:24 |
| fungi | i'll do some spot-checks from near the end of the list | 19:24 |
| clarkb | ++ | 19:25 |
| fungi | yeah, projects like zuul/zuul have it now | 19:26 |
| fungi | so i'll send out the announcement | 19:27 |
| clarkb | ++ | 19:27 |
| fungi | okay, announcement is out | 19:39 |
| clarkb | Email sent to repo discuss | 19:46 |
| clarkb | https://groups.google.com/g/repo-discuss/c/9SXeHpnnXrw/m/KiXzl1S7AAAJ | 19:47 |
| clarkb | fungi: any idea why git tag -s woudl fail on the test machine after I did a gpg --full-generate-key and made a dummy test key? | 19:55 |
| clarkb | there is a gpg agent running too | 19:55 |
| clarkb | oh I think maybe it is looking for a key for a specific name and email | 19:58 |
| clarkb | and I didn't generate my key that way | 19:58 |
| clarkb | I need to eat lunch and will look into this after | 19:58 |
| fungi | yeah, see git-tag(1) manpage: "By default, git tag in sign-with-default mode (-s) will use your committer identity (of the form Your Name <your@email.address>) to find a key." | 20:02 |
| fungi | there are cli and config options to work around that, whatever's most convenient for you | 20:02 |
| clarkb | I found GIT_TRACE=1 which gives me the output of the gpg command. Previously it was failing beacuse the identity was wrong. I set -u on git tag to the key id and I still get the git tag failure. Running the gpg command output when setting -u doesn't error though | 20:16 |
| clarkb | it just waits for data I think | 20:16 |
| clarkb | I expected a prompt to give my passphrase though and I'm not getting that | 20:17 |
| clarkb | let me try killing the gpg-agent | 20:17 |
| clarkb | ya I'm completely lost I have no idea what is going on now. I overrode the gpg-agent pinentry program to the curses one because I'm not getting prompted at all for a passphrase | 20:27 |
| clarkb | But that doesn't help anything | 20:27 |
| clarkb | fungi: any cahnce you can take a look? 158.69.73.149:/root/.gnupg and /root/test-tagging/test-project is where I'm trying to tag a commit (the initial one that has .gitreview in it) | 20:30 |
| clarkb | Once tagged my plan was to push it to gerrit with the acls removed to see gerrit say I need the createSignedTag acl and then add acl back in and have it succeed hopefully. But I can't figure out tagging at all | 20:30 |
| fungi | clarkb: i do this: env DISPLAY="" GPG_TTY=$(tty) git tag -s ... | 20:33 |
| clarkb | wow that worked. Why is this so impossible to work out of the box | 20:34 |
| clarkb | thanks | 20:34 |
| fungi | gnupg wants desperately to not trust your terminal unless it has no other choice | 20:34 |
| clarkb | I mean it seems to have completely failed here | 20:35 |
| fungi | yeah, it may have thought it was x-hosting pinentry-gtk2 to your desktop or something | 20:35 |
| fungi | absurd, i know | 20:35 |
| clarkb | ok after removing all the config to allow tag pushing I get ! [remote rejected] 0.0.1 -> 0.0.1 (prohibited by Gerrit: not permitted: create signed tag) | 20:39 |
| clarkb | Now I will just add createSignedTag back and see if it works | 20:39 |
| clarkb | * [new tag] 0.0.1 -> 0.0.1 | 20:40 |
| clarkb | I think that means my fix works | 20:40 |
| fungi | excellent! | 20:41 |
| clarkb | I've posted those results to the change upstream. I guess now we wait :/ | 20:44 |
| clarkb | I'll leave the instance held in case anyone else wants to test or if upstream asks for more info | 20:46 |
| clarkb | fungi: were you going to followup on the openstack-discuss thread on this too ? | 20:48 |
| clarkb | I guess I should try testing pushing of a normal tag | 20:49 |
| clarkb | and make sure it doesn't get detected as a signed tag? | 20:49 |
| clarkb | ! [remote rejected] 0.0.2 -> 0.0.2 (prohibited by Gerrit: not permitted: create) seems to work | 20:50 |
| fungi | oh, right following up now | 20:54 |
| clarkb | now to review the removal of createTag change for when this is working upstream again | 20:56 |
| clarkb | fungi: the followup change lgtm. I guess we really are just waiting for upstream now | 21:11 |
| fungi | yep | 21:18 |
| fungi | also there was some movement on our other gerrit bug about the gitweb config | 21:19 |
| fungi | https://bugs.chromium.org/p/gerrit/issues/detail?id=15589 | 21:19 |
| *** dviroel is now known as dviroel|afk | 21:38 | |
| *** rlandy|ruck is now known as rlandy|out | 23:48 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
