| ianw | i seem to have stuffed up the indenting acl change https://review.opendev.org/c/openstack/project-config/+/879906/3 | 00:51 |
|---|---|---|
| ianw | somehow i've pushed the other two patches ontop of an old version of it | 00:52 |
| opendevreview | Ian Wienand proposed openstack/project-config master: gerrit/acls : add NO_CODE_CHANGE https://review.opendev.org/c/openstack/project-config/+/880115 | 00:56 |
| opendevreview | Ian Wienand proposed openstack/project-config master: acl : remove NO_CODE_CHANGE from Allow-Post-Review https://review.opendev.org/c/openstack/project-config/+/880792 | 00:56 |
| opendevreview | Ian Wienand proposed openstack/project-config master: Indent Gerrit ACL options https://review.opendev.org/c/openstack/project-config/+/879906 | 00:56 |
| ianw | i will admit it's impossible to know what normalization rules apply ATM without just reading the source | 00:59 |
| opendevreview | Ian Wienand proposed openstack/project-config master: tools/normalize_acl.py: Add some human readable output https://review.opendev.org/c/openstack/project-config/+/880898 | 01:50 |
| opendevreview | Merged openstack/project-config master: gerrit/acls : add NO_CODE_CHANGE https://review.opendev.org/c/openstack/project-config/+/880115 | 02:00 |
| opendevreview | Merged openstack/project-config master: acl : remove NO_CODE_CHANGE from Allow-Post-Review https://review.opendev.org/c/openstack/project-config/+/880792 | 02:01 |
| opendevreview | Merged openstack/diskimage-builder master: Allow custom console=tty0 argument https://review.opendev.org/c/openstack/diskimage-builder/+/880008 | 04:27 |
| opendevreview | Ian Wienand proposed openstack/project-config master: tools/normalize_acl.py: Add some human readable output https://review.opendev.org/c/openstack/project-config/+/880898 | 04:32 |
| ianw | clarkb/fungi: i'm willing to manually apply the tab indent change https://review.opendev.org/c/openstack/project-config/+/879906 if you think ^ is enough to avert the problem of increasing complexity in the normalisation tool | 04:35 |
| opendevreview | Ching Kuo proposed opendev/system-config master: Update Hound to Use Python 3.11 Base Images https://review.opendev.org/c/opendev/system-config/+/880908 | 06:29 |
| opendevreview | Ian Wienand proposed opendev/zone-zuul-ci.org master: Add Jammy refresh NS records https://review.opendev.org/c/opendev/zone-zuul-ci.org/+/880909 | 06:30 |
| opendevreview | Ian Wienand proposed opendev/zone-zuul-ci.org master: Remove old nameservers https://review.opendev.org/c/opendev/zone-zuul-ci.org/+/880910 | 06:30 |
| opendevreview | Ian Wienand proposed opendev/zone-opendev.org master: Add Jammy refresh NS records https://review.opendev.org/c/opendev/zone-opendev.org/+/880577 | 06:38 |
| opendevreview | Ian Wienand proposed opendev/zone-opendev.org master: Remove old nameservers https://review.opendev.org/c/opendev/zone-opendev.org/+/880709 | 06:38 |
| genekuo | ianw: clarkb: Will you recommending update all containers to python 3.10 or later in one patch review or do it in multiple patches? I'm currently doing it in multiple patches. | 07:01 |
| ianw | ns04.opendev.org has address 162.253.55.23 | 07:02 |
| ianw | ns04.opendev.org has IPv6 address 2604:e100:1:0:f816:3eff:feff:bd1c | 07:02 |
| ianw | mnaser: ^ is there any chance we can get these ip's updated for reverse dns? | 07:03 |
| ianw | ns04 is a jammy server in vexxhost to replace the existing ns2 running there | 07:04 |
| ianw | genekuo: i would say one thing in one change. with gerrit and zuul we've very much about stacking smaller changes ontop of each other. it will all be combined and tested correctly | 07:05 |
| ianw | genekuo: i'm looking at the screenshots from the job you can see at https://zuul.opendev.org/t/openstack/build/074c1a1c9ae845988cbca3c891662764/artifacts | 07:09 |
| ianw | it does seem there's something up with the overlay | 07:10 |
| ianw | my next thought is to look at the build history for that job @ https://zuul.opendev.org/t/openstack/builds?job_name=system-config-run-codesearch&project=opendev/system-config | 07:10 |
| ianw | https://zuul.opendev.org/t/openstack/build/a5552fc4b8b045a9b0a244b5b48a3028/artifacts is a prior run of this job. looking at the screenshots there, it doesn't seem like it's got this overlay thing happening | 07:11 |
| ianw | which makes me a bit worried it's related to this change | 07:11 |
| ianw | which is weird because this is a go app. but i wonder if rebuilding it something has changed upstream? | 07:12 |
| genekuo | Is it possible to trigger a rebuild without my changes (updating python) and see the result? | 07:12 |
| genekuo | In this case we can see if there's something changed with the go app or my change actually cause the overlay issue. | 07:13 |
| ianw | yes, we often will push changes like this with a [dnm] prefix on the subject (do not merge) -- so do something like make a new branch and just update a comment in the dockerfile to get it to rebuild | 07:16 |
| opendevreview | Ching Kuo proposed opendev/system-config master: [dnm] Test Hound Build https://review.opendev.org/c/opendev/system-config/+/880912 | 07:18 |
| *** amoralej|off is now known as amoralej | 07:19 | |
| ianw | yep, just like that :) the dnm just flags it for us to know not to bother reviewing etc, it doesn't mean anything to zuul/gerrit | 07:19 |
| genekuo | Thanks for the tip | 07:21 |
| ianw | for this, we use the python base image to generate the hound config file with jeepyb IIRC | 07:22 |
| *** jpena|off is now known as jpena | 07:23 | |
| ianw | jeepyb (the jee is suppoed to sound like "g") helps manage gerrit and projects | 07:23 |
| ianw | https://opendev.org/opendev/jeepyb/src/branch/master/jeepyb/cmd/create_hound_config.py | 07:24 |
| ianw | hound itself is a go app, again, iirc. so the python update seems unlikely to be the cause of this really | 07:24 |
| ianw | the other thing it could be is selenium, which we use to take the screenshots. something going wrong there ... | 07:25 |
| ianw | ... if you find digging into this sort of thing interesting, then you're in the right place :) | 07:26 |
| ianw | selenium gets started by this playbook -> https://opendev.org/opendev/system-config/src/branch/master/playbooks/test-codesearch.yaml | 07:28 |
| ianw | which runs it from this role -> https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/run-selenium/tasks/main.yaml | 07:29 |
| ianw | and the screenshosts are taken in https://opendev.org/opendev/system-config/src/branch/master/testinfra/test_codesearch.py | 07:30 |
| genekuo | https://opendev.org/opendev/system-config/src/branch/master/testinfra/util.py | 07:31 |
| ianw | yep that's the implementation | 07:32 |
| ianw | so if they just pushed a new selenium container or something, that might be it... | 07:33 |
| genekuo | last push 16 days ago | 07:33 |
| ianw | https://hub.docker.com/r/selenium/standalone-firefox/ | 07:34 |
| genekuo | the last job is only 7 days | 07:34 |
| ianw | so one way to narrow it down would be to use an older tag for the selenium container in the DNM job | 07:35 |
| ianw | (if other things are making us think it is a generic selenium issue) | 07:35 |
| ianw | https://github.com/hound-search/hound/commit/d25b221872426b03d9be8cf6924327e5eab6c314 is quite suspicious | 07:37 |
| ianw | change Advanced options div ID | 07:37 |
| ianw | ps. please never write one line subject-only commit messages with no context :) | 07:38 |
| ianw | the thing that looks incorrectly overlayed is the advanced options | 07:38 |
| genekuo | from the commit I think it's only a name change for the tag | 07:38 |
| ianw | agree it seems unlikely. and also, we would have had this change in some of the prior runs, that look ok | 07:40 |
| ianw | the selenium logs from the last run, that looked good, are https://f314b8cc5a409254dc80-c21ba3df839786eb7baa2d6ac30080ab.ssl.cf5.rackcdn.com/876936/3/gate/system-config-run-codesearch/a5552fc/codesearch01.opendev.org/docker/musing_booth.txt | 07:42 |
| ianw | the "bad" log is https://a11e83f7bdd164f6a645-6d8ce6b05ca968f8ff87eab871edfddd.ssl.cf1.rackcdn.com/880908/1/check/system-config-run-codesearch/074c1a1/codesearch01.opendev.org/docker/dazzling_rubin.txt | 07:45 |
| ianw | they both say "Started Selenium Standalone 4.8.3 (revision b19b418e60)" | 07:46 |
| ianw | so that actually makes me think not much as changed there ... | 07:46 |
| ianw | the selenium container is running on the test codesearch node -- to see that log you need to go to "logs -> codesearch01 -> docker" | 07:47 |
| genekuo | I did a quick diff of both logs, I don't find any major difference besides timestamps and some IDs | 07:48 |
| ianw | clarkb/fungi: https://etherpad.opendev.org/p/2023-opendev-dns i think that is pretty much my final plan for the dns upgrades. a few ??? over who control some of the domains. other than that, i think we can get ns03/ns04 deployed, and do the domain changes after that | 07:49 |
| ianw | genekuo: if we can't figure it out from probing like this, we can make the job fail in the dnm test (just put a fake assert in testinfra) and tell zuul to hold the failed node, and we can log in and probe it, etc. | 07:51 |
| ianw | holding nodes on job failure is something an admin needs to setup, however. | 07:51 |
| genekuo | I see, let's wait for the dnm one build and see if the issue is there. | 07:52 |
| ianw | it's a bit of a race, but you can also look at the console output of the running job (currently https://zuul.opendev.org/t/openstack/stream/c75b44555a8c451da8636ad805e491cd?logfile=console.log) | 07:53 |
| ianw | that will give you the ip of the nodes being tested | 07:53 |
| ianw | in this case, 104.130.74.125 for codesearch01. there are a few minutes where this has deployed and is running testinfra, etc. where the host is up and you can try to connect | 07:54 |
| ianw | once the job finishes, zuul is going to destroy the node, but sometimes it's enough time to get an idea of what's going on | 07:54 |
| ianw | i'm afraid i've got to head off, but feel free to keep sending things through on that dnm change and poking and prodding. i know it's a little slow way to iterate, but gives the best view of what's going on | 07:57 |
| genekuo | no problem, let me try digging through by myself | 07:59 |
| ianw | thanks! appreciate you helping out! :) | 08:00 |
| genekuo | overlay still exist in the dnm build, so likely not related to python version update | 08:13 |
| opendevreview | Artem Goncharov proposed openstack/project-config master: Add sdk and cli xxx-service-core group https://review.opendev.org/c/openstack/project-config/+/880933 | 09:55 |
| opendevreview | Lucas Alvares Gomes proposed openstack/project-config master: Stop using Storyboard for ovn-bgp-agent https://review.opendev.org/c/openstack/project-config/+/880938 | 10:48 |
| *** amoralej is now known as amoralej|lunch | 11:01 | |
| opendevreview | Merged openstack/project-config master: Add sdk and cli xxx-service-core group https://review.opendev.org/c/openstack/project-config/+/880933 | 11:45 |
| gtema | dear infra, would you please add me as a member into following newly created gerrit groups: openstacksdk-service-core (33c03b5e000a75ffc476c51d484c5111648d399c) and | 12:15 |
| gtema | python-openstackclient-service-core (a5d3fd5d65aea9bb530989920f8a5664d5808cbc). | 12:15 |
| *** amoralej|lunch is now known as amoralej | 12:18 | |
| opendevreview | Lucas Alvares Gomes proposed openstack/project-config master: Stop using Storyboard for ovn-bgp-agent https://review.opendev.org/c/openstack/project-config/+/880938 | 12:24 |
| ykarel | Hi is there some issue with vexxhost mirrors, seeing Could not connect to mirror.ca-ymq-1.vexxhost.opendev.org:443 (2604:e100:1:0:f816:3eff:fe0c:e2c0). - connect (113: No route to host) Could not connect to mirror.ca-ymq-1.vexxhost.opendev.org:443 (199.204.45.149). - connect (113: No route to host) | 12:32 |
| ykarel | ex: https://89c01c9b8b932a721cbb-0a6eba071a2c1d142bff6fad0f66c65e.ssl.cf1.rackcdn.com/880461/6/check/neutron-tempest-plugin-openvswitch/22ded7e/job-output.txt | 12:32 |
| ykarel | happens randomly from quite some time | 12:35 |
| ykarel | mnaser, may be you know ^? | 12:36 |
| frickler | gtema: done | 12:50 |
| gtema | thanks a lot frickler | 12:51 |
| fungi | ykarel: might be worth looking at the routes we recorded for the node too, maybe something is shadowing the normal default route there (e.g. stray v6 router advertisements on the network) | 13:10 |
| fungi | so checking https://zuul.opendev.org/t/openstack/build/22ded7e2e1d244fbaa2173181fb27305/log/zuul-info/zuul-info.controller.txt#57-58 against the default routes in other successful builds in that region | 13:13 |
| fungi | skimming the syslog from that build, i don't see any indication of route changes during the short lifetime of that node: https://zuul.opendev.org/t/openstack/build/22ded7e2e1d244fbaa2173181fb27305/log/controller/logs/syslog.txt | 13:15 |
| ykarel | fungi, hmm it good looks there | 14:04 |
| fungi | are those the normal default gateways in that network then? | 14:05 |
| fungi | not some other random test node that leaked an ra out into the lan and got picked up by that node? | 14:05 |
| frickler | infra-root: seems gerrit is doing some weird handling of the ptl-approved label here https://review.opendev.org/c/openstack/releases/+/878864 | 14:05 |
| frickler | why is that condition not marked as satisfied? | 14:06 |
| ykarel | fungi, yes i see same in both good and bad case 2604:e100:1::/64 dev ens3 proto kernel metric 256 | 14:06 |
| ykarel | also with opensearch i see all the failures are on 4 hosts | 14:07 |
| fungi | frickler: i guess "-label:PTL-Approved=MIN" isn't doing what we expect? looks like it wants all reviewers to not have the minimum vote (which is 0) and two of them have a 0 vote | 14:07 |
| fungi | i guess we should have gone with max after all | 14:08 |
| frickler | fungi: that's what I thought at first, too, but compare to https://review.opendev.org/c/openstack/releases/+/878891 | 14:08 |
| fungi | huh, that one says it requires "label:Verified=MAX AND -label:Verified=MIN" | 14:09 |
| fungi | no, sorry, wrong condition | 14:09 |
| fungi | yeah, still " | 14:09 |
| fungi | -label:PTL-Approved=MIN" but there is indeed someone with a 0 vote | 14:10 |
| fungi | and yet it considered it satisfied | 14:10 |
| frickler | so maybe zuul's "PTL-Approved 0 (vote reset)" is different from not voting on that label at all? | 14:10 |
| fungi | frickler: i think the difference is that elodilles had added ptl-approved +1 and then removed it | 14:11 |
| fungi | yes, exactly, so it's seeing an explicit MIN vote rather than an implicit one | 14:11 |
| frickler | but one cannot tell the difference in the UI except by tracking the log, I guess | 14:12 |
| frickler | also zuul set the 0 after elodilles review, the PTL +1 is still there, so that looks like an issue with the zuul check, too | 14:12 |
| fungi | elodilles: commented explaining it as "note that Rafael's email address is different in gerrit and in governance projects.yaml, that's why it is not set" | 14:14 |
| fungi | maybe there was an address change? | 14:14 |
| elodilles | what i thought is that check-release-approval sets 0, and thus my PTL-Approved+1 is disregarded | 14:14 |
| elodilles | yes, probably, but that does not explain why my PTL-Approved+1 is disregarded | 14:15 |
| fungi | looks like the ptl's +1 happened on march 29, then the change sat untouched until a comment on april 6, so possibly the ptl's address changed sometime in the week between those events | 14:16 |
| fungi | elodilles: i think we're seeing a vote reset to 0 counting as a MIN vote, and the condition says to block when that's the case | 14:17 |
| fungi | so we probably have to go with MAX instead, and give up on the future idea of having zuul and release managers leave different vote ranges (+1 vs +2) | 14:18 |
| ykarel | fungi, added good/bad nodes https://paste.opendev.org/show/bduUQS4SuN50neyEgouP/ | 14:20 |
| frickler | "gerrit query 878864 --all-approvals" only shows the +1 for P-A, though | 14:20 |
| fungi | i agree. i wonder if this is gerrit inconsistently sometimes treating 0 as an implicit non-vote and sometimes as an explicit MIN vote | 14:22 |
| clarkb | vote reset to 0 isn't really a reset iirc. The default state is there are no votes and that is treated as 0 in the UI but really no votes have been recorded. When you vote 0 explicitly after say +/-1 that is an actual 0 vote recorded | 14:28 |
| frickler | ptl email seems indeed different in governance vs. gerrit | 14:31 |
| elodilles | clarkb: so this means, because check-release-approval once have set +1, and then reset it to 0, it remains there forever, even if we set PTL-Approved+1 by ourselves | 14:34 |
| frickler | maybe you can abort/restore or push a new PS to reset | 14:34 |
| clarkb | elodilles: I think whoever voted 0 needs to vote +1 that supercedes the zero | 14:35 |
| clarkb | infra-root I think I understand the issue causing errors with replication and it is the same issue causing us to leak edit- ref replication tasks | 14:36 |
| frickler | for zuul to vote +1 again I think the PTL would need to fix the email mismatch and then resubmit their CR+1 | 14:37 |
| elodilles | clarkb: which is in this case was the check-release-approval job (though it does not seem to appear in 'votes' but leaves the status as 'UNSETISFIED') | 14:38 |
| elodilles | frickler: actually 'fix the emai' is enough then any comment / review will trigger the check-release-approval job and will set it to +1 | 14:39 |
| frickler | ah, right. but that's likely the easier part anyway ;) | 14:45 |
| elodilles | yes :) | 14:48 |
| fungi | but current situation aside, i think using a not-min condition with a range where the min value is 0 isn't going to work quite as intended | 14:49 |
| clarkb | yes it will be confusing at the very least | 15:04 |
| fungi | the original idea was that we might give zuul 0..+2 permission and release managers 0..+1 and simply require at least one positive vote to merge, but it looks like this is not the way to express it (if there even is a way) | 15:05 |
| fungi | basically wanting to make it easy to tell a release manager override from a zuul vote at a glance, but i expect querying based on user instead is going to need to be enough | 15:07 |
| frickler | could one have "label:Verified>MIN" or is there only =/-= ? | 15:07 |
| clarkb | fungi: I'm not sure I understand why you would need +2 zuul or +1 human if zuul can also +1 | 15:09 |
| clarkb | its equivalent to just always +1 in that case | 15:09 |
| fungi | humans can't control zuul, so limiting release manager voting range to +1 means that a release manager's vote can't masquerade as zuul's vote | 15:10 |
| frickler | the idea was to be able to distinguish a "real" PTL-approved from an "release manager override" approval | 15:11 |
| fungi | they were looking for a way to easily identify what changes had been merged with a release manager override on that label and hoped they could do it by differentiating the vote number | 15:11 |
| clarkb | isn't the vote supplier the way you do that? | 15:11 |
| fungi | yes, i think they're going to need to do that by querying for votes left by not-zuul | 15:11 |
| fungi | and just use the same allowed range for both zuul and release managers | 15:12 |
| fungi | and then they can set the requirement to MAX instead of -MIN | 15:12 |
| fungi | also i'm not having any luck figuring out where in the gerrit docs the rule syntax is explained. doesn't seem to be in the access control docs | 15:14 |
| clarkb | its the query syntax | 15:15 |
| clarkb | I think the link to it from teh ACL docs | 15:15 |
| fungi | oh, those submit rules reuse query syntax... why hadn't i noticed that? | 15:16 |
| clarkb | I used that to debug the case issue | 15:16 |
| clarkb | you can just copy the conditions and query on them and see what gerrit thinks. It is nice for debugging | 15:17 |
| fungi | so yeah, seems like PTL-Approved>MIN should work | 15:19 |
| fungi | "Can also be used with the {MAX, MIN, ANY} label votes. All operators >, >=, <, ⇐ are supported. " | 15:20 |
| fungi | https://review.opendev.org/Documentation/user-search.html | 15:20 |
| clarkb | the entire motivation behind submit requirements was to shift from prolog that no one understood to a system that typical gerrit users that understand gerrit query language could implement | 15:21 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: PTL-Approved uses inequality rather than negation https://review.opendev.org/c/openstack/project-config/+/880985 | 15:24 |
| fungi | elodilles: frickler: clarkb: that ^ should in theory behave as intended | 15:24 |
| elodilles | fungi: thanks! looks good to me, though i might not see all the details / side cases how this will behave in our way of working / dashboard | 15:34 |
| fungi | well, yeah, clearly i didn't either since i was oblivious to the current issue | 15:36 |
| clarkb | https://bugs.chromium.org/p/gerrit/issues/detail?id=16867 https://bugs.chromium.org/p/gerrit/issues/detail?id=16868 | 15:56 |
| clarkb | I've asked on gerrit discord if anyone can take a look to tell me if I am on the right track and if so I can work on some patches | 15:57 |
| clarkb | the error spam we get is a subset of the second issue I think. Basically I strongly suspect that what happens is we stick a replication task in waiting/ for a ref or refs. Then the plugin filters all of those refs out of the list of things to replicate due to anonymous users permissions. In that specific case it tries to rename a new file in waiting/ for the hashed content of the | 15:58 |
| clarkb | empty list and tha tfails because it was never created in the first place | 15:58 |
| clarkb | That failure leaks the original file in waiting/ with the refs that get filtered. Then we restart gerrit and get a bunch of errors. | 15:59 |
| clarkb | I think my workaround change to clear out the entries on disk will only address a subset of those cases currently so we won't necessarily prevent all errorswith my workaround at this point | 15:59 |
| clarkb | The other variant of this situation is when only some refs are filtered leaving some refs to be replicated. In that case Ithink we create the new file in waiting orphaning the original but the new one is able to be moved and deleted normally | 16:00 |
| opendevreview | Merged openstack/project-config master: PTL-Approved uses inequality rather than negation https://review.opendev.org/c/openstack/project-config/+/880985 | 16:01 |
| corvus | clarkb: here's the updated container copy script i'm using for zuul: https://paste.opendev.org/show/biWc9zH6N73Lo6cOtAeB/ | 16:09 |
| clarkb | thanks! | 16:09 |
| clarkb | looks like I can edit that to do an image at a time too which will be good for figuring it out | 16:10 |
| corvus | clarkb: oops paste error: https://paste.opendev.org/show/bbEyBhgqn4eGJaipuonZ/ | 16:10 |
| clarkb | noted | 16:10 |
| corvus | i added a change_ filter so we don't copy leaked pre-promote tags | 16:11 |
| corvus | clarkb: also, i just temporarily added my personal account to the 'automation tools' group so i have write perms. will remove at end of process. | 16:11 |
| clarkb | hrm was that necessary? I thought our admin accounts had the bits necessary too but I would have to look at the setup again to refresh my memory | 16:12 |
| *** jpena is now known as jpena|off | 16:17 | |
| clarkb | it just occurred to me that most people using the replication plugin are probably doig replication for gerrit to gerrit replicas and/or HA. In those cases they do want to replicate everything and wouldn't run into these problems. | 16:43 |
| *** amoralej is now known as amoralej|off | 16:56 | |
| clarkb | infra-root it has been ~17 hours since new etherpad02 took over etherpad duties. What is our comfort level for cleaning up etherpad01? SHold I unwip those changes now? | 17:10 |
| clarkb | I've gone ahead and removed my WIP from https://review.opendev.org/c/opendev/system-config/+/880087 and https://review.opendev.org/c/opendev/zone-opendev.org/+/880169 | 17:21 |
| fungi | probably the only thing of value on the old server is my shell history for examples of calling the etherpad api, and if memory serves i eventually documented how to do that anyway | 17:37 |
| fungi | yeah, https://docs.opendev.org/opendev/system-config/latest/etherpad.html#manual-administrative-tasks has relevant examples already | 17:38 |
| clarkb | I was looking at clearing up a held message in service-discuss and hit f5 to refresh the page I had open from yesterday. I love that firefox warns about resending data to do that (since I had allowed a message through yesterday and didn't need to repost that). Also fungi appears to have taken care of the moderation for the newer email already | 17:44 |
| clarkb | long way to say browser features that keep users from doing a silly are great | 17:44 |
| fungi | oh, yep i usually try to check them all at least once a day | 17:46 |
| fungi | gonna disappear for a bit to grab early dinner out since the weather's nice, but will be back in an hour or so | 19:14 |
| clarkb | ya I'm having a lazy lunch here, but the weather is not nice | 19:30 |
| fungi | i'm back, and yeah the weather was just too hard to pass up | 20:31 |
| clarkb | I've just spot checked backups for etherpad02 and they look good. I was able to borg mount and head the mysql file and ls'ing the fs backups seems to show it working there too | 20:39 |
| clarkb | so ya I think I'm ready to land those two chagnes to cleanup etherpad01 whenever others are happy too | 20:39 |
| corvus | quick question on nameserver replacement -- ns03 and ns04 don't seem to be answering queries right now... what's the sequencing? | 20:39 |
| corvus | i ask because i'm reviewing ianw's change at https://review.opendev.org/880906 which adds them to gating.dev | 20:40 |
| corvus | but i think probably they should answer queries before we do that? | 20:40 |
| clarkb | corvus: https://etherpad.opendev.org/p/2023-opendev-dns captures the full plan and where we are at (completed steps are struck through) | 20:44 |
| clarkb | I was just reviewing it again to catch up on the latest updates | 20:45 |
| corvus | okay, so it looks like that change is step 7, and make new servers is step 6, and we're currently at step 4 | 20:46 |
| corvus | so that change should be considered wip until we get past step 6 | 20:46 |
| clarkb | yes | 20:48 |
| corvus | cool thx | 20:50 |
| fungi | okay, the acl update for openstack/releases still doesn't seem to have done what i thought it would... in https://review.opendev.org/878864 you can see that it thinks the "label:PTL-Approved>MIN" submit condition is not satisfied even though there is a +1 vote. does it need every vote to match that condition, not just one or more? | 21:00 |
| fungi | is so, then i don't think setting it to =MAX is going to work either | 21:01 |
| fungi | or do we need to add a count condition? | 21:01 |
| fungi | "label:PTL-Approved>MIN,count>0" | 21:02 |
| fungi | https://review.opendev.org/Documentation/user-search.html#labels seems to indicate that's the syntax at least | 21:02 |
| fungi | no, oddly =MAX seems to match, at least with the search window | 21:04 |
| fungi | https://review.opendev.org/q/change:878864+label:PTL-Approved%253DMAX | 21:04 |
| clarkb | I would use the search queries to figure that out | 21:04 |
| fungi | count>0 throws an error, count>=1 doesn't match for some reason | 21:04 |
| fungi | yeah, i should have tested it that way | 21:04 |
| fungi | "label:PTL-Approved>=1" also seems to work. this is a baffling parser | 21:06 |
| fungi | https://review.opendev.org/q/change:878864+label:PTL-Approved%253E%253D1 | 21:06 |
| fungi | "label:PTL-Approved>0" also works but "label:PTL-Approved>MIN" doesn't even though MIN is 0 for that label | 21:07 |
| fungi | i'll push a change in a few minutes unless someone points out i'm misreading this | 21:08 |
| clarkb | I think your experimenting is probably the best anyone understands it :) | 21:09 |
| fungi | :/ | 21:09 |
| opendevreview | Jeremy Stanley proposed openstack/project-config master: Switch PTL-Approved rule comparison from MIN to 0 https://review.opendev.org/c/openstack/project-config/+/881149 | 21:38 |
| fungi | elodilles: frickler: clarkb: ^ happy to shift approach if someone can actually figure out what's most appropriate | 21:38 |
| fungi | but at least that seems like it works | 21:38 |
| ianw | corvus: sorry for the confusion; i meant to -2 but forgot in the process of making all the changes | 22:11 |
| corvus | ianw: no prob, just making sure we're following the same script :) | 22:12 |
| clarkb | please weigh in on https://review.opendev.org/c/opendev/system-config/+/880087 if you think we're done with etherpad01 at this point :) also the dns record upate to removeetherpad01 recrods will conflict with ianw's dns zone updates for opendev. | 22:23 |
| clarkb | ianw: also you may be interested in the two bugs I filed against the replication plugin if you haven't seen them already (links in scrollback somewhere) | 22:24 |
| ianw | clarkb: have we double checked the backups for etherpad02? | 22:24 |
| clarkb | ianw: I borg mounted and ran head against the db backup and checked the fs backups using ls. They looked good to me. But having a second person check is a good idea | 22:25 |
| ianw | excellent, i didn't mount but double-checked hte logs and the remote side, lgtm | 22:28 |
| ianw | fungi there is a specific endpoint for testing -> https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#check-submit-requirement | 22:39 |
| corvus | infra-root: fyi we just merged a change to nodepool which will increase our log chattiness, but i think it's worth it. just fyi. | 22:39 |
| corvus | https://review.opendev.org/880675 change in question | 22:40 |
| ianw | {"name":"Code-Review","status":"UNSATISFIED","is_legacy":false,"submittability_expression_result":{"expression":"label:PTL-Approved\u003eMIN","fulfilled":false,"status":"FAIL","passing_atoms":[],"failing_atoms":["label:PTL-Approved\u003eMIN"]}} | 22:56 |
| ianw | curl -u user:pass -X POST -H 'Content-Type: application/json; charset=UTF-8' 'https://review.opendev.org/a/changes/openstack%2Freleases~master~I318e3abcb5aa4eae087a2e1f93d321734a75d796/check.submit_requirement' -d '{"name": "Code-Review", "submittability_expression": "label:PTL-Approved>MIN"}' | 22:59 |
| ianw | is the call i made. i do not know what that had to be authenticated, it doesn't mention it in the docs | 22:59 |
| fungi | neat. and >0 still works where >MIN does not | 23:00 |
| ianw | right, >0 makes it PASS | 23:01 |
| ianw | i guess the escaped unicode is a valid json thing | 23:01 |
| ianw | but surely this is a bug? | 23:02 |
| ianw | i guess https://gerrit-review.googlesource.com/Documentation/user-search.html does *explicitly* list >, < for MAX/MIN/ANY | 23:09 |
| ianw | does NOT i mean | 23:09 |
| fungi | huh, interesting. i must surely be misreading "Can also be used with the {MAX, MIN, ANY} label votes. All operators >, >=, <, ⇐ are supported." | 23:17 |
| clarkb | https://blog.pypi.org/posts/2023-04-20-introducing-trusted-publishers/ not sure how I feel about this | 23:18 |
| fungi | i guess those sentences being adjacent isn't intended to convey the meaning i took from their proximity to one another | 23:18 |
| fungi | clarkb: probably worth someone asking how well supported that is beyond github, or whether it's designed to create a special github ruling caste | 23:21 |
| clarkb | fungi: the hacker news thread alredy has a couple people calling that out | 23:21 |
| clarkb | https://docs.pypi.org/trusted-publishers/adding-a-publisher/ does make it look like only github is currently supported | 23:22 |
| clarkb | I've got a family dinner tonight in not too long but I'll plan to approve https://review.opendev.org/c/opendev/system-config/+/880087 tomorrow morning | 23:24 |
| fungi | unfortunately the cpython folk have grown substantial github tunnel vision since moving their code hosting and then their bug tracking, ci, cla management, governance, et cetera to be wholly dependent on it | 23:25 |
| fungi | they're also using some new github feature that provides virtual development systems so that their core devs no longer need local dev environments | 23:26 |
| opendevreview | Merged openstack/project-config master: Switch PTL-Approved rule comparison from MIN to 0 https://review.opendev.org/c/openstack/project-config/+/881149 | 23:28 |
| fungi | but yeah, i suppose the irony of calling that "trusted publishing" is sort of lost on them, since they do implicitly trust github over their own authentication systems | 23:29 |
| ianw | clarkb: i'd probably like to rebase the ns changes on it, so happy to merge and supervise 880087 if you like? | 23:31 |
| clarkb | ianw: works for me | 23:31 |
| clarkb | I just have to pop out soon for family time | 23:31 |
| ianw | sounds good to me, i'd like to try and get the new nameservers responding today, since we seem ok with the plan | 23:35 |
| opendevreview | Merged opendev/system-config master: Remove etherpad01 from inventory https://review.opendev.org/c/opendev/system-config/+/880087 | 23:44 |
| clarkb | yup the plan lgtm I left a couple more notes on the registrar and glue record stuff | 23:47 |
| clarkb | basically I don't think glue records hurt | 23:47 |
| ianw | i think the only thing is it means you can't replace the nameservers without updating those glue records | 23:52 |
| ianw | but i mean, that's why they're called glue records, heh :) | 23:52 |
| ianw | in the situation where we're rolling to new hosts with new ip's, it's much of a muchness | 23:53 |
| ianw | https://gerrit-review.googlesource.com/c/gerrit/+/370154 is another change to clarify the search situation. even if nobody reviews it at least it's there | 23:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!