| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Move grubenv to EFI dir https://review.opendev.org/c/openstack/diskimage-builder/+/804000 | 01:19 |
|---|---|---|
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Support grubby and the Bootloader Spec https://review.opendev.org/c/openstack/diskimage-builder/+/804002 | 01:19 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: RHEL/Centos 9 does not have package grub2-efi-x64-modules https://review.opendev.org/c/openstack/diskimage-builder/+/804816 | 01:19 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Add policycoreutils package mappings for RHEL/Centos 9 https://review.opendev.org/c/openstack/diskimage-builder/+/804817 | 01:19 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Add reinstall flag to install-packages, use it in bootloader https://review.opendev.org/c/openstack/diskimage-builder/+/804818 | 01:19 |
| opendevreview | Steve Baker proposed openstack/diskimage-builder master: Add DIB_YUM_REPO_PACKAGE as an alternative to DIB_YUM_REPO_CONF https://review.opendev.org/c/openstack/diskimage-builder/+/804819 | 01:19 |
| *** ysandeep|away is now known as ysandeep | 05:29 | |
| *** rpittau|afk is now known as rpittau | 07:38 | |
| *** jpena|off is now known as jpena | 07:38 | |
| *** ysandeep is now known as ysandeep|lunch | 07:45 | |
| ykarel | Hi jobs are failing while installing boto3 in ci jobs | 08:43 |
| ykarel | ERROR: Cannot install cinder==19.0.0.0b2.dev39 because these package versions have conflicting dependencies. | 08:43 |
| ykarel | example log https://955f32f8268e5d475e65-6c8f4c6e546a0854b4c11cc7c78829ca.ssl.cf5.rackcdn.com/802643/5/check/ironic-tempest-bfv/1ec156a/job-output.txt | 08:43 |
| ykarel | on checking found that issue is when using infra mirrors | 08:44 |
| ykarel | pip install --index-url=https://mirror.mtl01.inap.opendev.org/pypi/simple -c https://raw.githubusercontent.com/openstack/requirements/master/upper-constraints.txt boto3 | 08:44 |
| ykarel | with mirror it fails, and without mirrors it works fine, can someone check how it can resolved | 08:44 |
| *** ysandeep|lunch is now known as ysandeep | 08:53 | |
| frickler | ykarel: for me this is working fine now when testing your command locally. maybe we had cached an index? seems the pkg is pretty new. I can't really spot the exact cause for the failure in the job log, either. please double-check whether you still see the issue | 09:27 |
| opendevreview | Sorin Sbârnea proposed zuul/zuul-jobs master: Include podman installation with molecule https://review.opendev.org/c/zuul/zuul-jobs/+/803471 | 11:08 |
| ykarel | frickler, yes working fine now, but atleast some issue was there | 11:10 |
| ykarel | not sure it was cache or something else | 11:11 |
| ykarel | when the issue was there, it was fetching 1.18.1 not even versions release after that and before 1.18.24 | 11:11 |
| *** dviroel|ruck|out is now known as dviroel|ruck | 11:26 | |
| frickler | ykarel: I don't doubt that there was an issue, it's just that when it's gone now, it is difficult to debug it further | 11:30 |
| *** bhagyashris_ is now known as bhagyashris | 11:31 | |
| ykarel | frickler, ack let's see if it get's reproduced | 11:31 |
| frickler | ykarel: sure, if you see anything similar again, please mention it here using the keyword "infra-root" to ensure we can have a look as soon as possible | 11:34 |
| ykarel | sure will do | 11:34 |
| *** jpena is now known as jpena|lunch | 11:36 | |
| fungi | ykarel: keep in mind that pip's new dep solver will try earlier and earlier versions if it can't find a suitable wheel and then also fails to build something from sdist. what python version were you using, and do you have a log of the problem pip install attempt? | 11:48 |
| ykarel | fungi, https://955f32f8268e5d475e65-6c8f4c6e546a0854b4c11cc7c78829ca.ssl.cf5.rackcdn.com/802643/5/check/ironic-tempest-bfv/1ec156a/job-output.txt | 11:49 |
| fungi | ahh, you linked to an example log, let me try to figure out what the build result page was for that (the raw job logs are a pain to look through) | 11:49 |
| fungi | okay, so that was https://zuul.opendev.org/t/openstack/build/1ec156a09a9e43c2bafe0bc7a05212f4 | 11:51 |
| fungi | ykarel: unfortunately that job doesn't collect pip's debugging logs like we do in tox-based jobs, so i can't really tell what failed to get it to that point | 11:55 |
| fungi | it doesn't seem to say why it refused to use the constrained version | 11:56 |
| ykarel | it was not found that's why? | 11:57 |
| ykarel | actually when it was reproduced locally, the latest version available was 1.18.1 | 11:58 |
| fungi | one distinct possibility is the fastly cdn endpoint near inap-mtl was returning old copies of the pypi index, we've seen this occur in the past, though the only way we've managed to track it down is to excavate those copies from the apache cache and map them to retrieval times | 12:04 |
| fungi | for some reason it seems to hit most often in the montreal canada area providers | 12:04 |
| fungi | when it does happen, it's like 1 in 20 times when our proxy requests the page, the cdn returns very old instead of current content, which makes the problem seem to come and go at random | 12:06 |
| fungi | so at least in those cases the problem really is pypi (or their cdn anyway) but you'd only see it fail randomly if you were around that region of the world and repeatedly cleared your pip cache | 12:08 |
| fungi | if the problem does resurface, we can try to find proof that pypi is sometimes returning old index copies in some places, and have them try to flush those from their cdn network | 12:10 |
| ykarel | ack will post here if see this again | 12:23 |
| opendevreview | Merged opendev/elastic-recheck rdo: Make elastic recheck compatible with rdo elasticsearch https://review.opendev.org/c/opendev/elastic-recheck/+/803897 | 12:24 |
| ykarel | i should have checked other providers at same time | 12:25 |
| ykarel | but yes all the errors i seen were in inap-mtl01 | 12:29 |
| *** jpena|lunch is now known as jpena | 12:36 | |
| mordred | TIL that git-review is in homebrew for the mac users: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/git-review.rb ... and the formula for it totally looks legit | 12:57 |
| fungi | and looks like it does work, judging from the gerritbot message in #zuul | 13:03 |
| mordred | yup! | 13:05 |
| *** rpittau is now known as rpittau|afk | 13:55 | |
| opendevreview | Jing Li proposed openstack/diskimage-builder master: Add new element rocky https://review.opendev.org/c/openstack/diskimage-builder/+/802902 | 14:09 |
| *** ysandeep is now known as ysandeep|away | 14:31 | |
| opendevreview | Sorin Sbârnea proposed zuul/zuul-jobs master: Include podman installation with molecule https://review.opendev.org/c/zuul/zuul-jobs/+/803471 | 14:59 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Test lists.kc.io on focal https://review.opendev.org/c/opendev/system-config/+/805407 | 15:14 |
| clarkb | fungi: ^ fyi it occurred to me that we also want to check the ansible is happy with focal. | 15:14 |
| fungi | sure | 15:14 |
| clarkb | currently Non Interactive Users has proposal bot, openstack release bot, and usagestats in it | 15:25 |
| clarkb | grepping for that group name in openstack/project-config shows no results so acl use would have to be in all-projects alone? | 15:26 |
| clarkb | priority = batch group Non-Interactive Users says our documentation | 15:29 |
| clarkb | https://gerrit-review.googlesource.com/Documentation/access-control.html#capability_priority | 15:29 |
| clarkb | so the only extra privs they haev is using a different thread pool | 15:30 |
| clarkb | assuming we don't have undocumented use somewhere | 15:30 |
| fungi | makes sense, i think we ended up using a separate group to convey voting privileges for zuul | 15:47 |
| fungi | so it should be safe to include the different ci groups in non-interactive users, i guess | 15:48 |
| clarkb | Yup and I think this is a good reminder that we can tune the thread counts to balance resources between CI systems and interactive users if necessary | 15:49 |
| clarkb | right now the bulk of the interaction is using the interactive pool | 15:50 |
| clarkb | but we could split that (I suspect that the bots overall need more threads than interactive) | 15:50 |
| *** jpena is now known as jpena|off | 15:51 | |
| opendevreview | Sorin Sbârnea proposed zuul/zuul-jobs master: ensure-docker: enable centos-8-stream testing https://review.opendev.org/c/zuul/zuul-jobs/+/805432 | 16:06 |
| opendevreview | Sorin Sbârnea proposed zuul/zuul-jobs master: ensure-podman: enable testing of centos-8-stream https://review.opendev.org/c/zuul/zuul-jobs/+/805433 | 16:09 |
| clarkb | fungi: https://review.opendev.org/c/opendev/system-config/+/805407 passed testing so no anticipated issues with the ansible on a focal listserv | 16:29 |
| fungi | yay! | 16:31 |
| opendevreview | James E. Blair proposed opendev/system-config master: Matrix-eavesdrop: handle notices https://review.opendev.org/c/opendev/system-config/+/805439 | 16:36 |
| corvus | clarkb, fungi, mordred, tristanC: ^ that should get us our missing feature from eavesdrop | 16:36 |
| corvus | (i just tested that locally) | 16:36 |
| clarkb | left a note about thread safety as it came up before. tldr is file io isn't async io native so we don't risk being preempted and that should be safe | 16:41 |
| corvus | clarkb: i'm going to run some errands; hopefully that zuul change will land and i can restart after i get back | 17:04 |
| clarkb | sounds good | 17:05 |
| mordred | for the gerritbox-matrix we're running - it doesn't look like we're doing nick lookup in https://matrix.to/#/#test:opendev.org like the one in https://matrix.to/#/#gerritbot:matrix.org - is there something else we need to configure to make that work? | 17:06 |
| clarkb | tristanC: ^ | 17:08 |
| clarkb | mordred: is it looking up the nick of the person proposing changes? | 17:08 |
| corvus | yes | 17:09 |
| mordred | yeah. if you look in the gerritbot channel, you'll see that some of the lines resolve into links to the matrix user id | 17:09 |
| mordred | corvus is an excellent example case :) | 17:09 |
| corvus | mordred, clarkb: https://review.opendev.org/803396 | 17:09 |
| mordred | ah! | 17:10 |
| mordred | corvus: +A | 17:10 |
| mordred | (I note that you have created the needed token) | 17:11 |
| clarkb | How is it able to do that reliably? I guess it would only have problems if there was a name collision in the room(s) gerritbot is in? | 17:11 |
| mordred | it uses matrix identity - so one can register and publish an email address to be associated with your matrix identity - and then the identity api can look up "who is the matrix user with this published email" | 17:13 |
| clarkb | ah I didn't realize it allowed lookups by email like that | 17:13 |
| mordred | yah - but I believe only a 1:1 lookup in that direction. So - I have your email, what's your matrix id - the inverse doesn't work, I can't ask for all the emails your matrix id may have published | 17:14 |
| mordred | so you can opt-in to publishing your email as an identifying piece of data, or you can choose not to | 17:15 |
| fungi | neat, though i guess it relies on you associating your preferred gerrit address with your matrix id? (you can no longer lookup secondary addresses in gerrit unless you're a full admin) | 17:17 |
| corvus | The zuul matrix docs explain how to do that | 17:17 |
| fungi | cool | 17:18 |
| fungi | i'll make sure i do that | 17:18 |
| corvus | https://zuul-ci.org/docs/zuul/howtos/matrix-id.html | 17:18 |
| corvus | One of the optional appendices | 17:19 |
| fungi | thanks! | 17:19 |
| fungi | this weekend will likely be me getting around to replacing the shell server where i run weechat, since i would like to be on latest available when i try to get the matrix plugin for it working | 17:20 |
| *** mgoddard- is now known as mgoddard | 17:43 | |
| opendevreview | Merged opendev/system-config master: Add gerritbot-matrix identity lookup configuration https://review.opendev.org/c/opendev/system-config/+/803396 | 17:43 |
| opendevreview | Merged opendev/system-config master: Matrix-eavesdrop: handle notices https://review.opendev.org/c/opendev/system-config/+/805439 | 17:52 |
| clarkb | I just noticed that the infra-prod-service-eavesdrop job can run while the promote job for the image is running | 17:54 |
| clarkb | that may result in us not actually updating the image | 17:54 |
| fungi | oh, yep | 17:55 |
| fungi | or, rather, updating it on a delay | 17:56 |
| fungi | we won't get restarted on the new image until the subsequent infra-prod-service-eavesdrop build | 17:56 |
| fungi | eventually consistent, but not ideal | 17:56 |
| fungi | i guess that's input into our deployment dependencies rework | 17:57 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Run service-eavesdrop after promoting the matrix eavesdrop bot https://review.opendev.org/c/opendev/system-config/+/805446 | 17:58 |
| clarkb | fungi: I think ^ will fix it now | 17:58 |
| clarkb | but ya that becomes input to that whole thing | 17:58 |
| clarkb | one of the issues here is taht we sometimes do that in the pipeline and sometimes in the job definition. When we map this out coming up with a good set of rules for how to appraoch this would be good too | 17:59 |
| tristanC | clarkb: it seems like the matrix gerritbot is down, e.g. curl http://eavesdrop01.opendev.org:9001/ fails | 18:01 |
| fungi | tristanC: it may be restarting? | 18:01 |
| tristanC | perhaps its related to 803396 , could i get a copy of the docker logs? | 18:01 |
| fungi | tristanC: gerritbot-matrix: user error (Could not get hash details: MatrixError {meErrcode = "M_TERMS_NOT_SIGNED", meError = "Terms not signed", meRetryAfterMS = Nothing}) | 18:03 |
| clarkb | we need to accept matrix.org's terms and conditions ? | 18:03 |
| tristanC | fungi: thanks, i missed that steps in the instruction | 18:04 |
| fungi | looks like it's been trying in a loop since 17:47 | 18:04 |
| opendevreview | Tristan Cacqueray proposed opendev/system-config master: Add matrix term accept instruction https://review.opendev.org/c/opendev/system-config/+/805447 | 18:07 |
| tristanC | clarkb: yes, we would need to run these extra step ^ corvus has been runnning the one before from the bridge i think | 18:07 |
| tristanC | otherwise we can revert 803396 to restore the service | 18:07 |
| clarkb | previously we have had to accept terms and conditions for our hosted homeserver and its pretty straightforward. I expect that the tos for matrix.org is similar | 18:10 |
| corvus | Yeah I think we should run commands from tristanC . I'm not on a terminal with access now. | 18:12 |
| clarkb | I can do it | 18:12 |
| clarkb | https://matrix.org/legal/identity-server-privacy-notice-1 is the agreement which I'm skimming now | 18:13 |
| clarkb | reading that it almost seems like the bot itself is opting into the lookup | 18:15 |
| clarkb | eg to do lookups you yourself must opt into being look up able | 18:15 |
| clarkb | Our bots contact info should already be fairly public so I don't think that is an issue | 18:16 |
| clarkb | I don't see anything else on there that gives me any concern. | 18:16 |
| clarkb | corvus: fungi ^ you good with me accepting it now? | 18:16 |
| clarkb | oh this might be the biggest issue: the lookups live in their logs for half a year and that would be regardless of whether or not a user has opted into the lookup discovery on their end | 18:23 |
| clarkb | https://github.com/matrix-org/sydent/issues/189 is the linked issue which has been closed with an assertion that rotating logs is sufficient cleanup | 18:24 |
| clarkb | I'm personally ok with that, but want to amek sure others are before continuing | 18:24 |
| clarkb | tristanC: re ^ the bot will only do a lookup if it decides to post the notification to a channel? Will lookups be done for every event from the event stream? | 18:26 |
| tristanC | clarkb: only after the event match a channel | 18:30 |
| clarkb | then ya I'm comfortable with it if other infra-root are | 18:30 |
| tristanC | clarkb: and the lookup is performed with a hash of the email | 18:31 |
| tristanC | the bot is actually failing to start because of an error when getting the pepper used to salt the hashs | 18:33 |
| clarkb | oh that is interesting the terms don't say anything about a hash but that makes sense | 18:35 |
| clarkb | in that case I'm really comfortable with it | 18:35 |
| clarkb | I'll give fungi and corvus another little bit to chime in but I'll go ahead with that if I don't hear back in like 10-15 minutes | 18:35 |
| tristanC | clarkb: i need to go afk for an hour, i'll check again when i come back | 18:36 |
| fungi | yeah, sorry, had to take a break to catch up on a couple of chores but looking now | 18:38 |
| fungi | clarkb: seems fine to me, i'm good with it | 18:39 |
| clarkb | cool corvus already mentioned accepting it above so I'll goa head | 18:41 |
| clarkb | ok done. fungi was the container restarting in a loop? if so we should see it working again automatically? | 18:42 |
| fungi | gerritbot-matrix_1 | 2021-08-20 18:42:40.830 [ThreadId 19]: Connecting to review.opendev.org:29418 | 18:43 |
| fungi | yep | 18:43 |
| fungi | i guess someone could push a dnm change or something | 18:43 |
| fungi | to exercise it | 18:43 |
| clarkb | we can approve https://review.opendev.org/c/opendev/system-config/+/805447 and then it will report the merge | 18:45 |
| clarkb | I've +2'd the chagne as the commands worked for me | 18:45 |
| clarkb | fungi: ^ maybe double check it for bash correctness then +A? | 18:45 |
| fungi | yeah, done | 18:47 |
| opendevreview | Merged opendev/system-config master: Add matrix term accept instruction https://review.opendev.org/c/opendev/system-config/+/805447 | 19:19 |
| fungi | we got the irc report ^ | 19:50 |
| fungi | can someone monitoring matrix confirm it was echoed there by the new bot? | 19:50 |
| clarkb | fungi: yup we did | 19:50 |
| fungi | perfect! | 19:50 |
| fungi | was 805361 the change corvus was waiting to have in a new container before we restart? | 19:51 |
| clarkb | yes | 19:52 |
| clarkb | I think we're ready to restart as soon as corvus confirms he is ready | 19:52 |
| fungi | cool, i didn't see anything else mergeable which was already approved anyway | 19:52 |
| corvus | i am back now | 19:59 |
| corvus | should we go ahead and restart now? | 20:02 |
| fungi | i'm ready | 20:02 |
| fungi | i'll let #openstack-release know we're doing it shortl | 20:02 |
| fungi | y | 20:02 |
| clarkb | I'm here too | 20:02 |
| clarkb | though I think I odn't have keys loaded on this device I can fix that quickly if necessary | 20:02 |
| corvus | fungi: lemme know when you're ready | 20:03 |
| fungi | ready | 20:03 |
| fungi | you're initiating it? i'm on hand to test and help troubleshoot | 20:04 |
| corvus | i pulled images to be sure. and i'm restarting now | 20:04 |
| fungi | thanks | 20:04 |
| corvus | re-enqueueing | 20:14 |
| abhishekk | standard zuul restart? | 20:14 |
| corvus | #status log restarted all of zuul on commit 919c5a36546117c4ad869ff9b580455970ecd268 | 20:14 |
| opendevstatus | corvus: finished logging | 20:14 |
| fungi | abhishekk: yep! | 20:15 |
| fungi | just getting new features/fixes in advance of the next release being tagged | 20:16 |
| abhishekk | fungi, ack, is it daily activity ? | 20:16 |
| corvus | abhishekk: it's as-needed | 20:16 |
| abhishekk | corvus, ack, thank you | 20:17 |
| corvus | i believe we should see the lock cleanups after 60 minutes, so probably worth checking for log entries regarding those at 21:15 | 20:18 |
| fungi | it's been a bit more frequent lately since there's a lot of work getting merged leading up to zuul 5.0.0 and the scale-out (redundant) scheduler capability | 20:18 |
| corvus | "Removing stale lock" is what we'll be looking for | 20:18 |
| corvus | and hopefully not "Error cleaning up locks" | 20:19 |
| fungi | abhishekk: good news is by zuul v5 these scheduler restarts will likely no longer be noticed (at least once we deploy a second scheduler and load balance the dashboard | 20:19 |
| fungi | corvus: noted, i'll check the logs | 20:19 |
| corvus | we could add dedicated web servers now if we want. that might be a good idea to get ahead | 20:20 |
| abhishekk | yep | 20:20 |
| abhishekk | I noticed because, I just pushed couple of patches and suddenly those went out of the queue | 20:21 |
| abhishekk | now those are back in queue so no need to worry | 20:21 |
| corvus | re-enqueue complete | 20:22 |
| corvus | clarkb: did matrix-eavesdrop get restarted? | 20:22 |
| corvus | https://meetings.opendev.org/irclogs/%23test/latest.log.html looks like yes | 20:23 |
| clarkb | corvus: yup it seemed ot be in a fail loop and when the agreement was signed it startedup again | 20:27 |
| corvus | eavesdrop i mean | 20:27 |
| corvus | (not gerritbot) | 20:28 |
| corvus | but it looks like they're both good, because the eavesdrop log includes a notice and the notice includes a matrix id :) | 20:28 |
| clarkb | oh for eavesdrop I wasn't sure due to the issue that https://review.opendev.org/c/opendev/system-config/+/805446 should fix | 20:29 |
| clarkb | the promote and service jobs were running at the same time | 20:29 |
| clarkb | fungi: corvus ^ a quick double check on that change would be good then we can land it to avoid races in the future | 20:35 |
| corvus | clarkb mordred tristanC fungi i'm making the room now -- i'm thinking for this question i should set "Anyone" (but the default is the second option, members since selecting this option). i think Anyone might make it more compatible with room previews and the proposed matrix workflow of incrementally joining a room | 20:35 |
| * corvus uploaded an image: (25KiB) < https://matrix.org/_matrix/media/r0/download/acmegating.com/msHlWptRfQrhvbcxdobSyPam/image.png > | 20:35 | |
| corvus | (i can't remember what they call that right now, but it's the idea that you can "join" a room anonymously, and then if you want to start talking, you can get a nick and make an account) | 20:36 |
| corvus | anyway, what do you think? | 20:36 |
| clarkb | does encryption play into that? I think in our case because we don't need the room to be encrypted the anyone option is fine? | 20:37 |
| corvus | correct, and the room will not be encrypted | 20:37 |
| corvus | (and yes, encryption would alter the available choices there) | 20:37 |
| fungi | sounds like a neat feature, i'm in favor | 20:38 |
| clarkb | probably the only other consideration is how do we prune history should that be necessary. With eavesdrop it is easy (we just edit it). I'm sure matrix has some tooling for this but I don't know what it is | 20:39 |
| clarkb | and in most cases I think we would tell people to reset passwords instead of pretending they weren't disclosed | 20:39 |
| clarkb | basically I can't come up with a reason to not set it to anyone for a public development focused channel | 20:41 |
| fungi | "pruning history" is a fiction with irc too | 20:42 |
| clarkb | yup | 20:42 |
| fungi | redacting things in the published logs didn't remove them from people's own client logs/scrollback | 20:42 |
| corvus | in #test @admin just deleted a message from @corvus | 20:43 |
| corvus | if you want to see what that looks like. pretty easy to do in the web ui | 20:43 |
| clarkb | ah ok so login as admin and select a message to delete, straightforwatd | 20:43 |
| clarkb | stilla good idea to tell epople to not treat that as truly deleted, but we have the ability | 20:43 |
| clarkb | at least as much as we do with irc | 20:44 |
| fungi | yep, general guidance is still "once public, always public" (can't put the cat back in the bag, beans back in the can, whatever idiom you prefer) | 20:46 |
| mordred | beans in the cat? | 21:03 |
| fungi | no "Removing stale lock" in /var/log/zuul/debug.log yet, though it's still a bit early. i need to run a quick errand but will check again when i return if nobody else gets to it first | 21:04 |
| tristanC | corvus: anyone sounds good to me | 21:05 |
| clarkb | looks like users can delete their own messages too | 21:08 |
| clarkb | admin only necessary to delete someone else's messages | 21:09 |
| corvus | from my own private element -- if i do "explore public rooms" and then "add server" and add "opendev.org" i get a message saying i don't have permission to view the room list on that server. but it works if i do "ansible.im". any idea what we're missing? | 21:09 |
| corvus | mordred, clarkb: ^? | 21:09 |
| clarkb | I don't recall anything related to room listing when setting up the homeserver. Sorry I don't know | 21:10 |
| corvus | i wonder if it could be related to the client well-known file | 21:11 |
| clarkb | oh maybe? | 21:12 |
| clarkb | we didn't do the client one because we need to serve it with apache to set the right cors header iirc | 21:13 |
| clarkb | gitea will serve it but with the wrong cors headers. Apache is there and we can serve the files with it if we choose to | 21:14 |
| corvus | i checked my homeserver log -- it tried to make a request to 'opendev.org' | 21:15 |
| corvus | that seems like it might be well-known related | 21:16 |
| corvus | so sounds like apache may be the way to go | 21:16 |
| clarkb | sorry small correction we did do the client one, but gitea doesn't serve it with the correct cors header value | 21:17 |
| corvus | why is the apache proxy conditional? | 21:23 |
| clarkb | corvus: we weren't sure we would need it to start but then the ddos persisted so we switched to it | 21:25 |
| clarkb | I think we can probably stop using it conditionally now and just assume we'll have it | 21:25 |
| opendevreview | James E. Blair proposed opendev/system-config master: Serve matrix well-known files from apache https://review.opendev.org/c/opendev/system-config/+/805458 | 21:29 |
| opendevreview | James E. Blair proposed opendev/system-config master: Assume gitea reverse proxy https://review.opendev.org/c/opendev/system-config/+/805459 | 21:33 |
| opendevreview | James E. Blair proposed opendev/system-config master: Remove matrix well-known files from gitea image https://review.opendev.org/c/opendev/system-config/+/805460 | 21:34 |
| corvus | clarkb, fungi, mordred: ^ ianw fyi | 21:34 |
| clarkb | corvus: left a note on the second change that I think is worth double checking | 21:36 |
| corvus | agree | 21:37 |
| opendevreview | James E. Blair proposed opendev/system-config master: Assume gitea reverse proxy https://review.opendev.org/c/opendev/system-config/+/805459 | 21:38 |
| opendevreview | James E. Blair proposed opendev/system-config master: Remove matrix well-known files from gitea image https://review.opendev.org/c/opendev/system-config/+/805460 | 21:38 |
| clarkb | turning off the auto preview for urls makes things more readable in element when dealing with gerrit urls | 21:42 |
| clarkb | anyone know what sort of protocol gerrit needs to support to do those previews properly? | 21:43 |
| fungi | 2021-08-20 21:05:26,472 ERROR zuul.ExecutorQueue: Removing stale lock: /zuul/executor/unzoned/locks/e42339f7ad3844f499b6c4f4eeb8418f | 21:43 |
| fungi | 5352 of those logged with different lock names | 21:43 |
| opendevreview | James E. Blair proposed openstack/project-config master: Remove gerritbot from #zuul https://review.opendev.org/c/openstack/project-config/+/805463 | 21:43 |
| fungi | corvus: ^ seems like it worked! | 21:43 |
| corvus | fungi: excellent -- now the next question is whether we will see any further ones | 21:44 |
| clarkb | slack supports a bunch of different "unfurl" methods | 21:45 |
| opendevreview | James E. Blair proposed opendev/system-config master: Move #zuul from OFTC to Matrix https://review.opendev.org/c/opendev/system-config/+/805464 | 21:45 |
| clarkb | twitter and facebook use two different methods | 21:46 |
| clarkb | I suspect that Open Graph Data may be the thing though | 21:47 |
| opendevreview | James E. Blair proposed opendev/system-config master: Serve matrix well-known files from apache https://review.opendev.org/c/opendev/system-config/+/805458 | 21:47 |
| opendevreview | James E. Blair proposed opendev/system-config master: Assume gitea reverse proxy https://review.opendev.org/c/opendev/system-config/+/805459 | 21:47 |
| opendevreview | James E. Blair proposed opendev/system-config master: Remove matrix well-known files from gitea image https://review.opendev.org/c/opendev/system-config/+/805460 | 21:48 |
| corvus | clarkb: ^ updated based on your comment | 21:48 |
| fungi | another thing 805459 gives us is the option to stop using gitea itself to serve the opendev.org main page. that can now be something served directly by apache | 21:49 |
| clarkb | oh another though do we need to allow access to that doc root explicitly? | 21:49 |
| clarkb | fungi: corvus ^ | 21:49 |
| corvus | hrm maybe | 21:50 |
| corvus | maybe we can testinfra that | 21:50 |
| fungi | good idea | 21:50 |
| clarkb | ++ | 21:50 |
| corvus | this is literally what my ansiblefest talk is about so i better walk the walk | 21:55 |
| clarkb | fwiw it has been really helpful with testing things like the gerrit upgrade and downgrade between 3.2 and 3.3. Prior to that with getting the gitea upgrade for 1.15.0 sorted out | 21:56 |
| clarkb | re putting more data in the gerrit urls the content that is served is very limited and basically says preload a bunch of these api endpoints then run this script | 21:57 |
| corvus | if my talk has an audience you should be a plant :) | 21:57 |
| opendevreview | James E. Blair proposed opendev/system-config master: Serve matrix well-known files from apache https://review.opendev.org/c/opendev/system-config/+/805458 | 21:57 |
| clarkb | I wonder how difficult it would be for the gerrit server ti write out this metadata when so much of it is deferred to the client | 21:57 |
| corvus | clarkb, fungi: ^ how's that look? | 21:58 |
| * fungi will be a thallophytic plant-like organism | 21:58 | |
| corvus | fungi: you're fabulous | 21:58 |
| fungi | absolutely! | 21:58 |
| opendevreview | Tristan Cacqueray proposed opendev/system-config master: Update gerritbot-matrix to the latest version for on behalf annotation https://review.opendev.org/c/opendev/system-config/+/805465 | 21:58 |
| fungi | (fabulous fungi ii: absolutely fabulous fungi) | 21:59 |
| clarkb | corvus: that latest patchset lgtm | 21:59 |
| corvus | i wasn't 100% sure what apache would do with the value of the cors header so i omitted the "*" from the check, assuming that presence would be sufficient | 21:59 |
| corvus | we could tighten that up later if we want | 22:00 |
| corvus | (would the value be '""' or '' i dunno) | 22:00 |
| clarkb | ya I think we won't get the header at all otherwise so just asserting it is there is probably sufficient | 22:00 |
| corvus | oh it ate my stars | 22:00 |
| corvus | another weechat nit :) | 22:00 |
| corvus | weechat-matrix that is | 22:00 |
| fungi | so the matrix protocol expects *foo* to indicate some sort of emphasis, similar to _bar_? | 22:02 |
| corvus | fungi: no that's just matrix-weechat and doing a halfway job of dealing with it | 22:03 |
| fungi | i expect i'll eventually get annoyed enough by that to hack some sort of escaping into the plugin | 22:03 |
| mordred | we have 3x+2 on the apache change ... do I need to wait? | 22:04 |
| *** dviroel|ruck is now known as dviroel|out | 22:04 | |
| corvus | fungi: i should have typed: (would the value be "*" or * i dunno) | 22:04 |
| clarkb | the testinfra change should test the bits that matter | 22:04 |
| clarkb | mordred: ^ that means it should be safe to approve and it will fail if it odens't work for some reason | 22:04 |
| corvus | yeah, i feel like with testinfra there we can bombs-away | 22:04 |
| clarkb | corvus: going back to the separate zuul webs idea. Is there a reason to split them from the scheduler? or sould we plan to colocate those services when we run multiples of them? | 22:05 |
| fungi | agreed, i suppose there's a chance that we've all overlooked some error in the test itself which we might have caught by waiting for it to run and looking at the log, but i'm not overly concerne2d | 22:05 |
| corvus | fungi, clarkb, mordred: can you leave some votes on https://review.opendev.org/805463 and https://review.opendev.org/805464 ? i have WIPd those and will +W them tomorrow | 22:06 |
| corvus | clarkb: probably depends on what we want our new scheduler size to be. if we have an extra core or 2, then colocating them is probably a good idea for efficiency. if we max out the cpu for the scheduler process then split hosts. that's all i can think of. | 22:07 |
| fungi | though we could also consider that the web and fingergw processes are okay to leave as a spof because they're quick to restart | 22:08 |
| fungi | as long as they're entirely decoupled from the scheduler | 22:09 |
| clarkb | the scheduler is still largely only going to use a single cpu right? but do to memory needs and the way the clouds do flavors I think we should have extra cpu typically | 22:09 |
| clarkb | fungi: in my head every scheduler would also run a web and fingergw and then we can use haproxy or omething to point to them. But ya that is an option too I guess | 22:09 |
| corvus | yeah. probably like 1.2 cpus. | 22:09 |
| fungi | clarkb: leaving web and fingergw as spofs also means not needing to add a lb | 22:10 |
| mordred | corvus: I have +2d all the things - a few need to be rebased - want me to rebase them or want to wait? | 22:11 |
| fungi | i don't know which i prefer, just thinking through it, but the reduced complexity might outweigh the gains of hiding occasional brief outages | 22:11 |
| corvus | mordred: oh i'll rebase now | 22:11 |
| mordred | kk | 22:11 |
| mordred | also - I think +A as you feel | 22:11 |
| opendevreview | James E. Blair proposed opendev/system-config master: Assume gitea reverse proxy https://review.opendev.org/c/opendev/system-config/+/805459 | 22:11 |
| opendevreview | James E. Blair proposed opendev/system-config master: Remove matrix well-known files from gitea image https://review.opendev.org/c/opendev/system-config/+/805460 | 22:11 |
| corvus | the run-gitea job is running for the main change now | 22:12 |
| corvus | estimated time remaining 40m oy | 22:12 |
| mordred | that's a lot of m | 22:13 |
| corvus | so it'll be at least 1 hour 40m best case because of clean check :/ | 22:13 |
| corvus | which is like 100m | 22:13 |
| mordred | that's even more m | 22:13 |
| corvus | technically an order of magnitude more m | 22:14 |
| fungi | so not exactly a 100m dash | 22:14 |
| clarkb | its also a bit faster now that we use the api I think | 22:16 |
| clarkb | but ya it creates all the projects. We might be able to reduce that down to a representative sample now | 22:16 |
| clarkb | the cost is almost entirely in create all the projects then create them again to ensure we noop | 22:16 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update Gerrit images to most recent releases https://review.opendev.org/c/opendev/system-config/+/805471 | 22:56 |
| clarkb | infra-root ^ fyi I think landing and restarting on that can likely wait until Monday | 22:56 |
| corvus | sigh | 22:58 |
| opendevreview | James E. Blair proposed opendev/system-config master: Serve matrix well-known files from apache https://review.opendev.org/c/opendev/system-config/+/805458 | 22:59 |
| corvus | in about an hour, we'll find out the next typo :( | 22:59 |
| clarkb | oops | 22:59 |
| opendevreview | Merged opendev/system-config master: Update gerritbot-matrix to the latest version for on behalf annotation https://review.opendev.org/c/opendev/system-config/+/805465 | 23:12 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
