| mordred | corvus: more fail: https://zuul.opendev.org/t/openstack/build/84283ed96686489b965706323e5f3ccd | 00:47 |
|---|---|---|
| mordred | corvus: looks like baseurl and m.server swapped in the test | 00:48 |
| fungi | assert '"base_url": "https://opendev.ems.host"' in '{\n "m.server": "opendev.ems.host:443"\n}\n' | 01:28 |
| fungi | assert '"m.server": "opendev.ems.host:443"' in '{\n "m.homeserver": {\n "base_url": "https://opendev.ems.host"\n },\n "m.identity_server": {\n "base_url": "https://vector.im"\n }\n}\n' | 01:28 |
| fungi | yeah, i'll see if maybe the file contents are swapped | 01:29 |
| fungi | yep, either the file contents are backwards or the tests are backwards... | 01:30 |
| fungi | i think it's the tests which are backwards, the file seem to match what we're currently putting in the gitea containers | 01:32 |
| fungi | i'll try to fix it up | 01:32 |
| fungi | i think it's the urls which are switched? we want to see base_url in client with no cors header, and m.server in server with a cors header, yeah? | 01:36 |
| fungi | ahh, no, we're doing the cors header on client not server | 01:37 |
| fungi | so it's just the string matches which are swapped in the tests | 01:37 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Serve matrix well-known files from apache https://review.opendev.org/c/opendev/system-config/+/805458 | 01:39 |
| fungi | that ^ should hopefully solve it | 01:39 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Assume gitea reverse proxy https://review.opendev.org/c/opendev/system-config/+/805459 | 01:41 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Remove matrix well-known files from gitea image https://review.opendev.org/c/opendev/system-config/+/805460 | 01:41 |
| fungi | rebase those ^ on the revised one | 01:42 |
| fungi | er, rebased | 01:42 |
| fungi | closer at least... | 02:56 |
| fungi | AssertionError: assert 'Access-Control-Allow-Origin' in '* Added gitea99.opendev.org:3000:127.0.0.1 to DNS cache\n* Hostname gitea99.opendev.org was found in DNS cache\n* T...3 0 0 9562 0 --:--:-- --:--:-- --:--:-- 9562\n* Connection #0 to host gitea99.opendev.org left intact\n' | 02:56 |
| fungi | i think the header output may not be where the test expects? | 02:56 |
| fungi | mmm, no, using a similar curl command, i get "< Access-Control-Allow-Origin: *" from somewhere with it set, so maybe apache isn't correctly serving it? | 03:06 |
| fungi | mod_headers seems to be enabled, at leas on our production servers, so it's probably not that | 03:15 |
| fungi | should have a held node here shortly | 04:30 |
| corvus | wow that ended up being complex :/ | 04:45 |
| corvus | fungi: indeed, i don't see a cors header when i curl on that server | 04:48 |
| corvus | that's the job doing? it looks like it's just been sitting there for 16 minutes | 04:49 |
| corvus | well, gitea process seems to be busy, maybe this is the part of the job where it's creating repos | 04:50 |
| corvus | i think it's proxying to gitea | 04:51 |
| corvus | the ilikegitea cookie is being sent | 04:51 |
| corvus | oh i see the problem -- the test isn't hitting apache, it's hitting gitea directly | 05:01 |
| corvus | apache listens on 3081 | 05:01 |
| opendevreview | James E. Blair proposed opendev/system-config master: Serve matrix well-known files from apache https://review.opendev.org/c/opendev/system-config/+/805458 | 05:05 |
| opendevreview | James E. Blair proposed opendev/system-config master: Switch robots.txt test on gitea to proxy port https://review.opendev.org/c/opendev/system-config/+/805472 | 05:05 |
| corvus | fungi: ^ hopefully that should get it | 05:05 |
| corvus | thanks for pushing that along :) | 05:05 |
| opendevreview | James E. Blair proposed opendev/system-config master: Assume gitea reverse proxy https://review.opendev.org/c/opendev/system-config/+/805459 | 05:06 |
| opendevreview | James E. Blair proposed opendev/system-config master: Switch robots.txt test on gitea to proxy port https://review.opendev.org/c/opendev/system-config/+/805472 | 05:06 |
| opendevreview | James E. Blair proposed opendev/system-config master: Remove matrix well-known files from gitea image https://review.opendev.org/c/opendev/system-config/+/805460 | 05:06 |
| fungi | VirtualHost *:3081 | 05:07 |
| fungi | yep :/ | 05:07 |
| fungi | just got on the held node to check | 05:08 |
| fungi | i should have spotted that sooner | 05:08 |
| fungi | well, i guess we at least proved the context check against the files gitea's serving | 05:08 |
| fungi | so if they also work against apache then we can be more confident they're correct | 05:09 |
| corvus | ya | 05:09 |
| fungi | i'll clear the autohold and knock off for the night | 05:10 |
| fungi | thanks! | 05:10 |
| corvus | fungi: thanks! goodnight! | 05:11 |
| fungi | you too. i'll check back in after breakfast and see where we are | 05:11 |
| opendevreview | Merged opendev/system-config master: Serve matrix well-known files from apache https://review.opendev.org/c/opendev/system-config/+/805458 | 07:12 |
| yoctozepto | infra-root: ethercalc is down | 07:57 |
| yoctozepto | error 503 | 07:57 |
| yoctozepto | (the usual) | 07:57 |
| *** redrobot4 is now known as redrobot | 08:04 | |
| yoctozepto | up again | 09:46 |
| mordred | fungi, corvus: the patch is landed and the well-known file looks good, but I still can't browse opendev.org channels | 12:27 |
| fungi | mmm | 12:39 |
| mordred | I can join them | 12:40 |
| fungi | i can confirm https://opendev.org/.well-known/matrix/client has a "Access-Control-Allow-Origin: *" header served with it | 12:41 |
| mordred | https://github.com/matrix-org/synapse/blob/fd9856e4a98fb3fa9c139317b0a3b79f22aff1c7/docs/sample_config.yaml#L134 | 12:41 |
| mordred | one of the gnome guys suggested that there is a homeserver config setting ^^& | 12:41 |
| mordred | we might need to engage with our element friends to get them to set that? | 12:41 |
| opendevreview | Merged opendev/system-config master: Assume gitea reverse proxy https://review.opendev.org/c/opendev/system-config/+/805459 | 13:29 |
| yoctozepto | so the gnomes are real? ;o | 13:32 |
| fungi | yoctozepto: yes, every morning when i wake up, the software gnomes have added new bugs for me to troubleshoot | 13:33 |
| yoctozepto | fungi: :O no kidding, must be the reason there are so many bugs | 13:34 |
| fungi | infra-root: our shared inbox contains a couple of "please confirm your address" messages from element/matrix which arrived around 21:00 utc yesterday. just a heads up in case someone hasn't followed up on those. if they're no longer needed i'll move them to our element service archive mailbox | 13:37 |
| corvus | fungi: i probably triggered those; i don't know why it asked for them... :/ | 13:38 |
| * corvus uploaded an image: (40KiB) < https://matrix.org/_matrix/media/r0/download/acmegating.com/pZQwmZbrxdXxAYLKdqmbukNr/image.png > | 13:38 | |
| corvus | mordred: fungi do you think it could be the "Guest users" setting? | 13:39 |
| fungi | we already talked yesterday about turning that on, right? | 13:40 |
| fungi | seemed like there were no objections | 13:40 |
| corvus | fungi: i'm not 100% sure what that option entails, though i think generally it sounds like something we'd want? | 13:40 |
| fungi | the way it was described yesterday sounded like a net positive | 13:41 |
| corvus | based on chat history in #ems -- the 'guest users' feature would probably be needed in order to allow anonymous read access to rooms, so yeah, i think it's the next step in that and seems likely to be related to the lack of ability to browse lists | 13:45 |
| fungi | corvus: skimming back in discussion history, it looks like the confirmation e-mails may coincide with when you tried using an admin account to delete messages from the test channel | 13:45 |
| corvus | fungi: no it was when i logged into the ems dashboard and it kept asking me for my name | 13:46 |
| fungi | aha | 13:48 |
| fungi | so apparently that generates address confirmation e-mails | 13:48 |
| corvus | yeah; maybe our account got into a weird state or something. i haven't confirmed the email (again?), but i probably should | 13:49 |
| corvus | i asked in #ems if there's a setting that corresponds to the link from mordred | 13:50 |
| corvus | i'm assuming if we turn on the 'guest users' switch this will be enabled: https://github.com/matrix-org/synapse/blob/fd9856e4a98fb3fa9c139317b0a3b79f22aff1c7/docs/sample_config.yaml#L1218-L1222 | 13:57 |
| corvus | but maybe it will also enable the other option? | 13:57 |
| corvus | why don't i try enabling guest access and let's see what happens? | 14:04 |
| fungi | worth a try | 14:07 |
| fungi | skimming the config, room_list_publication_rules is likely applicable | 14:07 |
| corvus | i enabled it, and playing around with our (unpublicized) element instance, i am able to browse the list of rooms, and see a preview of #zuul without logging in. i get prompts to create an account if i want to join; of course that doesn't work since we don't allow registration. | 14:10 |
| fungi | ahh, yeah, i didn't consider that the prompt to register would point to registering with our homeserver | 14:11 |
| fungi | i wonder if there's a way to have it send people to matrix.org's registration page | 14:12 |
| corvus | okay, if i add 'opendev.org' as a server, i can browse the directory now, so it looks like this did enable the feature we want | 14:16 |
| corvus | it's a little unfortunate that typing in "#zuul:opendev.org" doesn't automatically do that, so you don't get feedback that it's correct. but this looks like the best that's possible right now | 14:16 |
| corvus | i think this is the real benefit of anonymous access -- visit this in a private tab: https://matrix.to/#/#zuul:opendev.org | 14:18 |
| corvus | that's the link we're going to have on the web site and docs | 14:18 |
| corvus | that lets you preview the room, and, happily, it's going through app.element.io, so the signup links are for the matrix.org homeserver | 14:19 |
| corvus | all in all, i think this setting is good for us | 14:19 |
| fungi | the privacy policy is interesting | 14:19 |
| fungi | "You must be at least 16 years old to use this Service (opendev.element.io)" | 14:19 |
| fungi | i guess this is a cookie-cutter policy adapted from element | 14:20 |
| fungi | but yeah, it's only a few clicks from that url to get to a view of the channel through the in-browser element client | 14:21 |
| fungi | actually the "sign in" and "sign up" links there default to matrix.org as the location for the account when i click them | 14:22 |
| corvus | yep | 14:22 |
| corvus | it's only if you do the same thing through our hidden element client that they default to opendev as the homeserver | 14:23 |
| fungi | ahh, so it's by virtue of using the matrix.to url that we get matrix.org as a default account provider | 14:23 |
| corvus | yep | 14:23 |
| fungi | and also i get to imagine i'm visiting a site in tonga | 14:25 |
| fungi | virtual vacation in the pacific islands | 14:26 |
| corvus | #status log enabled "Guest access" on matrix synapse server EMS control panel to allow anonymous read access to rooms which allow that, and for federated homeservers to be able to browse the published rooms list | 14:30 |
| opendevstatus | corvus: finished logging | 14:30 |
| opendevreview | Merged openstack/project-config master: Remove gerritbot from #zuul https://review.opendev.org/c/openstack/project-config/+/805463 | 14:47 |
| opendevreview | Merged opendev/system-config master: Remove matrix well-known files from gitea image https://review.opendev.org/c/opendev/system-config/+/805460 | 14:49 |
| mordred | corvus: \o/ | 14:53 |
| mordred | corvus: I confirm that in my element client I can now add opendev.org to the server list and browse the channels and it shows me the zuul channel | 14:53 |
| opendevreview | Merged opendev/system-config master: Move #zuul from OFTC to Matrix https://review.opendev.org/c/opendev/system-config/+/805464 | 14:57 |
| corvus | \o/ | 14:58 |
| mordred | corvus: it's happening | 15:09 |
| tristanC | it seems like the gerritbot didn't pick the zuul room config, is /var/lib/matrix-gerritbot/config/gerritbot.dhall updated? if it is, the autoreload seems broken and the compose should be restarted. if it isn't, there may be something wrong with the system-config ansible tasks | 15:40 |
| fungi | i'll look | 15:52 |
| fungi | /var/lib/matrix-gerritbot/config/gerritbot.dhall does not have the zuul room in it yet, just the test room still | 15:58 |
| fungi | did the deploy job complete? | 15:58 |
| mordred | fungi: it has not yet started | 16:00 |
| mordred | fungi: there are two ahead of it - and one of them is running quite long | 16:00 |
| fungi | that'd 'splain it | 16:00 |
| corvus | the eavesdrop job just finished | 16:03 |
| corvus | there's another project-config eavesdrop job to run too | 16:03 |
| corvus | oh sorry, that was for an older change... that's what you meant | 16:04 |
| corvus | so yeah, the job for the queue item for 805464 probably won't run for another 2 hours :( | 16:04 |
| corvus | good thing we didn't specify a cutover time :) | 16:04 |
| mordred | yeah | 16:05 |
| mordred | corvus: maybe the following system-config changes won't deploy *everything* ? | 16:06 |
| mordred | corvus, fungi : base failed on ns2 with a dpkg lock issue trying to run apt-get autoremove | 16:08 |
| mordred | there seems to be a dpkg process from aug 6 that's trying to remove old kernels | 16:09 |
| fungi | oh fun, i'll check it out | 16:09 |
| mordred | I'm thinking we shoudl kill that, clean the lock and re-run autoremove to get ns2 into a non-broken state | 16:10 |
| fungi | yeah, go for it | 16:10 |
| corvus | mordred: ++ | 16:10 |
| mordred | root 20417 20405 0 Aug06 pts/1 00:00:03 /usr/bin/dpkg --status-fd 22 --no-triggers --force-depends --remove linux-headers-4.15.0-147-generic:amd64 linux-headers-4.15.0-147:all linux-headers-4.15.0-45-generic:amd64 linux-headers-4.15.0-45:all linux-image-4.15.0-147-generic:amd64 linux-image-4.15.0-45-generic:amd64 linux-modules-4.15.0-147-generic:amd64 linux-modules-4.15.0-45-generic:amd64 | 16:10 |
| mordred | that's the hung process, fwiw. I'm betting there's a prompt :) | 16:10 |
| fungi | likely | 16:10 |
| mordred | "A new version of /boot/grub/menu.lst is available" | 16:11 |
| mordred | I'm leaning towards "install the package maintainer's version" - we don't actualy manage that file, yeah? | 16:12 |
| fungi | we don't | 16:12 |
| fungi | you can diff it if you're concerned | 16:12 |
| mordred | the new version removes references to initrd | 16:13 |
| mordred | so - seems like we should follow pacakge maint lead :) | 16:13 |
| mordred | k. ns2 should work next time | 16:14 |
| fungi | awesome, thanks! | 16:14 |
| mordred | fwiw - we've made it to remote-puppet-else | 16:15 |
| mordred | ooh! that went quick | 16:16 |
| mordred | we're on the second change now - and it's only doing gitea | 16:16 |
| fungi | clarkb put in some effort to speed these up | 16:18 |
| fungi | though there's still more we can do | 16:18 |
| mordred | yah. it's one of those forever tasks | 16:19 |
| mordred | corvus: your change is now queued for deployment. it is going to run ALL the jobs | 17:02 |
| mordred | which I'm guessing is largely due to inventory file matchers | 17:03 |
| fungi | yes, it's that it touches the inventory | 17:09 |
| opendevreview | Monty Taylor proposed opendev/system-config master: Restrict generic inventory matchers to inventory/base https://review.opendev.org/c/opendev/system-config/+/805487 | 17:13 |
| corvus | makes sense to me | 17:14 |
| opendevreview | Merged opendev/system-config master: Run service-eavesdrop after promoting the matrix eavesdrop bot https://review.opendev.org/c/opendev/system-config/+/805446 | 17:25 |
| tristanC | thank you for looking into that, it seems like the new matrix room is now configured for the gerritbot | 18:10 |
| corvus | eavesdrop is now running with the new config; i can see it on the text file; the html file should update the next time the cronjob runs | 18:18 |
| corvus | i think that means we have switched :) | 18:18 |
| mordred | corvus: how do we (gracefully) eject people from the old room? do we just ask nicely and then all just leave? | 18:20 |
| fungi | we could make the room invite only and then kick everyone | 18:21 |
| fungi | or we could just let everyone know and then add a welcome message and topic which both let folks know we're using #zuul:opendev.org via matrix instead | 18:22 |
| opendevreview | Tristan Cacqueray proposed opendev/system-config master: Update gerritbot-matrix version to fix a message typo https://review.opendev.org/c/opendev/system-config/+/805489 | 18:22 |
| corvus | fungi, mordred: i'm about to send out an email -- then i think we should do fungi's suggestion, maybe for the next week or so? then maybe see about further restriction (muting or invite only or something) | 18:23 |
| mordred | wfm | 18:26 |
| corvus | it looks like limnoria may still be logging, so we're sort of interleaving both rooms in the log right now | 18:35 |
| corvus | i'll double check its config and see if it needs to be restarted | 18:35 |
| corvus | erm, does anyone know where /var/lib/limnoria/opendev/conf/channels.conf comes from? | 18:38 |
| fungi | digging | 18:40 |
| corvus | i think maybe that's internal state from limnoria? | 18:41 |
| fungi | i think it must be generated, i can't find the string "lobotomized" in system-config nor project-config | 18:41 |
| corvus | i think our ansible creates limnoria.config, and that does not have #zuul in it | 18:41 |
| corvus | so i'm thinking we should restart limnoria and see if it changes | 18:42 |
| fungi | probably expects to restart the container, yes | 18:42 |
| fungi | i agree | 18:42 |
| fungi | now seems like a great time to do that | 18:42 |
| fungi | you want to or shall i? | 18:42 |
| corvus | i will | 18:42 |
| corvus | has it rejoined? i'm not sure i see all the events | 18:45 |
| corvus | oh yep, i see it in element | 18:46 |
| corvus | ah, and now i see log entries | 18:46 |
| corvus | wonder what the delay is there | 18:46 |
| corvus | looks like it's done joining, and it has not rejoined #zuul, so i think it's all good now | 18:48 |
| corvus | mordred, fungi: maybe a quick +3 on https://review.opendev.org/805489 ? | 18:51 |
| mordred | corvus: done | 18:57 |
| mordred | tristanC: docker: Error response from daemon: manifest for quay.io/software-factory/gerritbot-matrix:af7dc8c not found: manifest unknown: manifest unknown. | 19:10 |
| mordred | tristanC: https://zuul.opendev.org/t/openstack/build/486932f777074cfd964b73de37bee469 | 19:10 |
| mordred | tristanC: verified locally | 19:11 |
| mordred | tristanC: my local docker says: | 19:13 |
| mordred | quay.io/software-factory/gerritbot-matrix latest 75c78329e615 5 weeks ago 362MB | 19:13 |
| mordred | 5 weeks ago seems a little old - maybe something weird happened? | 19:13 |
| mordred | oh - I see what you're doing | 19:14 |
| mordred | is 0af1f8f the more right sha then? | 19:14 |
| opendevreview | Takashi Kajinami proposed openstack/project-config master: Telemetry: Switch back to launchpad from storyboard https://review.opendev.org/c/openstack/project-config/+/805492 | 19:16 |
| mordred | OH ... | 19:16 |
| mordred | tristanC: actually - 0af1f8f is the older one - I see that now. in quay.io that's the most recent one I see - I think afdc8c may have had a publication sad | 19:17 |
| tristanC | mordred: yeah, publication failed, the tag is now available | 19:24 |
| tristanC | mordred: it's a nix build i need to move in a zuul jobs. you might be interested to inspect the image, you'll see one layer per package | 19:31 |
| opendevreview | Merged opendev/system-config master: Update gerritbot-matrix version to fix a message typo https://review.opendev.org/c/opendev/system-config/+/805489 | 20:28 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
