| tonyb | 904151 Looks good to me. I have +2'd it and will +A tomorrow unless someone says stop | 00:38 |
|---|---|---|
| frickler | oh, I somehow had in my mind we'd emergencied the container downgrade already, but indeed we're running 10.2.3. so let's see if that "patch version" downgrade is enough | 07:35 |
| opendevreview | Merged opendev/system-config master: Temporarily pin Grafana to 10.2.2 https://review.opendev.org/c/opendev/system-config/+/904151 | 08:02 |
| tonyb | okay. | 08:02 |
| frickler | seems the downgrade has fixed the nodepool dashboards, yay | 08:39 |
| tonyb | okay awesome. I can play with an autohold to see if any of the identified issues are ours | 08:51 |
| *** mrunge_ is now known as mrunge | 08:54 | |
| *** dhill is now known as Guest12635 | 15:16 | |
| opendevreview | Clark Boylan proposed opendev/system-config master: Test gitea haproxy maxconn limits https://review.opendev.org/c/opendev/system-config/+/904500 | 16:34 |
| clarkb | reminder we said no meeting today. Have a happy new year! | 16:35 |
| fungi | same to you! | 16:35 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Upgrade to etherpad 1.9.5 https://review.opendev.org/c/opendev/system-config/+/904231 | 16:42 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM force etherpad failure to hold node https://review.opendev.org/c/opendev/system-config/+/840972 | 16:42 |
| clarkb | autohold is in place for ^ | 16:43 |
| clarkb | re the long queued job/ref in the periodic queue: if I had to guess one of our nodepool providers isn't functioning properly and we're not able to reject a node request from that job | 16:51 |
| fungi | speaking of which, the linaro cloud api cert is expiring in a couple of weeks | 16:54 |
| frickler | I think kevinz was taking care of those? doesn't seem to be around though | 16:56 |
| frickler | also www.openstack.org getting old | 16:56 |
| frickler | regarding periodic queue, the stuck job is a plain openstack-tox-py39 | 16:57 |
| clarkb | I did the last refresh of the linaro cloud cert | 16:57 |
| clarkb | we basically have to run the acme refresh script and then the kolla redeploy. There was email about it | 16:57 |
| frickler | I thought linode is what we did ourselves? or am I mixing this up? | 16:57 |
| clarkb | there is no linode | 16:58 |
| frickler | ah, inmotion is the other one | 16:59 |
| clarkb | looks like the certcheck for linaro is working now. That is good (one of the issues before was we were pointing to an old url and it wasn't checking the correct item) | 16:59 |
| fungi | linode has never been one of our cloud donors (i don't think they even use openstack, they used to be uml-based) | 16:59 |
| clarkb | the cert I get for www.openstack.org expires in may | 17:12 |
| clarkb | they must be serving different certs depending on region? | 17:12 |
| clarkb | since it is being served by cloudflare | 17:13 |
| fungi | is that the www.openstack.org cert or the openstack.org cert? | 17:20 |
| clarkb | www | 17:20 |
| clarkb | at least that is what certcheck is reporting expires in 23 days, but using openssl s_client from here I get a valid cert until may | 17:21 |
| fungi | https://paste.opendev.org/show/bBfjpUMwq6hBvyEPGDBN/ | 17:22 |
| fungi | the www.o.o cert is handled by cloudflare, the o.o cert is letsencrypt | 17:23 |
| clarkb | hrm that implies certcheck is hitting openstack.org but thinking it is looking at www.openstack.org? | 17:24 |
| fungi | that's my suspicion, without checking the config | 17:24 |
| fungi | looks like service-discuss just got spammed. investigating now | 17:46 |
| frickler | "openssl s_client -connect openstack.org:443" gives a cert with "subject=CN = www.openstack.org", that's exactly what certcheck is reporting I'd say. and that has "NotAfter: Jan 25 02:17:49 2024 GMT" | 17:47 |
| fungi | it also includes san for openstack.org, web1.openstack.org and web2.openstack.org which is why it's valid for openstack.org requests | 17:49 |
| fungi | i've deleted the spam post from the service-discuss archive and switched that user's queue processing to automatically discard future posts | 17:51 |
| fungi | note that this spam was sent through the hyperkitty webui, rather than via smtp | 17:52 |
| clarkb | is it possible to force web post user through the initial moderation but not email posters? | 17:53 |
| clarkb | seems like the bulk of the trouble is via web and not smtp so if we can apply the small barrier there that might be a good compromise | 17:53 |
| fungi | there's no separation between smtp and web injection, so it has to be set for everyone | 17:56 |
| fungi | the main reason i think we get most of it coming via web is that the hyperkitty workflow is very convenient for new users: click "reply" and you'll be automatically subscribed to the list with delivery initially disabled (until you validate your e-mail address). posting via smtp requires explicit subscription and validation steps first before your post will be accepted | 17:59 |
| fungi | one thing which might help (coming in the next mailman version i think?) is captcha support | 18:00 |
| clarkb | 104.130.253.220 is the held etherpad. I'm using the clarkb-test etherpad. It seems to work for me in ff just fine. Chrome is having connection resets. I want to say we saw this before and the assumption was it was related to cookie mismatches between the actual server and the dev one | 18:09 |
| clarkb | oh actually pulling up chrome dev tools it is because the socket.io connection is failing to validate the cert even though I told it to accept the cert. I guess in chrome they don't carry that over to websocket connections | 18:10 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Test gitea haproxy maxconn limits https://review.opendev.org/c/opendev/system-config/+/904500 | 18:12 |
| clarkb | the first ps of ^ used the latest haproxy image and the test failed which is what I expected. I've updated to undo the image update and go back to lts which should pass and make the change mergeable | 18:12 |
| clarkb | re etherpad I think the update lgtm. I'll let others check but we can probably merge that soon | 18:12 |
| fungi | working for me from firefox too | 18:13 |
| clarkb | the gitea upgrade is one that we should look at soon too, but I'm thinking today may not be best for that from me. Just because I may not fully be back yet | 18:16 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/904500 passes against the lts haproxy image. I think that means this is a good test for this problem | 19:34 |
| clarkb | and since it is a test only update it should be safe to land whenever reviewers are happy with it | 19:34 |
| frickler | https://github.com/haproxy/haproxy/issues/2395 is closed now, but it seems we need a 2.9.2 tag for the fix to show up in the docker image | 19:52 |
| clarkb | yes they seem to be backporting fixes to the 2.9 branch but no new release has been made yet | 19:52 |
| clarkb | once new images are ready we can confirm it fixes our problem using this test and an image update change | 19:53 |
| clarkb | it isn't clear to me how their branching and tagging work though. They say they backported to 2.9 but then there is no 2.9 branch. There is also no 2.9.1 tag | 19:55 |
| frickler | there is, but not on github | 19:55 |
| frickler | they only mirror the master branch from https://git.haproxy.org/ | 19:56 |
| clarkb | oh I see | 19:56 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!