| opendevreview | Jeremy Stanley proposed opendev/system-config master: Restart Mailman 3 containers when configs change https://review.opendev.org/c/opendev/system-config/+/891555 | 00:01 |
|---|---|---|
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Restart Mailman 3 containers when configs change https://review.opendev.org/c/opendev/system-config/+/891555 | 01:20 |
| opendevreview | Merged opendev/system-config master: Setup #openstack-charms IRC channel. https://review.opendev.org/c/opendev/system-config/+/890341 | 01:40 |
| fungi | looks like acme.sh is failing to renew the cert for insecure-ci-registry01 which expires today | 02:39 |
| fungi | huh, though that's no longer in dns, so why is it still trying to renew it? | 03:07 |
| fungi | insecure-ci-registry02 is in dns | 03:07 |
| Clark[m] | fungi: because it is still in the inventory. It's been on my list to delete it. I have a change up to pull it from inventory first that hasn't gotten any reviews yet | 03:22 |
| Clark[m] | We can put it in the emergency file instead but I was hoping for the proper removal | 03:22 |
| fungi | oh, i should have looked for that before i started writing one! | 03:25 |
| fungi | yours looks identical to what i was about to push | 03:26 |
| fungi | thanks! | 03:26 |
| opendevreview | Merged opendev/system-config master: Remove old insecure-ci-registry01 from our inventory https://review.opendev.org/c/opendev/system-config/+/887001 | 04:38 |
| *** dasm is now known as Guest9148 | 09:01 | |
| opendevreview | Stephen Finucane proposed openstack/project-config master: Add github sphinx-doc/sphinx to the project list https://review.opendev.org/c/openstack/project-config/+/891679 | 10:30 |
| opendevreview | Merged openstack/project-config master: Add github sphinx-doc/sphinx to the project list https://review.opendev.org/c/openstack/project-config/+/891679 | 12:10 |
| TheJulia | Would you awesome folks please put a job hold in for ironic-tempest-partition-uefi-ipmi-pxe-ovn I'm trying to figure out why I'm getting no logging out of zuul for OVN so I can troubleshoot the configuration and having no luck | 14:16 |
| fungi | TheJulia: for what repository? | 14:18 |
| TheJulia | ironic | 14:19 |
| TheJulia | openstack/ironic | 14:19 |
| TheJulia | change 885087 | 14:19 |
| fungi | added | 14:19 |
| TheJulia | much appreciated, thanks | 14:19 |
| fungi | yw | 14:19 |
| fungi | Clark[m]: are you done with the autohold for etherpad 1.9.1 testing now that we've upgraded? | 14:20 |
| fungi | frickler: are you still using the autohold for debugging dynamic routing? | 14:21 |
| frickler | fungi: yes | 14:21 |
| fungi | thanks, i'll clean that one up | 14:22 |
| frickler | ehm, still using means please don't clean it | 14:22 |
| fungi | oh, sorry :/ | 14:22 |
| Clark[m] | fungi: yes the etherpad 1.9.1 hold can be deleted. They just released 1.9.2 today too :) | 14:22 |
| fungi | frickler: apologies, i'll add you a new autohold, i missed that i asked you if you were using it rather than done with it. i'll add another hold | 14:23 |
| frickler | fungi: nevermind, I was mostly done, I guess I can also go on without it | 14:24 |
| fungi | okay, really sorry about that | 14:25 |
| fungi | Clark[m]: etherpad hold is gone now | 14:25 |
| fungi | Clark[m]: once you're more awake, did you want to look closer at the possibility of a gitea upgrade before the weekend? | 14:37 |
| fungi | also i think i got the mailman 3 restart handler updated with your recommendation | 14:38 |
| Clark[m] | Re the gitea upgrade I think the main thing is general review of the existing change and some thought on whether or not we care about configuring a bunch of oauth2 settings we ultimately end up disabling. | 14:50 |
| fungi | sounds good | 14:51 |
| Clark[m] | I'll take a look at the mm3 change shortly. | 15:03 |
| *** Guest9148 is now known as dasm | 15:11 | |
| clarkb | fungi: yup that looks correct. I half wonder if the condition in the handler is worth a comment but I've +2'd it. | 15:30 |
| fungi | there's no rush, but presumably we'd also need to merge another config change to actually trigger the restart, right? | 15:30 |
| clarkb | yes | 15:31 |
| fungi | oh, also i can certainly add a comment there | 15:31 |
| fungi | oh, or i can fudge a config change by manually rolling back the settings.py SITE_ID update, right? | 15:31 |
| clarkb | thinking about gitea, I don't think I'm in a rush there either and would prefer reviewers look over it carefully. This is the biggest gitea upgrade in terms of moving pieces on our side (but not features, weird how that works out) so adding confidence to it would be nice | 15:31 |
| clarkb | fungi: any update to the files that notify the handler should do it | 15:32 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Restart Mailman 3 containers when configs change https://review.opendev.org/c/opendev/system-config/+/891555 | 15:33 |
| clarkb | therefore for gitea I think its best if we focus on review for now and then decide when we're comfortable merging/deploying that change after reviews | 15:34 |
| fungi | sure, that seems like a safe way to proceed | 15:35 |
| fungi | so if you want to approve 891555 with the added comment, i'll manually switch SITE_ID back to 1 on the server and then when the change rolls out we should see the deploy job correct the value in settings.py back to 0 and then restart the containers after doing so | 15:37 |
| fungi | at which point the server name on the lists.zuul-ci.org archive pages should start to have the correct string | 15:38 |
| fungi | i changed the value in /var/lib/mailman/web/settings.py at 15:39 utc (what's now showing as the last modified timestamp) | 15:40 |
| clarkb | fungi: that change also removes the inventory file for insecure ci registry. was that intentional? | 15:48 |
| clarkb | I think git may handle that properly as long as it is identical to the change that landed yesterday | 15:48 |
| fungi | oh, no that was not. that was the mostly written change i was about to push last night and i forgot to hard reset when i did git-review -d just now | 15:49 |
| fungi | i'll fix it, thanks for spotting | 15:49 |
| clarkb | ok | 15:49 |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Restart Mailman 3 containers when configs change https://review.opendev.org/c/opendev/system-config/+/891555 | 15:52 |
| clarkb | I've approved ^ | 15:53 |
| fungi | thanks | 15:54 |
| clarkb | https://review.opendev.org/c/opendev/system-config/+/886993 is the gitea 1.20 change and https://104.239.143.111:3081/opendev/system-config is the resulting held node on the child change | 16:27 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Run bootstrap-bridge with empty nodeset https://review.opendev.org/c/opendev/system-config/+/891869 | 17:02 |
| clarkb | infra-root (and ianw if possible) I think that is a good optimization for our hourly job runs | 17:02 |
| fungi | oh, good catch | 17:05 |
| TheJulia | looks like hold 0000000239 is waiting, who would like my ssh pubkey | 17:10 |
| fungi | TheJulia: gimme | 17:11 |
| * TheJulia sees fungi laughing evilly | 17:12 | |
| TheJulia | :)_ | 17:12 |
| TheJulia | err :) | 17:12 |
| fungi | TheJulia: ssh root@104.130.219.43 | 17:14 |
| TheJulia | thanks | 17:16 |
| fungi | any time | 17:16 |
| TheJulia | and in, thanks! | 17:16 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Upgrade Etherpad to 1.9.2 https://review.opendev.org/c/opendev/system-config/+/891877 | 17:50 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM force etherpad failure to hold node https://review.opendev.org/c/opendev/system-config/+/840972 | 17:51 |
| clarkb | I've put a hold in place for ^ so that we can sanity check things pre upgrade | 17:52 |
| clarkb | zuul today looks like pre feature freeze is happening | 18:18 |
| fungi | yeah | 18:24 |
| fungi | there was a big spike in merger requests at 14:34 which i assume corresponded to a bunch of changes getting enqueued all at once. the node request backlog has been burning down for an hour or so at least | 18:25 |
| frickler | https://review.opendev.org/q/topic:release-health-check-cwi-bobcat | 18:27 |
| frickler | though that seems to have been submitted starting 14:55 | 18:28 |
| fungi | there was a much smaller spike around 14:57 which probably corresponds to those | 18:35 |
| fungi | infra-root: i've opened ticket #230817-ord-0001055 under our nodepool account in rackspace, requesting they delete the nearly 250 stuck error deleting instances so we can get back to using that third of our overall capacity | 18:47 |
| fungi | i'm also doing another manual image upload test to rax-iad, in hopes whatever the problem is with their glance in that region has cleared up | 18:49 |
| fungi | upload completed and the image import task was created in glance at 2023-08-17T18:53:02Z but i'm still looping waiting for it to appear in the image list | 19:10 |
| clarkb | was the upload any faster than before or was all the time in waiting for it show up in the listing? | 19:19 |
| clarkb | I've got kids gymnastics presentation thing this afternoon I'll pop out early for just a heads up | 19:19 |
| clarkb | and probably tomorrow morning I'm going to get out on the bike as our heat wave is ending | 19:19 |
| fungi | upload time was 4m30.807s but i don't think that's necessarily faster/slower than before when i was testing | 19:21 |
| fungi | the delay is pretty much entirely waiting for whatever happens between upload and appearing in the image list | 19:21 |
| fungi | bike ride sounds like a great idea. i think we're planning for a nice long walk first thing on saturday morning before it gets unbearably hot | 19:22 |
| clarkb | its been crazy hot since sunday here and even though today is a bit cooler I would've had to get out the door by like 8am for it to be bearable and I'm still too jet lagged for that | 19:25 |
| clarkb | tomorrow morning should be much better | 19:25 |
| fungi | the past month here has been either hot+humid with no breeze, or raining, so i'm way behind on yardwork as a result | 19:26 |
| fungi | appeared sometime between 19:26:48 and 19:27:48 (loop was polling once a minute) | 19:29 |
| fungi | so around 34-35 minutes after upload | 19:31 |
| fungi | keep in mind that's with our uploads to that region paused | 19:31 |
| fungi | so this was just a single upload (from bridge, not even a builder) | 19:31 |
| npanda_ | How do I get Genoa support. I am Using Openstack Zed release | 19:43 |
| npanda_ | qemu-system-x86_64 -cpu ?|grep -i genoa --> Doesn't return anything | 19:43 |
| npanda_ | libvirtd --version returns libvirtd (libvirt) 8.0.0 | 19:44 |
| Clark[m] | npanda_ this channel is dedicated to discussion of the development tools used to build openstack not openstack itself. I'm not sure what Genoa is or where to get help for that though. | 19:46 |
| fungi | npanda_: you may have more luck asking openstack questions in the #openstack channel or on the openstack-discuss@lists.openstack.org mailing list | 19:49 |
| fungi | (openDEV is not openSTACK) | 19:50 |
| fungi | i'm familiar with the city of genoa, and also their lovely sausages | 19:53 |
| fungi | looks like maybe genoa is the codename for one of amd's cpus | 19:53 |
| opendevreview | Merged opendev/system-config master: Restart Mailman 3 containers when configs change https://review.opendev.org/c/opendev/system-config/+/891555 | 19:55 |
| fungi | yay! now to await the deploy job | 19:56 |
| fungi | which, conveniently, is already running | 19:56 |
| fungi | it did update the /var/lib/mailman/web/settings.py file but the uwsgi processes still have a start time from tuesday | 19:58 |
| fungi | :( | 19:58 |
| fungi | RUNNING HANDLER [mailman3 : down containers] skipping: [lists01.opendev.org] => {"changed": false, "skip_reason": "Conditional result was False"} | 20:00 |
| fungi | the "mailman check for running containers" task did get an rc of 0 so it must be the docker-compose stdout that didn't match | 20:01 |
| fungi | now i wonder if that was supposed to be on stderr | 20:01 |
| clarkb | oh that could be I guess. Maybe test docker-compose up -d on the list server too? | 20:02 |
| fungi | yep, it's on stderr | 20:03 |
| * fungi sighs | 20:03 | |
| opendevreview | Jeremy Stanley proposed opendev/system-config master: Mailman3: check docker-compose stderr not stdout https://review.opendev.org/c/opendev/system-config/+/891903 | 20:05 |
| fungi | once more unto the breach! | 20:06 |
| fungi | https://grafana.opendev.org/d/a8667d6647/nodepool-rackspace shows the "deleting" count is rapidly falling over the past few minutes, so my smoke signal may have reached them | 20:10 |
| clarkb | I've approved that if you want to update the file on disk | 20:10 |
| fungi | thanks | 20:10 |
| fungi | manually rolled /var/lib/mailman/web/settings.py back to SITE_ID=1 again and new modified time on the file is 20:11 | 20:11 |
| fungi | nevermind abouit the deleting nodes, i was looking at the wrong graph | 20:12 |
| fungi | i don't think they've started working on that ticket yet | 20:12 |
| fungi | an interesting result of our speculative container builds... the fact that they all rely on a common opendev-buildset-registry job throws a serious wrench into zuul's ability to estimate buildset completion time | 20:28 |
| fungi | since that job will stay paused for varying amounts of time based on whatever the duration of the other jobs relying on it have | 20:29 |
| fungi | so for 891903 it thinks that the builds which need the registry are going to wrap up in another 2-3 minutes, but that the registry job itself needs another half hour after those finish | 20:30 |
| fungi | that said, i can't currently conceive of an elegant estimation algorithm to address that challenge | 20:31 |
| fungi | clarkb: looking at the held gitea upgrade sample, i noticed something that isn't new but is broken... if you go to https://opendev.org/opendev then at the top there's a "projects" tab which leads to https://opendev.org/opendev/-/projects and a 404 page | 20:36 |
| fungi | probably something we can comment out in the template | 20:37 |
| fungi | also while browsing, the ambiguous unicode character identification on https://opendev.org/opendev/system-config/src/branch/master/docker/gitea/custom/templates/home.tmpl is pretty nifty even if i can't immediately tell which characters on the affected lines are at fault | 20:41 |
| fungi | maybe the apostrophes aren't apostrophes but rather "smart" quotes? | 20:42 |
| clarkb | fungi: I think we can disable repo.projects in the config and that might be what we need to fix the first thing | 20:45 |
| fungi | https://github.com/go-gitea/gitea/blob/b658f2c61bc70f0ce79d1be7eb1d7654273716e3/templates/org/menu.tmpl#L9-L13 | 20:45 |
| fungi | i guess that's where it's coming from? | 20:46 |
| clarkb | ya looks like it. So can read projects is true but maybe disabling projects will flip that | 20:46 |
| clarkb | or its a bug and they need to check projects are neabled and readable | 20:46 |
| clarkb | I'm not sure I understand the second thing | 20:46 |
| clarkb | oh it annotates the problem lines that helps | 20:47 |
| fungi | the second thing wasn't a bug, i was just trying to figure out what specific characters i considered ambiguous in those lines | 20:47 |
| fungi | and yeah, you have to scroll down to get to them | 20:47 |
| clarkb | ah yup if you highlihgt the ' in we'll it tells you that is the problem | 20:47 |
| clarkb | U+2019 confusable with U+0060 | 20:47 |
| fungi | oh, indeed, i didn't notice there was a tooltip | 20:48 |
| clarkb | don't need to highlight just hover on it with your pointer | 20:48 |
| clarkb | DISABLED_REPO_UNITS = repo.issues,repo.pulls,repo.wiki,repo.projects,repo.actions <- we already set that and I think that is why it is a 404 | 20:48 |
| clarkb | so that is a bug that should be fixed upstream imo | 20:48 |
| fungi | right, but it's not clear where to hover since (at least in my browser) there's no marker other that at the beginning of the line. i guess slowly move the mouse cursor along the line until we get a popup tooltip | 20:49 |
| fungi | still pretty cool | 20:49 |
| clarkb | fungi: if you click the alert sign it just barely highlights the characters | 20:50 |
| fungi | oh, huh... | 20:50 |
| clarkb | its super missable on my laptop browser | 20:50 |
| fungi | yeah, neat | 20:50 |
| fungi | actually fairly noticeable on my browser since it's assuming dark mode from my defaults so the bright yellow box around it pops nicely | 20:51 |
| fungi | i can imagine it would disappear into a bright background easily though | 20:51 |
| clarkb | ah with the light mode its just a faint line. I only noticed bceaues the line length changes and it redraws the text area a bit | 20:51 |
| fungi | gonna run out to pick up takeout, should be back by the time the mm3 change merges | 20:52 |
| fungi | also the held gitea upgrade sample seems good to me. i didn't find anything broken which wasn't broken already in production anyway | 20:52 |
| clarkb | cool. I think the only other gitea question is how much do we hate confuguring secrets for unused oauth2 features. I've mostly come to terms with it. Fixing that doesn't seem to be a priority upstream and I don't have time to dig into it right now | 20:53 |
| fungi | yeah, i still need to look through the change | 20:58 |
| opendevreview | Merged opendev/system-config master: Mailman3: check docker-compose stderr not stdout https://review.opendev.org/c/opendev/system-config/+/891903 | 21:20 |
| fungi | looks like the deploy finished | 21:34 |
| fungi | new /var/lib/mailman/web/settings.py update time is 21:21 | 21:35 |
| fungi | uwsgi process start time is 21:21 | 21:35 |
| fungi | worked! | 21:35 |
| fungi | now for the most important part... | 21:35 |
| fungi | refreshed https://lists.zuul-ci.org/archives/ and it finally says "lists.zuul-ci.org" in the top-left corner! | 21:36 |
| fungi | and with that, i'm going to eat | 21:37 |
| fungi | no response on the stuck deleting nodes yet | 21:37 |
| clarkb | woohoo | 21:40 |
| clarkb | I mean for the mailman stuff. We just need patience on the other thing | 21:41 |
| clarkb | for som ereason I thought the zuul archive sstuff was already done? But I guess not you only did opendev then left things so that zuul could be semi automated in transition? | 21:41 |
| clarkb | setup up 23.253.164.4 as etherpad.opendev.org and hit clarkb-test pad for etherpad 1.9.2 testing. It seems to work for me with multiple browsers | 21:48 |
| fungi | ah, no, lists.opendev.org was the primary site name, and with SITE_ID=1 it appeared on all the archive pages for every domain hosted on that instance | 22:05 |
| clarkb | I see and automation fliped it back to 0 and made things work the wa we want? | 22:06 |
| fungi | we needed the SITE_ID=0 config change to merge to fix that (turning on domain-specific behavior), which merged yesterday | 22:06 |
| fungi | but the fact that we weren't restarting the containers meant that wasn't getting used by the running processes | 22:06 |
| clarkb | ah | 22:06 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!