| fungi | corvus: will do. lots of those were old old, some like 2019 old | 00:42 |
|---|---|---|
| fungi | now we have a baseline to see what/when is still happening | 00:43 |
| opendevreview | Jay Faulkner proposed openstack/diskimage-builder master: DNM: Testing Gentoo CI job against merged-usr profile https://review.opendev.org/c/openstack/diskimage-builder/+/892627 | 01:07 |
| opendevreview | Guillaume Boutry proposed openstack/project-config master: Add OpenStack K8S Magnum charms https://review.opendev.org/c/openstack/project-config/+/892650 | 08:57 |
| TheJulia | good morning | 13:29 |
| TheJulia | wrong channel, but still good to say :) | 13:30 |
| fungi | it's appropriate for any channel, good morning to you too! | 13:40 |
| *** dviroel_ is now known as dviroel | 14:03 | |
| clarkb | fungi: not sure what your day looks like. I've got a good stretch until this afternoon where I've got school prep stuff to do. Happy to try and upgrade gitea and or mailman3 or whatever if you think we're in a good spot for it | 14:57 |
| fungi | clarkb: yeah, i'm semi-distracted by painters at the moment but expect they'll be wrapping up for the day here shortly | 15:03 |
| *** TheJulia is now known as confused | 15:34 | |
| *** confused is now known as TheJulia | 15:34 | |
| opendevreview | Merged openstack/project-config master: Add OpenStack K8S Magnum charms https://review.opendev.org/c/openstack/project-config/+/892650 | 16:18 |
| clarkb | fungi: because I have a bad habit of rerevieing things: I've just (re)discovered a todo I left myself in https://review.opendev.org/c/opendev/system-config/+/886993/16/playbooks/roles/gitea/templates/app.ini.j2#4 which is to determine if that work path value matches what we've got on the current nodes since we aren't starting from scratch there | 16:30 |
| clarkb | https://docs.gitea.com/next/administration/config-cheat-sheet says there are a number of ways this can be set so I've got to cross check those methods against prod to see if the paths are the same on 1.19 as they will be in 1.20 with that change | 16:31 |
| fungi | aha | 16:34 |
| clarkb | I suspect it is fine and we're using a default that is consistent across the board but better safe than sorry (we could upgrade into empty repos) | 16:35 |
| clarkb | I haven't tracked it down in our config management yet. I will need to ssh into the hosts and inspect the running systems | 16:35 |
| fungi | looks like we map /var/gitea/data in as /data and there is a /var/gitea/data/gitea folder | 16:36 |
| fungi | so in theory /data/gitea inside the container ends up there | 16:36 |
| fungi | other than existing, what does appropriateness entail? | 16:37 |
| fungi | looks like it's all owned uid=1000 gid=1000 which is what we tell the gitea-web container to use, so its processes should be able to write there | 16:38 |
| fungi | anything else you think needs to be checked? | 16:38 |
| fungi | it's on the rootfs, but at least on gitea09 there's 113gb free | 16:39 |
| fungi | clarkb: other than revising the change to drop the todo comment, i guess it's fine? | 16:40 |
| clarkb | sorry went to grab my ssh keys to check myself and got distracted | 16:57 |
| clarkb | fungi: I think the main thing is to confirm the current 1.19 installs have work path set to /data/gitea and/or that the dir strucutures on the held node match our prod nodes | 16:57 |
| clarkb | fungi: it looks like the held gitea node logged the work path on startup. The logs for gitea14 seemed to have rolled over since last restart but I'm trying to double check that. We might be able to restart one of the prod nodes to have them log this info and confirm for us | 17:02 |
| clarkb | ya logs seemed to have all rolled over. | 17:03 |
| clarkb | fungi: look in the /proc/$PID/environ "file" for both gitea14 and the held gitea99. GITEA_WORK_DIR differs. It is /data/gitea on the held 1.20 node and /app/gitea in prod :/ | 17:05 |
| clarkb | fungi: ok I did some prior debugging of this too https://review.opendev.org/c/opendev/system-config/+/886993/16//COMMIT_MSG#37 | 17:08 |
| clarkb | wow gitea's documentation documents APP_DATA_PATH twice with two different sets of information | 17:11 |
| fungi | huh. odd | 17:14 |
| clarkb | I'm understanding this a bit better. Basically WORK_PATH is the default root that everything else is mounted to when using relative paths. Our config seems to use rooted paths for everything already in /data/gitea or /data/git | 17:14 |
| clarkb | the only thing in /app/gitea in prod is the gitea binary itself | 17:15 |
| clarkb | so on 1.19 we haven't created any new directories relative to the WORK_PATH | 17:15 |
| clarkb | I'll corss check the contents of /data/gitea on 1.20 with 1.19 to see if 1.20 is creating any new paths | 17:15 |
| clarkb | gitea 1.19 has /var/gitea/data/gitea/packages and 1.20 does not. gitea 1.20 has /var/gitea/data/gitea/sessions and 1.19 does not | 17:17 |
| clarkb | for the second item both 1.19 and 1.20 app.ini configs set session.PROVIDER_CONFIG = /data/gitea/sessions | 17:18 |
| clarkb | I'm guessing that is created on demand when sessions are created then? | 17:19 |
| clarkb | the packages dir is empty on 1.19. We disable packages on both 1.19 and 1.20. I think this may be a side effect of actually disabling more stuff | 17:21 |
| fungi | i don't see a /data/gitea/sessions on gitea09 | 17:21 |
| clarkb | fungi: ya I think gitea99 has one because I actually logged in as root | 17:21 |
| clarkb | and its probably created on demand when we login. We can test that by logging into gitea09 and seeing if we get a sessions dir | 17:21 |
| fungi | aha, so user login sessions to the webui (and not authenticated git push) | 17:22 |
| clarkb | I suspect both the sessions dir and packages dirs are fine. sessions beacuse we explciitly configure the path the same on both 1.19 and 1.20 and packages because we disable it and the contents are empty in 1.19 so even if it moves later that won't really affect us | 17:22 |
| clarkb | all that to say I think we are fine. If we had content in /app/gitea/ that was not the gitea binary we would be hurtying | 17:22 |
| clarkb | but we don't so yay | 17:22 |
| clarkb | and all the content in /var/gitea/data is bind mounted to /data so everything under that including /data/gitea and /data/git is preserved and that doesn't change | 17:24 |
| clarkb | all that to have a really long winded way of saying: I think this doesn't affect us because we haven't been relying on the default of appending relative paths to the WORK_DIR. Instead every subcomponent path has been explicitly stuck under /data/git or /data/gitea with fully rooted paths. The default prefix management being changed doesn't impact us as a result | 17:25 |
| clarkb | if that makes sense to you I think we can proceed. I'm happy to remove the todo in the app.ini file first if you prefer as well | 17:26 |
| clarkb | let me know if that all makes sense and what you think the best next step is | 17:29 |
| fungi | looks that way, yes. happy to approve as-is and then remove the todo comment separately | 17:33 |
| fungi | i can be around for a while still to check on the deployment | 17:33 |
| clarkb | that works for me. I too can be around if we want ot approve it now | 17:33 |
| clarkb | do you want to +A or should I? | 17:33 |
| fungi | i've approved it now | 17:35 |
| clarkb | you might also want to double check the secret was added on bridge. But I'm pretty confident I did that correctly :) | 17:35 |
| opendevreview | James E. Blair proposed opendev/system-config master: Update gerrit image to bookworm https://review.opendev.org/c/opendev/system-config/+/892699 | 17:38 |
| opendevreview | James E. Blair proposed opendev/system-config master: Update hound image to bookworm https://review.opendev.org/c/opendev/system-config/+/892700 | 17:40 |
| opendevreview | James E. Blair proposed opendev/system-config master: Update jinja-init image to bookworm https://review.opendev.org/c/opendev/system-config/+/892701 | 17:44 |
| opendevreview | James E. Blair proposed opendev/system-config master: Update gerrit image to bookworm https://review.opendev.org/c/opendev/system-config/+/892699 | 17:45 |
| opendevreview | James E. Blair proposed opendev/system-config master: Update zookeeper-statsd image to bookworm https://review.opendev.org/c/opendev/system-config/+/892702 | 17:47 |
| fungi | will do | 17:51 |
| fungi | yeah, i see a gitea_oauth2_jwt_secret set in the gitea group vars | 17:53 |
| fungi | which matches the variable name in the template | 17:53 |
| clarkb | in theory the value itself shouldnt matter | 17:53 |
| fungi | right, since we're just setting but not consuming it anywhere | 17:56 |
| opendevreview | James E. Blair proposed opendev/system-config master: Test hound image build https://review.opendev.org/c/opendev/system-config/+/892703 | 18:07 |
| opendevreview | James E. Blair proposed opendev/system-config master: Update gerrit image to bookworm https://review.opendev.org/c/opendev/system-config/+/892699 | 18:16 |
| opendevreview | James E. Blair proposed opendev/system-config master: Update gerrit image to bookworm https://review.opendev.org/c/opendev/system-config/+/892699 | 18:17 |
| opendevreview | James E. Blair proposed opendev/system-config master: Update jinja-init image to bookworm https://review.opendev.org/c/opendev/system-config/+/892701 | 18:18 |
| opendevreview | James E. Blair proposed opendev/system-config master: Update zookeeper-statsd image to bookworm https://review.opendev.org/c/opendev/system-config/+/892702 | 18:19 |
| opendevreview | James E. Blair proposed opendev/system-config master: Update hound image to bookworm https://review.opendev.org/c/opendev/system-config/+/892700 | 18:31 |
| clarkb | fungi: I'm not seeing starlingx matrix room logs on https://meetings.opendev.org/irclogs/ yet. Maybe nothing has happened since the bot joined up? | 18:46 |
| clarkb | but that may need further debugging | 18:46 |
| fungi | yeah, i wonder if it requires a restart | 18:56 |
| clarkb | the gitea change should merge any minute now if it isn't going to hit its timeout | 18:56 |
| fungi | root 1376910 0.0 1.0 138588 20468 ? Ssl Jul26 12:22 /usr/local/bin/python /usr/local/bin/eavesdrop | 18:58 |
| clarkb | probably a good hunch on needing to restart it then | 18:58 |
| opendevreview | Merged opendev/system-config master: Update to Gitea 1.20 https://review.opendev.org/c/opendev/system-config/+/886993 | 18:58 |
| fungi | -rw-r--r-- 1 root root 1203 Aug 23 15:26 /var/lib/matrix-eavesdrop/config/config.yaml | 18:59 |
| clarkb | here we go | 18:59 |
| fungi | so unless the daemon reads file updates automatically or gets soft reloaded somewhere, yeah i think we're missing a restart handler | 18:59 |
| clarkb | the gitea deploy job just started. It should update gitea09 first. I'm watching it | 19:00 |
| clarkb | https://gitea09.opendev.org:3081/opendev/system-config loads for me and reports the version I expect to see | 19:02 |
| clarkb | I'm able to clone the repo from that url as well | 19:02 |
| clarkb | I think the other major thing to check is replication once we've got things merging (we can also check refs/changes/ updates but not as easily) | 19:03 |
| clarkb | The mobile theming thing seems to work now that we have a real cert too | 19:04 |
| clarkb | (this was expected but difficult to test until we upgraded) | 19:04 |
| clarkb | 9,10, and 11 are updated at this point. And things look fine from here | 19:05 |
| clarkb | we are waiting for containers to stop on 12 | 19:06 |
| clarkb | one of three has stopped but the other two have not yet | 19:07 |
| clarkb | there they go | 19:07 |
| fungi | ayup | 19:09 |
| fungi | all of the above are working for me | 19:10 |
| clarkb | we are done with all but 14 now | 19:10 |
| clarkb | all six servers are done and the job reports success | 19:14 |
| clarkb | the service itself seems to work for me. Let's just keep an eye out for replication happenign | 19:14 |
| fungi | yeah, if i spot something merging soon i'll check behind it | 19:15 |
| clarkb | sounds good. Since nothing is obviously wrong I'm going to go eat a sandwich for lunch | 19:15 |
| fungi | i too am going to pop away for some refreshments and return soonish | 19:17 |
| opendevreview | Harry Kominos proposed openstack/diskimage-builder master: feat: Add new fail2ban elemenent https://review.opendev.org/c/openstack/diskimage-builder/+/892541 | 19:23 |
| fungi | okay, heading out, bbiab | 19:23 |
| opendevreview | James E. Blair proposed opendev/system-config master: Restart matrix-eavesdrop when config changes https://review.opendev.org/c/opendev/system-config/+/892713 | 19:59 |
| opendevreview | James E. Blair proposed opendev/system-config master: Update gerrit image to bookworm https://review.opendev.org/c/opendev/system-config/+/892699 | 20:01 |
| clarkb | corvus: there was some weird gerrit + java 17 thing I made a fix for but we decided to stick with 11. I think 11 is still available in bookworm too. Let me dig it up | 20:08 |
| clarkb | corvus: https://review.opendev.org/c/opendev/system-config/+/870877 | 20:08 |
| clarkb | its possible they fixed that problem too | 20:08 |
| clarkb | they did fix it. Neat https://issues.gerritcodereview.com/issues/40014855 | 20:09 |
| corvus | clarkb: okay cool, then my latest rev might work, let's see! i'm not married to any of this, just trying to see what's easy or needs more work :) | 20:13 |
| clarkb | corvus: reading the fix on that makes me wonder how helpful the java 17 chagne was. Basically you can't access a protected method directly when using reflection anymore. So you make a subclass that overrides the method and makes it public instead | 20:22 |
| clarkb | I don't do language design so I'm probably missing how it is better to make people do ^ than just do it directly | 20:22 |
| corvus | i guess more explicit is better? but yeah, there's probably more subtle stuff going on | 20:24 |
| fungi | it's been quiet. none of the channels i'm lurking in (and there are a lot) have gerritbot announcements of changes merged after the gitea upgrades | 20:30 |
| fungi | maybe that's a sign i should review some things | 20:30 |
| fungi | clarkb: 892700 looks straightforward and fairly low impact if it goes wrong somehow, care for being a second pair of eyes on it? | 20:32 |
| clarkb | done | 20:35 |
| fungi | i'm going to try to whip up a restart handler for the matrix-eavesdrop bot and notify it on config changes | 20:36 |
| clarkb | did we want to manually restart it to confirm that is the issue? | 20:36 |
| corvus | fungi: see https://review.opendev.org/892713 | 20:36 |
| corvus | maybe that will solve 2 problems at once :) | 20:36 |
| fungi | corvus: wow, it's like you traveled back in time and wrote the thing i was about to make. or maybe i'm just that far behind on reviews. yeah, probably that actually | 20:37 |
| corvus | it does not reload its configuration under any circumstances, so i strongly suspect it's the issue. i think restarting it to effectuate a fix is a good idea, or you could tag on a no-nop config file change after 892713 if you want to see it in action. | 20:37 |
| corvus | fungi: i ate lunch at my desk | 20:37 |
| corvus | there may be spaghetti sauce on that change, watch out | 20:37 |
| fungi | what i did to test a similar recent change to the mailman3 containers was manually roll back the configuration and let it get reapplied to trigger the handler | 20:38 |
| corvus | oh that's a good idea | 20:38 |
| fungi | even cheaper, i stuck a comment line at the top of /var/lib/matrix-eavesdrop/config/config.yaml on the server so ansible will see a difference and replace it | 20:40 |
| fungi | so once that merges and deploys here shortly, we should ideally see it restart the service | 20:41 |
| clarkb | and then we'll just need people to use the channels :) though it sounds like people are filtering in finally | 20:44 |
| fungi | oh, for starlingx? yeah there's been conversations in about half those channels in the past day | 20:44 |
| fungi | it's picking up steam nicely | 20:44 |
| clarkb | excellent | 20:44 |
| clarkb | fungi: for mm3 we run db migrations everytime the container starts iirc. Is that hte only upgrade step that needs to be done? | 20:45 |
| fungi | i had a lengthy talk in #starlingx-test:opendev.org about replicating zuul jobs locally, pointing out where the images we build can be found, our configuration for how we build images, what other issues to look out for... | 20:45 |
| clarkb | (also I think thos emigrations happen automatically) | 20:45 |
| fungi | clarkb: yes, db migrations are the only transformations of data. everything else is config changes we merge in ansible or updates to the software/defaults in docker image building | 20:46 |
| clarkb | ok cool. I have to pop out for school things in about 45 minutes, not sure if we want ot proceed with upgrading mm3 today. Happy to help if I can though | 20:47 |
| fungi | i'm around to keep an eye on the deployment if you approve it | 20:48 |
| clarkb | fungi: do we want to do that now or would you prefer waiting for tommorrow (your day is going to wind down soon too I think?) | 20:49 |
| fungi | i'm fine waiting for tomorrow. it's still possible we'll find something not quite right with today's gitea upgrade so spacing them out makes sense for our own peace of mind | 20:49 |
| clarkb | sounds good | 20:50 |
| fungi | for the record, there is a mailman 3 upgrade guide, but there have been no special steps since 3.3.1: https://docs.mailman3.org/en/latest/upgrade-guide.html | 20:52 |
| clarkb | good to know | 20:53 |
| fungi | and the standard migrate/compress/collectstatic/compilemessages steps are handled on container start | 20:53 |
| corvus | the gerrit image did actually pass tests with jdk17 and bookworm | 20:55 |
| fungi | https://opendev.org/opendev/system-config/src/branch/master/docker/mailman/web/docker-entrypoint.sh#L111-L126 is where we do the mailman automated upgrade steps | 20:56 |
| fungi | so everything in the upgrade guide seems to me to be covered | 20:57 |
| fungi | Python 3.11.5, 3.10.13, 3.9.18, and 3.8.18 just released, including some security fixes | 21:01 |
| fungi | https://discuss.python.org/t/32254 | 21:01 |
| * fungi takes a recompiling break | 21:01 | |
| opendevreview | Merged opendev/system-config master: Update hound image to bookworm https://review.opendev.org/c/opendev/system-config/+/892700 | 21:02 |
| fungi | okay let's see if that replicated | 21:03 |
| corvus | python:3.11-bullseye still appears to be .4 | 21:03 |
| clarkb | they usually rebuild within a day | 21:03 |
| fungi | c2ea42a86..1c5bee394 master -> origin/master | 21:03 |
| clarkb | the docker hub library images I mean | 21:03 |
| clarkb | https://opendev.org/opendev/system-config/commit/1c5bee394cc87e3a24e8150127b3c19a3101578d yup its there in gitea | 21:04 |
| fungi | head of master i just remote updated is Merge "Update hound image to bookworm" | 21:04 |
| fungi | so yes, seems to have replicated fine | 21:04 |
| opendevreview | Merged opendev/system-config master: Restart matrix-eavesdrop when config changes https://review.opendev.org/c/opendev/system-config/+/892713 | 21:04 |
| fungi | and after another remote update it's Merge "Restart matrix-eavesdrop when config changes" | 21:05 |
| fungi | now to see if we get a service restart when that deploys | 21:05 |
| fungi | my dummy comment is still present at the top of /var/lib/matrix-eavesdrop/config/config.yaml for the moment | 21:05 |
| clarkb | fungi: jobs are waiting behind the hourly deployment jobs | 21:06 |
| clarkb | we run eavesdrop hourl so the hourly job will do it and it may not have the update to restart containers in it? | 21:16 |
| clarkb | will do it -> update the config file you modified | 21:16 |
| clarkb | you might need to modify that file again if it doesn't restart things | 21:16 |
| clarkb | looks like it restarted 10 minutes ago | 21:28 |
| fungi | yeah, it worked | 21:28 |
| clarkb | and there are log files. THough the cron that spits out the nicer format stuff hasn't run yet | 21:28 |
| fungi | [and there was much rejoicing] | 21:29 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!