| clarkb | fungi: thank you for getting the meeting agenda sorted out. I'll be there (6am tomorrow relative to my local time) | 01:27 |
|---|---|---|
| clarkb | tonyb: I think the two tasks I have in mind are either A) continuing the server upgrade process (probably looking at mirrors next) or B) beginning the process of adding bookworm python container base images then converting services to those images | 02:10 |
| clarkb | B) may be a bit more interesting since you've gotten some of the server upgrade stuff already | 02:10 |
| tonyb | Yup, I'll try B as the primary task but still look at A as well. | 02:11 |
| tonyb | So we need to build and publish a bookworm-3.11 container and then use that instead of the bullseye-3.11 | 02:12 |
| tonyb | and I guess also update the 3.9 containers to 3.11 | 02:12 |
| clarkb | yup exactly. In the past we've done a 1:1 for the python versions so that we can drop the old debian stuff independent of the old python stuff | 02:12 |
| tonyb | containers in opendev.org/zuul/* should pull from quay.io but others should pull from docker.io | 02:13 |
| tonyb | and all from lines should be fully qualified | 02:13 |
| clarkb | https://opendev.org/opendev/system-config/src/branch/master/zuul.d/docker-images has three files that currnetly define all thos ejobs. I think we can copy them over to three new bookwork files then add the jobs in https://opendev.org/opendev/system-config/src/branch/master/zuul.d/project.yaml#L152-L160 | 02:13 |
| clarkb | ya that sound sright | 02:13 |
| clarkb | oh and here https://opendev.org/opendev/system-config/src/branch/master/zuul.d/project.yaml#L302-L310 | 02:14 |
| clarkb | In the past when I've done this I started out by seeing if I can drop the oldest python version. It may be worthwhile checking if anything is using 3.9 anymore | 02:14 |
| * clarkb looks at codesearch | 02:14 | |
| tonyb | there are a few, ptgbot for example | 02:15 |
| clarkb | https://codesearch.opendev.org/?q=3.9-bullseye&i=nope&literal=nope&files=&excludeFiles=&repos= ya looks like gerrit too | 02:15 |
| tonyb | Okay | 02:15 |
| clarkb | so ya maybe we decouple the two tasks | 02:15 |
| tonyb | Okay | 02:16 |
| clarkb | The real fun is in converting the services to those base images though. Since that will expose you to all the things consuming them | 02:17 |
| clarkb | Most things using containers have decent CI testing though so that should be less adventure and more straightforwatd I hope | 02:17 |
| clarkb | But the first step is the base image update and then we can use speculative testing from there. It is a great benefit when doing this stuff | 02:18 |
| tonyb | Yeah. The whole system design is pretty darn confidence inspiring :) | 02:21 |
| clarkb | tonyb: it is worth calling out how the base images work | 02:22 |
| clarkb | give me a minute to page that in so that I don't say the wrong then then will try to summarize | 02:22 |
| tonyb | Okay | 02:22 |
| tonyb | for users in the zuul project are there any reasons to *NOT* update to python 3.11 (assuming testing passes | 02:23 |
| clarkb | I don't think so | 02:24 |
| tonyb | Okay | 02:25 |
| clarkb | ok so the builder image has all the scripts in it. The `assemble script` basically installs python projects and tracks all of the necessary dependencies and record sthem. It also captures all of the wheels for the dependencies. Then you copy that info to the python-base image and install things using the recorded deps list and wheels. What this does is keeps all build deps on the | 02:26 |
| clarkb | throwaway builder image and reduces the size of the final image based on the base image | 02:26 |
| clarkb | The idea is that the two stage process keeps things minimal while still being accurate/complete | 02:27 |
| tonyb | That makes semse, and matches what I've seen in other places | 02:29 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Add Debian bookworm based python images https://review.opendev.org/c/opendev/system-config/+/888102 | 02:34 |
| tonyb | FWIW, I've started taking notes in https://etherpad.opendev.org/p/opendev-python-updates It's very very raw right now but hopefully it will be helpful next time | 02:37 |
| clarkb | sound sgood | 02:39 |
| clarkb | tonyb: I left a comment about one missed thing but I would wait on test results first | 02:39 |
| clarkb | it won't impact the check jobs | 02:39 |
| tonyb | fixed | 02:45 |
| clarkb | looks like the jobs are passing so you can probably push up the fixed version | 03:11 |
| clarkb | uwsgi depends on the other images so will go last if we want to wait | 03:11 |
| clarkb | doing code review in the heat is fun | 03:21 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Add Debian bookworm based python images https://review.opendev.org/c/opendev/system-config/+/888102 | 03:37 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Update accessbot to use the bookworm container https://review.opendev.org/c/opendev/system-config/+/888106 | 03:37 |
| tonyb | I have fixed the uwsgi issue also | 03:37 |
| tonyb | and 888106 is my best guess as to how to do the accessbot service | 03:37 |
| tonyb | I should read the infra-manual to try and come up with an internal model for which services can/should be grouped together | 03:38 |
| clarkb | tonyb: the other thing that may be helpful for that grouping is the playbooks in system-config because we tend to do roles per service then aggregate them on hosts | 03:38 |
| clarkb | *aggregate them on hosts using playbooks | 03:38 |
| tonyb | Okay. | 03:39 |
| clarkb | the accessbot change lgtm | 03:41 |
| tonyb | With my understanding of the whole speculative container testing, I'm assuming that even though the bookworm cotainers aren't promoted they'll be available to the later acessbot change. | 03:41 |
| clarkb | yes, they should be via the requires here https://review.opendev.org/c/opendev/system-config/+/888106/1/zuul.d/docker-images/accessbot.yaml I think | 03:42 |
| tonyb | based on: https://opendev.org/opendev/system-config/src/branch/master/zuul.d/infra-prod.yaml#L492-L494 I should do accesbot, ircbot and matrix-eavesdrop as one change | 03:48 |
| clarkb | ++ | 03:50 |
| clarkb | though you can still split them up if you prefer | 03:50 |
| clarkb | I don't think this will have a big or any impact on us but good to be aware of https://letsencrypt.org/2023/07/10/cross-sign-expiration.html | 04:25 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Add Debian bookworm based python images https://review.opendev.org/c/opendev/system-config/+/888102 | 05:05 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Update accessbot to use the bookworm container https://review.opendev.org/c/opendev/system-config/+/888106 | 05:05 |
| tonyb | clarkb: 888106 is failing with "manifest for opendevorg/python-base:3.11-bookworm not found: manifest unknown: manifest unknown" https://zuul.opendev.org/t/openstack/build/ab79e98cdd0242649cbc50593e87dae1/log/job-output.txt#723 | 05:27 |
| tonyb | I thought the idea was that it'd pull that from the buildset registry transparently | 05:29 |
| tonyb | is there something missing from the jobs to cause that or have I misuderstood? and we need 888102 to land before the new images are available at all | 05:30 |
| clarkb | hrm I expected that would work. But maybe we are missing something | 05:36 |
| clarkb | provides/requires does the cross buildset plumbing and then the job dependencies does the ordering within a buildset and I thought we had both | 05:37 |
| clarkb | I wonder if we need an explicit depends on for zuul to catch the relationship? except these are a normal git relationship which should be fine | 05:39 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Update accessbot,ircboot and matrix-eavesdrop to bookworm container https://review.opendev.org/c/opendev/system-config/+/888106 | 05:46 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Update to latest tag for Limnoria https://review.opendev.org/c/opendev/system-config/+/888110 | 05:46 |
| opendevreview | Tony Breeds proposed opendev/system-config master: Update ircbot and matrix-eavesdrop to python-3.11 https://review.opendev.org/c/opendev/system-config/+/888111 | 05:46 |
| tonyb | Okay I'll poke around and see if I can figure out what, if anything, is missing | 05:47 |
| tonyb | Once the testing passes that's kinda how I see things going for those services. | 05:48 |
| tonyb | Generally what else needs to happen? | 05:48 |
| * tonyb goes for a short walk | 05:48 | |
| clarkb | I think its basically iterating through and sorting out getting the various services to work on the new platform | 05:48 |
| clarkb | the irc bots are likely to be straightforward so are a good place to start. But something like gerrit might be more complicated | 05:49 |
| tonyb | yeah I figured so too. | 06:07 |
| tonyb | thanks clarkb | 06:07 |
| Clark[m] | And thank you! | 06:13 |
| opendevreview | Moritz Haase proposed zuul/zuul-jobs master: roles/ensure-python: Fix 'python_use_stow' option https://review.opendev.org/c/zuul/zuul-jobs/+/871822 | 06:17 |
| opendevreview | Merged opendev/infra-manual master: Trivial: fix image sizing in creators guide https://review.opendev.org/c/opendev/infra-manual/+/877554 | 13:13 |
| opendevreview | Merged opendev/infra-manual master: PyPI: clarify case where owner can't be removed https://review.opendev.org/c/opendev/infra-manual/+/878240 | 13:13 |
| opendevreview | Merged opendev/zone-opendev.org master: Replace ze04-ze06 https://review.opendev.org/c/opendev/zone-opendev.org/+/885514 | 14:17 |
| corvus | removed the fact cache for ze04-6 and approved the system-config change to replace them | 14:43 |
| fungi | thanks! | 14:50 |
| opendevreview | Merged opendev/system-config master: Replace ze04-ze06 https://review.opendev.org/c/opendev/system-config/+/885509 | 14:54 |
| slittle1 | Looking for a little help on starlingx-SDO-rv-service-core group. I'm a member, but for this one group I don't seem to have the power to add others. Not sure why. | 16:54 |
| fungi | looking | 16:54 |
| fungi | slittle1: somehow the group owner got set to a specific user rather than to the group itself | 16:56 |
| fungi | looks like it was done by user "Poornima Y N" | 16:56 |
| fungi | (who is one of the group's members) | 16:57 |
| fungi | i'll fix it back but it will take a moment | 16:57 |
| slittle1 | ok, well we need to fix that. Intel is no longer contributing to StarlingX in this area. I need to assign a few windriver guys to maintain this git in their absence. | 16:58 |
| fungi | slittle1: refresh and you should have control of it again | 16:59 |
| fungi | looks like it was probably done just after the group was created back in late 2020 and has just been that way ever since | 17:00 |
| corvus | #status log started zuul on replacement ze04-ze06 servers | 17:38 |
| opendevstatus | corvus: finished logging | 17:38 |
| corvus | so far they appear to be running normally | 17:38 |
| corvus | the first of my spot check builds completed successfully | 17:52 |
| corvus | fungi: i'm getting a lot of NDRs from lists; you seeing those? | 17:55 |
| fungi | i saw some earlier i think, someone spoofing abuse@lists.o.o to send messages to other addresses at lists.o.o (mainly mailing lists) | 17:59 |
| corvus | this is different; errors from django | 17:59 |
| fungi | oh, i'll check again in a sec | 18:00 |
| corvus | django's emailing root@localhost which is bouncing, so an ndr goes to root@lists.opendev.org which goes out to us | 18:00 |
| fungi | corvus: sorry, was trying to join a conference call but found it. one of the downsides on filtering those into a separate maildir is that i forget to check it regularly | 18:08 |
| corvus | i think this is a new behavior starting yesterday | 18:09 |
| fungi | looks like django doesn't have a 429 error page template, hence the traceback. the cause looks like someone trying to brute-force account signups | 18:11 |
| clarkb | fungi: corvus the address mm3 is configured to send those emails to is configurable if we need ot change that. But ya the underlyng error is the lack of a 429 template to be served | 18:48 |
| fungi | clarkb: oh, while the tc call is still going on, what are our resource and maintenance concerns with having an ever growing number of branches left open on projects? zuul scanning them for configuration (and them possibly containing broken configs), but what else? | 18:51 |
| fungi | also still having periodic jobs running (and perpetually failing) for all those branches | 18:54 |
| fungi | also i guess we need to be able to delete resources | 18:54 |
| fungi | images for distro versions contemporary with the original releases those branches are based on | 18:55 |
| clarkb | ya test resources that a contemporary is a big one | 18:57 |
| clarkb | people pushing changes to the branches and being completely ignored | 18:57 |
| clarkb | increased zuul configuration resolution times | 18:57 |
| clarkb | periodic jobs | 18:58 |
| clarkb | I think you captured the bulk of it | 18:58 |
| fungi | thanks | 18:59 |
| fungi | looks like i received 53 of those django traceback bounces between 08:51:24 and 18:52:58 so it's been an hour since the last one, but given that's a 10-hour span i don't know whether it's subsided or just bursty | 19:51 |
| corvus | tonyb clarkb it does look like there are no zuul artifacts listed in the inventory vars: https://zuul.opendev.org/t/openstack/build/ab79e98cdd0242649cbc50593e87dae1/log/zuul-info/inventory.yaml | 19:57 |
| clarkb | corvus: the change is a git child to the parent that adds the new images and has requires/provides set. IIRC that is what manipulates the artifacts across buildsets? | 20:00 |
| clarkb | but these are new images | 20:01 |
| corvus | yeah still pulling on threads | 20:01 |
| clarkb | ack thanks | 20:01 |
| corvus | clarkb: tonyb commented on the problem in https://review.opendev.org/888102 | 20:04 |
| corvus | basically looks ilke a copypasta typo | 20:04 |
| corvus | on the provides | 20:04 |
| fungi | wow, i would not have spotted that | 20:07 |
| tonyb | Ahh okay, and it's also missing from the existing bullseye ones so I missed it there | 20:08 |
| * tonyb will fix both | 20:08 | |
| corvus | yeah, can be tough to spot those once an image gets uploaded. we should make a tool. | 20:08 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!