| ianw | anyway, before all this nb03 was borked again. in a bit i'll try to see if i can get anything else there | 00:08 |
|---|---|---|
| ianw | i think it is leaked loop images | 01:04 |
| ianw | root@nb03:/tmp# losetup --list | wc -l | 01:04 |
| ianw | 9 | 01:04 |
| ianw | that ain't right | 01:04 |
| ianw | # mount /dev/loop7 /tmp/mnt | 01:07 |
| ianw | mount: /tmp/mnt: /dev/loop7 already mounted or mount point busy. | 01:07 |
| ianw | root@nb03:/opt/dib_tmp/dib_image.7BsVnO2D# mount | grep loop7 | 01:07 |
| ianw | ... nothing | 01:07 |
| ianw | so it's mounted/busy, but not mounted?! | 01:07 |
| ianw | ahh, hrm, /dev/mapper | 01:10 |
| ykarel | ianw, frickler can you please review https://review.opendev.org/c/zuul/zuul-jobs/+/846206 and https://review.opendev.org/c/zuul/zuul-jobs/+/846201 | 04:01 |
| opendevreview | Merged zuul/zuul-jobs master: Fix two testing problems https://review.opendev.org/c/zuul/zuul-jobs/+/846206 | 04:23 |
| *** pojadhav is now known as pojadhav|ruck | 04:25 | |
| ykarel | ianw, can you also check other one too please | 04:26 |
| ykarel | https://review.opendev.org/c/zuul/zuul-jobs/+/846201 | 04:26 |
| ianw | ykarel: sorry, that one doesn't make any sense to me. that variable has nothing to do with venv installation | 04:35 |
| ianw | i haven't quite figured out how dib is dying and leaving the dm devices around. it is the centos 9-stream build. i think it has to do with our current mirror woes | 04:43 |
| ianw | i'm going to clear out and reboot it again | 04:43 |
| ykarel | ianw, i replied, i agree with you on this | 04:43 |
| ykarel | but can we merge this to resolve issue with translation update jobs | 04:43 |
| ykarel | and solve the concerns with follow ups as i fixed what was wrong with the installations | 04:44 |
| ykarel | seems the var was just reused for venv package installation instead of introducing some duplicate | 04:45 |
| ianw | it seems like this came in via -> https://opendev.org/openstack/project-config/commit/c7d42980c649273eacf9361cfe20e5500ac3b6e7 | 04:47 |
| ianw | or somewhere around there | 04:47 |
| ykarel | yes right | 04:47 |
| ianw | i actually feel like reverting this change in zuul-jobs might be the most correct thing, rather than trying to fix it. whatever it breaks, should be fixed another way | 04:47 |
| ykarel | actually this seems to be mess there were series of patch to fix propose-updates job | 04:48 |
| ykarel | and then that broke translation-updates job which inherit from ^ as that have to run on bionic | 04:48 |
| ykarel | frickler, may have more context around orignal ones, /me just checked translation ones | 04:49 |
| opendevreview | Ian Wienand proposed zuul/zuul-jobs master: Revert "Install venv for all platforms in ensure-pip" https://review.opendev.org/c/zuul/zuul-jobs/+/846248 | 04:52 |
| ykarel | k Thanks, let's see | 04:53 |
| ianw | i would prefer we do something like ^ and then go back and re-evaluate what's going on | 04:53 |
| ykarel | k sure | 04:53 |
| ianw | i get the feeling that this comes down to having packaged python3.8 and python3.9 installed on the same system, something that ensure-pip was not written to understand | 04:53 |
| ykarel | likely this will break jobs those were using it | 04:55 |
| ykarel | where default python is something else | 04:55 |
| ianw | it might be that ensure-pip needs another argument like ensure_pip_python_package_versions: [python3.8, python3.9] to pull in all the right packages | 04:55 |
| ianw | but that would be orthogonal to the ensure_pip_from_upstream* arguments (it's *not* upstream, it's packaged) | 04:56 |
| * ykarel agrees | 04:57 | |
| ianw | i am in no way defending anything about ensure-pip. it has grown from a mess of mixed python2/3 systems, and mixing in installing upstream pip globally and using pip packages locally | 04:58 |
| ianw | all we can try and do is not make it any worse | 04:58 |
| ykarel | ianw, i think https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ensure-python/tasks/main.yaml would be more suitable | 05:04 |
| ykarel | for installinv -venv packages | 05:04 |
| ianw | there is probably an argument for doing it in there, in that venv could be considered part of any working python system so pulling the package in is reasonable | 05:06 |
| ianw | and also consistency, pretty sure rh distros install it by default with python, only debuntu splits it into the separate package | 05:07 |
| ykarel | yes ^ true for ubuntu | 05:07 |
| ykarel | so would need to install explicitly in this case | 05:07 |
| ianw | it's interesting that role uses the variable "python_version". that feels a little generic as a name to me | 05:09 |
| *** chkumar|rover is now known as chandankumar | 05:10 | |
| *** elodilles_pto is now known as elodilles | 06:46 | |
| *** jpena|off is now known as jpena | 07:01 | |
| *** arxcruz is now known as arxcruz|rover | 07:08 | |
| frickler | I must admit I focused on getting reqs updates working and didn't care much about translations. maybe split the common parent job into different ones now that they diverge so much | 07:31 |
| frickler | but then the zanata setup is doomed anyway | 07:31 |
| frickler | otoh splitting reqs updates into one job per python version and a finel merge job would be a good project, too | 07:32 |
| frickler | the job needs python3.8 + python3.9 installed, which is where the current mess started | 07:35 |
| frickler | #status log pushed openstack/requirements 846277,1 to gate in order to unblock neutron | 07:44 |
| opendevstatus | frickler: finished logging | 07:44 |
| frickler | ralonsoh: prometheanfire: ^^ | 07:44 |
| *** ysandeep|out is now known as ysandeep|afk | 08:10 | |
| *** rlandy|out is now known as rlandy | 10:30 | |
| *** ysandeep|afk is now known as ysandeep | 11:09 | |
| *** ysandeep is now known as ysandeep|afk | 11:47 | |
| fungi | ianw: if it helps, there's now a python3-full virtual package (and corresponding python3.9-full, et cetera) in debian as of bullseye and ubuntu as of impish (so also jammy), which does include -venv et cetera | 12:09 |
| fungi | doesn't solve bionic jobs though, of course | 12:13 |
| Clark[m] | ianw: ykarel: fungi: I don't think we can merge that revert without potentially breaking users. This being zuul-jobs we try to avoid doing that. We should fix the code in a way that doesn't change expected behavior. Then we can send email about potential breaking cleanups and land them after a waiting period | 12:15 |
| Clark[m] | I think that looks like ykarel's change to fix the loop, a follow-up to ensure-python to do similar, email telling people ensure-python needs to do it, then removing the code from ensure-pip later. But not upfront removal | 12:16 |
| ykarel | ^ sounds good to me | 12:17 |
| *** dviroel|out is now known as dviroel | 13:00 | |
| *** ysandeep|afk is now known as ysandeep | 13:16 | |
| BlaisePabon[m]1 | IDK much about zuul, but I am really into python.... (full message at https://matrix.org/_matrix/media/r0/download/matrix.org/DxqyHTfTrRbIzkvKAKGqzwiN) | 13:18 |
| *** pojadhav|ruck is now known as pojadhav|dinner | 13:18 | |
| frickler | can one instruct the matrix bridge not to hyperlink long messages? I really don't like having to use a http client every time someone send something with "full message at https://matrix.org/..." | 13:21 |
| ianw | Clark[m]: i don't want to say too much about the original as i'm missing the context of what happened, but it didn't update any documented behaviour -- if anything it broke the documented behaviour of the variable it's looping on because that was never supposed to be a key for venv packages | 13:31 |
| ianw | so imo it's more of a bug fix than a deprecation. that said, if yourself or others want to swizzle things around i won't argue. i do think now we've taken a closer look it's important we rework it, though | 13:32 |
| Clark[m] | Whether or not it is documented I think anyone relying on the -venv package being present for a single interpreter would break with that change. Multiple interpreters never worked due to the install failure | 13:36 |
| opendevreview | Lajos Katona proposed openstack/project-config master: Add github svinota/pyroute2 to project list https://review.opendev.org/c/openstack/project-config/+/846364 | 13:37 |
| Clark[m] | Blaise Pabon @blaisepabon:matrix.org: in many cases we're actually interested in the test platform as a whole including the shipped python. Using a separate source may make sense for some jobs but not generally. | 13:37 |
| Clark[m] | frickler: I'm not aware of one. I try to make IRC friendly messages when communicating via the bridge, but that may not be as apparent to users who don't see the IRC side | 13:38 |
| *** pojadhav|dinner is now known as pojadhav|ruck | 13:40 | |
| frickler | maybe we can make a note somewhere in our IRC docs. at least then I could point people at that ;) | 13:40 |
| frickler | Clark[m]: also note how the reply you sent has the matrix-side name of the user | 13:41 |
| ianw | Clark[m]: that argument only does something if ensure_pip_from_upstream=True. i agree that this never installed anything but "python3-venv". making ensure_pip_from_upstream_interpreters do anything without ensure_pip_from_upstream=True just ... doesn't make much sense | 13:43 |
| Clark[m] | frickler: yes because I know that user is on matrix, but when I highlighted you I used your IRC nick instead. It's mental overhead and I'm not sure there is a single best approach but I try | 13:44 |
| Clark[m] | ianw: I agree it should change, but it was added by users and I worry if we just change it they will break. Mostly just saying we should fix it in the more graceful method we try to use with zuul-jobs sending a warning and leaving some time before the change happens | 13:46 |
| Clark[m] | ianw: I wonder if https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ensure-pip/tasks/main.yaml#L79 is why it was set in that role | 13:50 |
| ianw | i'm sure there is context ... sorry getting a bit late for me to dig in. just we're committing "fixes" to this to "fix" ... what exactly? i don't think we know if the fix we're committing breaks anyone either, as it's not documented | 13:53 |
| ianw | this is why i feel like it's a grey area for deprecation. i agree with the overall principle of not pulling things out of zuul-jobs. but this doesn't feel like a working feature we're retiring | 13:54 |
| Clark[m] | Ya I think ykarel's change should maybe shift to ensure-python and install the -venv package for the requested versions there. Then separately deprecate and remove the block you are trying to remove | 13:54 |
| ianw | but as i said, also not going to argue if it gets rebased, etc. | 13:54 |
| ianw | https://pagure.io/centos-infra/issue/814 has had a couple of updates on the corrupt mirror package | 13:54 |
| Clark[m] | And ya it's late Friday for you. I don't think this is urgent enough to demand attention right now :) | 13:55 |
| ianw | this has vibes of people pulling packages on pypi. in the same way you can't pull a package, you have to release a new one, you can't just replace a package | 13:55 |
| ianw | yes, turning in now. just saw Penn & Teller, was a good show! | 13:56 |
| ianw | and only about 3 years or so after they announced they were coming to .au too :) | 13:56 |
| Clark[m] | Wow, sounds like fun! | 13:58 |
| Clark[m] | ykarel: another option on the consuming job side is to update your venv command to be python3 -m venv -p python3.x to select the right python version that way | 13:59 |
| Clark[m] | ykarel: that might be the easiest thing right now while we sort out zuul-jobs? | 13:59 |
| * ykarel just back, reading ^ | 14:00 | |
| *** dviroel is now known as dviroel|pto | 14:03 | |
| *** ysandeep is now known as ysandeep|out | 14:04 | |
| opendevreview | Merged openstack/project-config master: Add github svinota/pyroute2 to project list https://review.opendev.org/c/openstack/project-config/+/846364 | 14:05 |
| ykarel | Clark[m], if i try that locally it says venv: error: unrecognized arguments: -p | 14:06 |
| ykarel | on ubuntu focal | 14:06 |
| ykarel | Clark[m], just to fix on job side we can move https://opendev.org/openstack/project-config/src/branch/master/playbooks/proposal/pre.yaml#L8 | 14:10 |
| ykarel | to job definition, and override that in translation-update child jobs | 14:10 |
| ykarel | if that works for you? | 14:10 |
| Clark[m] | ykarel: it may need to be the virtualenv command to use the python version selector flag. But I still think that is a good approach. Basically just use what is there to configure a virtualenv how you want it | 14:16 |
| Clark[m] | Since I think we have those tools available it's just the interconnected roles and bars that aren't quite right | 14:17 |
| ykarel | Clark[m], okk that should work if virtualenv is installed | 14:19 |
| ykarel | me prepares patch | 14:19 |
| opendevreview | yatin proposed openstack/project-config master: Use virtualenv in translation update jobs https://review.opendev.org/c/openstack/project-config/+/846390 | 14:26 |
| *** pojadhav|ruck is now known as pojadhav|out | 14:26 | |
| ykarel | Clark[m], fungi done ^ | 14:27 |
| ykarel | can test in https://review.opendev.org/c/openstack/neutron/+/837454 once merged | 14:27 |
| ykarel | due to trusted not easy to test these before | 14:28 |
| Clark[m] | Lgtm. I'm about to do a school run so can't +2 properly. Will do that when I get back if it hasn't landed yet | 14:29 |
| ykarel | sure frickler if you can please ^ | 14:33 |
| *** marios is now known as marios|out | 15:25 | |
| opendevreview | Merged openstack/project-config master: Use virtualenv in translation update jobs https://review.opendev.org/c/openstack/project-config/+/846390 | 15:44 |
| opendevreview | Merged zuul/zuul-jobs master: Add the post-reboot-tasks role https://review.opendev.org/c/zuul/zuul-jobs/+/844704 | 15:46 |
| clarkb | two more zuul executors to restart, then the mergers then the schedulers | 16:03 |
| clarkb | this seems slower than the last few times. I guess it is very dependent on load | 16:03 |
| *** jpena is now known as jpena|off | 16:15 | |
| clarkb | ok that still fails because apparently virtualenv isn't installed. I thought it was, but we can fix that pretty easily | 16:25 |
| clarkb | and my local git-review install is sad (I think beacuse python just updated) as soon as that is fixed I'll push up a thing for ^ | 16:28 |
| fungi | we're on to ze12 now | 16:29 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Install virtualenv in proposal jobs https://review.opendev.org/c/openstack/project-config/+/846427 | 16:30 |
| clarkb | fwiw tumbleweed converted python3 from 3.8 to 3.10 which broke my old venvs | 16:31 |
| clarkb | I've created new venvs using python3.10 -m venv instead of python3 -m venv which means whne we update to 3.11 I should avoid this problem | 16:31 |
| fungi | yeah, one up-side to building local copies of python from source is that i get to decide when i upgrade the default one which i build my venvs from, and have scripted the rebuild of all those so it's fairly painless | 16:34 |
| fungi | that and also i'm not limited to the python versions my distro happens to have decided to provide | 16:35 |
| fungi | which means i'm currently able to easily test with 3.11.0b3 even | 16:36 |
| clarkb | the fix is fine, its just jarring when you get errors like "this package doesn't exist" running a command you run every day | 16:36 |
| clarkb | I just make a new venv and install git-review, reno, docker-compose, and tox | 16:37 |
| fungi | oh, i also use separate venvs for such things | 16:38 |
| fungi | though it probably doesn't buy me much | 16:38 |
| fungi | i have a dozen different python tools installed into individual venvs and their entrypoints symlinked in my ~/bin | 16:39 |
| fungi | mainly worried that they'll start conflicting over deps if i were to dump them into a single one | 16:40 |
| fungi | especially since one of those is python-openstackclient | 16:40 |
| clarkb | ya I've thought about that but so far they don't really care. | 16:40 |
| clarkb | I think beacuse they are so distinct. If I started mixing in a bunch of tools that have overlappign deps it would be worse | 16:40 |
| corvus | according to docker inspect, the image that nl01 is running was based on "org.zuul-ci.change_url": "https://review.opendev.org/846220" | 17:47 |
| corvus | that corresponds to the latest commit on master | 17:47 |
| fungi | yay! | 17:47 |
| corvus | so i think we've been running master since the last restart | 17:47 |
| corvus | (yesterday) | 17:47 |
| clarkb | yup when I checked yestesrday it looked like how we expect it to | 17:48 |
| corvus | what's the story with zuul restarts? | 17:49 |
| corvus | looks like we're still crawling through executors? | 17:50 |
| clarkb | corvus: yup almost done with the executors now | 17:50 |
| corvus | cool. i think we're ready for a nodepool release, but probably want to wait for this cycle to finish for zuul | 17:50 |
| fungi | probably in the next 2-3 hours we'll be done | 17:56 |
| clarkb | frickler: whats the story with mariadb and jammy and focal? Wondering if we should hold off on the upgrade of mariadb on review.o.o which will update from 10.4 to 10.6 using the upstream mariadb image with a containter host on focal | 17:57 |
| clarkb | seems like this is a ubuntu packaging specific problem so we are probably fine as we consume the image from the upstream mariadb image and not ubuntu packaging | 17:58 |
| frickler | clarkb: the issue happens when running mariadb in a jammy container on a focal host, because of the older kernel it seems | 18:10 |
| Clark[m] | Hrm maybe that is a problem for us then using the upstream amriadb image too? | 18:11 |
| frickler | if upstream uses jammy then yes | 18:11 |
| fungi | we're on 1/2 scheduler starts now | 18:22 |
| fungi | and the second scheduler is starting up | 18:37 |
| Clark[m] | frickler I think it may be debian based. Is it specific to jammy then? That is the part I don't understand. If it is just the newer mariadb then we have problems. But if it is specific to jammy I think we are ok | 18:54 |
| frickler | Clark[m]: it seems to be related to liburing2 vs. liburing1, the former is new to jammy https://jira.mariadb.org/browse/MDEV-28397 | 19:01 |
| * frickler eods | 19:01 | |
| fungi | and all done! | 19:03 |
| fungi | real 1637m43.306s | 19:03 |
| fungi | so 27h18m | 19:05 |
| clarkb | https://github.com/MariaDB/mariadb-docker/blob/c0d07be9ad5eb3bc212c6805cc8031308c56e9b6/10.6/Dockerfile the mariadb 10.6 docker image is based on focal so we should be fine | 19:20 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!