| opendevreview | Mohammed Naser proposed zuul/zuul-jobs master: ensure-kubernetes: fix missing 02-crio.conf https://review.opendev.org/c/zuul/zuul-jobs/+/835162 | 00:03 |
|---|---|---|
| opendevreview | Mohammed Naser proposed zuul/zuul-jobs master: run-buildset-registry: Drop extra install packages task https://review.opendev.org/c/zuul/zuul-jobs/+/835156 | 00:03 |
| fungi | looks like we're just waiting for ze01 at this point | 01:47 |
| *** ysandeep|out is now known as ysandeep | 04:58 | |
| *** chandankumar is now known as chkumar|rover | 05:52 | |
| *** ysandeep is now known as ysandeep|afk | 06:13 | |
| *** ysandeep|afk is now known as ysandeep | 07:01 | |
| *** ysandeep is now known as ysandeep|lunch | 08:04 | |
| *** marios is now known as marios|out | 08:07 | |
| *** arxcruz|ruck is now known as arxcruz | 08:10 | |
| *** jpena|off is now known as jpena | 08:11 | |
| *** marios|out is now known as marios | 08:18 | |
| *** marios is now known as marios|out | 08:27 | |
| *** mgoddard- is now known as mgoddard | 09:11 | |
| *** ysandeep|lunch is now known as ysandeep | 09:32 | |
| opendevreview | daniel.pawlik proposed opendev/system-config master: Add Opensearch Dashboard information https://review.opendev.org/c/opendev/system-config/+/833264 | 09:41 |
| ysandeep | fyi.. latest setuptools broke tripleo-ci check/gate jobs https://bugs.launchpad.net/tripleo/+bug/1966382, similiar issue to what's reported here: https://github.com/pypa/setuptools/issues/3197 , We are trying to workaround those issue as per github issue.. But just incase other projects hit this as well. | 09:46 |
| opendevreview | daniel.pawlik proposed opendev/system-config master: Add Opensearch Dashboard information https://review.opendev.org/c/opendev/system-config/+/833264 | 10:00 |
| *** rlandy|PTO is now known as rlandy | 10:28 | |
| priteau | Hello. Is anyone aware of issues with py39 tests today? blazar py39 tests are timing out in CI, they run fine locally | 10:45 |
| priteau | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_23e/833113/1/check/openstack-tox-py39/23ef991/job-output.txt | 10:45 |
| opendevreview | panticz.de proposed openstack/diskimage-builder master: Add interpolation note for dynamic-login password https://review.opendev.org/c/openstack/diskimage-builder/+/835206 | 11:00 |
| frickler | priteau: might be related to the setuptools issue tripleo is seeing? everyone else seems to be fine https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-py39&skip=0 | 11:02 |
| priteau | I am not sure, they have a specific error message | 11:02 |
| priteau | We haven't run jobs on master recently, so it broke sometime between March 10 and yesterday | 11:05 |
| priteau | Upper constraints shouldn't have changed much since then. Very odd. | 11:07 |
| fungi | i doubt a tox-based job would be picking up yesterday's setuptools yet. usually that needs new virtualenv first | 11:42 |
| fungi | yeah, looks like that job used 60.10.0 not 61.0.0 | 11:44 |
| fungi | also that first timed_out result was before the new setuptools release happened anyway | 11:47 |
| *** chkumar|rover is now known as raukadah | 12:01 | |
| *** rlandy is now known as rlandy|rover | 12:30 | |
| rlandy|rover | fungi: hello ... can you force merge a patch for us to clear the current tripleo gate blocker? https://review.opendev.org/c/openstack/tripleo-ci/+/835193 | 13:58 |
| rlandy|rover | we have ptl approval | 13:58 |
| rlandy|rover | ysandeep: jm1: ^^ | 13:59 |
| rlandy|rover | jpodivin: hello | 14:00 |
| rlandy|rover | just asked above for force merge | 14:00 |
| fungi | rlandy|rover: yep, working on it now | 14:03 |
| rlandy|rover | fungi: great - thanks | 14:03 |
| fungi | rlandy|rover: do you just need it moved to the front of the gate, or more? | 14:03 |
| rlandy|rover | fungi: it will fail in gate | 14:04 |
| rlandy|rover | we have a bunch more patches to merge | 14:04 |
| rlandy|rover | fungi: we also need the one in gate right now 835178,5 | 14:04 |
| rlandy|rover | we need that one to finish | 14:04 |
| rlandy|rover | it should pass | 14:05 |
| fungi | rlandy|rover: they can't be squashed together? they're for different repos i guess? | 14:05 |
| rlandy|rover | ack | 14:05 |
| rlandy|rover | collect logs and tripleo-ci | 14:05 |
| fungi | got it. | 14:05 |
| rlandy|rover | then we will have to address quickstart and a few otheres | 14:05 |
| rlandy|rover | lovely morning we are having :( | 14:05 |
| rlandy|rover | 835178,5 will also need to force merge if it fails - but it shouldn't | 14:07 |
| fungi | #status log Bypassed Zuul to force the merging of 835193,10 due to circular dependency on changes for related repositories addressing regressions in Setuptools 61.0.0 | 14:10 |
| opendevstatus | fungi: finished logging | 14:10 |
| *** artom__ is now known as artom | 14:14 | |
| rlandy|rover | fungi: https://review.opendev.org/c/openstack/tripleo-ci/+/835193 look sto have merged - thank you | 14:15 |
| fungi | yep, see above | 14:15 |
| rlandy|rover | fungi: awesome - much appreciated!! | 14:15 |
| fungi | you're welcome! | 14:15 |
| fungi | good luck untangling the rest | 14:15 |
| *** diablo_rojo_phone is now known as Guest318 | 14:42 | |
| *** ysandeep is now known as ysandeep|dinner | 14:46 | |
| corvus | fungi: i'm going to be afk for a chunk of the day... i could do the rest of the zuul restart now (schedulers/web), or do it after i return... which gives you the warm fuzzies? | 14:52 |
| fungi | corvus: now is fine! | 14:53 |
| corvus | making it so | 14:53 |
| fungi | i saw the mergers and executors finished but i didn't want to fire off the rest in case you had something specific you wanted to check | 14:53 |
| corvus | nah, more concerned with the unexpected (ie, a usual deployment) | 14:54 |
| fungi | i actually fell asleep before the first batch of executors finished last night anyway | 14:54 |
| Clark[m] | I'll be around today. Just trying to get started | 14:56 |
| corvus | 1/2 done | 15:03 |
| corvus | #status log rolling restart of all of zuul on at least 5.2.0 | 15:06 |
| opendevstatus | corvus: finished logging | 15:06 |
| corvus | frickler: is https://zuul.opendev.org/t/zuul/autohold/0000000030 still useful? | 15:07 |
| clarkb | fungi: for the setuptools issues sounds like the impacted groups are aware and addressing it. Otherwis ewe haven't seen problems? | 15:10 |
| frickler | corvus: I must admit that I don't even remember anymore what I was testing there, will clean up, thx for spotting | 15:13 |
| corvus | frickler: np, i can delete it if you want | 15:13 |
| corvus | is one click in web ui :) | 15:14 |
| frickler | ah, that was pluggy vs. py2. deleted. | 15:16 |
| corvus | the restart is completely complete now :) | 15:24 |
| clarkb | nice | 15:24 |
| frickler | btw., regarding zuul auth, do we have docs how the auth_token in zuul02:/root/.config/zuul/client.conf was created? better be prepared when it expires in 9 years | 15:24 |
| corvus | just the upstream: https://zuul-ci.org/docs/zuul/5.2.0/client.html#create-auth-token | 15:29 |
| fungi | clarkb: yeah, so far only tripleo is confirmed impacted, i think because they install setuptools directly and aren't pinning it. devstack still may be pinning setuptools, not sure. tox isn't using the new setuptools yet because it depends on virtualenv's vendored copy | 15:32 |
| clarkb | ah | 15:33 |
| *** ysandeep|dinner is now known as ysandeep | 15:38 | |
| *** ysandeep is now known as ysandeep|out | 15:44 | |
| clarkb | #status log Deleted 9 pastes from paste.o.o at the request of the users. | 16:04 |
| opendevstatus | clarkb: finished logging | 16:04 |
| frickler | corvus: is that the same type of token as described in https://zuul-ci.org/docs/zuul/5.2.0/authentication.html#manually-generating-a-jwt or are those two different things? | 16:11 |
| dtantsur | hey folks, rings any bells? mkswap: error: /root/swapfile is mounted; will not make swapspace | 16:27 |
| dtantsur | example: https://zuul.opendev.org/t/openstack/build/2b7c7331dc204e01beae014ffd7fbe75/logs | 16:28 |
| clarkb | no that is new to me | 16:28 |
| clarkb | the issues we've had in the past are using a sparse file which latest linux kernels (f35) don't support anymore so we switched to dd'ing zeros | 16:28 |
| dtantsur | https://review.opendev.org/c/openstack/ironic/+/835098 has more than a dozen RETRY_LIMITs | 16:29 |
| clarkb | dtantsur: I suspect that something else made a swapfile | 16:29 |
| clarkb | and then when that got there it failed | 16:29 |
| dtantsur | https://zuul.opendev.org/t/openstack/build/2b7c7331dc204e01beae014ffd7fbe75/log/job-output.txt#1017 and https://zuul.opendev.org/t/openstack/build/2b7c7331dc204e01beae014ffd7fbe75/log/job-output.txt#1238 | 16:29 |
| dtantsur | you're right | 16:29 |
| clarkb | yes that is what happened | 16:30 |
| clarkb | https://zuul.opendev.org/t/openstack/build/2b7c7331dc204e01beae014ffd7fbe75/log/job-output.txt#1017 | 16:30 |
| dtantsur | I don't get it... opendev.org/openstack/devstack/playbooks/pre.yaml@master is run, then opendev.org/openstack/devstack/playbooks/pre.yaml@stable/yoga? | 16:31 |
| dtantsur | it's not grenade, why are two versions executed? | 16:31 |
| clarkb | you probably have an explicit branch matcher | 16:33 |
| clarkb | and now that yoga is branched it is matching multiple branches | 16:33 |
| clarkb | explicit branch matchers should almost never be used on branched repos but people do add them and they create these problems when branches are made. The trick now is to hunt it down | 16:34 |
| dtantsur | we do have to use them sometimes, but not in this case? at least I cannot immediately spot anything that affects all jobs | 16:34 |
| dtantsur | (and I'm not even sure I understand "branch matchers" correctly) | 16:35 |
| clarkb | dtantsur: a specification on a job that indicates which branch the job should run against | 16:35 |
| clarkb | if your repo is branched you should rely on the implicit matching that says the definition on this branch runs against hte branch and not override that | 16:35 |
| clarkb | let me take a look at the job definitions | 16:36 |
| dtantsur | clarkb: if I understand it right, we do it for bugfix branches, but not for master | 16:36 |
| clarkb | dtantsur: https://opendev.org/openstack/devstack/src/branch/stable/yoga/.zuul.yaml#L1-L7 that is the problem. I feel like I had to fix this for devstack when xena was branched too :/ | 16:40 |
| clarkb | let me cross check against that and push a change up | 16:40 |
| dtantsur | ahhh. wow. I haven't seen this syntax before. | 16:40 |
| dtantsur | I thought you were talking about override-checkout or something like that | 16:41 |
| clarkb | dtantsur: not override-checkout but branches: | 16:41 |
| dtantsur | I see. Never used that, I think. | 16:41 |
| clarkb | but ya there is this other thing that devstack does and I thought we had made sure that the branching process included addressing this for next time | 16:41 |
| clarkb | dtantsur:remote: https://review.opendev.org/c/openstack/devstack/+/835276 Remove the zuul pragma to match master and feature/r1 | 16:44 |
| dtantsur | thanks! | 16:52 |
| dtantsur | do we need the same patch for stable/yoga? | 16:53 |
| clarkb | dtantsur: so my change is wrong because it went to mater because we haven't updated .gitreview yet. There is an approved change for this already though. I'll see if that needs help landing | 16:53 |
| clarkb | I'm going to abandon the chnage I just pushed | 16:53 |
| dtantsur | ack | 16:53 |
| clarkb | looks like it is in the gate so just a matter of waiting now. Depends-On should work too https://review.opendev.org/c/openstack/devstack/+/835221/ | 16:54 |
| fungi | should we promote it, or is there not much backlog? | 16:55 |
| clarkb | not much backlog. Its near the top and promoting would just prolong right now I think | 16:56 |
| fungi | yeah, if it was waaay down with a bunch of changes failing ahead on the same problem then it would be a different situation | 16:57 |
| *** jpena is now known as jpena|off | 17:06 | |
| clarkb | looks like nodepool (and I assume zuul soon too) are looking at doing python 3.10 testing but I don't think our opendev base image has a 3.10 variant yet. We might want to update as much as we can to python3.8 or 3.9 then drop 3.7 and add 3.10 images? | 17:49 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update gitea to v1.16.5 https://review.opendev.org/c/opendev/system-config/+/835307 | 18:55 |
| clarkb | if testing says ^ that is happy it should be a fairly safe update. But I also don't think it is super urgent | 18:56 |
| clarkb | the screenshots from 835307's deployment lgtm | 20:12 |
| clarkb | I'm reading up on the setuptools discovery stuff and I expect many things will be fine as they follow the flat-layout/adhoc structure | 21:02 |
| clarkb | Where we might have trouble is if we put python scripts in playbooks/ or roles/ dirs | 21:03 |
| clarkb | overall though I don't expect widespread fallout | 21:03 |
| clarkb | doc/ tools/ tests/ are all excluded by default. However that also means they will be excluded from sdists/wheels | 21:04 |
| clarkb | whcih we may want to avoid? | 21:04 |
| clarkb | I think system-config might break. But where it breaks should be limited: doc builds? | 21:08 |
| clarkb | something like `find ./ -type f -name '*.py' | grep -v ./doc/ | grep -v ./tools/ | grep -v './test[^/]*/'` where you also ignore $packagename dir to find problems | 21:10 |
| clarkb | I'm not able to make system-config tox -e docs fail if I upgrade the setuptools install in the .tox/docs dir and rerun | 21:13 |
| clarkb | maybe we are ok there then | 21:13 |
| clarkb | hrm nevermind if I manually construct a new virtualenv with up to date setuptools and try to pip install setuptools from there it fails | 21:21 |
| clarkb | *try tp pip install system-config form there | 21:21 |
| clarkb | fiddling with the setup.cfg to override auto discovery I'm not quite sure what the correct thing to do here is. The docs have a good example but for a completely different setup than ours | 21:32 |
| clarkb | I'm also not entirely sure the fix tripleo used is correct? setting py_modules to [] should result in emptyish packages? | 21:45 |
| clarkb | on the system-config side of things setting 'packages = ' seems to produce wheels that look like the wheels produced by old setuptools (and are very empty) | 21:46 |
| clarkb | I think beacuse pbr is looking for __init__.py in top level dir to find the source dir? | 21:46 |
| fungi | i think sticking with pbr means we won't need to care about setuptools discovery functionality | 21:47 |
| fungi | but system-config could be an outlier | 21:47 |
| fungi | pbr uses git's tracked files to build the manifest.in itself, right? | 21:48 |
| clarkb | ya it should and checking tripleo-quickstart using old setuptools and new setuptools against their current fixed code produces wheels iwth 300 files each and almost the same total size | 21:52 |
| clarkb | I suspect that in this case we can lie about the py_modules and/or packages lis tand pbr will figure it out | 21:53 |
| clarkb | It might be worth understanding that better and leaving comments in the setup.cfg/setup.py filesso there isn't confusion over this | 21:54 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!