| *** iurygregory is now known as iurygregory|holiday | 02:26 | |
| *** amoralej|off is now known as amoralej | 06:08 | |
| opendevreview | Merged openstack/releases master: [cloudkitty] Transition Xena to EM https://review.opendev.org/c/openstack/releases/+/878864 | 07:55 |
|---|---|---|
| frickler | reviewers please have a look at https://review.opendev.org/c/openstack/releases/+/880944 , this is one of the blockers for using latest oslo.db in openstack | 08:34 |
| opendevreview | Amit Uniyal proposed openstack/releases master: [yoga][zed][antelope] Nova stable branch release https://review.opendev.org/c/openstack/releases/+/881200 | 09:00 |
| opendevreview | Merged openstack/releases master: python-aodhclient 3.3.0 https://review.opendev.org/c/openstack/releases/+/880944 | 09:05 |
| *** amoralej is now known as amoralej|lunch | 12:43 | |
| opendevreview | Amit Uniyal proposed openstack/releases master: nova: Release yoga 25.1.1 https://review.opendev.org/c/openstack/releases/+/881209 | 12:45 |
| opendevreview | Amit Uniyal proposed openstack/releases master: nova: Release zed 26.1.1 https://review.opendev.org/c/openstack/releases/+/881210 | 12:45 |
| opendevreview | Amit Uniyal proposed openstack/releases master: nova: Release antelope 27.0.1 https://review.opendev.org/c/openstack/releases/+/881211 | 12:45 |
| hberaud | ttx, elodilles: is one of you is ok to replace me as chair for our meeting of Mai 5th? I fogot that my children will be in PTO during this same week so I'll be surely AFK this day. I'm ok to invert mine with one of your next weeks. | 12:48 |
| hberaud | s/Mai/May | 12:48 |
| elodilles | hberaud: i think i can, but let me check the dates | 12:53 |
| elodilles | hberaud: May 5th looks OK to me, added my name as chair. we haven't filled out all the chair slots yet, so no need to decide now which one is good for whom :) | 12:57 |
| elodilles | hberaud: if you have some time could you please review these, as some of them can be merged i think: https://review.opendev.org/q/topic:xena-em+is:open+label:PTL-Approved | 13:02 |
| hberaud | elodilles: thanks | 13:03 |
| hberaud | OMW | 13:03 |
| opendevreview | Merged openstack/releases master: [kolla] Transition Xena to EM https://review.opendev.org/c/openstack/releases/+/878883 | 13:14 |
| opendevreview | Merged openstack/releases master: [glance] Transition Xena to EM https://review.opendev.org/c/openstack/releases/+/878893 | 13:17 |
| elodilles | thanks! | 13:17 |
| opendevreview | Merged openstack/releases master: [winstackers] Transition Xena to EM https://review.opendev.org/c/openstack/releases/+/878876 | 13:20 |
| opendevreview | Elod Illes proposed openstack/releases master: [OpenStackAnsible] Transition Xena to EM https://review.opendev.org/c/openstack/releases/+/878887 | 13:22 |
| opendevreview | Merged openstack/releases master: [monasca] Transition Xena to EM https://review.opendev.org/c/openstack/releases/+/878898 | 13:26 |
| auniyal | hello, gate question, I have three patches and they all kind of stuck at same places | 13:33 |
| auniyal | https://zuul.openstack.org/stream/cb10cc84c07a476fb7e5ec2f5e920220?logfile=console.log | 13:34 |
| auniyal | https://zuul.openstack.org/stream/52c638dab78a4e5f876c21aab82728fc?logfile=console.log | 13:34 |
| auniyal | okay third one just went further, | 13:34 |
| auniyal | it took some time, but one is completed now, will just wait for others to complete | 13:36 |
| opendevreview | Merged openstack/releases master: [sahara] Transition Xena to EM https://review.opendev.org/c/openstack/releases/+/878888 | 13:37 |
| opendevreview | Merged openstack/releases master: [swift] Transition Xena to EM https://review.opendev.org/c/openstack/releases/+/878891 | 13:37 |
| auniyal | in case if logs get refreshed and you want want to know what happend | 13:42 |
| auniyal | " | 13:42 |
| auniyal | DEBUG: + cd /home/zuul/src/opendev.org/openstack/releases/.tox/validate/tmp/releases-acl5aa8w/openstack/nova | 13:42 |
| auniyal | DEBUG: + git checkout -f 47b6850bb1f3681e6af7d8248152a979bf5051d1 | 13:42 |
| auniyal | DEBUG: Previous HEAD position was 2bb8689835 Merge "Revert "Add logging to find test cases leaking libvirt threads"" | 13:42 |
| auniyal | DEBUG: HEAD is now at 47b6850bb1 [stable-only] Update TOX_CONSTRAINTS_FILE for stable/2023.1 | 13:43 |
| auniyal | DEBUG: cwd = /home/zuul/src/opendev.org/openstack/releases/.tox/validate/tmp/releases-acl5aa8w/openstack/nova | 13:43 |
| auniyal | DEBUG: $ python3 setup.py sdist | 13:43 |
| auniyal | " | 13:43 |
| *** amoralej|lunch is now known as amoralej | 13:49 | |
| opendevreview | Amit Uniyal proposed openstack/releases master: nova: Release yoga 25.1.1 https://review.opendev.org/c/openstack/releases/+/881209 | 13:51 |
| elodilles | auniyal: hmmm, good question, i'm looking, but not yet see why. we did not have such issue before | 13:55 |
| opendevreview | Amit Uniyal proposed openstack/releases master: nova: Release zed 26.1.1 https://review.opendev.org/c/openstack/releases/+/881200 | 14:00 |
| auniyal | elodilles, its tox-validate, I updated patch so it will run again | 14:05 |
| auniyal | again here - https://zuul.openstack.org/stream/982a6041e65a42d6ae00f156361625c1?logfile=console.log | 14:06 |
| fungi | catching up... looks like the build took ~34 minutes at this step: https://zuul.opendev.org/t/openstack/build/cb10cc84c07a476fb7e5ec2f5e920220/log/job-output.txt#19980-19981 | 14:08 |
| fungi | probably someone should profile sdist building for the releases package | 14:09 |
| fungi | oh, that's for openstack/nova actually | 14:09 |
| fungi | so creating an sdist of nova is taking a very long time | 14:10 |
| fungi | there was a new pip release over the weekend which i think changed some things about its dep solver, so i suppose that could be involved | 14:10 |
| fungi | we don't call setup.py when we upload sdists any more, we probably should switch to calling build in the validate job too? | 14:11 |
| fungi | https://pip.pypa.io/en/stable/news/#v23-1 | 14:13 |
| elodilles | i could not reproduce it locally even with the latest pip+virtualenv+tox | 14:23 |
| elodilles | on the otherhand, we were close to 1hr with the validation at Antelope release, but the timeout was way more than 1 hr: https://etherpad.opendev.org/p/release-job-timeouts | 14:25 |
| fungi | openstack_releases.pythonutils.build_sdist() already calls `python3 -m build --sdist --wheel` so i wonder where that `python3 setup.py sdist` from the build log is coming from. i can't seem to find it | 14:26 |
| elodilles | the timeout clearly changed btw: https://zuul.opendev.org/t/openstack/build/cb10cc84c07a476fb7e5ec2f5e920220/log/zuul-info/inventory.yaml#177 | 14:26 |
| elodilles | oh, i see, gate job has bigger timeout value than check job: https://zuul.opendev.org/t/openstack/build/2960d0d64469464289de83f5ccc5773e/log/zuul-info/inventory.yaml#214 | 14:28 |
| elodilles | so in this case, it seems with nova we reached the timeout | 14:29 |
| elodilles | on the otherhand, 'python3 setup.py sdist' should be replaced with the build command, yes | 14:31 |
| fungi | yeah, at this point i'm struggling to find where that's being called | 14:32 |
| elodilles | yes, me too. it seems to be coming from tools/clone_repo.sh maybe? | 14:33 |
| elodilles | nah, this must be it: https://opendev.org/openstack/releases/src/branch/master/openstack_releases/requirements.py#L94 | 14:34 |
| fungi | aha, yep that's it | 14:35 |
| fungi | get_requirements_at_ref is called around that point in the validate script | 14:35 |
| fungi | as an aside, we seem to build the same packages more than once in this job, which itself is inefficient | 14:36 |
| elodilles | :S | 14:36 |
| fungi | https://zuul.opendev.org/t/openstack/build/cb10cc84c07a476fb7e5ec2f5e920220/log/job-output.txt#6401 | 14:37 |
| fungi | for reference | 14:37 |
| fungi | that happened near the beginning of the job and completed quickly | 14:38 |
| fungi | i wonder if get_requirements_at_ref could reuse the already built sdist | 14:38 |
| opendevreview | Elod Illes proposed openstack/releases master: Replace old sdist and wheel build command in validate https://review.opendev.org/c/openstack/releases/+/881229 | 14:42 |
| elodilles | yes, reuse would be useful | 14:42 |
| opendevreview | Jon Bernard proposed openstack/releases master: Final Cinder team releases for stable/xena https://review.opendev.org/c/openstack/releases/+/881133 | 14:46 |
| opendevreview | Jon Bernard proposed openstack/releases master: [cinder] Transition Xena to EM https://review.opendev.org/c/openstack/releases/+/878882 | 14:54 |
| opendevreview | Slawek Kaplonski proposed openstack/releases master: New neutron-tempest-plugin release https://review.opendev.org/c/openstack/releases/+/881230 | 14:55 |
| opendevreview | Dmitrii Shcherbakov proposed openstack/releases master: New neutron-lib release https://review.opendev.org/c/openstack/releases/+/881231 | 15:00 |
| auniyal | elodilles, fungi tox validate failed for me with - ERROR: Could not find 26.1.1: Command '['git', 'show', '26.1.1']' returned non-zero exit status 128. | 15:31 |
| auniyal | I had same error earlier too, | 15:33 |
| fungi | i suppose that could be a problem fetching git refs | 15:36 |
| fungi | i've checked the cacti graphs for all our current gitea backends and am not seeing any signs of resource exhaustion nor load imbalance | 15:39 |
| fungi | oh, 26.1.1 is the release you're adding | 15:40 |
| fungi | it doesn't exist upstream | 15:40 |
| auniyal | so I think its trying to run show cmd in master branch instead of stable/zed - https://zuul.opendev.org/t/openstack/build/c595a37b5b30479a8a63155f9a54e3b0/log/tox/validate/validate-request-results.log?severity=0#62 | 15:41 |
| clarkb | git show doesn't care baout its current branch. It will find whatever ref you ask it to show if it is present | 15:43 |
| auniyal | okay | 15:43 |
| auniyal | its probably crrect but still asking, is it correct that its deleting the target tag and then checking for it again ? | 15:53 |
| auniyal | from here https://zuul.opendev.org/t/openstack/build/c595a37b5b30479a8a63155f9a54e3b0/log/tox/validate/validate-request-results.log?severity=0#13444 | 15:53 |
| elodilles | auniyal: that is a bit misleading, but it does not mean always that it is a 'real' ERROR. the validate scripts collects all the valid ERRORs and WARNINGs at the end of its run | 15:54 |
| *** amoralej is now known as amoralej|off | 15:56 | |
| auniyal | ack, so its just get timeout on python3 setup.py sdist | 15:56 |
| auniyal | no other issue w.r.t to patch | 15:57 |
| auniyal | elodilles, so I should just recheck again ? | 15:57 |
| elodilles | fungi: i've checked whether we could spare building the sdist again, and i think, we can't :( the get_requirements_at_ref command is called for different hashes and sdist needs to be built for both cases: https://opendev.org/openstack/releases/src/branch/master/openstack_releases/requirements.py#L38-L39 | 15:58 |
| elodilles | auniyal: not until this fix hasn't merged: https://review.opendev.org/c/openstack/releases/+/881229 | 15:59 |
| clarkb | do we know why it is taking so long? setup.py shouldn't use pip so no dep solving should happen. Is pbr spending a bunch of time dealing with git to figure out authors and version stuff ? | 15:59 |
| auniyal | ack | 15:59 |
| elodilles | auniyal: if that does not decrease the time, then we have to increase the timeout | 15:59 |
| auniyal | okay | 16:00 |
| clarkb | elodilles: well you should probably figure out hwy nova is slow too | 16:00 |
| elodilles | clarkb: no clue at all, for me locally, sdists builds in some seconds (even with latest pip+virtualenv) | 16:00 |
| clarkb | right i understand we don't currently no why. I'm just saying that understanding it before making a determination to increase timeouts is probably a good idea | 16:01 |
| clarkb | you can probably add instrumentation to the setup.py sdist command | 16:02 |
| fungi | maybe even just turn on some verbosity, but regardless directly calling setup.py sdist is deprecated and increasingly discouraged by setuptools as time goes on | 16:02 |
| elodilles | clarkb: true, i just mentioned the timeout value increase as last resort, otherwise yes, we should figure out why it is that slow in zuul | 16:03 |
| clarkb | fungi: ya or monkeypatch the print() function to add timestamp prefixes | 16:03 |
| fungi | if using build instead of directly invoking setuptools makes things faster, i'd be inclined to not look much deeper into misbehavior of something that its maintainers are telling people not to run | 16:04 |
| clarkb | ya I'm more concerned if the problem ends up in PBRs git handling | 16:04 |
| clarkb | because that will only get worse as more git commits are made | 16:05 |
| clarkb | elodilles: actually `PYTHONUNBUFFERED=1 python setup.py sdist` may be sufficient then in theory we would get the output logged by zuul with reasonable timestamps | 16:06 |
| elodilles | fungi is right though, as with 'build --sdist' sdist is built in ~30 secs | 16:06 |
| fungi | in the same job | 16:06 |
| elodilles | yes | 16:06 |
| fungi | basically it calls build and generates an sdist and a wheel in seconds, then calls setup.py to make an sdist and waits more than half an hour | 16:07 |
| fungi | so something clearly pathological is going on there | 16:07 |
| fungi | my money's on setuptools calling out to pip since it dropped easy_install a while back, and something's going sideways with pip's dep solving under that specific condition | 16:08 |
| clarkb | oh I didn't realize setuptools will exec pip now. | 16:08 |
| clarkb | that would be for setup requires only right? Still possible to have it go crazy though | 16:09 |
| fungi | yeah, at least for the setup_requires | 16:09 |
| fungi | i suppose install_requires isn't relevant when creating packages | 16:09 |
| opendevreview | Elod Illes proposed openstack/releases master: nova: Release zed 26.1.1 https://review.opendev.org/c/openstack/releases/+/881200 | 16:11 |
| opendevreview | Elod Illes proposed openstack/releases master: Speed up validate with skipping rebuilding sdist https://review.opendev.org/c/openstack/releases/+/881242 | 16:11 |
| opendevreview | Elod Illes proposed openstack/releases master: nova: Release zed 26.1.1 https://review.opendev.org/c/openstack/releases/+/881200 | 16:12 |
| elodilles | fungi: btw, i now see another weird PTL-Approved behavior :S https://review.opendev.org/c/openstack/releases/+/881200/4#message-9db0664ea54283bd6d8c8c0b5c42c57528147a53 | 16:13 |
| elodilles | fungi: my bad, it works properly | 16:14 |
| fungi | oh good ;) | 16:14 |
| elodilles | fungi: i forgot that auniyal was added as release liaison already o:) | 16:14 |
| fungi | and as for the delay in zuul believing that it could merge the problem change from yesterday, i suspect some sort of cached lookup from before we altered the submit rule | 16:15 |
| fungi | so it didn't evaluate it again until it thought the cached results were stale/invalidated | 16:15 |
| fungi | if we continue to see it on new changes, then we can look deeper | 16:16 |
| elodilles | ack, thanks for the info! | 16:16 |
| auniyal | thanks for looking into this elodilles fungi clarkb | 16:17 |
| auniyal | fungi++ elodilles++ clarkb++ | 16:17 |
| elodilles | auniyal: np | 16:23 |
| elodilles | fungi: 'build --sdist' did it's job: the job ran 8 mins instead of 1hr: https://zuul.opendev.org/t/openstack/build/93f0f4a1cd3842c1beb9273346a03e16 | 16:25 |
| fungi | was it consistently running longer before, or just occasionally? wondering if there's some additional outside factor | 16:25 |
| fungi | i guess it may just be slow for releases of specific projects, e.g. nova | 16:26 |
| fungi | so hard to tell by looking at the build history for the job itself | 16:27 |
| elodilles | as i checked back my etherpad about the long running jobs during Antelope release, it showed 57 mins for validate job and as far as i remember it was nova, so i think yes | 16:27 |
| fungi | fingers crossed this clears it up for good then | 16:28 |
| elodilles | fungi: yepp, it took long only for huge repositories afair | 16:28 |
| elodilles | and far worse was nova (again, afair) | 16:28 |
| opendevreview | Elod Illes proposed openstack/releases master: Set Xena status to Extended Maintenance https://review.opendev.org/c/openstack/releases/+/881254 | 17:37 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!