| *** mlavalle has quit IRC | 00:01 | |
| melwitt | oh hm. I've been having to go and manually set bugs to Fix Committed and Fix Released so I didn't realize that should have been working | 00:05 |
|---|---|---|
| fungi | yeah, this has always been running: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/gerrit/files/hooks/change-merged | 00:07 |
| fungi | melwitt: if you spot one which didn't get set to released when it should have, let me know and i can check logs (as long as it was recent enough to fit in our last month of gerrit error log retention) | 00:09 |
| *** dpawlik5 has quit IRC | 00:10 | |
| *** dpawlik7 has joined #opendev | 00:11 | |
| *** hamalq has quit IRC | 00:28 | |
| *** mailingsam has quit IRC | 00:41 | |
| *** kopecmartin has quit IRC | 00:57 | |
| *** dpawlik7 has quit IRC | 00:57 | |
| *** kopecmartin has joined #opendev | 00:57 | |
| *** dpawlik5 has joined #opendev | 00:57 | |
| *** dpawlik2 has joined #opendev | 01:23 | |
| *** dpawlik5 has quit IRC | 01:23 | |
| *** dpawlik2 has quit IRC | 01:33 | |
| *** dpawlik3 has joined #opendev | 01:36 | |
| *** brinzhang has joined #opendev | 01:40 | |
| *** brinzhang_ has joined #opendev | 01:56 | |
| *** brinzhang has quit IRC | 01:58 | |
| *** dpawlik9 has joined #opendev | 02:35 | |
| *** dpawlik3 has quit IRC | 02:36 | |
| openstackgerrit | xinliang proposed opendev/system-config master: Enable openEuler mirroring https://review.opendev.org/c/opendev/system-config/+/784874 | 03:23 |
| openstackgerrit | xinliang proposed opendev/system-config master: Enable openEuler mirroring https://review.opendev.org/c/opendev/system-config/+/784874 | 03:29 |
| *** ysandeep|away is now known as ysandeep | 03:44 | |
| *** ykarel has joined #opendev | 03:51 | |
| *** dpawlik9 has quit IRC | 04:35 | |
| *** dpawlik6 has joined #opendev | 04:35 | |
| *** whoami-rajat has joined #opendev | 04:38 | |
| *** marios has joined #opendev | 05:03 | |
| *** dpawlik7 has joined #opendev | 05:34 | |
| *** dpawlik6 has quit IRC | 05:35 | |
| *** kopecmartin has quit IRC | 05:36 | |
| *** DSpider has joined #opendev | 05:41 | |
| *** sboyron has joined #opendev | 06:01 | |
| *** sboyron has quit IRC | 06:03 | |
| *** ralonsoh has joined #opendev | 06:06 | |
| *** sboyron has joined #opendev | 06:09 | |
| *** eolivare has joined #opendev | 06:35 | |
| *** dpawlik5 has joined #opendev | 06:42 | |
| *** dpawlik7 has quit IRC | 06:42 | |
| *** SWAT has joined #opendev | 06:48 | |
| *** dpawlik5 has quit IRC | 06:52 | |
| *** dpawlik07 has joined #opendev | 06:55 | |
| *** dpawlik07 has quit IRC | 06:57 | |
| *** dpawlik5 has joined #opendev | 06:58 | |
| *** sshnaidm is now known as sshnaidm|afk | 06:58 | |
| *** dpawlik5 has quit IRC | 07:12 | |
| *** dpawlik5 has joined #opendev | 07:13 | |
| *** dpawlik5 is now known as dpawlik | 07:14 | |
| *** amoralej|off is now known as amoralej | 07:15 | |
| *** andrewbonney has joined #opendev | 07:19 | |
| *** sboyron has quit IRC | 07:21 | |
| *** sboyron has joined #opendev | 07:22 | |
| *** dpawlik has quit IRC | 07:28 | |
| *** dpawlik6 has joined #opendev | 07:28 | |
| *** avass has quit IRC | 07:29 | |
| *** avass has joined #opendev | 07:30 | |
| *** avass has quit IRC | 07:32 | |
| *** avass has joined #opendev | 07:34 | |
| *** tosky has joined #opendev | 07:38 | |
| *** dpawlik6 has quit IRC | 07:43 | |
| *** jpena|off is now known as jpena | 07:51 | |
| *** ysandeep is now known as ysandeep|lunch | 08:21 | |
| openstackgerrit | Takashi Kajinami proposed openstack/project-config master: Add puppet-cinder-core and puppet-glance-core https://review.opendev.org/c/openstack/project-config/+/784888 | 08:23 |
| openstackgerrit | Takashi Kajinami proposed openstack/project-config master: Add puppet-cinder-core and puppet-glance-core https://review.opendev.org/c/openstack/project-config/+/784888 | 08:28 |
| *** dtantsur|afk is now known as dtantsur | 08:45 | |
| *** ykarel is now known as ykarel|lunch | 08:50 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Use openstacksdk 0.45.0 for python2.7 https://review.opendev.org/c/zuul/zuul-jobs/+/784894 | 08:52 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Use openstacksdk 0.45.0 for python2.7 https://review.opendev.org/c/zuul/zuul-jobs/+/784894 | 08:54 |
| *** ysandeep|lunch is now known as ysandeep | 09:04 | |
| openstackgerrit | Merged zuul/zuul-jobs master: ensure-podman: Use official podman repos for ubuntu https://review.opendev.org/c/zuul/zuul-jobs/+/765177 | 09:17 |
| *** lpetrut has joined #opendev | 09:21 | |
| *** zoharm has joined #opendev | 09:22 | |
| *** DSpider has quit IRC | 10:05 | |
| *** sshnaidm|afk has quit IRC | 10:09 | |
| *** ykarel|lunch is now known as ykarel | 10:11 | |
| *** sshnaidm has joined #opendev | 10:18 | |
| *** kopecmartin has joined #opendev | 10:21 | |
| *** fdegir4 is now known as fdegir | 10:22 | |
| *** ykarel_ has joined #opendev | 10:32 | |
| *** ykarel has quit IRC | 10:34 | |
| *** ykarel_ is now known as ykarel | 11:13 | |
| *** jpena is now known as jpena|lunch | 11:31 | |
| *** brinzhang0 has joined #opendev | 11:38 | |
| *** brinzhang_ has quit IRC | 11:41 | |
| *** amoralej is now known as amoralej|lunch | 12:02 | |
| *** jpena|lunch is now known as jpena | 12:27 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Document algorithm var for remove-build-sshkey https://review.opendev.org/c/zuul/zuul-jobs/+/783988 | 12:50 |
| *** ykarel_ has joined #opendev | 12:53 | |
| *** ykarel has quit IRC | 12:56 | |
| *** zoharm has quit IRC | 13:03 | |
| *** amoralej|lunch is now known as amoralej | 13:07 | |
| *** mailingsam has joined #opendev | 13:26 | |
| priteau | Hello. I had a kayobe patch for stable/stein (https://review.opendev.org/c/openstack/kayobe/+/783621) which was working fine last Tuesday which is now completely broken. pip2 is trying to install the latest versions of packages, which are not compatible with python2. I've been trying to identify a package release which would explain this change of behaviour, but no luck. Has there been any change in infra since last Tuesday which would explain | 13:30 |
| priteau | it? | 13:30 |
| tosky | we are discussing about it on #openstack-qa | 13:31 |
| tosky | zbr: can you please repeat here what you discovered? | 13:31 |
| priteau | Heh, thanks :) | 13:31 |
| priteau | I see, it's a metadata issue due to mirrors themselves | 13:34 |
| tosky | the tl;dr, if I got it correctly, is that there was a major pypi outage over the weekend, which caused some infrastructural issue on their side | 13:34 |
| *** ykarel_ is now known as ykarel | 13:36 | |
| *** rh-jelabarre has joined #opendev | 13:37 | |
| priteau | zbr: I believe the python version requirement issue is resolved if using PyPI servers directly, is this correct? | 13:37 |
| zbr | my guess is that this may not solve from our mirrors, the holy-trinity of pip/setuptools/virtualenv all dropped support for py27, and it can become very tricky not to fail installing stuff on py27, for lots of reasons. | 13:38 |
| zbr | i would say that the only way to keep supporting py27 would be to manually constrain versions of these packages, always adding <x.y where x.y is the version that dropped support for py27. | 13:39 |
| priteau | At least in kayobe, our py27 jobs still use pip 20.3.4, which would install setuptools 44.1.1 and work fine | 13:40 |
| zbr | https://zuul.opendev.org/t/openstack/build/e98465dd84d84cd8b4da48aff000c7c8 -- based on what I read on that job, it used pip 9.0.3 :p | 13:41 |
| zbr | virtualenv creation failed and i do not know which version of it was used, clearly a system one | 13:44 |
| zbr | current versions of virtualenv do *not* download pip/setuptools/wheel from pypi unless you tell them to | 13:44 |
| priteau | Dunno about tempest, but our stable jobs used to work fine. See https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_fe4/783621/6/check/openstack-tox-py27/fe44c59/tox/py27-0.log for versions. | 13:47 |
| openstackgerrit | Merged zuul/zuul-jobs master: ensure-zookeeper: add use_tmpfs parameter https://review.opendev.org/c/zuul/zuul-jobs/+/783982 | 13:48 |
| *** avass has quit IRC | 13:48 | |
| *** avass has joined #opendev | 13:50 | |
| *** mlavalle has joined #opendev | 13:58 | |
| zbr | priteau: but i can see virtualenv being called with very specific options in your case, ones that are less likely to break. | 14:13 |
| fungi | zbr: tosky: priteau: the pypi maintainers have a fallback bandersnatch-based mirror which lacks python-requires metadata in its simple indices, so if fastly directs traffic there instead of to warehouse, you'll see pip pulling too-new versions of things which don't support older python interpreters. if that's happening again, we don't really have a solution on our end for this, it impacts everyone not just | 14:14 |
| fungi | our ci jobs | 14:14 |
| zbr | fungi: it is still happening today, long after the outdate from the last weekend. | 14:15 |
| priteau | I understand. But AFAIK PyPI is working normally again, but opendev mirrors aren't? | 14:15 |
| zbr | priteau: mirrors are static, which means they do work kinda similar to the pypi-fallback. still, my expectation was that once pypi.org issue was addressed, to be able to see the issue as fixed for us tool. | 14:17 |
| fungi | our pypi "mirrors" aren't mirrors at all, they're apache mod_proxy with a very short cache time on the indices | 14:21 |
| fungi | something like 5 minutes | 14:21 |
| avass | fungi: thanks that explains the issue in #zuul as well | 14:22 |
| fungi | so if the pypi outage is over, then the errors are likely unrelated | 14:22 |
| fungi | on the other hand, if what the pypi maintainers are saying is that the "outage" is over because they've pointed fastly at their bandersnatch mirror, then that's still a possibility | 14:23 |
| priteau | I can pip2 install fine when I use PyPI | 14:23 |
| fungi | priteau: remember there's a cdn involved, so what you see as "pypi" isn't necessarily the same pypi worldwide | 14:24 |
| tosky | the same jobs were working fine on April 1, and I'm pretty sure they (or one of their parents) haven't been updated since them | 14:24 |
| fungi | we've seen plenty of times where some of fastly's endpoints are serving from warehouse and some are serving from the bandersnatch mirror | 14:25 |
| zbr | fungi: what I recommend so far was to ensure pinning of pip/setuptools deps for py27 jobs, to "help" pick only compatible deps and avoid this issue | 14:29 |
| zbr | sadly doing this is not as easy as it appears, as many simple commands like "virtualenv ..." need updates. | 14:29 |
| fungi | you'd need to pin all dependencies | 14:29 |
| zbr | i mentioned that too, you fix one and fing 10 more broken ones. | 14:30 |
| zbr | if it would be up to me I would remove py27 from everywhere and allow whoever can afford the cost to attempt to fix it in a stable branch. | 14:31 |
| zbr | to keep the challenge fun, we can even use the #sisyphus hashtag to track the progress ;) | 14:32 |
| * mordred hands zbr a rock | 14:33 | |
| fungi | currently view-source:https://mirror.ord.rax.opendev.org/pypi/simple/setuptools/ is providing data-requires-python=">=3.5" with the setuptools-45.0.0-py2.py3-none-any.whl entry | 14:35 |
| priteau | fungi: Just tested it, this one works locally for me | 14:37 |
| priteau | But https://mirror.mtl01.inap.opendev.org/pypi/simple/ doesn't | 14:37 |
| fungi | "You are using pip version 9.0.3..." maybe that's too old to support requires-python metadata? | 14:37 |
| clarkb | if you examine the responses in your browser debug tools you get all the cache info and what pypi server served it | 14:38 |
| clarkb | we are very transparent about what you are getting | 14:38 |
| clarkb | you can take that info to pypi and say these frontends are still serving stale data if you confirm that the metadata is missing | 14:38 |
| fungi | view-source:https://mirror.mtl01.inap.opendev.org/pypi/simple/setuptools/ is also including data-requires-python entries | 14:40 |
| priteau | indeed, that's what I noticed too | 14:41 |
| fungi | https://pip.pypa.io/en/stable/news/#id531 "Implementation of pep-503 data-requires-python. When this field is present for a release link, pip will ignore the download when installing to a Python version that doesn’t satisfy the requirement." | 14:41 |
| fungi | so 9.0.3 is new enough for requires-python metadata | 14:41 |
| priteau | Well, of course it works now :D | 14:42 |
| fungi | are the failures consistent or random? | 14:42 |
| priteau | I had this in my pip.conf and I swear it wasn't working until minutes ago: | 14:42 |
| clarkb | note the ttl on indexes was 10 minutes last I checked and typically one of our mirrors will be served by at least 2 regions (and multiple servers) from the cdn | 14:42 |
| clarkb | this means every 10 mintues or so you roll the dice | 14:43 |
| priteau | [global] | 14:43 |
| priteau | index-url = https://mirror.mtl01.inap.opendev.org/pypi/simple/ | 14:43 |
| clarkb | it can go back to failing again in 10 minutes if the cdn is still stale | 14:43 |
| fungi | we've also seen some endpoints in the same region serve from different backends | 14:43 |
| fungi | yeah, that | 14:43 |
| fungi | so basically our "mirror" (proxy) requests an index and fastly sends it to the bandersnatch backend and we cache a copy without the metadata, a few minutes later our cache time is up and we request it from fastly again and this time we're served a copy from warehouse with the metadata included | 14:44 |
| fungi | depending on which fastly endpoint answers that particular request | 14:45 |
| fungi | we've tracked it down by manually searching the cache content on our proxies looking for copies of a particular index containing no metadata, and then checked the headers which indicate which fastly endpoint served that | 14:46 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Use openstacksdk 0.45.0 for python2.7 https://review.opendev.org/c/zuul/zuul-jobs/+/784894 | 14:55 |
| *** lbragstad has quit IRC | 14:58 | |
| *** hrw has joined #opendev | 14:59 | |
| hrw | morning | 14:59 |
| hrw | does someone remember/know how we are when it comes to Debian Bullseye support on nodes? | 14:59 |
| clarkb | hrw I think fungi has been working on it, not sure if images have been built and uploaded yet though | 15:00 |
| clarkb | you can check the labels available in zuul | 15:00 |
| hrw | zuul lacks bullseye nodes so I started looking | 15:00 |
| *** lbragstad has joined #opendev | 15:00 | |
| hrw | ok, I see that patches are in review. will look | 15:01 |
| hrw | noonedeadpunk and fungi work on it | 15:02 |
| fungi | yeah, i can't remember where we ended up with it... may have been waiting for a dib release? | 15:03 |
| hrw | iirc dib 3.8 had it but would need to check | 15:03 |
| noonedeadpunk | patch to dib hasn't merged | 15:06 |
| *** lpetrut has quit IRC | 15:06 | |
| noonedeadpunk | I stopped on this https://zuul.opendev.org/t/openstack/build/f37d1601ca0c4d16a69a1949978c5291/log/job-output.txt#25865 | 15:06 |
| noonedeadpunk | `Root partition of test-image does not appear to have grown: 1817051136 < 5000000000` | 15:06 |
| noonedeadpunk | didn't have time to look into the cause yet | 15:07 |
| hrw | thanks for info | 15:08 |
| melwitt | fungi: here's a recent example where the bug Status didn't get updated, it still says In Progress: https://bugs.launchpad.net/nova/+bug/1882094 when the patch merged on March 23 https://review.opendev.org/c/openstack/nova/+/733627 | 15:08 |
| openstack | Launchpad bug 1882094 in OpenStack Compute (nova) "[queens][OSP13] An overcloud reboot will sometimes leave nova_api broken" [Undecided,In progress] - Assigned to melanie witt (melwitt) | 15:08 |
| clarkb | noonedeadpunk: hrw: the growroot element is responsbile for that and the integration tests test it as it has had problems in the past | 15:08 |
| clarkb | if you need somewhere to look | 15:09 |
| noonedeadpunk | yeah, that's obviously growroot, just had no ideas on top of the head what exactly might be wrong with it | 15:10 |
| clarkb | the integration should should hopefully capture the console log for the instance which should log growroot | 15:11 |
| clarkb | (we have been looking at growroot in another context and the console has it there, btu we access it via journald so not sure if it ends up where nova can get it too) | 15:12 |
| *** hberaud has joined #opendev | 15:12 | |
| fungi | unrelated, but i think our arm64 centos images are still failing to boot | 15:14 |
| fungi | melwitt: thanks, this is what got logged... | 15:16 |
| fungi | [2021-03-23T16:56:16.503+0000] [HookQueue-1] ERROR com.googlesource.gerrit.plugins.hooks.HookTask : hook[change-merged] exited with error status: 2 [CONTEXT PLUGIN="hooks" SUBMISSION_ID="733627-reset_conf-1616518548794-559060ab" project="openstack/nova" ] | 15:16 |
| fungi | we'll probably need to do some local debugging to see why that's breaking | 15:16 |
| fungi | doesn't look like hook script stdout/stderr are getting included in the error log | 15:17 |
| *** lbragstad has quit IRC | 15:17 | |
| melwitt | fungi: ok, thanks for taking a look | 15:17 |
| fungi | i should be able to test manually running it, though it runs in the context of a container so testing it will be a little circuitous | 15:18 |
| *** lbragstad has joined #opendev | 15:19 | |
| yoctozepto | morning infra | 15:20 |
| yoctozepto | Zuul is acting weirdly on https://review.opendev.org/c/openstack/project-team-guide/+/784005 | 15:21 |
| yoctozepto | can't get it to gate | 15:21 |
| yoctozepto | and no logs :/ | 15:21 |
| hrw | long time since last time I used dib... | 15:22 |
| clarkb | yoctozepto: are there file filters on the jobs for that repo and if so does the chagne not trigger via those? | 15:26 |
| clarkb | yoctozepto: it ran the gate jobs. | 15:26 |
| yoctozepto | clarkb: sorry, I was not precise | 15:26 |
| yoctozepto | just saw what I wrote | 15:27 |
| yoctozepto | I meant it failed twice with no logs | 15:27 |
| clarkb | I see it is failing with a POST_FAILURE and has no logs | 15:27 |
| yoctozepto | yes, yes indeed | 15:27 |
| clarkb | we can check the logs on the zuul executors to see what happened there | 15:27 |
| clarkb | I'm in a meeting but can do that after | 15:27 |
| yoctozepto | thanks! :-) | 15:27 |
| clarkb | there are a number of post failures in the zuul status so likely not job specific | 15:28 |
| clarkb | fungi: ^ fyi | 15:28 |
| clarkb | I wonder if one of the regions we upload logs to is having issues | 15:29 |
| clarkb | as some recent job completions seem fine | 15:29 |
| fungi | yeah, i'm looking on ze07 where that build ran to see if i can spot what broke | 15:29 |
| yoctozepto | ah, yes, from Zuul status I can see lots of (middle) fingers | 15:30 |
| fungi | looks like the swift upload generated a traceback | 15:31 |
| fungi | HttpException: 503: Server Error for url: https://storage.bhs.cloud.ovh.net/v1/... | 15:33 |
| fungi | so ovh bhs swift raising 503 errors | 15:34 |
| fungi | we may need to disable log uploads there | 15:34 |
| fungi | 682 occurrences of that failure just on ze07 | 15:36 |
| fungi | most recent was 15:22:37 so probably still ongoing | 15:36 |
| yoctozepto | uhoh | 15:37 |
| fungi | no similar errors for storage.gra.cloud.ovh.net | 15:37 |
| openstackgerrit | Jeremy Stanley proposed opendev/base-jobs master: Temporarily disable log uploads to OVH BHS Swift https://review.opendev.org/c/opendev/base-jobs/+/784997 | 15:40 |
| fungi | infra-root: ^ urgent, will probably need to bypass zuul to merge that asap | 15:40 |
| clarkb | looking | 15:41 |
| clarkb | I've approved it if you want to do the merge (or wait for it to fail first?) | 15:41 |
| fungi | on it | 15:42 |
| openstackgerrit | Merged opendev/base-jobs master: Temporarily disable log uploads to OVH BHS Swift https://review.opendev.org/c/opendev/base-jobs/+/784997 | 15:45 |
| fungi | there we go | 15:45 |
| fungi | gertty is struggling to sync with gerrit at the moment | 15:45 |
| yoctozepto | gerrit struggles to sync with itself you mean | 15:46 |
| yoctozepto | it's slow this time of day | 15:46 |
| yoctozepto | Europe and Americas overlap time | 15:46 |
| *** d34dh0r53 has quit IRC | 15:47 | |
| fungi | #status notice POST_FAILURE results between 14:00 and 15:50 UTC can be safely rechecked, and were due to authentication problems in one of our storage donor regions | 15:52 |
| openstackstatus | fungi: sending notice | 15:52 |
| -openstackstatus- NOTICE: POST_FAILURE results between 14:00 and 15:50 UTC can be safely rechecked, and were due to authentication problems in one of our storage donor regions | 15:52 | |
| fungi | earliest error i found across all the executors was 14:01:26 utc | 15:53 |
| fungi | oh, technically not true with the end time there, it's job started before ~15:45 since that's when the storage location is decided | 15:54 |
| fungi | so some currently running jobs might still fail but jobs started after 15:45 utc should not try to use it | 15:55 |
| openstackstatus | fungi: finished sending notice | 15:55 |
| *** lpetrut has joined #opendev | 16:00 | |
| *** d34dh0r53 has joined #opendev | 16:08 | |
| *** ykarel has quit IRC | 16:11 | |
| *** hamalq has joined #opendev | 16:13 | |
| *** ysandeep is now known as ysandeep|away | 16:24 | |
| *** marios is now known as marios|out | 16:25 | |
| zbr | fungi: how crazy does it sound to drop py27 from pbr https://review.opendev.org/c/openstack/pbr/+/784938 ? | 16:46 |
| clarkb | as we have explained over and over and over again it is very crazy :) | 16:46 |
| clarkb | the problem is it is a setup_requires dep and you can't reliable set version specifers on those | 16:46 |
| clarkb | this means existing python2 and older python3 users of pbr will start getting a new pbr if you drop python2 and then break | 16:47 |
| clarkb | this includes things like openstack's stable branches | 16:47 |
| clarkb | but also individuals like mordred and fungi have indicated they are users of pbr with python2.7 code bases outside of openstack contexts | 16:47 |
| clarkb | I suspect that if openstack decides to make a python3 only pbr that it will require a fork and new package name that can be transitioned to | 16:48 |
| yoctozepto | alias python2='sudo rm -rf /' | 16:50 |
| mordred | yeah - basically I think there would need to be a REALLY good reason to do that. pbr itself doesn't change very much (and shouldn't) - there's no reason to actively remove py27 support given how far down the stack it sits | 16:52 |
| *** ysandeep|away is now known as ysandeep | 16:53 | |
| mordred | that said - I think minimizing what py27 testing is done should be perfectly acceptable | 16:54 |
| zbr | ... now includes a full deployment with py27, at least attempts to do it. | 16:56 |
| clarkb | yup because it wants to not break openstack train which is still a supported release? | 16:56 |
| zbr | TBH, i do like the idea of lowering the coverage on py27 without removing it. | 16:56 |
| mordred | I think it would be a good compromise | 16:56 |
| mordred | make sure it's not regressing syntactically (the biggest risk) | 16:56 |
| clarkb | is the problem that the train job doesn't work for other reasons? | 16:57 |
| zbr | most py27 jobs are failing to to the python_requires issue, all in different places. | 16:57 |
| clarkb | if so then ya, removing that job may make sense. If it is failing because pbr broke python2 compat then we shouldn't remove it as it caught an issue wen eed to cover in the unittests first :) | 16:57 |
| clarkb | zbr: not sure I understand | 16:57 |
| mordred | we're still officially supporting train on 2.7? | 16:57 |
| clarkb | is this the mirror issue? | 16:57 |
| clarkb | if so that should be fixed in the near future aiui | 16:57 |
| clarkb | mordred: extended maintenance and all that | 16:58 |
| mordred | yeah - the mirror issue isn't just a 2.7 thing | 16:58 |
| clarkb | mordred: the stable branches go way back these days | 16:58 |
| mordred | clarkb: gross | 16:58 |
| clarkb | mordred: also train was the last python2 release so I expect it will be the one people awnt to keep around if they have to make a choice | 16:58 |
| clarkb | particularly since centos 8 isn't an option into the future but centos 7 is | 16:58 |
| *** jpena is now known as jpena|off | 16:58 | |
| zbr | the lack of python_requires will it will never be fully fixed, when pypi goes into downgraded mode, that hits py27 users in particular. | 16:59 |
| clarkb | but makign this change doesn't help with that problem | 17:00 |
| clarkb | it just makes it worse because setup_requires isn't aware | 17:00 |
| clarkb | they are completely separate problems | 17:01 |
| zbr | in fact I suspect it will only go worse in one month https://status.python.org/ as I am not sure what is the status of SNI even with py27. I seen pip 9.0.3 used in some of the failing jobs... | 17:01 |
| clarkb | the fix for python_requires is either to use contraints or a lockfile or for pypi to actually solve the issue once and for all | 17:01 |
| clarkb | the fix for setup_requires is to transition everything to pyproject.toml which we know won't happen (so we are stuck with keeping pbr python2 support) | 17:02 |
| zbr | not sure who has the time to maintain those constraints, especially as you endup going down the dependency tree. Before you merge one fix, someone else will break your job by dropping py27 on another dependency. | 17:03 |
| *** amoralej is now known as amoralej|off | 17:03 | |
| clarkb | openstack does daily updates of constraints | 17:03 |
| zbr | these constraints must be inside the setup.cfg of each package, if you don't want the install to randomly fail. | 17:05 |
| zbr | while openstack-specific packages can use our constraints, we have a good number of generic libraries that are used outside openstack. | 17:05 |
| clarkb | ok, I'm just pointing out that these are completely separate concerns | 17:05 |
| clarkb | because pbr is a setup_requires | 17:05 |
| zbr | those need tuning of setup.cfg | 17:05 |
| clarkb | lets not conflate them and make things more confusing | 17:06 |
| fungi | there are likely also openstack users of pbr on python 2.7, as i pointed out in my comment, installing old versions of e.g. nova will still pull the latest release of pbr | 17:06 |
| clarkb | yup in particular train, which i expect to stick around due to its being the last python2 release and centos 8 being cancelled | 17:07 |
| fungi | i don't actually have current projects which support python 2.7 using pbr. plenty of people however have projects which use pbr and supported python 2.7 at some point in the past with releases their users may still be installing | 17:07 |
| *** whoami-rajat has quit IRC | 17:07 | |
| fungi | zbr: pip constraints don't apply to pbr | 17:08 |
| fungi | at least not with older setuptools versions which use easy_install as a backend | 17:09 |
| fungi | setuptools only started to use pip to retrieve packages in the past few years | 17:09 |
| fungi | you can't use new python ecosystem features to work around problems with old python ecosystem tools which pre-date those solutions | 17:10 |
| zbr | shortly pbr is expected to live in current form until a new version of python/pip/setuptools) will force its users to drop pbr in favour of the newer packaging system. | 17:13 |
| clarkb | I'm not sure I understand that statement, but roughly pbr must continue to support old software that relies on it. It can also support new software | 17:15 |
| clarkb | the trouble is when you try and drop old | 17:15 |
| *** andrewbonney has quit IRC | 17:16 | |
| fungi | zbr: "new version of python/pip/setuptools will force its users to drop pbr" but people using old python/pip/setuptools to install old package versions will still use the latest pbr release. making new things doesn't mean old things vanish or stop being used | 17:21 |
| fungi | priteau: are you still seeing new incidents of pip choosing too-new packages in your py27 jobs? wondering if pypi has cleared up whatever the cause was | 17:27 |
| *** eolivare has quit IRC | 17:30 | |
| clarkb | fungi: re centos arm64 images maybe we should email kevinz about trying fat32 config drives instead of iso? | 17:30 |
| *** dtantsur is now known as dtantsur|afk | 17:31 | |
| *** ralonsoh has quit IRC | 17:31 | |
| fungi | do we think that suddenly changed on ~friday? | 17:31 |
| fungi | http://travaux.ovh.net/?do=details&id=49850& seems to suggest that the swift problem from earlier should be resolved | 17:33 |
| fungi | is that how others read it? | 17:33 |
| clarkb | the images may have changed ~friday which caused them to not handle iso | 17:33 |
| clarkb | the other idea I had was to boot the upstream cloud arm64 image and see if it works | 17:33 |
| clarkb | fungi: "error rate back to normal at all locations" lgtm | 17:33 |
| fungi | i'll push up a revert | 17:34 |
| clarkb | if you want to double check you can reenable it on test (or did we not remove it there) and trigger the base-test path | 17:34 |
| openstackgerrit | Jeremy Stanley proposed opendev/base-jobs master: Revert "Temporarily disable log uploads to OVH BHS Swift" https://review.opendev.org/c/opendev/base-jobs/+/785019 | 17:34 |
| fungi | we didn't disable it on test, so yeah i can push a dnm change for that | 17:35 |
| clarkb | fungi: another idea is did we disable around that same time (maybe we made it happy in the process) | 17:35 |
| clarkb | just keep an eye out for issues when reenabling if we suspect ^ we may have been part of the fix | 17:35 |
| fungi | heh, maybe | 17:35 |
| fungi | i sure hope not | 17:35 |
| fungi | we merged it at 15:45, they indicated that errors seemed to have ceased as of 15:32, so i think it was clearing up while we were pushing that | 17:36 |
| clarkb | ++ | 17:37 |
| fungi | also i think the timestamps which don't say utc are in cest or similar | 17:37 |
| fungi | because that comment is timestamped 10 minutes from now | 17:38 |
| fungi | if it were utc | 17:38 |
| clarkb | ah | 17:39 |
| *** marios|out has quit IRC | 17:43 | |
| *** lpetrut has quit IRC | 18:13 | |
| *** rh-jelabarre has quit IRC | 18:31 | |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: [DNM] testing log uploads https://review.opendev.org/c/opendev/system-config/+/785025 | 18:39 |
| *** sboyron has quit IRC | 18:46 | |
| *** auristor has quit IRC | 18:53 | |
| *** ysandeep is now known as ysandeep|away | 18:56 | |
| *** auristor has joined #opendev | 18:58 | |
| priteau | fungi: there was still pip issues in my last recheck which ended one hour before your message. Rechecking again. | 19:09 |
| fungi | i can try to prod folks in #pypa-dev if nobody has done so yet | 19:10 |
| clarkb | I haven't | 19:11 |
| fungi | but to be clear, pretty much anyone can do that | 19:12 |
| priteau | I tried to pin some deps to known good versions, but then I hit issues with deps of deps which are not even in our requirements list | 19:13 |
| priteau | I mean the global-requirements list | 19:13 |
| fungi | https://status.python.org/incidents/rw171ylf8jw3 | 19:14 |
| clarkb | priteau: are you not using contraints? | 19:15 |
| clarkb | constraints should be global | 19:15 |
| fungi | "Unexpected behavior: Diverted requests to the Simple API internal mirror experienced additional confusing behavior due to missing data-requires-python metadata may have led to some users reliant on that feature receiving confusing errors or incorrect versions of unpinned dependencies." | 19:15 |
| fungi | clarkb: the errors seem to arise from installing too-new setuptools, which isn't covered by constraints | 19:21 |
| priteau | We are using upper-constraints of course. But for example this job failed because it pulled arrow 1.0.3: https://41a7b4b4e7581ee9052b-e2efce41b3edfa8b1781e11d852066bf.ssl.cf2.rackcdn.com/783621/8/check/kayobe-tox-ansible/e55e392/job-output.txt | 19:21 |
| priteau | arrow is not in stein u-c | 19:21 |
| clarkb | that seems like a bug in stein u-c constraints. But also makes sense that setuptools would cause problems too | 19:21 |
| fungi | er, at least some of the errors i saw were pip on python 2.7 installing setuptools 45.0.0 even though that requires python 3.5 or newer | 19:22 |
| fungi | so clearly not obeying the data-requires-python metadata (because it seems to have been missing from the indices sometimes) | 19:22 |
| fungi | another reassuring entry in that incident writeup... "Updates to Internal Mirror: Our internal mirror will be upgraded to mirror all current Simple API features so that in the event of future outages, installs will not be impacted." | 19:25 |
| fungi | so maybe we'll stop seeing this problem at some unspecified point in the future | 19:25 |
| clarkb | hopefully our previous reports of this issue helped them figure it out quickly when problems started today | 19:27 |
| clarkb | I know that when we first reported the behavior it was news to them | 19:28 |
| priteau | Maybe it's just luck but my recheck has not failed yet: https://zuul.openstack.org/status#783621 | 19:28 |
| clarkb | priteau: the issue above reports the fixed it | 19:28 |
| priteau | Well, that was updated yesterday, and our jobs were still failing just a few hours ago | 19:30 |
| clarkb | ah yup so someone should report to them that this was being observed today as well | 19:30 |
| fungi | right, they thought they fixed it yesterday, we're discussing in #pypa-dev currently | 19:31 |
| clarkb | fungi: thanks! | 19:31 |
| fungi | they're asking if we have a list of problem indices so they can manually purge them from the cdn | 19:32 |
| *** mailingsam has quit IRC | 19:33 | |
| clarkb | I suspect its likely to overlap with constraints though maybe not completely so | 19:33 |
| fungi | right | 19:34 |
| clarkb | since our jobs would've been pulling all those packages. Maybe add setuptools pbr and hacking/flake8/pycodestyle for things not in the list? | 19:34 |
| fungi | we're certainly not going to see the extent of the problem | 19:34 |
| fungi | it's more that they're asking for a list of which packages we've seen stale/incorrect indices for, since they'd be manually clearing each one in the cdn | 19:36 |
| clarkb | right, I'm suggesting that we don't bother trying to find that list and give them constraints + a few not in constraints instead | 19:37 |
| fungi | heh | 19:37 |
| clarkb | otherwise we'll just be doing whack a mole as we find new ones | 19:37 |
| clarkb | fungi: any indication if that API is still publicly accessible? | 19:37 |
| fungi | which api? | 19:37 |
| clarkb | because we could do it too if so (we had a purge api call in our old mirroring script iirc) | 19:38 |
| clarkb | the one to flush cdn data for particular packages | 19:38 |
| fungi | i don't think i'm going to ask them to manually clear hundreds of indices. at this point it could just be a few which are stuck for some reason. if we can demonstrate that it's more than a few, they'll likely look at more drastic measures | 19:38 |
| fungi | oh that api | 19:38 |
| fungi | i can ask | 19:38 |
| clarkb | basically the old mirror script would scan for errors, if found, purge, then retry | 19:38 |
| clarkb | and if that is what they intend on doing that maybe it is easier for us to script that up | 19:39 |
| clarkb | and/or address the specific problems as we find them rather than asking them to do it | 19:39 |
| fungi | looks like we did a purge request to the stale url, judging from https://opendev.org/opendev/puppet-bandersnatch/src/branch/master/files/run_bandersnatch.py | 19:43 |
| fungi | is that how you read it? | 19:43 |
| clarkb | ya | 19:43 |
| clarkb | curl -XPURGE https://pypi.org/simple/package/ and curl -XPURGE https://pypi.org/simple/package/json looks like | 19:44 |
| *** hamalq has quit IRC | 19:44 | |
| *** hamalq has joined #opendev | 19:45 | |
| fungi | clarkb: priteau: di says that still works if we just want to purge the ones we spot | 19:55 |
| clarkb | wfm | 19:56 |
| melwitt | clarkb: do you think this is good to approve now? the Depends-On patch has merged https://review.opendev.org/c/opendev/system-config/+/782540 | 20:21 |
| clarkb | melwitt: yes, I think we can go ahead with that. I need to pop out for a bit but can hit +A when I return | 20:22 |
| clarkb | or fungi can hit it if around | 20:22 |
| priteau | All jobs of https://review.opendev.org/c/openstack/kayobe/+/783621 passed, except one. New error this time: distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('pbr') | 20:31 |
| priteau | This might be a different issue | 20:31 |
| melwitt | cool thanks clarkb | 20:32 |
| *** rh-jelabarre has joined #opendev | 20:32 | |
| *** slaweq has quit IRC | 20:49 | |
| *** slaweq has joined #opendev | 20:51 | |
| fungi | priteau: yeah, i'm quickly getting lost in the spaghetti of tenks deployment scripting and ansible there | 20:53 |
| fungi | something is not finding pbr, that much is clear, but the logging from those scripts is not very thorough | 20:53 |
| fungi | when did this last work? | 20:54 |
| fungi | earlier today apparently, according to https://zuul.opendev.org/t/openstack/builds?job_name=kayobe-overcloud-centos&project=openstack/kayobe | 20:55 |
| *** slaweq has quit IRC | 20:57 | |
| openstackgerrit | Merged opendev/system-config master: Run update-bug on patchset-created again https://review.opendev.org/c/opendev/system-config/+/782540 | 21:23 |
| *** avass has quit IRC | 21:44 | |
| clarkb | melwitt: ^ fungi must've gotten it | 22:11 |
| melwitt | clarkb: he did :) thanks | 22:11 |
| clarkb | priteau: that error is coming from setuptools easy_install which is far less reliable than pip generally. Unfortunately it doesn't say which urls it attempted. However, from memory it doesn't use our mirrors at all and talks directly to pypi (since we dont' configure mirror setup for easy_install anymore | 22:46 |
| clarkb | fungi: ^ does that sound correct to you? | 22:46 |
| fungi | we used to configure it but i think we dropped that some time ago | 22:52 |
| clarkb | ya so I wonder if that is a case of the pypi cdn failing entirely | 22:53 |
| clarkb | I wonder if that easy_install failed on the sni brownouts | 23:10 |
| clarkb | that job ran on centos-7 and was using python2.7 | 23:13 |
| clarkb | centos-7 python2 is 2.7.5-89 according to a package listing | 23:15 |
| clarkb | 2.7.0-8 don't have sni according to https://github.com/pypa/pypi-support/issues/978 xenial is 2.7.12 as a comparison | 23:16 |
| clarkb | I have no idea if rhel/centos have backported sni support into python2 though | 23:16 |
| clarkb | (would be in things like urrlib? | 23:16 |
| clarkb | priteau: fungi: I'm beginning to suspect this is the issue since our mirror will bypass this on the pip side | 23:18 |
| clarkb | basically our mirror apache proxy doesn't care if you sni or not since we use different ports and dno't rely on sni to address vhosts | 23:18 |
| clarkb | then it will sni to the backend | 23:19 |
| clarkb | I also wonder if this is the sort of thing where rhel7/centos7 might update it does have 3 more years of shelf life | 23:21 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!