| ianw | so setuptools 50.0 has reimplemented the change from hhttp://lists.openstack.org/pipermail/openstack-discuss/2020-July/015779.html; see https://github.com/pypa/setuptools/commit/04e3df22df840c6bb244e9b27bc56750c44b7c85 | 00:59 |
|---|---|---|
| ianw | SETUPTOOLS_USE_DISTUTILS=distlib is an escape hatch ... | 01:00 |
| fungi | stdlib | 01:00 |
| ianw | sorry, yes | 01:00 |
| fungi | at least according to the changelog | 01:01 |
| ianw | for devstack, i think maybe putting that in pip_install | 01:01 |
| fungi | i didn't test it though, nor have i looked closely at the code | 01:01 |
| ianw | i don't think we'd want it set for things using pip, say, from their own virtualenv | 01:01 |
| fungi | nah, virtualenvs/venvs should be unaffected | 01:02 |
| fungi | this will just be places where we're running pip install into the system context as root, and likely only on debian/ubuntu systems | 01:03 |
| fungi | since it's clobbering debian's patched distutils which installs stuff into /usr/local instead of /usr | 01:03 |
| ianw | right, what i mean is i don't think we want the env var set in that case | 01:04 |
| fungi | i concur | 01:04 |
| ianw | so like a global export in stack.sh isn't quite right | 01:05 |
| ianw | or we remove our /usr/local/bin references, i guess | 01:05 |
| fungi | that's an option so long as we're not worried about the possibility the same commands could exist elsewhere in the path but be somehow wrong/broken | 01:07 |
| ianw | # nova defaults to /usr/local/bin, but fedora and suse pip like to | 01:07 |
| ianw | # install things in /usr/bin | 01:07 |
| ianw | iniset $NOVA_CONF DEFAULT bindir "/usr/bin" | 01:07 |
| ianw | we already have stuff like ^ scattered around | 01:08 |
| ianw | another option i guess is to use the packaged pip tools only. | 02:38 |
| ianw | that's a pita though, and i guess things like bionic pip don't understand manylinux2014 wheels | 02:38 |
| clarkb | why nit siwtch to /usr/bin? | 03:06 |
| clarkb | maybe not ideal but should work? | 03:06 |
| ianw | clarkb: that's not really the issue -- it installs all libraries to site-packages and the packaged python doesn't look in there | 03:07 |
| ianw | that's by debuntu design -- dist-packages for .deb packaged python, and site-packages if you like to ./configure; make your own python | 03:08 |
| clarkb | ah | 03:10 |
| ianw | yeah, so trying to use the packaged pip tools it blows up building things like https://pypi.org/project/grpcio/#files which have manylinux wheels that only later pips understand | 03:11 |
| ianw | i think there's something up with https://mirror.ca-ymq-1.vexxhost.opendev.org/pypi/simple/ | 04:05 |
| ianw | --- pypi.org ping statistics --- | 04:08 |
| ianw | 28 packets transmitted, 0 received, 100% packet loss, time 27628ms | 04:08 |
| ianw | it can't talk or ping to pypi. but i can ssh in, and it can wget/ping google.com | 04:08 |
| ianw | hrm, this looks like the db8 addresses again | 04:16 |
| ianw | i have run "ip -6 addr list | grep 2001:db8 | awk '{ print $2 }' | xargs -I {} -n1 ip addr del {} dev ens3" | 04:17 |
| ianw | mnaser, infra-root: ^ | 04:17 |
| *** stephenfin has quit IRC | 05:35 | |
| *** stephenfin has joined #opendev | 05:42 | |
| *** stephenfin has quit IRC | 05:47 | |
| *** stephenfin has joined #opendev | 05:56 | |
| frickler | mnaser: did you ever succeed in locating the sources for that? these were the current ones http://paste.openstack.org/show/797280/ | 05:59 |
| *** stephenfin has quit IRC | 06:01 | |
| *** stephenfin has joined #opendev | 06:02 | |
| *** fressi has joined #opendev | 06:05 | |
| *** ysandeep|away is now known as ysandeep | 06:08 | |
| *** stephenfin has quit IRC | 06:09 | |
| *** stephenfin has joined #opendev | 06:16 | |
| *** sshnaidm|afk is now known as sshnaidm | 06:17 | |
| *** stephenfin has quit IRC | 06:27 | |
| *** stephenfin has joined #opendev | 06:29 | |
| *** stephenfin has quit IRC | 06:50 | |
| *** stephenfin has joined #opendev | 06:57 | |
| ianw | frickler: you probably noticed everything broken :) | 07:03 |
| *** stephenfin has quit IRC | 07:04 | |
| ianw | frickler: two suggestions are to pin setuptools <50 https://review.opendev.org/#/c/748943/1 ; or try to set the config variable in appropriate places (something like https://review.opendev.org/748957 ... i've set it wip as i think there might be more) | 07:04 |
| ianw | i'm pretty much out for today, so ... yeah :/ | 07:04 |
| *** hashar has joined #opendev | 07:08 | |
| *** ysandeep is now known as ysandeep|lunch | 07:32 | |
| *** tosky has joined #opendev | 07:39 | |
| *** zigo has joined #opendev | 07:44 | |
| *** tosky has quit IRC | 08:00 | |
| *** sgw has quit IRC | 08:00 | |
| *** dirk has quit IRC | 08:00 | |
| *** moppy has quit IRC | 08:01 | |
| *** moppy has joined #opendev | 08:02 | |
| *** tosky has joined #opendev | 08:05 | |
| *** sgw has joined #opendev | 08:05 | |
| *** dirk has joined #opendev | 08:05 | |
| *** hemanth_n has joined #opendev | 08:14 | |
| lourot | o/ just checking, the last discussion(s) here are related to the py35 POST_FAILUREs we're seeing e.g. in https://review.opendev.org/#/c/748805/ , correct? | 08:16 |
| frickler | lourot: I guess they are, although that's a failure mode I haven't seen yet. infra-root: ^^ that would be an affected location outside of devstack | 08:19 |
| *** ysandeep|lunch is now known as ysandeep | 08:45 | |
| *** rgildein has joined #opendev | 09:01 | |
| *** rgildein has quit IRC | 09:01 | |
| ianw | "SystemError: Parent module 'setuptools' not loaded, cannot perform relative import" | 09:03 |
| ianw | i guess that is related to setuptools 50 if i had to guess, but not sure exactly what part of it | 09:03 |
| *** dtantsur|afk is now known as dtantsur | 09:11 | |
| frickler | #status notice due to a new release of setuptools (50.0.0), a lot of jobs are currently broken, please do not recheck blindly. see http://lists.openstack.org/pipermail/openstack-discuss/2020-August/016905.html | 09:15 |
| openstackstatus | frickler: sending notice | 09:15 |
| -openstackstatus- NOTICE: due to a new release of setuptools (50.0.0), a lot of jobs are currently broken, please do not recheck blindly. see http://lists.openstack.org/pipermail/openstack-discuss/2020-August/016905.html | 09:15 | |
| openstackstatus | frickler: finished sending notice | 09:18 |
| *** DSpider has joined #opendev | 09:49 | |
| *** hashar has quit IRC | 10:02 | |
| *** Gyuseok_Jung has joined #opendev | 10:03 | |
| *** tosky has quit IRC | 10:13 | |
| *** tosky has joined #opendev | 10:13 | |
| *** elod has quit IRC | 11:27 | |
| *** weshay is now known as weshay|ruck | 11:44 | |
| *** elod has joined #opendev | 11:52 | |
| *** sshnaidm has quit IRC | 11:57 | |
| *** sshnaidm has joined #opendev | 12:07 | |
| *** redrobot has joined #opendev | 12:34 | |
| *** ysandeep is now known as ysandeep|brb | 12:49 | |
| *** lpetrut has joined #opendev | 12:52 | |
| *** ysandeep|brb is now known as ysandeep | 12:57 | |
| *** kevinz has quit IRC | 13:59 | |
| *** ysandeep is now known as ysandeep|afk | 14:05 | |
| *** cloudnull has joined #opendev | 14:06 | |
| fungi | #status log zm07 was rebooted by the provider at 04:00z due to unspecified hypervisor host issues | 14:11 |
| openstackstatus | fungi: finished logging | 14:12 |
| fungi | looks like it's working fine at this point, so i'll close out that ticket | 14:12 |
| frickler | any puppet folk around? puppet-openstack-beaker-centos-8 seems to be running as non-voting job in gate | 14:12 |
| fungi | along with the one for zm04 last tuesday | 14:12 |
| *** fressi has quit IRC | 14:22 | |
| *** openstackgerrit has quit IRC | 14:37 | |
| clarkb | I'm trying to page back in the things that I had learned last time setuptools changed this behvaior nad I seem to recall that it was specific to the editable installs | 14:37 |
| clarkb | once I've got some tea and breakfast I was going to test if that is still the case and if so we may be able to avoid a lot of trouble simply by dropping the -e from pip install | 14:38 |
| fungi | it seems like it's any installs, not just editable | 14:41 |
| fungi | but they also provided an envvar this time we can use to tell setuptools to use the distutils from stdlib rather than its own copy | 14:41 |
| frickler | clarkb: dropping -e worked for devstack, but ianw's solution seemed better to me, see #-qa | 14:42 |
| frickler | https://review.opendev.org/748957 vs. https://review.opendev.org/749007 | 14:43 |
| clarkb | fungi: I'm rerunning my old reproducers fromthe original bug to test (but with different setuptools versions) | 14:45 |
| clarkb | frickler: ya for devstack editable installs are probably a reasonable feature to keep | 14:45 |
| clarkb | but in the general case I don't think they are necessary | 14:45 |
| *** ysandeep|afk is now known as ysandeep | 14:47 | |
| frickler | I haven't seen editable installs anywhere else yet. the os-testr failure on py35 seems a different thing, maybe updating setuptools after the venv was created? | 14:52 |
| clarkb | confirmed removing the -e preserves the old behavior | 14:52 |
| clarkb | fungi: ^ fyi it is editable install specific | 14:53 |
| fungi | oh, cool, so that does give us some additional options | 14:54 |
| clarkb | `sudo docker run -it --rm ubuntu:20.04 bash -c 'apt-get update && apt-get install -y python3-pip git && pip3 install -U pip && pip --version && pip install -U "setuptools" && git clone https://github.com/pypa/twine && pip install ./twine && which twine && pip uninstall -y twine && pip install ./twine && which twine'` is the non -e version | 14:54 |
| clarkb | and it uses twine because that was pypa's example package when reproducing before | 14:54 |
| clarkb | add a -e to the pip install to test the -e case | 14:55 |
| *** qchris has quit IRC | 14:57 | |
| *** qchris has joined #opendev | 15:10 | |
| *** sshnaidm is now known as sshnaidm|bbl | 15:11 | |
| *** lpetrut has quit IRC | 15:11 | |
| *** lpetrut has joined #opendev | 15:20 | |
| *** erbarr has joined #opendev | 15:23 | |
| *** ysandeep is now known as ysandeep|away | 15:24 | |
| johnsom | Not that it's as important an issue as the setuptools issue, but gerritbot doesn't seem to be happy. https://review.opendev.org/714004 posted an update that wasn't announced. | 15:58 |
| fungi | the bot may be dead. i'll check it in a sec | 15:59 |
| fungi | it's still running and getting events from gerrit according to its log, but maybe it fell off irc and didn't notice | 16:01 |
| johnsom | I got booted out of freenode over the weekend, so maybe there are freenode issues at play today as well | 16:01 |
| fungi | 2020-08-31 14:37:31 <-- openstackgerrit (~openstack@eavesdrop01.openstack.org) has quit (*.net *.split) | 16:01 |
| fungi | looks like it hit a split almost 1.5 hours ago | 16:02 |
| johnsom | The update I linked was about eight minutes ago | 16:03 |
| fungi | yeah, i think it was hit with a netsplit over an hour ago and hasn't returned | 16:05 |
| fungi | i'll try restarting it and see if it gets a connected server | 16:05 |
| fungi | #status log restarted gerritbot following a netsplit at 14:37:31 from which it never returned | 16:06 |
| openstackstatus | fungi: finished logging | 16:06 |
| yoctozepto | ah, you read my mind, I was about to report the gerrit bot missing ;d | 16:08 |
| *** priteau has joined #opendev | 16:09 | |
| fungi | johnsom is on the ball today | 16:09 |
| *** wolsen has joined #opendev | 16:37 | |
| *** dtantsur is now known as dtantsur|afk | 17:13 | |
| *** tosky has quit IRC | 17:14 | |
| *** priteau has quit IRC | 17:29 | |
| clarkb | morning meetings are done. I've reviewed setuptools things in devstack land and going to pop out on a bike ride now | 18:05 |
| clarkb | when I get back I'd like to make a gerritlib release | 18:05 |
| clarkb | (any idea if setuptools update will impact that?) | 18:06 |
| fungi | probably not, but if it does we can at least hopefully address it there since it's a lower impact project | 18:10 |
| *** diablo_rojo has joined #opendev | 18:11 | |
| fungi | doesn't look like openstack release management have approved any release requests since setuptools 50 happened | 18:11 |
| *** lpetrut has quit IRC | 18:20 | |
| *** mattd01 has joined #opendev | 18:36 | |
| *** mattd01 has quit IRC | 18:37 | |
| *** mattd01 has joined #opendev | 18:38 | |
| clarkb | infra-root how does tagging 6058d994b8bd36a8cf46b14ae2f05f806b895191 as 0.9.0 in gerritlib look? | 20:12 |
| clarkb | If we're happy with ^ I'll do that after I finish eating this sandwich | 20:12 |
| fungi | lookin' | 20:15 |
| *** tosky has joined #opendev | 20:15 | |
| fungi | 0.8.4 was the last release, 6058d99 is the master branch tip | 20:16 |
| fungi | all changes in that range are testing-related or bug fixes, except the last one which is a feature addition, so i agree with 0.9.0 | 20:17 |
| fungi | if not for "Add branches arg to createProject" i would have said 0.8.5, but yeah 0.9.0 there is best | 20:18 |
| clarkb | ok making that tag now | 20:23 |
| clarkb | and pushed | 20:26 |
| clarkb | once that is on pypi the next step is toe land the jeepyb change which should build new gerrit images with the gerritlib and jeepyb updates included | 20:27 |
| fungi | thanks! | 20:30 |
| fungi | i guess this will also tell us if new setuptools broke general release automation jobs | 20:30 |
| clarkb | release-openstack-python succeeded at least | 20:30 |
| fungi | a good sign, yep | 20:31 |
| clarkb | https://pypi.org/project/gerritlib/0.9.0/ is present as well | 20:31 |
| clarkb | fungi: https://review.opendev.org/#/c/741279/2 is the next thing to land, should I go ahead and do so now? | 20:32 |
| fungi | sure, you've got 2x +2 on there | 20:33 |
| clarkb | k just going to double check that that builds gerrit imgaes | 20:33 |
| clarkb | huh it triggers gerrit images but not for 2.14 | 20:34 |
| clarkb | not going to worry about that right now since we don't need 2.14 at the moment | 20:34 |
| *** mattd01 has quit IRC | 20:51 | |
| mnaser | HTTPError: 500 Server Error: Internal Server Error for url: https://opendev.org/openstack/requirements/raw/branch/stable/train/upper-constraints.txt | 20:58 |
| mnaser | other requests went through just fine | 20:58 |
| mnaser | but just a heads up in case something is brewing.. | 20:58 |
| *** openstackgerrit has joined #opendev | 21:04 | |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Pin setuptools<50 in our image venvs https://review.opendev.org/749154 | 21:04 |
| clarkb | ianw: ^ I think that is the workaround for python3.5 os-testr problems | 21:04 |
| clarkb | mnaser: which backend do you get that from (the cert names will tell you) | 21:04 |
| clarkb | all 8 gitea backends load that page for me | 21:05 |
| fungi | or was that in a job log? | 21:05 |
| openstackgerrit | Merged opendev/jeepyb master: Set repo HEAD on gerrit project creation https://review.opendev.org/741279 | 21:15 |
| clarkb | infra-root config-core ^ fyi that shouldn't impact new project creations but if it does let me know | 21:16 |
| clarkb | infra-root config-core https://review.opendev.org/749154 should fix the python3.5 job errors that people are reporting | 21:30 |
| clarkb | I'm happy to shepherd some image rebuilds if I can get reviews on that | 21:30 |
| fungi | lookin' | 21:36 |
| ianw | clarkb: are we 100% sure it's using tox from that virtulaenv these days? | 21:40 |
| ianw | and not something setup by ensure-tox? | 21:40 |
| clarkb | ianw: no, and really its just os-testr from the venv for it that I'm noticing failing | 21:40 |
| clarkb | I figured I'd apply the fix globally across the venvs though | 21:40 |
| clarkb | https://zuul.opendev.org/t/zuul/build/f0928f15f0e1423385c7cd5a9ebd454c is an example | 21:41 |
| clarkb | re ensure-tox I believe we set the tox path to our venv and use the venv | 21:41 |
| clarkb | but I'm not sure of that | 21:41 |
| ianw | https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ensure-tox/tasks/main.yaml#L7 ... i don't think so | 21:44 |
| ianw | also http://codesearch.openstack.org/?q=tox-env&i=nope&files=&repos= | 21:44 |
| ianw | anyway, kind of moot because it's not tox causing hte problem | 21:44 |
| clarkb | maybe we should set tox_executable, but ya not important for right now | 21:45 |
| openstackgerrit | Merged openstack/project-config master: Pin setuptools<50 in our image venvs https://review.opendev.org/749154 | 21:54 |
| clarkb | thanks! | 21:55 |
| clarkb | I've double checked and our new gerrit docker imag ehas gerritlib==0.9.0 in it as expected | 21:57 |
| *** Dmitrii-Sh1 has joined #opendev | 22:03 | |
| *** Dmitrii-Sh has quit IRC | 22:05 | |
| *** Dmitrii-Sh1 is now known as Dmitrii-Sh | 22:05 | |
| *** stephenfin has joined #opendev | 22:37 | |
| clarkb | I've started an ubuntu-xenial image build after confirming that project config has been updated on builders | 22:38 |
| clarkb | focal and bionic just got new builds prior to that change landing so will need to be triggered as well (though so far only python3.5 jobs on xenial seem affected) | 22:41 |
| *** tosky has quit IRC | 23:03 | |
| clarkb | ianw: should we keep the github third party ci meeting topic item? | 23:06 |
| ianw | clarkb: i think we can remove it for now. upstream said that full release automation was on their radar for "soon" ... of course we could do that all for them right now :) | 23:07 |
| clarkb | ianw: I did mean to ask why we only got a py35 arm64 wheel? Is that just a manylinux problem? | 23:07 |
| clarkb | or do we need to add more jobs etc | 23:08 |
| ianw | no, it's just that it's using the stable abi so it's forward compatible | 23:08 |
| clarkb | will pip discover that? | 23:08 |
| clarkb | if so maybe I need to resurrect my nodepool arm64 image jobs | 23:08 |
| clarkb | s/jobs/changes/ | 23:09 |
| ianw | yep, recent enough pip will pull it in, as it's the absolute lowest-common-denominator that works everywhere | 23:09 |
| clarkb | neat I didn't know that | 23:09 |
| ianw | yes, sorry i meant to ping to try that again | 23:09 |
| clarkb | ianw: feel free to update my changes or push new ones | 23:09 |
| clarkb | I'm running out of day and trying to get the agenda out and shepherd the vm image rebuilds | 23:09 |
| clarkb | I can likely look at nodepool tomorrow if you don't get there first | 23:09 |
| ianw | ok; yeah was on my todo, till everything was broken yesterday :) | 23:09 |
| clarkb | ianw: I think what we need/want is a revert chang eof the multi arch removal, then on top of that we can fiddle with mirror setup or whatever to speed things up | 23:11 |
| clarkb | (if necessary) | 23:11 |
| ianw | yeah, i'm not sure if ffi re-enabled their wheel build; do you remember if anything else was slowing it down? | 23:14 |
| ianw | off hand | 23:14 |
| clarkb | pynacl | 23:15 |
| fungi | so pip on py38 will pull in cp35 wheels? | 23:15 |
| fungi | i did not realize | 23:15 |
| fungi | i was sure i had seen pip under newer interpreters ignore wheels for older abis | 23:16 |
| ianw | fungi: yeah, stable api https://docs.python.org/3/c-api/stable.html | 23:18 |
| ianw | abi | 23:18 |
| ianw | note that, annoyingly, the bionic pip doesn't | 23:19 |
| clarkb | ya but how does pip know the cp35 annotated wheel is valid for cp38? I mean I expect that it does I just don't know the mechanism | 23:19 |
| ianw | it's the -abi3- in it i think | 23:20 |
| fungi | aha | 23:20 |
| fungi | i just hadn't seen one of those before | 23:20 |
| ianw | yes, i'm saying this like i'm not some sort of dilettante who had never built a manylinux wheel before last week :) | 23:21 |
| clarkb | you're the expert now | 23:22 |
| * fungi casts off one shoe | 23:23 | |
| ianw | we should be able to do pynacl | 23:24 |
| clarkb | for nodepool we should see if we can run quickly enough now with or without the openstack wheel mirrors | 23:26 |
| clarkb | but ya may need to add wheels for a small number of additional deps | 23:26 |
| fungi | i suppose we could try to implement our full wheel builder job on arm64? | 23:27 |
| fungi | might run for days though | 23:27 |
| ianw | yeah, i think targeted approach is probably better | 23:31 |
| clarkb | fwiw I think pynacl is in the openstack wheel builds the problem we hit with it in zuul context is zuul doesn't use openstack's constraints | 23:32 |
| fungi | honestly, while it's a little more work, maybe this is a great case for generalizing the wheel builder job to take arbitrary requirements lists | 23:32 |
| ianw | the thing with the manylinux builds though is i think it pretty much defeats any sort of generic approach to building | 23:32 |
| clarkb | maybe we can have openstack's wheels builds build the constraints and whatever is most recent | 23:32 |
| ianw | like the pyca one is using it's own custom docker image | 23:32 |
| clarkb | since we'll expect openstack to use the recent version anyway | 23:32 |
| fungi | so we could say "run the wheel builder job for the list including these versions of cryptography and pynacl" (for starters) | 23:32 |
| clarkb | basically do a forward looking build there then I think zuul's major issue with consuming from it to plug gaps will be addressed | 23:33 |
| ianw | yeah, i mean that was basically what we decided :) but it's also great if we can do something like work with upstream because now the cryptography problem is solved for everyone | 23:33 |
| clarkb | ++ | 23:33 |
| fungi | oh, i get ya... yes, building manylinux wheels is likely a separate task from maintaining a platform-specific wheel cache, agreed | 23:33 |
| clarkb | ya I'm not saying stop with the upstream builds | 23:33 |
| clarkb | more just a how do we make things better downstream too | 23:34 |
| ianw | yeah, agree. updating the building script to pull more arbitrary things is on my todo as well :) | 23:35 |
| ianw | looks like cryptography have realised that the rust extensions they want to pull in don't work nice with the python limited api/abi | 23:36 |
| fungi | are they building those with maturin or something like setuptools-rust? | 23:37 |
| wolsen | 4 | 23:50 |
| ianw | fungi: setuptools-rust and then ... umm ... i don't know what other wrappers are invovled | 23:52 |
| clarkb | the xenial image is built and uploading to ovh now (the other 6 upload workers are uploading the bionic image that built just a bit too early) | 23:54 |
| clarkb | I was hoping to recheck https://review.opendev.org/749098 before enqueuing rebuilds of the other images so I'll wait on bionic and focal and the debian images | 23:55 |
| clarkb | they don't seem to have trouble currently | 23:55 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!