| ianw | in a TIL, the way that docker sets the MTU means that on our linaro cloud, https connections to fastly don't work. i have to set it down to 1400 | 00:26 |
|---|---|---|
| ianw | not all connections. not http connections. just https connections, to fastly-based things, like, pypi, pythonhosted etc. | 00:27 |
| fungi | nuts | 00:27 |
| ianw | (connections *in* the container, outside works fine) | 00:27 |
| ianw | i'll have to put in something proper to zuul-jobs or something; https://github.com/pyca/cryptography/commit/652e1e11579237fcedc9fb3b1fe4edd96579abc5#diff-f4ee9b48fdd93cf4abfe6142818a03e4R11 works around it | 00:29 |
| ianw | clarkb/fungi/corvus: i'm happy with our POC for manylinux cryptography wheels -- i've updated the status @ https://github.com/pyca/cryptography/issues/5292#issuecomment-674331380 | 01:50 |
| fungi | ianw: brief reflection on that problem suggests it's probably a pmtud blackhole | 02:05 |
| fungi | i'm guessing other connections are negotiating a viable mss but connections to the fastly cdn endpoint may be dropping/eliding the ntf response or similar | 02:07 |
| openstackgerrit | Hirotaka Wakabayashi proposed openstack/diskimage-builder master: Adds Fedora 32 support https://review.opendev.org/746363 | 02:34 |
| *** seongsoocho has quit IRC | 02:55 | |
| *** mnasiadka has quit IRC | 02:55 | |
| *** mnasiadka has joined #opendev | 02:59 | |
| *** Open10K8S has quit IRC | 03:03 | |
| *** mnasiadka has quit IRC | 03:06 | |
| ianw | fungi: yeah ... it's also in a container, in a vm, behind god knows what on the other side of the cloud. it's a wonder packets get through at all really | 03:26 |
| *** mnasiadka has joined #opendev | 03:28 | |
| *** Open10K8S has joined #opendev | 03:28 | |
| *** seongsoocho has joined #opendev | 03:34 | |
| *** cloudnull has quit IRC | 04:58 | |
| *** cloudnull has joined #opendev | 04:59 | |
| jrosser | would we expect to find python-glanceclient 3.1.2 in https://mirror.ord.rax.opendev.org/wheel/ubuntu-20.04-x86_64/python-glanceclient/ ? | 07:23 |
| *** moppy has quit IRC | 08:01 | |
| *** moppy has joined #opendev | 08:01 | |
| *** DSpider has joined #opendev | 08:03 | |
| fungi | jrosser: normally, yes, if that version is in the upper-constraints.txt in openstack/requirements | 11:57 |
| * fungi hasn't looked yet | 11:57 | |
| jrosser | I think that’s in u-c for ussuri | 12:02 |
| *** tosky has joined #opendev | 12:21 | |
| fungi | jrosser: you're right https://opendev.org/openstack/requirements/src/branch/stable/ussuri/upper-constraints.txt#L201-L203 | 12:41 |
| fungi | so next to check the periodic build and see what's getting selected | 12:41 |
| fungi | added by https://review.opendev.org/745813 which merged 16:42 utc on wednesday | 12:43 |
| jrosser | I had a whole bunch of jobs blow up similarly and it was surprising that it didn’t fall back grabbing missing things to from pypi | 12:43 |
| fungi | so one of the wheel mirror updates in the last few days should have added it | 12:43 |
| fungi | oh, you know what, we publish wheels for that to pypi so we wouldn't have built a separate one for our binary wheel cache | 12:45 |
| fungi | https://pypi.org/project/python-glanceclient/3.1.2/#files has python_glanceclient-3.1.2-py3-none-any.whl | 12:46 |
| fungi | jrosser: have a link to the failure you saw? the job should have fetched that wheel from our caching pypi proxy, we don't (any more, and since some months) cache wheels for things which already have usable wheels on pypi | 12:48 |
| fungi | we don't separately cache those wheels, i mean | 12:48 |
| fungi | i should have remembered that sooner, but only just starting on my morning coffee | 12:49 |
| jrosser | fungi: heres a log https://zuul.opendev.org/t/openstack/build/10fd67f55b08464fb5260d66bf9306dc/log/logs/host/python_venv_build.log.txt#20921-21078 | 12:53 |
| jrosser | and as far as i can see thats looked in both https://mirror.ord.rax.opendev.org/pypi/simple/python-glanceclient/ and https://mirror.ord.rax.opendev.org/wheel/ubuntu-20.04-x86_64/python-glanceclient/ and wasnt able to find the right version in either | 12:54 |
| jrosser | i wonder if there was just some CDN issue with pypi at the time so it didnt find it via the caching proxy.... | 12:58 |
| fungi | yeah, it looks like it should have grabbed it from /pypi there | 13:29 |
| fungi | definitely looks like a stale pypi simple api index page | 13:33 |
| fungi | you can see in the log it outputs the versions it saw, 3.1.1 and 3.2.0 are there but 3.1.2 and 3.2.1 are missing | 13:35 |
| fungi | both 3.1.2 and 3.2.1 were published to pypi two days prior to when that job ran | 13:36 |
| fungi | yet were missing from the proxied pypi index | 13:36 |
| *** auristor has quit IRC | 14:32 | |
| *** auristor has joined #opendev | 14:57 | |
| *** qchris has quit IRC | 14:57 | |
| *** qchris has joined #opendev | 15:09 | |
| *** iurygregory has quit IRC | 15:36 | |
| *** tosky has quit IRC | 17:29 | |
| *** iurygregory has joined #opendev | 18:14 | |
| mnaser | infra-root: could someone check if there’s an extra ipv6 address on mirror in VEXXHOST | 18:18 |
| mnaser | (just like the issue we had a few days ago..) | 18:25 |
| frickler | mnaser: confirmed: http://paste.openstack.org/show/796858/ mirror01.ca-ymq-1.vexxhost.opendev.org | 18:28 |
| frickler | what did you do to fix that, just drop the 2001:db8 addrs? | 18:29 |
| frickler | would it help to run a tcpdump to see where the RAs are coming from? | 18:29 |
| mnaser | frickler: can you run a top dump in the background and remove those addresses? | 18:32 |
| mnaser | it’s a neutron bug :/ | 18:32 |
| mnaser | cc fungi logan- | 18:33 |
| frickler | mnaser: addresses removed, tcpdump so far only shows RAs for the correct subnet. is that a bug in neutron testing instances or in your deployment of neutron? | 18:36 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder master: update gentoo to allow building arm64 images https://review.opendev.org/746000 | 18:40 |
| frickler | anyway I'll go back to watching snooker, I'll leave the tcpdump running in a tmux in case it helps to identify some correlation when this happens again | 18:45 |
| mnaser | frickler: when neutron tests run they send out an RA that eventually gets picked up by openstack, so but in openstack which lets that slip out. Thanks for keeping the tcpdump up | 19:34 |
| fungi | good idea with the packet capture, we tried something similar when this cropped up in limestone, but then gave up after months of not reproducing the issue | 19:55 |
| fungi | i wonder if we can work out from the eui64 bits of the linklocal for the rogue gateway which node it originated from? | 19:56 |
| fungi | also the ttl on the route may tell us when it appeared | 19:56 |
| clarkb | I wonder if neutron expects security groups to guard against this | 20:18 |
| clarkb | (it shouldnt but) we open ours up andmanage rules on the hosts instead | 20:18 |
| clarkb | really even within a group you'd not want rogue RAs justlike you dont want rogue dhcp | 20:18 |
| fungi | i don't think they rely on security groups explicitly for guarding against this, but it's certainly possible that's why nobody notices | 20:19 |
| *** DSpider has quit IRC | 22:48 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!