| *** ianychoi has joined #openstack-swift | 00:14 | |
| *** openstackgerrit has joined #openstack-swift | 00:40 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: reconciler: Prevent problematic middlewares from being used https://review.opendev.org/c/openstack/swift/+/770522 | 00:40 |
|---|---|---|
| DHE | This is from an old-ish version of swift, but I ran into something. swift-ring-builder set_info doesn't update the version of the builder file. caused the composite ring builder to refuse to load the updates | 01:33 |
| *** tkajinam has quit IRC | 01:41 | |
| *** tkajinam has joined #openstack-swift | 01:42 | |
| *** baojg has joined #openstack-swift | 01:52 | |
| *** tkajinam has quit IRC | 02:09 | |
| *** tkajinam has joined #openstack-swift | 02:10 | |
| *** rcernin has quit IRC | 02:57 | |
| *** rcernin has joined #openstack-swift | 03:18 | |
| *** rcernin has quit IRC | 03:21 | |
| *** rcernin has joined #openstack-swift | 03:21 | |
| *** psachin has joined #openstack-swift | 03:34 | |
| *** rcernin has quit IRC | 04:35 | |
| *** rcernin has joined #openstack-swift | 04:35 | |
| openstackgerrit | Matthew Oliver proposed openstack/swift master: Reclaim shards when root container is reclaimed https://review.opendev.org/c/openstack/swift/+/770529 | 04:46 |
| *** gyee has quit IRC | 05:08 | |
| *** evrardjp has quit IRC | 05:20 | |
| *** evrardjp has joined #openstack-swift | 05:24 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-swift | 05:35 | |
| *** rcernin_ has joined #openstack-swift | 05:42 | |
| *** rcernin has quit IRC | 05:42 | |
| *** m75abrams has joined #openstack-swift | 06:05 | |
| *** timburke__ has quit IRC | 06:52 | |
| *** dsariel has joined #openstack-swift | 06:55 | |
| *** rcernin_ has quit IRC | 07:28 | |
| *** openstackgerrit has quit IRC | 07:47 | |
| *** diablo_rojo__ has quit IRC | 08:01 | |
| *** rcernin_ has joined #openstack-swift | 08:06 | |
| *** rpittau|afk is now known as rpittau | 08:25 | |
| *** rcernin_ has quit IRC | 08:26 | |
| *** dasp_ has quit IRC | 09:16 | |
| *** dasp has joined #openstack-swift | 09:17 | |
| *** hoonetorg has joined #openstack-swift | 09:44 | |
| *** openstackgerrit has joined #openstack-swift | 13:47 | |
| openstackgerrit | Alistair Coles proposed openstack/swift master: prefactor: Pass s3api Config to S3Requests https://review.opendev.org/c/openstack/swift/+/729091 | 13:47 |
| openstackgerrit | Alistair Coles proposed openstack/swift master: s3api: Make allowable clock skew configurable https://review.opendev.org/c/openstack/swift/+/770005 | 13:47 |
| *** dsariel has quit IRC | 14:11 | |
| *** dsariel has joined #openstack-swift | 14:12 | |
| *** klamath_atx has quit IRC | 15:23 | |
| *** psachin has quit IRC | 15:28 | |
| *** _david_sohonet has joined #openstack-swift | 15:30 | |
| acoles | timburke: I was tactical in fixing https://review.opendev.org/c/openstack/swift/+/729091 - there is more work needed on the conf handling, but I didn't want to take it all on while blocking https://review.opendev.org/c/openstack/swift/+/770005 | 15:37 |
| _david_sohonet | Since upgrading from 2.17 to 2.25.1 (including an OS upgrade to Ubuntu 18.04) we have been having proxy-server hangs, approximately once every 2 days. This is happening across multiple physical servers. When this occurs, the proxy-server does not respond to a TCP connection at all. The proxy-server is listening on localhost:80 and directly on the server, nc 127.0.0.1 80 times out. eventlet is 0.25.1-2ubuntu1~cloud0 and python ve | 15:39 |
| _david_sohonet | rsion is 3.6.9. tcpdump shows SYN sent over loopback but no accept() is occuring, no respsonce to SYN. | 15:39 |
| _david_sohonet | Can anyone advise? | 15:39 |
| _david_sohonet | We have a system 'hung' as above at the moment. Error log is not showing any issues. The last message was : Jan 13 09:41:06 SM-X9DBL4B-S-3-LAP12-US proxy-server: Client disconnected on read of.... That was a number of hours ago, nothing since then. | 15:50 |
| DHE | there's a hack that might work. edit /usr/lib/python3.6/logging/__init__.py and set logThreading = False near the top. path might not be exactly right but this should help you find it | 15:51 |
| DHE | correction: logThreads = false | 15:52 |
| _david_sohonet | would a restart of proxy-server be required ? | 15:52 |
| DHE | oh yes | 15:53 |
| _david_sohonet | A restart of the process will resolve the issue, so was hoping there was some data I could gather before doing that. | 15:54 |
| DHE | /proc/$(pid of a hung proxy-server process)/wchan does it mention futex ? | 15:54 |
| _david_sohonet | # cat /proc/31892/wchan | 15:56 |
| _david_sohonet | poll_schedule_timeout | 15:56 |
| _david_sohonet | # cat /proc/31899/wchan | 15:56 |
| _david_sohonet | futex_wait_queue_me | 15:56 |
| _david_sohonet | cat /proc/31900/wchan | 15:56 |
| _david_sohonet | futex_wait_queue_me | 15:56 |
| _david_sohonet | yes | 15:56 |
| DHE | yeah this is exactly what I think it is | 15:56 |
| DHE | just go ahead and restart it | 15:56 |
| _david_sohonet | $ curl http://localhost/healthcheck | 15:57 |
| _david_sohonet | OK | 15:57 |
| _david_sohonet | It recovered and is serving requests again. | 15:57 |
| DHE | https://bugs.launchpad.net/swift/+bug/1895739 you're hitting this | 15:57 |
| openstack | Launchpad bug 1895739 in OpenStack Object Storage (swift) "Proxy server sometimes deadlocks while logging client disconnect" [Undecided,In progress] | 15:57 |
| _david_sohonet | Thank you DHE | 15:57 |
| DHE | at least i'm 90% sure | 15:57 |
| _david_sohonet | AFAIK we always see client disconnect in the error logs before the hangs | 15:58 |
| *** m75abrams has quit IRC | 16:01 | |
| *** dsariel has quit IRC | 16:34 | |
| *** dsariel has joined #openstack-swift | 16:34 | |
| _david_sohonet | We have a misbehaving client application ( ffmpeg / libavfilter ) which appears to never properly close it's GETs ( IIRC they are range requests ), so creates lots of 499s GETs and "Client Disconnect" messages in the error log | 16:47 |
| *** prometheanfire has left #openstack-swift | 16:47 | |
| _david_sohonet | DHE: Will setting logThreads to false in /usr/lib/python3.6/logging/__init__.py mitigate the bug ? | 16:49 |
| *** gyee has joined #openstack-swift | 17:00 | |
| *** baojg has quit IRC | 17:41 | |
| *** baojg has joined #openstack-swift | 17:41 | |
| clayg | _david_sohonet: we've been having a lot more healthy proxies since we upgrade to eventlet > 0.28 with https://github.com/eventlet/eventlet/commit/9f49f0bbb9c18d7478d7c5e2e677ae6452845ab5 | 17:52 |
| clayg | i think that logging hang only effects py3 - the eventlet hang was also related to logging, but effected py2 as well | 17:54 |
| _david_sohonet | Thanks clayg | 17:59 |
| *** timburke has joined #openstack-swift | 17:59 | |
| *** ChanServ sets mode: +v timburke | 17:59 | |
| *** rpittau is now known as rpittau|afk | 18:05 | |
| *** baojg has quit IRC | 18:11 | |
| *** baojg has joined #openstack-swift | 18:11 | |
| *** timburke_ has joined #openstack-swift | 18:13 | |
| *** ChanServ sets mode: +v timburke_ | 18:13 | |
| *** timburke has quit IRC | 18:15 | |
| *** baojg has quit IRC | 18:18 | |
| *** baojg has joined #openstack-swift | 18:19 | |
| *** _david_sohonet has quit IRC | 18:23 | |
| DHE | I've been using this hack for months now successfully. but I realize messing with system packages like this is also frowned upon | 18:43 |
| DHE | clayg: so this fixes py3 hangs? | 18:44 |
| clayg | i think Tim is running with that fix in his py3 home cluster and still sees issues - so no it doesn't fix all the py3 hangs | 18:44 |
| clayg | i'm not even sure it fixes all the py2 hangs - but we were also rebooting hung proxies all the time until we tracked down and fixed the eventlet bug | 18:45 |
| *** _david_sohonet has joined #openstack-swift | 18:57 | |
| *** _david_sohonet has quit IRC | 19:12 | |
| *** _david_sohonet has joined #openstack-swift | 19:14 | |
| *** paladox has quit IRC | 19:34 | |
| *** baojg has quit IRC | 19:39 | |
| *** paladox has joined #openstack-swift | 19:39 | |
| *** baojg has joined #openstack-swift | 19:39 | |
| zigo | I got 14 unit test failures when building swift in Debian Sid: http://paste.openstack.org/show/801608/ | 19:55 |
| zigo | timburke_: Any idea what's going on? | 19:55 |
| *** hoonetorg has quit IRC | 20:04 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: reconciler: Prevent problematic middlewares from being used https://review.opendev.org/c/openstack/swift/+/770522 | 20:08 |
| timburke_ | zigo, not off hand -- what version of mock is it using? and which python 3.9? fwiw, i regularly test with py39 on my fedora dev box (mock==4.0.2, python==3.9.1 currently) | 20:15 |
| timburke_ | it's also weird that it doubles-up each test :-/ | 20:15 |
| timburke_ | ah! looks like upgrading to mock==4.0.3 lets me repro | 20:23 |
| zigo | timburke_: Mock 4.0.3, Python 3.9.1. | 20:28 |
| zigo | I found it weird too indeed. | 20:28 |
| timburke_ | i suspect (though haven't confirmed) that https://github.com/testing-cabal/mock/commit/902eea1 triggered it | 20:28 |
| zigo | timburke_: Should I try to revert this patch in the Mock package and try to build swift? | 20:37 |
| timburke_ | ah! it's a swift (test) bug. i'll have a patch up shortly | 20:38 |
| clayg | timburke_: you mean you think we did something UNSEEMLY in a TEST!? | 20:38 |
| clayg | 😱 | 20:39 |
| zigo | timburke_: Ah, great, ping me when you have it! :) | 20:42 |
| zigo | Thanks. | 20:42 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Fix __exit__ calls https://review.opendev.org/c/openstack/swift/+/770666 | 20:42 |
| timburke_ | zigo, ^^^ | 20:42 |
| * zigo tries the patch | 20:43 | |
| timburke_ | clayg, is the shocking part that the unseemly thing only affected tests? ;-) | 20:44 |
| zigo | timburke_: This fixes things for me indeed, thanks ! | 20:49 |
| timburke_ | 👍 | 20:49 |
| *** adeberg has quit IRC | 20:53 | |
| timburke_ | almost meeting time! | 20:57 |
| kota_ | good morning | 20:58 |
| mattoliverau | morning | 20:59 |
| mattoliverau | kota_: o/ | 21:00 |
| kota_ | mattoliverau: o/ | 21:01 |
| acoles | kota_: mattoliverau good morning | 21:01 |
| *** theintern_ has joined #openstack-swift | 21:02 | |
| *** hoonetorg has joined #openstack-swift | 21:18 | |
| *** hoonetorg has quit IRC | 21:43 | |
| *** baojg has quit IRC | 21:43 | |
| *** baojg has joined #openstack-swift | 21:44 | |
| openstackgerrit | Alistair Coles proposed openstack/swift master: WIP: ssync: sync non-durable fragments from handoffs https://review.opendev.org/c/openstack/swift/+/770047 | 21:49 |
| *** rcernin has joined #openstack-swift | 21:59 | |
| acoles | so in my virtual env, 'which swift-Object-server' gives me /vagrant/swift/.tox/py36/bin/swift-Object-server but 'ls /vagrant/swift/.tox/py36/bin/' only shows the lower case swift-object-server | 22:04 |
| *** klamath_atx has joined #openstack-swift | 22:11 | |
| *** dsariel has quit IRC | 22:15 | |
| *** baojg has quit IRC | 22:52 | |
| *** baojg has joined #openstack-swift | 22:53 | |
| clayg | yeah, my .tox dir is in ~/ - I use the vtox and venv wrappers which set the toxworkdir to get the heck out of that shared filesystem | 22:59 |
| *** timburke_ has quit IRC | 23:11 | |
| *** timburke_ has joined #openstack-swift | 23:11 | |
| *** ChanServ sets mode: +v timburke_ | 23:11 | |
| *** baojg has quit IRC | 23:33 | |
| *** baojg has joined #openstack-swift | 23:34 | |
| *** theintern_ has quit IRC | 23:34 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!