| *** rpittau|afk has quit IRC | 02:00 | |
| *** rpittau|afk has joined #openstack-swift | 02:01 | |
| *** gyee has quit IRC | 02:02 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-swift | 05:33 | |
| *** dsariel has joined #openstack-swift | 06:01 | |
| *** dsariel has quit IRC | 06:08 | |
| *** m75abrams has joined #openstack-swift | 07:29 | |
| *** rcernin has quit IRC | 07:31 | |
| *** dsariel has joined #openstack-swift | 08:35 | |
| *** rpittau|afk is now known as rpittau | 08:40 | |
| openstackgerrit | Merged openstack/swift master: Add a slightly more-useful repr for DiskFiles https://review.opendev.org/759929 | 09:09 |
|---|---|---|
| *** rcernin has joined #openstack-swift | 09:18 | |
| *** rcernin has quit IRC | 10:07 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift master: Imported Translations from Zanata https://review.opendev.org/760321 | 10:27 |
| *** rcernin has joined #openstack-swift | 11:15 | |
| *** whmcr has joined #openstack-swift | 12:21 | |
| *** ianychoi__ is now known as ianychoi | 12:34 | |
| whmcr | checking the requests that ormandj's referring to, the Host: header on the requests both match | 12:41 |
| *** cschwede has joined #openstack-swift | 12:50 | |
| *** ChanServ sets mode: +v cschwede | 12:50 | |
| kota_ | btw, could anyone recommend books that describe the modern file-system architecture and best practice? my colleague asked me about that, but I've not learned systematically so want to look for better books. | 13:11 |
| *** whmcr has quit IRC | 13:24 | |
| *** rcernin has quit IRC | 13:43 | |
| ormandj | kota_: it's a bit old but befs was pretty ahead of its time: https://www.haiku-os.org/legacy-docs/practical-file-system-design.pdf | 13:48 |
| ormandj | it's a good start though, dominic went on to design apfs @ apple | 13:48 |
| kota_ | ormandj: oh nice! I'll check it. | 13:48 |
| ormandj | it should give enough basics to know the right questions to ask to find the rest on google | 13:49 |
| kota_ | sounds pretty cool. | 13:50 |
| kota_ | much appreciated | 13:52 |
| ormandj | np! | 13:52 |
| zigo | Hi there! We're having a weird issue with the swift-proxy falling into timeouts (after 10 minutes), which likely is an issue with threadings in Ussuri. Has anyone experienced that? | 13:53 |
| *** whmcr has joined #openstack-swift | 13:53 | |
| ormandj | zigo: more clarity on the issue? we see timeouts in various places and run ussuri but we've also seen it with all versions :) | 13:54 |
| ormandj | what timeouts, specifically? | 13:54 |
| zigo | ormandj: It's our frontal haproxy that goes into timeout after no response from the swift-proxy. | 13:54 |
| ormandj | timburke: we can't figure out why one requet got the token/account/url rewrite and one did not :) | 13:54 |
| ormandj | we're wondering if it's related to the caching layer | 13:54 |
| zigo | We've configured it for long timeouts ... | 13:54 |
| ormandj | but perhaps no errors are thrown | 13:54 |
| ormandj | zigo: we don't generally experience timeouts talking to swift-proxy directly, just timeouts internal to swift-proxy and things it's doing | 13:55 |
| zigo | This happens after 6 hours of the swift-proxy restart. | 13:55 |
| zigo | ormandj: It doesn't look like the proxy is doing anything, it seems waiting for a lock or something. | 13:55 |
| ormandj | yeah, we don't have this issue. what happens if you run strace on a proxy process? | 13:56 |
| zigo | No timeout logged from other services like container or account. | 13:56 |
| ormandj | (making sure to follow forks/etc) | 13:56 |
| ormandj | i assume haproxy is logging which backend it makes the request to/times out | 13:56 |
| zigo | select(0, NULL, NULL, NULL, {tv_sec=0, tv_usec=8910}) = 0 (Timeout) | 13:56 |
| zigo | wait4(0, 0x7ffc3bd3d41c, WNOHANG, NULL) = 0 | 13:56 |
| zigo | select(0, NULL, NULL, NULL, {tv_sec=0, tv_usec=9980}) = 0 (Timeout) | 13:56 |
| zigo | wait4(0, 0x7ffc3bd3d41c, WNOHANG, NULL) = 0 | 13:56 |
| zigo | Stuff like that with starce ... | 13:56 |
| zigo | Our strace command was strace -f -p 1467 | 13:57 |
| zigo | (with 1467 being the PID of swift-proxy of course) | 13:57 |
| zigo | We also saw this when starting the proxy: http://paste.openstack.org/show/799509/ | 13:58 |
| zigo | Not sure if it is relevant to our issue... | 13:58 |
| ormandj | dmesg not showing anythingw eird, right? | 13:58 |
| ormandj | and yes that looks very odd in your startup | 13:58 |
| zigo | Nothing special in dmesg. | 13:59 |
| zigo | And this happens in all of our 6 swift proxies. | 13:59 |
| ormandj | yeah, something does not sound good :) | 13:59 |
| ormandj | perhaps: https://github.com/eventlet/eventlet/issues/592 | 14:01 |
| ormandj | we use 3.6 | 14:01 |
| zigo | I even tried reverting https://review.opendev.org/#/c/697653/ but it didn't fix anything. | 14:02 |
| patchbot | patch 697653 - swift - Replace all "with Chunk*Timeout" by a watchdog (MERGED) - 8 patch sets | 14:02 |
| zigo | Oh ... | 14:02 |
| ormandj | :) | 14:03 |
| zigo | ormandj: We're on Buster, so Python 3.7 ... | 14:03 |
| ormandj | yes, that could be an issue, i haven't run with 3.7. the swift devs might have more info if they've tested that | 14:03 |
| zigo | ormandj: Is there a fix for this eventlet bug? | 14:04 |
| zigo | Like: https://github.com/StackStorm/st2/pull/4855 ? | 14:04 |
| *** mattoliverau has joined #openstack-swift | 14:05 | |
| *** ChanServ sets mode: +v mattoliverau | 14:05 | |
| ormandj | zigo: again, i'm not sure this is even your issue, it was just a shot in the dark | 14:06 |
| zigo | Better than nothing, it's been blured for a few days already. | 14:07 |
| ormandj | https://docs.openstack.org/swift/latest/overview_replication.html#object-replication | 14:37 |
| zigo | Is it possible to run Swift with uwsgi ? | 14:37 |
| zigo | I mean, swift-proxy ... | 14:37 |
| zigo | Cause I'm tired of eventlet ... | 14:37 |
| seongsoocho | trivial question.. :) why is the name ssync ??? Alphabetical order?? rsync -> ssync ... | 14:37 |
| *** evrardjp has quit IRC | 14:38 | |
| *** evrardjp has joined #openstack-swift | 14:40 | |
| *** whmcr has quit IRC | 14:46 | |
| tdasilva_ | seongsoocho: i think soon you might start hearing about tsync :D | 14:52 |
| *** tdasilva_ is now known as tdasilva | 14:52 | |
| seongsoocho | tdasilva: :-) yeah, I heard about tsync this ptg | 14:53 |
| *** tkajinam has quit IRC | 15:05 | |
| *** takamatsu has quit IRC | 15:07 | |
| *** takamatsu has joined #openstack-swift | 15:09 | |
| *** m75abrams has quit IRC | 15:19 | |
| acoles | seongsoocho: I'm not certain but maybe s(wift)-sync or just alphabetical | 15:33 |
| ormandj | super-sync | 16:06 |
| timburke | https://review.opendev.org/#/c/741016/ | 16:15 |
| patchbot | patch 741016 - swift - sharding: Change default cleave_batch_size to 3 - 1 patch set | 16:15 |
| clayg | we have it now https://docs.openstack.org/swift/latest/deployment_guide.html#object-reconstructor | 16:52 |
| clayg | Al is the best: https://review.opendev.org/#/c/485569/ | 16:54 |
| patchbot | patch 485569 - swift - Add reconstructor section to deployment guide (MERGED) - 1 patch set | 16:54 |
| clayg | but no sharder!!! https://docs.openstack.org/swift/latest/deployment_guide.html#container-server-configuration | 16:54 |
| *** rpittau is now known as rpittau|afk | 16:57 | |
| acoles | clayg: I knew there was something about sharding that had been left undone....been bugging me all this time :D | 17:07 |
| ormandj | thank you again for the invite btw, it was great to meet you all | 17:08 |
| acoles | ormandj: good to meet you too, thanks for all your input | 17:11 |
| *** psachin has joined #openstack-swift | 17:13 | |
| acoles | clayg: no nam page for swift-manage-shard-ranges either | 17:19 |
| acoles | s/nam/man/ | 17:19 |
| clayg | 🙄 | 17:20 |
| *** psachin has quit IRC | 18:36 | |
| *** cschwede has quit IRC | 18:51 | |
| *** ccamel has quit IRC | 19:35 | |
| *** gyee has joined #openstack-swift | 19:57 | |
| *** camelCaser has joined #openstack-swift | 21:13 | |
| timburke | zigo, https://github.com/eventlet/eventlet/pull/611 should fix the start-up logging, but i don't think that's the cause for the hang | 21:14 |
| timburke | zigo, i've observed something similar, though; wrote up https://bugs.launchpad.net/swift/+bug/1895739 | 21:15 |
| openstack | Launchpad bug 1895739 in OpenStack Object Storage (swift) "Proxy server sometimes deadlocks while logging client disconnect" [Undecided,In progress] | 21:15 |
| zigo | timburke: Only with ussuri and on, right? | 21:15 |
| timburke | it seems to be some combination of python version, eventlet, and possibly swift (which is doing its own monkey-patching in places) | 21:16 |
| ormandj | timburke: did you see above link re: monkey patching? | 21:17 |
| timburke | iirc, DHE solved it by explicitly turning off logging.logThreads | 21:17 |
| zigo | timburke: DHE ? | 21:18 |
| zigo | Oh ... | 21:18 |
| zigo | timburke: How would I do that? | 21:18 |
| timburke | the other person i've seen report the same issue | 21:18 |
| zigo | timburke: How do I "explicitly turn off logging.logThreads" | 21:19 |
| zigo | ? | 21:19 |
| timburke | either hack up stdlib (https://github.com/python/cpython/blob/v3.7.9/Lib/logging/__init__.py#L66) or hack up swift to mokey-patch it (in swift/__init__.py or swift/common/utils.py, maybe?) | 21:20 |
| timburke | i've not abandoned all hope of finding a satisfactory fix (yet), but that's probably the fastest way to get to a more-stable proxy | 21:21 |
| zigo | Thanks, I'll try and let you know. | 21:21 |
| zigo | I did try to hack in swift/common/utils.py a little bit ... | 21:22 |
| zigo | eventlet.patcher.monkey_patch(all=False, socket=True, select=True, | 21:22 |
| zigo | thread=True) | 21:22 |
| zigo | + import __original_module_threading as orig_threading | 21:22 |
| zigo | + import threading | 21:22 |
| zigo | + orig_threading.current_thread.__globals__['_active'] = threading._active | 21:22 |
| zigo | This is what I did... | 21:22 |
| zigo | Just like in Nova. | 21:23 |
| zigo | But that's not fixing things. | 21:23 |
| timburke | if you want confirmation of the issue, you might try running https://github.com/swiftstack/python-stack-xray/blob/master/python-stack-xray against a stuck process and look for a stack with a double-call into eventlet.green.threading.current_thread | 21:25 |
| zigo | timburke: Is there a way to have swift-proxy run over uwsgi? | 21:25 |
| zigo | I'd very much welcome it, the multiple-interpreter thingy helps a log for this kind of pb. | 21:26 |
| timburke | probably? i seem to recall some non-eventlet-wsgi-server stuff in https://github.com/openstack/swift/tree/master/examples and https://docs.openstack.org/swift/latest/apache_deployment_guide.html that might be a starting point? | 21:28 |
| timburke | (fwiw, i also wrote up https://github.com/eventlet/eventlet/issues/647 to try to solicit some help/ideas -- haven't gotten around to trying temoto's code to try to repro yet, though) | 21:31 |
| zigo | Thanks, I'll try. | 21:33 |
| mattoliverau | What no man page for manage-shard-ranges. Now I not only have to trim a bunch but create one :p | 21:36 |
| DHE | zigo: me | 21:58 |
| zigo | DHE: Yeah, got that ! :) Sorry. | 22:07 |
| zigo | DHE: tmd tla ... :P | 22:07 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Add option to REPLICATE to just invalidate hashes https://review.opendev.org/758636 | 22:18 |
| DHE | and yes I edited logging/__init__.py and I do so with zero shame | 22:18 |
| DHE | then again this is a python build/install dedicated to swift so I don't mind | 22:19 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Do not require the close method for response iterator https://review.opendev.org/759602 | 22:40 |
| openstackgerrit | Tim Burke proposed openstack/swift master: tests: Replace FakeLogger with debug_logger in test_proxy_logging https://review.opendev.org/760466 | 22:40 |
| zigo | DHE: This is *not* a configuration file, from the distro perspective, so that's an annoying thing to do for me. I just did it in a production server though, to see how it goes, I'll be able to tell what the result is tomorrow morning. | 22:51 |
| zigo | If it fixes *or not*, I'll probably switch the swift packages to uwsgi anyways (if I can...). | 22:51 |
| *** rcernin has joined #openstack-swift | 22:58 | |
| *** rcernin has quit IRC | 23:00 | |
| *** rcernin has joined #openstack-swift | 23:00 | |
| openstackgerrit | Merged openstack/swift master: Imported Translations from Zanata https://review.opendev.org/760321 | 23:25 |
| openstackgerrit | Tim Burke proposed openstack/swift master: s3api: Add stats for error codes returned https://review.opendev.org/760469 | 23:28 |
| timburke | when we change the statsd prefix at https://github.com/openstack/swift/blob/2.26.0/swift/proxy/server.py#L537-L538 -- who's responsible for *undoing* that? that logger is shared between requests... how certain are we that it's anywhere near accurate? | 23:28 |
| timburke | 'cause if i don't reset the statsd prefix in that patch i just submitted, i *sometimes* get stats like proxy-server.s3api.400.AuthorizationHeaderMalformed and *sometimes* like proxy-server.account.s3api.400.AuthorizationHeaderMalformed ... and that's not great | 23:29 |
| *** dsariel has quit IRC | 23:57 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!