| *** mikecmpbll has joined #openstack-swift | 00:26 | |
| openstackgerrit | Merged openstack/swift-bench master: Change openstack-dev to openstack-discuss https://review.openstack.org/622832 | 00:38 |
|---|---|---|
| *** gyee has quit IRC | 00:45 | |
| timburke | i wish torgomatic was here :-/ | 01:39 |
| timburke | for a variety of reason, and nearly all of them good, but right now, i really wanna know why we did https://github.com/openstack/swift/blob/2.20.0/swift/common/utils.py#L1897-L1898 and carved out an exception for our prefixing iif we call logger.exception() | 01:40 |
| timburke | logger.error(..., exc_info=True)? sure, get a prefix. logger.exception(...)? nope. | 01:41 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Let developers/operators add watchers to object audit https://review.openstack.org/212824 | 01:43 |
| *** mikecmpbll has quit IRC | 01:44 | |
| *** psachin has joined #openstack-swift | 03:04 | |
| *** psachin has quit IRC | 03:07 | |
| *** psachin has joined #openstack-swift | 03:08 | |
| *** lifeless has joined #openstack-swift | 03:10 | |
| *** two_tired has joined #openstack-swift | 03:15 | |
| *** mahatic has quit IRC | 04:06 | |
| *** two_tired has quit IRC | 04:57 | |
| *** rcernin has quit IRC | 06:47 | |
| *** ccamacho has joined #openstack-swift | 08:01 | |
| *** mikecmpbll has joined #openstack-swift | 08:03 | |
| *** mikecmpbll has joined #openstack-swift | 09:06 | |
| *** [diablo] has joined #openstack-swift | 09:38 | |
| *** e0ne has joined #openstack-swift | 09:52 | |
| *** lifeless has quit IRC | 12:42 | |
| *** mvkr has quit IRC | 13:57 | |
| *** mvkr has joined #openstack-swift | 14:12 | |
| *** two_tired has joined #openstack-swift | 14:48 | |
| *** two_tired has quit IRC | 15:32 | |
| *** psachin has quit IRC | 16:32 | |
| *** itlinux has joined #openstack-swift | 16:33 | |
| *** e0ne has quit IRC | 16:36 | |
| timburke | good morning | 17:00 |
| *** mikecmpbll has quit IRC | 17:11 | |
| notmyname | good morning | 17:15 |
| *** e0ne has joined #openstack-swift | 17:43 | |
| *** e0ne has quit IRC | 17:44 | |
| *** mikecmpbll has joined #openstack-swift | 18:02 | |
| *** gyee has joined #openstack-swift | 18:04 | |
| *** ccamacho has quit IRC | 18:07 | |
| *** itlinux_ has joined #openstack-swift | 18:33 | |
| *** itlinux has quit IRC | 18:35 | |
| clayg | We run probetests in the gate right? I think I'd like to maybe have more EC disks (12 disks for a 4+2 policy instead of only 8) | 19:14 |
| timburke | clayg: yeah, we do | 19:17 |
| tdasilva | clayg, timburke: i think we just need to change this: https://github.com/openstack/swift/blob/master/tools/playbooks/multinode_setup/configure_loopback.yaml#L63 | 19:22 |
| clayg | Cool! I’ll see about getting tests working with extra disks first... then test the change. | 19:23 |
| *** itlinux_ has quit IRC | 19:40 | |
| *** lifeless has joined #openstack-swift | 20:12 | |
| zaitcev | Oh boy | 20:35 |
| zaitcev | Suddenly, the zero-copy send in object server... does not work on py3 with rather strange effects. | 20:36 |
| zaitcev | def __len__(self): return wsgi.MINIMUM_CHUNK_SIZE + 1 | 20:36 |
| timburke | wheee! | 20:40 |
| *** itlinux has joined #openstack-swift | 20:44 | |
| timburke | yeah, looking at https://review.openstack.org/#/c/102609/ i fell like its mostly black magic. maybe it comes down to a difference in sockets when we call `wsock = env['wsgi.input'].get_socket(); wsockfd = wsock.fileno()`? no idea | 20:52 |
| patchbot | patch 102609 - swift - Zero-copy object-server GET responses with splice() (MERGED) - 11 patch sets | 20:52 |
| zaitcev | I thought maybe corking... Because in py3 apparently bodies come out before headers. | 20:59 |
| timburke | whoa... ok, yeah, that'd make some sense... | 21:00 |
| timburke | does it work any better if we drop the corking entirely? | 21:01 |
| zaitcev | I'm going to test | 21:04 |
| zaitcev | One thing though, I don't understand what the corking is for to begin with. It bothers me, I don't want just slash and burn randomly. | 21:06 |
| timburke | reading the comments, it seems to try to avoid sending headers, then sending object data; rather, we want to send headers + start of object data, then rest of object data. just to try to cut down on # of packets. | 21:08 |
| timburke | i like "working" more than "broken but optimized" | 21:09 |
| zaitcev | Oooh, so it corks the rest of headers | 21:10 |
| zaitcev | Well, as you say. | 21:10 |
| timburke | i wonder if we should (now? maybe this wasn't an option before?) be doing something with env['eventlet.minimum_write_chunk_size'] instead of that crazy EventletPlungerString guy... | 21:14 |
| timburke | ugh, and if we keep it, it *definitely* needs to be subclassing bytes, not str... | 21:16 |
| zaitcev | modern eventlet does if isinstance(data, six.text_type): data = data.encode('ascii') | 21:17 |
| zaitcev | which is kinda amazing | 21:17 |
| timburke | ...so we lose the __len__ specialness! there we go | 21:18 |
| zaitcev | unfortunately if we set eventlet.minimum_write_chunk_size, then all bufferization is defeated | 21:18 |
| timburke | data *was* our plunger guy, then it was plain-old bytes, so towrite_size += len(data) doesn't do what we want | 21:19 |
| zaitcev | I didn't know how eventlet's wsgi worked | 21:19 |
| zaitcev | I thought that once you return from app, it posted headers, then cycled through chunks of what app returned (an iterator or whatever). | 21:20 |
| timburke | but all of eventlet's bufferization doesn't matter at this point, yeah? drop it to zero just before we call res(env, start_response) | 21:20 |
| timburke | yeah, there's no guarantee that an app will have actually *called* start_response until you get at least one non-empty chunk out of the iterator :-/ | 21:21 |
| zaitcev | But nope... It goes cycling right away. So, it receives our plunger string. THEN, it calls a fuction write()... which... is not a write at all. It's a big thing that looks if headers were sent, and if not, preps and sends headers. | 21:21 |
| timburke | wsgi servers get a lot of leeway when it comes to deciding when to send how much data | 21:21 |
| zaitcev | The core of the problem is, we want to splice in iterator, but we only want to do it once the header data is sent to socket. | 21:22 |
| zaitcev | So corking does not actually do anything | 21:23 |
| zaitcev | (it seems to me) | 21:23 |
| timburke | i swear, if the golang stuff doesn't move at some point, clayg's gonna go write a sweventlet or swsgi or something (a la swob) | 21:23 |
| timburke | zaitcev: yeah, i'm 99% sure it's the plunger inheriting from str, not bytes | 21:23 |
| zaitcev | well maybe twsgi right away, skip the s | 21:23 |
| clayg | eventlet is only terrible if you have to do blocking io 😢 | 21:24 |
| clayg | and it's not like lightweight concurrency isn't nice to have even when you DO have to do blocking io | 21:25 |
| clayg | golang is kinda great for that - despite all other evidence that it is also terrible | 21:25 |
| timburke | clayg: idk, you might still want to write your own wsgi server ;-) | 21:25 |
| clayg | i don't really like WSGI, it skipped some stuff that web servers have to think about that I later found out I cared about (like slow clients, and pipelined requests) | 21:26 |
| timburke | could still use eventlet. but it'd be nice if we didn't have to do something like EventletPlungerString | 21:26 |
| timburke | true... but the 3rd party middlewares :-( | 21:27 |
| clayg | yeah that's non-sense - but WSGI doesn't really like you to think to hard about zero-copy - again WSGI is also terrible | 21:28 |
| zaitcev | timburke: if I comment corking out, there's no change in behavior. I'll need to find out what's going on for real. | 21:28 |
| clayg | right... 3rd party middleware... nm WSGI is the only piece of software that isn't terrible! | 21:29 |
| clayg | zaitcev: no body uses that zero-copy non-sense - let's just deprecate it and say it's not supported in py3 | 21:29 |
| timburke | zaitcev: have you tried the bytes thing? EventletPlungerString().encode('ascii') will give you something whose len() is 0 again... | 21:29 |
| zaitcev | oh | 21:30 |
| zaitcev | haha | 21:30 |
| timburke | clayg: funny... i think i was talking with notmyname the other day about how we seem to be pretty defensive about making sure it'll work before we try it, so we ought to turn it on by default and try to get rid of the option :P | 21:30 |
| clayg | 🙄 | 21:31 |
| clayg | you guys like to make it complicated | 21:31 |
| zaitcev | timburke: thanks a lot, that worked | 21:31 |
| timburke | \o/ | 21:31 |
| timburke | apparently there's one other place where we subclass str... but it shouldn't be bytes. https://github.com/openstack/swift/blob/2.20.0/swift/common/middleware/s3api/s3response.py#L39 + non-ASCII headers scares me, though | 21:37 |
| timburke | [chr(x) for x in range(256) if chr(x) != chr(x).lower()] gives very different results on py2 vs py3... | 21:39 |
| *** itlinux_ has joined #openstack-swift | 21:56 | |
| *** itlinux has quit IRC | 21:59 | |
| *** itlinux_ has quit IRC | 22:24 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: s3api: Delete multipart uploads via multi-delete https://review.openstack.org/628701 | 23:07 |
| *** openstack has joined #openstack-swift | 23:45 | |
| *** ChanServ sets mode: +o openstack | 23:45 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: s3api: Delete multipart uploads via multi-delete https://review.openstack.org/628701 | 23:48 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!