| *** charlesw has quit IRC | 00:02 | |
| *** jeblair is now known as corvus | 00:19 | |
| *** lyrrad has quit IRC | 00:25 | |
| *** lpabon has joined #openstack-swift | 00:26 | |
| *** lpabon has quit IRC | 00:27 | |
| *** annegentle has joined #openstack-swift | 00:39 | |
| *** annegentle has quit IRC | 00:54 | |
| *** gsilvis has joined #openstack-swift | 00:59 | |
| *** B4rker has joined #openstack-swift | 01:00 | |
| *** annegentle has joined #openstack-swift | 01:02 | |
| *** B4rker has quit IRC | 01:05 | |
| *** annegentle has quit IRC | 01:07 | |
| *** gyee has quit IRC | 01:25 | |
| *** tdasilva has joined #openstack-swift | 01:30 | |
| *** torgomatic has quit IRC | 01:34 | |
| *** classicsnail has quit IRC | 01:34 | |
| *** StevenK has quit IRC | 01:35 | |
| *** lyrrad has joined #openstack-swift | 01:37 | |
| *** charlesw has joined #openstack-swift | 01:44 | |
| *** _bluev1 has quit IRC | 01:57 | |
| *** _bluev has joined #openstack-swift | 01:58 | |
| *** B4rker has joined #openstack-swift | 01:59 | |
| *** B4rker has quit IRC | 02:04 | |
| portante | swifterdarrell: have you considered trying out hummingbird branch? | 02:09 |
|---|---|---|
| zaitcev | portante: yes he did | 02:11 |
| portante | and what were the chunk sizes in use? | 02:12 |
| portante | if this is for small file, threads per disk seems like a bad way to go, but if the chunk size is large enough and the object sizes are large enough, 10gb networks etc. ... okay corner cases maybe? | 02:13 |
| * portante this coming from swifterwannabe | 02:13 | |
| *** lyrrad has quit IRC | 02:15 | |
| zaitcev | I dunno, I thought the speedup in case of threads per disk comes from the elevator on the rotating storage. | 02:17 |
| zaitcev | In which case smaller blocks should see a greater speed-up over a single-threaded case. | 02:18 |
| portante | but a single thread participating in the eventlet loop has to dole out I/O via logs and queues, which has an overhead cost | 02:18 |
| portante | for small I/O this cost is likely too high | 02:18 |
| portante | when the eventlet loop blocks on a mutex to sync with another thread, the entire engine will stop | 02:18 |
| zaitcev | oh, right | 02:18 |
| portante | so non-blocking I/O comes to a halt | 02:19 |
| portante | if we were using truly async I/O, that'd be another story | 02:19 |
| zaitcev | okay, so the workers per device is really what I'm thinking about | 02:19 |
| portante | this is one of the traps people fall into when thinking about non-blocking I/O as asynchronous, its not, its still synchronous | 02:19 |
| portante | we found that bumping the worker count high enough so that you cover all your disks (paper presented at a summit a year or so ago) we got a good speed up | 02:20 |
| *** lyrrad has joined #openstack-swift | 02:21 | |
| portante | sounds like swifterdarrell is taking that kind of direction, but trying to solve the port hand-off problem you get with pure object server counts where the kernel does not round robin between the eventlet processes (one eventlet can gobble up all the accepts and defeat the scheme) | 02:21 |
| zaitcev | Both him/Sam/Clay and RAX people complained about severe starvation observed in the field -- requests taking forever to service for no reason | 02:23 |
| *** josed has quit IRC | 02:30 | |
| *** lyrrad has quit IRC | 02:33 | |
| *** lyrrad has joined #openstack-swift | 02:34 | |
| MooingLemur | with thread pools per disk on or off? | 02:38 |
| portante | shouldn't matter | 02:49 |
| portante | if an object server starts accepting when it can't service requests, it will slow it all down | 02:50 |
| portante | in some sense, having a set of credits so that an object server won't issue an accept if it is not completing requests might help | 02:50 |
| *** _bluev1 has joined #openstack-swift | 02:52 | |
| portante | trying lowering the eventlet worker greenlets and raising the number of workers | 02:52 |
| portante | simple but effective | 02:52 |
| *** _bluev has quit IRC | 02:53 | |
| *** _bluev1 has quit IRC | 03:16 | |
| *** david-lyle has quit IRC | 03:25 | |
| *** david-lyle has joined #openstack-swift | 03:26 | |
| *** lpabon has joined #openstack-swift | 03:28 | |
| *** lpabon has quit IRC | 03:33 | |
| *** jamielennox is now known as jamielennox|away | 03:52 | |
| *** zaitcev has quit IRC | 03:59 | |
| *** charlesw has quit IRC | 04:17 | |
| *** lpabon has joined #openstack-swift | 04:30 | |
| *** lpabon has quit IRC | 04:46 | |
| *** SkyRocknRoll has joined #openstack-swift | 04:52 | |
| *** SkyRocknRoll has quit IRC | 06:13 | |
| *** zynisch_o7 has joined #openstack-swift | 06:26 | |
| *** SkyRocknRoll has joined #openstack-swift | 06:26 | |
| *** abcdef has joined #openstack-swift | 06:34 | |
| *** SkyRocknRoll has quit IRC | 06:46 | |
| *** abcdef has quit IRC | 06:57 | |
| *** StevenK has joined #openstack-swift | 06:58 | |
| *** SkyRocknRoll has joined #openstack-swift | 07:00 | |
| *** SkyRocknRoll has quit IRC | 07:06 | |
| *** zynisch_o7 has quit IRC | 07:06 | |
| *** silor has joined #openstack-swift | 07:15 | |
| *** SkyRocknRoll has joined #openstack-swift | 07:18 | |
| *** abcdef has joined #openstack-swift | 07:34 | |
| *** abcdef is now known as abcdefgh | 07:34 | |
| *** cdelatte has quit IRC | 09:52 | |
| *** classicsnail has joined #openstack-swift | 09:55 | |
| *** cdelatte has joined #openstack-swift | 10:02 | |
| *** abcdefgh has quit IRC | 10:05 | |
| *** kutija has quit IRC | 10:20 | |
| *** abcdefgh has joined #openstack-swift | 10:43 | |
| *** abcdefgh is now known as jhijij | 10:50 | |
| *** jhijij has quit IRC | 11:24 | |
| *** bkopilov has joined #openstack-swift | 12:06 | |
| *** SkyRocknRoll has quit IRC | 12:45 | |
| *** SkyRocknRoll has joined #openstack-swift | 12:59 | |
| *** proteusguy has quit IRC | 13:10 | |
| *** HiramAbif has joined #openstack-swift | 13:12 | |
| *** proteusguy has joined #openstack-swift | 13:22 | |
| *** charlesw has joined #openstack-swift | 14:00 | |
| *** breitz has quit IRC | 14:19 | |
| *** zynisch_o7 has joined #openstack-swift | 14:52 | |
| *** zynisch_o7 has quit IRC | 14:56 | |
| *** zynisch_o7 has joined #openstack-swift | 15:07 | |
| *** zynisch_o7 has quit IRC | 15:26 | |
| *** zynisch_o7 has joined #openstack-swift | 15:26 | |
| *** zynisch_o7 has quit IRC | 15:31 | |
| *** _bluev has joined #openstack-swift | 15:51 | |
| *** wbhuber has joined #openstack-swift | 15:59 | |
| *** _bluev has quit IRC | 16:06 | |
| *** wbhuber has quit IRC | 16:18 | |
| *** zynisch_o7 has joined #openstack-swift | 17:14 | |
| *** zynisch_o7 has quit IRC | 17:20 | |
| *** josed has joined #openstack-swift | 17:27 | |
| *** geaaru has joined #openstack-swift | 17:50 | |
| *** openstackgerrit has quit IRC | 17:51 | |
| *** openstackgerrit has joined #openstack-swift | 17:51 | |
| *** jrichli has joined #openstack-swift | 17:53 | |
| *** charlesw has quit IRC | 18:05 | |
| *** torgomatic has joined #openstack-swift | 18:22 | |
| *** ChanServ sets mode: +v torgomatic | 18:22 | |
| *** silor1 has joined #openstack-swift | 18:55 | |
| *** silor has quit IRC | 18:57 | |
| *** wbhuber has joined #openstack-swift | 19:18 | |
| *** _bluev has joined #openstack-swift | 19:25 | |
| *** josed has quit IRC | 19:26 | |
| *** charlesw has joined #openstack-swift | 19:51 | |
| *** _bluev has quit IRC | 19:55 | |
| *** _bluev has joined #openstack-swift | 20:02 | |
| *** silor1 has quit IRC | 20:16 | |
| *** zynisch_o7 has joined #openstack-swift | 20:16 | |
| *** _bluev has quit IRC | 20:21 | |
| *** zynisch_o7 has quit IRC | 20:21 | |
| *** wbhuber has quit IRC | 20:24 | |
| *** charlesw has quit IRC | 20:30 | |
| *** wbhuber has joined #openstack-swift | 20:32 | |
| *** wbhuber has quit IRC | 20:43 | |
| *** otoolee has quit IRC | 21:00 | |
| *** otoolee has joined #openstack-swift | 21:03 | |
| *** _bluev has joined #openstack-swift | 21:09 | |
| *** jrichli has quit IRC | 21:29 | |
| *** _bluev has quit IRC | 21:31 | |
| *** SkyRocknRoll has quit IRC | 21:44 | |
| *** zynisch_o7 has joined #openstack-swift | 22:04 | |
| *** zynisch_o7 has quit IRC | 22:08 | |
| *** _bluev has joined #openstack-swift | 22:12 | |
| *** jamielennox|away is now known as jamielennox | 22:17 | |
| *** wbhuber has joined #openstack-swift | 22:20 | |
| swifterdarrell | portante: I think you might be conflating 2+ things? | 22:38 |
| swifterdarrell | portante: for "normal" object-server, a single slow drive can hose an entire object server (Intel showed this pretty conclusively 1 to 3 conferences ago, the one before Hong Kong maybe?) | 22:39 |
| swifterdarrell | portante: that's because any object-server can issue an I/O to the crappy disk and starve that object-server eventlet hub, introducing latency into all other pending I/Os to all other disks that same object-server unix proc was handling | 22:40 |
| swifterdarrell | portante: that's crappy, and large clusters see this (as zaitcev mentioned) | 22:41 |
| swifterdarrell | portante: threads_per_disk gives full I/O isolation (a bad disk will only introduce latency into requests dealing with that disk), but the overhead is pretty high | 22:42 |
| swifterdarrell | portante: (too high, really) | 22:42 |
| swifterdarrell | portante: so the current idea is to have N object-server unix procs running which only handle reqs for a single disk (achieved by having unique ports for each disk in each server in the ring) | 22:43 |
| swifterdarrell | portante: this should give the same I/O isolation as thread_per_disk, with less overhead, and what I'm *trying* to get at is also increased utilization of the disks when under heavy load than even the "normal" (manifested as increased req/s, apples-to-apples) | 22:44 |
| swifterdarrell | portante: I'll have some numbers soon enough | 22:44 |
| *** geaaru has quit IRC | 22:53 | |
| openstackgerrit | Darrell Bishop proposed openstack/swift: Allow 1+ object-servers-per-disk deployment https://review.openstack.org/184189 | 23:02 |
| torgomatic | poo, looks like readv2() has not yet landed in Linux (if it's going to at all) | 23:04 |
| torgomatic | "git log -S readv2" doesn't turn up anything | 23:04 |
| * torgomatic was looking forward to that one | 23:04 | |
| *** wbhuber has quit IRC | 23:10 | |
| openstackgerrit | Darrell Bishop proposed openstack/swift: Allow 1+ object-servers-per-disk deployment https://review.openstack.org/184189 | 23:35 |
| swifterdarrell | torgomatic: fancy new kernel features look less exciting when you consider how long it is until they're available in some popular enterprise distros (*cough*RHEL*cough*) | 23:38 |
| swifterdarrell | torgomatic: :( | 23:38 |
| *** zynisch_o7 has joined #openstack-swift | 23:53 | |
| *** zynisch_o7 has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!