| *** haomaiwa_ has quit IRC | 00:01 | |
| *** haomaiwa_ has joined #openstack-swift | 00:01 | |
| *** resker has joined #openstack-swift | 00:08 | |
| *** asettle has joined #openstack-swift | 00:10 | |
| *** garthb has quit IRC | 00:11 | |
| *** esker has quit IRC | 00:12 | |
| *** chuck_ has joined #openstack-swift | 00:17 | |
| *** gyee has quit IRC | 00:26 | |
| *** lyrrad has quit IRC | 00:33 | |
| *** arch-nemesis has quit IRC | 00:43 | |
| *** esker has joined #openstack-swift | 00:47 | |
| kota_ | good morning | 00:49 |
|---|---|---|
| *** resker has quit IRC | 00:51 | |
| notmyname | hello kota_ | 00:52 |
| kota_ | notmyname: hello | 00:52 |
| *** asettle has quit IRC | 00:54 | |
| hosanai | kota_: notmyname: morning! & hello! | 00:54 |
| notmyname | hello hosanai | 00:54 |
| kota_ | hosanai: o/ | 00:54 |
| *** haomaiwa_ has quit IRC | 01:01 | |
| *** haomaiwang has joined #openstack-swift | 01:01 | |
| *** asettle has joined #openstack-swift | 01:04 | |
| *** haomai___ has joined #openstack-swift | 01:06 | |
| *** haomaiwang has quit IRC | 01:08 | |
| *** asettle has quit IRC | 01:08 | |
| *** klrmn has quit IRC | 01:12 | |
| *** esker has quit IRC | 01:14 | |
| *** chuck_ is now known as zu | 01:18 | |
| *** zu is now known as zul | 01:18 | |
| *** mingdang1 has joined #openstack-swift | 01:19 | |
| *** asettle has joined #openstack-swift | 01:23 | |
| *** takashi_ has joined #openstack-swift | 01:24 | |
| takashi_ | good morning :-) | 01:24 |
| *** jamielennox|away is now known as jamielennox | 01:29 | |
| *** bill_az has quit IRC | 01:34 | |
| *** mathiasb has quit IRC | 01:47 | |
| *** mathiasb has joined #openstack-swift | 01:47 | |
| *** StraubTW has joined #openstack-swift | 01:49 | |
| *** bill_az has joined #openstack-swift | 01:52 | |
| *** mingdang1 has quit IRC | 01:54 | |
| *** haomai___ has quit IRC | 02:01 | |
| *** haomaiwang has joined #openstack-swift | 02:01 | |
| *** haomaiwang has quit IRC | 02:03 | |
| *** 7F1AAMJH0 has joined #openstack-swift | 02:08 | |
| *** StraubTW has quit IRC | 02:09 | |
| *** rickyrem has quit IRC | 02:10 | |
| *** StraubTW has joined #openstack-swift | 02:11 | |
| *** 7F1AAMJH0 has quit IRC | 02:13 | |
| *** klrmn has joined #openstack-swift | 02:14 | |
| *** bill_az has quit IRC | 02:23 | |
| *** baojg has joined #openstack-swift | 02:29 | |
| *** lifeless has quit IRC | 02:43 | |
| *** lifeless has joined #openstack-swift | 02:44 | |
| *** baojg has quit IRC | 02:45 | |
| *** baojg has joined #openstack-swift | 02:46 | |
| *** haomaiwang has joined #openstack-swift | 02:52 | |
| *** sheel has joined #openstack-swift | 02:54 | |
| *** sgundur has left #openstack-swift | 02:57 | |
| *** sanchitmalhotra has joined #openstack-swift | 02:59 | |
| *** haomaiwang has quit IRC | 03:01 | |
| *** haomaiwang has joined #openstack-swift | 03:01 | |
| *** StraubTW has quit IRC | 03:04 | |
| *** dmorita has quit IRC | 03:11 | |
| *** links has joined #openstack-swift | 03:13 | |
| charz_ | morning | 03:25 |
| janonymous | o/ | 03:26 |
| *** dmorita has joined #openstack-swift | 03:44 | |
| *** dmorita has quit IRC | 03:48 | |
| *** asettle has quit IRC | 03:58 | |
| *** haomaiwang has quit IRC | 04:01 | |
| *** 7GHAAMTT3 has joined #openstack-swift | 04:01 | |
| *** sanchitmalhotra1 has joined #openstack-swift | 04:18 | |
| *** sanchitmalhotra has quit IRC | 04:20 | |
| *** SkyRocknRoll has joined #openstack-swift | 04:40 | |
| *** 7GHAAMTT3 has quit IRC | 05:01 | |
| *** haomaiwang has joined #openstack-swift | 05:01 | |
| *** pcaruana has quit IRC | 05:09 | |
| *** asettle has joined #openstack-swift | 05:12 | |
| *** sanchitmalhotra1 has quit IRC | 05:13 | |
| *** baojg has quit IRC | 05:24 | |
| *** baojg has joined #openstack-swift | 05:25 | |
| *** trifon has joined #openstack-swift | 05:28 | |
| *** silor has joined #openstack-swift | 05:35 | |
| *** asettle has quit IRC | 05:39 | |
| *** silor has quit IRC | 05:43 | |
| *** silor1 has joined #openstack-swift | 05:43 | |
| *** silor1 is now known as silor | 05:45 | |
| *** klrmn has quit IRC | 05:47 | |
| *** ChubYann has quit IRC | 05:51 | |
| *** silor has quit IRC | 05:54 | |
| *** silor has joined #openstack-swift | 05:54 | |
| *** asettle has joined #openstack-swift | 05:54 | |
| *** asettle has quit IRC | 05:59 | |
| *** haomaiwang has quit IRC | 06:01 | |
| *** haomaiwang has joined #openstack-swift | 06:01 | |
| *** tesseract has joined #openstack-swift | 06:19 | |
| *** tesseract is now known as Guest58782 | 06:19 | |
| *** haomaiwang has quit IRC | 07:01 | |
| *** haomaiwang has joined #openstack-swift | 07:01 | |
| *** rledisez has joined #openstack-swift | 07:11 | |
| *** zaitcev has quit IRC | 07:13 | |
| *** sanchitmalhotra has joined #openstack-swift | 07:15 | |
| *** jordanP has joined #openstack-swift | 07:21 | |
| *** dmellado|off is now known as dmellado | 07:36 | |
| *** rcernin has joined #openstack-swift | 07:36 | |
| *** mingdang1 has joined #openstack-swift | 07:37 | |
| *** haomaiwang has quit IRC | 07:39 | |
| *** mingdang1 has quit IRC | 07:39 | |
| *** mingdang1 has joined #openstack-swift | 07:40 | |
| *** haomaiwang has joined #openstack-swift | 07:43 | |
| *** SkyRocknRoll has quit IRC | 07:46 | |
| *** haomaiwang has quit IRC | 07:48 | |
| *** pcaruana has joined #openstack-swift | 07:48 | |
| *** daemontool has joined #openstack-swift | 07:50 | |
| *** haomaiwa_ has joined #openstack-swift | 07:53 | |
| *** SkyRocknRoll has joined #openstack-swift | 07:56 | |
| *** mmcardle has joined #openstack-swift | 07:56 | |
| *** daemontool_ has joined #openstack-swift | 07:58 | |
| *** haomaiwa_ has quit IRC | 08:01 | |
| *** haomaiwang has joined #openstack-swift | 08:01 | |
| *** daemontool has quit IRC | 08:02 | |
| openstackgerrit | Marek Kaleta proposed openstack/swift: Order devices in the output of swift-ring-builder https://review.openstack.org/277956 | 08:17 |
| *** joeljwright has joined #openstack-swift | 08:19 | |
| *** ChanServ sets mode: +v joeljwright | 08:19 | |
| *** mingdang1 has quit IRC | 08:21 | |
| *** jistr has joined #openstack-swift | 08:23 | |
| *** asettle has joined #openstack-swift | 08:26 | |
| *** jmccarthy has quit IRC | 08:38 | |
| *** jmccarthy has joined #openstack-swift | 08:38 | |
| *** daemontool_ has quit IRC | 08:43 | |
| *** daemontool_ has joined #openstack-swift | 08:43 | |
| *** mvk has joined #openstack-swift | 08:46 | |
| *** haomaiwang has quit IRC | 09:01 | |
| *** haomaiwang has joined #openstack-swift | 09:01 | |
| *** stantonnet has quit IRC | 09:02 | |
| *** stantonnet has joined #openstack-swift | 09:05 | |
| *** sileht has quit IRC | 09:07 | |
| *** sileht has joined #openstack-swift | 09:12 | |
| *** asettle has quit IRC | 09:27 | |
| *** takashi_ has quit IRC | 09:36 | |
| *** lifeless has quit IRC | 09:39 | |
| *** lifeless has joined #openstack-swift | 09:40 | |
| *** baojg has quit IRC | 09:42 | |
| *** baojg has joined #openstack-swift | 09:42 | |
| *** baojg has quit IRC | 09:44 | |
| *** baojg has joined #openstack-swift | 09:45 | |
| *** asettle has joined #openstack-swift | 09:52 | |
| *** daemontool_ has quit IRC | 09:55 | |
| *** km__ has joined #openstack-swift | 09:55 | |
| *** km__ has quit IRC | 09:55 | |
| *** daemontool_ has joined #openstack-swift | 09:55 | |
| *** daemontool_ has quit IRC | 09:56 | |
| *** daemontool_ has joined #openstack-swift | 09:56 | |
| *** km__ has joined #openstack-swift | 09:56 | |
| *** km__ is now known as Guest83701 | 09:57 | |
| *** asettle has quit IRC | 09:57 | |
| *** km has quit IRC | 09:58 | |
| *** haomaiwang has quit IRC | 10:01 | |
| *** haomaiwang has joined #openstack-swift | 10:01 | |
| *** Guest83701 has quit IRC | 10:07 | |
| *** km has joined #openstack-swift | 10:08 | |
| *** km has quit IRC | 10:11 | |
| *** km has joined #openstack-swift | 10:12 | |
| *** hosanai has quit IRC | 10:14 | |
| *** km has quit IRC | 10:17 | |
| *** km has joined #openstack-swift | 10:20 | |
| openstackgerrit | Merged openstack/swift: Fix py34 error of indexing 'dict_keys' object https://review.openstack.org/292096 | 10:21 |
| *** km has quit IRC | 10:27 | |
| *** km has joined #openstack-swift | 10:28 | |
| *** flaper87 has quit IRC | 10:36 | |
| *** flaper87 has joined #openstack-swift | 10:36 | |
| *** kei_yama has quit IRC | 10:37 | |
| *** km has quit IRC | 10:40 | |
| *** asettle has joined #openstack-swift | 10:56 | |
| *** asettle has quit IRC | 10:56 | |
| *** baojg has quit IRC | 10:59 | |
| *** haomaiwang has quit IRC | 11:01 | |
| *** haomaiwang has joined #openstack-swift | 11:01 | |
| *** dmorita has joined #openstack-swift | 11:12 | |
| *** mvk_ has joined #openstack-swift | 11:15 | |
| *** dmorita has quit IRC | 11:17 | |
| *** mvk has quit IRC | 11:19 | |
| *** mingdang1 has joined #openstack-swift | 11:33 | |
| *** mmcardle has quit IRC | 11:37 | |
| *** cbartz has joined #openstack-swift | 11:48 | |
| *** asettle has joined #openstack-swift | 11:56 | |
| *** haomaiwang has quit IRC | 12:01 | |
| *** haomaiwa_ has joined #openstack-swift | 12:01 | |
| *** mmcardle has joined #openstack-swift | 12:02 | |
| *** SkyRocknRoll has quit IRC | 12:06 | |
| *** baojg has joined #openstack-swift | 12:36 | |
| *** natarej has quit IRC | 12:39 | |
| *** dmellado is now known as dmellado|lunch | 12:40 | |
| *** StraubTW has joined #openstack-swift | 12:47 | |
| *** jmb___ has quit IRC | 12:53 | |
| *** kozhukalov has joined #openstack-swift | 12:56 | |
| *** StraubTW has quit IRC | 12:56 | |
| *** haomaiwa_ has quit IRC | 13:01 | |
| *** haomaiwang has joined #openstack-swift | 13:01 | |
| *** links has quit IRC | 13:02 | |
| tdasilva | good morning | 13:04 |
| *** natarej has joined #openstack-swift | 13:07 | |
| *** bill_az has joined #openstack-swift | 13:19 | |
| *** haomaiwang has quit IRC | 13:22 | |
| *** ametts has joined #openstack-swift | 13:22 | |
| *** StraubTW has joined #openstack-swift | 13:25 | |
| *** jmb__ has joined #openstack-swift | 13:27 | |
| pdardeau | good morning | 13:35 |
| vinsh_ | Good morning all. I just noticed, when running servers per disk - "lsof -i" of any given port in the ring shows something strange. I have 48 disks in this node, so 48 ports in the ring for that node 1 per disk. With servers per port setting of "1" I see 48 threads using lsof -i PER disk. | 13:40 |
| vinsh_ | If I bump servers per disk to 4 .. thats 236 threads open. | 13:40 |
| vinsh_ | so 48 threads PER port in the ring? | 13:40 |
| vinsh_ | I would expect to see just 1 or 4 based on the setting. | 13:41 |
| *** diogogmt has quit IRC | 13:43 | |
| vinsh_ | wonder if I am seeing https://bugs.launchpad.net/swift/+bug/1554233 ? | 13:44 |
| openstack | Launchpad bug 1554233 in OpenStack Object Storage (swift) "Servers-per-port can consume excessive OS threads" [High,In progress] - Assigned to Samuel Merritt (torgomatic) | 13:44 |
| vinsh_ | but the number of files open per port matches the number of disks each. | 13:44 |
| *** mingdang1 has quit IRC | 13:45 | |
| vinsh_ | I should clarify that its not threads I see.. its open files. (big difference) | 13:46 |
| vinsh_ | as ps -ef output shows what I would expect from servers per disk. I guess I just don't get the lsof -i output. :) | 13:46 |
| *** diogogmt has joined #openstack-swift | 13:47 | |
| vinsh_ | Not mounted: d175 on 24.26.92.216:6021 | 13:47 |
| vinsh_ | Not mounted: d174 on 24.26.92.216:6021 | 13:47 |
| vinsh_ | The reason I ask, is I see really bizzare stuff out of swift-recon on this deployment. Claiming ^^ | 13:47 |
| vinsh_ | where those disks are NOT even on that host. | 13:48 |
| vinsh_ | swift-ring-builder output confirms | 13:48 |
| *** asettle has quit IRC | 13:49 | |
| *** asettle has joined #openstack-swift | 13:55 | |
| *** asettle has quit IRC | 14:00 | |
| *** DevStok has joined #openstack-swift | 14:00 | |
| DevStok | hi all | 14:00 |
| DevStok | I have a env deployed with Fuel 7 | 14:01 |
| DevStok | HA | 14:01 |
| DevStok | glance + swift | 14:01 |
| DevStok | when I upload a lot of snapshots | 14:01 |
| DevStok | the folder /srv/node/ fill up the partition / | 14:01 |
| DevStok | I want to change swift conf /srv/node to other path with more space | 14:02 |
| DevStok | any ideas? | 14:02 |
| *** diogogmt has quit IRC | 14:05 | |
| *** daemontool_ is now known as daemontool | 14:06 | |
| *** trifon has quit IRC | 14:06 | |
| *** trifon has joined #openstack-swift | 14:06 | |
| *** sgundur has joined #openstack-swift | 14:12 | |
| *** trifon has quit IRC | 14:13 | |
| *** cbartz has quit IRC | 14:15 | |
| *** dmellado|lunch is now known as dmellado | 14:21 | |
| *** ajiang has left #openstack-swift | 14:30 | |
| *** baojg has quit IRC | 14:34 | |
| *** haomaiwang has joined #openstack-swift | 14:43 | |
| *** kozhukalov has quit IRC | 14:51 | |
| *** kozhukalov has joined #openstack-swift | 14:51 | |
| mmotiani | Hi Good morning! | 14:57 |
| mmotiani | I am planning to fill gaps in the swift client docs. | 14:58 |
| mmotiani | timburke: Would you be able to give me some suggestions? | 15:00 |
| *** haomaiwang has quit IRC | 15:01 | |
| *** haomaiwang has joined #openstack-swift | 15:01 | |
| *** arch-nemesis has joined #openstack-swift | 15:05 | |
| *** openstackgerrit has quit IRC | 15:06 | |
| *** arch-nemesis has quit IRC | 15:07 | |
| *** openstackgerrit has joined #openstack-swift | 15:07 | |
| joeljwright | mmotiani: hi, which gaps are you hoping to work on? | 15:10 |
| *** _JZ_ has joined #openstack-swift | 15:10 | |
| joeljwright | we have a patch in the pipeline to add in quite a bit of missing detail | 15:11 |
| mmotiani | joeljwright: I can see lot of docs only contains the heading and there is no content for them. | 15:11 |
| mmotiani | So, planning to work on them. | 15:11 |
| *** diogogmt has joined #openstack-swift | 15:11 | |
| joeljwright | let me point you at a current patch (there are still missing bits, but it's much more complete!) | 15:12 |
| joeljwright | just need to find it... | 15:12 |
| mmotiani | Could you please share the link of the patch | 15:12 |
| mmotiani | Yeah, that would be great. | 15:12 |
| *** dmorita has joined #openstack-swift | 15:12 | |
| joeljwright | https://review.openstack.org/#/c/288566/ | 15:14 |
| patchbot | joeljwright: patch 288566 - python-swiftclient - WIP: This patch adds a new doc structure for swift... | 15:14 |
| *** arch-nemesis has joined #openstack-swift | 15:14 | |
| notmyname | good morning | 15:15 |
| joeljwright | mmotiani: I'm working on the patch at the moment to address the existing comments | 15:15 |
| notmyname | mmotiani: great to see interest in improving the client docs! :-) | 15:15 |
| mmotiani | joeljwright: thanks! I will just go through the patch and comments to see how to get start with it. | 15:16 |
| mmotiani | notmyname: good morning :) | 15:16 |
| notmyname | wbhuber: on https://bugs.launchpad.net/swift/+bug/1563362 are you proposing to add full if-match etag support? I'm not entirely sure what that bug is reporting/proposing | 15:16 |
| openstack | Launchpad bug 1563362 in OpenStack Object Storage (swift) "Doing a PUT on a valid conditional request, If-None-Match, returns unclear statement" [Undecided,New] | 15:16 |
| *** siva_krishnan has left #openstack-swift | 15:16 | |
| *** dmorita has quit IRC | 15:17 | |
| *** diogogmt has quit IRC | 15:17 | |
| *** diogogmt has joined #openstack-swift | 15:18 | |
| joeljwright | mmotiani: I'll push up an update later today that addresses the existing comments from asettle (and adds another section+example to the swiftservice api part) | 15:19 |
| *** garthb has joined #openstack-swift | 15:19 | |
| *** siva_krishnan has joined #openstack-swift | 15:19 | |
| *** siva_krishnan has left #openstack-swift | 15:19 | |
| mmotiani | joeljwright: Ok, I will look into that too. Thanks! | 15:20 |
| *** siva_krishnan has joined #openstack-swift | 15:20 | |
| *** siva_krishnan has left #openstack-swift | 15:20 | |
| *** siva_krishnan1 has joined #openstack-swift | 15:21 | |
| *** siva_krishnan1 has left #openstack-swift | 15:21 | |
| *** siva_krishnan has joined #openstack-swift | 15:21 | |
| *** rcernin has quit IRC | 15:27 | |
| *** links has joined #openstack-swift | 15:32 | |
| *** gyee has joined #openstack-swift | 15:41 | |
| *** ametts has quit IRC | 15:59 | |
| *** haomaiwang has quit IRC | 16:01 | |
| *** haomaiwang has joined #openstack-swift | 16:01 | |
| *** esker has joined #openstack-swift | 16:03 | |
| *** Guest58782 has quit IRC | 16:03 | |
| *** links has quit IRC | 16:04 | |
| *** lyrrad has joined #openstack-swift | 16:05 | |
| *** lyrrad has quit IRC | 16:06 | |
| *** lyrrad has joined #openstack-swift | 16:07 | |
| *** esker has quit IRC | 16:13 | |
| *** esker has joined #openstack-swift | 16:13 | |
| *** jistr has quit IRC | 16:14 | |
| *** StraubTW has quit IRC | 16:18 | |
| vinsh_ | FYI I solved my recon problem I posted a few hours ago. I had empty dirs in /srv/node that were from a previous setup I had. Those were unused but still report back to recon. I removed them and all is well. | 16:18 |
| *** StraubTW has joined #openstack-swift | 16:18 | |
| *** nadeem has joined #openstack-swift | 16:18 | |
| *** nadeem has quit IRC | 16:19 | |
| *** nadeem has joined #openstack-swift | 16:19 | |
| mmotiani | joeljwright: notmyname: Hi, do we need to follow the same document structure for rest of the docs in Swift Client. I can see sdk.rst, index.rst, swift client.rst files have gaps. So do we need to follow the same convention for all which Joel is using in cli.rst? | 16:23 |
| *** haomaiwang has quit IRC | 16:32 | |
| *** dmorita has joined #openstack-swift | 16:32 | |
| *** haomaiwa_ has joined #openstack-swift | 16:34 | |
| *** StraubTW has quit IRC | 16:37 | |
| *** haomaiwa_ has quit IRC | 16:37 | |
| *** haomaiwang has joined #openstack-swift | 16:38 | |
| clayg | wat | 16:40 |
| clayg | vinsh_: oh good - i was curious | 16:41 |
| vinsh_ | I have 3 swift endpoints in one cluster now :) | 16:41 |
| clayg | whoa | 16:41 |
| vinsh_ | 3 different hardware vendors behind each. | 16:41 |
| vinsh_ | Yeah, cisco, supermicro and echostreams. | 16:41 |
| vinsh_ | all running EC | 16:41 |
| clayg | vinsh_: how'd you hack everyone to use different /etc/swift/swift.conf's? | 16:41 |
| *** rledisez has quit IRC | 16:42 | |
| vinsh_ | They all exist as distinct clusters. each registered with a different service name such as "swift" or "swift-perf" from keystone. | 16:42 |
| *** haomaiwang has quit IRC | 16:42 | |
| vinsh_ | they each have unique swift.confs. | 16:42 |
| joeljwright | mmotiani: sdk.rst has been removed | 16:42 |
| vinsh_ | So with puppet it's 3 different sets of roles. one per hardware type. | 16:42 |
| clayg | vinsh_: yeah - how you'd hack 'em to use different swift.conf's? stuff like the hash_suffix_prefix is hard coded to use /etc/swift/swift.conf - no "swift_dir" config option involved? | 16:43 |
| joeljwright | mmotiani: index.rst, cli.rst, client-api.rst and service-api.rst are the files that will continue to exist | 16:43 |
| joeljwright | mmotiani: oh, and introduction.rst | 16:44 |
| vinsh_ | @clayg Each of the 3 sets of nodes has a swift.conf deployed to it that is specific to its set of nodes (using puppet) | 16:44 |
| mmotiani | joeljwright: Alright! and what about apis.rst and swiftclient.rst ? | 16:45 |
| clayg | so.... different clusters? | 16:45 |
| vinsh_ | this is 3 pool of nodes each running their own ring. all registered to one main keystone. | 16:45 |
| vinsh_ | Yeah | 16:45 |
| clayg | ok, yeah i suppose multiple swift clusters in one keystone is probably a thing that happens - still coll tho! | 16:46 |
| pdardeau | vinsh_: each cluster running on separate machines? | 16:46 |
| vinsh_ | yeah | 16:46 |
| clayg | vinsh_: very cool! | 16:46 |
| clayg | vinsh_: now fill 'em up! | 16:46 |
| vinsh_ | each cluster diffrent hardware vendor. getting some EC stats to share at the summit with you guys/gals. | 16:46 |
| joeljwright | mmotiani: swiftclient.rst is still there, but it's a container for the autogenerated docstrings I think | 16:46 |
| clayg | vinsh_: awesome! | 16:46 |
| pdardeau | vinsh_: cool stuff. a federation of swift clusters! | 16:47 |
| vinsh_ | things like swift dispersion dont' understand multiple endpoints yet though.. so that needs work. | 16:47 |
| joeljwright | mmotiani: apis.rst was split up into cli/client-api/service-api | 16:47 |
| vinsh_ | Yeah :) | 16:47 |
| *** jordanP has quit IRC | 16:47 | |
| *** pcaruana has quit IRC | 16:48 | |
| mmotiani | joeljwright: Cool, thanks! | 16:48 |
| mmotiani | joeljwright: What would you suggest me to start with? | 16:48 |
| *** StraubTW has joined #openstack-swift | 16:49 | |
| vinsh_ | I wish for a session at the summit on "tuning ec chunksize and those kinda things" | 16:49 |
| joeljwright | mmotiani: There are CLI examples that still need to be completed | 16:49 |
| clayg | vinsh_: i thought dispersion operated on rings? what does it care about the swift endpoints - you mean just to pick out the right storage url? that's all like... service filters and junk? | 16:49 |
| vinsh_ | @clayg swift-disersion-report I should say | 16:50 |
| vinsh_ | It needs to know - usually from keystone what endpoint to operate on. | 16:50 |
| clayg | vinsh_: do you have some suggestions? I'm sure we can make time to discuss tuning EC if you have some suggestions | 16:50 |
| joeljwright | mmotiani: reviews and suggestions would also be gratefully received | 16:50 |
| vinsh_ | If there are multiple in a service catalog.. then it picks one only. | 16:50 |
| joeljwright | mmotiani: it's my plan to fill in the TODO sections in service-api.rst this week | 16:51 |
| clayg | vinsh_: yeah that makes sense to me - for a long time it didn't even support auth v2 - you could add some simple v1 admin auth middleware up in there - or maybe allow it to use internal_client? | 16:51 |
| vinsh_ | clayg: I wonder about how to determine what ec_chunk size and all those other chunks sizes in the pipeline for an ec cluster where object size is known/fixed. | 16:51 |
| clayg | when you have your rings and access to the storage nodes auth is for suckers - run you own proxy FTW | 16:51 |
| vinsh_ | ec_object_segment_size etc. | 16:52 |
| clayg | vinsh_: notmyname did played around with it for a while - it seemed to have a larger impact on memory than on performance (to a point) | 16:52 |
| vinsh_ | clayg: Agreed. Taking keystone out of the mix in the next iteration... less overhead for this use. | 16:52 |
| mmotiani | joeljwright: Ok, I will start from CLI and will look more into it and try to come up with some examples. | 16:52 |
| joeljwright | mmotiani: there are 3 suggestions for examples | 16:53 |
| joeljwright | mmotiani: but we want to keep it interesting if possible :) | 16:53 |
| vinsh_ | clayg: I got cosbench up and running (it's really nice!) I can use this and puppet to push different ec_object_segment_size out to the cluster.. find out what the sweet spot is. Lot of benchmarking to do on my end still. | 16:53 |
| openstackgerrit | Joel Wright proposed openstack/python-swiftclient: WIP: This patch adds a new doc structure for swiftclient https://review.openstack.org/288566 | 16:54 |
| mmotiani | joeljwright: Okay :) | 16:55 |
| *** mvk_ has quit IRC | 16:56 | |
| joeljwright | mmotiani: I have just pushed up the latest version from my laptop - that is my current working state | 16:56 |
| mmotiani | joeljwright: Yeah, I just saw that. | 16:56 |
| joeljwright | all existing comments are (hopefully) addressed, and all the TODO sections are 'real' | 16:56 |
| *** chsc has joined #openstack-swift | 16:56 | |
| mmotiani | Ok, thanks :) | 16:57 |
| *** ametts has joined #openstack-swift | 16:57 | |
| joeljwright | mmotiani: tahnks for helping! | 16:57 |
| *** dmorita has quit IRC | 16:57 | |
| *** _JZ_ has quit IRC | 17:05 | |
| *** zhiyan has quit IRC | 17:07 | |
| *** chsc has quit IRC | 17:07 | |
| *** patchbot has quit IRC | 17:07 | |
| *** sudorandom has quit IRC | 17:07 | |
| *** sw3 has quit IRC | 17:07 | |
| *** mathiasb has quit IRC | 17:07 | |
| *** dmsimard has quit IRC | 17:07 | |
| *** early has quit IRC | 17:07 | |
| *** ametts has quit IRC | 17:07 | |
| *** jmccarthy has quit IRC | 17:07 | |
| *** shakamunyi has quit IRC | 17:07 | |
| *** delatte has quit IRC | 17:07 | |
| *** saltsa has quit IRC | 17:07 | |
| *** briancline has quit IRC | 17:07 | |
| *** darrenc has quit IRC | 17:07 | |
| *** CrackerJackMack has quit IRC | 17:07 | |
| *** ejat has quit IRC | 17:07 | |
| *** sc has quit IRC | 17:07 | |
| *** pchng has quit IRC | 17:07 | |
| *** mmotiani has quit IRC | 17:07 | |
| *** mhu has quit IRC | 17:07 | |
| *** sgundur has quit IRC | 17:07 | |
| *** jamielennox has quit IRC | 17:07 | |
| *** nottrobin has quit IRC | 17:07 | |
| *** dfg has quit IRC | 17:07 | |
| *** JelleB has quit IRC | 17:07 | |
| *** Anticimex has quit IRC | 17:07 | |
| *** CrackerJackMack has joined #openstack-swift | 17:07 | |
| *** early has joined #openstack-swift | 17:07 | |
| *** delattec has joined #openstack-swift | 17:07 | |
| *** gyee has quit IRC | 17:07 | |
| *** _JZ__ has joined #openstack-swift | 17:07 | |
| *** barra204 has joined #openstack-swift | 17:07 | |
| *** patchbot` has joined #openstack-swift | 17:07 | |
| *** patchbot` is now known as patchbot | 17:07 | |
| *** sw3_ has joined #openstack-swift | 17:07 | |
| *** mathiasb_ has joined #openstack-swift | 17:07 | |
| openstackgerrit | Andreas Jaeger proposed openstack/swift: List system dependencies for running common tests https://review.openstack.org/298313 | 17:07 |
| *** dmsimard1 has joined #openstack-swift | 17:07 | |
| *** briancli1e has joined #openstack-swift | 17:07 | |
| *** Anticime1 has joined #openstack-swift | 17:07 | |
| *** sc__ has joined #openstack-swift | 17:07 | |
| *** saltsa_ has joined #openstack-swift | 17:07 | |
| *** ametts_ has joined #openstack-swift | 17:07 | |
| *** sudorandom_ has joined #openstack-swift | 17:07 | |
| *** sudorandom_ is now known as sudorandom | 17:07 | |
| *** ejat has joined #openstack-swift | 17:07 | |
| *** ejat has quit IRC | 17:07 | |
| *** ejat has joined #openstack-swift | 17:07 | |
| *** sw3_ is now known as sw3 | 17:07 | |
| openstackgerrit | Andreas Jaeger proposed openstack/swift: List system dependencies for running common tests https://review.openstack.org/298313 | 17:08 |
| notmyname | good morning, again | 17:08 |
| openstackgerrit | Andreas Jaeger proposed openstack/swift: List system dependencies for running common tests https://review.openstack.org/298313 | 17:08 |
| *** JelleB has joined #openstack-swift | 17:08 | |
| *** sgundur has joined #openstack-swift | 17:08 | |
| *** jmccarthy has joined #openstack-swift | 17:08 | |
| *** darrenc has joined #openstack-swift | 17:08 | |
| *** mmotiani has joined #openstack-swift | 17:08 | |
| *** dfg has joined #openstack-swift | 17:12 | |
| *** jamielennox has joined #openstack-swift | 17:12 | |
| *** daemontool has quit IRC | 17:14 | |
| *** pchng has joined #openstack-swift | 17:14 | |
| *** klrmn has joined #openstack-swift | 17:14 | |
| *** StraubTW has quit IRC | 17:14 | |
| *** nottrobin has joined #openstack-swift | 17:15 | |
| *** zhiyan has joined #openstack-swift | 17:17 | |
| *** chsc has joined #openstack-swift | 17:17 | |
| *** dmsimard1 is now known as dmsimard | 17:21 | |
| *** sgundur has left #openstack-swift | 17:21 | |
| *** lakshmiS has joined #openstack-swift | 17:30 | |
| *** dmorita has joined #openstack-swift | 17:39 | |
| *** sgundur has joined #openstack-swift | 17:39 | |
| *** StraubTW has joined #openstack-swift | 17:43 | |
| *** delatte has joined #openstack-swift | 17:47 | |
| *** ChubYann has joined #openstack-swift | 17:48 | |
| *** delattec has quit IRC | 17:51 | |
| *** zaitcev has joined #openstack-swift | 17:52 | |
| *** ChanServ sets mode: +v zaitcev | 17:52 | |
| *** delattec has joined #openstack-swift | 17:53 | |
| *** marcin12345_ has joined #openstack-swift | 17:56 | |
| *** delatte has quit IRC | 17:56 | |
| marcin12345_ | Does anybody know what to do with Swift with: object-server: ERROR container update failed with x.x.x.x:6001/sdb (saving for async update later): Timeout (3.0s) (txn: <....>) | 17:56 |
| marcin12345_ | ? | 17:56 |
| marcin12345_ | ? | 17:56 |
| *** dmorita has quit IRC | 17:59 | |
| *** asettle has joined #openstack-swift | 18:00 | |
| *** dmorita has joined #openstack-swift | 18:01 | |
| *** pcaruana has joined #openstack-swift | 18:02 | |
| clayg | marcin12345_: container server is down, maybe device is unmounted or full, but probably it's just a large container and the db was eating it's .pending inserts while that particular node was trying to update the container index - it'll process the async pending after awhile | 18:03 |
| marcin12345_ | well it is not disk related, I switched to ramdisk even. It is happening when using Swift with Gnocchi (time series db for Ceilometer), that is writing a lot of small objects all the time. The question is, is it safe to ignore those error? I think not, because it started happpending after adding more than 5 Vms. | 18:05 |
| marcin12345_ | container server is not down | 18:06 |
| tdasilva | marcin12345_: when you say you are using Swift with Gnocchi, do you mean you are sendind swift data to gnocchi, or is swift the data storage for gnocchi? | 18:07 |
| clayg | tdasilva: I think the latter! | 18:08 |
| marcin12345_ | the later, swift is data storage for gnocchi | 18:08 |
| clayg | tdasilva: i get POINTS | 18:08 |
| tdasilva | lol | 18:08 |
| marcin12345_ | ok you win ;) | 18:08 |
| *** cdelatte has joined #openstack-swift | 18:08 | |
| ahale | that sounds like an awesome way to make asyncs | 18:08 |
| clayg | ahale: lol | 18:08 |
| ahale | are they also expiring objects? :) | 18:09 |
| marcin12345_ | yes | 18:09 |
| ahale | :D | 18:09 |
| clayg | marcin12345_: you're probably going to need to think about the cardinality in your container schema - how quickly are you adding rows to a container - how long do you plan to keep that up? | 18:09 |
| clayg | if there answers are like 100/s and forever you're not gunna have a good time | 18:10 |
| marcin12345_ | but really with 5VMs+ it already does not scale? | 18:10 |
| *** asettle has quit IRC | 18:10 | |
| tdasilva | marcin12345_: probably a dumb question, have you checked that you can connect to your container node from that storage node? | 18:11 |
| *** StraubTW has quit IRC | 18:11 | |
| *** delattec has quit IRC | 18:11 | |
| clayg | marcin12345_: depends - on the cardinality in your container schema - the async_pending may not be an issue - but it depends on use case - and generally - yeah tons and tons and tons of teeny teeny tiny objects is the an object stores favorite use-case - sticking them all in one container makes swift panda's cry | 18:11 |
| clayg | is *not* the object stores favorite use-case | 18:12 |
| openstackgerrit | Merged openstack/python-swiftclient: Clean up some unnecessary variables https://review.openstack.org/296620 | 18:12 |
| clayg | tdasilva: he sure made it sound like he had pleanty of container updates working then at somepoint he started seeing timeouts - which sounds to me more like container contention than anything else - but sure - it could be other things | 18:13 |
| *** cdelatte has quit IRC | 18:13 | |
| clayg | marcin12345_: can you answer the question about how many objects you have in these container(s) and how fast they're having new objects added into them? | 18:13 |
| openstackgerrit | Merged openstack/swift: Ignore files in the devices directory when auditing objects https://review.openstack.org/295183 | 18:14 |
| clayg | ahale: lol - i missed the part about expiring - so some day they get twice as many inserts/s | 18:14 |
| *** arch-nemesis has quit IRC | 18:15 | |
| clayg | aerwin3: how goes the expiring work? didn't some guy push up some code for some go obj-expirier daemon? | 18:15 |
| marcin12345_ | clayg: 677 containers, 6442 objects, 46141712 Bytes, writing every 30 seconds | 18:16 |
| marcin12345_ | tdasilva: container node and storage node sits on the same box | 18:17 |
| clayg | so the containers on average have ... 10 objects in them? | 18:18 |
| *** cdelatte has joined #openstack-swift | 18:18 | |
| clayg | yeah this isn't a container contention problem - maybe some sort of nutso virtualization contention problem - something in the io scheduler - if you're container db's and objects are on the same devices there may be some weird io scheduling things going on in the host | 18:19 |
| clayg | ... only option I can think of that might have an effect is the db preallocate thing | 18:19 |
| clayg | tdasilva: you may have been right about network | 18:19 |
| tdasilva | clayg: so i get POINTS? ;) | 18:20 |
| tdasilva | lol | 18:20 |
| clayg | tdasilva: probably - it's not a zero sum game - we can both get points - hopefully even marcin12345_ can get some points | 18:21 |
| clayg | ... it's all about the points | 18:21 |
| *** StraubTW has joined #openstack-swift | 18:21 | |
| tdasilva | hehe, jk | 18:21 |
| clayg | marcin12345_: how many of those errors do you have? are the async_pendings keeping up (they get stored in /srv/node*/*/async_*) | 18:23 |
| *** StraubTW has quit IRC | 18:24 | |
| *** StraubTW has joined #openstack-swift | 18:24 | |
| *** arch-nemesis has joined #openstack-swift | 18:24 | |
| clayg | marcin12345_: do the errors include the /a/c path - any correlation on the specific containers? or ips? is it *only* when talking from the local object-server to local container-server but cross server werx? | 18:24 |
| marcin12345_ | it is both locally and remotely | 18:25 |
| marcin12345_ | at burst 15 errors/s | 18:26 |
| *** DevStock has joined #openstack-swift | 18:26 | |
| DevStock | hi | 18:29 |
| *** Nyar has joined #openstack-swift | 18:29 | |
| marcin12345_ | it shows also erros: ERROR __call__ error with PUT /sdb/<..>/AUTH_<...>/measure/<...>/<...>_<...> : LockTimeout (10s) /srv/node/sdb/containers/<...>/<...>/.lock (txn: tx<...>) | 18:29 |
| *** dmorita has quit IRC | 18:30 | |
| DevStock | the folder /srv/node/ can be configured in other path? | 18:30 |
| Nyar | Adding to what marcin12345_ has shared so far, increasing node_timeout to 10s in object-server.conf is "hiding" the problem with your current load | 18:31 |
| Nyar | We are however afraid, base on further testing, that this won't help once we scale to production load | 18:32 |
| *** dmorita has joined #openstack-swift | 18:32 | |
| Nyar | Moving the container and account services to use tmpfs devices also did not help. It doesn't seem like we are running into any hardware bottleneck here.. It seems like the object-server process is trying to communicate with container-server too fast | 18:37 |
| *** joeljwright has quit IRC | 18:39 | |
| Nyar | Are there any recommendations on the ratio of object-server concurrent workers vs container-server/updater concurrent workers? | 18:39 |
| jrichli | DevStock: you can change the devices value in the object server configuration http://docs.openstack.org/developer/swift/deployment_guide.html#object-server-configuration | 18:41 |
| DevStock | fuel mapped that path in the root | 18:44 |
| DevStock | so when I uploaded a lot of snapshot that path filled up the partition / | 18:45 |
| *** david-lyle has quit IRC | 18:45 | |
| *** dmorita has quit IRC | 18:46 | |
| *** david-lyle has joined #openstack-swift | 18:46 | |
| *** pauloewerton has joined #openstack-swift | 18:57 | |
| *** dmorita has joined #openstack-swift | 18:57 | |
| *** dmorita has quit IRC | 19:07 | |
| glange | DevStock: that's not the recommended way to run swift :) | 19:08 |
| *** dmorita has joined #openstack-swift | 19:08 | |
| *** dmorita has quit IRC | 19:09 | |
| *** dmorita has joined #openstack-swift | 19:09 | |
| clayg | Nyar: it's not normally a ratio of object -> container workers; in many clusters there's probably less container workers than object workers in total | 19:10 |
| clayg | marcin12345_: Nyar: are the LockTimeout new containers being created? | 19:11 |
| *** pauloewerton has quit IRC | 19:12 | |
| clayg | marcin12345_: Nyar: i guess that path looks like an object update | 19:12 |
| clayg | marcin12345_: Nyar: can you `swift stat` that AUTH_<...>/measure container - how many objects are in there again now? | 19:13 |
| Nyar | Containers in policy "gnocchi": 677 Objects in policy "gnocchi": 6084 Bytes in policy "gnocchi": 43794493 | 19:14 |
| Nyar | oh my bad, the container | 19:14 |
| Nyar | on it | 19:14 |
| Nyar | 0 object | 19:15 |
| Nyar | X-Container-Object-Count: 0 | 19:17 |
| *** trifon has joined #openstack-swift | 19:20 | |
| Nyar | Could someone shed some light on what the Swift pipeline is when an object is written to disk? To my understanding, the proxy-server receives the PUT request, determines to which server/partition to write the data to based on the ring consistent hashing algorithm | 19:22 |
| clayg | marcin12345_: Nyar: well, either that output is wrong/outdated or there's no good reason for that database to be taking that long to do *anything* | 19:22 |
| Nyar | an object-server process writes the actual data to disk | 19:22 |
| DevStock | ok not reccomended | 19:23 |
| Nyar | and then the object-server makes a call to the container-server? | 19:23 |
| DevStock | but i'm using glance + swift | 19:23 |
| clayg | Nyar: yup | 19:24 |
| DevStock | Is a wrong fuel conf | 19:25 |
| DevStock | set a qrong path | 19:25 |
| DevStock | wrong | 19:25 |
| DevStock | with small space | 19:25 |
| Nyar | So when we see object-server: ERROR container update failed with x.x.x.x:6001/sdb (saving for async update later): Timeout (3.0s), we are right to assume that the object-server is timeout while trying to communicate with the container-server after writing the data to disk? | 19:26 |
| clayg | Nyar: yeah that's definately what that error is saying | 19:26 |
| clayg | Nyar: specifically it's saying "I couldn't tell the container about this data, so I'm going to store this information in a temp file and the object-updater will try to send it to the container layer later" | 19:27 |
| *** silor has quit IRC | 19:27 | |
| Nyar | thanks clayg | 19:27 |
| clayg | Nyar: if these containers had millions of objects of them it'd all be basically SOP - but nothing about this is making any sense to me | 19:27 |
| clayg | Nyar: do you have a bunch of files in /srv/node/*/async_pending* | 19:28 |
| Nyar | We do indeed | 19:29 |
| clayg | Nyar: is the object-updater running? | 19:30 |
| Nyar | Since I am not seeing any high io wait or sign of io contention on the container devices | 19:30 |
| Nyar | I can only assume this is due to a too high number of requests made by the object-server to the container-server | 19:30 |
| Nyar | object-updater is running on all nodes yes | 19:31 |
| clayg | maybe grep the logs for messages from the object-updater see if he's having a bad time? | 19:31 |
| Nyar | The only container-server errors we (rarely) see are the LockTimeout | 19:32 |
| Nyar | I am going to try to collect more data before spamming you with more questions, hopefully finding a correlation between the object-server timeout errors and some relevant container-server logs | 19:34 |
| clayg | well what about the object-updater - he should log a status message every so often regardless - why isn't he claning up the async pendings - he should mention some kind error - `grep object-updater: /var/log/syslog` | 19:34 |
| clayg | Nyar: ok, but i'm worried we're not seeing the whole picture - try to get the async pending cleaned up - maybe stop the object-updater and run in in the foreground `swift-init object-updater stop` `swift-init object-updater once -nv` | 19:35 |
| clayg | gl | 19:35 |
| Nyar | Thank you clayg | 19:36 |
| *** daemontool has joined #openstack-swift | 19:38 | |
| *** sheel has quit IRC | 19:57 | |
| *** esker has quit IRC | 20:12 | |
| *** esker has joined #openstack-swift | 20:13 | |
| openstackgerrit | Paulo Ewerton Gomes Fragoso proposed openstack/python-swiftclient: Adding keystoneauth sessions support https://review.openstack.org/298968 | 20:29 |
| *** StraubTW has quit IRC | 20:29 | |
| *** gyee has joined #openstack-swift | 20:32 | |
| *** asettle has joined #openstack-swift | 20:43 | |
| jrichli | just checking : specifying x-newest against a cluster using an EC policy does nothing, right? | 20:43 |
| notmyname | jrichli: right. it's got to talk to all of the primaries anyway | 20:47 |
| clayg | it's sorta *always* x-newest | 20:47 |
| *** StraubTW has joined #openstack-swift | 20:51 | |
| jrichli | right, just making sure there wasn't some small thing that was different :-) | 20:51 |
| jrichli | thx! | 20:52 |
| *** StraubTW has quit IRC | 21:08 | |
| *** StraubTW has joined #openstack-swift | 21:19 | |
| *** lyrrad has quit IRC | 21:22 | |
| *** lakshmiS has quit IRC | 21:33 | |
| notmyname | FYI there's a summit schedule attached to http://lists.openstack.org/pipermail/openstack-dev/2016-March/090606.html | 21:33 |
| *** nadeem has quit IRC | 21:34 | |
| notmyname | there might be a few slots that are juggled for various projects, but it should be relatively stable | 21:34 |
| clayg | notmyname: boom - like the swift action | 21:35 |
| clayg | the fuel session stuck in the thursday afternoon slot is weird :\ | 21:35 |
| notmyname | tomorrow at the team meeting I want to kick off planning for it. I want to make this the most like a hackathon yet | 21:36 |
| *** StraubTW has quit IRC | 21:36 | |
| clayg | notmyname: can we offer to swap the 5-5:40 slot with them and just go do beers earlier on Thursday? | 21:36 |
| notmyname | clayg: good idea | 21:36 |
| notmyname | hmmm...it looks like the friday room might be shared | 21:38 |
| clayg | notmyname: ewwww with *glance* | 21:39 |
| clayg | :) | 21:39 |
| notmyname | be nice to glance :-) | 21:39 |
| clayg | oh i didn't even see the Swift slots in the other room on Wednesday morning | 21:40 |
| *** esker has quit IRC | 21:43 | |
| *** sgundur has left #openstack-swift | 21:43 | |
| *** lyrrad has joined #openstack-swift | 21:43 | |
| notmyname | actually, even if fuel could take the earlier 1:30 slot, that would be better for us too. more contiguous blocks of time | 21:44 |
| *** DevStock has quit IRC | 21:46 | |
| clayg | notmyname: like swift itself - we prefer to have large allocations of contiguous space | 21:46 |
| notmyname | ah, I can guess why we got that oddity. I'm giving a talk then. but I can certainly step out for that time, especially if we run it like the hackathon | 21:47 |
| clayg | notmyname: oh wow | 21:48 |
| clayg | notmyname: idk, we'll probably all just sit around staring at each other if you're not there | 21:48 |
| notmyname | yeah, probably ;-) | 21:48 |
| notmyname | bah. the first one is actually worse. I'm giving a talk then, too. (only 2 talks this summit.) but I'd want to be at the first of the working sessions | 21:49 |
| clayg | notmyname: someone needs to draw a venn diagram and say "Let's take a step back; what problem are we trying to solve" | 21:49 |
| notmyname | maybe we could do some working session prep during the fishbowl sessions | 21:49 |
| timburke | clayg: problems we're trying to solve -> O O <- what we're actually talking about | 21:51 |
| clayg | timburke: YES! | 21:53 |
| clayg | ascii venn diagram FTW | 21:53 |
| clayg | notmyname: nm, you're off the hook - you have timburke now | 21:53 |
| zaitcev | Venn diagram? I thought it was an emoticon. | 21:53 |
| timburke | problems that can be solved with Venn diagrams -> ( O <- problems that can be solved with ASCII Venn diagrams ) | 21:54 |
| clayg | zaitcev: no no, see how the two circles don't overlap - that's because we're not talking about the problems we're trying to solve - it's so genius | 21:54 |
| clayg | timburke: pushing it - but i'm still rofling over here | 21:55 |
| clayg | maybe we can add ascii art venn diagrams to container responses? | 21:55 |
| timburke | clayg: have a bit of fun with how sp/nbsp sort and you might be able to do it now | 21:58 |
| *** StraubTW has joined #openstack-swift | 22:02 | |
| *** mmcardle has quit IRC | 22:02 | |
| *** esker has joined #openstack-swift | 22:02 | |
| openstackgerrit | David Goetz proposed openstack/swift: go: fix the async logger https://review.openstack.org/299000 | 22:03 |
| openstackgerrit | Clay Gerrard proposed openstack/swift: WIP: Cleanup EC backend logging on disconnect https://review.openstack.org/297822 | 22:05 |
| *** daemontool has quit IRC | 22:05 | |
| *** ametts_ has quit IRC | 22:15 | |
| *** trifon has quit IRC | 22:43 | |
| *** jmb__ has quit IRC | 22:44 | |
| *** esker has quit IRC | 22:48 | |
| *** diogogmt has quit IRC | 22:48 | |
| *** esker has joined #openstack-swift | 22:57 | |
| *** km has joined #openstack-swift | 23:05 | |
| *** esker has quit IRC | 23:11 | |
| *** jmb__ has joined #openstack-swift | 23:13 | |
| *** jmb__ has quit IRC | 23:18 | |
| Nyar | @clayg: I haven't been able to collect more logs pertinent to the object-server errors we are seeing when Gnocchi writes to Swift :( | 23:22 |
| Nyar | The object-updater is doing its job and never complaining | 23:23 |
| *** arch-nemesis has quit IRC | 23:23 | |
| notmyname | Nyar: are the container DBs on flash or on spinning drives? | 23:24 |
| Nyar | They are currently on spinning drives but we have also tried to move them to tmpfs to confirm if it could be due to i/o contention | 23:25 |
| Nyar | The result was the same: object-server spamming the following a few seconds after gnocchi starts sending PUT request to Swift | 23:26 |
| *** mingdang1 has joined #openstack-swift | 23:26 | |
| Nyar | object-server: ERROR container update failed with x.x.x.x:6001/sdb (saving for async update later): Timeout (3.0s) | 23:26 |
| Nyar | We also see container-server complaining about a LockTimeout (10s) once in a while | 23:27 |
| notmyname | ok. thanks for helping me catch up. :-) | 23:28 |
| Nyar | IO wait on the container devices is within acceptable range (~20 max) | 23:28 |
| Nyar | :) | 23:28 |
| notmyname | and IIRC you've got very small containers (small number of obejcts in them) and relatively low request rates, right? | 23:29 |
| Nyar | I am losing my mind on this. I have tried tweaking the number of object-server workers down and cranking up the number of container-worker up | 23:29 |
| notmyname | yeah, Iw as about to ask about that | 23:30 |
| notmyname | are these co-located? | 23:30 |
| notmyname | what is your current worker count? | 23:30 |
| Nyar | Gnocchi containers have 9 objects in them, we currently have 1518 of them | 23:30 |
| Nyar | Proxy and storage services are all running on the same nodes indeed | 23:30 |
| Nyar | My current crazy test was 8 object-servers for 128 container-server | 23:30 |
| Nyar | :D | 23:30 |
| notmyname | how many cores do you have on one of those machines? | 23:31 |
| Nyar | 16 hyperthreaded | 23:31 |
| Nyar | so 32 from a kernel standpoint | 23:31 |
| Nyar | Originally, our workers count was 32 object-server and 32 account-server | 23:32 |
| notmyname | are you using servers per disk? | 23:32 |
| notmyname | err... servers_per_port | 23:32 |
| Nyar | Hehe, old habits? :) | 23:32 |
| notmyname | or threads_per_disk? | 23:33 |
| Nyar | I was reading about that, we do not currently | 23:33 |
| *** kei_yama has joined #openstack-swift | 23:33 | |
| Nyar | But would not I see high IO waits if that was my issue? | 23:33 |
| notmyname | well, really, I was just trying to get a picture of where you are | 23:33 |
| notmyname | did you turn the worker counts back down? | 23:34 |
| Nyar | What's really troubling me here is that it doesn't seem we are hitting any hardware bottleneck, it seems like object-server is timeout while trying to communicate with container-server after an object-write. But all objects are written to disk succesffully | 23:34 |
| Nyar | Went back to 32 proxy, 32 object, 32 account, 32 container | 23:35 |
| notmyname | have you been running this workload for a while or is it a pretty new thing that you just turned on? IOW, is this a problem that emerged after time, or did it start immediately? | 23:35 |
| Nyar | That's brand new, this Swift cluster was only (under)used by Glance | 23:35 |
| Nyar | We started observing this as soon as Gnocchi entered the mix | 23:35 |
| notmyname | do you have db_preallocation set in the container server config? | 23:35 |
| notmyname | ok | 23:35 |
| Nyar | I do not, let me look that up | 23:36 |
| Nyar | And thank you so much already for your time! Much appreciated! | 23:36 |
| notmyname | it's normally a good thing on account/container DBs deployed on spinning drives. it helps with disk fragmentation over time | 23:36 |
| notmyname | so, just to throw it out there, it is *always* recommended that you use flash for accounts and containers | 23:37 |
| notmyname | :-) | 23:37 |
| notmyname | (and with flash drives, you'd want to turn db_preallocation off) | 23:37 |
| notmyname | but that's not where we are, so let's figure it out | 23:38 |
| *** bill_az has quit IRC | 23:38 | |
| Nyar | That's going to happen very soon. I must admit our original design was supposed to remain small (glance only) so we went for the cheap and easy. | 23:38 |
| Nyar | We are planning to dedicate flash drives for accounts and containers as soon as we understood what's going on here :) | 23:39 |
| notmyname | what's the gnocchi request rate? soemthing like 30/s right? | 23:39 |
| *** mingdang1 has quit IRC | 23:40 | |
| notmyname | Nyar: what version of swift are you using? | 23:41 |
| Nyar | That sounds about right, getting the exact numbers as we speak | 23:41 |
| Nyar | 2.5.0.1 Liberty | 23:42 |
| clayg | notmyname: thanks for jumping in | 23:42 |
| notmyname | do you have container_update_timeout set in the object server? | 23:43 |
| clayg | Nyar: it doesn't make sense to me that you have a bunch of async_pending files piling up while the object-updater is also successfully processing async_pendings | 23:43 |
| clayg | Nyar: is there anything in /var/log/messages or syslog about network? nfconntrack or dropped something - some kind of system/kernal warning message? | 23:44 |
| Nyar | notmyname: I do not. One way to "hide" those error messages I found was to set node_timeout to 10s though | 23:45 |
| clayg | Nyar: can you find one of these containers on disk? and check it's size and the size of it's .pending? | 23:45 |
| Nyar | But that was only a work around, the error messages appeared again as soon as we had Gnocchi writing more data (by adding more nodes monitored by Ceilometer) | 23:45 |
| notmyname | clayg: yup. that's the right next step | 23:46 |
| Nyar | clayg: no network error logs | 23:46 |
| clayg | notmyname: i feel like there's more objects in these containers than we're seeing - the LockTimeout doesn't make any sense at all on these tiny containers | 23:46 |
| clayg | Nyar: are you still on the tmpfs stuff or we're back on spinning rust? | 23:47 |
| Nyar | we are back on spinning drives | 23:47 |
| notmyname | yeah. I'd think it might be some really big un-vacuumed DB | 23:47 |
| Nyar | Since it did not make a difference | 23:47 |
| *** hosanai has joined #openstack-swift | 23:47 | |
| *** ChanServ sets mode: +v hosanai | 23:47 | |
| Nyar | looking for a container on disk | 23:48 |
| clayg | Nyar: do the glace containers ever [lock]timeout - or always the gnnochi containers? | 23:48 |
| Nyar | I confirm the ~50+ requests/second from Gnocchi by the way | 23:48 |
| Nyar | Only the gnocchi containers but I am not sure how pertinent that is knowing that Glance is just sitting on its ass all day | 23:48 |
| Nyar | We write an image per week to it.. | 23:49 |
| clayg | 50 requests per second for a day is 4M objects - could there really be that much data hiding in async_pendings? | 23:50 |
| notmyname | of they are expiring, then maybe in unused rows? | 23:50 |
| notmyname | I don't remember what the right word is for "unused" | 23:50 |
| clayg | notmyname: ah! | 23:51 |
| clayg | delted = 1 - of course | 23:51 |
| clayg | deleted | 23:51 |
| clayg | i call them tombstone rows sometimes | 23:51 |
| clayg | notmyname: I like the tombstone row idea a lot | 23:51 |
| notmyname | great. so 4M tombstone rows + 3 active rows. try to PUT 50/s and get an async | 23:51 |
| Nyar | Gnocchi is deleting A LOT of objects, the number of objects remains stable and proportional to the number of instances monitored by Ceilometer | 23:51 |
| clayg | gb containers here we come! | 23:51 |
| Nyar | The current counts: | 23:51 |
| Nyar | Containers in policy "glance": 1 Objects in policy "glance": 63 Bytes in policy "glance": 16774897152 Containers in policy "gnocchi": 677 Objects in policy "gnocchi": 7906 Bytes in policy "gnocchi": 46494042 | 23:52 |
| Nyar | erf, shitty formating, my bad | 23:52 |
| clayg | Nyar: that's fine - you need to find a container on disk | 23:52 |
| notmyname | Nyar: if you can find a gnocchi DB on disk and do `du` on it, that will tell us something | 23:52 |
| notmyname | Nyar: do you know how to do that? I can walk you through it | 23:52 |
| clayg | `swift-get-nodes /etc/swift/container.ring.gz gnocchi_account gnocchi_container` | 23:53 |
| Nyar | I do not know no, but since we are using storage policies, the only containers on those devices should be Gnocchi's | 23:54 |
| Nyar | oh nice! | 23:54 |
| Nyar | thanks! | 23:54 |
| notmyname | clayg: yeah, the delete=1 rows is what I was thinking when thinking about the fragmentation on disk (db_preallocation=off). it would exacerbate the problem | 23:54 |
| clayg | notmyname: gah - i don't know why I didn't think about tombstone rows! good call notmyname | 23:55 |
| clayg | notmyname: you know a lot about swift | 23:55 |
| Nyar | I just realized after typing this command that my sentence about storage policies was stupid | 23:55 |
| Nyar | Please ignore. | 23:56 |
| Nyar | I do not know a lot about Swift :] | 23:56 |
| *** DevStok has quit IRC | 23:56 | |
| notmyname | you're running it prod. that's what counts :-) | 23:56 |
| Nyar | But I am glad the project lead knows a lot about it :D | 23:56 |
| notmyname | (and that counts for a lot and is really awesome) | 23:56 |
| Nyar | SO | 23:57 |
| Nyar | 20K 1afd06e9acafdff124db7e72955421bb.db | 23:58 |
| Nyar | 0 1afd06e9acafdff124db7e72955421bb.db.pending | 23:58 |
| clayg | GD! | 23:58 |
| notmyname | well that seems reasonable. but totally not helpful to solving our problem!@ | 23:58 |
| Nyar | I figured as much ^^ | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!