| *** tqtran has quit IRC | 00:09 | |
| *** diogogmt has joined #openstack-swift | 00:11 | |
| kota_ | good morning | 00:14 |
|---|---|---|
| openstackgerrit | Spencer Krum proposed openstack/swift: Hi zuul, it's me, nibz https://review.openstack.org/355676 | 00:21 |
| mattoliverau | kota_: morning | 00:23 |
| kota_ | mattoliverau: morning | 00:24 |
| redbo | /win 3 | 00:29 |
| *** Suyi has quit IRC | 00:35 | |
| *** jamielennox is now known as jamielennox|away | 00:48 | |
| *** jamielennox|away is now known as jamielennox | 00:49 | |
| *** csmart has quit IRC | 00:52 | |
| *** csmart has joined #openstack-swift | 00:53 | |
| *** m_kazuhiro has joined #openstack-swift | 01:02 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift: Follow up delayed reap probe test https://review.openstack.org/355683 | 01:08 |
| *** tqtran has joined #openstack-swift | 01:09 | |
| *** tqtran has quit IRC | 01:13 | |
| *** airdisa has quit IRC | 01:16 | |
| *** asettle has joined #openstack-swift | 01:22 | |
| *** manous has quit IRC | 01:25 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift: Don't run probe tests if resetswift failed https://review.openstack.org/355687 | 01:32 |
| *** asettle has quit IRC | 01:32 | |
| *** klrmn has quit IRC | 01:57 | |
| *** jamielennox is now known as jamielennox|away | 02:07 | |
| openstackgerrit | Tim Burke proposed openstack/swift: Autovivify X-Versions-Location container https://review.openstack.org/265015 | 02:10 |
| *** dmorita_ has quit IRC | 02:15 | |
| *** jamielennox|away is now known as jamielennox | 02:26 | |
| *** asettle has joined #openstack-swift | 02:33 | |
| mahatic_ | good morning | 02:34 |
| mattoliverau | mahatic_: morning! | 02:36 |
| openstackgerrit | Merged openstack/swift: Added a test for delayed reap. https://review.openstack.org/321041 | 02:36 |
| *** asettle has quit IRC | 02:40 | |
| *** klrmn has joined #openstack-swift | 03:01 | |
| *** ChubYann has quit IRC | 03:11 | |
| *** dmorita has joined #openstack-swift | 03:15 | |
| *** SkyRocknRoll has joined #openstack-swift | 03:16 | |
| *** dmorita has quit IRC | 03:20 | |
| *** ChubYann has joined #openstack-swift | 03:24 | |
| *** _JZ_ has quit IRC | 03:26 | |
| timburke | kota_: fyi, i think i'll have a new patchset for you before long | 03:28 |
| kota_ | timburke: okay, thanks for your info! | 03:31 |
| *** diogogmt has quit IRC | 03:34 | |
| *** asettle has joined #openstack-swift | 03:56 | |
| *** asettle has quit IRC | 04:01 | |
| openstackgerrit | Tim Burke proposed openstack/swift: Add "history" mode to versioned_writes middleware https://review.openstack.org/214922 | 04:08 |
| kota_ | timburke: so quick! | 04:08 |
| *** tqtran has joined #openstack-swift | 04:10 | |
| timburke | kota_: i'd been think about what needed to change and already had a lot of it done before i pinged you :) | 04:10 |
| kota_ | timburke: hehe, nice. i'm feeling it takes a day-ish when saying "before long" on my poor English | 04:11 |
| timburke | i think timezones may also play into it :-) speaking of, i should go home | 04:13 |
| notmyname | what?! | 04:13 |
| timburke | but i really like where things are going with defaulter + history mode + auto-vivifying version locations :-) | 04:13 |
| notmyname | timburke: go home! why are you in the office? | 04:13 |
| timburke | notmyname: ...whiskey? | 04:13 |
| timburke | i mean, uh... | 04:13 |
| notmyname | ah, yes. sales team meetings | 04:13 |
| *** tqtran has quit IRC | 04:14 | |
| kota_ | timburke: i agree with notmyname you should go home, I am going to run tests instead of you in JST | 04:15 |
| notmyname | wheee. time to send another email | 04:18 |
| *** _JZ_ has joined #openstack-swift | 04:19 | |
| *** links has joined #openstack-swift | 04:20 | |
| *** klrmn has quit IRC | 04:23 | |
| *** bkopilov has joined #openstack-swift | 04:25 | |
| *** bkopilov is now known as bkopilov_wfh | 04:25 | |
| *** nadeem has joined #openstack-swift | 04:26 | |
| timburke | hrm. judging by http://lists.openstack.org/pipermail/openstack-dev/2016-August/101411.html i didn't do what i thought i did in https://github.com/openstack/python-swiftclient/commit/aa0edd0 | 04:30 |
| timburke | ...or requests wasn't behaving as i thought it did? | 04:31 |
| *** abhitechie has joined #openstack-swift | 04:51 | |
| *** SkyRocknRoll has quit IRC | 04:57 | |
| *** psachin has joined #openstack-swift | 04:59 | |
| *** nadeem has quit IRC | 05:12 | |
| *** nadeem has joined #openstack-swift | 05:23 | |
| *** klrmn has joined #openstack-swift | 05:28 | |
| *** SkyRocknRoll has joined #openstack-swift | 05:33 | |
| *** klrmn has quit IRC | 05:52 | |
| *** portante has quit IRC | 06:04 | |
| *** portante has joined #openstack-swift | 06:04 | |
| *** tqtran has joined #openstack-swift | 06:11 | |
| *** rcernin has joined #openstack-swift | 06:14 | |
| *** tqtran has quit IRC | 06:15 | |
| *** nadeem has quit IRC | 06:16 | |
| *** maestropandy has joined #openstack-swift | 06:28 | |
| *** Jeffrey4l has joined #openstack-swift | 06:30 | |
| *** maestropandy has quit IRC | 06:32 | |
| *** maestropandy has joined #openstack-swift | 06:32 | |
| *** maestropandy has left #openstack-swift | 06:33 | |
| *** admin6 has joined #openstack-swift | 06:36 | |
| *** nadeem has joined #openstack-swift | 06:39 | |
| *** bkopilov_wfh is now known as bkopilov | 06:39 | |
| *** oshritf has joined #openstack-swift | 06:45 | |
| *** SkyRocknRoll has quit IRC | 06:55 | |
| *** maestropandy1 has joined #openstack-swift | 07:05 | |
| *** mingyu has joined #openstack-swift | 07:05 | |
| *** ppai has joined #openstack-swift | 07:09 | |
| *** SkyRocknRoll has joined #openstack-swift | 07:11 | |
| *** maestropandy1 has quit IRC | 07:17 | |
| *** maestropandy has joined #openstack-swift | 07:17 | |
| *** maestropandy has left #openstack-swift | 07:22 | |
| *** rledisez has joined #openstack-swift | 07:23 | |
| *** takashi has joined #openstack-swift | 07:34 | |
| *** SkyRocknRoll has quit IRC | 07:49 | |
| *** chlong has quit IRC | 07:50 | |
| openstackgerrit | Charles Hsu proposed openstack/swift: Ignore auditor status files to prevent replicator reports errors https://review.openstack.org/353648 | 07:52 |
| *** bebule has joined #openstack-swift | 07:55 | |
| *** bebule has quit IRC | 08:01 | |
| *** bebule has joined #openstack-swift | 08:01 | |
| *** m_kazuhiro has quit IRC | 08:04 | |
| *** nadeem has quit IRC | 08:08 | |
| *** joeljwright has joined #openstack-swift | 08:09 | |
| *** ChanServ sets mode: +v joeljwright | 08:09 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/swift: [WIP] Testing latest u-c https://review.openstack.org/318441 | 08:10 |
| *** dmorita has joined #openstack-swift | 08:17 | |
| *** asettle has joined #openstack-swift | 08:20 | |
| *** dmorita has quit IRC | 08:21 | |
| *** mvk has quit IRC | 08:29 | |
| *** manous has joined #openstack-swift | 08:33 | |
| *** maestropandy has joined #openstack-swift | 08:35 | |
| *** maestropandy1 has joined #openstack-swift | 08:36 | |
| *** maestropandy has quit IRC | 08:40 | |
| *** SkyRocknRoll has joined #openstack-swift | 08:46 | |
| *** maestropandy1 has left #openstack-swift | 08:49 | |
| *** acoles_ is now known as acoles | 08:49 | |
| kota_ | hi, acoles. thanks for commenting on delay reap probe tests. | 08:52 |
| kota_ | acoles: i did wrong but i found another issue that why i did so. | 08:52 |
| acoles | kota_: hi kota_ NP sounds like not failing when resetswift fails cost you a lot of time :/ | 08:53 |
| kota_ | yes, that's what i'd like to resolve at patch 355687 | 08:54 |
| patchbot | kota_: https://review.openstack.org/#/c/355687/ - swift - Don't run probe tests if resetswift failed | 08:54 |
| acoles | kota_: yep | 08:54 |
| *** mvk has joined #openstack-swift | 08:55 | |
| acoles | kota_: I see same thing with sudo .probetests | 08:58 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift: Updated from global requirements https://review.openstack.org/88736 | 09:01 |
| *** stantonnet_ has quit IRC | 09:03 | |
| *** nikivi has joined #openstack-swift | 09:05 | |
| *** oshritf has quit IRC | 09:05 | |
| *** ouchkernel has joined #openstack-swift | 09:09 | |
| * kota_ is going to dinner table. | 09:13 | |
| *** _JZ_ has quit IRC | 09:13 | |
| *** ppai has quit IRC | 09:34 | |
| *** admin6 has quit IRC | 09:39 | |
| *** ppai has joined #openstack-swift | 09:46 | |
| *** dmellado has quit IRC | 09:46 | |
| *** dmellado has joined #openstack-swift | 09:56 | |
| *** tqtran has joined #openstack-swift | 10:12 | |
| *** oshritf has joined #openstack-swift | 10:14 | |
| *** tqtran has quit IRC | 10:17 | |
| *** mingyu has quit IRC | 10:21 | |
| *** asettle has quit IRC | 10:22 | |
| *** mingyu has joined #openstack-swift | 10:22 | |
| *** mvk has quit IRC | 10:25 | |
| *** mvk has joined #openstack-swift | 10:26 | |
| *** mingyu has quit IRC | 11:05 | |
| *** mingyu has joined #openstack-swift | 11:06 | |
| *** asettle has joined #openstack-swift | 11:09 | |
| *** mingyu has quit IRC | 11:10 | |
| openstackgerrit | zheng yin proposed openstack/swift: simplify code https://review.openstack.org/355858 | 11:27 |
| *** takashi has quit IRC | 11:29 | |
| *** asettle has quit IRC | 11:30 | |
| *** abhinavtechie has joined #openstack-swift | 11:38 | |
| *** abhitechie has quit IRC | 11:38 | |
| *** asettle has joined #openstack-swift | 11:44 | |
| *** ukaynar has joined #openstack-swift | 11:44 | |
| *** ukaynar has quit IRC | 11:50 | |
| *** asettle has quit IRC | 11:51 | |
| *** ukaynar has joined #openstack-swift | 11:51 | |
| *** asettle has joined #openstack-swift | 11:52 | |
| *** airdisa has joined #openstack-swift | 12:03 | |
| *** bebule has quit IRC | 12:04 | |
| *** ukaynar has quit IRC | 12:08 | |
| *** ukaynar has joined #openstack-swift | 12:09 | |
| *** psachin has quit IRC | 12:10 | |
| *** ukaynar_ has joined #openstack-swift | 12:16 | |
| *** ukaynar has quit IRC | 12:16 | |
| *** ukaynar_ has quit IRC | 12:18 | |
| *** dmorita has joined #openstack-swift | 12:18 | |
| *** dmorita has quit IRC | 12:22 | |
| *** abhinavtechie has quit IRC | 12:33 | |
| *** abhitechie has joined #openstack-swift | 12:35 | |
| *** ppai has quit IRC | 12:49 | |
| *** ChubYann has quit IRC | 12:51 | |
| *** asettle has quit IRC | 12:55 | |
| *** asettle has joined #openstack-swift | 12:58 | |
| *** kei_yama has quit IRC | 12:59 | |
| *** ppai has joined #openstack-swift | 13:03 | |
| *** ChubYann has joined #openstack-swift | 13:07 | |
| *** lifeless has quit IRC | 13:11 | |
| *** manoelrui has joined #openstack-swift | 13:18 | |
| *** lifeless has joined #openstack-swift | 13:29 | |
| *** klamath has joined #openstack-swift | 13:30 | |
| *** klamath has quit IRC | 13:31 | |
| *** ppai has quit IRC | 13:31 | |
| *** klamath has joined #openstack-swift | 13:32 | |
| *** diogogmt has joined #openstack-swift | 13:33 | |
| *** catintheroof has joined #openstack-swift | 13:36 | |
| *** airdisa_ has joined #openstack-swift | 13:40 | |
| *** diogogmt has quit IRC | 13:42 | |
| *** ppai has joined #openstack-swift | 13:43 | |
| *** airdisa has quit IRC | 13:43 | |
| *** catintheroof has quit IRC | 13:43 | |
| *** abhitechie has quit IRC | 13:45 | |
| *** abhitechie has joined #openstack-swift | 13:46 | |
| *** abhitechie has quit IRC | 13:52 | |
| *** abhitechie has joined #openstack-swift | 13:53 | |
| *** sbelous__ has joined #openstack-swift | 13:53 | |
| *** permalac has joined #openstack-swift | 13:53 | |
| *** sbelous__ has left #openstack-swift | 13:53 | |
| *** jistr is now known as jistr|debug | 14:03 | |
| *** ppai has quit IRC | 14:05 | |
| *** mingyu has joined #openstack-swift | 14:06 | |
| *** mingyu has quit IRC | 14:10 | |
| *** tqtran has joined #openstack-swift | 14:14 | |
| openstackgerrit | Alistair Coles proposed openstack/swift: EC - eliminate .durable files https://review.openstack.org/355958 | 14:17 |
| *** asettle has quit IRC | 14:18 | |
| *** tqtran has quit IRC | 14:18 | |
| *** asettle has joined #openstack-swift | 14:18 | |
| *** ppai has joined #openstack-swift | 14:19 | |
| *** joeljwright1 has joined #openstack-swift | 14:22 | |
| *** joeljwright1 has quit IRC | 14:22 | |
| *** asettle has quit IRC | 14:23 | |
| *** joeljwright has quit IRC | 14:24 | |
| *** asettle has joined #openstack-swift | 14:35 | |
| *** diogogmt has joined #openstack-swift | 14:37 | |
| *** links has quit IRC | 14:39 | |
| *** ppai has quit IRC | 14:39 | |
| *** jistr|debug is now known as jistr | 14:50 | |
| *** arch-nemesis has joined #openstack-swift | 14:51 | |
| *** abhitechie has quit IRC | 14:51 | |
| *** wer has quit IRC | 14:55 | |
| *** wer has joined #openstack-swift | 14:56 | |
| *** tongli has joined #openstack-swift | 14:56 | |
| *** permalac has quit IRC | 14:58 | |
| *** oshritf has quit IRC | 15:10 | |
| *** cdelatte has joined #openstack-swift | 15:15 | |
| *** tongli has quit IRC | 15:27 | |
| *** tongli has joined #openstack-swift | 15:27 | |
| openstackgerrit | Alistair Coles proposed openstack/swift: Improve doc for using container-sync with large objects https://review.openstack.org/356012 | 15:36 |
| notmyname | good morning | 15:36 |
| *** rcernin has quit IRC | 15:46 | |
| acoles | notmyname: good morning. is it ok for us to manually mark bugs with fix committed/released? I often see email from an openstack bot doing that but in this case the patch did not reference the bug so there is no record | 15:52 |
| acoles | i.e. https://bugs.launchpad.net/swift/+bug/1487791 | 15:52 |
| openstack | Launchpad bug 1487791 in OpenStack Object Storage (swift) "POST to DLO squashes data without fast-POST" [Undecided,In progress] - Assigned to Janie Richling (jrichli) | 15:52 |
| notmyname | acoles: yep. IIRC it should be set to "fix released" | 15:55 |
| notmyname | that is, I don't think the bot goes along at release time to move stuff from committed to released | 15:56 |
| notmyname | and I think the bot these days just puts stuff into "Released" | 15:56 |
| openstackgerrit | Donagh McCabe proposed openstack/swift: Corrections for the API specifications in api-ref https://review.openstack.org/354767 | 16:01 |
| acoles | notmyname: k. thanks | 16:06 |
| *** admin6 has joined #openstack-swift | 16:09 | |
| *** asettle has quit IRC | 16:13 | |
| *** SkyRocknRoll has quit IRC | 16:15 | |
| *** dmorita has joined #openstack-swift | 16:22 | |
| *** rledisez has quit IRC | 16:24 | |
| *** dmorita has quit IRC | 16:25 | |
| *** dmorita has joined #openstack-swift | 16:26 | |
| *** Suyi has joined #openstack-swift | 16:28 | |
| *** _JZ_ has joined #openstack-swift | 16:36 | |
| *** thumpba has joined #openstack-swift | 16:43 | |
| openstackgerrit | Alistair Coles proposed openstack/swift: Add test for HeaderKeyDict https://review.openstack.org/353940 | 16:54 |
| timburke | good morning | 17:06 |
| *** david-lyle_ has joined #openstack-swift | 17:09 | |
| *** Suyi has quit IRC | 17:10 | |
| *** david-lyle has quit IRC | 17:13 | |
| *** david-lyle_ is now known as david-lyle | 17:13 | |
| *** Suyi has joined #openstack-swift | 17:14 | |
| *** tqtran has joined #openstack-swift | 17:19 | |
| *** klrmn has joined #openstack-swift | 17:20 | |
| manoelrui | hello guys | 17:24 |
| manoelrui | I have noted a high latency after rebalancing ring operation. The system keeps high latency even after rebalancing operation is completed. | 17:25 |
| manoelrui | The swift environment was deployed with 1 proxy node and 6 storage nodes. | 17:25 |
| manoelrui | Do you know what is happening? | 17:25 |
| *** sunden has joined #openstack-swift | 17:28 | |
| *** gyee has joined #openstack-swift | 17:30 | |
| *** acoles is now known as acoles_ | 17:36 | |
| *** ukaynar_ has joined #openstack-swift | 17:39 | |
| *** manoelrui has quit IRC | 17:51 | |
| *** manoel_rui has joined #openstack-swift | 17:53 | |
| *** tqtran has quit IRC | 17:55 | |
| *** tqtran has joined #openstack-swift | 17:58 | |
| *** dmorita has quit IRC | 18:00 | |
| *** rcernin has joined #openstack-swift | 18:01 | |
| *** dmorita has joined #openstack-swift | 18:03 | |
| *** tqtran has quit IRC | 18:03 | |
| *** wer has quit IRC | 18:10 | |
| *** tqtran has joined #openstack-swift | 18:12 | |
| *** tqtran has quit IRC | 18:19 | |
| *** wer has joined #openstack-swift | 18:24 | |
| *** zaitcev has joined #openstack-swift | 18:26 | |
| *** ChanServ sets mode: +v zaitcev | 18:26 | |
| *** Jeffrey4l has quit IRC | 18:31 | |
| *** tqtran has joined #openstack-swift | 18:32 | |
| *** mvk has quit IRC | 18:38 | |
| *** dmorita has quit IRC | 18:56 | |
| *** dmorita has joined #openstack-swift | 18:56 | |
| clayg | patch 198633 looks cool! nice work kota_ acoles_ and tdasilva | 18:57 |
| patchbot | clayg: https://review.openstack.org/#/c/198633/ - swift - Last-Modified header support on HEAD/GET container (MERGED) | 18:57 |
| *** ukaynar_ has quit IRC | 18:59 | |
| timburke | oh, sweet! i hadn't noticed that it merged; I kept meaning to take a look at it. I suppose there's still time to review https://review.openstack.org/#/c/198634 though! | 18:59 |
| patchbot | timburke: patch 198634 - swift - Support last modified on listing containers | 19:00 |
| *** dmorita has quit IRC | 19:04 | |
| *** dmorita has joined #openstack-swift | 19:05 | |
| *** fifieldt has quit IRC | 19:06 | |
| *** asettle has joined #openstack-swift | 19:14 | |
| *** admin6 has quit IRC | 19:16 | |
| *** lcurtis has joined #openstack-swift | 19:18 | |
| *** fifieldt has joined #openstack-swift | 19:19 | |
| lcurtis | hello all...seeing certain devices continue to fill up even after rebalancing...is it a bad thing to have full disks in Swift? | 19:20 |
| *** asettle has quit IRC | 19:21 | |
| *** nikivi has quit IRC | 19:27 | |
| clayg | lcurtis: yeah most storage systems get sorta bummed about running out of capacity - swift's no magic there - although a few things have got in that made it less bad over the years | 19:30 |
| clayg | lcurtis: you running replicated storage policy? Probably need to start tuning your replicators. | 19:30 |
| *** tqtran has quit IRC | 19:34 | |
| *** tongli has quit IRC | 19:35 | |
| lcurtis | @clayg yes..running replicated | 19:35 |
| lcurtis | thank you | 19:35 |
| lcurtis | i think we ran for a period where our ingest was faster than replication | 19:35 |
| lcurtis | and also we had too many rsyncs running which killed performance | 19:36 |
| *** ukaynar_ has joined #openstack-swift | 19:36 | |
| *** manous has quit IRC | 19:40 | |
| clayg | ok, this is write-affinity cluster? | 19:40 |
| clayg | I'd recommend you migrate to rsync module per disk for better max_connections tuning to limit the effects of tarpit disks - then give http://docs.openstack.org/developer/swift/admin_guide.html?highlight=full#preventing-disk-full-scenarios a read | 19:42 |
| clayg | most of the time you can turn the replicators concurrency tunable up, more-so if you have dedicated replication object-server's | 19:43 |
| lcurtis | yes...we did turn it up | 19:43 |
| lcurtis | it was set to zero back from when we first set up cluster | 19:43 |
| lcurtis | doh | 19:43 |
| clayg | this is also a great strategy for monitoring overall rebalance progress -> http://docs.openstack.org/developer/swift/admin_guide.html?highlight=full#checking-handoff-partition-distribution | 19:44 |
| clayg | lcurtis: well that's good! | 19:44 |
| lcurtis | :) | 19:44 |
| lcurtis | many thanks clayg | 19:45 |
| *** tqtran has joined #openstack-swift | 19:47 | |
| *** manous has joined #openstack-swift | 19:49 | |
| notmyname | FYI the tc meeting is in 5 minutes in #openstack-meeting. among other things, they're talking about the tc-goals resolution and accepting storlets as an openstack project | 19:55 |
| notmyname | I'll be there, at least lurking. not sure how much I'll need to speak up | 19:56 |
| *** asettle has joined #openstack-swift | 19:59 | |
| *** asettle has quit IRC | 19:59 | |
| *** tqtran has quit IRC | 19:59 | |
| *** tqtran has joined #openstack-swift | 20:05 | |
| *** Trixboxer has quit IRC | 20:09 | |
| *** tqtran has quit IRC | 20:16 | |
| manoel_rui | clayg notmyname hi, can you help me ? | 20:17 |
| notmyname | manoel_rui: I'm in a meeting right now | 20:17 |
| manoel_rui | notmyname no problem, thanks | 20:18 |
| clayg | manoel_rui: what's up? | 20:18 |
| *** cdelatte has quit IRC | 20:19 | |
| manoel_rui | clayg I have noted a high latency after rebalancing ring operation. The system keeps high latency even after rebalancing operation is completed. | 20:20 |
| manoel_rui | clayg The swift environment was deployed with 1 proxy node and 6 storage nodes. | 20:20 |
| manoel_rui | clayg do you know what is happening or any idea ? | 20:21 |
| clayg | how are you measuring "rebalancing operation is complete" - how are you measuring latency for that matter? | 20:22 |
| *** nikivi has joined #openstack-swift | 20:23 | |
| clayg | it is not uncommon that large capacity adjustments - a rebalance - will add load to the cluster - which can impact overall quality of service - it's a trade off how fast you want the cluster topology to move vs how much you want to smooth out the service quality for clients - it's not always cut and dry | 20:23 |
| clayg | do you monior system stats - particularlly disk load? is this triple replicated rsync policy? seperate a/c tier? | 20:24 |
| notmyname | info: the tc-goals proposal has been approved as-is, with a comment in the meeting that wording changes can be considered separately as follow-on patches | 20:25 |
| ahale | do you think projects that fail to adequately document why they didnt achieve one of the targets will all be treated the same? even ones that have had the tc members write rants about not playing nicely with others | 20:26 |
| *** Trixboxer has joined #openstack-swift | 20:26 | |
| *** jordanP has joined #openstack-swift | 20:27 | |
| manoel_rui | clayg I am using statsd, monitoring some metrics like "object-replicator.suffix.syncs.count" | 20:27 |
| manoel_rui | clayg I am using cosbench for measuring latency | 20:28 |
| clayg | I think that suffix syncing should continue even after a rebalance completes = you should consider monitoring parition placement with something like dispersion-(populate|report) or http://docs.openstack.org/developer/swift/admin_guide.html#checking-handoff-partition-distribution | 20:29 |
| *** tqtran has joined #openstack-swift | 20:31 | |
| clayg | is the cluster fairly new then? there's a few implications of file system layout that don't really show up until you get enough objects in the cluster - the performance of a empty disk may not be the same as a cluster that's been running for years all other things being the same | 20:31 |
| manoel_rui | clayg what metric do you refer about disk load ? I am using 3 replicas for object and 2 replicas per a/c. All replicas are in same 6 storage nodes | 20:32 |
| *** tqtran has quit IRC | 20:34 | |
| clayg | how many devices per node? all storage tiers on all devices? | 20:35 |
| clayg | for disk load interactively I use iostat mostly I guess? but I have all alot of that stuff plumbed to graphite dashboards via collectd | 20:36 |
| *** ouchkernel has quit IRC | 20:37 | |
| manoel_rui | clayg The cluster is an experimental environment that I can delete all objects, so is a new cluster | 20:37 |
| manoel_rui | clayg one device per node and all tiers are spread to all devices | 20:39 |
| *** brnelson has quit IRC | 20:39 | |
| clayg | manoel_rui: it's not clear to me how this is related to rebalance then - you ran a cosbench and saw one latency - you ran it again and saw a different number? But there's lots of things that could cause that - how do you want to narrow it down? | 20:39 |
| clayg | i hate to ask... but are these vms? | 20:40 |
| *** tqtran has joined #openstack-swift | 20:42 | |
| notmyname | ahale: you may understand why I've been picky about the proposed wording ;-) | 20:43 |
| manoel_rui | clayg Basicly, I'm running the workload with a constant number of clients along the time without any performance loss. | 20:43 |
| ahale | :) | 20:43 |
| manoel_rui | clayg when I update the ring while system is running I can observe that performance gets down | 20:44 |
| manoel_rui | clayg so, after I wait for a long time the performance keeps worse. | 20:45 |
| clayg | ok, then - that's inline with my experience as well - rebalance moves data - moving data produces load - load from consistency engine can put backpressure on client facing quality of service - there's lots of tradeoffs available | 20:45 |
| clayg | how do you measure "a long time" - like wait two weeks? a day? I think you should observe your replicators and partition movement closely after introducing a ring change | 20:46 |
| manoel_rui | clayg I was hopping the performance would came back one moment. | 20:46 |
| manoel_rui | clayg and yes, I'm using VMs | 20:47 |
| *** tqtran has quit IRC | 20:48 | |
| clayg | a coworker of mine likes to remind us "RINGS ARE NOT MAGIC PIXY DUST THAT TRANSPORT DATA!!!" - I think his point is just that rebalancing clusers requires bytes to be read from disk, transfered over a wire and written onto a another disk - these operations take significantly longer than it does to rework the ring data structure - you should investigate this more closely and see if it matches your expectations/understanding/obs | 20:49 |
| *** jordanP has quit IRC | 20:52 | |
| ahale | i spent about 6 or 7 hours today poking replication in a cluster and was happy a moved a 'where data is and where it should be' guage by about 0.1% | 20:52 |
| *** ouchkernel has joined #openstack-swift | 20:55 | |
| manoel_rui | claygs I'm using a small workload, so need just few hours | 20:59 |
| *** ouchkernel has quit IRC | 21:00 | |
| *** ouchkernel has joined #openstack-swift | 21:01 | |
| *** arch-nemesis has quit IRC | 21:06 | |
| *** sunden has quit IRC | 21:06 | |
| clayg | manoel_rui: that's possible, maybe double check your logs, if replication is finishing and all partitions have correctly rebalanced then it's not clear to me why performance would be appreciably worse than before the rebalance - unless the total avaiable cluster resources have been reduced? | 21:06 |
| *** tqtran has joined #openstack-swift | 21:07 | |
| openstackgerrit | Tim Burke proposed openstack/swift: Drop X-Auth-Token from all versioned_writes subrequests https://review.openstack.org/307929 | 21:11 |
| manoel_rui | clayg I see, I will try explore closely this issues, may be I'm not seeing what resource is overloaded, thank you for you help | 21:12 |
| *** airdisa_ has quit IRC | 21:17 | |
| *** gyee has quit IRC | 21:20 | |
| *** mvk has joined #openstack-swift | 21:23 | |
| *** jordanP has joined #openstack-swift | 21:23 | |
| *** gyee has joined #openstack-swift | 21:23 | |
| *** rcernin has quit IRC | 21:27 | |
| *** jordanP has quit IRC | 21:32 | |
| *** tqtran has quit IRC | 21:37 | |
| *** tqtran has joined #openstack-swift | 21:37 | |
| *** tqtran has quit IRC | 21:42 | |
| *** nikivi has quit IRC | 21:52 | |
| *** tqtran has joined #openstack-swift | 21:54 | |
| *** tqtran has quit IRC | 21:59 | |
| *** tqtran has joined #openstack-swift | 22:10 | |
| *** chrichip has quit IRC | 22:13 | |
| mattoliverau | Morning | 22:16 |
| *** klamath has quit IRC | 22:20 | |
| *** tqtran has quit IRC | 22:20 | |
| *** tqtran has joined #openstack-swift | 22:27 | |
| *** admin6 has joined #openstack-swift | 22:33 | |
| *** tqtran has quit IRC | 22:36 | |
| *** tqtran has joined #openstack-swift | 22:39 | |
| *** tqtran has quit IRC | 22:43 | |
| *** thumpba has quit IRC | 22:46 | |
| *** thumpba has joined #openstack-swift | 22:49 | |
| *** thumpba has quit IRC | 22:49 | |
| *** admin6 has quit IRC | 22:54 | |
| *** chrichip has joined #openstack-swift | 22:58 | |
| *** thumpba has joined #openstack-swift | 22:59 | |
| *** thumpba has quit IRC | 22:59 | |
| *** thumpba has joined #openstack-swift | 22:59 | |
| *** asettle has joined #openstack-swift | 23:05 | |
| *** tqtran has joined #openstack-swift | 23:05 | |
| *** chrichip has quit IRC | 23:07 | |
| *** lcurtis has quit IRC | 23:08 | |
| *** asettle has quit IRC | 23:09 | |
| *** kei_yama has joined #openstack-swift | 23:16 | |
| *** tqtran has quit IRC | 23:19 | |
| *** ukaynar_ has quit IRC | 23:29 | |
| *** thumpba_ has joined #openstack-swift | 23:37 | |
| *** thumpba has quit IRC | 23:39 | |
| *** tqtran has joined #openstack-swift | 23:46 | |
| *** manous has quit IRC | 23:49 | |
| *** tqtran has quit IRC | 23:51 | |
| zaitcev | Everyone working on Swift becomes a master of stuffing API extensions into unexpected nooks and cracks. | 23:51 |
| zaitcev | Eventually. | 23:51 |
| *** arch-nemesis has joined #openstack-swift | 23:54 | |
| *** tqtran has joined #openstack-swift | 23:54 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!