| *** lcurtis has quit IRC | 00:00 | |
| mattoliverau | ho_: morning | 00:00 |
|---|---|---|
| ho_ | mattoliverau: morning! | 00:00 |
| openstackgerrit | Jonathan Hinson proposed openstack/swift: Automatic refresh of memcache config settings https://review.openstack.org/218490 | 00:07 |
| *** mragupat has quit IRC | 00:09 | |
| clayg | timburke: fwiw I can confirm the object-auditor is in a sad state - did you open a bug already? | 00:09 |
| timburke | clayg: not yet | 00:09 |
| clayg | timburke: well we better start there | 00:10 |
| timburke | yep | 00:10 |
| clayg | notmyname: I marked lp bug #1533002 critical - but if timburke isn't able to stick with it up to the pending release you might bump it down to HIGH - I'll be sad, but not offended | 00:13 |
| openstack | Launchpad bug 1533002 in OpenStack Object Storage (swift) "object-auditor skips EC fragments" [Critical,Confirmed] https://launchpad.net/bugs/1533002 | 00:13 |
| *** badari has quit IRC | 00:15 | |
| notmyname | clayg: ack. timburke, thanks for diving into that one! | 00:15 |
| clayg | timburke is the best! three cheers for timburke! | 00:15 |
| mattoliverau | Cheers! Cheers! Cheers! | 00:18 |
| openstackgerrit | Clay Gerrard proposed openstack/swift: functest for x-timestamp validation https://review.openstack.org/266085 | 00:26 |
| *** km___ has joined #openstack-swift | 00:32 | |
| *** km_ has quit IRC | 00:32 | |
| *** shakamunyi has quit IRC | 00:35 | |
| *** shakamunyi has joined #openstack-swift | 00:35 | |
| *** diogogmt has quit IRC | 00:37 | |
| kota_ | good morning | 00:43 |
| *** m_kazuhiro has joined #openstack-swift | 00:44 | |
| kota_ | clayg: thanks for reviewing my patch 265154 :) | 00:45 |
| patchbot | kota_: https://review.openstack.org/#/c/265154/ - Add note COPY with conditional headers | 00:45 |
| mattoliverau | kota_: morning | 00:54 |
| *** gyee has quit IRC | 00:55 | |
| kota_ | mattoliverau: morning :) | 00:56 |
| *** barra204 has joined #openstack-swift | 00:56 | |
| *** shakamunyi has quit IRC | 00:57 | |
| openstackgerrit | Tim Burke proposed openstack/swift: Autovivify X-Versions-Location container https://review.openstack.org/265015 | 00:58 |
| *** garthb__ has quit IRC | 01:09 | |
| *** natarej_ has quit IRC | 01:11 | |
| *** barra204 has quit IRC | 01:14 | |
| *** shakamunyi has joined #openstack-swift | 01:15 | |
| ho_ | kota_: morning! | 01:19 |
| kota_ | ho: morning! | 01:20 |
| takashi | good morning | 01:23 |
| kota_ | takashi: o/ | 01:24 |
| *** Jeffrey4l has quit IRC | 01:25 | |
| *** km___ has quit IRC | 01:26 | |
| *** km_ has joined #openstack-swift | 01:26 | |
| *** barra204 has joined #openstack-swift | 01:31 | |
| *** shakamunyi has quit IRC | 01:32 | |
| *** shakamunyi has joined #openstack-swift | 01:36 | |
| *** barra204 has quit IRC | 01:38 | |
| *** MVenesio has joined #openstack-swift | 01:38 | |
| *** barra204 has joined #openstack-swift | 01:38 | |
| *** asettle has quit IRC | 01:38 | |
| *** shakamunyi has quit IRC | 01:39 | |
| *** MVenesio has quit IRC | 01:42 | |
| *** shakamunyi has joined #openstack-swift | 01:44 | |
| *** barra204 has quit IRC | 01:45 | |
| *** haomaiwang has quit IRC | 01:48 | |
| *** shakamunyi has quit IRC | 01:48 | |
| *** 14WAAN277 has joined #openstack-swift | 01:48 | |
| *** shakamunyi has joined #openstack-swift | 01:48 | |
| *** asettle has joined #openstack-swift | 01:49 | |
| *** 14WAAN277 has quit IRC | 01:50 | |
| clayg | timur: yay! I owe you a beer and you owe one to mattoliverau ! | 01:53 |
| *** shakamunyi has quit IRC | 01:54 | |
| timur | clayg: woot! thanks for the reviews | 02:02 |
| timur | and thanks mattoliverau -- happy to buy that beer :) | 02:03 |
| *** sgundur has joined #openstack-swift | 02:05 | |
| *** sgundur has left #openstack-swift | 02:05 | |
| *** shakamunyi has joined #openstack-swift | 02:08 | |
| *** shakamunyi has quit IRC | 02:10 | |
| *** badari has joined #openstack-swift | 02:11 | |
| *** shakamunyi has joined #openstack-swift | 02:12 | |
| *** asettle has quit IRC | 02:16 | |
| *** asettle has joined #openstack-swift | 02:17 | |
| *** klrmn has quit IRC | 02:20 | |
| *** asettle has quit IRC | 02:22 | |
| *** shakamunyi has quit IRC | 02:22 | |
| *** shakamunyi has joined #openstack-swift | 02:23 | |
| mattoliverau | keep the cool patches comin | 02:24 |
| *** mingdang1 has joined #openstack-swift | 02:28 | |
| *** shakamunyi has quit IRC | 02:32 | |
| *** diogogmt has joined #openstack-swift | 02:33 | |
| *** asettle has joined #openstack-swift | 02:47 | |
| *** sanchitmalhotra has joined #openstack-swift | 02:49 | |
| *** venkat has joined #openstack-swift | 03:14 | |
| *** mragupat has joined #openstack-swift | 03:18 | |
| *** janonymous has joined #openstack-swift | 03:20 | |
| openstackgerrit | venkatesh proposed openstack/swift: check_object_creation: method call is redundant https://review.openstack.org/266131 | 03:23 |
| *** klrmn has joined #openstack-swift | 03:25 | |
| *** dimasot has joined #openstack-swift | 03:28 | |
| *** badari_ has joined #openstack-swift | 03:36 | |
| *** MVenesio has joined #openstack-swift | 03:39 | |
| *** nadeem has joined #openstack-swift | 03:39 | |
| *** badari has quit IRC | 03:39 | |
| *** ppai has joined #openstack-swift | 03:39 | |
| *** MVenesio has quit IRC | 03:44 | |
| *** janonymous has quit IRC | 03:47 | |
| *** badari_ has quit IRC | 03:50 | |
| pdardeau | clayg: thanks for review on dev id holes patch | 04:06 |
| *** asettle has quit IRC | 04:15 | |
| *** dmorita has quit IRC | 04:25 | |
| *** dmorita has joined #openstack-swift | 04:26 | |
| *** links has joined #openstack-swift | 04:26 | |
| *** janonymous has joined #openstack-swift | 04:32 | |
| openstackgerrit | venkatesh proposed openstack/swift: check_object_creation: method call is redundant https://review.openstack.org/266131 | 04:37 |
| *** nadeem has quit IRC | 04:42 | |
| *** asettle has joined #openstack-swift | 04:46 | |
| *** janonymous_ has joined #openstack-swift | 04:48 | |
| *** janonymous has quit IRC | 04:50 | |
| *** asettle has quit IRC | 04:52 | |
| *** asettle has joined #openstack-swift | 04:52 | |
| *** itlinux has joined #openstack-swift | 05:04 | |
| *** itlinux has quit IRC | 05:04 | |
| *** klrmn has quit IRC | 05:29 | |
| *** trifon has joined #openstack-swift | 05:30 | |
| *** janonymous_ has quit IRC | 05:32 | |
| *** mragupat has quit IRC | 05:32 | |
| *** janonymous has joined #openstack-swift | 05:35 | |
| *** MVenesio has joined #openstack-swift | 05:39 | |
| *** MVenesio has quit IRC | 05:44 | |
| openstackgerrit | venkatamahesh proposed openstack/swift: Improvements in python conditional code https://review.openstack.org/266185 | 05:49 |
| openstackgerrit | Hisashi Osanai proposed openstack/swift: Fix posting accounts behavior when half of account servers downed https://review.openstack.org/266190 | 05:57 |
| openstackgerrit | Hisashi Osanai proposed openstack/swift: Fix posting containers behavior when half of container servers downed https://review.openstack.org/266193 | 06:00 |
| *** bkumar has joined #openstack-swift | 06:01 | |
| openstackgerrit | Hisashi Osanai proposed openstack/swift: Fix putting containers behavior during migration of accounts https://review.openstack.org/266197 | 06:02 |
| *** dmorita has quit IRC | 06:02 | |
| openstackgerrit | Hisashi Osanai proposed openstack/swift: Fix deleting containers behavior https://review.openstack.org/266199 | 06:04 |
| *** Jeffrey4l has joined #openstack-swift | 06:16 | |
| *** asettle has quit IRC | 06:17 | |
| *** 6A4ABMVDU has joined #openstack-swift | 06:26 | |
| *** dmorita has joined #openstack-swift | 06:26 | |
| *** dmorita has quit IRC | 06:28 | |
| *** dmorita has joined #openstack-swift | 06:28 | |
| *** ChubYann has quit IRC | 06:43 | |
| *** esker has joined #openstack-swift | 06:49 | |
| *** natarej has joined #openstack-swift | 06:49 | |
| *** esker has quit IRC | 06:53 | |
| *** dmorita has quit IRC | 06:58 | |
| *** silor has joined #openstack-swift | 06:59 | |
| *** 6A4ABMVDU has quit IRC | 07:03 | |
| *** haomaiwang has joined #openstack-swift | 07:05 | |
| *** dmorita has joined #openstack-swift | 07:07 | |
| *** zaitcev has quit IRC | 07:08 | |
| *** ntt has joined #openstack-swift | 07:25 | |
| *** haomaiwang has quit IRC | 07:25 | |
| *** haomaiwang has joined #openstack-swift | 07:26 | |
| *** SkyRocknRoll has joined #openstack-swift | 07:28 | |
| *** MVenesio has joined #openstack-swift | 07:40 | |
| takashi | ho_: are you available now? | 07:44 |
| *** MVenesio has quit IRC | 07:44 | |
| openstackgerrit | Reedip proposed openstack/swift-bench: Remove support for py33/py26 https://review.openstack.org/264486 | 07:45 |
| *** haomaiwang has quit IRC | 07:56 | |
| *** haomaiwa_ has joined #openstack-swift | 07:58 | |
| *** haomaiwa_ has quit IRC | 07:59 | |
| *** haomaiwa_ has joined #openstack-swift | 07:59 | |
| *** SkyRocknRoll has quit IRC | 08:00 | |
| *** haomaiwa_ has quit IRC | 08:01 | |
| *** ntt has quit IRC | 08:01 | |
| *** haomaiwa_ has joined #openstack-swift | 08:01 | |
| *** arnox has joined #openstack-swift | 08:03 | |
| *** jistr has joined #openstack-swift | 08:06 | |
| *** jistr is now known as jistr|doc | 08:07 | |
| openstackgerrit | janonymous proposed openstack/swift: Added GreenAsyncPile txn_id logging https://review.openstack.org/266239 | 08:08 |
| openstackgerrit | Merged openstack/swift: Fix IPv6 handling in MemcacheConnPool. https://review.openstack.org/258704 | 08:09 |
| *** rledisez has joined #openstack-swift | 08:09 | |
| *** dmorita has quit IRC | 08:09 | |
| *** ntt has joined #openstack-swift | 08:11 | |
| *** esker has joined #openstack-swift | 08:12 | |
| *** SkyRocknRoll has joined #openstack-swift | 08:14 | |
| *** ntt has quit IRC | 08:15 | |
| *** esker has quit IRC | 08:17 | |
| *** dmorita has joined #openstack-swift | 08:23 | |
| *** haomaiwa_ has quit IRC | 08:26 | |
| *** dmorita has quit IRC | 08:32 | |
| *** asettle has joined #openstack-swift | 08:32 | |
| *** ntt has joined #openstack-swift | 08:32 | |
| *** haomaiwa_ has joined #openstack-swift | 08:33 | |
| *** venkat has quit IRC | 08:36 | |
| ho_ | takashi: yeah but i'm in meeting so quick response is difficult :) | 08:40 |
| *** haomaiwa_ has quit IRC | 08:42 | |
| *** haomaiwa_ has joined #openstack-swift | 08:44 | |
| *** geaaru has joined #openstack-swift | 08:44 | |
| *** venkat has joined #openstack-swift | 08:47 | |
| *** jmccarthy has quit IRC | 08:48 | |
| *** jmccarthy has joined #openstack-swift | 08:50 | |
| takashi | ho_: OK. I posted a comment on patch 266197, so please check that. | 08:53 |
| patchbot | takashi: https://review.openstack.org/#/c/266197/ - Fix putting containers behavior during migration o... | 08:53 |
| *** esker has joined #openstack-swift | 08:53 | |
| *** ppai_ has joined #openstack-swift | 08:56 | |
| *** esker has quit IRC | 08:58 | |
| *** ppai has quit IRC | 08:58 | |
| *** jistr|doc has quit IRC | 08:59 | |
| *** haomaiwa_ has quit IRC | 09:01 | |
| *** 77CAAI2UM has joined #openstack-swift | 09:01 | |
| ho_ | takashi: interesting! I couldn't read all the history but you already know current my concern behavior difference b/w put_object and put_container. | 09:05 |
| *** jordanP has joined #openstack-swift | 09:08 | |
| *** daemontool has joined #openstack-swift | 09:11 | |
| *** joeljwright has joined #openstack-swift | 09:14 | |
| *** ChanServ sets mode: +v joeljwright | 09:14 | |
| takashi | ho_: yes. I think that behavior works was decided because swift in old version does not user handoff so actively for account/container, | 09:17 |
| *** openstackgerrit has quit IRC | 09:17 | |
| takashi | s/user/use | 09:17 |
| takashi | but nowadays swift uses handoff effectively for account, container, based on request_node_count | 09:17 |
| *** openstackgerrit has joined #openstack-swift | 09:18 | |
| takashi | and I wondered if we should keep that behavior now, when writing that patch. (sorry for my inactivity...) | 09:18 |
| *** jistr has joined #openstack-swift | 09:20 | |
| *** jordanP has quit IRC | 09:22 | |
| *** jordanP has joined #openstack-swift | 09:33 | |
| *** esker has joined #openstack-swift | 09:35 | |
| *** proteusguy has quit IRC | 09:35 | |
| *** aix has joined #openstack-swift | 09:37 | |
| *** esker has quit IRC | 09:40 | |
| *** km_ has quit IRC | 09:51 | |
| *** km_ has joined #openstack-swift | 09:51 | |
| *** proteusguy has joined #openstack-swift | 09:52 | |
| *** ho_ has quit IRC | 09:52 | |
| *** sileht has quit IRC | 09:52 | |
| *** venkat has quit IRC | 09:54 | |
| *** ho has joined #openstack-swift | 09:55 | |
| *** ho is now known as Guest47795 | 09:55 | |
| *** SkyRocknRoll has quit IRC | 09:55 | |
| *** 77CAAI2UM has quit IRC | 10:01 | |
| *** sileht has joined #openstack-swift | 10:02 | |
| Guest47795 | takashi: i'm back. | 10:03 |
| *** Guest47795 is now known as ho_ | 10:03 | |
| ho_ | takashi: ^^^ ho_ | 10:03 |
| *** acoles_ is now known as acoles | 10:04 | |
| *** jordanP has quit IRC | 10:04 | |
| acoles | good morning | 10:06 |
| ho_ | ho_: i will check more but current my thought is it's better to make a fix in proxy side rather than container-server. because each container-server doesn't have all account servers (some) | 10:06 |
| *** venkat has joined #openstack-swift | 10:06 | |
| ho_ | takashi: :-) | 10:06 |
| *** SkyRocknRoll has joined #openstack-swift | 10:07 | |
| *** haomaiwa_ has joined #openstack-swift | 10:07 | |
| ho_ | takashi: therefore container-server can not handle correct status for the account. i think it's a role for proxy-server. | 10:08 |
| ho_ | takashi: i might have mis-understanding but container/server.py:L234 in your patch means "ERROR the account is missing on SOME account servers" i think. | 10:11 |
| ho_ | acoles: morning! | 10:12 |
| *** esker has joined #openstack-swift | 10:16 | |
| *** vinsh_ has joined #openstack-swift | 10:16 | |
| *** AbyssOne_ has joined #openstack-swift | 10:19 | |
| *** vinsh has quit IRC | 10:19 | |
| *** AbyssOne has quit IRC | 10:19 | |
| *** gmmaha has quit IRC | 10:19 | |
| *** haomaiwa_ has quit IRC | 10:20 | |
| *** gmmaha has joined #openstack-swift | 10:20 | |
| mahatic | acoles: good morning | 10:21 |
| *** esker has quit IRC | 10:21 | |
| *** SkyRocknRoll has quit IRC | 10:23 | |
| *** ntt has quit IRC | 10:26 | |
| takashi | acoles: good morning | 10:26 |
| takashi | ho_: you are right. I think I shoud change that message like this. ERROR the all account assigned to me is missing | 10:27 |
| takashi | s/the all account/all the accounts/ | 10:29 |
| takashi | s/is/are | 10:29 |
| *** aix has quit IRC | 10:32 | |
| takashi | ho_: If we determine to overwrite responses from container server based on the whole situation, I think we have to do that in proxy-server. | 10:33 |
| ho_ | takashi: overrides works only split brain case. | 10:34 |
| *** venkat has quit IRC | 10:35 | |
| *** jordanP has joined #openstack-swift | 10:35 | |
| takashi | ho_: I NOW understand. That's good! | 10:36 |
| acoles | takashi: are you Takashi Kajinami? | 10:36 |
| takashi | acoles: yes | 10:36 |
| acoles | takashi: i just saw that you registered for the hackathon :) | 10:37 |
| takashi | acoles: Yes! I'm now working with kota_, and he told me about the hackathon :-) | 10:39 |
| acoles | takashi: great | 10:39 |
| kota_ | acoles, takashi: we will be in Bristol ;) | 10:40 |
| takashi | kota_, acoles: yeah, and this is my first swift hackathon, and also my first visit to England. I'm very looking forward to it. | 10:43 |
| acoles | takashi: nice | 10:45 |
| *** venkat has joined #openstack-swift | 10:48 | |
| *** aix has joined #openstack-swift | 10:53 | |
| *** esker has joined #openstack-swift | 10:58 | |
| ho_ | takashi: unn... difficult. both approaches are fine for me. my patch can handle migration state but you patch's behavior is same as object ... | 10:58 |
| *** esker has quit IRC | 11:02 | |
| ho_ | takashi: do you have a plan to re-open the patch? | 11:13 |
| *** bkumar has quit IRC | 11:14 | |
| takashi | ho_: I'll re-open it if necessory. I think I should rebase it. | 11:16 |
| *** openstackgerrit has quit IRC | 11:17 | |
| ho_ | takashi: k, I put my +1 on your patch so I will abandoned my patch after you will re-open it. | 11:18 |
| *** openstackgerrit has joined #openstack-swift | 11:18 | |
| takashi | ho_: Could you give me a time for that? I think I have to think more to decide which one is better. | 11:19 |
| takashi | ho_: I think I can take some time for that tommorow | 11:20 |
| ho_ | takashi: np! if you rebase your patch, please remove L233-234 in server.py. your approach doesn't need to care the case. this is the reason why i choose yours :-) | 11:21 |
| takashi | ho_: thanks! | 11:22 |
| *** ho_ has quit IRC | 11:24 | |
| *** takashi has quit IRC | 11:25 | |
| *** dmorita has joined #openstack-swift | 11:33 | |
| *** dmorita has quit IRC | 11:37 | |
| *** sanchitmalhotra has quit IRC | 11:42 | |
| *** venkat has quit IRC | 11:43 | |
| *** mingdang1 has quit IRC | 11:46 | |
| *** mingdang1 has joined #openstack-swift | 11:47 | |
| *** McMurlock has quit IRC | 11:51 | |
| *** mingdang1 has quit IRC | 11:51 | |
| *** venkat has joined #openstack-swift | 11:54 | |
| *** MVenesio has joined #openstack-swift | 11:57 | |
| *** McMurlock has joined #openstack-swift | 11:58 | |
| *** asettle has quit IRC | 12:03 | |
| *** asettle has joined #openstack-swift | 12:04 | |
| *** asettle has quit IRC | 12:08 | |
| *** kei_yama has quit IRC | 12:19 | |
| kota_ | am heading for my home | 12:22 |
| *** ppai_ has quit IRC | 12:24 | |
| *** ho_away has joined #openstack-swift | 12:26 | |
| *** aix has quit IRC | 12:29 | |
| *** aix has joined #openstack-swift | 12:32 | |
| *** ppai_ has joined #openstack-swift | 12:38 | |
| *** mingdang1 has joined #openstack-swift | 12:39 | |
| *** ho_away has quit IRC | 12:39 | |
| *** d0ugal has quit IRC | 12:42 | |
| *** d0ugal has joined #openstack-swift | 12:43 | |
| *** d0ugal is now known as Guest58385 | 12:43 | |
| *** Guest58385 is now known as d0ugal | 12:45 | |
| *** d0ugal has quit IRC | 12:45 | |
| *** d0ugal has joined #openstack-swift | 12:45 | |
| *** m_kazuhiro has quit IRC | 12:46 | |
| *** haomaiwang has joined #openstack-swift | 12:56 | |
| *** km_ has quit IRC | 12:57 | |
| *** haomaiwang has quit IRC | 13:01 | |
| *** ppai_ has quit IRC | 13:01 | |
| venkat | Hi all | 13:02 |
| venkat | what is the use of X-Backend-Obj-Multiphase-Commit header? | 13:02 |
| venkat | In general, X-Backend-Obj-* headers ? | 13:02 |
| venkat | these are passed from proxy right. not from the user?? | 13:03 |
| *** shakamunyi has joined #openstack-swift | 13:04 | |
| *** haomaiwang has joined #openstack-swift | 13:06 | |
| onovy | clayg: hi. i replied to your comment for ionice/nice support. it's bit longer, sorry for it. I just want to explain everything correctly. That review is really long so i made summary. | 13:07 |
| *** haomaiwang has quit IRC | 13:11 | |
| *** lpabon_ has joined #openstack-swift | 13:15 | |
| *** ppai_ has joined #openstack-swift | 13:15 | |
| *** venkatesh has joined #openstack-swift | 13:15 | |
| *** lpabon has quit IRC | 13:15 | |
| *** venkat has quit IRC | 13:15 | |
| *** john_b has left #openstack-swift | 13:17 | |
| *** ho_away has joined #openstack-swift | 13:21 | |
| *** haomaiwang has joined #openstack-swift | 13:23 | |
| *** mingdang_ has joined #openstack-swift | 13:23 | |
| *** mingdang1 has quit IRC | 13:24 | |
| *** diogogmt has quit IRC | 13:27 | |
| *** diogogmt has joined #openstack-swift | 13:27 | |
| *** haomaiwang has quit IRC | 13:28 | |
| *** bkumar has joined #openstack-swift | 13:32 | |
| *** bkumar has quit IRC | 13:34 | |
| *** dimasot has quit IRC | 13:34 | |
| *** bkumar has joined #openstack-swift | 13:35 | |
| *** john_bar has joined #openstack-swift | 13:36 | |
| *** ppai_ has quit IRC | 13:38 | |
| acoles | venkatesh: any header name starting X-Backend- is reserved for swift internal use and they are removed from any client request. | 13:43 |
| *** diogogmt has quit IRC | 13:43 | |
| acoles | venkat: ^^ (same person, two nicks??) | 13:44 |
| acoles | X-Backend-Obj-Multiphase-Commit header is used in erasure coded object PUT to inform the object server that there are two phases to the put (send data, then commit) | 13:45 |
| venkatesh | acoles: Thanks, just now I have seen gatekeeper is removing those headers | 13:45 |
| venkatesh | these headers will be saved with object?? | 13:46 |
| acoles | venkatesh: yes, removed by gatekeeper, no not saved with object. | 13:46 |
| acoles | venkatesh: internal metadata that needs to be saved with object uses x-object-sysmeta-* which is also removed by gatekeeper | 13:47 |
| venkatesh | acoles: ya, x-object-sysmeta-* it is removing | 13:48 |
| acoles | x-backend- headers are used proxy->backend but also between obj servers for example during replication | 13:48 |
| venkatesh | acoles: Ok, Thanks | 13:49 |
| *** bkumar has quit IRC | 13:49 | |
| venkatesh | Hi | 13:54 |
| venkatesh | what is the use of X-Obj-Metadata-Footer? | 13:54 |
| openstackgerrit | Merged openstack/swift: Add note COPY with conditional headers https://review.openstack.org/265154 | 13:54 |
| venkatesh | it seems custom-metata used object PUT | 13:54 |
| *** bkumar has joined #openstack-swift | 13:55 | |
| *** haomaiwa_ has joined #openstack-swift | 13:59 | |
| *** haomaiwa_ has quit IRC | 14:01 | |
| *** haomaiwang has joined #openstack-swift | 14:01 | |
| *** venkatesh has quit IRC | 14:04 | |
| *** bkumar has quit IRC | 14:05 | |
| *** bkumar has joined #openstack-swift | 14:06 | |
| *** bkumar has quit IRC | 14:10 | |
| *** yatin has joined #openstack-swift | 14:16 | |
| *** tongli has joined #openstack-swift | 14:18 | |
| *** mingdang_ has quit IRC | 14:20 | |
| *** dimasot has joined #openstack-swift | 14:23 | |
| *** mingdang1 has joined #openstack-swift | 14:24 | |
| *** mrmoje has joined #openstack-swift | 14:26 | |
| acoles | sometimes the proxy needs to send the object server 'headers' after the object body as 'footers' - X-Obj-Metadata-Footer is used by the obj server to let the proxy know that its ok to send footers. | 14:26 |
| acoles | sometimes == for example during a erasure coded object put | 14:27 |
| *** esker has joined #openstack-swift | 14:29 | |
| *** petertr7_away is now known as petertr7 | 14:30 | |
| *** esker has quit IRC | 14:32 | |
| *** aix has quit IRC | 14:33 | |
| *** esker has joined #openstack-swift | 14:33 | |
| *** aix has joined #openstack-swift | 14:33 | |
| *** esker has quit IRC | 14:38 | |
| *** lcurtis has joined #openstack-swift | 14:45 | |
| *** sgundur has joined #openstack-swift | 14:45 | |
| *** ho_away has quit IRC | 14:45 | |
| *** blmartin has joined #openstack-swift | 14:48 | |
| *** links has quit IRC | 14:50 | |
| *** yatin has quit IRC | 14:54 | |
| *** hseipp has joined #openstack-swift | 14:57 | |
| *** lpabon has joined #openstack-swift | 15:05 | |
| *** esker has joined #openstack-swift | 15:05 | |
| *** esker has quit IRC | 15:06 | |
| *** esker has joined #openstack-swift | 15:07 | |
| *** trifon has quit IRC | 15:08 | |
| *** petertr7 is now known as petertr7_away | 15:10 | |
| *** diazjf has joined #openstack-swift | 15:11 | |
| *** petertr7_away is now known as petertr7 | 15:18 | |
| *** mingdang1 has quit IRC | 15:20 | |
| *** haomaiwang has quit IRC | 15:24 | |
| *** diazjf has quit IRC | 15:27 | |
| *** breitz has quit IRC | 15:32 | |
| *** breitz has joined #openstack-swift | 15:32 | |
| *** mragupat has joined #openstack-swift | 15:37 | |
| *** diazjf has joined #openstack-swift | 15:38 | |
| *** yatin has joined #openstack-swift | 15:56 | |
| *** yatin_ has joined #openstack-swift | 15:57 | |
| *** yatin has quit IRC | 16:01 | |
| *** SkyRocknRoll has joined #openstack-swift | 16:03 | |
| *** aix has quit IRC | 16:07 | |
| *** garthb__ has joined #openstack-swift | 16:08 | |
| *** nadeem has joined #openstack-swift | 16:10 | |
| openstackgerrit | Jonathan Hinson proposed openstack/swift: Automatic refresh of memcache config settings https://review.openstack.org/218490 | 16:14 |
| *** asti has joined #openstack-swift | 16:18 | |
| *** diogogmt has joined #openstack-swift | 16:20 | |
| *** asti has left #openstack-swift | 16:23 | |
| *** badari has joined #openstack-swift | 16:23 | |
| *** diogogmt has quit IRC | 16:27 | |
| *** diazjf has quit IRC | 16:29 | |
| *** diogogmt has joined #openstack-swift | 16:29 | |
| *** badari has quit IRC | 16:33 | |
| *** badari has joined #openstack-swift | 16:33 | |
| *** jamielennox is now known as jamielennox|away | 16:37 | |
| *** diazjf has joined #openstack-swift | 16:38 | |
| notmyname | good morning | 16:40 |
| *** petertr7 is now known as petertr7_away | 16:41 | |
| joeljwright | morning | 16:41 |
| pdardeau | good morning | 16:41 |
| joeljwright | (well, I wish it was still morning - then I might get everything done today!) | 16:41 |
| notmyname | heh | 16:41 |
| notmyname | interesting ML discussion on bug 1512207 (and more generally about bugs/patches like that) | 16:42 |
| openstack | bug 1512207 in tuskar "Fix usage of assertions" [Undecided,In progress] https://launchpad.net/bugs/1512207 - Assigned to Swapnil Kulkarni (coolsvap) | 16:42 |
| *** petertr7_away is now known as petertr7 | 16:42 | |
| *** jordanP has quit IRC | 16:43 | |
| *** jordanP has joined #openstack-swift | 16:43 | |
| notmyname | seems to be a good bit of frustration. see the ML thread with the subject "spam of patches" | 16:44 |
| *** nadeem has quit IRC | 16:48 | |
| *** nadeem has joined #openstack-swift | 16:49 | |
| mahatic | notmyname: morning! | 16:50 |
| *** SkyRocknRoll has quit IRC | 16:53 | |
| *** zaitcev has joined #openstack-swift | 16:54 | |
| *** ChanServ sets mode: +v zaitcev | 16:54 | |
| *** CaioBrentano has joined #openstack-swift | 16:58 | |
| *** gyee has joined #openstack-swift | 17:01 | |
| acoles | notmyname: yeah, if pointless > 0: wont_fix = pointless | 17:01 |
| *** Jeffrey4l has quit IRC | 17:02 | |
| acoles | notmyname: or should that be if pointless: <slap> | 17:02 |
| notmyname | :-) | 17:05 |
| *** tongli has quit IRC | 17:07 | |
| *** yatin has joined #openstack-swift | 17:12 | |
| notmyname | good point here. we really should be considering the implications of py4. http://astrofrog.github.io/blog/2016/01/12/stop-writing-python-4-incompatible-code/ | 17:13 |
| notmyname | I know that's especially important to torgomatic_ | 17:13 |
| notmyname | also, why stop there. can we go ahead and make sure our code is py5 and py6 compatible? | 17:14 |
| *** yatin_ has quit IRC | 17:16 | |
| *** daemontool has quit IRC | 17:22 | |
| *** bdrich has joined #openstack-swift | 17:25 | |
| timburke | notmyname: snark aside, it looks like swiftclient would stop printing *anything* if neither six.PY2 nor six.PY3 is True: https://github.com/openstack/python-swiftclient/blob/2.7.0/swiftclient/multithreading.py#L74-L80 | 17:26 |
| *** bdrich has left #openstack-swift | 17:27 | |
| *** yatin has quit IRC | 17:28 | |
| notmyname | yeah, the advice int eh article isn't bad. it's just funny to see it presented like it was (Py4) | 17:28 |
| *** dmorita has joined #openstack-swift | 17:34 | |
| *** geaaru has quit IRC | 17:35 | |
| *** dmorita has quit IRC | 17:35 | |
| pdardeau | notmyname: i created a couple of bps related to ring | 17:36 |
| *** petertr7 is now known as petertr7_away | 17:37 | |
| *** esker has quit IRC | 17:39 | |
| *** dmorita has joined #openstack-swift | 17:39 | |
| *** garthb has joined #openstack-swift | 17:41 | |
| *** coreycb has joined #openstack-swift | 17:43 | |
| *** garthb__ has quit IRC | 17:43 | |
| coreycb | is there a new swift release coming out anytime soon? | 17:43 |
| *** rledisez has quit IRC | 17:47 | |
| *** jordanP has quit IRC | 17:48 | |
| onovy | notmyname: ^ question for you i think :) | 17:49 |
| clayg | ^ this is the burning question on everyones mind! | 17:49 |
| onovy | clayg: hi. can we continue with our discussion about ionice? or do you want to talk about it at meeting tomorrow? | 17:54 |
| onovy | coreycb: can i ask you why are you asking? :) you are on debian openstack channel, so is it related? :) | 17:55 |
| coreycb | onovy, nope not related I don't think. we just have 2.5.0 in liberty and mitaka at this point and was wondering if there's a new version coming out soon that would correspond to mitaka. | 17:57 |
| onovy | ah :) no mitaka swift release yet, thats right | 17:57 |
| *** klrmn has joined #openstack-swift | 17:58 | |
| *** diazjf has quit IRC | 17:58 | |
| *** dmorita has quit IRC | 18:01 | |
| *** xzvk has quit IRC | 18:01 | |
| *** dmorita has joined #openstack-swift | 18:01 | |
| *** dmorita has quit IRC | 18:01 | |
| *** diogogmt has quit IRC | 18:02 | |
| *** jistr has quit IRC | 18:03 | |
| *** dmorita has joined #openstack-swift | 18:04 | |
| clayg | onovy: i was about to be afk - i will be at the meeting tmrw | 18:05 |
| clayg | onovy: did you add some feedback for me to the review - i'm sure we can make progress async - although I'm happy to discuss higher bandwidth at some point too | 18:06 |
| openstackgerrit | Jonathan Hinson proposed openstack/swift: Functional tests for if-match with multiple etags https://review.openstack.org/266516 | 18:07 |
| *** diogogmt has joined #openstack-swift | 18:07 | |
| clayg | onovy: FWIW, I think I can see why the feature might be "nice to have" but I probably rate the value lower and the cost higher than you do - it's a judgement call no doubt - i'm sure there exists a reasonable argument; i'm less sure there exists a fully objective quantification of it it "should" go in? | 18:08 |
| *** dmorita has quit IRC | 18:08 | |
| clayg | onovy: maybe ML post to sample some other swift deployer/operator's "gut feel" would be the most direct route? | 18:08 |
| *** dmorita has joined #openstack-swift | 18:10 | |
| clayg | onovy: well I just read your comments, and I'm not sure I follow some of the rebuttles you laid out. | 18:11 |
| onovy | rebuttles? what do you mean? | 18:13 |
| clayg | onovy: one thing that might help me *greatly* is if you could point to some prior art? What other software systems do you know of that have native ionice config options like this? postgresql or something like that? | 18:14 |
| openstackgerrit | Ondřej Nový proposed openstack/swift: Change schedule priority of daemon/server in config https://review.openstack.org/238799 | 18:14 |
| onovy | clayg: what other systems you know that have something like swift-init? :) | 18:15 |
| onovy | if we don't have swift-init, i'm fine with ionice/nice inside init script | 18:15 |
| onovy | because admin can't restart daemon with correct ionice | 18:15 |
| onovy | but we have | 18:15 |
| onovy | excatly can, but it's possible to do it wrong with init script + swift-init combination | 18:16 |
| onovy | i'm ok with nice/ionice in init script, but that's means don't use swift-init in production at all | 18:16 |
| clayg | onovy: that argument tells me we should just get rid of swift-init | 18:16 |
| onovy | clayg: that's one way | 18:17 |
| onovy | or just say: don't use it in production, use it for SAIO for example | 18:17 |
| clayg | onovy: we recently worked on our systemd support and swift-init is basically imcompatible with all that best I can tell | 18:17 |
| onovy | but from other point, swift-init all restart <-- this is really helful | 18:17 |
| onovy | hmm | 18:17 |
| onovy | and another point: how we can do that "next" optimalization i mention? | 18:18 |
| onovy | for example set header for account-reaper for all DEL reqs | 18:18 |
| clayg | onovy: maybe alias swift-init or something, idk, yeah it's a useful tool - but if it's getting in the way of operations we should kick it to the curb | 18:18 |
| clayg | onovy: i'm not really sure - ionice is a neat thing, but it seems like in the application itself we have access to better methods for applying ratelimiting and turning | 18:19 |
| *** hseipp has quit IRC | 18:19 | |
| clayg | ionice is blunt instrument to whack at a process like it's a black box - I think we could do better than that if we desire to | 18:20 |
| onovy | hmm, and nice? | 18:20 |
| onovy | what should be better to tell kernel: prioritize this to this | 18:20 |
| onovy | than | 18:20 |
| onovy | ratelimiting is fine, but it's another usecase | 18:21 |
| onovy | for example: i want to do object-audit as fast as possible | 18:21 |
| onovy | BUT don't slow down our users | 18:21 |
| onovy | this is not possible on application level | 18:21 |
| onovy | exactly it's, but what is cost :) | 18:21 |
| onovy | there we have few lines of codes which can do it | 18:21 |
| clayg | onovy: well, i need to sign off for a bit - but if you could please find a few other storage-ish system (perferably opensource) that do something like what you are proposing - and make me a list - it would go a good way to impress on me it's worth pursuing | 18:22 |
| onovy | clayg: ok, thank you | 18:22 |
| *** dmorita has quit IRC | 18:22 | |
| clayg | onovy: and I don't nessecarily know why you say it's not possible at the application level - but current "army of independent processes" design in swift makes it's inconvient; and the fact that we have a distributed system makes co-ordination a non trivial problem always - but if we're sure this is the biggest problem - i'm sure we could put a dent in it | 18:24 |
| *** badari has quit IRC | 18:26 | |
| clayg | onovy: the other way to demonstrate results would be with some benchmarks - taking the "convience" of having swift-init do the nicing aside for a moment - could you do an a/b comparisson of cluster with your suggest nice settings and w/o and show there's a big difference in how the system is able to respond to user iops while compeating with replication? | 18:26 |
| *** badari has joined #openstack-swift | 18:27 | |
| clayg | if they're indeed significant and reproducible it may be very attractive indeed to support the feature in swift process management first class - but in my mind this is TBD | 18:27 |
| *** joeljwright has quit IRC | 18:28 | |
| clayg | notmyname: ahale: have you ever heard of a swift deployment where the operators are agressively tuning nice settings per process? | 18:28 |
| *** badari has quit IRC | 18:29 | |
| blmartin | :wq | 18:30 |
| blmartin | maybe I should make an alias so that ':wq' in irc changes to 'Hello all!' | 18:30 |
| blmartin | so that it isn't so awkward for me | 18:31 |
| onovy | so compare object-server with object-replicator? | 18:32 |
| onovy | and metrick is "latency"? | 18:32 |
| onovy | clayg: http://engineering.spilgames.com/openstack-swift-lots-small-files/ | 18:33 |
| onovy | so all of you already heard :) | 18:33 |
| onovy | i will graph object-server and object-auditor. It's much simpler (no rsync here) | 18:38 |
| *** dmorita has joined #openstack-swift | 18:38 | |
| *** dmorita has quit IRC | 18:39 | |
| *** dmorita has joined #openstack-swift | 18:39 | |
| *** badari has joined #openstack-swift | 18:40 | |
| *** diazjf has joined #openstack-swift | 18:40 | |
| ahale | clayg: i have some vague recollection that we considered it sometimes ages ago but never bothered. didnt one of the comments say that ibm were doing it ? | 18:45 |
| ahale | on that nice'ing processes change | 18:45 |
| onovy | 21:34:45 <briancline> onovy: I actually asked our ops folks about this recently. we already control these in production through other means, so we're fine with this going in as long as its optional, which it certainly seems to be | 18:49 |
| onovy | this=nice/ionice? | 18:49 |
| *** ChubYann has joined #openstack-swift | 18:49 | |
| onovy | 21:35:34 <briancline> i can see situations later where it'll help us to have it in config | 18:50 |
| onovy | (from swift meeting logs) | 18:50 |
| *** petertr7_away is now known as petertr7 | 18:52 | |
| lcurtis | hello all....question on the ring and rebalancing...it is generally recommended to add weight slowly...will that mean that a partition gets moved twice as weight is brought up or that the weight just delays the chances of moving a partition | 18:52 |
| notmyname | good morning (again) | 18:54 |
| notmyname | clayg: yeah, the ops feedback so far on the ionice thing is "seems reasonable". onovy's use case is more around cdn/streaming that a lot of the other use cases that are vocally represented in reviews | 18:56 |
| notmyname | as for releases, check https://bugs.launchpad.net/swift for critical bugs. yes, I want to do a release asap. but we've got a few worrisome bugs we shouldn't release without | 18:56 |
| *** diazjf has quit IRC | 18:58 | |
| *** daemontool has joined #openstack-swift | 19:00 | |
| onovy | lcurtis: no | 19:01 |
| onovy | it mean that only few of partion will be moved | 19:01 |
| onovy | and then another few | 19:02 |
| onovy | (in ideal world :) | 19:02 |
| onovy | and hi | 19:02 |
| notmyname | hi onovy | 19:02 |
| onovy | notmyname: hi :) | 19:03 |
| *** dmorita has quit IRC | 19:04 | |
| lcurtis | thanks onovy | 19:04 |
| onovy | lcurtis: np. if you have small cluster and adding capacity, always add weight slowly | 19:05 |
| onovy | add, push to cluster, wait and then add more weight, push to cluster... | 19:05 |
| *** trifon has joined #openstack-swift | 19:08 | |
| notmyname | patch 256306 is one I want to see land before a release (fixes one of those critical bugs listed) | 19:08 |
| patchbot | notmyname: https://review.openstack.org/#/c/256306/ - Fix ClientException handling in Container Sync | 19:08 |
| onovy | notmyname: thanks for review | 19:08 |
| onovy | but after today discussion i think swift-init drop is near :) | 19:08 |
| notmyname | timburke: are you working on a patch for https://bugs.launchpad.net/swift/+bug/1533002 | 19:08 |
| openstack | Launchpad bug 1533002 in OpenStack Object Storage (swift) "object-auditor skips EC fragments" [Critical,Confirmed] | 19:08 |
| *** arnox has quit IRC | 19:11 | |
| notmyname | (the in-person answer to that question was "yes, he's working on a patch today") | 19:12 |
| notmyname | thanks, timburke! | 19:12 |
| *** gyee has quit IRC | 19:13 | |
| *** dmorita has joined #openstack-swift | 19:21 | |
| *** itlinux has joined #openstack-swift | 19:21 | |
| *** yarkot has joined #openstack-swift | 19:22 | |
| *** lpabon has quit IRC | 19:23 | |
| *** lpabon has joined #openstack-swift | 19:23 | |
| *** lpabon has quit IRC | 19:23 | |
| *** yarkot has quit IRC | 19:25 | |
| lcurtis | all...what is genuine best way to tell when a cluster is finished rebalancing | 19:26 |
| lcurtis | something in particular to look for in logs? | 19:26 |
| notmyname | lcurtis: the secret is that a "cluster" doesn't rebalance (or if it does, it never finishes). each individual storage node does, and that's what you track. | 19:27 |
| *** acoles is now known as acoles_ | 19:27 | |
| *** dmorita has quit IRC | 19:27 | |
| notmyname | lcurtis: the replicators will log when they are done. I'll have to remind myself what the log line is | 19:27 |
| onovy | https://github.com/openstack/swift/blob/master/swift/obj/replicator.py#L785 this line | 19:28 |
| onovy | use swift-recon for this | 19:28 |
| notmyname | onovy: thanks :-) | 19:29 |
| onovy | lcurtis: | 19:29 |
| onovy | Oldest completion was 2016-01-12 19:12:33 (16 minutes ago) by sdn-swift-store5.ng.seznam.cz:6000. | 19:29 |
| onovy | Most recent completion was 2016-01-12 19:28:20 (41 seconds ago) by sdn-swift-store4.ko.seznam.cz:6000. | 19:29 |
| onovy | lcurtis: swift-recon :) you can see, that replication part was finished 41s <-> 15 minutes ago on all servers | 19:30 |
| onovy | so if i pushed ring more than 16 minutes ago, my cluster should be balanced | 19:30 |
| onovy | but be carefull, when you push ring, all replicator will restart (=finish in recon) | 19:31 |
| lcurtis | wow thank you very much onovy | 19:32 |
| onovy | you are welcome | 19:32 |
| onovy | AND use swift-dispersion-report | 19:32 |
| onovy | this is really usefull when you adding new devices | 19:33 |
| *** dmorita has joined #openstack-swift | 19:33 | |
| onovy | lcurtis: then you can have this nice graph: http://s14.postimg.org/nmuind7ep/staz_eny_soubor.png | 19:34 |
| *** diazjf has joined #openstack-swift | 19:34 | |
| onovy | 100% = everything balanced | 19:34 |
| notmyname | onovy: what coverage percent are you using in the dispersion report? | 19:35 |
| onovy | notmyname: need to look | 19:35 |
| notmyname | please be 100. please be 100. please be 100 | 19:35 |
| notmyname | ;-) | 19:35 |
| clayg | timburke: hit me up for a review when you think you're close on the ec-frag-audit bug | 19:35 |
| onovy | cont_pct_found = round(data['container']['pct_found'], 2) | 19:36 |
| onovy | cont_retries = data['container']['retries'] | 19:36 |
| onovy | cont_overlapping = data['container']['overlapping'] | 19:36 |
| onovy | obj_pct_found = round(data['object']['pct_found'], 2) | 19:36 |
| onovy | obj_retries = data['object']['retries'] | 19:36 |
| onovy | obj_overlapping = data['object']['overlapping'] | 19:36 |
| onovy | from JSON: swift-dispersion-report -j | 19:36 |
| onovy | and coverage is 1-2 % i think | 19:37 |
| notmyname | ah. some other deployers have found that it's a really good idea to have 100% of the partition space covered with the dispersion report | 19:38 |
| onovy | hmm, do you have any explaination for it? :) | 19:38 |
| openstackgerrit | Peter Chng proposed openstack/swift: Ensure sysmeta is written/updated on (fast) POST https://review.openstack.org/266545 | 19:40 |
| *** janonymous has quit IRC | 19:40 | |
| notmyname | I'm sure there's a curve somewhere of probability of finding issues vs percent coverage you have in a cluster of a given size (drives and partitions). but basically it comes down to relatively small storage cost traded against the probability of finding errors | 19:40 |
| ahale | yeah we run both 100% and default - cos sometimes its really important to really know. and other times who wants to wait that long to find out | 19:40 |
| onovy | ahale: so you have two accounts for swift_mon? | 19:40 |
| onovy | one with 1 % and one with 100 %? | 19:40 |
| ahale | yeah | 19:40 |
| notmyname | would be interesting to do the math and figure out when you cross 50% or 95% or 99% | 19:40 |
| onovy | i don't want to touch/GET 524288 objects from swift every 5 minutes just for graph :) | 19:41 |
| notmyname | heh | 19:41 |
| onovy | but that solutions with two acc looks good | 19:42 |
| onovy | even now i'm geting dispersion only every 15 minutes, because i don't want to bother swift with it | 19:42 |
| notmyname | ahale: how often do you run the 100% one? every day? hour? week? | 19:43 |
| onovy | we are using disp-report only for monitoring of balance progress | 19:43 |
| onovy | is there anything more what can i get from it? | 19:43 |
| ahale | manually notmyname | 19:43 |
| notmyname | once-every-ahale | 19:44 |
| onovy | :] | 19:44 |
| ahale | oh hell no, way more often than an ahale :) | 19:44 |
| notmyname | onovy: balance progress, yes. also you can see background failure issues with it if you have 100% coverage. eg if a drive fails, you'll see that get replicated to handoffs and back as the drive is replaced | 19:45 |
| dimasot | hi all | 19:45 |
| notmyname | hello | 19:45 |
| onovy | notmyname: if drive fails, i can see it in swift-recon in drive-audit as umounted | 19:45 |
| dimasot | clayg: following our yesterday conversation - https://github.com/DmitrySot/data_mover | 19:46 |
| onovy | but exactly it's funny, we didn't have any disk failures yet :) | 19:46 |
| ahale | ...today? :) | 19:46 |
| notmyname | lol | 19:46 |
| onovy | ahale: hey! no. | 19:47 |
| onovy | but it's ok. i'm developer, not admin. not my problem | 19:47 |
| *** dmorita has quit IRC | 19:47 | |
| ahale | :D | 19:48 |
| *** silor has quit IRC | 19:52 | |
| *** dmorita has joined #openstack-swift | 20:03 | |
| Zyric | Good morning | 20:06 |
| notmyname | hello | 20:07 |
| *** dmorita has quit IRC | 20:08 | |
| *** dmorita has joined #openstack-swift | 20:10 | |
| notmyname | sarafraj: welcome! | 20:10 |
| clayg | dimasot: I think it would be useful to udpate the docstring of build_moving_map with a literal sample of the structure of the return value | 20:11 |
| clayg | dimasot: the key in particular seems confusing -> https://github.com/DmitrySot/data_mover/blob/master/data_mover.py#L110 | 20:11 |
| * notmyname needs to go pick up some lunch | 20:11 | |
| *** zhill has joined #openstack-swift | 20:12 | |
| dimasot | clayg: agree I just wanted it to be unique - to allow fast check if a given device shoudl care about a given partition, or it will be done by anothe server/device | 20:19 |
| *** dmorita has quit IRC | 20:19 | |
| dimasot | clayg: so how you suggest to change it? - or you want me to add some comment that discribes how it would look like ? | 20:20 |
| clayg | dimasot: honstly i'm looking at it now (moving_map_dump.txt) and it's not any more clear :P | 20:21 |
| clayg | dimasot: the keys seem to be maybe... <replica>_<partnum> ? | 20:22 |
| clayg | dimasot: nope that's not it - maybe <devid>_<partnum | 20:22 |
| dimasot | <device_id>_<partitions_id> | 20:23 |
| clayg | dimasot: and the value is maybe [<part>, <old_dev_id>, <new_dev_id>] ? | 20:23 |
| dimasot | yep, now I got your comment, the variable names are really not self contein | 20:24 |
| clayg | dimasot: in the common case as I understand it job['nodes'] is always a list of one in this impelemntation? | 20:24 |
| *** john_bar has quit IRC | 20:24 | |
| blmartin | mattoliverau: I found the other bug that was causing rsync to drop existing pivots. The pull request is setup against your repo. | 20:24 |
| dimasot | yep, this is the legacy from replicaton code - should be removed | 20:24 |
| clayg | dimasot: hrmm... it seems like move spawns update, but then - oic if the spawned update fails it appends to the retrie_list and run_once will give it a few more shots | 20:27 |
| clayg | .. but all the work is expected to be handled by the time you return from move | 20:27 |
| clayg | dimasot: yeah these data structure manipulation feels somewhat disorganized :\ | 20:29 |
| openstackgerrit | Ondřej Nový proposed openstack/swift: Allow to change auditor sleep interval in config https://review.openstack.org/266561 | 20:29 |
| openstackgerrit | Jonathan Hinson proposed openstack/swift: Functional tests for if-match with multiple etags https://review.openstack.org/266516 | 20:30 |
| onovy | can someone look to https://review.openstack.org/#/c/251151/ pls? | 20:31 |
| dimasot | I added this retrie_list since I observed that at some cases I have ~10 out of ~40k partitions failed - so it was a path to catchup that failures | 20:33 |
| dimasot | agree it shoul dbe done in more accurate way | 20:33 |
| clayg | dimasot: yeah that's fine, not really germaine | 20:34 |
| clayg | dimasot: I think ignoring the sync with other primaries during rebalance is interesting, I think preseeding the new ring by over-replicating to a staging dir is interesting | 20:36 |
| dimasot | I am working now on a slide deck that represents the performance compare with replication based rebalancing | 20:36 |
| dimasot | so i wil lshare it with you too, when it will be ready | 20:37 |
| clayg | dimasot: once the data_mover finishes how do you go about slaming the directory tree from data_mover into the regular datadir paths and publishing the new rings - what's your process? | 20:37 |
| dimasot | this the part that I am working on - I am doing it by runnign an ansible | 20:38 |
| *** trifon has quit IRC | 20:38 | |
| clayg | dimasot: well... I don't think it's really a fair "comparison" - well... I mean you can pitch it however you want - but I don't htink it's terribly interesting to discuss it in that context | 20:39 |
| dimasot | but I plan to make a function that will run locally and check if there exists "mover" directory on swift device and move its content to "objects" | 20:39 |
| dimasot | regards the unfair comparison - what is another option if I what to add 25% capacity to my cluster? r | 20:40 |
| clayg | dimasot: patience :P | 20:41 |
| clayg | dimasot: i'm kidding - I understand the problem space and use case | 20:42 |
| dimasot | :) | 20:42 |
| ahale | i think there is a huge amount of value in having replication tools that are aware of previous rings :) | 20:42 |
| clayg | ok, that may be all the brain cells I can devote to it today - i'd encourage you to take another whack at the mover data structure to make it more obvious | 20:43 |
| redbo_ | I'm not so sure. We can tell what's not supposed to be on a drive pretty fast. | 20:43 |
| dimasot | ok, will work on it :) thanks for you help | 20:44 |
| clayg | redbo_: one idea that's in dimasot's poc that I hadn't consider is the possible value of not having to connect to or discuss a partition with other primaries while focusing on rebalancing | 20:44 |
| dimasot | will ping you when I will make more progress | 20:44 |
| redbo_ | Sounds a lot like "move handoffs first" mode | 20:45 |
| clayg | redbo_: currently in the python replicator a handoff partition will skip the pre-flight REPLCIATION request (which is nice to avoid hash suffix recalc and listdir io) - but it still calls rsync agains the remote node | 20:45 |
| dimasot | any other comments except the styling/naming? | 20:45 |
| redbo_ | Oh, I see what you're saying. | 20:45 |
| clayg | redbo_: I feel pretty strongly it's a variation on that theme - and continue to have interest in patch 215867 | 20:45 |
| patchbot | clayg: https://review.openstack.org/#/c/215867/ - Make handoffs_first a more useful "mode" | 20:45 |
| redbo_ | So only sync to new locations for that partition for a little while | 20:46 |
| clayg | redbo_: well that's one of the main ideas that I see dimasot exploring - and probably matches what ahale is thinking about | 20:47 |
| clayg | I'm not entirely sure a richer replication protocol wouldn't be able to make it self-evident with some additional context beyond having to pre-calculate it or distribute old versions of the ring | 20:48 |
| clayg | redbo_: maybe it's no so nuts - if rebalance shat out a move_map that you distributed with your rings - i could imagine that replicators might slurp that up and begin chewing on it (kickout out entries for freshly rehomed parts as it processes them) and once the file is consumed it returns to normal sync mode | 20:51 |
| clayg | I don't want a crazy five step rebalance process - but a totally optional self managing handoff_first mode at the cost of distributing a new file is starting to sound like it may be on it's way to a not terrible idea | 20:51 |
| *** daemontool has quit IRC | 20:52 | |
| onovy | good night | 20:53 |
| clayg | redbo_: yeah you might have to talk me out of this one - I think sending out new ring immediately means you could process the move map directly into the datadirs - which I like for avoiding dentry updates | 20:54 |
| *** bsdkurt has quit IRC | 20:54 | |
| redbo_ | I'm not sure you'd want to distribute a new file. The way it is as an outside controller right now, it prevents every drive in the cluster from trying to sync to the new drive. | 20:54 |
| clayg | redbo_: YES! +2 | 20:54 |
| clayg | redbo_: handoffs_first sucks for that reason! | 20:54 |
| ahale | oh yeah , i mean im talking vague unexplored ideas, not really dimascots specific thing.. if anything out-of-replication-band tools to brute force specific things to happen right-now | 20:54 |
| ahale | and sidestep rsycd max connections etc etc | 20:55 |
| clayg | redbo_: I've been bitter lately that replicators "restart" when a new ring comes out (intead of just continuing from the part they're currently syncing but you know - using the right nodes because rings already reload) | 20:55 |
| clayg | redbo_: in move_map situation where replicators keep running you might let them drop into mover mode after the current cycle finishes - giving some amout of distribution | 20:56 |
| clayg | redbo_: that plus max_connections (or getting in the datapath) might allow things to move along a little quicker without eating all the i/o on filling drives | 20:57 |
| clayg | well... new drives might just be hozed regardless | 20:57 |
| *** dslevin has joined #openstack-swift | 21:03 | |
| *** dmorita has joined #openstack-swift | 21:03 | |
| *** zhill has quit IRC | 21:04 | |
| *** dslevin has quit IRC | 21:04 | |
| *** eranrom has joined #openstack-swift | 21:05 | |
| *** MVenesio has quit IRC | 21:07 | |
| *** nadeem has quit IRC | 21:13 | |
| *** eranrom has quit IRC | 21:15 | |
| *** MVenesio has joined #openstack-swift | 21:18 | |
| *** trifon has joined #openstack-swift | 21:19 | |
| *** zhill has joined #openstack-swift | 21:20 | |
| *** dmorita has quit IRC | 21:22 | |
| *** dmorita has joined #openstack-swift | 21:24 | |
| *** diazjf has quit IRC | 21:27 | |
| *** dslevin has joined #openstack-swift | 21:30 | |
| *** esker has joined #openstack-swift | 21:30 | |
| *** esker has quit IRC | 21:31 | |
| *** esker has joined #openstack-swift | 21:31 | |
| *** diazjf has joined #openstack-swift | 21:37 | |
| *** gyee has joined #openstack-swift | 21:44 | |
| *** petertr7 is now known as petertr7_away | 21:47 | |
| *** dmorita has quit IRC | 21:47 | |
| *** blmartin has quit IRC | 21:48 | |
| *** trifon has quit IRC | 21:55 | |
| mattoliverau | morning | 21:56 |
| *** diogogmt has quit IRC | 21:56 | |
| pdardeau | hi mattoliverau | 21:56 |
| mattoliverau | pdardeau: hey man! | 21:56 |
| *** petertr7_away is now known as petertr7 | 21:57 | |
| *** dmorita has joined #openstack-swift | 21:58 | |
| *** dmorita has quit IRC | 21:58 | |
| *** dmorita has joined #openstack-swift | 21:58 | |
| *** bkmz_ has joined #openstack-swift | 22:03 | |
| *** bkmz has quit IRC | 22:06 | |
| *** dmorita has quit IRC | 22:09 | |
| *** zhill has quit IRC | 22:09 | |
| *** dmorita has joined #openstack-swift | 22:10 | |
| *** petertr7 is now known as petertr7_away | 22:11 | |
| dimasot | clayg: some refactoring done, more comments added - so now I hope it will b emore clear | 22:14 |
| dimasot | so you can take a look when you have time for it | 22:15 |
| *** jamielennox|away is now known as jamielennox | 22:17 | |
| *** diogogmt has joined #openstack-swift | 22:20 | |
| *** dmorita has quit IRC | 22:30 | |
| *** zhill has joined #openstack-swift | 22:30 | |
| openstackgerrit | Tim Burke proposed openstack/swift: Make object-auditor storage-policy-aware https://review.openstack.org/266599 | 22:30 |
| timburke | clayg: ^^ | 22:30 |
| *** mingdang1 has joined #openstack-swift | 22:32 | |
| *** dmorita has joined #openstack-swift | 22:32 | |
| *** dmorita has quit IRC | 22:34 | |
| *** dmorita has joined #openstack-swift | 22:42 | |
| *** sarafraj has left #openstack-swift | 22:53 | |
| *** dmorita has quit IRC | 22:53 | |
| *** nadeem has joined #openstack-swift | 22:53 | |
| *** dmorita has joined #openstack-swift | 22:54 | |
| *** dmorita has quit IRC | 22:54 | |
| *** dmorita has joined #openstack-swift | 22:55 | |
| *** km_ has joined #openstack-swift | 22:56 | |
| openstackgerrit | Bill Huber proposed openstack/swift: Re-format the SLO manifest file on new multipart-manifest GET call https://review.openstack.org/263902 | 23:08 |
| *** nadeem has quit IRC | 23:11 | |
| *** diazjf has quit IRC | 23:14 | |
| *** mragupat has quit IRC | 23:15 | |
| *** mrmoje has quit IRC | 23:16 | |
| *** dmorita has quit IRC | 23:19 | |
| *** mingdang1 has quit IRC | 23:22 | |
| *** kei_yama has joined #openstack-swift | 23:29 | |
| *** asettle has joined #openstack-swift | 23:30 | |
| openstackgerrit | Tim Burke proposed openstack/python-swiftclient: Accept gzip-encoded API responses https://review.openstack.org/184956 | 23:40 |
| *** logan2 has joined #openstack-swift | 23:42 | |
| openstackgerrit | Tim Burke proposed openstack/python-swiftclient: Use bulk-delete middleware when available https://review.openstack.org/190887 | 23:42 |
| *** logan2 is now known as logan- | 23:42 | |
| *** ho has joined #openstack-swift | 23:42 | |
| *** ho is now known as Guest55994 | 23:43 | |
| *** dmorita has joined #openstack-swift | 23:44 | |
| *** dmorita has quit IRC | 23:45 | |
| openstackgerrit | Tim Burke proposed openstack/python-swiftclient: Drop testtools from test-requirements.txt https://review.openstack.org/253678 | 23:47 |
| *** dmorita has joined #openstack-swift | 23:48 | |
| *** Guest55994 is now known as ho_ | 23:49 | |
| ho_ | good morning! | 23:53 |
| notmyname | hello | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
