| *** adriant has joined #openstack-swift | 00:19 | |
| clayg | timburke: had you already confirmed that apache mod proxy won't handle passing expect 100 continue responses from the proxied app to the client? | 00:26 |
|---|---|---|
| timburke | clayg: i haven't done any real investigation into apache and 100-continue | 00:27 |
| clayg | timburke: I managed to get the testFileSizeLimit request going through to swift - and I'm getting the 413 error logged - but apache is still taking it's own damn time to respond with 400 | 00:27 |
| clayg | sorry I thought I saw you say something yesterday - thanks | 00:27 |
| timburke | that was pure spitballing | 00:27 |
| clayg | well good fucking guess genius - I think that's more or less the problem ;) | 00:28 |
| timburke | so what all did you do to get it to 413? | 00:28 |
| clayg | just feed it more bytes | 00:28 |
| timburke | ah, sure -- get it to have enough in its buffer to flush and send data to swift. remind me: as part of the 413, does swift send a connection: close and immediately hang up? | 00:30 |
| clayg | but it's not just tests -> https://gist.github.com/clayg/8a0eb56c8984c0cf9b58bb5a409aedf8 | 00:30 |
| timburke | i wonder if apache is trying to be "nice" and keep the data flowing, with the assumption that the client wants to pipeline another request... | 00:31 |
| clayg | yeah idk... swift isn't sending Connection: close - there's a bunch of ML postings and things about apache and expect: 100-continue support - it's possible swift or the request could be changed to make it work? | 00:33 |
| timburke | what do we do if you try to pipeline requests and the first one is too big? does eventlet throw away $content_length bytes, or does it assume that the client never sent any of them? 'cause one of those is still http and the other isn't | 00:36 |
| clayg | this is weird, I took the timeout out again, swift has logged the 413, apache has logged the 400, but the test is still hung waiting on the response? Maybe http_client is being particularly stupid and annoying? | 00:36 |
| timburke | have you tried sending *all* the bytes? | 00:36 |
| clayg | fuck no :p | 00:37 |
| clayg | we should close the connection after we respond 4xx after expect: 100-continue - that's the only correct behavior | 00:37 |
| *** lucasxu has joined #openstack-swift | 00:42 | |
| *** lucasxu has quit IRC | 00:44 | |
| *** itlinux has joined #openstack-swift | 00:45 | |
| *** lucasxu has joined #openstack-swift | 00:45 | |
| *** lucasxu has quit IRC | 00:47 | |
| kota_ | morning | 00:47 |
| kota_ | clayg: thanks for your ack for mahatic's patch. I'll be back to the patch today | 00:48 |
| kota_ | notmyname: that https://review.openstack.org/#/c/439572/ is my priority today | 00:48 |
| patchbot | patch 439572 - swift - Limit number of revert tombstone SSYNC requests | 00:48 |
| notmyname | kota_: thanks! | 00:48 |
| kota_ | notmyname: to tell truth, I saw the comment from clayg briefly and i'm feeling what i can do is just pushing my +2 +A | 00:50 |
| *** lucasxu has joined #openstack-swift | 00:50 | |
| kota_ | but before doing so, I want to look up it again | 00:50 |
| notmyname | :-) | 00:50 |
| *** ukaynar has joined #openstack-swift | 00:51 | |
| kota_ | clayg: for https://review.openstack.org/#/c/439572/, can i update the commit message and give +A for that? I think current commit message is lack to describe what actually the patch want to improve | 00:51 |
| patchbot | patch 439572 - swift - Limit number of revert tombstone SSYNC requests | 00:51 |
| kota_ | but I want to modify only just a commit message even we could accept the number of sync nodes as current | 00:52 |
| *** zhengyin has joined #openstack-swift | 00:54 | |
| notmyname | kota_: I'm ok with you updating the commit message and adding your +2/+A to a patch | 00:54 |
| clayg | No worries. | 00:54 |
| kota_ | notmyname, clayg: thanks, probably i can finish up the patch! | 00:55 |
| *** itlinux has quit IRC | 00:55 | |
| *** itlinux has joined #openstack-swift | 00:56 | |
| *** itlinux has quit IRC | 00:56 | |
| *** lucasxu has quit IRC | 00:56 | |
| *** lucasxu has joined #openstack-swift | 00:58 | |
| *** lucasxu has quit IRC | 01:05 | |
| *** tovin07_ has joined #openstack-swift | 01:05 | |
| openstackgerrit | Lingyong Xu proposed openstack/swift master: Using assertIsNone() instead of assertEqual(None) https://review.openstack.org/471522 | 01:08 |
| openstackgerrit | Thiago da Silva proposed openstack/pyeclib master: Release 1.5.0 https://review.openstack.org/471524 | 01:11 |
| *** gyee has quit IRC | 01:25 | |
| *** zhurong has joined #openstack-swift | 01:28 | |
| clarkb | btw I think the two changes to simplify the swift functional test setup finally merged and applied so if you notice any weirdness in the gate func test let me know. But I tested it and it should be happy | 01:31 |
| notmyname | clarkb: thank you | 01:31 |
| *** itlinux has joined #openstack-swift | 01:39 | |
| *** itlinux has quit IRC | 01:45 | |
| *** chlong has quit IRC | 01:50 | |
| openstackgerrit | Merged openstack/swift master: Using assertIsNone() instead of assertEqual(None) https://review.openstack.org/471522 | 01:55 |
| *** itlinux has joined #openstack-swift | 01:57 | |
| *** itlinux has quit IRC | 02:05 | |
| kota_ | clayg: still? | 02:09 |
| *** mingyu has joined #openstack-swift | 02:14 | |
| kota_ | clayg: I added a thought for https://review.openstack.org/#/c/439572/ | 02:21 |
| patchbot | patch 439572 - swift - Limit number of revert tombstone SSYNC requests | 02:21 |
| kota_ | clayg: I basically agree with your thought we can have less primaries than k to push the tombstone if we assume primary partners can propagate the tombstone asap | 02:22 |
| kota_ | and it looks to work as you expected when *no ec duplication case* | 02:23 |
| kota_ | however, in my final review, I found another corner case, I don't suppose it's in your expectation so I still wanna get your opinion that. | 02:24 |
| kota_ | ec_duplication is much durable (can be readable if a few frags are not available), (probably) we've expected | 02:25 |
| kota_ | than we expected | 02:25 |
| *** zhurong has quit IRC | 02:30 | |
| *** mat128 has quit IRC | 02:36 | |
| *** mat128 has joined #openstack-swift | 02:39 | |
| openstackgerrit | Hieu LE proposed openstack/swift master: OSprofiler in OpenStack Swift https://review.openstack.org/468316 | 02:43 |
| timburke | clayg: interesting... try something like this | 02:44 |
| timburke | curl -H content-length:999999999999 -X PUT -d 1234 http://saio:8090/v1/AUTH_test/test-container/big -X PUT -d 1234 http://saio:8090/v1/AUTH_test/test-container/little | 02:44 |
| timburke | (maybe throw in a -v) | 02:45 |
| *** mat128 has quit IRC | 02:47 | |
| *** mat128 has joined #openstack-swift | 02:48 | |
| *** links has joined #openstack-swift | 02:52 | |
| *** mat128 has quit IRC | 02:52 | |
| *** itlinux has joined #openstack-swift | 03:22 | |
| *** mingyu has quit IRC | 03:27 | |
| *** mingyu has joined #openstack-swift | 03:29 | |
| *** mingyu has quit IRC | 03:33 | |
| *** winggundamth has joined #openstack-swift | 03:34 | |
| *** winggundamth has quit IRC | 03:37 | |
| openstackgerrit | Lingyong Xu proposed openstack/swift master: Using assertIsNone() instead of assertEqual(None) https://review.openstack.org/471570 | 03:37 |
| *** winggundamth has joined #openstack-swift | 03:42 | |
| *** ukaynar has quit IRC | 03:43 | |
| *** psachin has joined #openstack-swift | 03:44 | |
| *** mingyu has joined #openstack-swift | 03:46 | |
| *** mingyu has quit IRC | 03:50 | |
| *** aselius has quit IRC | 03:55 | |
| *** kei_yama has quit IRC | 03:57 | |
| *** zhurong has joined #openstack-swift | 04:00 | |
| *** kei_yama has joined #openstack-swift | 04:01 | |
| *** hoonetorg has quit IRC | 04:29 | |
| *** zhurong has quit IRC | 04:30 | |
| *** adriant has quit IRC | 04:35 | |
| *** kei_yama has quit IRC | 04:35 | |
| mahatic | clayg: thanks for your comments on patch 439572! | 04:35 |
| patchbot | https://review.openstack.org/#/c/439572/ - swift - Limit number of revert tombstone SSYNC requests | 04:35 |
| mahatic | hmm I didn't get a notification | 04:36 |
| *** adriant has joined #openstack-swift | 04:39 | |
| *** zhurong has joined #openstack-swift | 04:40 | |
| *** adriant has quit IRC | 04:40 | |
| *** adriant has joined #openstack-swift | 04:41 | |
| *** hoonetorg has joined #openstack-swift | 04:46 | |
| *** itlinux has quit IRC | 04:49 | |
| *** mingyu has joined #openstack-swift | 04:50 | |
| *** mingyu has quit IRC | 04:55 | |
| *** pcaruana has joined #openstack-swift | 04:56 | |
| *** zhengyin has quit IRC | 05:06 | |
| *** zhengyin has joined #openstack-swift | 05:07 | |
| *** pcaruana has quit IRC | 05:14 | |
| *** bkopilov has joined #openstack-swift | 05:14 | |
| kota_ | mahatic: how do you think of the number of primary nodes to push the tombstone? | 05:15 |
| *** kei_yama has joined #openstack-swift | 05:20 | |
| *** skudlik has joined #openstack-swift | 05:32 | |
| *** cshastri has joined #openstack-swift | 05:43 | |
| mahatic | kota_: thanks for your comments as well. I just read them. I'm still thinking on the EC duplication case | 05:49 |
| *** qwertyco has joined #openstack-swift | 05:50 | |
| zaitcev | mahatic: https://wiki.openstack.org/wiki/Meetings/Swift does not have your meeting, could you please remind me what time it starts? | 05:51 |
| kota_ | mahatic: thanks, exactly, (replicas - k + 1) may be too many primaries... not sure. | 05:52 |
| mahatic | zaitcev: hi, it's on the right column 0700. We've agreed on having a diff chair each time for this meeting (alphabetically). So acoles is going to chair the next one | 05:53 |
| mahatic | zaitcev: please feel free to update the page with topics, if you have any | 05:53 |
| mahatic | zaitcev: next one is on June 14th | 05:54 |
| zaitcev | mahatic: thank you, I see the time in the column header now | 05:54 |
| *** winggundamth has quit IRC | 06:05 | |
| openstackgerrit | Lingyong Xu proposed openstack/swift master: Using assertIsNone() instead of assertEqual(None) https://review.openstack.org/471570 | 06:06 |
| *** jaosorior_away is now known as jaosorior | 06:11 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift master: Follow up for account quota exclude containers patch https://review.openstack.org/471592 | 06:12 |
| *** bkopilov has quit IRC | 06:17 | |
| *** rcernin has joined #openstack-swift | 06:21 | |
| *** cshastri has quit IRC | 06:33 | |
| *** hseipp has joined #openstack-swift | 06:40 | |
| *** saltsa has quit IRC | 06:40 | |
| *** cshastri has joined #openstack-swift | 06:49 | |
| *** mingyu has joined #openstack-swift | 06:52 | |
| *** saltsa has joined #openstack-swift | 06:53 | |
| *** mingyu has quit IRC | 06:54 | |
| *** klrmn has quit IRC | 06:57 | |
| *** afazekas has quit IRC | 07:01 | |
| *** afazekas has joined #openstack-swift | 07:01 | |
| *** mingyu has joined #openstack-swift | 07:01 | |
| *** mingyu has quit IRC | 07:02 | |
| *** mingyu has joined #openstack-swift | 07:02 | |
| *** pcaruana has joined #openstack-swift | 07:03 | |
| *** tesseract has joined #openstack-swift | 07:11 | |
| *** cshastri has quit IRC | 07:17 | |
| *** bkopilov has joined #openstack-swift | 07:20 | |
| *** oshritf has joined #openstack-swift | 07:21 | |
| *** geaaru has joined #openstack-swift | 07:21 | |
| *** cshastri has joined #openstack-swift | 07:34 | |
| *** mingyu has quit IRC | 07:35 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift master: WIP: use config_number method instead of node_id + 1 https://review.openstack.org/471613 | 07:45 |
| *** rcernin has quit IRC | 07:52 | |
| *** rcernin has joined #openstack-swift | 07:52 | |
| *** cbartz has joined #openstack-swift | 07:55 | |
| rledisez | notmyname: nothing you missed, alecuyer just had no time to work on it. it's still on todo list | 07:56 |
| rledisez | clayg: reconstructor concurency, right now I run 1 REVERT process + 1 SYNC process for N devices (N being 1 or 3 depending on the IO load I observe). in config file I set concurrency=1 | 07:58 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift master: Follow up for affinity config per policy https://review.openstack.org/467920 | 08:15 |
| openstackgerrit | Christopher Bartz proposed openstack/swift master: Exclude containers for account quota https://review.openstack.org/414232 | 08:18 |
| acoles | good morning | 08:21 |
| acoles | kota_: hi! did you change patch 467920 or just rebase? | 08:21 |
| patchbot | https://review.openstack.org/#/c/467920/ - swift - Follow up for affinity config per policy | 08:21 |
| kota_ | acoles: hi, just rebased | 08:22 |
| kota_ | acoles: sorry not yet checking your comments | 08:22 |
| acoles | kota_: ok. oh, I just saw your comment on gerrit :) | 08:22 |
| kota_ | acoles: :) | 08:22 |
| acoles | kota_: would be good to get that merged, and not have too many 'follow-up' patches hanging around | 08:23 |
| kota_ | acoles: yup, maybe i will have more time to look at my patches tommorow | 08:23 |
| acoles | kota_: ok | 08:26 |
| *** mingyu has joined #openstack-swift | 08:36 | |
| *** cshastri has quit IRC | 08:40 | |
| *** mingyu has quit IRC | 08:41 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift master: Use config_number method instead of node_id + 1 https://review.openstack.org/471613 | 08:44 |
| *** zhurong has quit IRC | 08:47 | |
| *** kei_yama has quit IRC | 08:49 | |
| *** jamielennox is now known as jamielennox|away | 08:50 | |
| *** cshastri has joined #openstack-swift | 08:54 | |
| *** cshastri has quit IRC | 08:55 | |
| *** cshastri has joined #openstack-swift | 08:56 | |
| *** mvk has joined #openstack-swift | 08:57 | |
| kota_ | https://review.openstack.org/#/c/471613/ is for timburke | 08:58 |
| patchbot | patch 471613 - swift - Use config_number method instead of node_id + 1 | 08:58 |
| kota_ | timburke: it seems like we already have the mapping helper method from node to config number, that is wisdom of our predecessors | 08:59 |
| *** zhurong has joined #openstack-swift | 09:02 | |
| *** mingyu has joined #openstack-swift | 09:04 | |
| *** jamielennox|away is now known as jamielennox | 09:10 | |
| *** qwertyco has quit IRC | 09:14 | |
| cbartz | kota: Thx for your review. Would you like to take a look over patch 454716 ? | 09:29 |
| patchbot | https://review.openstack.org/#/c/454716/ - swift - Allow DLO PUT to prefix-based tempurls | 09:29 |
| openstackgerrit | Lingyong Xu proposed openstack/python-swiftclient master: Drop py34 target in tox.ini https://review.openstack.org/471685 | 09:43 |
| *** zhurong has quit IRC | 09:51 | |
| *** qwertyco has joined #openstack-swift | 09:55 | |
| *** mvk has quit IRC | 09:55 | |
| *** mingyu has quit IRC | 10:00 | |
| *** mingyu has joined #openstack-swift | 10:00 | |
| *** psachin has quit IRC | 10:08 | |
| *** silor has joined #openstack-swift | 10:08 | |
| *** psachin has joined #openstack-swift | 10:09 | |
| *** psachin has quit IRC | 10:09 | |
| *** mvk has joined #openstack-swift | 10:27 | |
| *** tovin07_ has quit IRC | 10:29 | |
| openstackgerrit | Alistair Coles proposed openstack/swift master: Improve domain_remap docs https://review.openstack.org/471712 | 10:35 |
| *** kong has joined #openstack-swift | 10:38 | |
| *** zhengyin has quit IRC | 11:08 | |
| *** zhengyin has joined #openstack-swift | 11:11 | |
| *** psachin has joined #openstack-swift | 11:40 | |
| *** mingyu has quit IRC | 11:43 | |
| *** zhengyin has quit IRC | 11:49 | |
| openstackgerrit | Lingxian Kong proposed openstack/swift master: Write-affinity aware object deletion https://review.openstack.org/470158 | 11:57 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift master: Updated from global requirements https://review.openstack.org/88736 | 12:01 |
| *** chlong has joined #openstack-swift | 12:02 | |
| *** mat128 has joined #openstack-swift | 12:08 | |
| *** tovin07 has joined #openstack-swift | 12:44 | |
| *** klamath has joined #openstack-swift | 13:26 | |
| *** links has quit IRC | 13:30 | |
| *** chlong has quit IRC | 13:43 | |
| *** mingyu has joined #openstack-swift | 13:47 | |
| *** mingyu has quit IRC | 13:51 | |
| *** zhurong has joined #openstack-swift | 13:54 | |
| *** gkadam has joined #openstack-swift | 13:58 | |
| *** gkadam has quit IRC | 14:13 | |
| clayg | rledisez: 1 or 3 depending on observed io sounds pretty sophisticated!? | 14:13 |
| clayg | rledisez: is it always 1 revert per drive? Is the tuning for sync "drives_per_worker"? Or do you just have a high/low mode? | 14:15 |
| *** zhurong has quit IRC | 14:17 | |
| cbartz | timburke: Could you please take a final look over patch 414232 ? Thx. | 14:18 |
| patchbot | https://review.openstack.org/#/c/414232/ - swift - Exclude containers for account quota | 14:18 |
| *** mingyu has joined #openstack-swift | 14:25 | |
| *** jaosorior is now known as jaosorior_away | 14:31 | |
| *** lucasxu has joined #openstack-swift | 14:44 | |
| *** cshastri has quit IRC | 14:45 | |
| *** mingyu has quit IRC | 14:49 | |
| *** mingyu has joined #openstack-swift | 14:52 | |
| openstackgerrit | Alistair Coles proposed openstack/swift master: Update Global EC docs with reference to composite rings https://review.openstack.org/468011 | 14:58 |
| *** mat128 has quit IRC | 15:00 | |
| openstackgerrit | Thiago da Silva proposed openstack/liberasurecode master: re-org of README https://review.openstack.org/471813 | 15:00 |
| *** cebruns has quit IRC | 15:00 | |
| *** psachin has quit IRC | 15:01 | |
| *** cebruns has joined #openstack-swift | 15:02 | |
| *** aselius has joined #openstack-swift | 15:02 | |
| *** mingyu has quit IRC | 15:11 | |
| *** rcernin has quit IRC | 15:11 | |
| *** skudlik has quit IRC | 15:13 | |
| *** ujjain has joined #openstack-swift | 15:20 | |
| *** ujjain has joined #openstack-swift | 15:20 | |
| *** qwertyco has quit IRC | 15:23 | |
| *** chlong has joined #openstack-swift | 15:27 | |
| *** cbartz has quit IRC | 15:29 | |
| *** lucasxu has quit IRC | 15:30 | |
| *** klrmn has joined #openstack-swift | 15:33 | |
| *** itlinux has joined #openstack-swift | 15:34 | |
| *** gyee has joined #openstack-swift | 15:41 | |
| *** itlinux has quit IRC | 15:42 | |
| *** Sukhdev has joined #openstack-swift | 15:58 | |
| *** gkadam has joined #openstack-swift | 16:04 | |
| *** klrmn has quit IRC | 16:08 | |
| timburke | good morning | 16:13 |
| clayg | timburke: so you're going to track down the thing with closing the connection harder/better on 413? | 16:13 |
| *** oshritf has quit IRC | 16:19 | |
| *** hseipp has quit IRC | 16:22 | |
| *** mat128 has joined #openstack-swift | 16:27 | |
| *** lucasxu has joined #openstack-swift | 16:28 | |
| *** mat128 has quit IRC | 16:29 | |
| rledisez | clayg: by "observed io", i mean i checked the state of my cluster and i updated the value manually. nothing fancy :) | 16:33 |
| clayg | phew | 16:33 |
| clayg | and what is the value exactly? like how is it plumbed? does it effect "devices_per_worker" for SYNC only or REVERT too? | 16:34 |
| *** tesseract has quit IRC | 16:34 | |
| clayg | I'm also a little curious what exactly your plumbing looks like for handoff_only revert/sync - i'm guessing it's a little different than upstream? | 16:36 |
| *** chsc has joined #openstack-swift | 16:36 | |
| *** chsc has quit IRC | 16:36 | |
| *** chsc has joined #openstack-swift | 16:36 | |
| *** Sukhdev has quit IRC | 16:42 | |
| *** silor1 has joined #openstack-swift | 16:45 | |
| *** ediardo has quit IRC | 16:45 | |
| *** silor has quit IRC | 16:46 | |
| *** silor1 is now known as silor | 16:46 | |
| *** itlinux has joined #openstack-swift | 16:50 | |
| notmyname | good morning | 16:59 |
| timburke | clayg: http://paste.openstack.org/show/611748/ will fix it... i think it should be a sane way to go about it, too? i can't imagine a situation where we *wouldn't* want to immediately close the connection after sending a 413 | 17:00 |
| timburke | without it, connection's just kinda sitting there; netstat looks like | 17:03 |
| timburke | tcp 0 0 saio:8090 192.168.8.1:63408 ESTABLISHED | 17:03 |
| timburke | i'll wait a while, see if that changes | 17:04 |
| clayg | idk, feels a little buried... in the status *setter* - magic integer value? but i haven't looked for a better place... if you think that's really the best maybe it's fine | 17:06 |
| clayg | isn't there some places in swob where we already have some sort of "finalize" kind of thing that makes sure some important headers are filled in and sane for various responses... idk, like 'content-length' or accept something something? | 17:07 |
| *** geaaru has quit IRC | 17:08 | |
| rledisez | clayg: "value" is the number of device per process. eg: If I have 36 devices and I choose 3, i'll have 12 REVERT and 12 SYNC process, 3 devices for each. (looks like concurrency is working in auditor) | 17:09 |
| rledisez | yes it's different than upstream. I have 2 params for reconstructor: —revert-only and —sync-only. And I pass the list of devices with —device | 17:10 |
| *** mvk has quit IRC | 17:10 | |
| timburke | clayg: _response_iter? maybe... | 17:11 |
| *** klrmn has joined #openstack-swift | 17:11 | |
| clayg | so it's not *quite* like auditor - auditor would say "concurrency = 12" (which should maybe be "workers", but w/e) then if you had 36 devices you get 3 devices per worker. | 17:12 |
| clayg | rledisez: which makes sense to me... but doesn't sound like exactly how you want to be thinking about it? I think i'm ok with either way with a slight preference to whatever we think is easier to understand | 17:13 |
| clayg | er... easier to *explain* or document or whatever | 17:13 |
| acoles | clayg: timburke iirc swob.Response._response_iter does a bunch of 'finalize.-like stuff | 17:14 |
| *** skudlik has joined #openstack-swift | 17:14 | |
| acoles | timburke: oh, you found it :) | 17:14 |
| rledisez | clayg: tbh, i chose the easiest way to implement, not especially the best way. i never though of upstream when i wrote it. it's mainly an init-script stuff, the patch in the reconstructor itself is very short | 17:17 |
| clayg | rledisez: ok well - upstream is going to get support for something *like* this and it would be super great if it landed in a condition where you could consume/migrate to it? So if you wanna have input into the design/implementation now is your chance! | 17:18 |
| rledisez | i feel like defining how much device per process is easier to understand, because you don't have to know how many devices have each of your servers. it will auto adapt the number of process. but I guess everybody know use a config management like puppet & co, so may be not a valid point | 17:18 |
| clayg | worst case scenario is I implement something "close" to what you're doing and you opt to not use it and prefer to keep doing your own thing leading to more bifrication | 17:18 |
| *** ChubYann has joined #openstack-swift | 17:19 | |
| rledisez | yeah, i don't want that neither | 17:19 |
| rledisez | clayg: did i miss a link to etherpad or review about that? | 17:20 |
| *** ChubYann has quit IRC | 17:20 | |
| *** ChubYann has joined #openstack-swift | 17:20 | |
| *** ChubYann has quit IRC | 17:21 | |
| rledisez | what really matters to me is to separate REVERT and SYNC. SYNC jobs are really slow, so I don"t want to slow down my rebalances because of that | 17:21 |
| *** ChubYann has joined #openstack-swift | 17:21 | |
| clayg | no, i haven't finished writing it yet | 17:21 |
| clayg | of course! | 17:21 |
| *** itlinux has quit IRC | 17:24 | |
| clayg | it's not obvious to me I want to run the same # of sync and revert workers either | 17:24 |
| *** ChubYann has quit IRC | 17:25 | |
| *** ChubYann has joined #openstack-swift | 17:26 | |
| *** mat128 has joined #openstack-swift | 17:27 | |
| rledisez | I actually have this possibility in my implementation, but never used it for now | 17:27 |
| rledisez | doesn't mean it's a bad idea, just that my cluster is not in situation it would really make sense | 17:29 |
| *** rcernin has joined #openstack-swift | 17:31 | |
| openstackgerrit | Thiago da Silva proposed openstack/pyeclib master: update to README https://review.openstack.org/471872 | 17:46 |
| *** silor has quit IRC | 17:51 | |
| *** itlinux has joined #openstack-swift | 17:54 | |
| *** hseipp has joined #openstack-swift | 17:58 | |
| *** gkadam has quit IRC | 17:59 | |
| *** itlinux has quit IRC | 18:05 | |
| *** hseipp has quit IRC | 18:07 | |
| *** itlinux has joined #openstack-swift | 18:11 | |
| *** hseipp has joined #openstack-swift | 18:19 | |
| *** hseipp has quit IRC | 18:19 | |
| *** tonanhngo has joined #openstack-swift | 18:20 | |
| *** qwertyco has joined #openstack-swift | 18:33 | |
| *** Sukhdev has joined #openstack-swift | 18:37 | |
| *** qwertyco has quit IRC | 18:47 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Don't rehash primaries in reconstructor handoffs_only mode https://review.openstack.org/428408 | 18:55 |
| *** itlinux has quit IRC | 19:00 | |
| *** itlinux has joined #openstack-swift | 19:00 | |
| clayg | i feel like it should be useful/reasonable to have more than one reconstructor per-disk if you're in a hurry and perhaps less sensitive than rledisez to front end latency - but I can also see running *less* workers than the number of disks. Maybe the option is workers_per_disk = 0.33 or something... | 19:06 |
| clayg | sync_workers_per_disk and handoff_workers_per_disk maybe? | 19:06 |
| *** rcernin has quit IRC | 19:17 | |
| *** mat128 has quit IRC | 19:32 | |
| *** rcernin has joined #openstack-swift | 19:35 | |
| timburke | how much do we care that the @wsgify decorator breaks functional tests? it bit me in https://review.openstack.org/#/c/449394/, it bit Tovin in https://review.openstack.org/#/c/468316/, and it would have bit cbartz in https://review.openstack.org/#/c/414232/ if account_quotas was left of auth | 19:37 |
| *** klrmn has quit IRC | 19:37 | |
| patchbot | No data found for patch 449394 | 19:37 |
| patchbot | No data found for patch 468316 | 19:37 |
| patchbot | No data found for patch 414232 | 19:37 |
| *** klrmn has joined #openstack-swift | 19:38 | |
| timburke | also, when it bites, the failure is not at all obvious: http://logs.openstack.org/94/449394/3/check/gate-swift-dsvm-functional-ubuntu-xenial/cfa35d9/console.html.gz#_2017-05-19_02_59_21_629331 | 19:41 |
| tdasilva | anybody else seeing a lot of gerrit errors? | 19:46 |
| timburke | patchbot certainly did :-) | 19:53 |
| *** Sukhdev has quit IRC | 20:02 | |
| -openstackstatus- NOTICE: The Gerrit service on review.openstack.org is being restarted now to clear some excessive connection counts while we debug the intermittent request failures reported over the past few minutes | 20:06 | |
| notmyname | kota_: timburke: tdasilva: so you think https://review.openstack.org/#/c/414232/ is a good idea? (regardless of code quality, it's good to have that feature in swift?) | 20:15 |
| patchbot | patch 414232 - swift - Exclude containers for account quota | 20:15 |
| timburke | notmyname: i'm still torn | 20:15 |
| tdasilva | notmyname: hard to say, it is such a use-case specific feature...I think cbartz has a use case for this, but maybe we can ask him to discuss in one of the swift meetings ??? | 20:17 |
| tdasilva | notmyname: but it would be hard for me to tell him that no, he should not do that. so I'm ok supporting it, documenting and then it is up to deployers how they want to use it. | 20:18 |
| notmyname | yeah, I can understand that position. and that's why, like timburke, I'm torn. I don't want to tell anyone that their use case is invalid (it's not), and I understand the one given in the commit message | 20:19 |
| notmyname | but that's not the same thing as wanting to commit to maintaining it forever | 20:19 |
| notmyname | account quotas are such a weird thing, given swift's eventual consistency on them, so adding more conditions doesn't seem like the best way to get an "allowed here but not there" sort of use | 20:20 |
| notmyname | if I had a customer asking for something similar, I'd probably recommend tracking quotas outside of swift itself | 20:21 |
| notmyname | anyway, that's why I'm torn on it | 20:21 |
| notmyname | but the longer we don't say "no" on a patch, the harder it is to eventually say no. just because we've all spent that much more time on it (especially the author!) | 20:22 |
| tdasilva | well...but what are you maintaining? I share the feeling of keeping users from shooting themselves in the foot, but you are always going to have some of that. operators are able to create 1-replica systems for example | 20:22 |
| notmyname | sure | 20:22 |
| notmyname | I mean maintaining the logic of the feature in the future | 20:22 |
| notmyname | just like any other feature in swift | 20:22 |
| tdasilva | yeah yeah, that's what i mean. the logic of the feature is really not the hard part to maintain right | 20:23 |
| timburke | (indeed, with composite rings, we now have a perfectly valid use-case for one-replica rings) | 20:23 |
| tdasilva | it's more the: 'man I started using this thing, and I didn't know what I was doing, and now look what i got" | 20:23 |
| notmyname | sure, and I can't (with a straight face) argue against this because of additional complexity when we just landed composite rings? ;-) | 20:23 |
| tdasilva | heh | 20:24 |
| notmyname | but at the same time, we *should* argue against adding features that are best implemented outside of swift itelse | 20:24 |
| tdasilva | notmyname: right, so back to that...honest question: why is it better to implement outside? don't you fall into the same issue? | 20:25 |
| timburke | ...like having account quotas at all? | 20:25 |
| tdasilva | i mean, you are getting the data from swift to implement it, so you have the issue of eventual conssitency all the same | 20:25 |
| notmyname | I don't like having quotas in swift. :-) | 20:25 |
| timburke | my point was more that by satisfying one use-case, we may be laying the groundwork for other interesting features that we haven't even thought of yet | 20:26 |
| notmyname | IMO quotas are much better implemented when they're tightly integrated with auth and billing | 20:26 |
| tdasilva | notmyname: isn't keystone doing quotas? | 20:26 |
| tdasilva | :) | 20:26 |
| notmyname | timburke: definitely! those are the sorts of things that *should* be added to swift. the building blocks so that deployers can build cathedrals | 20:26 |
| notmyname | tdasilva: well, they're talking about it a lot. and yes, I like that better :-) | 20:27 |
| tdasilva | timburke: cough cough.. let's land symlinks ;) | 20:27 |
| timburke | like https://review.openstack.org/#/c/342857, which can get you that scratch space | 20:27 |
| patchbot | patch 342857 - swift - Add defaulter middleware | 20:27 |
| timburke | yes! or symlinks! i need to find time to look at that again... | 20:27 |
| tdasilva | timburke: just came across your 'data protection' gist the other day... | 20:28 |
| timburke | tdasilva: yeah, that'd be *so* much happier with symlink-backed versioning. still gotta think harder about what to do for the expirer... | 20:30 |
| timburke | ...and object names that are close to the length limit... | 20:32 |
| *** pcaruana has quit IRC | 20:53 | |
| *** pcaruana has joined #openstack-swift | 20:53 | |
| mattoliverau | Morning, just an FYI, wife is sick this morning so I'll be at meeting but might be somewhat distracted looking after Lucy at the same time. | 20:57 |
| kota_ | good morning | 20:57 |
| kota_ | mattoliverau: take care of your wife and daughter | 20:58 |
| notmyname | mattoliverau: no worries. we'll just get lucy to pick up a few patches | 20:58 |
| *** m_kazuhiro has joined #openstack-swift | 20:58 | |
| notmyname | swift team meeting time in #openstack-meeting | 20:59 |
| *** klamath_ has joined #openstack-swift | 21:10 | |
| *** klamath has quit IRC | 21:10 | |
| *** pcaruana has quit IRC | 21:16 | |
| *** klamath has joined #openstack-swift | 21:23 | |
| *** seongsoocho_ has joined #openstack-swift | 21:24 | |
| *** cargonza_ has joined #openstack-swift | 21:24 | |
| *** Sukhdev has joined #openstack-swift | 21:25 | |
| *** jarbod___ has joined #openstack-swift | 21:26 | |
| *** mtreinish has quit IRC | 21:29 | |
| *** cargonza has quit IRC | 21:29 | |
| *** seongsoocho has quit IRC | 21:29 | |
| *** torgomatic has quit IRC | 21:29 | |
| *** adriant has quit IRC | 21:29 | |
| *** f0o has quit IRC | 21:29 | |
| *** dja has quit IRC | 21:29 | |
| *** jarbod has quit IRC | 21:29 | |
| *** klamath_ has quit IRC | 21:29 | |
| *** hoonetorg has quit IRC | 21:29 | |
| *** edausq has quit IRC | 21:29 | |
| *** seongsoocho_ is now known as seongsoocho | 21:29 | |
| *** cargonza_ is now known as cargonza | 21:29 | |
| *** torgomatic has joined #openstack-swift | 21:29 | |
| *** ChanServ sets mode: +v torgomatic | 21:29 | |
| *** f0o has joined #openstack-swift | 21:30 | |
| *** mtreinish has joined #openstack-swift | 21:30 | |
| *** dja has joined #openstack-swift | 21:30 | |
| *** hoonetorg has joined #openstack-swift | 21:30 | |
| *** edausq has joined #openstack-swift | 21:30 | |
| *** rcernin has quit IRC | 21:44 | |
| openstackgerrit | Lingxian Kong proposed openstack/swift master: Write-affinity aware object deletion https://review.openstack.org/470158 | 21:44 |
| *** lucasxu has quit IRC | 21:45 | |
| *** catintheroof has joined #openstack-swift | 21:47 | |
| kota_ | clayg: so the point is "once > k fragments are replaced with tombstones it's not possible to service the request [2]." | 22:02 |
| kota_ | you said on the patch | 22:02 |
| clayg | kota_: I recognize that (k + 1) * duplication won't *always* hit enough nodes to *guarantee* you would over-write enough fragments to prevent a stale read - but... realistically... those nodes probably have tombstones already... I still think it's not about tombstone propagation as much as "realistically sure if we remove this tombstone some other primaries have it" | 22:02 |
| clayg | I think I'd need to draw a table to understand "replicas - (k + 1)" vs "(k + 1) * duplication" | 22:03 |
| kota_ | ok, if that is. just m+1 is small enough? | 22:03 |
| clayg | any number > 3 and < replicas is an improvement | 22:03 |
| kota_ | currently (m+1) * dup we are pushing | 22:03 |
| clayg | I don't know what the "perfect" answer is | 22:03 |
| kota_ | yup, me too. | 22:04 |
| clayg | I think the change says "(k + 1) * d" - where k is data frags? | 22:04 |
| clayg | oh... wait | 22:04 |
| kota_ | no m+1 | 22:04 |
| clayg | you're right "(m + 1) * d" | 22:04 |
| kota_ | and m is the number of parity fragments | 22:05 |
| clayg | so yeah, pretty good chance in the duplicated case that could hit well below the number needed to prevent stale read | 22:05 |
| *** m_kazuhiro has quit IRC | 22:05 | |
| clayg | i still like that it's smaller | 22:05 |
| clayg | max(3, replicas / 2) maybe!? | 22:05 |
| clayg | idk | 22:05 |
| acoles | rledisez: I'll try to catch you tomorrow in EU time to discuss that ssync/expired object bug some more | 22:06 |
| kota_ | if *in design* duplicate case could hit stale read well, just (m + 1) would be small enough i felt | 22:06 |
| clayg | rledisez: acoles: I think letting the reconstructor reconstruct is the best thing - make sure the fetch frags guy sends x-backend-replication header and make sure object server responds inspite of the metadata | 22:06 |
| kota_ | if we want to prevent the stale read as possible, we should go to (replicas - k + 1) | 22:07 |
| kota_ | always kill enough number of frags that is not decode-able | 22:07 |
| kota_ | so i didn't find the *intended* desing yesterday | 22:08 |
| acoles | clayg: that would at least save us reasoning about the consistency corner cases that might arise from doing nothing | 22:08 |
| clayg | acoles: i like i! | 22:09 |
| clayg | *it | 22:09 |
| * kota_ is feeling (replicas - k + 1) is too many though | 22:09 | |
| *** klamath has quit IRC | 22:10 | |
| acoles | clayg: would you vote for using x-backend-replication header or a new header e.g. x-ignore-delete-at | 22:10 |
| clayg | kota_: I see what you're saying about (replicas - k + 1) | 22:10 |
| clayg | kota_: I'm ok with that tho | 22:10 |
| clayg | if later we decide for the duplicated case it's too expensive we can drop it down again | 22:11 |
| clayg | it's a big improvement over what's there for ec duplciation = 1 IMHO | 22:11 |
| clayg | let's do it! (?) | 22:11 |
| clayg | acoles: ugh, not x-ignore-delete-at I don't think :\ | 22:11 |
| kota_ | kk | 22:11 |
| clayg | smaller surface area on the api I think | 22:12 |
| clayg | but w/e works | 22:13 |
| *** Sukhdev has quit IRC | 22:20 | |
| kota_ | whooa??? | 22:23 |
| kota_ | once, I tried to change the number of nodes | 22:23 |
| kota_ | the test says # of replicas = 6, # of part nodes = 6, but # of ec_ndata... 10!? | 22:24 |
| acoles | clayg: got iy | 22:24 |
| acoles | it* | 22:24 |
| kota_ | weird statement... | 22:25 |
| kota_ | it looks like misconsistency between testing storage policy and fabricated_ring | 22:27 |
| kota_ | clayg: it seems like I need more time to fix it but I'll push the newer version (to change the number of primaries as (replicas - k + 1) with overwritten on the patch today | 22:28 |
| * kota_ have to prepare breakfast for his kids | 22:29 | |
| *** itlinux has quit IRC | 22:31 | |
| *** chlong has quit IRC | 22:32 | |
| *** itlinux has joined #openstack-swift | 22:42 | |
| *** skudlik has quit IRC | 22:45 | |
| *** catintheroof has quit IRC | 22:46 | |
| *** vint_bra has quit IRC | 22:47 | |
| *** itlinux has quit IRC | 22:54 | |
| clayg | timburke: where's that follow up with config number cleanup stuff - which i know have context for :) | 22:59 |
| timburke | clayg: it's to do with kota_'s comments at https://review.openstack.org/#/c/302494/9/test/probe/test_db_replicator.py@89 | 23:00 |
| patchbot | patch 302494 - swift - Sync metadata in 'rsync_then_merge' in db_replicator | 23:00 |
| timburke | iirc | 23:01 |
| clayg | yeah i know - that's the context I have now have - don't have the patch tho... | 23:01 |
| timburke | oh, gotcha. https://review.openstack.org/#/c/471613/ | 23:01 |
| patchbot | patch 471613 - swift - Use config_number method instead of node_id + 1 | 23:01 |
| *** openstack has joined #openstack-swift | 23:14 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Cleanup db replicator probetest https://review.openstack.org/471957 | 23:22 |
| *** kei_yama has joined #openstack-swift | 23:29 | |
| *** chsc has quit IRC | 23:39 | |
| clayg | notmyname: patch 470158 is shaping up pretty good - I'm not even sure it's ready already - need to look at it again real soon (tm) | 23:46 |
| patchbot | https://review.openstack.org/#/c/470158/ - swift - Write-affinity aware object deletion | 23:46 |
| notmyname | clayg: yay! | 23:46 |
| clayg | does anyone know the author's nick? | 23:47 |
| notmyname | I do not | 23:48 |
| *** tonanhngo has quit IRC | 23:54 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Limit number of revert tombstone SSYNC requests https://review.openstack.org/439572 | 23:58 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!