| klrmn1 | notmyname: is hot for the random module today | 00:00 |
|---|---|---|
| clayg | notmyname: it's more like while True: if client_is_slow(): conn -= 1 elif replication_is_slow(): conn += 1 else: sleep(1) | 00:00 |
| notmyname | yet again, clayg patches my code by replacing it with better code ;-) | 00:01 |
| *** haomaiwa_ has quit IRC | 00:01 | |
| clayg | lol | 00:01 |
| *** haomaiwang has joined #openstack-swift | 00:01 | |
| clayg | notmyname: dfg_ was talking about some crazyness where you might try adding capacity and pushing rings *just* to object servers (letting the proxy push stuff to the old primary - now handoff) until your new capacity looks full-ish then pushing the rings out to the proxies too. | 00:03 |
| notmyname | that's interesting | 00:03 |
| clayg | unrelated to rsync max connections - but it's a general idea of how you might let a drive get hammered during capacity adjustment w/o killing on % of client requests quite so bad | 00:03 |
| notmyname | sounds similar to what we were talking about in austin | 00:03 |
| *** ho has joined #openstack-swift | 00:04 | |
| *** hrou has joined #openstack-swift | 00:04 | |
| notmyname | somehow being able to ingest a node and get data to it without it actually being used by the proxy | 00:04 |
| clayg | notmyname: I agree - sounds crazy - but it might could work - I keep thinking there's a few things we want to do with that have to with cluster having a copy of the "old" ring (part-power adjustment has a similar need) | 00:04 |
| clayg | notmyname: I must have missed that side bar - but yeah that's exactly the idea | 00:04 |
| clayg | totally crazy stuff | 00:05 |
| notmyname | it was phrased as disabling/enabling a drive | 00:05 |
| clayg | meh, not really how I'm thinking about it - there's a few nice things of writing to the old location of the part instead of just some random handoff because one of the current primaries is "disabled" | 00:06 |
| notmyname | sounds like it would be dependent on (1) the placement algorithm doing a good job wrt keeping handoffs relatively stable and not moving too many primaries (2) really really good monitoring about how well replication is running in the cluster and (3) sharp ops team that has things tuned well and the patience to wait for replication cycles before rebalancing rings | 00:06 |
| notmyname | yeah, I agree about wanting something better than a simple "drive is disabled try again" code path. the old location is better than that | 00:08 |
| notmyname | hmm..it essentially makes the ring rollout into a process that takes a lot longer (days instead of minutes) | 00:08 |
| clayg | notmyname: idk, maybe - it's an interesting thought exercise | 00:09 |
| clayg | but anyway | 00:09 |
| openstackgerrit | Peter Chng proposed openstack/swift: Append crypto-meta to user-meta https://review.openstack.org/245885 | 00:09 |
| clayg | serverascode: the reason this is all relevant is that running with a high max connections won't really bite you until you add some fresh disks - then client request that hit that drive while replication is accidently the whole iops is gunna make some requests sad | 00:10 |
| clayg | serverascode: if that's a "quickish" sort of pain because you aren't adding that much capacity it's entirely possilbe no one will care | 00:11 |
| clayg | but as the cluster gets bigger (multiple PB) you'll be back in channel asking what notmyname and I were talking about :P | 00:11 |
| serverascode | clayg: ok, thanks again | 00:12 |
| clayg | there's less risk going with smaller max connections - except that when you *do* add capacity you'll want to have some inside into your replication logs or recon data to understand just how long it's going to take if only 4-10 parts can be moving onto a drive at a time :\ | 00:12 |
| ho | good morning! | 00:15 |
| notmyname | hello ho | 00:15 |
| ho | notmyname: hello :-) | 00:15 |
| mattoliverau | ho: Morning | 00:16 |
| ho | mattoliverau: morning! welcome back :-) | 00:16 |
| *** joeljwright has quit IRC | 00:17 | |
| gmmaha | peluse: were you able to get past the issue? Went back and checked my notes and i dont see to have run into that | 00:17 |
| mattoliverau | ho: thanks :) | 00:19 |
| mattoliverau | hey gmmaha | 00:20 |
| peluse | gmmaha: no, I'm convinced this whole python pbr setuptools easy-install.pth non-sense was all create to drive me nutso :) | 00:23 |
| peluse | gmmaha: but I'm off to the grocery store so will dork with it later.... | 00:24 |
| openstackgerrit | Merged openstack/swift: Missing log_name option added https://review.openstack.org/245444 | 00:26 |
| *** jrichli has joined #openstack-swift | 00:28 | |
| notmyname | torgomatic_: seems your 503 etag bug only applies to replicas | 00:28 |
| torgomatic_ | notmyname: true | 00:28 |
| torgomatic_ | I think the bug says as much | 00:28 |
| notmyname | ah, yes. in the comments | 00:29 |
| notmyname | I, like, have to *scroll* to find that | 00:29 |
| notmyname | torgomatic_: anyway, thanks for working on that | 00:29 |
| torgomatic_ | notmyname: sure. 5xx errors are pretty important, after all | 00:30 |
| *** jerrygb has quit IRC | 00:40 | |
| *** jerrygb has joined #openstack-swift | 00:42 | |
| *** jerrygb has quit IRC | 00:42 | |
| *** siva_krishnan has joined #openstack-swift | 00:46 | |
| *** m_kazuhiro has joined #openstack-swift | 00:51 | |
| *** haomaiwang has quit IRC | 01:01 | |
| *** haomaiwang has joined #openstack-swift | 01:01 | |
| *** wbhuber has quit IRC | 01:01 | |
| *** wbhuber_ has quit IRC | 01:10 | |
| *** wbhuber_ has joined #openstack-swift | 01:11 | |
| *** siva_krishnan has quit IRC | 01:11 | |
| *** wbhuber_ has quit IRC | 01:15 | |
| *** dmorita has joined #openstack-swift | 01:30 | |
| openstackgerrit | Peter Chng proposed openstack/swift: Append crypto-meta to user-meta https://review.openstack.org/245885 | 01:38 |
| *** aix has quit IRC | 01:46 | |
| lifeless | peluse: whats up? | 01:48 |
| *** haomaiwang has quit IRC | 01:49 | |
| *** gyee has quit IRC | 01:55 | |
| peluse | lifeless: just having issues getting everything updated (setuptools, pbr, pip) and then getting keystone to install. complains about something related that I don't "get". Have to run now, if you're around tomorrow and think you can help I'll hit ya up! | 01:58 |
| lifeless | peluse: this may help - https://rbtcollins.wordpress.com/2015/07/12/bootstrapping-developer-environments-for-openstack/ | 02:01 |
| lifeless | peluse: but sure, ping me tomorrow | 02:02 |
| peluse | lifeless: thanks, will take a look! | 02:06 |
| *** alejandrito has joined #openstack-swift | 02:08 | |
| *** alejandrito has quit IRC | 02:08 | |
| *** alejandrito has joined #openstack-swift | 02:08 | |
| *** klrmn1 has quit IRC | 02:12 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift: Updated from global requirements https://review.openstack.org/88736 | 02:16 |
| *** alejandrito has quit IRC | 02:21 | |
| *** thucdx has joined #openstack-swift | 02:25 | |
| *** thucdx has left #openstack-swift | 02:28 | |
| *** haomaiwang has joined #openstack-swift | 02:31 | |
| *** alejandrito has joined #openstack-swift | 02:33 | |
| *** haomaiwang has quit IRC | 02:36 | |
| *** haomaiwa_ has joined #openstack-swift | 02:36 | |
| *** zhill has quit IRC | 02:48 | |
| *** sanchitmalhotra has joined #openstack-swift | 02:53 | |
| *** venkat_p has joined #openstack-swift | 02:59 | |
| *** SkyRocknRoll has joined #openstack-swift | 02:59 | |
| *** nadeem has joined #openstack-swift | 03:01 | |
| *** bapalm has quit IRC | 03:06 | |
| openstackgerrit | venkatesh proposed openstack/swift: remove unused parameters in the method https://review.openstack.org/243401 | 03:12 |
| jrichli | timburke: dont suppose you are around? some slorange stuff is not mixing well with crypto. i have some questions. | 03:12 |
| *** m_kazuhiro has quit IRC | 03:17 | |
| *** balajir has quit IRC | 03:17 | |
| *** m_kazuhiro has joined #openstack-swift | 03:18 | |
| *** AbyssOne has quit IRC | 03:18 | |
| *** balajir has joined #openstack-swift | 03:18 | |
| *** mwheckmann has joined #openstack-swift | 03:24 | |
| *** janonymous has joined #openstack-swift | 03:25 | |
| *** AbyssOne has joined #openstack-swift | 03:26 | |
| *** janonymous has quit IRC | 03:34 | |
| *** wbhuber has joined #openstack-swift | 03:45 | |
| *** tsg has joined #openstack-swift | 03:51 | |
| *** klrmn1 has joined #openstack-swift | 03:57 | |
| *** alejandrito has quit IRC | 04:16 | |
| *** trifon has joined #openstack-swift | 04:20 | |
| *** sanchitmalhotra1 has joined #openstack-swift | 04:29 | |
| *** sanchitmalhotra has quit IRC | 04:31 | |
| *** mahatic has joined #openstack-swift | 04:37 | |
| *** sanchitmalhotra has joined #openstack-swift | 04:39 | |
| *** ppai has joined #openstack-swift | 04:40 | |
| *** sanchitmalhotra1 has quit IRC | 04:41 | |
| *** jrichli has quit IRC | 04:45 | |
| *** trifon has quit IRC | 04:52 | |
| *** sanchitmalhotra1 has joined #openstack-swift | 04:52 | |
| *** nottrobin_ has joined #openstack-swift | 04:53 | |
| *** nikhil|afk has joined #openstack-swift | 04:54 | |
| *** sanchitmalhotra has quit IRC | 04:54 | |
| *** matt___ has joined #openstack-swift | 04:55 | |
| *** baffle has joined #openstack-swift | 04:55 | |
| *** redbo_ has joined #openstack-swift | 04:56 | |
| *** MooingLe1ur has joined #openstack-swift | 04:56 | |
| *** timur has joined #openstack-swift | 04:56 | |
| *** haypo_ has joined #openstack-swift | 04:56 | |
| *** zacksh has joined #openstack-swift | 04:57 | |
| *** hrou has quit IRC | 04:57 | |
| *** mwheckmann has quit IRC | 04:58 | |
| *** SkyRocknRoll_ has joined #openstack-swift | 04:59 | |
| *** CrackerJackMack has quit IRC | 05:00 | |
| *** charz has quit IRC | 05:00 | |
| *** serverascode has quit IRC | 05:00 | |
| *** xnox has quit IRC | 05:00 | |
| *** swifterdarrell has quit IRC | 05:00 | |
| *** nottrobin has quit IRC | 05:00 | |
| *** baffle_ has quit IRC | 05:00 | |
| *** redbo has quit IRC | 05:00 | |
| *** zacksh_ has quit IRC | 05:00 | |
| *** nikhil has quit IRC | 05:00 | |
| *** MooingLemur has quit IRC | 05:00 | |
| *** jamielennox has quit IRC | 05:00 | |
| *** jroll has quit IRC | 05:00 | |
| *** wasmum has quit IRC | 05:00 | |
| *** timur_ has quit IRC | 05:00 | |
| *** peluse has quit IRC | 05:00 | |
| *** notmyname has quit IRC | 05:00 | |
| *** mattoliverau has quit IRC | 05:00 | |
| *** haypo has quit IRC | 05:00 | |
| *** jroll has joined #openstack-swift | 05:00 | |
| *** CrackerJackMack has joined #openstack-swift | 05:00 | |
| *** MooingLe1ur is now known as MooingLemur | 05:00 | |
| *** wasmum- has joined #openstack-swift | 05:00 | |
| *** jamielennox has joined #openstack-swift | 05:01 | |
| *** jerrygb has joined #openstack-swift | 05:01 | |
| *** zaitcev has quit IRC | 05:01 | |
| *** peluse has joined #openstack-swift | 05:01 | |
| *** jroll has quit IRC | 05:02 | |
| *** jroll has joined #openstack-swift | 05:02 | |
| *** jroll is now known as Guest89040 | 05:02 | |
| *** xnox has joined #openstack-swift | 05:02 | |
| *** nottrobin_ is now known as nottrobin | 05:02 | |
| *** bill_az has quit IRC | 05:02 | |
| *** swifterdarrell has joined #openstack-swift | 05:02 | |
| *** ChanServ sets mode: +v swifterdarrell | 05:02 | |
| *** notmyname has joined #openstack-swift | 05:03 | |
| *** ChanServ sets mode: +v notmyname | 05:03 | |
| *** charz has joined #openstack-swift | 05:03 | |
| *** jamielennox is now known as jamielennox|away | 05:06 | |
| *** serverascode has joined #openstack-swift | 05:07 | |
| *** sanchitmalhotra1 has quit IRC | 05:10 | |
| *** sanchitmalhotra has joined #openstack-swift | 05:13 | |
| *** sanchitmalhotra1 has joined #openstack-swift | 05:16 | |
| openstackgerrit | Merged openstack/swift: New swauth URL https://review.openstack.org/245578 | 05:16 |
| *** wbhuber has quit IRC | 05:17 | |
| *** sanchitmalhotra has quit IRC | 05:18 | |
| *** onder has quit IRC | 05:25 | |
| openstackgerrit | Daisuke Morita proposed openstack/swift-specs: Changing Policies spec https://review.openstack.org/168761 | 05:26 |
| *** bill_az has joined #openstack-swift | 05:28 | |
| *** onder has joined #openstack-swift | 05:30 | |
| *** sanchitmalhotra has joined #openstack-swift | 05:34 | |
| *** wbhuber has joined #openstack-swift | 05:35 | |
| *** sanchitmalhotra1 has quit IRC | 05:37 | |
| *** jerrygb has quit IRC | 05:38 | |
| *** klrmn1 has quit IRC | 05:49 | |
| *** wbhuber has quit IRC | 05:54 | |
| *** nadeem has quit IRC | 05:56 | |
| *** trifon has joined #openstack-swift | 05:59 | |
| *** matt___ is now known as mattoliverau | 05:59 | |
| *** ChanServ sets mode: +v mattoliverau | 06:00 | |
| *** NM has joined #openstack-swift | 06:01 | |
| mahatic | mattoliverau: hello, welcome back! | 06:06 |
| mattoliverau | mahatic: thanks :) | 06:07 |
| openstackgerrit | Merged openstack/swift: Remove references to now-missing havana docs https://review.openstack.org/245326 | 06:11 |
| *** rcernin has joined #openstack-swift | 06:22 | |
| *** sanchitmalhotra1 has joined #openstack-swift | 06:28 | |
| *** sanchitmalhotra has quit IRC | 06:31 | |
| *** jerrygb has joined #openstack-swift | 06:39 | |
| *** sanchitmalhotra has joined #openstack-swift | 06:45 | |
| *** jerrygb has quit IRC | 06:45 | |
| *** sanchitmalhotra1 has quit IRC | 06:47 | |
| *** bill_az has quit IRC | 06:47 | |
| *** nakagawamsa has joined #openstack-swift | 06:48 | |
| *** sileht has joined #openstack-swift | 06:49 | |
| *** nakagawamsa has quit IRC | 06:49 | |
| *** SkyRocknRoll_ has quit IRC | 06:54 | |
| *** kei_yama has joined #openstack-swift | 06:56 | |
| *** dhellmann has quit IRC | 07:01 | |
| *** dhellmann has joined #openstack-swift | 07:02 | |
| *** SkyRocknRoll has quit IRC | 07:05 | |
| *** sanchitmalhotra has quit IRC | 07:07 | |
| openstackgerrit | Merged openstack/swift: Fix 503 on zero-byte replicated PUT with incorrect Etag https://review.openstack.org/246010 | 07:08 |
| *** tsg has quit IRC | 07:08 | |
| *** SkyRocknRoll_ has joined #openstack-swift | 07:11 | |
| *** sssagar has joined #openstack-swift | 07:13 | |
| *** sssagar has left #openstack-swift | 07:14 | |
| *** SkyRocknRoll has joined #openstack-swift | 07:19 | |
| *** sanchitmalhotra has joined #openstack-swift | 07:19 | |
| *** takashi has joined #openstack-swift | 07:21 | |
| *** SkyRocknRoll has quit IRC | 07:33 | |
| *** zhill has joined #openstack-swift | 07:43 | |
| *** zhill has quit IRC | 07:48 | |
| *** arnox has joined #openstack-swift | 08:00 | |
| *** SkyRocknRoll has joined #openstack-swift | 08:02 | |
| *** JelleB is now known as a1|away | 08:14 | |
| *** a1|away is now known as JelleB | 08:14 | |
| *** ho has quit IRC | 08:15 | |
| *** rledisez has joined #openstack-swift | 08:16 | |
| *** sanchitmalhotra1 has joined #openstack-swift | 08:32 | |
| *** sanchitmalhotra has quit IRC | 08:34 | |
| *** hseipp has joined #openstack-swift | 08:36 | |
| *** jerrygb has joined #openstack-swift | 08:40 | |
| *** geaaru has joined #openstack-swift | 08:43 | |
| *** jerrygb has quit IRC | 08:46 | |
| *** jordanP has joined #openstack-swift | 08:51 | |
| *** venkat_p has quit IRC | 08:55 | |
| *** sssagar has joined #openstack-swift | 08:57 | |
| *** jmccarthy has quit IRC | 08:57 | |
| *** jmccarthy has joined #openstack-swift | 08:58 | |
| *** sssagar has left #openstack-swift | 08:59 | |
| *** kei_yama has quit IRC | 09:02 | |
| *** kei_yama has joined #openstack-swift | 09:05 | |
| *** SkyRocknRoll has quit IRC | 09:16 | |
| *** ekarlso has quit IRC | 09:22 | |
| *** dmorita has quit IRC | 09:23 | |
| *** zhill has joined #openstack-swift | 09:25 | |
| *** zhill has quit IRC | 09:26 | |
| *** openstackgerrit has quit IRC | 09:31 | |
| *** openstackgerrit has joined #openstack-swift | 09:31 | |
| *** ekarlso has joined #openstack-swift | 09:37 | |
| *** ekarlso has quit IRC | 09:38 | |
| *** ekarlso has joined #openstack-swift | 09:38 | |
| *** esker has joined #openstack-swift | 09:49 | |
| *** joeljwright has joined #openstack-swift | 09:53 | |
| *** ChanServ sets mode: +v joeljwright | 09:53 | |
| *** esker has quit IRC | 09:54 | |
| *** esker has joined #openstack-swift | 09:55 | |
| *** esker has quit IRC | 09:55 | |
| *** ekarlso has quit IRC | 09:57 | |
| *** haypo_ is now known as haypo | 10:02 | |
| *** kei_yama has quit IRC | 10:10 | |
| *** kei_yama has joined #openstack-swift | 10:11 | |
| *** daemontool has joined #openstack-swift | 10:12 | |
| *** daemontool has quit IRC | 10:14 | |
| *** daemontool has joined #openstack-swift | 10:15 | |
| *** haomaiwa_ has quit IRC | 10:31 | |
| *** openstackgerrit has quit IRC | 10:31 | |
| *** openstackgerrit has joined #openstack-swift | 10:31 | |
| *** haomaiwang has joined #openstack-swift | 10:32 | |
| *** sanchitmalhotra1 has quit IRC | 10:32 | |
| *** ekarlso has joined #openstack-swift | 10:36 | |
| *** ekarlso has quit IRC | 10:39 | |
| *** jerrygb has joined #openstack-swift | 10:43 | |
| *** m_kazuhiro has quit IRC | 10:46 | |
| *** Leigh507 has joined #openstack-swift | 10:47 | |
| *** haomaiwang has quit IRC | 10:47 | |
| *** jerrygb has quit IRC | 10:47 | |
| *** haomaiwang has joined #openstack-swift | 10:49 | |
| *** haomaiwang has quit IRC | 10:49 | |
| *** haomaiwa_ has joined #openstack-swift | 10:49 | |
| Leigh507 | Hi all, Is this the right place to ask for some help with Icehouse swift? | 10:49 |
| *** daemontool has quit IRC | 11:11 | |
| *** daemontool has joined #openstack-swift | 11:11 | |
| *** daemontool has quit IRC | 11:23 | |
| *** daemontool has joined #openstack-swift | 11:24 | |
| *** mahatic_ has joined #openstack-swift | 11:25 | |
| *** daemontool has quit IRC | 11:25 | |
| *** xnox has left #openstack-swift | 11:26 | |
| *** daemontool has joined #openstack-swift | 11:26 | |
| *** Guest53741 has quit IRC | 11:27 | |
| *** mahatic has quit IRC | 11:28 | |
| *** kei_yama has quit IRC | 11:44 | |
| onovy | Leigh507, hi, don't ask to ask, just ask :) | 11:45 |
| onovy | there are (almost) all swift (core) devs here | 11:45 |
| Leigh507 | :) | 11:47 |
| *** NM has quit IRC | 11:47 | |
| Leigh507 | Ok. So I am trying to set up Swift as an object store authenticating with keystone. | 11:48 |
| Leigh507 | When running a swift stat I get a 401. | 11:48 |
| Leigh507 | The user has a correct token | 11:48 |
| Leigh507 | And the Swift URL returned is valid for the tennant. | 11:49 |
| Leigh507 | IE the service catalog URL has the tennant ID appended. | 11:49 |
| Leigh507 | I think my issue may reside in ('Www-Authenticate', 'Keystone uri=http://127.0.0.1:35357'), | 11:49 |
| Leigh507 | My keystone is on a remote box | 11:49 |
| Leigh507 | So I am not sure why localhost is being returned. | 11:50 |
| Leigh507 | From what I have seen this is returned when you do not set the auth uri in the proxy conf | 11:53 |
| Leigh507 | However this has been set. | 11:53 |
| Leigh507 | Onvoy, Also when I curled the swift URL retruned from the service catalog with the auth token provided by keystone I get: | 11:57 |
| Leigh507 | < HTTP/1.1 401 Unauthorized HTTP/1.1 401 Unauthorized < Content-Length: 0 Content-Length: 0 < Content-Type: text/plain Content-Type: text/plain < Www-Authenticate: Keystone uri=http://127.0.0.1:35357 Www-Authenticate: Keystone uri=http://127.0.0.1:35357 < X-Trans-Id: tx68e4a70cdf1c40b4acb25-00564b159f X-Trans-Id: tx68e4a70cdf1c40b4acb25-00564b159f < Date: Tue, 17 Nov 2015 11:55:15 GMT Date: Tue, 17 Nov 2015 11:55:15 GMT | 11:57 |
| *** venkat_p has joined #openstack-swift | 12:02 | |
| venkat_p | How to integrate keystone with SAIO? | 12:03 |
| *** pushkarajthorat has joined #openstack-swift | 12:04 | |
| *** venkat_p has quit IRC | 12:11 | |
| *** links has joined #openstack-swift | 12:12 | |
| *** links has quit IRC | 12:12 | |
| *** haomaiwa_ has quit IRC | 12:14 | |
| *** NM has joined #openstack-swift | 12:28 | |
| *** jerrygb has joined #openstack-swift | 12:31 | |
| *** Guest89040 is now known as jroll | 12:36 | |
| *** alejandrito has joined #openstack-swift | 12:42 | |
| *** SkyRocknRoll_ has quit IRC | 12:45 | |
| *** pushkarajthorat has quit IRC | 13:03 | |
| *** ppai has quit IRC | 13:06 | |
| *** tsg has joined #openstack-swift | 13:41 | |
| *** haomaiwang has joined #openstack-swift | 13:47 | |
| *** bill_az has joined #openstack-swift | 13:52 | |
| *** haomaiwang has quit IRC | 13:52 | |
| peluse | lifeless: FYI my issue was with however I upgraded pbr, something didn't work right and keystone dependencies wouldn't install. I used easy_install to redo pbr and that fixed whatever was wrong... thanks though! | 13:56 |
| *** ekarlso has joined #openstack-swift | 14:01 | |
| *** joeljwright has quit IRC | 14:09 | |
| *** joeljwright has joined #openstack-swift | 14:10 | |
| *** ChanServ sets mode: +v joeljwright | 14:10 | |
| *** tsg has quit IRC | 14:19 | |
| *** wbhuber_ has joined #openstack-swift | 14:25 | |
| *** wbhuber__ has joined #openstack-swift | 14:26 | |
| *** wbhuber__ is now known as wbhuber | 14:27 | |
| *** proteusguy_ has quit IRC | 14:29 | |
| *** wbhuber_ has quit IRC | 14:29 | |
| *** pushkarajthorat has joined #openstack-swift | 14:33 | |
| *** dustins has joined #openstack-swift | 14:33 | |
| *** hrou has joined #openstack-swift | 14:33 | |
| *** breitz has quit IRC | 14:36 | |
| *** breitz has joined #openstack-swift | 14:36 | |
| *** haomaiwang has joined #openstack-swift | 14:39 | |
| *** ekarlso has quit IRC | 14:40 | |
| *** proteusguy_ has joined #openstack-swift | 14:46 | |
| *** wbhuber_ has joined #openstack-swift | 14:47 | |
| *** rvasilets__ has joined #openstack-swift | 14:47 | |
| *** wbhuber has quit IRC | 14:50 | |
| *** wbhuber_ is now known as wbhuber | 14:50 | |
| *** diazjf has joined #openstack-swift | 15:00 | |
| *** chsc has joined #openstack-swift | 15:01 | |
| *** trifon has quit IRC | 15:03 | |
| *** blmartin has joined #openstack-swift | 15:06 | |
| openstackgerrit | Peter Chng proposed openstack/swift: Append crypto-meta to user-meta https://review.openstack.org/245885 | 15:06 |
| *** mac_ified has joined #openstack-swift | 15:07 | |
| *** tongli has joined #openstack-swift | 15:11 | |
| *** siva_krishnan has joined #openstack-swift | 15:15 | |
| *** dustins has quit IRC | 15:22 | |
| *** dustins has joined #openstack-swift | 15:27 | |
| *** tsg has joined #openstack-swift | 15:30 | |
| *** jrichli has joined #openstack-swift | 15:33 | |
| *** daemontool has quit IRC | 15:34 | |
| *** siva_krishnan has quit IRC | 15:35 | |
| *** siva_krishnan has joined #openstack-swift | 15:35 | |
| *** pushkarajthorat has quit IRC | 15:42 | |
| *** bill_az has quit IRC | 15:44 | |
| *** minwoob has joined #openstack-swift | 15:48 | |
| *** proteusguy__ has joined #openstack-swift | 15:49 | |
| gmmaha | :wq | 15:51 |
| *** proteusguy_ has quit IRC | 15:51 | |
| glange | vi user ^^ | 15:51 |
| *** dustins has quit IRC | 15:52 | |
| gmmaha | :D | 15:52 |
| gmmaha | sorry wrong focus window. :( Need to stop windwos from switching focux when i get a msg | 15:52 |
| glange | everybody has done that -- no worries :) | 15:52 |
| *** dustins has joined #openstack-swift | 15:52 | |
| *** daemontool has joined #openstack-swift | 15:57 | |
| *** daemontool has quit IRC | 15:59 | |
| *** daemontool has joined #openstack-swift | 15:59 | |
| *** rcernin has quit IRC | 16:04 | |
| *** mzhou has joined #openstack-swift | 16:10 | |
| timburke | good morning! jrichli, still need help? | 16:27 |
| *** jlhinson has joined #openstack-swift | 16:31 | |
| *** mwheckmann has joined #openstack-swift | 16:35 | |
| jrichli | timburke: good morning! Crypto support for sending etag in a footer is now having issues with the change made to line 454 in https://github.com/openstack/swift/blame/feature/crypto/swift/common/request_helpers.py | 16:37 |
| jrichli | timburke: that conditional check used to also require "and first_byte is None and last_byte is None". I suppose that was not true in our crypto version of things before, so this didn't raise before. | 16:39 |
| timburke | it still does that (essentially), just earlier -- line 426 | 16:40 |
| timburke | am i right in thinking the trouble is that we've got the etag for the encrypted segment, not the plaintext segment? | 16:41 |
| timburke | hopefully, the content length will ensure we don't trip the "Too many bytes..." error and leave us ready to receive some footers, but i'm not actually sure... | 16:43 |
| lifeless | peluse: how weird, that must be the first time in the universe easy_install has fixed something | 16:43 |
| jrichli | timburke: I assume that is the issue. I need to verify what etags are what in this method. I just know that this wasn't satisfying that conditional before (no raise before). | 16:46 |
| timburke | jrichli: so how does one read footers? i don't think i've ever done that...but presumably we should now do it before we close the seg_resp.app_iter on line 452 | 16:48 |
| peluse | lifeless: just lcuky I guess :) | 16:48 |
| *** zaitcev has joined #openstack-swift | 16:52 | |
| *** ChanServ sets mode: +v zaitcev | 16:52 | |
| *** tsg has quit IRC | 16:54 | |
| *** tsg has joined #openstack-swift | 16:55 | |
| *** gyee has joined #openstack-swift | 16:55 | |
| openstackgerrit | Michael Barton proposed openstack/swift: go: fix problem with using single config file https://review.openstack.org/246500 | 16:58 |
| *** mfalatic has joined #openstack-swift | 16:59 | |
| *** SkyRocknRoll has joined #openstack-swift | 17:03 | |
| *** daemontool has quit IRC | 17:04 | |
| *** daemontool has joined #openstack-swift | 17:04 | |
| *** tsg has quit IRC | 17:06 | |
| *** mzhou has quit IRC | 17:07 | |
| *** mahatic_ has quit IRC | 17:07 | |
| *** mzhou has joined #openstack-swift | 17:10 | |
| jrichli | timburke: I gotta find out which etag is seg_resp.etag. You see, in crypto world, there are 3 etags: 1) ciphertext MD5; 2) plaintext MD5 encrypted with obj key; 3) plaintext MD5 encrypted with cont key | 17:10 |
| *** mwheckmann has quit IRC | 17:10 | |
| *** tsg has joined #openstack-swift | 17:11 | |
| timburke | jrichli: extra fun! i've not been keeping up :) | 17:11 |
| *** rledisez has quit IRC | 17:12 | |
| *** janonymous has joined #openstack-swift | 17:16 | |
| *** nadeem has joined #openstack-swift | 17:17 | |
| *** arnox has quit IRC | 17:18 | |
| timburke | jrichli: remind me, what's the pipeline look like? and can you currently PUT a SLO manifest with crypto enabled? | 17:24 |
| *** wbhuber has quit IRC | 17:24 | |
| *** rcernin has joined #openstack-swift | 17:36 | |
| *** petertr7 is now known as petertr7_away | 17:39 | |
| jrichli | timburke: pipeline = catch_errors gatekeeper healthcheck proxy-logging cache bulk tempurl ratelimit crossdomain tempauth staticweb container-quotas account-quotas slo dlo versioned_writes decrypter trivial_keymaster encrypter proxy-logging proxy-server | 17:39 |
| jrichli | timburke: no, no SLO is working in the version with footers support. when using fake_footers instead, I believe this is working. | 17:41 |
| *** geaaru has quit IRC | 17:44 | |
| *** SkyRocknRoll has quit IRC | 17:45 | |
| *** Leigh507 has quit IRC | 17:48 | |
| *** klrmn1 has joined #openstack-swift | 17:54 | |
| *** haomaiwang has quit IRC | 17:55 | |
| *** mwheckmann has joined #openstack-swift | 17:55 | |
| *** mfalatic_ has joined #openstack-swift | 17:56 | |
| *** joeljwright1 has joined #openstack-swift | 17:57 | |
| *** mac_ifie_ has joined #openstack-swift | 17:57 | |
| *** zhill has joined #openstack-swift | 17:57 | |
| *** joeljwright has quit IRC | 17:58 | |
| *** baffle has quit IRC | 17:59 | |
| *** rvasilets__ has quit IRC | 17:59 | |
| *** mfalatic has quit IRC | 17:59 | |
| *** mac_ified has quit IRC | 17:59 | |
| *** jlhinson_ has joined #openstack-swift | 18:00 | |
| *** BAKfr has quit IRC | 18:00 | |
| *** baffle has joined #openstack-swift | 18:00 | |
| *** jordanP has quit IRC | 18:01 | |
| *** BAKfr has joined #openstack-swift | 18:02 | |
| *** jlhinson has quit IRC | 18:03 | |
| minwoob | I am trying to create a Swift daemon like the container-sharder that mattoliverau created, but swift-init isn't picking up on the conf nor the process. Are there any set of instructions that document this? | 18:03 |
| *** diazjf has quit IRC | 18:07 | |
| *** mahatic has joined #openstack-swift | 18:15 | |
| jrichli | minwoob: Maybe look at swift/common/manager.py at the ALL_SERVERS | 18:16 |
| jrichli | seems like you would modify that class | 18:17 |
| *** siva_krishnan has quit IRC | 18:31 | |
| *** mahatic has quit IRC | 18:35 | |
| *** hseipp has quit IRC | 18:35 | |
| *** daemontool_ has quit IRC | 18:37 | |
| minwoob | jrichli: That didn't seem to do it, but thanks for the suggestion. | 18:38 |
| minwoob | I'll dig further. | 18:39 |
| *** petertr7_away is now known as petertr7 | 18:41 | |
| *** joeljwright1 has quit IRC | 18:41 | |
| notmyname | minwoob: did you add it to setup.cfg? | 18:42 |
| *** daemontool has quit IRC | 18:44 | |
| minwoob | notmyname: I did not. Will go ahead and do that. | 18:51 |
| minwoob | Thanks. | 18:51 |
| *** tsg has quit IRC | 18:53 | |
| *** blmartin has quit IRC | 18:55 | |
| *** lpabon has joined #openstack-swift | 18:56 | |
| *** blmartin has joined #openstack-swift | 18:56 | |
| *** jlhinson_ has quit IRC | 18:58 | |
| *** ssagar has joined #openstack-swift | 18:59 | |
| *** sc68cal_ is now known as sc68cal | 19:00 | |
| *** ssagar has quit IRC | 19:01 | |
| notmyname | I'm making some assertions about swift. https://review.openstack.org/246558 based on the tags defined at http://governance.openstack.org/reference/tags/index.html | 19:04 |
| *** siva_krishnan has joined #openstack-swift | 19:04 | |
| timburke | notmyname: sweet! are all of the testing requirements already satisfied by the grenade gate? | 19:11 |
| *** siva_krishnan has quit IRC | 19:13 | |
| notmyname | timburke: I think | 19:15 |
| notmyname | timburke: colloquially, swift is always used as the examples for these, so i think/hope we're fine :-) | 19:15 |
| *** siva_krishnan has joined #openstack-swift | 19:15 | |
| timburke | notmyname: oh, i know we definitely adhere to all of these principles. just making sure we *actually* have tests for them :P | 19:16 |
| *** ekarlso has joined #openstack-swift | 19:24 | |
| *** petertr7 is now known as petertr7_away | 19:27 | |
| tdasilva | notmyname: how do these tags work? or what effect do they have? | 19:29 |
| notmyname | they "work" by being in a yaml file | 19:29 |
| notmyname | and I think the effect is that lists get built from them | 19:30 |
| notmyname | (I'm not trying to be cynical about it, really) | 19:30 |
| notmyname | so eg the foundation has the new project overview site they talked about at the summit | 19:30 |
| notmyname | and the tags will be shown there | 19:31 |
| notmyname | I think it will be like "these projects can do rolling upgrades" | 19:31 |
| tdasilva | lol...i know..probably not a good question...just trying to understand the context...what's their use, or where do I see them being displayed or something? is this part of the marketplace thing? | 19:31 |
| tdasilva | oh yeah, the overview site...that's what i was thinking | 19:31 |
| tdasilva | got it! | 19:31 |
| notmyname | the goal of the tags generally is to better communicate what each project can do. to ops, to end users, and overall much better than "integrated release" did | 19:31 |
| notmyname | there's an email. I'll look for the link | 19:31 |
| notmyname | here | 19:32 |
| notmyname | http://lists.openstack.org/pipermail/openstack-dev/2015-November/079542.html | 19:32 |
| tdasilva | "Project Navigator" i think it is called | 19:32 |
| notmyname | yeah, that | 19:32 |
| notmyname | so that's one place I guess they'll be used | 19:33 |
| torgomatic_ | oh, so *you* type up a diff with "assert" in it and suddenly you're "helping out project navigator" and "being a good openstack citizen", but when I do it I'm "lazy" and "littering the codebase with garbage" | 19:33 |
| *** petertr7_away is now known as petertr7 | 19:33 | |
| notmyname | assert:has-cranky-core-devs | 19:34 |
| tdasilva | lol | 19:34 |
| torgomatic_ | heh | 19:34 |
| tdasilva | i really don't understand swift's maturity being 3 out 5??? | 19:35 |
| notmyname | 3? | 19:35 |
| notmyname | we were 4 | 19:35 |
| tdasilva | i guess we got downgraded | 19:36 |
| tdasilva | http://www.openstack.org/software/releases/liberty/components/swift | 19:36 |
| notmyname | oh weird. we used to have something like 18 different SDKs listed | 19:37 |
| notmyname | also, I completely disagree that "only 62% of respondents use swift therefore it's not mature enough" is valid in any way | 19:38 |
| notmyname | tdasilva: oh, interesting. maybe none of these SDKs have been updated in this release cycle | 19:39 |
| tdasilva | yeah...really don't get that | 19:39 |
| tdasilva | notmyname: maybe that means maturity | 19:39 |
| tdasilva | oh, i see: "Status is an integer - the number of SDKs updated within the release cycle that support the service project " | 19:40 |
| tdasilva | that doesn't seem right | 19:40 |
| torgomatic_ | well, according to my arbitrary metrics that I am making up right meow, Swift is 11/10 on the cat-picture scale, beating out every other openstack project by at least 2 units | 19:40 |
| notmyname | yeah | 19:40 |
| *** torgomatic_ is now known as torgomatic | 19:40 | |
| *** ChanServ sets mode: +v torgomatic | 19:40 | |
| notmyname | ok, I've got a way to bring it up. I'll have to compose an email | 19:43 |
| *** diazjf has joined #openstack-swift | 19:43 | |
| blmartin | Hi guys, do you know of a good way to list containers in a hidden account? like .test_AUTH_Account? | 19:44 |
| *** jamielennox|away is now known as jamielennox | 19:45 | |
| *** jlhinson_ has joined #openstack-swift | 19:48 | |
| glange | blmartin: https://github.com/gholt/swiftly <-- you could install that tool on one of your systems "inside" of swift (one with the rings and access to all the swift servers) | 19:48 |
| *** zhill has quit IRC | 19:48 | |
| glange | and then run swiftly -D /v1/.test_AUTH_Account get | 19:49 |
| notmyname | glange: thanks. I was thinking "use swiftly" ;-) | 19:50 |
| glange | that's what we do! :) | 19:50 |
| blmartin | awseome glange! Thanks a lot! | 19:50 |
| *** jamielennox is now known as jamielennox|away | 19:52 | |
| *** jlhinson_ has quit IRC | 19:54 | |
| *** diazjf has quit IRC | 19:55 | |
| *** diazjf has joined #openstack-swift | 19:56 | |
| *** tsg has joined #openstack-swift | 20:03 | |
| *** eariasn has joined #openstack-swift | 20:04 | |
| eariasn | what would be a downside of switching Swift to Cassandra ? | 20:04 |
| tdasilva | eariasn: isn't Cassandra a nosql database??? doesn't seem right to compare to swift? | 20:07 |
| tdasilva | two different use cases, no? | 20:07 |
| *** zhill has joined #openstack-swift | 20:08 | |
| eariasn | no I meant like put Swift not to use SQLite but Cassandra | 20:08 |
| tdasilva | oohhh | 20:08 |
| tdasilva | where's that plugable backends patch again??? | 20:09 |
| *** jamielennox|away is now known as jamielennox | 20:09 | |
| glange | sqlite helps swift scale because there are many, many independed sqlite dbs in each swift cluster | 20:09 |
| glange | you don't want a central db because of that | 20:09 |
| glange | independent | 20:09 |
| glange | each account is a db, each container is a db | 20:10 |
| eariasn | so the use case is a single container with millions of objects | 20:10 |
| glange | you could do that with something simpler than swift | 20:11 |
| *** dkleute has joined #openstack-swift | 20:14 | |
| clayg | blmartin: you can also use an internal client -> https://gist.github.com/clayg/9f3078413f5b242034fd | 20:14 |
| clayg | eariasn: no reason to avoid a few million objects in a swift container today - maybe if a few million is a working set that's turned over every few weeks - tens of millions is less attractive - but not as unattractive as scaling a single database to the size of a cluster; or managing rebalance/sharding of multiple database servers | 20:18 |
| *** eranrom has joined #openstack-swift | 20:18 | |
| clayg | eariasn: sqlite is pretty great at solving container listings for swift - there's some pain points - but there's simpler ways to address them than throwing the baby out with bathwater (e.g. https://review.openstack.org/#/c/218738/) | 20:19 |
| blmartin | clayg: I'll play with that next, never hurts to be more familiar with the inner workings swift. Swiftly worked great for a quick find operation, I must say. | 20:22 |
| blmartin | Thanks for writing the gist! | 20:22 |
| clayg | blmartin: yeah it's pretty great | 20:23 |
| *** khoah64 has joined #openstack-swift | 20:23 | |
| clayg | blmartin: it was no trouble putting the gist together - i have dozens of snippets like that laying around for dev/test - i just grabbed the closest thing and added a usage string ;) | 20:23 |
| eranrom | Greetings, I am trying to prioritize, and was wondering what would be more helpful at this point in time: (1) Clay's gigantic ring builder patch or (2) Swift client auth improvements. Independently I will work with acoles on fast post as soon as he reaches a verdict | 20:25 |
| *** petertr7 is now known as petertr7_away | 20:26 | |
| clayg | eranrom: no my patch for sure - you should totally hold out on me until I get back to container-sync! | 20:27 |
| *** blmartin has quit IRC | 20:28 | |
| clayg | eranrom: maybe patch 202411 patch 214206 patch 160877 or patch 117710 | 20:29 |
| patchbot | clayg: https://review.openstack.org/#/c/202411/ - Add functional test for access control (RBAC) with... | 20:29 |
| patchbot | clayg: https://review.openstack.org/#/c/214206/ - Modify functional tests to use testr | 20:29 |
| patchbot | clayg: https://review.openstack.org/#/c/160877/ - Add storage policy support for sorting method | 20:29 |
| patchbot | clayg: https://review.openstack.org/#/c/117710/ - Add concurrent reads option to proxy | 20:29 |
| eranrom | clayg: I ownder how seriously I should take this :-) | 20:29 |
| eranrom | but alright. | 20:29 |
| eranrom | so just to make sure, since I was a way for some time - the swiftclient auth improvements were not addrressed yet, right? | 20:30 |
| clayg | eranrom: hey! fair is fair! social capitol is the only leverage we have | 20:30 |
| *** petertr7_away is now known as petertr7 | 20:30 | |
| eranrom | :-) | 20:30 |
| clayg | eranrom: well that and shame... like the one time i ebarassed redbo and he wrote patch 246500 | 20:31 |
| patchbot | clayg: https://review.openstack.org/#/c/246500/ - go: fix problem with using single config file | 20:31 |
| notmyname | clayg is all over the priorities ;-) | 20:31 |
| clayg | eranrom: I don't | 20:31 |
| clayg | ack! | 20:31 |
| redbo_ | :) | 20:31 |
| clayg | eranrom: I don't know what's up with swiftclient auth - I think the priority on swiftclient is probably patch 223319 - but joel has the as wip | 20:32 |
| patchbot | clayg: https://review.openstack.org/#/c/223319/ - New API documentation for python-swiftclient | 20:32 |
| notmyname | eranrom: nothings been done yet on the client auth. but the list clayg gave is a good one (beyong what you mentioned about the rings and fast-post) | 20:32 |
| clayg | eranrom: we also have... something about reverse listings - that or acoles' ec patches are on my radar today | 20:32 |
| eranrom | ok got it, thanks! | 20:33 |
| clayg | notmyname: I think for the moment we just wait for torgomatic's ring-fuzzer review on patch 241571 | 20:33 |
| patchbot | clayg: https://review.openstack.org/#/c/241571/ - Put part-replicas where they go | 20:33 |
| *** elligottmc has joined #openstack-swift | 20:33 | |
| clayg | notmyname: and of course I'd love to hear what cschwede thinks we should do for that patch review wise - and see someone from rax add a comment that they ran their rings over it and don't *necessarily* have to kill me immediately (like they can wait until the mid-cycle or whatever) | 20:34 |
| notmyname | hurricanerix janonymous have been looking at the testr change, so I think that's covered for now | 20:35 |
| notmyname | clayg: yes! | 20:35 |
| notmyname | eranrom: probably the simplest "get something useful done soon" review you could do is for patch 202411 | 20:36 |
| patchbot | notmyname: https://review.openstack.org/#/c/202411/ - Add functional test for access control (RBAC) with... | 20:36 |
| dkleute | Please reference me to outside archived discussions if the answer is out there. Two questions: | 20:36 |
| dkleute | 1. Has a database outside of SQLite been considered for much larger scale containers (Talking billions or even trillions of objects in a “single” container.) Riak, Cassandra, Redis… even a full clustered SQL etc. | 20:36 |
| dkleute | 2. Has any thought been put into a pluggable container architecture which would allow the underlying database to be swapped out at will to fit specific workloads? | 20:36 |
| dkleute | Both these questions come from client requirements in terms of scalability. I hope to generate a discussion as the objections I’ve heard to some of these “databases” may well be out of date. I apologize if I’m re-hashing an old subject. | 20:36 |
| *** wbhuber has joined #openstack-swift | 20:36 | |
| notmyname | didn't eariasn just ask about that? | 20:37 |
| eranrom | notmyname: ok sure. | 20:37 |
| clayg | notmyname: that review isn't simple - because keystone? | 20:37 |
| notmyname | dkleute: what particular scale questions are you getitng? | 20:37 |
| notmyname | clayg: *shhh* | 20:37 |
| dkleute | notmyname: He may have. I work with him trying to solve this issue. | 20:37 |
| hurricanerix | notmyname yeah, i looked at it some and it seems to be working. i wasn't sure what was up with the ec errors, but i thought i remember a comment from you that you expected those. | 20:38 |
| clayg | hurricanerix: not anymore - timburke fixed the EC functest errors | 20:38 |
| notmyname | hurricanerix: any EC errors I was seeing should be resolved by now. timburke fixed it | 20:38 |
| dkleute | We need to reach scale of billions of objects without significant (read that as any) performance degredation. | 20:38 |
| notmyname | hurricanerix: is there anything more you're waiting for or expecting to do on that patch then? | 20:38 |
| clayg | dkleute: billions of objects isn't a challange per say - but why do they need to be in the same container? | 20:39 |
| notmyname | dkleute: the common answer is to use a lot of containers. but if you're looking for billions of object in one container, then the container sharding work that mattoliverau is working on is what you want to track | 20:39 |
| hurricanerix | notmyname does timburke have a diff patch that he is going to push? i was looking at http://logs.openstack.org/06/214206/3/check/gate-swift-dsvm-functional/6952177/console.html.gz | 20:39 |
| hurricanerix | notmyname which is from the patch you pushed | 20:40 |
| glange | dkleute: don't put a billion objects in one container :) | 20:40 |
| dkleute | notmyname: Yes, I'm aware of the sharding work. The problem is sharding for this performance and scale will lead to 100,000 containers. | 20:40 |
| notmyname | dkleute: the number of objects is only half of the question. the other is the create/delete rate of those objects too | 20:40 |
| clayg | hurricanerix: i don't think that's from after timburke's fixed was merged - recheck no bug | 20:40 |
| notmyname | dkleute: is 100000 containers a problem? | 20:40 |
| clayg | notmyname: 10M container is not a problem | 20:41 |
| dkleute | And, I wrote a hashring to distribute load across thousands of containers. But, these are ultimately short term solutions as they present data expansion and migration challenges. | 20:41 |
| hurricanerix | clayg ahh, so it just needs to be rebased? let me do that and see what happens. | 20:41 |
| notmyname | clayg: righ | 20:41 |
| clayg | notmyname: I have no idea if 100M container is problem - I think 1B containers might be... well that would be interesting | 20:41 |
| dkleute | We haven't gotten to the point where we can test 100,000 containers. Hopefully soon. But my guess is it will be for operations like list. | 20:41 |
| clayg | dkleute: not if mattoliverau does his job right! | 20:42 |
| * clayg cracks the whip | 20:42 | |
| notmyname | clayg: right. dkleute was saying that sharding would leak to 100k contianers. I don't see the problem with that (depending on use case) | 20:42 |
| clayg | notmyname: well if they're just a hash of the name you can't do anything smart about prefix listing and subdir path queries and the like | 20:42 |
| clayg | notmyname: jumble jumble | 20:42 |
| notmyname | sure, but with billions of objects, I'm guessing that listing operations aren't something that's done often | 20:43 |
| mattoliverau | Even if they are a listing will only be 10000 at a time, so you never hit all containers | 20:43 |
| clayg | notmyname: idk, -infra had that use-case where they wanted to do like path queries but they wanted lots of things in a single container because of cdn/management overhead of lots and lots of containers | 20:44 |
| notmyname | dkleute: yeah, so the work mattoliverau is doing is in principle similar to you creating 100k containers on the client side | 20:44 |
| dkleute | We haven't tried 100k containers yet with that code either. | 20:44 |
| *** nadeem has quit IRC | 20:44 | |
| notmyname | clayg: because of wanting to duplicate filesystem semantics with nested directories (or metadata on those pseudo dirs) | 20:44 |
| dkleute | But, still, why do all this sharding work when there are scaleable databases out there that can have billions of records? | 20:45 |
| clayg | notmyname: yeah I mean *thats* the rub - if you're putting 100M objects in a container you're doing some sort of pseudo dir higharchy - at least that's what I've seen when evaluating s3 datasets for migrations | 20:45 |
| clayg | dkleute: you ever ran a HA cassandra cluster? Do you want to run 1000 of them? Or shard load/accounts and migrate between them? | 20:46 |
| clayg | dkleute: the pre-cursor to swift had central db pools - some other older object storage systems have architecture like that - it'd be a step backwards | 20:47 |
| *** nadeem has joined #openstack-swift | 20:47 | |
| dkleute | no, I haven't. And that may be the answer. Are you saying cassandra can't handle this kind of load at scale? Or Riak... etc. | 20:47 |
| *** nadeem has quit IRC | 20:47 | |
| clayg | sqlite is great! doing all the sharding stuff is just to reduce friction on clients - going with a different metadata architecture would need to be driven by something more than "but I don't *wanna* rewrite my app code to use a dozen containers for 10M objects" | 20:48 |
| *** nadeem has joined #openstack-swift | 20:48 | |
| mattoliverau | The awesome thing about sqlite is they can be treated like objects and thus get the durability and separation as normal objects. | 20:48 |
| glange | there are more sqlite databases in the solar system than any other kind of database | 20:49 |
| dkleute | Has anyone tried it? I'd love to see some metrics. | 20:50 |
| clayg | dkleute: I'm pretty keen on Cassandra for like document orientented storage/queries - I don't think it's an ideal solution for swift containers (because they're so simple and screw concensus) - but yeah I think trying to consolidate metadata at swift's scale is a step backwards - it's less of a knock on any other design - just that swift is solving swift's problems | 20:50 |
| elligottmc | I do recall in the latest spec I think there was discussion of sharding the database | 20:50 |
| clayg | dkleute: tried what? storing a billion documents in cassandra? sure | 20:51 |
| elligottmc | as an option being explored | 20:51 |
| clayg | glange: probably a half doezn on mars alone | 20:51 |
| elligottmc | which is not the same as going to new database necessarily but seems relevant | 20:52 |
| glange | it's a true fact! | 20:52 |
| clayg | comparing sqlite to a real database server/process/cluster is like apples to oranges - it's just totally different - just because your django app one time ran on sqlite and then you used mysql does not make them equivilant in any useful sense | 20:53 |
| notmyname | clayg: also unfair since the question implies that sqlite isn't a "real" DB ;-) | 20:54 |
| clayg | notmyname: it's not | 20:54 |
| dkleute | For cassandra, I think centralizing is the wrong word. Personally, I would still isolate containers using keyspaces and column families. I'm just surprised someone would want to handle sharding logic before doing something like that. | 20:55 |
| clayg | notmyname: well I don't know -- I guess database can mean different things | 20:55 |
| elligottmc | you can shard sqllite correct? | 20:55 |
| clayg | elligottmc: no - everything in sqlite is a file | 20:55 |
| elligottmc | k | 20:55 |
| clayg | dkleute: well... surprise! | 20:55 |
| elligottmc | so you would have to switch db's to shard | 20:55 |
| clayg | elligottmc: no - you can shard at the application level | 20:56 |
| glange | dkleute: there was an early version of "swift" that had a central postgres db | 20:56 |
| glange | and it didn't work/scale after a while | 20:56 |
| dkleute | Postgres is a central database. Cassandra/Riak are not. | 20:56 |
| glange | plus what happens when your db server goes down or whatever | 20:56 |
| clayg | dkleute: so here's the thing - for a good deal of deployments it's unclear that even a well tuned cassandra system could serve the swift keyspace for an entire cluster well | 20:56 |
| clayg | dkleute: the challange of concensus at scale is problematic to the eventually consistent heavily available design goals of swift | 20:57 |
| dkleute | clayg: ok, I'd love to know more about that. Doesn't something like cassandra meet those needs out of the box? There's no master, it's distributed, highly available. | 20:58 |
| elligottmc | https://www.sqlite.org/sharedcache.html | 20:59 |
| elligottmc | seems kinda shardish not the real mccoy | 20:59 |
| clayg | dkleute: Riak is not relevant - it's an object storage system like swift except that their listing api is... well... they say don't use it | 20:59 |
| elligottmc | but maybe helpful | 20:59 |
| elligottmc | if not already in code of so apologies | 20:59 |
| glange | dkleute: if you want to use casandra, go for it :) | 20:59 |
| dkleute | clayg: yes, cassandra is my "obvious" choice. Is there something other than "it may not scale" holding back trying it out? | 20:59 |
| notmyname | it would be a whole lot of work? | 21:00 |
| dkleute | more than making sharding stable at similar scale? (this all leads to my second question. pluggable storage architecture) | 21:00 |
| glange | haha | 21:00 |
| clayg | notmyname: well that and it may not work - column families have the same problem as sqlite indexes - you can't just throw shit in them forever w/o index updates getting slower | 21:01 |
| *** mzhou_ has joined #openstack-swift | 21:01 | |
| clayg | we have such a simple use-case for the datastore - there's no reason we can't do smart shit at the application level and use small simple lean and mean storage systems | 21:01 |
| dkleute | clayg: yes, there is a scaling limit to everything. but let's say we hit that. wouldn't it still be easier to shard far fewer times against different cassandra clusters? | 21:02 |
| *** mzhou has quit IRC | 21:02 | |
| *** mzhou_ is now known as mzhou | 21:02 | |
| notmyname | why would that be easier? | 21:02 |
| clayg | dkleute: notmyname: it's *known* to be *harder* | 21:02 |
| notmyname | you'd have the same work to do in swift, but operationally you'd have a huge amount of new complexity | 21:02 |
| dkleute | Because growing and shrinking clusters with thousands or millions of shards is difficult. | 21:03 |
| *** zacksh has quit IRC | 21:03 | |
| *** zacksh has joined #openstack-swift | 21:03 | |
| glange | matt oliver is going to make all that work | 21:03 |
| glange | out of the box | 21:03 |
| dkleute | I understand that. I may be assisting with that work. Bit I also need to look at alternatives. | 21:04 |
| clayg | dkleute: awesome! | 21:04 |
| clayg | dkleute: you should start with cassandra - i'm curious if you think a single instance could handle a cluster reasonably well. | 21:05 |
| clayg | dkleute: good luck | 21:05 |
| dkleute | I doubt it. But maintaining a small cluster of cassandra nodes is easier, to me, than implementing sharding. | 21:05 |
| glange | dkleute: why not just use cassandra without swift? | 21:05 |
| glange | or use ceph? | 21:05 |
| dkleute | I wouldn't use cassandra for storing the actual objects. | 21:06 |
| glange | what's the maximum value it can store? | 21:06 |
| dkleute | I'm just trying to solve the bottleneck of SQLite while still maintaining masterless and high availability eventual consistency. | 21:06 |
| glange | bottleneck being that sqlite can't handle a billion objects in a container? | 21:07 |
| dkleute | While also keeping code fast and clean. less code = less room for bugs. | 21:07 |
| glange | or what is the bottleneck? | 21:07 |
| dkleute | In our tests sqlite performance starts degrading between 1M and 10M objects. | 21:07 |
| glange | what kind of disks are your container db's on? | 21:08 |
| glange | spinning? | 21:08 |
| dkleute | I believe that test was SSD's. but I don't know for sure. | 21:08 |
| glange | ok, and just sharding your objects on the client side across many containers isn't an option? | 21:09 |
| notmyname | what aspect of sqlite performance was degrading? | 21:09 |
| *** blmartin has joined #openstack-swift | 21:09 | |
| glange | insertion time | 21:09 |
| glange | probably any operation on the db | 21:09 |
| eariasn | SSD's are being used now. | 21:09 |
| mattoliverau | But more op overhead (addition of Cassandra clusters). So what is better, getting a solution working built in or leaving it to the operators. But sure a pluggable backend is interesting.. But is a lot of work as well. | 21:10 |
| dkleute | I wasn't part of the original test. I will try and find all the specifics. Sharding across containers is an option and is being done. But growing a sharded environment is yet another complexity. | 21:10 |
| glange | sharding is pretty easy client side, pick a big number of containers to start with | 21:11 |
| dkleute | And we are talking about growth of Petabytes per month. | 21:11 |
| clayg | dkleute: you don't need these jerks to give you approval - you should just do it and publish the code - that'll show 'em | 21:11 |
| glange | petabytes per month for how long? | 21:11 |
| glange | you might end up with the biggest swift cluster ever pretty quickly | 21:11 |
| clayg | glange: 10PB next month 1EB by Q1 | 21:11 |
| dkleute | Forever. And gorwing | 21:11 |
| dkleute | *growing | 21:11 |
| glange | you are going to run into all sorts of problems then | 21:12 |
| *** NM has quit IRC | 21:12 | |
| dkleute | Yep. | 21:12 |
| glange | more than just sqlite :) | 21:12 |
| glange | like replication might be too slow at that point | 21:12 |
| dkleute | Yep. Hence looking at places to eliminate complexity and increase scalability. | 21:12 |
| glange | db replication is fast -- it's object replication that is the sticking point | 21:12 |
| clayg | dkleute: awesome - gl - keep us posted | 21:13 |
| glange | also, do you know about the object distribution variablity problem ? | 21:13 |
| dkleute | Yes, right now, the bottleneck we have is SQLite. So, that is what I'm working on. | 21:13 |
| dkleute | Nope. One problem at a time for me. ;) | 21:13 |
| *** zaitcev has quit IRC | 21:13 | |
| glange | well, you should consider that one too | 21:13 |
| glange | it's bad | 21:13 |
| mattoliverau | dkleute: clayg is right, give it a go | 21:13 |
| notmyname | which is why it's great to have you in here :-) ( dkleute and elligottmc) | 21:13 |
| clayg | dkleute: that's how you do it - priortize the list - knock off the first one - reprioritze - rinse/repeat | 21:13 |
| glange | matt, don't try to get out of solving these problems :() | 21:13 |
| glange | :) | 21:13 |
| dkleute | Matt, I would love to try this. But my knowledge of swift architecture is next to nothing at the moment. | 21:14 |
| glange | dkleute: write object distribution variability problem on your list | 21:14 |
| notmyname | dkleute: https://swiftstack.com/openstack-swift/ is a reasonable single-page overview | 21:14 |
| openstackgerrit | Tim Burke proposed openstack/python-swiftclient: Use bulk-delete middleware when available https://review.openstack.org/190887 | 21:15 |
| *** zaitcev has joined #openstack-swift | 21:15 | |
| *** ChanServ sets mode: +v zaitcev | 21:15 | |
| *** eranrom has quit IRC | 21:15 | |
| glange | dkleute: I don't think you can push 1EB into a swift cluster in 3 months | 21:16 |
| dkleute | That is a good overview of swift. But that's not what I meant. I meant from a coding perspective. | 21:16 |
| glange | that's another problem | 21:16 |
| clayg | glange: and you have some personal investment in preventing him from *trying*? | 21:17 |
| notmyname | dkleute: somewhat dated, but https://swiftstack.com/blog/2013/02/12/swift-for-new-contributors/ might help | 21:17 |
| clayg | glange: like you just wanna build up a reserve of i told you so's? | 21:17 |
| glange | haha, I'm just trying to help -- I see problems, I say something, I'm sorry :) | 21:17 |
| glange | I'm a helpful person | 21:17 |
| notmyname | glange: http://i.imgur.com/ES6sP.jpg ? | 21:18 |
| dkleute | Anyone want to spend an hour doing a "What if" code scenario? | 21:18 |
| dkleute | ;) | 21:18 |
| *** siva_krishnan has quit IRC | 21:18 | |
| clayg | dkleute: no | 21:18 |
| glange | dkleute: you should do everything in middleware | 21:18 |
| dkleute | didn't think so. ;) | 21:18 |
| dkleute | And my time limit just expired. Thank you so much for the help! I'll dig into it and see what I find. | 21:19 |
| glange | dkleute: you might need to hire a bunch of swift experts for high salaries | 21:19 |
| dkleute | We have some help. But, right now, I don't even understand what in the code needs to change. | 21:19 |
| clayg | glange is a swift expert | 21:20 |
| *** wbhuber_ has joined #openstack-swift | 21:20 | |
| glange | and I like money | 21:20 |
| kota_ | clayg: you too | 21:21 |
| clayg | kota_: I know next to nothing about swift architecture | 21:21 |
| glange | swift was designed by monkeys | 21:21 |
| glange | (I kid) | 21:21 |
| tdasilva | in the desert? | 21:21 |
| *** wbhuber has quit IRC | 21:24 | |
| mattoliverau | dkleute: sorry it's early for me and I haven't had coffee, but if you want to discuss in more detail I'm happy to listen, I like good ideas. Just email me :) | 21:24 |
| dkleute | Sure! Matt, I would love to talk to you about this. | 21:24 |
| dkleute | Do you have phone time? | 21:24 |
| clayg | <groan> | 21:24 |
| *** diazjf has quit IRC | 21:25 | |
| notmyname | dkleute: what scale of swift have you tested so far? | 21:26 |
| openstackgerrit | Richard Hawkins proposed openstack/swift: Modify functional tests to use testr https://review.openstack.org/214206 | 21:27 |
| notmyname | hurricanerix: thanks. that's a rebase? | 21:28 |
| hurricanerix | notmyname yeah | 21:28 |
| hurricanerix | i re-ran it and everything seemed good | 21:28 |
| mattoliverau | dkleute: email to start is best, you can write your idea down and I can respond in my own time. And point out areas that would need to change. Etc. But of brain storming. | 21:28 |
| hurricanerix | so hopefully this one passes the gates =) | 21:28 |
| dkleute | I think 10 million objects. Over a million saw performance degradation on a private cluster. | 21:28 |
| dkleute | Matt, works for me | 21:28 |
| dkleute | I have to sign off. Thanks for all the feedback! | 21:30 |
| notmyname | dkleute: there are many swift clusters today with billions of objects. I'd suggest working with swift's design as you grow. and I'd love to help you get involved in upstream dev work! our best contributions come from people solving production problems! | 21:31 |
| *** dkleute has quit IRC | 21:34 | |
| mattoliverau | Now that breakfast is over, morning! :) | 21:34 |
| notmyname | hi mattoliverau! | 21:35 |
| *** diazjf has joined #openstack-swift | 21:36 | |
| *** jlhinson has joined #openstack-swift | 21:42 | |
| *** mzhou_ has joined #openstack-swift | 21:42 | |
| *** mzhou has quit IRC | 21:44 | |
| *** mzhou_ is now known as mzhou | 21:44 | |
| *** haypo has left #openstack-swift | 21:45 | |
| *** diazjf has quit IRC | 21:45 | |
| *** diazjf has joined #openstack-swift | 21:48 | |
| *** dustins has quit IRC | 21:48 | |
| notmyname | hurricanerix: seems to work for me | 21:48 |
| *** tongli has quit IRC | 21:50 | |
| *** zaitcev has quit IRC | 21:55 | |
| *** hrou has quit IRC | 21:59 | |
| *** khoah64 has quit IRC | 22:08 | |
| *** minwoob has quit IRC | 22:21 | |
| *** rcernin has quit IRC | 22:21 | |
| *** petertr7 is now known as petertr7_away | 22:22 | |
| *** zaitcev has joined #openstack-swift | 22:28 | |
| *** ChanServ sets mode: +v zaitcev | 22:28 | |
| *** diazjf1 has joined #openstack-swift | 22:30 | |
| *** diazjf1 has quit IRC | 22:30 | |
| *** petertr7_away is now known as petertr7 | 22:31 | |
| *** mwheckmann has quit IRC | 22:33 | |
| *** diazjf has quit IRC | 22:33 | |
| *** zhill has quit IRC | 22:37 | |
| *** zhill has joined #openstack-swift | 22:40 | |
| *** diazjf has joined #openstack-swift | 22:42 | |
| *** thumpba has joined #openstack-swift | 22:54 | |
| onovy | hi, https://review.openstack.org/#/c/243040/ // final +A someone pls? :) | 22:56 |
| clayg | onovy: I think there's a bug open for that! | 22:59 |
| *** diazjf has quit IRC | 23:02 | |
| onovy | clayg, can't found it | 23:02 |
| *** tsg has quit IRC | 23:02 | |
| clayg | onovy: https://bugs.launchpad.net/swift/+bug/1508228 | 23:02 |
| openstack | Launchpad bug 1508228 in OpenStack Object Storage (swift) "replicator should warn when there's no local devices" [Undecided,In progress] - Assigned to Syed Ahsan Shamim Zaidi (ahsanmohsin04) | 23:02 |
| onovy | clayg, ah, thanks! | 23:02 |
| *** km has joined #openstack-swift | 23:04 | |
| openstackgerrit | Ondřej Nový proposed openstack/swift: Log error if a local device not identified https://review.openstack.org/243040 | 23:05 |
| *** km is now known as Guest6043 | 23:05 | |
| onovy | added to commit message | 23:05 |
| *** blmartin has quit IRC | 23:05 | |
| *** thumpba has quit IRC | 23:07 | |
| *** wbhuber_ has quit IRC | 23:07 | |
| *** wbhuber has joined #openstack-swift | 23:08 | |
| *** thumpba has joined #openstack-swift | 23:08 | |
| openstackgerrit | Jonathan Hinson proposed openstack/swift: Transition to 'keymaster' https://review.openstack.org/244904 | 23:09 |
| onovy | clayg, i think peter just implemented Darrell Bishop suggestion from https://review.openstack.org/#/c/180893/2 | 23:11 |
| clayg | swifterdarrell: ^ review onvoy's patch https://review.openstack.org/#/c/180893/2 | 23:11 |
| clayg | er... | 23:11 |
| clayg | swifterdarrell: patch 243040 | 23:11 |
| patchbot | clayg: https://review.openstack.org/#/c/243040/ - Log error if a local device not identified | 23:11 |
| onovy | Peter's patch :) | 23:12 |
| *** wbhuber has quit IRC | 23:12 | |
| clayg | ah - sorry | 23:13 |
| onovy | np, i was my suggestion, peter's code :) | 23:13 |
| onovy | it | 23:13 |
| clayg | onovy: oic - you have *people* for this sort of work? | 23:14 |
| clayg | ;) | 23:14 |
| onovy | we have simple workflow in seznam. i say: implement this and peter will send review. life is good :) | 23:14 |
| onovy | you gonna do what you gonna do... | 23:14 |
| onovy | gotta sry | 23:15 |
| *** jlhinson has quit IRC | 23:16 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-swiftclient: Updated from global requirements https://review.openstack.org/173982 | 23:18 |
| *** kei_yama has joined #openstack-swift | 23:26 | |
| *** wuhg has joined #openstack-swift | 23:26 | |
| *** mac_ifie_ has quit IRC | 23:27 | |
| *** alejandrito has quit IRC | 23:31 | |
| *** petertr7 is now known as petertr7_away | 23:32 | |
| *** thumpba has quit IRC | 23:34 | |
| *** diazjf has joined #openstack-swift | 23:40 | |
| *** diazjf1 has joined #openstack-swift | 23:43 | |
| *** diazjf has quit IRC | 23:46 | |
| *** chsc has quit IRC | 23:50 | |
| *** mwheckmann has joined #openstack-swift | 23:52 | |
| *** nadeem has quit IRC | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
