| *** itlinux has joined #openstack-swift | 00:00 | |
| *** itlinux has quit IRC | 00:04 | |
| *** hoonetorg has joined #openstack-swift | 00:08 | |
| *** JimCheung has joined #openstack-swift | 00:09 | |
| openstackgerrit | Alistair Coles proposed openstack/swift master: Add Composite Ring Functionality https://review.openstack.org/441921 | 00:12 |
|---|---|---|
| openstackgerrit | Alistair Coles proposed openstack/swift master: Experimental swift-ring-composer CLI to build composite rings https://review.openstack.org/451507 | 00:12 |
| acoles | kota_: ^^ we had +2 form clayg and myself, so I squashed the docs in to this patch | 00:14 |
| *** JimCheung has quit IRC | 00:14 | |
| *** NM has quit IRC | 00:14 | |
| clayg | acoles: I think cschwede_ took some issue with using `swift-ring-builder component.builder rebalance --force` | 00:19 |
| clayg | maybe he can offer a better alternative - we can merge that too! | 00:21 |
| *** gyee has quit IRC | 00:22 | |
| *** _JZ_ has quit IRC | 00:24 | |
| *** Sukhdev_ has quit IRC | 00:46 | |
| *** klrmn has quit IRC | 00:58 | |
| *** klrmn has joined #openstack-swift | 02:13 | |
| *** kei_yama has quit IRC | 02:25 | |
| *** kei_yama has joined #openstack-swift | 02:31 | |
| *** itlinux has joined #openstack-swift | 02:49 | |
| *** kei_yama_ has joined #openstack-swift | 02:59 | |
| *** kei_yama_ has quit IRC | 03:02 | |
| *** itlinux has quit IRC | 03:18 | |
| *** itlinux has joined #openstack-swift | 03:22 | |
| *** links has joined #openstack-swift | 03:30 | |
| *** Sukhdev has joined #openstack-swift | 03:38 | |
| *** oshritf_ has joined #openstack-swift | 03:40 | |
| *** kei_yama has quit IRC | 03:53 | |
| *** kei_yama has joined #openstack-swift | 04:02 | |
| *** JimCheung has joined #openstack-swift | 04:21 | |
| *** JimCheung has quit IRC | 04:25 | |
| *** oshritf_ has quit IRC | 04:30 | |
| *** kei_yama has quit IRC | 04:51 | |
| *** Sukhdev has quit IRC | 04:53 | |
| *** kei_yama has joined #openstack-swift | 05:09 | |
| *** jamielennox is now known as jamielennox|away | 05:10 | |
| *** SkyRocknRoll has quit IRC | 05:17 | |
| *** jamielennox|away is now known as jamielennox | 05:30 | |
| *** links has quit IRC | 05:33 | |
| *** links has joined #openstack-swift | 05:41 | |
| *** cshastri has joined #openstack-swift | 05:42 | |
| *** ChubYann has quit IRC | 05:54 | |
| *** itlinux has quit IRC | 05:56 | |
| *** itlinux has joined #openstack-swift | 06:01 | |
| *** itlinux has quit IRC | 06:12 | |
| *** tovin07_ has joined #openstack-swift | 06:17 | |
| *** pcaruana has joined #openstack-swift | 06:22 | |
| *** cshastri has quit IRC | 06:32 | |
| *** rcernin has joined #openstack-swift | 06:32 | |
| *** cshastri has joined #openstack-swift | 06:44 | |
| *** jaosorior has quit IRC | 06:53 | |
| *** adriant has quit IRC | 06:57 | |
| *** klrmn has quit IRC | 07:02 | |
| *** JimCheung has joined #openstack-swift | 07:03 | |
| *** jamielennox is now known as jamielennox|away | 07:07 | |
| *** JimCheung has quit IRC | 07:08 | |
| *** cshastri has quit IRC | 07:24 | |
| *** cshastri has joined #openstack-swift | 07:28 | |
| *** jamielennox|away is now known as jamielennox | 07:34 | |
| *** jaosorior has joined #openstack-swift | 07:43 | |
| *** silor has joined #openstack-swift | 08:01 | |
| *** geaaru has joined #openstack-swift | 08:07 | |
| *** mvk has quit IRC | 08:19 | |
| *** jaosorior is now known as jaosorior_lunch | 08:20 | |
| *** cbartz has joined #openstack-swift | 08:22 | |
| *** oshritf has joined #openstack-swift | 08:46 | |
| *** mvk has joined #openstack-swift | 08:51 | |
| *** JimCheung has joined #openstack-swift | 08:51 | |
| *** JimCheung has quit IRC | 08:56 | |
| *** zaitcev has quit IRC | 09:01 | |
| *** zaitcev has joined #openstack-swift | 09:02 | |
| *** ChanServ sets mode: +v zaitcev | 09:02 | |
| *** jaosorior_lunch is now known as jaosorior | 09:15 | |
| *** ejat has quit IRC | 09:25 | |
| *** ejat has joined #openstack-swift | 09:26 | |
| *** ejat has quit IRC | 09:26 | |
| *** ejat has joined #openstack-swift | 09:26 | |
| *** cshastri has quit IRC | 09:32 | |
| *** tovin07_ has quit IRC | 09:56 | |
| *** mvk has quit IRC | 10:06 | |
| *** mvk has joined #openstack-swift | 10:12 | |
| *** cshastri has joined #openstack-swift | 10:15 | |
| *** aulaitshooter has joined #openstack-swift | 10:23 | |
| *** aulaitshooter has left #openstack-swift | 10:25 | |
| *** JimCheung has joined #openstack-swift | 10:40 | |
| *** JimCheung has quit IRC | 10:44 | |
| dewanee | Hi all | 11:05 |
| dewanee | Hope the summit was nice. | 11:05 |
| dewanee | I have a weird problem maybe some of you can help shed light on: | 11:06 |
| dewanee | swift-object-replicator report between 50K and 100k failures evenly distributed across the devices | 11:07 |
| dewanee | on both nodes of a cluster (albeit with a different failure count for the two) | 11:07 |
| dewanee | any idea what could be causing that? the deployment is way under used with about 300 GB of data per device | 11:09 |
| *** asettle_ is now known as asettle | 11:13 | |
| *** NM has joined #openstack-swift | 12:36 | |
| *** lan has joined #openstack-swift | 12:43 | |
| *** klamath has joined #openstack-swift | 13:00 | |
| *** klamath has quit IRC | 13:00 | |
| *** klamath has joined #openstack-swift | 13:01 | |
| *** _JZ_ has joined #openstack-swift | 13:21 | |
| openstackgerrit | Christian Schwede proposed openstack/swift master: Add support to increase object ring partition power https://review.openstack.org/337297 | 13:42 |
| *** rcernin has quit IRC | 13:48 | |
| briancli1e | dewanee: you might turn up the log level to see what's going on, or possibly run one of them interactively with --once | 13:48 |
| *** rcernin has joined #openstack-swift | 13:49 | |
| briancli1e | dewanee: also, check syslog for any potential errors reported by rsyncd | 13:49 |
| briancli1e | and of course make sure each of them can connect to the other's rsyncd port (873) | 13:50 |
| *** briancli1e is now known as briancline | 13:52 | |
| *** NM has quit IRC | 13:52 | |
| *** jaosorior is now known as jaosorior_away | 14:00 | |
| *** NM has joined #openstack-swift | 14:02 | |
| *** kei_yama has quit IRC | 14:19 | |
| *** vint_bra has joined #openstack-swift | 14:39 | |
| *** cshastri has quit IRC | 14:39 | |
| *** tovin07 has joined #openstack-swift | 14:40 | |
| *** links has quit IRC | 14:58 | |
| *** itlinux has joined #openstack-swift | 15:09 | |
| *** itlinux has quit IRC | 15:18 | |
| *** LeoZhang has joined #openstack-swift | 15:35 | |
| *** oshritf has quit IRC | 15:48 | |
| notmyname | good morning | 15:50 |
| *** LeoZhang has quit IRC | 15:50 | |
| *** tonanhngo has joined #openstack-swift | 15:54 | |
| *** NM has quit IRC | 16:00 | |
| notmyname | cschwede_: I keep seeing you be mentioned on https://review.openstack.org/#/c/441921/21/swift/common/ring/composite_builder.py (line 86). can you comment? | 16:08 |
| patchbot | patch 441921 - swift - Add Composite Ring Functionality | 16:08 |
| cschwede_ | notmyname: Hi! it's already on my list, but unfortunately not today | 16:09 |
| notmyname | cschwede_: no worries. just wanted to make sure you saw it. thanks. :-) | 16:09 |
| notmyname | I know it's already late for you | 16:09 |
| notmyname | we are hoping to land that patch this week, though | 16:10 |
| *** sileht has quit IRC | 16:14 | |
| *** sileht has joined #openstack-swift | 16:20 | |
| *** tovin07 has left #openstack-swift | 16:23 | |
| notmyname | I'm keeping an eye on a couple of conversations happening on the mailing list right now | 16:26 |
| notmyname | one is on the cross-project request id. ie passing a request id from project to project.the gerrit patch is https://review.openstack.org/#/c/464746/ and there is a big thread on the ML | 16:26 |
| patchbot | patch 464746 - oslo-specs - global request id spec | 16:26 |
| notmyname | I'm planning on writing something (not sure if gerrit or ML) about how swift does this, in part because the proposal as-is doesn't work for swift | 16:27 |
| notmyname | the other conversation is about the application-specific passwords in keystone. there's been a lot of good points made in the ML thread, and some of it reminds me of tempurls. not sure if our experience with tempurls could be useful there, but maybe | 16:28 |
| notmyname | hmm... there's another thread too. the TC has been talking about changing the way they communicate. they've talked about both dropping the weekly tc meeting and also setting up a separate #openstack-tc irc channel (and maybe even a separate mailing list) | 16:30 |
| notmyname | while these ideas are somewhat distinct, they are related to each other | 16:30 |
| notmyname | eg the -tc channel idea is in the context of not having a tc meeting and instead doing ad-hoc conversations | 16:30 |
| notmyname | IMO these are some of the conversations to pay attention to right now in the broader openstack community | 16:32 |
| *** pcaruana has quit IRC | 16:53 | |
| *** rcernin has quit IRC | 16:53 | |
| *** SkyRocknRoll has joined #openstack-swift | 16:53 | |
| *** cbartz has quit IRC | 17:06 | |
| *** geaaru has quit IRC | 17:10 | |
| *** sileht has quit IRC | 17:19 | |
| *** sileht has joined #openstack-swift | 17:19 | |
| *** klrmn has joined #openstack-swift | 17:21 | |
| *** ChubYann has joined #openstack-swift | 17:27 | |
| clayg | briancline: great advice for dewanee - thanks! | 17:29 |
| notmyname | timburke: https://review.openstack.org/#/c/464746/ | 17:30 |
| patchbot | patch 464746 - oslo-specs - global request id spec | 17:30 |
| *** amit213 has quit IRC | 17:43 | |
| *** amit213 has joined #openstack-swift | 17:45 | |
| *** SkyRocknRoll has quit IRC | 17:50 | |
| *** mvk has quit IRC | 18:01 | |
| *** gyee has joined #openstack-swift | 18:06 | |
| *** zul has quit IRC | 18:09 | |
| *** openstackgerrit has quit IRC | 18:17 | |
| *** Sukhdev has joined #openstack-swift | 18:29 | |
| *** NM has joined #openstack-swift | 18:31 | |
| *** zul has joined #openstack-swift | 18:37 | |
| *** NM has quit IRC | 18:59 | |
| *** NM has joined #openstack-swift | 19:13 | |
| *** Sukhdev has quit IRC | 19:22 | |
| *** silor has quit IRC | 19:26 | |
| *** ujjain has quit IRC | 20:32 | |
| *** Sukhdev has joined #openstack-swift | 20:36 | |
| *** Sukhdev has quit IRC | 20:37 | |
| *** Sukhdev has joined #openstack-swift | 20:38 | |
| *** ujjain has joined #openstack-swift | 20:44 | |
| *** ujjain has joined #openstack-swift | 20:44 | |
| *** skudlik has joined #openstack-swift | 21:03 | |
| *** mwheckmann has joined #openstack-swift | 21:23 | |
| mwheckmann | Hello. Anyone know of a way to speed up container PUTs on a large global cluster with many sites (regions)? The problem is that there is no write_affinity for the container ring | 21:24 |
| mwheckmann | dropping the replica count would in turn lower the require quorum, but that's only a partial solution and it would mean spreading things a little thin | 21:25 |
| clayg | is the latency between the regions really slow - or is creating a container a common operation (e.g. client always PUT container before PUT object)? | 21:27 |
| clayg | mwheckmann: or is it the container update after an object PUT/DELETE that is slow? | 21:28 |
| mwheckmann | it's possibly that. | 21:28 |
| mwheckmann | You're probably right, but our first by latency on writes has also gone up. drastically in fact. As we've been adding regions | 21:29 |
| mwheckmann | first byte write latency I mean | 21:29 |
| *** mvk has joined #openstack-swift | 21:29 | |
| clayg | how many regions? | 21:30 |
| mwheckmann | 4: NCSA-east, EMEA-west, EMEA-east, APAC | 21:30 |
| clayg | any you're running 4 a&c replicas? | 21:30 |
| clayg | one in each region? or mix & match of 3 replicas | 21:31 |
| mwheckmann | 9 since we wanted to have at least 2 copies per region. | 21:31 |
| mwheckmann | in fact, it was 9 when we still had 3 regions. (3x per region). We haven't yet adjusted after the latest region as I want to figure out the best thing to do. | 21:32 |
| clayg | that's a good direction to go I think... | 21:32 |
| mwheckmann | 3x per region? | 21:33 |
| clayg | 2-3 per region - yes | 21:33 |
| mwheckmann | ok. But how to deal with the fact that we have no write affinity for a&c ? | 21:34 |
| clayg | there's not currently anyway to limit the number or replicas that are updated synchronously on write | 21:34 |
| clayg | number/locality | 21:34 |
| mwheckmann | meaning, it's (n +1) // 2? | 21:35 |
| notmyname | how many object replicas do you have? | 21:35 |
| mwheckmann | depends on the policy. For a policy with 3 regions, it's 6 replicas | 21:36 |
| mwheckmann | of course, the overload factor is adjusted if necessary to provide an even distribution of replicas | 21:36 |
| notmyname | the object PUT path updates the containers. so (for easy math), if you have 3 object replicas and 12 container replicas, each object server will be updating 4 container servers (12/3) | 21:37 |
| notmyname | the container update that the object server does shouldn't affect end-user latency. if all the updates don't happen in `container_update_timeout` seconds, then the updates a queued for later (https://github.com/openstack/swift/blob/master/etc/object-server.conf-sample#L64) | 21:39 |
| notmyname | oh! what's your memcache look like? | 21:39 |
| notmyname | do you have a separate memcache pool for each region? | 21:39 |
| mwheckmann | yes, we do | 21:39 |
| notmyname | oh ok | 21:39 |
| notmyname | that would have been an easy explanation :-) | 21:39 |
| mwheckmann | I'll double check those configs as things can get pretty dynamic with controllers coming and going (locally within a region), but there is no cross-region memcache traffic, it wouldn't even work from an L3 routing perspective | 21:40 |
| mwheckmann | memcache sits on the backend | 21:41 |
| *** NM has quit IRC | 21:41 | |
| *** alenavolk has joined #openstack-swift | 21:42 | |
| *** NM has joined #openstack-swift | 21:43 | |
| *** NM has quit IRC | 21:44 | |
| mwheckmann | hmm. would something have changed with respect to the proxy talking to the container DBs on pre/post PUT since 2.5.0 ? | 21:45 |
| mwheckmann | 'cause I'm seeing a lot of traffic to remote container devices early on an upload of a large object | 21:46 |
| mwheckmann | I'm running 2.5.0 on this cluster (I know, old, sorry) | 21:47 |
| mwheckmann | to clarify, I'm talking about traffic from the proxy server to remote container servers | 21:51 |
| mwheckmann | going to create a pcap and open it in wireshark | 21:51 |
| *** alenavolk has quit IRC | 21:55 | |
| mwheckmann | I'm seeing a lot of misplace_objects requests. I have to go, but looks like I'm going to have to dig deeper.. | 21:59 |
| *** openstackgerrit has joined #openstack-swift | 22:00 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Ring doc cleanups https://review.openstack.org/465184 | 22:00 |
| *** mwheckmann has quit IRC | 22:01 | |
| *** mwheckmann has joined #openstack-swift | 22:02 | |
| *** Sukhdev has quit IRC | 22:11 | |
| *** klamath has quit IRC | 22:12 | |
| *** tonanhngo has quit IRC | 22:13 | |
| *** _JZ_ has quit IRC | 22:17 | |
| *** itlinux has joined #openstack-swift | 22:19 | |
| *** tonanhngo has joined #openstack-swift | 22:27 | |
| *** tonanhngo has quit IRC | 22:32 | |
| *** vint_bra has quit IRC | 22:35 | |
| onovy | fyi, Swift go go :) http://imgur.com/a/glVGK | 22:45 |
| notmyname | onovy: cool! | 22:45 |
| *** Sukhdev has joined #openstack-swift | 22:45 | |
| skudlik | onovy: my graph! :) | 22:45 |
| notmyname | onovy: I see "100G". is that 100 gbit/sec? | 22:45 |
| notmyname | what are the colors? | 22:45 |
| onovy | notmyname: of course :) | 22:46 |
| onovy | notmyname: servers | 22:46 |
| onovy | skudlik: :) | 22:46 |
| notmyname | nice | 22:46 |
| onovy | tbh there is varnish before swift | 22:46 |
| notmyname | looks like you got a popular customer :-) | 22:46 |
| notmyname | what happened? | 22:46 |
| onovy | notmyname: new crazy show released | 22:47 |
| openstackgerrit | Tim Burke proposed openstack/swift master: domain_remap: stop mangling client-provided paths https://review.openstack.org/464217 | 22:47 |
| notmyname | onovy: it seems to be popular ;-) | 22:48 |
| onovy | it's, our over all record of traffic | 22:48 |
| onovy | it was something about 1/3 of whole czech internet traffic | 22:49 |
| skudlik | we'll see tomorrow .. now it is 0:49 am here .. | 22:49 |
| onovy | yes, we will brake czech internet | 22:50 |
| onovy | gn | 22:51 |
| skudlik | ;-) we were planning to do some tests with Pavel tomorrow (with one locality switch off) .. not a good idea with this show | 22:53 |
| notmyname | wow | 22:54 |
| notmyname | is that from swift itself or is that the varnish layer in front of swift? | 22:55 |
| *** adriant has joined #openstack-swift | 22:56 | |
| skudlik | from varnish | 22:56 |
| notmyname | ah ok | 22:56 |
| skudlik | nginx->varnish->swift | 22:56 |
| skudlik | varnish is caching everything that comes from swift | 22:57 |
| notmyname | that's similar to how wikipedia uses swift IIRC | 22:58 |
| notmyname | makes a lot of sense for public content | 22:58 |
| notmyname | onovy: skudlik: that's really cool to see. thanks for sharing :-) | 23:00 |
| skudlik | sum 4.2k requests/s peek to object servers | 23:01 |
| skudlik | gn | 23:05 |
| *** skudlik has left #openstack-swift | 23:05 | |
| *** alenavolk has joined #openstack-swift | 23:11 | |
| *** alenavolk has quit IRC | 23:12 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: fixup! Add container sharding to Swift containers https://review.openstack.org/450491 | 23:20 |
| *** bkopilov has quit IRC | 23:28 | |
| *** bkopilov has joined #openstack-swift | 23:29 | |
| *** Sukhdev has quit IRC | 23:40 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!