Monday, 2014-06-30

« Sunday, 2014-06-29 Index Tuesday, 2014-07-01 »
*** openstack has joined #openstack-swift13:49
*** psharma has quit IRC13:57
portanteacoles: are they kinda accurate?14:00
acolesportante: yes. the only one i questioned was cornish but i have family roots there.14:01
acolesportante: have you found a US equivalent?14:02
*** redbo_ has quit IRC14:09
*** kenhui has quit IRC14:11
*** kenhui has joined #openstack-swift14:11
*** rook_ has joined #openstack-swift14:16
*** bkopilov_ has joined #openstack-swift14:43
*** mlipchuk has quit IRC14:46
*** kopparam_ has quit IRC14:47
*** kopparam has joined #openstack-swift14:48
*** kevinc_ has joined #openstack-swift14:51
*** kopparam has quit IRC14:52
*** swills has quit IRC14:54
*** zhiyan is now known as zhiyan_14:55
*** zhiyan_ is now known as zhiyan14:56
*** Mikalv is now known as meeh214:57
*** zhiyan is now known as zhiyan_14:59
*** swills has joined #openstack-swift15:00
*** swills has quit IRC15:01
*** swills has joined #openstack-swift15:01
*** mlipchuk has joined #openstack-swift15:03
openstackgerritgholt proposed a change to openstack/swift: SimpleClient http proxying  https://review.openstack.org/10259815:05
*** thebloggu has joined #openstack-swift15:05
*** mitz- has quit IRC15:07
*** mitz has joined #openstack-swift15:07
*** kevinc_ has quit IRC15:09
*** kevinc_ has joined #openstack-swift15:11
theblogguI have a 5 zone, 3 replicas (1 Proxy Node, 5 Storage Nodes)  swift cluster with thousands of small objects which I haven't touched for 2 weeks. no objects were added or deleted. is it normal that I have the object replicator opening thousands of rsyncs between the nodes?15:13
*** mitz has quit IRC15:15
*** mitz has joined #openstack-swift15:16
*** chandan_kumar has quit IRC15:18
openstackgerritAlistair Coles proposed a change to openstack/python-swiftclient: Add keystone v3 auth support  https://review.openstack.org/9178815:23
*** mlipchuk has quit IRC15:27
*** byeager has joined #openstack-swift15:31
*** foexle has quit IRC15:33
*** kevinc_ has quit IRC15:50
*** nacim has quit IRC15:52
*** gyee has joined #openstack-swift15:52
*** kevinc_ has joined #openstack-swift15:53
*** dANOKELOFF has quit IRC15:55
*** chandan_kumar has joined #openstack-swift15:59
*** meeh2 is now known as Meeh15:59
*** byeager has quit IRC16:02
*** byeager has joined #openstack-swift16:03
*** mwstorer has joined #openstack-swift16:08
notmynamegood morning16:23
*** amandap_ is now known as amandap16:40
*** miqui has quit IRC16:46
notmynamecreiht: portante: peluse_: briancline: how's Swift 2.0 in the lab going? since I haven't heard anything, I'm assuming we're still good to go for a final release this week (finish QA on Wed and release on Thurs)16:48
*** byeager has quit IRC16:49
*** annegent_ has joined #openstack-swift16:55
peluse_notmyname:  I should be done with rolling upgrade tests today, so far so good16:56
notmynameawesome. great to hear16:56
*** nsquare has quit IRC16:57
peluse_yeah, got started Tue last week then went and took vacation for the rest of the week and somehow not much got done on it :)16:57
notmynameFYI, for those of you interested in openstack governance things, tomorrow's TC meeting (at 2000UTC, 1pm pacific, in #openstack-meeting) is focused on the DefCore stuff16:58
notmyname(all TC meetings are open meetings and anyone can attend and speak. only TC members can vote)16:59
*** thebloggu has quit IRC17:00
*** thebloggu has joined #openstack-swift17:02
theblogguI have a 5 zone, 3 replicas (1 Proxy Node, 5 Storage Nodes)  swift cluster with thousands of small objects which I haven't touched for 2 weeks. no objects were added or deleted. is it normal that I have the object replicator opening thousands of rsyncs between the nodes?17:02
*** mmcardle has quit IRC17:04
*** Midnightmyth has joined #openstack-swift17:04
notmynamethebloggu: it depends. "thousands" sounds excessive, but there still could be some activity. in general, if you need to limit that, take a look at the max_connections setting in rsyncd.conf17:05
notmynamethebloggu: also, and probably first, take a look at your logs and see what's getting reported there. is there data actually getting moved around? look for the log lines that say a replication pass is finished17:06
notmynamethebloggu: and if you have any monitoring integration set up with statsd, look at the replication timing. or look at the output of `swift-recon --replication` to see stats there17:08
thebloggunotmyname: I have a firewall between the proxy node and 1 storage node (same real machine) and the other 4 storage nodes. the firewall logs all the traffic and i got out of space. upon inspection it was the traffic between the real machine and the others on port 600017:09
notmynamethebloggu: ah, so not rsync?17:09
thebloggunotmyname: I saw somthing like a thousand requests per second and also, i sniffed the traffic for 4/5 seconds17:10
notmynamethebloggu: that makes total sense then. the replication process will ping the object servers (default port 6000) with the REPLICATE verb to check if things need to be replicated (it checks a high-level checksum). if data is out of sync (checksums don't match), then it fires up rsync to move the data17:10
thebloggunotmyname: ah ok, in the traffic i sniffed there was REPLICATE requests, so you're probably right17:12
notmynamethebloggu: so you can limit that traffic with the config options in the [object-replicator] section of /etc/swift/object-server.conf (see the sample config for lots of docs on the settings)17:12
notmynamethebloggu: for example, you may want to lower the concurrency or raise the run_pause17:12
thebloggunotmyname: but if this happens in this setup with something like 10 containers and a few thousand files how will it scale later?17:13
thebloggunotmyname: will the process not became saturated?17:14
*** annegent_ has quit IRC17:15
thebloggus/became/become17:15
thebloggunotmyname: and BTW, is that ammount of traffic normal for my setup?17:18
*** kopparam has joined #openstack-swift17:22
*** nsquare has joined #openstack-swift17:22
*** andyandy has joined #openstack-swift17:25
*** kopparam has quit IRC17:26
andyandyI am using swiftbrowser, when I upload a file it says: "Unauthorized" but from command line works perfectly, any suggestions is welcome, thank you very much17:27
*** miqui has joined #openstack-swift17:29
notmynamecschwede may be able to help that out andyandy17:32
notmynamethebloggu: it depends somewhat on the partition power you have and how many partitions are on each drive (or server)17:33
*** kenhui has quit IRC17:34
notmynamethebloggu: here's how it works: each partition is on disk as a directory and has a "splaying directory" structure one level underneath it17:34
*** kenhui has joined #openstack-swift17:34
notmynamethebloggu: that is, the partition is a number, and there are 4096 possible directories as child directories of the partition directory17:34
andyandythank you notmyname17:34
notmynamethebloggu: swift maintains a checksum of the data in each of those 4096 directories. in the normal replication process, the REPLICATE verb is just checking that the checksums match for the local and remote copy17:35
notmynamethebloggu: so as your cluster grows, you'll actually have less partitions assigned to each drive, which means that each drive has to do less of the checking, which means that basically things get better as your cluster gets bigger17:36
*** angelastreeter has joined #openstack-swift17:37
*** andyandy has quit IRC17:43
*** angelastreeter has quit IRC17:48
notmynamethebloggu: in fact, this is part of the tension about choosing the right part power when you first build your cluster. you can't change the part power after it's set, so you need something big enough to handle how big your cluster will get, but you also need it to be small enough so that you aren't overwhelmed by this sort of system overhead17:48
thebloggunotmyname: hmm, interesting17:48
thebloggunotmyname: one more question though, the object-replicator seems to be consuming much CPU even tough the cluster is not being used. is it normal?17:51
notmynamethebloggu: same issue. spamming REPLICATE requests and managing the net connections17:52
thebloggunotmyname: ok, thank you once again :)17:53
notmynamethebloggu: and I'm not saying I'm proud of how much CPU overhead all that takes. but it's not surprising based on the other things you've said17:53
notmynamethebloggu: so...another thing you could look in to is using the "experimental" ssync feature of swift. it replaces the rsync transport with swift primitives. it will be the future default, but it's still a "try it out and see how it works for you" thing right now. there are some very large clusters using it. but getting your feedback too would be great for all of us17:55
thebloggunotmyname: so I can take from that that the ammount of traffic I'm getting may be higher than I should expect right?17:55
thebloggunotmyname: hmm, ok, I'll take a look into it17:56
*** zaitcev has joined #openstack-swift17:58
*** ChanServ sets mode: +v zaitcev17:58
*** angelastreeter has joined #openstack-swift17:59
*** angelastreeter has quit IRC18:02
thebloggunotmyname: I've got to go. I want to thank you for your help.18:03
thebloggunotmyname: tomorrow I'll try to figure this all better. I'm not sure if the ammount of traffic I get is normal or not.18:04
*** thebloggu has quit IRC18:05
notmynamethebloggu: you know, in some sense "normal" doesn't matter. what you are seeing does matter, and if your hardware can handle it matters. that is, it doesn't matter what rackspace or swiftstack clusters are doing because you need to keep yours running18:05
openstackgerritSamuel Merritt proposed a change to openstack/swift: Buffer DiskfileWriter writes  https://review.openstack.org/9113318:16
openstackgerritJosh Gachnang proposed a change to openstack/python-swiftclient: Adding Swift Temporary URL support  https://review.openstack.org/10263218:41
openstackgerritJosh Gachnang proposed a change to openstack/python-swiftclient: Adding Swift Temporary URL support  https://review.openstack.org/10263218:47
creihtnotmyname: still in progress18:52
notmynamecreiht: thanks18:54
*** mmcardle has joined #openstack-swift19:04
*** annegent_ has joined #openstack-swift19:05
openstackgerritTushar Gohad proposed a change to openstack/swift: EC: Add support for policy types, 'erasure_coding' policy  https://review.openstack.org/10363219:10
*** tsg has joined #openstack-swift19:10
notmynametsg: did you mean to propose that to master or to feature/ec?19:15
peluse_heh, I just commenmted on that :)19:16
*** tsg_ has joined #openstack-swift19:18
*** nsquare has quit IRC19:19
*** tsg has quit IRC19:19
* tsg_ working on fixing .. notmyname thanks for pointing out19:20
*** joeljwright has quit IRC19:23
*** kevinc_ has quit IRC19:23
*** kenhui has quit IRC19:24
*** mmcardle has quit IRC19:26
*** sileht_ has quit IRC19:34
*** sileht has joined #openstack-swift19:35
*** annegent_ has quit IRC19:35
openstackgerritTushar Gohad proposed a change to openstack/swift: EC: Add support for policy types, 'erasure_coding' policy  https://review.openstack.org/10363219:39
*** annegent_ has joined #openstack-swift19:39
*** annegent_ has quit IRC19:40
*** annegent_ has joined #openstack-swift19:40
notmynametsg_: still to master19:41
tsg_notmyname: not sure what's up with my tree19:41
notmynametsg_: ok, so you need `git review feature/ec` rather than just `git review`19:42
tsg_notmyname: that wasn't necessary earlier?19:42
notmynametsg_: peluse_: also, someone needs to update the .gitreview file on the feature/ec branch to default to feature ec. since the branch was recreated, it's currently set to master19:42
* notmyname is in a meeting19:43
notmynameI can get to it later if nobody else does it first19:43
tsg_notmyname: gotcha - that must be it19:43
tsg_notmyname: in the meantime, I will submit specifying the branch explicitly19:44
notmynamethanks19:44
openstackgerritTushar Gohad proposed a change to openstack/swift: EC: Add support for policy types, 'erasure_coding' policy  https://review.openstack.org/10364419:49
notmynametsg_: yay!19:49
tsg_<notmyname>:  :) didn't realize I had to abandon the previous changeset19:50
notmynametsg_: ah. I wondered how that was handled. I guess it wasn't19:50
tsg_notmyname: it would have been, if I had the ability to force-push to gerrit ;)19:50
*** chandan_kumar has quit IRC19:52
*** nsquare has joined #openstack-swift19:52
*** kevinc_ has joined #openstack-swift20:02
openstackgerritJohn Dickinson proposed a change to openstack/swift: default git review to push to feature/ec branch  https://review.openstack.org/10364820:13
notmynametsg_: peluse_: ^^ done20:14
peluse_notmyname:  cool thx.  BTW, updating my 2.0 test compelte ETA to tomorrow, too many INTs today....20:16
notmynamepeluse_: ack. thanks20:16
*** byeager has joined #openstack-swift20:17
*** Midnightmyth has quit IRC20:20
*** fifieldt has quit IRC20:25
*** fifieldt has joined #openstack-swift20:26
*** wasmum is now known as zz_wasmum20:27
*** kopparam has joined #openstack-swift20:28
*** zz_wasmum is now known as wasmum20:30
*** kopparam has quit IRC20:33
*** wasmum is now known as zz_wasmum20:36
*** byeager has quit IRC20:37
*** byeager has joined #openstack-swift20:38
*** tdasilva has quit IRC20:39
*** byeager has quit IRC20:42
*** byeager has joined #openstack-swift20:42
*** byeager_ has joined #openstack-swift20:45
*** byeager has quit IRC20:49
*** zz_wasmum is now known as wasmum21:24
*** cm2_ has joined #openstack-swift21:27
*** mkollaro1 has joined #openstack-swift21:28
*** mkollaro has quit IRC21:28
*** nthacker_ has joined #openstack-swift21:29
*** wasmum is now known as zz_wasmum21:31
*** cm2 has quit IRC21:34
*** cm2_ is now known as cm221:34
*** nthacker has quit IRC21:35
*** rook_ has quit IRC21:37
*** mtreinish has joined #openstack-swift21:42
*** annegent_ has quit IRC21:45
*** annegent_ has joined #openstack-swift21:46
*** annegent_ has quit IRC21:51
nthacker_hey guys good job on the swift 2.0!!21:57
*** byeager_ has quit IRC21:58
notmynamenthacker_: thanks! it's a huge effort from a lot of people. quite a few of the core devs really spent a lot of time on it to make it happen (see also the mailing list link in the topic)22:00
*** byeager has joined #openstack-swift22:01
mattoliverauMorning all22:09
notmynamemattoliverau: howdy22:10
notmynamemattoliverau: is it 8am or 9am where you are (in sydney?)?22:10
*** marcusvrn has quit IRC22:11
mattoliveraunotmyname: 8am in Melbourne Australia :)22:11
notmynamegotcha. thanks22:11
openstackgerritA change was merged to openstack/swift: default git review to push to feature/ec branch  https://review.openstack.org/10364822:23
notmynametsg_: peluse_: ^ there :-)22:24
*** byeager has quit IRC22:47
openstackgerritA change was merged to openstack/swift-specs: fix missing link for py-modules on index  https://review.openstack.org/10291322:48
*** annegent_ has joined #openstack-swift22:53
*** kevinc_ has quit IRC22:59
*** kevinc_ has joined #openstack-swift23:04
*** mkollaro1 has quit IRC23:13
notmynamehttp://bit.ly/swift_review_dashboard23:14
*** ChanServ changes topic to "Currently in Swift 2.0 RC phase -- read all about it at http://lists.openstack.org/pipermail/openstack-dev/2014-June/038357.html | master is open for dev work | http://bit.ly/swift_review_dashboard"23:15
*** annegent_ has quit IRC23:17
*** openstackgerrit has quit IRC23:19
*** kevinc_ has quit IRC23:20
notmynameFYI that dashboard URL was created with https://review.openstack.org/#/c/103683/23:21
*** openstackgerrit has joined #openstack-swift23:21
*** kevinc_ has joined #openstack-swift23:24
*** zhiyan_ is now known as zhiyan23:26
*** zhiyan is now known as zhiyan_23:27
*** zhiyan_ is now known as zhiyan23:29
mattoliveraunotmyname: Awesome dashboard, that'll make life a while lot easier :)23:30
mattoliverau*whole23:30
*** zhiyan is now known as zhiyan_23:30
notmynameI hope so23:30
notmynamethe point is to call attention to the right things and not get overwhelmed by all the rest23:31
notmynameso I'm pretty sure we can find ways to further improve it. but I like it for now. I'll certainly be using it23:31
claygnotmyname: new dashboard is pretty cool!23:35
notmynameclayg: cool. hope it's useful to you23:36
notmynamepeluse_: tsg_: note that there is a section for feature/ec at the bottom of that dashboard. the other sections are only branch:master23:36
notmynameclayg: ^23:36
mattoliveraunotmyname: none of my patch sets in review appear in the list, I'm obviously special :P23:37
*** kevinc_ has quit IRC23:38
peluse_notmyname:  very nice23:45
peluse_clayg:  have you by any chance tried tsg's policy patch yet?  I'm having some env issue with my SAIO that he's not seeing... wondering if you could lend a hand23:46
peluse_clayg:  never mind...23:53
« Sunday, 2014-06-29 Index Tuesday, 2014-07-01 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!