| *** physcx has joined #openstack-swift | 00:08 | |
| *** Masahiro has joined #openstack-swift | 00:10 | |
| notmyname | glange_: this looks like something you'd find useful http://www.linuxjournal.com/content/autokey-shorthand-typists?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+linuxjournalcom+%28Linux+Journal+-+The+Original+Magazine+of+the+Linux+Community%29 | 00:12 |
|---|---|---|
| *** sungju has joined #openstack-swift | 00:12 | |
| *** Masahiro has quit IRC | 00:14 | |
| tdasilva_ | does anybody have a suggestion for an irc client for ios? | 00:16 |
| *** openstackgerrit has quit IRC | 00:18 | |
| *** openstackgerrit has joined #openstack-swift | 00:19 | |
| notmyname | tdasilva_: I have LimeChat on my phone. I rarely use it, but when I do it seems sufficient | 00:19 |
| jokke_ | tdasilva_: no, I tend not to irc from the switches ;) | 00:19 |
| mattoliverau | tdasilva_: I use quassel, you install the server somewhere (rackspace cloud for me) and then you can install clients on what ever you want and you can access the same session from anywhere. Laptop, phone etc. I use an android client on my phone, but there is an iphone one i think. | 00:20 |
| notmyname | tdasilva_: but I'm also pre-disposed to that one since I also use LimeChat on OSX | 00:20 |
| notmyname | mattoliverau: that sounds like an irc bouncer | 00:20 |
| mattoliverau | notmyname: yeah it is.. but easier to set up :) | 00:20 |
| tdasilva_ | thanks for all the tips...i've thought about setting up an irc bouncer, but haven't got to do that yet | 00:21 |
| mattoliverau | but it does mean my phone buzzes when pinged even when on my laptop :P | 00:22 |
| tdasilva_ | mattoliverau: what's the cost of running a vm for that on a rackspace cloud? | 00:22 |
| notmyname | tdasilva_: get a bouncer! ;-) | 00:23 |
| notmyname | I use znc | 00:23 |
| tdasilva_ | notmyname: do you run it on ec2 or rackspace? | 00:24 |
| glange_ | notmyname: that looks interesting | 00:25 |
| mattoliverau | tdasilva_: I'd have to look it up. You could use my quassel vm.. but it's in an Oz region, so might be a little slow for you. | 00:26 |
| mattoliverau | tdasilva_: if you want to have a play, I can give you an account on the VM so you can set a username and password in the quassel server, the server is easy, its irc.oliver.net.au :) | 00:28 |
| notmyname | tdasilva_: I have znc running my a RAX server (only because I have credit there) | 00:29 |
| tdasilva_ | mattoliverau: I will take a look at setting up znc, but thanks for the offer! :-) | 00:30 |
| mattoliverau | tdasilva_: nps | 00:32 |
| *** dmorita has joined #openstack-swift | 00:36 | |
| openstackgerrit | Xiang Hui proposed openstack/swift: Fix getaddrinfo if dnspython is installed. https://review.openstack.org/116618 | 00:37 |
| *** physcx has quit IRC | 00:51 | |
| openstackgerrit | Xiang Hui proposed openstack/swift: Fix getaddrinfo if dnspython is installed. https://review.openstack.org/116618 | 00:52 |
| *** bkopilov has joined #openstack-swift | 00:53 | |
| *** Masahiro has joined #openstack-swift | 01:06 | |
| *** jwang__ has joined #openstack-swift | 01:11 | |
| clayg | irssi in screen on my aws tiny gets me by and by | 01:16 |
| clayg | not recommended | 01:16 |
| *** yuanz has quit IRC | 01:32 | |
| *** yuanz has joined #openstack-swift | 01:32 | |
| *** peluse__ has joined #openstack-swift | 01:32 | |
| *** addnull has joined #openstack-swift | 01:33 | |
| *** peluse_ has quit IRC | 01:35 | |
| *** bill_az has quit IRC | 01:46 | |
| *** nosnos has joined #openstack-swift | 01:52 | |
| *** haomaiwang has joined #openstack-swift | 01:59 | |
| *** kajinamit has joined #openstack-swift | 01:59 | |
| openstackgerrit | Matthew Oliver proposed openstack/swift: Add concurrent reads option to proxy https://review.openstack.org/117710 | 02:17 |
| *** Masahiro has quit IRC | 02:22 | |
| *** Masahiro has joined #openstack-swift | 02:24 | |
| *** addnull has quit IRC | 02:35 | |
| *** lpabon has quit IRC | 02:36 | |
| *** nosnos has quit IRC | 02:48 | |
| notmyname | mattoliverau: just trying to grok the high-level thing you've done with that patch | 02:50 |
| notmyname | if concurrent_reads is on, then wait concurrent_timeout (default: 0) between subsequent calls to _make_node_request | 02:51 |
| notmyname | and as soon as you get any response, then stop iterating and do cleanup and return the answer | 02:52 |
| notmyname | mattoliverau: is that right? | 02:52 |
| mattoliverau | notmyname: yup | 02:53 |
| mattoliverau | that's about right :) | 02:53 |
| mattoliverau | As soon as you get a good answer (source) | 02:54 |
| notmyname | are you using self.sources as an instance variable so that you create it in method A, modify it in calls to B, and read it back in A? | 02:58 |
| notmyname | cause it seems to work, but also seems like it might be something left over from refactoring out the new method? | 02:58 |
| openstackgerrit | Hisashi Osanai proposed openstack/swift: Enable Object Replicator's failure count in recon https://review.openstack.org/138342 | 03:00 |
| notmyname | mattoliverau: I don't mean that as a negative review at this point. I just looked at it and that was my first impression | 03:01 |
| notmyname | (the "fun" with text-based communication) | 03:01 |
| notmyname | I'm more curious, actually | 03:01 |
| mattoliverau | Yeah, so the green threads can place the reponses. So it has now become and instance variable. The GetOrHeadHandler is created per request so it should be "safe" | 03:02 |
| mattoliverau | I could dump them in results, but newest requires a return value.. I could rework it I guess :) | 03:04 |
| notmyname | mattoliverau: no, don't go do that at this point. I'm only looking at the diff one file at a time | 03:04 |
| notmyname | like I said, I'm more curious and just trying to understand | 03:04 |
| notmyname | mattoliverau: ok, next question about it... | 03:06 |
| mattoliverau | shoot | 03:06 |
| notmyname | let's just assume (for the sake of typing) that concurrent_timeout is 0 | 03:06 |
| notmyname | so you concurrently request up to replica_count nodes. so normally 3. and then only ask for more on a node_timeout basis (10 sec IIRC) | 03:07 |
| notmyname | oh. wait. I get it | 03:08 |
| notmyname | ok, so if one of the primary nodes fails fast, you'll still quickly go to a handoff. ie its the faster of however long it takes to return from .waitall() and the node_timeout | 03:08 |
| mattoliverau | notmyname: clayg noticed that if a, in this case, 4th request was sent, it would get stuck in the queue and result in a 404, whereby putting the timeout in once the queue is full will hopefully elievate this.. this probably needs more testing. | 03:09 |
| notmyname | point is, if a primary is offline (drive or server), then it will very quickly ask a handoff as well. and actually, it just keeps replica_count connections going at any one time | 03:09 |
| notmyname | ah ok | 03:09 |
| mattoliverau | yup | 03:10 |
| notmyname | and in the case of defaults and a missing file, it will pretty much ask for 3, 3, 3 (primary + 2*primary handoffs = 9). so 3 batches of 3 in a healthy cluster | 03:11 |
| mattoliverau | yeah, depending on the speed of the reponse, but yeah due to node_time, should be batches of 3, which is better then batches of 1 :P | 03:12 |
| notmyname | "better" ;-) | 03:12 |
| mattoliverau | we could increase the window to be replicas *2 or something | 03:13 |
| mattoliverau | but its replicas atm | 03:13 |
| notmyname | replicas makes sense to me | 03:13 |
| notmyname | but I have zero numbers to back up one way over another | 03:14 |
| mattoliverau | Yeah me too, can always make it tunable if thats what the people want later ;) | 03:14 |
| notmyname | less tunable == better (n general) | 03:14 |
| mattoliverau | Or we could implement the TCP windowing protocal and wait to lost requests.. how could that fail :P | 03:15 |
| mattoliverau | *s/to/for | 03:15 |
| notmyname | why don't we just use raw sockets and have our own protocol. it could be so much more efficient! | 03:15 |
| mattoliverau | lol, perfect, I'll get on that :P | 03:16 |
| clayg | replica count is the correct answer, but I don't think the requests end up being as batchy as you might imagine, and the 404 node_timeout thing was really about making subsequent requests after you select your source, the async pile always limited you to having replica count in fligth at one time | 03:16 |
| notmyname | I hadn't sussed out the 404 edge case you're talking about yet. you're way ahead of me | 03:17 |
| clayg | notmyname: i think matt already fixed it | 03:17 |
| mattoliverau | clayg has spoken... do you guys ever sleep? | 03:17 |
| notmyname | ok, way way ahead of me ;-) | 03:17 |
| notmyname | mattoliverau: heh, I'm at home | 03:18 |
| notmyname | clayg: and your family just left my house, so you should head that way too ;-) | 03:18 |
| notmyname | mattoliverau: I'm just happy to be in front of IRC at some time when we're both awake and I'm not in some meeting ;-) | 03:18 |
| notmyname | mattoliverau: it's only about 7:20pm here | 03:19 |
| mattoliverau | notmyname: same here :) but LCA is coming up soon, so we can add beer to the mix ;) | 03:19 |
| notmyname | what could go wrong?! | 03:19 |
| mattoliverau | we solve all the things! | 03:19 |
| mattoliverau | clayg: I've been working on a baked in container sharding approach (cause I didn't believe in holidays apparanetly) based on what we talked about at summit, once I flesh it out some more I'd love if you could take a gander to point and laugh and my failings :) | 03:21 |
| clayg | heh | 03:21 |
| clayg | mattoliverau: maybe a spec would better for hashing out some of the highlevel stuff, unless you just prefer to type up the code and see what sticks... did your reverse listing thing ever merge? I know everyone was pretty down on the filter by dates and content-types and such? | 03:22 |
| clayg | mattoliverau: I think it was notmyname that said that things like the versioned objects middleware could acctually use reverse listings with limits as an optimization... | 03:23 |
| *** gyee has quit IRC | 03:25 | |
| mattoliverau | clayg: yeah, I have a braindump I plan to turn into a spec :) but I also like hacking to see if I'm talking nonsense :P | 03:25 |
| mattoliverau | the reverse listings is good (?), but not merged. Sitting there without reviews, I'd better poke it and make sure is actaully still works. The other releted patch was just extra stuff that people mentioned would be good in passing in reivews... but aren't indexed, so I think that change can probably go bye byes, I think it's soon to be in the abandoned list if it isn't already :) | 03:28 |
| clayg | mattoliverau: well wtf has the reverse listing been showing up on my review dashboard... | 03:28 |
| notmyname | /kidswerescreaminginthebathtub | 03:29 |
| notmyname | /wifewasyellingformetoo | 03:29 |
| notmyname | clayg: ya, I want to play with the dashboard tools some to see what I can do to make a better one :/ | 03:30 |
| mattoliverau | clayg: its a magic change apparently, it's quite old, I might rebase it which might bring it more life :) | 03:30 |
| notmyname | mattoliverau: http://littlemexico.co.nz advertises "the largest collection of tequila in Auckland" | 03:30 |
| clayg | omg i still have kota's patch on my list too! | 03:30 |
| mattoliverau | notmyname: nice, sounds like a place that should be visited ;) | 03:31 |
| clayg | a'ight i'm offline for a bit | 03:31 |
| mattoliverau | notmyname: mikal has organised an openstack ATC meetup on the Sunday before LCA (I probably wont arrive in NZ until sunday arvo I think [still haven't got my flights]) | 03:32 |
| mattoliverau | clayg: night.. unless you come back :) | 03:33 |
| *** tellesnobrega_ has quit IRC | 03:33 | |
| *** sungju has quit IRC | 03:34 | |
| clayg | mattoliverau: if concurrent reads and inflight > nodes: concurreny else node maybe? | 03:35 |
| clayg | idk either way is probably fine | 03:35 |
| mattoliverau | oh yeah, though inflight < nodes would work better :P | 03:37 |
| *** Masahiro has quit IRC | 03:38 | |
| notmyname | mattoliverau: ya, I land sunday early am, and I registered for the meetup. I likely go | 03:43 |
| notmyname | *I'll likely go | 03:44 |
| *** Masahiro has joined #openstack-swift | 03:44 | |
| mattoliverau | I will too, as I have registered for it, but I may be late.. depending on flights | 03:45 |
| notmyname | mattoliverau: nice. that restaurant is right next to the uni and walking distance from my hotel | 03:45 |
| notmyname | mattoliverau: are you staying on campus or on a hotel? | 03:46 |
| notmyname | I opted for a hotel this time | 03:46 |
| mattoliverau | notmyname: I think us Aussie rackers are planning on staying at some serviced apartments somewhere close by. So the hotel route as well :) | 03:47 |
| notmyname | nice | 03:47 |
| notmyname | although I think part of the "charm" of LCA is the dorm housing, after 2 years I figured I'll get a hotel and have my own bathroom this time :-) | 03:48 |
| mattoliverau | notmyname: lol, I totally agree, out of the last 6 years of LCA I've been too, I've done hotel only once, cause the dorms are fun. :) | 03:49 |
| *** Masahiro has quit IRC | 03:49 | |
| notmyname | mattoliverau: thanks for helping me understand the high-level of the concurrent reads. and thanks for working on it | 03:51 |
| openstackgerrit | Jeremy Stanley proposed openstack/python-swiftclient: Workflow documentation is now in infra-manual https://review.openstack.org/139384 | 03:51 |
| openstackgerrit | Jeremy Stanley proposed openstack/swift: Workflow documentation is now in infra-manual https://review.openstack.org/139393 | 03:52 |
| openstackgerrit | Jeremy Stanley proposed openstack/swift-specs: Workflow documentation is now in infra-manual https://review.openstack.org/139394 | 03:52 |
| mattoliverau | notmyname: my pleasure, it was fun and was interesting to work with a more core part of the swift code :) | 03:52 |
| notmyname | I'm out for a while. good night | 03:53 |
| mattoliverau | Looks like fungi is patching infra-manual which touches our docs :) | 03:53 |
| mattoliverau | notmyname: kk, night, thanks for keeping me company in channel :) | 03:54 |
| *** addnull has joined #openstack-swift | 03:58 | |
| *** Masahiro has joined #openstack-swift | 03:59 | |
| *** ppai has joined #openstack-swift | 04:11 | |
| *** ahonda has quit IRC | 04:15 | |
| *** ahonda has joined #openstack-swift | 04:22 | |
| openstackgerrit | Matthew Oliver proposed openstack/swift: Workflow documentation is now in infra-manual https://review.openstack.org/139393 | 04:30 |
| openstackgerrit | Matthew Oliver proposed openstack/python-swiftclient: Workflow documentation is now in infra-manual https://review.openstack.org/139384 | 04:33 |
| mattoliverau | ^^ Fixed up the url link and some documentation on behalf of fungi | 04:34 |
| *** sungju has joined #openstack-swift | 04:48 | |
| *** addnull has quit IRC | 04:57 | |
| *** addnull has joined #openstack-swift | 05:12 | |
| *** Masahiro has quit IRC | 05:12 | |
| *** Masahiro has joined #openstack-swift | 05:27 | |
| *** kopparam has joined #openstack-swift | 05:37 | |
| mattoliverau | Right, well it's friday arvo, so I'm going to take an early mark, have a great weekend all. | 05:37 |
| *** addnull has quit IRC | 05:49 | |
| openstackgerrit | Hisashi Osanai proposed openstack/swift: Fix the behavior of swift-ring-builder list_parts before rebalance https://review.openstack.org/139523 | 06:00 |
| openstackgerrit | Hisashi Osanai proposed openstack/swift: Fix the behavior of swift-ring-builder list_parts before rebalance https://review.openstack.org/139523 | 06:09 |
| *** kopparam has quit IRC | 06:28 | |
| *** kopparam has joined #openstack-swift | 06:30 | |
| *** Masahiro has quit IRC | 06:35 | |
| *** Masahiro has joined #openstack-swift | 06:39 | |
| *** reed has quit IRC | 06:39 | |
| *** addnull has joined #openstack-swift | 06:40 | |
| *** sungju has quit IRC | 06:41 | |
| *** openstackgerrit has quit IRC | 06:49 | |
| *** openstackgerrit has joined #openstack-swift | 06:49 | |
| *** zul has quit IRC | 07:14 | |
| *** addnull has quit IRC | 07:20 | |
| *** addnull has joined #openstack-swift | 07:26 | |
| *** SkyRocknRoll has joined #openstack-swift | 07:44 | |
| *** SkyRocknRoll has joined #openstack-swift | 07:44 | |
| *** SkyRocknRoll has quit IRC | 07:48 | |
| *** SkyRocknRoll has joined #openstack-swift | 07:48 | |
| *** SkyRocknRoll has joined #openstack-swift | 07:48 | |
| *** tellesnobrega_ has joined #openstack-swift | 08:00 | |
| *** tellesnobrega_ has quit IRC | 08:04 | |
| *** zul has joined #openstack-swift | 08:07 | |
| *** rledisez has joined #openstack-swift | 08:18 | |
| *** bkopilov has quit IRC | 08:20 | |
| *** bkopilov has joined #openstack-swift | 08:22 | |
| *** Masahiro has quit IRC | 08:36 | |
| *** kopparam has quit IRC | 08:39 | |
| *** oomichi has quit IRC | 08:41 | |
| *** Masahiro has joined #openstack-swift | 08:47 | |
| openstackgerrit | Hisashi Osanai proposed openstack/swift: Fix the behavior of swift-ring-builder list_parts before rebalance https://review.openstack.org/139523 | 08:52 |
| *** ppai has quit IRC | 08:53 | |
| *** jordanP has joined #openstack-swift | 09:00 | |
| *** sluo_wfh has quit IRC | 09:02 | |
| *** kopparam has joined #openstack-swift | 09:05 | |
| *** sluo_wfh has joined #openstack-swift | 09:14 | |
| *** jistr has joined #openstack-swift | 09:14 | |
| *** sluo_wfh has quit IRC | 09:20 | |
| *** addnull has quit IRC | 09:29 | |
| *** dmorita has quit IRC | 09:32 | |
| *** tellesnobrega_ has joined #openstack-swift | 09:35 | |
| *** foexle has joined #openstack-swift | 09:45 | |
| *** tellesnobrega_ has quit IRC | 10:00 | |
| *** kopparam has quit IRC | 10:04 | |
| openstackgerrit | Merged openstack/python-swiftclient: Workflow documentation is now in infra-manual https://review.openstack.org/139384 | 10:12 |
| *** aix has joined #openstack-swift | 10:16 | |
| *** acoles_away is now known as acoles | 10:19 | |
| *** Masahiro has quit IRC | 10:33 | |
| *** haomaiwang has quit IRC | 10:37 | |
| *** kopparam has joined #openstack-swift | 10:56 | |
| *** Masahiro has joined #openstack-swift | 11:34 | |
| *** Masahiro has quit IRC | 11:39 | |
| *** sandywalsh has quit IRC | 12:02 | |
| *** sandywalsh has joined #openstack-swift | 12:05 | |
| *** kopparam has quit IRC | 12:14 | |
| *** dmsimard_away is now known as dmsimard | 12:18 | |
| *** mahatic has joined #openstack-swift | 12:24 | |
| *** mahatic has quit IRC | 12:25 | |
| *** mahatic has joined #openstack-swift | 12:28 | |
| *** ahonda has quit IRC | 12:29 | |
| *** kopparam has joined #openstack-swift | 12:42 | |
| *** SkyRocknRoll has quit IRC | 12:53 | |
| *** Masahiro has joined #openstack-swift | 13:08 | |
| *** Masahiro has quit IRC | 13:13 | |
| mandarine | Good morning | 13:24 |
| mandarine | I think I have a question about "partition power" when creating the ring files | 13:25 |
| mandarine | I know of http://rackerlabs.github.io/swift-ppc/ to calculate the right number ... BUT how comes it doesn't depend on the size of my drives ? | 13:25 |
| mandarine | I have 8 drives and ~3.5TB per device. | 13:26 |
| mandarine | I am about to put a partition power to 20 ... but I'm not even sure | 13:29 |
| ahale | morning, partition power is more about prtitions of the md5 hash space than disk partition size | 13:31 |
| mandarine | Yeah, but it's used as a unit for replication and I'm not much into rsyncing 5Go each time I have to do a partition replica ? | 13:36 |
| mandarine | GB* | 13:37 |
| *** miqui_ has joined #openstack-swift | 14:04 | |
| tdasilva_ | mandarine: I believe that's where the device's weight comes in, check out this link: http://docs.openstack.org/developer/swift/overview_ring.html | 14:11 |
| mandarine | The weight is relative to the number of partitions on a specific device. But the number of partition of the cluster is defined while creating the ring.builder file with the <part_power> argument | 14:15 |
| mandarine | I shall put a 20 in here, even if I'm not quite sure. | 14:15 |
| openstackgerrit | Thiago da Silva proposed openstack/swift: versioned writes middleware https://review.openstack.org/134347 | 14:16 |
| jbonjean | Could someone tell me if this patch would some side effects http://pastebin.com/4nvaRvL8 ? Basically it makes the proxy treat connection timeouts as errors | 14:47 |
| jbonjean | I don't understand why it is not the default behaviour | 14:48 |
| *** gfidente has joined #openstack-swift | 14:54 | |
| gfidente | hi guys, pardon the stupid question | 14:54 |
| gfidente | is there something I can use to monitor the status of a cluster as in how many nodes are making up a cluster, which service is running on which node or things like that? | 14:55 |
| gfidente | (something bundled with swift I mean) | 14:55 |
| *** Masahiro has joined #openstack-swift | 14:57 | |
| *** Masahiro has quit IRC | 15:01 | |
| swifterdarrell | notmyname: fyi https://bugs.launchpad.net/python-swiftclient/+bug/1399678 | 15:30 |
| glange_ | gfidente: nothing like that bundled with swift | 15:32 |
| gfidente | glange_, I am seeing something interesting with swift-ring-builder | 15:33 |
| gfidente | but it is not good for monitoring I suppose | 15:33 |
| *** reed has joined #openstack-swift | 15:34 | |
| glange_ | with the ring builder you are "telling" swift about the systems that make up a cluster | 15:35 |
| gfidente | yeah but it can show that too | 15:40 |
| gfidente | the nodes making up a ring at least | 15:41 |
| glange_ | true | 15:42 |
| ahale | recon is probably the closest thing in swift core to something that returns info about nodes in the cluster | 15:43 |
| gfidente | I'll check that too, thanks | 15:44 |
| *** mahatic has quit IRC | 15:46 | |
| *** mahatic_ has joined #openstack-swift | 15:46 | |
| notmyname | good morning | 16:07 |
| mahatic_ | good morning | 16:11 |
| *** rdaly2 has joined #openstack-swift | 16:23 | |
| notmyname | mahatic_: how's it going? do you have any questions about what you're working on? | 16:31 |
| mahatic_ | notmyname, hey. I was not working for the past couple of days. Had to wrap up some things at work. I'm done with that now. So back here :) | 16:34 |
| notmyname | mahatic_: great! | 16:35 |
| notmyname | mahatic_: do you want to schedule an official "office hours" time every week, or would you prefer to leave it ad-hoc for now? what will work best for you? | 16:37 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-swiftclient: Updated from global requirements https://review.openstack.org/89250 | 16:40 |
| *** mahatic_ has quit IRC | 16:41 | |
| *** mahatic_ has joined #openstack-swift | 16:41 | |
| *** Masahiro has joined #openstack-swift | 16:46 | |
| *** Masahiro has quit IRC | 16:50 | |
| *** sfineberg has joined #openstack-swift | 17:03 | |
| *** erlon has quit IRC | 17:09 | |
| *** rledisez has quit IRC | 17:18 | |
| *** mahatic_ has quit IRC | 17:23 | |
| *** mahatic_ has joined #openstack-swift | 17:23 | |
| *** pberis has joined #openstack-swift | 17:23 | |
| *** mahatic_ has quit IRC | 17:23 | |
| *** mahatic_ has joined #openstack-swift | 17:24 | |
| mahatic_ | notmyname, the coming up Wednesday, 10th, I have a plan to travel that night to my hometown, is it possible we shift the 1 hour sometime in the week? | 17:25 |
| notmyname | mahatic_: yes. if we don't talk before then, I'll find you later | 17:29 |
| mahatic_ | notmyname, alright, you'll find me before and after :) | 17:30 |
| *** jordanP has quit IRC | 17:30 | |
| *** foexle has quit IRC | 17:35 | |
| *** jistr has quit IRC | 17:45 | |
| *** tellesnobrega_ has joined #openstack-swift | 17:54 | |
| *** david-lyle has joined #openstack-swift | 17:57 | |
| *** tellesnobrega_ has quit IRC | 17:59 | |
| ctennis | Is there a reason the replicators scan over all suffixes and rsync them by individual name vs. just rsyncing the partition directory and letting it grab the suffixes as part of the sync process? | 18:11 |
| notmyname | ctennis: there can be a lot of objects in a partition (and suffix) directory, so syncing suffix dirs allows dividing up the work into smaller chunks | 18:14 |
| ctennis | ok, I don't see where it actually divides it up though | 18:15 |
| peluse__ | ctennis, look un update() where the suffixes list is created based on differnces in hashes.pkl files, just the subset that different is sync | 18:16 |
| peluse__ | ctennis, FYI I'm in the middle of doing the EC reconstructor and picking these things apart so its fresh if you wanna walk through any of it w/me :) | 18:17 |
| ctennis | ok, I see it in update now..looks like update_deleted doesn't use that though? | 18:17 |
| peluse__ | udpate_deleted moves partitions that don't belong on the current node (not individual suffix dirs) | 18:18 |
| ctennis | at least, it just uses the whole suffix list I mean | 18:18 |
| peluse__ | so you're reading it correctly... when moving data is part by part, when sync'ing between nodes that are supposed to have the same copy its suffix dir by suffix dir | 18:19 |
| ctennis | peluse__: right..however, if we're going to sync the whole partition and could avoid looping over all of the suffixes, it would help performance there | 18:19 |
| peluse__ | ctennis, which loop are you looking at? | 18:19 |
| ctennis | peluse__: I'm diving into this a bit after fighting a customer cluster who had way out of whack replication | 18:19 |
| ctennis | peluse__: "tpool_get_suffixes" | 18:20 |
| notmyname | line 189 I think | 18:20 |
| peluse__ | I'm looking at my EC branch, which file/function? | 18:20 |
| peluse__ | ahh, thats not a loop. that's trying to avoid moving stuff that's not a suffix dir | 18:21 |
| peluse__ | well, its a list comprehensision so yeah its proc time... | 18:21 |
| ctennis | peluse__: I know, but os.listdir(path) can be very expensive on a burdened system | 18:21 |
| peluse__ | one sec... | 18:21 |
| *** kopparam has quit IRC | 18:21 | |
| notmyname | ctennis: "is very expensive." FTFY ;-) | 18:22 |
| *** kopparam has joined #openstack-swift | 18:22 | |
| ctennis | On this one system, where there should be about 16k partitions on the node spread across 36 disks there were almost 600,000. the replicator would take almost 6 hours just to startup. | 18:22 |
| ctennis | I know that doesn't impact this particular place, it just has me looking for some places to improve the cycle times in general | 18:23 |
| *** gyee has joined #openstack-swift | 18:23 | |
| peluse__ | heh, so I've changed some things on the branch I'm looking at. let me pull up master and refresh my memory on a few things... give me a couple of mins | 18:24 |
| peluse__ | ctennis, and your customer is using rsync right? | 18:24 |
| ctennis | peluse__: yeah | 18:25 |
| *** gyee has quit IRC | 18:25 | |
| *** gyee has joined #openstack-swift | 18:25 | |
| *** rdaly2 has quit IRC | 18:27 | |
| *** rdaly2_ has joined #openstack-swift | 18:27 | |
| peluse__ | ctennis, so yeah if you look at rsync() it takes a suffix list that it loops on which is needed for how update() works so that's likely why update_delete() is building a list and passing it in. Seems like there could be an oppt to have update_delete() use the part dir instead with not too many tweaks | 18:27 |
| peluse__ | opportunity that is (not sure where oppt came from) | 18:28 |
| ctennis | ok sounds good peluse__ - I'm going to try and see if there's an real gain or advantage to doing it that way, I would think rsync may be a little more optimal by specifying one source argument vs potentially hundreds. | 18:28 |
| ctennis | but certainly I think we'd benefit by not doing the os.listdir if we can help it there | 18:29 |
| peluse__ | ctennis, certainly worth looking into w/what I know about it at least :) The EC stuff will always use ssync so I may jot down a note to look clser there once the rest of it is working... cool | 18:29 |
| *** peluse__ is now known as peluse | 18:31 | |
| *** ChanServ sets mode: +v peluse | 18:31 | |
| *** kopparam has quit IRC | 18:33 | |
| *** Masahiro has joined #openstack-swift | 18:35 | |
| *** gyee has quit IRC | 18:35 | |
| *** gyee has joined #openstack-swift | 18:38 | |
| *** gyee has quit IRC | 18:38 | |
| *** gyee has joined #openstack-swift | 18:39 | |
| *** Masahiro has quit IRC | 18:39 | |
| openstackgerrit | Alistair Coles proposed openstack/swift: Update container on fast-POST https://review.openstack.org/135380 | 18:53 |
| openstackgerrit | Alistair Coles proposed openstack/swift: Make ssync compatible with fast-post meta files https://review.openstack.org/138498 | 18:53 |
| acoles | clayg: ^^ mainly backfilling tests for ssync patch, next i will add the triple timestamp stuff | 18:54 |
| *** acoles is now known as acoles_away | 18:58 | |
| *** rdaly2_ has quit IRC | 18:58 | |
| *** lpabon has joined #openstack-swift | 19:00 | |
| *** nshaikh has joined #openstack-swift | 19:01 | |
| *** gfidente has quit IRC | 19:10 | |
| peluse | clayg, I'm taking off for a liquid lunch but FYI the changes we talked about are looking pretty cool... excited to spend some time this weekend or Mon on them... | 19:10 |
| *** tellesnobrega has quit IRC | 19:29 | |
| openstackgerrit | Thiago da Silva proposed openstack/swift: versioned writes middleware https://review.openstack.org/134347 | 19:32 |
| *** gyee has quit IRC | 19:34 | |
| *** zaitcev has joined #openstack-swift | 19:44 | |
| *** ChanServ sets mode: +v zaitcev | 19:44 | |
| clayg | peluse: acoles_away: ok, i'll take a look as soon as I can - maybe torgomatic could peek too | 19:49 |
| *** gordc has joined #openstack-swift | 20:07 | |
| gordc | had a random q about proxy server... in icehouse, the etc/swift/proxy-server.conf has a pipeline like this: http://logs.openstack.org/42/130142/5/check/check-tempest-dsvm-full/e732515/logs/etc/swift/proxy-server.conf.txt.gz but for some reason, in actual proxy log, it's noticeably different: http://logs.openstack.org/42/130142/5/check/check-tempest-dsvm-full/e732515/logs/screen-s-proxy.txt.gz | 20:09 |
| gordc | is the pipeline not taken from proxy-server.conf in icehouse? | 20:09 |
| notmyname | gordc: it would seem that there is some other config file that it was actually started with. | 20:14 |
| notmyname | but let me check one thing | 20:14 |
| gordc | notmyname: cool cool. i could just be misunderstanding it and the logging is just renaming middlewares when it prints | 20:16 |
| notmyname | gordc: ya, that's what I was checking. that is possible | 20:16 |
| openstackgerrit | Jesse J. Cook proposed openstack/python-swiftclient: Do not override specified URL or token https://review.openstack.org/139728 | 20:19 |
| notmyname | gordc: I think I know what's happening. testing | 20:20 |
| *** lpabon has quit IRC | 20:21 | |
| gordc | notmyname: awesome. thanks for looking at it. | 20:21 |
| *** Masahiro has joined #openstack-swift | 20:23 | |
| notmyname | gordc: nope. not what I thought | 20:24 |
| notmyname | but yeah, it seems like it might be the names getting printed differently. but I couldn't duplicate that in my saio | 20:25 |
| gordc | :( i'll try debugging it. just seemed really strange that they were different. | 20:26 |
| *** nshaikh has left #openstack-swift | 20:26 | |
| notmyname | gordc: the changes are cache->memcache, keystoneauth->big, ceilometer->swift, and proxy-server->proxy | 20:26 |
| *** Masahiro has quit IRC | 20:27 | |
| gordc | notmyname: hmm.. yeah i'll take a look and see why those changed... | 20:28 |
| *** NobodyCam has joined #openstack-swift | 20:28 | |
| openstackgerrit | Merged openstack/swift: Fix the behavior of swift-ring-builder list_parts before rebalance https://review.openstack.org/139523 | 20:29 |
| NobodyCam | good afternoon all, I have a question: can I set Temp-url-key in a conf file vs the "post -m" example | 20:30 |
| notmyname | NobodyCam: no, it's stored on the account metadata (2 keys actually, so you can do rotations) | 20:30 |
| NobodyCam | :) thank you :) | 20:30 |
| *** rdaly2 has joined #openstack-swift | 20:38 | |
| *** tellesnobrega_ has joined #openstack-swift | 20:43 | |
| *** tellesnobrega_ has quit IRC | 20:43 | |
| *** cdelatte has joined #openstack-swift | 21:05 | |
| *** cdelatte has quit IRC | 21:28 | |
| clayg | mattoliverau: latest changeset on concurrent reads looks pretty good! I think I'll probably have a diff on the unittests, but you may have the right idea. | 21:33 |
| clayg | torgomatic: you were asking about fun ones? https://review.openstack.org/#/c/117710/14 | 21:33 |
| clayg | tdasilva_: thanks for doing all those reviews as of late! I had remembered talking to notmyname that one usecase of reverse container listings would be something like object versions, so when I remembered I've been needing to look at https://review.openstack.org/#/c/120709/ I thought of you too! | 21:36 |
| *** kopparam has joined #openstack-swift | 21:57 | |
| *** gordc has left #openstack-swift | 21:57 | |
| *** gyee has joined #openstack-swift | 22:03 | |
| *** rdaly2 has quit IRC | 22:04 | |
| *** Masahiro has joined #openstack-swift | 22:12 | |
| mattoliverau | clayg: cool, thanks I look forward to seeing the diff :) | 22:15 |
| *** Masahiro has quit IRC | 22:17 | |
| clayg | mattoliverau: don't hold your breath/you should badger torgomatic for a review | 22:26 |
| torgomatic | badger badger badger xattr | 22:29 |
| *** kopparam has quit IRC | 22:29 | |
| mattoliverau | torgomatic: badger badger badger concurrent reads :p | 22:31 |
| torgomatic | badgebabadgeradgrer | 22:31 |
| notmyname | snake! | 22:31 |
| mattoliverau | Lol, you tube has ruined us all :p | 22:33 |
| notmyname | mattoliverau: did you see that youtube had a video that broke the 32 bit limit for the views counter? | 22:34 |
| mattoliverau | Haha, really! Awesome, geeky awesome, but awesome none the less ;p | 22:36 |
| clayg | notmyname: I thought that was sort of a myth, that they went to 64bit ints long before psy's video ever acctually rolled over? | 22:38 |
| notmyname | don't confuse us with facts! | 22:38 |
| notmyname | actually I have no idea | 22:38 |
| notmyname | saw it somewhere last week maybe? | 22:39 |
| notmyname | clayg: nat according to BBC (and the liked official youtub google+ page) http://www.bbc.com/news/world-asia-30288542 | 22:40 |
| notmyname | "a couple months ago" is when they fixed it | 22:41 |
| notmyname | wow. doesn't look like I can type | 22:41 |
| notmyname | *the *not *linked | 22:41 |
| clayg | oh yeah the bbc knows | 22:41 |
| mattoliverau | Lol | 22:41 |
| mattoliverau | Sorry for slow responses, is Saturday and I'm in the middle of an archery field. Yay tech | 22:42 |
| notmyname | mattoliverau: maybe not the "middle"? | 22:42 |
| clayg | torgomatic: can more than one policy use the same instance of an obj_ring? I guess you could copy object to object-1 if you really wanted the same nodes, but under the hood I think the StoragePolicyCollection is going to have two different Ring instances in hand... | 22:43 |
| notmyname | cool! I figured out how to set my calendar to have the right time for the swift team meetings, even with the time change shenanigans | 22:43 |
| torgomatic | clayg: yeah, they'll be different instances | 22:43 |
| notmyname | if your calendar doesn't allow setting an event in UTC, then set it to Icelandic time. they are GMT+0 and don't do a time change | 22:44 |
| mattoliverau | Well not in the middle except for when collecting arrows... Which is when I ignore irc :p | 22:44 |
| notmyname | (and then subscribe to icelandic news and learn icelandic to see if they start observing some daylight savings time thing) | 22:44 |
| mattoliverau | Lol, yeah that's much easier :p I just remember its either 5am or 6am :p | 22:47 |
| clayg | notmyname: which do you think will happen first a) everyone uses metric system b) everyone uses UTC? | 22:48 |
| notmyname | UTC. who doesn't use UTC on their servers today? | 22:48 |
| notmyname | I'm mean, honestly? | 22:49 |
| clayg | yeah... god that's sad | 22:49 |
| clayg | mattoliverau: you bushmen use metric right? is the us the only hold out? | 22:49 |
| notmyname | clayg: https://twitter.com/joe_jag/status/510048646482894848 | 22:49 |
| notmyname | ^ what with the recent news of NASA sending humans to Mars... | 22:50 |
| clayg | notmyname: so is UTC the same on mars or only within the light cone? | 22:50 |
| notmyname | well, it calls itself "universal coordinated time". so I'm guessing it's got to be for everyone everywhere. it says so right there in the name! | 22:51 |
| notmyname | also, the phrase "light cone" is cool | 22:51 |
| clayg | notmyname: well maybe that's only between universes that can coordinate with entangled particles - otherwise I just don't see how it would even be possible | 22:52 |
| notmyname | heh | 22:52 |
| notmyname | clayg: ya, UTC is dependent on a 24 hour day. not true on mars | 22:52 |
| * notmyname proposes s/UTC/ETC/g | 22:52 | |
| mattoliverau | Yup, metric all the way, makes life so much easier! | 22:52 |
| notmyname | heh | 22:54 |
| notmyname | "This abbreviation arose from a desire by the International Telecommunication Union and the International Astronomical Union to use the same abbreviation in all languages. English speakers originally proposed CUT (for "coordinated universal time"), while French speakers proposed TUC (for "temps universel coordonné"). The compromise that emerged was UTC" | 22:54 |
| *** tkay has joined #openstack-swift | 22:59 | |
| torgomatic | anyone have a system with a weird filesystem on it that can test something for me? like ZFS, UFS2... basically not XFS and not ext4 | 23:02 |
| torgomatic | basically, run this https://gist.github.com/smerritt/a415008402d82b03ceb4 and tell me what it says | 23:03 |
| openstackgerrit | Merged openstack/swift: Workflow documentation is now in infra-manual https://review.openstack.org/139393 | 23:05 |
| torgomatic | notmyname: don't you have some ZFS thingy around somewhere? | 23:05 |
| notmyname | torgomatic: running it now on that | 23:05 |
| torgomatic | :) | 23:05 |
| *** dmsimard is now known as dmsimard_away | 23:07 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift: ec get for the lolz https://review.openstack.org/139752 | 23:07 |
| notmyname | torgomatic: I just gotta figure out the best way to get xattr installed on open illumos | 23:07 |
| torgomatic | notmyname: ooh... yeah | 23:07 |
| notmyname | torgomatic: hey remember that time (yesterday) when clayg was talking about how hard EC get would be and it would take a long time for him to submit a patch about it? | 23:08 |
| torgomatic | notmyname: no, the gin helps with that | 23:08 |
| *** quack_quack_ has joined #openstack-swift | 23:08 | |
| clayg | notmyname: this one is only for the lolz | 23:08 |
| clayg | but the md5 matched so I say ship it! | 23:08 |
| clayg | where ship it ~= type git review and get read for the party tonight! | 23:09 |
| clayg | whoot whoot! | 23:09 |
| notmyname | torgomatic: I'm going to file this under "No". xattr depends on cffi, which means I'd have to figure out how to build that on this crazy old system | 23:15 |
| torgomatic | notmyname: fair enough. thanks for trying! | 23:16 |
| notmyname | torgomatic: FWIW, on OSX (which I'm sure you already "tested"), FS supports really big extended attributes (over 1 MiB) | 23:17 |
| notmyname | briancline: do you have any ZFS boxes around? to test out torgomatic's thing? | 23:17 |
| notmyname | torgomatic: what have you tested so far? | 23:19 |
| torgomatic | notmyname: got XFS and ext4. I have a vagrant freebsd box attempting to come up, but the base box download is slooooooowwwww | 23:20 |
| torgomatic | so that'll get me ZFS, I believe | 23:20 |
| torgomatic | I figure that XFS, ZFS, ext4, and UFS2 are about all anyone will really care about, and I'm not sure about UFS2 | 23:21 |
| * notmyname gets nervous when asked "How many lines of code make up Swift?" | 23:23 | |
| notmyname | "enough"? | 23:23 |
| notmyname | I wonder if the only reason people track line count is because all our editors have the number conveniently printed next to every line. so someone looked over a dev's shoulder at one point and though it was a god thing to track | 23:26 |
| torgomatic | well, you can't count quality, but you can count lines | 23:31 |
| notmyname | and more importantly, you can count how many change | 23:32 |
| *** mahatic_ has quit IRC | 23:46 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!