Monday, 2015-02-16

*** kragniz_ is now known as kragniz00:00
notmynamemattoliverau: jrichli_ I just found something interesting that is probably pretty small that can be looking into00:14
notmyname*looked00:14
jrichli_notmyname: sounds great, what is it?00:15
notmynamea weird, and I think extraneous we're doing00:15
notmynameso reset your SAIO, then get an auth token from tempauth00:15
notmynamethat's it00:15
notmynamenow look at your logs00:15
notmynameyou'll see one like this:00:15
notmynameFeb 16 00:10:31 saio swift_proxy: - - 16/Feb/2015/00/10/31 HEAD /v1/v1.0 HTTP/1.0 204 - Swift - - - - tx88f2881a8c994748aeb84-0054e13577 Host:%20saio:8080%0AUser-Agent:%20Swift%0AX-Trans-Id:%20tx88f2881a8c994748aeb84-0054e13577 0.0151 GET_INFO - 1424045431.786942005 1424045431.802002907 -00:16
notmynameso there's 2 things there00:16
notmynamefirst, the GET_INFO swift source isn't documented on http://docs.openstack.org/developer/swift/logs.html00:16
notmynamebut, more interesting is why are we making a HEAD request to "/v1/v1.0"???!00:16
notmynamethat's the odd part00:17
jrichli_ok, I will take a look.00:17
notmynameit's line 524 in proxy/controllers/base.py that's making the call00:17
notmynamebut I don't know why the path is set and we're doing that00:17
notmynamejrichli_: thanks :-)00:17
notmynameI've got to get offline for a bit (I think my family is starting to ask why I'm on my computer this afternoon :-)00:18
jrichli_np.  I will let you know what I find out.00:18
notmynamebut, for anyone else (acoles_away, daisuke when he gets online, slogging is updated to work with current swift)00:18
mattoliveraunotmyname: I'll let daisuke now when he turns up00:19
notmynamemattoliverau: thanks00:19
mattoliverausorry in a meeting00:19
notmynamehttps://github.com/notmyname/slogging  #link00:19
mattoliverauta00:19
notmynamebe back later :-)00:20
*** ho has joined #openstack-swift00:43
hogood morning guys!00:44
notmynamejrichli_: did you find anything?00:44
notmynameho: how was your trip home?00:44
honotmyname: good. I have no jet lag!00:45
notmynamethat's great!00:45
honotmyname: thanks!00:46
jrichli_notmyname: I had to do some things.  I haven't gotten far yet in the investigating.00:46
notmynamejrichli_: no worries00:46
*** ho has quit IRC00:47
*** ho has joined #openstack-swift00:51
notmynameseems that anything after the auth_prefix in the path is getting treated like a swift account and a HEAD is done00:52
jrichli_seems like it's doing the HEAD in order to set the info in the environment and in memcache (because the info was cached)01:13
jrichli_*not cached01:13
jrichli_ah, and you are saying the info wasn't found in cache because the wrong path was used?01:14
openstackgerritJohn Dickinson proposed openstack/swift: added swift_source to ratelimit info calls  https://review.openstack.org/15609501:15
notmynamejrichli_: that's one issue ^01:15
notmynamejrichli_: the swift_source wasn't set on a call to get_account_info in the ratelimit middleware01:15
jrichli_i saw that had to have been None for GET_INFO to be used01:16
notmynameright01:16
notmynamehmm...so here's the situation now01:17
notmynameit's not tempauth (although that's where I saw it first)01:17
notmynameany request to `http://saio:8080/foo/bar` will cause "Feb 16 01:17:15 saio swift_proxy: - - 16/Feb/2015/01/17/15 HEAD /v1/bar HTTP/1.0 204 - Swift - - - - txbdb842e7fec64662b79d5-0054e1451b Host:%20saio:8080%0AUser-Agent:%20Swift%0AX-Trans-Id:%20txbdb842e7fec64662b79d5-0054e1451b 0.0091 RL - 1424049435.027575016 1424049435.036645889 -"01:18
notmynamealong with the associated account server requests to up to 9 different servers (3 + 6 handoffs)01:18
notmynamewhich is the real issue01:18
notmynamewhich is coming from the ratelimit middleware (the "RL" -- with my patch)01:20
jrichli_I see.  Makes sense.  How is the swift_source chosen?  I mean, RL sounds good for something called ratelimit, but does each calling class just use any string?01:23
notmynamejrichli_: basically :-)01:23
jrichli_ok01:23
notmynamejrichli_: along with the "please consult and add yours to http://docs.openstack.org/developer/swift/ratelimit.html"01:23
notmynameI mean http://docs.openstack.org/developer/swift/logs.html01:23
jrichli_oh, yes.  good to know!01:25
jrichli_I will add your fix and run again.  Just maybe I will give you a +1!01:27
notmynamethanks.01:32
* notmyname off to cook dinner01:32
*** kajinamit has joined #openstack-swift01:34
*** kajinamit has quit IRC01:36
openstackgerritHisashi Osanai proposed openstack/swift: Enable Object Replicator's failure count in recon  https://review.openstack.org/13834201:47
*** jamielennox is now known as jamielennox|away02:19
*** jamielennox|away is now known as jamielennox02:27
*** doxavore has joined #openstack-swift02:43
*** doxavore has quit IRC02:46
*** doxavore has joined #openstack-swift02:46
*** ahonda has quit IRC02:47
*** sluo_wfh has quit IRC02:49
*** sluo_wfh has joined #openstack-swift02:50
*** sluo_wfh has quit IRC02:55
*** sluo_wfh has joined #openstack-swift02:56
*** doxavore has quit IRC03:45
*** wer_ has quit IRC04:12
*** wer has quit IRC04:12
*** wer has joined #openstack-swift04:12
*** wer_ has joined #openstack-swift04:12
*** BAKfr has quit IRC04:32
*** BAKfr has joined #openstack-swift04:32
*** ppai has joined #openstack-swift04:35
*** BAKfr has quit IRC04:37
*** SkyRocknRoll has joined #openstack-swift04:44
*** SkyRocknRoll has joined #openstack-swift04:46
*** BAKfr has joined #openstack-swift04:48
*** SkyRocknRoll has quit IRC04:52
torgomaticnotmyname: we could use a pastedeploy composite to route auth down a different popeline than non-auth05:19
torgomaticpipeline, even05:19
torgomaticalthough now I'm wondering what a popeline is05:20
notmynametorgomatic: that's not a bad idea05:20
notmynametorgomatic: it probably involves some sort of smoke05:20
notmynamethe response codes are color-coded05:20
notmynameyou have to be patient, though. latency is a killer05:20
notmynametorgomatic: is this sane? Request\.blank\('/[^(v1)]/05:21
notmynameie to match Request.blank('/a/c/o05:21
notmyname'cause I wanna change it to Request.blank('/v1/a/c/o05:22
torgomaticnotmyname: Request.blank with a regex pattern? I don't think I would know what that means05:24
torgomaticoh wait05:24
torgomaticyeah, that's probably okay05:25
notmynametorgomatic: I got it. actually jrichli_ got it, but still05:25
notmynameit wasn't ok actually05:25
notmynameRequest\.blank\('/[^(v1)]*/05:25
torgomaticother than the fact you're parsing Python with a regex, but neglecting that, it's okay05:25
notmynameheh05:25
torgomatic"Other than that, Mrs. Lincoln, how was the play?"05:25
notmynameactually no. just for grep or my editor to find stuff in the tests05:25
torgomaticah, fair enough :)05:25
*** jrichli_ has quit IRC05:36
notmynamewow. this test is relying on a bug (in the test) in order to pass05:42
mattoliverauTest... Relying on a bug.. A test! Wow05:57
notmynameanyone seen or know anything about keystone's authenticated tokens? from the ML it seems they're faster than uuid tokens05:59
notmynameand bounded in size06:00
openstackgerritMerged openstack/swift: added swift_source to ratelimit info calls  https://review.openstack.org/15609506:09
*** kota_ has joined #openstack-swift06:10
notmynamekota_: how was your trip home?06:10
kota_ya, it was good. I got very good health in Japan.06:11
notmynamegreat! glad you made it ok06:11
kota_Thanks! How was the dinner at the last day of hackathon?06:11
mattoliverauDelicious :) we went out for Thai06:13
notmynamegreat! the best part was meeting joearnold on top of a hill in the city with his binoculars and looking at jupiter, orion, the pliedes (subaru cluster), and even the andromeda galexy06:13
mattoliverau+1 to urban astronomy :)06:13
notmynamehttps://twitter.com/joearnold/status/56610143423234048006:13
notmynameit was pretty cool06:13
kota_Wow, so beautiful!06:14
notmynametorgomatic: mattoliverau: I (briefly) looked at adding a test for swift_source to that ratelimit patch that just added. looks like there's some others that test that (with FakeSwift). but test_ratelimit doesn't use FakeSwift, so getting that plumbed through is a little more difficult06:15
*** ahonda has joined #openstack-swift06:15
mattoliverauBut we got it in 1 patchset.. Don't break the record :p06:21
notmynameheh06:22
notmynametorgomatic: the reason I updated the logs doc on the fist patchset is because I'm the one who originally had to troll through the code to write that doc int he first place. the pain isn't easily forgotten ;-)06:25
openstackgerritOpenStack Proposal Bot proposed openstack/swift: Imported Translations from Transifex  https://review.openstack.org/15596706:30
honotmyname: AE tokens? thanks for the info. I will read the spec. BTW does swift have a problem like performance with keystone so far?06:40
notmynameho: generally swift doesn't have the performance problem ;-)06:40
honotmyname: i see.06:41
notmynameho: but yea, keystone has historically struggled with how to manage a large number of keys. swift, in normal operation, can use a huge number of keys, so keystone becomes a problem06:42
*** kota_ has quit IRC06:42
notmynameho: and the PKI tokens have the problem with being unbounded in size. and in general just being really big (multiple KB). and that makes things really hard when you have to send that for every swift request. especially since PKI tokens aren't cacheble06:42
honotmyname: i think so too. I have been developing policy based RBAC with keystone. if there are a lot of tokens, roles in wire increase.06:44
notmynameso all of that is why I'm pretty interested in the AE tokens they talked about.06:45
*** kota_ has joined #openstack-swift06:48
honotmyname: yeah. thanks for the info again!06:50
notmynamehere it is https://review.openstack.org/#/c/130050/06:50
*** kota_ has quit IRC06:57
notmynamegood night, all06:59
honotmyname: good night!07:00
*** SkyRocknRoll has joined #openstack-swift07:20
*** SkyRocknRoll has joined #openstack-swift07:20
*** SkyRocknRoll has quit IRC07:21
*** chlong has quit IRC07:51
*** nellysmitt has joined #openstack-swift07:53
openstackgerritMerged openstack/swift: Imported Translations from Transifex  https://review.openstack.org/15596707:58
*** jistr has joined #openstack-swift08:08
*** rledisez has joined #openstack-swift08:12
*** joeljwright has joined #openstack-swift08:12
*** silor has joined #openstack-swift08:26
*** jordanP has joined #openstack-swift08:42
*** jordanP has quit IRC08:44
*** jordanP has joined #openstack-swift08:44
*** openstackgerrit has quit IRC08:57
*** openstackgerrit has joined #openstack-swift08:57
*** jistr has quit IRC08:59
*** mmcardle has joined #openstack-swift09:01
*** madhuri_ has quit IRC09:05
donagh /server Aquarius.panicbnc.net 1337 donagh/freenode:2fe6CDf809:09
*** madhuri_ has joined #openstack-swift09:12
*** ho has quit IRC09:14
*** madhuri_ has quit IRC09:31
*** madhuri_ has joined #openstack-swift09:32
*** madhuri_ has quit IRC09:37
remix_tjdonagh: ehm.09:52
openstackgerritMerged openstack/swift: Add functional tests for container TempURLs  https://review.openstack.org/15551309:53
*** sluo_wfh has quit IRC10:04
*** ppai has quit IRC10:27
*** ppai has joined #openstack-swift10:41
*** aix has joined #openstack-swift10:44
*** miqui_ has quit IRC11:18
*** miqui has joined #openstack-swift11:18
*** anticw has quit IRC11:19
*** anticw has joined #openstack-swift11:20
*** aix has quit IRC11:27
*** mwilliams_ct has joined #openstack-swift11:35
*** mwilliams_ct has left #openstack-swift11:37
*** ppai has quit IRC12:31
*** aix has joined #openstack-swift12:32
*** silor has quit IRC12:34
*** silor has joined #openstack-swift12:35
*** ppai has joined #openstack-swift12:44
*** mahatic has joined #openstack-swift12:46
*** EmilienM is now known as EmilienM|afk13:20
*** silor has quit IRC13:28
*** silor has joined #openstack-swift13:29
*** silor1 has joined #openstack-swift13:37
*** silor has quit IRC13:40
*** sandywalsh_ has quit IRC13:52
*** tdasilva has joined #openstack-swift13:59
*** acoles_away is now known as acoles14:29
*** doxavore has joined #openstack-swift14:30
doxavoreI had another quick question on replica placement - this weekend I learned that Swift distributes evenly among failure domains of a given tier, so if I had 3 regions, each would get 1 replica. Does this happen even if the weight of the regions aren't equal?14:33
doxavoreI'm thinking of a cluster where drives/nodes are periodically needing to be removed - it seems if it takes total weight of each region into account, then it'd be nearly impossible to ensure you have 1 replica of everything in each region?14:34
*** ppai has quit IRC14:52
*** jrichli_ has joined #openstack-swift15:01
*** sweeper has joined #openstack-swift15:05
sweeperdoxavore: it uses a weighted-nearest neighbor round robin to distribute availability on a cyclomatic basis15:06
*** EmilienM|afk is now known as EmilienM15:16
*** sandywalsh has joined #openstack-swift15:47
*** dmsimard_away is now known as dmsimard15:47
*** haomaiwang has joined #openstack-swift15:47
*** nellysmitt has quit IRC15:51
*** abhirc has joined #openstack-swift15:54
*** jistr has joined #openstack-swift15:58
*** david-lyle_afk is now known as david-lyle16:01
*** rdaly2 has joined #openstack-swift16:02
*** jrichli_ has quit IRC16:09
*** jrichli_ has joined #openstack-swift16:10
*** zigo has quit IRC16:26
*** delattec has joined #openstack-swift16:30
*** zigo has joined #openstack-swift16:31
*** jistr has quit IRC16:32
*** delattec has quit IRC16:33
*** delatte has quit IRC16:33
notmynamegood morning16:40
notmynamedonagh: might want to change your password now16:41
donaghnotmyname: Yeah, I noticed right after I did it. Thanks for pointing it out (also thanks cschede: for same)16:43
notmynamedonagh: we missed you last week16:43
donaghI was tracking indirecty. Reports from Alistair and Gerry were very good.16:44
donaghs/were/are/16:44
acolesmattoliverau: donagh exists ^^ :D16:45
*** wer has quit IRC16:45
*** wer_ has quit IRC16:45
donaghacoles: I guess that's in in joke!16:45
acolesdonagh: see hipchat16:46
donaghAh16:46
acolesnotmyname: i am missing the sfo climate16:46
*** wer_ has joined #openstack-swift16:47
notmynameit was warm this weekend. about 70-75 (F)16:47
*** IRTermite has quit IRC16:47
*** wer has joined #openstack-swift16:47
notmynamemattoliverau: look at acoles talking to himself!16:47
acolesnotmyname: lol16:49
notmynameacoles: donagh: do you know anything about the authenticated encryption tokens that keystone is recently talking about?16:49
acolesnotmyname: not yet16:49
acolesnotmyname: do you have any link?16:50
notmynamehttp://dolphm.com/benchmarking-openstack-keystone-token-formats/ and https://review.openstack.org/#/c/130050/ is what I've seen16:50
acolesthx16:52
donaghnotmyname: dito (i.e., no)16:54
*** silor1 has quit IRC17:05
acolesclayg: tried devstack install. failed :( http://paste.openstack.org/show/175328/ will keep banging head against wall17:05
doxavorenotmyname: when you mentioned this weekend that swift distributes replicas "evenly" across failure domains, did you mean with or without weights? it seems it's be difficult to keep 1 copy in 3 different regions (or zones) unless they were _exactly_ the same total weight if that's how works, no?17:18
doxavoreas long as my domains were roughly the same size, it would be likely that _most_ partitions existed in all 3 regions, but if it's expecting them to be perfectly divided, but that doesn't really satisfy a requirement on making sure there's a copy of everything :-/17:22
*** mahatic has quit IRC17:25
*** tellesnobrega has quit IRC17:25
*** arnaud_o has quit IRC17:25
*** MooingLemur has quit IRC17:25
*** mtreinish has quit IRC17:25
*** Anticimex has quit IRC17:25
*** joearnold has quit IRC17:25
*** JoshNang has quit IRC17:25
*** klrmn has quit IRC17:25
*** bobby2 has quit IRC17:25
*** notmyname has quit IRC17:25
*** eikke has quit IRC17:25
*** mikehn_ has quit IRC17:25
*** portante has quit IRC17:25
*** ahale has quit IRC17:25
*** rdaly2 has quit IRC17:25
*** rdaly2 has joined #openstack-swift17:25
*** rledisez has quit IRC17:28
*** notmyname has joined #openstack-swift17:31
*** ChanServ sets mode: +v notmyname17:31
*** mahatic has joined #openstack-swift17:31
*** tellesnobrega has joined #openstack-swift17:31
*** arnaud_o has joined #openstack-swift17:31
*** klrmn has joined #openstack-swift17:31
*** MooingLemur has joined #openstack-swift17:31
*** mtreinish has joined #openstack-swift17:31
*** Anticimex has joined #openstack-swift17:31
*** joearnold has joined #openstack-swift17:31
*** JoshNang has joined #openstack-swift17:31
*** bobby2 has joined #openstack-swift17:31
*** eikke has joined #openstack-swift17:31
*** mikehn_ has joined #openstack-swift17:31
*** portante has joined #openstack-swift17:31
*** ahale has joined #openstack-swift17:31
mahaticI see. I just read the wiki page on it and checked back here, your explanation is clearly crisp :D17:31
notmynamemahatic: and now it just fixed itself17:32
mahaticnotmyname, I see. I just read the wiki page on it and checked back here, your explanation is clearly crisp :D17:32
mahaticyeah :D17:32
openstackgerritJohn Dickinson proposed openstack/swift-specs: added clarification on how to ask questions  https://review.openstack.org/15629117:33
notmynametdasilva: ^^ there17:34
tdasilvanotmyname: nice, thanks!17:36
tdasilvanotmyname, acoles: yes, really missing SFO climate. woke up to coldest morning in 11 years here with -35F for wind chill17:38
*** acoles is now known as acoles_away17:38
notmynamewow!!17:38
*** mkerrin has quit IRC17:39
*** nellysmitt has joined #openstack-swift17:40
notmynamemahatic: how's your patch progress coming?17:41
mahaticnotmyname, I wrote a test and it's throwing up an error (trying to fix it). Will push that if I don't get to resolve.17:43
notmynameok, great17:44
notmynamemahatic: I'll be out (my) thursday and friday this week. so if you have questions, please let me know sooner than later17:44
*** abhirc has quit IRC17:44
mahaticnotmyname, yeah sure. I remember that17:44
notmyname:-)17:45
notmynameFYI. last friday I had a really interesting phone call with a company that is going to open-source a tape-library connector for Swift that they've written17:49
notmynameI had several conversations with companies in Paris about this topic, so I know there are other people out there looking to do something similar17:49
remix_tjnotmyname: you say using swift as VTL?17:49
notmynameso if you or people at your company are involved in this in any way, please get in touch with me17:50
notmynameremix_tj: no, not exactly. using swift to store data on tape rather than magnetic memory. so they've taken swift's DiskFile abstraction and written one that works with their VTL. I don't know yet how generic it is or if there are any proprietary hooks (I think it's pretty general, though)17:51
notmynameremix_tj: so you can imagine a deployment that has a tape policy and a drive policy and a flash policy, all in different regions17:51
remix_tjoh, so you say using a tape drive as tier17:52
*** briancline has quit IRC17:52
*** briancline has joined #openstack-swift17:52
notmynamebriancline: TIL that .zone is a valid TLD17:54
*** mmcardle has quit IRC17:54
*** acoles_away is now known as acoles18:01
*** rdaly2 has quit IRC18:02
*** reed has joined #openstack-swift18:02
*** rdaly2 has joined #openstack-swift18:03
*** EmilienM is now known as EmilienM|afk18:07
acolestdasilva: ouch (-35)18:14
*** rdaly2 has quit IRC18:22
tdasilvaacoles: yeah, that's a bit colder than SFO :-)18:24
*** rdaly2 has joined #openstack-swift18:25
tdasilvaI wrote a hackathon trip report to be sent internally and ended up converting to a blog post, Here's the link in case anyone is interested: https://thiagodasilvablog.wordpress.com/2015/02/16/openstack-swift-mid-cycle-report/18:26
notmynametdasilva: I'm planning on going skiing next weekend. but I'm worried that there won't be enough snow18:26
notmynametdasilva: thanks for the link18:26
tdasilvaAlso feel free to send me corrections and I'll be glad to update18:27
tdasilvanotmyname: come over to NE, we got plenty of snow here :P18:27
notmynametdasilva: that's much more detailed than my blog post I rushed out on friday morning! :-)18:27
notmynametdasilva: you have Intel listed twice in the first paragraph18:29
jrichli_tdasilva: Thanks for sharing the write up.  It is a great summary!18:29
notmynameit is a great summary18:29
acolestdasilva: yeah, i was just wondering if there had been a merger with intel i hadn't heard about ;)18:29
notmynameya, he 2nd Intel is HP18:30
tdasilvaoops! thanks!18:30
notmynameacoles: surprise! that company split everyone is talking about? now it's intel :-)18:30
acolestdasilva: great write up! thanks18:31
*** Trixboxer has joined #openstack-swift18:32
notmynametdasilva: and the appropriate twitter accounts I have access to have now mentioned it :-)18:34
tdasilvaacoles, jrichli_, notmyname: thanks!18:34
acolesjrichli_: guess what, my day is gone and i didn't get round to the encryption spec updates but its on the agenda for tomorrow18:36
jrichli_acoles: np.  I have been playing catch up all day as well.  I hope to have some code to push in a few days.  We shall see.18:37
*** MasterPiece has joined #openstack-swift18:38
*** jordanP has quit IRC18:40
*** rdaly2_ has joined #openstack-swift18:43
*** rdaly2 has quit IRC18:43
*** IRTermite has joined #openstack-swift18:55
*** IRTermite has quit IRC18:57
*** IRTermite has joined #openstack-swift18:59
*** IRTermite has quit IRC19:01
*** rdaly2 has joined #openstack-swift19:03
*** rdaly2_ has quit IRC19:04
*** IRTermite has joined #openstack-swift19:07
*** acoles is now known as acoles_away19:11
*** aix has quit IRC19:11
*** IRTermite has quit IRC19:15
*** rdaly2_ has joined #openstack-swift19:23
*** rdaly2 has quit IRC19:23
sweeperok, so this ring deal. if I have two physical regions, with one zone in region B and 2 zones in region A, and I set a replication level of 3, will I end up with the expected distribution of replications (one per zone, 1-B1, 2-A1, 3-A2)?19:41
notmynamesweeper: are the zones the same size?19:42
sweepernotmyname: I'd prefer not to have to care about that19:42
notmynamehmm...actually I don't think that matters19:43
*** rdaly2_ has quit IRC19:43
notmynameI think you have to care about relative sizes. swift cannot evenly distribute data across failure domains if those failure domains don't match in capacity19:44
*** rdaly2 has joined #openstack-swift19:44
notmyname...but let's come back to that later19:44
notmynamesweeper: what will happen in you scenario is that you'll end up with one replica in on region, and 2 replicas in the other region until the smaller region is filled up. then you'll end with 3 replicas in the large region for the remainder of the capacity available19:45
notmynameI think. torgomatic, is that right?19:45
notmynameor cschwede?19:45
torgomaticeh... kind of?19:45
sweepernotmyname: mkay, what if the zone sizes are ever (or perhaps I fudge using overload?)19:46
sweepers/ever/even/19:46
torgomaticif your smaller region is exactly 1/2 of the larger region, then it's 1/3 of the total capacity, and you get each partition with 1 replica in the smaller and 2 in the larger19:46
torgomaticif it's slightly more than 1/3 of the total capacity, then you get most partitions with 1 replica in the smaller and 2 in the larger19:46
torgomaticthe remainer have 2 in the smaller and 1 in the larger19:47
torgomatic*remainder19:47
sweeperok. is there a simpler way to do this? basically I have a datacenter where we use the data, and an offsite backup location, I need two copies at the DC and one offsit19:47
notmynametorgomatic: ah. thanks for the correction19:47
sweeperit's not really acceptable to have even a few partitions with 2 replicas in the smaller region19:49
*** MasterPiece has quit IRC19:49
sweepertorgomatic: what if we overloaded the larger region?19:50
torgomaticsweeper: well, overload is a global setting, not per-device19:50
torgomaticalso I don't think it'd help19:50
torgomaticif your smaller region were less than 1/3 of the total, then normally you'd get some parts with all 3 replicas in the larger19:51
torgomaticsetting an overload would alleviate that19:51
*** abhirc has joined #openstack-swift19:51
sweepertorgomatic: so a slightly smaller backup region plus overload, gives me what I want?19:53
torgomaticsweeper: I believe so19:53
sweeperawesome, thanks19:53
torgomaticof course, your smaller region will fill faster than the larger since its disks will be proportionally fuller19:53
torgomaticso watch out for that19:53
sweeperbut it should only need to be asymptotically smaller, to force the ring builder to distribute the right way?19:54
sweepere.g. I start with 20gb in region A and 9gb in region B, then I can go to 200gb in region A and 99gb in region B, while maintaining the same availability profile?19:56
torgomaticsweeper: yes, but you'll probably want a little padding for if you lose some disks in the larger region too, since you don't ever want 2 replicas of any partition in the smaller19:57
torgomaticso like 100 disks / 50 disks is probably bad; you'd want more like 100 disks / 45 disks so you can lose a few in the larger region without upsetting the delicate balance19:57
torgomaticthat's all I'm saying19:57
sweepertorgomatic: hmm. so are we just doing this at too small of a scale? it seems like a reasonable first architecture, but one that is pretty complex to effect19:58
torgomaticsweeper: well, you're sort of fighting the ring builder here; Swift's ring builder was really designed with the idea of having a bunch of zones that data could be evenly spread across20:00
torgomaticpaying attention to disk weights (sizes), of course20:00
torgomaticit's much more about just standing up a bunch of storage in various failure domains and letting data spread around as far as it can20:01
torgomaticsomething like you're doing, where you're forcing 2 replicas on the left and 1 on the right, is sort of fighting against what the ring builder wants20:01
torgomaticit's possible, but it's a little fragile20:01
torgomaticthat said, if you want to do this with only a few disks, you can20:02
sweeperso would 3 zones be a better fit then?20:02
torgomaticjust maintain the property that sum(big-region-weights) >= 2 * sum(little-region-weights)20:02
sweeperI mean, basically we have 2 physical failure domains, that we have to spread across, and within one physical failure domain we have two network/power domains20:03
*** rdaly2_ has joined #openstack-swift20:03
torgomaticso if big-region loses a disk and you want to carve it out of the rings, you may have to fudge the other weights upwards to keep that invariant20:03
torgomaticsweeper: yeah, if you did 3 zones and set overload=999999999999999, you'd get 1 replica per zone20:04
*** rdaly2 has quit IRC20:04
*** EmilienM|afk is now known as EmilienM20:04
*** abhirc has quit IRC20:05
*** rdaly2_ has quit IRC20:06
*** rdaly2 has joined #openstack-swift20:06
sweepertorgomatic: I almost feel like there should be layers of rings? I mean I get the notion of evenly distributing data, but it feels like the ring builder throws out any notions of intra-zone availability in favor of global availability?20:07
torgomaticsweeper: yeah, the assumption is that all regions are equal, and all zones within a region are equal, and all nodes, etc etc20:09
torgomaticthis sort of primary-and-backup scenario is one of the places where that falls down20:09
sweeperwell even something like "dc in us and smaller dc in au" seems like it would be problematic20:10
torgomaticwell, only if you add in the requirement of 2 replicas in .us and 1 in .au20:11
torgomaticif you let the replicas fall where they may, then it works out okay20:11
sweeperyea I suppose20:12
sweeperhmm. I guess one could just write one's own ring-building logic...20:12
torgomatictrue; once the ring-builder writes out the .ring.gz file, it's just a static mapping20:12
torgomaticthe running Swift code knows nothing of how the partitions got that way, just that there's this big fat array to index into20:13
sweeperaight. well I don't particularly feel led to start mucking around in internals at this point, but I'm pretty happy it's not too scary an enterprise20:22
sweepertorgomatic: thanks for the help \o20:22
torgomaticno problem :)20:36
*** rdaly2 has quit IRC20:43
*** mahatic has quit IRC20:49
*** mahatic has joined #openstack-swift20:57
*** Nadeem_ has joined #openstack-swift20:58
*** devlaps has joined #openstack-swift21:01
mattoliverauMorning all21:09
notmynamehi mattoliverau21:09
mattoliverauacoles_away: I'm impressed with your ability to use 2 irc clients at once, very impressed :P21:09
notmynamelol21:09
tdasilvamattoliverau: hey! good morning21:10
mattoliverautdasilva: Morning, so your not frozen (just read scroll back)! I'm about to read your blog post :)21:10
tdasilvamattoliverau: not frozen, but you did remind me of a Frozen (the disney movie) meme I saw the other day21:16
jrichli_mattoliverau: good morning21:17
mattoliverautdasilva: you can't just dangle the mention of a meme and not share it :P21:18
tdasilvamattoliverau: looking for it21:18
mattoliveraujrichli_: Morning! You came back again, we obviously aren't as scary as we thought21:18
jrichli_so far, so good :-)21:20
*** ChanServ sets mode: +v mattoliverau21:25
tdasilvamattoliverau: some good ones here: https://www.facebook.com/MassachusettsMemes21:26
panbalaghi.I'm trying to configure swift as the backend for Glance. The glance.conf has a parameter 'swift_store_key'. Any idea where I can find this?21:32
mattoliverautdasilva: lol, if it makes you feel any better its going to be only 86 here today, so a cooler day :P21:33
notmynamepanbalag: is that something to do with auth credentials so glance can talk to swift?21:33
panbalagnotmyname, yeah21:34
notmynamepanbalag: ok. I'd guess it's something set up by or in keystone then21:34
panbalagnotmyname, ok. let me try that21:34
notmynametdasilva: can you give me a rhel/cent version of https://launchpad.net/ubuntu/+source/swift21:37
notmynamehmm...why don't we have rsync in the requirements file?21:40
notmynameI guess since it's not a python thing?21:52
doxavoreare there any recommendations for swift 2.2.2 installation on ubuntu? it doesn't appear anyone has released packages and the documentation points to Ubuntu Cloud Archive still at 2.2.0. (2.2.2 was the first time that overload setting was made available I take it?)21:55
notmynamedoxavore: installation isn't hard. or rather, there's nothing special. but if you aren't building your own packages, you'll be dependent on whatever your distro gives you21:57
notmynamemaybe I'm splitting hairs there21:57
openstackgerritJohn Dickinson proposed openstack/swift: update the getting started doc  https://review.openstack.org/15638822:00
*** os1 has joined #openstack-swift22:01
notmynamedoxavore: most deployers I know do build and manage their own packages of swift so that they can stay up to date. the distros normally only update a couple of times a year22:01
os1Hi22:01
os1Is there a reason why 'swift-init all status' would return that container-server and object-server are not running,22:01
notmynameos1: do you reason to believe that they are running?22:02
os1but doing a 'ps aux | grep swift' reveals processes for both the container-server and object-server?22:02
notmynameah ok :-)22:02
notmynameos1: probably the pid file was lost.22:02
*** aix has joined #openstack-swift22:02
os1notmyname : Could you be more specific?22:02
*** jamielennox is now known as jamielennox|away22:03
os1notmyname : Doing a 'swift-init {container-server | object-server} restart', then, should work, but it fails whenever trying to bring up one of those services.22:05
notmynameos1: the process id (pid) is written down in a file, but if that file is lost or overwritten, then swift-init doesn't know where to look to query a process.22:05
notmynameos1: why does it fail? config or maybe because there's already something bound on the port22:05
notmynameos1: you might be able to use `swift-orphans` or `swift-oldies` to help you find old processes that need to be cleaned up22:07
notmynameos1: and if you keep having trouble starting those processes, then either start them directly or look in the logs to see if there's something it's reporting22:07
os1notmyname : Okay. By 'start them directly', which utility are you referring to?22:11
os1Thank you.22:11
*** jamielennox|away is now known as jamielennox22:14
notmynameos1: `swift-init` is a tool that has some aliases and finds some config files. but ultimately, you can use eg `swift-proxy-server /etc/swift/proxy-server.conf -v`22:14
os1notmyname : I see. Thank you.22:16
notmynameanyone else going to the operators summit in Philadelphia next month?22:18
notmynameerr...wow. it's already the middle of february. so like in a few weeks22:18
*** abhirc has joined #openstack-swift22:22
mattoliverauI think I'm travelled out.. until vancouver at least :P22:23
tdasilvanotmyname: do you mean swift packages for rhel/centos ?22:29
tdasilvanotmyname: I assuming your are mentioning RDO, which you can find it here: https://repos.fedorapeople.org/repos/openstack/22:30
*** nellysmitt has quit IRC22:37
notmynametdasilva: thanks. it has to do with https://review.openstack.org/#/c/156388/22:38
*** nellysmitt has joined #openstack-swift22:38
notmynametdasilva: I linked https://openstack.redhat.com/Repositories which seems fairly general22:38
*** nellysmitt has quit IRC22:42
tdasilvanotmyname: cool, just added some comments there22:45
notmynamethanks22:45
notmynametdasilva: what? you didn't know you work for shuttleworth now? ;-)22:46
tdasilvalol22:46
tdasilvaI rode in an elevator with him in Paris and he didn't say anything22:46
tdasilva:-)22:46
tdasilvanotmyname: ubuntu already released packages for swift 2.2.2 huh?22:47
notmynameseems that way22:48
notmynameI only found that out myself today when I clicked that link22:48
StevenK swift | 2.2.2-0ubuntu1    | vivid                     | source22:48
StevenKYeah22:48
openstackgerritJohn Dickinson proposed openstack/swift: update the getting started doc  https://review.openstack.org/15638822:48
notmynametdasilva: done22:49
*** fbo has quit IRC22:49
*** fbo has joined #openstack-swift22:54
mattoliverauphew, expenses report done, that wasn't a waste of a morning at all :P22:55
notmyname:-)22:58
notmynameso I just saw this job posting on twitter: IBM looking for someone to run Swift. http://searchamericanjobs.com/display-job/29898533/SoftLayer-OpenStack-Swift-Lead-Architect.html22:59
notmynamethat's cool, but they're looking for someone with 5+ years experience running Swift at scale22:59
notmynamehmmm22:59
mattoliveraulol23:00
jrichli_interesting23:00
notmynameso they are targeting about 4 of the ops guys at rackspace? who themselves only have about 4.5 years experience with swift in prod at scale :-)23:00
notmyname(counting from may 2010)23:00
mahaticI did this first git pull origin master; it showed me I have an updated file and would I want it to be merged, I said yes. This is what git log looked like - >http://paste.openstack.org/show/175531/23:03
mahaticmy today's changes are committed with git commit --amend (which amended the wrong commit - the merge one). I then did a git reset --soft HEAD~123:05
mahaticI now have a ton of files to committed23:05
mahaticwrong channel to post this? :)23:05
notmynamemahatic: no, it's fine. you've got problems with swift, so let's get it sorted23:06
notmyname(but I'm going to go pour myself some coffee first)23:06
mahaticgreat, thanks!23:06
mahaticsure23:06
*** Nadeem_ has quit IRC23:07
*** chlong has joined #openstack-swift23:09
mahaticgit status shows this -> http://paste.openstack.org/show/175532/23:14
*** doxavore has quit IRC23:14
notmynamemahatic: I'm back23:14
mahaticyay23:14
notmynamedo you have changes locally that you're trying to keep?23:14
notmynameah I see23:14
notmynameyou amended the merge commit23:14
mahaticyeah!23:15
notmynamebut you need to undo that and amend the right one23:15
mahaticand then i did this  git reset --soft HEAD~123:15
notmynamehave you pushed anything (I'm guessing no since I didn't see a bot message in here)23:15
mahaticno I didn't23:15
notmynameok, that's good23:15
notmynamehow big was your change?23:16
notmynameor, how sad would you be if you had to redo it?23:16
mahaticI do have a back up.23:16
notmynameah good23:16
mahaticso nothing really sad :)23:16
notmyname:-)23:16
notmynameok, then `git reset --hard origin/master` will get you back to the consistent view of what's upstream23:16
mahaticokay, on it23:17
*** rdaly2 has joined #openstack-swift23:18
notmynamemahatic: I've changed my git workflow slightly so that I can avoid weird stuff like that23:18
notmynamemahatic: instead of using `git pull`, I use `git fetch --all`. that pulls down changes, but it doesn't update my local branches or the working directory23:19
notmynamemahatic: then, if everything is ok, I can easily `git merge origin/master` and be just fine23:19
mahaticnotmyname, oh okay. I will note this23:19
mahaticbut now, how do I amend my existing commit?23:19
notmynameso splitting the pull into the fetch+merge basically gives me a nice place where I can eg track your changes without having to affect my own branches23:20
*** joeljwright has quit IRC23:20
notmynamemahatic: is the commit that you want to amend on its own brach?23:20
notmynamebranch23:20
mahaticnope, master23:21
*** jrichli_ has quit IRC23:21
notmynamewell that's not good. but we can fix that23:21
notmynameis this amending a patch you have already submitted upstream?23:21
*** jrichli has joined #openstack-swift23:21
mahaticnotmyname, https://review.openstack.org/#/c/153617/ it's not in the upstream yet23:22
notmynameok23:22
notmynamemahatic: so we need to do a little surgery on your git repo so that things keep working.23:23
mahaticI see23:23
notmynamehere's what's happened: you wrote a change on master and submitted it, but it hasn't landed before other patches landed upstream. so now the upstream repo is out of sync with your local master branch. and that's why the `git pull` tried to do the merge23:24
notmynamemake sense?23:24
mahaticyup23:24
mahaticso I did merge23:25
notmynameagain?23:25
mahaticno, this merge was a couple of days back23:25
notmynameoh23:25
mahaticwhat you said happened when I tried to pull a couple of days back (and I merged)23:26
notmynameok23:26
mahaticand now, today after my changes I did this git commit --amend (so the prev merge commit was amended)23:26
notmynamegot it23:26
mahaticokay23:26
notmynametry to keep your local master clean. that is, the only time you update it is when you merge from origin. which should always be a fast-forward merge. all of your work happens on your local branches23:27
mahaticoh okay. I should create a local branch for my local changes23:27
notmynameyes23:28
notmynameand in this case, to fix this issue, it will be really easy23:28
notmynamesince you've already pushed it to gerrit23:28
notmynamethe first version. you'll still have to redo your updates to it23:28
mahaticokay23:28
notmynamesound ok? ready to get started on that?23:28
mahaticyeah, sounds good23:28
mahaticready!23:29
notmynamemahatic: ok. you already did the git reset? everything is back to normal on master?23:29
mahaticyeah, that's right23:29
mahaticgit reset --hard origin/master23:29
mahaticdid that23:30
notmynameok23:30
notmynameok, now we'll ask git-review to download the patch set and it will create the branch for you: `git review -d 153617`23:30
notmyname(that's instead of rewinding and doing all the git stuff locally. this way you just pull the branch from something external. much easier)23:31
notmynamethat will give you some weirdly-named branch23:31
notmynameie it downloads that patch into a branch and checks out that branch23:31
mahaticokay23:31
mahaticthat's right. this sounds a bit familiar (I really need to get a hang of this!)23:31
notmynamewhat branch did it create?23:31
notmynamedo `git branch` and look for the one with a * next to it23:32
mahaticit's still downloading23:32
notmynameoh23:32
mahatic* review/mahati/15361723:32
mahaticso that one23:32
notmynameok23:33
mahaticmy changes are back!23:33
notmynameso now, let's rename it something that's easier for you to deal with (I'm guessing you don't want to type `git checkount review/mahati/153617` every time you need to work on it23:33
notmynameto rename the current branch, do this: `git branch -m <new name here>`23:34
mahaticsure23:34
notmynameso maybe "checker" or something. whatever you want.23:34
*** rdaly2 has quit IRC23:34
notmynamemahatic: what OS are you using?23:34
mahaticshouldn't I be giving a generic one? So that going forward I can use this same local branch for all my other changes?23:35
mahaticfedora23:35
notmynameno. you'll have a new branch for every patch23:35
mahaticfedora 2023:35
notmynameand you can safely delete it after it's landed23:35
mahaticah okay23:35
mahaticI should make this a practice23:36
notmynameso there's a tool called gitk which might help you understand what's going on with git. it shows the git commit history visually.23:36
mahatic* ringchecker (that's my branch)23:37
mahaticoh23:37
notmynameon the mac, I use gitx. it's basically the same thing. here's a screenshot: http://d.not.mn/gitx.png23:37
mahaticnice23:37
notmynameso you can see that right now I've got a update_docs, fix_get_info, and master branches locally. and also the origin and gerrit remotes23:37
notmynameok, so one more thing before you redo your changes23:38
notmynamenow that you want to make updates to your ringchecker branch, you need to make sure it is still ok with the current state of master. ie there won't be any merge conflicts when it lands23:38
notmynamemake sense?23:38
mahaticyeah23:39
mahaticand how do i do that?23:39
notmynameeasy23:39
notmynameyou're on your branch23:39
notmynameso `git rebase master`. that will replay the commits on your current branch (ringchecker) on top of the master branch. in your case, you've only got one patch23:40
notmynameit will warn if there are conflicts. if so, then you'll need to resolve those and I can help you with that23:40
notmynamebut normally it will go without any errors23:40
mahaticoh okay23:41
mahaticon it23:41
mahaticand this step can be done after I redo my changes too?23:41
notmynameit's a lot easier if you do the rebase first23:42
mahaticas in, is this a step that needs to be taken every time I make changes on my local branch?23:42
notmynameyou only really need to do the rebase if there are merge conflicts in getting it onto master23:43
*** tdasilva has quit IRC23:43
mahaticoh okay23:43
mahaticdone23:43
mahaticno conlicts23:43
mahaticconflicts23:43
notmynamegreat23:45
notmynameso now put your patches onto teh current state of the ringchecker branch23:45
notmynameand then `git commit --amend -a` to add them to the existing commit23:45
mahaticmy earlier question was: say for my next patch, i create a new branch, should i have to run that rebase master command?23:45
notmynamethen `git review` for all the rest of us to se23:45
mahatic:)23:45
notmynamemahatic: in general no23:46
mahaticit should happen when I commit, right23:46
mahatic?23:46
notmynamewhat will?23:46
mahaticwhen I submit for review and get approval, the merge to master happens23:46
mahaticsorry, when I do my next new patch and commit, it should be on master (without me running git rebase), correct?23:47
notmynamecorrect23:48
notmynamewait23:48
notmynamecorrect to "when I submit for review and get approval, the merge to master happens"23:48
notmynamewhen you do another patch later, you'll create a new branch for it (git checkout -b cool_feature) and then do your work there23:48
mahaticyes and then my cool_feature does need an explicit rebase by git rebase master?23:49
notmynamemahatic: no, not when you create it. you'll create it off of master23:50
notmynamemahatic: you'll only need to do a rebase if your patch hasn't landed and there have been conflicting changes on master23:50
notmynamesummit conference session voting has started! https://www.openstack.org/vote-vancouver/23:51
mahaticah okay23:51
notmynamelooks like a lot of good swift talks proposed23:51
*** ChanServ changes topic to "Summit voting: https://www.openstack.org/vote-vancouver/ | riority Reviews: https://wiki.openstack.org/wiki/Swift/PriorityReviews | Review Dashboard: http://goo.gl/r2mxbe | Ideas: https://wiki.openstack.org/wiki/Swift/ideas"23:53
notmynamelink is in the topic now23:53
*** treyd has joined #openstack-swift23:56
notmynamemahatic: and the last important point is that when you do the amend, do not change the change-id line. that ties it all together in gerrit23:56
mahaticnotmyname, sure. I actually never amend that. I thought it was generated automatically23:57
mahaticamended23:57
notmynameit is via a commit hook you have locally. git review set that up for you at the beginning when you did `git review -s`23:57
*** jkugel has quit IRC23:58
mahaticah okay23:58
openstackgerritMahati proposed openstack/swift: Ring checker in swift-recon  https://review.openstack.org/15361723:59
mahaticfinally there it is!23:59
mahaticthanks so much notmyname !23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!