| *** setmason has quit IRC | 00:03 | |
| ho | good morning! | 00:04 |
|---|---|---|
| openstackgerrit | Samuel Merritt proposed openstack/swift: EC: don't 503 on marginally-successful PUT https://review.openstack.org/180795 | 00:05 |
| mattoliverau | ho: morning | 00:05 |
| kota_ | ho, mattoliverau: morning :) | 00:07 |
| mattoliverau | kota_: moring! | 00:07 |
| *** jamielennox|away is now known as jamielennox | 00:29 | |
| *** dmorita has joined #openstack-swift | 00:31 | |
| openstackgerrit | Merged openstack/swift: X-Auth-Token should be a bytestring. https://review.openstack.org/180098 | 00:46 |
| notmyname | clayg: I'm running reconstructor now and seeing errors in the logs. liberasure tracebacks and 404s | 00:46 |
| notmyname | here's a fun one: | 00:47 |
| notmyname | May 8 00:02:00 localhost.localdomain object-reconstructor: 192.168.12.15:6003/d33/2179 EXCEPTION in replication.Sender: #012Traceback (most recent call last):#012 File "/usr/lib/pymodules/python2.7/swift/obj/ssync_sender.py", line 72, in __call__#012 self.connect()#012 File "/usr/lib/pymodules/python2.7/swift/obj/ssync_sender.py", line 133, in connect#012 self.node['index'])#012KeyError: 'index' | 00:47 |
| clayg | nah lp bug #1452619 is old news | 00:50 |
| openstack | Launchpad bug 1452619 in OpenStack Object Storage (swift) "object-reconstructor crash in ssync_sender" [Undecided,Confirmed] https://launchpad.net/bugs/1452619 | 00:50 |
| clayg | notmyname: but the other ones might be something | 00:50 |
| notmyname | clayg: same as we saw on the GETs: Reconstruct ERROR: Insufficient number of fragments. | 00:51 |
| *** erlon has quit IRC | 00:51 | |
| notmyname | also saw the invalid literal for base 16 error | 00:53 |
| notmyname | clayg: hmm..this seems to kill the reconstructor | 00:55 |
| clayg | what do you mean by "this" and "kill" | 00:55 |
| clayg | i'm not terribly surprised fragment *rebuild* isn't making great progress (yet?) given the fragment placement of the object I was looking at last night | 00:57 |
| notmyname | clayg: are logged in doing stuff? | 00:57 |
| notmyname | onto 01 | 00:57 |
| clayg | notmyname: I just logged in and started tailing the all.log | 00:57 |
| notmyname | ah ok. my grep on that box started looking weird | 00:58 |
| notmyname | "who's this swiftqa user doing this grep?!" | 00:58 |
| clayg | but I'm referencing my notes from last night about the crazy fragments everywhere - and thinking about the rebuild handling in the reconstructor | 00:58 |
| ho | mattoliverau: kota_: morning! thanks! | 00:58 |
| clayg | how did I change what *your* grep was looking like? | 00:58 |
| notmyname | clayg: so by "this" and "kill" I mean that after a while the process ends | 00:58 |
| notmyname | as in: "this kills the crab" http://i.imgur.com/gMmnR5p.jpg | 01:00 |
| clayg | gross | 01:01 |
| clayg | this is probably darrells segfault bug! he was basically in the same state - fragments everywhere | 01:01 |
| clayg | notmyname: so lp bug #1452553 | 01:01 |
| openstack | Launchpad bug 1452553 in OpenStack Object Storage (swift) "don't rebuild existing fragments" [Undecided,New] https://launchpad.net/bugs/1452553 | 01:01 |
| notmyname | so the one lesson I'm repeatedly learning in the world of swift: if there's a bug, everyone has seen it. doesn't matter who you are, we all see the same thing ;-) | 01:02 |
| clayg | notmyname: too bad you picked isa-l - jerasure would have worked fine! | 01:02 |
| notmyname | "too bad" | 01:02 |
| * clayg jabs at peluse for not responding to my email to him and tsg and kmgreen yet | 01:02 | |
| notmyname | ok, so leave reconstruction off for a next, low-concurrency, run? | 01:03 |
| notmyname | all in a new container? | 01:03 |
| clayg | notmyname: nice thing about working in the open! | 01:03 |
| notmyname | :-) | 01:03 |
| clayg | notmyname: wait no - finding out if the cluster can right it self is way more intresting than benchmarks !!!! :D | 01:03 |
| clayg | oh oh oh! | 01:04 |
| clayg | handoffs_first! | 01:04 |
| notmyname | I'm gonna go with "no" to that question | 01:04 |
| notmyname | oh? | 01:04 |
| notmyname | wat wat? | 01:04 |
| clayg | if we can get reverts to fix things with the data we have - then we will segfault less - segfault less is good! | 01:04 |
| notmyname | I can agree with that! | 01:04 |
| clayg | i feel like reverts should be making some progress tho - i'm not seeing any "Removing partition" in the logs tho... | 01:05 |
| notmyname | so you think 2.conf should have handoffs_first? | 01:05 |
| clayg | notmyname: I'm going to try it - it's only "qa" - what's the worst that could happen? it won't make a black hole or anything | 01:05 |
| * clayg goes to get a beer first | 01:05 | |
| notmyname | all I ask is that you do it exactly the same on every server. I had to fix something of that earlier. makes updates later really hard when the configs are different | 01:06 |
| notmyname | maybe we should actually use some config management for this... | 01:06 |
| clayg | notmyname: I know some people that could help you with that... | 01:08 |
| notmyname | well, it is about 9am in taipei. charz should be online soon ;-) | 01:08 |
| clayg | man, how did I leave that lexographical revert sync sort in there :\ | 01:09 |
| clayg | i would have swore I came up with something better... gd | 01:10 |
| notmyname | my kids are getting crazy and need to be fed. I'm getting out of the qa cluster. I'll check in later about it | 01:10 |
| mattoliverau | notmyname: time to teach your kids how to work with the qa cluster ;) | 01:11 |
| mattoliverau | Sorry I've been distracted with sharding at large scale issues. so only leaving drive by sarcastic comments | 01:11 |
| notmyname | bah, I accidentally killed the screen session | 01:12 |
| clayg | now every time I type reconstructor I think of the object-retransmogifier | 01:13 |
| clayg | zaitcev is such a comedian | 01:13 |
| notmyname | what's the command to start a screen session and have the output go into a file? | 01:13 |
| clayg | there's a *command* for that? | 01:13 |
| clayg | typescript? | 01:14 |
| notmyname | I have no idea | 01:14 |
| notmyname | like how did the screen output go to screenlog.0 on bm01? | 01:14 |
| mattoliverau | clayg: shame april fools is over, otherwise a patch to change the name of the reconstructor would be in roder | 01:14 |
| mattoliverau | *order | 01:14 |
| clayg | maybe just "script" | 01:14 |
| clayg | notmyname: charz is a ninja - ninja's log | 01:15 |
| notmyname | yeah, I tried that. didn't see anything. a sneaky ninja | 01:15 |
| *** pberis has joined #openstack-swift | 01:17 | |
| clayg | see object-reconstructor: 192.168.12.13:6003/d22/855 Early disconnect | 01:17 |
| clayg | ^ that means the remote node isn't being a good replication server | 01:17 |
| clayg | what node is that? | 01:17 |
| clayg | qa0X? | 01:18 |
| mattoliverau | notmyname: looks like screen -L | 01:18 |
| notmyname | clayg: last quad of the IP is 13, so that's qa03 | 01:18 |
| notmyname | they ae .{11..15} | 01:18 |
| *** pberis has quit IRC | 01:19 | |
| notmyname | mattoliverau: yup. that was it. thanks! | 01:19 |
| notmyname | ok, gotta run now. I'll try to peek in later | 01:20 |
| mattoliverau | notmyname: have fun | 01:21 |
| *** aix has quit IRC | 01:31 | |
| *** rmcall has quit IRC | 01:45 | |
| charz | morning | 01:50 |
| *** km has quit IRC | 02:06 | |
| charz | notmyname: I'm going to do that (config managment) today. | 02:07 |
| *** km has joined #openstack-swift | 02:08 | |
| *** kei_yama has quit IRC | 02:08 | |
| *** panbalag has quit IRC | 02:08 | |
| *** kei_yama has joined #openstack-swift | 02:09 | |
| *** david-lyle has joined #openstack-swift | 02:10 | |
| *** tamizh_geek has joined #openstack-swift | 02:14 | |
| *** annegentle has joined #openstack-swift | 02:15 | |
| clayg | redbo: lol - cross post - should have left it to the master as I am ye verily but the student | 02:20 |
| *** annegentle has quit IRC | 02:20 | |
| clayg | redbo: not used to seeing you on the mailing list tho - I thought someone should say something ;) | 02:21 |
| redbo | I was hoping it'd just go away, but Chuck brought it back to life. | 02:24 |
| clayg | rofl | 02:25 |
| clayg | i just like pictured him sneazing and everyone's heads popping up over the cubicles | 02:26 |
| zaitcev | and the rest reaching for their guns | 02:26 |
| redbo | he's in a little conference room now, so it's contained | 02:27 |
| redbo | Plus it took me a few days to not be snippy. I feel like all these people think they get a vote about whether or not I can work on something :) | 02:28 |
| clayg | yeah I don't know - i was surpirsed when notmyname started the thread - but I guess it's better to be like "ok, let's get it out of your system people because this is happening" | 02:29 |
| *** jkugel has joined #openstack-swift | 02:31 | |
| *** wbhuber has joined #openstack-swift | 02:32 | |
| clayg | also - me likes redo's snippy | 02:33 |
| clayg | \:-| | 02:33 |
| clayg | no... maybe not that - I find it humerous when redbo makes snarky comments | 02:34 |
| zaitcev | At least you didn't find it humerus. | 02:35 |
| clayg | ^ see comedian | 02:37 |
| redbo | I have direct i/o implemented in a branch, maybe I'll do that once we finish some of the cleanup. | 02:38 |
| zaitcev | Sounds nice. | 02:38 |
| *** annegentle has joined #openstack-swift | 02:40 | |
| clayg | shit handoff_first doesn't work right because of the way we yield partitions... maybe... | 02:41 |
| *** jkugel has left #openstack-swift | 02:44 | |
| clayg | gah poor ssync - why am I getting 409 - the missing check shouldn't be asking me to ship these objects - crap I need acoles :D | 02:51 |
| clayg | ok, must be swapped - missing check says yeah I want that frag index, but PUT 409s because of another frag same ts - and since revert to handoff isn't working... | 02:54 |
| *** wbhuber has quit IRC | 03:11 | |
| *** bill_az has quit IRC | 03:12 | |
| *** kei_yama has quit IRC | 03:26 | |
| *** km has quit IRC | 03:27 | |
| *** km_ has joined #openstack-swift | 03:27 | |
| *** kei_yama has joined #openstack-swift | 03:27 | |
| *** fanyaohong has joined #openstack-swift | 03:41 | |
| *** rmcall has joined #openstack-swift | 03:42 | |
| *** annegentle has quit IRC | 03:43 | |
| *** annegentle has joined #openstack-swift | 03:44 | |
| *** tamizh_geek has quit IRC | 03:50 | |
| *** vinsh has quit IRC | 03:58 | |
| *** annegentle has quit IRC | 04:02 | |
| clayg | charz: notmyname: ok I added a patch to avoid the segfault to lp bug #1452553 and applied it to all the nodes in the qa cluster - seem to be segfaulting less | 04:20 |
| openstack | Launchpad bug 1452553 in OpenStack Object Storage (swift) "don't rebuild existing fragments" [Undecided,New] https://launchpad.net/bugs/1452553 | 04:20 |
| clayg | still seeing a lot of data on handoff nodes - but it may just be swapping fragments | 04:21 |
| charz | clayg: Did you add this patch to all ndoes in qacluster? | 04:21 |
| clayg | charz: yeah | 04:21 |
| clayg | "all" | 04:21 |
| clayg | heh | 04:21 |
| clayg | it's only 5 | 04:21 |
| clayg | acoles_away: so I ended up wanting to fix the keyerror and leave the 409 in place | 04:21 |
| charz | clayg: Oh, that's why reconstractors are working well | 04:22 |
| charz | clayg: I was confusing in that moment. :-) | 04:22 |
| clayg | peluse: you know how we were thinking swaps should basically almost *never* happen - I'm not sure what notmyname did when this cluster was getting setup - and I know the reconstructor wasn't running *at all* for a number of rebalances... | 04:23 |
| clayg | but swaps happen ;) | 04:23 |
| kota_ | notmyname, clayg: I might know something the decoding error at liberasurecode. | 04:33 |
| kota_ | notmyname: I saw the decoding error when I was making a ton of POST request (i.e. post as copy) | 04:34 |
| kota_ | notmyname: At that time, POST as COPY seemed to make some handoff frags in the cluster because of some reason likes connection timeout. | 04:35 |
| kota_ | notmyname: and then, when I tried GET object, sometimes it went to fail because proxy retrieved fragments from both primaries and handoffs (maybe on the GET also hits connection timeout) | 04:37 |
| kota_ | notmyname: " ERROR: Insufficient number of fragments." will occur when the number of fragments is less than k (ndata). | 04:39 |
| notmyname | yes. I saw that a lot | 04:39 |
| kota_ | notmyname: current Swift proxy ensure n stream for decode but if the framents duplicated each other | 04:40 |
| kota_ | notmyname: i.e. [0,0,1,2,3] for k=5 | 04:40 |
| kota_ | notmyname: that's example, it causes the Insufficient number of fragments. | 04:41 |
| kota_ | nomyname: the integer means Fragment Index. | 04:41 |
| kota_ | notmyname: e.g. k=5, m=2, primaries have [0, 1, 2, 3, 4, 5, 6] (the integer means fragment index) | 04:46 |
| kota_ | notmyname: assuming POST as copy (or Update with same object) and a node (e.g. 0) is failed to connect | 04:46 |
| kota_ | notmyname: data layout will be primaries [0, 1, 2, 3, 4, 5] and handoff [0] | 04:47 |
| kota_ | notmyname: assuming no recconstructor is running. | 04:47 |
| *** ppai has joined #openstack-swift | 04:48 | |
| kota_ | notmyame: no, data layout will be [0, 1, 2, 3, 4, 5, 6] and handoff [0] | 04:49 |
| kota_ | notmyname: and then when trying to GET and some nodes (e.g. 4,5,6) are failed to connect | 04:50 |
| kota_ | notmyname: proxy will get k nodes as [0, 1, 2, 3, 0(handoff)] for decode. | 04:50 |
| kota_ | notmyname: however acutually only 4 fragments are gathered even though 5 *different* fragments needed for decoding. | 04:52 |
| kota_ | notmyname: liberasurecode will raises "ERROR: Insufficient number of fragments." at such a situation. | 04:53 |
| kota_ | notmyname: for now, I'm thinking whether we need a validation for the fragment index uniqueness for decoding or not. | 04:55 |
| kota_ | imo, we need some imlementation to handle the decoding failure... | 04:57 |
| kota_ | hum.... | 04:57 |
| *** annegentle has joined #openstack-swift | 05:03 | |
| notmyname | kota_: that sounds very similar to what I saw. it wasn't POST-as-COPY, but the result was effectively the same. reading older data after a ring change. and the reconstructor wasn't running (because of aforementioned segfaults) | 05:05 |
| notmyname | but mostly I've been feeding clayg info on all this and being his rubber duck for debugging. ;-) | 05:06 |
| notmyname | seems like he has some patches and ideas on the next step for clearing these up | 05:06 |
| kota_ | notmyname: cool | 05:07 |
| kota_ | notmyname: patches are already in gerrit? I'd love to see | 05:07 |
| *** annegentle has quit IRC | 05:09 | |
| *** SkyRocknRoll has joined #openstack-swift | 05:15 | |
| *** zaitcev has quit IRC | 05:22 | |
| *** bkopilov has quit IRC | 05:27 | |
| *** bkopilov has joined #openstack-swift | 05:44 | |
| *** vinsh has joined #openstack-swift | 05:46 | |
| *** fanyaohong has quit IRC | 05:49 | |
| *** vinsh has quit IRC | 05:51 | |
| *** tamizh_geek has joined #openstack-swift | 06:03 | |
| *** annegentle has joined #openstack-swift | 06:04 | |
| *** annegentle has quit IRC | 06:10 | |
| *** thumpba has joined #openstack-swift | 06:32 | |
| openstackgerrit | Christian Schwede proposed openstack/swift: Update my mailmap https://review.openstack.org/181305 | 06:42 |
| *** bkopilov has quit IRC | 06:46 | |
| ho | cschwede: hello, is there any rule to add email addresses to the mailmap? | 06:56 |
| *** annegentle has joined #openstack-swift | 07:05 | |
| *** annegentle has quit IRC | 07:10 | |
| *** krykowski has joined #openstack-swift | 07:17 | |
| *** rmcall has quit IRC | 07:35 | |
| *** jamielennox is now known as jamielennox|away | 07:36 | |
| *** geaaru has joined #openstack-swift | 07:44 | |
| *** thumpba has quit IRC | 07:53 | |
| *** tamizh_geek has quit IRC | 07:55 | |
| *** tobe has joined #openstack-swift | 08:05 | |
| *** annegent_ has joined #openstack-swift | 08:06 | |
| *** annegent_ has quit IRC | 08:11 | |
| *** proteusguy has joined #openstack-swift | 08:17 | |
| *** chlong has quit IRC | 08:20 | |
| *** tobe has quit IRC | 08:31 | |
| *** hunius has joined #openstack-swift | 08:46 | |
| *** hunius_ has joined #openstack-swift | 08:50 | |
| *** bkopilov has joined #openstack-swift | 08:51 | |
| *** hunius has quit IRC | 08:51 | |
| *** acoles_away is now known as acoles | 08:51 | |
| *** rvasilets has joined #openstack-swift | 08:55 | |
| *** tamizh_geek has joined #openstack-swift | 08:56 | |
| *** tamizh_geek has quit IRC | 09:00 | |
| acoles | notmyname: clayg: so i can see from a distance you guys are working hard on reconstructor stuff which is great! | 09:05 |
| *** annegentle has joined #openstack-swift | 09:07 | |
| acoles | notmyname: clayg some thoughts from over here FWIW: 'the invalid literal for base 16' error gets logged when ssync closes connection early on failures (i.e. its kinda 'normal' although noisy) - *some* of them get cleaned up by patch 177836 | 09:08 |
| patchbot | acoles: https://review.openstack.org/#/c/177836/ | 09:08 |
| cschwede | ho: indeed, the .mailmap is used by git, so the definition is in the manpage: ftp://www.kernel.org/pub/software/scm/git/docs/git-shortlog.html - and because i’ll cuse my @redhat.com address from now on i added this | 09:09 |
| acoles | clayg: and i am working my way towards avoiding ssync 409s when rx has different FI by having the rx return 'not in sync but don't send' semantic in missing_check response | 09:10 |
| acoles | clayg: but not quite there yet - will build on patch 138498 | 09:11 |
| patchbot | acoles: https://review.openstack.org/#/c/138498/ | 09:11 |
| *** hunius_ has quit IRC | 09:11 | |
| acoles | notmyname: clayg thats ^^ not meant to be self-promotion of my patches! just trying to help :) | 09:12 |
| *** annegentle has quit IRC | 09:12 | |
| *** tamizh_geek has joined #openstack-swift | 09:22 | |
| *** thumpba has joined #openstack-swift | 09:24 | |
| *** vinsh has joined #openstack-swift | 09:24 | |
| *** thumpba has quit IRC | 09:28 | |
| *** vinsh has quit IRC | 09:28 | |
| *** krykowski has quit IRC | 09:29 | |
| *** krykowski has joined #openstack-swift | 09:29 | |
| ho | cschwede: i see. thanks for the info. | 09:46 |
| acoles | ho: hello! are you coming to vancover? | 09:47 |
| cschwede | ho: you’re welcome! | 09:48 |
| cschwede | acoles: i hope he will do | 09:48 |
| acoles | me too | 09:48 |
| cschwede | acoles: when are you arriving? | 09:49 |
| acoles | cschwede: heh just typed same question for you | 09:49 |
| acoles | cschwede: saturday | 09:49 |
| acoles | wondering if the 'panel' could meet sunday evening to prepare? | 09:50 |
| cschwede | acoles: :) i’ll arrive on sunday, will be in Denmark next week, coming back on saturday. will be at home then for exactly 12 hours ;) | 09:50 |
| cschwede | acoles: that’s a very good idea! | 09:50 |
| acoles | cschwede: oh wow! Denmark for work or vacation? | 09:51 |
| kota_ | acoles, cschwede: hi | 09:51 |
| acoles | kota_: hi! | 09:51 |
| kota_ | acoles, cschwede: FYI, I'll arrive on sunday | 09:51 |
| cschwede | acoles: vacation. kids are off school next week, and it’s just a few hours drive - nice for relaxing :) | 09:52 |
| cschwede | kota_: hello Kota! | 09:52 |
| acoles | so maybe we can have a very jet lagged meetup sunday evening | 09:52 |
| cschwede | acoles: yes, let’s do that! | 09:52 |
| acoles | cschwede: sounds good. and makes sense you have been writing specs this week ;) | 09:53 |
| cschwede | i wanted to visit the capilano suspension bridge on sunday, hopefully i’m not to jetlagged then. if anyone else is interested: http://www.capbridge.com/ | 09:53 |
| acoles | kota_: cschwede which hotels are you in? (thinking about venue selection for possible dinner) | 09:54 |
| cschwede | acoles: yes, wanted to have something to discuss in vancouver, and specs are a good start (hopefully) | 09:54 |
| acoles | definitely! | 09:54 |
| kota_ | ya, absolutely | 09:55 |
| acoles | cschwede: hang on, so what time does your flight get in on sunday?? | 09:55 |
| kota_ | I'll stay at Pinnacle Hotel Vancouver Harbourfront. | 09:55 |
| cschwede | acoles: i’m in the „days inn vancouver downtown“, 921 West Pender Street | 09:55 |
| cschwede | acoles: 11:50 local time | 09:56 |
| acoles | ah early from europe, i'm late saturday | 09:56 |
| cschwede | kota: i’m just 5 minutes walking distance from your hotel | 09:57 |
| cschwede | probably it’s the easiest thing to meet at the convention center entrance? | 09:57 |
| kota_ | cschwede: great! | 09:57 |
| kota_ | I'll arrived on vancouver at 10:15 am local time. | 09:57 |
| *** proteusguy has quit IRC | 09:58 | |
| kota_ | cschwede: that sounds good to meet at the convention center. | 09:58 |
| acoles | kota_: cschwede: i'll ask around the others and then send email with suggested time | 09:58 |
| kota_ | acoles: ok, thanks :D | 09:59 |
| acoles | cschwede: i have been to capilano! its good, iirc (was ~20 years ago) | 09:59 |
| *** krykowski has quit IRC | 09:59 | |
| cschwede | acoles: nice, good to know! | 10:00 |
| acoles | cschwede: my current plan is to visit friends on sunday though | 10:00 |
| kota_ | wow, capbridge looks so interesting... | 10:01 |
| *** annegent_ has joined #openstack-swift | 10:08 | |
| *** annegent_ has quit IRC | 10:13 | |
| *** kota_ has quit IRC | 10:20 | |
| *** panbalag has joined #openstack-swift | 10:23 | |
| *** krykowski has joined #openstack-swift | 10:24 | |
| *** krykowski has quit IRC | 10:47 | |
| *** krykowski has joined #openstack-swift | 10:53 | |
| *** dmorita has quit IRC | 10:57 | |
| *** rvasilets has quit IRC | 11:01 | |
| ho | acoles: hello, I will come to vancouver! yay! | 11:07 |
| *** annegent_ has joined #openstack-swift | 11:08 | |
| ho | let me join the dinner on sunday :-) | 11:12 |
| *** vinsh has joined #openstack-swift | 11:13 | |
| *** annegent_ has quit IRC | 11:13 | |
| ho | good night all! have a nice weekend | 11:14 |
| *** ho has quit IRC | 11:14 | |
| *** vinsh has quit IRC | 11:17 | |
| *** cdelatte has joined #openstack-swift | 11:17 | |
| *** delattec has joined #openstack-swift | 11:17 | |
| *** ppai has quit IRC | 11:20 | |
| *** ppai has joined #openstack-swift | 11:32 | |
| *** fthiagogv has joined #openstack-swift | 11:59 | |
| *** annegentle has joined #openstack-swift | 12:00 | |
| *** delattec has quit IRC | 12:01 | |
| *** cdelatte has quit IRC | 12:01 | |
| *** thurloat_isgone is now known as thurloat | 12:09 | |
| *** MVenesio has joined #openstack-swift | 12:14 | |
| *** SkyRocknRoll has quit IRC | 12:27 | |
| *** chlong has joined #openstack-swift | 12:46 | |
| *** ppai has quit IRC | 12:47 | |
| *** NM has joined #openstack-swift | 12:47 | |
| *** ppai has joined #openstack-swift | 13:00 | |
| *** annegentle has quit IRC | 13:00 | |
| *** vinsh has joined #openstack-swift | 13:01 | |
| *** annegentle has joined #openstack-swift | 13:02 | |
| *** vinsh has quit IRC | 13:07 | |
| *** EmilienM|afk is now known as EmilienM | 13:15 | |
| *** km_ has quit IRC | 13:22 | |
| *** kei_yama has quit IRC | 13:26 | |
| *** jrichli has joined #openstack-swift | 13:39 | |
| *** tamizh_geek has quit IRC | 13:40 | |
| *** esker has joined #openstack-swift | 13:40 | |
| *** esker has quit IRC | 13:41 | |
| *** esker has joined #openstack-swift | 13:41 | |
| *** esker has quit IRC | 13:41 | |
| *** esker has joined #openstack-swift | 13:41 | |
| *** erlon has joined #openstack-swift | 13:44 | |
| *** annegentle has quit IRC | 13:44 | |
| *** annegentle has joined #openstack-swift | 13:45 | |
| *** MVenesio has quit IRC | 13:50 | |
| *** openstackgerrit has quit IRC | 13:51 | |
| *** openstackgerrit has joined #openstack-swift | 13:51 | |
| *** lpabon has joined #openstack-swift | 13:59 | |
| *** vinsh has joined #openstack-swift | 14:08 | |
| *** vinsh has quit IRC | 14:08 | |
| *** annegentle has quit IRC | 14:16 | |
| *** ppai has quit IRC | 14:19 | |
| *** shakamunyi has joined #openstack-swift | 14:25 | |
| *** vinsh has joined #openstack-swift | 14:27 | |
| openstackgerrit | Alistair Coles proposed openstack/swift: Add POST capability to ssync for .meta files https://review.openstack.org/138498 | 14:30 |
| openstackgerrit | Alistair Coles proposed openstack/swift: Don't ssync fragments that conflict with another fragment https://review.openstack.org/181407 | 14:30 |
| acoles | clayg: ^^ that gets rid of some ssync 409s | 14:31 |
| acoles | clayg: somewhere in that patch chain i have a TODO inviting an argument about the message format ;) | 14:32 |
| *** Guest58709 has joined #openstack-swift | 14:41 | |
| *** bkopilov has quit IRC | 14:46 | |
| *** minwoob has joined #openstack-swift | 14:48 | |
| *** wbhuber has joined #openstack-swift | 14:58 | |
| *** proteusguy has joined #openstack-swift | 15:08 | |
| *** proteusguy has quit IRC | 15:09 | |
| *** proteusguy has joined #openstack-swift | 15:09 | |
| *** wbhuber_ has joined #openstack-swift | 15:20 | |
| *** wbhuber has quit IRC | 15:23 | |
| *** shakamunyi has quit IRC | 15:25 | |
| *** silor has joined #openstack-swift | 15:32 | |
| *** Guest58709 is now known as annegentle | 15:40 | |
| *** krykowski has quit IRC | 15:48 | |
| *** gyee has joined #openstack-swift | 15:52 | |
| *** openstackstatus has quit IRC | 15:56 | |
| *** openstackstatus has joined #openstack-swift | 15:58 | |
| *** ChanServ sets mode: +v openstackstatus | 15:58 | |
| notmyname | good morning | 15:58 |
| *** annegentle has quit IRC | 16:02 | |
| notmyname | cschwede: ping | 16:02 |
| cschwede | notmyname: good morning! | 16:03 |
| notmyname | hello | 16:03 |
| notmyname | cschwede: on https://review.openstack.org/#/c/181305/ ... if you update the .mailmap you also need to update AUTHORS | 16:03 |
| notmyname | cschwede: normally I'd just do that for you, but since you started.... ;-) | 16:03 |
| cschwede | notmyname: ah, thx for the info - i knew i probably forgot something ;) | 16:04 |
| cschwede | notmyname: i wanted to make it a tad easier for you, but then… | 16:05 |
| openstackgerrit | Christian Schwede proposed openstack/swift: Update my mailmap entry https://review.openstack.org/181305 | 16:05 |
| notmyname | cschwede: this has my own path names hard coded in it, but I use https://github.com/notmyname/git-stats/blob/master/new_authors.sh to keep that stuff up to date | 16:07 |
| notmyname | cschwede: thanks. looks good now | 16:07 |
| cschwede | new_authors.sh looks interesting - looks like you automated all the recurring PTL duties :) | 16:08 |
| notmyname | not quite :-) | 16:09 |
| notmyname | extra awesome would be to actually patch AUTHORS. I still do that by hand | 16:09 |
| *** InAnimaTe has joined #openstack-swift | 16:10 | |
| *** dencaval has joined #openstack-swift | 16:10 | |
| *** aerwin has joined #openstack-swift | 16:13 | |
| *** shakamunyi has joined #openstack-swift | 16:15 | |
| *** gyee has quit IRC | 16:16 | |
| *** wbhuber has joined #openstack-swift | 16:16 | |
| *** wbhuber_ has quit IRC | 16:22 | |
| *** gyee has joined #openstack-swift | 16:22 | |
| *** Gues_____ has joined #openstack-swift | 16:22 | |
| *** tamizh_geek has joined #openstack-swift | 16:23 | |
| *** nadeem has joined #openstack-swift | 16:24 | |
| minwoob | Hi. Are there some performance results for ISA-L compared to other EC libs (liberasurecode, Jerasure, etc) ? | 16:24 |
| *** nadeem has quit IRC | 16:27 | |
| notmyname | minwoob: no, not yet. at least not that I've seen other than Intel marketing video ;-) /cc peluse | 16:29 |
| *** Gues_____ has quit IRC | 16:30 | |
| wbhuber | I'd like to add Minwoo's point. In the latest relese notes (kilo), there's a known issue published that EC support is lacking outside swiftstack. | 16:31 |
| notmyname | wbhuber: ? | 16:32 |
| wbhuber | And there is not much of performance characterization going on. How can we make a contribution to that content? | 16:32 |
| wbhuber | https://wiki.openstack.org/wiki/ReleaseNotes/Kilo | 16:32 |
| notmyname | whew. "swiftstack" isn't found on that page | 16:32 |
| wbhuber | Is there anything of preliminary performance testnig that had been done or not? | 16:33 |
| *** Gues_____ has joined #openstack-swift | 16:33 | |
| wbhuber | Pardon me. Disregard "swiftstack". | 16:33 |
| *** shakamunyi has quit IRC | 16:33 | |
| notmyname | wbhuber: yes, it's ongoing. if you want to help, the basic idea is to set up a cluster and run some tests. I know that I'm doing that with clayg on the community qa cluster, and we'll continue to be open with any results from that | 16:33 |
| notmyname | wbhuber: ;-) | 16:33 |
| wbhuber | what is the configuration like? beyond a single SAIO environment? if you have some documentation to lead us, that would be utmost terrfic. | 16:34 |
| *** thumpba has joined #openstack-swift | 16:34 | |
| notmyname | wbhuber: and mattoliverau said he was getting it in a cluster (IIRC) and looking. kota_ has done some stuff (I know because he's seen the same issues we've seen). I hope acoles has started looking at HP. and if you could look at IBM that would be wonder ful | 16:34 |
| notmyname | wbhuber: the community qa cluster is 5 nodes, 12 drives each. we've configure 4 storage policies (20 drives for only replication, 20 drives for both repl and EC, and 20 drives for only EC) | 16:35 |
| wbhuber | 5 nodes in addition to proxy and container nodes? | 16:36 |
| notmyname | wbhuber: there's isn't a particular config other than "deploy a cluster like normal". then we're all just adding an EC policy to it | 16:36 |
| notmyname | no, it's 5 nodes total. they all run all the processes. I know that's not great for performance, but it's sufficient for relative tests and basic validation | 16:36 |
| *** thumpba_ has joined #openstack-swift | 16:37 | |
| notmyname | I've got another larger cluster that I'll use later for some more testing after most of the major issues are resolve and we have some baselines | 16:37 |
| wbhuber | i am still digesting the whole cluster configuration to see if we could "replicate" it on our end for further testing and performance number profiling. | 16:38 |
| notmyname | I think the most helpful thing to start with is not to try to get every ounce of performance out of EC to see what's possible, but to explore it from the perspective of "when does EC make sense vs replication". ie what data size, what operations, etc | 16:39 |
| notmyname | and further to see where the particular hardware bottlenecks are so we have a general idea of cluster sizing. eg is it CPU or disk bound first? how do memory requirements compare? etc | 16:40 |
| *** thumpba has quit IRC | 16:40 | |
| wbhuber | do you have some kind of performance design with questions that need to be answered or some sorts that we can use as a direction? | 16:40 |
| notmyname | I'm planning on sharing any results we have at the summit. I hope we'll get the current issues sorted out soon (today?) so we can have some interesting relative numbers to share | 16:41 |
| notmyname | wbhuber: two important questions that don't have good answers yet: 1) at what object size does it make sense to use EC vs replication? 2) given a particular hardware configuration, what are the appropriate EC parameters? | 16:42 |
| notmyname | I don't know the answer to either of those, but they are some of the first questions asked. so that's my current goal | 16:43 |
| notmyname | getting data points for that | 16:43 |
| wbhuber | Speaking of 1), I have been hearing that Replication makes most sense when it is used for less than 1MB objects and EC for larger objects. I have to get to the sources and research on them further. | 16:43 |
| notmyname | that is, beyond "does this work?" | 16:43 |
| notmyname | yeah, replication is definitely better at smaller objects. but what is the switch point? 5MB? 50MB? 100MB? | 16:44 |
| wbhuber | Good questions. | 16:44 |
| *** harlowja has quit IRC | 16:44 | |
| notmyname | and is it different for read-heavy workloads vs write-heavy workloads? | 16:44 |
| *** harlowja has joined #openstack-swift | 16:44 | |
| acoles | jrichli: ping | 16:45 |
| notmyname | wbhuber: so unfortunately, the best answer I have right now is "go deploy swift with EC and see what happens" :-) | 16:46 |
| wbhuber | :-) | 16:46 |
| *** shakamunyi has joined #openstack-swift | 16:47 | |
| notmyname | wbhuber: or rather, "I'm really looking forward to seeing your own test numbers in Vancouver at the summit" ;-) | 16:47 |
| clayg | notmyname: I thought that's what the release notes said - that's what we're doing - what is everyone else doing? | 16:48 |
| notmyname | clayg: I think we're all doing the same thing | 16:48 |
| clayg | go team swift! | 16:48 |
| notmyname | and one of the cool parts is that we're all seeing the same thing | 16:48 |
| wbhuber | notmyname: Wish we could have done that sooner but most likely before the next summit after Vancouver. | 16:48 |
| *** Gues_____ has quit IRC | 16:48 | |
| * clayg goes to see if the qa cluster unhozed it self with my patches | 16:48 | |
| notmyname | well don't only do that for the next six months | 16:48 |
| notmyname | there's nothing stopping you from sharing in here as soon as you have numbers :-) | 16:49 |
| wbhuber | we will tune in to you guys every now and then before 6 months | 16:49 |
| wbhuber | :) | 16:49 |
| clayg | notmyname: we have to benchmark go, and object-server per disk, and pypy apparently :'( | 16:49 |
| *** acoles is now known as acoles_away | 16:49 | |
| notmyname | the only difference with the summit is that at the summit we talk face to face. | 16:49 |
| minwoob | wbhuber: It seems that EC generally performs better for writes than for reads. | 16:50 |
| notmyname | clayg: yeah, testing Go is important. and object-server-per-disk (I expected to already have that done on the other lab cluster, but this ec has taken longer). | 16:50 |
| wbhuber | notmyname: yes, we'd like to get your results not only at the summit but also here. i'll ask jrichli to pick up the materials if the numbers aren't posted by then. | 16:51 |
| notmyname | wbhuber: they'll all be online | 16:51 |
| wbhuber | minwoob: is there also research statistics on when migrating larger objects from Replication to EC? | 16:52 |
| wbhuber | notmyname: sounds good. | 16:52 |
| minwoob | Haven't looked into migration yet. | 16:52 |
| *** proteusguy has quit IRC | 16:53 | |
| *** annegentle has joined #openstack-swift | 16:59 | |
| *** zhill_ has joined #openstack-swift | 17:05 | |
| *** bkopilov has joined #openstack-swift | 17:06 | |
| *** shakamunyi has quit IRC | 17:07 | |
| *** wasmum has quit IRC | 17:09 | |
| *** shakamunyi has joined #openstack-swift | 17:13 | |
| *** barra204 has joined #openstack-swift | 17:14 | |
| *** shakamunyi has quit IRC | 17:14 | |
| *** whydidyoustealmy has joined #openstack-swift | 17:16 | |
| *** barra204 has quit IRC | 17:17 | |
| *** whydidyoustealmy has quit IRC | 17:17 | |
| *** shakamunyi has joined #openstack-swift | 17:18 | |
| *** tamizh_g_ has joined #openstack-swift | 17:19 | |
| clayg | ok, so the cluster did not unhoze - issue seems to be the 409's causing early disconnect means your inability to make progress with cleanup (because early disconnect fails the whole ssync) will eventually lead to a large variety of farg indexes and a part and almost any node you might try to sync with have a conflict at some point | 17:19 |
| *** tamizh_geek has quit IRC | 17:20 | |
| *** tamizh_geek has joined #openstack-swift | 17:21 | |
| *** annegentle has quit IRC | 17:21 | |
| *** shakamunyi has quit IRC | 17:23 | |
| *** tamizh_g_ has quit IRC | 17:23 | |
| *** shakamunyi has joined #openstack-swift | 17:34 | |
| *** bkopilov has quit IRC | 17:39 | |
| *** fthiagogv has quit IRC | 17:39 | |
| *** tamizh_g_ has joined #openstack-swift | 17:39 | |
| *** tamizh_geek has quit IRC | 17:41 | |
| *** annegentle has joined #openstack-swift | 17:43 | |
| *** sandywalsh has left #openstack-swift | 17:44 | |
| jrichli | acoles: just got back from lunch. reading scrollback. | 17:46 |
| jrichli | acoles_away ^^ | 17:49 |
| openstackgerrit | Merged openstack/swift: Update my mailmap entry https://review.openstack.org/181305 | 17:50 |
| *** esker has quit IRC | 17:52 | |
| *** Guest___ has joined #openstack-swift | 17:56 | |
| *** tamizh_geek has joined #openstack-swift | 18:08 | |
| *** tamizh_g_ has quit IRC | 18:10 | |
| *** wasmum has joined #openstack-swift | 18:11 | |
| *** esker has joined #openstack-swift | 18:18 | |
| *** zhill_ has quit IRC | 18:23 | |
| *** tamizh_g_ has joined #openstack-swift | 18:28 | |
| *** tamizh_geek has quit IRC | 18:28 | |
| *** annegentle has quit IRC | 18:31 | |
| *** zhill_ has joined #openstack-swift | 18:31 | |
| *** wasmum has quit IRC | 18:33 | |
| *** hub_cap has joined #openstack-swift | 18:35 | |
| hub_cap | hello friendos. QQ about swift w/ memcache. Is it necessary? Is it stupid if I dont use it? I guess I dont really understand its need/use in the cluster | 18:36 |
| hub_cap | feel free to tell me im a dummy :D | 18:37 |
| redbo | If you don't run memcache, every operation will have to do a lookup on the container, to see if you have permissions at the minimum. A few features won't work at all like rate limiting. | 18:47 |
| redbo | You'll have to hit the auth system for every request to see if the user is valid. Stuff like that. | 18:48 |
| redbo | I guess that might not be true for those giant encrypted tokens | 18:49 |
| *** Guest___ has quit IRC | 18:53 | |
| hub_cap | My use case is stupid small, like a 3 node cluster with almost no operations... I just need replication, an s3 api, and code that attacks split brain stupidity :) | 18:55 |
| hub_cap | thx redbo for the info | 18:57 |
| *** Guest___ has joined #openstack-swift | 19:01 | |
| *** wasmum has joined #openstack-swift | 19:07 | |
| *** lpabon has quit IRC | 19:14 | |
| *** tamizh_geek has joined #openstack-swift | 19:15 | |
| *** zaitcev has joined #openstack-swift | 19:17 | |
| *** ChanServ sets mode: +v zaitcev | 19:17 | |
| *** tamizh_geek has quit IRC | 19:17 | |
| *** tamizh_g_ has quit IRC | 19:17 | |
| openstackgerrit | Minwoo Bae proposed openstack/swift: The hash_cleanup_listdir function should only be called when necessary. https://review.openstack.org/178317 | 19:34 |
| *** Guest___ has quit IRC | 19:40 | |
| *** Guest___ has joined #openstack-swift | 19:47 | |
| *** bkopilov has joined #openstack-swift | 19:49 | |
| *** Guest___ has quit IRC | 19:50 | |
| *** zaitcev has quit IRC | 19:56 | |
| *** rmcall has joined #openstack-swift | 20:14 | |
| *** shakamunyi has quit IRC | 20:17 | |
| *** breitz has quit IRC | 20:17 | |
| *** breitz has joined #openstack-swift | 20:17 | |
| *** thurloat is now known as thurloat_isgone | 20:18 | |
| *** jkugel has joined #openstack-swift | 20:20 | |
| *** annegentle has joined #openstack-swift | 20:25 | |
| *** annegentle has quit IRC | 20:26 | |
| *** annegentle has joined #openstack-swift | 20:26 | |
| jrichli | Hi all. I was wanting to learn more about account creation and deletion under the hood. I know that the identity API is used. But I wanted to know more about the role that the middleware can play. | 20:32 |
| jrichli | For example, I know that some accounts are "autocreated", and so an account PUT will not pass through the pipeline for that. | 20:32 |
| jrichli | Does an account DELETE request pass through the pipeline? I noticed that there is a direct_delete_account, which sounds like it would be deleting without any help from the pipeline. Is this right? | 20:33 |
| *** silor has quit IRC | 20:36 | |
| *** esker has quit IRC | 20:38 | |
| openstackgerrit | Merged openstack/swift: Functional test for SLO PUT overwriting one of its own segments https://review.openstack.org/174557 | 20:40 |
| openstackgerrit | Merged openstack/swift: Bump up a timeout in a test https://review.openstack.org/179956 | 20:43 |
| *** dencaval has quit IRC | 20:43 | |
| torgomatic | jrichli: anything in direct_client.py speaks to the backend servers without going through the proxy | 20:46 |
| torgomatic | but usually, an account DELETE will go through the proxy | 20:46 |
| *** shakamunyi has joined #openstack-swift | 20:47 | |
| *** bejorg has quit IRC | 20:50 | |
| jrichli | torgomatic: ok, thx. we can talk more along these lines at summit. I wanted to be sure that planning on some middleware actions for acct DELETE is reasonable. | 20:52 |
| *** shakamunyi has quit IRC | 21:03 | |
| jrichli | torgomatic: is there any way for a client to create an account with user-meta without going through the proxy? | 21:04 |
| torgomatic | jrichli: not that I know of | 21:04 |
| jrichli | k, thx | 21:05 |
| *** breitz has quit IRC | 21:08 | |
| *** aerwin has quit IRC | 21:16 | |
| *** rmcall has quit IRC | 21:17 | |
| *** shakamunyi has joined #openstack-swift | 21:21 | |
| openstackgerrit | Tim Burke proposed openstack/swift: Properly re-raise exceptions in proxy_logging https://review.openstack.org/181566 | 21:26 |
| *** esker has joined #openstack-swift | 21:36 | |
| jlk | hey folks. I've got some weird stuff going on where if I have both authtoken and keystoneauth in my proxy pipeline I get failures to auth a token. BUT if I take out keystoneauth and just leave authtoken, things work fine. I'm having trouble wrapping my brain around where the misconfiguration can be, can somebody lend me a hand? | 21:39 |
| jlk | notmyname: you were very helpful last time, but I've got a new cluster with a new problem, hoping you'll be able to help again :) ^^ | 21:41 |
| *** erlon has quit IRC | 21:41 | |
| *** NM has quit IRC | 21:48 | |
| *** jrichli has quit IRC | 21:50 | |
| *** jkugel has left #openstack-swift | 21:51 | |
| *** wbhuber has quit IRC | 22:03 | |
| *** shakamunyi has quit IRC | 22:04 | |
| *** barra204 has joined #openstack-swift | 22:04 | |
| peluse | wrt earlier conversation on EC perf - we (intel) are also setting up 2 14 node clusters with varrying CPUs and running a set of tests w/both 3x and EC and plan to have those results available sometime after the summit unfortunately | 22:08 |
| *** EmilienM is now known as EmilienM|afk | 22:08 | |
| *** zhill_ has quit IRC | 22:20 | |
| *** annegentle has quit IRC | 22:23 | |
| *** zhill_ has joined #openstack-swift | 22:27 | |
| *** annegentle has joined #openstack-swift | 22:36 | |
| *** vinsh has quit IRC | 22:42 | |
| notmyname | wheeee. been in meetings since 10:30 am | 23:09 |
| *** annegentle has quit IRC | 23:09 | |
| notmyname | finally done | 23:09 |
| notmyname | the longest one was a new hire training, so that was fun :-) | 23:09 |
| *** zaitcev has joined #openstack-swift | 23:21 | |
| *** ChanServ sets mode: +v zaitcev | 23:21 | |
| *** barra204 has quit IRC | 23:22 | |
| *** jrichli has joined #openstack-swift | 23:42 | |
| InAnimaTe | notmyname: do you work at swiftstack? | 23:43 |
| InAnimaTe | lol k just answered my own question | 23:44 |
| InAnimaTe | thank you for having proper whois information | 23:44 |
| zaitcev | oops | 23:44 |
| *** zhill_ has quit IRC | 23:54 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
