Thursday, 2015-07-30

*** haomaiwang has quit IRC00:01
*** haomaiwang has joined #openstack-swift00:02
*** darrenc_afk is now known as darrenc00:14
*** jasondotstar has quit IRC00:23
*** dmorita has joined #openstack-swift00:25
*** marcusvrn has quit IRC00:25
*** tellesnobrega has joined #openstack-swift00:33
*** tellesnobrega has quit IRC00:34
*** h00327910__ has joined #openstack-swift00:58
*** jasondotstar has joined #openstack-swift00:58
*** haomaiwang has quit IRC01:01
*** haomaiwang has joined #openstack-swift01:02
*** kei_yama has quit IRC01:02
*** kei_yama has joined #openstack-swift01:03
*** janonymous_ has joined #openstack-swift01:14
*** chlong has joined #openstack-swift01:29
*** ccavanna has joined #openstack-swift01:30
*** ccavanna has quit IRC01:33
*** jasondotstar has quit IRC01:35
flwang1ho: still around?01:45
flwang1clayg: ping01:45
flwang1anybody around?01:59
hoflwang!: yes01:59
hosorry s/flwang!/flwang1/01:59
flwang1ho: cooool01:59
flwang1i just give it a try for my 1st use case02:00
flwang1i mean add a role which has the swift only access02:00
flwang1it 'works'02:00
flwang1but the problem is, swift will take all the roles in operator_roles as an admin02:00
*** haomaiwang has quit IRC02:01
flwang1that said, the container acl won't work for this role02:01
*** haomaiwang has joined #openstack-swift02:02
*** Kennan has quit IRC02:02
*** Kennan has joined #openstack-swift02:02
hoflwang1: i understand. anonymous access with acl is a possibility02:04
flwang1ho: so is it possible to create a swift-only role, and make it honour container ACL?02:04
hoflwang1: let me summarize your situation. wait a minute.02:05
flwang1ho: ok02:05
*** annegentle has quit IRC02:06
hoflwang1: you created two roles for swift. one is for read, other one is for write. then you configured them in operator_roles in proxy-server.conf02:07
janonymous_o/02:08
flwang1ho: no, for now, I just created one role in Keystone, named swift_only, and i add it for "operator_roles" in proxy-server.conf02:08
flwang1because all the other openstack components don't know this new role, so it's swift only02:09
hoflwang1: then you access to a container which allows read only access with a user who has a role for write.02:09
hoflwang1: but the user will be allowed to access because the role in operator_roles.02:10
flwang1ho: i found all the user has 'swift-only' role can delete any object02:10
flwang1ho: yep02:10
*** kota_ has joined #openstack-swift02:10
*** ChanServ sets mode: +v kota_02:10
flwang1so i'm trying to figure out if there is a way to create a role in keystone and configure it in swift, which can still honour the container ACL02:11
flwang1that's my current question02:11
mattoliverauflwang1: anyone user in keystone with a role defined in operator_roles, will be an admin for there own account.02:11
flwang1mattoliverau: yep, i just realized that02:11
mattoliverauflwang1: so the ACL wont work when accessing there own account02:11
flwang1mattoliverau: so my new question is "so i'm trying to figure out if there is a way to create a role in keystone and configure it in swift, which can still honour the container ACL"02:11
mattoliverauflwang1: if that user was to access another account, or its another user in keystone, then the ACL will work.02:12
hrouflwang1:  Just create another tenant in keystone => different account in swift, and try to access that from the original tenant/account, you won't be able to without ACL02:13
mattoliverauflwang1: what hrou said ^ :)02:13
hrouWhich is what mattoliverau was saying too, I just reworded it ; )02:14
flwang1hrou: any way we don't have to create a new tenant?02:14
flwang1because it will may the billing process hard02:14
mattoliverauyou can create a user in keystone that doesn't have an role in operator_roles, then the ACLs will work on them.02:14
flwang1mattoliverau: but what's the role of the new user?02:15
hrouAh but without a new tenant, your going to be using the same account in swift, or try to change the role.02:15
flwang1ok, guys, let me repeat my user case02:15
*** nakagawamsa has joined #openstack-swift02:15
mattoliverauflwang1: so long as its a valid keystone user (so the token can be verified) and no roles that are in operator_roles, then anything02:15
flwang11. we want a swift only role, all the users with that role can only access swift, no other openstack components02:16
flwang12. based on #1, we would like to set read only access for some containers02:16
*** janonymous_ has quit IRC02:17
flwang1mattoliverau: ok02:17
hoflwang1: you create userA who has access role and userB who doesn't have access role in a project. you use userB for access and configure the userB info on a container.02:17
flwang1ho: access role = swift_only role?02:20
hoflwang1: yes.02:21
hoflwang1: the result said swift can't distinguish users' access by acl if the users are configured in operator-roles.02:21
flwang1ho: so seems we don't have to add the swift_only role in to the proxy-server.conf, right?02:24
mattoliverauflwang1: so until ho's patch merges, to give someone a read/write account they must have a role in operator_roles (and access to others via ACLs). Everyoneelse  in keystone can have access to different containers, etc, by ACLs.02:25
hoflwang1: i said so.02:25
flwang1mattoliverau: i'm confused now02:26
mattoliverauflwang1: you'll need to create containers and set the ACLs, so you need someone to have an account.02:26
flwang1mattoliverau: based on my test with the devstack master branch02:26
flwang1anyway, I will do the test again, unless i missed something. but before that, i'm going to write down my steps, so that you guys can review it02:27
mattoliveraugood idea, maybe we're all getting confused ;)02:27
flwang11. create role 'swift_only' in keystone02:28
flwang12. add the role in operator_role of proxy-server.conf, and restart proxy-server service02:28
flwang13. keystone user-create --name user_A --tenant demo02:30
flwang14. keystone user-create --name user_B --tenant demo02:30
*** bill_az has quit IRC02:30
flwang15. keystone user-role-add --user user_A --tenant demo --role swiftoperator02:31
*** jasondotstar has joined #openstack-swift02:31
flwang16. keystone user-role-add --user user_B --tenant demo --role swiftoperator02:31
flwang1i'm using swiftoperator instead of 'swift_only' sorry :)02:31
mattoliverauflwang1: dont you want to add them to swift_only02:31
mattoliveraulol02:31
flwang1hah02:31
mattoliverauflwang1: but you only want to add one of them to swift_only02:31
flwang1mattoliverau: really?02:32
flwang1so I don't have to add user_B for the role, right?02:32
hoflwang1: you dont need step 602:33
mattoliverauflwang1: well if you want to test ACLs02:33
flwang1mattoliverau: ok, until now, what i can see from keystone for user_B is like this02:33
flwang1http://paste.openstack.org/show/406331/02:34
mattoliveraubecause user_A will create a container and set the ACLs for suer_B. then user_B can access it02:34
flwang1it has a ROLE '_member_'02:34
flwang1so that means user_B can play with the other openstack components02:34
flwang1i think user_A has the similar issue02:35
*** jrichli has joined #openstack-swift02:35
*** yuanzz has quit IRC02:37
flwang1now i'm going to remove the _member_ role from user_A and user_B, any concern? guys?02:37
*** yuanzz has joined #openstack-swift02:37
hoflwang1: dont need to do it. i think swift doesn't care02:37
flwang1ho: yep, i believe swift doesn't care, but the other components care02:38
flwang1that's my basic line02:38
hoflwang1: i see02:38
flwang1we need a role which is swift only, that means the user with that role can't talk with the other components02:38
flwang1does that make any sense?02:38
flwang1we have several customers requiring that02:39
hoflwang1: make sense for me. you mange roles for each component.02:41
flwang1the result is shit02:41
*** jasondotstar has quit IRC02:41
flwang1for now, user_A only has one role, swiftoperator, but it can login, and create instance :(02:41
flwang1anybody know what i missed?02:42
flwang1seems by default, the policy of openstack components give a default user too much permissions02:42
hoflwang1: what is login? you use horizon something?02:43
flwang1ho: horizon02:43
flwang1i think command line will do the same thing02:43
flwang1and i just checked the policy.json of nova,  i got "compute:create": "",02:43
flwang1i think it means any user can create instance02:44
hoflwang1: i'm not sure but it's better to re-configure __menber__ on userA for horizon :)02:44
flwang1ho: __member__ has been removed from user_A02:44
flwang1now user_A only has the swiftoperator role02:44
hoflwang1: yeah, but swift can work with the user but other component not right?02:46
flwang1ho: i haven't try swift side :)02:46
flwang1let's do it now02:46
flwang1i will figure out how to keep the base line02:46
hoflwang1: better to separator the problem :-)02:46
openstackgerritTim Burke proposed openstack/swift: Improve content negotiation  https://review.openstack.org/20727602:47
flwang1actually, i confirmed my way with some keystone cores, they told me it works, but seems no02:47
flwang1ho: let's rock on02:47
*** changbl has quit IRC02:47
flwang1ho: now i'm going to create container_demo with user_A02:47
*** annegentle has joined #openstack-swift02:50
flwang1swift post --header 'x-container-read: AUTH_user_b' container_C02:50
flwang1is this the right command i should use?02:50
hoflwang1: you will create container_c?02:51
flwang1i have created02:51
flwang1and i'm going to set its acl02:52
flwang1with x-container-read02:52
hoflwang1: 'x-container-read: <project_id for demo>:<user_id for user_b> is better02:52
flwang1cool02:53
flwang1ho: http://paste.openstack.org/show/406332/02:54
flwang1it should be user_B, i have fixed it02:55
*** breitz has quit IRC02:55
hoflwang1: one question you specify id for project and user. i see "demo:user_b" for read-acl02:56
flwang1ho: oh, ok02:57
hoflwang1: both cases, name and id, work for your test but better to use id for keystone v302:58
flwang1ho: ok02:59
*** haomaiwang has quit IRC03:01
*** haomaiwang has joined #openstack-swift03:01
openstackgerritTim Burke proposed openstack/swift: Improve content negotiation  https://review.openstack.org/20727603:01
flwang1ho: yep, it woks03:01
flwang1so in my previous test, the key step is i shouldn't add role 'swiftoperator' for user_B03:02
*** SkyRocknRoll has quit IRC03:02
hoflwang1: i think so. the access to be authorized by acl in swift.03:03
flwang1ho: and I think we even don't need add the 'swiftoperator' into proxy-server.conf03:04
flwang1but there is another problem03:04
hoflwang1: yep!03:04
flwang1from the swift side, we still can't only accept a role access swfit03:05
flwang1because we don't have the policy.json support03:05
flwang1so we have to 'ask' the other components reject to accept a specific role, we named it as 'swiftonly' or something like that03:06
flwang1right?03:06
*** changbl has joined #openstack-swift03:06
hoflwang1: i have same understanding with you now.03:07
flwang1ho: cool, happy to see we're on the same page03:07
*** breitz has joined #openstack-swift03:07
flwang1ho: btw, do you think it's possible to hack something in keystone to achieve that?03:07
flwang1s/keystone/swift03:09
*** sanchitmalhotra has joined #openstack-swift03:09
hoflwang1: sorry achieve that means policy support?03:09
flwang1ho: we need it in juno :(03:12
flwang1so if there is no good way to achieve that, we may have to hack it a bit03:12
hoflwang1: commit to upstream then backport to juno. the change of  current code is almost keystoneauth.py so it's possible i think :-)03:14
flwang1ho: ok, will see.03:14
hoi have a business trip this afternoon so i will out of office soon. thanks for the info (what you want to do). it's useful for me. thanks!03:14
flwang1ho: thank you soooooooooooooooooo much for your support on my question03:15
flwang1have a good trip03:15
*** annegentle has quit IRC03:20
*** kei_yama has quit IRC03:21
*** kei_yama has joined #openstack-swift03:21
*** annegentle has joined #openstack-swift03:26
*** annegentle has quit IRC03:26
*** sanchitmalhotra1 has joined #openstack-swift03:29
*** sanchitmalhotra has quit IRC03:31
notmynamehello, again03:33
*** annegentle has joined #openstack-swift03:40
*** jamielennox|away is now known as jamielennox03:47
janonymousHi, Please vote on : https://www.openstack.org/summit/tokyo-2015/vote-for-speakers/Presentation/4570   , Thanks in advance  :)03:50
*** esker has joined #openstack-swift03:53
*** sanchitmalhotra has joined #openstack-swift03:59
*** haomaiwang has quit IRC04:01
*** sanchitmalhotra1 has quit IRC04:01
*** 16WAAEKLZ has joined #openstack-swift04:02
*** ppai has joined #openstack-swift04:04
*** aluria has quit IRC04:05
*** tamizh_g_ has joined #openstack-swift04:09
*** jrichli has quit IRC04:14
*** proteusguy has quit IRC04:16
mattoliveraunotmyname:  hi again!04:19
claygoh my good ness04:19
mattoliverauclayg: whats up?04:19
claygmattoliverau: no i was saw a bunch of scrollback04:20
notmynamethis was my view tonight https://twitter.com/notmyname/status/62658897675372134504:20
mattoliveraunotmyname:  wow, hard life your leading04:21
notmynamelol04:21
notmynameit's a really incredible view04:22
portantewould that be mount hood?04:22
mattoliverauit really is04:22
notmynamemt ranier04:22
portanteactive volcano,right?04:22
notmynamenot yet ;-)04:22
claygnotmyname: don't get dead04:23
notmynamesuper clear today and as the sun was starting to go down the colors really popped04:23
portante"Geologists consider this mountain to be an 'episodically active' volcano, meaning one that will erupt again some time in the future even though it may be quiet now. Mount Rainier is the tallest volcano and fifth highest peak in the contiguous United States."04:23
notmynameclayg: I'll do my best04:23
mattoliverauLooks a bit like fuji-san (mount fuji)04:23
portantemay you have no episodes04:24
notmynameportante: if you want to be really freaked out, read http://www.newyorker.com/magazine/2015/07/20/the-really-big-one04:24
mattoliveraumay you be up wind if it does (and far enough away)04:25
portantenotmyname: okay, I really really love New England04:28
notmynamelol04:28
*** ccavanna has joined #openstack-swift04:30
notmynameok, so what do I need to look at in swift tonight before going to bed?04:30
notmynameah, ccavanna just joined. his patch would be easy to review04:30
ccavannahi.04:30
ccavannaYes, I was hoping for that :-)04:30
mattoliverauccavanna: o/04:30
ccavannaActually, I have two patches.04:30
ccavannaOne for documentation, another for the improvements to statsD.04:31
ccavannaAnd I wanted to ask for some suggestions on what I could review myself.04:31
*** blmartin has joined #openstack-swift04:32
notmynameccavanna: do you see the "review dashboard" link in the channel topic?04:33
ccavannaNow that you mention it, yes.04:33
notmynameccavanna: when you open that, you'll see a bunch of swift-related patches. the top section there is "starred patches". those are the priority patches to review04:33
*** jamielennox is now known as jamielennox|away04:33
ccavannaGreat. Thanks.04:34
openstackgerritJohn Dickinson proposed openstack/swift: Swift documentation for first-time contributors.  https://review.openstack.org/20686104:40
notmynameccavanna: it looks great04:40
notmynameI took the RDO link fix that tdasilva had and rolled it in04:41
notmynameand approved04:42
*** sakaYK has joined #openstack-swift04:42
ccavannaGreat, thank you.04:43
ccavannaOne question, how do I move forward my other change?04:43
ccavannaI addressed all the comments as well.04:43
ccavannaDo I just wait?04:44
*** annegentle has quit IRC04:44
notmynameyup04:44
notmynameand if/when you don't see anything on it, you come in here and bug people ;-)04:44
ccavannanotmyname: That was the dilemma, how much "bugging" was reasonable :-)04:46
claygccavanna: if you don't make it to targeted it will sorta roll around04:48
clayglike if you jump in channel every day and say "hey can someone look at xyz" that will normally spur someone who is feeling particularlly guilty about being behind04:49
claygif someone leaves a review and you address *their* comments in particular - it's not really "bugging" if you say "can you validate that I addressed your concern"04:49
claygit's just a friendly and helpful reminder04:49
claygwhich we all need from time to time04:49
* clayg hopes i'm not setting myself up to review ccavanna's patch tonite04:50
ccavannaWell ... I can be very convincing ... and since you offered to ... :-) but fortunately for you it is late in Toronto.04:52
*** darrenc is now known as darrenc_afk04:56
*** blmartin has quit IRC04:58
ccavannaWell, in case somebody (more awake than I am) sees this ...04:58
ccavannaCan anyone review https://review.openstack.org/#/c/202657/ ?04:58
*** tamizh_g_ has quit IRC04:58
ccavannaIt's a proposed change to improve statistics in Graphite.04:59
mattoliveraunotmyname: if you still want another patch to review, patch 195131 is important for my fellow rackspace private cloud guys. (they've poked me).04:59
patchbotmattoliverau: https://review.openstack.org/#/c/195131/04:59
ccavannaclayg: thanks for the tips. I'm still getting used to the dynamic.04:59
notmynamemattoliverau: ah, thanks for the info on that. (to be fair, I'm not going to be reviewing any keystone stuff tonight--too tired to do the necessary setup to test it)05:00
mattoliverauccavanna: I'll take a look, I'm suppose to be more awake.. but not really firing on all cylinders :)05:00
mattoliveraulol05:00
mattoliveraufair enough :)05:00
*** 16WAAEKLZ has quit IRC05:01
ccavannamattoliverau: me neither, long week + it's late.05:01
mattoliverauccavanna: well its only 3pm here05:01
*** haomaiwang has joined #openstack-swift05:02
claygthis seems right, but really needs some functional testing.05:04
claygw/c05:04
ccavannamattoliverau: I don't want to look at my watch ... :-)05:04
*** flwang1 has quit IRC05:04
ccavannaAnyway, I'm really going to sleep. Up early tomorrow. On course all day. Thanks for the tips.05:05
claygmattoliverau: does devstack setup v3 and domains and stuff?05:05
ccavannaI'll start reviewing one patch.05:05
mattoliverauccavanna: night05:05
*** tamizh_g_ has joined #openstack-swift05:08
mattoliverauclayg: devstack.. *shudder* I'm not sure, I just set up my own keystone servers. (yay public cloud), which I re-use for testing.05:09
notmynamemattoliverau: have you talked to alex or lana lately? there was talk in vancouver about reviewing the swift docs and getting a plan for improving them, but I haven't seen anything05:10
claygdo you have a playbook or something that will setup install keystone - or you just set it up once and then use it?05:10
claygwhoa - double barrell questions05:11
hrouclayg yep !  Fairly sure it does, you just need to change openrc to use the V3 end points, I'll give it a go.05:11
clayghrou: whoa - awesome - I think05:12
mattoliverauclayg: dplph had a github repo what set up keystone (but it is old) so used that and updated it a little to make it work. I'll clone it in my github and push up my fixes so it can be used to set up keystone ready to play with swift.05:13
*** sakaYK has quit IRC05:14
claygmattoliverau: sure - could work05:15
hroumattoliverau, that'd be great, seems a lot of folks start off with swift SAIO, and it's nicer just to add keystone.05:15
openstackgerritCharles Hsu proposed openstack/swift: Remove error_suppression_interval, error_suppression_limit options.  https://review.openstack.org/20729605:15
hrouIn theory this should do it but I haven't tried it:  https://github.com/Open-I-Beam/swift-install05:15
claygcharz: remove options!?  that's not nearly as fun as adding them!05:15
mattoliveraunotmyname: alex is holidaying in vietnam.. that's probably the delay05:16
notmynameok05:16
mattoliverauI can poke lana tho ;)05:16
notmynamethis "holiday" concept. what is that? seems that all the non-US people seem to think it's a thing05:16
notmynameand clayg and I are here after 10pm05:16
notmyname(and I'm technically on half-day PTO this week) ;-)05:17
claygnotmyname: we're the weird ones tho - the rest of the world is normal05:17
mattoliveraulol, yeah you guys are crazy05:17
mattoliverauwe Aussies love our holidays05:17
notmynamemattoliverau: but it's my only chance to talk to you05:17
* notmyname is sad that his work hours rarely overlap with cschwede these days05:17
charzclayg: That's in account-server.conf-sample. I think that's belong to proxy-server.05:18
claygcharz: yeah i think so too - are their other references - man pages, docs?05:19
claygthey're in the proxy conf example yeah?05:19
charzclayg: yes, they're in the proxy conf, and deployment docs is good.05:19
claygcharz: did you see my notes on the .durable bug thread?05:20
claygwell... *your* the one taking notes - I'm just throwing out ideas from the side lines :\05:20
charzclayg: yeah, I saw that.05:20
claygcharz: you buying it?05:20
charzclayg: yes, I'm thinking how to repro that in a small cluster even VM.05:21
hrouclayg, yep just replace OS_AUTH_URL=*/v3 and OS_IDENTITY_API_VERSION=v3 .  Of course the keystone CLI doesn't support v2, only the openstack CLI does.05:22
notmynamecharz: looks like those options are also in man/account-server.conf.5 and need to be removed there too...or I could just fix the patch05:22
charzclayg: first, we need to figure out why the .durables missing in high concurrency or load, and able to reproduce that.05:23
charznotmyname: I can do that.05:23
notmynamecharz: no, you do the EC stuff with clayg. that's more important ;-)05:23
notmynameI've already got this open05:23
hrou* keystone CLI only supports v2.05:23
charznotmyname: k05:23
claygcharz: you want me to fix the man page - or do you wanna fix it and I'll +2 :P05:24
claygcharz: for patch 20729605:24
patchbotclayg: https://review.openstack.org/#/c/207296/05:24
claygnotmyname: do you have any idea how to build manpages?  there's all that weird syntax - every time I touch one I try to copy whats near by - but I have no idea05:24
charzclayg: I do that.05:24
clayghrou: nice!05:25
claygnotmyname: oh you beat me too it05:25
notmynameclayg: nope. I do the same05:25
claygnotmyname: how did they even get there!?05:25
claygnotmyname: marcelo?05:25
notmynameyup :-)05:25
claygnice05:25
claygwhat was his handle... btorch?05:26
openstackgerritJohn Dickinson proposed openstack/swift: Remove error_suppression_interval, error_suppression_limit options.  https://review.openstack.org/20729605:26
notmynameyes05:26
claygredbo: what ever happened to btorch05:26
notmynamefortunately in this case it was just deleting some parts of the man pages05:26
notmynameit's really easy to delete stuff05:26
charznotmyname: you're winner. :)05:26
claygnotmyname: also - correction - charz is not doing "EC stuff with clayg" - charz is doing "critical EC testing" - clayg is goofing off - as usual05:26
*** SkyRocknRoll has joined #openstack-swift05:26
notmyname:-)05:27
notmynameI just didn't want to distract the conversation with this. bigger fish, ya know?05:27
claygBIG OL FREAKING BASS BABY!05:27
charzclayg: I'll try to test in with benchmark tool on VMs and disable reconstructors.05:27
*** zaitcev has quit IRC05:28
claygcharz: is that our best bet?  did we alreay repurpose everyhting in the lab?  I had assumed those 5 nodes were still available to you.05:28
claygmaybe it's cheap and easy to reproduce on vms - in which case we'll do that05:29
mattoliveraunotmyname: well in a week and a bit well all be locking in an airconditioned room in Texas ;) (sorry for late reply got stuck on a call).05:29
claygI could probably crank up some knobs on my swift-bench config until my saio starts dropping .durables05:29
charzclayg: yeah, if I can reproduce in VMs, you can fix it in a easy way. is it correct?05:30
notmynameclayg: the 5 nodes are back in as gate checks (and charz is adding another for functests w/ default EC) right now05:30
claygumm... yeah - let's go with that05:30
claygwell..05:30
notmynameso they could be used for the EC stuff again, but they aren't set up for that currently05:30
claygi mean I really just want full transaction log for one of those requests that resulted in a frag with no durable05:31
claygnotmyname: charz: ok sure np05:31
notmynamecharz: any chance we could pull it from the archived logs? I'm not sure if we have enough to find it05:32
claygnotmyname: i was thinking you'd have to work your way up from a hashdir with no .durable -> swift-object-info -> name -> log -> transaction_id05:32
claygyou know - like you do05:32
notmynameas one does05:33
charznotmyname: I'll try, but no guarantee. Maybe I'll turn qacluster to test this if I need.05:34
claygcharz: do what you think is best05:34
notmynamecharz: no, we'd have to start with data in the cluster. no need to start with logs to guess at something like this05:34
redboclayg: He's still around.  He moved to austin and works remote.05:34
claygredbo: nice!  does he know how man pages work?  can he come to the hack-a-thon and do a brown bag.  you're up late.05:35
*** kei_yama_ has joined #openstack-swift05:36
*** kei_yama has quit IRC05:36
redboIt's not late yet.  I'll be up late here in a few hours.05:36
claygwell look at that - a GET that 404's has a body (html w/e) and a content-length (obviously) but if you make the same request with the verb HEAD - you get a different content-length (0 this time, obviously)05:37
charznotmyname: got it.05:38
claygi guess the rule about GET's and HEAD's having the same headers only applies to 200 series responses05:38
redboisn't that a SHOULD, not a MUST?05:39
*** darrenc_afk is now known as darrenc05:40
claygredbo: having the same headers - yeah that's a should - but I don't think it applies to 404's?  or yeah - maybe it just doesn't matter05:47
notmynameit's late for me. I'm going to bed. talk to you tomorrow05:48
*** ppai has quit IRC05:51
openstackgerritAzhagu Selvan SP proposed openstack/swift: Respect 'Accept' header in error responses  https://review.openstack.org/20419605:54
*** tamizh_g_ has quit IRC05:57
*** tamizh_g_ has joined #openstack-swift06:00
*** haomaiwang has quit IRC06:01
*** jith_ has quit IRC06:01
*** haomaiwang has joined #openstack-swift06:02
*** Kennan2 has joined #openstack-swift06:03
*** Kennan has quit IRC06:03
*** ppai has joined #openstack-swift06:05
claygkota_: looking at the fix content-lenght on 416 for ec06:06
claygkota_: working on a test to describe teh failure i'm seeing06:07
*** hrou has quit IRC06:09
claygdid we ever get to run functests against non-default policy or do we still just change the default policy?06:13
*** esker has quit IRC06:13
claygok SWIFT_TEST_POLICY=ec seems to do the trick06:18
*** sakaYK has joined #openstack-swift06:25
kota_clayg: I noticed your comment now.06:26
kota_clayg: night man06:26
kota_oh yeah, the trick recently landed on master.06:29
kota_i guess, we did but not sure. ok let's run functional agaisnt to ec now.06:30
kota_or if you have failure tests already, please feel free to comment on gerrit. I wanna try to make sure also in my env.06:33
*** sanchitmalhotra1 has joined #openstack-swift06:43
*** david-lyle has quit IRC06:43
*** sanchitmalhotra has quit IRC06:45
claygkota_: oh hi :)06:47
*** david-lyle has joined #openstack-swift06:49
kota_i just was reading versioned_writes middleware.06:50
kota_in paralell, I launched an all-in-one with vagrant at AWS for the functional checking :P06:51
kota_clayg: and hi :) (might be too late)06:52
*** bkopilov has quit IRC06:52
claygkota_: nice ;)06:53
claygkota_: do you have ... oh AWS + vagrant!?  you're in crazy town man!06:53
kota_yey06:55
kota_clayg: based on your vagrant-all-in-one06:55
claygheh, i've never tried the vagrant cloud stuff - very cool06:55
*** bkopilov has joined #openstack-swift06:55
kota_clayg: here, https://github.com/bloodeagle40234/vagrant-swift-all-in-one/tree/aws06:56
kota_clayg: note I didn't make enough documentation for that :(06:57
clayghey - it has a commit message06:57
kota_ya, just memo for me :P06:58
claygkota_: looks pretty sweet - i bet we could get that baked into the mainline with just a few tweeks and some extra localrc options?07:00
claygneway - yeah versioned middleware - good call07:00
*** haomaiwang has quit IRC07:01
claygI'm fartin' around with haypo's patch now - let me know if you have some thoughts on the 416 ec range etag thing (i left you a comment to ponder)07:01
kota_which one?07:01
kota_mine?07:01
*** haomaiwang has joined #openstack-swift07:02
kota_oh, you mean you moved from the 416 thing to haypos07:02
*** mahatic has joined #openstack-swift07:02
claygkota_: yessir07:03
claygkota_: I'm in pep8 crazy town now!07:03
kota_clayg: oic, I think so too, that's crazy town looking at today's meeting.07:04
claygyeah it was pretty nuts07:05
*** rledisez has joined #openstack-swift07:10
haypohello07:17
kota_haypo: hello07:20
haypodoes swift juno support Python 2.6?07:22
haypohum, i still see python26 on https://review.openstack.org/#/c/207132/ ("Fixes for mock 1.1" for juno)07:22
haypooh, i remember that mock 1.1 dropped python 2.6 support, but i came back later :)07:23
hayposo it's ok, nevermind ;)07:23
*** mahatic has quit IRC07:23
kota_i think so. python2.6 support should drop after Kilo release.07:24
kota_oh, interesting for patch 20713207:24
patchbotkota_: https://review.openstack.org/#/c/207132/07:24
*** chlong has quit IRC07:25
haypokota_: "python26: SUCCESS", it's ok ;)07:25
openstackgerritOndrej Novy proposed openstack/python-swiftclient: Test auth params together with --help option.  https://review.openstack.org/20732207:25
kota_lol07:26
kota_haypo: sounds good.07:26
haypokota_: yeah, i reviewed my own patch, it looks good to me :-D07:27
haypokota_: (well, it's a backport made by notmyname, and juno is different)07:27
*** ho has quit IRC07:28
claygso is jchroll not the guy for pep8 anymore?  PyCQA?07:29
openstackgerritVictor Stinner proposed openstack/swift: Fix pep8 E265 warning of hacking 0.10  https://review.openstack.org/20724207:34
openstackgerritVictor Stinner proposed openstack/swift: Fix warning pep8 E128 warning of hacking 0.10  https://review.openstack.org/20724307:34
openstackgerritVictor Stinner proposed openstack/swift: Update hacking to 0.10.0  https://review.openstack.org/20597707:34
openstackgerritVictor Stinner proposed openstack/swift: Fix pep8 E warning for hacking 0.10  https://review.openstack.org/20723707:34
haypoclayg: you are right for "# ####" and E265: https://review.openstack.org/207242 -- i modified my patch07:35
onovyhi, just quick review, someone pls? :) https://review.openstack.org/#/c/207322/07:35
clayghaypo: yeah but I +2'd already - now I gotta go back and do it again - i was hoping to sleep tonite07:36
kota_wow, I'm running out of time to stay my office, see you again.07:36
haypoclayg: it's the morning for me :)07:37
haypoclayg: next time, approve directly the patch :-D07:38
*** ppai has quit IRC07:38
haypoclayg: (thanks for reviewing these stuff)07:38
haypoclayg: what's your timezone? (where do you live?)07:38
haypoi live at the south of france (CEST)07:39
onovyCEST and morning here too :)07:39
haypoi'm asking because i remember someone saying that the swift meeting was at 02:00pm if i recall correctly07:40
haypoi hate timezones :)07:40
haypo21:22:13 <peluse_> Im' tired too - its 2:21pm here :)07:41
claygnotmyname: acoles_away: if I get frustrated and start +A'ing hypos stuff you guys can kick me out of the club07:41
*** kota_ has quit IRC07:41
claygi'ts 7:50 UTC here same as everywhere else07:41
claygmy residence is in SF, CA, USA - I don't keep very normalized hours07:42
haypoah wait, pm if before noon. i also hate am/pm, sorry :)07:42
haypoclayg: haha07:42
-openstackstatus- NOTICE: Our CI system is broken again today, jobs are not getting processed at all.07:42
*** ChanServ changes topic to "Our CI system is broken again today, jobs are not getting processed at all."07:42
*** flwang1 has joined #openstack-swift07:46
openstackgerritOndrej Novy proposed openstack/python-swiftclient: Test auth params together with --help option.  https://review.openstack.org/20732207:49
*** flwang1 has quit IRC07:50
-openstackstatus- NOTICE: CI system is broken and very far behind. Please do not approve any changes for a while.07:52
*** ChanServ changes topic to "CI system is broken and very far behind. Please do not approve any changes for a while."07:52
*** ppai has joined #openstack-swift07:52
*** geaaru has joined #openstack-swift07:54
*** jordanP has joined #openstack-swift07:55
openstackgerritOndrej Novy proposed openstack/python-swiftclient: Block comment PEP8 fix.  https://review.openstack.org/20584907:56
*** jamielennox|away is now known as jamielennox07:57
*** tamizh_g_ has quit IRC08:00
*** haomaiwang has quit IRC08:01
*** haomaiwang has joined #openstack-swift08:02
*** ppai has quit IRC08:05
*** tamizh_g_ has joined #openstack-swift08:07
*** ho has joined #openstack-swift08:11
*** acoles_away is now known as acoles08:15
acolesclayg: np i'm going to look at them again today08:19
*** ppai has joined #openstack-swift08:19
acoleshaypo: ack will review today. thanks again for the work.08:19
haypoacoles: no problem08:20
haypoacoles: i'm sorry for my behaviour, i put too much energy on python3, i'm getting tired. i'm in holiday in two days, i want to reach the milestone of a voting py34 gate for swift :-p08:21
acoleshaypo: no need for apologies! sounds like a good time for a holiday :)08:23
*** SkyRocknRoll has quit IRC08:25
*** jistr has joined #openstack-swift08:25
haypoacoles: it takes me extra efforts to justify my work, and acecpt that other developers want to modify the code differently than me :)08:25
clayghaypo: acoles puts up with me - so you're probably fine08:25
haypoacoles: i only fixed E checks, 2 F checks and many H checks are still ignored. are you fine with that?08:26
haypoacoles: i don't know if the 2 F checks are new or not08:27
acoleshaypo: i expect i will be fine with that08:27
acoles;)08:27
acolesclayg: up late again?08:28
haypo(note: it looks like the CI is sick, the infra team asked to "not approve any changes for a while" :-p)08:29
acoleshaypo: if you have N swift developers in a room you will get at least N+1 opinions :P08:29
haypoacoles: lol08:29
acoleshaypo: and I'm probably responsible for the extra 108:29
haypoi may propose a patch to sort and group imports. when i had to add six imports, it took me a while to find where to put it08:30
haypobut i know that some people hate coding styles :)08:33
*** ppai has quit IRC08:45
*** aix has joined #openstack-swift08:47
*** ppai has joined #openstack-swift08:57
*** sasi has joined #openstack-swift08:57
*** sasi has left #openstack-swift08:58
*** haomaiwang has quit IRC09:01
*** ChanServ changes topic to "Review Dashboard: http://goo.gl/8IUcKl | Summary Dashboard: http://goo.gl/qHus5v | Hackathon topics: https://etherpad.openstack.org/p/swift-midcycle-aug-2015 | https://etherpad.openstack.org/p/swift_encryption_issues | Logs: http://eavesdrop.openstack.org/irclogs/%23openstack-swift/"09:01
-openstackstatus- NOTICE: CI is back online but has a huge backlog. Please be patient and if possible delay approving changes until it has caught up.09:01
*** haomaiwang has joined #openstack-swift09:02
*** silor has joined #openstack-swift09:02
*** chlong has joined #openstack-swift09:06
*** jamielennox is now known as jamielennox|away09:25
*** tamizh_geek has quit IRC09:27
*** ppai is now known as ppai|afk09:29
*** ig0r_ has joined #openstack-swift09:31
*** tamizh_geek has joined #openstack-swift09:33
*** sakaYK has quit IRC09:34
*** sanchitmalhotra has joined #openstack-swift09:41
*** tamizh_g_ has quit IRC09:41
*** tamizh_g_ has joined #openstack-swift09:41
*** sanchitmalhotra1 has quit IRC09:43
*** marzif_ has joined #openstack-swift09:57
*** haomaiwang has quit IRC10:01
*** haomaiwang has joined #openstack-swift10:01
openstackgerritHiroshi Miura proposed openstack/python-swiftclient: Update flake8 config  https://review.openstack.org/20737210:02
acolesclayg: you still here?10:08
*** SkyRocknRoll has joined #openstack-swift10:12
*** jasondotstar has joined #openstack-swift10:15
*** ho has quit IRC10:19
*** sanchitmalhotra1 has joined #openstack-swift10:27
*** sanchitmalhotra has quit IRC10:29
openstackgerritHiroshi Miura proposed openstack/python-swiftclient: Update flake8 config  https://review.openstack.org/20737210:33
*** sanchitmalhotra has joined #openstack-swift10:37
*** aix has quit IRC10:39
*** sanchitmalhotra1 has quit IRC10:39
*** sanchitmalhotra1 has joined #openstack-swift10:42
*** sanchitmalhotra has quit IRC10:44
*** sanchitmalhotra has joined #openstack-swift10:49
*** sanchitmalhotra1 has quit IRC10:51
*** haomaiwang has quit IRC11:01
*** 7GHAAT9JB has joined #openstack-swift11:02
*** sanchitmalhotra1 has joined #openstack-swift11:09
*** sanchitmalhotra has quit IRC11:11
*** aix has joined #openstack-swift11:13
*** jasondotstar has quit IRC11:19
openstackgerritAlistair Coles proposed openstack/swift: Refactor diskfile  https://review.openstack.org/19842911:31
openstackgerritAlistair Coles proposed openstack/swift: Add ability for GET path to see/select alternate frag archs  https://review.openstack.org/20716511:31
acolespeluse: ^^ fixed a merge conflict. I guess if you like the refactor you could just abandon your existing review and adopt this one.11:32
*** cdelatte has quit IRC11:35
*** dmorita has quit IRC11:38
*** swat30 has quit IRC11:45
*** jordanP has quit IRC11:46
*** swat30 has joined #openstack-swift11:46
*** marcusvrn has joined #openstack-swift11:47
*** jordanP has joined #openstack-swift11:47
*** tsubic has quit IRC11:51
*** cdelatte has joined #openstack-swift11:51
*** delattec has joined #openstack-swift11:51
peluseacoles, thanks.  that's the plan as soon as I get some tie to go through it - looking like tomorrow or this weekend11:52
*** 7GHAAT9JB has quit IRC12:01
*** haomaiwang has joined #openstack-swift12:02
*** cdelatte has quit IRC12:03
*** delattec has quit IRC12:03
*** tsubic has joined #openstack-swift12:05
*** lpabon has joined #openstack-swift12:07
*** km has quit IRC12:09
tsubic`2`12:16
*** tsubic has left #openstack-swift12:17
*** jasondotstar has joined #openstack-swift12:22
*** SkyRocknRoll has quit IRC12:35
*** mahatic has joined #openstack-swift12:35
*** jamielennox|away is now known as jamielennox12:35
*** ccavanna has quit IRC12:49
*** ppai|afk is now known as ppai12:56
*** jkugel has joined #openstack-swift12:59
*** haomaiwang has quit IRC13:01
*** haomaiwang has joined #openstack-swift13:02
*** jamielennox is now known as jamielennox|away13:06
*** nakagawamsa has quit IRC13:09
*** ig0r_ has quit IRC13:10
*** hrou has joined #openstack-swift13:12
*** lpabon has quit IRC13:13
*** ig0r_ has joined #openstack-swift13:16
*** NM has joined #openstack-swift13:16
*** marzif_ has quit IRC13:21
*** marzif_ has joined #openstack-swift13:22
*** marzif_ has quit IRC13:22
*** lpabon has joined #openstack-swift13:22
*** rbrooker has joined #openstack-swift13:23
*** bill_az has joined #openstack-swift13:25
*** jasondotstar has quit IRC13:30
*** ig0r_ has quit IRC13:31
*** kei_yama_ has quit IRC13:33
*** ccavanna has joined #openstack-swift13:36
*** annegentle has joined #openstack-swift13:43
*** joeljwright has joined #openstack-swift13:51
*** ChanServ sets mode: +v joeljwright13:51
*** annegentle has quit IRC13:55
*** haomaiwang has quit IRC14:01
*** thurloat_isgone is now known as thurloat14:02
*** haomaiwang has joined #openstack-swift14:02
thurloathas anyone launched a reasonably scaled swift setup using SSDs for storage?14:02
*** sanchitmalhotra1 has quit IRC14:04
*** Kennan2 has quit IRC14:16
*** Kennan has joined #openstack-swift14:16
*** breitz has quit IRC14:22
*** ppai has quit IRC14:23
*** annegentle has joined #openstack-swift14:24
*** ig0r_ has joined #openstack-swift14:31
*** mahatic has quit IRC14:34
*** jlhinson has joined #openstack-swift14:35
*** rbrooker has quit IRC14:38
*** jrichli has joined #openstack-swift14:42
*** joeljwright has quit IRC14:43
*** bapalm_ has joined #openstack-swift14:43
*** lcurtis has joined #openstack-swift14:52
*** breitz has joined #openstack-swift14:55
*** minwoob has joined #openstack-swift14:59
*** haomaiwang has quit IRC15:01
*** rbrooker has joined #openstack-swift15:02
*** haomaiwang has joined #openstack-swift15:02
*** ig0r_ has quit IRC15:03
*** ig0r__ has joined #openstack-swift15:03
*** ig0r_ has joined #openstack-swift15:08
*** logan2 has quit IRC15:08
*** ig0r__ has quit IRC15:09
*** ig0r__ has joined #openstack-swift15:12
*** ig0r__ has quit IRC15:15
lcurtisq for all...if i lose a drive...does swift auditor detect the failure and start replicating elswhere or do i have to replace drive first and remount as same device for replication to start occuring15:15
*** ig0r_ has quit IRC15:15
jrichlilcurtis: replication to the handoff node is triggered at that point.  so you don't have to replace the drive to get full replication again.15:19
lcurtisjrichli...what exact action kicks off that replication15:20
lcurtisi am checking logs on server with failed drive15:20
lcurtisdont see anything replicating15:21
glangethe replicator daemon is the part that does replication -- it walks the disk and tries to push everything it finds to the right places15:21
glangereplication on a particular partition happens when it is found on disk15:22
lcurtisokay...so replicator on another server would find partition and notice missing and replicate?15:22
redboMaybe.  The drive has to be unmounted, and you have to have mount checks enabled.15:23
glangeit would check the to see if the file is in the right three places, and try to push to places where the file should be15:23
redboAlso having data on handoff nodes for long periods of time isn't ideal.  You should replace the drive or remove it from the ring.15:23
lcurtisdefinitely...we are replacing immediately15:23
glangethen the files should be written back to it when a replication pass is done15:24
glangenothing really "knows" that the drive is gone or empty -- the replicator just tries to dumbly push objects to the right place over and over again15:24
lcurtisbut the replicator might run on another server and check that replica is not available on deviceX?15:26
lcurtisand THAT would force a replication?15:26
tdasilvaglange: so until you replace that drive, you might only have 2 copies?15:26
lcurtissorry...trying to understand in detail15:26
redboIf the replicator sees that a drive is unmounted, it'll switch to the handoff.15:26
lcurtisthe replicator on that server with failed drive or from another15:27
redbofrom another server that has a copy of that data15:27
lcurtisawesome15:28
lcurtisthank you15:28
lcurtisvery much folks15:28
glangeit's always best to replace the drive fast :)15:28
lcurtisyep! we are indeed15:28
*** cppforlife_ has quit IRC15:29
lcurtiswe do have mount checks enabled15:29
notmynamegood morning15:29
*** zul has quit IRC15:31
lcurtishello notmyname!15:33
pelusemornin'15:36
peluseso FYI Mahati and one of our guys here did some S3 compatiblity testing with Swift and Ceph and have engaged the swift3 folks (Kota and others) as well as the S3 test repo maintainer...15:37
peluseinteresting results, they're still comparing notes on configs, etc., but once that's straightened out and there's some confidence in the results we'll post here somewhere15:37
peluseclayg, take a look at the missing .durables bug when you get online.  looks like we might be seeing the same thing here in Phx on the performance cluster15:38
lcurtis@all...what would be most reliable way to determine all partitions back to normal on a replaced drive?15:41
*** jasondotstar has joined #openstack-swift15:42
*** rbrooker has quit IRC15:45
lcurtisor i should say objects15:46
notmynamelcurtis: you can't really know that, since that would require finding things that are not there. so in practice you look at the replication logs15:49
*** joeljwright has joined #openstack-swift15:49
*** ChanServ sets mode: +v joeljwright15:49
lcurtisnotmyname okay great...thats what i am doing...just wanted to make sure not overlooking something15:50
glangelcurtis: there is also something called the dispersion report -- basically you seed a bunch of small objects all over your cluser and it can check to see if they are in the right place15:51
lcurtisthank you15:51
lcurtisglange...ah okay15:51
lcurtismakes sense15:51
glangehttp://docs.openstack.org/developer/swift/admin_guide.html <-- look at the "cluster health" section15:51
*** ig0r_ has joined #openstack-swift15:52
notmynameoh yeah. the dispersion report would be good for that15:54
ahalegeneral system graphs are useful too, seeing a drives disk usage increase and plateau with ring weight increases etc.15:54
lcurtismake sense15:55
*** Protux has quit IRC15:55
*** h00327910__ has quit IRC15:55
*** marcusvrn has quit IRC15:55
*** rbrooker has joined #openstack-swift15:58
*** minwoob_ has joined #openstack-swift15:59
*** minwoob has quit IRC16:00
*** haomaiwang has quit IRC16:01
*** haomaiwa_ has joined #openstack-swift16:02
*** thumpba has joined #openstack-swift16:04
*** nadeem has joined #openstack-swift16:04
*** nadeem has quit IRC16:05
*** nadeem has joined #openstack-swift16:06
*** rbrooker has quit IRC16:06
*** mfalatic has quit IRC16:07
*** rbrooker has joined #openstack-swift16:08
*** zul has joined #openstack-swift16:10
tdasilvapeluse: were the s3 tests a comparison between the compatibility differences between swift and ceph rgw? or just swift3 and the latest s3 api?16:12
*** rbrooker has quit IRC16:14
*** bhakta has quit IRC16:16
*** zul has quit IRC16:16
*** ajiang has quit IRC16:16
*** logan2 has joined #openstack-swift16:19
*** rledisez has quit IRC16:21
*** ajiang has joined #openstack-swift16:21
*** bhakta has joined #openstack-swift16:22
*** jasondotstar has quit IRC16:23
*** mfalatic has joined #openstack-swift16:23
*** jistr has quit IRC16:24
openstackgerritMichael Barton proposed openstack/swift: go: fix expecttransport memory usage  https://review.openstack.org/20627616:25
*** zhill has joined #openstack-swift16:25
*** marcusvrn has joined #openstack-swift16:26
*** zaitcev has joined #openstack-swift16:26
*** ChanServ sets mode: +v zaitcev16:26
*** zul has joined #openstack-swift16:28
*** aix has quit IRC16:31
*** cppforlife_ has joined #openstack-swift16:33
*** zul has quit IRC16:33
*** jordanP has quit IRC16:36
*** zul has joined #openstack-swift16:42
*** zhill has quit IRC16:48
claygpeluse: yeah I think we're onto something - dunno why with all this hashdirs that don't have durables in them we can't get back to the transaction ids that put them there?16:52
claygacoles: nope16:53
*** jordanP has joined #openstack-swift16:55
acolesclayg: heh. np it wasn't important16:56
claygacoles: k16:57
claygacoles: did we merge any pep8/flake8/hacking fixes last night!?16:57
acolesclayg: I +A'd all the pep8 ones and i'm +2 on the hacking version bump patch 20597716:59
patchbotacoles: https://review.openstack.org/#/c/205977/16:59
acolestorgomatic had a +2 on that a few versions back ^^16:59
acolesdon't think i have seen any land but zuuls been struggling17:00
claygacoles: oh right17:00
*** haomaiwa_ has quit IRC17:01
*** haomaiwang has joined #openstack-swift17:02
acoleslooks like 205977 is the only one on haypo's pep8 topic that is not WIP and not approved17:03
acolesyet17:03
*** robefran has joined #openstack-swift17:04
notmyname12 patches in the gate queue17:05
haypoacoles: yes17:05
haypoi will rebase my other pep8 patches when this first patch serie is merged17:06
*** silor has quit IRC17:07
*** Protux has joined #openstack-swift17:08
*** h00327910__ has joined #openstack-swift17:08
peluseclayg, yeah, odd.  Just asked our BM guy to up the timeouts (conn and node both) and restart the tests, will see if that makes our errors go away17:13
peluseBTW you may recall some discussion ages ago wrt the ecrecon at least about possibly bumping these based on increased connections.  there's even still a comment in the code there in the ecrecon...17:15
peluse        # defaults subject to change after beta17:16
peluse        self.conn_timeout = float(conf.get('conn_timeout', 0.5))17:16
peluse        self.node_timeout = float(conf.get('node_timeout', 10))17:16
acolesho: i took a look at your test patch - can you tell me which scenario you are concerned about?17:16
*** bapalm_ has quit IRC17:25
*** zhill_desktop has quit IRC17:29
claygpeluse: I don't think it's the reconstructor that's iming out tho?17:32
openstackgerritThiago da Silva proposed openstack/swift: versioned writes middleware  https://review.openstack.org/13434717:34
claygwhy does everything think their linting changes are helping?  https://review.openstack.org/#/c/207372/217:34
claygi'm heading in17:34
*** zhill_desktop has joined #openstack-swift17:36
redboI made a thing like hacking one time that all it did was look for "if not x == y:" and told you to change it to "if x != y:".  I think we should gate on that.17:39
*** joeljwright has quit IRC17:40
*** robefran_ has joined #openstack-swift17:41
*** robefran_ has quit IRC17:41
*** acoles is now known as acoles_away17:42
openstackgerritMinwoo Bae proposed openstack/swift: EC: Handoff node to push existing fragment to the correct location.  https://review.openstack.org/19684817:56
*** haomaiwang has quit IRC18:01
*** haomaiwang has joined #openstack-swift18:02
*** nadeem has quit IRC18:14
*** nadeem has joined #openstack-swift18:15
*** ig0r__ has joined #openstack-swift18:24
*** ccavanna_ has joined #openstack-swift18:29
*** jasondotstar has joined #openstack-swift18:30
*** thumpba_ has joined #openstack-swift18:30
*** briancurtin_ has joined #openstack-swift18:31
*** bill_az_ has joined #openstack-swift18:33
*** marcusvrn has quit IRC18:33
*** blair has quit IRC18:34
*** briancurtin has quit IRC18:34
*** acoles_away has quit IRC18:34
*** tdasilva has quit IRC18:34
*** ccavanna has quit IRC18:34
*** bill_az has quit IRC18:34
*** thumpba has quit IRC18:34
*** briancurtin_ is now known as briancurtin18:34
*** blair has joined #openstack-swift18:40
*** changbl has quit IRC18:43
ccavanna_Hi. I proposed a stats improvement change a few days ago (to show stats by policy) and addressed all comments. Does anyone have a moment to re-review? It's a small change.18:48
ccavanna_https://review.openstack.org/#/c/202657/18:48
*** ig0r_ has quit IRC18:50
*** changbl has joined #openstack-swift18:56
*** bapalm_ has joined #openstack-swift19:00
*** haomaiwang has quit IRC19:01
*** geaaru has quit IRC19:01
*** haomaiwang has joined #openstack-swift19:02
*** ig0r_ has joined #openstack-swift19:04
*** bapalm_ has quit IRC19:05
*** lpabon has quit IRC19:10
*** marcusvrn has joined #openstack-swift19:14
*** jasondotstar has quit IRC19:29
*** tdasilva has joined #openstack-swift19:29
*** ChanServ sets mode: +v tdasilva19:29
*** acoles_away has joined #openstack-swift19:29
*** acoles_away is now known as acoles19:30
*** ChanServ sets mode: +v acoles19:30
*** openstackgerrit has quit IRC19:46
*** openstackgerrit has joined #openstack-swift19:46
*** jasondotstar has joined #openstack-swift19:50
*** wer_ has joined #openstack-swift19:53
*** wer_ has quit IRC19:54
*** ig0r_ has quit IRC19:56
*** haomaiwang has quit IRC20:01
*** haomaiwang has joined #openstack-swift20:02
*** thurloat is now known as thurloat_isgone20:10
*** annegentle has quit IRC20:13
*** annegentle has joined #openstack-swift20:13
*** openstackgerrit has quit IRC20:16
*** openstackgerrit has joined #openstack-swift20:16
openstackgerritMinwoo Bae proposed openstack/swift: EC: Handoff node to push existing fragment to the correct location.  https://review.openstack.org/19684820:21
*** blmartin has joined #openstack-swift20:22
*** si1v3r has joined #openstack-swift20:26
si1v3rOut of curiousity, is there any preference against using raid cards when they are available?20:27
ctennisyes, use them in jbod mode20:28
si1v3rWhy not raid 10?20:28
ctennisbecause swift works better when it knows about each individual disk so it can place data specifically.20:29
si1v3rDoes it?20:29
ctenniswhen you have a failed drive, swift will work around it.  when you have a failed raid drive, swift doesn't know.20:29
occupantI do that with an LSI raid card and I ran into weirdo performance issues with the CentOS kernel that I could never get anyone to care about. Ended up using the elrepo v3 kernel instead.20:29
occupantA lot of raid cards won't let you have a raid array AND use jbods at the same time.20:30
si1v3rctennis: I understand that, but replacing the drive in the same manner as any of our other machines would make swift's working around completly unnessicary.20:30
si1v3rI'm seeing big performance numbers in my head.20:30
occupantwhen your app can directly address drives, there's no performance benefit in creating a logical group20:31
occupantthe bandwidth of your controller is the only thing that matters20:31
si1v3rWell, it's a sas controller, and I'm pretty sure you can write to a raid faster than 2 drives of the same speed.20:32
si1v3rUsing the controllers data queing20:32
si1v3r(having a bad typing day here...)20:32
ctennisi don't recommend that either20:33
si1v3rWhy?20:33
ctennisbecause you have a time window where you could lose data20:33
*** robefran has quit IRC20:33
si1v3rWe have that already20:33
ctennisI guess in the end it depends on your use case.  Swift is designed for durability, not necessarily write performance.20:33
si1v3rWould the data not be spread to 3 raids?20:34
si1v3rWe have 30 machines with 12 drives each. The sas controllers are useless in this configuration.20:34
ctennisyes, so present all 360 disks individually to swift20:35
si1v3rWhy20:35
si1v3ralso, did. It sucks.20:35
ctennissorry to hear20:35
si1v3rWouldn't swift be happy as a clam never knowing of drive failure?20:36
ctenniscan you be more specific?20:36
ctennissure, but then why use Swift at that point?20:36
si1v3rI want to know why many single drive jbods would be better than raid.20:36
occupantused to work at a place that wrote data raw to disk. no filesystem, no parition even.20:36
ctennisnotmyname: can you articulate this better than I? ^20:37
ctenniscan you quantify how it sucked?20:37
si1v3rSwift uses the kernel to tell if a drive is bad. It's causing false positives.20:37
si1v3rI'd rather trust LSI.20:38
ctennisOkay, well, there's no technical reason you *can't* do it20:38
si1v3rSo instead of each box being a zone with 12 drives, we could just make each a zone with one drive.20:38
si1v3rIs there a technical reason I should not?20:39
si1v3rall I'm hearing is that it's abnormal and not as designed.20:39
si1v3rhas anyone even tried?20:39
occupantyou're getting false kernel errors about your drives?20:39
ctennissure, I've seen people do lots of things.  I've seen clusters of all raid0 single drives.20:39
ctennisyou may never have an issue.20:39
ctennisI'd say if you have good success with it, it would be an interesting blog post or story to read.20:40
*** annegentle has quit IRC20:40
si1v3rSo there is no reason to believe that it works better with individual drives then?20:40
ctennisI guess it depends on what you mean by "works better"20:41
*** ccavanna_ has quit IRC20:41
si1v3rIf this is unexplored territory that is fine, but using hardware wrong based on bad assumptions is a damn shame.20:41
ctennisIt sounds like you have a specific error you're trying to avoid20:41
occupantI dunno, maybe you know better than the people who wrote this shit20:41
ctenniswhich is okay20:41
si1v3rctennis: I'm trying to make it go faster than 1gbps.20:42
si1v3rit takes forever to upload 400GB right now.20:42
ctennisthat may be doable with just some performance tuning20:42
*** openstackgerrit has quit IRC20:46
*** openstackgerrit has joined #openstack-swift20:46
occupantif you're trying to upload large files, you'd probably be happier doing more aggressive chunking20:47
occupantthat way your writes are more distributed with less chance of saturating the IO of any one drive20:47
ctennisyou should easily be able to 1Gbps, if not 10.20:47
si1v3ryeah these boxes are on 40gb links I think, they should be able to go a lot faster.20:48
*** jasondotstar has quit IRC20:56
*** jamielennox|away is now known as jamielennox20:57
*** thumpba_ has quit IRC21:00
*** haomaiwang has quit IRC21:01
*** openstack has joined #openstack-swift21:10
*** annegentle has joined #openstack-swift21:16
*** hrou has quit IRC21:18
*** robefran has joined #openstack-swift21:18
*** zul has joined #openstack-swift21:18
*** wer_ has joined #openstack-swift21:19
*** wer_ has quit IRC21:19
claygswifterdarrell: https://bugs.launchpad.net/swift/+bug/147997221:23
openstackLaunchpad bug 1479972 in OpenStack Object Storage (swift) "HUP signal doesn't shutdown wsgi servers" [Undecided,New]21:23
*** marzif_ has joined #openstack-swift21:26
*** Fin1te has joined #openstack-swift21:28
*** marzif_ has quit IRC21:32
*** blmartin has quit IRC21:33
*** marzif_ has joined #openstack-swift21:34
*** Fin1te has quit IRC21:35
*** marzif_ has quit IRC21:35
*** marzif_ has joined #openstack-swift21:36
openstackgerritDarrell Bishop proposed openstack/swift: Fix regression in WSGI server SIGHUP behavior  https://review.openstack.org/20763721:37
openstackgerritDarrell Bishop proposed openstack/swift: Fix regression in WSGI server SIGHUP behavior  https://review.openstack.org/20763721:40
*** jrichli has quit IRC21:40
swifterdarrellacoles: clayg: cschwede: glange: mattoliverau: notmyname: peluse: redbo: tdasilva: torgomatic: zaitcev:  Gah! I broke SIGHUP for the WSGI servers  https://review.openstack.org/#/c/207637/221:42
redbowell stop it.  oh you can't.21:43
swifterdarrellredbo: haha, train. station. left.21:43
swifterdarrellredbo: fix is ready for review, though21:43
clayganyone got any problems getting this merged quickly?  It's a) really gross and b) fairly obvious fix - we can do a post morten after we start working on new packages.21:45
redbodone21:45
*** jasondotstar has joined #openstack-swift21:48
*** annegentle has quit IRC21:49
*** ig0r__ has quit IRC21:51
peluseneat21:53
*** marzif_ has quit IRC21:54
claygpeluse: no - NOT neat21:55
*** jlhinson has quit IRC21:57
MooingLemurthis reminds me of having to bounce between event loop handlers every couple hundred ms in a very old project.21:58
clayg:'(21:58
claygI honestly don't think I really understood how the *non*-green os.wait was acctually *blocking* the parent pids event loop and the pythons moving to sig handlers HUP was falling out out of the wait21:59
claygit's all very crazy21:59
claygthe way it's written now I acctually understand - and if I'd have thought about the old implementation I would have realized I don't know how it would work if the wait blocks with no timeout - but that "exactly" what the old code did!  :'(22:00
*** annegentle has joined #openstack-swift22:00
*** haomaiwang has quit IRC22:01
*** NM has quit IRC22:01
claygbut I guess that's the EINTR :'(22:01
clayggod damn it22:01
*** haomaiwang has joined #openstack-swift22:02
redbodid someone say python's getting rid of eintr?  that'll be nice.22:02
redbowell hiding it22:02
clayglol22:02
*** jkugel has quit IRC22:06
redboI've had problems on modwsgi before where signals interrupt i/o calls, causing an unrecoverable exception.  But apache uses signals to talk between workers, so you can't just disable them.22:08
claygcan't win for loosing :'(22:08
*** jamielennox is now known as jamielennox|away22:15
*** jordanP has quit IRC22:22
claygcan't win for loosing :'(22:37
clayg^ or for alt-tab-up-entering22:37
*** annegentle has quit IRC22:44
openstackgerritClay Gerrard proposed openstack/swift: Add a probetest for HUP/reload  https://review.openstack.org/20765622:52
clayglove me some probetests22:59
*** haomaiwang has quit IRC23:01
*** km has joined #openstack-swift23:01
*** haomaiwa_ has joined #openstack-swift23:02
*** ho has joined #openstack-swift23:14
*** kei_yama has joined #openstack-swift23:17
mattoliverauMorning23:22
openstackgerritpaul luse proposed openstack/swift: Add ability for GET path to see/select alternate frag archs  https://review.openstack.org/20716523:32
peluseacoles, ^ EC func tests fixed23:34
*** marcusvrn has quit IRC23:43
*** lcurtis has quit IRC23:46
claygpeluse: so is patch 201283 to be abandonded?23:48
patchbotclayg: https://review.openstack.org/#/c/201283/23:48
peluseyeah, acoles made one that had diskfile refactor as a dependecny so we moved to that one23:49
peluseso he is the owner, same patch name, etc.  All working/rebased/bla bla bla.  Ready for eyes23:49
peluseclayg, BTW our perf cluster has been running a while now w/large (30 sec) timeouts and no strange partial PUT failures that lead to Ec recon failures.  Testing continues of course23:50
peluseclayg, might be worth charles trying to crank those up also as a test23:51
pelusenotmyname, or anyone really... does the Openstack community generally consider Swift as HA?  I don't see it mentioned in http://docs.openstack.org/high-availability-guide/high-availability-guide.pdf23:58
pelusebut seems to me that we're already HA...23:58
peluse... no single POF, rolling upgrades w/o downtime, etc., etc.23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!