Tuesday, 2014-02-25

*** richm has quit IRC00:14
*** lbragstad has joined #openstack-keystone00:21
*** nachi has quit IRC00:25
*** arunkant has quit IRC00:26
*** arunkant has joined #openstack-keystone00:27
*** lnxnut has joined #openstack-keystone00:30
*** lnxnut has quit IRC00:34
*** achampion has quit IRC00:35
*** lnxnut has joined #openstack-keystone00:35
*** lnxnut has quit IRC00:40
*** david_lyle has quit IRC00:44
*** henrynash has joined #openstack-keystone00:49
*** lnxnut has joined #openstack-keystone00:50
*** hogepodge has quit IRC00:51
*** henrynash has quit IRC00:53
*** lnxnut has quit IRC00:54
*** lnxnut has joined #openstack-keystone00:54
*** lnxnut has quit IRC00:59
*** lnxnut has joined #openstack-keystone01:01
*** henrynash has joined #openstack-keystone01:04
*** browne has quit IRC01:08
*** lnxnut has quit IRC01:38
*** lnxnut has joined #openstack-keystone01:39
*** gokrokve has quit IRC01:40
*** gokrokve has joined #openstack-keystone01:41
*** gokrokve_ has joined #openstack-keystone01:42
*** gokrokve_ has quit IRC01:43
*** lnxnut has quit IRC01:44
*** gokrokve has quit IRC01:46
*** devlaps has quit IRC01:52
*** achampion has joined #openstack-keystone01:57
*** lnxnut has joined #openstack-keystone01:57
*** marcoemorais1 has joined #openstack-keystone02:15
*** marcoemorais has quit IRC02:15
*** gokrokve has joined #openstack-keystone02:19
*** lnxnut has quit IRC02:39
*** marcoemorais1 has quit IRC02:39
*** lnxnut has joined #openstack-keystone02:40
*** lnxnut has quit IRC02:44
*** henrynash has quit IRC03:13
ayoungbknudson, dolphm_503 jamielennox really needs to move ahead.  https://review.openstack.org/#/c/71455/  Needs to  be in icehouse03:18
jamielennoxayoung: looking03:19
*** lnxnut has joined #openstack-keystone03:19
ayoungjamielennox, thanks.  It makes client the owner of the common code....03:19
jamielennoxayoung: what are we still doing the environment stuff for if we're using client cms?03:20
ayoungjamielennox, ask dstanek about that!03:20
ayoungjamielennox, when the monkey patching takes effect, it changes the class returned by the exception03:20
ayoungif we don't do the environment stuff, we look for the wrong exception03:21
jamielennoxwrong exception from where?03:22
dstanekjamielennox: you mean where we catch the exception?03:22
ayoungjamielennox, popen03:22
ayoungjamielennox, dstanek figured that out, which is why is a coauthor03:22
ayoungcuz he's smaht!03:23
jamielennoxayoung: popen shouldn't be affected, because if you are doing the eventlet monkey patch then it will always raise the subprocess errror03:23
* dstanek blushes03:23
jamielennoxdstanek: i don't think you're wrong i'm just wondering03:23
jamielennoxenvironment.subprocess error shouldn't really be needed any more03:23
ayoungjamielennox, OK...so we were not properly monkeypatching in the client before03:24
dstanekjamielennox: the exception thrown from the eventlet subprocess is not the same as the one from the stdlib03:24
jamielennoxayoung: well we shouldn't be monkeypatching the client03:24
ayoungjamielennox, no, we need to detect monkeypatching in the client03:24
jamielennoxdstanek: when you do a from environment.green import subprocess03:24
ayoungand call the write code accordingly03:24
dstanekjamielennox: check out the last 3 classes here: https://review.openstack.org/#/c/71455/5/keystone/tests/test_token_provider.py03:24
dstanekthe problem (not really a problem) is that we are not monkey patching subprocess03:25
jamielennoxoh, i see what you mean03:25
dstanekso everyone has to import the same one to interoperate03:25
jamielennoxi hadn't seen cms.set_subprocess03:26
dstanekyeah, so environment has always been our shim to do that03:26
jamielennoxwe could probably have just relied on eventlet patching to do that for us03:26
ayoungjamielennox, nope03:26
ayoungpopen is .... called by the monkeypatched version of popen03:26
ayoungso you cna;t just monkeypatch popen,  you have to call the right version...its wonderful code03:26
dstanekgotta love magic!03:27
jamielennoxergh, i'm gonna assume you both know what's happening there03:27
* ayoung having trouble finding the change03:28
ayoungjamielennox, hold on.  it was a one character fix03:28
ayoung.get('os'):  OK a little more than one character03:29
ayoungbut with that, we properly identified that the client should use the eventlet version....see, most of the openstack clients don't call monkeypatch for os....03:29
jamielennoxayoung most of the *clients* shouldn't monkeypatch03:30
jamielennoxit should only ever happen from a server03:30
ayoungjamielennox, but for Eventlet, they need to use the monkeypatched version of popen, or they block03:30
ayoungthat is what I mean by clients03:30
ayoungauth token clients03:30
*** marcoemorais has joined #openstack-keystone03:32
jamielennoxayoung: surely (and i guess i mean hopefully) when you do     eventlet.patcher.monkey_patch(os=False, select=True, socket=True,03:32
jamielennox                                  thread=monkeypatch_thread, time=True,03:32
jamielennox                                  psycopg=False, MySQLdb=False)03:32
jamielennox it patches popen03:32
ayoungLurvely, eh Wot?03:33
*** marcoemorais has quit IRC03:34
*** marcoemorais has joined #openstack-keystone03:35
*** marcoemorais has quit IRC03:39
*** morganfainberg is now known as morganfainberg_Z03:39
*** lnxnut has quit IRC03:55
*** lnxnut has joined #openstack-keystone03:56
*** lnxnut has quit IRC04:00
*** stevemar has quit IRC04:09
*** lnxnut has joined #openstack-keystone04:11
*** theocean154 has joined #openstack-keystone04:12
*** harlowja is now known as harlowja_away04:12
*** lnxnut has quit IRC04:16
theocean154noob question: I have a hello world api sitting in keystone/keystone/contrib, how to i hook it in to the routers list. do i need to modify keystone/service.py04:19
ayoungtheocean154, not necessarily04:22
ayoungfor an extension, add it to the pipeline, so it gets auto-registered. See how  the others are done...I think it was oauth that we did that way first04:22
*** david_lyle_ has joined #openstack-keystone04:23
ayoungtheocean154, extension.register_admin_extension  or the public variant should wire it up04:24
*** lnxnut has joined #openstack-keystone04:26
ayoungjamielennox, OK,  so I started working on a client extension for revocation events, and decided that before auth_token supports it, I want to do a CLI04:27
ayoungsomething like keystone validate_token $TOKEN04:28
ayoungit will do the cms call, then the revocation check04:28
ayoungonce I get that working, we can integrate with AT-middle04:28
jamielennoxayoung: sounds fair04:29
ayoungjamielennox, I was trying to figure out how to add an option to the CLI04:29
jamielennoxno it means you are limited to the v2 api which doesn't have the revocation events04:29
theocean154@ayoung is the path given by the alias key in core.py?04:30
ayoungtheocean154, sure04:30
theocean154how do you cite a name in limechat anyone?04:30
ayoungtheocean154, I think....I wrote this so long ago, I would have to read to code to remember.04:30
theocean154i see your todos haha. i copied the basic code from the contrib/ec2 files04:31
jamielennoxthe path to an extension is a funny concept that im not sure is ever properly fleshed out04:31
*** harlowja_away is now known as harlowja04:32
ayoungjamielennox, you know, when I first read that, I thought you were talking about client extensions,  but realize you meant server.  I think it pretty much applies to anything I've touched in Keystone04:33
jamielennoxayoung: client doesn't have proper extensions, i was going to do that until the /extensions path was rejected04:33
jamielennox(someone came looking for that earlier this moning)04:34
ayoungyeah. yeah.04:34
jamielennoxi don't know what happened to the IBM guys who wanted to do client extensions04:34
jamielennoxanyway for CLI don't you need v3 for token revocation extensions?04:35
ayoungyep  V304:35
jamielennoxso no can do on the CLI04:36
ayoungI figured I start just a "list"04:36
ayoungooh.  can't we do --api-clinet-version=304:36
jamielennoxthat never worked04:36
ayoungI see some real ugliness in there04:37
ayounghttps://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/shell.py#L255 <--------- jamielennox what is that04:38
ayoungI know I should ask Gabe Hurley04:39
ayoungbut I'm guessing he just inherited it from where ever....04:40
jamielennoxayoung: somewhere between that and https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/shell.py#L432 is hwere it says ignore what the user asked for and use the v2 api04:40
*** lnxnut has quit IRC04:40
ayoungpretty much the line I just showed you guaran-damn-tees it04:40
ayoungbecause you are ALWAYS getting a KeyError there04:41
jamielennoxyea, your one doesn't give a warning :)04:41
jamielennoxayoung: i'm currently trying to figure out if it's worth trying to reuse the existing clients or do a rewrite of them for the SDK04:42
jamielennoxrewrite is looking better and better04:42
*** achampio1 has joined #openstack-keystone04:42
jamielennoxi fought kindof hard to reuse them, but ughh04:42
ayoungnah, we can fix this04:42
ayounga rewrite is rarely worth it04:43
jamielennoxayoung: the question i suppose is do we want to04:43
ayoungNo, I want to get the client supporting v3, get ATM using the client code, and move on with my life04:43
*** achampion has quit IRC04:45
ayoungAuth token middleware.  ATM04:46
jamielennoxmakes sense04:47
jamielennoxi've got some patches later in my review list that have ATM using the client04:48
ayoungyeah...need that04:48
jamielennoxayoung: does this vaguely make sense as an SDK pattern: https://github.com/jamielennox/python-mysdk/blob/master/mysdk/api/identity.py04:49
ayoungI started off with ATM for revocations, and realized I needed to fetch the list...which lead me looking at sessions...and thats how I ended up looking at the CLI04:49
jamielennoxi'm not sure exactly why i'm asking it just seems a bit weird04:49
*** achampio1 has quit IRC04:50
ayoungjamielennox, its going to end up like the current client. .... the current client is a mess because it is a messy business.  I'll take a look, but I'm none too hopeful that anything sane can stand04:50
theocean154ayoung: thanks, it worked. The path was /v3/tfa.  is the v3 appended to the path because my controller is a V3Controller class?04:50
ayoungtheocean154, you going to submit what you are working on upstream?04:50
theocean154im part of open academy04:51
theocean154we're doing the two factor authentication stuff04:51
ayoungyou might be looking at the wrong code then04:51
theocean154oh goody haha04:51
ayoungthat probably should be an Auth plugin04:51
theocean154we've been looking at soooo much code04:51
ayoungtry to work within the auth controller under keystone/auth04:51
*** achampion has joined #openstack-keystone04:52
ayoungand just add a plugin to that....same thing the Federation work had to do04:52
ayoungIm just guessing, and can't say for sure until I see your code, though04:52
ayoungwe need to rework the auth architecture into more of a pipeline, and ...  its really plugable in two places right now.  Auth plugins and the token provider.. the seond is way too course-grained a thing to replcae04:54
*** KanagarajM_ has joined #openstack-keystone04:54
theocean154ok, i assume i can just move my code from the controllers, routers, and core in contrib to those in plugins?04:54
ayoungtheocean154, post it as a WIP review and I'll look at it shortly, and I'll try to make it clear.  I need to know the scope before I can be certain04:56
ayoungadd me to the review once you've posted it04:56
ayounggoing to bed now04:59
*** ayoung has quit IRC04:59
lbragstaddolphm_503: identity v3 api endpoint doc cleanup: https://review.openstack.org/#/c/76086/105:03
jamielennoxlbragstad: commented05:05
jamielennoxlbragstad: you can leave it to others though05:06
lbragstadendpoint ids shouldn't be public?05:06
lbragstadjamielennox: thanks for the review05:06
jamielennoxlbragstad: we were having this conversation about something the other day05:07
jamielennoxi don't see why they should be and i don't *think* they are in a service catalog05:08
lbragstadI must have missed the conversation?05:08
lbragstadeither way, dolphm_503 and I were talking about it earlier because they had the 'name' attribute.05:09
lbragstadwhich isn't accessible through the clients05:09
lbragstadso I was working on cleaning that out of the docs, and dolphm mentioned that the --service--id and '...' could be replaced with realistic values.05:09
lbragstadjamielennox:  like how this uses an actual ID in the example: https://github.com/openstack/identity-api/blob/master/openstack-identity-api/v3/src/markdown/identity-api-v3.md#endpoints-v3endpoints05:10
lbragstadbut it doesn't remain consistent here: https://github.com/openstack/identity-api/blob/master/openstack-identity-api/v3/src/markdown/identity-api-v3.md#endpoints05:10
jamielennoxlbragstad: yea, i would still show it, i think it's good other wise05:11
jamielennoxlooking at the way i read that doc an endpoint should have an id05:11
lbragstadjamielennox: yeah, do you know if there is a record of the #openstack-keystone channel?05:12
jamielennoxlbragstad: i think there should be05:13
jamielennoxi don't know how far back you would need to go05:13
jamielennoxprobably it was still in -dev05:13
lbragstadahh ok05:13
lbragstadwas gonna try and dig it up05:14
lbragstada lot of the resources in the doc have that same thing going on05:14
lbragstadmost of the examples use '...' or '--some--attribute--'05:14
jamielennoxlbragstad: might be that i'm completely wrong05:15
jamielennoxthis is a token i just got from v3: http://paste.openstack.org/show/69267/05:15
jamielennoxand there is definetly an endpoint id in them05:15
lbragstadit is in catalog05:16
lbragstadand it has legacy_endpoint_id and region05:16
jamielennoxi definetly meant the catalog05:16
lbragstadand the spec doesn't look like it has 'type'?05:17
jamielennoxhmm, type we need05:18
jamielennoxi went to +2, now back to -105:19
lbragstadI'll add that to the review05:19
*** theocean154 has quit IRC05:19
*** theocean154 has joined #openstack-keystone05:21
lbragstadtype should be optional05:23
*** marcoemorais has joined #openstack-keystone05:29
*** marcoemorais1 has joined #openstack-keystone05:32
*** marcoemorais has quit IRC05:33
jamielennoxlbragstad: i don't think so - type is like 'public' or 'admin' without it were kinda stuffed05:35
jamielennoxergh, hang on that's called interface in the catalog05:35
*** henrynash has joined #openstack-keystone05:36
jamielennox'type' is like 'compute'05:36
lbragstadyeah. interface is apparently either 'admin', 'public', or 'internal'05:36
lbragstadI feel like 'type' should be in service05:36
jamielennoxbut that's not a per endpoint thing apparently05:36
jamielennox... i have no good advice today ...05:36
*** lnxnut has joined #openstack-keystone05:36
lbragstadyou *should* be able to associate a type with a service... which is correctly reflected in the doc05:37
jamielennoxand an endpoint needs a service_id05:37
lbragstadbut I was a little mixed up when I saw your paste05:37
jamielennoxand that should be enough05:37
lbragstadyour paste listed endpoints05:37
lbragstadbut they had 'type'?05:37
* lbragstad might be missing something 05:37
jamielennoxno they don't it's just not that well formatted05:39
jamielennoxcatalog endpoints are a list of dictionaries05:39
*** theocean154 has left #openstack-keystone05:40
jamielennoxwith a 'endpoints', 'type', and 'id' in them05:40
lbragstadI see it now05:40
*** lnxnut has quit IRC05:41
lbragstadservices don't seem to reflect the 'enabled' field. I thought bknudson was working on something like that for endpoints too?05:48
lbragstadjamielennox: ^05:48
jamielennoxlbragstad: i'm not sure05:48
lbragstadI can investigate that tomorrow05:48
jamielennoxswift's client is completely flat - there is support the swift bin and nothing else05:49
*** lbragstad is now known as lbragstad_05:53
*** harlowja is now known as harlowja_away06:01
*** chandan_kumar has joined #openstack-keystone06:07
*** saju_m has joined #openstack-keystone06:21
*** topol has joined #openstack-keystone06:21
*** topol has quit IRC06:23
*** topol has joined #openstack-keystone06:23
*** lnxnut has joined #openstack-keystone06:26
*** saju_m has quit IRC06:29
*** lnxnut has quit IRC06:31
*** gyee has quit IRC06:42
*** saju_m has joined #openstack-keystone06:42
*** bvandenh has joined #openstack-keystone06:59
*** topol has quit IRC07:01
*** gokrokve has quit IRC07:06
*** gokrokve has joined #openstack-keystone07:09
*** gokrokve_ has joined #openstack-keystone07:11
*** saju_m has quit IRC07:13
*** saju_m has joined #openstack-keystone07:13
*** gokrokve has quit IRC07:14
*** lnxnut has joined #openstack-keystone07:26
*** henrynash has quit IRC07:31
*** lnxnut has quit IRC07:31
*** leseb has joined #openstack-keystone08:13
*** dstanek has quit IRC08:25
*** lnxnut has joined #openstack-keystone08:27
*** gokrokve_ has quit IRC08:28
*** lnxnut has quit IRC08:31
*** gokrokve has joined #openstack-keystone09:12
*** gokrokve has quit IRC09:17
*** lnxnut has joined #openstack-keystone09:26
*** lnxnut has quit IRC09:31
*** marcoemorais1 has quit IRC09:40
*** gokrokve has joined #openstack-keystone10:12
*** gokrokve has quit IRC10:17
*** david_lyle_ has quit IRC10:21
*** lnxnut has joined #openstack-keystone10:26
*** lnxnut has quit IRC10:31
*** KanagarajM__ has joined #openstack-keystone10:50
*** KanagarajM_ has quit IRC10:53
*** leseb has quit IRC10:58
*** leseb has joined #openstack-keystone11:08
*** gokrokve has joined #openstack-keystone11:12
*** gokrokve has quit IRC11:17
*** lnxnut has joined #openstack-keystone11:26
*** lnxnut has quit IRC11:31
*** gokrokve has joined #openstack-keystone12:12
*** gokrokve has quit IRC12:17
*** lnxnut has joined #openstack-keystone12:26
*** lnxnut has quit IRC12:31
*** leseb has quit IRC12:39
*** leseb has joined #openstack-keystone12:54
*** gokrokve has joined #openstack-keystone13:14
*** gokrokve has quit IRC13:18
*** dstanek has joined #openstack-keystone13:19
*** ChanServ sets mode: +v dstanek13:19
*** topol has joined #openstack-keystone13:19
bknudsonlbragstad_: there's no enabled field in the schema. the enabled data goes in the extra json13:23
*** lnxnut has joined #openstack-keystone13:26
*** lnxnut has quit IRC13:31
*** dolphm_503 is now known as dolphm13:34
*** browne has joined #openstack-keystone13:40
dstanekbknudson: it's like our private nosql database inside our sql database!13:40
bknudsondstanek: it's webscale13:41
dstanekdolphm: nice catch on the import13:46
dolphmdstanek: the same thing resulted in a long convo at some point :P13:48
dstanekdolphm: i would not have even thought of that, but now that i see it...13:49
*** saju_m has quit IRC13:50
dstanekdolphm: i just came across https://review.openstack.org/#/c/75816 and i'm a little sad that we want to use a 409 on a PUT13:53
dolphmdstanek: me too- i think we can safely change that to a 200/201 within the wording of the api compatibility guidelines13:54
dstanekdolphm: the api docs explicitly say 40913:55
dolphmdstanek: yeah, IIRC it's one of those implementation bugs that made it's way back into the spec13:56
dstanekdolphm: ah, i'd really like to see the POST just take an id and 409, leaving the PUT to just update13:57
dolphmdstanek: i'd rather not use PUTs, unless they're following the actual semantics of PUT (replace)13:57
dolphmdstanek: or create-or-replace13:57
dolphmPUT /domains/my-custom-domain-id13:58
dstanekdolphm: no i agree, that's what i meant - or we can just leave it out since patch is defined13:58
*** nkinder_ has quit IRC13:59
dolphmdstanek: ++14:00
dstanekdolphm: do we already have code doing real PUTs?14:01
*** marcoemorais has joined #openstack-keystone14:02
*** lnxnut has joined #openstack-keystone14:02
dolphmdstanek: only for role assignments, v3 region creation, and some federation calls (IdP & protocol crud)14:02
dolphmdstanek: regions is a bit more flexible in that it allows...14:04
dolphmauto assigned ID's: POST /v3/regions14:04
dolphmand user-specified ID's:14:04
dolphmPUT /v3/regions/custom-region-id14:04
dolphmPOST /v3/regions {"region": {"id": "custom-region-id"}14:04
*** lbragstad_ has quit IRC14:05
*** marcoemorais1 has joined #openstack-keystone14:05
dolphmthere's a third combination of method / resource / body it allows, but that's sort of a subtle inconsequential bug in the implementation14:05
*** marcoemorais has quit IRC14:06
dstaneki'm going to submit a review for this to the identity-api14:07
dstanekdoes this break backward compatibility rules?14:07
*** achampion has quit IRC14:08
dolphmdstanek: nope! i've always wanted user specified ID's across the entire API :) i made sure there was room in the spec for them to be introduced later14:08
dolphmdstanek: the only caveat is that i would raise a 400 if user_specified_id != urllib.quote(user_specified_id), i think?14:09
dolphmespecially if you support POST w/ ID's in the body -- it'd be easy to slip in something that's not URL friendly, and not realize it14:10
dstanekwhy is that? you only want web safe IDs?14:10
dstanekactually the id will have to be something that is path safe anyway14:11
dolphmdstanek: preferably14:11
*** gokrokve has joined #openstack-keystone14:12
*** leseb has quit IRC14:15
*** leseb has joined #openstack-keystone14:16
dstanekdolphm: i haven't looked at the impl yet, but the federation docs imply that it's using PUT as create only14:17
dolphmdstanek: correct - that should be true in the impl as well14:17
*** gokrokve has quit IRC14:17
dstanekwhat happens if the resource already exists?14:18
dolphmdstanek: L64 https://review.openstack.org/#/c/71353/38/keystone/contrib/federation/routers.py14:18
*** KanagarajM__ has quit IRC14:20
*** jraim has quit IRC14:22
*** jraim_ has joined #openstack-keystone14:22
*** topol_ has joined #openstack-keystone14:23
*** stevemar has joined #openstack-keystone14:24
*** ChanServ sets mode: +v stevemar14:24
*** topol has quit IRC14:24
*** topol_ is now known as topol14:25
dstanekdolphm: this may be an interesting discussion for today's meeting14:26
dstanekdolphm: even though there is room for user specified IDs if we remove PUT or change the semantics to be RESTful i think that's an API change14:27
dolphmdstanek: remove PUT from what?14:27
dolphmdstanek: nothing uses PUT except role assignments and regions in the core API14:28
dolphmdstanek: https://github.com/openstack/identity-api/blob/master/openstack-identity-api/v3/src/markdown/identity-api-v3.md#create-region-post-regions14:29
dstanekdolphm: right, in regions14:29
dolphmdstanek: what would you want to change about regions?14:29
dstanekdolphm: removing the 409 conflict14:30
dstanekactually is apparently doesn't 409 right now is 500s - that's what the review is trying to fix14:31
*** lnxnut has quit IRC14:31
dolphmdstanek: i can't find the wiki article at the moment, but we're free to tweak error codes and turn error codes into 200's14:32
dolphmdstanek: but then we can't go back (from 2xx to 4xx, for example)14:32
*** marekd|away is now known as marekd14:33
dolphmso, 500 -> 409 -> 200 would be a valid evolution (if desired) (or we can just skip to the 200)14:33
dstanekdolphm:  the thing that worries me isn't just the status code, it's that we'd be updating (replacing) when previously we didn't touch the data14:33
dstaneki'm definitely OK with it if you are :-)14:34
dolphmdstanek: i guess i'm not following... example?14:34
*** marcoemorais1 has quit IRC14:34
dstanekdolphm: i don't have a usecase where this makes sense but right now if you PUT one you get a resource; if you then PUT again to the same URL you are given an error code14:35
dstanekthat implies "change your ID and try again" - change it to a 200 may break a user if they actually depend on a failure14:36
dstanektheir data would be replace - so two different user picking the same region to create "my-region"14:36
dolphmdstanek: ah, so you're worried about PUT to create-or-replace vs PUT to create-only14:37
*** lbragstad has joined #openstack-keystone14:38
dstanekyes, if should be less of an issue for this because we give a server error code, but i wanted to bring it up14:38
dolphmdstanek: i wonder if there are buggy clients out there that include null or blank ID's in POST requests - we might break them14:38
dstanekwith POST we create the ID14:40
dstanekalso federation doesn't behave as i expected - http://paste.openstack.org/show/69432/ - the second put() fails with a 40914:41
dolphmdstanek: yeah - i'd *prefer* a 200, but i could have told you it was a 409 :P14:43
dolphmdstanek: we *could* make that configurable :-/14:43
dolphmdstanek: allow_replace_on_create or something?14:43
dstaneknah, i think that would make is hard to interoperate on the client side - taking a script designed for CloudA and then pointing it to CloudB14:44
*** ayoung has joined #openstack-keystone14:45
*** lnxnut has joined #openstack-keystone14:48
*** lnxnut has quit IRC14:48
dstanekdolphm: sounds like the right thing to do might be just to make it 40914:49
dolphmdstanek: i'm fine with that14:50
*** nkinder_ has joined #openstack-keystone14:52
*** lnxnut has joined #openstack-keystone14:53
ayoungarunkant, this is cool  https://review.openstack.org/#/c/72026/2114:56
*** gokrokve has joined #openstack-keystone15:02
*** bknudson1 has joined #openstack-keystone15:02
*** bknudson has quit IRC15:05
*** bknudson1 has quit IRC15:07
*** bknudson has joined #openstack-keystone15:10
*** achampion has joined #openstack-keystone15:15
ayoungtox is hanging again.  Specifically, tox -epy27    or tox -edocs  both with and with out -r and with and without an existing .tox subdir...-v produces no output.  What the Tox?15:20
*** david_lyle_ has joined #openstack-keystone15:26
stevemarayoung, can you ping pypi? maybe trouble downloading libs?15:27
*** david_lyle_ is now known as david_lyle15:27
ayoungstevemar, I'm running a "watch" on find . -print of the tox dir.  They are growing, but slowly15:27
ayoungstevemar, for example15:28
ayoungstevemar,  find py27/ -print | wc -l   gives 1139815:28
ayoungits growing by ... I'm guessing one python package  ever 4 seconds or so15:29
ayoungUp to 1219815:29
ayoungI'm running the commands for -epy27 and -edocs in parallel15:29
ayoungstevemar, how big is you .tox/py27?  I'd like to know how long I'll be waiting15:30
stevemarayoung, maybe it's not hanging, just slow?15:30
*** marcoemorais has joined #openstack-keystone15:30
stevemarchecking debug directory, its the same as py2715:30
*** david_lyle_ has joined #openstack-keystone15:31
stevemarayoung, /opt/stack/keystone/.tox$ find debug/ -print | wc -l15:31
stevemarpy27 is 697515:32
ayoungwow...much smaller15:32
stevemarthats weird15:32
ayoungand now mine is down to 776115:33
ayoungmust be some interim processing that gets cleaned up15:33
ayoungthis has been my general experience iwth tox, and why I kindof hate it15:33
*** marcoemorais1 has joined #openstack-keystone15:34
stevemarayoung, i guess it's running the tests now15:34
ayoungstevemar, it musrt be.  it looks like it is spewing the certificate dumps15:35
*** david_lyle has quit IRC15:35
ayoungGenerating RSA private key, 2048 bit long modulus and so forth15:35
*** david_lyle_ is now known as david_lyle15:35
stevemarso not hanging, just slow15:35
ayoungpainfully slow.  I've been at this over an hour15:35
dolphmayoung: i thought you were going to clean up that stdout at some point?15:35
*** marcoemorais has quit IRC15:35
ayounggranted I restarted15:35
ayoungdolphm, right now, it is the only indicator I have that things haven't hung15:36
*** david_lyle_ has joined #openstack-keystone15:36
stevemarayoung, where does that stuff come from?15:37
dolphmstevemar: keystone.common.cms or keystone.cli, i think15:37
ayoungthe pki generation?  oneof the setup tests, I think.  I haven't looked, but I don't think it was something I wrote....I just remember noticing it at somepoint.15:38
ayoungits probably a side effect of doing a shell out to the openssl commands15:38
dstanekayoung: those extra files are probably from pip installing15:38
*** marcoemorais1 has quit IRC15:38
ayoungdstanek, yeah, and now I know the high water mark is around 12000 files, and then it curts down to 7K  I can keep track of progress.15:39
dstanekwhen pip installs stuff it extracts the package and deps into venv/build15:39
dolphmstevemar: ayoung: i'm guessing keystone.common.openssl15:39
*** david_lyle has quit IRC15:39
ayoungdolphm, yeah.  I want to change how we do certificates anyway.  I want to user certmonger, and have it use the selfsign for this use case, or talk to a real CA.  But there is something I need to work out in the selfsigned use case first, and haven't gotten to.15:40
ayoungselfsign doesn't generate a CA cert, and I need to test if we can get through the openssl cms call using the same selfsigned cert for both15:41
ayoung(both signing cert and ca cert)15:41
ayoungif so, it means common/openssl can go away15:41
*** david-lyle has joined #openstack-keystone15:41
dolphmjust tried to repro, and it appears that keystone is refusing to use my keystone.conf [signing] config15:42
dstanekthere was a patch to get rid of that output a while ago, but i don't remember what happened to it15:43
dolphmdstanek: i thought so too15:43
dstaneki think it died because of some subprocess issues15:44
ayoungdolphm, is that the change to use provider instead of the old config value?15:45
*** david_lyle_ has quit IRC15:45
ayoungactually, that should not factor into the generation15:45
dolphmthat's weird... pki_setup fails with "EnvironmentError: makedirs('/etc/keystone/ssl/certs'): Permission denied" until i enabled debug+verbose, and then it succeeds15:45
dolphmbecause it stops trying to wriet to /etc/ and uses my keystone.conf instead15:46
ayoungdolphm, perms on the directory?  Did you run it as root before?15:46
ayoungdolphm, sudo and then set the group as a CL option15:46
dolphmayoung: i din't run it as root, and i don't want it to write to /etc/15:46
dolphmayoung: it's also not configured to write to /etc/15:46
ayoungdolphm, its going to pick up conf from the search paths.  I assume you don't have an /etc/keystone/keystone.conf15:48
dolphmayoung: no i don't, and it otherwise picks up my conf just fine15:49
ayoungetc is the default, I think15:49
dolphmayoung: i'm using etc in cwd15:50
ayoung/etc/keystone/ssl/private/cakey.pem  and so on15:50
dolphmayoung: but that's *not* where i have keystone.conf configured to put those files15:51
dolphmayoung: and it's ignoring my conf and trying to write there anyway15:52
ayoungdolphm, I am guessing it is skipping "all" of the config15:52
ayoungtry running with pdb.15:52
dolphmayoung: options like "debug" take effect just fine15:52
dolphmayoung: http://pasteraw.com/1570jfzh8geg7o5t1yu2ywwkmflcpi115:53
dolphmooh, there is a / i missed in there...15:54
dolphmYAY ayoung: fixed, my fault15:55
dolphmi must have made the exact same mistake twice -- i wiped my keystone.conf and started fresh, made the edits again, and ran into the same outcome15:55
ayoungI noticed that some had etc/ and some /etc/15:56
ayoungI was still checking the openssl code to make sure it wasn't something we were doing15:56
ayounganyway... dolphm that is what certmonger should be doing for us15:56
ayoungOK...let me try this15:56
*** devlaps has joined #openstack-keystone15:58
stevemaris there a reason why we call our own utils here: https://github.com/openstack/keystone/blob/master/keystone/policy/backends/rules.py#L56.L58 instead of https://github.com/openstack/keystone/blob/master/keystone/openstack/common/policy.py#L196.L212 ?16:08
ayoungok, it works....I'll write it up16:08
ayoungstevemar, probably just has not been updated...but I want to move that logic to the client anyway16:16
dolphmstevemar: no - i don't think that existed at the time16:17
*** thedodd has joined #openstack-keystone16:20
*** thedodd has quit IRC16:20
stevemardolphm, hopefully unrelated, but was there a change that added a whole lot more to stdout when running tests? i'm getting a lot of db setup related stuff at the beginning http://paste.openstack.org/show/69466/16:22
dstanekstevemar: you get the output even when all of your tests pass?16:24
stevemardstanek, the tests are still running, i'm running all of them16:25
dstanekstevemar: ping me tomorrow when their done running :-)16:25
stevemardstanek, hehe, few more minutes, test_v3_auth takes a while16:26
dolphmstevemar: hmm, might be oslo.db related?16:26
dstanekstevemar: i usually only get output like that for failing tests16:26
dolphmpaste isn't load for me..16:26
stevemardolphm, that's what i was thinking16:26
dstanekdolphm: it's just a ton of SQLAlchemy INFO/DEBUG logging16:26
stevemari have killed paste16:26
*** gyee has joined #openstack-keystone16:37
dstanekdolphm: your crazy fast :-)16:39
dolphmdstanek: ?16:39
dstaneki just saw the email about the 'unimplemented v3 get token' review and when i got to the bug that caused it i saw that it was modified by dolphm 23 seconds ago16:40
dolphmdstanek: ah, i built a dashboard thing to monitor the gate with a raspberry pi16:41
dolphmdstanek: shows failed merges http://i.imgur.com/zGo55Hw.jpg16:42
dstanekdolphm: nice. is that in a shareable state?16:43
dolphmdstanek: not particularly :) but after icehouse releases i'll probably spend another weekend cleaning it up and making it project agnostic16:44
dolphmdstanek: right now it's very much hardcoded to keystone, and i don't have any automated deployment for it16:44
dolphmdstanek: it's basically chrome in kiosk mode + bootstrap + jquery + flask + celery + redis polling zuul, launchpad & gerrit16:46
dstanekdolphm: when you publish let me know so i can saltstack it up and deploy to my pi16:49
*** chandankumar_ has joined #openstack-keystone16:53
*** chandan_kumar has quit IRC16:54
dolphmdstanek: should i keep the ansible playbooks in a separate repo then? :P16:57
*** fabiog has joined #openstack-keystone17:07
*** chandankumar_ has quit IRC17:10
*** marcoemorais has joined #openstack-keystone17:10
*** chandan_kumar has joined #openstack-keystone17:10
ayoungdolphm, morganfainberg_Z the latest Token Revocation Extension review has support for persisted KVS  https://review.openstack.org/#/c/55908/56  as it was fairly trivial to implement17:14
*** amcrn has joined #openstack-keystone17:23
*** browne has quit IRC17:29
*** marekd is now known as marek|away17:32
*** leseb has quit IRC17:39
*** browne has joined #openstack-keystone17:43
*** richm has joined #openstack-keystone17:45
*** amerine has quit IRC17:50
*** amerine has joined #openstack-keystone17:51
*** fmarco76 has joined #openstack-keystone17:53
*** fmarco76 has left #openstack-keystone17:54
*** harlowja_away is now known as harlowja17:55
dolphmbknudson: cc- deprecating XML in keystone, is there a patch for that? or shall i propose one?17:56
bknudsondolphm: I haven't had the time to propose a patch17:56
bknudsonso go ahead if you have time17:56
*** henrynash has joined #openstack-keystone17:58
*** morganfainberg_Z is now known as morganfainberg17:59
morganfainbergayoung, cool17:59
ayoungmorganfainberg, yeah, would appreciate it if you bled on it18:00
morganfainbergayoung, will do today.18:00
morganfainbergayoung, it'll be about the only upstream thing i can do today (still doing internal things) but I'll hit it right now18:00
ayoungfeels silly to switch to #openstack-meeting18:00
* dolphm yay, blood18:00
morganfainbergwell post meeting that is18:00
dolphmayoung: it generates meeting summaries *shrug*18:00
morganfainbergwe could just add the meetbot to this channel :P18:01
ayoungyeah...its all good, and it gives us a reason to bug people to get off our lawn18:01
morganfainbergGET OFF OUR LAWN18:01
ayoungits time, right?18:01
*** topol has quit IRC18:02
*** gokrokve has quit IRC18:03
*** david_lyle_ has joined #openstack-keystone18:03
*** david_lyle_ is now known as david_lyle18:04
*** david-lyle has quit IRC18:07
*** nkinder_ has quit IRC18:07
*** arunkant has quit IRC18:07
*** arunkant has joined #openstack-keystone18:08
*** bvandenh has quit IRC18:12
*** nkinder has joined #openstack-keystone18:15
*** henrynash has quit IRC18:20
*** achampion has quit IRC18:22
*** chandan_kumar has quit IRC18:22
*** chandankumar_ has joined #openstack-keystone18:22
*** achampion has joined #openstack-keystone18:23
*** thedodd has joined #openstack-keystone18:30
*** chandankumar_ has quit IRC18:30
*** chandan_kumar has joined #openstack-keystone18:31
*** devlaps1 has joined #openstack-keystone18:32
*** gokrokve has joined #openstack-keystone18:33
*** devlaps has quit IRC18:33
*** browne has quit IRC18:36
*** jamielen- has joined #openstack-keystone18:39
*** leseb has joined #openstack-keystone18:40
*** jamielennox has quit IRC18:42
*** leseb has quit IRC18:45
*** browne has joined #openstack-keystone18:46
ayounglast summits etherpads https://wiki.openstack.org/wiki/Summit/Icehouse/Etherpads#Keystone19:01
ayoungwe don't need refactoring assingments19:01
ayoungFedrated one session this time, I think19:01
dolphmjamielen-: i was hoping the sdk meeting would be moved to today..19:01
ayoungI want to propose on one on "distributing policy files"19:01
gyeeayoung, we need to discuss the signature stuff, ec2, s3, etc19:01
stevemarayoung, yep, 1 session19:01
dolphmayoung: GET /v3/policies19:02
gyeeespecially for ec2, scoping a key to a trust seem like a bad idea19:02
jamielen-dolphm: i think in general it is but they were skipping this week - i don't know why19:02
ayounggyee, how about a crypto session?19:02
dolphmayoung: it just needs to be implemented in oslo policy :)19:02
morganfainbergi would like to make sure we touch on performance et al, but not as an official session, unconference/dev lounge stuff19:02
gyeeayoung, sure, krypto19:02
*** jamielen- is now known as jamielennox19:02
*** ChanServ sets mode: +v jamielennox19:02
dolphmjamielen-: ah, we have a conference going on at rackspace19:02
morganfainbergperformance, concurrency, etc19:02
dolphmjamielennox: i was looking forward to it, though19:03
gyeedolphm, which conference?19:03
ayoungdolphm, oslo needs to tell us where to put it, client can fetch it, but endpoint  neeeds to know its identity to ask for the right policy file.19:03
jamielennoxdolphm: i definetly have some queestions and want to see what's happening with it19:03
dolphmgyee: internal rackspace conference19:03
stevemargyee rax.io i think?19:03
dolphmstevemar: ++19:03
ayoungeither url or endpouint ID, I'm leaning toward Id19:03
jamielennoxbknudson, gyee: can we have another round of auth plugins? i really want to get this figured out before the SDK stuff gathers much more steam19:03
ayoungjamielennox, client....one session or maybe two19:04
dolphmjamielennox: not much - the gate is wedged, that's all i know :P19:04
gyeejamielennox, I was looking at them last night, I am fine with them as long as bknudson go through the spelling and grammars :)19:04
jamielennoxayoung: there will be a lot of cross client stuff this time around19:04
gyeebknudson, sorry man, I was just kidding19:04
ayoung2 sessions....lets find a way to differentiate them19:04
jamielennoxdolphm: yea saw that, but i haven't seen any info coming out from the blueprints or anything either19:04
*** alaski has joined #openstack-keystone19:05
alaskimorganfainberg: hi19:05
morganfainbergalaski, is here to talk user_id length changes w/ us!19:05
morganfainbergannd henrynash isn't here19:05
gyeejamielennox, is there a plan to push the session and httpclient to Oslo?19:05
jamielennoxgyee: not oslo, but hopefully somewhere19:05
morganfainbergalaski, so we're proposing a change to increase the length of the user id, user_id@@domain19:06
ayoungI'm here19:06
jamielennoxgyee: i would still like a base client library for all this stuf19:06
morganfainbergalaski, this means we will exceed the current UUID length19:06
gyeejamielennox, why not, as we want all the clients to share authentication and session management19:06
ayounggyee, leave in the keystone client and make everyone bind to the keystoneclient19:06
morganfainbergalaski, not sure if this would cause any issues within nova, quotas, etc19:06
jamielennoxgyee: i've been thinking about rewriting the Managers/Resources part of the clients as well19:06
gyeeayoung, right now the stuff is all over the place19:06
gyeeI looked at all the clients, they are all doing their own stuff19:07
morganfainbergalaski, i don't remember nova using the userid beyond maybe auditing, but ... we don't want to break things :)19:07
jamielennoxgyee: yea, but oslo is copy and paste and that's bad for clients that will often share the same process19:07
ayounggyee, if we get it right, and solve the compression issue for swift, we should be in good position19:07
jamielennoxit means everyone has a copy of the session which is completely wrong19:07
morganfainbergalaski, it's part of https://review.openstack.org/#/c/74214/1219:07
alaskimorganfainberg: what would be the max length of user_id@domain?19:07
ayoungand a security nightmare , jamielennox19:07
jamielennoxgyee: yea, i've been going through them as well trying to figure out how much of a problem it will be to convert them19:07
alaskimorganfainberg: right now user_id is stored in a varchar(255) in the nova db19:07
morganfainbergalaski, i think we're moving it to varchar(255)19:07
dolphmmorganfainberg: we'd be at max 130 chars (64 + 2 + 64), right?19:07
jamielennoxgyee: essentially - it's big19:07
dolphm(in terms of an actual migration)19:08
ayoungdolphm, yes19:08
jamielennoxgyee: swift is the most interesting one - python-swiftclient is nothing but a CLI app, there is no library19:08
alaskiokay, so that wouldn't involve a migration then.19:08
gyeejamielennox, ayoung, I really think we should push this to Oslo and press others to adopt it19:08
dolphmjamielennox: ?19:08
dolphmjamielennox: i've used the library...19:08
alaskiuser_id is used to store per user quotas for some resources19:08
ayounggyee, nah..this way is working...watch19:08
ayounggyee, https://review.openstack.org/#/c/76109/19:09
alaskiso we'd need to find a way to update those appropriately19:09
morganfainbergyes, we'd move to a varchar 255 https://review.openstack.org/#/c/74214/12/keystone/identity/backends/sql.py (top of new file, line 28)19:09
ayoungalaski, can you bump the size of the field in the DB?19:09
jamielennoxdolphm: is it just flat then?19:09
morganfainbergso in theory, nova would need to support the same19:09
ayoungin practice, too19:09
jamielennoxi took the fact that there was no class objects to mean they were donig the CLI thing and discovering commands19:09
morganfainbergayoung, :P19:10
morganfainbergayoung, in practice i think 130 is the reality19:10
morganfainbergbut 255 limit means 25519:10
gyeejamielennox, take a look at swiftclient/client.py19:10
alaskiayoung: we store it in a varchar(255) already, so a bump isn't needed19:10
gyeejamielennox, it just doing import of keystoneclient.v2_019:11
jamielennoxgyee: hmm, ok, i took the fact that they are recreating a connection on every command if not passed in as evidence that they were doing one off calls19:11
morganfainbergayoung, so i htinke a ML topic to -dev would be in order for the uid size bump19:12
morganfainbergbut my guess is most people are using varchar25519:12
gyeejamielennox, that's what I meant, if push the stuff to Oslo, it will eliminate these kind of stuff19:12
jamielennoxgyee, dolphm: lets go with swift just went way out left field on there client19:12
jamielennoxgyee: right, but session and stuff can just be imported from keystone or some other base library19:13
alaskimorganfainberg: this will break quotas, though not all of them.  we key some of them off of project_id and user_id19:13
jamielennoxgyee: but the base client object and stuff can go somewhere19:13
gyeejamielennox, take a look at oslo-incubator/openstack/common/apiclient19:14
jamielennoxgyee: i don't particularly want keystoneclient to be the special flower here, other than owning the auth19:14
alaskimorganfainberg: we could probably just strip the @domain part for now to keep everything working, but eventually we'd want a better solution19:14
jamielennoxgyee: oh - i know it well19:14
morganfainbergalaski, we're not planning on changing existing ids.19:14
gyeewe want authentication and session management in there19:14
alaskimorganfainberg: oh, excellent19:14
jamielennoxgyee: not in oslo19:14
morganfainbergalaski, just making sure new ids are a new format. -- except in an explicit deployer "i want new id format and i use LDAP as the default domain" setup19:14
jamielennoxgyee: i want this stuff to be imported once by everyone but not copied around everywhere19:15
gyeejamielennox, sure, just do the import thing in Oslo19:15
gyeedef get_session() { from keystonclient import session, ...}19:15
jamielennoxgyee: ok, can import from there, just don't have the code live there - for the baseclient sure it can go to oslo19:16
gyeejamielennox, the reason for Oslo is consistency19:16
jamielennoxgyee: yes and no - the reason for oslo is at least partially that the projects can still do overrides locally for various oslo bits19:17
jamielennoxi don't know if we want that for the clients19:17
jamielennox(beyond normal inheritance)19:17
gyeejamielennox, we can't prevent overrides19:18
jamielennoxgyee: i know19:18
gyeewe can only educate :)19:18
jamielennoxgyee: have you looked through the code in oslo/apiclient? that base client object is kind of a mess19:18
gyeejamielennox, oh yeah, I spent a great deal of time staring at it19:20
jamielennoxgyee: so i've been considering the idea of rewriting the Managers, any thoughts?19:20
gyeeI understand your pain19:20
alaskimorganfainberg: if the new format only applies to new users then everything should be fine with quotas.  And everywhere else that user_id is used, except on spot that stores it as a varchar(36).19:20
morganfainbergalaski, ok so we should toss up a ML topic19:20
morganfainbergalaski, and get that one spot fixed19:20
gyeejamielennox, ++, I thought the keystoneclient layering design is awesome!19:21
alaskimorganfainberg: yeah.  the volume_usage_cache would need updating.19:21
gyeethat why I was asking to push the stuff to Oslo19:21
jamielennoxgyee: apparently now that KDS, auth plugins, and pecan are stuck on review i need some other completely immovable thing to bang my head against19:21
jamielennoxgyee: oh yea, i definetly want to push that to the other clients - it's just whether we'll launch it from oslo or keystoneclient19:21
gyeejamielennox, I am all code review this week! sorry I got stuck in the HP internal stuff the past couple of weeks19:22
jamielennoxgyee: nah, it's not stuff that moves very quickly - particularly around feature freeze19:22
gyeeyeah, just need to pick our battles, too many changes19:23
jamielennoxgyee: i'm not sure exactly what i want to fix yet in managers other than remove passing all those objects being created via **kwargs and build_url19:23
gyeejamielennox, I did a prototype sometime back to do away with all the CRUD managers19:25
gyeehave one config file describing the API and the data types19:25
jamielennoxgyee: i'd love to see it19:25
jamielennoxhmm, not sure about config file - i still think we should do it in code19:26
gyeethat way, you can add any API and any attributes19:26
jamielennoxbut possibly something with a bit more structure to it19:26
gyeeby updating the config file19:26
gyeemost of the CRUD manager are doing nothing but data type validation19:26
gyeethat's pretty much it19:26
jamielennoxgyee: i was messing around with the SDK concepts and got as far as: https://github.com/jamielennox/python-mysdk/blob/master/mysdk/api/identity.py19:27
jamielennoxgyee: then i started looking at other clients and got disheartened again19:27
jamielennoxgyee: agree on the validation - but if you do it via config it means that the user can modify the structure and i don't think thats a good idea19:28
jamielennoxkeystoneclient version X should support exactly the following things and having people tweak the local config files seems wrong19:28
*** marek|away is now known as marekd19:29
gyeejamielennox, we have a need to support extensions, I was trying to find way to do that without code changes19:29
jamielennoxgyee: yea, extensions :( i don't know what to do about that19:29
gyeejamielennox, I was thinking schema discovery19:30
gyeebut didn't get very far with it19:30
jamielennox /extensions is dead and i don't know if the clients can really support a full blown manager discovery approach19:30
jamielennoxgyee: yea that's tricky19:30
morganfainbergayoung, dolphm, do you want me to send the email for user_id size increase?19:31
jamielennoxgyee: did you read the blog i did on the session object the other day?19:31
morganfainbergayoung, dolphm, or do we want henrynash ?19:31
gyeejamielennox, not yet, still catching up on my emails19:32
ayoungmorganfainberg, my thought is go ahead and send it.19:32
jamielennoxgyee: that's ok, you should know most of it anyway - it's just showing examples of how to use the session: http://www.jamielennox.net/blog/2014/02/24/client-session-objects/19:32
jamielennox(when it's complete)19:32
morganfainbergayoung, i'll get something sent today.19:33
morganfainbergi'll tag it as [all] since it's not a keystone issue19:33
gyeejamielennox, like I say, I love the current design19:33
morganfainbergwell [all][keystone]19:33
jamielennoxgyee: yep, it's giving dtroyer some grief i think and i wanted to get it in front of the SDK guys as well19:34
gyeejamielennox, what part of that design dtroyer doesn't like?19:35
ayoungjamielennox, I think he means the whole auth problem is giving dtroyer grief19:37
*** leseb has joined #openstack-keystone19:41
dtroyer:)   mostly I need something that works…ideally the knowledge isn't forced too low in the stack (i think it is again) but I can live with it if we get the next level up right19:41
*** leseb has quit IRC19:42
*** leseb_ has joined #openstack-keystone19:43
gyeedtroyer, all you need is Session right? it pretty does it all19:43
gyeepretty much19:43
gyeeversion discovery, auth, token management, caching19:44
dtroyerit does too much right now and sounds like folk want it to do more…I want Session to be JUST a wrapper around requests.Session that knows how to the the extra bits we want, ie headers, debug, logging, etc.  Maybe the JSON e19:45
dtroyerthat is too low for the caching as to do the cache right you have to know something about the data being passes, that isn't a transport layer problem19:45
dtroyerdiscovery uses Session but also isn't a transport layer problem19:45
gyeeI don't view Session as a transport layer19:46
jamielennoxdtroyer: i agree on the caching - i don't want that there19:46
jamielennoxdtroyer: did you see the blog post i wrote (linked above)19:47
dtroyerI haven't read it all yet, been in DevStack-weekend-catchup mode19:47
jamielennoxi don't disagree on the transport level isolation, but i'm fairly convinced that something like the session object is needed19:48
gyeeanything in Session is customizable anyway19:48
jamielennoxand given that we are 100% on requests now i'm not sure what begin able to swap out a transport layer gives us19:48
dtroyerswap out?  ;)   I'm on the start-over warpath19:48
jamielennoxwell what's in session is pretty much a start-over19:49
dtroyerright…now for the next layer up19:49
morganfainbergayoung, dolphm, sent http://lists.openstack.org/pipermail/openstack-dev/2014-February/028125.html19:49
morganfainbergjamielennox, ^19:49
jamielennoxi was looking at whether i could have a requests transport layer and a httplib transport layer for the swift/cinder guys but i don't think it's worth it19:49
morganfainbergi'm leaving out the deployer choice to change the format for LDAP default domains, that is an exceptional case19:50
jamielennoxdtroyer: ok so moving up a layer19:51
jamielennoxthat gets us to the base client object right?19:51
jamielennoxdtroyer: i wrote https://github.com/jamielennox/python-mysdk/blob/master/mysdk/api/identity.py yesterday trying to abstract v2 and v3 APIs19:52
jamielennox(it's pretty rough)19:52
gyeejamielennox, line 15-32 is eye sore19:53
jamielennoxbut from line 70 and 85 i think the baseclient object in keystone is enough19:53
jamielennoxgyee: yep, i wanted to see what it would take to completely replace v2_0.client and v3.client19:53
jamielennoxthe only thing that they did that baseclient didn't was 35-87  and i had to import everything again19:54
jamielennoxbut yea the imports are exactly what was in v2_0.client and v3.client19:54
jamielennox(also i didn't intend to show this to people yet so it's very rough)19:56
gyeejamielennox, yeah, that prototype won't support extensions19:58
jamielennoxgyee: yea, that one i think we don't want to19:58
jamielennoxgyee: that was looking specifically at the openstack-SDK effort and whether we can use the existing clients and just have SDK do the abstraction (which i think is the right approach)19:59
jamielennox(though it will take a lot of effort to get the clients to that point)19:59
dtroyerFWIW, there isn't a lot of sympathy for using the existing clients, at least in their current form20:00
jamielennoxanything that cannot cross API version boundaries doesn't belong in SDK IMO20:00
gyeedtroyer, you think?!!! :)20:00
jamielennoxdtroyer: yea, i know20:00
jamielennoxdtroyer: but i really don't like the duplication of effort20:00
*** leseb_ has quit IRC20:01
jamielennoxbecause we will continue to support the python-*clients for a fair while and it means that for every new api we need to add it to both places20:01
dtroyerI don't either, which is one reason I want to use your new lower-level stuff from keystoneclient, I think it can be generalize easily enough to work for all of it20:01
jamielennoxyep, my thought was if i can make a good enough baseclient then the SDK will push people towards the session object for there clients20:02
dtroyerbut not as an import from ksc, absorb it directly20:02
jamielennoxand i think that the base i've got in keystoneclient is pretty good20:03
gyeebesides, version discovery, auth, token management, and caching, the CRUD managers are doing just plain data validations20:03
jamielennoxdtroyer: i don't like oslo for this, i'd prefer that to be in it's own library - but sure20:03
dtroyerI've already turned it into an sdk-like package (without auth yet through) and it seemed to go well enough20:03
dtroyernot oslo either20:03
dtroyerthe SDK is going to be built as a unit with multiple layers (packaging tbd)20:04
dtroyerone of the biggest pain points for any client is the installation nightmare we have now20:04
gyeeOslo is one package20:05
dtroyeractually, it isn't, there are multiple…oslo-incubator is one package, not everything goes through it though20:05
gyeebut what is Oslo really, its purpose, is for everything that is common or shared isn't it20:06
jamielennoxdtroyer: ++ on installation nightmare, but if we go down that route then we should kill of the python-*clients altogether20:07
dtroyerjamielennox: I want to make the alternative compelling enough that they will die a natural death, pain-free, unlike their users20:07
jamielennoxdtroyer: this will mean though that whatever package assumes that umbrella is going to have to take changes for every new API in OpenStack20:08
dtroyershould those projects make that decision, yes.  but everyone wants extensions too…that doesn't mean they can't maintain extensions, but on top of a clean base20:09
dtroyerI have a mechanism for anyone to add to OSC just by installing a properly written package20:09
dtroyerso at that level, we are there today20:09
jamielennoxdtroyer: right the plugin approach which SDK has decided against20:10
dtroyerhowever, to push it down into the SDK we need a new mechanism20:10
dtroyerit needs the same effect but a different process20:10
dtroyerRax will assure that works20:10
dtroyerthe point is that the SDK will have to take on a lot of stuff, the same people can help do the work, but it will be vetted for consistency and sanity across a wider surface than a single project20:11
*** leseb has joined #openstack-keystone20:13
jamielennoxdtroyer: do you have any idea what that process is? - i would prefer we didn't rewrite setuptools20:14
dtroyerI think they do plan on not using setuptools, so there needs to be something.  I'm not the python guru, gaynor and noller jointly hold that title in that room...20:15
dtroyerentry points are a problem in a number of ways, it totally prevents decent pacakging of OSC for use on, say, windoes20:16
jamielennoxdtroyer: yea, one of the writers of the packaging format for python works in my office and he essentially pleaded not to write a competitor20:16
dtroyerjamielennox: I don't want a competitor, but jeez, python packaging makes CPAN look totally sane20:17
jamielennoxdtroyer: yea, i know20:18
jamielennoxdtroyer: so do you think we drop the 'manager' concept for clients?20:19
dolphmjamielennox: morganfainberg: talking kite in openstack meeting20:19
dolphmTC meeting20:19
dtroyerah, a better topic!   I want to try, I don't think we need both managers and resources to do the pivot20:19
dtroyerand resources can actually be useful, sometimes.  thats why the experiment with the object commands.  I wanted to do another set with the usual CRUD operations to see how bad it would be20:21
dtroyerbut, you know, time and all20:21
*** marekd is now known as marekd|email-me20:23
jamielennoxi think the resources are very useful, i would much prefer all these things (resources) to exists as actual objects that you can do some sort of validation of rather than the passing of dicts that we do now20:24
jamielennoxhowever the command objects would appear to work better for the CLI where you are mapping command -> object20:24
jamielennoxin the library case the CRUD operations are going to be common for most resources20:25
*** theocean154 has joined #openstack-keystone20:25
*** leseb has quit IRC20:27
*** theocean154 has quit IRC20:28
*** theocean154 has joined #openstack-keystone20:29
*** gyee has quit IRC20:30
*** leseb has joined #openstack-keystone20:30
*** alaski has left #openstack-keystone20:36
dtroyerjamielennox: I think I lost the browser tab with your blog post in it, hit me with the url again please?20:42
jamielennoxdtroyer: http://www.jamielennox.net/blog/2014/02/24/client-session-objects/20:42
*** fabiog has quit IRC20:50
*** jraim_ is now known as jraim20:56
ayoungthe automated sample conf file should not be checked into git.  Autogenerated files should be autogenerated.  Its causing rebase issues on every single one of my patches.  Anything that touches config.21:01
*** david_lyle is now known as david-lyle21:01
*** thedodd has quit IRC21:02
*** dolphm is now known as dolphm_50321:18
*** topol has joined #openstack-keystone21:19
*** stevemar has quit IRC21:19
*** thedodd has joined #openstack-keystone21:21
*** chandan_kumar has quit IRC21:27
ayoungwhat is that other stuff in tox?  like debug helper?  And why is there both py27 and venv?   Why is sample_config in its own env?21:30
*** gyee has joined #openstack-keystone21:34
*** amcrn has quit IRC21:40
*** lnxnut_ has joined #openstack-keystone21:48
*** lnxnut has quit IRC21:51
morganfainbergsimo, ping21:52
*** gokrokve_ has joined #openstack-keystone21:55
*** leseb has quit IRC21:57
*** gokrokve has quit IRC21:57
*** theocean154 has quit IRC21:57
*** lnxnut_ has quit IRC22:03
*** lnxnut has joined #openstack-keystone22:04
*** leseb has joined #openstack-keystone22:05
*** lnxnut has quit IRC22:08
simomorganfainberg: pong22:14
*** topol has quit IRC22:14
bknudsonfor some reason nova-api gets multiple auth_token token caches.22:14
bknudsonso one cache might have the token cached and it works for 5 mins and another doesn't have it cache so the operation fails.22:15
bknudsonmaybe there's one per "thread"22:15
morganfainbergsimo, so i understand your concern about storing plain-text keys, but if I want to extract a hexdigest of the data, why shouldn't that be available w/o having to load everything into hashlib separately?22:15
morganfainbergsimo, re https://review.openstack.org/#/c/7225922:15
morganfainbergsimo, perhaps a separate method that does the same thing but is not claiming to be a "secure password hashing"?22:16
simomorganfainberg: it is not really difficult to run a blob of binary data through an hexifier22:19
simoyou do not need to use hashlib for that22:19
morganfainbergsimo, i just feel that rather than having to do that separately, we already have a mechanism that does in the lib you're relying on22:20
simoI am conservative with crypto stuff22:20
simothis is not necessary, and unnecessary stuff just increase the chances stuff can go wrong22:20
simomore code to review, more bloat, etc...22:20
morganfainbergsimo, see i disagree that this here is not needed, adding in a "go make a hex digest" puts that code duplication on the consuming projects22:21
morganfainbergsimo, and makes me feel like the provided lib is lacking in some base functionality that the underlying lib provides, why bother using it22:21
morganfainbergsimo, /puts on the obnoxious dev hat ;)22:22
simomorganfainberg: and I have actually already been asked why the heck I separated  the extract and expand phases in 2 functions, ideally we should think about coalescing them in a single function indeed22:22
morganfainbergsimo, see that i would be a different landscape, i'd still argue a "digest" of the data should be available22:22
morganfainbergsimo, it's useful functionality.22:23
morganfainbergobv. i can't force ya to accept something like this22:23
simoI do not deny it could be useful to someone22:23
simomorganfainberg: what do you need it for right now ?22:23
bknudsondoes a token_cache_time of 300 make sense? a token can be valid for 5 mins after it's revoked?22:23
morganfainbergsimo, hexdigest of the data for memcache keys, i can't use binary data for the memcache key22:24
morganfainbergsimo, and i was aiming to use the lib as a sign/encrypt mechanism for the backend.  i wasn't expecting this hexdigest to be used as a key anywhere.22:24
morganfainbergsimo, it's related to https://review.openstack.org/#/c/72291/22:24
morganfainbergsimo, so the back-end of a dogpile store can be validated as "good data" or "secure" data.  the encryption key separate from the digest used as a cache/storage key for the Key Value Store22:25
simomorganfainberg: you are going to store derived keys in memcached ?22:25
morganfainbergsimo, it's strictly as a unique memcache key for looking up the data22:26
morganfainbergsimo, a digest of the signed or encrypted data22:26
*** lnxnut has joined #openstack-keystone22:26
simowhy don't you just use a hash for that ?22:26
simowhy do you need a key derivation function as a key handle ?22:26
morganfainbergsimo, your key derivation function is just a hash ;)22:26
morganfainbergsimo, and the data was already there.22:27
simono, not just a hash, a hash with specific properties22:27
simowhich is uselessly slow for your use case22:27
morganfainbergsimo, in either case doesn't matter. i don't have time to work on this at the moment, i was wondering what your reasoning for not allowing a hexdigest was22:28
simoif you have a real use case I can be convinced to say yes, but you have my opinion above :)22:28
morganfainbergsimo, sure.22:28
morganfainbergsimo, it was more curiosity than anything else22:28
*** marcoemorais has quit IRC22:29
*** dolphm_503 is now known as dolphm22:29
*** lbragstad has quit IRC22:36
*** theocean154 has joined #openstack-keystone22:39
morganfainbergi think we have some potential concerns about the aggregate user_id stuff22:41
morganfainbergon the mailing list.22:41
morganfainbergi think i have all the responses i need but i might want dolphm, henrynash, or ayoung to weigh in as well22:42
*** leseb has quit IRC22:42
morganfainbergand it looks like heat and nova are the only two so far with table column length issues22:43
morganfainbergif someone else doesn't jump on in response, i'll do so later today/tonight22:43
*** thedodd has quit IRC22:44
morganfainbergsimo, if i have a better formed use-case i'll let you know when i can circle back on that oslo thing22:45
morganfainbergsimo, i might also punt the crypto bit until a second patchset anyway, so i can habe you and other more crypto minded people involved22:46
morganfainbergand less focused on the whole dogpile abstraction bit22:46
dolphmmorganfainberg: we might be able to have the best of both worlds, sort of as jay described22:55
morganfainbergdolphm, perhaps22:56
dolphmmorganfainberg: uuid.uuid4().hex[:CONFIGURABLE_LEN] + '@@' uuid.uuid4().hex[:CONFIGURABLE_LEN]22:56
morganfainbergdolphm, so always use a UUID and do a mapping table?22:56
dolphmmorganfainberg: oh, i just mean provide some means to produce shorter IDs, so the combined UUID ends up being 32 or less22:57
morganfainbergdolphm, because remember, we do things like use bits of a DN for LDAP users22:57
dolphmmorganfainberg: or under 64, at least?22:57
morganfainbergdolphm, i dunno if we can do that w/ the LDAP stuff included22:57
morganfainbergdolphm, it's an issue with the differing things each backend provides us22:58
dolphmmorganfainberg: yeah, i guess not...22:58
morganfainbergdolphm, if we _always_ used a uuid and did internal mapping (maybe not the best idea?)22:58
morganfainbergdolphm, it becomes a non-issue22:59
morganfainbergbut it adds overhead to each lookup22:59
dolphmmorganfainberg: absolutely, but it solves the problem with zero impact...22:59
morganfainbergdolphm, ++22:59
dolphmmorganfainberg: and could be cached quite well22:59
morganfainbergdolphm, i'm thinking its likely a way better way.22:59
morganfainbergdolphm, create an internal map that simply says "take ID and get id@@domain"23:00
morganfainbergor well (id, domain) back or some such23:00
morganfainbergold ids can be dn partials or whatever, and we add a mapping bit in the middle23:00
dolphmmorganfainberg: when would that table be populated?23:00
dolphmmorganfainberg: on auth?23:00
dolphmpost- successful auth?23:00
morganfainbergdolphm, on migration for upgrade, for new IDs on auth?23:01
morganfainbergdolphm, it feels icky to hold data about ephemeral users though23:01
dolphmmorganfainberg: very...23:01
morganfainbergdolphm, perhaps we can make the map a reversible something or another so ephemeral user comes in, and the unique id (name?) we use + domain can be hashed/unhashed23:02
morganfainbergdolphm, so we can ensure it's always the same?23:02
dolphmmorganfainberg: we talked about that... it'd have to be encrypted?23:03
dolphmor b64?23:03
morganfainbergdolphm, hm. b64 i think would be fine in this case23:03
morganfainbergdolphm, for federated users23:03
*** marcoemorais has joined #openstack-keystone23:03
dolphmmorganfainberg: are we solving for federation or multildap?23:04
morganfainbergdolphm, both23:04
morganfainbergdolphm, federated would likely need a uid_transform method23:04
morganfainbergbut if it was something we could subclass/call based on IDP info23:05
morganfainbergwe could do something like uuid for LDAP, and add to the mapping table, and federated does b64 magic?23:05
*** openstack has joined #openstack-keystone23:05
morganfainbergor if is_uuid -> lookup in mapping, if b64 decode?23:06
dolphmmorganfainberg: L77 https://review.openstack.org/#/c/71353/42/keystone/auth/plugins/saml2.py23:06
morganfainbergdolphm, aye ok so yes23:07
dolphmmorganfainberg: name and, indirectly, ID are entirely defined by the mapping23:07
dolphmmorganfainberg: ID = url safe ( name )23:07
morganfainbergdolphm, alternatively we could b64 ALL new ids instead, and we can decode from that?23:08
dolphmmorganfainberg: there's no guarantees as to uniqueness or anything about the name there23:08
morganfainbergdolphm, which is why we'd need something to include IDPID in those cases and be able to decode it23:08
morganfainbergmulti-ldap would be eaiser with an internal mapping table, but could be done in the same manner23:08
dolphmmorganfainberg: idp id is included in the token, ultimately23:08
morganfainbergdolphm, sure.23:09
dolphmmorganfainberg: as is protocol id, but not a reference to the mapping23:09
morganfainbergdolphm, but we need to hold the idpid in the user_id for assignment grant table stuff23:09
morganfainbergwell, effectively23:09
morganfainbergbecause it needs uniqueness since we already know we can't trust an IDP to provide unique data23:09
dolphmmorganfainberg: i really don't want to support user-based role assignments on ephemeral users :(23:09
morganfainbergthe whole reason we want to include the idpid on the user_id.23:10
morganfainbergdolphm, i was on board with that, but i thought we lost that argument23:10
dolphmmorganfainberg: unless we're just randomly assigning uuid's -- then it's viable but still dumb23:10
morganfainbergdolphm, maybe the way to provide unique IDs is a b64 encode of "name:idpid" basically.23:11
dolphmmorganfainberg: i -2'd the two reviews allowing assignments on non-existent users23:11
morganfainbergdolphm, ah ++ cool then23:11
dolphmmorganfainberg: (just last week, i think)23:11
morganfainbergdolphm, it would (potentially) work the same for ldap users.23:11
morganfainbergdolphm, multi-ldap that is23:11
morganfainbergif we can guarantee a maximum size of user-id data derived from ldap23:12
morganfainbergand sql is easy, but if it works the same, it keeps thing simple23:12
morganfainbergdolphm, if we can do the user+idp info w/ zero impact to the id field, i'd be happiest (I agree w/ jay/shardy on this)23:13
morganfainbergdolphm, though that we already support 64 and some people are limited to 32, that should be fixed to match what we currently allow as the maximum field size23:14
morganfainbergrather than us truncating down.23:14
dolphmmorganfainberg: so, we'd literally have like an 'ephemeral_user' table? with (uuid, source) ?23:14
morganfainbergdolphm, if we used b64, we could do the same id generation and add idp23:14
morganfainbergand it's reversible23:15
morganfainbergdolphm, if we did something like an internal sql, yes.23:15
dolphmmorganfainberg: who's limited to 32, btw?23:15
*** dstanek has quit IRC23:15
morganfainbergone table in nova (36?) and heat23:16
morganfainbergthey are 6423:16
morganfainbergread comment and didn't read code. comment says 32 is the max size, but they have acolumn at 6423:16
dolphmlen(str(uuid.uuid4())) == 3623:16
morganfainbergso yeah nova has a single column that is 3623:17
morganfainbergi'll propose a fix this weekend to solve that23:17
morganfainbergalready talked to russtle about that23:17
dolphmmorganfainberg: reduce to 32 or increase to 64?23:17
morganfainbergincrease to 6423:17
morganfainbergmatch our schema23:17
dolphmmorganfainberg: why not reduce ours to 32?23:18
morganfainbergor to 255 if we go forward with the current proposed impl for henrynash23:18
morganfainbergdolphm, or if we reduce to 32, we can propose a fix to others23:18
dolphmor 25 if you encode uuid's to base36... or...23:18
morganfainbergi don't care which way, as long as our schema matches the other schemas23:18
morganfainbergif we set a max size of 64, no one should expect less data than that23:19
morganfainbergthey should keep it in sync.23:19
morganfainbergwe shouldn't be changing it often enouigh to warrant this discussion more than once in a very long time23:19
morganfainbergif ever again23:19
morganfainbergin fact...23:19
morganfainberg-2 if we ever need to talk about this again for the life of V323:20
morganfainbergdolphm, anyway. the question is... what do we do to uniquely identify a user in a way that we can determine the IDP (federated, multi-ldap, etc) with zero change to our schema23:24
morganfainbergdolphm, is it possible?23:24
dolphmmorganfainberg: zero change to API or schema?23:24
morganfainbergdolphm, erm zero change to the maximum user_id length (64) as per our schema23:24
morganfainbergthe API, i am assuming, wont care.23:25
morganfainbergthe REST api that is23:25
dolphmmorganfainberg: identity_api.get_domain_id(user_id) # SELECT domain_id FROM user_domain_map WHERE user_id={user_id};23:27
morganfainbergdolphm, ok.23:28
morganfainbergdolphm, i'm fine with that23:28
morganfainbergdolphm, do we want this for I? https://review.openstack.org/#/c/74214/ would probably need it23:29
dolphmmorganfainberg: i'm not really solving for domain_id vs idp_id though -- do we need to solve for both?23:29
morganfainbergdolphm, we can probably ignore the idp issue.... probably23:29
dolphmmorganfainberg: yeah, i think i think it'd be the critical piece to that patch23:29
morganfainbergdolphm, though, i'd rather not have different logic for the two mechanisms23:30
*** achampion has quit IRC23:30
dolphmmorganfainberg: right23:30
*** theocean154 has quit IRC23:35
*** huats_ has quit IRC23:36
*** david-lyle has quit IRC23:38
dolphmmorganfainberg: want to respond on -dev or shall i?23:44
morganfainbergdolphm, i can respond later tonight but not atm23:45
morganfainbergdolphm, happy to let you respond23:45
morganfainbergdolphm, in the middle of setting up some infrastructure nodes for internal company CI stuff23:45
dolphmmorganfainberg: YAY CI23:45
dolphmmorganfainberg: i'm about to eat dinner anyway23:46
morganfainbergdolphm, well i'll response to -dev if you haven't once i'm done with this task23:46
*** huats has joined #openstack-keystone23:54
*** nkinder has quit IRC23:54
*** browne has quit IRC23:57

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!