18:02:18 <dolphm> #startmeeting keystone
18:02:19 <openstack> Meeting started Tue Jan 15 18:02:18 2013 UTC.  The chair is dolphm. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:02:20 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:02:20 <gyee> full house today :)
18:02:22 <openstack> The meeting name has been set to 'keystone'
18:02:35 <dolphm> #topic Team Membership Updates
18:02:43 <dolphm> official welcome to henrynash and gyee :)
18:02:51 <henrynash> thank you, kindly
18:03:02 <gyee> look forward to do some serious damage
18:03:02 <ayoung> Good job on code reviews this past week.  Much appreciated you two
18:03:14 <dolphm> keep it up :)
18:03:19 <henrynash> :)
18:03:26 <dolphm> #topic Test Coverage
18:03:30 <dolphm> ayoung: any updates?
18:03:34 <ayoung> topol, want to join the weekly Keystone meeting?
18:03:40 <topol> yes
18:03:57 <topol> this is it correct?
18:04:01 <ayoung> dolphm,  no major changes in test coverage has happened this week
18:04:09 <ayoung> dolphm, going back one step
18:04:20 <joesavak> jorgew and myself are here too. : )
18:04:23 <ayoung> topol is going to be helping out on LDAP support
18:04:30 <dolphm> yay!
18:04:41 <ayoung> topol, who is the other dev with you?
18:04:46 <topol> sahdev will be helping too
18:04:51 <topol> say hi sahdev
18:04:53 <spzala> ayoung: hi Adam, I would like to join keystone weekly meeting as well.
18:05:20 <ayoung> cool
18:05:21 <spzala> Hi all!
18:05:26 <dolphm> spzala: /salute
18:05:30 <gyee> welcome
18:05:37 <ayoung> This is starting to feel like a real project.
18:05:39 <topol> Thanks
18:05:41 <spzala> :-) thanks!
18:06:05 <ayoung> so, back to test coverage.  Lets open a Jenkins bug to rerun the test coverage results
18:06:16 <dolphm> ayoung: a bug?
18:06:31 <dolphm> ayoung: i.e. we lost coverage on something?
18:06:34 <ayoung> dolphm, wasn't it there before and got removed?
18:06:48 <ayoung> I thought you said last week that we used to run coverage on each Jenkins run
18:06:58 <ayoung> otherwise RFE
18:07:32 <ayoung> Should be as simple as adding the flag to run_tests.sh, but maybe jenkins doesn;t use that
18:07:39 <dolphm> ah that kind of coverage
18:07:52 <ayoung> yeah, unit test coverage
18:08:00 <dolphm> i don't see the charting anymore on jenkins, but we have the infrastructure for it as far as i can tell
18:08:16 <dolphm> keystoneclient too
18:08:25 <dolphm> see tox.ini
18:09:19 <ayoung> dolphm, so, for example, http://logs.openstack.org/18519/5/check/gate-keystone-python27/2010/
18:09:27 <dolphm> ayoung: want to tackle that?
18:09:34 <ayoung> dolphm, sure
18:09:56 <ayoung> #action ayoung get jenkins to generate unit test coverage report
18:09:57 <dolphm> #action ayoung to trace down jenkins coverage reporting
18:10:06 <dolphm> and do it twice!
18:10:09 <joesavak> ha
18:10:19 <dolphm> #topic critical issues
18:10:49 <dolphm> i know we have a security-related fix in the works, anything else on fire?
18:11:06 <ayoung> http://wiki.openstack.org/Meetings/KeystoneMeeting
18:11:36 <dolphm> ayoung: yeah, criticalness appears to have dissappeared from the agenda
18:11:54 <ayoung> For everyone new, Agenda is ^^
18:12:05 <dolphm> i noticed yesterday that db_sync no longer supports specifying a migration number -- need to file a bug on taht
18:12:29 <dolphm> #action dolphm to file a bug on lack of db_sync migration number option
18:12:42 <ayoung> dolphm, I don't think it is critical, but the is an issue identified by tempest WRT roles that is coming up as a bug.  I'll take it for now
18:13:11 <ayoung> #link http://paste.openstack.org/show/29472/
18:13:24 <ayoung> jaypipes is opening/assigning
18:13:34 <dolphm> cool
18:13:52 <dolphm> i'll assume that's it for now
18:13:55 <dolphm> #topic "Default" domain migration
18:14:09 <dolphm> so, this is a topic i've only scratched the surface on, but i wanted to give everyone a heads up on what i'm doing
18:14:14 <dolphm> this is probably bp worthy, so...
18:14:22 <dolphm> #action dolphm to write default-domain blueprint
18:14:51 <ayoung> dolphm, how does that tie in with henrynash 's domain blueprint...
18:15:10 <dolphm> anyway, we'll be creating a data migration to create a Default domain (id=default, name=Default) that will contain user/tenant resources on the V2 API
18:15:10 <ayoung> #link https://blueprints.launchpad.net/keystone/+spec/domain-name-spaces
18:15:15 <henrynash> dolphm: I'm already monkeying around with domains, so feel free to assign it to me :-)
18:15:37 <dolphm> i assume it'll use private namespaces once that bp is implemented
18:15:47 <ayoung> +1
18:16:11 <gyee> henrynash, I meant to response to your domain namespace proposal
18:16:12 <dolphm> any operation on the v2 API will automatically work with default domain resources (creating a new tenant on v2 will create a project in v3 in the default domain)
18:16:24 <dolphm> we may also need to special case the domain so that users can't accidentally delete it from the v3 api
18:16:52 <dolphm> and i'd also like to make the default_domain_id a configuration option so that users can override the behavior and delete the domain we create if they'd prefer
18:17:04 <dolphm> any objections?
18:17:08 <dolphm> or questions
18:17:14 <ayoung> dolphm, no.
18:17:20 <ayoung> dolphm, a suggestion
18:17:23 <dolphm> or suggestions
18:17:26 <gyee> +1 on default_domain_id
18:17:30 <henrynash> dolphm: we need to also define what happens when (in v3) a user creates entities without specify a domain (e.ff. users)
18:17:34 <ayoung> we should probably gues the default domain from the DNS domain name.
18:17:39 <ayoung> But that is an install issue
18:18:00 <henrynash> do they go in the default domain?
18:18:04 <dolphm> henrynash: my thoughts-- automatically create the user in the creating user's domain
18:18:12 <dolphm> same for projects
18:18:16 <joesavak> +1
18:18:28 <joesavak> (autlo create the userin the creating user's domain)
18:18:56 <ayoung> why users domain and not "default" domain?
18:19:02 <topol> the same domain you plan to use for v2 tenants or are they separate?
18:19:12 <ayoung> Or,  if there is more than one domain, why not force them to specify?
18:19:20 <dolphm> ayoung: as a v3 user, that doesn't seem intuitive to me
18:19:31 <dolphm> ayoung: as a v2 user, i don't care about domains, but the implementation has to
18:19:54 <ayoung> dolphm, so the question is how should the V2 API behave?
18:20:02 <dolphm> ayoung: in terms of..?
18:20:12 <ayoung> if no domain is specified.
18:20:22 <dolphm> ayoung: v2 api isn't domain-aware
18:20:22 <ayoung> V2 create user needs to guess the domain
18:20:29 <ayoung> OK,  I can go with "same as user"
18:20:51 <ayoung> for V3, lets make it default if there is only one domain, and must be specified if there is more than one?
18:20:59 <dolphm> ayoung: that's an interesting thought...
18:21:07 <dolphm> ayoung: we can apply that behavior across both API's
18:21:19 <dolphm> ayoung: or, hrm
18:21:55 <dolphm> ayoung: can users not in the default domain authenticate on the v2 api? they absolutely can't with a privately namespaced domain
18:22:02 <ayoung> dolphm, I'm ok with guessing for V2.  We don;t want to break existing tools, and matching the existing user seems to make the most sense
18:22:07 <dolphm> ..without using ID's
18:22:18 <joesavak> FYI - jorgew and i have a hard stop at 12:30 and would like to discuss multi-project/token scope
18:22:35 <dolphm> i'll write the BP and ping the list
18:22:50 <henrynash> dolphm: +1
18:22:56 <dolphm> #topic multi-project tokens in V3
18:23:06 <joesavak> dolphm - thank you
18:23:23 <ayoung> joesavak, any change in your thinking from last discussion?
18:23:31 <dolphm> #link https://review.openstack.org/#/c/19014/
18:23:53 <joesavak> We've updated to include the roles under tokens per your suggestion ayoung
18:24:17 <ayoung> joesavak, +1
18:24:19 <henrynash> so I think the proposal hangs together fine….as long as we are convinced we need the capability…sorry for keeping on asking about that...
18:24:41 <joesavak> henrynash: the rax use case may be unique but it could happen again. We had a swift product runnning under 1 tenant. We acquired a company (mosso) that is the compute product running under a different tenant structure
18:24:41 <jorgew> So we have two major reasons for the feature
18:24:44 <ayoung> henrynash, it should not be a general use thing, but only for specific use cases that span multiple tenants
18:24:59 <jorgew> and joe just mentioned one of them :-)
18:25:13 <joesavak> so to do a back up of a compute instance and to store in a swift instance, we need a token capable of seeing 2 projects
18:25:13 <jorgew> ayoung, +1
18:25:15 <ayoung> the question is how to consume them
18:25:35 <jorgew> The other is to keep in mind that V2 supported the feature at least in the contract
18:25:52 <ayoung> My thinking is that we should make a multi-project token look like a single project token, but provide the data to the application from the middleware
18:26:11 <jorgew> ayoung,  I agree.
18:26:14 <ayoung> auth_token will just say "yep, this is OK"
18:26:32 <ayoung> the unmarshalled token data should then be in the context
18:26:41 <jorgew> in fact because a resource can belong in a single project, then and the validate call can check for that the change is minimal
18:26:44 <ayoung> Now, RBAC will come into play
18:27:08 <dolphm> joesavak: v2 supported this feature in the XML contract, but the feature was lost in the JSON interpretation of that contract
18:27:31 <jorgew> dolphm,  I think our implementation went ahead and added the support in JSON :-)
18:27:52 <jorgew> +1 on adding support for JSON schema :-)
18:28:00 <dwchadwick> seems the obvious answer, fix JSON
18:28:01 <joesavak> +1 ; )
18:28:10 <dolphm> ayoung: we don't have a way to provide multiple projects to services underlying auth_token today, i wrote a comment in the review about how to go about supporting that
18:28:45 <ayoung> dolphm, right.
18:29:02 <joesavak> any questions on the use cases presented?
18:29:30 <dolphm> dwchadwick: which will immediately break all clients written against the current implementation; it's too late to go back and match the intent at this point on v2
18:29:34 <dwchadwick> We have an alternative use case based on federated identities
18:29:35 <ayoung> dolphm, my point is that it should be possible to use a multi-project token like a single project token, perhaps based on the ordering of the projects requested.  Some projects are more equal than others.
18:29:42 <dolphm> dwchadwick: please share
18:30:16 <dwchadwick> When you present a set of identity attributes from the federated infrastructure, then these are mapped into local tenants/domains/roles
18:30:16 <ayoung> dwchadwick, that uses a different middleware, correct?
18:30:20 <henrynash> joesavak:…so once we different services based on different trees, I think I get the need
18:30:20 <dwchadwick> yes
18:30:42 <dwchadwick> so the client is presented with a set of tenants to choose from
18:30:55 <gyee> ayoung, I am not sure about the ordering
18:31:01 <dwchadwick> currently he picks just one, but can swap to another one later if he wants to
18:31:25 <gyee> depending on ordering is too easy for implementations to screw up
18:31:36 <dwchadwick> so the client has a set of accounts he can use from his single authentication session. Thus it should be possible to support this with keystone tokens
18:31:42 <dolphm> simply migrating a resource from ownership by project A to ownership by project B requires authz on both projects (which is a use case requested on the mailing list recently)
18:32:08 <dwchadwick> that is a different case to the one I am talking about
18:32:15 <dolphm> granted you could implement it with multiple tokens
18:32:25 <ayoung> dolphm, I have to say, I really don't like the _build_user_headers  aspect of auth_token middleware.  It feels wrong
18:32:45 <ayoung> But, if we must...
18:32:59 <ayoung> X_TENANT_IDS,  X_TENANT_NAMES ...
18:33:02 <ayoung> comma separated
18:33:15 <ayoung> X_ROLES...ugh
18:33:16 <dolphm> ayoung: what if the name includes a comma?
18:33:35 <gyee> or in chinese? :)
18:33:47 <dolphm> gyee: right
18:33:58 <ayoung> <lando>This deal is getting worse all the time! </lando>
18:34:05 <dolphm> i think you need to JSON encode a list of strings (w/ unicode support)
18:34:43 <ayoung> dolphm, and, since this is all consumed by the WSGI app after the auth_token middleware, it should all be in the context, and not in custom headers
18:35:34 <gyee> I think various context are built from the headers right now
18:35:38 <dolphm> ayoung: i'm not aware of another way to communicate between middleware layers except through the environment
18:36:20 <gyee> we can stash the entire token access into the env
18:36:27 <ayoung> gyee, yeah
18:36:29 * dolphm runs
18:36:43 <ayoung> gyee, change "we can" to "we should"
18:36:50 <gyee> amen
18:37:02 <joesavak> we are diving quite deep into the implementation for multi-projects/token. Is that needed to agree on https://review.openstack.org/#/c/19014/ ?
18:37:08 <dolphm> then you're expecting underlying services to understand identity api v2, v3, v4, etc
18:37:17 <dolphm> which is not their responsibility
18:38:00 <gyee> we version the tokens
18:38:03 <gyee> might as well :)
18:38:05 <dolphm> joesavak: the only snag i see is that we'll also need to support multi-domain tokens since we're also going down the domain-scoped route
18:38:06 <dwchadwick> You need to move to an extensible design for the API if you can, so that new features can be added without changing the api
18:38:11 <ayoung> dolphm, multi project tokens are breaking the xcontract regadless of who we break it
18:38:15 <jorgew1> dolphm,  really it's the middle ware that needs to understand this.  Note that a resoruce still exists in a single project
18:38:36 <dwchadwick> the attribute type-value approach solves at least part of this
18:38:58 <dwchadwick> so domains would have just become yet another attribute in the set, and the API would not have needed to change to cater for domains
18:39:15 <joesavak> +1 dwchadwick
18:39:31 <gyee> dwchadwick, what does the attribute for project_ids like?
18:39:45 <dolphm> jorgew1: the implementation spans client + server + middleware though
18:39:49 <dwchadwick> project-ID=11324
18:39:54 <gyee> services still need to be aware of the attributes right?
18:40:02 <dwchadwick> project-ID=456578
18:40:13 <gyee> for multi-project scoping?
18:40:21 <dwchadwick> yes they do, but that then becomes a config issue of the service
18:40:35 <henrynash> so I think we may be getting off in the weeds here….
18:40:36 <ayoung> dwchadwick, the question is not whether we can solve this in the general case.  The question is can we solve this with the existing contract.  I'm all for attribute based, but that still means changing the other projects consumption of the tokens
18:40:43 <gyee> how does that different from X-Project-IDs?
18:41:19 <ayoung> We've left the weeds behind.  We are in the woods, crossing a stream
18:41:23 <dolphm> ;)
18:41:27 <ayoung> and I think I spotted a deer tick
18:41:27 <joesavak> lol
18:41:45 <dolphm> alright...
18:41:48 <dolphm> #topic SQL Testing
18:41:58 <dolphm> #link https://review.openstack.org/#/c/18519/
18:42:05 <ayoung> joesavak, I'm OK with the concept of multi-proejct tokens, but I don;'t think the existing projects can consume them yet.  Any thing consuming them needs to be a one off.
18:42:08 <dolphm> ayoung: ^
18:42:24 <ayoung> OK, so the review above again has a test fix
18:42:28 <dolphm> ayoung: don't think = can't :)
18:42:33 <gyee> ayoung, a great topic for the next summit
18:42:36 <joesavak> ayoung -  ok. thanks for everyone's attention on it.
18:42:45 <ayoung> https://review.openstack.org/#/c/18519/5/keystone/common/sql/migrate_repo/versions/009_normalize_identity_migration.py
18:42:55 <ayoung> This downgrade was not tested
18:43:10 <dolphm> my only realy issue with https://review.openstack.org/#/c/18519/ is that it's effectively stepping on the toes of tempest, which is already built for this
18:43:14 <ayoung> it probably should have its own unit test, but it gets run as part of the downgrade
18:43:17 <henrynash> dolpm: fyi ,we missed the v3 token discussion topic on the agenda
18:43:36 <ayoung> dolphm, disagree
18:44:01 <jaypipes> ayoung: done: https://bugs.launchpad.net/keystone/+bug/1099966
18:44:03 <uvirtbot> Launchpad bug 1099966 in keystone "Race condition when rapidly deleting and creating tokens" [Undecided,New]
18:44:11 <ayoung> tempest should be feeding us test that we should be working into the unit test framework,  but we should be able to  run against live DBs as part of development
18:44:27 <dolphm> henrynash: ah, i'll hit that next/last
18:44:29 <ayoung> A developer shouldn't need to know about Tempest to check that Keystone works
18:44:40 <jaypipes> ayoung: lemme know if the information in the bug isn't enough... happy to provide more.
18:44:50 <ayoung> and while sqlite is a good first approximateion,   We should be able to test against a RDBMS
18:45:15 <ayoung> jaypipes, would you say it is a critical?
18:45:26 <gyee> I have bad experiences where stuff appeared to work in sqlite but blow up in mysql
18:45:47 <ayoung> gyee, and I don;t trust MySQL.  I want PostgrSQL
18:45:56 <gyee> so be able to test against different DBs is definitely useful
18:46:08 <jaypipes> ayoung: meh.. I'm just ignoring the failures in my local tempest env. But it would be nice to fix.
18:46:09 <gyee> ayoung, that's the point
18:46:15 <ayoung> It was essential for LDAP, too
18:46:17 <jaypipes> ayoung: I could give it a go myself.
18:46:24 <jaypipes> ayoung: also, has nothing to do with MySQL.
18:46:38 <dolphm> ayoung: i don't disagree, but the same issue applies to all of openstack... hence, it's a project-wide initiative... ergo, tempest
18:46:40 <ayoung> jaypipes, nah, that is part of the ongoing discussion, not on that bug
18:46:42 <jaypipes> ayoung: and we do run tempest against an env with a PG backend.
18:47:33 <jaypipes> ayoung: https://jenkins.openstack.org/view/Tempest/job/gate-tempest-devstack-vm-postgres/
18:47:45 <ayoung> dolphm, so the remaining changes in that bug (other than the fix to 009) are just to use teardown instead of assuming a blank database.  We could make that conditional on the URL, and assume blank fro sqlite, use teardown otherwise
18:48:00 <gyee> dolphm, somebody's going to do it
18:48:11 <ayoung> jaypipes, we are discussing https://review.openstack.org/#/c/18519/
18:48:49 <ayoung> dolphm, I think that the same approach (check the URL) probably makes sense for the rest of the unit tests.  But that will be a separate commit.  It has other issues to work through.
18:48:50 <jaypipes> ayoung: oh, sorry man.
18:49:03 * jaypipes goes back into his hole.
18:49:33 <ayoung> dolphm, should I move  https://review.openstack.org/#/c/18519/5/keystone/common/sql/migrate_repo/versions/009_normalize_identity_migration.py into its own review like I did the other fix?
18:50:38 <dolphm> jaypipes: lol your input is appreciated
18:51:26 <dolphm> ayoung: i would say so
18:51:40 <dolphm> ayoung: not that it has backport potential, but it'll still merge faster as a discrete fix
18:51:47 <ayoung> dolphm, OK,  will do so.  Then I can redo the above for both upgrade test and unit tests.
18:52:51 <dolphm> ayoung: cool
18:52:56 <dolphm> #topic v3 tokens/authn/authz etc.
18:53:12 <dolphm> i didn't add this, but i assume this was the API discussion in gyee's thread?
18:53:12 <henrynash> guang: I added this so you had a chance to discuss
18:53:28 <ayoung> good idea
18:53:34 <gyee> excellent
18:53:53 <ayoung> OK,  so I still don't really want to build multiple auth support into Keystone if we get it from Apache for free, but...
18:54:03 <ayoung> I do want to be able to use basic_auth and HTML
18:54:12 <henrynash> gyee: vested interest…since I need to make changes to this for both domain-scoping and name-spaces
18:54:23 <dolphm> ayoung: not everyone runs behind apache
18:54:26 <ayoung> And we need a V3 API for Authentication
18:54:33 <ayoung> dolphm, not yet they don't...
18:54:48 <gyee> ayoung, they said you can get java for free before oracle comes in :)
18:54:51 * ayoung twirls mustache
18:55:50 <ayoung> the use case I can see as most important is
18:56:02 <ayoung> keep ID in Mysql but auth against LDAP
18:56:16 <ayoung> I'd like that to be the first supported
18:56:34 <ayoung> assume that syncing users from LDAP to MySQL is done externally somehow...
18:56:37 <dolphm> ayoung: how does that impact the API?
18:56:49 <ayoung> dolphm, that makes LDAP the first supported Auth Mechanism
18:56:58 <ayoung> instead of Google whateveritis
18:57:02 <gyee> with auth plugin, I don't care where you store LDAP users
18:57:10 <gyee> Keystone can just be a proxy for all I care
18:57:12 <dolphm> gyee: +1
18:57:30 <ayoung> gyee,  right.  just understand that this is the single most demanded feature right now
18:57:32 <gyee> since we are going to abstract auth, token issuance, and token validation
18:57:36 <dolphm> gyee: have you worked on a python API for authentication plugins?
18:57:58 <gyee> you can effectively have your own backends
18:58:20 <gyee> dolphm, not formally
18:58:30 <gyee> just have a bunch of brain farts so far
18:58:36 <dolphm> gyee: {'auth': {'password'}} x.authenticate(**contents_of_object_x)
18:58:38 <dolphm> whoops
18:59:20 <ayoung> one question:  how are we going to decide which auth mechanism to use for a given request?
18:59:26 <gyee> dolphm, I'll send out a proposal for the interfaces this week, assuming we all agree on the direction
18:59:30 <dolphm> gyee: {'auth': {'password': {'username': 'a', 'password': 'b'}} -> password_plugin.authenticate(username='a', password='b')
18:59:31 <ayoung> Domain is the most viable option
18:59:49 <dolphm> something like, **explode the contents of hte request that map to the name of the plugin
19:00:05 <gyee> yep
19:00:14 <dolphm> ayoung: according to gyee's proposal, it depends on the contents of the request
19:00:37 <ayoung> dolphm, that is too wide open
19:00:42 <gyee> mechanism maps to the plugin seem simple enough
19:01:00 <dolphm> plugins[plugin_name].authenticate(**credentials) -> resulting in -> plugins['password'].authenticate(username='a', password='b')
19:01:08 <dolphm> ayoung: what is too wide open for what
19:01:15 <ayoung> dolphm, "based on the request"
19:01:22 <ayoung> that implies a whole policy like rules engine
19:01:45 <ayoung> I'd prefer it to be something more implementable\
19:01:48 <gyee> I am not seeing any problem
19:02:05 <gyee> if we don't support a given auth mechanism, we return a 401
19:02:09 <ayoung> like list the auth mechanisms in the config file, and then the domain entry has an auth mechanism specified in it
19:02:27 <dolphm> ah, we're over time
19:02:31 <dolphm> switch to #openstack-dev ;)
19:02:34 <dolphm> #endmeeting