21:00:54 #startmeeting Hierarchical Multitenancy 21:00:55 Meeting started Fri Jan 31 21:00:54 2014 UTC and is due to finish in 60 minutes. The chair is vishy. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:56 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:58 The meeting name has been set to 'hierarchical_multitenancy' 21:01:15 #link https://wiki.openstack.org/wiki/HierarchicalMultitenancy 21:01:26 #topic Role Call 21:01:32 who is here for the meeting? 21:01:39 \o 21:01:42 o/ 21:01:44 hi 21:01:46 hi 21:01:55 o/ 21:01:56 hi 21:02:34 did everyone get a chance to read the wiki? 21:02:36 https://wiki.openstack.org/wiki/HierarchicalMultitenancy 21:02:42 yes 21:03:10 yes, had a look 21:03:12 hi 21:03:13 i think the use case there gives us a good starting point 21:03:17 had a look 21:03:24 #info Martha, the owner of ProductionIT provides it services to multiple Enterprise clients. She would like to offer cloud services to Joe at WidgetMaster, and Sam at SuperDevShop. Joe is a Development Manager for WidgetMaster and he has multiple QA and Development teams with many users. Joe needs the ability create users projects, and quotas, to list and delete resources across WidgetMaster. Martha needs to be able to set 21:03:25 quotas for WidgetMaster and SuperDevShop, manage users, projects, and objects across the entire system, and set quotas for the client companies as a whole. She also needs to ensure that Joe can't see or mess with anything owned by Sam. 21:04:12 i think that clearly states the use case that I am trying to enable 21:04:56 and what part of that doesn't work now 21:05:24 the biggest problem in what exists today regarding roles and domains is that there is no way to enable the "domain admins" to control resources from their domain 21:05:31 If I understand well, the idea behind the US, would be to have some kind of recursive domain ? 21:05:38 they would have to manually join and login to each project 21:05:49 florentflament: recursive tenants/projects 21:05:51 in order to control them 21:05:52 we have the inherit extension now 21:06:00 dolphm, agree 21:06:20 ayoung: so inheritance allows for some basic functionality 21:06:29 i.e. the domain admin could have a role in every project 21:06:49 vishy: yes 21:06:49 is there a use case for more than 2 levels deep? 21:06:54 but the only way of listing all instances in a domain would be to get a list of all projects from keystone and list all of them individually and aggregate 21:07:04 gyee: i think there is but I'm leaving that aside for the moment 21:07:10 are these hierarchical roles? 21:07:11 al iinstances of a vm? 21:07:27 yes all virtual machines 21:07:41 so under Potential Solutions I have this listed 21:07:49 "Remove Cross-Project Functionality" 21:08:05 we could just say, sorry you always have to authenticate per project for resources 21:08:16 gyee: yes, we do have such use cases 21:08:17 which I think is horrible user experience but it roughly works today 21:08:24 bknudson: no 21:08:44 so I personally believe that multiple levels is what we really want 21:08:55 but I want to prove that this can work before we have that discussion 21:08:57 vishy: we could bring domain scoped token to openstack, that improves the user experience 21:09:24 tiamar: I think that is the wrong approach personally, but let me expand on my proposal for now 21:09:36 fwiw, I'm with vishy that multiple levels is the meets-all-needs solution, but that getting there is a lot harder and some intermediate steps might be helpful 21:10:08 so one of the issues is figuring out how a hierarchical structure would work inside of the projects 21:10:17 regardless of whether this is domain.project 21:10:24 or project.project.project etc. 21:10:39 i.e assuming I have a broader grouping of objects than a project 21:10:44 vishy, then there's really no distinction between domain and project then 21:10:44 how do I support that in code 21:10:50 gyee: correct 21:11:05 gyee: which is why i think hierarchical projects removes the need for domains 21:11:09 vishy: ++ 21:11:15 but that is something that is orthogonal to this proposal 21:11:20 domains are the namespaces not really grouping 21:11:20 a "domain" basically becomes a project with no parent project 21:11:24 vishy, I suppose there's always a root project, which is formerly known as domain? 21:11:27 a "root project" 21:11:36 dolphm, ha 21:11:44 gyee: we would have a root project for sure 21:11:46 are you going to allow users with the same name in different projects? 21:11:50 but again not important to decide now 21:12:13 i don't want to digress into domains vs. projects because it is unimportant for this discussion 21:12:21 bknudson: identity domains and authorization domains are really two separate issues -- i'm hoping we can ignore identity domains for the sake of this conversation 21:12:26 i want to prove that multiple levels of ownership works in the projects 21:12:38 currently we have a single "owner" field in nova 21:12:51 it is called project_id in most places 21:12:52 gyee: "domain" has never been anything more than an arbitrary different name for a 2-level hierarchy, instead of moving to an N-level hierarchy 21:13:19 gabrielhurley, I am not critical on names :) 21:13:24 +1 21:13:28 gabrielhurley: it has a few unique qualities as well, shuch as namespacing unique users 21:13:36 call it whatever, as long as it functions consistently 21:13:37 fair 21:13:46 the obvious but inelegant solution is to expose a call in keystone to ask "is this resource's owner a subset of this authorization scope?" (letting keystone track the hierarchy) 21:13:46 ok so here is the basic plan i have 21:14:05 that would get hammered a lot, but it could be a cacheable HEAD request 21:14:06 I think, its look ok to scrap domain and have a hierachical structure of projects...but I guess moving to this is quite difficult than having domains and having maximum of 3 levels of hierachy i.e domains -> Tenants -> Users at this moment 21:14:06 dolphm: yeah I think that is a nasty performance choice 21:14:23 dolphm: let me cover my proposal 21:14:34 vishy: go for it, i just wanted to get that out of the way :) 21:14:38 so i want to make a branch of devstack/nova 21:14:49 that can test this without actually modifying keystone 21:15:06 i will create some feaux heirarchies using . as a separator 21:15:19 so I will have a project called companya.foo 21:15:27 another one called companyb.foo 21:15:37 companya.bar 21:15:41 companya 21:15:44 companyb 21:15:52 and corresponding users 21:16:04 so this doesn't deal with role inheritance or anything like this 21:16:31 but i want to prove that a user who authenticates to companya can list/delete in companya.bar and companya.foo 21:16:34 vishy, that would make . forbidden in any names 21:16:35 but nothing in companyb 21:16:37 etc. 21:16:44 gyee: I am aware of that 21:16:50 but this is just a prototype 21:17:01 keep in mind that the reall enforcer will actually be the id 21:17:12 so they will have matching . 21:17:13 ids 21:17:14 etc. 21:17:29 so the real requirement is that . is not allowed in ids 21:17:31 But what about delegate some user as admin at companya.bar so that he can further creates sub projects...if you go like this, then hierachy keeps on growing and implementing through policy.json will be difficult 21:17:34 so, tracking and enforcement of the hierarchy remains distributed 21:17:52 dolphm: correct 21:17:56 uuid have usability cost, but sure since this is a prototype 21:18:12 so there are two things that need to be done in the service to make this work 21:18:37 first of all the enforcement of project_id = object.project_id needs to be expanded to a substring/regex match 21:19:00 and secondly (the harder one) is to change the enforcement of list operations to actually pass filters into the db 21:19:20 so perhaps there is a new type of enforcer in policy similar to rule: 21:19:35 that might be regex: or substring: or something 21:19:46 so the admin_or_owner check would actually use 21:19:55 a partial match on project_id instead of an exact match 21:20:14 then to optimize the list case this needs to be preconverted into some kind of sql filter 21:20:17 you're saying they try to list all instances and they only get some instances? 21:20:25 bknudson: correct 21:20:33 if you outhenticate to companya 21:20:45 you get all instances from companya companya.foo companya.bar 21:20:51 but none from company.b 21:21:06 bknudson, that's how authorization is usually implement, via resource filtering 21:21:10 currently the --all-tenants flag just says are you an admin? ok then send everything 21:21:15 vishy: so, how do you *really* list all instances? 21:21:21 there is no scoping 21:21:32 nova list --all-tenants 21:22:00 dolphm: we could leave the "superadmin" role in for filtering but if we convert everything to this model 21:22:06 * for no filtering 21:22:06 then what about somebody authenticating with scope of companya.bar wants to list instances only under this (not at parent project companya) 21:22:17 the key distinction here is that a user expects list operations to default to "show me everything I have access to unless instructed otherwise" (e.g. the highest scope) and that create operations should default to "create this just for me unless instrcuted otherwise" (e.g. the lowest scope. 21:22:20 then it makes sense for the superadmin to actually be the root of the tree 21:22:23 i.e. 21:22:28 you can always specify more specific list or create scopes 21:22:29 openstack.companya.foo 21:22:32 but the defaults are crucial 21:22:37 where openstack is the root 21:22:51 gabrielhurley: ++ 21:22:57 but that would be a later change. For now we can just leave the superadmin role in place that does no filtering 21:23:02 vishy: +1 on "openstack" being the root 21:23:30 gabrielhurley: having a read vs write scope is interesting but initially i think forcing someone to auth at a certain level isn't too horrible 21:23:51 gabrielhurley: so short term i would just take that approach 21:24:04 whether the scope is set by explicit auth or API param I don't really care 21:24:07 i.e. i can authenticate in project foo 21:24:16 or i can go authenticate in openstack and list everything 21:24:36 I'm just telling you what a naive user expects 21:24:44 stray thought: this makes default tenancy in keystone suddenly attractive again, as it's likely that all my projects share a single parent 21:24:46 gabrielhurley: i can see people getting confused by seeing stuff from other projects in a list 21:24:55 quite the opposite 21:25:07 people are confused by openstack's rigid scoping and all the context switching 21:25:16 makes it hard find things 21:25:23 gabrielhurley: i guess that is true 21:25:26 forces users to keep track of all their scopes in their head 21:25:30 higher mental load/ burden 21:25:32 gabrielhurley: you could have something similar to a cwd though 21:25:37 representing your scope 21:25:50 anyway side point for the future 21:25:54 ++, that's a cool idea 21:26:09 the good news about this approach is that it will work in the projects with fairly minimal changes 21:26:09 users are people, people are self-centered foremost, group-centered secondmost 21:26:14 but this is not cirtical 21:26:18 * morganfainberg is here now 21:26:24 and it doesn't force keystone to implement new features before we are sure they are actually valuable 21:26:28 yep 21:26:43 so I create an instance and set its project to openstack.companyA.groupC 21:26:45 vishy, ++ on backward compat 21:26:47 vishy: except for maybe user-defined project ID's? and authorization around doing so? 21:27:03 vishy, i do like backwards compat 21:27:11 now there are a bunch of other issues that will come up as we try and implement and we will know what we need to fix 21:27:26 there are unsolved problems like quotas 21:27:28 vishy: and about domains? Will it supersede them? 21:27:37 dolphm: the prototype version i'm going to just hack the id manually in the db 21:27:49 leandrorosa: that is a decision we can make in atlanta 21:28:06 the output of this experiment would be a summit discussion and a clear proposal 21:28:32 leandrorosa, it might supersede some aspects of domains but likely not all, we will need to evaluate the prototype to see what ends up 21:28:41 dolphm: if the experiment is successful than i see keystone walking the hierarchy and sending a parameter called scope 21:28:46 vishy: +1 21:28:54 vishy: CERN wishes to implement quota delegation - I think it comes handy now 21:28:56 which would include the dot.separated.hierarchy 21:28:59 vishy: sending where / when ? 21:29:00 I thought the token already had the scope. 21:29:09 dolphm: along with project_id 21:29:16 maybe it needs a different name 21:29:27 tiamar: correct 21:29:27 basically the entire hierarchy of the project 21:29:33 which could be domain.project today 21:29:42 or it could be project.project.project 21:29:48 vishy, scope is heavly used already. so a different name would be good, or we supplant the current "scope" concept 21:29:56 openstack.company.business_unit.project 21:29:57 ++ to all this scope stuff 21:30:02 vishy: would that not just be the project_id = grandparentproject.parentproject.project ? 21:30:05 name notwithstanding 21:30:09 fair enough call it hierarchical_context 21:30:15 dolphm: that is exactly what it is 21:30:33 my point is that the project in the database doesn't need the . in that case 21:30:46 ah 21:30:50 it is constructed by keystone and sent along with roles / etc. 21:31:03 so user-defined project ids isn't necessary 21:31:23 so what are you thinking the auth request looks like? 21:31:37 domain is a top level container that can contain projects and users. Projects can be hierarchical without removing the usefulness of domains. 21:31:41 bknudson: which auth request? communication with keystone? 21:31:49 ayoung, ++ 21:31:50 vishy: yes, to keystone to get the token. 21:31:52 dolphm, any idea what does it look like potentially, for service-scope? 21:32:00 ayoung: that is fine by me, as I said beyond the scope of this experiment 21:32:09 I think we would still keep the idea that VMs etc must be in a projectr, not just a domain 21:32:28 ayoung: sure that is ok if you want distinction there 21:32:39 then we just scope this down to "hierarchical projects" 21:32:55 gyee: i don't think this impacts service-scoped tokens, if that's what you're referring to 21:33:07 gyee, i think that it wont change anything we're talking about on that front. "service scope" is not exactly related to what nova is doing here. remember service-scope was non-project related 21:33:08 ayoung: fine by me 21:33:22 service scoped tokens could benefit from this as well. Put services inside a project.... 21:33:34 everything that needs to be managed goes in a projects 21:33:34 ayoung: is there any use case that cannot be covered with purely hierarchical projects ? 21:33:42 ayoung: i think to make this valuable the projects need to inherit some of the functionality that you stuck in domains though 21:33:59 i.e. i think the right answer is that every user needs to have a maximum hierarchical scope 21:34:00 schwicke, it solves all things for all users everywhere using all languages on all platforms 21:34:13 but it shouldn't be a separate concept 21:34:16 vishy: no delegation outside that scope? 21:34:21 ayoung: my rationale is the following 21:34:30 you use domains to control adding users to projects yes? 21:34:41 you need to use the heirarchy for that 21:34:53 "assigning user sroels in projects" 21:34:58 yes 21:35:22 so you want to be able to delegate the ability to delegate 21:35:22 but anyway we could rathole on that particular point forever 21:35:36 ayoung: right the ability to delegate should be a role in the hierarchy 21:35:43 in my could, you can create a project, and assign admin rights in the project to dolphm 21:35:48 for example assign_role should be a role that you get somewhere in the tree 21:36:03 and you have that role for anything below that location in the tree 21:36:27 and if you go that far, i "think" the only thing that domain gives you is namespacing of users 21:36:28 +1 21:36:36 vishy, you mean project-specific role definition? 21:36:47 and you could do namespacing of users via something like the highest point in the tree where they have a role 21:36:55 or even an explicit role called namespace or some such 21:36:58 vishy: thus you can assign/delegate that role to any other user for any subset of the project hierarchy? 21:37:06 domain is a top level namespace. We could have implemented domains as nested projects. I think someone even suggested that. 21:37:35 that was me that duggested that 21:37:39 *suggested 21:37:42 ayoung: we both did :) 21:37:54 both supported* it lol 21:37:56 ayoung: i think the best approach would be a) see if this nested idea works in the non-keystone services b) implement nested projects if it works c) consider replacing domains if they are no longer necessary 21:38:09 +1 21:38:14 or else we risk pissing people off :) 21:38:17 vishy, domains will stay. We have other reasons now for the,m 21:38:19 them 21:38:20 the property of namespacing users and such is an interesting case 21:38:32 vishy: timeline on your devstack prototype? 21:38:32 fair enough 21:38:34 there may be other properties like that 21:38:36 ayoung: i'm not sure we do 21:38:36 for example: 21:38:46 jgriffith: I will "start" it this weekend 21:38:50 and post on github 21:38:56 oh yes we do....but do you want me to explain now, or save it for the summit? Its long 21:39:04 i hope to have the absolute minimum done my monday 21:39:25 ayoung: identity providers have the opportunity to supersede domains on the authentication side, and hierarchical multitenancy has the opportunity to supersede domains on the authorization side 21:39:29 I would appreciate help from others trying it out, refining it, discovering all the random bugs we will fine 21:40:07 vishy: I think we can help there, Vinod to confirm 21:40:09 vishy: sounds like the best next steps.. thanks 21:40:15 yes 21:40:24 any other comments or concerns? 21:40:24 keen on it actually 21:40:25 and yeah, I'll try and help where I can 21:40:26 vishy, so the big thing to watch out for is the logic in querying nested things. Databases don't do that well 21:40:35 #action vishy to prototype devstack/nova this weekend 21:40:42 ayoung: yeah i have some concerns about that 21:40:45 But i do agree with ayoung, that domains are necessary 21:40:45 and applying things based on hierarchy can be nasty....there are a lot of details to get right 21:41:05 i'm going to try something simple and see what breaks 21:41:09 ayoung, you mean like performance? :) 21:41:11 Hey, lets rename projects to domains and mess everyone up 21:41:16 i'm hoping to make minimul changes to the policy checks in nova 21:41:24 but i think i will have to invert some of them 21:41:24 vishy: can you provide a link in the original -dev thread? 21:41:29 ayoung: +1 21:41:35 a link to the github? 21:41:38 ys 21:41:39 yes 21:41:40 absolutely 21:41:59 yes 21:42:01 vishy, so long as project are referred to by ID and not by name, there should be no change 21:42:21 the calculation of roles for a user on a project will be more expensive at token creation time 21:42:40 ayoung: for usability in this version I'm going to use the "hack" of showing the heirarchy in the project name 21:42:52 vishy..do you mean that you will implement this and post it on github and do testing from our side 21:42:58 vishy, that is fine, but policy enforcement is done on projectId not name 21:43:10 ayoung: right, but login is done via name 21:43:18 so i need to modify both for the prototype 21:43:24 vishy, can be done by either iirc 21:43:33 vishy, leave users in domains out of it, and let people log in using their existing mechanisms 21:43:48 authorization and authentication are two separate things and should be managed separately 21:44:00 +1 21:44:08 +1 21:44:16 the authentication problem spent too much of its history being painful already 21:44:16 +1 21:44:18 IR. let the user create an account (self registration) and then add that account to the project via a role 21:45:25 vishy, you with me? See the distinction? 21:45:46 But then authentication should check only user is valid or not and give some token or something with his roles etc information. The services do validate this token and then apply RBAC rules to check the authorization and also to decide the scope 21:45:48 vishy: is momentarily distracted in the office here 21:45:54 ayoung: so a new user would by default not be able to do anything until he gets a role assigned, right ? 21:45:55 he'll be back in 60 seconds 21:45:57 that is one advantage of cross domain assignments. 21:46:25 schwicke....yes.. 21:46:29 schwicke, yes, although you can actually pre-assign the role before the user is created. In the federation case, that will be the norm 21:46:41 makes sense 21:46:44 +1 21:46:45 in a self-service world a user should have a default "just them" project and role... in a more controlled world they'd have nothing. depends on the deployment 21:46:46 but ideally speaking he has atleast role of a member and we should atleast allow to query his own things 21:46:57 sorry boss walked by :) 21:47:21 I'm working on a proposal to allow the domain backends to control one portion of the userid and the other portion will be the domain id. That way, two IdPs can both assign a userid without them potentially conflicting 21:47:41 ayoung: if domains are a per authn thing that is fine 21:48:03 so if you know your domain ID, and you know the ayoung user is coming, you could create a role assign for ayoung** for your project 21:48:34 vishy, they can also be the top level container for projects without breaking anything, so long as we have the delegation you are recommending. I like this approach 21:49:04 ayoung: yes that is true but it sounds like it might be better to just keep them as a grouping for users and leave projects out of it 21:49:04 If you have role r on project p you also have role x on all sub projects of p 21:49:16 vishy, only if we hate out users 21:49:21 lets no make them relearn 21:49:37 the only exception is a domain might have a maximum scope in the hierarchy 21:49:50 i don't like them at the top and here is why 21:49:53 domain is a project with no parent. Its ok to give it a different name 21:49:58 say i'm a service provider 21:50:03 its really no different than the difference between hostname and URL 21:50:11 i provide services to two companies each with their own idm 21:50:27 as the service provider i need to be able list/delete in both companies 21:50:32 so i need a root above the domain 21:50:43 but they should be limited to their side of the tree 21:51:06 I agree with vishy 21:51:10 ayoung I think there should be a root to the tree 21:51:21 vishy: ++ 21:51:22 works for me 21:51:24 I've been saying that for ages 21:51:28 domains at the top means you have more than one root 21:51:29 No you can either be a super user/admin or have a role as Domain Admin in both companies 21:51:44 VINOD_: you could do it that way but why 21:51:51 when you can stick it all under one tree 21:51:59 VINOD_: where does the super-admin role get assigned? currently roles are tied to projects/domains 21:52:03 and then it functions the same way at all levels 21:52:13 gabrielhurley: makes a good point there ^^ 21:52:15 no role shoud grant you premissions above/outside the thing it is granted on 21:52:22 gabrielhurley: +++++++++++++++++++++++++ 21:52:44 you could do that with a single domain 21:52:48 currently the user with role "admin" is super user 21:52:56 one domain, and sub project for each customer 21:53:09 VINOD_, we are fixing that 21:53:13 currently even if a user has been assigned a role of "admin" in a tenant, he actually becomes the super admin 21:53:16 ayoung: then each customer is limited to a single project 21:53:18 ayoung: but that kills all the reasons you'd want to ahve multiple domains 21:53:19 VINOD_: currently the role admin exists on a project which means we have this annoying habit of creating a special admin project 21:53:20 VINOD_, lets leave that out, that is a policy thing 21:53:22 ayoung ok 21:53:23 ayoung: they can't have their own multitenancy isolation 21:53:24 which is so gross 21:53:30 dolphm, not with hierarchical projects 21:53:43 the "admin domain" and "admin tenant" are terrible hacks that need to die a brutal death 21:53:53 dolphm, with hierarchical projects... ayoung said it 21:54:11 ok good so my intention here isn't to get keystone to start working on hierarchical projects right away, but its nice to know you guys aren't opposed to the idea. 21:54:12 dolphm, agreed. if we have a single root-domain though we've just implmented the same thing 21:54:14 dolphm, if the service provider is going to manage the scope for the customer, whether we call it a domain or a project is irrelevant 21:54:20 dolphm: ++ 21:54:23 root is the domain. 21:54:38 would it be easier to implement hierarchical projects in keystone? 21:54:45 if you want a single root, have a single domain. The hierarchy stuff works fine 21:54:47 than to fake up something else? 21:54:55 bknudson, they need to be in Keystone 21:54:58 ayoung: sure but i thought you wanted separate idms 21:55:02 for each customer 21:55:05 bknudson: the distributed authorization enforcement is the harder problem that needs to be tackled first 21:55:05 with namespaced users 21:55:09 vishy, assignment backend is split from Identity 21:55:25 ayoung: ok were fine then 21:55:41 yes we are...thanks for driving this forward 21:55:46 cool 21:55:54 will email everyone when i have a basic prototype 21:56:02 it will likely only support one or two calls 21:56:03 vishy, cool. 21:56:06 and will do ugly db hacks 21:56:09 great! 21:56:14 vishy, lets talk 21:56:15 and modify openstack.common in place 21:56:19 yay db hacks! 21:56:23 we can, I think, do better than that 21:56:29 you have been warned! 21:56:41 ayoung: you mean on the db hacks? 21:56:46 Both LDAP and SQL have different ways of supporting hierarchy, but both drivers can be made to work 21:56:48 yeah 21:57:00 as could KVS... 21:57:08 ayoung: Ok, this will be 10 minutes of the prototype work so no need to do that now 21:57:11 all these changes should be restricted to the assignments backend 21:57:16 ayoung: this doesn't impact ldap 21:57:19 vishy, that is what scares me 21:57:23 dolphm, tell that to CERN 21:57:30 ayoung: CERN agrees 21:57:33 LDAP does this bettr than SQL 21:57:34 ayoung: rememember? 21:57:37 ayoung: we are here :) 21:57:40