16:00:04 #startmeeting keystone 16:00:05 Meeting started Tue Jul 3 16:00:04 2018 UTC and is due to finish in 60 minutes. The chair is lbragstad. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:06 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:06 ping ayoung, breton, cmurphy, dstanek, gagehugo, hrybacki, knikolla, lamt, lbragstad, lwanderley, kmalloc, rodrigods, samueldmq, spilla, aselius, dpar, jdennis, ruan_he, wxy, sonuk 16:00:08 The meeting name has been set to 'keystone' 16:00:09 o/ 16:00:12 #link https://etherpad.openstack.org/p/keystone-weekly-meeting 16:00:14 agenda ^ 16:00:25 o/ 16:00:38 o/ 16:01:01 o/ 16:01:16 o/ 16:01:57 we'll give it another minute 16:02:44 BEEP BEEP BEEP *hits snooze button* 16:03:02 #topic release information 16:03:14 #info reminder that feature freeze is July 13th 16:03:21 so - just under two weeks away 16:03:35 with that, we have a few efforts that still need reviews 16:03:55 mainly capability lists, unified limits, and mfa receipts 16:04:13 unified limits is mostly ready afaict, just needs eyes. MFA receipts needs in-depth reviewing, and capability lists should be pretty light 16:04:19 o/ 16:04:46 afaik the capability lists stuff needs opinions on the database migrations 16:04:50 friday the 13th is perfect for a release 16:04:51 Yes. 16:04:55 i think that's what jgrassler wanted feedback on 16:04:59 That is kind of blocking all the rest :-/ 16:05:01 i 16:05:06 'll review them post meeting 16:05:13 o/ 16:05:18 #link https://review.openstack.org/#/c/572776/ 16:05:20 WhadIMiss? 16:05:24 * kmalloc is on a tear about migrations and ensuring we don't have wonk-i-ness 16:05:42 so i'll cover migration reviews here [hopefully not too crazy] 16:05:55 so far - that's the only review i'm tracking for capability lists (jgrassler correct me if i'm wrong) 16:05:57 kmalloc: thanks! 16:06:04 lbragstad: yes, it is. 16:06:07 ayoung: 3 initiatives need review - capability lists, unified limits, and mfa receipts 16:06:10 good deal 16:06:28 unified limits has a *whole* bunch of moving pieces 16:06:34 capabilities list is kinda something I made happen, so I'll keep an eye on that, too 16:06:34 but they are pretty isolated 16:06:35 lbragstad: I can't write any of the rest before the data model has ceased to be a moving target :-/ 16:06:37 as for mutable config, we need a way to get events about file updates 16:06:46 uwsgi conveniently has one https://uwsgi-docs.readthedocs.io/en/latest/PythonDecorators.html#uwsgidecorators.filemon 16:06:52 but for other we need a different mechanism 16:06:57 knikolla: right, we can also simply look at mtime 16:07:31 #topic release goals 16:07:49 jgrassler, I'm not a fan of the two table approach now that I look. Lets discuss that later 16:08:02 i just realized i doubled up topics - let's move on to release goals though 16:08:10 we don't have to worry about mox 16:08:13 kmalloc: would we do that every request? 16:08:20 knikolla: and i'm looking at a way to make all our drivers/controllers/etc share state (see the __shared_state__ bit, borg pattern)- if we do that we can simply instantiate a new object as needed. the main issue we've had in the past is that state is explicit and could end up with partial loading. 16:08:28 knikolla: mtime check is cheap. 16:08:29 but we still have yet to decide on how to approach #link https://governance.openstack.org/tc/goals/rocky/enable-mutable-configuration.html 16:08:30 ayoung: sure 16:08:35 knikolla: so we could. 16:08:44 knikolla: [we already do that for policy.json] 16:08:57 alright, i'll push something today during office hours 16:09:02 knikolla: sweet 16:09:07 jgrassler: personally i found it helpful to write things the other way around, e.g. starting with the controller and working backwards https://review.openstack.org/#/c/524423/1 16:09:18 but that's just me 16:09:24 cmurphy: ++ 16:09:31 cmurphy: or at least the manager layer 16:09:36 ya or that 16:10:01 post AppCreds, I want to revisit unified delegation 16:10:24 knikolla: lets chat post meeting re: shared state and config 16:10:34 knikolla: do you want to take an action for the release goal? 16:10:42 so will the flask stuff help us with the mutable config thing? 16:10:50 i was wondering that, too... 16:10:51 cmurphy: it should in some ways. 16:10:57 we've got too many mechanisms, and we should really have one. Now that cmurphy has built one, and jgrassler is about to, we'll have the critical mass to tackle reworking it 16:11:00 because controllers are instnatiated on-demand 16:11:27 so...services should be able to re-read configus without a restart. kill -1 semantics 16:11:39 all we need to do is make a call to oslo.config with the new config. the hard part is knowing when to (which is where the mtime check comes in place) 16:11:43 seems like it should be an oslo-config feature. 16:11:46 driver loading and api loading on-demand if we share state(s) [see https://review.openstack.org/#/c/571955/ as a lead in] 16:11:49 could be done easily 16:12:06 ayoung: oslo-config is doing it, but we have some issues with how we hold state based upon config 16:12:19 kmalloc, like which drivers we've initialized, I take it? 16:12:19 even "debug" - we need to load consistently across the board for the config object 16:12:32 ayoung: and if they have locally stored bits about config 16:12:33 some do 16:12:41 like the LDAP stuff 16:12:48 debug is a low barrier, but generally most of our configs could be made mutable 16:13:32 since we're mostly atomic/stateless between requests [MFA receipts aside, and that is still stateless] 16:13:41 I wonder if we can take LDAP out of the config altogether and use domain specific config mechansim for that and for mappings...hmmmm 16:13:50 debug log is enough to complete the goal 16:13:54 right. 16:13:57 yep 16:14:12 right now i'm just concerned with getting that done ;0 16:14:14 :) 16:14:16 so, anyway, the key is that all keystone processes need to know ocnfig changed and we don't use oslo-service 16:14:22 So we need to do an audit and build a checklist of places where we hold state. Do we have that? 16:14:24 so mtime check is likely the best. 16:14:48 we can't lean on uwsgi alone because not everyone deploys with uwsgi 16:14:52 (though they should) 16:14:55 i'll push something today. there's air conditioning at work (and not at home) so... 16:15:06 knikolla: hehe, ouch and sounds good 16:15:06 knikolla: nice - thanks 16:15:15 actually, I think we cleaned up LDAP when we went to domain specific 16:15:20 anything else on release goals? 16:15:22 kmalloc: hmm, could we though? 16:15:27 lean on uwsgi i mean 16:15:31 cmurphy: i would love to 16:15:43 if uwsgi solves the problem can we just say ops who care about this should deploy with uwsgi 16:15:55 i don't think any of the downstream packages use it 16:15:57 sounds like a good carrot 16:16:04 wrm. 16:16:15 (though I don't understand how uwsgi solves the problem) 16:16:24 yeah, lets pull in the Apache/mod_wsgi smart people and get a solution based on that, too 16:16:28 cmurphy: it has built-in watcher constructs 16:16:29 uwsgi has a filemon decorator https://uwsgi-docs.readthedocs.io/en/latest/PythonDecorators.html#uwsgidecorators.filemon 16:16:49 so instead of checking mtime every request, we decorate a function that mutates the config on file change 16:17:10 now, we'd need to stub that out to address the no-uwsgi form. 16:17:17 because afaik we can't make uwsgi a dep of keystone 16:17:24 I don't see us changing RDO/OSP to a different WSGI. 16:17:38 But I do see us getting a solution based on what we do use 16:17:47 ayoung: i hope RDO/OSP moves to apache/mod_uwsgi 16:17:49 honestly 16:18:05 Apache has a re-read config option. I wonder if that is passed to mods? 16:18:06 mod_wsgi has a lot of quirks that are painful in comparison to uwsgi 16:18:21 and asking apache to re-read config, it's a reload of the app and causes other weirdness. 16:18:41 basically don't expect apache to reload the app unless it's very deliberate "e.g. I deployed a new config, go do that" 16:18:48 I think that is the only safe way, though 16:18:54 mod_wsgi also is not great with python3 16:18:57 if you make changes that have to be applied across the board 16:19:17 if you don't restart, and have old process in the 1/2 done state, it could cause consistency problems 16:19:32 * kmalloc is still of the opinion we should lean on etcd for shared mutable configs. 16:19:35 Keystone should be a fast restart. I wonder if this is a real problem on the Keystone side? 16:19:52 it is a community goal to allow for mutable config loading [debug flag] 16:20:01 What config options, beside debug, would we want to change on the fly? 16:20:11 #link https://governance.openstack.org/tc/goals/rocky/enable-mutable-configuration.html 16:20:16 right now it's just debug 16:20:29 I know that page, but it might not make sense across the board 16:20:30 i'm inclined to say we make that a separate effort 16:20:40 ayoung: i could see token TTL, allow_expiry_window, etc. but def separate initiative 16:21:07 anything that is live-change I say we move into the database 16:21:08 ayoung: profiler changes. 16:21:20 ayoung: anyway. for now, just debug. 16:21:21 read from config, but hold a cached version 16:21:41 (log levels come to mind) 16:21:57 regardless, it sounds like we have a couple paths forward 16:22:00 there's a packed agenda, let's keep it going or move it to -keystone for office hours 16:22:05 only can be updated via keystone-manage command 16:22:10 ++ 16:22:14 #topic oslo_policy types 16:22:15 kmalloc: o/ 16:22:25 ok. so i've been digging around in oslo-policy 16:22:27 a lot. 16:22:35 notably for the RBACEnforcer changes for flask. 16:22:45 it's made me realzie we're missing a few oslo-policy check types. 16:23:18 example: does a key exist in the target_dict, but not in the creds dict 16:23:23 heh 16:23:29 e.g. "must have a filter when doing a list" 16:24:04 right now the DSL cannot make that work because all values from the first part of the colon are explicit key lookups in creds[] 16:24:19 I reopened https://review.openstack.org/#/c/165908/ due to that kind of error 16:24:24 so saying something like "EXISTS():user_filter" 16:24:38 so a user can't list users without a filter of some sort. 16:24:44 not high prio, but useful 16:24:51 kmalloc: ++ 16:24:52 had a case when looking in the token for a value that was in different locations in V2 and V3] 16:24:59 similar with NOTEXISTS 16:25:03 I hit it when adding project_id filter for limits 16:25:16 "you cannot supply a given filter with this role" 16:25:33 e.g. no "list_servers?all_projects=True 16:25:41 this feels like business logic 16:25:47 you should be able to write a rule that, if it is based on a false assumption, will just fail, not cause an API 500 error 16:26:09 for example - https://bugs.launchpad.net/keystone/+bug/1750660 16:26:09 Launchpad bug 1750660 in OpenStack Identity (keystone) "The v3 project API should account for different scopes" [High,Triaged] 16:26:11 lbragstad: sortof, but most apps do not have insight into roles 16:26:23 once you're past the "can i do X" bit 16:26:33 OK, so missing key is a big one. 16:26:37 Any others? 16:26:41 list_users_for_all_domains example 16:26:46 if that is a query-param 16:26:56 we should be able to check if it exists or notexists in the policy dsl 16:27:00 the last one is an explicit match 16:27:10 which would have made ayoung's is_admin_project easier 16:27:21 so...if jamielennox were here, he would state something about it has to be in oslo-context, right? 16:27:22 MATCH(value):TargetDictKey 16:27:38 right now you can do literal matches for Cred dict 16:27:46 you cannot do literal matches for target dict 16:28:03 Keystone was like the privileged child for policy checks, as we had the token data in our WSGI context already, but other services were forced to use oslo-context 16:28:08 use: Action must be performed with user_id X 16:28:09 we finally forced Keystone to use Oslo 16:29:03 we may also want some "ValueIn" and "ValueNotIn" list operators 16:29:11 but those seem further into the weeds 16:30:03 so...mising key, could we treat those as a fail? 16:30:07 being able to explicitly check if a value (any value) is or is not in the target dict [key check] AND being able to check for an explicit value in the target dict. 16:30:23 assume the key is there. If it is missing, check fails? 16:30:29 ayoung: so the idea would be: list_users but role X needs filters 16:30:34 it fails without a filter 16:30:39 "sorry, no filters passed" 16:30:41 key doesn't exist 16:30:59 the converse is "filtering is deemed expensive in my deplpoyment, you may list everything but not filter" 16:30:59 righty, so instead of error. 16:31:07 "filter is present, so error/fail" 16:31:33 and the last one is "filtering on the username FOO is bad, so your username=Foo filter causes a fail" 16:31:40 contrived examples 16:32:08 mostly wanted to seed the idea, there are cases where inspecting the target dict without a corresponding cred value is useful 16:32:41 i'm wondering if this is going to push people to carry more custom policy changes than what they do today 16:32:52 it might. 16:33:40 i might need a bit more time to digest the idea 16:33:40 just in testing alone, being able to do these things without craaaazy mock would be awesome. 16:34:15 even if we do an oslo_policy fixture that enables these added DSL things would help. 16:34:26 but i don't think this is anywhere an emergency to develop 16:34:39 i just had the ideas fresh in my head due to major refactors 16:34:42 what does it mean if the filter is missing 16:34:48 so - maybe a good thing to review in denver 16:35:23 ayoung: your role, "project_sub_admin" may only list users if you provide a filter of some sort, e.g. by username 16:35:38 ayoung: aka list_users?username_starts_with=Foo 16:36:40 if you don't provide a filter, we do not allow that to go through. anyway, 16:36:43 * kmalloc hands mic off 16:36:48 done on this topic 16:36:51 ok - i'll think more on it 16:36:56 thanks kmalloc 16:37:03 #topic token provider API refactor 16:37:08 ok - i'll make this quick 16:37:18 i have a series up that does a ton of refactoring 16:37:24 around how we handle tokens 16:37:37 ideally - it makes the interface for token providers sane and super explicit 16:37:49 what kind of refactorings 16:37:51 feel free to check it out, it should help wtih JWT in the future 16:38:10 i've remove the token_data dict we pass *everywhere* in favor of an object 16:38:15 ayoung: following up on the KeystonetokenModel work we did before 16:38:16 removed* 16:38:27 and i redefined the token provider interface 16:38:37 so...I'd like use to start making a semantic distinction 16:38:38 and finalizing it/making it a real thing vs just a dict-ish thing 16:38:41 a token is opaque 16:38:49 it points to the auth data 16:39:00 and we should keep the two concepts separate in naming 16:39:08 internal to keystone it is an immutable object once minted except in 2 cases 16:39:10 3(* 16:39:13 this is all auth-data 16:39:19 1) Validation -> token model 16:39:28 yeah 16:39:36 2) Token Model -> json (user) 16:39:37 I get a token, I give you auth data, that is validation 16:39:48 3) token model -> oslo_policy enforcer (it needs a dict) 16:39:51 json is marshalling of the auth data 16:40:17 tokens should only be used in reference to the different formats like fernet, JWT, PKIZ, UUID, EIEIE! 16:40:18 internal to keystone it is now an object, attributes etc, loaded from the backend as needed/cached 16:40:48 and the -> JSON, FErnet_ID, JWT, dict is all a marshal stage 16:40:56 that is what lbragstad has done 16:40:57 the blur has been cause, IIUC, because the tokens each had a different core set of data. 16:41:06 and tha tis no longer the case. 16:41:10 so we're fixing it 16:41:12 and it was modified all up and down the stack 16:41:15 good 16:41:21 the tranlation happens in one place now 16:41:34 anywho.. just wanted to advertise it 16:41:34 I think I'm on board. I might be a stickler for some name changes, but the concept is sound 16:41:48 #link https://bugs.launchpad.net/keystone/+bug/1778945 16:41:48 Launchpad bug 1778945 in OpenStack Identity (keystone) "Complexity in token provider APIs" [Medium,In progress] - Assigned to Lance Bragstad (lbragstad) 16:41:58 starts here #link https://review.openstack.org/#/q/topic:bug/1778945+status:open 16:42:14 i'm good to move on - but if you have questions or want to get into the design, just ask me :) 16:42:17 I think a big part of that legacy is from the UUID tokens, which serialized EVERYTHING in a blob 16:42:25 yep, and yay, dead. 16:42:30 this actually ties in with unified delegation 16:42:42 the idea is a that a token points to some delegation agreement 16:43:24 anyway, this is all internal-to-keystone code, so no functional change outside. which is a good starting place for what you want ayoung, lets get this change in and then work on buiding on top of it :) 16:43:24 ayoung: this might help with that, want to pick it up in -keystone afterwords? 16:43:26 the data in the token itself should be limited to identity, time, delegation agreement 16:43:34 WFM 16:43:37 good job 16:43:38 awesome 16:43:41 and i totally think this change is the start of that 16:43:43 #topic unified limits 16:43:53 I +2ed a couple of those this morning 16:43:55 the DB migrations should both be gating... 16:43:57 there is a bunch of things for this 16:44:03 we'll keep on them 16:44:15 wxy|: has been doing a bunch of the server side bits 16:44:24 but the implementation would be good to get eyes on 16:44:31 we also have client support in ksc now 16:44:37 which was release last week i think? 16:44:42 excellent 16:44:47 and we have patches to consume it in osc 16:44:52 #link https://review.openstack.org/#/q/project:openstack/oslo.limit+status:open 16:44:52 need to make sure it gets into shade 16:45:14 sorry 16:45:16 so we can do this from ansible...I've learned the hardway that Ansible is the finish line for affecting change in Keystone, at least for me 16:45:18 these are the osc reviews 16:45:19 # https://review.openstack.org/#/q/status:open+project:openstack/python-openstackclient+branch:master+topic:bp/unified-limits 16:45:25 ayoung: you mean openstacksdk 16:45:35 yep 16:45:37 :) 16:45:53 kmalloc, I'll stop throwing shade...around 16:46:07 i started working on the oslo.limit bits yesterday 16:46:08 #link https://review.openstack.org/#/q/project:openstack/oslo.limit+status:open 16:46:10 * ayoung can hear the groans through your open TCP connections 16:46:15 i still think shade needed a .throw method >.> 16:46:21 curious to get feedback there - i need to incorporate the ksa bits though 16:46:26 * kmalloc cuts down on the snark. 16:46:44 but the basic context maanger approach should be there 16:46:57 just so folks know, oslo.limit core consists of oslo-core and keystone-core 16:47:03 ++ 16:47:20 can we grab oslo.policy, too? Right now, I might be the only core on that 16:47:26 that is also core on Keystone, I mean 16:47:31 wxy|: do you have anything in particular you want to share with respect to unified limits 16:47:42 ayoung: uh i think we do have oslo-policy, don't we? 16:47:44 oslo.policy core has keystone core 16:47:48 https://review.openstack.org/#/admin/groups/556,members 16:47:58 the keystone-core group in included there 16:47:58 Excellent, I missed when that happend. Cool 16:48:19 lbragstad: no, all goes well. Just need eyes on. ;) 16:48:27 wxy|: awesome 16:48:43 at some point in the next couple weeks, we'll need to assess making it stable 16:48:53 since we had a check point for that at the beginning of the release 16:49:13 sounds like we can move on though 16:49:20 #topic policy management tool 16:49:22 ayoung: o/ 16:49:40 So, thanks to kmalloc for getting me started in FLask, I have the kernel of a new project 16:49:59 right now, I can upload policy.json and .yaml files and put them into a database table. 16:50:01 w00t 16:50:16 migrations are Alembic, DB is MYsql 16:50:19 FLask REST 16:50:36 the idea is that it would be used prior to deployment, 16:50:51 gather the policy files from the different locations, upload, munge, regenerate, distribute 16:51:03 tool would take you as far as regenerate 16:51:11 here's where I need help 16:51:25 what are the transforms we need to make it easy to work with policy? 16:52:00 Ideally, we would take a look at people that have modified their policy files in the past, and implement tooling to automate those transitions 16:52:10 for example, what would it mean to "enforce scope" 16:52:22 or to implement a "read only role" 16:52:26 and so on 16:52:43 so, if you have ideas, feel free to drop them in the github tools 16:52:56 issues, pull requests, etc 16:53:02 nice 16:53:06 I want to try and have a workable demo for the Berlin summit 16:53:19 if anyone wants to co-present, please point them my way 16:53:54 I'm thinking about implementing a git-style revision mechanism, so you can roll back to an old version, etc 16:54:14 would love some help on thinking that through, too. 16:54:36 Is anyone here interested in helping? Is there anyone in the larger community that wants to participate? 16:54:58 once we have something that works, I'll propose formally moving it under the identity program in OpenStack 16:55:10 or whatever the proper name is for the group Lance is PTL of 16:55:13 :) 16:55:14 i wouldn't mind keeping tabs on it 16:55:31 I'll try to set up a demo in a week or two 16:55:31 i'll read up on the post you have 16:55:47 if you record something that'd be good, too 16:55:58 Good idea. I'll try to do that 16:56:16 i did that with the system scope stuff, organizing a time is hard to do 16:56:58 alright - final topic 16:57:01 #topic policy meetings 16:57:09 i sent a note about this a while ago 16:57:10 #link http://lists.openstack.org/pipermail/openstack-dev/2018-May/130116.html 16:57:17 responses were light though 16:57:31 i think we're at a point where we can disband the meeting 16:57:59 POlicy is only one cross cutting concern. we should have a policy section of the Keystone meeting, and a policy section of the cross-project meeting 16:58:19 having it in its own meeting gets only the intersection, and we want the union 16:58:35 sure - i think we get pretty good policy coverage in this meeting 16:59:50 if no one has objections - i'll propose the removal 16:59:58 ping me if you would like to discuss! 17:00:16 otherwise - thanks for the time and getting through all the topics 17:00:22 see folks in office hours! 17:00:25 #endmeeting