Wednesday, 2019-02-20

*** markvoelker has quit IRC00:15
*** itlinux has joined #openstack-keystone00:21
*** jamesmcarthur has quit IRC00:22
*** jamesmcarthur has joined #openstack-keystone00:23
imacdonnseems I just needed a new python2-rfc3986 .. OK now00:25
*** jamesmcarthur has quit IRC00:27
*** lbragstad has quit IRC00:34
*** ileixe has joined #openstack-keystone00:53
*** lbragstad has joined #openstack-keystone01:01
*** ChanServ sets mode: +o lbragstad01:01
lbragstadmnaser kmalloc yeah - that is likely bit rot01:02
*** dave-mccowan has joined #openstack-keystone01:08
*** markvoelker has joined #openstack-keystone01:12
*** itlinux has quit IRC01:28
*** Dinesh_Bhor has joined #openstack-keystone01:41
*** markvoelker has quit IRC01:44
*** itlinux has joined #openstack-keystone02:09
*** whoami-rajat has joined #openstack-keystone02:31
*** jamesmcarthur has joined #openstack-keystone02:37
*** markvoelker has joined #openstack-keystone02:41
*** markvoelker has quit IRC03:15
*** jamesmcarthur has quit IRC03:21
*** jamesmcarthur has joined #openstack-keystone03:22
*** jamesmcarthur has quit IRC03:27
*** jamesmcarthur has joined #openstack-keystone03:44
openstackgerritLance Bragstad proposed openstack/keystone master: WIP: Additional work for testing assignment protection
*** itlinux has quit IRC03:45
*** jamesmcarthur has quit IRC03:59
*** jamesmcarthur has joined #openstack-keystone04:00
*** itlinux has joined #openstack-keystone04:02
*** ileixe has quit IRC04:03
*** jamesmcarthur has quit IRC04:05
*** jamesmcarthur has joined #openstack-keystone04:09
*** itlinux has quit IRC04:10
*** markvoelker has joined #openstack-keystone04:12
*** itlinux has joined #openstack-keystone04:13
*** itlinux has quit IRC04:18
*** jamesmcarthur has quit IRC04:22
*** itlinux has joined #openstack-keystone04:23
*** jamesmcarthur has joined #openstack-keystone04:25
*** itlinux has quit IRC04:28
*** jamesmcarthur has quit IRC04:30
*** dave-mccowan has quit IRC04:31
openstackgerritMerged openstack/oslo.limit master: add python 3.7 unit test job
*** itlinux has joined #openstack-keystone04:34
*** lbragstad_ has joined #openstack-keystone04:36
*** ChanServ sets mode: +o lbragstad_04:36
*** lbragstad has quit IRC04:37
*** markvoelker has quit IRC04:44
*** jamesmcarthur has joined #openstack-keystone04:53
*** ileixe has joined #openstack-keystone04:59
*** jamesmcarthur has quit IRC04:59
*** shyamb has joined #openstack-keystone05:07
*** shyamb has quit IRC05:12
*** lbragstad has joined #openstack-keystone05:23
*** ChanServ sets mode: +o lbragstad05:23
*** itlinux_ has joined #openstack-keystone05:25
*** lbragstad_ has quit IRC05:25
*** itlinux has quit IRC05:27
*** lbragstad_ has joined #openstack-keystone05:29
*** ChanServ sets mode: +o lbragstad_05:29
*** lbragstad has quit IRC05:31
*** shyamb has joined #openstack-keystone05:34
*** itlinux has joined #openstack-keystone05:40
*** lbragstad has joined #openstack-keystone05:41
*** ChanServ sets mode: +o lbragstad05:41
*** markvoelker has joined #openstack-keystone05:42
*** itlinux_ has quit IRC05:43
*** lbragstad_ has quit IRC05:43
*** itlinux has quit IRC05:56
*** lbragstad_ has joined #openstack-keystone05:57
*** ChanServ sets mode: +o lbragstad_05:57
*** tkajinam_ has joined #openstack-keystone05:57
*** lbragstad has quit IRC05:58
*** tkajinam has quit IRC05:59
*** shyamb has quit IRC06:05
*** shyamb has joined #openstack-keystone06:06
*** markvoelker has quit IRC06:15
*** lbragstad has joined #openstack-keystone06:20
*** ChanServ sets mode: +o lbragstad06:20
*** lbragstad_ has quit IRC06:22
*** itlinux has joined #openstack-keystone06:23
*** itlinux has quit IRC06:28
*** itlinux has joined #openstack-keystone06:33
*** itlinux has quit IRC06:50
*** shyamb has quit IRC06:58
*** markvoelker has joined #openstack-keystone07:01
*** itlinux has joined #openstack-keystone07:02
*** shyamb has joined #openstack-keystone07:06
*** itlinux has quit IRC07:07
*** jamesmcarthur has joined #openstack-keystone07:18
*** jamesmcarthur has quit IRC07:22
*** itlinux has joined #openstack-keystone07:26
*** itlinux has quit IRC07:36
*** shyamb has quit IRC07:50
*** itlinux has joined #openstack-keystone07:56
*** pcaruana has joined #openstack-keystone07:59
*** itlinux has quit IRC08:01
*** itlinux has joined #openstack-keystone08:03
*** awalende has joined #openstack-keystone08:06
*** tkajinam_ has quit IRC08:14
*** lbragstad has quit IRC08:15
*** itlinux has quit IRC08:16
*** yan0s has joined #openstack-keystone08:21
*** rcernin has quit IRC08:29
*** shyamb has joined #openstack-keystone08:52
*** Dinesh_Bhor has quit IRC09:41
*** shyamb has quit IRC09:46
*** shyamb has joined #openstack-keystone09:53
*** markvoelker has quit IRC10:01
*** markvoelker has joined #openstack-keystone10:02
*** shyamb has quit IRC10:06
*** shyamb has joined #openstack-keystone10:06
*** markvoelker has quit IRC10:06
*** shyamb has quit IRC10:12
*** shyamb has joined #openstack-keystone10:53
*** ileixe has quit IRC10:59
*** markvoelker has joined #openstack-keystone11:03
*** markvoelker has quit IRC11:36
*** shyamb has quit IRC11:44
*** awalende has quit IRC11:49
*** awalende has joined #openstack-keystone11:50
*** awalende has quit IRC11:51
*** awalende has joined #openstack-keystone11:51
*** shyamb has joined #openstack-keystone11:52
*** raildo has joined #openstack-keystone12:30
*** shyamb has quit IRC12:31
*** markvoelker has joined #openstack-keystone12:33
*** awalende has quit IRC12:34
*** awalende has joined #openstack-keystone12:35
*** awalende has quit IRC12:36
*** awalende has joined #openstack-keystone12:36
openstackgerritJose Castro Leon proposed openstack/keystone master: Adds caching of credentials
*** markvoelker has quit IRC13:05
*** jamesmcarthur has joined #openstack-keystone13:18
*** jamesmcarthur has quit IRC13:22
*** jamesmcarthur has joined #openstack-keystone13:22
*** jamesmcarthur has quit IRC13:33
*** jdennis has quit IRC13:41
*** spsurya has joined #openstack-keystone13:44
*** jmlowe has quit IRC13:47
yan0susing apache2 mellon plugin for keystone federation13:50
yan0sI'm having trouble to log out13:51
*** jamesmcarthur has joined #openstack-keystone13:51
*** jamesmcarthur has quit IRC13:51
*** jamesmcarthur has joined #openstack-keystone13:52
yan0sactually I'm logging out from OpenStack dashboard but then on next login I'm not being asked to enter credentials from the IDP13:52
*** vishakha has joined #openstack-keystone13:55
*** markvoelker has joined #openstack-keystone14:02
*** jdennis has joined #openstack-keystone14:11
*** dave-mccowan has joined #openstack-keystone14:15
*** erus has joined #openstack-keystone14:28
*** jmlowe has joined #openstack-keystone14:30
*** lbragstad has joined #openstack-keystone14:32
*** ChanServ sets mode: +o lbragstad14:32
*** awalende has quit IRC14:35
*** markvoelker has quit IRC14:36
*** awalende has joined #openstack-keystone14:36
*** lbragstad_ has joined #openstack-keystone14:40
*** ChanServ sets mode: +o lbragstad_14:40
*** awalende has quit IRC14:41
*** lbragstad has quit IRC14:41
*** erus has quit IRC14:50
*** jamesmcarthur has quit IRC14:53
*** lbragstad has joined #openstack-keystone14:53
*** ChanServ sets mode: +o lbragstad14:53
*** erus has joined #openstack-keystone14:53
*** erus has quit IRC14:54
*** lbragstad_ has quit IRC14:54
*** lbragstad_ has joined #openstack-keystone14:56
*** ChanServ sets mode: +o lbragstad_14:56
*** lbragstad has quit IRC14:58
*** lbragstad_ has quit IRC14:58
*** lbragstad has joined #openstack-keystone15:01
*** ChanServ sets mode: +o lbragstad15:01
*** lbragstad_ has joined #openstack-keystone15:12
*** ChanServ sets mode: +o lbragstad_15:12
*** lbragstad has quit IRC15:13
*** lbragstad_ is now known as lbragstad15:18
*** markvoelker has joined #openstack-keystone15:33
*** markvoelker has quit IRC16:06
openstackgerritLance Bragstad proposed openstack/keystone master: Remove domain policies from policy.v3cloudsample.json
*** yan0s has quit IRC16:34
lbragstadkmalloc curious if you'd be able to revisit
kmalloclbragstad: it just needs to have an updated default policy string16:36
kmalloclbragstad: so folks can use it without custom policy.16:36
kmalloclbragstad: the concern with what gyee proposed first was adding in all the system reader stuff.16:37
kmallocso, this is fine, just needs to cover credential ownership (admin_or_owner) not just ADMIN_RULE16:37
lbragstadi don't think the bug is that the default needs to change16:41
lbragstadthat API has always been rule:admin_required16:41
lbragstadat least until merged16:43
*** gyee has joined #openstack-keystone16:45
lbragstadi think gyee just wants to make sure they can use policies like what was in policy.v3cloudsample.json16:47
gyeelbragstad, I was taking a day off yesterday, looking at the review comments now16:47
gyeethanks for the review btw16:47
lbragstadthanks for the fix16:47
kmalloclbragstad: ah, i was looking at the comparison to master vs rocky16:48
gyeelbragstad, I was thinking along the line as you, that we don't want to change the default policies. Just want to make sure we don't break customize policies.16:49
kmallocand figured we wanted self-service16:49
kmallocthen it's fine as is.16:49
gyeelet me fix up the grammar16:49
kmallocit's fine as is16:49
kmallocdon't respin.16:49
gyeekmalloc, thanks!16:49
kmallocunles there is a bigger need.16:49
lbragstadi had to step through it a bit to make sure it all made sense16:49
kmalloclbragstad: yeah the code is fine as is. it was a concern about defaults16:50
lbragstadkicked it through16:50
gyeelbragstad, I've also started working on, didn't realized it got assigned to someone else16:55
openstackLaunchpad bug 1813057 in OpenStack Identity (keystone) "The tokenless/x509 authentication documentation is opaque" [Medium,Triaged] - Assigned to Adam Heczko (aheczko-mirantis)16:55
gyeedon't want to duplicate the work if Adam had put some significant work in already16:56
lbragstadi'm not sure if Adam is on IRC right now - but if I see him come through I can ask how that is going16:59
lbragstadi don't see anything in review, yet16:59
gyeek, please let me know. I can also help review the changes whenever Adam have something ready.17:01
lbragstadcool - that sounds good17:01
*** markvoelker has joined #openstack-keystone17:03
*** pcaruana has quit IRC17:13
openstackgerritLance Bragstad proposed openstack/keystone master: WIP: Additional work for testing assignment protection
lbragstadvishakha ^ i added a few more tests and filled in some of the others17:16
lbragstadbut - the ones that i added are specific to the system reader role17:17
lbragstadi'll work on some other testing patches that test system member, system admin, domain reader, domain member, domain admin, etc...17:17
gagehugosummit schedule is out17:23
* kmalloc nods.17:29
kmalloclooks like ayoung's and my talk was accepted17:29
ayoungboth of them!17:29
ayounglbragstad, we gonna be talking about JWT!17:30
lbragstadthat's the word on the grapevine17:30
*** markvoelker has quit IRC17:36
* lbragstad slowly turns up CCR on the radio17:39
mnaseri know this is a bit silly but is there a way of disabling the in-process token cache deprecation message when running tests in ksm?17:39
mnaserlet me correct myself, running tests against an app that uses ksm17:39
lbragstadnot that i'm aware of?17:40
vishakhalbragstad: thanks. I will check in the morning.17:41
mnaserokay, i guess ill live with the messages17:41
kmalloci thought there was a deprecation warning suppression in oslo_log17:42
lbragstadi originally thought log level might do it - but probably not for deprecation messages17:42
kmallocmnaser: for testing (like unit test), I'd turn off in-process caching.17:42
kmallocmnaser: for staging/load testing/etc, I'd stand up a real cache.17:42
kmallocmnaser: in-process cache will totally bite you if you have more than one process running.17:43
mnaserkmalloc: oh interesting, so there's an option to turn off caching for ksm i think that's where im missing17:43
kmalloctokens may validate / not validate / time out oddly between requests.17:43
mnaseryeah of course, this is just for functional tests against a flask app17:43
kmallocmnaser: yeah totally should be. let me help find it.17:43
kmallocwow.. that is a gap in the code17:44
kmallocthere should be a way to disable caching in ksm.17:44
ayoungknikolla, lbragstad, cmurphy BTW, I'm totally planning on hijacking you for the policy lab, if you are available17:44
kmallocmnaser: i guess there is no way.17:44
mnaserkmalloc: yeah the way i saw it, that warning is pushed if memcached_servers == []17:45
kmallocmnaser: unless you replace the driver with a NOOP driver (Another thing we should support)17:45
* kmalloc eyerolls.17:45
mnaserit's okay, it's just noise D:17:45
kmallocksm caching has long been on my todo list to fix17:45
kmallocit's kindof bad.17:46
mnaseri guess we can kill that there17:46
mnaseror something17:46
lbragstadayoung what day?17:48
*** jaosorior has quit IRC17:49
*** jaosorior has joined #openstack-keystone17:49
ayounglbragstad, ,
ayounglbragstad, is the schedule set for those yet?17:49
lbragstad^ that looks like it17:51
knikollaayoung: nice!17:51
lbragstad3:50 - 5:20 on Monday17:51
ayoungAnd our other talk is on Wednesday.17:52
kmallocmnaser: that is the swift-pass-a-cache in. I guess you could pass a no-op cache into KSM.17:53
kmallocmnaser: and it would suppress the logs17:53
kmallocmnaser:  i haven't looked into how that works in a long time.17:53
mnaserkmalloc: ill have to find some time to dig into it but don't have time to look into those warnings now :)17:54
kmallocmnaser: ++17:55
lbragstadit never fails cmurphy17:59
lbragstadwe always seem to have conflicting talks17:59
*** markvoelker has joined #openstack-keystone18:33
melwittlbragstad: re: your reply to my ML thread about system-scope tokens. how does a user obtain a system-scope token if they have role:Operator, for example? I'm trying to understand how we could approach the problem with scopes18:35
melwittcurrently, the only thing I understand is if we wanted to remove the hard-coded db check, we'd have to go through all our server APIs and add the server project/user as a target for the policy checks, to get policy enforcement. and I don't understand how to use scopes instead of that18:36
lbragstadmelwitt o/18:36
lbragstadlet me grab an example18:36
lbragstadsystem role assignments exist just list role assignments on a domain or a project (it's just a special entity we can use to protect system-specific APIs)18:38
lbragstadbut - a user can ask for a system-scoped token like this
lbragstadinstead of asking for a project or domain in the scope of their request, they can ask for "system"18:38
melwittI see. so they (the user) would have to know to do that before calling nova18:39
lbragstadthis is what the response would look like18:39
melwittour server APIs default to project scope but we want it to be that users can go across projects if they have a certain role18:39
lbragstadsure - that makes sense18:40
lbragstadwhich is Operator?18:40
melwittwould that mean they should all be declared as project and system? (I'm looking at this patch of yours
melwittin this example, yes. and it's configured in the policy.json18:40
melwittby the deployer18:40
lbragstadok - imo that would be a system specific API18:40
lbragstadand a project specific API18:41
lbragstadlet me grab another example18:41
melwittok, so all of our server APIs should be declared as system? I wasn't sure that sounded right18:41
lbragstadwell - if I have a role on a project, i should be able to call server APIs, right?18:42
lbragstadbut I should only see instances within that project?18:42
lbragstadbut - if i'm a system operator or admin, i should be able to call GET /v2/servers and get all servers in the deployment18:43
lbragstad(for the sake of the example)18:43
lbragstadin that case - scope_types = ['system', 'project']18:43
lbragstadhere is an example of how we do something very similar in keystone -
lbragstadbut - the idea is that you specify the types of tokens you expect to be called with that API using scope_types18:44
melwittI see18:44
lbragstadthen you can craft a default policy that defines who is supposed to see what with each scope18:45
lbragstadso - for the server example18:45
lbragstada check string like 'role:reader and system_scope:all' would make it so anyone with a 'reader' role on the system could list all instances across projects18:45
lbragstadand - if i remember correctly18:47
lbragstadnova subclasses oslo_contect for RequestContext objects18:47
lbragstadkeystone will validate tokens in middleware and set header values that oslo.context will check when loading from a request environment18:48
lbragstadtl;dr - oslo.context will set scope attributes automatically18:48
lbragstadso nova could do something like `if nova_context_obj.system_scoped: # get entire instance list and apply filtering if present`18:49
melwittok. I think I'm missing, how are policy rules related to token scopes? or is it that you can either scope a token or configure it via policy?18:49
lbragstadpolicy rules are what deployers can configure - but we want to try and pull all the scope information away from them18:50
melwitt*configure API behavior via policy18:50
lbragstadinstead - we want to move towards a pattern where scope is defined in the service and not a configurable file18:50
lbragstadso - token scopes go hand-in-hand with policy checks18:51
melwittand then train the user to get the right scope token for what they want18:51
melwitthrm.. the last one is going to be hard I think18:51
melwittfor us18:51
lbragstadgiven the historical mess this has been - the migration will slow18:51
lbragstadremember - oslo.policy won't enforce scope checking by default18:52
lbragstadbut operators can opt into that if they have everything setup and once the services have code to support the various scopes18:52
melwittyeah, makes sense. especially in our case where all server APIs will be ['project', 'system'] it would have no way of knowing what to do depending on scope18:53
melwittI guess it wouldn't even if it were only project either. ignore me18:53
lbragstadin an ideal world18:53
lbragstadif nova sees the request is being made with a project-scoped token, because it pulls that information off the context object, it should make sure the request contains only information relevant for that project18:54
lbragstad(e.g., project A and not project B)18:54
lbragstadbut if the request is being made with a system-scoped token, it can query for all instances and apply filters (if the request was made with one)18:55
melwittso today, for us, all tokens are project-scoped right? legacy way18:56
lbragstadessentially, yes18:56
lbragstadwe support domain-scoped tokens, but they are really only used in keystone18:56
melwittand what deployers are doing is adding role:Operator to relevant projects that they want to be able to list instances across projects, etc18:56
melwittI see, ok18:56
lbragstadright - so the system construct should make having a super special is-admin project irrelevant18:57
lbragstadand hopefully less overloaded18:57
lbragstadit should also make things easier for operators to setup18:58
lbragstadif they don't have to write a custom policy rule for role:operator against a single project to elevate privileges18:58
melwittyeah, ok. thanks, I think I understand now about the scopes and how it's a fundamental change we want to get to, but can't be added as the immediate, only way to do it. should be added as an opt-in18:58
lbragstadright - the migration is going to be a total pain18:59
lbragstadit's going to require services to support different scopes effectively18:59
lbragstadand it'll also require operators to go through their deployment and audit their assignment tables18:59
melwittand now I understand on the spec mriedem linked in the thread, if we fix the role:Operator problem using the scopes, that could motivate users to go toward to scoped tokens18:59
lbragstadyes - exactly18:59
melwittyay, I finally get it \o/19:00
melwittthank you19:00
lbragstadwith the huge payoff being - i can have admin on Project A and i don't see anything in Project B19:00
lbragstadideally - we shouldn't have to check for "does this user have the admin role"?19:01
lbragstadinstead - we have all the underlying plumbing in oslo.policy, oslo.context, keystonemiddleware, and keystone to say "did this user call this API with the right scope and the right role?"19:01
melwittand the role is checked by keystone when the user asks for a token, right?19:02
lbragstadthe role information is relaying in the token body itself19:02
lbragstadso when keystonemiddleware validates the token - it'll set that information19:03
melwittok, and then nova checks the role19:03
lbragstadwhich eventually makes it into the request context object from oslo.context19:03
lbragstad(you should be able to do something like oslo_context.RequestContext.roles)19:03
lbragstadbut - even still, that might not be necessary for nova to check19:03
lbragstadif nova has a context object, you can literally just pass that to oslo.policy during for authorize() or enforce() call and it will know what to do with it19:04
melwitthm, ok. I think that's where I'm a bit fuzzy19:04
lbragstadwhich part is fuzzy?19:05
melwittbut the idea is that if someone has role:Operator and system scope token, they are allowed to list all instances, for example19:05
melwittand not all with system scope token are allowed. that makes sense. I was initially confused about how to use the roles in the existence of token scopes19:05
lbragstadoh - sure19:06
lbragstadi suppose in the past roles have been a pretty 2D thing...19:06
*** markvoelker has quit IRC19:06
lbragstadbecause we didn't have something to denote if the context should be elevated19:06
lbragstadso - we had to make custom roles called Operator for people managing the deployment19:07
lbragstadone way i think about it is that we're no longer just checking if a user has a role in order to do something19:07
lbragstadwe're making sure they have the role and they have it on the thing (e.g., project, domain, system) we expect them to have it on19:08
lbragstadwhich adds complexity, but allows us to reuse things like the admin role so you can give admin to customers on a project and not risk them being able to list hypervisors19:09
lbragstador delete instances in other projects19:11
lbragstadi have a post-it note to dig into the nova tests today, but i was curious if nova has API protection tests that ensure certain users can or can't do specific things?19:16
*** marst has joined #openstack-keystone19:17
lbragstadayoung you'll need to modify the description of the JWT talk19:18
*** jaosorior has quit IRC19:18
lbragstadwe're not talking about Java19:19
*** jmlowe has quit IRC19:19
melwittlbragstad: I'm not sure. I don't think so. lemme take a quick look19:20
lbragstadi imagine something like that would be pretty useful to have in place before starting a refactor to pull all the policy check stuff out of the database19:20
lbragstads/database/database layer/19:21
lbragstadi proposed a few test cases to cinder when we were in denver19:22
lbragstadthat showed how services could add testing like that without relying on keystone19:23
lbragstadwe also have - which describes how all that works19:24
lbragstadwithout nova having to mock keystone tokens or anything like that19:25
lbragstadbut still gives you a level of protection testing with different user personas19:25
melwittyeah, looks like we have no tests, being that my WIP patch passed nearly everything19:25
lbragstad(making sure project users can't delete instance in another project)19:25
melwittthese tempest tests are the only thing that failed
melwittand when you said earlier "make sure certain users can't do specific things" you mean, if using default policy, right? because how it behaves will depend on policy used19:26
lbragstadyes - exaclty19:27
lbragstadi think we should be testing the default policy provided19:27
melwittok, yeah, testing with default policy will be big but doable. testing with other policies would be a lot bigger19:27
melwittin addition to19:28
lbragstadi think it's reasonable to say if an operator roll their own crazy policy, its on them to make sure it does what it should19:28
*** jaosorior has joined #openstack-keystone19:28
* melwitt nods19:28
lbragstadbut - i think patrole (a tempest plugin) attempts to make that easier19:28
ayounglbragstad, I know, I need to see if I can still edit it19:28
lbragstadmelwitt at the same time - all of this work, while slow moving, i would image makes custom policy less painful and less needed19:29
*** jmlowe has joined #openstack-keystone19:30
melwittcool, I'll look at patrole. though I can't help but think the tests should be local functional tests to not burden the gate so much19:31
lbragstadyeah - there is certainly value to having it locally19:31
lbragstadwe're actually re-doing a bunch of our testing in keystone to be more exhaustive on this front19:32
lbragstad <- has some examples19:32
lbragstador anything in
lbragstadcinder wanted to add this because they accidentally changed a default a couple releases ago that introduced a regression, but there weren't any tests for it19:33
ayoungbug 96869619:34
openstackbug 968696 in OpenStack Identity (keystone) ""admin"-ness not properly scoped" [High,In progress] - Assigned to Lance Bragstad (lbragstad)19:34
melwittlbragstad: when you say cinder wanted to add this, you mean they are adding similar to their tree?19:36
lbragstadmelwitt yeah -
lbragstadthey wanted to at least introduce protection testing that tested their default policies19:36
melwittok, thanks. I'm on the lookout for something to copy :P19:37
lbragstad is the keystone specific stuff19:37
lbragstadbut it follows the same high-level pattern as
lbragstad just uses oslo.context so that cinder/nova don't need to rely on keystone for a functional/unit test19:37
lbragstadbut you just need to setup the context object to model a keystone token - which isn't too difficult once you do it a couple times19:38
melwittI like the sound of that19:38
lbragstadmelwitt has a bunch of protection tests19:40
lbragstadyiken was adding a bunch19:40
melwittthanks for all these examples19:41
ayoungI thought we did that in KSM19:41
lbragstadmelwitt i lied - they are here
ayoungAh for the unit tests only, right19:42
* ayoung still reading up19:42
lbragstadayoung yep19:42
lbragstadayoung there is a lot of scroll back :)19:42
ayoungmelwitt, thanks for tackling this19:43
ayoungit really is hard to cover the entire openstack ecosphere on a policy change this huge19:43
lbragstadi feel bad that we have seasoned openstack vets struggling with the concept of scopes19:44
lbragstadi'd really like to find a way to make it easier for people to digest this concept19:44
melwittI haven't tackled anything yet, just asked a question on the ML. but from this convo, I've learned what concrete things to do on the path to fixing the problem I originally asked about19:44
melwittlbragstad: fwiw, I think the biggest gap is connecting the dots from what we have today to where we want to be (scopes). I wasn't understanding how to solve the use case with scopes, especially the part about training users to ask for scoped tokens, I completely missed. but now it seems like a step-by-step, add protection test cases, add scopes to APIs, add scope checks to APIs19:51
lbragstadmelwitt ++19:51
lbragstadmelwitt would have helped, or should we elaborate on that doc even more?19:52
*** vishakha has quit IRC19:54
lbragstador maybe ?19:54
melwittlbragstad: in my case, no. I think I understood scoped tokens (at least at a high level) but didn't know how to get from A to B with what we have in nova. I'd have needed a "dummies" guide for, "how to migrate your project from legacy policy checking to scopes" and spell out the "change user behavior" as needed19:55
lbragstad is a place we could put that19:56
lbragstadit's already written for other openstack developers19:56
melwittit could just be me, so don't assume there's anything missing in docs :)19:57
lbragstadno - we get a lot of questions about it19:57
lbragstadmriedem was asking about it a bunch when he was doing the policy stuff for placement last release19:58
*** jamesmcarthur has joined #openstack-keystone20:02
*** markvoelker has joined #openstack-keystone20:03
*** jmlowe has quit IRC20:06
*** jmlowe has joined #openstack-keystone20:08
* pas-ha lbragstad: have time to chat about fernet keys rotation and distribution? sort of like
openstackLaunchpad bug 1702230 in OpenStack Identity (keystone) "fernet token fails with keystone HA" [Undecided,Invalid] - Assigned to PRAVIN (jarvisopenstack)20:11
pas-haI think I've found how the docs may be improved20:11
pas-hathe most common way AFAIK to distribute keys after rotation is rsync (os-ansible does that anyway) and see what happens during rsync with keys on secondary node
pas-haIf I understand correctly, the at the time marked as NOW! the token issued on primary controller is encrypted with key with content C but if such token arrives at the moment marked as NOW! to the secondary node it has no key with content C in the key repo.20:15
pas-hathis is by default rsync first deletes the missing files and then copies changed and new files i alphanum order20:16
pas-hamarked as NO! on the secondary that is20:16
pas-hathis is what we experience right now on a multi-node - some (only some) operations immediately during rotation + rsync are failing with 401 and the Not a valid fernet token in keystone logs.20:18
*** erus has joined #openstack-keystone20:21
*** markvoelker has quit IRC20:36
*** spsurya has quit IRC20:42
*** raildo has quit IRC20:52
*** jmlowe has quit IRC20:53
*** raildo has joined #openstack-keystone20:54
lbragstadpas-ha checking21:03
*** raildo has quit IRC21:04
*** jamesmcarthur has quit IRC21:05
pas-haso IMO this caveat of key distribution should be mentioned in the docs somehow21:05
*** erus has quit IRC21:05
pas-ha(if I correctly understood the process of course :-) )21:06
*** jdennis has quit IRC21:07
lbragstadpas-ha did you happen to see ?21:08
lbragstador is that the document you're referencing?21:08
pas-hayes, it just vaguely says 'leave it to configmgmt'21:09
pas-hathe distribution that is21:09
pas-hawe have token lifetime 2h, and key rotation every 1h, so according to formula for max_keys, 3 is sufficient and the problems we see should not be due to key being outrotated.21:10
lbragstadthat makes sense21:11
lbragstadbut you're still seeing 401s?21:11
lbragstadand you're sure it's due to over-rotation of the keys?21:11
pas-hano, I am sure it is because the distribution procedure is 'uploading' the keys in the wrong order - effectively in the opposite order key files were changed during rotation21:13
pas-haand distribution is
lbragstadok - so you're using osa for key rotation and distribution21:14
pas-hanot osa :-) but the same rsync command21:14
lbragstadyou index your key name differently21:15
pas-hathat's say after 100 rotations have happened, and letters denote actual content of files (which is all that matters)21:16
lbragstadah - sure21:16
lbragstadi see what you mean21:16
*** jamesmcarthur has joined #openstack-keystone21:17
pas-hafirst half shows how files are changing on prrimary node during rotation, second how they are changed on secondary during distribution21:17
lbragstadso - at what point are you performing the distribution phase?21:18
lbragstadis that marked as NEW on the primary node?21:18
pas-haNEW - from this time AFAIU the tokens issued on primary node will be encrypted with 102 key with content of C, right?21:19
pas-haafter rotation is complete on primary we do `rsync --delete <fernet-key-repo-folder>`  to the secondary node21:20
pas-haand due to default order of operations rsync does there is a moment marked as NO! on the secondary node21:21
*** jdennis has joined #openstack-keystone21:21
pas-haat this tiny window there is no key with content of "C" on secondary node, so if the token issued on primary since NEW is being validated by secondary node exactly at this moment, it fails21:22
lbragstadis this because you're attempting to do distribution super close to rotation?21:23
lbragstadthat's the part i'm missing i think21:24
lbragstadi can't tell if this is an issue with rsync or timing between rotation and distribution21:24
pas-haI don't think so. it is because by default this rsync command first deletes files (not an issue) but then copies new/changed files in the alphanum order, so it will first copy changed staged key 0 and then new primary key.21:26
pas-haif the staged key was named `z` not `0` this would not be an issue :-)21:26
lbragstadoh - since primary keys are always the ones with the highest index, they are the last to be copied over by rsync?21:27
lbragstadthat's kind of a crazy timing/ordering issue21:29
lbragstadyou hit this in an actual deployment?21:29
*** takamatsu_ has joined #openstack-keystone21:31
*** takamatsu has quit IRC21:31
pas-hayes, our internal Queens-based development cloud, running CI on that so there is a constant churn, and time to time we get all of a sudden nova gets 401 from neutron, there are 'Not a valid fernet token' logs in Keystone - and this happens exactly in that couple of seconds the distribution script runs rsync21:31
lbragstadhave you worked around it somehow?21:32
*** markvoelker has joined #openstack-keystone21:33
pas-hahave an idea and will try tomorrow and will keep you posted :-) also need to come up with more cohesive longer description of what's happening21:33
lbragstadpas-ha have you opened a bug against keystone, yet?21:33
pas-hanot really as it is obviously not a bug in keystone itself.21:34
lbragstadok - i'm going to open one just to at least get the ordering written down21:35
pas-haI'd say just mentioning smth like "during key distribution ensure the new primary key is distributed first" would suffice in this fernet FAQ doc21:35
pas-haK, almost midnight here, have a good day y'all :-)21:37
melwittlbragstad: is there a doc about how you (administrator) setup who is allowed to obtain a system-scoped token? I see that you can pass '--os-system-scope all' to OSC to make it request a system-scoped token21:40
lbragstadpas-ha thanks for the information21:40
lbragstadmelwitt yep21:40
lbragstadsystem is just another authorization target, so it's just like a project or a domain21:40
lbragstadyou give users authorization o the system through role assignments21:41
lbragstad`openstack role add --user melwitt --system all admin``21:41
lbragstadfor example ^21:41
melwittok, let me look at the role doc again
lbragstad looks outdated21:42
* lbragstad makes a note21:42
lbragstadmelwitt if you're using devstack - there should be a system-scope entry in clouds.yaml21:45
lbragstadso - you could do something like ``openstack list domains --os-cloud devstack-system-admin``21:45
*** takamatsu_ has quit IRC21:45
lbragstador - ``openstack token issue --os-cloud devstack-system-admin``21:45
lbragstadwhich should give you a system scoped token and you can pass that to nova21:45
melwittlbragstad: I'm actually just trying to write up some high level steps of what operators should do to have non-admin be able to do server actions with a system token in the desired end state21:46
*** takamatsu has joined #openstack-keystone21:46
melwittand I'm thinking, what will an end user have to do, what will the operator have to do, for all of it to work21:46
lbragstadso - is the end users going to be given access to the system?21:46
lbragstadare server actions a system-scoped operation, rather?21:47
melwittwell, the RFE I have is simply, "allow non-admin user to live migrate a server"21:47
melwittand I began with, take out the hard-coded database check that prevents that21:47
melwittthen, I was informed our policy checks don't use the server as a target in many cases, so that would have to be fixed21:47
lbragstadgotcha - doesn't a live migrate require the caller to know which hypervisor to migrate to?21:48
melwittthen, I learned (from you) that system scoped tokens could be a way to solve this21:48
melwittand I'm thinking, what will the end user have to do to live migrate in a world where that's how non-admin are allowed to do it. and I was thinking, the deployer configures their role appropriately, and then they (end user) need to be able to request a system scoped token to call nova API with21:49
lbragstadyou could have a 'live-migrate' role21:50
lbragstadin keystone21:50
melwittnot necessarily, you can specify a host but you don't have to21:50
lbragstadoh - interesting21:50
lbragstadi was always curious about that - i never really knew if you needed to know which host to live migrate to21:50
lbragstadwhich felt weird for project users to know details about the underlying hypervisor21:51
lbragstadso - in that case21:51
lbragstadyou might just be able to use a project-scoped token21:51
lbragstadif you set scope_types on compute:live-migrate to be ['system', 'project']21:51
melwittyeah. it really is a privileged action, it's just that we have customers who don't want to make those users admin. they are "Operator"21:52
lbragstadbut then in nova's API, you can make sure if the API is in fact called with a project-scoped token, it can't specify the specific compute host or whatever21:52
melwittyeah, maybe. that adds some complexity. first step would probably be just make it 'system' only?21:54
lbragstadyeah - that's true21:54
lbragstadand then just have a custom role called "live-migrate" or something21:55
lbragstadthen whoever has that on the system can perform live-migrations but nothing else21:55
lbragstadbut leave it open for system admins to live migrate21:55
*** jamesmcarthur has quit IRC21:55
melwittyeah. in this example, the custom role is "Operator" and it's specified in policy.json for the relevant APIs they want those users to be able to do21:58
melwittI was thinking though, that a system token would be allowed without requiring a custom role, else we would have what we have today21:58
lbragstadyou still have the ability to check a role in the policy check string, through?21:59
*** jamesmcarthur has joined #openstack-keystone21:59
melwittyeah. I guess what I really mean is 'system' would be the default. because yeah, admin should be able to configure how they want21:59
lbragstadso - question22:01
melwittI'm just getting confused trying to write down "you would need to do these things" to make it work in the end state. how to configure keystone role, how to get system token, etc22:01
*** dave-mccowan has quit IRC22:01
lbragstadif i have this Operator role on the system - but i'm not really a system administrator, i should still only be able to live migrate instances within a specific project, right?22:01
melwittnot sure. my initial reaction to that is you would be limited to your project if you had role Operator on the project, no? if you have Operator on the system, you could do for any project (forgive if I'm just not understanding scopes again)22:04
lbragstadnope - you nailed it22:04
lbragstadso - what that tells me is that the live migrate policy check in nova is going to have to support both...22:04
lbragstadotherwise - it's going to be hard for nova to know if an operator system-scoped token really is coming from a system administrator or someone who should be operating on a project22:05
melwittoh, I see your point22:05
lbragstadas a bad actor, i might get this operator role on the system and start live migrating stuff i shouldn't be22:06
*** markvoelker has quit IRC22:07
lbragstadthe policy/api validation layer in nova would essentially check if the live migration request was made with a project-scoped token and if it was, then the instance being live migrated needs to belong to the project from the token's scope22:07
lbragstadotherwise - 40322:07
melwittyeah, ok. I've written this up like this so far:
lbragstadif the live migrate API was called with a system-scoped token, then that check just falls though22:08
melwittso I think I've actually covered that without realizing it22:08
* lbragstad reads22:08
melwittthinking of the system-scoped scenario, I was thinking, how to summarize how the usage will look for the end user and the person setting up keystone roles22:10
melwittto give people an idea of what to expect in the end22:10
lbragstadright - so that would be a system administrator?22:10
lbragstadin that case22:11
lbragstadif the live-migrate functionality is limited to a specific role22:11
lbragstad(e.g., Operator)22:11
lbragstadthen that person is going to have to make sure that policy is overridden in nova's policy file22:11
melwittmaybe. tbh I don't know that much detail about what they're doing, just that they have a role that means the people in it get a subset of the normal admin privilege22:12
lbragstadand the right project users have that role on projects they want to live migrate instances in22:12
lbragstadgot it22:13
melwittyeah, today they have role on projects set up that way and role allowed for the live migrate API22:14
melwittand want to allow it across projects22:14
lbragstadbut that gives users to only live migrate within their project - that's what they want?22:14
melwittand I was thinking, if we add token scoping, we could allow any system scoped token to live migrate across projects. then all they have to do is configure roles the right way and then ask for a system token before they call live migrate22:15
lbragstadideally - i think, with the right steps and coordination, this should work all out of the box22:15
melwittthey want to live migrate different projects, not only within their project22:16
lbragstadok - so that's system specific22:16
melwittthe only thing preventing what they want from working is our hard-coded database layer project_only=True check22:16
lbragstadoh - ah22:16
lbragstadsorry - i was getting ahead of myself there22:17
melwittbut it's also not as easy as removing it, because a lot of our correct "admin_or_owner" behavior is coming from that project_only=True. so we'd need to patch those holes22:17
lbragstadi completely agree with #1 in your writeup22:17
melwittand I'm trying to get it straight in my head how to patch it with the way forward that we want (token scope) rather than the legacy way (add policy targets of server project/user)22:19
lbragstad#2 could include several substeps i think22:19
melwittyeah, it definitely could22:20
lbragstadbut once you have protection testing that ensures different users assert different behavior, its safe to start setting the scope_types variable on existing policies22:20
lbragstadadditionally, you can evolve the check strings to be like
lbragstadat that point - customers can technically use the functionality in the way you're asking22:23
lbragstadwhich goes hand-in-hand with #3 i think22:23
melwittyeah, I think I'm following you22:24
lbragstadbut depending on how #2 and #3 are done in nova, your customers might not even need to make any policy or role changes22:24
lbragstadbecause we offer an `admin`, `member`, and `reader` role in keystone22:24
melwittthat's what I meant in #3 without yet knowing the details of how you implement it22:24
melwitt(I think)22:25
lbragstadif nova consumes scopes properly, and assumes those roles exist, then nova could offer this functionality to users with the `admin` role on a project by default22:25
melwitthm, I thought if they had the admin role this would have worked for them already22:26
lbragstadwell - today yeah22:27
lbragstadbecause of the hardcoded admin check22:27
lbragstadbut if we have scopes22:27
*** rcernin has joined #openstack-keystone22:27
lbragstadit would mean users with admin on project A would by default be able to live migrate instances *within* the project22:28
melwittoh, you mean preserve that working. yeah22:28
lbragstadwithout security concerns22:28
melwittyeah, I definitely don't want to break things that currently work when scopes are added22:28
lbragstadbut - users with admin on the system can live migrate whatever they want, right?22:29
lbragstadso - we'd be reusing the admin role across different scopes but without violating security concerns22:29
melwittyeah maybe it's unavoidable to break that, if we want scopes to make sense22:30
lbragstadright - so if a user has been given admin on a project but they are a customer and you trust them to not break anything, you'll need to have the admin functionality for project users to live migrate in order for them to not be broken22:31
*** whoami-rajat has quit IRC22:31
lbragstadbut if we make live-migrate a system-only API, they'll be broken unless you give them access to the system22:32
melwittyeah... will have to figure out what to do there22:33
lbragstadthat's where the audit comes into place22:33
* melwitt nods22:34
lbragstadi had to write all this down22:34
lbragstadfor the different keystone APIs that broke this22:34
openstackLaunchpad bug 1748027 in OpenStack Identity (keystone) "The v3 users API should account for different scopes" [High,In progress] - Assigned to Lance Bragstad (lbragstad)22:34
lbragstadwe broke the work up by resource/api22:37
melwittyeah, this is where role:admin gets to do everything across projects
lbragstadyeah - pretty much22:38
lbragstadit would be nice to work towards making a lot of the openstack services more "self-service"22:38
lbragstadyour live-migrate case reminds me of that22:39
*** gagehugo has quit IRC22:41
lbragstadif we start breaking the role:admin check into (role:admin and system_scope:all) or (role:admin and then we can start offering more functionality to end users eventually22:41
melwittin that bug, was it that you broke users and are fixing it now? or that you provided backward compat with existing usage and are working to phase it out and facilitate user migration?22:41
lbragstadwe're phasing it out22:41
lbragstadthis is where the series started -
lbragstadeach patches adds functionality or tests for a role/scope permutation22:42
lbragstadbut oslo.policy has plumbing to deprecate policies, which allows for easier upgrades22:43
lbragstadso - operators will be able to keep things working while they upgrade, then they're prompted to perform an audit22:43
lbragstadif they want to keep things working the way they do today, they can copy/paste the policy check as an override and maintain it manually22:44
melwittoh, cool22:44
lbragstadotherwise - they need to correct the role assignments in their deployment22:44
lbragstadfor context - the bug is more of an RFE22:46
lbragstadin that a lot of keystone's API (and OpenStack APIs in general) are really tailored to system administrator, but we should be able to delegate a lot of that functionality down to other users22:47
*** jmlowe has joined #openstack-keystone22:47
lbragstadmaking it easier for users to do more things themselves, instead of relying on tickets to administrators to do things for them because you can't trust a user with the keys to your deployment22:48
melwittI see, yeah22:48
*** erus has joined #openstack-keystone22:52
*** tkajinam has joined #openstack-keystone22:55
*** markvoelker has joined #openstack-keystone23:03
*** rcernin has quit IRC23:06
*** rcernin has joined #openstack-keystone23:08
*** imacdonn has quit IRC23:32
*** imacdonn has joined #openstack-keystone23:33
*** markvoelker has quit IRC23:35
*** marst has quit IRC23:41
*** erus has quit IRC23:41
*** erus has joined #openstack-keystone23:42
*** jamesmcarthur has quit IRC23:55
*** jamesmcarthur has joined #openstack-keystone23:56

Generated by 2.15.3 by Marius Gedminas - find it at!