| *** markvoelker has quit IRC | 00:15 | |
| *** itlinux has joined #openstack-keystone | 00:21 | |
| *** jamesmcarthur has quit IRC | 00:22 | |
| *** jamesmcarthur has joined #openstack-keystone | 00:23 | |
| imacdonn | seems I just needed a new python2-rfc3986 .. OK now | 00:25 |
|---|---|---|
| *** jamesmcarthur has quit IRC | 00:27 | |
| *** lbragstad has quit IRC | 00:34 | |
| *** ileixe has joined #openstack-keystone | 00:53 | |
| *** lbragstad has joined #openstack-keystone | 01:01 | |
| *** ChanServ sets mode: +o lbragstad | 01:01 | |
| lbragstad | mnaser kmalloc yeah - that is likely bit rot | 01:02 |
| *** dave-mccowan has joined #openstack-keystone | 01:08 | |
| *** markvoelker has joined #openstack-keystone | 01:12 | |
| *** itlinux has quit IRC | 01:28 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 01:41 | |
| *** markvoelker has quit IRC | 01:44 | |
| *** itlinux has joined #openstack-keystone | 02:09 | |
| *** whoami-rajat has joined #openstack-keystone | 02:31 | |
| *** jamesmcarthur has joined #openstack-keystone | 02:37 | |
| *** markvoelker has joined #openstack-keystone | 02:41 | |
| *** markvoelker has quit IRC | 03:15 | |
| *** jamesmcarthur has quit IRC | 03:21 | |
| *** jamesmcarthur has joined #openstack-keystone | 03:22 | |
| *** jamesmcarthur has quit IRC | 03:27 | |
| *** jamesmcarthur has joined #openstack-keystone | 03:44 | |
| openstackgerrit | Lance Bragstad proposed openstack/keystone master: WIP: Additional work for testing assignment protection https://review.openstack.org/636825 | 03:44 |
| *** itlinux has quit IRC | 03:45 | |
| *** jamesmcarthur has quit IRC | 03:59 | |
| *** jamesmcarthur has joined #openstack-keystone | 04:00 | |
| *** itlinux has joined #openstack-keystone | 04:02 | |
| *** ileixe has quit IRC | 04:03 | |
| *** jamesmcarthur has quit IRC | 04:05 | |
| *** jamesmcarthur has joined #openstack-keystone | 04:09 | |
| *** itlinux has quit IRC | 04:10 | |
| *** markvoelker has joined #openstack-keystone | 04:12 | |
| *** itlinux has joined #openstack-keystone | 04:13 | |
| *** itlinux has quit IRC | 04:18 | |
| *** jamesmcarthur has quit IRC | 04:22 | |
| *** itlinux has joined #openstack-keystone | 04:23 | |
| *** jamesmcarthur has joined #openstack-keystone | 04:25 | |
| *** itlinux has quit IRC | 04:28 | |
| *** jamesmcarthur has quit IRC | 04:30 | |
| *** dave-mccowan has quit IRC | 04:31 | |
| openstackgerrit | Merged openstack/oslo.limit master: add python 3.7 unit test job https://review.openstack.org/637698 | 04:32 |
| *** itlinux has joined #openstack-keystone | 04:34 | |
| *** lbragstad_ has joined #openstack-keystone | 04:36 | |
| *** ChanServ sets mode: +o lbragstad_ | 04:36 | |
| *** lbragstad has quit IRC | 04:37 | |
| *** markvoelker has quit IRC | 04:44 | |
| *** jamesmcarthur has joined #openstack-keystone | 04:53 | |
| *** ileixe has joined #openstack-keystone | 04:59 | |
| *** jamesmcarthur has quit IRC | 04:59 | |
| *** shyamb has joined #openstack-keystone | 05:07 | |
| *** shyamb has quit IRC | 05:12 | |
| *** lbragstad has joined #openstack-keystone | 05:23 | |
| *** ChanServ sets mode: +o lbragstad | 05:23 | |
| *** itlinux_ has joined #openstack-keystone | 05:25 | |
| *** lbragstad_ has quit IRC | 05:25 | |
| *** itlinux has quit IRC | 05:27 | |
| *** lbragstad_ has joined #openstack-keystone | 05:29 | |
| *** ChanServ sets mode: +o lbragstad_ | 05:29 | |
| *** lbragstad has quit IRC | 05:31 | |
| *** shyamb has joined #openstack-keystone | 05:34 | |
| *** itlinux has joined #openstack-keystone | 05:40 | |
| *** lbragstad has joined #openstack-keystone | 05:41 | |
| *** ChanServ sets mode: +o lbragstad | 05:41 | |
| *** markvoelker has joined #openstack-keystone | 05:42 | |
| *** itlinux_ has quit IRC | 05:43 | |
| *** lbragstad_ has quit IRC | 05:43 | |
| *** itlinux has quit IRC | 05:56 | |
| *** lbragstad_ has joined #openstack-keystone | 05:57 | |
| *** ChanServ sets mode: +o lbragstad_ | 05:57 | |
| *** tkajinam_ has joined #openstack-keystone | 05:57 | |
| *** lbragstad has quit IRC | 05:58 | |
| *** tkajinam has quit IRC | 05:59 | |
| *** shyamb has quit IRC | 06:05 | |
| *** shyamb has joined #openstack-keystone | 06:06 | |
| *** markvoelker has quit IRC | 06:15 | |
| *** lbragstad has joined #openstack-keystone | 06:20 | |
| *** ChanServ sets mode: +o lbragstad | 06:20 | |
| *** lbragstad_ has quit IRC | 06:22 | |
| *** itlinux has joined #openstack-keystone | 06:23 | |
| *** itlinux has quit IRC | 06:28 | |
| *** itlinux has joined #openstack-keystone | 06:33 | |
| *** itlinux has quit IRC | 06:50 | |
| *** shyamb has quit IRC | 06:58 | |
| *** markvoelker has joined #openstack-keystone | 07:01 | |
| *** itlinux has joined #openstack-keystone | 07:02 | |
| *** shyamb has joined #openstack-keystone | 07:06 | |
| *** itlinux has quit IRC | 07:07 | |
| *** jamesmcarthur has joined #openstack-keystone | 07:18 | |
| *** jamesmcarthur has quit IRC | 07:22 | |
| *** itlinux has joined #openstack-keystone | 07:26 | |
| *** itlinux has quit IRC | 07:36 | |
| *** shyamb has quit IRC | 07:50 | |
| *** itlinux has joined #openstack-keystone | 07:56 | |
| *** pcaruana has joined #openstack-keystone | 07:59 | |
| *** itlinux has quit IRC | 08:01 | |
| *** itlinux has joined #openstack-keystone | 08:03 | |
| *** awalende has joined #openstack-keystone | 08:06 | |
| *** tkajinam_ has quit IRC | 08:14 | |
| *** lbragstad has quit IRC | 08:15 | |
| *** itlinux has quit IRC | 08:16 | |
| *** yan0s has joined #openstack-keystone | 08:21 | |
| *** rcernin has quit IRC | 08:29 | |
| *** shyamb has joined #openstack-keystone | 08:52 | |
| *** Dinesh_Bhor has quit IRC | 09:41 | |
| *** shyamb has quit IRC | 09:46 | |
| *** shyamb has joined #openstack-keystone | 09:53 | |
| *** markvoelker has quit IRC | 10:01 | |
| *** markvoelker has joined #openstack-keystone | 10:02 | |
| *** shyamb has quit IRC | 10:06 | |
| *** shyamb has joined #openstack-keystone | 10:06 | |
| *** markvoelker has quit IRC | 10:06 | |
| *** shyamb has quit IRC | 10:12 | |
| *** shyamb has joined #openstack-keystone | 10:53 | |
| *** ileixe has quit IRC | 10:59 | |
| *** markvoelker has joined #openstack-keystone | 11:03 | |
| *** markvoelker has quit IRC | 11:36 | |
| *** shyamb has quit IRC | 11:44 | |
| *** awalende has quit IRC | 11:49 | |
| *** awalende has joined #openstack-keystone | 11:50 | |
| *** awalende has quit IRC | 11:51 | |
| *** awalende has joined #openstack-keystone | 11:51 | |
| *** shyamb has joined #openstack-keystone | 11:52 | |
| *** raildo has joined #openstack-keystone | 12:30 | |
| *** shyamb has quit IRC | 12:31 | |
| *** markvoelker has joined #openstack-keystone | 12:33 | |
| *** awalende has quit IRC | 12:34 | |
| *** awalende has joined #openstack-keystone | 12:35 | |
| *** awalende has quit IRC | 12:36 | |
| *** awalende has joined #openstack-keystone | 12:36 | |
| openstackgerrit | Jose Castro Leon proposed openstack/keystone master: Adds caching of credentials https://review.openstack.org/636645 | 12:55 |
| *** markvoelker has quit IRC | 13:05 | |
| *** jamesmcarthur has joined #openstack-keystone | 13:18 | |
| *** jamesmcarthur has quit IRC | 13:22 | |
| *** jamesmcarthur has joined #openstack-keystone | 13:22 | |
| *** jamesmcarthur has quit IRC | 13:33 | |
| *** jdennis has quit IRC | 13:41 | |
| *** spsurya has joined #openstack-keystone | 13:44 | |
| *** jmlowe has quit IRC | 13:47 | |
| yan0s | using apache2 mellon plugin for keystone federation | 13:50 |
| yan0s | I'm having trouble to log out | 13:51 |
| *** jamesmcarthur has joined #openstack-keystone | 13:51 | |
| *** jamesmcarthur has quit IRC | 13:51 | |
| *** jamesmcarthur has joined #openstack-keystone | 13:52 | |
| yan0s | actually I'm logging out from OpenStack dashboard but then on next login I'm not being asked to enter credentials from the IDP | 13:52 |
| *** vishakha has joined #openstack-keystone | 13:55 | |
| *** markvoelker has joined #openstack-keystone | 14:02 | |
| *** jdennis has joined #openstack-keystone | 14:11 | |
| *** dave-mccowan has joined #openstack-keystone | 14:15 | |
| *** erus has joined #openstack-keystone | 14:28 | |
| erus | o/ | 14:28 |
| *** jmlowe has joined #openstack-keystone | 14:30 | |
| *** lbragstad has joined #openstack-keystone | 14:32 | |
| *** ChanServ sets mode: +o lbragstad | 14:32 | |
| *** awalende has quit IRC | 14:35 | |
| *** markvoelker has quit IRC | 14:36 | |
| *** awalende has joined #openstack-keystone | 14:36 | |
| *** lbragstad_ has joined #openstack-keystone | 14:40 | |
| *** ChanServ sets mode: +o lbragstad_ | 14:40 | |
| *** awalende has quit IRC | 14:41 | |
| *** lbragstad has quit IRC | 14:41 | |
| *** erus has quit IRC | 14:50 | |
| *** jamesmcarthur has quit IRC | 14:53 | |
| *** lbragstad has joined #openstack-keystone | 14:53 | |
| *** ChanServ sets mode: +o lbragstad | 14:53 | |
| *** erus has joined #openstack-keystone | 14:53 | |
| *** erus has quit IRC | 14:54 | |
| *** lbragstad_ has quit IRC | 14:54 | |
| *** lbragstad_ has joined #openstack-keystone | 14:56 | |
| *** ChanServ sets mode: +o lbragstad_ | 14:56 | |
| *** lbragstad has quit IRC | 14:58 | |
| *** lbragstad_ has quit IRC | 14:58 | |
| *** lbragstad has joined #openstack-keystone | 15:01 | |
| *** ChanServ sets mode: +o lbragstad | 15:01 | |
| *** lbragstad_ has joined #openstack-keystone | 15:12 | |
| *** ChanServ sets mode: +o lbragstad_ | 15:12 | |
| *** lbragstad has quit IRC | 15:13 | |
| *** lbragstad_ is now known as lbragstad | 15:18 | |
| *** markvoelker has joined #openstack-keystone | 15:33 | |
| *** markvoelker has quit IRC | 16:06 | |
| openstackgerrit | Lance Bragstad proposed openstack/keystone master: Remove domain policies from policy.v3cloudsample.json https://review.openstack.org/605876 | 16:30 |
| *** yan0s has quit IRC | 16:34 | |
| lbragstad | kmalloc curious if you'd be able to revisit https://review.openstack.org/#/c/637341/ | 16:34 |
| kmalloc | lbragstad: it just needs to have an updated default policy string | 16:36 |
| kmalloc | lbragstad: so folks can use it without custom policy. | 16:36 |
| kmalloc | lbragstad: the concern with what gyee proposed first was adding in all the system reader stuff. | 16:37 |
| kmalloc | so, this is fine, just needs to cover credential ownership (admin_or_owner) not just ADMIN_RULE | 16:37 |
| lbragstad | i don't think the bug is that the default needs to change | 16:41 |
| lbragstad | that API has always been rule:admin_required | 16:41 |
| lbragstad | at least until https://review.openstack.org/#/c/594547/ merged | 16:43 |
| *** gyee has joined #openstack-keystone | 16:45 | |
| lbragstad | i think gyee just wants to make sure they can use policies like what was in policy.v3cloudsample.json | 16:47 |
| gyee | lbragstad, I was taking a day off yesterday, looking at the review comments now | 16:47 |
| gyee | thanks for the review btw | 16:47 |
| lbragstad | anytime | 16:47 |
| lbragstad | thanks for the fix | 16:47 |
| kmalloc | lbragstad: ah, i was looking at the comparison to master vs rocky | 16:48 |
| gyee | lbragstad, I was thinking along the line as you, that we don't want to change the default policies. Just want to make sure we don't break customize policies. | 16:49 |
| kmalloc | and figured we wanted self-service | 16:49 |
| kmalloc | then it's fine as is. | 16:49 |
| lbragstad | cool | 16:49 |
| gyee | let me fix up the grammar | 16:49 |
| kmalloc | it's fine as is | 16:49 |
| kmalloc | +2. | 16:49 |
| kmalloc | don't respin. | 16:49 |
| gyee | kmalloc, thanks! | 16:49 |
| kmalloc | unles there is a bigger need. | 16:49 |
| lbragstad | i had to step through it a bit to make sure it all made sense | 16:49 |
| kmalloc | lbragstad: yeah the code is fine as is. it was a concern about defaults | 16:50 |
| lbragstad | kicked it through | 16:50 |
| gyee | yay! | 16:52 |
| gyee | lbragstad, I've also started working on https://bugs.launchpad.net/keystone/+bug/1813057, didn't realized it got assigned to someone else | 16:55 |
| openstack | Launchpad bug 1813057 in OpenStack Identity (keystone) "The tokenless/x509 authentication documentation is opaque" [Medium,Triaged] - Assigned to Adam Heczko (aheczko-mirantis) | 16:55 |
| gyee | don't want to duplicate the work if Adam had put some significant work in already | 16:56 |
| lbragstad | i'm not sure if Adam is on IRC right now - but if I see him come through I can ask how that is going | 16:59 |
| lbragstad | i don't see anything in review, yet | 16:59 |
| gyee | k, please let me know. I can also help review the changes whenever Adam have something ready. | 17:01 |
| lbragstad | cool - that sounds good | 17:01 |
| *** markvoelker has joined #openstack-keystone | 17:03 | |
| *** pcaruana has quit IRC | 17:13 | |
| openstackgerrit | Lance Bragstad proposed openstack/keystone master: WIP: Additional work for testing assignment protection https://review.openstack.org/636825 | 17:16 |
| lbragstad | vishakha ^ i added a few more tests and filled in some of the others | 17:16 |
| lbragstad | but - the ones that i added are specific to the system reader role | 17:17 |
| lbragstad | i'll work on some other testing patches that test system member, system admin, domain reader, domain member, domain admin, etc... | 17:17 |
| gagehugo | o/ | 17:22 |
| gagehugo | summit schedule is out | 17:23 |
| * kmalloc nods. | 17:29 | |
| kmalloc | looks like ayoung's and my talk was accepted | 17:29 |
| ayoung | w00t! | 17:29 |
| ayoung | both of them! | 17:29 |
| kmalloc | :) | 17:29 |
| ayoung | lbragstad, we gonna be talking about JWT! | 17:30 |
| lbragstad | that's the word on the grapevine | 17:30 |
| *** markvoelker has quit IRC | 17:36 | |
| * lbragstad slowly turns up CCR on the radio | 17:39 | |
| mnaser | i know this is a bit silly but is there a way of disabling the in-process token cache deprecation message when running tests in ksm? | 17:39 |
| mnaser | let me correct myself, running tests against an app that uses ksm | 17:39 |
| lbragstad | not that i'm aware of? | 17:40 |
| vishakha | lbragstad: thanks. I will check in the morning. | 17:41 |
| mnaser | aw | 17:41 |
| mnaser | okay, i guess ill live with the messages | 17:41 |
| kmalloc | i thought there was a deprecation warning suppression in oslo_log | 17:42 |
| lbragstad | i originally thought log level might do it - but probably not for deprecation messages | 17:42 |
| kmalloc | mnaser: for testing (like unit test), I'd turn off in-process caching. | 17:42 |
| kmalloc | mnaser: for staging/load testing/etc, I'd stand up a real cache. | 17:42 |
| kmalloc | mnaser: in-process cache will totally bite you if you have more than one process running. | 17:43 |
| mnaser | kmalloc: oh interesting, so there's an option to turn off caching for ksm i think that's where im missing | 17:43 |
| kmalloc | tokens may validate / not validate / time out oddly between requests. | 17:43 |
| mnaser | yeah of course, this is just for functional tests against a flask app | 17:43 |
| kmalloc | mnaser: yeah totally should be. let me help find it. | 17:43 |
| kmalloc | wow.. that is a gap in the code | 17:44 |
| kmalloc | there should be a way to disable caching in ksm. | 17:44 |
| ayoung | knikolla, lbragstad, cmurphy BTW, I'm totally planning on hijacking you for the policy lab, if you are available | 17:44 |
| kmalloc | mnaser: i guess there is no way. | 17:44 |
| mnaser | kmalloc: yeah the way i saw it, that warning is pushed if memcached_servers == [] | 17:45 |
| mnaser | https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token/_cache.py#L56-L67 | 17:45 |
| kmalloc | mnaser: unless you replace the driver with a NOOP driver (Another thing we should support) | 17:45 |
| * kmalloc eyerolls. | 17:45 | |
| kmalloc | sorry. | 17:45 |
| mnaser | it's okay, it's just noise D: | 17:45 |
| kmalloc | ksm caching has long been on my todo list to fix | 17:45 |
| kmalloc | it's kindof bad. | 17:46 |
| mnaser | https://github.com/openstack/keystonemiddleware/blob/caa899b93d845985277277860f90a40cbf7722a2/keystonemiddleware/auth_token/__init__.py#L945-L952 | 17:46 |
| mnaser | i guess we can kill that there | 17:46 |
| mnaser | or something | 17:46 |
| lbragstad | ayoung what day? | 17:48 |
| *** jaosorior has quit IRC | 17:49 | |
| *** jaosorior has joined #openstack-keystone | 17:49 | |
| ayoung | lbragstad, , https://www.openstack.org/summit/denver-2019/call-for-presentations/preview/23005 | 17:49 |
| ayoung | lbragstad, is the schedule set for those yet? | 17:49 |
| lbragstad | https://www.openstack.org/summit/denver-2019/summit-schedule/#day=2019-04-29 | 17:50 |
| lbragstad | https://www.openstack.org/summit/denver-2019/summit-schedule/events/23005/access-control-policy-hands-on-lab | 17:50 |
| lbragstad | ^ that looks like it | 17:51 |
| knikolla | ayoung: nice! | 17:51 |
| lbragstad | 3:50 - 5:20 on Monday | 17:51 |
| ayoung | And our other talk is on Wednesday. | 17:52 |
| kmalloc | mnaser: that is the swift-pass-a-cache in. I guess you could pass a no-op cache into KSM. | 17:53 |
| kmalloc | mnaser: and it would suppress the logs | 17:53 |
| kmalloc | mnaser: i haven't looked into how that works in a long time. | 17:53 |
| mnaser | kmalloc: ill have to find some time to dig into it but don't have time to look into those warnings now :) | 17:54 |
| kmalloc | mnaser: ++ | 17:55 |
| lbragstad | it never fails cmurphy | 17:59 |
| lbragstad | we always seem to have conflicting talks | 17:59 |
| *** markvoelker has joined #openstack-keystone | 18:33 | |
| melwitt | lbragstad: re: your reply to my ML thread about system-scope tokens. how does a user obtain a system-scope token if they have role:Operator, for example? I'm trying to understand how we could approach the problem with scopes | 18:35 |
| melwitt | currently, the only thing I understand is if we wanted to remove the hard-coded db check, we'd have to go through all our server APIs and add the server project/user as a target for the policy checks, to get policy enforcement. and I don't understand how to use scopes instead of that | 18:36 |
| lbragstad | melwitt o/ | 18:36 |
| lbragstad | let me grab an example | 18:36 |
| lbragstad | system role assignments exist just list role assignments on a domain or a project (it's just a special entity we can use to protect system-specific APIs) | 18:38 |
| lbragstad | but - a user can ask for a system-scoped token like this https://developer.openstack.org/api-ref/identity/v3/index.html?expanded=password-authentication-with-scoped-authorization-detail#system-scoped-example | 18:38 |
| lbragstad | instead of asking for a project or domain in the scope of their request, they can ask for "system" | 18:38 |
| melwitt | I see. so they (the user) would have to know to do that before calling nova | 18:39 |
| lbragstad | right | 18:39 |
| lbragstad | this is what the response would look like | 18:39 |
| lbragstad | https://developer.openstack.org/api-ref/identity/v3/index.html?expanded=password-authentication-with-scoped-authorization-detail#id11 | 18:39 |
| melwitt | our server APIs default to project scope but we want it to be that users can go across projects if they have a certain role | 18:39 |
| lbragstad | sure - that makes sense | 18:40 |
| lbragstad | which is Operator? | 18:40 |
| melwitt | would that mean they should all be declared as project and system? (I'm looking at this patch of yours https://review.openstack.org/525772) | 18:40 |
| melwitt | in this example, yes. and it's configured in the policy.json | 18:40 |
| melwitt | by the deployer | 18:40 |
| lbragstad | ok - imo that would be a system specific API | 18:40 |
| lbragstad | and a project specific API | 18:41 |
| lbragstad | let me grab another example | 18:41 |
| melwitt | ok, so all of our server APIs should be declared as system? I wasn't sure that sounded right | 18:41 |
| lbragstad | well - if I have a role on a project, i should be able to call server APIs, right? | 18:42 |
| lbragstad | but I should only see instances within that project? | 18:42 |
| melwitt | yeah | 18:42 |
| lbragstad | but - if i'm a system operator or admin, i should be able to call GET /v2/servers and get all servers in the deployment | 18:43 |
| lbragstad | (for the sake of the example) | 18:43 |
| melwitt | yes | 18:43 |
| lbragstad | in that case - scope_types = ['system', 'project'] | 18:43 |
| lbragstad | here is an example of how we do something very similar in keystone - https://git.openstack.org/cgit/openstack/keystone/tree/keystone/common/policies/domain.py#n59 | 18:43 |
| melwitt | thanks | 18:44 |
| lbragstad | but - the idea is that you specify the types of tokens you expect to be called with that API using scope_types | 18:44 |
| melwitt | I see | 18:44 |
| lbragstad | then you can craft a default policy that defines who is supposed to see what with each scope | 18:45 |
| lbragstad | so - for the server example | 18:45 |
| lbragstad | a check string like 'role:reader and system_scope:all' would make it so anyone with a 'reader' role on the system could list all instances across projects | 18:45 |
| lbragstad | and - if i remember correctly | 18:47 |
| lbragstad | nova subclasses oslo_contect for RequestContext objects | 18:47 |
| melwitt | yes | 18:47 |
| lbragstad | keystone will validate tokens in middleware and set header values that oslo.context will check when loading from a request environment | 18:48 |
| lbragstad | tl;dr - oslo.context will set scope attributes automatically | 18:48 |
| lbragstad | so nova could do something like `if nova_context_obj.system_scoped: # get entire instance list and apply filtering if present` | 18:49 |
| melwitt | ok. I think I'm missing, how are policy rules related to token scopes? or is it that you can either scope a token or configure it via policy? | 18:49 |
| lbragstad | policy rules are what deployers can configure - but we want to try and pull all the scope information away from them | 18:50 |
| melwitt | *configure API behavior via policy | 18:50 |
| lbragstad | instead - we want to move towards a pattern where scope is defined in the service and not a configurable file | 18:50 |
| lbragstad | so - token scopes go hand-in-hand with policy checks | 18:51 |
| melwitt | and then train the user to get the right scope token for what they want | 18:51 |
| lbragstad | correct | 18:51 |
| melwitt | hrm.. the last one is going to be hard I think | 18:51 |
| melwitt | for us | 18:51 |
| lbragstad | given the historical mess this has been - the migration will slow | 18:51 |
| lbragstad | remember - oslo.policy won't enforce scope checking by default | 18:52 |
| lbragstad | but operators can opt into that if they have everything setup and once the services have code to support the various scopes | 18:52 |
| melwitt | yeah, makes sense. especially in our case where all server APIs will be ['project', 'system'] it would have no way of knowing what to do depending on scope | 18:53 |
| melwitt | I guess it wouldn't even if it were only project either. ignore me | 18:53 |
| lbragstad | in an ideal world | 18:53 |
| lbragstad | if nova sees the request is being made with a project-scoped token, because it pulls that information off the context object, it should make sure the request contains only information relevant for that project | 18:54 |
| lbragstad | (e.g., project A and not project B) | 18:54 |
| lbragstad | but if the request is being made with a system-scoped token, it can query for all instances and apply filters (if the request was made with one) | 18:55 |
| melwitt | yeah | 18:55 |
| melwitt | so today, for us, all tokens are project-scoped right? legacy way | 18:56 |
| lbragstad | essentially, yes | 18:56 |
| lbragstad | we support domain-scoped tokens, but they are really only used in keystone | 18:56 |
| melwitt | and what deployers are doing is adding role:Operator to relevant projects that they want to be able to list instances across projects, etc | 18:56 |
| melwitt | I see, ok | 18:56 |
| lbragstad | right - so the system construct should make having a super special is-admin project irrelevant | 18:57 |
| lbragstad | and hopefully less overloaded | 18:57 |
| lbragstad | it should also make things easier for operators to setup | 18:58 |
| lbragstad | if they don't have to write a custom policy rule for role:operator against a single project to elevate privileges | 18:58 |
| melwitt | yeah, ok. thanks, I think I understand now about the scopes and how it's a fundamental change we want to get to, but can't be added as the immediate, only way to do it. should be added as an opt-in | 18:58 |
| lbragstad | right - the migration is going to be a total pain | 18:59 |
| lbragstad | it's going to require services to support different scopes effectively | 18:59 |
| lbragstad | and it'll also require operators to go through their deployment and audit their assignment tables | 18:59 |
| melwitt | and now I understand on the spec mriedem linked in the thread, if we fix the role:Operator problem using the scopes, that could motivate users to go toward to scoped tokens | 18:59 |
| lbragstad | yes - exactly | 18:59 |
| melwitt | yay, I finally get it \o/ | 19:00 |
| melwitt | thank you | 19:00 |
| lbragstad | with the huge payoff being - i can have admin on Project A and i don't see anything in Project B | 19:00 |
| lbragstad | :) | 19:00 |
| melwitt | gotcha | 19:00 |
| lbragstad | ideally - we shouldn't have to check for "does this user have the admin role"? | 19:01 |
| lbragstad | instead - we have all the underlying plumbing in oslo.policy, oslo.context, keystonemiddleware, and keystone to say "did this user call this API with the right scope and the right role?" | 19:01 |
| melwitt | and the role is checked by keystone when the user asks for a token, right? | 19:02 |
| lbragstad | the role information is relaying in the token body itself | 19:02 |
| lbragstad | so when keystonemiddleware validates the token - it'll set that information | 19:03 |
| melwitt | ok, and then nova checks the role | 19:03 |
| lbragstad | which eventually makes it into the request context object from oslo.context | 19:03 |
| lbragstad | (you should be able to do something like oslo_context.RequestContext.roles) | 19:03 |
| lbragstad | but - even still, that might not be necessary for nova to check | 19:03 |
| lbragstad | if nova has a context object, you can literally just pass that to oslo.policy during for authorize() or enforce() call and it will know what to do with it | 19:04 |
| melwitt | hm, ok. I think that's where I'm a bit fuzzy | 19:04 |
| lbragstad | which part is fuzzy? | 19:05 |
| melwitt | but the idea is that if someone has role:Operator and system scope token, they are allowed to list all instances, for example | 19:05 |
| lbragstad | right | 19:05 |
| melwitt | and not all with system scope token are allowed. that makes sense. I was initially confused about how to use the roles in the existence of token scopes | 19:05 |
| lbragstad | oh - sure | 19:06 |
| lbragstad | i suppose in the past roles have been a pretty 2D thing... | 19:06 |
| *** markvoelker has quit IRC | 19:06 | |
| lbragstad | because we didn't have something to denote if the context should be elevated | 19:06 |
| lbragstad | so - we had to make custom roles called Operator for people managing the deployment | 19:07 |
| lbragstad | one way i think about it is that we're no longer just checking if a user has a role in order to do something | 19:07 |
| melwitt | yeah | 19:07 |
| lbragstad | we're making sure they have the role and they have it on the thing (e.g., project, domain, system) we expect them to have it on | 19:08 |
| lbragstad | which adds complexity, but allows us to reuse things like the admin role so you can give admin to customers on a project and not risk them being able to list hypervisors | 19:09 |
| lbragstad | or delete instances in other projects | 19:11 |
| lbragstad | i have a post-it note to dig into the nova tests today, but i was curious if nova has API protection tests that ensure certain users can or can't do specific things? | 19:16 |
| *** marst has joined #openstack-keystone | 19:17 | |
| lbragstad | ayoung you'll need to modify the description of the JWT talk | 19:18 |
| *** jaosorior has quit IRC | 19:18 | |
| lbragstad | we're not talking about Java | 19:19 |
| *** jmlowe has quit IRC | 19:19 | |
| melwitt | lbragstad: I'm not sure. I don't think so. lemme take a quick look | 19:20 |
| lbragstad | i imagine something like that would be pretty useful to have in place before starting a refactor to pull all the policy check stuff out of the database | 19:20 |
| lbragstad | s/database/database layer/ | 19:21 |
| melwitt | aye | 19:22 |
| lbragstad | i proposed a few test cases to cinder when we were in denver | 19:22 |
| lbragstad | that showed how services could add testing like that without relying on keystone | 19:23 |
| lbragstad | https://review.openstack.org/#/c/602489/ | 19:23 |
| lbragstad | we also have https://docs.openstack.org/oslo.policy/latest/user/usage.html#testing-default-policies - which describes how all that works | 19:24 |
| lbragstad | without nova having to mock keystone tokens or anything like that | 19:25 |
| lbragstad | but still gives you a level of protection testing with different user personas | 19:25 |
| melwitt | yeah, looks like we have no tests, being that my WIP patch passed nearly everything | 19:25 |
| lbragstad | (making sure project users can't delete instance in another project) | 19:25 |
| melwitt | these tempest tests are the only thing that failed https://github.com/openstack/tempest/blob/master/tempest/api/compute/servers/test_servers_negative.py#L581 | 19:25 |
| lbragstad | off | 19:26 |
| lbragstad | oof* | 19:26 |
| melwitt | and when you said earlier "make sure certain users can't do specific things" you mean, if using default policy, right? because how it behaves will depend on policy used | 19:26 |
| lbragstad | yes - exaclty | 19:27 |
| lbragstad | i think we should be testing the default policy provided | 19:27 |
| melwitt | ok, yeah, testing with default policy will be big but doable. testing with other policies would be a lot bigger | 19:27 |
| melwitt | in addition to | 19:28 |
| lbragstad | i think it's reasonable to say if an operator roll their own crazy policy, its on them to make sure it does what it should | 19:28 |
| *** jaosorior has joined #openstack-keystone | 19:28 | |
| * melwitt nods | 19:28 | |
| lbragstad | but - i think patrole (a tempest plugin) attempts to make that easier | 19:28 |
| ayoung | lbragstad, I know, I need to see if I can still edit it | 19:28 |
| lbragstad | melwitt at the same time - all of this work, while slow moving, i would image makes custom policy less painful and less needed | 19:29 |
| *** jmlowe has joined #openstack-keystone | 19:30 | |
| lbragstad | imagine* | 19:30 |
| melwitt | cool, I'll look at patrole. though I can't help but think the tests should be local functional tests to not burden the gate so much | 19:31 |
| lbragstad | yeah - there is certainly value to having it locally | 19:31 |
| lbragstad | we're actually re-doing a bunch of our testing in keystone to be more exhaustive on this front | 19:32 |
| lbragstad | https://review.openstack.org/#/q/status:open+project:openstack/keystone+branch:master+topic:implement-default-roles <- has some examples | 19:32 |
| lbragstad | or anything in https://git.openstack.org/cgit/openstack/keystone/tree/keystone/tests/unit/protection/v3 | 19:32 |
| lbragstad | cinder wanted to add this because they accidentally changed a default a couple releases ago that introduced a regression, but there weren't any tests for it | 19:33 |
| ayoung | bug 968696 | 19:34 |
| openstack | bug 968696 in OpenStack Identity (keystone) ""admin"-ness not properly scoped" [High,In progress] https://launchpad.net/bugs/968696 - Assigned to Lance Bragstad (lbragstad) | 19:34 |
| melwitt | lbragstad: when you say cinder wanted to add this, you mean they are adding similar to their tree? | 19:36 |
| lbragstad | melwitt yeah - https://review.openstack.org/#/c/602489/ | 19:36 |
| lbragstad | they wanted to at least introduce protection testing that tested their default policies | 19:36 |
| melwitt | ok, thanks. I'm on the lookout for something to copy :P | 19:37 |
| lbragstad | https://review.openstack.org/#/q/status:open+project:openstack/keystone+branch:master+topic:implement-default-roles is the keystone specific stuff | 19:37 |
| lbragstad | but it follows the same high-level pattern as https://review.openstack.org/#/c/602489/ | 19:37 |
| lbragstad | https://review.openstack.org/#/c/602489/ just uses oslo.context so that cinder/nova don't need to rely on keystone for a functional/unit test | 19:37 |
| lbragstad | but you just need to setup the context object to model a keystone token - which isn't too difficult once you do it a couple times | 19:38 |
| melwitt | I like the sound of that | 19:38 |
| lbragstad | melwitt http://git.openstack.org/cgit/openstack/cinder/tree/cinder/tests/unit/api/v3 has a bunch of protection tests | 19:40 |
| lbragstad | yiken was adding a bunch | 19:40 |
| melwitt | thanks for all these examples | 19:41 |
| ayoung | I thought we did that in KSM | 19:41 |
| lbragstad | melwitt i lied - they are here http://git.openstack.org/cgit/openstack/cinder/tree/cinder/tests/unit/policies | 19:42 |
| ayoung | Ah for the unit tests only, right | 19:42 |
| * ayoung still reading up | 19:42 | |
| lbragstad | ayoung yep | 19:42 |
| lbragstad | ayoung there is a lot of scroll back :) | 19:42 |
| ayoung | melwitt, thanks for tackling this | 19:43 |
| ayoung | it really is hard to cover the entire openstack ecosphere on a policy change this huge | 19:43 |
| lbragstad | i feel bad that we have seasoned openstack vets struggling with the concept of scopes | 19:44 |
| lbragstad | i'd really like to find a way to make it easier for people to digest this concept | 19:44 |
| melwitt | I haven't tackled anything yet, just asked a question on the ML. but from this convo, I've learned what concrete things to do on the path to fixing the problem I originally asked about | 19:44 |
| melwitt | lbragstad: fwiw, I think the biggest gap is connecting the dots from what we have today to where we want to be (scopes). I wasn't understanding how to solve the use case with scopes, especially the part about training users to ask for scoped tokens, I completely missed. but now it seems like a step-by-step, add protection test cases, add scopes to APIs, add scope checks to APIs | 19:51 |
| lbragstad | melwitt ++ | 19:51 |
| lbragstad | melwitt would https://docs.openstack.org/keystone/latest/admin/tokens-overview.html have helped, or should we elaborate on that doc even more? | 19:52 |
| *** vishakha has quit IRC | 19:54 | |
| lbragstad | or maybe http://specs.openstack.org/openstack/keystone-specs/specs/keystone/queens/system-scope.html#problem-description ? | 19:54 |
| melwitt | lbragstad: in my case, no. I think I understood scoped tokens (at least at a high level) but didn't know how to get from A to B with what we have in nova. I'd have needed a "dummies" guide for, "how to migrate your project from legacy policy checking to scopes" and spell out the "change user behavior" as needed | 19:55 |
| lbragstad | aha | 19:55 |
| lbragstad | https://docs.openstack.org/keystone/latest/contributor/services.html#authorization-scopes is a place we could put that | 19:56 |
| lbragstad | it's already written for other openstack developers | 19:56 |
| melwitt | it could just be me, so don't assume there's anything missing in docs :) | 19:57 |
| lbragstad | no - we get a lot of questions about it | 19:57 |
| lbragstad | mriedem was asking about it a bunch when he was doing the policy stuff for placement last release | 19:58 |
| *** jamesmcarthur has joined #openstack-keystone | 20:02 | |
| *** markvoelker has joined #openstack-keystone | 20:03 | |
| *** jmlowe has quit IRC | 20:06 | |
| *** jmlowe has joined #openstack-keystone | 20:08 | |
| * pas-ha lbragstad: have time to chat about fernet keys rotation and distribution? sort of like https://bugs.launchpad.net/keystone/+bug/1702230 | 20:11 | |
| openstack | Launchpad bug 1702230 in OpenStack Identity (keystone) "fernet token fails with keystone HA" [Undecided,Invalid] - Assigned to PRAVIN (jarvisopenstack) | 20:11 |
| pas-ha | I think I've found how the docs may be improved | 20:11 |
| pas-ha | the most common way AFAIK to distribute keys after rotation is rsync (os-ansible does that anyway) and see what happens during rsync with keys on secondary node https://termbin.com/5pa7 | 20:12 |
| pas-ha | If I understand correctly, the at the time marked as NOW! the token issued on primary controller is encrypted with key with content C but if such token arrives at the moment marked as NOW! to the secondary node it has no key with content C in the key repo. | 20:15 |
| pas-ha | this is by default rsync first deletes the missing files and then copies changed and new files i alphanum order | 20:16 |
| pas-ha | marked as NO! on the secondary that is | 20:16 |
| pas-ha | this is what we experience right now on a multi-node - some (only some) operations immediately during rotation + rsync are failing with 401 and the Not a valid fernet token in keystone logs. | 20:18 |
| *** erus has joined #openstack-keystone | 20:21 | |
| erus | :D | 20:27 |
| *** markvoelker has quit IRC | 20:36 | |
| *** spsurya has quit IRC | 20:42 | |
| *** raildo has quit IRC | 20:52 | |
| *** jmlowe has quit IRC | 20:53 | |
| *** raildo has joined #openstack-keystone | 20:54 | |
| lbragstad | pas-ha checking | 21:03 |
| *** raildo has quit IRC | 21:04 | |
| *** jamesmcarthur has quit IRC | 21:05 | |
| pas-ha | so IMO this caveat of key distribution should be mentioned in the docs somehow | 21:05 |
| *** erus has quit IRC | 21:05 | |
| pas-ha | (if I correctly understood the process of course :-) ) | 21:06 |
| *** jdennis has quit IRC | 21:07 | |
| lbragstad | pas-ha did you happen to see https://docs.openstack.org/keystone/latest/admin/fernet-token-faq.html ? | 21:08 |
| lbragstad | or is that the document you're referencing? | 21:08 |
| pas-ha | yes, it just vaguely says 'leave it to configmgmt' | 21:09 |
| pas-ha | the distribution that is | 21:09 |
| lbragstad | yeah | 21:10 |
| pas-ha | we have token lifetime 2h, and key rotation every 1h, so according to formula for max_keys, 3 is sufficient and the problems we see should not be due to key being outrotated. | 21:10 |
| lbragstad | that makes sense | 21:11 |
| lbragstad | but you're still seeing 401s? | 21:11 |
| pas-ha | yes | 21:11 |
| lbragstad | and you're sure it's due to over-rotation of the keys? | 21:11 |
| pas-ha | no, I am sure it is because the distribution procedure is 'uploading' the keys in the wrong order - effectively in the opposite order key files were changed during rotation | 21:13 |
| lbragstad | ah | 21:14 |
| pas-ha | and distribution is https://github.com/openstack/openstack-ansible-os_keystone/blob/4c9dda7dc4e90ade994eb05928a2d636c6747f43/tasks/keystone_fernet_keys_distribute.yml#L20 | 21:14 |
| lbragstad | ok - so you're using osa for key rotation and distribution | 21:14 |
| pas-ha | not osa :-) but the same rsync command | 21:14 |
| lbragstad | oh! | 21:15 |
| lbragstad | you index your key name differently | 21:15 |
| lbragstad | https://termbin.com/5pa7 | 21:15 |
| pas-ha | that's say after 100 rotations have happened, and letters denote actual content of files (which is all that matters) | 21:16 |
| lbragstad | ah - sure | 21:16 |
| lbragstad | i see what you mean | 21:16 |
| *** jamesmcarthur has joined #openstack-keystone | 21:17 | |
| pas-ha | first half shows how files are changing on prrimary node during rotation, second how they are changed on secondary during distribution | 21:17 |
| lbragstad | so - at what point are you performing the distribution phase? | 21:18 |
| lbragstad | is that marked as NEW on the primary node? | 21:18 |
| pas-ha | NEW - from this time AFAIU the tokens issued on primary node will be encrypted with 102 key with content of C, right? | 21:19 |
| pas-ha | after rotation is complete on primary we do `rsync --delete <fernet-key-repo-folder>` to the secondary node | 21:20 |
| pas-ha | and due to default order of operations rsync does there is a moment marked as NO! on the secondary node | 21:21 |
| *** jdennis has joined #openstack-keystone | 21:21 | |
| pas-ha | at this tiny window there is no key with content of "C" on secondary node, so if the token issued on primary since NEW is being validated by secondary node exactly at this moment, it fails | 21:22 |
| lbragstad | is this because you're attempting to do distribution super close to rotation? | 21:23 |
| lbragstad | that's the part i'm missing i think | 21:24 |
| lbragstad | i can't tell if this is an issue with rsync or timing between rotation and distribution | 21:24 |
| pas-ha | I don't think so. it is because by default this rsync command first deletes files (not an issue) but then copies new/changed files in the alphanum order, so it will first copy changed staged key 0 and then new primary key. | 21:26 |
| lbragstad | m | 21:26 |
| pas-ha | if the staged key was named `z` not `0` this would not be an issue :-) | 21:26 |
| lbragstad | oh - since primary keys are always the ones with the highest index, they are the last to be copied over by rsync? | 21:27 |
| pas-ha | yes | 21:27 |
| lbragstad | damn... | 21:28 |
| lbragstad | that's kind of a crazy timing/ordering issue | 21:29 |
| pas-ha | yep | 21:29 |
| lbragstad | you hit this in an actual deployment? | 21:29 |
| *** takamatsu_ has joined #openstack-keystone | 21:31 | |
| *** takamatsu has quit IRC | 21:31 | |
| pas-ha | yes, our internal Queens-based development cloud, running CI on that so there is a constant churn, and time to time we get all of a sudden nova gets 401 from neutron, there are 'Not a valid fernet token' logs in Keystone - and this happens exactly in that couple of seconds the distribution script runs rsync | 21:31 |
| lbragstad | wow | 21:31 |
| lbragstad | have you worked around it somehow? | 21:32 |
| *** markvoelker has joined #openstack-keystone | 21:33 | |
| pas-ha | have an idea and will try tomorrow and will keep you posted :-) also need to come up with more cohesive longer description of what's happening | 21:33 |
| lbragstad | pas-ha have you opened a bug against keystone, yet? | 21:33 |
| pas-ha | not really as it is obviously not a bug in keystone itself. | 21:34 |
| lbragstad | ok - i'm going to open one just to at least get the ordering written down | 21:35 |
| pas-ha | I'd say just mentioning smth like "during key distribution ensure the new primary key is distributed first" would suffice in this fernet FAQ doc | 21:35 |
| pas-ha | K, almost midnight here, have a good day y'all :-) | 21:37 |
| melwitt | lbragstad: is there a doc about how you (administrator) setup who is allowed to obtain a system-scoped token? I see that you can pass '--os-system-scope all' to OSC to make it request a system-scoped token | 21:40 |
| lbragstad | pas-ha thanks for the information | 21:40 |
| lbragstad | melwitt yep | 21:40 |
| lbragstad | system is just another authorization target, so it's just like a project or a domain | 21:40 |
| lbragstad | you give users authorization o the system through role assignments | 21:41 |
| lbragstad | `openstack role add --user melwitt --system all admin`` | 21:41 |
| lbragstad | for example ^ | 21:41 |
| melwitt | ok, let me look at the role doc again https://docs.openstack.org/keystone/latest/admin/cli-manage-projects-users-and-roles.html#assign-a-role | 21:41 |
| lbragstad | https://developer.openstack.org/api-ref/identity/v3/index.html#system-role-assignments | 21:42 |
| lbragstad | https://docs.openstack.org/keystone/latest/admin/cli-manage-projects-users-and-roles.html#assign-a-role looks outdated | 21:42 |
| * lbragstad makes a note | 21:42 | |
| lbragstad | melwitt if you're using devstack - there should be a system-scope entry in clouds.yaml | 21:45 |
| lbragstad | https://review.openstack.org/#/c/629235/2 | 21:45 |
| lbragstad | so - you could do something like ``openstack list domains --os-cloud devstack-system-admin`` | 21:45 |
| *** takamatsu_ has quit IRC | 21:45 | |
| lbragstad | or - ``openstack token issue --os-cloud devstack-system-admin`` | 21:45 |
| lbragstad | which should give you a system scoped token and you can pass that to nova | 21:45 |
| melwitt | lbragstad: I'm actually just trying to write up some high level steps of what operators should do to have non-admin be able to do server actions with a system token in the desired end state | 21:46 |
| *** takamatsu has joined #openstack-keystone | 21:46 | |
| melwitt | and I'm thinking, what will an end user have to do, what will the operator have to do, for all of it to work | 21:46 |
| lbragstad | aha | 21:46 |
| lbragstad | so - is the end users going to be given access to the system? | 21:46 |
| lbragstad | are server actions a system-scoped operation, rather? | 21:47 |
| melwitt | well, the RFE I have is simply, "allow non-admin user to live migrate a server" | 21:47 |
| melwitt | and I began with, take out the hard-coded database check that prevents that | 21:47 |
| melwitt | then, I was informed our policy checks don't use the server as a target in many cases, so that would have to be fixed | 21:47 |
| lbragstad | gotcha - doesn't a live migrate require the caller to know which hypervisor to migrate to? | 21:48 |
| melwitt | then, I learned (from you) that system scoped tokens could be a way to solve this | 21:48 |
| melwitt | and I'm thinking, what will the end user have to do to live migrate in a world where that's how non-admin are allowed to do it. and I was thinking, the deployer configures their role appropriately, and then they (end user) need to be able to request a system scoped token to call nova API with | 21:49 |
| lbragstad | right... | 21:49 |
| lbragstad | you could have a 'live-migrate' role | 21:50 |
| lbragstad | in keystone | 21:50 |
| melwitt | not necessarily, you can specify a host but you don't have to | 21:50 |
| lbragstad | oh - interesting | 21:50 |
| lbragstad | i was always curious about that - i never really knew if you needed to know which host to live migrate to | 21:50 |
| lbragstad | which felt weird for project users to know details about the underlying hypervisor | 21:51 |
| lbragstad | so - in that case | 21:51 |
| lbragstad | you might just be able to use a project-scoped token | 21:51 |
| lbragstad | if you set scope_types on compute:live-migrate to be ['system', 'project'] | 21:51 |
| melwitt | yeah. it really is a privileged action, it's just that we have customers who don't want to make those users admin. they are "Operator" | 21:52 |
| lbragstad | but then in nova's API, you can make sure if the API is in fact called with a project-scoped token, it can't specify the specific compute host or whatever | 21:52 |
| melwitt | yeah, maybe. that adds some complexity. first step would probably be just make it 'system' only? | 21:54 |
| lbragstad | yeah - that's true | 21:54 |
| lbragstad | and then just have a custom role called "live-migrate" or something | 21:55 |
| lbragstad | then whoever has that on the system can perform live-migrations but nothing else | 21:55 |
| lbragstad | but leave it open for system admins to live migrate | 21:55 |
| *** jamesmcarthur has quit IRC | 21:55 | |
| lbragstad | hmm | 21:57 |
| melwitt | yeah. in this example, the custom role is "Operator" and it's specified in policy.json for the relevant APIs they want those users to be able to do | 21:58 |
| melwitt | I was thinking though, that a system token would be allowed without requiring a custom role, else we would have what we have today | 21:58 |
| lbragstad | you still have the ability to check a role in the policy check string, through? | 21:59 |
| lbragstad | though* | 21:59 |
| *** jamesmcarthur has joined #openstack-keystone | 21:59 | |
| melwitt | yeah. I guess what I really mean is 'system' would be the default. because yeah, admin should be able to configure how they want | 21:59 |
| lbragstad | so - question | 22:01 |
| melwitt | I'm just getting confused trying to write down "you would need to do these things" to make it work in the end state. how to configure keystone role, how to get system token, etc | 22:01 |
| *** dave-mccowan has quit IRC | 22:01 | |
| lbragstad | if i have this Operator role on the system - but i'm not really a system administrator, i should still only be able to live migrate instances within a specific project, right? | 22:01 |
| melwitt | not sure. my initial reaction to that is you would be limited to your project if you had role Operator on the project, no? if you have Operator on the system, you could do for any project (forgive if I'm just not understanding scopes again) | 22:04 |
| lbragstad | nope - you nailed it | 22:04 |
| lbragstad | so - what that tells me is that the live migrate policy check in nova is going to have to support both... | 22:04 |
| lbragstad | otherwise - it's going to be hard for nova to know if an operator system-scoped token really is coming from a system administrator or someone who should be operating on a project | 22:05 |
| melwitt | oh, I see your point | 22:05 |
| lbragstad | as a bad actor, i might get this operator role on the system and start live migrating stuff i shouldn't be | 22:06 |
| *** markvoelker has quit IRC | 22:07 | |
| lbragstad | the policy/api validation layer in nova would essentially check if the live migration request was made with a project-scoped token and if it was, then the instance being live migrated needs to belong to the project from the token's scope | 22:07 |
| lbragstad | otherwise - 403 | 22:07 |
| melwitt | yeah, ok. I've written this up like this so far: http://paste.openstack.org/show/745523/ | 22:08 |
| lbragstad | if the live migrate API was called with a system-scoped token, then that check just falls though | 22:08 |
| melwitt | so I think I've actually covered that without realizing it | 22:08 |
| * lbragstad reads | 22:08 | |
| lbragstad | nice | 22:10 |
| melwitt | thinking of the system-scoped scenario, I was thinking, how to summarize how the usage will look for the end user and the person setting up keystone roles | 22:10 |
| melwitt | to give people an idea of what to expect in the end | 22:10 |
| lbragstad | right - so that would be a system administrator? | 22:10 |
| lbragstad | in that case | 22:11 |
| lbragstad | if the live-migrate functionality is limited to a specific role | 22:11 |
| lbragstad | (e.g., Operator) | 22:11 |
| lbragstad | then that person is going to have to make sure that policy is overridden in nova's policy file | 22:11 |
| melwitt | maybe. tbh I don't know that much detail about what they're doing, just that they have a role that means the people in it get a subset of the normal admin privilege | 22:12 |
| lbragstad | and the right project users have that role on projects they want to live migrate instances in | 22:12 |
| lbragstad | got it | 22:13 |
| melwitt | yeah, today they have role on projects set up that way and role allowed for the live migrate API | 22:14 |
| lbragstad | ok | 22:14 |
| melwitt | and want to allow it across projects | 22:14 |
| lbragstad | but that gives users to only live migrate within their project - that's what they want? | 22:14 |
| melwitt | and I was thinking, if we add token scoping, we could allow any system scoped token to live migrate across projects. then all they have to do is configure roles the right way and then ask for a system token before they call live migrate | 22:15 |
| lbragstad | += | 22:15 |
| lbragstad | right | 22:15 |
| lbragstad | ideally - i think, with the right steps and coordination, this should work all out of the box | 22:15 |
| melwitt | they want to live migrate different projects, not only within their project | 22:16 |
| lbragstad | ok - so that's system specific | 22:16 |
| melwitt | the only thing preventing what they want from working is our hard-coded database layer project_only=True check | 22:16 |
| lbragstad | oh - ah | 22:16 |
| lbragstad | sorry - i was getting ahead of myself there | 22:17 |
| melwitt | but it's also not as easy as removing it, because a lot of our correct "admin_or_owner" behavior is coming from that project_only=True. so we'd need to patch those holes | 22:17 |
| lbragstad | right | 22:17 |
| lbragstad | i completely agree with #1 in your writeup | 22:17 |
| melwitt | and I'm trying to get it straight in my head how to patch it with the way forward that we want (token scope) rather than the legacy way (add policy targets of server project/user) | 22:19 |
| lbragstad | yeah | 22:19 |
| lbragstad | #2 could include several substeps i think | 22:19 |
| melwitt | yeah, it definitely could | 22:20 |
| lbragstad | but once you have protection testing that ensures different users assert different behavior, its safe to start setting the scope_types variable on existing policies | 22:20 |
| lbragstad | additionally, you can evolve the check strings to be like http://git.openstack.org/cgit/openstack/keystone/tree/keystone/common/policies/domain.py#n46 | 22:22 |
| lbragstad | at that point - customers can technically use the functionality in the way you're asking | 22:23 |
| lbragstad | which goes hand-in-hand with #3 i think | 22:23 |
| melwitt | yeah, I think I'm following you | 22:24 |
| lbragstad | but depending on how #2 and #3 are done in nova, your customers might not even need to make any policy or role changes | 22:24 |
| lbragstad | because we offer an `admin`, `member`, and `reader` role in keystone | 22:24 |
| melwitt | that's what I meant in #3 without yet knowing the details of how you implement it | 22:24 |
| melwitt | (I think) | 22:25 |
| lbragstad | if nova consumes scopes properly, and assumes those roles exist, then nova could offer this functionality to users with the `admin` role on a project by default | 22:25 |
| melwitt | hm, I thought if they had the admin role this would have worked for them already | 22:26 |
| lbragstad | well - today yeah | 22:27 |
| lbragstad | because of the hardcoded admin check | 22:27 |
| lbragstad | but if we have scopes | 22:27 |
| *** rcernin has joined #openstack-keystone | 22:27 | |
| lbragstad | it would mean users with admin on project A would by default be able to live migrate instances *within* the project | 22:28 |
| melwitt | oh, you mean preserve that working. yeah | 22:28 |
| lbragstad | without security concerns | 22:28 |
| melwitt | yeah, I definitely don't want to break things that currently work when scopes are added | 22:28 |
| lbragstad | but - users with admin on the system can live migrate whatever they want, right? | 22:29 |
| melwitt | yeah | 22:29 |
| lbragstad | so - we'd be reusing the admin role across different scopes but without violating security concerns | 22:29 |
| melwitt | yeah maybe it's unavoidable to break that, if we want scopes to make sense | 22:30 |
| lbragstad | right - so if a user has been given admin on a project but they are a customer and you trust them to not break anything, you'll need to have the admin functionality for project users to live migrate in order for them to not be broken | 22:31 |
| *** whoami-rajat has quit IRC | 22:31 | |
| lbragstad | but if we make live-migrate a system-only API, they'll be broken unless you give them access to the system | 22:32 |
| melwitt | yeah... will have to figure out what to do there | 22:33 |
| lbragstad | that's where the audit comes into place | 22:33 |
| lbragstad | play* | 22:33 |
| * melwitt nods | 22:34 | |
| lbragstad | i had to write all this down | 22:34 |
| lbragstad | for the different keystone APIs that broke this | 22:34 |
| lbragstad | https://bugs.launchpad.net/keystone/+bug/1748027 | 22:34 |
| openstack | Launchpad bug 1748027 in OpenStack Identity (keystone) "The v3 users API should account for different scopes" [High,In progress] - Assigned to Lance Bragstad (lbragstad) | 22:34 |
| lbragstad | we broke the work up by resource/api | 22:37 |
| melwitt | yeah, this is where role:admin gets to do everything across projects https://github.com/openstack/nova/blob/c6218428e9b29a2c52808ec7d27b4b21aadc0299/nova/policies/base.py#L26 | 22:37 |
| lbragstad | yeah - pretty much | 22:38 |
| lbragstad | it would be nice to work towards making a lot of the openstack services more "self-service" | 22:38 |
| lbragstad | your live-migrate case reminds me of that | 22:39 |
| *** gagehugo has quit IRC | 22:41 | |
| lbragstad | if we start breaking the role:admin check into (role:admin and system_scope:all) or (role:admin and project.id:%s) then we can start offering more functionality to end users eventually | 22:41 |
| melwitt | in that bug, was it that you broke users and are fixing it now? or that you provided backward compat with existing usage and are working to phase it out and facilitate user migration? | 22:41 |
| lbragstad | we're phasing it out | 22:41 |
| melwitt | ok | 22:42 |
| lbragstad | this is where the series started - https://review.openstack.org/#/c/605485/18 | 22:42 |
| lbragstad | each patches adds functionality or tests for a role/scope permutation | 22:42 |
| lbragstad | but oslo.policy has plumbing to deprecate policies, which allows for easier upgrades | 22:43 |
| lbragstad | so - operators will be able to keep things working while they upgrade, then they're prompted to perform an audit | 22:43 |
| lbragstad | if they want to keep things working the way they do today, they can copy/paste the policy check as an override and maintain it manually | 22:44 |
| melwitt | oh, cool | 22:44 |
| lbragstad | otherwise - they need to correct the role assignments in their deployment | 22:44 |
| lbragstad | for context - the bug is more of an RFE | 22:46 |
| lbragstad | in that a lot of keystone's API (and OpenStack APIs in general) are really tailored to system administrator, but we should be able to delegate a lot of that functionality down to other users | 22:47 |
| *** jmlowe has joined #openstack-keystone | 22:47 | |
| lbragstad | making it easier for users to do more things themselves, instead of relying on tickets to administrators to do things for them because you can't trust a user with the keys to your deployment | 22:48 |
| melwitt | I see, yeah | 22:48 |
| *** erus has joined #openstack-keystone | 22:52 | |
| *** tkajinam has joined #openstack-keystone | 22:55 | |
| *** markvoelker has joined #openstack-keystone | 23:03 | |
| *** rcernin has quit IRC | 23:06 | |
| *** rcernin has joined #openstack-keystone | 23:08 | |
| *** imacdonn has quit IRC | 23:32 | |
| *** imacdonn has joined #openstack-keystone | 23:33 | |
| *** markvoelker has quit IRC | 23:35 | |
| *** marst has quit IRC | 23:41 | |
| *** erus has quit IRC | 23:41 | |
| *** erus has joined #openstack-keystone | 23:42 | |
| *** jamesmcarthur has quit IRC | 23:55 | |
| *** jamesmcarthur has joined #openstack-keystone | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!