Tuesday, 2018-08-07

openstackgerritLance Bragstad proposed openstack/keystone master: Allow for more robust config checking with keystone-manage  https://review.openstack.org/58930800:02
*** gyee has quit IRC00:06
openstackgerritLance Bragstad proposed openstack/keystone master: Allow for more robust config checking with keystone-manage  https://review.openstack.org/58930800:14
*** zhurong has joined #openstack-keystone00:22
*** harlowja has quit IRC00:26
*** mvkr has quit IRC00:26
*** mvkr has joined #openstack-keystone00:27
*** mvkr has quit IRC00:39
*** mvkr has joined #openstack-keystone00:40
*** mvkr has quit IRC00:50
*** mvkr has joined #openstack-keystone00:51
*** mvkr has quit IRC01:01
*** mvkr has joined #openstack-keystone01:02
openstackgerritMerged openstack/keystone master: Add a release note for bug 1785164  https://review.openstack.org/58924101:11
openstackbug 1785164 in OpenStack Identity (keystone) "Identity API v3: POST method for "Create Limits" is abnormal for a domain-id" [Undecided,Fix released] https://launchpad.net/bugs/1785164 - Assigned to wangxiyuan (wangxiyuan)01:11
openstackgerritMerged openstack/oslo.policy master: import zuul job settings from project-config  https://review.openstack.org/58870301:25
*** zhurong has quit IRC01:26
*** lbragstad has quit IRC02:18
*** jhesketh_ is now known as jhesketh03:14
*** dave-mccowan has quit IRC03:31
openstackgerritwangxiyuan proposed openstack/keystone master: Remove redundant get_project call  https://review.openstack.org/58902703:50
*** spsurya has joined #openstack-keystone03:54
*** sonuk has joined #openstack-keystone04:40
*** sonuk has quit IRC04:56
*** sonuk has joined #openstack-keystone04:59
vishakhawxy-xiyuan , cmurphy : Hi,  Regarding this bug https://bugs.launchpad.net/keystone/+bug/1473292. Can I propose a blueprint for "Implementing trust_flush via keystone-manage". Is this seems valid ? Thanks.05:08
openstackLaunchpad bug 1473292 in OpenStack Identity (keystone) "Cannot delete or show a trust with an expired date" [Wishlist,Triaged] - Assigned to Vishakha Agarwal (vishakha.agarwal)05:08
*** shyambiradar has joined #openstack-keystone05:12
*** jaosorior has quit IRC05:26
*** shyambiradar has quit IRC05:39
*** shyambiradar has joined #openstack-keystone05:39
*** nicolasbock has joined #openstack-keystone05:46
openstackgerritwangqi proposed openstack/oslo.policy master: Remove PyPI downloads  https://review.openstack.org/58936806:00
*** jaosorior has joined #openstack-keystone06:05
openstackgerritwangxiyuan proposed openstack/keystone master: Remove redundant get_project call  https://review.openstack.org/58902706:27
*** hoonetorg has quit IRC06:33
*** pcaruana has joined #openstack-keystone06:36
openstackgerritVishakha Agarwal proposed openstack/keystone master: Implement Trust Flush via keystone-manage.  https://review.openstack.org/58937806:48
*** hoonetorg has joined #openstack-keystone06:49
openstackgerritVishakha Agarwal proposed openstack/keystone master: Implement Trust Flush via keystone-manage.  https://review.openstack.org/58937806:50
cmurphyvishakha: we don't really use blueprints, I would just propose a patch with closes-bug (looks like that's what you already did)06:54
vishakhacmurphy: thanks :)06:55
vishakhacmurphy: https://review.openstack.org/#/c/588211/ need one more +2. Pl review.06:59
*** shyambiradar has quit IRC07:00
*** shyambiradar has joined #openstack-keystone07:00
*** shyambiradar has quit IRC07:13
*** shyambiradar has joined #openstack-keystone07:23
wxy-xiyuanvishakha: Closing the bug is fine. I have to reread the bug context before looking your fix. Thanks for picking it up.07:29
*** shyambiradar has quit IRC07:47
*** jaosorior has quit IRC07:49
*** slunkad has quit IRC07:58
*** jaosorior has joined #openstack-keystone08:02
*** rcernin has quit IRC08:11
*** Emine has joined #openstack-keystone08:16
*** sayalilunkad has joined #openstack-keystone08:31
*** shyambiradar has joined #openstack-keystone08:47
*** jaosorior has quit IRC09:02
*** josecastroleon has joined #openstack-keystone09:14
vishakhawxy-xiyuan: thanx for the reply09:49
*** shyambiradar has quit IRC09:50
openstackgerritMerged openstack/oslo.policy master: Remove PyPI downloads  https://review.openstack.org/58936809:50
vishakhawxy-xiyuan, cmurphy : I am a little confused for this https://review.openstack.org/#/c/588211/. Is this on hold for now?09:50
*** shyambiradar has joined #openstack-keystone09:51
cmurphyvishakha: no, there's nothing preventing someone else from merging it09:53
vishakhaok thanks09:54
*** shyambiradar has quit IRC10:34
*** jaosorior has joined #openstack-keystone10:46
*** shyambiradar has joined #openstack-keystone10:49
*** dave-mccowan has joined #openstack-keystone10:51
*** zeus has quit IRC10:53
*** shyambiradar has quit IRC10:54
*** shyambiradar has joined #openstack-keystone11:41
*** s10 has joined #openstack-keystone11:57
*** jaosorior has quit IRC11:59
*** raildo has joined #openstack-keystone12:02
*** raildo has quit IRC12:03
*** raildo has joined #openstack-keystone12:03
*** edmondsw has joined #openstack-keystone12:04
*** raildo has quit IRC12:04
*** raildo has joined #openstack-keystone12:05
*** raildo has quit IRC12:06
*** raildo has joined #openstack-keystone12:06
*** raildo has quit IRC12:07
*** s10 has quit IRC12:09
*** s10 has joined #openstack-keystone12:10
*** s10 has quit IRC12:10
*** raildo has joined #openstack-keystone12:12
*** raildo has quit IRC12:14
*** raildo has joined #openstack-keystone12:15
*** jaosorior has joined #openstack-keystone12:18
*** jaosorior has quit IRC12:27
*** jaosorior has joined #openstack-keystone12:42
*** glb has quit IRC12:50
*** shyambiradar has quit IRC12:54
*** dklyle has quit IRC12:59
*** bigdogstl has joined #openstack-keystone13:00
*** shyambiradar has joined #openstack-keystone13:01
*** shyambiradar has quit IRC13:10
*** lbragstad has joined #openstack-keystone13:11
*** ChanServ sets mode: +o lbragstad13:11
*** s10 has joined #openstack-keystone13:15
*** bigdogstl has quit IRC13:36
*** _ix has quit IRC14:06
kmalloccmurphy: ++ to figuring out how to manage group names vs IDs.14:11
cmurphykmalloc: any suggestions?14:13
cmurphyhow important is it that the saml assertion use names instead of IDs?14:14
*** jrist has quit IRC14:15
*** nicolasbock has quit IRC14:18
openstackgerritLance Bragstad proposed openstack/keystone master: Allow for more robust config checking with keystone-manage  https://review.openstack.org/58930814:24
*** mbuil has joined #openstack-keystone14:26
*** jrist has joined #openstack-keystone14:27
*** dklyle has joined #openstack-keystone14:33
*** josecastroleon has quit IRC14:35
mbuilhello Keystone community! I am planning to run a proof of concept with the architecture you guys suggested to the edge computing group. As far as I understood, a central openstack will act as a IdP for various openstack deployments with limited resources. Is there any link with useful documentation where I could start? I already have two openstack deployments that can ping each other :P14:35
*** josecastroleon has joined #openstack-keystone14:36
lbragstadmbuil: o/14:42
*** _ix has joined #openstack-keystone14:42
lbragstadare you familiar with federated identity at all?14:42
lbragstador federation in general?14:43
lbragstadif not - https://docs.openstack.org/keystone/latest/advanced-topics/federation/federated_identity.html might be a good place to start14:43
cmurphymbuil: o/ in addition to the official docs i have some notes here that might be useful http://www.gazlene.net/demystifying-keystone-federation.html14:44
mbuillbragstad: I have just watched https://www.youtube.com/watch?v=jwXenfEOSOk and read http://www.gazlene.net/demystifying-keystone-federation.html. I guess Keystone to Keystone is what I am after14:44
lbragstadlol - perfect14:44
lbragstadkmalloc: you want me to base the policy conversion on https://review.openstack.org/#/c/589282/ ?14:46
lbragstadmbuil: yeah - it sounds like k2k has been the popular target for edge discussions14:46
mbuillbragstad, cmurphy: let me read that and try to come back with good questions. I guess I need to figure out how to configure keystone_1 as IdP and keystone_2 as Federated pointing to keystone_1 for IdP14:47
mbuildoes that make sense? Or I am totally confused?14:47
lbragstadright - that's the gist of it14:47
lbragstadyou'll need to setup one keystone deployment to be the identity provider14:48
cmurphythere's also been an idea about using regular federation and then using application credentials on the service providers14:48
lbragstadyeah, that's right14:48
lbragstadmbuil: the other keystone will be the service provider, so you'll need to set it up to trust the identity provider of your deployment (the fact that both deployments are keystone is the keystone-to-keystone part)14:50
lbragstadthere is always the option to setup federation the other identity providers, or anything that speaks SAML (like ADFS)14:51
mbuilcmurphy what does application mean in keystone context?14:52
mbuillbragstad: ok, thanks. I think we will start with Keystone to Keystone as first step14:53
cmurphymbuil: we have a special credential type called 'application credentials' https://docs.openstack.org/keystone/latest/user/application_credentials.html so it means any automated thing but a human could use it too if they wanted14:53
*** josecastroleon has quit IRC14:55
lbragstadcmurphy: can you jog my memory on how we were going to work that into federated use cases?14:55
*** wxy|xiyuan has joined #openstack-keystone14:57
*** hoonetorg has quit IRC14:57
*** josecastroleon has joined #openstack-keystone14:57
cmurphylbragstad: i think the idea is during the times your sites are connected you use your user to create application credentials on the keystones in different sites, so that during the times the sites aren't connected you can use the application credential to log in locally14:57
*** Emine has quit IRC14:58
lbragstadaha - ok, that's right14:58
*** jaosorior has quit IRC15:01
*** lamt has joined #openstack-keystone15:02
mbuilcmurphy: I am confused now. If keystone_2 is federated and uses keystone_1 as IdP, when the connection is broken, how can keystone_2 provide tokens if it cannot validate users?15:03
*** nicolasbock has joined #openstack-keystone15:07
cmurphylbragstad: kmalloc knikolla what was the idea there ^ ? shadow users on keystone_2?15:09
lbragstadi think so15:09
lbragstadmbuil: when a user authenticates using federation, the keystone acting as a service provider creates something called a "shadow user"15:10
lbragstadwhich is stored in the keystone SP database15:10
mbuilaaah I see. Thanks15:11
*** pcaruana has quit IRC15:11
lbragstadthe theory behind it is that if a federated user creates an application credential, the application credential will be associated to the shadow user, meaning keystoen SP won't have to make a round trip to the identity provider15:11
*** s10 has quit IRC15:12
*** s10 has joined #openstack-keystone15:15
knikollareading up15:29
*** gyee has joined #openstack-keystone15:32
kmalloclbragstad: yeah15:33
knikollayeah, with regards to authN that was the plan15:33
lbragstadkmalloc: cool15:33
knikollawith regards to authZ there are two options15:33
knikollause groups and get permissions at authentication time based on the groups you have.15:34
kmalloccmurphy: i defer to knikolla, but i don't like transmitting only group_ids, what do those ids mean... we need some kind of translation15:34
knikollawe discussed about having "refreshable" application credentials which store the permissions at authentication time15:34
kmalloclbragstad: i made some formatting changes in the __init__, etc so that we are less-likely to conflict and can parallelize some of the work.15:35
knikollaor just have a concrete role assignment on the shadow user15:35
kmallocknikolla: i think we wanted both, refreshable app-cred and concrete role assignment15:35
kmalloctwo different use-cases15:35
kmalloclocal-only and possibly disconnected sync case.15:36
knikollayup, that's why i said to options, depending on the use case15:37
knikollai'll start writing a spec for stein15:38
*** s10 has quit IRC15:39
*** s10 has joined #openstack-keystone15:40
kmalloclbragstad: note that some /policy bits are in end-point-policy15:47
kmalloclbragstad: but just extract those parts when you port /policy [i messed up the _path_prefix thing for endpoint policy]15:47
*** ayoung has joined #openstack-keystone16:03
*** wxy|xiyuan has quit IRC16:05
*** spilla has joined #openstack-keystone16:21
kmallocAnd now, for your viewing pleasure...16:28
cmurphyomg she got huge16:29
*** devx has quit IRC16:29
cmurphywhere did the mini floofball go16:29
*** devx has joined #openstack-keystone16:31
*** lbragstad[m] has quit IRC16:32
kmallocyeah, she's gotten big16:34
kmallocshe's like 20lbs now.16:34
kmallocand still not even full size (~4.5mo)16:35
kmallocprob will be growing still for another month or so, then will fill out, we're guessing she's going to be close to 30lbs16:35
kmallocshe's a very silly dog.16:36
*** s10 has quit IRC16:38
openstackgerritMorgan Fainberg proposed openstack/keystone master: Pass path into full_url and base_url  https://review.openstack.org/58954616:43
*** Emine has joined #openstack-keystone16:55
lbragstad#startmeeting keystone-office-hours17:00
openstackMeeting started Tue Aug  7 17:00:17 2018 UTC and is due to finish in 60 minutes.  The chair is lbragstad. Information about MeetBot at http://wiki.debian.org/MeetBot.17:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.17:00
openstackThe meeting name has been set to 'keystone_office_hours'17:00
lbragstadcmurphy: i suppose this is an inconvenient time for you17:00
cmurphynot at all17:01
* lbragstad is all ears17:01
cmurphyso this is the bug that captures it i think https://bugs.launchpad.net/keystone/+bug/147020517:01
openstackLaunchpad bug 1470205 in OpenStack Identity (keystone) "Keystone IdP SAML metadata insufficient for websso flow" [Wishlist,Triaged]17:01
knikollathough we're not in the business of being a webapp17:02
cmurphywhen keystone is set up as a SAML IdP it isn't capable of doing the WebSSO profile, all it does is this very weird IdP-initiated ECP flow which is not a real profile17:02
knikollaand the websso flow kinda requires that.17:02
knikollawe'd need a form to login, and session cookies.17:03
lbragstadso we have gaps in our federated IdP implementation specifically17:03
kmallocso, fwiw, with flask, i'll be moving us towards having an actual non-json page17:03
kmallocif possible17:03
kmallocwhich leads towards possibly being able to render websso things more cleanly17:03
knikollakmalloc: that would be really helpful17:03
kmallocflask opens doors in ways webob was a REAL pain17:03
kmalloci already have some logic for "accept header" variations and rendering different content17:04
kmallocso, we can head down that path17:04
lbragstadwhich i think we needed for json/home17:04
kmallocin flask that was a lot easier to do than in webob17:04
kmallocand once we're 100% on flask-restful, i'll be making a push to go to restplus17:05
kmallocso we get swagger documentation17:05
kmallocif we already have that, we can line up some views for web-rendering that can be webapp/sso flow compatible17:05
kmallocso, think in S release we might be able to head down that path.17:05
knikollasounds reasonable17:05
kmallocdepending on how the rest of flask conversion goes17:06
lbragstadcmurphy: you have colleagues asking for this specifically?17:06
lbragstador customers?17:06
kmalloclbragstad: change of plans, use https://review.openstack.org/#/c/589546/ [if it passes check] as the base for /policy API17:06
lbragstadkmalloc: ack17:06
kmalloclbragstad: i changed the way full_url and base_url work17:06
kmallocso you can pass in a path to them and get /v3 added automatically17:07
cmurphylbragstad: colleages/project managers17:07
cmurphyignoring the SAML part, the other part I have a hard time wrapping my head around is how a not-openstack application would know how to use keystone fernet tokens - i guess it would have to use keystonemiddleware or have some other kind of middleware that knows to redirect users to keystone for auth and requires the x-auth-token header for all requests17:07
cmurphyand then what does it do with information about projects? what would not-openstack do with projects and role assignments17:08
knikollaif we implemented openid connect, we would already be compatible with most apps17:08
lbragstadwe also have a derivative of RBAC that's isn't exaclty an industry standard17:08
knikollathe problem is the RBAC, more or less.17:09
lbragstadi agree17:09
lbragstadi think this same kinda problem came up when we talked about oauth2 support17:09
lbragstadi guess it depends on what's using keystone and what they are trying to protect (without sounds overly vague)17:11
lbragstadhow are the resources grouped?17:11
lbragstaddepending on the answers, could you generalize keystone enough to work for each?17:12
knikollaoauth2.0 has the concept of scopes (think roles). but nothing really project based17:14
knikollai'm also not familiar with other implementations of scoped rbac like in openstack.17:15
lbragstadi'm not aware of any other implementations of scoped-rbac17:15
lbragstadi want to say scoped-rbac is openstack-specific17:16
knikollain that case, does the application care about the project at all?17:16
lbragstadwhich application?17:16
knikollaexternal application that want to integrate with keystone17:16
lbragstadi think the answer to that depends on what the application is responsible for17:16
knikollaif all they care is users and roles (and have no concept of scoped rbac), but just plain rbac17:16
*** orange_julius has joined #openstack-keystone17:16
knikollathat should be doable in a generalizable way17:17
cmurphyknikolla: hmm so maybe there'd be a dummy project and the application only pays attention to the role name?17:17
lbragstadif the application in question is something like nova that provides APIs for managing all sorts of things across different authorization levels, then i think keystone needs to care about scoped-rbac17:17
orange_juliusHey guys. Had a quick question about Keystone + LDAP. I have 3 controllers configured and Keystone seems to be sending an authentication request from Horizon off to LDAP 3 times, assumingly one per controller. This is causing an issue because our lockout policy in LDAP is 3 bad passwords... so if a user enters 1 bad password we get an instant locko17:18
orange_juliusut. Is this expected behaviour? Why would keystone be sending the authentication request 3 times for the same request?17:18
knikollacmurphy: yeah, something like that17:18
lbragstadthat's essentially what we started with, i think17:19
lbragstaddeploy keystone with a single "level" of authorization and only use that layer17:19
cmurphyorange_julius: your haproxy should be configured to forward requests to just one keystone, three keystones making the same request at once would indeed all go to ldap17:20
knikolladepends on how they want to do the integration. if they care about users, use a dummpy project. if they care about projects (as in the webapp provides services to extend an existing openstack deployemnt therefore project isolation matters), we can send the project_id as the username in the assertion.17:20
*** spilla has quit IRC17:20
_ixMaybe this is actually a quick question... if I'm using passwords, is there a reason that I'd specify `v3password` instead of `password`?17:20
cmurphylbragstad: heh yeah i guess this sort of takes us back to pre-multitenancy17:20
_ixI don't know why, but some of the services I'm working on have a v3password specified in the conf files.17:20
cmurphy_ix: in theory they should both be the same, `password` i think either looks at the url or does a discovery step to figure out what version to use17:21
knikollacmurphy: so the question is, in the webapp, do you want to act on behalf of the user (username in assertion = user_id, webapp will have no concept of projects). do you want to act on behalf of a project (username in assertion is project_id, webapp will have no concept of individual users)17:22
_ixAh. So if I'm explicit with the endpoint, I'll be in good shape. Thanks!17:22
cmurphyknikolla: that's a good way to look at it17:23
lbragstadknikolla: in the first example, i'm inclined to say we already support that pretty easily17:23
*** spilla has joined #openstack-keystone17:24
* lbragstad feels like we're coming up with various deployment models17:24
lbragstadprotecting apps the have no concept of scope or individual projects is a pretty low bar since most of that can be done between keystone and ksm17:25
lbragstadif an app starts caring about tenancy, then it becomes a question of where that logic lives, imo17:26
knikollalbragstad: in the case that the app cares about tenancy, that would probably be tied to an attribute/claim of the assertion.17:26
knikollawe could make that configurable17:27
knikollaKeycloak for example allows you to map user attributes -> claims for each specific client which is using it for auth.17:28
lbragstadbut the app being protected needs to understand those claims, right?17:28
knikollaassuming it's already using a standard openid connect / saml idp. it probably already does17:29
*** harlowja has joined #openstack-keystone17:31
knikollawe just need to document the mapping between keystone terminology -> openid connect / saml, and deployment strategies.17:31
knikollalike the two cases i mentioned above.17:32
lbragstadand the resources managed by the webapp?17:32
knikollait's up to the webapp, based on the roles keystone says u have17:33
knikollanot any different than what it is now17:33
lbragstadtoday - that's done by policy17:33
lbragstador our specific policy engine17:33
lbragstadbecause that's where we say "instances are project resources, endpoints are system resources"17:34
knikollayou can flatten that into the webapp checking instances require the project_member role, endpoint require the system_member role.17:34
lbragstadyeah - using a policy file is an implementation detail17:35
lbragstadi guess what i'm seeing is that the resource can make a difference in that mapping depending on the app17:36
knikollalbragstad: you could also say that scope is an implementation detail17:37
knikollauser has access to project which owns resource17:37
knikollauser acts on behalf of project which owns resource.17:38
knikollalike i don't think most openstack project care about users17:38
knikollathey store resources with a project_id in their database17:38
knikollatherefore given (project_id, role) a webapp has enough information to enforce policy17:40
knikollai also remember the early discussions about making system a part of the project tree17:41
*** harlowja has quit IRC17:43
lbragstadwe decided against that because we thought there was a difference between the resources in openstack's case17:44
orange_juliusCould keystone be picking up the authentication requests from RabbitMQ? It doesn't seem like HAProxy is forwarding the requests to all 3 keystone servers17:45
*** markvoelker_ has quit IRC17:45
lbragstadorange_julius: keystone doesn't support authentication via the message bus17:47
orange_juliushmmm ok. Didn't think so but am running out of ideas.17:47
lbragstaddoes the problem go away if you only route requests so a single keystone node?17:48
orange_juliusNot sure yet. We have to schedule an outage for that sort of test.17:51
*** ayoung has quit IRC17:53
lbragstadi suppose you could try and recreate it by making the same request to a single keystone node, too17:57
lbragstadas opposed to taking nodes out of the pool17:57
* lbragstad grabs lunch quick17:57
kmalloclbragstad: our tests are still spewing the "scope check" fail message17:59
kmalloclbragstad: do we have a fix for that?17:59
kmallocthis is a LOT of extra logging that seems like it is not actionable atm.17:59
kmallocbecause this makes me want to push a change to squash all logging from oslo_context18:01
kmallocby default18:01
*** spilla has quit IRC18:15
openstackgerritMorgan Fainberg proposed openstack/keystone master: Allow wrap_member and wrap_collection to specify target  https://review.openstack.org/58928818:16
*** spilla has joined #openstack-keystone18:25
*** fiddletwix has quit IRC18:37
*** fiddletwix has joined #openstack-keystone18:38
*** openstackgerrit has quit IRC18:49
lbragstadkmalloc: i had a wip series of patches for that18:58
lbragstadand https://review.openstack.org/#/c/551411/118:59
*** s10 has joined #openstack-keystone19:17
lbragstadkmalloc: i don't need to depend on https://review.openstack.org/#/c/589288/2 to convert the policy API? just https://review.openstack.org/#/c/589546/1 ?19:28
orange_juliusSo it doesn't look like HAProxy is forwarding the keystone authentication requests to all keystone controllers. I tested with a similiar configuration and saw an HTTP request hit just one HTTP server sitting behind haproxy with similiar configs19:35
lbragstadbut the user is still getting locked out?19:40
*** openstackgerrit has joined #openstack-keystone19:49
openstackgerritLance Bragstad proposed openstack/oslo.policy master: Move _capture_stdout to a common place  https://review.openstack.org/53444019:49
orange_juliusYes the user is still getting locked out19:50
openstackgerritLance Bragstad proposed openstack/keystone master: Update release notes for the rocky release  https://review.openstack.org/58957819:52
kmalloclbragstad: just convert the api19:54
lbragstadok - working on that now19:54
kmalloclbragstad: and you can base on the patch preceeding that one you linked19:55
kmallocThe one that changes base_url and full_url19:55
lbragstadorange_julius: are your users in sql or ldap?19:56
lbragstadorange_julius: you're using the [security_compliance] section of keystone.conf, yeah?19:56
orange_juliusUsers are in active directory. Not using security_compliance. The only lockout policy is defined in active directory19:57
orange_juliusI see two possibilites. Let me know if I am wrong. Either a single keystone instance is sending multiple requests to AD, or all 3 keystone instances are seeing the login attempt and then asking AD for authorization19:58
lbragstadare you able to pin point which user is getting locked out? is it the user that's authenticating or the bind user being used to query ldap?19:58
orange_juliusIts the user that is authenticating and its actually any user. We were able to replicate with my account (that sucked)19:59
lbragstaddo you have a different value for that?19:59
lbragstader - does your account have a different lockout setting?19:59
orange_juliusNo, its a "global" policy. 3 bad attempts20:00
lbragstadand just to confirm, this is what users are doing POST /v3/auth/tokens ?20:00
orange_juliusYes. well its through horizon and a POST /v3/auth/tokens is seen in the logs when they attempt. I believe we were seeing that come through the keystone.log on all three controllers20:00
lbragstadok - that makes sense20:01
lbragstadhorizon is building a request to keystone to get a token for the user trying to log in20:01
lbragstadand they are using usernames and passwords?20:01
*** raildo has quit IRC20:02
lbragstadorange_julius: which release are you using?20:02
lbragstadmaster, queens, pike?20:02
orange_juliusnewton actually =(20:03
lbragstadoh ok20:03
lbragstadluckily, i don't think the ldap bits of the identity driver have changed drastically since then20:03
orange_juliusWe will be turning on verbose logging on keystone + ldap which will hopefully help identify the problem. For now we don't have a whole lot unfortunately. Its a "production" system so we can't bounce things without an approval process20:04
orange_juliusThat won't happen until Friday though20:04
lbragstadright now does the user just get an 'Invalid user / password' error?20:05
orange_juliusCorrect. They log into their windows workstation, go to horizon, fatfinger the horizon password, and then their AD account is locked out20:06
orange_juliusand receive bad password on horiozn20:06
*** raildo has joined #openstack-keystone20:07
kmallochorizon might be making multiple attempts20:07
lbragstadat a high level - we start getting into that logic here https://github.com/openstack/keystone/blob/582cab391a10714a4ec8bab1a6cce9b49867f8d4/keystone/identity/core.py#L90720:08
lbragstadwhich calls into the ldap specific implementation https://github.com/openstack/keystone/blob/582cab391a10714a4ec8bab1a6cce9b49867f8d4/keystone/identity/backends/ldap/core.py#L60-L7820:09
lbragstadand there are some common utilities for getting connections to ldap20:09
*** ckonstanski has joined #openstack-keystone20:10
lbragstadbut afaict, i just see one connection getting initiatied?20:10
orange_juliusSweet that is going to be very helpful. Horizon access logs only show up on the controller with the VIP which is only showing one POST to /dashboard/auth/login.20:11
lbragstadorange_julius: do you know if you're setting this in your deployment?20:13
orange_juliusNo we are not. At least not explicitly20:13
lbragstadi wonder if that would have an impact20:13
lbragstadit attempts to reconnect 3 times before failing20:14
lbragstadif using connection pooling (which appears to be enabled by default)20:14
lbragstadso - if that's the case, that could be where your 3 attempts are getting used up20:14
orange_juliusWould the fact that the POST to auth/tokens showing up on all controllers keystone.log indicate that keystone is probably making the request 3 times? one per controller node?20:14
orange_juliusWouldn't it only attempt to reconnect if it failed?20:14
lbragstadright - that's the part i'm curious about20:15
lbragstadif the password is wrong, are we attempting to reconnect?20:16
lbragstadand burning password attempts accidentally?20:16
orange_juliusAhh I see20:16
orange_juliusSo we could set that to... 50. and have our max lockouts be also set at 50 and receive the same result20:17
lbragstadorange_julius: do you have the ability to stand up a test node and point it to your adfs?20:17
lbragstadorange_julius: or just disable it..20:17
orange_juliusI'll start standing up a test infra to validate. Won't be able to test disabling until Friday at the earliest20:18
lbragstadsure - that makes sense20:18
lbragstadjust to be clear, i have no idea if this is actually the problem :)20:18
lbragstadbut it looks suspicious20:18
lbragstadand might just be an unfortunate coincidence20:18
lbragstadthat our default for that particular setting happens to match your lockout setting in AD20:19
lbragstadi guess if you have read access to your AD deployment, a single keystone node would be able to recreate this20:19
lbragstadby keeping those default set, and trying to authenticate using POST /v3/auth/tokens with the wrong password20:20
orange_juliusNo worries. Its worth investigating for sure. Is the retry logic around those links you sent earlier? I see it is supposed to raise an invalid credentials error, but not sure how that is handled20:20
openstackgerritDoug Hellmann proposed openstack/oslo.limit master: add python 3.6 unit test job  https://review.openstack.org/58959920:21
openstackgerritDoug Hellmann proposed openstack/oslo.policy master: add python 3.6 unit test job  https://review.openstack.org/58960320:21
lbragstadwe use another library for ldappool functionality, but i think if retries fail, that exception bubbles up to the user20:21
lbragstadand we pass that configuration option into the constructor for that object https://github.com/openstack/keystone/blob/582cab391a10714a4ec8bab1a6cce9b49867f8d4/keystone/identity/backends/ldap/common.py#L71120:24
*** ckonstanski has left #openstack-keystone20:24
orange_juliusOk thanks. I have some things to investigate =D20:27
lbragstadhopefully it helps20:27
*** raildo has quit IRC20:28
lbragstadi'm curious to know if that's actually the problem20:29
*** hoonetorg has joined #openstack-keystone20:29
orange_juliusI will report back once I have the test case.20:30
orange_juliusexcept ldap.LDAPError as error: I wonder if an ldap authentication error is of type LDAPError20:31
orange_juliusIt is!20:36
orange_juliusdamnit bragstad20:37
orange_juliusYea so that is totally what is happening right? LDAPError is the base class, so that exception will catch all errors including auths. ldappool will retry 3 times in .3 seconds and lock a user out instantly20:38
lbragstadit's looking to be more and more the case20:39
lbragstadtotally and unfortunate combination of configuration options between two systems lol20:39
orange_juliusI'm just surprised this hasn't come up before20:39
lbragstadyeah... me, to20:39
lbragstadbut i guess it depends on what people are setting their locks out to on AD20:40
lbragstadif they set it to 5, then the likelihood of hitting this drops significantly (i would assume)20:40
orange_juliusYea seems like it. I'm going to set up a VM and play with the connection pooling settings to see if I can replicate. If I can... what do you want me to submit for this?20:42
lbragstadi mean - we could document this as a bug and just write down the coincidence and mark it as invalid20:42
-openstackstatus- NOTICE: Due to a bug, Zuul has been unable to report on cherry-picked changes over the last 24 hours. This has now been fixed; if you encounter a cherry-picked change missing its results (or was unable to merge), please recheck now.20:42
lbragstadthat would make it discoverable in case other people are randomly hitting this20:43
lbragstadotherwise, i'm not sure we can adjust the defaults in keystone, just because what would you choose?20:43
lbragstadit becomes a guessing game as to what you'd expect a reasonable retry to be and assuming that's what people are doing20:44
lbragstadbut the net would be20:44
orange_juliusWell why would you ever retry on a bad password?20:44
orange_juliusI guess that would be something for ldappool20:44
lbragstadyou could teach ldappool to try and distinguish the difference20:44
lbragstadso - a bug to ldappool is probably more appropriate20:45
orange_juliusI think getting it documented would be a great start. You are right, you can't increase the number of retries or you might break more people. 3 is fairly low already20:45
*** nicolasbock has quit IRC20:45
lbragstadwe can just document the workaround in the ldappool bug20:46
orange_juliusSounds good to me20:46
lbragstad"if you're relying ldap to enforce user lock outs, just be sure to set retry_max to 0 until ldappool knows how to distinguish different authentication failure types"20:47
lbragstadotherwise you risk locking users out prematurely20:47
orange_juliusYou also could introduce strange behaviors which might have been masked by the retries, but its better than an instant lock out.. especially when people have to call to get their account unlocked =p20:48
lbragstadright - i can see that being totally frustrating20:49
lbragstadwant me to open a bug or have you already started?20:49
orange_juliusI can open it if you are busy, but I havn't started yet.20:50
lbragstadkmalloc: we're putting ldappool bugs here - yeah? https://bugs.launchpad.net/ldappool20:51
kmallockmalloc: i think so20:52
kmallockmalloc: i mean, that is where i'd put them ;)20:52
lbragstadme too, we just haven't had one opened yet ;)20:52
orange_juliusoh man nice! A first!20:52
* lbragstad cringes watching ldappool paint suffer it's first scratch20:53
lbragstadi'll leave the honors to orange_julius heh20:54
orange_juliusMaybe I'm just dumb, but I don't see a button to report bugs. Its usually on the far rightyea?20:58
orange_juliusw00t thanks20:59
lbragstadyou should see a Report Bug hiding in the upper right corning of the page20:59
orange_juliusThanks for helping troubleshoot this lbragstad. Much appreciated21:09
orange_juliusSubmitted: https://bugs.launchpad.net/ldappool/+bug/178589821:10
openstackLaunchpad bug 1785898 in ldappool "Connection Pooling Retries Failed Passwords" [Undecided,New]21:10
orange_juliusI had to reallllly restrain myself from putting "first" somewhere in there =P21:10
*** spilla has quit IRC21:12
openstackgerritMorgan Fainberg proposed openstack/keystone master: Convert regions API to flask native dispatching  https://review.openstack.org/58964021:15
openstackgerritMorgan Fainberg proposed openstack/keystone master: Convert services api to flask native dispatching  https://review.openstack.org/58964121:15
openstackgerritMorgan Fainberg proposed openstack/keystone master: Convert endpoints api to flask native dispatching  https://review.openstack.org/58964221:15
kmalloclbragstad: ^ annnnnnnnd there is another subsystem done.21:15
kmalloclbragstad: i.. i think we're down to the "really hard"(tm) ones21:15
lbragstadorange_julius: anytime - i'll wait to confirm the bug once we know it's recreateable with your system21:24
*** edmondsw has quit IRC21:29
orange_juliusSounds good. I'll probably get around to reproducing tomorrow in a test environment and then hopefully we can our change request approved for Friday. Once I confirm it there I'll update the bug21:29
orange_juliushopefully we can get our change request approved**21:30
*** spilla has joined #openstack-keystone21:33
*** rcernin has joined #openstack-keystone22:20
*** spilla has quit IRC22:50
*** edmondsw has joined #openstack-keystone22:54
*** edmondsw has quit IRC22:59
*** _ix has quit IRC23:05
openstackgerritMerged openstack/keystone master: Convert OS-AUTH1 paths to flask dispatching  https://review.openstack.org/58806423:16
*** s10 has quit IRC23:37
*** s10 has joined #openstack-keystone23:38
*** s10 has quit IRC23:38
*** s10 has joined #openstack-keystone23:39
*** s10 has quit IRC23:39
*** s10 has joined #openstack-keystone23:39
*** s10 has quit IRC23:40
*** s10 has joined #openstack-keystone23:40
*** s10 has quit IRC23:41
*** s10 has joined #openstack-keystone23:41
*** s10 has quit IRC23:41
openstackgerritMorgan Fainberg proposed openstack/keystone master: Convert regions API to flask native dispatching  https://review.openstack.org/58964023:53
openstackgerritMorgan Fainberg proposed openstack/keystone master: Convert services api to flask native dispatching  https://review.openstack.org/58964123:54
openstackgerritMorgan Fainberg proposed openstack/keystone master: Convert services api to flask native dispatching  https://review.openstack.org/58964123:54

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!