Wednesday, 2017-08-30

feefifoit seems like the domain token doesn't come with a service catalog00:01
feefifois there a way to configure keystone to include it?00:02
*** kbaegis has joined #openstack-keystone00:03
*** edmondsw has joined #openstack-keystone00:08
*** edmondsw has quit IRC00:12
*** edmondsw has joined #openstack-keystone00:15
feefifonever mind00:16
feefifoi think the issue i ran into is the implied roles doesn't map to domains00:16
*** edmondsw_ has joined #openstack-keystone00:17
*** edmondsw has quit IRC00:19
*** edmondsw_ has quit IRC00:21
*** thorst_afk has joined #openstack-keystone00:22
*** thorst_afk has quit IRC00:29
*** thorst_afk has joined #openstack-keystone00:30
*** thorst_afk has quit IRC00:34
*** edmondsw has joined #openstack-keystone00:40
*** lbragstad has joined #openstack-keystone00:44
*** ChanServ sets mode: +o lbragstad00:44
*** edmondsw has quit IRC00:44
*** zhurong has joined #openstack-keystone00:45
*** Shunli has joined #openstack-keystone00:46
feefifohi lbragstad00:47
lbragstadfeefifo: o/00:48
feefifodo you have a few mins for some q's?00:48
lbragstadfeefifo: sure00:48
feefifothanks00:48
feefifoi've been running some tests on the cloud admin policy file00:48
feefifoand i've been finding some issues with some of the rules00:49
feefifoeg. https://github.com/openstack/keystone/blob/master/etc/policy.v3cloudsample.json#L4000:49
feefifothis rule seems to suggest that listing projects is granted to cloud admins or domain admins00:49
feefifobut keystone doesn't seem to accept a domain token for the projects endpoint00:50
lbragstadfeefifo: there are some issues with that policy file00:50
lbragstadthe test coverage isn't as good as the testing of the default policy file, unfortunately00:50
feefifoah i see00:50
lbragstadfeefifo: also00:51
lbragstadwhile it's possible to get domain scoped tokens, they aren't really accepted anywhere, yet00:51
*** basilAB has left #openstack-keystone00:51
lbragstadwe have quite a bit of work to do across the various services to get them honored consistently in openstack00:51
feefifoi see00:51
feefifook that's really good to know00:52
lbragstadcertainly something we're hoping to close the gap on00:52
feefifoso what purpose does the domain token serve today?00:53
lbragstadfeefifo: not a whole lot because it hasn't really be implemented anywher e00:54
lbragstadideally - a service would recognize a domain scoped token and perform the API according to the domain scoped instead of a project00:55
lbragstaddomain scope*00:55
lbragstada good example would be list instance00:55
lbragstadif you pass a project scoped token to nova and ask for a list of instances, you expect to see all the instances owned by that project, right?00:55
feefifoyes00:55
feefifoit would be great to have that for domains too00:56
lbragstadright00:56
lbragstadbut the list consists of all instances owned by all projects within that specific domain00:56
feefiforight00:56
lbragstadyeah - it'd be awesome to have that00:57
feefifodefinitely agree00:57
feefifoi'm guessing there's also a lot of work to support domain tokens in keystone?00:57
lbragstadbut - we have to do quite a bit of work with other projects to consistently make those changes00:57
feefifolike listing projects in a domain00:57
lbragstadfeefifo: i'm sure there are still some gaps in the keystone api00:57
lbragstadwith respect to domain scoped tokens00:57
lbragstadi don't believe domain scoped tokens work that way in keystone today00:58
feefifoi see00:58
feefifowow, i have a feeling that it will be a few releases before we see full support across all components00:58
lbragstadfeefifo: likely00:59
lbragstadfeefifo: there are a bunch of things we have to work on from a policy perspective00:59
lbragstadpolicy/RBAC00:59
feefifoi see00:59
lbragstadonce we have a few of the bigger things fixed and integrated into other projects, it will likely pave the way to do stuff better with domain-scoped tokens01:00
feefifowhat are some of the bigger things you guys are working on?01:00
lbragstadwell - we currently have some issues with admin-ness in openstack01:01
lbragstadare we're currently working on addressing that01:01
lbragstadwe're also moving all policy default into code and documenting them (much like how we handle configuration options)01:01
feefifoi see01:01
*** thorst_afk has joined #openstack-keystone01:02
lbragstadwe've also talked about associating different levels of scope to various operations in openstack01:02
lbragstadwhich will play a big part in getting traction with domain scoped tokens i think01:02
*** jamesbenson has joined #openstack-keystone01:02
feefifoyes that seems crucial for scope01:03
*** thorst_afk has quit IRC01:03
lbragstadbut once all policy in code and documented (which is community goal for all applicable projects this release) we should be in a better place to make that change01:04
lbragstad(e.g. listing hypervisors in nova requires global scope, but listing instances only requires project or domain scope)01:04
feefifogot it01:05
feefifoi'd be interested in helping out on some of that01:05
lbragstadfeefifo: awesome :)01:06
feefifois there a blueprint or ticket somewhere?01:06
lbragstadyeah - let me grab you some link s01:06
feefifofor tracking the overall progress01:06
feefifocool thanks!01:06
lbragstadfeefifo: so - this is the infamous bug report https://bugs.launchpad.net/keystone/+bug/96869601:06
openstackLaunchpad bug 968696 in OpenStack Identity (keystone) ""admin"-ness not properly scoped" [High,In progress] - Assigned to Lance Bragstad (lbragstad)01:06
*** jamesbenson has quit IRC01:07
lbragstadthat essentially details the admin-ness issue01:07
feefifoOk01:07
lbragstadin an effort to help lay things out and make it easy for others to get up-to-speed with the problem01:08
lbragstadi wrote a couple other documents01:08
lbragstadhttps://review.openstack.org/#/c/460344/ and https://review.openstack.org/#/c/462733/1101:08
feefifooh perfect, thanks!01:08
lbragstadthe first one is a pretty general document with a high level view01:08
lbragstadthe second just elaborates on one way we can improve security by fixing some policy issues01:09
lbragstadthen there is https://review.openstack.org/#/c/464763/501:09
lbragstadif you really want to dig into context and more discussion, there is a thread on the mailing list that describes a couple approaches01:10
lbragstadhttp://lists.openstack.org/pipermail/openstack-dev/2017-May/117419.html01:10
lbragstadbut there is a bunch of good information in ^01:10
feefifohmm do global roles overlap with inherited roles?01:11
lbragstadfeefifo: umm - not really01:11
feefifook i'll read deeper into the doc01:11
lbragstadinherited roles are meant to be applied in project hierarchies01:11
lbragstadif project B is a child of project A and I have a role on project A then it gets inherited to project B01:12
lbragstad(is the basic gist of it)01:12
lbragstadfeefifo: right now, in openstack, when you make a role assignment you have to supply an actor (e.g. user or group) and a target, right?01:13
feefifoyes01:14
feefifoeither a project or domain01:14
lbragstadthe target always has to be a project or a domain01:14
lbragstadyep01:14
lbragstadglobal roles makes it possible to assign someone a role globally01:14
lbragstadintroducing another level of scope, if you will01:14
feefifoah okay01:14
lbragstad(that implementation is here https://review.openstack.org/#/q/status:open+project:openstack/keystone+branch:master+topic:bp/global-roles _01:15
feefifothat makes sense01:15
lbragstad^ i'm working on getting that in shape for the PTG01:15
feefifoso if i was an admin globally, that doesn't mean i'm admin of all domains and projects01:15
feefifounless the inherit extension was modified to support that i'm guessing01:15
lbragstadfeefifo: it could01:15
lbragstadif you have the admin role globally - you could be considered the operator or global administrator of the cloud01:16
feefifoi see01:16
lbragstadbut - if you had admin on a project, you could create sub projects under your project and stuff like that (for example)01:16
lbragstador if you had admin on a domain01:16
lbragstadyou should be able to setup projects in your domain, create users in your domain, assign them roles on the right projects, etc...01:17
feefiforight01:17
lbragstadright now - we're missing the global scope bit01:17
lbragstad(and the bug report i linked above details that)_01:17
feefifoi see, ok i'll take a closer look01:18
feefifothanks a lot for your time @lbragstad01:19
feefiforeally appreciate it!01:20
feefifohope i can help out in the future01:20
lbragstadfeefifo: anytime - i'm stoked you're interested in helping01:20
lbragstadfeefifo: do you have any other questions?01:20
feefifonot at the moment, i think i'm good01:20
lbragstadfeefifo: good deal - are you going to be at the PTG?01:20
feefifonot sure yet, there's a chance i might go to the summit01:21
*** jmlowe has quit IRC01:21
feefifoif i do i'll definitely be at the keystone meets01:21
*** jmlowe has joined #openstack-keystone01:21
lbragstadok - we are planning a bunch of policy discussion during the PTG in Denver01:21
lbragstadhttps://etherpad.openstack.org/p/keystone-queens-ptg is our schedule01:21
lbragstadin case that helps you get a lay of the land as far as what we're going to be discussing as a larger group01:22
feefifocool thanks, this is good to have01:22
lbragstaddon't hesitate to ping if you have more questions01:22
feefifowill do, thanks again :)01:23
lbragstadanytime01:23
feefifohave a good day / night!01:23
lbragstadyou too!01:23
*** jmlowe has quit IRC01:24
*** jmlowe has joined #openstack-keystone01:25
*** edmondsw has joined #openstack-keystone01:29
*** edmondsw has quit IRC01:33
*** itlinux has joined #openstack-keystone01:34
*** aselius has quit IRC01:34
*** otleimat has quit IRC01:37
*** feefifo has quit IRC01:37
*** edmondsw has joined #openstack-keystone01:42
*** edmondsw has quit IRC01:46
*** kbaegis has quit IRC01:51
*** kbaegis has joined #openstack-keystone01:51
*** edmondsw has joined #openstack-keystone01:56
*** kukacz has quit IRC02:00
*** edmondsw has quit IRC02:00
*** kukacz has joined #openstack-keystone02:01
*** mjax has quit IRC02:03
*** mjax has joined #openstack-keystone02:04
*** thorst_afk has joined #openstack-keystone02:04
*** mjax has quit IRC02:06
*** mjax has joined #openstack-keystone02:06
*** lbragstad has quit IRC02:07
*** mjax has quit IRC02:08
*** thorst_afk has quit IRC02:09
*** edmondsw has joined #openstack-keystone02:20
*** edmondsw has quit IRC02:23
*** edmondsw has joined #openstack-keystone02:24
*** edmondsw has quit IRC02:28
*** edmondsw has joined #openstack-keystone02:31
*** edmondsw has quit IRC02:35
openstackgerritMerged openstack/keystone master: Copy specific distro pages for install guide  https://review.openstack.org/49860502:39
*** sapd_ has joined #openstack-keystone02:45
*** mjax has joined #openstack-keystone03:03
*** mjax has quit IRC03:05
*** thorst_afk has joined #openstack-keystone03:05
*** thorst_afk has quit IRC03:10
*** chlong has quit IRC03:21
*** kbaegis has quit IRC03:22
*** kbaegis has joined #openstack-keystone03:22
*** kbaegis has quit IRC03:23
*** jmlowe has quit IRC03:24
*** jmlowe has joined #openstack-keystone03:24
*** mjax has joined #openstack-keystone03:33
*** mjax has quit IRC03:35
*** lbragstad has joined #openstack-keystone03:37
*** ChanServ sets mode: +o lbragstad03:37
*** dave-mccowan has quit IRC03:40
*** links has joined #openstack-keystone03:40
*** gyee has quit IRC03:56
*** abhi89 has joined #openstack-keystone03:57
*** thorst_afk has joined #openstack-keystone04:06
*** lbragstad has quit IRC04:09
*** thorst_afk has quit IRC04:10
*** edmondsw has joined #openstack-keystone04:32
*** edmondsw has quit IRC04:36
*** jmlowe has quit IRC04:47
*** aojea has joined #openstack-keystone04:52
*** zsli_ has joined #openstack-keystone04:56
*** aojea has quit IRC04:57
*** Shunli has quit IRC04:58
*** jamesbenson has joined #openstack-keystone05:00
*** jamesbenson has quit IRC05:04
*** thorst_afk has joined #openstack-keystone05:07
*** thorst_afk has quit IRC05:11
*** abhi89 has quit IRC05:13
*** abhi89 has joined #openstack-keystone05:13
*** zsli_ has quit IRC05:27
*** aojea has joined #openstack-keystone05:56
*** cfriesen_ has quit IRC05:57
openstackgerritTin Lam proposed openstack/keystone master: Add database migration for project tags  https://review.openstack.org/48445605:58
*** aojea has quit IRC06:04
*** thorst_afk has joined #openstack-keystone06:07
*** thorst_afk has quit IRC06:12
*** sapd_ has quit IRC06:15
*** josecastroleon has quit IRC06:21
*** edmondsw has joined #openstack-keystone06:32
*** edmondsw has quit IRC06:37
*** rajalokan has joined #openstack-keystone06:38
*** rcernin has joined #openstack-keystone06:42
*** pcaruana has joined #openstack-keystone06:43
*** itlinux has quit IRC06:51
*** edmondsw has joined #openstack-keystone06:55
*** hoonetorg has quit IRC07:05
*** david-lyle has quit IRC07:08
*** david-lyle has joined #openstack-keystone07:08
*** thorst_afk has joined #openstack-keystone07:08
*** jamielennox has quit IRC07:09
*** gus has quit IRC07:09
*** gus has joined #openstack-keystone07:10
*** thorst_afk has quit IRC07:13
*** jamielennox has joined #openstack-keystone07:14
*** tesseract has joined #openstack-keystone07:19
*** aojea has joined #openstack-keystone07:20
*** hoonetorg has joined #openstack-keystone07:21
*** josecastroleon has joined #openstack-keystone07:36
*** edmondsw has quit IRC07:36
*** josecastroleon has quit IRC07:41
*** ioggstream has joined #openstack-keystone07:44
*** josecastroleon has joined #openstack-keystone07:59
*** thorst_afk has joined #openstack-keystone08:09
*** thorst_afk has quit IRC08:14
*** jaosorior has quit IRC08:55
*** jaosorior has joined #openstack-keystone09:00
*** StefanPaetowJisc has joined #openstack-keystone09:01
*** StefanPaetowJisc has quit IRC09:04
*** thorst_afk has joined #openstack-keystone09:10
*** StefanPaetowJisc has joined #openstack-keystone09:11
*** StefanPaetowJisc has quit IRC09:13
*** thorst_afk has quit IRC09:14
*** dims has quit IRC09:28
*** edmondsw has joined #openstack-keystone09:37
*** kbaegis has joined #openstack-keystone09:37
*** edmondsw has quit IRC09:42
*** kbaegis has quit IRC09:49
*** kbaegis has joined #openstack-keystone09:49
*** dims has joined #openstack-keystone09:58
*** jamesbenson has joined #openstack-keystone09:59
*** jamesbenson has quit IRC10:03
*** jmlowe has joined #openstack-keystone10:10
*** thorst_afk has joined #openstack-keystone10:11
*** kbaegis1 has joined #openstack-keystone10:14
*** thorst_afk has quit IRC10:15
*** kbaegis has quit IRC10:17
*** abhi89 has quit IRC10:25
*** dave-mccowan has joined #openstack-keystone10:29
*** kbaegis1 has quit IRC10:33
*** zhurong has quit IRC10:52
*** thorst_afk has joined #openstack-keystone11:12
*** thorst_afk has quit IRC11:16
*** abhi89 has joined #openstack-keystone11:20
*** edmondsw has joined #openstack-keystone11:38
*** edmondsw has quit IRC11:43
*** jmlowe_ has joined #openstack-keystone11:45
*** jmlowe has quit IRC11:47
*** thorst_afk has joined #openstack-keystone11:56
*** raildo has joined #openstack-keystone12:02
*** kbaegis has joined #openstack-keystone12:02
*** raildo has quit IRC12:07
*** abhi89 has quit IRC12:07
*** lifeless has quit IRC12:07
*** brad[] has quit IRC12:07
*** abhi89 has joined #openstack-keystone12:07
*** brad[]` has joined #openstack-keystone12:07
*** john5223 has quit IRC12:09
*** tesseract has quit IRC12:11
*** john5223 has joined #openstack-keystone12:14
*** dougshelley66 has joined #openstack-keystone12:22
*** lifeless_ has joined #openstack-keystone12:22
*** raildo has joined #openstack-keystone12:22
*** tesseract has joined #openstack-keystone12:26
*** abhi89 has quit IRC12:39
*** abhi89 has joined #openstack-keystone12:59
*** links has quit IRC13:00
openstackgerritColleen Murphy proposed openstack/keystone master: Fix endpoint update example in api-ref  https://review.openstack.org/49914113:06
*** catintheroof has joined #openstack-keystone13:15
*** edmondsw has joined #openstack-keystone13:39
*** edmondsw has quit IRC13:43
*** lbragstad has joined #openstack-keystone13:44
*** ChanServ sets mode: +o lbragstad13:44
*** ducttape_ has joined #openstack-keystone13:47
knikollao/13:48
*** lucasxu has joined #openstack-keystone13:48
lbragstado/13:49
*** mnaser has joined #openstack-keystone13:50
mnaseri'm doing some testing for pike+keystone deployed via puppet (upgrades) and post upgrade i see a bunch of "Forbidden: You are not authorized to perform the requested action." (i guess policy somehow is blocking calls to /v3/auth/tokens).. i noticed the original policy from ocata was there and replaced it by "{}" (for policy in code stuff) and i'm still seeing this13:55
mnaserwould anyone have any ideas?13:55
mnaseri'm just wondering if the puppet modules are missing something or if there are things being carried over which should be be marked absent in puppet13:55
*** gyee has joined #openstack-keystone13:56
lbragstadmnaser: o/13:57
mnaserhey lbragstad :)13:58
lbragstadmnaser: we moved all policy into code - so if the policy files contained defaults, you can remove them13:58
mnaserlbragstad that was my first step, so now my policy is just {} yet still getting that13:58
lbragstadmnaser: when you try to validate a token?13:59
mnaserlbragstad when i try to authenticate13:59
lbragstadhuh - really?!13:59
lbragstadhttps://github.com/openstack/keystone/blob/e0a8780b63dfe611c587a654855158968b86373b/keystone/auth/routers.py#L3013:59
mnaserhttp://paste.openstack.org/show/619895/13:59
mnaseri mean unless i'm on the wrong place but14:00
mnaserthat error seems like a policy denial14:00
lbragstadhttps://github.com/openstack/keystone/blob/e0a8780b63dfe611c587a654855158968b86373b/keystone/auth/controllers.py#L10714:00
mnasernot a unauthorized14:00
mnaser(unless i'm wrong?)14:00
lbragstadwe apply the policy protection using a decorator at the controller layer14:00
lbragstadhttps://github.com/openstack/keystone/blob/e0a8780b63dfe611c587a654855158968b86373b/keystone/auth/controllers.py#L29914:00
lbragstadfor example ^14:00
mnaserbut that error is a policy error (just to confirm right?)14:01
lbragstadbut we don't even protect the authentication endpoint14:01
mnaseroh14:01
mnaserwell that's what i thought14:01
lbragstadchecking the trace14:01
mnaserit wouldn't even make sense for it to be14:01
lbragstadright14:01
lbragstadfwiw - that error is our generic Forbidden error14:01
lbragstadwhich is what's used in policy enforcement14:01
mnaseroh so it could be an auth issue14:02
mnaseri wonder if it's memcache not being restarted after the upgrade14:02
lbragstadhttps://github.com/openstack/keystone/blob/e0a8780b63dfe611c587a654855158968b86373b/keystone/common/policy.py#L6014:02
mnasernope14:02
lbragstadhow would that lead to 403s?14:03
mnaserlbragstad ive seen sometimes where memcache might have data that doesn't make sense to the newer openstack release14:03
mnasermaybe not in keystone specifically but in other services (i cant remember)14:03
lbragstadah14:03
mnaserlbragstad ok. so that request was magnum making trust auth that was messing up14:04
*** links has joined #openstack-keystone14:05
mnaseri just tried a normal auth (looks like keystone v2 was configured on this dashboard)14:05
mnaserAuthorization failed. The request you have made requires authentication. from 192.168.0.250: Unauthorized: The request you have made requires authentication.14:05
mnaserok, basic troubleshooting: keystone-manage doctor shows "Credential key repository is empty."14:06
*** links has quit IRC14:06
*** pcaruana has quit IRC14:06
lbragstadyeah - that should be fine14:06
lbragstadwe built in a null key that will be used if the key repository isn't populated14:07
*** edmondsw has joined #openstack-keystone14:07
mnaserkey repo is readable by user (does keystone do a failsafe "i wont auth anything" if it is?)14:07
*** gyee has quit IRC14:08
lbragstadif the process running keystone can't read that location - you'll see a failure for sure14:08
lbragstadwe do that if the fernet token provider is used and the key repository isn't readable https://github.com/openstack/keystone/blob/e0a8780b63dfe611c587a654855158968b86373b/keystone/token/providers/fernet/core.py#L33-L4514:09
mnaserlbragstad certainly not that, sudo -u keystone cat /etc/ekystone/fernet-keys/X works well14:11
lbragstadmnaser: ok - so you have keys14:12
lbragstadare you authenticating against v2.0 or v3?14:13
mnaserin this case the failure is happening against both (v2.0 from dashboard, v3 from magnum-conductor with trust auth)14:13
lbragstadso you're scoping a trust14:14
mnaserdb revision is 109 which matches the latest as well hm14:14
*** rbrndt has joined #openstack-keystone14:14
mnaserat this point i simplified it to osc client and that is failing auth too14:14
mnaseri wonder if it has to do with the password hashing changes14:14
lbragstadi just thought of that14:15
mnaser"It is recommended passwords be changed after upgrade to Pike."14:15
mnaserdoesnt imply it should be but..14:16
lbragstadmnaser: there was an upgrade path built in14:16
lbragstadhttps://review.openstack.org/#/c/438701/14:16
mnaserwonder if that didnt kick in somehow14:16
*** jamesbenson has joined #openstack-keystone14:16
lbragstadmnaser: do you see the compat option here - https://review.openstack.org/#/c/438701/17/keystone/conf/identity.py ?14:17
lbragstadline 17114:17
mnaserlbragstad it is not present in keystone.conf but it is present (but commented out) in keystone.conf.rpmnew14:17
mnaser(this was a rpm upgrade with rdo pkgs)14:18
lbragstadah - then it would be False14:18
mnaserin this case, i turned off all the other keystone instances14:18
mnaserso only 1 is running14:18
lbragstadok14:19
mnaseri see contract/expand... is that a keystone-manage command or is it integrated with keystone-managed?14:19
mnasers/managed/manage/14:19
lbragstadyeah - so we have keystone-manage db_sync14:20
lbragstadbut we also have keystone-manage db_sync --expand14:20
lbragstadkeystone-manage db_sync --migrate, and keystone-manage db_sync --contract14:20
lbragstadby default, if you just run keystone-manage db_sync it will perform all three14:20
*** jamesbenson has quit IRC14:21
lbragstadthe --expand, --migrate, --contract, bits are specific to doing a rolling upgrade14:21
mnaserokay i see, so it's no tthat14:21
lbragstadcc kmalloc ^14:21
mnaserok with password_hash_algorithm being unset wont it default to bcrypt .. is it possible it's trying to test passwords using bcrypt?14:22
mnaserbecause technically keystone couldn't possibly reencrypt (unless the hashing method is stored in the db?)14:23
mnaseror _get_hasher_from_ident is not doing it's job possibly14:24
cmurphythe hashing method is stored with the password14:25
mnaserhmm14:25
mnasershould the password be stored in password or password_hash14:26
*** jamesbenson has joined #openstack-keystone14:26
lbragstadhttps://github.com/openstack/keystone/blob/e0a8780b63dfe611c587a654855158968b86373b/keystone/common/password_hashing.py#L37-L4214:26
lbragstadmnaser: new passwords should be stored in password_hash i believe14:26
mnaserwhat i see in doing "select * from password;" is that password_hash is null for all except admin user (which is failing to auth)14:26
lbragstadlet me double check the change14:26
mnaserand then password is not null for the rest14:27
*** ducttap__ has joined #openstack-keystone14:28
*** ducttape_ has quit IRC14:28
lbragstadyeah - it returns password_hash if it exists in the backend, otherwise it returns the hash stored in password14:28
lbragstadhttps://review.openstack.org/#/c/438701/17/keystone/identity/backends/sql_model.py14:28
*** mvk has joined #openstack-keystone14:29
*** cfriesen_ has joined #openstack-keystone14:31
mnaserlbragstad ok something is odd here14:32
mnasermanually generated bcrypt for 'foobar'14:32
mnaserset it in password_hash14:32
mnaserit auth'd14:32
mnaseri wonder what happened exactly and how to prevent it14:32
*** aojea has quit IRC14:32
lbragstadhmm14:33
lbragstadjust to double check14:33
lbragstadyou're currently running pike14:33
mnaseryes, the release hash14:33
lbragstadand you have version 109 in the migrate repo14:33
mnaseryep, verified that too14:33
lbragstadthe expand, migrate, and contract repos are all at version 24, right?14:33
mnaserlet me check that14:33
mnaseryes14:33
lbragstadok14:34
mnaserthe only weird thing i saw happen running puppet was.. the package was upgraded, httpd was not restarted yet, and puppet started configuring resources14:34
lbragstadconfiguring resources?14:34
mnaserpuppet can create keystone users etc by calling openstack CLI14:35
mnaserso maybe the fact that the old codebase was still running and it might have started to try and configure the passwords14:35
lbragstadah - so it started doing that before httpd was restarted to effectively doing that against ocata instead of pike14:35
mnaseri suspect that is the case, let me pull out the puppet logs14:36
lbragstadok14:36
lbragstadif you get a token now - can you create a new user and try authenticating?14:36
lbragstadnow that you've resolved the hash issue manually14:36
mnaserindeed, that is the case that it started running it against it14:38
lbragstadaha14:38
mnaserso we need to make sure that this doesn't happen14:39
mnaserlet me verify i can add users properly etc14:39
lbragstadmnaser: can you forcible bounce httpd immediately after the upgrade?14:39
mnaserlbragstad we do have 'anchor' points where we can control order14:39
lbragstadthat sounds promising14:40
mnaserok14:41
mnasersomething weird is happening14:41
mnaserso it was working, i reran puppet, puppet tried to remove the debug flag i enabled, because it did that, it triggered a db_sync, fernet_setup and bootstrap exec's14:41
mnaserand by that point i couldn't authenticate again14:42
mnaserlet me see if i can repro..14:42
lbragstadbut the database should have been upgraded, right?14:42
*** ducttape_ has joined #openstack-keystone14:42
mnaserlbragstad it is, but puppet doesn't know that so it's kindof a noop14:43
mnaserat least we assume that in puppet land14:43
lbragstadyeah14:43
lbragstadbut it does invoke bootstrap14:44
mnaseris bootstrap not noop as well?14:44
mnasermaybe that's what's happening14:45
lbragstadmnaser: bootstrap should be idempotent14:45
lbragstadhttps://github.com/openstack/keystone/blob/e0a8780b63dfe611c587a654855158968b86373b/keystone/cmd/cli.py#L216-L26114:45
*** ducttap__ has quit IRC14:45
mnaserlbragstad ok i think i found the culprit14:46
mnaserlooking at the logs14:46
mnaser2017-08-30 14:39:48.074 19211 INFO keystone.cmd.cli [req-447bb23f-7018-47bc-92f3-57ecb6721491 - - - - -] Reset password for user admin.14:46
mnaserso it looks like the bootstrap is resetting the password14:46
lbragstadhttps://github.com/openstack/keystone/blob/e0a8780b63dfe611c587a654855158968b86373b/keystone/cmd/cli.py#L245-L25214:46
lbragstadso - do you know if that password reset is happening against ocata or pike?14:47
mnaserthe password reset has to happen again pike, i did a yum update at this point on this vm14:48
lbragstadif the password reset happens on pike, shouldn't things work?14:48
lbragstadis the issue that the password get reset using a different hash from ocata that pike doesn't understand?14:48
mnaserlbragstad or maybe it's a puppet bug and the way it supplies the password to the bootstrap too possibly14:49
lbragstadmnaser: after pike is running, can you use bootstrap to reset the admin password?14:50
lbragstadand authenticate with the admin user after that14:50
*** ducttap__ has joined #openstack-keystone14:50
lbragstadthat case *should* work14:50
mnaserok i think this is a puppet issue.  it looks like if it doesn't get admin_password supplied, it uses the value of admin_token14:51
mnaserwhich means that the password is reset to the value of the token14:51
lbragstadoh - interesting14:52
mnaserwhich really means it flip/flops because the resource which creates the admin user is setting a different password than the one bootstrap is using14:52
lbragstadright14:52
lbragstadthat's a long password to remember ;)14:52
mnaserso bootstrap sets admin user to $token, then later puppet runs openstack user set password <the_actual_password>14:52
lbragstadinteresting14:53
*** ducttape_ has quit IRC14:53
mnaserlbragstad sorry for the noise :( looks like that was issue the whole time14:54
mnaseri have no idea why it's manifested itself *just now*14:55
lbragstadmnaser: yeah - that's surprising14:56
lbragstadmnaser: no worries - glad you were able to get it figured out :)14:56
*** edmondsw has quit IRC14:56
mnaserlbragstad personal goal is to get pike deployed asap and looks like keystone will be one of the first up :) thanks for the help14:56
lbragstadmnaser: good deal! let us know if you run into anything else.14:56
mnaseri think this seems to be it for now! merci beaucoup14:57
lbragstadmnaser: anytime!14:57
*** rcernin has quit IRC15:09
*** thegreenhundred has joined #openstack-keystone15:10
*** kbaegis has quit IRC15:14
*** mvk has quit IRC15:15
*** jamesbenson has quit IRC15:31
*** gyee has joined #openstack-keystone15:33
*** kbaegis has joined #openstack-keystone15:33
*** kbaegis has quit IRC15:38
*** aojea has joined #openstack-keystone15:39
*** aojea has quit IRC15:39
*** aojea has joined #openstack-keystone15:39
*** kbaegis has joined #openstack-keystone15:40
*** itlinux has joined #openstack-keystone15:47
*** itlinux has quit IRC15:48
*** thegreenhundred has quit IRC15:48
openstackgerritGage Hugo proposed openstack/keystone master: Refactor removal of duplicate projects/domains  https://review.openstack.org/49157415:48
*** manjaroi3 has joined #openstack-keystone15:48
*** pcaruana has joined #openstack-keystone15:54
*** manjaroi3 has quit IRC15:56
*** thegreenhundred has joined #openstack-keystone15:56
*** thegreenhundred has quit IRC15:58
*** thegreenhundred has joined #openstack-keystone15:58
*** jmlowe_ has quit IRC16:02
*** abhi89 has quit IRC16:09
openstackgerritSamuel Pilla proposed openstack/python-keystoneclient master: Add project tags to keystoneclient  https://review.openstack.org/48122316:14
lbragstadcmurphy: do you run full devstack (including nova) on your x1?16:15
*** aojea has quit IRC16:15
openstackgerritOmar Tleimat proposed openstack/keystone master: Add unit tests to mapping_purge  https://review.openstack.org/40830416:17
cmurphylbragstad: i usually just keystone + nova + *-api but a full devstack is possible16:18
cmurphyneeds an 8GB vm16:19
gagehugolbragstad I used a full devstack for that tempest random failing test bug, it worked fine imo16:20
lbragstadcmurphy: ah - you run in a virtual machine16:20
lbragstadcmurphy: gagehugo have either of you tried running devstack in a container? https://docs.openstack.org/devstack/latest/guides/lxc.html16:21
cmurphylbragstad: heh no16:21
cmurphyi have a whole virsh workflow that i'm pretty settled on16:21
lbragstadnice16:21
openstackgerritOmar Tleimat proposed openstack/keystone master: Add unit tests to mapping_purge  https://review.openstack.org/40830416:23
*** otleimat has joined #openstack-keystone16:24
gagehugolbragstad no, I've used openstack-helm before to test some keystone changes16:31
gagehugobut haven't done a full stack yet16:31
*** itlinux has joined #openstack-keystone16:31
*** rbrndt has left #openstack-keystone16:32
*** markvoelker has joined #openstack-keystone16:32
*** tesseract has quit IRC16:33
gagehugohttp://openstack-helm.readthedocs.io/en/latest/install/developer/all-in-one.html16:36
*** aselius has joined #openstack-keystone16:46
*** jamesbenson has joined #openstack-keystone16:53
*** jamesbenson has quit IRC16:57
*** stlbigdog has joined #openstack-keystone17:02
*** markvoelker has quit IRC17:07
lbragstadgagehugo: cmurphy interesting - i hit an issue where the container doesn't start because of a missing character device file on the host17:11
lbragstadi guess it's needed so that the container can actually use kvm17:12
*** aojea has joined #openstack-keystone17:12
*** dims has quit IRC17:14
*** dims has joined #openstack-keystone17:18
gagehugohmm17:18
*** stlbigdog has quit IRC17:23
gagehugolbragstad it looks like there is some concern about the project tag character size limit of 60, would there be any reason to object upping it?17:25
lbragstadgagehugo: i don't think so - but i think we limited it based on the fact nova limits it and we wanted to maintain consistency17:26
gagehugolbragstad I think we got 60 from the nova spec, but it looks like they upped it to 80 for the implementation17:26
lbragstadgagehugo: i'd be fine updating our spec to maintain consistency then17:27
lbragstadi don't think we should deviate17:27
gagehugoyeah, I've been updating the spec as things change17:27
*** ioggstream has quit IRC17:28
*** stlbigdog has joined #openstack-keystone17:28
lbragstadgagehugo: maybe run it by the nova folks and see if they'd be interested in upping their limit,too?17:28
gagehugolbragstad 80 would be fine imo17:30
*** homeski has joined #openstack-keystone17:30
lbragstadi think consistency would be good17:30
homeskiQuestion about Keystone with Active Directory: How is the syncing happening?17:31
gagehugoI'll probably break up the current change into controller/manager/backend anyway so I can change that as well17:31
homeskiCan't find any documentation about it17:31
*** eandersson has quit IRC17:31
lbragstadhomeski: i'm not sure we have much documentation on Active Directory specifically17:34
lbragstadhttps://wiki.openstack.org/wiki/HowtoIntegrateKeystonewithAD17:34
lbragstadbut that does use the ldap backend for keystone17:34
*** mjax has joined #openstack-keystone17:34
lbragstadhttps://github.com/openstack/keystone/blob/e0a8780b63dfe611c587a654855158968b86373b/keystone/conf/identity.py#L67-L7917:34
*** kbaegis has quit IRC17:35
homeskiIf I add a new user on the LDAP side, is there any sort of syncing that needs to happen from Keystone, so that I can log in?17:35
*** kbaegis has joined #openstack-keystone17:36
lbragstadhomeski: no, so long as the user exists in AD you should be able to log in17:38
lbragstadthe syncing will happen after the authentication happens using AD17:38
homeskiso it queries LDAP, makes sure it's valid credentials, if the user doesn't exist in local identity management, it will create user and default roles17:39
homeski^ for example, if I add a user to LDAP, then immediately try to login with Horizon17:40
mjaxlbragstad: I'm also curious about that, when/which module do roles and projects get assigned to the user that is authenticated on keystone through ldap for the first time?17:43
lbragstadif ldap is configured and you authenticate a shadow user will be created17:44
lbragstadwhich lives in keystone17:44
lbragstadit's referred to as a non local user17:44
lbragstadfrom there you can actually assign the non local user roles on projects and so on17:44
lbragstadand you can operate on various projects like you would if the user lived natively in keystone as a local sql user17:45
*** ducttap__ has quit IRC17:49
mjaxlbragstad: Sorry, i'm not very familiar with ldap, but am curious. Can ldap contain information about the user's domain, roles, and projects for keystone to extract, or is this all going to have to be done manually through the client after authenticating and creating a shadow user?17:52
*** rcernin has joined #openstack-keystone17:52
lbragstadmjax: keystone use to support assignment and resource read/write backends for ldap - but that is no longer supported17:53
lbragstadcurrently, read-only is supported17:53
lbragstadthe assignments should be created manually after the user authenticates17:53
lbragstador is shadowed in keystone17:53
mjaxi see17:55
mjaxwhat do you mean by shadowed in keystone?17:55
*** edmondsw has joined #openstack-keystone17:55
lbragstadwhen keystone authenticates a user against an ldap backend, a user gets created in keystone's sql backend to model it17:55
lbragstador shadow it17:55
lbragstadso when you give that user role assignments to projects, you're going to be giving those assignments to that shadow user17:56
mjaxright17:56
lbragstadit helps in the case where you're hooking your openstack deployment up to a corporate ldap17:58
lbragstadbut you don't have write access to it17:59
lbragstadso you can control the role assignments to a user persisted in ldap without having to open a bunch of tickets to the team that manages your corporate ldap17:59
mjaxI see, so then the role assignments and authorization stuff are going to be only on keystone, while ldap is used only for authenticating the user. Did I understand that correctly?18:00
lbragstadyep18:01
mjaxI guess that leads me to the question, what's the difference between how shadow users and local users are represented in keystone? Does one have special features or is it mostly just so that shadow users are more easily identifiable18:03
*** ducttape_ has joined #openstack-keystone18:05
openstackgerritColleen Murphy proposed openstack/keystone master: Fix endpoint examples in api-ref  https://review.openstack.org/49914118:07
*** jmlowe has joined #openstack-keystone18:08
homeskilbragstad: thanks for information18:09
homeskivery helpful18:09
*** markvoelker has joined #openstack-keystone18:10
homeskiexactly what I needed to know :)18:10
*** jamesbenson has joined #openstack-keystone18:22
*** aojea has quit IRC18:35
lbragstadhomeski: glad i could help18:36
*** markvoelker has quit IRC18:43
*** edmondsw has quit IRC19:04
*** markvoelker has joined #openstack-keystone19:06
*** edmondsw has joined #openstack-keystone19:06
*** edmondsw has quit IRC19:10
*** rama_y has joined #openstack-keystone19:11
*** edmondsw has joined #openstack-keystone19:14
*** jamesben_ has joined #openstack-keystone19:14
*** jamesbenson has quit IRC19:15
*** edmondsw has quit IRC19:18
*** dims has quit IRC19:28
*** ducttape_ has quit IRC19:28
*** ducttape_ has joined #openstack-keystone19:28
*** catintheroof has quit IRC19:28
*** dims has joined #openstack-keystone19:31
lbragstadwow - devstack actually runs in a container19:33
cmurphyheh19:33
lbragstadi tripped over some cinder stuff with lvm, but...19:33
*** markvoelker has quit IRC19:37
*** markvoelker has joined #openstack-keystone19:38
*** jamesben_ has quit IRC19:45
*** jamesbenson has joined #openstack-keystone19:45
clarkblbragstad: nova compute also has to use the userland iscsi driver stuff in libvirt19:50
clarkbbut thats probably the biggest two items you'll hit with a vanilla devstack run19:50
lbragstadclarkb: yeah - i attempted to follow https://docs.openstack.org/devstack/latest/guides/lxc.html#limitations but that didn't seem to help the lvm issue i was hitting19:51
lbragstadit looked like it couldn't create the volumn group19:52
lbragstadvolume*19:52
*** markvoelker has quit IRC19:56
*** edmondsw has joined #openstack-keystone20:00
*** edmondsw_ has joined #openstack-keystone20:01
*** edmondsw has quit IRC20:04
*** edmondsw_ has quit IRC20:05
*** aojea has joined #openstack-keystone20:09
*** jmlowe has quit IRC20:12
*** jmlowe has joined #openstack-keystone20:15
*** kbaegis has quit IRC20:22
*** stlbigdog has quit IRC20:27
*** jamesbenson has quit IRC20:28
*** catintheroof has joined #openstack-keystone20:29
*** jose-phillips has joined #openstack-keystone20:29
*** nicolasbock has joined #openstack-keystone20:32
*** rama_y has quit IRC20:34
*** rama_y has joined #openstack-keystone20:36
*** jamesbenson has joined #openstack-keystone20:37
*** pcaruana has quit IRC20:40
*** edmondsw has joined #openstack-keystone20:43
*** edmondsw has quit IRC20:48
*** lucasxu has quit IRC20:49
*** ducttape_ has quit IRC21:02
*** ducttape_ has joined #openstack-keystone21:04
*** jmlowe has quit IRC21:04
*** thorst_afk has quit IRC21:12
*** sapd has quit IRC21:26
*** jamesbenson has quit IRC21:27
*** jamesbenson has joined #openstack-keystone21:30
*** jamesbenson has quit IRC21:34
*** ducttape_ has quit IRC21:38
*** edmondsw has joined #openstack-keystone21:39
*** edmondsw_ has joined #openstack-keystone21:40
*** edmondsw has quit IRC21:44
*** jmlowe has joined #openstack-keystone21:44
*** edmondsw has joined #openstack-keystone21:45
*** edmondsw_ has quit IRC21:45
*** ducttape_ has joined #openstack-keystone21:46
*** thegreenhundred has quit IRC21:56
*** edmondsw has quit IRC22:00
*** aojea has quit IRC22:02
*** aojea has joined #openstack-keystone22:02
*** aojea has quit IRC22:06
*** raildo has quit IRC22:08
otleimatcmurphy, lbragstad: reflected those changes here https://review.openstack.org/#/c/408304/22:12
*** dave-mccowan has quit IRC22:12
*** thegreenhundred has joined #openstack-keystone22:18
*** edmondsw has joined #openstack-keystone22:22
*** itlinux has quit IRC22:23
*** edmondsw has quit IRC22:26
*** efried is now known as efried_off22:28
*** rcernin has quit IRC22:28
*** edmondsw has joined #openstack-keystone22:39
*** thegreenhundred has quit IRC22:39
*** edmondsw has quit IRC22:43
*** ducttap__ has joined #openstack-keystone22:49
*** ducttape_ has quit IRC22:52
*** edmondsw has joined #openstack-keystone23:01
*** edmondsw has quit IRC23:06
*** ducttap__ has quit IRC23:12
*** stlbigdog has joined #openstack-keystone23:20
*** ducttape_ has joined #openstack-keystone23:21
*** itlinux has joined #openstack-keystone23:22
*** stlbigdog has quit IRC23:23
*** ducttape_ has quit IRC23:26
*** markvoelker_ has joined #openstack-keystone23:45
*** edmondsw has joined #openstack-keystone23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!