Friday, 2018-03-09

*** r-daneel has quit IRC00:03
*** dtruong has quit IRC00:05
*** dtruong has joined #openstack-keystone00:06
*** sapd_ has quit IRC00:12
*** dtruong has quit IRC00:12
*** dtruong has joined #openstack-keystone00:12
*** david-lyle has quit IRC00:17
*** david-lyle has joined #openstack-keystone00:17
*** david-lyle has quit IRC00:18
*** guys has quit IRC00:24
*** oikiki has quit IRC00:35
*** itlinux has joined #openstack-keystone00:39
*** guys has joined #openstack-keystone01:07
*** anyone is now known as eschwartz01:08
*** annp has joined #openstack-keystone01:09
*** gyee has quit IRC01:13
*** lifeless has quit IRC01:27
*** jrist has joined #openstack-keystone02:06
*** sapd has joined #openstack-keystone02:14
*** harlowja has quit IRC02:21
lbragstadadriant: i thought knikolla his some issue like that02:46
lbragstadjamielennox: ok - back02:46
*** zhurong has joined #openstack-keystone02:47
adriantlbragstad: and this is a weird one. when I assign a role to a domain I created, the scope in the assignment is 'project', when it's to the 'default' domain, the scope is 'domain'. :/02:49
lbragstaduh - i bet it's the is_admin_project stuff02:49
adriantno, wait, nvm02:51
adriantsomething was being weird earlier02:51
adriantok, that's interesting...02:52
adriant'openstack role add test --user admin --project test_domain' throws and error, it can't find the project02:53
adriant'openstack role add test --user admin --project 8ea96eef932b46e4a9abe5f60285bac4' doesn't02:53
adriantand the latter sets the scope to 'project' scope, even though the project is a domain02:53
lbragstad8ea96eef932b46e4a9abe5f60285bac4 is a domain, then?02:54
adriant'openstack role add test --user admin --domain test_domain'  technically does the same thing, but actually sets the scope to domain02:54
adriantthat is the id of a domain :P02:54
lbragstadright - osc should specify the domain in the path for the role assignment02:54
lbragstadthe --domain bit should take care of that02:55
adriantI think it's just the API itself allows setting domain ID for project scope since well... projects are domains.02:56
adriantbut the role assignment stuff doesn't know the difference, so it uses the scope it was given rather than checking, is this actually a domain02:57
adrianthence assignment.scope being:  {u'project': {u'id': u'8ea96eef932b46e4a9abe5f60285bac4'}} or {u'domain': {u'id': u'8ea96eef932b46e4a9abe5f60285bac4'}}02:57
adriantpotentially, when a project is a domain, what we should see is: {u'domain': {u'id': u'8ea96eef932b46e4a9abe5f60285bac4'}, u'project': {u'id': u'8ea96eef932b46e4a9abe5f60285bac4'}}02:58
adriantboth the scope is technically both02:58
lbragstadwell - a domain is a top-level projects02:59
lbragstadnot every project is actually a domain02:59
adriantoh yeah02:59
adriantcorrectly I said it the wrong way around02:59
adriantdomains are projects*03:00
adriantso when a role is assigned to a domain the scope should be both domain and project03:00
adriantalthough now I'm curious how the effective role stuff interprets that03:01
adriantdoes it do anything different if the scope if domain vs project on the same domain project...03:01
lbragstadmight be worth a test...03:02
lbragstadmost of that stuff blows my mind03:02
* adriant finds so many of the weird edge cases03:02
*** namnh has joined #openstack-keystone03:03
adriantok, it looks like or at least feels like being able to set project scope on a domain is a bug03:05
adriantwhen I set 'domain scope' properly, I can do include_names03:05
*** lifeless has joined #openstack-keystone03:06
lbragstadjamielennox: ok - i think i figured out the oslo_context bit (maybe?)
lbragstadthat seems to pass things to the oslo.policy library properly03:17
lbragstadayoung: might be interested in the ^ too03:20
openstackgerritmelissaml proposed openstack/oslo.policy master: Update links in README
*** mburrows has joined #openstack-keystone03:45
jamielennoxlbragstad: sorry, was gone longer than i said03:49
jamielennoxlbragstad:that's the basics yea, does auth_token really set HTTP_X_SYSTEM?03:50
hrybackicmurphy: sorry for the delay -- updated03:53
aningcmurphy: You are right, the keystone "user" table has all the users in it.03:56
*** oikiki has joined #openstack-keystone04:01
*** dave-mccowan has quit IRC04:12
*** sapd has quit IRC04:18
*** oikiki_ has joined #openstack-keystone04:19
*** oikiki has quit IRC04:21
adriantlbragstad: I think there isn't a difference between scope assigned/domain assigned for a role on a domain project, at least not that I can see. It's just that internally the 'scope' if either to the project or the domain, which means I have to explicitly check 'is this a domain scope' rather than just treat them all as project scope assignments.04:26
*** sapd has joined #openstack-keystone04:33
*** germs has quit IRC04:33
*** germs has joined #openstack-keystone04:34
*** germs has quit IRC04:34
*** germs has joined #openstack-keystone04:34
*** germs has quit IRC04:34
*** germs has joined #openstack-keystone04:35
*** germs has quit IRC04:35
*** germs has joined #openstack-keystone04:35
*** zhurong has quit IRC04:36
*** felipemonteiro has joined #openstack-keystone04:40
ayounglbragstad, sorry, I got pulled in to family mode right when you asked your question04:47
ayoungnot sure if your question was inside of keystone or the other services, about populating  the context object04:49
*** zhurong has joined #openstack-keystone04:49
ayoungadriant, all domains ARE projects04:51
ayoungFor the default domain, I can add a role on it, as a project that is then inherited by all subordinat projects04:52
ayoungopenstack role add --user CloudAdmin --user-domain Default --project Default --project-domain Default --inherited admin04:52
ayoungSo for 'openstack role add test --user admin --project test_domain  you need  --project-domain test_domain04:53
ayoungadriant, so you can have both a domain scoped role and a project scoped role on  test_domain04:54
ayoungthey can even be the same named role, like admin or _member_04:54
ayoungbut they mean different things, as the policy will look for which way it is scoped.  I think that is what you were aiming for, you just missesd the need to specify the domain for the project when setting it as a project scoped role04:55
adriantayoung: Oh, I see what you mean. That's... interesting. I wonder if anyone actually uses the distinction between a project/domain scope in their policies. It feels like extra complexity for no real gain in essence.04:58
ayoungadriant, read my blog post04:59
*** germs has quit IRC04:59
ayoungit was really useful in that case04:59
*** germs has joined #openstack-keystone05:00
*** germs has quit IRC05:00
*** germs has joined #openstack-keystone05:00
ayoungYOu can assign a user a role on the domain-as-a-project and they inherit it to all projects underneath it.  Instead of having to use admin to cross project boundaries...looking at cmurphy 's blog, that was one of the things discussed at the PTG05:00
adriantayoung: but that's just inheritance, not any specific difference between domain/project scope05:01
ayoungcloudsample makes the distinction05:01
ayounga few people have used that05:01
adriantI honestly was kind of hoping 'domain scope' wasn't a distinction we cared about since then everything becomes simpler and is just project scope :(05:01
ayoung  for list projects for example05:01
ayoungdomains suck05:02
ayoungreally, they should not exist.05:02
ayoungdomains for users and groups should be IdPs05:02
ayoungdomains for projects should just be projects05:02
ayoungghost of battles lost05:02
adriantyeah, I was kind of hoping I could pretend domains didn't exist and just use 'projects' but the fact that the distinction still exists even in code is annoying05:03
* adriant shrugs05:03
adriantoh well05:03
adriantayoung: basically all I was hoping for as a thing we could do, is make it so when you assign a domain scope, it automatically ALSO assigns a project scope to the project.05:05
adriantbut if people are actually using those two different cases in policy, then... *shrug*05:06
*** felipemonteiro has quit IRC05:12
*** mburrows has quit IRC05:20
ayoungadriant, they are different, and should be05:25
*** harlowja has joined #openstack-keystone05:25
ayoungI could write your own policy and completely ignore the domain aspect of it, but...I thin that the problem comes when you want to do operations that can onluy be done on domains05:26
ayounglike add a user,  if you supplied a project scoped token,  it wouldn't know what to match05:27
adriantbut it would know what the root of that project was05:27
adriantwhich is a 'domain'05:27
ayoungnot in a deep hierarchy05:27
ayoungsay tyou hav domain d, and p1, under that p205:27
ayoungparent of p2 is p1, domaing for both is d05:27
ayoungit would have to be smart enough to say "I can accept a project scoped token where the domain is the same as the project id"05:28
ayoungnot impossible05:28
ayoungdomain is always on the token, regardless of how deeply nested things are05:28
ayoungso you could probably make it work, but you would have some strangeness if you made a role specific for adding users05:29
ayoungpeople could assigne that role to a user-on-a-project-that-is-not-a-domain05:29
ayoungand...well you have to make sure they don't add a user by accident05:30
adriantbut that's how keystone is right now, or was05:30
adriantyou can assign 'admin' anywhere05:30
ayoungSo, yeah, you could probably get away without domain scoped roles at all05:30
ayoungyou'd just have some funky policy rules to enforce for create_user create_group and the like05:31
adriantoh, so the use case is: only allow user create when role X and on scope Domain05:31
adriantrather than: allow user create when role is X05:32
ayoungand so if you had a useradmin role you could, as you said, assign that to a user on a project-that-is-a-domain and then scope check would be...05:32
adriantbut couldn't that also be solved without domain with: allow user create when role X on root05:32
ayounguser.domain_id == token.domain_id and token.domain_id == token.project_id05:33
ayoungwhich is kinda hackish05:33
ayoungthere is no root.  there is Default domain05:33
ayoungall projects live under a domain05:33
adriantby root I mean, root of the given project tree05:33
ayoungyes, that is pretty much what I said there05:34
ayounguser.domain_id == token.domain_id and token.domain_id == token.project_id05:34
ayoungso user  should be target or something05:34
ayoungnot the user in the token05:34
ayoungtarget.domain_id == token.domain_id and token.domain_id == token.project_id05:34
ayoungadriant, that enough for you?  I need to crash.  1/2 past midnight here, and kids are up for school regardless of when I go to bed05:35
adriantgo sleep, I think we're on the same page, but on slightly different topics anyway :P05:36
*** jaosorior has joined #openstack-keystone05:37
*** germs has quit IRC05:45
*** itlinux has quit IRC05:48
*** germs has joined #openstack-keystone05:48
*** germs has quit IRC05:48
*** germs has joined #openstack-keystone05:48
*** germs has quit IRC05:53
*** oikiki_ has quit IRC05:56
*** oikiki has joined #openstack-keystone05:56
*** harlowja has quit IRC06:00
*** itlinux has joined #openstack-keystone06:06
*** itlinux has quit IRC06:09
*** harlowja has joined #openstack-keystone06:19
*** itlinux has joined #openstack-keystone06:20
*** mburrows has joined #openstack-keystone06:23
*** germs has joined #openstack-keystone06:24
*** germs has quit IRC06:24
*** germs has joined #openstack-keystone06:24
*** masber has quit IRC06:26
*** germs has quit IRC06:32
*** jrist has quit IRC06:43
*** jrist has joined #openstack-keystone07:01
*** pcichy has joined #openstack-keystone07:08
*** oikiki has quit IRC07:14
*** pcaruana has joined #openstack-keystone07:27
*** d0ugal_ has quit IRC07:37
*** martinus__ has joined #openstack-keystone07:40
adriantayoung, lbragstad: I've sent an email to the mailing list. ayoung, I think we misunderstood each other, and I think I've found some issues that I can't entirely make sense of. :(07:43
adriantPlease have a look and help me confirm if I'm crazy or not :P07:43
adriantAnd better still, if you can reproduce my issues.07:43
*** namnh has quit IRC07:46
*** d0ugal_ has joined #openstack-keystone07:54
*** sileht has left #openstack-keystone07:56
*** rcernin has quit IRC08:00
*** harlowja has quit IRC08:08
*** tesseract has joined #openstack-keystone08:38
*** dangtrinhnt has joined #openstack-keystone08:46
*** threestrands_ has quit IRC08:48
*** d0ugal_ has quit IRC09:16
*** d0ugal has joined #openstack-keystone09:17
*** d0ugal has quit IRC09:17
*** d0ugal has joined #openstack-keystone09:17
*** zhurong has quit IRC09:18
*** masber has joined #openstack-keystone09:25
*** d0ugal has quit IRC09:32
*** d0ugal has joined #openstack-keystone09:34
*** d0ugal has quit IRC09:42
*** mvk has joined #openstack-keystone09:48
*** d0ugal has joined #openstack-keystone09:56
*** annp has quit IRC10:30
*** mburrows has quit IRC11:04
*** jaosorior has quit IRC11:16
*** jaosorior has joined #openstack-keystone11:16
*** jaosorior has quit IRC11:21
*** jaosorior_ has joined #openstack-keystone11:21
*** jaosorior_ has quit IRC11:23
*** jaosorior__ has joined #openstack-keystone11:23
*** jaosorior__ has quit IRC11:24
*** jaosorior has joined #openstack-keystone11:25
*** jaosorior has quit IRC11:25
*** jaosorior has joined #openstack-keystone11:25
*** dangtrinhnt has quit IRC12:08
*** raildo has joined #openstack-keystone12:17
*** dave-mccowan has joined #openstack-keystone12:21
*** jmlowe_ has quit IRC12:25
*** dangtrinhnt has joined #openstack-keystone12:29
*** dangtrinhnt has quit IRC12:56
*** germs has joined #openstack-keystone13:06
*** germs has quit IRC13:06
*** germs has joined #openstack-keystone13:06
*** germs has quit IRC13:07
*** idlemind has quit IRC13:19
*** pcichy has quit IRC13:31
*** pcichy has joined #openstack-keystone13:49
*** jrist has quit IRC13:55
*** AlexeyAbashkin has joined #openstack-keystone13:56
*** edmondsw has joined #openstack-keystone13:59
*** AlexeyAbashkin has quit IRC14:01
*** AlexeyAbashkin has joined #openstack-keystone14:03
*** jrist has joined #openstack-keystone14:07
*** AlexeyAbashkin has quit IRC14:07
*** jaosorior has quit IRC14:09
* lbragstad is still reading scroll back14:12
ayounglbragstad, it would take a while to untangle all that14:25
ayoungthere appears to be at least two bugs14:25
lbragstadyeah - that's not good14:25
ayoungand quite a few mistakes in typing.  For example, his last:  openstack project create test5 --domain domain114:25
ayoungopenstack role add test --user alice --project 86a8b3dc1b8844fd8c2af8dd50cc2138614:25
ayounghe managed to add an additional 8 on to the id14:26
ayoungit should only be 86a8b3dc1b8844fd8c2af8dd50cc213814:26
ayounger...he added a 614:26
lbragstadhere you mean?
lbragstadthose are the same ids14:26
*** dansmith is now known as superdan14:26
lbragstadoh - wait14:27
lbragstadyeah - 86a8b3dc1b8844fd8c2af8dd50cc2138 is the correct one14:27
ayoungand things like ule:admin_and_matching_domain_id",14:27
ayoung    - "cloud_admin": "role:admin and (is_admin_project:True or14:27
ayoung domain_id:  is how it is reported in the token if it is a domain scoped role14:27
ayoungversus project_domain_id for a project scoped token14:27
ayoungdomains suck14:27
ayoungthey really need to be taken out to the back 40 and left there14:28
lbragstadi agree14:28
lbragstadmunging them with projects was where things really got complicated14:28
openstackgerritJohannes Grassler proposed openstack/keystone-specs master: Add whitelist-extension-for-app-creds
ayoungKeystone API V4 has no domains.14:30
ayoungI think we could actually make a set of policy rules that would mean you never need domain scoped roles14:31
lbragstadyep - a root project doesn't have parent, and that's how you tell14:31
ayoungeverything is a project scoped role...juyst the policy has to check that the project IS A domain for certain operations14:31
lbragstadi just noticed this the other day, but projects have 'is_domain', 'parent_id', and 'parent_project_id' attributes14:31
ayounglbragstad, yeah.  parent_project_id is null if the parent is a domain14:32
ayoungI think that is the differentiator14:32
lbragstadright - and it's *one* thing, versus a combination of three attributes that change depending on if it's a domain or a project14:33
lbragstadi guess if we ever supported microversions, that'd be an area to fix14:34
lbragstader "fix" because you can't actually remove it, but you could hide some of the ugly14:34
lbragstadfor users that opt into using microversions14:34
lbragstadayoung: ok - so what were the two bugs specifically in your opinion (re: adriant's note)?14:35
ayoungthe ones that casue stack traces....14:36
lbragstadso - not being able to remove a role assignment that's made on a project and a domain with the same ID14:36
lbragstadi question if you should even be able to do that...14:36
ayoungor 500s14:37
lbragstadwhy would you need to make a project assignment on a domain?14:37
ayoungyou need both domain admin and project admin.  domain admin on the domain, project admin inherited on all the sub projects14:38
lbragstadso - i would expect that when we merged domain and project resources ( we forgot to clean that up14:39
lbragstador make it so that you could remove those assignments14:40
lbragstadayoung: historical context check - domains were certainly a thing before we munged them together with projects, right?14:40
ayoungthe one() call I think assumes there is only one role in the results, but there are multiple, so it is ambiguous which you want to remove14:40
ayoungdomains were added by gyee, first year I was on the project IIRC14:41
lbragstadok - that sounds right, just making sure14:41
lbragstadand we munged them with projects when we started trying to solve reseller usecases?14:41
ayoungthey were one of the distinguishing factors of the V3 API, so as long as that has been about14:41
ayoungYeah, the HTM code required a root for the tree14:42
lbragstadand we decided to call that root a domain14:42
*** AlexeyAbashkin has joined #openstack-keystone14:43
lbragstadand we punted on the ability to have a domain anywhere but the root of the tree14:43
lbragstadbecause we didn't have anyone with a clear enough use case14:43
*** r-daneel has joined #openstack-keystone14:44
lbragstadso - just to recap... we need to open a bug for the two 500s here
lbragstadwhich could be solved with the same patch14:45
lbragstadare there any other things we need bugs open for?14:45
lbragstador things that I'm missing?14:45
*** AlexeyAbashkin has quit IRC14:47
*** AlexeyAbashkin has joined #openstack-keystone14:51
*** jmlowe has joined #openstack-keystone14:52
cmurphyany other PTG recaps I should link to in the weekly update? knikolla gagehugo raildo hrybacki wxy15:00
raildocmurphy, you and lbragstad already made a great job on that recaps post blogs :)15:01
*** david-lyle has joined #openstack-keystone15:01
knikollacmurphy: my procrastination skills exceeded my writing skills.15:05
knikollai hope to finish it over the weekend.15:06
knikollayou and lbragstad did a great job recapping. so that's more than plenty for the weekly update :)15:07
*** germs has joined #openstack-keystone15:08
*** germs has quit IRC15:08
*** germs has joined #openstack-keystone15:08
ayounglbragstad, we could probably merge domains into projects15:11
ayoungthe only thing that distinguises a domain from a project is the fact that it can own users and groups15:12
ayoungcmurphy, that recap was fantabulous.15:12
*** guys has quit IRC15:12
*** germs has quit IRC15:13
lbragstadcmurphy: i need to read yours, i haven't yet15:16
* lbragstad digs for a link15:16
cmurphyit's probably mostly the same as yours15:17
knikollacmurphy: do you do hand-code html or use a static site generator?15:17
cmurphyexcept i forgot to mention the JWT discussion, will need to update it15:17
cmurphyknikolla: i use pelican15:17
cmurphybut i created the theme myself15:17
ayounglbragstad, so...scope.  For admins15:18
ayoungWHat I am seeing in CloudForms is that we want to say HMT works, and that should be used to scope how a client app interacts with various services15:18
ayoungand this is more than just Keystone15:19
knikollacmurphy: looks pretty cool! i played around a bit with pelican, but github pages compiling jekyll automatically makes it too convenient.15:19
ayoungwe could say "to list all projects in a tree, you need to use a service scoped token" but that feels like giving away too much15:19
ayounglike, Amazon would not let us get service scoped tokens to talk to their service, right?15:19
ayoungAnd I kindof like what I read in the summary about "lets punt on real reseller, and just go 2 level "15:20
ayoung"On Behalf Of" Header...I think might not really fly15:20
ayoungI'd rather it be something based on the mechanisms we already if you have the right role on a parent projects, you can see everything underneath it when doing "list projects"15:21
ayoungand that might be the reason to use domains:15:21
ayoungif I have...domain_manager, I know that the token I have is going to match all of the tokens issued for that domain15:22
ayoungthat was poorly stated15:22
ayoungIf I have domain_admin,  I know that the domain on my token is going to match the domain for the project that manages any resource15:22
ayoungjust,   the remote services don't know about that relationship15:22
ayoungi.e.  a network object in neutron has a project id, but does not know the domain id.15:23
ayoungHowever...with unified limits, it looks like you are tackling that sync?15:23
ayoungSo maybe we can say "now you have the ability to link the project back to its domain, use the domain as a way to group projects for resource_list"15:24
ayoungit was a big part of the discussion back when we did hmt, just ask raildo and henrynash15:24
lbragstadi'm not sure i understand the amazon bit15:29
lbragstadamazon doesn't expose system-level apis in the way we do15:29
lbragstadalso - the On Behalf Of thing seems much more applicable to a private cloud deployment15:30
lbragstadif i have an account on AWS and i hose something up - that's my fault, and getting someone to come fix it is going to be hard15:30
lbragstadin theory, if i have a "reader" role on a domain, i should be able to get a domain-scoped token and use it to view all instances belonging to all projects in that domain15:33
raildolbragstad, ayoung ok, so the idea behind that ability to add a "project role" to a domain it was a way to make other projects handle with the domain stuff, like for example to set the quota for that domain in the other services15:33
raildo(which probably is not necessary any more with the unified limits api)15:34
raildoso, it was basically like, if I want to do domain actions in keystone with this domain, I got a domain scoped token but I'm looking for to do project stuff with that domain outside keystone, I'll request a project scoped token and deal with it as a project15:35
raildoand as ayoung said before, the difference is that that domain is a root project always, with no parent_id15:35
raildolbragstad, I'm gonna to take a look on that bug request and I hope to send a patch set for it soon15:36
lbragstadraildo: that'd be awesome15:36
lbragstadraildo: it's here if you haven't seen it yet -
openstackLaunchpad bug 1754677 in OpenStack Identity (keystone) "Unable to remove an assignment from domain and project" [High,Triaged] - Assigned to Raildo Mascena de Sousa Filho (raildo)15:36
lbragstadoh - nice15:36
lbragstadnevermind, it looks like you found it already15:37
raildolbragstad, I was faster that you :)15:37
raildolbragstad, also I like the idea to make a microversion and push all that stuff for that15:37
raildolbragstad, that would make easier to deal with all that stuff15:38
lbragstadyeah - it just popped up in my head as i was thinking about it... if we did do microversions, that'd be something we might be able to smooth out with it15:38
lbragstadthere was an action item last release to come up with a list of things like that15:38
raildolbragstad, ++ maybe we can discuss about it in the next ptg, or something like that15:38
lbragstadto help people weigh the pros and cons of microversions15:39
raildolbragstad, yeah, I'm fine with do everything to make the operators' life easier when we are talking about deploy and configure OpenStack15:40
ayounglbragstad, so, I think the short of it is, get unified limits working, and use that as a way to deal with access to the subtrees15:40
raildoayoung, yeah, that would be the best option at this moment15:42
*** AlexeyAbashkin has quit IRC15:45
lbragstadnice summary cmurphy15:49
ayoungBTW, cmurphy I didn't realize you were on the TC.  That is awesome, both for you, and for Keystone.  Shows how out of it I have been.  So let me formally congratulate you now.15:49
cmurphythanks ayoung :)15:50
lbragstadbug queue has bloated by about 15 new bugs in the last week15:57
cmurphyptg broke keystone15:58
* lbragstad laces up a fresh pair of bug stompers15:58
raildoso, on Keystone we got the bugs from the East15:59
*** marst has left #openstack-keystone16:01
*** pcaruana has quit IRC16:02
*** felipemonteiro has joined #openstack-keystone16:14
*** ioni has joined #openstack-keystone16:40
ionihello guys16:40
ionii operate a 5 region openstack newton with a single keystone. I want to upgrade from newton to ocata and I was wondering if newton components can still use keystone that is on ocata16:41
ionior keystone must be upgraded last16:41
ioniocata compoents and keystone newton16:42
*** spilla has joined #openstack-keystone16:48
*** masber has quit IRC16:54
*** Faster-Fanboi has quit IRC17:00
*** harlowja has joined #openstack-keystone17:00
mnaserhas anyone seen a case where keystone auth takes 150+ seconds..17:00
mnaser[Fri Mar  9 16:57:35 2018] POST /v3//auth/tokens => generated 11881 bytes in 144442 msecs (HTTP/1.1 201) 6 headers in 386 bytes (1 switches on core 0)17:01
*** germs has joined #openstack-keystone17:09
*** germs has quit IRC17:09
*** germs has joined #openstack-keystone17:09
*** germs has quit IRC17:14
ayoungioni, can you upgrade Keystone first?  You would probably be much happier17:23
ayoungioni, I assume you are doing something like an ansible/scripted deployment, not using Tripleo or the like.17:23
ayoungKeystone works hard to be backwards compatible, so you should be able to run an Ocata Keystone for a Newton cluster.  But, yes, I think the reverse would work.17:24
openstackgerritLance Bragstad proposed openstack/keystone master: Update context middleware and token models for system
openstackgerritLance Bragstad proposed openstack/keystone master: Add system scope project protection tests
lbragstadhrybacki: cmurphy ^17:27
lbragstada feeble attempt at getting system scope coverage with the three default roles we talked about in dublin17:28
lbragstadi feel like a lot of stuff is repeated17:28
lbragstadwhich kinda sucks, but i guess i was aiming for readability17:28
*** gyee has joined #openstack-keystone17:43
*** harlowja has quit IRC17:54
*** harlowja has joined #openstack-keystone17:56
*** harlowja has quit IRC18:00
*** germs has joined #openstack-keystone18:06
*** germs has quit IRC18:06
*** germs has joined #openstack-keystone18:06
*** germs has quit IRC18:07
*** germs has joined #openstack-keystone18:08
*** germs has quit IRC18:08
*** germs has joined #openstack-keystone18:08
*** dmellado has quit IRC18:16
*** oikiki has joined #openstack-keystone18:33
*** dmellado has joined #openstack-keystone18:42
*** itlinux has quit IRC18:50
*** harlowja has joined #openstack-keystone19:00
*** tesseract has quit IRC19:05
aningWe are doing upgrade from Newton to Pike, what's the consequce if we skip keystone-manage db_sync --contract, all or partially?19:16
aningthe reason is we ocasionally get deadloack at 01419:19
*** pcichy has quit IRC19:19
*** blake has joined #openstack-keystone19:27
openstackgerritLance Bragstad proposed openstack/keystone master: Add system scope project protection tests
lbragstadaning: are you running on postgres19:35
aningwe have postgres as the backend db19:40
*** oikiki has quit IRC19:48
*** oikiki has joined #openstack-keystone19:48
lbragstadcmurphy: was helping someone with that previously19:53
aninglbragstad: you mean he ever helped with the deadlock issue?19:54
aningcmurphy: is there a solution already for postgres deadlock issue during upgrade?20:01
lbragstadaning: she helped debug it a bit, iirc20:01
aninglbradstad, cmurphy: I have more info if you want to look at it again...20:02
aningwe've been struggling with this for a while ... so we are thinking to skip 014 where the deadlock happens.20:03
aningas a workaround.20:03
cmurphylbragstad: i don't think that was me, you might be thinking about the mariadb bug we had20:07
cmurphywas there a bug report?20:07
cmurphyaning: do you have the error message it gives when it locks up?20:10
aningYes ... one second.20:11
aning2018-02-22T21:26:47.000 controller-1 postgres[76959]: warning [2-1] db=keystone,user=admin-keystone ERROR:  deadlock detected20:12
aning2018-02-22T21:26:47.000 controller-1 postgres[76959]: warning [2-2] db=keystone,user=admin-keystone DETAIL:  Process 76959 waits for AccessExclusiveLock on relation 17886 of database 16401; blocked by process 76955.20:12
aning2018-02-22T21:26:47.000 controller-1 postgres[76959]: warning [2-3]     Process 76955 waits for AccessShareLock on relation 17776 of database 16401; blocked by process 76959.20:12
aning2018-02-22T21:26:47.000 controller-1 postgres[76959]: warning [2-4]     Process 76959: ALTER TABLE local_user ADD CONSTRAINT local_user_user_id_fkey FOREIGN KEY(user_id, domain_id) REFERENCES "user" (id, domain_id) ON DELETE CASCADE ON UPDATE CASCADE20:12
aning2018-02-22T21:26:47.000 controller-1 postgres[76959]: warning [2-5]     Process 76955: SELECT AS local_user_id, local_user.user_id AS local_user_user_id, local_user.domain_id AS local_user_domain_id, AS local_user_name, local_user.failed_auth_count AS local_user_failed_auth_count, local_user.failed_auth_at AS local_user_failed_auth_at, anon_1.user_id AS anon_1_user_id, anon_1.user_domain_id AS anon_1_user_domain_id20:12
aning2018-02-22T21:26:47.000 controller-1 postgres[76959]: warning [2-6]     FROM (SELECT "user".id AS user_id, "user".domain_id AS user_domain_id20:12
aning2018-02-22T21:26:47.000 controller-1 postgres[76959]: warning [2-7]     FROM "user"20:12
aning2018-02-22T21:26:47.000 controller-1 postgres[76959]: warning [2-8]     WHERE "user".id = '20bd4216910340bc8e6019f6d826f9d7') AS anon_1 JOIN local_user ON anon_1.user_id = local_user.user_id AND anon_1.user_domain_id = local_user.domain_id ORDER BY anon_1.user_id, anon_1.user_domain_id20:12
aning2018-02-22T21:26:47.000 controller-1 postgres[76959]: warning [2-9] db=keystone,user=admin-keystone HINT:  See server log for query details.20:12
aning2018-02-22T21:26:47.000 controller-1 postgres[76959]: warning [2-10] db=keystone,user=admin-keystone STATEMENT:  ALTER TABLE local_user ADD CONSTRAINT local_user_user_id_fkey FOREIGN KEY(user_id, domain_id) REFERENCES "user" (id, domain_id) ON DELETE CASCADE ON UPDATE CASCADE20:12
aningSorry, did I just flood the chat?20:12
cmurphyaning: yep, please use next time :)20:12
aningHow to use that? I'm rather new to IRC20:13
*** oikiki has quit IRC20:13
cmurphyaning: it's a website, go to to create the paste and then just drop the URL here :)20:13
*** oikiki has joined #openstack-keystone20:14
*** itlinux_ has joined #openstack-keystone20:14
aningHere it is:
lbragstadi've seen that error before20:21
*** threestrands has joined #openstack-keystone20:22
*** threestrands has quit IRC20:22
*** threestrands has joined #openstack-keystone20:22
aningWe are not alone:)20:22
lbragstadugh - where did i see that?20:22
lbragstadcmurphy: sorry - i got mixed up20:22
cmurphylbragstad: aning sorry i don't have any answers, maybe if lbragstad can figure out where we've seen that before we can get some hints20:24
* lbragstad digs20:24
knikollaaning: do you have multiple keystone servers?20:25
aningYes, we have two20:26
lbragstadit's a race bug iirc20:26
aninglbragstad: is it a reported bug already?20:27
knikollaaning: what is the exact procedure you're following?20:30
aningOh that's from us. kansim-wrs is our tech lead.20:32
lbragstadnevermind then20:32
*** itlinux_ has quit IRC20:35
*** felipemonteiro has quit IRC20:36
*** itlinux has joined #openstack-keystone20:38
lbragstadaning: were you able to make progress with the caching bit then?20:39
lbragstador shutting of the policy for list_users?20:39
lbragstadto circumvent the read operation causing the deadlock20:39
aningNo. we don't want to enalbe caching or changing policy for now.20:40
aningwe are thinking of skipping 014 as workaround20:40
aningSo what will be the consequence if we skip 014?20:41
aningknikolla: on controller0 (active) we upgrade controller1(standby) including db_sync --expand and --migrate, then we make controller1 as active, upgrade controller0 (now it's standby), then we call keystone-manage db_sync --contract on active controller120:41
*** raildo has quit IRC20:42
*** itlinux has quit IRC20:43
knikollaaning: and the contract at this very last step is causing the error20:44
aningand it happens only ocasionally.20:45
lbragstadit's a race condition20:45
knikollayes, only when there's a read going on. because the contract is requesting an ACCESS EXCLUSIVE lock, which also locks out reads.20:45
lbragstadwell - these two fks won't be created in that case -
aningwithout these two fks, my understanding is that, functional wise nothing will be broken, but there in very rare occations, the two keys may lose sync between user table and local_user table.20:48
knikollai think we do cascade deletes based on domain?20:49
cmurphyi think that's a case where the foreign keys are pretty important20:50
knikollai'm confused. is it even possible to support online upgrades if upgrading the database requires access exclusive locks, which also blocks reads :/20:54
aningknikolla: the CASCADE on DELETE will gurantee if we delete a row from one table, the referenced raw in the other table will be deleted as well?20:54
knikollaaning: yes. in that case between the table user and the table local_user, and nonlocal_user and user.21:02
openstackgerritayoung proposed openstack/keystone-specs master: Add whitelist-extension-for-app-creds
ayoungjgr, ^^ is just getting tox to pass21:02
aningwithout fk association between two table, potentially I could update the column in one table, leaving the  related column in the other table unchanged, am I right?21:03
aningthen the sync needs to be guruanteed by the user application, again am I right?21:05
*** felipemonteiro has joined #openstack-keystone21:05
adriantayoung, lbragstad: thanks for helping untangle that mess!21:06
adriantayoung: I read through the scoll back here and I see how the policy works. (domain_id vs project_domain_id)21:07
lbragstadaning: right - it would be possible to delete a domain and leave users orphaned in the system21:07
ayoungadriant, you are now the smartest person I know21:07
adriantugh at that policy, that's painful subtle enough to shoot someone in the foot21:07
lbragstadas are most things with keystone :)21:07
jgrayoung: ah, I had this feeling that I'd forgotten something when I hit `git review`...thanks :-)21:07
adriantayoung: I don't UNDERSTAND it, I'm not that bright :P21:08
knikollalbragstad: that's actually the foreign key that maps local_user, nonlocal_user to user.21:08
ayoungusually we give you a dull spoon with which to cut off your own foot.21:08
knikolladeleting domains shouldn't leave orphaned things.21:08
ayoungjgr, I am going through now, and going to suggest you define  two entities instead of one21:08
lbragstadknikolla: but deleting a user will leave fragments of that user in other tables, right?21:08
ayoungthe first is the common part:  Service, URL pattern, VERB21:09
knikollalbragstad: yes. and depending on how we handle the user delete, might result in 500 on user listing.21:09
adriantthat said, ayoung, regarding the alice example where I made a typo. The point wasn't the error, the fact that when I didn't make the typo just now that it worked is worse :P
ayoungthe second is the actual whitelist.  But I think that means the whitelist can be added to the app credential in an optional field21:09
adriantayoung: alice has roles in projects that are not her domain. is that intended?21:09
knikollabut if we delete the row in user first, the orphaned fragments should be innocuous.21:10
ayoungyes, that is fine21:10
ayoungdomains "own" the user record21:10
adriantcorrection, is allowing that intended?21:10
adriantOh, but not the scope21:10
ayoungrole assignemtns can cross domain boundaries21:10
knikollauser will be a subset of what's available. so it won't query about the orphaned rows.21:10
knikollaunless something weird is happening with joins.21:10
knikollastill i would argue against continuing without that migration.21:11
knikollathere be dragons.21:11
adriantayoung: thanks btw, apologises for potentially silly questions, it's just that trying to work out what is intended/unintended is a little weird sometimes.21:12
adriantbut is role assignments can be cross domain, and you can scope to other domains... that opens up some very interesting uses cases for me...21:13
adriantbut if*21:13
aningknikolla: so you don't think it's good idea of skipping the fk creation?21:15
knikollaaning: yes, bad idea.21:16
knikollaat best everything will work fine. at worst you're going to get server error when you list users.21:17
*** threestrands has quit IRC21:22
knikollawhen listing users, an outerjoin is done on tables user and local_user21:23
knikollathat'll probably break.21:24
ayoungadriant, no problem at all.  I really appreciate you taking the time to work through this.  Keystone is the better for your efforts.21:27
*** dtruong has quit IRC21:43
openstackgerritNicolas Helgeson proposed openstack/keystone master: Extend comparator support for project list by tags
*** mburrows has joined #openstack-keystone21:52
*** r-daneel has quit IRC21:56
*** awestin1 has quit IRC22:27
*** jamespage has quit IRC22:27
*** edmondsw has quit IRC22:27
*** jamespage has joined #openstack-keystone22:28
*** awestin1 has joined #openstack-keystone22:28
*** dave-mccowan has quit IRC23:09
*** martinus__ has quit IRC23:15
*** masber has joined #openstack-keystone23:27
*** masber has quit IRC23:30
*** masber has joined #openstack-keystone23:30
*** edmondsw has joined #openstack-keystone23:49
*** blake has quit IRC23:52
*** edmondsw has quit IRC23:54

Generated by 2.15.3 by Marius Gedminas - find it at!