18:00:03 #startmeeting keystone 18:00:04 Meeting started Tue Aug 8 18:00:03 2017 UTC and is due to finish in 60 minutes. The chair is lbragstad. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:05 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:00:07 The meeting name has been set to 'keystone' 18:00:07 ping ayoung, breton, cmurphy, dstanek, edmondsw, gagehugo, henrynash, hrybacki, knikolla, lamt, lbragstad, lwanderley, notmorgan, rderose, rodrigods, samueldmq, spilla, aselius, dpar 18:00:17 #link https://etherpad.openstack.org/p/keystone-weekly-meeting 18:00:19 o/ 18:00:19 agenda ^ 18:00:19 o/ 18:00:21 o/ 18:00:21 o/ 18:00:22 hi o/ 18:01:01 o/ 18:01:35 #topic announcements 18:01:44 #info rc-1 is targeted for this week 18:02:10 #info planning for the PTG should be underway 18:02:37 i'm going to start going through the etherpad and organizing the items into like categories 18:02:45 I plan to start doing that as soon as we can cut rc1 18:02:52 lbragstad: are there any critical/high priority things for rc1? 18:03:00 i.e bugs 18:03:21 we do have one release blocker 18:03:33 https://bugs.launchpad.net/keystone/+bug/1705081 18:03:33 Launchpad bug 1705081 in OpenStack Identity (keystone) "DELETE project API is failing in forbidden(403) error message" [High,In progress] - Assigned to Lance Bragstad (lbragstad) 18:03:35 #link https://bugs.launchpad.net/keystone/+bug/1705081 18:03:49 and the fix for that should be gating 18:03:57 #link https://review.openstack.org/#/c/491546/ 18:04:23 #topic rc1 homestretch 18:04:39 since we were just talking about it 18:04:41 #link https://goo.gl/ygTNJF 18:05:15 lbragstad: what does that filter say? 18:05:17 #link https://bugs.launchpad.net/keystone/+bug/1705081 has a patch in review 18:05:17 Launchpad bug 1705081 in OpenStack Identity (keystone) "DELETE project API is failing in forbidden(403) error message" [High,In progress] - Assigned to Lance Bragstad (lbragstad) 18:05:43 samueldmq: that's a list of all bugs that are open and targeted to RC1 18:05:58 so - it's a list of things we need to complete before we can release a candidate for pike 18:06:07 ok seems a pretty decent amount of job for the office hours today? 18:06:09 :) 18:06:24 #link https://bugs.launchpad.net/keystone/+bug/1674676 is a doc fix - per dstanek's comment 18:06:24 Launchpad bug 1674676 in OpenStack Identity (keystone) "The URL listed against the details of identity resources returns 404 Not Found error" [Medium,Triaged] 18:06:30 ^ that'd be a good one to get gating today 18:06:50 anyone feel like picking that up today? 18:06:59 or submitting a doc patch to close that one? 18:08:16 *crickets* 18:08:20 Is that about those relationship links? 18:08:23 :) 18:08:27 yes 18:08:28 i've never really understood those relationship links 18:08:34 ^ 18:08:39 that's exactly why we need a docs patch 18:08:58 they are not suppose to resolve - so the point of the patch would be to document why they don't resolve 18:09:20 essentially consolidating https://bugs.launchpad.net/keystone/+bug/1674676/comments/3 18:09:20 Launchpad bug 1674676 in OpenStack Identity (keystone) "The URL listed against the details of identity resources returns 404 Not Found error" [Medium,Triaged] 18:09:20 me too, I tried working on the relationship links, but I'm not able to understand those 18:09:48 ahaa sjain_ ^ 18:09:59 oh 18:10:07 oops I'm late. yes that's explaining what those relationship URLs are for. we had that in our TODO 18:11:00 So we want to change the docs to described what those links really are then 18:11:21 gagehugo: yeah - according to dstanek, a blanket statement to the api-ref would suffice 18:11:38 just explaining what they are and why they're in the documentation 18:11:45 lbragstad I can tackle that then I think, shouldn't be too bad if that's just the case 18:11:47 that should be in the api-docs 18:11:52 since that's where those links appear 18:12:03 #action gagehugo to pickup https://bugs.launchpad.net/keystone/+bug/1674676 18:12:03 Launchpad bug 1674676 in OpenStack Identity (keystone) "The URL listed against the details of identity resources returns 404 Not Found error" [Medium,Triaged] 18:12:04 Put them wherever relationship links are imo 18:12:05 thanks gagehugo 18:12:24 next 18:12:26 #link https://bugs.launchpad.net/keystone/+bug/1689644 18:12:27 Launchpad bug 1689644 in OpenStack Identity (keystone) "Keystone does not report microversion headers" [Medium,In progress] - Assigned to Rohan Arora (ra271w) 18:12:32 that ^ one has a fix in review 18:12:46 #link https://review.openstack.org/#/c/468189/ 18:13:07 and it's moving along - it'd be nice to have that gating no later than tomorrow afternoon 18:13:15 so keystone doesn't report a microversion header - but we also don't have microversions 18:13:25 lbragstad I'll let those working on that know 18:13:26 so i was kind of confused about the point of that one 18:13:30 lbragstad: would be nice to have mordred looking at that review too 18:13:39 rarora ^ 18:13:44 and cmurphy who has been in sync with some microversion stuff 18:13:46 we support one microversion at a time 18:14:01 cmurphy: right - so the implementation needs to be able to take that into account 18:14:05 calling that a microversion is not really accurate 18:14:33 edmondsw: totally agree. 18:15:24 macroversion? 18:15:41 #link https://bugs.launchpad.net/keystone/+bug/1689644/comments/2 sums up the need pretty well 18:15:41 Launchpad bug 1689644 in OpenStack Identity (keystone) "Keystone does not report microversion headers" [Medium,In progress] - Assigned to Rohan Arora (ra271w) 18:16:15 well that comment is saying we should support microversions, that's not what this patch does 18:17:05 cmurphy: not really. that comment is saying we should report a microversion. so the microversion selection code in the client doesn't need a special case for keystone. 18:17:19 the patch does the bare minimum to allow clients to apply code evenly across openstack services without committing to microversions 18:18:08 i'm not sure the bug suggests an opinion on microversions one way or another 18:19:00 lbragstad that comment you referenced says "by not supporting microversions" not "by not returning microversion headers" 18:20:19 right 18:20:45 so do we want to pull this from rc1 queue until we discuss the possibility of microversions further? 18:21:48 I think that is appropriate. We don't support microversions as of today 18:22:07 because i don't expect to have a productive conversation on microversions until the ptg 18:23:26 I'm leery of starting to return a header that seems to indicate keystone supports microversions, when we don't 18:23:37 and wondering if that will bite us down the road 18:23:38 that's fair 18:23:40 +1 18:24:36 the tough part is that the microversion ship has mostly sailed 18:24:45 we're one of the few projects that haven't adopted it 18:24:55 but that's a discussion for the PTG 18:25:02 yep 18:25:05 pulling it from the queue 18:25:55 cc morgan ^ 18:27:16 i'm never going to be a fan nor support microversions 18:27:18 i wont block it 18:27:37 should I update the patch set's commit message or anything like that? 18:27:53 and keystone should NEVER emit a header indicating microversion support unless it does 18:28:23 that is a hard -2 on additional data. you can support microversions and indicate it, but you cannot emit data indicating support for it if there is no support 18:28:52 yep that's the concern 18:29:18 morgan I'd agree... and then you'll want to drop a -2 on https://review.openstack.org/#/c/468189/ 18:29:32 rarora: placed a procedural -2 on the patch for now so that we are aware of the status 18:29:50 lbragstad: ok sounds good 18:30:04 alright - moving on 18:30:09 #link https://bugs.launchpad.net/keystone/+bug/1694525 18:30:09 Launchpad bug 1694525 in OpenStack Identity (keystone) "keystone reports 404 User Not Found during grenade tests" [Medium,Triaged] 18:30:34 clarkb: ^ originally brought that to our attention a while ago and i targeted it to RC1 since it seemed to be affecting the gate quite a bit at the time 18:30:45 done 18:31:06 edmondsw: commented saying i'd remove the -2 when we support microversions 18:31:12 the logs from the occurrence have since expired and the logstash query is kind of useless since it returns a lot of false positives 18:31:19 morgan +! 18:31:21 ya I wasn't sure if it was a regression in upgrades or not as well 18:31:21 edmondsw: i wont +2/+1 microversion support but def. wont block implementation 18:31:33 (-1/-2 on that basis) 18:31:45 clarkb: it is a regression from previous releases as far as we know 18:32:36 well "user not found" isn't necessarily an indication of a problem 18:32:43 the way osc is designed it just does that 18:33:05 it could also be a test asserting something about the state of a user 18:33:27 well in this case it was failing to boot instances because the user being used went away in the upgrade iirc 18:33:41 I'd have to go reread things to remember properly though 18:34:18 lbragstad: likely 18:34:50 clarkb: do you want to do that in -keystone after the meeting? 18:34:59 we have office hours and it might be a good time to investigate 18:35:08 if your day is busy though, i understand 18:36:45 ya I'm currently unleaking nova instances in one of our clouds 18:36:56 it turns out nova can do that apparently so not sure if otday is a good day 18:37:10 clarkb: no worries - ping me if you end up wanting to go through it 18:37:27 otherwise i can try and find another occurrance of it 18:37:42 alright - next 18:37:51 #link https://bugs.launchpad.net/keystone/+bug/1700852 18:37:51 Launchpad bug 1700852 in OpenStack Identity (keystone) "Slow listing projects for user with many role assignments" [Medium,In progress] - Assigned to Lance Bragstad (lbragstad) 18:38:01 ^ that's a performance issue with effective role assignments 18:38:16 it seemed like you were waiting for feedback from the bug reporter on that one? 18:38:31 or do you want to un-wip it and get it in? 18:38:50 cmurphy: i think the patch i proposed is failing tests 18:38:58 which i need to investigate 18:39:15 so - it boils down to this 18:39:29 the role assignment api for listing effective role assignments is terribly slow 18:39:38 it's also super dense python logic 18:40:10 from the perspective of someone who hasn't spent a lot of time in the assignment API - it's pretty confusing 18:40:50 i suspect the problem is that we're doing some processing in python that is causing performance to degrade - specifically for effective assignemtn 18:41:01 my knee-jerk reaction was to implement caching for it 18:41:32 as a way to stop the bleeding until we can get more time to unwind and understand the role assignment API 18:41:56 and look for ways to improve performance by using/implementing better python 18:42:29 does anyone object to that approach? or should this wait until queens where we can do that investigation 18:42:42 so.. 18:42:45 I won't have time to unwind the entire role assignment api by the time rc1 rolls around 18:43:07 if we go for caching, we need to be really really careful. if we fail to invalidate in a single case (tehre are many) 18:43:08 caching sounds like a good start but i'm not sure it can get much better than that, it's basically a O(N^M) op because it has to map every user and every project 18:43:21 we'll be leaking roles to tokens, which is not fun 18:43:30 samueldmq: that's the other concerns 18:43:32 cmurphy: right 18:43:45 cmurphy: it's a lot of processing in python 18:43:59 with no real good way to offload it 18:44:07 lbragstad: yes it is. I've worked a lot in that code in the past 18:44:20 and that was per design, we decided to go and expand users into groups 18:44:35 and then we have project hierarchies, and then inherited role assingments 18:45:00 all that logic gets processed in those methods for role assignments, since all those mechanisms are for assigning roles anyways 18:45:23 yeah 18:45:29 question is, how bad is it? 18:45:32 * samueldmq looks at the bug 18:45:38 the performance numbers are in the bug 18:46:00 i'll see if i can get the patch passing tests and propose it 18:46:22 if we get too close to rc1 we'll cut the release with out it and backport the fix when we have it 18:46:43 (we backported a bunch of caching fixes in the mitaka timeframe for the same reasons when we dealt with fernet) 18:47:04 #action lbragstad to get https://review.openstack.org/#/c/487143/ passing tests 18:47:28 the next two seems related and have been causing issues in the gate 18:47:36 #link https://bugs.launchpad.net/keystone/+bug/1702211 18:47:36 Launchpad bug 1702211 in OpenStack Identity (keystone) "test_password_history_not_enforced_in_admin_reset failed in tempest test" [Medium,Confirmed] 18:47:43 #link https://bugs.launchpad.net/keystone/+bug/1703917 18:47:43 Launchpad bug 1703917 in OpenStack Identity (keystone) "Sometimes test_update_user_password fails with Unauthorized" [Medium,Triaged] 18:48:02 those mostly need investigation 18:48:19 Spent a day looking at them and came up short. 18:48:23 failed in a tempest test? 18:48:36 same here 18:48:42 when I see "sometimes" in a bug report, I feel the pain 18:48:52 seems like a race condition but i couldn't understand why 18:48:53 lbragstad: remember when we were trying to debug some fernet/tokens stuff 18:49:03 yeah - those are like that 18:49:24 i've reproduced the admin_reset one by running it over and over but for some reason not the regular one 18:50:07 cmurphy: sounds about par for the course, there was a bug i found acting similar and it required a sleep to repro... the one that you can repro should be the focus 18:50:23 the other one *shrug* best we can do is keep an eye on it if we can repro it 18:50:31 yeah 18:50:42 morgan: the sleep was because of the fernet weirdness, but i've seen it at least once with uuid 18:50:45 can't repro it* 18:50:55 cmurphy: same think in uuid at times. 18:51:07 just dealing with a much smaller window 18:51:09 cmurphy: sometimes ... things are racy and pain. 18:51:14 heh 18:51:54 ok - if we get to RC1 without understanding what's going on we'll cut it without fixes and backport them if we can 18:52:03 next 18:52:07 #link https://bugs.launchpad.net/keystone/+bug/1705072 18:52:07 Launchpad bug 1705072 in OpenStack Identity (keystone) "clearing default project_id from users using wrong driver implementation" [Medium,Triaged] 18:52:28 last i heard praskre was working on a fix for this 18:52:31 but i haven't seen anything 18:52:40 I had a fix for it. 1 line change 18:52:55 make the LDAP driver do a "pass" 18:53:08 samueldmq: this is a different bug i think 18:53:17 hm 18:53:18 this is if you have multiple domains and different identity backends for each 18:53:33 the default project id is only cleared for the default domain backend 18:53:38 it should be "if_readonly: pass" 18:53:44 not just "if ldap" 18:53:51 ldap driver explicitly says read-only 18:53:54 the logic to unset default project ids should be applied to all identity backends 18:53:56 so, focus on that. 18:54:05 morgan: that's being handled in a separate patch 18:54:18 this is specifically referencing multi-domain support 18:54:25 ++ 18:54:31 h,. 18:54:32 hm. 18:54:36 just do a for known backend, as opposed to the default one 18:54:59 if a project is deleted the default project id is only cleared for the default domain backend 18:55:02 you'd have to iterate 18:55:07 right 18:55:13 in as many words, default_project_id is a dumb construct 18:55:21 iterate and check if it's read_only or not 18:55:21 find all available drivers and call unset_default_project_id() on each 18:55:31 eh. 18:55:34 yeah 18:55:39 or just call them and let the driver decide if there is something to be done 18:55:49 that'd be better maybe. the driver knows the backend. 18:56:04 anyways, looks like we agree in a solution 18:56:25 anyone interested in proposing a fix and a test? 18:56:38 i haven't seen anything from praskre yet 18:57:29 i can take a stab at it 18:57:54 #action lbragstad to propose a fix for https://bugs.launchpad.net/keystone/+bug/1705072 18:57:54 Launchpad bug 1705072 in OpenStack Identity (keystone) "clearing default project_id from users using wrong driver implementation" [Medium,Triaged] 18:58:11 last one before the meeting is done 18:58:15 #link https://bugs.launchpad.net/keystone/+bug/1673157 18:58:15 Launchpad bug 1673157 in OpenStack Identity (keystone) "type: local must be set in order to get domain parsed when mapping federated users" [Low,In progress] - Assigned to Colleen Murphy (krinkle) 18:58:21 that one ^ has a fix in review 18:58:25 just needs another +2 18:58:30 it's a documentation fix 18:58:35 (thanks cmurphy) 18:59:11 #link https://review.openstack.org/#/c/491478/ 18:59:22 * mordred wves - has opened review from earlier 18:59:37 alright - we're out of time but we can continue this in office hours 18:59:49 mordred: we're headed to -keystone for office hours 18:59:57 #endmeeting