18:00:14 #startmeeting keystone 18:00:15 Meeting started Tue Sep 13 18:00:14 2016 UTC and is due to finish in 60 minutes. The chair is stevemar. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:16 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:00:18 The meeting name has been set to 'keystone' 18:00:20 hey everyone! 18:00:23 o/ 18:00:24 o/ 18:00:28 o/ 18:00:28 o/ 18:00:29 o/ 18:00:29 o/ 18:00:34 o/ 18:00:36 o/ 18:00:47 olá! o/ 18:01:08 o/ 18:01:12 topol: joining us today :) 18:01:20 o/ 18:01:20 henrynash is AFK, on a plane 18:01:28 no wifi? 18:01:31 o/ 18:01:37 gagehugo: excuses excuses eh 18:01:42 $10 for a flight ?! pshhh 18:02:00 #link agenda: https://etherpad.openstack.org/p/keystone-weekly-meeting 18:02:11 hi 18:02:15 hey bknudson 18:02:29 i think we're at a close enough consensus 18:02:39 we need ayoung and lbragstad eventually 18:02:40 quorum 18:02:45 o/ 18:02:48 I was here first 18:02:48 anteaya: tru tru 18:02:51 stevemar i'm here homes 18:02:53 ayoung: damn 18:02:59 i give up 18:03:06 o/ 18:03:08 stevemar need a nap? 18:03:08 2 laws of thermo 18:03:12 #topic project stuff 18:03:19 1. you can't win, best you can do is break even 18:03:25 2. you can't break even 18:03:40 if you're interested in serving as PTL, send a review request 18:03:59 i think to here: https://github.com/openstack/election 18:04:01 o/ 18:04:02 https://review.openstack.org/#/q/status:open+project:openstack/election 18:04:06 mailing list is also appreciated / helps advertise your candidacy, but it's not a required step anymore 18:04:13 dolphm: ++ 18:04:34 #topic newton release status 18:04:56 we'll be tagging RC1 this week 18:05:03 hi keystoners 18:05:03 so keep an eye out for critical bugs 18:05:32 where are we WRT KSA and KC releases relative to the Newton release? 18:05:33 looks like requirements is incorrect 18:05:39 so should get that fixed before release 18:05:39 ayoung: those are closed out 18:06:04 bknudson: requirements meaning the blocking out a specific ksm version? 18:06:13 i saw a bug for that this morning 18:06:13 yes. 18:06:21 bknudson: which requirements are incorrect? they're frozen now 18:06:27 bknudson: we can ask for that, but they are frozen 18:06:59 then we need to revert a bunch of changes that require a newer version of ksm 18:06:59 someone want to create a patch for that? and push it to requirements? 18:07:32 https://bugs.launchpad.net/keystone/+bug/1623091 18:07:34 Launchpad bug 1623091 in OpenStack Identity (keystone) "keystonemidleware dependency should be > 4.0.0" [Undecided,In progress] - Assigned to tamil vanan (tamilhce) 18:07:47 someone locked the python-requests-Kerberos version out, but only for Py3+ ...more KSA brokeness due to global deps 18:08:28 OK, i guess i'll submit a patch to requirements 18:09:07 bknudson: let's hope the requirements team allows it 18:09:36 dolphm: you opened bug 1623117 recently, is that needed? 18:09:37 bug 1623117 in OpenStack Identity (keystone) "Prevent keystone from serving requests when schema or data migrations are not up to date" [Medium,New] https://launchpad.net/bugs/1623117 18:09:42 probably won't like it since this will affect several projects 18:10:07 stevemar: not strictly, but it's something that if we have a fix ready for ocata, i'd like to slip it into the newton release IF we have an actual release blocker to warrant a respin / etc 18:10:36 I like that one 18:10:42 dolphm: fair enough, i'll mark it as rc-potential 18:10:49 stevemar: ++ 18:10:50 it will catch pain at a reportable point in the deploy process 18:10:53 ayoung: ++ 18:11:16 last one is bug 1621200 18:11:17 bug 1621200 in OpenStack Identity (keystone) "MySQLOpportunisticIdentityDriverTestCase.test_change_password fails in UTC+N timezone" [Undecided,In progress] https://launchpad.net/bugs/1621200 - Assigned to Ron De Rose (ronald-de-rose) 18:11:21 "It is never a keystone bug" 18:11:35 rderose and breton have been hacking away at it 18:11:44 whats the news there you two 18:12:11 I have fix that address that bug and another issue with the timestamp default 18:12:25 rderose: is the bug not reflective of the change? 18:12:34 cause i thought it was just a test issue 18:12:35 https://review.openstack.org/#/c/367374/ 18:13:01 stevemar: no, the other issue is that the created_at value may not be UTC with the default datetime 18:13:11 I'll update the bug to include that 18:13:56 rderose: yeah, i'll mark it as rc-potential, i'm not sure its a blocker 18:13:57 the patch is mostly ready, but we're having an issue with the migration tests running out of order 18:14:14 server_default=sqlalchemy.func.now() that is new to me 18:14:40 stevemar: dstanek is working on that issue. my patch will likely be based off of his fix 18:15:05 rderose: okay, we could probably slip it into rc2 if needed, i'll re-evaluate once you update the bug 18:15:23 stevemar: rderose: i have a much simpler fix 18:15:25 rderose: i'm not working on the timestamp issue at all. is that what you are talking about? 18:15:26 ayoung: recommended by zzzeek at the time, but problematic with some versions of mysql 18:15:44 stevemar: rderose: and actually i am not sure which bug rderose tries to fix :p 18:15:44 dstanek: no, tests running out of order issue 18:15:55 yeah, looks strange. Why but default is not sufficient because not-null needs a value ? 18:16:06 ayoung: pretty much 18:16:13 Toy 18:16:19 rderose: yeah, that i'm looking into 18:16:34 breton stevemar: https://bugs.launchpad.net/keystone/+bug/1621200 + the UTC issue 18:16:35 Launchpad bug 1621200 in OpenStack Identity (keystone) "MySQLOpportunisticIdentityDriverTestCase.test_change_password fails in UTC+N timezone" [Undecided,In progress] - Assigned to Ron De Rose (ronald-de-rose) 18:17:06 rule 1 of keystone: ALWAYS store/convert/workwith date time objects in UTC/TZ agnostic forms 18:17:13 rule 2 of keystone: see rule 1 18:17:26 i proposed https://review.openstack.org/#/c/367374/ and it worked for me, since i reported the bug :p 18:17:52 so maybe a new bugreport needs to be opened for what Ron's doing 18:18:02 rule 3: if you're relying on the system clock to be consistent in tests, you're doing it wrong. control the clock in tests directly 18:18:03 breton: I don't trust that it will work for all versions of mysql 18:18:05 maybe our unit tests could set a random timezone 18:18:22 notmorgan: agree 18:18:30 why is that not: server_default=datetime.datetime.utcnow 18:18:41 ayoung: because that is calculated once iirc 18:19:09 server_default is calculated on mysql server 18:19:11 ayoung: server_default=datetime.datetime.utcnow doesn't work for setting the server default 18:19:23 bknudson: ah wait that is passed to the server yeah no doesn'tmwork then 18:19:26 it's not python 18:19:34 usually i use something like server_default='CURRENT_TIMESTAMP' 18:19:47 but...wait 18:19:50 OK, no. 18:19:54 i don't think python code will work there 18:19:56 that is not what is meant by not null 18:20:00 so, uhm. this is a bad plan to set a datetime default at the server level 18:20:01 you might want to look at what sqlalchemy generates for the SQL 18:20:03 that is making a null value that is just trash 18:20:06 don't do that. 18:20:13 it says "the user can insert a row and if they don' 18:20:25 t provide a value, one will be provided for them" 18:20:25 this is going to make you sad because you're getting whatever TZ the server is in 18:20:29 which is not what we want 18:20:47 so...no there should not be a server_default 18:20:48 dstanek: ^ 18:20:53 dstanek: server_default='CURRENT_TIMESTAMP' will automatically update the timestamp column whenever any other column is updated 18:21:10 it should be a Not Null column, with a value required to be passed in when a record is created...am I wrong? 18:21:19 ayoung: or a server default can be set to a known bad value (0?) that we can directly check for. 18:21:25 but same argument 18:21:27 Or does SQL A somehow make it painful 18:21:35 you can't add a not null column without a default value 18:21:44 ayoung: my latest patch is removing the server_default 18:21:47 notmorgan, I don't want to "check" I want it to be required at dataload 18:21:51 rderose, excellent 18:21:53 maybe you can drop the default value after adding the column. 18:22:27 bknudson: unlikely 18:22:32 anyway...not for this meeting...drive on 18:22:38 shout if it is readyu for review 18:22:40 btw: the bug i reported happens to me only on unit tests 18:22:42 * ayoung adding myself 18:22:49 not in production 18:22:50 thank you ayoung 18:23:00 let's discuss the timezone thing in -keystone 18:23:01 breton: do you have a different timezone in production servers? 18:23:23 again, our code and tests are wrong if we rely on system TZ ever. 18:23:27 we should make sure we aren't 18:23:37 but carry on in -keystone after meeting 18:23:40 bknudson: my localhost production keystone doesn't suffer that :p 18:23:52 rderose: sounds like you need more investigation here 18:24:05 move to -keystone for this later, bah 18:24:09 #topic encrypted credentials update 18:24:22 so, while painful, this is not a stop ship. It sounds like a test only issue, and should be fixed in the tests 18:24:22 stevemar: will do 18:24:30 just a quick note that i wanted to give lbragstad a round of applause for saving our bacon last week 18:24:43 most of the patches are throug Tripleo, so I think I'm good on encrypted creds 18:24:58 tripleo / rdo was broken cause of the credential encryption work, and required key setup 18:25:04 of course, that is a brushoff to the rest of the installers 18:25:21 fwiw - we added support to grenade for a proper update 18:25:26 but lbragstad came up huge by including a null key if no fernet keys are seen 18:25:35 so thanks! :D 18:25:46 lbragstad: ++ 18:25:53 lbragstad: thanks 18:25:57 https://review.openstack.org/#/q/topic:keystone/credentials all merged and I wanted to keep lance's work quiet so they got through 18:25:59 more great work from lbragstad 18:26:08 it was dolphm's idea, i was just the gnome that did the finger work ;) 18:26:11 it made ayoung happy, you know it was hard! 18:26:30 well thanks to dolphm too :) 18:26:44 go lbragstad go 18:26:46 glad we got something worked out, thanks for all the reviews 18:26:52 Actually, forcint Tripleo to be able to support encrypted creds and fernet made me happy, I just didn't want to be too obvious about it 18:27:10 ayoung: :) 18:27:14 ayoung: you're up next 18:27:18 and possibly jamielennox 18:27:21 #topic Bug 968696 update 18:27:25 bug 968696 in Glance ""admin"-ness not properly scoped" [High,In progress] https://launchpad.net/bugs/968696 - Assigned to Sharat Sharma (sharat-sharma) 18:27:36 * stevemar cue the daunting music, DUM DUM DUM 18:27:44 So...according to Jamie, we stil have 2-3 services that are not compliant 18:27:49 968696 I know that number 18:27:53 i've lost track of what is left to do for this one 18:27:59 to get this bug fixed required changes to Keystone (long done) 18:28:02 then oslo-context 18:28:07 also long sinve done 18:28:13 stevemar: ++ 18:28:21 ayoung: all good so far :) 18:28:22 i think we have the majority of services not compliant 18:28:26 ayoung: could you give a quick recap in what's the plan ? what's missing ? 18:28:31 and then each of the services needed to load all of the value from oslo conf to policy check 18:28:42 all the oslo.context work is done, and most took on the helper functions that are needed 18:28:53 so there are patches out for nova nca cinder, and one not even written for neutron 18:29:06 bottom line, we are not going to have this closed out for Newton 18:29:22 the last step is to make them use context.to_policy_values() as the dict, however this is a bit more of a concern for most projects as we are changing the values that go into the policy engine 18:29:30 https://review.openstack.org/#/c/340195/ 18:29:32 and keeping compatibility there is harder than you would think 18:29:41 and https://review.openstack.org/#/c/341905/ are examples of what needs to happen 18:30:38 ayoung: jamielennox simple enough changes 18:31:06 ayoung & jamielennox what do you need from the keystone team, or is this striclty an update? 18:31:13 they are, but technically backwards incompatible because currently everyone drops context.to_dict into the enforcement 18:31:21 do you need some of us to patch other projects? 18:31:23 ah 18:31:39 jamielennox, why is is_admin_project not part of that? 18:31:44 can i help with that by allocating a fishbowl to this topic? 18:31:48 Yes 18:32:00 ayoung: is_admin_project comes from the base context object 18:32:11 stevemar: i've had a request to do an oslo.context as a cross-project already 18:32:23 stevemar: i need to follow up on that, just not sure with whom at the moment 18:32:48 I still don't get it 18:33:11 jamielennox: sounds good to me 18:33:14 do we need to inject is_admin_project into context.to_dict ? Why would that break anything? 18:33:45 ayoung: https://github.com/openstack/oslo.context/blob/master/oslo_context/context.py#L136 18:34:09 ayoung: no, i want to stop people using to_dict because it throws a whole lot of crap into policy enforcement that there is no reasonable way you would ever want to use 18:34:41 i just need to convince people it's ok to let that stuff go 18:34:43 ayoung & jamielennox thanks for the update 18:35:06 can we move on? 18:35:13 sure 18:35:25 #topic The identity v3 gate (non-voting) is broken 18:35:37 looks like raildo already volunteered to look at this 18:35:50 it's been failing for a month :( 18:35:53 this is the devstack job that does not run v2? 18:35:53 its a devstack job 18:36:06 dolphm: i believe so 18:36:27 yes, i`ve been working on it in the last months, and I want to help on it 18:36:28 having it not have teeth is the same as not testing. it will continue to break unless we get it working. 18:36:37 thanks raildo, i hadn't noticed this 18:36:49 and make it voting 18:36:51 jamielennox: np 18:37:09 dolphm: yes, it is 18:37:16 #link bug that is seen: https://bugs.launchpad.net/tempest/+bug/1620999 18:37:17 Launchpad bug 1620999 in tempest "gate-tempest-dsvm-neutron-identity-v3-only-full-nv 100% failure rate" [Undecided,New] 18:37:38 they are posting the removal since clarkb is moving it from trusty to xenial 18:37:44 postponing* 18:37:57 stevemar: ++ 18:38:05 as the goal is now to get it voting 18:38:11 we need to make sure they're stable 18:38:14 raildo_: i'll leave it to you to look at, i'll put it on next weeks agenda for an update 18:38:22 the problem is when the devstack try to get the image in glance, it`s not related to keystone v3 18:38:25 otherwise project teams won't want to add it as voting 18:38:39 stevemar: ok, thanks 18:38:53 right, the problem is this one gets broken because other teams don't configure for v3 18:39:01 raildo_: i suggest you see what went into glance around the time it started to fail 18:39:03 samueldmq: ++ the idea is to make this voting asap 18:39:03 ok, is this service catalog nonsense? 18:39:03 the intent was to get it voting to stop that from happening 18:39:05 raildo_: if it is not related to keystone v3, there is a proejct that CI is broken for 1+ month? 18:39:21 I notice that, at least in tripleo, we append versions to the URLs in the service catalog 18:39:29 so the identiyt one ends int /v2.0 18:39:47 ayoung: no one has any clue what the cause of this is, we're just guessing, no one has actually done any investigation 18:39:58 samueldmq: CI is broken because they couldn`t set the job properly, since this glance image issue 18:40:05 i'd rather not toss darts blindly, it was just a PSA for someone to look at it 18:40:11 raildo_: ok 18:40:30 samueldmq: but i have to investigate deeper 18:40:32 raildo_: I suggest to keep a very close eye on the jobs if we want to get them voting :) 18:40:35 raildo_: thank you! 18:40:43 raildo_: kk gotcha, thanks for working on it 18:40:46 we should make that job voting around summit time :) 18:40:47 dolphm: ++ thanks for looking at it raildo_ 18:40:51 samueldmq: i wil :) 18:41:08 devstack doesn't put /v2.0 on the identity endpoint anymore 18:41:18 YAY! 18:41:18 moving on, i like this next topic 18:41:21 dolphm: stevemar no problem, I hope to get it fixed asap 18:41:33 #topic Newton post-mortem 18:41:42 He was killed by an apple 18:41:51 the Neutron team is having one: https://review.openstack.org/#/c/360207/12 18:41:56 ba dum tsh 18:42:00 ayoung: thats actually pretty funny, +1 18:42:14 stevemar: ayoung: +1 18:42:20 i usually just clean up the spec repo, but i like this idea 18:42:25 ++ 18:42:34 We want to have it prior to the summit, I assume 18:42:47 i've always found it frustrating that we just skimp on docs, testing, and client changes, when we mark the BP for the server complete 18:43:13 i think having a post mortem would hold feet to the fire, before we continue implementing new features 18:43:34 identifying technical debt? 18:43:42 *cough* domain specific config support for keystoneclient 18:43:49 That was my original complaint about specs. I felt that that they should have bee written as docs, and had a lifespan accordingly 18:43:50 bknudson: is that what we're calling it now? :) 18:44:04 thoughts on trying this out? 18:44:13 ++ 18:44:20 +1 18:44:20 are there rules to the game? 18:44:29 ayoung: no rules, and the points don't count! 18:44:44 ayoung: honestly, none yet, i think we just want to identify where we missed 18:44:50 "Alex, I'll take lighting things on fire for 500" 18:44:58 I think of a postmortem as "what went well, what didn't go well" 18:45:00 and tack on names for things that missed 18:45:17 http://www.au.af.mil/au/awc/awcgate/army/tc_25-20/tc25-20.pdf 18:45:19 bknudson: i'm used to that definition too 18:45:42 what's missing is a task management system of some sort 18:45:49 bknudson: maybe port mortem isn't the right term here, but it would be "did we actually deliver on what we initially scoped" 18:45:54 dstanek: yeah 18:45:55 one thing that the AAR format suggests is a neutral 3rd party to moderate 18:45:57 we software engineer without the engineer... so do we just software? 18:46:23 OK, let me take this as a TODO and repropose this in a few days 18:46:27 ++ 18:46:31 i'll think of the outcomes I want to see 18:46:32 stevemar: ++ 18:46:37 dstanek please software responsibly 18:46:42 keep an eye on the mailing list 18:46:49 lbragstad: ++ 18:47:00 https://review.openstack.org/#/c/360207/12/specs/newton/postmortem/postmortem.rst -- here's the doc 18:47:23 #topic Mitaka status 18:47:57 we've got 2 different attempts to backport dstanek's cache fix 18:48:07 i am working on it 18:48:11 breton: awesome 18:48:12 and have only 2 tests failing 18:48:18 that sounds promising 18:48:24 a lot of things changed from mitaka to newton. 18:48:35 it's hard to duplicate awesome 18:48:57 ... which one is closer? 18:48:58 lol 18:49:29 we've got a few changes in stable/mitaka that haven't been released yet: https://github.com/openstack/keystone/compare/9.1.0...stable/mitaka 18:50:03 so it would be nice to get the cache fix in, and we release 9.2.0 18:50:21 mine i guess 18:50:35 since in a few weeks it'll be n-2 and only security fixes get merged :( 18:50:46 breton: cool, i'll review that one today then 18:50:52 thanks dstanek 18:51:19 todo, breton to code, dstanek tor review, stevemar to propose new release when ready 18:51:36 lbragstad: got 9 minutes! 18:51:42 #topic Update on making fernet the default 18:51:54 well - here it is 18:51:55 #link https://review.openstack.org/#/c/345688/19 18:52:02 i've seen a lot of action for this lately, thanks for picking this one up 18:52:16 that patch is dependent on everything needed to make fernet the default token provider 18:52:39 looks to be passing the transient issue we noticed the first time we made the switch 18:52:50 which appears to be a mysql rounding thing 18:53:05 I've doc'd that here - https://bugs.launchpad.net/keystone/+bug/1622010 18:53:06 Launchpad bug 1622010 in OpenStack Identity (keystone) "MySQL rounds timestamps" [High,In progress] - Assigned to Lance Bragstad (lbragstad) 18:53:17 lbragstad: you merged a fix for that (mysql rounding up the timestamp) ? 18:53:30 samueldmq i didn't merge it - i just proposed one solution 18:53:36 happy to see others though 18:53:43 lbragstad: ++ 18:53:49 so, other solutions could be : 18:53:51 Can we hack the tests in Tempest instead? 18:53:57 store subsecond precision in another column 18:54:12 "we never promised that a revoked token would be seen immeditatly, your tests are too aggresive" 18:54:33 ayoung: ++ 18:54:39 we couldn't promise that anyways due to caching 18:54:47 "revocations will not be visible for 5 minutes. Deal with it" 18:54:47 yep 18:54:49 or just have a long int column storing the time 18:54:50 ayoung the issue is when mysql would round a timestamp up 18:54:56 ayoung: i think that is fair 18:55:02 lbragstad: and that is an issue 18:55:15 samueldmq: if you go that route, it'd be far easier to just store a bigger integer in one column and divide it by 10^x to get the subsecond precision you want 18:55:51 dolphm: yes, just listed as the second option .. ^ have a long int column storing the time (from 1970s I think) 18:56:10 lbragstad: have you spoke with sdague and the qa team about this? 18:56:21 give them plenty of heads up 18:56:30 and we can be as precise as we want to be, without worring about rounding/truncating anything 18:56:44 stevemar about making fernet the default? 18:56:51 lbragstad: yes 18:56:55 stevemar I figured we would only consider that once ocata opens 18:57:05 lbragstad: that'll be in 2 weeks or so :P 18:57:10 but yes - we'll for sure need to give them a heads up 18:57:14 but OK, i guess it can wait 18:57:41 stevemar i have a list under the topic on the meeting etherpad asking who we need to update 18:57:44 or reach out to, 18:57:57 is anyone else aware of other groups that need to be on that list? 18:58:34 I think a cp thread would be nice 18:58:41 to let everyone know what our intentions are 18:58:52 talking with the the install guide folks will be an interesting talk 18:58:59 it does complicate setup 18:59:14 stevemar: ++ 18:59:20 can we do the same null-keys trick with fernet? 18:59:29 it's nice we have dedicated teams for those thigns 18:59:32 that would be dangerous 18:59:48 bknudson: you'll still be left with a bearer token 19:00:03 lbragstad: i'll add to the list if i think of more 19:00:10 lbragstad: probably RDO / tripleo 19:00:13 yes, you token is : role:admin,project:admin 19:00:14 #endmeeting