18:02:54 #startmeeting keystone 18:02:54 Meeting started Tue Nov 10 18:02:54 2015 UTC and is due to finish in 60 minutes. The chair is stevemar_. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:02:55 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:02:57 The meeting name has been set to 'keystone' 18:03:01 stevemar_: courtesy ping o/ 18:03:06 i was looking at email, doh! 18:03:17 hehe 18:03:17 courtesy ping for ajayaa, amakarov, ayoung, breton, browne, davechen, david8hu, dolphm, dstanek, ericksonsantos, geoffarnold, gyee, henrynash, hogepodge, htruta, jamielennox, joesavak, lbragstad, lhcheng, marekd, morganfainberg, nkinder, raildo, rharwood, rodrigods, roxanaghe, samueldmq, shaleh, stevemar, tsymanczyk, topol, vivekd, wanghong, claudiub, rderose, samleon, xek 18:03:26 o/ 18:03:30 o/ 18:03:31 lots on the agenda today 18:03:32 \o 18:03:43 agenda: https://wiki.openstack.org/wiki/Meetings/KeystoneMeeting 18:03:45 #link https://wiki.openstack.org/wiki/Meetings/KeystoneMeeting 18:03:45 o/ 18:03:50 i never understood why you put that name list in -meeting rather than in -keystone 18:03:51 o/ 18:04:02 the point would be to get people to join -meeting 18:04:04 if we are here, why ping us 18:04:06 :-) 18:04:23 to wake you up from idling! 18:04:25 that's why I removed myself from the list. I'm already here. 18:04:28 pingbot 18:04:41 bknudson: you're a rebel like that 18:04:54 lbragstad: you're up 18:04:59 #topic Converting from MySQL's DATETIME to integers for timestamps 18:05:10 from* timestamps 18:05:17 alright, so this was something we discussed at the summit 18:05:27 and i ended up visiting a bit with ops about it 18:05:35 odyssey4me around? 18:05:36 looks like you have a patch already: https://review.openstack.org/#/c/243742/ 18:05:44 stevemar_ yep, so feel free to review away 18:05:56 stevemar_ addressing your comments. 18:06:07 this will get us the precision we need? 18:06:11 lbragstad, that a good oops or bad oops 18:06:18 can we not add new migrations in the extensions? move it to base first 18:06:22 gyee: OPs 18:06:39 I'd also like to have jaypipes take a look at it since from an "on-line" perspective 18:06:56 stevemar_ yep, it will give us the precision we need 18:07:19 bknudson: we can move the revoke stuff to the top of the 'extensions to core', or just review those patches i have up 18:07:30 lbragstad: sounds delightful 18:07:30 (sorry I’m late) 18:07:42 lbragstad: why use BigInteger instead of Integer? (just curious) 18:08:14 stevemar_ Integer isn't big enough to hold the integer timestamp that represents a microsecond precision int 18:08:22 lbragstad: neat 18:08:35 well, i don't have any issue with this, anyone else? 18:08:40 lbragstad: and zzzeek, too. 18:08:41 1447178915610956 18:08:51 stevemar_ that's an example ^ 18:08:58 jaypipes o/ 18:09:03 so, 18:09:19 this could also help people who want to use a non-MySQL setup, right? 18:09:22 we also use sql.DateTime in the trust schema 18:09:41 shaleh right now, the changes are specific to backends/sql.py 18:09:50 lbragstad: I mean like pg or something 18:09:51 it's kinda stupid that we have to work around mysql, but it is the main target so if we have to do it i'm good 18:10:25 http://stackoverflow.com/questions/7604966/maximum-and-minimum-values-for-ints 18:10:26 shaleh we would have to move the conversion logic up to models.py or core.py 18:10:34 jamielennox: yeah, we gotta get around it, so it's a necessary change 18:10:46 "Note that in Python 3 the int type is basically the same as the long type in Python 2" 18:11:00 lbragstad: well, it is a positive step forward 18:11:20 it's not really "working arond MySQL"... the MySQL DATETIME column type has *never* supported microseconds. 18:11:58 we also use sql.DateTime in trusts, but I figured this was the critical path for getting fernet to default 18:12:15 so one option is to not have microseconds 18:12:25 store seconds since epoch rather than microseconds 18:12:28 and we use it in the token table :( 18:12:32 jaypipes: ? as of 5.6.4 IIRC http://dev.mysql.com/doc/refman/5.7/en/fractional-seconds.html 18:12:36 * ayoung still catching up on Fernet stuff 18:13:15 since fernet has seconds-precision and apparently it's not causing problems. 18:13:18 https://github.com/openstack/keystone/blob/master/keystone/token/persistence/backends/sql.py#L37 18:13:35 bknudson well, we can't have sub-second precision in one and not the other. 18:13:38 bknudson: it's failing tempest, if that counts as a problem 18:14:03 if we introduce sub-second precision in sql we have to do the same for Fernet, upstream 18:14:11 trusts can happen later. Trust unit tests are not failing, and will not fail if we change the revocation table 18:14:12 otherwise we will hit race conditions 18:14:32 bknudson which is why i opened up - https://bugs.launchpad.net/keystone/+bug/1513541 18:14:32 Launchpad bug 1513541 in OpenStack Identity (keystone) "Support sub-second accuracy in Fernet's creation timestamp" [Undecided,New] 18:14:33 * dolphm has a pull request to github.com/fernet/spec to add microseconds 18:14:43 bknudson so we can track that, however it gets done. 18:14:55 reaperhulk seemed to be fine with adding the support in cryptography 18:15:03 but that's not really the problem 18:15:07 lbragstad, we are going to put the micros into a separate field, right? 18:15:11 we need to get a hold of the maintainers of the Fernet SPec 18:15:18 we need to ensure we use the DocImpact on those patches 18:15:18 ayoung that's the other option 18:15:26 I think we need to say that in the release notes 18:15:51 (precision of comparison for token revokation) 18:15:53 lbragstad, it was admittedly a hack, but considering how important fernet is,I think we need to streamline the implmentation to things we control only 18:16:08 I think we should wait until fernet either gets microsecond support or it's rejected. 18:16:08 revocation* 18:16:19 bknudson, no 18:16:27 bknudson, we can;t wait that long 18:16:28 well, I guess we can always fall back to seconds 18:16:35 bknudson: why the push back? 18:16:36 if we start with microseconds 18:16:45 if we support microseconds internally, we can always round in APIs and whatnot 18:16:58 that's a good point 18:16:59 (or truncate, whatever is more reasonable) 18:17:05 the envelope need to meet the fernet spec. The payload is ours to control. Checking revocation is also ours to contorl. 18:17:15 so...basically what dolphm just said 18:17:38 we'd need to be willing to make the fernet tokens bigger 18:18:00 lbragstad, are you doing the timestamp all in one column, or in two? 18:18:10 one for existing datetime, one for millis? 18:18:19 ayoung the integer that is replacing the sql.DateTime? 18:18:37 ayoung that's in one column 18:18:37 shaleh: I'm not pushing back. 18:18:44 lbragstad, yeah. I think it is OK to do that for revocation events 18:18:47 that is our API 18:18:58 ayoung cool 18:19:00 and then Fernet won't have a database table anyway...so we shoiuld be good 18:19:07 right 18:19:22 stevemar_ that's my update 18:19:28 lbragstad: :) 18:19:35 stevemar_ I can try and take an action item though for the Fernet Spec stuff? 18:19:41 or just trying to get a hold of those guys 18:19:48 bknudson: sorry if I misunderstood. It seemed to me like you were asking for this work to slow down and wait. 18:20:06 not sure if dolphm has made any progress on that front either, but it had been a while since we sync'd on it 18:20:13 lbragstad: i think proceed with the patch, and the plan you already had at the summit 18:20:17 shaleh: I am asking for the migration to be done after we've moved the extension to core. 18:20:31 no progress since summit; my branch is here https://github.com/dolph/fernet-spec#timestamp 18:20:43 bknudson: how much longer until those land? Plenty of things circling around that work. 18:20:45 dolphm cool, thanks 18:21:13 the last thing on the spec i need to work out is how to generate the negative test cases - not sure how to "correctly" produce the failing fernet tokens (there's no docs on how the current negative tests are really produced) 18:21:28 shaleh: could take forever, people have to review it... just like they'll have to review this new patch. 18:21:52 #action everyone review more 18:21:56 * stevemar_ cracks whip 18:21:58 stevemar_: ++ 18:22:16 bknudson: you're up next 18:22:19 #topic Prefer built-in xml rather than lxml 18:22:28 i reallllllly like this patch 18:22:33 so the proposal is to switch from lxml to built-in xml 18:22:38 stevemar_: ++ on reviews 18:22:41 for the adfs and saml2 auth plugins 18:22:43 in keystoenauth 18:22:44 for KSA: https://review.openstack.org/#/c/242512/ 18:22:51 me 2, but we have to be careful with namespaces 18:22:52 so that they don't have to be in "extras" 18:23:06 the signature depends on the namespace 18:23:14 ++ on less dependence on lxml. It ca be a pain to get installed. 18:24:06 so i don't really know how xml namespaces work, i thought they were like a shortener and that you could use any namespace values 18:24:09 if the tests are inadequate as far as covering namespaces then we should fix the tests. 18:24:33 we fixed a bug awhile back where the signature is invalidate when reloading the xml doc 18:24:35 I thought the tests were pretty extensive. 18:24:40 gyee or marekd can one of you try out the patch? 18:24:42 basically the namespace got regenerated in the wrong order 18:24:59 stevemar_, yes, I'll try it out before +2 it 18:25:20 gyee: cool, it would be awesome to have this in, and we don't have to use extras 18:25:29 jamielennox: I think the wrinkle is when the namespace is referenced in the value (rather than in the tag name) 18:26:08 i think marekd is away, but i'm sure he would be eager to try this 18:26:16 ok - if they're properly equivalent then i'm all for this 18:26:53 we'll have an awesome 0.6.0 release if that's the case 18:27:01 hell yeah 18:27:02 with kerb support too :) 18:27:18 and we can get rid of the extra repos 18:27:18 bah, who uses kerberos any more :-) 18:27:24 Heh 18:27:29 ^^ 18:27:36 kerb will still need extras 18:27:39 * ayoung assumes all kerberos barbs are directed at him 18:27:47 gyee: you're up 18:27:52 #topic Custom project ID at project creation 18:28:08 so we have a request to allow custom project ID at creation 18:28:12 stevemar_, so...there are two use cases for this. 18:28:12 jamielennox: the kerberos change is proposed here: https://review.openstack.org/#/c/240458/ -- it uses extras 18:28:20 hope you like scaling huge mountains 18:28:25 One is to synchronize between two OpenStack Keystone instances... 18:28:31 but author had indicated that they need this so they can maintain it out-of-tree 18:28:38 and the other is to re-enable a deleted project 18:28:48 I don't know if we have a policy for this sort of request 18:29:16 are we talking about more than 1 thing at the same time , 18:29:19 ? 18:29:22 * samueldmq is confused 18:29:23 I thought our policy was to just do everything and then have to maintain the code forever. 18:29:39 bknudson, so you are in support of this approach then? 18:29:41 bknudson: or longer 18:30:00 gyee: is there a patch up for custom project id ? 18:30:04 this particular request seems pretty easy 18:30:11 I had one. 18:30:12 for context, our motivation for this was right now we're running Juno and can't do federation 18:30:16 Just mark everything deprecated from the very beginning 18:30:20 gyee: it still has to be a UUID, right? The user just provides it? 18:30:21 samueldmq, yes 18:30:22 so our options are either database replication of hacking keystone 18:30:22 it seems easy, but then why would we allow it for project and not everything else 18:30:29 samueldmq, https://review.openstack.org/#/c/203852/ 18:30:35 project IDs don't have to be UUIDs 18:30:37 ayoung: thnks 18:30:41 project_ids are globally unqiue, you can't alwasys satisfy that condition with custom project_id. 18:30:58 haneef_, LDAP Assignment already broke that one 18:31:09 ryanpetrello: i don't understand how this prevents you from doing federationy things in Juno 18:31:17 But in this case, it would be an admin only API, used for migrations, sync, and the like 18:31:27 my understanding is that Juno keystone doesn't support replication 18:31:30 haneef_: the goal is to NOT have them globally unique (i.e. to replicte a project into another keystone) 18:31:32 not something exposed to ordinary users 18:31:36 we're working with clusters across different OpenStack clusters with distinct keystone installs 18:31:38 we could put it in keystone-manage project-create 18:31:41 haneef_: which ahs me worried 18:31:52 henrynash, they are separate clouds 18:31:53 right, what ayoung said ^ 18:32:04 bknudson: not a horrible idea 18:32:07 gyee: I know…. 18:32:15 henrynash: we can't replicate if there is already a project with that id 18:32:51 is there a way to "re-ID" a project? So if a conflict does occur it can be fixed? 18:32:59 Nope 18:33:01 bknudson, I like the keystone-manage approach 18:33:07 project ID is immutable, and needs to stay that way 18:33:07 haneef_: I don’t actually mind it for your use case….it is more the conept that has me worried…someone wmight use this in the wrong way nad really have two clashing project_ids 18:33:10 shaleh: and how does the use know that the id has changed ? 18:33:41 henrynash: putting it in keystone-manage is a nice way to indicate the purpose 18:34:05 samueldmq: agreed. Asking the questions to walk all the way around the problem. 18:34:15 shaleh: yeah, I agree that’s better 18:34:15 shaleh: ++ 18:34:44 with keystone-manage, we provide a reasonable workaround without having to change the API 18:34:57 i don't think we should allow this from the API, i have no real objection to having it in keystone-manage 18:35:01 do we have tooling to check for unique before the replication runs? Even with UUID there is the remote possibility of conflicts. 18:35:11 jamielennox: ++ 18:35:12 henrynash: the original design for id is, if id is uuid then it is immutable, and if it is then it is mutable. We will break that too 18:35:16 project id column should have a unique constraint 18:35:32 for what it's worth, it's not nearly as useful for us if it's not available via the API :/ 18:35:41 When you create an entry in the database, it will fail due to an uniqueness check. Is that not early enough? 18:35:57 Anything earliers would risk a race condition...but we are micro-optimizing 18:35:57 we avoided using PUT calls throughout our API to make room for this request eventually. PUT /v3/projects/{user_specified_project_id} can have it's own RBAC, and the new controller method can be called by the *old* controller method (which randomly assigns a project ID) 18:36:33 dolphm, ++ 18:36:48 the implementation is easy - understanding and accepting the consequences of user-specified, non-globally unique IDs is worth spending time on 18:36:48 i feel like this will open the door to other APIs having user defined ids 18:36:49 dolphm, yes, that'll work 18:37:15 stevemar_: definitely a concern 18:37:20 stevemar_: /v3/regions already does this basic pattern 18:37:30 would it not be possible to update the API with validation that just rejects invalid formatted uuids (and those that already exist)? 18:37:32 and the federation stuff too 18:37:33 dolphm: yes and that already created a ton of confusion 18:37:47 dolphm: though, thats mainly cause 'regions' are confusing 18:37:48 stevemar_: yep, we will probably endup by allowing everyhting to be customized (IDs) 18:38:33 ryanpetrello how does keystone determine what's invalid? 18:38:35 gyee: ryanpetrello are user defined IDs any string? or a UUID? 18:38:35 We should have been using FQDNs for unique resource naming anyway 18:38:47 stevemar_: I'm suggesting just a uuid 18:38:50 ryanpetrello configuration options? 18:38:55 right now keystone is calling uuid4() 18:38:58 stevemar_, yes, should be a string 18:39:03 just like UUID 18:39:06 seems reasonable to just allow the API to take a uuid4 string 18:39:09 and reject anything else 18:39:18 I'm speaking of the project creation API, that is 18:39:23 ryanpetrello, um...no 18:39:51 Migrations for people using Keystone LDAP Assignments would benefit from having the existing project IDs supported 18:40:10 ayoung: good point 18:40:22 if we accept > uuids we'll probably have to change the column type 18:40:47 ayoung: how does this work for re-creating a deleted project? 18:40:56 we'll have to limit the inputs in some way 18:41:11 stevemar_, you pass in the deleted-project-id and it is recreated 18:41:14 right now project.id is a varchar(64) 18:41:18 bknudson: yeah, no special weird characters 18:41:39 bknudson, yes, we need to limit the size and type 18:41:40 lbragstad, bknudson I think we can stick with that limitation for now. It is what the other proejcts use, so LDAP assignment would not work for longer values anyway 18:41:40 ayoung: but anything associated with the project is gone no? how does this help? 18:41:48 stevemar_, not in the remote services... 18:42:02 nothing is actually listening for Keystone events yet...excetp ceilometer. 18:42:07 from an operators perspective, it seems disjointed that i can specify a UUID for glance, but not for keystone. this change would make those two clients more in line 18:42:10 So this is a way an admin could clean things up remotely. 18:42:57 stevemar_: if the roles still exist the admin re-assigns them, right? 18:43:14 how many projects assume project_ids as uuid ( openstack client)? 18:43:33 shaleh, those would need to be reassiged; we clean them up on delete 18:43:42 haneef_, none of the others 18:43:54 haneef_, we had LDAP assignment for a few years now, and that did not do UUID 18:43:56 http://git.openstack.org/cgit/openstack/keystone/tree/keystone/resource/backends/sql.py#n249 -- id and name already have the same column type 18:44:01 well. did not force UUID 18:44:43 ayoung: but openstack client never worked for that case, since it assumes id as UUID 18:44:59 is there a reason we cannot fix the CLI? 18:45:06 I thought we worked around that.... 18:45:12 no need. the CLI is dead 18:45:18 really?!!! OSC specifically check for UUID? 18:45:22 could make the id checking a configurable regex, and default it to look for "uuid-ish", like ([a-f0-9])}{32} 18:45:25 ayoung: I mean OSC 18:45:39 dolphm ++ we wanted to move in that direction with keystone api validation layer 18:46:22 what happens if the project id is not globally unique? 18:46:45 that seems to be the most likely snag that people will hit 18:47:08 409 18:47:10 shaleh, nothing 18:47:11 shaleh: gyee we can always fix osc, that's not that big of an issue 18:47:20 we're expecting the project ID to not be globally unique... this is so you can have the same project ID in multiple clouds 18:47:22 globally meaning across two diffrerent OpenStack deployments? 18:47:24 stevemar_: agreed. 18:47:26 we're almost out of time 18:47:32 cuz otherwise, there is only "unique across Keystone" 18:47:36 shaleh henrynash gyee: so is specifying the project ID via an admin API totally out? because that's sort of the reason I created this spec in the first place; using keystone-manage doesn't really solve *our* issue :/ 18:47:40 henrys got 3 more items :( 18:48:12 so what's the verdict? PUT? 18:48:22 I'm fine with PUT 18:48:26 ryanpetrello: I think it’s just a big step…which is why we need to mull on it 18:48:31 gyee, the verdict is get a spec written and let us hash it out there 18:48:35 gyee: table the conversation for a sec, put up a spec for now 18:48:42 henrynash: ready to go? 18:48:47 #topic Let's create a new assignment driver (V9) 18:49:00 ok, so I had reason to do this! 18:49:13 I think I am the first to rev one of our driver API versiions 18:49:15 henrynash, Service catalog scoped roles? 18:49:19 my curent attempt: https://review.openstack.org/#/c/242853/ 18:49:27 dstanek: morgan? around? 18:49:48 we need to have tests that show that old code will work with the new driver 18:49:58 bknudson: yep, included those 18:50:31 basically - is what I have done what we had in mind? 18:50:46 cool 18:51:11 I’m ok if people want to review and come back to me….just want to get a collective view on how we do this, so we can copy this approach for other divers 18:51:36 what's in the V9 driver? 18:51:39 henrynash you'll have to put me in that camp, I haven't looked at it yet 18:52:06 bknudson: so we are removing a bnch of old methods now list_role_assignemnt is much more powerful 18:52:22 so we don’t duplicate processing of inheritance in multiple palces 18:52:30 oh, it's going to translate the old methods to list_role_assignments? 18:52:41 bknduson: yep 18:52:44 ok 18:53:06 stevemar:ok, let’s let people review it and comment on the patch 18:53:06 henrynash: overall i'm okay with the idea 18:53:13 henrynash: that's a great cleanup 18:53:23 great 18:53:30 #topic Enhancing list_role_assignments 18:53:35 Ok, so this is https://review.openstack.org/#/c/240466/ 18:53:38 i think this is less contentious 18:53:54 only question is: do we augment with just names or full entities 18:53:59 just: &include_names and just the names 18:54:10 gyee: I think you felt stronly on entities 18:54:17 i'm OK with just names 18:54:22 yes he is :) 18:54:25 we are not consistent with out APIs 18:54:37 in some cases we return the links but in others we don't 18:54:46 UX will suffer 18:55:04 we either going to do it consistently or we don't 18:55:29 unless there's a good reason for the inconsistency... 18:55:35 are links useful in this case? 18:55:46 i dont see what the whole reference buys us? 18:55:55 btw, I thought we were going to drop role IDs anyways. 18:55:56 the return is going to be HUGE 18:56:22 stevemar_: I guess that’s the question….is the additional large payload worth it? 18:56:39 I thought the whole point with links is that clients don't need to reconstruct anything, they just follow links 18:56:55 gyee: but as it stands now, no one uses links 18:57:08 stevemar_, they we should drop all the links 18:57:10 Morning 18:57:15 API consistency matters 18:57:18 gyee: Link is useless now as it always returns public endpoint 18:57:48 my point is we either going to consistently support it, or we don't 18:58:04 we should not be doing it halfway 18:58:09 gyee: i agree with your point, but picking that battle now, over this minor API change seems weird 18:58:26 stevemar_, like I said, that's not a deal breaker 18:58:40 so its not really a battle 18:58:41 gyee: you are more than welcome to clean up the APIs :) 18:58:46 more like bitching 18:58:53 hehe 18:58:59 dolphm: heh, learn something new every day... 18:59:03 dolphm: cheers 18:59:13 henrynash: i think you are all clear 18:59:15 jaypipes: o/ 18:59:18 we're at time 18:59:22 gyee, stevemar_: so sounds line include_name is OK for now…and maybe we add include_entities later if we celan things up 18:59:22 fwiw, I'd still recommend storing in a bigint anyway 18:59:24 pl 18:59:27 ok 18:59:29 #endmeeting