| *** EinstCrazy has joined #openstack-keystone | 00:03 | |
| * notmorgan continues on campaign to fix caching ick | 00:04 | |
| *** shoutm has joined #openstack-keystone | 00:05 | |
| *** EinstCrazy has quit IRC | 00:08 | |
| *** dims_ has joined #openstack-keystone | 00:19 | |
| *** shoutm_ has joined #openstack-keystone | 00:20 | |
| *** dims has quit IRC | 00:22 | |
| *** shoutm has quit IRC | 00:22 | |
| notmorgan | jamielennox: http://paste.openstack.org/show/484783/ the kwarg aware funciton_key_generator | 00:27 |
|---|---|---|
| notmorgan | and it has to be py2.6 compat so... dict(()) version of dict comprehensions | 00:28 |
| *** rcernin has quit IRC | 00:30 | |
| notmorgan | some minor fixes i've done since. but it's essentially ... the same | 00:31 |
| notmorgan | just changes the self detection | 00:31 |
| notmorgan | oh that is horrifying | 00:34 |
| notmorgan | the multi key thing is unlikely to ever work. | 00:34 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/keystone: Updating sample configuration file https://review.openstack.org/269479 | 00:34 |
| *** dslev has joined #openstack-keystone | 00:34 | |
| *** boris-42 has joined #openstack-keystone | 00:39 | |
| *** simondodsley has quit IRC | 00:48 | |
| *** lhcheng has quit IRC | 00:56 | |
| *** henrynash has joined #openstack-keystone | 01:00 | |
| *** ChanServ sets mode: +v henrynash | 01:00 | |
| *** dims_ has quit IRC | 01:00 | |
| *** dslev has quit IRC | 01:05 | |
| *** lhcheng has joined #openstack-keystone | 01:15 | |
| *** ChanServ sets mode: +v lhcheng | 01:15 | |
| openstackgerrit | henry-nash proposed openstack/keystone: Verify project unique constraints for projects acting as domains https://review.openstack.org/158372 | 01:16 |
| *** shoutm_ has quit IRC | 01:19 | |
| *** shoutm has joined #openstack-keystone | 01:20 | |
| *** shoutm_ has joined #openstack-keystone | 01:27 | |
| *** dims has joined #openstack-keystone | 01:27 | |
| *** dims has quit IRC | 01:27 | |
| *** shoutm has quit IRC | 01:28 | |
| *** davechen1 has joined #openstack-keystone | 01:31 | |
| *** hogepodge has joined #openstack-keystone | 01:34 | |
| notmorgan | stevemar: we should deprecate the current template driver (catalog) and do a quick implementation of a YAML or JSON based templated thing | 01:35 |
| notmorgan | stevemar: so people can still do CMS managed disk-based catalog just not the horrible templated driver we have | 01:35 |
| *** davechen has joined #openstack-keystone | 01:36 | |
| *** davechen1 has quit IRC | 01:38 | |
| jamielennox | notmorgan: i essentially volunteered for this at summit | 01:38 |
| *** hogepodge has quit IRC | 01:38 | |
| *** markvoelker has joined #openstack-keystone | 01:42 | |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone: Make AuthContext depend on auth_token middleware https://review.openstack.org/255686 | 01:44 |
| *** boltR has quit IRC | 01:45 | |
| *** boltR has joined #openstack-keystone | 01:46 | |
| *** markvoelker has quit IRC | 01:46 | |
| notmorgan | jamielennox: so what is needed for ksm to use oslo.context properly and keystone to work with that now? | 01:57 |
| notmorgan | jamielennox: cause, i would like to see that soooooon | 01:57 |
| jamielennox | notmorgan: so ksm doesn't have much to do with it, ksm is a producer | 01:57 |
| jamielennox | oslo.context is created based on the output of auth_token | 01:58 |
| notmorgan | right. we just want to make sure we're getting an oslo.context in keystone | 01:58 |
| jamielennox | keystone integration with auth_token is ready to go now that ksm has had a release | 01:58 |
| notmorgan | and if that is relying on ksm, what is needed? | 01:58 |
| jamielennox | oslo.context hasn't released the from_environ patch afaik | 01:58 |
| notmorgan | right, did a rebase/waiting for check | 01:58 |
| notmorgan | when did it land? | 01:58 |
| jamielennox | not long | 01:58 |
| notmorgan | like since mitaka-2? | 01:58 |
| notmorgan | and we should explicitly ask for a release if they'd be open to it so we can unblock our work | 01:59 |
| notmorgan | i think it is a reaosnable request "hey this is super duper important for us" | 01:59 |
| jamielennox | notmorgan: it's not a blocker, it's a helper function we can rely on when it's there | 01:59 |
| notmorgan | ok | 01:59 |
| *** Ephur has quit IRC | 01:59 | |
| jamielennox | we already construct an oslo.context in keystone | 01:59 |
| notmorgan | so if it isn't there can we still get to the oslo_context stuff? | 01:59 |
| notmorgan | ahhhh when did that change? | 01:59 |
| jamielennox | we have to to do logging and notifications | 01:59 |
| jamielennox | but there's nothing very useful in it | 01:59 |
| notmorgan | oh neato | 01:59 |
| notmorgan | hey, then i can already leverate it | 02:00 |
| notmorgan | without anything else | 02:00 |
| notmorgan | since there is a thread.local, right? | 02:00 |
| notmorgan | leverage it* | 02:00 |
| jamielennox | from memory you would need to fill it | 02:00 |
| notmorgan | i expect i need to anyway | 02:00 |
| jamielennox | https://review.openstack.org/#/c/255686/5/keystone/middleware/auth.py L160 | 02:01 |
| notmorgan | ok hmm. | 02:01 |
| notmorgan | so i don't care about the other data in the oslo.context | 02:01 |
| notmorgan | just so i have a well-defined place to stash these lookups | 02:02 |
| jamielennox | i wasn't thinking of using the threadlocal part of oslo.context - but i wouldn't really object, i just don't particularly like thread locals :) | 02:02 |
| jamielennox | notmorgan: you should care about the other data | 02:02 |
| notmorgan | jamielennox: not in what i'm looking to do ;) | 02:02 |
| notmorgan | jamielennox: the other data is useful for otherthings | 02:02 |
| jamielennox | notmorgan: context.user == user_id etc so you can use that instead of doing lookups | 02:02 |
| notmorgan | right, except we still need to do a user_lookup one time | 02:03 |
| jamielennox | notmorgan: nope | 02:03 |
| notmorgan | and if it is populated via the manager ever [inc. from building the context] my data will be populated as verified | 02:03 |
| jamielennox | notmorgan: auth_token middleware handles token validation, that's it | 02:03 |
| notmorgan | so if the ATM does the manager call [i assume it does in this case?] | 02:04 |
| notmorgan | it would still populate the data the way i want to do this | 02:04 |
| notmorgan | i want to hook into dogpile so any call down populates the lookup for us. | 02:04 |
| notmorgan | and it's effectively a 2-layer cache | 02:04 |
| notmorgan | thread.local cache and the shared-cache | 02:04 |
| notmorgan | the thread.local cache is for this request, don't ever ask the driver again, or memcache, or anything | 02:05 |
| notmorgan | the shared cache is the typical caching layer | 02:05 |
| jamielennox | notmorgan: i would subclass oslo.context's twice | 02:05 |
| jamielennox | one for CRUD where you are given a token, one for token creation where you construct a token | 02:06 |
| notmorgan | oh sure. | 02:06 |
| jamielennox | the CRUD should be exactly the same as everyone else | 02:06 |
| notmorgan | one from authenticate and one for authenticate_for_token | 02:06 |
| jamielennox | if you want to do fancy cache integration for token creation then whatever | 02:06 |
| notmorgan | or whatever it is | 02:06 |
| * jamielennox would still like to see those two services logically split | 02:07 | |
| notmorgan | right | 02:07 |
| notmorgan | i have some ideas on some helper funcs we can add to make that more explicit | 02:07 |
| notmorgan | rather than .get_user()['id'] all the time we make a .get_user_id() which is smart enough to look in the context if it's there. | 02:07 |
| notmorgan | and then passes through and down, if it hits the backend query, it would populate the user_data anyway | 02:08 |
| jamielennox | i think you've got that backwards | 02:08 |
| jamielennox | you should always ask the context and let it fill it in if not present | 02:08 |
| notmorgan | how does the context fill it in ? .get_user()['id']? | 02:09 |
| notmorgan | bad example cause we use user_ids. | 02:09 |
| notmorgan | user_project_id | 02:09 |
| notmorgan | it still has to talk to the manager | 02:09 |
| notmorgan | at some point | 02:09 |
| jamielennox | hmm, maybe for token creation that's not valid | 02:10 |
| notmorgan | the way i see it is we make a helper function that knows how to look this up, if it can't find it it just hits the backend, and that populates the cache | 02:10 |
| notmorgan | and subsequent .get_user() for any data would be a local-in-mem cache hit | 02:10 |
| notmorgan | if it is in context, we don't even need to worry about the cache hit | 02:10 |
| notmorgan | that way asking for id becomes more explicit. | 02:11 |
| notmorgan | and because we have cache hits we can expand the methods to be less "ask for the SQLA thing" and "ask for the element we care about" | 02:11 |
| notmorgan | the manager can handle that logic | 02:11 |
| notmorgan | and we'll always look to our request-specific cache first or where appropriate the context from auth_middleware | 02:12 |
| jamielennox | i'd prefer not to mix those concepts | 02:12 |
| notmorgan | so for example: .get_user_project_id() - if in_context, return, if not, context populates it, but also populates a local cache so all future "give me user_project_thing" requests hits the request local cache | 02:13 |
| notmorgan | for the life of the request | 02:13 |
| jamielennox | construction can be/has to be a special case, but there's no reason you should need to ask the manager for a project_id when you've been through auth_token middleware | 02:13 |
| * notmorgan doesn't care if the "get_useR_project" is a functionl of the context or not | 02:13 | |
| notmorgan | it can always be ask the context and the context will ask the manager if needed | 02:14 |
| notmorgan | but there are other things we do, operations that involve not-my-project and not-my-domain etc, it can always be cached in the same way under the hood. | 02:15 |
| notmorgan | so maybe what we need is 3 subclasses? | 02:16 |
| notmorgan | base[with cache] | 02:16 |
| notmorgan | token_constructor | 02:16 |
| notmorgan | and CRUD | 02:16 |
| jamielennox | notmorgan: yep we will always need to do some of that - but those calls never happen in a token construct right? | 02:16 |
| notmorgan | well in token construct they do, we do a lot of "look to see if X thing is there" | 02:16 |
| notmorgan | over and over | 02:16 |
| jamielennox | you will always need a way to do get_project() for listing or whatever | 02:17 |
| notmorgan | right | 02:17 |
| notmorgan | so, i think the cache concept makes sense regardless | 02:17 |
| notmorgan | some times you'll ask the manager, sometimes you'll ask the context. | 02:17 |
| jamielennox | yea, it's just figuring out a pattern | 02:17 |
| jamielennox | context is always current user, manager is always give me general info | 02:17 |
| notmorgan | so my thought is lets cache it all, then worry about asking context appropriatelyu | 02:17 |
| jamielennox | cache goes behind that regardless | 02:17 |
| notmorgan | yeah | 02:18 |
| notmorgan | exactly | 02:18 |
| notmorgan | so if your thing lands i can do my thing pretty low-cost ways | 02:18 |
| jamielennox | notmorgan: we can do the thread local stuff anyway as oslo.cache is already there | 02:19 |
| notmorgan | just build a dogpile proxy and always apply it and then make it so disable simply always returns NO_VALUE from the backend so regeneration always occurs | 02:19 |
| notmorgan | oh we already have it? | 02:19 |
| jamielennox | we would need buy in on a blueprint to use that object though rather than the current context dict | 02:19 |
| jamielennox | notmorgan: https://github.com/openstack/keystone/blob/master/keystone/middleware/auth.py#L147 | 02:19 |
| jamielennox | so defined for all CRUD operations | 02:20 |
| notmorgan | jamielennox: for all requests | 02:20 |
| notmorgan | even auth | 02:20 |
| notmorgan | right? | 02:20 |
| notmorgan | does that mean the thread.local thing is there? because it means i can do a speculative change to implement this immediately for testing performance gains | 02:21 |
| jamielennox | notmorgan: sorry yes | 02:21 |
| notmorgan | fantastic | 02:21 |
| notmorgan | so i don't need to change anything except make a subclass that i can store in and access from | 02:21 |
| notmorgan | sweet. | 02:21 |
| notmorgan | i am going to bet this is going to be a fairly large improvement | 02:22 |
| notmorgan | i'm just going to need to be *very* sure we GC the request correctly | 02:22 |
| notmorgan | so we don't memleak | 02:22 |
| jamielennox | notmorgan: it will be overriden on each request | 02:23 |
| notmorgan | jamielennox: right, but it's about how refs are stacked/held onto | 02:23 |
| notmorgan | if we're not careful we might not be able to GC out of the context even as a thread.local | 02:23 |
| jamielennox | notmorgan: if at all possible don't store objects on the request | 02:24 |
| jamielennox | store basic types | 02:24 |
| notmorgan | it'll be dicts mostly | 02:24 |
| jamielennox | i had another review that got reverted | 02:24 |
| notmorgan | since that is what we use outside of the SQL-A backends | 02:24 |
| notmorgan | i just need to be sure i don't do something silly | 02:25 |
| notmorgan | we don't pass SQL-A objects up beyond the drivers unless we have a bug. | 02:25 |
| jamielennox | notmorgan: https://review.openstack.org/#/c/264021/ got reverted because heat does silly things | 02:25 |
| notmorgan | so it should always be basic types | 02:25 |
| notmorgan | well python-native | 02:25 |
| jamielennox | notmorgan: but in general that's how contexts are used in projects and it'd be good if we stick to that | 02:26 |
| jamielennox | so don't save user = {'domain_id': 'xyz', name: 'abc', ... } | 02:26 |
| notmorgan | the only issue i see is we don't know the flattend values people want | 02:26 |
| notmorgan | yeah that isn't going to work for what i need | 02:26 |
| jamielennox | save context.user_name = 'abc' | 02:26 |
| jamielennox | context.user_id = 'def' | 02:26 |
| jamielennox | user_domain_id = .. | 02:27 |
| notmorgan | unless we make the managers always ask for user_name not user['name'] | 02:27 |
| jamielennox | etc | 02:27 |
| notmorgan | i can probably write someting to flatten the dict out and do some magic to make it look like a dict | 02:27 |
| notmorgan | so get_user() can reconstruct from flattend properties | 02:28 |
| jamielennox | notmorgan: is it mostly because of extras? | 02:28 |
| notmorgan | no because we say identity_api.get_user('id') | 02:28 |
| notmorgan | then we act on the dict | 02:28 |
| notmorgan | and check verisou things | 02:28 |
| notmorgan | but i can do this by making properties like <calculated_cache_key>_<prop_name> | 02:29 |
| notmorgan | and then reconstruct it to a dict on .get | 02:29 |
| notmorgan | or i can just store the native dict | 02:29 |
| jamielennox | i don't think that's going to be a problem | 02:29 |
| notmorgan | it actually will be pretty straightforward code in either case | 02:30 |
| notmorgan | so should i flatten the dicts down to properties? | 02:31 |
| notmorgan | or should i just store the dict | 02:31 |
| jamielennox | notmorgan: i think it is two seperate problems | 02:33 |
| jamielennox | context is always the current user, what you add to that should be flattened but you should only need basic properties about the current uesr | 02:33 |
| jamielennox | you will still have the dict available for token building etc | 02:33 |
| *** EinstCrazy has joined #openstack-keystone | 02:33 | |
| notmorgan | jamielennox: i'm trying to offload the calls to the backends | 02:33 |
| notmorgan | as well | 02:33 |
| notmorgan | so we stop saying .get_user() many times | 02:33 |
| notmorgan | or .get_domain() just to see if it's around | 02:34 |
| notmorgan | like .get_project calls .get_domain | 02:34 |
| notmorgan | why not store the result of .get_domain() so all subsequent calls know it's valid | 02:34 |
| notmorgan | we only need to check once really unless there is an explicit invalidate on the current request | 02:34 |
| notmorgan | or a global invalidate on the region (role_assignments, catalog) | 02:35 |
| notmorgan | the only difference is i'm storing the result in the context for the life of the request | 02:35 |
| jamielennox | i think we shouldn't be checking that in the drivers | 02:35 |
| notmorgan | it's not in the driver | 02:35 |
| notmorgan | it's in the manager | 02:35 |
| jamielennox | and in the managers we should be smart enough to know that it's already been checked | 02:35 |
| jamielennox | but i know we don't | 02:35 |
| notmorgan | that is wht i'm trying to do | 02:35 |
| notmorgan | ;) | 02:35 |
| notmorgan | but rather than needing to always make the manager "smart enough" | 02:36 |
| notmorgan | we cache things. why not leverage the caching layer that already knows how to do this | 02:36 |
| notmorgan | it also limits the volume of code change to say "oh uh... we checked this already" | 02:36 |
| notmorgan | we can improve the managers but i don't know where you're going to stash the "this was already checked" and we still have cases where we use the resulting data from the backend multiple times | 02:37 |
| notmorgan | in subsequent calls | 02:37 |
| notmorgan | here let me run my errand and spin up a change tonight to try this out | 02:38 |
| *** EinstCrazy has quit IRC | 02:40 | |
| *** shoutm_ has quit IRC | 02:41 | |
| jamielennox | notmorgan: yea, i think try it out | 02:45 |
| notmorgan | jamielennox: so first pass, flatten the returns out or store the dict? i am thinking flatten so there is no way someone can accidently "modify" the stored data | 02:46 |
| jamielennox | notmorgan: i think for arbitrary data (not current user) there shouldn't be much advantage in the context | 02:47 |
| notmorgan | jamielennox: i think you'll be surprised. | 02:48 |
| jamielennox | notmorgan: i don't mind if you keep dicts, just that i want to do something like that review i showed so assume that context.user and context.project is the string id | 02:48 |
| jamielennox | you can do whatever you like beyond that | 02:48 |
| notmorgan | oh right, this is stuff that should never ever be touched outside of this code i'm writing | 02:48 |
| notmorgan | all 100% internal | 02:48 |
| jamielennox | i don't like reconstituting dicts from flattened attributes | 02:48 |
| notmorgan | my only concern is if someone does a .get_domain then sets a value on it | 02:49 |
| *** shoutm has joined #openstack-keystone | 02:49 | |
| notmorgan | and i want to avoid needing to copy.deepcopy a lot | 02:49 |
| notmorgan | though that might be less expensive than flatten/unflatten | 02:50 |
| jamielennox | i think the deepcopy will be cheaper | 02:51 |
| notmorgan | well i can start there. it's def simplier code | 02:52 |
| *** shoutm has quit IRC | 02:56 | |
| ayoung | jamielennox, please give https://review.openstack.org/#/c/242614/ your blessing, or at least rage at it again. There is a long line of Henry's work that is based on it, and I am worried about that stuff geting dropped because no one can review it until late | 02:58 |
| ayoung | this is maybe imperfect, but should be good enough | 02:58 |
| ayoung | and I would not like to break the revision record on this review if possible | 02:59 |
| ayoung | getting close | 02:59 |
| *** shoutm has joined #openstack-keystone | 03:00 | |
| *** davechen1 has joined #openstack-keystone | 03:25 | |
| *** davechen has quit IRC | 03:27 | |
| notmorgan | jamielennox: ... so quick first pass, this looks like it's speeding up tests like 50+ | 03:28 |
| notmorgan | jamielennox: some failures but things are happening much faster | 03:28 |
| notmorgan | jamielennox: visual guessing though | 03:29 |
| notmorgan | 50%+ that is | 03:29 |
| notmorgan | i think i have an isolation bug though.. need to do an explicit clear of the cache in core test case(s) | 03:30 |
| *** richm has joined #openstack-keystone | 03:32 | |
| *** davechen1 is now known as davechen | 03:40 | |
| *** markvoelker has joined #openstack-keystone | 03:43 | |
| *** markvoelker has quit IRC | 03:47 | |
| notmorgan | bah isolation bug was speeding things up | 03:54 |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone: WIP/DNM: Use requst local caching https://review.openstack.org/271885 | 04:00 |
| *** dslev has joined #openstack-keystone | 04:01 | |
| openstackgerrit | Dave Chen proposed openstack/keystone: Consolidate `test_contrib_ec2.py` into `test_credential.py` https://review.openstack.org/271886 | 04:04 |
| openstackgerrit | Dave Chen proposed openstack/keystone: Consolidate `test_contrib_ec2.py` into `test_credential.py` https://review.openstack.org/271886 | 04:06 |
| *** shoutm_ has joined #openstack-keystone | 04:06 | |
| *** dslev has quit IRC | 04:06 | |
| *** shoutm has quit IRC | 04:09 | |
| *** shoutm has joined #openstack-keystone | 04:10 | |
| *** shoutm_ has quit IRC | 04:11 | |
| *** richm has quit IRC | 04:13 | |
| openstackgerrit | Merged openstack/keystoneauth: Use positional library instead of our own copy https://review.openstack.org/267300 | 04:17 |
| *** shoutm has quit IRC | 04:28 | |
| *** shoutm has joined #openstack-keystone | 04:30 | |
| openstackgerrit | henry-nash proposed openstack/keystone: Projects acting as domains https://review.openstack.org/231289 | 04:32 |
| *** su_zhang has joined #openstack-keystone | 04:48 | |
| davechen | notmorgan: what does "check experimental" means? | 04:49 |
| notmorgan | davechen: i'm using one of your patchsets as a baseline test ;) | 04:49 |
| notmorgan | davechen: it runs the check jobs in the experimental queue | 04:49 |
| davechen | notmorgan: where i can see it? | 04:49 |
| notmorgan | this is actually a case where i want to see the rally output . first and only time | 04:49 |
| notmorgan | look at zuul.openstack.org and the 'Experimental' queue | 04:50 |
| davechen | got it. | 04:50 |
| notmorgan | you'll see a couple things running | 04:50 |
| notmorgan | i'm profiling a change to try and offload queries to the backends/DB for a given request | 04:50 |
| notmorgan | not sure how much benefit there is | 04:51 |
| boris-42 | notmorgan: it's really bad that you guys removed rally job | 04:51 |
| davechen | notmorgan: ha, i saw it, this the first time I know this. | 04:51 |
| notmorgan | boris-42: no it isn't | 04:51 |
| notmorgan | boris-42: the job has been useless. not changed since it was implemented. fix it and make it useful and we'll add it back | 04:51 |
| boris-42 | notmorgan: it is, it was smarter to learn how to use it | 04:51 |
| boris-42 | notmorgan: you didn't get our patches | 04:51 |
| notmorgan | boris-42: running it in isolation like it is is not very useful | 04:51 |
| boris-42 | notmorgan: amaretsky was working on it | 04:51 |
| notmorgan | since every node has different performance profile and we're still not tracking data ovet time | 04:52 |
| *** chlong has joined #openstack-keystone | 04:52 | |
| boris-42 | notmorgan: it is, if you used it fully you won't make the stuff with removing cache and blocking whole openstack development for few days | 04:52 |
| notmorgan | for the record, that was mostly an overblown reaction | 04:52 |
| notmorgan | mostly | 04:52 |
| notmorgan | it was impactful but there are a lot of otherthings going on too | 04:53 |
| boris-42 | notmorgan: yep agree | 04:53 |
| notmorgan | but again, i can't know if rally being slow is that node, or more global without looking at every patchset | 04:53 |
| notmorgan | it's not really useful | 04:53 |
| notmorgan | if the data is cenrtalized, it becomes very useful | 04:53 |
| boris-42 | notmorgan: so usually from what i saw for 3 years | 04:53 |
| notmorgan | centralized/searchable/not a pain in the butt to get / correlate changes on | 04:54 |
| boris-42 | notmorgan: performance can be different about 2 times* | 04:54 |
| boris-42 | notmorgan: put sla with avg * 2 and you'll get failing job when something is going badly wrong | 04:54 |
| notmorgan | again, make it so i can see the difference between runs. i disagree with rally as it stands. | 04:54 |
| boris-42 | notmorgan: I understand that you would like to get everything instantly and for free | 04:54 |
| notmorgan | for a unit/check job | 04:55 |
| boris-42 | notmorgan: with prediciton & showing where code fails | 04:55 |
| boris-42 | notmorgan: but it won't happen immediately | 04:55 |
| notmorgan | it's why i didn't remove the thing and made it expiermental, run it on demand. literally not a single person in the meeting had ever used the data | 04:56 |
| notmorgan | that means there was a failure in garnering adoption/making it useful. | 04:56 |
| notmorgan | to us in general | 04:57 |
| *** lhcheng has quit IRC | 04:57 | |
| notmorgan | i will jump for joy when we have it being useful and it goes back in. | 04:57 |
| boris-42 | notmorgan: ok | 04:57 |
| boris-42 | notmorgan: maybe I should make a video demo | 04:57 |
| notmorgan | i didn't delete it because i didn't want it to be totally gone. | 04:57 |
| boris-42 | notmorgan: about rally in gates | 04:57 |
| notmorgan | boris-42: you need to make the data easy to get at | 04:57 |
| *** roxanagh_ has joined #openstack-keystone | 04:58 | |
| notmorgan | boris-42: if the data is easy to get at / understand that will go a long way too. | 04:58 |
| notmorgan | i think i asked a year and a half ago for a central view of over-time w/ rally + searchable. | 04:58 |
| notmorgan | when we threatend to remove it last | 04:58 |
| boris-42 | notmorgan: yep it is quite old task | 04:59 |
| notmorgan | but i did leave it experimental because it allows someone to use it but it's not consuming resources (a real premium atm) on every check run | 04:59 |
| boris-42 | notmorgan: ok | 04:59 |
| boris-42 | notmorgan: but not sure that somebody will find it now =( | 05:00 |
| notmorgan | btw: rally would not have picked up the issue with the caching | 05:00 |
| notmorgan | that was not a keystone server issue | 05:00 |
| notmorgan | that was a keystone middleware issue, keystonemiddleware never ever ever is run against keystone | 05:00 |
| boris-42 | notmorgan: it was middlerware I know | 05:00 |
| notmorgan | erm as keystone | 05:00 |
| boris-42 | notmorgan: it will catch actually (because we have scenarios that tests caching) | 05:00 |
| notmorgan | but ... it can't catch caching changes on keystonemiddleware for the keystonejob | 05:01 |
| notmorgan | thats all | 05:01 |
| notmorgan | since middleware isn't involved | 05:01 |
| boris-42 | notmorgan: and now you are running only keystone? | 05:01 |
| boris-42 | notmorgan: in dsvm jobs? | 05:01 |
| boris-42 | notmorgan: we are testing caching (from different projects e.g. nova/cinder) | 05:02 |
| notmorgan | the job we made experiemental wouldn't have caught this bug | 05:02 |
| notmorgan | that was my point | 05:02 |
| *** roxanagh_ has quit IRC | 05:02 | |
| boris-42 | notmorgan: so we have exatcly scenarios that tests glance/cinder/nova token caching =) | 05:02 |
| boris-42 | notmorgan: jfyi | 05:02 |
| notmorgan | i also wont say it is not useful in nova | 05:02 |
| notmorgan | it is not useful in keystone specifically | 05:02 |
| notmorgan | i wouldn't advocate making the job experimental elsewhere | 05:02 |
| notmorgan | i haven't looked to see how useful it is | 05:03 |
| * stevemar is pretty sure notmorgan has worked on keystone all weekend | 05:03 | |
| notmorgan | but you get a more end-to-end view with other services mixed in | 05:03 |
| notmorgan | so since we only ran token creates in our job | 05:03 |
| notmorgan | and very limited other things | 05:03 |
| notmorgan | it shows me it wasn't a useful job | 05:03 |
| notmorgan | stevemar: i have sortof | 05:03 |
| stevemar | notmorgan: i am close to finalizing the mid cycle schedule | 05:04 |
| notmorgan | stevemar: nice | 05:04 |
| boris-42 | notmorgan: ok | 05:04 |
| notmorgan | boris-42: so mayyyybe having a job that is doing specific per-project things isn't useful maybe we need a baseline "here are all the things we check no matter what" set | 05:05 |
| davechen | stevemar: will this bug is included in the schedule? (https://bugs.launchpad.net/keystone/+bug/1501740) | 05:05 |
| openstack | Launchpad bug 1501740 in OpenStack Identity (keystone) "Creating a region without request parameters failed." [Medium,In progress] - Assigned to Dave Chen (wei-d-chen) | 05:05 |
| notmorgan | boris-42: and then some project-spefici things | 05:05 |
| davechen | stevemar: different reviwers has different views on this. | 05:05 |
| boris-42 | notmorgan: so basically we are able to specify specific things per job | 05:05 |
| notmorgan | boris-42: again the problem with the keystone job is it was purely isolated and not showing a heck of a lot. plus a lot of "this isn't super useful to look at as is" | 05:05 |
| boris-42 | notmorgan: with SLA | 05:05 |
| stevemar | davechen: it's definitely one of the items i want to discuss, just trying to find a slot for it | 05:06 |
| notmorgan | boris-42: ok but i don't want to have to see an update to keystone every single time we need to add things, and nova and glance and cinder | 05:06 |
| boris-42 | notmorgan: so except trending which is really hard and requires hardware that I don't have | 05:06 |
| notmorgan | boris-42: look at bandit. it has some basic profiles | 05:06 |
| notmorgan | boris-42: that we can then override/modify | 05:06 |
| notmorgan | that means when bandit adds a new thing, we get it for free [yeah linter != rally, but you get the idea] | 05:07 |
| davechen | stevemar: cool, if notmorgan's comments got buy in from the mid cycle, we will check the APIs instead. | 05:07 |
| notmorgan | boris-42: lets circle up w/ infra, maybe we can do some simple graphite magic or some sort of "make this data something we can correlate" with ELK or the like | 05:08 |
| notmorgan | boris-42: i'd see that as super useful | 05:08 |
| boris-42 | notmorgan: it's very hard task, we can keep all results in single db and fetch it | 05:08 |
| boris-42 | notmorgan: however it's hard to normalize results | 05:08 |
| notmorgan | boris-42: we can probably also trend the checks with graphite and run a post job to show outliers | 05:09 |
| boris-42 | notmorgan: it want stop merging bad code | 05:09 |
| boris-42 | won't* | 05:09 |
| notmorgan | boris-42: no it wont. but we can for sure show at a glance when something went wonky or show that a given changeset is horrrrribly out of whack | 05:09 |
| boris-42 | notmorgan: okay I will work on this for a bit | 05:10 |
| notmorgan | boris-42: i'm happy to help chat with infra to see if we can get some resources for centralizing it in a sane way | 05:10 |
| boris-42 | notmorgan: so what we need is to have just isolated resources (but they say that they don't have) | 05:10 |
| notmorgan | this would be like the log server | 05:11 |
| notmorgan | a place where we can inject the raw data and show the over-time changes. maybe the over-time changes is a gate-only job, and the check job shows comparison to the over time changes | 05:11 |
| notmorgan | boris-42: lets plan to talk more post keystone midcycle and i'm happy to help you plan this out | 05:12 |
| boris-42 | notmorgan: i mean we already have DB | 05:12 |
| boris-42 | notmorgan: in Rally | 05:12 |
| boris-42 | notmorgan: that stores all results | 05:12 |
| notmorgan | boris-42: a persistent db? | 05:12 |
| boris-42 | notmorgan: yep | 05:12 |
| notmorgan | is it available to query? | 05:12 |
| boris-42 | notmorgan: it's SQL db | 05:12 |
| boris-42 | notmorgan: so what we need is just to have single DB instead of each time new one | 05:13 |
| notmorgan | right | 05:13 |
| boris-42 | notmorgan: and run post jobs and store results | 05:13 |
| notmorgan | yes | 05:13 |
| notmorgan | something like that | 05:13 |
| boris-42 | notmorgan: and I will make simple reporting for that | 05:13 |
| notmorgan | heck we do it with some subunit2sql now too | 05:13 |
| boris-42 | notmorgan: I mean Rally DB exists since the begging of Rally ;) | 05:13 |
| notmorgan | right. but it's per run. not central for everything | 05:14 |
| notmorgan | so each run is a clean / new db? | 05:14 |
| notmorgan | as in local to the dsvm run | 05:14 |
| boris-42 | notmorgan: yep it's current implementation | 05:14 |
| notmorgan | ok | 05:14 |
| boris-42 | notmorgan: of dsvm job for rally | 05:14 |
| boris-42 | notmorgan: for post job we will need a bit different job | 05:14 |
| notmorgan | so lets work on either making that a central resource we can keep data in or finding a way to push the data to a central resource we can use in a post jb | 05:15 |
| notmorgan | job* | 05:15 |
| notmorgan | whether it's SQL or something else. | 05:15 |
| notmorgan | and i think that will make rally significantly more interesting. | 05:15 |
| notmorgan | i also still recommend making a set of baseline scenarios we get for free that are just part of rally, like how bandit works | 05:16 |
| boris-42 | notmorgan: so it's very hard... because it really depends on project | 05:16 |
| boris-42 | notmorgan: we are working on OPS stuff | 05:16 |
| boris-42 | notmorgan: https://github.com/openstack/rally/tree/master/certification/openstack | 05:17 |
| boris-42 | notmorgan: but not sure that this is suitable for your case | 05:17 |
| notmorgan | boris-42: right but you know that genrrally you can do a nova boot x instances for the generic rally job | 05:17 |
| notmorgan | so maybe that should be always there | 05:17 |
| notmorgan | where the add_remove_user_role job [keystone] is really a keystone specific thing | 05:17 |
| notmorgan | since it wont be impacted by anything nova does | 05:18 |
| boris-42 | notmorgan: so we can make large-ops-job-like-task | 05:18 |
| boris-42 | notmorgan: and keep it in rally | 05:18 |
| boris-42 | notmorgan: but where you would like to use it ? | 05:18 |
| * notmorgan is trying to give suggestions that makes rally more interesting/useful without asking for each and every change to require a change to keystone and nova and glance etc to enable a scenario | 05:18 | |
| notmorgan | but i think the central data is really going to be killer addon for the gate :) | 05:19 |
| boris-42 | notmorgan: ok I will manage to work on that | 05:19 |
| boris-42 | notmorgan: however need to finish osprofiler first | 05:20 |
| notmorgan | sure | 05:20 |
| *** EinstCrazy has joined #openstack-keystone | 05:23 | |
| *** markvoelker has joined #openstack-keystone | 05:44 | |
| notmorgan | jamielennox: oh my | 05:44 |
| notmorgan | jamielennox: so... deepcopy is REALLY expensive | 05:44 |
| notmorgan | jamielennox: insanely so | 05:44 |
| notmorgan | jamielennox: http://paste.openstack.org/show/484793/ | 05:46 |
| notmorgan | jamielennox: let me try spinning my patch with a flatten/reconstruct bit of code | 05:47 |
| notmorgan | and i mean that thing i did there was so bloody synthetic i don't know what to say ^ | 05:47 |
| notmorgan | and that is far far simpler than what i was doing. | 05:48 |
| *** markvoelker has quit IRC | 05:48 | |
| *** jaosorior has joined #openstack-keystone | 05:52 | |
| *** rcernin has joined #openstack-keystone | 05:54 | |
| *** tyagiprince has joined #openstack-keystone | 05:55 | |
| tyagiprince | Hii. I want to know why is it necessary to run neutron-ovs-cleanup command on nodes before starting the openstack services like dhcp and l3-agent. Please help me. | 05:56 |
| *** su_zhang has quit IRC | 05:56 | |
| jamielennox | notmorgan: wow, that's kind of nutes | 05:57 |
| notmorgan | jamielennox: right? | 05:58 |
| notmorgan | want to see the comparison on pickle? | 05:58 |
| notmorgan | jamielennox: http://paste.openstack.org/show/484794/ | 05:58 |
| jamielennox | tyagiprince: i have no idea, probably best to try the openstack-neutron channel | 05:58 |
| notmorgan | jamielennox: it also tells me we may be suffering copy.deepcopy things on the cache-isolating proxy for testing | 05:59 |
| notmorgan | might be adding a significant overhead for our unit tests | 05:59 |
| jamielennox | i don't know if it's going to cause problems though | 05:59 |
| *** roxanagh_ has joined #openstack-keystone | 05:59 | |
| *** EinstCrazy has quit IRC | 05:59 | |
| tyagiprince | jamielennox: Tried that.. Been asking there since 2-3 days.. no one talks there :P | 05:59 |
| jamielennox | also if you have actual objects you can add __copy__ to speed up the guessing | 05:59 |
| notmorgan | jamielennox: i'm going to run a similar synthetic test with json serialize now | 06:00 |
| jamielennox | tyagiprince: well last 2-3 days have been the weekend :) i'm surprised though - anyway networking is all just magic stuff, can't help | 06:00 |
| tyagiprince | jamielennox: the problem I am facing is when I restarted my connet (controller + network) node, the dhcp agent status goes down. | 06:01 |
| tyagiprince | okay.. Thanks :) | 06:01 |
| jamielennox | notmorgan: gotta run out for a bit, will see result later though | 06:01 |
| *** ajayaa has joined #openstack-keystone | 06:02 | |
| tyagiprince | notmorgan: you might be able to help me out. I see you there in openstack-neutron:P | 06:03 |
| *** EinstCrazy has joined #openstack-keystone | 06:03 | |
| *** henrynash has quit IRC | 06:03 | |
| *** roxanagh_ has quit IRC | 06:04 | |
| notmorgan | tyagiprince: i lurk in a lot of channels | 06:05 |
| notmorgan | :P | 06:06 |
| notmorgan | and i mean a lot | 06:06 |
| notmorgan | mostly for when people need keystone advice | 06:06 |
| notmorgan | not sure what neutron-ovs-cleanup does, but my guess is it removes the networking stuff that isn't needed / cleans up all the linvering definitions so neutron can repopulate them sanely | 06:06 |
| tyagiprince | notmorgan: Yup, you have helped me alot earlier :) | 06:06 |
| notmorgan | jamielennox: 7usec per loop on json deserialize | 06:07 |
| notmorgan | jamielennox: so json deserialize is 2x faster than deepcopy is | 06:07 |
| notmorgan | 0.o | 06:07 |
| *** tyagiprince1 has joined #openstack-keystone | 06:11 | |
| *** Nirupama has joined #openstack-keystone | 06:11 | |
| *** tyagiprince has quit IRC | 06:14 | |
| *** tyagiprince1 is now known as tyagiprince | 06:14 | |
| notmorgan | oh interesting... revoke tree has circular references! | 06:19 |
| * notmorgan glares. | 06:19 | |
| stevemar | notmorgan: why shouldn't it! | 06:19 |
| notmorgan | i'll bet that has something to do with.. $OMGSLOW$ | 06:19 |
| notmorgan | things you learn when you try and serialize an object | 06:19 |
| ajayaa | Hi. Is this blueprint https://blueprints.launchpad.net/keystone/+spec/pagination-backend-support still relevant? | 06:21 |
| ajayaa | I am interested in picking it up. | 06:21 |
| notmorgan | ajayaa: you don't want tpo ask me that question | 06:21 |
| notmorgan | cause i have very strong opinions on pagination | 06:21 |
| notmorgan | stevemar: can atest to that | 06:22 |
| ajayaa | okay. notmorgan, please pretend the you didn't see the above question. :) | 06:22 |
| *** jaosorior has quit IRC | 06:22 | |
| ajayaa | s/the/that | 06:22 |
| notmorgan | stevemar: so.. i think we have a problem in revoke events =/, if this is really a circular reference it's non-GC'able | 06:23 |
| notmorgan | stevemar: so we're leaking memory in keystone | 06:23 |
| * notmorgan puts on Oh GOD IT MAKES MY EYES BLEED debugging hat | 06:23 | |
| stevemar | notmorgan: we had some folks looking at revoke events internally, and they def had criticism about it | 06:24 |
| notmorgan | stevemar: i have a sweeping change i want to make | 06:25 |
| stevemar | maybe a tree wasn't the best impl. for it | 06:25 |
| notmorgan | stevemar: dump the entire tree thing and just rely on the backend [SQL] to do the heavy lifting | 06:25 |
| notmorgan | it does mean a lot more indexes | 06:25 |
| stevemar | notmorgan: sweeping changes in mitaka-3, no big deal | 06:25 |
| notmorgan | but i mean, we can do a simple SQL query and say "oh something matches" | 06:25 |
| stevemar | true | 06:25 |
| notmorgan | and it'll be destroying the interfaces | 06:25 |
| notmorgan | that were unfortunately "public" | 06:26 |
| notmorgan | there is no way to maintain compat. | 06:26 |
| notmorgan | afaict | 06:26 |
| notmorgan | but it's on objects no one should ever ever ever ever ever ever ever ever be touching | 06:26 |
| notmorgan | so it should be ok | 06:26 |
| stevemar | notmorgan: what about just changing the impl. without affecting the manager/driver interfaces? | 06:27 |
| notmorgan | thats the issue | 06:27 |
| notmorgan | well we need to change the driver interface | 06:27 |
| notmorgan | because we have to pass a lot more data down or we need to do driver -> manager calls | 06:27 |
| notmorgan | which we said was bad (no cross driver calls) | 06:28 |
| notmorgan | but that is mostly versionable | 06:28 |
| notmorgan | it's the model that isn't | 06:28 |
| notmorgan | but no one should ever get a model unless then do ._get_revoke_tree | 06:28 |
| *** chlong has quit IRC | 06:28 | |
| notmorgan | it's just the revoke tree is "technically" public | 06:29 |
| notmorgan | and we will be throwing it out | 06:29 |
| notmorgan | and never using it and breaking compat on being able to use it | 06:29 |
| *** jaosorior has joined #openstack-keystone | 06:30 | |
| * notmorgan tries to figure out how the hell revoke tree has circular references | 06:30 | |
| *** jamielennox is now known as jamielennox|away | 06:31 | |
| stevemar | notmorgan: double check that it actually does, like i said, we had folks internally who took a long look at and found issues, but never circular references | 06:33 |
| *** jamielennox|away is now known as jamielennox | 06:33 | |
| notmorgan | oh look at that we can use the gc module to deal with freeing it | 06:34 |
| notmorgan | ugh | 06:34 |
| notmorgan | stevemar: this is based on trying to serialize it. the error is "circular refernce found" so i am thinking it realy is one | 06:34 |
| notmorgan | the fact that it does recursive self modification it doesn't surprise me | 06:34 |
| notmorgan | it isn't going to be "easy"to see where the ref happens | 06:35 |
| *** su_zhang has joined #openstack-keystone | 06:36 | |
| *** henrynash has joined #openstack-keystone | 06:40 | |
| *** ChanServ sets mode: +v henrynash | 06:40 | |
| *** gildub has quit IRC | 06:41 | |
| *** belmoreira has joined #openstack-keystone | 06:44 | |
| *** lhcheng has joined #openstack-keystone | 06:46 | |
| *** ChanServ sets mode: +v lhcheng | 06:46 | |
| *** EinstCrazy has quit IRC | 06:47 | |
| openstackgerrit | Steve Martinelli proposed openstack/python-keystoneclient: use positional library instead of utils https://review.openstack.org/271906 | 06:49 |
| stevemar | poke notmorgan and jamielennox https://review.openstack.org/#/c/271906/ | 06:49 |
| notmorgan | stevemar: -1 | 06:50 |
| stevemar | notmorgan: oh noes | 06:50 |
| *** lhcheng has quit IRC | 06:50 | |
| notmorgan | stevemar: can't remove the import versions of ksm require it in .utils | 06:50 |
| jamielennox | stevemar: what he said | 06:51 |
| notmorgan | so it'll need to live there for a long time | 06:51 |
| stevemar | :( | 06:51 |
| notmorgan | like... forever | 06:51 |
| notmorgan | so maybe we need a # NOTE(): DO NOT REMOVE | 06:51 |
| jamielennox | it's not a big deal, just put a comment in there and we'll remove it later | 06:51 |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone: Removed deprecated revoke KVS backend https://review.openstack.org/267777 | 06:51 |
| notmorgan | stevemar: ^ that fixes the test cases | 06:51 |
| stevemar | notmorgan: oh thanks | 06:52 |
| stevemar | jamielennox: abandon: https://review.openstack.org/#/c/179703/4 ? | 06:52 |
| jamielennox | stevemar: oh -yep | 06:52 |
| openstackgerrit | Steve Martinelli proposed openstack/python-keystoneclient: use positional library instead of utils https://review.openstack.org/271906 | 06:54 |
| stevemar | notmorgan: added a comment | 06:54 |
| openstackgerrit | henry-nash proposed openstack/keystone: Removes project.domain_id FK https://review.openstack.org/233274 | 06:56 |
| notmorgan | henrynash: oh that's in lifght of using project as a domain? | 06:56 |
| henrynash | notmorgan: yes, the phase 1 stuff | 06:57 |
| notmorgan | henrynash: we may not want to be rid of the FK though | 06:58 |
| notmorgan | it might just need to be FK(project.id) | 06:58 |
| notmorgan | not sure if that is valid in mysql, but i think it's valid SQL | 06:58 |
| notmorgan | another example would be employee.manager is an employee but a FK | 06:58 |
| notmorgan | since you want the relationship | 06:58 |
| stevemar | notmorgan: whats up with "self.config_fixture.config(group='revoke', driver='sql')", sql is the default now | 06:59 |
| henrynash | notmorgan: conceptually I agree - however, since domain_id can be None (for projects acting as domains), taht could be tricky | 06:59 |
| notmorgan | henrynash: a check on the SQL-A model bind to ensure project.id != domain.id | 06:59 |
| notmorgan | henrynash: this is something that needs to be added to the SQL-Model itself. | 06:59 |
| henrynash | notmorgan: and in fact we store a special value instead of None in teh domain_id column, so as to still use uniqueness constaints | 06:59 |
| notmorgan | uh | 07:00 |
| notmorgan | wait what? | 07:00 |
| notmorgan | i don;t like that at all | 07:00 |
| notmorgan | stevemar: we can remove them but we have them lingering elsewhere | 07:00 |
| henrynash | which part :-) | 07:00 |
| openstackgerrit | Steve Martinelli proposed openstack/keystone: Removed deprecated revoke KVS backend https://review.openstack.org/267777 | 07:00 |
| notmorgan | henrynash: a not none for uniqueness constraints? | 07:01 |
| notmorgan | why does there need to be uniqueness constraints? | 07:01 |
| henrynash | notmorgan: we still want name+domain_id to be unique | 07:02 |
| notmorgan | domain names are globally unique | 07:02 |
| notmorgan | so name+None is unique | 07:02 |
| notmorgan | well name+null | 07:02 |
| henrynash | notmorgan: and for projects actinsg as domains, the domain_id is None | 07:02 |
| notmorgan | so thats fine | 07:02 |
| henrynash | notmorgan: yep | 07:02 |
| notmorgan | so no reason to store a special value | 07:02 |
| notmorgan | and the FK can exist, but be NULL | 07:02 |
| notmorgan | so column is FK -> project.id and nullable=True | 07:03 |
| henrynash | notmorgan: although I don;t think sql supoer for a unquness constarints with a null value is that stadnard | 07:03 |
| henrynash | notmotgan: at least not according to the literature | 07:03 |
| notmorgan | we can also make a special row called "root" that is self-referring | 07:03 |
| openstackgerrit | Steve Martinelli proposed openstack/keystone: remove deprecated revoke_by_expiration function https://review.openstack.org/271135 | 07:04 |
| henrynash | notmorgan: you could…but see the patch for what I am proposing: https://review.openstack.org/#/c/264533/ | 07:04 |
| notmorgan | i'm checking if null is part of a constraing when doing unique() now | 07:04 |
| *** EinstCrazy has joined #openstack-keystone | 07:05 | |
| notmorgan | henrynash: solution | 07:05 |
| notmorgan | might work... | 07:05 |
| notmorgan | give me a sec | 07:05 |
| notmorgan | ah null is not distinct | 07:06 |
| openstackgerrit | Steve Martinelli proposed openstack/keystone: remove deprecated revoke_by_expiration function https://review.openstack.org/271135 | 07:07 |
| notmorgan | actually.. we can work around that | 07:08 |
| henrynash | notmorgan: and we have to deal with legacy drivers who might not supprt None as domain_id | 07:08 |
| notmorgan | so we do something evil | 07:09 |
| openstackgerrit | henry-nash proposed openstack/keystone: Projects acting as domains https://review.openstack.org/231289 | 07:09 |
| notmorgan | we create a row that is called "root" that has it'self as domain.id | 07:09 |
| notmorgan | or __keystone_root__ | 07:09 |
| notmorgan | and we just strip it out in all cases | 07:09 |
| notmorgan | make it's id like Default, a string vs uuid | 07:10 |
| notmorgan | so we know it's static | 07:10 |
| notmorgan | anything that is a domain is parented to __keystone_root__ | 07:10 |
| openstackgerrit | Steve Martinelli proposed openstack/keystone: Remove eventlet support https://review.openstack.org/249486 | 07:10 |
| notmorgan | and we can handle that internally | 07:10 |
| notmorgan | i don't like using a non-FK for this because it should be enforced to exist.. alternatively you can use a bind in the SQL-A driver to enforce existence. but that seems like a lot of extra work | 07:11 |
| notmorgan | and the driver itself can handle the adding/removing of the __keystone_root__ id | 07:11 |
| stevemar | notmorgan: since you're on a 'decrease-test-time-kick', henrynash has a patch that'll speed things up here: https://review.openstack.org/#/c/271106/ | 07:12 |
| henrynash | notmorgan: so understand your approach, not yet convinced it’s less evil than mine…but let me think about it! | 07:12 |
| notmorgan | henrynash: i am worried you're going to get inconsistencies in the db because somehow because we're not ensuring a cascade delete | 07:12 |
| notmorgan | henrynash: and orphaned projects are relatively concerning if somehow a domain is deleted with projects still attached to it | 07:13 |
| notmorgan | henrynash: the cases we removed the FKs before we move the things into other backends. so we know we had x-driver actions | 07:13 |
| notmorgan | this time we're doing something a bit odd and referencing something in our own table | 07:14 |
| notmorgan | this one worries me a bit that is all. | 07:15 |
| notmorgan | because until now it was impossible to orphan a project | 07:15 |
| henrynash | notmorgan: so we are not changing the Fk for child project to it’s parent, that’s still presnet - Iguess youre argument is tha t we shold do the same for domain_id, with the project table | 07:15 |
| notmorgan | right | 07:15 |
| henrynash | notmorgan: ok, understand the concern….let em mull in it | 07:16 |
| notmorgan | yeah need to figure out the right way to solve it, since unique() shouldn't have nullable columns | 07:16 |
| notmorgan | but then how do we represent a domain properly | 07:16 |
| notmorgan | and avoid the non-unique domain name issue | 07:16 |
| *** shoutm_ has joined #openstack-keystone | 07:17 | |
| stevemar | making a call to cancel the keystone meeting on tuesday, any dissenters? | 07:19 |
| notmorgan | ah | 07:19 |
| notmorgan | henrynash: have a workaround | 07:19 |
| henrynash | stevemar: nix it | 07:19 |
| henrynash | notmorgan: ok.... | 07:19 |
| stevemar | ++ | 07:19 |
| *** shoutm has quit IRC | 07:20 | |
| notmorgan | henrynash: ignore the :trigger: | 07:20 |
| notmorgan | we don't want that | 07:20 |
| notmorgan | http://paste.openstack.org/show/484798/ | 07:20 |
| notmorgan | taken from https://bugs.mysql.com/bug.php?id=17825 [towards the bottom] | 07:20 |
| notmorgan | and we can use SQL-A bind or the driver to enforce that the dummy [or in our case is_domain] is only set for domains | 07:21 |
| notmorgan | this would be mysql-specific. we'd need to see how pgsql and db2 handles it | 07:21 |
| *** lhcheng has joined #openstack-keystone | 07:21 | |
| *** ChanServ sets mode: +v lhcheng | 07:21 | |
| notmorgan | but basically the constraint would be (name, is_domain) | 07:22 |
| notmorgan | for mysql | 07:22 |
| *** tyagiprince1 has joined #openstack-keystone | 07:22 | |
| notmorgan | and the domain_id would be FK project.id | 07:22 |
| henrynash | notmorgan: although that’s not enough…since we want name+domain_id for all other domains to unique as well | 07:22 |
| notmorgan | that is fine. | 07:22 |
| notmorgan | you cna still unqiue(name, domain.id) | 07:23 |
| notmorgan | now in non-mysql dialects it probalbly is sufficient to unique name domain.id | 07:23 |
| henrynash | ah, right - so two different uniqueness constaints | 07:23 |
| notmorgan | also maybe domains use themselves as the domain.id | 07:23 |
| notmorgan | ? | 07:24 |
| *** tyagiprince has quit IRC | 07:24 | |
| *** tyagiprince1 is now known as tyagiprince | 07:24 | |
| notmorgan | i actually think the root row of __keystone_root_domain__ is probably the best answer | 07:24 |
| notmorgan | that way all the enforcement is handled in a sane way | 07:24 |
| henrynash | notmorgan: …I think that is better than your last suggestion, yes…. | 07:25 |
| stevemar | notmorgan: fixes the revert from the eventlet gate failure: https://review.openstack.org/#/c/271851/1 | 07:27 |
| *** tyagiprince has quit IRC | 07:28 | |
| notmorgan | henrynash: mysql> insert into test (`id`, `parent`) VALUES(10, 10); | 07:30 |
| notmorgan | henrynash: Query OK, 1 row affected (0.01 sec) | 07:30 |
| notmorgan | mysql> insert into test (`id`, `parent`) VALUES(10, 12); | 07:30 |
| notmorgan | ERROR 1452 (23000): Cannot add or update a child row: a foreign key constraint fails (`ks`.`test`, CONSTRAINT `fk_parent` FOREIGN KEY (`parent`) REFERENCES `test` (`id`) ON DELETE CASCADE ON UPDATE CASCADE) | 07:30 |
| notmorgan | henrynash: so self-reference works well. | 07:30 |
| notmorgan | henrynash: my test was create table `test` ( `id` int unsigned not null auto_increment, `parent` int unsigned default null, primary key(`id`), constraint `fk_parent` FOREIGN KEY (`parent`) references `test` (`id`) on delete cascade on update cascade) engine=InnoDB; | 07:31 |
| henrynash | notmorgan: good…and we use that already for the child->parent project Id | 07:31 |
| notmorgan | henrynash: could make parent Not Null | 07:31 |
| henrynash | notmorgan: certainly making a “root” row, would be one wayt to do this | 07:31 |
| notmorgan | henrynash: and the backend driver can just strip it out | 07:32 |
| notmorgan | and automatically add it back in where needed | 07:32 |
| notmorgan | so the business logic code can ignore it | 07:32 |
| notmorgan | and we get None as expected | 07:32 |
| notmorgan | or the SQL-A model bind does the magic for that | 07:32 |
| henrynash | notmorgan: (which is how I implemted the special value…it’s all in the driver) | 07:32 |
| openstackgerrit | Steve Martinelli proposed openstack/python-keystoneclient: remove CLI from keystoneclient https://review.openstack.org/258181 | 07:32 |
| henrynash | notmorgan: so same idea, but agree yours would provide more referential integrity | 07:33 |
| notmorgan | henrynash: see line ~143 and such https://review.openstack.org/#/c/269693/4/keystone/common/sql/core.py if you want to see something neat/hacky we can do to make it baked into the model | 07:33 |
| notmorgan | i think we can do that too with the sql-a model not just a type | 07:33 |
| notmorgan | but we could also make it project_domain_id type that does the work so the driver isn't even responsible. | 07:34 |
| notmorgan | henrynash: but in the driver is good for me. | 07:34 |
| henrynash | notmorgan: yep, maybe…although maybe keeping it more standard might give us confidence it will work acorss all sqls | 07:35 |
| notmorgan | henrynash: ok posted a comment on the review | 07:36 |
| notmorgan | -1 as well. | 07:36 |
| notmorgan | sorry | 07:36 |
| henrynash | notmorgan: nothing wround with that :-) | 07:37 |
| henrynash | (wrong with) | 07:37 |
| notmorgan | stevemar: +2/+A on henrynash's stop doing duplicate tests | 07:37 |
| *** shoutm_ has quit IRC | 07:37 | |
| stevemar | notmorgan: yay | 07:38 |
| notmorgan | i have to pack bags, do laundry, get hair cut, and send things back to amazon tomorrow | 07:38 |
| notmorgan | so tuesday i can hop on a plain stupid early | 07:38 |
| *** shoutm has joined #openstack-keystone | 07:39 | |
| openstackgerrit | Steve Martinelli proposed openstack/keystone: Enhance manager list_role_assignments to support group listing https://review.openstack.org/265650 | 07:40 |
| stevemar | henrynash: rebased ^ for you | 07:40 |
| notmorgan | stevemar: i don't know if we want https://review.openstack.org/#/c/215715/ because it dirty touches internal interfaces | 07:40 |
| henrynash | stevemar: thx | 07:40 |
| notmorgan | stevemar: but it's the only way i think we're landing role caching this cycle. | 07:40 |
| notmorgan | i am trying to figure out how to make that friendly in upstream dogpile | 07:41 |
| stevemar | notmorgan: hehe your comment is funny: "YES!" | 07:41 |
| notmorgan | but i'm not sure if there is an easy way. | 07:41 |
| notmorgan | but that patch [except my own -1 for the sake of "damn it bad comment"] is able to pass/go | 07:42 |
| stevemar | notmorgan: role_assignments are pretty expensive | 07:42 |
| notmorgan | and it can fix catalog too | 07:42 |
| notmorgan | which we need to basically rip out the caching in that requires region.invalidate or we need the same fix | 07:42 |
| notmorgan | and backport the fix | 07:42 |
| notmorgan | stevemar: https://bugs.launchpad.net/keystone/+bug/1537617 | 07:42 |
| openstack | Launchpad bug 1537617 in OpenStack Identity (keystone) "caching of the catalog does not invalidate across processes" [High,Triaged] | 07:43 |
| stevemar | notmorgan: y, i saw that in the devstack patch | 07:43 |
| *** EinstCrazy has quit IRC | 07:43 | |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone: Apply invalidation proxy to the catalog cache region https://review.openstack.org/271536 | 07:44 |
| *** markvoelker has joined #openstack-keystone | 07:44 | |
| openstackgerrit | henry-nash proposed openstack/keystone: WIP: Remove domain table references https://review.openstack.org/165936 | 07:44 |
| notmorgan | stevemar: so i personally think region.invalidate means we did something horrrrrribly wrong | 07:45 |
| notmorgan | in our design | 07:45 |
| notmorgan | but. it's hard to refactor all of it to solve the issue. | 07:45 |
| *** topol has quit IRC | 07:45 | |
| notmorgan | i have an alternative approach, but it's also kindof ... ugly | 07:46 |
| notmorgan | we can also use a version id of somesort we can share across instances [somehow?] and pass that in as an argument or somehow utilize it as part of the cachekey building. | 07:47 |
| *** roxanagh_ has joined #openstack-keystone | 07:47 | |
| notmorgan | and an invalidate simply changes that key's value. | 07:47 |
| *** topol_ has joined #openstack-keystone | 07:47 | |
| notmorgan | still will suffer from much of the same issues of "if shared store dies" or whatever. | 07:47 |
| notmorgan | but.. | 07:47 |
| notmorgan | at the very least it doesn't patch internal interfaces. | 07:47 |
| openstackgerrit | henry-nash proposed openstack/keystone: Bye Bye Domain Table https://review.openstack.org/161854 | 07:48 |
| *** markvoelker has quit IRC | 07:49 | |
| openstackgerrit | Steve Martinelli proposed openstack/python-keystoneclient: remove oslo-incubator apiclient https://review.openstack.org/257127 | 07:49 |
| *** EinstCrazy has joined #openstack-keystone | 07:50 | |
| *** roxanagh_ has quit IRC | 07:51 | |
| *** henrynash has quit IRC | 07:51 | |
| * stevemar yawns | 07:57 | |
| stevemar | notmorgan: lets let ayoung push https://review.openstack.org/#/c/267777/ through, revoke is his baby | 07:58 |
| notmorgan | stevemar: nope, gonna +A that the moment it passed | 07:58 |
| notmorgan | passes | 07:58 |
| notmorgan | if i'm awake | 07:58 |
| notmorgan | he's already good with it | 07:59 |
| stevemar | notmorgan: oh, in that case, it's ready | 07:59 |
| notmorgan | yeah i was just waiting for jenkins +1 but feel free to +A now | 07:59 |
| stevemar | oh wait, zuul is failing it | 07:59 |
| notmorgan | you BROKE SOMETHING | 08:00 |
| notmorgan | :P | 08:00 |
| notmorgan | uhm | 08:00 |
| stevemar | ./keystone/tests/unit/test_revoke.py:193:1: E302 expected 2 blank lines, found 1 | 08:00 |
| notmorgan | what | 08:00 |
| notmorgan | still has .kvs entries? | 08:00 |
| stevemar | i just changed the commit msg :P | 08:00 |
| stevemar | notmorgan: i think you missed this one: https://github.com/openstack/keystone/blob/c0c1227c30457fcb0c9e4bb104aaa89292aede9e/keystone/tests/unit/test_v3_auth.py#L3193 | 08:02 |
| *** shoutm_ has joined #openstack-keystone | 08:02 | |
| notmorgan | dealing with it now | 08:02 |
| *** shoutm has quit IRC | 08:03 | |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone: Removed deprecated revoke KVS backend https://review.openstack.org/267777 | 08:03 |
| jamielennox | notmorgan, stevemar: can you give your thoughts on https://review.openstack.org/#/c/271929/ when you have a minute | 08:04 |
| notmorgan | jamielennox: i love it.. no wait i hate it ... no wait... CHANGE PLACES | 08:05 |
| notmorgan | 10/16 lets have a tea party | 08:05 |
| *** EinstCrazy has quit IRC | 08:05 | |
| notmorgan | oer erm 10/6 | 08:06 |
| jamielennox | notmorgan: maybe i should ask again tomorrow instead ... | 08:06 |
| notmorgan | jamielennox: lol | 08:06 |
| *** EinstCrazy has joined #openstack-keystone | 08:06 | |
| notmorgan | i'm not opposed to this really | 08:06 |
| stevemar | jamielennox: follow the yellow brick road | 08:06 |
| notmorgan | stevemar: Who.... are.... yooooouuuu? | 08:06 |
| notmorgan | stevemar: Follow the yellow brick road? | 08:07 |
| notmorgan | follow the yellow brick road! | 08:07 |
| jamielennox | i'm not sure if architecturely it should be an oslo.policy credentials_dict_from_context method - but either way i would like to get a standard list | 08:07 |
| stevemar | just a scared tin man | 08:07 |
| notmorgan | stevemar: if you only had a heart. | 08:07 |
| notmorgan | stevemar: is it bad that i'm catching up to you on -2's for mitaka | 08:08 |
| notmorgan | jamielennox: yeah not sure if that is right, but standardizing would be fantastic | 08:09 |
| stevemar | jamielennox: i | 08:10 |
| stevemar | jamielennox: i'll keep it open in my tab | 08:10 |
| notmorgan | jamielennox: quick +2/+A before it escapes https://review.openstack.org/#/c/271906/ | 08:11 |
| *** EinstCrazy has quit IRC | 08:11 | |
| stevemar | jamielennox: and it's child patch | 08:12 |
| stevemar | cause that's a lot of rebasing :P | 08:12 |
| openstackgerrit | Steve Martinelli proposed openstack/keystone: remove deprecated revoke_by_expiration function https://review.openstack.org/271135 | 08:13 |
| stevemar | jamielennox: i'll hit up that oslo patch tomorrow, bed time for me | 08:16 |
| jamielennox | stevemar: yea, whenever you can - thanks | 08:16 |
| jamielennox | stevemar: night | 08:16 |
| *** EinstCrazy has joined #openstack-keystone | 08:30 | |
| openstackgerrit | Merged openstack/keystone: Revert "skip test_get_token_id_error_handling to get gate passing" https://review.openstack.org/271851 | 08:39 |
| openstackgerrit | Merged openstack/keystone: Remove duplicate LDAP test class https://review.openstack.org/271106 | 08:39 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/keystone: Updating sample configuration file https://review.openstack.org/269479 | 08:41 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/keystone: Updating sample configuration file https://review.openstack.org/269479 | 08:42 |
| *** fhubik has joined #openstack-keystone | 08:42 | |
| *** fhubik is now known as fhubik_brb | 08:42 | |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone: WIP/DNM: Use requst local caching https://review.openstack.org/271885 | 08:42 |
| *** roxanagh_ has joined #openstack-keystone | 08:48 | |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone: Deprecate keystone.common.kvs https://review.openstack.org/271948 | 08:49 |
| *** roxanagh_ has quit IRC | 08:52 | |
| *** fhubik_brb is now known as fhubik | 08:52 | |
| *** pnavarro has joined #openstack-keystone | 08:56 | |
| *** topol_ is now known as topol | 09:00 | |
| *** ChanServ sets mode: +v topol | 09:00 | |
| *** su_zhang has quit IRC | 09:00 | |
| *** fhubik has quit IRC | 09:06 | |
| openstackgerrit | Merged openstack/python-keystoneclient: Use positional library instead of local code https://review.openstack.org/268379 | 09:06 |
| openstackgerrit | Merged openstack/python-keystoneclient: use positional library instead of utils https://review.openstack.org/271906 | 09:07 |
| *** shoutm has joined #openstack-keystone | 09:07 | |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone: Removed deprecated revoke KVS backend https://review.openstack.org/267777 | 09:09 |
| *** shoutm_ has quit IRC | 09:10 | |
| *** tjcocozz has quit IRC | 09:12 | |
| *** bapalm has quit IRC | 09:12 | |
| *** shoutm_ has joined #openstack-keystone | 09:13 | |
| *** shoutm has quit IRC | 09:13 | |
| *** shoutm_ has quit IRC | 09:18 | |
| *** EinstCra_ has joined #openstack-keystone | 09:19 | |
| *** EinstCrazy has quit IRC | 09:21 | |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone: remove deprecated revoke_by_expiration function https://review.openstack.org/271135 | 09:22 |
| *** mhickey has joined #openstack-keystone | 09:22 | |
| *** jistr has joined #openstack-keystone | 09:23 | |
| odyssey4me | notmorgan thank you for your feedback on https://review.openstack.org/271357 :) | 09:23 |
| notmorgan | odyssey4me: happy to help | 09:25 |
| notmorgan | :) | 09:25 |
| *** mattt has joined #openstack-keystone | 09:25 | |
| openstackgerrit | Dave Chen proposed openstack/keystone: test_credential.py work with python34 https://review.openstack.org/271965 | 09:28 |
| odyssey4me | notmorgan and we're happy to find bugs :) | 09:31 |
| openstackgerrit | Dave Chen proposed openstack/keystone: test_credential.py work with python34 https://review.openstack.org/271965 | 09:32 |
| odyssey4me | and especially happy to find them within the development cycle, not during the stable cycle | 09:32 |
| *** markvoelker has joined #openstack-keystone | 09:45 | |
| *** markvoelker has quit IRC | 09:50 | |
| *** davechen has left #openstack-keystone | 09:54 | |
| *** vgridnev has joined #openstack-keystone | 09:56 | |
| arif-ali | Hi all, | 10:00 |
| arif-ali | was wondering the reason that keystone-salt-formula project wasn't maintained anymore? | 10:00 |
| *** roxanagh_ has joined #openstack-keystone | 10:00 | |
| *** roxanagh_ has quit IRC | 10:05 | |
| openstackgerrit | Dina Belova proposed openstack/keystone: Integrate OSprofiler in Keystone https://review.openstack.org/103368 | 10:06 |
| *** EinstCra_ has quit IRC | 10:08 | |
| *** EinstCrazy has joined #openstack-keystone | 10:18 | |
| *** jaosorior has quit IRC | 10:22 | |
| *** jaosorior has joined #openstack-keystone | 10:23 | |
| *** jaosorior has quit IRC | 10:29 | |
| *** jaosorior has joined #openstack-keystone | 10:34 | |
| *** tyagiprince has joined #openstack-keystone | 10:36 | |
| *** tyagiprince1 has joined #openstack-keystone | 10:43 | |
| *** tyagiprince has quit IRC | 10:43 | |
| *** tyagiprince1 is now known as tyagiprince | 10:43 | |
| *** tyagiprince1 has joined #openstack-keystone | 10:46 | |
| *** tyagiprince has quit IRC | 10:47 | |
| *** tyagiprince1 is now known as tyagiprince | 10:47 | |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone: WIP/DNM: Use requst local caching https://review.openstack.org/271885 | 10:49 |
| *** ajayaa has quit IRC | 10:49 | |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone: WIP/DNM: Use requst local caching [full cache] https://review.openstack.org/272007 | 10:50 |
| *** tyagiprince1 has joined #openstack-keystone | 10:50 | |
| *** tyagiprince has quit IRC | 10:52 | |
| *** tyagiprince1 has quit IRC | 10:54 | |
| notmorgan | arif-ali: because no one has stepped up to maintain it. Also we [upstream] afaik haven't had a team maintaining salt formulas | 10:55 |
| *** shoutm has joined #openstack-keystone | 10:55 | |
| notmorgan | arif-ali: if someone wants to step up, i'm sure no one will be upset by it. | 10:56 |
| *** bapalm has joined #openstack-keystone | 10:57 | |
| *** tjcocozz has joined #openstack-keystone | 10:57 | |
| *** tyagiprince has joined #openstack-keystone | 11:01 | |
| *** tyagiprince has quit IRC | 11:03 | |
| *** EinstCrazy has quit IRC | 11:22 | |
| *** EinstCrazy has joined #openstack-keystone | 11:23 | |
| arif-ali | notmorgan, thanks, I am about to convert all our configs to use keystone v3 API, and noticed it wasn't there, and did some hackery that allowed me to convert a v2 site v3, but not able to provision a v3 from scratch | 11:29 |
| arif-ali | I will have a look, and see if I can do to help that along, and bring it to life again | 11:29 |
| *** pnavarro is now known as pnavarro|lunch | 11:45 | |
| *** markvoelker has joined #openstack-keystone | 11:46 | |
| *** roxanagh_ has joined #openstack-keystone | 11:48 | |
| *** markvoelker has quit IRC | 11:51 | |
| *** roxanagh_ has quit IRC | 11:53 | |
| *** dims has joined #openstack-keystone | 11:57 | |
| *** ajayaa has joined #openstack-keystone | 12:07 | |
| openstackgerrit | Morgan Fainberg proposed openstack/keystone: WIP/DNM: Use requst local caching [full cache] https://review.openstack.org/272007 | 12:10 |
| *** dims_ has joined #openstack-keystone | 12:22 | |
| *** dims has quit IRC | 12:23 | |
| *** raildo-afk is now known as raildo | 12:25 | |
| *** hogepodge has joined #openstack-keystone | 12:27 | |
| *** gordc has joined #openstack-keystone | 12:30 | |
| *** hogepodge has quit IRC | 12:32 | |
| openstackgerrit | Dave Chen proposed openstack/keystone: test_credential.py work with python34 https://review.openstack.org/271965 | 12:44 |
| *** jdennis has joined #openstack-keystone | 12:44 | |
| *** ericksonsantos has quit IRC | 12:49 | |
| *** lhcheng has quit IRC | 12:53 | |
| *** pauloewerton has joined #openstack-keystone | 13:05 | |
| *** vgridnev has quit IRC | 13:05 | |
| *** vgridnev has joined #openstack-keystone | 13:08 | |
| *** markvoelker has joined #openstack-keystone | 13:16 | |
| *** eandersson has joined #openstack-keystone | 13:16 | |
| eandersson | Morning | 13:16 |
| eandersson | I am having an odd issue. We have two sites, A and B... and on Site B everything is fine, but on Site A authentication is failing with Key length is > 250. | 13:17 |
| eandersson | I see these warnings in Keystone | 13:19 |
| eandersson | > Fernet token created with length of 268 characters, which exceeds 255 characters | 13:20 |
| *** Nirupama has quit IRC | 13:23 | |
| *** jsavak has joined #openstack-keystone | 13:25 | |
| *** dims_ has quit IRC | 13:28 | |
| *** bill_az has joined #openstack-keystone | 13:31 | |
| *** edmondsw has joined #openstack-keystone | 13:32 | |
| *** pnavarro|lunch has quit IRC | 13:32 | |
| *** dims has joined #openstack-keystone | 13:33 | |
| *** pnavarro has joined #openstack-keystone | 13:34 | |
| *** roxanagh_ has joined #openstack-keystone | 13:36 | |
| *** dstanek has quit IRC | 13:41 | |
| *** fesp has joined #openstack-keystone | 13:42 | |
| *** dstanek has joined #openstack-keystone | 13:42 | |
| *** ChanServ sets mode: +v dstanek | 13:42 | |
| *** ninag has joined #openstack-keystone | 13:42 | |
| *** roxanagh_ has quit IRC | 13:42 | |
| *** dslev has joined #openstack-keystone | 13:52 | |
| *** mattt has left #openstack-keystone | 13:52 | |
| *** dslev_ has joined #openstack-keystone | 13:53 | |
| *** dslev has quit IRC | 13:57 | |
| *** fesp has quit IRC | 13:58 | |
| eandersson | It must be some sort of cache error. I disabled memcached in Nova and it worked. | 14:13 |
| openstackgerrit | Brant Knudson proposed openstack/keystone: Parameter to return audit ids only in revocation list https://review.openstack.org/260153 | 14:14 |
| *** vgridnev has quit IRC | 14:16 | |
| *** richm has joined #openstack-keystone | 14:16 | |
| *** vgridnev has joined #openstack-keystone | 14:18 | |
| *** davechen has joined #openstack-keystone | 14:19 | |
| openstackgerrit | Brant Knudson proposed openstack/python-keystoneclient: Remove Babel from requirements.txt https://review.openstack.org/272112 | 14:23 |
| *** hogepodge has joined #openstack-keystone | 14:28 | |
| *** mdavidson has joined #openstack-keystone | 14:29 | |
| *** davechen1 has joined #openstack-keystone | 14:31 | |
| lbragstad | eandersson are IDs in your deployment uuid4 format? Or are they non-uuid strings (maybe some from LDAP?) | 14:32 |
| *** hogepodge has quit IRC | 14:33 | |
| *** davechen has quit IRC | 14:33 | |
| *** shoutm has quit IRC | 14:34 | |
| *** hogepodge has joined #openstack-keystone | 14:34 | |
| openstackgerrit | Brant Knudson proposed openstack/keystonemiddleware: Remove Babel from requirements.txt https://review.openstack.org/272114 | 14:34 |
| eandersson | lbragstad: Yea, the user domain is using LDAP. | 14:34 |
| lbragstad | eandersson gotcha | 14:34 |
| eandersson | The odd thing is that if I change the region from A to B it works. | 14:35 |
| eandersson | Both are identical setups. | 14:35 |
| lbragstad | eandersson keystone will attempt to convert uuid.uuid4().hex (which is the format of our ID strings) to bytes - because they result in smaller tokens | 14:35 |
| lbragstad | when you have ID strings that are external - the result could be larger tokens because keystone isn't able to compress something non-uuid format to uuid.uuid4().bytes | 14:36 |
| stevemar | o/ | 14:37 |
| stevemar | morning | 14:37 |
| eandersson | So I did some debugging and the token at this stage is the same format (and length) token_ids, cached = self._token_cache.get(token) | 14:37 |
| eandersson | in the keystonemiddleware | 14:37 |
| lbragstad | eandersson so your tokens from region B are larger than tokens from region A? | 14:37 |
| eandersson | As far as I can see they are both identical. | 14:47 |
| eandersson | in length | 14:48 |
| lbragstad | eandersson ah - so tokens from one region aren't being validated properly? | 14:48 |
| eandersson | Yep | 14:48 |
| eandersson | We are actually testing Kilo to Liberty migration at the moment. | 14:48 |
| lbragstad | eandersson is there anything in the logs except the message about lenght/ | 14:48 |
| eandersson | http://paste.openstack.org/show/nvpQykrJU5KIFb4rOfYg/ | 14:50 |
| eandersson | I added a lot of additional logging, but nothing that stands out. Like I mentioned the "token" variable looks almost identical before _validate_token | 14:51 |
| eandersson | https://github.com/openstack/keystonemiddleware/blob/stable/kilo/keystonemiddleware/auth_token/__init__.py#L728 | 14:51 |
| lbragstad | eandersson ah - interesting. In keystone we don't actually fail the token creation if the length is greater than 250, we just emit a warning - https://github.com/openstack/keystone/blob/ce549fda8cadba4fa99dc800175a66e763adbb53/keystone/token/providers/fernet/token_formatters.py#L166-L175 | 14:51 |
| eandersson | Yea, this is a warning from the memcached library | 14:51 |
| eandersson | So maybe the memcached version? but the again why fail on only one side :p | 14:51 |
| lbragstad | eandersson that seems pretty self explanatory | 14:52 |
| lbragstad | eandersson are the memcached versions the same in both regions? | 14:52 |
| eandersson | Yes. | 14:52 |
| lbragstad | and they are configured the exact same? | 14:52 |
| eandersson | Both the same memcached library and binaries. | 14:52 |
| eandersson | Yes. | 14:52 |
| lbragstad | hm | 14:52 |
| lbragstad | notmorgan this is interesting ^ | 14:53 |
| eandersson | I even manually verified the code. | 14:53 |
| lbragstad | eandersson ok, and you did say that the tokens from both regions are the same length | 14:53 |
| eandersson | Removing the memcached line in Nova/Neutron fixed it :P | 14:53 |
| eandersson | Holdon, I'll add a len() to verify. | 14:54 |
| *** ngupta has joined #openstack-keystone | 14:54 | |
| eandersson | Yep, both are 268 characters. | 14:56 |
| *** pushkaru has joined #openstack-keystone | 14:57 | |
| lbragstad | eandersson that's strange - both regions A and B issue tokens of the same length, but region A works fine with memcached and region B doesn't. And, removing memcached from the equation in region B fixes the problem. | 14:58 |
| *** davechen1 has left #openstack-keystone | 15:03 | |
| lbragstad | that feels like a memcached issue? | 15:05 |
| lbragstad | where something is different between region A and region B | 15:05 |
| bknudson | we always used to hash the pki token when we put it in memcache. | 15:05 |
| *** rbak has joined #openstack-keystone | 15:07 | |
| *** vivekd has joined #openstack-keystone | 15:11 | |
| *** erhudy has joined #openstack-keystone | 15:12 | |
| *** GB21 has joined #openstack-keystone | 15:13 | |
| *** rderose has joined #openstack-keystone | 15:13 | |
| *** vgridnev has quit IRC | 15:18 | |
| *** vgridnev has joined #openstack-keystone | 15:19 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:20 | |
| eandersson | lbragstad, after debugging line by line I found the issue | 15:22 |
| eandersson | No clue how this happened | 15:22 |
| eandersson | but one side is running a slightly different version | 15:22 |
| eandersson | according to rpm they are both running the same version | 15:23 |
| bknudson | if fernet tokens can get larger than 250 chars we should hash those too | 15:23 |
| lbragstad | eandersson one region is running a slightly different version of memcached? | 15:23 |
| eandersson | nope, it was missing this commit https://github.com/openstack/keystonemiddleware/commit/518e9c3a7f34f7f4b86d82da0f9f3a2923358753 | 15:23 |
| eandersson | both servers are showing python-keystonemiddleware-1.5.1-1.el7.noarch | 15:24 |
| lbragstad | bknudson looks like that is what https://github.com/openstack/keystonemiddleware/commit/518e9c3a7f34f7f4b86d82da0f9f3a2923358753 did | 15:24 |
| eandersson | must have been a bad upgrade/downgrae or something | 15:24 |
| lbragstad | eandersson so - if you fix the version of keystonemiddleware, both regions work fine, right? | 15:25 |
| eandersson | yep | 15:25 |
| lbragstad | eandersson awesome | 15:26 |
| *** timcline has joined #openstack-keystone | 15:26 | |
| *** chlong has joined #openstack-keystone | 15:27 | |
| *** phalmos has joined #openstack-keystone | 15:29 | |
| openstackgerrit | Steve Martinelli proposed openstack/keystone: remove deprecated revoke_by_expiration function https://review.openstack.org/271135 | 15:34 |
| openstackgerrit | Steve Martinelli proposed openstack/keystone: remove KVS backend for keystone.contrib.revoke https://review.openstack.org/272134 | 15:38 |
| eandersson | lbragstad, so it looks like RDO does not have that patch for kilo yet. | 15:38 |
| lbragstad | eandersson ah | 15:39 |
| *** aix_ has quit IRC | 15:39 | |
| lbragstad | eandersson are you using RDO for one region and something else for the other? | 15:39 |
| eandersson | I think someone might have used pip to upgrade it manually | 15:39 |
| eandersson | which is why it didnt show up when checking rpm | 15:39 |
| *** chlong has quit IRC | 15:40 | |
| *** chlong has joined #openstack-keystone | 15:40 | |
| *** chlong has quit IRC | 15:43 | |
| *** simondodsley has joined #openstack-keystone | 15:43 | |
| *** chlong has joined #openstack-keystone | 15:44 | |
| *** dslev_ has quit IRC | 15:45 | |
| *** timcline has quit IRC | 15:46 | |
| *** slberger has joined #openstack-keystone | 15:47 | |
| *** timcline has joined #openstack-keystone | 15:48 | |
| *** doug-fish has joined #openstack-keystone | 15:50 | |
| notmorgan | so what is going on with the validationy things? | 15:52 |
| lbragstad | notmorgan eandersson was missing the backported patch you had to keystonemiddleware that hashed tokens, ensuring a 250 character length | 15:53 |
| eandersson | RDO currently comes with keystonemiddleware 1.5.1 which is missing https://github.com/openstack/keystonemiddleware/commit/518e9c3a7f34f7f4b86d82da0f9f3a2923358753 | 15:53 |
| notmorgan | lbragstad: ah | 15:53 |
| notmorgan | yep | 15:53 |
| notmorgan | that would do it | 15:53 |
| notmorgan | was going to say there was a hashing backport | 15:53 |
| notmorgan | ;) | 15:53 |
| eandersson | A rogue pip upgrade caused a great deal of frustration :p | 15:53 |
| notmorgan | lbragstad: https://review.openstack.org/#/c/272007/ ;) | 15:54 |
| jorge_munoz | @ayoung @stevemar When ever you guys have a chance, could you look at patch https://review.openstack.org/#/c/269824/. lbragstad told me your input would be good for these changes. | 15:54 |
| lbragstad | notmorgan ah nice! i'll check that out today | 15:55 |
| stevemar | jorge_munoz: sure thing boss | 15:55 |
| notmorgan | lbragstad: on rax vms unit tests were down to 200s runtime + setup | 15:55 |
| notmorgan | lbragstad: local laptop ~20s saving | 15:56 |
| notmorgan | lbragstad: some dsvms were as many at 10m savings | 15:56 |
| notmorgan | lbragstad: if we like it (cc dolphm ^) i can un wip it and then there is a minor change needed to keystone common.config to make it a standard thing | 15:56 |
| jorge_munoz | stevemar: thanks | 15:56 |
| eandersson | besides this issue, are there any known concerns with running Keystone/Liberty with everything else still running Kilo? | 15:57 |
| *** henrynash has joined #openstack-keystone | 15:57 | |
| *** ChanServ sets mode: +v henrynash | 15:57 | |
| notmorgan | eandersson: not really afaik | 15:57 |
| eandersson | awesome, been running great so far | 15:57 |
| notmorgan | typicaly keystone can be ahead of the rest of the services easily | 15:57 |
| eandersson | will make the upgrade to liberty a lot easier | 15:58 |
| *** markvoelker_ has joined #openstack-keystone | 16:01 | |
| *** markvoelker has quit IRC | 16:02 | |
| openstackgerrit | Lance Bragstad proposed openstack/keystone: Add checks for domain scoped data creep https://review.openstack.org/253671 | 16:02 |
| notmorgan | stevemar: https://review.openstack.org/#/c/271948/ found another deprecation | 16:04 |
| notmorgan | stevemar: needs minor massaging to pass | 16:05 |
| *** markvoelker has joined #openstack-keystone | 16:05 | |
| *** markvoelker_ has quit IRC | 16:07 | |
| *** vgridnev has quit IRC | 16:12 | |
| *** chlong has quit IRC | 16:15 | |
| *** belmoreira has quit IRC | 16:15 | |
| *** vivekd_ has joined #openstack-keystone | 16:15 | |
| *** chlong has joined #openstack-keystone | 16:15 | |
| *** vivekd has quit IRC | 16:17 | |
| *** vivekd_ is now known as vivekd | 16:17 | |
| *** vgridnev has joined #openstack-keystone | 16:20 | |
| *** woodster_ has joined #openstack-keystone | 16:23 | |
| *** tonytan4ever has joined #openstack-keystone | 16:23 | |
| *** jorge_munoz_ has joined #openstack-keystone | 16:28 | |
| *** jorge_munoz has quit IRC | 16:28 | |
| *** jorge_munoz_ is now known as jorge_munoz | 16:28 | |
| *** pgbridge has joined #openstack-keystone | 16:29 | |
| *** vgridnev has quit IRC | 16:30 | |
| ayoung | jorge_munoz, https://review.openstack.org/#/c/269824/9/keystone/token/providers/common.py looks wrong | 16:31 |
| *** su_zhang has joined #openstack-keystone | 16:31 | |
| ayoung | why are we looking at the roles for the user if it is a trust? | 16:31 |
| ayoung | if r in trust_role_ids and r not in roles_id: | 16:31 |
| *** rcernin has quit IRC | 16:31 | |
| *** jsavak has quit IRC | 16:31 | |
| ayoung | why is the and r not in roles_id in there? | 16:31 |
| *** jsavak has joined #openstack-keystone | 16:32 | |
| breton | ayoung: stevemar: https://review.openstack.org/#/c/258741/3 an easy one for you to re-approve | 16:33 |
| *** spandhe has joined #openstack-keystone | 16:36 | |
| stevemar | breton: thanks for the heads up | 16:37 |
| *** jsavak has quit IRC | 16:37 | |
| *** jsavak has joined #openstack-keystone | 16:39 | |
| *** spzala has joined #openstack-keystone | 16:41 | |
| *** freerunner has quit IRC | 16:44 | |
| jorge_munoz | @ayoung A check to avoid adding duplicate roles | 16:44 |
| *** freerunner has joined #openstack-keystone | 16:45 | |
| ayoung | jorge_munoz, put them in a set. DOn't compare them to the roles requested off the user. | 16:45 |
| *** tsymanczyk has joined #openstack-keystone | 16:45 | |
| *** tsymanczyk is now known as Guest67058 | 16:46 | |
| jorge_munoz | @ayoung Ok, I’ll make the change. | 16:46 |
| ayoung | jorge_munoz, no. I want to understand the rationale | 16:46 |
| ayoung | I might be wrong here | 16:46 |
| *** peter-hamilton has joined #openstack-keystone | 16:46 | |
| ayoung | why'd you do it that way? | 16:46 |
| ayoung | and if the tests pass there, it might be due to pure chance. but they do pass | 16:47 |
| *** mhickey has quit IRC | 16:48 | |
| jorge_munoz | @ayoung I added it because a user could use a scope token with trust and wanted to avoid adding existing roles, but I might be wrong. When a user authenticates with trust keystone, is the new trust token rescoped? | 16:50 |
| *** gyee has quit IRC | 16:52 | |
| *** gyee has joined #openstack-keystone | 16:56 | |
| *** ChanServ sets mode: +v gyee | 16:56 | |
| *** r-daneel has joined #openstack-keystone | 17:00 | |
| *** Madkiss has quit IRC | 17:00 | |
| *** su_zhang has quit IRC | 17:00 | |
| openstackgerrit | Steve Martinelli proposed openstack/keystone: remove deprecated revoke_by_expiration function https://review.openstack.org/271135 | 17:01 |
| openstackgerrit | Merged openstack/keystone: Removed deprecated revoke KVS backend https://review.openstack.org/267777 | 17:01 |
| *** chlong has quit IRC | 17:02 | |
| *** chlong has joined #openstack-keystone | 17:03 | |
| *** chlong has quit IRC | 17:04 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/keystone: Updating sample configuration file https://review.openstack.org/269479 | 17:04 |
| *** chlong has joined #openstack-keystone | 17:05 | |
| jorge_munoz | @ayoung Just tested it, you are right there is no reason why to check the roles that could have come from a token_domain or token_project. The trusted scope token should only have the roles from the trust. | 17:06 |
| *** amakarov has quit IRC | 17:08 | |
| ayoung | jorge_munoz, team meeting right now...back after it is done | 17:10 |
| notmorgan | stevemar: ugh... that moment you realize you stayed up till 3am cause insomnia... and code | 17:13 |
| *** GB21 has quit IRC | 17:17 | |
| stevemar | notmorgan: i know that feel very well | 17:22 |
| notmorgan | stevemar: but fun code out the other end | 17:22 |
| *** su_zhang has joined #openstack-keystone | 17:22 | |
| stevemar | notmorgan: more clean up for kvs and revoke | 17:22 |
| stevemar | damn kvs crap | 17:22 |
| notmorgan | no this | 17:22 |
| notmorgan | https://review.openstack.org/#/c/272007/ | 17:22 |
| *** _cjones_ has joined #openstack-keystone | 17:23 | |
| notmorgan | and i could speed it up a bit more if i moved to json. | 17:23 |
| notmorgan | vs. msgpack. | 17:23 |
| notmorgan | but msgpack was much much easier. | 17:24 |
| *** lhcheng has joined #openstack-keystone | 17:24 | |
| *** ChanServ sets mode: +v lhcheng | 17:24 | |
| *** ajayaa has quit IRC | 17:24 | |
| *** su_zhang has quit IRC | 17:26 | |
| *** Ephur has joined #openstack-keystone | 17:27 | |
| *** spzala has quit IRC | 17:29 | |
| *** spzala has joined #openstack-keystone | 17:30 | |
| *** jsavak has quit IRC | 17:30 | |
| *** spzala_ has joined #openstack-keystone | 17:30 | |
| *** Ephur has quit IRC | 17:32 | |
| *** spzala has quit IRC | 17:34 | |
| eandersson | Good old Domain-tokens | 17:34 |
| *** spzala_ has quit IRC | 17:35 | |
| *** vgridnev has joined #openstack-keystone | 17:35 | |
| openstackgerrit | henry-nash proposed openstack/keystone: Enhance manager list_role_assignments to support group listing https://review.openstack.org/265650 | 17:35 |
| eandersson | As a domain admin I am currently getting the following when trying to modify users under my domain/project "You are not authorized to perform the requested action: identity:list_role_assignments" | 17:35 |
| *** jsavak has joined #openstack-keystone | 17:35 | |
| eandersson | I modified the policy for list_role_assignments to take rule:admin_or_cloud_admin, but I assume that isn't best practices | 17:36 |
| eandersson | is this an issue with the latest Horizon patch, or just some incompatibility with the default v3domain policy? | 17:38 |
| *** jsavak has quit IRC | 17:39 | |
| *** jsavak has joined #openstack-keystone | 17:40 | |
| *** timcline has quit IRC | 17:42 | |
| *** jsavak has quit IRC | 17:45 | |
| *** jsavak has joined #openstack-keystone | 17:45 | |
| *** EinstCrazy has quit IRC | 17:46 | |
| *** agireud has quit IRC | 17:46 | |
| *** vgridnev has quit IRC | 17:47 | |
| *** vgridnev has joined #openstack-keystone | 17:47 | |
| *** agireud has joined #openstack-keystone | 17:48 | |
| *** Guest67058 has quit IRC | 17:49 | |
| *** ajayaa has joined #openstack-keystone | 17:50 | |
| *** jistr has quit IRC | 17:51 | |
| openstackgerrit | Merged openstack/python-keystoneclient: Replace TestResponse with requests_mock https://review.openstack.org/258741 | 17:52 |
| *** vivekd has quit IRC | 17:56 | |
| *** tsymanczyk has joined #openstack-keystone | 18:00 | |
| *** tsymanczyk is now known as Guest88772 | 18:00 | |
| *** henrynash has quit IRC | 18:01 | |
| *** timcline has joined #openstack-keystone | 18:01 | |
| *** browne has joined #openstack-keystone | 18:06 | |
| eandersson | So looks like this issue https://review.openstack.org/#/c/180846/ | 18:12 |
| eandersson | Anyone aware of an alternative fix? | 18:13 |
| *** vgridnev has quit IRC | 18:14 | |
| openstackgerrit | Lance Bragstad proposed openstack/keystone: Add checks for project scoped data creep to tests https://review.openstack.org/253670 | 18:22 |
| *** jaosorior has quit IRC | 18:23 | |
| *** su_zhang has joined #openstack-keystone | 18:23 | |
| openstackgerrit | Lance Bragstad proposed openstack/keystone: Reuse project scoped token check for trusts https://review.openstack.org/253672 | 18:24 |
| *** jaosorior has joined #openstack-keystone | 18:24 | |
| *** rderose has quit IRC | 18:25 | |
| stevemar | notmorgan: does nothing use the revoke model now? | 18:27 |
| stevemar | oh nvm, it's fine cause you kept the reference | 18:27 |
| lbragstad | jorge_munoz respun those ^ patches to have negative tests | 18:28 |
| lbragstad | for the data creep | 18:28 |
| *** chlong has quit IRC | 18:29 | |
| *** chlong has joined #openstack-keystone | 18:30 | |
| *** doug-fish has quit IRC | 18:30 | |
| openstackgerrit | Merged openstack/python-keystoneclient: Remove argparse from requirements https://review.openstack.org/270386 | 18:30 |
| openstackgerrit | Merged openstack/keystonemiddleware: Remove Babel from requirements.txt https://review.openstack.org/272114 | 18:31 |
| *** chlong has quit IRC | 18:32 | |
| *** chlong has joined #openstack-keystone | 18:32 | |
| *** daemontool has joined #openstack-keystone | 18:33 | |
| stevemar | lbragstad: long live stevermar | 18:33 |
| stevemar | lbragstad: dolphm heads up, mgagne has fernet questions | 18:34 |
| *** lhcheng_ has joined #openstack-keystone | 18:34 | |
| dolphm | mgagne: o/ | 18:34 |
| stevemar | saw him on -operators | 18:34 |
| lbragstad | what's up? | 18:36 |
| mgagne | lbragstad dolphm: I'm currently profiling fernet token validation with artificial latency at the database. We expect at least 100ms but I'm testing with 1s to exaggerate the effect. I found that token validation now takes forever (+1m) so I suspect a lot of SQL queries are executed in the process. | 18:36 |
| mgagne | what I found with ngrep: https://gist.github.com/mgagne/a80ee892ed099d0bc4ef | 18:36 |
| *** lhcheng has quit IRC | 18:36 | |
| lbragstad | mgagne those are all the calls made on a single validate token call? | 18:37 |
| mgagne | each single query takes at least 1s, there are ~75 of them, and ~30 for the connection pool: ping and rollback | 18:37 |
| mgagne | lbragstad a fresh never validated token | 18:37 |
| lbragstad | sql calls made on a single validate token call* | 18:37 |
| mgagne | lbragstad with all caching patches applied, it takes 0.04s with no query once token is validated once | 18:37 |
| lbragstad | oh - token create | 18:37 |
| mgagne | no create | 18:38 |
| mgagne | token validation | 18:38 |
| mgagne | curl http://identity:35357/v3/auth/tokens?nocatalog -H "X-Subject-Token: XXX" -H "X-Auth-Token: XXX" | 18:38 |
| lbragstad | oh - a token that hasn't had anything cached yet | 18:38 |
| mgagne | yes | 18:38 |
| lbragstad | wow | 18:38 |
| mgagne | token is created on a different keystone node. I'm testing against a 2nd node which is freshly started | 18:39 |
| mgagne | so what I found about those SELECT 1: https://github.com/openstack/oslo.db/blob/master/oslo_db/sqlalchemy/engines.py#L53-L82 | 18:39 |
| mgagne | the behavior is hardcoded in oslo_db based on pessimistic recommendations from sqlalchemy http://docs.sqlalchemy.org/en/latest/core/pooling.html#disconnect-handling-pessimistic | 18:39 |
| *** jsavak has quit IRC | 18:40 | |
| *** jsavak has joined #openstack-keystone | 18:40 | |
| mgagne | rollback are issued due to connection pool engine rollbacking anything that *could* still be in progress although the application is done with the connection and is returning it to the connection pool | 18:41 |
| dolphm | mgagne: there's obviously some redundant queries in there, but have you compared against uuid? | 18:41 |
| mgagne | lbragstad with mentioned queries, validation takes ~1m18s | 18:41 |
| mgagne | dolphm I didn't, I'm not interested in using UUID due to token persistence | 18:42 |
| dolphm | (solution to redundant queries is to actually use our "auth context", which is not fully populated nor consistently utilized) | 18:42 |
| mgagne | dolphm we don't want to store them anymore | 18:42 |
| dolphm | mgagne: so what are you comparing this 1m8s benchmark against? | 18:42 |
| mgagne | dolphm there is no way for keystone to get those info from the context as there is none afaik | 18:42 |
| mgagne | dolphm nothing for now. We are currently using PKI and are planning to move away from it | 18:43 |
| mgagne | dolphm in fact | 18:43 |
| dolphm | mgagne: have you benchmarked against PKI? | 18:43 |
| mgagne | dolphm yes, we did compare PKI with fernet | 18:43 |
| mgagne | dolphm let me dig the numbers | 18:43 |
| *** spzala has joined #openstack-keystone | 18:45 | |
| mgagne | a coworker did the tests but we can't find those numbers :-/ | 18:45 |
| *** jasonsb has quit IRC | 18:45 | |
| mgagne | we only tested with nova list and fernet. One is with a local database, no latency. The other with latency. | 18:45 |
| mgagne | Nova list: 2 sec vs 7 sec. Neutron net-list: 1.5 sec vs 6 sec. Keystone token-get: 0.6 sec vs 3 sec | 18:46 |
| *** EinstCrazy has joined #openstack-keystone | 18:46 | |
| mgagne | if we could have a way to only issue 1-2 queries, that would be great as I know we won't be able to not issue any queries at all | 18:50 |
| stevemar | looks like we have a test failure in the gate, use of self.skip instead of self.skipTest | 18:50 |
| lbragstad | stevemar yeah - i just hit that locally I think? | 18:51 |
| *** pcaruana has joined #openstack-keystone | 18:51 | |
| stevemar | lbragstad: i got a patch up, 1 sec | 18:51 |
| *** EinstCrazy has quit IRC | 18:52 | |
| openstackgerrit | Steve Martinelli proposed openstack/keystone: use self.skipTest instead of self.skip https://review.openstack.org/272223 | 18:52 |
| stevemar | lbragstad: ^ | 18:52 |
| lbragstad | stevemar awesome - thanks | 18:53 |
| lbragstad | stevemar i thought i hosed my environment | 18:54 |
| stevemar | lbragstad: looks like it just started | 18:54 |
| mgagne | but then you could have non-SQL backends for some resources so you can't really easily fetch all of them without checking which ones are SQL or not :-/ | 18:54 |
| lbragstad | stevemar was that caused by a bump of something? | 18:54 |
| lbragstad | mgagne how are you introducing the latency again? | 18:56 |
| mgagne | tc | 18:56 |
| mgagne | let me SSH into the node to get the command, forgot to remove latency so now SSH takes forever :-/ | 18:56 |
| mgagne | tc qdisc change dev eth0 root netem delay 1000ms | 18:57 |
| *** jaosorior has quit IRC | 18:57 | |
| lbragstad | you're adding a second worth of latency? | 18:57 |
| mgagne | yes, as mentioned above | 18:57 |
| dolphm | lbragstad: per query | 18:58 |
| lbragstad | hm ok | 18:58 |
| dolphm | i assume that includes SELECT 1; etc | 18:58 |
| mgagne | any packets I would say | 18:58 |
| dolphm | mgagne: is anything about your results surprising? | 18:58 |
| mgagne | so yea, side effects is 1s per query | 18:58 |
| stevemar | lbragstad: probably this: https://pypi.python.org/pypi/testtools | 18:58 |
| mgagne | dolphm the number of queries is surprising | 18:58 |
| stevemar | lbragstad: 1.9.0 released on 2016-01-25 | 18:58 |
| mgagne | dolphm as close to half could be removed | 18:58 |
| lbragstad | stevemar oh - that would make sense | 18:58 |
| dolphm | mgagne: are you interested in helping to refactor our backends and usage thereof to work towards eliminating redundant queries? | 18:59 |
| * dolphm runs to meeting | 18:59 | |
| lbragstad | mgagne actually - dstanek has a pretty cool idea of how to do that | 18:59 |
| mgagne | dolphm I already identified the source of those queries. Some are hardcoded in oslo_db based on sqlalchemy recommendations and some are builtin to sqlalchemy. | 19:00 |
| mgagne | low hanging fruit IMO are located close to sqlalchemy, not keystone itself | 19:00 |
| mgagne | but keystone could greatly benefit from better settings/optimisations | 19:01 |
| *** openstackgerrit has quit IRC | 19:02 | |
| *** openstackgerrit has joined #openstack-keystone | 19:02 | |
| openstackgerrit | Jorge Munoz proposed openstack/keystone: Fix trust redelegation and associated test https://review.openstack.org/269824 | 19:04 |
| *** pnavarro has quit IRC | 19:05 | |
| dstanek | mgagne: i don't think we can get rid of the "ping" queries | 19:06 |
| dstanek | mgagne: i was working on some code that would reduce some of the redundant application queries though | 19:07 |
| bknudson | we could mock the db to always return success for the ping queries | 19:07 |
| mgagne | dstanek this means all keystone operations will take at least twice the time to complete. | 19:07 |
| mgagne | bknudson I think this could be a setting | 19:07 |
| mgagne | to oslo_db | 19:07 |
| *** doug-fish has joined #openstack-keystone | 19:07 | |
| dstanek | mgagne: aren't those queries necessary for the way pooling works? | 19:07 |
| mgagne | pessimist_connection_ping=True/False | 19:07 |
| dstanek | mgagne: probably a good thing to talk to zzzeek about | 19:08 |
| mgagne | dstanek read sqlalchemy, it's a pessimist recommendation. I believe there could be a better way to do it | 19:08 |
| mgagne | http://docs.sqlalchemy.org/en/latest/core/pooling.html#disconnect-handling-pessimistic | 19:08 |
| mgagne | there is the optimistic way: issue the request anyway and hope for the best. | 19:09 |
| *** jsavak has quit IRC | 19:10 | |
| *** doug-fis_ has joined #openstack-keystone | 19:11 | |
| mgagne | but I don't know how to implement it. I suspect it's done this way due to the complexity introduced by transactional queries | 19:11 |
| *** jsavak has joined #openstack-keystone | 19:12 | |
| mgagne | but what I'm seeing is multiple readonly single-statement transactions | 19:12 |
| *** doug-fish has quit IRC | 19:12 | |
| *** doug-fi__ has joined #openstack-keystone | 19:12 | |
| stevemar | dolphm: dstanek can one of y'all punt this through: https://review.openstack.org/#/c/272223/ it's busting le gate | 19:12 |
| mgagne | so I'm not sure all the overhead introduced to make sure the transaction is well rollbacked and all is necessary | 19:12 |
| stevemar | i just looked at zuul, and it's passing py27 now | 19:12 |
| dstanek | stevemar: done | 19:13 |
| stevemar | ty | 19:14 |
| *** doug-fis_ has quit IRC | 19:15 | |
| *** doug-fish has joined #openstack-keystone | 19:16 | |
| *** doug-fi__ has quit IRC | 19:16 | |
| *** doug-fish has quit IRC | 19:17 | |
| *** doug-fish has joined #openstack-keystone | 19:17 | |
| openstackgerrit | Jorge Munoz proposed openstack/keystone: Fix trust redelegation and associated test https://review.openstack.org/269824 | 19:18 |
| notmorgan | lbragstad: ping. | 19:24 |
| notmorgan | Can't reply in gerrit cause new gerrit is broken on phones :( | 19:24 |
| *** dims has quit IRC | 19:28 | |
| notmorgan | mgagne: if we add in the role caching and I just proposed a fix that should effectively eliminate redundant queries per request via a thread.local cache. | 19:29 |
| notmorgan | mgagne: you should see performance benefits. | 19:30 |
| notmorgan | mgagne: we can also def use improvements on our drivers | 19:30 |
| *** pcaruana has quit IRC | 19:32 | |
| *** pcaruana has joined #openstack-keystone | 19:36 | |
| *** ajayaa has quit IRC | 19:37 | |
| *** pcaruana has quit IRC | 19:40 | |
| *** mhickey has joined #openstack-keystone | 19:42 | |
| *** pcaruana has joined #openstack-keystone | 19:43 | |
| *** rcernin has joined #openstack-keystone | 19:44 | |
| *** EinstCrazy has joined #openstack-keystone | 19:49 | |
| stevemar | doh, rechecked a patch but the fix isn't in yet | 19:50 |
| *** dslevin1 has joined #openstack-keystone | 19:53 | |
| *** EinstCrazy has quit IRC | 19:54 | |
| notmorgan | lbragstad: responded to your comments now that i am home | 19:54 |
| *** slberger1 has joined #openstack-keystone | 19:55 | |
| *** maxabidi has joined #openstack-keystone | 19:55 | |
| *** pcaruana has quit IRC | 19:56 | |
| *** slberger has quit IRC | 19:56 | |
| notmorgan | lbragstad: https://review.openstack.org/#/c/215715/ does the cross-worker cache invalidations | 20:01 |
| notmorgan | lbragstad: fwiw | 20:01 |
| *** jsavak has quit IRC | 20:02 | |
| *** jsavak has joined #openstack-keystone | 20:03 | |
| openstackgerrit | Steve Martinelli proposed openstack/python-keystoneclient: remove CLI from keystoneclient https://review.openstack.org/258181 | 20:22 |
| *** doug-fish has quit IRC | 20:22 | |
| lbragstad | notmorgan awesome - thanks! | 20:28 |
| notmorgan | lbragstad: the cross-process invalidates is ugly cause it patches internal interfaces | 20:29 |
| notmorgan | lbragstad: on dogpile | 20:30 |
| *** su_zhang has quit IRC | 20:30 | |
| *** su_zhang has joined #openstack-keystone | 20:31 | |
| *** su_zhang has quit IRC | 20:31 | |
| *** doug-fish has joined #openstack-keystone | 20:32 | |
| *** jsavak has quit IRC | 20:32 | |
| notmorgan | lbragstad: i expect to cover "do we want cross process invalidates" while i work on upstream fixes to dogpile | 20:33 |
| notmorgan | at the midcycle | 20:34 |
| *** pnavarro has joined #openstack-keystone | 20:36 | |
| openstackgerrit | Steve Martinelli proposed openstack/keystone: Remove eventlet support https://review.openstack.org/249486 | 20:36 |
| lbragstad | notmorgan that sounds good to me | 20:36 |
| notmorgan | and i'll seriously talk w/ zzzeek about co-maintaining dogpile after the midcycle | 20:37 |
| notmorgan | so we have a few more active eyes on it | 20:38 |
| *** timcline has quit IRC | 20:38 | |
| *** maxabidi has quit IRC | 20:45 | |
| *** timcline has joined #openstack-keystone | 20:45 | |
| *** pauloewerton has quit IRC | 20:49 | |
| *** jsavak has joined #openstack-keystone | 20:51 | |
| *** henrynash has joined #openstack-keystone | 20:52 | |
| *** ChanServ sets mode: +v henrynash | 20:52 | |
| *** jsavak has quit IRC | 20:54 | |
| *** jsavak has joined #openstack-keystone | 20:54 | |
| openstackgerrit | Brant Knudson proposed openstack/python-keystoneclient: Remove Babel from requirements.txt https://review.openstack.org/272112 | 20:55 |
| openstackgerrit | Merged openstack/keystone: use self.skipTest instead of self.skip https://review.openstack.org/272223 | 21:00 |
| *** raildo is now known as raildo-afk | 21:00 | |
| henrynash | notmorgan: hi | 21:01 |
| notmorgan | henrynash: heyya | 21:01 |
| bknudson | what caused self.skip to stop working? | 21:01 |
| notmorgan | bknudson: uhm. no idea, but .skipTest iirc is more correct | 21:01 |
| *** su_zhang has joined #openstack-keystone | 21:02 | |
| notmorgan | maybe .skip was deprecated | 21:02 |
| henrynash | notorgan: on the FK issue…..we can certainly do as we discussed…although I realized that we will already have an FK integretity link for every project, since every project (except root domains) now has a parent link to it’s ancestor | 21:02 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/keystone: Updating sample configuration file https://review.openstack.org/269479 | 21:03 |
| notmorgan | henrynash: still have the issue of the unique() constraint where nulls [in mysql] are not considered unique | 21:03 |
| notmorgan | which is debatable as to if it conforms to the ANSI standard | 21:03 |
| bknudson | new testtools? | 21:04 |
| notmorgan | bknudson: perhaps? | 21:04 |
| henrynash | notmorgan: agreed…we need a “special valus” of some type (hidden within the driver layer), the only question is whetehr we make that speack value be | 21:04 |
| henrynash | oops | 21:04 |
| notmorgan | right | 21:04 |
| henrynash | notmorgan: ..be used as an FK link to a “root”, or is just a avlue | 21:04 |
| notmorgan | i'd go with a FK since we can ensure you don't end up with non-special values | 21:05 |
| notmorgan | and randomly orphaned/inweirdstate projects | 21:05 |
| notmorgan | it just ensures [esp. if its a NOT NULL] it is always populated and sane. | 21:06 |
| notmorgan | and i like enforcement on the schema where we can | 21:06 |
| henrynash | notmorgan: that last point is true, it stops you have mutliple special values….I don;t think we get any extra integrity protection, since as I said, every non-domain project already has an FK to its parent | 21:07 |
| notmorgan | except we rely on domain_id for other things | 21:07 |
| notmorgan | if that was bogus it could result in strange behaviors | 21:07 |
| notmorgan | and out-of-tree drivers i'm not going to argue too much about. | 21:07 |
| *** chlong has quit IRC | 21:07 | |
| notmorgan | since it's on them to write enforcement but we should do our best to avoid those weird cases | 21:08 |
| *** su_zhang has quit IRC | 21:08 | |
| *** chlong has joined #openstack-keystone | 21:08 | |
| henrynash | notmorgan: indeed…ok, I think we have a plan…one slight issue is that we can’t add the FK we are suggesting until we add all the domains in as projects acting as domains, so I still may need to split up the “special value” part and the FK part, otherwise we end up with a v large patch | 21:09 |
| notmorgan | henrynash: unless we would rather chase up the tree to determine domain_id if domain_id is bogus. it seems like a FK is just easier | 21:09 |
| *** EinstCrazy has joined #openstack-keystone | 21:09 | |
| henrynash | notmorgan: wherever possible, let’s let the more hardened SQL products do our work for us | 21:09 |
| notmorgan | i'm ok with a migration being included in the patch that adds the new functionality | 21:10 |
| notmorgan | defer the move domains -> projects if at all possible until then? | 21:10 |
| notmorgan | rest of the scaffolding can be added before the final move | 21:10 |
| notmorgan | or make it a bigger patch? | 21:10 |
| notmorgan | i mean at some point we will have big-ish patches for feature swings like this | 21:10 |
| notmorgan | but we can work hard to minimize them | 21:11 |
| henrynash | notmorgan: yep. that has been the plan | 21:11 |
| stevemar | bknudson: yes, new testtools | 21:13 |
| stevemar | bknudson: released today | 21:13 |
| bknudson | I guess they don't follow semver | 21:13 |
| *** shaleh has joined #openstack-keystone | 21:13 | |
| stevemar | bknudson: they follow keystonemiddleware method of releasing | 21:13 |
| *** EinstCrazy has quit IRC | 21:14 | |
| bknudson | I assume stable is broken? | 21:14 |
| bknudson | oh, this is our own problem. | 21:15 |
| notmorgan | stevemar: break things and pray no one notices, screw version numbers? | 21:16 |
| notmorgan | stevemar: gee, sounds like openstack | 21:17 |
| bknudson | it's because we have the guard for calling deprecated function and that function is now deprecated. | 21:17 |
| *** jsavak has quit IRC | 21:18 | |
| *** jsavak has joined #openstack-keystone | 21:19 | |
| bknudson | working-as-designed | 21:20 |
| notmorgan | oi | 21:20 |
| notmorgan | c | 21:20 |
| *** spzala has quit IRC | 21:21 | |
| *** lhcheng has joined #openstack-keystone | 21:21 | |
| *** ChanServ sets mode: +v lhcheng | 21:21 | |
| *** lhcheng_ has quit IRC | 21:24 | |
| *** dims has joined #openstack-keystone | 21:27 | |
| *** shaleh has quit IRC | 21:29 | |
| bknudson | y, stable is broken too. I don't think we need the deprecated function call check in stable, but it's easy to just backport the change | 21:29 |
| bknudson | both liberty and kilo :( | 21:33 |
| *** iurygregory has quit IRC | 21:33 | |
| *** iurygregory has joined #openstack-keystone | 21:34 | |
| notmorgan | fun | 21:37 |
| mgagne | notmorgan reading your change now. I'm not sure what it changes, how I can test it or how it improves the scenario I mentioned earlier. | 21:38 |
| openstackgerrit | Lance Bragstad proposed openstack/keystone-specs: Add spec for multifactor authentication https://review.openstack.org/272287 | 21:38 |
| *** chlong has quit IRC | 21:39 | |
| notmorgan | mgagne: so my change stores every query we can potentially cache after the first time | 21:39 |
| mgagne | notmorgan I just tested again my scenario but I also tried to validate a 2nd new token for the same project/user and I know for sure service_provider and trusts aren't cached. | 21:39 |
| notmorgan | mgagne: so in a given request we will ony look up for example domain_id once | 21:39 |
| notmorgan | now, you do need to enable caching to make it happen | 21:39 |
| mgagne | I didn't hit this use case | 21:39 |
| *** chlong has joined #openstack-keystone | 21:39 | |
| mgagne | I have cache enabled already | 21:40 |
| notmorgan | ok so this just means we don't duplicate any queries that are cached to the backend | 21:40 |
| notmorgan | even to memcache | 21:40 |
| notmorgan | there are systems we have not enabled caching on within keystone | 21:40 |
| notmorgan | so it wont help there | 21:40 |
| mgagne | according to ngrep, cachable information are already cached whenever possible, except for trust and service_provider | 21:41 |
| notmorgan | somewhat | 21:41 |
| mgagne | and I'm not sure why ALL trusts are loaded, why is there not a more specific WHERE clause | 21:41 |
| notmorgan | what this does it also isolates the data to in-process rather than needing to hit memcache, making the number of calls to memcache go down | 21:41 |
| notmorgan | because of limited design on the trust backend | 21:42 |
| mgagne | I don't have any problem with memcached, memcached is local to our region and works very well | 21:42 |
| openstackgerrit | henry-nash proposed openstack/keystone: Projects acting as domains https://review.openstack.org/231289 | 21:42 |
| mgagne | and what a limit that is :O | 21:42 |
| notmorgan | all fixable | 21:42 |
| notmorgan | but adding caching is difficult because there are very few people who understand caching and the caching libs | 21:43 |
| *** Guest88772 has quit IRC | 21:43 | |
| mgagne | is there any design limitations? | 21:43 |
| notmorgan | maybe i'd need to go look | 21:43 |
| notmorgan | the APIs may assume some broad things we pass down to the driver | 21:43 |
| notmorgan | that makes it hard to cache | 21:43 |
| notmorgan | i am working on improving the cache-key generator as well in the upstream project so it is more flexible | 21:43 |
| mgagne | notmorgan there is already cache in keystone. telling nobody/dev understands it mean I would have to disable it completely as it would be unreliable or badly coded. | 21:43 |
| notmorgan | mgagne: there are very few people in openstack that understand caching | 21:44 |
| notmorgan | and how to write it | 21:44 |
| mgagne | or you meant there is only 2 people in the team that knows how to code it | 21:44 |
| notmorgan | more like 3-5 | 21:44 |
| notmorgan | it's why i try and spend a large amount of time reviewing any cache changes | 21:44 |
| notmorgan | i'm trying to help folks understand it better | 21:45 |
| notmorgan | so i'm not the only one who spends time adding it | 21:45 |
| mgagne | cool, thanks for that =) | 21:45 |
| notmorgan | there may be more but it's also limited in what folks are working on | 21:45 |
| notmorgan | writing code to manage caching is specialied and if you're not thinking about it regularly it takes longer to do | 21:45 |
| dstanek | notmorgan: i think a lot of that is because we make it much harder than it needs to be | 21:46 |
| notmorgan | caching is one of the hardest things to get right in computer science. [specifically cache coherency] | 21:46 |
| notmorgan | dstanek: part of it is that, part of it is people still don't understand how / when to invalidate | 21:46 |
| notmorgan | dstanek: and using a raw memcache interface doesn't make that easier. | 21:46 |
| notmorgan | and in most cases "eventual consistency" is a a disaster for how openstack is written | 21:47 |
| bknudson | if it's hard then we're going to be constantly breaking it | 21:47 |
| mgagne | what have I started... | 21:47 |
| bknudson | stop changing your catalog all the time. | 21:48 |
| notmorgan | bknudson: we do a reasonable job of testing our invalidation code | 21:48 |
| notmorgan | i think we've had 2-3 real bugs in caching | 21:48 |
| *** tsymanczyk has joined #openstack-keystone | 21:48 | |
| notmorgan | not inc. the keystonemiddleware "turn it off" type bug | 21:48 |
| *** tsymanczyk is now known as Guest83292 | 21:48 | |
| bknudson | we don't have a gate job running multiple keystone instances with caching | 21:49 |
| notmorgan | bknudson: soon | 21:49 |
| notmorgan | bknudson: soon it'll be every dsvm | 21:49 |
| notmorgan | have a change up for that | 21:49 |
| notmorgan | https://review.openstack.org/#/c/271892/ | 21:50 |
| notmorgan | since we have a shared cache, it's trivial to enable it now | 21:50 |
| notmorgan | bknudson: well every mitaka + | 21:51 |
| notmorgan | bknudson: doesn't mean it'll be easier to write the caching code. though i think it's gotten much much better than the first pass that was done | 21:53 |
| notmorgan | the hard part are the invalidates tbh | 21:53 |
| bknudson | some caching is easier than others | 21:53 |
| *** jsavak has quit IRC | 21:54 | |
| bknudson | caching the catalog should be easy | 21:54 |
| bknudson | (but it's not) | 21:54 |
| notmorgan | bknudson: yes. well part of that is poor design on our interfaces for the catalog | 21:54 |
| notmorgan | bknudson: but we can start fixing that. | 21:54 |
| *** jsavak has joined #openstack-keystone | 21:55 | |
| *** rderose has joined #openstack-keystone | 21:56 | |
| *** rderose has quit IRC | 21:58 | |
| *** jsavak has quit IRC | 22:00 | |
| *** ninag has quit IRC | 22:01 | |
| *** timcline has quit IRC | 22:01 | |
| *** aix_ has joined #openstack-keystone | 22:02 | |
| *** pnavarro has quit IRC | 22:07 | |
| *** su_zhang has joined #openstack-keystone | 22:07 | |
| *** mhickey has quit IRC | 22:10 | |
| *** Guest83292 has quit IRC | 22:15 | |
| *** dims has quit IRC | 22:16 | |
| *** dims has joined #openstack-keystone | 22:17 | |
| *** hongbin has joined #openstack-keystone | 22:17 | |
| *** tonytan4ever has quit IRC | 22:21 | |
| *** dims has quit IRC | 22:27 | |
| *** jsavak has joined #openstack-keystone | 22:30 | |
| *** gildub has joined #openstack-keystone | 22:31 | |
| *** rcernin has quit IRC | 22:31 | |
| *** tsymancz1k has joined #openstack-keystone | 22:35 | |
| stevemar | notmorgan: poke https://review.openstack.org/#/c/272134/ | 22:48 |
| stevemar | dstanek: poke for you too https://review.openstack.org/#/c/272134/ | 22:48 |
| *** jsavak has quit IRC | 22:49 | |
| dstanek | stevemar: need a bp? | 22:49 |
| stevemar | dstanek: it was left over from when you removed the main one :O | 22:49 |
| stevemar | i could link it if you want | 22:49 |
| openstackgerrit | Steve Martinelli proposed openstack/keystone: remove KVS backend for keystone.contrib.revoke https://review.openstack.org/272134 | 22:50 |
| stevemar | dstanek: done | 22:50 |
| dstanek | stevemar: i mean in the commit message | 22:50 |
| dstanek | thx :-) | 22:50 |
| *** jsavak has joined #openstack-keystone | 22:51 | |
| stevemar | np :) | 22:52 |
| openstackgerrit | henry-nash proposed openstack/keystone: Allow project domain_id to be nullable at the manager level https://review.openstack.org/264533 | 22:53 |
| *** peter-hamilton has quit IRC | 22:53 | |
| openstackgerrit | henry-nash proposed openstack/keystone: Verify project unique constraints for projects acting as domains https://review.openstack.org/158372 | 22:54 |
| openstackgerrit | henry-nash proposed openstack/keystone: Removes project.domain_id FK https://review.openstack.org/233274 | 22:55 |
| *** doug-fish has quit IRC | 22:56 | |
| openstackgerrit | henry-nash proposed openstack/keystone: Projects acting as domains https://review.openstack.org/231289 | 22:59 |
| *** jsavak has quit IRC | 22:59 | |
| *** iurygregory has quit IRC | 22:59 | |
| stevemar | notmorgan: so, looking at https://review.openstack.org/#/c/271948 | 23:00 |
| *** iurygregory has joined #openstack-keystone | 23:00 | |
| stevemar | notmorgan: looks like it's conflicting with the deprecation checks i have for LDAP identity create/update/delete | 23:00 |
| *** jsavak has joined #openstack-keystone | 23:00 | |
| stevemar | notmorgan: looks like it's failing here: https://github.com/openstack/keystone/blob/master/keystone/tests/unit/test_backend_ldap.py#L733-L745 | 23:01 |
| *** jsavak has quit IRC | 23:01 | |
| stevemar | notmorgan: because i emit a deprecation here: https://github.com/openstack/keystone/blob/master/keystone/identity/backends/ldap.py#L171-L173 | 23:01 |
| notmorgan | stevemar: sure | 23:01 |
| stevemar | notmorgan: i think you may be able to make the tests pass by checking whats in args and kwargs? args, _kwargs = mock_deprecator.call_args | 23:02 |
| * notmorgan shrugs | 23:02 | |
| notmorgan | i haven't circled back to look at it | 23:02 |
| notmorgan | since i wrote it in like 15 mins | 23:02 |
| stevemar | notmorgan: well i'm doing some circling for you :) | 23:02 |
| notmorgan | hehe | 23:02 |
| stevemar | just giving you a heads up! | 23:02 |
| *** chlong has quit IRC | 23:02 | |
| stevemar | bigger question is, why is it using that backend... | 23:03 |
| notmorgan | i ... don't want to know tbh | 23:03 |
| stevemar | it means something in our tests is using keystone.common.kvs | 23:03 |
| *** chlong has joined #openstack-keystone | 23:03 | |
| stevemar | so either we untangle it now or later when we try to remove it :) | 23:03 |
| notmorgan | well yes | 23:04 |
| notmorgan | core is | 23:04 |
| notmorgan | cause it needs to reset it | 23:04 |
| notmorgan | and test_kvs is | 23:04 |
| *** chlong has quit IRC | 23:04 | |
| *** phalmos has quit IRC | 23:05 | |
| notmorgan | and test_backend_kvs is | 23:05 |
| notmorgan | those are the only "kvs" strings in all of keystone's tests | 23:05 |
| notmorgan | well and test_kvs, which is expected | 23:05 |
| *** chlong has joined #openstack-keystone | 23:05 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:07 | |
| *** pushkaru has quit IRC | 23:15 | |
| *** pushkaru has joined #openstack-keystone | 23:16 | |
| mgagne | notmorgan just to clarify, is the invalidation proxy only proposed to add invalidation support for in-memory cache? What I mean is: does someone using memcached need this patch or is .invalidate() doing what's expected in that context? | 23:19 |
| notmorgan | mgagne: no that proxy does all the in-memory cache management | 23:20 |
| *** pushkaru has quit IRC | 23:20 | |
| notmorgan | mgagne: so it's not just for invalidate it also does the cache store/retrieve from the in-memory locaiton and falls through to the normal backend if it's not in memory | 23:20 |
| mgagne | notmorgan let me reword that question: with memcached, do I need this patch? | 23:20 |
| notmorgan | mgagne: i've seen a benefit even with memcache, it depends on how loaded your memcache server is. | 23:21 |
| mgagne | notmorgan I'm trying to read the function description and I have difficulties understanding its purpose (the one in dogpile.cache) | 23:22 |
| notmorgan | mgagne: since this is all in-memory for the life of the request, there is less open-socket-ask-memcache-for-data-reconstruct-data-to-in-mem-object | 23:22 |
| notmorgan | mgagne: basically this sits between the @memoize decorator and the memcache backend | 23:22 |
| notmorgan | and it provies a second teir cache for the request itself | 23:22 |
| *** hongbin has left #openstack-keystone | 23:23 | |
| openstackgerrit | Steve Martinelli proposed openstack/keystone: remove deprecated revoke_by_expiration function https://review.openstack.org/271135 | 23:23 |
| notmorgan | that only handles the data for that request at the end of the request the memory is freed and you're back at an empty cache. it has no bearing on memcache's caching (which happens as well) | 23:23 |
| notmorgan | so what this is designed to do is move the cached data in-process for the request. meaning that subsequent calls should be faster for the same lookup | 23:23 |
| mgagne | notmorgan alright, 2nd tier of cache. will someone only using memcached see staled cached entries? | 23:23 |
| notmorgan | no. | 23:23 |
| notmorgan | should be the same, just a closer-to-the-processing cache | 23:24 |
| notmorgan | since it is in-python and doesn't require network transit | 23:24 |
| mgagne | notmorgan makes perfect sense. A bit like the in-memory/per-request cache the Nova API introduced a couple of years ago to avoid round-trip to the database for extensions. | 23:25 |
| notmorgan | it costs you a little more CPU, a little more memory (gc will free this eventually) and less network io, when scaling up to many many many keystone processes it should help keep the memcache server's network usage down | 23:25 |
| mgagne | agreed | 23:26 |
| notmorgan | mgagne: sortof, though the nova cache is a little longer lived iirc than just the request | 23:26 |
| notmorgan | but not much | 23:26 |
| mgagne | it is only the request last time I checked | 23:26 |
| mgagne | since I introduced the same to Cinder =) | 23:26 |
| notmorgan | i thought thye used memorycache [in-mem dict] | 23:26 |
| notmorgan | vs. a thread.local type thing | 23:26 |
| mgagne | I think it's in the request context or something | 23:26 |
| notmorgan | then it's the same as what i'm doing here, except that keystone already has a lot of cache hooks | 23:27 |
| notmorgan | so the impact is potentially greater | 23:27 |
| mgagne | true | 23:27 |
| notmorgan | since it affects any place we already cache | 23:27 |
| notmorgan | by design | 23:27 |
| notmorgan | i did opt for a slightly more expensive serializer/deserializer for ease of code clarity | 23:28 |
| notmorgan | i could move to a much faster (10x) serializer/deseralizer with a little work | 23:28 |
| notmorgan | but we're talking 7 usec on vs 60 usec on a i7 5660u | 23:28 |
| mgagne | trying to cache service provider/federation (against kilo) https://git.dev.iweb.com/gist/mgagne/9ae6e8417dc95815a201 | 23:28 |
| *** chlong has quit IRC | 23:29 | |
| notmorgan | and i don't think 50 microseconds is worth battling over | 23:29 |
| mgagne | if the cost is only paid once per request and is not in the order of ms/s, I'm fine with it | 23:29 |
| notmorgan | it washes out even more in xeons and even full desktop i7s | 23:29 |
| mgagne | I'm battling over seconds now =) | 23:29 |
| notmorgan | well the deserialize is paied every time | 23:29 |
| notmorgan | because i didn't want to copy.deepcopy | 23:29 |
| *** chlong has joined #openstack-keystone | 23:29 | |
| notmorgan | that turned out to be nightmarishly hard to evaluate performance / benefit on | 23:30 |
| notmorgan | so i stash it in the thread.local as a msgpack | 23:30 |
| notmorgan | and deserialize on get | 23:30 |
| mgagne | =) | 23:30 |
| notmorgan | the reason is to avoid someone accidently doing .get_domain(id) ref['name'] = blah and now that cached value is "blah" named instead of "default" | 23:30 |
| notmorgan | oops | 23:30 |
| notmorgan | cost is about the same as the SQL-A object book-keeping or pickle (50% faster, but again 30 usec is "meh") | 23:31 |
| *** EinstCrazy has joined #openstack-keystone | 23:31 | |
| notmorgan | and the general gate runs are showing a few minutes to 10 minutes savings on the fastest nodes | 23:31 |
| notmorgan | and on the faster nodes making the keystone unit tests 200s runtime (real run - setup) vs 300 or 400 | 23:32 |
| notmorgan | it has a tangible benefit | 23:32 |
| notmorgan | just not ground breaking. | 23:32 |
| *** slberger1 has left #openstack-keystone | 23:32 | |
| notmorgan | and that is layerd on top of an actual full-cache | 23:32 |
| mgagne | I got 99 problems and in-memory cache performance ain't one =) | 23:32 |
| notmorgan | so anyway. will this help? likely will it help when i start applying this work to other openstack projects? yes even more so | 23:33 |
| notmorgan | but we will need to add more cache hooks into keystone for say trusts and fix the catalog one | 23:33 |
| notmorgan | and finish the role_assignment one | 23:33 |
| notmorgan | [catalog and role assignment are pending some chatting this week @ the midcycle) | 23:33 |
| mgagne | notmorgan unfortunately, it won't as all queries done by keystone are different when token is validated. You can see queries I captured with ngrep. | 23:33 |
| *** rbak has quit IRC | 23:34 | |
| notmorgan | mgagne: the query might be different but we also do a lot of bookkeeping around similar calls | 23:34 |
| mgagne | I already applied catalog and assignment cache patches. It works very well when a 2nd token for the same user/project is validated. it doesn't change much when the user/project hits the API for the very first time in the last ~10m | 23:34 |
| notmorgan | i would extend the resource cache to ~600s or 1200s tbh | 23:35 |
| notmorgan | vs 300 | 23:35 |
| mgagne | doesn't change the 1st hit | 23:35 |
| notmorgan | no | 23:35 |
| notmorgan | but first hit will always be a miss | 23:35 |
| mgagne | our users aren't super heavy users | 23:35 |
| notmorgan | always | 23:35 |
| notmorgan | then make the cache 86400 ;) | 23:35 |
| mgagne | so they don't hit the API every seconds/minutes | 23:35 |
| mgagne | notmorgan this is a verrry short term solution IMO =) | 23:36 |
| notmorgan | the only case where that doesn't work is token validation | 23:36 |
| mgagne | notmorgan we are talking about token validation here | 23:36 |
| *** EinstCrazy has quit IRC | 23:36 | |
| notmorgan | you should turn on middlewaretoken validation cache then | 23:36 |
| mgagne | notmorgan token is created in a single centralized keystone endpoint with 0 latency to database | 23:36 |
| mgagne | already done | 23:36 |
| *** gordc has quit IRC | 23:36 | |
| notmorgan | then you're doing it all 100% right | 23:37 |
| notmorgan | that cache should be ~300s | 23:37 |
| mgagne | I've done some homework first =) | 23:37 |
| notmorgan | max or so | 23:37 |
| notmorgan | the role_assignments and catalog cache can be very hight TTLs | 23:37 |
| notmorgan | and will make many of the hits better | 23:37 |
| notmorgan | like those can probably be in the order of "days" for the TTL | 23:38 |
| mgagne | will check on that side too | 23:38 |
| mgagne | I'm really looking into first that 1st hit performance issue | 23:38 |
| notmorgan | which means you have few "first hit" misses [since i assume users use the APIs more often than days] | 23:38 |
| mgagne | they don't use it often TBH | 23:38 |
| notmorgan | first hit is always going to be slow(er), and honestly i'm not sure why you're seeing minutes | 23:39 |
| notmorgan | for validation | 23:39 |
| notmorgan | something seems very broken if that is the case | 23:39 |
| mgagne | notmorgan I tested with fake latency: 1s. we expect 100ms. | 23:39 |
| notmorgan | oh | 23:39 |
| notmorgan | ooooh you're setting artificial latency | 23:39 |
| notmorgan | that can expand response time exponentially. | 23:40 |
| mgagne | yea, I explained it in my original question, might have forgot since or I was talking with someone else =) | 23:40 |
| notmorgan | i didn't see all the convos | 23:40 |
| mgagne | notmorgan yea and I find it unfortunate | 23:40 |
| notmorgan | i expect if you're legitmately at 1s+ latency per query on a DB you need to increase the cache in the DB and/or expand the hardware | 23:41 |
| mgagne | notmorgan story is: we use PKI with centralized keystones. We are evaluating Fernet with regional keystone nodes. Initial auth is still done from the central one but services will use regional nodes to validate tokens. | 23:41 |
| notmorgan | well good, cause PKI tokens are terribad | 23:41 |
| mgagne | notmorgan cache in DB or hardware won't fix your transatlantic link latency | 23:41 |
| notmorgan | actually >.> | 23:42 |
| notmorgan | you could solve this in a dirty way... | 23:42 |
| mgagne | notmorgan so we are testing the effect of moving to fernet and with the expected 100ms latency. we see 4-6x performance hit to the API | 23:42 |
| notmorgan | you could implement a proxy like i did that pushes the validated data to the transatlantic cache server :P | 23:42 |
| mgagne | notmorgan super expensive solution is with galera in all our regions. | 23:42 |
| mgagne | notmorgan I don't have the time budget for it now =) | 23:42 |
| notmorgan | so a design that we (some of us) are working on is a central galera | 23:42 |
| notmorgan | and region nodes are big ram boxes that are read-only | 23:43 |
| notmorgan | so queries are as fast as possible | 23:43 |
| notmorgan | writes are all central | 23:43 |
| mgagne | notmorgan agreed, that's the final solution | 23:43 |
| notmorgan | i'm planning on hacking in read-slave support into keystone in the next week or so | 23:43 |
| notmorgan | to support this | 23:43 |
| mgagne | notmorgan we are looking into it too | 23:43 |
| mgagne | notmorgan but found writes to still occur when validating, maybe due to periodic_tasks or I don't know what | 23:44 |
| *** david-lyle has quit IRC | 23:44 | |
| notmorgan | if someone signs out of horizon it'll write | 23:44 |
| notmorgan | and with PKI you def. will see writes sinc ehte service user needs an active token | 23:45 |
| notmorgan | so it'll issue/create a token | 23:45 |
| mgagne | writes will occur to our centralized keystone server as we don't plan on hooking it to our regional nodes. | 23:45 |
| notmorgan | as needed | 23:45 |
| mgagne | PKI is out once we fix the token validation issue we have | 23:45 |
| notmorgan | revocations need to be written to the central server if you want them at all | 23:45 |
| notmorgan | otherwise you can just turn them off (effectively) | 23:46 |
| notmorgan | but tokens will always be valid | 23:46 |
| notmorgan | until they expire | 23:46 |
| mgagne | they should be unless services actively revoke tokens, maybe Heat | 23:46 |
| notmorgan | if you issue a revoke at the regional keystone that revoke must be written to the central keystone | 23:46 |
| notmorgan | if you want that token to be invalidated everywhere | 23:46 |
| notmorgan | that was the only sticking point we were running into with our design | 23:47 |
| mgagne | the regional nodes are still hooked to the centralized database | 23:47 |
| *** edmondsw has quit IRC | 23:47 | |
| notmorgan | right | 23:47 |
| notmorgan | for writes | 23:47 |
| mgagne | you mean with slaves? | 23:47 |
| notmorgan | you can have a r/o slave | 23:47 |
| notmorgan | but any writes still need to go to the central cluster | 23:47 |
| notmorgan | and the main source of writes from a regional node is "logout" or "password change" | 23:48 |
| notmorgan | or some such | 23:48 |
| openstackgerrit | Steve Martinelli proposed openstack/keystone: Reuse project scoped token check for trusts https://review.openstack.org/253672 | 23:48 |
| notmorgan | where you need a event everywhere to revoke | 23:48 |
| mgagne | yes | 23:49 |
| notmorgan | it's hard to explain via irc/text | 23:49 |
| *** tsymancz1k has quit IRC | 23:50 | |
| mgagne | only "users" of regional nodes will be keystonemiddleware and services, we won't allow our API users to use it. So writes "should" go to central one | 23:51 |
| *** spandhe has left #openstack-keystone | 23:51 | |
| mgagne | only issue I could see is cache invalidation in regional nodes which will all have their own memcached server. | 23:52 |
| *** spandhe has joined #openstack-keystone | 23:54 | |
| spandhe | l | 23:54 |
| *** su_zhang has quit IRC | 23:57 | |
| *** EinstCrazy has joined #openstack-keystone | 23:57 | |
| *** bknudson has left #openstack-keystone | 23:57 | |
| *** chlong has quit IRC | 23:58 | |
| *** chlong has joined #openstack-keystone | 23:59 | |
| *** shoutm has joined #openstack-keystone | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!