Thursday, 2019-02-07

*** markvoelker has joined #openstack-keystone00:11
*** itlinux has joined #openstack-keystone00:26
*** itlinux has quit IRC00:33
*** itlinux has joined #openstack-keystone00:35
*** itlinux has quit IRC00:38
*** itlinux has joined #openstack-keystone00:39
*** markvoelker has quit IRC00:44
*** itlinux has quit IRC00:44
*** itlinux has joined #openstack-keystone00:46
*** itlinux has quit IRC00:48
*** ileixe has joined #openstack-keystone00:49
*** ileixe has quit IRC00:51
*** ileixe has joined #openstack-keystone00:51
*** itlinux has joined #openstack-keystone01:07
*** itlinux has quit IRC01:10
*** gyee has quit IRC01:22
*** dklyle_ has joined #openstack-keystone01:27
*** david-lyle has quit IRC01:31
*** itlinux has joined #openstack-keystone01:45
*** itlinux has quit IRC01:47
*** itlinux has joined #openstack-keystone01:49
*** itlinux has quit IRC01:52
*** itlinux has joined #openstack-keystone01:56
*** itlinux has quit IRC02:01
*** david-lyle has joined #openstack-keystone02:12
*** dklyle_ has quit IRC02:15
*** lbragstad has quit IRC02:26
*** lbragstad has joined #openstack-keystone02:33
*** ChanServ sets mode: +o lbragstad02:33
*** rcernin has joined #openstack-keystone03:07
*** Dinesh_Bhor has joined #openstack-keystone03:08
openstackgerritYang Youseok proposed openstack/keystonemiddleware master: Add auth invalidation in auth_token for identity endpoint update  https://review.openstack.org/63369503:15
*** adriant has quit IRC03:18
*** adriant has joined #openstack-keystone03:27
*** Dinesh_Bhor has quit IRC03:30
*** Dinesh_Bhor has joined #openstack-keystone03:39
*** itlinux has joined #openstack-keystone03:41
*** rcernin has quit IRC04:35
*** lbragstad has quit IRC04:38
*** erus1 has quit IRC04:38
*** erus1 has joined #openstack-keystone04:39
*** itlinux has quit IRC04:44
*** itlinux_ has joined #openstack-keystone04:45
*** itlinux_ has quit IRC04:45
*** Dinesh_Bhor has quit IRC04:48
*** itlinux has joined #openstack-keystone04:59
*** Dinesh_Bhor has joined #openstack-keystone05:00
*** itlinux has quit IRC05:03
*** itlinux has joined #openstack-keystone05:04
*** itlinux has quit IRC05:26
*** erus1 has quit IRC05:26
*** erus1 has joined #openstack-keystone05:26
*** itlinux has joined #openstack-keystone05:40
*** Ebukha has joined #openstack-keystone05:40
*** vishakha has joined #openstack-keystone05:55
*** adriant has quit IRC06:06
*** markvoelker has joined #openstack-keystone06:07
*** adriant has joined #openstack-keystone06:07
*** Ebukha has quit IRC06:10
*** markvoelker has quit IRC06:11
*** pcaruana has joined #openstack-keystone07:02
*** Ebukha has joined #openstack-keystone07:08
*** markvoelker has joined #openstack-keystone07:08
*** Ebukha has quit IRC07:30
*** markvoelker has quit IRC07:41
openstackgerritVishakha Agarwal proposed openstack/keystone master: Fix wrong example for direct_maps  https://review.openstack.org/63544407:46
*** itlinux has quit IRC07:50
*** itlinux has joined #openstack-keystone07:59
*** Emine has joined #openstack-keystone08:02
*** awalende has joined #openstack-keystone08:05
*** itlinux_ has joined #openstack-keystone08:06
*** itlinux has quit IRC08:09
*** tkajinam has quit IRC08:15
*** itlinux_ has quit IRC08:22
*** shyamb has joined #openstack-keystone08:30
*** markvoelker has joined #openstack-keystone08:38
*** xek__ has joined #openstack-keystone08:49
*** shyamb has quit IRC08:55
*** Ebukha has joined #openstack-keystone08:59
*** markvoelker has quit IRC09:12
*** shyamb has joined #openstack-keystone09:36
*** yan0s has joined #openstack-keystone10:03
*** shyamb has quit IRC10:05
*** markvoelker has joined #openstack-keystone10:09
*** shyamb has joined #openstack-keystone10:09
*** vishakha has quit IRC10:41
*** erus1 has quit IRC10:41
*** markvoelker has quit IRC10:41
*** Dinesh_Bhor has quit IRC10:41
*** erus1 has joined #openstack-keystone10:41
*** shyamb has quit IRC10:57
*** ileixe has quit IRC11:20
*** markvoelker has joined #openstack-keystone11:38
*** dave-mccowan has joined #openstack-keystone12:03
*** markvoelker has quit IRC12:11
*** mgheorghe has joined #openstack-keystone12:17
*** raildo has joined #openstack-keystone12:30
*** markvoelker has joined #openstack-keystone13:08
*** markvoelker has quit IRC13:41
*** raildo has quit IRC13:42
*** erus1 has quit IRC13:42
*** erus1 has joined #openstack-keystone13:43
*** imus has joined #openstack-keystone13:47
*** raildo has joined #openstack-keystone13:49
*** lbragstad has joined #openstack-keystone13:58
*** ChanServ sets mode: +o lbragstad13:58
*** jmlowe has quit IRC14:09
*** mgheorghe has quit IRC14:14
*** jmlowe has joined #openstack-keystone14:30
*** irclogbot_1 has joined #openstack-keystone14:30
*** markvoelker has joined #openstack-keystone14:38
*** raildo has quit IRC14:42
knikollao/14:45
lbragstad\o14:47
*** raildo has joined #openstack-keystone14:55
*** david-lyle has quit IRC14:57
*** Ebukha has quit IRC15:00
*** Ebukha has joined #openstack-keystone15:04
*** awalende has quit IRC15:04
*** awalende has joined #openstack-keystone15:04
*** awalende has quit IRC15:08
*** markvoelker has quit IRC15:11
*** dansmith has quit IRC15:13
*** imacdonn has quit IRC15:13
*** dansmith has joined #openstack-keystone15:14
gagehugoo/15:20
*** Ebukha has quit IRC15:25
kmalloco/15:46
kmalloc\o15:46
kmalloc /o\15:46
*** markvoelker has joined #openstack-keystone16:08
*** yan0s has quit IRC16:09
*** gyee has joined #openstack-keystone16:13
*** lbragstad has quit IRC16:15
*** lbragstad has joined #openstack-keystone16:16
*** ChanServ sets mode: +o lbragstad16:16
*** pcaruana has quit IRC16:21
*** markvoelker has quit IRC16:41
*** imus has quit IRC16:53
*** awalende has joined #openstack-keystone17:05
*** erus1 has quit IRC17:05
*** erus1 has joined #openstack-keystone17:05
*** awalende has quit IRC17:09
kmalloco/17:16
kmallocgoing to run out to get some breakfast and some food stock in case we have an extra snowy weekend (as is expected in the forecast)17:17
*** itlinux has joined #openstack-keystone17:19
*** itlinux_ has joined #openstack-keystone17:26
*** itlinux has quit IRC17:27
gagehugolbragstad: I will be out tomorrow, just a heads up17:29
*** dklyle has joined #openstack-keystone17:29
lbragstadsounds good17:29
lbragstadthanks for the heads up17:29
*** xek__ has quit IRC17:30
*** xek has joined #openstack-keystone17:30
*** jmlowe has quit IRC17:37
*** markvoelker has joined #openstack-keystone17:38
lbragstadi need a quick sanity check17:55
*** erus1 has quit IRC17:55
lbragstadwe use web servers to manage keystone processes17:55
lbragstadand we've removed eventlet support17:56
*** erus1 has joined #openstack-keystone17:56
lbragstadevery new request coming in instantiates a new request context to handle that request17:56
*** itlinux_ has quit IRC17:58
lbragstadwe only support the eventlet case for testing18:00
lbragstadhttps://git.openstack.org/cgit/openstack/keystone/tree/keystone/server/wsgi.py#n2218:01
*** erus1 has quit IRC18:01
lbragstadotherwise - it looks like we fire off one of those for every request18:01
lbragstadwhich loads backends, configs, etc...18:02
lbragstadif we wanted to address wxy-xiyuan's concern about loading jws keys only when they are changed18:02
*** erus1 has joined #openstack-keystone18:02
lbragstadwe'd have to preserve key repository state across processes/threads, right?18:02
*** aojea has joined #openstack-keystone18:03
*** markvoelker has quit IRC18:11
*** mvkr has quit IRC18:12
lbragstadah - nevermind...18:15
lbragstadlooks like we load a flask app and then return that to the web server?18:15
lbragstadwe load all backends and whatnot before returning the app to serve to the web server18:18
gagehugodoes the jws loading issue apply to fernet keys as well?18:36
lbragstadyeah18:43
lbragstadwe read the keys from the filesystem on every request18:43
lbragstadbut - we instantiate a provider for each process/thread18:44
*** erus1 has quit IRC18:44
lbragstadat least according to what i'm seeing locally18:44
*** erus1 has joined #openstack-keystone18:45
lbragstadand apparently we have a downstream team that hit performance issues reading keys from disk on every request18:46
lbragstadi suppose i could see that being a problem if you have keypairs for 100s on nodes18:46
lbragstadof nodes*18:46
lbragstadiiuc - the trick is going to be implementing something that watches for changes on the directory without reading files18:47
lbragstadto recreate in a local environment - make sure you disable caching and cache_on_issue18:49
lbragstadso you force the keys to be read every time18:49
lbragstadi was wondering why token validation wasn't being run - but it was because the token was already cached18:50
*** jmlowe has joined #openstack-keystone18:59
*** markvoelker has joined #openstack-keystone19:08
lbragstad`19:30
*** lbragstad has quit IRC19:32
*** lbragstad has joined #openstack-keystone19:32
*** ChanServ sets mode: +o lbragstad19:32
kmalloclbragstad: we don't use eventlet even for testing.19:37
kmallocwe use wsgiref19:37
kmalloclbragstad: yes, you need to assume each process (uwsgi is the runner in most cases) will maintain state.19:37
kmalloclbragstad: as long as mtime is enabled for the key repository or similar, you're fine and can stat the files/directory without needing to do the complete reload19:38
lbragstadright19:38
lbragstadi was testing os.path.getmtime() but it didn't seem to work the way i was expecting it to19:38
lbragstador i was using it wrong19:38
kmalloclbragstad: it is a bit weird.19:38
kmallocyou could also just be aggressive on cycling processes.19:39
lbragstadthat would be up to the web server though, right?19:39
kmallocyeah19:39
kmallocthe other thing that could be done is have an internal timer for reloading19:39
lbragstadhmm19:40
kmallocso 1) load on new process, 2) load after X period, 3) if unseen key, load with a delay if the same key is seen and still doesn't load in19:40
lbragstadwhat about using pynotify?19:40
kmallocpynotify may or may not work well.19:40
kmalloci've had mixed results with inotify like stuff19:41
kmallocso what i would do is keep the fingerprint of the key in cache (in-process), and if we haven't seen a key before refresh the data from the repo, if the key *still* doesn't exist, Negative-Cache it for ~5 or 10m19:42
lbragstadwhat do you generate the fingerprint off of?19:42
lbragstadthe file name?19:42
*** markvoelker has quit IRC19:42
kmallocor the key data?19:42
lbragstadmm19:42
kmallocthis is pki19:42
kmallocyou can get fingerprints for the .pem19:43
kmallocand should be something we can derive from the data in the JSE19:43
lbragstadit would be nice to have a reusable solution for fernet keys, too19:43
kmallocfernet would be a slightly different workflow, but ultimately doable19:43
kmallocif key is not found, try and load from the repo, set timout before we try and reload again19:44
kmallocmake the cache/fingerprint/nx-cache code backend specific19:44
kmallocand if it isn't implemented, reload on all requests19:44
kmallocsince Asym and Sym crypto work differently we can't derive a fingerprint from the ciphertext (necessarily) in the fernet/symetric cases19:45
lbragstadright19:45
lbragstadthe fingerprint would have to be something else19:45
kmallocin asym, you should always be able to derive the fingerprint, so we can be more efficient at caching.19:45
lbragstadlike a hash of the key contents or something19:45
kmallocright, now, if we encoded a key fingerprint (sha?) outside the ciphertext payload in fernet19:46
kmalloc(requires new formatter) we could leverage similar code, just fingerprint the keys.19:46
kmallocit's still going to be differnet code paths, but ultimately pretty straightforward19:47
lbragstadtoday - the fernet repository just loads a list of Fernet keys19:47
lbragstader - key contents that it passes as a list to cryptography19:47
kmallocright, so we'd load tuples in-memory, fingerprint: key19:47
lbragstadyeah19:47
lbragstadwe could have a key abstraction on top of that19:47
kmallocand if we put the fingerprint in the payload [larger fernet tokens] we could avoid needing to load the repo every request19:47
kmallocyeh19:47
kmalloci like this plan.19:48
kmalloci could probably mock this up pretty quickly19:48
kmallocfor JSE i see this as a big win.19:48
lbragstadok - so you have a cache of Key objects19:48
lbragstadthe expose a .contents property19:48
lbragstadand a .fingerprint property19:49
kmallocsomething like that19:49
lbragstadhow do we detect new changes to the directory without loading all the keys again (because that's what we do today and we're back to the same performance)19:49
kmallocand we check token-crypto fingerprint (fernet, data is outside of ciphertext; JSE do direct fingerprint)19:49
kmallocno if you see an unknown key, load the repo, if the key *still* doesn't exist, negatively cache it19:50
kmallocso the workflow is: Check if fingerprint is a loaded key, use it. check if it is negatively cached (and cache is still in-play), invalid, load repo and use key if it exists or negatively cache it19:51
kmallocwe always load the repo when we load it19:51
kmalloccatch all changes19:51
kmallocthe caveat is that we need an explicit timeout where we load the repo to cache keys removed from the repo19:51
kmallocso we need to set a window we're comfortable -- like 300s, where we will reload the repo *anyway*19:52
kmallocif we haven't reloaded due to new-key being found in a token19:52
lbragstadright - what we need is a thing that tells us of 1.) a key was added under a new name 2.) a key was added with the same name (contents changed) 3.) a key was removed19:52
kmallocsince we're leaning on a fingerprint of the key not file name, we don't care about case 219:53
kmalloccase 2 would be the same as case 1.19:53
lbragstadi was thinking that would be generic enough for fernet19:53
kmallocwe should place some cryptographic information about which key was used in the fernet payload (unencrypted) if we want to use similar code to JSE19:54
kmallocif not, we implement it as file-specific for fernet and do:19:54
kmalloc1) see if data decrypts with any key in memory, 2) load repo if no decryption (set timeout between forced loads), 3) guaranteed load after expiry of in-memory cache19:55
kmallocit would be different code paths and that is because fernet is fundamentally different code.19:55
lbragstadya19:56
kmalloci prefer the add some data about the key in the fernet payload so we can19:58
kmallocmake the process faster/more reliable19:58
kmallocif we are willing to expand token sizes (JSE) we should consider the same for fernet19:59
lbragstadso - the key fingerprint has to be derived from key contents20:01
lbragstadi think20:01
kmallocyeh20:01
lbragstadbecause it's possible for keys to be renamed20:01
kmallocit could be some kind of sha -- or something*20:02
lbragstadright20:02
kmallocit will expand the payload20:02
lbragstadok - i think i was missing that piece of information before20:02
kmallocsize20:02
lbragstadi wasn't assuming we would solve this with additional data from the token itself20:02
kmallocbut it will save loading from disk as much and speed up validation, since you don't have to try each key20:02
lbragstadi was assuming all directory state changes would get picked up by inspecting the filesystem20:03
kmallocyeah, lets not try and do that20:03
kmallocthe reason is it allows us to lean on tech like vault20:03
kmallocor any thing else to load the keys20:03
lbragstadsure20:03
lbragstadthat's a good point20:03
kmallocnot locking us into a on-disk repo or "write your own code"20:03
lbragstadalthough i was thinking a solution under that assumption would also be reuseable for knikolla's mutable config change20:04
kmallocpossibly20:04
*** erus1 has quit IRC20:04
*** erus1 has joined #openstack-keystone20:05
lbragstadok - security question20:06
kmallocsure20:06
lbragstadwe can't put the fingerprint in the payload of the fernet token20:06
lbragstadbecause that would mean we would have to cycle through the keys to decrypt it to figure out the fingerprint20:07
lbragstadat which point, we've already lost20:07
kmallocput it outside of the ciphertext20:07
lbragstadso - we have to keep it outside the ciphertext20:07
kmallocyeah20:07
kmallocit would be a new formatter20:07
lbragstadis it problematic to expose that to end users?20:07
kmalloci don't think so... if we use a secure 1-way-hash20:07
lbragstador is it no more problematic than expose ciphertext20:08
lbragstadyeah - i suppose20:08
kmallocwe could use a PBKDF and drop the salt after hashing if we wanted20:08
lbragstadciphertext is arguable more valuable to a bad actor than a one-way has20:08
lbragstadhash*20:08
kmallocit becomes just a way we identify the key in memory20:08
kmallocas long as we use the same data (consistently) we're solid.20:09
kmallocit could even be a partial hash or any number of things as long as it's something reproducable20:10
lbragstadthe new token formatter is going to have to handle both types of tokens for a release, and that's not assuming a fast forward upgrade case20:10
kmallocsure. thats easy though20:10
lbragstadanother security thing..20:10
lbragstadas an attacker20:11
lbragstadi could generate a whole bunch of tokens20:11
lbragstadand group tokens by their owning key20:11
lbragstadwhich gives me a group of cipher text that i know was create with the same token20:11
lbragstads/token/key/20:11
kmallocyep. but that is something you can assume within a given window anyway20:12
lbragstad(e.g., isolating tokens issued by a specific keystone service behind a load balancer)20:12
kmallocknowing how fernet works.20:12
kmallocunless someone does the weird thing where they have multiple keystones using the same keys in different orders20:13
*** erus1 has quit IRC20:13
lbragstadi suppose that would be more applicable to jws20:13
*** erus1 has joined #openstack-keystone20:13
lbragstadi could target a specific keystone service if i find tokens with the same fingerprint20:13
lbragstadserver*20:13
kmallocyeah20:13
kmallocasymmetric crypto assumes you know what key was used.20:14
kmallocwell some/many forms do20:15
lbragstadi think you can make that assumption between intended audiences (keystone servers)20:15
lbragstadsince they assume the token they are getting was signed with a token in their possession20:16
kmallocsymmetric crypto may assume a priori exchange of keys/knowing outside of the ciphertext what key was used.20:16
kmallocrather than communicating along with the ciphertext what key was used.20:16
kmallocin keystone we brute-force that by trying all of our keys20:16
lbragstadright20:16
lbragstadwell - that's an implementation detail of pyca/cryptography20:17
lbragstadbut yeah20:17
lbragstadi wouldn't be surprised if they have thought about this exact problem20:17
kmallocwith fernet, not expecting if anyone thought about this at this point20:18
kmallocit's pretty much dead.20:18
kmallocexcept for us20:18
lbragstadoh - i'm talking about the pyca folks20:21
kmallocoh yeah20:21
lbragstadi don't think the multi key fernet thing was a detail of the heroku specification20:21
kmallocI would prefer to push folks towards JSE as we get it implemented.20:21
kmalloci am also 100% ok with saying JSE does the "don't reload from disk every time"20:22
kmallocand fernet leans on pyca so we just do what they do20:22
kmallocit can totally be an implementation detail of the token formatter/provider20:22
lbragstadsure20:24
lbragstadinitially - i suppose20:24
* lbragstad heads to #cryptography quick 20:24
*** aojea has quit IRC20:26
lbragstadoops #cryptography-dev actually20:27
* knikolla will read back later. in a meeting20:33
*** blake has joined #openstack-keystone20:35
*** markvoelker has joined #openstack-keystone20:39
*** raildo has quit IRC20:46
*** markvoelker has quit IRC21:11
*** xek has quit IRC21:22
*** blake has quit IRC21:55
*** erus1 has quit IRC21:55
*** dave-mccowan has quit IRC21:56
*** erus1 has joined #openstack-keystone21:56
*** markvoelker has joined #openstack-keystone22:08
*** markvoelker has quit IRC22:42
*** tkajinam has joined #openstack-keystone22:55
*** Emine has quit IRC22:59
openstackgerritLance Bragstad proposed openstack/keystone master: Implement JWS token provider  https://review.openstack.org/61454923:02
openstackgerritLance Bragstad proposed openstack/keystone master: Clarify cache_on_issue configuration option help text  https://review.openstack.org/63569023:02
*** whoami-rajat has quit IRC23:03
openstackgerritLance Bragstad proposed openstack/keystone master: Clarify cache_on_issue configuration option help text  https://review.openstack.org/63569023:03
*** mvkr has joined #openstack-keystone23:03
*** takamatsu has quit IRC23:19
*** markvoelker has joined #openstack-keystone23:39

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!