| *** itlinux has quit IRC | 00:08 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/keystonemiddleware master: Updated from global requirements https://review.openstack.org/500005 | 00:19 |
|---|---|---|
| openstackgerrit | OpenStack Proposal Bot proposed openstack/pycadf master: Updated from global requirements https://review.openstack.org/470137 | 00:27 |
| *** Shunli has joined #openstack-keystone | 00:31 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/keystonemiddleware master: Updated from global requirements https://review.openstack.org/500005 | 00:35 |
| *** zhurong has joined #openstack-keystone | 00:41 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/pycadf master: Updated from global requirements https://review.openstack.org/470137 | 00:43 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/keystonemiddleware master: Updated from global requirements https://review.openstack.org/500005 | 01:00 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/pycadf master: Updated from global requirements https://review.openstack.org/470137 | 01:08 |
| *** aselius has quit IRC | 01:14 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/keystonemiddleware master: Updated from global requirements https://review.openstack.org/500005 | 01:16 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/pycadf master: Updated from global requirements https://review.openstack.org/470137 | 01:24 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/keystonemiddleware master: Updated from global requirements https://review.openstack.org/500005 | 02:30 |
| *** jamesbenson has joined #openstack-keystone | 02:31 | |
| *** jamesbenson has quit IRC | 02:35 | |
| *** aojea has joined #openstack-keystone | 02:37 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/pycadf master: Updated from global requirements https://review.openstack.org/470137 | 02:38 |
| *** aojea has quit IRC | 02:42 | |
| *** erlon has quit IRC | 02:46 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/keystonemiddleware master: Updated from global requirements https://review.openstack.org/500005 | 02:50 |
| *** thorst has quit IRC | 02:51 | |
| *** thorst has joined #openstack-keystone | 02:51 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 02:54 | |
| *** thorst has quit IRC | 02:56 | |
| *** Dinesh_Bhor has quit IRC | 02:57 | |
| openstackgerrit | Merged openstack/keystonemiddleware master: Fix gate error caused by mocked URLs https://review.openstack.org/504487 | 02:57 |
| *** Dinesh_Bhor has joined #openstack-keystone | 02:57 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/pycadf master: Updated from global requirements https://review.openstack.org/470137 | 02:58 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/keystonemiddleware master: Updated from global requirements https://review.openstack.org/500005 | 03:06 |
| *** dave-mcc_ has quit IRC | 03:09 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/pycadf master: Updated from global requirements https://review.openstack.org/470137 | 03:14 |
| *** itlinux has joined #openstack-keystone | 03:19 | |
| *** itlinux has quit IRC | 03:22 | |
| *** itlinux has joined #openstack-keystone | 03:23 | |
| *** Dinesh_Bhor has quit IRC | 03:24 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/keystonemiddleware master: Updated from global requirements https://review.openstack.org/500005 | 03:27 |
| *** Dinesh_Bhor has joined #openstack-keystone | 03:28 | |
| *** Dinesh_Bhor has quit IRC | 03:34 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/pycadf master: Updated from global requirements https://review.openstack.org/470137 | 03:35 |
| *** itlinux has quit IRC | 03:35 | |
| *** itlinux has joined #openstack-keystone | 03:37 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 03:46 | |
| *** thorst has joined #openstack-keystone | 03:53 | |
| *** itlinux has quit IRC | 04:00 | |
| *** mtreinish has quit IRC | 04:24 | |
| *** cfriesen has quit IRC | 04:25 | |
| *** cfriesen_ has joined #openstack-keystone | 04:25 | |
| *** mtreinish has joined #openstack-keystone | 04:34 | |
| *** nkinder has quit IRC | 05:17 | |
| *** aojea has joined #openstack-keystone | 05:43 | |
| *** cfriesen_ has quit IRC | 06:02 | |
| *** thorst has quit IRC | 06:03 | |
| *** jamesbenson has joined #openstack-keystone | 06:07 | |
| *** jamesbenson has quit IRC | 06:11 | |
| *** jaosorior has quit IRC | 06:12 | |
| *** pcaruana has joined #openstack-keystone | 06:20 | |
| *** zhurong has quit IRC | 06:22 | |
| *** jaosorior has joined #openstack-keystone | 06:35 | |
| *** rcernin has joined #openstack-keystone | 06:47 | |
| *** thorst has joined #openstack-keystone | 06:59 | |
| *** belmoreira has joined #openstack-keystone | 07:11 | |
| *** tesseract has joined #openstack-keystone | 07:14 | |
| *** belmoreira has quit IRC | 07:19 | |
| *** ioggstream has joined #openstack-keystone | 07:19 | |
| *** belmoreira has joined #openstack-keystone | 07:26 | |
| *** belmoreira has quit IRC | 07:29 | |
| *** belmoreira has joined #openstack-keystone | 07:45 | |
| *** belmoreira has quit IRC | 07:59 | |
| *** aloga has quit IRC | 08:04 | |
| *** aloga has joined #openstack-keystone | 08:04 | |
| *** markvoelker has quit IRC | 08:06 | |
| *** zhurong has joined #openstack-keystone | 08:20 | |
| *** mvk has quit IRC | 08:26 | |
| *** Shunli has quit IRC | 08:27 | |
| *** Shunli has joined #openstack-keystone | 08:28 | |
| *** belmoreira has joined #openstack-keystone | 08:29 | |
| *** belmoreira has quit IRC | 08:30 | |
| *** mvk has joined #openstack-keystone | 08:54 | |
| *** belmoreira has joined #openstack-keystone | 08:59 | |
| *** aojea_ has joined #openstack-keystone | 09:02 | |
| *** aojea has quit IRC | 09:05 | |
| *** belmoreira has quit IRC | 09:12 | |
| *** mvk has quit IRC | 09:15 | |
| *** shengping has joined #openstack-keystone | 09:19 | |
| *** shengping has quit IRC | 09:20 | |
| *** shengping has joined #openstack-keystone | 09:21 | |
| *** mvk has joined #openstack-keystone | 09:28 | |
| *** Shunli has quit IRC | 09:31 | |
| *** shengping has quit IRC | 09:35 | |
| *** jamesbenson has joined #openstack-keystone | 09:43 | |
| *** jamesbenson has quit IRC | 09:48 | |
| *** zhurong has quit IRC | 09:52 | |
| *** markvoelker has joined #openstack-keystone | 10:07 | |
| *** breton has quit IRC | 10:10 | |
| *** breton has joined #openstack-keystone | 10:11 | |
| *** zhurong has joined #openstack-keystone | 10:35 | |
| *** markvoelker has quit IRC | 10:41 | |
| *** nicolasbock has joined #openstack-keystone | 11:07 | |
| *** thorst has quit IRC | 11:31 | |
| *** zhurong has quit IRC | 11:35 | |
| *** zhurong has joined #openstack-keystone | 11:39 | |
| *** markvoelker has joined #openstack-keystone | 11:39 | |
| *** ioggstream has quit IRC | 11:39 | |
| *** nkinder has joined #openstack-keystone | 12:03 | |
| *** dave-mccowan has joined #openstack-keystone | 12:07 | |
| *** nkinder has quit IRC | 12:09 | |
| *** dave-mcc_ has joined #openstack-keystone | 12:10 | |
| *** thorst has joined #openstack-keystone | 12:11 | |
| *** dave-mccowan has quit IRC | 12:12 | |
| *** markvoelker has quit IRC | 12:12 | |
| *** edmondsw has joined #openstack-keystone | 12:14 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/keystonemiddleware master: Updated from global requirements https://review.openstack.org/500005 | 12:24 |
| *** raildo has joined #openstack-keystone | 12:28 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/pycadf master: Updated from global requirements https://review.openstack.org/470137 | 12:32 |
| *** markvoelker has joined #openstack-keystone | 12:32 | |
| *** shewless has joined #openstack-keystone | 12:36 | |
| shewless | Hello. I'm trying to improve the performance of interacting with the openstack APIs. For example I notice when I do a "openstack stack list" it takes 10 seconds to return | 12:37 |
| shewless | Must of this time is spend in keystone | 12:37 |
| shewless | with fernet token auth "openstack stack list" takes 10 seconds | 12:37 |
| shewless | with user/pass auth (ldap) the same command takes 5 seconds | 12:37 |
| shewless | I would have thought fernet tokens would have been faster | 12:37 |
| shewless | first: even 5 seconds is too long. any recommendations on how I can make this subsecond? | 12:37 |
| shewless | second: any hints to make fernet faster? I can play with lowering crypt_strength in keystone.conf perhaps.. | 12:38 |
| *** ioggstream has joined #openstack-keystone | 12:39 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/keystonemiddleware master: Updated from global requirements https://review.openstack.org/500005 | 12:40 |
| *** zhurong has quit IRC | 12:46 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/pycadf master: Updated from global requirements https://review.openstack.org/470137 | 12:47 |
| *** panbalag has joined #openstack-keystone | 12:50 | |
| *** panbalag has left #openstack-keystone | 12:50 | |
| *** panbalag has joined #openstack-keystone | 12:50 | |
| *** erlon has joined #openstack-keystone | 12:57 | |
| *** Dinesh_Bhor has quit IRC | 12:59 | |
| *** lucasxu has joined #openstack-keystone | 13:07 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/keystonemiddleware master: Updated from global requirements https://review.openstack.org/500005 | 13:08 |
| *** Dinesh_Bhor has joined #openstack-keystone | 13:08 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/pycadf master: Updated from global requirements https://review.openstack.org/470137 | 13:16 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/keystonemiddleware master: Updated from global requirements https://review.openstack.org/500005 | 13:24 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/pycadf master: Updated from global requirements https://review.openstack.org/470137 | 13:32 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/oslo.policy master: http/https check rules as stevedore extensions https://review.openstack.org/507098 | 13:39 |
| *** cfriesen_ has joined #openstack-keystone | 13:53 | |
| *** belmoreira has joined #openstack-keystone | 13:55 | |
| *** belmoreira has quit IRC | 14:01 | |
| *** belmoreira has joined #openstack-keystone | 14:04 | |
| *** raildo has quit IRC | 14:06 | |
| knikolla | o/ | 14:09 |
| *** raildo has joined #openstack-keystone | 14:09 | |
| *** jamesbenson has joined #openstack-keystone | 14:25 | |
| *** belmoreira has quit IRC | 14:28 | |
| *** lucasxu has quit IRC | 14:30 | |
| *** lbragstad has quit IRC | 14:44 | |
| *** lbragstad has joined #openstack-keystone | 14:54 | |
| *** ChanServ sets mode: +o lbragstad | 14:54 | |
| lbragstad | shewless: crypt_strength will certainly haven an impact on performance, but that would be independent of the token provider | 14:55 |
| lbragstad | shewless: what does your caching configuration look like? | 14:55 |
| *** gyee has joined #openstack-keystone | 14:57 | |
| *** gyee has quit IRC | 14:58 | |
| *** gyee has joined #openstack-keystone | 14:58 | |
| samueldmq | o/ | 15:12 |
| *** lucasxu has joined #openstack-keystone | 15:13 | |
| lbragstad | morning! | 15:13 |
| *** jamesbenson has quit IRC | 15:14 | |
| *** Suramya__ has joined #openstack-keystone | 15:15 | |
| *** jamesbenson has joined #openstack-keystone | 15:15 | |
| gagehugo | o/ | 15:15 |
| samueldmq | lbragstad: Suramya__ : hey ! o/ | 15:15 |
| samueldmq | lbragstad: so... Suramya__ is interested to work with us during next Outreachy internship | 15:16 |
| samueldmq | for documentation | 15:16 |
| samueldmq | she even went ahead and submitted a first patch https://review.openstack.org/#/c/505135/ | 15:16 |
| samueldmq | which is great :) | 15:16 |
| lbragstad | Suramya__: o/ welcome! | 15:16 |
| kmalloc | lbragstad: o/ | 15:17 |
| lbragstad | kmalloc: o/ | 15:17 |
| lbragstad | gagehugo: o/ | 15:17 |
| gagehugo | welcome Suramya__ o/ | 15:17 |
| samueldmq | lbragstad: Suramya__: just quick introductions :-) I have to go now, have a talk in 10 minutes | 15:17 |
| kmalloc | lbragstad: and expect bcrypt to be slower than sha512_crypt. | 15:17 |
| kmalloc | shewless: ^cc | 15:17 |
| kmalloc | if you're using pike | 15:17 |
| Suramya__ | hello everyone :).I am a potential outreachy applicant for this round interested to contribute to keystone project for documentation.I am in final year and will be available full time after dec. to work on the project :) | 15:20 |
| shewless | @lbragstad: do you mean the "memcached" section? | 15:21 |
| shewless | @lbragstad: I'm using fernet tokens.. would 1000 be noticably faster than 10000? Also I would like to ensure I have caching enabled and optimized but I'm not sure how to do that | 15:22 |
| lbragstad | shewless: 1000 or 10000 users? projects? | 15:23 |
| shewless | @lbragstad: I meant the crypt_strength value | 15:23 |
| lbragstad | oh - yes, i would expect that to make a signifiant difference, but kmalloc is more familiar with the different hashing algorithms and the impact of performance with respect to rounds | 15:23 |
| kmalloc | shewless: what release of openstack are you using? | 15:24 |
| shewless | @lbragstad, kmalloc: mitaka | 15:24 |
| lbragstad | (i believe he did document some of that in the configuration options when we updated keystone to support better hashing techniques) | 15:24 |
| kmalloc | ok, then my advice for a lot of advanced tuning isn't useful | 15:24 |
| shewless | kmalloc: doh | 15:24 |
| kmalloc | since i've been focused on bcrypt,scrypt, and pbkdf2 (changes that came in pike and have more performance impact than the algo used in mitaka and before, possibly much slower) | 15:25 |
| shewless | in keystone.conf I have my memcache servers set.. but the rest of the config is default | 15:25 |
| kmalloc | shewless: the default hashing rounds (crypt_strength) should be mostly ok for almost any environment in mitaka | 15:25 |
| shewless | kmalloc: it it common that fernet token auth would take twice as long as ldap auth? | 15:26 |
| shewless | also, I'm running a private cloud and I can sacrifice some security for performance | 15:26 |
| kmalloc | that seems like something weird is going on | 15:27 |
| kmalloc | lbragstad: didn't we have some massive performance fixes in newton and ocata? | 15:27 |
| kmalloc | lbragstad: for fernet | 15:27 |
| lbragstad | shewless: yes | 15:27 |
| lbragstad | kmalloc: yes | 15:27 |
| kmalloc | thats what i thought | 15:27 |
| lbragstad | shewless: if you switch to uuid tokens, do you see the same performance footpritn? | 15:28 |
| lbragstad | footprint* | 15:28 |
| shewless | @lbragstad: I haven't tried that. are uuid tokens stored in the database? | 15:28 |
| lbragstad | shewless: yes | 15:28 |
| lbragstad | shewless: it would point out if the performance impact you're seeing is directly associated to fernet tokens or not | 15:29 |
| lbragstad | if not - then tinkering with crypt strength might be more useful, if it is attributable to fernet tokens, we need to look at caching | 15:29 |
| shewless | @lbragstad: can we look at caching now? Do you mean the [cache] section in keystone.conf? | 15:30 |
| kmalloc | shewless: ok so caching can help a lot | 15:30 |
| kmalloc | you'll also want to make sure caching is enabled for the other services [nova, neutron, etc] in the authtoken config | 15:30 |
| lbragstad | ++ | 15:30 |
| kmalloc | and typically you'll want to make sure the non-keystone services share a memcache pool | 15:31 |
| lbragstad | you need to make sure caching is configured is a few different areas | 15:31 |
| kmalloc | [same memcache servers] as caching the token in nova will impact glance that way | 15:31 |
| lbragstad | keystone implements caching on a per-subsystem basis | 15:31 |
| kmalloc | keystone always uses it's own hash-key for caching, so.... you benefit only in keystone for that | 15:31 |
| shewless | okay. I do have my services sharing a memcache pool. | 15:31 |
| kmalloc | shewless: good! | 15:31 |
| lbragstad | (e.g. you can turn caching on for users and tokens but leave caching of catalogs off) | 15:31 |
| kmalloc | lbragstad: by default if you turn on caching in keystone, you get it for everything | 15:32 |
| kmalloc | thankfully. | 15:32 |
| shewless | how do I turn on caching in keystone? | 15:32 |
| lbragstad | kmalloc: yep | 15:32 |
| shewless | in the [cache] section? | 15:32 |
| kmalloc | shewless: in the [cache] section i think. | 15:32 |
| lbragstad | shewless: look in the [cache] section | 15:32 |
| * lbragstad finds a link | 15:32 | |
| * kmalloc admits it has been a while since configuring this | 15:32 | |
| shewless | okay.. the "enabled = false" section I assume has to be changed :) | 15:32 |
| kmalloc | lbragstad: can we talk about making memcache a hard-requirement (or at least some-kind-of-caching) sometime soon (like... Rocky?) | 15:32 |
| kmalloc | shewless: yep | 15:33 |
| kmalloc | enabled = True | 15:33 |
| shewless | and then I can set the memcache_servers in the [cache] section as well | 15:33 |
| kmalloc | if you have the memcache servers configured, it should be significantly faster. | 15:33 |
| kmalloc | yeah. | 15:33 |
| kmalloc | hm. hold on | 15:33 |
| kmalloc | lets make sure that is right | 15:33 |
| shewless | sure. | 15:33 |
| shewless | to clarify in keystone.conf there is a [memcache] section where I have my servers = set to my memcache servers | 15:34 |
| lbragstad | https://github.com/openstack/keystone/blob/fdb6adf055d72db17be76bd59a71e159116f3b96/etc/keystone.conf.sample#L518] | 15:34 |
| lbragstad | https://github.com/openstack/keystone/blob/fdb6adf055d72db17be76bd59a71e159116f3b96/etc/keystone.conf.sample#L518 | 15:34 |
| shewless | but I don't have anything set in the [cache] section | 15:34 |
| kmalloc | https://www.irccloud.com/pastebin/vqTxGxrs/ | 15:34 |
| kmalloc | the memcache section is old | 15:34 |
| kmalloc | and it really shouldn't be used. | 15:34 |
| kmalloc | you *can* use it, but it is there for backwards compat only | 15:34 |
| shewless | even on mitaka? | 15:35 |
| kmalloc | since we moved to dogpile. | 15:35 |
| kmalloc | which was... icehouse? | 15:35 |
| kmalloc | or juno, or kilo or something | 15:35 |
| kmalloc | it's been quite a while | 15:35 |
| kmalloc | if you were running something old enough that required the [memcache] section, you couldn't cache user data, etc | 15:36 |
| shewless | okay cool. So I just need to focus on configuring the [cache] section and I should see amazing stuff happen! | 15:36 |
| kmalloc | that is our general plan | 15:36 |
| kmalloc | since you're using LDAP auth --- here is a warning** | 15:36 |
| lbragstad | unicorns and rainbows around! | 15:36 |
| kmalloc | The cache will not reflect (assuming read only ldap) user changes | 15:36 |
| *** Suramya__ has quit IRC | 15:36 | |
| shewless | maybe getting ahead of myself here... but do I need to change my nova, neutron configs? | 15:36 |
| shewless | kmalloc: what is a "user change" i nthis context? | 15:37 |
| kmalloc | if you change it. i tmight lag by ~5m or so | 15:37 |
| kmalloc | update any user data | 15:37 |
| kmalloc | if ldap is read-only, since keystone doens't manage it | 15:37 |
| kmalloc | it can't invalidate on user changes | 15:37 |
| kmalloc | so the cache will potentially persist for _cache_ttl_ [not the option] before the change is reflected | 15:38 |
| kmalloc | we can work on nova/neutron/cinder/etc once keystone is happily caching | 15:38 |
| shewless | kmalloc: okay that's probably not a big deal | 15:38 |
| kmalloc | it usually isn't | 15:38 |
| kmalloc | but just be aware | 15:38 |
| kmalloc | it's the typical cache caveat for non-managed data within an app. | 15:39 |
| kmalloc | :) | 15:39 |
| shewless | kmalloc: give me a moment and I'll try that keystone change. Do I need to set the memcache_servers variable in the [cache] section? | 15:39 |
| kmalloc | let me confirm the change is correct for mitaka | 15:39 |
| kmalloc | sec | 15:39 |
| shewless | kmalloc: thanks the default args I see are different from the snippet I think | 15:39 |
| kmalloc | ok mitaka uses oslo_cache | 15:40 |
| kmalloc | so the snippet should be correct. | 15:40 |
| kmalloc | you'll need to change the "backend_argumenbt" to reflect your memcache server(s) | 15:40 |
| kmalloc | if you want | 15:41 |
| kmalloc | you can use memcache_servers | 15:41 |
| kmalloc | there too it looks like. | 15:41 |
| kmalloc | so | 15:41 |
| kmalloc | https://www.irccloud.com/pastebin/CJTO72Vz/ | 15:42 |
| kmalloc | also, are you running keystone in mod_wsgi? or something else? | 15:42 |
| shewless | mod_wsgi | 15:42 |
| kmalloc | ok | 15:43 |
| kmalloc | cool | 15:43 |
| kmalloc | you're 100% fine then, no other tweaks really needed | 15:43 |
| shewless | so.. don't set memcache_servers at all in [cache]? | 15:44 |
| kmalloc | you *can* | 15:44 |
| shewless | also.. the backend_argument url.. shoudl that be localhost or my memcached servers? | 15:44 |
| kmalloc | use memcache_servers, sorry we have it for compat and i gave bad advice | 15:44 |
| kmalloc | it's easier to use than backend_url | 15:44 |
| kmalloc | see my second snippet, and use your memcache server | 15:45 |
| kmalloc | not localhost | 15:45 |
| kmalloc | or 127.0.0.1 | 15:45 |
| shewless | I only see 1 snippet | 15:45 |
| kmalloc | https://www.irccloud.com/pastebin/CJTO72Vz/ | 15:46 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/keystonemiddleware master: Updated from global requirements https://review.openstack.org/500005 | 15:47 |
| kmalloc | the reason is you can use list args. backend_arg is ... mostly so you can pass things to say REDIS instead of memcache backends | 15:47 |
| shewless | kmalloc: cool. Let me try this | 15:47 |
| kmalloc | if you only have a single memcache, obviously you wont have multiple servers in the list | 15:48 |
| shewless | kmalloc: yes I only have a single one ATM.. Okay I tried the change, restarted apache, and am running my "openstack stack list" as a test. so far I don't see the improvement | 15:49 |
| shewless | hmm.. actually I think it shaved about 3 seconds total off of my 10 second request (for the token case) | 15:50 |
| kmalloc | that is a big improvement | 15:50 |
| shewless | yup | 15:50 |
| kmalloc | also you're hitting non-keystone things | 15:50 |
| shewless | and I think "heat" is a slightly bigger contributor now.. hard to tell for sure though | 15:50 |
| kmalloc | yeah heat is probably most of that now | 15:51 |
| kmalloc | but we can make sure you're caching keystone things (tokens) in heat, nova, etc as well | 15:51 |
| kmalloc | ok so in heat/nova/neutron/cinder/etc configs, where auth_token is configured | 15:53 |
| kmalloc | add memcache_servers=<memcache> | 15:55 |
| kmalloc | and you should see benefits (make sure the memcache is shared between non-keystone services) | 15:56 |
| kmalloc | and the order of servers in tha tlist is the same | 15:56 |
| kmalloc | that should help w/ non-keystone services | 15:56 |
| shewless | kmalloc: hmm. so in heat.conf I have memcache_servers already set | 15:57 |
| shewless | in this section: rom keystonemiddleware.auth_token | 15:58 |
| kmalloc | yeah | 15:58 |
| kmalloc | then you should already be benefitting there | 15:59 |
| kmalloc | it just forces caching of the keystone tokens for ~5m | 15:59 |
| kmalloc | saves extra round-trips to keystone, and it adds up | 15:59 |
| *** sbezverk has quit IRC | 16:00 | |
| shewless | kmalloc: thanks for the help so far. My "openstack stack list" takes about 7 seconds now for token. I'll try ldap now.. I bet it's faster.. Is there anything else I can do to speed it up before I through another curve ball at you? | 16:00 |
| kmalloc | upgrade to newton | 16:01 |
| kmalloc | or ocata | 16:01 |
| * kmalloc is serious | 16:01 | |
| *** jistr is now known as jistr|mtg | 16:01 | |
| kmalloc | there are some large speed improvements for keystone around fernet | 16:01 |
| shewless | kmalloc: working on it.. | 16:01 |
| kmalloc | i recommend ocata, | 16:01 |
| shewless | okay.. so ldap is now 3 seconds | 16:01 |
| kmalloc | keep in mind you can [usually] upgrade keystone before other services | 16:01 |
| shewless | kmalloc: interesting idea | 16:02 |
| kmalloc | we are mostly isolated from other services behind the rest api, some cases it is wonky, but 1-2 releases has historically be fine | 16:02 |
| shewless | so no other tricks to make fernet tokens faster? even at the risk of security? :) | 16:02 |
| kmalloc | not really | 16:02 |
| kmalloc | internal to keystone we added a lot of improvements in later releases | 16:02 |
| kmalloc | well... i mean you could hand-code a token provider that forgoes encryption and hmac... | 16:03 |
| kmalloc | it would be faster | 16:03 |
| kmalloc | >.> | 16:03 |
| kmalloc | <.< | 16:03 |
| lbragstad | so - did caching help? | 16:03 |
| shewless | kmalloc: I know some people that would think that is a good idea :) | 16:03 |
| kmalloc | we're trying to eliminate the myriad of config options for tuning (that doesn't benefit most deploys). | 16:04 |
| kmalloc | the goal is to have sane behavior that doesn't need tuning | 16:04 |
| shewless | @lbragstad: caching did help. shaved 2-3 seconds off of my openstack stack list command | 16:04 |
| kmalloc | lbragstad: i'll take a 30% improvement | 16:04 |
| *** rcernin has quit IRC | 16:05 | |
| shewless | so..I also have another region running.. using the same keystone. So the keystone benefits are there.. but the other regions heat won't be using the same shared memcache pool. any ideas? | 16:05 |
| lbragstad | shewless: do you have a lot of revocation events in the revocation_event table? | 16:05 |
| kmalloc | lbragstad: ah good point | 16:06 |
| shewless | also: are there any tweaks to be made in horizon to leverage caching? | 16:06 |
| shewless | @lbragstad: Let me look. | 16:06 |
| lbragstad | that made a huge difference in performance when we fixed that | 16:06 |
| kmalloc | not really on hoprizon | 16:06 |
| kmalloc | and as long as heat in the other region has it's own memcache it's fine | 16:06 |
| lbragstad | we did make changes to horizon so that it didn't revoke tokens on projects switches | 16:06 |
| lbragstad | which bloated the revocation event table and slowed token validation down... | 16:06 |
| kmalloc | lbragstad: yeah, that is mostly just a "deploy a later openstack" solution though | 16:06 |
| lbragstad | yeah - that's a newton improvement i think | 16:07 |
| lbragstad | i'd have to double check with robcresswell | 16:07 |
| shewless | @lbragstad: there are 12,000 rows in revocation_event | 16:07 |
| lbragstad | oooof | 16:07 |
| lbragstad | ouch | 16:07 |
| shewless | will clearing that make things faster? | 16:07 |
| lbragstad | that's probably contributing to slow performance | 16:07 |
| shewless | or is it the thing adding the entries that's causing the problem? | 16:08 |
| lbragstad | we've since indexed that table and stopped writing unnecessary events to it | 16:08 |
| * lbragstad fetches another link | 16:08 | |
| lbragstad | https://review.openstack.org/#/c/382107/ | 16:08 |
| lbragstad | ^ was a newton thing, too | 16:08 |
| lbragstad | performance numbers are in the commit message | 16:09 |
| shewless | hmm | 16:09 |
| lbragstad | the gist of it is that keystone persists a revocation event when a token is invalidate (e.g. password reset or deleting a specific token) | 16:09 |
| shewless | how do you even delete a token? | 16:10 |
| lbragstad | we used to pull all revocation events out of sql and compare them in python | 16:10 |
| lbragstad | DELETE /v3/auth/tokens with the token in the X-Subject-Token header | 16:10 |
| lbragstad | comparing in python was super slow and it used a tree structure | 16:10 |
| shewless | @lbragstad: I doubt my users are doing that.. is there something that grants/revokes in horizon? | 16:10 |
| lbragstad | shewless: whenever a user switches projecst | 16:10 |
| shewless | @lbragstad: I see | 16:11 |
| lbragstad | the previous project token is revoked by horizon | 16:11 |
| *** mvk has quit IRC | 16:11 | |
| lbragstad | or it was - until we fixed that in horizon | 16:11 |
| shewless | So.. can I delete all rows in there without causing a problem? | 16:11 |
| *** jamesbenson has quit IRC | 16:11 | |
| lbragstad | well - it will mean that tokens that were invalid might be valid again | 16:11 |
| shewless | I suppose I could index the table as well manually | 16:11 |
| lbragstad | shewless: yes - which is another thing we did upstream | 16:11 |
| lbragstad | https://review.openstack.org/#/q/topic:bug/1524030+(status:open+OR+status:merged) | 16:11 |
| lbragstad | https://review.openstack.org/#/c/376523/ | 16:12 |
| shewless | @lbragstad: thanks.. I'm just trying to think of a way to fix this quickly for Mitaka so we can benefit before we upgrade | 16:12 |
| lbragstad | we indexed the revocation_event table and we made a smarted sql query to give us revocation events directly instead of handling all the comparisons in python | 16:12 |
| lbragstad | smarter* | 16:12 |
| lbragstad | shewless: ack - you could go through and attempt to remove old revocation events | 16:13 |
| lbragstad | that would help to some extent | 16:13 |
| lbragstad | the worst possible case would be that a token that was previously invalid is now valid again until it expires | 16:13 |
| shewless | @lbragstad: okay I don't think that would be a big deal in my case | 16:13 |
| shewless | @lbragstad: do you think https://review.openstack.org/#/c/376523/ would apply relatively cleanly in mitaka? | 16:14 |
| kmalloc | lbragstad: truncate <revocation table> :P | 16:14 |
| shewless | kmalloc, @lbragstad: thank you for the help. the 30% improvement is a great start. I'd like to explore this revocation_event table thing a bit further. I need to go AFK at the moment but I'm really hoping there is a patch I can apply to improve the revocation_event interaction in mitaka | 16:16 |
| cfriesen_ | lbragstad: I spoke with you at the PTG about optionally restricting the service catalog when authenticating...here's the spec: https://review.openstack.org/#/c/505345/ | 16:17 |
| shewless | I'd love to know if I can safely truncate that table | 16:17 |
| shewless | I don't really have a security problem if there are some revoked tokens that are valid... but I'm not sure if there are functionality problems if I do that | 16:17 |
| *** jistr|mtg is now known as jistr | 16:18 | |
| kmalloc | shewless: be super careful with backports, if we didn't backport it to the release there may be edge cases you run into in future upgrades | 16:19 |
| kmalloc | especially if it affects the schema | 16:19 |
| lbragstad | shewless: ack - that sounds good, i'm unsure if that patch would be backportable | 16:24 |
| lbragstad | shewless: but you could manually manage the revocation event table if needed until you upgrade to Newton, which would contain all of those improvements | 16:25 |
| lbragstad | cfriesen_: awesome - thanks for posting the spec | 16:25 |
| lbragstad | cc kmalloc ^ you'll probably have opinions/suggestions there | 16:25 |
| *** r-daneel has joined #openstack-keystone | 16:27 | |
| kmalloc | lbragstad: oh that. | 17:00 |
| kmalloc | lbragstad: i have a lot of views on that | 17:00 |
| kmalloc | lbragstad: ah. that isn't what i thought it was. | 17:01 |
| kmalloc | lbragstad: it might actually be more compute intensive to limit the response fwiw | 17:01 |
| kmalloc | on the server side | 17:01 |
| kmalloc | on the client side, it might be less. | 17:01 |
| kmalloc | cfriesen_: ^cc | 17:01 |
| kmalloc | lbragstad: i'll do some more in depth review shortly | 17:02 |
| *** rcernin has joined #openstack-keystone | 17:11 | |
| *** nkinder has joined #openstack-keystone | 17:15 | |
| *** ayoung has quit IRC | 17:24 | |
| *** raildo has quit IRC | 17:25 | |
| *** spilla has joined #openstack-keystone | 17:29 | |
| *** ioggstream has quit IRC | 17:36 | |
| shewless | kmalloc: seriously can I truncate <revocation table> without killing my openstack? Worst case is my revoked tokens are no longer revoked? | 17:37 |
| *** raildo has joined #openstack-keystone | 17:37 | |
| *** tesseract has quit IRC | 17:37 | |
| kmalloc | shewless: you can. it means any tokens that were revoked that were still valid are no longer revoked | 17:40 |
| kmalloc | shewless: be very careful you're only truncating the revocation table | 17:40 |
| openstackgerrit | Kristi Nikolla proposed openstack/keystonemiddleware master: Document endpoint interface and region behavior https://review.openstack.org/505396 | 17:45 |
| *** jmlowe has quit IRC | 17:46 | |
| *** lbragstad has quit IRC | 17:47 | |
| shewless | kmalloc: just revocation_event right? | 17:50 |
| * cmurphy will miss the meeting today | 17:51 | |
| *** lbragstad has joined #openstack-keystone | 17:51 | |
| *** ChanServ sets mode: +o lbragstad | 17:51 | |
| *** panbalag has quit IRC | 17:53 | |
| kmalloc | shewless: uhm... i *think* so | 17:56 |
| kmalloc | shewless: sorry a little distracted atm | 17:56 |
| kmalloc | but that should be the right table | 17:57 |
| lbragstad | kmalloc: shewless they will be unrevoked until they expire | 17:57 |
| lbragstad | so if you token expiration is set to 1 hour and you have a token that was revoked 30 minutes ago | 17:57 |
| lbragstad | it will be valid for 30 minutes until it is expired if you truncate the revocation event table | 17:57 |
| lbragstad | keystone checks token expiration separately from revocation event comparisons | 17:58 |
| *** jamesbenson has joined #openstack-keystone | 18:00 | |
| *** jamesbenson has quit IRC | 18:04 | |
| *** aselius has joined #openstack-keystone | 18:07 | |
| *** ayoung has joined #openstack-keystone | 18:10 | |
| *** lucasxu has quit IRC | 18:11 | |
| *** dave-mcc_ is now known as dave-mccowan | 18:12 | |
| *** jmlowe has joined #openstack-keystone | 18:19 | |
| gagehugo | I have a meeting but I'll be back for office hours in about an hour | 18:19 |
| -openstackstatus- NOTICE: The infra team is continuing work to bring Zuul v3 online; expect service disruptions and please see https://docs.openstack.org/infra/manual/zuulv3.html for more information. | 18:24 | |
| *** lucasxu has joined #openstack-keystone | 18:52 | |
| *** ayoung has quit IRC | 18:55 | |
| *** jamesbenson has joined #openstack-keystone | 18:55 | |
| lbragstad | #startmeeting keystone-office-hours | 18:58 |
| openstack | Meeting started Tue Sep 26 18:58:04 2017 UTC and is due to finish in 60 minutes. The chair is lbragstad. Information about MeetBot at http://wiki.debian.org/MeetBot. | 18:58 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 18:58 |
| *** openstack changes topic to " (Meeting topic: keystone-office-hours)" | 18:58 | |
| *** ChanServ changes topic to "Queens release schedule: https://releases.openstack.org/queens/schedule.html | Meeting agenda: https://etherpad.openstack.org/p/keystone-weekly-meeting | Bugs that need triaging: http://bit.ly/2iJuN1h" | 18:58 | |
| openstack | The meeting name has been set to 'keystone_office_hours' | 18:58 |
| gagehugo | I think the gerrit changes broke the office hours bookmark | 19:00 |
| shewless | HOLY CRAP: truncate revocation_event | 19:01 |
| shewless | openstack stack list now takes <1s | 19:01 |
| shewless | token or ldap | 19:01 |
| shewless | THANK you kmalloc and @lbragstad | 19:01 |
| shewless | so good | 19:01 |
| shewless | much better than 10 seconds | 19:01 |
| gagehugo | https://review.openstack.org/#/c/504084/ stable/pike doc change that closes 2 bugs | 19:01 |
| *** ianw|pto is now known as ianw | 19:01 | |
| *** spilla has quit IRC | 19:03 | |
| lbragstad | shewless: awesome - that should be much more consistent once you have Newton deployed | 19:04 |
| lbragstad | since the index will be in place and we write a *lot* less events to that table | 19:04 |
| *** mvk has joined #openstack-keystone | 19:06 | |
| *** jaosorior has quit IRC | 19:09 | |
| hrybacki | https://review.openstack.org/#/c/507434/1 stable/octa backport that also closes bug | 19:19 |
| shewless | yes looking forward to newton/ocata. Almost done upgrading in our staging environment | 19:22 |
| *** rcernin has quit IRC | 19:34 | |
| *** d0ugal has joined #openstack-keystone | 19:41 | |
| *** d0ugal has quit IRC | 19:41 | |
| *** d0ugal has joined #openstack-keystone | 19:41 | |
| lbragstad | shewless: each revocation event has an attribute called 'revoked_at' | 19:44 |
| lbragstad | you could programmatically remove revocation events based on that attribute | 19:44 |
| lbragstad | if the revoked_at time is older than time.now() - CONF.token.expiration_time, then you should be good to safely remove the revocation event | 19:45 |
| shewless | @lbragstad: thanks! | 19:52 |
| *** jmlowe has quit IRC | 19:56 | |
| *** lucasxu has quit IRC | 19:57 | |
| *** pcaruana has quit IRC | 20:05 | |
| *** dave-mcc_ has joined #openstack-keystone | 20:10 | |
| *** ayoung has joined #openstack-keystone | 20:11 | |
| *** dave-mccowan has quit IRC | 20:12 | |
| *** dave-mccowan has joined #openstack-keystone | 20:23 | |
| *** belmoreira has joined #openstack-keystone | 20:25 | |
| *** dave-mcc_ has quit IRC | 20:25 | |
| *** jmlowe has joined #openstack-keystone | 20:30 | |
| kmalloc | lbragstad: or we could publish the driver that disables rev events | 20:45 |
| kmalloc | :P | 20:45 |
| lbragstad | kmalloc: driver that disabled revocation events? | 20:47 |
| kmalloc | lbragstad: we talked about it in the past | 20:51 |
| kmalloc | basically just didn't store rev events at all | 20:51 |
| kmalloc | for deployments that don't want it | 20:51 |
| *** itlinux has joined #openstack-keystone | 20:53 | |
| *** Suramya has joined #openstack-keystone | 21:04 | |
| *** jmlowe has quit IRC | 21:07 | |
| *** jmlowe has joined #openstack-keystone | 21:09 | |
| *** belmoreira has quit IRC | 21:12 | |
| *** thorst has quit IRC | 21:16 | |
| *** thorst has joined #openstack-keystone | 21:16 | |
| *** raildo has quit IRC | 21:17 | |
| *** jmlowe has quit IRC | 21:18 | |
| *** belmoreira has joined #openstack-keystone | 21:19 | |
| *** thorst has quit IRC | 21:21 | |
| *** belmoreira has quit IRC | 21:22 | |
| *** raildo has joined #openstack-keystone | 21:23 | |
| lbragstad | edmondsw: updated https://review.openstack.org/#/c/500141/4 | 21:32 |
| edmondsw | lbragstad ack | 21:33 |
| lbragstad | i tried bending my head around the best way to document the use cases | 21:33 |
| lbragstad | i ended up using examples to illustrate the point | 21:33 |
| *** r-daneel has quit IRC | 21:43 | |
| *** d0ugal has quit IRC | 21:44 | |
| *** edmondsw has quit IRC | 21:46 | |
| *** edmondsw_ has joined #openstack-keystone | 21:49 | |
| lbragstad | edmondsw_: done with https://review.openstack.org/#/c/500207/1 too | 21:53 |
| *** edmondsw_ has quit IRC | 21:54 | |
| openstackgerrit | Lance Bragstad proposed openstack/keystone-specs master: Clarify backlog instructions and add ideas dir https://review.openstack.org/505057 | 22:00 |
| *** raildo has quit IRC | 22:01 | |
| *** jamesbenson has quit IRC | 22:02 | |
| *** Suramya has quit IRC | 22:08 | |
| openstackgerrit | Gage Hugo proposed openstack/keystone master: Refactor test_backend_ldap tests https://review.openstack.org/507694 | 22:11 |
| *** aojea_ has quit IRC | 22:16 | |
| *** jmlowe has joined #openstack-keystone | 22:24 | |
| *** dave-mccowan has quit IRC | 22:36 | |
| *** itlinux has quit IRC | 22:37 | |
| *** lbragstad has quit IRC | 22:40 | |
| *** panbalag has joined #openstack-keystone | 22:41 | |
| *** thorst has joined #openstack-keystone | 23:04 | |
| *** thorst has quit IRC | 23:08 | |
| openstackgerrit | Gage Hugo proposed openstack/keystone master: WIP - Refactor test_backend_ldap tests https://review.openstack.org/507694 | 23:14 |
| *** panbalag has quit IRC | 23:26 | |
| *** jamesbenson has joined #openstack-keystone | 23:43 | |
| *** jamesbenson has quit IRC | 23:47 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!