| clarkb | mordred: ^ I think kjorg50's quetion might be answerable by you if those are pbrx built? | 00:02 |
|---|---|---|
| clarkb | kjorg50: while I don't know for sure my hunch is that the -base image is the base install image then the other three layer on top of that and set the CMD value to each of those commands | 00:03 |
| * mordred has no idea - I do not believe anybody is using pbrx for image builds anymore | 00:03 | |
| *** dmellado has quit IRC | 00:03 | |
| mordred | I'm guessing those are loci images? | 00:03 |
| *** dmellado has joined #openstack-keystone | 00:04 | |
| kjorg50 | clarkb mordred - thanks for the responses. I think you are right about the keystone-base image. I am referring to these kolla image definitions https://github.com/openstack/kolla/tree/master/docker/keystone | 00:14 |
| clarkb | kjorg50: you'll oprobably have better luck asking in the kolla hcannel | 00:14 |
| kjorg50 | I asked there too :) thanks | 00:15 |
| *** hoonetorg has quit IRC | 00:24 | |
| *** hoonetorg has joined #openstack-keystone | 00:26 | |
| *** markvoelker has joined #openstack-keystone | 00:49 | |
| *** ileixe has joined #openstack-keystone | 00:56 | |
| *** itlinux has joined #openstack-keystone | 00:57 | |
| *** whoami-rajat has joined #openstack-keystone | 01:09 | |
| *** gyee has quit IRC | 01:21 | |
| *** markvoelker has quit IRC | 01:22 | |
| *** kjorg50 has quit IRC | 01:34 | |
| *** itlinux_ has joined #openstack-keystone | 01:39 | |
| *** itlinux_ has quit IRC | 01:40 | |
| *** itlinux has quit IRC | 01:41 | |
| *** jhesketh has quit IRC | 01:57 | |
| *** lbragstad has quit IRC | 01:58 | |
| *** jhesketh has joined #openstack-keystone | 01:58 | |
| *** ileixe has quit IRC | 02:05 | |
| *** ileixe has joined #openstack-keystone | 02:08 | |
| *** markvoelker has joined #openstack-keystone | 02:19 | |
| *** lbragstad has joined #openstack-keystone | 02:51 | |
| *** ChanServ sets mode: +o lbragstad | 02:51 | |
| *** markvoelker has quit IRC | 02:52 | |
| *** markvoelker has joined #openstack-keystone | 03:49 | |
| *** markvoelker has quit IRC | 04:22 | |
| *** vishakha has joined #openstack-keystone | 05:11 | |
| *** shyamb has joined #openstack-keystone | 05:16 | |
| *** hoonetorg has quit IRC | 05:33 | |
| *** johnthetubaguy has quit IRC | 05:49 | |
| *** hoonetorg has joined #openstack-keystone | 05:50 | |
| *** johnthetubaguy has joined #openstack-keystone | 05:56 | |
| *** shyamb has quit IRC | 06:43 | |
| *** shyamb has joined #openstack-keystone | 06:48 | |
| *** markvoelker has joined #openstack-keystone | 06:49 | |
| *** lbragstad has quit IRC | 07:17 | |
| *** markvoelker has quit IRC | 07:21 | |
| *** shyamb has quit IRC | 07:33 | |
| *** shyamb has joined #openstack-keystone | 07:44 | |
| *** shyamb has quit IRC | 07:49 | |
| *** tkajinam has quit IRC | 08:09 | |
| *** awalende has joined #openstack-keystone | 08:15 | |
| *** markvoelker has joined #openstack-keystone | 08:19 | |
| *** pcaruana has joined #openstack-keystone | 08:29 | |
| *** shyamb has joined #openstack-keystone | 08:48 | |
| *** xek has joined #openstack-keystone | 08:49 | |
| *** markvoelker has quit IRC | 08:52 | |
| *** markvoelker has joined #openstack-keystone | 09:49 | |
| *** shyamb has quit IRC | 09:51 | |
| openstackgerrit | Merged openstack/keystone master: Drop py35 jobs https://review.openstack.org/639909 | 09:54 |
| *** shyamb has joined #openstack-keystone | 10:19 | |
| *** markvoelker has quit IRC | 10:22 | |
| *** shyamb has quit IRC | 10:40 | |
| openstackgerrit | Merged openstack/python-keystoneclient master: Make tests pass in 2020 https://review.openstack.org/640024 | 10:44 |
| *** xek has quit IRC | 10:58 | |
| *** xek_ has joined #openstack-keystone | 10:58 | |
| *** FlorianFa has quit IRC | 11:01 | |
| *** FlorianFa has joined #openstack-keystone | 11:02 | |
| *** pcaruana has quit IRC | 11:04 | |
| *** shyamb has joined #openstack-keystone | 11:13 | |
| *** markvoelker has joined #openstack-keystone | 11:19 | |
| *** shyamb has quit IRC | 11:21 | |
| *** shyamb has joined #openstack-keystone | 11:21 | |
| *** pcaruana has joined #openstack-keystone | 11:32 | |
| *** shyamb has quit IRC | 11:43 | |
| *** shyamb has joined #openstack-keystone | 11:47 | |
| *** dave-mccowan has joined #openstack-keystone | 11:47 | |
| *** markvoelker has quit IRC | 11:52 | |
| *** xek_ has quit IRC | 11:54 | |
| *** raildo has joined #openstack-keystone | 11:59 | |
| *** mvkr has joined #openstack-keystone | 12:13 | |
| *** markvoelker has joined #openstack-keystone | 12:49 | |
| *** mchlumsky has joined #openstack-keystone | 13:02 | |
| *** TheJulia_sick is now known as TheJulia | 13:05 | |
| *** shyamb has quit IRC | 13:15 | |
| *** markvoelker has quit IRC | 13:22 | |
| *** phasespace has joined #openstack-keystone | 13:36 | |
| phasespace | Getting errors like this: "NoMatches: No 'keystone.auth.saml2' driver found, looking for 'keystone.auth.plugins.mapped.Mapped'" | 13:41 |
| phasespace | Anyone know what the issue is? | 13:41 |
| phasespace | Saw it mentioned in this channel previously: http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-keystone.2018-09-22.log.html | 13:41 |
| phasespace | errr, kmalloc : you guys were discussing this issue. did you figure out what it was? | 13:42 |
| *** jamesmcarthur has joined #openstack-keystone | 13:45 | |
| *** pcaruana has quit IRC | 13:51 | |
| *** pcaruana has joined #openstack-keystone | 14:01 | |
| *** FlorianFa has quit IRC | 14:09 | |
| openstackgerrit | Colleen Murphy proposed openstack/keystone master: Add keystone's technical vision reflection https://review.openstack.org/641374 | 14:13 |
| *** jamesmcarthur has quit IRC | 14:18 | |
| *** markvoelker has joined #openstack-keystone | 14:19 | |
| *** lbragstad has joined #openstack-keystone | 14:20 | |
| *** ChanServ sets mode: +o lbragstad | 14:20 | |
| *** erus has joined #openstack-keystone | 14:34 | |
| erus | o/ | 14:34 |
| *** jamesmcarthur has joined #openstack-keystone | 14:35 | |
| *** jamesmcarthur has quit IRC | 14:36 | |
| *** jamesmcarthur_ has joined #openstack-keystone | 14:36 | |
| lbragstad | o/ | 14:42 |
| *** markvoelker has quit IRC | 14:52 | |
| knikolla | o/ | 14:59 |
| cmurphy | phasespace: we have a bug for that i think https://bugs.launchpad.net/keystone/+bug/1793845 but if you're just using the saml2 plugin all you need to do is add saml2 to [auth]/methods and not set saml2=anything | 15:10 |
| openstack | Launchpad bug 1793845 in OpenStack Identity (keystone) "Federation Protocol saml2 fails on Rocky" [Medium,Triaged] | 15:10 |
| *** awalende has quit IRC | 15:15 | |
| *** awalende has joined #openstack-keystone | 15:16 | |
| *** awalende has quit IRC | 15:20 | |
| phasespace | thanks, got it working | 15:28 |
| *** xek_ has joined #openstack-keystone | 15:30 | |
| *** xek__ has joined #openstack-keystone | 15:32 | |
| *** xek_ has quit IRC | 15:35 | |
| *** markvoelker has joined #openstack-keystone | 15:50 | |
| *** pcaruana has quit IRC | 15:53 | |
| *** erus has quit IRC | 15:56 | |
| *** erus has joined #openstack-keystone | 15:56 | |
| *** pcaruana has joined #openstack-keystone | 16:06 | |
| *** erus has quit IRC | 16:15 | |
| *** erus has joined #openstack-keystone | 16:19 | |
| *** markvoelker has quit IRC | 16:22 | |
| *** ileixe has quit IRC | 16:33 | |
| *** imacdonn has quit IRC | 16:36 | |
| *** imacdonn has joined #openstack-keystone | 16:36 | |
| *** pcaruana has quit IRC | 16:55 | |
| *** gyee has joined #openstack-keystone | 17:06 | |
| *** awalende has joined #openstack-keystone | 17:17 | |
| *** markvoelker has joined #openstack-keystone | 17:19 | |
| *** awalende has quit IRC | 17:22 | |
| knikolla | lbragstad: never has been doing code reviews on test cases been so easy thanks to "with self.test_client() as c" | 17:22 |
| kmalloc | knikolla: :) | 17:25 |
| kmalloc | glad to have been able to make that possible or at least semi-possible | 17:25 |
| knikolla | glory to kmalloc | 17:25 |
| kmalloc | flask has improved a few things for keystone | 17:26 |
| lbragstad | yeah - it saved a few key strokes for me | 17:44 |
| * lbragstad actually just summarized all the policy work and is about to send that note to the mL | 17:45 | |
| *** jamesmcarthur_ has quit IRC | 17:50 | |
| *** jamesmcarthur has joined #openstack-keystone | 17:50 | |
| *** jamesmcarthur has quit IRC | 17:50 | |
| *** jamesmcarthur has joined #openstack-keystone | 17:51 | |
| *** markvoelker has quit IRC | 17:52 | |
| *** markvoelker has joined #openstack-keystone | 18:50 | |
| *** jamesmcarthur has quit IRC | 18:50 | |
| *** jamesmcarthur has joined #openstack-keystone | 18:51 | |
| *** jamesmcarthur has quit IRC | 18:55 | |
| lbragstad | http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003552.html | 19:05 |
| openstackgerrit | Merged openstack/keystonemiddleware master: Fix debug tox environment https://review.openstack.org/640183 | 19:05 |
| * lbragstad finds lunch-like objects | 19:06 | |
| *** erus has quit IRC | 19:07 | |
| *** erus has joined #openstack-keystone | 19:07 | |
| *** erus has quit IRC | 19:08 | |
| *** markvoelker has quit IRC | 19:22 | |
| openstackgerrit | Lance Bragstad proposed openstack/keystone master: Add service developer documentation for scopes https://review.openstack.org/638563 | 19:33 |
| *** xek has joined #openstack-keystone | 19:42 | |
| *** xek has quit IRC | 19:44 | |
| *** xek_ has joined #openstack-keystone | 19:44 | |
| *** xek__ has quit IRC | 19:45 | |
| *** lbragstad has quit IRC | 20:00 | |
| *** lbragstad has joined #openstack-keystone | 20:00 | |
| *** ChanServ sets mode: +o lbragstad | 20:00 | |
| *** raildo has quit IRC | 20:04 | |
| *** raildo has joined #openstack-keystone | 20:08 | |
| *** markvoelker has joined #openstack-keystone | 20:19 | |
| openstackgerrit | Lance Bragstad proposed openstack/keystone master: Allow domain users to access the limit API https://review.openstack.org/621023 | 20:28 |
| openstackgerrit | Lance Bragstad proposed openstack/keystone master: Add tests for project users interacting with limits https://review.openstack.org/621024 | 20:28 |
| openstackgerrit | Lance Bragstad proposed openstack/keystone master: Remove limit policies from policy.v3cloudsample.json https://review.openstack.org/621025 | 20:28 |
| *** jamesmcarthur has joined #openstack-keystone | 20:30 | |
| *** vishakha has quit IRC | 20:35 | |
| *** markvoelker has quit IRC | 20:53 | |
| *** raildo has quit IRC | 20:59 | |
| *** irclogbot_1 has joined #openstack-keystone | 21:09 | |
| *** nkinder has joined #openstack-keystone | 21:09 | |
| openstackgerrit | Gage Hugo proposed openstack/keystone master: WIP - Add flask hook for authentication timings https://review.openstack.org/634826 | 21:13 |
| *** whoami-rajat has quit IRC | 21:18 | |
| *** jamesmcarthur has quit IRC | 21:24 | |
| *** jamesmcarthur has joined #openstack-keystone | 21:24 | |
| *** irclogbot_1 has quit IRC | 21:28 | |
| *** jamesmcarthur has quit IRC | 21:28 | |
| timothyb89 | hi all, bit of a performance question... should token revocation lists be cached? | 21:33 |
| timothyb89 | I've been debugging some api slowness and based on some profiles, it seems to be hitting the database for the revocation list 2x every request, and as far as I can tell never touches the cache | 21:35 |
| timothyb89 | for reference: https://drive.google.com/file/d/16U2KOJOa2q7Fu57l6zokmvF6IaVI2ro9/view?usp=sharing | 21:35 |
| lbragstad | timothyb89 interesting.. if you have ``keystone.conf [oslo_cache] caching=True`` then by default token revocation should be on | 21:37 |
| timothyb89 | yup, that's set, and everything else seems to be hitting the cache as it ought to | 21:37 |
| timothyb89 | also have a debug log with cache debugging enabled: http://paste.openstack.org/show/747377/ | 21:38 |
| timothyb89 | don't see anything involving 'revoke' or 'token_events', but perhaps that won't show up so obviously | 21:38 |
| lbragstad | so - this is the token revocation list API logic | 21:40 |
| lbragstad | it looks like we only cache one method - https://git.openstack.org/cgit/openstack/keystone/tree/keystone/revoke/core.py#n53 | 21:40 |
| lbragstad | for listing all token revocation events | 21:40 |
| lbragstad | but the token provider API calls a different method for validating a token against a set of revocation events | 21:42 |
| lbragstad | https://git.openstack.org/cgit/openstack/keystone/tree/keystone/token/provider.py#n133 | 21:42 |
| lbragstad | which isn't cached - https://git.openstack.org/cgit/openstack/keystone/tree/keystone/revoke/core.py#n125 | 21:42 |
| lbragstad | =/ | 21:42 |
| timothyb89 | ah, that would do it | 21:42 |
| lbragstad | misleading, though | 21:42 |
| lbragstad | i can understand the confusion | 21:42 |
| lbragstad | several releases ago we had even worse performance with checking tokens against a list of revocation events because we were doing everything in python | 21:43 |
| lbragstad | we took another approach to take the important data from the token and build smarter SQL queries so we leveraged more the database for that check | 21:43 |
| timothyb89 | possible followup question, maybe my profile data is misleading, but it looks like check_validation gets called twice per request? | 21:46 |
| lbragstad | what API are you calling? | 21:46 |
| timothyb89 | just listing all projects repeatedly in a benchmark | 21:46 |
| timothyb89 | apologies, check_revocation is called twice | 21:47 |
| * lbragstad squints at the performance chart | 21:48 | |
| lbragstad | ok - so middle of the page? left side and middle columns? | 21:49 |
| timothyb89 | recommend opening the svg in a browser if you aren't already, it's interactive and much more readable that way | 21:49 |
| timothyb89 | yup | 21:49 |
| lbragstad | oh - sweet | 21:50 |
| *** markvoelker has joined #openstack-keystone | 21:50 | |
| lbragstad | so 27% of the request is spent in check_revocation? | 21:50 |
| timothyb89 | the first time, yes | 21:50 |
| timothyb89 | with another call later costing an additional 22% | 21:51 |
| lbragstad | and 22.7% | 21:51 |
| lbragstad | got it | 21:51 |
| lbragstad | what tool did you use to generate this? | 21:51 |
| timothyb89 | https://github.com/benfred/py-spy | 21:51 |
| lbragstad | i'll make a note to read this | 21:53 |
| *** mchlumsky has quit IRC | 21:53 | |
| timothyb89 | good to hear - thanks! | 21:54 |
| lbragstad | hmm | 21:54 |
| lbragstad | validate_token is getting called twice, too | 21:54 |
| lbragstad | (which makes sense) | 21:54 |
| lbragstad | considering check_revocation is getting called twice and it's only used in that API | 21:55 |
| lbragstad | well - the first token validation is actually happening in middleware | 21:56 |
| lbragstad | https://git.openstack.org/cgit/openstack/keystone/tree/keystone/server/flask/request_processing/middleware/auth_context.py#n243 | 21:58 |
| lbragstad | ^ that gets invoked from keystonemiddleware i believe | 21:58 |
| *** phasespace has quit IRC | 22:00 | |
| lbragstad | huh | 22:01 |
| timothyb89 | hmm, that seems to match the data at least | 22:01 |
| lbragstad | so - it looks like... | 22:01 |
| lbragstad | keystone overrides a hook from keystonemiddleware | 22:01 |
| lbragstad | (so that keystonemiddleware doesn't attempt to put the token on the wire to a keystone service to validate like it would for another service) | 22:02 |
| lbragstad | that's this code | 22:02 |
| lbragstad | https://git.openstack.org/cgit/openstack/keystone/tree/keystone/server/flask/request_processing/middleware/auth_context.py#n241 | 22:02 |
| lbragstad | which is overriding the keystonemiddleware auth_token implementation - https://git.openstack.org/cgit/openstack/keystone/tree/keystone/server/flask/request_processing/middleware/auth_context.py#n233 | 22:03 |
| lbragstad | so - that's the first validation | 22:03 |
| lbragstad | the second is when we get a little further in request processing, but still in middleware | 22:03 |
| lbragstad | and we're creating a context object (so we can do things like policy enforcement based on the authorization associated to the token) - and we validate the token in that, too | 22:04 |
| lbragstad | so - i'm not sure how feasible this might be | 22:10 |
| lbragstad | but iiuc - we could attempt to eliminate the time spent in that middle column if we try and reuse the first token validation somehow? | 22:10 |
| lbragstad | that might be tricky though | 22:10 |
| timothyb89 | that would make sense, passing that data around sounds fun, though | 22:17 |
| openstackgerrit | Lance Bragstad proposed openstack/keystone master: WIP: Only validate tokens once per request https://review.openstack.org/641499 | 22:19 |
| lbragstad | timothyb89 ^ does that help? | 22:19 |
| lbragstad | i'll be honest, i didn't test it | 22:20 |
| lbragstad | so i'm not sure if it'll work 100% | 22:20 |
| timothyb89 | oh, cool! I'll try it out and see | 22:20 |
| lbragstad | i'm just curious if that speeds up requests by about ~20% | 22:20 |
| *** phasespace has joined #openstack-keystone | 22:21 | |
| *** markvoelker has quit IRC | 22:22 | |
| lbragstad | do you happen to have measurements in time in addition to percentages? | 22:23 |
| timothyb89 | I have some stats on me now, but could rerun my benchmark | 22:25 |
| lbragstad | percentages are good | 22:26 |
| lbragstad | i was just curious what kind of times you were seeing | 22:26 |
| timothyb89 | mean response time for list projects was 29ms, 22ms min, 100ms max, std dev of 11.8ms - over 100 requests | 22:26 |
| lbragstad | is the client local to the benchmark box? | 22:26 |
| timothyb89 | in this case yes | 22:26 |
| * lbragstad nods | 22:26 | |
| timothyb89 | some more comprehensive data: https://docs.google.com/spreadsheets/d/1AaIR4a1JNzngjyO3F0emw9T2ZsdcZprE_MXuM2MsTvM/edit?usp=sharing | 22:29 |
| lbragstad | hah - just a little bit | 22:29 |
| timothyb89 | seemed to hit a latency floor of ~20ms no matter how many resources we threw at it :) | 22:30 |
| timothyb89 | vs uuid tokens which could return in < 10ms | 22:30 |
| lbragstad | interesting | 22:31 |
| *** erus has joined #openstack-keystone | 22:31 | |
| lbragstad | you're using pypy? | 22:32 |
| timothyb89 | tried it, at least, but it wasn't really faster and was less consistent | 22:32 |
| lbragstad | huh | 22:33 |
| timothyb89 | turned out we were IO bound due to mysql latency | 22:33 |
| lbragstad | last i tried running anything with pypy was about 4 years ago | 22:33 |
| lbragstad | so avg response time for requests when using uuid was ~20 ms? | 22:34 |
| timothyb89 | well, plain token validation (no project list) was 7ms +/- 6ms | 22:35 |
| lbragstad | got it | 22:35 |
| timothyb89 | token validation with fernet is 34ms +/- 16ms | 22:36 |
| timothyb89 | so the best case was much better | 22:36 |
| lbragstad | i'm noticing only slightly better performance locally | 22:51 |
| lbragstad | without my patch GET /v3/projects took 0.06 seconds, with the patch i'm noticing about 0.054 or 0.047 in the best case | 22:52 |
| *** rcernin has joined #openstack-keystone | 22:52 | |
| lbragstad | maybe that's in line with your numbers, though? | 22:52 |
| timothyb89 | currently trying to deploy the patch myself, will hopefully be able to say for certain | 22:54 |
| *** tkajinam has joined #openstack-keystone | 23:01 | |
| *** dave-mccowan has quit IRC | 23:06 | |
| timothyb89 | lbragstad: alright, tentative result is response time down to 22ms +/- 11.5 ms, with a new record low response time of 15.5 ms | 23:13 |
| timothyb89 | lbragstad: new profile result as well: https://drive.google.com/file/d/1fSQaB-sDGAdeV9bSLSne7chSiFugOtgb/view?usp=sharing | 23:14 |
| timothyb89 | savings appears to be right about as expected, roughly 23% faster :) | 23:18 |
| *** markvoelker has joined #openstack-keystone | 23:19 | |
| timothyb89 | (disclaimer, though, I applied the patch to our stable/rocky tree, so YMMV on master) | 23:28 |
| *** markvoelker has quit IRC | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!