| *** spzala has joined #openstack-keystone | 00:02 | |
| *** markvoelker has joined #openstack-keystone | 00:04 | |
| *** bigdogstl has quit IRC | 00:11 | |
| *** bigdogstl has joined #openstack-keystone | 00:11 | |
| *** spzala has quit IRC | 00:12 | |
| *** nicolasbock has joined #openstack-keystone | 00:19 | |
| *** bigdogstl has quit IRC | 00:21 | |
| *** bigdogstl has joined #openstack-keystone | 00:27 | |
| *** bigdogstl has quit IRC | 00:31 | |
| *** edmondsw has joined #openstack-keystone | 00:37 | |
| *** bigdogstl has joined #openstack-keystone | 00:41 | |
| *** edmondsw has quit IRC | 00:42 | |
| *** bigdogstl has quit IRC | 00:46 | |
| *** bigdogstl has joined #openstack-keystone | 00:50 | |
| *** markvoelker has quit IRC | 00:53 | |
| *** markvoelker has joined #openstack-keystone | 00:54 | |
| *** threestrands has quit IRC | 00:58 | |
| *** markvoelker has quit IRC | 00:58 | |
| *** itlinux has joined #openstack-keystone | 01:01 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 01:01 | |
| *** threestrands has joined #openstack-keystone | 01:02 | |
| *** threestrands has quit IRC | 01:03 | |
| *** threestrands has joined #openstack-keystone | 01:04 | |
| *** markvoelker has joined #openstack-keystone | 01:04 | |
| *** markvoelker has quit IRC | 01:09 | |
| *** threestrands has quit IRC | 01:11 | |
| *** markvoelker_ has joined #openstack-keystone | 01:13 | |
| *** afred312 has quit IRC | 01:22 | |
| *** afred312 has joined #openstack-keystone | 01:24 | |
| *** threestrands has joined #openstack-keystone | 01:24 | |
| *** threestrands has quit IRC | 01:24 | |
| *** threestrands has joined #openstack-keystone | 01:24 | |
| *** threestrands has quit IRC | 01:25 | |
| *** threestrands has joined #openstack-keystone | 01:26 | |
| *** afred312 has quit IRC | 01:26 | |
| *** gyee has quit IRC | 01:26 | |
| *** zhurong has joined #openstack-keystone | 01:45 | |
| *** dave-mccowan has joined #openstack-keystone | 01:46 | |
| *** threestrands has quit IRC | 01:54 | |
| openstackgerrit | wangqiang-bj proposed openstack/keystone master: fix wrong url link of User trusts https://review.openstack.org/533032 | 01:55 |
|---|---|---|
| *** afred312 has joined #openstack-keystone | 01:56 | |
| *** threestrands has joined #openstack-keystone | 01:56 | |
| *** kukacz has quit IRC | 02:00 | |
| *** afred312 has quit IRC | 02:01 | |
| *** kukacz_ has joined #openstack-keystone | 02:01 | |
| lbragstad | wxy: looks like https://review.openstack.org/#/c/524082/20 is struggling with zuul :-/ | 02:02 |
| wxy | yeah. zuul restarts everyday... | 02:03 |
| lbragstad | i noticed a few emails about that | 02:03 |
| *** afred312 has joined #openstack-keystone | 02:03 | |
| lbragstad | hopefully that last recheck does the trick | 02:03 |
| wxy | and there were almost 270+ patch wait for check yesterday. | 02:03 |
| lbragstad | oof | 02:03 |
| lbragstad | hopefully the break over the weekend lets things catch up | 02:04 |
| wxy | even it seems that zuulv3 is not stable enough these days. | 02:04 |
| lbragstad | sounds like they're flushing out issues, which is good | 02:04 |
| lbragstad | outside of zuul issues that are out of my control - is there anything i can help with? | 02:06 |
| lbragstad | i plan to review the unified limit controller tomorrow | 02:06 |
| lbragstad | i was caught up with the manager bits and morgan reviewed it | 02:06 |
| lbragstad | i spent most of today on application credentials, so i should be free to look through all the controller pieces for limits | 02:07 |
| wxy | Thanks. I've refreshed the patch locally already. I'll submit it later. | 02:11 |
| wxy | I have a question about system scope. https://review.openstack.org/#/c/526197/ | 02:12 |
| wxy | how could this patch passed py27, since it changed the scope to only system. | 02:12 |
| wxy | lbragstad: the test shows that it only uses project scoped token. https://github.com/openstack/keystone/blob/master/keystone/tests/unit/test_v3_policy.py#L37-L41 | 02:15 |
| wxy | Maybe I missed something? | 02:15 |
| lbragstad | oh - sure | 02:15 |
| lbragstad | that's because we landed this in oslo policy | 02:15 |
| lbragstad | https://review.openstack.org/#/c/529372/ | 02:16 |
| lbragstad | and https://review.openstack.org/#/c/530263/ | 02:16 |
| lbragstad | so - a project has to be configured to enforce scope | 02:16 |
| lbragstad | s/project/service/ | 02:16 |
| wxy | oh, the default value is False. | 02:17 |
| lbragstad | yeah - at least for now for backwards compatibility | 02:18 |
| wxy | clear~ | 02:18 |
| lbragstad | a warning will be logged through | 02:18 |
| lbragstad | though* | 02:18 |
| *** markvoelker_ has quit IRC | 02:31 | |
| *** annp has joined #openstack-keystone | 02:39 | |
| *** Dinesh_Bhor has quit IRC | 02:44 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 02:47 | |
| ayoung | https://review.openstack.org/#/c/515215/ cmurphy care to pull the trigger? Time to get this stack moving. | 02:56 |
| *** Dinesh_Bhor has quit IRC | 03:02 | |
| *** bigdogstl has quit IRC | 03:14 | |
| *** edmondsw has joined #openstack-keystone | 03:14 | |
| *** bigdogstl has joined #openstack-keystone | 03:18 | |
| *** edmondsw has quit IRC | 03:18 | |
| *** itlinux has quit IRC | 03:19 | |
| *** bigdogstl has quit IRC | 03:33 | |
| *** dave-mccowan has quit IRC | 03:33 | |
| *** bigdogstl has joined #openstack-keystone | 03:48 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 03:56 | |
| *** bigdogstl has quit IRC | 03:57 | |
| *** bigdogstl has joined #openstack-keystone | 03:58 | |
| *** Dinesh_Bhor has quit IRC | 03:58 | |
| *** ayoung has quit IRC | 03:58 | |
| openstackgerrit | chenpengzi proposed openstack/keystone master: Modify users-list-response links: users list response links not contain identity https://review.openstack.org/533063 | 04:00 |
| *** zhurong has quit IRC | 04:03 | |
| openstackgerrit | chenpengzi proposed openstack/keystone master: Modify users-list-response links: https://review.openstack.org/533063 | 04:04 |
| *** bigdogstl has quit IRC | 04:05 | |
| *** bigdogstl has joined #openstack-keystone | 04:07 | |
| *** markvoelker has joined #openstack-keystone | 04:11 | |
| *** bigdogstl has quit IRC | 04:12 | |
| *** edmondsw has joined #openstack-keystone | 04:18 | |
| *** nicolasbock has quit IRC | 04:21 | |
| *** edmondsw has quit IRC | 04:22 | |
| *** links has joined #openstack-keystone | 04:31 | |
| *** bigdogstl has joined #openstack-keystone | 04:32 | |
| *** itlinux has joined #openstack-keystone | 04:32 | |
| *** bigdogstl has quit IRC | 04:37 | |
| *** namnh has joined #openstack-keystone | 04:51 | |
| *** bigdogstl has joined #openstack-keystone | 05:04 | |
| *** bigdogstl has quit IRC | 05:15 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 05:39 | |
| *** zhurong has joined #openstack-keystone | 05:42 | |
| *** Dinesh_Bhor has quit IRC | 05:52 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 05:52 | |
| *** Dinesh_Bhor has quit IRC | 06:07 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 06:08 | |
| *** Dinesh_Bhor has quit IRC | 06:12 | |
| openstackgerrit | wangqiang-bj proposed openstack/keystone master: adjust response code in order of credentials.inc https://review.openstack.org/533082 | 06:12 |
| *** bigdogstl has joined #openstack-keystone | 06:17 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 06:18 | |
| *** bigdogstl has quit IRC | 06:22 | |
| *** bigdogstl has joined #openstack-keystone | 06:23 | |
| *** harlowja has quit IRC | 06:35 | |
| *** bigdogstl has quit IRC | 06:41 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/keystone master: Imported Translations from Zanata https://review.openstack.org/533093 | 06:42 |
| openstackgerrit | wangqiang-bj proposed openstack/keystone master: put response code in table of ''domains.inc'' https://review.openstack.org/533097 | 06:53 |
| *** Dinesh_Bhor has quit IRC | 06:54 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 06:55 | |
| *** Dinesh_Bhor has quit IRC | 06:56 | |
| *** daidv has quit IRC | 06:57 | |
| *** annp has quit IRC | 06:57 | |
| *** annp has joined #openstack-keystone | 06:58 | |
| *** daidv has joined #openstack-keystone | 06:58 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 07:01 | |
| *** rha has quit IRC | 07:03 | |
| *** harlowja has joined #openstack-keystone | 07:10 | |
| *** harlowja has quit IRC | 07:10 | |
| openstackgerrit | wangqiang-bj proposed openstack/keystone master: adjust response code order in ''domains-config-v3.inc'' https://review.openstack.org/533103 | 07:15 |
| *** bigdogstl has joined #openstack-keystone | 07:16 | |
| *** itlinux has quit IRC | 07:20 | |
| *** bigdogstl has quit IRC | 07:20 | |
| *** rcernin has quit IRC | 07:28 | |
| *** pcaruana has joined #openstack-keystone | 07:29 | |
| *** Dinesh_Bhor has quit IRC | 07:35 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 07:37 | |
| *** Dinesh_Bhor has quit IRC | 07:41 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/keystonemiddleware master: Imported Translations from Zanata https://review.openstack.org/533114 | 07:44 |
| *** bigdogstl has joined #openstack-keystone | 07:46 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 07:46 | |
| *** threestrands has quit IRC | 07:56 | |
| *** bigdogstl has quit IRC | 07:57 | |
| *** tesseract has joined #openstack-keystone | 08:08 | |
| *** bigdogstl has joined #openstack-keystone | 08:12 | |
| *** bigdogstl has quit IRC | 08:16 | |
| *** AlexeyAbashkin has joined #openstack-keystone | 08:23 | |
| *** Dinesh_Bhor has quit IRC | 08:45 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 08:47 | |
| openstackgerrit | wanghui proposed openstack/keystone master: adjust response code order in ''policies.inc'' https://review.openstack.org/533130 | 08:50 |
| *** bigdogstl has joined #openstack-keystone | 08:58 | |
| openstackgerrit | Thomas Bechtold proposed openstack/keystone-tempest-plugin master: Use openstackdocstheme for docs and release notes https://review.openstack.org/531097 | 09:10 |
| *** bigdogstl has quit IRC | 09:10 | |
| *** zhurong has quit IRC | 09:20 | |
| *** vaishali has quit IRC | 09:21 | |
| *** eglute has quit IRC | 09:21 | |
| *** sxc731 has joined #openstack-keystone | 09:21 | |
| *** dikonoor has joined #openstack-keystone | 09:25 | |
| *** Dinesh_Bhor has quit IRC | 09:25 | |
| *** eglute has joined #openstack-keystone | 09:26 | |
| *** vaishali has joined #openstack-keystone | 09:26 | |
| *** bigdogstl has joined #openstack-keystone | 09:28 | |
| *** Dinesh_Bhor has joined #openstack-keystone | 09:28 | |
| *** bigdogstl has quit IRC | 09:33 | |
| *** Dinesh_Bhor has quit IRC | 09:40 | |
| openstackgerrit | Merged openstack/keystone master: Add db operation for unified limit https://review.openstack.org/524082 | 09:52 |
| *** bigdogstl has joined #openstack-keystone | 10:00 | |
| *** daidv has quit IRC | 10:10 | |
| *** bigdogstl has quit IRC | 10:12 | |
| *** StefanPaetowJisc has joined #openstack-keystone | 10:14 | |
| *** namnh has quit IRC | 10:15 | |
| *** wxy has quit IRC | 10:16 | |
| *** StefanPaetowJisc has quit IRC | 10:19 | |
| *** sambetts|afk is now known as sambetts | 10:23 | |
| openstackgerrit | Colleen Murphy proposed openstack/keystone master: Add Application Credentials manager https://review.openstack.org/524747 | 10:29 |
| *** markvoelker has quit IRC | 10:44 | |
| *** mvk has quit IRC | 10:52 | |
| *** wxy has joined #openstack-keystone | 10:54 | |
| wxy | ping cmurphy: I downloaded your controller patch and test with RESTful request. | 10:55 |
| cmurphy | wxy: you mean that's where you saw the db reference error? | 10:57 |
| wxy | yeah | 10:57 |
| cmurphy | wxy: okay i'll see if i can reproduce that way, thanks! | 10:57 |
| wxy | part of pbr test:http://paste.openstack.org/show/643751/ | 10:57 |
| *** AlexeyAbashkin has quit IRC | 10:57 | |
| wxy | s/pdb | 10:57 |
| cmurphy | i thought this would work since trusts does it the same way but maybe that's why trusts doesn't use a foreign key for roles | 10:58 |
| wxy | cmurphy: not sure if it happened only in my env. mysql version: Ver 14.14 Distrib 5.7.19, for Linux (x86_64) using EditLine wrapper | 11:02 |
| *** edmondsw has joined #openstack-keystone | 11:03 | |
| *** mvk has joined #openstack-keystone | 11:07 | |
| openstackgerrit | wangxiyuan proposed openstack/keystone master: Add limit provider https://review.openstack.org/524109 | 11:07 |
| openstackgerrit | wangxiyuan proposed openstack/keystone master: Implement policies for limits https://review.openstack.org/530143 | 11:07 |
| openstackgerrit | wangxiyuan proposed openstack/keystone master: Expose unified limit APIs https://review.openstack.org/524110 | 11:07 |
| wxy | cmurphy: I have to go home now. leave some information to you if needed: SQLAlchemy 1.1.15, oslo.db 4.31.0. Bye. | 11:07 |
| *** edmondsw has quit IRC | 11:08 | |
| cmurphy | wxy: alright, have a good night! | 11:09 |
| *** AlexeyAbashkin has joined #openstack-keystone | 11:12 | |
| *** bigdogstl has joined #openstack-keystone | 11:16 | |
| *** odyssey4me has quit IRC | 11:19 | |
| *** bigdogstl has quit IRC | 11:28 | |
| *** nicolasbock has joined #openstack-keystone | 11:36 | |
| *** odyssey4me has joined #openstack-keystone | 11:42 | |
| *** odyssey4me has quit IRC | 11:45 | |
| *** odyssey4me has joined #openstack-keystone | 11:45 | |
| *** odyssey4me has quit IRC | 11:53 | |
| *** odyssey4me has joined #openstack-keystone | 11:54 | |
| *** odyssey4me has quit IRC | 11:58 | |
| *** odyssey4me has joined #openstack-keystone | 11:58 | |
| *** StefanPaetowJisc has joined #openstack-keystone | 11:59 | |
| *** raildo has joined #openstack-keystone | 12:02 | |
| *** StefanPaetowJisc has quit IRC | 12:04 | |
| *** annp has quit IRC | 12:07 | |
| *** edmondsw has joined #openstack-keystone | 12:11 | |
| *** odyssey4me has quit IRC | 12:13 | |
| *** odyssey4me has joined #openstack-keystone | 12:13 | |
| *** bigdogstl has joined #openstack-keystone | 12:16 | |
| *** bigdogstl has quit IRC | 12:30 | |
| *** sxc731 has quit IRC | 12:35 | |
| *** markvoelker has joined #openstack-keystone | 12:45 | |
| openstackgerrit | Colleen Murphy proposed openstack/keystone master: Add application credentials driver https://review.openstack.org/524928 | 12:45 |
| openstackgerrit | Colleen Murphy proposed openstack/keystone master: Add Application Credentials manager https://review.openstack.org/524747 | 12:45 |
| openstackgerrit | Colleen Murphy proposed openstack/keystone master: Extract expiration validation to utils https://review.openstack.org/532257 | 12:45 |
| *** dave-mccowan has joined #openstack-keystone | 12:54 | |
| *** bigdogstl has joined #openstack-keystone | 13:02 | |
| *** hrybacki has quit IRC | 13:10 | |
| *** hrybacki has joined #openstack-keystone | 13:13 | |
| *** bigdogstl has quit IRC | 13:19 | |
| *** bigdogstl has joined #openstack-keystone | 13:19 | |
| *** markvoelker has quit IRC | 13:20 | |
| *** sxc731 has joined #openstack-keystone | 13:28 | |
| *** efried is now known as fried_rice | 13:38 | |
| *** bigdogstl has quit IRC | 13:40 | |
| *** panbalag has joined #openstack-keystone | 13:41 | |
| *** panbalag has quit IRC | 13:48 | |
| *** david-lyle has quit IRC | 13:49 | |
| bhagyashris | iawn: Hi, are you around? | 13:54 |
| bhagyashris | sorry for spell mistake | 13:55 |
| bhagyashris | ianw: | 13:55 |
| cmurphy | bhagyashris: he lives in australia so you might have to wait a few hours, you're also more likely to catch him in #openstack-infra than here | 13:57 |
| cmurphy | bhagyashris: if there's something we can help you with go ahead and ask your question and if someone's around who can answer we will | 13:57 |
| bhagyashris | cmurphy: thank you for the inputs :) just request to review one of my patch https://review.openstack.org/#/c/527907/ | 14:00 |
| cmurphy | bhagyashris: #openstack-qa would be a better place to ask for that, the keystone team can't help much with devstack :) | 14:01 |
| *** panbalag has joined #openstack-keystone | 14:01 | |
| bhagyashris | cmurphy: ok thank you :) actually he was give +2 on my patch earlier but becuase of some review comments it's removed so i am trying to contact him but yeah thank you for your suggestion to contact the devstack team | 14:03 |
| *** panbalag has left #openstack-keystone | 14:05 | |
| *** bigdogstl has joined #openstack-keystone | 14:10 | |
| *** bigdogstl has quit IRC | 14:15 | |
| *** edmondsw has quit IRC | 14:19 | |
| *** edmondsw has joined #openstack-keystone | 14:19 | |
| *** ayoung has joined #openstack-keystone | 14:22 | |
| *** edmondsw has quit IRC | 14:24 | |
| *** bigdogstl has joined #openstack-keystone | 14:26 | |
| *** odyssey4me has quit IRC | 14:30 | |
| *** odyssey4me has joined #openstack-keystone | 14:31 | |
| *** odyssey4me has quit IRC | 14:32 | |
| *** odyssey4me has joined #openstack-keystone | 14:33 | |
| *** edmondsw has joined #openstack-keystone | 14:52 | |
| *** markvoelker has joined #openstack-keystone | 14:52 | |
| *** bigdogstl has quit IRC | 15:09 | |
| *** bigdogstl has joined #openstack-keystone | 15:10 | |
| *** sxc731 has quit IRC | 15:12 | |
| *** sxc731 has joined #openstack-keystone | 15:13 | |
| knikolla | o/ | 15:21 |
| *** abhi89 has quit IRC | 15:26 | |
| *** dansmith is now known as superdan | 15:31 | |
| *** AlexeyAbashkin has quit IRC | 15:34 | |
| lbragstad | o/ | 15:36 |
| lbragstad | kmalloc: apparently stable/pike is broken https://review.openstack.org/#/c/532984/2 | 15:36 |
| lbragstad | ^ that should fix it pending what zuul says | 15:37 |
| lbragstad | i've ask smcginnis to review and push it through if you don't have the time to look into it | 15:37 |
| lbragstad | asked* | 15:37 |
| *** david-lyle has joined #openstack-keystone | 15:38 | |
| gagehugo | o/ | 15:43 |
| sxc731 | lbragstad: hello! | 15:45 |
| lbragstad | sxc731: o/ | 15:46 |
| lbragstad | how goes it? | 15:46 |
| sxc731 | lbragstad: not too bad, thank you and yourself? | 15:46 |
| lbragstad | well - it's friday :0 | 15:46 |
| lbragstad | :) | 15:46 |
| sxc731 | True enough ;-) | 15:46 |
| sxc731 | Remember you gave me that trick on enabling debugging for KS caching? So I now know my caching isn't working but it seems the cause is that different keys are generated for the same token... | 15:47 |
| lbragstad | huh | 15:48 |
| sxc731 | Simple test: try to validate the same token twice in a row. I get: | 15:48 |
| sxc731 | CACHE_GET: key1 | 15:49 |
| sxc731 | CACHE_GET: key1 | 15:49 |
| sxc731 | CACHE_PUT: key1 | 15:49 |
| sxc731 | then 2nd validation: | 15:49 |
| sxc731 | CACHE_GET: key2 | 15:49 |
| sxc731 | CACHE_GET: key2 | 15:49 |
| sxc731 | CACHE_PUT: key2 | 15:49 |
| sxc731 | I have compared the content of the token, which is output at CACHE_PUT time and it's exactly the same | 15:49 |
| sxc731 | This is within a single process. What could causing this? | 15:49 |
| sxc731 | (the issue is that key1 != key2) | 15:50 |
| sxc731 | (not sure why I get 2x GET but not too big a deal I gather) | 15:50 |
| lbragstad | is this for authenticating or only validating a token? | 15:51 |
| sxc731 | Validating | 15:51 |
| lbragstad | so you're doing this with GET /v3/auth/tokens and X-Auth-Token and X-Subject-Token in the headers | 15:51 |
| sxc731 | Yep, this is still taken from dolph's benchmark | 15:52 |
| lbragstad | do you have your config (sans sensitive data) handy? | 15:53 |
| sxc731 | Sure (it's not a prod env anyway). Let me paste it | 15:53 |
| lbragstad | cool - i want to try and recreate locally with caching debug enabled | 15:53 |
| sxc731 | lbragstad: http://paste.openstack.org/show/643865/ | 15:57 |
| *** cburgess has quit IRC | 15:57 | |
| sxc731 | (essentially this is a standard OSA cfg, with the cache debugging stuff added) | 15:58 |
| *** dikonoor has quit IRC | 15:58 | |
| lbragstad | cool - so not much has changed since the last time i looked | 15:58 |
| sxc731 | Indeed | 15:58 |
| lbragstad | let me set this up locally and see what happens | 15:58 |
| sxc731 | Would you like a quick look at the log excerpt; perhaps you can spot smth odd? | 15:59 |
| *** cburgess has joined #openstack-keystone | 15:59 | |
| *** dikonoor has joined #openstack-keystone | 15:59 | |
| sxc731 | I'm just wondering: could this explain the perceived regression between our ~150ms validation time compared with Dolph's ~6ms from Feb 2015? Surely someone would have noticed, right?? | 16:01 |
| sxc731 | Dolph's catalog is quite a bit slimmer than mine but that doesn't explain. Do you guys have tests that validate caching? | 16:02 |
| lbragstad | so - i can validate that caching is on | 16:10 |
| lbragstad | but i'm not seeing the debug out put | 16:10 |
| sxc731 | Hmm, that's weird; have you looked in the main keystone.log? Let me paste my sytemd Exec line | 16:11 |
| sxc731 | First you need the following 4 lines to enable cache debug output (took me a while to figure this out too ;-) | 16:12 |
| *** itlinux has joined #openstack-keystone | 16:12 | |
| lbragstad | http://paste.openstack.org/raw/643876/ | 16:13 |
| sxc731 | [DEFAULT] | 16:13 |
| sxc731 | debug = True | 16:13 |
| sxc731 | default_log_levels = ... , oslo.cache=DEBUG,... | 16:13 |
| sxc731 | [cache] | 16:13 |
| sxc731 | debug_cache_backend = True | 16:13 |
| lbragstad | checking those values | 16:13 |
| lbragstad | ah - i'm missing the default_log_levels bit | 16:14 |
| lbragstad | adding that | 16:14 |
| sxc731 | Yes, your cache seems to make a difference from ~200ms down to 50 | 16:14 |
| sxc731 | Yeah, the default_log_levels thing was a real treat to figure out | 16:15 |
| sxc731 | (it's always obvious in hindsight ;-) | 16:15 |
| lbragstad | granted - i'm running everything locally | 16:16 |
| lbragstad | ok - so i have two different tokens | 16:18 |
| lbragstad | one for X-Auth-Token and one for X-Subject-Token | 16:19 |
| lbragstad | and i see two different cache hits | 16:19 |
| sxc731 | Yep my test was the same and I guess what you see shows it's working ok | 16:19 |
| lbragstad | http://paste.openstack.org/raw/643884/ | 16:19 |
| lbragstad | key 57c045e2c35baec049d532a74edba38bd5328ae7 is x-auth-token | 16:20 |
| sxc731 | Let me paste my output | 16:20 |
| lbragstad | and key 8edd04a60920836974fc8dbefc9b58badffdfda9 is the x-subject-token | 16:20 |
| *** itlinux has quit IRC | 16:20 | |
| *** hoonetorg has quit IRC | 16:22 | |
| sxc731 | First validation round: http://paste.openstack.org/show/643886/ | 16:22 |
| sxc731 | 2nd round (about 2 secs later): http://paste.openstack.org/show/643887/ | 16:23 |
| openstackgerrit | ayoung proposed openstack/keystone master: Implement controller logic for system group assignments https://review.openstack.org/524017 | 16:25 |
| lbragstad | weird... | 16:25 |
| sxc731 | (this output is obviously quite distilled as I struggled to make sense of it otherwise). If you compare the subject tokens which are output at CACHE_SET time (I have) you'll see the token is exactly the same; yet the cache key is different | 16:25 |
| sxc731 | I'm running an early-ish Pike baseline; I gather you're running master? | 16:25 |
| lbragstad | it's the exact same token? | 16:25 |
| lbragstad | yeah - i just installed master, but the code should be the same for each with respect to the caching stuff | 16:26 |
| sxc731 | Yep, I've done some sed'ing and JSON pretty formatting | 16:26 |
| lbragstad | sxc731: are you still using the benchmark script? | 16:27 |
| sxc731 | When we last spoke, you mentioned you thought "order of 150ms was about right" according to some past (recent) benchmarks. Yet what you've just shown me is order of 50ms and I suppose this is on a dev box as opposed to prod-class HW? | 16:27 |
| lbragstad | right - i'm just running a container on my laptop | 16:28 |
| sxc731 | I'm just sending through single curl's (actually 'ab -n1' as this is what was in the bench) | 16:28 |
| lbragstad | when i had access to production-ish hardware, i was seeing about 120 ms response times with about 120k requests a minute | 16:28 |
| ayoung | lbragstad, first 2 patches of your current service role series are +2Aed | 16:29 |
| sxc731 | AH! Mine is a single request on unloaded HW | 16:29 |
| lbragstad | ayoung: thank you, sir | 16:29 |
| lbragstad | sxc731: can you export a token to an env variable and run this? | 16:30 |
| lbragstad | s/token/two tokens/ | 16:30 |
| lbragstad | curl -X GET -H "X-Auth-Token: $TOKEN" -H "X-Subject-Token: $TOKEN2" http://192.168.122.160:35357/v3/auth/tokens -w "\n\n%{time_total}\n\n" | 16:30 |
| lbragstad | replacing your endpoint, too | 16:30 |
| sxc731 | OK | 16:30 |
| lbragstad | i just want to make sure we're using the same technique, i'll take another look at the benchmark script | 16:31 |
| sxc731 | The BM script uses 'ab' (Apache Bench). Should be functionally equivalent but I'll double-check... | 16:31 |
| *** jistr has quit IRC | 16:37 | |
| *** sapd_ has quit IRC | 16:37 | |
| *** cloudnull has quit IRC | 16:37 | |
| *** asettle has quit IRC | 16:37 | |
| *** lamt has quit IRC | 16:37 | |
| *** charz has quit IRC | 16:37 | |
| *** lamt has joined #openstack-keystone | 16:37 | |
| *** charz has joined #openstack-keystone | 16:37 | |
| *** sapd_ has joined #openstack-keystone | 16:37 | |
| *** asettle has joined #openstack-keystone | 16:37 | |
| *** cloudnull has joined #openstack-keystone | 16:37 | |
| *** asettle is now known as Guest31532 | 16:37 | |
| *** lamt is now known as Guest65662 | 16:37 | |
| sxc731 | OK, I still get same orders of magnitude: 0.379s; let me check the cache debugging | 16:38 |
| *** links has quit IRC | 16:38 | |
| *** hoonetorg has joined #openstack-keystone | 16:39 | |
| *** itlinux has joined #openstack-keystone | 16:39 | |
| *** jistr has joined #openstack-keystone | 16:40 | |
| *** links has joined #openstack-keystone | 16:43 | |
| lbragstad | sxc731: even with verifying the same tokens are being passed to keystone? | 16:45 |
| -openstackstatus- NOTICE: Zuul has been restarted and lost queue information; changes in progress will need to be rechecked. | 16:45 | |
| lbragstad | sxc731: let me stand up a pike environment quick | 16:46 |
| sxc731 | lbragstad: not sure what you mean? I'm using the same curl cmdline with the env vars as you suggested so it should be the same token, right? Beside, the CACHE_SET lines explode the content of the token and they're identical. | 16:46 |
| sxc731 | I think the cache is behaving as expected. What seems broken is the cache key generation part. I presume this uses some sort of hashing? What could possibly throw this off? | 16:48 |
| lbragstad | hmm | 16:49 |
| lbragstad | here's what i've done | 16:49 |
| lbragstad | i dropped my database, created a new one, installed stable/pike (d0721d7cf4dc808946a7016b0ca2830c8850d5d9), re-bootstrapped the deployment | 16:49 |
| lbragstad | generated a couple new tokens | 16:50 |
| lbragstad | and this is my output | 16:50 |
| lbragstad | v | 16:50 |
| lbragstad | http://paste.openstack.org/raw/643903/ | 16:50 |
| sxc731 | Looks alright! | 16:52 |
| sxc731 | My KS baseline 6a67918f9d5f39564af8eacc57b80cba98242683 # HEAD of "stable/pike" as of 28.09.2017 (which is what OSA 16.0.1 delivers). I'm assuming you would be aware if a significant bug like this had been reoslved? | 16:53 |
| sxc731 | I've actually been wondering why OSA uses commits as baselines rahter than tags... | 16:53 |
| sxc731 | Guess I need to roll fwd my Keystone and retest | 16:54 |
| sxc731 | (sorry that's OSA 16.0.2) | 16:55 |
| sxc731 | lbragstad: Gonna be afk for 15-20 mins; will pick up then. Thank you for all your help so far!! | 16:57 |
| lbragstad | sxc731: anytime, pasting my logs here in a bit | 16:58 |
| *** sxc731_ has joined #openstack-keystone | 16:58 | |
| *** sxc731 has quit IRC | 17:02 | |
| cmurphy | from ttx https://twitter.com/pilgrimstack/status/951860289141641217 | 17:03 |
| *** pcaruana has quit IRC | 17:03 | |
| lbragstad | sxc731_: here is my entire uwsgi log http://paste.openstack.org/show/643918/ | 17:03 |
| lbragstad | cmurphy: mmm | 17:04 |
| lbragstad | cmurphy: is the tools documentation referencing our documentation? | 17:06 |
| cmurphy | lbragstad: i think they mean their customers' tools don't have documentation indicating either way | 17:07 |
| lbragstad | cmurphy: ok - that's what i thought, but wasn't sure | 17:07 |
| *** fried_rice is now known as fried_rolls | 17:07 | |
| openstackgerrit | Gage Hugo proposed openstack/keystone master: WIP - Add functional testing gate https://review.openstack.org/531014 | 17:10 |
| *** tesseract has quit IRC | 17:12 | |
| *** AlexeyAbashkin has joined #openstack-keystone | 17:12 | |
| *** sxc731_ has quit IRC | 17:16 | |
| *** sxc731 has joined #openstack-keystone | 17:16 | |
| *** AlexeyAbashkin has quit IRC | 17:17 | |
| *** AlexeyAbashkin has joined #openstack-keystone | 17:21 | |
| lbragstad | sxc731: actually - what happens if you modify your keystone configuration to go to only one cache? | 17:21 |
| sxc731 | lbragstad: let me try that for fun ;-) | 17:21 |
| lbragstad | i wonder if keystone it querying different memcached backends and getting misses | 17:22 |
| lbragstad | so - technically it would be storing the same token in three different places... | 17:23 |
| sxc731 | BINGO | 17:23 |
| sxc731 | Down to order of 0.04s | 17:23 |
| sxc731 | Tha'ts quite a bit nicer for sure! | 17:24 |
| *** links has quit IRC | 17:24 | |
| sxc731 | Still doesn't explain why the keys would be different... | 17:24 |
| sxc731 | Surely if different keys are generated for the same subject token (even if different cache instances) that would qualify as a bug? | 17:25 |
| sxc731 | I can't imagine I'm the only one running with multiple cache instances though... | 17:25 |
| *** AlexeyAbashkin has quit IRC | 17:25 | |
| lbragstad | i wouldn't think you are the only one either | 17:26 |
| lbragstad | i'm not really familiar with how those keys are generated | 17:26 |
| sxc731 | Can you reproduce the degenerate behaviour with multiple caches? | 17:26 |
| lbragstad | I would assume it is generated from data in memcache? | 17:26 |
| sxc731 | That key hashing would have to be done client-side because that's before you've managed to retrieve the data, right? | 17:27 |
| *** spzala has joined #openstack-keystone | 17:28 | |
| lbragstad | cc kmalloc ^ | 17:31 |
| sxc731 | lbragstad: getting some higher-level (wife!) interrupts as it's already 6:30 pm here in Europe. Looking at the KS baselines, it seems there was only one for Pike: 12.0.0, right? | 17:31 |
| lbragstad | baselines? | 17:32 |
| sxc731 | I mean "tag" | 17:32 |
| lbragstad | oh - yeah | 17:32 |
| sxc731 | How should I proceed if I'm to raise a clean ticket on this? Update my baseline or is what I have ok? | 17:34 |
| sxc731 | or perhaps if you could confirm that you can reproduce the issue with multiple caches it might be useful? | 17:35 |
| lbragstad | what commit do you have installed locally? | 17:35 |
| sxc731 | 6a67918f9d5f39564af8eacc57b80cba98242683 # HEAD of "stable/pike" as of 28.09.2017 (per OSA 16.0.2) | 17:35 |
| lbragstad | technically - the worst case scenario would be you suffer a full performance hit for each cache in the deployment | 17:35 |
| lbragstad | yeah - you are only three commits behind what i have | 17:36 |
| lbragstad | http://paste.openstack.org/show/643946/ | 17:36 |
| lbragstad | and i don't think those three commits would affect this at all | 17:36 |
| lbragstad | and i'm almost positive this could be recreated on master | 17:37 |
| lbragstad | if i stand up multiple memcached instances | 17:37 |
| sxc731 | Interesting... so maybe there's a genuine issue here? Been wondering about the syntax of that 'backend_argument' config line as it looks a little weird... | 17:37 |
| sxc731 | backend_argument = url:172.20.120.247,172.20.68.222,172.20.112.234:11211 | 17:38 |
| sxc731 | But I double-checked it somewhere else and it looked similar | 17:38 |
| lbragstad | from what i can tell, that's suppose to aggregate the caches and shard data across them | 17:38 |
| *** sxc731_ has joined #openstack-keystone | 17:38 | |
| lbragstad | so if you have 3 keystone containers and 3 memcache instances, you can share data across all of them equally | 17:38 |
| *** sxc731_ has quit IRC | 17:39 | |
| *** sxc731_ has joined #openstack-keystone | 17:40 | |
| lbragstad | validating a token on keystone-1 will result in a validation on keystone-3 pulling the token from the cache instead of having to set it in the cache like it would on the first hit | 17:40 |
| *** sxc731 has quit IRC | 17:41 | |
| *** AlexeyAbashkin has joined #openstack-keystone | 17:42 | |
| *** sxc731_ has quit IRC | 17:43 | |
| *** sxc731_ has joined #openstack-keystone | 17:44 | |
| *** AlexeyAbashkin has quit IRC | 17:44 | |
| sxc731_ | My debugging really suggested the issue was key generation... | 17:44 |
| lbragstad | right - let me try and recreate locally and track down the key generation stuff | 17:45 |
| sxc731_ | Not sure how that’s impacted by multiple caches but certainly seems related... | 17:46 |
| lbragstad | yeah - me either | 17:46 |
| kmalloc | lbragstad: that isn't how it works | 17:48 |
| kmalloc | it relies on python-memcache code. | 17:48 |
| kmalloc | oh wait, yes it shoud share data | 17:48 |
| kmalloc | it doesn't cause data replication | 17:48 |
| kmalloc | at all | 17:48 |
| kmalloc | and python-memcache sucks. | 17:49 |
| kmalloc | a lot of issues around how it does the scanning etc. but there are ways to optimise for cases when a memcache is down | 17:49 |
| kmalloc | but in short, we're still leaning on a bad and unmaintained code base | 17:49 |
| sxc731_ | Kmalloc:sounds bad! | 17:50 |
| kmalloc | the key generation should be mostly consistent, but is based upon args passed to the methods | 17:50 |
| kmalloc | so, if an arg is slightly different for some reason | 17:50 |
| kmalloc | the key will be different | 17:51 |
| sxc731_ | So could the presence of multiple cache back-ends cause different keys to be generated? That’s certainl y what this looks like... | 17:51 |
| kmalloc | can you verify that the keys are actually different (Cache-key) or if python-memcache is sending the request to a differtent memcache server | 17:52 |
| sxc731_ | Our older Kilo/Fuel instance did behave better in this respect though so it feels like a regression... | 17:52 |
| kmalloc | my guess is it is the latter | 17:52 |
| kmalloc | my guess (zero basis) is that the issue is a dict hash issue, where we pass a dict to the backend and the order of the memcache servers is not consistent | 17:53 |
| kmalloc | and it is possibly an issue with oslo.cache's processing vs the keystone-implemented one before... | 17:54 |
| sxc731_ | The keys are different because the Oslo.cache logging shows them to be. Is the chosen cache instance exposed that far into app code? | 17:54 |
| kmalloc | nah. the keys should be consistent | 17:54 |
| kmalloc | and shouldn't be impacted by the backend unless someone set a diffewrent key-gen algo | 17:54 |
| lbragstad | i'm standing up an env with master and 3 memcached instances | 17:54 |
| sxc731_ | ^^ that sounds a likely reason!! When was that switch made roughly? | 17:54 |
| kmalloc | if the keys are really different it's not the backend code. | 17:55 |
| kmalloc | it is either a different keygen (somehow) or somehow data is different between requests. | 17:55 |
| kmalloc | the keys should never be impacted in our case, we hard set to a SHA1 hash of args | 17:55 |
| *** mvk has quit IRC | 17:55 | |
| sxc731_ | The app..level payload (ks token) is the same | 17:56 |
| kmalloc | and explicitly ban kwargs to avoid difference between dict hashing | 17:56 |
| kmalloc | if oslo.cache is showing the cache-key is different, then it's hashing the values differently and somehow breaking memoization | 17:56 |
| * kmalloc tosses original theory in the round file. | 17:56 | |
| kmalloc | ok. so something wierd with how we're key-hashing. | 17:57 |
| kmalloc | that makes me a bit more sad. it's harder to debug. | 17:57 |
| kmalloc | are you using memcache-pool or just memcache backend? | 17:57 |
| kmalloc | (this is just to eliminate one other wonky-bit of code/look at the interactions) | 17:58 |
| sxc731_ | I think it’s memcached-backend (see a little earlier; sorry I’m on an ipad in a bus :-( | 17:58 |
| kmalloc | no worries | 17:59 |
| kmalloc | that is what i expected. but figured i'd ask | 17:59 |
| *** markvoelker has quit IRC | 18:00 | |
| kmalloc | lbragstad: let me know what you dig up, i'm looking at the oslo.cache code, but ... hmm. weird | 18:03 |
| lbragstad | just about to try and recreate | 18:04 |
| sxc731_ | Thanks guys! I really appreciate your help. Going to be afk for a moment but will definitely check in later! | 18:11 |
| *** david-lyle has quit IRC | 18:12 | |
| lbragstad | weird | 18:12 |
| lbragstad | kmalloc: so - i have pike installed and i have two tokens exported to env vars | 18:13 |
| kmalloc | ok | 18:13 |
| lbragstad | and doing a constant token validate call | 18:13 |
| lbragstad | curl -X GET -H "X-Auth-Token: $AUTH_TOKEN" -H "X-Subject-Token: $SUBJECT_TOKEN" http://192.168.122.160:35357/v3/auth/tokens -w "\n\n%{time_total}\n\n" | 18:13 |
| lbragstad | but i set CACHE_SET everywhere in the keystone.log | 18:13 |
| lbragstad | technically - there should only be two set | 18:13 |
| lbragstad | two SETs | 18:13 |
| lbragstad | one for the initial auth token and one for the initial subject token, right? | 18:14 |
| kmalloc | and you have a memcache? | 18:14 |
| lbragstad | yeah - i have three of them | 18:14 |
| lbragstad | and this is my configuration in keystone | 18:14 |
| lbragstad | http://paste.openstack.org/show/643971/ | 18:15 |
| sxc731_ | Lbragstad: all these SETs: exactly what I saw. That’s because every GET is a miss based on the fact that the keys are always different! | 18:16 |
| kmalloc | lbragstad: aND If you reconfig for 1 memcache, you get one set and multiple gets? | 18:16 |
| lbragstad | checking | 18:16 |
| lbragstad | if i reconfigure with a single memcache instance (backend_argument = url:192.168.122.243:11211) i don't see *any* CACHE_SETs and I only see CACHE_GETs | 18:19 |
| lbragstad | but token validation looks right given the configuration http://paste.openstack.org/raw/643977/ | 18:20 |
| kmalloc | ok | 18:20 |
| kmalloc | weird. | 18:20 |
| lbragstad | 0.337 for the first call 0.022 after that and so on | 18:20 |
| kmalloc | ok so somehow the multiple backends is leaking into the cache key | 18:20 |
| kmalloc | weeeeirrrrdd | 18:20 |
| kmalloc | it's an oslo.cache detail then. | 18:20 |
| kmalloc | looking at the region, | 18:20 |
| lbragstad | but where in the world does the first key get set?! | 18:20 |
| lbragstad | or the second... | 18:21 |
| lbragstad | technically it should be putting both in the cache since the tokens in the call are different | 18:21 |
| kmalloc | it might be pre-seeded | 18:21 |
| kmalloc | ? | 18:21 |
| lbragstad | let me recheck | 18:22 |
| kmalloc | also... wonky | 18:22 |
| * lbragstad swiftly kicks memcache and keystone | 18:22 | |
| kmalloc | you're not getting any GETs in the multi-setup | 18:22 |
| kmalloc | because you should see GET/SET/GET/SET/GET/SET | 18:22 |
| kmalloc | afaict | 18:22 |
| *** sxc731_ has quit IRC | 18:23 | |
| lbragstad | yeah | 18:24 |
| lbragstad | ok - so if i kick memcache and keystone | 18:24 |
| lbragstad | and redo the test | 18:24 |
| lbragstad | i see the sets for everything | 18:25 |
| lbragstad | (catalog, roles, users, etc..) | 18:25 |
| lbragstad | and eventually the token | 18:25 |
| lbragstad | so it must be pre-seeded | 18:25 |
| kmalloc | there are some cases we did some magic for that | 18:26 |
| lbragstad | i know we did for authenticate | 18:26 |
| kmalloc | also, the SET might be hidden right after the first GET | 18:26 |
| lbragstad | we put the token in the cache after creation | 18:26 |
| kmalloc | so, if you just kick memcache over and validate you should see a set. | 18:26 |
| kmalloc | anyway. | 18:27 |
| lbragstad | yeah - i do | 18:27 |
| kmalloc | that aside. | 18:27 |
| lbragstad | configuring multiple backends seems hosed | 18:27 |
| kmalloc | weird. | 18:27 |
| kmalloc | i don't know why this would change the key. | 18:27 |
| lbragstad | me either | 18:27 |
| lbragstad | we don't accept kwargs on the method we wrap specifically for that reason | 18:28 |
| kmalloc | right, even though we have a kwarg mangle now. | 18:28 |
| kmalloc | bnut that aside | 18:28 |
| lbragstad | def _validate_token(self, token_id): | 18:28 |
| *** sxc731 has joined #openstack-keystone | 18:29 | |
| lbragstad | that's ^ the method we wrap | 18:29 |
| kmalloc | and the keys look like SHA1 hashes right? | 18:31 |
| kmalloc | when you're looking at them? | 18:31 |
| kmalloc | hold on trying to setup a VM really quickly to test this. | 18:32 |
| kmalloc | s/really quickly/looking for an image i can import/ | 18:32 |
| lbragstad | let me grab a snippet of the logs | 18:32 |
| lbragstad | http://paste.openstack.org/show/643995/ | 18:34 |
| *** vaishali has quit IRC | 18:34 | |
| *** hugokuo has quit IRC | 18:34 | |
| *** dstanek has quit IRC | 18:34 | |
| *** errr has quit IRC | 18:34 | |
| *** d34dh0r53 has quit IRC | 18:34 | |
| *** sxc731 has quit IRC | 18:34 | |
| *** d34dh0r53 has joined #openstack-keystone | 18:34 | |
| *** dstanek has joined #openstack-keystone | 18:34 | |
| lbragstad | CACHE_GET and CACHE_SET all in ^ | 18:34 |
| *** errr has joined #openstack-keystone | 18:35 | |
| *** sxc731 has joined #openstack-keystone | 18:35 | |
| *** hugokuo has joined #openstack-keystone | 18:35 | |
| lbragstad | the cache key for the one of the tokens being validated was c9019b39555c4c860ec8c4e95a605bb328c441e1 | 18:36 |
| lbragstad | it gets set once and then it gets a successful hit every time after that (so working as expected) | 18:36 |
| lbragstad | with a single cache | 18:36 |
| kmalloc | yeah that looks fine. | 18:36 |
| lbragstad | grabbing some logs with multi-cache configured | 18:37 |
| kmalloc | ~10minutes left on the image download. | 18:40 |
| lbragstad | weird | 18:41 |
| kmalloc | or... now 30m.. damn | 18:41 |
| lbragstad | every single CACHE_GET results in a dogpile.cache.api.NoValue object | 18:41 |
| lbragstad | so it attempts to set it | 18:41 |
| kmalloc | that is fine. that is normal operation if the backend is cache missing | 18:41 |
| kmalloc | NoValue is a magic "not set" value | 18:42 |
| lbragstad | only up the the total number of cache instances | 18:42 |
| lbragstad | right | 18:42 |
| lbragstad | technically your worst case would be 3 cache misses | 18:42 |
| lbragstad | if you hit every cache with the same key before hitting one that has it | 18:42 |
| kmalloc | this is sounding like a bug in python-memcache... | 18:43 |
| *** vaishali has joined #openstack-keystone | 18:43 | |
| lbragstad | either we're generating keys weird or yeah.. this is a bug in python-memcached | 18:43 |
| lbragstad | what about oslo.cache? | 18:43 |
| kmalloc | is the key consistent on requests? | 18:43 |
| *** sxc731 has quit IRC | 18:43 | |
| kmalloc | so c9xxxxxx is the same key on the GET | 18:44 |
| lbragstad | let me check quick | 18:44 |
| kmalloc | lets elminiate oslo/dogpile issues or memcache issues | 18:44 |
| *** vaishali has quit IRC | 18:44 | |
| *** vaishali has joined #openstack-keystone | 18:45 | |
| lbragstad | does paste.o.o have a limit size? | 18:45 |
| *** sxc731 has joined #openstack-keystone | 18:45 | |
| lbragstad | latest run with multi-cache enabled http://paste.openstack.org/show/644008/ | 18:47 |
| lbragstad | http://paste.openstack.org/raw/644008/ | 18:48 |
| kmalloc | hmm | 18:48 |
| kmalloc | looks like the cache key is consistent? | 18:49 |
| kmalloc | or am i mis-reading this | 18:49 |
| kmalloc | i wonder if this is a squashed exception somewhere | 18:50 |
| *** gyee has joined #openstack-keystone | 18:52 | |
| lbragstad | hah! | 18:52 |
| lbragstad | keys are getting generated differently | 18:52 |
| lbragstad | i dont' think my log showed it because i maxed out the paste limit | 18:52 |
| lbragstad | but if you look elsewhere in the logs | 18:53 |
| lbragstad | i can see the same token getting set | 18:53 |
| lbragstad | because of the audit-ids | 18:53 |
| lbragstad | and the audit-id chain | 18:53 |
| lbragstad | but they have different cache keys | 18:53 |
| kmalloc | wait. ... | 18:53 |
| kmalloc | that shouldn't be the case | 18:53 |
| kmalloc | if you are doing a GET on a single token with a consistent token | 18:53 |
| kmalloc | audit ids should be 100% the same | 18:53 |
| kmalloc | if you keep issuing a new token, yes it will have a chain id issue | 18:54 |
| lbragstad | right - they are | 18:54 |
| lbragstad | i'm validating the same token over and over | 18:54 |
| lbragstad | so the audit-ids are the same | 18:54 |
| kmalloc | ok, so... uh | 18:54 |
| lbragstad | which proves the cache key is different | 18:54 |
| lbragstad | for the same token | 18:54 |
| kmalloc | can you show me? | 18:54 |
| kmalloc | i don't see how that is possible | 18:54 |
| kmalloc | the audit_id and chain_id should be the same | 18:54 |
| lbragstad | let me find a paste service that works for logs | 18:54 |
| kmalloc | email me a log? | 18:55 |
| kmalloc | or hangout+shard screen | 18:55 |
| kmalloc | just looking for quick | 18:55 |
| lbragstad | https://hangouts.google.com/call/wkibMF0R0DX2D6mgkoSNAAEE | 18:57 |
| ayoung | lbragstad, are System scope assignments going to honor the implied roles mechanisms? | 19:00 |
| lbragstad | http://paste.openstack.org/show/644030/ | 19:01 |
| *** phalmos has joined #openstack-keystone | 19:04 | |
| lbragstad | ^ if anyone else is interested in debugging cache problems | 19:05 |
| *** sxc731 has quit IRC | 19:07 | |
| *** phalmos_ has joined #openstack-keystone | 19:08 | |
| *** phalmos has quit IRC | 19:09 | |
| *** sambetts is now known as sambetts|afk | 19:09 | |
| kmalloc | lbragstad: https://github.com/openstack/oslo.cache/blob/master/oslo_cache/core.py#L233 | 19:12 |
| *** gyee has quit IRC | 19:16 | |
| *** sxc731 has joined #openstack-keystone | 19:20 | |
| lbragstad | kmalloc: locals: {'key': 'keystone.token.provider:_validate_token|gAAAAABaWQd9Vr5NSlgloAbyGS-A7nkx660QtRKZVMblDSqUjwT0J_N4a9k-zDodsF9vvmmeSWAeyeRjVgDQjX96_Xe2ProJvyepPKsUL45rw5_i2KYNZznim2-yyw3sA0C2SVLT4_D4nKFROgR-HFtV6pbYhtbOA7BBaIdFEKK5t1Q i5D1zHXY:@+L\xdby(.\xb4\x03\xb6'} | 19:25 |
| *** sxc731 has quit IRC | 19:25 | |
| *** AlexeyAbashkin has joined #openstack-keystone | 19:25 | |
| *** abhi89 has joined #openstack-keystone | 19:26 | |
| *** blake has joined #openstack-keystone | 19:27 | |
| *** harlowja has joined #openstack-keystone | 19:27 | |
| kmalloc | lbragstad: https://bitbucket.org/zzzeek/dogpile.cache/src/4adfbe99ab6b5c9d67d367b28395dd6bc4a8764a/dogpile/cache/util.py?at=master&fileviewer=file-view-default#util.py-7:43 | 19:28 |
| *** david-lyle has joined #openstack-keystone | 19:33 | |
| *** jose-phi_ has quit IRC | 19:38 | |
| kmalloc | inspect.getargspec(fn | 19:38 |
| *** jose-phillips has joined #openstack-keystone | 19:39 | |
| *** AlexeyAbashkin has quit IRC | 19:42 | |
| *** sxc731 has joined #openstack-keystone | 19:47 | |
| *** bigdogstl has quit IRC | 19:54 | |
| *** blake has quit IRC | 19:55 | |
| *** sxc731 has quit IRC | 20:02 | |
| *** harlowja has quit IRC | 20:04 | |
| *** sxc731 has joined #openstack-keystone | 20:05 | |
| *** sxc731_ has joined #openstack-keystone | 20:05 | |
| *** sxc731_ has quit IRC | 20:08 | |
| openstackgerrit | Colleen Murphy proposed openstack/keystone master: Extract expiration validation to utils https://review.openstack.org/532257 | 20:09 |
| sxc731 | lbragstad: so you've managed to reproduce the issue and you're looking for a fix with kmalloc, right? Would it be useful if I raised a ticket? | 20:10 |
| lbragstad | ayoung: eventually | 20:10 |
| lbragstad | sxc731: well - it's strange... | 20:10 |
| kmalloc | yeah standing up a VM now. | 20:10 |
| kmalloc | it's wonky | 20:10 |
| sxc731 | In my experience the absence of cache hits due to different keys seems 100% reproducible with multiple caches. I'm just surprised nobody else noticed because what I used was the off-the-shelf OSA deployment pattern. And the performance impact is _very_ noticeable | 20:11 |
| *** bigdogstl has joined #openstack-keystone | 20:13 | |
| lbragstad | yeah - we're trying to figure out if it is a keystone issue or not | 20:14 |
| lbragstad | keystone is passing the same key, but when dogpile or oslo.cache goes to generate a cache key, it comes up different each time | 20:14 |
| lbragstad | with multiple cache backends configured | 20:15 |
| lbragstad | which is the weird part | 20:15 |
| sxc731 | kmalloc mentioned a switch from keystone-based interface to memcache to oslo.cache in the recent-ish past... | 20:15 |
| kmalloc | yeah a while ago | 20:16 |
| kmalloc | we swapped to oslo.cache | 20:16 |
| kmalloc | at some point | 20:16 |
| sxc731 | I didn't experience | 20:16 |
| lbragstad | yeah - that was pre-pike | 20:16 |
| lbragstad | for sure | 20:16 |
| sxc731 | ... this issue in Kilo | 20:16 |
| sxc731 | Just speculating, not having read the code at all but I don't suppose the cache instance being has any bearing on the key generation? | 20:17 |
| *** bigdogstl has quit IRC | 20:19 | |
| lbragstad | that's the part i'm about to start digging into | 20:23 |
| *** bigdogstl has joined #openstack-keystone | 20:25 | |
| sxc731 | I'm still baffled that nobody noticed this before... | 20:26 |
| lbragstad | me too | 20:27 |
| lbragstad | sxc731: solid find, sir... solid find | 20:28 |
| *** bigdogstl has quit IRC | 20:30 | |
| *** bigdogstl has joined #openstack-keystone | 20:33 | |
| lbragstad | kmalloc: i can confirm that this is what is getting called | 20:38 |
| lbragstad | https://bitbucket.org/zzzeek/dogpile.cache/src/4adfbe99ab6b5c9d67d367b28395dd6bc4a8764a/dogpile/cache/util.py?at=master&fileviewer=file-view-default#util.py-42 | 20:38 |
| kmalloc | yeah | 20:38 |
| kmalloc | i knew that much | 20:38 |
| kmalloc | installing keystone now | 20:38 |
| lbragstad | http://paste.openstack.org/show/644099/ | 20:39 |
| sxc731 | Here's the defect for committing against: https://bugs.launchpad.net/keystone/+bug/1743036 | 20:39 |
| openstack | Launchpad bug 1743036 in OpenStack Identity (keystone) "Multiple memcached back-end instances breaks caching" [Undecided,New] | 20:39 |
| lbragstad | so - the key that is generated there doesn't have garbage at the end... | 20:40 |
| lbragstad | like we were seeing in oslo.cache logs | 20:40 |
| lbragstad | and that is called from https://bitbucket.org/zzzeek/dogpile.cache/src/4adfbe99ab6b5c9d67d367b28395dd6bc4a8764a/dogpile/cache/region.py?at=master&fileviewer=file-view-default#region.py-1241 | 20:45 |
| lbragstad | well - https://bitbucket.org/zzzeek/dogpile.cache/src/4adfbe99ab6b5c9d67d367b28395dd6bc4a8764a/dogpile/cache/region.py?at=master&fileviewer=file-view-default#region.py-1242 specifically | 20:45 |
| *** sxc731 has quit IRC | 20:45 | |
| *** sxc731_ has joined #openstack-keystone | 20:46 | |
| *** openstackstatus has quit IRC | 20:46 | |
| *** openstack has joined #openstack-keystone | 20:47 | |
| *** ChanServ sets mode: +o openstack | 20:47 | |
| *** openstackstatus has joined #openstack-keystone | 20:48 | |
| *** ChanServ sets mode: +v openstackstatus | 20:48 | |
| lbragstad | kmalloc: the garbarge get amended here - http://paste.openstack.org/show/644108/ | 20:52 |
| lbragstad | https://github.com/openstack/keystone/blob/master/keystone/common/cache/core.py#L87 | 20:52 |
| kmalloc | ugh | 20:52 |
| kmalloc | yeah that might actually cause issues with multiple processes. | 20:52 |
| lbragstad | ipdb> invalidation_manager.region_id | 20:52 |
| lbragstad | '!X\xbd\xc2=q\xe0\x86\x8e\xf2' | 20:52 |
| kmalloc | let me check waht region_id actually is. | 20:53 |
| kmalloc | yeah, what is THAT supposed to be? | 20:53 |
| lbragstad | good question | 20:53 |
| lbragstad | dstanek: might know :) | 20:53 |
| lbragstad | we somehow stub this in https://github.com/openstack/keystone/blob/master/keystone/common/cache/core.py#L31 | 20:54 |
| kmalloc | lame | 20:55 |
| kmalloc | return os.urandom(10) | 20:55 |
| kmalloc | ok THAT is ... | 20:55 |
| lbragstad | yes... | 20:55 |
| kmalloc | i just don't get it | 20:56 |
| lbragstad | we seem to like calling it | 20:56 |
| lbragstad | and we use it when we invalidate a region | 20:56 |
| kmalloc | yeah that is an issue | 20:56 |
| kmalloc | like... THAT SHOULD JUST BE NAMESPACE changing | 20:56 |
| kmalloc | w.t.g | 20:56 |
| kmalloc | what in the ... ugh | 20:57 |
| kmalloc | this is icky | 20:57 |
| *** panbalag has joined #openstack-keystone | 20:57 | |
| kmalloc | ok i think this is a keystone bug, and it's going to need some rather serious unwinding. | 20:58 |
| kmalloc | we're setting random crap on the cache keys | 20:59 |
| kmalloc | and each process is going to cache it differently afaict | 20:59 |
| kmalloc | this is... bad news. | 20:59 |
| lbragstad | so - this was the patch we wrote to handle distributed cache invalidation https://github.com/openstack/keystone/commit/42eda48c78f1153081b4c193dc13c88561409fd3 | 21:01 |
| *** fried_rolls is now known as fried_rice | 21:01 | |
| kmalloc | yeah | 21:01 |
| kmalloc | the fact we generate random junk is a serious issue | 21:02 |
| *** itlinux has quit IRC | 21:02 | |
| kmalloc | because we hard set this to in-memory | 21:02 |
| lbragstad | the weird part is that we generate it on the key... when it looks region specific | 21:03 |
| kmalloc | yeah it's so we can force a whole region to be invalidated at once | 21:03 |
| lbragstad | at least that's weird to me... | 21:03 |
| kmalloc | it was/is a hack around bugs in dogpile | 21:03 |
| kmalloc | i think this is no longer needed with modern dogpile | 21:03 |
| kmalloc | but... i need to dig into to make sure we didn't just land the same broken thing to dogpile | 21:03 |
| kmalloc | i *think*. | 21:03 |
| kmalloc | this is going to take a chunk of time to unwind. | 21:04 |
| lbragstad | ok - i want to understand how this works when a single backend is there but it doesn't for multiple | 21:05 |
| lbragstad | where is that case in our keystone code? | 21:05 |
| kmalloc | this might actually be ok code wise. | 21:06 |
| kmalloc | yeah ok, this is mostly ok. | 21:07 |
| lbragstad | https://github.com/openstack/keystone/blob/master/keystone/common/cache/core.py#L161 | 21:08 |
| lbragstad | that looks suspicious | 21:08 |
| * lbragstad is suspicious of everything now | 21:08 | |
| kmalloc | thats fine | 21:10 |
| kmalloc | ok this is an oke patch, but i really dislike we use a random() function instead of a real string | 21:12 |
| kmalloc | ... we should fix that to be something like uuid.uuid4.hex | 21:12 |
| kmalloc | and we should replace the namespace NOT tack random crap on the end. | 21:13 |
| kmalloc | but... | 21:13 |
| kmalloc | that is a different story | 21:13 |
| kmalloc | this should be using the same mechanism for setting up backends as the primary config | 21:13 |
| kmalloc | so, it should be perfectly fine. | 21:13 |
| lbragstad | https://bitbucket.org/zzzeek/dogpile.cache/src/4adfbe99ab6b5c9d67d367b28395dd6bc4a8764a/dogpile/cache/region.py?at=master&fileviewer=file-view-default#region.py-537 | 21:13 |
| kmalloc | yeah | 21:14 |
| kmalloc | we have to override it | 21:14 |
| lbragstad | yeah looks like it uses the method from dog pile | 21:14 |
| lbragstad | it's blowing my mind how this works for a single backend but not three | 21:14 |
| lbragstad | because all the backend stuff is being handled by dogpile as far as i can tell | 21:15 |
| *** lintonv has quit IRC | 21:15 | |
| kmalloc | yeh | 21:15 |
| *** panbalag has joined #openstack-keystone | 21:15 | |
| kmalloc | unless we're getting some inconsistency in memcache backends. | 21:15 |
| kmalloc | in how we're passing backends. | 21:16 |
| lbragstad | so - do we conditionally build regions based on backends? | 21:16 |
| kmalloc | we do the buildconf | 21:17 |
| kmalloc | from the oslo.conf data | 21:17 |
| kmalloc | which turns into a dict that is passed to the backend(s) | 21:17 |
| *** spzala has quit IRC | 21:18 | |
| lbragstad | yep - that makes sense | 21:19 |
| lbragstad | and from what i can tell, that seems sane | 21:19 |
| *** spzala has joined #openstack-keystone | 21:19 | |
| kmalloc | wait let me see your config again | 21:20 |
| kmalloc | your config is bad | 21:20 |
| kmalloc | this is incorrect: backend_argument = url:192.168.122.243,192.168.122.241,192.168.122.144:11211 | 21:21 |
| lbragstad | http://paste.openstack.org/show/644116/ | 21:21 |
| kmalloc | you're missing ports on the first 2 elements | 21:21 |
| lbragstad | hmm | 21:21 |
| lbragstad | that's weird | 21:21 |
| lbragstad | is that format documented somewhere? | 21:22 |
| kmalloc | so, my guess is you're hitting a case where the servers are being marked down and erroring | 21:22 |
| kmalloc | i think so* | 21:22 |
| kmalloc | but the format is......... | 21:22 |
| kmalloc | sec | 21:22 |
| lbragstad | i think that was part of the OSA template | 21:22 |
| kmalloc | cfg.MultiStrOpt('backend_argument', default=[], secret=True, | 21:23 |
| kmalloc | help='Arguments supplied to the backend module. ' | 21:23 |
| kmalloc | 'Specify this option once per argument to be ' | 21:23 |
| kmalloc | 'passed to the dogpile.cache backend. Example ' | 21:23 |
| kmalloc | 'format: "<argname>:<value>".'), | 21:23 |
| kmalloc | so: <key>:<value> | 21:23 |
| lbragstad | so | 21:23 |
| lbragstad | backend_argument = url:192.168.122.243:11211,192.168.122.241:11211,192.168.122.144:11211 | 21:23 |
| kmalloc | yes | 21:23 |
| lbragstad | or .. | 21:23 |
| kmalloc | oor | 21:23 |
| kmalloc | you could use memcached_servers opt | 21:23 |
| kmalloc | memcache_servers*( | 21:23 |
| *** spzala has quit IRC | 21:23 | |
| kmalloc | i'll be this is a case where we are hitting issues with servers being marked as dead and the ring distribution is breaking in python-memcached | 21:24 |
| lbragstad | backend_argument = url:192.168.122.243:11211 | 21:24 |
| lbragstad | backend_argument = url:192.168.122.241:11211 | 21:24 |
| lbragstad | backend_argument = url:192.168.122.144:11211 | 21:24 |
| kmalloc | nope | 21:24 |
| kmalloc | you must combine all values for a single key on a single line | 21:24 |
| *** spzala has joined #openstack-keystone | 21:24 | |
| lbragstad | mmm | 21:25 |
| kmalloc | backend_argument may be multiple backend arguments | 21:25 |
| lbragstad | that's misleading when the option is defined with a name of MultiStrOpt | 21:25 |
| kmalloc | it is because you might have url, expiry, password | 21:25 |
| kmalloc | etc | 21:25 |
| kmalloc | all different options that need to be passed in a backend-specific way | 21:25 |
| kmalloc | the multistropt is about allowing that | 21:25 |
| kmalloc | url is just *one* of them | 21:26 |
| lbragstad | {"error": {"message": "An unexpected error prevented the server from fulfilling your request: Unable to parse connection string: \"192.168.122.243:11211,192.168.122.241:11211,192.168.122.144:11211\". (Disable insecure_debug mode to suppress these details.)", "code": 500, "title": "Internal Server Error"}} | 21:26 |
| *** harlowja has joined #openstack-keystone | 21:26 | |
| kmalloc | wait.. wut. | 21:26 |
| kmalloc | use memcache_servers | 21:27 |
| kmalloc | it's a list opt | 21:27 |
| kmalloc | use that for the moment | 21:27 |
| kmalloc | see if it fixes the bug | 21:27 |
| kmalloc | we can chase the other issues down after | 21:27 |
| kmalloc | https://bitbucket.org/zzzeek/dogpile.cache/src/4adfbe99ab6b5c9d67d367b28395dd6bc4a8764a/dogpile/cache/backends/memcached.py?at=master&fileviewer=file-view-default#memcached.py-57:59 | 21:28 |
| kmalloc | yeah. the backend argument bit is somewhat broken somehow | 21:28 |
| kmalloc | it's going to need to be re-worked | 21:28 |
| kmalloc | yeah that is going to be just broken. | 21:29 |
| kmalloc | in short, don't use that mechanism to configure for now | 21:29 |
| *** spzala has quit IRC | 21:30 | |
| kmalloc | i'll bet this has been broken for quite a while. | 21:30 |
| lbragstad | so - memcache_servers = 192.168.122.243:11211,192.168.122.241:11211,192.168.122.144:11211 | 21:30 |
| kmalloc | lbragstad: yeah. | 21:31 |
| kmalloc | annnnnd | 21:31 |
| kmalloc | actually it might be multiple lines, for the multistropt, id' need to poke at oslo.config to see what it does. | 21:31 |
| kmalloc | you're example with backend_argument = url:<xxxx> multiple times is correct | 21:31 |
| kmalloc | your* | 21:31 |
| lbragstad | but with memcache_servers? | 21:32 |
| lbragstad | with memcache_servers = 192.168.122.243:11211,192.168.122.241:11211,192.168.122.144:11211 i'm still only seeing traffic to the first server | 21:32 |
| *** itlinux has joined #openstack-keystone | 21:32 | |
| kmalloc | that is going to depend on how the ring is parsed in python-memcached | 21:33 |
| kmalloc | is it still missing on 100% of the calls? | 21:33 |
| kmalloc | that's all i care about to start. | 21:33 |
| lbragstad | caching is working | 21:33 |
| kmalloc | try with multiple lines on the multistropt | 21:33 |
| lbragstad | oh - wait | 21:33 |
| kmalloc | if memcache_servers is working | 21:34 |
| *** spzala has joined #openstack-keystone | 21:34 | |
| lbragstad | so - caching works with memcache_servers = 192.168.122.243:11211,192.168.122.241:11211,192.168.122.144:11211 | 21:35 |
| lbragstad | but i only see traffic to 243 | 21:35 |
| lbragstad | tailing syslog on each memcache instance and that is the only one reporting GET and SET traffic | 21:36 |
| *** edmondsw has quit IRC | 21:36 | |
| kmalloc | that is related to how the buckets work internal to python-memcache | 21:37 |
| kmalloc | i don't have a lot of insight | 21:37 |
| kmalloc | try with the multi-line url: bit | 21:37 |
| kmalloc | for backend-argument | 21:37 |
| kmalloc | my guess is it will be the same | 21:37 |
| kmalloc | https://github.com/linsomniac/python-memcached/blob/master/memcache.py#L169 | 21:39 |
| *** spzala has quit IRC | 21:39 | |
| lbragstad | ok - this is what i have | 21:39 |
| lbragstad | http://paste.openstack.org/show/644127/ | 21:39 |
| kmalloc | yep lets try that | 21:40 |
| kmalloc | if that works like i think it will... | 21:40 |
| kmalloc | but i am not 100% sure | 21:40 |
| lbragstad | it only sends traffic to the first one in config | 21:42 |
| lbragstad | confirmed by switching them around | 21:42 |
| kmalloc | hm. | 21:43 |
| kmalloc | can you debug what the config ends up looking like | 21:43 |
| lbragstad | huh | 21:44 |
| lbragstad | nevermind | 21:44 |
| kmalloc | ? | 21:44 |
| lbragstad | bumped up the logging | 21:44 |
| kmalloc | is it working as expected? | 21:44 |
| lbragstad | with memcache_servers = 192.168.122.243:11211,192.168.122.241:11211,192.168.122.144:11211 | 21:44 |
| lbragstad | running a loop so i can see traffic | 21:44 |
| lbragstad | on each | 21:44 |
| kmalloc | ok backend-argument is broken i think | 21:44 |
| kmalloc | the more i look at this | 21:44 |
| kmalloc | it... never should ahve worked | 21:45 |
| kmalloc | tl;dr, use memcache_servers | 21:45 |
| kmalloc | it is the only way that consistently works atm | 21:45 |
| kmalloc | we're going to need to totally re-work the processing for backend-arguemnt | 21:46 |
| lbragstad | yep - ok cool | 21:46 |
| lbragstad | i can confirm | 21:46 |
| kmalloc | but is in oslo.cache | 21:46 |
| kmalloc | bug* | 21:46 |
| lbragstad | using memcached servers in a list works as expected | 21:46 |
| lbragstad | and it sends traffic to each memcache server | 21:46 |
| kmalloc | yep | 21:46 |
| lbragstad | and it looks like they are all responding within reasonable time | 21:46 |
| kmalloc | and not getting endless misses | 21:46 |
| kmalloc | yeah | 21:46 |
| lbragstad | cc sxc731_^ | 21:46 |
| kmalloc | ok, i know what needs to happen. but very short, backend-arguemnt is totally busted. | 21:47 |
| kmalloc | we have to add a bunch of hint processing for the backends to have a clue what it should be | 21:47 |
| lbragstad | example response times with a cluster of 3 memcache servers | 21:47 |
| openstackgerrit | Gage Hugo proposed openstack/keystone master: WIP - Add functional testing gate https://review.openstack.org/531014 | 21:47 |
| lbragstad | http://paste.openstack.org/show/644131/ | 21:47 |
| kmalloc | so possiby something like "list:delim🔑value" | 21:47 |
| kmalloc | or <type>|<delim>|<key>|<value> | 21:48 |
| kmalloc | something so we know what the type should be + delimiter if relevant | 21:48 |
| kmalloc | it's going to be a lot of re-working | 21:48 |
| kmalloc | so it probably should look like backend-argument = type:list:,|url|192.168.122.243:11211,192.168.122.244:11211 | 21:49 |
| kmalloc | and the default behavior if the there is no "type" hint, is to process as a string like it is today | 21:50 |
| lbragstad | yeah | 21:52 |
| lbragstad | well - that was fun | 21:52 |
| *** harsha has joined #openstack-keystone | 21:52 | |
| *** r-daneel has joined #openstack-keystone | 21:52 | |
| * lbragstad heads over to osa | 21:52 | |
| kmalloc | go ahead and assign me the bug. | 21:52 |
| kmalloc | it's going to take a bit of work to re-do this | 21:52 |
| openstackgerrit | Gage Hugo proposed openstack/keystone master: WIP - Add functional testing gate https://review.openstack.org/531014 | 21:53 |
| harsha | Hello, can anyone let me know what's the best configuration for the number of WSGIDaemonProcess process and threads that can be configured for keystone | 21:53 |
| *** spzala has joined #openstack-keystone | 21:56 | |
| lbragstad | kmalloc: i don't have the power to do that for oslo.cache | 21:57 |
| lbragstad | https://bugs.launchpad.net/oslo.cache/+bug/1743036 | 21:57 |
| openstack | Launchpad bug 1743036 in oslo.cache "Multiple memcached back-end instances breaks caching" [Undecided,Confirmed] | 21:57 |
| lbragstad | but i marked it as invalid for keystone | 21:57 |
| kmalloc | k | 21:57 |
| lbragstad | harsha: that's going to totally depend on the hardware you have | 21:57 |
| lbragstad | harsha: that openstack-ansible commuity might be able to point you in the first direction since they do some math when automating the deployment of keystone | 21:58 |
| *** panbalag has quit IRC | 21:58 | |
| harsha | @lbragstad thanks you | 21:59 |
| sxc731_ | Lbragstadt, kmalloc: thank you for your hard work guys. Not only have you figured out the issue but you even have a workaround. Outstanding! | 22:03 |
| *** spzala has quit IRC | 22:04 | |
| *** spzala has joined #openstack-keystone | 22:05 | |
| lbragstad | sxc731_: anytime - i have the workaround proposed to osa - https://review.openstack.org/#/c/533314/ | 22:06 |
| lbragstad | that should hopefully get you around the issue until we get to the bottom of the python-memcache issues | 22:06 |
| lbragstad | or oslo.cache issues | 22:06 |
| kmalloc | it's def oslo.cache issues | 22:06 |
| sxc731_ | lbragstad: also explains why this wasn’t widely experienced before. If you used the old-style config, you’d be fine... Only too bad for OSA deployments... | 22:07 |
| lbragstad | right | 22:07 |
| *** itlinux has quit IRC | 22:10 | |
| *** spzala has quit IRC | 22:17 | |
| *** sxc731_ has quit IRC | 22:21 | |
| *** phalmos_ has quit IRC | 22:26 | |
| openstackgerrit | Colleen Murphy proposed openstack/keystone master: Add application credentials db migration https://review.openstack.org/524927 | 22:27 |
| openstackgerrit | Colleen Murphy proposed openstack/keystone master: Add application credentials driver https://review.openstack.org/524928 | 22:27 |
| openstackgerrit | Colleen Murphy proposed openstack/keystone master: Add Application Credentials manager https://review.openstack.org/524747 | 22:27 |
| openstackgerrit | Colleen Murphy proposed openstack/keystone master: Add Application Credentials controller https://review.openstack.org/524423 | 22:27 |
| openstackgerrit | Colleen Murphy proposed openstack/keystone master: WIP Add application credential auth plugin https://review.openstack.org/525346 | 22:27 |
| *** harsha has quit IRC | 22:27 | |
| *** dikonoor has quit IRC | 22:36 | |
| *** dikonoor has joined #openstack-keystone | 22:36 | |
| *** spzala has joined #openstack-keystone | 22:37 | |
| *** dave-mccowan has quit IRC | 22:39 | |
| *** spzala has quit IRC | 22:42 | |
| *** AlexeyAbashkin has joined #openstack-keystone | 22:54 | |
| *** AlexeyAbashkin has quit IRC | 22:59 | |
| *** panbalag has joined #openstack-keystone | 23:00 | |
| *** harlowja has quit IRC | 23:01 | |
| *** spzala has joined #openstack-keystone | 23:01 | |
| *** raildo has quit IRC | 23:04 | |
| *** panbalag has quit IRC | 23:09 | |
| *** spzala has quit IRC | 23:10 | |
| *** spzala has joined #openstack-keystone | 23:13 | |
| *** spzala has quit IRC | 23:20 | |
| *** itlinux has joined #openstack-keystone | 23:22 | |
| *** spzala has joined #openstack-keystone | 23:25 | |
| *** spzala has quit IRC | 23:30 | |
| *** fried_rice is now known as efried | 23:35 | |
| *** r-daneel has quit IRC | 23:35 | |
| *** spzala has joined #openstack-keystone | 23:37 | |
| *** itlinux has quit IRC | 23:39 | |
| *** spzala has quit IRC | 23:46 | |
| *** itlinux has joined #openstack-keystone | 23:57 | |
| *** spzala has joined #openstack-keystone | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!