| *** erhudy has quit IRC | 00:00 | |
| *** lamt has joined #openstack-keystone | 00:08 | |
| *** yingwei has joined #openstack-keystone | 00:38 | |
| *** markvoelker has joined #openstack-keystone | 00:41 | |
| *** markvoelker has quit IRC | 00:46 | |
| *** browne has quit IRC | 01:06 | |
| *** afred312 has joined #openstack-keystone | 01:09 | |
| *** hoonetorg has quit IRC | 01:15 | |
| *** erlon has joined #openstack-keystone | 01:19 | |
| *** hoonetorg has joined #openstack-keystone | 01:20 | |
| *** spzala has quit IRC | 01:22 | |
| *** spzala has joined #openstack-keystone | 01:23 | |
| *** spzala has quit IRC | 01:27 | |
| *** markvoelker has joined #openstack-keystone | 01:38 | |
| *** spzala has joined #openstack-keystone | 01:44 | |
| *** spzala has quit IRC | 01:49 | |
| *** spzala has joined #openstack-keystone | 01:50 | |
| *** spzala has quit IRC | 01:55 | |
| *** zhurong has joined #openstack-keystone | 02:05 | |
| *** phalmos has quit IRC | 02:05 | |
| *** Nakato has quit IRC | 02:06 | |
| *** Nakato has joined #openstack-keystone | 02:09 | |
| *** spzala has joined #openstack-keystone | 02:11 | |
| *** edmondsw has joined #openstack-keystone | 02:13 | |
| *** spzala has quit IRC | 02:16 | |
| *** edmondsw has quit IRC | 02:18 | |
| *** dave-mccowan has quit IRC | 02:27 | |
| *** spzala has joined #openstack-keystone | 02:32 | |
| *** Nakato has quit IRC | 02:32 | |
| *** Nakato has joined #openstack-keystone | 02:33 | |
| *** spzala has quit IRC | 02:36 | |
| *** thorst has quit IRC | 02:36 | |
| *** prashkre has joined #openstack-keystone | 02:44 | |
| *** spzala has joined #openstack-keystone | 02:48 | |
| *** prashkre has quit IRC | 02:49 | |
| *** openstackgerrit has joined #openstack-keystone | 02:50 | |
| openstackgerrit | ChangBo Guo(gcb) proposed openstack/keystone master: Remove usage of enforce_type https://review.openstack.org/455391 | 02:50 |
|---|---|---|
| *** spzala has quit IRC | 02:53 | |
| *** hoonetorg has quit IRC | 02:59 | |
| *** spzala has joined #openstack-keystone | 03:01 | |
| *** gcb has joined #openstack-keystone | 03:02 | |
| *** thorst has joined #openstack-keystone | 03:07 | |
| openstackgerrit | Shan Guo proposed openstack/keystone master: Remove unused log translation code https://review.openstack.org/457916 | 03:09 |
| *** thorst has quit IRC | 03:11 | |
| *** hoonetorg has joined #openstack-keystone | 03:15 | |
| *** david-lyle is now known as dklyle | 03:19 | |
| *** dklyle is now known as david-lyle | 03:20 | |
| *** david-lyle is now known as dklyle | 03:21 | |
| *** nicolasbock has quit IRC | 03:22 | |
| *** dklyle is now known as david-lyle | 03:23 | |
| *** thorst has joined #openstack-keystone | 03:38 | |
| *** zhurong has quit IRC | 03:38 | |
| *** aojea has joined #openstack-keystone | 03:38 | |
| *** thorst has quit IRC | 03:45 | |
| *** aojea has quit IRC | 03:45 | |
| openstackgerrit | Merged openstack/keystone master: Readability enhancements to architecture doc https://review.openstack.org/422375 | 03:48 |
| *** gagehugo has quit IRC | 03:58 | |
| *** gagehugo has joined #openstack-keystone | 03:59 | |
| *** thorst has joined #openstack-keystone | 04:41 | |
| *** thorst has quit IRC | 04:45 | |
| *** zhurong has joined #openstack-keystone | 05:01 | |
| *** gyee has quit IRC | 05:03 | |
| openstackgerrit | zhengliuyang proposed openstack/keystone master: use '&' instead of '?' to connect parameters in url https://review.openstack.org/460826 | 05:05 |
| *** erlon has quit IRC | 05:13 | |
| *** mkrai_ has joined #openstack-keystone | 05:29 | |
| mkrai_ | Hi I am facing some error in keystone while running devstack setup | 05:30 |
| mkrai_ | logs http://paste.openstack.org/show/608279/ | 05:31 |
| mkrai_ | Can anyone help? | 05:31 |
| mkrai_ | samueldmq: Hi there | 05:34 |
| *** richm has quit IRC | 05:43 | |
| *** edmondsw has joined #openstack-keystone | 05:50 | |
| *** yingwei has quit IRC | 05:53 | |
| *** edmondsw has quit IRC | 05:54 | |
| *** lamt has quit IRC | 05:56 | |
| *** ducttap__ has joined #openstack-keystone | 06:00 | |
| *** zhurong has quit IRC | 06:04 | |
| *** ducttap__ has quit IRC | 06:05 | |
| *** adrian_otto has joined #openstack-keystone | 06:05 | |
| *** zhurong has joined #openstack-keystone | 06:12 | |
| *** adrian_otto has quit IRC | 06:15 | |
| *** Shunli has joined #openstack-keystone | 06:16 | |
| *** zhurong has quit IRC | 06:23 | |
| *** voelzmo has joined #openstack-keystone | 06:24 | |
| *** pnavarro has quit IRC | 06:32 | |
| *** pcaruana has joined #openstack-keystone | 06:36 | |
| *** Aqsa has joined #openstack-keystone | 06:41 | |
| *** thorst has joined #openstack-keystone | 06:42 | |
| *** thorst has quit IRC | 06:46 | |
| *** tesseract has joined #openstack-keystone | 07:05 | |
| *** ducttape_ has joined #openstack-keystone | 07:10 | |
| *** ducttape_ has quit IRC | 07:14 | |
| *** aojea has joined #openstack-keystone | 07:18 | |
| *** edmondsw has joined #openstack-keystone | 07:38 | |
| *** edmondsw has quit IRC | 07:42 | |
| *** thorst has joined #openstack-keystone | 07:43 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-keystone | 08:00 | |
| *** andreaf has quit IRC | 08:02 | |
| *** thorst has quit IRC | 08:02 | |
| *** andreaf has joined #openstack-keystone | 08:04 | |
| *** aojea has quit IRC | 08:08 | |
| openstackgerrit | zhengliuyang proposed openstack/keystone master: Add filter explain in api ref about parents_as_list and subtree_as_list https://review.openstack.org/458307 | 08:23 |
| openstackgerrit | ChangBo Guo(gcb) proposed openstack/keystone master: Remove test_metadata_invalid_contact_type https://review.openstack.org/460873 | 08:27 |
| *** pnavarro has joined #openstack-keystone | 08:28 | |
| openstackgerrit | Merged openstack/keystone master: Remove unused CONF https://review.openstack.org/459041 | 08:44 |
| openstackgerrit | Merged openstack/keystone master: Remove unused LOG https://review.openstack.org/459038 | 08:44 |
| *** jaosorior_away is now known as jaosorior | 08:49 | |
| openstackgerrit | ChangBo Guo(gcb) proposed openstack/keystone master: Remove test_minimum_password_age_and_password_expires_days_deactivated https://review.openstack.org/460879 | 08:49 |
| *** thorst has joined #openstack-keystone | 08:59 | |
| *** aojea has joined #openstack-keystone | 08:59 | |
| *** thorst has quit IRC | 09:10 | |
| *** MasterOfBugs has quit IRC | 09:14 | |
| *** markvoelker has quit IRC | 09:16 | |
| *** odyssey4me_ is now known as odyssey4me | 09:20 | |
| *** adriant_ has quit IRC | 09:23 | |
| *** edmondsw has joined #openstack-keystone | 09:26 | |
| *** Shunli has quit IRC | 09:26 | |
| *** edmondsw has quit IRC | 09:30 | |
| *** jaosorior is now known as jaosorior_away | 10:00 | |
| *** pnavarro has quit IRC | 10:03 | |
| *** nicolasbock has joined #openstack-keystone | 10:03 | |
| *** tesseract has quit IRC | 10:04 | |
| *** thorst has joined #openstack-keystone | 10:07 | |
| *** tesseract has joined #openstack-keystone | 10:08 | |
| *** thorst has quit IRC | 10:11 | |
| *** richm has joined #openstack-keystone | 10:15 | |
| *** markvoelker has joined #openstack-keystone | 10:17 | |
| *** markvoelker has quit IRC | 10:22 | |
| *** raildo has joined #openstack-keystone | 11:02 | |
| lbragstad | mkrai_ it looks like something similar to the devstack failures we saw when the switch to uwsgi was made | 11:27 |
| lbragstad | mkrai_ maybe ask in the -qa channel to see if it's similar to what they were seeing last week? | 11:27 |
| *** thorst has joined #openstack-keystone | 11:44 | |
| *** tesseract has quit IRC | 11:50 | |
| lbragstad | rodrigods i responded to your comment here - https://review.openstack.org/#/c/455391/4 | 11:54 |
| lbragstad | rodrigods does that help? | 11:54 |
| *** dave-mccowan has joined #openstack-keystone | 11:56 | |
| *** aojea has quit IRC | 12:01 | |
| *** d0ugal_ has joined #openstack-keystone | 12:02 | |
| *** d0ugal has quit IRC | 12:04 | |
| *** tesseract has joined #openstack-keystone | 12:05 | |
| rodrigods | lbragstad, thanks | 12:11 |
| *** pnavarro has joined #openstack-keystone | 12:11 | |
| *** chlong has quit IRC | 12:21 | |
| *** d0ugal_ is now known as d0ugal | 12:21 | |
| *** d0ugal is now known as Guest8049 | 12:21 | |
| *** Guest8049 has quit IRC | 12:21 | |
| *** d0ugal_ has joined #openstack-keystone | 12:22 | |
| *** edmondsw has joined #openstack-keystone | 12:22 | |
| *** edmondsw has quit IRC | 12:24 | |
| *** edmondsw has joined #openstack-keystone | 12:25 | |
| *** edmondsw has quit IRC | 12:29 | |
| *** edmondsw has joined #openstack-keystone | 12:34 | |
| *** markvoelker has joined #openstack-keystone | 12:38 | |
| *** gcb has quit IRC | 12:40 | |
| *** ducttape_ has joined #openstack-keystone | 12:57 | |
| *** catintheroof has joined #openstack-keystone | 13:01 | |
| *** catintheroof has quit IRC | 13:04 | |
| *** catintheroof has joined #openstack-keystone | 13:04 | |
| *** lamt has joined #openstack-keystone | 13:04 | |
| *** Aqsam has joined #openstack-keystone | 13:07 | |
| *** Aqsa has quit IRC | 13:07 | |
| *** lamt has quit IRC | 13:09 | |
| *** ducttap__ has joined #openstack-keystone | 13:13 | |
| *** lamt has joined #openstack-keystone | 13:14 | |
| *** ducttape_ has quit IRC | 13:17 | |
| *** catinthe_ has joined #openstack-keystone | 13:18 | |
| *** catintheroof has quit IRC | 13:21 | |
| *** arturb has quit IRC | 13:23 | |
| *** d0ugal_ has quit IRC | 13:26 | |
| *** d0ugal has joined #openstack-keystone | 13:27 | |
| *** d0ugal has quit IRC | 13:27 | |
| *** d0ugal has joined #openstack-keystone | 13:27 | |
| *** aojea has joined #openstack-keystone | 13:36 | |
| *** Guest24728 is now known as zeus | 13:37 | |
| *** zeus has quit IRC | 13:37 | |
| *** zeus has joined #openstack-keystone | 13:37 | |
| openstackgerrit | Merged openstack/keystone master: Remove usage of enforce_type https://review.openstack.org/455391 | 13:37 |
| efried | mordred I can't tell for sure, but I think we may have some operators who still want multiple [glance]api_servers - see http://lists.openstack.org/pipermail/openstack-dev/2017-April/116028.html | 13:38 |
| *** ducttap__ has quit IRC | 13:40 | |
| *** edmondsw_ has joined #openstack-keystone | 13:42 | |
| *** edmondsw has quit IRC | 13:43 | |
| mordred | lemme go read - thanks! | 13:43 |
| mordred | oh dear god | 13:45 |
| *** Dinesh_Bhor has quit IRC | 13:46 | |
| efried | mordred Will there be screaming now? | 13:46 |
| efried | <rubs hands together> | 13:47 |
| mordred | efried: well, I'm going to start with not-screaming :) | 13:47 |
| efried | A sane approach, always. | 13:47 |
| *** lamt has quit IRC | 13:49 | |
| *** lamt has joined #openstack-keystone | 13:51 | |
| *** chlong has joined #openstack-keystone | 13:53 | |
| *** ducttape_ has joined #openstack-keystone | 14:05 | |
| *** chlong has quit IRC | 14:07 | |
| *** ducttape_ has quit IRC | 14:10 | |
| *** jaosorior_away has quit IRC | 14:10 | |
| *** chlong has joined #openstack-keystone | 14:20 | |
| mordred | efried: ok- I just sent a VERY long reponse | 14:21 |
| efried | mordred Looking forward to reading it. | 14:21 |
| *** rvba has quit IRC | 14:30 | |
| *** adrian_otto has joined #openstack-keystone | 14:31 | |
| *** adrian_otto has quit IRC | 14:44 | |
| *** ducttape_ has joined #openstack-keystone | 14:56 | |
| *** phalmos has joined #openstack-keystone | 14:57 | |
| *** voelzmo has quit IRC | 15:02 | |
| *** nle5223__ has joined #openstack-keystone | 15:03 | |
| *** ducttape_ has quit IRC | 15:06 | |
| *** Aqsam has quit IRC | 15:07 | |
| *** rvba has joined #openstack-keystone | 15:13 | |
| *** rvba has quit IRC | 15:13 | |
| *** rvba has joined #openstack-keystone | 15:13 | |
| *** mkrai has joined #openstack-keystone | 15:15 | |
| mkrai | Hi I am facing issue in keystone while installing devstack | 15:16 |
| mkrai | http://paste.openstack.org/show/608341/ | 15:16 |
| mkrai | Can anyone help? | 15:16 |
| lbragstad | mkrai how old is your devstack clone? did you just pull it? | 15:16 |
| lbragstad | mkrai the devstack project recently switched keystone to uwsgi instead of apache | 15:17 |
| mkrai | latest | 15:17 |
| mkrai | I cloned it right now | 15:17 |
| mkrai | yes issue is related to that only | 15:17 |
| mkrai | Keystone service is not running | 15:17 |
| *** openstackgerrit has quit IRC | 15:17 | |
| mkrai | lbragstad: Is there any workaround? | 15:17 |
| lbragstad | mkrai can you check your keystone logs? | 15:18 |
| mkrai | sure | 15:18 |
| lbragstad | mkrai you should be able to find them in /var/log/keystone or somewhere around there | 15:18 |
| mkrai | lbragstad: http://paste.openstack.org/show/608342/ | 15:18 |
| mkrai | Is this log helpful? | 15:18 |
| lbragstad | mkrai i haven't seen that error specifically | 15:19 |
| lbragstad | mkrai this might be related - but the change looks correct to me https://github.com/openstack-dev/devstack/commit/6ed53156b6198e69d59d1cf3a3497e96f5b7a870 | 15:22 |
| lbragstad | mkrai you're not setting WSGI_MODE anywhere are you? | 15:23 |
| mkrai | No I am not | 15:23 |
| lbragstad | mkrai have you checked with the devstack folks in #openstack-qa? | 15:24 |
| mkrai | No not yet | 15:24 |
| mkrai | I should check with devstack or QA team? | 15:24 |
| lbragstad | mkrai that might be a good place to start, it certainly seems deployment related | 15:24 |
| *** ravelar has joined #openstack-keystone | 15:26 | |
| *** pcaruana has quit IRC | 15:28 | |
| *** ducttape_ has joined #openstack-keystone | 15:33 | |
| *** chlong has quit IRC | 15:34 | |
| *** ducttape_ has quit IRC | 15:45 | |
| *** gyee has joined #openstack-keystone | 15:52 | |
| *** ducttape_ has joined #openstack-keystone | 15:56 | |
| *** david-lyle has quit IRC | 16:00 | |
| *** gcb has joined #openstack-keystone | 16:02 | |
| gcb | lbragstad, I think we need more work from keystone side to make unit tests pass with oslo.config 4.0 | 16:04 |
| gcb | lbragstad, please check details in http://lists.openstack.org/pipermail/openstack-dev/2017-April/116051.html | 16:04 |
| gcb | lbragstad, hope keystone folks can help dig and fix them, I just tried the simple one | 16:05 |
| *** knasim-wrs has joined #openstack-keystone | 16:05 | |
| lbragstad | gcb you're specifically referencing https://review.openstack.org/#/c/455391/ ? | 16:06 |
| *** spzala has quit IRC | 16:06 | |
| *** spzala has joined #openstack-keystone | 16:06 | |
| *** knasim-wrs has quit IRC | 16:06 | |
| *** aojea has quit IRC | 16:06 | |
| gcb | lbragstad, that fixes most of failures , we still get failures in http://logs.openstack.org/11/459411/1/check/gate-cross-keystone-python27-ubuntu-xenial/8a6879b/testr_results.html.gz | 16:07 |
| *** aojea has joined #openstack-keystone | 16:07 | |
| *** prajeesh-wrs has joined #openstack-keystone | 16:07 | |
| lbragstad | gcb ok - checking | 16:07 |
| lbragstad | gcb seeing if i can reproduce locally | 16:07 |
| gcb | lbragstad, sure, that's related about the keystone domain knowledge, as you know I'm not keystone expert :-), just fix two in https://review.openstack.org/#/q/status:open+project:openstack/keystone+branch:master+topic:bug/1686921 | 16:09 |
| *** adrian_otto has joined #openstack-keystone | 16:09 | |
| lbragstad | gcb some of those actually look like oslo.db errors | 16:10 |
| *** spzala has quit IRC | 16:11 | |
| *** aojea has quit IRC | 16:11 | |
| gcb | lbragstad, yeah, the first one 'keystone.tests.unit.common.test_notifications.NotificationsTestCase' looks like about oslo.db, I can help if oslo.db has issue | 16:12 |
| gcb | lbragstad, I recorded failures and fixes in https://etherpad.openstack.org/p/keystone_enforce_type_issues | 16:13 |
| lbragstad | gcb still digging into it - but it might be from a type being enforced | 16:13 |
| gcb | lbragstad, that may be helpful to track what we have been doing | 16:13 |
| *** openstackgerrit has joined #openstack-keystone | 16:14 | |
| openstackgerrit | Merged openstack/keystone master: use '&' instead of '?' to connect parameters in url https://review.openstack.org/460826 | 16:14 |
| samueldmq | mkrai_: hi, I am around now | 16:15 |
| gcb | lbragstad, take it easy, we don't bump oslo.config to 4.0 before we fixed in consuming projects, just raise the failures, hope we can fix them together :-O | 16:16 |
| gcb | :-) | 16:16 |
| *** chlong has joined #openstack-keystone | 16:16 | |
| lbragstad | gcb yeah - that makes sense, I appreciate the help :) | 16:17 |
| prajeesh-wrs | https://thepasteb.in/p/8qhO107xN4DF0 | 16:18 |
| prajeesh-wrs | Hi , I have an issue with Keystone memory leak | 16:18 |
| prajeesh-wrs | I found something interesting while doing a quick load test of keystone / newton . | 16:18 |
| prajeesh-wrs | When I started the load test the memory usage for keystone processes (admin and public wsgi) went up – and it never came down | 16:18 |
| prajeesh-wrs | Also, found that many functions in resource/backends/sql.py are not closing the sessions once open . | 16:19 |
| prajeesh-wrs | Do we need to close the sessions explicitly ? Is that the reason for persistent high memory usage ? | 16:19 |
| prajeesh-wrs | 2017-04-28 14:17:21.001 653 ERROR keystone.common.wsgi [req-2208fddc-6801-4a9c-a6fd-22cfd310427d - - - - -] QueuePool limit of size 1 overflow 10 reached, connection timed out, timeout 30 | 16:19 |
| prajeesh-wrs | 2017-04-28 14:17:21.001 653 ERROR keystone.common.wsgi Traceback (most recent call last): | 16:19 |
| prajeesh-wrs | 2017-04-28 14:17:21.001 653 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/wsgi.py", line 225, in __call__ | 16:19 |
| prajeesh-wrs | 2017-04-28 14:17:21.001 653 ERROR keystone.common.wsgi result = method(req, **params) | 16:19 |
| prajeesh-wrs | 2017-04-28 14:17:21.001 653 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/auth/controllers.py", line 397, in authenticate_for_token | 16:19 |
| prajeesh-wrs | 2017-04-28 14:17:21.001 653 ERROR keystone.common.wsgi [req-2208fddc-6801-4a9c-a6fd-22cfd310427d - - - - -] QueuePool limit of size 1 overflow 10 reached, connection timed out, timeout 30 | 16:19 |
| lbragstad | prajeesh-wrs from what i can tell - most of the sql connections are in a context manager that should close them once it exits the context | 16:20 |
| prajeesh-wrs | is the error . I think we can do a work around to solve this error . However , it will not address the memory / session open issue | 16:20 |
| prajeesh-wrs | @lbragstad - Thanks . | 16:20 |
| openstackgerrit | ayoung proposed openstack/keystone master: Refactor is_admin https://review.openstack.org/387710 | 16:21 |
| prajeesh-wrs | What I'm noticing is , memroy usage is still high even after 3 hrs after I stopped the load testing | 16:21 |
| *** ducttape_ has quit IRC | 16:24 | |
| *** aojea has joined #openstack-keystone | 16:27 | |
| lbragstad | prajeesh-wrs do you have steps detailing what you did to reproduce the issue? | 16:27 |
| lbragstad | or a script? | 16:28 |
| *** ducttape_ has joined #openstack-keystone | 16:28 | |
| prajeesh-wrs | yeah ... in a python script - | 16:28 |
| lbragstad | prajeesh-wrs are you able to share it | 16:29 |
| lbragstad | ? | 16:29 |
| prajeesh-wrs | I will launch 50 threads , and each thread will hit keystone public endpoint to get the project list / user list - | 16:29 |
| prajeesh-wrs | sure . let me share it - | 16:29 |
| prajeesh-wrs | its a plain python program | 16:29 |
| prajeesh-wrs | we noticed this issue with a regular insallation - and able to reproduce it with this script | 16:30 |
| gcb | samueldmq, just replied your comments, feel free to update my patches if you have better solution :-) | 16:30 |
| prajeesh-wrs | https://thepasteb.in/p/76hElKORnp5FV | 16:31 |
| prajeesh-wrs | @lbragstad - Please see the code - | 16:32 |
| lbragstad | prajeesh-wrs will do | 16:32 |
| prajeesh-wrs | when execute it , change the IP - | 16:32 |
| lbragstad | prajeesh-wrs thanks for sharing | 16:32 |
| prajeesh-wrs | and monitor the memory usage | 16:32 |
| *** ducttape_ has quit IRC | 16:33 | |
| *** sjain has joined #openstack-keystone | 16:33 | |
| samueldmq | gcb: re-replied, I understand your motivation for those patches, and perhaps our code needs to be updated if oslo already does the checks for us | 16:35 |
| samueldmq | and yes, re-replied sounds odd | 16:36 |
| samueldmq | :-) | 16:36 |
| *** dave-mccowan has quit IRC | 16:36 | |
| gcb | samueldmq, yeah, just talked with lbragstad about the failures in http://logs.openstack.org/11/459411/1/check/gate-cross-keystone-python27-ubuntu-xenial/8a6879b/testr_results.html.gz | 16:38 |
| samueldmq | gcb: are those new failures? | 16:38 |
| samueldmq | I think we've been those tests for a while now | 16:38 |
| gcb | samueldmq, oslo.config 4.0 make set_override with enforce_type=True by default, test failed with oslo.config 4.0 | 16:39 |
| gcb | samueldmq, we have some invalid tests, now oslo.config 4.0 finds them :-), please see http://lists.openstack.org/pipermail/openstack-dev/2017-April/116051.html | 16:41 |
| gcb | for more details and background | 16:41 |
| samueldmq | gcb: interesting ... that'd have avoided us to misuse oslo configs | 16:42 |
| samueldmq | gcb: I can give a respin on those patches if you want, as you don't have keystone expertise as you stated in the review | 16:42 |
| gcb | samueldmq, yeah, we use some invalid config option value in keystone, just do it :-) | 16:43 |
| samueldmq | gcb: are the oslo versions capped? | 16:44 |
| samueldmq | gcb: in projects like keystone ? /me looks | 16:44 |
| samueldmq | gcb: if not, that may cause issues in current deployments which set that value to 0 (which means disabled) | 16:44 |
| samueldmq | if people update their oslo config version | 16:45 |
| *** tesseract has quit IRC | 16:45 | |
| gcb | samueldmq, look at https://review.openstack.org/#/c/459411/, we still can't bump oslo.config to 4.0 | 16:45 |
| samueldmq | gcb: until we make sure everything passes ... | 16:46 |
| samueldmq | but projects should be warned that somethign may break | 16:46 |
| samueldmq | if it's not tested enough :-) | 16:47 |
| gcb | samueldmq, it's all about tests, will be okay in runtime now, what we do is making test same value both test and runtime | 16:47 |
| gcb | samueldmq, it's a long story about enforce_type=True, just look at https://bugs.launchpad.net/oslo.config/+bug/1517839 | 16:48 |
| openstack | Launchpad bug 1517839 in tacker "Make CONF.set_override with parameter enforce_type=True by default" [Undecided,In progress] - Assigned to Ji.Wei (jiwei) | 16:48 |
| samueldmq | gcb: ++ I will have a look so I have a better understanding of it | 16:49 |
| samueldmq | but I got your point | 16:49 |
| *** david-lyle has joined #openstack-keystone | 16:50 | |
| gcb | cool | 16:51 |
| *** dave-mccowan has joined #openstack-keystone | 16:59 | |
| *** thorst has quit IRC | 17:01 | |
| *** prashkre has joined #openstack-keystone | 17:04 | |
| openstackgerrit | ayoung proposed openstack/keystone master: Refactor is_admin https://review.openstack.org/387710 | 17:05 |
| *** ducttape_ has joined #openstack-keystone | 17:06 | |
| *** harlowja has quit IRC | 17:06 | |
| lbragstad | prajeesh-wrs can you open a bug for that issue? | 17:06 |
| prajeesh-wrs | Ok . Sure . | 17:07 |
| lbragstad | prajeesh-wrs https://bugs.launchpad.net/keystone/+filebug | 17:07 |
| lbragstad | prajeesh-wrs i just don't want to lose the context of the problem | 17:07 |
| lbragstad | in irc | 17:07 |
| lbragstad | that will also get eyes on it from other folks, too | 17:07 |
| prajeesh-wrs | yeah.. understand . | 17:07 |
| openstackgerrit | Merged openstack/keystone master: Revise doc about python 3.4 https://review.openstack.org/460385 | 17:08 |
| *** aojea has quit IRC | 17:08 | |
| *** thorst has joined #openstack-keystone | 17:11 | |
| *** pnavarro has quit IRC | 17:16 | |
| prajeesh-wrs | @lbragstad - I added a bug - https://bugs.launchpad.net/keystone/+bug/1687073 | 17:22 |
| openstack | Launchpad bug 1687073 in OpenStack Identity (keystone) "Keystone Memory usage remains high " [Undecided,New] | 17:22 |
| openstackgerrit | ayoung proposed openstack/keystone master: Add is_admin_project check to policy https://review.openstack.org/257636 | 17:22 |
| openstackgerrit | ayoung proposed openstack/keystone master: Refactor is_admin https://review.openstack.org/387710 | 17:22 |
| *** gcb has quit IRC | 17:24 | |
| openstackgerrit | ayoung proposed openstack/keystone master: Fernet token formatter with explicit role https://review.openstack.org/310074 | 17:25 |
| *** nle5223__ has quit IRC | 17:35 | |
| *** Aqsa has joined #openstack-keystone | 17:35 | |
| *** aojea has joined #openstack-keystone | 17:37 | |
| *** sjain has quit IRC | 17:38 | |
| *** aojea_ has joined #openstack-keystone | 17:38 | |
| *** thorst has quit IRC | 17:41 | |
| openstackgerrit | Merged openstack/keystoneauth master: Uncomment warning-is-error for doc building https://review.openstack.org/459673 | 17:41 |
| *** aojea has quit IRC | 17:41 | |
| *** aojea_ has quit IRC | 17:43 | |
| openstackgerrit | Merged openstack/python-keystoneclient master: Remove unused log https://review.openstack.org/459607 | 17:59 |
| *** thorst has joined #openstack-keystone | 18:03 | |
| *** catintheroof has joined #openstack-keystone | 18:04 | |
| openstackgerrit | Merged openstack/keystone master: Add notes in inherit.inc https://review.openstack.org/459932 | 18:07 |
| *** catinthe_ has quit IRC | 18:08 | |
| openstackgerrit | Merged openstack/keystone master: Explicitly set 'builders' option https://review.openstack.org/457969 | 18:18 |
| openstackgerrit | Merged openstack/keystonemiddleware master: Bump the token deferral message from info to debug https://review.openstack.org/441549 | 18:21 |
| *** voelzmo has joined #openstack-keystone | 18:23 | |
| *** david-lyle_ has joined #openstack-keystone | 18:24 | |
| *** david-lyle has quit IRC | 18:26 | |
| *** harlowja has joined #openstack-keystone | 18:38 | |
| *** Aqsa has quit IRC | 18:39 | |
| *** odyssey4me has quit IRC | 18:44 | |
| *** evrardjp has quit IRC | 18:46 | |
| *** odyssey4me has joined #openstack-keystone | 18:46 | |
| *** evrardjp has joined #openstack-keystone | 18:47 | |
| openstackgerrit | ayoung proposed openstack/keystone-specs master: Tokens with subsets of roles https://review.openstack.org/186979 | 18:57 |
| openstackgerrit | ayoung proposed openstack/keystone-specs master: Tokens with subsets of roles https://review.openstack.org/186979 | 19:06 |
| *** david-lyle_ is now known as david-lyle | 19:09 | |
| ayoung | dstanek, https://review.openstack.org/#/c/387710/18 can you +2 that now, please? Changed the function name as you requested | 19:14 |
| *** ducttape_ has quit IRC | 19:20 | |
| openstackgerrit | ayoung proposed openstack/keystone master: Refactor Authorization: https://review.openstack.org/387161 | 19:26 |
| openstackgerrit | ayoung proposed openstack/keystone master: Refactor is_admin https://review.openstack.org/387710 | 19:26 |
| openstackgerrit | ayoung proposed openstack/keystone master: Add is_admin_project check to policy https://review.openstack.org/257636 | 19:30 |
| *** voelzmo has quit IRC | 19:33 | |
| *** ducttape_ has joined #openstack-keystone | 19:36 | |
| *** adrian_otto has quit IRC | 19:37 | |
| *** ducttape_ has quit IRC | 19:37 | |
| *** ducttape_ has joined #openstack-keystone | 19:37 | |
| *** Aqsa has joined #openstack-keystone | 19:42 | |
| mordred | ayoung: it's likely completely valid to auth without a project scope if the thing you're planning on doing is registering services or endpoints isn't it? | 19:46 |
| ayoung | mordred, I don't know. | 19:46 |
| ayoung | mordred, probably not | 19:46 |
| ayoung | mordred, that is an admin operation, and you should have to specify the admin project for it | 19:47 |
| ayoung | mordred, speaking of which, gues which Windmill I'm tilting at today>? | 19:47 |
| mordred | cool. fielding a issue from someone - they're getting a traceback due to an empty service catalog - but of course they get an empty service catalog because no project | 19:47 |
| mordred | ayoung: oh golly - there are so many fun ones... | 19:47 |
| ayoung | mordred, how many bugs do you know by number? THat have been printed on T-Shirts? THat can be played as a melody on a keyboard? | 19:48 |
| mordred | ayoung: wow. I don't know any bugs by number. I feel deficient now :) | 19:49 |
| *** thorst has quit IRC | 19:49 | |
| *** thorst has joined #openstack-keystone | 19:50 | |
| *** thorst_ has joined #openstack-keystone | 19:51 | |
| ayoung | https://www.youtube.com/watch?v=2h1CY-XCbic&feature=youtu.be | 19:52 |
| *** thorst has quit IRC | 19:54 | |
| *** thorst_ has quit IRC | 19:56 | |
| *** thorst has joined #openstack-keystone | 20:07 | |
| *** dave-mccowan has quit IRC | 20:10 | |
| *** raildo has quit IRC | 20:25 | |
| mordred | ayoung: ahhhhhh. have fun with that | 20:28 |
| ayoung | gagehugo, you! | 20:32 |
| ayoung | Change-Id: I035fe570972764b9c9342d1851654634d681ac5e | 20:33 |
| ayoung | People, please make sure you understand the changes you are making before you make them | 20:33 |
| ayoung | Bug: #1652012 | 20:33 |
| openstack | bug 1652012 in OpenStack Identity (keystone) "token model assumes a token is is_admin_project" [Low,Fix released] https://launchpad.net/bugs/1652012 - Assigned to Gage Hugo (gagehugo) | 20:33 |
| ayoung | That "Bug Fix" Made it harder/impossible to actually fix the real damn bug | 20:34 |
| breton | ayoung: revert it :) | 20:37 |
| ayoung | breton, I am so flipping angry right now | 20:38 |
| breton | ayoung: and leave a better comment so that we don't break it again :) | 20:38 |
| ayoung | breton, I did better than that | 20:38 |
| ayoung | I had a goddammn bug fix that went ignored excpet for nit picking meanwhile this kind of crap | 20:38 |
| breton | ayoung: what's the review number? | 20:39 |
| ayoung | Mine? | 20:40 |
| breton | ayoung: yes | 20:40 |
| ayoung | https://review.openstack.org/#/c/257636/ and the two it depends on | 20:40 |
| ayoung | yak shedding | 20:40 |
| edmondsw_ | ayoung just to polish off your day, you're going to love my response to https://review.openstack.org/#/c/257636/ | 20:43 |
| edmondsw_ | sorry | 20:43 |
| edmondsw_ | breton, I'm not sure you understood what he was saying in https://bugs.launchpad.net/keystone/+bug/1684994 | 20:44 |
| openstack | Launchpad bug 1684994 in OpenStack Identity (keystone) "POST v3/auth/tokens API is returning unexpected 500 error when ldap credentials are incorrect" [Undecided,Invalid] | 20:44 |
| breton | edmondsw_: what is he saying? | 20:44 |
| edmondsw_ | breton we're giving 500 for invalid creds, but we're giving 504 for the LDAP timeout. We should give 500 for both | 20:44 |
| edmondsw_ | and the message for both should say that we couldn't reach the identity repo, without mentioning LDAP or why we couldn't reach it | 20:45 |
| edmondsw_ | as it is today, we leak that this is an LDAP config if there is an LDAP timeout | 20:45 |
| ayoung | edmondsw_, you are lucky you are not within arms reach | 20:46 |
| edmondsw_ | ayoung... I did say sorry... | 20:46 |
| ayoung | edmondsw_, you are so wrong | 20:46 |
| edmondsw_ | ayoung I thought my logic was pretty ironclad | 20:46 |
| ayoung | there is nothing in Keystone that actually requires Admin on project | 20:46 |
| ayoung | Now, I can understand why you think there sjhould be | 20:46 |
| ayoung | that would actually make sense | 20:46 |
| edmondsw_ | if you agree to that much, then you should agree overall | 20:47 |
| ayoung | but it was not how things were implemented in the past. And this is a don't break anything | 20:47 |
| edmondsw_ | don't put in a nasty hack that will make that harder in future | 20:47 |
| edmondsw_ | and this is a nasty hack | 20:47 |
| ayoung | IIRC is was your nasty hack in the first place | 20:47 |
| edmondsw_ | good luck to folks trying to figure out where that is / how things work | 20:47 |
| ayoung | Keystone is a nasty hack | 20:47 |
| ayoung | edmondsw_, that is why I was so pissed yesterday | 20:47 |
| ayoung | hard coding this is just not the right approach | 20:48 |
| ayoung | and I recall you saying it wouldn't derail this effort | 20:48 |
| edmondsw_ | again i don't think you understand what has been hardcoded (much less than you think) | 20:48 |
| ayoung | so flipping angry right now | 20:48 |
| edmondsw_ | they coded defaults... they are still overrideable | 20:48 |
| ayoung | edmondsw_, the rules from the default policy.json for role checking were hard coded | 20:48 |
| ayoung | edmondsw_, are you going to do the work to fix this? I recall you signed up to do the Nova work, which is still malingereing | 20:49 |
| edmondsw_ | ayoung I said I would try to help on that, and sorry that I got pulled off and haven't had time to do much... I did some work on it, but all I've had time for lately on that front is talking to johnthetubaguy about how to fix it and hoping he will | 20:50 |
| ayoung | Feel free to rewrite the patch, once it is working | 20:50 |
| edmondsw_ | if I can get back to it I will | 20:50 |
| ayoung | I don't have time for this either. Keystone is no longer my full time job | 20:50 |
| ayoung | Lead, follow, or get out of the way. | 20:50 |
| edmondsw_ | ayoung is this how we operate now... we accept nasty hacks and hope that someone will do it right later, rather than fixing it right to begin with? | 20:50 |
| edmondsw_ | why do we even have a review process, then? | 20:50 |
| ayoung | edmondsw_, this is the right fix | 20:51 |
| ayoung | this does not change the existing policy | 20:51 |
| ayoung | if you want to do a follow on that does better default policy, go for it | 20:52 |
| ayoung | but don't hold up the bug fix | 20:52 |
| openstackgerrit | ayoung proposed openstack/keystone master: Add is_admin_project check to policy https://review.openstack.org/257636 | 20:52 |
| breton | edmondsw_: i see now. LDAPServerConnectionError should be inherited from UnexpectedError, not from Error | 20:52 |
| ayoung | edmondsw_, If it makes you happy, I will add a follow on commit that will allow a non-global admin user to perform assign-role-to-user-on-project | 20:53 |
| ayoung | that is the only case where that should be loosened up that I am aware of. Do you know of any others? | 20:53 |
| edmondsw_ | breton yeah, and a) stop explicitly coding 504 and b) stop mentioning LDAP in its message | 20:54 |
| breton | edmondsw_: we can still mention LDAP, it will be displayed only in insecure_debug afaik | 20:54 |
| edmondsw_ | and once those changes are made, it could well be used for ldap.INVALID_CREDENTIALS as well as where it's already used | 20:54 |
| edmondsw_ | breton I think that message actually comes back on the API response | 20:54 |
| edmondsw_ | breton so not just in logs | 20:55 |
| breton | edmondsw_: ok, i am going to file a new bugreport now | 20:55 |
| breton | it is also debug_message_format vs message_format | 20:55 |
| ayoung | and edmondsw_ I want that t-shirt back | 20:57 |
| edmondsw_ | ayoung lots of things that need to work for folks who aren't is_admin_project:True are using rule:admin_required | 20:58 |
| edmondsw_ | list_users, get_group, etc. | 20:58 |
| edmondsw_ | ayoung those are domain-scoped, not global-scoped | 20:59 |
| ayoung | edmondsw_, Nope | 21:00 |
| edmondsw_ | ? | 21:00 |
| ayoung | That is cloudsample thinking, but not how the policy.json default worked | 21:00 |
| ayoung | edmondsw_, you are talking about writing new default policy | 21:00 |
| ayoung | not a bad idea | 21:00 |
| ayoung | but way beyond the scope of this patch | 21:01 |
| edmondsw_ | ayoung your hack doesn't just affect folks with a single domain | 21:01 |
| ayoung | this is to make it possible | 21:01 |
| edmondsw_ | ayoung what you're proposing will break anyone with multiple domains, and since it's hardcoded they have no recourse | 21:01 |
| ayoung | edmondsw_, it breaks no one | 21:01 |
| dstanek | ayoung: sure i can take a look | 21:02 |
| ayoung | it is not enabled by default | 21:02 |
| edmondsw_ | I agree with hardcoding, but you have to do it in a way that doesn't break multi-cloud cases | 21:02 |
| breton | so tell me, how does ayoung's fix interfere with https://review.openstack.org/#/c/438035/ ? | 21:02 |
| ayoung | edmondsw_, that is a follow on patch | 21:02 |
| ayoung | please feel free to write it | 21:02 |
| ayoung | breton, OK here is how that happened | 21:02 |
| edmondsw_ | ayoung but you want people to use it, right? So the "not on by default" argument doesn't hold. We need a solution that works for folks that do what we want them to do, which is turn this on | 21:02 |
| ayoung | henry was looking at the code and commented on it. I explained to him the rationale, and he agree, but opened a bug to track the issue. Then, someone saw it as low hanging fruit and fixed the bug without understand what they were doing | 21:03 |
| ayoung | edmondsw_, as I said, feel free to write a follow on patch | 21:03 |
| ayoung | meanwhile, there is no reason to give domain admins any more leeway than project admins | 21:04 |
| ayoung | the problem was one of scope in the past | 21:04 |
| ayoung | IE scope was ignored | 21:04 |
| *** Aqsa has quit IRC | 21:05 | |
| ayoung | domains were a mistake anyway, but saying we should have better default policy in no way changes the scope of what this patch should do. It is as minimal as possible, and shouild have gone in in January. | 21:05 |
| breton | ok, wait, so this happened. What consesquences did this change from True to False have? | 21:06 |
| edmondsw_ | breton ayoung is right about https://review.openstack.org/#/c/438035/ being premature... we need to do that someday, but we can't do it today | 21:06 |
| ayoung | instead of -1 my reviews on other project please just fix them https://review.openstack.org/#/c/384655/ | 21:06 |
| edmondsw_ | breton is_admin_project has to default to False if an admin_project isn't configured for backward compatibility, so that we can add checks for is_admin_project:True and not have them block folks that haven't configured an admin_project | 21:07 |
| *** thorst_ has joined #openstack-keystone | 21:08 | |
| edmondsw_ | gagehugo ^ | 21:08 |
| gagehugo | edmondsw_ thanks | 21:08 |
| gagehugo | my chat notifications are borked | 21:09 |
| *** prashkre has quit IRC | 21:09 | |
| breton | well, all of that should have gone to the note and bugreport | 21:09 |
| breton | this is what we get for being lazy and not putting all information where it needs to be | 21:09 |
| *** thorst has quit IRC | 21:09 | |
| breton | lets revert it | 21:09 |
| edmondsw_ | I will heartily agree that keystone code is not commented well enough | 21:10 |
| breton | (also, for not reviewing -- that patch was there for 2+ weeks) | 21:10 |
| dstanek | edmondsw_: in the code? | 21:11 |
| gagehugo | ayoung if that change was made too early, please do revert it. But imo it does seem dangerous to leave it defaulting to true | 21:11 |
| ayoung | gagehugo, you think? | 21:12 |
| edmondsw_ | dstanek which topic are you referring to? | 21:12 |
| ayoung | gagehugo, you do realize that it is implicitly DEFAULTED TO TRUE RIGHT NOW! | 21:12 |
| ayoung | bug 968696 | 21:12 |
| openstack | bug 968696 in OpenStack Identity (keystone) ""admin"-ness not properly scoped" [High,In progress] https://launchpad.net/bugs/968696 - Assigned to Adam Young (ayoung) | 21:12 |
| edmondsw_ | gagehugo nobody likes having that True. There's just no way around it at the moment | 21:12 |
| ayoung | gagehugo, so, this is just codifying the existing behavior so we can fix it | 21:12 |
| ayoung | and gagehugo I don't fault you | 21:12 |
| *** thorst_ has quit IRC | 21:12 | |
| ayoung | I fault the people that +2ed your review and should have known better | 21:13 |
| ayoung | Especially Henry Nash | 21:13 |
| ayoung | Instead of bike shedding, can we put some effort into actually solving the problems around Keystone? | 21:13 |
| gagehugo | ayoung yeah it's a mess, but I didn't think that would add to the problem | 21:14 |
| *** ayoung is now known as ayoung-ragequit | 21:14 | |
| dstanek | edmondsw_: comments | 21:14 |
| *** catintheroof has quit IRC | 21:14 | |
| ayoung-ragequit | gagehugo, UI hereby commission you to write new default policy to edmondsw_ standards on top of my last patch | 21:14 |
| gagehugo | ayoung I will take a look, I am behind though on the progress you've made though | 21:16 |
| *** harlowja has quit IRC | 21:17 | |
| *** harlowja has joined #openstack-keystone | 21:17 | |
| *** aojea has joined #openstack-keystone | 21:21 | |
| edmondsw_ | breton you said you were opening a new bug for the LDAP thing... number? | 21:22 |
| *** thorst has joined #openstack-keystone | 21:29 | |
| edmondsw_ | breton nm, see it | 21:30 |
| openstackgerrit | Merged openstack/keystone master: Correct oauth create_request_token documentation https://review.openstack.org/459114 | 21:32 |
| *** thorst has quit IRC | 21:33 | |
| *** aojea has quit IRC | 21:35 | |
| *** harlowja has quit IRC | 21:36 | |
| *** chlong has quit IRC | 21:39 | |
| *** ayoung-ragequit is now known as ayoung | 21:49 | |
| ayoung | edmondsw_, OK, I think I see the disconnect. You were thinking that the fix there is the new default, and it is not. It is another interim step. Here is what needs to happen | 21:50 |
| ayoung | 1. get this kind of fix into the proejcts: | 21:50 |
| ayoung | 2. enable an admin project in devstack | 21:50 |
| ayoung | meanwhile keeping Tempest running | 21:50 |
| ayoung | then, we can tighten up the rules, over time, so long as the tests keep running | 21:51 |
| *** cmurphy has quit IRC | 21:59 | |
| *** cmurphy has joined #openstack-keystone | 22:04 | |
| *** edmondsw_ has quit IRC | 22:06 | |
| *** thorst has joined #openstack-keystone | 22:11 | |
| *** thorst has quit IRC | 22:12 | |
| *** catintheroof has joined #openstack-keystone | 22:17 | |
| *** adrian_otto has joined #openstack-keystone | 22:28 | |
| *** phalmos_ has joined #openstack-keystone | 22:36 | |
| *** phalmos has quit IRC | 22:39 | |
| *** thorst has joined #openstack-keystone | 22:43 | |
| *** adrian_otto has quit IRC | 22:46 | |
| *** adrian_otto has joined #openstack-keystone | 22:49 | |
| morgan | ugh. | 22:51 |
| morgan | i think i found a security flaw =/ | 22:51 |
| * morgan grumps. | 22:51 | |
| morgan | nope | 22:54 |
| morgan | nvm | 22:54 |
| *** thorst has quit IRC | 22:59 | |
| *** lamt has quit IRC | 23:01 | |
| *** lamt has joined #openstack-keystone | 23:02 | |
| *** lamt has quit IRC | 23:08 | |
| *** adrian_otto has quit IRC | 23:09 | |
| *** catintheroof has quit IRC | 23:11 | |
| *** adrian_otto has joined #openstack-keystone | 23:11 | |
| *** adrian_otto has quit IRC | 23:15 | |
| *** harlowja has joined #openstack-keystone | 23:30 | |
| *** edmondsw has joined #openstack-keystone | 23:32 | |
| *** edmondsw has quit IRC | 23:37 | |
| *** harlowja has quit IRC | 23:55 | |
| *** thorst has joined #openstack-keystone | 23:56 | |
| *** harlowja has joined #openstack-keystone | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!