Monday, 2016-09-26

*** sdake has quit IRC00:22
*** spzala has joined #openstack-keystone00:24
*** roxanaghe has joined #openstack-keystone00:28
*** sdake has joined #openstack-keystone00:32
*** roxanaghe has quit IRC00:33
*** tqtran has joined #openstack-keystone00:58
*** davechen has joined #openstack-keystone00:58
*** tqtran has quit IRC01:02
*** ngupta has joined #openstack-keystone01:05
*** ngupta has quit IRC01:06
*** ngupta has joined #openstack-keystone01:07
*** ngupta_ has joined #openstack-keystone01:08
*** ngupta has quit IRC01:11
*** EinstCrazy has joined #openstack-keystone01:20
*** EinstCra_ has joined #openstack-keystone01:22
*** ngupta_ has quit IRC01:22
*** EinstCrazy has quit IRC01:25
*** ngupta has joined #openstack-keystone01:26
openstackgerritHa Van Tu proposed openstack/keystone: Refactor Keystone admin-tokens and admin-users v2  https://review.openstack.org/36988301:26
*** roxanaghe has joined #openstack-keystone01:30
*** roxanaghe has quit IRC01:34
*** ngupta_ has joined #openstack-keystone01:45
*** ngupta has quit IRC01:47
*** ngupta has joined #openstack-keystone02:01
*** ngupta_ has quit IRC02:02
*** ngupta_ has joined #openstack-keystone02:17
*** ngupta has quit IRC02:17
*** ngupta_ has quit IRC02:22
*** stevemar has quit IRC02:30
*** stevemar has joined #openstack-keystone02:31
*** spzala has quit IRC02:32
*** lamt has joined #openstack-keystone02:33
*** ChanServ sets mode: +o stevemar02:35
*** sdake has quit IRC02:35
openstackgerritDave Chen proposed openstack/keystone: Deprecate endpoint filter catalog and make it as default  https://review.openstack.org/37593102:39
stevemardavechen: thanks for that ^ :)02:45
stevemarsamueldmq: thanks for the remarks, and thanks for running :)02:45
*** GB21 has quit IRC02:56
*** david-lyle has quit IRC03:04
* davechen looking into the failure testcases...03:06
davechenstevemar: happy to work on that. :)03:07
*** roxanaghe has joined #openstack-keystone03:15
*** roxanaghe has quit IRC03:20
*** brad[]` has quit IRC03:38
*** markvoelker has joined #openstack-keystone03:48
*** markvoelker has quit IRC03:53
*** tonytan_brb has joined #openstack-keystone04:17
*** tonytan4ever has quit IRC04:19
openstackgerritOpenStack Proposal Bot proposed openstack/keystoneauth: Updated from global requirements  https://review.openstack.org/37611504:21
openstackgerritOpenStack Proposal Bot proposed openstack/keystonemiddleware: Updated from global requirements  https://review.openstack.org/37611604:21
openstackgerritJamie Lennox proposed openstack/keystonemiddleware: Extract token fetching into method  https://review.openstack.org/37612304:23
openstackgerritOpenStack Proposal Bot proposed openstack/oslo.policy: Updated from global requirements  https://review.openstack.org/37612904:26
openstackgerritOpenStack Proposal Bot proposed openstack/python-keystoneclient: Updated from global requirements  https://review.openstack.org/37613704:26
openstackgerritOpenStack Proposal Bot proposed openstack/python-keystoneclient-kerberos: Updated from global requirements  https://review.openstack.org/37368604:26
*** roxanaghe has joined #openstack-keystone04:31
*** spzala has joined #openstack-keystone04:32
*** roxanaghe has quit IRC04:35
*** spzala has quit IRC04:37
*** sdake has joined #openstack-keystone04:41
*** sdake has quit IRC04:45
*** markvoelker has joined #openstack-keystone04:49
*** markvoelker has quit IRC04:53
*** GB21 has joined #openstack-keystone04:53
*** tqtran has joined #openstack-keystone04:59
openstackgerritQiming Teng proposed openstack/keystone: Tweak api-ref doc for v3 roles status codes  https://review.openstack.org/36779405:00
*** tqtran has quit IRC05:04
*** jidar_ has joined #openstack-keystone05:13
*** crinkle_ has joined #openstack-keystone05:13
*** ctracey_ has joined #openstack-keystone05:13
*** auggy_ has joined #openstack-keystone05:13
*** adriant has quit IRC05:14
*** DuncanT_ has joined #openstack-keystone05:15
*** ktychkova_ has joined #openstack-keystone05:15
*** serverascode_ has joined #openstack-keystone05:15
*** X-Istence has joined #openstack-keystone05:16
*** Guest66676 has joined #openstack-keystone05:16
*** woodburn1 has joined #openstack-keystone05:16
*** pleia2_ has joined #openstack-keystone05:16
*** chrome0_ has joined #openstack-keystone05:19
*** dgonzalez_ has joined #openstack-keystone05:19
openstackgerritQiming Teng proposed openstack/keystone: Tweak api-ref doc for v3 roles  https://review.openstack.org/37616905:19
*** jlk` has joined #openstack-keystone05:19
*** jlvillal_ has joined #openstack-keystone05:19
*** amoralej_ has joined #openstack-keystone05:20
*** clayton_ has joined #openstack-keystone05:20
*** timburke_ has joined #openstack-keystone05:20
*** sileht has quit IRC05:20
*** woodburn has quit IRC05:21
*** serverascode has quit IRC05:21
*** tsufiev has quit IRC05:21
*** mrhillsman has quit IRC05:21
*** ctracey has quit IRC05:21
*** dmellado has quit IRC05:21
*** ayoung has quit IRC05:21
*** chrome0 has quit IRC05:21
*** jlvillal has quit IRC05:21
*** timburke has quit IRC05:21
*** jlk has quit IRC05:21
*** arunkant has quit IRC05:21
*** Guest29913 has quit IRC05:21
*** vkmc has quit IRC05:21
*** clayton has quit IRC05:21
*** auggy has quit IRC05:21
*** mnaser has quit IRC05:21
*** DuncanT has quit IRC05:21
*** amoralej has quit IRC05:21
*** iurygregory_ has quit IRC05:21
*** kragniz has quit IRC05:21
*** hogepodge has quit IRC05:21
*** jidar has quit IRC05:21
*** crinkle has quit IRC05:21
*** dgonzalez has quit IRC05:21
*** timss has quit IRC05:21
*** pleia2 has quit IRC05:21
*** ktychkova has quit IRC05:21
*** rha has quit IRC05:21
*** Guest66666 has quit IRC05:21
*** jamiec has quit IRC05:21
*** x58 has quit IRC05:21
*** SpamapS has quit IRC05:21
*** dgonzalez_ is now known as dgonzalez05:21
*** tonytan_brb has quit IRC05:21
*** clayton_ is now known as clayton05:21
*** rha_ has joined #openstack-keystone05:21
*** dmellado has joined #openstack-keystone05:22
*** tsufiev has joined #openstack-keystone05:22
*** jlvillal_ is now known as jlvillal05:23
*** vkmc has joined #openstack-keystone05:23
*** jlvillal is now known as Guest9712805:23
*** ctracey_ is now known as ctracey05:25
*** auggy_ is now known as auggy05:25
*** mnaser has joined #openstack-keystone05:26
*** ayoung has joined #openstack-keystone05:27
*** ChanServ sets mode: +v ayoung05:27
*** arunkant has joined #openstack-keystone05:27
*** mrhillsman has joined #openstack-keystone05:27
*** timss has joined #openstack-keystone05:27
*** kragniz has joined #openstack-keystone05:28
*** Guest29913 has joined #openstack-keystone05:28
*** X-Istence is now known as x5805:28
*** iurygregory_ has joined #openstack-keystone05:28
*** SpamapS has joined #openstack-keystone05:28
*** sileht has joined #openstack-keystone05:29
*** rcernin has joined #openstack-keystone05:30
*** serverascode_ is now known as serverascode05:31
*** DuncanT_ is now known as DuncanT05:36
bretonmorning, keystone05:36
openstackgerritMerged openstack/python-keystoneclient: Updated from global requirements  https://review.openstack.org/37613705:50
*** haplo37_ has quit IRC05:56
*** haplo37_ has joined #openstack-keystone05:58
openstackgerritMerged openstack/keystoneauth: Updated from global requirements  https://review.openstack.org/37611506:07
openstackgerritMerged openstack/oslo.policy: Updated from global requirements  https://review.openstack.org/37612906:13
openstackgerritHieu LE proposed openstack/keystone: Remove default=None when set value in config  https://review.openstack.org/37619306:13
openstackgerritMerged openstack/keystonemiddleware: Updated from global requirements  https://review.openstack.org/37611606:23
*** jidar_ is now known as jidar06:24
darrencany keystone gurus around?06:40
*** pcaruana has joined #openstack-keystone06:45
openstackgerritJamie Lennox proposed openstack/keystonemiddleware: Raise NotImplementedError instead of NotImplemented  https://review.openstack.org/36550906:49
openstackgerritChangBo Guo(gcb) proposed openstack/keystonemiddleware: Use method constant_time_compare from oslo.utils  https://review.openstack.org/37623507:11
*** rha_ is now known as rha07:16
*** rha has joined #openstack-keystone07:16
*** roxanaghe has joined #openstack-keystone07:19
*** roxanaghe has quit IRC07:24
*** jaosorior has joined #openstack-keystone07:29
*** amoralej_ is now known as amoralej07:43
*** zzzeek has quit IRC08:00
*** zzzeek has joined #openstack-keystone08:01
*** Guest29913 is now known as amakarov08:08
*** tonytan4ever has joined #openstack-keystone08:22
*** tonytan4ever has quit IRC08:27
*** asettle has joined #openstack-keystone08:30
*** TonyXu has quit IRC08:32
*** TonyXu has joined #openstack-keystone08:33
*** jlwhite has quit IRC08:36
openstackgerritCao Xuan Hoang proposed openstack/keystone: Using assertIsNone() instead of assertIs(None)  https://review.openstack.org/37629508:36
*** jlwhite has joined #openstack-keystone08:36
openstackgerritAlexander Makarov proposed openstack/keystone: Verbose 401/403 debug responses  https://review.openstack.org/37243308:40
*** ChanServ sets mode: +v henrynash08:50
openstackgerritCao Xuan Hoang proposed openstack/keystone: Use assertEqual() instead of assertDictEqual()  https://review.openstack.org/37630708:54
openstackgerritMerged openstack/keystonemiddleware: Raise NotImplementedError instead of NotImplemented  https://review.openstack.org/36550908:55
*** acoles_ is now known as acoles09:01
*** acoles has left #openstack-keystone09:01
*** code-R has joined #openstack-keystone09:04
*** code-R_ has joined #openstack-keystone09:07
*** code-R has quit IRC09:10
*** jistr has quit IRC09:20
*** pjm6_ is now known as pjm609:36
*** andreykurilin_ has left #openstack-keystone09:36
*** flaper87 has joined #openstack-keystone09:39
*** flaper87 has quit IRC09:39
*** flaper87 has joined #openstack-keystone09:39
*** daemontool has joined #openstack-keystone09:54
*** haplo37_ has quit IRC09:55
*** haplo37_ has joined #openstack-keystone09:57
*** sdake has joined #openstack-keystone10:06
*** EinstCra_ has quit IRC10:16
*** code-R_ has quit IRC10:18
*** code-R has joined #openstack-keystone10:32
*** rodrigods has quit IRC10:43
*** rodrigods has joined #openstack-keystone10:43
*** dikonoor has joined #openstack-keystone10:45
dikonoorhenrynash:hi, are you around?10:46
*** roxanaghe has joined #openstack-keystone10:48
*** nicolasbock has joined #openstack-keystone10:50
dikonoordolphm: bknudson: anyone around?10:50
*** roxanaghe has quit IRC10:52
*** code-R has quit IRC10:54
*** code-R has joined #openstack-keystone10:54
*** haplo37_ has quit IRC10:56
bretondikonoor: what stops you from just asking the question? maybe someone else will know the answer10:59
*** haplo37_ has joined #openstack-keystone10:59
dikonoorbreton: Thanks for responding :) https://bugs.launchpad.net/keystone/+bug/162769611:00
openstackLaunchpad bug 1627696 in OpenStack Identity (keystone) "keystonemiddleware throws OSError " [Undecided,New]11:00
dikonoorI keep running into the problem mentioned in that bug (which I just opened)..11:00
*** tqtran has joined #openstack-keystone11:01
*** sdake_ has joined #openstack-keystone11:04
*** sdake has quit IRC11:04
*** nicolasbock has quit IRC11:04
*** tqtran has quit IRC11:05
*** davechen has quit IRC11:19
*** nicolasbock has joined #openstack-keystone11:27
openstackgerritMerged openstack/python-keystoneclient: Correct output for Implied Roles  https://review.openstack.org/36849811:37
*** amoralej is now known as amoralej|lunch11:43
*** sdake_ has quit IRC11:43
*** artmr has joined #openstack-keystone11:53
*** tonytan4ever has joined #openstack-keystone11:54
*** raildo has joined #openstack-keystone11:58
*** tonytan4ever has quit IRC11:58
*** asettle has quit IRC12:03
*** GB21 has quit IRC12:06
*** iurygregory_ is now known as iurygregory12:11
*** asettle has joined #openstack-keystone12:14
*** GB21 has joined #openstack-keystone12:19
*** markvoelker has joined #openstack-keystone12:20
*** code-R has quit IRC12:21
*** lamt has quit IRC12:27
*** asettle has quit IRC12:31
*** asettle has joined #openstack-keystone12:32
*** pnavarro has joined #openstack-keystone12:35
*** evrardjp has quit IRC12:36
*** evrardjp has joined #openstack-keystone12:36
*** roxanaghe has joined #openstack-keystone12:36
*** daemontool has quit IRC12:38
*** roxanaghe has quit IRC12:41
*** markvoelker has quit IRC12:41
*** edmondsw has joined #openstack-keystone12:43
*** asettle has quit IRC12:43
*** asettle has joined #openstack-keystone12:43
*** daemontool has joined #openstack-keystone12:44
*** david-lyle has joined #openstack-keystone12:56
*** markvoelker has joined #openstack-keystone12:57
*** sdake has joined #openstack-keystone12:57
*** amoralej|lunch is now known as amoralej12:58
*** andrewbogott has quit IRC13:01
*** andrewbogott has joined #openstack-keystone13:01
*** daemontool has quit IRC13:01
*** sdake_ has joined #openstack-keystone13:03
openstackgerritMerged openstack/oslo.policy: Doc: declare YAML/JSON support  https://review.openstack.org/37463213:03
*** daemontool has joined #openstack-keystone13:03
*** sdake has quit IRC13:06
openstackgerritMerged openstack/keystone: Add domain check in domain-specific role implication  https://review.openstack.org/37446313:13
*** GB21 has quit IRC13:14
*** ngupta has joined #openstack-keystone13:14
*** code-R has joined #openstack-keystone13:16
*** code-R_ has joined #openstack-keystone13:18
*** jaosorior has quit IRC13:19
*** jaosorior has joined #openstack-keystone13:19
*** code-R has quit IRC13:21
*** LamT_ has joined #openstack-keystone13:23
*** lamt has joined #openstack-keystone13:26
dikonooranyone has any clue on why https://bugs.launchpad.net/keystone/+bug/1627696 could be happening ?13:26
openstackLaunchpad bug 1627696 in OpenStack Identity (keystone) "keystonemiddleware throws OSError " [Undecided,New]13:26
*** GB21 has joined #openstack-keystone13:27
*** Guest97128 is now known as jlvillal13:36
*** BrAsS_mOnKeY is now known as g2`13:38
*** GB21 has quit IRC13:44
*** jlk` is now known as jlk13:44
*** jlk has quit IRC13:44
*** jlk has joined #openstack-keystone13:44
*** code-R_ has quit IRC13:45
*** code-R has joined #openstack-keystone13:46
dstanek_dikonoor: just looking at the stack trace it appears there is something wrong with creating/using the temp signing directory13:46
*** brad[] has joined #openstack-keystone13:47
*** sdake has joined #openstack-keystone13:47
*** woodster_ has joined #openstack-keystone13:48
openstackgerritMerged openstack/keystone: Remove default=None when set value in config  https://review.openstack.org/37619313:49
dikonoordstanek_: yeah.. we get into this flow of verify_signing_dir() only when there's an exception inside https://github.com/openstack/keystonemiddleware/blob/stable/newton/keystonemiddleware/auth_token/_signing_dir.py#L4513:49
*** sdake_ has quit IRC13:50
dikonoordstanek_:but when I put a debugger on, it never gets into ththe except flow:13:50
dikonoorhttps://github.com/openstack/keystonemiddleware/blob/stable/newton/keystonemiddleware/auth_token/_signing_dir.py#L5313:50
*** sdake_ has joined #openstack-keystone13:51
dikonoordstanek_: I also find it strange that the /tmp/ directory does not have any keystone-signing-* created by neutron and glance even in cases where everything works13:52
*** sdake has quit IRC13:54
*** tonytan4ever has joined #openstack-keystone13:55
*** ravelar has joined #openstack-keystone13:55
dstanek_dikonoor: does the user running keystone have the ability to create directories there?13:56
dikonoordstanek_: The file permissions of the /tmp/ directory is such that anyone can create directories inside it..I see keystone-signing-* directories inside temp for nova, swift, cinder, ceilometer but not for neutron and glance..13:58
*** tonytan4ever has quit IRC13:59
*** tonytan4ever has joined #openstack-keystone13:59
openstackgerritRon De Rose proposed openstack/keystone: Add indexes to the revocation_event table  https://review.openstack.org/37652314:00
ayoungdikonoor,  _verify_signing_dir   you using PKI tokens?14:01
ayoungPLease don't.14:01
*** GB21 has joined #openstack-keystone14:01
*** ravelar has quit IRC14:02
*** ravelar has joined #openstack-keystone14:02
dikonoorayoung: No, using fernet tokens14:04
ayoungdikonoor, hmmm, we need to kill the signing-directory code.  Is it triggered by a config option?  Maybe not...14:04
dikonoorayoung: I am using the default config ..no specific changes made14:05
ayoungdikonoor, well here is what is happening14:05
openstackgerritLance Bragstad proposed openstack/keystone: Add release note for fernet tokens  https://review.openstack.org/37652614:05
ayoungdikonoor, bacjk in the days of PKI tokens, we needed a way to distributed certificates14:05
*** GB21 has quit IRC14:06
dikonoorayoung: ok14:06
ayoungat start up, middleware creates a temp dir, and fetches the certs on demand, puts them in the temp dir14:06
ayoungat startupi, you system is throwing an exception trying to create the temp dir14:06
dikonoorayoung : and these certificates are applicable only if the token provider is pki..Then why does the keystonemiddleware even get into this flow for fernet?14:07
ayoungdikonoor, because middleware does not know what kind of tokens it is going to receive14:08
ayoungdikonoor, this has not come up in a long time, not sure why you are seeing it.  What did you do differently?14:08
lbragstadstevemar i added a release note for making fernet default and made the grenade change dependent on it14:08
lbragstadstevemar otherwise - everything in that series to make fernet the default should be good to go14:09
lbragstadstevemar across all impacting projects14:09
dikonoorayoung: I don't seem to have done anything differently..14:09
ayoungdikonoor, I can't seem to get to launchpad right now to look further at the stacktrace14:10
dikonoorayoung: This is the first time I am hitting this..and I can't reproduce this on any other setup14:10
ayoungdikonoor, its a permissions or SELinux type issue, I guess14:10
dikonoorayoung : Well..I kind of feel that too..but I couldn't find any selinux errors14:10
dikonoorayoung: I have selinux in enforcing..So, next time I hit it I will put it in the permissive mode and check if that helps14:11
dikonoorayoung: But when this is hit, if I restart the neutron serivce, the problem disappears14:11
dikonoorayoung: very strangely.. and even when I hit this, I can't reproduce it from a python prompt if I try to use the same tempfil and os.makedirs api14:12
ayoungdikonoor, using the keystone user that the wsgi script runs as?14:13
dikonoorayoung : I hit this when I make a neutron rest api call for eg. v2.0/ports ..so it would be neutron user..or I did not follow your question14:14
dikonoorayoung : neutron user running neutron service invoking keystonemiddleware trying to create a tmp directory ?14:15
ayoungno, you are right...neutron14:15
ayoungbut the bug is filed against Keystone....14:15
ayoungshould be against keystonemiddleware14:15
dikonoorayoung:I can fix that :)14:16
*** spedione|AWAY is now known as spedione14:17
dikonoorayoung: Can't access LP now14:17
*** ngupta has quit IRC14:21
*** ngupta has joined #openstack-keystone14:21
*** roxanaghe has joined #openstack-keystone14:24
*** Guest92615 is now known as zeus14:26
*** zeus has quit IRC14:26
*** zeus has joined #openstack-keystone14:26
*** ngupta has quit IRC14:26
*** roxanaghe has quit IRC14:29
*** artmr has quit IRC14:36
openstackgerritStephen Finucane proposed openstack/oslo.policy: Add sphinx extension to build sample policy  https://review.openstack.org/37654414:38
*** dikonoor has quit IRC14:39
*** code-R has quit IRC14:40
ayoungdstanek_, doing a Keystone server build of tag 8.1.2  gets an error building cryptography (something with openssl).  It even happend when I used requirements.txt and test-requirements.txt from master.  So the binaries are not in the venv...they are the same.  But the python buld fails.  Puzzled...14:40
rderosebknudon: you around?14:42
ravelarbknudson: is there any additional changes you would like me to address for https://review.openstack.org/#/c/359371/14:42
rderose:)14:42
bknudsonrderose: I'm around.14:42
rderosebknudson: was going to ask you about 35937114:42
bknudsonI haven't had a chance to look at it again.14:43
rderosesaw your comments, were they in regards to clint's?14:43
bknudsonyes, they were mostly in regards to SpamapS feedback.14:43
rderosebknudson: okay, cool14:44
bknudsonthe point is that validation  is taking way too long14:44
bknudsonand scaling keystone isn't going to help it. even with 1 system it takes too long14:44
bknudsonso we need this or something else that improves single-system validation performance14:45
rderosebknudson: right14:45
bknudsonravelar: I would like to see unit tests similar to how  http://git.openstack.org/cgit/openstack/keystone/tree/keystone/tests/unit/identity/test_backend_sql.py does it14:46
dstanek_ayoung: what's the failure?14:46
bknudsonravelar: actually, more like http://git.openstack.org/cgit/openstack/keystone/tree/keystone/tests/unit/identity/backends/test_sql.py14:46
rderosebknudson: good point, doesn't matter if we add 10 servers, it still takes too long14:46
ravelarbknudson: will look into those to see if I need to add anything else. In the meantime, I have added 10 additional tests that call the new is_revoke directly by removing the old is_revoke and matches and then replacing the calls in the test with the new one14:47
bknudsonrderose: if there was some idea to speed things up via some other method then we can go with that.14:47
*** spzala has joined #openstack-keystone14:48
bknudsonbut I can't think of anything.14:48
rderoseme neither :)14:48
bknudsonI don't know how much it would help to have a simpler structure stored in memcache14:48
*** gagehugo has joined #openstack-keystone14:49
*** woodburn1 has quit IRC14:50
bknudsonor if there's some way to leverage last_fetch so that there isn't so much work done by the server14:50
*** edtubill has joined #openstack-keystone14:51
rderosebknudson: well, the sql approach should solve it, right? at least your and ravelar's numbers looks pretty good14:51
bknudsonrderose: I'm happy with the numbers I've seen on a dev system, but I'm still wary about scaling.14:51
bknudsonAt the scales I'm thinking we'll be running at, I think the sql approach will work.14:52
rderosebknudson: maybe "solve" it is not the right words :)  but definitely improves it.14:52
rderosebknudson: yeah, I would think it would scale14:53
bknudsonat high enough scales there's going to be way too much going to the database, I think; and no way to reduce it.14:53
rderosebknudson: other than indexing, we could also flush the revocation_events table more often if need be14:54
bknudsonmaybe database does a good job of caching, too.14:54
bknudsonI doubt that indexing will give us anything. Would have to measure that.14:54
rderosebknudson: I'm thinking that as well, indexing has only helped me when we're talking millions of rows14:54
bknudsonindex would have to give us better results than just scanning the table.14:55
rderosewrites would be slower14:55
dstanek_rderose: with that many rows is where i would usually turn off indexing14:55
bknudsonthe table isn't very wide, so lots of rows fit on a page.14:55
rderosedstanek: that would be where I re-index :)14:55
*** tonytan_brb has joined #openstack-keystone14:56
rderosebknudson dstanek: how often to we write compared to read on this table?14:56
rderoseis it like 1:10 read:write?14:57
bknudsonI would guess the ratio is around 20:1 ?14:57
*** daemontool_ has joined #openstack-keystone14:57
rderoseokay14:57
bknudson1:10 to 1:20 I would expect. I haven't measured it.14:57
bknudsonthere's caching in auth_token middleware.14:57
*** richm has joined #openstack-keystone14:57
bknudsonand tokens don't get re-used as much as we'd like.14:57
rderoseI see14:57
dstanek_bknudson: that's an understatement :-)14:57
*** hogepodge has joined #openstack-keystone14:58
bknudsondepending on applications in the cloud you could see 1:1000 or 1:2 .14:58
bknudsonyou might run applications that never revoke tokens.14:59
*** tonytan4ever has quit IRC14:59
bknudsonor you run applications that think it's better to revoke every token they get.14:59
*** daemontool has quit IRC14:59
*** sdake_ has quit IRC14:59
rderosebknudson: so it could be dramatically different between operators, hmm...14:59
bknudsondstanek_: at least with fernet the token table doesn't grow with every new token.15:00
bknudsonrderose: another way to have a shorter revocation event list would be to change keystone to validate project status vs the project table rather than have an event for it.15:01
*** Marcellin__ has joined #openstack-keystone15:01
rderoseravelar: does your other patch do this ^15:02
*** asettle has quit IRC15:02
bknudsonThat should be a separate patch15:02
bknudsonwe need to figure out how to handle the event API if we make that change.15:03
bknudsonI doubt anyone is using the event api.15:03
rderoseI see15:03
bknudsonbut we can't just change it. Maybe have a config option or something.15:03
dstanek_bknudson: do we have to support all of the existing events to deal with pki tokens?15:03
ravelarrderose yes for domains and projects but lbragstads patch is looking into doing away with projects and domains altogether15:04
bknudsondstanek_: what do you mean to deal with pki tokens?15:04
bknudsonpki tokens can go to the token table for revocations, don't need events15:04
dstanek_bknudson: do events like project changes need to still be captured?15:04
ravelarrderose in revocation events that is15:04
rderoseravelar: I see, thanks15:05
bknudsondstanek_: I think events like project changes need to be captured to support the revocation events API.15:05
bknudsonMaybe they could be in a separate table?15:05
*** ngupta has joined #openstack-keystone15:06
bknudsondstanek_: the only problem I see with getting rid of project events is that something might be using the revocation event API. Nothing we've written (auth_token) uses the event API.15:06
*** jlvillal has quit IRC15:07
ayoungdstanek_, :    error: command 'gcc' failed with exit status 115:07
ayoung  start of the trace is here15:07
ayoungbuild/temp.linux-x86_64-2.7/_openssl.c:697:6: error: conflicting types for ‘BIO_new_mem_buf’15:07
*** mvk has quit IRC15:07
dstanek_ayoung: have you tried checking out bindep to see if you have all the correct system deps?15:08
ayoungdstanek_, nope.  Never heard of it before15:09
ayoungdstanek_, I'm running Fedora 24.  I wouldn't be surprised if they were off, but the master branch builds15:10
dstanek_ayoung: http://docs.openstack.org/infra/bindep/readme.html15:10
ayoungits just the old one that fails...even with the master's requirements.  What else could affect the build process this way?15:10
dstanek_ayoung: i don't think that looks at strange verison issues though15:10
dstanek_ayoung: hmm... have you tried to delete any temporary build dirs and make sure there is no lingering object files?15:11
ayoungtox -r15:12
*** twouters_ has joined #openstack-keystone15:14
*** roxanaghe has joined #openstack-keystone15:15
*** sdake has joined #openstack-keystone15:15
ayoungdstanek_, is there anything other than .tox?15:17
*** woodburn has joined #openstack-keystone15:18
ayoungand it build when I switch to master, and did n't buld when I switch back.  weeeeeird.15:18
dstanek_ayoung: maybe ~/.pip? not really sure anymore. i thought everything has been moved to .tox thought15:20
ayoungdstanek_, just killed .tox and reran, same thing?15:20
dstanek_s/t$//15:20
ayounglet me see15:20
ayoungdstanek_, only thing there was ~/.tox/distshare/oslo.context-2.9.1.dev3.zip15:21
ayoungdstanek_, can you build off that tag?15:22
dstanek_ayoung: sure. i think i alredy have a f24 cloud node15:25
ayoungdstanek_, so, it is from build/temp.linux-x86_64-2.7/_openssl.c15:26
ayoungIn file included from /usr/include/openssl/asn1.h:65:0,15:26
*** openstackstatus has joined #openstack-keystone15:27
*** ChanServ sets mode: +v openstackstatus15:27
ayoung/usr/include/openssl/bio.h:692:6: note: previous declaration of ‘BIO_new_mem_buf’ was here15:27
ayoungthose both look like system libraries15:27
ayoungthe asn one is this line:  #  include <openssl/bio.h>15:28
ayoungmust be getting it from a different location, or with a different flag15:29
ayoung/usr/include/openssl/bio.h  seems to be the only one on the system, though15:29
*** akrzos has quit IRC15:30
ayoungdstanek_, there are other error in the traceback, too.  SSLv2_method  SSLv2_client_method SSLv2_server_method  redeclared15:31
*** akrzos has joined #openstack-keystone15:32
dstanek_bindep doesn't seem to work on f2415:37
*** spilla has joined #openstack-keystone15:39
bretonhm15:40
bretonis something wrong with gerrit and lp integration? I am getting "Provider is not supported, or was incorrectly entered."15:41
spillabreton launchpad is currently down, i know people have been running into some issues with gerrit, most likely because of that15:42
-openstackstatus- NOTICE: Earlier job failures for "zuul-cloner: error: too few arguments" should now be solved, and can safely be rechecked15:42
bretonyeah, things are broken15:43
-openstackstatus- NOTICE: Launchpad SSO is offline, preventing login to https://review.openstack.org/, https://wiki.openstack.org/ and many other sites; no ETA has been provided by the LP admin team15:47
*** adrian_otto has joined #openstack-keystone15:47
*** ChanServ changes topic to "Launchpad SSO is offline, preventing login to https://review.openstack.org/, https://wiki.openstack.org/ and many other sites; no ETA has been provided by the LP admin team"15:47
*** rcernin has quit IRC15:52
*** browne has joined #openstack-keystone15:55
openstackgerritBoris Bobrov proposed openstack/keystone: Remove support for PKI and PKIz tokens  https://review.openstack.org/37447916:01
dstanek_ayoung: so i'm seeing the same thing as you16:02
ayoungdstanek_, I thought pip freeze showed the set of package/versions,16:05
ayoungbut16:05
ayoungInstalling collected packages: cffi, cryptography16:05
ayoung  Found existing installation: cffi 1.3.116:05
ayoung    Uninstalling cffi-1.3.1:16:05
ayoung      Successfully uninstalled cffi-1.3.116:05
ayoungSuccessfully installed cffi-1.8.3 cryptography-1.5.116:05
ayoungbut then not seeing crypto in pip freeze16:05
openstackgerritAlexey Yelistratov proposed openstack/keystone: Add DB operations tracing  https://review.openstack.org/29453516:08
*** ChanServ changes topic to "Summit Brainstorm: https://etherpad.openstack.org/p/keystone-ocata-summit-brainstorm | Meeting Agenda https://etherpad.openstack.org/p/keystone-weekly-meeting | Newton retrospective: https://etherpad.openstack.org/p/keystone-newton-retrospective"16:09
-openstackstatus- NOTICE: Launchpad SSO logins are confirmed working correctly again16:09
dstanek_ayoung: pip freeze should show all of the python packages installed in that environment16:10
ayoungdstanek_, is it lying, or is the pip install lying?16:11
openstackgerritAlexander Makarov proposed openstack/keystone: OAuth1 driver for unified delegation  https://review.openstack.org/37096516:12
dstanek_ayoung: i'm not sure... i see it OK in my environment; are you using the same pip for installing and freezing?16:13
*** david-lyle has quit IRC16:13
*** david-lyle has joined #openstack-keystone16:14
stevemaro/16:15
stevemarsemi-online today in case someone is looking16:15
openstackgerritAlexander Makarov proposed openstack/keystone: Unified delegation model  https://review.openstack.org/20848816:15
*** daemontool_ has quit IRC16:17
*** dikonoor has joined #openstack-keystone16:17
openstackgerritBoris Bobrov proposed openstack/keystone: Remove support for PKI and PKIz tokens  https://review.openstack.org/37447916:22
*** slberger has joined #openstack-keystone16:23
ayoungdstanek_, I thought I was16:24
ayoungdstanek_, but you are seeing it too>?16:24
*** lamt has quit IRC16:25
dstanek_ayoung: yes, works on master, but fails on 8.1.216:26
ayoungdstanek_, and we are still supporting 8.1.2, so that means any commit there would fail, right?16:27
*** lamt has joined #openstack-keystone16:27
dstanek_ayoung: unless the devstack magic installs the right things. i'm not sure what this issue would be16:27
ayoungdstanek_, ah true, that should install things via packages for the most part16:28
openstackgerritRon De Rose proposed openstack/keystone: Add indexes to the revocation_event table  https://review.openstack.org/37652316:28
ayoungdoes not appear to install cryptography that way, though16:29
*** pcaruana has quit IRC16:29
openstackgerritRon De Rose proposed openstack/keystone: Add indexes to the revocation_event table  https://review.openstack.org/37652316:31
dstanek_ayoung: it's not installed on master for yoU?16:31
*** ezpz_ has joined #openstack-keystone16:36
*** ezpz_ is now known as ezpz16:36
openstackgerritSamuel Pilla proposed openstack/keystone: Domain included for role in list_role_assignment  https://review.openstack.org/37351616:36
*** ravelar has quit IRC16:36
stevemarayoung: dstanek_ that's liberty right? it's only supported for a few more days/weeks16:37
ayoungstevemar, yeah, well, its supported now, and I need to backport to it16:38
stevemarayoung: what are you backporting?16:38
ayoungstevemar, LDAP AD fixes16:39
bretonayoung: link to review?16:39
stevemarayoung: critical and security fixes only for upstream, but let's see the reviews anyway16:39
ayoungbreton, I don't have it yet, as I was trying to run a unit test first16:39
bretonstevemar: btw was mitaka cut? Patch https://review.openstack.org/374245 is still not merged :(16:40
ayounghttps://review.openstack.org/376627 Enable support for posixGroups in LDAP     is one of 216:40
ayoungdon't have the second backported yet16:40
stevemarbreton: it was already cut :(16:41
ayoungstevemar, we have a policy in RDO to not ship patches that have not been at least posted upstream fro16:41
bretonstevemar: that's bad. Should i abandon the change?16:41
stevemarbreton: no, i think we'll cut one more release eventually16:41
*** slberger has quit IRC16:42
stevemarbreton: you can argue that it's a security fix since it relates to auth16:42
ayounganything in keystone is a security fix16:43
bretonsold!16:43
stevemarayoung: the posixGroup fix is rather feature-ish16:44
stevemari'll get slapped from the stable team16:44
ayoungstevemar, almost certainly16:44
ayoungstevemar, that was my response internally, too.16:44
stevemarah okay, as long as we're on the same page :(16:44
ayoungBut since we only just got a Newton based build shipped, and we have people doing long burnin process on Mitaka, telling them to start over now will not fly16:45
ayoungDownstream mitaka based distro took a lot longer than expected16:45
openstackgerritAlexander Makarov proposed openstack/keystone: Verbose 401/403 debug responses  https://review.openstack.org/37243316:45
*** ravelar has joined #openstack-keystone16:47
*** pleia2_ is now known as pleia216:53
*** lamt has quit IRC16:54
*** ngupta has quit IRC16:54
*** ngupta has joined #openstack-keystone16:55
*** ngupta has quit IRC16:55
*** ngupta has joined #openstack-keystone16:55
*** esp has joined #openstack-keystone16:56
openstackgerritRon De Rose proposed openstack/keystone: Add indexes to the revocation_event table  https://review.openstack.org/37652316:59
*** slberger has joined #openstack-keystone17:00
*** mah has joined #openstack-keystone17:02
*** electrichead is now known as redrobot17:03
*** gyee has joined #openstack-keystone17:05
*** slberger has quit IRC17:06
*** jaosorior has quit IRC17:07
openstackgerritSteve Martinelli proposed openstack/keystone: create release notes for removed functionality  https://review.openstack.org/37591417:07
openstackgerritRon De Rose proposed openstack/keystone: Add docs for PCI-DSS  https://review.openstack.org/37442217:07
*** tqtran has joined #openstack-keystone17:11
*** LamT_ has quit IRC17:11
*** Guest68174 is now known as med_17:14
*** med_ has quit IRC17:14
*** med_ has joined #openstack-keystone17:14
*** morgan has joined #openstack-keystone17:18
openstackgerritRon De Rose proposed openstack/keystone: Change python code revocation search to sql  https://review.openstack.org/35937117:21
*** harlowja has joined #openstack-keystone17:30
*** slberger has joined #openstack-keystone17:30
*** dikonoor has quit IRC17:32
* morgan sighs.17:34
morgano/17:34
*** tonytan_brb is now known as tonytan4ever17:45
stevemarwhats up morgan17:50
stevemarmorgan: why the sigh, guy17:51
*** ravelar has quit IRC17:55
*** ravelar has joined #openstack-keystone17:56
*** amoralej is now known as amoralej|off17:58
*** spzala has quit IRC18:01
*** spzala has joined #openstack-keystone18:02
*** spzala_ has joined #openstack-keystone18:03
*** spzala has quit IRC18:07
openstackgerritRon De Rose proposed openstack/keystone: Add indexes to the revocation_event table  https://review.openstack.org/37652318:07
*** spzala_ has quit IRC18:08
openstackgerritRichard Avelar proposed openstack/keystone: Change python code revocation search to sql  https://review.openstack.org/35937118:09
*** spzala has joined #openstack-keystone18:11
*** browne has quit IRC18:15
openstackgerritMerged openstack/keystone: Using assertIsNone() instead of assertIs(None)  https://review.openstack.org/37629518:17
*** markvoelker_ has joined #openstack-keystone18:18
*** markvoelker has quit IRC18:19
*** spedione is now known as spedione|AWAY18:20
*** markvoelker has joined #openstack-keystone18:21
*** markvoelker_ has quit IRC18:24
*** thumpba has joined #openstack-keystone18:30
openstackgerritRodrigo Duarte proposed openstack/keystone: PCI-DSS functional tests  https://review.openstack.org/37701018:36
*** ngupta has quit IRC18:36
*** ngupta has joined #openstack-keystone18:36
rodrigodsstevemar, rderose ^18:36
openstackgerritAlexander Makarov proposed openstack/keystone: Unified delegation assignment driver  https://review.openstack.org/29131818:37
openstackgerritAlexander Makarov proposed openstack/keystone: Move dependency-related trust logic to manager  https://review.openstack.org/36073518:37
openstackgerritAlexander Makarov proposed openstack/keystone: Unified delegation trust driver  https://review.openstack.org/29187118:37
rderoserodrigods: cool and thanks! will take a looks here shortly.18:37
openstackgerritAlexander Makarov proposed openstack/keystone: OAuth1 driver for unified delegation  https://review.openstack.org/37096518:37
*** ngupta has quit IRC18:41
*** ngupta has joined #openstack-keystone18:41
*** browne has joined #openstack-keystone18:45
morganstevemar: my bouncer is dead.18:48
stevemarmorgan: womp womp18:48
morganstevemar: like... the whole server is dead18:48
morganlooking at options and considering ditching the server18:48
*** adrian_otto has quit IRC18:50
openstackgerritTony Xu proposed openstack/pycadf: Clean oslo.i18n  https://review.openstack.org/37452218:51
openstackgerritRodrigo Duarte proposed openstack/keystone: PCI-DSS functional tests  https://review.openstack.org/37701018:57
*** adrian_otto has joined #openstack-keystone19:07
*** spedione|AWAY is now known as spedione19:14
*** slberger has left #openstack-keystone19:19
*** jamielennox|away has quit IRC19:26
*** jamielennox|away has joined #openstack-keystone19:29
*** jamielennox|away is now known as jamielennox19:30
*** ChanServ sets mode: +v jamielennox19:30
*** sdake has quit IRC19:33
*** TonyXu has quit IRC19:41
*** haplo37_ has quit IRC19:45
*** crinkle_ is now known as crinkle19:45
*** sdake has joined #openstack-keystone19:46
*** haplo37_ has joined #openstack-keystone19:48
*** thiagolib has joined #openstack-keystone19:48
openstackgerritEric Brown proposed openstack/keystone: Update man page for Ocata release version and date  https://review.openstack.org/37706820:04
*** spedione is now known as spedione|AWAY20:05
*** Kimmo_ has quit IRC20:24
*** code-R has joined #openstack-keystone20:32
*** adriant has joined #openstack-keystone20:35
*** code-R_ has joined #openstack-keystone20:35
*** spzala has quit IRC20:35
*** spzala has joined #openstack-keystone20:36
*** code-R has quit IRC20:38
bknudsonrderose: ravelar: the index on user_id sped things up significantly!20:38
bknudsonToken validated in 0:00:00.031574. -- used to be 0.120:38
bknudsonand it's more like 0.4 with the original code (not doing sql filtering)20:39
*** spzala has quit IRC20:40
bknudsonhere's docs on explain output: https://dev.mysql.com/doc/refman/5.7/en/explain-output.html20:41
rderosebknudson: sweet!20:41
bknudsonhere's docs on using explain: https://dev.mysql.com/doc/refman/5.7/en/using-explain.html20:41
bknudsonI don't know how we would find what indexes we need... not sure if mysql can use multiple indexes on a table.20:42
rderosebknudson: nice. adding a couple indexes here:https://review.openstack.org/#/c/376523/20:42
rderosebut leery about adding more20:42
rderosebknudson: in can, but with too many we can confuse the optimizer and slow down writes20:43
bknudsonrderose: based on the docs looks like it's not useful to have multiple indexes if 1 works.20:43
ravelarbknudson that's good news!20:43
rderoseso have to be careful20:43
*** ngupta has quit IRC20:43
bknudsonhttps://dev.mysql.com/doc/refman/5.7/en/explain-output.html looks like it picks one index for a table.20:43
*** code-R_ has quit IRC20:43
bknudsonpossible_keys == "The possible indexes to choose" -> key == "The index actually chosen"20:44
rderosebknudson: but if you do have multiple indexes, you can give HINTS on which to use: http://dev.mysql.com/doc/refman/5.7/en/index-hints.html20:44
bknudsonrderose: how should mysql choose which index to use in this case?20:45
bknudsonI chose user_id because every query is going to have user_id20:46
bknudsonand potentially multiple times.20:46
rderoseyeah, so user_id is a good one20:46
bknudsonwhat's the point of having an index on user_id and an index on issued_before ?20:47
rderoseI'm adding that with issued_before (composite) because issued_before is always included20:47
bknudsonthe query is always going to have both those fields20:47
rderosenot always, right20:47
bknudsonevery token has a user_id20:47
rderosehmm...20:47
ravelarbut not every revocation20:48
*** mah has quit IRC20:48
bknudsonyes, in my case none of the revocation events had a user_id20:49
bknudsonprobably why the index was so successful20:49
bknudsonI mean the event had a user_id but it's NULL20:49
rderoseravelar bknudson: right, so the revocation event may not match a user_id, but will always check issued_before20:49
bknudsonrderose: what does that have to do with use of an index or not?20:50
bknudsonthe query always looks the same20:50
rderosebknudson: you mean we're always querying the same data?20:51
bknudsonrderose: the query is always the same. The data can be different.20:51
bknudsonMight have a lot of user revocations. In my case I only had token revocations.20:51
*** ebalduf has joined #openstack-keystone20:52
rderosebknudson: but I think the where clause should be different based on data in the event. in other words, if we are checking if user_id = '123456', we shouldn't check if user_id also null20:53
rderosebknudson: if we only have values for user_id, shouldn't it just check for user_id = that value20:54
rderose?20:54
*** mvk has joined #openstack-keystone20:54
bknudsonrderose: if that's the case then the code in the review is completely wrong.20:54
bknudsonrderose: the token always has a user_id20:54
rderosebknudson: true, I think ravelar is working on that20:55
rderosebknudson: but logically, it will still work. just the index is not so useful20:55
rderose:)20:55
rderosebknudson: so we should index all columns?20:58
bknudsonrderose: experiment to see what indexes help.20:58
bknudsonwe shouldn't have to guess about this.20:58
rderosebknudson: true20:59
bknudsonmaybe we can add a token validation / revocation events test to lbragstad's perf tester?21:00
*** gyee has quit IRC21:00
*** esp has left #openstack-keystone21:03
*** raildo has quit IRC21:09
*** ebalduf has quit IRC21:13
*** catintheroof has joined #openstack-keystone21:15
openstackgerritRodrigo Duarte proposed openstack/keystone: PCI-DSS functional tests  https://review.openstack.org/37701021:19
*** spilla has quit IRC21:22
ravelarbknudson the query you are seeing is checking for null when it cannot find a value for token['trustor_id'] or trustee_id21:22
ravelarotherwise itll show the full check for null or comparison if token[key] is not None21:22
*** ngupta has joined #openstack-keystone21:25
ravelarbut the reason the is None is there is because, like in matches, the purpose is to ignore the comparison if the event column is None. But if it is not then a rev event column with a non None value and a token value with none should give back False.21:25
*** ayoung has quit IRC21:26
*** pnavarro has quit IRC21:32
*** sdake has quit IRC21:33
*** lamt has joined #openstack-keystone21:35
*** Kimmo_ has joined #openstack-keystone21:37
*** adrian_otto has quit IRC21:38
*** Aiduehb has joined #openstack-keystone21:43
*** Aiduehb_ has joined #openstack-keystone21:44
*** Aiduehb has quit IRC21:47
*** nicolasbock has quit IRC21:51
*** Aiduehb_ has quit IRC21:52
*** ezpz has quit IRC21:55
bknudsonravelar: I'm not complaining about the query. That was rderose.21:57
rderose:)21:57
rderoseyeah, I'm finally understanding the query now21:58
ravelarbknudson haha oh okay whoops, just trying to clarify21:58
bknudsonthere only needs to be one check for if the value is null. The other checks are redundant21:59
*** edtubill has quit IRC21:59
bknudsonquery optimizer should take care of it.21:59
bknudsonbut people reading the code and trying to figure out what it's doing are going to wonder why it's written the way it is.22:00
*** iurygregory_ has joined #openstack-keystone22:00
*** nicolasbock has joined #openstack-keystone22:00
rderosebknudson: yeah, this is hard, because we're trying to search for project_id or (project_id and domain_id) or (project_id and domain_id and user_id) based on what's in the token data22:03
rderosebknudson: so we always need to OR if null22:03
rderosecomplicated...22:04
bknudsonthe code doesn't have to obfuscate things.22:04
rderoseright22:04
bknudsonalso we can have unit tests that show that the code is correct given different inputs.22:04
rderosebknudson: I know, it's just trying to make sense of a query that is doing all of these different combinations22:06
*** ravelar has quit IRC22:06
rderosebknudson: anyway, playing with this now to see if we can eliminate the extra null checks22:07
*** dstanek_ is now known as dstanek22:08
*** ChanServ sets mode: +v dstanek22:08
*** ravelar has joined #openstack-keystone22:10
*** iurygregory_ has quit IRC22:12
*** ayoung has joined #openstack-keystone22:14
*** ChanServ sets mode: +v ayoung22:14
bknudsonactually, in my testing looks like the indexes didn't help much. Couldn't tell just by looking at the numbers.22:23
*** iurygregory_ has joined #openstack-keystone22:25
*** ngupta has quit IRC22:26
*** ngupta has joined #openstack-keystone22:27
*** lamt has quit IRC22:31
*** ngupta has quit IRC22:31
morganbknudson: how large of a table?22:34
*** spzala has joined #openstack-keystone22:34
morganbknudson: because at small row counts indexews will have no real value22:34
bknudsonmorgan: I usually get a couple thousand rows in there.22:34
*** ngupta has joined #openstack-keystone22:34
morganhm22:34
bknudsonalthough it's faster now so might have been more.22:34
* morgan nods22:34
*** spzala has quit IRC22:36
*** spzala has joined #openstack-keystone22:37
*** thumpba has quit IRC22:40
*** thumpba has joined #openstack-keystone22:40
*** spzala has quit IRC22:41
*** thumpba has quit IRC22:42
*** ngupta has quit IRC22:56
*** ngupta has joined #openstack-keystone22:57
*** ngupta has quit IRC22:57
*** ngupta has joined #openstack-keystone22:58
*** edmondsw has quit IRC23:01
*** catintheroof has quit IRC23:04
*** adrian_otto has joined #openstack-keystone23:06
*** ngupta has quit IRC23:13
*** ngupta has joined #openstack-keystone23:13
*** ngupta has quit IRC23:17
*** adu has joined #openstack-keystone23:27
*** lamt has joined #openstack-keystone23:28
*** adrian_otto has quit IRC23:38
*** adrian_otto has joined #openstack-keystone23:40
*** adrian_otto has quit IRC23:41
*** sdake has joined #openstack-keystone23:44
*** timburke_ is now known as timburke23:46
*** haplo37_ has quit IRC23:51
*** haplo37_ has joined #openstack-keystone23:53

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!