Friday, 2018-07-27

*** david-lyle has joined #openstack-keystone00:03
*** harlowja has joined #openstack-keystone00:05
*** dklyle has quit IRC00:06
*** linkmark has quit IRC00:07
*** dklyle has joined #openstack-keystone00:12
*** dklyle_ has joined #openstack-keystone00:14
*** david-lyle has quit IRC00:14
*** itlinux has joined #openstack-keystone00:15
*** dklyle has quit IRC00:17
*** gyee has quit IRC00:23
*** itlinux has quit IRC00:25
openstackgerritMerged openstack/keystone master: Deprecate [token] infer_roles=False  https://review.openstack.org/57486900:49
*** felipemonteiro has quit IRC00:55
*** harlowja has quit IRC01:17
*** dklyle has joined #openstack-keystone01:49
*** dklyle_ has quit IRC01:50
*** david-lyle has joined #openstack-keystone02:01
*** dklyle has quit IRC02:02
*** david-lyle has quit IRC02:03
*** dklyle_ has joined #openstack-keystone02:04
*** gongysh has joined #openstack-keystone02:22
*** dave-mccowan has quit IRC03:24
kmallocknikolla: irccloud isn't great on that front either.03:27
knikollakmalloc: it's definitely better though03:28
*** annp has quit IRC03:31
*** annp has joined #openstack-keystone03:32
*** abhi89 has joined #openstack-keystone03:51
*** links has joined #openstack-keystone03:52
*** gongysh has quit IRC03:52
gagehugoknikolla I must have clicked the non-v3-only one04:17
gagehugonice that it auto-generates a folder for that04:17
gagehugowxy I retested the notification issue, something weird is going on http://paste.openstack.org/show/726725/04:21
gagehugoit appears to be populating the initiator id correctly now for all cases04:22
gagehugothe main issue was we were seeing an event when logging into horizon, but with the initiator id was random, which isn't good04:28
*** pcaruana has joined #openstack-keystone04:28
*** pcaruana has quit IRC04:30
openstackgerritMerged openstack/keystone master: Address nits in strict-two-level implementation  https://review.openstack.org/58537904:40
*** shyamb has joined #openstack-keystone04:46
*** gongysh has joined #openstack-keystone04:50
*** abhi89 has quit IRC04:56
*** abhi89 has joined #openstack-keystone04:57
*** flwang1 has quit IRC04:59
*** fiddletwix has joined #openstack-keystone05:03
*** josecastroleon has joined #openstack-keystone05:48
*** annp has quit IRC05:51
*** annp has joined #openstack-keystone05:52
*** zigo_ has joined #openstack-keystone05:53
*** zigo has quit IRC05:53
*** martinus__ has joined #openstack-keystone05:59
*** shyamb has quit IRC06:15
*** shyamb has joined #openstack-keystone06:15
*** shyamb has quit IRC06:22
*** sapd has quit IRC06:22
*** sapd has joined #openstack-keystone06:23
*** shyamb has joined #openstack-keystone06:35
*** shyambiradar has joined #openstack-keystone06:44
abhi89wxy: in continuation with our discussion the other day regarding identity.project.created events not reaching panko database.. the event sure gets generated at https://github.com/openstack/keystone/blob/master/keystone/resource/core.py#L230, https://github.com/openstack/keystone/blob/master/keystone/notifications.py#L721-L74206:46
abhi89wxy: but, in oslo_messaging, it doesn't reach here - https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/notify/notifier.py#L31206:47
*** shyamb has quit IRC06:47
abhi89this condition fails - https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/notify/notifier.py#L31806:47
abhi89so the event doesn't reach message queue at all06:48
abhi89this a bug?06:48
*** elibrokeit is now known as ztrawhcse06:51
*** tesseract has joined #openstack-keystone06:52
*** rcernin has quit IRC07:00
*** ispp has joined #openstack-keystone07:00
wxyabhi89: only project.created events is blocked? other events in keystone works well?07:04
abhi89wxy: all keystone related events suffer this problem07:11
*** dtantsur|afk is now known as dtantsur07:21
*** ispp has quit IRC07:27
*** AlexeyAbashkin has joined #openstack-keystone07:29
wxyabhi89: emm I guess it's maybe your config problem. gagehugo is testing identity.authenticate notification, the whole workflow at least seems works well.07:30
*** kfox1111 has quit IRC07:36
wxygagehugo: a little werid. I tested again. the id is still a auto-generated uuid, not user_id.07:41
*** shyambiradar has quit IRC07:47
*** ispp has joined #openstack-keystone07:48
*** shyambiradar has joined #openstack-keystone07:48
*** dtantsur is now known as dtantsur|bbl08:00
abhi89wxy: i had tested for project.created & updated.. checked again for identity.authenticate.success, same issue still.. reg config problem, might be, but is there any specific event related config properties we set, i am not sure.. asaik, events are generated by default except for audit events..08:06
wxyabhi89: I suggest to register a bug. Add your env and config info there.08:13
abhi89wxy: ok08:13
*** shyambiradar has quit IRC08:16
*** flwang1 has joined #openstack-keystone08:34
*** jaosorior has quit IRC08:38
*** flwang1 has quit IRC08:46
*** shyamb has joined #openstack-keystone08:48
*** lifeless has quit IRC08:54
*** jaosorior has joined #openstack-keystone08:58
wxyabhi89: here is my test for project.created: {"oslo.message": "{\"priority\": \"INFO\", \"_unique_id\": \"b0ad03af1c9d442f87ad7209e1e3bd05\", \"event_type\": \"identity.project.created\", \"timestamp\": \"2018-07-27 08:59:39.008041\", \"publisher_id\": \"identity.devstack\", \"payload\": {\"typeURI\": \"http://schemas.dmtf.org/cloud/audit/1.0/event\", \"initiator\": {\"typeURI\": \"service/security/account/user\",09:01
wxy\"host\": {\"agent\": \"python-keystoneclient\", \"address\": \"10.3.150.25\"}, \"project_id\": \"3a705b9f56bb439381b43c4fe59dccce\", \"user_id\": \"9a7e43333cc44ef4b988f05fc3d3a49d\", \"id\": \"9a7e43333cc44ef4b988f05fc3d3a49d\"}, \"target\": {\"typeURI\": \"data/security/project\", \"id\": \"e2c18e2ff5f541278f7d16ce0e7be362\"}, \"observer\": {\"typeURI\": \"service/security\", \"id\": \"09a09:01
wxyabhi89: It works well.09:01
abhi89wxy: i got this object too.. but did it reach here - https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/notify/notifier.py#L31209:06
wxyabhi89: looking.09:07
*** d0ugal has joined #openstack-keystone09:08
*** flwang1 has joined #openstack-keystone09:09
*** josecastroleon has quit IRC09:09
*** lifeless has joined #openstack-keystone09:11
wxyabhi89: It's called by notifer.info() of cause it reached there.09:12
wxyabhi89: I got the project.created notification from rabbitmq09:13
abhi89wxy: yes, its called by notifier.info..it even reached https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/notify/notifier.py#L298.. but for me its not going in https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/notify/notifier.py#L310.. and hence not reaching the message queue..09:14
wxyabhi89: did you config the [oslo_messaging_notifications] section in keystone.conf?09:14
abhi89wxy: its all commented in [oslo_messaging_notifications]09:16
wxyabhi89: well, you should update "transport_url" and "driver" options in that group09:16
abhi89wxy: oh ok.. let me check..09:17
wxydriver = messagingv2  transport_url = rabbit://user_name:password@ip:port/09:17
*** josecastroleon has joined #openstack-keystone09:18
wxyif keystone connects to rabbitmq success, you can get a topic named "keystone" by default using `rabbitmqctl list_exchanges` command09:19
wxyabhi89: have to leave now. If it still doesn't work. Feel free to leave message or register a bug. Bye.09:32
abhi89wxy: i had already registered a bug.. will cancel it if this works.. thanks.. bye..09:32
*** andymccr- has joined #openstack-keystone09:47
*** jaosorior has quit IRC09:49
*** andymccr_ has quit IRC09:50
*** johnthetubaguy has quit IRC09:52
*** flwang1 has quit IRC09:55
*** flwang1 has joined #openstack-keystone09:56
*** abhi89 has quit IRC09:57
*** abhi89 has joined #openstack-keystone09:57
*** flwang1 has quit IRC10:00
*** andymccr has quit IRC10:04
*** andymccr- is now known as andymccr10:05
*** liuzz_ has quit IRC10:09
*** ispp has quit IRC10:09
*** flwang1 has joined #openstack-keystone10:13
*** abhi89 has quit IRC10:17
*** abhi89 has joined #openstack-keystone10:18
*** neha_alhat has joined #openstack-keystone10:26
*** shyamb has quit IRC10:32
*** shyamb has joined #openstack-keystone10:32
*** ispp has joined #openstack-keystone10:33
*** shyamb has quit IRC10:33
*** shyamb has joined #openstack-keystone10:34
*** shyamb has quit IRC10:39
*** gongysh has quit IRC10:47
*** dtantsur|bbl is now known as dtantsur10:49
*** abhi89 has quit IRC11:01
*** dave-mccowan has joined #openstack-keystone11:06
*** pooja_jadhav has quit IRC11:08
*** shyamb has joined #openstack-keystone11:15
*** s10 has joined #openstack-keystone11:15
*** flwang1 has quit IRC11:22
*** flwang1 has joined #openstack-keystone11:26
*** neha_alhat has quit IRC11:27
*** pooja_jadhav has joined #openstack-keystone11:29
*** flwang1 has quit IRC11:33
*** flwang1 has joined #openstack-keystone11:34
*** d0ugal has quit IRC11:41
*** csatari has joined #openstack-keystone11:52
csatariknikolla I started to configure a Shibboleth for keystone-tempest-plugin. If I get it right I need a metadata file from the SP, what is Keystone under test now. Do you have any idea how can I get this?11:55
*** linkmark has joined #openstack-keystone12:03
*** ispp has quit IRC12:08
*** shyamb has quit IRC12:16
*** edmondsw has joined #openstack-keystone12:17
*** johnthetubaguy has joined #openstack-keystone12:17
*** ispp has joined #openstack-keystone12:20
knikollacsatari: that will be $HOST/Shibboleth.sso/Metadata12:25
knikollaSince the authentication for the SP is handled by the Shibboleth module for Apache12:26
*** annp has quit IRC12:27
csatariWow. Got it.12:27
csatariThanks.12:28
knikolla:)12:29
*** ispp has quit IRC12:36
*** shyamb has joined #openstack-keystone12:37
*** flwang1 has quit IRC12:43
*** flwang1 has joined #openstack-keystone12:46
*** ispp has joined #openstack-keystone13:03
*** flwang1 has quit IRC13:14
*** shyamb has quit IRC13:17
*** shyamb has joined #openstack-keystone13:24
*** flwang1 has joined #openstack-keystone13:31
*** jistr is now known as jistr|mtg13:32
*** spilla has joined #openstack-keystone13:42
*** shyamb has quit IRC13:49
*** zioproto has joined #openstack-keystone13:51
zioprotoedmondsw: hello there, are you online?13:51
zioprotoI am looking at a bug where you were involved13:52
edmondswzioproto hi13:52
edmondswwhat bug?13:52
zioprotohttps://bugs.launchpad.net/keystone/+bug/166276213:52
openstackLaunchpad bug 1662762 in OpenStack Identity (keystone) ocata "Authentication for LDAP user fails at MFA rule check" [High,Fix released] - Assigned to Matthew Edmonds (edmondsw)13:52
zioprotohi :)13:52
zioprotoI am upgrading keystone from newton to ocata13:52
zioprotoI see the following in my log files: http://paste.openstack.org/show/726752/13:52
zioprotoit looks similar to that bug but not exactly the same13:53
zioprotokeystone package version from the ubuntu distro is 2:11.0.3-0ubuntu1~cloud0 so that patch to that bug should be included13:53
edmondswzioproto what version of OpenStack are you running?13:53
zioprotoedmondsw: 2:11.0.3-0ubuntu1~cloud013:54
zioprotoOpenstack Newton, I am now upgrading Keystone to Ocata as the first step of the upgrade13:54
edmondswcan you translate that to an OpenStack release for me? Pike? Queens? etc.13:54
edmondswah, newton13:54
edmondswI don't think that bug was fixed in newton, was it?13:54
zioproto11.0.3 is Ocata13:54
zioprotokeystone package version is 2:11.0.3-0ubuntu1~cloud0 ( Ocata )13:55
edmondswah, ocata... yeah, looks like it was fixed in ocata13:55
edmondswbut via backport... so it's possible that Ubuntu didn't pick up the fix13:55
edmondswdid you check the code to see if the change was present?13:56
zioprotolook carefully13:56
zioprotoI get a stacktrace in keystone/auth/core.py", line 37713:56
zioprotooh you are right13:56
zioprotothat is the very same stacktrace13:56
edmondswthis was the fix in ocata: https://git.openstack.org/cgit/openstack/keystone/commit/?id=95160d1812104d90ff096cb2d32390b065bfeaae13:57
openstackgerritNeha Alhat proposed openstack/keystonemiddleware master: Register 'split_loggers' conf option  https://review.openstack.org/57800813:57
edmondswzioproto check to see if those changes are on your ocata install... if not, then you need Ubuntu to backport that13:57
edmondswor make the changes yourself13:57
zioprotogive me a minute to check13:57
zioprotohttps://review.openstack.org/#/c/437998/1 says the change is included in 11.0.1 already, but lets double check13:58
zioprotoedmondsw: I confirm the fix implemented in https://review.openstack.org/#/c/437998 is in my packages14:02
edmondswok, so seems like there is some other case that the fix didn't cover14:02
zioprotoyes that is what I mean from the beginning ;)14:03
zioprotoI am not sure how to make a request to reproduce that14:03
zioprotoI just see it very frequently in the logs14:03
zioprotoit is a testing system to test the upgrade14:03
zioprotoso I can play with it14:03
edmondswbest thing to do might be to recreate with devstack and then see if one of the keystone guys can help14:04
edmondswI'm pretty busy elsewhere these days14:04
edmondswlbragstad is there anyone still around who's knowledgable about ldap?14:05
csatariknikolla> Next thing what I do not find if the pem file for signature validation.14:05
zioprotobut I dont have any LDAP. I would like to make that clear14:06
edmondswzioproto... oh, no ldap?14:06
zioprotoexactly14:06
edmondswah... so the ldap fix definitely isn't what you need :)14:06
edmondswwhat are you using instead of ldap?14:06
zioprotothe mysql database that comes with keystone14:07
zioprotoand UUID tokens14:07
zioprotoI get this stuff just on the admin port 3535714:09
zioprotoI mean all requests that have this stacktrace are requests going to the admin port14:10
*** links has quit IRC14:10
edmondswzioproto I wonder if this is an upgrade issue...14:13
zioprotoI would not be surprised :) I am trying to understand how to reproduce a request that causes the backtrace14:14
zioprotofrom the keystone logs I just see a POST request to the admin port, but the headers are not logged, so I dont know much about the request14:14
zioprotothe request comes from nova14:16
edmondswzioproto the trace indicates that you're getting this when you try to authenticate (i.e. get a token)14:16
zioprotosorry does not come from nova14:17
zioprotoI mean14:17
zioprotoI get a line14:17
zioprotobefore the stacktrace that looks like this:14:17
edmondswto recreate, see if you can get a token with any user that was created prior to the upgrade14:17
zioproto2018-07-27 13:33:07.783 26 INFO keystone.common.wsgi [req-9d09a563-bb73-4e32-a309-6fef7ef96901 - - - - -] POST https://keystone.scloud.switch.ch:35357/v3/auth/tokens14:17
edmondswdo you know how to do that?14:17
zioprotoopenstack token issue14:17
zioprotoit works14:17
edmondswfor what kind of user did you try it?14:18
zioprotoboth with normal user and admin14:18
*** jlvacation is now known as jlvillal14:18
edmondswtry it with one of the service users14:18
edmondswe.g. the user configured in the keystone_authtoken section of nova.conf14:18
*** r-daneel has joined #openstack-keystone14:18
*** felipemonteiro has joined #openstack-keystone14:19
zioprotoedmondsw: it works I successfully get a token14:24
edmondswI think you might have to setup a debugger to dig into this14:25
edmondswzioproto sorry I haven't been more help, but I need to get back to my day job14:25
zioprotoedmondsw: no problem ! thanks. I updated the bug. I hope someone picks it up :)14:26
edmondswzioproto I suggest you open a new bug... this one has been fixed, so it's not going to get attention14:26
edmondswand that bug was for LDAP, which is not your case14:26
edmondswso they're related, but different bugs14:26
edmondswzioproto mention the old bug in the new one that you open, but be clear that your case is NOT using LDAP14:27
ayoungknikolla, can you pull the trigger on https://review.openstack.org/#/c/583278/  please?14:28
ayoungI confirmed that the move from the old directory to the new maintained the original logic. The rest is just Flask specific changes which you should be able to verify quickly14:28
ayoungand I am afraid of it bit-rotting.14:28
*** links has joined #openstack-keystone14:29
kmallocedmondsw: shh ayoung  and I are still around, but we pretend we don't hear LDAP ever. Fwiw, that trace looks like maybe dbsync needs to be run.14:31
kmalloczioproto: ^ did you run dbsync when upgrading?14:31
kmallocayoung: it shouldn't be too risky to bitrot, I'm actively working on the chain still, auth is just taking a lot longer because things like os-federation is involved. Though I would like if that landed sooner to avoid rebase ick. :)14:32
kmallocknikolla: ^cc14:32
*** breton has quit IRC14:33
ayoungkmalloc, auth is some messy code14:35
ayoungyeah, the rebase/bitrot/things around it changin effect is what I want to avoid14:36
kmallocYep. And we have a lot.of bits that directly referenced auth.controllers magically, which kindof violates "web framework" best practices.. so... Working to fix that too. :(14:36
ayoungpretty stoked about how this is going to look when it is done.  Kind of like a long await remodel of a house14:37
zioprotokmalloc: you are right, looking at my Kibana looks like these messages appeared when I upgraded, i had few hundreds, and then they disappeared completely14:38
zioprotokmalloc: probably db_sync fixed the problem14:38
kmalloczioproto: :).14:38
zioprotoI am looking in Kibana for : "mfa_rules = user_ref['options'].get(ro.MFA_RULES_OPT.option_name, [])"14:39
*** jistr|mtg is now known as jistr14:39
kmallocI figured as much, but glad it is resolved now.14:39
zioprotoI have 111 hits when I upgraded packages, and then I had run db_sync by hand14:39
zioprotoit is weird db_sync did not produce any output, and db_version is still the same14:40
kmallocThat I don't know, I haven't used Ubuntu packaged OpenStack ever.14:40
zioprotook no worries14:40
zioprotoedmondsw: looks like it is sorted out14:41
*** bhagyashris has quit IRC14:41
*** flwang1 has quit IRC14:41
zioprotoI will try to reproduce it again tomorrow, rollback to newton again a14:41
zioprotoand check if all requests fail like that before db_sync14:41
zioprotothanks !14:41
*** felipemonteiro_ has joined #openstack-keystone14:42
lbragstad_https://review.openstack.org/#/c/545450/ is ready for some reviews ;)14:45
*** felipemonteiro has quit IRC14:46
edmondswzioproto glad you got it sorted14:48
*** flwang1 has joined #openstack-keystone14:50
*** imacdonn has quit IRC14:52
*** imacdonn has joined #openstack-keystone14:52
*** breton has joined #openstack-keystone14:56
*** shyamb has joined #openstack-keystone14:58
*** links has quit IRC15:02
*** josecastroleon has quit IRC15:07
*** r-daneel_ has joined #openstack-keystone15:08
*** r-daneel has quit IRC15:09
*** r-daneel_ is now known as r-daneel15:09
*** andymccr has quit IRC15:34
*** andymccr has joined #openstack-keystone15:35
-openstackstatus- NOTICE: A zuul config error slipped through and caused a pile of job failures with retry_limit - a fix is being applied and should be back up in a few minutes15:38
*** ChanServ changes topic to "A zuul config error slipped through and caused a pile of job failures with retry_limit - a fix is being applied and should be back up in a few minutes"15:38
lbragstad_hrybacki: fyi - i'm going through the trello board and updating cards15:39
lbragstad_there are somethings we might not need anymore15:39
lbragstad_liek https://trello.com/c/qCeaLYqx/45-write-specification-for-external-pdp-pep-enforcement15:39
*** AlexeyAbashkin has quit IRC15:46
*** gyee has joined #openstack-keystone15:52
*** flwang1 has quit IRC15:55
*** r-daneel_ has joined #openstack-keystone15:58
*** flwang1 has joined #openstack-keystone16:00
*** r-daneel has quit IRC16:00
*** r-daneel_ is now known as r-daneel16:00
openstackgerritMerged openstack/keystone-specs master: Update the default roles spec to include Rocky details  https://review.openstack.org/58267316:00
*** openstackgerrit has quit IRC16:04
*** openstackgerrit has joined #openstack-keystone16:09
openstackgerritGage Hugo proposed openstack/keystone master: Expose random uuid bug in cadf notifications  https://review.openstack.org/58078016:09
gagehugolbragstad_ added info from the bug report16:09
*** lbragstad_ is now known as lbragstad16:11
lbragstadgagehugo: thanks!16:11
*** ispp has quit IRC16:11
*** flwang1 has quit IRC16:12
*** links has joined #openstack-keystone16:14
*** links has quit IRC16:17
*** links has joined #openstack-keystone16:17
*** links has quit IRC16:23
*** harlowja has joined #openstack-keystone16:27
*** flwang1 has joined #openstack-keystone16:28
*** tesseract has quit IRC16:32
*** shyamb has quit IRC16:33
*** felipemonteiro__ has joined #openstack-keystone16:52
*** felipemonteiro_ has quit IRC16:52
*** felipemonteiro_ has joined #openstack-keystone17:06
*** felipemonteiro__ has quit IRC17:10
*** dtantsur is now known as dtantsur|afk17:10
*** felipemonteiro__ has joined #openstack-keystone17:18
*** felipemonteiro_ has quit IRC17:18
*** jmlowe has joined #openstack-keystone17:24
*** harlowja has quit IRC17:31
*** felipemonteiro_ has joined #openstack-keystone17:34
*** felipemonteiro__ has quit IRC17:37
*** harlowja has joined #openstack-keystone18:15
*** jmlowe has quit IRC18:31
*** flwang1 has quit IRC18:41
*** flwang1 has joined #openstack-keystone18:42
*** r-daneel_ has joined #openstack-keystone18:47
*** r-daneel has quit IRC18:47
*** r-daneel_ is now known as r-daneel18:47
*** s10 has quit IRC18:48
*** r-daneel has quit IRC19:04
*** lbragstad_ has joined #openstack-keystone19:40
*** lbragstad has quit IRC19:41
*** mchlumsky_ has quit IRC19:47
*** mchlumsky has joined #openstack-keystone19:50
*** lbragstad_ is now known as lbragstad19:56
*** mchlumsky has quit IRC19:56
*** mchlumsky has joined #openstack-keystone19:58
*** itlinux has joined #openstack-keystone20:03
*** mchlumsky has quit IRC20:05
*** mchlumsky has joined #openstack-keystone20:07
*** s10 has joined #openstack-keystone20:09
*** dtruong_ has quit IRC20:23
*** dtruong_ has joined #openstack-keystone20:26
*** mchlumsky has quit IRC20:37
*** felipemonteiro_ has quit IRC20:40
*** felipemonteiro_ has joined #openstack-keystone20:40
*** david-lyle has joined #openstack-keystone20:54
*** dklyle_ has quit IRC20:57
*** martinus__ has quit IRC20:59
*** edmondsw has quit IRC21:04
*** edmondsw has joined #openstack-keystone21:05
*** r-daneel has joined #openstack-keystone21:05
*** edmondsw has quit IRC21:10
*** r-daneel has quit IRC21:16
*** spilla has quit IRC21:49
*** itlinux has quit IRC22:05
*** felipemonteiro_ has quit IRC22:06
*** jmlowe has joined #openstack-keystone22:07
*** harlowja has quit IRC23:09
*** gongysh has joined #openstack-keystone23:43
*** itlinux has joined #openstack-keystone23:50
*** itlinux has quit IRC23:50

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!