Wednesday, 2019-04-10

*** erus has quit IRC00:44
*** lbragstad has quit IRC00:45
*** gyee has quit IRC01:03
*** whoami-rajat has joined #openstack-keystone01:12
*** jamesmcarthur has joined #openstack-keystone01:20
*** jamesmcarthur has quit IRC01:23
*** openstackgerrit has quit IRC01:30
cmurphyhogepodge: i wouldn't usually think of a 503 error as a performance error, are there tracebacks in the logs?01:43
hogepodgecmurphy: tracking things down, I think it's an openstack client issue now01:57
hogepodgerolling back to Rocky didn't change anything01:57
hogepodgeoddly enough if I turn the verbosity of the openstack client up I get much better reliability01:58
cmurphyhogepodge: hmm, not sure how openstackclient could cause server side issues02:00
hogepodgethat's the thing, I'm not seeing any server side logs (I thought maybe it was a uwsgi host thing)02:00
hogepodgeI'm putting keystone under some mild load using the openstack client across a bunch of different containers, and the number of 503s I'm getting back is alarming.02:01
cmurphyhogepodge: not seeing *any* server side logs, or not seeing any errors in the logs?02:02
hogepodgeI'm not seeing any errors in the keystone logs02:05
cmurphyah hmm02:05
hogepodgeI actually still don't know if it's uwsgi or docker-proxy causing the issue. I'm going to disable the userland proxy and see if that changes anything.02:20
*** adriant has joined #openstack-keystone02:38
*** erus has joined #openstack-keystone02:45
*** openstackgerrit has joined #openstack-keystone03:00
openstackgerritColleen Murphy proposed openstack/keystone master: Fix werkzeug imports for version 0.15.x  https://review.openstack.org/65143003:00
cmurphyprometheanfire: ^ i like that the "deprecation warning" causes a fatal error03:00
*** vishakha has joined #openstack-keystone03:09
prometheanfirelol, yep03:11
*** dave-mccowan has quit IRC04:15
*** erus has quit IRC04:30
*** openstackstatus has quit IRC04:35
*** openstackstatus has joined #openstack-keystone04:36
*** ChanServ sets mode: +v openstackstatus04:36
*** pcaruana has joined #openstack-keystone05:06
*** shyamb has joined #openstack-keystone06:01
*** shyamb has quit IRC06:40
*** shyamb has joined #openstack-keystone06:52
openstackgerritVishakha Agarwal proposed openstack/keystone master: Fix sphinx for requirements.txt  https://review.openstack.org/65144406:58
*** awalende has joined #openstack-keystone07:08
*** shyamb has quit IRC07:10
*** shyamb has joined #openstack-keystone07:10
*** shyamb has quit IRC07:26
*** pcaruana has quit IRC07:34
*** pcaruana has joined #openstack-keystone07:35
*** shyamb has joined #openstack-keystone07:37
*** shyamb has quit IRC07:58
*** tkajinam has quit IRC08:04
*** phasespace has quit IRC08:08
*** evrardjp has quit IRC08:18
*** evrardjp has joined #openstack-keystone08:19
openstackgerritVishakha Agarwal proposed openstack/keystone master: WIP : Missing packages in lower-contraints.txt  https://review.openstack.org/65146208:46
openstackgerritVishakha Agarwal proposed openstack/keystone master: Fix sphinx for requirements.txt  https://review.openstack.org/65144408:47
*** shyamb has joined #openstack-keystone09:07
*** phasespace has joined #openstack-keystone09:30
*** rcernin has quit IRC09:47
*** shyamb has quit IRC10:35
openstackgerritVishakha Agarwal proposed openstack/keystone master: WIP : Missing packages in lower-contraints.txt  https://review.openstack.org/65146210:44
*** shyamb has joined #openstack-keystone10:47
*** shyamb has quit IRC10:55
*** shyamb has joined #openstack-keystone11:12
*** sapd1 has quit IRC11:29
*** sapd1 has joined #openstack-keystone11:29
*** aloga has quit IRC11:35
*** aloga has joined #openstack-keystone11:36
*** ybunker has joined #openstack-keystone11:55
*** raildo has joined #openstack-keystone11:56
ybunkerHi all, I've a question regarding keystone (Juno) upgrade process,... i'm using keystone with swift, and someone at my team do an upgrade of the swift cluster from Juno to Rocky on the data nodes, and left the Proxy nodes with Queens release, so at this point we have:  keystone (Juno), swift-data (rocky) and swift-proxy (queens),.. keystone is using old token process, but i know that on rocky that keystone mechanism is deprecated and has11:58
ybunker been removed right?, so at this point I need to upgrade de Keystone from Juno to Rocky, is there a process doc to do this? any advice tips? Thanks in advance all11:58
*** sapd1 has quit IRC12:00
*** sapd1 has joined #openstack-keystone12:02
*** shyamb has quit IRC12:06
*** dave-mccowan has joined #openstack-keystone12:16
*** shyamb has joined #openstack-keystone12:19
fricklerso I made a patch in devstack dropping the creation of the admin endpoint, but it's still exploding big time. mostly because ksa still seems to default to using the admin interface. is there any plan to mitigate this? http://logs.openstack.org/92/651492/2/check/devstack/357b6f0/controller/logs/screen-g-api.txt.gz#_Apr_10_11_28_00_91507312:23
*** awalende has quit IRC12:26
*** awalende has joined #openstack-keystone12:26
*** awalende has quit IRC12:31
*** lbragstad has joined #openstack-keystone12:58
*** ChanServ sets mode: +o lbragstad12:58
*** irclogbot_1 has joined #openstack-keystone13:03
*** altlogbot_2 has joined #openstack-keystone13:07
*** dklyle has joined #openstack-keystone13:17
*** mvkr has quit IRC13:19
*** shyamb has quit IRC13:26
ybunkeranyone?13:34
*** phasespace has quit IRC13:36
lbragstadybunker what token format are you using?13:52
*** mvkr has joined #openstack-keystone13:53
ybunkerlbragstad:  PKI token13:54
*** vishakha has quit IRC13:54
lbragstadand you're upgrading to?13:55
lbragstadQueens or Rocky?13:55
ybunkerlbragstad: Rocky13:55
lbragstadok - the only token provider available in that release is Fernet13:56
lbragstadhttps://git.openstack.org/cgit/openstack/keystone/tree/setup.cfg?h=stable/rocky#n14613:56
lbragstadwhen you mentioned "old token process" in your original ping, were you referencing token providers or something else? http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-keystone.2019-04-10.log.html#t2019-04-10T11:58:4813:56
ybunkerlbragstad: got it, and is possible to move from Juno (PKI) to Rocky (Fernet) is 'minimal' disruption?13:56
ybunkerlbragstad:  provider=keystone.token.providers.pkiz.Provider13:58
lbragstadok - i just wanted to make sure we were both referencing the same thing13:58
lbragstadare you familiar with Fernet?13:59
ybunkerlbragstad: not at all :-(14:00
lbragstadok - that's fine14:00
lbragstadwe have documentation that introduces the idea and describes how to migrate from old token providers to fernet14:01
lbragstadhttps://www.lbragstad.com/blog/what-you-need-to-know-about-keystones-new-default-token-format14:01
*** mchlumsky has joined #openstack-keystone14:01
lbragstadkeep in mind, that post references ocata a bunch because we made Fernet the default token provider that release, but it should be relevant to your situation if you're upgrading to rocky14:01
ybunkerlbragstad: oh nice, thanks :-) let me take a look on that14:02
lbragstadone you get through that - https://docs.openstack.org/keystone/latest/admin/fernet-token-faq.html will be easier to parse14:03
ybunkerlbragstad: thanks, also i was hopping to approach with the upg, first creating newly VMs with keystone rocky (setup fernet key repo), take a dump backup of the actual keystone-db, apply that backup on keystone-rocky VMs and run the db_sync.. is that even possible?14:05
*** phasespace has joined #openstack-keystone14:05
ybunkerlbragstad: since i dont want to make an in-place upgrade on the actual keystone VM14:06
lbragstadwell - one big difference is that fernet tokens aren't persisted14:12
lbragstadso they're not actually in the database at all14:12
*** openstackgerrit has quit IRC14:14
lbragstadybunker describes some of that, too https://www.youtube.com/watch?v=702SRZHdNW814:17
*** erus has joined #openstack-keystone14:18
lbragstadi'm not sure if you have separate database instances, but you could point your rocky keystone nodes to the database and they should work fine14:21
lbragstadgranted... you're PKI tokens aren't going to be validateable on rocky nodes, and fernet tokens aren't going to be validateable on your juno nodes14:22
ybunkerlbragstad: oh i see, let me check the video, thanks!14:25
*** awalende has joined #openstack-keystone14:27
*** sapd1 has quit IRC14:27
*** sapd1 has joined #openstack-keystone14:27
*** awalende has quit IRC14:31
gagehugoo/14:37
eruso/14:45
lbragstadybunker yep - let us know if you have additional questions14:46
ybunkerlbragstad: thanks a lot :-) I really appreciate your help with this14:47
lbragstadanytime!14:47
cmurphyo/14:57
*** sapd1 has quit IRC15:09
*** sapd1 has joined #openstack-keystone15:09
*** dave-mccowan has quit IRC15:23
*** gyee has joined #openstack-keystone15:26
mnaserhmm15:26
mnaseris it possible that as part of the upgrade to stein, user:admin/role:admin gets scope:system15:27
mnaserso it looks like the user:admin gets the role:admin with system scope = all15:28
mnaserhowever it looks like a few groups we had defined that has role:admin on project:admin didn't get those so im trying to figure out where that happens (and if it should happen or not)15:29
cmurphymnaser: did you re-run keystone-manage bootstrap as part of the upgrade?15:29
mnasercmurphy: yeah, OSA runs bootstrap on every run15:29
cmurphythat will do it15:29
mnasercmurphy: I see, did I maybe miss a release note that said 'make sure you update your roles accordingly'?15:31
cmurphymnaser: the admin user should still have the admin role on the admin project15:32
cmurphyyou shouldn't have to update anything15:32
*** jamesmcarthur has joined #openstack-keystone15:32
*** starborn has joined #openstack-keystone15:33
mnasercmurphy: right, okay, I guess maybe this is a horizon bug, I have some users that can't access certain portions of horizon with some super cryptic traceback that doesn't give any info at all15:34
mnaseronly difference between them and the admin user was that the admin user has system_scope:all role:admin setup... but I just did that and it's still oddly not showing things up15:35
* mnaser goes back to dealing with js15:35
mnaserthanks15:35
cmurphymnaser: hmm okay, let us know if it starts looking more like a keystone issue again15:35
mnasercmurphy: will do.. this is stein so it could be something no one has ever ran into as well, the small price to pay :P15:36
cmurphymnaser: thanks for being our guinea pig :)15:36
mnaser:>15:37
*** shyamb has joined #openstack-keystone15:44
*** jamesmcarthur has quit IRC16:06
*** jamesmcarthur has joined #openstack-keystone16:08
knikollao/16:13
erushi knikolla16:18
knikollahey erus :)16:18
erushow are you? :)16:18
*** sapd1 has quit IRC16:32
*** sapd1 has joined #openstack-keystone16:32
*** jdennis has quit IRC16:35
*** jdennis has joined #openstack-keystone16:36
*** shyamb has quit IRC16:38
*** altlogbot_2 has quit IRC16:45
*** phasespace has quit IRC16:47
knikollaerus: i'm good, how are you?17:06
knikollacmurphy: the federated_domain_name conf option doesn't do anything anymore since we have per idp domains17:06
*** openstackgerrit has joined #openstack-keystone17:06
openstackgerritColleen Murphy proposed openstack/keystone master: Fix werkzeug imports for version 0.15.x  https://review.openstack.org/65143017:06
cmurphyknikolla: sorry, context?17:08
knikollacmurphy: https://bugs.launchpad.net/keystone/+bug/175404817:09
openstackLaunchpad bug 1754048 in OpenStack Identity (keystone) "Federated domain is reported when validating a federated token" [Medium,Triaged]17:09
*** mvkr has quit IRC17:09
knikollai'm assuming we need to go through the normal deprecation cycle regardless?17:10
cmurphyi think so, just to warn people that they can take it out of their config17:10
openstackgerritKristi Nikolla proposed openstack/keystone master: Deprecate [federation] federated_domain_name  https://review.openstack.org/65161417:23
knikolla^^17:24
knikollacmurphy: fix-werkzeug-for-real, lol17:28
knikollareminds me of most commit message in most repos17:28
*** sapd1 has quit IRC17:29
*** sapd1 has joined #openstack-keystone17:30
cmurphyi had another local branch that was fix-werkzeug where we just pinned it17:33
cmurphyknikolla: do you have any advice for this https://bugs.launchpad.net/keystone/+bug/182384717:34
openstackLaunchpad bug 1823847 in OpenStack Identity (keystone) "Multiple rules in a mapping is not working with type: "local" attribute" [Undecided,Invalid]17:34
knikollacmurphy: hmmm... i guess the only way to handle this with a non breaking change is to have a different type "local_if_exists"17:37
cmurphythat's what i was thinking :/17:38
cmurphyhate to add more complexity to the mapping api though17:39
knikollacmurphy: a better approach would be https://bugs.launchpad.net/keystone/+bug/181607617:40
openstackLaunchpad bug 1816076 in OpenStack Identity (keystone) "RFE: Extend user API to support federated attributes" [Wishlist,In progress]17:40
knikollathis way they would be able to create ephemeral users ahead of time and just map to them ephemerally17:40
knikollaif they don't exist they'll be created17:41
knikollaif they do, they'll be mapped17:41
knikollai was thinking of taking that on for train17:41
knikollabut just for the ephemeral user creation, none of the other list/update/etc features ron had planned17:41
cmurphythat would be awesome17:42
*** jamesmcarthur has quit IRC17:42
knikollai'll brush up the spec today17:43
*** sapd1 has quit IRC18:00
*** sapd1 has joined #openstack-keystone18:00
*** awalende has joined #openstack-keystone18:11
*** jamesmcarthur has joined #openstack-keystone18:12
*** awalende has quit IRC18:15
erusknikolla i'm good :)18:37
*** canori01 has joined #openstack-keystone18:49
canori01Hello, is it possible to allow non-admin users to create projects via policy.json?18:50
-openstackstatus- NOTICE: Restarting Gerrit on review.openstack.org to pick up new configuration for the replication plugin19:05
*** ybunker has quit IRC19:15
*** erus has quit IRC19:19
lbragstadcanori01 it depends on if you have a custom policy file, some of the defaults in policy.v3cloudsample.json attempt to do that but we're actually fixing those issues directly in keystone19:20
lbragstadfixes for those resources (projects being one of them) will be available in Stein19:21
canori01lbragstad: I don't curerntly have a policy.json. I was looking at policy.v3cloudsample.json. Is there somewhere I could see what the default rules are?19:24
canori01I'm running queens and just taking defaults I guess since my policy.json is blank19:25
lbragstadcorrect19:25
lbragstadyou can generate the defaults from code using ``oslopolicy-sample-generator --namespace keystone``19:25
canori01thank you19:26
*** awalende has joined #openstack-keystone19:28
*** erus has joined #openstack-keystone19:34
*** nkinder has joined #openstack-keystone19:57
nkindercmurphy, let me know if my comment here addresses your question - https://review.openstack.org/#/c/649177/6/keystone/tests/unit/test_backend_ldap.py19:58
*** jamesmcarthur has quit IRC19:59
cmurphynkinder: i think so, still need to stare at it for a bit20:03
cmurphyalso added gyee as reviewer20:03
*** starborn has quit IRC20:04
*** sapd1 has quit IRC20:10
*** sapd1 has joined #openstack-keystone20:16
*** awalende has quit IRC20:19
*** awalende has joined #openstack-keystone20:19
*** awalende has quit IRC20:24
*** sapd1 has quit IRC20:34
*** phasespace has joined #openstack-keystone20:36
*** sapd1 has joined #openstack-keystone20:37
openstackgerriterus proposed openstack/keystone master: Add new attribute to the federation protocol API  https://review.openstack.org/63730520:46
*** mvkr has joined #openstack-keystone20:52
openstackgerriterus proposed openstack/keystone master: Add new attribute to the federation protocol API  https://review.openstack.org/63730520:53
openstackgerriterus proposed openstack/keystone master: Add new attribute to the federation protocol API  https://review.openstack.org/63730520:59
*** raildo has quit IRC21:06
*** sapd1 has quit IRC21:14
*** sapd1 has joined #openstack-keystone21:19
openstackgerritayoung proposed openstack/keystone master: Predictable IDs for Roles  https://review.openstack.org/65165521:44
openstackgerriterus proposed openstack/keystone master: Add new attribute to the federation protocol API  https://review.openstack.org/63730521:46
*** sapd1 has quit IRC21:50
*** sapd1 has joined #openstack-keystone21:54
*** erus has quit IRC22:01
*** erus has joined #openstack-keystone22:03
*** rcernin has joined #openstack-keystone22:06
*** sapd1 has quit IRC22:41
*** sapd1 has joined #openstack-keystone22:42
*** erus has quit IRC22:44
*** whoami-rajat has quit IRC22:51
*** tkajinam has joined #openstack-keystone22:53
*** sapd1 has quit IRC22:55
*** sapd1 has joined #openstack-keystone22:58
cmurphylbragstad: http://git.openstack.org/cgit/openstack-infra/reviewstats/23:03
lbragstadoh - nice, it supports that?23:03
lbragstader - collecting all reviews for a given release?23:03
cmurphyi couldn't get it to do per release but i used the number of days since stein opened23:04
cmurphyso the number i ended up with is across all branches23:04
cmurphywhich i think is fine, it's still activity that happened since the last time we gave an update23:04
lbragstadright - it's close enough23:10
*** sapd1 has quit IRC23:23
*** sapd1 has joined #openstack-keystone23:24
*** david-lyle has joined #openstack-keystone23:36
*** dklyle has quit IRC23:36
*** david-lyle has quit IRC23:46
*** sapd1 has quit IRC23:48
*** sapd1 has joined #openstack-keystone23:50
*** rcernin has quit IRC23:52

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!