Friday, 2019-08-23

*** markvoelker has joined #openstack-keystone00:00
*** markvoelker has quit IRC00:05
*** dtruong has quit IRC00:07
*** dtruong has joined #openstack-keystone00:07
*** markvoelker has joined #openstack-keystone00:22
*** gyee has quit IRC00:36
*** markvoelker has quit IRC00:47
*** spatel has joined #openstack-keystone01:23
*** dave-mccowan has joined #openstack-keystone02:35
*** markvoelker has joined #openstack-keystone02:40
*** markvoelker has quit IRC02:46
*** markvoelker has joined #openstack-keystone03:20
*** markvoelker has quit IRC03:25
*** markvoelker has joined #openstack-keystone04:40
*** markvoelker has quit IRC04:45
*** jaosorior has joined #openstack-keystone05:01
*** spatel has quit IRC05:15
*** dave-mccowan has quit IRC05:16
*** adriant has joined #openstack-keystone05:23
*** markvoelker has joined #openstack-keystone06:40
*** markvoelker has quit IRC06:45
*** shyamb has joined #openstack-keystone06:49
*** dancn has joined #openstack-keystone06:52
*** jawad_axd has joined #openstack-keystone07:07
*** takamatsu has joined #openstack-keystone07:08
*** rcernin has quit IRC07:15
*** trident has quit IRC07:25
*** trident has joined #openstack-keystone07:31
*** sapd1_ has joined #openstack-keystone07:33
*** xek has joined #openstack-keystone07:37
*** sapd1 has quit IRC07:37
*** dancn has quit IRC07:39
*** dancn has joined #openstack-keystone07:45
*** shyamb has quit IRC07:51
*** tkajinam has quit IRC08:19
mordredkmalloc, cmurphy: I can get behind a keystoneauth2 in U. I could also get behind some minor cleanups of transitional stuff. I feel like there's one or two parameter defaults we've wanted to change but can't for backwards compat reasons. might make sense to take a pass through, make an etherpad of things we think we'd like to change in a 2 and see which ones we're comfortable with08:19
*** ivve has joined #openstack-keystone08:26
*** dancn has quit IRC08:28
*** dancn has joined #openstack-keystone08:34
*** shyamb has joined #openstack-keystone08:43
*** shyamb has quit IRC08:56
*** shyamb has joined #openstack-keystone09:04
*** dancn has quit IRC09:31
*** dancn has joined #openstack-keystone09:36
*** shyamb has quit IRC09:57
*** jaosorior has quit IRC10:10
*** dancn has quit IRC10:11
*** shyamb has joined #openstack-keystone10:22
*** jaosorior has joined #openstack-keystone10:27
*** shyamb has quit IRC10:44
*** shyamb has joined #openstack-keystone11:04
openstackgerritMerged openstack/keystone master: Fix translated response  https://review.opendev.org/67711711:10
*** tesseract has joined #openstack-keystone11:12
*** jaosorior has quit IRC11:26
*** jaosorior has joined #openstack-keystone11:43
*** markvoelker has joined #openstack-keystone11:57
*** dave-mccowan has joined #openstack-keystone12:06
*** shyamb has quit IRC12:21
*** shyamb has joined #openstack-keystone12:25
*** shyamb has quit IRC12:41
*** spatel has joined #openstack-keystone12:43
*** spatel has quit IRC12:48
*** jaosorior has quit IRC13:14
*** spatel has joined #openstack-keystone13:14
*** spatel has quit IRC13:17
*** ivve has quit IRC13:24
*** lbragstad_ has joined #openstack-keystone13:24
*** lbragstad has quit IRC13:25
knikollao/13:29
lbragstad_o/13:31
*** lbragstad_ is now known as lbragstad13:31
*** bnemec has joined #openstack-keystone13:34
*** gagehugo has quit IRC13:34
*** bnemec is now known as beekneemech13:35
*** jawad_axd has quit IRC13:49
*** jawad_axd has joined #openstack-keystone13:53
*** jawad_axd has quit IRC13:58
lbragstadkmalloc not sure if you saw the updates to https://review.opendev.org/#/c/676648/3 yet - but it doesn't sound like the approach we talked about yseterday will work14:18
*** soumith_ has joined #openstack-keystone14:18
soumith_Hello all, I am unable reach cloud when I source rc file and get the status of servers,images,etc... I get the error as "Failed to discover available identity versions when contacting... Unable to establish connection to http://192.168.100.3:5000/v3/auth/tokens: ('Connection aborted.', BadStatusLine('No status line received - the server has closed the connection',))14:23
soumith_". Can anyone help me know why this issue is raised? Thank you!14:23
*** jawad_axd has joined #openstack-keystone14:46
*** jawad_axd has quit IRC14:51
*** xek has quit IRC14:58
*** cmurphy is now known as cmorpheus15:09
*** jawad_axd has joined #openstack-keystone15:11
*** jawad_axd has quit IRC15:16
*** dave-mccowan has quit IRC15:20
kmalloclbragstad: i hadn't seen it yet15:21
*** gagehugo has joined #openstack-keystone15:24
*** soumith_ has quit IRC15:28
*** gyee has joined #openstack-keystone15:46
*** jamesmcarthur has joined #openstack-keystone15:47
openstackgerritMorgan Fainberg proposed openstack/keystoneauth master: Allow initializing session with connection retries  https://review.opendev.org/67664815:55
lbragstadfwiw - i'm trying to recreate ^ locally15:59
kmalloclbragstad, cmorpheus ^ see the changes here. fixed the concerns i had with the tests and fixed the releasenote. If everyone is fine with this, we can roll with it as is. if we would prefer the call-args default (vs. in-built session default), I'll quickly spin that up15:59
kmalloclbragstad: i've been unable to recreate it locally no matter how hard I try15:59
kmalloclbragstad: but i don't doubt it occurs.15:59
lbragstadhmm16:00
*** lbragstad is now known as elbragstad16:00
*** kmalloc is now known as needscoffee16:00
*** needscoffee is now known as needsSoMuchMoreC16:00
*** needsSoMuchMoreC is now known as needscoffee16:00
needscoffeebah16:00
needscoffeeanyway16:00
*** AJaeger has left #openstack-keystone16:01
elbragstadi'm trying to get this to timeout using get_endpoint() with the ksa adapter, per one of the comments16:06
elbragstadno luck so far16:06
elbragstadnot sure if i'm doing something wrong - but this is what i'm doing to try and recreate this https://pasted.tech/pastes/2fd7d369468c40fd831bef830c27467f208eea7a.raw16:07
*** beekneemech has quit IRC16:07
needscoffeehm, yeah i think it's a case where it's keystone being slightly overloaded and blocking a connection waiting on DB or something16:08
needscoffeeor an endpoint being slow to respond for the version discovery16:08
needscoffeein general16:08
needscoffeebut in all seriousness, i can't duplicate it16:08
needscoffeemaybe run a keystone with a single thread/worker and have a bunch of these hammer it?16:09
*** tesseract has quit IRC16:11
elbragstadok - trying that now16:20
*** bnemec has joined #openstack-keystone16:20
needscoffeecoreycb: responded to your email. I expect I'll be able to help more directly next week on unblocking the patch16:29
needscoffeecoreycb: the comment was definitely not directed at you.16:29
needscoffeecoreycb: unless the original submitter can address the concerns.16:29
coreycbneedscoffee: thanks. ok well either way i think it's a good point. :)16:29
needscoffee^_^16:30
needscoffeecoreycb: i am sure you can see why i was frustrated there. I re-wrote the first email at least 10 times to try and not sound like I was too grumpy/unwilling to work with folks.16:31
needscoffeeanyway. have a great rest of your friday and i'll see about following up next week so we can get the py3 stuff unblocked16:32
coreycbneedscoffee: i completely side-tracked that email. thanks for clearing it up. and yes i completely get it. have a good weekend and thanks!16:34
*** markvoelker has quit IRC16:39
*** markvoelker has joined #openstack-keystone16:47
elbragstadneedscoffee i think i was able to recreate it?16:49
elbragstadi set the timeout on the session to be really short (0.1 seconds)16:49
needscoffeeah16:49
needscoffeeso it is a case where the server is somewhat overloaded16:50
elbragstadhmm16:50
elbragstadi mean - i guess that's what it's trying to simulate?16:50
cmorpheusneedscoffee: re 676648, do you believe it's a good idea to store connection retry settings in the session object where it is likely to be used for all services the session is used for? if so, why not also add all of the other retry and concurrency settings from the adapter to the session?16:51
elbragstadneedscoffee https://pasted.tech/pastes/ab7b4cea5fada59c63062a67dfd0a3957986a896.raw16:51
*** bnemec has quit IRC16:53
needscoffeecmorpheus: i really don't like it there.16:57
needscoffeecmorpheus: but i get that they can't really shove it in the adapter for all cases16:57
elbragstadso - i have some questions about that16:58
needscoffeebut overall, as long as the call args overrides the session-level set16:58
needscoffeei'm fine with allowing a session-wide set of retries16:58
elbragstadit sounds like heat needs to use a session because the data changes and can't be relied on/cached?16:58
needscoffeeand my new test shows it does, properly16:58
cmorpheuselbragstad: i think it's because they get trust tokens for different trust users16:59
cmorpheustrustor users*16:59
needscoffee^ that is my understanding16:59
elbragstadok - so they need a session because the use changes16:59
elbragstad?16:59
elbragstaduser*16:59
cmorpheusi'm still unclear on why they can't use an adapter tbh16:59
needscoffeeit16:59
needscoffee's why i've held off on +217:00
needscoffeebut the +1 is because the code should be fine as it sits17:00
elbragstadok - so the alternative you proposed cmorpheus was to have an adapter and reach into the session to adjust the auth data?17:01
needscoffeethat sounds... weird.17:01
cmorpheusthat was my proposal but they face the same issue because it goes through the session which doesn't have retries17:01
needscoffeelike breaking the session/adapter divide17:01
cmorpheusbecause get_access and get_auth_ref don't expose it17:01
elbragstadoh17:01
elbragstadwell - neutron actually uses that approach17:02
elbragstadhttps://opendev.org/openstack/python-neutronclient/src/commit/ab426a791ad1937ea2cf3b340202b3968a378978/neutronclient/client.py#L37917:02
cmorpheusit is a little weird17:04
needscoffeeyeau17:04
needscoffeei think it probably is an option that should go on session ... or maybe in the auth plugins?17:05
needscoffeei really want to encourage not just setting session-wide retries17:05
needscoffeeit seems blunt. where adapters are intended to do this.17:05
needscoffeeor even just allow get_auth_ref to handle a retry case.17:06
cmorpheusyeah i am trying to figure out whether it's more awkward to add it to the session or the auth plugin17:06
elbragstadi have another dumb question - it's not specific to retries from keystone?17:06
needscoffee++17:06
needscoffeeit is mostly keystone-specific, but it is in a path that is awkward to work with17:06
elbragstadthe traceback in the bug report is between neutron and nova i think17:07
elbragstadso - it's failing because it's using a trust-scoped token to do things with resources in a stack?17:07
cmorpheusthat was a weird traceback to use as an example because that actually is using the adapter17:07
cmorpheusthe actual issue in the bug report is about get_auth-ref17:07
elbragstadah17:07
cmorpheusthat was part of my confusion and why i withdrew from the discussion a bit on the review17:08
cmorpheusif it's applied to the session, and not overridden in session.request() by the adapter or anything else, then it will be used on all services17:09
*** jamesmcarthur has quit IRC17:11
*** jamesmcarthur has joined #openstack-keystone17:11
cmorpheuswhich could mean ksa itself will start hammering all of the services even more, and there's no rate semaphore in the session to control the retry delay17:13
elbragstadin that case, would a timeout for a nova call result in a retry for another service?17:14
needscoffeepossibly17:14
cmorpheusmm i don't think it would be that bad17:14
cmorpheusit would just keep retrying nova17:14
elbragstadthe retry logic is still just using the adapter retry lofic17:15
elbragstadlogic*17:15
elbragstadso - it would be service-specific(?)17:15
elbragstadit's just exposing it through the Session17:15
* elbragstad is thinking out loud 17:16
cmorpheusso - you get a session object, s = session.Session(auth=auth, connect_retries=10), all good - then do s.request(novaurl) - if nova is healthy then everything is fine, if the nova service is flaky then it will be retried and maybe eventually succeed, or maybe contribute to nova being overloaded17:19
cmorpheusother services wouldn't be involved, but my point was that if the reason you're setting connect_retries=10 is because you expect keystone to be flaky, but then as a side effect get retries for the nova service, that is kind of weird17:20
elbragstadhmm17:20
elbragstadin the original bug report or early review comments it sounded like keystone was timing out but there were plenty of keystone processes available to handle requests17:20
cmorpheusand gets you in the situation where if nova is actually down and you didn't expect it (and you can't set the retriable status codes because we haven't exposed that) then you're waiting for 10 retries for no reason17:20
elbragstadhttps://bugs.launchpad.net/keystoneauth/+bug/1840235/comments/217:21
openstackLaunchpad bug 1840235 in keystoneauth "[Errno 110] ETIMEDOUT with endpoint discovery and token request" [Undecided,In progress] - Assigned to Morgan Fainberg (mdrnstm)17:21
elbragstad"When updating large heat stacks with number of nested stacks, we create auth plugins and request number of trust tokens concurrently for every nested stack, and we see ETIMEDOUT errors[1], in spite of having large number of keystone workers. "17:22
*** jamesmcarthur has quit IRC17:22
elbragstadi guess now i'm curous what's causing the timeouts in that particular case if keystone isn't maxed out (and is assumed to be healthy enough to respond to incoming requests)17:22
cmorpheusthat's a good question17:23
elbragstadam i understanding the properly?17:23
cmorpheusi think you are17:24
elbragstadhmm17:27
elbragstadok - commented on the bug17:27
elbragstadi think i'm still missing some context around the heat flow17:28
*** zaneb has joined #openstack-keystone17:32
elbragstadoh - well there he is17:32
zanebelbragstad: this is probably a better place to discuss :)17:33
elbragstadi was just about to say that i was talking to zaneb in -dev but it looks like he made the move here for us ;)17:33
elbragstadzaneb agreed17:33
zanebso Rabi was working on it directly, I only know second-hand info17:33
zanebbut AIUI keystone was one of the things that was timing out17:34
elbragstadthat's fair - i was pinging him a bit yesterday, but i know our timezones don't really line up17:34
zanebat that scale everything starts to timeout some (small but usally fatal ;) portion of the time17:34
elbragstadthe wording in the bug report makes it seem like the initial timeout aren't related to the amount of keystone resources17:35
elbragstad"When updating large heat stacks with number of nested stacks, we create auth plugins and request number of trust tokens concurrently for every nested stack, and we see ETIMEDOUT errors[1], in spite of having large number of keystone workers."17:36
elbragstadso - just wondering if there was any insight into what's causing those if keystone is relatively healthy at that point17:36
elbragstador - should i be interpreting that as "even with a high number of keystone nodes, we still overload them with requests when updating tons of stacks"17:37
zanebelbragstad: yes, the last thing you said17:38
elbragstadok - i so i was misunderstanding it then17:38
elbragstadcool17:38
zanebwith s/nodes/workers/17:38
zanebbasically you only get one undercloud node to run everything on, and they scale the workload out until things start failing. then fix those things and scale it up again17:38
zanebright now what's failing is that even with plenty of workers, they've hit the limit and some requests are now timing out17:39
zanebwhich is fine; we just need graceful ways for clients to handle that17:39
elbragstadand https://review.opendev.org/#/c/678039/ does that for nearly all other clients heat uses17:40
zanebhttps://review.opendev.org/#/c/678193/ is also necessary for the keystone bits17:41
zanebI think we'll backport those two, because we can't backport a requirements change to bump keystoneauth anyway17:42
elbragstadyeah17:43
zanebbut it would be nice to have a way to specify retries on the keystone get token/catalog on master at least17:43
elbragstadso https://review.opendev.org/#/c/678193/3 fixes the issue, but just not with ksa directly17:43
zanebyeah. arguably it's a workaround ;)17:44
elbragstadspecifically so you can backport it, yeah?17:44
zanebright. we would drop that on master in favour of a fix in ksa if it were available17:45
* elbragstad nods17:46
elbragstadso retries are needed here because you're calling get_access https://review.opendev.org/#/c/678193/3/heat/common/context.py@30617:47
elbragstadand get_endpoint is susceptible to the same thing17:47
zanebthat's my understanding17:47
elbragstadhttps://review.opendev.org/#/c/678193/3/heat/engine/clients/client_plugin.py,unified@94 looks like keystone-specific code17:48
elbragstaddumb question: but if it's keystone-specific could it use an adapter?17:48
zanebit's in the base class for all of the client plugins. not sure if it's keystone-specific in the sense that you mean it17:50
elbragstadah17:50
* zaneb has a sketchy understanding of keystone auth stuff17:51
elbragstadat first glance, i was thinking that method was specific to only connecting to keystone, but i think the keystone-specific stuff i was assuming is only for parsing a service url out of a catalog17:51
elbragstadit looks like it could handle something like url_for(interface='public', service_name='nova')17:54
elbragstador whatnot17:54
zaneblooks like it17:55
elbragstadiiuc - it looks like the minimum to fix this in ksa would be to expose connection retries through get_access() and get_endpoint()17:56
elbragstadbut the current implementation does that via the session (covering both of those cases?)17:57
elbragstadcmorpheus needscoffee thoughts?17:57
needscoffeei'm more inclined to have it be explicit for get_access and get_endpoint, or to enhance to have explicit auth retries independent of the service communication18:01
needscoffeebut really, i'm kindof fine with whatever solution is selected, noting that none of these features can be backported18:01
needscoffeein general.18:02
*** cp has joined #openstack-keystone18:02
needscoffeemy general view is: give an auth-path retry that is session-specific and leaned on for all auth-bits unless overidden18:03
needscoffeesecond choice, session-global that can be overidden18:03
needscoffeei am worried that get_access and get_endpoint having separate retry values would be awful to use18:03
elbragstadyeah - that could be strange18:06
elbragstadif there are session-wide concerns, then option #1 makes sense?18:18
elbragstadbut i know the least about ksa out of everyone having this discussion, so i'll wait for others to weigh in ;)18:20
*** gyee has quit IRC18:42
*** gyee has joined #openstack-keystone18:46
*** xek has joined #openstack-keystone18:58
openstackgerritGage Hugo proposed openstack/keystonemiddleware master: Comment html_static_path entry in docs conf.py  https://review.opendev.org/67784019:04
*** hrybacki has joined #openstack-keystone19:05
*** bnemec has joined #openstack-keystone19:10
*** bnemec is now known as beekneemech19:11
cmorpheuseasy review https://review.opendev.org/67784019:23
*** jamesmcarthur has joined #openstack-keystone19:28
*** jamesmcarthur has quit IRC19:32
*** jamesmcarthur has joined #openstack-keystone19:32
*** jamesmcarthur has quit IRC19:43
openstackgerritMerged openstack/keystonemiddleware master: Comment html_static_path entry in docs conf.py  https://review.opendev.org/67784019:44
*** jamesmcarthur has joined #openstack-keystone19:45
*** jamesmcarthur has quit IRC19:50
*** ivve has joined #openstack-keystone20:03
*** jamesmcarthur has joined #openstack-keystone20:14
*** jamesmcarthur has quit IRC20:18
*** jamesmcarthur has joined #openstack-keystone20:19
needscoffeecmorpheus: I have a working migration and realized i needed to do some research on orm.relationship backrefs20:25
needscoffeecmorpheus: it's getting close to having a base implementation with a mixin.20:25
needscoffeecmorpheus: there is some additional wonky tooling around json_schema stuff, but should be straightforward.20:26
cmorpheusneedscoffee: o720:28
elbragstadi know we recently refreshed our service token docs21:13
elbragstadbut... do the steps in https://review.opendev.org/#/c/672145/2/doc/source/configuration/block-storage/service-token.rst@43 look accurate to everyone else?21:13
* elbragstad didn't think services needed to roll out their own configuration option in order get that functionality?21:14
cmorpheusi didn't think so either, but i know nova has similar instructions21:17
elbragstadoh - really?21:17
elbragstadi was under the impression that services would build clients with auth information from the [keystone_authtoken] section anyway21:18
cmorpheusyeah i am unclear on why they are not doing that21:25
* elbragstad grabs a copy of cinder21:25
*** xek has quit IRC21:25
elbragstadgrepping the cinder source i don't actually see that configuration used?21:26
elbragstadmaybe i should just head over there21:26
needscoffeeok i ran into a problem21:32
*** needscoffee is now known as kmalloc21:32
kmalloci guess i'm just going to create separate options table for each resource.21:33
kmalloctrying to be clever and do some join magic to load them with the ORM.relationship bit is just not working.21:33
kmalloccmorpheus: let me respin this, I'll just do roles for the moment so we can do immutable there.21:34
cmorpheuskmalloc: okay21:34
kmalloc*sigh*, this is going to mean we need to be very deliberate where we want resource options21:35
kmallocotherwise we get an explosion ot DB tables.21:35
kmallocof*21:35
kmallocso, roles... and anything else should be lumped in? project? domain?21:35
kmallocadding tables is the easy part here really.21:36
*** jamesmcarthur has quit IRC21:46
*** jamesmcarthur has joined #openstack-keystone21:48
*** jamesmcarthur has quit IRC21:50
*** jamesmcarthur has joined #openstack-keystone21:51
*** takamatsu has quit IRC21:55
*** jamesmcarthur has quit IRC22:07
*** jamesmcarthur has joined #openstack-keystone22:07
*** rcernin has joined #openstack-keystone22:11
*** jamesmcarthur has quit IRC22:13
cmorpheuskmalloc: roles, users, projects, and domains are what i'm concerned about for immutable22:15
*** ivve has quit IRC22:17
beekneemechelbragstad: Heh, I didn't realize that we're coworkers again. And for the same company as last time.22:21
beekneemechWhat a long, strange trip it's been. :-)22:21
*** jamesmcarthur has joined #openstack-keystone22:31
*** jamesmcarthur has quit IRC22:37
*** jamesmcarthur has joined #openstack-keystone22:39
*** jamesmcarthur has quit IRC22:46
*** jamesmcarthur has joined #openstack-keystone22:50
kmalloccmorpheus: almost have the Resource Options patch ready, running unit tests now.22:57
kmalloccmorpheus: looks like i have a little debugging to do, but it's pretty straightforward debugging.22:58
*** jamesmcarthur has quit IRC23:01
kmallocelbragstad: wow, i haven't run our whole unit tests on my X1C in a looong time.23:07
kmallocgah it's slow23:07
cmorpheusit's even longer if you have the opportunistic db tests turned on23:08
kmalloccmorpheus: yeah i don't think i have those enabled.23:08
kmallocbut when check_system_role is taking >10s to run23:08
kmallocwe have issues :(23:08
kmalloctest_user_can_revoke_project_scoped_token should not take 10seconds to run23:09
*** rcernin has quit IRC23:11
*** rcernin has joined #openstack-keystone23:12
*** jamesmcarthur has joined #openstack-keystone23:15
*** jamesmcarthur has quit IRC23:17
*** jamesmcarthur has joined #openstack-keystone23:19
kmallocwow...23:20
kmallocSum of execute time for each test: 10104.3671 sec.23:20
kmalloccmorpheus: ^ is that... consistent with what you see?23:20
kmallocthat seems... excessive23:20
kmallocmakes me wondr if my laptop is having issues23:21
*** jamesmcarthur has quit IRC23:22
*** jamesmcarthur has joined #openstack-keystone23:22
* beekneemech does the math23:26
*** beekneemech is now known as keanu23:26
keanuwhoa23:26
*** keanu is now known as beekneemech23:27
cmorpheuskmalloc: not sure because i never run them serially23:32
*** jamesmcarthur has quit IRC23:32
cmorpheuswithout the opportunistic tests it's around 15-20 minutes for me with tox23:32
cmorpheusas the ci has been showing it can be upwards of 40 minutes with the opportunistic tests on a slow node23:32
*** jamesmcarthur has joined #openstack-keystone23:38
*** jamesmcarthur has quit IRC23:42
kmallocouch23:49
kmallochmm...23:51
kmalloci think my tests took a lot longer23:51
kmallocok i'm posting the change it is not passing tests yet, have a couple things to fix.23:55
kmallocbut just so it has some eyes. this does not implemnet immutable yet23:56
openstackgerritMorgan Fainberg proposed openstack/keystone master: Implement resource options for roles and projects  https://review.opendev.org/67832223:56

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!