| *** cshen has quit IRC | 00:03 | |
| *** btully has joined #openstack-horizon | 00:24 | |
| *** btully has quit IRC | 00:29 | |
| *** gyee has quit IRC | 00:36 | |
| *** cshen has joined #openstack-horizon | 01:59 | |
| openstackgerrit | pengyuesheng proposed openstack/horizon master: Name is outside the page range https://review.openstack.org/614382 | 02:02 |
|---|---|---|
| *** cshen has quit IRC | 02:04 | |
| *** itlinux has joined #openstack-horizon | 02:45 | |
| *** itlinux has quit IRC | 02:56 | |
| *** btully has joined #openstack-horizon | 04:02 | |
| *** btully has quit IRC | 04:07 | |
| *** itlinux has joined #openstack-horizon | 05:05 | |
| *** pcaruana|elisa| has joined #openstack-horizon | 05:29 | |
| *** pcaruana|elisa| has quit IRC | 05:37 | |
| *** btully has joined #openstack-horizon | 05:51 | |
| *** btully has quit IRC | 05:56 | |
| *** itlinux has quit IRC | 06:04 | |
| *** e0ne has joined #openstack-horizon | 06:16 | |
| *** ChanServ sets mode: +o e0ne | 06:16 | |
| *** e0ne has quit IRC | 06:18 | |
| *** btully has joined #openstack-horizon | 07:40 | |
| *** btully has quit IRC | 07:44 | |
| *** pcaruana|elisa| has joined #openstack-horizon | 07:45 | |
| *** rdopiera has joined #openstack-horizon | 08:02 | |
| *** e0ne has joined #openstack-horizon | 08:12 | |
| *** ChanServ sets mode: +o e0ne | 08:12 | |
| *** jtomasek has joined #openstack-horizon | 08:18 | |
| *** jpich has joined #openstack-horizon | 08:28 | |
| *** cshen has joined #openstack-horizon | 08:43 | |
| *** pcaruana|elisa| has quit IRC | 09:13 | |
| *** pcaruana|elisa| has joined #openstack-horizon | 09:15 | |
| *** e0ne has quit IRC | 09:16 | |
| *** e0ne has joined #openstack-horizon | 09:25 | |
| *** ChanServ sets mode: +o e0ne | 09:25 | |
| openstackgerrit | Merged openstack/horizon master: Remove setup.py check from pep8 job https://review.openstack.org/614253 | 09:39 |
| *** lennyb has quit IRC | 09:48 | |
| *** lennyb has joined #openstack-horizon | 09:49 | |
| *** ianychoi has quit IRC | 09:51 | |
| *** ianychoi has joined #openstack-horizon | 09:52 | |
| openstackgerrit | Damon Li proposed openstack/horizon master: Add status update for image after creation operation https://review.openstack.org/614459 | 10:24 |
| *** mvkr has quit IRC | 10:25 | |
| *** e0ne has quit IRC | 11:15 | |
| *** e0ne has joined #openstack-horizon | 11:15 | |
| *** ChanServ sets mode: +o e0ne | 11:15 | |
| *** jtomasek has quit IRC | 11:16 | |
| *** e0ne has quit IRC | 11:27 | |
| *** cshen has quit IRC | 11:30 | |
| *** jtomasek has joined #openstack-horizon | 11:38 | |
| *** mvkr has joined #openstack-horizon | 11:40 | |
| *** cshen has joined #openstack-horizon | 11:46 | |
| openstackgerrit | Radomir Dopieralski proposed openstack/horizon master: Properly calculate auth_url for WEBSSO from POST data https://review.openstack.org/611387 | 12:28 |
| *** cshen has quit IRC | 12:30 | |
| *** cshen has joined #openstack-horizon | 12:31 | |
| *** cshen has quit IRC | 12:36 | |
| *** cshen has joined #openstack-horizon | 12:39 | |
| *** e0ne has joined #openstack-horizon | 13:06 | |
| *** ChanServ sets mode: +o e0ne | 13:06 | |
| *** btully has joined #openstack-horizon | 13:07 | |
| *** v12aml has quit IRC | 13:10 | |
| *** lennyb has quit IRC | 13:10 | |
| *** btully has quit IRC | 13:12 | |
| *** lennyb has joined #openstack-horizon | 13:12 | |
| openstackgerrit | Ivan Kolodyazhny proposed openstack/tempest-horizon master: Remove deprecated and unused 'login_url' config option https://review.openstack.org/614501 | 13:29 |
| openstackgerrit | Ivan Kolodyazhny proposed openstack/tempest-horizon master: Remove deprecated and unused 'login_url' config option https://review.openstack.org/614501 | 13:32 |
| openstackgerrit | Radomir Dopieralski proposed openstack/horizon master: Properly calculate auth_url for WEBSSO from POST data https://review.openstack.org/611387 | 13:49 |
| openstackgerrit | Merged openstack/tempest-horizon master: Cleanup requirements.txt https://review.openstack.org/591234 | 14:08 |
| *** lblanchard has joined #openstack-horizon | 14:14 | |
| *** rdopiera has quit IRC | 14:15 | |
| openstackgerrit | Merged openstack/tempest-horizon master: fix tox python3 overrides https://review.openstack.org/606569 | 14:17 |
| *** e0ne has quit IRC | 14:19 | |
| *** e0ne has joined #openstack-horizon | 14:26 | |
| *** ChanServ sets mode: +o e0ne | 14:26 | |
| *** rdopiera has joined #openstack-horizon | 14:28 | |
| *** dayou has quit IRC | 14:36 | |
| *** e0ne has quit IRC | 14:41 | |
| *** e0ne has joined #openstack-horizon | 14:44 | |
| *** ChanServ sets mode: +o e0ne | 14:44 | |
| *** Leo_m has joined #openstack-horizon | 14:52 | |
| *** btully has joined #openstack-horizon | 14:55 | |
| *** ianychoi has quit IRC | 15:00 | |
| *** btully has quit IRC | 15:00 | |
| *** gyee has joined #openstack-horizon | 15:01 | |
| e0ne | hi. is anybody going to joint horizon meeting? | 15:01 |
| *** dayou has joined #openstack-horizon | 15:25 | |
| *** itlinux has joined #openstack-horizon | 15:47 | |
| *** cshen has quit IRC | 16:12 | |
| *** dnguyen has joined #openstack-horizon | 16:21 | |
| *** pcaruana|elisa| has quit IRC | 16:37 | |
| *** gyee has quit IRC | 16:50 | |
| *** jpich has quit IRC | 16:59 | |
| *** e0ne has quit IRC | 17:08 | |
| mgagne | I have a weird behavior with Horizon Queens. When restarting Apache, the first ever request always fails to login. Or if user is already logged and refresh, session is lost and login page shows. But for the later, if you try to access any page in Horizon, the session is in fact still active. Also if you are logged, restart Apache, start a new session in a different browser, it fails. But the other user that was already logged won't lose | 17:20 |
| mgagne | its session. | 17:20 |
| mgagne | Does it ring any bell for anyone? | 17:20 |
| mgagne | I'm using the cache backend with memcache as the session engine. | 17:20 |
| mgagne | I'm currently using mod_wsgi with processes=2 and threads=1. If I increase processes to 20, I'm having a LOT more issues with lost sessions. | 17:25 |
| *** Leo_m has quit IRC | 17:29 | |
| *** Leo_m has joined #openstack-horizon | 17:43 | |
| *** gyee has joined #openstack-horizon | 17:44 | |
| *** mvkr has quit IRC | 17:49 | |
| *** rdopiera has quit IRC | 18:04 | |
| *** cshen has joined #openstack-horizon | 18:07 | |
| *** cshen has quit IRC | 18:20 | |
| *** cshen has joined #openstack-horizon | 18:22 | |
| *** cshen has quit IRC | 18:26 | |
| *** e0ne has joined #openstack-horizon | 18:31 | |
| *** ChanServ sets mode: +o e0ne | 18:31 | |
| *** e0ne has quit IRC | 18:32 | |
| *** btully has joined #openstack-horizon | 18:33 | |
| *** mvkr has joined #openstack-horizon | 18:36 | |
| *** cshen has joined #openstack-horizon | 18:36 | |
| *** btully has quit IRC | 18:38 | |
| *** e0ne has joined #openstack-horizon | 19:44 | |
| *** ChanServ sets mode: +o e0ne | 19:44 | |
| *** betherly has joined #openstack-horizon | 19:45 | |
| *** e0ne has quit IRC | 19:45 | |
| *** betherly has quit IRC | 19:49 | |
| *** rdopiera has joined #openstack-horizon | 20:17 | |
| *** btully has joined #openstack-horizon | 20:22 | |
| *** rdopiera has quit IRC | 20:24 | |
| *** rdopiera has joined #openstack-horizon | 20:24 | |
| *** btully has quit IRC | 20:27 | |
| *** e0ne has joined #openstack-horizon | 20:30 | |
| *** ChanServ sets mode: +o e0ne | 20:30 | |
| *** e0ne has quit IRC | 20:50 | |
| *** lblanchard has quit IRC | 21:03 | |
| *** betherly has joined #openstack-horizon | 21:27 | |
| *** betherly has quit IRC | 21:32 | |
| adriant | mgagne: yes! We get this error and it is annoying. When horizon loads up first for some reason it can't process old sessions right away | 21:42 |
| mgagne | adriant: plz tell me more =) | 21:43 |
| adriant | we had this error because uwsgi was setup to kill/restart workers and sometimes on such a restart users would get "logged out" | 21:43 |
| adriant | but if you waited and refreshed the page you'd be fine | 21:44 |
| mgagne | adriant: is there a solution to that? | 21:44 |
| adriant | mgagne: we never got to the bottom of it, but if others are hitting it as well we should try and get the horizon team to look at it | 21:44 |
| adriant | mgagne: yes... don't restart horizon, or don't allow it to serve requests until it is fully alive. | 21:45 |
| mgagne | how can I make sure it's fully alive? | 21:45 |
| adriant | alternatively, and I haven't looked into this yet, it may be an issue session handling. What session handling are you using? | 21:45 |
| mgagne | I'm not sure what config this refers to but I'm using the cache session engine with memcached as cache backend. | 21:46 |
| adriant | we just take it out of loadbalancer when restarting for some redefined period that seemed to be safe | 21:46 |
| adriant | cool, same as us | 21:46 |
| adriant | ok, what memcached library? | 21:46 |
| mgagne | python-memcached | 21:47 |
| adriant | again, same as us... try switching to PyLibMCCache and pylibmc (which needs an underlying c libary) | 21:47 |
| adriant | If you can switch that is | 21:47 |
| mgagne | I'm open to alternative at this point if it fixes the issue | 21:48 |
| adriant | python-memcached is actually unmaintained and I'm working with upstream Django to replace it with an alternative pymemcache | 21:48 |
| adriant | Just be aware, this probably isn't a fix, I'm just speculating. python-memcached doesn't load up right away, it talks to memcached only when the first request comes in, while pylibmc NEEDS a memcached server setup on startup. | 21:49 |
| adriant | So I'm jumping to a running conclusion that the differences in the libraries may cause some oddness. If that proves not to be the case, then it's a problem in Horizon. | 21:50 |
| mgagne | ok, I can try to switch to a file based backend and see if it fixes the session issue. I will rebuild with pylibmc later | 21:51 |
| mgagne | filebased looks to be the same | 21:53 |
| adriant | bah | 21:53 |
| adriant | I'm just spinning up a docker env I've got and telling gunicorn to kill/restart workers for horizon every 5 requests, to see if I can replicate the issue, and then switch memcached library | 21:54 |
| *** itlinux has quit IRC | 21:57 | |
| adriant | mgagne: yeah, same problem | 21:58 |
| adriant | takes me back to login page and I see: | 21:58 |
| adriant | You do not have permission to access the resource: | 21:58 |
| adriant | Login as different user or go back to home page | 21:58 |
| mgagne | yes | 21:58 |
| mgagne | sometimes you are still considered logged openstack_auth.user is_authenticated=True | 21:59 |
| mgagne | but I guess something raises NotAuthorized somewhere so you get redirected | 22:00 |
| adriant | yeah | 22:00 |
| adriant | it's like it hasn't been able to verify your session fully | 22:01 |
| mgagne | in fact | 22:01 |
| mgagne | sometimes, is_authenticated is True but NotAuthenticated is raised | 22:01 |
| adriant | either it just redirects you and doesn't give you that message (we see that), or you get that message | 22:01 |
| mgagne | I'm currently trying to trace in horizon/middleware/base.py process_exception | 22:01 |
| adriant | it's as if some parts of horizon aren't quite ready to start processing requests but it is anyway | 22:02 |
| adriant | mgagne: if you find anything, I'll glady help/review, and if you open a bug for it, I can add some notes to it as well | 22:06 |
| mgagne | just trying to find the source of the problem atm ;) | 22:06 |
| adriant | mgagne: this is the message I see before being kicked out: https://github.com/openstack/horizon/blob/master/horizon/static/framework/framework.module.js#L132 | 22:11 |
| adriant | so we're hitting: https://github.com/openstack/horizon/blob/master/horizon/static/framework/framework.module.js#L147 | 22:11 |
| adriant | and it seems to either redirect us, or actually sometimes force a logout | 22:12 |
| adriant | the question is, why is that js being triggered | 22:12 |
| adriant | and why only during start up | 22:12 |
| mgagne | I don't see any much on my side. did you see it in the developer console? | 22:13 |
| adriant | I think it may depend on the page you're on... that looks like angular error handling. I think this is still a symptom | 22:14 |
| mgagne | I'm getting a server side redirection on my side (302) | 22:14 |
| mgagne | Im testing with /horizon/project/api_access/ | 22:15 |
| adriant | I get a notification (very briefly) from horizon in the top right corner | 22:15 |
| adriant | let me try with the same as well | 22:15 |
| mgagne | which shouldn't have any angular | 22:15 |
| mgagne | and I'm really getting horizon.exceptions.NotAuthenticated | 22:15 |
| adriant | yep there we go, 'logged out' and no error in the top left on api_access | 22:16 |
| mgagne | I'm not sure how to better debug but that's how I get it: https://gist.github.com/mgagne/5c4f561dfaa605ef6cd69bf111159d99 | 22:16 |
| *** gyee has quit IRC | 22:19 | |
| mgagne | https://docs.djangoproject.com/en/1.11/topics/http/middleware/#process-exception | 22:20 |
| mgagne | "Again, middleware are run in reverse order during the response phase, which includes process_exception." | 22:20 |
| adriant | yeah, and on the images page (an angular page) I see notifications for "policy check failed" and "Unauthorised: Redirecting to Login" | 22:20 |
| mgagne | horizon.middleware.HorizonMiddleware is the last loaded middleware in my case, I removed most of them. | 22:20 |
| mgagne | ok, will enable debug for policy then | 22:20 |
| mgagne | hmm I'm not sure if I'm doing the right thing to enable full debug logging | 22:25 |
| mgagne | I removed all policies for horizon config, still the same. | 22:29 |
| adriant | it's unlikely to be a policy issue I think | 22:32 |
| adriant | the root cause will most likely be something to do with the existing session not being handled during a restart/start of the app | 22:33 |
| adriant | I tried digging into it myself, and couldn't find anything useful :( | 22:33 |
| mgagne | I'm annoyed because enabling logging reveals nothing in the log file... except lookup errors in templates | 22:35 |
| adriant | Yeah... openstack logging is lacking at times | 22:40 |
| mgagne | why can't I get django to enable logging everywhere? I mean, I can't believe no modules at all try to log something | 22:41 |
| adriant | It's quite likely there isn't any | 22:48 |
| adriant | most of the logging in horizon doesn't go that deep. | 22:49 |
| *** itlinux has joined #openstack-horizon | 22:50 | |
| mgagne | why not =( should propagate to underlying modules... | 22:51 |
| adriant | Django logging is just python logging in a dict format, so you can in theory turn on the logging for any library you are using provided that library has logging | 22:52 |
| adriant | but I have a feeling this error is happening in horizon, and it doesn't really have logging output where it is happening :/ | 22:53 |
| mgagne | yea... | 22:53 |
| mgagne | with file based caching, when reloading the page after a restart, there is no network connection performed by Horizon. I'm redirected to login page directly. | 22:53 |
| *** dnguyen has quit IRC | 22:53 | |
| mgagne | so the problem is self-contained on the server/app. | 22:54 |
| adriant | yeah | 22:54 |
| adriant | it can't unwrap the session, or something and then throws it away pretty much | 22:54 |
| *** cshen has quit IRC | 22:54 | |
| mgagne | could be an issue with serialization where data isn't properly signed anymore. but it doesn't explain that you can restore it on a 2nd refresh. | 22:55 |
| adriant | it could even be a django issue, but I somewhat doubt it | 22:55 |
| adriant | or maybe horizon can't yet talk to keystone to validate the token from the session? | 22:55 |
| mgagne | could be it because if I refresh again, it now tries to contact keystone to list projects I have access to. | 22:56 |
| mgagne | but it could also mean that django/horizon can't restore the session and therefore the existing token | 22:57 |
| *** rdopiera has quit IRC | 23:01 | |
| adriant | yep, lots of things that could be going wrong | 23:02 |
| adriant | but... only for the first (or first few) request(s) | 23:03 |
| mgagne | looks like I'm getting a horizon.exceptions.NotAuthenticated before process_response or even process_request from SessionMiddleware is called | 23:15 |
| mgagne | which makes absolutely no sense to me | 23:16 |
| mgagne | ok, missed a log entry | 23:17 |
| *** ianychoi has joined #openstack-horizon | 23:35 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!