Thursday, 2018-11-01

mgagneso far... in horizon.decorators request.user=AnonymousUser when calling require_auth. that's all I found for now.00:04
adriantmgagne: At the very least, open a bug for it with the notes of what you've found00:32
mgagneI think I found the issue, just tracking the git history00:32
adriantooo?!00:32
mgagnehttps://github.com/openstack/horizon/blob/master/openstack_auth/backend.py#L7200:33
mgagnerequest is not available and there is a note about monkey patching00:33
*** betherly has joined #openstack-horizon00:35
mgagneusage of self.request goes up to original commit in django_openstack_auth: https://github.com/openstack/django_openstack_auth/commit/9962375e0c5209ff82b7d6b5fda92bfec8dc4e1c00:35
mgagneso I'm not sure how it ever worked00:36
mgagnebecause once you are logged, it looks like you don't go through that code path, can't find my debug log00:36
mgagnethere is patch_middleware_get_user in openstack_auth which monkey patches the get_user of the backend and injects the request attribute. I'm not sure what's supposed to call it.00:39
adriantinteresting00:39
mgagnebut your name is in the comments so I feel you should know something :P00:39
*** betherly has quit IRC00:39
mgagnehttps://github.com/openstack/horizon/blob/master/openstack_auth/utils.py#L6300:39
mgagnecalled here https://github.com/openstack/horizon/blob/master/openstack_auth/urls.py#L2000:40
adrianthuh00:41
adriantso it is00:41
adriantOh, I know why. It's because I was doing the work to migrate between Django versions00:41
mgagneand here https://github.com/openstack/horizon/blob/master/openstack_dashboard/settings.py#L19600:41
mgagneI don't know what in Django/Horizon loads it00:42
adrianthttps://github.com/openstack/django_openstack_auth/commit/601e1ad44c18b9665c2ec6886438299de72d376100:43
mgagnewell, I do use Django 1.11 and it's broken ¯\_(ツ)_/¯00:43
adriantI'm curious if this bug existed in earlier versions00:44
mgagnehttps://github.com/openstack/horizon/blob/master/openstack_dashboard/urls.py#L5200:45
adriantso it does get called00:46
adriantjust, in a very roundabout way00:46
mgagnenope, only "introspected"00:46
adriantOH00:46
adriantffs00:47
mgagnebut if you access any pages under auth/ in a different browser, you don't get the issue anymore00:47
adriantso until it gets called at least once, that code path isn't executed00:47
mgagnebecause now it gets loaded for real00:47
mgagneyea00:47
adriantso we need to move that function call somewhere else00:47
mgagneI suppose ;)00:47
*** cshen has joined #openstack-horizon00:50
*** cshen has quit IRC00:54
*** betherly has joined #openstack-horizon00:56
adriantnope00:57
adrianttried moving it to openstack_dashboard/urls.py00:58
adriantsame error00:58
mgagneI tried to move it around and I get other errors from Django01:01
*** betherly has quit IRC01:01
mgagnecould be that django.contrib.auth.middleware.AuthenticationMiddleware is already loaded at that moment01:03
mgagnebut at this point, I'm like that dog in the lab, I don't know what I'm doing.01:04
adriantyeah, that's about where I got to when chasing this last time. :/01:06
mgagneso django.middleware.common.CommonMiddleware is early enough but might not be the right place. :P01:06
mgagneI might just create a dummy middleware and load it from there01:08
mgagnesomeone else had a similar idea https://opensourcehacker.com/2010/08/12/applying-monkey-patches-in-django-middleware-layer/01:09
mgagneI'm gonna be that guy https://gist.github.com/mgagne/185776102e73766571978ac295861baa01:13
*** dayou has quit IRC01:38
adriantdoes it work?01:47
mgagneyes01:47
mgagneI can restart with 20 processes, no issue.01:48
*** betherly has joined #openstack-horizon01:48
mgagnejust going to reset my environment to make sure01:48
mgagneall looks fine, currently repackaging on my side01:50
adriantjust adding the same to my code and will test01:51
adriantwhere in the middleware classes list did you put it?01:51
mgagne1st one01:51
*** betherly has quit IRC01:52
adriantworks!01:55
adriantI've limited gunicorn to 1 worker, and am having it restart after 5 requests01:55
adrianti've not seen a logout01:56
mgagne=)01:56
adriantmgagne: I expect to see a patch asap, and I would still file a bug report to help the horizon team understand the bug01:58
adriantmgagne: thanks sooooo much01:59
adriantI can't wait to port this to our build01:59
mgagnehehe, it bothers me too01:59
adriantmostly because... horizon has a memory leak problem, and we solve that by restarting it often...02:00
adriantwhich makes our users get "logged out"02:00
mgagneouch02:00
mgagneyea, we had to restart Horizon too, OOM. but running Kilo in some environments.02:00
adriantso the memory problem is mostly due to the memoized cached02:01
mgagnethere is a bug already: https://bugs.launchpad.net/horizon/+bug/176462202:01
openstackLaunchpad bug 1764622 in OpenStack Dashboard (Horizon) "Restarting the web server causes users to get kicked out" [Undecided,Confirmed]02:01
adriantand the solution (sort of) to the memory leak issue: https://review.openstack.org/#/c/605024/ and https://review.openstack.org/#/c/60880102:02
adriantoh cool, not sure how I missed that bug02:02
openstackgerritAdrian Turjak proposed openstack/horizon master: rework memoized as a LRU cache with expiry  https://review.openstack.org/60880102:12
mgagneok, added details in bug, I will propose a fix tomorrow as it's 10pm here. you can propose a fix too if you want or have a better idea.02:17
adriantnah, that fix is perfect02:19
adriantmgagne: but do add me to review :)02:19
adriantand if the memory leak affects you, feel free to review/test the above two patches02:20
mgagneit took a couple of weeks/months to get the OOM :P02:20
mgagneor maybe it crashed and restarted by itself with Puppet. who knows02:21
adriantah, and no many users probably?02:22
adriantnot*02:22
openstackgerritMathieu Gagné proposed openstack/horizon master: Fix django.contrib.auth.middleware monkey patching  https://review.openstack.org/61467302:36
mgagnewe have multiple web servers too so I don't know for sure.02:37
openstackgerritMathieu Gagné proposed openstack/horizon master: Fix django.contrib.auth.middleware monkey patching  https://review.openstack.org/61467302:43
*** Leo_m has quit IRC02:45
*** Leo_m has joined #openstack-horizon02:50
*** cshen has joined #openstack-horizon02:51
*** cshen has quit IRC02:55
*** betherly has joined #openstack-horizon03:21
*** betherly has quit IRC03:26
*** btully has joined #openstack-horizon03:38
*** btully has quit IRC03:42
*** betherly has joined #openstack-horizon03:53
openstackgerritAndy Botting proposed openstack/horizon master: Support setting more visibility options for edit/create image  https://review.openstack.org/60246803:55
openstackgerritAndy Botting proposed openstack/horizon master: Better support for community images  https://review.openstack.org/61468803:55
*** betherly has quit IRC03:57
openstackgerritAndy Botting proposed openstack/horizon master: Support setting more visibility options for edit/create image  https://review.openstack.org/60246803:58
*** dayou has joined #openstack-horizon03:59
*** betherly has joined #openstack-horizon04:24
*** betherly has quit IRC04:29
*** cshen has joined #openstack-horizon04:51
*** cshen has quit IRC04:56
openstackgerritAdrian Turjak proposed openstack/horizon master: Fix django.contrib.auth.middleware monkey patching  https://review.openstack.org/61467304:58
adriantmgagne: I've submitted some additional cleanup to your patch rather than leaving notes because I thought it would make your life a little easier than chasing comments. :)05:12
*** betherly has joined #openstack-horizon05:16
*** betherly has quit IRC05:21
*** ircuser-1 has quit IRC05:24
*** btully has joined #openstack-horizon05:26
*** btully has quit IRC05:31
openstackgerritAdrian Turjak proposed openstack/horizon master: IGNORE THIS  https://review.openstack.org/61469705:37
*** betherly has joined #openstack-horizon05:37
*** betherly has quit IRC05:42
*** betherly has joined #openstack-horizon05:57
*** betherly has quit IRC06:02
*** cshen has joined #openstack-horizon06:15
*** cshen has quit IRC06:21
*** cshen has joined #openstack-horizon06:43
*** cshen has quit IRC07:08
*** cshen has joined #openstack-horizon07:14
*** cshen has quit IRC07:19
*** cshen has joined #openstack-horizon07:34
*** cshen has quit IRC07:39
*** pcaruana|elisa| has joined #openstack-horizon07:40
openstackgerritOpenStack Proposal Bot proposed openstack/horizon master: Imported Translations from Zanata  https://review.openstack.org/61470407:50
*** pcaruana|elisa| has quit IRC07:59
*** pcaruana has joined #openstack-horizon08:05
*** cshen has joined #openstack-horizon08:06
*** cshen has quit IRC08:38
*** cshen has joined #openstack-horizon08:48
*** rdopiera has joined #openstack-horizon08:55
*** jpich has joined #openstack-horizon09:00
*** e0ne has joined #openstack-horizon09:43
*** ChanServ sets mode: +o e0ne09:43
*** v12aml has joined #openstack-horizon09:44
*** phuongnh has joined #openstack-horizon10:15
*** phuongnh has quit IRC10:31
*** e0ne has quit IRC10:32
*** e0ne has joined #openstack-horizon10:36
*** ChanServ sets mode: +o e0ne10:36
*** jtomasek has quit IRC10:43
*** jtomasek has joined #openstack-horizon10:45
*** dave-mccowan has joined #openstack-horizon10:45
*** btully has joined #openstack-horizon10:53
*** btully has quit IRC10:57
*** pcaruana has quit IRC11:05
*** cshen has quit IRC11:50
*** cshen has joined #openstack-horizon11:52
*** pcaruana has joined #openstack-horizon11:52
mgagneadriant: go ahead12:25
adriantmgagne: I already did, and it seems to have actually fixed a failure :)12:26
adriantto test that I pushed https://review.openstack.org/#/c/614697/ to see if the same failure happened again, and it did12:27
mgagnewas it mostly related to how tests were done?12:27
adriantmore so to how the middleware was used12:27
adriantit needed to be able to be called12:28
mgagnein tests right? because I didn't see any issue so far in my setup12:28
adriantwhen I included the monkey patching middleware, the tests kept failing because something kept trying to call the middleware12:28
adriantso adding that extra logic to the middleware which does pretty much nothing fixed it12:29
mgagne+112:29
* adriant isn't exactly sure what he fixed, but it works so *shrug*12:29
adriantand now we aren't being silly and also calling the middleware in weird places12:30
adriantI mean12:30
adriantcalling the monkeypatch12:30
mgagneI'm not sure I fully understand the fix because there is no __init__ in the other middlewares12:31
mgagnehmm12:32
mgagnefound one in MiddlewareMixin12:32
mgagneok, so be it12:32
adrianthttps://docs.djangoproject.com/en/1.11/topics/http/middleware/12:33
adriantdjango's example middleware has it12:33
adriantthat's pretty much what I based it on12:33
adriantlet me revert my code a sec and rerun the tests, I can show you the failure12:33
mgagneit's fine, I saw the failure yesterday12:35
mgagnesomething like object() takes no parameters12:36
adriantmgagne: http://paste.openstack.org/show/733773/12:36
adriantyeah, that12:36
mgagnestill weird because I didn't get any issue in unit tests on my side with queens. Django==1.11.912:38
mgagneor it doesn't test what I think it's testing :D12:38
adriantit only happened for me when I added the middleware to the openstack_auth test settings.py12:40
adriantbut the extra code in the middleware effectively does nothing, so it doesn't matter anyway and I guess is now compatible with whatever different ways it can be used12:41
adriantI'm still just so happy and thankful you found the bug!12:42
mgagne=) finally12:42
adriantwith that gone our memory leak problems, while still not idle, are mostly fixed with having uwsgi or gunicorn restart the workers more often.12:42
mgagnenow... how hard would it be to fix that "bug" ? ;) https://bugs.launchpad.net/horizon/+bug/160977712:45
openstackLaunchpad bug 1609777 in OpenStack Dashboard (Horizon) "'can_set_password' ignored in the new launch instance experience" [Medium,In progress] - Assigned to Pascal Boutin (pboutin)12:45
adriantmgagne: amusingly enough... I was actually talking about that recently with my boss. It causes us a bit of pain for windows vms.12:47
mgagneus too, we had to revert to the legacy launch instance12:48
adriantwe just get people to setup their windows machines via the web console12:48
mgagnePascal Boutin is no longer working with us so there won't be an update on that change12:49
mgagneadriant: not really an option because there is a random admin password generated by Nova that Horizon doesn't show on boot.12:50
adriant"As a user, when I launch a new instance, I want to be able to define a password for its default user, so that I can log in to it from the web console."12:51
adriantjust digging through our UX related backlog12:51
adriantwe actually would be interested in redoing the entirety of the launch instance workflow... but that's not an easy task and it's too low priority for us12:52
adrianthuh, this is an interesting one too:12:53
mgagneat this point, I just want an admin password field :P12:53
adriantAs a cloud user, when I request the cloud to generate an SSH key on my behalf, I want to download the private key by default and have an option to copy it, so that on my Windows desktop I don't have a broken key.12:53
adriantAcceptance criteria:12:53
adriantDownload option must be the default12:53
adriantThe plain text key should be hidden by default12:53
adriantIf the user clicks on the copy to clipboard action, then the plain text key is shown and the key is copied to the clipboard12:53
adriantwe kept having issues with the copy/paste action in horizon for generated keys broke when using tools like putty and such on windows12:54
mgagnewhat's a broken key?12:54
mgagneoh, right. the key is not in the putty format?12:54
adriantextra spaces, tabs, etc, windows was interpreting it weirdly12:54
adriantthe content was fine if you cleaned it up manually, but something about windows and copy/paste doesn't work12:55
mgagne:-/12:55
*** e0ne has quit IRC12:55
mgagneI think our user base still use/prefer passwords although I agree it might not be for the best.12:56
adriantwe mostly try to disallow it, but... windows VMs that's not an option12:57
mgagneso yea, if we could get an admin password field, that would be great ™12:57
adriantI'll add a note to our ticket for that so if we do anything related to it, to follow up with you as well.12:58
mgagnethanks!12:58
adriantI'm not a horizon core dev though, I just happen to be the person from our company who tends to deal with its code the most12:59
mgagnewe have that feature in our backlog as well, marked as impeded for ~1 year =)12:59
adriantfor us it's most just soooooo low a priority12:59
*** cshen has quit IRC13:01
mgagnehehe, just found that that you are from catalyst and I think we still have a story in our backlog related to some of your optimizations we need to test.13:02
adriantmgagne: to be fair, a lot of those aren't entirely all that helpful. The problem ultimately is just too many API calls. But at least some I'm vaguely tempted to do a little about if I can get https://review.openstack.org/#/c/608801/ merged. Mostly add a little bit of extra caching globally (extension lists), or per project.13:09
adriantlike for example: https://review.openstack.org/#/c/511095/ nova extensions don't change often, we can cache that once, refresh it every 24hrs, and that's maybe one less API call.13:10
mgagneyes, this one was in our list ^13:11
mgagnebut like you, Horizon optimization is very low on our list, mostly best effort.13:11
adriantyep13:12
adriantalthough in recent weeks... the memory leak has become a priority13:12
mgagneuntil I'm annoyed enough that I just take the time to fix it13:12
adriantwe... have been having to restart horizon 1-2 times a day, and carefully because of that logout bug13:13
adriantwe used to just let uwsgi handle it, but we started getting too many users so it restarted workers too often and oddly logged out people13:13
mgagne:-/13:14
adriantHorizon shouldn't really eat 4gb of memory per host when it's only running 4 uwsgi workers13:16
mgagne¯\_(ツ)_/¯13:16
adriantexactly13:16
*** belmoreira has joined #openstack-horizon13:17
adriantanyway, off to sleep. Can you test that my changes still solve the logout issue for you?13:19
adriantand I'll do the same tomorrow with my gunicorn setup13:20
mgagneI can test and report in the change.13:22
adriantmgagne: ty! I'm looking forward to seeing it merged, and backported to our current horizon package13:22
mgagne+113:22
adriantwell, cherrypicked to our internal repo13:22
*** cshen has joined #openstack-horizon13:24
mgagnesmall nit for django 1.11, commented in change13:29
*** cshen has quit IRC13:29
*** tolisbar has joined #openstack-horizon13:34
*** lblanchard has joined #openstack-horizon13:36
*** cshen has joined #openstack-horizon13:40
*** e0ne has joined #openstack-horizon13:40
*** ChanServ sets mode: +o e0ne13:40
tolisbarhello all. I've deployed openstack rocky through ansible-kolla 7.0.0 with internal/external networking, and the network_interface, neutron_external_interface, and kolla_external_vip_interface defined in the invenrory. My globals is this(https://pastebin.com/HicQXYaC). When I try to access horizon I get redirected to /horizon/auth/login/?next=/ which is a 404. My horizon.lon and horizon-access.log13:44
tolisbarare here(https://pastebin.com/u9E8hLF0) and here(https://pastebin.com/AN4rM9N6). Any idea what might be causing this ?13:44
*** Leo_m has quit IRC13:44
*** cshen has quit IRC13:48
*** e0ne_ has joined #openstack-horizon14:09
*** ChanServ sets mode: +o e0ne_14:09
*** e0ne has quit IRC14:10
*** Leo_m has joined #openstack-horizon14:31
*** btully has joined #openstack-horizon14:31
*** Leo_m has quit IRC14:31
*** Leo_m has joined #openstack-horizon14:32
*** cshen has joined #openstack-horizon14:33
*** btully has quit IRC14:36
*** mvkr has quit IRC14:40
*** mvkr has joined #openstack-horizon15:11
*** kukacz has quit IRC15:12
*** itlinux has quit IRC15:13
*** kukacz has joined #openstack-horizon15:19
*** gyee has joined #openstack-horizon15:33
*** e0ne_ has quit IRC15:34
*** e0ne has joined #openstack-horizon15:34
*** ChanServ sets mode: +o e0ne15:34
*** belmoreira has quit IRC16:11
*** jpich has quit IRC16:13
*** dnguyen has joined #openstack-horizon16:13
*** itlinux has joined #openstack-horizon16:13
*** dnguyen has quit IRC16:18
*** ircuser-1 has joined #openstack-horizon16:20
*** cshen has quit IRC16:25
*** e0ne has quit IRC16:26
*** cshen has joined #openstack-horizon16:41
*** cshen has quit IRC16:47
*** cshen has joined #openstack-horizon16:51
*** cshen has quit IRC16:56
*** belmoreira has joined #openstack-horizon17:02
*** belmoreira has quit IRC17:19
*** belmoreira has joined #openstack-horizon17:20
*** e0ne has joined #openstack-horizon17:31
*** ChanServ sets mode: +o e0ne17:31
*** tolisbar has quit IRC17:37
*** jpich has joined #openstack-horizon17:38
*** e0ne has quit IRC17:47
*** cshen has joined #openstack-horizon17:48
*** e0ne has joined #openstack-horizon17:59
*** ChanServ sets mode: +o e0ne17:59
*** e0ne has quit IRC18:02
*** e0ne has joined #openstack-horizon18:05
*** ChanServ sets mode: +o e0ne18:05
*** e0ne has quit IRC18:07
*** btully has joined #openstack-horizon18:08
*** btully has quit IRC18:13
*** mvkr has quit IRC18:47
*** belmoreira has quit IRC18:48
*** jpich has quit IRC18:59
*** itlinux has quit IRC19:01
*** pcaruana has quit IRC19:05
*** belmoreira has joined #openstack-horizon19:32
*** itlinux has joined #openstack-horizon19:39
*** mvkr has joined #openstack-horizon19:51
openstackgerritAndrew Bogott proposed openstack/horizon master: Server overview: display hypervisor name if available  https://review.openstack.org/61432819:54
*** cshen has quit IRC19:57
*** btully has joined #openstack-horizon19:58
*** btully has quit IRC20:02
*** cshen has joined #openstack-horizon20:09
*** e0ne has joined #openstack-horizon20:11
*** ChanServ sets mode: +o e0ne20:11
*** belmoreira has quit IRC20:12
*** itlinux has quit IRC20:14
*** itlinux has joined #openstack-horizon20:14
*** e0ne has quit IRC20:28
*** dave-mccowan has quit IRC20:32
*** belmoreira has joined #openstack-horizon20:35
*** belmoreira has quit IRC20:35
*** rdopiera has quit IRC20:50
*** lblanchard has quit IRC20:53
*** btully has joined #openstack-horizon21:46
*** btully has quit IRC21:51
openstackgerritAdrian Turjak proposed openstack/horizon master: Fix django.contrib.auth.middleware monkey patching  https://review.openstack.org/61467321:53
adriantmgagne: ^ see if that works for you22:14
mgagneadriant: I didn't get any error after adding the optional parameter to constructor. So I suspect middleware doesn't need to be a callable in Django 1.11.22:15
mgagnelet me test22:16
mgagneto make sure22:16
*** threestrands has joined #openstack-horizon22:17
mgagneno issue on my side22:17
mgagneadriant: ^22:18
adriantcool, just backporting it to my horizon to test22:20
adriantsweet, seems to work22:28
*** Leo_m has quit IRC22:36
*** cshen has quit IRC22:56
*** lblanchard has joined #openstack-horizon22:58
*** mvkr has quit IRC23:22
*** mvkr has joined #openstack-horizon23:23
*** btully has joined #openstack-horizon23:35
*** btully has quit IRC23:39
*** gyee has quit IRC23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!