| *** logan- has quit IRC | 00:02 | |
| *** logan- has joined #ara | 00:02 | |
| openstackgerrit | David Moreau Simard proposed openstack/ara master: Add support for configuring sqlalchemy pool size, timeout and recycle https://review.openstack.org/524427 | 02:59 |
|---|---|---|
| openstackgerrit | David Moreau Simard proposed openstack/ara master: Improve memory usage when dealing with large amount of data https://review.openstack.org/559840 | 02:59 |
| *** harlowja has quit IRC | 03:16 | |
| *** bcoca has quit IRC | 03:45 | |
| *** harlowja has joined #ara | 03:57 | |
| *** gvincent has joined #ara | 05:22 | |
| *** harlowja has quit IRC | 06:40 | |
| *** gvincent has quit IRC | 07:49 | |
| *** gvincent has joined #ara | 07:50 | |
| *** gvincent has quit IRC | 07:52 | |
| *** gvincent has joined #ara | 07:52 | |
| *** DrWaluigi has quit IRC | 08:45 | |
| *** mihai_ansible has joined #ara | 10:24 | |
| mihai_ansible | hi, I just read about your project and used the anisble role to install ara and the apache server | 10:24 |
| mihai_ansible | now I would like to run ansible plabooks for a mac laptop, and record the output to the server mentioned before | 10:25 |
| mihai_ansible | I seem just not to find how to exactly do that | 10:25 |
| mihai_ansible | is it required that I install ara on the mac? or is enough to install the callback and modules? | 10:25 |
| mihai_ansible | I would not like to install too many packages | 10:25 |
| mihai_ansible | thanks! | 10:25 |
| mihai_ansible | aand made a pull request for a small typo atb the handler name | 10:29 |
| *** rvgate has quit IRC | 10:40 | |
| *** rvgate has joined #ara | 10:40 | |
| *** sshnaidm is now known as sshnaidm|bbl | 10:47 | |
| *** sshnaidm|bbl has quit IRC | 10:47 | |
| *** rvgate has quit IRC | 11:12 | |
| *** sshnaidm|bbl has joined #ara | 11:27 | |
| mihai_ansible | is this project abandoned? | 11:30 |
| dmsimard | mihai_ansible: it's not at all. I was just sleeping :) | 11:40 |
| mihai_ansible | :) Hi | 11:40 |
| mihai_ansible | glad you woke up | 11:40 |
| mihai_ansible | I was about to abandon it | 11:40 |
| mihai_ansible | dmsimard: I had installed the default on a cent virtual machine, managed to see the interface | 11:41 |
| mihai_ansible | dmsimard: now I wanted to run an ansible playbook from my mac laptop | 11:41 |
| mihai_ansible | dmsimard: and I am stuck, | 11:42 |
| mihai_ansible | dmsimard: don't have time to understand the data base configurations and all, but I assume it just uses the sqlite in the centos | 11:42 |
| dmsimard | mihai_ansible: the callback is what records the data in a database. It needs to be installed wherever Ansible is running. The web interface reads from that database. | 11:43 |
| mihai_ansible | dmsimard: what is needed to do to use it ansible playbook from laptop, ara on its own server? | 11:43 |
| mihai_ansible | I can also update the role afterwards | 11:44 |
| mihai_ansible | dmsimard: even though the pull requests are denied .... | 11:44 |
| dmsimard | Sure, I'd love to get some help on the role | 11:44 |
| mihai_ansible | dmsimard: thanks! | 11:44 |
| dmsimard | There are docs on how to contribute: https://github.com/openstack/ara/blob/master/README.rst#contributing-testing-issues-and-bugs | 11:45 |
| dmsimard | We just don't use GitHub for code contribution, it's used for mirroring | 11:45 |
| mihai_ansible | dmsimard: ouch ... everything is overly complicated these days :) | 11:47 |
| mihai_ansible | dmsimard: even that with tools like ansible we have the means to just provide a command | 11:47 |
| mihai_ansible | dmsimard: it would of been great if I could of set it up today, | 11:48 |
| dmsimard | It's pretty simple when you get used to it, I prefer it to the GitHub workflow by far :D | 11:48 |
| dmsimard | I'd be happy to help you get started but I need to properly wake up first lol | 11:48 |
| mihai_ansible | dmsimard: ok. get your uppers see u later | 11:49 |
| mihai_ansible | dmsimard: just ping me pls | 11:49 |
| dmsimard | In the meantime, maybe this FAQ can help you understand: http://ara.readthedocs.io/en/latest/faq.html#can-ansible-with-ara-run-on-a-different-server-than-the-web-application | 11:49 |
| dmsimard | If you're installing ARA on your laptop (or even just Ansible) I highly recommend using virtual environments | 11:50 |
| dmsimard | I'll be back in an hour or so. | 11:51 |
| *** sshnaidm|bbl is now known as sshnaidm | 11:53 | |
| *** weshay_pto is now known as weshay | 12:03 | |
| *** openstackgerrit has quit IRC | 12:04 | |
| *** bcoca has joined #ara | 12:50 | |
| *** bcoca has joined #ara | 12:50 | |
| ara-slack | <dmsimard> @logan.attwood this patch limits the memory usage down to more reasonable levels: https://review.openstack.org/#/c/559840/ | 12:50 |
| ara-slack | <dmsimard> It's still a bit spiky but I'll iterate on it | 12:51 |
| ara-slack | <dmsimard> You can install it this way: http://paste.openstack.org/raw/718815/ | 12:51 |
| dmsimard | mihai_ansible: ok so what have you tried and what doesn't work ? | 12:52 |
| *** logan- has quit IRC | 12:55 | |
| *** logan- has joined #ara | 12:56 | |
| *** mihai_ansible has quit IRC | 13:15 | |
| ara-slack | <logan.attwood> <3 | 13:50 |
| ara-slack | <dmsimard> @logan.attwood no love until you've confirmed it works :P | 13:51 |
| ara-slack | <logan.attwood> any idea how i get something i can apply with patch? | 13:51 |
| ara-slack | <logan.attwood> ah. nm | 13:52 |
| ara-slack | <logan.attwood> now i'm worried about load balancer timeouts :,) | 13:58 |
| ara-slack | <dmsimard> I gave you the command to run in that paste if you hadn't seen it | 13:59 |
| ara-slack | <dmsimard> (I try not to use the slack paste thing, people on IRC can't see those) | 13:59 |
| ara-slack | <logan.attwood> yeah i used that | 14:00 |
| ara-slack | <logan.attwood> oh | 14:00 |
| ara-slack | <logan.attwood> memory is still spiking :S | 14:00 |
| ara-slack | <logan.attwood> 5GB | 14:00 |
| ara-slack | <logan.attwood> I was trying to setup a wsgi memory profiler middleware but no luck | 14:01 |
| ara-slack | <logan.attwood> spiked, crashed :( | 14:02 |
| *** tbielawa has joined #ara | 14:02 | |
| ara-slack | <logan.attwood> it's | 14:04 |
| ara-slack | <dmsimard> ok so | 14:04 |
| ara-slack | <logan.attwood> reports/ajax/plays | 14:04 |
| ara-slack | <dmsimard> sec | 14:04 |
| ara-slack | <dmsimard> In that patch, there's a constant here at the top: https://review.openstack.org/#/c/559840/2/ara/views/reports.py | 14:04 |
| ara-slack | <dmsimard> Can you put 50 or something and see if it spikes less ? | 14:04 |
| ara-slack | <logan.attwood> sure | 14:07 |
| ara-slack | <logan.attwood> dumb python question | 14:07 |
| ara-slack | <logan.attwood> File uploaded https://ara-community.slack.com/files/UA2H3AQ1Z/FA4BGHT9T/image.png / https://slack-files.com/T6VAB05L7-FA4BGHT9T-dfeab5f180 | 14:07 |
| ara-slack | <logan.attwood> how do I log the size of `results` | 14:07 |
| ara-slack | <logan.attwood> no hits for "log" in that file | 14:07 |
| ara-slack | <dmsimard> @logan.attwood I can add some verbosity for profiling real quick, hang on | 14:11 |
| ara-slack | <logan.attwood> what i wanted to get working was this but i had no idea how to: https://github.com/mgedmin/dozer | 14:12 |
| ara-slack | <logan.attwood> I'd get it working but the `_dozer/index` url didn't work | 14:13 |
| ara-slack | <logan.attwood> just did this instead | 14:14 |
| ara-slack | <logan.attwood> (sorry irc folks) | 14:14 |
| ara-slack | <logan.attwood> File uploaded https://ara-community.slack.com/files/UA2H3AQ1Z/FA44NQ8BD/image.png / https://slack-files.com/T6VAB05L7-FA44NQ8BD-be5968c917 | 14:14 |
| *** zxiiro has quit IRC | 14:14 | |
| *** zxiiro has joined #ara | 14:14 | |
| ara-slack | <logan.attwood> what i'm hoping | 14:15 |
| ara-slack | <logan.attwood> is that it will OOM and print out the size it OOM'd at | 14:15 |
| ara-slack | <logan.attwood> unless it's OOMing inside `jsonify` | 14:15 |
| ara-slack | <logan.attwood> then all hope is lost | 14:15 |
| *** mnaser has quit IRC | 14:15 | |
| *** mnaser has joined #ara | 14:16 | |
| ara-slack | <logan.attwood> well | 14:23 |
| ara-slack | <logan.attwood> stats load now | 14:23 |
| ara-slack | <logan.attwood> halfa meg of json :,) | 14:23 |
| ara-slack | <dmsimard> with YIELD_PER at 50 ? | 14:27 |
| ara-slack | <logan.attwood> yeah | 14:27 |
| ara-slack | <logan.attwood> timeout reading headers this time | 14:28 |
| ara-slack | <logan.attwood> I think I may have found the problem | 14:28 |
| ara-slack | <logan.attwood> I think sqlalchemy is eagerly loading the relationships? | 14:29 |
| ara-slack | <dmsimard> I've read something along those lines when I found about yield_per | 14:29 |
| ara-slack | <dmsimard> I can't pretend to know too much about how it works | 14:29 |
| ara-slack | <logan.attwood> trying this | 14:30 |
| ara-slack | <logan.attwood> File uploaded https://ara-community.slack.com/files/UA2H3AQ1Z/FA4C14X7X/image.png / https://slack-files.com/T6VAB05L7-FA4C14X7X-922b2fc07c | 14:30 |
| ara-slack | <logan.attwood> well the ones that succeeded loaded hella quick | 14:31 |
| ara-slack | <dmsimard> hmm, I'll definitely check it out | 14:32 |
| ara-slack | <dmsimard> got rabbitholed by adding debug logs | 14:32 |
| ara-slack | <logan.attwood> yeah i think it's going to timeout again :S | 14:32 |
| ara-slack | <dmsimard> @logan.attwood you likely need to add that to the other queries as well | 14:34 |
| ara-slack | <logan.attwood> getting up into `results 7500` before crash | 14:50 |
| ara-slack | <dmsimard> I found something interesting while fixing the logging things | 14:57 |
| dmsimard | running this on my laptop is killing it lol | 15:13 |
| dmsimard | you can hear the fans spinning | 15:13 |
| ara-slack | <logan.attwood> so it eventually succeeds | 15:16 |
| ara-slack | <logan.attwood> after nearly 30 minutes | 15:16 |
| ara-slack | <logan.attwood> i think pagination might be the solution :( | 15:16 |
| ara-slack | <dmsimard> but hey, it didn't go OOM :) | 15:18 |
| ara-slack | <dmsimard> so that's something | 15:19 |
| ara-slack | <dmsimard> you managed to not go OOM only with eagerloads(false) ? | 15:19 |
| ara-slack | <logan.attwood> i never let it run w/o eagerloads(false), got impatient | 15:21 |
| ara-slack | <logan.attwood> (being honest) | 15:21 |
| ara-slack | <dmsimard> that's totally fair | 15:23 |
| ara-slack | <dmsimard> still looking at this, improving logging will help us look in the right direction | 15:24 |
| ara-slack | <logan.attwood> trying with YIELD_PER set to 2000 | 15:24 |
| ara-slack | <logan.attwood> man it's a *lot* faster with eager load false | 15:24 |
| dmsimard | "The loading of relationships falls into three categories; lazy loading, eager loading, and no loading. Lazy loading refers to objects are returned from a query without the related objects loaded at first. When the given collection or reference is first accessed on a particular object, an additional SELECT statement is emitted such that the requested collection is loaded." | 15:25 |
| dmsimard | "Eager loading refers to objects returned from a query with the related collection or scalar reference already loaded up front. The Query achieves this either by augmenting the SELECT statement it would normally emit with a JOIN to load in related rows simultaneously, or by emitting additional SELECT statements after the primary one to load collections or scalar references at once." | 15:25 |
| ara-slack | <logan.attwood> what sent me down that direction was connecting to postgres and seeing the queries issued by ara constantly change | 15:26 |
| ara-slack | <logan.attwood> 2k OOMs | 15:26 |
| ara-slack | <logan.attwood> :( | 15:26 |
| dmsimard | yeah yield_per limits the size of the buffer between each flush | 15:27 |
| dmsimard | if you're setting that number higher, the buffer (and memory usage) is higher | 15:27 |
| ara-slack | <logan.attwood> still though- should be able to get 2k rows into memory | 15:27 |
| ara-slack | <dmsimard> @harlowja do you have an opinion on lazy and eager loading ? | 15:27 |
| ara-slack | <dmsimard> @logan.attwood right, it shouldn't be that big of a deal, I agree | 15:28 |
| ara-slack | <logan.attwood> even 500 OOMs :( | 15:45 |
| ara-slack | <logan.attwood> i'm calling `sys.getsizeof` on the results list, will be curious to see where it ends up | 15:53 |
| dmsimard | almost finished fixing logging.. | 15:54 |
| dmsimard | there's a lot of these queries: http://paste.openstack.org/raw/718850/ | 16:00 |
| dmsimard | (with different UUIDs) | 16:00 |
| ara-slack | <logan.attwood> looks like loading that makes me sad :( | 16:05 |
| ara-slack | <logan.attwood> probably makes more sense (dunno if sqlalchemy supports this, i know doctrine on php does) | 16:05 |
| ara-slack | <logan.attwood> to write a custom query and have sqlalchemy hydrate it | 16:05 |
| ara-slack | <logan.attwood> eager loads might actually help <_< | 16:06 |
| *** openstackgerrit has joined #ara | 16:08 | |
| openstackgerrit | David Moreau Simard proposed openstack/ara master: WIP: Fix logging https://review.openstack.org/560069 | 16:08 |
| ara-slack | <dmsimard> @logan.attwood ^ you can use that as a basis for printing things to log if you'd like.. need to flush ~/.ara/logging.yml (if it exists, the defaults were updated) | 16:08 |
| ara-slack | <dmsimard> and then logs are available in ~/.ara/ara.log | 16:08 |
| ara-slack | <logan.attwood> i uh, kinda prefer them to come to stdout on apache :slightly_smiling_face: | 16:09 |
| dmsimard | Yeah that's okay, I'm using that to help me find the issue | 16:10 |
| ara-slack | <logan.attwood> I think the solution is going to be "don't use an ORM when fetching 50k rows" | 16:12 |
| dmsimard | lol | 16:13 |
| dmsimard | give me a chance to take a stab at it, ironing this out would be great | 16:14 |
| ara-slack | <harlowja> @dmsimard no opinion; i personally don't like ORMs though, haha | 16:24 |
| ara-slack | <dmsimard> Didn't you write oslo.db ? T_T | 16:24 |
| ara-slack | <harlowja> nope, lol | 16:24 |
| ara-slack | <harlowja> oslo.db is just like a helper over sqlalchemy though (imho most of it's code should just be in sqlalchemy itself as like a helper module) | 16:25 |
| ara-slack | <harlowja> sqlachemy-core i think is ok, though idk, i haven't used it in a while | 16:26 |
| ara-slack | <harlowja> http://docs.sqlalchemy.org/en/latest/core/ | 16:26 |
| ara-slack | <harlowja> i also just like regular SQL but ya, nice things can't always be had, lol | 16:28 |
| ara-slack | <harlowja> i just always have these weird nightmares about ORMs where u get a 33MB database that takes 6+GB ram :P | 16:40 |
| ara-slack | <harlowja> *alwayyyyys* lol | 16:40 |
| ara-slack | <harlowja> which in no world imho should take more than like 33MB * 10, lol | 16:41 |
| ara-slack | <harlowja> (cause the whole DB could just be in memory, lol) | 16:41 |
| dmsimard | yeah :/ | 16:41 |
| *** DrWaluigi has joined #ara | 16:51 | |
| *** tbielawa is now known as tbielawa|lunch | 17:07 | |
| ara-slack | <harlowja> but it's ok, u live u learn :P | 17:17 |
| *** mihai_ansible has joined #ara | 17:20 | |
| mihai_ansible | dmsimard: hi, back for some minutes | 17:20 |
| mihai_ansible | dmsimard: as I was saying, not very good with virtuaenv | 17:20 |
| mihai_ansible | dmsimard: what I did, ran the ansible ara role against a centos 7.4 machine with the apache and the module enabled | 17:21 |
| mihai_ansible | dmsimard: have ansible on my mac laptop and I would like to send the playbook to the centos7 server | 17:21 |
| mihai_ansible | dmsimard: is not very clear what and how do I need to configure n the laptop | 17:21 |
| mihai_ansible | dmsimard: tried pip install ara and succeded but the rest of the python instructions complain about missing setup module | 17:23 |
| mihai_ansible | dmsimard: maybe there is a simpler way? | 17:23 |
| *** mihai_ansible has quit IRC | 17:27 | |
| *** sshnaidm is now known as sshnaidm|bbl | 17:27 | |
| *** openstackgerrit has quit IRC | 17:34 | |
| *** tbielawa|lunch is now known as tbielawa|appt | 17:43 | |
| *** gvincent has quit IRC | 17:51 | |
| ara-slack | <harlowja> @dmsimard so what patches should i be trying? | 18:25 |
| ara-slack | <harlowja> i've got some time to try some stuff | 18:25 |
| ara-slack | <harlowja> in between other things :P | 18:25 |
| *** harlowja has joined #ara | 18:25 | |
| ara-slack | <dmsimard> Nothing for now, @logan.attwood tried it and it still needs some work. I'm trying something and I'll report back if it works. | 18:29 |
| ara-slack | <logan.attwood> i'm thinking you need eager loading, which means not using `yield_per` | 18:34 |
| ara-slack | <logan.attwood> which means server side pagination | 18:34 |
| dmsimard | yeah that's what I'm working on | 18:41 |
| dmsimard | just need to get the javascript datatable madness to work with it | 18:41 |
| ara-slack | <logan.attwood> <3 | 18:42 |
| ara-slack | <logan.attwood> oh damn. thanks to some ansible failures I got some smaller result runs | 18:42 |
| dmsimard | javascript sucks | 18:42 |
| ara-slack | <logan.attwood> 8k tasks is 3.8MB of json :,) | 18:42 |
| ara-slack | <logan.attwood> so extrapolating 8k to 50k | 18:42 |
| ara-slack | <logan.attwood> almost a 24MB blob | 18:43 |
| ara-slack | <logan.attwood> ahaha | 18:43 |
| *** sshnaidm|bbl is now known as sshnaidm | 18:48 | |
| *** gvincent has joined #ara | 18:49 | |
| dmsimard | hmmm | 19:01 |
| dmsimard | come to think of it | 19:01 |
| dmsimard | this is more complicated than I thought | 19:02 |
| dmsimard | because the search field is executed and rendered client side, if we don't push all the data it won't be able to search | 19:02 |
| dmsimard | bleh | 19:02 |
| *** tbielawa|appt is now known as tbielawa | 19:15 | |
| dmsimard | the fact that there's entire python modules dedicated to doing what I'm trying to do is kind of depressing | 19:29 |
| ara-slack | <logan.attwood> seems pretty simple https://stackoverflow.com/a/43104868 | 19:54 |
| ara-slack | Action: logan.attwood ducks | 19:54 |
| ara-slack | <logan.attwood> the search does get tricky though | 19:54 |
| dmsimard | We already have pagination like that | 19:56 |
| dmsimard | It's not a fancy dynamic javascript thing though | 19:56 |
| dmsimard | https://github.com/openstack/ara/blob/master/ara/views/reports.py#L54-L59 | 19:57 |
| dmsimard | https://github.com/openstack/ara/blob/master/ara/templates/report_list.html#L22-L46 | 19:57 |
| dmsimard | Doing a pagination like that would not allow for things like client-side searching, filtering and sorting | 19:57 |
| dmsimard | There are plenty of fancy javascript things I'm sure we'd be able to do to get datatables to poke a paginated endpoint but I am not a javascript person :( | 19:58 |
| dmsimard | Looked at different options but none of them appear to be appropriate, maintained or actually work | 19:58 |
| dmsimard | i.e, https://pypi.python.org/pypi/sqlalchemy-datatables/ | 19:59 |
| dmsimard | or https://datatables.net/examples/server_side/pipeline.html | 19:59 |
| dmsimard | If you search about datatables and server-side pagination, it's a seemingly endless void of stackoverflow and datatables forums questions | 20:00 |
| *** tbielawa is now known as tbielawa|brb | 20:00 | |
| *** gvincent has quit IRC | 20:21 | |
| *** gvincent has joined #ara | 20:29 | |
| *** gvincent has quit IRC | 20:53 | |
| *** tbielawa|brb is now known as tbielawa | 21:03 | |
| *** jrist has quit IRC | 21:14 | |
| *** jrist has joined #ara | 21:41 | |
| ara-slack | <dmsimard> @logan.attwood, @harlowja I tried a bunch of different things today and I know what the problem is but I can't find a reasonable solution that is not 1.0 (or 2.0) | 21:53 |
| harlowja | it's al good | 21:53 |
| ara-slack | <dmsimard> The best I can do for the time being is the yield_per patch which at least limits the RAM spikes but has a performance impact for larger runs | 21:53 |
| ara-slack | <dmsimard> I've learned a lot from this though and we'll definitely put this into application in 1.0/2.0 where we're allowed to break backwards compat | 21:54 |
| ara-slack | <logan.attwood> :( thanks for digging in | 21:55 |
| ara-slack | <dmsimard> If you folks know anyone who would like to dig into this further, I'd be happy to look at a patch that doesn't involve refactoring everything (because that's what we're already doing in the future release with the API and etc) | 21:57 |
| ara-slack | <dmsimard> For the time being this admittedly exceeds my skills, I'm more of a sysadmin than a dev :S | 21:57 |
| dmsimard | I'll chat with gvincent (who's actually good) tomorrow to make sure we get this right, he's helping with 2.0 | 21:58 |
| ara-slack | <harlowja> wfm | 22:01 |
| *** openstackgerrit has joined #ara | 22:03 | |
| openstackgerrit | David Moreau Simard proposed openstack/ara master: Do not merge: profiler implementation https://review.openstack.org/560166 | 22:03 |
| dmsimard | ^ Just an example memory profiler implementation which provides an output like this: paste.openstack.org/show/718864/ | 22:04 |
| dmsimard | http://paste.openstack.org/show/718864/ * | 22:04 |
| *** spiette has quit IRC | 22:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
