Friday, 2018-04-06

openstackgerritDavid Moreau Simard proposed openstack/ara master: Update integration jobs and fix broken unit tests  https://review.openstack.org/55895700:31
*** harlowja has quit IRC01:21
openstackgerritDavid Moreau Simard proposed openstack/ara master: 1.0 backport: Add helper modules/scripts to know where ARA is  https://review.openstack.org/55894801:28
openstackgerritDavid Moreau Simard proposed openstack/ara master: 1.0 backport: Refactor configuration and instanciate it  https://review.openstack.org/55894901:28
openstackgerritDavid Moreau Simard proposed openstack/ara master: Disable database automatic creation/migration in the sqlite middleware  https://review.openstack.org/55894601:28
openstackgerritDavid Moreau Simard proposed openstack/ara master: Add a bit of verbose debug to the sqlite middleware  https://review.openstack.org/55894701:28
openstackgerritMerged openstack/ara master: Update integration jobs and fix broken unit tests  https://review.openstack.org/55895701:53
openstackgerritMerged openstack/ara master: Disable database automatic creation/migration in the sqlite middleware  https://review.openstack.org/55894603:44
openstackgerritMerged openstack/ara master: Add a bit of verbose debug to the sqlite middleware  https://review.openstack.org/55894703:44
*** bcoca has quit IRC03:47
*** bcoca has joined #ara03:47
*** bcoca has joined #ara03:47
*** harlowja has joined #ara03:53
*** bcoca has quit IRC04:46
*** harlowja has quit IRC05:13
*** rvgate has quit IRC08:09
*** rvgate has joined #ara09:38
*** jrm_ has joined #ara10:01
*** jrm_ has quit IRC10:07
ara-slack<dmsimard> @harlowja did you end up trying the autocreate database off setting ?11:47
*** jrm_ has joined #ara12:22
jrm_Is it possible to integrate AWX with ARA?12:22
dmsimardjrm_: hey, you mean to have ARA reporting while using AWX ?12:40
jrm_yeah12:42
dmsimardjrm_: it's a question we get from time to time but I've never tried it yet personally12:43
ara-slack<dmsimard> @faraz ^ did you end up making it work ?12:43
dmsimardjrm_: I replied this a while back https://storyboard.openstack.org/#!/story/200156212:44
dmsimardjrm_: I should probably really try it12:44
jrm_I'll have a look over that stuff and see what happens :)12:45
dmsimardjrm_: let me know if it works!12:47
dmsimardI'm spinning one up to check it out.12:48
*** tbielawa has joined #ara13:11
*** cliles has quit IRC13:20
*** bcoca has joined #ara13:36
*** bcoca has joined #ara13:36
*** hwoarang has quit IRC13:51
jrm_@dmsimard it looks like I've got ARA and AWX talking when I run a playbook, but it throws an exception saying "No module named xstatic.main"13:52
dmsimardjrm_: sounds like ARA isn't installed but the callback is loaded13:54
dmsimardAWX and ARA probably aren't using the same python interpreter ?13:54
dmsimardDid you install ARA with pip somewhere ?13:55
*** hwoarang has joined #ara13:55
jrm_I activated the ansible virtualenv and ran pip from there?13:56
jrm_but looking in the directory I don't seem to have an ara folder13:56
dmsimardjrm_: what directory? $VIRTUAL_ENV/lib/py27/site-packages/ara ?13:57
jrm_I ended up in some other directory. I do have an ara folder in /var/lib/awx/venv/ansible/lib/python2.7/site-packages13:59
jrm_which is where ansible is looking for it13:59
dmsimardok, and we know awx is using ara because it's failing with that exception14:00
dmsimardhmm14:00
bcocahttps://github.com/ansible/ansible/pull/38403 <= dmsimard this might interest you14:01
dmsimardwith that venv sourced can you run something like: python -c "from ara.webapp import create_app; create_app()" ?14:02
dmsimardjrm_: ^14:02
dmsimardbcoca: looking14:02
dmsimardbcoca: neat14:04
jrm_it doesn't fail, but it doesn't output anything14:04
dmsimardjrm_: right, it's not supposed to .. we can actually make it print something to make sure it works but that one liner is supposed to exercise that xstatic.main import14:06
dmsimardjrm_: python -c "from ara.webapp import create_app; app = create_app(); print(app.config['XSTATIC'])"14:06
jrm_comes back with a JSON string14:08
jrm_y14:10
dmsimardok so it works14:10
dmsimardand by that I mean14:18
dmsimardara by itself works and is able to bootstrap itself14:19
dmsimardhowever it looks like ansible might use a different python interpreter14:19
dmsimardjrm_: you could probably confirm by somehow running ansible-playbook manually from within the venv and the ARA configuration enabled ?14:19
jrm_I get a different error trying that, "Import Error: no module named flask"14:22
dmsimardjrm_: if you do "which ansible-playbook" what do you get ?14:27
dmsimardjrm_: like, what is the actual path of ansible-playbook ?14:27
jrm_usr/bin/ansible-playbook14:29
dmsimardis that /usr/bin/ansible-playbook ? or $VIRTUAL_ENV/usr/bin/ansible-playbook ?14:30
jrm_the former14:30
dmsimardOk so that particular ansible-playbook doesn't use $VIRTUAL_ENV/bin/python then, unless there's a specific #!/$VIRTUAL_ENV/bin/python shebang at the top for some reason14:31
dmsimardand so it doesn't have access to ara14:31
dmsimardso ansible is installed globally then ?14:32
jrm_pass, awx is a docker container, it appears that whenever a playbook is run it's launched by /usr/bin/ans...14:33
dmsimardI just got an AWX going with docker compose, let me take a look14:34
dmsimardjrm_: so it's the task container right ?14:39
jrm_I believe so14:39
jrm_I did add it to the awx_web but that made no difference14:40
dmsimardthat's ... interesting14:53
dmsimardso there's /usr/bin/awx-python14:53
dmsimardwhich is really just a wrapper that sources the virtualenv first and then is a passthrough14:53
dmsimardthe venv path I'm seeing is /var/lib/awx/venv/awx14:53
dmsimardIs that what you have ?14:54
dmsimardjrm_: ^14:54
jrm_I understood it as the venv was /var/lib/awx/venv/ansible14:54
dmsimardhmm, can you try using the awx one ?14:54
dmsimardOH14:55
dmsimardwait14:55
dmsimardthe ansible-playbook command has a shebang for /usr/bin/python2 :/14:55
jrm_ooo... I might have it working14:57
jrm_I have it not erroring at least...14:57
jrm_if you create a new venv, as per https://github.com/ansible/awx/blob/devel/docs/custom_virtualenvs.md14:58
jrm_then when you ask AWX to use it, it runs the playbook in the venv14:58
dmsimardjrm_: is that a setting you need to put in your playbooks or something ?15:00
* dmsimard is an AWX/Tower noob15:00
jrm_to run a playbook in AWX you create a 'job template' and it's a setting there15:00
dmsimardjrm_: so you'd need to set the custom venv for all your job templates ? :/15:01
dmsimardjrm_: does it tell you what venv it is using by default ?15:01
dmsimardah "By default, this virtualenv is located at /var/lib/awx/venv/ansible on the file system."15:01
dmsimardok let me try..15:02
dmsimardmeh, I got some mtu config issues with this docker thing, hang on15:02
jrm_so it wrote stuff into ansible.sqlite15:18
jrm_but when I run a runserver up, I get 'macros' is undefined when browsing to it pointing at the db15:19
*** rvgate has quit IRC15:25
*** openstackgerrit has quit IRC15:33
dmsimardjrm_: hmmm, that sort of rings me a bell15:38
jrm_I have to leave now. I'll pop back on monday.16:12
*** jrm_ has quit IRC16:13
*** gvincent has quit IRC16:32
*** tbielawa is now known as tbielawa|g0n316:35
*** jparrill has quit IRC17:14
ara-slack<harlowja> @dmsimard ya, i turned that on, though i gotta turn back off the max-requests to see what happens17:17
ara-slack<dmsimard> Huh17:20
ara-slack<dmsimard> The maximum number of requests a worker will process before restarting.  Any value greater than zero will limit the number of requests a work will process before automatically restarting. This is a simple method to help limit the damage of memory leaks.17:21
ara-slack<logan.attwood> I'm getting OOM on massive plays18:28
ara-slack<logan.attwood> :(18:28
ara-slack<logan.attwood> any plans for adding pagination?18:32
ara-slack<logan.attwood> yeah. just gave docker for mac 10GB ram and they won't load :S18:42
*** uncas has joined #ara18:50
uncashi.  in my env. ara-manage runserver does not pickup ARA_DIR defined in ansible.cfg18:50
uncasit always start uses default location ~/.ara/18:50
uncasThis is how I define ARA_DIR in ansible.cfg18:51
uncas# grep ARA_DIR ansible.cfg ARA_DIR = "/opt/ara"18:51
*** logan- has quit IRC18:56
*** logan- has joined #ara18:56
uncasIm just trying to change location of ARA database and log.  I figured I can just set ARA_DIR in ansible.cfg and that is all.18:56
uncasseems ARA is not reading my ansible.cfg file.19:01
uncasonly way I know to workaround is to export environment variable ARA_DIR before starting the server19:01
uncasand before running ansible-playbook commands.19:01
ara-slack<dmsimard> @logan.attwood hey o/ how are you running ara ?19:08
ara-slack<logan.attwood> I've tried both as a wsgi app and via runserver19:09
ara-slack<dmsimard> okay, and you mentioned docker, so you're running it within docker ?19:09
ara-slack<logan.attwood> yeah19:09
ara-slack<dmsimard> ok, how massive are we talking about -- can you give an order of scale ?19:09
ara-slack<logan.attwood> this is happening both on ECS & Docker for Mac.19:09
ara-slack<dmsimard> in terms of hosts/tasks19:09
ara-slack<logan.attwood> yeah19:09
ara-slack<logan.attwood> one sec19:09
ara-slack<logan.attwood> File uploaded https://ara-community.slack.com/files/UA2H3AQ1Z/FA2QWRF37/image.png / https://slack-files.com/T6VAB05L7-FA2QWRF37-70e772014519:09
ara-slack<logan.attwood> (sorry to irc users who can't see the screenshot)19:10
ara-slack<logan.attwood> File uploaded https://ara-community.slack.com/files/UA2H3AQ1Z/FA2E4B7T6/image.png / https://slack-files.com/T6VAB05L7-FA2E4B7T6-b6bc7274e219:10
ara-slack<dmsimard> cool, thanks -- is that sqlite ?19:10
ara-slack<logan.attwood> postgres19:10
ara-slack<logan.attwood> I wouldn't do that to sqlite :slightly_smiling_face:19:10
ara-slack<dmsimard> you'd be surprised19:11
ara-slack<dmsimard> but beyond the point19:11
ara-slack<dmsimard> @logan.attwood just to confirm, that's only when browsing the interface right ? Not when running ansible-playbook ?19:13
ara-slack<logan.attwood> correct19:13
ara-slack<logan.attwood> ansible-playbook has been running fine19:13
ara-slack<logan.attwood> ``` ara_1  | 172.21.0.1 - - [06/Apr/2018 18:39:26] "GET /reports/ajax/stats/13e97e41-a41f-404d-9c5c-3047d31998d5.txt?_=1523039838801 HTTP/1.1" 200 - ara_1  | 172.21.0.1 - - [06/Apr/2018 18:39:26] "GET /reports/ajax/parameters/e7d1579c-f4e1-49a6-a760-5f69e7f192f3.txt?_=1523039838804 HTTP/1.1" 200 - ara_1  | 172.21.0.1 - - [06/Apr/2018 18:40:33] "GET /reports/ajax/stats/e7d1579c-f4e1-49a6-a760-5f69e7f192f3.txt?_=1523039838805 HTTP/1.1" 2019:13
ara-slack- ara_1  | 172.21.0.1 - - [06/Apr/2018 18:40:34] "GET /reports/ajax/plays/e7d1579c-f4e1-49a6-a760-5f69e7f192f3.txt?_=1523039838806 HTTP/1.1" 200 - ara_1  | 2018/04/06 18:41:42.922823 [INFO] (runner) child process died ara_1  | 2018/04/06 18:41:42.924911 [ERR] (cli) child process died with exit code 247 ```19:13
ara-slack<dmsimard> ok ... so there appears to be a memory leak somewhere but I don't know where yet. @harlowja seems to have worked around it for the time being with gunicorn's --max-requests (which automatically restarts processes after N requests)19:14
ara-slack<dmsimard> he has a dockerfile too, sec19:14
ara-slack<dmsimard> File uploaded https://ara-community.slack.com/files/U6V9WRRTM/FA21JHUE9/-.txt / https://slack-files.com/T6VAB05L7-FA21JHUE9-101cb3c890 - prob this is also useful for folks to...19:15
ara-slack<logan.attwood> it's not getting a single one iirc19:15
ara-slack<logan.attwood> (of the big reports)19:15
ara-slack<logan.attwood> verifying now19:16
ara-slack<logan.attwood> oh19:16
ara-slack<logan.attwood> it managed to load the parameters on a big one19:16
*** uncas has quit IRC19:17
ara-slack<bavdicev> Hi.  decided to Join Slack. much better than IRC19:17
dmsimard¯\_(ツ)_/¯19:17
dmsimardsubjective19:17
ara-slack<logan.attwood> yeah19:19
ara-slack<logan.attwood> will try w/ gunicorn19:19
bcocaslack is like irc but requiring a client that uses much more ram/cpu19:19
bcocaand makes it harder to ignore people posting cat videos19:19
ara-slack<logan.attwood> it served 25 requests before getting killed :S19:19
ara-slack<dmsimard> @logan.attwood so the long story short is that I'm aware there's a memory leak but haven't had the opportunity to find where yet. I'm actually in the process of merging some patches to improve performance in other places but that memory leak is on my todo.19:20
ara-slack<dmsimard> Something I'd be curious to know is if it reproduces with sqlite19:20
ara-slack<dmsimard> It would help narrow down where the issue could be19:21
ara-slack<logan.attwood> that gets tricky because we run ansible on jenkins19:21
ara-slack<logan.attwood> and want to run ara on ECS19:22
ara-slack<logan.attwood> will gunicorn serve up static assets? i come from the land of java and php19:22
ara-slack<dmsimard> it does but it's not very good at it, usually you put something like nginx in front19:22
ara-slack<dmsimard> just so you have something returning HTTP 304's instead of hitting the disk each time19:22
ara-slack<logan.attwood> iow, if it's not just an internal tool19:23
ara-slack<logan.attwood> @dmsimard any idea where `requirements.txt` lives?19:24
ara-slack<dmsimard> in the source19:24
ara-slack<dmsimard> https://github.com/openstack/ara/blob/master/requirements.txt19:25
ara-slack<logan.attwood> sorry, wasn't sure where it was in the context of the dockerfile19:25
ara-slack<logan.attwood> since his patches are missing from the repo too :S19:25
ara-slack<dmsimard> oh, I'm not sure either -- I suppose he copies it into the docker build directory or something19:25
ara-slack<dmsimard> @logan.attwood if you can't find a workaround or a fix, I should have time for this sometime next week19:26
ara-slack<dmsimard> @bavdicev ara uses the same configuration loading mechanism as ansible19:27
ara-slack<dmsimard> So if ansible is loading ara (because you have callback_plugins = <something>) then it should also load the configuration under the [ara] block.19:27
ara-slack<dmsimard> @bavdicev oh wait, I realize where the issue is19:28
ara-slack<dmsimard> ARA_DIR is the name of the environment variable19:28
ara-slack<logan.attwood> @dmsimard if you're going to dig into the memory leak i'll leave our dockerfile alone then :slightly_smiling_face:19:28
ara-slack<dmsimard> In your ansible.cfg file you need to have [ara] and then "dir = something"19:28
ara-slack<logan.attwood> fyi, an average ansible run over all our infrastructure is around ~50.38k tasks19:28
ara-slack<logan.attwood> :|19:28
ara-slack<logan.attwood> (we use it like puppet)19:28
ara-slack<dmsimard> you can see this here: https://ara.readthedocs.io/en/latest/configuration.html#parameters-and-their-defaults19:29
ara-slack<bavdicev> oh19:29
ara-slack<bavdicev> I get it now19:29
ara-slack<bavdicev> ARA_DIR does NOT go into ansible.cfg19:29
ara-slack<bavdicev> the paramter is simply ‘dir’19:29
ara-slack<bavdicev> got it.19:29
ara-slack<dmsimard> @logan.attwood that's cool, it's a scale I'm familiar with but in a bigger decentralized volume19:30
ara-slack<dmsimard> as in, way more playbooks but smaller ones :slightly_smiling_face:19:30
ara-slack<logan.attwood> hindsight is 20/20- I went with ansible because some of the folks here were familiar with it already. Should have just used puppet :man-facepalming:19:30
ara-slack<dmsimard> do you know puppetdb ? :slightly_smiling_face:19:30
ara-slack<logan.attwood> yeah19:30
ara-slack<dmsimard> er, or rather puppetboard19:31
ara-slack<logan.attwood> i do19:31
ara-slack<logan.attwood> set it up at a previous job19:31
ara-slack<dmsimard> yeah ara is slightly inspired by puppetboard19:31
ara-slack<dmsimard> I also did a lot of puppet in a previous life :slightly_smiling_face:19:31
dmsimardbcoca: there's always a relevant xkcd https://xkcd.com/1782/19:32
dmsimardoh god I had never seen the alt text19:33
dmsimard2078: he announces that he switches from screen+irssi to tmux+weechat19:33
dmsimardI'm a screen user but mostly because I've never taken the time to learn tmux properly :/19:33
* bcoca is guy floating in space19:34
ara-slack<dmsimard> yup19:35
ara-slack<dmsimard> just like Ansible, you'll have "callback_plugins" vs ANSIBLE_CALLBACK_PLUGINS19:35
dmsimardbcoca: there was a funny thread in some k8s ML about potentially moving away from slack because they're cutting the irc bridge19:35
dmsimardit was glorious19:35
dmsimardhttps://groups.google.com/forum/#!topic/kubernetes-dev/8-Lj7Gp9b4A is the one19:37
bcocaim still using irc bridge from slack, once it is gone ...19:39
dmsimardThere's a weechat-slack thing, apparently it works well but I haven't tried it19:39
dmsimard(I happen to use weechat so..)19:39
dmsimardhttps://github.com/wee-slack/wee-slack19:40
bcocai know, i found it for some cooworkers, also pidgin has one19:41
dmsimardpidgin is horrible in terms of UI/UX though :(19:41
dmsimardunless it's changed since I last usedit19:42
dmsimardwhen I was on OSX a few years back I really liked adium19:42
bcocadmsimard:  finch!19:47
bcocafinch is curses version, which i used, have not used gui pidgin in years19:48
dmsimardTIL19:51
bcocatmux + finch <= how i've done messaging for long time19:52
dmsimardI use a variation of https://hveem.no/a-modern-IRC-experience19:56
*** viewlogic is now known as Brainspackle20:11
*** DrWaluigi has quit IRC20:35
ara-slack<dmsimard> @faraz you might be interested in https://twitter.com/ARA_Community/status/98235541089096089720:38
ara-slack<dmsimard> I'll document how to do it.20:38
ara-slack<faraz> woah20:44
ara-slack<faraz> yesss!!20:44
ara-slack<faraz> thanks for checking awx out... i got fairly close before but had to abandon ship for work stuff20:44
ara-slack<dmsimard> np, I get this question a lot so I figured I'd spend the time20:44
ara-slack<dmsimard> The gist of it is that it doesn't seem like awx actually uses /var/lib/awx/venv/ansible ?20:45
ara-slack<dmsimard> So you need to either create a custom venv (and then make sure that it's selected in your job templates)20:45
ara-slack<faraz> hmm20:45
ara-slack<dmsimard> By default AWX appears to use the system-wide python interpreter20:45
ara-slack<faraz> Yeah, it appears their paid product Tower uses the venv20:46
ara-slack<faraz> but the awx stuff is just using the systemwise interpreter20:46
ara-slack<dmsimard> and then in the settings, you have a field where you can add callbacks -- so I added /usr/lib/python2.7/site-packages/ara/plugins/callbacks to it20:46
ara-slack<dmsimard> oh ? maybe awx is different then, I don't know.20:46
ara-slack<faraz> yeah, the deployment is very different for tower vs awx20:47
ara-slack<dmsimard> :S20:47
ara-slack<faraz> no but this is good though..20:47
ara-slack<dmsimard> are you using tower or awx ?20:47
ara-slack<faraz> at work i use both..20:47
ara-slack<dmsimard> o_O20:47
ara-slack<faraz> recently switched gears to awx for development envs20:47
ara-slack<faraz> but tower for prod stuff20:48
ara-slack<dmsimard> ah, makes sense20:48
ara-slack<dmsimard> less expensive too :slightly_smiling_face:20:48
ara-slack<faraz> lol yeah20:48
ara-slack<dmsimard> so I'll document how I think it works in AWX but if it's different in Tower...20:48
ara-slack<dmsimard> ¯\_(ツ)_/¯20:48
ara-slack<faraz> yeah, ill try to see if tower can work too some day20:49
ara-slack<faraz> redhat just needs to pay you or something and make ara the default way how awx/tower shows job history20:49
ara-slack<faraz> :)20:49
ara-slack<faraz> it sucks right now20:49
ara-slack<faraz> scrolling through all the output20:49
ara-slack<dmsimard> red hat is already paying me :P20:49
ara-slack<dmsimard> but I mean, not for ARA specifically20:49
ara-slack<dmsimard> it's a side project, not my full time job20:50
ara-slack<faraz> oh? you talk to the Tower devs to see if the ara project can be merged or something at some point in the future?20:50
ara-slack<dmsimard> yeah, bcoca and matburt are interested in seeing some sort of integration but we're not sure what that means yet. I'd like ara to remain a standalone project with the same simplicity and narrow use case.20:51
ara-slack<dmsimard> The next major release will provide an API so maybe it will go through that, I don't know20:51
*** DrWaluigi has joined #ara20:51
ara-slack<dmsimard> They seem keen on keeping third party projects standalone as well, at least from the discussions I've seen about molecule recently20:52
ara-slack<faraz> that makes sense. Right perhaps the API could help... quite honestly we pay a LOT for tower and I am not convinced all the money is worth it.. its a mess trying to debug what went wrong in a play20:52
ara-slack<faraz> i work for IBM, so IBM has a horse in the race with urbancode20:52
ara-slack<faraz> but ansible usage is widespread20:52
ara-slack<dmsimard> never heard of urbancode before :slightly_smiling_face:20:53
ara-slack<faraz> lol exactly :)20:53
ara-slack<dmsimard> looks complicated20:54
ara-slack<dmsimard> I'll stick to ansible and ara :P20:54
ara-slack<faraz> lol it is20:54
ara-slack<faraz> haha20:54
ara-slack<dmsimard> anyway, tower is more than just reporting though -- ACLs and stuff are pretty nice if you just want to hand out a "deploy" button or something like that20:56
ara-slack<faraz> yeah, thats kind of the reason we pay for it20:57
ara-slack<faraz> audit etc20:57
ara-slack<faraz> im sure it will get better over time..20:57
ara-slack<bavdicev> Thanks. with a bit of tinkering, I can now see my Jenkins playbook runs in ARA21:14
dmsimardyay \o/21:15
ara-slack<dmsimard> @faraz btw I filed an issue about the python interpreter thing: https://github.com/ansible/awx/issues/173721:16
bcocadmsimard:  i want ara still independant, i would just like awx to allow for 'drop ins' and ara be able to be 'dropped in'21:17
bcocato integrate with auth/rbac/menus21:17
bcocaand of course, drop the callback into every execution21:17
dmsimardbcoca: it seems like it wouldn't be too hard to get going if it wasn't for that issue I just created21:18
dmsimardbcoca: tl;dr /var/lib/awx/venv/ansible/bin/pip install ara21:18
dmsimardset the extra callback in settings -> jobs21:18
dmsimardand boom, it's enabled for all the jobs21:18
dmsimardWe could even set it up to use the embedded postgre21:18
dmsimardbut because of that issue ^ we either need to 1) install ara system-wide or 2) create a new venv and manually select it for every template/job21:20
*** spiette has quit IRC21:28
*** DrWaluigi has quit IRC22:03
*** DrWaluigi has joined #ara22:16

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!