| dmsimard | wwriverrat: yeah I think tweaking settings here and there can definitely help. How many hosts and tasks are in a typical playbook ? | 00:23 |
|---|---|---|
| wwriverrat | For today: I had 44 servers in each pod of 10 pods. | 00:24 |
| wwriverrat | I originally ran the whole group, but cancelled | 00:24 |
| dmsimard | Ohhh, you might really want to bump the amount of forks and probably tweak the polling interval | 00:24 |
| dmsimard | If you're not meant to run on too many hosts simultaneously you can do like serial: 10 or something like that | 00:25 |
| harlowja | don't kill the VM host with to many forks, kthx :-P | 00:25 |
| dmsimard | Even 25 forks is a small amount | 00:25 |
| wwriverrat | yeah, we’re running this through slack bot. It’s use could be as few as one and as many as thousands | 00:25 |
| harlowja | `MAX_FORKS = 25` | 00:25 |
| harlowja | lol | 00:25 |
| dmsimard | Forks are not necessarily what you think | 00:25 |
| harlowja | ya, i've seen it run out of memory though :( | 00:26 |
| harlowja | ^ which is why i put that limit in, lol | 00:26 |
| harlowja | its a 8GB vm now i think | 00:26 |
| harlowja | *8GB ram i think | 00:26 |
| harlowja | something like that, ha | 00:26 |
| dmsimard | And then people complain Ansible doesn't scale and is slow :( | 00:27 |
| harlowja | ya, i'm not sure where the blame lies | 00:27 |
| dmsimard | There's a couple things | 00:27 |
| harlowja | but def saw vm crap the bucket (oom kiler and bad things) when to many forks, lol | 00:27 |
| dmsimard | Anyway we're running like 70 concurrent ansible playbook runs with zuul and ara (sqlite) | 00:28 |
| harlowja | ya, maybe can try to boost it again, idk | 00:28 |
| harlowja | RAM | 00:28 |
| harlowja | 8192 MB | 00:28 |
| harlowja | Cores | 00:28 |
| harlowja | 4 | 00:28 |
| harlowja | thats the vm 'info' | 00:28 |
| dmsimard | We're not running against hundreds of hosts though | 00:28 |
| harlowja | ya, i have a feeling its the inventory parsing and ... or something | 00:29 |
| wwriverrat | To me, it’s about going the right speed. I like to be able to run a few Canary runs to make sure the change is going to do what I want. Then I like to go fast enough to get it done, but slow enough so if shiz-hits-the-fan, I can stop without too much damage | 00:29 |
| harlowja | was the porridge just right wwriverrat ? | 00:29 |
| harlowja | or to hot | 00:29 |
| harlowja | :-P | 00:29 |
| wwriverrat | given I get to spec “forks=whateva-the-river-rat-wants”, yes, it’s just right :D | 00:30 |
| harlowja | u should write a childrens book | 00:30 |
| harlowja | lol | 00:30 |
| wwriverrat | Once upon a time, there was a bald developer… | 00:30 |
| wwriverrat | :-o | 00:30 |
| harlowja | u are bald now? | 00:31 |
| wwriverrat | That thing on my head “may” be a rug? maybe not? | 00:31 |
| wwriverrat | The emojis are all bald, so that’s what I went with | 00:31 |
| harlowja | lol | 00:32 |
| dmsimard | I like where this is going | 00:32 |
| dmsimard | harlowja: so something else worth exploring in 1.0 is to drop the data on a message bus instead of directly writing to the database so that things get processed asynchronously | 00:34 |
| dmsimard | There'll be the notion of "input" drivers | 00:34 |
| dmsimard | The current callback is folded back as such | 00:34 |
| harlowja | hmmmm | 00:35 |
| harlowja | hmmmmm | 00:35 |
| harlowja | more rabbit? :-P | 00:36 |
| harlowja | please no, lol | 00:36 |
| harlowja | awk is using celery, u can use that :-P | 00:36 |
| harlowja | ^ and prob rabbit :( | 00:36 |
| harlowja | dmsimard interesting though, why would that just be upstream in ansible though, not in ara :-/ | 00:38 |
| harlowja | ara streaming output/results plugin to some <bus> | 00:38 |
| harlowja | (sorry ansible, not ara) | 00:39 |
| dmsimard | Yeah but ara needs a way to import that data in a format it knows about | 00:41 |
| harlowja | sounds like a common-interchange format :-P | 00:41 |
| harlowja | protobuf or other, ha | 00:41 |
| dmsimard | Anyway, I don't have the skills to write that kind of stuff but it seems some of the openstack-infra folks are interested in that | 00:42 |
| harlowja | i believe in your skillz | 00:42 |
| dmsimard | Right now we're doing pretty fine with sqlite tbh but we don't have the use case of keeping a long history, it's just one report per job | 00:42 |
| harlowja | ya | 00:42 |
| dmsimard | lol protobuff, Monty totally lost me in that oaktree email recently | 00:43 |
| dmsimard | Yeah, yeah, I know some of those words.jpg | 00:43 |
| harlowja | ya, he never responded to more of my questions :( | 00:43 |
| harlowja | the important questions were just starting, lol | 00:43 |
| harlowja | like ok, u have a shade executor that runs in threads, that doesn't work usually across machines/processes... | 00:44 |
| harlowja | it just starting getting fun :-P | 00:44 |
| harlowja | *started | 00:45 |
| wwriverrat | hrmmmm… oslo has a min_pool_size=1 and max_pool_size=5 (rather than SQLALCHEMY_POOL_SIZE) | 00:47 |
| harlowja | ya, idk why they picked those | 00:52 |
| *** jrist has quit IRC | 02:22 | |
| *** jrist has joined #ara | 02:23 | |
| *** bcoca has quit IRC | 03:56 | |
| *** jparrill has joined #ara | 06:59 | |
| *** openstackgerrit has quit IRC | 10:33 | |
| *** nicovs_be has joined #ara | 13:44 | |
| -openstackstatus- NOTICE: gerrit has been restarted to get it back to its normal speed. | 13:50 | |
| ara-slack1 | <alain> @dmsimard Quick question : where can we ask for feature requests ? Or check if they have already been asked ? | 14:03 |
| ara-slack1 | <dmsimard> @alain in the ARA bug tracker: https://storyboard.openstack.org/#!/project/843 | 14:05 |
| ara-slack1 | <alain> thanks. | 14:05 |
| ara-slack1 | <dmsimard> @alain there's a quick how-to here: https://ara.readthedocs.io/en/latest/contributing.html#set-up-your-ubuntu-launchpad-account | 14:05 |
| ara-slack1 | <dmsimard> you can also ask me and I'll tell you if it's something that is logged (or in my head, because I'm not super good at keeping the tracker up to date) | 14:06 |
| *** ara-slack1 has quit IRC | 14:06 | |
| *** ara-slack has joined #ara | 14:06 | |
| *** ara-slack has joined #ara | 14:06 | |
| *** ChanServ sets mode: +v ara-slack | 14:06 | |
| ara-slack | <alain> In the Web UI - you can sort quickly and you can search - how about a quick filter drop down on the columns ? Eg: I’m just looking a tasks on single host. | 14:07 |
| ara-slack | <alain> Though I suspect the host list may be large for some systems | 14:07 |
| ara-slack | <dmsimard> you mean kind of like in excel ? | 14:08 |
| ara-slack | <dmsimard> because you can already search for a hostname and it'll filter that too | 14:08 |
| ara-slack | <alain> So yes, and I know. but I was trying to filter for failed “mst-01” … it showed me nothing - but I know there is one… | 14:09 |
| ara-slack | <dmsimard> Is that in the hosts tab ? Or in the tasks tab ? | 14:10 |
| ara-slack | <alain> tasks | 14:11 |
| ara-slack | <alain> it uses the space. As I type “mst-01” it filters the host properly, but then if I put “mst-01 ” nothing shows up… | 14:11 |
| ara-slack | <alain> can you do multi term search ? | 14:13 |
| ara-slack | <dmsimard> @alain it's actually a regex search | 14:16 |
| ara-slack | <alain> let me try | 14:16 |
| ara-slack | <dmsimard> So if you go here for example: http://logs.openstack.org/50/454450/20/check/openstack-ansible-deploy-aio-opensuse-423/517a149/ara/reports/16972fd1-53f7-4d2a-a45a-844eec7b6dea.html | 14:17 |
| ara-slack | <dmsimard> You could search for (ipv4|ipv6) | 14:17 |
| ara-slack | <alain> Ok … Cool | 14:19 |
| ara-slack | <alain> works for what I want ! Thanks | 14:19 |
| ara-slack | <dmsimard> yay \o/ | 14:19 |
| ara-slack | <alain> Then I will say - add a “tool tip” or a :grey_question: on the side | 14:20 |
| ara-slack | <alain> but that’s just candy | 14:20 |
| ara-slack | <dmsimard> @alain There's already a "?" at the top right of the panels :slightly_smiling_face: | 14:35 |
| ara-slack | <alain> Again this is just sugar on top - placing a contextual tip near the search box - on a 27in screen they are far apart … | 14:36 |
| *** tbielawa has joined #ara | 14:37 | |
| ara-slack | <dmsimard> Ah, that's fair | 14:38 |
| ara-slack | <alain> I’m not complaining … :slightly_smiling_face: | 14:38 |
| ara-slack | <dmsimard> I'd love a UI/UX/Frontend pro to contribute to ARA :slightly_smiling_face: | 14:38 |
| ara-slack | <alain> I’m not one… but I stayed at holliday inn express once | 14:38 |
| ara-slack | <alain> https://www.youtube.com/watch?v=eHCTaUFXpP8 | 14:41 |
| ara-slack | <dmsimard> ahaha :slightly_smiling_face: | 14:52 |
| *** bcoca has joined #ara | 15:07 | |
| *** jparrill has quit IRC | 15:33 | |
| *** wwriverrat_ has joined #ara | 16:09 | |
| *** wwriverrat has quit IRC | 16:12 | |
| *** wwriverrat_ is now known as wwriverrat | 16:12 | |
| *** Bakey has joined #ara | 16:15 | |
| *** nicovs_be has quit IRC | 16:28 | |
| *** tbielawa is now known as tbielawa|lunch | 16:28 | |
| *** nicovs_be has joined #ara | 16:29 | |
| *** nicovs_be has quit IRC | 16:33 | |
| dmsimard | wwriverrat: I replied to your comment btw | 16:51 |
| wwriverrat | cool. Going to check out when some of my coffee gets deeper into my system | 16:52 |
| dmsimard | wwriverrat: ok, you haven't seen the issue reproduce since you upped your forks ? I guess if you have a good way to reproduce it with a lower amount of forks, it's worth a try to see if tweaking the pool settings also help. A combination of the two would probably be good enough. | 16:54 |
| wwriverrat | hmmm… yes/no. The run that I did yesterday spanned production and had some longer running tasks. Now that is done, trying to think whether we have a test environment and a playbook that would behave the same. | 16:56 |
| *** weshay is now known as weshay_mtg | 16:57 | |
| *** nicovs_be has joined #ara | 17:17 | |
| *** nicovs_be has quit IRC | 17:18 | |
| *** nicovs_be has joined #ara | 17:18 | |
| *** nicovs_be has quit IRC | 17:49 | |
| *** nicovs_be has joined #ara | 17:49 | |
| *** nicovs_be has quit IRC | 17:54 | |
| *** weshay_mtg is now known as weshay_bbiab | 18:05 | |
| *** jparrill has joined #ara | 18:38 | |
| *** tbielawa|lunch is now known as tbielawa | 18:39 | |
| *** nicovs_be has joined #ara | 19:46 | |
| *** weshay_bbiab is now known as weshay | 20:06 | |
| *** weshay is now known as weshay_interview | 20:12 | |
| *** nicovs_be has quit IRC | 20:20 | |
| *** nicovs_be has joined #ara | 20:25 | |
| *** nicovs_be has quit IRC | 20:30 | |
| *** openstackgerrit has joined #ara | 20:39 | |
| openstackgerrit | Nicolas Hicher proposed openstack/ara master: Move ansible linter tests outside run_tests.sh https://review.openstack.org/524723 | 20:39 |
| *** weshay_interview is now known as weshay | 21:19 | |
| *** tbielawa has quit IRC | 21:46 | |
| *** nicovs_be has joined #ara | 21:59 | |
| *** nicovs_be has quit IRC | 22:22 | |
| *** nicovs_be has joined #ara | 22:26 | |
| *** nicovs_be has quit IRC | 22:31 | |
| harlowja | dmsimard we are testing that change, now deployed, will see how it goes | 23:48 |
| harlowja | *fyi | 23:48 |
| *** nicovs_be has joined #ara | 23:57 | |
| harlowja | wwriverrat can verify/confirm that, ha | 23:58 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!