Wednesday, 2017-07-05

« Tuesday, 2017-07-04 Index Thursday, 2017-07-06 »
*** iceyao has joined #ara00:20
*** iceyao has quit IRC00:25
*** iceyao has joined #ara00:41
*** iceyao has quit IRC00:46
*** cuongnv has joined #ara00:48
*** iceyao has joined #ara01:02
*** iceyao has quit IRC01:06
*** iceyao has joined #ara01:22
*** cuongnv_ has joined #ara01:37
*** cuongnv has quit IRC01:39
*** iceyao has quit IRC02:02
*** iceyao has joined #ara02:03
*** cuongnv_ is now known as cuongnv02:12
*** iceyao has quit IRC03:53
*** iceyao has joined #ara04:15
*** iceyao has quit IRC04:20
*** iceyao has joined #ara04:33
*** iceyao has quit IRC04:38
*** iceyao has joined #ara04:42
*** iceyao has quit IRC05:39
*** iceyao has joined #ara05:58
*** iceyao has quit IRC06:02
*** iceyao has joined #ara06:09
*** iceyao has quit IRC06:13
*** sshnaidm has quit IRC06:16
*** iceyao has joined #ara06:36
*** iceyao has quit IRC06:41
*** sshnaidm has joined #ara07:19
*** iceyao has joined #ara08:40
*** iceyao has quit IRC10:08
*** cuongnv has quit IRC10:14
*** iceyao has joined #ara10:19
*** iceyao has quit IRC10:23
*** iceyao has joined #ara10:46
*** iceyao has quit IRC10:50
*** iceyao has joined #ara11:22
*** sshnaidm has quit IRC12:44
*** tbielawa has joined #ara12:53
*** sshnaidm has joined #ara13:49
*** jparrill has joined #ara13:53
dmsimardsshnaidm: forgot you were here, let's move here instead14:48
dmsimardso, to roll back discussion14:49
dmsimardI feel like https://review.openstack.org/#/c/479882/2/roles/collect-logs/library/ara_graphite.py should be a good addition to ara upstream14:49
dmsimardbut I feel like it's better suited to have a callback implementation instead of a module so the data is fed to graphite in real-time and not at the end of the playbook in bulk14:50
sshnaidmdmsimard, it's better to have it configurable14:50
dmsimardWe've had this idea of drivers in ARA where ara could collect data from other places (i.e if upstream zuul publishes ansible tasks to mqtt, ara could pick up messages from there instead of from the callback)14:50
dmsimardsshnaidm: ara is configurable :)14:51
sshnaidmdmsimard, yeah :) I mean now there is requirement to publish only when playbook finished successfully14:51
dmsimardsshnaidm: ah, hrm14:52
dmsimardI guess one does not exclude the other14:52
sshnaidmdmsimard, so we can use either callback or final "publish all"14:52
dmsimardwe can have it both as a module and as a callback14:52
sshnaidmdmsimard, yeah14:52
sshnaidmdmsimard, I'm more concerned about mapping task names to graphite paths14:53
dmsimardsshnaidm: that can end up being in ansible.cfg14:53
sshnaidmdmsimard, as part of ara config?14:53
dmsimardsshnaidm: yeah, ara can be configured inside ansible.cfg under the [ara] block14:54
dmsimardsshnaidm: and fyi ansible 2.4 is moving from inifile format to yaml format14:54
dmsimardfor ansible.cfg14:54
sshnaidmoh, nice14:55
sshnaidmdmsimard, do you know how you are going to implement these publishers or you have any example..?14:56
dmsimardsshnaidm: It's mostly just an idea for the time being. In practice, the current implementation would become a driver (sql) and there would be a module for each additional driver (graphite/elasticsearch/etc)15:00
dmsimardsshnaidm: https://github.com/openstack/ara/blob/master/ara/plugins/callbacks/log_ara.py would be made generic and publish to the appropriate driver instead of being hardcoded to sql15:01
dmsimardsshnaidm: so that people not interested in the web UI could not even publish to a sql database if they don't want to15:01
dmsimardsshnaidm: basically ara would really become an ansible indexer, if you will15:09
sshnaidmdmsimard, ok.. so for now I leave it as is, or you want me to integrate this anywhere..?15:11
dmsimardsshnaidm: If this is blocking you from moving forward, feel free to carry it for the time being but let's consider that we'll need to move it eventually15:29
dmsimardsshnaidm: does it add any requirements that are not in the standard library ?15:29
sshnaidmdmsimard, no, everything in std15:31
*** tbielawa is now known as tbielawa|lunch15:57
dmsimardsshnaidm: left some comments in your review15:59
dmsimardsshnaidm: up to you if you want to move it to ara right now or wait until later -- if we merge it, it would only be available in trunk/master until we cut a release15:59
dmsimardIt'd land in 0.14 which would not be for a while still16:00
dmsimard0.14 is bound to be the version that lands ansible 2.4 compatibility16:00
*** tbielawa|lunch has quit IRC16:22
*** jparrill has quit IRC16:53
*** iceyao has quit IRC17:03
*** iceyao has joined #ara17:30
*** iceyao has quit IRC17:34
*** tbielawa has joined #ara18:02
*** tbielawa is now known as tbielawa|covfefe18:03
*** tbielawa|covfefe is now known as tbielawa18:28
*** iceyao has joined #ara18:35
*** iceyao has quit IRC18:39
*** iceyao has joined #ara19:02
*** iceyao has quit IRC19:06
*** tbielawa is now known as tbielawa|VMs19:08
*** tbielawa|VMs is now known as tbielawa19:22
*** tbielawa is now known as tbielawa|tour19:41
*** openstackgerrit has joined #ara19:49
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/ara master: [WIP] Add subunit output support  https://review.openstack.org/48071619:49
*** tbielawa|tour has quit IRC20:11
*** tbielawa has joined #ara20:11
*** tbielawa has quit IRC20:32
*** jparrill has joined #ara20:34
*** iceyao has joined #ara21:48
*** iceyao has quit IRC21:53
harlowjadmsimard whatever came with that output recording/capturing stuffs?21:55
dmsimardharlowja: it was released in 0.13.2 I think21:55
harlowjathe idea was i guess to do `ara playbook list` then run a followup playbook to record data directly for that other playbook right21:59
harlowjapretty sure that was the idear21:59
harlowjadmsimard  ^ ?22:02
harlowjaif so ,  is there anyway for me to force/specify/pick the uuid that will be used for initial recording (ie the playbook that will do the work) so that i can ensure that i attach the stdout to the right playbook in the db  (which afaik is based on a uuid)22:04
dmsimardharlowja: Yeah the uuid is currently generated. You can retrieve it at runtime by registering the output of an ara_record or ara_read task under playbook_id.22:07
harlowjaor an initial `ara create play` that will give me a uuid that i can then use later for all recording and running22:07
harlowjaright dmsimard that assumes say that i have one ansible thing happening at the same time22:07
harlowjaso wondering how i would figure out the uuid in a multi-process-ara/ansible usage22:08
dmsimardharlowja: you run concurrent ansible-playbook ?22:08
harlowjain theory, yes22:08
dmsimardharlowja: I still haven't addressed that but it is a known issue :/22:08
dmsimardhttps://storyboard.openstack.org/#!/story/200085322:09
harlowjaie, 'deploy glance staging' and 'deploy nova <somewhere-else>' seems like they should be concurrent (or at least possible so)22:09
dmsimardharlowja: the concurrent use case and the clash of metadata for ara_record/ara_read is something I want to fix.. in the meantime, let me try and think22:10
harlowjaeven a simple `ara init playbook record` that could be used later would work22:10
harlowja^ command would output a uuid to use22:10
dmsimardharlowja: yeah but it's kind of an ugly workaround to solving the real problem22:11
dmsimardnot going to implement something like that22:11
harlowjaya, depends on how ugly u want this to be, lol22:12
harlowjasome kind of way to tag playbooks would work also, so that i can at least find the tag later (when running the followup playbook)22:13
dmsimardharlowja: well I mean the current real problem is that concurrent runs can clash, if this would be resolved you could accurately retrieve the uuid22:13
dmsimardand there'd be no need for workarounds22:14
dmsimardthere's maybe an easy fix.. let me look22:14
* harlowja run one playbook at a time, lol22:14
harlowjawith a lock22:14
harlowjalol22:14
dmsimardharlowja: it sucks because ansible doesn't have something like a seed, a hash or whatever could uniquely represent a run22:16
harlowja(though i don't want to do that, especially if u have better  idears)22:16
harlowjaya, i mean, a 'ara_tag: <tags>' would work to22:16
harlowjathough still a pita22:16
harlowjawhere the tags would be said seed that i could later find22:16
harlowjabut ya, if ansible had something, that'd be nice, lol22:17
dmsimardharlowja: hm, wait, maybe things don't clash against each others after all22:18
dmsimardharlowja: yeah I think this could be a non-issue, maybe you could confirm22:18
dmsimardharlowja: ansible generates a new tmpdir for each ansible run22:19
dmsimardharlowja: and ara stores it's file in that tmpdir22:19
* harlowja puts ara stuff into a sqlite db22:19
harlowja(unless u mean a different file)22:20
dmsimardharlowja: https://paste.fedoraproject.org/paste/nU2dWl62AlH~ArZwH8bnXQ/raw22:20
dmsimardharlowja: regardless of database backend, the playbook uuid is persisted in an ansible tmpfile on the control host22:21
dmsimardex:22:21
dmsimard$ cat ./ansible-local-2215499oovY/ara.json22:21
dmsimard{"playbook": {"id": "3849c552-6bd7-4c04-95ee-107316fc4e1b"}}22:21
dmsimardThis is how ara_record and ara_read modules are able to refer back to the parent playbook22:21
harlowjahow would i find which  ./ansible-local-$blah to use?22:22
dmsimardharlowja: you don't, the modules know where to find it in the loaded in-memory context22:22
dmsimardharlowja: do this: https://gist.githubusercontent.com/dmsimard/f04758bbc5d9c3e96703d506e85141a7/raw/5b0f2f7fb55221293a1d4b9cd9e5013bfcef589d/task.yml22:23
dmsimardharlowja: you'll see there is a playbook_id key in the return output that you can use22:24
dmsimardin that particular example it'd be record.playbook_id22:24
harlowjahmmmm, intersting22:24
harlowjaok that may work, will try22:25
dmsimardharlowja: it's a bit complicated but basically the modules load the flask application context that the callback initially started in which the ansible tmpdir was created (with a known path inside that context)22:25
dmsimardon Ansible's end, it's basically just a mktemp so it's random, but we have access to the path inside the context22:26
harlowjadoesn't that mean those tempdirs can never be deleted?22:27
dmsimard(the flask application context is a super powerful feature I didn't know existed before)22:27
dmsimardharlowja: they can be deleted, they're only useful during the actual playbook run is in progress22:27
harlowjaah, k22:27
harlowjagotcha22:27
dmsimardharlowja: if you want to run ara_record on a playbook *after* it's finished, you *need* to pass a playbook id22:28
dmsimardand that's the feature I implemented for you :P22:28
harlowjayup, gonna see how i can plug it all together now, ha22:28
harlowjaright now our ansible stdout/stderr get shoved into a gist on our github, lol22:28
dmsimardgotta do what you gotta do22:29
dmsimardharlowja: but hey, no guarantee on this, however it *should* work22:29
harlowja:-P22:29
dmsimardconcurrent runs is not something that is even tested right now22:29
harlowjasomeones gotta do it, lol22:30
dmsimardyeah, I can see it being a legit use case22:30
dmsimardharlowja: tbh, it's not so much concurrent runs that's a problem22:30
dmsimardharlowja: it's concurrent runs *from the same host*22:30
dmsimardlike, I know there are users with multiple "control hosts" and feed data to a central mysql database22:30
dmsimardthat works super fine22:31
dmsimardthere might be edge cases (I would venture so far as saying within ansible itself?) by running ansible concurrently as the same user on the same box22:31
harlowjaimpassible, lol22:32
dmsimardimpossibru ?22:32
harlowjalol22:32
harlowjaya22:32
harlowjathat22:32
dmsimardharlowja: your bot could launch ephemeral isolated containers to run ansible-playbook from or something :p22:33
harlowjafml22:33
harlowjalol22:33
dmsimardharlowja: with cool technology22:33
dmsimardharlowja: like k8s22:33
harlowja:(22:35
harlowjato much tech22:35
harlowjadon't need waffle-iron to deploy openstack22:35
harlowjalol22:35
harlowjawaffle iron powered by k8s22:36
*** jparrill has quit IRC22:36
harlowjadmsimard what was the hack that http://logs.openstack.org/54/342354/21/check/gate-kolla-python35/99c1d4b/_zuul_ansible/ did, i forget22:49
harlowjathey basically patchd ansible (i forget?)22:50
*** iceyao has joined #ara22:50
dmsimardharlowja: oh basically they hijacked the whole command module22:52
*** klindgren_ has joined #ara22:52
dmsimardharlowja: and then run everything in there22:52
harlowjaklindgren_ hallo22:52
klindgren_o/22:52
*** klindgren_ is now known as klindgren22:52
harlowjaklindgren_ also at godaddy, was asking me some questions on wtf i was going to do for recording, lol22:52
dmsimardharlowja: basically it's to (amongst other things) send data to a file so that the zuul console can read from it and people can stream it22:52
harlowjaand then i was starting to channel/proxy dmsimard and i dislike channeling people (i'm not into voodoo)22:53
dmsimardharlowja: otherwise ansible doesn't output the result of a task until it's completed22:53
dmsimardharlowja: so zuul hijacked the module in order to stream the output as it is available22:53
dmsimardklindgren: hai22:53
dmsimardharlowja: https://github.com/openstack-infra/zuul/blob/master/zuul/ansible/library/command.py22:54
dmsimardhttps://github.com/openstack-infra/zuul/blob/master/zuul/ansible/library/command.py#L127-L15722:54
*** iceyao has quit IRC22:54
dmsimardhttps://github.com/openstack-infra/zuul/blob/master/zuul/ansible/library/zuul_log.py22:55
dmsimardand finally https://github.com/openstack-infra/zuul/blob/master/zuul/ansible/library/zuul_console.py#L15822:55
harlowjaya , man22:55
harlowjaare the ansible folks just gonna make stdout/stderr redirection easier, lol22:55
klindgrenSo one thing I was trying to figure out with ara_record stuff.  Is with ansible you can have 2 types of playbooks.  A playbook of playbooks and a playbook of tasks.22:55
dmsimardharlowja: it's not a matter of redirection, it's a matter of buffering the task output until it's completed22:56
* harlowja was telling klindgren we'd need something like https://gist.github.com/harlowja/994b761f15269db29a8527713bb3f679 on playbooks 22:56
dmsimardharlowja: streaming the live output of a task when you're running one task against one host is easy, but if you're running one task against 10, 100, 1000 hosts ...22:56
klindgrenIf we made the ara_record and output fetching stuff its own set of playbooks and then jsut executed those before and after our other ansible role stuff - would that be fine22:56
harlowjathe ara_record one + debug needs to happen from the main playbook that does all the work22:57
harlowja(afaik)22:57
harlowjacause that's the one i want to record stdout/stderr of22:57
klindgrenor does it need to run as a task in the main playbook doing all the work22:57
klindgrenWell what I dont understand from an internal ansible perspective is how is a playbook of playbooks actually treated.22:57
klindgrenbecause you can pass variables/host and stuff between them22:58
dmsimardklindgren: so the fact that your one playbook ends up including a set of playbooks is not important from the perspective of ara, there will be one unique playbook run and it's the "main" playbook, so there will be one playbook UUID and one run recorded in ara22:58
dmsimardklindgren: in ARA, one "ansible-playbook" command == one playbook uuid22:59
klindgrenkk22:59
dmsimardklindgren: so, if you want to stash the playbook uuid for that particular playbook, you do it anywhere within that playbook, can be in any file, not necessarily in the "main" playbook file22:59
klindgrenthats what I was figuring/hoping so we can do a generic ara_record/ara_output thingy and include that on all our main runs, and it will work for everything.23:00
dmsimardklindgren: what I did for ARA's integration tests when I added the feature for harlowja was fairly simple since the execution order is predictable23:00
dmsimard1) Save the output somewhere: https://github.com/openstack/ara/blob/master/run_tests.sh#L95-L9623:01
dmsimard2) Get the playbook id I'm interested in https://github.com/openstack/ara/blob/master/run_tests.sh#L11423:01
dmsimard3) Attach it to the playbook: https://github.com/openstack/ara/blob/master/run_tests.sh#L13023:01
harlowja4) profit23:01
dmsimardharlowja: nooooooo23:01
dmsimard4) ???23:01
dmsimard5) profit23:01
harlowjaklindgren the issue that i was discussing earlier was https://github.com/openstack/ara/blob/master/run_tests.sh#L114 won't work, especially if we run at any kind of concurrency level23:02
harlowjaso that's where including https://gist.github.com/harlowja/994b761f15269db29a8527713bb3f679 somewhere in the 'main playbook' will dump it (so that it can be later found)23:03
dmsimardklindgren: so, in your context, if you need a reliable way of retrieving the playbook uuid (step #2), you can use a "sort-of" dummy ara_record task and the registered return provides the playbook_id23:03
dmsimardOnce you have the playbook_id, you can do whatever you want with it, dump it to a file at a predictable location or something.23:03
dmsimardand then do step #3 not much differently than in the example I provided23:03
harlowjadmsimard did u ever poke the ansible-tower folks, from what i remember they are also capturing output somehow23:04
dmsimardharlowja: they wrap around ansible-playbook with like pexpect23:04
harlowjak23:04
dmsimardbasically you don't run the ansible-playbook command23:04
dmsimardyou run ansible-tower (or whatever it is)23:05
dmsimardand in the background what it really does is run ansible-playbook and captures the output23:05
harlowjaya, then i guess they shove it in there db and associate it however23:05
klindgrenso stupid question - can we generate a UUID and pass it to ara somehow?23:05
harlowjanice try!23:05
harlowjadid u read the logs,l lol23:05
* harlowja not sure there are logs :-P23:06
dmsimardI ideally don't want to wrap ansible-playbook under an ara command, want to avoid requiring users to change their workflows23:06
dmsimardharlowja: yeah eavesdrop23:06
harlowjaklindgren already asked, ha23:06
dmsimardklindgren: no there's no way, it's generated when the run starts23:06
klindgrenread logs = pshaw - who does that these days23:06
klindgrenis this uuid generation something that ara is doing or is this something internally that ansible is doing and ara is just using it?23:07
dmsimardklindgren: the UUID ends up being used all over the place, leaving an option to make it user-supplied worries me :)23:07
harlowjahttps://github.com/openstack/ara/blob/master/ara/models.py#L32 :-P23:07
dmsimardklindgren: it's something that ara does, ansible has no concept of uniquely identifying an ansible run23:07
dmsimardharlowja: yeah, and you know what, there was this openstack-dev thread a while back where people were advising against using uuid as primary keys.. I TIL'd that day. I'll address that sometime.23:08
dmsimardklindgren: is recovering and using the uuid the way I explained particularly bothersome ?23:09
klindgrenharlowja, so - when we run this stuff via daddy is a new tmpdir for each playbook run done?23:10
dmsimardklindgren: yeah, that part is abstracted by ansible23:10
dmsimardklindgren: https://paste.fedoraproject.org/paste/nU2dWl62AlH~ArZwH8bnXQ/raw23:10
*** iceyao has joined #ara23:11
klindgrendmsimard, its just: https://github.com/openstack/ara/blob/master/run_tests.sh#L95-L96 make it so where if we are doing like 4 parallel runs, all refrencing the same tee dir that they will start blowing each other up23:11
klindgrenI guess we could do a uuid per run ahead of time and try on our end to link our uuid run to a ara uuid23:12
dmsimardklindgren: well I don't have context on how you're running things.. but surely you have some means of knowing which run this is referring to somehow23:12
dmsimardand pipe the output to a file appropriate for that run23:12
dmsimardor you could, I don't know, LOGFILE=$(mktemp) ansible-playbook |tee $LOGFILE ?23:13
klindgrenI think I have an idea - I will work with josh on it.23:14
dmsimardklindgren: happy to hear about it when you got something going23:14
dmsimardI have to head out before A) my gf divorces me B) I digest myself23:14
*** iceyao has quit IRC23:15
*** iceyao has joined #ara23:31
*** iceyao has quit IRC23:36
*** iceyao has joined #ara23:52
*** iceyao has quit IRC23:56
« Tuesday, 2017-07-04 Index Thursday, 2017-07-06 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!