Monday, 2014-01-20

*** nosnos has joined #savanna01:24
mattffyi, reading nova, cinder and keystone code, they do name|id on the client side!02:26
*** IlyaE has joined #savanna02:47
*** vodopianow has joined #savanna04:09
*** vodopianow has quit IRC04:14
*** akuznetsov has joined #savanna04:16
*** IlyaE has quit IRC04:52
*** aignatov has joined #savanna05:12
*** aignatov has quit IRC05:17
*** IlyaE has joined #savanna05:17
*** DinaBelova_ is now known as DinaBelova05:40
*** SergeyLukjanov_ is now known as SergeyLukjanov05:40
*** akuznetsov has quit IRC05:46
*** nosnos has quit IRC05:52
*** nosnos_ has joined #savanna05:52
openstackgerritJenkins proposed a change to openstack/savanna: Imported Translations from Transifex  https://review.openstack.org/6763306:07
SergeyLukjanovmattf, sure, names should be resolved on client side06:07
SergeyLukjanovfor cli06:07
*** akuznetsov has joined #savanna06:13
*** akuznetsov has quit IRC06:13
*** akuznetsov has joined #savanna06:13
*** DinaBelova has quit IRC06:28
*** DinaBelova has joined #savanna06:31
*** nosnos_ has quit IRC06:52
*** nosnos has joined #savanna06:53
*** SergeyLukjanov is now known as SergeyLukjanov_a06:59
*** SergeyLukjanov_a is now known as SergeyLukjanov_07:00
*** SergeyLukjanov_ is now known as SergeyLukjanov07:08
*** SergeyLukjanov is now known as SergeyLukjanov_07:18
*** IlyaE has quit IRC07:42
*** NikitaKonovalov_ is now known as NikitaKonovalov07:43
*** SergeyLukjanov_ is now known as SergeyLukjanov07:53
openstackgerritA change was merged to openstack/savanna: Imported Translations from Transifex  https://review.openstack.org/6763308:08
*** SergeyLukjanov is now known as SergeyLukjanov_08:47
*** DinaBelova is now known as DinaBelova_08:48
*** dmitryme has joined #savanna08:58
*** aignatov_ has joined #savanna09:24
*** SergeyLukjanov_ is now known as SergeyLukjanov09:43
*** SergeyLukjanov is now known as SergeyLukjanov_09:48
*** IvanBerezovskiy has joined #savanna09:49
aignatov_mattf, ping09:59
openstackgerritSergey Reshetnyak proposed a change to openstack/savanna: Add code to configure cluster for external hdfs  https://review.openstack.org/4782810:11
openstackgerritSergey Reshetnyak proposed a change to openstack/savanna: Add alembic migration tool to sqlalchemy  https://review.openstack.org/6403210:23
*** _ruhe is now known as ruhe10:42
*** Ch00k has joined #savanna11:55
Ch00khi11:56
Ch00kI'm having troubles starting my job in my savanna cluster11:56
Ch00kit gets into KILLED state all the time and I get the following in oozie.log: https://gist.github.com/Ch00k/851882011:57
Ch00kThe job I try to run is this one https://github.com/openstack/savanna-extra/tree/master/edp-examples/pig-job11:57
aignatov_hi, Ch00k, from the ooze log it's only clear that job doesn't exist when ooze tries to run it12:02
Ch00kaignatov_: but it is in savanna dashboard12:03
*** dmitryme has quit IRC12:03
aignatov_can you make sure that pig script you are trying to run is uploaded to the hdfs?12:03
Ch00kaignatov_: how can I check that?12:03
aignatov_try to run job again and collect debug logs from savanna12:04
Ch00kok12:05
*** ylobankov has joined #savanna12:07
Ch00kaignatov_: https://gist.github.com/Ch00k/8518942/raw/ad60b9cd80fab435ef8235b5cee64f6faa134d27/debug12:09
*** ruhe is now known as _ruhe12:09
Ch00kI don't see any errors there12:12
Ch00kand I see my binaries are uploaded to hdfs12:13
aignatov_yes, I see, indeed there is no exceptions or some suspicious logs12:15
aignatov_another way to check is to run ooze job not from savanna and manually, maybe there are some ooze issues itself12:17
Ch00kI see12:17
Ch00kin savanna dashboard my cluster reports oozie url as http://172.24.4.228:1100012:24
Ch00kbut when I run oozie jobs --oozie http://172.24.4.228:1100012:25
Ch00kI get Error: HTTP error code: 404 : Not Found12:25
Ch00kcurl http://172.24.4.228:11000 works fine12:27
*** _ruhe is now known as ruhe12:30
openstackgerritYaroslav Lobankov proposed a change to openstack/savanna: Fixed HDP plugin to support Heat engine  https://review.openstack.org/6598112:34
Ch00kaignatov_: is that the correct url to use?12:35
aignatov_you did curl from the same net when your savanna installed?12:38
aignatov_not sure, do you have security groups with opened access to port 11000 in your openstack installation?12:39
aignatov_it seems that there is some networking problem and savanna is not able to launch jobs via oozie12:40
Ch00kI am running oozie on the same machine where it is installed12:40
*** qwerty_nor has joined #savanna12:41
Ch00kand this same url is also available from the machine where openstack is installed12:41
Ch00kso no networking issues12:41
*** dmitryme has joined #savanna12:52
*** Ch00k has quit IRC12:54
dmitrymeHey Ch00k, do you run 'curl' and 'ooze' utilities from the same machine?12:56
*** SergeyLukjanov_ is now known as SergeyLukjanov12:58
*** DinaBelova_ is now known as DinaBelova12:58
*** Ch00k has joined #savanna12:59
dmitrymeHey Ch00k, seems like you've missed my question. Do you run 'curl' and 'ooze' utilities from the same machine?13:05
Ch00kdmitryme: yes13:05
Ch00kdmitryme: and this is the machine that oozie server is running on13:06
Ch00kdmitryme: https://gist.github.com/Ch00k/8519574 this is the output of both13:07
Ch00kdmitryme: and this is how oozie was started, maybe it will help https://gist.github.com/Ch00k/851958113:08
dmitrymeCh00k: got it, let me check it on my setup13:09
*** ruhe is now known as _ruhe13:11
aignatov_Ch00k: how many datanodes are  you using in your setup?13:13
mattfaignatov_, pong13:15
Ch00kaignatov_: one13:15
aignatov_mattf: I'm again about https://review.openstack.org/#/c/64210/13:16
aignatov_:)13:16
mattfthat the heat stuff?13:16
aignatov_yes, heat13:16
Ch00kaignatov_: one node: oozie, jobtracker, namenode, the other node: datanode, tasktracker13:16
mattfgotcha, thanks for the ping13:17
aignatov_Ch00k: there might be a problem that Oozzie is not started properly because of 1 DataNode and dfs.replication=313:18
aignatov_Oozie tries to put libs during the start to the hdfs and they are not replicated in time and than Error13:18
Ch00kaignatov_: hm, do I need 3 datanodes?13:18
aignatov_Ch00k: yes, or decrease replication factor to 113:19
aignatov_and use 1 DataNode in that case13:19
Ch00kI'll try, thanks13:20
Ch00kaignatov_: wait, where are you seeing this replication setting?13:20
aignatov_I know, 3 is by default :)13:20
Ch00kI mean where do I need to change it? :)13:21
aignatov_during the creating cluster template in the HDFS setting tab you will see it13:21
Ch00kah, ok, so I need to recreate the cluster, right?13:22
aignatov_yes, recreating cluster itself could help you ^^13:22
*** SergeyLukjanov is now known as SergeyLukjanov_13:26
*** _ruhe is now known as ruhe13:29
*** tmckay has joined #savanna13:29
*** Ch00k has quit IRC13:33
*** Ch00k has joined #savanna13:34
dmitrymeCh00k: looks like I understood what is wrong: the oozie URL should have a path, i.e.13:34
dmitryme./oozie jobs -oozie http://172.24.4.228:11000/oozie13:34
Ch00kdmitryme: yep, it works with /oozie, thanks13:36
dmitrymeCh00k, no problem13:36
Ch00kaignatov_: dmitryme after re-creating the cluster woth replication = 1 the exception in oozie log is no longer shown but the job still gets into KILLED state13:36
Ch00kI will try it manually13:37
Ch00kok, looks like I need to specify -config option for the job, where do I find that config file?13:39
aignatov_Ch00k: job.properties, actually savanna creates it internally you no need to create this one manually13:44
aignatov_buit if you want to run jobs not from savanna, just on the created cluster the following page will help you https://oozie.apache.org/docs/4.0.0/DG_Examples.html13:44
aignatov_*but13:46
*** _crobertsrh is now known as crobertsrh13:53
*** SergeyLukjanov_ is now known as SergeyLukjanov14:04
aignatov_mattf, I've found that test files in savannaclient have "nova" in path  https://github.com/openstack/python-savannaclient/tree/master/savannaclient/tests/nova14:04
aignatov_is this expected?14:05
*** moisha has joined #savanna14:08
Ch00kaignatov_: I am having the same org.apache.oozie.command.CommandException: E0604: Job does not exist [None] exception with a 3- and 4-worker node cluster14:10
Ch00kso the problem is not the replication14:10
mattfaignatov_, it is. i left it there because those tests are mostly taken from nova. i expect i'll remove the "nova" bit at some point in the future.14:11
*** moisha has quit IRC14:11
aignatov_ok, please file the bug about that, maybe someone from the community will fix that before you :) it's low-hanging-fruit14:12
mattfhmm bug to say "we copied client tests from nova and stored them in a dir named nova" ?14:14
mattfwhat would you prefer the lay to be?14:14
mattfaignatov_, it doesn't seem specifically wrong to me to have "nova" in a path name, so i'm having trouble formulating a bug statement14:18
aignatov_just 'Rename/remove nova in unit tests path'14:20
aignatov_Ch00k: but was you able to run ooze job manual?14:20
aignatov_*manually14:21
Ch00kaignatov_: well, I did not try actually14:21
openstackgerritSergey Reshetnyak proposed a change to openstack/savanna: WIP Add integration tests to Intel plugin WIP  https://review.openstack.org/6681214:21
Ch00kaignatov_: I don't quite understranf how to create a job.properties file14:25
aignatov_in the ooze node you can find /opt/oozie/oozie-examples-4.0.0.tar.gz14:25
aignatov_untag it, there are many examples of jobs14:26
aignatov_*untar14:26
aignatov_in the above link https://oozie.apache.org/docs/4.0.0/DG_Examples.html there is an example of how to run simple map-reduce example14:27
openstackgerritIvan Berezovskiy proposed a change to openstack/savanna-image-elements: Remove image size fixing for Fedora cloud image  https://review.openstack.org/6783914:28
Ch00kaignatov_: is there any way to get the job.properties that savanna is using to run the job?14:28
aignatov_yes, i can find it in /user/hadoop/spaces/45148468-3a16-4988-a0ea-e6ecc6a98e6d14:30
aignatov_this path I've took from your logs above14:30
aignatov_savanna logs I  mean14:30
aignatov_each job creates new uuid like path in the tail, you can see all jobs submitted to the ozies in the cluster's hdfs14:32
aignatov_in /user/hadoop/14:32
aignatov_the command is "hadoop fs -ls /user/hadoop/NAME_OF_YOUR_JOB/SOME_UUID"14:33
*** Ch00k has quit IRC14:33
*** Ch00k has joined #savanna14:35
Ch00kaignatov_: but there is no job.properties there because they must be stored on the local fs not hdfs14:35
tmckayaignatov_, ping14:39
aignatov_tmckay: pong14:40
tmckayaignatov_, hi.  The good news is that my Java integration test job launched, but I must have something wrong.  I got a "KILLED".  Is there a way to get at the Oozie logs for a job that ran on savanna-ci?14:41
tmckayIf it's killed, then Oozie threw an error somewhere14:42
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Refactor display of node group template  https://review.openstack.org/6766914:43
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Refactor display of cluster template  https://review.openstack.org/6767014:43
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Refactor display of cluster  https://review.openstack.org/6767114:43
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Add cluster-template-show command to CLI  https://review.openstack.org/6766514:43
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Add --json option to node-group-template-show  https://review.openstack.org/6767214:43
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Add --json option to cluster-template-show  https://review.openstack.org/6767314:43
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Add --json option to cluster-show  https://review.openstack.org/6767414:43
aignatov_tmckay: I'm afraid it is imposible to take that, because savanna-ci kills clusters after each tests runninig14:44
tmckayaignatov_, hehe, okay.  So I have to play detective.  Thanks.14:45
*** dmitryme has quit IRC14:48
*** dmitryme has joined #savanna14:49
*** SergeyLukjanov is now known as SergeyLukjanov_a14:51
*** SergeyLukjanov_a is now known as SergeyLukjanov_14:52
*** nosnos has quit IRC15:06
*** Ch00k has quit IRC15:06
openstackgerritJonathan Maron proposed a change to openstack/savanna: Update Ambari Repo location and services refactoring  https://review.openstack.org/6757715:11
*** DinaBelova is now known as DinaBelova_15:16
*** SergeyLukjanov_ is now known as SergeyLukjanov15:19
openstackgerritTrevor McKay proposed a change to openstack/savanna: Ignore key/value pairs with empty keys in workflow generation  https://review.openstack.org/6749715:19
*** DinaBelova_ is now known as DinaBelova15:20
openstackgerritTrevor McKay proposed a change to openstack/savanna: Ignore key/value pairs with empty keys in workflow generation  https://review.openstack.org/6749715:21
*** mascotte has joined #savanna15:25
*** mascotte has quit IRC15:32
openstackgerritA change was merged to openstack/savanna: Add support for HBase in HDP plugin  https://review.openstack.org/6061615:42
*** 23LAAXST3 has joined #savanna15:45
*** lukout has joined #savanna15:45
*** pivenherlitz has joined #savanna15:46
*** IlyaE has joined #savanna15:46
*** NikitaKonovalov is now known as NikitaKonovalov_15:47
*** Ch00k has joined #savanna15:51
*** lukout has quit IRC15:52
*** pivenherlitz has quit IRC15:53
*** 23LAAXST3 has quit IRC15:53
*** dmitryme has quit IRC15:58
openstackgerritYaroslav Lobankov proposed a change to openstack/savanna: Fixed HDP plugin to support Heat engine  https://review.openstack.org/6598116:00
*** SergeyLukjanov is now known as SergeyLukjanov_16:09
*** DinaBelova is now known as DinaBelova_16:17
*** dmitryme has joined #savanna16:23
*** schmigirilov has joined #savanna16:26
*** Jayunit100 has joined #savanna16:27
Jayunit100mattf: hello sir16:27
mattfJayunit100, howdy16:27
Jayunit100http://www.youtube.com/watch?v=OVnN4S52F3k16:27
Jayunit100thats an MLK video for steve16:27
Jayunit100my bad.  anyways...16:27
Jayunit100How do "outputS" actually get glued into to the jobs.? Normally in hadoop you specify output as part of the job binary.16:27
*** schmigirilov has quit IRC16:28
mattftmckay, you're in that code right now, afaik, can you field ^^ ?16:28
tmckayJayunit100, hi16:29
Jayunit100hi tmckay !16:29
tmckayJayunit100, so for all job types except "Java" which was recently added, you specify an input data_source and output data_source16:30
tmckayData_Sources are objects that wrap a swift URL, or currently on review, an hdfs url.16:30
tmckayWhen Savanna generates the Oozie workflow for you, it takes apart the input and output data source and sets the values in the workflow.xml16:31
Jayunit100ohhh okay . so really its all ooze based.16:31
Jayunit100mattf: i think at open stack CT i might focus some on how oozie works .16:32
*** NikitaKonovalov_ is now known as NikitaKonovalov16:32
tmckayyes, totally Oozie based.  We're working on adding more Oozie actions, including raw Oozie that you upload yourself16:32
Jayunit100okay that makes sense.16:32
*** silvertabby has joined #savanna16:33
mattfJayunit100, assuming everyone gets the overall picture. that makes sense if you've deep knowledge about oozie. we made a decision to use oozie as our workflow interface to the cluster.16:34
*** silvertabby has quit IRC16:35
*** makuta has joined #savanna16:36
Jayunit100i wonder if ooze supports, or ever will support, OSGI style apps16:37
*** makuta has quit IRC16:37
*** DinaBelova_ is now known as DinaBelova16:39
*** aignatov_ has quit IRC16:50
openstackgerritTrevor McKay proposed a change to openstack/savanna: Read Swift credentials from input_data OR output_data  https://review.openstack.org/6750916:50
*** bordache has joined #savanna16:56
*** ruhe is now known as _ruhe16:59
*** dmitryme has quit IRC17:00
*** bordache has quit IRC17:09
*** IlyaE has quit IRC17:23
mattfIvanBerezovskiy, welcome back17:23
openstackgerritA change was merged to openstack/python-savannaclient: Make plugin commands singular  https://review.openstack.org/6766017:25
openstackgerritA change was merged to openstack/python-savannaclient: Add node-group-template-show command to CLI  https://review.openstack.org/6766317:25
mattfand so it begins17:26
*** aignatov_ has joined #savanna17:41
*** dmitryme has joined #savanna17:45
*** _ruhe is now known as ruhe17:45
*** dmitryme has quit IRC17:46
*** aignatov_ has quit IRC17:47
*** IlyaE has joined #savanna17:47
openstackgerritA change was merged to openstack/savanna-image-elements: Remove image size fixing for Fedora cloud image  https://review.openstack.org/6783917:47
openstackgerritTrevor McKay proposed a change to openstack/savanna: Change configs["args"] to be a list for Pig jobs  https://review.openstack.org/6758817:48
*** SergeyLukjanov_ is now known as SergeyLukjanov17:52
openstackgerritTrevor McKay proposed a change to openstack/savanna: Change configs["args"] to be a list for Pig jobs  https://review.openstack.org/6758817:53
*** ruhe is now known as _ruhe17:57
*** NikitaKonovalov is now known as NikitaKonovalov_18:05
*** nadya__ has joined #savanna18:16
*** nadya__ has quit IRC18:17
*** mattf is now known as _mattf18:22
*** SergeyLukjanov is now known as SergeyLukjanov_a18:24
*** SergeyLukjanov_a is now known as SergeyLukjanov_18:25
*** _mattf is now known as mattf18:27
*** tmckay has left #savanna18:27
mattfSergeyLukjanov_, can i talk you into approving https://review.openstack.org/#/c/67665/ and https://review.openstack.org/#/c/67669/ ?18:30
mattfif anyone happens in CT and is interested in savanna - http://www.meetup.com/Openstack-Connecticut/events/156660582/18:32
*** IlyaE has quit IRC18:37
openstackgerritAndrew Lazarev proposed a change to openstack/savanna: [Vanilla] Updated docs to point to icehouse images  https://review.openstack.org/6756918:38
*** tmckay has joined #savanna18:40
*** dmitryme has joined #savanna18:45
*** IlyaE has joined #savanna18:55
*** SergeyLukjanov_ is now known as SergeyLukjanov19:02
*** aignatov_ has joined #savanna19:03
*** IlyaE has quit IRC19:03
*** IlyaE has joined #savanna19:23
*** SergeyLukjanov is now known as SergeyLukjanov_a19:31
*** SergeyLukjanov_a is now known as SergeyLukjanov_19:32
*** SergeyLukjanov_ is now known as SergeyLukjanov19:35
*** nadya__ has joined #savanna19:37
SergeyLukjanovmattf, done19:38
SergeyLukjanovmattf, nice to see that there are events about savanna :019:38
mattfprops to Jayunit100 for coming up to speed in a few days!19:39
SergeyLukjanovmattf, please, take a look on https://review.openstack.org/#/c/66722/ and apprv it ;)19:39
mattfbtw, i hope it isn't just me, but the patchsets i've been sending are the units that i like to see and review19:39
mattfdone19:40
SergeyLukjanovmattf, I've already reviewed them earlier19:40
mattfi'm saying i'm a fan of small, distinct changes19:41
mattfi groan every time i see a 1000+ loc review19:41
SergeyLukjanovagreed19:44
*** tmckay has quit IRC19:48
*** nadya__ has quit IRC19:50
crobertsrhakuznetsov:  I'm testing out https://review.openstack.org/#/c/47828/ and came across some issues.  I'm hoping that you might be able to steer me in the right direction (my questions are in the review notes).19:53
*** rnirmal has joined #savanna19:55
*** kiselar has joined #savanna19:58
*** _ruhe is now known as ruhe19:59
*** tmckay has joined #savanna20:03
*** tmckay has left #savanna20:04
*** tmckay has joined #savanna20:05
*** nadya__ has joined #savanna20:05
*** nadya__ has quit IRC20:10
*** aignatov_ has quit IRC20:16
*** aignatov_ has joined #savanna20:19
*** SergeyLukjanov is now known as SergeyLukjanov_20:22
*** NikitaKonovalov_ is now known as NikitaKonovalov20:45
*** dmitryme has quit IRC20:49
* tmckay is learning how to run integration tests on his laptop cluster from README.rst and hacking :)20:52
*** IlyaE has quit IRC20:54
*** IlyaE has joined #savanna20:55
*** rnirmal has quit IRC20:56
*** DinaBelova is now known as DinaBelova_21:00
openstackgerritA change was merged to openstack/python-savannaclient: Add cluster-template-show command to CLI  https://review.openstack.org/6766521:18
openstackgerritA change was merged to openstack/python-savannaclient: Refactor display of node group template  https://review.openstack.org/6766921:18
openstackgerritA change was merged to openstack/python-savannaclient: Refactor display of cluster template  https://review.openstack.org/6767021:18
openstackgerritA change was merged to openstack/python-savannaclient: Refactor display of cluster  https://review.openstack.org/6767121:18
openstackgerritA change was merged to openstack/python-savannaclient: Add --json option to node-group-template-show  https://review.openstack.org/6767221:18
openstackgerritA change was merged to openstack/python-savannaclient: Add --json option to cluster-template-show  https://review.openstack.org/6767321:19
openstackgerritA change was merged to openstack/python-savannaclient: Add --json option to cluster-show  https://review.openstack.org/6767421:19
openstackgerritA change was merged to openstack/python-savannaclient: Remove version from setup.cfg  https://review.openstack.org/6672221:19
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Add node-group-template-create to CLI  https://review.openstack.org/6796021:20
openstackgerritAndrew Lazarev proposed a change to openstack/savanna: [IDH] Added config controlling hadoop-swift.jar URL  https://review.openstack.org/6582121:20
mattfaignatov_, fyi, reading nova, cinder and keystone code, they do name|id on the client side!21:23
*** IlyaE has quit IRC21:32
*** aignatov_ has quit IRC21:34
*** ruhe is now known as _ruhe21:34
*** IlyaE has joined #savanna21:37
*** mattf is now known as _mattf21:44
*** NikitaKonovalov is now known as NikitaKonovalov_21:52
*** kiselar has quit IRC21:53
*** rnirmal has joined #savanna22:00
*** _mattf is now known as mattf22:01
*** crobertsrh is now known as _crobertsrh22:04
openstackgerritTrevor McKay proposed a change to openstack/savanna: Add integration test for Oozie java action  https://review.openstack.org/6641522:22
*** Ch00k has quit IRC22:25
*** Ch00k has joined #savanna22:26
*** tmckay has quit IRC22:26
*** Ch00k has quit IRC23:09
openstackgerritJonathan Maron proposed a change to openstack/savanna: Update Ambari Repo location and services refactoring  https://review.openstack.org/6757723:32
*** qwerty_nor has quit IRC23:50

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!