Friday, 2014-01-24

« Thursday, 2014-01-23 Index Saturday, 2014-01-25 »
openstackgerritAndrew Lazarev proposed a change to openstack/savanna: Fixed typo in unit tests utility method  https://review.openstack.org/6880500:03
mattfi just figured out what bugs me about "data-sources" -- we use them for both input (source) and output (sink)00:10
*** rnirmal has quit IRC00:12
*** matsuhashi has joined #savanna00:27
openstackgerritAndrew Lazarev proposed a change to openstack/savanna: Fixed potential problems in test_periodic.py  https://review.openstack.org/6881400:46
openstackgerritAndrew Lazarev proposed a change to openstack/savanna: Removed unused parameter in periodic methods  https://review.openstack.org/6881500:52
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Refactor display of a single data-source  https://review.openstack.org/6881901:16
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Add data-source-create to CLI  https://review.openstack.org/6882001:16
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Add data-source-create to CLI  https://review.openstack.org/6882001:23
*** xuhanp has joined #savanna01:27
openstackgerritJenkins proposed a change to openstack/savanna: Updated from global requirements  https://review.openstack.org/6882401:31
*** nosnos has joined #savanna01:35
*** alazarev has quit IRC01:43
*** jcooley_ has joined #savanna01:57
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Send credentials only when they are provided  https://review.openstack.org/6882902:00
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Add data-source-create to CLI  https://review.openstack.org/6882002:00
*** jcooley_ has quit IRC02:00
*** jcooley_ has joined #savanna02:02
*** jcooley_ has quit IRC02:17
*** jcooley_ has joined #savanna02:18
*** jcooley_ has quit IRC02:19
*** xuhanp has quit IRC02:33
*** jcooley_ has joined #savanna02:35
*** xuhanp has joined #savanna02:47
*** jcooley_ has quit IRC02:49
*** matsuhashi has quit IRC03:14
*** sballe has quit IRC03:33
*** sballe has joined #savanna03:34
*** xuhanp has quit IRC03:36
*** xuhanp has joined #savanna03:53
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Refactor display of a job binary  https://review.openstack.org/6884203:55
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Add job-binary-create to CLI  https://review.openstack.org/6884303:55
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Refactor display of job template  https://review.openstack.org/6884403:55
*** sballe has quit IRC03:56
*** alazarev has joined #savanna03:58
*** jcooley_ has joined #savanna04:13
*** jcooley_ has joined #savanna04:13
*** matsuhashi has joined #savanna04:39
*** akuznetsov has joined #savanna05:11
*** SergeyLukjanov_ is now known as SergeyLukjanov05:22
*** IvanBerezovskiy has quit IRC06:06
openstackgerritJenkins proposed a change to openstack/savanna: Imported Translations from Transifex  https://review.openstack.org/6833606:06
*** jcooley_ has quit IRC06:13
*** SergeyLukjanov is now known as SergeyLukjanov_06:29
*** nadya__ has joined #savanna06:49
*** Ch00k has joined #savanna06:49
*** alazarev has quit IRC06:50
*** nadya__ has quit IRC06:52
*** alazarev has joined #savanna07:02
*** jcooley_ has joined #savanna07:07
*** alazarev has quit IRC07:10
*** jcooley_ has quit IRC07:12
*** aignatov_ is now known as aignatov07:13
*** jcooley_ has joined #savanna07:14
*** NikitaKonovalov_ is now known as NikitaKonovalov07:16
*** DinaBelova_ is now known as DinaBelova07:18
*** jcooley_ has quit IRC07:18
*** SergeyLukjanov_ is now known as SergeyLukjanov07:22
*** SergeyLukjanov is now known as SergeyLukjanov_a07:39
*** SergeyLukjanov_a is now known as SergeyLukjanov_07:40
*** matsuhashi has quit IRC07:42
*** nosnos has quit IRC07:43
*** aignatov is now known as aignatov_07:47
*** SergeyLukjanov_ is now known as SergeyLukjanov07:49
*** Ch00k has quit IRC08:00
*** jcooley_ has joined #savanna08:02
*** jcooley_ has quit IRC08:36
*** aignatov_ is now known as aignatov08:40
*** aignatov is now known as aignatov_08:42
openstackgerritYaroslav Lobankov proposed a change to openstack/savanna: Fixed HDP plugin to support Heat engine  https://review.openstack.org/6598108:44
*** aignatov_ is now known as aignatov08:44
openstackgerritA change was merged to openstack/savanna: Fixed typo in unit tests utility method  https://review.openstack.org/6880508:48
*** IvanBerezovskiy has joined #savanna09:00
*** dmitryme has joined #savanna09:13
*** Ch00k has joined #savanna09:21
*** jcooley_ has joined #savanna09:26
*** jcooley_ has quit IRC09:28
*** jcooley_ has joined #savanna09:29
*** SergeyLukjanov is now known as SergeyLukjanov_a09:31
*** SergeyLukjanov_a is now known as SergeyLukjanov_09:32
*** jcooley_ has quit IRC09:35
*** DinaBelova is now known as DinaBelova_09:42
*** nadya__ has joined #savanna09:52
*** Ch00k has quit IRC09:53
*** Ch00k has joined #savanna09:54
*** nadya__ has quit IRC09:57
*** Ch00k has quit IRC09:58
*** Ch00k has joined #savanna09:59
*** DinaBelova_ is now known as DinaBelova10:10
*** Ch00k has quit IRC10:10
*** Ch00k has joined #savanna10:11
*** aignatov is now known as aignatov_10:13
*** Ch00k has quit IRC10:16
*** Ch00k has joined #savanna10:19
*** jcooley_ has joined #savanna10:20
*** jcooley_ has quit IRC10:25
openstackgerritYaroslav Lobankov proposed a change to openstack/savanna: Fixed HDP plugin to support Heat engine  https://review.openstack.org/6598110:31
*** aignatov_ is now known as aignatov10:41
*** Ch00k has quit IRC10:44
*** Ch00k has joined #savanna10:46
*** xuhanp has quit IRC10:50
*** DinaBelova is now known as DinaBelova_10:54
*** Ch00k has quit IRC11:04
*** Ch00k has joined #savanna11:05
openstackgerritA change was merged to openstack/savanna: Update install guide  https://review.openstack.org/6836411:11
openstackgerritA change was merged to openstack/savanna: Updated from global requirements  https://review.openstack.org/6882411:11
openstackgerritA change was merged to openstack/savanna: Make error logging more safe  https://review.openstack.org/6413211:11
*** jcooley_ has joined #savanna11:15
*** Ch00k has quit IRC11:18
*** jcooley_ has quit IRC11:19
*** Ch00k has joined #savanna11:20
*** Ch00k has quit IRC11:33
*** Ch00k has joined #savanna11:35
*** Ch00k has quit IRC11:46
*** Ch00k has joined #savanna11:47
*** _ruhe is now known as ruhe11:49
*** NikitaKonovalov is now known as NikitaKonovalov_11:49
*** jcooley_ has joined #savanna12:09
*** Ch00k has quit IRC12:11
*** Ch00k has joined #savanna12:12
*** jcooley_ has quit IRC12:18
*** Ch00k has quit IRC12:43
*** Ch00k has joined #savanna12:44
*** dmitryme has quit IRC12:51
*** NikitaKonovalov_ is now known as NikitaKonovalov12:56
*** Ch00k has quit IRC13:00
*** DinaBelova_ is now known as DinaBelova13:00
*** Ch00k has joined #savanna13:01
*** SergeyLukjanov_ is now known as SergeyLukjanov13:01
*** SergeyLukjanov is now known as SergeyLukjanov_13:02
*** SergeyLukjanov_ is now known as SergeyLukjanov13:03
*** jcooley_ has joined #savanna13:04
*** NikitaKonovalov is now known as NikitaKonovalov_13:06
*** NikitaKonovalov_ is now known as NikitaKonovalov13:06
*** Ch00k has quit IRC13:07
*** ruhe is now known as _ruhe13:07
*** Ch00k has joined #savanna13:08
*** jcooley_ has quit IRC13:08
*** SergeyLukjanov is now known as SergeyLukjanov_13:11
*** dmitryme has joined #savanna13:12
*** Ch00k has quit IRC13:15
*** Ch00k has joined #savanna13:16
*** _ruhe is now known as ruhe13:18
*** Ch00k has quit IRC13:23
*** NikitaKonovalov is now known as NikitaKonovalov_13:23
*** SergeyLukjanov_ is now known as SergeyLukjanov13:24
*** Ch00k has joined #savanna13:26
*** NikitaKonovalov_ is now known as NikitaKonovalov13:26
*** Ch00k has quit IRC13:30
*** qwerty_nor has joined #savanna13:31
*** Ch00k has joined #savanna13:33
*** tmckay has quit IRC13:33
*** openstackgerrit has quit IRC13:42
*** _crobertsrh is now known as crobertsrh13:42
*** mattf has quit IRC13:43
*** mattf has joined #savanna13:44
*** DinaBelova is now known as DinaBelova_13:50
*** DinaBelova_ is now known as DinaBelova13:52
*** Ch00k has quit IRC13:55
*** Ch00k has joined #savanna13:55
*** jcooley_ has joined #savanna13:58
*** jcooley_ has quit IRC14:02
*** SergeyLukjanov is now known as SergeyLukjanov_14:07
*** SergeyLukjanov_ is now known as SergeyLukjanov14:09
*** sballe has joined #savanna14:21
Jayunit100good morning savanna14:24
mattfmorning jay14:29
*** tmckay has joined #savanna14:32
ruheJayunit100: morning, and fyi for some people here it's evening already :)14:33
*** SergeyLukjanov is now known as SergeyLukjanov_14:41
*** SergeyLukjanov_ is now known as SergeyLukjanov14:41
*** alazarev has joined #savanna14:51
*** SergeyLukjanov is now known as SergeyLukjanov_14:51
*** jcooley_ has joined #savanna14:52
*** Ch00k has quit IRC14:54
*** Ch00k has joined #savanna14:55
*** Ch00k has quit IRC14:55
*** jcooley_ has quit IRC14:57
*** aignatov is now known as aignatov_15:14
*** SergeyLukjanov_ is now known as SergeyLukjanov15:19
*** aignatov_ is now known as aignatov15:22
aignatovyeah, evening15:24
tmckayaignatov, thoughts on mapreduce streaming?  Review: Choice is new job type, or extra fields in a mapreduce Job indicating streaming mapper/reducer.  UI could treat it differently, even if the same job type is used.  crobertsrh I'm sure has opinions too.15:31
tmckaydifference in the workflow is just one new section, <streaming/> with a mapper and a reducer15:32
crobertsrhNew job type might be easiest from UI side, but it should be workable either way15:32
tmckayAnd, to work well, it needs files and archives tags too (but that would apply to all job types potentially)15:32
tmckayI could go either way.  Thinking about userdocs too -- what would be better for a user?15:33
tmckayit really just comes down to a difference in specifying mapper/reducer, and uploading those if they aren't already in hdfs15:34
aignatovtmckay: for user it would be better to reflect our actions in edp to ooze native actions15:38
aignatovimho15:38
tmckayaignatov, that would argue then for using the MapReduce job type with extensions for streaming15:44
aignatovtmckay: exactly :)15:45
*** jcooley_ has joined #savanna15:46
tmckayaignatov, crobertsrh, okay, lets go with same job type, and I'll work on defining the "streaming" stuff in the Job15:47
crobertsrhtmckay:  ack15:48
aignatovtmckay: approved, did you filed bp for that?15:49
*** ylobankov has left #savanna15:49
*** jcooley_ has quit IRC15:51
*** NikitaKonovalov is now known as NikitaKonovalov_16:00
*** jcooley_ has joined #savanna16:03
*** jcooley_ has quit IRC16:05
*** jcooley_ has joined #savanna16:05
*** jcooley_ has quit IRC16:10
*** jcooley_ has joined #savanna16:12
*** openstackgerrit has joined #savanna16:12
Jayunit100tmckay: for map reduce streaming you should (IMO) have options, not new job type.  The reason is that if you start creating new job types, you are imposing an artificial hierarchy into the map reduce programming paradigm.  Streaming is an option for configuring a standard job… ui should defiantly reflect that .16:13
tmckayaignatov, yes, https://blueprints.launchpad.net/savanna/+spec/edp-oozie-streaming-mapreduce16:15
tmckayJayunit100, agreed16:15
aignatovtmckay: thx16:15
Jayunit100:)16:16
Jayunit100jayunit100  will try keep an eye on savanna project from now on and provide unsolicited opinions16:16
tmckayJayunit100, please, do.  I aimed at aignatov and crobertsrh but I was secretly hoping for comments from everybody.  Otherwise, it shows up in CR reviews later ;-)16:17
*** NikitaKonovalov_ is now known as NikitaKonovalov16:25
*** IvanBerezovskiy has left #savanna16:28
Jayunit100unsolicited request for contributions : any java/mapreduce/hadoop/maven  hackers in here want to help me with https://github.com/jayunit100/bigpetstore  ?  if so ping and we can set up a hangout.16:29
*** NikitaKonovalov is now known as NikitaKonovalov_16:33
*** SergeyLukjanov is now known as SergeyLukjanov_16:37
*** DinaBelova is now known as DinaBelova_16:41
*** aignatov is now known as aignatov_16:45
*** aignatov_ is now known as aignatov16:53
*** aignatov is now known as aignatov_16:54
*** jcooley_ has quit IRC16:57
*** aignatov_ is now known as aignatov16:59
mattf^^ looks like bip has become popular16:59
*** aignatov is now known as aignatov_17:00
*** ruhe is now known as _ruhe17:04
*** alazarev has quit IRC17:07
*** dmitryme has quit IRC17:14
*** DinaBelova_ is now known as DinaBelova17:16
*** SergeyLukjanov_ is now known as SergeyLukjanov17:17
*** qwerty_nor has quit IRC17:19
openstackgerritTrevor McKay proposed a change to openstack/savanna: Add a config flag to disable cluster deletion after integration test.  https://review.openstack.org/6868817:20
openstackgerritTrevor McKay proposed a change to openstack/savanna: Add a config flag to disable cluster deletion after integration test  https://review.openstack.org/6868817:23
*** qwerty_nor has joined #savanna17:24
*** rnirmal has joined #savanna17:26
openstackgerritJonathan Maron proposed a change to openstack/savanna: Enable EDP on private neutron networks  https://review.openstack.org/6351717:28
*** DinaBelova is now known as DinaBelova_17:41
*** SergeyLukjanov is now known as SergeyLukjanov_a17:42
*** DinaBelova_ is now known as DinaBelova17:42
*** SergeyLukjanov_a is now known as SergeyLukjanov_17:43
*** mattf is now known as _mattf17:46
*** _mattf is now known as mattf17:47
*** mattf has quit IRC17:48
*** mattf has joined #savanna17:48
*** alazarev has joined #savanna17:48
*** aignatov_ is now known as aignatov17:49
*** aignatov is now known as aignatov_17:51
*** DinaBelova is now known as DinaBelova_17:57
openstackgerritAndrew Lazarev proposed a change to openstack/savanna: Fixed potential problems in test_periodic.py  https://review.openstack.org/6881418:01
openstackgerritAndrew Lazarev proposed a change to openstack/savanna: Fixed potential problems in test_periodic.py  https://review.openstack.org/6881418:03
openstackgerritAndrew Lazarev proposed a change to openstack/savanna: Fixed potential problems in test_periodic.py  https://review.openstack.org/6881418:04
*** Ch00k has joined #savanna18:07
openstackgerritAndrew Lazarev proposed a change to openstack/savanna: Fixed reading topology file with newline at the end  https://review.openstack.org/6852518:11
*** SergeyLukjanov_ is now known as SergeyLukjanov18:12
SergeyLukjanovmattf, I'm using znc :)18:13
SergeyLukjanovmattf, and I've make accounts for some folks18:13
mattfahh, that splains it18:17
tmckaycrobertsrh, ping, thinking out loud18:23
tmckaycrobertsrh, so for streaming mapreduce, there is a case where neither libs nor mains are required.  This is when the mapper and reducer executables are already in hdfs.  So the JSON validation potentially changes for MapReduce jobs.18:24
crobertsrhOk.  As long as it's sane, I should be able to make it happen.  Or rather the client library should be able to make it happen.18:25
tmckayI'm imagining a checkbox on the job creation UI, "[] streaming" that is greyed out except if job type is set to mapreduce18:25
tmckayIf you select mapreduce and check the box, it sets a boolean in the Job object on the api side and eases validation.  impacts libs/mains (still working on that)18:26
*** DinaBelova_ is now known as DinaBelova18:26
crobertsrhthat is doable18:26
tmckayThen, on job launch, we need to detect that and get the streaming stuff into hints.  Sound okay?18:26
tmckayso at that point the streaming mapper/reducer would be set instead of the current mapred.mapper.class or whatever it is18:27
crobertsrhI think that sounds reasonable18:38
tmckaycrobertsrh, hmm, as an aside it looks like we can get rid of the 'Jar' -> 'MapReduce' compat stuff.  The UI does MapReduce now, correct?18:38
crobertsrhYes.  Does the backend convert it in the database for old jobs?18:39
tmckayah, that's the rub, isn't it?  I wonder if that can be done with Alembic18:39
crobertsrhI bet it could be18:40
*** nadya__ has joined #savanna18:40
tmckayI know you can change columns, I wonder if you can modify values.  Thanks for reminding me.18:40
* tmckay goes to add a bug18:41
openstackgerritJonathan Maron proposed a change to openstack/savanna: Enable EDP on private neutron networks  https://review.openstack.org/6351719:10
*** jcooley_ has joined #savanna19:11
*** jcooley_ has quit IRC19:30
*** akuznetsov has quit IRC19:47
*** ErikB has joined #savanna19:52
*** nadya__ has quit IRC19:56
*** jcooley_ has joined #savanna19:59
*** Ch00k has quit IRC20:08
*** nadya__ has joined #savanna20:11
tmckaymattf, ping20:32
tmckayre https://review.openstack.org/#/c/68829/120:32
mattftmckay, pong20:38
tmckaymattf, hey there.  just making sure you saw aignatov's comment on patch set 1, I missed it because of the trivial rebase20:40
mattfi did miss it20:41
mattfadding default=None is a good idea20:41
tmckayI didn't understand his "not much different" comment on patch set 2 :)20:41
*** DinaBelova is now known as DinaBelova_20:42
*** crobertsrh is now known as _crobertsrh20:42
*** jcooley_ has quit IRC20:46
*** jcooley_ has joined #savanna20:48
*** _ruhe is now known as ruhe20:53
*** jcooley_ has quit IRC20:54
mattftmckay, thanks for the pointer. i'm trying to verify the new change, but i've lost connectivity to my cluster. argh.20:55
tmckaynp20:55
*** jcooley_ has joined #savanna20:57
mattfopenstack-config --set /etc/savanna/savanna.conf DEFAULT os_auth_host $(hostname)21:00
*** jcooley_ has quit IRC21:00
mattfmaybe helpful21:00
mattfweird, i can access it w/ nc, but not the cli21:08
*** nadya__ has quit IRC21:09
*** jcooley_ has joined #savanna21:12
*** jcooley_ has quit IRC21:12
*** ErikB has quit IRC21:24
openstackgerritTrevor McKay proposed a change to openstack/savanna: Require "libs" for MapReduce and Java jobs and disallow "mains"  https://review.openstack.org/6901921:25
* mattf starts to suspect his isp is messing with him21:26
*** ruhe is now known as _ruhe21:33
*** SergeyLukjanov is now known as SergeyLukjanov_21:39
*** qwerty_nor has quit IRC21:43
*** ErikB has joined #savanna21:50
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Send credentials only when they are provided  https://review.openstack.org/6882922:03
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Refactor display of job template  https://review.openstack.org/6884422:03
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Refactor display of a job binary  https://review.openstack.org/6884222:03
openstackgerritMatthew Farrellee proposed a change to openstack/python-savannaclient: Add job-binary-create to CLI  https://review.openstack.org/6884322:03
*** jcooley_ has joined #savanna22:19
*** jcooley_ has quit IRC22:29
openstackgerritTrevor McKay proposed a change to openstack/savanna: Allow boolean "streaming" in Job JSON  https://review.openstack.org/6903522:34
*** tmckay has left #savanna22:39
openstackgerritJenkins proposed a change to openstack/python-savannaclient: Updated from global requirements  https://review.openstack.org/6904422:41
openstackgerritJenkins proposed a change to openstack/savanna: Updated from global requirements  https://review.openstack.org/6904522:41
*** IlyaE has joined #savanna22:45
*** ErikB has quit IRC23:07
*** ErikB has joined #savanna23:36
*** Ch00k has joined #savanna23:39
*** Ch00k has quit IRC23:41
*** Ch00k_ has joined #savanna23:41
*** rnirmal has quit IRC23:52
*** ErikB has quit IRC23:56
« Thursday, 2014-01-23 Index Saturday, 2014-01-25 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!