Monday, 2013-08-05

« Sunday, 2013-08-04 Index Tuesday, 2013-08-06 »
*** lastidiot has joined #savanna00:02
*** NikitaKonovalov has joined #savanna00:38
*** NikitaKonovalov has quit IRC00:43
*** lastidiot has quit IRC00:46
*** NikitaKonovalov has joined #savanna01:39
*** NikitaKonovalov has quit IRC01:43
*** ben_duyujie has joined #savanna01:55
*** lastidiot has joined #savanna02:05
*** NikitaKonovalov has joined #savanna02:39
*** NikitaKonovalov has quit IRC02:44
*** IlyaE has quit IRC03:39
*** SergeyLukjanov has joined #savanna03:39
*** NikitaKonovalov has joined #savanna03:40
*** NikitaKonovalov has quit IRC03:45
openstackgerritSergey Lukjanov proposed a change to stackforge/savanna: Move path manipulations into function  https://review.openstack.org/4014803:54
openstackgerritA change was merged to stackforge/savanna: Move path manipulations into function  https://review.openstack.org/4014804:06
*** ben_duyujie has quit IRC04:15
*** NikitaKonovalov has joined #savanna04:41
*** NikitaKonovalov has quit IRC04:45
*** lastidiot has quit IRC04:47
*** IlyaE has joined #savanna04:56
*** dina_belova has joined #savanna05:22
*** dina_belova has quit IRC05:25
*** NikitaKonovalov has joined #savanna05:32
*** NikitaKonovalov has quit IRC05:33
*** SergeyLukjanov has quit IRC05:40
*** ben_duyujie has joined #savanna06:14
*** ben_duyujie has quit IRC06:55
*** nprivalova has joined #savanna07:13
*** IlyaE has quit IRC07:40
*** ruhe has joined #savanna08:50
*** leeuwenrjj has joined #savanna09:07
openstackgerritSergey Reshetnyak proposed a change to stackforge/savanna: Refactoring hdp plugin  https://review.openstack.org/3973309:09
leeuwenrjjHi, I am trying to setup Savanna on our test OpenStack environment and I run into a traceback related to getting the IP network: http://pastebin.com/dnC1fdGf09:12
ruheleeuwenrjj, hi09:13
leeuwenrjjsavanna/service/networks.py seems to give the name of the network and it looks like it is expecting a cidr instead09:13
ruheare you using quantum?09:13
leeuwenrjjYes09:13
ruhesavanna doesn't support quantum/neutron yet. but it's in our plans: https://blueprints.launchpad.net/savanna/+spec/add-neutron-support09:14
leeuwenrjjOkay, is it just here that it needs a modification or is it all over the place?09:15
ruheit's in networks.py mostly. but it requires some modifications to make it work with floating IPs assigned by neutron.09:16
leeuwenrjjThx, if we decide to work on the blueprint I'll let you know. Have to talk to some people first here.09:19
*** NikitaKonovalov has joined #savanna09:37
*** dina_belova has joined #savanna09:37
*** SergeyLukjanov has joined #savanna09:38
*** NikitaKonovalov has quit IRC09:38
*** NikitaKonovalov has joined #savanna09:38
openstackgerritDmitry Mescheryakov proposed a change to stackforge/savanna: Fix foreign keys in new model  https://review.openstack.org/4018609:42
openstackgerritSergey Reshetnyak proposed a change to stackforge/savanna: Refactoring remote utils  https://review.openstack.org/3904309:49
*** NikitaKonovalov has quit IRC09:51
*** NikitaKonovalov has joined #savanna10:04
openstackgerritDmitry Mescheryakov proposed a change to stackforge/savanna: Fix foreign keys and table names in new model  https://review.openstack.org/4018610:04
*** dina_belova has quit IRC10:11
*** ruhe has quit IRC10:15
*** nprivalova has quit IRC10:16
openstackgerritDmitry Mescheryakov proposed a change to stackforge/savanna: Fix foreign keys and table names in new model  https://review.openstack.org/4018610:20
openstackgerritDmitry Mescheryakov proposed a change to stackforge/savanna: Fix foreign keys and table names in new model  https://review.openstack.org/4018610:26
openstackgerritA change was merged to stackforge/savanna: Fix foreign keys and table names in new model  https://review.openstack.org/4018610:31
*** nprivalova has joined #savanna10:35
*** key has joined #savanna10:38
openstackgerritSergey Lukjanov proposed a change to stackforge/savanna: Docs build fixed  https://review.openstack.org/4020010:38
openstackgerritSergey Lukjanov proposed a change to stackforge/savanna: Docs build fixed  https://review.openstack.org/4020010:40
*** dmitryme has joined #savanna10:40
*** akuznetsov has joined #savanna10:40
openstackgerritSergey Reshetnyak proposed a change to stackforge/savanna: Refactoring remote utils  https://review.openstack.org/3904310:42
*** ruhe has joined #savanna10:43
openstackgerritA change was merged to stackforge/savanna: Docs build fixed  https://review.openstack.org/4020010:45
*** ben_duyujie has joined #savanna10:59
*** dina_belova has joined #savanna11:02
*** qwerty_nor has joined #savanna11:04
*** NikitaKonovalov has quit IRC11:23
*** NikitaKonovalov has joined #savanna11:26
*** dina_belova has quit IRC11:59
openstackgerritNikita Konovalov proposed a change to stackforge/savanna: Conductor methods added  https://review.openstack.org/3990212:00
*** dina_belova has joined #savanna12:09
openstackgerritNadya Privalova proposed a change to stackforge/savanna: Workflow creator  https://review.openstack.org/3990612:18
openstackgerritNikita Konovalov proposed a change to stackforge/savanna: Conductor methods added  https://review.openstack.org/3990212:24
openstackgerritNadya Privalova proposed a change to stackforge/savanna: Workflow creator  https://review.openstack.org/3990612:26
*** _crobertsrh is now known as crobertsrh12:26
*** ruhe has quit IRC12:27
*** ruhe has joined #savanna12:27
*** akuznetsov has quit IRC12:59
openstackgerritDmitry Mescheryakov proposed a change to stackforge/savanna: Created savanna-db-manage script for new DB  https://review.openstack.org/4022013:01
openstackgerritNadya Privalova proposed a change to stackforge/savanna: Workflow creator  https://review.openstack.org/3990613:02
*** dmitryme has quit IRC13:03
openstackgerritNikita Konovalov proposed a change to stackforge/savanna: Conductor methods added  https://review.openstack.org/3990213:06
openstackgerritNikita Konovalov proposed a change to stackforge/savanna: Conductor methods added  https://review.openstack.org/3990213:08
openstackgerritA change was merged to stackforge/savanna: Conductor methods added  https://review.openstack.org/3990213:12
*** NikitaKonovalov has quit IRC13:14
openstackgerritNadya Privalova proposed a change to stackforge/savanna: Workflow creator  https://review.openstack.org/3990613:16
openstackgerritNadya Privalova proposed a change to stackforge/savanna: Workflow creator  https://review.openstack.org/3990613:17
*** ruhe has quit IRC13:18
*** ben_duyujie has quit IRC13:20
*** dina_belova has quit IRC13:20
*** dina_belova has joined #savanna13:21
*** ruhe has joined #savanna13:21
openstackgerritSergey Reshetnyak proposed a change to stackforge/savanna: Refactoring remote utils  https://review.openstack.org/3904313:21
*** dina_belova has quit IRC13:25
*** dina_belova has joined #savanna13:25
*** _tstclair is now known as tstclair13:26
*** NikitaKonovalov has joined #savanna13:28
*** dina_belova has quit IRC13:30
*** dmitryme has joined #savanna13:32
*** ruhe has quit IRC13:35
*** nprivalova has quit IRC13:41
openstackgerritJonathan Maron proposed a change to stackforge/savanna: Added validation checks to HDP plugin  https://review.openstack.org/3997313:41
*** ruhe has joined #savanna13:44
*** tmckayrh has joined #savanna13:46
*** nprivalova has joined #savanna13:47
*** dina_belova has joined #savanna13:48
*** NikitaKonovalov has quit IRC13:49
openstackgerritAlexander Kuznetsov proposed a change to stackforge/savanna: Added first version of model for EDP  https://review.openstack.org/4022713:56
tmckayrhnprivalova, I have a question for you about the Job Origin api13:58
*** ruhe has quit IRC14:00
tmckayrhor SergeyLukjanov, or others :)14:00
*** nprivalova has quit IRC14:01
*** dina_belova has quit IRC14:04
*** dmitryme has quit IRC14:07
*** nprivalova has joined #savanna14:13
nprivalovahi tmckayrh!14:13
tmckayrhnprivalova, hi!  Something is still a little fuzzy for me.  This quote from the blueprints:14:14
tmckayrh"* Get Job code, this code will be interact with plugin responsible for job source implementation"14:14
tmckayrhfrom Job Origin (formerly Job Source)14:14
tmckayrhnprivalova, do you know what "get" means precisely?  Are we talking about copying to hdfs?14:15
tmckayrhOr, maybe specify a destination for copy?14:15
tmckayrhOr, do we actually return a script in a JSON object? (in the case of non-binaries)14:15
tmckayrhSomething else?14:15
tmckayrhThis is one of the Job Origin component's methods, btw14:16
*** akuznetsov has joined #savanna14:16
*** rnirmal has joined #savanna14:17
*** dina_belova has joined #savanna14:17
tmckayrhakuznetsov, oh, you're here too now :)  Can you see the scrollback or should I repaste a question to nprivalova?14:17
*** dina_belova has quit IRC14:18
*** NikitaKonovalov has joined #savanna14:19
*** dina_belova has joined #savanna14:20
*** lastidiot has joined #savanna14:20
akuznetsovtmckayrh get means that Job Origin component has ability to put job code to HDFS, so it will be available for Oozie14:22
*** NikitaKonovalov has quit IRC14:23
*** NikitaKonovalov has joined #savanna14:23
tmckayrhakuznetsov, okay, thanks.  Always hdfs?  And, does the "get" call specify the destination, or does the result object tell the caller where the job went?14:23
akuznetsovtmckayrh another option is to use a swift for storing job code14:25
akuznetsovhdfs is better because in theory open stack cluster can work with swift14:26
akuznetsovand the data for elastic processing can be stored for example in NoSQL database14:27
tmckayrhakuznetsov, okay, so hdfs first maybe.  But the idea is that the Job Origin component copies a job from storage (git, savanna db, another hdfs, etc) to hdfs (or swift someday) where it can be read directly for execution by hadoop.  Correct?14:27
*** key4 has joined #savanna14:27
*** ruhe has joined #savanna14:27
*** key has quit IRC14:28
akuznetsovtmckayrh yes14:29
akuznetsovtmckayrh for first it should be able copy job code from Swift to HDFS14:30
*** lastidiot has quit IRC14:30
tmckayrhakuznetsov, okay, thanks.  My last question, then -- how much control does the caller of "get" have over where the file is copied to?  Does the caller specify the whole destination path, or part of the path?14:31
akuznetsovtmckayrh I think the caller should provide a path to hdfs dir where the job code will be stored14:32
tmckayrhakuznetsov, okay, with the filename preserved.  That makes sense.  Thank you, this fills in some gaps in the blueprint/api/sequence.  I'll see about adding extra detail in a few places.14:34
*** nprivalova has quit IRC14:46
*** dmitryme has joined #savanna14:46
openstackgerritSergey Reshetnyak proposed a change to stackforge/savanna: Refactoring remote utils  https://review.openstack.org/3904314:59
*** NikitaKonovalov has quit IRC15:03
openstackgerritA change was merged to stackforge/savanna: Fix Ganglia service start failure  https://review.openstack.org/3994615:04
openstackgerritA change was merged to stackforge/savanna: Workflow creator  https://review.openstack.org/3990615:06
*** nprivalova has joined #savanna15:08
*** rnirmal_ has joined #savanna15:12
openstackgerritDmitry Mescheryakov proposed a change to stackforge/savanna: Created savanna-db-manage script for new DB  https://review.openstack.org/4022015:12
*** rnirmal has quit IRC15:13
*** rnirmal_ is now known as rnirmal15:13
*** lastidiot has joined #savanna15:19
openstackgerritAlexander Kuznetsov proposed a change to stackforge/savanna: Added first version of model for EDP  https://review.openstack.org/4022715:20
*** nprivalova has quit IRC15:21
*** NikitaKonovalov has joined #savanna15:21
*** ruhe has quit IRC15:25
*** ruhe has joined #savanna15:32
*** tstclair has quit IRC15:34
*** tstclair has joined #savanna15:47
*** ruhe has quit IRC15:52
openstackgerritSergey Lukjanov proposed a change to stackforge/savanna: Temporarily use sphinx 1.2b1  https://review.openstack.org/4024215:53
*** dmitryme_ has joined #savanna15:54
*** dmitryme has quit IRC15:55
*** dmitryme_ is now known as dmitryme15:55
openstackgerritSergey Lukjanov proposed a change to stackforge/savanna: Temporarily use sphinx 1.2b1  https://review.openstack.org/4024215:56
openstackgerritSergey Reshetnyak proposed a change to stackforge/savanna: Refactoring remote utils  https://review.openstack.org/3904316:06
*** akuznetsov has quit IRC16:09
*** SergeyLukjanov has quit IRC16:17
*** NikitaKonovalov has quit IRC16:17
*** dina_belova has quit IRC16:21
*** dmitryme has quit IRC16:25
*** ruhe has joined #savanna16:26
tmckayrhstill honing in on the correct definitions for Job Origin Object on https://etherpad.openstack.org/savanna_API_draft_EDP_extensions16:33
tmckayrhIs anybody here to discuss?16:33
tmckayrhThere still seems to be a  mix in the Job Origin object example of values that 1) describe an endpoint, that is a storage depot and 2) a particular job16:35
tmckayrhThe example value ""URI": "swift://test/hello_word.q" names a specific job but I believe that should be ""URI": "swift://test/"16:35
tmckayrhAnd "job_type" probably can be a list, because more than 1 type may be stored at a location16:36
tmckayrhokay, modified https://etherpad.openstack.org/savanna_API_draft_EDP_extensions.  Optimistically assuming I am correct :)16:45
*** IlyaE has joined #savanna16:51
tmckayrhruhe, from our discussion on Thursday, we postulated a new component for storing uncompiled job source, and a build mechanism.  Should these be the same component, and what do you think about a name?16:53
tmckayrhruhe, if they are the same, we could call it "Job Build Component".  Trying to make a blueprint...16:54
tmckayrhI'm not sure I want to reuse "Job Source Component" since it used to mean something else....16:54
*** ruhe has quit IRC16:54
tmckayrhdoh, something I said :)16:54
*** dina_belova has joined #savanna17:08
*** NikitaKonovalov has joined #savanna17:18
*** NikitaKo_ has joined #savanna17:22
*** NikitaKonovalov has quit IRC17:22
*** NikitaKo_ has quit IRC17:22
*** SergeyLukjanov has joined #savanna17:22
*** NikitaKonovalov has joined #savanna17:22
*** NikitaKonovalov has quit IRC17:27
*** ruhe has joined #savanna17:31
ruhetmckayrh, hi17:33
ruhei think that JoSource should be a separate component from JobOrigin17:33
ruheand we need to find a new name for JobSource. new name should clearly indicate purpose of this component17:35
tmckayrhruhe, hi17:36
tmckayrhI created a blueprint called "Job Build  Component" but the question remains, should JobSource (uncompiled code management) and JobBuild be under one component or two?17:37
SergeyLukjanovbtw https://blueprints.launchpad.net/savanna/+spec/edp-job-build-component17:37
tmckayrhthanks17:37
SergeyLukjanovtmckayrh, hi!17:37
ruhetmckayrh, i guess we need pros and cons for two different approaches. if they're 2 separate component, there should be something above to manage them both17:39
tmckayrhI could see it either way.  In a two-component scenario, I suppose a Job Build Component would pull source code from a Job Source Component and then write the compiled output to the Job Origin Component.17:40
ruheif it's the same component, then it'll handle two different things: 1. Source code storages (git, svn, etc), 2. build systems: make, mvn, gradle, etc17:40
tmckayrhagreed.17:40
*** dmitryme has joined #savanna17:40
tmckayrhI wonder if that's too much under one umbrella17:40
ruhei think - too much17:40
tmckayrhIt would make the Job Storage Component very simple -- the analog to Job Origin.  Just register job source endpoints with location information.17:41
tmckayrhoops, ^^ I meant "Job Source"17:41
ruheit should also have a plugin mechanism for different storages. right?17:41
tmckayrhWhatever the new name is.17:42
tmckayrhyes17:42
ruheit should be easier for native english speakers to find a proper name :)17:42
*** IlyaE has quit IRC17:42
tmckayrhI am almost wondering if "Job Source" and "Job Origin" have the same code base.  The only difference is binary vs text in most cases17:42
tmckayrhhehe17:42
tmckayrhHow about "Raw Storage Component"17:43
tmckayrhor, rather boring, but "Uncompiled Source Component"17:43
ruhemaybe "SourceCodeComponent" ?17:44
ruheah17:44
ruhehmm17:44
tmckayrhOkay, I like that.  It's different enough from "Job Source" to not be confused with the past.17:44
ruhewe can add JobSourceCodeComponent to make it more clear17:45
ruhe*prepend Job17:45
tmckayrhokay.  So I should create a second blueprint and edit the description.17:46
ruhegreat17:46
tmckayrhruhe, what do you think about common code for JobSourceCodeCompnent and JobOrigin?  It seems like add/list/delete/get would be almost if not completely the same, just the file type is different.17:47
tmckayrhwell, "get" for JO copies from storage to hdfs, for example.  Not sure JobSourceCC has that function17:48
ruhehmm. i'm not sure about this. for SourceCode we have git, svn, etc. for JobOrigin - Swift, Gluster, HDFS, etc. API might be the same, but implementation is absolutely different17:48
tmckayrhruhe, I thought we also were considering git/svn/mercurial etc for JobOrigin.  With a copy to a hadoop compatible file system for execution.17:49
ruhewell, that might be useful in some scenarios. but who would store binaries in VCS?17:51
tmckayrhfrom the blueprint...  Maybe we have migrating ideas here :)17:51
SergeyLukjanovmaybe it'll be better to make build process transparent for end-user?17:51
tmckayrh"We should provide a different way to get a job source code for  execution. It can be stored in Git, Mercurial, Internal Savanna Database  and etc. The support of different component is done via plugin  mechanism. "17:51
tmckayrhI wonder if the Job Source component started off meaning raw job source code, and then evolved into "binaries", and then we renamed it "origin", and now we are talking about source code again, hehe17:52
tmckayrhruhe, SergeyLukjanov, distributed development impact ^^.  We need akuznetsov here to verify.17:54
ruhei'm not sure if he'll show up today. it's about 10PM here17:55
ruhegood point to coordinate is etherpad17:55
tmckayrhokay.  I think we have some historical confusion on terms/components, but I think we are almost there.  We might need to tweak the blueprints.17:56
*** dina_belova has quit IRC17:57
ruheyes. we absolutely need to update all blueprints to reflect current understanding of components17:57
tmckayrhOkay, I'll add something to the component etherpad17:57
*** IlyaE has joined #savanna17:57
ruhethanks17:57
ruhei'll tell akuznetsov about our discussion17:58
tmckayrh:) np.  I like to put ducks in a row.17:58
*** ruhe has quit IRC18:05
*** mattf is now known as _mattf18:07
*** _mattf is now known as mattf18:12
*** NikitaKonovalov has joined #savanna18:14
*** dina_belova has joined #savanna18:20
*** NikitaKonovalov has quit IRC18:32
*** NikitaKonovalov has joined #savanna18:33
openstackgerritA change was merged to stackforge/savanna: Added validation checks to HDP plugin  https://review.openstack.org/3997318:48
*** qwerty_nor has quit IRC18:50
*** NikitaKonovalov has quit IRC18:50
*** NikitaKonovalov has joined #savanna18:50
*** NikitaKonovalov has quit IRC19:06
*** NikitaKonovalov has joined #savanna19:07
openstackgerritSergey Lukjanov proposed a change to stackforge/savanna: Improve coverage calculation  https://review.openstack.org/4026919:12
tmckayrhSergeyLukjanov, ping19:13
SergeyLukjanovtmckayrh, pong19:13
tmckayrhSergeyLukjanov, still updating notes and things and thinking about how this works.  Is there any reason that we want to be able to list job storage locations (like directories)?  Or only the individual jobs?19:17
tmckayrhSergeyLukjanov, somewhere along the line I got the idea that we wanted to manage directories (add/list/delete locations for storing jobs), but I think maybe that was a misunderstanding.19:18
SergeyLukjanovhmm19:18
*** NikitaKonovalov has quit IRC19:18
tmckayrhI think maybe we just want the Job Origin Component to store binaries for individual jobs.19:19
*** NikitaKonovalov has joined #savanna19:19
tmckayrhThis would explain some of the collisions in the etherpads, etc.19:19
SergeyLukjanovaffair we want to store jobs and then be able to run some job on some cluster19:19
tmckayrhyes, ultimately19:20
tmckayrhI'm not sure we care about being able to list storage locations (such as "I have an hdfs filesystem over here, and this type of stuff is stored there")19:21
SergeyLukjanovI think that listing storage locations could be useful for end-user in UI19:27
tmckayrhSergeyLukjanov, yes, just like a filesystem.  Where might I find things?  Where could I put things?  I agree.19:30
*** NikitaKonovalov has quit IRC19:30
*** NikitaKonovalov has joined #savanna19:30
tmckayrhSo maybe there are 2 aspects to the UI: 1) store and retrieve specific job binaries by id and 2) find out about "filesystems"19:31
tmckayrhI think I was mixing the two together, but I think the current api as outlined should be/is #1 ^^19:31
tmckayrh#2 maybe needs other functions19:31
*** ruhe has joined #savanna19:38
tmckayrhSergeyLukjanov (and ruhe) maybe the Job Origin api just needs to be a little richer... Maybe the list function for example could do path globbing, or specify "storage type" or "job type" as a search filter.  that would narrow the results returned.19:40
*** ruhe has quit IRC19:41
tmckayrhIt still might return too much information, though, if you were trying to find filesystems.19:42
*** dina_belova has quit IRC19:43
openstackgerritA change was merged to stackforge/savanna: Improve coverage calculation  https://review.openstack.org/4026920:01
SergeyLukjanovtmckayrh, I hope we'll added filter support for all list ops20:02
SergeyLukjanovadd*20:03
tmckayrhagreed.  I'm trying to imagine how one might request the list of all binaries from the job origin component and then filter it to learn about possible filesystems to use for binary storage.20:04
tmckayrhWithout putting specific records in the savanna db to describe storage systems.20:05
*** SergeyLukjanov has quit IRC20:07
tmckayrhI suppose it should be possible based on URI, but then there is the question of depth20:07
*** dmitryme has quit IRC20:08
*** dmitryme has joined #savanna20:10
*** NikitaKonovalov has quit IRC20:29
*** NikitaKonovalov has joined #savanna20:30
*** NikitaKonovalov has quit IRC20:35
*** tstclair has quit IRC20:36
*** tstclair has joined #savanna20:40
*** dina_belova has joined #savanna20:44
*** dina_belova has quit IRC20:48
*** dina_belova has joined #savanna20:54
*** key4 has quit IRC20:57
*** key has joined #savanna20:57
*** dina_belova has quit IRC20:58
*** crobertsrh is now known as _crobertsrh21:08
*** tmckayrh has quit IRC21:26
*** dmitryme has quit IRC21:28
*** NikitaKonovalov has joined #savanna21:35
*** NikitaKonovalov has quit IRC21:39
*** qwerty_nor has joined #savanna21:45
*** rnirmal has quit IRC21:46
*** dina_belova has joined #savanna21:54
*** dina_belova has quit IRC21:59
*** mattf is now known as _mattf22:18
*** lastidiot has quit IRC22:23
*** NikitaKonovalov has joined #savanna22:36
*** IlyaE has quit IRC22:38
*** NikitaKonovalov has quit IRC22:40
*** IlyaE has joined #savanna22:43
*** IlyaE has quit IRC22:46
*** IlyaE has joined #savanna22:54
*** dina_belova has joined #savanna22:55
*** dina_belova has quit IRC22:59
*** esmute has quit IRC23:03
*** esmute has joined #savanna23:03
*** jcooley has quit IRC23:03
*** jcooley has joined #savanna23:06
*** lastidiot has joined #savanna23:21
*** NikitaKonovalov has joined #savanna23:35
*** NikitaKonovalov has quit IRC23:40
*** qwerty_nor has quit IRC23:52
*** dina_belova has joined #savanna23:55
« Sunday, 2013-08-04 Index Tuesday, 2013-08-06 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!