| *** IlyaE has joined #savanna | 00:03 | |
| *** SlickNik has quit IRC | 00:13 | |
| *** SlickNik has joined #savanna | 00:14 | |
| *** matsuhashi has joined #savanna | 00:25 | |
| *** nosnos has joined #savanna | 00:56 | |
| *** SlickNik has quit IRC | 01:08 | |
| *** SlickNik has joined #savanna | 01:09 | |
| *** nosnos has quit IRC | 01:10 | |
| *** nosnos has joined #savanna | 01:11 | |
| *** IlyaE has quit IRC | 01:48 | |
| *** SergeyLukjanov has joined #savanna | 01:53 | |
| *** SergeyLukjanov has quit IRC | 02:45 | |
| *** IlyaE has joined #savanna | 02:57 | |
| *** IlyaE has quit IRC | 03:11 | |
| *** IlyaE has joined #savanna | 04:26 | |
| *** matsuhashi has quit IRC | 05:20 | |
| *** matsuhashi has joined #savanna | 05:21 | |
| *** akuznetsov has joined #savanna | 05:50 | |
| *** SergeyLukjanov has joined #savanna | 06:00 | |
| *** nosnos has quit IRC | 06:00 | |
| *** nosnos_ has joined #savanna | 06:00 | |
| *** matsuhas_ has joined #savanna | 06:01 | |
| *** matsuhashi has quit IRC | 06:01 | |
| *** akuznetsov has quit IRC | 06:10 | |
| *** akuznetsov has joined #savanna | 06:12 | |
| *** akuznetsov has quit IRC | 06:15 | |
| *** IlyaE has quit IRC | 06:23 | |
| *** IlyaE has joined #savanna | 06:30 | |
| *** akuznetsov has joined #savanna | 06:34 | |
| *** matsuhas_ has quit IRC | 06:44 | |
| *** matsuhashi has joined #savanna | 06:45 | |
| *** nosnos has joined #savanna | 06:47 | |
| *** nosnos_ has quit IRC | 06:48 | |
| *** matsuhashi has quit IRC | 06:49 | |
| *** matsuhashi has joined #savanna | 07:02 | |
| *** IlyaE has quit IRC | 07:05 | |
| *** SlickNik has quit IRC | 07:24 | |
| *** SlickNik has joined #savanna | 07:24 | |
| *** akuznetsov has quit IRC | 07:34 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 07:37 | |
| *** _SergeyLukjanov has quit IRC | 07:38 | |
| *** SergeyLukjanov has joined #savanna | 07:44 | |
| *** SergeyLukjanov has quit IRC | 07:49 | |
| *** akuznetsov has joined #savanna | 08:19 | |
| *** matsuhashi has quit IRC | 08:27 | |
| *** matsuhashi has joined #savanna | 08:28 | |
| *** nosnos has quit IRC | 09:01 | |
| *** nosnos has joined #savanna | 09:01 | |
| *** nprivalova has joined #savanna | 09:25 | |
| *** aignatov has joined #savanna | 09:43 | |
| *** akuznetsov has quit IRC | 10:02 | |
| *** matsuhashi has quit IRC | 10:32 | |
| *** matsuhashi has joined #savanna | 10:32 | |
| *** matsuhashi has quit IRC | 10:37 | |
| *** akuznetsov has joined #savanna | 10:42 | |
| venza | do someone know if there is someone working on a Spark plugin for Savanna? | 11:31 |
|---|---|---|
| mattf | venza, no one that i'm aware of. we have a reference plugin, an hdp, there's an idh one ongoing and also a cdh one iirc | 11:33 |
| mattf | Dmitry Mescheryakov around? | 11:33 |
| venza | thanks! | 11:33 |
| mattf | venza, you thinking about doing one? | 11:34 |
| *** nprivalova has quit IRC | 11:34 | |
| venza | yes, we are planning to start work on it | 11:34 |
| mattf | curious, who's we, and how are you thinking about integrating? | 11:35 |
| venza | mattf: we are a small team from Eurecom, a research institution in southern France | 11:38 |
| venza | mattf: we have a small openstack cluster for research on Hadoop | 11:38 |
| mattf | venza, gotcha. if you open a blueprint to describe what you're going to do with spark & savanna it'll help us know what's coming and let others who might be working on spark & savanna find you to collaborate - https://blueprints.launchpad.net/specs/+new | 11:40 |
| venza | we plan to make a plugin for Spark taking the Vanilla one as an example | 11:40 |
| venza | mattf: ok, we will do that thanks for the suggestion | 11:41 |
| mattf | sounds like you might be onto a very interesting approach w/ savanna! | 11:41 |
| venza | mattf: I'm sure you'll hear about us again, with more questions... | 11:43 |
| mattf | venza, looking forward to it! | 11:47 |
| *** nprivalova has joined #savanna | 11:47 | |
| *** ruhe has joined #savanna | 11:48 | |
| aignatov | venza, hi, great news about Spark for Savanna! Here is the page for plugin spi http://docs.openstack.org/developer/savanna/devref/plugin.spi.html. And yes, you can use Vanilla plugin as reference implementation | 11:51 |
| venza | aignatov: thanks! | 11:55 |
| *** dmitryme has joined #savanna | 12:01 | |
| venza | I have another question, regarding data sources | 12:02 |
| venza | we would like to have a "data layer" cluster with hdfs and no compute | 12:02 |
| venza | that can be used by several compute-only clusters, vanilla or spark or others | 12:03 |
| venza | is there a "hdfs" data source planned? | 12:04 |
| aignatov | venza, yes, we have a blueprint https://blueprints.launchpad.net/savanna/+spec/epd-data-source-existing-hadoop-cluster | 12:07 |
| aignatov | there are few words about it, but it's data source as HDFS defenitely | 12:07 |
| aignatov | we are planning to include it in Savanna 0.3.1 | 12:08 |
| venza | aignatov: ok thanks, we don't want to duplicate efforts | 12:08 |
| aignatov | I'm I right that you want HDFS Data Source as feature of Savanna's EDP -Elastic Data Processing? | 12:09 |
| venza | aignatov: yes, but we need also to be able to launch a compute-only cluster, with no name nodes or data nodes and refer to an existing hdfs cluster | 12:11 |
| venza | from what I understand currently this is not possible | 12:12 |
| aignatov | venza, Savanna already is able do this | 12:12 |
| aignatov | you can deploy cluster only with HDFS | 12:12 |
| dmitryme | aignatov: but you can't deploy compute-only cluster which will be configured to use an existing HDFS | 12:13 |
| dmitryme | right now if you bring up compute-only cluster you will be able to work with Swift only | 12:13 |
| venza | yes, that is what we are looking for, compute-only plus hdfs | 12:14 |
| ruhe | dmitryme, isn't it possible to pass details about external HDFS through cluster and nogegroup tempaltes? | 12:14 |
| dmitryme | ruhe, hm, good question | 12:14 |
| venza | there are hard coded rules that will not let you laung a cluster without name node and data nodes | 12:15 |
| venza | *launch | 12:15 |
| dmitryme | aha, I see | 12:18 |
| dmitryme | yes, that seems to be too restrictive for your case | 12:18 |
| aignatov | venza, that's a good point to contribute, yes, Savanna has strict validation on it, you can file a bug :-) but I think solution would be if we are deploing only compute Hadoop services we need have an ability to set HDFS for that compute during deployment. | 12:18 |
| dmitryme | I guess you can launch a compute cluster having NameNode deployed on master | 12:19 |
| dmitryme | and no DataNode processes | 12:19 |
| dmitryme | or as aignatov suggested, slightly change Savanna validation rules | 12:19 |
| dmitryme | I believe the thing me and aignatov are unsure of right now is that: if launch a compute cluster, will you be able to configure your jobs to use external HDFS | 12:21 |
| dmitryme | ? | 12:21 |
| aignatov | another case - once Savanna has only HDFS deployed, some time later that cluster can be scaled with tasktrackers and jobtracker only | 12:21 |
| dmitryme | The blueprint aignatov pointed to aims to support such case for EDP | 12:21 |
| *** ruhe has quit IRC | 12:22 | |
| venza | in our use case we will not re-scale the hdfs cluster, but yes I see the problem | 12:23 |
| dmitryme | I am not sure if external HDFS will work if you run jobs directly on the cluster | 12:23 |
| venza | we will temporarily relax the rules and see what can be done | 12:24 |
| venza | thanks everyone | 12:24 |
| aignatov | venza, great, looking forward for your blueprints and solutions! :-) | 12:26 |
| *** ruhe has joined #savanna | 12:29 | |
| *** ruhe has quit IRC | 12:29 | |
| mattf | dmitryme, do you know how to get savanna to register its service name w/ keystone so SAVANNA_URL isn't needed and the SAVANNA_SERVICE lookup will work in savannadashboard/api/client.py ? | 12:48 |
| *** ruhe has joined #savanna | 12:48 | |
| dmitryme | mattf, savanna can't do it automatically and I think it should not | 12:48 |
| dmitryme | so far we say that user must manually add the endpoint | 12:49 |
| mattf | what's the goal of the service code? | 12:50 |
| dmitryme | you mean, why just not use SAVANNA_URL always? | 12:50 |
| mattf | i'd like to default SAVANNA_URL, but that'll prevent the SAVANNA_SERVICE code from ever executing. if there's no point to the SAVANNA_SERVICE code then defaulting makes sense. if there's a point then i need to find another solution. | 12:51 |
| mattf | that other solution may be to actually have savanna register w/ keystone. i'm curious why the code to go a service lookup exists, but there's no code to do a service register. | 12:52 |
| mattf | ...so what't the goal of the service code? no goal, mind if i remove it? | 12:53 |
| dmitryme | as for automatic registration: what if you have several Savanna services. For instance our engineers use their own Savanna against single lab | 12:53 |
| dmitryme | in that case you will have several endpoints registered for different Savanna instances | 12:54 |
| dmitryme | I believe that adding an endpoint must be a conscious action performed by an admin | 12:55 |
| mattf | sure could. the only way to have multiple "heads" on a single infrastructure is to have multiple horizons | 12:55 |
| dmitryme | as for removing that code: I am against it | 12:56 |
| dmitryme | Though I can't think of an example where it is really needed | 12:56 |
| mattf | there are a fwe ways to proceed imho. 0) savanna registers itself by default, 1) savanna registers itself by admin request, 2) savanna does not have the ability to register itself | 12:56 |
| dmitryme | I think we implemented it because we wanted to be OpenStack-like | 12:56 |
| mattf | i'm looking at this from the perspective of configuring horizon= | 12:57 |
| dmitryme | each core openstack project have an endpoint registered in keystone | 12:57 |
| mattf | right now every horizon deployment needs to have SAVANNA_URL added to its config | 12:57 |
| mattf | that url recently changed and probably will again in the future. it's a necessarily manual task right now. | 12:57 |
| mattf | i'll argue that the common deployment case for savanna is to have a single savanna instance on some openstack infrastructure | 12:58 |
| mattf | as such, i want to remove the manual configuration step | 12:58 |
| mattf | (btw, it's also a pita for upgrade) | 12:58 |
| dmitryme | if I were an admin, I would prefer to as little implicit magic as possible | 12:59 |
| dmitryme | * to have | 12:59 |
| mattf | to remove the manual config i can either default SAVANNA_URL (http://localhost:8386/v1.1) or enable SAVANNA_SERVICE | 12:59 |
| dmitryme | actually, I prefer explicit over implicit everywhere :-) | 13:00 |
| mattf | if i default SAVANNA_URL then the SAVANNA_SERVICE code will never be reached | 13:00 |
| mattf | explicit has resulted in 3 open questions since we did the 0.3 release | 13:00 |
| mattf | i've had a few other folks ask me questions directly that resulted in "edit local_settings and change v1.0 to v1.1" | 13:01 |
| dmitryme | I believe that is a normal thing for upgrade | 13:01 |
| mattf | the more i think about this the more it makes sense to have savanna register itself by default. it'll ease the common case. then admin who want to do "advanced" configurations, such as having multiple savanna instances talking to the same openstack infra, take the burden of extra config. | 13:02 |
| mattf | imho it's a common model to make the common case as simple as possible | 13:04 |
| dmitryme | from my point of view, we can have such a thing, but only if it is turned off by default | 13:04 |
| mattf | so long as SAVANNA_URL isn't defaulted, the advanced case admins can still get the deployment they want | 13:04 |
| mattf | well, i guess they could in either case | 13:05 |
| mattf | why is it a good thing to make someone deploying horizon w/ the savanna app edit the local_settings config? | 13:05 |
| dmitryme | it is not mandatory right now. I am looking in the code right now, and if I am not mistaken, it is enough to add 'mapreduce' endpoint to keystone | 13:06 |
| mattf | (fyi, i live in a world where the vast majority of users will not read release notes or follow their instructions very closely) | 13:06 |
| dmitryme | https://github.com/openstack/savanna-dashboard/blob/master/savannadashboard/api/client.py#L67 | 13:07 |
| mattf | it is, though i'd call it savanna instead | 13:07 |
| ruhe | i'd prefer to do the same other similar projects do | 13:08 |
| mattf | ruhe, hard to argue with that! | 13:08 |
| dmitryme | as for endpoint name, actually it is not exactly the name, it is 'type' | 13:09 |
| *** nosnos has quit IRC | 13:09 | |
| dmitryme | endpoint has two properties - name and type | 13:09 |
| dmitryme | type must be 'mapreduce' and 'name' - anything | 13:09 |
| *** nosnos has joined #savanna | 13:10 | |
| dmitryme | we called it 'map reduce' by default because that is the way other services are registered in keystone | 13:10 |
| dmitryme | do | 13:10 |
| dmitryme | keystone service-list | 13:10 |
| dmitryme | here nova is registered as endpoint with name 'nova' and type 'compute' | 13:11 |
| *** nosnos has quit IRC | 13:14 | |
| mattf | looks like name, type, desc = savanna, dataprocessing, OpenStack Savanna service ? | 13:14 |
| openstackgerrit | Yaroslav Lobankov proposed a change to openstack/savanna: Decreasing integration test for cluster configs https://review.openstack.org/52921 | 13:14 |
| dmitryme | agree that 'data-processing' as type is more appropriate right now | 13:15 |
| dmitryme | the 'mapreduce' was set several month ago, when there was no data processing | 13:17 |
| *** tmckay has joined #savanna | 13:23 | |
| openstackgerrit | John Speidel proposed a change to openstack/savanna: Add support for Hive related services https://review.openstack.org/52923 | 13:24 |
| openstackgerrit | John Speidel proposed a change to openstack/savanna: Add support for Hive related services https://review.openstack.org/52923 | 13:26 |
| mattf | dmitryme, ruhe, do you know if other services auto-register their service w/ keystone? | 13:26 |
| dmitryme | the installation tells that you need even register keystone endpoint manually: | 13:29 |
| dmitryme | http://docs.openstack.org/havana/install-guide/install/yum/content/keystone-services.html | 13:29 |
| dmitryme | and other services also require that | 13:30 |
| dmitryme | for instance, see Compute or Image Service in the same guide | 13:30 |
| *** dmitryme has quit IRC | 13:47 | |
| *** ruhe has quit IRC | 13:48 | |
| *** macjack has joined #savanna | 13:48 | |
| *** nprivalova is now known as nadya | 13:53 | |
| *** DennyZhang has joined #savanna | 13:56 | |
| *** IlyaE has joined #savanna | 14:01 | |
| *** ruhe has joined #savanna | 14:04 | |
| *** dmitryme has joined #savanna | 14:33 | |
| openstackgerrit | John Speidel proposed a change to openstack/savanna: Add support for Hive related services https://review.openstack.org/52923 | 14:36 |
| *** rnirmal has joined #savanna | 14:37 | |
| openstackgerrit | Yaroslav Lobankov proposed a change to openstack/savanna: Content for integration tests was added https://review.openstack.org/52937 | 14:52 |
| openstackgerrit | Yaroslav Lobankov proposed a change to openstack/savanna: Content for integration tests was added https://review.openstack.org/52937 | 14:58 |
| openstackgerrit | Yaroslav Lobankov proposed a change to openstack/savanna: Content for integration tests was added https://review.openstack.org/52937 | 15:01 |
| *** ruhe has quit IRC | 15:32 | |
| *** ruhe has joined #savanna | 15:33 | |
| *** ruhe has quit IRC | 15:53 | |
| *** ruhe has joined #savanna | 15:55 | |
| *** dmitryme has quit IRC | 16:01 | |
| *** dmitryme has joined #savanna | 16:02 | |
| *** ruhe has quit IRC | 16:04 | |
| *** sreshetnyak has joined #savanna | 16:04 | |
| *** vrovachev has joined #savanna | 16:05 | |
| *** akuznetsov has quit IRC | 16:08 | |
| *** rnirmal_ has joined #savanna | 16:15 | |
| *** nadya has quit IRC | 16:17 | |
| *** DennyZhang has quit IRC | 16:18 | |
| *** rnirmal_ has quit IRC | 16:20 | |
| *** rnirmal has quit IRC | 16:20 | |
| *** rnirmal has joined #savanna | 16:20 | |
| *** IlyaE has quit IRC | 16:27 | |
| *** dmitryme has quit IRC | 16:33 | |
| *** macjack has quit IRC | 16:36 | |
| *** macjack has joined #savanna | 16:37 | |
| *** dmitryme has joined #savanna | 16:37 | |
| *** dmitryme has quit IRC | 16:40 | |
| *** ruhe has joined #savanna | 16:41 | |
| *** rnirmal_ has joined #savanna | 16:47 | |
| *** rnirmal has quit IRC | 16:50 | |
| *** rnirmal_ is now known as rnirmal | 16:50 | |
| *** ruhe has quit IRC | 16:55 | |
| *** rnirmal has quit IRC | 17:06 | |
| *** rnirmal has joined #savanna | 17:06 | |
| *** ruhe has joined #savanna | 17:07 | |
| *** SergeyLukjanov has joined #savanna | 17:15 | |
| *** IlyaE has joined #savanna | 17:20 | |
| *** akuznetsov has joined #savanna | 17:27 | |
| *** sacharya has joined #savanna | 17:29 | |
| *** rnirmal has quit IRC | 17:30 | |
| *** rnirmal has joined #savanna | 17:31 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 17:35 | |
| *** _SergeyLukjanov has quit IRC | 17:35 | |
| *** sreshetnyak1 has joined #savanna | 17:45 | |
| *** ruhe has quit IRC | 17:45 | |
| *** SergeyLukjanov has joined #savanna | 17:52 | |
| *** SergeyLukjanov has quit IRC | 17:53 | |
| *** SergeyLukjanov has joined #savanna | 17:54 | |
| *** ruhe has joined #savanna | 17:57 | |
| *** ruhe has quit IRC | 18:00 | |
| *** rnirmal has quit IRC | 18:12 | |
| *** rnirmal has joined #savanna | 18:12 | |
| *** mattf is now known as _mattf | 18:18 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 18:27 | |
| *** _SergeyLukjanov has quit IRC | 18:28 | |
| *** SergeyLukjanov has joined #savanna | 18:32 | |
| *** _mattf is now known as mattf | 18:32 | |
| *** macjack has quit IRC | 18:32 | |
| *** macjack_ has joined #savanna | 18:32 | |
| *** nadya has joined #savanna | 18:34 | |
| *** ruhe has joined #savanna | 18:38 | |
| *** rnirmal has quit IRC | 18:41 | |
| *** rnirmal has joined #savanna | 18:41 | |
| *** ruhe has quit IRC | 18:43 | |
| *** rnirmal_ has joined #savanna | 19:02 | |
| *** rnirmal_ has quit IRC | 19:03 | |
| *** rnirmal_ has joined #savanna | 19:03 | |
| *** rnirmal has quit IRC | 19:03 | |
| *** rnirmal_ is now known as rnirmal | 19:03 | |
| openstackgerrit | Jonathan Maron proposed a change to openstack/savanna: Enable network operations over neutron private nets https://review.openstack.org/52997 | 19:12 |
| *** macjack_ has quit IRC | 19:29 | |
| openstackgerrit | Jonathan Maron proposed a change to openstack/savanna: Enable network operations over neutron private nets https://review.openstack.org/52997 | 19:33 |
| * mattf smashes head into desk trying to figure out what layer of the stack is caching some bad endpoint data | 19:34 | |
| *** nadya has quit IRC | 19:52 | |
| *** nadya has joined #savanna | 20:09 | |
| openstackgerrit | Jonathan Maron proposed a change to openstack/savanna: Enable network operations over neutron private nets https://review.openstack.org/52997 | 20:13 |
| *** nadya has quit IRC | 20:17 | |
| *** akuznetsov has quit IRC | 20:24 | |
| *** sreshetnyak1 has quit IRC | 20:30 | |
| *** mattf is now known as _mattf | 21:47 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 22:15 | |
| *** _SergeyLukjanov is now known as SergeyLukjanov | 22:15 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 22:28 | |
| *** _SergeyLukjanov is now known as SergeyLukjanov | 22:28 | |
| *** tmckay has quit IRC | 22:32 | |
| *** IlyaE has quit IRC | 22:35 | |
| *** IlyaE has joined #savanna | 22:39 | |
| *** SergeyLukjanov has quit IRC | 22:45 | |
| *** SergeyLukjanov has joined #savanna | 22:46 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 22:46 | |
| *** _SergeyLukjanov has quit IRC | 22:48 | |
| *** _mattf is now known as mattf | 22:53 | |
| *** rnirmal has quit IRC | 23:01 | |
| *** SergeyLukjanov has joined #savanna | 23:10 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 23:33 | |
| *** _SergeyLukjanov is now known as SergeyLukjanov | 23:33 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 23:34 | |
| *** _SergeyLukjanov is now known as SergeyLukjanov | 23:34 | |
| openstackgerrit | John Speidel proposed a change to openstack/savanna: Add support for Hive related services https://review.openstack.org/52923 | 23:43 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!