18:05:24 <nprivalova> #startmeeting Savanna
18:05:25 <openstack> Meeting started Thu May 30 18:05:24 2013 UTC.  The chair is nprivalova. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:05:26 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:05:29 <openstack> The meeting name has been set to 'savanna'
18:05:41 <SergeyLukjanov> o/
18:05:45 <nprivalova> here is our agenda for today
18:05:54 <nprivalova> 1. Update on each blueprint which is in progress.
18:06:02 <nprivalova> 2. Q&A
18:06:22 <nprivalova> that's it :)
18:06:52 <jmaron> sounds ok
18:07:06 <nprivalova> I'd like to describe in a few words the plugins we are working on. Please remember your questions and ask them after update, because we are limited in time
18:07:14 <aignatov> this week we published a lot of blueprints related to Savanna project
18:07:50 <nprivalova> #topic https://blueprints.launchpad.net/savanna/+spec/instance-interop-helper
18:08:11 <nprivalova> Alex I, please give an update
18:08:33 <aignatov> ok
18:09:13 <aignatov> intstance-interop-helper is a former name of VMManager
18:09:32 <aignatov> this feature is completely done I think
18:10:24 <aignatov> but not integrated with the trunk
18:10:34 <SergeyLukjanov> I would like to clarify that currently only basic functions has been implemented
18:10:35 <mattf> i don't see ref VMManager in the source. what was it?
18:11:05 <nprivalova> do you mean link? or sources?
18:11:11 <SergeyLukjanov> it'll be available when the core plugin mechanism will be merged into the trunk
18:11:24 <mattf> nprivalova, both would be ideal
18:11:42 <aignatov> yeah, only basic functions - setup_connection, execute_remote_command, interactive remote invoking and read/write remote file
18:11:49 <nprivalova> ok, let's continue
18:11:57 <nprivalova> #topic Horizon UI for Savanna https://blueprints.launchpad.net/savanna/+spec/savanna-dashboard
18:12:22 <nprivalova> Nikita, please update
18:12:27 <nkonovalov> Currently the savanna-dashboard plugin is at the begining of developement
18:13:07 <nkonovalov> We are bulding our tables and workflows for nodegroup templates, clusters templates and clusters
18:13:45 <nprivalova> #topic Cluster Security https://blueprints.launchpad.net/savanna/+spec/cluster-security
18:14:21 <nprivalova> Alex K, please :)
18:14:29 <mattf> the contents of the savanna-dashboard is "Horizon plugin for Savanna. \n It should provide an ability to fully control Savanna using UI." - can we get more detail on it?
18:15:20 <nprivalova> Matt, we will answer after, ok? I've remembered your question
18:15:35 <mattf> ok, after all updates, not after each -- oops
18:15:51 <akuznetsov> Now we re-create the vm provision mechanism corresponding to the new object model
18:17:01 <akuznetsov> it will also contain a improvement for cluster security, it will possible login to the clusters machine via user keypair
18:17:58 <akuznetsov> each cluster will have it own private key for passwordl-ess login
18:18:00 <SergeyLukjanov> Savanna will generate unique keypairs per clusters
18:18:17 <nprivalova> #topic Image Registry https://blueprints.launchpad.net/savanna/+spec/hadoop-image-registry
18:18:42 <nprivalova> Sergey L, please update
18:18:55 <SergeyLukjanov> ok, let's see… it consists of two parts
18:19:10 <SergeyLukjanov> python SPI to use in plugins and REST API to interop with it externally
18:19:19 <SergeyLukjanov> as for SPI - it's already done
18:19:44 <SergeyLukjanov> here it is - https://review.openstack.org/#/c/29936/14/savanna/utils/openstack/images.py
18:20:17 <SergeyLukjanov> I think we'll add such helpers (like Image Registry) to the plugins context
18:20:29 <mattf> queue for later: blueprint for image-registry is "Hadoop images should be registered in Savanna to provide an ability to specify additional info about Hadoop and etc." - more detail please
18:20:48 <SergeyLukjanov> I think that's all from my side for this blueprint
18:21:20 <SergeyLukjanov> it described better in pluggable provisioning mechanism blueprints at wiki.o.o
18:21:30 <SergeyLukjanov> I'll add link to it to the bp
18:21:40 <SergeyLukjanov> let's go to the next point
18:21:40 <nprivalova> #topic Image Builder. https://blueprints.launchpad.net/savanna/+spec/savanna-image-elements
18:22:36 <nprivalova> Ivan B is working on this, but he is not online. Alex I, please help us :)
18:23:43 <aignatov> We have created a new repository savanna-extra which is place of Savanna related things not included into the main repo: scpecial scripts, infra related. Imagebuilder elements is the first thing which will be checked in to savanna-extra.
18:23:43 <aignatov> Moreover, we have the first CR related to creating Ubuntu images with jdk and Apache Hadoop inside.
18:23:43 <aignatov> Links to repo and the first CR
18:23:44 <aignatov> #link https://github.com/stackforge/savanna-extra
18:23:44 <aignatov> #link https://review.openstack.org/#/c/31033/
18:23:44 <aignatov> mattf, I know you were interested in this feature, please review this patch and welcome for contributing!))
18:24:16 <SergeyLukjanov> we are planning to merge this CR tomorrow
18:24:17 * mattf deletes question about "where's the source"
18:24:27 <SergeyLukjanov> :)
18:24:31 <aignatov> xD
18:24:35 <mattf> aignatov, please link the bp and the source
18:25:08 <jmaron> bp?
18:25:13 <mattf> blueprint
18:25:18 <jmaron> ah :)
18:25:22 <aignatov> ok, will do
18:25:28 <nprivalova> #topic Swift Integration https://blueprints.launchpad.net/savanna/+spec/savanna-swift-integration
18:25:39 <mattf> fyi - https://github.com/stackforge/savanna-extra is empty
18:26:08 <ruhe> mattf it'll be there once code is reviewed and approved
18:26:17 <nprivalova> It's my favourite topic
18:26:29 <nprivalova> Now we are investigating Swift and Hadoop side patches and think how to test data-locality feature on our cluster.  But now we are able to copy data between hdfs and swift using 'hadoop distcp' and run hive-queries on data in swift.  On next week the patch for Savanna should be done.
18:26:56 <SergeyLukjanov> I think that we'll integrate it with Savanna asap
18:27:22 <nprivalova> #topic Vanilla Plugin https://blueprints.launchpad.net/savanna/+spec/vanilla-hadoop-plugin
18:27:34 <aignatov> the status of vanilla-hadop-pugin is in progress))
18:28:03 <nprivalova> #topic Ambari plugin
18:28:07 <aignatov> just trying to reimplement the same behavior as it was in Savanna 0.1
18:28:38 <nprivalova> Jon, John, Eric,  do you have any update on your unpublished blueprint :)?
18:29:00 <jmaron> yes - it will be published shortly ;)
18:29:00 <jspeidel> Erik is working on it
18:29:01 <jmaron> :)
18:29:19 <SergeyLukjanov> jmaron, it's great
18:29:24 <jmaron> Erik just responded to Sergey's email
18:30:17 <nprivalova> do you have any update regarding plugin itself?
18:30:48 <SergeyLukjanov> guys, I think it'll be great if you share your plugin code with us when you'll be ready, maybe w/o change requests
18:30:54 <jmaron> we are progressing with our implementation.  We are essentially modifying our existing code to align with the SPI
18:30:54 <mattf> queue for later - what's the difference between the vanilla hadoop plugin and the ambari plugin?
18:31:56 <nprivalova> mattf, ok, will discuss
18:32:33 <jmaron> matt - vanilla plugin is essentially restructuring existing savanna code that installs apache hadoop.  Ambari plugin will leverage Ambari to install HDP
18:32:37 <jspeidel> SergeyLukjanov,  we should be able to post our plugin code mid next week
18:32:50 <nprivalova> #topic Node placement controll https://blueprints.launchpad.net/savanna/+spec/node-placement-control
18:32:51 <SergeyLukjanov> jspeidel, that's great!
18:32:56 <aignatov> mattf, vanilla plugin supports vanilla Apache Hadoop and Ambari plugin supports for Hortonworks Apache Ambari Distribution
18:33:21 <nprivalova> Alex K, please update
18:33:43 <akuznetsov> Node group will contain a property anti affinity group that allows to schedule a vm with data node on a different compute nodes. This work is near to the end and code will be merge in the trunk at the next week.
18:33:55 <mattf> jmaron, aignatov, does that mean the ambari plugin won't be able to install unbranded apache hadoop?
18:34:28 <jspeidel> mattf, it could, but it wont
18:35:10 <nprivalova> #topic Plugabble Provisioning https://blueprints.launchpad.net/savanna/+spec/pluggable-cluster-provisioning
18:35:17 <nprivalova> so the main topic
18:35:18 <jspeidel> the hdp plugin uses ambari to configure/install/manage the hadoop cluster
18:35:33 <SergeyLukjanov> ok
18:35:49 <SergeyLukjanov> it consists of tons of complicated parts:)
18:35:51 <jspeidel> mattf, ambari could work with vanilla hadoop, but that is not in our plans now
18:36:19 <nprivalova> Guys, lets discuss after update please
18:36:31 <SergeyLukjanov> first of all, db model and data access layer are ready
18:36:40 <mattf> jspeidel, can we discuss over in #savanna?
18:36:52 <jspeidel> sure
18:36:57 <SergeyLukjanov> the overall helpers for building REST API are ready too
18:37:09 <SergeyLukjanov> and currently I'm working on implementing REST API functions
18:37:25 <SergeyLukjanov> I'm planning to merge this code by the end of this week
18:37:49 <SergeyLukjanov> #link https://review.openstack.org/#/c/29936/
18:38:19 <SergeyLukjanov> I think that that's generally all
18:38:22 <SergeyLukjanov> from my side
18:38:28 <nprivalova> #topic Q&A
18:39:02 <jmaron> interop helper and image registry - how does the plugin access them?  will they be accessible via the cluster reference?
18:39:03 <SergeyLukjanov> guys, I think that we should clarify difference between vanilla plugin and Ambari one
18:39:49 <SergeyLukjanov> jmaron, they we'll be accessible by the cluster ref or context that'll be available be calling ctx() anywehere)
18:40:43 <nprivalova> Are there any questions more about vanilla and ambari?
18:40:54 <jmaron> ah…you've added ctx as an argument to the spi methods?
18:41:20 <jmaron> I see it now
18:41:26 <mattf> re vanilla vs ambari -- what are the differences?
18:41:41 <SergeyLukjanov> oops, this part of code is outdated :)
18:42:01 <jspeidel> mattf, to answer your question about ambari managing non-branded hadoop
18:42:08 <mattf> nprivalova, re VMManager, it's being replaced, what was it?
18:42:47 <SergeyLukjanov> we just name it "Instance interop helper"
18:42:51 <jspeidel> mattf, technically it could, but that is not currently in the plans for the plugin.  we are currently making certain assumptions regarding hdp
18:43:12 <SergeyLukjanov> VMManager is not very good name
18:43:23 <jspeidel> mattf, so the short answer is that it is possible, but for now the plugin will assume hdp
18:43:28 <jmaron> Sergey:  so ctx isn't a parameter passed to SPI methods?
18:43:30 <SergeyLukjanov> because in feature we are planning to support bare metal provisioning:)
18:43:32 <mattf> nprivalova, re dashboard - the contents of the savanna-dashboard is "Horizon plugin for Savanna. \n It should provide an ability to fully control Savanna using UI." - can we get more detail on it?
18:43:39 <SergeyLukjanov> jmaron, yep
18:43:47 <nprivalova> mattf, it's just a helper for interacting with VMs. Are you finding where is it described?
18:44:00 <ruhe> mattf, we created these blueprints to allow community to monitor current progress. i agree that we need to add more details to blueprints. we'll do that gradually
18:44:01 <SergeyLukjanov> mattf, we are published mockups
18:44:04 <mattf> SergeyLukjanov, is it the code currently in cluster_ops and friends for managing ssh sessions and exec'ing commands?
18:44:19 <jmaron> is the assumption that we extends a base class and invoke a ctx() method to access the context?
18:44:21 <ruhe> Dmitry published mockups this week
18:44:22 <mattf> jspeidel, i see, ok
18:44:31 <SergeyLukjanov> mattf, yes
18:44:46 <ruhe> we only need to add links to mockups in the corresponding bp
18:45:03 <nprivalova> mattf, https://wiki.openstack.org/wiki/Savanna/PluggableProvisioning#VM_Manager
18:46:14 <ruhe> mattf, here is link to email with mockups https://lists.launchpad.net/savanna-all/msg00107.html
18:46:37 <jmaron> still wondering about ctx…
18:46:41 <SergeyLukjanov> Nadya, are there any other questions?
18:46:42 <aignatov> btw, instance-interop-helper bp in the savanna has a link to the spec which Nadya just posted
18:47:07 <nprivalova> question from mattf:queue for later: blueprint for image-registry is "Hadoop images should be registered in Savanna to provide an ability to specify additional info about Hadoop and etc." - more detail please
18:47:19 <SergeyLukjanov> jmaron, ctx will hold db session, helpers and other misc
18:47:43 <SergeyLukjanov> I hope that I'll update code tomorrow
18:47:50 <ruhe> mattf, about your question about image registry. we'll add link to corresponding wiki page to the bp
18:48:05 <jmaron> Sergey:  still wondering how I get a reference as a plugin developer?  can you post to list when code is updated so I can take a look?
18:48:13 <mattf> ruhe, ok
18:48:33 <ruhe> mattf, thanks for asking. that'll help to keep blueprints in a good shape :)
18:48:41 <SergeyLukjanov> jmaron, you can subscribe to the change request or to the gerrit project to receive notifications
18:48:51 <jmaron> k
18:49:03 <mattf> ruhe, the dashboard bp is also empty of details. i'm particularly interested in it.
18:49:22 <ruhe> sure, it'll be updated
18:49:22 <jspeidel> mattf, to clarify, HDP really is Apache Hadoop, and Ambari is an Apache project also.  So, you can obviously use Ambari to manage Apache Hadoop.  At this time, the savanna plugin is assuming that certain services are available that are in the HDP distribution.
18:50:16 <aignatov> btw, evryone can join this project and implement it's own plugin)
18:50:24 <mattf> jspeidel, specifically what services?
18:50:35 <jspeidel> mattf, examples are Ganglia and Nagios
18:50:52 <SergeyLukjanov> guys, are there any other unanswered questions?
18:51:10 <mattf> SergeyLukjanov, i'm pretty good. assuming bp's get more content soon!
18:51:34 <mattf> jspeidel, anything else come to mind?
18:51:54 <jspeidel> mattf, we are also assuming that some hdp puppet scripts exist
18:52:38 <nprivalova> #action ruhe add more details to savanna-dashboard blueprint
18:52:41 <mattf> those are puppet scripts provided by HDP instead of being provided by Ambari?
18:53:00 <SergeyLukjanov> #action SergeyLukjanov to update pluggable provisioning mechanism CR with latest ctx vision
18:53:48 <jspeidel> mattf, good point, looking at our code, I see only ambari puppet scripts referenced
18:54:39 <nprivalova> mattf, may I assign action about bp creation on you?
18:54:54 <mattf> nprivalova, for which bp?
18:55:24 <mattf> jspeidel, in my mind it seems like if you provide a repo with all the bits ambari expects the ambari plugin would work, if that repo is from hdp or from cloudera or from bigtop
18:55:25 <nprivalova> mattf, oops :) I meant Ambari plugin
18:55:35 <nprivalova> mattf, sorry
18:56:31 <mattf> nprivalova, you could, but i'm not sure what the action is
18:56:43 <nprivalova> #action EricB create bp for Ambari
18:56:51 <mattf> oh, ha!
18:57:12 <ErikB> nprivalova - got it
18:58:03 <nprivalova> So thanks guys! We are out of time. I hope I was not too boring :)
18:58:19 <mattf> ttyl
18:58:25 <jmaron> bye
18:58:28 <ruhe> bye
18:58:31 <SergeyLukjanov> bye)
18:58:33 <mattf> jspeidel, back to #savanna?
18:58:41 <nprivalova> #endmeeting