18:06:09 <Nadya> #startmeeting savanna
18:06:10 <openstack> Meeting started Thu Jun  6 18:06:09 2013 UTC.  The chair is Nadya. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:06:11 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:06:13 <openstack> The meeting name has been set to 'savanna'
18:06:29 <Nadya> here is our agenda for today:
18:07:01 <Nadya> 1. Action items from previous meeting
18:07:10 <Nadya> 2. Updates
18:07:18 <Nadya> 3. convert() method
18:07:29 <Nadya> 4. IP discovery
18:08:07 <Nadya> #topic Action items
18:08:31 <Nadya> So, we had the following:
18:08:32 * Nadya ruhe add more details to savanna-dashboard blueprint (nprivalova, 18:52:38)
18:08:32 * Nadya SergeyLukjanov to update pluggable provisioning mechanism CR with latest ctx vision (SergeyLukjanov, 18:53:00)
18:08:32 * Nadya EricB create bp for Ambari (nprivalova, 18:56:43)
18:08:45 <Nadya> Sergey L, please update
18:08:50 <SergeyLukjanov> yep, thank you
18:09:19 <SergeyLukjanov> ok, we are working on the core part and I think that the beta version will be available tomorrow
18:09:38 <SergeyLukjanov> oh, actions update)
18:09:40 <SergeyLukjanov> sorry
18:10:02 <SergeyLukjanov> we was added details to BPs (links to mockups)
18:10:11 <Nadya> it is usefull info, anyway :)
18:10:31 <SergeyLukjanov> and I think that info about beta version of core part is actual for this question
18:10:43 <jmaron> ctx updates?
18:11:02 <SergeyLukjanov> aignatov send the letter with some details about using helpers
18:11:25 <SergeyLukjanov> ctx is partially implemented currently
18:11:28 <jmaron> ah..ok
18:11:36 <aignatov2> I'v just replied on your question, jon
18:11:46 <SergeyLukjanov> if you have some specific question, you're welcome :)
18:12:40 <SergeyLukjanov> additionally, I think that we can make a call between me and plugin writer to discuss some things
18:12:48 <Nadya> We saw that Eric publiched blueprint so let`s move on
18:12:52 <SergeyLukjanov> I mean integration we the core part
18:13:02 <jmaron> ok
18:13:43 <Nadya> #topic Updates
18:14:18 <Nadya> Alex I, please update
18:14:48 <aignatov2> ok, I'm working on vanilla plugin implementation
18:14:59 <aignatov2> it's draft version on gerrit
18:15:05 <aignatov2> you can observe it
18:15:23 <aignatov2> #link https://review.openstack.org/#/c/31532/
18:15:24 <SergeyLukjanov> it could be used as an example of using tools
18:16:04 <aignatov2> jon, regarding your question, in my implemntation i'm operating only with cluster object
18:16:37 <SergeyLukjanov> generally, there is no need to directly operate with context write know
18:16:39 <jmaron> remind me which question that was?  ;)
18:16:55 <jmaron> oh…ok.  now I understand.
18:17:02 <aignatov2> Plugin Context object thread in savanna-all
18:17:31 <Nadya> From my side: I'm preparing HOWTO document about Hadoop-Swift integration. Will start to work on Savanna integration tomorrow
18:18:17 <aignatov2> another my update: also we have updated CR about fedora elements
18:18:38 <aignatov2> there are some issues with installing rpm packages
18:18:41 <SergeyLukjanov> mattf, thank you for contributing there
18:19:00 <ruhe> mattf, did you have a chance to build image with Fedora?
18:19:03 <aignatov2> packages is not installing properly
18:19:04 <mattf> my pleasure
18:19:17 <mattf> i didn't see the update, i must have fumbled on following the repo
18:19:48 <aignatov2> just working on it to resolve issues
18:19:51 <ruhe> it's not there yet, we faced some issues, which could be caused by image-builder itself
18:20:24 <aignatov2> link to CR
18:20:30 <aignatov2> #link https://review.openstack.org/#/c/31218/
18:20:45 <SergeyLukjanov> are there any other updates?
18:20:45 <SergeyLukjanov> jmaron, maybe from your side?
18:20:47 <mattf> i didn't complete building a fedora image after i learned you were already working on it
18:21:14 <Nadya> does anyone else have updates?
18:21:35 <mattf> i'll take a look at 31218
18:21:35 <jmaron> we are progressing well with the ambari plugin.  we're at the point where we need to make sure the integration with savanna api is complete and working, hence my questions about setting up a virtual env
18:22:11 <jmaron> and we'll need to migrate some of our functionality to the ctx helpers, though I imagine we could keep some of that as is for the first iteration
18:22:22 <jmaron> (direct use of ssh etc)
18:22:45 <SergeyLukjanov> jmaron, I think we can have a talk with you about it, let's discuss it after the meeting
18:22:49 <aignatov2> mattf, please feel free to comment it, Ivan sent several questions to diskimage-builder guys, but there is no answeres yet (
18:23:16 <jmaron> ok
18:23:39 <mattf> aignatov2, have a pointer to the dib questions?
18:24:01 <aignatov2> one sec
18:24:20 <SergeyLukjanov> here it is
18:24:23 <SergeyLukjanov> #link http://lists.openstack.org/pipermail/openstack-dev/2013-June/009938.html
18:25:00 <Nadya> #topic convert() method
18:25:26 <Nadya> Sergey L, please
18:25:30 <SergeyLukjanov> mm, I want to take a small story about how we see this functionality
18:25:54 <SergeyLukjanov> in todays discussions we are extend templates to include all information from cluster
18:26:10 <aignatov2> mattf, It will be great to have rhel and/or centos instructions for creating cloud images to work with Savanna
18:26:25 <SergeyLukjanov> so, we want to make user able to create cluster from cluster template specifying only cluster name and user key
18:26:49 <aignatov2> do you have any thoughts? As I know DIB doesn't support these distros(
18:26:56 <SergeyLukjanov> due to this update it'll be cool if plugins would convert plugin-specific config files to cluster template
18:27:10 <mattf> aignatov2, #link https://github.com/mattf/savanna/blob/RDO-quickstart/doc/source/rdo_quickstart.rst <- first draft of single node setup, rdo has changed a bit recently that's slowed production of the multi-node instructions
18:27:12 <SergeyLukjanov> it provide an absolutely wide flexibility of using them
18:27:48 <SergeyLukjanov> btw cluster template could contain node groups description w/o node groups templates
18:28:01 <mattf> aignatov2, i have dib working on fedora 17 to produce ubuntu savanna images and vanilla fedora images. i had to submit a few patches to dib to get it working.
18:28:04 <Nadya> by the way, we are discussing convert() method for the flow when user wants to create cluster from some configuration file which cannot be parsed by Savana. So it is needed to convert it into Template or cluster object
18:28:05 <SergeyLukjanov> I'll describe it with additional details tomorrow
18:28:50 <SergeyLukjanov> jamaron, do you have any thoughts on adjusting convert to converting config file to cluster template?
18:28:59 <SergeyLukjanov> sorry, jmaron
18:29:14 <jmaron> just so I understand:  you are proposing that the convert method be changed from populating the cluster object to actually creating a savanna cluster template?
18:29:46 <SergeyLukjanov> yes, because we want to make cluster templates fully describe cluster
18:30:01 <aignatov2> mattf, that's really cool instruction. can we add these instruction to the main savanna repo in future?
18:30:24 <mattf> absolutely
18:30:28 <SergeyLukjanov> so, if this cluster template will provide some configs that could be overrided while cluster creation than it will be an additional flexibility
18:30:31 <aignatov2> great!
18:30:54 <jmaron> once the cluster object is populated, it would be a simple matter for the controller/UI to relate the information in the returned cluster object, the process it goes through to create a cluster object model anyhow
18:30:55 <SergeyLukjanov> and it could provide an ability for users to make some configs changes
18:32:01 <SergeyLukjanov> jmaron, do you have plans to extract some config params from config file to cluster object>
18:32:02 <SergeyLukjanov> ?
18:32:06 <jmaron> whether we are using standard or provider specific templates, remember that both are essentially tied to a specific provider
18:32:31 <SergeyLukjanov> yep, cluster templates are provider-specific too
18:32:52 <SergeyLukjanov> I mean that you can extract some parameter from the provier-specific config
18:32:56 <Nadya> jmaron, have you already started to implement convert()?
18:32:56 <jmaron> that's what convert() does, per the email exchange from a few days ago...
18:33:32 <jmaron> config properties in the provider template are populated into the cluster user_inputs
18:33:42 <SergeyLukjanov> jmaron, I'm very sorry, there are tons of discussions on this topic...
18:33:51 <ruhe> jmaron, convert() reruns cluster object which is essentially the same as cluster template
18:34:07 <jmaron> agreed
18:34:07 <SergeyLukjanov> rune, yes, that's I mean :)
18:34:59 <jmaron> we'd still prefer to return a populated cluster object.  persisting the contents of that cluster object into a template is probably an option the UI could support
18:35:39 <ruhe> so, if we make convert() to return cluster template, we don't loose anything. we just get all the features which templates have - user will have a lot of options to view/edit them
18:35:50 <Nadya> jmaron, have you already started to implement convert()?
18:35:57 <jmaron> we've completed it
18:36:16 <SergeyLukjanov> jmaron, I think it'll be clearer to use only one flow for cluster creation - from templates and supporting cluster template creation from provider-specific config file
18:36:27 <SergeyLukjanov> you still can use extra in it
18:36:33 <dmitryme> jmaron, it would be great if you can already share your code
18:36:57 <dmitryme> in that case we could find differences between your and our code earlier
18:37:12 <jmaron> perhaps early next week.  we still need to refine some and John, who wrote it, is out for the next couple of days
18:37:25 <SergeyLukjanov> provider-specific templates is in general equals to our cluster templates
18:39:06 <jmaron> I don't understand why we need to return a cluster template rather than a cluster.  a template is simply a representation of the cluster object.  call __repr__ if you like to get the template out of it.  why generate a file as part of the runtime functionality?
18:39:33 <SergeyLukjanov> I think that codebase will not be changed if convert to cluster templates instead of cluster
18:40:01 <ruhe> jmaron, you don't need to generate a file. you only need to generate an object representing cluster template
18:40:27 <jmaron> which is basically the cluster object!
18:41:00 <SergeyLukjanov> but cluster is an instance of cluster template
18:41:17 <jmaron> so?
18:41:18 <SergeyLukjanov> and user can create cluster template from pconfig file
18:41:32 <SergeyLukjanov> and then just use it to creating clusters
18:41:48 <jmaron> you want the conversion to occur prior to the creation flow?
18:41:49 <SergeyLukjanov> I think that we should discuss it when you share your code
18:42:11 <SergeyLukjanov> jmaron, yes
18:42:27 <Nadya> jmaron, I see your point. Let`s postpone this discussion, we need to look at code
18:42:33 <ruhe> let's move this conversation to the mailing list or to #savanna
18:42:45 <jmaron> I think we're just confusing the users more.  We should be aiming to streamline the flow into a simple "pick your template" whether it's a savanna or provider one, iterate thru the steps, and submit
18:42:45 <SergeyLukjanov> to not create some different flows of cluster creation that could be not clear for end-users
18:43:33 <SergeyLukjanov> I think that it's great that we share that vision with team
18:43:47 <SergeyLukjanov> and let's continue discussions later
18:44:04 <jmaron> I would say we've just added a separate flow (convert your template).  the UI can be structured in such a way that the steps are clear and unified
18:44:37 <jmaron> ok
18:44:47 <Nadya> jmaron, will we have 2 different cluster-creatin UI-flow?
18:45:31 <Nadya> jmaron, I understood, 1 flow :)
18:45:37 <SergeyLukjanov> in fact we just want to make some small changes in UI :)
18:45:40 <SergeyLukjanov> but let's go on
18:45:41 <SergeyLukjanov> :)
18:45:48 <Nadya> #topic IP discovery
18:46:10 <SergeyLukjanov> I propose the BP for the simple implementation of it
18:46:30 <SergeyLukjanov> here it is
18:46:31 <SergeyLukjanov> #link https://blueprints.launchpad.net/savanna/+spec/ips-discovery-mechanism
18:46:46 <SergeyLukjanov> are there any thoughts on it?
18:47:03 <jmaron> we took a look at that implementation.  at the moment that looks acceptable
18:47:19 <rnirmal> why define them again in savanna.. can't they be defined in nova and in savanna just refer to which network maps to management / private and public
18:47:25 <mattf> i've some, but they're incomplete
18:48:08 <rnirmal> and savanna can use the appropriate ips for the instances provided by nova
18:48:15 <jmaron> that's the problem - we've encountered deployments where that private/public distinction isn't clear
18:48:17 <SergeyLukjanov> rnirmal, it's not clear to determine which addresses should be used for different purposes
18:48:26 <mattf> the mananagement ip makes sense if you think about it as = "internal" or "public", not necessarily by itself. i don't think savanna generates enough traffic to warrant its own network
18:49:04 <rnirmal> SergeyLukjanov: well won't that be part of the provider or who ever is setting it up to sync that
18:49:09 <mattf> i'm not bought into the idea of "discovering" via savanna config - it appears it's duplicate config with the authoritative existing in nova-network / quantum
18:49:33 <mattf> i.e. if you change floating ips in nova-network you also have to inform savanna -> leads to complications for admins
18:49:57 <SergeyLukjanov> nova-network/quantum are both doesn't provide an ability to understand which ip is available for user
18:50:09 <mattf> fyi, my current floating range isn't easily expressed as x.y.z/p
18:50:27 <SergeyLukjanov> and which should be used for communication between Hadoop nodes
18:50:29 <mattf> quite possible we should be requesting enhancement in nn/q then
18:50:36 <rnirmal> not sure don't all the ips provided by nova-network or quantum accessible by the user
18:51:01 <jmaron> that's great and the ideal - but we're seeing deployments where the private, public, float ip distinction isn't made clear via nova and there is reluctance to change the networking
18:51:25 <mattf> i was hoping to look into the ability to tag interfaces, i.e. pub=10.0.0.1,priv=192.168.0.0 on instances, then config savanna to have internal_ip=priv, public_id=pub -- but i haven't had a chance yet
18:52:05 <mattf> rnirmal, that's probably true, it'd be scoped to the tenant, but savanna knows tenant -- that's another good reason to not use savanna config
18:52:18 <mattf> ...you'll end up needing [networks] for each potential tenant?
18:52:26 <SergeyLukjanov> if fact we can demand on networks names or specify networks in savanna
18:52:26 <rnirmal> mattf: agree
18:52:53 <mattf> i'm also not clear on why the internal_ip would be added to /etc/hosts
18:53:10 <rnirmal> that would be for name resolution for hadoop I suppose
18:53:20 <dmitryme> rnimal: exactly
18:53:24 <mattf> (fyi, this is all half formed, i was hoping to think it through more before responding on savanna-all)
18:53:26 <jmaron> but don't all those os based solutions assume that the given OS provider is willing to modify their networking to accommodate the solution?
18:54:02 <rnirmal> jmaron: but not sure how defining them within savanna is going to help either
18:54:16 <jmaron> they have to be defined somewhere
18:54:25 <jmaron> and if you can't control the OS stack
18:54:27 <mattf> jmaron, there's definitely an issue around best practices. if an OS deployment isn't naming interfaces in a consistent / discoverable way, it's hard to do discovery
18:54:31 <jmaron> savanna is your only choice
18:54:42 <rnirmal> nova provides tagged ips as such public=10.127.1.15; private=192.168.0.15
18:54:45 <jmaron> and that's exactly what we're finding
18:54:49 <rnirmal> or whatever be their names
18:54:53 <SergeyLukjanov> generally, I do not like this approach too, but I have no ideas to implement this functionality...
18:55:05 <jmaron> rnirmal:  not in a deployment I saw this week
18:55:06 <rnirmal> and in savanna just map them to what savanna understands as internal and public ips
18:55:25 <SergeyLukjanov> rnirmal, our OS installed in dev lab doesn't provide such tags
18:55:40 <jmaron> neither does the deployment at a customer site
18:55:54 <mattf> and the one in my site is different from jmaron's
18:55:58 <jmaron> and the customer will not change the deployment
18:56:05 <SergeyLukjanov> jmaron, +1
18:56:15 <aignatov2> guys, 5 minutes left
18:56:17 <rnirmal> hmm ok so how would you see implementing that within savanna
18:56:26 <mattf> there's still the idea of being able to discover by querying the tenant
18:56:33 <jmaron> with a sub-optimal approach ;)
18:56:50 <jmaron> mattf:  I'd need to understand that better
18:57:00 <mattf> i tend to de-dup config, because config is so easy for people to get wrong
18:57:07 <mattf> jmaron, me too
18:57:20 <mattf> it's definitely a complex topic and worthy of some focused time
18:57:28 <rnirmal> :) but on instance create does savanna pass in network info ?
18:57:53 <mattf> +1 SergeyLukjanov for generating a bp so we have a place to discuss, but i'm skeptical w/ the current proposal
18:57:53 <SergeyLukjanov> mattf, can you point us to API that provides info about networks destination?
18:58:16 <mattf> SergeyLukjanov, not atm, that's what i haven't gotten to yet in forming my understanding
18:58:44 <ruhe> if there is a way to to get that info about tenant from nova/quantum, it would be great
18:58:53 <rnirmal> another topic to add to this discussion would be private networks support for the hadoop internal ips
18:58:58 <jmaron> that would be the ideal
18:59:09 <mattf> if there isn't, it's an opportunity for the savanna community to contribute to the openstack networking community
18:59:16 <SergeyLukjanov> BP creation is ok
18:59:31 <SergeyLukjanov> mattf, we want to support not only the trunk version of openstack
19:00:07 <mattf> SergeyLukjanov, well, short term there's no feature, long term we should aim for closer integration w/ os networking
19:00:19 <mattf> in between, who knows
19:00:45 <jmaron> savanna filtering approach could be viewed as "last resort" but workable in some deployments, I suppose
19:00:53 <mattf> we should continue on #savanna and savanna-all
19:00:59 <Nadya> yes
19:01:12 <Nadya> we are out of time :(
19:01:23 <Nadya> Thank you all
19:01:28 <mattf> thanks all
19:01:31 <jmaron> thanks
19:01:34 <aignatov2> thanks
19:01:35 <ruhe> thanks
19:01:36 <SergeyLukjanov> thank you guys
19:01:43 <Nadya> #endmeeting