18:04:13 #startmeeting Savanna 18:04:14 Meeting started Thu May 23 18:04:13 2013 UTC. The chair is aignatov2. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:04:15 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:04:18 The meeting name has been set to 'savanna' 18:05:38 We are working on extension and templates blueprints last week 18:05:40 This week we updated bluprints related for Pluggable Provisioning Mechanism 18:06:05 Now they have more details 18:06:28 Nadya can you post links to this documents? 18:07:40 yes, sure https://wiki.openstack.org/wiki/Savanna/PluggableProvisioning 18:08:01 #link https://wiki.openstack.org/wiki/Savanna/PluggableProvisioning 18:10:06 and the second updated wiki page is about Templates #link https://wiki.openstack.org/wiki/Savanna/Templates 18:10:33 #link https://wiki.openstack.org/wiki/Savanna/Templates 18:10:48 I just posted some thoughts about templates to the savanna-all list 18:12:55 I'd welcome comments/thoughts 18:12:58 we are looking through it 18:13:02 thanks! 18:17:36 besides your mail Jon, I'd like to notice another items. 18:18:13 such parameters can be place at cluster config, not at a node group config 18:18:14 We are mostly finished with instructions for diskimage-builder to construct images with Apache Hadoop inside 18:18:33 It will support both Ubuntu and CentOS distros 18:18:46 possibly we should add for cluster config a nested layout 18:18:54 for example HDFS 18:19:21 it should contain parameters applicable for all nodes in clusters 18:19:33 e.g. replication factor 18:20:48 In Jon's example on checkpoint.period, are you saying that this should be in the cluster config? 18:21:11 I'm not sure what you mean. service configuration properties can be defined at both the cluster level as well as the node group level 18:21:54 so any 'GENERAL" or service specific property can be specified at both cluster and node level 18:22:17 for examples mapred child heap size is only applicable to the node group 18:22:31 aignatov2, when you have something even mostly working re diskimage-builder, fire it my way. i'll take it for a test drive through RDO. fyi, i still think you should check the recipe into the savanna.git repo. 18:22:40 it can be specified globally at the cluster, but overridden at the node level if desired 18:22:58 if specified at the cluster, it's essentailly a global default 18:24:10 mattf, we just have some troubles with DIB and not well tested it 18:24:29 sometimes clusters are not working with it :) 18:25:03 mostly it is the same DIB elements which I sent you last week) 18:25:25 mattf, we're going to create a new repo in stackforge for all the tools we build for Savanna 18:25:40 if they aren't working in that the instance doesn't get a vm priv ip, i found a fix for that. i needed it to get ubuntu instances running on rdo. 18:26:18 ruhe, what's the hesitation for including the image creation in the savanna repo? 18:27:24 in our view the user would have a cluster configuration panel and a node group configuration panel. In both instances the same properties can be specified. they'll end up in a cluster or node group template. In both cases, they'd be associated to a "GENERAL" groping or a service grouping 18:27:41 mattf, we would like keep current repo only for savanna core. savanna-python-client, savanna-horizon-plugin will be separate projects. that's how all the OS projects manage additional components 18:27:44 mattf, I think these scripts are the separate project like savanna-pythonclient, savanna-horizon etc 18:28:25 ruhe +1 18:29:26 currently there's a tight coupling between what the image and savanna-api. until there's a cleaner separation, it seems wise to keep the two together. the hope is they'll more easily be kept in sync. 18:29:56 but node group can override parameters like replication factor 18:30:03 in you case 18:30:20 take for instance the default root password. if it's changed in the image builder it must also be changed in savanna-api. best to do it as a unit in one repo. 18:30:31 in the future we may have more scripts and instructions, elements and so forth, and savanna repo is an independent project with it's integration tests, for instance 18:30:53 Jon, in your model user can redefine a property like HDFS replication factor in Node Group 18:30:55 mattf, we're definitely going to decouple these things in ongoing implementation of plugins 18:31:05 mattf, root password is a temporary solution, we plan to reset it after deployment for security reasons 18:31:07 akuznetsov: yes. do you object to that capability? 18:31:18 in the duture releases 18:31:21 this looks like a way to shoot in ones foot 18:31:23 *future 18:31:37 Ot 18:31:38 yes because the replication factor is cluster parameter 18:32:16 ok. I suppose we could add the scope parameter for such properties 18:32:35 root pw is just a specific example. w/o defined expectations between savanna-api <-> instance, we risk the two getting out of sync if they live apart. 18:33:32 mattf, there is already set of integration tests we run on each commit. we'll know that something is wrong instantly :) 18:33:45 so i propose keeping them together, until the time there's a well defined interface, or they're entirely stable. 18:33:57 akuznetsov - do you see the issue that was brought up by Jon on the property collision that is possible in the current proposal? 18:34:17 ruhe, do those integration tests build a new image for testing w/ or use a stock image? 18:34:30 Erik, sure we see it 18:34:41 that was just one example. there are others as well. we believe the correct grouping is at a service level 18:34:44 yes we see it 18:34:57 actually we were thinking to avoid it in the following way: 18:35:40 when plugin specifies config applicable_target="service:hdfs", we demonstrate it in the UI once in Node Group 18:35:47 to be clear: properties are defined as associated to services or general, scoped to node or cluster 18:36:50 it would also have to be presented only once at the cluster level, if applicable 18:36:56 i.e. the do not going to provide user a way to specify it twice in a single Node Group 18:37:12 Jon, yep, for cluster as well 18:38:23 mattf, currently we don't build images for each commit. but i think that in a couple of weeks we'll have image-builder decoupled from any hard-code 18:38:26 that would fix that issue, but I'm wondering about the templates: would properties still be grouped by component? 18:40:22 component = process? 18:40:26 Jon, component is "task tracker", "datanode", right? 18:40:26 yes 18:40:34 aha, I see 18:40:59 ruhe, how would you do an integration test that involves image creation if the integration test is for savanna-api (lives w/ savanna-api repo) and the image creation lives in a separate repo? 18:43:57 Jon, we want some properties to be grouped by process (like processes heap sizes) and some - by service (like fs.checkpoint.dir) 18:45:44 the problem is that we actually don't even have that information in most instances. Hadoop configuration is essentially service based, so there is actually no documentation about the component/process association. we've spoken to internal developers, and the indication is that that would be a fairly significant documentation effort that isn't planned currently 18:46:31 so there are two fundamental problems: 1. grouping by component isn't generally done by hadoop and 2. the information isn't even available 18:47:58 but as UI developers, given the information associated with the config object, you can make some design decisions (pagination, filtering, etc) to make a usable interface. Trying to make the hadoop model align with the UI requirements is not the way to do that, IMHO 18:50:01 mattf, that's a good question. i think we should have a pool of images. i understand your point, but we really want to keep savanna similar to other OS projects. for instance, every project has python client which is kept separate from the main code. and at the same time python client can be used internally to handle requests from Horizon UI 18:51:35 ruhe, python client makes sense. it forces you to have a stable API for each project/repo. i think it's premature to say the image & savanna-api is stable enough to live apart. it's also not entirely clear to me that there's major motivation to stabilize that interface right now. 18:53:23 consider keeping them together for now. in any event, i'm keen to try it out asap. i've the email instructions to work from atm, would love a repo to work with too -- i'm guessing there will be rdo specific changes i'll have to make. 18:54:16 mattf, sure. we'll try to get that repo asap 18:56:24 Jon, an example. we have 2 mashines: big and small. We run a service mapred. So it appears 2 process tt on 1st and on 2nd. It have different amount of tasks to which may be processed 18:57:57 so we need to configure that on 1st machine we may run 10 tasks and on second - 5. But it is the one service 18:58:40 *map tasks 18:58:55 I'm not sure you can even do that based on the selection of one flavor per node group. How do you propose to have different machine sizes as part of the same node group? 18:59:34 so in this case, you'd define a big machine group of one, a small machine group of one, and configure the service parameters for each accordingly 18:59:56 service may be ran on 2 machines which belong to different node groups 19:01:00 hey folks 19:01:03 guys, we are out of time 19:01:08 in that case you can specify cluster level parameters for the service as default values, and then node group overrides per node 19:01:19 node group 19:01:21 #endmeeting savanna