18:10:37 <SergeyLukjanov> #startmeeting savanna 18:10:37 <openstack> Meeting started Thu Jan 9 18:10:37 2014 UTC and is due to finish in 60 minutes. The chair is SergeyLukjanov. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:10:38 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:10:40 <openstack> The meeting name has been set to 'savanna' 18:10:41 <jmaron> yep 18:10:46 <SergeyLukjanov> openstack, thank you sir 18:10:52 <SergeyLukjanov> #topic Agenda 18:10:56 <SergeyLukjanov> #link https://wiki.openstack.org/wiki/Meetings/SavannaAgenda 18:11:32 <SergeyLukjanov> #topic Action items from the last meeting 18:11:43 <SergeyLukjanov> roadmap improvements are still in my backlog 18:11:47 <SergeyLukjanov> heh 18:11:59 <SergeyLukjanov> #action SergeyLukjanov to check that all blueprints created and ping guys to make them if not 18:12:04 <SergeyLukjanov> #action SergeyLukjanov add links to the blueprints to roadmap 18:12:12 <SergeyLukjanov> #topic News / updates 18:12:19 <SergeyLukjanov> guys, please 18:12:33 <SergeyLukjanov> I'm still in vacation, so, not so many news from my side 18:12:58 <aignatov> nothing specific from me, just reviewed some code today and continue working on adding anti affinity to scaling with heat 18:13:07 <crobertsrh> I'm currently working on putting the "Java" job type into the dashboard (based on tmckay's Oozie Java action work). 18:13:10 <aignatov> the first day after holidays :) 18:13:16 <jspeidel> any update regarding HBASE support? 18:13:29 <mattf> i've been chipping away at the savanna cli. comments still welcome on https://blueprints.launchpad.net/python-savannaclient/+spec/python-savannaclient-cli - especially around how to handle template creation. 18:13:33 <alazarev> I've finished IDH plugin initial version: https://review.openstack.org/#/c/56105/ 18:13:44 <SergeyLukjanov> jspeidel, not yet, I was on holidays/vacation, we'll continue at Monday 18:14:02 <jspeidel> @SergeyLukjanov ok, thanks 18:14:04 <SergeyLukjanov> jspeidel, and it's still -1'ed 18:14:10 <alazarev> jspeidel: as I see IDH waits for your review :) 18:14:19 <jspeidel> yep, I know :) 18:14:19 <aignatov> yeh, I see that this patch is WIP 18:14:23 * mattf jumps on jspeidel too 18:14:26 * mattf smiles 18:15:08 <SergeyLukjanov> any other updates? 18:15:09 <aignatov> alazarev: patch lgtm to be merged ;) 18:15:14 <jmaron> edp over neutron private nets is paused while I wait on our neutron setup to resurrect from the dead so I can perform functional testing 18:15:25 <alazarev> also, I was able to run Pig job via EDP on IDH cluster (https://review.openstack.org/#/c/64323/) 18:15:29 <mattf> jmaron, neutron issues? 18:15:37 <mattf> alazarev, awesome 18:15:39 <jmaron> IT/host issues 18:15:59 <SergeyLukjanov> also I'd like to mention Spark plugin efforts in ML 18:16:04 <SergeyLukjanov> let me find the link 18:16:10 <jspeidel> @mattf #link https://review.openstack.org/#/c/56105/ 18:16:14 <mattf> https://blueprints.launchpad.net/savanna/+spec/spark-plugin 18:16:31 <aignatov> #link https://blueprints.launchpad.net/savanna/+spec/spark-plugin 18:16:42 <SergeyLukjanov> #link http://lists.openstack.org/pipermail/openstack-dev/2014-January/thread.html#23806 18:16:53 <aignatov> corrected you mattf :P 18:17:14 <mattf> aignatov, not the first, won't be the last! 18:17:38 <SergeyLukjanov> #topic Roadmap update/cleanup 18:17:46 <SergeyLukjanov> action items are on me 18:17:51 <SergeyLukjanov> I don't think that we have any updates to roadmap so far 18:17:58 <aignatov> SergeyLukjanov: these is awesome news, one more plugin, i'm impressed 18:18:03 <mattf> should we scan the roadmap and give feedback? i've not looked at it since summit. 18:18:21 <SergeyLukjanov> mattf, I'm afraid that nothing changed 18:18:52 <SergeyLukjanov> mattf, but it should, I still hope to return back to it and ensure blueprints existence 18:19:09 <mattf> ack 18:19:19 <SergeyLukjanov> #topic General discussion 18:19:42 <SergeyLukjanov> #info the icehouse-2 release will be Jan 24 18:19:43 <mattf> anyone have suggestions on how to create node group / cluster templates from the cli? 18:19:50 <SergeyLukjanov> so, be prepared for the code freeze Jan 22 18:20:15 <alazarev> mattf: request json? :) 18:20:17 <mattf> SergeyLukjanov, i'll do another oslo sync for next week, please add an action for me (remind me to do it next thurs) 18:20:29 <aignatov> mattf, not atm, the first thing is yes -json 18:20:37 <SergeyLukjanov> mattf, you can find some of my thoughts in the CLI blueprint 18:20:38 <jmaron> savanna template-create < template.json ? 18:20:43 <mattf> alazarev, i was thinking about that actually. maybe creating only by piping in a json file? 18:20:52 <jspeidel> alazarev +1 for json request bodhy 18:20:54 <jspeidel> body 18:20:56 <aignatov> jmaron: smthg like this 18:21:31 <mattf> afaict, it's complicated by the fact that template json is plugin specific 18:21:34 <aignatov> mattf: you can get an idea from heat ;) 18:21:38 <SergeyLukjanov> #action mattf to sync oslo right before the i2 code freeze 18:21:44 <mattf> for instance, process names aren't consistent between plugins 18:22:06 <mattf> and hdp supports more processes than vanilla 18:22:17 <SergeyLukjanov> I'm ok with piping json template for the first time at least 18:22:29 <alazarev> mattf: and they will not be consistent because different distros have different namings 18:22:31 <mattf> i've an action on myself to work out the best way to list processes to help in contructing json, but...that's not very userfiendly 18:22:45 <SergeyLukjanov> #action SergeyLukjanov to unsure that we'rein sync with global requirements 18:22:46 <jmaron> there's a precedent for plugin specific template processing in the plugin SPI 18:22:51 <SergeyLukjanov> #undo 18:22:52 <openstack> Removing item from minutes: <ircmeeting.items.Action object at 0x30f5350> 18:22:55 <mattf> alazarev, we could have some consistency tho, NameNode vs NAMENODE etc 18:23:02 <SergeyLukjanov> #action SergeyLukjanov to ensure that we're in sync with global requirements 18:23:28 <mattf> (someone needs to fix #undo to print the object string0 18:23:39 <mattf> not it 18:23:43 <SergeyLukjanov> mattf, alazarev, that's a problem of choosing vendor names vs. our common names 18:24:15 <mattf> seems plugins could maintain a simple map from common name to vendor name 18:24:30 <jspeidel> mattf, I don't think that we can expect consistency across providers. Each provider should expose names that are meaningful for their stack 18:24:42 <mattf> set of common names would have to be union of all vendor features tho 18:24:47 <aignatov> jspeidel: +1 18:25:12 <jmaron> vendor CLI plugins for template processing (akin to SPI)? 18:25:15 <alazarev> some services could be splited into severel processes, some services are specific for distro, I don't think we need to force namings 18:25:15 <mattf> jspeidel, so far the examples are somewhat silly tho, the primary difference is all caps for HDP 18:25:50 <alazarev> mattf: this is because we have only HDP and vanilla 18:25:52 <jmaron> vanilla: "oozie" HDP: OOZiE_SERVER, OOZIE_CLIENT 18:26:04 <mattf> so the proposal (well liked) is currently to simple accept json files. that means to actually create a template the user must essentially use horizon and then export it. 18:26:12 <jspeidel> mattf, agreed that the current differences are simple, but I think that assuming a consistency is a slippery slope 18:26:56 <mattf> jmaron, thanks for finding a nice example. in this case it seems like the vanilla is simply being less specific 18:27:01 <alazarev> if we have vanilla, HDP, IDH, cloudera, etc - there will be much more differences 18:27:02 <jmaron> (to point out less trivial differences) 18:27:41 <aignatov> don't forget about incoming Spark plugin, I think there will be another set of processes :) 18:27:45 <mattf> my view on this is we should make using savanna simple and consistent for users, no matter what plugin they are using on the backend 18:27:58 <mattf> ^^ my guiding rule, which eventually has exceptions 18:28:14 <mattf> aignatov, yeah, set would have to be union of all plugins 18:28:33 <jspeidel> mattf, agree with usability but don't think the we can dictate to providers what to call components 18:28:57 <jspeidel> mattf, each plugins users know the names of the components associated with the vendors stack 18:28:59 <mattf> jspeidel, the vendor would not have to call them something internally, only when representing them out to savanna 18:29:06 <jmaron> we could follow this already established precedent/convention: https://savanna.readthedocs.org/en/latest/devref/plugin.spi.html#convert-config-plugin-name-version-template-name-cluster-template-create 18:30:31 <mattf> imho, no need to make a decision here. seems to me the pressure is for no change to how the plugins work at the moment. i want to make sure folks are aware of the differences and that they can impact usability. 18:30:43 <mattf> (they're impacting usability in how i can make the cli already!) 18:30:49 <jspeidel> mattf, agreed 18:30:59 <alazarev> mattf: +1 18:31:30 * mattf gets off soapbox 18:32:07 <mattf> another related topic - destroy or delete - what's the proper verb for our cli? 18:32:18 <mattf> imho we should just be consistent w/ other clis 18:32:34 <jmaron> annihilate 18:32:40 <mattf> e.g. cluster-destroy or cluster-delete, node-group-template-destroy or node-group-template-delete 18:32:43 <mattf> obliterate? 18:32:52 <aignatov> jmaron: lol! 18:33:09 <aignatov> mattf, +1 to be consistent with other clis 18:33:30 <alazarev> I vote on delete 18:33:36 <mattf> aignatov, clis aren't consistent, how would you rank them (nova then keystone then cinder?) 18:33:49 <aignatov> cluster-delete and node-group-template-delete my vote 18:34:00 <jmaron> +1 for delete 18:34:07 <jspeidel> mattf, +1 for delete 18:34:12 * mattf starts to see himself s/destroy/delete/g'ing 18:34:36 <jspeidel> mattf, should have asked sooner ;) 18:34:45 <jmaron> s/destroy/euthanize/g 18:34:47 * mattf hangs head in shame 18:34:53 <mattf> jmaron, you're on fire 18:35:04 <jmaron> :) 18:35:05 <mattf> ok, i'll tend to delete and do another pass over other clis 18:35:13 <tmckay> just checked the REST apis, looks like it is all "delete" there too 18:35:15 <aignatov> mattf, yep, rely on nova first, imo 18:35:16 <tmckay> +1 delete 18:35:24 <mattf> tmckay, delete is the REST verb 18:35:51 <SergeyLukjanov> I'm ok with both destoy and delete ;) 18:36:02 <mattf> ok, thanks for the input folks! 18:36:05 <tmckay> well, true. I was looking at method names. Doubly consistent :) 18:36:32 <jmaron> SergeyLukjanov: node-template-destroy-and-delete? 18:36:41 <mattf> fyi - i'm finding all sorts of rough edges in the client, so i'll start writing them up for client v2 18:37:04 <mattf> (kinda glad i'm finding rough edges, one reason for doing the cli was to evaluate the usability of the api!) 18:37:28 <alazarev> destroy-delete-and-remove-completely 18:37:51 * mattf hopes someone brings up another topic before we get to exterminate 18:37:53 <aignatov> destroy-delete-and-remove-completely-and-run-away 18:38:11 * mattf waits for Aliens reference 18:38:12 <alazarev> during IDH plugin development I faced with problem that auth token is expired because of long inactivity. Auth token was only needed to take username from image tags. So, I proposed caching for username in node group.extra (https://review.openstack.org/#/c/65402/). 18:38:39 <alazarev> SergeyLukjanov: you left a comment that this need to be discussed 18:38:40 <jmaron> *get away from my template you b****!* 18:38:58 <mattf> jmaron, lol, i was thinking nuke from orbit 18:39:20 <jmaron> :) 18:39:23 <mattf> jspeidel, what on earth do you guys have in your break rooms!? 18:39:26 <aignatov> xD 18:39:29 <SergeyLukjanov> jmaron, heh, that's a good option :) 18:39:57 <SergeyLukjanov> alazarev, my concern is extra field ressurection 18:40:24 <alazarev> SergeyLukjanov: what wrong with it? extra is good 18:40:36 <mattf> alazarev, SergeyLukjanov, aignatov, does this become a non-issue w/ heat? 18:41:00 <SergeyLukjanov> alazarev, it's a kind of necrophilia 18:41:12 <SergeyLukjanov> mattf, extra field? 18:41:13 * mattf covers eyes 18:41:26 <mattf> SergeyLukjanov, need storing username on image 18:41:33 <mattf> need for* 18:41:34 <alazarev> mattf: yes, heat removes the issue, but it would be good to get plugin work before full switch to heat 18:41:37 <dmitryme> mattf, indeed it is not an issue with the heat 18:41:54 <mattf> alazarev, how long is "long inactivity"? 18:42:03 <aignatov> SergeyLukjanov: do you know that we already use extra for hadoop keypairs? 18:42:04 <mattf> and who's inactivity? 18:42:07 <mattf> whose* 18:42:08 <dmitryme> for those who don't know: all instances provisioned by Heat have ec2-user 18:42:21 <dmitryme> i.e. static username for all images 18:42:24 <alazarev> mattf: 30+ mins, savanna waits for manager to install something 18:42:32 <mattf> dmitryme, we could change that via userdata, right? 18:42:40 <mattf> alazarev, that's not very long imho 18:42:51 <dmitryme> mattf: change what? :-) 18:42:52 <mattf> not very long -> up priority 18:43:07 <mattf> dmitryme, os-user vs ec2-user. it's just ec2-user beacuse that's the cloud-init default? 18:43:08 <alazarev> mattf: I didn't dig why token expires, but it did 18:43:11 <SergeyLukjanov> aignatov, yup, I remember 18:43:35 <dmitryme> mattf: it is Heat default 18:43:36 <mattf> SergeyLukjanov, alazarev, what about letting in alazarev's change and filing a but to remove it once we go to heat? 18:43:44 <mattf> dmitryme, ahh 18:43:56 <mattf> eesh bug* 18:44:07 <dmitryme> Heat pushes onto instance not only userdata provided by the user, but also some additional scripts 18:44:19 <dmitryme> one of them sets up ec2-user 18:44:29 <mattf> dmitryme, i didn't know that, thanks 18:44:33 <aignatov> dmitry, thx, with heat we will remove username field from registering image 18:44:40 <alazarev> and, once again… what wrong with extra? engines could use it whatever they need 18:44:57 <dmitryme> mattf: no problem, it was a surprise for me actually 18:45:03 <mattf> alazarev, that's kinda what's wrong with it 18:45:34 <mattf> dmitryme, i'm sure you just saved me a ton of pain later on when i'd be trying to figure out why things are happening when i didn't explicitly ask for them to happenm 18:45:54 <aignatov> alazarev: I have nothing against extra 18:45:56 <mattf> alazarev, i'm flexible, ok w/ adding so long as we record the fact that we should later remove it 18:46:15 <dmitryme> alazarev: I would prefer one more specific field in node group, like 'username', not just 'extra' 18:46:43 <SergeyLukjanov> alazarev, my concern is that we're resurrecting extra for some objects in different patches instead of thinking about the common way to store additional info for objects 18:46:52 <SergeyLukjanov> custom/optional info 18:46:54 <alazarev> dmitryme: heat doesn't need such field, what's why I used extra 18:46:58 <SergeyLukjanov> plugin-specific maybe too 18:47:18 <SergeyLukjanov> heat needs it too 18:47:23 <SergeyLukjanov> but it'll be ec2-user 18:47:31 <alazarev> SergeyLukjanov: extra is a common way, no? 18:47:37 <SergeyLukjanov> I think that could make it configurable for example 18:47:43 <SergeyLukjanov> heat guys * 18:48:11 <SergeyLukjanov> alazarev, not really, we're adding it for each object when we're need it 18:48:50 <dmitryme> alazarev: but the plugin does not know which engine is used, Heat or Direct 18:49:07 <tmckay> hmmm, we have an "extra" in job_binary for EDP. Maybe I should try to stamp that out too 18:49:11 <dmitryme> so it does not know how to get the username in a uniform way 18:49:16 <alazarev> SergeyLukjanov: exactly! and I don't see problems here, let's extra be 18:49:39 <aignatov> tmckay: it is specific EDP extra, don't worry about it :) 18:49:51 <tmckay> is "extra" extra good or extra bad? That is the question 18:50:18 <mattf> "extra" is a red flag for me, means we missed something in the design 18:50:18 <aignatov> in EDP it is good 18:50:34 <alazarev> dmitryme: yes, that's why we have corresponding method in engine. Engine could use extra or don't use 18:50:39 <tmckay> :) a Python programmer's best friend 18:50:53 <SergeyLukjanov> tmckay, yup 18:51:14 <dmitryme> alazarev: we can simply unconditionally set the username field in node group before passing the cluster to the plug 18:51:24 <dmitryme> IMHO that is much simplier 18:51:49 <jmaron> dmitryme: I think I like that 18:51:50 <mattf> dmitryme, nice thought 18:52:02 <alazarev> mattf: if we keep something important in extra - yes, if it is used for caching or passing params - it's Ok for me 18:52:18 <SergeyLukjanov> dmitryme, +1 18:52:52 <alazarev> ok, will change to username field, extra was used because of heat only 18:53:26 <SergeyLukjanov> alazarev, yup, username field looks good 18:53:51 <aignatov> ok, agreed with username field 18:53:52 <SergeyLukjanov> probably we can add something to the orchestration engines to be able to override get_username behaviour 18:54:13 <aignatov> but what should we do with current extra field options in cluster obis? 18:54:15 <aignatov> https://github.com/openstack/savanna/blob/master/savanna/plugins/vanilla/config_helper.py#L222-L232 18:54:26 <mattf> alazarev, thanks for bring this up during the meeting. it helped me understand what you were after w/ that cr 18:55:10 <tmckay> so, while we're here... 18:55:28 <tmckay> I think I have just enough time to do more workflow extensions before Jan 22 18:55:43 <tmckay> what would people like to see? I know there is a roadmap note somewhere 18:55:53 <mattf> spark? 18:56:02 <tmckay> Maybe raw oozie workflows, or streaming 18:56:29 <tmckay> mattf, I am unaware of oozie/spark integratoin 18:56:47 <mattf> me too, more a ref to the inbound spark plugin 18:56:49 <alazarev> aignatov: I vote to move hadoop_private_ssh_key and hadoop_public_ssh_key into fields and remove extra… to be consistent 18:57:10 <tmckay> also, there is a shell workflow that might be nice 18:57:21 <aignatov> tmckay: maybe integration tests support of Java added action, updated rest api docs about new action? 18:57:22 <mattf> alazarev, btw, what do we need the public part of that key for? 18:57:42 <SergeyLukjanov> alazarev, it's only used by the vanilla plugin now I think, so, it's not good to make it a field 18:57:48 <tmckay> aignatov, definitely, I was thinking maybe I have enough time after that for another... 18:58:11 <aignatov> tmckay: what will be after Jan 22? 18:58:26 <tmckay> Oh, that's just the freeze 18:58:34 <SergeyLukjanov> tmckay, code freeze for i2 18:58:37 <dmitryme> alazarev: do we really need to store these keys for the Vanilla plugin? 18:58:50 <dmitryme> I can't imagine why? 18:59:35 <aignatov> dmitryme: we need, it simplify hadoop debugging after install 18:59:45 <alazarev> SergeyLukjanov: this means that plugins could store data in cluster.extra but can't in node group.extra, that looks strange for me 19:00:07 <alazarev> dmitryme: to put them on machine during scaling I believe 19:00:16 <mattf> alazarev, you're not wrong 19:00:42 <SergeyLukjanov> guys, I'd like to ask you to go through the https://launchpad.net/savanna/+milestone/icehouse-2 and decide what you'd like to do in i2 19:00:46 <SergeyLukjanov> we're out of time 19:00:48 <SergeyLukjanov> thanks all 19:00:49 <aignatov> tmckay: another thing I thought in EDP would be nice is to implement multiple workflows 19:00:55 <SergeyLukjanov> #endmeeting