18:10:37 <SergeyLukjanov> #startmeeting savanna
18:10:37 <openstack> Meeting started Thu Jan  9 18:10:37 2014 UTC and is due to finish in 60 minutes.  The chair is SergeyLukjanov. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:10:38 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:10:40 <openstack> The meeting name has been set to 'savanna'
18:10:41 <jmaron> yep
18:10:46 <SergeyLukjanov> openstack, thank you sir
18:10:52 <SergeyLukjanov> #topic Agenda
18:10:56 <SergeyLukjanov> #link https://wiki.openstack.org/wiki/Meetings/SavannaAgenda
18:11:32 <SergeyLukjanov> #topic Action items from the last meeting
18:11:43 <SergeyLukjanov> roadmap improvements are still in my backlog
18:11:47 <SergeyLukjanov> heh
18:11:59 <SergeyLukjanov> #action SergeyLukjanov to check that all blueprints created and ping guys to make them if not
18:12:04 <SergeyLukjanov> #action SergeyLukjanov add links to the blueprints to roadmap
18:12:12 <SergeyLukjanov> #topic News / updates
18:12:19 <SergeyLukjanov> guys, please
18:12:33 <SergeyLukjanov> I'm still in vacation, so, not so many news from my side
18:12:58 <aignatov> nothing specific from me, just reviewed some code today and continue working on adding anti affinity to scaling with heat
18:13:07 <crobertsrh> I'm currently working on putting the "Java" job type into the dashboard (based on tmckay's Oozie Java action work).
18:13:10 <aignatov> the first day after holidays :)
18:13:16 <jspeidel> any update regarding HBASE  support?
18:13:29 <mattf> i've been chipping away at the savanna cli. comments still welcome on https://blueprints.launchpad.net/python-savannaclient/+spec/python-savannaclient-cli - especially around how to handle template creation.
18:13:33 <alazarev> I've finished IDH plugin initial version: https://review.openstack.org/#/c/56105/
18:13:44 <SergeyLukjanov> jspeidel, not yet, I was on holidays/vacation, we'll continue at Monday
18:14:02 <jspeidel> @SergeyLukjanov ok, thanks
18:14:04 <SergeyLukjanov> jspeidel, and it's still -1'ed
18:14:10 <alazarev> jspeidel: as I see IDH waits for your review :)
18:14:19 <jspeidel> yep, I know :)
18:14:19 <aignatov> yeh, I see that this patch is WIP
18:14:23 * mattf jumps on jspeidel too
18:14:26 * mattf smiles
18:15:08 <SergeyLukjanov> any other updates?
18:15:09 <aignatov> alazarev: patch lgtm to be merged ;)
18:15:14 <jmaron> edp over neutron private nets is paused while I wait on our neutron setup to resurrect from the dead so I can perform functional testing
18:15:25 <alazarev> also, I was able to run Pig job via EDP on IDH cluster (https://review.openstack.org/#/c/64323/)
18:15:29 <mattf> jmaron, neutron issues?
18:15:37 <mattf> alazarev, awesome
18:15:39 <jmaron> IT/host issues
18:15:59 <SergeyLukjanov> also I'd like to mention Spark plugin efforts in ML
18:16:04 <SergeyLukjanov> let me find the link
18:16:10 <jspeidel> @mattf #link https://review.openstack.org/#/c/56105/
18:16:14 <mattf> https://blueprints.launchpad.net/savanna/+spec/spark-plugin
18:16:31 <aignatov> #link https://blueprints.launchpad.net/savanna/+spec/spark-plugin
18:16:42 <SergeyLukjanov> #link http://lists.openstack.org/pipermail/openstack-dev/2014-January/thread.html#23806
18:16:53 <aignatov> corrected you mattf  :P
18:17:14 <mattf> aignatov, not the first, won't be the last!
18:17:38 <SergeyLukjanov> #topic Roadmap update/cleanup
18:17:46 <SergeyLukjanov> action items are on me
18:17:51 <SergeyLukjanov> I don't think that we have any updates to roadmap so far
18:17:58 <aignatov> SergeyLukjanov: these is awesome news, one more plugin, i'm impressed
18:18:03 <mattf> should we scan the roadmap and give feedback? i've not looked at it since summit.
18:18:21 <SergeyLukjanov> mattf, I'm afraid that nothing changed
18:18:52 <SergeyLukjanov> mattf, but it should, I still hope to return back to it and ensure blueprints existence
18:19:09 <mattf> ack
18:19:19 <SergeyLukjanov> #topic General discussion
18:19:42 <SergeyLukjanov> #info the icehouse-2 release will be Jan 24
18:19:43 <mattf> anyone have suggestions on how to create node group / cluster templates from the cli?
18:19:50 <SergeyLukjanov> so, be prepared for the code freeze Jan 22
18:20:15 <alazarev> mattf: request json? :)
18:20:17 <mattf> SergeyLukjanov, i'll do another oslo sync for next week, please add an action for me (remind me to do it next thurs)
18:20:29 <aignatov> mattf, not atm, the first thing is yes -json
18:20:37 <SergeyLukjanov> mattf, you can find some of my thoughts in the CLI blueprint
18:20:38 <jmaron> savanna template-create < template.json ?
18:20:43 <mattf> alazarev, i was thinking about that actually. maybe creating only by piping in a json file?
18:20:52 <jspeidel> alazarev +1 for json request bodhy
18:20:54 <jspeidel> body
18:20:56 <aignatov> jmaron: smthg like this
18:21:31 <mattf> afaict, it's complicated by the fact that template json is plugin specific
18:21:34 <aignatov> mattf: you can get an idea from heat ;)
18:21:38 <SergeyLukjanov> #action mattf to sync oslo right before the i2 code freeze
18:21:44 <mattf> for instance, process names aren't consistent between plugins
18:22:06 <mattf> and hdp supports more processes than vanilla
18:22:17 <SergeyLukjanov> I'm ok with piping json template for the first time at least
18:22:29 <alazarev> mattf: and they will not be consistent because different distros have different namings
18:22:31 <mattf> i've an action on myself to work out the best way to list processes to help in contructing json, but...that's not very userfiendly
18:22:45 <SergeyLukjanov> #action SergeyLukjanov to unsure that we'rein sync with global requirements
18:22:46 <jmaron> there's a precedent for plugin specific template processing in the plugin SPI
18:22:51 <SergeyLukjanov> #undo
18:22:52 <openstack> Removing item from minutes: <ircmeeting.items.Action object at 0x30f5350>
18:22:55 <mattf> alazarev, we could have some consistency tho, NameNode vs NAMENODE etc
18:23:02 <SergeyLukjanov> #action SergeyLukjanov to ensure that we're in sync with global requirements
18:23:28 <mattf> (someone needs to fix #undo to print the object string0
18:23:39 <mattf> not it
18:23:43 <SergeyLukjanov> mattf, alazarev, that's a problem of choosing vendor names vs. our common names
18:24:15 <mattf> seems plugins could maintain a simple map from common name to vendor name
18:24:30 <jspeidel> mattf, I don't think that we can expect consistency across providers.  Each provider should expose names that are meaningful for their stack
18:24:42 <mattf> set of common names would have to be union of all vendor features tho
18:24:47 <aignatov> jspeidel: +1
18:25:12 <jmaron> vendor CLI plugins for template processing (akin to SPI)?
18:25:15 <alazarev> some services could be splited into severel processes, some services are specific for distro, I don't think we need to force namings
18:25:15 <mattf> jspeidel, so far the examples are somewhat silly tho, the primary difference is all caps for HDP
18:25:50 <alazarev> mattf: this is because we have only HDP and vanilla
18:25:52 <jmaron> vanilla:  "oozie"  HDP:  OOZiE_SERVER, OOZIE_CLIENT
18:26:04 <mattf> so the proposal (well liked) is currently to simple accept json files. that means to actually create a template the user must essentially use horizon and then export it.
18:26:12 <jspeidel> mattf, agreed that the current differences are simple, but I think that assuming a consistency is a slippery slope
18:26:56 <mattf> jmaron, thanks for finding a nice example. in this case it seems like the vanilla is simply being less specific
18:27:01 <alazarev> if we have vanilla, HDP, IDH, cloudera, etc - there will be much more differences
18:27:02 <jmaron> (to point out less trivial differences)
18:27:41 <aignatov> don't forget about incoming Spark plugin, I think there will be another set of processes :)
18:27:45 <mattf> my view on this is we should make using savanna simple and consistent for users, no matter what plugin they are using on the backend
18:27:58 <mattf> ^^ my guiding rule, which eventually has exceptions
18:28:14 <mattf> aignatov, yeah, set would have to be union of all plugins
18:28:33 <jspeidel> mattf, agree with usability but don't think the we can dictate to providers what to call components
18:28:57 <jspeidel> mattf, each plugins users know the names of the components associated with the vendors stack
18:28:59 <mattf> jspeidel, the vendor would not have to call them something internally, only when representing them out to savanna
18:29:06 <jmaron> we could follow this already established precedent/convention:  https://savanna.readthedocs.org/en/latest/devref/plugin.spi.html#convert-config-plugin-name-version-template-name-cluster-template-create
18:30:31 <mattf> imho, no need to make a decision here. seems to me the pressure is for no change to how the plugins work at the moment. i want to make sure folks are aware of the differences and that they can impact usability.
18:30:43 <mattf> (they're impacting usability in how i can make the cli already!)
18:30:49 <jspeidel> mattf, agreed
18:30:59 <alazarev> mattf: +1
18:31:30 * mattf gets off soapbox
18:32:07 <mattf> another related topic - destroy or delete - what's the proper verb for our cli?
18:32:18 <mattf> imho we should just be consistent w/ other clis
18:32:34 <jmaron> annihilate
18:32:40 <mattf> e.g. cluster-destroy or cluster-delete, node-group-template-destroy or node-group-template-delete
18:32:43 <mattf> obliterate?
18:32:52 <aignatov> jmaron: lol!
18:33:09 <aignatov> mattf, +1 to be consistent with other clis
18:33:30 <alazarev> I vote on delete
18:33:36 <mattf> aignatov, clis aren't consistent, how would you rank them (nova then keystone then cinder?)
18:33:49 <aignatov> cluster-delete and node-group-template-delete my vote
18:34:00 <jmaron> +1 for delete
18:34:07 <jspeidel> mattf, +1 for delete
18:34:12 * mattf starts to see himself s/destroy/delete/g'ing
18:34:36 <jspeidel> mattf, should have asked sooner ;)
18:34:45 <jmaron> s/destroy/euthanize/g
18:34:47 * mattf hangs head in shame
18:34:53 <mattf> jmaron, you're on fire
18:35:04 <jmaron> :)
18:35:05 <mattf> ok, i'll tend to delete and do another pass over other clis
18:35:13 <tmckay> just checked the REST apis, looks like it is all "delete" there too
18:35:15 <aignatov> mattf, yep, rely on nova first, imo
18:35:16 <tmckay> +1 delete
18:35:24 <mattf> tmckay, delete is the REST verb
18:35:51 <SergeyLukjanov> I'm ok with both destoy and delete ;)
18:36:02 <mattf> ok, thanks for the input folks!
18:36:05 <tmckay> well, true.  I was looking at method names.  Doubly consistent :)
18:36:32 <jmaron> SergeyLukjanov: node-template-destroy-and-delete?
18:36:41 <mattf> fyi - i'm finding all sorts of rough edges in the client, so i'll start writing them up for client v2
18:37:04 <mattf> (kinda glad i'm finding rough edges, one reason for doing the cli was to evaluate the usability of the api!)
18:37:28 <alazarev> destroy-delete-and-remove-completely
18:37:51 * mattf hopes someone brings up another topic before we get to exterminate
18:37:53 <aignatov> destroy-delete-and-remove-completely-and-run-away
18:38:11 * mattf waits for Aliens reference
18:38:12 <alazarev> during IDH plugin development I faced with problem that auth token is expired because of long inactivity. Auth token was only needed to take username from image tags. So, I proposed caching for username in node group.extra (https://review.openstack.org/#/c/65402/).
18:38:39 <alazarev> SergeyLukjanov: you left a comment that this need to be discussed
18:38:40 <jmaron> *get away from my template you b****!*
18:38:58 <mattf> jmaron, lol, i was thinking nuke from orbit
18:39:20 <jmaron> :)
18:39:23 <mattf> jspeidel, what on earth do you guys have in your break rooms!?
18:39:26 <aignatov> xD
18:39:29 <SergeyLukjanov> jmaron, heh, that's a good option :)
18:39:57 <SergeyLukjanov> alazarev, my concern is extra field ressurection
18:40:24 <alazarev> SergeyLukjanov: what wrong with it? extra is good
18:40:36 <mattf> alazarev, SergeyLukjanov, aignatov, does this become a non-issue w/ heat?
18:41:00 <SergeyLukjanov> alazarev, it's a kind of necrophilia
18:41:12 <SergeyLukjanov> mattf, extra field?
18:41:13 * mattf covers eyes
18:41:26 <mattf> SergeyLukjanov, need storing username on image
18:41:33 <mattf> need for*
18:41:34 <alazarev> mattf: yes, heat removes the issue, but it would be good to get plugin work before full switch to heat
18:41:37 <dmitryme> mattf, indeed it is not an issue with the heat
18:41:54 <mattf> alazarev, how long is "long inactivity"?
18:42:03 <aignatov> SergeyLukjanov: do you know that we already use extra for hadoop keypairs?
18:42:04 <mattf> and who's inactivity?
18:42:07 <mattf> whose*
18:42:08 <dmitryme> for those who don't know: all instances provisioned by Heat have ec2-user
18:42:21 <dmitryme> i.e. static username for all images
18:42:24 <alazarev> mattf: 30+ mins, savanna waits for manager to install something
18:42:32 <mattf> dmitryme, we could change that via userdata, right?
18:42:40 <mattf> alazarev, that's not very long imho
18:42:51 <dmitryme> mattf: change what? :-)
18:42:52 <mattf> not very long -> up priority
18:43:07 <mattf> dmitryme, os-user vs ec2-user. it's just ec2-user beacuse that's the cloud-init default?
18:43:08 <alazarev> mattf: I didn't dig why token expires, but it did
18:43:11 <SergeyLukjanov> aignatov, yup, I remember
18:43:35 <dmitryme> mattf: it is Heat default
18:43:36 <mattf> SergeyLukjanov, alazarev, what about letting in alazarev's change and filing a but to remove it once we go to heat?
18:43:44 <mattf> dmitryme, ahh
18:43:56 <mattf> eesh bug*
18:44:07 <dmitryme> Heat pushes onto instance not only userdata provided by the user, but also some additional scripts
18:44:19 <dmitryme> one of them sets up ec2-user
18:44:29 <mattf> dmitryme, i didn't know that, thanks
18:44:33 <aignatov> dmitry, thx, with heat we will remove username field from registering image
18:44:40 <alazarev> and, once again… what wrong with extra? engines could use it whatever they need
18:44:57 <dmitryme> mattf: no problem, it was a surprise for me actually
18:45:03 <mattf> alazarev, that's kinda what's wrong with it
18:45:34 <mattf> dmitryme, i'm sure you just saved me a ton of pain later on when i'd be trying to figure out why things are happening when i didn't explicitly ask for them to happenm
18:45:54 <aignatov> alazarev: I have nothing against extra
18:45:56 <mattf> alazarev, i'm flexible, ok w/ adding so long as we record the fact that we should later remove it
18:46:15 <dmitryme> alazarev: I would prefer one more specific field in node group, like 'username', not just 'extra'
18:46:43 <SergeyLukjanov> alazarev, my concern is that we're resurrecting extra for some objects in different patches instead of thinking about the common way to store additional info for objects
18:46:52 <SergeyLukjanov> custom/optional info
18:46:54 <alazarev> dmitryme: heat doesn't need such field, what's why I used extra
18:46:58 <SergeyLukjanov> plugin-specific maybe too
18:47:18 <SergeyLukjanov> heat needs it too
18:47:23 <SergeyLukjanov> but it'll be ec2-user
18:47:31 <alazarev> SergeyLukjanov: extra is a common way, no?
18:47:37 <SergeyLukjanov> I think that could make it configurable for example
18:47:43 <SergeyLukjanov> heat guys *
18:48:11 <SergeyLukjanov> alazarev, not really, we're adding it for each object when we're need it
18:48:50 <dmitryme> alazarev: but the plugin does not know which engine is used, Heat or Direct
18:49:07 <tmckay> hmmm, we have an "extra" in job_binary for EDP.  Maybe I should try to stamp that out too
18:49:11 <dmitryme> so it does not know how to get the username in a uniform way
18:49:16 <alazarev> SergeyLukjanov: exactly! and I don't see problems here, let's extra be
18:49:39 <aignatov> tmckay: it is specific EDP extra, don't worry about it :)
18:49:51 <tmckay> is "extra" extra good or extra bad?  That is the question
18:50:18 <mattf> "extra" is a red flag for me, means we missed something in the design
18:50:18 <aignatov> in EDP it is good
18:50:34 <alazarev> dmitryme: yes, that's why we have corresponding method in engine. Engine could use extra or don't use
18:50:39 <tmckay> :)  a Python programmer's best friend
18:50:53 <SergeyLukjanov> tmckay, yup
18:51:14 <dmitryme> alazarev: we can simply unconditionally set the username field in node group before passing the cluster to the plug
18:51:24 <dmitryme> IMHO that is much simplier
18:51:49 <jmaron> dmitryme:  I think I like that
18:51:50 <mattf> dmitryme, nice thought
18:52:02 <alazarev> mattf: if we keep something important in extra - yes, if it is used for caching or passing params - it's Ok for me
18:52:18 <SergeyLukjanov> dmitryme, +1
18:52:52 <alazarev> ok, will change to username field, extra was used because of heat only
18:53:26 <SergeyLukjanov> alazarev, yup, username field looks good
18:53:51 <aignatov> ok, agreed with username field
18:53:52 <SergeyLukjanov> probably we can add something to the orchestration engines to be able to override get_username behaviour
18:54:13 <aignatov> but what should we do with current extra field options in cluster obis?
18:54:15 <aignatov> https://github.com/openstack/savanna/blob/master/savanna/plugins/vanilla/config_helper.py#L222-L232
18:54:26 <mattf> alazarev, thanks for bring this up during the meeting. it helped me understand what you were after w/ that cr
18:55:10 <tmckay> so, while we're here...
18:55:28 <tmckay> I think I have just enough time to do more workflow extensions before Jan 22
18:55:43 <tmckay> what would people like to see?  I know there is a roadmap note somewhere
18:55:53 <mattf> spark?
18:56:02 <tmckay> Maybe raw oozie workflows, or streaming
18:56:29 <tmckay> mattf, I am unaware of oozie/spark integratoin
18:56:47 <mattf> me too, more a ref to the inbound spark plugin
18:56:49 <alazarev> aignatov: I vote to move hadoop_private_ssh_key and hadoop_public_ssh_key into fields and remove extra… to be consistent
18:57:10 <tmckay> also, there is a shell workflow that might be nice
18:57:21 <aignatov> tmckay: maybe integration tests support of Java added action, updated rest api docs about new action?
18:57:22 <mattf> alazarev, btw, what do we need the public part of that key for?
18:57:42 <SergeyLukjanov> alazarev, it's only used by the vanilla plugin now I think, so, it's not good to make it a field
18:57:48 <tmckay> aignatov, definitely, I was thinking maybe I have enough time after that for another...
18:58:11 <aignatov> tmckay: what will be after Jan 22?
18:58:26 <tmckay> Oh, that's just the freeze
18:58:34 <SergeyLukjanov> tmckay, code freeze for i2
18:58:37 <dmitryme> alazarev: do we really need to store these keys for the Vanilla plugin?
18:58:50 <dmitryme> I can't imagine why?
18:59:35 <aignatov> dmitryme: we need, it simplify hadoop debugging after install
18:59:45 <alazarev> SergeyLukjanov: this means that plugins could store data in cluster.extra but can't in node group.extra, that looks strange for me
19:00:07 <alazarev> dmitryme: to put them on machine during scaling I believe
19:00:16 <mattf> alazarev, you're not wrong
19:00:42 <SergeyLukjanov> guys, I'd like to ask you to go through the https://launchpad.net/savanna/+milestone/icehouse-2 and decide what you'd like to do in i2
19:00:46 <SergeyLukjanov> we're out of time
19:00:48 <SergeyLukjanov> thanks all
19:00:49 <aignatov> tmckay: another thing I thought in EDP would be nice is to implement multiple workflows
19:00:55 <SergeyLukjanov> #endmeeting