18:09:02 <SergeyLukjanov> #startmeeting savanna
18:09:03 <openstack> Meeting started Thu Feb  6 18:09:02 2014 UTC and is due to finish in 60 minutes.  The chair is SergeyLukjanov. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:09:04 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:09:06 <openstack> The meeting name has been set to 'savanna'
18:09:08 <SergeyLukjanov> #topic Agenda
18:09:10 <SergeyLukjanov> #link https://wiki.openstack.org/wiki/Meetings/SavannaAgenda
18:09:22 <SergeyLukjanov> #topic News / updates
18:09:28 <SergeyLukjanov> folks, please
18:09:32 <SergeyLukjanov> short update section
18:09:39 <SergeyLukjanov> ylobankov1, ping,
18:09:46 <mattf> i caused some headaches by filing a bunch of blueprints <- count as an update?
18:09:49 <SergeyLukjanov> could you please make a QA update?
18:09:53 <ylobankov1> I am here
18:10:10 <SergeyLukjanov> mattf, it's an update for both of us :)
18:10:27 <ylobankov1> HDP plugin works with Heat
18:10:34 <mattf> it looks like the cli is in place and supports --name, so we have a solid base for 0.5. i still need to fully vet an edp use case tho
18:10:48 <SergeyLukjanov> anyway, I have a good news - now we have fully functional async GATE!!
18:10:49 <aignatov> ylobankov1: cool, I saw SUCCEED
18:10:56 <SergeyLukjanov> mattf, that's awsome!
18:11:06 <crobertsrh> I've been working on streaming MapReduce in the dashboard.  Along with a few bug fixes to a few bugs that have been around awhile.  I have a question about one of them.
18:11:17 <SergeyLukjanov> mattf, I think that it could be released in 1-2 weeks
18:11:19 <mattf> SergeyLukjanov, awesome!
18:11:22 <SergeyLukjanov> I mean client 0.5.0
18:11:25 <mattf> re gate
18:11:33 <aignatov> I'm worked all the week on the adding savanna resource to heat
18:11:36 <bob_nettleton> I'm working on a separate task this week, so not much progress on the diskimage-builder images for HDP.  I hope to submit a patch review within a week.
18:11:38 <tmckay> modified java actions are finally rebased on something stable and pass jenkins/ci.  Streaming mapreduce works through the api, but we are including dotted job types to better support the UI.
18:11:45 <aignatov> prepared the first patch there
18:11:48 <NikitaKonovalov> I've sent the change introducing pecan/wsme stuff to start v2 api
18:11:55 <SergeyLukjanov> me and ylobankov1 are working now on extending basic API tests to tempest
18:12:14 <sreshetnyak> i worked on the integration tests for IDH plugin
18:12:18 <mattf> update for venza - we have the beginnings of the spark contribution in the dib elements repo
18:12:19 <SergeyLukjanov> bob_nettleton, great, looking forward for it
18:12:28 <aignatov> I hope to make series of 3 patches: cluster create on template, full cluster create with all fields and cluster update
18:12:36 <mattf> NikitaKonovalov, i hope to try that out today/tomorrow
18:12:43 <tmckay> I started reviewing the v2 API (CR and email thread) and want to discuss 'cancel' of a job during general topics
18:12:50 <mattf> NikitaKonovalov, if you have any pointers that'll make it easier for me, i'm all ears
18:13:01 <alazarev> I started investigation on what is needed to support hadoop 2.x
18:13:20 <SergeyLukjanov> folks, please, note that the v2 api CR is just an addition of framework with samples, not the new api :)
18:13:44 <NikitaKonovalov> mattf, here is the link https://review.openstack.org/#/c/63908/
18:13:46 <SergeyLukjanov> mattf, what's your estimate for cli stuff to be able to release 0.5.0 client and consume it everywhere?
18:13:48 <tmckay> it's a target to shoot at :)
18:14:19 <mattf> SergeyLukjanov, i'll check the bps again, but i think we have a good v0 of the cli now
18:14:31 <mattf> or v11 is maybe a better way to put it
18:14:42 <aignatov> SergeyLukjanov: I'd like to see full unit tests set for 0.5 release of savannaclient
18:14:47 <aignatov> #link https://blueprints.launchpad.net/python-savannaclient/+spec/python-savannaclient-unit-tests
18:15:09 <tmckay> mattf, with the java action changes currently staged, we can drop "job_exec_data" from the client create() as long as we sync the merges between savanna api/UI and the client
18:15:19 <mattf> crobertsrh, ^^
18:15:42 <mattf> tmckay, i need to catch up on that, help me after this meeting?
18:15:43 <crobertsrh> Yeah, I'm on it.
18:15:51 <tmckay> mattf, sure.
18:15:58 <crobertsrh> I think I have that change ready to go.
18:16:10 <tmckay> My plan was to start looking at cli tests this week, but I was mired in rebases
18:16:20 <tmckay> Maybe now I have a chance
18:16:28 <alazarev> I like how novaclient is tested by UT, probably we need to do the same thing
18:16:32 <SergeyLukjanov> aignatov, it sounds reasonable, but we have a good integration tests coverage for latest released version, probably, we should adopt the jobs to test client too
18:17:02 <aignatov> News/updaes section of this meeting turned to General discussions :-)
18:17:03 <SergeyLukjanov> aignatov, I'm not sure that we have enough bandwidth to resolve it before the mid of i3
18:17:15 <SergeyLukjanov> and to the i3 dev status
18:17:20 <SergeyLukjanov> #topic i3 dev status
18:17:34 <aignatov> SergeyLukjanov: I can make it possibly
18:17:34 <SergeyLukjanov> #link https://launchpad.net/savanna/+milestone/icehouse-3
18:17:51 <SergeyLukjanov> aignatov, It'll be great to have a first release of client with tests :)
18:18:03 <SergeyLukjanov> and probably enable support of py3k, I can help with it
18:18:16 <mattf> i'm still looking for +1s and +2s on the api endpoints - https://review.openstack.org/#/c/70627/
18:18:17 <SergeyLukjanov> folkd, please, take a look at https://launchpad.net/savanna/+milestone/icehouse-3 and say what's missed
18:18:41 <SergeyLukjanov> or what should be postponed so far
18:19:00 <SergeyLukjanov> jmaron, jspeidel, ping
18:19:06 <SergeyLukjanov> heh
18:19:25 <mattf> SergeyLukjanov, i'll review and make suggestions on #savanna later
18:19:32 <SergeyLukjanov> thx
18:19:46 <SergeyLukjanov> I'd like to ask all of us to check their blueprints statuses
18:20:09 <SergeyLukjanov> and if you're working on blueprint that isn't targeted to i3, please, ping me
18:20:35 <aignatov> hm, it seems already implemented https://blueprints.launchpad.net/savanna/+spec/epd-data-source-existing-hadoop-cluster
18:20:42 <SergeyLukjanov> we'll enable soft feature freeze after i3
18:21:21 <SergeyLukjanov> aignatov, agreed, both server side and dashboard side merged
18:21:35 <aignatov> sreshetnyak: here?
18:21:35 <SergeyLukjanov> need to confirm it with Trevor and Chad
18:21:56 <sreshetnyak> here
18:22:26 <aignatov> You was assigned on external hdfs. is it fully completed?
18:22:52 <crobertsrh> I believe it is completed
18:23:31 <sreshetnyak> need to add integration tests
18:23:34 <tmckay> I haven't used it for an external cluster, but I have run it with hdfs data sources on the same running cluster
18:24:01 <mattf> +1.5
18:24:25 <aignatov> I ran it on external hdfs with earliest patches of this commit ;)
18:24:27 <SergeyLukjanov> ok, I've mark it implemented
18:24:33 <aignatov> it worked
18:24:37 <SergeyLukjanov> any questions?
18:24:44 <aignatov> on my laptop ;-)
18:24:52 <mattf> heh
18:24:59 <mattf> +.5
18:25:45 <aignatov> matff, don't you trust my laptop?
18:25:55 <tmckay> I think all my currently approved blueprints will be implemented very soon, with maybe the exception of https://blueprints.launchpad.net/savanna/+spec/edp-oozie-files-and-archives
18:26:30 <tmckay> files and archives are not strictly necessary -- streaming mapreduce can use 'libs' to bundle files if a script needs to be uploaded
18:26:39 <tmckay> but they would be nice on a long-running cluster
18:27:20 <mattf> aignatov, just getting it to +2
18:27:34 <crobertsrh> I have a question regarding https://bugs.launchpad.net/savanna/+bug/1212354  It's a bug against the UI that happens when you empty out a parameter from the HDFS config tab for a cluster template (or probably most tabs).  The client treats it as ok because the plugin returns a config set that sets those values as optional.  Who would be the right person to talk to about this?
18:29:18 <crobertsrh> I should say that the dashboard treats it as ok ^^^
18:29:59 <mattf> jmaron, welcome
18:30:20 <jmaron> sorry I'm late
18:30:33 <SergeyLukjanov> jmaron, hey, any updates?
18:30:39 <jmaron> lost track of time...
18:30:50 <mattf> all that time in the break room
18:30:51 <jmaron> putting finishing touched on HDP 2 integration
18:30:58 <jmaron> touches
18:31:08 <jmaron> should have something for review today or tomorrow
18:31:29 <jmaron> I'd like to commit an initial patch that addresses functionality
18:31:41 <aignatov> crobertsrh: I'll try to take a  look on this bug tomorrow
18:31:43 <jmaron> with the understanding that there'll be a follow up commit with some needed refactorings
18:32:18 <mattf> jmaron, you know i'm a fan of phasing changes
18:32:20 <crobertsrh> aignatov:  Thanks.  Let me know what you think about it.  I may be able to tweak the dashboard if necessary, but I think it would be rather ugly.
18:32:31 <jmaron> I think doing the refactorings at this time will make the patch unwieldy…
18:32:50 <jmaron> so no comments about duplicate code ;)
18:33:03 <mattf> fair enough
18:33:23 <aignatov> crobertsrh: agreed, when I'll figure out whats happening I'll comment to the bug or to you in channel :)
18:33:31 <crobertsrh> Sounds good
18:34:00 <mattf> bob_nettleton, btw, i'm eagerly awaiting your dib patches. are you planning to separate out the java & ssh elements at the same time?
18:35:41 <SergeyLukjanov> ok, let's move on
18:35:44 <bob_nettleton> mattf, not in the initial patch submission, but I do think it makes sense to break these out into common elements.  My hope was to get the initial submission in, and then I could refactor the scripts once the common elements were available.
18:36:12 <mattf> bob_nettleton, should the refactoring bps be assigned to you?
18:37:06 <bob_nettleton> mattf, sure, that would be ok for now.  if my task assignments change, we might need to re-assign, but I could certainly take a look at this.
18:37:21 <mattf> bob_nettleton, ok
18:37:22 <bob_nettleton> mattf, should the common elements go in DIB itself, or in savanna-image-elements?
18:37:49 <mattf> imho, we should try to get all our elements into dib itself. only truly general ones will make the cut.
18:37:51 <SergeyLukjanov> bob_nettleton, I think the first step is complete them in s-i-e
18:38:09 <mattf> so you can file a review against both. in the past i'ev filed against dib and when rejected filed against savanna
18:38:30 <bob_nettleton> SergeyLukjanov, mattf, ok, sounds fine.  thanks!
18:38:46 <SergeyLukjanov> #topic Project naming collision
18:38:56 <SergeyLukjanov> #link https://etherpad.openstack.org/p/savanna-renaming
18:39:03 <SergeyLukjanov> there are some proposed options alreadt
18:39:17 <SergeyLukjanov> let's continue discussion of new name in this etherpad
18:39:27 <mattf> do we have a deadline?
18:39:44 <SergeyLukjanov> I'm thinking about setting up a voting to cut top XX options
18:39:56 <SergeyLukjanov> in 1-1.5 weeks
18:40:01 <aignatov> we need much more proposed names
18:40:06 <mattf> we do
18:40:18 <mattf> is openstack legal giving us a deadline?
18:40:21 <aignatov> top XX means 20?
18:40:25 <SergeyLukjanov> I'd like to select the new name around the i3
18:40:32 <SergeyLukjanov> aignatov, I hope 5
18:40:40 <alazarev> SergeyLukjanov: did you ask about Savannah?
18:40:41 <SergeyLukjanov> mattf, there is now hard deadline
18:40:59 <mattf> are there milestones set, such as needing to be fully transitioned by i ga?
18:41:03 <SergeyLukjanov> mattf, but if we'd like to graduate from the incubation :)
18:41:03 <dmitryme> Hey, by the way, are you going to present all the options on the voting, or we will filter out some of them before?
18:41:19 <mattf> SergeyLukjanov, you speak the magic words...
18:41:53 <mattf> so we need to fully rename by I. we need to get through legal w/ the new name before we rename, and we need to prune the list before giving it to legal?
18:41:53 <SergeyLukjanov> dmitryme, I'd like to have several iterations
18:42:10 <SergeyLukjanov> mattf, yup
18:42:18 <mattf> are there any estimates on how long renaming takes and how long legal needs?
18:42:36 <dmitryme> SergeyLukjanov: yes, seems like a good idea. So at the first iteration we will review all the options and select, say 20 of them, right?
18:42:47 <SergeyLukjanov> mattf, and it'll be the best option to have a new name around i3 == before the graduation review
18:42:55 <aignatov> dmitryme: I guess so
18:43:14 <SergeyLukjanov> mattf, <=1w for legals I hope, ~1-2w for renaming
18:43:24 <mattf> SergeyLukjanov, so i3 - legal time == our deadline
18:43:55 <SergeyLukjanov> mattf, yup, that's the best case
18:44:06 <mattf> i just started soliciting names at rht, so i'd like to go on the 1.5-2wk for vote end
18:44:32 <SergeyLukjanov> i3 == Mar 6
18:44:43 <SergeyLukjanov> so, we have 4 w
18:45:34 <mattf> is legal going to output a single name or a further pruned list?
18:46:15 <mattf> ...do we need a second round of name voting...
18:46:20 <aignatov> "noooo, please, please let's us leave our beautiful name Savanna..." - we will write to legals :)
18:46:42 <mattf> aignatov, i'll +1 that name
18:46:54 <SergeyLukjanov> mattf, legal could check a short list of names for us
18:47:40 <mattf> so 4w - 1w voting round 2 - 1.5w legal - 1w voting round 1 == we need to start voting in a week. ok.
18:48:28 <mattf> i retract my request for 2weeks, thx
18:48:55 <SergeyLukjanov> I'd like to start voting right after the next meeting
18:49:09 <SergeyLukjanov> probably, we'll need to discuss something
18:49:33 <aignatov> SergeyLukjanov: you mean first round of voting, right?
18:49:42 <SergeyLukjanov> and I'll setup up for a week, but if it'll collect enough votes, it could be ended earlier
18:49:44 <SergeyLukjanov> aignatov, yup
18:49:49 <aignatov> ok
18:50:32 <SergeyLukjanov> honestly, I don't see any good options now :( Savanna is too own...
18:50:39 <SergeyLukjanov> heh
18:50:52 <SergeyLukjanov> #topic General discussion
18:52:14 <SergeyLukjanov> any questions/ideas?
18:52:15 <mattf> aignatov, alazarev, tmckay, crobertsrh, any comments on the base v2 api?
18:52:26 <tmckay> I would like to reconsider job "cancel" in v2 api
18:52:28 <tmckay> The reason:
18:52:37 <alazarev> mattf: looks good for me
18:52:50 <tmckay> crobertsrh notes that job relaunch is implemented on the job executions tab
18:52:54 <SergeyLukjanov> mattf, probably I missed it, but do you have any general api changes section?
18:52:59 <mattf> tmckay, reconsider? nothing is decided on it atm, just not listed in base.
18:53:03 <crobertsrh> I tend to agree with tmckay.
18:53:03 <SergeyLukjanov> mattf, like don't include tenant-id to url?
18:53:04 <tmckay> so, the use case is cancel a job, tweak the data or configs, and relaunch
18:53:19 <tmckay> mattf, my mistake.  I thought missing meant gone :)
18:53:20 <mattf> SergeyLukjanov, i focused on the endpoints
18:53:26 <SergeyLukjanov> mattf, ok
18:54:05 <mattf> tmckay, so it's a stop + modify + restart use case?
18:54:06 <aignatov> for long running jobs the 'cancel' operation is extremely needed
18:54:11 <tmckay> mattf, yes
18:54:21 <mattf> aignatov, do you see cancel == stop ?
18:54:55 <aignatov> yep, stop it running on cluster
18:55:21 <mattf> aignatov, stop cluster or stop job?
18:55:27 <aignatov> stop job
18:55:47 <mattf> as in, for a transient cluster. does the clsuter survive the stop + restart?
18:55:59 <tmckay> good question
18:56:14 <aignatov> transient custer is another case :)
18:56:16 <crobertsrh> I would say that transient cluster goes away
18:56:18 <SergeyLukjanov> it'll be killed in current code
18:56:43 <aignatov> cluster should be killed, yes
18:56:44 <mattf> if it's killed, the stop + edit + restart ~= delete + edit + launch new
18:57:22 <tmckay> except that delete wipes it from the database, so you have to start from scratch.  Maybe a little more typing
18:57:53 <mattf> another way is "clone" a running job. so "clone w/ edit" then delete old.
18:58:24 <aignatov> I said about 'cancel' jobs mostly targeted to persistent clusters (not transient)
18:58:29 <mattf> i'm just tossing out ideas that don't require us to add another verb, but give us the same use case.
18:58:56 <mattf> fyi, clone can be done entirely client side
19:00:01 <mattf> may be out of time, -> #savana
19:00:04 <mattf> may be out of time, -> #savanna
19:00:08 <SergeyLukjanov> exactly
19:00:12 <SergeyLukjanov> #endmeeting