18:09:02 <SergeyLukjanov> #startmeeting savanna 18:09:03 <openstack> Meeting started Thu Feb 6 18:09:02 2014 UTC and is due to finish in 60 minutes. The chair is SergeyLukjanov. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:09:04 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:09:06 <openstack> The meeting name has been set to 'savanna' 18:09:08 <SergeyLukjanov> #topic Agenda 18:09:10 <SergeyLukjanov> #link https://wiki.openstack.org/wiki/Meetings/SavannaAgenda 18:09:22 <SergeyLukjanov> #topic News / updates 18:09:28 <SergeyLukjanov> folks, please 18:09:32 <SergeyLukjanov> short update section 18:09:39 <SergeyLukjanov> ylobankov1, ping, 18:09:46 <mattf> i caused some headaches by filing a bunch of blueprints <- count as an update? 18:09:49 <SergeyLukjanov> could you please make a QA update? 18:09:53 <ylobankov1> I am here 18:10:10 <SergeyLukjanov> mattf, it's an update for both of us :) 18:10:27 <ylobankov1> HDP plugin works with Heat 18:10:34 <mattf> it looks like the cli is in place and supports --name, so we have a solid base for 0.5. i still need to fully vet an edp use case tho 18:10:48 <SergeyLukjanov> anyway, I have a good news - now we have fully functional async GATE!! 18:10:49 <aignatov> ylobankov1: cool, I saw SUCCEED 18:10:56 <SergeyLukjanov> mattf, that's awsome! 18:11:06 <crobertsrh> I've been working on streaming MapReduce in the dashboard. Along with a few bug fixes to a few bugs that have been around awhile. I have a question about one of them. 18:11:17 <SergeyLukjanov> mattf, I think that it could be released in 1-2 weeks 18:11:19 <mattf> SergeyLukjanov, awesome! 18:11:22 <SergeyLukjanov> I mean client 0.5.0 18:11:25 <mattf> re gate 18:11:33 <aignatov> I'm worked all the week on the adding savanna resource to heat 18:11:36 <bob_nettleton> I'm working on a separate task this week, so not much progress on the diskimage-builder images for HDP. I hope to submit a patch review within a week. 18:11:38 <tmckay> modified java actions are finally rebased on something stable and pass jenkins/ci. Streaming mapreduce works through the api, but we are including dotted job types to better support the UI. 18:11:45 <aignatov> prepared the first patch there 18:11:48 <NikitaKonovalov> I've sent the change introducing pecan/wsme stuff to start v2 api 18:11:55 <SergeyLukjanov> me and ylobankov1 are working now on extending basic API tests to tempest 18:12:14 <sreshetnyak> i worked on the integration tests for IDH plugin 18:12:18 <mattf> update for venza - we have the beginnings of the spark contribution in the dib elements repo 18:12:19 <SergeyLukjanov> bob_nettleton, great, looking forward for it 18:12:28 <aignatov> I hope to make series of 3 patches: cluster create on template, full cluster create with all fields and cluster update 18:12:36 <mattf> NikitaKonovalov, i hope to try that out today/tomorrow 18:12:43 <tmckay> I started reviewing the v2 API (CR and email thread) and want to discuss 'cancel' of a job during general topics 18:12:50 <mattf> NikitaKonovalov, if you have any pointers that'll make it easier for me, i'm all ears 18:13:01 <alazarev> I started investigation on what is needed to support hadoop 2.x 18:13:20 <SergeyLukjanov> folks, please, note that the v2 api CR is just an addition of framework with samples, not the new api :) 18:13:44 <NikitaKonovalov> mattf, here is the link https://review.openstack.org/#/c/63908/ 18:13:46 <SergeyLukjanov> mattf, what's your estimate for cli stuff to be able to release 0.5.0 client and consume it everywhere? 18:13:48 <tmckay> it's a target to shoot at :) 18:14:19 <mattf> SergeyLukjanov, i'll check the bps again, but i think we have a good v0 of the cli now 18:14:31 <mattf> or v11 is maybe a better way to put it 18:14:42 <aignatov> SergeyLukjanov: I'd like to see full unit tests set for 0.5 release of savannaclient 18:14:47 <aignatov> #link https://blueprints.launchpad.net/python-savannaclient/+spec/python-savannaclient-unit-tests 18:15:09 <tmckay> mattf, with the java action changes currently staged, we can drop "job_exec_data" from the client create() as long as we sync the merges between savanna api/UI and the client 18:15:19 <mattf> crobertsrh, ^^ 18:15:42 <mattf> tmckay, i need to catch up on that, help me after this meeting? 18:15:43 <crobertsrh> Yeah, I'm on it. 18:15:51 <tmckay> mattf, sure. 18:15:58 <crobertsrh> I think I have that change ready to go. 18:16:10 <tmckay> My plan was to start looking at cli tests this week, but I was mired in rebases 18:16:20 <tmckay> Maybe now I have a chance 18:16:28 <alazarev> I like how novaclient is tested by UT, probably we need to do the same thing 18:16:32 <SergeyLukjanov> aignatov, it sounds reasonable, but we have a good integration tests coverage for latest released version, probably, we should adopt the jobs to test client too 18:17:02 <aignatov> News/updaes section of this meeting turned to General discussions :-) 18:17:03 <SergeyLukjanov> aignatov, I'm not sure that we have enough bandwidth to resolve it before the mid of i3 18:17:15 <SergeyLukjanov> and to the i3 dev status 18:17:20 <SergeyLukjanov> #topic i3 dev status 18:17:34 <aignatov> SergeyLukjanov: I can make it possibly 18:17:34 <SergeyLukjanov> #link https://launchpad.net/savanna/+milestone/icehouse-3 18:17:51 <SergeyLukjanov> aignatov, It'll be great to have a first release of client with tests :) 18:18:03 <SergeyLukjanov> and probably enable support of py3k, I can help with it 18:18:16 <mattf> i'm still looking for +1s and +2s on the api endpoints - https://review.openstack.org/#/c/70627/ 18:18:17 <SergeyLukjanov> folkd, please, take a look at https://launchpad.net/savanna/+milestone/icehouse-3 and say what's missed 18:18:41 <SergeyLukjanov> or what should be postponed so far 18:19:00 <SergeyLukjanov> jmaron, jspeidel, ping 18:19:06 <SergeyLukjanov> heh 18:19:25 <mattf> SergeyLukjanov, i'll review and make suggestions on #savanna later 18:19:32 <SergeyLukjanov> thx 18:19:46 <SergeyLukjanov> I'd like to ask all of us to check their blueprints statuses 18:20:09 <SergeyLukjanov> and if you're working on blueprint that isn't targeted to i3, please, ping me 18:20:35 <aignatov> hm, it seems already implemented https://blueprints.launchpad.net/savanna/+spec/epd-data-source-existing-hadoop-cluster 18:20:42 <SergeyLukjanov> we'll enable soft feature freeze after i3 18:21:21 <SergeyLukjanov> aignatov, agreed, both server side and dashboard side merged 18:21:35 <aignatov> sreshetnyak: here? 18:21:35 <SergeyLukjanov> need to confirm it with Trevor and Chad 18:21:56 <sreshetnyak> here 18:22:26 <aignatov> You was assigned on external hdfs. is it fully completed? 18:22:52 <crobertsrh> I believe it is completed 18:23:31 <sreshetnyak> need to add integration tests 18:23:34 <tmckay> I haven't used it for an external cluster, but I have run it with hdfs data sources on the same running cluster 18:24:01 <mattf> +1.5 18:24:25 <aignatov> I ran it on external hdfs with earliest patches of this commit ;) 18:24:27 <SergeyLukjanov> ok, I've mark it implemented 18:24:33 <aignatov> it worked 18:24:37 <SergeyLukjanov> any questions? 18:24:44 <aignatov> on my laptop ;-) 18:24:52 <mattf> heh 18:24:59 <mattf> +.5 18:25:45 <aignatov> matff, don't you trust my laptop? 18:25:55 <tmckay> I think all my currently approved blueprints will be implemented very soon, with maybe the exception of https://blueprints.launchpad.net/savanna/+spec/edp-oozie-files-and-archives 18:26:30 <tmckay> files and archives are not strictly necessary -- streaming mapreduce can use 'libs' to bundle files if a script needs to be uploaded 18:26:39 <tmckay> but they would be nice on a long-running cluster 18:27:20 <mattf> aignatov, just getting it to +2 18:27:34 <crobertsrh> I have a question regarding https://bugs.launchpad.net/savanna/+bug/1212354 It's a bug against the UI that happens when you empty out a parameter from the HDFS config tab for a cluster template (or probably most tabs). The client treats it as ok because the plugin returns a config set that sets those values as optional. Who would be the right person to talk to about this? 18:29:18 <crobertsrh> I should say that the dashboard treats it as ok ^^^ 18:29:59 <mattf> jmaron, welcome 18:30:20 <jmaron> sorry I'm late 18:30:33 <SergeyLukjanov> jmaron, hey, any updates? 18:30:39 <jmaron> lost track of time... 18:30:50 <mattf> all that time in the break room 18:30:51 <jmaron> putting finishing touched on HDP 2 integration 18:30:58 <jmaron> touches 18:31:08 <jmaron> should have something for review today or tomorrow 18:31:29 <jmaron> I'd like to commit an initial patch that addresses functionality 18:31:41 <aignatov> crobertsrh: I'll try to take a look on this bug tomorrow 18:31:43 <jmaron> with the understanding that there'll be a follow up commit with some needed refactorings 18:32:18 <mattf> jmaron, you know i'm a fan of phasing changes 18:32:20 <crobertsrh> aignatov: Thanks. Let me know what you think about it. I may be able to tweak the dashboard if necessary, but I think it would be rather ugly. 18:32:31 <jmaron> I think doing the refactorings at this time will make the patch unwieldy… 18:32:50 <jmaron> so no comments about duplicate code ;) 18:33:03 <mattf> fair enough 18:33:23 <aignatov> crobertsrh: agreed, when I'll figure out whats happening I'll comment to the bug or to you in channel :) 18:33:31 <crobertsrh> Sounds good 18:34:00 <mattf> bob_nettleton, btw, i'm eagerly awaiting your dib patches. are you planning to separate out the java & ssh elements at the same time? 18:35:41 <SergeyLukjanov> ok, let's move on 18:35:44 <bob_nettleton> mattf, not in the initial patch submission, but I do think it makes sense to break these out into common elements. My hope was to get the initial submission in, and then I could refactor the scripts once the common elements were available. 18:36:12 <mattf> bob_nettleton, should the refactoring bps be assigned to you? 18:37:06 <bob_nettleton> mattf, sure, that would be ok for now. if my task assignments change, we might need to re-assign, but I could certainly take a look at this. 18:37:21 <mattf> bob_nettleton, ok 18:37:22 <bob_nettleton> mattf, should the common elements go in DIB itself, or in savanna-image-elements? 18:37:49 <mattf> imho, we should try to get all our elements into dib itself. only truly general ones will make the cut. 18:37:51 <SergeyLukjanov> bob_nettleton, I think the first step is complete them in s-i-e 18:38:09 <mattf> so you can file a review against both. in the past i'ev filed against dib and when rejected filed against savanna 18:38:30 <bob_nettleton> SergeyLukjanov, mattf, ok, sounds fine. thanks! 18:38:46 <SergeyLukjanov> #topic Project naming collision 18:38:56 <SergeyLukjanov> #link https://etherpad.openstack.org/p/savanna-renaming 18:39:03 <SergeyLukjanov> there are some proposed options alreadt 18:39:17 <SergeyLukjanov> let's continue discussion of new name in this etherpad 18:39:27 <mattf> do we have a deadline? 18:39:44 <SergeyLukjanov> I'm thinking about setting up a voting to cut top XX options 18:39:56 <SergeyLukjanov> in 1-1.5 weeks 18:40:01 <aignatov> we need much more proposed names 18:40:06 <mattf> we do 18:40:18 <mattf> is openstack legal giving us a deadline? 18:40:21 <aignatov> top XX means 20? 18:40:25 <SergeyLukjanov> I'd like to select the new name around the i3 18:40:32 <SergeyLukjanov> aignatov, I hope 5 18:40:40 <alazarev> SergeyLukjanov: did you ask about Savannah? 18:40:41 <SergeyLukjanov> mattf, there is now hard deadline 18:40:59 <mattf> are there milestones set, such as needing to be fully transitioned by i ga? 18:41:03 <SergeyLukjanov> mattf, but if we'd like to graduate from the incubation :) 18:41:03 <dmitryme> Hey, by the way, are you going to present all the options on the voting, or we will filter out some of them before? 18:41:19 <mattf> SergeyLukjanov, you speak the magic words... 18:41:53 <mattf> so we need to fully rename by I. we need to get through legal w/ the new name before we rename, and we need to prune the list before giving it to legal? 18:41:53 <SergeyLukjanov> dmitryme, I'd like to have several iterations 18:42:10 <SergeyLukjanov> mattf, yup 18:42:18 <mattf> are there any estimates on how long renaming takes and how long legal needs? 18:42:36 <dmitryme> SergeyLukjanov: yes, seems like a good idea. So at the first iteration we will review all the options and select, say 20 of them, right? 18:42:47 <SergeyLukjanov> mattf, and it'll be the best option to have a new name around i3 == before the graduation review 18:42:55 <aignatov> dmitryme: I guess so 18:43:14 <SergeyLukjanov> mattf, <=1w for legals I hope, ~1-2w for renaming 18:43:24 <mattf> SergeyLukjanov, so i3 - legal time == our deadline 18:43:55 <SergeyLukjanov> mattf, yup, that's the best case 18:44:06 <mattf> i just started soliciting names at rht, so i'd like to go on the 1.5-2wk for vote end 18:44:32 <SergeyLukjanov> i3 == Mar 6 18:44:43 <SergeyLukjanov> so, we have 4 w 18:45:34 <mattf> is legal going to output a single name or a further pruned list? 18:46:15 <mattf> ...do we need a second round of name voting... 18:46:20 <aignatov> "noooo, please, please let's us leave our beautiful name Savanna..." - we will write to legals :) 18:46:42 <mattf> aignatov, i'll +1 that name 18:46:54 <SergeyLukjanov> mattf, legal could check a short list of names for us 18:47:40 <mattf> so 4w - 1w voting round 2 - 1.5w legal - 1w voting round 1 == we need to start voting in a week. ok. 18:48:28 <mattf> i retract my request for 2weeks, thx 18:48:55 <SergeyLukjanov> I'd like to start voting right after the next meeting 18:49:09 <SergeyLukjanov> probably, we'll need to discuss something 18:49:33 <aignatov> SergeyLukjanov: you mean first round of voting, right? 18:49:42 <SergeyLukjanov> and I'll setup up for a week, but if it'll collect enough votes, it could be ended earlier 18:49:44 <SergeyLukjanov> aignatov, yup 18:49:49 <aignatov> ok 18:50:32 <SergeyLukjanov> honestly, I don't see any good options now :( Savanna is too own... 18:50:39 <SergeyLukjanov> heh 18:50:52 <SergeyLukjanov> #topic General discussion 18:52:14 <SergeyLukjanov> any questions/ideas? 18:52:15 <mattf> aignatov, alazarev, tmckay, crobertsrh, any comments on the base v2 api? 18:52:26 <tmckay> I would like to reconsider job "cancel" in v2 api 18:52:28 <tmckay> The reason: 18:52:37 <alazarev> mattf: looks good for me 18:52:50 <tmckay> crobertsrh notes that job relaunch is implemented on the job executions tab 18:52:54 <SergeyLukjanov> mattf, probably I missed it, but do you have any general api changes section? 18:52:59 <mattf> tmckay, reconsider? nothing is decided on it atm, just not listed in base. 18:53:03 <crobertsrh> I tend to agree with tmckay. 18:53:03 <SergeyLukjanov> mattf, like don't include tenant-id to url? 18:53:04 <tmckay> so, the use case is cancel a job, tweak the data or configs, and relaunch 18:53:19 <tmckay> mattf, my mistake. I thought missing meant gone :) 18:53:20 <mattf> SergeyLukjanov, i focused on the endpoints 18:53:26 <SergeyLukjanov> mattf, ok 18:54:05 <mattf> tmckay, so it's a stop + modify + restart use case? 18:54:06 <aignatov> for long running jobs the 'cancel' operation is extremely needed 18:54:11 <tmckay> mattf, yes 18:54:21 <mattf> aignatov, do you see cancel == stop ? 18:54:55 <aignatov> yep, stop it running on cluster 18:55:21 <mattf> aignatov, stop cluster or stop job? 18:55:27 <aignatov> stop job 18:55:47 <mattf> as in, for a transient cluster. does the clsuter survive the stop + restart? 18:55:59 <tmckay> good question 18:56:14 <aignatov> transient custer is another case :) 18:56:16 <crobertsrh> I would say that transient cluster goes away 18:56:18 <SergeyLukjanov> it'll be killed in current code 18:56:43 <aignatov> cluster should be killed, yes 18:56:44 <mattf> if it's killed, the stop + edit + restart ~= delete + edit + launch new 18:57:22 <tmckay> except that delete wipes it from the database, so you have to start from scratch. Maybe a little more typing 18:57:53 <mattf> another way is "clone" a running job. so "clone w/ edit" then delete old. 18:58:24 <aignatov> I said about 'cancel' jobs mostly targeted to persistent clusters (not transient) 18:58:29 <mattf> i'm just tossing out ideas that don't require us to add another verb, but give us the same use case. 18:58:56 <mattf> fyi, clone can be done entirely client side 19:00:01 <mattf> may be out of time, -> #savana 19:00:04 <mattf> may be out of time, -> #savanna 19:00:08 <SergeyLukjanov> exactly 19:00:12 <SergeyLukjanov> #endmeeting