18:00:13 <SergeyLukjanov> #startmeeting sahara 18:00:13 <openstack> Meeting started Thu Apr 24 18:00:13 2014 UTC and is due to finish in 60 minutes. The chair is SergeyLukjanov. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:14 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:00:16 <openstack> The meeting name has been set to 'sahara' 18:00:17 <SergeyLukjanov> hey sahara folks 18:00:28 <aignatov> howdy 18:00:50 <NikitaKonovalov> o/ 18:00:54 <SergeyLukjanov> let's wait for a few mins for quorum 18:01:33 <crobertsrh> hello 18:01:40 <elmiko> hi 18:01:48 <bob_nettleton> Hi All! 18:02:04 <elmiko> sounds like a good day for bob ;) 18:02:23 <aignatov> elmiko: why? :) 18:02:31 <bob_nettleton> :) 18:02:43 <elmiko> i dunno, he was very emphatic with his greeting 18:03:43 <aignatov> I see :) 18:04:09 * SergeyLukjanov looking for the agenda link 18:04:13 <SergeyLukjanov> #link https://wiki.openstack.org/wiki/Meetings/SaharaAgenda 18:04:25 <SergeyLukjanov> traditionally, let's start with ... 18:04:27 <SergeyLukjanov> #topic News / updates 18:04:30 <SergeyLukjanov> folks, please 18:04:56 <SergeyLukjanov> #info Draft schedule for Sahara design summit published - http://junodesignsummit.sched.org/overview/type/sahara+%28ex-savanna%29 18:05:12 <bob_nettleton> working on HDP image fixes, considering adding resizing support for HDP CentOS images. 18:05:41 <crobertsrh> Dashboard merge is back to having some traction. Hoping to see some reviews of these soon before I do the remainder of the panels. #link https://review.openstack.org/#/q/status:open+project:openstack/horizon+branch:master+topic:bp/merge-sahara-dashboard,n,z 18:06:20 <SergeyLukjanov> #info sahara-dashboard -> horizon reviews https://review.openstack.org/#/q/status:open+project:openstack/horizon+branch:master+topic:bp/merge-sahara-dashboard,n,z 18:06:43 <alazarev> I’m preparing to “performance of hadoop on openstack” talk on summit 18:06:43 <aignatov> let’s all us go and vote +1 for Chad!!! 18:06:44 <SergeyLukjanov> crobertsrh, I've answered in ML 18:07:34 <crobertsrh> Yes. Pretty much everyone said what I thought they would say (and what I agree with). I'm not sure what else really made sense to cause such a stir. 18:08:22 <aignatov> I fixed one bug with broken novanetwork working when aoutassignment is turned off 18:08:26 <SergeyLukjanov> crobertsrh, ping if you'll need support on political questions ;) 18:09:14 <crobertsrh> Will do. I think we should be ok for now. Although, my patches are currently blocked on a bug that I discovered this morning (horizon bug, not sahara bug)...only blocked for tests passing though, code runs fine. 18:09:39 <SergeyLukjanov> crobertsrh, nice, have you prepared some doc re testing your changes? 18:09:54 <aignatov> and started implementing some pig job for as a second option for demo at summit, also made several draft slides for this talk 18:09:55 <SergeyLukjanov> crobertsrh, you publish it on wiki for ex. 18:10:19 <crobertsrh> I added a few quick notes to the bp. I can do something more grand if necessary. 18:10:35 <SergeyLukjanov> crobertsrh, ok, cool, thx 18:10:50 <aignatov> tmckay: I thrown away an idea to do sloccount on openstack sources and decided to make some log analyser :) 18:11:04 <aignatov> for openstack logs in tempest 18:11:26 <SergeyLukjanov> any other news/updates? 18:11:34 <tmckay> aignatov, okay, great. We should talk about Summit soon 18:11:47 <SergeyLukjanov> #topic Action items from the last meeting 18:12:06 <SergeyLukjanov> #info [DONE] alazarev file blueprint re hadoop version upgrades, aggregate thoughts and details, link to the design summit session 18:12:13 <SergeyLukjanov> #link https://blueprints.launchpad.net/sahara/+spec/hadoop-upgrade 18:12:28 <SergeyLukjanov> #info [DONE] dmitryme to align roadmap/Icehouse with release 18:12:34 <SergeyLukjanov> #link https://wiki.openstack.org/wiki/Sahara/Roadmap 18:12:39 <SergeyLukjanov> thx guys 18:12:44 <aignatov> hey folks , did you see my mail about retrospective questions? 18:12:55 <alazarev> aignatov: yes 18:13:11 <SergeyLukjanov> #action aignatov prepare data for retrospective on summit 18:13:20 <aignatov> I’m waiting for your answers :) 18:13:23 <SergeyLukjanov> #topic Design summit 18:13:45 <SergeyLukjanov> #link http://junodesignsummit.sched.org/overview/type/sahara+%28ex-savanna%29 18:13:54 <aignatov> don’t hestiate, all results will be anonymous like it was done for 0.3https://wiki.openstack.org/wiki/Savanna/Release_0.3_Retrospective 18:13:57 <aignatov> https://wiki.openstack.org/wiki/Savanna/Release_0.3_Retrospective 18:13:59 <SergeyLukjanov> I've pushed the draft schedule for our design summit 18:14:26 <SergeyLukjanov> not all session have good description, I'm still working on it 18:14:29 <SergeyLukjanov> but topics / order is ready 18:14:44 <SergeyLukjanov> folks, please, take a look on it till the tomorrow noon 18:14:52 <SergeyLukjanov> and share comments / suggestions 18:15:04 <SergeyLukjanov> I'll cleanup it tomorrow and publish final schedule 18:15:45 <SergeyLukjanov> I expect tmckay and croberts to lead http://junodesignsummit.sched.org/event/be842178a085fe95b7665a653f8ab541 18:15:59 <SergeyLukjanov> aignatov - http://junodesignsummit.sched.org/event/dfa603324c0bbf29c2f09a77efb82d1d 18:16:04 <sreshetnyak> o/ 18:16:05 <SergeyLukjanov> mattf - http://junodesignsummit.sched.org/event/a64f771cf28ed3ad637730db828668ff 18:16:21 <SergeyLukjanov> any premature objections? :) 18:17:09 <aignatov> Future of EDP should lead tmckay first then me :) he did huge efforts during icehouse 18:17:38 <SergeyLukjanov> aignatov, oops, I missed tmckay here, I'm absolutely agree with ot 18:18:13 <SergeyLukjanov> #info tmckay and croberts to lead "UX improvements" session 18:18:33 <SergeyLukjanov> #info tmckay and aignatov to lead "Future of EDP: plugins, SPI, Oozie" session 18:18:37 <tmckay> thanks :) Now I need ideas 18:19:06 <SergeyLukjanov> #info mattf to lead "Next major REST API - v2" session 18:19:11 <aignatov> tmckay: we should conduct the ideas in the etherpads 18:19:15 * mattf waves 18:19:41 <crobertsrh> I'm happy to talk about UX stuff 18:19:55 <SergeyLukjanov> aignatov, tmckay, I'll add etherpads to sessions tomorrow 18:19:59 <SergeyLukjanov> mattf, o/ 18:20:04 <tmckay> k 18:20:07 <SergeyLukjanov> for mattf: 18:20:09 <SergeyLukjanov> [22:19:06] <SergeyLukjanov> #info mattf to lead "Next major REST API - v2" session 18:20:24 <aignatov> SergeyLukjanov: great, I’ll start documenting my ideas 18:20:56 <SergeyLukjanov> aignatov, don't lost the notebook 18:21:10 <SergeyLukjanov> lose* 18:21:24 <aignatov> SergeyLukjanov: wh00t do you mean? :\ 18:21:29 <mattf> no immediate objection from me 18:22:15 <SergeyLukjanov> fyi we'll have a pad for sahara for all 5 days of summit 18:22:20 <mattf> i still have to look over the summit sessions 18:22:33 <SergeyLukjanov> with ~10 chairs and whiteboard 18:22:53 <aignatov> with beer? 18:23:04 <crobertsrh> No cots for napping? 18:23:24 <mattf> alazarev, did your perf talk get in? 18:23:26 <aignatov> crobertsrh: lol 18:23:30 <crobertsrh> If we had cots, we wouldn't need hotel rooms 18:23:39 <alazarev> mattf: yes 18:23:41 <mattf> brave brave crobertsrh 18:24:00 <SergeyLukjanov> crobertsrh, unfortunately no cots ;( 18:24:13 <mattf> alazarev, ahh, under "savanna" i was searching for "sahara" 18:24:15 <mattf> congrats 18:24:26 <aignatov> SergeyLukjanov: and what’s about beer? 18:24:30 <alazarev> mattf: :) 18:24:40 <SergeyLukjanov> mattf, crobertsrh, tmckay, in which hotel will you live? 18:24:51 <mattf> apparently crobertsrh will be on a cot 18:24:52 <SergeyLukjanov> aignatov, only water and snacks afaik 18:25:00 <mattf> it's warm in atl in the summer 18:25:01 <tmckay> didn't actually make my reservatoin yet :) 18:25:11 <tmckay> so maybe I'll be camping 18:25:18 <mattf> "warm" being an understatement 18:25:23 <crobertsrh> I think I went with the Omni 18:25:24 <elmiko> tmckay: bring a cot ;) 18:25:36 <tmckay> camping wouldn't be so bad 18:25:37 <SergeyLukjanov> heh, it's too cheap to bring my cot with me through the Atlantic ;) 18:25:56 <tmckay> if all the rooms are gone close by, I'll rent a car and drive ;-) 18:25:56 <elmiko> better beer at the hotel anyway 18:26:08 <SergeyLukjanov> :) 18:26:24 <SergeyLukjanov> in Hong Kong there was a party each day after the summit 18:26:30 <SergeyLukjanov> with free beer/wine 18:26:34 <elmiko> nice 18:26:40 <SergeyLukjanov> and even snacks sometimes 18:27:20 <SergeyLukjanov> alazarev, mattf, aignatov, jspeidel, have you all received invination to the core reviewers party? 18:27:21 <alazarev> SergeyLukjanov: in Portland there were several parties each day, actually 18:27:44 <mattf> not sure, who's it from? i still have an email backlog atm. 18:27:47 <alazarev> SergeyLukjanov: one official and some unofficial 18:27:57 * mattf may have "accidentally" junked it if it looked generic 18:28:03 <alazarev> SergeyLukjanov: no invitation 18:28:23 <SergeyLukjanov> topic: "You're invited to Core Reviewer Party at OpenStack Juno Summit (May 14, 2014)" 18:28:23 <aignatov> I only recieved the … “OpenStack VIP Party: Atlanta Metro meets Block-Party Retro” 18:28:40 <SergeyLukjanov> I've received "You're invited to Core Reviewer Party at OpenStack Juno Summit (May 14, 2014)" at March 15 18:28:41 <aignatov> did anyone recieve the same? 18:28:46 <SergeyLukjanov> yup 18:29:02 <SergeyLukjanov> I'll contact organizers of core rev. party 18:29:04 <elmiko> i haven't received any party invites :( 18:29:11 <mattf> SergeyLukjanov, i don't see that one in my inbox 18:29:22 <SergeyLukjanov> elmiko, are you sure that you registered for summit? :) 18:29:31 <elmiko> SergeyLukjanov: i did 18:29:41 <SergeyLukjanov> elmiko, mostly all of the parties were open 18:30:10 <SergeyLukjanov> I hope that we'll have an ATC lounge this time too 18:30:16 <SergeyLukjanov> with unlimited red bull 18:31:02 <mattf> hong kong hangover -> develop a taste for red bull -> feel shame 18:31:27 <SergeyLukjanov> :) 18:31:51 <SergeyLukjanov> any other questions re design summit? 18:31:55 <SergeyLukjanov> or summit at all? 18:32:04 <mattf> you're taking feedback until tomorrow? 18:32:25 <SergeyLukjanov> #info our slots: thu afternoon + fri morning 18:33:02 <SergeyLukjanov> mattf, yup, I'd like to start conversations in etherpads asap 18:33:16 <SergeyLukjanov> anyway, we could change something on the go 18:33:45 <SergeyLukjanov> like "Oh, crap, we forgot to include **** to our design summit" 18:34:01 * mattf nods 18:34:32 <SergeyLukjanov> comparing to the last summit sessions and hot topics current schedule looks actual 18:34:52 <SergeyLukjanov> additionally we'll be able to plan several small "chats" in other time 18:35:06 <SergeyLukjanov> using our pod 18:35:14 <SergeyLukjanov> I hope that it'll work good 18:35:59 <SergeyLukjanov> next topics in meeting agenda are: 18:36:00 <SergeyLukjanov> When to start backward compat? 18:36:00 <SergeyLukjanov> Backward compat for Hadoop version 18:36:16 <SergeyLukjanov> any preferences what to discuss next? 18:36:38 <mattf> i'm skeptical about supporting multiple 2.x streams 18:36:57 <SergeyLukjanov> #topic Backward compat for Hadoop version 18:37:04 <mattf> and i'm curious what folks have in mind for "upgrading" clusters 18:37:14 <SergeyLukjanov> folks, please, share your thoughts 18:37:32 <mattf> i didn't get a chance to read last weeks notes, so if it is there tell me to go rtfn first 18:37:34 <alazarev> I think we should keep all reseased versions 18:37:38 <SergeyLukjanov> alazarev, you've prepared the bp, so, you should share your thoughts ;) 18:37:40 <aignatov> mattf: but typical use case is when user has 2.3.0 cluster for examle 18:37:45 <alazarev> and remove them after deprication period 18:37:46 <aignatov> long-living cluster 18:37:56 <aignatov> and then he needs to scale it 18:38:17 <alazarev> no upgrade of existing cluster 18:38:25 <mattf> scaling and edp are the two places where we touch the inside of a cluster and may break compatibility? 18:38:26 <alazarev> only data migration 18:38:26 <aignatov> but suddenly he has sahara with new hadoop version 18:39:24 <alazarev> and for now, add 2.4 without removing 2.3 18:39:39 <NikitaKonovalov> agree to alazarev, the only problem is to move data to a cluster with a newer version 18:40:14 <alazarev> NikitaKonovalov: EDP supports external HDFS 18:40:16 <mattf> would you agree that supporting compat w/ a running cluster is different than building images for previous versions of hadoop? 18:40:32 <alazarev> NikitaKonovalov: don’t see problems in data migration 18:41:21 <NikitaKonovalov> alazarev: it does, but the HDFS itself may change it's own api 18:41:27 <aignatov> mattf: agree, actually users can build images with any hadoop versions atm and work with it like with 2.3.0 18:41:37 <mattf> data migration is an upgrade issue when someone wants to go from a 2.3 cluster to a 2.4 cluster, which is different from sahara going from 2014.1 to 2014.2, imho 18:41:55 <mattf> data migration seems like it is outside our scope atm 18:42:03 <alazarev> NikitaKonovalov: HDFS is API, it will be backward compatibility 18:42:10 <SergeyLukjanov> my position: don't remove prev. plugin's version while adding new ones (we released 2.3 in I, so, it should be J, but could be removed in K), add docs about data migration 18:42:15 <mattf> solved by standard migration procedures.... distcp from old to new.... 18:42:16 <NikitaKonovalov> alazarev: I hope so 18:42:47 <alazarev> mattf: agree 18:42:54 <SergeyLukjanov> IMO we shouldn't implement migration, only add docs about possible data migration approaches 18:42:54 <mattf> for long running clusters the migration plan should be whatever the plugin vendor recommends. imho. 18:43:01 <SergeyLukjanov> mattf, agree re scope 18:43:02 <alazarev> mattf: we can document this process, to make things clea 18:43:04 <mattf> hopefully they're all pretty much the same 18:43:55 <aignatov> data can be migrated to swift and back, anyway, agree to only document possible migrations procedurs 18:44:00 * SergeyLukjanov booking 10 mins for Open Discussion 18:44:16 <mattf> as for the scaling & edp compatibility, our exposure should be limited to the tools we chose for scaling and edp (standard shell scripts & oozie). hopefully that's a low risk, or we chose poorly 18:45:09 <mattf> we should have a test procedure to detect if we're breaking compat between sahara releases, not necessarily w/i 18:45:39 <mattf> so sahara 2014.1 has 2.3 and 2014.2 has 2.4, if 2014.2 can't operate on 2.3 we add compat code to 2014.2 18:45:51 <mattf> that's generally where my head is 18:46:21 <mattf> i'm rather against the idea of supporting image building for old versions. you either have those registered somewhere or you use the new version. 18:46:36 * mattf pauses to breath 18:47:11 <aignatov> mattf: in general I agree with you 18:47:53 <mattf> also, if hadoop breaks mgmt interfaces between 2.3 and 2.4 we yell at them 18:48:20 <mattf> (and we probably won't be the only group yelling) 18:49:02 <mattf> bob_nettleton, do you have any comments about the stability of hadoop w/i the 2.x series or the stability of the oozie interface we're using? 18:49:23 <aignatov> also I think we will not need in compat code with hadoop vesions because diffrences between minor versions like 2.3, 2.4 is changes in configurations files 18:49:46 <aignatov> only these files coud affect sahara’s hadoop confgiuring 18:50:04 <bob_nettleton> mattf, not sure if I have a specific comment regarding the 2.x interfaces. I'm trying to understand why we'd drop support for older versions in the images though. Is this to simplify the requirements on the plugin? 18:50:06 <aignatov> but scaling and edp will work I believe 18:50:12 <mattf> agree, it should be low risk. it's good to have an ide aof what we'll do if hadoop folks mess up. 18:50:32 <alazarev> aignatov: I agree that code can be compacted, but this is technical questions, not political 18:50:52 <mattf> bob_nettleton, we should be validating all the images we produce w/ diskimage-create 18:50:58 <SergeyLukjanov> sounds like we have a direction 18:51:11 <mattf> bob_nettleton, imho we should need a really good reason to double the liability there 18:51:30 <SergeyLukjanov> 8 mins left 18:51:34 <mattf> bob_nettleton, would you like to support the plugin w/ hdp 2.0 and 2.1 as separate? 18:51:37 <SergeyLukjanov> switching topic 18:51:38 <SergeyLukjanov> #topic Open discussion 18:52:08 <SergeyLukjanov> the next meeting day in May 1, it's a non-working day in Russia 18:52:18 <SergeyLukjanov> and I'm starting traveling May 2 18:52:32 <mattf> that's a looooong flight 18:52:36 <tmckay> heh 18:52:36 <SergeyLukjanov> so, I'm proposing to move it to cancel it 18:52:53 <bob_nettleton> mattf, I guess it depends. I understand your concern regarding a growing testing gap, with respect to how many images we generate. I do worry about users not being able to adopt Sahara due to a requirement to use a specific Hadoop version though. 18:52:55 <SergeyLukjanov> I'll be in CA for a week before ATL 18:53:01 <mattf> could take it to os-dev -- if there are topics of interest 18:53:03 <SergeyLukjanov> aignatov too 18:53:11 <mattf> so you guys won't need the red bull 18:53:11 <aignatov> me too, actually I’ll start travelling on Monday 18:53:16 <bob_nettleton> mattf, I certainly agree that we can't support all versions of Hadoop as time goes on, just not sure where the line should be drawn. 18:53:33 <SergeyLukjanov> I'll write about canceling the meeting to ML, good idea mattf 18:53:36 <tmckay> aignatov, this Monday? 18:53:49 <SergeyLukjanov> we could move it to anouther day too 18:54:05 <SergeyLukjanov> bob_nettleton ++ 18:54:06 <aignatov> before summit I’ll bring Sahara to masses at Moscow confernce about BigData OpenStack 18:54:06 <mattf> bob_nettleton, there's probably a line we should draw, but it should be between upstream and downstream providers. if downstream has longer maint requirements that's more a downstream issue than an upstream. 18:54:16 <aignatov> tmckay: yes 18:54:27 <tmckay> ah, I see. another conference 18:54:32 <SergeyLukjanov> oh, one more note - I'll be on vacation for a week after the summit 18:54:57 <mattf> SergeyLukjanov, well earned 18:55:50 <aignatov> guys, plese dont forget about retrospective :) 18:56:00 <SergeyLukjanov> anyone living in CA would like to have a beer before summit? 18:56:06 <SergeyLukjanov> or milk 18:56:26 <mattf> milk stouts are good 18:56:36 * mattf is boston (right coast) 18:56:39 <bob_nettleton> mattf, just seems to me that even dot releases in Hadoop these days include such large amounts of change that it might be difficult to settle on just one 2.x version, for example. 18:56:43 <aignatov> it’ll be helpful to discuss what should be improved in our community relating to your feadbacks 18:57:16 <SergeyLukjanov> yup, folks, please, don't forget about retrospective, it could be very usefull 18:57:20 <mattf> bob_nettleton, imho sahara should adopt tip of 2.x and we probably support previous-tip from our last release 18:57:46 <mattf> aignatov, did we do much w/ the last one? 18:58:00 <mattf> (6 mo is too long ago to remember) 18:58:31 <SergeyLukjanov> 1 min left 18:58:42 <aignatov> mattf: I think yes 18:58:51 * mattf smiles 18:59:13 <mattf> continue on #openstack-sahara? 18:59:14 <bob_nettleton> mattf, I think I'd need to understand the additional burdens on the plugin providers to support multiple 2.x versions, before I have a stronger opinion on this one. 18:59:25 <aignatov> anyway, I’m waiting for your responses 18:59:31 <SergeyLukjanov> mattf, we do, there was comments re review, ci, etc. 18:59:32 <mattf> bob_nettleton, i'm hoping for fewer versions to support, fyi 18:59:43 <SergeyLukjanov> yup, let's move to our home channel 18:59:45 <mattf> bob_nettleton, at least upstream... 18:59:46 <SergeyLukjanov> #endmeeting