18:00:13 <SergeyLukjanov> #startmeeting sahara
18:00:13 <openstack> Meeting started Thu Apr 24 18:00:13 2014 UTC and is due to finish in 60 minutes.  The chair is SergeyLukjanov. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:00:14 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:00:16 <openstack> The meeting name has been set to 'sahara'
18:00:17 <SergeyLukjanov> hey sahara folks
18:00:28 <aignatov> howdy
18:00:50 <NikitaKonovalov> o/
18:00:54 <SergeyLukjanov> let's wait for a few mins for quorum
18:01:33 <crobertsrh> hello
18:01:40 <elmiko> hi
18:01:48 <bob_nettleton> Hi All!
18:02:04 <elmiko> sounds like a good day for bob ;)
18:02:23 <aignatov> elmiko: why? :)
18:02:31 <bob_nettleton> :)
18:02:43 <elmiko> i dunno, he was very emphatic with his greeting
18:03:43 <aignatov> I see :)
18:04:09 * SergeyLukjanov looking for the agenda link
18:04:13 <SergeyLukjanov> #link https://wiki.openstack.org/wiki/Meetings/SaharaAgenda
18:04:25 <SergeyLukjanov> traditionally, let's start with ...
18:04:27 <SergeyLukjanov> #topic News / updates
18:04:30 <SergeyLukjanov> folks, please
18:04:56 <SergeyLukjanov> #info Draft schedule for Sahara design summit published - http://junodesignsummit.sched.org/overview/type/sahara+%28ex-savanna%29
18:05:12 <bob_nettleton> working on HDP image fixes, considering adding resizing support for HDP CentOS images.
18:05:41 <crobertsrh> Dashboard merge is back to having some traction.  Hoping to see some reviews of these soon before I do the remainder of the panels.  #link https://review.openstack.org/#/q/status:open+project:openstack/horizon+branch:master+topic:bp/merge-sahara-dashboard,n,z
18:06:20 <SergeyLukjanov> #info sahara-dashboard -> horizon reviews https://review.openstack.org/#/q/status:open+project:openstack/horizon+branch:master+topic:bp/merge-sahara-dashboard,n,z
18:06:43 <alazarev> I’m preparing to “performance of hadoop on openstack” talk on summit
18:06:43 <aignatov> let’s all us go and vote +1 for Chad!!!
18:06:44 <SergeyLukjanov> crobertsrh, I've answered in ML
18:07:34 <crobertsrh> Yes.  Pretty much everyone said what I thought they would say (and what I agree with).  I'm not sure what else really made sense to cause such a stir.
18:08:22 <aignatov> I fixed one bug with broken novanetwork working when aoutassignment is turned off
18:08:26 <SergeyLukjanov> crobertsrh, ping if you'll need support on political questions ;)
18:09:14 <crobertsrh> Will do.  I think we should be ok for now.  Although, my patches are currently blocked on a bug that I discovered this morning (horizon bug, not sahara bug)...only blocked for tests passing though, code runs fine.
18:09:39 <SergeyLukjanov> crobertsrh, nice, have you prepared some doc re testing your changes?
18:09:54 <aignatov> and started implementing  some pig job for as a second option for demo at summit, also made several draft slides for this talk
18:09:55 <SergeyLukjanov> crobertsrh, you publish it on wiki for ex.
18:10:19 <crobertsrh> I added a few quick notes to the bp.  I can do something more grand if necessary.
18:10:35 <SergeyLukjanov> crobertsrh, ok, cool, thx
18:10:50 <aignatov> tmckay: I thrown away an idea to do sloccount on openstack sources and decided to make some log analyser :)
18:11:04 <aignatov> for openstack logs in tempest
18:11:26 <SergeyLukjanov> any other news/updates?
18:11:34 <tmckay> aignatov, okay, great.  We should talk about Summit soon
18:11:47 <SergeyLukjanov> #topic Action items from the last meeting
18:12:06 <SergeyLukjanov> #info [DONE] alazarev file blueprint re hadoop version upgrades, aggregate thoughts and details, link to the design summit session
18:12:13 <SergeyLukjanov> #link https://blueprints.launchpad.net/sahara/+spec/hadoop-upgrade
18:12:28 <SergeyLukjanov> #info [DONE] dmitryme to align roadmap/Icehouse with release
18:12:34 <SergeyLukjanov> #link https://wiki.openstack.org/wiki/Sahara/Roadmap
18:12:39 <SergeyLukjanov> thx guys
18:12:44 <aignatov> hey folks , did you see my mail about retrospective questions?
18:12:55 <alazarev> aignatov: yes
18:13:11 <SergeyLukjanov> #action aignatov prepare data for retrospective on summit
18:13:20 <aignatov> I’m waiting for your answers :)
18:13:23 <SergeyLukjanov> #topic Design summit
18:13:45 <SergeyLukjanov> #link http://junodesignsummit.sched.org/overview/type/sahara+%28ex-savanna%29
18:13:54 <aignatov> don’t hestiate, all results will be anonymous like it was done for 0.3https://wiki.openstack.org/wiki/Savanna/Release_0.3_Retrospective
18:13:57 <aignatov> https://wiki.openstack.org/wiki/Savanna/Release_0.3_Retrospective
18:13:59 <SergeyLukjanov> I've pushed the draft schedule for our design summit
18:14:26 <SergeyLukjanov> not all session have good description, I'm still working on it
18:14:29 <SergeyLukjanov> but topics / order is ready
18:14:44 <SergeyLukjanov> folks, please, take a look on it till the tomorrow noon
18:14:52 <SergeyLukjanov> and share comments / suggestions
18:15:04 <SergeyLukjanov> I'll cleanup it tomorrow and publish final schedule
18:15:45 <SergeyLukjanov> I expect tmckay and croberts to lead http://junodesignsummit.sched.org/event/be842178a085fe95b7665a653f8ab541
18:15:59 <SergeyLukjanov> aignatov - http://junodesignsummit.sched.org/event/dfa603324c0bbf29c2f09a77efb82d1d
18:16:04 <sreshetnyak> o/
18:16:05 <SergeyLukjanov> mattf - http://junodesignsummit.sched.org/event/a64f771cf28ed3ad637730db828668ff
18:16:21 <SergeyLukjanov> any premature objections? :)
18:17:09 <aignatov> Future of EDP should lead tmckay first then me :) he did huge efforts during icehouse
18:17:38 <SergeyLukjanov> aignatov, oops, I missed tmckay here, I'm absolutely agree with ot
18:18:13 <SergeyLukjanov> #info tmckay and croberts to lead "UX improvements" session
18:18:33 <SergeyLukjanov> #info tmckay and aignatov to lead "Future of EDP: plugins, SPI, Oozie" session
18:18:37 <tmckay> thanks :)  Now I need ideas
18:19:06 <SergeyLukjanov> #info mattf to lead "Next major REST API - v2" session
18:19:11 <aignatov> tmckay: we should conduct the ideas in the etherpads
18:19:15 * mattf waves
18:19:41 <crobertsrh> I'm happy to talk about UX stuff
18:19:55 <SergeyLukjanov> aignatov, tmckay, I'll add etherpads to sessions tomorrow
18:19:59 <SergeyLukjanov> mattf, o/
18:20:04 <tmckay> k
18:20:07 <SergeyLukjanov> for mattf:
18:20:09 <SergeyLukjanov> [22:19:06]  <SergeyLukjanov>	 #info mattf to lead "Next major REST API - v2" session
18:20:24 <aignatov> SergeyLukjanov: great, I’ll start documenting my ideas
18:20:56 <SergeyLukjanov> aignatov, don't lost the notebook
18:21:10 <SergeyLukjanov> lose*
18:21:24 <aignatov> SergeyLukjanov: wh00t do you mean?  :\
18:21:29 <mattf> no immediate objection from me
18:22:15 <SergeyLukjanov> fyi we'll have a pad for sahara for all 5 days of summit
18:22:20 <mattf> i still have to look over the summit sessions
18:22:33 <SergeyLukjanov> with ~10 chairs and whiteboard
18:22:53 <aignatov> with beer?
18:23:04 <crobertsrh> No cots for napping?
18:23:24 <mattf> alazarev, did your perf talk get in?
18:23:26 <aignatov> crobertsrh: lol
18:23:30 <crobertsrh> If we had cots, we wouldn't need hotel rooms
18:23:39 <alazarev> mattf: yes
18:23:41 <mattf> brave brave crobertsrh
18:24:00 <SergeyLukjanov> crobertsrh, unfortunately no cots ;(
18:24:13 <mattf> alazarev, ahh, under "savanna" i was searching for "sahara"
18:24:15 <mattf> congrats
18:24:26 <aignatov> SergeyLukjanov: and what’s about beer?
18:24:30 <alazarev> mattf: :)
18:24:40 <SergeyLukjanov> mattf, crobertsrh, tmckay, in which hotel will you live?
18:24:51 <mattf> apparently crobertsrh will be on a cot
18:24:52 <SergeyLukjanov> aignatov, only water and snacks afaik
18:25:00 <mattf> it's warm in atl in the summer
18:25:01 <tmckay> didn't actually make my reservatoin yet :)
18:25:11 <tmckay> so maybe I'll be camping
18:25:18 <mattf> "warm" being an understatement
18:25:23 <crobertsrh> I think I went with the Omni
18:25:24 <elmiko> tmckay: bring a cot ;)
18:25:36 <tmckay> camping wouldn't be so bad
18:25:37 <SergeyLukjanov> heh, it's too cheap to bring my cot with me through the Atlantic ;)
18:25:56 <tmckay> if all the rooms are gone close by, I'll rent a car and drive ;-)
18:25:56 <elmiko> better beer at the hotel anyway
18:26:08 <SergeyLukjanov> :)
18:26:24 <SergeyLukjanov> in Hong Kong there was a party each day after the summit
18:26:30 <SergeyLukjanov> with free beer/wine
18:26:34 <elmiko> nice
18:26:40 <SergeyLukjanov> and even snacks sometimes
18:27:20 <SergeyLukjanov> alazarev, mattf, aignatov, jspeidel, have you all received invination to the core reviewers party?
18:27:21 <alazarev> SergeyLukjanov: in Portland there were several parties each day, actually
18:27:44 <mattf> not sure, who's it from? i still have an email backlog atm.
18:27:47 <alazarev> SergeyLukjanov: one official and some unofficial
18:27:57 * mattf may have "accidentally" junked it if it looked generic
18:28:03 <alazarev> SergeyLukjanov: no invitation
18:28:23 <SergeyLukjanov> topic: "You're invited to Core Reviewer Party at OpenStack Juno Summit (May 14, 2014)"
18:28:23 <aignatov> I only recieved the … “OpenStack VIP Party: Atlanta Metro meets Block-Party Retro”
18:28:40 <SergeyLukjanov> I've received "You're invited to Core Reviewer Party at OpenStack Juno Summit (May 14, 2014)" at March 15
18:28:41 <aignatov> did anyone recieve the same?
18:28:46 <SergeyLukjanov> yup
18:29:02 <SergeyLukjanov> I'll contact organizers of core rev. party
18:29:04 <elmiko> i haven't received any party invites :(
18:29:11 <mattf> SergeyLukjanov, i don't see that one in my inbox
18:29:22 <SergeyLukjanov> elmiko, are you sure that you registered for summit? :)
18:29:31 <elmiko> SergeyLukjanov: i did
18:29:41 <SergeyLukjanov> elmiko, mostly all of the parties were open
18:30:10 <SergeyLukjanov> I hope that we'll have an ATC lounge this time too
18:30:16 <SergeyLukjanov> with unlimited red bull
18:31:02 <mattf> hong kong hangover -> develop a taste for red bull -> feel shame
18:31:27 <SergeyLukjanov> :)
18:31:51 <SergeyLukjanov> any other questions re design summit?
18:31:55 <SergeyLukjanov> or summit at all?
18:32:04 <mattf> you're taking feedback until tomorrow?
18:32:25 <SergeyLukjanov> #info our slots: thu afternoon + fri morning
18:33:02 <SergeyLukjanov> mattf, yup, I'd like to start conversations in etherpads asap
18:33:16 <SergeyLukjanov> anyway, we could change something on the go
18:33:45 <SergeyLukjanov> like "Oh, crap, we forgot to include **** to our design summit"
18:34:01 * mattf nods
18:34:32 <SergeyLukjanov> comparing to the last summit sessions and hot topics current schedule looks actual
18:34:52 <SergeyLukjanov> additionally we'll be able to plan several small "chats" in other time
18:35:06 <SergeyLukjanov> using our pod
18:35:14 <SergeyLukjanov> I hope that it'll work good
18:35:59 <SergeyLukjanov> next topics in meeting agenda are:
18:36:00 <SergeyLukjanov> When to start backward compat?
18:36:00 <SergeyLukjanov> Backward compat for Hadoop version
18:36:16 <SergeyLukjanov> any preferences what to discuss next?
18:36:38 <mattf> i'm skeptical about supporting multiple 2.x streams
18:36:57 <SergeyLukjanov> #topic Backward compat for Hadoop version
18:37:04 <mattf> and i'm curious what folks have in mind for "upgrading" clusters
18:37:14 <SergeyLukjanov> folks, please, share your thoughts
18:37:32 <mattf> i didn't get a chance to read last weeks notes, so if it is there tell me to go rtfn first
18:37:34 <alazarev> I think we should keep all reseased versions
18:37:38 <SergeyLukjanov> alazarev, you've prepared the bp, so, you should share your thoughts ;)
18:37:40 <aignatov> mattf: but typical use case is when user has 2.3.0 cluster for examle
18:37:45 <alazarev> and remove them after deprication period
18:37:46 <aignatov> long-living cluster
18:37:56 <aignatov> and then he needs to scale it
18:38:17 <alazarev> no upgrade of existing cluster
18:38:25 <mattf> scaling and edp are the two places where we touch the inside of a cluster and may break compatibility?
18:38:26 <alazarev> only data migration
18:38:26 <aignatov> but suddenly he has sahara with new hadoop version
18:39:24 <alazarev> and for now, add 2.4 without removing 2.3
18:39:39 <NikitaKonovalov> agree to alazarev, the only problem is to move data to a cluster with a newer version
18:40:14 <alazarev> NikitaKonovalov: EDP supports external HDFS
18:40:16 <mattf> would you agree that supporting compat w/ a running cluster is different than building images for previous versions of hadoop?
18:40:32 <alazarev> NikitaKonovalov: don’t see problems in data migration
18:41:21 <NikitaKonovalov> alazarev: it does, but the HDFS itself may change it's own api
18:41:27 <aignatov> mattf: agree, actually users can build images with any hadoop versions atm and work with it like with 2.3.0
18:41:37 <mattf> data migration is an upgrade issue when someone wants to go from a 2.3 cluster to a 2.4 cluster, which is different from sahara going from 2014.1 to 2014.2, imho
18:41:55 <mattf> data migration seems like it is outside our scope atm
18:42:03 <alazarev> NikitaKonovalov: HDFS is API, it will be backward compatibility
18:42:10 <SergeyLukjanov> my position: don't remove prev. plugin's version while adding new ones (we released 2.3 in I, so, it should be J, but could be removed in K), add docs about data migration
18:42:15 <mattf> solved by standard migration procedures.... distcp from old to new....
18:42:16 <NikitaKonovalov> alazarev: I hope so
18:42:47 <alazarev> mattf: agree
18:42:54 <SergeyLukjanov> IMO we shouldn't implement migration, only add docs about possible data migration approaches
18:42:54 <mattf> for long running clusters the migration plan should be whatever the plugin vendor recommends. imho.
18:43:01 <SergeyLukjanov> mattf, agree re scope
18:43:02 <alazarev> mattf: we can document this process, to make things clea
18:43:04 <mattf> hopefully they're all pretty much the same
18:43:55 <aignatov> data can be migrated to swift and back, anyway, agree to only document possible migrations procedurs
18:44:00 * SergeyLukjanov booking 10 mins for Open Discussion
18:44:16 <mattf> as for the scaling & edp compatibility, our exposure should be limited to the tools we chose for scaling and edp (standard shell scripts & oozie). hopefully that's a low risk, or we chose poorly
18:45:09 <mattf> we should have a test procedure to detect if we're breaking compat between sahara releases, not necessarily w/i
18:45:39 <mattf> so sahara 2014.1 has 2.3 and 2014.2 has 2.4, if 2014.2 can't operate on 2.3 we add compat code to 2014.2
18:45:51 <mattf> that's generally where my head is
18:46:21 <mattf> i'm rather against the idea of supporting image building for old versions. you either have those registered somewhere or you use the new version.
18:46:36 * mattf pauses to breath
18:47:11 <aignatov> mattf: in general I agree with you
18:47:53 <mattf> also, if hadoop breaks mgmt interfaces between 2.3 and 2.4 we yell at them
18:48:20 <mattf> (and we probably won't be the only group yelling)
18:49:02 <mattf> bob_nettleton, do you have any comments about the stability of hadoop w/i the 2.x series or the stability of the oozie interface we're using?
18:49:23 <aignatov> also I think we will not need in compat code with hadoop vesions because diffrences between  minor versions like 2.3, 2.4 is changes in configurations files
18:49:46 <aignatov> only these files coud affect sahara’s hadoop confgiuring
18:50:04 <bob_nettleton> mattf, not sure if I have a specific comment regarding the 2.x interfaces.  I'm trying to understand why we'd drop support for older versions in the images though.  Is this to simplify the requirements on the plugin?
18:50:06 <aignatov> but scaling and edp will work I believe
18:50:12 <mattf> agree, it should be low risk. it's good to have an ide aof what we'll do if hadoop folks mess up.
18:50:32 <alazarev> aignatov: I agree that code can be compacted, but this is technical questions, not political
18:50:52 <mattf> bob_nettleton, we should be validating all the images we produce w/ diskimage-create
18:50:58 <SergeyLukjanov> sounds like we have a direction
18:51:11 <mattf> bob_nettleton, imho we should need a really good reason to double the liability there
18:51:30 <SergeyLukjanov> 8 mins left
18:51:34 <mattf> bob_nettleton, would you like to support the plugin w/ hdp 2.0 and 2.1 as separate?
18:51:37 <SergeyLukjanov> switching topic
18:51:38 <SergeyLukjanov> #topic Open discussion
18:52:08 <SergeyLukjanov> the next meeting day in May 1, it's a non-working day in Russia
18:52:18 <SergeyLukjanov> and I'm starting traveling May 2
18:52:32 <mattf> that's a looooong flight
18:52:36 <tmckay> heh
18:52:36 <SergeyLukjanov> so, I'm proposing to move it to cancel it
18:52:53 <bob_nettleton> mattf, I guess it depends.  I understand your concern regarding a growing testing gap, with respect to how many images we generate.  I do worry about users not being able to adopt Sahara due to a requirement to use a specific Hadoop version though.
18:52:55 <SergeyLukjanov> I'll be in CA for a week before ATL
18:53:01 <mattf> could take it to os-dev -- if there are topics of interest
18:53:03 <SergeyLukjanov> aignatov too
18:53:11 <mattf> so you guys won't need the red bull
18:53:11 <aignatov> me too, actually I’ll start travelling on Monday
18:53:16 <bob_nettleton> mattf, I certainly agree that we can't support all versions of Hadoop as time goes on, just not sure where the line should be drawn.
18:53:33 <SergeyLukjanov> I'll write about canceling the meeting to ML, good idea mattf
18:53:36 <tmckay> aignatov, this Monday?
18:53:49 <SergeyLukjanov> we could move it to anouther day too
18:54:05 <SergeyLukjanov> bob_nettleton ++
18:54:06 <aignatov> before summit I’ll bring Sahara to masses at Moscow confernce about BigData OpenStack
18:54:06 <mattf> bob_nettleton, there's probably a line we should draw, but it should be between upstream and downstream providers. if downstream has longer maint requirements that's more a downstream issue than an upstream.
18:54:16 <aignatov> tmckay: yes
18:54:27 <tmckay> ah, I see.  another conference
18:54:32 <SergeyLukjanov> oh, one more note - I'll be on vacation for a week after the summit
18:54:57 <mattf> SergeyLukjanov, well earned
18:55:50 <aignatov> guys, plese dont forget about retrospective :)
18:56:00 <SergeyLukjanov> anyone living in CA would like to have a beer before summit?
18:56:06 <SergeyLukjanov> or milk
18:56:26 <mattf> milk stouts are good
18:56:36 * mattf is boston (right coast)
18:56:39 <bob_nettleton> mattf, just seems to me that even dot releases in Hadoop these days include such large amounts of change that it might be difficult to settle on just one 2.x version, for example.
18:56:43 <aignatov> it’ll be helpful to discuss what should be improved in our community relating to your feadbacks
18:57:16 <SergeyLukjanov> yup, folks, please, don't forget about retrospective, it could be very usefull
18:57:20 <mattf> bob_nettleton, imho sahara should adopt tip of 2.x and we probably support previous-tip from our last release
18:57:46 <mattf> aignatov, did we do much w/ the last one?
18:58:00 <mattf> (6 mo is too long ago to remember)
18:58:31 <SergeyLukjanov> 1 min left
18:58:42 <aignatov> mattf: I think yes
18:58:51 * mattf smiles
18:59:13 <mattf> continue on #openstack-sahara?
18:59:14 <bob_nettleton> mattf, I think I'd need to understand the additional burdens on the plugin providers to support multiple 2.x versions, before I have a stronger opinion on this one.
18:59:25 <aignatov> anyway, I’m waiting for your responses
18:59:31 <SergeyLukjanov> mattf, we do, there was comments re review, ci, etc.
18:59:32 <mattf> bob_nettleton, i'm hoping for fewer versions to support, fyi
18:59:43 <SergeyLukjanov> yup, let's move to our home channel
18:59:45 <mattf> bob_nettleton, at least upstream...
18:59:46 <SergeyLukjanov> #endmeeting