18:03:51 <SergeyLukjanov> #startmeeting sahara
18:03:51 <notmyname> heh, ok :-)
18:03:51 <openstack> Meeting started Thu Jul 10 18:03:51 2014 UTC and is due to finish in 60 minutes.  The chair is SergeyLukjanov. Information about MeetBot at http://wiki.debian.org/MeetBot.
18:03:52 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
18:03:55 <openstack> The meeting name has been set to 'sahara'
18:04:13 <SergeyLukjanov> notmyname, btw you are fast ;)
18:04:23 <SergeyLukjanov> #link https://wiki.openstack.org/wiki/Meetings/SaharaAgenda#Next_meetings
18:04:32 <SergeyLukjanov> #topic News / updates
18:04:35 <SergeyLukjanov> folks, please
18:04:41 <notmyname> SergeyLukjanov: I'm "swift"? ;-)
18:04:53 <SergeyLukjanov> notmyname :)
18:05:07 <SergeyLukjanov> mattf, hey
18:05:09 <alazarev> uppercase go before lowercase :)
18:05:29 <mattf> SergeyLukjanov, hey
18:05:37 <dmitryme> I did minor changes + review
18:05:38 * mattf is happy to have found his way back to irc
18:05:54 <SergeyLukjanov> mattf, we're happy to see you again too ;)
18:06:05 <SergeyLukjanov> mattf, and it's time to sync oslo
18:06:06 * mattf smiles
18:06:08 <SergeyLukjanov> (no kidding)
18:06:12 <mattf> overdue actually!
18:06:21 <SergeyLukjanov> mattf, yeah
18:06:23 <elmiko> i've been investigating the code base in preparation for the swift/auth bp. starting to make mock ups of the keystone trust delegation and swift client interactions. also getting some reviews in.
18:06:28 <mattf> i'll put it in my evening catchup queue
18:06:34 <SergeyLukjanov> mattf, awesome, thx
18:06:40 <mattf> give me an action for the meeting so i'm definitely on the hook
18:06:49 <SergeyLukjanov> #action mattf to sync oslo
18:07:15 <SergeyLukjanov> crobertsrh, we have a separated topic for you re dashboard2horizon ;)
18:07:26 <crobertsrh> Yes, I feel so special
18:07:35 <mattf> 2horizon or not 2horizon
18:07:58 <SergeyLukjanov> the only new thing from me is that I've pushed patches to make sahara using i18n
18:08:06 <SergeyLukjanov> mattf, :)
18:08:20 <alazarev> I’ve fixed some heat engine staff
18:08:20 <SergeyLukjanov> I mean oslo.i18n
18:08:24 <tmckay> working on spark EDP via ssh and spark-submit
18:08:41 <tmckay> I have some questions about plugin configs for general topics ...
18:08:49 <alazarev> and pushed error handing refactoring
18:09:25 <SergeyLukjanov> tmckay, yup, I've seen your questions
18:09:42 <SergeyLukjanov> tmckay, let's discuss it before the open discussion
18:09:51 <crobertsrh> I've been putting together a list of dashboard to-do items and trying to think about whether or not they will fit for Juno.  #link https://etherpad.openstack.org/p/sahara-juno-post-merge-changes
18:09:53 <tmckay> okay
18:09:59 <SergeyLukjanov> any other news/updates?
18:10:03 <SergeyLukjanov> crobertsrh, great
18:10:22 <crobertsrh> My Juno cycle is probably going to be truncated by a couple of weeks.  I will most likely be gone for a couple weeks starting August 15th.
18:10:33 <mattf> i'm planning to submit a talk about sahara & spark for summit
18:10:45 <tmckay> crobertsrh, why is that? ;-)
18:10:58 <crobertsrh> Because I am a slacker :)
18:10:58 <alazarev> crobertsrh: it would be great to get at least something working in horizon by juno :)
18:11:21 <SergeyLukjanov> #link https://etherpad.openstack.org/p/sahara-juno-post-merge-changes
18:11:29 <dmitryme> alazarev: actually there are couple of pages in horizon alredy :-)
18:11:33 <crobertsrh> alazarev:  the current "drop dead" date is j-2 (July 24), so I hope so too.
18:11:45 <SergeyLukjanov> #topic sahara-dashboard @ horizon status (croberts)
18:12:00 <crobertsrh> The merge is still limping along at a snail's pace.  It looked like the next panel was close to being merged the other day, but a small problem was found and it's back to the waiting queue again.  2 weeks from today is the deadline.  At the current pace of reviews, we will not make it.
18:12:00 * mattf hunts for a box of tissues
18:12:03 <SergeyLukjanov> #link https://etherpad.openstack.org/p/sahara-juno-post-merge-changes
18:12:20 <crobertsrh> I have an agenda item each week in their meeting where they say that the Sahara reviews are a priority, but I'm just not feeling the love (at least not as much as I'd like to make me comfortable).
18:13:01 <crobertsrh> On the plus side, several of the recent -1s have been for things as small as a period at the end of a message.
18:13:09 <crobertsrh> I think they are running out of ways to keep us out.
18:13:22 * mattf half smiles
18:13:41 <mattf> anyone from horizon here now?
18:14:11 <alazarev> mattf: don’t think so
18:14:13 <crobertsrh> Yeah
18:14:40 <crobertsrh> wrong window...sorry
18:15:01 <SergeyLukjanov> okay, let's move on
18:15:09 <SergeyLukjanov> #topic Action items from the last meeting
18:15:12 <mattf> i can re-ping the rht folks
18:15:20 <mattf> SergeyLukjanov, can you re-ping the mirantis folks?
18:15:26 <mattf> sballe, will you re-ping the hp folks?
18:15:31 <SergeyLukjanov> I'm now have drafts for both my action items, but it's still WIP
18:15:44 <SergeyLukjanov> mattf, I think that our folks are well pinged ;)
18:15:48 <sballe> mattf, Yes will do so now
18:15:55 <mattf> thx
18:16:01 <SergeyLukjanov> dmitryme, could you please talk with Tatyana tomorrow?
18:16:15 <SergeyLukjanov> #action SergeyLukjanov to create bp with steps to enable heat be default
18:16:26 <SergeyLukjanov> #action SergeyLukjanov to create bp about removing/hiding username@image for heat based provisioning
18:16:38 <SergeyLukjanov> #action aignatov create bp re moving/updating rest samples docs and do it
18:16:39 <dmitryme> SergeyLukjanov: I will, but she actually does review
18:16:49 <SergeyLukjanov> #action dmitryme to remind aignatov and SergeyLukjanov to make their action items
18:16:50 <SergeyLukjanov> :)
18:17:04 <SergeyLukjanov> #topic Juno 2 (July 24)
18:17:13 <alazarev> saturday is the best time to talk with Tatyana :)
18:17:16 <SergeyLukjanov> so, the next dev milestone is in two weeks
18:17:20 <dmitryme> mine action item is done, but I did it only once. Probably I should have beed more pushy
18:17:46 <dmitryme> alazarev: am, tomorrow is friday actually
18:17:48 <SergeyLukjanov> #link https://wiki.openstack.org/wiki/Juno_Release_Schedule
18:18:46 <alazarev> dmitryme: I used to have Jul 4 each week and consider friday as saturday, sorry :)
18:19:46 <SergeyLukjanov> so, folks, please, note that the next week is the last week to land something for j2
18:20:40 <SergeyLukjanov> due to the current status of sahara 2 dashboard merge I think that we'll need to wait for some additional time after the j2
18:20:54 <SergeyLukjanov> but the merge will be completed in time for Juno release
18:21:09 <mattf> the current plan is to abort and release from sahara so we have SOME opportunity to enhance it in juno
18:21:12 <SergeyLukjanov> I mean that it's not just our hope
18:21:19 <mattf> shall we re-evaluate next week?
18:21:46 <SergeyLukjanov> mattf, we have two weeks to look on a progress
18:21:55 <mattf> i'm very concerned that we're not going to be able to make necessary changes, even if we get it merged
18:22:11 <SergeyLukjanov> mattf, yup, it's possible too
18:22:46 <mattf> i know it's the horizon folks intent to have integrated projects under the horizon umbrella, but the pace of merge is not promising
18:22:47 <crobertsrh> I'm hoping that the wait for changes after the merge is shorter (smaller patches should be easier to review), but there is still a rather large back log of reviews in horizon.
18:23:23 <SergeyLukjanov> crobertsrh, yeah
18:23:37 <mattf> we may be fooling ourselves here
18:23:56 <SergeyLukjanov> that's why I think that we should ensure that our sahara-dashboard is working after j2 and be ready to rollback to it
18:23:59 <mattf> but we should wait a week+ before we have the meeting to decide
18:24:08 <sballe> mattf, I just sent the email. I will follow later in person
18:24:16 <mattf> sballe, thank you
18:24:16 <SergeyLukjanov> but IMO we should double our patches to both -dashboard and horizon now
18:24:31 <SergeyLukjanov> sballe, thank you
18:24:41 <mattf> SergeyLukjanov, let's make that decision w/ the review in a week+
18:25:02 <SergeyLukjanov> sballe, re your questions - ping us after the meeting in sahara channel - it's a good time to catch the team
18:25:08 <SergeyLukjanov> mattf, yeah
18:25:48 <SergeyLukjanov> #action team to review dashboard2horizon merge and decide to rollback or not and when
18:25:58 <sballe> SergeyLukjanov, ok thx
18:26:29 <SergeyLukjanov> #topic Pilot sahara-specs
18:26:30 <crobertsrh> #link https://review.openstack.org/#/q/status:open+project:openstack/horizon+branch:master+topic:bp/merge-sahara-dashboard,n,z to see the outstanding patches for the merge
18:26:49 <SergeyLukjanov> crobertsrh, thx
18:26:59 <SergeyLukjanov> so, we have a bunch of open specs - https://review.openstack.org/#/q/sahara-specs+AND+status:open,n,z
18:27:22 <SergeyLukjanov> and some already merged specs - https://review.openstack.org/#/q/sahara-specs+AND+status:merged,n,z
18:27:43 <SergeyLukjanov> and I'm happy to see that it's really helping to make blueprints much more detailed
18:27:59 <elmiko> SergeyLukjanov: +1
18:28:42 <SergeyLukjanov> #topic Plugin-specific sahara configs
18:28:50 <SergeyLukjanov> tmckay, let's discuss it here
18:29:07 <SergeyLukjanov> so, you need to add spark plugin specific option to sahara.conf?
18:29:19 <crobertsrh> Was it removed before because we didn't want them or because we just didn't use/need them at the time?
18:29:26 <tmckay> well, that's the question.  In short, spark EDP needs to know SPARK_HOME
18:29:32 <tmckay> but, the nodes don't
18:29:50 <SergeyLukjanov> crobertsrh, I've removed them while migrating from handmade plugin framework to stevedore
18:29:52 <tmckay> So, should that be a cluster config, or something only on the Sahara side?
18:30:02 <tmckay> Maybe a cluster config is better
18:30:05 <mattf> tmckay, isn't SPARK_HOME an attribute of the cluster?
18:30:13 <SergeyLukjanov> tmckay, IMO it's cluster config
18:30:25 <SergeyLukjanov> tmckay, it could be diff. value for diff. images
18:31:04 <tmckay> right, it's fixed in the image.  Okay, I agree, it should be a cluster config with a default in the spark plugin
18:31:11 <mattf> i think we all just came to the same conclusion at the same moment
18:31:17 * tmckay has been poking at this for a little while
18:31:44 <SergeyLukjanov> sounds like time for open disc
18:31:49 <tmckay> so we don't need plugin specific configs in sahara.conf for now :)
18:31:58 <tmckay> But if we do, I found a simple way :)
18:32:08 <SergeyLukjanov> anyone wants personal topics?
18:32:23 <elmiko> i've got a question or two about the swift/auth changes
18:32:27 <SergeyLukjanov> tmckay, it's really easy to add support for it
18:32:43 <tmckay> You can create on oslo.cfg.ConfigOpts with a specific section ("spark") and reparse the default config files at plugin load, then reference that object only in the plugin
18:32:56 <tmckay> isolated, and no reparse of the main CONF object
18:32:57 <SergeyLukjanov> elmiko, it's important, so, let's have a topic for it
18:33:09 <elmiko> k
18:33:33 <SergeyLukjanov> #topic Swift auth discussion
18:33:39 <SergeyLukjanov> elmiko, please
18:33:58 <elmiko> ok, so i'll need some help understanding the hadoop-swift fs component
18:34:04 <elmiko> who is the best person to bug about that?
18:34:13 <SergeyLukjanov> elmiko, alazarev could help with it
18:34:19 <elmiko> noted
18:34:32 <SergeyLukjanov> elmiko, he's not the author but I think that he knows it well
18:34:40 <elmiko> also, should i wait for the spec to be merged before i start making a branch and setting up reviews?
18:35:15 <SergeyLukjanov> elmiko, I think that you should wait for merge just to be sure that we're all agree on the approach
18:35:16 <alazarev> elmiko: yes, I can
18:35:30 <elmiko> ack, i'll keep investigating
18:35:34 <SergeyLukjanov> elmiko, but if you have something already done, it's ok to propose reviews
18:35:52 <mattf> def no +A tho
18:36:48 <elmiko> also, we will need to use the swift storageURL for access to the token based auth. would it be acceptable to change the url we pass in workflow.xml or better to put the url in place with the tokens?
18:37:22 <SergeyLukjanov> mattf, aignatov, tmckay, please, review https://review.openstack.org/#/c/104647/ (and let's approve it on the next week to make folks able to review it)
18:38:09 <tmckay> elmiko, unclear on the workflow.xml url issue.
18:38:51 <tmckay> elmiko, hadoop will need something to identify the object, and something else to handle the auth.
18:39:03 <tmckay> I should probably review your spec :)
18:39:03 <elmiko> my understanding is that we pass a swift:// url to the nodes in the workflow.xml (along with the creds)
18:39:09 <tmckay> I saw it in etherpad form
18:39:31 <tmckay> yes, and the fs plugin takes that url apart
18:39:33 <elmiko> if we want to use the token based authentication we will need the swift storageURL to gain access
18:39:53 <elmiko> which looks more like, http://ip:port/AUTH_uuid/container/object
18:40:00 <tmckay> does that still point to the particular object?
18:40:05 <tmckay> oh, so it does
18:40:19 <elmiko> we can still pull the container/object out, but the base url needs to be different
18:40:51 <elmiko> my question is, will it be acceptable to change the url that gets deployed to the workflow?
18:41:01 <elmiko> (i will note this in the spec)
18:41:14 <tmckay> I think that should be okay.  In general, we need all swift urls to not require credentials, so swift access in Sahara (for getting binaries) and in hadoop should work the same
18:41:26 <tmckay> I think "yes"
18:41:34 <elmiko> k
18:41:47 <tmckay> Hmm, well, it might depend on the patch
18:42:00 <tmckay> I think it identifies the fs type by the schema
18:42:14 <tmckay> hdfs://, swift://, ec
18:42:26 <elmiko> we could still preface with swift://, and just replace with http:// during usage
18:42:30 <tmckay> so http might throw a wrench. alazarev may know better
18:42:35 <tmckay> ack
18:42:48 <elmiko> we just can't do, swift://container/object
18:42:51 <tmckay> at that point, I think the details could be handled in the swift plugin part
18:43:00 <elmiko> we need the actual storage url for the token to work
18:43:06 <tmckay> gotcha
18:43:22 <alazarev> yes, it schema is registered in hadoop
18:43:25 <elmiko> yea, also why i need some help with the swift plugin part. my java is a little rusty
18:43:51 <tmckay> Mine has moth holes.
18:43:55 <elmiko> lol
18:44:08 <tmckay> but I can read it
18:44:33 <SergeyLukjanov> alazarev, wake up, please :)
18:44:42 <elmiko> alazarev: when i get some specific questions about the swift plugin, i'll bring them to you
18:44:52 <alazarev> elmiko: sure
18:44:53 <tmckay> so elmiko, in short, the only thing that use that url in the workflow is hadoop itself
18:45:15 <elmiko> ok, good to know
18:45:28 <tmckay> we have the same form url for sahara binary retrieval, but those can be changed separately.  And we might have to change validation in the REST stuff for the form of the url, not sure.
18:45:57 <elmiko> the alternative is for sahara to distribute the storageURL when it distributes the trust tokens
18:46:27 <alazarev> elmiko: but this url is used by generic hadoop driver, .container was used to bypass its usual workflow
18:47:02 <tmckay> elmiko, I think it might depend on user impact
18:47:03 <elmiko> alazarev: but the swift:// url is only used by the swift plugin, right?
18:47:24 <tmckay> elmiko, we still want people in the UI or cli to say "swift://container/object"
18:47:30 <tmckay> without any reference to auth
18:47:38 <elmiko> tmckay: yes, that doesn't change
18:47:59 <elmiko> internally, sahara will associate the storageURL endpoint with the container and object from the user
18:48:31 <tmckay> elmiko, any security issue having that http url exposed in the workflow?
18:48:42 <elmiko> that url can only be used with authentication
18:48:59 <elmiko> so, i don't think so
18:49:12 <tmckay> elmiko, we could also send multiple urls (not sure if there's a reason)
18:49:13 <alazarev> elmiko: right
18:49:30 <elmiko> alazarev: ack
18:49:57 <elmiko> i think that satisfies my questions.... for now
18:50:10 <tmckay> lot of inside baseball, I think the other people went to sleep :)
18:50:23 <elmiko> this always happens when i start talking about this
18:50:54 <SergeyLukjanov> :)
18:50:58 <tmckay> elmiko, someday spark will support swift and we may have the whole discussion again
18:51:10 <SergeyLukjanov> #topic Open discussion
18:51:16 <SergeyLukjanov> 9 mins left
18:51:21 <elmiko> tmckay: i'm ok with that, i will be a keystone/swift wizard by then :D
18:52:22 <elmiko> just to echo SergeyLukjanov's comment from before, i think the new -spec system is really nice and helps to define the blueprints in greater detail.
18:52:44 <tmckay> +2, I've written a couple and it was great
18:53:17 <SergeyLukjanov> I'm writing two and facing some language barrier to fully describe
18:53:23 <SergeyLukjanov> and time barrier too ;)
18:55:11 <SergeyLukjanov> 5 mins left
18:55:23 <crobertsrh> nothing from me
18:55:37 * SergeyLukjanov ending the meeting in 1 min
18:56:22 <SergeyLukjanov> #endmeeting