18:03:51 <SergeyLukjanov> #startmeeting sahara 18:03:51 <notmyname> heh, ok :-) 18:03:51 <openstack> Meeting started Thu Jul 10 18:03:51 2014 UTC and is due to finish in 60 minutes. The chair is SergeyLukjanov. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:03:52 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:03:55 <openstack> The meeting name has been set to 'sahara' 18:04:13 <SergeyLukjanov> notmyname, btw you are fast ;) 18:04:23 <SergeyLukjanov> #link https://wiki.openstack.org/wiki/Meetings/SaharaAgenda#Next_meetings 18:04:32 <SergeyLukjanov> #topic News / updates 18:04:35 <SergeyLukjanov> folks, please 18:04:41 <notmyname> SergeyLukjanov: I'm "swift"? ;-) 18:04:53 <SergeyLukjanov> notmyname :) 18:05:07 <SergeyLukjanov> mattf, hey 18:05:09 <alazarev> uppercase go before lowercase :) 18:05:29 <mattf> SergeyLukjanov, hey 18:05:37 <dmitryme> I did minor changes + review 18:05:38 * mattf is happy to have found his way back to irc 18:05:54 <SergeyLukjanov> mattf, we're happy to see you again too ;) 18:06:05 <SergeyLukjanov> mattf, and it's time to sync oslo 18:06:06 * mattf smiles 18:06:08 <SergeyLukjanov> (no kidding) 18:06:12 <mattf> overdue actually! 18:06:21 <SergeyLukjanov> mattf, yeah 18:06:23 <elmiko> i've been investigating the code base in preparation for the swift/auth bp. starting to make mock ups of the keystone trust delegation and swift client interactions. also getting some reviews in. 18:06:28 <mattf> i'll put it in my evening catchup queue 18:06:34 <SergeyLukjanov> mattf, awesome, thx 18:06:40 <mattf> give me an action for the meeting so i'm definitely on the hook 18:06:49 <SergeyLukjanov> #action mattf to sync oslo 18:07:15 <SergeyLukjanov> crobertsrh, we have a separated topic for you re dashboard2horizon ;) 18:07:26 <crobertsrh> Yes, I feel so special 18:07:35 <mattf> 2horizon or not 2horizon 18:07:58 <SergeyLukjanov> the only new thing from me is that I've pushed patches to make sahara using i18n 18:08:06 <SergeyLukjanov> mattf, :) 18:08:20 <alazarev> I’ve fixed some heat engine staff 18:08:20 <SergeyLukjanov> I mean oslo.i18n 18:08:24 <tmckay> working on spark EDP via ssh and spark-submit 18:08:41 <tmckay> I have some questions about plugin configs for general topics ... 18:08:49 <alazarev> and pushed error handing refactoring 18:09:25 <SergeyLukjanov> tmckay, yup, I've seen your questions 18:09:42 <SergeyLukjanov> tmckay, let's discuss it before the open discussion 18:09:51 <crobertsrh> I've been putting together a list of dashboard to-do items and trying to think about whether or not they will fit for Juno. #link https://etherpad.openstack.org/p/sahara-juno-post-merge-changes 18:09:53 <tmckay> okay 18:09:59 <SergeyLukjanov> any other news/updates? 18:10:03 <SergeyLukjanov> crobertsrh, great 18:10:22 <crobertsrh> My Juno cycle is probably going to be truncated by a couple of weeks. I will most likely be gone for a couple weeks starting August 15th. 18:10:33 <mattf> i'm planning to submit a talk about sahara & spark for summit 18:10:45 <tmckay> crobertsrh, why is that? ;-) 18:10:58 <crobertsrh> Because I am a slacker :) 18:10:58 <alazarev> crobertsrh: it would be great to get at least something working in horizon by juno :) 18:11:21 <SergeyLukjanov> #link https://etherpad.openstack.org/p/sahara-juno-post-merge-changes 18:11:29 <dmitryme> alazarev: actually there are couple of pages in horizon alredy :-) 18:11:33 <crobertsrh> alazarev: the current "drop dead" date is j-2 (July 24), so I hope so too. 18:11:45 <SergeyLukjanov> #topic sahara-dashboard @ horizon status (croberts) 18:12:00 <crobertsrh> The merge is still limping along at a snail's pace. It looked like the next panel was close to being merged the other day, but a small problem was found and it's back to the waiting queue again. 2 weeks from today is the deadline. At the current pace of reviews, we will not make it. 18:12:00 * mattf hunts for a box of tissues 18:12:03 <SergeyLukjanov> #link https://etherpad.openstack.org/p/sahara-juno-post-merge-changes 18:12:20 <crobertsrh> I have an agenda item each week in their meeting where they say that the Sahara reviews are a priority, but I'm just not feeling the love (at least not as much as I'd like to make me comfortable). 18:13:01 <crobertsrh> On the plus side, several of the recent -1s have been for things as small as a period at the end of a message. 18:13:09 <crobertsrh> I think they are running out of ways to keep us out. 18:13:22 * mattf half smiles 18:13:41 <mattf> anyone from horizon here now? 18:14:11 <alazarev> mattf: don’t think so 18:14:13 <crobertsrh> Yeah 18:14:40 <crobertsrh> wrong window...sorry 18:15:01 <SergeyLukjanov> okay, let's move on 18:15:09 <SergeyLukjanov> #topic Action items from the last meeting 18:15:12 <mattf> i can re-ping the rht folks 18:15:20 <mattf> SergeyLukjanov, can you re-ping the mirantis folks? 18:15:26 <mattf> sballe, will you re-ping the hp folks? 18:15:31 <SergeyLukjanov> I'm now have drafts for both my action items, but it's still WIP 18:15:44 <SergeyLukjanov> mattf, I think that our folks are well pinged ;) 18:15:48 <sballe> mattf, Yes will do so now 18:15:55 <mattf> thx 18:16:01 <SergeyLukjanov> dmitryme, could you please talk with Tatyana tomorrow? 18:16:15 <SergeyLukjanov> #action SergeyLukjanov to create bp with steps to enable heat be default 18:16:26 <SergeyLukjanov> #action SergeyLukjanov to create bp about removing/hiding username@image for heat based provisioning 18:16:38 <SergeyLukjanov> #action aignatov create bp re moving/updating rest samples docs and do it 18:16:39 <dmitryme> SergeyLukjanov: I will, but she actually does review 18:16:49 <SergeyLukjanov> #action dmitryme to remind aignatov and SergeyLukjanov to make their action items 18:16:50 <SergeyLukjanov> :) 18:17:04 <SergeyLukjanov> #topic Juno 2 (July 24) 18:17:13 <alazarev> saturday is the best time to talk with Tatyana :) 18:17:16 <SergeyLukjanov> so, the next dev milestone is in two weeks 18:17:20 <dmitryme> mine action item is done, but I did it only once. Probably I should have beed more pushy 18:17:46 <dmitryme> alazarev: am, tomorrow is friday actually 18:17:48 <SergeyLukjanov> #link https://wiki.openstack.org/wiki/Juno_Release_Schedule 18:18:46 <alazarev> dmitryme: I used to have Jul 4 each week and consider friday as saturday, sorry :) 18:19:46 <SergeyLukjanov> so, folks, please, note that the next week is the last week to land something for j2 18:20:40 <SergeyLukjanov> due to the current status of sahara 2 dashboard merge I think that we'll need to wait for some additional time after the j2 18:20:54 <SergeyLukjanov> but the merge will be completed in time for Juno release 18:21:09 <mattf> the current plan is to abort and release from sahara so we have SOME opportunity to enhance it in juno 18:21:12 <SergeyLukjanov> I mean that it's not just our hope 18:21:19 <mattf> shall we re-evaluate next week? 18:21:46 <SergeyLukjanov> mattf, we have two weeks to look on a progress 18:21:55 <mattf> i'm very concerned that we're not going to be able to make necessary changes, even if we get it merged 18:22:11 <SergeyLukjanov> mattf, yup, it's possible too 18:22:46 <mattf> i know it's the horizon folks intent to have integrated projects under the horizon umbrella, but the pace of merge is not promising 18:22:47 <crobertsrh> I'm hoping that the wait for changes after the merge is shorter (smaller patches should be easier to review), but there is still a rather large back log of reviews in horizon. 18:23:23 <SergeyLukjanov> crobertsrh, yeah 18:23:37 <mattf> we may be fooling ourselves here 18:23:56 <SergeyLukjanov> that's why I think that we should ensure that our sahara-dashboard is working after j2 and be ready to rollback to it 18:23:59 <mattf> but we should wait a week+ before we have the meeting to decide 18:24:08 <sballe> mattf, I just sent the email. I will follow later in person 18:24:16 <mattf> sballe, thank you 18:24:16 <SergeyLukjanov> but IMO we should double our patches to both -dashboard and horizon now 18:24:31 <SergeyLukjanov> sballe, thank you 18:24:41 <mattf> SergeyLukjanov, let's make that decision w/ the review in a week+ 18:25:02 <SergeyLukjanov> sballe, re your questions - ping us after the meeting in sahara channel - it's a good time to catch the team 18:25:08 <SergeyLukjanov> mattf, yeah 18:25:48 <SergeyLukjanov> #action team to review dashboard2horizon merge and decide to rollback or not and when 18:25:58 <sballe> SergeyLukjanov, ok thx 18:26:29 <SergeyLukjanov> #topic Pilot sahara-specs 18:26:30 <crobertsrh> #link https://review.openstack.org/#/q/status:open+project:openstack/horizon+branch:master+topic:bp/merge-sahara-dashboard,n,z to see the outstanding patches for the merge 18:26:49 <SergeyLukjanov> crobertsrh, thx 18:26:59 <SergeyLukjanov> so, we have a bunch of open specs - https://review.openstack.org/#/q/sahara-specs+AND+status:open,n,z 18:27:22 <SergeyLukjanov> and some already merged specs - https://review.openstack.org/#/q/sahara-specs+AND+status:merged,n,z 18:27:43 <SergeyLukjanov> and I'm happy to see that it's really helping to make blueprints much more detailed 18:27:59 <elmiko> SergeyLukjanov: +1 18:28:42 <SergeyLukjanov> #topic Plugin-specific sahara configs 18:28:50 <SergeyLukjanov> tmckay, let's discuss it here 18:29:07 <SergeyLukjanov> so, you need to add spark plugin specific option to sahara.conf? 18:29:19 <crobertsrh> Was it removed before because we didn't want them or because we just didn't use/need them at the time? 18:29:26 <tmckay> well, that's the question. In short, spark EDP needs to know SPARK_HOME 18:29:32 <tmckay> but, the nodes don't 18:29:50 <SergeyLukjanov> crobertsrh, I've removed them while migrating from handmade plugin framework to stevedore 18:29:52 <tmckay> So, should that be a cluster config, or something only on the Sahara side? 18:30:02 <tmckay> Maybe a cluster config is better 18:30:05 <mattf> tmckay, isn't SPARK_HOME an attribute of the cluster? 18:30:13 <SergeyLukjanov> tmckay, IMO it's cluster config 18:30:25 <SergeyLukjanov> tmckay, it could be diff. value for diff. images 18:31:04 <tmckay> right, it's fixed in the image. Okay, I agree, it should be a cluster config with a default in the spark plugin 18:31:11 <mattf> i think we all just came to the same conclusion at the same moment 18:31:17 * tmckay has been poking at this for a little while 18:31:44 <SergeyLukjanov> sounds like time for open disc 18:31:49 <tmckay> so we don't need plugin specific configs in sahara.conf for now :) 18:31:58 <tmckay> But if we do, I found a simple way :) 18:32:08 <SergeyLukjanov> anyone wants personal topics? 18:32:23 <elmiko> i've got a question or two about the swift/auth changes 18:32:27 <SergeyLukjanov> tmckay, it's really easy to add support for it 18:32:43 <tmckay> You can create on oslo.cfg.ConfigOpts with a specific section ("spark") and reparse the default config files at plugin load, then reference that object only in the plugin 18:32:56 <tmckay> isolated, and no reparse of the main CONF object 18:32:57 <SergeyLukjanov> elmiko, it's important, so, let's have a topic for it 18:33:09 <elmiko> k 18:33:33 <SergeyLukjanov> #topic Swift auth discussion 18:33:39 <SergeyLukjanov> elmiko, please 18:33:58 <elmiko> ok, so i'll need some help understanding the hadoop-swift fs component 18:34:04 <elmiko> who is the best person to bug about that? 18:34:13 <SergeyLukjanov> elmiko, alazarev could help with it 18:34:19 <elmiko> noted 18:34:32 <SergeyLukjanov> elmiko, he's not the author but I think that he knows it well 18:34:40 <elmiko> also, should i wait for the spec to be merged before i start making a branch and setting up reviews? 18:35:15 <SergeyLukjanov> elmiko, I think that you should wait for merge just to be sure that we're all agree on the approach 18:35:16 <alazarev> elmiko: yes, I can 18:35:30 <elmiko> ack, i'll keep investigating 18:35:34 <SergeyLukjanov> elmiko, but if you have something already done, it's ok to propose reviews 18:35:52 <mattf> def no +A tho 18:36:48 <elmiko> also, we will need to use the swift storageURL for access to the token based auth. would it be acceptable to change the url we pass in workflow.xml or better to put the url in place with the tokens? 18:37:22 <SergeyLukjanov> mattf, aignatov, tmckay, please, review https://review.openstack.org/#/c/104647/ (and let's approve it on the next week to make folks able to review it) 18:38:09 <tmckay> elmiko, unclear on the workflow.xml url issue. 18:38:51 <tmckay> elmiko, hadoop will need something to identify the object, and something else to handle the auth. 18:39:03 <tmckay> I should probably review your spec :) 18:39:03 <elmiko> my understanding is that we pass a swift:// url to the nodes in the workflow.xml (along with the creds) 18:39:09 <tmckay> I saw it in etherpad form 18:39:31 <tmckay> yes, and the fs plugin takes that url apart 18:39:33 <elmiko> if we want to use the token based authentication we will need the swift storageURL to gain access 18:39:53 <elmiko> which looks more like, http://ip:port/AUTH_uuid/container/object 18:40:00 <tmckay> does that still point to the particular object? 18:40:05 <tmckay> oh, so it does 18:40:19 <elmiko> we can still pull the container/object out, but the base url needs to be different 18:40:51 <elmiko> my question is, will it be acceptable to change the url that gets deployed to the workflow? 18:41:01 <elmiko> (i will note this in the spec) 18:41:14 <tmckay> I think that should be okay. In general, we need all swift urls to not require credentials, so swift access in Sahara (for getting binaries) and in hadoop should work the same 18:41:26 <tmckay> I think "yes" 18:41:34 <elmiko> k 18:41:47 <tmckay> Hmm, well, it might depend on the patch 18:42:00 <tmckay> I think it identifies the fs type by the schema 18:42:14 <tmckay> hdfs://, swift://, ec 18:42:26 <elmiko> we could still preface with swift://, and just replace with http:// during usage 18:42:30 <tmckay> so http might throw a wrench. alazarev may know better 18:42:35 <tmckay> ack 18:42:48 <elmiko> we just can't do, swift://container/object 18:42:51 <tmckay> at that point, I think the details could be handled in the swift plugin part 18:43:00 <elmiko> we need the actual storage url for the token to work 18:43:06 <tmckay> gotcha 18:43:22 <alazarev> yes, it schema is registered in hadoop 18:43:25 <elmiko> yea, also why i need some help with the swift plugin part. my java is a little rusty 18:43:51 <tmckay> Mine has moth holes. 18:43:55 <elmiko> lol 18:44:08 <tmckay> but I can read it 18:44:33 <SergeyLukjanov> alazarev, wake up, please :) 18:44:42 <elmiko> alazarev: when i get some specific questions about the swift plugin, i'll bring them to you 18:44:52 <alazarev> elmiko: sure 18:44:53 <tmckay> so elmiko, in short, the only thing that use that url in the workflow is hadoop itself 18:45:15 <elmiko> ok, good to know 18:45:28 <tmckay> we have the same form url for sahara binary retrieval, but those can be changed separately. And we might have to change validation in the REST stuff for the form of the url, not sure. 18:45:57 <elmiko> the alternative is for sahara to distribute the storageURL when it distributes the trust tokens 18:46:27 <alazarev> elmiko: but this url is used by generic hadoop driver, .container was used to bypass its usual workflow 18:47:02 <tmckay> elmiko, I think it might depend on user impact 18:47:03 <elmiko> alazarev: but the swift:// url is only used by the swift plugin, right? 18:47:24 <tmckay> elmiko, we still want people in the UI or cli to say "swift://container/object" 18:47:30 <tmckay> without any reference to auth 18:47:38 <elmiko> tmckay: yes, that doesn't change 18:47:59 <elmiko> internally, sahara will associate the storageURL endpoint with the container and object from the user 18:48:31 <tmckay> elmiko, any security issue having that http url exposed in the workflow? 18:48:42 <elmiko> that url can only be used with authentication 18:48:59 <elmiko> so, i don't think so 18:49:12 <tmckay> elmiko, we could also send multiple urls (not sure if there's a reason) 18:49:13 <alazarev> elmiko: right 18:49:30 <elmiko> alazarev: ack 18:49:57 <elmiko> i think that satisfies my questions.... for now 18:50:10 <tmckay> lot of inside baseball, I think the other people went to sleep :) 18:50:23 <elmiko> this always happens when i start talking about this 18:50:54 <SergeyLukjanov> :) 18:50:58 <tmckay> elmiko, someday spark will support swift and we may have the whole discussion again 18:51:10 <SergeyLukjanov> #topic Open discussion 18:51:16 <SergeyLukjanov> 9 mins left 18:51:21 <elmiko> tmckay: i'm ok with that, i will be a keystone/swift wizard by then :D 18:52:22 <elmiko> just to echo SergeyLukjanov's comment from before, i think the new -spec system is really nice and helps to define the blueprints in greater detail. 18:52:44 <tmckay> +2, I've written a couple and it was great 18:53:17 <SergeyLukjanov> I'm writing two and facing some language barrier to fully describe 18:53:23 <SergeyLukjanov> and time barrier too ;) 18:55:11 <SergeyLukjanov> 5 mins left 18:55:23 <crobertsrh> nothing from me 18:55:37 * SergeyLukjanov ending the meeting in 1 min 18:56:22 <SergeyLukjanov> #endmeeting