14:01:37 #startmeeting sahara 14:01:37 Meeting started Thu Jan 14 14:01:37 2016 UTC and is due to finish in 60 minutes. The chair is elmiko. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:01:38 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:01:40 The meeting name has been set to 'sahara' 14:02:03 #link https://wiki.openstack.org/wiki/Meetings/SaharaAgenda 14:02:11 o/ 14:02:39 question, do we still need the sahara@horizon agenda item? 14:02:43 oops, 2 mins late 14:02:50 (it looked like most of the links on the etherpad are completed) 14:02:54 #chair SergeyLukjanov 14:02:54 Current chairs: SergeyLukjanov elmiko 14:02:57 np SergeyLukjanov 14:02:59 seems like time to remove it elmiko 14:03:04 the floor is yours =) 14:03:09 ack, i'll cut it 14:03:11 elmiko thx :) 14:03:20 there should be no more merging issues ;) 14:03:24 any objections? 14:03:44 none here 14:03:45 nothing 14:04:06 #agreed to remove separated dashboard discussion item from meeting regular agenda 14:04:14 #agree to remove separated dashboard discussion item from meeting regular agenda 14:04:23 hi 14:04:29 #topic News / updates 14:04:34 (including dashboard :) ) 14:05:14 I'm working on support running sahara-api via apache 14:05:27 i've been working on distributed periodic tasks implementation 14:05:28 i've got the last part of the improved-secret-store spec up for review (cdh passwords), and am bringing the api v2 wiki into shape. i've also been investigating some security issues with sahara related to our bandit gate. 14:05:36 I'll be in CA for the next 2-3 month starting this weekends, so, will be available in PST time zone 14:05:52 made some researches for health checks, writing spec is in the progress 14:06:02 i am working on moving sahara scenario tests 14:06:05 elmiko: EDP log enhancement spec updated need your review https://review.openstack.org/#/c/245571/ 14:06:07 esikachev is currently actively working on the scenario tests separation and hopefully we'll get it soon 14:06:09 SergeyLukjanov: wow, nice long stay. getting away from the snow? 14:06:21 huichun: ack, i will take a look today 14:06:37 elmiko, yeah, it was -20 C few days ago 14:06:43 yeeeouch! 14:07:31 esikachev could you please send a link to the CR for tests? 14:07:43 I'm almost finished with internal tests, will be moving to dashboard activities soon 14:07:50 SergeyLukjanov EDP scheduler patch has been long time reviewed, and the integration tests has been passed. Ready to merge https://review.openstack.org/#/c/182310/ 14:07:53 #info we'll have Mitaka-2 next week 14:08:01 #link https://review.openstack.org/#/c/267543/ 14:08:05 SergeyLukjanov: ^^ 14:08:40 huichun ack 14:08:56 SergeyLukjanov: other patches merged 14:09:51 elmiko: suspend edp patch has been updated, long time reviewed, need your review too ^ ^ https://review.openstack.org/#/c/201448/ 14:09:58 esikachev: thanks for the work on moving scenario tests (and thanks more for keeping the history) 14:10:11 huichun: thanks, i'll add it to my queue ;) 14:10:16 tosky: np) 14:11:32 actually regarding the tests move 14:11:55 I've pushed the cleaned-up by esikachev base code to the sahara-scenario repo 14:12:08 so, please, do not merge into the scenario tests in the sahara repo 14:12:18 SergeyLukjanov: thanks 14:12:39 and we're planning to rename repo on the next maintainence window from sahara-scenario to sahara-tests 14:13:06 cool 14:13:18 when is it planned more or less? 14:13:23 rename? 14:13:32 yes; one week, two weeks... ? 14:13:42 no idea, hopefully no more then few weeks 14:13:46 ok 14:14:04 we have a request for one more rename, so, probably earlier than later, will decide next Tue on the infra meeting 14:14:32 #topic Action items from the last meeting 14:14:34 we have one 14:14:39 tosky to comment on the scenario separation spec about git grafting the history 14:14:53 but I think it's fully resolved ;) 14:15:06 tosky could you please ack if you're ok with how it looks like now? 14:15:22 SergeyLukjanov: yes, it's good, thanks 14:15:32 ack here or on the review? 14:15:54 here is enough 14:15:55 thx 14:16:04 #topic Separated launchpad for sahara-dashboard 14:16:25 so, I remember there were questions about where to manage sahara-dashboard blueprints and etc. 14:16:52 IMO for now it's better to keep it in sahara project, because we have the same release cycle, specs and etc. 14:16:56 any thoughts? 14:17:19 are there any examples of other projects tracking their dashboard separately? 14:17:20 I agree to keep them into sahara; you can easily search by tags, as it was done up to this point 14:17:35 i think trove just decided to make a separate lp for their dashboard 14:17:46 I agree with SergeyLukjanov position 14:17:57 and iirc, there are a few others that manage their dashboard through a separate lp 14:18:06 but, i'm ok with keeping it in the same 14:18:24 NikitaKonovalov, nope, I've never seen such 14:18:38 the one nice point about having the dashboard stuff in a separate lp is that we can add horizon cores to that project 14:18:38 It will be friendly for users in most cases, bug manager will decide correct scope of the bug 14:18:41 elmiko, oh, I've missed this (re trove) 14:18:41 SergeyLukjanov: look at trove 14:18:55 SergeyLukjanov: they just decided that last week 14:18:59 ok 14:19:13 vgridnev: https://review.openstack.org/#/c/201448/ suspend EDP patch updated according to your review comments 14:19:34 huichun, ok 14:19:43 elmiko, what's the profit of having horizon-cores added at launchpad? 14:19:55 we can have them at gerrit and it's not related 14:20:13 elmiko: add horizon core to the project, does it change something? 14:20:25 ups, too slow to type :) 14:20:52 tosky: i meant, add them to the lp project for just the dashboard. (i'm not even sure they'd want to, but it was an interesting topic brought up in the trove discussion) 14:21:11 ok, but as SergeyLukjanov pointed out, gerrit permissions are not related 14:21:20 right 14:21:38 and don't get me wrong, i'm not arguing to make our dashboard separate. i think it's fine how it is 14:22:04 no, I was really trying to understand if there is some difference for some ACL, for example 14:22:20 SergeyLukjanov: the point that was made by the trove team was that having a separate lp for dashboard would give them more granularity to allow the horizon cores to manage the dashboard lp, if necessary 14:22:29 tosky only bugs and bp management 14:22:29 tosky: i'm not exactly sure on that point 14:22:54 yea, i think they mainly were interested in bug mgmt by horizon cores, if necessary 14:23:10 elmiko unfortunately I don't know why horizon folks will want to manage lp of some external dashboard :) 14:23:19 SergeyLukjanov: me either ;) 14:23:21 which I don't think they (horizon core) would ever do 14:23:23 exactly 14:23:27 tosky: right... 14:23:38 I think they a lot of things to do in their project :) 14:23:45 so, i'm +1 for keeping it in the same lp 14:23:53 could be 14:24:11 so, seems like we all don't see any reasons for separating right now 14:24:30 let's keep this item for the next meeting if some one will find out why we should do it 14:24:46 sounds good 14:24:58 okay 14:25:02 #topic clouds.yaml support, #link https://review.openstack.org/#/c/236712/ 14:25:07 #link https://review.openstack.org/#/c/236712/ 14:25:26 I don't know who added this item, but it's the thing I was planning to discuss too ;) 14:25:32 i added it =) 14:25:49 and i know you saw the review, i was just curious if we had someone who will take up this work? 14:26:32 I think apavlov can do it 14:26:36 elmiko: is it just for the CLI? Given that we are migrating to openstackclient, does it make sense to add support to the old CLI client? 14:26:39 AndreyPavlov ^^ 14:26:40 SergeyLukjanov: ^ 14:27:01 tosky: right... it may not even be worth it 14:27:11 I think only for the new CLI 14:27:15 right 14:27:30 so, if we are planning to fully integrate with openstackclient then we might not need to 14:27:40 and as I understand it should be easy to do 14:27:57 yea, the patch looked relatively small 14:28:06 the line "Many projects provide openstackclient extensions rather than their own client, so are covered already." sounds like you don't need to do anything, but I don't know the internal there 14:28:20 on this topic though, will we be planning to integrate with the openstacksdk project? 14:28:36 tosky: yea, it would mainly be if we planned to port it into the sahara cli tool 14:29:55 tosky, I have a feeling that it should be anyway checked 14:30:15 sure 14:30:17 it's a nice convenience for our users to have 14:30:18 just wondering 14:30:28 I totally agree it's useful 14:32:33 AndreyPavlov please check it 14:32:53 thanks SergeyLukjanov and AndreyPavlov =) 14:33:11 SergeyLukjanov: sure, i will 14:33:17 #topic API v2 progress 14:33:21 hey 14:33:26 so, i've got a few updates here 14:33:38 1. please, more reviews on #link https://review.openstack.org/#/c/212172/ 14:33:50 i need to add something about tempest tests, but this spec is quite old now... 14:34:07 2. i'm adding more content into #link https://wiki.openstack.org/wiki/Sahara/api-v2 14:34:17 i will soon start adding the individual work items to that page 14:34:21 and, finally 14:34:45 3. i'm getting an experimental version of the /v2 endpoint up and running with the project id removed from the uri 14:34:52 cool! 14:34:55 any questions, comments, or concerns? 14:34:58 =) 14:34:59 elmiko great work 14:35:10 I've added CR to my list 14:35:19 sweet, thanks SergeyLukjanov ! 14:35:50 that's all i had 14:35:57 anything to discuss now, elmiko ? 14:36:01 :) 14:36:05 not unless there are questions 14:36:28 #topic Open discussion 14:36:51 trove and hbase, i hope everyone has taken a look at this #link http://lists.openstack.org/pipermail/openstack-dev/2016-January/083422.html 14:37:09 i've commented as much as i can, but if we have some hbase experts it would be nice if they could read that 14:37:29 trove is proposing adding hbase standalone and pseudo-distrib. as datastores 14:37:43 by themselves, these modes are not production ready 14:37:59 so the next question is, where will this go with trove, and can we help with some sort of integration? 14:38:15 i don't think they want to do that currently, but still we should help to expose the issues they might run into 14:41:13 SergeyLukjanov: I have seen many data pipeline solution (aws, LinkedIn'solution..) compared with our Sahara EDP engine, I think maybe next release we should talk about this integration,IMO, data pipe line is more useful and practical 14:41:14 different topic, has anyone been looking at openstacksdk, and i think we should start planning to integrate =) 14:41:17 elmiko: 14:41:53 interesting, i'm not that familiar with data pipeline 14:42:12 huichun: do you have any links that we could read about it? 14:42:50 elmiko, re trove vs hbase - I don't know how to say that many of HBase users using not only Hbase, but other parts of Hadoop world 14:43:13 elmiko, re standalone and pseudo == 100% useless 14:43:17 SergeyLukjanov: right... 14:43:24 elmiko: http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/what-is-datapipeline.html 14:43:24 and that was brought up on the ML 14:43:29 huichun: thanks! 14:44:07 SergeyLukjanov: anyways, it would be nice to have more voices in that thread 14:44:10 elmiko re sdk - I haven't seen yet, it'll be cool if you can make some short overview 14:44:21 SergeyLukjanov: ack, i'll do that for next meeting =) 14:44:29 huichun do you mean to support pipelines from sahara or implement in sahara? 14:44:32 elmiko, thx! 14:44:45 #action elmiko do short overview of openstacksdk for next meeting 14:45:27 SergeyLukjanov: we have two option,one is create a standalone project named data pipeline(use Sahara) 14:46:04 Second is redesign current EDP engine to support data pipeline 14:47:39 Actually we did so many discussions internally abotu data pipeline idea, we thought it could be another project in OpenStack and equivalent with AWS Data Pipeline. 14:48:06 interesting 14:48:22 SergeyLukjanov: many company use data pipeline to solve their problem with EMR 14:48:27 yeah, we were having a lot of discussions about it in Mirantis too 14:48:47 LinkedIn. Netflix etc 14:48:53 sounds like it should live one more layer upper 14:49:29 like some project that will use trove, sahara, murano as a providers of building blocks and actions and mistral to define and execute flows 14:50:09 Yes 14:50:20 Yes, agree with SergeyLukjanov. 14:50:43 But it's a huge change, actually redesign the EDP parts 14:51:16 And if it is possible Sahara can also be used in Murano for better support. 14:51:53 An example like AWS, Data Pipeline also call EMR to build a Hadoop cluster. 14:52:14 i like the idea of higher level openstack apps 14:52:24 But Data Pipeline's capabilities are not only for Hadoop cluster, but also for the data flow control. 14:52:47 we have some ideas of implementing sahara support in murano to able to use all sahara capabilities from murano 14:52:51 That's why we thought this should be a high level openstack apps. 14:52:58 but last time we checked murano was too limited for it 14:53:31 ah, too bad 14:53:33 in theory it fits the murano goals as it related to the app catalog and it itself is a set of building blocks 14:53:58 are you talking about a.o.o or murano as app-catalog? =) 14:54:05 elmiko it was around half of year ago, so, I think we'll need to re-avaluate it in Mitaka for sure 14:54:13 SergeyLukjanov: +1 14:54:18 Our idea is Sahara can keep the focus for provisioning a Hadoop/Spark cluster and we can have a higher level project to control the data flow. 14:54:52 weiting: i'm actually working on something similar, albeit smaller in scope, to propose as a talk for openstack summit in austin 14:55:03 kzaitsev_mb re murano as catalog, a.o.o is just a list of apps 14:55:31 elmiko, great to hear this info. 14:55:32 some parts of data flow could be (and I think should be) supported directly by sahara 14:55:45 but only the BigData related parts and not too upper layer 14:55:55 agreed 14:56:28 in general - the data flows are super wide term and it includes every resource and capabilities that could be provided by openstack 14:56:57 including integration with AWS for example for multicloud usage 14:57:44 yea, this is perfect for higher order applications 14:57:55 we should keep sahara focused on its specialties 14:58:29 agree 14:58:42 1 min left 14:58:49 any last minute q? 14:59:05 everyone having a nice new year? ;) 14:59:44 so far so good 14:59:48 #endmeeting