| *** dina_belova has joined #savanna | 00:09 | |
| *** dina_belova has quit IRC | 00:14 | |
| *** NikitaKonovalov has joined #savanna | 00:16 | |
| *** NikitaKonovalov has quit IRC | 00:20 | |
| *** dina_belova has joined #savanna | 01:10 | |
| *** crobertsrh has joined #savanna | 01:14 | |
| *** crobertsrh is now known as _crobertsrh | 01:15 | |
| *** dina_belova has quit IRC | 01:15 | |
| *** NikitaKonovalov has joined #savanna | 01:16 | |
| *** NikitaKonovalov has quit IRC | 01:20 | |
| *** sanek has quit IRC | 01:58 | |
| *** sanek has joined #savanna | 01:58 | |
| *** dina_belova has joined #savanna | 02:10 | |
| *** dina_belova has quit IRC | 02:15 | |
| *** NikitaKonovalov has joined #savanna | 02:16 | |
| *** NikitaKonovalov has quit IRC | 02:21 | |
| *** IlyaE has joined #savanna | 02:55 | |
| *** dina_belova has joined #savanna | 03:11 | |
| *** dina_belova has quit IRC | 03:16 | |
| *** NikitaKonovalov has joined #savanna | 03:17 | |
| *** NikitaKonovalov has quit IRC | 03:22 | |
| *** IlyaE has quit IRC | 03:32 | |
| *** SergeyLukjanov has joined #savanna | 03:59 | |
| *** dina_belova has joined #savanna | 04:11 | |
| *** dina_belova has quit IRC | 04:16 | |
| *** NikitaKonovalov has joined #savanna | 04:18 | |
| *** NikitaKonovalov has quit IRC | 04:22 | |
| *** dina_belova has joined #savanna | 04:33 | |
| *** dina_belova has quit IRC | 04:36 | |
| *** lastidiot has quit IRC | 04:41 | |
| *** IlyaE has joined #savanna | 04:55 | |
| *** Nadya has joined #savanna | 04:58 | |
| openstackgerrit | Sergey Lukjanov proposed a change to stackforge/savanna: A Resource implementation for Conductor https://review.openstack.org/40400 | 05:05 |
|---|---|---|
| *** dina_belova has joined #savanna | 05:05 | |
| *** Nadya has quit IRC | 05:15 | |
| *** NikitaKonovalov has joined #savanna | 05:18 | |
| *** dina_belova has quit IRC | 05:22 | |
| *** NikitaKonovalov has quit IRC | 05:22 | |
| *** dina_belova has joined #savanna | 05:24 | |
| *** SergeyLukjanov has quit IRC | 05:48 | |
| *** akuznetsov has joined #savanna | 05:57 | |
| *** NikitaKonovalov has joined #savanna | 06:17 | |
| *** NikitaKonovalov has quit IRC | 06:21 | |
| openstackgerrit | Alexander Kuznetsov proposed a change to stackforge/savanna: Added first version of model for EDP https://review.openstack.org/40227 | 06:31 |
| openstackgerrit | Alexander Kuznetsov proposed a change to stackforge/savanna: Added first version of model for EDP https://review.openstack.org/40227 | 06:34 |
| *** NikitaKonovalov has joined #savanna | 06:51 | |
| *** Nadya has joined #savanna | 07:02 | |
| *** NikitaKonovalov has quit IRC | 07:05 | |
| *** ben_duyujie has joined #savanna | 07:26 | |
| *** NikitaKonovalov has joined #savanna | 07:29 | |
| *** SergeyLukjanov has joined #savanna | 07:30 | |
| *** dina_belova has joined #savanna | 07:40 | |
| *** Nadya has quit IRC | 07:43 | |
| *** IlyaE has quit IRC | 07:51 | |
| *** ben_duyujie has quit IRC | 07:54 | |
| openstackgerrit | Sergey Lukjanov proposed a change to stackforge/savanna: Migrate to pbr https://review.openstack.org/37480 | 08:02 |
| openstackgerrit | Sergey Lukjanov proposed a change to stackforge/savanna: Made Ambari RPM location configurable https://review.openstack.org/40479 | 08:03 |
| *** dmitryme has joined #savanna | 08:13 | |
| *** nprivalova has joined #savanna | 08:23 | |
| openstackgerrit | Nikita Konovalov proposed a change to stackforge/savanna: Tests module refactoring https://review.openstack.org/40572 | 08:25 |
| openstackgerrit | Alexander Kuznetsov proposed a change to stackforge/savanna: Added first version of model for EDP https://review.openstack.org/40227 | 08:33 |
| openstackgerrit | Alexander Kuznetsov proposed a change to stackforge/savanna: Added first version of model for EDP https://review.openstack.org/40227 | 08:35 |
| *** ruhe has joined #savanna | 08:37 | |
| openstackgerrit | Alexander Kuznetsov proposed a change to stackforge/savanna: Added first version of model for EDP https://review.openstack.org/40227 | 08:41 |
| openstackgerrit | Nikita Konovalov proposed a change to stackforge/savanna: Tests module refactoring https://review.openstack.org/40572 | 09:03 |
| *** akuznetsov has quit IRC | 09:08 | |
| *** dina_belova has quit IRC | 09:12 | |
| *** ruhe has quit IRC | 09:15 | |
| *** ruhe has joined #savanna | 09:17 | |
| openstackgerrit | Sergey Reshetnyak proposed a change to stackforge/savanna: Refactoring hdp plugin https://review.openstack.org/39733 | 09:23 |
| *** dina_belova has joined #savanna | 09:42 | |
| *** dina_belova has quit IRC | 09:51 | |
| openstackgerrit | Dmitry Mescheryakov proposed a change to stackforge/savanna: A Resource implementation for Conductor https://review.openstack.org/40400 | 10:04 |
| *** NikitaKonovalov has quit IRC | 10:21 | |
| *** ruhe has quit IRC | 10:25 | |
| *** nprivalova has quit IRC | 10:31 | |
| *** dina_belova has joined #savanna | 10:31 | |
| *** NikitaKonovalov has joined #savanna | 10:51 | |
| *** _crobertsrh is now known as crobertsrh | 10:53 | |
| *** NikitaKonovalov has quit IRC | 10:53 | |
| *** NikitaKonovalov has joined #savanna | 10:53 | |
| *** ruhe has joined #savanna | 10:55 | |
| *** nprivalova has joined #savanna | 10:55 | |
| *** akuznetsov has joined #savanna | 10:58 | |
| *** ruhe has quit IRC | 11:02 | |
| *** nprivalova has quit IRC | 11:14 | |
| openstackgerrit | Alexander Ignatov proposed a change to stackforge/savanna: Added basic helper for map-reduce actions https://review.openstack.org/40590 | 11:18 |
| openstackgerrit | Dmitry Mescheryakov proposed a change to stackforge/savanna: A Resource implementation for Conductor https://review.openstack.org/40400 | 11:18 |
| openstackgerrit | A change was merged to stackforge/savanna: Tests module refactoring https://review.openstack.org/40572 | 11:22 |
| openstackgerrit | Alexander Kuznetsov proposed a change to stackforge/savanna: Added conductor API for DataSource object https://review.openstack.org/40591 | 11:22 |
| *** dmitryme has quit IRC | 11:36 | |
| *** dmitryme has joined #savanna | 11:39 | |
| openstackgerrit | Yaroslav Lobankov proposed a change to stackforge/savanna: IT updating for "HDP" plugin https://review.openstack.org/40369 | 11:40 |
| *** bogdando has quit IRC | 11:43 | |
| *** bogdando has joined #savanna | 11:45 | |
| openstackgerrit | Dmitry Mescheryakov proposed a change to stackforge/savanna: A Resource implementation for Conductor https://review.openstack.org/40400 | 11:45 |
| *** nprivalova has joined #savanna | 11:50 | |
| openstackgerrit | Yaroslav Lobankov proposed a change to stackforge/savanna: IT updating for "HDP" plugin https://review.openstack.org/40369 | 11:56 |
| *** ruhe has joined #savanna | 11:58 | |
| openstackgerrit | Alexander Kuznetsov proposed a change to stackforge/savanna: Added first version of model for EDP https://review.openstack.org/40227 | 11:59 |
| openstackgerrit | Alexander Kuznetsov proposed a change to stackforge/savanna: Added conductor API for DataSource object https://review.openstack.org/40591 | 11:59 |
| *** akuznetsov has quit IRC | 12:03 | |
| *** NikitaKonovalov has quit IRC | 12:03 | |
| *** NikitaKonovalov has joined #savanna | 12:07 | |
| *** ranjan has joined #savanna | 12:18 | |
| openstackgerrit | Alexander Kuznetsov proposed a change to stackforge/savanna: Added conductor API for DataSource object https://review.openstack.org/40591 | 12:29 |
| openstackgerrit | Alexander Kuznetsov proposed a change to stackforge/savanna: Added conductor API for DataSource object https://review.openstack.org/40591 | 12:32 |
| ranjan | Hi, all, where can i find documentation on creating a CDH plugin for savanna? | 12:34 |
| openstackgerrit | Dmitry Mescheryakov proposed a change to stackforge/savanna: A Resource implementation for Conductor https://review.openstack.org/40400 | 12:37 |
| SergeyLukjanov | ranjan, hi | 12:37 |
| ranjan | SergeyLukjanov, hi, | 12:37 |
| ranjan | SergeyLukjanov, how are you :) | 12:37 |
| SergeyLukjanov | we have two implemented plugins, you can use them as ref impl | 12:38 |
| SergeyLukjanov | additionally we have docs here - http://savanna.rtfd.org | 12:38 |
| SergeyLukjanov | http://savanna.readthedocs.org/en/latest/devref/plugins.html http://savanna.readthedocs.org/en/latest/devref/plugin.spi.html | 12:38 |
| SergeyLukjanov | we are now working on updating db access layer, so, code around db we be changed | 12:39 |
| SergeyLukjanov | the new db code located in savanna/db_new module, I hope that will finish migration to it to the end of this week | 12:40 |
| ranjan | SergeyLukjanov, but other than vanilla hadoop which other plugin is there, and where is it available? | 12:42 |
| SergeyLukjanov | HDP plugin, https://github.com/stackforge/savanna/tree/master/savanna/plugins/hdp | 12:43 |
| *** nprivalova has quit IRC | 12:43 | |
| ranjan | SergeyLukjanov, okie. | 12:46 |
| ranjan | let me check | 12:47 |
| *** akuznetsov has joined #savanna | 12:47 | |
| SergeyLukjanov | ranjan, you're welcome | 12:47 |
| SergeyLukjanov | ping us here or in mailing list if you'll have questions | 12:47 |
| ranjan | SergeyLukjanov, sure, thank you so much for the kind help and support. | 12:48 |
| openstackgerrit | Dmitry Mescheryakov proposed a change to stackforge/savanna: A Resource implementation for Conductor https://review.openstack.org/40400 | 12:57 |
| *** nprivalova has joined #savanna | 12:57 | |
| *** _tstclair is now known as tstclair | 13:00 | |
| openstackgerrit | Nikita Konovalov proposed a change to stackforge/savanna: Unit test for Conductor Manager improved https://review.openstack.org/40616 | 13:15 |
| openstackgerrit | A change was merged to stackforge/savanna: A Resource implementation for Conductor https://review.openstack.org/40400 | 13:16 |
| *** akuznetsov has quit IRC | 13:20 | |
| openstackgerrit | Nikita Konovalov proposed a change to stackforge/savanna: Unit test for Conductor Manager improved https://review.openstack.org/40616 | 13:28 |
| *** NikitaKonovalov has quit IRC | 13:28 | |
| *** dina_belova has quit IRC | 13:38 | |
| *** ruhe has quit IRC | 13:42 | |
| *** ruhe has joined #savanna | 13:44 | |
| *** ruhe has quit IRC | 13:45 | |
| *** dmitryme has quit IRC | 13:55 | |
| *** NikitaKonovalov has joined #savanna | 13:58 | |
| tmckayrh | SergeyLukjanov, I have some general savanna execution path / db questions. Are you available? | 13:59 |
| * tmckayrh says, ah, the declarative_base, that's where the missing constructors went | 14:02 | |
| *** NikitaKonovalov has quit IRC | 14:03 | |
| *** dina_belova has joined #savanna | 14:05 | |
| nprivalova | tmckayrh, hi! | 14:11 |
| tmckayrh | nprivalova, hello! | 14:12 |
| nprivalova | tmckayrh, do you know where JobOrigin in etherpad? I can't find it | 14:12 |
| tmckayrh | yes. hold on... web history :) | 14:12 |
| tmckayrh | https://etherpad.openstack.org/savanna_API_draft_EDP_extensions | 14:13 |
| nprivalova | oh… it called JobCodeStorage: | 14:13 |
| tmckayrh | nprivalova, are you looking for JobCodeStorage> | 14:14 |
| tmckayrh | ? | 14:14 |
| nprivalova | I thought that https://etherpad.openstack.org/edp_v3_components is up-to-date | 14:15 |
| *** ruhe has joined #savanna | 14:15 | |
| tmckayrh | nprivalova, yes, JobCodeStorage is here https://etherpad.openstack.org/edp_v3_components | 14:15 |
| nprivalova | but it doesn't contain JobOrigin description | 14:16 |
| tmckayrh | But I think that JobCodeStorage is a relic :) | 14:16 |
| *** lastidiot has joined #savanna | 14:16 | |
| tmckayrh | nprivalova, ack, we should maybe move/copy JobOrigin to edp components | 14:16 |
| tmckayrh | this page ^^ is the object model, after all | 14:16 |
| nprivalova | yes. could you please do it? you are expert in JobOrigin :) | 14:17 |
| tmckayrh | nprivalova, looks like we have some duplication (on datasource object, too) | 14:17 |
| tmckayrh | haha, expert, yes! | 14:17 |
| tmckayrh | nprivalova, okay, let's do this -> I'll update the api draft page to refer back to the components page for the object model, and put only api functions on that api page. | 14:18 |
| tmckayrh | all objects go on the components page under the object model section | 14:18 |
| nprivalova | great! | 14:19 |
| tmckayrh | nprivalova, I have a question about the execution path and organization in savanna. Do you know much about the api -> database interactions? | 14:19 |
| SergeyLukjanov | tmckayrh, I'm here now :) | 14:20 |
| SergeyLukjanov | tmckayrh, what questions do you have? | 14:20 |
| tmckayrh | SergeyLukjanov, thanks! still learning who does what in savanna :) (both people and modules, haha) | 14:21 |
| tmckayrh | okay, here is my question | 14:21 |
| *** NikitaKonovalov has joined #savanna | 14:21 | |
| tmckayrh | for cluster creation, the flow looks like it goes "rest api -> savanna service module -> database layer" when a cluster is defined | 14:21 |
| tmckayrh | And that includes spinning up the cluster in the service layer | 14:22 |
| SergeyLukjanov | yep, looks like this | 14:22 |
| tmckayrh | But for something like adding a JobOrigin, there really is no service interaction to start up, I think. | 14:22 |
| SergeyLukjanov | btw we are updating db layer now | 14:22 |
| tmckayrh | So, how should the flow proceed? "api -> db", where the api module references the db layer directly? Or something in between? | 14:23 |
| SergeyLukjanov | I think that we'll call conductor directly from the api | 14:23 |
| tmckayrh | I don't want to break an abstraction, or introduce imports, etc, that are not wanted | 14:23 |
| tmckayrh | ah, the conductor. What is that exactly? | 14:23 |
| SergeyLukjanov | it's an abstraction around db ops | 14:23 |
| SergeyLukjanov | it'll be working locally for all services for now | 14:23 |
| SergeyLukjanov | but in future we'll maybe transform it to separated service | 14:24 |
| SergeyLukjanov | if we'll need it | 14:24 |
| tmckayrh | okay, I'll take another look. Thanks. | 14:24 |
| SergeyLukjanov | so conductor module could be used anywhere in the code | 14:24 |
| SergeyLukjanov | and db module we'll be used actually only in the conductor | 14:25 |
| SergeyLukjanov | https://wiki.openstack.org/wiki/Savanna/NextGenArchitecture | 14:25 |
| SergeyLukjanov | ^^ here are some details about it | 14:25 |
| SergeyLukjanov | I hope that we'll move our current core code to use conductor till the end of week | 14:26 |
| tmckayrh | Thanks, I'll re-read. It may make more sense to me now. Okay. So, to create a JobOrigin, ultimately I need a validation (which means defining a json schema somewhere), and a db model for the type, and a template to render the json return result. Is this right? | 14:26 |
| *** dmitryme has joined #savanna | 14:28 | |
| *** IlyaE has joined #savanna | 14:29 | |
| *** lastidiot has quit IRC | 14:31 | |
| SergeyLukjanov | tmckayrh, yes, validation isn't very important for the first time | 14:49 |
| tmckayrh | SergeyLukjanov, thanks, ack, just checking to make sure I've got the complete picture. | 14:50 |
| *** dina_belova has quit IRC | 14:54 | |
| openstackgerrit | Nikita Konovalov proposed a change to stackforge/savanna: Unit test for Conductor Manager improved https://review.openstack.org/40616 | 14:58 |
| *** dina_belova has joined #savanna | 15:03 | |
| openstackgerrit | Sergey Reshetnyak proposed a change to stackforge/savanna: Refactoring remote utils https://review.openstack.org/39043 | 15:07 |
| *** key has joined #savanna | 15:07 | |
| *** dmitryme has quit IRC | 15:08 | |
| *** NikitaKonovalov has quit IRC | 15:10 | |
| *** lastidiot has joined #savanna | 15:11 | |
| *** NikitaKonovalov has joined #savanna | 15:14 | |
| *** rnirmal has joined #savanna | 15:18 | |
| *** dina_belova has quit IRC | 15:20 | |
| *** SergeyLukjanov has quit IRC | 15:20 | |
| *** NikitaKonovalov has quit IRC | 15:21 | |
| *** key4 has joined #savanna | 15:29 | |
| *** key has quit IRC | 15:32 | |
| *** gkleiman has joined #savanna | 15:38 | |
| *** ruhe has quit IRC | 15:41 | |
| *** NikitaKonovalov has joined #savanna | 15:52 | |
| *** NikitaKonovalov has quit IRC | 15:55 | |
| *** ruhe has joined #savanna | 15:59 | |
| *** tmckayrh is now known as _tmckayrh | 16:00 | |
| ruhe | tmckayrh, hi | 16:01 |
| *** key has joined #savanna | 16:01 | |
| *** _tmckayrh is now known as tmckayrh | 16:02 | |
| tmckayrh | ruhe, hi | 16:02 |
| tmckayrh | ruhe, what's up? I was just going to grab some lunch. | 16:02 |
| ruhe | i had a discussion with our field guys today. and they told me something, i need to apologise to you for. It turns out that many OS deployments don't have Swift installed | 16:03 |
| *** key4 has quit IRC | 16:03 | |
| tmckayrh | :) okay. | 16:03 |
| ruhe | i've pushed for that approach recently. but now it seems like we need to pick something else | 16:03 |
| ruhe | yesterday we discussed another option - store binaries in DB | 16:04 |
| tmckayrh | ruhe, I was keeping an open mind. nprivalova has been advocating job binary storage in hdfs at first, to avoid the complexities of copying from switf -> hdfs, or the issue of running from swift. | 16:04 |
| tmckayrh | ruhe, yes. | 16:05 |
| ruhe | tmckayrh, we need to keep binaries somewhere until the cluster starts. and HDFS will not be available until the cluster starts | 16:05 |
| ruhe | that's all i wanted to say. now you're free to go for lunch :) | 16:06 |
| openstackgerrit | Jonathan Maron proposed a change to stackforge/savanna: Made Ambari RPM location configurable https://review.openstack.org/40479 | 16:07 |
| tmckayrh | ruhe, so that the collection of binaries can be established/maintained outside of savanna, is that the goal? Populate it prior, or delete the cluster and create a new one, or populate hdfs on multiple clusters from the same external source? | 16:08 |
| tmckayrh | It could be an external hdfs.... outside of savanna. Ultimately we just need to be able to copy to the savanna cluster, right? | 16:09 |
| tmckayrh | ruhe, but I have no problem with savanna db storage. Just which one to pursure first. | 16:10 |
| ruhe | right, we need some place to store binaries and we also need to be able to transfer those binaries into provisioned cluster | 16:10 |
| *** dmitryme has joined #savanna | 16:10 | |
| nprivalova | tmckayrh, we are considering the scenario that user uploads all binaries from his desktop. we need to place to store them | 16:10 |
| tmckayrh | ruhe, okay. I am still sneaking up on add/list/delete, "get" which implies copy to local hdfs will be last. I think the first 3 are the same for any type.... | 16:11 |
| ruhe | right | 16:11 |
| tmckayrh | nprivalova, okay. | 16:11 |
| tmckayrh | ruhe, nprivalova, thus far I don't think there is a "put binary" method in the proposed api. Using the savanna db, we would need that. | 16:12 |
| ruhe | yes | 16:12 |
| tmckayrh | ruhe, nprivalova, should that be part of the JobOrigin api: "store to savanna db" | 16:13 |
| nprivalova | I'm preparing a document with it | 16:13 |
| tmckayrh | I'm not sure we want to wrap upload functions for all storage types.... other things that already have an interface, like swift, don't need a savanna api to wrap it | 16:13 |
| tmckayrh | just my thought | 16:14 |
| openstackgerrit | Ivan Berezovskiy proposed a change to stackforge/savanna-extra: Remove unnecessary variable 'WORKAROUND_BUG_1204824' https://review.openstack.org/40667 | 16:14 |
| tmckayrh | okay, thanks for the heads up. Lunch, and then I will try to create something :) I have been digging into sqlalchemy, I like to know how things work ;-) | 16:14 |
| *** tmckayrh is now known as tmckayrh_lunch | 16:15 | |
| *** key has quit IRC | 16:42 | |
| *** key has joined #savanna | 16:42 | |
| *** dmitryme has quit IRC | 16:59 | |
| *** ruhe has quit IRC | 17:01 | |
| *** ranjan has quit IRC | 17:02 | |
| *** dmitryme has joined #savanna | 17:03 | |
| nprivalova | guys, please take a look on updated https://etherpad.openstack.org/edp_v3_components | 17:09 |
| nprivalova | you are very welcome for comments | 17:10 |
| *** dmitryme has quit IRC | 17:10 | |
| nprivalova | please keep in mind that it is only a proposal but is is based on the latest discussions | 17:10 |
| *** ruhe has joined #savanna | 17:11 | |
| *** IlyaE has quit IRC | 17:17 | |
| *** nprivalova has quit IRC | 17:22 | |
| *** lastidiot has quit IRC | 17:24 | |
| openstackgerrit | Jonathan Maron proposed a change to stackforge/savanna: Made Ambari RPM location configurable https://review.openstack.org/40479 | 17:24 |
| *** sacharya1 has joined #savanna | 17:24 | |
| *** sacharya1 has quit IRC | 17:24 | |
| *** ruhe has quit IRC | 17:30 | |
| *** mattf is now known as _mattf | 17:31 | |
| *** IlyaE has joined #savanna | 17:34 | |
| *** sa_ has joined #savanna | 17:35 | |
| *** _mattf is now known as mattf | 17:35 | |
| *** ruhe has joined #savanna | 17:39 | |
| *** ruhe has quit IRC | 17:46 | |
| *** tmckayrh_lunch is now known as tmckayrh | 17:49 | |
| *** sa_ has quit IRC | 17:54 | |
| *** sacharya has joined #savanna | 17:55 | |
| *** dina_belova has joined #savanna | 18:06 | |
| *** dina_belova has quit IRC | 18:14 | |
| *** dina_belova has joined #savanna | 18:28 | |
| tmckayrh | crobertsrh, I hit a problem similar to what I think you hit yesterday. tox invocation complains about the version of python-keystoneclient. | 18:31 |
| *** SergeyLukjanov has joined #savanna | 18:31 | |
| crobertsrh | Yeah | 18:31 |
| tmckayrh | do you know how to hack it? | 18:31 |
| crobertsrh | Your best bet is to blow away your .tox directory and to tools/install_venv again | 18:31 |
| tmckayrh | okay, thanks | 18:31 |
| crobertsrh | You did a git pull, I assume | 18:32 |
| tmckayrh | yes, always want to be current :) | 18:32 |
| SergeyLukjanov | crobertsrh, tmckayrh, hi | 18:34 |
| SergeyLukjanov | it should works ok now | 18:35 |
| tmckayrh | SergeyLukjanov, I recreated tox as crobertsrh suggested, it seems to be working. Should I pull again? | 18:39 |
| crobertsrh | Yeah, Sergey, I was able to get it working this morning again. | 18:39 |
| crobertsrh | I did have to blow away my .tox dir and recreate. Just running tools/install_venv wasn't enough by itself for some reason. | 18:39 |
| SergeyLukjanov | tox caches envs and doesn't handle versions changes | 18:40 |
| SergeyLukjanov | it's sad :( | 18:43 |
| tmckayrh | SergeyLukjanov, simple question. Would the savanna convention for URLs say "/job-origins" or "job_origins" ? I see "/data-sources" in v11.py but on the etherpad it is "/data_sources". | 18:49 |
| tmckayrh | dash or underscore, basically | 18:49 |
| SergeyLukjanov | tmckayrh, dash in urls | 18:49 |
| tmckayrh | okey doke, dash it is | 18:50 |
| SergeyLukjanov | tmckayrh, for example, now we have cluster-templates, etc. | 18:50 |
| *** Nadya has joined #savanna | 18:57 | |
| *** ruhe has joined #savanna | 18:58 | |
| *** NikitaKonovalov has joined #savanna | 19:02 | |
| *** Nadya_ has joined #savanna | 19:05 | |
| *** Nadya has quit IRC | 19:06 | |
| tmckayrh | hmmm, my job origin api methods on https://etherpad.openstack.org/savanna_API_draft_EDP_extensions are ambiguous for "copy to destination" | 19:06 |
| tmckayrh | GET /v1.1/{tenant_id}/job-origins/{job_origin_id} # Fetch details about job origin | 19:07 |
| *** NikitaKonovalov has quit IRC | 19:07 | |
| tmckayrh | GET /v1.1/tenant_id}/job-origins/{job_origin_id} # Copy a job from storage to a specified desination | 19:07 |
| tmckayrh | That's not going to work | 19:07 |
| tmckayrh | SergeyLukjanov. ruhe, Nadya_, crobertsrh, how about this? | 19:08 |
| * tmckayrh drumroll | 19:08 | |
| tmckayrh | POST /v1.1/tenant_id}/job-origins/{job_origin_id} # Takes dest url as json data, copy a job from storage to a specified desination | 19:09 |
| Nadya_ | I need to read irc logs :) | 19:10 |
| crobertsrh | So, this would be an action that could be performed on an already defined job_origin? | 19:10 |
| crobertsrh | Or is this something that would only be done as part of a job execution at execute time? | 19:10 |
| tmckayrh | Yes. It kind of implies that "update" is not supported. "update" would have be delete and recreate | 19:10 |
| tmckayrh | Well, ultimately its point is execution. | 19:11 |
| tmckayrh | You have to get the job binary from storage to hdfs where it can be run. | 19:11 |
| tmckayrh | But I don't have a way yet to say "copy" | 19:11 |
| tmckayrh | I suppose it could be POST /v1.1/tenant_id}/job-origins/copy/{job_origin_id} | 19:12 |
| crobertsrh | I'm ok with the REST call. I'm just wondering what, if anything in the UI winds-up using it. | 19:12 |
| tmckayrh | to be more explicit | 19:12 |
| tmckayrh | crobertsrh, I'm not sure the UI needs it. | 19:12 |
| tmckayrh | If someone really wants to suck job binaries out of storage via savanna, they can use http from the command line | 19:13 |
| tmckayrh | not sure there is a UI use case for it | 19:13 |
| crobertsrh | Right....just making sure that I understood things correctly | 19:13 |
| crobertsrh | Maybe this could mirror the tagging of an image API | 19:13 |
| tmckayrh | as a web kind of guy, what do you think -- include the extra word to be more self documenting? | 19:13 |
| crobertsrh | image/{image_id}/tag where POST data is the tag | 19:14 |
| tmckayrh | because POST /v1.1/tenant_id}/job-origins/{job_origin_id} could someday legitimately be a mod to an existing record | 19:14 |
| crobertsrh | job_origin/{job_origin_id}/copy POST is the URL | 19:14 |
| tmckayrh | ie, change the credentials or the URI, etc | 19:14 |
| tmckayrh | crobertsrh, +1 | 19:15 |
| *** sacharya has quit IRC | 19:15 | |
| *** sacharya has joined #savanna | 19:16 | |
| tmckayrh | crobertsrh, okay, changed the pad to POST /v1.1/{tenant_id}/job-origins/{job_origin_id}/copy # Copy a job from storage to a specified desination (url) | 19:16 |
| * tmckayrh feels powerful because savanna is responding to my urls with canned strings :) | 19:18 | |
| crobertsrh | cool :) | 19:19 |
| tmckayrh | walk before we run, I always say ;-) | 19:19 |
| Nadya_ | I have one concern about this. We need to create user-specific directory in hdfs for every job | 19:20 |
| crobertsrh | Yeah, that's what I've been doing on the UI side. I have ugly impls for jobs, job origins and data sources. | 19:20 |
| tmckayrh | Nadya_, please elaborate | 19:21 |
| Nadya_ | I do not have strong opinion about this, just to remind this :) not sure that tenant and job_origin_id is enough to determine user-specific directory | 19:22 |
| Nadya_ | but maybe it isok for now...It means that if we use shared cluster (not specific for job) all users from tenant has access to jobs | 19:24 |
| Nadya_ | is my point clear? | 19:24 |
| tmckayrh | ah, so we do not have a way to scope the visibility of jobs to a particular user? | 19:25 |
| Nadya_ | yes | 19:25 |
| tmckayrh | in other words, you can see all the job binaries that I define in the system, and I can see yours, as long as we each have a token for the same tenant | 19:25 |
| tmckayrh | hmmm | 19:26 |
| tmckayrh | Nadya_, off the top of my head, it sounds like some kind of ACL or permissions associated with the job binary | 19:26 |
| tmckayrh | I wonder if openstack has a general solution to this? | 19:27 |
| Nadya_ | unfortunately I dont know. we may discuss this on tomorrow meeting | 19:27 |
| tmckayrh | swift might have that, with the "account" concept | 19:27 |
| tmckayrh | sql storage probably does.... | 19:27 |
| tmckayrh | I wonder what hdfs has... | 19:27 |
| tmckayrh | good idea | 19:28 |
| tmckayrh | For the moment I will assume we are all nice and like to share :) | 19:28 |
| Nadya_ | I'm ok with it | 19:28 |
| tmckayrh | Nadya_, btw, I looked at your proposal on the pad, it looks good to me. I will look again more closely before tomorrow's meeting | 19:29 |
| *** ruhe has quit IRC | 19:30 | |
| tmckayrh | crobertsrh, party tricks :) http://fpaste.org/30665/13759038/ | 19:31 |
| SergeyLukjanov | btw currently we have an OpenStack-style ACL | 19:32 |
| SergeyLukjanov | with only tenant-specific objects | 19:32 |
| SergeyLukjanov | I think that we should add much more complex scoping especially for EDP needs | 19:33 |
| tmckayrh | SergeyLukjanov, so the assumption is that if you are authorized for the tenant, it's okay that you can see all the jobs? | 19:33 |
| tmckayrh | ah, I see. now ====> future | 19:33 |
| SergeyLukjanov | tmckayrh, for now it's ok | 19:34 |
| Nadya_ | yes, let it be per tenant | 19:34 |
| tmckayrh | brb | 19:34 |
| *** IlyaE has quit IRC | 19:36 | |
| SergeyLukjanov | I have some thoughts about scoping and it looks like that we have 2 categories of permissions - read and write, I mean, that we need to be able to specify how can read data and how can write | 19:36 |
| SergeyLukjanov | so for both of them we can have several scopes - creator, creator's tenant and public | 19:37 |
| Nadya_ | what component do you mean\? | 19:38 |
| *** IlyaE has joined #savanna | 19:46 | |
| SergeyLukjanov | Nadya_, is it a question for me?^) | 19:51 |
| Nadya_ | SergeyLukjanov,yes | 19:51 |
| SergeyLukjanov | I mean that all savanna objects could have such scoping options | 19:53 |
| SergeyLukjanov | we need to discuss it, but theoretically it's possible :) | 19:54 |
| *** IlyaE has quit IRC | 19:57 | |
| *** NikitaKonovalov has joined #savanna | 20:02 | |
| *** NikitaKonovalov has quit IRC | 20:07 | |
| *** gkleiman has quit IRC | 20:17 | |
| *** Nadya_ has quit IRC | 20:26 | |
| *** dina_belova has quit IRC | 20:34 | |
| *** IlyaE has joined #savanna | 20:39 | |
| *** IlyaE has quit IRC | 20:41 | |
| *** crobertsrh is now known as _crobertsrh | 20:52 | |
| *** NikitaKonovalov has joined #savanna | 21:03 | |
| *** NikitaKonovalov has quit IRC | 21:07 | |
| *** SergeyLukjanov has quit IRC | 21:11 | |
| *** mattf is now known as _mattf | 21:35 | |
| *** _mattf is now known as mattf | 21:43 | |
| *** dina_belova has joined #savanna | 21:44 | |
| *** tstclair is now known as _tstclair | 21:45 | |
| *** dina_belova has quit IRC | 21:47 | |
| *** NikitaKonovalov has joined #savanna | 22:03 | |
| *** NikitaKonovalov has quit IRC | 22:08 | |
| *** sacharya has quit IRC | 22:16 | |
| *** NikitaKonovalov has joined #savanna | 23:04 | |
| *** NikitaKonovalov has quit IRC | 23:09 | |
| *** sacharya has joined #savanna | 23:24 | |
| *** sacharya has quit IRC | 23:24 | |
| *** IlyaE has joined #savanna | 23:25 | |
| *** rnirmal has quit IRC | 23:45 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!