| *** NikitaKonovalov has joined #savanna | 00:31 | |
| *** NikitaKonovalov has quit IRC | 00:35 | |
| *** aignatov_ has quit IRC | 00:46 | |
| *** dina_belova has joined #savanna | 00:54 | |
| *** dina_belova has quit IRC | 00:58 | |
| *** benl has quit IRC | 01:08 | |
| *** aignatov has quit IRC | 01:10 | |
| *** aignatov has joined #savanna | 01:11 | |
| *** NikitaKonovalov has joined #savanna | 01:31 | |
| *** NikitaKonovalov has quit IRC | 01:36 | |
| *** benl has joined #savanna | 01:39 | |
| *** benl has quit IRC | 01:48 | |
| *** dina_belova has joined #savanna | 01:54 | |
| *** dina_belova has quit IRC | 01:59 | |
| *** NikitaKonovalov has joined #savanna | 02:32 | |
| *** NikitaKonovalov has quit IRC | 02:36 | |
| *** IlyaE has joined #savanna | 02:50 | |
| *** dina_belova has joined #savanna | 02:55 | |
| *** shredder12 has quit IRC | 02:59 | |
| *** mattf is now known as _mattf | 03:00 | |
| *** dina_belova has quit IRC | 03:00 | |
| *** NikitaKonovalov has joined #savanna | 03:32 | |
| *** NikitaKonovalov has quit IRC | 03:37 | |
| *** SergeyLukjanov has joined #savanna | 03:46 | |
| *** dina_belova has joined #savanna | 03:55 | |
| *** dina_belova has quit IRC | 04:00 | |
| *** SergeyLukjanov has quit IRC | 04:11 | |
| *** dina_belova has joined #savanna | 04:14 | |
| *** dina_belova has quit IRC | 04:29 | |
| *** NikitaKonovalov has joined #savanna | 04:33 | |
| *** NikitaKonovalov has quit IRC | 04:38 | |
| *** aignatov_ has joined #savanna | 04:44 | |
| *** aignatov_ has quit IRC | 05:03 | |
| *** Nadya has joined #savanna | 05:22 | |
| *** bmahe has joined #savanna | 05:33 | |
| *** Nadya has quit IRC | 05:39 | |
| *** IlyaE has quit IRC | 06:11 | |
| *** bmahe has quit IRC | 06:20 | |
| *** IlyaE has joined #savanna | 06:21 | |
| *** bmahe has joined #savanna | 06:33 | |
| *** NikitaKonovalov has joined #savanna | 06:34 | |
| *** NikitaKonovalov has quit IRC | 06:34 | |
| *** NikitaKonovalov has joined #savanna | 06:36 | |
| *** NikitaKonovalov has quit IRC | 06:36 | |
| *** NikitaKonovalov has joined #savanna | 06:37 | |
| *** IlyaE has quit IRC | 07:04 | |
| *** Nadya has joined #savanna | 07:12 | |
| *** dina_belova has joined #savanna | 07:15 | |
| *** dina_belova has quit IRC | 07:16 | |
| *** Nadya has quit IRC | 07:26 | |
| *** NikitaKonovalov has quit IRC | 07:27 | |
| *** NikitaKonovalov has joined #savanna | 07:45 | |
| *** SergeyLukjanov has joined #savanna | 07:49 | |
| *** nmakhotkin has joined #savanna | 07:52 | |
| *** dmitryme has joined #savanna | 08:01 | |
| *** nmakhotkin has quit IRC | 08:02 | |
| *** dina_belova has joined #savanna | 08:08 | |
| openstackgerrit | A change was merged to stackforge/savanna: Implement to_dict() method for Resource https://review.openstack.org/40904 | 08:27 |
|---|---|---|
| openstackgerrit | Yaroslav Lobankov proposed a change to stackforge/savanna: Integration test updating for "HDP" plugin https://review.openstack.org/40369 | 08:30 |
| *** nprivalova has joined #savanna | 08:37 | |
| *** dina_belova has quit IRC | 08:55 | |
| *** dmitryme_ has joined #savanna | 09:00 | |
| *** dmitryme has quit IRC | 09:01 | |
| *** dmitryme_ is now known as dmitryme | 09:01 | |
| *** dina_belova has joined #savanna | 09:14 | |
| *** dina_belova has quit IRC | 09:14 | |
| *** dina_belova has joined #savanna | 09:20 | |
| *** dina_belova has quit IRC | 09:36 | |
| *** NikitaKonovalov has quit IRC | 09:37 | |
| *** NikitaKonovalov has joined #savanna | 09:42 | |
| openstackgerrit | Alexander Kuznetsov proposed a change to stackforge/savanna: Added first version of model for EDP https://review.openstack.org/40227 | 09:48 |
| openstackgerrit | Alexander Kuznetsov proposed a change to stackforge/savanna: Added conductor API for DataSource object https://review.openstack.org/40591 | 09:48 |
| *** bmahe has quit IRC | 09:52 | |
| *** NikitaKonovalov has quit IRC | 10:03 | |
| *** dina_belova has joined #savanna | 10:13 | |
| *** dina_belova has quit IRC | 10:15 | |
| *** dina_belova has joined #savanna | 10:15 | |
| *** nprivalova has quit IRC | 10:24 | |
| *** dina_belova has quit IRC | 10:27 | |
| *** nprivalova has joined #savanna | 10:28 | |
| openstackgerrit | Sergey Reshetnyak proposed a change to stackforge/savanna: Refactoring hdp plugin https://review.openstack.org/39733 | 10:34 |
| *** nprivalova has quit IRC | 10:40 | |
| *** dina_belova has joined #savanna | 10:41 | |
| *** dina_belova has quit IRC | 10:41 | |
| *** NikitaKonovalov has joined #savanna | 10:51 | |
| *** NikitaKonovalov has quit IRC | 10:55 | |
| *** NikitaKonovalov has joined #savanna | 11:01 | |
| *** dina_belova has joined #savanna | 11:06 | |
| *** nprivalova has joined #savanna | 11:06 | |
| *** NikitaKonovalov has quit IRC | 11:12 | |
| *** NikitaKonovalov has joined #savanna | 11:14 | |
| *** benl_ has joined #savanna | 11:22 | |
| *** akuznetsov has joined #savanna | 11:31 | |
| *** dina_belova has quit IRC | 11:33 | |
| *** dmitryme has quit IRC | 11:34 | |
| *** dina_belova has joined #savanna | 11:42 | |
| *** ruhe has joined #savanna | 11:45 | |
| *** tmckayrh has quit IRC | 11:49 | |
| openstackgerrit | Nikita Konovalov proposed a change to stackforge/savanna: Migration to Conductor https://review.openstack.org/40902 | 11:52 |
| *** akuznetsov has quit IRC | 11:57 | |
| *** dina_belova has quit IRC | 11:57 | |
| *** NikitaKonovalov has quit IRC | 11:57 | |
| *** ruhe has quit IRC | 12:01 | |
| *** SergeyLukjanov has quit IRC | 12:08 | |
| *** dmitryme has joined #savanna | 12:14 | |
| openstackgerrit | A change was merged to stackforge/savanna: Added first version of model for EDP https://review.openstack.org/40227 | 12:21 |
| *** nprivalova has quit IRC | 12:26 | |
| *** dina_belova has joined #savanna | 12:28 | |
| *** nprivalova has joined #savanna | 12:31 | |
| *** ruhe has joined #savanna | 12:31 | |
| *** _mattf is now known as mattf | 12:31 | |
| *** dina_belova has quit IRC | 12:36 | |
| *** dmitryme_ has joined #savanna | 12:49 | |
| *** dmitryme has quit IRC | 12:52 | |
| *** dmitryme_ is now known as dmitryme | 12:53 | |
| *** NikitaKonovalov has joined #savanna | 12:54 | |
| *** ruhe has quit IRC | 12:58 | |
| *** NikitaKonovalov has quit IRC | 12:59 | |
| openstackgerrit | Jonathan Maron proposed a change to stackforge/savanna: Made Ambari RPM location configurable https://review.openstack.org/40479 | 13:05 |
| *** nprivalova has quit IRC | 13:27 | |
| *** _tstclair is now known as tstclair | 13:30 | |
| *** tmckayrh has joined #savanna | 13:38 | |
| tmckayrh | hello savanna folks. I have a question on debugging savanna-ci results. I have a failure on https://review.openstack.org/40936 and I've grepped the console log for "fail", "debug", "error" and I can't find the reason for the failure. Any suggestions on how to understand what savanna-ci is telling me? | 13:40 |
| * tmckayrh grabs snacks | 13:40 | |
| dmitryme | tmckayrh: As you see, CI consists of 5 different jobs, of which two fail | 13:44 |
| dmitryme | don't mind coverage job for now | 13:44 |
| tmckayrh | ah, maybe all of this wasn't my fault. File "/var/lib/jenkins/jobs/gate-savanna-coverage/workspace/savanna/db/model_base.py", line 19, in <module> from sqlalchemy.ext import declarative ImportError: No module named ext | 13:45 |
| dmitryme | it is experimental and actually should never fail | 13:45 |
| dmitryme | hmm, where do you see output you've just posted? | 13:46 |
| tmckayrh | that's from coverage, actually.... hold on | 13:46 |
| dmitryme | forget about coverage | 13:46 |
| dmitryme | it is broken | 13:46 |
| tmckayrh | okay. | 13:46 |
| dmitryme | that leaves only integration tests | 13:46 |
| dmitryme | to understand, what is failing go to jenkins via the link | 13:47 |
| tmckayrh | okay http://jenkins.savanna.mirantis.com/job/gate-savanna-integration/532/console | 13:47 |
| dmitryme | in the menu on left click on 'Console Output' [raw] | 13:48 |
| dmitryme | specifically, click on [raw] | 13:48 |
| tmckayrh | it ends with what looks like a failure on "test_swift". | 13:48 |
| dmitryme | yep, that is it | 13:48 |
| tmckayrh | I did change an EDP enum value of "Swift" to lowercase but that seems not likely... still | 13:49 |
| dmitryme | actually that is a known problem, timeouts for that job are not finely tuned yet, so it fails from time to time | 13:49 |
| *** nprivalova has joined #savanna | 13:49 | |
| dmitryme | EDP should not affect the test | 13:49 |
| tmckayrh | dmitryme, can I run the same test from my dev environment? I'd like to see the test log itself | 13:50 |
| dmitryme | at all | 13:50 |
| dmitryme | I don't think it is your fault swift test failed | 13:50 |
| dmitryme | as I said, it fails from time to time | 13:50 |
| tmckayrh | okay. this is what I find confusing... there is an error here, but no pointer to a detailed log | 13:51 |
| dmitryme | as for running tests manually, I don't know how to do that | 13:51 |
| tmckayrh | I have to repatch anyway, I'll upload a new change set and see what happens. Thanks. | 13:51 |
| dmitryme | and I don't see any folks who know around | 13:51 |
| dmitryme | yep, I think that is the best approach | 13:51 |
| tmckayrh | I appreciate the help. | 13:52 |
| dmitryme | no problem :-) | 13:52 |
| tmckayrh | nprivalova, ping | 13:52 |
| *** nprivalova has quit IRC | 14:01 | |
| *** dmitryme has quit IRC | 14:08 | |
| *** benl_ has quit IRC | 14:08 | |
| *** dmitryme has joined #savanna | 14:09 | |
| *** alazarev has joined #savanna | 14:12 | |
| *** benl_ has joined #savanna | 14:12 | |
| *** akuznetsov has joined #savanna | 14:15 | |
| *** nprivalova has joined #savanna | 14:16 | |
| nprivalova | hi tmckayrh | 14:16 |
| *** ruhe has joined #savanna | 14:16 | |
| tmckayrh | akuznetsov, I saw your note on credentials for edp -- nprivalova also mentioned alternate auth methods yesterday. I am looking at how to use "oneof" in a json schema so that we can have different credential forms but still validate | 14:17 |
| tmckayrh | that would be ideal | 14:17 |
| tmckayrh | If I can't make it work, I'll just loosen the restrictions | 14:17 |
| nprivalova | tmckayrh, I think we should consider only user+password case | 14:19 |
| *** alazarev has left #savanna | 14:19 | |
| akuznetsov | tmckayrh I think for now it will enough to object type for this field, as for data source component | 14:20 |
| nprivalova | tmckayrh, I told you about trust-mechanism but we need to investigate it first | 14:21 |
| tmckayrh | nprivalova, well, proof of concept then? I can always leave "oneof" with a single option, for later expansion. akuznetsov mentioned "what about tokens in the future?" as you did. | 14:21 |
| *** alazarev has joined #savanna | 14:21 | |
| *** alazarev has quit IRC | 14:21 | |
| *** benl_ has quit IRC | 14:24 | |
| nprivalova | tmckayrh, proof, yes | 14:26 |
| *** ruhe has quit IRC | 14:30 | |
| *** IlyaE has joined #savanna | 14:32 | |
| *** benl_ has joined #savanna | 14:33 | |
| *** ruhe has joined #savanna | 14:37 | |
| openstackgerrit | Nikolay Mahotkin proposed a change to stackforge/savanna: Added Hive configuration, new nodeprocess - hiveserver https://review.openstack.org/40847 | 14:48 |
| *** akuznetsov has quit IRC | 14:49 | |
| aignatov | tmckayrh, ping | 14:57 |
| aignatov | could you take a look at my patch https://review.openstack.org/#/c/40590/ | 14:58 |
| tmckayrh | aignatov, yes, sure | 14:59 |
| tmckayrh | otp at the moment | 15:00 |
| *** benl_ has quit IRC | 15:02 | |
| * tmckayrh has triumped with the "oneOf" keyword in json | 15:02 | |
| tmckayrh | triumphed even | 15:02 |
| openstackgerrit | Nikolay Mahotkin proposed a change to stackforge/savanna: Added Hive configuration, new nodeprocess - hiveserver https://review.openstack.org/40847 | 15:02 |
| tmckayrh | aignatov, oops, I missed the email where I was assigned to be a reviewer :) I should change my mail filter... | 15:03 |
| aignatov | not a problem, anyway I'll ping you if needed ^_^ | 15:09 |
| *** bogdando has quit IRC | 15:23 | |
| *** sacharya has joined #savanna | 15:26 | |
| openstackgerrit | Nadya Privalova proposed a change to stackforge/savanna: Added conductor API for Job Object https://review.openstack.org/41119 | 15:27 |
| openstackgerrit | Nadya Privalova proposed a change to stackforge/savanna: Added conductor API for Job Object https://review.openstack.org/41119 | 15:31 |
| openstackgerrit | Nadya Privalova proposed a change to stackforge/savanna: Added conductor API for Job Object https://review.openstack.org/41119 | 15:37 |
| *** rnirmal has joined #savanna | 15:38 | |
| openstackgerrit | Dmitry Mescheryakov proposed a change to stackforge/savanna: Conductor objects are re-populated on update https://review.openstack.org/41122 | 15:38 |
| sacharya | Hi all | 15:42 |
| tmckayrh | hi | 15:42 |
| sacharya | Is there a more flushed out version of this https://etherpad.openstack.org/savanna_API_draft_EDP_extensions | 15:42 |
| tmckayrh | sacharya, work in progress. In fact, I am just committing the JobOrigin api now. The best doc at the moment is code | 15:43 |
| tmckayrh | sacharya, do you have a specific question? Maybe I know the answer :) | 15:44 |
| openstackgerrit | Trevor McKay proposed a change to stackforge/savanna: Add initial version of the REST api for the job origin component https://review.openstack.org/40936 | 15:44 |
| sacharya | Its hard to say without looking at the request response for each of those ;) | 15:44 |
| sacharya | There are so many different ways of executing a job in Hadoop | 15:45 |
| tmckayrh | sacharya, yes, I see your point :) Okay, so the response will be dependent on the database object that is defined. I am about to start working on defining the JobOrigin object, I think the other edp objects do not have database models yet either. | 15:46 |
| tmckayrh | as you can see. the EDP work in savanna is very new | 15:46 |
| sacharya | Yeah, thats fine. I just wanted to make sure I was not missing some secret doc somewhere | 15:47 |
| aignatov | btw, sacharya, here is another doc https://etherpad.openstack.org/edp_v3_components | 15:47 |
| tmckayrh | nope :) We are zeroing in on what reality will be | 15:47 |
| sacharya | Thanks aignatov | 15:48 |
| aignatov | regarding different ways of job executing, we'll try to support most popular ones - pig, hive, jar files) | 15:48 |
| sacharya | how about streaming ? | 15:49 |
| sacharya | python scripts | 15:49 |
| *** nprivalova has quit IRC | 15:49 | |
| aignatov | not sure about python scrits, but streaming and pips should be there :) | 15:50 |
| aignatov | *pipes | 15:50 |
| sacharya | that will cover it | 15:50 |
| aignatov | ah, ok | 15:50 |
| sacharya | stupid question tmckayrh | 15:52 |
| sacharya | what is a data-source? say a swift endpoint or a particular container in swift that will act as a hadoop input ? | 15:52 |
| tmckayrh | sacharya, not a stupid question :) A data-source is a particular file, for lack of a better term. Maybe a dir, for data (I could imagine processing a whole dir) | 15:53 |
| *** akuznetsov has joined #savanna | 15:53 | |
| aignatov | i think is's a particular container | 15:53 |
| sacharya | In that case, do we need another api call to list supported data-source types eg swift, hdfs, db etc? | 15:55 |
| tmckayrh | sacharya, could be something other than swift (hdfs). But in swift terms, I would think it was a container or an object. Same for output. | 15:55 |
| tmckayrh | sacharya, we have talked about that -- an ability to browse "endpoints" that the system currently knows about | 15:56 |
| tmckayrh | to answer the question "hey, I'm new here. Where can I find stuff to process, and where can I put my results?" | 15:56 |
| tmckayrh | similar question for "where can I store my job binaries anyway?" | 15:57 |
| tmckayrh | not strictly necessary at first during development -- you just "know" what you can use, or ask the site admin :) | 15:57 |
| sacharya | yeah, i agree | 15:58 |
| tmckayrh | sacharya, feel free to post comments on the etherpad -- that's not a formal doc | 15:58 |
| tmckayrh | just throw up an irc nick or an email so we can find you again :) | 15:59 |
| sacharya | ok sure | 15:59 |
| aignatov | right, but don't forget to add tag with your nickname) | 15:59 |
| * tmckayrh wonders if he has done that everywhere.... | 15:59 | |
| aignatov | haha :) | 16:00 |
| *** akuznetsov has quit IRC | 16:00 | |
| aignatov | tmckayrh, I have to go now, but I'll look at your patchset more deeply on the weekend) | 16:01 |
| tmckayrh | aignatov, okay, thanks. I'm going to start working on the db object definition.... | 16:01 |
| *** ruhe has quit IRC | 16:02 | |
| *** benl_ has joined #savanna | 16:02 | |
| *** benl_ has quit IRC | 16:09 | |
| openstackgerrit | Nikolay Mahotkin proposed a change to stackforge/savanna: Added Hive configuration, new nodeprocess - hiveserver https://review.openstack.org/40847 | 16:11 |
| *** dmitryme has quit IRC | 16:11 | |
| sacharya | tmckayrth, i am assuming these job apis can be configured before user has a cluster? only the execute action needs a cluster. | 16:21 |
| sacharya | tmckayrh | 16:21 |
| tmckayrh | sacharya, yes, that sounds right | 16:26 |
| sacharya | i was wondering why you need to store the job binaries in database, that looks like the reason. | 16:26 |
| tmckayrh | sacharya, we are postulating that jobs (binaries at first) can be stored in swift or in the savanna sqlite database, or an external hdfs and then will be moved to the hadoop cluster for execution | 16:27 |
| tmckayrh | yes, the db stores the location information so that savanna can be used to move things around later | 16:28 |
| sacharya | makes sense | 16:29 |
| tmckayrh | 2 things going on there -- 1) the job record, which contains a uri of some sort and 2) in some instances, the actual job binary itself will also be in the db (but the preferred method is to have it somewhere else) | 16:29 |
| *** ruhe has joined #savanna | 16:52 | |
| *** ruhe has quit IRC | 16:52 | |
| *** ruhe has joined #savanna | 17:00 | |
| *** ruhe has quit IRC | 17:05 | |
| *** IlyaE has quit IRC | 17:12 | |
| openstackgerrit | John Speidel proposed a change to stackforge/savanna: Allow Ambari users to be specified in configuration https://review.openstack.org/41146 | 17:14 |
| openstackgerrit | John Speidel proposed a change to stackforge/savanna: Allow Ambari users to be specified in configuration https://review.openstack.org/41146 | 17:29 |
| *** akuznetsov has joined #savanna | 17:38 | |
| *** akuznetsov has quit IRC | 17:49 | |
| *** IlyaE has joined #savanna | 17:55 | |
| *** benl_ has joined #savanna | 17:58 | |
| *** IlyaE has quit IRC | 18:02 | |
| *** dmitryme has joined #savanna | 18:03 | |
| *** Nadya has joined #savanna | 18:04 | |
| *** akuznetsov has joined #savanna | 18:04 | |
| mattf | so much for back to talk in the AM ET. sorry folks, i've been otp all day. | 18:06 |
| *** dmitryme has quit IRC | 18:25 | |
| *** dmitryme has joined #savanna | 18:34 | |
| *** akuznetsov has quit IRC | 18:40 | |
| *** hub_cap has left #savanna | 18:51 | |
| *** tmckayrh has quit IRC | 18:51 | |
| *** dmitryme has quit IRC | 19:45 | |
| *** tmckayrh has joined #savanna | 19:52 | |
| *** benl_ has quit IRC | 20:21 | |
| *** dmitryme has joined #savanna | 20:32 | |
| *** IlyaE has joined #savanna | 20:38 | |
| *** benl_ has joined #savanna | 20:46 | |
| *** rnirmal has quit IRC | 21:24 | |
| *** tstclair is now known as _tstclair | 21:36 | |
| *** sacharya has quit IRC | 21:40 | |
| *** dmitryme has quit IRC | 21:43 | |
| tmckayrh | anyone around handy with sqlalchemy and know why I get this? I added a table, but apparently not completely :) | 21:52 |
| tmckayrh | File "/home/tmckay/src/savanna/.tox/venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1397, in add | 21:52 |
| tmckayrh | raise exc.UnmappedInstanceError(instance) | 21:52 |
| tmckayrh | UnmappedInstanceError: Class 'savanna.db.models_edp.JobOrigin' is not mapped | 21:52 |
| tmckayrh | :) got it. Forgot to derive from SavannaBase | 21:57 |
| tmckayrh | __tablename__ attribute was missing | 21:57 |
| Nadya | tmckayrh, I believe you forget to add JobOrigin in one of 4(or 6) layers... | 21:57 |
| Nadya | oh. ok :) | 21:57 |
| tmckayrh | 4 or 6 layers? uh oh :) | 21:57 |
| tmckayrh | So, I apparently created a record. | 21:57 |
| tmckayrh | I wonder if it's persisting | 21:58 |
| Nadya | look at patches from me or from Alex K | 21:59 |
| Nadya | there is tests on models | 21:59 |
| tmckayrh | ah, okay, thanks | 21:59 |
| Nadya | I think it may help | 21:59 |
| tmckayrh | It's late there.... | 22:00 |
| tmckayrh | or early :) | 22:01 |
| Nadya | late...:) | 22:09 |
| tmckayrh | is there a controlled way to stop savanna-api running in tox, or is it ctrl-c safe? | 22:13 |
| tmckayrh | my firefox SQLiteManager plugin is showing the JobOrigin, yay | 22:14 |
| tmckayrh | bbl maybe | 22:21 |
| tmckayrh | _tmckayrh | 22:21 |
| Nadya | ctrl-c is perfect! | 22:43 |
| *** _tstclair has quit IRC | 22:47 | |
| *** Nadya has quit IRC | 22:47 | |
| *** IlyaE has quit IRC | 22:49 | |
| *** _tstclair has joined #savanna | 22:52 | |
| *** _tstclair has quit IRC | 22:57 | |
| *** IlyaE has joined #savanna | 23:08 | |
| *** benl_ has quit IRC | 23:09 | |
| *** tstclair has joined #savanna | 23:18 | |
| *** IlyaE has quit IRC | 23:35 | |
| *** sacharya has joined #savanna | 23:47 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!