| *** matsuhashi has joined #savanna | 00:48 | |
| *** sacharya has joined #savanna | 00:56 | |
| *** IlyaE has joined #savanna | 00:59 | |
| *** IlyaE has quit IRC | 01:01 | |
| *** nosnos has joined #savanna | 01:05 | |
| *** sacharya has quit IRC | 01:16 | |
| *** sacharya has joined #savanna | 01:19 | |
| *** SergeyLukjanov has joined #savanna | 03:33 | |
| *** sacharya has quit IRC | 03:53 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 03:58 | |
| *** sacharya has joined #savanna | 03:58 | |
| *** _SergeyLukjanov has quit IRC | 03:58 | |
| *** sacharya has quit IRC | 04:45 | |
| *** matsuhashi has quit IRC | 05:11 | |
| *** SergeyLukjanov has joined #savanna | 05:11 | |
| *** matsuhashi has joined #savanna | 05:11 | |
| *** matsuhashi has quit IRC | 05:13 | |
| *** matsuhashi has joined #savanna | 05:14 | |
| *** matsuhashi has quit IRC | 05:22 | |
| *** matsuhashi has joined #savanna | 05:23 | |
| *** matsuhashi has quit IRC | 05:27 | |
| *** matsuhashi has joined #savanna | 05:34 | |
| *** akuznetsov has quit IRC | 07:28 | |
| *** dafter has joined #savanna | 07:38 | |
| *** dafter has joined #savanna | 07:38 | |
| *** SergeyLukjanov has quit IRC | 07:46 | |
| *** DinaBelova has joined #savanna | 08:03 | |
| *** akuznetsov has joined #savanna | 08:14 | |
| *** matsuhas_ has joined #savanna | 08:14 | |
| *** matsuhashi has quit IRC | 08:15 | |
| *** tvb|afk has joined #savanna | 08:15 | |
| *** matsuhashi has joined #savanna | 08:15 | |
| *** dafter has quit IRC | 08:19 | |
| *** matsuhas_ has quit IRC | 08:19 | |
| *** tvb|afk has quit IRC | 08:49 | |
| *** DinaBelova has quit IRC | 09:08 | |
| *** tmckay has joined #savanna | 09:30 | |
| *** openstackgerrit has quit IRC | 09:38 | |
| *** dafter has joined #savanna | 09:42 | |
| *** dafter has quit IRC | 09:43 | |
| *** DinaBelova has joined #savanna | 09:45 | |
| *** tvb|afk has joined #savanna | 09:49 | |
| *** akuznetsov has quit IRC | 10:09 | |
| *** akuznetsov has joined #savanna | 10:10 | |
| *** nprivalova has quit IRC | 10:22 | |
| *** DinaBelova has quit IRC | 10:27 | |
| *** nosnos has quit IRC | 10:28 | |
| *** nosnos has joined #savanna | 10:28 | |
| *** nosnos has quit IRC | 10:33 | |
| *** tvb|afk has quit IRC | 10:37 | |
| *** dafter has joined #savanna | 10:37 | |
| *** tmckay has quit IRC | 10:42 | |
| *** tvb|afk has joined #savanna | 10:42 | |
| *** dafter has quit IRC | 10:43 | |
| *** nprivalova has joined #savanna | 10:52 | |
| *** akuznetsov has quit IRC | 10:53 | |
| *** DinaBelova has joined #savanna | 10:59 | |
| *** tvb|afk has quit IRC | 11:01 | |
| *** dafter has joined #savanna | 11:02 | |
| *** nprivalova has quit IRC | 11:20 | |
| *** dmitryme has joined #savanna | 11:27 | |
| *** tvb|afk has joined #savanna | 11:28 | |
| *** dafter_ has joined #savanna | 11:29 | |
| *** dafter_ has joined #savanna | 11:30 | |
| *** dafter has quit IRC | 11:32 | |
| *** dafter_ has quit IRC | 11:32 | |
| *** tvb|afk has quit IRC | 11:33 | |
| *** nprivalova has joined #savanna | 11:36 | |
| *** dmitryme has quit IRC | 11:38 | |
| *** akuznetsov has joined #savanna | 11:51 | |
| *** dafter has joined #savanna | 11:52 | |
| *** tvb|afk has joined #savanna | 11:57 | |
| *** tvb|afk has joined #savanna | 11:57 | |
| *** akuznetsov has quit IRC | 12:00 | |
| *** dafter has quit IRC | 12:00 | |
| *** dmitryme has joined #savanna | 12:09 | |
| *** nprivalova has quit IRC | 12:13 | |
| *** nprivalova has joined #savanna | 12:21 | |
| *** akuznetsov has joined #savanna | 12:34 | |
| *** openstackgerrit has joined #savanna | 13:02 | |
| *** tvb|afk has quit IRC | 13:04 | |
| *** dafter has joined #savanna | 13:07 | |
| *** ruhe has joined #savanna | 13:11 | |
| *** _crobertsrh is now known as crobertsrh | 13:13 | |
| *** dafter has quit IRC | 13:15 | |
| *** dafter has joined #savanna | 13:15 | |
| *** tvb|afk has joined #savanna | 13:18 | |
| *** dafter has quit IRC | 13:20 | |
| *** dmitryme has quit IRC | 13:26 | |
| *** tmckay has joined #savanna | 13:28 | |
| *** tmckay has left #savanna | 13:28 | |
| *** tmckay has joined #savanna | 13:29 | |
| *** ruhe has quit IRC | 13:29 | |
| *** ruhe has joined #savanna | 13:30 | |
| tmckay | So I was starting up a cluster from savanna. The instances were created (1 master, 1 worker) and the cluster was in the Waiting status for a long time. Eventually it went to error. When I use the REST api to retrieve the cluster info, it contains this: | 13:32 |
|---|---|---|
| tmckay | "status": "Error", | 13:33 |
| tmckay | "status_description": "Unauthorized (HTTP 401)", | 13:33 |
| tmckay | Does anybody know what this means? | 13:33 |
| tmckay | crobertsrh, this is where I ended up last evening ^^ | 13:33 |
| * tmckay goes to look through devstack logs | 13:33 | |
| crobertsrh | I haven't seen that one | 13:38 |
| tmckay | something is unauthorized, but I don't know what... | 13:40 |
| tmckay | maybe I should try again, it's been looping all night. | 13:40 |
| crobertsrh | security rules set up to allow ssh? | 13:41 |
| *** ruhe has quit IRC | 13:43 | |
| *** DinaBelova has quit IRC | 13:46 | |
| *** ruhe has joined #savanna | 13:49 | |
| *** tmckay has quit IRC | 13:51 | |
| aignatov | tmckay, may be token is expired? | 13:54 |
| *** nprivalova is now known as nadya | 13:56 | |
| *** dmitryme has joined #savanna | 14:03 | |
| *** ruhe has quit IRC | 14:06 | |
| *** ruhe has joined #savanna | 14:11 | |
| *** tmckay has joined #savanna | 14:15 | |
| *** SergeyLukjanov has joined #savanna | 14:16 | |
| *** akuznetsov has quit IRC | 14:24 | |
| *** sacharya has joined #savanna | 14:27 | |
| *** SergeyLukjanov has quit IRC | 14:32 | |
| *** ruhe has quit IRC | 14:33 | |
| *** matsuhashi has quit IRC | 14:34 | |
| *** ruhe has joined #savanna | 14:35 | |
| *** matsuhashi has joined #savanna | 14:35 | |
| *** akuznetsov has joined #savanna | 14:36 | |
| *** SergeyLukjanov has joined #savanna | 14:38 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 14:38 | |
| *** matsuhashi has quit IRC | 14:39 | |
| *** _SergeyLukjanov has quit IRC | 14:39 | |
| *** sacharya has quit IRC | 14:41 | |
| *** SergeyLukjanov has joined #savanna | 14:52 | |
| *** rnirmal has joined #savanna | 15:02 | |
| *** sacharya has joined #savanna | 15:13 | |
| *** mattf is now known as _mattf | 15:18 | |
| *** ruhe has quit IRC | 15:26 | |
| *** sreshetnyak has quit IRC | 15:31 | |
| *** ruhe has joined #savanna | 15:33 | |
| openstackgerrit | Alexander Ignatov proposed a change to openstack/savanna: Added REST API v1.1 section https://review.openstack.org/50946 | 15:36 |
| *** DennyZhang has joined #savanna | 15:36 | |
| openstackgerrit | A change was merged to openstack/savanna: Update end time in job execution after job complete https://review.openstack.org/51809 | 15:40 |
| openstackgerrit | A change was merged to openstack/savanna: Unconditionally monkey patch nova.keypairs.get https://review.openstack.org/51693 | 15:42 |
| *** akuznetsov has quit IRC | 15:52 | |
| crobertsrh | Did the validation rules for job binary file names change? hadoop-examples-1.2.1.jar is not a valid file name? Are "-" no longer allowed? | 15:52 |
| *** DennyZhang has quit IRC | 15:55 | |
| *** crobertsrh is now known as crobertsrh-lunch | 15:56 | |
| *** tmckay has quit IRC | 15:57 | |
| aignatov | mmm, let me check | 16:01 |
| aignatov | yes, "error_message": "u'hadoop-examples-1.2.1.jar' is not a 'valid_name'", | 16:02 |
| aignatov | also the name "hadoopexamples1.1jar" is not allowed | 16:04 |
| aignatov | because of digit after dot | 16:04 |
| *** nadya has quit IRC | 16:04 | |
| aignatov | I think we shoud change it, but it seems not in 0.3 release | 16:05 |
| *** ruhe has quit IRC | 16:05 | |
| SergeyLukjanov | oops, yeah, looks like too strong validation | 16:06 |
| *** dmitryme has quit IRC | 16:08 | |
| *** tvb|afk has quit IRC | 16:16 | |
| *** SergeyLukjanov has quit IRC | 16:20 | |
| *** _mattf is now known as mattf | 16:21 | |
| mattf | (OperationalError) no such column: clusters.trust_id <-- anyone remember off the top of their head how to resolve this? it's a savanna update. the db is old. | 16:22 |
| mattf | savanna-db-manage db-sync is no help | 16:24 |
| *** SergeyLukjanov has joined #savanna | 16:41 | |
| *** crobertsrh-lunch is now known as crobertsrh | 16:41 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 16:42 | |
| *** _SergeyLukjanov is now known as SergeyLukjanov | 16:42 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 16:42 | |
| *** _SergeyLukjanov is now known as SergeyLukjanov | 16:42 | |
| *** ruhe has joined #savanna | 16:43 | |
| *** akuznetsov has joined #savanna | 17:07 | |
| *** dmitryme has joined #savanna | 17:18 | |
| *** DennyZhang has joined #savanna | 17:25 | |
| *** tmckay has joined #savanna | 17:26 | |
| *** akuznetsov has quit IRC | 17:26 | |
| *** dafter has joined #savanna | 17:27 | |
| *** dafter has quit IRC | 17:32 | |
| openstackgerrit | Dmitry Mescheryakov proposed a change to openstack/savanna: Remove KeypairManager.get monkey patch https://review.openstack.org/51734 | 17:37 |
| *** openstackgerrit has quit IRC | 17:37 | |
| *** openstackgerrit has joined #savanna | 17:38 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/savanna-extra: Add example edp jobs folder and pig example https://review.openstack.org/51892 | 17:47 |
| *** sreshetnyak has joined #savanna | 17:47 | |
| *** DennyZhang has quit IRC | 17:49 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/savanna-extra: Add simple tox.ini to make gate working https://review.openstack.org/51895 | 17:50 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/savanna-image-elements: Add simple tox.ini to make gate working https://review.openstack.org/51896 | 17:50 |
| openstackgerrit | A change was merged to openstack/savanna-image-elements: Add simple tox.ini to make gate working https://review.openstack.org/51896 | 17:51 |
| openstackgerrit | A change was merged to openstack/savanna-extra: Add simple tox.ini to make gate working https://review.openstack.org/51895 | 17:52 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/savanna-extra: Add example edp jobs folder and pig example https://review.openstack.org/51892 | 17:53 |
| tmckay | hey folks, I have 2 instances from a cluster launced from savanna, which I can ssh to, but the cluster status is still "Waiting" and has been for a long time. What is savanna trying to do to connect to them? | 18:23 |
| tmckay | this is where I'm getting to lately, and ultimately it errors out | 18:23 |
| SergeyLukjanov | tmckay, hi | 18:24 |
| SergeyLukjanov | tmckay, possible it could be related to the floating ips config | 18:24 |
| SergeyLukjanov | which version of OS are you using? | 18:24 |
| SergeyLukjanov | and nova-network or neutroin | 18:24 |
| SergeyLukjanov | neutron* | 18:24 |
| tmckay | my host is F19, my images are fedora | 18:24 |
| dmitryme | can you post your logs via paste | 18:25 |
| tmckay | I think I'm using the default network | 18:25 |
| tmckay | unfortunately I have savanna logging to the screen | 18:25 |
| tmckay | I should change that | 18:25 |
| dmitryme | yep | 18:25 |
| tmckay | openstack I have logging to disk | 18:25 |
| dmitryme | also can you rebase to the latest master | 18:25 |
| dmitryme | it has additional logging | 18:25 |
| tmckay | okay. Is there a place in the savanna code you can point me for the command it is trying to use now to connect? If it's a shell command maybe I can debug it. | 18:26 |
| dmitryme | Since it is hanging in "Waiting" state, I think it should be in the following method: https://github.com/openstack/savanna/blob/master/savanna/service/instances.py#L37 | 18:27 |
| tmckay | cool, thanks. Hmm, not sure if I have any floating ips assigned. how can I check? Each instance only shows an address from the "fixed" range in horizon | 18:28 |
| tmckay | ah, I see, maybe stuck in _await_active (hence no floating ip) | 18:29 |
| SergeyLukjanov | tmckay, are instance avaiable from the host where savanna installed through the internal ips? | 18:30 |
| dmitryme | another question, how value 'use_floating_ips' in config is set to? | 18:31 |
| tmckay | yes. I have a fixed range of 10.0.3.x, and I can ssh -l fedora from the savanna host to 10.0.3.2 and 10.0.3.3 | 18:31 |
| tmckay | dmitryme, commented out | 18:32 |
| tmckay | #use_floating_ips=True | 18:32 |
| *** ruhe has quit IRC | 18:34 | |
| dmitryme | I think it should start working if you uncomment it and set to false | 18:34 |
| tmckay | okay, I'll try | 18:34 |
| SergeyLukjanov | yeah, the default value is true | 18:34 |
| tmckay | why would that break it? | 18:35 |
| tmckay | _check_if_active looks pretty simple | 18:35 |
| SergeyLukjanov | it waits for floating ips to access vms | 18:35 |
| *** SergeyLukjanov has quit IRC | 18:35 | |
| tmckay | so maybe I'm stuck in _assign_floating_ips? or _await_networks? | 18:36 |
| dmitryme | it would work your current config if you had nova-network and auto_assign_floating_ip=True in your nova.conf | 18:36 |
| dmitryme | in that case floating IPs would be assigned automatically by Nova | 18:37 |
| tmckay | okay, thanks. I'll check that. crobertsrh was just mentioning something about that too | 18:37 |
| dmitryme | if there is no auto assignment, then as far as I recall you need to pass pool_id, or something like that | 18:38 |
| dmitryme | with each node group | 18:38 |
| dmitryme | Savanna will itself assign a floating IP taken from that pool | 18:39 |
| *** SergeyLukjanov has joined #savanna | 18:39 | |
| tmckay | okay. So next question, is it safe to "delete cluster" to get this out of the waiting state? | 18:39 |
| *** ruhe has joined #savanna | 18:41 | |
| tmckay | alright, retrying with use_floating_ips=False | 18:44 |
| tmckay | dmitryme, oh my! It's configuring! | 18:46 |
| tmckay | And Active! Are you kidding me? | 18:46 |
| dmitryme | great to hear that :-) | 18:47 |
| SergeyLukjanov | tmckay, it's safe enough to delete cluster in waiting state | 18:47 |
| tmckay | SergeyLukjanov, thanks. It was my floating_ip config. Thank you very much. I think I had other issues earlier with nova sec groups, etc. Maybe I can run a job now. | 18:49 |
| tmckay | SergeyLukjanov, also, thanks for the pointer to the "create_cluster" routine. Seems like a useful place for debug. | 18:51 |
| crobertsrh | While trying to run the pig demo, I get the following (in both fedora and ubuntu image, so it seems like something I'm doing incorrectly): http://fpaste.org/46992/81863631/ (oozie log). My job get to RUNNING state briefly, then KILLED. Thanks for any tips! | 19:01 |
| *** ruhe has quit IRC | 19:04 | |
| *** dmitryme has quit IRC | 19:07 | |
| mattf | SergeyLukjanov, https://review.openstack.org/#/c/51892 -- should the output be checked in? | 19:34 |
| crobertsrh | aignatov: is there anything special required from your cluster config to successfully run your pig demo? Both tmckay and I are trying to run it, but our job winds-up as "KILLED". | 19:39 |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 19:46 | |
| *** _SergeyLukjanov has quit IRC | 19:47 | |
| openstackgerrit | A change was merged to openstack/savanna: Remove KeypairManager.get monkey patch https://review.openstack.org/51734 | 19:49 |
| *** ruhe has joined #savanna | 19:53 | |
| crobertsrh | Hmm, the pig job fails with error message: Workflow failed, error message[Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]] | 19:57 |
| *** SergeyLukjanov has joined #savanna | 19:58 | |
| *** ruhe has quit IRC | 20:00 | |
| tmckay | crobertsrh, yes, I was just looking at main class messages before an emgency dog walk | 20:10 |
| *** SergeyLukjanov has quit IRC | 20:11 | |
| crobertsrh | tmckay: A useful tool...the oozie web UI http://<your master ip>:11000/oozie/ | 20:14 |
| tmckay | crobertsrh, where did you find that PigMain error? | 20:15 |
| crobertsrh | click down a few levels in the web UI | 20:15 |
| crobertsrh | I actually originally found it in oozie.log, but the web UI is nicer | 20:15 |
| tmckay | ah, there we go. JA018 | 20:23 |
| tmckay | that's got to mean something | 20:24 |
| crobertsrh | Yep, same code I see | 20:24 |
| crobertsrh | I wonder if our images are just out of date? | 20:24 |
| *** rnirmal has quit IRC | 20:26 | |
| tmckay | found my way to jobdetails. oozie.launcher.error counter is 1 on Map. I would guess this means that Oozie doesn't know how to find the main class for Map. | 20:30 |
| *** dmitryme has joined #savanna | 20:45 | |
| crobertsrh | Ah, finally got my job to run.....question. Is it a bug or a feature that the name that I give my "job binary" is actually used in the execution of the job? My problem was that I called gave udf.jar a name of "udf-jar" when I created my binary. When I called it "udf.jar", it started working properly. | 20:51 |
| *** mattf is now known as _mattf | 20:54 | |
| *** SergeyLukjanov has joined #savanna | 21:12 | |
| *** crobertsrh is now known as _crobertsrh | 21:13 | |
| *** dmitryme has quit IRC | 21:18 | |
| *** sreshetnyak has quit IRC | 21:20 | |
| *** sacharya has quit IRC | 22:28 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 22:34 | |
| *** _SergeyLukjanov has quit IRC | 22:34 | |
| *** SergeyLukjanov has joined #savanna | 22:36 | |
| *** SergeyLukjanov has quit IRC | 22:44 | |
| *** akuznetsov has joined #savanna | 23:20 | |
| *** tmckay has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!