| *** goldyfruit has quit IRC | 01:36 | |
| *** goldyfruit has joined #openstack-qinling | 02:06 | |
| *** irclogbot_1 has quit IRC | 02:26 | |
| *** irclogbot_3 has joined #openstack-qinling | 02:29 | |
| goldyfruit | lxkong, +1 from Zuul https://review.opendev.org/#/c/660789/ | 02:48 |
|---|---|---|
| goldyfruit | Wait and see the comments from Kolla core reviewers! | 02:49 |
| lxkong | goldyfruit: thanks for the contribution | 02:49 |
| lxkong | and thanks for answering questions when i'm offline :-) | 02:49 |
| goldyfruit | You're welcome! | 02:50 |
| goldyfruit | The least I can do | 02:50 |
| lxkong | i have an meeting at 15:00, will be back soon | 02:50 |
| goldyfruit | 11pm here :) | 02:50 |
| *** goldyfruit has quit IRC | 04:06 | |
| *** hrw has joined #openstack-qinling | 13:14 | |
| hrw | morning | 13:14 |
| *** goldyfruit has joined #openstack-qinling | 13:16 | |
| hrw | your README.rst is py3 unfriendly :( | 13:17 |
| hrw | http://logs.openstack.org/75/642375/23/check/kolla-build-ubuntu-source/280dece/logs/build/000_FAILED_qinling-base.txt.gz | 13:18 |
| hrw | http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006544.html | 13:28 |
| goldyfruit | hrw I got the +1 form Zuul, I fixed my Readme | 13:37 |
| hrw | where? | 13:39 |
| goldyfruit | https://review.opendev.org/#/c/660789 | 13:39 |
| goldyfruit | hrw, sorry I think we are not speaking about the same thing ! | 13:39 |
| hrw | goldyfruit: that job was run with py2 | 13:39 |
| hrw | goldyfruit: which gave you +1 | 13:39 |
| hrw | goldyfruit: https://review.opendev.org/642375 check | 13:40 |
| goldyfruit | yes indeed ! | 13:41 |
| hrw | ;) | 13:41 |
| goldyfruit | This is related to tʃinliŋ | 13:44 |
| hrw | yes | 13:44 |
| goldyfruit | I'll try to fix it | 13:45 |
| hrw | thank you | 13:45 |
| goldyfruit | I guess we could go Tchinling | 13:53 |
| hrw | englished ipa? | 13:54 |
| goldyfruit | I asked to a colleague who speaks mandarin | 13:57 |
| hrw | cool | 13:58 |
| goldyfruit | I hope lxkong will be OK with that :D | 13:58 |
| hrw | but you can wait for https://review.opendev.org/#/c/564874 | 13:59 |
| hrw | as this may solve it globally | 13:59 |
| goldyfruit | Last update, June 2018 | 13:59 |
| hrw | http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006547.html | 14:00 |
| hrw | bbl | 14:00 |
| openstackgerrit | Gaëtan Trellu proposed openstack/qinling master: README: make it us-ascii https://review.opendev.org/661005 | 14:03 |
| hrw | thanks | 14:53 |
| hrw | commented and +1 | 14:53 |
| goldyfruit | Oh! hrw = Marcin | 14:54 |
| hrw | yes | 14:54 |
| goldyfruit | Didn't know, we met few times at the summit | 14:54 |
| goldyfruit | <- Ormuco | 14:54 |
| hrw | never been at summit ;D | 14:54 |
| hrw | maybe at Snowpenstack in Dublin | 14:55 |
| goldyfruit | Oh, wrong Name then :( | 14:55 |
| goldyfruit | (sorry) | 14:55 |
| hrw | happens ;D | 14:55 |
| hrw | np | 14:55 |
| hrw | I do not have good face memory | 14:55 |
| *** goldyfruit has quit IRC | 17:03 | |
| *** hrw has quit IRC | 17:21 | |
| *** goldyfruit has joined #openstack-qinling | 17:36 | |
| nsherry4 | goldyfruit: I've made some progress since yesterday, but I think I've hit another road block, do you have some time to take a look? | 18:38 |
| goldyfruit | i could try | 18:39 |
| nsherry4 | I've been able to create a function, but now when I try to run it, qinling-engine gets a 500 error back | 18:39 |
| nsherry4 | which manifests as it not being able to decode a non-existent json payload | 18:40 |
| goldyfruit | JSONDecodeError: Expecting value: line 1 column 1 (char 0)? | 18:40 |
| nsherry4 | yup | 18:40 |
| goldyfruit | Like this bug: https://storyboard.openstack.org/#!/story/2005711 ? | 18:40 |
| goldyfruit | Are you using Python 2.7 or Python 3 ? | 18:41 |
| goldyfruit | I got error with Python 2.7 | 18:41 |
| goldyfruit | Which is why I opened the bug :p | 18:41 |
| nsherry4 | openstackqinling/python3-runtime | 18:42 |
| goldyfruit | hum | 18:42 |
| goldyfruit | What is your code ? | 18:42 |
| goldyfruit | of the function | 18:42 |
| nsherry4 | Just the 'Hello World' function you mentioned | 18:42 |
| nsherry4 | I added some debug output to engine, and got "The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application." | 18:42 |
| goldyfruit | Could you please paste the log ? | 18:43 |
| nsherry4 | sure thing | 18:43 |
| nsherry4 | https://pastebin.com/fkRzi7FS | 18:44 |
| goldyfruit | You could try to check the pod logs | 18:45 |
| goldyfruit | kubectl get pod -o wide -n qinling | 18:45 |
| goldyfruit | kubectl logs 35bafe75-f5fa-440e-af09-ae2fe97c8e5f-849fcdd9cb-v2dhg worker -n qinlin | 18:46 |
| goldyfruit | Do the same for the 3 pods | 18:46 |
| nsherry4 | https://pastebin.com/ZSFqwKWa | 18:49 |
| goldyfruit | Failed to download function package from http://<IP>:7070/v1/functions/eda8a29b-dbc1-47e0-9ed5-a133d2ddac30?download=true | 18:49 |
| goldyfruit | Failed to establish a new connection: [Errno 113] Host is unreachable | 18:50 |
| goldyfruit | Which means that your worker container doesn't have access to Qinling API | 18:50 |
| nsherry4 | Firewall issue? | 18:50 |
| goldyfruit | Could be | 18:51 |
| goldyfruit | As I mentioned yesterday, I would have disable all the possible failure stuff for the tests | 18:51 |
| goldyfruit | firewall, selinux | 18:51 |
| goldyfruit | just for tests | 18:51 |
| nsherry4 | Disabled iptables, same result | 18:53 |
| nsherry4 | I can get a shell into the worker node with exec, right? | 18:54 |
| goldyfruit | nop | 18:54 |
| goldyfruit | but you could create a simple pod (not related to qinling) | 18:55 |
| goldyfruit | https://kubernetes.io/docs/tasks/debug-application-cluster/get-shell-running-container/ | 18:55 |
| goldyfruit | nginx image has a shell | 18:55 |
| nsherry4 | okay, so I _was_ able to shell into the worker node, but it doesn't have ping/telnet/etc. It does have ssh, and with the -v flag, I can see it returning html 400 markup when I try to ssh to port 7070, so it _can_ connect | 19:00 |
| goldyfruit | but port 7070 is not ssh | 19:01 |
| goldyfruit | It's HTTP | 19:01 |
| goldyfruit | Do you have curl/wget in the nginx image ? | 19:02 |
| nsherry4 | I don't have an nginx image | 19:02 |
| goldyfruit | ohh ok | 19:03 |
| nsherry4 | I do have curl on the worker, though | 19:03 |
| nsherry4 | curl http://10.64.12.232:7070 | 19:03 |
| goldyfruit | yep | 19:03 |
| nsherry4 | {"versions": [{"status": "CURRENT", "id": "v1.0", "links": [{"href": "http://10.64.12.232:7070/v1", "target": "v1", "rel": "self"}]}]} | 19:03 |
| goldyfruit | Just in case, could you please list the network policy | 19:03 |
| goldyfruit | kubectl get netpol -n qinling | 19:04 |
| goldyfruit | should be empty | 19:04 |
| goldyfruit | except if qinling-engine has been restarted since yesterday | 19:04 |
| nsherry4 | ah, yes, I've restarted it | 19:04 |
| nsherry4 | allow-qinling-engine-only <none> 127m | 19:04 |
| goldyfruit | kubectl delete netpol allow-qinling-engine-only -n qinling | 19:05 |
| nsherry4 | I have to delete this with every restart? | 19:05 |
| goldyfruit | yeah :( | 19:05 |
| goldyfruit | because of the bug I sent you yesterday | 19:05 |
| goldyfruit | https://storyboard.openstack.org/#!/story/2005710 | 19:05 |
| nsherry4 | Same error | 19:06 |
| nsherry4 | wait | 19:07 |
| nsherry4 | same error in the engine | 19:07 |
| nsherry4 | different error on the worker | 19:07 |
| goldyfruit | Ah | 19:07 |
| nsherry4 | This time it can't serialize a "Read timed out. (read timeout=5)" | 19:08 |
| goldyfruit | could you please paste the error | 19:08 |
| nsherry4 | https://pastebin.com/t3NPV5Mq | 19:09 |
| goldyfruit | Yeah, still the same, unable to reach the Qinling API | 19:09 |
| nsherry4 | curling that URL from the container shell gives me a 401 Unauthorized | 19:11 |
| goldyfruit | Do you see the call in qinling-api.log ? | 19:11 |
| nsherry4 | 2 lines, creating execution and downloading function | 19:12 |
| goldyfruit | When you create your function, try to specify a bigger timeout | 19:14 |
| goldyfruit | --timeout | 19:14 |
| nsherry4 | Wait, the logs seem to show the 'Downloading function' line showing up after the execution fails | 19:14 |
| nsherry4 | I'll try the timeout change now | 19:15 |
| nsherry4 | Set the timeout to 300s, but got the same result. Didn't take 300s to fail | 19:22 |
| nsherry4 | looking at the created/updated times, it took 5 seconds, which is the time in the read timeout error in the worker | 19:23 |
| goldyfruit | I never got this issue :/ | 19:27 |
| goldyfruit | Could be related to microk8s | 19:28 |
| goldyfruit | even with Iptables disable, did you apply the FORWARD | 19:28 |
| nsherry4 | I'm seeing what happens when I add a timeout value to the python requests post call in worker:server.py to fetch the script | 19:29 |
| goldyfruit | kubectl logs 35bafe75-f5fa-440e-af09-ae2fe97c8e5f-849fcdd9cb-v2dhg sidecar -n qinling | 19:29 |
| goldyfruit | Check the sidecar container | 19:29 |
| goldyfruit | Which is the one responsible to download the package | 19:29 |
| nsherry4 | sidecar shows a number of 5000ms download calls, but I don't see any errors | 19:33 |
| nsherry4 | "POST /download => generated 222 bytes in 5018 msecs (HTTP/1.1 500) 2 headers in 99 bytes (1 switches on core 0)" | 19:35 |
| nsherry4 | so the worker node calls the sidecar, which in turn downloads the function package from qinling-api? | 19:37 |
| goldyfruit | Is the engine bind on an IP ? | 19:38 |
| goldyfruit | https://github.com/openstack/qinling/blob/bc0e64b94a83b8b433a1e6161b60e9490d76f5f3/qinling/config.py#L86-L121 | 19:38 |
| nsherry4 | "Failed to request url http://10.64.12.232:31380/execute" | 19:39 |
| nsherry4 | I don't have any settings in the [engine] section set in the config | 19:40 |
| goldyfruit | Ok, so by default it listens everywhere | 19:40 |
| goldyfruit | I'm running out of ID | 19:41 |
| goldyfruit | idea | 19:41 |
| nsherry4 | Does the sidecar have to auth to download the function package? | 19:41 |
| nsherry4 | Our auth system in this toy environment is pretty slow | 19:42 |
| goldyfruit | No the authentication is made by Qinling API | 19:42 |
| goldyfruit | When the call goes thru the engine it means that it has already been authenticated | 19:42 |
| nsherry4 | got it | 19:43 |
| goldyfruit | https://github.com/openstack/qinling/blob/master/runtimes/sidecar/sidecar.py | 19:43 |
| goldyfruit | I think I said crap about the auth and the sidecar | 19:43 |
| goldyfruit | https://github.com/openstack/qinling/blob/master/runtimes/sidecar/sidecar.py#L55-L56 | 19:44 |
| nsherry4 | line 59: timeout=5 | 19:44 |
| goldyfruit | :D | 19:44 |
| nsherry4 | any way I can change that through kubectl? | 19:44 |
| goldyfruit | You mean kubectl exec ? | 19:45 |
| goldyfruit | Going into the sidecar container and change it manually ? | 19:46 |
| nsherry4 | a-hah | 19:46 |
| nsherry4 | sidecar doesn't have bash, but it does have sh | 19:46 |
| nsherry4 | I may have to restart the containers for this to work | 19:48 |
| goldyfruit | yep | 19:48 |
| goldyfruit | remove the pyc too before | 19:48 |
| goldyfruit | (if there is) | 19:48 |
| goldyfruit | nsherry4, I just build a new one with 60s as timeout | 19:51 |
| goldyfruit | https://hub.docker.com/r/goldyfruit/sidecar | 19:51 |
| goldyfruit | In your qinling.conf | 19:51 |
| goldyfruit | sidecar_image = goldyfruit/sidecar:0.0.3 | 19:52 |
| goldyfruit | under the [engine section] | 19:52 |
| nsherry4 | restart the engine? | 19:52 |
| goldyfruit | yes, and then delete the network policy | 19:53 |
| goldyfruit | You will have to create a new runtime | 19:53 |
| goldyfruit | runtime create the two containers | 19:53 |
| goldyfruit | https://github.com/openstack/qinling/blob/bc0e64b94a83b8b433a1e6161b60e9490d76f5f3/qinling/orchestrator/kubernetes/templates/deployment.j2 | 19:54 |
| nsherry4 | just to keep things simple, I'm going to set the replicas to 1 | 19:54 |
| goldyfruit | +1 | 19:55 |
| nsherry4 | okay, new runtime is created, creating a new function for it | 19:57 |
| nsherry4 | haha, success | 20:00 |
| goldyfruit | Yeah !!! | 20:00 |
| nsherry4 | thank you so much | 20:00 |
| goldyfruit | Your are welcome | 20:01 |
| goldyfruit | You* | 20:01 |
| nsherry4 | I'm going to play around with this for a while and see what it can do now! | 20:02 |
| goldyfruit | That's great ! | 20:02 |
| nsherry4 | Long term, if this timeout issue follows us into a production environment, what would our best bet be? | 20:06 |
| goldyfruit | You could build your own sidecar with a very high timeout | 20:07 |
| goldyfruit | Like I did | 20:07 |
| nsherry4 | and that's just out of the runtimes folder in the git repo? | 20:08 |
| goldyfruit | Sorry | 20:58 |
| goldyfruit | https://paste.api-zulu.com/usorediyej.bash | 20:59 |
| goldyfruit | I created a story about the 5 seconds https://storyboard.openstack.org/#!/story/2005769 | 21:07 |
| lxkong | good morning/afternoon/evening, goldyfruit, nsherry4 | 21:12 |
| lxkong | just start my day and see your discussions here | 21:12 |
| goldyfruit | :) | 21:12 |
| goldyfruit | Morning | 21:13 |
| lxkong | i will spend some time today to try to fix the bugs you opened, goldyfruit | 21:13 |
| lxkong | goldyfruit: i have a question, have you done some evaluation for Knative? | 21:13 |
| goldyfruit | Don't know what it is lxkong :D | 21:14 |
| lxkong | hah | 21:14 |
| goldyfruit | Pretty old in the OpenStack world but pretty new un the Kubernetes one | 21:14 |
| lxkong | Knative is the serverless solution in k8s world | 21:15 |
| goldyfruit | Ohh | 21:15 |
| lxkong | it's becoming so popular in k8s community, i have plan to integrate with Knative at some point | 21:15 |
| goldyfruit | So knative is used as standalone ? | 21:16 |
| goldyfruit | no need of k8s cluster ? | 21:16 |
| goldyfruit | or it's running on top of it ? | 21:16 |
| lxkong | on top of it | 21:16 |
| lxkong | and need to use together with Istio for more advanced features | 21:16 |
| goldyfruit | lxkong, what would be the advantage compare to what we currently have ? | 21:17 |
| lxkong | more popular, more developer, more features :-) | 21:17 |
| lxkong | but it provides k8s API | 21:17 |
| goldyfruit | Sure, could be a very nice to have ! | 21:18 |
| openstackgerrit | Gaëtan Trellu proposed openstack/qinling master: Increase request timeout for sidecar https://review.opendev.org/661122 | 21:19 |
| openstackgerrit | Gaëtan Trellu proposed openstack/qinling master: Increase request timeout for sidecar https://review.opendev.org/661122 | 21:20 |
| goldyfruit | (forgot the story/task in the commit message...) | 21:21 |
| lxkong | goldyfruit: will review that patch today | 21:21 |
| goldyfruit | No rush ! | 21:22 |
| lxkong | and for `tchinling` thing, i need to ask my colleagues for a pronunciation :-) | 21:22 |
| goldyfruit | Sure | 21:22 |
| goldyfruit | I did a demo to my CEO | 21:24 |
| goldyfruit | He loved it :D | 21:24 |
| lxkong | really?! | 21:24 |
| lxkong | amazing! | 21:24 |
| goldyfruit | Took two devs to show them the potential ! | 21:24 |
| goldyfruit | Yeah | 21:24 |
| lxkong | if you decide to make some investment on Qinling, i'm very happy to support | 21:25 |
| goldyfruit | Oh we will | 21:26 |
| goldyfruit | Which is why I'm integrating it to Kolla | 21:26 |
| goldyfruit | which will help us to maintain/upgrade it in production | 21:26 |
| lxkong | i believe my boss would also love to see there are some other people interested in Qinling after two dev cycles | 21:26 |
| goldyfruit | Yeah ! | 21:27 |
| goldyfruit | Hope the kolla integration will help for the adoption | 21:27 |
| goldyfruit | The doc needs to be improved too, how to support Magnum, microk8s, etc... | 21:28 |
| goldyfruit | And then people will discover the potential | 21:28 |
| goldyfruit | Like nsherry4 :p | 21:28 |
| lxkong | never heard of microk8s before, but could start with improving the doc related to Magnum | 21:28 |
| goldyfruit | Me too until yesterday :D | 21:29 |
| openstackgerrit | Merged openstack/qinling master: README: make it us-ascii https://review.opendev.org/661005 | 22:34 |
| *** goldyfruit has quit IRC | 22:38 | |
| *** goldyfruit has joined #openstack-qinling | 22:53 | |
| *** goldyfruit has quit IRC | 23:22 | |
| *** goldyfruit has joined #openstack-qinling | 23:42 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!