| *** altlogbot_2 has quit IRC | 00:10 | |
| *** altlogbot_0 has joined #openstack-qinling | 00:12 | |
| *** goldyfruit has quit IRC | 01:06 | |
| *** goldyfruit has joined #openstack-qinling | 01:31 | |
| *** goldyfruit has quit IRC | 02:54 | |
| *** heychirag has quit IRC | 05:29 | |
| *** heychirag has joined #openstack-qinling | 05:33 | |
| *** goldyfruit has joined #openstack-qinling | 13:07 | |
| goldyfruit | lxkong, second part: https://review.opendev.org/#/c/660789/ | 18:03 |
|---|---|---|
| nsherry4 | I'm following the install instructions here (https://docs.openstack.org/qinling/latest/admin/install/install_ubuntu.html) but on RHEL7. I hit the finalize step, and the systemd service files are referring to files in /usr/lical/bin which don't seem to exist. Anyone know what these should be symlinked to? | 18:52 |
| goldyfruit | nsherry4, where is you qinling-api file ? | 19:28 |
| nsherry4 | goldyfruit: I put the qinling-api service file in the suggested location '/etc/systemd/system/qinling-api.service' | 19:30 |
| nsherry4 | systemd picks it up, but fails at the exec stage, not finding the binary | 19:31 |
| goldyfruit | But you did install qinling ? | 19:31 |
| nsherry4 | I installed it as the documentation suggested by doing a `git clone` | 19:32 |
| goldyfruit | and "pip install -e ." ? | 19:32 |
| nsherry4 | yes | 19:32 |
| goldyfruit | So if you ran the pip command then the binary should be in /usr/local/bin | 19:32 |
| goldyfruit | No error during the pip install ? | 19:33 |
| nsherry4 | # pip list | grep -i qinling | 19:33 |
| nsherry4 | qinling (1.0.1.dev5, /var/lib/qinling/qinling) | 19:33 |
| nsherry4 | no errors that I saw | 19:33 |
| goldyfruit | find / -name qinling-api | 19:34 |
| nsherry4 | since then, I went ahead and symlinked the files in `/var/lib/qinling/qinling/qinling/cmd/` | 19:34 |
| nsherry4 | Looks like it landed in /usr/bin/qinling-api | 19:35 |
| goldyfruit | Cool | 19:35 |
| nsherry4 | thanks | 19:35 |
| goldyfruit | So you just have to link it there | 19:35 |
| goldyfruit | same for qinling-engine | 19:35 |
| nsherry4 | That's picking it up now, great | 19:38 |
| goldyfruit | Cool! | 19:40 |
| goldyfruit | nsherry4, how things are going ? | 20:31 |
| nsherry4 | I'm using microk8s as my backend and have no idea what to do here... | 20:36 |
| goldyfruit | Qinling needs to connect to Kubernetes and to etcd | 20:37 |
| goldyfruit | microk8s setup an etcd server ? | 20:37 |
| nsherry4 | I've installed microk8s with snap and etcd from yum | 20:38 |
| goldyfruit | Ok | 20:38 |
| goldyfruit | etcd is running with or without SSL ? | 20:38 |
| nsherry4 | ahh | 20:38 |
| nsherry4 | I don't know. I never enabled it, and all the URLs in the config file are http:// | 20:39 |
| goldyfruit | Did you use this documentation https://docs.openstack.org/install-guide/environment-etcd-rdo.html ? | 20:39 |
| nsherry4 | I did | 20:39 |
| goldyfruit | Then there is no SSL, easier for now :) | 20:40 |
| nsherry4 | except it didn't seem to like ETCD_NAME="controller" | 20:40 |
| goldyfruit | Doesn't matter | 20:40 |
| goldyfruit | In qinling.conf you need to set your etcd | 20:40 |
| goldyfruit | https://paste.api-zulu.com/fipexiyabi.ini | 20:40 |
| goldyfruit | Where host is equal to your IP of course | 20:41 |
| goldyfruit | both api and engine need to connect to the etcd server | 20:41 |
| goldyfruit | Then the next step will be to configure the engine to speak with your Kubernetes cluster | 20:41 |
| goldyfruit | I guess your Kubernest API will be http://YOUR-IP:8080 | 20:43 |
| nsherry4 | microk8s.config seems to indicate it's https://MY-IP:16443 | 20:43 |
| nsherry4 | which is what I've set the kube_host value to | 20:44 |
| goldyfruit | If you "curl https://MY-IP:16443 -I" what do you have ? | 20:44 |
| nsherry4 | short version: 'curl: (60) Peer's Certificate issuer is not recognized.' | 20:45 |
| goldyfruit | add -k | 20:45 |
| nsherry4 | HTTP/1.1 401 Unauthorized Content-Type: application/json Www-Authenticate: Basic realm="kubernetes-master" Date: Wed, 22 May 2019 20:45:58 GMT Content-Length: 165 | 20:45 |
| goldyfruit | Ok, so this is your endpoint | 20:46 |
| goldyfruit | In /var/microk8s/current/certs you should have certificates | 20:46 |
| nsherry4 | I don't have that folder | 20:46 |
| nsherry4 | I assume because it was installed via snap | 20:46 |
| nsherry4 | looks like its in /snap/microk8s/current/certs | 20:47 |
| nsherry4 | all I see in there is csr.conf.template | 20:47 |
| goldyfruit | Oki | 20:47 |
| goldyfruit | Nothing in /var/snap/microk8s/current ? | 20:47 |
| nsherry4 | oh, yes | 20:48 |
| nsherry4 | ca.crt ca.key ca.srl csr.conf kubelet.crt kubelet.key server.crt server.csr server.key serviceaccount.key | 20:48 |
| goldyfruit | cool | 20:48 |
| goldyfruit | So I guess you will need: ca.crt, server.crt and server.key | 20:51 |
| nsherry4 | those match ssl_ca_cert, cert_file, and key_file respectively? | 20:52 |
| nsherry4 | Okay, I don't see any more connection/ssl errors in journalctl for qinling-engine or qinling-api | 21:01 |
| nsherry4 | and the openstack runtime/function list commands return empty strings instead of errors | 21:01 |
| goldyfruit | Yes, for the question above | 21:01 |
| goldyfruit | You should see: "Function mapping handler started." | 21:02 |
| goldyfruit | "Starting engine..." | 21:02 |
| goldyfruit | In the engine log | 21:02 |
| nsherry4 | yup | 21:03 |
| goldyfruit | Could you please run this command: kubectl get netpol -n qinling | 21:03 |
| nsherry4 | NAME POD-SELECTOR AGE | 21:03 |
| goldyfruit | This is created by qinling-engine when it's connected to the Kubernetes cluster | 21:03 |
| nsherry4 | allow-qinling-engine-only <none> 10m | 21:03 |
| goldyfruit | Cool | 21:03 |
| goldyfruit | qinling-engine and your Kubernetes are speaking togheter | 21:04 |
| nsherry4 | nice | 21:04 |
| nsherry4 | So would the next step be adding a runtime? | 21:05 |
| goldyfruit | This network could be an issue as mentioned here: https://storyboard.openstack.org/#!/story/2005710 | 21:05 |
| goldyfruit | So to avoid any issue right, I invite you to remove it (it will be re-created when qinling-engine will restart) | 21:05 |
| goldyfruit | kubectl delete networkpolicy allow-qinling-engine-only -n qinling | 21:05 |
| goldyfruit | Yes, next step is the engine | 21:06 |
| goldyfruit | I got error with python2 runtime, I used the python3 | 21:06 |
| goldyfruit | openstack runtime create openstackqinling/python3-runtime --name python3 | 21:06 |
| nsherry4 | are these included in the git repo in the 'runtime' folder? | 21:07 |
| goldyfruit | openstackqinling/python3-runtime from Docker hub | 21:07 |
| goldyfruit | https://hub.docker.com/r/openstackqinling/python3-runtime | 21:07 |
| nsherry4 | just `openstack runtime create openstackqinling/python3-runtime`? | 21:08 |
| goldyfruit | yep | 21:08 |
| goldyfruit | kubectl get pod -n qinling | 21:09 |
| goldyfruit | should returns pods | 21:09 |
| goldyfruit | starting with the ID of your runtime (openstack runtime list) | 21:10 |
| nsherry4 | neat, looks like it's created 3 of them | 21:10 |
| nsherry4 | or, creating | 21:10 |
| goldyfruit | Yeah, because of the replicaset | 21:10 |
| goldyfruit | https://github.com/openstack/qinling/blob/bc0e64b94a83b8b433a1e6161b60e9490d76f5f3/qinling/orchestrator/kubernetes/templates/deployment.j2 | 21:11 |
| goldyfruit | https://github.com/openstack/qinling/blob/bc0e64b94a83b8b433a1e6161b60e9490d76f5f3/qinling/config.py#L144-L148 | 21:11 |
| nsherry4 | okay, so we could change that later if we wanted to | 21:12 |
| goldyfruit | yep, inside the [kubernetes] section | 21:12 |
| goldyfruit | Is your runtime in Running state ? | 21:12 |
| nsherry4 | Is that going to be an initial size which would scale with demand? | 21:13 |
| nsherry4 | or is it more of a static allocation setting | 21:13 |
| goldyfruit | The replicaset if more for failure perspective | 21:13 |
| nsherry4 | like host machine failures? | 21:14 |
| goldyfruit | yeah | 21:14 |
| nsherry4 | okay, so that'll make more sense once we scale k8s a bit | 21:15 |
| goldyfruit | https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/ | 21:15 |
| nsherry4 | so now that the runtime is defined and available in k8s, the next step is to define a function? | 21:16 |
| goldyfruit | Correct | 21:16 |
| nsherry4 | I'll actually have to do that, first, I guess. | 21:16 |
| nsherry4 | write one | 21:16 |
| goldyfruit | openstack function create --name hello_world --runtime YOUR-RUNTIME-ID --entry hello_world.main --file hello_world.py | 21:18 |
| goldyfruit | hello_world.py content: https://paste.api-zulu.com/iribukopof.py | 21:18 |
| nsherry4 | where will the arguments come from in this case? Are they translated from a REST call? | 21:21 |
| goldyfruit | That is just a function without args | 21:21 |
| goldyfruit | Just to test the function/runtime | 21:21 |
| goldyfruit | When you need to set arguments, there is the --input option in the "openstack function execution" CLI | 21:22 |
| goldyfruit | That is a nice example of how to use Python runtime with package: https://medium.com/@n.neerja28/tutorial-on-how-to-create-a-python-function-with-libraries-in-a-package-d8a9b2f5e46 | 21:22 |
| nsherry4 | on `openstack function create` I see Unable to establish connection to http://<IP>:7070/v1/functions: ('Connection aborted.', BadStatusLine("''",)) | 21:27 |
| goldyfruit | Is your Kubernetes cluster able to reach the Qinling API endpoint ? | 21:28 |
| nsherry4 | I'm not sure... They're both on the same machine, but the keystone service/endpoints are all through an HAProxy server | 21:29 |
| nsherry4 | I've opened up 7070 on it, and can run other commands, but I don't know if/how to configure k8s to handle that | 21:30 |
| goldyfruit | qinling-api is listening on all the interfaces or only on 127.0.0.1 ? | 21:31 |
| goldyfruit | ss -plantu | grep 7070 | 21:31 |
| nsherry4 | looks like it's listening on it's 10.x.x.x IP address | 21:32 |
| nsherry4 | don't see localhost | 21:32 |
| goldyfruit | if you run "curl http://<IP>:7070/v1/functions" ? | 21:33 |
| goldyfruit | 401 ? | 21:33 |
| nsherry4 | yup | 21:34 |
| goldyfruit | OK, so it's in your Kubernest cluster | 21:34 |
| goldyfruit | What do you have in qinling logs ? | 21:35 |
| goldyfruit | If you shut your firewall ? | 21:38 |
| nsherry4 | https://pastebin.com/QXCSrXqD | 21:38 |
| goldyfruit | Check if your forward is enable ? | 21:38 |
| goldyfruit | iptables -P FORWARD ACCEPT | 21:38 |
| nsherry4 | blank output | 21:38 |
| goldyfruit | iptables -P FORWARD ACCEPT enables FORWARD :_ | 21:39 |
| nsherry4 | same error | 21:41 |
| goldyfruit | You have deleted the network policy right ? | 21:41 |
| goldyfruit | From Kubernetes | 21:41 |
| nsherry4 | trying to delete it again with the same command gives 'not found' | 21:42 |
| goldyfruit | ok | 21:42 |
| nsherry4 | I'm using the dns name of the haproxy server in all the config rather than the IP address. Could it be that microk8s isn' | 21:44 |
| nsherry4 | isn't or can't do name resolution outside of itself? | 21:44 |
| goldyfruit | But your Qinling API endpoint is an IP ? | 21:44 |
| nsherry4 | no, our setup is all done using hostnames rather than IPs, so I followed that convention | 21:45 |
| goldyfruit | http://<IP>:7070/v1/functions: ('Connection aborted.', BadStatusLine("''",)) | 21:45 |
| goldyfruit | So here it's not and <IP> but <DNS> ? | 21:45 |
| nsherry4 | yes | 21:45 |
| nsherry4 | sorry | 21:46 |
| goldyfruit | oh | 21:46 |
| goldyfruit | From https://microk8s.io/docs/ | 21:47 |
| goldyfruit | There are few steps about firewall/dns | 21:47 |
| goldyfruit | sudo ufw allow in on cbr0 && sudo ufw allow out on cbr0 | 21:47 |
| goldyfruit | I guess you are using firewalld | 21:47 |
| goldyfruit | microk8s.inspect | 21:49 |
| nsherry4 | you want me to send the tar file it generated? | 21:51 |
| nsherry4 | or just look at some of the output | 21:51 |
| goldyfruit | https://github.com/ubuntu/microk8s/issues/75 | 21:51 |
| goldyfruit | I don't know microk8s | 21:52 |
| goldyfruit | Did you try to enable the DNS in microk8s ? | 21:55 |
| goldyfruit | microk8s.enable dns | 21:55 |
| nsherry4 | It's already on | 21:55 |
| nsherry4 | I think I'm going to go back and change all the hostname entries in the db/config to ip addresses, which will take a while | 21:56 |
| nsherry4 | It's getting to be end of day here, so I think I'll pick this up tomorrow | 21:56 |
| goldyfruit | 6pm here | 21:56 |
| nsherry4 | same | 21:56 |
| nsherry4 | Thank you for all your help, I'm definitely further ahead than I would be otherwise | 21:57 |
| goldyfruit | You're welcome, you're very close to have Qinling working :) | 21:57 |
| nsherry4 | I'm looking forward to it. it definitely has a bit of a cool factor | 21:58 |
| nsherry4 | bye for now | 21:58 |
| goldyfruit | See you | 21:58 |
| *** goldyfruit_ has joined #openstack-qinling | 22:28 | |
| *** goldyfruit has quit IRC | 22:30 | |
| *** goldyfruit_ has quit IRC | 22:33 | |
| *** goldyfruit has joined #openstack-qinling | 22:42 | |
| *** goldyfruit has quit IRC | 23:21 | |
| *** goldyfruit has joined #openstack-qinling | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!