| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: Move kibana to OSH infra https://review.openstack.org/529068 | 00:10 |
|---|---|---|
| *** marst has quit IRC | 00:11 | |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: Move kibana to OSH infra https://review.openstack.org/529068 | 00:27 |
| *** yangyapeng has quit IRC | 00:30 | |
| *** XueFeng has quit IRC | 00:44 | |
| *** XueFeng has joined #openstack-helm | 00:44 | |
| *** yangyapeng has joined #openstack-helm | 01:21 | |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: Move fluentbit and fluentd configs to values.yaml https://review.openstack.org/527489 | 01:21 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: Move fluentbit and fluentd configs to values.yaml https://review.openstack.org/527489 | 01:25 |
| openstackgerrit | Merged openstack/openstack-helm master: Dev-Deploy: setup gateway to use default route device https://review.openstack.org/530145 | 01:31 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: Move fluentbit and fluentd configs to values.yaml https://review.openstack.org/527489 | 01:39 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Dev-Deploy: Deploy Nova with native kvm when possible https://review.openstack.org/530144 | 01:46 |
| *** ljjjustin has joined #openstack-helm | 01:55 | |
| *** adrianreza has joined #openstack-helm | 02:01 | |
| *** yamamoto has joined #openstack-helm | 02:07 | |
| *** XueFeng has quit IRC | 02:08 | |
| *** XueFeng has joined #openstack-helm | 02:08 | |
| *** XueFeng has quit IRC | 02:35 | |
| *** XueFeng has joined #openstack-helm | 02:35 | |
| *** XueFeng has quit IRC | 03:07 | |
| *** XueFeng has joined #openstack-helm | 03:07 | |
| *** yamamoto has quit IRC | 03:31 | |
| *** XueFeng has quit IRC | 03:49 | |
| *** XueFeng has joined #openstack-helm | 03:49 | |
| *** mdih has joined #openstack-helm | 04:00 | |
| *** yamamoto has joined #openstack-helm | 04:08 | |
| *** XueFeng has quit IRC | 04:10 | |
| *** XueFeng has joined #openstack-helm | 04:10 | |
| *** yamamoto has quit IRC | 04:33 | |
| *** yamamoto has joined #openstack-helm | 04:34 | |
| *** mdih has quit IRC | 04:39 | |
| *** yamamoto has quit IRC | 04:40 | |
| *** XueFeng has quit IRC | 04:55 | |
| *** XueFeng has joined #openstack-helm | 04:55 | |
| *** mdih has joined #openstack-helm | 04:56 | |
| *** mdih has quit IRC | 05:01 | |
| *** yamamoto has joined #openstack-helm | 05:08 | |
| *** yamamoto has joined #openstack-helm | 05:10 | |
| *** marst has joined #openstack-helm | 06:07 | |
| *** marst has quit IRC | 06:12 | |
| *** adrianreza has quit IRC | 06:32 | |
| *** krypto has joined #openstack-helm | 07:22 | |
| *** krypto has quit IRC | 07:24 | |
| *** krypto has joined #openstack-helm | 07:24 | |
| *** krypto has quit IRC | 07:36 | |
| *** yamamoto has quit IRC | 08:02 | |
| *** yamamoto has joined #openstack-helm | 09:02 | |
| *** yamamoto has quit IRC | 09:09 | |
| *** yamamoto has joined #openstack-helm | 09:54 | |
| *** ljjjustin has quit IRC | 10:09 | |
| *** yamamoto has quit IRC | 10:18 | |
| osh-chatbot4 | <abhishekjaiswal> help needed, keystone stucked, | 10:52 |
| osh-chatbot4 | <abhishekjaiswal> Initialized False - an hour ContainersNotInitialized containers with incomplete status: [init] Ready False - an hour ContainersNotReady containers with unready status: [keystone-api] PodScheduled True - an hour - - | 10:52 |
| osh-chatbot4 | <abhishekjaiswal> File uploaded https://kubernetes.slack.com/files/U6CG24GSK/F8KHPGNTC/screen_shot_2017-12-27_at_4.22.40_pm.png / https://slack-files.com/T09NY5SBT-F8KHPGNTC-c5148900af | 10:54 |
| osh-chatbot4 | <abhishekjaiswal> env: minikube | 10:55 |
| *** yamamoto has joined #openstack-helm | 11:05 | |
| *** yamamoto has quit IRC | 11:09 | |
| *** yamamoto has joined #openstack-helm | 11:11 | |
| osh-chatbot4 | <abhishekjaiswal> File uploaded https://kubernetes.slack.com/files/U6CG24GSK/F8KN0Q2TF/-.txt / https://slack-files.com/T09NY5SBT-F8KN0Q2TF-8f85ca3e2a | 11:16 |
| *** yamamoto has quit IRC | 11:30 | |
| *** yamamoto has joined #openstack-helm | 11:34 | |
| *** yamamoto has quit IRC | 11:51 | |
| *** yamamoto has joined #openstack-helm | 12:13 | |
| *** MarkBaker has joined #openstack-helm | 12:17 | |
| *** MarkBaker has quit IRC | 12:33 | |
| openstackgerrit | Matt McEuen proposed openstack/openstack-helm master: [WIP] Add Swift Chart https://review.openstack.org/508780 | 13:05 |
| openstackgerrit | Matt McEuen proposed openstack/openstack-helm master: Fixed ceph mon resource definition https://review.openstack.org/530217 | 13:12 |
| osh-chatbot4 | <abhishekjaiswal> can anyone help on mariadb is not coming up , deploying in minikube environment | 13:19 |
| osh-chatbot4 | <abhishekjaiswal> WSREP: Failed to resolve tcp://mariadb-1.mariadb-discovery:4567 | 13:19 |
| *** yangyapeng has quit IRC | 13:37 | |
| *** mdih has joined #openstack-helm | 13:49 | |
| *** mdih has quit IRC | 13:57 | |
| *** yangyapeng has joined #openstack-helm | 14:05 | |
| mattmceuen | abhishekjaiswal, are all your your mariadb and ceph pods running and healthy? Can you share output of `kubectl get pods --all-namespaces` ? | 14:28 |
| *** yamamoto_ has joined #openstack-helm | 14:43 | |
| *** yamamoto has quit IRC | 14:47 | |
| portdirect | using ceph in minikube probably wont work | 14:51 |
| portdirect | in addition to the command mattmceuen suggested could you ssh into the minikube vm, and see if the ceph rbd kernel module is available | 14:52 |
| portdirect | if its not - then we can talk through running with a local volume, or nfs for mariadb :) | 14:53 |
| *** yangyapeng has quit IRC | 15:00 | |
| *** yangyapeng has joined #openstack-helm | 15:00 | |
| osh-chatbot4 | <abhishekjaiswal> @portdirect can you provide me steps to start a local volume for mariadb | 15:10 |
| portdirect | sure | 15:10 |
| osh-chatbot4 | <abhishekjaiswal> File uploaded https://kubernetes.slack.com/files/U6CG24GSK/F8K0CTGKA/-.txt / https://slack-files.com/T09NY5SBT-F8K0CTGKA-032e8a5c7c | 15:10 |
| osh-chatbot4 | <abhishekjaiswal> @portdirect even mass is not coming up | 15:12 |
| osh-chatbot4 | <portdirect> can you run `helm ls`? | 15:12 |
| osh-chatbot4 | <abhishekjaiswal> File uploaded https://kubernetes.slack.com/files/U6CG24GSK/F8KPZUA93/-.txt / https://slack-files.com/T09NY5SBT-F8KPZUA93-ab2a89a315 | 15:13 |
| osh-chatbot4 | <portdirect> did you label the nodes? | 15:13 |
| osh-chatbot4 | <abhishekjaiswal> yes | 15:14 |
| osh-chatbot4 | <portdirect> anyway - so can you remove the mairadb deployment: `helm delete --purge mariadb` | 15:14 |
| osh-chatbot4 | <portdirect> and then deploy as follows: | 15:16 |
| osh-chatbot4 | <portdirect> ``` helm install ./mariadb \ --namespace=ucp \ --name=mariadb \ --set pod.replicas.server=1 \ --set volume.enabled=false ``` | 15:16 |
| osh-chatbot4 | <portdirect> which should get you a db up with a non-persistant backend | 15:17 |
| osh-chatbot4 | <abhishekjaiswal> why this new namespace will this not work with openstack namepsce | 15:18 |
| osh-chatbot4 | <abhishekjaiswal> @portdirect can you help me with maas and drydock | 15:21 |
| mattmceuen | do the maas logs have any telltale errors? | 15:22 |
| osh-chatbot4 | <abhishekjaiswal> Failed to pull image "quay.io/attcomdev/maas-cache:master": rpc error: code = Unknown desc = Error response from daemon: unauthorized: access to the requested resource is not authorized Error syncing podError syncing pod | 15:24 |
| osh-chatbot4 | <abhishekjaiswal> maas-region-o | 15:24 |
| osh-chatbot4 | <abhishekjaiswal> i build that maas image on my host machine, and then did make charts | 15:25 |
| *** marst has joined #openstack-helm | 15:31 | |
| osh-chatbot4 | <abhishekjaiswal> Failed to pull image "quay.io/attcomdev/maas-cache:master": rpc error: code = Unknown desc = Error response from daemon: unauthorized: access to the requested resource is not authorized Error syncing podError syncing pod | 15:31 |
| osh-chatbot4 | <abhishekjaiswal> keystone bootstrap and keystone api stuck on PODINIT | 15:44 |
| osh-chatbot4 | <abhishekjaiswal> any help | 15:44 |
| osh-chatbot4 | <abhishekjaiswal> '"keystone-api" in pod "keystone-api-74c774d448-c4nsg" is waiting to start: PodInitializing' | 15:46 |
| mattmceuen | I've never installed the maas chart, so I'm not an expert - I've sent your error to sthussey in case he's seen it before | 15:47 |
| osh-chatbot4 | <abhishekjaiswal> @mattmceuen no i am talking about keystone | 15:48 |
| osh-chatbot4 | <abhishekjaiswal> i am trying to install keystone chart | 15:48 |
| mattmceuen | what do you see with `kubectl -n openstack describe pod keystone-api-74c774d448-c4nsg` ? | 15:49 |
| osh-chatbot4 | <abhishekjaiswal> File uploaded https://kubernetes.slack.com/files/U6CG24GSK/F8K0TE0MN/-.txt / https://slack-files.com/T09NY5SBT-F8K0TE0MN-f26d974062 | 15:50 |
| osh-chatbot4 | <abhishekjaiswal> started container | 15:50 |
| mattmceuen | Re: mariadb -- it looks like above you installed it in the `ucp` namespace for UCP purposes, have you also installed a copy in `openstack` namespace for OpenStack purposes? | 15:51 |
| osh-chatbot4 | <abhishekjaiswal> no i have mariadb in openstack namespace | 15:51 |
| osh-chatbot4 | <abhishekjaiswal> File uploaded https://kubernetes.slack.com/files/U6CG24GSK/F8LN2PXTR/-.txt / https://slack-files.com/T09NY5SBT-F8LN2PXTR-1157b75e90 | 15:52 |
| mattmceuen | please share the full output of the kubectl above, as well as `kubectl get pods --all-namespaces` | 15:52 |
| osh-chatbot4 | <abhishekjaiswal> output of `helm ls --all` | 15:52 |
| osh-chatbot4 | <abhishekjaiswal> File uploaded https://kubernetes.slack.com/files/U6CG24GSK/F8K4ZFF25/-.txt / https://slack-files.com/T09NY5SBT-F8K4ZFF25-bb3604b4f0 | 15:53 |
| osh-chatbot4 | <abhishekjaiswal> so mariadb is in openstack | 15:54 |
| mattmceuen | and `kubectl -n openstack describe pod keystone-bootstrap-dds67` | 15:55 |
| osh-chatbot4 | <abhishekjaiswal> File uploaded https://kubernetes.slack.com/files/U6CG24GSK/F8KQFA5TP/-.txt / https://slack-files.com/T09NY5SBT-F8KQFA5TP-e4e51c6218 | 15:55 |
| osh-chatbot4 | <abhishekjaiswal> ` Normal Started 15m kubelet, minikube Started container` | 15:56 |
| osh-chatbot4 | <mattmceuen> The text above is truncated - can you grab the whole output? | 15:56 |
| osh-chatbot4 | <abhishekjaiswal> File uploaded https://kubernetes.slack.com/files/U6CG24GSK/F8K50TGMP/-.txt / https://slack-files.com/T09NY5SBT-F8K50TGMP-5b30c40b93 | 15:56 |
| osh-chatbot4 | <abhishekjaiswal> i have not installeed ceph and my environment is minikube, | 15:58 |
| osh-chatbot4 | <abhishekjaiswal> will i have to pass `volume.enabled=false` | 15:58 |
| srwilkers | for any container that's in an init state, you can get the logs of the init container with `kubectl logs <pod name> -n <namespace> -c <init>` | 15:59 |
| srwilkers | i dont see memcached in the list of the pods you have running, and memcached is one of the dependencies the keystone chart will wait for before entrypoint marks it as ready to proceed | 16:00 |
| srwilkers | https://github.com/openstack/openstack-helm/blob/master/keystone/values.yaml#L74-L75 | 16:00 |
| *** yamamoto_ has quit IRC | 16:00 | |
| osh-chatbot4 | <abhishekjaiswal> @srwilkers what should i pass with `<init>` | 16:02 |
| srwilkers | strike the angulars -- just do `-c init` | 16:02 |
| osh-chatbot4 | <abhishekjaiswal> and will it not resolved it dependency for memcache | 16:02 |
| osh-chatbot4 | <abhishekjaiswal> it says Resolving dependency Service memcached in namespace openstack failed: endpoints "memcached" not found | 16:03 |
| srwilkers | right | 16:04 |
| srwilkers | so, you'll need to install the memcache chart | 16:04 |
| srwilkers | as kubernetes entrypoint will not complete until it verifies there's an endpoint that can be resolved for the services defined as dependencies for a given chart (keystone in this case) | 16:04 |
| osh-chatbot4 | <mattmceuen> abhishekjaiswal - for getting the dependencies right, I suggest walking through the dev install guide - https://docs.openstack.org/openstack-helm/latest/install/developer/all-in-one.html Sine you're using minikube rather than the recommended kubeadm, there will be some differences (like the need to avoid persistent mariadb that you discovered) | 16:06 |
| *** yamamoto has joined #openstack-helm | 16:07 | |
| osh-chatbot4 | <abhishekjaiswal> @srwilkers thanks after memcache it came up | 16:10 |
| srwilkers | no problem -- happy to help | 16:10 |
| *** yamamoto has quit IRC | 16:12 | |
| *** yamamoto has joined #openstack-helm | 16:22 | |
| *** yamamoto has quit IRC | 16:27 | |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: Move fluentbit and fluentd configs to values.yaml https://review.openstack.org/527489 | 16:31 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: Move fluentbit and fluentd configs to values.yaml https://review.openstack.org/527489 | 16:35 |
| openstackgerrit | Elise Eiden proposed openstack/openstack-helm master: [WIP] Add Tempest Chart https://review.openstack.org/498929 | 16:58 |
| *** yamamoto has joined #openstack-helm | 17:07 | |
| *** yamamoto has quit IRC | 17:12 | |
| *** yamamoto has joined #openstack-helm | 17:22 | |
| *** mwynne has joined #openstack-helm | 18:06 | |
| mwynne | Hi guys. I'm following the AIO guide, and I'm wondering if the IP addresses in the "Depoloy Ceph" section "--set network.public=172.17.0.1/16" etc, need to be modified to match my IP address? | 18:08 |
| openstackgerrit | Merged openstack/openstack-helm-infra master: Add prometheus annotations to calico-nodes for felix metrics https://review.openstack.org/530129 | 18:14 |
| *** yamamoto has joined #openstack-helm | 18:22 | |
| d|k | @mynne yep. ceph uses host networking, so you'll want to pick a network on some attached interface. | 18:24 |
| d|k | ooops, misdirected that. @mwynne: yep, and because ceph uses host networking, you'll want to pick a net on some attached interface. | 18:25 |
| d|k | @mwynne also: you probably want to use the same network for OSD_CLUSTER_NETWORK and OSD_PUBLIC_NETWORK unless you've got a strong reason not to. | 18:26 |
| *** yamamoto has quit IRC | 18:28 | |
| mwynne | d|k: Ok, sounds good. Thanks :) | 18:29 |
| mwynne | d|k: If I used a wrong IP can I just fix it and re-run the scripts? Or will I need to start from scratch? | 18:30 |
| d|k | you SHOULD be fine doing a helm delete --purge and then a reinstall. | 18:30 |
| mwynne | d|k: Cool, thanks. | 18:31 |
| d|k | that's worked for me when i've botched precisely that configuration (though in my case it was because they were empty; can't imagine that'd be a significant difference.) | 18:32 |
| mwynne | d|k: What's the "release" I want to delete? | 18:32 |
| mwynne | All of them I'm assuming? | 18:32 |
| mwynne | Everything from `helm list`? | 18:32 |
| d|k | if you've already tried to install mariadb and the rest of openstack, you'll _probably_ want to delete those guys | 18:33 |
| d|k | but at a minimum, you'll need to delete ceph | 18:33 |
| mwynne | d|k: Or, actually, just everything from ceph and after? | 18:33 |
| mwynne | yeah | 18:33 |
| mwynne | sounds good | 18:33 |
| d|k | what you said | 18:33 |
| mwynne | mariadb failed | 18:33 |
| mwynne | So, yeah.. | 18:33 |
| mwynne | Thanks | 18:33 |
| d|k | yeah, i'd expect that. mariadb uses ceph-backed PVCs for the database, and it'll be profoundly unhappy if ceph isn't functional. | 18:34 |
| mwynne | d|k: Does that IP/network need to be public? | 18:41 |
| d|k | nope, only the participants in the ceph cluster need to talk to it. | 18:51 |
| d|k | so it should work just fine to, for example, use a privately-addressed subinterface. | 18:51 |
| d|k | but even if only a subset of your hosts will be running ceph, you DO need to ensure that anything that runs a pod that might USE ceph can reach that net. | 18:52 |
| mwynne | d|k: But.... But why is it called public :S haha | 18:52 |
| d|k | bleah, accident of history | 18:52 |
| d|k | it is possible to separate them so that, in particular, the heavier-IO traffic can be hived off to a different network. | 18:53 |
| mwynne | So It should be safe to just use my only NIC's IP? | 18:54 |
| d|k | yep! | 18:54 |
| mwynne | This is mostly just a test AIO deployment so I can get familiar with it. (As suggested here) | 18:54 |
| d|k | makes sense. if you're not overly worried about performance or security, i don't think there should really be a material difference between using the same interface for everything. | 18:56 |
| d|k | ... and using a dedicated ceph interface. | 18:56 |
| mwynne | d|k: wait. I need a dedicated ceph interface? | 18:58 |
| d|k | nonono, sorry. | 18:59 |
| d|k | i just meant to append that to the previous statement: there shouldn't be a material difference between using a shared and separate interface. | 19:00 |
| mwynne | Ah, ok. | 19:01 |
| mwynne | d|k: Do I need a seperate subnet for OSH_BR_EX_ADDR and OSH_EXT_SUBNET? | 19:02 |
| mwynne | d|k: The example uses 172.24.4.0/24 which isn't referenced anywhere else afaik | 19:05 |
| d|k | i shouldn't think so. if you've got one interface on your boxes facing out toward the internet, that's the net you should use for anything that needs a native address, i think. | 19:06 |
| mwynne | hmm, ok. | 19:07 |
| d|k | obviously, your CNI pod and service networks likely want to be different from that, though. | 19:07 |
| d|k | well, hm, actually: | 19:07 |
| d|k | the br_ex addr you're going to need to be careful with if you've only got one interface. | 19:08 |
| mwynne | d|k: Ok, well I have more interfaces. | 19:08 |
| d|k | just as in a native openstack deployment, when you drop that interface into br_ex, it'll fall off the net, at least until the address is added to the bridge. | 19:08 |
| mwynne | Ok, so I have an interface that I can use as my provider network. | 19:09 |
| d|k | probably sensible to keep that separate. it'd _work_ to have them all on the same interface, but it'd probably take some planning to make sure that bringing up br_ex didn't break the rest of your net. | 19:10 |
| mwynne | Ok It would be nice if this documentation specified a min number of NICs etc like other deployment guides typically do | 19:12 |
| *** bryan_att has quit IRC | 19:12 | |
| *** zioproto has quit IRC | 19:12 | |
| *** srwilkers has quit IRC | 19:12 | |
| *** v1k0d3n has quit IRC | 19:12 | |
| *** larainema has quit IRC | 19:12 | |
| *** jayahn has quit IRC | 19:13 | |
| *** srwilkers_ has quit IRC | 19:13 | |
| *** alanmeadows has quit IRC | 19:13 | |
| *** powerds0111 has quit IRC | 19:13 | |
| *** hogepodge has quit IRC | 19:13 | |
| *** portdirect has quit IRC | 19:13 | |
| *** cargonza has quit IRC | 19:14 | |
| *** aimeeu has quit IRC | 19:14 | |
| *** kong has quit IRC | 19:14 | |
| *** rwellum has quit IRC | 19:14 | |
| *** wxy has quit IRC | 19:14 | |
| *** serverascode has quit IRC | 19:14 | |
| *** mattmceuen has quit IRC | 19:14 | |
| d|k | fair point. i do think the "min number" _is_ actually 1, but it likely is a lot cleaner and easier to use at least two. | 19:15 |
| mwynne | d|k: Yeah. Or, since it's an AIO guide, just decide on a min for the guide, and stick to it. | 19:15 |
| d|k | ... and don't rely on what i say about this overmuch: while i've deployed OSH a mess of times, i'm not the expert other folks are. | 19:16 |
| mwynne | Something like that is nice, and would mean people like me wouldn't have to come in here and bother you all... | 19:16 |
| *** andreaf has quit IRC | 19:16 | |
| d|k | but then who would we TALK to? | 19:18 |
| *** andreaf has joined #openstack-helm | 19:18 | |
| mwynne | haha fair enough | 19:19 |
| mwynne | As long as you're happy to have to answer these questions, then I'm fine haha | 19:19 |
| *** yamamoto has joined #openstack-helm | 19:24 | |
| *** yamamoto has quit IRC | 19:29 | |
| mwynne | d|k: ingress-api and ingress-error-pages now have IPs on different subnets... | 19:39 |
| *** jayahn has joined #openstack-helm | 19:39 | |
| *** alanmeadows has joined #openstack-helm | 19:39 | |
| *** ChanServ sets mode: +o alanmeadows | 19:39 | |
| *** srwilkers has joined #openstack-helm | 19:39 | |
| mwynne | api is using the docker0 bridge ip "172.17.0.1" | 19:39 |
| mwynne | And mariadb isn't coming up... 0/1 ready | 19:40 |
| *** kong has joined #openstack-helm | 19:40 | |
| *** bryan_att has joined #openstack-helm | 19:42 | |
| *** serverascode has joined #openstack-helm | 19:42 | |
| *** wxy has joined #openstack-helm | 19:43 | |
| *** cargonza has joined #openstack-helm | 19:43 | |
| *** aimeeu has joined #openstack-helm | 19:43 | |
| *** hogepodge has joined #openstack-helm | 19:53 | |
| *** zioproto has joined #openstack-helm | 19:53 | |
| *** v1k0d3n has joined #openstack-helm | 19:53 | |
| *** ChanServ sets mode: +o v1k0d3n | 19:53 | |
| *** srwilkers_ has joined #openstack-helm | 19:54 | |
| mwynne | d|k: Do you know if the pod_subnet needs to be the same as a subnet I have configured on my host? | 19:59 |
| d|k | mwynne: i'm a lot less familiar with the AIO case than the multinode case, but at least in the latter, the pod subnet is typically distinct from any net attached to the host -- a truly private network for the pods. | 20:06 |
| *** portdirect has joined #openstack-helm | 20:15 | |
| *** ChanServ sets mode: +o portdirect | 20:15 | |
| *** yamamoto has joined #openstack-helm | 20:25 | |
| *** yamamoto has quit IRC | 20:30 | |
| openstackgerrit | Merged openstack/openstack-helm master: Fixed ceph mon resource definition https://review.openstack.org/530217 | 20:36 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Dev-Deploy: Setup iptables for use with docker >= 1.13.1 https://review.openstack.org/530262 | 20:48 |
| *** rwellum has joined #openstack-helm | 21:02 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Dev-Deploy: Setup iptables for use with docker >= 1.13.1 https://review.openstack.org/530262 | 21:14 |
| openstackgerrit | Merged openstack/openstack-helm master: Dev-Deploy: Deploy Nova with native kvm when possible https://review.openstack.org/530144 | 21:18 |
| *** powerds0111 has joined #openstack-helm | 21:25 | |
| *** larainema has joined #openstack-helm | 21:26 | |
| *** yamamoto has joined #openstack-helm | 21:27 | |
| openstackgerrit | Merged openstack/openstack-helm-infra master: Flannel: Fix RBAC definitions https://review.openstack.org/530132 | 21:28 |
| openstackgerrit | Merged openstack/openstack-helm-infra master: Move kibana to OSH infra https://review.openstack.org/529068 | 21:28 |
| *** mattmceuen has joined #openstack-helm | 21:30 | |
| mwynne | d|k: Sounds good, thanks. | 21:30 |
| mwynne | That's what I assumed | 21:30 |
| *** yamamoto has quit IRC | 21:32 | |
| mwynne | d|k: How can I debug a pod that isn't coming up? | 21:33 |
| mwynne | I'm just following the AIO guide but it's not coming up for me. | 21:33 |
| mwynne | I've increased the timeout but it's been a long time, and nothing's happening. | 21:34 |
| mwynne | d|k: No logs or anything either that I can find... | 21:40 |
| mwynne | portdirect: Are you around at all? | 21:55 |
| portdirect | mwynne: o/ | 21:56 |
| mwynne | Howdy. I'm looking into the AIO deployment we talked about last week I think. | 21:57 |
| mwynne | You said I could just spin up a 16.04 VM and run the scripts essentially. | 21:57 |
| mwynne | `mariadb-0 0/1 Running 1 1h 192.168.98.210 controller-9` | 21:57 |
| portdirect | should be able to for a single node | 21:57 |
| mwynne | This pod won't come up to save my life | 21:57 |
| mwynne | Any ideas? | 21:58 |
| portdirect | can you do a `kubectl describe -n openstack pods mariadb-0` | 21:58 |
| mwynne | Warning Unhealthy 1m (x146 over 1h) kubelet, controller-9 Readiness probe failed: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES) | 21:58 |
| mwynne | Could not SELECT 1 | 21:58 |
| portdirect | hmm | 21:59 |
| portdirect | lets clean up and try again | 21:59 |
| portdirect | can you `helm delete --purge mariadb` | 21:59 |
| portdirect | then we will clean up the pv and pvc | 21:59 |
| mwynne | portdirect: Ok, it's deleted | 22:00 |
| portdirect | `kubectl delete -n openstack pvc mysql-data-mariadb-0` | 22:00 |
| mwynne | portdirect: Done | 22:00 |
| portdirect | cool can you check to see if `kubectl get pv` returns anything | 22:00 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: Move fluentbit and fluentd configs to values.yaml https://review.openstack.org/527489 | 22:00 |
| portdirect | if it does delete the `pv` | 22:01 |
| mwynne | One entry | 22:01 |
| mwynne | want to see it? | 22:01 |
| mwynne | http://paste.openstack.org/show/632595/ | 22:01 |
| portdirect | nice, lets remove it | 22:01 |
| portdirect | `kubectl delete pv pvc-ad0abc9e-eb45-11e7-ad57-fa163e853c77` | 22:02 |
| mwynne | Error from server (NotFound): persistentvolumes "pvc-ad0abc9e-eb45-11e7-ad57-fa163e853c77" not found | 22:02 |
| mwynne | But now, get pv returns `no resources found` | 22:02 |
| mwynne | Which is good, I'm assuming? | 22:02 |
| portdirect | odd - but lets take a win when we can :) | 22:03 |
| mwynne | portdirect: What's next? Running 06-mariadb.sh? | 22:04 |
| portdirect | so lets try again - https://docs.openstack.org/openstack-helm/latest/install/developer/all-in-one.html#deploy-mariadb | 22:04 |
| portdirect | yes | 22:04 |
| portdirect | it should take about 5 mins for the db to come up | 22:04 |
| mwynne | Ok, it's running now. | 22:05 |
| mwynne | Anything I can do to monitor it? | 22:05 |
| portdirect | if you open another terminal i usually have `watch kubectl get --all-namespaces pods` running to keep an eye on things | 22:05 |
| mwynne | Perfect, thanks. | 22:06 |
| portdirect | and once its running `kubectl logs -n openstack -f mariadb-0` should tail the logs | 22:07 |
| portdirect | for reference this is what my mairadb logs look like: http://paste.openstack.org/show/632596/ | 22:08 |
| portdirect | it so happens I'm literally at the point of deploying it in a new env :D | 22:08 |
| mwynne | [Warning] Access denied for user 'root'@'localhost' (using password: YES) | 22:09 |
| mwynne | Started watching the logs and got that ^ | 22:09 |
| mwynne | portdirect: And I just get that over and over now | 22:11 |
| portdirect | :( | 22:11 |
| portdirect | what is the spec of your vm? | 22:11 |
| mwynne | 16GB ram, 8 cpus | 22:12 |
| mwynne | 80 gig drive | 22:12 |
| portdirect | could you paste the full output from the mariadb pod? | 22:14 |
| mwynne | you want the output of `kubectl logs -n openstack -f mariadb-0`? | 22:15 |
| portdirect | yup | 22:15 |
| mwynne | I just killed it before you asked, it's re-building now... | 22:16 |
| portdirect | did you clean up the pv? | 22:17 |
| mwynne | yeah | 22:17 |
| portdirect | cool | 22:17 |
| mwynne | followed your steps from before | 22:17 |
| mwynne | :) | 22:17 |
| portdirect | and just to double check, your working from master? | 22:17 |
| mwynne | I've literally just followed the steps from that guide | 22:18 |
| mwynne | so, yeah. | 22:18 |
| mwynne | Your branch is up-to-date with 'origin/master'. | 22:18 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: WIP: Ceph: update dependency logic to be more strict https://review.openstack.org/530266 | 22:20 |
| mwynne | portdirect: I should be working from master? | 22:22 |
| mwynne | portdirect: It's going again. If if fails I'll get you the logs. | 22:24 |
| portdirect | mwynne: yeah - master is gated, so should (oh no - you know I just jinxed it) always work | 22:24 |
| mwynne | portdirect: http://paste.openstack.org/show/632602/ | 22:25 |
| mwynne | No dice :( | 22:25 |
| portdirect | mwynne - thats super weird | 22:27 |
| portdirect | thats the full log? | 22:27 |
| portdirect | https://www.irccloud.com/pastebin/eGaXlunc/ | 22:27 |
| portdirect | ^^ even on restart I'd expect to see the following lines :/ | 22:28 |
| mwynne | hmm | 22:28 |
| mwynne | let me check | 22:28 |
| *** yamamoto has joined #openstack-helm | 22:29 | |
| mwynne | There's more :) | 22:30 |
| mwynne | One sec | 22:30 |
| mwynne | portdirect: http://paste.openstack.org/show/632603/ | 22:31 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: WIP: Ceph: update dependency logic to be more strict https://review.openstack.org/530266 | 22:34 |
| *** yamamoto has quit IRC | 22:35 | |
| mwynne | portdirect: Any ideas? | 22:36 |
| portdirect | mwynne: for some reason it looks like its reusing the same pv - and removing it it having no joy :/ | 22:36 |
| mwynne | portdirect: I've also started from scratch several times | 22:37 |
| mwynne | All resulting in the same thing | 22:37 |
| mwynne | (starting from scratch == new vm each time) | 22:37 |
| portdirect | ok - lets use a horrible hack to get you through this stage | 22:37 |
| portdirect | clean up again | 22:37 |
| portdirect | and this time use the following to launch mariadb | 22:38 |
| portdirect | https://www.irccloud.com/pastebin/EFxkggUz/ | 22:39 |
| portdirect | this will just use a dir that lasts the lifespan of the pod, rather than ceph to back the db | 22:39 |
| mwynne | Ok, that's running now. | 22:42 |
| mwynne | portdirect: same thing | 22:43 |
| mwynne | access denied for root | 22:43 |
| portdirect | mwynne: I must confess I'm pretty confused now | 22:44 |
| portdirect | could you paste the full output from `kubectl describe pod -n openstack mariadb-0` | 22:47 |
| portdirect | though at this stage I'm pretty stumped | 22:47 |
| mwynne | portdirect: http://paste.openstack.org/show/632605/ | 22:48 |
| portdirect | just to sanity check myself, did you change to root password or any other value in the values.yaml? | 22:51 |
| mwynne | portdirect: Nope | 22:53 |
| mwynne | portdirect: It looks like that password might not be getting changed to root|password | 22:57 |
| mwynne | if that's what's supposed to be happening | 22:57 |
| portdirect | mwynne: as a last ditch attempt could you up the value of this: https://github.com/openstack/openstack-helm/blob/master/mariadb/templates/statefulset.yaml#L74 to 300 | 23:05 |
| mwynne | portdirect: Ok. Deleting and trying again.. | 23:08 |
| mwynne | portdirect: There isn't anything else I need to do that isn't documented in that page, is there? | 23:08 |
| mwynne | the AIO page | 23:08 |
| portdirect | if that doesnt work I'm really out of ideas :/ | 23:08 |
| portdirect | there should not be | 23:08 |
| *** marst has quit IRC | 23:08 | |
| portdirect | its based of this: https://github.com/portdirect/sydney-workshop | 23:09 |
| portdirect | which I used in sydney to get people up and running on 110 r3.xlarge vms on ec2 that were provisioned using this cloud-init script: https://github.com/portdirect/sydney-workshop/blob/master/provision/cloud-init.sh | 23:10 |
| mwynne | Ok, looks reasonable. | 23:11 |
| mwynne | running with the increased timeout now.. let's see :S | 23:16 |
| mwynne | portdirect: If I use `mysql_secure_installation` to change the root password to 'password', I can run /tmp/readiness.sh successfully. Otherwise it fails... | 23:23 |
| anticw | mwynne: wading is late here sorry, so forgive me if i ask stupid questions | 23:25 |
| portdirect | even with that timeout increased? | 23:25 |
| anticw | mwynne: pvc/pv working as expected? | 23:25 |
| mwynne | with timeout: yes | 23:25 |
| portdirect | anticw: it was, but I suggested disabling it to rule that out. | 23:25 |
| anticw | would need a 'general' sc then | 23:25 |
| anticw | mwynne: `kubectl get pv` looks ok? | 23:26 |
| mwynne | It came up this time. | 23:27 |
| mwynne | It's running... | 23:27 |
| mwynne | finally | 23:27 |
| anticw | is it possible this system is very slow? | 23:27 |
| anticw | like a laptop or something? | 23:27 |
| mwynne | anticw: *sigh*... who knows. Possible. | 23:27 |
| mwynne | It's running in a VM in an openstack stack | 23:28 |
| mwynne | with some good compute nodes | 23:28 |
| mwynne | Might be busy now though | 23:28 |
| mwynne | I'll have to check on it. That's very possible though. Not running on a laptop though :) | 23:28 |
| anticw | some of this causes a lot of 'seeky synchronious' io | 23:29 |
| portdirect | from looking at your logs, I think you may be starved for iops... | 23:29 |
| anticw | which really grinds hard if you have rotating disks and is sometimes worse with raid | 23:29 |
| portdirect | hence why increasing the readiness from 60s to 300, and removing ceph got you going. | 23:30 |
| mwynne | portdirect: I meant to ask about the Neutron part: OSH_BR_EX_ADDR="172.24.4.1/24"... can that be an ip address that would be on the same subnet as the interface that br-ex connects to? | 23:30 |
| mwynne | Ah, ok. That would make sense. We have spinners + raid | 23:30 |
| mwynne | Great... | 23:30 |
| *** yamamoto has joined #openstack-helm | 23:30 | |
| portdirect | mwynne: yep | 23:31 |
| anticw | mwynne: what happens if that db operations tend to be small and synchronous ... which means (in some not all) cases you have really expensive RMW operations | 23:32 |
| mwynne | portdirect: Where do I set which nic connects to br-ex? I have 3 nics on this VM. | 23:32 |
| portdirect | the guide is written assuming the use of the standard docker0 iface ip to simplify network setup, and this heat template for your external network: https://github.com/openstack/openstack-helm/blob/master/tools/gate/files/heat-public-net-deployment.yaml | 23:32 |
| mwynne | anticw: Good to know. Thanks :) | 23:32 |
| portdirect | it does not assume that you'll setup a provider network, but just nats from br-ex to the outside world | 23:33 |
| mwynne | Ahh, ok. | 23:33 |
| mwynne | So does it matter which IP I give br-ex then? | 23:33 |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: Retrigger the doc publish job https://review.openstack.org/530270 | 23:34 |
| anticw | fwiw i don't run using docker0 most of the tiem | 23:34 |
| anticw | time | 23:34 |
| portdirect | you can change it to whatever you like, provided you update this accordingly as well: https://github.com/openstack/openstack-helm/blob/master/tools/deployment/developer/19-use-it.sh#L22-L23 | 23:34 |
| anticw | i set network.interface.external to the physical interface that will be used by libvirt bridges | 23:35 |
| *** yamamoto has quit IRC | 23:37 | |
| mwynne | Thanks for the help guys. I appreciate it. | 23:39 |
| mwynne | Keystone's failing now too haha | 23:40 |
| mwynne | Probably for similar reasons as mysql | 23:40 |
| anticw | db error? | 23:40 |
| anticw | in some (not all) cases those will restart and come right | 23:40 |
| anticw | db-sync? | 23:40 |
| mwynne | (2003, "Can't connect to MySQL server on 'mariadb.openstack.svc.cluster.local' ([Errno 111] Connection refused)") | 23:41 |
| anticw | check mariadb-0 pod | 23:41 |
| anticw | describe ... did it get restarted or glitch? | 23:41 |
| mwynne | doesn't look like it | 23:42 |
| mwynne | but I can't connect to it in the docker container either | 23:42 |
| anticw | docker ps doesn't see it? | 23:44 |
| mwynne | I can connect to it with docker exec | 23:45 |
| mwynne | root@mariadb-0:/# mysql | 23:45 |
| mwynne | ERROR 2003 (HY000): Can't connect to MySQL server on 'localhost' (111 "Connection refused") | 23:45 |
| mwynne | That wasn't failing a while ago | 23:45 |
| anticw | dmesg|tail -20 | 23:47 |
| anticw | make sure that doesn't look bad | 23:47 |
| anticw | if it looks iffy maybe pastebin all dmesg somewhere, check for OOM | 23:47 |
| mwynne | anticw: in the container? | 23:48 |
| mwynne | or host | 23:48 |
| anticw | host | 23:48 |
| mwynne | aufs au_opts_verify:1597:dockerd[13458]: dirperm1 breaks the protection by the permission bits on the lower branch | 23:48 |
| mwynne | Lots of those | 23:48 |
| anticw | nothing else? | 23:49 |
| anticw | those are 'normal' (for some irritating definition of normal) | 23:49 |
| anticw | i'm guessing you have ub 1604 host(s) ? | 23:49 |
| mwynne | 1604 yeah | 23:51 |
| mwynne | nothing else | 23:51 |
| mwynne | anticw: http://paste.openstack.org/show/632613/ | 23:52 |
| mwynne | in syslog | 23:52 |
| anticw | "Stat fs failed. Error: no such file or directory" isn't something i've seen before | 23:54 |
| anticw | no storage weirdness? nothing got unmounted or wrecked? | 23:54 |
| *** openstackgerrit has quit IRC | 23:54 | |
| mwynne | not from what I can tell | 23:55 |
| anticw | kubeget get pods works? | 23:55 |
| anticw | gut docker ps doesn't? | 23:55 |
| anticw | err, kubectl ... whatever, i dumb you have to translate from idiot to real sorry sometimes | 23:56 |
| mwynne | they both work | 23:56 |
| mwynne | I can get to the container | 23:56 |
| mwynne | http://paste.openstack.org/show/632616/ | 23:56 |
| mwynne | This is running in the container for some reason... | 23:56 |
| mwynne | mysql logs are empty in the container | 23:59 |
| anticw | you can't connect to db? | 23:59 |
| anticw | still? | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
