| mwynne | no | 00:00 |
|---|---|---|
| mwynne | ERROR 2003 (HY000): Can't connect to MySQL server on 'localhost' (111 "Connection refused") | 00:00 |
| *** jaypipes has quit IRC | 00:00 | |
| anticw | telnet mariadb.openstack.svc.cluster.local 3306 | 00:00 |
| anticw | fails? | 00:00 |
| mwynne | telnet: Unable to connect to remote host: Connection refused | 00:02 |
| anticw | kubes thinks the pod is up and healthy though? | 00:02 |
| *** openstackgerrit has joined #openstack-helm | 00:17 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: WIP: Ceph: update dependency logic to be more strict https://review.openstack.org/530266 | 00:17 |
| *** yangyapeng has quit IRC | 00:22 | |
| *** yangyapeng has joined #openstack-helm | 00:23 | |
| *** yangyapeng has quit IRC | 00:27 | |
| *** yamamoto has joined #openstack-helm | 00:33 | |
| *** yamamoto has quit IRC | 00:39 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Ceph: update dependency logic to be more strict https://review.openstack.org/530266 | 00:45 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Dev-Deploy: Setup iptables for use with docker >= 1.13.1 https://review.openstack.org/530262 | 00:51 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Ceph: update dependency logic to be more strict https://review.openstack.org/530266 | 00:51 |
| mwynne | anticw: Yeah | 00:54 |
| mwynne | Sorry, I had to run and take care of something | 00:55 |
| mwynne | anticw: Weird. The container restarted and it's back up now | 01:00 |
| *** yangyapeng has joined #openstack-helm | 01:01 | |
| anticw | i would check to see that you're not badly IO starved which could be causing health checks to stab things | 01:01 |
| mwynne | Yeah, I will. | 01:01 |
| mwynne | If I want to re-run the keystone stuff do I just need to delete the pods? | 01:02 |
| mwynne | `helm delete --purge <keystone pods>`? | 01:02 |
| anticw | not exactly | 01:07 |
| *** yangyapeng has quit IRC | 01:07 | |
| anticw | helm will do it ... it will delete the charts ... and the resources | 01:07 |
| anticw | which wil delete the pods | 01:07 |
| anticw | deleting pods (directly) from k8s usually doesn't help as it puts them back | 01:07 |
| anticw | helm delete --purge | 01:07 |
| anticw | is probably what you want | 01:07 |
| anticw | with a chart name | 01:08 |
| mwynne | That's what I meant :) | 01:08 |
| mwynne | Like you said, translate from idiot... haha | 01:08 |
| mwynne | (1130, u"Host '192.168.98.226' is not allowed to connect to this MariaDB server") | 01:10 |
| mwynne | It never ends, does it? haha | 01:10 |
| anticw | that's from the mariadb-0 logs? | 01:12 |
| mwynne | No | 01:14 |
| mwynne | anticw: keystone | 01:14 |
| mwynne | Looks like external connections are disabled for mysql | 01:14 |
| anticw | telnet mariadb.openstack.svc.cluster.local 3306 | 01:15 |
| anticw | ? | 01:15 |
| *** yangyapeng has joined #openstack-helm | 01:15 | |
| *** yangyapeng has quit IRC | 01:15 | |
| mwynne | anticw: HHost '192.168.100.9' is not allowed to connect to this MariaDB serverConnection closed by foreign host. | 01:15 |
| mwynne | Nothing external works | 01:16 |
| *** yangyapeng has joined #openstack-helm | 01:16 | |
| mwynne | anticw: from the container I can connect to mysql | 01:16 |
| mwynne | the mysql container, that is. | 01:16 |
| mwynne | But not from any other container, or my host. | 01:16 |
| mwynne | I'm assuming portdirect might know more about this. | 01:17 |
| anticw | portdirect: rbac? | 01:17 |
| portdirect | shouldnt be :/ | 01:17 |
| portdirect | mwynne: what is the ip of your vm? | 01:17 |
| mwynne | 192.168.100.9 | 01:17 |
| portdirect | oh *expletive* | 01:18 |
| portdirect | so that we need to add a warning for | 01:18 |
| portdirect | calico will be trying to use 192.168.0.0/16 for its network | 01:18 |
| portdirect | and thats gonna cause trouble | 01:18 |
| mwynne | portdirect: Can I change that in pod_subnet: 192.168.0.0/16 | 01:18 |
| mwynne | ? | 01:19 |
| portdirect | you should be able to yes | 01:19 |
| mwynne | Perfect. I'll start from scratch again ;) | 01:19 |
| portdirect | mwynne: your patience is commendable dude, I'd buy you a drink if you were in STL | 01:19 |
| mwynne | I have to run out right now. I'll be looking at this again later though. Thanks for all yoru help today portdirect and anticw :) | 01:20 |
| mwynne | I really appreciate all the help | 01:20 |
| mwynne | I'd be happy to help with docs or something if I can | 01:20 |
| portdirect | mwynne: ++ | 01:20 |
| mwynne | Gotta run though. Have a good night/day :) | 01:20 |
| anticw | portdirect: we've hit this before, i wonder about a sanity checking chart? | 01:26 |
| portdirect | I was just about to add a note to the docs, but a chart would be nice | 01:26 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Horizon: move to kolla image with bugs fixed https://review.openstack.org/530279 | 01:29 |
| *** yamamoto has joined #openstack-helm | 01:35 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Dev-Docs: Add warning about IP ranges consumed by K8s and Calico https://review.openstack.org/530280 | 01:35 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Dev-Docs: Add warning about IP ranges consumed by K8s and Calico https://review.openstack.org/530280 | 01:36 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Dev-Docs: Add warning about IP ranges consumed by K8s and Calico https://review.openstack.org/530280 | 01:37 |
| *** yamamoto has quit IRC | 01:39 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Dev-Docs: Add warning about IP ranges consumed by K8s and Calico https://review.openstack.org/530280 | 01:44 |
| anticw | SamYaple: portdirect either of you have experience running docker/k8s without iptables? | 02:03 |
| portdirect | anticw: I've run it (the docker daemon) without before | 02:04 |
| portdirect | though never in anger... you dont need it enabled if your not gonna use docker itself | 02:05 |
| portdirect | and you can set up the tables for simple masquerade or snat pretty easily | 02:06 |
| portdirect | though you need them if you want to do the port binding fu automagily... | 02:06 |
| *** yamamoto has joined #openstack-helm | 02:35 | |
| *** yamamoto has quit IRC | 02:41 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: WIP: Cinder: Enable backup https://review.openstack.org/530286 | 02:56 |
| *** yamamoto has joined #openstack-helm | 03:37 | |
| *** yamamoto has quit IRC | 03:43 | |
| openstackgerrit | Merged openstack/openstack-helm master: Retrigger the doc publish job https://review.openstack.org/530270 | 04:25 |
| *** yamamoto has joined #openstack-helm | 04:39 | |
| *** yamamoto has quit IRC | 04:44 | |
| *** yangyapeng has quit IRC | 04:51 | |
| *** yangyapeng has joined #openstack-helm | 04:51 | |
| *** gagehugo has quit IRC | 04:59 | |
| *** yamamoto has joined #openstack-helm | 05:26 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: WIP: Cinder: Enable backup https://review.openstack.org/530286 | 05:41 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Glance: Fix RBAC object creation for storage-init job https://review.openstack.org/530293 | 05:50 |
| openstackgerrit | Renis Makadia proposed openstack/openstack-helm-infra master: Creating loopback devices through Ansible and Systemd with reboot support https://review.openstack.org/530302 | 06:30 |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: Updating security context for magnum chart https://review.openstack.org/529125 | 07:00 |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: [DO NOT MERGE] Barbican with Dogtag backend https://review.openstack.org/503399 | 07:02 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: WIP: Cinder: Enable backup https://review.openstack.org/530286 | 07:06 |
| anticw | portdirect: for this use case i do *not* want all the insanery that's done with iptables | 07:13 |
| portdirect | anticw: to clarify, you want no iptables fun anywhere? not from docker, or kube-proxy? | 07:14 |
| * portdirect is sad that anticw is not wanting to spend hours trying to trace chains of increasingly cryptic and convoluted iptablings | 07:16 | |
| *** yamamoto has quit IRC | 07:54 | |
| *** yamamoto has joined #openstack-helm | 07:55 | |
| *** yamamoto has quit IRC | 08:00 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: WIP: Cinder: Enable backup https://review.openstack.org/530286 | 08:06 |
| *** yamamoto has joined #openstack-helm | 08:18 | |
| *** yamamoto has joined #openstack-helm | 08:30 | |
| *** yamamoto has quit IRC | 08:35 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: WIP: Cinder: Enable backup https://review.openstack.org/530286 | 08:38 |
| *** yamamoto has joined #openstack-helm | 09:14 | |
| *** yamamoto has quit IRC | 09:19 | |
| osh-chatbot4 | <abhishekjaiswal> hi , i am trying new multinode setup with kubespray, i am little confused with these values `export OSD_CLUSTER_NETWORK=10.26.0.0/26 export OSD_PUBLIC_NETWORK=10.26.0.0/26` can any one tell me what value should be here my machines ips are | 09:47 |
| osh-chatbot4 | <abhishekjaiswal> 192.168.3.119 192.168.3.120 192.168.3.121 192.168.3.122 192.168.3.123 | 09:47 |
| *** yamamoto has joined #openstack-helm | 10:15 | |
| *** yamamoto has quit IRC | 10:21 | |
| openstackgerrit | Tin Lam proposed openstack/openstack-helm-infra master: RBAC: Fix issue with inclusion of empty jobs or daemonsets https://review.openstack.org/530337 | 10:32 |
| *** jayahn has quit IRC | 10:37 | |
| *** jayahn has joined #openstack-helm | 10:38 | |
| *** yamamoto has joined #openstack-helm | 11:17 | |
| *** yamamoto has quit IRC | 11:23 | |
| *** yamamoto has joined #openstack-helm | 11:57 | |
| *** yamamoto has quit IRC | 12:02 | |
| *** yangyapeng has quit IRC | 12:10 | |
| *** yangyapeng has joined #openstack-helm | 12:10 | |
| *** yamamoto has joined #openstack-helm | 12:14 | |
| *** yangyapeng has quit IRC | 12:15 | |
| *** yamamoto has joined #openstack-helm | 12:28 | |
| *** yamamoto has quit IRC | 12:33 | |
| *** yangyapeng has joined #openstack-helm | 12:38 | |
| *** jaypipes has joined #openstack-helm | 13:15 | |
| *** yamamoto has joined #openstack-helm | 13:30 | |
| *** yamamoto has quit IRC | 13:35 | |
| osh-chatbot4 | <portdirect> @abhishekjaiswal if those are the only interfaces on your machines, then you shoule use `192.168.3.0/24` for both the `OSD_CLUSTER_NETWORK` and `OSD_PUBLIC_NETWORK` | 14:14 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Docs: update ceph troubleshooting with luminious data https://review.openstack.org/530362 | 14:25 |
| openstackgerrit | Merged openstack/openstack-helm master: Dev-Docs: Add warning about IP ranges consumed by K8s and Calico https://review.openstack.org/530280 | 14:28 |
| d|k | portdirect: along the lines of that doc change to the ceph troubleshooting guide: the multinode docs currently suggest installing ceph-common from the ubuntu repos. given that'll get people jewel and the attendant reboot-blocking bug, should that doc recommend installing luminous from the ceph repos? | 14:29 |
| portdirect | d|k: can you confirm that this is fixed in the luminous release? | 14:29 |
| portdirect | if so then we should be updating the multinode guide and AIO accordingly :) | 14:30 |
| d|k | portdirect: that's the claim from the redhat folks. i've only done limited testing, and the bug doesn't manifest on _every_ reboot, but i haven't seen it on my most recent luminous hosts. | 14:31 |
| portdirect | and thats with rbd volumes attached? | 14:31 |
| *** yamamoto has joined #openstack-helm | 14:31 | |
| d|k | yah, but i could try some additional bounces to be sure. | 14:31 |
| portdirect | awesome - thanks dude | 14:32 |
| d|k | well, "sure". like i said, it's always going to be a bit shaky as the bug only manifests some of the time. | 14:32 |
| portdirect | for me I've been unmounting all volumes in a systemd unit fired on shutdow, and with that I was never hit by this | 14:32 |
| d|k | seems like maybe it probably should recommend luminous _anyway_, as that's what's in the containers. | 14:33 |
| d|k | ... though jewel works ok. | 14:33 |
| portdirect | you have a link to the ppa? | 14:33 |
| d|k | lemme find it, one sec. | 14:33 |
| d|k | hmmm, the ceph install docs recommend adding the repo key from their git. that's... not ideal. | 14:34 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: Move fluentbit and fluentd configs to values.yaml https://review.openstack.org/527489 | 14:35 |
| d|k | owait, i take it back, was looking at old docs | 14:36 |
| *** yamamoto has quit IRC | 14:36 | |
| d|k | install docs are here: http://docs.ceph.com/master/install/get-packages, could prolly just link to those? | 14:38 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: WIP: Use OpenStack-Helm-Infra log collection playbook in zuul https://review.openstack.org/530369 | 15:04 |
| *** yamamoto has joined #openstack-helm | 15:16 | |
| *** yamamoto_ has joined #openstack-helm | 15:28 | |
| *** yamamoto has quit IRC | 15:32 | |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: Move fluentbit and fluentd configs to values.yaml https://review.openstack.org/527489 | 15:48 |
| *** yamamoto_ has quit IRC | 15:56 | |
| *** Guest99680 is now known as dansmith | 16:12 | |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: Move fluentbit and fluentd configs to values.yaml https://review.openstack.org/527489 | 16:37 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm master: WIP - Add docs for LMA services https://review.openstack.org/528488 | 16:37 |
| *** yamamoto has joined #openstack-helm | 16:57 | |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: Elasticsearch: Add curator snapshot action and PVC for fs repo https://review.openstack.org/527539 | 17:01 |
| *** yamamoto has quit IRC | 17:15 | |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: Rename elasticsearch configmaps and provide config via toYaml https://review.openstack.org/530391 | 17:24 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm master: WIP - Add docs for LMA services https://review.openstack.org/528488 | 17:26 |
| openstackgerrit | Merged openstack/openstack-helm master: Glance: Fix RBAC object creation for storage-init job https://review.openstack.org/530293 | 17:31 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: Elasticsearch: Add curator snapshot action and PVC for fs repo https://review.openstack.org/527539 | 17:35 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: Rename elasticsearch configmaps and provide config via toYaml https://review.openstack.org/530391 | 17:35 |
| openstackgerrit | Merged openstack/openstack-helm master: Horizon: move to kolla image with bugs fixed https://review.openstack.org/530279 | 18:13 |
| osh-chatbot4 | <abhishekjaiswal> thanks | 18:17 |
| openstackgerrit | Merged openstack/openstack-helm master: Dev-Deploy: Setup iptables for use with docker >= 1.13.1 https://review.openstack.org/530262 | 18:26 |
| osh-chatbot4 | <kelvin633> which version/branch of openstack helm is expected to work with kolla 6.0.0 images? | 18:29 |
| *** yamamoto has joined #openstack-helm | 18:45 | |
| anticw | portdirect: no iptables anywhere of anyones making but mine | 18:54 |
| anticw | portdirect: there is a need to run containers (for control plane elements) in places where iptables is already in use or can't be used | 18:54 |
| anticw | systemd-nspawn may or may not work, runc for some reason doesn't work | 18:55 |
| mwynne | portdirect: I've change the pod subnet to something that won't conflict with my range, but I'm still getting "Host '192.168.100.11' is not allowed to connect to this MariaDB serverConnection closed by foreign host." when I try to telnet to it :S | 19:07 |
| mwynne | anticw: Should I be able to telnet to that host/port from my VM? | 19:10 |
| anticw | mwynne: the ip:port for mysql? usually yes | 19:12 |
| *** yamamoto has quit IRC | 19:13 | |
| *** yamamoto has joined #openstack-helm | 19:13 | |
| anticw | mwynne: `kubectl get pods -o wide --all-namespaces | grep ceph` says? | 19:13 |
| *** yamamoto has quit IRC | 19:14 | |
| mwynne | anticw: Looks good. All running. | 19:15 |
| mwynne | anticw: Looks like remote root access is disabled. I'm going to try manually fixing that and continuing with keystone... | 19:16 |
| anticw | 'remote root' means what in this case? | 19:17 |
| anticw | quite a few of the containers are so spartan they are near useless and a few don't have root entrypoints (arguably a good thing but for development quite frustrating) | 19:17 |
| mwynne | anticw: root access to the DB is only allowed on the localhost (mysql docker container) | 19:19 |
| mwynne | so I can't connect to it from my vm or the keystone container | 19:19 |
| anticw | mwynne: oh, not sure why it won't work using the defaults from the keystone container | 19:21 |
| mwynne | anticw: No idea. After allowing remote root access keystone is doing a db sync now. | 19:21 |
| mwynne | anticw: And keystone came up just fine it seems.. | 19:28 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: Rename elasticsearch configmaps and provide config via toYaml https://review.openstack.org/530391 | 19:30 |
| mwynne | How can I get to horizon once it's been deployed? | 19:37 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: Elasticsearch: Add curator snapshot action and PVC for fs repo https://review.openstack.org/527539 | 19:42 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm-infra master: Gate: simplify helm release log gathering https://review.openstack.org/530398 | 19:48 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: Rename elasticsearch configmaps and provide config via toYaml https://review.openstack.org/530391 | 19:48 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm-infra master: Gate: simplify helm release log gathering https://review.openstack.org/530398 | 19:49 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm master: WIP - Add docs for LMA services https://review.openstack.org/528488 | 19:50 |
| openstackgerrit | Merged openstack/openstack-helm master: Docs: update ceph troubleshooting with luminious data https://review.openstack.org/530362 | 20:00 |
| mwynne | portdirect: Deploying openvswitch I got "containers failed to start.", but they're all running, and everything seems fine.... | 20:00 |
| *** yamamoto has joined #openstack-helm | 20:14 | |
| *** yamamoto has quit IRC | 20:23 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: WIP: Cinder: Enable backup https://review.openstack.org/530286 | 20:36 |
| openstackgerrit | Elise Eiden proposed openstack/openstack-helm master: [WIP] Add Tempest Chart https://review.openstack.org/498929 | 20:37 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: WIP: Use OpenStack-Helm-Infra log collection playbook in zuul https://review.openstack.org/530369 | 20:39 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm-infra master: Gate: simplify helm release log gathering https://review.openstack.org/530398 | 20:40 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: WIP: Use OpenStack-Helm-Infra log collection playbook in zuul https://review.openstack.org/530369 | 20:40 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm master: Add out-of-date warning to openstack-helm gates README https://review.openstack.org/530400 | 20:52 |
| mwynne | portdirect: Are you sure that this AIO deployment actually changes the mysql root password? I'm still getting access denied :( | 21:05 |
| mwynne | (even with that timeout increased) | 21:06 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: Add elasticsearch exporter https://review.openstack.org/523634 | 21:21 |
| openstackgerrit | Merged openstack/openstack-helm master: Ceph: update dependency logic to be more strict https://review.openstack.org/530266 | 21:55 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: WIP: Cinder: Enable backup https://review.openstack.org/530286 | 22:01 |
| *** gagehugo has joined #openstack-helm | 22:09 | |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: WIP: Cinder: Enable backup https://review.openstack.org/530286 | 22:27 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: WIP: Cinder: Enable backup https://review.openstack.org/530286 | 23:03 |
| mwynne | anticw or portdirect you guys around? | 23:13 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Cinder: Enable backup https://review.openstack.org/530286 | 23:35 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Docs: remove potentially confusing references to versions and RBAC https://review.openstack.org/530412 | 23:46 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Docs: remove potentially confusing references to versions and RBAC https://review.openstack.org/530412 | 23:47 |
| openstackgerrit | Pete Birley proposed openstack/openstack-helm master: Cinder: Enable backup https://review.openstack.org/530286 | 23:48 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!