Monday, 2017-08-21

« Sunday, 2017-08-20 Index Tuesday, 2017-08-22 »
openstackgerritPete Birley proposed openstack/openstack-helm master: Nova: Break out NoVNC assets from the proxy process  https://review.openstack.org/49566000:13
*** iceyao has joined #openstack-helm00:36
*** iceyao has quit IRC00:38
*** iceyao has joined #openstack-helm00:39
*** iceyao has quit IRC00:43
*** iceyao has joined #openstack-helm01:00
openstackgerritPete Birley proposed openstack/openstack-helm master: WIP: LOCI Image Support.  https://review.openstack.org/47296101:24
*** gouthamr has quit IRC01:26
openstackgerritJawon Choo proposed openstack/openstack-helm master: pod resources mismatched in ingress chart  https://review.openstack.org/49567602:49
*** felipemonteiro_ has joined #openstack-helm03:48
*** felipemonteiro__ has joined #openstack-helm03:50
*** felipemonteiro_ has quit IRC03:53
*** felipemonteiro__ has quit IRC03:55
*** iceyao has quit IRC03:57
*** iceyao has joined #openstack-helm04:13
*** iceyao has quit IRC04:18
*** iceyao has joined #openstack-helm04:39
*** iceyao has quit IRC06:33
*** iceyao has joined #openstack-helm06:34
*** iceyao has quit IRC06:38
openstackgerritMateusz Blaszkowski proposed openstack/openstack-helm master: Fix issues with [trustee] section of heat.conf  https://review.openstack.org/47182806:58
*** jaypipes has joined #openstack-helm07:14
*** MarkBaker has joined #openstack-helm07:18
*** pprokop has joined #openstack-helm07:22
*** MarkBaker has quit IRC07:28
*** MarkBaker has joined #openstack-helm07:42
*** MarkBaker has quit IRC08:15
*** larainema has quit IRC08:18
*** larainema has joined #openstack-helm08:21
*** xek has joined #openstack-helm08:28
*** anticw has quit IRC08:44
openstackgerritGrzegorz Grasza (xek) proposed openstack/openstack-helm master: Add mq-init.py script [WiP]  https://review.openstack.org/49178908:49
*** anticw has joined #openstack-helm08:51
*** MarkBaker has joined #openstack-helm09:04
*** dims has quit IRC09:50
*** pprokop has quit IRC10:02
*** pprokop has joined #openstack-helm10:03
*** pprokop has quit IRC10:04
*** pprokop has joined #openstack-helm10:10
*** pprokop has quit IRC10:12
*** gixxer1k has joined #openstack-helm10:45
openstackgerritMateusz Blaszkowski proposed openstack/openstack-helm master: heat-engine as Deployment instead of StatefulSet  https://review.openstack.org/47225610:58
*** openstackgerrit has quit IRC11:17
osh-chatbot<dporru> I have question about the Ceph charts.  I'm trying to initialize my Ceph cluster with multiple OSDs. If I'm not mistaken, the current behavior of the Ceph charts is to create one OSD per Kubernetes node in `/var/lib/openstack-helm/ceph/osd`.  I would like to let the charts initialize my OSD devices (eg. /dev/sda, /dev/sdb, ...). I don't see a way to let the Helm charts initialize OSDs and put them in the Ceph cluster without changing11:35
osh-chatbotthe charts (so by setting only values). Maybe I'm missing something, if so, please let me know.  I would like to propose a change to the Ceph charts to make the following possible with only overwriting values:  - Multiple OSDs per Kubernetes node - Use the OSD_TYPE=disk to prepare and activate an OSD - Use Bluestore with OSD_BLUESTORE=1  I'd like to implement these changes but I have some questions:  1. Do you see any problems with11:35
osh-chatbotimplementing these changes? 2. Would you want this in the main repo?  I'm open for other suggestions, any advise or feedback.11:35
*** julim has quit IRC11:57
*** schwicht has joined #openstack-helm12:23
*** schwicht has quit IRC12:33
*** schwicht_at_work has joined #openstack-helm12:34
*** aric49 has joined #openstack-helm12:56
*** schwicht_at_work has quit IRC12:56
*** julim has joined #openstack-helm12:57
osh-chatbot<portdirect> @dporru 1) no. 2) yes! :slightly_smiling_face:13:12
osh-chatbot<portdirect> there is some work going on for disc support thats pretty close to the finish line: https://review.openstack.org/#/c/457754/13:13
osh-chatbot<portdirect> would be great to give it a whirl and get your feedback13:13
*** schwicht has joined #openstack-helm13:13
osh-chatbot<portdirect> it doesnt currently used bluestore but once in it would be great to have a ps that adds that as an option.13:14
*** mattmceuen has joined #openstack-helm13:16
*** dims has joined #openstack-helm13:21
maciejjozefczykHellol13:39
v1k0d3nhello maciejjozefczyk13:40
maciejjozefczykGuys, I found something strange with Helm deployment and ingress-api; In keystone/templates/ingress-api.yaml its defined that ingress-api should create vhost on both names: short one and long one, in my example its keystone and keystone.openstack13:40
maciejjozefczykbut I found when i'm trying to use openstack client with my deployment my openstackclient is trying to reach 'keystone-api' host13:41
maciejjozefczykDEBUG (session:375) REQ: curl -g -i -X GET http://keystone-api.openstack:80/v3 -H "Accept: application/json" -H "User-Agent: nova keystoneauth1/3.1.0 python-requests/2.18.4 CPython/2.7.12"13:42
maciejjozefczykDEBUG (connectionpool:208) Starting new HTTP connection (1): keystone-api.openstack13:42
maciejjozefczykDEBUG (connectionpool:396) http://keystone-api.openstack:80 "GET /v3 HTTP/1.1" 503 21313:42
maciejjozefczykDEBUG (session:423) RESP: [503] Server: nginx/1.13.1 Date: Mon, 21 Aug 2017 13:38:23 GMT Content-Type: text/html Content-Length: 213 Connection: keep-alive Strict-Transport-Security: max-age=15724800; includeSubDomains; Y13:42
maciejjozefczyk503 is because server-name is not defined in nginx ingress-api13:42
maciejjozefczykAhh, I just have found it. Created openrc file from horizon set keystone-api.openstack as AUTH_URL; istead keystone.openstack13:44
*** xek has quit IRC13:47
*** xek has joined #openstack-helm13:47
*** felipemonteiro_ has joined #openstack-helm13:48
*** iceyao has joined #openstack-helm13:48
*** felipemonteiro__ has joined #openstack-helm13:49
*** openstackgerrit has joined #openstack-helm13:51
openstackgerritPete Birley proposed openstack/openstack-helm master: WIP: Gate: Heat functional test  https://review.openstack.org/49271913:51
*** felipemonteiro_ has quit IRC13:52
*** iceyao has quit IRC14:35
*** jaypipes has quit IRC14:41
*** iceyao has joined #openstack-helm14:44
osh-chatbot<kubernaut> Hi, is someone running OpenStack-Helm on CentOS 7.3, docker 1.13.1 and kubernetes 1.7.3?14:47
osh-chatbot<dporru> @portdirect Thanks, that's exactly the kind kind of direction I was hoping for. I'll give it a try. Adding Bluestore doesn't look very complicated.14:47
v1k0d3nthat's very specific @kubernaut. not exactly those versions, but i am starting to attempt to use CoreOS...beginning this exploration this week in more depth.14:48
osh-chatbot<portdirect> @kubernaut we've tested with centos 7.3, and I've done some light testing with 1.7.x though have no idea of the docker version in use14:50
osh-chatbot<portdirect> That said I dont think you should encounter any difficulty with that combo14:51
v1k0d3noh sorry...CentOS and NOT CoreOS...sorry @kubernaut, i read that wrong :)14:51
v1k0d3nwe've had API version issues with later versions of docker...so just be aware of this when you're testing.14:51
v1k0d3nyou will see API errors in the log if this is the case.14:52
osh-chatbot<portdirect> yeah - the docker provides packages are not to great atm :S14:53
osh-chatbot<portdirect> *provided14:53
v1k0d3nyup. typical.14:53
osh-chatbot<kubernaut> @v1k0d3n @matt-porter okay thx :slightly_smiling_face: the main problems which we have is ceph-osd and mariadb:  Error from server (BadRequest): container "mariadb" in pod "mariadb-0" is waiting to start: ContainerCreating  [root@csky04 ~]# kubectl logs ceph-osd-0ncrb -n ceph  2017-08-21 14:53:55  /entrypoint.sh: SUCCESS forego   | starting ceph-0.1 on port 5000 ceph-0.1 | starting osd.0 at :/0 osd_data /var/lib/ceph/osd/ceph-014:55
osh-chatbot/var/lib/ceph/osd/ceph-0//journal ceph-0.1 | 2017-08-21 14:53:55.956114 7fb59f5df8c0 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway ceph-0.1 | 2017-08-21 14:53:56.063229 7fb59f5df8c0 -1 osd.0 6196 log_to_monitors {default=true}14:55
osh-chatbot<kubernaut> what is the recommended docker version? 1.12.x? on Ubuntu?14:59
v1k0d3nthe docker version is really decided on by kubeadm, since that is what OSH is establishing...kubeadm for AIO container, etc.14:59
v1k0d3nhowever, the project should be designed to allow users to install kubernetes however they need/want...given that RBAC is applied correctly (a lot of people get confused on RBAC, so OSH puts in place an RBAC policy that is extremely loose...just for development purposes for now).15:01
v1k0d3nthere are a lot of moving parts, so as you can imagine...the kubeadm-aio is useful for most people, and that helps to control the deployments and limit errors/questions from people.15:02
v1k0d3nbut look into the type of deployment you _want_ to use...and determine how the docker version applies for that deployment type.15:02
osh-chatbot<kubernaut> we tried with a multi-node kubespray env. (kube 1.6.7) and  a 2 node kubeadm env. on CentOS 7.3 with not so much luck, now we'll switch to Ubuntu and I guess we'll have more luck :slightly_smiling_face:15:06
*** gixxer1k has quit IRC15:56
openstackgerritPete Birley proposed openstack/openstack-helm master: WIP: Gate: Heat functional test  https://review.openstack.org/49271915:59
*** iceyao has quit IRC16:23
*** mattmceuen has quit IRC16:23
*** MarkBaker has quit IRC16:33
*** iceyao has joined #openstack-helm16:49
*** gixxer1k has joined #openstack-helm16:52
osh-chatbot<portdirect> @kubernaut have you set selinux to permissive mode?16:53
osh-chatbot<portdirect> or is it enforcing?16:53
*** iceyao has quit IRC16:53
*** julim has quit IRC17:32
*** julim has joined #openstack-helm17:39
openstackgerritPete Birley proposed openstack/openstack-helm master: Gate: Heat functional test  https://review.openstack.org/49271918:56
*** gixxer1k has quit IRC19:14
*** gixxer1k has joined #openstack-helm19:14
*** gixxer1k has quit IRC19:17
*** gixxer1k has joined #openstack-helm19:18
*** marst has joined #openstack-helm19:19
*** gixxer1k has quit IRC19:21
*** gixxer1k has joined #openstack-helm19:21
*** gixxer1k has quit IRC19:24
*** gixxer1k has joined #openstack-helm19:25
*** gixxer1k has quit IRC19:28
*** gixxer1k has joined #openstack-helm19:28
*** gixxer1k has quit IRC19:31
*** gixxer1k has joined #openstack-helm19:32
*** julim has quit IRC19:41
*** wxy has quit IRC19:44
*** wxy has joined #openstack-helm19:46
*** gouthamr has joined #openstack-helm20:03
portdirect^^ If some people could give this a look it would be awesome20:07
portdirectdrives the gate functional test with heat as well as the cli if available, and tidies up a lot of the knobs to a "vars.sh" file20:08
openstackgerritPete Birley proposed openstack/openstack-helm master: Fix issues with [trustee] section of heat.conf  https://review.openstack.org/47182820:10
osh-chatbot<kubernaut> @portdirect yes selinux is set to permissive20:11
osh-chatbot<portdirect> :( @kubernaut damn - that would have been the easy fix (if not...)20:12
osh-chatbot<portdirect> can you dump `kubectl get --all-namespaces all --show-all` to paste.openstack.org ?20:12
openstackgerritMerged openstack/openstack-helm master: pod resources mismatched in ingress chart  https://review.openstack.org/49567620:15
osh-chatbot<kubernaut> @portdirect I rebooted the 2 servers and the dump is here (mariadb is vanished): http://paste.openstack.org/show/618952/20:19
osh-chatbot<portdirect> cool20:20
osh-chatbot<portdirect> it is ok if we start again - just to make sure everythings a ok?20:20
osh-chatbot<portdirect> also - it looks like you are not running a cni? is that correct?20:22
osh-chatbot<portdirect> take it just flanneld on the host? or some other external network kung-fu?20:23
*** gagehugo has quit IRC20:50
osh-chatbot<kubernaut> we're running calico: [root@csky04 ~]# docker ps | grep -i calico 21ec5dfc2b12        calico/node:v1.1.3                                                                                                                       "start_runit"            57 minutes ago      Up 57 minutes                           calico-node20:50
osh-chatbot<portdirect> right - gotch20:51
osh-chatbot<portdirect> *a20:51
osh-chatbot<portdirect> so you ok to start over?20:52
osh-chatbot<portdirect> though its late where you are?20:52
osh-chatbot<kubernaut> start over with?20:53
portdirectceph20:53
osh-chatbot<kubernaut> ah, purge ceph and start over?20:53
osh-chatbot<portdirect> yup20:54
osh-chatbot<kubernaut> okay20:54
osh-chatbot<portdirect> right - as well as ```helm delete --purge ceph``` lets run a few more things20:54
osh-chatbot<portdirect> https://www.drupal.org/files/project-images/nuke-from-orbit.jpg :)20:55
osh-chatbot<kubernaut> :-920:55
osh-chatbot<portdirect> also run: ``` kubectl delete ns ceph ```20:55
osh-chatbot<portdirect> and20:55
osh-chatbot<kubernaut> [root@csky04 ~]# helm delete --purge ceph release "ceph" deleted20:55
osh-chatbot<portdirect> Run this on each of your nodes as well: ``` sudo rm -rf /var/lib/openstack-helm ```20:56
osh-chatbot<kubernaut> on master too?20:57
osh-chatbot<kubernaut> done on master and worker20:58
osh-chatbot<portdirect> to be safe i would20:58
osh-chatbot<portdirect> ok nice20:58
osh-chatbot<portdirect> so now lets check you have the required stuff on each host20:59
osh-chatbot<portdirect> can you run: ``` sudo yum install -y \       ceph ``` on each host20:59
osh-chatbot<portdirect> to ensure that you have the utils required for mounting the rbd volumes21:00
osh-chatbot<kubernaut> Error: Package: 1:python-cephfs-0.80.7-0.8.el7.x86_64 (epel)            Requires: python-rados = 1:0.80.7            Installed: 1:python-rados-0.94.5-1.el7.x86_64 (@base)                python-rados = 1:0.94.5-1.el721:00
osh-chatbot<portdirect> hmm :S21:01
*** gouthamr has quit IRC21:01
osh-chatbot<portdirect> can you update/upgrade and see if that resolves it?21:01
osh-chatbot<kubernaut> okay21:02
osh-chatbot<kubernaut> did yum update, but didn't help: [root@csky04 ~]# cat /etc/redhat-release CentOS Linux release 7.3.1611 (Core)21:04
*** gixxer1k has quit IRC21:04
osh-chatbot<kubernaut> [root@csky04 ~]# sudo yum install -y ceph --skip-broken Packages skipped because of dependency problems:     1:ceph-0.80.7-0.8.el7.x86_64 from epel     gperftools-libs-2.4-8.el7.x86_64 from base     leveldb-1.12.0-11.el7.x86_64 from epel     1:libcephfs1-0.80.7-0.8.el7.x86_64 from epel     2:libunwind-1.1-5.el7_2.2.x86_64 from base     python-babel-0.9.6-8.el7.noarch from base     1:python-cephfs-0.80.7-0.8.el7.x86_64 from epel21:04
osh-chatbot1:python-flask-0.10.1-4.el7.noarch from extras     python-itsdangerous-0.23-2.el7.noarch from extras     python-jinja2-2.7.2-2.el7.noarch from base     python-markupsafe-0.11-10.el7.x86_64 from base     python-werkzeug-0.9.1-2.el7.noarch from extras21:04
osh-chatbot<portdirect> ok21:05
osh-chatbot<portdirect> can you add the ceph-jewel repo and see if that fixes your issues:21:06
osh-chatbot<portdirect> https://github.com/CentOS-Storage-SIG/centos-release-ceph-jewel/blob/master/CentOS-Ceph-Jewel.repo21:06
osh-chatbot<portdirect> or just ```yum install centos-release-ceph-jewel```21:07
openstackgerritHuamin Chen proposed openstack/openstack-helm master: Ceph OSD disk targeting  https://review.openstack.org/45775421:07
osh-chatbot<kubernaut> Dependencies Resolved  ===================================================================================================================================================================================  Package                                          Arch                              Version                                     Repository                                    Size21:10
osh-chatbot=================================================================================================================================================================================== Updating:  ceph-common                                      x86_64                            1:10.2.7-0.el7                              centos-ceph-jewel                             17 M21:10
osh-chatbot<portdirect> awesome21:10
osh-chatbot<portdirect> so with that repo enabled you are able to install the ```ceph``` package?21:11
osh-chatbot<kubernaut> YEHHHH :slightly_smiling_face:21:12
osh-chatbot<kubernaut> nstalled:   ceph.x86_64 1:10.2.7-0.el7  Dependency Installed:   boost-iostreams.x86_64 0:1.53.0-26.el7    boost-program-options.x86_64 0:1.53.0-26.el7    boost-random.x86_64 0:1.53.0-26.el7        boost-regex.x86_64 0:1.53.0-26.el7   ceph-base.x86_64 1:10.2.7-0.el7           ceph-mds.x86_64 1:10.2.7-0.el7                  ceph-mon.x86_64 1:10.2.7-0.el7             ceph-osd.x86_64 1:10.2.7-0.el7   ceph-selinux.x86_64 1:10.2.7-0.el721:13
osh-chatbot      fcgi.x86_64 0:2.4.0-25.el7                      fuse-libs.x86_64 0:2.9.2-7.el7             gperftools-libs.x86_64 0:2.4-8.el7   leveldb.x86_64 0:1.12.0-11.el7            libbabeltrace.x86_64 0:1.2.4-3.el7              libcephfs1.x86_64 1:10.2.7-0.el7           libradosstriper1.x86_64 1:10.2.7-0.el7   librgw2.x86_64 1:10.2.7-0.el7             libunwind.x86_64 2:1.1-5.el7_2.2                lttng-ust.x86_64 0:2.4.1-4.el721:13
osh-chatbotpython-babel.noarch 0:0.9.6-8.el7   python-cephfs.x86_64 1:10.2.7-0.el7       python-flask.noarch 1:0.10.1-4.el7              python-itsdangerous.noarch 0:0.23-2.el7    python-jinja2.noarch 0:2.7.2-2.el7   python-markupsafe.x86_64 0:0.11-10.el7    python-setuptools.noarch 0:0.9.8-4.el7          python-werkzeug.noarch 0:0.9.1-2.el7       userspace-rcu.x86_64 0:0.7.16-1.el7  Dependency Updated:   ceph-common.x86_64 1:10.2.7-0.el721:13
osh-chatbotlibrados2.x86_64 1:10.2.7-0.el7   librbd1.x86_64 1:10.2.7-0.el7   python-rados.x86_64 1:10.2.7-0.el7   python-rbd.x86_64 1:10.2.7-0.el7  Complete!21:13
osh-chatbot<portdirect> ok - now on to the next step in our adventure21:13
osh-chatbot<portdirect> so can you dump the output of ```ip addr```21:13
osh-chatbot<portdirect> to a paste21:13
osh-chatbot<portdirect> also worth noting that for anything thats more than a line of output a pate makes it to much easier for our friends on irc to parse ;)21:14
osh-chatbot<portdirect> *paste21:14
osh-chatbot<portdirect> so from a quick pm we know its the `10.0.0.0/24` subnet we want to use here for ceph things21:17
osh-chatbot<portdirect> @kubernaut can you pull the latest version of osh from git21:18
osh-chatbot<portdirect> just to eleimiate that as a source of confusion21:18
osh-chatbot<kubernaut> yo21:18
osh-chatbot<kubernaut> to adapt the values?21:18
osh-chatbot<portdirect> no nothing to do with that just yet21:18
osh-chatbot<portdirect> just make sure you are on the master branch21:19
osh-chatbot<kubernaut> okay21:19
osh-chatbot<portdirect> and run `git pull`21:19
osh-chatbot<kubernaut> [root@csky04 ~]# cd openstack-helm/ [root@csky04 openstack-helm]# git pull remote: Counting objects: 96, done. remote: Total 96 (delta 70), reused 71 (delta 70), pack-reused 25 Unpacking objects: 100% (96/96), done. From https://github.com/openstack/openstack-helm    a958450..9f7e6ed  master     -> origin/master Updating a958450..9f7e6ed Fast-forward  barbican/templates/bin/_barbican-test.sh.tpl            |  37 +++++++++++++21:19
osh-chatbotbarbican/templates/configmap-bin.yaml                   |   2 +  barbican/templates/configmap-etc.yaml                   |   3 +-  barbican/templates/pod-test.yaml                        |  52 +++++++++++++++++++21:19
osh-chatbot<kubernaut> ...21:19
osh-chatbot<portdirect> once you have done that - can you then run make21:20
osh-chatbot<portdirect> also please (pretty please) use paste for posting output :)21:20
osh-chatbot<kubernaut> w/o adapting the nova/values.yaml?21:20
osh-chatbot<portdirect> inline with messages is horrible to read21:20
osh-chatbot<kubernaut> okay, sorry21:20
osh-chatbot<portdirect> why do you need to do that? (adapt the nova/values.yaml?)21:21
osh-chatbot<kubernaut> I was told to do that for meta.data ip, etc..21:21
osh-chatbot<portdirect> you should not need to change anything in the values.yaml we can do that all via set while deploying21:22
osh-chatbot<portdirect> but for now lets just deal with ceph21:22
osh-chatbot<kubernaut> okay21:22
osh-chatbot<portdirect> so can you run `make`21:22
osh-chatbot<kubernaut> make gives: http://paste.openstack.org/show/618955/21:22
osh-chatbot<portdirect> ok - so can you run `helm serve &`21:23
osh-chatbot<portdirect> and then run `make` again21:23
osh-chatbot<kubernaut> yeh: 1 chart(s) linted, no failures21:24
osh-chatbot<portdirect> should be a lot more than one chart?21:25
osh-chatbot<portdirect> can you paste the output?21:25
osh-chatbot<kubernaut> okay21:25
osh-chatbot<kubernaut> pasted only the last line, here is the full output: http://paste.openstack.org/show/618958/21:26
osh-chatbot<portdirect> nice21:26
osh-chatbot<portdirect> so now lets go over what you have21:27
osh-chatbot<portdirect> 2 nodes21:27
osh-chatbot<portdirect> 1 master21:27
osh-chatbot<portdirect> 1 worker21:27
osh-chatbot<kubernaut> yes, 1 master 1 node21:27
osh-chatbot<portdirect> correct?21:27
osh-chatbot<portdirect> and where do you want ceph running?21:27
osh-chatbot<kubernaut> on the node21:27
osh-chatbot<kubernaut> workernode21:27
osh-chatbot<portdirect> ok21:27
osh-chatbot<portdirect> whats it called? (`kubectl get nodes`)21:28
osh-chatbot<kubernaut> hmm, now I can't get the node: [root@csky04 ~]# kubectl get nodes The connection to the server localhost:8080 was refused - did you specify the right host or port?21:29
osh-chatbot<portdirect> is it beacuse you are root?21:31
osh-chatbot<kubernaut> [kubernaut@csky04 ~]$ kubectl get nodes The connection to the server localhost:8080 was refused - did you specify the right host or port?21:32
osh-chatbot<portdirect> nothing we have done should have had any effect on the cluster21:33
osh-chatbot<kubernaut> I think some other colleagues have chaned the installation w/o informing me :(21:33
osh-chatbot<portdirect> ok21:34
osh-chatbot<portdirect> feel free to ping me when you can get access to the cluster again21:34
osh-chatbot<kubernaut> thx so much!21:35
osh-chatbot<portdirect> np21:36
*** schwicht has quit IRC21:39
*** aric49 has quit IRC21:44
openstackgerritMerged openstack/openstack-helm master: Gate: Heat functional test  https://review.openstack.org/49271921:46
osh-chatbot<kubernaut> @portdirect I want to start a new install on new bare-metal, shall I go with kubeadm w/ kube 1.7.3 or kubespray with w/ kube 1.6.8?21:50
openstackgerritPete Birley proposed openstack/openstack-helm master: Ceph: CephFS external provisioner and client helm tests  https://review.openstack.org/49357921:50
osh-chatbot<portdirect> @kubernaut should not make a difference21:51
osh-chatbot<kubernaut> okay, thx!21:51
osh-chatbot<portdirect> I'm currently working on a PS for helm 2.6 and k8s 1.721:52
osh-chatbot<portdirect> though all our testing is around helm 2.5.1 and k8s 1.6 atm21:52
osh-chatbot<portdirect> Seems to work fine so far21:52
osh-chatbot<portdirect> really looking forward to statefulset upgrades though :)21:52
*** schwicht has joined #openstack-helm22:01
openstackgerritPete Birley proposed openstack/openstack-helm master: Nova: Move Libvirt into its own chart  https://review.openstack.org/49373822:04
*** schwicht has quit IRC22:05
openstackgerritPete Birley proposed openstack/openstack-helm master: Neutron: Move OpenVSwitch into its own chart  https://review.openstack.org/49374922:11
openstackgerritLarry Rensing proposed openstack/openstack-helm-addons master: WIP - MaaS chart  https://review.openstack.org/47238122:11
openstackgerritPete Birley proposed openstack/openstack-helm master: Neutron: Move OpenVSwitch into its own chart  https://review.openstack.org/49374922:11
*** schwicht has joined #openstack-helm22:15
osh-chatbot<kubernaut> @portdirect installed a fresh kubespray w/ kube 1.6.7 [root@csky06 ~]# kubectl get nodes NAME      STATUS                     AGE       VERSION csky06    Ready,SchedulingDisabled   3m        v1.6.7+coreos.0 csky07    Ready                      3m        v1.6.7+coreos.022:15
openstackgerritPete Birley proposed openstack/openstack-helm master: Neutron: Move OpenVSwitch into its own chart  https://review.openstack.org/49374922:16
osh-chatbot<portdirect> @kubernaut nice - and you have installed the`ceph` package on each of those nodes?22:17
osh-chatbot<kubernaut> yes22:19
openstackgerritPete Birley proposed openstack/openstack-helm master: Ceph OSD disk targeting  https://review.openstack.org/45775422:21
osh-chatbot<kubernaut> and did a git clone on master: [root@csky06 ~]# git clone https://github.com/openstack/openstack-helm.git22:21
osh-chatbot<portdirect> ok - so can you set up helm and run `make` again22:21
osh-chatbot<kubernaut> okay22:21
osh-chatbot<portdirect> and also relax the rbac: http://openstack-helm.readthedocs.io/en/latest/install/multinode.html#setting-up-rbac22:22
osh-chatbot<kubernaut> hmm ??:  Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster. Happy Helming! [root@csky06 ~]# helm version Client: &version.Version{SemVer:"v2.6+unreleased", GitCommit:"a2323f8658b29dedc8485020730556793f5d14e0", GitTreeState:"clean"} Error: cannot connect to Tiller22:28
osh-chatbot<kubernaut> [root@csky06 ~]# kubectl update -f https://raw.githubusercontent.com/openstack/openstack-helm/master/tools/kubeadm-aio/assets/opt/rbac/dev.yaml W0822 00:26:59.648094    7858 cmd.go:361] update is DEPRECATED and will be removed in a future version. Use replace instead. clusterrolebinding "cluster-admin" replaced22:29
*** schwicht has quit IRC22:31
osh-chatbot<portdirect> @kubernaut Not sure :S but running a beta helm version may not help.22:33
osh-chatbot<portdirect> or may be a red herring, you never know ;)22:33
osh-chatbot<kubernaut> [root@csky06 linux-amd64]# helm version Client: &version.Version{SemVer:"v2.6.0", GitCommit:"5bc7c619f85d74702e810a8325e0a24f729aa11a", GitTreeState:"clean"} Server: &version.Version{SemVer:"v2.6.0", GitCommit:"5bc7c619f85d74702e810a8325e0a24f729aa11a", GitTreeState:"clean"}22:42
portdirectlooking better22:43
osh-chatbot<kubernaut> lable nodes?22:43
portdirecthave you got dns working on your hosts22:43
*** felipemonteiro__ has quit IRC22:43
osh-chatbot<kubernaut> I've the hosts in resolve.conf22:47
osh-chatbot<kubernaut> or shall they resolve to fqdn?22:48
osh-chatbot<kubernaut> [root@csky06 linux-amd64]# kubectl get svc --all-namespaces NAMESPACE     NAME            CLUSTER-IP     EXTERNAL-IP   PORT(S)         AGE default       kubernetes      10.233.0.1     <none>        443/TCP         31m kube-system   kube-dns        10.233.0.3     <none>        53/UDP,53/TCP   31m kube-system   tiller-deploy   10.233.16.34   <none>        44134/TCP       17m22:48
osh-chatbot<kubernaut> did helm serve& and make, went well22:57
osh-chatbot<kubernaut> helm list retrievs nothing22:57
*** marst has quit IRC22:59
osh-chatbot<kubernaut> is that right?  [root@csky06 openstack-helm]# export osd_cluster_network=10.0.0.0/24 [root@csky06 openstack-helm]# export osd_public_network=10.0.0.0/24  Or shall I set the public network to the x.x.x.x/26 which is defined on the switch?23:04
portdirectwhy do you have a dif subnet on the switch?23:13
portdirectand for dns you will want to edit the resolv.conf to setup the search path and use the k8s dns23:14
osh-chatbot<kubernaut> we have a public subnet and a 10.0.0.0 sunbet23:14
portdirectoh ok - but the nodes have direct connectivitity on the 10.0.0.0/24 subnet23:15
portdirect?23:15
osh-chatbot<kubernaut> yes23:15
portdirectcan you dump the output of /etc/resolv.conf23:15
osh-chatbot<kubernaut> http://paste.openstack.org/show/618966/23:18
portdirectok23:19
portdirectis 10.233.0.3 your k8s dns server?23:19
osh-chatbot<kubernaut> the nameserver in resolve.conf for kube-dns is from k8s dns 10.233.0.3, yes23:20
portdirectright23:20
portdirect2 mins23:20
portdirectcan you delete the resolv.conf symlink on your hosts23:24
portdirectsudo rm -f /etc/resolv.conf23:24
portdirectand replace them with this file23:24
osh-chatbot<portdirect> File uploaded https://kubernetes.slack.com/files/portdirect/F6R5CESL9/-.txt / https://slack-files.com/T09NY5SBT-F6R5CESL9-a6a5b5bb1e23:24
osh-chatbot<kubernaut> done23:25
osh-chatbot<portdirect> nice23:25
osh-chatbot<portdirect> to check thats worked can you run this from each of your hosts23:26
osh-chatbot<portdirect> ``` ping -c1 -t1 kubernetes.default ```23:26
osh-chatbot<portdirect> and see that its resolved an ip23:27
osh-chatbot<portdirect> most likley `10.233.0.1` in your setup23:27
osh-chatbot<portdirect> though the ping itself will fail23:27
osh-chatbot<kubernaut> Yes :slightly_smiling_face: [root@csky06 openstack-helm]# ping -c1 -t1 kubernetes.default PING kubernetes.default.svc.cluster.local (10.233.0.1) 56(84) bytes of data.23:28
osh-chatbot<portdirect> sweet23:28
osh-chatbot<portdirect> ok - so you installed the ceph package on those hosts as well?23:28
osh-chatbot<kubernaut> yes23:28
osh-chatbot<portdirect> right - so now can you run `helm repo add local http://localhost:8879/charts`23:29
osh-chatbot<portdirect> and then `make`23:29
osh-chatbot<portdirect> and then `helm repo remove stable`23:29
osh-chatbot<portdirect> before posting the output of `helm list` to a paste23:30
osh-chatbot<kubernaut> dome, helm list retrieves nothing: http://paste.openstack.org/show/618967/23:33
osh-chatbot<kubernaut> I'm root23:34
osh-chatbot<portdirect> can you run `helm search`23:34
osh-chatbot<portdirect> and then list again23:35
osh-chatbot<kubernaut> :slightly_smiling_face: http://paste.openstack.org/show/618968/23:35
osh-chatbot<portdirect> lol - toolate in the day for me23:36
osh-chatbot<kubernaut> but:23:36
osh-chatbot<kubernaut> [root@csky06 openstack-helm]# helm list23:36
osh-chatbot<portdirect> thats cool :slightly_smiling_face:23:36
osh-chatbot<portdirect> was meaning helm search when i was asking for helm list - doh23:36
osh-chatbot<kubernaut> ahhh23:36
osh-chatbot<portdirect> so anyway - lets label the nodes23:36
osh-chatbot<portdirect> can you give me the output of `kubectl get nodes`23:37
osh-chatbot<kubernaut> [root@csky06 openstack-helm]# kubectl get nodes NAME      STATUS                     AGE       VERSION csky06    Ready,SchedulingDisabled   1h        v1.6.7+coreos.0 csky07    Ready                      1h        v1.6.7+coreos.023:37
osh-chatbot<portdirect> you know if you click the + next to the message you can post it as a code block? makes it much easier to read23:38
osh-chatbot<kubernaut> okay thx, didn't know23:38
osh-chatbot<portdirect> File uploaded https://kubernetes.slack.com/files/portdirect/F6RJXCR0T/-.txt / https://slack-files.com/T09NY5SBT-F6RJXCR0T-6e15b4535d23:38
osh-chatbot<portdirect> can you run those commands to label your node23:38
*** schwicht has joined #openstack-helm23:39
osh-chatbot<kubernaut> [root@csky06 openstack-helm]# kubectl label nodes ceph-mon=enabled csky07 error: all resources must be specified before label changes: csky07 See 'kubectl label -h' for help and examples. +23:39
osh-chatbot<portdirect> ah23:40
osh-chatbot<portdirect> File uploaded https://kubernetes.slack.com/files/portdirect/F6RM4962W/-.txt / https://slack-files.com/T09NY5SBT-F6RM4962W-24650c6bca23:40
osh-chatbot<portdirect> and then from the openstack-helm directory run:23:42
osh-chatbot<portdirect> File uploaded https://kubernetes.slack.com/files/portdirect/F6SPV1BKR/-.txt / https://slack-files.com/T09NY5SBT-F6SPV1BKR-9f4fc3753823:42
osh-chatbot<kubernaut> oh wow :slightly_smiling_face: http://paste.openstack.org/show/618969/23:44
osh-chatbot<portdirect> ok :slightly_smiling_face:23:44
osh-chatbot<portdirect> now lets see whats going on23:44
osh-chatbot<portdirect> `watch kubectl get -n ceph pods`23:45
osh-chatbot<kubernaut> http://paste.openstack.org/show/618970/23:45
osh-chatbot<portdirect> hmm - you should not have 2 osds and mons running :S23:46
osh-chatbot<portdirect> did you label the master node as well?23:46
osh-chatbot<kubernaut> yes, I labled before with --all23:46
osh-chatbot<portdirect> oh23:46
osh-chatbot<kubernaut> sorry23:47
osh-chatbot<portdirect> np23:47
osh-chatbot<portdirect> you will want to delete and purge the helm ceph deployment23:47
osh-chatbot<portdirect> remove the labels from the master node23:47
osh-chatbot<portdirect> remove the /var/lib/openstack-helm dir23:48
osh-chatbot<portdirect> delte the ceph ns23:48
osh-chatbot<portdirect> and then run the command above to launch ceph again23:48
osh-chatbot<kubernaut> okay23:48
osh-chatbot<kubernaut> http://paste.openstack.org/show/618971/23:55
osh-chatbot<portdirect> you still have the labels on the mast node it looks like?23:56
osh-chatbot<portdirect> *master23:56
osh-chatbot<portdirect> can you dump `kubectl describe nodes`?23:57
*** schwicht has quit IRC23:59
« Sunday, 2017-08-20 Index Tuesday, 2017-08-22 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!