| *** jmccarthy has quit IRC | 00:15 | |
| *** jmccarthy has joined #kolla | 00:16 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Reduce gate resource usage https://review.openstack.org/280029 | 00:18 |
|---|---|---|
| sdake | SamYaple around? | 00:19 |
| sdake | SamYaple did a charger get left in the office there? | 00:19 |
| sdake | usb charger type thing | 00:20 |
| sdake | 3 plugs iirc | 00:20 |
| sdake | anker iirc | 00:20 |
| *** salv-orlando has quit IRC | 00:21 | |
| *** tzn has quit IRC | 00:25 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Reduce gate resource usage https://review.openstack.org/280029 | 00:39 |
| SamYaple | sdake: doubtful | 00:40 |
| SamYaple | we cleaned up the office | 00:40 |
| *** sdake has quit IRC | 00:42 | |
| *** iNeilus has quit IRC | 00:42 | |
| openstackgerrit | Merged openstack/kolla-mesos: Zookeeper expects strings as values https://review.openstack.org/279566 | 00:45 |
| *** XuChao918 has joined #kolla | 00:49 | |
| *** tzn has joined #kolla | 00:50 | |
| *** iceyao has joined #kolla | 00:51 | |
| *** tzn has quit IRC | 00:57 | |
| *** sdake has joined #kolla | 00:58 | |
| asalkeld | SamYaple: i am testing that netns:shared and am getting an error from docker, just wondered if you have seen this | 01:00 |
| asalkeld | docker: Error response from daemon: Cannot start container db594243927bb066f788eb19e91b286aa6f28f0d7499c3737ea914c76e1cac43: Path /run/netns is mounted on /run but it is not a shared mount.. | 01:00 |
| asalkeld | this has both /run and /run/netns:shared | 01:01 |
| SamYaple | asalkeld: yes i have seen that | 01:01 |
| SamYaple | you need to update the docker daemon (systemctl) | 01:01 |
| asalkeld | ok, I did a yum update, guess i need to restart it too | 01:02 |
| SamYaple | hold on asalkeld | 01:02 |
| SamYaple | ill get the patch | 01:02 |
| sdake | yum update restarts docker daemon asalkeld | 01:02 |
| SamYaple | asalkeld: https://review.openstack.org/#/c/272715/ | 01:03 |
| asalkeld | looking | 01:03 |
| asalkeld | btw: http://paste.openstack.org/show/486941/ (marathon file) | 01:03 |
| SamYaple | asalkeld: look at the commit message | 01:03 |
| asalkeld | [Service] | 01:04 |
| asalkeld | MountFlags=shared | 01:04 |
| asalkeld | thanks! | 01:04 |
| SamYaple | anytime! | 01:04 |
| SamYaple | docs patch tonight for it | 01:04 |
| *** alisonh has left #kolla | 01:16 | |
| *** alisonh has joined #kolla | 01:26 | |
| *** salv-orlando has joined #kolla | 01:29 | |
| *** salv-orlando has quit IRC | 01:31 | |
| *** sdake_ has joined #kolla | 01:34 | |
| *** neilus has joined #kolla | 01:36 | |
| *** sdake has quit IRC | 01:36 | |
| *** neilus has quit IRC | 01:40 | |
| *** Jeffrey4l has joined #kolla | 01:47 | |
| *** sdake_ has quit IRC | 01:53 | |
| *** achanda has joined #kolla | 01:58 | |
| *** tzn has joined #kolla | 02:13 | |
| *** tzn has quit IRC | 02:17 | |
| *** sdake has joined #kolla | 02:31 | |
| sdake | SamYaple is that mountflags=shared the same problem as the kolla toolbox problem? | 02:53 |
| SamYaple | sdake: different issues | 03:07 |
| sdake | thanks | 03:10 |
| *** coolsvap|away is now known as coolsvap | 03:36 | |
| openstackgerrit | Merged openstack/kolla: Make pep8 *the* linting interface https://review.openstack.org/279384 | 03:47 |
| *** klint has joined #kolla | 03:48 | |
| openstackgerrit | Merged openstack/kolla: elevates privileges and grants access to /dev/mapper to cinder backup https://review.openstack.org/276990 | 03:52 |
| *** cfarquhar has quit IRC | 03:52 | |
| *** cfarquhar has joined #kolla | 03:54 | |
| *** cfarquhar has quit IRC | 03:54 | |
| *** cfarquhar has joined #kolla | 03:54 | |
| *** coolsvap is now known as coolsvap|away | 03:59 | |
| *** dave-mccowan has quit IRC | 04:00 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Reduce gate resource usage https://review.openstack.org/280029 | 04:07 |
| openstackgerrit | Angus Salkeld proposed openstack/kolla-mesos: Add neutron config https://review.openstack.org/265109 | 04:08 |
| openstackgerrit | Angus Salkeld proposed openstack/kolla-mesos: Respect the enable_<service> variables https://review.openstack.org/280058 | 04:08 |
| *** coolsvap|away is now known as coolsvap | 04:13 | |
| openstackgerrit | Merged openstack/kolla: Designate ubuntu binary container https://review.openstack.org/276676 | 04:16 |
| *** obuisson has quit IRC | 04:17 | |
| *** neilus has joined #kolla | 04:18 | |
| *** kbyrne has quit IRC | 04:19 | |
| *** kbyrne has joined #kolla | 04:19 | |
| openstackgerrit | Angus Salkeld proposed openstack/kolla-mesos: Respect the enable_<service> variables https://review.openstack.org/280058 | 04:22 |
| openstackgerrit | Angus Salkeld proposed openstack/kolla-mesos: Add neutron config https://review.openstack.org/265109 | 04:22 |
| asalkeld | nihilifer: that should be working ^, the mounting of volumes works. I had an issue connceting to rabbit but i think it's my setup issue | 04:24 |
| *** neilus has quit IRC | 04:24 | |
| *** Marga_ has joined #kolla | 04:24 | |
| *** Marga_ has joined #kolla | 04:25 | |
| asalkeld | nihilifer: note that "cleaning containers" part of the cleanup script is getting stuck | 04:26 |
| *** dims has joined #kolla | 04:28 | |
| *** dims has quit IRC | 04:30 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Reduce gate resource usage https://review.openstack.org/280029 | 04:43 |
| openstackgerrit | Angus Salkeld proposed openstack/kolla-mesos: Add neutron config https://review.openstack.org/265109 | 04:44 |
| openstackgerrit | Angus Salkeld proposed openstack/kolla-mesos: Move mariadb mount from default file https://review.openstack.org/280061 | 04:44 |
| *** salv-orlando has joined #kolla | 04:44 | |
| *** salv-orlando has quit IRC | 04:49 | |
| *** sdake has quit IRC | 04:57 | |
| *** daneyon_ has joined #kolla | 05:01 | |
| *** daneyon has quit IRC | 05:02 | |
| nihilifer | asalkeld: ok | 05:02 |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Reduce gate resource usage https://review.openstack.org/280029 | 05:04 |
| SamYaple | man i am out of it tonight | 05:04 |
| SamYaple | look at the diff history for that patch. i know all of those things :/ | 05:04 |
| openstackgerrit | Merged openstack/kolla-mesos: Respect the enable_<service> variables https://review.openstack.org/280058 | 05:06 |
| *** chandankumar has joined #kolla | 05:19 | |
| mdnadeem | ajafo, SamYaple , nihilifer , did any one notice this error while kolla-ansible deploy "msg: The requested image does not exist: localhost:4000/kollaglue/centos-binary-keystone:2.0.0" | 05:28 |
| SamYaple | mdnadeem: thats a pretty clear message | 05:28 |
| nihilifer | mdnadeem: this message means that it cannot pull the image from registry | 05:28 |
| SamYaple | the image doesnt exist in the registry | 05:28 |
| nihilifer | please check whether | 05:28 |
| mdnadeem | SamYaple, Yes, i am thinking is this a bug? | 05:29 |
| nihilifer | 1) registry works | 05:29 |
| nihilifer | 2) the image is present here | 05:29 |
| SamYaple | mdnadeem: no, its user error in this case | 05:29 |
| nihilifer | nope, it's not a bug... | 05:29 |
| mdnadeem | even this image does not exist in kollaglue repo | 05:29 |
| SamYaple | correct | 05:29 |
| SamYaple | you have to build it | 05:29 |
| mdnadeem | SamYaple, ok | 05:31 |
| *** achanda has quit IRC | 05:32 | |
| mdnadeem | SamYaple, I had run kolla-build before may be due to some error it is not build keystone image, However i cant get the log to debug the build error for keystone image | 05:40 |
| *** achanda has joined #kolla | 05:41 | |
| mdnadeem | did any one know, from where i can get kolla-build log to debug the error for the failed images? | 05:41 |
| SamYaple | mdnadeem: it outputs it when you build | 05:45 |
| *** tzn has joined #kolla | 05:45 | |
| SamYaple | make sure to pull the latest master | 05:45 |
| mdnadeem | SamYaple, Now i lost the ouput screen, however as i remember in output it shows only image failed , but not give any usefull info to debug it, like why it is failing | 05:47 |
| SamYaple | it gives all of the output that docker returns | 05:50 |
| *** tzn has quit IRC | 05:50 | |
| mdnadeem | SamYaple, ok, i will try to freshly build keystone image alone. | 05:50 |
| mdnadeem | SamYaple, It will be nice if error messages is properly logged in some log file. May be in future we can do this :) | 05:51 |
| mdnadeem | SamYaple, Now i get the actual error which cause keystone image build fail --> Error unpacking rpm package httpd-2.4.6-40.el7.centos.x86_64 | 05:56 |
| mdnadeem | which | 05:56 |
| kejlly_ | d o | 05:57 |
| kejlly_ | sorry, type in wrong place. | 05:57 |
| *** severion has joined #kolla | 06:05 | |
| *** alisonh has quit IRC | 06:05 | |
| *** v1k0d3n_ has quit IRC | 06:07 | |
| *** mdnadeem_wb has joined #kolla | 06:08 | |
| *** alisonh has joined #kolla | 06:08 | |
| *** mdnadeem has quit IRC | 06:10 | |
| *** mdnadeem has joined #kolla | 06:11 | |
| *** sdake has joined #kolla | 06:25 | |
| openstackgerrit | Merged openstack/kolla-mesos: Move mariadb mount from default file https://review.openstack.org/280061 | 06:26 |
| *** salv-orlando has joined #kolla | 06:30 | |
| *** Allen_Gao has quit IRC | 06:36 | |
| *** aginwala has joined #kolla | 06:38 | |
| *** sdake has quit IRC | 06:39 | |
| *** aginwala has quit IRC | 06:43 | |
| *** jasonsb has quit IRC | 06:43 | |
| *** Allen_Gao has joined #kolla | 06:49 | |
| *** achanda has quit IRC | 06:58 | |
| *** Marga_ has quit IRC | 07:01 | |
| *** neilus has joined #kolla | 07:01 | |
| *** dolpher has quit IRC | 07:03 | |
| *** vincent_vdk has quit IRC | 07:03 | |
| *** vincent_vdk has joined #kolla | 07:03 | |
| *** nihilifer has quit IRC | 07:03 | |
| *** Kennan_Vacation has quit IRC | 07:03 | |
| *** dolpher has joined #kolla | 07:04 | |
| *** nihilifer has joined #kolla | 07:04 | |
| *** Kennan_Vacation has joined #kolla | 07:04 | |
| *** neilus has quit IRC | 07:07 | |
| nihilifer | asalkeld: i see that new service format doesn't include "volumes" parameter in marathon. am i right? | 07:11 |
| nihilifer | https://mesosphere.github.io/marathon/docs/native-docker.html like in the 1st listing | 07:11 |
| nihilifer | if not, i'm going to add it. passing volumes directly as a docker parameter doesn't make them cleaned up by mesos, generally | 07:12 |
| nihilifer | so passing volumes as a docker param should be imo used only as a woekaround for named volumes | 07:12 |
| asalkeld | nihilifer: it does | 07:26 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: fix the custom profile raise exception issue https://review.openstack.org/272022 | 07:26 |
| asalkeld | just it doesn't support "shared" | 07:26 |
| asalkeld | so we use parameters section for the shared one | 07:26 |
| *** chandankumar has quit IRC | 07:28 | |
| *** achanda has joined #kolla | 07:34 | |
| *** tzn has joined #kolla | 07:34 | |
| *** chandankumar has joined #kolla | 07:35 | |
| *** tzn has quit IRC | 07:39 | |
| *** salv-orlando has quit IRC | 07:47 | |
| *** mdnadeem_ has joined #kolla | 07:51 | |
| *** iceyao has quit IRC | 07:52 | |
| *** iceyao has joined #kolla | 07:52 | |
| *** mdnadeem has quit IRC | 07:55 | |
| *** mdnadeem has joined #kolla | 07:55 | |
| *** mdnadeem_ has quit IRC | 07:57 | |
| *** fgimenez has joined #kolla | 08:01 | |
| *** fgimenez has quit IRC | 08:01 | |
| *** fgimenez has joined #kolla | 08:01 | |
| nihilifer | asalkeld: ah, ok | 08:09 |
| ajafo | o/ | 08:10 |
| *** iceyao_ has joined #kolla | 08:11 | |
| *** subscope has joined #kolla | 08:13 | |
| *** iceyao has quit IRC | 08:14 | |
| *** mbound has joined #kolla | 08:20 | |
| *** achanda has quit IRC | 08:26 | |
| *** mikelk has joined #kolla | 08:27 | |
| *** achanda has joined #kolla | 08:28 | |
| *** chandankumar has quit IRC | 08:30 | |
| *** Jeffrey4l has quit IRC | 08:30 | |
| *** neilus has joined #kolla | 08:32 | |
| *** achanda has quit IRC | 08:42 | |
| *** Jeffrey4l has joined #kolla | 08:42 | |
| ajafo | anyone had error like this 500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.22/containers/kolla_toolbox/start' ? | 08:43 |
| *** Serlex has joined #kolla | 08:44 | |
| *** chandankumar has joined #kolla | 08:45 | |
| *** neilus has quit IRC | 08:48 | |
| *** salv-orlando has joined #kolla | 08:59 | |
| *** salv-orlando has quit IRC | 08:59 | |
| *** salv-orlando has joined #kolla | 08:59 | |
| *** kproskurin has joined #kolla | 09:04 | |
| *** athomas has joined #kolla | 09:13 | |
| *** fgimenez has quit IRC | 09:13 | |
| *** fgimenez has joined #kolla | 09:14 | |
| *** dolpher has quit IRC | 09:16 | |
| *** dolpher has joined #kolla | 09:16 | |
| *** chandankumar has quit IRC | 09:17 | |
| *** chandankumar has joined #kolla | 09:17 | |
| *** athomas has quit IRC | 09:26 | |
| *** athomas has joined #kolla | 09:26 | |
| *** v1k0d3n_ has joined #kolla | 09:27 | |
| *** iNeilus has joined #kolla | 09:28 | |
| *** kproskurin has quit IRC | 09:28 | |
| *** severion has quit IRC | 09:28 | |
| *** elemoine- has quit IRC | 09:28 | |
| *** subscope has quit IRC | 09:29 | |
| *** elemoine has joined #kolla | 09:30 | |
| *** iNeilus has quit IRC | 09:33 | |
| *** subscope has joined #kolla | 09:37 | |
| *** akwasnie has joined #kolla | 09:51 | |
| *** tzn has joined #kolla | 09:58 | |
| *** tzn has quit IRC | 09:59 | |
| *** tzn has joined #kolla | 09:59 | |
| Jeffrey4l | ajafo, are u using pull action? | 10:01 |
| *** akwasnie1 has joined #kolla | 10:01 | |
| ajafo | Jeffrey4l: just sudo kolla-ansible deploy | 10:03 |
| *** salv-orl_ has joined #kolla | 10:03 | |
| ajafo | Jeffrey4l: and just reproduce it on second env | 10:03 |
| *** akwasnie has quit IRC | 10:04 | |
| ajafo | reproduced* | 10:04 |
| *** salv-orlando has quit IRC | 10:06 | |
| Jeffrey4l | ajafo, what's the docker-py and docker version? | 10:07 |
| ajafo | docker-engine-1.10.1-1.el7.centos.x86_64 and docker-py 1.7.0 | 10:07 |
| Jeffrey4l | hmm.. | 10:09 |
| Jeffrey4l | oh. | 10:09 |
| Jeffrey4l | got it. | 10:10 |
| Jeffrey4l | ajafo, it should be this issue. https://review.openstack.org/#/c/279064/ | 10:10 |
| ajafo | Jeffrey4l: checking | 10:11 |
| Jeffrey4l | ajafo, manul fix should: `rm -f /run/kolla/log && docker restart rsyslog && docker rm kolla-toolbox` | 10:11 |
| Jeffrey4l | then run kolla-ansible deploy again. | 10:12 |
| *** jmccarthy has quit IRC | 10:12 | |
| *** jmccarthy has joined #kolla | 10:14 | |
| ajafo | Jeffrey4l: it works :) thanks! | 10:15 |
| Jeffrey4l | ajafo, :D | 10:15 |
| *** gfidente has joined #kolla | 10:18 | |
| *** macsz has joined #kolla | 10:21 | |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Add neutron config https://review.openstack.org/265109 | 10:33 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Make Mesos resources configurable https://review.openstack.org/280144 | 10:33 |
| nihilifer | asalkeld: fyi ^^ | 10:33 |
| nihilifer | hardcoded 512 mb ram for everything doesn't fit well for standard vm size | 10:33 |
| *** alyson_ has joined #kolla | 10:38 | |
| *** akwasnie1 has quit IRC | 10:40 | |
| *** Jeffrey4l has quit IRC | 10:45 | |
| *** subscope has quit IRC | 11:02 | |
| *** Jeffrey4l has joined #kolla | 11:08 | |
| *** akwasnie has joined #kolla | 11:11 | |
| mdnadeem | @anyone , deploy fail with Error "response from daemon: Container mariadb is restarting, wait until the container is running" | 11:19 |
| mdnadeem | docker logs mariadb shows--> /usr/bin/mysqld_safe: line 139: /var/lib/mysql//kolla.err: Permission denied | 11:20 |
| mdnadeem | ^^ Any idea guys ?? | 11:21 |
| mdnadeem | nihilifer, jeffrey4l ^^ | 11:26 |
| *** subscope has joined #kolla | 11:30 | |
| ajafo | just notice similar error half hour ago now trying to recreate it | 11:31 |
| mdnadeem | ajafo, :(, i always stuck at some point while deploying kolla, hope fully i get success soon :) | 11:33 |
| ajafo | mdnadeem: ubuntu or centos? | 11:34 |
| mdnadeem | ajafo, ubuntu, source | 11:34 |
| mdnadeem | ajafo, u? | 11:34 |
| nihilifer | mdnadeem: you can try to do | 11:35 |
| nihilifer | docker exec -t <name_of_mariadb_container> /bin/bash | 11:36 |
| nihilifer | and check this command inside | 11:36 |
| nihilifer | stat -c %U:%G /var/lib/mysql | 11:36 |
| ajafo | ubuntu but few minutes ago notice some digitalocean mirror conecting problem | 11:36 |
| ajafo | nihilifer: it probably doesn't go in my case container was just restarting | 11:36 |
| mdnadeem | nihilifer, ok, i will try | 11:37 |
| nihilifer | maybe "-R" is needed in chown in mariadb's extend_start... | 11:38 |
| mdnadeem | nihilifer, can not start mriadb container , docker exec ----- fails --> Error response from daemon: Container mariadb is restarting, wait until the container is running | 11:41 |
| mdnadeem | nihilifer, time to catch my bus, will debug it more from home | 11:43 |
| *** limamauricio has joined #kolla | 11:43 | |
| *** fgimenez has quit IRC | 12:03 | |
| *** fgimenez has joined #kolla | 12:03 | |
| limamauricio | Hello there, I ran kolla-ansible deploy and I have a new error now in TASK: [keystone | Creating the admin project, user and role] | 12:05 |
| limamauricio | logs ->http://pastebin.com/rd5H0Ktb | 12:05 |
| limamauricio | :/ | 12:06 |
| *** rhallisey has joined #kolla | 12:06 | |
| *** iceyao_ has quit IRC | 12:08 | |
| *** iceyao has joined #kolla | 12:08 | |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla: Upgrade cinder playbook https://review.openstack.org/278049 | 12:10 |
| *** sdake has joined #kolla | 12:21 | |
| *** openstackgerrit_ has joined #kolla | 12:27 | |
| *** akwasnie has quit IRC | 12:28 | |
| *** coolsvap is now known as coolsvap|away | 12:36 | |
| openstackgerrit | venkatamahesh proposed openstack/kolla: Use uppercase 'S' in word "OpenStack" https://review.openstack.org/276230 | 12:37 |
| *** dims_ has joined #kolla | 12:42 | |
| *** jtriley has joined #kolla | 12:43 | |
| *** akwasnie has joined #kolla | 12:55 | |
| *** salv-orl_ has quit IRC | 13:04 | |
| *** Serlex has quit IRC | 13:05 | |
| *** iNeilus has joined #kolla | 13:05 | |
| *** athomas has quit IRC | 13:06 | |
| *** subscope has quit IRC | 13:10 | |
| *** iNeilus has quit IRC | 13:10 | |
| *** athomas has joined #kolla | 13:12 | |
| *** fgimenez has quit IRC | 13:12 | |
| *** JoseMello has joined #kolla | 13:13 | |
| *** dave-mccowan has joined #kolla | 13:14 | |
| ajafo | limamauricio: what kind of build have you got? ubuntu/centos binary/source? | 13:14 |
| *** fgimenez has joined #kolla | 13:15 | |
| *** fgimenez has quit IRC | 13:15 | |
| *** fgimenez has joined #kolla | 13:15 | |
| *** subscope has joined #kolla | 13:17 | |
| *** shakamunyi has quit IRC | 13:20 | |
| *** jtriley has quit IRC | 13:21 | |
| *** sbezverk has joined #kolla | 13:22 | |
| sbezverk | Hello, I am trying to deploy AIO on rhel 7.2 and I am getting this prechecks error. fatal: [192.168.80.25] => One or more undefined variables: 'dict object' has no attribute u'ansible_tenants-b0' | 13:24 |
| sbezverk | any suggestions where could be a misconfiguration? | 13:24 |
| *** dims_ has quit IRC | 13:33 | |
| sdake | SamYaple | 13:33 |
| sdake | re PDE | 13:33 |
| sdake | you had asked what it meant | 13:33 |
| sdake | I looked into it a bit | 13:33 |
| *** dims has joined #kolla | 13:33 | |
| sdake | one problem with PDE is it skews dashboarding to companies that have multiple contributors versus just one (like you with Servosity or me with Cisco) | 13:34 |
| sbezverk | Here is globals.yml, appreciate if somebody could check. http://paste.openstack.org/show/486981/ | 13:38 |
| *** salv-orlando has joined #kolla | 13:40 | |
| *** _tzn has joined #kolla | 13:45 | |
| *** tzn has quit IRC | 13:48 | |
| *** dwalsh has joined #kolla | 13:49 | |
| limamauricio | ajafo: centos/source | 13:51 |
| *** dims has quit IRC | 13:53 | |
| *** dims has joined #kolla | 13:57 | |
| *** klint has quit IRC | 13:57 | |
| sdake | morning cats | 13:57 |
| sdake | limamauricio still having trouble deploying? | 13:58 |
| limamauricio | yes | 13:58 |
| limamauricio | :/ | 13:58 |
| sdake | we fixed a critical (non-functional) bug in master for centos | 13:58 |
| limamauricio | TASK: [keystone | Creating the admin project, user and role] ? | 13:58 |
| sdake | if your on centos/source please git pull | 13:58 |
| sdake | sbezverk same goes | 13:58 |
| sdake | limamauricio yes that is the one we fixed | 13:58 |
| limamauricio | Wow!!!! | 13:58 |
| limamauricio | :) | 13:59 |
| sdake | the deal is keystone deprecated an option in their middleware | 13:59 |
| sdake | and changed the default middleware auth in the process of the deprecation | 13:59 |
| sdake | with a deprecation your not supposed to change default behavior | 13:59 |
| sdake | but projects do what they want :) | 13:59 |
| sdake | and we roll with it | 13:59 |
| sdake | still it is a bit disruptive | 14:00 |
| sdake | limamauricio just clarifying here, your on master correct? | 14:00 |
| limamauricio | yes | 14:00 |
| *** fthiagogv has joined #kolla | 14:00 | |
| limamauricio | just a git pull? | 14:00 |
| sdake | sbezverk did you build centos/binary to deploy on RHEL 7.2? | 14:00 |
| *** dwalsh has quit IRC | 14:00 | |
| *** dims has quit IRC | 14:00 | |
| sdake | sbezverk your issue is brand new, never been reported (this prechecks problem) | 14:01 |
| sdake | sbezverk I'm not quite sure where to begin, just woke up about 30 mins ago - let me get my day going and we can debug together if you have time | 14:01 |
| *** chandankumar has quit IRC | 14:02 | |
| *** dims has joined #kolla | 14:02 | |
| sdake | sbezverk just got through your globals.yml | 14:02 |
| sdake | sbezverk can you ip link show | fpaste please | 14:03 |
| sdake | sbezverk ansible --version | fpaste | 14:05 |
| sdake | sbezverk pip docker-py show | fpaste | 14:05 |
| sdake | pip show docker-py | fpaste that is | 14:05 |
| *** dwalsh has joined #kolla | 14:06 | |
| sdake | lima git pull --rebase will do the job | 14:06 |
| sdake | limamauricio ^^ | 14:06 |
| sbezverk | sdake I am making a backup of "deployment" vm to have something to roll back as soon as it is completed I will be able to debug it. Just to give an idea of a scenario here. I have a server, which is supposed to become AIO server, on that server I am already running qemu kolla deployment VM. | 14:06 |
| sdake | so you already running a deployment and want a second deployment? | 14:07 |
| sdake | let me reread tha tfor a minute - its not clear to me ;) | 14:08 |
| ajafo | morning sdake | 14:09 |
| *** iceyao has quit IRC | 14:09 | |
| sdake | ajafo morning | 14:10 |
| sdake | sbezverk i gave you some fpaste commands above to run - I need to see those please | 14:10 |
| *** kproskurin_ has joined #kolla | 14:11 | |
| sdake | sbezverk so if I understand correctly, you want to deploy openstack AIO using Kolla in a VM, then once its deployed, keep a copy of the qcow2 file around, and restore it immediately after testing openstack services | 14:13 |
| sbezverk | sdake requested commands: http://paste.openstack.org/show/486986/ | 14:18 |
| ajafo | sdake: I found solution on my last problems it was authplugin for ubuntu I saw that sby made patch for it too but it will be discussed because of compatibility with ubuntu/liberty, the same problem we have here https://bugs.launchpad.net/kolla/+bug/1533620 imho if's for liberty to have compatibility could be ok | 14:18 |
| openstack | Launchpad bug 1533620 in kolla "Nova-compute upgrade level to auto doesn't work on Ubuntu" [Undecided,New] - Assigned to Artur Zarzycki (azarzycki) | 14:18 |
| sdake | your prechecks are failing because eht0 is the only network interface yoru system has sbezverk | 14:18 |
| *** athomas has quit IRC | 14:19 | |
| *** mbound_ has joined #kolla | 14:20 | |
| *** JoseMello_ has joined #kolla | 14:21 | |
| *** Serlex has joined #kolla | 14:23 | |
| sbezverk | sdake right, kolla runs inside of a vm with FS located on a san storage, so any server can be used to host kolla vm for AIO installation. | 14:23 |
| *** athomas has joined #kolla | 14:23 | |
| sdake | ajafo i updated bug | 14:24 |
| *** dave-mccowan has quit IRC | 14:24 | |
| *** JoseMello has quit IRC | 14:24 | |
| *** mbound has quit IRC | 14:24 | |
| sdake | ajafo does that answer your question? | 14:24 |
| *** dave-mccowan has joined #kolla | 14:24 | |
| *** JoseMello_ is now known as JoseMello | 14:25 | |
| sdake | morning daneyon_ | 14:25 |
| sdake | dave-mccowan | 14:25 |
| dave-mccowan | good morning sdake | 14:25 |
| sdake | wtb implementation of https://blueprints.launchpad.net/kolla/+spec/diag-env | 14:26 |
| sdake | should take 5 minutes :) | 14:26 |
| sdake | plus days of review time :) | 14:26 |
| *** fgimenez has quit IRC | 14:27 | |
| sdake | dave-mccowan can you rebase your patch on master 279707 | 14:27 |
| sdake | so we can see if the gate works correctly with it | 14:27 |
| *** salv-orlando has quit IRC | 14:27 | |
| sdake | dave-mccowan https://review.openstack.org/#/c/279707/ | 14:27 |
| sdake | git git pull --rebase | 14:28 |
| sdake | git review will do the trick ;) | 14:28 |
| dave-mccowan | yep. will do. | 14:28 |
| sdake | i'm not sure when our master fixes merged to fix the gate | 14:28 |
| sdake | but it may ahve been less then 24 hours ago | 14:28 |
| sdake | i think it was about 12 hours ago | 14:28 |
| *** dims has quit IRC | 14:28 | |
| sdake | sbezverk so basically your globals.yml you are specifying a bunch of network interfaces that don't exist in the system | 14:29 |
| sdake | kolla doesn't create network interfaces on its own | 14:29 |
| sdake | if your using a vm, add a network interface for vm2 | 14:30 |
| *** fgimenez has joined #kolla | 14:30 | |
| sdake | rather for vm | 14:30 |
| sdake | then you will have eth0 and eth1 | 14:30 |
| sdake | use one for your mgmt network and one for your provider network | 14:30 |
| sdake | change provider_b0 or whateveryou have in globals.yml to eth1 | 14:30 |
| sdake | and mgmt_b0 to eth0 | 14:30 |
| sdake | or alternatively change eth0/eth1 to mgmt_b0, however that is done in ubuntu | 14:31 |
| openstackgerrit | Allen Gao proposed openstack/kolla: Remove the role path from the template src https://review.openstack.org/280262 | 14:31 |
| *** Allen_Gao has quit IRC | 14:33 | |
| ajafo | sdake: yes, thanks | 14:33 |
| *** dmsimard has joined #kolla | 14:33 | |
| *** jtriley has joined #kolla | 14:36 | |
| *** dims has joined #kolla | 14:36 | |
| *** macsz has quit IRC | 14:37 | |
| *** akwasnie has quit IRC | 14:38 | |
| *** akwasnie has joined #kolla | 14:39 | |
| sdake | elemoine are you around? | 14:40 |
| d_code | morning sdake | 14:41 |
| d_code | looks like over the weekend source builds were fixed? | 14:41 |
| sdake | d_code morning fine sir | 14:41 |
| d_code | the Keystone issue? | 14:41 |
| sdake | roger | 14:41 |
| d_code | w00t | 14:41 |
| d_code | binary? | 14:41 |
| d_code | still debugging the dhcp? | 14:41 |
| sdake | hang tight | 14:41 |
| sdake | talkingwith wife | 14:41 |
| sbezverk | sdake but these interface do exist on a target physical server, what os the point of kolla checking local VM for these interfaces, I am not deploying AIO in VM, I guess I am missing something here.. | 14:41 |
| d_code | kk | 14:42 |
| *** banix has joined #kolla | 14:42 | |
| *** akwasnie has quit IRC | 14:42 | |
| *** akwasnie has joined #kolla | 14:44 | |
| dave-mccowan | sdake spinning up kolla is on my to-do list today. i have a centos7 vm and an ubuntu 14.04 vm. should either of those work for a one node install? | 14:45 |
| sdake | dave-mccowan sounds good either shoudl work note i am having trouble with centos atm | 14:48 |
| sdake | sbezverk they need to exist in the vm | 14:48 |
| *** akwasnie has quit IRC | 14:48 | |
| sdake | d_code we think there is an issue remaining possiblly with dhcp - not sure, I cna't g et past toolbox deploy on my gear | 14:48 |
| sdake | i intended to debug that today | 14:48 |
| sdake | but centos is deploying in the gate correctly | 14:49 |
| openstackgerrit | Dave McCowan proposed openstack/kolla: Use variables to specify http or https when constructing URLs https://review.openstack.org/279707 | 14:49 |
| d_code | cool. I’m just gonna start a fresh build and wanted to see my options | 14:49 |
| sdake | so if you got as far as dhcp it may deploy for for | 14:49 |
| d_code | I’ll rock the source | 14:49 |
| d_code | :) | 14:49 |
| sdake | d_code ^^ | 14:49 |
| sdake | git pull --rebase | 14:49 |
| sdake | for you that is | 14:49 |
| d_code | yup | 14:49 |
| sdake | ok folks I need to head to airport for 45 mins to drop wife off for ansible fest or whatever she is going to | 14:49 |
| sdake | i'll be back | 14:49 |
| *** Jeffrey4l has quit IRC | 14:53 | |
| *** iNeilus has joined #kolla | 14:54 | |
| *** chandankumar has joined #kolla | 14:58 | |
| *** iNeilus has quit IRC | 14:58 | |
| *** dims has quit IRC | 14:59 | |
| *** sdake has quit IRC | 15:02 | |
| *** cloudnautique has joined #kolla | 15:05 | |
| *** haplo37 has joined #kolla | 15:07 | |
| *** dims has joined #kolla | 15:10 | |
| sbezverk | precheck docker version check fails. All version looks right, could someobdy check: http://paste.openstack.org/show/487000/ | 15:14 |
| *** cloudnautique has quit IRC | 15:15 | |
| *** cloudnautique has joined #kolla | 15:22 | |
| *** cloudnautique has quit IRC | 15:22 | |
| *** cloudnautique has joined #kolla | 15:23 | |
| *** cloudnautique has quit IRC | 15:23 | |
| *** cloudnautique has joined #kolla | 15:23 | |
| ajafo | sbezverk: you've probably old docker version 1.8.2? | 15:26 |
| ajafo | master is compatible with 1.10 | 15:26 |
| sbezverk | ajafo ok, then please change to doc page, it explicitely ask to downgrade to 1.8.2 | 15:28 |
| *** subscope has quit IRC | 15:30 | |
| ajafo | sbezverk: as I see in table is docker 1.10 (https://github.com/openstack/kolla/blob/master/doc/quickstart.rst) but later I see section with downgrade - it fresh change so probably nobody changed it, if nobody do this earlier I'll make changes today later | 15:32 |
| *** chandankumar is now known as chandankumar|out | 15:33 | |
| rhallisey | ajafo, downgrade is only for stable liberty | 15:33 |
| *** chandankumar|out is now known as chandan|brb | 15:33 | |
| rhallisey | I can fix the doc to make that more clear | 15:33 |
| rhallisey | master use 1.9 or higher | 15:33 |
| *** alyson_ has quit IRC | 15:34 | |
| *** blahRus has joined #kolla | 15:34 | |
| *** alyson_ has joined #kolla | 15:37 | |
| sbezverk | rhallisey with 1.10.1 docker, precheck has completed sucessfully. Now I need to deploy AIO on physical server,using kolla VM. For this scenario, would it be sufficient just to add physcial server ip to ansible inventory file? | 15:38 |
| rhallisey | ya I think for AIO that should be fine | 15:39 |
| sbezverk | <rhallisey> it is strange, it tries to install AIO on a local host which is my kolla VM, instead of deploying it on a physical server. What am I missing here? | 15:41 |
| *** subscope has joined #kolla | 15:41 | |
| dmsimard | SamYaple: You might have to do this earlier than you think: https://github.com/openstack/kolla/blob/d346545922fa3dfda742a9acc6ca6e9f140786ab/docker/keystone/Dockerfile.j2#L66 | 15:47 |
| *** thomasem has quit IRC | 15:50 | |
| *** thomasem has joined #kolla | 15:52 | |
| *** daneyon_ has quit IRC | 15:56 | |
| *** daneyon has joined #kolla | 15:56 | |
| rhallisey | sbezverk, maybe I'm confused with what your doing then | 15:59 |
| *** daneyon has quit IRC | 15:59 | |
| *** daneyon has joined #kolla | 16:00 | |
| *** jmccarthy has quit IRC | 16:03 | |
| SamYaple | dmsimard: looking | 16:04 |
| SamYaple | dmsimard: im doing it right now | 16:04 |
| *** daneyon_ has joined #kolla | 16:04 | |
| SamYaple | dmsimard: that was just a temp patch to get the gate and dev rolling again | 16:04 |
| dmsimard | SamYaple: I pulled the trigger a bit too quickly -- but keystone actually removed admin_auth_token from the default pipeline and broke a buncha things | 16:04 |
| SamYaple | yea dmsimard that sed lines _adds_ it back to the efault pipeline | 16:05 |
| dmsimard | yeah | 16:05 |
| SamYaple | its just deprecated not fully removed | 16:05 |
| *** jmccarthy has joined #kolla | 16:05 | |
| dmsimard | SamYaple: For RDO we'll do that right inside the packaging for this cycle | 16:07 |
| dmsimard | We submitted a partial revert to add it back into the default pipeline but I'm not sure it will go through at this point | 16:07 |
| *** daneyon has quit IRC | 16:07 | |
| *** mbound_ has quit IRC | 16:08 | |
| *** stvnoyes has quit IRC | 16:09 | |
| *** stvnoyes has joined #kolla | 16:09 | |
| sbezverk | <rhallisey> I have VM with kolla, I want to initiate AIO deployment from kolla VM onto a physical server. | 16:09 |
| SamYaple | dmsimard: honestly? Its deprecated. That means not-default in my opinon | 16:10 |
| SamYaple | dmsimard: im very happy with the change, but i understand your concern | 16:10 |
| *** jtriley has quit IRC | 16:11 | |
| dmsimard | SamYaple: it's just not a proper deprecation if it's removed in our opinion | 16:11 |
| dmsimard | SamYaple: "fix your stuff in packaging/installer" is the usual answer we get but this one didn't roll very good with us | 16:11 |
| dmsimard | like, okay, we'll create a nova api database now because you've been telling us since kilo | 16:12 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Add manila perms to /var/cache/manila https://review.openstack.org/280306 | 16:12 |
| *** sdake has joined #kolla | 16:12 | |
| dmsimard | but devstack switched to bootstrap back in december .. | 16:12 |
| *** jtriley has joined #kolla | 16:14 | |
| SamYaple | dmsimard: fair enough, but a day of breakage in the gate isn't worth a revert in my opinion | 16:14 |
| dmsimard | SamYaple: it broke everything leveraging puppet modules | 16:15 |
| dmsimard | users and installers | 16:15 |
| SamYaple | i understand what yo uare saying dmsimard | 16:15 |
| SamYaple | it broke kolla too | 16:15 |
| SamYaple | we fixed it and moved on. it is master after all | 16:15 |
| dmsimard | We pick our battles :) | 16:15 |
| SamYaple | i get it. i think im on the other side of the fence on this battle though | 16:16 |
| SamYaple | rhallisey: master is 1.10 | 16:16 |
| dmsimard | Yeah, *shrugs* it looks like there is disagreement even between the keystone cores on that change | 16:17 |
| dmsimard | The topic has been going for almost an hour now. | 16:17 |
| rhallisey | SamYaple, oh right because of the neutron thin containers | 16:17 |
| SamYaple | rhallisey: indeed | 16:17 |
| rhallisey | forgot about that | 16:17 |
| SamYaple | dmsimard: ive logged in. i like reading the drama! | 16:18 |
| dmsimard | SamYaple: you've missed a lot, look at the eavesdrop lol | 16:18 |
| dmsimard | seriously though, distros are getting fed up with upstream shovelling issues in our yard | 16:19 |
| SamYaple | dmsimard: eh its a two edged sword | 16:21 |
| SamYaple | new features new changes | 16:21 |
| SamYaple | it happens | 16:21 |
| SamYaple | this keystone-manage bootstrap thing is something I actually asked for a year ago | 16:22 |
| SamYaple | dmsimard: check these logs | 16:23 |
| SamYaple | ##ansible-shade.2015-07-07.log:03:16 < SamYaple> mordred: I was speaking about the SERVICE_TOKEN (its called admin_token in the conf file). The one that lets you setup the initial endpoints and users. Ill look into it, but its no rush on that now. Thanks for the info | 16:23 |
| SamYaple | ##ansible-shade.2015-07-07.log:03:49 < SamYaple> mordred: from a security standpoint, I went a different direction. I used the token to bootstrap just once from withinside the container. I dont wnt ansible spitting out that service token | 16:23 |
| SamYaple | ##ansible-shade.2015-07-07.log:03:58 < mordred> SamYaple: yah. actaully, morgan was talking about making keystone-manage able to handle this without needing the admin token middleware at all | 16:23 |
| openstackgerrit | Merged openstack/kolla: Use uppercase 'S' in word "OpenStack" https://review.openstack.org/276230 | 16:23 |
| rhallisey | sbezverk, so wait your using kolla to create kolla on a physical server? | 16:24 |
| rhallisey | from a vm | 16:24 |
| *** _tzn has quit IRC | 16:27 | |
| *** guillaume_ has joined #kolla | 16:27 | |
| d_code | I’m still getting the keystone errors on centos source build | 16:28 |
| sbezverk | rhallisey yes, basically at the end I want to have physical server running bunch of docker containers and offerring openstack services | 16:28 |
| d_code | i did a “git pull —rebase” this morning | 16:28 |
| guillaume_ | hello all, I got an error about when starting neutron_dhcp_agent : Path /run/netns is mounted on /run but it is not a shared mount. | 16:28 |
| rhallisey | sbezverk, what I don't understand though is what the kolla vm has to do with getting to that | 16:28 |
| d_code | on “Creating the admin project, user and role” | 16:28 |
| SamYaple | guillaume_: moment | 16:30 |
| *** chandan|brb is now known as chandankumar | 16:30 | |
| SamYaple | guillaume_: i have a docs patch started around that issue | 16:30 |
| guillaume_ | thx | 16:30 |
| SamYaple | guillaume_: for now, read this commit message https://review.openstack.org/#/c/272715/ | 16:31 |
| *** mbound has joined #kolla | 16:31 | |
| *** jasonsb has joined #kolla | 16:33 | |
| sbezverk | <rhallisey> Kolla VM will be re-used for other deployments. Example: I need to bring up openstack AIO, do something, then clean it and bring another AIO. I was hoping to Kolla VM as a deployment engine. So when I need to bring up new physical server, I just install OS, install docker container engine and the initiate deployment from Kolla VM, make sense? | 16:33 |
| rhallisey | sbezverk, so your talking about using that vm like an undercloud | 16:34 |
| *** tzn has joined #kolla | 16:35 | |
| *** tzn has quit IRC | 16:35 | |
| SamYaple | sbezverk: this is how I use kolla | 16:36 |
| sbezverk | SamYaple Glad to hear that it make sense to other folks. | 16:38 |
| guillaume_ | SamYaple : I dont understand where I need to put the [service] thing | 16:38 |
| guillaume_ | in /etc/init/docker.conf? | 16:38 |
| *** fgimenez has quit IRC | 16:39 | |
| rhallisey | sbezverk, cool | 16:39 |
| SamYaple | guillaume_: /usr/lib/systemd/system/docker.service | 16:39 |
| SamYaple | sudo sed -i 's|^MountFlags=.*|MountFlags=shared|' /usr/lib/systemd/system/docker.service | 16:39 |
| SamYaple | ^ that will do it for you guillaume_ | 16:39 |
| guillaume_ | thx a lot | 16:39 |
| SamYaple | youll need to reload systemctl and restart the service | 16:40 |
| guillaume_ | in ubuntu it is : /lib/systemd/system/docker.service | 16:41 |
| sbezverk | SamYapple could you share how you initiate deployment from VM to a physical host. When I try, it always try to deploy on a local host. | 16:41 |
| d_code | SamYaple: I’d recommed using /etc/systemd/system/docker.service instead. When you reinstall the RPM/DEB there’s no guarantees that your changes will be preserved | 16:41 |
| d_code | the /etc/ dir masks the /lib dir… this is its purpose | 16:41 |
| *** subscope has quit IRC | 16:42 | |
| *** iNeilus has joined #kolla | 16:42 | |
| SamYaple | d_code: i cant speak for centos, but on ubuntu that file does not exist | 16:43 |
| SamYaple | and the multi-user target is a symlink | 16:43 |
| *** fgimenez has joined #kolla | 16:43 | |
| d_code | hmm… weird | 16:43 |
| d_code | on CentOS…that’s the way to do it…it’s in the freedesktop spec…there should be a local directory that is intended for administrator modification of the service/socket/whatever files | 16:44 |
| SamYaple | d_code: may be bad packaging... what you suggest works on ubuntu too (its a systemd thing) but its not setup by default | 16:44 |
| d_code | ah…makes sense | 16:44 |
| *** mbound has quit IRC | 16:45 | |
| guillaume_ | SamYaple : in ubuntu it is : /lib/systemd/system/docker.service | 16:45 |
| *** jtriley has quit IRC | 16:46 | |
| *** jtriley has joined #kolla | 16:46 | |
| *** subscope has joined #kolla | 16:46 | |
| SamYaple | guillaume_: it is in centos too, what d_code was saying is /etc/systemd/system/docker.service survives upgrades (and he is right) | 16:46 |
| SamYaple | but you would have to add that file yourself on ubuntu | 16:47 |
| guillaume_ | ok | 16:47 |
| *** iNeilus has quit IRC | 16:47 | |
| guillaume_ | so I copy the file in /lib to this file? | 16:47 |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Reduce gate resource usage https://review.openstack.org/280029 | 16:50 |
| SamYaple | guillaume_: you can do that. or once youve made the appropriate change, you can just leave it alone | 16:51 |
| SamYaple | remember, that service file can change on upgrades | 16:51 |
| SamYaple | you don't want a stale service file around | 16:51 |
| guillaume_ | SamYaple, I now have this : docker start kolla_toolbox | 16:51 |
| guillaume_ | Error response from daemon: Cannot start container 6383443305d2335305924e1b708fcc45d6a0aff3b9b8d8e916bb595ac9830b58: [9] System error: not a directory | 16:51 |
| guillaume_ | Error: failed to start containers: kolla_toolbox | 16:51 |
| SamYaple | guillaume_: _that_ is an entirely different issue. | 16:51 |
| SamYaple | we know about this, it just started a few days ago. we have no confirmed causes yet | 16:52 |
| guillaume_ | ok strange | 16:52 |
| guillaume_ | For me it appears to be there since the change | 16:52 |
| SamYaple | what I would suggest is to stop the docker service and then restart your machine | 16:52 |
| guillaume_ | of the file | 16:52 |
| guillaume_ | before I was able to go further than that | 16:52 |
| *** ByPasS has quit IRC | 16:52 | |
| SamYaple | yea we are still tracking down the cause | 16:52 |
| guillaume_ | ok | 16:52 |
| *** ByPasS has joined #kolla | 16:54 | |
| sdake | morning | 16:55 |
| sdake | back | 16:55 |
| sdake | reading scrollback | 16:55 |
| *** mikelk has quit IRC | 16:57 | |
| sdake | d_code you need to rebuild your containers | 16:57 |
| d_code | sdake: I did cleanup-images | 16:57 |
| d_code | I’ll give it another go | 16:57 |
| sdake | SamYaple the docs shoudl prboably just have us overrride /etc | 16:59 |
| sdake | guilaume_ that problem is known and unsolved, we aree working on it | 16:59 |
| sdake | d_code sorry was reading scrollback | 16:59 |
| sdake | ok justa minute | 17:00 |
| sdake | let me think plz :) | 17:00 |
| sdake | d_code I have a suspicion there is some kind of container leak (atleast on centos) in the build system | 17:01 |
| *** Marga_ has joined #kolla | 17:01 | |
| sdake | can you run docker ps -a | fpaste | 17:01 |
| sdake | d_code (what would solve this short term is to use the nuclear option on docker cleanup :) | 17:01 |
| sdake | d_code but if you could put out the odcker ps -a output that would be helpful so we can confirm it | 17:02 |
| SamYaple | sdake: overriding /etc is the issue | 17:02 |
| SamYaple | ubuntu doesnt package /etc | 17:02 |
| SamYaple | but ill post some docs stuff and well get it figured out in review | 17:03 |
| sdake | sounds good | 17:03 |
| *** dims_ has joined #kolla | 17:03 | |
| sdake | if you could get somethign in queue soon that would be helpful to point people at :) | 17:03 |
| sdake | (re docs change) | 17:03 |
| SamYaple | sdake: yea thatll be tonight | 17:03 |
| sdake | sbezverk are you sorted? | 17:03 |
| sbezverk | sdake nope | 17:04 |
| SamYaple | was going to be last night, but then it didnt happen sinec it was sunday | 17:04 |
| sdake | right | 17:04 |
| SamYaple | sdake: btw, i had that mount issue on ubuntu | 17:04 |
| SamYaple | i rebooted and it worked | 17:04 |
| sdake | interesting | 17:04 |
| *** dims has quit IRC | 17:04 | |
| SamYaple | i think it may be a problem with changing the shared mount flag thing without a hard reboot | 17:04 |
| sdake | i have a debug log around here on one of my vms of docker running | 17:04 |
| sdake | i'll try that SamYaple | 17:05 |
| sdake | and see if i can get a deploy in my vm | 17:05 |
| SamYaple | if that _is_ the problem, i bet we can figure out a fix without a reboot | 17:05 |
| sdake | i can see why a reboot there would work using your speculation ;) | 17:06 |
| *** salv-orlando has joined #kolla | 17:06 | |
| *** Marga_ has quit IRC | 17:07 | |
| d_code | sdake: I blew it up…I’ll rebuild the source and see where I get | 17:09 |
| SamYaple | going to lunch | 17:10 |
| SamYaple | bbiab | 17:10 |
| *** salv-orlando has quit IRC | 17:11 | |
| sdake | restarting docker with mountflags shared doesn't fix toolbox problem trying reboot now | 17:12 |
| sdake | d_code any chance you can get your centos patch in the queue? | 17:12 |
| sdake | the host setup script | 17:12 |
| d_code | yeah, no problem. I was gonna wait until it worked :) | 17:13 |
| sdake | ;-) | 17:13 |
| d_code | can I just fork on github and do a pull request? | 17:13 |
| sdake | nope | 17:13 |
| d_code | or is there something different | 17:13 |
| d_code | lol | 17:13 |
| sdake | we use gerrit | 17:13 |
| d_code | k | 17:13 |
| sdake | doc links coming | 17:13 |
| d_code | k | 17:13 |
| sdake | groan can't find docs | 17:15 |
| sdake | just a moment need to use special google skillset ;) | 17:15 |
| sdake | http://docs.openstack.org/infra/manual/developers.html | 17:15 |
| sdake | d_code ^^ | 17:17 |
| sdake | when your ready | 17:17 |
| sdake | its pretty easy to setup - takes about 15 mins, if you need help ping here or openstack-infra | 17:17 |
| sdake | once your setup your setup forever | 17:17 |
| d_code | cool | 17:17 |
| sdake | well until the robots take over | 17:18 |
| *** thomasem has quit IRC | 17:18 | |
| sdake | SamYaple a reboot with sahred flag does not fix the kolla toolbox issue | 17:20 |
| *** kproskurin_ has quit IRC | 17:20 | |
| sbezverk | sdake it is still not clear how to initiate a deployment from Kolla VM onto a physical server. SamYaple mentioned that he is using this approach.. | 17:20 |
| sdake | sbezverk you deploy either on either a bare metal server or onto a vm | 17:20 |
| sdake | you could deploy from a vm to a bare metal server if you want | 17:20 |
| sdake | for that you will need a registry | 17:20 |
| SamYaple | sbezverk: oh perhpas i misunderstood your issue. can you elaborate? | 17:21 |
| SamYaple | I am fairly certain you can do what you want... but just to be clear | 17:21 |
| sdake | sbezverk here is an easy question to answer, do you want openstack running on the bare metal, or on the vm? | 17:21 |
| * sdake thinks we need a non-vagrant non-heat use-a-vm guide ;) | 17:22 | |
| sbezverk | Ok, I have VM hosted on my lab network which is running kolla and registery. I want to be able to deploy OpenStack (in a form of docker containers) on a physical server running ONLY OS and Docker engine. | 17:22 |
| sdake | or make vagrant more usable alteneratively | 17:23 |
| sdake | sbezverk roger got it | 17:23 |
| d_code | sdake: http://ur1.ca/oiv93 | 17:23 |
| *** thomasem has joined #kolla | 17:23 | |
| d_code | k. just finished source build | 17:23 |
| SamYaple | sbezverk: yea just setup your inventory with the physical hosts (not your kolla vm) and you should be good | 17:23 |
| *** fgimenez has quit IRC | 17:23 | |
| SamYaple | sbezverk: now this depends on your networking, obviously your vm and phsycial hosts need to talk to each other | 17:23 |
| SamYaple | but its that straight forward | 17:23 |
| sdake | d_code good to see nothign leaked there - that was with the ps -a flag right? | 17:24 |
| d_code | yes | 17:24 |
| sdake | cool mabye it was during deploy it leaks | 17:24 |
| sbezverk | SamYaple connectivity is not an issue, when I initiate deployment Kolla tries to deploy on a local host, even though it does not exist in ansible inventory file | 17:24 |
| sdake | I only saw it the one time i deployed on sunday (a non work dya for me :) | 17:25 |
| d_code | so, gonna deploy | 17:25 |
| d_code | `time kolla-ansible deploy -vvv` | 17:25 |
| SamYaple | sbezverk: how are you specifying your inventory? (exact kolla-ansible command) | 17:25 |
| SamYaple | d_code: that command for me deploys in under 2 minutes on multinode ;) | 17:25 |
| SamYaple | sdake: i got the gate down to an 8 minute deploy with ubuntu (though it averages 12m) | 17:26 |
| sdake | sbezverk there is an inventory file in ansible/inventory/all_in_one | 17:26 |
| sdake | SamYaple how | 17:26 |
| sdake | sbezverk it contains "localhost" | 17:26 |
| SamYaple | sdake: magic | 17:26 |
| sdake | you need to modify it or provide your own of the same format with the hosts specified | 17:26 |
| SamYaple | sdake: mirrors and dropping some non-used build containers and services in the deploy gate (ceph horizon heat) | 17:27 |
| d_code | failing on admin project again, looks like | 17:27 |
| sdake | sbezverk kolla-ansible takes an -i flag | 17:27 |
| sdake | ya mirrors probaby make a big difference | 17:27 |
| sdake | I did notice the huge delta between ubuntu gating speed and centos gating speed | 17:27 |
| SamYaple | mirrors got me to 12, the other changes took me to 8 | 17:27 |
| d_code | The error was: TypeError: Value of unknown type: <class 'keystoneauth1.exceptions.http.Unauthorized'>, The request you have made requires authentication. (HTTP 401) (Request-ID: req-03152288-1ff9-4faa-99eb-73de33c97fd4)" | 17:27 |
| SamYaple | sdake: no rpm mirrors yet. im kkeeping tabs on the issue in the infra | 17:28 |
| d_code | 2m19s | 17:28 |
| SamYaple | d_code: grab the latest master code and rebuild your images | 17:28 |
| SamYaple | this was fixed and merged last night | 17:28 |
| sdake | 8ef7434770846e81aa04009838bd1df69cc29ac8 | 17:28 |
| sdake | d_code do you ahve that commit in your repo? | 17:29 |
| sdake | git pull --rebase should have pulled it in | 17:29 |
| d_code | just updated to ‘84f4314’ | 17:29 |
| sdake | hrm | 17:29 |
| SamYaple | sdake: even if its in the repo doesnt mean his kolla image for keystone has it (old build) | 17:29 |
| d_code | k | 17:30 |
| d_code | gonna nuke this whole box | 17:30 |
| d_code | well. | 17:30 |
| sdake | SamYaple he wiped out /var/lib/kolala 20 seconds ago and rebuilt | 17:30 |
| d_code | docker | 17:30 |
| sdake | d_code hang tight | 17:30 |
| SamYaple | *boom* | 17:30 |
| sdake | rather 20 mins ago | 17:30 |
| SamYaple | ah | 17:30 |
| sdake | rather /var/lib/docker | 17:30 |
| SamYaple | probably wants head of master then | 17:30 |
| SamYaple | wasnt* | 17:30 |
| sdake | the commit he is on is head of master | 17:30 |
| sdake | 84f314 is that OpenStack change thing | 17:31 |
| SamYaple | sdake: 17:29:53 < d_code> just updated to | 17:31 |
| sdake | I don't know if it acutally works baremetal because I can't get past toolbox | 17:31 |
| SamYaple | he said he just updated | 17:31 |
| sdake | by just he menas about an hour ago | 17:31 |
| SamYaple | he said it 2 minutes ago | 17:31 |
| sdake | he did it an hour ago | 17:31 |
| SamYaple | alrighty | 17:31 |
| sdake | d_code ? | 17:31 |
| SamYaple | its fine | 17:31 |
| *** aginwala has joined #kolla | 17:32 | |
| *** subscope has quit IRC | 17:33 | |
| SamYaple | sdake: https://jenkins02.openstack.org/job/gate-kolla-dsvm-deploy-centos-binary/284/console | 17:34 |
| SamYaple | sdake: that fails on kolla_toolbox | 17:34 |
| SamYaple | it happens in the gate... SOMETIMES | 17:34 |
| sdake | interesting | 17:34 |
| d_code | sdake: I updated when you told me to do the rebase | 17:35 |
| d_code | then rebuilt | 17:35 |
| d_code | then I updated again…so there were new changes that came in | 17:35 |
| sdake | i see | 17:35 |
| *** tzn has joined #kolla | 17:35 | |
| sdake | well when i told you to do the rebase the centos changes wer ein | 17:35 |
| sdake | but your on tip of master atm | 17:35 |
| *** salv-orlando has joined #kolla | 17:36 | |
| sdake | SamYaple well at this point I'd go harass the docker devs for advice if Icould get into the damn channel | 17:36 |
| sdake | but -ETOOMANYPASSWORDS | 17:36 |
| SamYaple | sdake: already on it. no worries | 17:37 |
| SamYaple | ive been talking with lk4kd(?) about it for a few days | 17:37 |
| sdake | ya | 17:37 |
| *** cloudnautique has quit IRC | 17:37 | |
| sdake | he rocks | 17:37 |
| SamYaple | he walked me through the initial issues for rsyslog to pump out that fix | 17:37 |
| *** jtriley has quit IRC | 17:37 | |
| SamYaple | indeed | 17:37 |
| d_code | sooo….. nuke it? | 17:37 |
| sdake | you mean the shared fix? | 17:37 |
| SamYaple | im almost positive i know the breaking commit, but why does it work _sometimes_ | 17:37 |
| sdake | let me know the ommit, i'll revert in my vm and let you know for sure ;-) | 17:38 |
| sdake | sometimes = race condition typically | 17:38 |
| sdake | threads ftw! | 17:38 |
| sdake | SamYaple if you can point out the commit that you think causes the problem, I can assist in debug | 17:39 |
| SamYaple | no its the same one from rsyslog issue | 17:39 |
| SamYaple | but i think its modifying /dev perms | 17:40 |
| sdake | docker has special magic code to mount /dev | 17:40 |
| *** jtriley has joined #kolla | 17:40 | |
| sdake | asked for it in 1.6.0 time | 17:40 |
| sdake | i guess we got it ;-) | 17:40 |
| sdake | so which commit SamYaple so I can test if it is indeed the problem | 17:41 |
| sdake | i dont know which rsyslog commit was the problem | 17:41 |
| sdake | if we can narrow it down to a specific commit - it narrows the search area for the problem | 17:41 |
| sdake | sounds like you think its the bindmounting of /dev | 17:41 |
| sdake | yup not bindmounting dev fixes the problem | 17:43 |
| sdake | SamYaple ^^ | 17:43 |
| SamYaple | sdake: thats not it at all | 17:44 |
| sdake | I removed /dev:/dev from bindmount of kolla-toolbox and we are cooking now | 17:44 |
| SamYaple | i mean the special magic code | 17:44 |
| SamYaple | and you can also make it work without binding in /run | 17:44 |
| sdake | i'll try that | 17:44 |
| SamYaple | but its a comlicated issue im almost possitive is related to permissions in /dev | 17:44 |
| *** openstackgerrit has quit IRC | 17:45 | |
| *** blahRus1 has joined #kolla | 17:46 | |
| *** openstackgerrit has joined #kolla | 17:46 | |
| *** neophy has joined #kolla | 17:46 | |
| sdake | SamYaple removing /run:/run has zero effect - internal server error 500 | 17:46 |
| SamYaple | sdake: http://paste.openstack.org/show/487028/ | 17:46 |
| SamYaple | like i said, its _probably_ related to that | 17:47 |
| SamYaple | and its _probably_ a permissions issue | 17:47 |
| SamYaple | but lots of people are out in docker-dev today | 17:47 |
| sdake | git SamYaple ya i pasted that a couple days ago | 17:47 |
| *** aginwala has quit IRC | 17:47 | |
| sdake | git/ya | 17:47 |
| sdake | the sandbox thing is where it begins | 17:47 |
| *** blahRus has quit IRC | 17:48 | |
| sdake | rhallisey can you go bug dan walsh to join channel plz | 17:48 |
| sdake | he implemented the dev:dev bindmounting | 17:48 |
| rhallisey | ya I'll see if he's around | 17:48 |
| sdake | thx | 17:49 |
| d_code | SamYaple: the MountFlags=shared addresses the dhcp issue? | 17:50 |
| SamYaple | yea | 17:50 |
| d_code | k…that’s the hurldle I was running into on centos binary…I’ll retry with that | 17:52 |
| d_code | except I just nuked docker, so I’ll rebuild :) | 17:52 |
| sdake | ok it has nothign to do with run | 17:52 |
| sdake | bindmounting /dev:/dev followed by bindmounting /etc/:/dev/log | 17:53 |
| sdake | triggers problem | 17:53 |
| sdake | its a double bindmount on the dev fs | 17:53 |
| sdake | 100% reproducible on my gear | 17:53 |
| SamYaple | sdake: yea thats what me and inc0 were looking at last night | 17:53 |
| SamYaple | but the question is _why_ it only breaks sometimes | 17:54 |
| sdake | well you said it was run, it has nothing to do with run ;) | 17:54 |
| SamYaple | maybe all the time out your gear, but its a race looking at the gate failures | 17:54 |
| sdake | yup roger on the race | 17:54 |
| sdake | definately looks like a race condition | 17:54 |
| sdake | but could also be environmental | 17:54 |
| SamYaple | 17:45:00 < SamYaple> and you can also make it work without binding in /run | 17:54 |
| sdake | what are gate permissions on /dev/log - any idea? | 17:54 |
| SamYaple | oops */dev | 17:55 |
| SamYaple | look at context, i meant /dev sorry | 17:55 |
| sdake | got it | 17:55 |
| sdake | ok well i'm up to speed now | 17:55 |
| SamYaple | so we have another option | 17:55 |
| SamYaple | i didnt want to mad rush into it | 17:55 |
| SamYaple | but we dont _have_ to bind to /dev/log | 17:55 |
| SamYaple | we can configure most (all?) service to log to any other socket | 17:55 |
| SamYaple | wanted to flesh out the cause first though | 17:56 |
| sdake | when dan comes around we can get some free consulting on why double dev bindmounts dont work ;-) | 17:56 |
| SamYaple | wont work sometimes | 17:56 |
| sdake | which version of docker are you running SamYaple ? | 17:57 |
| SamYaple | 1.10 | 17:57 |
| sdake | i've got 1.10.1 | 17:57 |
| SamYaple | 1.10.1 | 17:57 |
| SamYaple | * | 17:57 |
| sdake | well its clearly a docker problem | 17:57 |
| sdake | either docker doesn't support that model of bindmounting (not intended in the design) or its broken in its implementation | 17:58 |
| *** blakec has joined #kolla | 17:58 | |
| sdake | how long has this problem been around? | 17:58 |
| sdake | just on docker 1.10? | 17:58 |
| *** cloudnautique has joined #kolla | 17:59 | |
| *** cloudnautique has quit IRC | 17:59 | |
| *** cloudnautique has joined #kolla | 17:59 | |
| SamYaple | again, its a race | 17:59 |
| *** cloudnautique has quit IRC | 17:59 | |
| SamYaple | likely the race is trying to bind to /dev/log before /dev is bound | 18:00 |
| SamYaple | and I would be willing to bet I can fix that | 18:00 |
| SamYaple | infact im almost positive I can now | 18:00 |
| *** cloudnautique has joined #kolla | 18:00 | |
| SamYaple | patch incoming | 18:00 |
| SamYaple | its not a race | 18:00 |
| SamYaple | its an unsorted dict! | 18:00 |
| sdake | oh well that would explain it | 18:00 |
| SamYaple | sometimes /run/kolla/log:/dev/log is before /dev/:/dev/ | 18:00 |
| SamYaple | i bet you thats it actually | 18:01 |
| SamYaple | error message makes sense too | 18:01 |
| sdake | paste patch I'll test in my 100% reprodubiel environemnt :) | 18:01 |
| sdake | docker shouldn't implode as it does - it leaks a container in this scneario | 18:03 |
| sdake | but thats a minor problem if its just a ordering issue on the mounting operation | 18:03 |
| SamYaple | sdake: check that patch | 18:04 |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Fix sandbox issues on some hosts https://review.openstack.org/280347 | 18:04 |
| SamYaple | sdake: that _may_ not fix it | 18:05 |
| SamYaple | since im reading and ansible dict | 18:05 |
| SamYaple | actually | 18:05 |
| SamYaple | no its a list | 18:05 |
| SamYaple | anyway i think this is the right path | 18:06 |
| sdake | so i think sort should be after the : | 18:07 |
| *** Marga_ has joined #kolla | 18:07 | |
| sdake | rather the nthe full string | 18:07 |
| sdake | does this patch do that? | 18:08 |
| SamYaple | im not sure what you are asking. this keeps the dict ordered | 18:10 |
| sdake | doesn't work | 18:10 |
| SamYaple | so when docker-py loops over it it sets it up in the right order | 18:10 |
| *** tzn has quit IRC | 18:10 | |
| SamYaple | ok sdake, lets go another direction here. do it break with _only_ dev? | 18:10 |
| sdake | git right | 18:10 |
| sdake | right | 18:10 |
| SamYaple | what combination breaks it? | 18:10 |
| sdake | i dont know let me try that line of thinking | 18:10 |
| sdake | ls | 18:11 |
| *** achanda has joined #kolla | 18:11 | |
| SamYaple | if its /dev and /run/olla/log:/dev/log then think its still an ordering issue | 18:11 |
| sdake | right | 18:11 |
| sdake | just needs to be ssorted after the : ;) | 18:11 |
| SamYaple | sorted is the wrong approach | 18:11 |
| SamYaple | ordered is the right | 18:12 |
| sdake | right | 18:12 |
| SamYaple | ordered keeps them in order as they get added, sorted rearrages | 18:12 |
| SamYaple | im heading back to office will check back in in a minute | 18:12 |
| sdake | chagning order of dev moutns in ansible file has no effect | 18:13 |
| sdake | possible ansible sorts it | 18:13 |
| *** dwalsh has quit IRC | 18:14 | |
| SamYaple | sdake: no ansible has a list | 18:19 |
| SamYaple | lists arent sorted | 18:19 |
| SamYaple | docker-py gets a dict | 18:20 |
| SamYaple | dicts are based on hashes | 18:20 |
| SamYaple | its possible its chaning there | 18:20 |
| sdake | yup | 18:21 |
| sdake | can you try sorting from the : onwards | 18:21 |
| sdake | and i'll tes tthat out | 18:21 |
| sdake | so rax just announced RHEL OSP as a service | 18:22 |
| sdake | 99.99% uptime | 18:22 |
| sdake | lol | 18:22 |
| sdake | 4 9's availability unlikely imo ;-) | 18:22 |
| sdake | SamYaple i've got a 30 min meeting but i'll be back, if you could work up a patch with a sort after the first : if dev is matched in the string, I'll try that | 18:23 |
| sdake | if not, i'll write it myself and try it when I get back ;) | 18:23 |
| sdake | kind of hacky but dev has special magic for moutning and clearly we are doing something wrong ;-) | 18:24 |
| sdake | too bad danwalsh isn't around | 18:25 |
| sdake | he could explain better the dev bindmounting process - its different then other filesystems | 18:25 |
| sdake | i just dont have the details to offer up the differences | 18:25 |
| sdake | but i am certain its different because i looked at the patch the docker cats and rht put together for it in libcontainer | 18:26 |
| sdake | and said | 18:26 |
| sdake | "go" | 18:26 |
| sdake | "dont have time for that" :) | 18:26 |
| *** jtriley has quit IRC | 18:26 | |
| *** dwalsh has joined #kolla | 18:31 | |
| d_code | guess who’s got two thumbs and the latest build of centos binary kolla running on his box? | 18:32 |
| d_code | this guy | 18:32 |
| sdake | it ran? | 18:34 |
| sdake | wish it worked for me ;-) | 18:35 |
| sdake | what did you do differently ;) | 18:35 |
| sdake | you didn't get stuck on the kolla_toolbox thing? | 18:35 |
| sdake | d_code this is a normal state of kolla (master always runs) its this recent change to 1.10 which has been a little disurptive but necessary ;) | 18:36 |
| d_code | :) | 18:36 |
| sdake | d_code the main problem is our gates are not voting | 18:36 |
| sdake | the reason we cna't make our gates voting is because the upstream repos go down and then our gate gets blocked for days on end | 18:36 |
| d_code | I added the MountFlags=shared to the /etc/systemd/system/docker.service file | 18:36 |
| sdake | we are working on solving that | 18:36 |
| d_code | can you setup like a Pulp instance that mirrors upstream? | 18:37 |
| d_code | would make your builds much faster too | 18:37 |
| d_code | sdake: now that it’s working… I’ll try to replicate it using my instructions and work on submitting a patch | 18:39 |
| d_code | for the build script | 18:39 |
| sbezverk | SamYaple I got my VM back on track. For the deployment we talked earlier, in globals.yml, do I specify values for my remote physical server or for VM? | 18:40 |
| sdake | sbezverk the internal_address should be a free VIP on your network | 18:41 |
| sdake | sbezverk the network interface names should be the names on your physical vm | 18:41 |
| sbezverk | sdake but there is no physical vm, there is physical server OR VM, what is it then? | 18:43 |
| d_code | sdake: so…neutron_external_interface… by default that is NATing public traffic out? | 18:43 |
| sdake | replace physical vm in above line with physical bar emetal | 18:43 |
| *** Marga_ has quit IRC | 18:43 | |
| sdake | however neutron works, I really dont know ;) | 18:44 |
| sdake | its the "br-ex" in devstack | 18:44 |
| sdake | iirc | 18:44 |
| d_code | the network for that interface has a gateway of 192.168.1.1/24 | 18:44 |
| d_code | yes…that’s correct | 18:44 |
| sdake | init-runonce | 18:44 |
| sdake | is what I use to setup my network | 18:44 |
| d_code | ah | 18:44 |
| d_code | I’ll poke into that | 18:44 |
| sdake | thats tailored to my environment | 18:44 |
| sdake | you will have to tailor to your personal network | 18:44 |
| sdake | but its pretty close | 18:44 |
| rhallisey | sdake, that /dev issue is solved? | 18:45 |
| rhallisey | thought I read you guys figured it out | 18:45 |
| *** Marga_ has joined #kolla | 18:49 | |
| *** Marga_ has quit IRC | 18:49 | |
| d_code | sdake: cool. just had to create the right floating IPs and it worked | 18:51 |
| d_code | it…lunch then back to tinkering | 18:51 |
| sdake | rhallisey unsolved thus far | 18:52 |
| sdake | we know that moutnign /dev and /dev/log at same time result in docker cratering | 18:52 |
| sdake | we are not sure if ordering of the moutning matters | 18:52 |
| sbezverk | sdake I adjusted globals.yml. now /usr/share/kolla/ansible/inventory/all-in-one, there is line localhost ansible_connection=local | 18:52 |
| *** Marga_ has joined #kolla | 18:52 | |
| sbezverk | sdake do I change it for my server ip address and connection ssh? | 18:52 |
| *** achanda has quit IRC | 18:54 | |
| *** subscope has joined #kolla | 18:55 | |
| openstackgerrit | imain proposed openstack/kolla: A few silly grammar fixes. https://review.openstack.org/280363 | 18:58 |
| *** banix has quit IRC | 18:59 | |
| *** vhosakot has joined #kolla | 19:00 | |
| sbezverk | sdake Started deployment and it fails: I pasted logs from the server and from VM here: http://paste.openstack.org/show/487032/ Please check. | 19:03 |
| limamauricio | TASK: [neutron | Starting neutron-dhcp-agent container] *********************** failed: [localhost] => {"changed": true, "failed": true} msg: APIError(HTTPError(u'500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.22/containers/neutron_dhcp_agent/start',),) | 19:04 |
| *** tzn has joined #kolla | 19:07 | |
| sbezverk | never mind I figured out.. deployment is still going.. | 19:08 |
| dave-mccowan | hello SamYaple | 19:12 |
| Serlex | sbezverk I had a similar error but durin vagrant multi-mode deployment | 19:14 |
| Serlex | limamauricio - is that a centos AIO deployment? | 19:15 |
| limamauricio | yes | 19:15 |
| sbezverk | Serlex that one I figured out, my baremetal server did not have insecure access to registry | 19:15 |
| Serlex | If so, myself and few others here have come across same problem | 19:15 |
| Serlex | Is it resolved? I'm not sure, sdake knows best | 19:15 |
| sbezverk | Serlex but now the deployment is failing because mariadb does not start on the server | 19:15 |
| Serlex | oh yeah, I remember now sbezverk, you are correct! It took me few hours to figure it out | 19:16 |
| sbezverk | failed: [192.168.80.25] => {"attempts": 6, "changed": false, "cmd": ["docker", "exec", "mariadb", "ls", "/var/lib/mysql/mariadb.pid"], "delta": "0:00:00.051707", "end": "2016-02-15 14:11:32.886853", "failed": true, "rc": 1, "start": "2016-02-15 14:11:32.835146", "stdout_lines": [], "warnings": []} | 19:16 |
| sbezverk | stderr: Error response from daemon: Container mariadb is restarting, wait until the container is running | 19:16 |
| Serlex | whats the status in docker ps -a | 19:16 |
| *** fthiagogv_ has joined #kolla | 19:16 | |
| Serlex | limamauricio - wait for sdake to confirm a work around. Did you build using binary or source? | 19:17 |
| sbezverk | Serlex http://paste.openstack.org/show/487035/ | 19:18 |
| Serlex | hmm I'm not an expert on docker or kolla I'm afraid | 19:19 |
| sdake | was in meeting | 19:19 |
| sdake | whats up | 19:19 |
| *** fthiagogv has quit IRC | 19:19 | |
| *** limamauricio has quit IRC | 19:19 | |
| Serlex | hi | 19:19 |
| sdake | let me read scrollback | 19:19 |
| sbezverk | sdake mariadb container is flapping like a toilet seat | 19:19 |
| sdake | so d_code was able to deploy with the shared workaround | 19:20 |
| Serlex | limamauricio - is having the same ' Starting neutron-dhcp-agent container' | 19:20 |
| sdake | sbezverk haven't seen that before | 19:20 |
| sdake | Serlex ya thats fixed | 19:20 |
| Serlex | ahh awesome | 19:20 |
| Serlex | great, i can get back to playing around with kolla | 19:20 |
| Serlex | how do I apply the fix? | 19:20 |
| sdake | pull master | 19:20 |
| SamYaple | dave-mccowan: hey | 19:20 |
| sdake | blow away /var/lib/docker | 19:20 |
| sdake | stop docker | 19:21 |
| sdake | blow away /var/lib/docker | 19:21 |
| sdake | start docker | 19:21 |
| sdake | and start a fresh would be my recommendation ;) | 19:21 |
| Serlex | ahhhhh | 19:21 |
| d_code | Serlex: I had to edit the openstack.service file too | 19:21 |
| Serlex | :'( | 19:21 |
| sdake | if you need to keep /var/lib/docker | 19:21 |
| sdake | d_code which one? | 19:21 |
| sdake | then just delete all containers shown in docker ps -a | 19:21 |
| sdake | docker rm -f ab | 19:21 |
| dave-mccowan | SamYaple I have some time over the next couple weeks to help out with TLS endpoints. do you have a design in mind, or a proposal on how to collaborate? | 19:21 |
| d_code | sdake: the change seems to have been the mount options | 19:21 |
| sdake | where ab is teh first 2 or 3 characters of the id | 19:21 |
| Serlex | I will try again, sorry for more dump questions, maste you mean clone again using git? | 19:21 |
| SamYaple | sdake: re: "after the split" thats not what happening | 19:22 |
| SamYaple | ordereddicts need ot be setup this way | 19:22 |
| d_code | my /etc/systemd/system/docker.service (on CentOS 7.2) http://ur1.ca/oivwz | 19:22 |
| SamYaple | what do you think is going to occur? | 19:22 |
| sdake | serlex that works or git pull --rebase will do it too | 19:22 |
| d_code | that was the fix for the neutron dhcp thing | 19:22 |
| Serlex | ok cheers | 19:22 |
| d_code | I also rebased | 19:22 |
| sdake | ok time for a break | 19:22 |
| sdake | bbiaf | 19:22 |
| d_code | Serlex: so that was a binary install too | 19:22 |
| *** chandankumar has quit IRC | 19:22 | |
| SamYaple | dave-mccowan: its fairly simply, if external_ssl is enabled then we push a user provided ssl cert to haproxy and set that up | 19:23 |
| d_code | the source one was still giving me issues with Keystone | 19:23 |
| Serlex | d_code openstack.service or docker.service? | 19:23 |
| SamYaple | all in all im thinking under 20 lines of code | 19:23 |
| d_code | oh | 19:23 |
| Serlex | I will do a fresh binary bare-metal install tomorrow | 19:23 |
| d_code | docker.service | 19:23 |
| Serlex | ok | 19:23 |
| Serlex | I will keep a record of your copy | 19:23 |
| SamYaple | i think the biggest patch was the one you already submited dave-mccowan | 19:23 |
| SamYaple | well no. because we need the second public vip for haproxy. but again not to much work | 19:24 |
| d_code | Serlex: here’s the steps I did -> https://gist.github.com/dcode/aea450c1648034067637 | 19:24 |
| dave-mccowan | yea, in lines of code that's the bulk. there's a few more tweaks needs for some services, especially horizon. | 19:24 |
| d_code | I’m gonna verify those work on a clean OS install (CentOS 7.2 minimal) and I’ll work on submitting a patch to Kolla | 19:24 |
| *** neilus has joined #kolla | 19:24 | |
| d_code | in the case of that gist, 192.168.4.210 is my interface IP and .200 is the VIP | 19:25 |
| dave-mccowan | also, we need a way to test it. do you think a new gate job would be in order? | 19:26 |
| sdake | dave-mccowan our gate doens't do functiona ltesting atm | 19:27 |
| sbezverk | any chance the issue I see with mariadb might be related to docket 1.10.1?? | 19:27 |
| sdake | so the answer is yes we need gating ;) | 19:28 |
| dave-mccowan | SamYaple haproxy.cfg.j2 also needs some work. this is where the 2 VIPs discussion comes in. | 19:28 |
| *** neilus has quit IRC | 19:29 | |
| *** salv-orlando has quit IRC | 19:30 | |
| *** jtriley has joined #kolla | 19:30 | |
| dave-mccowan | sdake samyaple line 38 of https://etherpad.openstack.org/p/kolla-mitaka-midcycle-ssl is how i expect haproxy.cfg to look. thoughts on this? (line 26 is how it looks now) | 19:32 |
| sdake | serlex is this --insecure-registry flag not documented? | 19:33 |
| sdake | the error lima sees is what SamYaple and I are workign on atm | 19:34 |
| *** achanda has joined #kolla | 19:35 | |
| d_code | sdake: https://github.com/openstack/kolla/blob/master/doc/quickstart.rst#deploy-a-registry-required-for-multinode | 19:36 |
| d_code | it’s documented | 19:36 |
| sdake | yap retty sure i put that in ;) | 19:36 |
| *** banix has joined #kolla | 19:36 | |
| sdake | i really dont know what to do when people don't read the quickstart ;) | 19:37 |
| sdake | i would get it if the quickstart were wrong, which it is at the moment | 19:37 |
| sdake | because it is mising the docker service file seding | 19:37 |
| sdake | sbezverk ok lets debug mariadb container plz | 19:38 |
| *** jtriley has quit IRC | 19:38 | |
| SamYaple | dave-mccowan: haproxy.cfg.j2 is super simple to tweak | 19:39 |
| SamYaple | you can listen on multiple address | 19:39 |
| *** tzn has quit IRC | 19:39 | |
| sdake | sbezverk ping me when your ready for that | 19:40 |
| sdake | so d_code is in good shape | 19:40 |
| sdake | who else is suffering :) | 19:40 |
| d_code | :thumbsup: | 19:40 |
| sdake | Serlex your going to try a fresh install then? | 19:41 |
| sdake | and lima is jammed up on the same issue i am | 19:41 |
| sdake | which is blocking development | 19:41 |
| sdake | d_code you just change the docker.service file, removed /var/lib/docker and started fresh? | 19:42 |
| SamYaple | sdake: can you review https://review.openstack.org/#/c/280029/ (note that designate-base is failing for centos-binary, unrelated to this patch) | 19:42 |
| sdake | d_code centos source or centos_binary? | 19:42 |
| sbezverk | sdake any time | 19:42 |
| sbezverk | I removed all containers and tried to deploy again | 19:43 |
| sdake | sbezverk run docker ps -a | fpaste | 19:43 |
| sdake | or paste it | 19:43 |
| sbezverk | and now kolla-toolbox is misbehaving | 19:43 |
| sbezverk | TASK: [common | Starting kolla-toolbox container] ***************************** | 19:44 |
| sbezverk | failed: [192.168.80.25] => {"changed": true, "failed": true} | 19:44 |
| sbezverk | msg: APIError(HTTPError('500 Server Error: Internal Server Error',),) | 19:44 |
| *** limamauricio has joined #kolla | 19:44 | |
| sbezverk | docker ps -a | 19:44 |
| sbezverk | CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES | 19:44 |
| sbezverk | 550d4b451d2c 192.168.80.240:4000/kollaglue/centos-source-kolla-toolbox:2.0.0 "/bin/sleep infinity" About a minute ago Created kolla_toolbox | 19:44 |
| sbezverk | 463373e04640 192.168.80.240:4000/kollaglue/centos-source-rsyslog:2.0.0 "kolla_start" About a minute ago Up About a minute rsyslog | 19:44 |
| d_code | sdake: I did binary | 19:44 |
| d_code | source failed | 19:44 |
| limamauricio | Serlex | 19:44 |
| d_code | w/ the Keystone API issue | 19:44 |
| d_code | sdake: ^^ | 19:44 |
| d_code | sdake: yes, I started fresh with the latest checkout of master (at the time), and the updated service file | 19:45 |
| sdake | d_code interesting | 19:45 |
| sdake | sbezverk please use a paste service, if your on centos you can install fpaste from epel-release | 19:45 |
| sdake | and just pipe it right into fpaste | 19:45 |
| sdake | sbezverk your stuck on the ssame point as i am koolla-toolbox failing to start | 19:46 |
| sdake | notice "Created" but no uptime on the toolbox container | 19:46 |
| sdake | sbezverk we are working on this problem, not sure if its a regression or what but its clearly creating alot of chaos ;) | 19:47 |
| sdake | 5+ people have reported in last couple days | 19:47 |
| sbezverk | http://paste.fedoraproject.org/323169/55657151 | 19:48 |
| sbezverk | sdake interestigly it was working for the first time as you can see from my previous paste | 19:49 |
| sdake | yup | 19:49 |
| sdake | it would probably work again further if you reboot your baremetal | 19:49 |
| sdake | but clearly that is not an acceptable solution long term | 19:50 |
| sbezverk | sdake let me try, but it does not explain mariadb issue right? | 19:50 |
| sdake | mariadb issue is new to me | 19:50 |
| *** blakec has quit IRC | 19:52 | |
| SamYaple | sbezverk: what mariadb issue? | 19:53 |
| sbezverk | SamYaple mariadb was kept restarting | 19:54 |
| SamYaple | sbezverk: you most likely had an unclean environment | 19:55 |
| SamYaple | when redeploying you need to make sure you clean up previous containers and volumes | 19:56 |
| *** Slower is now known as Slow | 19:57 | |
| sdake | it was cratering on something related to pid checking | 19:57 |
| *** Slow is now known as Slower | 19:57 | |
| sdake | slowest | 19:58 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla: Upgrade cinder playbook https://review.openstack.org/278049 | 19:58 |
| sdake | bbiaf need to recharge laptop and care for children (today is day off from school) | 19:59 |
| limamauricio | TASK: [neutron | Starting neutron-dhcp-agent container] ******** msg: APIError(HTTPError(u'500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.22/containers/neutron_dhcp_agent/start',),) | 20:00 |
| limamauricio | Centos, Source, Docker version 1.10.1, build 9e83765, ansible 1.9.4 | 20:00 |
| SamYaple | limamauricio: docker 1.10? MountFalgs=shared? | 20:00 |
| *** vhosakot has quit IRC | 20:01 | |
| limamauricio | docker 1.10 | 20:01 |
| limamauricio | MountFlags i don'i know | 20:01 |
| SamYaple | yea ive got docs coming tonight to explain that | 20:01 |
| limamauricio | MountFlags i don't* know | 20:01 |
| SamYaple | check this commit message for now https://review.openstack.org/#/c/272715/ | 20:01 |
| sbezverk | sdake rebooting baremetal did not help with toolbox | 20:03 |
| *** Serlex has quit IRC | 20:03 | |
| SamYaple | sbezverk: we are working on toolbox issue. unfortunately I do not have access to a box with this issue to test patches for it | 20:04 |
| SamYaple | so im reduced to submitting patches that others then pull and test :/ | 20:04 |
| SamYaple | really slows troubleshooting | 20:04 |
| limamauricio | SamYaple, thanks | 20:04 |
| limamauricio | I add MountFlags=shared :) | 20:04 |
| SamYaple | limamauricio: anytime. docs patch coming tonight | 20:05 |
| limamauricio | :) | 20:05 |
| sbezverk | SamYaple let me know if I can help, I do not mind to test it | 20:05 |
| sbezverk | SamYaple, if you want access my server via web conferencing, that could work too.. | 20:06 |
| *** tzn has joined #kolla | 20:07 | |
| *** Serlex has joined #kolla | 20:09 | |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla: Upgrade cinder playbook https://review.openstack.org/278049 | 20:12 |
| *** mbound has joined #kolla | 20:12 | |
| rhallisey | SamYaple, ^ that should be good. I don't what was up with that rebase.. | 20:12 |
| limamauricio | SamYaple, one doubt, i must install httpd? | 20:13 |
| *** achanda has quit IRC | 20:15 | |
| limamauricio | I've seen somewhere a command that install it, but it does not have seen it in official documentation | 20:15 |
| *** achanda has joined #kolla | 20:16 | |
| *** openstackgerrit has quit IRC | 20:17 | |
| *** openstackgerrit_ is now known as openstackgerrit | 20:17 | |
| *** openstackgerrit has quit IRC | 20:17 | |
| *** openstackgerrit_ has joined #kolla | 20:17 | |
| *** openstackgerrit_ is now known as openstackgerrit | 20:18 | |
| *** openstackgerrit_ has joined #kolla | 20:18 | |
| SamYaple | limamauricio: for what? | 20:20 |
| SamYaple | im not sure where youll need httpd | 20:20 |
| SamYaple | elemoine_: ping | 20:20 |
| limamauricio | I just saw | 20:20 |
| limamauricio | :) | 20:20 |
| limamauricio | sam | 20:20 |
| limamauricio | after add MountFlags=shared i must run systemctl restart docker.service | 20:21 |
| SamYaple | yes | 20:21 |
| limamauricio | ? | 20:21 |
| limamauricio | :) | 20:21 |
| limamauricio | ok | 20:21 |
| limamauricio | this should work right? if you give the same error, I must rebuild it? | 20:22 |
| SamYaple | you should not need to rebuild | 20:22 |
| limamauricio | ok | 20:22 |
| SamYaple | you may need to remove the containers/rekick | 20:22 |
| limamauricio | i take a snapshot without containers | 20:23 |
| limamauricio | :) | 20:23 |
| SamYaple | rhallisey: centos binary is failing on designate-base | 20:23 |
| SamYaple | rhallisey: https://review.openstack.org/278049 | 20:23 |
| SamYaple | that patch is ready but you see gate.. | 20:23 |
| d_code | limamauricio: after you change the mountflags, you have to run `systemctl daemon-reload` first | 20:24 |
| d_code | that makes systemd reparse the service file | 20:24 |
| limamauricio | ok | 20:24 |
| sbezverk | sdake does neutron or any oher opensatck containers expose in any way its .conf files so it can be changed and the container restarted for changes to become active? | 20:24 |
| limamauricio | systemctl daemon-reload and systemctl restart docker.service | 20:25 |
| limamauricio | ? | 20:25 |
| SamYaple | yes limamauricio | 20:25 |
| SamYaple | sbezverk: you are asking if you can make a config change and have the configs take affect? | 20:25 |
| SamYaple | if so we are working on some playbooks for that. our midcycle (last week) covered that topic | 20:26 |
| *** neilus has joined #kolla | 20:26 | |
| rhallisey | SamYaple, I see... | 20:26 |
| sbezverk | SamYaple, yes, example if I add to cinder.conf new backstore or other possible changes | 20:26 |
| SamYaple | sbezverk: as of right now, you can remove teh cinder-api container and rerun the playbooks for the changes to take affect | 20:27 |
| sbezverk | how I can access cinder.conf within a container and would restarting a container do the trick | 20:27 |
| SamYaple | it depends | 20:27 |
| SamYaple | if you have the config set to COPY_ALWAYS you can change it external to the container | 20:27 |
| SamYaple | the default is the container itself will only ever hold one container (immutability) | 20:28 |
| SamYaple | but its configurable to allow it to change its contfig on container restart | 20:28 |
| SamYaple | 20:28:47 < SamYaple> the default is the container itself will only ever hold one container (immutability) | 20:28 |
| SamYaple | will only ever hold one config* | 20:28 |
| sbezverk | SamYaple I see, but how one would access cinder.conf if we taker cinder container as an example? | 20:29 |
| SamYaple | sbezverk: on the cinder-api host, /etc/kolla/cinder-api/cinder.conf | 20:30 |
| SamYaple | with kolla_config_strategy set to COPY_ALWAYS changes you make there will take affect in hte container on contianer restart | 20:30 |
| sdake | sbezverk yup didn't think it would | 20:33 |
| sbezverk | SamYaple, I think I got it. so COPY_ALWAYS is set, then the container check everytime when it restart for possible changes in the configuration, correct? | 20:33 |
| sdake | sbezverk I am starting to suspect changing mountflags requires a complete removal of docker /var/lib/docker | 20:34 |
| sdake | but I don't want to damage my vm so I can actually get to the bottom of the problem | 20:34 |
| sdake | sbezverk d_code did an rm -rf of /var/lib/docker after stopping docker and rebooting and that got his centos binary going | 20:34 |
| d_code | well…I didn’t reboot, but the rest is true | 20:34 |
| SamYaple | sbezverk: correct | 20:35 |
| SamYaple | sdake: MountFlags is not a docker thing | 20:35 |
| SamYaple | its a systemd thing | 20:35 |
| sdake | i see | 20:35 |
| SamYaple | ubuntu 14.04 isn't involved here | 20:35 |
| SamYaple | but, say, 14.10 would need to set that | 20:35 |
| sdake | taht is the delta between my gear and d_code's gear | 20:36 |
| sdake | (the removal of /var/lib/docker) | 20:36 |
| sdake | but i guess it still happens randomly after that point | 20:36 |
| sdake | just got back from break, i'm digging into it now | 20:37 |
| SamYaple | its probably not random, but im going the opposite way | 20:37 |
| SamYaple | im removing the /dev/log binding | 20:37 |
| SamYaple | it was enevitable just annoying | 20:37 |
| sbezverk | sdake just removing /var/lib/docker did not help me, trying wit reboot | 20:39 |
| *** salv-orlando has joined #kolla | 20:42 | |
| SamYaple | sdake: can I jump onto your box to check this snadbox thing? | 20:52 |
| SamYaple | (you said its 100% reproduceable? | 20:52 |
| sdake | its a vm | 20:52 |
| *** vhosakot has joined #kolla | 20:52 | |
| sbezverk | sdake rm and reboot did the trick, so I am back to my flapping mariadb container | 20:52 |
| sdake | i'm not sure how I can make it happen SamYaple | 20:52 |
| SamYaple | so? | 20:52 |
| SamYaple | oh | 20:52 |
| SamYaple | you can port forward | 20:53 |
| SamYaple | bbut i dont feel like walking you through that | 20:53 |
| sdake | from my mac laptop? | 20:53 |
| SamYaple | yea but its ok | 20:53 |
| sdake | i dont need walking through - I c an set it up if you like | 20:53 |
| SamYaple | more trouble than its worth | 20:53 |
| sdake | just rather try this sorting thing first | 20:53 |
| sdake | see if that resolves the problem | 20:53 |
| SamYaple | sdake: its dorting properly | 20:53 |
| SamYaple | sorting* | 20:53 |
| SamYaple | what you are suggesting does not do what you think it does | 20:53 |
| sdake | but sbezverk above rebooted and rm rf | 20:54 |
| SamYaple | rather its ordering correctly, not sorting | 20:54 |
| SamYaple | you are suggesting sorted (which is bad since that reorders) | 20:54 |
| sdake | i was planning to keep everything intact unless it had a dev target | 20:54 |
| SamYaple | i think we are talking about two different things | 20:56 |
| SamYaple | the patch i submitted does what ithink it should do, that doesnt work according to you | 20:56 |
| SamYaple | you should submit one if you have a different vision | 20:56 |
| SamYaple | i cant reproduce | 20:56 |
| SamYaple | or at least not consistently to test | 20:57 |
| sdake | SamYaple if I had one written I would submit ;) | 20:57 |
| sdake | hard to irc and code at same time ;-) | 20:58 |
| vhosakot | Hey all, I'm a dev looking to get involved in kolla, where do I get started ? I have experience with OpenStack, Docker, Ansible, Python, Vagrant, Shell Scripting, etc... I have read about kolla and really like what is does.. Let me know how I can get started! | 20:58 |
| SamYaple | vhosakot: first things first, deploy kolla! | 21:00 |
| sdake | if you can ;) | 21:00 |
| SamYaple | thats a good way to get involved, deploying and figuring out what works/doesnt and submitting patches based on that | 21:00 |
| SamYaple | right now there is a fairly crippling issue for some issues due to docker 1.10 | 21:00 |
| SamYaple | so we are working on that ATM | 21:00 |
| vhosakot | so, Should I get going with an all-in-one kolla dev environment with Vagrant ? http://docs.openstack.org/developer/kolla/vagrant-dev-env.html | 21:01 |
| sdake | people complain alot about vagrant and end up deploying in their own vm of their own creation | 21:01 |
| sdake | i dont know what hte issue is with vagrant since i dont use it | 21:01 |
| sdake | either the docs or implementation need imrprovement or both | 21:01 |
| sdake | that would probably be a good way to get started if you a a vagrant dude :) | 21:02 |
| sdake | we have a million things to do after you get up to speed on how kolla deploys | 21:02 |
| sdake | not really a mllion | 21:02 |
| sdake | less then 1,000 :) | 21:03 |
| SamYaple | vhosakot: dont listen to him. the overhead isn't high | 21:03 |
| sdake | 107 open bugs in bug tracker ;) | 21:03 |
| sdake | hundreds of blueprints | 21:03 |
| sdake | busy busy project :) | 21:03 |
| sdake | wear a seat belt ;) | 21:03 |
| sdake | SamYaple you use vagrant? | 21:03 |
| SamYaple | sdake: i went through it. its not work creating your own vm in my opinion | 21:04 |
| sdake | after d_code submits his host setup patch, I agree, the vm setup shoudl be pretty easy | 21:05 |
| sdake | atm following the quickstart guide seems challenging ;) | 21:05 |
| vhosakot | So, doesn't kolla work out of the box on day 1 for a developer/operator.... I'd like to fix it if the team thinks it needs to be fixed soon :) | 21:06 |
| SamYaple | vhosakot: it does work out of the box, youve come on a bad day | 21:06 |
| sdake | vhosakot unfortunately I think only a super experienced kolla developer can fix this problem | 21:06 |
| SamYaple | this probably popped up last night | 21:06 |
| sdake | but once its fixed you will be golden | 21:06 |
| vhosakot | :) | 21:06 |
| SamYaple | yea its been fun. keysotne broke us, docker broke us, we all get broken! | 21:06 |
| SamYaple | but we fixed the keystoen problem, working through the docker one now | 21:07 |
| sdake | ya dependency management FTW | 21:07 |
| sdake | get the anestaphine from the medlock!! | 21:07 |
| vhosakot | I understand... totally.. dependent code.... ok, so, then, shall I get started with deploying manually then (non-Vagrant way) ? | 21:07 |
| vhosakot | I have a Ubuntu now that I use for DevStack | 21:08 |
| sdake | you can't deploy kolla on the same node as devstack | 21:08 |
| sdake | or i guess you could but you really wouldn't want to | 21:08 |
| sdake | you would damage your devstack, which is painful to fix | 21:08 |
| SamYaple | vhosakot: perfect | 21:08 |
| sdake | you will want a new vm to start with | 21:08 |
| SamYaple | vhosakot: ubuntu works well | 21:08 |
| SamYaple | 14.04 esspecially at this time | 21:09 |
| sdake | ya no systemd ftw | 21:09 |
| vhosakot | cool, I can create new VM.. no prob... I do it daily :) using vagrant :) | 21:09 |
| vhosakot | yes... I'm on 14.04 | 21:09 |
| sdake | vhosakot if you really ar e rockin at vagrant try the vagrant docs | 21:09 |
| SamYaple | vhosakot: yup new vm just follow the deploy guide and it _should_ just work | 21:09 |
| sdake | maybe second | 21:09 |
| sdake | after deploying manually with the quickstart guide | 21:09 |
| sdake | vagrant is meant more for devleopment then eval | 21:10 |
| sdake | quickstart is meant for eval | 21:10 |
| SamYaple | vhosakot: couple of things to remember, need ansible 1.9.4, docker 1.10+, docker-py 1.6+ | 21:10 |
| SamYaple | after that we handle the rest | 21:10 |
| sdake | to build you need a bunch of deps as well | 21:10 |
| vhosakot | SamYaple: deploy guide meaning http://docs.openstack.org/developer/kolla/vagrant-dev-env.html ?? | 21:10 |
| *** blakec has joined #kolla | 21:10 | |
| sdake | install test-requirements.txt and requirements.txt on your host | 21:10 |
| sdake | rather on your vm | 21:10 |
| guillaume_ | failed: [localhost] => {"changed": true, "failed": true} | 21:10 |
| guillaume_ | msg: APIError(HTTPError(u'500 Server Error: Internal Server Error for url: http+docker://localunixsocket/v1.22/containers/kolla_toolbox/start',),) | 21:10 |
| guillaume_ | FATAL: all hosts have already failed -- aborting | 21:10 |
| guillaume_ | Still no solution? | 21:10 |
| sdake | vhosakot those are the docs, there is also a quickstart | 21:10 |
| sdake | guillaume_ yes we know | 21:11 |
| vhosakot | sdake: please point me to the doc to install kolla... I am searching the repo as well... | 21:11 |
| SamYaple | guillaume_: we are working on it. if you _really_ want a solution, use ubuntu 14.04 | 21:11 |
| SamYaple | it wont have this issue | 21:11 |
| SamYaple | otherwise give it a day at most | 21:11 |
| guillaume_ | we alredy use 14.04 and have the problem | 21:11 |
| SamYaple | really? | 21:11 |
| sdake | vhosakot try http://docs.openstack.org/developer/kolla/quickstart.html | 21:11 |
| SamYaple | i cannot reproduce this on ubuntu at all | 21:11 |
| SamYaple | i dont have a machine where I can reproduce this either :/ | 21:12 |
| guillaume_ | root@kolla-guillaume:~# uname -a | 21:12 |
| guillaume_ | Linux kolla-guillaume 3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | 21:12 |
| sdake | 3.13 | 21:12 |
| SamYaple | 3.13, i think we had a reason why that didnt work | 21:12 |
| vhosakot | sdake: thanks! so, is that link about installing kolla manually from scratch ? | 21:12 |
| SamYaple | guillaume_: whats your backend for docker ? `docker info` | 21:12 |
| sdake | vhosakot right - on bare metal or in a vm | 21:12 |
| guillaume_ | root@kolla-guillaume:~# docker info | 21:12 |
| guillaume_ | Containers: 34 | 21:12 |
| guillaume_ | Running: 20 | 21:12 |
| guillaume_ | Paused: 0 | 21:12 |
| guillaume_ | Stopped: 14 | 21:12 |
| guillaume_ | Images: 413 | 21:12 |
| guillaume_ | Server Version: 1.10.1 | 21:13 |
| guillaume_ | Storage Driver: aufs | 21:13 |
| sdake | i'm using overly atm | 21:13 |
| guillaume_ | Root Dir: /var/lib/docker/aufs | 21:13 |
| guillaume_ | Backing Filesystem: extfs | 21:13 |
| guillaume_ | Dirs: 297 | 21:13 |
| sdake | groan don't paste in channnel plz | 21:13 |
| guillaume_ | Dirperm1 Supported: false | 21:13 |
| guillaume_ | Execution Driver: native-0.2 | 21:13 |
| guillaume_ | Logging Driver: json-file | 21:13 |
| sdake | unless its a 1-2 liner | 21:13 |
| guillaume_ | Plugins: | 21:13 |
| guillaume_ | Volume: local | 21:13 |
| guillaume_ | Network: bridge null host | 21:13 |
| vhosakot | sdake: nice! ok, I know what to do now.... thanks! I will let the team know what happens | 21:13 |
| guillaume_ | Kernel Version: 3.13.0-77-generic | 21:13 |
| guillaume_ | Operating System: Ubuntu 14.04.3 LTS | 21:13 |
| guillaume_ | OSType: linux | 21:13 |
| guillaume_ | Architecture: x86_64 | 21:13 |
| SamYaple | sdake: ahhhhh i think i know the delta. im using btrfs. i saw this on aufs | 21:13 |
| guillaume_ | CPUs: 8 | 21:13 |
| SamYaple | its the overlay | 21:13 |
| guillaume_ | Total Memory: 15.67 GiB | 21:13 |
| guillaume_ | Name: kolla-guillaume | 21:13 |
| guillaume_ | ID: WUWT:AWGH:JP4A:XOPF:PVXA:NH4S:WX3L:2LAL:3UFC:ESOS:JLBU:HJPU | 21:13 |
| guillaume_ | WARNING: No swap limit support | 21:13 |
| guillaume_ | sry | 21:13 |
| vhosakot | once I'm up with kolla, I can get started... you guys can let me know what to implement/fix | 21:13 |
| sdake | interesting, so graph driver is the culprit | 21:14 |
| sdake | vhosakot sounds like a plan | 21:14 |
| sdake | this channel runs 24/7 | 21:14 |
| SamYaple | sdake: maybe | 21:14 |
| vhosakot | sdake: cool! | 21:14 |
| sdake | someone will be around to help you ;) | 21:14 |
| vhosakot | I will be here in the channel everyday | 21:14 |
| vhosakot | cool! | 21:14 |
| *** rajathagasthya has joined #kolla | 21:15 | |
| *** blakec has quit IRC | 21:15 | |
| sdake | i guess i could try a different storage backend | 21:15 |
| sdake | d_code what graph driver are you using? | 21:15 |
| sdake | I'd like to actually fix the problem rather then hack around it | 21:15 |
| d_code | novnc? | 21:15 |
| d_code | wait | 21:16 |
| d_code | storage | 21:16 |
| sdake | graph driver is a docker thing | 21:16 |
| d_code | doing the default | 21:16 |
| sdake | on centos? | 21:16 |
| d_code | yup | 21:16 |
| d_code | so…filesystem? | 21:16 |
| sdake | that is lvm fat driver | 21:16 |
| d_code | which, we talked about is a bad idea | 21:16 |
| d_code | but it worked | 21:16 |
| sdake | SamYaple you ought to give overlayfs a spin | 21:16 |
| sdake | might get you a reproducible environment | 21:17 |
| sdake | the thing is d_code everyone else in here with this problem is using the same graph driver as you | 21:17 |
| sdake | SamYaple the graphdriver in the gate is diffferent then what everyone else is using as well for centos | 21:19 |
| sdake | which may explain why centos in the gate works most of th etime | 21:20 |
| SamYaple | btrfs in the gate | 21:20 |
| sdake | yup | 21:20 |
| SamYaple | but if its a race btrfs is _much_ faster than overlayfs when it comes to mounting | 21:20 |
| sdake | it probably doesn't matter what order we specify to api, its probably going to roll in different threads | 21:21 |
| sdake | threading = evil | 21:21 |
| sbezverk | sdake figured out mariadb issue, was misconfig. used the same ip address, so mariadb failed to start the service.. | 21:21 |
| sdake | sbezverk you mean you VIP was your host's IP? | 21:22 |
| sdake | sounds like a good precheck | 21:22 |
| SamYaple | sdake: its in the prechecks | 21:22 |
| sdake | if kolla_internal_address in list_of_nodes - explode | 21:22 |
| sdake | ok folks i am going to focus for a bit on this problem, dropping away from irc | 21:23 |
| sdake | if I seem non-responsive, thats why :) | 21:23 |
| *** cloudnautique has quit IRC | 21:27 | |
| vhosakot | Is the heading "Bare Metal Deployment of Kolla" in http://docs.openstack.org/developer/kolla/quickstart.html.... shouldn't it be "Deployment of kolla on a bare metal node or a virtual machine" ? :) | 21:28 |
| SamYaple | vhosakot: docs patches are always welcome | 21:28 |
| vhosakot | I will find all issues in the quickstart (if any) and combine all of them into a single doc-fix merge | 21:28 |
| vhosakot | ok, cool | 21:28 |
| *** achanda has quit IRC | 21:29 | |
| sdake | SamYaple just struck gold | 21:32 |
| sdake | hang tight log incoming | 21:32 |
| *** aginwala has joined #kolla | 21:34 | |
| sdake | http://paste.fedoraproject.org/323210/14555720/ | 21:34 |
| sdake | first order works | 21:35 |
| sdake | second order doesn't work | 21:35 |
| sdake | repeated several times same results | 21:35 |
| sdake | logs same result about the sandbox container nonsense | 21:35 |
| sdake | that atleast proves its an ordering problem | 21:37 |
| guillaume_ | i changed my kernel to 4.2.0-27 and still have the issue with the kolla-toolbox | 21:37 |
| sdake | the gate flakiness may demonstrate its a race condition | 21:37 |
| sdake | but that is a bit of speculation | 21:38 |
| SamYaple | yea but not an ordering problem i think we can fix unfortunately | 21:38 |
| SamYaple | its not a race | 21:38 |
| SamYaple | thats wrong | 21:38 |
| SamYaple | its a... hashing? problem | 21:38 |
| SamYaple | when a dict is created its keys are hashed | 21:38 |
| openstackgerrit | Shaun Smekel proposed openstack/kolla: Adds support for cleaning up bootstrap containers in the toos/cleanup-containers script. https://review.openstack.org/279961 | 21:38 |
| *** cloudnautique has joined #kolla | 21:38 | |
| *** cloudnautique has joined #kolla | 21:39 | |
| *** cloudnautique has quit IRC | 21:39 | |
| sdake | asalkeld you around | 21:39 |
| openstackgerrit | Shaun Smekel proposed openstack/kolla: Adds support for cleaning up bootstrap containers in the toos/cleanup-containers script. https://review.openstack.org/279961 | 21:39 |
| *** cloudnautique has joined #kolla | 21:40 | |
| sdake | SamYaple join #heat | 21:41 |
| sdake | zaneb is super expert at python | 21:41 |
| sdake | he will e able to give us a concrete answer if there is a solution | 21:41 |
| *** aginwala has quit IRC | 21:46 | |
| *** subscope has quit IRC | 21:48 | |
| *** aginwala has joined #kolla | 21:48 | |
| sdake | ok sam, your patch looks right | 21:50 |
| sdake | but doesn't work | 21:50 |
| *** mdnadeem has quit IRC | 21:50 | |
| sdake | could we try in reverse order see if that changes anything? | 21:50 |
| *** mdnadeem has joined #kolla | 21:50 | |
| *** zaneb has joined #kolla | 21:53 | |
| zaneb | sdake, SamYaple: o/ | 21:53 |
| SamYaple | as i said before, dont worry about it, ive got the long way around patch comming | 21:53 |
| SamYaple | thanks zaneb | 21:53 |
| sdake | hey zaneb | 21:53 |
| sdake | ya thanks zaneb for helping clear that up | 21:54 |
| zaneb | I lurk here so we can avoid spamming the other channel | 21:54 |
| zaneb | s/I/I'll/ | 21:54 |
| SamYaple | sdake: this is broken because of the 1.10 patch i submitted which was required to fix /dev bindmount | 21:54 |
| SamYaple | for whatever reason on my ubuntu it hashes in the right order (out of luck) most of the time | 21:54 |
| SamYaple | ive got a bigger fix coming that ignores this whole reordering thing | 21:55 |
| sdake | i guess the question i have is what is hasing ;) | 21:55 |
| * zaneb is still confused but if y'all have it figured out that's great ;) | 21:55 | |
| SamYaple | sdake: https://wiki.python.org/moin/DictionaryKeys | 21:55 |
| zaneb | sdake: a dict is just a hash table | 21:55 |
| sdake | yes i know that | 21:55 |
| sdake | i did not know it could be ordered | 21:56 |
| sdake | that is what the question was about | 21:56 |
| SamYaple | zaneb: so the issue is we have a list of mounts that need to happen in a certain order, between kolla and docker-py that list is becoming unordered | 21:56 |
| SamYaple | even when using ordered dicts | 21:56 |
| SamYaple | so the fix im doing is to remove the dependacy on those mounts being in a certain order | 21:56 |
| sdake | wfm | 21:57 |
| zaneb | are you sure it's between kolla and docker-py, or between docker-py and docker? | 21:57 |
| sdake | zaneb not sure | 21:57 |
| SamYaple | zaneb: havent dug in, looks like docker-py to docker to be honest | 21:57 |
| sdake | i'd like to find out to get the bug fixed | 21:57 |
| SamYaple | im passing an ordered dict (confirmed that) so it might be docker-py to docker | 21:57 |
| sdake | but master is broken so that takes priority atm | 21:57 |
| *** rajathagasthya has quit IRC | 21:58 | |
| SamYaple | master is broken on some systems | 21:58 |
| SamYaple | not mine for some reason :/ | 21:58 |
| SamYaple | hard to test fies | 21:58 |
| SamYaple | fixes* | 21:58 |
| *** Serlex has quit IRC | 21:58 | |
| sdake | now that I understand the issue, I am happy to test any fix you have SamYaple | 21:58 |
| zaneb | SamYaple: wait, what is self.module.params in kolla_docker.py? is that a dict too? | 21:58 |
| *** macsz has joined #kolla | 21:59 | |
| zaneb | SamYaple: self.module.params['volumes'] I mean | 21:59 |
| *** gtt116_ has joined #kolla | 22:00 | |
| SamYaple | zaneb: list | 22:00 |
| SamYaple | that was my thought earlier too | 22:01 |
| *** alyson_ has quit IRC | 22:02 | |
| SamYaple | zaneb: found it. think its a docker-py bug | 22:02 |
| SamYaple | at this point in the code the volumes will be a list (no matter if passed in as a dict) and this code runs every time | 22:03 |
| SamYaple | https://github.com/docker/docker-py/blob/bd658f7f7fdbefd6fd96907b2800583eaa2db373/docker/utils/utils.py#L841 | 22:03 |
| *** salv-orl_ has joined #kolla | 22:03 | |
| *** macsz has quit IRC | 22:03 | |
| *** gtt116__ has quit IRC | 22:04 | |
| SamYaple | sdake: you willing to test a docker-py code change? | 22:04 |
| sdake | absolutely | 22:05 |
| sdake | fpaste the diff please | 22:05 |
| sdake | or paste | 22:05 |
| sdake | or whatever | 22:05 |
| zaneb | ewww | 22:05 |
| *** salv-orlando has quit IRC | 22:05 | |
| sdake | whats the eww about zaneb ;) | 22:06 |
| zaneb | about https://github.com/docker/docker-py/blob/bd658f7f7fdbefd6fd96907b2800583eaa2db373/docker/utils/utils.py#L841 | 22:06 |
| SamYaple | sdake: make it look like this http://paste.openstack.org/show/487043/ | 22:06 |
| SamYaple | im not going to diff it because i didnt build a patch | 22:06 |
| SamYaple | ill be forking docker-py in a moment though | 22:06 |
| sdake | will do | 22:06 |
| sdake | bbiaf | 22:06 |
| SamYaple | you can test my fork i suppose | 22:06 |
| sdake | i'll ust hack my env up - its no big deal easy to fix | 22:06 |
| SamYaple | zaneb: at that point the volumes stuff could have looked like this, string->list->dict->string->list->dict | 22:07 |
| SamYaple | zaneb: how awesome is that! | 22:07 |
| SamYaple | before it was ever used! | 22:07 |
| zaneb | lol | 22:07 |
| zaneb | SamYaple: that code only runs on a list, so you could pass an OrderedDict to it and not have to fork docker-py | 22:08 |
| zaneb | if you have control over the code that calls that, at least | 22:09 |
| SamYaple | zaneb: it gets converted to a list before then :( | 22:09 |
| SamYaple | in docker-py | 22:09 |
| SamYaple | zaneb: it gets converted to list here https://github.com/docker/docker-py/blob/bd658f7f7fdbefd6fd96907b2800583eaa2db373/docker/utils/utils.py#L295 | 22:09 |
| SamYaple | then returned _back_ to kolla_docker, then its expected to be either a string or list | 22:10 |
| SamYaple | i suppose I could override it... but then i lose the ability for docker-py to validate/change my dict to work with docker | 22:10 |
| *** banix has quit IRC | 22:10 | |
| d_code | so, while my clean build is running… tunnel_interface is globals.yml is for the vxvlan traffic between compute/controller hosts, right? | 22:12 |
| SamYaple | d_code: correct, it defaults to network_interface variable | 22:12 |
| d_code | api_interface is where the actual web ui and API endpoints for running openstack cli cients, right? | 22:12 |
| SamYaple | yes, and a few other things like mysql traffic | 22:13 |
| d_code | and storage, is obviously for storage…in the default install, is that ceph? | 22:13 |
| sdake | zaneb thoughts on SamYaple's analysis? | 22:13 |
| SamYaple | it should be ceph, but i fear that variable is entirely unused atm | 22:13 |
| SamYaple | d_code: that code is untested by most, but thats the idea behind the variables (ill be gating them at some point) | 22:14 |
| d_code | so, it would go over api_interface, probably? | 22:14 |
| SamYaple | probably | 22:14 |
| *** neophy has quit IRC | 22:14 | |
| zaneb | SamYaple: host_config['Binds'] = OrderedDict((k, dict()) for k in host_config['Binds'])? | 22:14 |
| SamYaple | zaneb: yes that would work sure, but my point was docker-py validates that stuff | 22:14 |
| SamYaple | it has all the code about deprecation and what not | 22:14 |
| SamYaple | we would have to give it up | 22:14 |
| SamYaple | also stuff like https://github.com/docker/docker-py/blob/bd658f7f7fdbefd6fd96907b2800583eaa2db373/docker/utils/utils.py#L295 zaneb | 22:14 |
| zaneb | ah, ok. I am missing the big picture of the data flow here so just spitballing | 22:15 |
| SamYaple | basically docker-py sanatizes it into a list. while I _could_ pass a dict, I would have to sanatize it in kolla_docker | 22:15 |
| SamYaple | but yes, technically possible | 22:15 |
| openstackgerrit | Sam Yaple proposed openstack/kolla: [WIP] Move socket binding to named_volume https://review.openstack.org/280429 | 22:16 |
| SamYaple | sdake: theres a big patch to hopefully sidestep this whole issue | 22:16 |
| sdake | be cool if we can fix in docker-py for other folks too :) | 22:17 |
| sdake | i'll get to teting your patch in a moment | 22:17 |
| sdake | ever seen south park? | 22:17 |
| sdake | they are coming right for me! | 22:17 |
| vhosakot | I need the VM to have two interfaces, right ? eth0 (network_interface) and eth1 (neutron_external_interface) /etc/kolla/globals.yml | 22:18 |
| sdake | so I think a plan of action is to fix this in docker-py so we aren't afflicted by it in the future, possibly workaround in whatever way is most sanitary, and get a patch merged asap :) | 22:18 |
| SamYaple | vhosakot: yes. it is technically possible to do kolla with one interface but it is complicated and not-documented | 22:19 |
| sdake | vhosakot yes 2 ips ;) | 22:19 |
| SamYaple | sdake: you confirm it works and ill submit a PR for docker-py | 22:19 |
| sdake | will do sam, working on getting my env going | 22:19 |
| vhosakot | SamYaple: ok, two interfaces, two IPs, two networks | 22:19 |
| sdake | no ip on the second interface | 22:19 |
| vhosakot | ok | 22:19 |
| vhosakot | what tool do you guys use ? VirtualBox ? I use virt-manager and it does not allow me to add more than one interface per guest VM | 22:20 |
| sdake | atleast we are to the root cause of the issue after two days :) | 22:20 |
| sdake | i use bare metal when its available, in remodel atm so using virutalbox | 22:20 |
| sdake | vhosakot wwith virt-manager you ahve to creat a second network | 22:21 |
| vhosakot | I dont have a bare-metal server | 22:21 |
| SamYaple | vhosakot: virt-manager you can hot add a second interface | 22:21 |
| sdake | that is a little complicated and undocumented :) | 22:21 |
| vhosakot | sdake: ah, second network in virt-manager | 22:21 |
| vhosakot | thought so.. ok, thanks! | 22:21 |
| sdake | sam have your pate open | 22:22 |
| sdake | wher edoes line 1 go | 22:22 |
| SamYaple | i dont know what version of docker you have so the line numbers change | 22:23 |
| sdake | which file/function | 22:23 |
| SamYaple | but its in docker/utils/utils.py | 22:23 |
| sdake | thanks | 22:23 |
| *** bmace has quit IRC | 22:23 | |
| *** bmace has joined #kolla | 22:23 | |
| SamYaple | the function is create_container_config() I *think* | 22:24 |
| sdake | got it | 22:24 |
| sdake | i'll give ita go | 22:24 |
| vhosakot | sdake: I see I can create a second virtual network, but, when creating the VM, I can attach it to one one network... Shouldn't I attach the VM to both the networks.. | 22:24 |
| vhosakot | in virt-manager | 22:24 |
| *** mbound has quit IRC | 22:26 | |
| *** blahRus1 has quit IRC | 22:26 | |
| *** dwalsh has quit IRC | 22:26 | |
| *** JoseMello has quit IRC | 22:28 | |
| *** achanda has joined #kolla | 22:30 | |
| sdake | cd SamYaple that 2 liner failed | 22:30 |
| sdake | same results | 22:30 |
| sdake | looked good though | 22:30 |
| SamYaple | hmm. i wish I could be in the environemnt | 22:31 |
| sdake | i'll see if I can figure that out | 22:31 |
| SamYaple | sdake: did you use the docker-py with the kolla patch i submited earlier? | 22:31 |
| sdake | just keep sorting out your workaround | 22:31 |
| sdake | which one? | 22:31 |
| sdake | oh no | 22:31 |
| sdake | duh | 22:31 |
| sdake | let me do that | 22:31 |
| sdake | same result | 22:32 |
| sdake | i can either debug this or setup my env for you to get in | 22:33 |
| SamYaple | well idk. i need to pdb at this point | 22:33 |
| sdake | which would you prefer | 22:33 |
| SamYaple | if you setup your environment ill log in | 22:33 |
| SamYaple | but i cant give anymore pointers | 22:33 |
| sdake | anymore pointers? | 22:34 |
| SamYaple | as to what the issue might be or possible fixes | 22:34 |
| sdake | you mean about environnmental setup | 22:34 |
| SamYaple | about the issue at hand | 22:34 |
| SamYaple | this patch should make it better though https://review.openstack.org/280429 | 22:34 |
| SamYaple | avoids the whole issue | 22:34 |
| sdake | right | 22:35 |
| SamYaple | if you _still_ hit the issue, this is a different problem | 22:35 |
| sdake | let me drop off the vpn so i can get you access if I can figure out how to get my vm an ip address ;) | 22:35 |
| SamYaple | it might be worth it to give it a spin | 22:35 |
| sdake | ok i'll try it now moment | 22:35 |
| SamYaple | youll need to rebuild | 22:36 |
| sdake | ya that works | 22:37 |
| sdake | deploys atleast | 22:38 |
| SamYaple | you rebuild already? | 22:38 |
| sdake | let me rebuild didn't notice the docker changes | 22:38 |
| SamYaple | well yea | 22:38 |
| SamYaple | i mean that will _work_ but logging is broken until you rebuild | 22:38 |
| sdake | ya i see that | 22:38 |
| SamYaple | this is much cleaner anyway | 22:38 |
| SamYaple | name_volumes to share teh socket | 22:39 |
| sdake | ya i liek your change | 22:39 |
| sdake | have you tested it? | 22:39 |
| SamYaple | yea | 22:39 |
| sdake | works for you? | 22:39 |
| SamYaple | full deploy with logging | 22:39 |
| SamYaple | yea | 22:39 |
| sdake | ok | 22:39 |
| sdake | building - t-45 minutes out | 22:39 |
| SamYaple | its WIP because no bug/bp | 22:39 |
| sdake | right | 22:39 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Make Mesos resources configurable https://review.openstack.org/280144 | 22:42 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Add neutron config https://review.openstack.org/265109 | 22:42 |
| *** athomas has quit IRC | 22:44 | |
| sdake | ok SamYaple need to eat - bbiaf afte rbuild done with results | 22:47 |
| *** tzn has quit IRC | 22:52 | |
| *** aginwala has quit IRC | 22:56 | |
| *** haplo37 has quit IRC | 22:58 | |
| *** aginwala has joined #kolla | 22:58 | |
| sdake | named volumes are pretty sweet tech | 22:59 |
| asalkeld | nihilifer: https://review.openstack.org/#/c/280144/2 | 23:02 |
| asalkeld | mem/cpu are supported | 23:02 |
| vhosakot | sdake: ok, I created a VM with two network interfaces (eth0 and eth1) using virt-manager... I expected virt-manager allows me to attach two networks when creating teh VM... it does not.. so, I created the VM with one interface, added the second interface, and rebooted the VM.. | 23:03 |
| sdake | that should do it | 23:03 |
| vhosakot | sdake: may be I'm used to how OpenStack horizon allows the user add a VM to multiple networks when creating the VM.. :) | 23:04 |
| vhosakot | sdake: cool | 23:04 |
| sdake | ymmv with libvirt and two networks and actually getting neutron to work | 23:04 |
| sdake | I never operate kolla that way | 23:04 |
| sdake | I always operate on bare metal | 23:04 |
| sdake | been doing vms for about a month since my house is disassembled | 23:05 |
| sdake | its painful | 23:05 |
| sdake | i'm almsot to the point of moving my lab into my bedroom | 23:05 |
| vhosakot | so, on the bare metal, you have two physical NICs or virtual NICs ? | 23:05 |
| sdake | except my 1.2tb pcie ssd is busted and needs to be shipped to intel for repair | 23:05 |
| vhosakot | wow | 23:05 |
| sdake | i have two 10gige nics | 23:05 |
| vhosakot | nice | 23:05 |
| nihilifer | asalkeld: ok, i see | 23:12 |
| d_code | sdake: so, if I want compute services on two hosts, do I need to change the mariadb role to just be one? I tried to deploy with two control hosts and it died on mariadb | 23:12 |
| sdake | ha requires 3 hostss | 23:13 |
| d_code | ah | 23:14 |
| d_code | that makes sense | 23:14 |
| sdake | although it can be dpeloyed with 2 | 23:14 |
| sdake | pretty sure SamYaple deploys with 2 | 23:14 |
| sdake | but tbh I only test with 3 | 23:14 |
| d_code | well…I just fell back to 1 control host for now, 2 compute hosts | 23:14 |
| sdake | if you paste your inventory file i could take a look | 23:14 |
| sdake | it could be a bug in master | 23:14 |
| sdake | only a couple people in the community have gear to test ha | 23:15 |
| d_code | k. holding out fo the deploy to complete | 23:15 |
| sdake | its my default setup | 23:15 |
| sdake | the mariadb role shoudl not be on a compute host though | 23:15 |
| sdake | rereading your question, I'm not sure i parsed it correctly the first time | 23:15 |
| sdake | been a long long day | 23:15 |
| d_code | I’ve got….a unique deployment scenario | 23:15 |
| sdake | you can run control and compute on two nodes | 23:16 |
| sdake | and sort of get ha ;) | 23:16 |
| dave-mccowan | i'm following the quick start guide and i'm at the tools/build.py step. some of my containers are failing to build. the doc just says keep trying... any other advice? looks like the centos httpd package is related to the root cause. | 23:16 |
| sdake | dave-mccowan httpd should pull in fine | 23:16 |
| d_code | sdake: http://ur1.ca/oiwym | 23:16 |
| sdake | dave-mccowan what container ? | 23:16 |
| d_code | I had ddi001 and ddi002 on all roles but storage | 23:17 |
| dave-mccowan | several, including keystone and horizon | 23:17 |
| d_code | and maria failed | 23:17 |
| sdake | d_code that should absolutely work | 23:17 |
| sdake | dave-mccowan it is possible the mirrors are busted | 23:18 |
| * d_code shrugs | 23:18 | |
| d_code | I’ll circle back to it | 23:18 |
| sdake | what I would recommend is to rebuild your base container and keystone and horizon | 23:18 |
| sdake | buidl.py keystone horizon will do the job | 23:18 |
| sdake | rather | 23:18 |
| sdake | build.py --no-cache keystone horizon | 23:18 |
| sdake | sometimes the upstream repos change and files become unavailable from the repo that is installed in teh base image | 23:19 |
| sdake | rebuilding the base image fixes this problem | 23:19 |
| sdake | since the mirrors are propogating over the internets all the time | 23:19 |
| *** jtriley has joined #kolla | 23:19 | |
| sdake | and basically your mirror has either been deleted or is repropogating | 23:19 |
| sdake | dave-mccowan does that make sense? | 23:20 |
| d_code | hmmm…. what provides the neutron command? | 23:20 |
| d_code | b/c….that’s aint workin | 23:20 |
| d_code | python-openstack? | 23:20 |
| dave-mccowan | yep. i'll do that. if it still fails, i should wait and try agin? maybe the mirrors will be less busy later tonight. | 23:21 |
| d_code | client | 23:21 |
| sdake | python-neutronclient | 23:21 |
| sdake | python-openstackclient may ahve equivalents | 23:21 |
| sdake | it is in rapid development | 23:21 |
| sdake | i use the old school clients because old dog new tricks cant teach and all that ;) | 23:22 |
| sdake | d_code ^^ | 23:22 |
| d_code | thanks | 23:22 |
| dave-mccowan | sdake i'm not clear on the resolution of the 2 VIP direction. can you give you your view of next steps on that? | 23:22 |
| d_code | weird…they were there when I did the allinone…but not now | 23:22 |
| sdake | dave-mccowan SamYaple agreed that 2 VIPs managed by HAproxy is the way to go from my parsing | 23:23 |
| sdake | and apparentl ythis was discussed at the midcycle | 23:23 |
| sdake | i only type 110 WPM :) | 23:24 |
| *** vhosakot has quit IRC | 23:24 | |
| *** achanda has quit IRC | 23:25 | |
| dave-mccowan | sdake seems reasonable. tls changes will be easier once that is done. | 23:25 |
| d_code | sdake: yeah, so iptables is totally in the way :p | 23:27 |
| d_code | or did I screw myself and kolla attempts to configure firewalld? | 23:27 |
| sdake | kolla doesn't configure firewalld | 23:28 |
| sdake | if you have host ip tables, I generally turn my rules off | 23:29 |
| sdake | we probably need a document that describes how to setu pa host | 23:29 |
| sdake | i wanted this this cycle | 23:29 |
| sdake | but it hasn't happened yet | 23:29 |
| d_code | k. I’ll have to automate that for my deployment | 23:29 |
| *** yuanying_flu is now known as yuanying | 23:29 | |
| d_code | sdake: the default init networks | 23:31 |
| d_code | is the idea the vm gets an IP from “demo-net” (10.0.0.0/24) | 23:31 |
| *** neilus has quit IRC | 23:31 | |
| d_code | and 10.0.2.0/24 are “floating IPs”? | 23:31 |
| d_code | which would be available on the nova_external interface? | 23:32 |
| sdake | d_code yes | 23:32 |
| d_code | cool | 23:32 |
| d_code | thanks | 23:32 |
| sdake | demo-net is a private network | 23:33 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Make Mesos resources configurable https://review.openstack.org/280144 | 23:33 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Add neutron config https://review.openstack.org/265109 | 23:33 |
| sdake | private as in neutron private | 23:33 |
| *** achanda has joined #kolla | 23:33 | |
| d_code | sdake: https://gist.github.com/dcode/aea450c1648034067637 | 23:35 |
| d_code | that’s what finally worked for me | 23:35 |
| d_code | I’ll get around to submitting it as you asked | 23:35 |
| openstackgerrit | Angus Salkeld proposed openstack/kolla-mesos: Update docs on adding a new service https://review.openstack.org/280444 | 23:35 |
| asalkeld | nihilifer: ^ | 23:35 |
| d_code | I’m out…time to make dinner | 23:36 |
| sdake | d_code I'd remove line 106+ | 23:36 |
| sdake | but the rest looks beutiful | 23:36 |
| sdake | I shoudl have done that long ago | 23:36 |
| *** Jeffrey4l has joined #kolla | 23:39 | |
| d_code | :) | 23:39 |
| sdake | dave-mccowan fwiw I just rebuilt and onl ythe containers that shouldn't build dont on centos binary | 23:39 |
| sdake | these include designate, mistral, gnocchi, and murano | 23:40 |
| sdake | these don't build because there is no upstream RPM for them | 23:40 |
| sdake | notice there is one for magnum ;-) | 23:40 |
| *** dims_ has quit IRC | 23:41 | |
| asalkeld | sorry, nihilifer getting a bit picky :-( | 23:42 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla: Ensure that /var/lib/zookeeper is owned by user https://review.openstack.org/280445 | 23:42 |
| dave-mccowan | sdake those failed for me too. but also keystone and horizon? | 23:43 |
| sdake | keystone should work and horizon should work | 23:44 |
| sdake | did you tyr the no-cache thing? | 23:44 |
| *** banix has joined #kolla | 23:45 | |
| sdake | ./build.py --no-cache keystone horizon | 23:45 |
| sdake | that will buid only keystone and horizon | 23:45 |
| sdake | (and its dependencies) | 23:45 |
| *** jtriley has quit IRC | 23:46 | |
| nihilifer | asalkeld: good idea with conf profiles | 23:47 |
| dave-mccowan | yep, that failed. httpd.x86_64 0:2.4.6-40.el7.centos is the reported failing package. | 23:48 |
| sdake | SamYaple your patch is deploying atm, i'll let you know | 23:48 |
| sdake | dave-mccowan is there any more data available? | 23:48 |
| sdake | if you run script | 23:48 |
| sdake | you can run the build | 23:49 |
| sdake | type eixt | 23:49 |
| sdake | a file typesciript will be created | 23:49 |
| sdake | samyaple http://paste.fedoraproject.org/323244/80315145 | 23:52 |
| *** jtriley has joined #kolla | 23:52 | |
| sdake | need more anestaphine from the medlock!! | 23:52 |
| *** tzn has joined #kolla | 23:52 | |
| dave-mccowan | sdake http://paste.openstack.org/show/487051/ | 23:52 |
| *** banix has quit IRC | 23:55 | |
| SamYaple | sdake: so it deployed, but broke with mariadb? | 23:55 |
| sdake | kolla-toolbox looks like it is missing jinja2 | 23:56 |
| sdake | dave-mccowan fwiw my keystone container has that httpd in it | 23:56 |
| sdake | from your base image, remove line cd | 23:56 |
| SamYaple | sdake: _binary_ _centos_ kolla-toolbox is busted for you. thats important | 23:56 |
| *** dims has joined #kolla | 23:57 | |
| SamYaple | its a big difference in that container | 23:57 |
| sdake | it works for d_code | 23:57 |
| asalkeld | this is irritating bug 1544287 | 23:57 |
| openstack | bug 1544287 in kolla-mesos "unable to launch containers, Unexpected Env format for 'ContainerConfig.Env'" [Critical,Confirmed] https://launchpad.net/bugs/1544287 | 23:57 |
| *** tzn has quit IRC | 23:57 | |
| SamYaple | asalkeld: what version of docker-py? | 23:58 |
| asalkeld | mesos is using docker inspect to validate ENV | 23:58 |
| sdake | dave-mccowan throw in RUN rpm -e yum-plugin-fastestmirrors | 23:58 |
| asalkeld | the issue seems to be a newer version of mesos | 23:58 |
| sdake | dave-mccowan in the keystone container | 23:58 |
| sdake | then build.py keystone | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!