| openstackgerrit | Allen Gao proposed openstack/kolla: Move the install base type check to kolla-build https://review.openstack.org/274629 | 00:03 |
|---|---|---|
| *** Allen_Gao has joined #kolla | 00:05 | |
| *** salv-orl_ has quit IRC | 00:08 | |
| *** dims has quit IRC | 00:10 | |
| *** dave-mcc_ has joined #kolla | 00:27 | |
| *** dave-mccowan has quit IRC | 00:28 | |
| *** ajafo has quit IRC | 00:32 | |
| *** ajafo has joined #kolla | 00:32 | |
| *** absubram has quit IRC | 00:41 | |
| *** unicell has joined #kolla | 00:52 | |
| *** jasonsb has joined #kolla | 00:53 | |
| *** unicell1 has joined #kolla | 01:02 | |
| *** unicell has quit IRC | 01:04 | |
| *** kjelly has joined #kolla | 01:11 | |
| *** diogogmt has joined #kolla | 01:18 | |
| *** daneyon_ has quit IRC | 01:36 | |
| *** daneyon has joined #kolla | 01:36 | |
| *** kjelly has quit IRC | 01:39 | |
| openstackgerrit | Allen Gao proposed openstack/kolla: Move the install base type check to kolla-build https://review.openstack.org/274629 | 01:39 |
| *** ajafo has quit IRC | 01:40 | |
| *** ajafo has joined #kolla | 01:40 | |
| *** ssurana has quit IRC | 01:41 | |
| *** weiyu has joined #kolla | 01:49 | |
| *** tfukushima has joined #kolla | 01:54 | |
| *** Jeffrey4l has joined #kolla | 02:03 | |
| *** dave-mcc_ has quit IRC | 02:09 | |
| *** dave-mccowan has joined #kolla | 02:10 | |
| *** dims has joined #kolla | 02:11 | |
| *** Marga_ has quit IRC | 02:20 | |
| *** Marga_ has joined #kolla | 02:22 | |
| *** coolsvap|away is now known as coolsvap | 02:26 | |
| *** Marga_ has quit IRC | 02:27 | |
| *** kjelly has joined #kolla | 02:29 | |
| openstackgerrit | Allen Gao proposed openstack/kolla: Move the install base type check to kolla-build https://review.openstack.org/274629 | 02:29 |
| *** kejlly_ has joined #kolla | 02:32 | |
| *** kjelly has quit IRC | 02:32 | |
| *** kejlly_ is now known as kjelly | 02:32 | |
| *** ssurana has joined #kolla | 02:46 | |
| *** ssurana has quit IRC | 02:48 | |
| *** chandankumar has joined #kolla | 02:50 | |
| *** thumpba has quit IRC | 02:51 | |
| *** coolsvap is now known as coolsvap|away | 02:59 | |
| *** chandankumar has quit IRC | 03:04 | |
| *** kjelly has quit IRC | 03:14 | |
| *** tfukushima has quit IRC | 03:14 | |
| *** tfukushima has joined #kolla | 03:16 | |
| *** chandankumar has joined #kolla | 03:19 | |
| *** yuanying_ has joined #kolla | 03:21 | |
| *** yuanying has quit IRC | 03:24 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Playbook for horizon upgrade https://review.openstack.org/274827 | 03:29 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Playbook for horizon upgrade https://review.openstack.org/274827 | 03:30 |
| *** unicell1 has quit IRC | 03:31 | |
| *** chandankumar has quit IRC | 03:32 | |
| *** ssurana has joined #kolla | 03:34 | |
| *** sdake has joined #kolla | 03:35 | |
| *** yuanying has joined #kolla | 03:41 | |
| *** Marga_ has joined #kolla | 03:42 | |
| *** achanda has joined #kolla | 03:43 | |
| *** Marga_ has joined #kolla | 03:43 | |
| *** yuanying_ has quit IRC | 03:44 | |
| *** Marga_ has quit IRC | 03:45 | |
| *** Marga_ has joined #kolla | 03:46 | |
| *** chandankumar has joined #kolla | 03:48 | |
| *** dims has quit IRC | 03:53 | |
| *** ssurana has quit IRC | 03:53 | |
| *** klint has joined #kolla | 03:54 | |
| *** CheKoLyN has joined #kolla | 03:55 | |
| *** achanda has quit IRC | 04:04 | |
| *** yuanying has quit IRC | 04:05 | |
| *** yuanying has joined #kolla | 04:06 | |
| *** coolsvap|away is now known as coolsvap | 04:07 | |
| *** yuanying_ has joined #kolla | 04:07 | |
| *** yuanying has quit IRC | 04:07 | |
| *** achanda has joined #kolla | 04:11 | |
| *** chandankumar has quit IRC | 04:12 | |
| *** Marga_ has quit IRC | 04:25 | |
| *** Marga_ has joined #kolla | 04:25 | |
| *** chandankumar has joined #kolla | 04:26 | |
| *** chandankumar has quit IRC | 04:37 | |
| *** weiyu has quit IRC | 04:43 | |
| *** tfukushima has quit IRC | 04:43 | |
| *** sdake has quit IRC | 05:05 | |
| *** unicell has joined #kolla | 05:07 | |
| *** dave-mccowan has quit IRC | 05:10 | |
| *** weiyu has joined #kolla | 05:16 | |
| *** chandankumar has joined #kolla | 05:22 | |
| *** chandankumar has quit IRC | 05:25 | |
| *** tfukushima has joined #kolla | 05:34 | |
| *** weiyu has quit IRC | 05:41 | |
| *** chandankumar has joined #kolla | 05:42 | |
| *** dratushnyy has joined #kolla | 05:43 | |
| dratushnyy | Hi guys | 05:44 |
| *** Guest51478 has quit IRC | 05:48 | |
| *** chandankumar has quit IRC | 05:49 | |
| *** kjelly has joined #kolla | 05:50 | |
| *** weiyu has joined #kolla | 05:52 | |
| *** CheKoLyN has quit IRC | 06:00 | |
| *** Marga_ has quit IRC | 06:00 | |
| *** dwalsh has joined #kolla | 06:11 | |
| *** Allen_Gao has quit IRC | 06:12 | |
| *** chandankumar has joined #kolla | 06:22 | |
| *** chandankumar has quit IRC | 06:23 | |
| *** chandankumar has joined #kolla | 06:23 | |
| *** ajafo has quit IRC | 06:24 | |
| *** ajafo has joined #kolla | 06:25 | |
| *** Allen_Gao has joined #kolla | 06:28 | |
| *** Marga_ has joined #kolla | 06:30 | |
| openstackgerrit | Allen Gao proposed openstack/kolla: Playbook for upgrade memcached https://review.openstack.org/275035 | 06:54 |
| openstackgerrit | Allen Gao proposed openstack/kolla: Playbook for rabbitmq upgrade https://review.openstack.org/275036 | 06:54 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Upgrade the ansible version to 2 https://review.openstack.org/275037 | 06:54 |
| openstackgerrit | Allen Gao proposed openstack/kolla: Playbook for upgrade murano https://review.openstack.org/275038 | 07:04 |
| *** dwalsh has quit IRC | 07:05 | |
| *** Allen_Gao has quit IRC | 07:13 | |
| *** coolsvap is now known as coolsvap|away | 07:16 | |
| *** Allen_Gao has joined #kolla | 07:31 | |
| *** ppetit has joined #kolla | 07:31 | |
| *** achanda has quit IRC | 07:40 | |
| *** achanda has joined #kolla | 07:43 | |
| *** tfukushima has quit IRC | 07:47 | |
| *** tfukushima has joined #kolla | 07:49 | |
| *** achanda has quit IRC | 07:55 | |
| *** kexiaodong has joined #kolla | 07:59 | |
| *** ppetit has quit IRC | 08:09 | |
| *** sdake has joined #kolla | 08:09 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Upgrade the ansible version to 2 https://review.openstack.org/275037 | 08:12 |
| *** fgimenez has joined #kolla | 08:13 | |
| *** akwasnie has joined #kolla | 08:14 | |
| kjelly | Hi guys, it seems that kolla-build script never read `/etc/kolla/kolla-build.conf`. Does anyone have the same issue? | 08:15 |
| *** chandankumar has left #kolla | 08:17 | |
| *** Marga_ has quit IRC | 08:24 | |
| *** dwalsh has joined #kolla | 08:25 | |
| *** tobe has joined #kolla | 08:28 | |
| *** kproskurin has joined #kolla | 08:29 | |
| *** athomas has joined #kolla | 08:31 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Validate that all the containers are started https://review.openstack.org/275067 | 08:33 |
| *** akwasnie has quit IRC | 08:41 | |
| *** ajafo has quit IRC | 08:48 | |
| *** ajafo has joined #kolla | 08:48 | |
| *** dwalsh has quit IRC | 08:50 | |
| *** akwasnie has joined #kolla | 08:53 | |
| *** weiyu has quit IRC | 08:55 | |
| *** dwalsh has joined #kolla | 08:58 | |
| *** tfukushima has quit IRC | 08:58 | |
| *** tfukushima has joined #kolla | 08:58 | |
| *** weiyu has joined #kolla | 09:02 | |
| *** mbound has joined #kolla | 09:02 | |
| *** tfukushima has quit IRC | 09:03 | |
| *** tobe has quit IRC | 09:16 | |
| *** dratushnyy has quit IRC | 09:24 | |
| *** achanda has joined #kolla | 09:25 | |
| *** achanda has quit IRC | 09:30 | |
| *** rmart04 has joined #kolla | 09:34 | |
| *** phong_ has joined #kolla | 09:35 | |
| *** ppetit has joined #kolla | 09:50 | |
| *** mbound has quit IRC | 09:52 | |
| *** mbound has joined #kolla | 09:52 | |
| *** weiyu has quit IRC | 09:53 | |
| *** rhallisey has joined #kolla | 09:53 | |
| *** weiyu has joined #kolla | 09:54 | |
| *** mbound_ has joined #kolla | 09:55 | |
| *** mbound has quit IRC | 09:55 | |
| phong_ | I am trying to deploy, developing multi-node Kolla on an OpenStack cloud environment that supports Heat, is there any step-by-step guide? I found it is hard to image deployment model | 10:04 |
| phong_ | thanks | 10:04 |
| *** kproskurin has quit IRC | 10:09 | |
| ccesario | hello.... following this page... http://docs.openstack.org/developer/kolla/deploy-all-in-one-node.html ... when I try deploy I'm getting this error | 10:09 |
| ccesario | eploying Playbook : ansible-playbook -i ansible/inventory/all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml ansible/site.yml | 10:09 |
| ccesario | ERROR: merge_configs is not a legal parameter in an Ansible task or handler | 10:09 |
| *** dwalsh has quit IRC | 10:09 | |
| ccesario | does someone have any idea about fix it ? | 10:10 |
| Allen_Gao | ccesario: what your ansible version? | 10:12 |
| *** ajafo has quit IRC | 10:14 | |
| *** openstackgerrit has quit IRC | 10:17 | |
| *** openstackgerrit has joined #kolla | 10:17 | |
| *** akwasnie has quit IRC | 10:18 | |
| pbourke | ccesario: please check your software versions against the table on this page http://docs.openstack.org/developer/kolla/quickstart.html | 10:18 |
| ccesario | let me check... | 10:18 |
| *** akwasnie has joined #kolla | 10:18 | |
| *** akwasnie1 has joined #kolla | 10:20 | |
| *** akwasnie has quit IRC | 10:20 | |
| *** akwasnie1 has quit IRC | 10:22 | |
| openstackgerrit | Antoine Roux proposed openstack/kolla: Fix wrong behavior when the source is a directory https://review.openstack.org/275104 | 10:24 |
| *** macsz has left #kolla | 10:26 | |
| *** macsz has joined #kolla | 10:27 | |
| *** akwasnie has joined #kolla | 10:31 | |
| *** ajafo has joined #kolla | 10:31 | |
| ccesario | well... ansible is 1.9.2 | 10:32 |
| ccesario | ansible --version | 10:32 |
| ccesario | ansible 1.9.2 | 10:32 |
| pbourke | Jeffrey4l: what functionally are you changing here? https://review.openstack.org/#/c/275067/ | 10:33 |
| pbourke | ccesario: min is 1.9.4 | 10:34 |
| ccesario | does the git trunk branch is is compatible with ubuntu-14.04.3 and liberty version!? | 10:34 |
| ccesario | pbourke, yes... I ḿ upgrading it | 10:34 |
| openstackgerrit | weiyu proposed openstack/kolla: fix miss rabbitmq config when hostname include special character https://review.openstack.org/275111 | 10:44 |
| *** akwasnie has quit IRC | 10:45 | |
| *** kproskurin has joined #kolla | 10:49 | |
| *** akwasnie has joined #kolla | 10:53 | |
| *** weiyu has quit IRC | 10:57 | |
| *** ppetit has quit IRC | 10:58 | |
| *** dave-mccowan has joined #kolla | 10:59 | |
| *** dims_ has joined #kolla | 11:02 | |
| Jeffrey4l | pbourke, old logical is: if the ansible deploy success, then it is succe. new logical is: not only the ansible deploy success, and the docker containers are all up status(not exited, not created status) | 11:08 |
| *** fthiagogv has quit IRC | 11:09 | |
| Jeffrey4l | pbourke, check this log http://logs.openstack.org/38/275038/1/check/gate-kolla-dsvm-deploy-centos-source/caa9df5/console.html#_2016-02-02_07_36_07_184 | 11:10 |
| Jeffrey4l | neutron_openvswitch_agent container exited. But the test is success. | 11:10 |
| *** ajafo has quit IRC | 11:12 | |
| *** ajafo has joined #kolla | 11:13 | |
| *** dave-mccowan has quit IRC | 11:17 | |
| pbourke | any idea why the --filter=exited didn't pick that up? | 11:17 |
| *** fgimenez has quit IRC | 11:19 | |
| *** fgimenez has joined #kolla | 11:20 | |
| Jeffrey4l | pbourke, what? the filter is should be used like ` --filter status=exited` | 11:25 |
| pbourke | Jeffrey4l: sorry, whatever is in the existing code | 11:26 |
| *** phong_ has quit IRC | 11:26 | |
| pbourke | Jeffrey4l: it seems to me it should pick it up? | 11:26 |
| pbourke | Jeffrey4l: you have just added an extra filter for created | 11:26 |
| kproskurin | pbourke: Do you think we could merge heat trusts? https://review.openstack.org/#/c/236198/ Or we need someone else vote there? | 11:26 |
| Jeffrey4l | I don't know what do your mean | 11:26 |
| pbourke | kproskurin: I was going to wait for sdake to approve as he's more familiar with heat | 11:28 |
| Jeffrey4l | you want to know why I add `--filter status=created`? | 11:29 |
| pbourke | yes | 11:29 |
| *** jmccarthy has quit IRC | 11:29 | |
| *** achanda has joined #kolla | 11:29 | |
| kproskurin | pbourke: got it, tx | 11:29 |
| *** jmccarthy has joined #kolla | 11:30 | |
| Jeffrey4l | I just see this on my laptop. It seems the container stick at the create stage. I have no idea why and how it happend. | 11:30 |
| Jeffrey4l | Adding it just in case. | 11:31 |
| Jeffrey4l | pbourke, ^ | 11:31 |
| pbourke | Jeffrey4l: I understand, but here's my question | 11:32 |
| pbourke | in the log you linked, the openvswitch container is not stuck in created, its exited | 11:32 |
| pbourke | so the created filter wont catch this | 11:32 |
| pbourke | ? | 11:32 |
| *** fthiagogv has joined #kolla | 11:33 | |
| Jeffrey4l | pbourke, the log linked is another PS. without my PS. | 11:33 |
| pbourke | I know, but I dont see how your PS will fix it | 11:34 |
| *** achanda has quit IRC | 11:34 | |
| Jeffrey4l | pbourke, check this http://logs.openstack.org/67/275067/1/check/gate-kolla-dsvm-deploy-ubuntu-source/ff693b2/console.html#_2016-02-02_10_14_12_682 | 11:35 |
| Jeffrey4l | this is the log with my PS. | 11:35 |
| Jeffrey4l | The root cause is the this line `tools/kolla-ansible deploy || print_failure` . The print_failure never be executed if the deploy success. | 11:36 |
| pbourke | ah | 11:37 |
| Jeffrey4l | So seperate it into two line. And print the failed container and exit with code 1 | 11:37 |
| pbourke | I see | 11:37 |
| pbourke | thanks! | 11:37 |
| Jeffrey4l | :D | 11:37 |
| Jeffrey4l | Maybe I should file a bug to desc this? | 11:38 |
| pbourke | Jeffrey4l: I think a better commit message would be great, e.g. "...bash uses lazy evaluation so if the deploy succeeded the print_failure never runs" or something like that | 11:39 |
| pbourke | Jeffrey4l: these things help when you're reviewing 20 patches at a time :) | 11:39 |
| Jeffrey4l | pbourke, yes. You are right. I will fix this. Thanks a lot. | 11:40 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Validate that all the containers are started https://review.openstack.org/275067 | 11:47 |
| ccesario | pbourke, after upgrade..... the deploys process pass.... but now... with other error | 11:50 |
| ccesario | http://pastebin.com/0q1Rk5ZG | 11:50 |
| *** tzn has joined #kolla | 11:52 | |
| pbourke | ccesario: can you paste the versions of docker and docker-py | 11:54 |
| ccesario | # docker --version | 12:00 |
| ccesario | Docker version 1.9.1, build a34a1d5 | 12:00 |
| ccesario | Name: docker-py | 12:00 |
| ccesario | Version: 1.6.0 | 12:00 |
| pbourke | hmm seems correct | 12:01 |
| akwasnie | ccesario: it looks that you do not have this image (192.168.16.228:4000/kollaglue/ubuntu-source-data:1.0.0-liberty) | 12:01 |
| akwasnie | ccesario: if you do docker images, can you find it there? | 12:02 |
| pbourke | try: 'docker pull 192.168.16.228:4000/kollaglue/ubuntu-source-data:1.0.0-liberty' | 12:02 |
| ccesario | docker pull 192.168.16.228:4000/kollaglue/ubuntu-source-data:1.0.0-liberty | 12:04 |
| ccesario | Pulling repository 192.168.16.228:4000/kollaglue/ubuntu-source-data | 12:04 |
| ccesario | Error while pulling image: Get http://192.168.16.228:4000/v1/repositories/kollaglue/ubuntu-source-data/images: dial tcp 192.168.16.228:4000: connection refused | 12:04 |
| *** chandankumar has joined #kolla | 12:04 | |
| ccesario | ps aux | grep dock | 12:04 |
| ccesario | root 1639 0.2 0.9 317596 38484 ? Ssl 10:00 0:00 /usr/bin/docker daemon --insecure-registry 192.168.16.228:4000 | 12:04 |
| pbourke | is your registry running? | 12:09 |
| *** rhallisey has quit IRC | 12:13 | |
| *** weiyu has joined #kolla | 12:16 | |
| ccesario | hmmm... let me check | 12:17 |
| openstackgerrit | Antoine Roux proposed openstack/kolla: Fix wrong behavior when the source is a directory https://review.openstack.org/275104 | 12:23 |
| *** fthiagogv has quit IRC | 12:26 | |
| *** fthiagogv has joined #kolla | 12:27 | |
| *** kproskurin has quit IRC | 12:36 | |
| *** macsz has quit IRC | 12:36 | |
| *** fgimenez has quit IRC | 12:36 | |
| *** macsz1 has joined #kolla | 12:36 | |
| *** fgimenez has joined #kolla | 12:38 | |
| *** fgimenez has quit IRC | 12:38 | |
| *** fgimenez has joined #kolla | 12:38 | |
| openstackgerrit | Allen Gao proposed openstack/kolla: Move the install base type check to kolla-build https://review.openstack.org/274629 | 12:41 |
| ccesario | pbourke, it is running ... http://pastebin.com/Eb0kUMW9 | 12:44 |
| ccesario | pbourke, what branch git is recommended to use ? master ? stable/liberty!? | 12:44 |
| pbourke | ccesario: ok, and did you push your images to it? via the --push arg | 12:44 |
| pbourke | ccesario: either should be fine | 12:44 |
| ccesario | I 'm removing all and re-start the build process | 12:45 |
| ccesario | master or stable/liberty branch ? | 12:52 |
| openstackgerrit | Allen Gao proposed openstack/kolla: Playbook for upgrade mistral https://review.openstack.org/275176 | 12:58 |
| *** anteaya has joined #kolla | 13:00 | |
| *** logan- has quit IRC | 13:00 | |
| *** logan- has joined #kolla | 13:00 | |
| *** weiyu has quit IRC | 13:02 | |
| *** anteaya has quit IRC | 13:06 | |
| *** ajafo has quit IRC | 13:18 | |
| *** ajafo has joined #kolla | 13:24 | |
| *** Marga_ has joined #kolla | 13:25 | |
| *** achanda has joined #kolla | 13:33 | |
| *** dwalsh has joined #kolla | 13:37 | |
| *** dwalsh has joined #kolla | 13:37 | |
| *** achanda has quit IRC | 13:38 | |
| *** thumpba has joined #kolla | 13:42 | |
| *** tzn has quit IRC | 13:44 | |
| *** tzn has joined #kolla | 13:47 | |
| *** dwalsh has quit IRC | 13:51 | |
| *** dwalsh has joined #kolla | 13:51 | |
| *** Allen_Gao has quit IRC | 13:52 | |
| *** Marga__ has joined #kolla | 13:53 | |
| *** ppetit has joined #kolla | 13:57 | |
| *** Marga_ has quit IRC | 13:57 | |
| *** klint has quit IRC | 13:59 | |
| *** salv-orlando has joined #kolla | 14:01 | |
| *** chandankumar has quit IRC | 14:02 | |
| *** thumpba has quit IRC | 14:06 | |
| *** tzn has quit IRC | 14:07 | |
| *** tzn has joined #kolla | 14:07 | |
| *** sdake has quit IRC | 14:11 | |
| *** thomasem-away is now known as thomasem | 14:13 | |
| *** rhallisey has joined #kolla | 14:13 | |
| *** dave-mccowan has joined #kolla | 14:15 | |
| *** dmsimard has quit IRC | 14:15 | |
| *** fgimenez has quit IRC | 14:16 | |
| *** achanda has joined #kolla | 14:18 | |
| *** fgimenez has joined #kolla | 14:19 | |
| *** fgimenez has quit IRC | 14:19 | |
| *** fgimenez has joined #kolla | 14:19 | |
| *** diogogmt has quit IRC | 14:26 | |
| *** achanda has quit IRC | 14:27 | |
| openstackgerrit | Alicja Kwasniewska proposed openstack/kolla: Added Elasticearch and its deployment. https://review.openstack.org/267714 | 14:29 |
| pbourke | ccesario: I think you will have best results with master but stable/liberty is perfectly fine | 14:31 |
| pbourke | ccesario: so whatever suits you | 14:31 |
| *** akwasnie has quit IRC | 14:32 | |
| *** rmart04 has quit IRC | 14:32 | |
| *** rmart04_ has joined #kolla | 14:32 | |
| *** rmart04_ is now known as rmart04 | 14:32 | |
| openstackgerrit | Merged openstack/kolla: Fix wrong behavior when the source is a directory https://review.openstack.org/275104 | 14:33 |
| *** kproskurin_ has joined #kolla | 14:37 | |
| *** absubram has joined #kolla | 14:41 | |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Use docker_remove_delay for auto-removing containers https://review.openstack.org/270213 | 14:42 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Remove rsyslog from all profiles https://review.openstack.org/270151 | 14:42 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Use private network in default settings https://review.openstack.org/269545 | 14:42 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Use volume for MariaDB https://review.openstack.org/270252 | 14:42 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Categorize variables in etc/globals.yml https://review.openstack.org/274560 | 14:42 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Add neutron config https://review.openstack.org/265109 | 14:42 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Remove named volumes https://review.openstack.org/275237 | 14:42 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Remove unexisting data_files in setup.cfg https://review.openstack.org/275238 | 14:42 |
| *** rmart04 has quit IRC | 14:42 | |
| *** absubram_ has joined #kolla | 14:42 | |
| *** rmart04 has joined #kolla | 14:46 | |
| *** absubram has quit IRC | 14:46 | |
| *** absubram_ is now known as absubram | 14:46 | |
| *** jeh has joined #kolla | 14:49 | |
| *** jtriley has joined #kolla | 14:50 | |
| *** rmart04 has quit IRC | 14:50 | |
| *** dwalsh has quit IRC | 14:52 | |
| openstackgerrit | Merged openstack/kolla: fix miss rabbitmq config when hostname include special character https://review.openstack.org/275111 | 14:55 |
| *** inc0 has joined #kolla | 15:00 | |
| inc0 | good morning | 15:01 |
| *** dmsimard has joined #kolla | 15:02 | |
| pbourke | hey inc0 | 15:04 |
| pbourke | inc0: can I ask you a few questions on your nova upgrade patch? Or would you rather just add them all to the review | 15:05 |
| inc0 | pbourke, shoot | 15:06 |
| *** fthiagogv has quit IRC | 15:06 | |
| *** fthiagogv has joined #kolla | 15:07 | |
| pbourke | inc0: ok. first of all, why are you using docker labels in upgrade/conductors.yml, but not the others | 15:07 |
| inc0 | pbourke, so conductors are stopped, all of them | 15:08 |
| inc0 | this is only service that is not rolled-upgraded | 15:08 |
| *** mbound_ has quit IRC | 15:08 | |
| inc0 | and I want to make sure that if I stop every conductor, I have damn good reason to do it | 15:08 |
| inc0 | and don't do it every time I run upgrade play | 15:08 |
| inc0 | I am ok with rebuilding and reruning every other container, as it is rolled upgrade | 15:09 |
| pbourke | can you have more than one conductor running on a node? | 15:11 |
| *** ppetit has quit IRC | 15:11 | |
| inc0 | technically you can, but we don't do this with kolla | 15:11 |
| inc0 | I mean you should be able to, I'm not 100% sure, in any case that would be antipattern at best | 15:12 |
| *** rmart04 has joined #kolla | 15:12 | |
| pbourke | so why not just upgrade it the usual way | 15:12 |
| inc0 | because nova is wired in a way that conductors, all of them, has to be up-to-date | 15:12 |
| inc0 | they do all the message backporting and everything | 15:12 |
| *** thumpba has joined #kolla | 15:13 | |
| inc0 | so if one conductor is newer than other, if it projects a message and other consume it, finding out that it's too new for him, he will raise an error | 15:13 |
| inc0 | thats why stop all the conductors and start new ones | 15:13 |
| pbourke | ok I get that, but what part does the label play in that | 15:13 |
| inc0 | so we had to have some place to put version in | 15:14 |
| inc0 | because minor upgrades doesn't need full stop on conductors | 15:14 |
| inc0 | only major ones | 15:14 |
| inc0 | so I need to be sure that upgrade I'm doing is a major one, and I can't do this by simply inspecting image | 15:15 |
| inc0 | this is related to record-version bp | 15:15 |
| pbourke | the final part of the task though is going to restart if there's a new image / openstack_release var set regardless | 15:16 |
| pbourke | just for reference, I'm talking about this file: https://review.openstack.org/#/c/254395/13/ansible/roles/nova/tasks/upgrade/conductors.yml | 15:16 |
| inc0 | yes but I don't stop it anywhere | 15:17 |
| inc0 | which means I'm rebuilding running container | 15:17 |
| pbourke | so stop+start is different to restart? | 15:18 |
| inc0 | yes because I'm stopping all of them | 15:18 |
| inc0 | remember that task is run on every node before moving to next one | 15:19 |
| inc0 | whereas restarting is somewhat async | 15:19 |
| *** jmccarthy has quit IRC | 15:19 | |
| *** jmccarthy has joined #kolla | 15:19 | |
| inc0 | there is still slight chance that at very short time there will be downtime of all conductors, but that's acceptable imho | 15:20 |
| *** blahRus has joined #kolla | 15:20 | |
| *** dmsimard has quit IRC | 15:21 | |
| *** rmart04 has quit IRC | 15:22 | |
| *** mbound has joined #kolla | 15:23 | |
| inc0 | does that make sense pbourke ? | 15:24 |
| pbourke | not yet :( Maybe I dont understand fully how Ansible executes tasks | 15:24 |
| inc0 | ok | 15:24 |
| inc0 | well let me put it this way | 15:25 |
| ccesario | pbourke, memcached container error ..... http://pastebin.com/37GzHryq | 15:25 |
| inc0 | if you run play, it won't go to next task before previous task is finished right? | 15:25 |
| pbourke | ok | 15:25 |
| pbourke | across all nodes | 15:26 |
| inc0 | on the other hand, thanks to thread option in play (default 8) it will process more than one host during each run | 15:26 |
| inc0 | across all nodes | 15:26 |
| inc0 | but inside single task, there is async | 15:26 |
| inc0 | so if you run less threads than number of conductors, you are 100% sure that there will be no downtime | 15:26 |
| inc0 | because it will restart one at the time | 15:27 |
| inc0 | if you run more, there is still good chance that downtime will now overlap, because container restart is so fast that it will be less than a second of effective downtime | 15:27 |
| pbourke | ok | 15:28 |
| inc0 | and stop->start ensures that there is downtime and there is no conflicting conductors running at the same time all across cluster | 15:28 |
| pbourke | yeah I see it now | 15:28 |
| inc0 | because it will stop first and only after all nodes are stopped, it start new ones | 15:28 |
| inc0 | hence "when" on stop, we don't want to do this unless we have good reasons to | 15:28 |
| *** fgimenez has quit IRC | 15:28 | |
| pbourke | makes sense | 15:29 |
| pbourke | thanks | 15:29 |
| *** fgimenez has joined #kolla | 15:29 | |
| *** fgimenez has joined #kolla | 15:29 | |
| inc0 | np | 15:30 |
| pbourke | so maybe this feeds into the other thing I was asking before | 15:30 |
| *** jmccarthy has quit IRC | 15:30 | |
| pbourke | which is why you've split the restart out into multiple tasks | 15:30 |
| pbourke | it makes sense for the conductor but the rest seem vanilla | 15:31 |
| inc0 | because only conductors have to be up-to-date | 15:31 |
| inc0 | because conductors does backporing of objects and they are "translators" between versions | 15:31 |
| pbourke | right, but for the other services | 15:31 |
| pbourke | why not just include start.yml as in keystone | 15:31 |
| inc0 | other nova services? hmm...good point | 15:32 |
| *** rmart04 has joined #kolla | 15:33 | |
| inc0 | only reason I can think of right now is order of upgrade | 15:33 |
| inc0 | we need to start from conductor | 15:33 |
| pbourke | yeah I was thinking that but I think it could be done with some refactoring of start.yml | 15:33 |
| pbourke | but its not a major deal right now | 15:34 |
| inc0 | yeah it should be as simple as changing order of start | 15:34 |
| inc0 | I'll look into it, thanks | 15:34 |
| pbourke | so the final question is around the sighup | 15:34 |
| pbourke | is it just premature optimisation | 15:34 |
| pbourke | it feels hacky | 15:34 |
| inc0 | well, it's what nova guys told me to do | 15:34 |
| inc0 | nova have upgrade_version=auto | 15:35 |
| inc0 | (or whatever this config is called) | 15:35 |
| inc0 | this means it will autodetect lowest version running and backport messages to it | 15:35 |
| pbourke | umm | 15:35 |
| inc0 | sighup tells nova to re-read this and hopefully notice that it's newer than it used to be | 15:35 |
| pbourke | you're talking about this file? https://review.openstack.org/#/c/254395/13/ansible/roles/nova/tasks/upgrade/reload.yml | 15:35 |
| inc0 | yeha | 15:36 |
| inc0 | http://lists.openstack.org/pipermail/openstack-dev/2015-December/081591.html | 15:37 |
| pbourke | I would be surprised if restarting the container wouldnt have the same effect | 15:38 |
| *** JoseMello has joined #kolla | 15:43 | |
| *** jmccarthy has joined #kolla | 15:45 | |
| *** rmart04 has quit IRC | 15:45 | |
| *** rmart04 has joined #kolla | 15:47 | |
| *** dims has joined #kolla | 15:54 | |
| *** dims_ has quit IRC | 15:54 | |
| SamYaple | morning all | 15:55 |
| SamYaple | inc0: you know I didn't mention this last night, with a few very small tweaks we would be able to allow the user to choose between containerized libvirt and libvirt on the host system (they would do all host configuration) | 15:57 |
| SamYaple | that would fit kollas motto still (dont touch the host) | 15:57 |
| openstackgerrit | Proskurin Kirill proposed openstack/kolla-mesos: Add memcached support https://review.openstack.org/266477 | 15:57 |
| openstackgerrit | Proskurin Kirill proposed openstack/kolla-mesos: Support deploy without configs in tests https://review.openstack.org/275287 | 15:57 |
| *** ssurana has joined #kolla | 15:58 | |
| inc0 | SamYaple, that might be good idea to do as well | 15:58 |
| inc0 | I have few ideas how to deal with this problem now, night sleep helped | 15:58 |
| inc0 | however give me some more time to test/check them out | 15:58 |
| SamYaple | inc0: yea sure. but just letting you know. since we go over tcp its even easier. but we would need a single option (libvirt_on_host: False) and that would be _it_. We can just skip the nova_libvirt container and it would be on the user to setup libvirtd.conf correctly (we can provided needed options in docs) | 16:00 |
| SamYaple | ill submit a patch to this effect for you today | 16:00 |
| SamYaple | it doesnt fix the issue, but some people might prefer that | 16:01 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Remove named volumes https://review.openstack.org/275237 | 16:01 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Use docker_remove_delay for auto-removing containers https://review.openstack.org/270213 | 16:01 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Remove rsyslog from all profiles https://review.openstack.org/270151 | 16:01 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Use private network in default settings https://review.openstack.org/269545 | 16:02 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Use volume for MariaDB https://review.openstack.org/270252 | 16:02 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Categorize variables in etc/globals.yml https://review.openstack.org/274560 | 16:02 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Add neutron config https://review.openstack.org/265109 | 16:02 |
| *** sdake has joined #kolla | 16:21 | |
| sdake | inc0 ping | 16:21 |
| sdake | when on earth did anyone agree we would be oding rolllbacks | 16:22 |
| sdake | roll forwrd is hard enough | 16:22 |
| sdake | personallly i think rollbacks are a near impossibility | 16:22 |
| SamYaple | sdake: morning | 16:22 |
| sdake | but maybe down the road it can happen | 16:22 |
| sdake | morning SamYaple | 16:22 |
| SamYaple | sdake: please for the love of god merge this! https://review.openstack.org/#/c/236198/ | 16:22 |
| sdake | i am on half day pto today | 16:22 |
| SamYaple | its being held for you ::) | 16:23 |
| sdake | SamYaple it has 2 +2s | 16:23 |
| SamYaple | sdake: waiting on you | 16:23 |
| sdake | let me read full log | 16:23 |
| SamYaple | you are heat guy | 16:23 |
| sdake | tesst it | 16:24 |
| sdake | got it | 16:24 |
| SamYaple | I (try) to hold patches for those most familiar with the issue | 16:24 |
| sdake | ok well put a workflow -1 on the thing | 16:24 |
| SamYaple | its not my patch and it works for me | 16:24 |
| sdake | i think i am interruption free for the rest of th work | 16:24 |
| sdake | did you use wait conditions? | 16:24 |
| SamYaple | but me and you disagree on what works in heat :) | 16:24 |
| SamYaple | probably not, im not a heat guy | 16:25 |
| sdake | I am not usre if I can test it with my lab torn apart | 16:25 |
| SamYaple | barely got a template from the interwebs working | 16:25 |
| sdake | i am not sure i can get wcs to work in a vm | 16:25 |
| sdake | wcs=waitconditions | 16:25 |
| SamYaple | well nihilifer says its working. we can merge it. its not any worse than it is now | 16:25 |
| sdake | that is why trusts are needed | 16:25 |
| SamYaple | if you cant test it really dont want to hold longer (look at age of patch) | 16:25 |
| sdake | lets confirm with nihilifer if wcs work | 16:25 |
| sdake | if not we can get a confirmation from asalkeld | 16:26 |
| SamYaple | nihilifer: ^^ | 16:26 |
| sdake | he is also a heat dude | 16:26 |
| SamYaple | fair enough | 16:26 |
| SamYaple | just so its on your radara | 16:26 |
| sdake | his labi isn't afu atm | 16:26 |
| sdake | sory guys i knwo i've been out a bit in january as well as dec was a total loss | 16:28 |
| sdake | the pain will end soon with the remodel | 16:28 |
| sdake | and hten i'll be back at 18 hour days just like you like it :) | 16:28 |
| SamYaple | sdake: named volumes are done btw so thats out of your mind | 16:29 |
| sdake | sweet | 16:29 |
| sdake | nice job guys | 16:29 |
| *** daneyon_ has joined #kolla | 16:29 | |
| sdake | ar enamed volumes also slow like container ovlluems? | 16:30 |
| sdake | like data containers | 16:30 |
| SamYaple | its the same mechanism | 16:30 |
| elemoine | sdake, are you available for the IRC meeting tomorrow? | 16:30 |
| SamYaple | a mechanism ive never found slow | 16:30 |
| sdake | elemoine i will make time | 16:30 |
| SamYaple | but yea it would be the same | 16:30 |
| elemoine | there's currently no agenda on https://wiki.openstack.org/wiki/Meetings/Kolla | 16:30 |
| sdake | SamYaple if you try booting nova from the data container its alot slower | 16:30 |
| sdake | elemoine whoever wants an agenda item fixes the agenda meeting calendar :) | 16:31 |
| sdake | it typically just takes care of itself | 16:31 |
| elemoine | sdake, makes sense | 16:31 |
| sdake | i never touch it and its always ready to rock :) | 16:31 |
| sdake | ok i think our vote is done for hte time to select our new meeting | 16:31 |
| sdake | but i need some time toe coordinate that with the infra team, so that wont start until next week | 16:31 |
| openstackgerrit | Merged openstack/kolla: Playbook for upgrade murano https://review.openstack.org/275038 | 16:32 |
| sdake | ok i actually have to prep for this webinar interview for hte foundation and get a shower and get my contractors sorted out and maybe fit in a hair cut! | 16:32 |
| inc0 | sdake, pong | 16:32 |
| sdake | inc0 rollback - we ren't doing it | 16:32 |
| sdake | i wasn't clear on your email, were ou suggesting not doing thin containers as well | 16:33 |
| *** daneyon has quit IRC | 16:33 | |
| elemoine | sdake, I updated the wiki page | 16:33 |
| inc0 | yeah, what I mean is to remove code that is not docker 1.8 compatible | 16:33 |
| elemoine | I kept SamYaple's items as he wasn't there last time IIRC | 16:33 |
| sdake | SamYaple plesae try to be at next meeting yu had some agenda items | 16:34 |
| inc0 | quick question guys | 16:35 |
| inc0 | if we rebuild contrainer with vms | 16:35 |
| inc0 | vms will get not only restarted, but actually removed right? | 16:36 |
| sdake | so there is a bug | 16:36 |
| sdake | vms are restartedi if the container is restarted | 16:36 |
| sdake | but in liberty i tested it iand the vms for certain did not restart | 16:36 |
| inc0 | and container is restarted when docker is restarted | 16:36 |
| sdake | they stayed active and libvirt connected to them | 16:36 |
| sdake | i have multiple people owkring on this problem in the docker community | 16:36 |
| inc0 | which version of docker? | 16:36 |
| sdake | they recognize the implications of vm downtime | 16:37 |
| sdake | inc0 the docs for liberty say 1.7.0 and i was doing the 2 weeks of making that vm shit work at the end of hte cycle | 16:37 |
| sdake | mostly l3/rc1 | 16:37 |
| *** daneyon has joined #kolla | 16:37 | |
| sdake | it otok me 2 weeks | 16:37 |
| sdake | we have identified a regression in the new version of docker | 16:37 |
| sdake | or in how we are using it | 16:37 |
| inc0 | so currently I can't get pid=host stuff work on docker restart | 16:37 |
| sdake | something is broken and htat needs to be fixed, but we can upgrade docker daemon without vm downtime and upgrade libvirt container without vm downtime in liverty | 16:38 |
| inc0 | and we don't have migration plan from data container to named volume | 16:38 |
| inc0 | sdake, not the case any more it seems | 16:38 |
| sdake | inc0 we can get a migration plan for data container to named volume | 16:38 |
| sdake | inc0 i know as i said i have multiple cats working on it | 16:38 |
| sdake | we haev to trust they will get the job done | 16:38 |
| sdake | they are seriously the #1 experts in all this pid=host space | 16:39 |
| sdake | and they agreed current behavior is broken | 16:39 |
| inc0 | ok, will we get this before mitaka release? | 16:39 |
| sdake | god i hope so | 16:39 |
| sdake | otherwise we wont be upgrading the libvirt container | 16:39 |
| inc0 | and will we get 1.8 ->1.11 without restarting vms? | 16:39 |
| sdake | kill -9 docker still keeps the vms intact | 16:39 |
| sdake | samyaple tested and confirmed that | 16:39 |
| *** daneyon_ has quit IRC | 16:40 | |
| inc0 | I'll check out kill -9 | 16:40 |
| inc0 | service docker restart breaks stuff | 16:40 |
| sdake | kill -9 works | 16:40 |
| sdake | everything else not sure | 16:40 |
| sdake | our upgrade is kill -9 all nodes of docker, upgrade rpm, start docker daemons | 16:41 |
| sdake | sorry all docker dameon nodes | 16:41 |
| inc0 | if that will keep vm intact, I'm ok | 16:42 |
| openstackgerrit | Merged openstack/kolla: Playbook for upgrade memcached https://review.openstack.org/275035 | 16:42 |
| inc0 | I mean we still won't be able to work with data container | 16:42 |
| inc0 | but that's different issue to tackle | 16:42 |
| inc0 | ok, so kill -9 does preserve vms, but starting docker removes it | 16:44 |
| *** intr1nsic has quit IRC | 16:44 | |
| inc0 | which can be solved with 1.11 if your docker ppl will take that into account | 16:44 |
| inc0 | so being able to run container on pid that's running | 16:44 |
| SamYaple | sdake: im -2 on kill -9 period | 16:45 |
| SamYaple | that can corrupt the docker database | 16:45 |
| SamYaple | inc0: the issue here is not solved with docker 1.11 | 16:46 |
| inc0 | and docker start breaks stuff anyway | 16:46 |
| SamYaple | you still need to upgrade docker contaienr for qemu | 16:46 |
| SamYaple | nova_libvirt | 16:46 |
| sdake | ok well lets figure out what the hell the requireements are and give them to me and i'll get them done in docker | 16:46 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Remove unexisting data_files in setup.cfg https://review.openstack.org/275238 | 16:46 |
| inc0 | but we can decouple qemu upgrade from kolla upgrade | 16:46 |
| SamYaple | from nova upgrade, not kolla upgrade | 16:46 |
| SamYaple | but i agree | 16:46 |
| SamYaple | it can be decoupled | 16:46 |
| SamYaple | maybe | 16:47 |
| sdake | yes decouple | 16:47 |
| SamYaple | nova does have libvirt requirements, and those change | 16:47 |
| inc0 | it can be decoupled from kolla upgrade as kolla code | 16:47 |
| openstackgerrit | Merged openstack/kolla: Validate that all the containers are started https://review.openstack.org/275067 | 16:47 |
| SamYaple | i know the supported list of libvirt and qemu changes | 16:47 |
| inc0 | yes but unless it changes we don't have to do it | 16:47 |
| SamYaple | but it does every release | 16:47 |
| *** dims_ has joined #kolla | 16:47 | |
| inc0 | but you can do it later | 16:47 |
| inc0 | not all at once | 16:47 |
| inc0 | and that's important | 16:48 |
| SamYaple | right, but you wont be in a supported package state until you do | 16:48 |
| SamYaple | and that might break service agreements | 16:48 |
| SamYaple | just something to keep in mind | 16:48 |
| inc0 | all I'm saying, let's provide option not force it on ops | 16:48 |
| inc0 | we need to upgrade qemu and docker at some point - agree | 16:49 |
| SamYaple | decouple i am ok with, but its still a requirement for a full upgrade from one version to another of kolla | 16:49 |
| SamYaple | are we in agreement on these points? | 16:49 |
| inc0 | but what I don't agree with is to upgrade it at the very same day we upgrade openstack | 16:49 |
| *** dims has quit IRC | 16:49 | |
| openstackgerrit | Proskurin Kirill proposed openstack/kolla-mesos: Add memcached support https://review.openstack.org/266477 | 16:49 |
| inc0 | problem is, due to named volumes and thin containers, you can't do this | 16:49 |
| SamYaple | cant do what? | 16:49 |
| inc0 | upgrade kolla without vm downtime | 16:50 |
| sdake | otp brb | 16:50 |
| SamYaple | i know | 16:50 |
| inc0 | because new kolla won't work | 16:50 |
| inc0 | at all | 16:50 |
| inc0 | so no, you can't decouple it | 16:50 |
| kproskurin_ | sdake: Could you pls look into heat trusts patch? https://review.openstack.org/#/c/236198/ I think it ready to be merged | 16:50 |
| SamYaple | well you can | 16:50 |
| inc0 | and old kolla won't work with new docker so you can't upgrade docker and then upgrade kolla | 16:50 |
| SamYaple | you just have to decouple the qemu named volume switch too | 16:50 |
| SamYaple | but youre right, this is a full stop of the vms for the named volume and qemu switch at this time | 16:51 |
| sdake | sounding good sam | 16:51 |
| inc0 | there is no version of kolla that supports 1.8 and 1.10 | 16:51 |
| inc0 | at the same time | 16:51 |
| SamYaple | uhhh | 16:51 |
| SamYaple | yea there is | 16:51 |
| SamYaple | liberty runs just fine on 1.10 | 16:51 |
| SamYaple | you just cant run our playbooks | 16:51 |
| SamYaple | you can upgrade docker in liberty | 16:52 |
| SamYaple | then run the new playbooks for mitaka | 16:52 |
| inc0 | but during docker upgrade you don't have kolla plays to help you | 16:53 |
| sdake | yes agree this is the approach we should tkae as we discussed friday | 16:53 |
| SamYaple | inc0: this is true, but the way we have built the containers by default they will restart | 16:53 |
| *** rhallisey has quit IRC | 16:53 | |
| SamYaple | if the default restart policy is changed then its on the operator to restart them | 16:53 |
| SamYaple | but they should be aware of that as they changed teh option | 16:54 |
| *** absubram has quit IRC | 16:55 | |
| sdake | ok so what do we need | 16:55 |
| sdake | we need a docker that doesn't restart vms on a container upgrade | 16:56 |
| sdake | which used to work | 16:56 |
| sdake | and no longer does | 16:56 |
| sdake | what ELSE do we need | 16:56 |
| SamYaple | sdake: that is an option in docker that doesnt exist | 16:56 |
| SamYaple | im working that part out | 16:56 |
| sdake | SamYaple it WAS working | 16:56 |
| SamYaple | ill hit up the apprpriate channel | 16:56 |
| sdake | i will get my people to getit working again | 16:56 |
| SamYaple | sdake: actually it _wasnt_ according to docker | 16:56 |
| *** mbound has quit IRC | 16:56 | |
| SamYaple | that was a bug | 16:56 |
| sdake | oh that was a bug? | 16:56 |
| SamYaple | thats why the pid=host containres couldn't fully stop | 16:56 |
| *** achanda has joined #kolla | 16:57 | |
| sdake | and they fixed the bug and broke the function we depended on? | 16:57 |
| sdake | hae a issue tracker #? | 16:57 |
| SamYaple | google knows i do not | 16:57 |
| sdake | did you see it though or just guessing | 16:57 |
| SamYaple | but anyway, we need that as a feature _and_ we need docker daemon restart without stopping containers | 16:57 |
| SamYaple | sdake: no thats what happened | 16:57 |
| sdake | so your speculating? :) | 16:57 |
| SamYaple | no thats what actually happened | 16:58 |
| SamYaple | there are bugs out there for this | 16:58 |
| sdake | so you saw evidence | 16:58 |
| sdake | ok | 16:58 |
| SamYaple | i dont know them its been like a year | 16:58 |
| sdake | good you saw evidence of a bug, i can work from there | 16:58 |
| sdake | ok docker daemon restart without topping containers, the answer to that is runc | 16:58 |
| SamYaple | inc0: so what we need is a shared pid namespace. and docker is working on that | 16:58 |
| sdake | that wont be ready for mitaka | 16:58 |
| SamYaple | inc0: with a shared pid namepsace we will be solid | 16:59 |
| SamYaple | sdake: the docker daemon restart will be ready for mitaka | 16:59 |
| *** blahRus1 has joined #kolla | 16:59 | |
| SamYaple | sdake: the "qemu process survives container stop and recreate" will not be ready | 16:59 |
| sdake | SamYaple docker palns to solve this problem with shared pid namespace? | 17:00 |
| sdake | dwalsh ^^ | 17:00 |
| sdake | damn | 17:00 |
| sdake | not here | 17:00 |
| SamYaple | no it doesnt plan anything to "solve" this issue | 17:00 |
| inc0 | they have "hot upgrades" in roadmap | 17:00 |
| inc0 | not sure what that means | 17:00 |
| SamYaple | inc0: thats teh docker daemon restart | 17:00 |
| sdake | ok with the host=pid thing here is what is going to happen | 17:00 |
| sdake | mark my words, mark them well | 17:01 |
| sdake | we will get an option in docker | 17:01 |
| inc0 | I know, not sure if that's about shared pid namespace | 17:01 |
| *** blahRus has quit IRC | 17:01 | |
| inc0 | but that still needs kill -9 mind you | 17:01 |
| sdake | the option will be "let all the processes be managed by pid=1 | 17:01 |
| inc0 | and Sam has point - this can break things | 17:01 |
| inc0 | otherwise docker explicitly kills container | 17:01 |
| sdake | not kill -9 | 17:01 |
| inc0 | yes, but having option in new docker doesnt help us | 17:02 |
| inc0 | because we still need to turn off 1.8 | 17:02 |
| sdake | so if this option is used, docker will give up responsibility for killing pids with pid=host | 17:02 |
| inc0 | and that kills containers | 17:02 |
| sdake | if as SamYaple claims they fixed bugs to make that happen | 17:02 |
| sdake | inc0 got it | 17:03 |
| sdake | kill -9 wfm ;) | 17:03 |
| SamYaple | sdake: nevertheless im -2 | 17:03 |
| sdake | the docker database is consistent | 17:03 |
| SamYaple | thats a good way to loose all of your data | 17:03 |
| sdake | if docker can't handl a process crash we are doome | 17:03 |
| sdake | d | 17:03 |
| sdake | a kill -9 is a process crash just different uncatchable signal | 17:04 |
| sdake | if segv hits docker | 17:04 |
| sdake | i highly highlly doubt the docker db woudl be corrupted | 17:04 |
| sdake | in such a way as to totally destroy the system | 17:04 |
| sdake | if it is, then we are fucked | 17:04 |
| inc0 | if it does, we have much bigger issue to solve here | 17:05 |
| sdake | ok gotta do some stuff | 17:05 |
| inc0 | ok | 17:05 |
| SamYaple | well im telling you that _can_ happen | 17:05 |
| SamYaple | we don't want it to | 17:05 |
| sdake | i'll be back but not to the channel until after 3-4pm | 17:05 |
| openstackgerrit | Merged openstack/kolla: Playbook for upgrade mistral https://review.openstack.org/275176 | 17:05 |
| inc0 | ok I'll work on migrating data container to named volume | 17:05 |
| *** achanda has quit IRC | 17:06 | |
| sdake | i hae to prep for haircutting and taking lucnh to my kid at school and webinar | 17:06 |
| inc0 | I might have super-hacky idea | 17:06 |
| sdake | inc0 good that is something we need | 17:06 |
| sdake | super hhacky is good | 17:06 |
| sdake | we are doing it one time | 17:06 |
| sdake | we may carry it cuple releases | 17:06 |
| sdake | openstack recommends 2 releasses before deprecating old stuff | 17:06 |
| sdake | atleast that is how it used to be not sure now | 17:06 |
| *** dwalsh has joined #kolla | 17:06 | |
| inc0 | I guess if we have 1 release for us it's ok | 17:07 |
| inc0 | but anyway, we need deprecation process and docker migration process | 17:07 |
| inc0 | kill -9 keeps vm, we will test it out to make sure it won't destroy your data | 17:07 |
| inc0 | then we wait till docker enables shared pid namespace | 17:07 |
| inc0 | and attach to running pids | 17:08 |
| inc0 | and we'll do this along with migration to data container | 17:08 |
| SamYaple | inc0: i have the named migration covered | 17:08 |
| Jeffrey4l | But the binary are still in the old container, right? | 17:08 |
| inc0 | SamYaple, do share plz | 17:08 |
| SamYaple | inc0: as i stated in the initial named volume patch | 17:08 |
| SamYaple | inc0: so there are two paths | 17:09 |
| SamYaple | 1) lookup old path via docker inspect and do a host-level `mv` operation | 17:09 |
| SamYaple | 2) new container with old volume and new named volume and do mv inside container | 17:10 |
| inc0 | that was hacky way I had in mind | 17:10 |
| SamYaple | i like 2) | 17:10 |
| inc0 | yeah 2 seems better | 17:10 |
| *** mbound has joined #kolla | 17:10 | |
| inc0 | we can do "migration volume" | 17:10 |
| inc0 | "migration container" | 17:10 |
| SamYaple | no we dont actually need a new container for this | 17:10 |
| inc0 | however that will also cause downtime | 17:10 |
| SamYaple | but yes it will cause downtime | 17:10 |
| inc0 | yeah it's just ubuntu container with one command | 17:10 |
| SamYaple | right | 17:11 |
| SamYaple | ill just have to introduce the command option back to kolla_docker, not an issue | 17:11 |
| inc0 | so if we do symlink instead | 17:11 |
| inc0 | it might not cause downtime | 17:11 |
| SamYaple | wont work on all underlying filysystems | 17:11 |
| inc0 | symlink on host level | 17:11 |
| SamYaple | also i worry about the overlyafs stuff too | 17:11 |
| SamYaple | so i want to only use 1) as a last resort | 17:12 |
| inc0 | yeah, well, this problem should be solvable tho | 17:12 |
| inc0 | we just need to figure out how to pretend that container data is volume | 17:12 |
| SamYaple | host level things are going to make that hard/impossible | 17:12 |
| SamYaple | we cant account for every host level change and symlink doesnt atually move the data | 17:13 |
| SamYaple | what if docker doesn't allow symlink (or worse, they allow it now and it breaks in the future) | 17:13 |
| inc0 | how about we ask docker to introduce yet another thing - make volume out of existing data | 17:13 |
| SamYaple | yea but thats not going to happen for Mitaka | 17:13 |
| SamYaple | and then it doesnt matter anyway | 17:13 |
| SamYaple | btw what you are asking for is volume rename | 17:13 |
| SamYaple | and it has been asked before | 17:13 |
| inc0 | SamYaple, how angry would you be at me if I'd move us back to data containers? | 17:14 |
| inc0 | because if we do that | 17:14 |
| inc0 | we can say that "run mitaka on 1.8" | 17:14 |
| sdake | we are not going back to data container | 17:14 |
| *** salv-orlando has quit IRC | 17:14 | |
| sdake | ok this is definately midcycle topic | 17:15 |
| inc0 | and by the next release we'll have all the docker stuff we need | 17:15 |
| sdake | i really have to run guys | 17:15 |
| inc0 | ok, let's discuss it in midcycle | 17:15 |
| sdake | i will bbe back in about 5 hours | 17:15 |
| sdake | full blast rest of day | 17:15 |
| sdake | keep discussing now | 17:15 |
| sdake | get the ideas out there | 17:15 |
| SamYaple | inc0: not about angry, data contaienrs dont solve any issues | 17:15 |
| inc0 | don't bring weapons, bare hands will have less chance to kill us | 17:15 |
| sdake | but just pointing ut we will need to rehash this at midcycle | 17:15 |
| inc0 | SamYaple, will allow us to keep docker 1.8 till next release | 17:15 |
| *** sdake has quit IRC | 17:15 | |
| SamYaple | no it wont | 17:16 |
| SamYaple | we need 1.10 | 17:16 |
| inc0 | for thin containers | 17:16 |
| inc0 | I know | 17:16 |
| SamYaple | for thin container | 17:16 |
| inc0 | but we can hold this till next release as well | 17:16 |
| inc0 | what I'm asking is deprecation period for 1.8 | 17:16 |
| inc0 | docker 1.8 | 17:16 |
| SamYaple | lol like i said reverting an entire cycle | 17:16 |
| inc0 | of a single release | 17:16 |
| SamYaple | and you still dont solve the data contaienr issue | 17:17 |
| inc0 | no, kolla_docker will actually save our asses | 17:17 |
| inc0 | because it works on all of these | 17:17 |
| SamYaple | inc0: https://bugs.launchpad.net/kolla/+bug/1521231 | 17:17 |
| openstack | Launchpad bug 1521231 in kolla "rabbitmq fail to restart on image update/upgrade" [Critical,In progress] - Assigned to Vladislav Belogrudov (vlad-belogrudov) | 17:17 |
| SamYaple | data contaienrs are a flawed idea we should have never used | 17:17 |
| inc0 | ok, host bind instead of named volume then | 17:17 |
| inc0 | for 1 release | 17:17 |
| inc0 | I know it's against our policy of clean host | 17:18 |
| SamYaple | we still need to move the data... | 17:18 |
| SamYaple | same issue | 17:18 |
| inc0 | well, not exactly | 17:18 |
| SamYaple | how so? | 17:19 |
| inc0 | you can do it slowly and one container at the time | 17:19 |
| inc0 | whereas we propose small revolution | 17:19 |
| SamYaple | same with named volumes though | 17:19 |
| SamYaple | slowly move some data | 17:19 |
| SamYaple | not all at once | 17:19 |
| inc0 | but you have to upgrade docker | 17:19 |
| inc0 | in between | 17:19 |
| *** fgimenez has quit IRC | 17:19 | |
| SamYaple | no you update docker then pull mitaka playbooks | 17:20 |
| SamYaple | updating docker is done on liberty | 17:20 |
| SamYaple | so docker is updated and ready for this | 17:20 |
| inc0 | so bottom line, we make sure that docker can reattach itself to running pid | 17:20 |
| SamYaple | we cant get around these downtimes. but honestly downtimes were _always_ expected. we said hard downtimes during tokyo was expected | 17:21 |
| SamYaple | inc0: its not docker its libvirt | 17:21 |
| SamYaple | libvirt can do that | 17:21 |
| inc0 | well my point is next containers we run will have same VMs we had | 17:21 |
| inc0 | before | 17:21 |
| SamYaple | we want docker to disown the processes | 17:21 |
| SamYaple | and forget about them | 17:21 |
| inc0 | which can happen if we kill -9 it | 17:21 |
| SamYaple | but not on a start again | 17:21 |
| SamYaple | like you said | 17:21 |
| inc0 | but that can be done with docker part | 17:22 |
| SamYaple | really what is wanted is a complete disowning of the process | 17:22 |
| SamYaple | no tracking. completely forget it | 17:22 |
| inc0 | or being able to take on running process from previous docker | 17:22 |
| inc0 | either one works | 17:22 |
| SamYaple | doesnt solve the liberty-to-mitaka thing at all | 17:22 |
| inc0 | not killing it - tacking ownership | 17:22 |
| SamYaple | so well save that for conversation for another time | 17:23 |
| inc0 | midcycle | 17:23 |
| SamYaple | midcycle should be problems at hand mostly | 17:23 |
| inc0 | remember, no weapons or sharp objects | 17:23 |
| SamYaple | i think we have enough of those | 17:23 |
| inc0 | that is problem at hand | 17:23 |
| inc0 | that is upgrade path | 17:23 |
| inc0 | discussion | 17:23 |
| SamYaple | ok but im talking liberty-mitaka upgrade | 17:23 |
| SamYaple | we do that discussion first | 17:23 |
| SamYaple | then if time mitaka-newton | 17:23 |
| SamYaple | but well have time at summit for that too | 17:24 |
| SamYaple | ok im going to lunch | 17:24 |
| SamYaple | inc0: ill get you that libvirt_on_host patch today | 17:24 |
| inc0 | I'm ok with moving libvirt stack to host | 17:25 |
| SamYaple | not by defaut | 17:25 |
| SamYaple | as an option | 17:25 |
| SamYaple | if you want it on host, go for it | 17:25 |
| inc0 | ok, whatever, as upgrade | 17:25 |
| SamYaple | no | 17:25 |
| inc0 | then later we can try to attach to it | 17:25 |
| SamYaple | no im saying an option to allow you to run livirt_on_host if you choose | 17:25 |
| SamYaple | but not the default | 17:26 |
| inc0 | as upgrade with option I personally will recommend, may not be default but I'm fairly sure will be pretty populart | 17:26 |
| SamYaple | you might be right | 17:26 |
| SamYaple | i dont disagree | 17:26 |
| *** salv-orlando has joined #kolla | 17:26 | |
| SamYaple | its been asked before | 17:26 |
| inc0 | also I'm ok with having new vms running in container as we do now | 17:26 |
| inc0 | just...we need docker to solve this stuff | 17:26 |
| SamYaple | my only point is it wont be default, options are great | 17:26 |
| SamYaple | but by default we dont touch the hose | 17:27 |
| SamYaple | host* | 17:27 |
| inc0 | what I'd like to see is also migration plan container->host | 17:27 |
| SamYaple | thats easy too | 17:27 |
| SamYaple | if we do 2) of the migration plan | 17:27 |
| inc0 | it should be | 17:27 |
| SamYaple | its time consuming potentially though | 17:27 |
| inc0 | well we won't really have to | 17:27 |
| SamYaple | potentialyl thats moving cross file-system | 17:27 |
| inc0 | but we can do this lm-style | 17:27 |
| inc0 | and somehow reconnect qemu to new file after we're done | 17:28 |
| inc0 | and if we use ceph, we're golden | 17:28 |
| SamYaple | ceph is golden, moving qcow2 is slow | 17:28 |
| SamYaple | we arent going to send qemu commands over qmp to move the files cross file-system though | 17:28 |
| inc0 | well ceph is default anyway so I don't really care that much | 17:28 |
| inc0 | ephemeral is emphemeral, disks die | 17:28 |
| SamYaple | ephemeral is on ceph too ;) | 17:28 |
| inc0 | it's not that ephemeral | 17:29 |
| SamYaple | anyway moving on to other issue. this has solutions | 17:29 |
| inc0 | I mean if you use ephemeral in ceph, they're not files in fs | 17:29 |
| SamYaple | which is the default | 17:29 |
| inc0 | so if you use ceph for ephemerals, which we do by default and suggest | 17:29 |
| *** thumpba has quit IRC | 17:29 | |
| inc0 | there is no data to migrate | 17:30 |
| SamYaple | nova backed by ceph IS ephemeral backed by ceph | 17:30 |
| inc0 | I know, so we zombify qemu processes, move libvirt to host and reattach to qemu | 17:30 |
| inc0 | and then upgrade docker with API downtime but VMs stays alive | 17:30 |
| inc0 | sounds like a plan? | 17:31 |
| SamYaple | that wont work | 17:31 |
| SamYaple | but you can try | 17:31 |
| inc0 | how so? | 17:32 |
| SamYaple | docker wil lstill kill qemu | 17:32 |
| *** jasonsb has quit IRC | 17:32 | |
| *** openstackgerrit has quit IRC | 17:32 | |
| *** openstackgerrit has joined #kolla | 17:32 | |
| inc0 | well I assume old version will not (kill -9) and new version will not as we'll push sdake to use hardest whip he can to make sure it won't | 17:32 |
| inc0 | I wanted to learn golang and I'll just start writing this stuff if that's the case | 17:33 |
| *** Jeffrey4l has quit IRC | 17:33 | |
| SamYaple | focus man | 17:33 |
| SamYaple | you have no focus | 17:33 |
| inc0 | I'm ok with saying on summit "this is docker release preview, to do upgrades we need features from it so please hold on till it's released" | 17:34 |
| SamYaple | it wont be in 1.11 | 17:34 |
| inc0 | noone deploy openstack on day 1 anyway | 17:34 |
| ccesario | pbourke, it is running :) | 17:35 |
| pbourke | ccesario: great :) | 17:35 |
| pbourke | ccesario: did you figure out what was your original problem? | 17:35 |
| *** kproskurin_ has quit IRC | 17:38 | |
| ccesario | the first one it was the ansible version... the second it was the registry service.... and the last one kolla_install_type: "source" ... by default binary it was used .... I'm still validating all services.... | 17:38 |
| ccesario | only mismatches caused by user side | 17:40 |
| *** unicell has quit IRC | 17:41 | |
| *** mbound has quit IRC | 17:42 | |
| *** sdake has joined #kolla | 17:48 | |
| *** sdake has quit IRC | 17:50 | |
| *** salv-orl_ has joined #kolla | 17:50 | |
| *** rhallisey has joined #kolla | 17:53 | |
| *** salv-orlando has quit IRC | 17:54 | |
| limamauricio | SamYaple, i'm doing the last tests with the Manila and now I get this error -> OSError: [Errno 13] Permission denied: '/var/lib/manila/tmp' | 17:57 |
| SamYaple | limamauricio: can you post a patch | 17:58 |
| SamYaple | limamauricio: that is for the oslo_lock stuff | 17:58 |
| limamauricio | [oslo_concurrency] lock_path = /var/lib/manila/tmp | 17:59 |
| SamYaple | yea but the containter clearly has no perms there. if you are working on the most recent code, that would make sense | 18:00 |
| *** pbourke has quit IRC | 18:00 | |
| *** pbourke has joined #kolla | 18:00 | |
| limamauricio | tmp is a dir or a file? | 18:01 |
| SamYaple | dir | 18:02 |
| SamYaple | but manila user should have /var/lib/manila perms | 18:02 |
| *** jmccarthy has quit IRC | 18:02 | |
| limamauricio | I add a manila_sudoers file on manila-base dir | 18:02 |
| limamauricio | this error happens in share_container | 18:03 |
| SamYaple | hold on limamauricio, ill submit the patch for you to fix this issue (rather than trying to describe it further) | 18:03 |
| limamauricio | ok | 18:04 |
| *** rmart04 has quit IRC | 18:04 | |
| ccesario | pbourke, does kolla support upgrades ? Kilo->Liberty by example ? if yes ... is there any document about it ? | 18:04 |
| *** rhallisey has quit IRC | 18:05 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Fix manila perms https://review.openstack.org/275359 | 18:05 |
| SamYaple | limamauricio: https://review.openstack.org/275359 | 18:06 |
| *** tzn has quit IRC | 18:06 | |
| *** athomas has quit IRC | 18:10 | |
| *** athomas has joined #kolla | 18:11 | |
| *** athomas has quit IRC | 18:12 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Rename kolla_ansible to kolla_toolbox https://review.openstack.org/274154 | 18:13 |
| *** intr1nsic has joined #kolla | 18:13 | |
| *** unicell has joined #kolla | 18:16 | |
| *** achanda has joined #kolla | 18:19 | |
| *** thumpba has joined #kolla | 18:21 | |
| *** thumpba has quit IRC | 18:27 | |
| *** salv-orl_ has quit IRC | 18:28 | |
| *** salv-orlando has joined #kolla | 18:28 | |
| *** salv-orlando has quit IRC | 18:31 | |
| *** salv-orlando has joined #kolla | 18:32 | |
| *** dwalsh has quit IRC | 18:36 | |
| *** thumpba has joined #kolla | 18:42 | |
| *** salv-orlando has quit IRC | 18:43 | |
| *** dwalsh has joined #kolla | 18:45 | |
| *** salv-orlando has joined #kolla | 18:45 | |
| *** absubram has joined #kolla | 18:46 | |
| *** rmart04 has joined #kolla | 18:46 | |
| *** tzn has joined #kolla | 18:47 | |
| *** tzn has quit IRC | 18:48 | |
| SamYaple | hey inc0 user namespace remapping has landed in docker 1.10 | 18:49 |
| SamYaple | all the drop-root work we did was for naught! | 18:49 |
| *** rmart04 has quit IRC | 18:56 | |
| *** whydidyoustealmy has joined #kolla | 19:03 | |
| limamauricio | SamYaple: I have the same error | 19:05 |
| SamYaple | limamauricio: did you fully rebuild your contianers? | 19:05 |
| limamauricio | yes :/ | 19:06 |
| SamYaple | hmm im not sure. was there a typo? the fact is if the container is running as the manila user it should have access to /var/lib/manila with my patch | 19:06 |
| *** vincent_1dk has joined #kolla | 19:08 | |
| limamauricio | I accessed the container and the tmp directory does not exist. | 19:10 |
| *** Kennan has quit IRC | 19:12 | |
| *** barra204 has quit IRC | 19:12 | |
| *** kbyrne has quit IRC | 19:12 | |
| *** Slower has quit IRC | 19:12 | |
| *** vincent_vdk has quit IRC | 19:12 | |
| *** Slower has joined #kolla | 19:14 | |
| *** kbyrne has joined #kolla | 19:15 | |
| *** Kennan has joined #kolla | 19:16 | |
| *** dwalsh has quit IRC | 19:25 | |
| *** salv-orlando has quit IRC | 19:25 | |
| *** dmsimard has joined #kolla | 19:31 | |
| *** ppetit has joined #kolla | 19:34 | |
| *** dmsimard has quit IRC | 19:37 | |
| *** dmsimard has joined #kolla | 19:41 | |
| *** dmsimard has quit IRC | 19:41 | |
| *** dmsimard has joined #kolla | 19:46 | |
| openstackgerrit | Qiu Yu proposed openstack/kolla: Update register playbooks to use per-role endpoint address https://review.openstack.org/275386 | 19:48 |
| *** whydidyoustealmy is now known as barra204 | 19:53 | |
| *** Marga__ has quit IRC | 20:00 | |
| *** thumpba has quit IRC | 20:00 | |
| *** ppetit has left #kolla | 20:01 | |
| SamYaple | limamauricio: is that the error you get? | 20:05 |
| SamYaple | it should be creaintg the tmp dir for you | 20:05 |
| *** achanda has quit IRC | 20:11 | |
| *** salv-orlando has joined #kolla | 20:28 | |
| *** sdake has joined #kolla | 20:47 | |
| *** thumpba has joined #kolla | 20:51 | |
| *** achanda has joined #kolla | 20:52 | |
| *** thumpba has quit IRC | 20:56 | |
| *** thumpba has joined #kolla | 20:59 | |
| *** salv-orl_ has joined #kolla | 21:04 | |
| *** salv-orlando has quit IRC | 21:04 | |
| *** salv-orl_ has quit IRC | 21:08 | |
| *** ssurana has quit IRC | 21:08 | |
| *** salv-orlando has joined #kolla | 21:08 | |
| *** CheKoLyN has joined #kolla | 21:15 | |
| *** tzn has joined #kolla | 21:16 | |
| *** Marga_ has joined #kolla | 21:22 | |
| *** Guest40848 has joined #kolla | 21:23 | |
| openstackgerrit | Merged openstack/kolla: Add an ansible role for mongodb https://review.openstack.org/269393 | 21:25 |
| *** sdake has quit IRC | 21:29 | |
| *** kproskurin has joined #kolla | 21:31 | |
| openstackgerrit | Merged openstack/kolla: Fix Swift for multinode P+ACO deployment https://review.openstack.org/274215 | 21:39 |
| *** inc0 has quit IRC | 21:48 | |
| *** achanda has quit IRC | 21:48 | |
| *** thumpba has quit IRC | 21:54 | |
| *** sdake has joined #kolla | 21:57 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Create nova-api database https://review.openstack.org/274928 | 21:57 |
| sdake | yo | 21:57 |
| SamYaple | yo sdake | 21:59 |
| SamYaple | btw nova has been broken for a week if you werent aware | 21:59 |
| *** achanda has joined #kolla | 21:59 | |
| kproskurin | Hi sdake, could you pls look into heat trusts review? https://review.openstack.org/#/c/236198/ | 22:07 |
| SamYaple | kproskurin: sdakes lab is down. I have tested it but i did not excersize it | 22:08 |
| SamYaple | kproskurin: sdake said if you get asalkeld sign off on the patch we can merge it | 22:08 |
| SamYaple | just let me know and ill +W | 22:08 |
| SamYaple | since asalkeld has +1'd the patch I would assume its good, but i just want to ask the question to him directly first | 22:09 |
| kproskurin | SamYaple, well asalkeld gave his +1 and I did some testing too, for kolla mesos... | 22:09 |
| kproskurin | ok, no problme | 22:09 |
| SamYaple | did you test wait conditions? | 22:09 |
| * SamYaple is not a heat guy, im just repeating sdakes questions | 22:09 | |
| kproskurin | Im not a heat guy too, so I not really sure what wait condition is. I deployed it like 20 times, check some heat cli calls and so on | 22:10 |
| SamYaple | yea i did too | 22:10 |
| SamYaple | apparently thats not enough (or something) so i was going to ask asalkeld about wait conditions and merge | 22:11 |
| kproskurin | Ok, so lets wait for asalkeld response than, no problem | 22:11 |
| SamYaple | fyi kproskurin https://bugs.launchpad.net/kolla-mesos/+bug/1540579 | 22:12 |
| openstack | Launchpad bug 1540579 in kolla-mesos "nova-api -- No sql_connection parameter is established" [Undecided,New] | 22:12 |
| SamYaple | the kolla-ansible fix is here https://review.openstack.org/#/c/274928/ | 22:13 |
| *** inc0 has joined #kolla | 22:13 | |
| SamYaple | the kolla-ansible fix will need to land then you can fix mesos | 22:13 |
| inc0 | sdake, around? | 22:13 |
| SamYaple | yo inc0 | 22:13 |
| sdake | otp but will be soon | 22:13 |
| inc0 | day of meetings \o/ | 22:13 |
| kproskurin | SamYaple, nova is in merge conflict right now, waiting for asalkeld to fix it, so I didnt test it yet for mesos | 22:14 |
| SamYaple | ok kproskurin ill let asalkeld know then by commenting on review | 22:14 |
| kproskurin | ok, tx | 22:15 |
| *** bmace has joined #kolla | 22:18 | |
| *** ssurana has joined #kolla | 22:19 | |
| *** sdake has quit IRC | 22:24 | |
| *** sdake has joined #kolla | 22:26 | |
| sdake | inc0 ok i'm off the phone | 22:26 |
| sdake | whats up | 22:26 |
| *** kproskurin has quit IRC | 22:27 | |
| inc0 | so, how good are these docker folks of yours? because we might have idea | 22:29 |
| inc0 | SamYaple, correct me plz | 22:29 |
| SamYaple | inc0: me correct you? | 22:29 |
| inc0 | but if we kill -9 docker 1.8 vm keeps going | 22:29 |
| SamYaple | pffff | 22:29 |
| inc0 | we end up with pid on host and some data | 22:30 |
| inc0 | in container | 22:30 |
| inc0 | problem is, if we start docker again, it will kill running pids | 22:30 |
| *** achanda has quit IRC | 22:30 | |
| SamYaple | inc0: this may be of interest to you | 22:31 |
| inc0 | so if we explicitly tell docker not to do it, we'll have effectively qemu on host, outside of container | 22:31 |
| SamYaple | "20:35:48 < cpuguy83> SamYaple: As for disowning a process like this, I don't think that will ever be supported... for being able to restart a daemon w/o having to kill all the processes in a container, yes, most definitely that should be supported at some point." | 22:31 |
| SamYaple | cpuguy83 said no | 22:31 |
| SamYaple | already asked this question | 22:31 |
| inc0 | ok, but if we teach docker to, instead of killing, restore containers from pids | 22:31 |
| inc0 | we're in even better position | 22:31 |
| SamYaple | conversation was a bit longer than that, but pid existing outside of docker daemon is a bug. why would they reattach to existing ones rather than kill | 22:32 |
| inc0 | bottom line, docker will kill vms on startup (if we previously kill -9 it), this can't happen | 22:32 |
| SamYaple | (according to them) | 22:32 |
| inc0 | which is way to go in my opinion as well | 22:32 |
| SamYaple | "20:39:08 < cpuguy83> SamYaple: I think it's outside the Docker use-case. You could look at runc for this." | 22:32 |
| SamYaple | again this was asked and rejected. maybe presure will help but i doubt it | 22:33 |
| inc0 | well we are where we are | 22:33 |
| SamYaple | this _is_ outside teh docker use-case | 22:33 |
| inc0 | docker startup kills vm | 22:33 |
| SamYaple | this is the desired affect inc0 | 22:33 |
| inc0 | not by us it's not | 22:33 |
| SamYaple | the only hope I can think of is shared pid-namespace | 22:33 |
| SamYaple | well we can "look at runc for this" ;) | 22:33 |
| SamYaple | inc0: with shared pid namespaces we get "20:40:24 < SamYaple> so instead of host its another docker container holding on the qemu processes like shared volumes_from | 22:35 |
| SamYaple | inc0: sdake see https://github.com/docker/docker/issues/10163 | 22:35 |
| SamYaple | thats the issue that may solve this for us long term | 22:35 |
| SamYaple | so i would think preassure is best applied there | 22:35 |
| SamYaple | i was shutdown pretty hard suggesting the disown thing :/ | 22:36 |
| inc0 | I also thing that might be way to go | 22:36 |
| inc0 | reattaching to running containers will make our job easier | 22:36 |
| SamYaple | shared pid namespace? | 22:37 |
| inc0 | yeah | 22:37 |
| inc0 | with reattach option | 22:37 |
| SamYaple | ok that will work if we can guide development of that feature | 22:37 |
| SamYaple | because it could very well not work too | 22:37 |
| SamYaple | like each container tracks its own pid and there is no ownership sharing | 22:37 |
| inc0 | I agree, it might work, that's where sdake's guys come in | 22:38 |
| inc0 | also there is that: https://github.com/docker/docker/issues/2658 | 22:38 |
| inc0 | this is roadmap, and that implies that docker will have to have some sort of re-attach | 22:38 |
| SamYaple | yea but thats happening | 22:38 |
| SamYaple | reattach to namespaces is different | 22:38 |
| SamYaple | we are talking processes inside the naemspace | 22:39 |
| inc0 | wonderful, we can help in a way so we can laverage reattach | 22:39 |
| inc0 | ok, what I mean is | 22:39 |
| inc0 | to achieve hot restart of docker | 22:39 |
| SamYaple | yea but thats happening. that is supposed to be in 1.11 | 22:39 |
| inc0 | docker daemon after restart needs to be able to reconnect to running containers | 22:39 |
| inc0 | SamYaple, great, I know, it is possible to have it in Mitaka at least in form of RC | 22:39 |
| SamYaple | thats not _inside_ the namespace though, its just not killing the namespace on stop/start | 22:39 |
| inc0 | which is exactly what we need really | 22:40 |
| *** dwalsh has joined #kolla | 22:40 | |
| SamYaple | the pid sharing feature would need to be guided to allow non-watched pids to be transient between the two containers i think | 22:40 |
| SamYaple | thats only part of the issue inc0 | 22:40 |
| SamYaple | the qemu container restart is the bigger one | 22:40 |
| SamYaple | no getting around that one ever | 22:40 |
| inc0 | point is, it potentially might help us | 22:41 |
| asalkeld | o/ | 22:41 |
| SamYaple | docker upgarde is a great solution to a problem. but thats not _the_ problem | 22:41 |
| SamYaple | yo yo asalkeld | 22:41 |
| inc0 | it very well may not, but it can, and we should move heaven so it will | 22:41 |
| SamYaple | asalkeld: if you could just confirm trusts are _actually_ working with this patch ill merge it https://review.openstack.org/#/c/236198/ | 22:41 |
| inc0 | SamYaple, problem for us right now is that docker upgrade causes qemu restart | 22:41 |
| inc0 | upgrades of qemu is different problem I don't want to tackle now | 22:42 |
| SamYaple | inc0: we have different opinions on that. we have a hard stop of qemu for file moves anyway. period | 22:42 |
| inc0 | I want to be able to upgrade openstack with kolla without ever touching qemu | 22:42 |
| asalkeld | SamYaple: looking | 22:42 |
| SamYaple | any upgrade,ever point releases, are going to upgrade nova_libvirt | 22:42 |
| inc0 | with ceph you don't | 22:42 |
| inc0 | not playbook I've written | 22:42 |
| SamYaple | then you aren't doing upgrade and oyu need to add that | 22:43 |
| inc0 | and I will argue against it | 22:43 |
| inc0 | no I don't | 22:43 |
| SamYaple | its a requirement... | 22:43 |
| SamYaple | and you will yes | 22:43 |
| inc0 | I want to do it in separate thing | 22:43 |
| SamYaple | its needed for upgrades | 22:43 |
| inc0 | infra upgrades may be separate | 22:43 |
| SamYaple | its not an infra upgrade. | 22:43 |
| SamYaple | its a container we built | 22:43 |
| SamYaple | it has to be upgraded | 22:43 |
| inc0 | it does not | 22:44 |
| SamYaple | you are absolutely wrong | 22:44 |
| inc0 | it might at some point if nove requires it | 22:44 |
| SamYaple | all containers are on the same base images for a reason | 22:44 |
| inc0 | wonderful, you can just not touch it | 22:44 |
| inc0 | or you can upgrade it PRIOR to openstack upgrade | 22:44 |
| inc0 | or after | 22:44 |
| SamYaple | you can do that if _you_ want, but the default is to upgrade the container | 22:45 |
| inc0 | my point, you dont have to do it at this very moment | 22:45 |
| SamYaple | you do have to do it for the upgrade to be complete | 22:45 |
| inc0 | no, default is 2 separate plays | 22:45 |
| inc0 | I'm -2 on forcing upgrade qemu along with openstack | 22:45 |
| inc0 | but you can do it in separate time-scale | 22:45 |
| SamYaple | you arent going to block upgrade by being stuborn inc0 | 22:45 |
| *** jtriley has quit IRC | 22:45 | |
| inc0 | I'm going to block imposing forced downtime | 22:46 |
| SamYaple | qemu containres must change when we build new base images | 22:46 |
| inc0 | but running ones don't have to be touched | 22:46 |
| *** asalkeld has quit IRC | 22:46 | |
| SamYaple | you cannot unilaterally make a change to what Kolla has always been | 22:46 |
| inc0 | well, upgrades are part of it | 22:46 |
| inc0 | with minimal downtime | 22:47 |
| inc0 | I think less downtime is worth breaking from policy | 22:47 |
| SamYaple | i dont care man, you can argue all day long but we are keeping all containers on the same base images | 22:47 |
| inc0 | if we provide good way to get consistent with it | 22:47 |
| *** asalkeld has joined #kolla | 22:47 | |
| SamYaple | until that happens upgrade is not complete | 22:47 |
| inc0 | at *separate* timescale | 22:47 |
| inc0 | even if that means 6 months | 22:47 |
| SamYaple | nope | 22:48 |
| SamYaple | wont work | 22:48 |
| SamYaple | needs to stay on same base image | 22:48 |
| SamYaple | you are diverging from the same base image | 22:48 |
| inc0 | why is that exactly? | 22:48 |
| asalkeld | :headdesk: updated distro, "do you want to reboot now or later" - /me presses the wrong button - nooo | 22:48 |
| inc0 | having separate requirement space is exactly what contianers give youy | 22:48 |
| SamYaple | you feel free to lock in nova_libvirt to a certain version in your globals.yml, thats why these options exist | 22:48 |
| SamYaple | its not teh default | 22:48 |
| inc0 | thats exactly why you want containers in upgrade | 22:49 |
| inc0 | but I don't have to include redeploy of nova_libvirt in upgrade play | 22:49 |
| inc0 | and I will not include it there | 22:49 |
| inc0 | I will prepare sepratate strategy for this one | 22:49 |
| SamYaple | you will | 22:49 |
| SamYaple | you just need to pin it | 22:49 |
| SamYaple | you don't understand kolla man | 22:50 |
| SamYaple | theres one option for this | 22:50 |
| SamYaple | but the container will have the nova_libvirt start | 22:50 |
| inc0 | upgrade play won't | 22:50 |
| inc0 | nova-upgrade play anyway | 22:50 |
| SamYaple | it will | 22:50 |
| SamYaple | its a requirement | 22:50 |
| inc0 | there will be libvirt upgrade play | 22:50 |
| inc0 | later on, called per node | 22:51 |
| SamYaple | but again you dont understand is my guess | 22:51 |
| SamYaple | ugh you just dont get it | 22:51 |
| SamYaple | if you dont do it in the upgrade play, it will just happen next time you run deploy | 22:51 |
| SamYaple | you just dont understand man | 22:51 |
| inc0 | I know it will | 22:51 |
| inc0 | that's precisely why you won't do this | 22:51 |
| SamYaple | holy crap | 22:51 |
| SamYaple | it doesnt need to be this way | 22:51 |
| SamYaple | we already have the infrastrucute in place to fix the issue you are talking about | 22:52 |
| SamYaple | _you_ need to set one option | 22:52 |
| inc0 | if you want to restart all the vms everytime ANYTHING in base image version of anything change...well | 22:52 |
| SamYaple | jeez dude | 22:52 |
| inc0 | I'm taling about default | 22:52 |
| SamYaple | no | 22:52 |
| inc0 | we have bigger issue then it seems | 22:52 |
| SamYaple | nova_libvirt is upgraded by default | 22:52 |
| SamYaple | you want to change kolla all by yourself | 22:52 |
| SamYaple | thats the issue | 22:52 |
| SamYaple | we have options for a reason | 22:52 |
| inc0 | you did multiple of times | 22:53 |
| SamYaple | you can deploy and manage it the way you want | 22:53 |
| SamYaple | not by myself | 22:53 |
| SamYaple | i got the community behind it first | 22:53 |
| inc0 | let's discuss it on midcycle | 22:53 |
| SamYaple | do you know which option to set to do what you want (AND STILL MAKE deploy WORK!) | 22:53 |
| inc0 | because don't worry, I won't +2 my own patches, so by definition I need community behind this | 22:53 |
| inc0 | yes, but I KNOW KOLLA and 99% people DONT | 22:54 |
| inc0 | people run default | 22:54 |
| inc0 | and they cry | 22:54 |
| inc0 | I don't want to see people cry | 22:54 |
| SamYaple | ok if we are on the same page, what is the option im talking about? | 22:54 |
| inc0 | you can specify image | 22:54 |
| SamYaple | specifcally just the tag... | 22:55 |
| SamYaple | to pin the image... | 22:55 |
| SamYaple | this is why this option exists | 22:55 |
| SamYaple | its for rabbitmq. and mariadb. and all the rest | 22:55 |
| inc0 | if you have doubts that I know this kind of basics, why did you vote me for core in the first place? | 22:55 |
| SamYaple | its why i put the option there in the first place | 22:55 |
| SamYaple | because you want to change the fundementals of kolla and think 2 core members +2 is the "community behind it" | 22:56 |
| inc0 | I think that's why core members have +2 power | 22:56 |
| SamYaple | that doesnt mean the community is behind it | 22:56 |
| SamYaple | thats my point | 22:56 |
| inc0 | how many +2 it takes to "have community behind"? | 22:56 |
| inc0 | I'm willing to ask cores to wait with +2 for that many _2 | 22:57 |
| inc0 | +2s | 22:57 |
| SamYaple | you said ive changed kolla many times yes? the biggest was the ansible change that i brought form yaodu before your time | 22:57 |
| SamYaple | that required a spec | 22:57 |
| SamYaple | go see the "community behind" that one | 22:57 |
| SamYaple | thats what im talking about | 22:57 |
| SamYaple | this beviour is in place for a reason | 22:57 |
| inc0 | but we did agree for spec-less process | 22:57 |
| inc0 | I actually would like to reintroduce specs | 22:57 |
| SamYaple | you are changing that basic behaviour under the radar man | 22:58 |
| SamYaple | you need to for this | 22:58 |
| inc0 | because forced upgrade we found ourselves in, I hate that | 22:58 |
| inc0 | no, I simply add new playbook | 22:58 |
| inc0 | deploy will stay EXACTLY the same | 22:58 |
| inc0 | and will restart VMs every time base image got rebuilt | 22:58 |
| SamYaple | just submit a patch so i can -1 you there. you arent going to change this behaviour with jsut two cores approval | 22:59 |
| inc0 | which means you really don't want to re-run deploy now when I think about it | 22:59 |
| SamYaple | i jsut said that | 22:59 |
| SamYaple | glad you are catching up | 22:59 |
| inc0 | which renders your argument invalid | 23:00 |
| SamYaple | you are wrong | 23:00 |
| inc0 | because I don't care about deploy play | 23:00 |
| inc0 | upgrade play MAY not touch nova_libvirt at all | 23:00 |
| inc0 | it wont unless I write task for it, which I won't | 23:00 |
| SamYaple | it has to to be complete | 23:00 |
| SamYaple | otherwise when you rerun deploy you change things | 23:00 |
| SamYaple | thats bad | 23:00 |
| inc0 | reruning deploy at all will break stuff unless you don't touch your images at all | 23:01 |
| SamYaple | no, you should be able to run upgrade, then deploy and deploy change nothing | 23:01 |
| SamYaple | thats the whole point | 23:01 |
| SamYaple | upgrade is finished | 23:01 |
| inc0 | which basically mean that I'll put huge red letter "don't run deploy unless you a. upgraded openstack and b. upgraded infra underneath" | 23:01 |
| SamYaple | or just good documentation on the deploy process (like every other openstack project does with release notes) | 23:02 |
| *** CheKoLyN has quit IRC | 23:02 | |
| *** dwalsh has quit IRC | 23:02 | |
| inc0 | admins don't care if upgrade is finished in 3 months if that what it takes, unless cloud stays up all the time | 23:02 |
| SamYaple | if you want to pin nova_libvirt to prevent undesireable behaviour here is how "set one option in globals.yml" | 23:02 |
| inc0 | so they can upgrade services | 23:02 |
| inc0 | and then qemu one by one | 23:02 |
| * SamYaple sigh | 23:03 | |
| inc0 | I'm not pinning nova libvirt, I'm just not upgrading it in same play as nova_compute | 23:03 |
| SamYaple | then you don't have a competeled upgrade | 23:03 |
| SamYaple | because as soon as you run deploy you are screwed | 23:03 |
| SamYaple | and thats what operators will do | 23:03 |
| inc0 | I'm out, cya | 23:03 |
| *** inc0 has quit IRC | 23:03 | |
| *** unicell1 has joined #kolla | 23:09 | |
| *** unicell has quit IRC | 23:10 | |
| *** salv-orlando has quit IRC | 23:13 | |
| *** salv-orlando has joined #kolla | 23:13 | |
| *** tzn has quit IRC | 23:18 | |
| *** alisonh has quit IRC | 23:26 | |
| SamYaple | asalkeld: did you say that the heat patch was good? i lost the scrollback | 23:32 |
| asalkeld | SamYaple: i haven't tested | 23:32 |
| SamYaple | ok | 23:33 |
| SamYaple | well just ping me when you say its solid and ill merge | 23:33 |
| asalkeld | i am just busy beating a patch into submission | 23:33 |
| SamYaple | no rush | 23:33 |
| SamYaple | asalkeld: i left a comment on your nova mesos patch. you need the nova database and the nova_api database (thats a thing!) | 23:34 |
| asalkeld | ok, thanks | 23:34 |
| *** thumpba has joined #kolla | 23:34 | |
| *** stvnoyes has quit IRC | 23:40 | |
| *** stvnoyes has joined #kolla | 23:41 | |
| *** tzn has joined #kolla | 23:44 | |
| *** tzn has quit IRC | 23:45 | |
| *** alisonh has joined #kolla | 23:45 | |
| *** sdake has quit IRC | 23:48 | |
| *** blahRus1 has quit IRC | 23:52 | |
| *** jeh has quit IRC | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!