| *** jmccarthy has quit IRC | 00:15 | |
| *** jmccarthy has joined #kolla | 00:15 | |
| *** Jeffrey4l has joined #kolla | 00:20 | |
| *** salv-orlando has quit IRC | 00:25 | |
| *** aginwala has quit IRC | 00:29 | |
| *** aginwala has joined #kolla | 00:32 | |
| *** aginwala has quit IRC | 00:33 | |
| *** aginwala has joined #kolla | 00:33 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Do not pull rabbitmq-data image https://review.openstack.org/278851 | 00:39 |
|---|---|---|
| *** jasonsb has joined #kolla | 00:39 | |
| *** iNeilus has joined #kolla | 00:46 | |
| *** iNeilus has quit IRC | 00:52 | |
| mandre | sbezverk, the cpio cap_set_file error makes me think of an issue with your docker storage driver | 00:52 |
| mandre | are you using aufs by any chance? | 00:53 |
| mandre | I suggest you switch to overlayfs or btrfs. devicemapper is the default on centos I believe, it's known to work well too. | 00:56 |
| mandre | don't bother with a regex when building images, just set the build profile to default like this: "kolla-build -b centos -t binary --profile default" | 00:57 |
| mandre | this way it will only build the images that are required for a kolla deploy and will save you a lot of build time | 00:58 |
| mandre | d_code, ^ this is for you too | 00:58 |
| mandre | d_code: you generate the kolla-build.conf with "tox -e genconfig" | 01:02 |
| *** ayoung has joined #kolla | 01:02 | |
| mandre | http://docs.openstack.org/developer/kolla/image-building.html | 01:02 |
| openstackgerrit | Ruslan Kamaldinov proposed openstack/kolla: Output image statuses to log instead of return value of main function https://review.openstack.org/278853 | 01:04 |
| mandre | d_code, sbezverk: after you've build the image you can do a "kolla-ansible prechecks" to ensure your environment meets the requirements for deploying kolla | 01:08 |
| *** aginwala has quit IRC | 01:10 | |
| *** aginwala has joined #kolla | 01:14 | |
| *** aginwala has quit IRC | 01:15 | |
| *** aginwala has joined #kolla | 01:15 | |
| *** dolpher has quit IRC | 01:17 | |
| *** dolpher has joined #kolla | 01:23 | |
| openstackgerrit | Naren Narendra proposed openstack/kolla: Clarify Ansible installation for distros in docs. https://review.openstack.org/279356 | 01:33 |
| *** ssurana has quit IRC | 01:35 | |
| openstackgerrit | Naren Narendra proposed openstack/kolla: Clarify Ansible installation for distros in docs. https://review.openstack.org/279356 | 01:36 |
| *** jtriley has joined #kolla | 01:37 | |
| openstackgerrit | Merged openstack/kolla: Make pep8 *the* linting interface https://review.openstack.org/278616 | 01:38 |
| *** tzn has quit IRC | 01:39 | |
| SamYaple | mandre: can you just merge this in? d11678a09ef5 registry:8182/kollaglue/ubuntu-source-glance-api:2.0.0 "kolla_start" 7 hours ago Up 16 minutes glance_api | 01:41 |
| SamYaple | oops | 01:41 |
| SamYaple | mandre: https://review.openstack.org/#/c/277184/ | 01:41 |
| SamYaple | its been rechecked for a while | 01:41 |
| SamYaple | its solid, but normal gate issue in play | 01:41 |
| openstackgerrit | Naren Narendra proposed openstack/kolla: Clarify Ansible installation for distros in docs. https://review.openstack.org/279356 | 01:41 |
| *** alisonh has quit IRC | 01:42 | |
| *** aginwala has quit IRC | 01:43 | |
| mandre | SamYaple, ok if you merge https://review.openstack.org/#/c/275573/ | 01:43 |
| mandre | Jeffrey4l got his br-ex patch working | 01:44 |
| *** sdake_ is now known as sdake | 01:47 | |
| SamYaple | mandre: merged | 01:48 |
| SamYaple | now you! | 01:48 |
| sdake | sup nerds | 01:48 |
| sdake | can someone merge that centos doc patch plz | 01:48 |
| SamYaple | link | 01:48 |
| sdake | so this br-ex patch, jeffrey has fixed it ? | 01:48 |
| SamYaple | sdake: he put in a proper wait, seems to resolve issue | 01:49 |
| sdake | it just scrolled sam | 01:49 |
| sdake | ok wfm | 01:49 |
| sdake | as long as its fixed :) | 01:49 |
| openstackgerrit | Merged openstack/kolla: Ensuring the openvswitch_db is ready before creating bridges https://review.openstack.org/275573 | 01:49 |
| sdake | i didn't have a chance to look at the review queue today - been busy with first day back from travel nonsense | 01:50 |
| *** aginwala has joined #kolla | 01:50 | |
| openstackgerrit | Merged openstack/kolla: Use Infra provided mirrors in gate https://review.openstack.org/277184 | 01:51 |
| sdake | and debugging | 01:53 |
| sdake | 7pm time to ptfo | 01:53 |
| sdake | so tierd | 01:53 |
| mandre | so is registry:v1 broken with docker 1.10? | 01:54 |
| sdake | tx SamYaple | 01:54 |
| sdake | mandre roger | 01:54 |
| sdake | mandre i got soemone rewriting the docs atm | 01:54 |
| sdake | patch shoudl hit tomorrow | 01:54 |
| sdake | for that problem | 01:54 |
| mandre | good sdake. docs need some love for sure. | 01:55 |
| mandre | i | 01:55 |
| mandre | i've seen the midcycle has been productive | 01:55 |
| sdake | not rewriting the docs | 01:55 |
| sdake | just that registry section | 01:55 |
| sdake | since its out of date | 01:55 |
| sdake | I sent em a mail and they said htey would do the job | 01:55 |
| sdake | ya we rocked it at the midcycle | 01:56 |
| sdake | got almost all openstack service updates done | 01:56 |
| mandre | excellent! what is left? | 01:56 |
| sdake | heat, working on | 01:56 |
| sdake | all infrastructure components | 01:56 |
| SamYaple | neutron working on it | 01:56 |
| sdake | we have a full plan and docs on how to do infra | 01:56 |
| sdake | oh right and neutron | 01:56 |
| sdake | i did work on heat, I finished the domain users | 01:57 |
| sdake | but haven't had a chnace ot test | 01:57 |
| sdake | so not really finsihed | 01:57 |
| sdake | need the second patch on top of it to test it really | 01:57 |
| sdake | mandre this is all work that needs to be done | 01:58 |
| sdake | https://etherpad.openstack.org/p/kolla-mitaka-midcycle-infrastructure-upgrades | 01:58 |
| mandre | did you confirm docker is not killing the VMs during upgrade? | 01:58 |
| *** shakamunyi has joined #kolla | 01:58 | |
| sdake | we are punting that to redhat to sort out I think | 01:58 |
| sdake | but there was some analysis at the midcycle about that | 01:59 |
| sdake | docker cannot keep track of cgroups libvirt creates | 01:59 |
| *** daneyon has joined #kolla | 01:59 | |
| mandre | yeah, had a look at the etherpad... looks good | 01:59 |
| sdake | suspicion is libvirt changed how they manage qemu cgroups | 01:59 |
| SamYaple | sdake: thats not what it was | 02:00 |
| sdake | oh did that get root caused? | 02:01 |
| SamYaple | oh wait you are refering to qemu vs kvm | 02:01 |
| sdake | yes | 02:01 |
| SamYaple | yea qemu is the hpervisor with kvm extentions | 02:01 |
| sdake | i didn't know it was a qemu vs kvm thing | 02:01 |
| SamYaple | when running with kvm extentions it is in a different cgroup | 02:01 |
| sdake | i thought i t was a all libvirt qemu processes bust | 02:01 |
| SamYaple | when running with tcg (software) then its killed by docker | 02:02 |
| sdake | without kvm extensions same? | 02:02 |
| *** daneyon_ has quit IRC | 02:02 | |
| SamYaple | with kvm everything is a-ok | 02:02 |
| sdake | nice | 02:02 |
| SamYaple | software only its borked | 02:02 |
| *** alisonh has joined #kolla | 02:02 | |
| sdake | when was tha figured out | 02:02 |
| SamYaple | days before the midcycle | 02:02 |
| SamYaple | and reiterated during | 02:02 |
| mandre | thanks for the explanation SamYaple | 02:03 |
| sdake | ya that was during our pair programming | 02:03 |
| mandre | this is reassuring | 02:03 |
| sdake | ok i'm off for the night guys | 02:03 |
| SamYaple | night sdake | 02:03 |
| mandre | night sdake | 02:03 |
| SamYaple | me and paul are about to ekko it up | 02:03 |
| sdake | super beat, up since 6 am, walked both ways uphill to work today, etc | 02:03 |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Fix non-root deploys https://review.openstack.org/276887 | 02:08 |
| *** shakamunyi has quit IRC | 02:12 | |
| *** cloudnau_ has joined #kolla | 02:15 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Fix detect_distro https://review.openstack.org/279363 | 02:16 |
| *** cloudnau_ has quit IRC | 02:20 | |
| *** tzn has joined #kolla | 02:23 | |
| *** aginwala has quit IRC | 02:26 | |
| *** daneyon has quit IRC | 02:29 | |
| *** aginwala has joined #kolla | 02:30 | |
| *** daneyon has joined #kolla | 02:30 | |
| *** shakamunyi has joined #kolla | 02:32 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Fix detect_distro https://review.openstack.org/279363 | 02:32 |
| *** daneyon has quit IRC | 02:35 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Fix detect_distro https://review.openstack.org/279363 | 02:36 |
| *** sdake has quit IRC | 02:40 | |
| *** sdake has joined #kolla | 02:44 | |
| *** salv-orlando has joined #kolla | 02:53 | |
| *** dave-mccowan has quit IRC | 02:54 | |
| *** salv-orlando has quit IRC | 02:55 | |
| *** aginwala has quit IRC | 02:57 | |
| *** aginwala has joined #kolla | 03:00 | |
| *** aginwala has quit IRC | 03:02 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Fix detect_distro https://review.openstack.org/279363 | 03:02 |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Fix non-root deploys https://review.openstack.org/276887 | 03:02 |
| openstackgerrit | Merged openstack/kolla: Clarify Ansible installation for distros in docs. https://review.openstack.org/279356 | 03:09 |
| *** dims has quit IRC | 03:11 | |
| *** dims has joined #kolla | 03:11 | |
| *** iNeilus has joined #kolla | 03:29 | |
| *** dims has quit IRC | 03:31 | |
| *** iNeilus has quit IRC | 03:33 | |
| *** dave-mccowan has joined #kolla | 03:35 | |
| *** sdake has quit IRC | 03:37 | |
| *** achanda has joined #kolla | 03:39 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Fix detect_distro https://review.openstack.org/279363 | 03:39 |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Fix non-root deploys https://review.openstack.org/276887 | 03:39 |
| *** unicell has quit IRC | 03:40 | |
| *** achanda has quit IRC | 03:41 | |
| *** sdake has joined #kolla | 03:44 | |
| *** achanda has joined #kolla | 03:46 | |
| *** sdake has quit IRC | 04:03 | |
| *** dave-mccowan has quit IRC | 04:28 | |
| *** salv-orlando has joined #kolla | 04:37 | |
| *** salv-orlando has quit IRC | 04:42 | |
| *** alyson_ has quit IRC | 05:09 | |
| *** alyson_ has joined #kolla | 05:09 | |
| *** Slower has quit IRC | 05:11 | |
| *** Slower has joined #kolla | 05:11 | |
| *** iNeilus has joined #kolla | 05:17 | |
| *** Jeffrey4l has quit IRC | 05:19 | |
| *** iNeilus has quit IRC | 05:21 | |
| *** aginwala has joined #kolla | 05:25 | |
| *** achanda has quit IRC | 05:38 | |
| *** Jeffrey4l has joined #kolla | 05:41 | |
| *** tzn has quit IRC | 05:44 | |
| *** unicell has joined #kolla | 05:52 | |
| *** salv-orlando has joined #kolla | 05:54 | |
| *** salv-orlando has quit IRC | 05:56 | |
| *** achanda has joined #kolla | 05:57 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix image plugin functionality for oslo.config https://review.openstack.org/268211 | 06:00 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: fix the custome profile raise exception issue https://review.openstack.org/272022 | 06:00 |
| *** aginwala has quit IRC | 06:01 | |
| *** aginwala has joined #kolla | 06:04 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Make pep8 *the* linting interface https://review.openstack.org/279384 | 06:07 |
| *** achanda has quit IRC | 06:09 | |
| *** achanda has joined #kolla | 06:11 | |
| *** achanda has quit IRC | 06:20 | |
| *** achanda has joined #kolla | 06:21 | |
| *** aginwala has quit IRC | 06:25 | |
| *** achanda has quit IRC | 06:28 | |
| *** opennode has joined #kolla | 06:33 | |
| *** aginwala has joined #kolla | 06:35 | |
| *** tzn has joined #kolla | 06:40 | |
| *** aginwala has quit IRC | 06:58 | |
| *** aginwala has joined #kolla | 07:00 | |
| *** iNeilus has joined #kolla | 07:05 | |
| *** iNeilus has quit IRC | 07:10 | |
| *** salv-orlando has joined #kolla | 07:12 | |
| *** tzn has quit IRC | 07:14 | |
| *** salv-orlando has quit IRC | 07:23 | |
| ajafo | o/ | 07:46 |
| *** opennode has quit IRC | 07:51 | |
| elemoine_ | o/ | 07:53 |
| elemoine_ | ajafo: how is going? | 07:53 |
| ajafo | still fight with kolla to make it working :) | 07:53 |
| ajafo | but got some ideas so I'll test it today :) | 07:54 |
| ajafo | and what about you? | 07:54 |
| elemoine_ | good thanks | 07:56 |
| *** achanda has joined #kolla | 07:58 | |
| *** fgimenez has joined #kolla | 08:03 | |
| *** tzn has joined #kolla | 08:11 | |
| *** opennode has joined #kolla | 08:15 | |
| *** aginwala has quit IRC | 08:17 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Remove the openstack_release option in the globals.yml file https://review.openstack.org/274408 | 08:20 |
| *** opennode has quit IRC | 08:21 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Remove the openstack_release option in the globals.yml file https://review.openstack.org/274408 | 08:21 |
| *** athomas has joined #kolla | 08:25 | |
| elemoine_ | Jeffrey4l: can I easily reproduce https://bugs.launchpad.net/kolla/+bug/1544545? I'd like to understand when/how this problem occurs. | 08:27 |
| openstack | Launchpad bug 1544545 in kolla "kolla-toolbox container failed and stick on create stage" [Undecided,In progress] - Assigned to Jeffrey Zhang (jeffrey4l) | 08:27 |
| elemoine_ | thanks | 08:27 |
| *** akwasnie has joined #kolla | 08:28 | |
| *** salv-orlando has joined #kolla | 08:28 | |
| Jeffrey4l | elemoine_, yes. I am writing on the lauchpad. I will ping back you when i finished. | 08:28 |
| elemoine_ | ok, sorry for insisting | 08:28 |
| Jeffrey4l | np | 08:29 |
| elemoine_ | I will run in the same kind of problems with Heka and the log Unix socket, so I want to fully understand this | 08:29 |
| Jeffrey4l | got it. | 08:30 |
| *** aginwala has joined #kolla | 08:31 | |
| *** gfidente has joined #kolla | 08:33 | |
| *** gfidente has joined #kolla | 08:33 | |
| *** macsz has joined #kolla | 08:34 | |
| *** salv-orlando has quit IRC | 08:35 | |
| *** aginwala has quit IRC | 08:36 | |
| *** mbound has joined #kolla | 08:38 | |
| *** tzn has quit IRC | 08:44 | |
| *** jmccarthy1 has quit IRC | 08:48 | |
| elemoine_ | do we have unit tests for kolla_docker.py? | 08:53 |
| *** jmccarthy1 has joined #kolla | 08:55 | |
| Jeffrey4l | elemoine_, ^ check it. | 08:56 |
| elemoine_ | thanks, I'll have a look | 08:57 |
| Jeffrey4l | we have no unittest for kolla_docker.py file now. But we should and need add ut for it. | 08:58 |
| elemoine_ | ok | 08:58 |
| elemoine_ | I found a bug in kolla_docker.py and I was wondering if I should write tests | 08:58 |
| Jeffrey4l | elemoine_, you can just fix the bug only. But it will be perfect if you can add some ut for this. | 09:00 |
| openstackgerrit | Alicja Kwasniewska proposed openstack/kolla: Added Elasticearch and its deployment. https://review.openstack.org/267714 | 09:05 |
| *** salv-orlando has joined #kolla | 09:05 | |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Fix kolla_docker check_volume https://review.openstack.org/279429 | 09:06 |
| elemoine_ | Jeffrey4l: fix submitted | 09:07 |
| *** sbezverk has quit IRC | 09:11 | |
| *** sbezverk has joined #kolla | 09:12 | |
| *** aginwala has joined #kolla | 09:14 | |
| *** aginwala has quit IRC | 09:18 | |
| *** achanda has quit IRC | 09:27 | |
| *** Jeffrey4l has quit IRC | 09:28 | |
| *** kproskurin has joined #kolla | 09:36 | |
| kproskurin | Guys, looks like we have a problem with compatibility between kolla and kolla-mesos. Kolla *always* runs kolla_extend_start: https://github.com/openstack/kolla/blob/master/docker/base/start.sh and we cant workaround it in any way. And, for example, in kolla_extend_start of horizon we have a md5 sum thingy with horizon configs. But in kolla-mesos we dont have any configs yet, they about to be created after. | 09:39 |
| kproskurin | I’d really like to see this kolla_extend_start somehow triggered by ansible var or something, so we could skip it | 09:40 |
| *** unicell has quit IRC | 09:47 | |
| *** neilus has joined #kolla | 09:47 | |
| openstackgerrit | Marek Zawadzki proposed openstack/kolla: Added note about root permissions, removed sudos. https://review.openstack.org/278532 | 09:48 |
| openstackgerrit | Marek Zawadzki proposed openstack/kolla: Added note about root permissions, removed sudos. https://review.openstack.org/278532 | 09:51 |
| *** salv-orlando has quit IRC | 09:52 | |
| *** neilus has quit IRC | 09:52 | |
| *** sdake has joined #kolla | 09:57 | |
| *** sdake_ has joined #kolla | 10:00 | |
| *** rhallisey has quit IRC | 10:01 | |
| *** sdake has quit IRC | 10:03 | |
| *** salv-orlando has joined #kolla | 10:03 | |
| *** athomas has quit IRC | 10:20 | |
| sdake_ | morning | 10:25 |
| *** sdake_ is now known as sdake | 10:25 | |
| *** openstackgerrit has quit IRC | 10:32 | |
| *** openstackgerrit has joined #kolla | 10:32 | |
| *** tzn has joined #kolla | 10:34 | |
| *** aginwala has joined #kolla | 10:35 | |
| *** aginwala has quit IRC | 10:40 | |
| *** salv-orl_ has joined #kolla | 10:41 | |
| *** dims has joined #kolla | 10:41 | |
| *** akwasnie has quit IRC | 10:42 | |
| *** salv-orlando has quit IRC | 10:45 | |
| kproskurin | sdake: hi | 10:48 |
| ajafo | does neutron and nova should use v2 or v3 in communication with keystone? | 10:49 |
| ajafo | sdake: morning | 10:49 |
| kproskurin | sdake: I found a silly problem with kolla vs kolla-mesos compatibility. Kolla *always* runs kolla_extend_start: https://github.com/openstack/kolla/blob/master/docker/base/start.sh and we cant workaround it in any way. And, for example, in kolla_extend_start of horizon we have a md5 sum thingy with horizon configs. But in kolla-mesos we dont have any configs yet, they about to be created after. | 10:49 |
| openstackgerrit | Steven Dake proposed openstack/kolla-mesos: Remove a hanging whitespace https://review.openstack.org/279465 | 10:50 |
| openstackgerrit | Merged openstack/kolla-mesos: Make pep8 *the* linting interface https://review.openstack.org/278619 | 10:50 |
| sdake | ok folks | 10:50 |
| sdake | 3am | 10:50 |
| sdake | give me a minute to get my day rolling | 10:50 |
| sdake | woke up early for some reason today | 10:50 |
| elemoine_ | 3am!? | 10:50 |
| kproskurin | %-) | 10:50 |
| kproskurin | “a bit early” | 10:50 |
| sdake | probably has something to do with passing out at 6pm last night from too much travel :) | 10:50 |
| elemoine_ | not a good time for bothering sdake with questions I guess :) | 10:51 |
| sdake | time is ok just dont expect brilliance ;) | 10:51 |
| sdake | so extend start | 10:51 |
| sdake | is always meant to be run | 10:51 |
| sdake | it is meant to be overridden per container from the main start script | 10:52 |
| sdake | the problem kolla-mesos has is what, it is doing some work on configs that are not present yet? | 10:52 |
| sdake | did I hear that right | 10:52 |
| kproskurin | Yep, we will fetch configs from ZK after, from kolla_mesos_start | 10:52 |
| sdake | ajafo all of kolla is keystone v3, but nova should be using keystone v3 with neutron auth, but I'll double check the default configs - moment | 10:53 |
| kproskurin | And sometimes we maybe want to have a bit different bootstrap of our own, but right now we forced to always run kollas first :-) | 10:53 |
| sdake | so I believe at one time we agreed in custom bootstrap provided by the infrastructure of choice | 10:54 |
| sdake | working in the same way as config-external works now | 10:54 |
| sdake | in mesos case, we sort of do config-internal there, but its the same concept | 10:54 |
| ajafo | sdake: thx, I've little problems in ubuntu it's looks like neutron/nova try to communicate with v2 and I don't know why at this moment | 10:55 |
| sdake | so we agreed, but there wasn't a pressing need to do the job | 10:55 |
| kproskurin | Well, horizon broke things for us right now | 10:55 |
| sdake | ajafo do you mean when nova launches a vm it communicates with nova in keystone v2? | 10:55 |
| sdake | kproskurin the answer is a prototype implementation | 10:56 |
| sdake | the works like config-external | 10:56 |
| sdake | except with the extend_start script | 10:56 |
| sdake | i've got an awful lot on my plte but I could prototype it for you, then you could port the tree | 10:56 |
| ajafo | sdake: no, when I try to login to horizon it tries to communicate with nova and neutron, and then I see in neutron and nova logs that it use v2 links and got errors | 10:56 |
| sdake | I think finishing by the 4th will be difficult | 10:56 |
| sdake | ajafo roger, so horizon is using keystone v2 then? | 10:57 |
| sdake | can you nova boot a vm? | 10:57 |
| sdake | keystone v2 has been removed from opentack iiuc | 10:57 |
| ajafo | but horizon as I see in logs is using v3 it's why I'm little confused | 10:57 |
| sdake | let me double check configs | 10:58 |
| sdake | second pulling repo | 10:58 |
| kproskurin | sdake: I not 100% sure what are you talking about with “config-external” thingy. Maybe I missed this disscussion. Im really open to any ideas about this situation. I’d liek to hear nihilifer opinion about this too | 10:58 |
| sdake | ajafo can you give me a few minutes to work this out with kproskurin, then I'll be all ours? | 10:58 |
| sdake | yours? | 10:58 |
| ajafo | I'm trying to setup centos env to compare environments my suspection is about api-paste.ini | 10:59 |
| ajafo | sdake: no problem, take your time just asked :) | 10:59 |
| ajafo | and thanks :) | 10:59 |
| sdake | https://github.com/openstack/kolla/blob/master/ansible/roles/heat/templates/heat-engine.json.j2 | 11:00 |
| sdake | kproskurin read that file please | 11:00 |
| kproskurin | sdake: done... | 11:01 |
| sdake | what happens on bootstrap is ansible reads that file and copies all the stuff in config_files | 11:01 |
| sdake | the proposal we had at the tokyo summit was to add extend_start to that file so it could be externally bootstrapped | 11:01 |
| sdake | and the tool thatcopies that file inside the vm would run the extend_start code | 11:01 |
| kproskurin | sdake: sounds good for me | 11:02 |
| sdake | asalkeld asked for it, we agreed to it, nobody ever did the work | 11:02 |
| kproskurin | so it will be ansible-onlu thing | 11:02 |
| sdake | thats where its at at the moment | 11:02 |
| sdake | no, kolla-mesos also uses these json files I think | 11:02 |
| sdake | just copies them from a different source | 11:02 |
| sdake | atleast kolla-mesos should be using these json | 11:03 |
| kproskurin | We copy it in a bit different way, so we copy all tmpl from kolla, but populate them via kolla_mesos_start | 11:03 |
| kproskurin | so we dont have this jsons | 11:03 |
| kproskurin | only files from this jsons | 11:03 |
| kproskurin | https://github.com/openstack/kolla-mesos/blob/master/services/mariadb/mariadb.yml.j2 | 11:04 |
| kproskurin | As you can see, we run bootstrap here too, and will this kolla one we run bootstrap twice %-) | 11:05 |
| sdake | kproskurin what is this https://github.com/openstack/kolla-mesos/blob/master/services/mariadb/mariadb.yml.j2#L13 | 11:05 |
| sdake | what is the meaning of bootstrap: https://github.com/openstack/kolla-mesos/blob/master/services/mariadb/mariadb.yml.j2#L11 | 11:06 |
| kproskurin | It’s our call of bootstrap script. So then kolla-mesos-start starts, it reads this config, and run all commands by their dependancies. So mariadb need mariadb/bootstrap first, so it look this bootstrap section and run this script with this env. So we litteraly run bootstraping twice in kolla-mesos. First time via start.sh from kolla, and second time via our start script. | 11:07 |
| kproskurin | I just found that yesterday | 11:07 |
| sdake | that needs to be fixed | 11:08 |
| kproskurin | Thats why im here :-) | 11:08 |
| sdake | completely unsuitable solution | 11:08 |
| sdake | so it runs this bootstrap section one time right? | 11:09 |
| kproskurin | yes | 11:09 |
| *** akwasnie has joined #kolla | 11:09 | |
| kproskurin | with run_once: True | 11:09 |
| sdake | this yml file is run by ansible? | 11:09 |
| sdake | or this is anguss custom config script like the json above i linked | 11:09 |
| kproskurin | So if you move this extend_start to this json thing, it will be totaly ok for us if I understand everything right | 11:09 |
| kproskurin | this file is not run by ansible at all | 11:10 |
| sdake | you use an ansible namespace name run_once | 11:10 |
| sdake | why I ask :) | 11:10 |
| kproskurin | it stored in ZK node and fetched by our start script | 11:10 |
| sdake | does kolla-mesos need start.sh overridden as well? | 11:10 |
| kproskurin | Well, start.sh fetch our start script and runs it - all ok. But in the middle of it it runs extend_start which is not ok. :-) | 11:11 |
| sdake | got a link to the code that runs extend_start | 11:12 |
| sdake | kproskurin file a blueprint please | 11:12 |
| sdake | call it "custom-extend-start" | 11:12 |
| kproskurin | https://github.com/openstack/kolla/blob/master/docker/base/start.sh#L12 | 11:12 |
| sdake | not sure if we can delive rby the 4th got alot on our plate | 11:12 |
| kproskurin | Ok, but I think I should disscuss it with nihilifer first, not sure if he will be online today | 11:13 |
| kproskurin | Since he is core in both projects | 11:13 |
| sdake | kolla-mesos bootstrap code is running kolla_extend_start? | 11:13 |
| sdake | its one project - kolla | 11:14 |
| sdake | two repos | 11:14 |
| kproskurin | Ok sry. We could run extend_start or any other script if we want. | 11:14 |
| kproskurin | And call it bootstrap or anything | 11:14 |
| kproskurin | it’s just a chain of commands with deps | 11:14 |
| sdake | if you look here: | 11:15 |
| kproskurin | but yes 95% of the time its just a kolla bootstrap | 11:15 |
| sdake | https://github.com/openstack/kolla/blob/master/docker/base/start.sh#L7 | 11:15 |
| sdake | lets talk about general case not bootstrap | 11:15 |
| sdake | bootstrap is wrong way to look at it | 11:15 |
| sdake | there are othe rcases where a custom kolla_extend_start is needed | 11:15 |
| sdake | so line 7 | 11:16 |
| sdake | what does that in kolla-mesos? | 11:16 |
| sdake | or does that kolla_set_configs contain mesos's bootstrap codebase as well? | 11:16 |
| kproskurin | Line 7 is kolla_set_configs. We launch this container with this: KOLLA_CONFIG={ "command": "kolla_mesos_start", "config_files": [ { "source": "zk://172.20.9.27:2181/kolla/common/kolla_mesos_start.py", "dest": "/usr/local/bin/kolla_mesos_start", "owner": "root", "perm": "0755" } ]} | 11:16 |
| kproskurin | So line 7 fetch out start script from ZK node | 11:16 |
| kproskurin | it’s all ok | 11:17 |
| kproskurin | After at line 12 it runs kolla bootstrap and we cant avoid it. And in the like 15 it launch our start script | 11:17 |
| kproskurin | our start scrip fetch configs related to this app | 11:17 |
| kproskurin | And run commands | 11:17 |
| kproskurin | We add bootstrap commands in most of the apps, since we missed what its allready forced running in start.sh | 11:18 |
| kproskurin | So we run them twice | 11:18 |
| sdake | give me 5 minutes to process all that | 11:19 |
| kproskurin | Solution what I like is to move this extend_start from start.sh to somethere else, there only ansible run could do it. So we could use our own calls of bootstrap(using kolla scrips in 95% of cases and ouw own if needed) | 11:19 |
| kproskurin | Maybe kolla_start should run extend_start first for example | 11:20 |
| sdake | kproskurin you said line 7 of kolla_set_configs launches the container, I'm looking and line 7 of kolaset_configs nd its a comment | 11:21 |
| kproskurin | no, I said line 7 of start.sh fetch our start script | 11:21 |
| sdake | https://github.com/openstack/kolla/blob/master/docker/base/set_configs.py#L7 | 11:22 |
| kproskurin | https://github.com/openstack/kolla/blob/master/docker/base/start.sh#L7 | 11:22 |
| kproskurin | :-) | 11:22 |
| sdake | ok well break it down one more level for me please | 11:22 |
| kproskurin | kproskurin: Line 7 is kolla_set_configs. We launch this container with this: KOLLA_CONFIG={ "command": "kolla_mesos_start", "config_files": [ { "source": "zk://172.20.9.27:2181/kolla/common/kolla_mesos_start.py", "dest": "/usr/local/bin/kolla_mesos_start", "owner": "root", "perm": "0755" } ]} | 11:22 |
| sdake | ya you dont have to cut and paste | 11:22 |
| sdake | what I eman is where in the set-configs is said container launched | 11:22 |
| sdake | 4:20am... still booting | 11:22 |
| kproskurin | my english is not perfect too, sry, maybe I descripe it bad | 11:23 |
| sdake | your doing well | 11:23 |
| sdake | keep at it | 11:23 |
| kproskurin | Could you pls tell me what confuses you about all this? | 11:23 |
| sdake | so you gave me abunch of detail about whwat line 7 does | 11:23 |
| kproskurin | And I will go in detail about it | 11:23 |
| kproskurin | ok, 1 min | 11:23 |
| sdake | but line 7 launches set_configs.py | 11:24 |
| sdake | if I could get some links there, that would be helpful | 11:24 |
| sdake | (in docker/base directory) | 11:24 |
| kproskurin | So kolla has start.sh in base container. And line 7 of this script https://github.com/openstack/kolla/blob/master/docker/base/start.sh#L7 is call of kolla_set_config. | 11:24 |
| kproskurin | We run each container with this env: KOLLA_CONFIG={ "command": "kolla_mesos_start", "config_files": [ { "source": "zk://172.20.9.27:2181/kolla/common/kolla_mesos_start.py", "dest": "/usr/local/bin/kolla_mesos_start", "owner": "root", "perm": "0755" } ]} | 11:24 |
| kproskurin | So kolla_set_config is fetch our kolla_mesos_start script | 11:25 |
| kproskurin | All ok for now | 11:25 |
| *** akwasnie has quit IRC | 11:25 | |
| sdake | how does that fetching work | 11:25 |
| sdake | pointers to line #s | 11:25 |
| sdake | I need to understand how kolla-mesos is bootstrapping | 11:25 |
| sdake | before I can offer a proper solution | 11:25 |
| sdake | and we will get here by you answering th equestion about how set_configs launches the container | 11:26 |
| kproskurin | kolla_set_config checks env here: https://github.com/openstack/kolla/blob/master/docker/base/set_configs.py#L206 and we pass this env var to container with this data: KOLLA_CONFIG={ "command": "kolla_mesos_start", "config_files": [ { "source": "zk://172.20.9.27:2181/kolla/common/kolla_mesos_start.py", "dest": "/usr/local/bin/kolla_mesos_start", "owner": "root", "perm": "0755" } ]} | 11:27 |
| kproskurin | Here it understands what this is ZK path: https://github.com/openstack/kolla/blob/master/docker/base/set_configs.py#L69 | 11:27 |
| kproskurin | and it need to fetch it from ZK | 11:28 |
| kproskurin | And we pass all needed info for this in this env | 11:28 |
| *** achanda has joined #kolla | 11:28 | |
| kproskurin | So it fetch our script: kolla_mesos_start and put it in /usr/local/bin/kolla_mesos_start | 11:28 |
| kproskurin | This part is ok? Or need some details? | 11:28 |
| sdake | where is the fetching done | 11:29 |
| kproskurin | This func: https://github.com/openstack/kolla/blob/master/docker/base/set_configs.py#L124 | 11:29 |
| sdake | need more details i'll ask qs ou answer, we will get here | 11:29 |
| kproskurin | and this check: https://github.com/openstack/kolla/blob/master/docker/base/set_configs.py#L135 | 11:29 |
| sdake | so this here https://github.com/openstack/kolla/blob/master/docker/base/set_configs.py#L138 | 11:30 |
| kproskurin | yes | 11:30 |
| kproskurin | Should I go on? | 11:30 |
| sdake | need few minutes to process | 11:30 |
| kproskurin | ok | 11:30 |
| sdake | i had always envisioned the json file as being common between different implementations of underlays | 11:32 |
| *** Serlex1 has joined #kolla | 11:32 | |
| sdake | asalkeld has made a forked version | 11:32 |
| sdake | can you link his mariadb script again | 11:32 |
| kproskurin | one sec | 11:33 |
| kproskurin | https://github.com/openstack/kolla-mesos/blob/master/services/mariadb/mariadb.yml.j2 | 11:33 |
| sdake | https://github.com/openstack/kolla-mesos/blob/master/services/mariadb/mariadb.yml.j2#L19 | 11:34 |
| sdake | this is the json parsed by set_configs | 11:34 |
| *** achanda has quit IRC | 11:34 | |
| sdake | (w euse json because it requires no yml depenedency on the host) | 11:34 |
| sdake | I dont mind if asalkeld uses yml | 11:34 |
| sdake | here is the problem | 11:35 |
| sdake | https://github.com/openstack/kolla-mesos/blob/master/services/mariadb/mariadb.yml.j2#L13 | 11:35 |
| sdake | needs to go in lines 20-23 | 11:35 |
| kproskurin | Well, not really. Its just a bit different structure. | 11:36 |
| sdake | *no* | 11:36 |
| sdake | galera.cnf.j2 is the ssame json file format as ansible | 11:36 |
| sdake | I want that ABI to be the same between projects | 11:36 |
| sdake | at midcycle we agreed with nihilifer in room that anywhere we could have common architecture we would between repos or commo ncode we would | 11:37 |
| sdake | if we couldn't then variance is permitted | 11:37 |
| sdake | this is a case where there can be common code | 11:37 |
| Serlex1 | wow sdake, do you not sleep? | 11:38 |
| kproskurin | oh, yeah, thats to asalkeld | 11:38 |
| Serlex1 | which time zone you live in? | 11:38 |
| sdake | Serlex1 super man!! | 11:38 |
| Serlex1 | haha | 11:38 |
| sdake | mst | 11:38 |
| sdake | i went to bed at 6-7 last night | 11:38 |
| sdake | and wake up super early unfortunately | 11:38 |
| Serlex1 | 4:38am there -_- | 11:38 |
| sdake | kproskurin no I was talking to you | 11:38 |
| kproskurin | sdake: I understand, but asalkeld is the man who do such calls in kolla-mesos | 11:39 |
| sdake | do which calls | 11:39 |
| sdake | you mean code calls | 11:39 |
| sdake | or decision making | 11:39 |
| kproskurin | second | 11:39 |
| sdake | decision making happens by consensus in community not by asalkeld's calls :) | 11:40 |
| kproskurin | :-) | 11:40 |
| sdake | we made decision by consensus that architecture where it can be should be the same | 11:40 |
| sdake | which means line 13 needs to go to line 20 | 11:40 |
| sdake | in some way | 11:40 |
| kproskurin | In new to opensource, was 10 years in corporate enslavery, so maybe I chose not the right words. :-) | 11:42 |
| sdake | dude no sweat | 11:42 |
| sdake | ok so lets get a blueprint filed | 11:42 |
| sdake | so we can start getting work items filed on it | 11:42 |
| sdake | firstwork item is unifyign command across kolla-mesos and kolla-ansible | 11:43 |
| sdake | kproskurin will you file or shall I | 11:43 |
| Serlex1 | In a AIO environment, docs says I need two interfaces. I have a host-only adapter and NAT adapter. However I'm not sure how the networking on globals.yml should be configured | 11:43 |
| sdake | host-only adapter is connected tothe internets? | 11:44 |
| kproskurin | sdake, I thinks it’s better if you file it | 11:45 |
| kproskurin | if you dont mind | 11:45 |
| sdake | kproskurin your wish is my command ;) | 11:46 |
| sdake | bbiaf | 11:46 |
| kproskurin | %-) | 11:46 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Add Heka log decoder for RabbitMQ https://review.openstack.org/276668 | 11:46 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Add Heka log decoder infrastructure https://review.openstack.org/276667 | 11:46 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Make Heka collect RabbitMQ logs https://review.openstack.org/275844 | 11:46 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Add Heka log decoder for OpenStack https://review.openstack.org/275845 | 11:46 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Make Heka collect Nova logs https://review.openstack.org/275846 | 11:46 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Add Heka log decoder for MariaDB https://review.openstack.org/275847 | 11:46 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Add Heka log decoder for Keystone/Apache https://review.openstack.org/279192 | 11:46 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Add a Dockerfile for Heka https://review.openstack.org/275840 | 11:46 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Make Heka collect Keystone logs https://review.openstack.org/279193 | 11:46 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Add Heka to common role https://review.openstack.org/275841 | 11:46 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Make Heka collect HAProxy and Keepalived logs https://review.openstack.org/275842 | 11:46 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Make Heka collect kolla-toolbox logs https://review.openstack.org/279191 | 11:46 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Make Heka collect MariaDB logs https://review.openstack.org/275848 | 11:46 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Make Heka collect Glance logs https://review.openstack.org/276760 | 11:46 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Make Heka collect Neutron logs https://review.openstack.org/279489 | 11:46 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Make Heka collect Swift logs https://review.openstack.org/279490 | 11:46 |
| Serlex1 | Adapter 1 is host-only with dhcp enabled on default range 172.28.128.X. Adapter 2 is NAT with 10.0.3.X. Services are bind to 172.28.128.3 IP and docker is registered to 10.0.3.15 | 11:48 |
| Serlex1 | Also do I need to do clean up or some sort of reset if a kolla-deploy fails? | 11:49 |
| Serlex1 | I can see that it skips pass anything that is already configured, however it seems to have failed on starting neutron-dhcp-agent and disconnected my ssh session | 11:49 |
| *** sbezverk has quit IRC | 11:51 | |
| *** JoseMello has joined #kolla | 11:51 | |
| *** aginwala has joined #kolla | 11:53 | |
| *** aginwala has quit IRC | 11:58 | |
| sdake | kproskurin https://blueprints.launchpad.net/kolla/+spec/custom-extend-start | 11:59 |
| kproskurin | sdake: tx! | 12:00 |
| sdake | Serlex1 bio then i'm with ajafo for a bit then i'll be with you ok? | 12:00 |
| sdake | kproskurin please read through the blueprint make corrections where necessary or edit the whiteboard with comments | 12:00 |
| *** Marga__ has joined #kolla | 12:00 | |
| Serlex1 | yeah thats fine sdake, I'm in no rush to get it working | 12:02 |
| *** Marga_ has quit IRC | 12:03 | |
| sdake | ajafo still around? | 12:04 |
| ajafo | sdake: I need little investigation to have question because at this moment I've 3 env's and have different error on everyone :/ | 12:04 |
| ajafo | sdake: so at this moment I'll not take your time, and if you let me I'll try ask some later | 12:05 |
| ajafo | if you don't mind | 12:05 |
| sdake | ajafo ok wfm | 12:05 |
| sdake | you may hae to work with someone else i have a dr appointment this morning | 12:05 |
| sdake | but i'm sur eosmeone will be around that can get you going | 12:05 |
| sdake | ok serlex thats get rolling | 12:05 |
| sdake | Serlex1 | 12:05 |
| ajafo | sdake: ok thx | 12:05 |
| sdake | Serlex1 you havea couple issues | 12:07 |
| sdake | lets work em one by one | 12:07 |
| sdake | Serlex1 you said docker is running on 10.0.3.z | 12:07 |
| sdake | but your publi interface is 172.x.y.z | 12:07 |
| sdake | docker needs to run on the public interface | 12:07 |
| sdake | once kolla starts up, your 10.0.3 network will be used by neutron and unusable by anything else | 12:08 |
| sdake | 10.0.3 i think is what you plan to use as your neutron interface, is that correct? | 12:08 |
| *** dave-mccowan has joined #kolla | 12:08 | |
| sdake | morning dave-mccowan | 12:09 |
| dave-mccowan | good morning sdake! | 12:09 |
| Serlex1 | hang on let me check few things sdake | 12:09 |
| *** kproskurin has quit IRC | 12:10 | |
| *** kproskurin has joined #kolla | 12:10 | |
| Serlex1 | Ok I was wrong, 10.0.3. interface is my default route out | 12:12 |
| Serlex1 | 172.x.y.z is the host-only adapter with DHCP enabled | 12:12 |
| Serlex1 | [root@localhost kolla]# route | 12:19 |
| Serlex1 | Kernel IP routing table | 12:19 |
| Serlex1 | Destination Gateway Genmask Flags Metric Ref Use Iface | 12:19 |
| Serlex1 | default 10.0.3.2 0.0.0.0 UG 0 0 0 enp0s8 | 12:19 |
| Serlex1 | 10.0.3.0 0.0.0.0 255.255.255.0 U 0 0 0 enp0s8 | 12:19 |
| Serlex1 | 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 | 12:19 |
| sdake | paste your globals.yml please | 12:19 |
| sdake | not in channel | 12:19 |
| sdake | use a past service | 12:19 |
| Serlex1 | its the default at the moment | 12:19 |
| sdake | ok well default doesn't work, you need atleast 4 config options | 12:19 |
| sdake | first Q, are you using a registry | 12:19 |
| Serlex1 | I've restored the vm to a snapshot post kolla-build, pre deploy | 12:20 |
| Serlex1 | local registry | 12:20 |
| *** Jeffrey4l has joined #kolla | 12:22 | |
| sdake | did all your containers build? | 12:22 |
| *** intr1nsic has quit IRC | 12:22 | |
| Serlex1 | sure | 12:23 |
| Serlex1 | docker image count is 42 | 12:23 |
| sdake | ajafo coudl you do me a favor please | 12:23 |
| ajafo | sdake: yes? | 12:23 |
| sdake | ajafo do you hae a current deployment that is active? | 12:24 |
| sdake | doesn't have tow ork just has to be active | 12:24 |
| ajafo | docker images | grep ubuntu | wc -l | 12:24 |
| ajafo | 72 | 12:24 |
| ajafo | sdake: yes | 12:24 |
| Serlex1 | haha guessing now all is built then | 12:24 |
| sdake | ajafo dave-mccowan really needs the file from the haproxy container haproxy.cfg | 12:24 |
| Serlex1 | deploy failed on ip configuration not lack of images though | 12:24 |
| ajafo | sdake: one second | 12:25 |
| sdake | ajafo if you need help with docker commands to get that file otu let me knwo | 12:25 |
| *** intr1nsic has joined #kolla | 12:25 | |
| ajafo | sdake: http://paste.openstack.org/show/486804/ | 12:25 |
| sdake | Serlex1 if your on centos, you can yum install fpaste and then use fpaste /etc/globals.yml | 12:25 |
| dave-mccowan | sdake ajafo nevermind. the template is fine. i found it. | 12:25 |
| ajafo | dave-mccowan:http://paste.openstack.org/show/486804/ | 12:26 |
| ajafo | ok | 12:26 |
| openstackgerrit | Proskurin Kirill proposed openstack/kolla-mesos: Improve logging of cleanup script https://review.openstack.org/279006 | 12:27 |
| sdake | dave-mccowan so i think what that comment means in external-ssl is the communication to the haproxy endpoint from internally is nto private | 12:27 |
| sdake | that is why its an attack vector | 12:27 |
| sdake | if someone is sniffing your public HAPproxy VIP | 12:28 |
| sdake | sometimes they will get plaintext passwords | 12:28 |
| sdake | the internal services don't just magically get ssl because the haproxy server is setup right? | 12:29 |
| Serlex1 | http://pastebin.com/wBWziEjg | 12:29 |
| Serlex1 | brb | 12:29 |
| dave-mccowan | right, the internal services still use plaintext, but should be using an internalURL. | 12:31 |
| sdake | i am pretty sure they use internal-url | 12:31 |
| sdake | the problem I think kolla suffers from is we don't hvae an internl-url endpoint for haproxy that is different from the external endpoint | 12:32 |
| sdake | its all one endpoint (an external one) | 12:32 |
| sdake | I'm not totallly convinced using two VIPs makes total sense though | 12:33 |
| sdake | one for internal and one for external | 12:33 |
| sdake | the risk is small, I was just curious on your thinkingon the matter | 12:34 |
| sdake | if soemone has managed to sniffy your external api endpoint from the internal network, you probably havebig troubles ahead | 12:34 |
| *** aginwala has joined #kolla | 12:35 | |
| sdake | ajafo thanks bro | 12:36 |
| sdake | Serlex1 ping me when yoru back and we can get your globals.yml into shape | 12:37 |
| ajafo | sdake: n/p | 12:37 |
| sdake | dave-mccowan ^^ | 12:37 |
| dave-mccowan | sdake i just noticed that looking at kolla's template. i'm used to seeing two networks. the public network, with the public URLs, and an internal network with the admin and internal URLs. | 12:38 |
| sdake | interesting | 12:38 |
| openstackgerrit | Merged openstack/kolla-mesos: We forgot to add mesos section to config https://review.openstack.org/279012 | 12:38 |
| sdake | so our model is you put a NAT in front of the VIP and that gives you security | 12:38 |
| sdake | only ports are accessible via the NAT to the haproxy API endpoint ports | 12:39 |
| sdake | which should be all rock solid secure | 12:39 |
| *** aginwala has quit IRC | 12:39 | |
| *** liyi has joined #kolla | 12:41 | |
| dave-mccowan | keeping the same VIP for internal and external URLs, mean that you'll need to use different port numbers. keystone public-secure will bind to 5000 and keystone internal-open will then bind to 5001, for example. | 12:41 |
| *** rhallisey has joined #kolla | 12:41 | |
| dave-mccowan | i like the approach of two VIPs better. then only standard port numbers are needed. use TLS only on the public VIP, and use plaintext only on the internal/admin VIP. | 12:43 |
| liyi | hi, today kolla deployment require target host has python&docker-py installed. I am wondering if this could be removed by using kolla_toolbox as the jumpstart container. I.e. running ansible script inside the container. Is this feasible or just crazy? | 12:46 |
| *** aginwala has joined #kolla | 12:48 | |
| sdake | dave-mccowan i'm not sure that is how keystone binds | 12:48 |
| openstackgerrit | Proskurin Kirill proposed openstack/kolla-mesos: Fix ubuntu config path search https://review.openstack.org/278963 | 12:49 |
| sdake | we bind to 5000 and 355somethinroanother | 12:49 |
| sdake | llyi 5:49 am | 12:49 |
| sdake | llyi brain booting | 12:49 |
| ajafo | 35357 | 12:50 |
| sdake | ajafo roger | 12:50 |
| dave-mccowan | public_ip:5000, internal_ip:5000, admin_ip:35357 | 12:50 |
| rhallisey | liyi, I mean it's not completely crazy because we technically use kolla_toolbox similarly, but I think there we would need to have quite an evaluation for that change given it will affect the way we operate | 12:51 |
| sdake | llyi so i can tell you mitaka-3 is out of the question for such a change | 12:51 |
| rhallisey | liyi, I mean docker-py cant be so bad :) | 12:51 |
| sdake | newton, hey, anything can happen ;) | 12:51 |
| dave-mccowan | kolla is making the requirement that public_ip==internal_ip==admin_ip. right? | 12:51 |
| liyi | thanks guy! | 12:51 |
| sdake | rhallisey distros like coreos don't have a python runtime | 12:52 |
| liyi | i am happy that i was not the only crazy one | 12:52 |
| liyi | :) | 12:52 |
| liyi | i believe evetually, the host could use OS like Atomic | 12:52 |
| sdake | dave-mccowan nope my public_endpoint is broked.self-ip.net, my internal_endpoint is my IVP endpoint, and admin_ip uses my VIP endpoint | 12:53 |
| rhallisey | liyi, yes and just have docker there | 12:53 |
| sdake | the downside of toolbox container to launch all that stuff is it would be docker on docker | 12:53 |
| rhallisey | liyi, it's possible. It would require an extensive evaluation and refactor, but I think it could work | 12:53 |
| *** aginwala has quit IRC | 12:53 | |
| liyi | there will a very neat system, right rhallisey? | 12:53 |
| liyi | :) | 12:53 |
| rhallisey | yess | 12:53 |
| liyi | looking forward to seeing that to happen :D | 12:54 |
| sdake | broked.self-ip.net is a NAT gateway which forwrads to my internal endpoint IP | 12:54 |
| sdake | dave-mccowan make sense? | 12:54 |
| liyi | sdake, I know it is too late for mitaka3. but do we have a plan for this already, right? | 12:55 |
| sdake | llyi sounds like a bluepritn needs to be filed and placed in the discussion state | 12:56 |
| sdake | llyi doesn't atomic have a python runtime? | 12:56 |
| dave-mccowan | so the TLS termination needs to happen between the NAT gateway and the internal endpoint. | 12:56 |
| sdake | this is just my setup | 12:56 |
| sdake | but i think this is a secure model | 12:57 |
| sdake | but nto being a networking rocket scientist, not 100% certain if this is how people secure their networks | 12:57 |
| liyi | i am not sure about atomic. But i know other systems in that style only have docker engine | 12:57 |
| sdake | or just suck itu p and burn wo ips addresses | 12:57 |
| sdake | llyi right | 12:57 |
| liyi | for example docker-machine, rancherOS | 12:57 |
| sdake | when did docker-machine turn into an OS :) | 12:58 |
| *** salv-orl_ has quit IRC | 12:58 | |
| liyi | sorry, i took it wrong | 12:58 |
| sdake | llyi do yo uknwo how ot file a blueprint | 12:58 |
| liyi | but rancheros is | 12:59 |
| liyi | yes, kind of | 12:59 |
| liyi | do u want me to do it? | 12:59 |
| rhallisey | liyi, ya file a bp | 12:59 |
| sdake | rhallisey need to know for certain if atomic has a python runtime as well as docker-py - since /usr is readonly | 13:00 |
| sdake | can you verify that | 13:00 |
| rhallisey | ya | 13:01 |
| liyi | no problem, I will give it a try and back to u later. | 13:01 |
| sdake | dave-mccowan most people htat deploy kolla really suffer through the "what the hell is a VIP" thing :) | 13:01 |
| sdake | throwing two in the mix sounds like more pain for them | 13:01 |
| *** dwalsh has joined #kolla | 13:01 | |
| sdake | with two VIPs I guess we could have one on an external network and one on an internal network, and not have to worry about NAT | 13:01 |
| sdake | dave-mccowan is that the thinking between the two VIPS? | 13:02 |
| dave-mccowan | sdake: i added an example: https://etherpad.openstack.org/p/kolla-mitaka-midcycle-ssl | 13:02 |
| dave-mccowan | did i get this right? | 13:02 |
| dave-mccowan | Serlex1: what is the output of your "keystone endpoint-list | grep 5000" ? | 13:04 |
| sdake | dave-mccowan i dont have a deployment active atm unfortunately | 13:04 |
| sdake | dave-mccowan i have been in dev and my env is afu | 13:04 |
| sdake | but that looks corrrect | 13:04 |
| sdake | that looks like a 1 VIP model | 13:05 |
| sdake | not 2 VIP example | 13:05 |
| sdake | what is server operator? | 13:05 |
| sdake | dave-mccowan ok one delta | 13:06 |
| sdake | publicURL on my machine is broked.self-ip.net | 13:06 |
| sdake | publicURL should be a DNS address | 13:06 |
| sdake | broked.self-ip.net is some address on the internet (my gige connection) | 13:07 |
| sdake | and my wireless NATs to 10.10.10.254 | 13:07 |
| sdake | (its actualy 192.168.1.149 in my environment but same idea) | 13:07 |
| dave-mccowan | sdake the import thing is what keystone says the public url is. | 13:09 |
| sdake | dave-mccowan note the above woudl work (your publicUrl) but then the haproxy endpoint would be totally wide open and require firewall | 13:09 |
| sdake | i have extensively worked on that code | 13:10 |
| sdake | in kolla | 13:10 |
| sdake | I know for certain it will say broked.selfip.net:5000 | 13:10 |
| sdake | if I make it 10.10.10.254 its on the ame network as my management network | 13:11 |
| sdake | then there is no way to isolate the management network and the API network | 13:11 |
| sdake | openstack uses internalurl to communicate internally, but when you first connect to openstack it uses publicurl to find the correct endpoint into the cluster | 13:12 |
| sdake | you probably already know that :)_ | 13:12 |
| sdake | the only way it could find my network for my bare metal gear from my laptop is to do that NAT thing i spoke up | 13:13 |
| sdake | of | 13:13 |
| sdake | also we use keystone v3 | 13:13 |
| sdake | so those configs ren't quite right | 13:13 |
| *** thumpba has joined #kolla | 13:13 | |
| ajafo | ok I've question maybe someone will know | 13:13 |
| rhallisey | sdake, docker-py is on atomic | 13:14 |
| *** thumpba has quit IRC | 13:14 | |
| sdake | rhallisey thanks | 13:14 |
| *** thumpba has joined #kolla | 13:14 | |
| sdake | then we dont hae to worry about atoic with illy's case | 13:14 |
| rhallisey | python-docker-py-1.1.0-2.fc22.noarch in fedora-atomic | 13:14 |
| sdake | just rancheros and coreos | 13:14 |
| rhallisey | ya | 13:14 |
| rhallisey | maybe we could have a container to handle his case | 13:15 |
| sdake | i dont think atomic should have a python runtime fwiw | 13:15 |
| rhallisey | but I feel like if we go down that road why not make it universal | 13:15 |
| ajafo | when I log in into horizon I get CRITICAL keystonemiddleware.auth_token [-] Unable to validate token: Unable to establish connection to https://127.0.0.1:35357 it looks like identity_uri is not set insted of it we've auth_url and auth_uri | 13:15 |
| sdake | but i didn't make i t ;) | 13:15 |
| ajafo | the same in nova | 13:15 |
| sdake | ajafo please paste your globals.yml in a paste service | 13:16 |
| * rhallisey brb | 13:16 | |
| sdake | i am pretty sure identity_uri is deprecated | 13:16 |
| sdake | ajafo can you paste the top part of the logs and see if we are getting any config depcration warnings | 13:17 |
| ajafo | http://paste.openstack.org/show/486811/ | 13:17 |
| sdake | vagrant | 13:19 |
| * sdake groans | 13:19 | |
| dave-mccowan | sdake. i updated the etherpad. the public url is "kolla_external_address". but, per haproxy.cfg.j2, haproxy is not listening on that address? | 13:19 |
| *** thumpba has quit IRC | 13:19 | |
| sdake | dave-mccowan right, its on the operator to sort out how to get the external address to map to the internal address | 13:19 |
| sdake | via dns, nat, whatever | 13:20 |
| ajafo | sdake: right it is in logs http://paste.openstack.org/show/486812/ | 13:20 |
| dave-mccowan | by the same logic, TLS termination would also be on the operator. | 13:20 |
| sdake | ajafo are you on stable/liberty or master? | 13:21 |
| ajafo | sdake: I'm using master with ubuntu build so images build for liberty | 13:22 |
| sdake | dave-mccowan i wasn't make a logic statement, I was stating where the community is on this point | 13:22 |
| ajafo | sdake: but sthg like 1 month ago it was working | 13:22 |
| sdake | I think its illogical personally :) | 13:22 |
| sdake | but touching on the point of ssl termination on operator | 13:23 |
| sdake | this is because the certs must match the host names? | 13:23 |
| dave-mccowan | yea, that's cool. i didn't mean it to be a judgement. trying to wrap my head around how to make it work. | 13:23 |
| sdake | i htink the problem we will suffer with is the certs wont match the dns names | 13:24 |
| sdake | and there will be problems with that | 13:24 |
| dave-mccowan | sdake... definitely, the certs need to match the dns. | 13:24 |
| dave-mccowan | also, sdake somehow real-public-address needs to map to something different than kolla_internal_address | 13:25 |
| sdake | so with nat, how does that work? | 13:25 |
| sdake | where is real-public-address from? | 13:25 |
| sdake | kolla_internal_address is a minomer, it is actually the kolla VIP | 13:26 |
| dave-mccowan | kolla_external_address | 13:26 |
| sdake | dave-mccowan right it does in my environment ;) | 13:26 |
| sdake | but I use NAT | 13:26 |
| sdake | so here is my thinking on all of this | 13:26 |
| sdake | we want what is best for the operators | 13:26 |
| sdake | if the operators want two VIPs one external one internal I htink we should proceed that way | 13:27 |
| sdake | if operators only one one VIP with NAT as their only option, we should roll that way | 13:27 |
| britthouser | sdake: typically NAT rewrites the DNS packets | 13:27 |
| britthouser | Two VIPs is what is recommended by security guide | 13:27 |
| sdake | yes, I had asked earlier if nat damages SSL | 13:27 |
| sdake | britthouser have a link | 13:27 |
| * britthouser digs | 13:28 | |
| sdake | because if nat rewrites dns in some way that makes external SSL not work, then we have to go to two VIPs | 13:28 |
| sdake | which I can live with | 13:28 |
| britthouser | http://docs.openstack.org/security-guide/networking/architecture.html | 13:28 |
| britthouser | have a look at teh BIG diagram | 13:28 |
| sdake | britthouser thanks | 13:28 |
| britthouser | API and Mgmt are two different networks | 13:29 |
| britthouser | But I agree how operators actually run should be our deciding factor. | 13:30 |
| sdake | email going out to operator's list right now | 13:30 |
| *** achanda has joined #kolla | 13:32 | |
| *** tzn has quit IRC | 13:33 | |
| dave-mccowan | sdake if we want to support TLS with a one-VIP option, then we'll need to get creative with ports. it looks like now publicURL and internalURL are aliases of one another. if we want one to support TLS and one to not, then they need to be different... either different IP or different port. | 13:34 |
| *** _tzn has joined #kolla | 13:35 | |
| *** achanda has quit IRC | 13:38 | |
| sdake | public_IP is not supposed to be a different ip | 13:38 |
| sdake | its supposed to be a different dns name | 13:38 |
| sdake | sorry public_url | 13:38 |
| sdake | the operators I believe are supposed to sort out how to get that DNS name to map to the internal VIP | 13:38 |
| sdake | so that requirement you mention above dave-mccowan is already mentioned (but probably not in the docs ;) | 13:39 |
| sdake | publicURL and internalURL should *not* be aliased to one another except for developers playingaround | 13:39 |
| *** kproskurin has quit IRC | 13:39 | |
| sdake | or people doing evals | 13:39 |
| dave-mccowan | sdake the requirement is not to get it to the internalVIP it is getting the the internal representation of the public VIP. | 13:40 |
| sdake | they need to be isolated networks to maintain security | 13:40 |
| sdake | yes dave-mccowan familiar with how keystone works | 13:40 |
| *** aginwala has joined #kolla | 13:40 | |
| sdake | the reason they need to be different is to isoalte the networks | 13:41 |
| sdake | ajafo file bugs witht he deprecation warnigns please | 13:42 |
| sdake | ajafo we fix deprecations quickly :) | 13:42 |
| *** aginwala has quit IRC | 13:44 | |
| sdake | elemoine_ yes please please write unit tests for all python code | 13:46 |
| sdake | it should be a requirement going forward | 13:47 |
| sdake | but I realize we were ina bit of emergency management mode with the whole docker 1.8.2 and needing docker 1.10 | 13:47 |
| sdake | dave-mccowan reread your statement avove and parsed it slighlty differently | 13:48 |
| sdake | I think what your syaing there is in my environment, my NAT does the mapping of publicURL to internalip | 13:48 |
| dave-mccowan | sdake yes | 13:49 |
| sdake | so our current design relies on NAT absolutely and completely with no other options | 13:49 |
| sdake | I went over this with SamYaple back and forth for months on end and he said he didn't really know how the networks were setup but it was on the operators | 13:49 |
| sdake | so I came up with nat, and was happy with that | 13:49 |
| sdake | but if nat isn't going to work with external SSL - that is no-beuno | 13:49 |
| sdake | SamYaple and I never discussed that I recall two VIPs, one for internal and one for public | 13:51 |
| sdake | dave-mccowan will external-ssl work in a NAT environment? | 13:52 |
| dave-mccowan | sdake NAT is no problem, and perhaps a distraction to our discussion. | 13:53 |
| dave-mccowan | sdake. check the etherpad. i have examples of a one VIP and VIP solution. | 13:54 |
| Serlex1 | dave-mccowan - I've discarded the earlier deploy went retored snapshot after kolla-built | 13:56 |
| Serlex1 | sdake - I'm ready to sort out this globals.yml | 13:57 |
| *** dwalsh has quit IRC | 13:57 | |
| *** clayton has quit IRC | 14:00 | |
| sdake | dave-mccowan what si your public email address | 14:01 |
| sdake | i am going to cc you on this operator thread | 14:01 |
| dave-mccowan | i use my work address, dmccowan@ | 14:01 |
| d_code | morning | 14:06 |
| dave-mccowan | sdake do the examples i added to the etherpad help clarify? | 14:06 |
| sdake | can you link the etherpad | 14:06 |
| sdake | i've got like a million windows open not sure which one you mean | 14:06 |
| dave-mccowan | https://etherpad.openstack.org/p/kolla-mitaka-midcycle-ssl | 14:06 |
| sdake | oh duh that one | 14:07 |
| openstackgerrit | Merged openstack/kolla: Do not pull rabbitmq-data image https://review.openstack.org/278851 | 14:08 |
| d_code | so, I checked out master yesterday on a CentOS7 box, built from source without a registry, then tried to deploy to all-in-one (using kolla-ansible deploy), and it failed on “Starting neutron-dhcp-agent” | 14:08 |
| sdake | processing igve me fe wminutes | 14:08 |
| d_code | :( | 14:08 |
| sdake | Serlex1 i'll be with you in a moment ok? | 14:08 |
| d_code | running `docker logs neutron_dhcp_agent` produces nothing. | 14:08 |
| Serlex1 | thats cool | 14:08 |
| d_code | error is: msg: APIError(HTTPError('500 Server Error: Internal Server Error',),) | 14:08 |
| sdake | d_code so the container didn't even start? | 14:09 |
| sdake | did you yum update before doing that? | 14:10 |
| d_code | sdake: as far as I can tell, I guess…. | 14:10 |
| sdake | what type of file storage backend are you using? | 14:10 |
| d_code | didn’t do a yum update prior to that | 14:11 |
| ajafo | d_code: what version docker and docker_py have you got? | 14:11 |
| d_code | storage backend……whatever is default? | 14:11 |
| ajafo | it's full error or is sthg else? | 14:11 |
| d_code | Docker version 1.10.0, build 590d5108 | 14:11 |
| d_code | docker-py Version: 1.7.0 | 14:12 |
| *** clayton- has joined #kolla | 14:12 | |
| ajafo | I'm trying to build centos containers becuase ubuntu stop working for me | 14:12 |
| d_code | the rest is just ansible telling me it failed and changed = true | 14:12 |
| ajafo | but I downgrade docker to 1.9.1 and docker-py to 1.6.0 | 14:12 |
| *** salv-orlando has joined #kolla | 14:13 | |
| sdake | dave-mccowan ya gotta say I don't like lines 58-68 one bit | 14:14 |
| sdake | the implications of that are massive | 14:14 |
| sdake | as in pain producing :) | 14:14 |
| britthouser | dave-mccowan: would a 4th alternative be enabling TLS for internal as well? that would simplify 56-68, right? | 14:15 |
| d_code | sdake: looks like journalctl has more info: https://gist.github.com/dcode/a8f26f325d5321963542 | 14:15 |
| britthouser | but then I imagine all containers would need the keys | 14:16 |
| sdake | d_code type "df" | fpaste | 14:16 |
| sdake | without the quotes | 14:17 |
| sdake | Serlex1 can you get your globals.yml in an fpaste | 14:17 |
| sdake | Serlex1 i've lost all the context we had in our discussion earlier because of the drop off | 14:17 |
| sdake | i've got 1.5 hours then need to jet to dr appointment for a bit | 14:17 |
| dave-mccowan | sdake i don't see pain. pain to kolla? the keystone catalog manages the internal vs. external endpoints for us, it just a matter of touching up all the registers. | 14:18 |
| d_code | sdake: http://ur1.ca/oig40 | 14:18 |
| d_code | wow…fpaste is new to me…neat | 14:18 |
| *** jtriley has quit IRC | 14:18 | |
| Serlex1 | pastebin if thats ok http://pastebin.com/LnS20TLg | 14:19 |
| sdake | d_code getenforce ? | 14:20 |
| d_code | Enforcing | 14:20 |
| sdake | no bueno | 14:20 |
| sdake | turn that off please | 14:20 |
| d_code | that’s no in the instructions ;-) | 14:20 |
| d_code | *not | 14:20 |
| sdake | /etc/selinux/config | 14:20 |
| sdake | it used to be | 14:20 |
| d_code | yes, I’m well aware of how to do it | 14:20 |
| sdake | ok | 14:20 |
| sdake | well i'll make sure the selinux thing makees it back into the instructions | 14:21 |
| d_code | for production release, is there any effort to get selinux policies functional? | 14:21 |
| sdake | d_code we want to start down that path soon | 14:21 |
| sdake | the reason is not what you migh tthink | 14:21 |
| sdake | keystone community has indicated they will use kolla for R&D of keystone (rather then devstack) | 14:21 |
| sdake | they are blocked on lack of selinux | 14:22 |
| sdake | this is a use case we want to explore | 14:22 |
| sdake | but not our primary objective | 14:22 |
| d_code | it’s a requirement for my production build when I get there in July | 14:22 |
| Serlex1 | 10.0.3.15 (interface enp0s8 which is NAT) is how I reach out to the internet. enp0s8 (host-only adapter with DHCP enabled on 172.17.0.1) | 14:22 |
| d_code | I’m pretty well acquainted with selinux | 14:22 |
| d_code | not so much with openstack, but given that it seems to have made it that far | 14:22 |
| sdake | can't promise working selinux by july - we only have one dev with exeprience with it , rhallisey and he is pretty swamped with other htings atm | 14:23 |
| sdake | d_code we do take commits and I can help you learn how to commit properly and all that ifyou sort out the patchess | 14:23 |
| sdake | i just dont knwo selinux | 14:23 |
| sdake | that is why i can't promise it | 14:23 |
| sdake | because I don't know the scope of the work | 14:23 |
| d_code | understood | 14:23 |
| d_code | also…there are no selinux denials…so that didn’t cause my issue… neutron still failed to run | 14:25 |
| d_code | I’ll look into the code to see if I can trace this error | 14:25 |
| *** dmsimard has quit IRC | 14:25 | |
| sdake | busy busy | 14:26 |
| sdake | Serlex1 and you did buidl oraclelinux distro iamges from source? | 14:26 |
| Serlex1 | binary | 14:27 |
| sdake | ya binary sorry | 14:27 |
| sdake | ok first problem line 21 | 14:27 |
| sdake | kolla_internal_address | 14:27 |
| sdake | This should be a VIP, an unused IP on your network | 14:27 |
| sdake | do yo uhave any unused ip addresses on your network? | 14:28 |
| sdake | it should not be your host's IP address | 14:28 |
| Serlex1 | i set an from the DHCP interface, done | 14:28 |
| Serlex1 | an IP | 14:28 |
| sdake | ok so you have an unused I Padddress, and set line 21 to it? | 14:28 |
| Serlex1 | yep | 14:28 |
| sdake | delete line 30 | 14:29 |
| sdake | WHOEVER KEEPS TELLING PEOPLE TO TURN OFF HAPROXY PLEASE STOP THAT | 14:29 |
| sdake | that is not how kolla works and it makes setup and debug harder | 14:29 |
| Serlex1 | lol ok deleted | 14:29 |
| sdake | if you want haproxy to be optional, write the code to make it all work optioanlly | 14:29 |
| sdake | that is all :) | 14:29 |
| sdake | ok | 14:30 |
| Serlex1 | haha ok ok | 14:30 |
| sdake | registry | 14:30 |
| sdake | Serlex1 that was targeted at the cor ereviewers not you | 14:30 |
| sdake | they read the logs | 14:30 |
| sdake | you have a docker registry? | 14:30 |
| sdake | line 49 should be network_interface: "whatever 10.0.3.15 is on" | 14:31 |
| Serlex1 | ok | 14:31 |
| sdake | line 61 should be whatever your other 172 network is on | 14:31 |
| sdake | i see yoru running stable/liberty | 14:31 |
| sdake | is that correct? | 14:32 |
| Serlex1 | how do I check sorry | 14:32 |
| sdake | how did yo uget the software | 14:32 |
| Serlex1 | git | 14:32 |
| sdake | go to git directory where code is located | 14:33 |
| sdake | and type git status | fpaste | 14:33 |
| *** macsz has left #kolla | 14:33 | |
| openstackgerrit | Proskurin Kirill proposed openstack/kolla-mesos: Zookeeper expects strings as values https://review.openstack.org/279566 | 14:34 |
| *** akwasnie has joined #kolla | 14:35 | |
| sdake | hey akwasnie | 14:35 |
| Serlex1 | i dont have fpaste setup on this oracle box | 14:35 |
| Serlex1 | hang on | 14:35 |
| Serlex1 | [root@localhost kolla]# git status | 14:36 |
| Serlex1 | # On branch master | 14:36 |
| Serlex1 | nothing to commit, working directory clean | 14:36 |
| sdake | akwasnie can you give me a run down of where you stand with diags and elemoine_ integration? | 14:36 |
| sdake | Serlex1 ok docker images | fpaste | 14:37 |
| sdake | d_code | 14:38 |
| sdake | are you using registry v1? | 14:38 |
| akwasnie | hey sdake | 14:39 |
| Serlex1 | sdake im sligtly confused. "(2:31:15 PM) sdake: line 49 should be network_interface: "whatever 10.0.3.15 is on" | 14:39 |
| Serlex1 | But globals.yml says | 14:39 |
| Serlex1 | When running an All-In-One | 14:39 |
| Serlex1 | 19 # without haproxy and keepalived, this should be the first IP on your | 14:39 |
| Serlex1 | 20 # 'network_interface' as set in the Networking section below. | 14:39 |
| Serlex1 | 21 kolla_internal_address: "172.28.128.4" | 14:39 |
| sdake | look at your paste | 14:40 |
| Serlex1 | shouldn't network_interface be whatever 172.28.1.x IP | 14:40 |
| sdake | not at the globals.yml in master | 14:40 |
| sdake | a network interface i slike eth0, enp0s4, etc | 14:40 |
| sdake | ip link show will list your interface | 14:40 |
| sdake | its the interface names | 14:40 |
| akwasnie | sdake: I prepared Elasticsearch patch (Dockerfile+ansible) and discussed with Eric that we will begin its integration with Heka after Elasticsearch merge | 14:41 |
| sdake | akwasnie ok what do we need to get unblocked then? | 14:41 |
| sdake | a review of your work? | 14:41 |
| akwasnie | in the meantime I prepared also Logstash, so we can test both solutions (Heka and Logstash) | 14:41 |
| akwasnie | sdake: yes, review | 14:41 |
| akwasnie | https://review.openstack.org/#/c/267714/ | 14:41 |
| d_code | sdake: I switched to registry v2, but I was having issues at another point in the deploy…so I switched to the docker container like you suggested | 14:42 |
| sdake | zomg big patch | 14:42 |
| sdake | dont have time before dr, but i'll review today akwasnie | 14:42 |
| sdake | docker registery 2.3? | 14:42 |
| akwasnie | sdake: ok, thanks | 14:42 |
| sdake | i have seen your error with older docker registries | 14:43 |
| sdake | but 2.3 should deploy | 14:43 |
| sdake | you may have to rebuild unfortunately | 14:43 |
| sdake | doc fixes are in progress a tm | 14:43 |
| *** cfarquhar has quit IRC | 14:43 | |
| sdake | you said you switched to the docker container like i suggested, you mean 2.3? | 14:44 |
| sdake | dcode ^^ | 14:44 |
| sdake | d_code ^^ | 14:44 |
| d_code | yes | 14:44 |
| Serlex1 | hmm | 14:44 |
| d_code | maybe I should just clear out the whole box and start from scratch | 14:44 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Add neutron config https://review.openstack.org/265109 | 14:45 |
| sdake | d_code | 14:45 |
| sdake | lets keep as is | 14:45 |
| sdake | yo uturned off selinux right? | 14:45 |
| d_code | I’m keeping a record of what I do…in the hopes I’ll eventually have something that works | 14:45 |
| sdake | turn off firewalld | 14:45 |
| d_code | yes | 14:45 |
| d_code | k | 14:45 |
| *** cfarquhar has joined #kolla | 14:45 | |
| *** cfarquhar has quit IRC | 14:45 | |
| *** cfarquhar has joined #kolla | 14:45 | |
| *** dwalsh has joined #kolla | 14:45 | |
| sdake | turn off iptables | 14:45 |
| d_code | well…no firewalld, but I have iptables | 14:45 |
| sdake | i thought this was all documented | 14:45 |
| sdake | i think people must have removed it | 14:45 |
| *** jtriley has joined #kolla | 14:46 | |
| sdake | your on centos without firewalld? | 14:46 |
| sdake | minimal install ? | 14:46 |
| *** _tzn has quit IRC | 14:46 | |
| d_code | yes | 14:46 |
| sdake | ok you need to yum udpate your kernel | 14:46 |
| d_code | already did that | 14:46 |
| sdake | i'd just yum updat everything | 14:46 |
| Serlex1 | sdake - what should neutron_external_interface be? | 14:46 |
| sdake | Serlex1 whatever your 172 (not your normal internet address) is mapped to in your host | 14:47 |
| sdake | ip link show | grep 172 | 14:47 |
| sdake | that should give you a few choices to look at :) | 14:47 |
| *** akwasnie has quit IRC | 14:47 | |
| Serlex1 | ok | 14:47 |
| sdake | d_code cool | 14:47 |
| d_code | sdake: I basically did this script, starting from a minimal install, removed firewalld, replaced with iptables-services, and yum updated, then reboot | 14:47 |
| d_code | https://gist.github.com/dcode/aea450c1648034067637 | 14:47 |
| d_code | gonna add turn off iptables to that for now | 14:47 |
| sdake | i actually leave iptables on | 14:48 |
| sdake | is it regisstery 2.3.0 or registery 2.3 on line 30? | 14:48 |
| sdake | I'm not sure :) | 14:48 |
| *** JoseMello has quit IRC | 14:48 | |
| sdake | let me look at dockerhub | 14:48 |
| Serlex1 | ok and then deploy? :D | 14:48 |
| sdake | serlex paste your globals.yml pls | 14:48 |
| d_code | sdake: I copied and pasted what you put last night | 14:49 |
| Serlex1 | ok | 14:49 |
| d_code | but it installed 2.3.0 | 14:49 |
| Serlex1 | i must say you are a machine sdake | 14:49 |
| Serlex1 | thanks for the hlep | 14:49 |
| Serlex1 | http://pastebin.com/JvHjVdRY | 14:50 |
| d_code | looks like 2.3.0 is the latest | 14:50 |
| sdake | d_code 2.3 and 2.3.0 are both versions | 14:51 |
| d_code | ah…see that now | 14:51 |
| sdake | in the future, i'd switch that script around to 2.3 | 14:51 |
| d_code | k…will do | 14:51 |
| sdake | but for now | 14:51 |
| sdake | keep as is | 14:51 |
| sdake | ok clenaup-containers | 14:51 |
| sdake | run script | 14:51 |
| sdake | run kolla-ansible deploy | 14:51 |
| sdake | exit | 14:51 |
| sdake | paste typescript file created | 14:51 |
| d_code | I don’t need to redo the build, right? everything seemed like it built from source just fine in the local docker cache | 14:52 |
| d_code | like I said, I had issues pushing to registry | 14:52 |
| sdake | Serlex1 need your docker images | fpaste | 14:52 |
| sdake | d_code shouldn't need to | 14:52 |
| sdake | oh your stuck on ./build --push? | 14:52 |
| d_code | I figured if I could deploy all in one, then I could figure out the registry thing and get it working on multiple hosts | 14:52 |
| d_code | I skipped push since I’m doing all in one | 14:53 |
| d_code | does that not work | 14:53 |
| d_code | ? | 14:53 |
| sdake | with master without a registry? | 14:53 |
| Serlex1 | http://pastebin.com/efBUPX6p | 14:53 |
| sdake | serlex1 do tools/cleanup-containers then try a deploy | 14:54 |
| sdake | your config looks good | 14:54 |
| Serlex1 | ok | 14:54 |
| d_code | sdake: yes…master without registry to do all-in-one | 14:54 |
| sdake | d_code ok there was a recent change to master | 14:54 |
| d_code | haha…. bleeding edge….love it | 14:54 |
| sdake | I am nto sure, but someone may ahve removed the ability to run without a registry | 14:54 |
| sdake | I dont know | 14:55 |
| sdake | so lets try it | 14:55 |
| sdake | then I'll know | 14:55 |
| *** clayton- is now known as clayton | 14:55 | |
| sdake | I wasn't sure - I saw the patch review fly by my email | 14:55 |
| d_code | k…just lost connection to my host… lemme switch modes to fixing that :p | 14:55 |
| sdake | the core team is punshing me with vacation this week :) | 14:55 |
| *** mbound has quit IRC | 14:55 | |
| sdake | d_code yu will want to do cleanup-containers tho | 14:56 |
| d_code | k | 14:56 |
| sdake | just ot blow away whatever mess you have with the selinux stuff atm | 14:56 |
| sdake | i think selinux was your problem | 14:56 |
| sdake | your script is essentially what I do when I setup a host but I do it manually | 14:56 |
| *** Marga__ has quit IRC | 14:56 | |
| sdake | having that script in the repo would be great :) | 14:56 |
| d_code | I’m not so sure, but I’ll do it without and see where it goes | 14:56 |
| rhallisey | I'll see if I can come up with some policy for us real quick | 14:56 |
| d_code | I didn’t see any denials | 14:56 |
| d_code | sdake: happy to contribute…I’ve done the same for a couple of other projects | 14:57 |
| *** Marga_ has joined #kolla | 14:57 | |
| *** dave-mccowan has quit IRC | 15:01 | |
| sdake | d_code what I want is that as a playbook but I'll take a script to begin ;) | 15:02 |
| d_code | well…I can do that too | 15:04 |
| d_code | generally how I write my playbooks….do it manually…record it all in a text file….then automate with tool like Ansible | 15:05 |
| d_code | well…Ansible or Chef…depending on what I’m writing…obviously ansible makes more sense for this :) | 15:05 |
| *** dwalsh has quit IRC | 15:08 | |
| d_code | sdake: cleanup-images too? | 15:08 |
| Serlex1 | failed: [localhost] => {"changed": false, "cmd": ["docker", "exec", "haproxy", "/usr/local/bin/kolla_ensure_haproxy_latest_config"], "delta": "0:00:00.643014", "end": "2016-02-12 15:07:59.630687", "rc": 137, "start": "2016-02-12 15:07:58.987673", "stdout_lines": [], "warnings": []} | 15:09 |
| d_code | so, by default, storage will just go somewhere in /var/lib/docker/ as a file, right? | 15:09 |
| sdake | d_code keep images in shape | 15:09 |
| sdake | d_code right | 15:09 |
| sdake | d_code dont delete images | 15:09 |
| d_code | oh boy….yeah…so something on my box isn’t quite right | 15:09 |
| sdake | ok i know how to fix | 15:10 |
| sdake | sudo systemctl stop docker | 15:10 |
| sdake | sudo rm -rf /var/lib/docker | 15:10 |
| d_code | I lost connectivity because got a general protection fault in the kernel when cleaning containers | 15:10 |
| sdake | sudo reboot | 15:10 |
| sdake | ya your docker cache is busted | 15:10 |
| d_code | wonderful | 15:10 |
| sdake | you probably were on docker 1.8.3 and 1.10 at different times | 15:10 |
| d_code | that….may have happened | 15:10 |
| sdake | docker blows at version upgrades | 15:10 |
| d_code | wonderful | 15:10 |
| sdake | but getting better | 15:11 |
| sdake | if you had 1.10 and then ran 1.83 with same /var/lib/docker | 15:11 |
| sdake | your cache was corrupted | 15:11 |
| d_code | k…makes sense | 15:11 |
| sdake | you can just delete /var/lib/docker | 15:11 |
| d_code | k… rebooting :) | 15:11 |
| sdake | and it will recreate teh files | 15:11 |
| d_code | then I’ll clean it up | 15:11 |
| sdake | and rebuild | 15:11 |
| sdake | you just cleaned it up :) | 15:11 |
| sdake | thats the nuclear option with docker dev :) | 15:12 |
| *** iNeilus has joined #kolla | 15:12 | |
| sdake | much faste rthen cleanup-c-ontainers too if you dont mind losing your images :) | 15:12 |
| Serlex1 | sorry sdake, typo, deploy still running | 15:12 |
| d_code | no…I had a kernel panic…I’m rebooting so that I can cleanup | 15:12 |
| sdake | oh roger | 15:12 |
| sdake | ya kernel panic | 15:13 |
| sdake | until runc comes around | 15:13 |
| sdake | not much kolla can do about kernel and docker misbheaviors | 15:13 |
| Serlex1 | I see there is a new docker version just pushed out | 15:14 |
| sdake | a stable version? | 15:14 |
| Serlex1 | [stack@localhost kolla]$ docker -v | 15:14 |
| Serlex1 | Docker version 1.10.1, build 9e83765 | 15:14 |
| sdake | cool should be better | 15:15 |
| Serlex1 | ok my deploy failed again on neutron-dhcp-agent | 15:15 |
| d_code | yeah…that’s what I was having | 15:15 |
| Serlex1 | I'm clearly a chump on understanding the IPs | 15:15 |
| *** tzn has joined #kolla | 15:15 | |
| Serlex1 | :'( | 15:15 |
| sdake | Serlex1 paste ip link show and your globals.yml file plz | 15:16 |
| sdake | serlex1 what type of failure did you get on the neutron-dhcp-agent? | 15:16 |
| sdake | it is possible neutron-dhcp-agent is broken | 15:16 |
| Serlex1 | TASK: [neutron | Starting neutron-dhcp-agent container] *********************** | 15:16 |
| Serlex1 | failed: [localhost] => {"changed": true, "failed": true} | 15:16 |
| Serlex1 | msg: APIError(HTTPError('500 Server Error: Internal Server Error',),) | 15:16 |
| *** iNeilus has quit IRC | 15:16 | |
| sdake | two reproductions | 15:16 |
| sdake | seems like a legit problem | 15:16 |
| sdake | folks I can't test any of kolla atm because my lab is bust | 15:17 |
| sdake | I can test one node vm | 15:17 |
| *** dave-mccowan has joined #kolla | 15:17 | |
| sdake | but been so busy on the irc channel this last week haven't had an opportunity to build | 15:17 |
| sdake | serlex1 type getenforce | 15:17 |
| d_code | we appreciate your presence, sdake | 15:17 |
| Serlex1 | ditto | 15:17 |
| Serlex1 | disabled sdake | 15:18 |
| sdake | i've got to go to the doc soon, let me start a build | 15:18 |
| Serlex1 | http://pastebin.com/71mtxbUR | 15:18 |
| sdake | serlex1 can you ping 8.8.8.8 from your vm? | 15:19 |
| sdake | or whateveryour deploying from | 15:19 |
| Serlex1 | yep | 15:20 |
| Serlex1 | neutron_external_interface: "enp0s3" <--- this interface is a host-only interface with DHCP enabled on virtualbox | 15:20 |
| Serlex1 | I hope that helpss | 15:20 |
| Serlex1 | 172.X range | 15:20 |
| *** sdake_ has joined #kolla | 15:22 | |
| sdake_ | serlex1 which images are you building, centos ones? | 15:23 |
| sdake_ | rather d_code | 15:23 |
| d_code | centos | 15:23 |
| *** dwalsh has joined #kolla | 15:23 | |
| d_code | k…/var/lib/docker is detonated….rebooting and trying again | 15:24 |
| Serlex1 | my images are oraclelinux | 15:24 |
| *** sdake has quit IRC | 15:24 | |
| Serlex1 | Where can I find more verbose logs for that error? | 15:25 |
| d_code | Serlex1: I couldn’t find anything besides what was logged in journalctl | 15:29 |
| d_code | which just said what it was doing, then it failed | 15:29 |
| Serlex1 | bloody hell | 15:30 |
| Serlex1 | which images you using? | 15:30 |
| d_code | CentOS | 15:30 |
| sdake_ | i am building now | 15:30 |
| sdake_ | it is possible this is a centos specific ergression | 15:30 |
| d_code | I’m not an expert on oracle, but there really aren’t that many difference between the two, afaik | 15:31 |
| *** absubram has joined #kolla | 15:31 | |
| sdake_ | our gate is busted for centos so we have been ignroing the deploy of centos a bit until we can fix the gate | 15:31 |
| d_code | bah | 15:31 |
| d_code | k…I built my images with selinux on…I’ve turned it off, blown away docker, I’ll try rebuilding from source again | 15:31 |
| Serlex1 | they are both based on rhel right? so yeah I dont think there is much difference | 15:31 |
| d_code | right | 15:32 |
| *** daneyon has joined #kolla | 15:32 | |
| *** absubram_ has joined #kolla | 15:32 | |
| elemoine_ | akwasnie, sdake, it would be good to merge Elasticsearch to be able to start integration, Heka is ready | 15:32 |
| sdake_ | d_code building images with selinux on will make no difference | 15:34 |
| sdake_ | d_code so you have built from source and built from binary and hae this same regression? | 15:34 |
| d_code | I thought binary build from CentOS isn’t supported currently | 15:35 |
| d_code | there’s a missing package | 15:35 |
| d_code | or couple | 15:35 |
| sdake_ | well kolla doesn't really support anything | 15:35 |
| d_code | :) | 15:35 |
| sdake_ | organizations provide support for kolla | 15:35 |
| sdake_ | we implement | 15:35 |
| d_code | I thought binary build using CentOS was not currently functional | 15:35 |
| *** absubram has quit IRC | 15:36 | |
| *** absubram_ is now known as absubram | 15:36 | |
| sdake_ | it may not be because of a problem in nova_api - the gate fails on deploy | 15:36 |
| sdake_ | but that may be a gate problem rather the na nova problem | 15:36 |
| d_code | lemme check the logs…someone said that RDO changed a package, I think | 15:36 |
| sdake_ | i'll let you know | 15:36 |
| *** achanda has joined #kolla | 15:36 | |
| rhallisey | back | 15:36 |
| d_code | oh yeah…it’s that the RPM creates a dir that ansible expects to create and errors | 15:36 |
| rhallisey | also building.. | 15:36 |
| d_code | I applied the patch that’s in launchpad, but it didn’t fix my issue | 15:36 |
| sdake_ | d_code got link to bug ? | 15:37 |
| d_code | yeah…sec…it’s one of these open tabs :p | 15:37 |
| *** absubram has quit IRC | 15:37 | |
| sdake_ | i've got to leave in 12 minutes | 15:37 |
| *** rhallisey has quit IRC | 15:38 | |
| *** rhallisey has joined #kolla | 15:38 | |
| rhallisey | d_code, I'm building centos binary images now I'll see if I hit the issue | 15:38 |
| Serlex1 | sdake do you think I should continue troubleshooting this or ack its an issue on kolla side? | 15:38 |
| sdake_ | serlex1 two peopel have same problem | 15:39 |
| sdake_ | seems like a bug should be filed and mark confirmed to me | 15:39 |
| sdake_ | that is typicallly how it goes ;) | 15:39 |
| sdake_ | i suspect the kolla_docker module is busted | 15:39 |
| Serlex1 | ok - up for it d_code? | 15:39 |
| Serlex1 | :D | 15:39 |
| sdake_ | but it could be a missing bindmount for centos thatis present on ubuntu | 15:40 |
| sdake_ | who knows | 15:40 |
| sdake_ | it could still be user error | 15:40 |
| d_code | sdake_: https://bugs.launchpad.net/kolla/+bug/1543417 | 15:40 |
| openstack | Launchpad bug 1543417 in kolla "nova-compute bootstrap failing on centos-binary" [Critical,Fix released] - Assigned to Martin André (mandre) | 15:40 |
| sdake_ | but i think you both are doing everything i know how to do to fix things up | 15:41 |
| *** achanda has quit IRC | 15:41 | |
| d_code | looks like that was commited | 15:42 |
| d_code | also…just want to express…. Launchpad is awful….that is all….I won’t mention it again | 15:43 |
| Serlex1 | lol | 15:43 |
| *** dmsimard has joined #kolla | 15:43 | |
| elemoine_ | d_code are you a GitHub user? ;) | 15:43 |
| d_code | I am | 15:43 |
| Serlex1 | ok sdake I will ping back here next week or something | 15:43 |
| Serlex1 | ? | 15:43 |
| d_code | sdake_: I’ll build from source again and post the bug | 15:44 |
| sdake_ | d_code thanks | 15:47 |
| *** JoseMello has joined #kolla | 15:47 | |
| sdake_ | ok dr appointment | 15:48 |
| sdake_ | back in hour or two | 15:48 |
| sdake_ | i should have results by then | 15:48 |
| Serlex1 | ok | 15:48 |
| sdake_ | d_code when yo u file bug link i nchannel and i'll sort out getting it set rightin the tracker | 15:48 |
| d_code | kk | 15:49 |
| *** aginwala has joined #kolla | 15:54 | |
| *** tzn has quit IRC | 15:58 | |
| *** aginwala has quit IRC | 15:58 | |
| *** sdake_ has quit IRC | 15:58 | |
| *** salv-orlando has quit IRC | 16:01 | |
| *** tzn has joined #kolla | 16:04 | |
| *** stvnoyes has quit IRC | 16:07 | |
| *** neilus has joined #kolla | 16:08 | |
| *** stvnoyes has joined #kolla | 16:08 | |
| *** blahRus has joined #kolla | 16:08 | |
| *** tzn has quit IRC | 16:10 | |
| *** dwalsh has quit IRC | 16:11 | |
| d_code | k. built centos from source, no errors on build…also pushed to registry (version 2.3) | 16:14 |
| d_code | trying deploy | 16:14 |
| d_code | well…prechecks first | 16:14 |
| rhallisey | d_code, k cool | 16:15 |
| rhallisey | I built with centos binary just fine | 16:15 |
| rhallisey | not sure what you were hitting | 16:15 |
| rhallisey | well so far.. | 16:15 |
| d_code | rhallisey: it was a bug that ws commited 2 days ago | 16:15 |
| Serlex1 | problems were at deploy | 16:15 |
| rhallisey | oh I thought it was around building | 16:16 |
| rhallisey | getting there | 16:16 |
| d_code | rhallisey: like I said, it looks like the change was commited…there was a conflict between ansible trying to create a dir and the RPM already having created it | 16:19 |
| rhallisey | gotcha | 16:19 |
| *** neilus has quit IRC | 16:20 | |
| d_code | and #fail | 16:21 |
| d_code | during “Creating the admin project, user, and role” | 16:21 |
| d_code | HTTP 401 | 16:21 |
| d_code | try again w/ -vvv | 16:22 |
| *** dwalsh has joined #kolla | 16:24 | |
| *** jmccarthy1 has quit IRC | 16:29 | |
| d_code | so…anything I try to do it gives me a HTTP 401 | 16:30 |
| *** tzn has joined #kolla | 16:32 | |
| rhallisey | d_code, which service gives you that? Sorry I missed your discussion earlier with sdake | 16:32 |
| d_code | I assume it’s keystone | 16:33 |
| *** CheKoLyN has joined #kolla | 16:33 | |
| d_code | when I run with -vvv… I see that it tries 10 times | 16:34 |
| d_code | here’s the final output: http://paste.fedoraproject.org/321851/45529483/ | 16:34 |
| rhallisey | so a keystone perms error | 16:37 |
| rhallisey | let me see if I can get that too..just finsihing build | 16:37 |
| Serlex1 | oh i thought we had the same issue d_code never mind | 16:43 |
| Serlex1 | mine is a neutron agent container failing tos tart | 16:43 |
| d_code | Serlex1: that’s the error that I did have, then I tried again and got…well…I think I got past it | 16:43 |
| openstackgerrit | Merged openstack/kolla-mesos: Remove a hanging whitespace https://review.openstack.org/279465 | 16:47 |
| Serlex1 | can I ask what the solution was? | 16:48 |
| d_code | uh…I did a couple things. 1. Shutdown docker, 2. rm -rf /var/lib/docker/* | 16:49 |
| d_code | I set selinux to permissive mode | 16:49 |
| d_code | yum update -y | 16:49 |
| d_code | rebooted | 16:49 |
| *** dwalsh_ has joined #kolla | 16:50 | |
| Serlex1 | hmm I've disabled selinux, rebooted, cleaned up containers and tried to deploy | 16:50 |
| Serlex1 | same error | 16:50 |
| d_code | built from scratch using source and pushed to local repo running using docker registry 2.3 | 16:50 |
| Serlex1 | hmm mine is binary and registry 2.3 | 16:51 |
| Serlex1 | I see what Steve can do | 16:51 |
| Serlex1 | I think binary deployment is standard practise | 16:51 |
| *** dwalsh has quit IRC | 16:52 | |
| d_code | I’m happy to try that…just ran into roadblocks…but since that’s fixed, I’ll give that a go | 16:52 |
| d_code | so… kolla-build --base centos --type binary --registry 192.168.4.210:4000 --push --tag=2.0.0 | 17:03 |
| d_code | here we go | 17:03 |
| Serlex1 | thats what failed or succeeded? | 17:11 |
| *** fgimenez has quit IRC | 17:14 | |
| *** tzn has quit IRC | 17:14 | |
| *** Jeffrey4l has quit IRC | 17:15 | |
| *** openstackgerrit has quit IRC | 17:17 | |
| *** openstackgerrit has joined #kolla | 17:17 | |
| d_code | it’s building now | 17:17 |
| *** tzn has joined #kolla | 17:25 | |
| *** aginwala has joined #kolla | 17:25 | |
| *** tzn has quit IRC | 17:26 | |
| *** haplo37 has joined #kolla | 17:27 | |
| openstackgerrit | Ruslan Kamaldinov proposed openstack/kolla: Output image statuses to log instead of return value of main function https://review.openstack.org/278853 | 17:35 |
| *** achanda has joined #kolla | 17:38 | |
| d_code | Serlex1: failed on neutron-dhcp-agent | 17:39 |
| d_code | APIError(HTTPError('500 Server Error: Internal Server Error',),) | 17:39 |
| *** aginwala has quit IRC | 17:39 | |
| *** achanda has quit IRC | 17:39 | |
| *** achanda has joined #kolla | 17:40 | |
| d_code | bbl…. lunch | 17:41 |
| Serlex1 | ok thanks for the update d_code | 17:41 |
| *** achanda has quit IRC | 17:41 | |
| *** achanda has joined #kolla | 17:42 | |
| *** achanda has quit IRC | 17:43 | |
| *** dwalsh_ has quit IRC | 17:43 | |
| *** sdake has joined #kolla | 17:44 | |
| sdake | hello | 17:44 |
| sdake | d_code stil lgoing at it? | 17:44 |
| sdake | back from dr | 17:44 |
| Serlex1 | all well? | 17:47 |
| *** sdake_ has joined #kolla | 17:48 | |
| *** Serlex1 has left #kolla | 17:49 | |
| *** Serlex has joined #kolla | 17:49 | |
| *** sdake has quit IRC | 17:50 | |
| *** gfidente has quit IRC | 17:51 | |
| *** achanda has joined #kolla | 17:54 | |
| *** neilus has joined #kolla | 17:55 | |
| *** jasonsb has quit IRC | 17:58 | |
| *** neilus has quit IRC | 18:00 | |
| openstackgerrit | Dave McCowan proposed openstack/kolla: Use variables to specify http or https when constructing URLs https://review.openstack.org/279707 | 18:00 |
| dave-mccowan | samyaple stdake ^^ i was in a hacking mood | 18:02 |
| dave-mccowan | sdake_ | 18:02 |
| sdake_ | dave-mccowan big patch reviewing | 18:04 |
| sdake_ | is SamYaple in today - the operator cats seem to be uninimous on openstack-operators mailing list they want two vips | 18:04 |
| sdake_ | damn i wish I had recognized this earlier | 18:04 |
| sdake_ | now its a damn fire drill | 18:04 |
| *** achanda has quit IRC | 18:06 | |
| *** aginwala has joined #kolla | 18:12 | |
| sdake_ | dave-mccowan reviewed, errors but looks good | 18:13 |
| *** salv-orlando has joined #kolla | 18:16 | |
| rhallisey | sdake_, how were you able to get your deployment going on a vm? Having issues with the database | 18:17 |
| *** athomas has joined #kolla | 18:18 | |
| sdake_ | i didn't say i did get it going | 18:18 |
| sdake_ | still workign on it | 18:18 |
| rhallisey | roger | 18:18 |
| sdake_ | i had it going last week though | 18:18 |
| sdake_ | but that was stable | 18:18 |
| sdake_ | my suspicion is master is busted. | 18:18 |
| rhallisey | I think my env is wrong here.. | 18:18 |
| rhallisey | sdake_, ya looks like were seeing an issue with neutron | 18:19 |
| rhallisey | my issues is likely vm related.. | 18:19 |
| * dave-mccowan listening. i'm about to start up my first kolla deployment, but i only have a VM. is that known to work, or should i try to track down some real hardware? | 18:19 | |
| sdake_ | my build is faling | 18:19 |
| rhallisey | dave-mccowan, It might work. I'm seeing an issue starting the database | 18:20 |
| *** neilus has joined #kolla | 18:20 | |
| sdake_ | vms work | 18:20 |
| rhallisey | sdake_, where is it failing? | 18:20 |
| sdake_ | rhallisey i am not sure the whole thing blew up | 18:20 |
| sdake_ | but i think my wife unplugged my laptop and it went to sleep so that may have not helped ;) | 18:20 |
| rhallisey | I've never run into the wife bug yet XD | 18:21 |
| sdake_ | dave-mccowan if you want to play around use stable/liberty | 18:21 |
| sdake_ | it works well enough | 18:21 |
| sdake_ | if you want to see bleeding edge, use ubuntu source on master | 18:21 |
| sdake_ | that may work better then centos binary or source | 18:22 |
| sdake_ | you wont notice much difference between liberty and mitaka | 18:23 |
| sdake_ | the code is totally different and much tidier | 18:23 |
| *** ssurana has joined #kolla | 18:23 | |
| sdake_ | but from an interactive perspective the experience is about the same | 18:23 |
| *** dwalsh_ has joined #kolla | 18:24 | |
| *** aginwala has quit IRC | 18:24 | |
| *** openstack has joined #kolla | 18:25 | |
| *** aginwala has joined #kolla | 18:28 | |
| *** achanda has joined #kolla | 18:30 | |
| openstackgerrit | Dave McCowan proposed openstack/kolla: Use variables to specify http or https when constructing URLs https://review.openstack.org/279707 | 18:31 |
| Serlex | hey sdake | 18:32 |
| Serlex | what should I do | 18:32 |
| sdake_ | can you give me a little more context | 18:33 |
| sdake_ | i think your blocked on master until i deploy or dont deploy it | 18:33 |
| sdake_ | another option is you can try ubuntu source build | 18:33 |
| sdake_ | sam doesn't test centos and he was the last one to touch that container | 18:34 |
| sdake_ | it is a possibility he broke it | 18:34 |
| sdake_ | our centos gate is broken | 18:34 |
| sdake_ | so it wouldn't have been caught by the gate | 18:34 |
| *** aginwala has quit IRC | 18:35 | |
| sdake_ | I bought both my kids one of these for christmas and it finally arrived: http://www.digitalstorm.com/aventum-3.asp | 18:36 |
| sdake_ | its almost 4 feet tall | 18:36 |
| sdake_ | and weighs 250 pounds | 18:36 |
| *** Marga_ has quit IRC | 18:37 | |
| *** aginwala has joined #kolla | 18:37 | |
| *** Marga_ has joined #kolla | 18:38 | |
| d_code | sdake_: I was able to get the source build up until it tries to configure the admin project, but I get 401 errors, like it isn’t authenticated to keystone | 18:39 |
| d_code | I retried the binary build, but stuck at neutron-dhcp-agent | 18:40 |
| sdake_ | source doesn't have the dhcp agent problem in centos? | 18:41 |
| d_code | nope | 18:42 |
| d_code | sdake_: https://paste.fedoraproject.org/321940/14553027/ | 18:45 |
| d_code | here’s the verbose output | 18:45 |
| d_code | I assume “keystone | Creating the admin project, user and role” happens after neutron-dhcp-agent | 18:45 |
| d_code | aside from source and binary, the config was the same….allinone, push to registry 2.3, selinux and firewalls off | 18:46 |
| d_code | though, I don’t think the firewall is relevant | 18:46 |
| *** openstackgerrit has quit IRC | 18:47 | |
| *** openstackgerrit has joined #kolla | 18:47 | |
| *** Marga_ has quit IRC | 18:48 | |
| *** Serlex has quit IRC | 18:49 | |
| *** Marga_ has joined #kolla | 19:05 | |
| *** iNeilus has joined #kolla | 19:08 | |
| *** neilus has quit IRC | 19:09 | |
| *** Marga_ has quit IRC | 19:09 | |
| *** Serlex has joined #kolla | 19:10 | |
| *** Serlex has left #kolla | 19:10 | |
| *** Serlex1 has joined #kolla | 19:10 | |
| *** dmsimard has quit IRC | 19:10 | |
| Serlex1 | Ok i will try the source | 19:11 |
| Serlex1 | got to go | 19:11 |
| *** Marga_ has joined #kolla | 19:15 | |
| *** Serlex1 has quit IRC | 19:15 | |
| *** Marga__ has joined #kolla | 19:16 | |
| *** Marga_ has quit IRC | 19:16 | |
| *** aginwala has quit IRC | 19:22 | |
| *** dmsimard has joined #kolla | 19:25 | |
| *** Marga__ has quit IRC | 19:28 | |
| *** Marga_ has joined #kolla | 19:29 | |
| *** sdake_ has quit IRC | 19:31 | |
| *** aginwala has joined #kolla | 19:36 | |
| *** dmsimard has quit IRC | 19:36 | |
| *** mdnadeem has quit IRC | 19:40 | |
| *** mdnadeem has joined #kolla | 19:40 | |
| *** dwalsh_ has quit IRC | 19:54 | |
| *** aginwala has quit IRC | 20:00 | |
| *** aginwala has joined #kolla | 20:02 | |
| *** aginwala has quit IRC | 20:03 | |
| *** achanda has quit IRC | 20:04 | |
| *** aginwala has joined #kolla | 20:07 | |
| *** JoseMello has quit IRC | 20:08 | |
| *** achanda has joined #kolla | 20:10 | |
| *** aginwala has quit IRC | 20:12 | |
| *** aginwala has joined #kolla | 20:13 | |
| *** AJaeger has joined #kolla | 20:24 | |
| AJaeger | could somebody review this liberty change, please? https://review.openstack.org/#/c/279384/ it's the second part of moving from linters to pep8 | 20:25 |
| *** neilus has joined #kolla | 20:43 | |
| *** iNeilus has quit IRC | 20:47 | |
| *** dave-mccowan has quit IRC | 20:48 | |
| *** opennode has joined #kolla | 20:50 | |
| *** aginwala has quit IRC | 20:57 | |
| *** dwalsh has joined #kolla | 20:58 | |
| *** aginwala has joined #kolla | 20:59 | |
| *** salv-orlando has quit IRC | 21:03 | |
| *** dwalsh has quit IRC | 21:15 | |
| *** haplo37 has quit IRC | 21:18 | |
| *** jtriley_ has joined #kolla | 21:22 | |
| *** jtriley has quit IRC | 21:26 | |
| *** aginwala has quit IRC | 21:31 | |
| *** sdake has joined #kolla | 21:35 | |
| *** jtriley_ has quit IRC | 21:42 | |
| -openstackstatus- NOTICE: The infrastructure team is taking gerrit offline for maintenance this afternoon, beginning at 22:00 utc. We should have it back online around 23:00 utc. http://lists.openstack.org/pipermail/openstack-dev/2016-February/086195.html | 21:45 | |
| *** achanda has quit IRC | 21:46 | |
| *** alyson_ has quit IRC | 21:52 | |
| openstackgerrit | Ruslan Kamaldinov proposed openstack/kolla: Output image statuses to log instead of return value of main function https://review.openstack.org/278853 | 21:52 |
| *** aginwala has joined #kolla | 21:54 | |
| sdake | d_code just got back from dentist | 21:54 |
| sdake | give me a moment to read your paste | 21:54 |
| d_code | k | 21:55 |
| sdake | keystone is one of the first things to happen | 21:57 |
| sdake | neutron-dhcp-agent happens later | 21:57 |
| *** achanda has joined #kolla | 21:57 | |
| sdake | it would appear dhcp is broken in centos-binary | 21:57 |
| sdake | sincey ou tried centos-source | 21:57 |
| sdake | would you paste your globals.yml please? | 21:58 |
| sdake | lets see if we can atleast get you workgin with source builds of centos | 21:58 |
| sdake | you ended up with an unauthorized | 21:58 |
| sdake | quick q, are you running ansible deploy as root? | 21:58 |
| d_code | yes | 21:59 |
| sdake | good | 21:59 |
| d_code | gimme a few…I’ll catch up | 21:59 |
| sdake | sounds good | 21:59 |
| -openstackstatus- NOTICE: Gerrit is offline for maintenacne until 23:00 utc | 22:02 | |
| *** ChanServ changes topic to "Gerrit is offline for maintenacne until 23:00 utc" | 22:02 | |
| *** jtriley has joined #kolla | 22:05 | |
| d_code | sdake: https://gist.github.com/dcode/aea450c1648034067637#file-etc-kolla-globals-yml | 22:05 |
| *** ssurana has quit IRC | 22:05 | |
| d_code | I haven’t retried since I did the binary attempt | 22:05 |
| d_code | lemme clean those out and deploy source builds | 22:05 |
| sdake | d_code what is the IP of your machine | 22:05 |
| sdake | hang tight a sec | 22:05 |
| d_code | 192.168.4.210 | 22:06 |
| sdake | ok | 22:06 |
| sdake | that internal address | 22:06 |
| sdake | needs to be a free empty ip address on your network | 22:06 |
| sdake | maybe I should make the docs capitalized there :) | 22:06 |
| d_code | even for all in one? | 22:06 |
| sdake | even for all in one | 22:06 |
| d_code | okeydokey | 22:06 |
| d_code | no problem | 22:06 |
| d_code | sdake: so, I should turn haproxy back on, then? | 22:10 |
| sdake | oh ya | 22:11 |
| d_code | k..did that… | 22:11 |
| sdake | paste again plz | 22:11 |
| d_code | here we go…running | 22:11 |
| sdake | i misseed that part of yoru config | 22:12 |
| sdake | just let me double check one mor etime | 22:12 |
| d_code | https://gist.github.com/dcode/aea450c1648034067637#file-allinone-globals-yml | 22:12 |
| d_code | haproxy disable was hiding at the bottom | 22:12 |
| d_code | I commented it out | 22:12 |
| d_code | sdake: back to keystone error | 22:14 |
| sdake | eno0 is your main interface ip address? | 22:14 |
| d_code | eno1 | 22:14 |
| sdake | right | 22:14 |
| d_code | eno1 = 192.168.4.210 | 22:14 |
| sdake | eno3 is your neutron interface? | 22:14 |
| d_code | eno3 is a 192.168.1.1/24 network that has a router to the interwebs | 22:15 |
| sdake | does eno3 have an IP associated with it? | 22:15 |
| d_code | no | 22:15 |
| d_code | not an IPv4 address…it has the normal default IPv6 that is auto-assigned for the link | 22:15 |
| sdake | you cleaned up your deployment (cleainup-containers) ? | 22:15 |
| d_code | yes | 22:16 |
| sdake | and still getting the error with kolla's toolbox? | 22:16 |
| sdake | about keystone unauthorized | 22:16 |
| d_code | yes | 22:16 |
| * sdake groans | 22:16 | |
| sdake | let me see if my vm ever built the damn containers just a minute | 22:16 |
| d_code | :) thanks | 22:17 |
| sdake | binary didn't build right, building again | 22:18 |
| sdake | should take about 15 mins | 22:18 |
| sdake | laptop went to sleep, that makes build not work ;) | 22:18 |
| sdake | wife keeps unplugging my laptop for whatever reason when i depart the house | 22:18 |
| d_code | lol | 22:19 |
| sdake | i think she is jacking my power | 22:19 |
| sdake | she works at ansible | 22:19 |
| sdake | and we have same power cable but she has her own | 22:19 |
| sdake | "but boy that one is right there..." | 22:20 |
| d_code | hahaha…. my wife does the same | 22:20 |
| sdake | cool | 22:20 |
| sdake | not many women there | 22:20 |
| sdake | i heard only 2 | 22:20 |
| *** bmace has quit IRC | 22:22 | |
| *** bmace has joined #kolla | 22:22 | |
| *** haplo37 has joined #kolla | 22:24 | |
| sdake | d_code what does your wife do may I ask? | 22:25 |
| d_code | she’s a pediatrician…she just has a macbook air and steals my power cables | 22:26 |
| sdake | oh i thought you meant she worked at ansible | 22:27 |
| sdake | but what you meant is she steals your power ;) | 22:27 |
| d_code | right | 22:27 |
| d_code | lol | 22:27 |
| *** rhallisey has quit IRC | 22:30 | |
| sdake | openstack-base is not building for me atm | 22:33 |
| * sdake growls | 22:33 | |
| sdake | moment let me debug this | 22:33 |
| sdake | this explains why my whole build is nto working | 22:33 |
| *** aginwala has quit IRC | 22:38 | |
| sdake | intereesting there is an overlayfs rpm plugin now | 22:39 |
| sdake | d_code I just learned something | 22:39 |
| sdake | if your vm goes to sleep | 22:39 |
| sdake | docker has to be stopped and started | 22:40 |
| sdake | goes to sleep as in you close the lid or something | 22:40 |
| *** achanda has quit IRC | 22:41 | |
| *** aginwala has joined #kolla | 22:42 | |
| SamYaple | sdake: dont freak out about the vip | 22:46 |
| SamYaple | youre blowing up when this was always teh situation | 22:46 |
| SamYaple | this is literally nothing new | 22:46 |
| sdake | i haven't blown anything up | 22:46 |
| SamYaple | once we agreed to the haproxy ssl termintation we _must_ use two vips | 22:47 |
| sdake | wfm | 22:47 |
| SamYaple | before we said the external address was user managed | 22:47 |
| SamYaple | we had this conversation before liberty | 22:47 |
| sdake | SamYaple you may be confused with my frustration with the fact that kolla is afu atm | 22:47 |
| SamYaple | how so? | 22:47 |
| SamYaple | liberty you mean? | 22:47 |
| sdake | not sure, dhcp master doesn't work on centos-binary | 22:47 |
| sdake | keystone master doesn't work on centos source | 22:47 |
| sdake | and its blocking several people from using kolla | 22:48 |
| SamYaple | :( | 22:48 |
| sdake | been debugging since 3am | 22:48 |
| SamYaple | i never use centos | 22:48 |
| SamYaple | i dont know man | 22:48 |
| sdake | thats what I'm frustrated about :) | 22:48 |
| SamYaple | ubuntus working great :( | 22:48 |
| sdake | i understand you dont test centos | 22:48 |
| sdake | i'll sort out centos | 22:48 |
| sdake | but if we are good with two vips, wfm, that is secondary to this centos concern | 22:48 |
| SamYaple | i mean ubuntu stays sorted because I am on top of it. but i dont know anyone "on top of" centos on a regular basis | 22:48 |
| sdake | which I will handle | 22:48 |
| sdake | typically thats me sam | 22:49 |
| SamYaple | two vips is something we that we discussed | 22:49 |
| sdake | but ya, been spotty lately as admitted | 22:49 |
| SamYaple | basically we cant ensure firewalls are firewalling properly | 22:49 |
| sdake | right | 22:49 |
| sdake | cool | 22:49 |
| SamYaple | most people care about that and thats outside our control | 22:49 |
| sdake | at one point you said you didn't want to do it when i uggested it | 22:49 |
| SamYaple | because of the reason above | 22:49 |
| sdake | the operators seem to care about using two vips on the operator list | 22:50 |
| SamYaple | ubt i said if we do haproxy ssl termination (what we agreed to at midcycle) then we MUST do two vips | 22:50 |
| sdake | cool sounds good then | 22:50 |
| SamYaple | sure, they can use two vips | 22:50 |
| SamYaple | do they care about _KOLLA_ managing two vips | 22:50 |
| sdake | i missed hhe two vip requirement on the external ssl | 22:50 |
| SamYaple | that was always teh questions | 22:50 |
| sdake | that wasn't in the notes i took | 22:50 |
| SamYaple | nah it was a condition from liberty midcycle | 22:50 |
| SamYaple | i dont think it was brought back up | 22:50 |
| SamYaple | since i said I would handle it | 22:51 |
| sdake | could have missed it, we made alot of decisions, I tried to capture everthing in notes | 22:51 |
| sdake | but you guys talk faster then i type ;) | 22:51 |
| SamYaple | ill hit up the mailing list to disperse any confusion | 22:51 |
| sdake | sounds good | 22:52 |
| sdake | i posted to the operator list not dev list | 22:52 |
| sdake | not sure if you saw that thread | 22:52 |
| *** tzn has joined #kolla | 22:52 | |
| sdake | all the ops want two vips | 22:52 |
| sdake | whether we manage your right, i didn't ask unfortunately | 22:52 |
| sdake | c_code i've got a binary build going | 22:53 |
| sdake | d_code i've got a binary build going | 22:53 |
| sdake | and its finally working | 22:53 |
| sdake | the docker restart fixed the docker proxy which I htink goes out to lucn hof a vm sleepover | 22:53 |
| sdake | but that is just speculation | 22:53 |
| sdake | certain vm sleep causes some kind of docker problem though | 22:54 |
| SamYaple | i see everythign! | 22:54 |
| SamYaple | sdake: ubuntu recovers from vm/laptop sleep | 22:54 |
| SamYaple | just fyi | 22:54 |
| sdake | imo take more superman pills | 22:54 |
| SamYaple | im aware of this issue | 22:54 |
| sdake | yet its not in the docs :) | 22:55 |
| *** ssurana has joined #kolla | 22:55 | |
| *** daneyon has quit IRC | 22:56 | |
| sdake | d_code have 1 hr meeting which is about how long the images will take to build | 22:56 |
| sdake | i'll let you know where its at when they are done | 22:56 |
| *** tzn has quit IRC | 22:57 | |
| -openstackstatus- NOTICE: Gerrit is offline for maintenance, ETA updated to 23:30 utc | 23:03 | |
| *** ChanServ changes topic to "Gerrit is offline for maintenance, ETA updated to 23:30 utc" | 23:03 | |
| d_code | thanks sdake | 23:06 |
| *** CheKoLyN has quit IRC | 23:07 | |
| *** aginwala has quit IRC | 23:13 | |
| *** blahRus has quit IRC | 23:15 | |
| *** aginwala has joined #kolla | 23:19 | |
| *** neilus has quit IRC | 23:25 | |
| *** neilus has joined #kolla | 23:28 | |
| *** neilus has quit IRC | 23:32 | |
| *** achanda has joined #kolla | 23:36 | |
| -openstackstatus- NOTICE: Gerrit is offline for maintenance, ETA updated to 23:59 utc | 23:38 | |
| *** ChanServ changes topic to "Gerrit is offline for maintenance, ETA updated to 23:59 utc" | 23:38 | |
| *** salv-orlando has joined #kolla | 23:44 | |
| *** achanda has quit IRC | 23:48 | |
| sdake | d_code my containers have built | 23:51 |
| *** salv-orlando has quit IRC | 23:53 | |
| sdake | d_code deploying now | 23:54 |
| *** opennode has quit IRC | 23:54 | |
| d_code | k | 23:55 |
| *** opennode has joined #kolla | 23:55 | |
| *** opennode has quit IRC | 23:55 | |
| sdake | d_code neutron dhcp agent fails to start confirmed | 23:57 |
| sdake | definately some type of docker bug or kolla bug | 23:57 |
| d_code | oh…. oops…. I didn’t open that bug :-/ | 23:57 |
| d_code | cooking dinner now, can get to it later, if needed | 23:58 |
| sdake | i'll open it thanks | 23:58 |
| d_code | sdake: one thing that we were having problems with is getting more info…the run doesn’t produce much and not much in journallctl | 23:58 |
| d_code | bbl | 23:59 |
| sdake | d_code to get the right info you have to ru ndocker in debug mode | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!