Friday, 2016-02-12

*** jmccarthy has quit IRC00:15
*** jmccarthy has joined #kolla00:15
*** Jeffrey4l has joined #kolla00:20
*** salv-orlando has quit IRC00:25
*** aginwala has quit IRC00:29
*** aginwala has joined #kolla00:32
*** aginwala has quit IRC00:33
*** aginwala has joined #kolla00:33
openstackgerritJeffrey Zhang proposed openstack/kolla: Do not pull rabbitmq-data image
*** jasonsb has joined #kolla00:39
*** iNeilus has joined #kolla00:46
*** iNeilus has quit IRC00:52
mandresbezverk, the cpio cap_set_file error makes me think of an issue with your docker storage driver00:52
mandreare you using aufs by any chance?00:53
mandreI suggest you switch to overlayfs or btrfs. devicemapper is the default on centos I believe, it's known to work well too.00:56
mandredon't bother with a regex when building images, just set the build profile to default like this: "kolla-build -b centos -t binary --profile default"00:57
mandrethis way it will only build the images that are required for a kolla deploy and will save you a lot of build time00:58
mandred_code, ^ this is for you too00:58
mandred_code: you generate the kolla-build.conf with "tox -e genconfig"01:02
*** ayoung has joined #kolla01:02
openstackgerritRuslan Kamaldinov proposed openstack/kolla: Output image statuses to log instead of return value of main function
mandred_code, sbezverk: after you've build the image you can do a "kolla-ansible prechecks" to ensure your environment meets the requirements for deploying kolla01:08
*** aginwala has quit IRC01:10
*** aginwala has joined #kolla01:14
*** aginwala has quit IRC01:15
*** aginwala has joined #kolla01:15
*** dolpher has quit IRC01:17
*** dolpher has joined #kolla01:23
openstackgerritNaren Narendra proposed openstack/kolla: Clarify Ansible installation for distros in docs.
*** ssurana has quit IRC01:35
openstackgerritNaren Narendra proposed openstack/kolla: Clarify Ansible installation for distros in docs.
*** jtriley has joined #kolla01:37
openstackgerritMerged openstack/kolla: Make pep8 *the* linting interface
*** tzn has quit IRC01:39
SamYaplemandre: can you just merge this in? d11678a09ef5        registry:8182/kollaglue/ubuntu-source-glance-api:2.0.0                  "kolla_start"            7 hours ago         Up 16 minutes                                  glance_api01:41
SamYapleits been rechecked for a while01:41
SamYapleits solid, but normal gate issue in play01:41
openstackgerritNaren Narendra proposed openstack/kolla: Clarify Ansible installation for distros in docs.
*** alisonh has quit IRC01:42
*** aginwala has quit IRC01:43
mandreSamYaple, ok if you merge
mandreJeffrey4l got his br-ex patch working01:44
*** sdake_ is now known as sdake01:47
SamYaplemandre: merged01:48
SamYaplenow you!01:48
sdakesup nerds01:48
sdakecan someone merge that centos doc patch plz01:48
sdakeso this br-ex patch, jeffrey has fixed it ?01:48
SamYaplesdake: he put in a proper wait, seems to resolve issue01:49
sdakeit just scrolled sam01:49
sdakeok wfm01:49
sdakeas long as its fixed :)01:49
openstackgerritMerged openstack/kolla: Ensuring the openvswitch_db is ready before creating bridges
sdakei didn't have a chance to look at the review queue today - been busy with first day back from travel nonsense01:50
*** aginwala has joined #kolla01:50
openstackgerritMerged openstack/kolla: Use Infra provided mirrors in gate
sdakeand debugging01:53
sdake7pm time to ptfo01:53
sdakeso tierd01:53
mandreso is registry:v1 broken with docker 1.10?01:54
sdaketx SamYaple01:54
sdakemandre roger01:54
sdakemandre i got soemone rewriting the docs atm01:54
sdakepatch shoudl hit  tomorrow01:54
sdakefor that problem01:54
mandregood sdake. docs need some love for sure.01:55
mandrei've seen the midcycle has been productive01:55
sdakenot rewriting the docs01:55
sdakejust that registry section01:55
sdakesince its out  of date01:55
sdakeI sent em a mail and they said htey would do the job01:55
sdakeya we rocked it at the midcycle01:56
sdakegot almost all openstack service updates done01:56
mandreexcellent! what is left?01:56
sdakeheat, working on01:56
sdakeall infrastructure components01:56
SamYapleneutron working on it01:56
sdakewe have a full plan and docs on how to do infra01:56
sdakeoh right and neutron01:56
sdakei did work on heat, I finished the domain users01:57
sdakebut haven't had a chnace ot test01:57
sdakeso not really finsihed01:57
sdakeneed the second patch on top of it to test it really01:57
sdakemandre this is all work that needs to be done01:58
mandredid you confirm docker is not killing the VMs during upgrade?01:58
*** shakamunyi has joined #kolla01:58
sdakewe are punting that to redhat to sort out I think01:58
sdakebut there was some analysis at the midcycle about that01:59
sdakedocker cannot keep track of cgroups libvirt creates01:59
*** daneyon has joined #kolla01:59
mandreyeah, had a look at the etherpad... looks good01:59
sdakesuspicion is libvirt changed how they manage qemu cgroups01:59
SamYaplesdake: thats not what it was02:00
sdakeoh did that get root caused?02:01
SamYapleoh wait you are refering to qemu vs kvm02:01
SamYapleyea qemu is the hpervisor with kvm extentions02:01
sdakei didn't know it was a qemu vs kvm thing02:01
SamYaplewhen running with kvm extentions it is in a different cgroup02:01
sdakei thought i t was a all libvirt qemu processes bust02:01
SamYaplewhen running with tcg (software) then its killed by docker02:02
sdakewithout kvm extensions same?02:02
*** daneyon_ has quit IRC02:02
SamYaplewith kvm everything is a-ok02:02
SamYaplesoftware only its borked02:02
*** alisonh has joined #kolla02:02
sdakewhen was tha figured out02:02
SamYapledays before the midcycle02:02
SamYapleand reiterated during02:02
mandrethanks for the explanation SamYaple02:03
sdakeya that was during our pair programming02:03
mandrethis is reassuring02:03
sdakeok i'm off for the night guys02:03
SamYaplenight sdake02:03
mandrenight sdake02:03
SamYapleme and paul are about to ekko it up02:03
sdakesuper beat, up since 6 am, walked both ways uphill to work today, etc02:03
openstackgerritSam Yaple proposed openstack/kolla: Fix non-root deploys
*** shakamunyi has quit IRC02:12
*** cloudnau_ has joined #kolla02:15
openstackgerritSam Yaple proposed openstack/kolla: Fix detect_distro
*** cloudnau_ has quit IRC02:20
*** tzn has joined #kolla02:23
*** aginwala has quit IRC02:26
*** daneyon has quit IRC02:29
*** aginwala has joined #kolla02:30
*** daneyon has joined #kolla02:30
*** shakamunyi has joined #kolla02:32
openstackgerritSam Yaple proposed openstack/kolla: Fix detect_distro
*** daneyon has quit IRC02:35
openstackgerritSam Yaple proposed openstack/kolla: Fix detect_distro
*** sdake has quit IRC02:40
*** sdake has joined #kolla02:44
*** salv-orlando has joined #kolla02:53
*** dave-mccowan has quit IRC02:54
*** salv-orlando has quit IRC02:55
*** aginwala has quit IRC02:57
*** aginwala has joined #kolla03:00
*** aginwala has quit IRC03:02
openstackgerritSam Yaple proposed openstack/kolla: Fix detect_distro
openstackgerritSam Yaple proposed openstack/kolla: Fix non-root deploys
openstackgerritMerged openstack/kolla: Clarify Ansible installation for distros in docs.
*** dims has quit IRC03:11
*** dims has joined #kolla03:11
*** iNeilus has joined #kolla03:29
*** dims has quit IRC03:31
*** iNeilus has quit IRC03:33
*** dave-mccowan has joined #kolla03:35
*** sdake has quit IRC03:37
*** achanda has joined #kolla03:39
openstackgerritSam Yaple proposed openstack/kolla: Fix detect_distro
openstackgerritSam Yaple proposed openstack/kolla: Fix non-root deploys
*** unicell has quit IRC03:40
*** achanda has quit IRC03:41
*** sdake has joined #kolla03:44
*** achanda has joined #kolla03:46
*** sdake has quit IRC04:03
*** dave-mccowan has quit IRC04:28
*** salv-orlando has joined #kolla04:37
*** salv-orlando has quit IRC04:42
*** alyson_ has quit IRC05:09
*** alyson_ has joined #kolla05:09
*** Slower has quit IRC05:11
*** Slower has joined #kolla05:11
*** iNeilus has joined #kolla05:17
*** Jeffrey4l has quit IRC05:19
*** iNeilus has quit IRC05:21
*** aginwala has joined #kolla05:25
*** achanda has quit IRC05:38
*** Jeffrey4l has joined #kolla05:41
*** tzn has quit IRC05:44
*** unicell has joined #kolla05:52
*** salv-orlando has joined #kolla05:54
*** salv-orlando has quit IRC05:56
*** achanda has joined #kolla05:57
openstackgerritJeffrey Zhang proposed openstack/kolla: Fix image plugin functionality for oslo.config
openstackgerritJeffrey Zhang proposed openstack/kolla: fix the custome profile raise exception issue
*** aginwala has quit IRC06:01
*** aginwala has joined #kolla06:04
openstackgerritJeffrey Zhang proposed openstack/kolla: Make pep8 *the* linting interface
*** achanda has quit IRC06:09
*** achanda has joined #kolla06:11
*** achanda has quit IRC06:20
*** achanda has joined #kolla06:21
*** aginwala has quit IRC06:25
*** achanda has quit IRC06:28
*** opennode has joined #kolla06:33
*** aginwala has joined #kolla06:35
*** tzn has joined #kolla06:40
*** aginwala has quit IRC06:58
*** aginwala has joined #kolla07:00
*** iNeilus has joined #kolla07:05
*** iNeilus has quit IRC07:10
*** salv-orlando has joined #kolla07:12
*** tzn has quit IRC07:14
*** salv-orlando has quit IRC07:23
*** opennode has quit IRC07:51
elemoine_ajafo: how is going?07:53
ajafostill fight with kolla to make it working :)07:53
ajafobut got some ideas so I'll test it today :)07:54
ajafoand what about you?07:54
elemoine_good thanks07:56
*** achanda has joined #kolla07:58
*** fgimenez has joined #kolla08:03
*** tzn has joined #kolla08:11
*** opennode has joined #kolla08:15
*** aginwala has quit IRC08:17
openstackgerritJeffrey Zhang proposed openstack/kolla: Remove the openstack_release option in the globals.yml file
*** opennode has quit IRC08:21
openstackgerritJeffrey Zhang proposed openstack/kolla: Remove the openstack_release option in the globals.yml file
*** athomas has joined #kolla08:25
elemoine_Jeffrey4l: can I easily reproduce  I'd like to understand when/how this problem occurs.08:27
openstackLaunchpad bug 1544545 in kolla "kolla-toolbox container failed and stick on create stage" [Undecided,In progress] - Assigned to Jeffrey Zhang (jeffrey4l)08:27
*** akwasnie has joined #kolla08:28
*** salv-orlando has joined #kolla08:28
Jeffrey4lelemoine_, yes. I am writing on the lauchpad. I will ping back you when i finished.08:28
elemoine_ok, sorry for insisting08:28
elemoine_I will run in the same kind of problems with Heka and the log Unix socket, so I want to fully understand this08:29
Jeffrey4lgot it.08:30
*** aginwala has joined #kolla08:31
*** gfidente has joined #kolla08:33
*** gfidente has joined #kolla08:33
*** macsz has joined #kolla08:34
*** salv-orlando has quit IRC08:35
*** aginwala has quit IRC08:36
*** mbound has joined #kolla08:38
*** tzn has quit IRC08:44
*** jmccarthy1 has quit IRC08:48
elemoine_do we have unit tests for
*** jmccarthy1 has joined #kolla08:55
Jeffrey4lelemoine_, ^ check it.08:56
elemoine_thanks, I'll have a look08:57
Jeffrey4lwe have no unittest for file now. But we should and need add ut for it.08:58
elemoine_I found a bug in and I was wondering if I should write tests08:58
Jeffrey4lelemoine_, you can just fix the bug only. But it will be perfect if you can add some ut for this.09:00
openstackgerritAlicja Kwasniewska proposed openstack/kolla: Added Elasticearch and its deployment.
*** salv-orlando has joined #kolla09:05
openstackgerritEric Lemoine proposed openstack/kolla: Fix kolla_docker check_volume
elemoine_Jeffrey4l: fix submitted09:07
*** sbezverk has quit IRC09:11
*** sbezverk has joined #kolla09:12
*** aginwala has joined #kolla09:14
*** aginwala has quit IRC09:18
*** achanda has quit IRC09:27
*** Jeffrey4l has quit IRC09:28
*** kproskurin has joined #kolla09:36
kproskurinGuys, looks like we have a problem with compatibility between kolla and kolla-mesos. Kolla *always* runs kolla_extend_start: and we cant workaround it in any way. And, for example, in kolla_extend_start of horizon we have a md5 sum thingy with horizon configs. But in kolla-mesos we dont have any configs yet, they about to be created after.09:39
kproskurinI’d really like to see this kolla_extend_start somehow triggered by ansible var or something, so we could skip it09:40
*** unicell has quit IRC09:47
*** neilus has joined #kolla09:47
openstackgerritMarek Zawadzki proposed openstack/kolla: Added note about root permissions, removed sudos.
openstackgerritMarek Zawadzki proposed openstack/kolla: Added note about root permissions, removed sudos.
*** salv-orlando has quit IRC09:52
*** neilus has quit IRC09:52
*** sdake has joined #kolla09:57
*** sdake_ has joined #kolla10:00
*** rhallisey has quit IRC10:01
*** sdake has quit IRC10:03
*** salv-orlando has joined #kolla10:03
*** athomas has quit IRC10:20
*** sdake_ is now known as sdake10:25
*** openstackgerrit has quit IRC10:32
*** openstackgerrit has joined #kolla10:32
*** tzn has joined #kolla10:34
*** aginwala has joined #kolla10:35
*** aginwala has quit IRC10:40
*** salv-orl_ has joined #kolla10:41
*** dims has joined #kolla10:41
*** akwasnie has quit IRC10:42
*** salv-orlando has quit IRC10:45
kproskurinsdake: hi10:48
ajafodoes neutron and nova should use v2 or v3 in communication with keystone?10:49
ajafosdake: morning10:49
kproskurinsdake: I found a silly problem with kolla vs kolla-mesos compatibility. Kolla *always* runs kolla_extend_start: and we cant workaround it in any way. And, for example, in kolla_extend_start of horizon we have a md5 sum thingy with horizon configs. But in kolla-mesos we dont have any configs yet, they about to be created after.10:49
openstackgerritSteven Dake proposed openstack/kolla-mesos: Remove a hanging whitespace
openstackgerritMerged openstack/kolla-mesos: Make pep8 *the* linting interface
sdakeok folks10:50
sdakegive me a minute to get my day rolling10:50
sdakewoke up early for some reason today10:50
kproskurin“a bit early”10:50
sdakeprobably has something to do with passing out at 6pm last night from too much travel :)10:50
elemoine_not a good time for bothering sdake with questions I guess :)10:51
sdaketime is ok just dont expect brilliance ;)10:51
sdakeso extend start10:51
sdakeis always meant to be run10:51
sdakeit is meant to be overridden per container from the main start script10:52
sdakethe problem kolla-mesos has is what, it is doing some work on configs that are not present yet?10:52
sdakedid I hear that right10:52
kproskurinYep, we will fetch configs from ZK after, from kolla_mesos_start10:52
sdakeajafo all of kolla is keystone v3, but nova should be using keystone v3 with neutron auth, but I'll double check the default configs - moment10:53
kproskurinAnd sometimes we maybe want to have a bit different bootstrap of our own, but right now we forced to always run kollas first :-)10:53
sdakeso I believe at one time we agreed in custom bootstrap provided by the infrastructure of choice10:54
sdakeworking in the same way as config-external works now10:54
sdakein mesos case, we sort of do config-internal there, but its the same concept10:54
ajafosdake: thx, I've little problems in ubuntu it's looks like neutron/nova try to communicate with v2 and I don't know why at this moment10:55
sdakeso we agreed, but there wasn't a pressing need to do the job10:55
kproskurinWell, horizon broke things for us right now10:55
sdakeajafo do you mean when nova launches a vm it communicates with nova in keystone v2?10:55
sdakekproskurin the answer is a prototype implementation10:56
sdakethe works like config-external10:56
sdakeexcept with the extend_start script10:56
sdakei've got an awful lot on my plte but I could prototype it for you, then you could port the tree10:56
ajafosdake: no, when I try to login to horizon it tries to communicate with nova and neutron, and then I see in neutron and nova logs that it use v2 links and got errors10:56
sdakeI think finishing by the 4th will be difficult10:56
sdakeajafo roger, so horizon is using keystone v2 then?10:57
sdakecan you nova boot a vm?10:57
sdakekeystone v2 has been removed from opentack iiuc10:57
ajafobut horizon as I see in logs is using v3 it's why I'm little confused10:57
sdakelet me double check configs10:58
sdakesecond pulling repo10:58
kproskurinsdake: I not 100% sure what are you talking about with “config-external” thingy. Maybe I missed this disscussion. Im really open to any ideas about this situation. I’d liek to hear nihilifer opinion about this too10:58
sdakeajafo can you give me a few minutes to work this out with kproskurin, then I'll be all ours?10:58
ajafoI'm trying to setup centos env to compare environments my suspection is about api-paste.ini10:59
ajafosdake: no problem, take your time just asked :)10:59
ajafoand thanks :)10:59
sdakekproskurin read that file please11:00
kproskurinsdake: done...11:01
sdakewhat happens on bootstrap is ansible reads that file and copies all the stuff in config_files11:01
sdakethe proposal we had at the tokyo summit was to add extend_start to that file so it could be externally bootstrapped11:01
sdakeand the tool thatcopies that file inside the vm would run the extend_start code11:01
kproskurinsdake: sounds good for me11:02
sdakeasalkeld asked for it, we agreed to it, nobody ever did the work11:02
kproskurinso it will be ansible-onlu thing11:02
sdakethats where its at at the moment11:02
sdakeno, kolla-mesos also uses these json files I think11:02
sdakejust copies them from a different source11:02
sdakeatleast kolla-mesos should be using these json11:03
kproskurinWe copy it in a bit different way, so we copy all tmpl from kolla, but populate them via kolla_mesos_start11:03
kproskurinso we dont have this jsons11:03
kproskurinonly files from this jsons11:03
kproskurinAs you can see, we run bootstrap here too, and will this kolla one we run bootstrap twice %-)11:05
sdakekproskurin what is this
sdakewhat is the meaning of bootstrap:
kproskurinIt’s our call of bootstrap script. So then kolla-mesos-start starts, it reads this config, and run all commands by their dependancies. So mariadb need mariadb/bootstrap first, so it look this bootstrap section and run this script with this env. So we litteraly run bootstraping twice in kolla-mesos. First time via from kolla, and second time via our start script.11:07
kproskurinI just found that yesterday11:07
sdakethat needs to be fixed11:08
kproskurinThats why im here :-)11:08
sdakecompletely unsuitable solution11:08
sdakeso it runs this bootstrap section one time right?11:09
*** akwasnie has joined #kolla11:09
kproskurinwith run_once: True11:09
sdakethis yml file is run by ansible?11:09
sdakeor this is anguss custom config script like the json above i linked11:09
kproskurinSo if you move this extend_start to this json thing, it will be totaly ok for us if I understand everything right11:09
kproskurinthis file is not run by ansible at all11:10
sdakeyou use an ansible namespace name run_once11:10
sdakewhy I ask :)11:10
kproskurinit stored in ZK node and fetched by our start script11:10
sdakedoes kolla-mesos need overridden as well?11:10
kproskurinWell, fetch our start script and runs it - all ok. But in the middle of it it runs extend_start which is not ok. :-)11:11
sdakegot a link to the code that runs extend_start11:12
sdakekproskurin file a blueprint please11:12
sdakecall it "custom-extend-start"11:12
sdakenot sure if we can delive rby the 4th got alot on our plate11:12
kproskurinOk, but I think I should disscuss it with nihilifer first, not sure if he will be online today11:13
kproskurinSince he is core in both projects11:13
sdakekolla-mesos bootstrap code is running kolla_extend_start?11:13
sdakeits one project - kolla11:14
sdaketwo repos11:14
kproskurinOk sry. We could run extend_start or any other script if we want.11:14
kproskurinAnd call it bootstrap or anything11:14
kproskurinit’s just a chain of commands with deps11:14
sdakeif you look here:11:15
kproskurinbut yes 95% of the time its just a kolla bootstrap11:15
sdakelets talk about general case not bootstrap11:15
sdakebootstrap is wrong way to look at it11:15
sdakethere are othe rcases where a custom kolla_extend_start is needed11:15
sdakeso line 711:16
sdakewhat does that in kolla-mesos?11:16
sdakeor does that kolla_set_configs contain mesos's bootstrap codebase as well?11:16
kproskurinLine 7 is kolla_set_configs. We launch this container with this: KOLLA_CONFIG={ "command": "kolla_mesos_start", "config_files": [ { "source": "zk://", "dest": "/usr/local/bin/kolla_mesos_start", "owner": "root", "perm": "0755" } ]}11:16
kproskurinSo line 7 fetch out start script from ZK node11:16
kproskurinit’s all ok11:17
kproskurinAfter at line 12 it runs kolla bootstrap and we cant avoid it. And in the like 15 it launch our start script11:17
kproskurinour start scrip fetch configs related to this app11:17
kproskurinAnd run commands11:17
kproskurinWe add bootstrap commands in most of the apps, since we missed what its allready forced running in start.sh11:18
kproskurinSo we run them twice11:18
sdakegive me 5 minutes to process all that11:19
kproskurinSolution what I like is to move this extend_start from to somethere else, there only ansible run could do it. So we could use our own calls of bootstrap(using kolla scrips in 95% of cases and ouw own if needed)11:19
kproskurinMaybe kolla_start should run extend_start first for example11:20
sdakekproskurin  you said line 7  of kolla_set_configs launches the container, I'm looking and line 7 of kolaset_configs nd its a comment11:21
kproskurinno, I said line 7 of fetch our start script11:21
sdakeok well break it  down one more level for me please11:22
kproskurinkproskurin: Line 7 is kolla_set_configs. We launch this container with this: KOLLA_CONFIG={ "command": "kolla_mesos_start", "config_files": [ { "source": "zk://", "dest": "/usr/local/bin/kolla_mesos_start", "owner": "root", "perm": "0755" } ]}11:22
sdakeya you dont have to cut and paste11:22
sdakewhat I eman is where in the set-configs is said container launched11:22
sdake4:20am... still booting11:22
kproskurinmy english is not perfect too, sry, maybe I descripe it bad11:23
sdakeyour doing well11:23
sdakekeep at it11:23
kproskurinCould you pls tell me what confuses you about all this?11:23
sdakeso you gave me abunch of detail about whwat line 7 does11:23
kproskurinAnd I will go in detail about it11:23
kproskurinok, 1 min11:23
sdakebut line 7 launches set_configs.py11:24
sdakeif I could get some links there, that would be helpful11:24
sdake(in docker/base directory)11:24
kproskurinSo kolla has in base container. And line 7 of this script is call of kolla_set_config.11:24
kproskurinWe run each container with this env: KOLLA_CONFIG={ "command": "kolla_mesos_start", "config_files": [ { "source": "zk://", "dest": "/usr/local/bin/kolla_mesos_start", "owner": "root", "perm": "0755" } ]}11:24
kproskurinSo kolla_set_config is fetch our kolla_mesos_start script11:25
kproskurinAll ok for now11:25
*** akwasnie has quit IRC11:25
sdakehow does that fetching work11:25
sdakepointers to line #s11:25
sdakeI need to understand how kolla-mesos is bootstrapping11:25
sdakebefore I can offer a proper solution11:25
sdakeand we will get here by you answering th equestion about how set_configs launches the container11:26
kproskurinkolla_set_config checks env here: and we pass this env var to container with this data: KOLLA_CONFIG={ "command": "kolla_mesos_start", "config_files": [ { "source": "zk://", "dest": "/usr/local/bin/kolla_mesos_start", "owner": "root", "perm": "0755" } ]}11:27
kproskurinHere it understands what this is ZK path:
kproskurinand it need to fetch it from ZK11:28
kproskurinAnd we pass all needed info for this in this env11:28
*** achanda has joined #kolla11:28
kproskurinSo it fetch our script: kolla_mesos_start and put it in /usr/local/bin/kolla_mesos_start11:28
kproskurinThis part is ok? Or need some details?11:28
sdakewhere is the fetching done11:29
kproskurinThis func:
sdakeneed more details i'll ask qs ou answer, we will get here11:29
kproskurinand this check:
sdakeso this here
kproskurinShould I go on?11:30
sdakeneed few minutes to process11:30
sdakei had always envisioned the json file as being common between different implementations of underlays11:32
*** Serlex1 has joined #kolla11:32
sdakeasalkeld has made a forked version11:32
sdakecan you link his mariadb script again11:32
kproskurinone sec11:33
sdakethis is the json parsed by set_configs11:34
*** achanda has quit IRC11:34
sdake(w euse json because it requires no yml depenedency on the host)11:34
sdakeI dont mind if asalkeld uses yml11:34
sdakehere is the problem11:35
sdakeneeds to go in lines 20-2311:35
kproskurinWell, not really. Its just a bit different structure.11:36
sdakegalera.cnf.j2 is the ssame json file format as ansible11:36
sdakeI want that ABI to be the same between projects11:36
sdakeat midcycle we agreed with nihilifer in room that anywhere we could have common architecture we would between repos or commo ncode we would11:37
sdakeif we couldn't then variance is permitted11:37
sdakethis is a  case where there can be common code11:37
Serlex1wow sdake, do you not sleep?11:38
kproskurinoh, yeah, thats to asalkeld11:38
Serlex1which time zone you live in?11:38
sdakeSerlex1 super man!!11:38
sdakei went to bed at 6-7 last night11:38
sdakeand wake up super early unfortunately11:38
Serlex14:38am there -_-11:38
sdakekproskurin no I was talking to you11:38
kproskurinsdake: I understand, but asalkeld is the man who do such calls in kolla-mesos11:39
sdakedo which calls11:39
sdakeyou mean code calls11:39
sdakeor decision making11:39
sdakedecision making happens by consensus in community not by asalkeld's calls :)11:40
sdakewe made decision by consensus that architecture where it can be should be the same11:40
sdakewhich means line 13 needs to go to line 2011:40
sdakein some way11:40
kproskurinIn new to opensource, was 10 years in corporate enslavery, so maybe I chose not the right words. :-)11:42
sdakedude no sweat11:42
sdakeok so lets get a blueprint filed11:42
sdakeso we can start getting work items filed on it11:42
sdakefirstwork item is unifyign command across kolla-mesos and kolla-ansible11:43
sdakekproskurin will you file or shall I11:43
Serlex1In a AIO environment, docs says I need two interfaces. I have a host-only adapter and NAT adapter. However I'm not sure how the networking on globals.yml should be configured11:43
sdakehost-only adapter is connected tothe internets?11:44
kproskurinsdake, I thinks it’s better if you file it11:45
kproskurinif you dont mind11:45
sdakekproskurin your wish is my command ;)11:46
openstackgerritEric Lemoine proposed openstack/kolla: Add Heka log decoder for RabbitMQ
openstackgerritEric Lemoine proposed openstack/kolla: Add Heka log decoder infrastructure
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect RabbitMQ logs
openstackgerritEric Lemoine proposed openstack/kolla: Add Heka log decoder for OpenStack
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect Nova logs
openstackgerritEric Lemoine proposed openstack/kolla: Add Heka log decoder for MariaDB
openstackgerritEric Lemoine proposed openstack/kolla: Add Heka log decoder for Keystone/Apache
openstackgerritEric Lemoine proposed openstack/kolla: Add a Dockerfile for Heka
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect Keystone logs
openstackgerritEric Lemoine proposed openstack/kolla: Add Heka to common role
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect HAProxy and Keepalived logs
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect kolla-toolbox logs
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect MariaDB logs
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect Glance logs
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect Neutron logs
openstackgerritEric Lemoine proposed openstack/kolla: Make Heka collect Swift logs
Serlex1Adapter 1 is host-only with dhcp enabled on default range 172.28.128.X. Adapter 2 is NAT with 10.0.3.X. Services are bind to IP and docker is registered to
Serlex1Also do I need to do clean up or some sort of reset if a kolla-deploy fails?11:49
Serlex1I can see that it skips pass anything that is already configured, however it seems to have failed on starting neutron-dhcp-agent and disconnected my ssh session11:49
*** sbezverk has quit IRC11:51
*** JoseMello has joined #kolla11:51
*** aginwala has joined #kolla11:53
*** aginwala has quit IRC11:58
kproskurinsdake: tx!12:00
sdakeSerlex1 bio then i'm with ajafo for a bit then i'll be with you ok?12:00
sdakekproskurin please read through the blueprint make corrections where necessary or edit the whiteboard with comments12:00
*** Marga__ has joined #kolla12:00
Serlex1yeah thats fine sdake, I'm in no rush to get it working12:02
*** Marga_ has quit IRC12:03
sdakeajafo still around?12:04
ajafosdake: I need little investigation to have question because at this moment I've 3 env's and have different error on everyone :/12:04
ajafosdake: so at this moment I'll not take your time, and if you let me I'll try ask some later12:05
ajafoif you don't mind12:05
sdakeajafo ok wfm12:05
sdakeyou may hae to work with someone else i have a dr appointment this morning12:05
sdakebut i'm sur eosmeone will be around that can get you going12:05
sdakeok serlex thats get rolling12:05
ajafosdake: ok thx12:05
sdakeSerlex1  you havea couple issues12:07
sdakelets work em one by one12:07
sdakeSerlex1 you said docker is running on 10.0.3.z12:07
sdakebut your publi interface is 172.x.y.z12:07
sdakedocker needs to run on the public interface12:07
sdakeonce kolla starts up, your 10.0.3 network will be used by neutron and unusable by anything else12:08
sdake10.0.3 i think is what you plan to use as your neutron interface, is that correct?12:08
*** dave-mccowan has joined #kolla12:08
sdakemorning dave-mccowan12:09
dave-mccowangood morning sdake!12:09
Serlex1hang on let me check few things sdake12:09
*** kproskurin has quit IRC12:10
*** kproskurin has joined #kolla12:10
Serlex1Ok I was wrong, 10.0.3. interface is my default route out12:12
Serlex1172.x.y.z is the host-only adapter with DHCP enabled12:12
Serlex1[root@localhost kolla]# route12:19
Serlex1Kernel IP routing table12:19
Serlex1Destination     Gateway         Genmask         Flags Metric Ref    Use Iface12:19
Serlex1default         UG    0      0        0 enp0s812:19
Serlex110.0.3.0   U     0      0        0 enp0s812:19
Serlex1172.17.0.0     U     0      0        0 docker012:19
sdakepaste your globals.yml please12:19
sdakenot in channel12:19
sdakeuse a past service12:19
Serlex1its the default at the moment12:19
sdakeok well default doesn't work, you need atleast 4 config options12:19
sdakefirst Q, are you using a registry12:19
Serlex1I've restored the vm to a snapshot post kolla-build, pre deploy12:20
Serlex1local registry12:20
*** Jeffrey4l has joined #kolla12:22
sdakedid all your containers build?12:22
*** intr1nsic has quit IRC12:22
Serlex1docker image count is 4212:23
sdakeajafo coudl you do me a favor please12:23
ajafosdake: yes?12:23
sdakeajafo do you hae a current deployment that is active?12:24
sdakedoesn't have tow ork just has to be active12:24
ajafodocker images | grep ubuntu | wc -l12:24
ajafosdake: yes12:24
Serlex1haha guessing now all is built then12:24
sdakeajafo dave-mccowan really needs the file from the haproxy container haproxy.cfg12:24
Serlex1deploy failed on ip configuration not lack of images though12:24
ajafosdake: one second12:25
sdakeajafo if you need help with docker commands to get that file otu let me knwo12:25
*** intr1nsic has joined #kolla12:25
sdakeSerlex1 if your on centos, you can yum install fpaste and then use fpaste /etc/globals.yml12:25
dave-mccowansdake ajafo  nevermind.  the template is fine.  i found it.12:25
openstackgerritProskurin Kirill proposed openstack/kolla-mesos: Improve logging of cleanup script
sdakedave-mccowan so i think what that comment means in external-ssl is the communication to the haproxy endpoint from internally is nto private12:27
sdakethat is why its an attack vector12:27
sdakeif someone is sniffing your public HAPproxy VIP12:28
sdakesometimes they will get plaintext passwords12:28
sdakethe internal services don't just magically get ssl because the haproxy server is setup right?12:29
dave-mccowanright, the internal services still use plaintext, but should be using an internalURL.12:31
sdakei am pretty sure they use internal-url12:31
sdakethe problem I think kolla suffers from is we don't hvae an internl-url endpoint for haproxy that is different from the external endpoint12:32
sdakeits all one endpoint (an external one)12:32
sdakeI'm not totallly convinced using two VIPs makes total sense though12:33
sdakeone for internal and one for external12:33
sdakethe risk is small, I was just curious on your thinkingon the matter12:34
sdakeif soemone has managed to sniffy your external api endpoint from the internal network, you probably havebig troubles ahead12:34
*** aginwala has joined #kolla12:35
sdakeajafo thanks bro12:36
sdakeSerlex1 ping me when yoru back and we can get your globals.yml into shape12:37
ajafosdake: n/p12:37
sdakedave-mccowan  ^^12:37
dave-mccowansdake i just noticed that looking at kolla's template.  i'm used to seeing two networks.  the public network, with the public URLs, and an internal network with the admin and internal URLs.12:38
openstackgerritMerged openstack/kolla-mesos: We forgot to add mesos section to config
sdakeso our model is you put a NAT in front of the VIP and that gives you security12:38
sdakeonly ports are accessible via the NAT to the haproxy API endpoint ports12:39
sdakewhich should be all rock solid secure12:39
*** aginwala has quit IRC12:39
*** liyi has joined #kolla12:41
dave-mccowankeeping the same VIP for internal and external URLs, mean that you'll need to use different port numbers.  keystone public-secure will bind to 5000 and keystone internal-open will then bind to 5001, for example.12:41
*** rhallisey has joined #kolla12:41
dave-mccowani like the approach of two VIPs better.  then only standard port numbers are needed.  use TLS only on the public VIP, and use plaintext only on the internal/admin VIP.12:43
liyihi, today kolla deployment require target host has python&docker-py installed. I am wondering if this could be removed by using kolla_toolbox as the jumpstart container. I.e. running ansible script inside the container. Is this feasible or just crazy?12:46
*** aginwala has joined #kolla12:48
sdakedave-mccowan i'm not sure that is how keystone binds12:48
openstackgerritProskurin Kirill proposed openstack/kolla-mesos: Fix ubuntu config path search
sdakewe bind to 5000 and 355somethinroanother12:49
sdakellyi 5:49 am12:49
sdakellyi brain booting12:49
sdakeajafo roger12:50
dave-mccowanpublic_ip:5000, internal_ip:5000, admin_ip:3535712:50
rhalliseyliyi, I mean it's not completely crazy because we technically use kolla_toolbox similarly, but I think there we would need to have quite an evaluation for that change given it will affect the way we operate12:51
sdakellyi so i can tell you mitaka-3 is out of the question for such a change12:51
rhalliseyliyi, I mean docker-py cant be so bad :)12:51
sdakenewton, hey, anything can happen ;)12:51
dave-mccowankolla is making the requirement that public_ip==internal_ip==admin_ip.  right?12:51
liyithanks guy!12:51
sdakerhallisey distros like coreos don't have a python runtime12:52
liyii am happy that i was not the only crazy one12:52
liyii believe evetually, the host could use OS like Atomic12:52
sdakedave-mccowan nope my public_endpoint is, my internal_endpoint is my IVP endpoint, and admin_ip uses my VIP endpoint12:53
rhalliseyliyi, yes and just have docker there12:53
sdakethe downside of toolbox container to launch all that stuff is it would be docker on docker12:53
rhalliseyliyi, it's possible.  It would require an extensive evaluation and refactor, but I think it could work12:53
*** aginwala has quit IRC12:53
liyithere will a very neat system, right rhallisey?12:53
liyilooking forward to seeing that to happen :D12:54 is a NAT gateway which forwrads to my internal endpoint IP12:54
sdakedave-mccowan make sense?12:54
liyisdake, I know it is too late for mitaka3. but  do we have a plan for this already, right?12:55
sdakellyi sounds like a bluepritn needs to be filed and placed in the discussion state12:56
sdakellyi doesn't atomic have  a python runtime?12:56
dave-mccowanso the TLS termination needs to happen between the NAT gateway and the internal endpoint.12:56
sdakethis is just my setup12:56
sdakebut i think this is a secure model12:57
sdakebut nto being a networking rocket scientist, not 100% certain if this is how people secure their networks12:57
liyii am not sure about atomic. But i know other systems in that style only have docker engine12:57
sdakeor just suck  itu p and burn wo ips addresses12:57
sdakellyi right12:57
liyifor example docker-machine, rancherOS12:57
sdakewhen did docker-machine turn into an OS :)12:58
*** salv-orl_ has quit IRC12:58
liyisorry, i took it wrong12:58
sdakellyi do yo uknwo how ot file a blueprint12:58
liyibut rancheros is12:59
liyiyes, kind of12:59
liyido u want me to do it?12:59
rhalliseyliyi, ya file a bp12:59
sdakerhallisey need to know for certain if atomic has a python runtime as well as docker-py - since /usr is readonly13:00
sdakecan you verify that13:00
liyino problem, I will give it a try and back to u later.13:01
sdakedave-mccowan most people htat deploy kolla really suffer through the "what the hell is a VIP" thing :)13:01
sdakethrowing two in the mix sounds like more pain for them13:01
*** dwalsh has joined #kolla13:01
sdakewith two VIPs I guess we could have one on an external network and one on an internal network, and not have to worry about NAT13:01
sdakedave-mccowan is that the thinking between the two VIPS?13:02
dave-mccowansdake: i added an example:
dave-mccowandid i get this right?13:02
dave-mccowanSerlex1: what is the output of your "keystone endpoint-list | grep 5000" ?13:04
sdakedave-mccowan i dont have a deployment active atm unfortunately13:04
sdakedave-mccowan i have been in dev and my env is afu13:04
sdakebut that looks corrrect13:04
sdakethat looks like a 1 VIP model13:05
sdakenot 2 VIP example13:05
sdakewhat is server operator?13:05
sdakedave-mccowan ok one delta13:06
sdakepublicURL on my machine is broked.self-ip.net13:06
sdakepublicURL should be a DNS address13:06 is some address on the internet (my gige connection)13:07
sdakeand my wireless NATs to
sdake(its actualy in my environment but same idea)13:07
dave-mccowansdake the import thing is what keystone says the public url is.13:09
sdakedave-mccowan note the above woudl work (your publicUrl) but then the haproxy endpoint would be totally wide open and require firewall13:09
sdakei have extensively worked on that code13:10
sdakein kolla13:10
sdakeI know for certain it will say
sdakeif I make it its on the ame network as my management network13:11
sdakethen there is no way to isolate the management network and the API network13:11
sdakeopenstack uses internalurl to communicate internally, but when you first connect to openstack it uses publicurl to find the correct endpoint into the cluster13:12
sdakeyou probably already know that :)_13:12
sdakethe only way it could find my network for my bare metal gear from my laptop is to do that NAT thing i spoke up13:13
sdakealso we use keystone v313:13
sdakeso those configs ren't quite right13:13
*** thumpba has joined #kolla13:13
ajafook I've question maybe someone will know13:13
rhalliseysdake, docker-py is on atomic13:14
*** thumpba has quit IRC13:14
sdakerhallisey thanks13:14
*** thumpba has joined #kolla13:14
sdakethen we dont hae to worry about atoic with illy's case13:14
rhalliseypython-docker-py-1.1.0-2.fc22.noarch in fedora-atomic13:14
sdakejust rancheros and coreos13:14
rhalliseymaybe we could have a container to handle his case13:15
sdakei dont think atomic should have a python runtime fwiw13:15
rhalliseybut I feel like if we go down that road why not make it universal13:15
ajafowhen I log in into horizon I get CRITICAL keystonemiddleware.auth_token [-] Unable to validate token: Unable to establish connection to   it looks like identity_uri is not set insted of it we've auth_url and auth_uri13:15
sdakebut i didn't make i t ;)13:15
ajafothe same in nova13:15
sdakeajafo please paste your globals.yml in a paste service13:16
* rhallisey brb13:16
sdakei am pretty sure identity_uri is deprecated13:16
sdakeajafo can you paste the top part of the logs and see if we are getting any config depcration warnings13:17
* sdake groans13:19
dave-mccowansdake.  i updated the etherpad.  the public url is "kolla_external_address".  but, per haproxy.cfg.j2, haproxy is not listening on that address?13:19
*** thumpba has quit IRC13:19
sdakedave-mccowan right, its on the operator to sort out how to get the external address to map to the internal address13:19
sdakevia dns, nat, whatever13:20
ajafosdake: right it is in logs
dave-mccowanby the same logic, TLS termination would also be on the operator.13:20
sdakeajafo are you on stable/liberty or master?13:21
ajafosdake: I'm using master with ubuntu build so images build for liberty13:22
sdakedave-mccowan  i wasn't make a logic statement, I was stating where the community is on this point13:22
ajafosdake: but sthg like 1 month ago it was working13:22
sdakeI think its illogical personally :)13:22
sdakebut touching on the point of ssl termination on operator13:23
sdakethis is because the certs must match the host names?13:23
dave-mccowanyea, that's cool.  i didn't mean it to be a judgement.  trying to wrap my head around how to make it work.13:23
sdakei htink the problem we will suffer with is the certs wont match the dns names13:24
sdakeand there will be problems with that13:24
dave-mccowansdake... definitely, the certs need to match the dns.13:24
dave-mccowanalso, sdake somehow real-public-address needs to map to something different than kolla_internal_address13:25
sdakeso with nat, how does that work?13:25
sdakewhere is real-public-address from?13:25
sdakekolla_internal_address is a minomer, it is actually the kolla VIP13:26
sdakedave-mccowan  right it does in my environment ;)13:26
sdakebut I use NAT13:26
sdakeso here is my thinking on all of this13:26
sdakewe want what is best for the operators13:26
sdakeif the operators want two VIPs one external one internal I htink we should proceed that way13:27
sdakeif operators only one one VIP with NAT as their only option, we should roll that way13:27
britthouser sdake: typically NAT rewrites the DNS packets13:27
britthouserTwo VIPs is what is recommended by security guide13:27
sdakeyes, I had asked earlier if nat damages SSL13:27
sdakebritthouser have a link13:27
* britthouser digs13:28
sdakebecause if nat rewrites dns in some way that makes external SSL not work, then we have to go to two VIPs13:28
sdakewhich I can live with13:28
britthouserhave a look at teh BIG diagram13:28
sdakebritthouser  thanks13:28
britthouserAPI and Mgmt are two different networks13:29
britthouserBut I agree how operators actually run should be our deciding factor.13:30
sdakeemail going  out to operator's list right now13:30
*** achanda has joined #kolla13:32
*** tzn has quit IRC13:33
dave-mccowansdake if we want to support TLS with a one-VIP option, then we'll need to get creative with ports.  it looks like now publicURL and internalURL are aliases of one another.  if we want one to support TLS and one to not, then they need to be different... either different IP or different port.13:34
*** _tzn has joined #kolla13:35
*** achanda has quit IRC13:38
sdakepublic_IP is not supposed to be a different ip13:38
sdakeits supposed to be a different dns name13:38
sdakesorry public_url13:38
sdakethe operators I believe are supposed to sort out how to get that DNS name to map to the internal VIP13:38
sdakeso that requirement you mention above dave-mccowan is already mentioned (but probably not in the docs ;)13:39
sdakepublicURL and internalURL should *not* be aliased to one another except for developers playingaround13:39
*** kproskurin has quit IRC13:39
sdakeor people doing evals13:39
dave-mccowansdake the requirement is not to get it to the internalVIP it is getting the the internal representation of the public VIP.13:40
sdakethey need to be isolated networks to maintain security13:40
sdakeyes dave-mccowan familiar with how keystone works13:40
*** aginwala has joined #kolla13:40
sdakethe reason they need to be different is to isoalte the networks13:41
sdakeajafo file bugs witht he deprecation warnigns please13:42
sdakeajafo we fix deprecations quickly :)13:42
*** aginwala has quit IRC13:44
sdakeelemoine_ yes please please write unit tests for all python code13:46
sdakeit should be a requirement going forward13:47
sdakebut I realize we were ina bit of emergency management mode with the whole docker 1.8.2 and needing docker 1.1013:47
sdakedave-mccowan  reread your statement avove and parsed it slighlty differently13:48
sdakeI think what your syaing there is in my environment, my NAT does the mapping of publicURL to internalip13:48
dave-mccowansdake yes13:49
sdakeso our current design relies on NAT absolutely and completely with no other options13:49
sdakeI went over this with SamYaple back and forth for months on end and he said he didn't really know how the networks were setup but it was on the operators13:49
sdakeso I came up with nat, and was happy with that13:49
sdakebut if nat isn't going to work with external SSL - that is no-beuno13:49
sdakeSamYaple and I never discussed that I recall two VIPs, one for internal and one for public13:51
sdakedave-mccowan will external-ssl work in a NAT environment?13:52
dave-mccowansdake NAT is no problem, and perhaps a distraction to our discussion.13:53
dave-mccowansdake.  check the etherpad.  i have examples of a one VIP and VIP solution.13:54
Serlex1dave-mccowan - I've discarded the earlier deploy went retored snapshot after kolla-built13:56
Serlex1sdake - I'm ready to sort out this globals.yml13:57
*** dwalsh has quit IRC13:57
*** clayton has quit IRC14:00
sdakedave-mccowan what si your public email address14:01
sdakei am going to cc you on this operator thread14:01
dave-mccowani use my work address, dmccowan@14:01
dave-mccowansdake do the examples i added to the etherpad help clarify?14:06
sdakecan you link the etherpad14:06
sdakei've got like a million windows open not sure which one you mean14:06
sdakeoh duh that one14:07
openstackgerritMerged openstack/kolla: Do not pull rabbitmq-data image
d_codeso, I checked out master yesterday on a CentOS7 box, built from source without a registry, then tried to deploy to all-in-one (using kolla-ansible deploy), and it failed on “Starting neutron-dhcp-agent”14:08
sdakeprocessing igve me fe wminutes14:08
sdakeSerlex1 i'll be with you in a moment ok?14:08
d_coderunning `docker logs neutron_dhcp_agent` produces nothing.14:08
Serlex1thats cool14:08
d_codeerror is: msg: APIError(HTTPError('500 Server Error: Internal Server Error',),)14:08
sdaked_code so the container didn't even start?14:09
sdakedid you yum update before doing that?14:10
d_codesdake: as far as I can tell, I guess….14:10
sdakewhat type of file storage backend are you using?14:10
d_codedidn’t do a yum update prior to that14:11
ajafod_code: what version docker and docker_py have you got?14:11
d_codestorage backend……whatever is default?14:11
ajafoit's full error or is sthg else?14:11
d_codeDocker version 1.10.0, build 590d510814:11
d_codedocker-py Version: 1.7.014:12
*** clayton- has joined #kolla14:12
ajafoI'm trying to build centos containers becuase ubuntu stop working for me14:12
d_codethe rest is just ansible telling me it failed and changed = true14:12
ajafobut I downgrade docker to 1.9.1 and docker-py to 1.6.014:12
*** salv-orlando has joined #kolla14:13
sdakedave-mccowan ya gotta say I don't like lines 58-68 one bit14:14
sdakethe implications of that are massive14:14
sdakeas in pain producing :)14:14
britthouserdave-mccowan: would a 4th alternative be enabling TLS for internal as well?  that would simplify 56-68, right?14:15
d_codesdake: looks like journalctl has more info:
britthouserbut then I imagine all containers would need the keys14:16
sdaked_code type "df" | fpaste14:16
sdakewithout the quotes14:17
sdakeSerlex1 can you get your globals.yml in an fpaste14:17
sdakeSerlex1 i've lost all the context we had in our discussion earlier because of the drop off14:17
sdakei've got 1.5 hours then need to jet to dr appointment for a bit14:17
dave-mccowansdake i don't see pain.  pain to kolla?  the keystone catalog manages the internal vs. external endpoints for us, it just a matter of touching up all the registers.14:18
d_codewow…fpaste is new to me…neat14:18
*** jtriley has quit IRC14:18
Serlex1pastebin if thats ok
sdaked_code getenforce ?14:20
sdakeno bueno14:20
sdaketurn that off please14:20
d_codethat’s no in the instructions ;-)14:20
sdake /etc/selinux/config14:20
sdakeit used to be14:20
d_codeyes, I’m well aware of how to do it14:20
sdakewell i'll make sure the selinux thing makees it back into the instructions14:21
d_codefor production release, is there any effort to get selinux policies functional?14:21
sdaked_code we want to start down that path soon14:21
sdakethe reason is not what you migh tthink14:21
sdakekeystone community has indicated they will use kolla for R&D of keystone (rather then devstack)14:21
sdakethey are blocked on lack of selinux14:22
sdakethis is a use case we want to explore14:22
sdakebut not our primary objective14:22
d_codeit’s a requirement for my production build when I get there in July14:22
Serlex110.0.3.15 (interface enp0s8 which is NAT) is how I reach out to the internet. enp0s8 (host-only adapter with DHCP enabled on
d_codeI’m pretty well acquainted with selinux14:22
d_codenot so much with openstack, but given that it seems to have made it that far14:22
sdakecan't promise working selinux by july - we only have one dev with exeprience with it , rhallisey and he is pretty swamped with other htings atm14:23
sdaked_code we do take commits and I can help you learn how to commit properly and all that ifyou sort out the patchess14:23
sdakei just dont knwo selinux14:23
sdakethat is why i can't promise it14:23
sdakebecause I don't know the scope of the work14:23
d_codealso…there are no selinux denials…so that didn’t cause my issue…  neutron still failed to run14:25
d_codeI’ll look into the code to see if I can trace this error14:25
*** dmsimard has quit IRC14:25
sdakebusy busy14:26
sdakeSerlex1 and you did buidl oraclelinux distro iamges from source?14:26
sdakeya binary sorry14:27
sdakeok first problem line 2114:27
sdakeThis should be a VIP, an unused IP on your network14:27
sdakedo yo uhave any unused ip addresses on your network?14:28
sdakeit should not be your host's IP address14:28
Serlex1i set an from the DHCP interface, done14:28
Serlex1an IP14:28
sdakeok so you have an unused I Padddress, and set line 21 to it?14:28
sdakedelete line 3014:29
sdakethat is not how kolla works and it makes setup and debug harder14:29
Serlex1lol ok deleted14:29
sdakeif you want haproxy to be optional, write the code to make it all work optioanlly14:29
sdakethat is all :)14:29
Serlex1haha ok ok14:30
sdakeSerlex1 that was targeted at the cor ereviewers not you14:30
sdakethey read the logs14:30
sdakeyou have a docker registry?14:30
sdakeline 49 should be network_interface: "whatever is on"14:31
sdakeline 61 should be whatever your other 172 network is on14:31
sdakei see yoru running stable/liberty14:31
sdakeis that correct?14:32
Serlex1how do I check sorry14:32
sdakehow did yo uget the software14:32
sdakego to git directory where code is located14:33
sdakeand type git status | fpaste14:33
*** macsz has left #kolla14:33
openstackgerritProskurin Kirill proposed openstack/kolla-mesos: Zookeeper expects strings as values
*** akwasnie has joined #kolla14:35
sdakehey akwasnie14:35
Serlex1i dont have fpaste setup on this oracle box14:35
Serlex1hang on14:35
Serlex1[root@localhost kolla]# git status14:36
Serlex1# On branch master14:36
Serlex1nothing to commit, working directory clean14:36
sdakeakwasnie can you give me a run down of where you stand with diags and elemoine_ integration?14:36
sdakeSerlex1 ok docker images | fpaste14:37
sdakeare you using registry v1?14:38
akwasniehey sdake14:39
Serlex1sdake im sligtly confused. "(2:31:15 PM) sdake: line 49 should be network_interface: "whatever is on"14:39
Serlex1But globals.yml says14:39
Serlex1When running an All-In-One14:39
Serlex1     19 # without haproxy and keepalived, this should be the first IP on your14:39
Serlex1     20 # 'network_interface' as set in the Networking section below.14:39
Serlex1     21 kolla_internal_address: ""14:39
sdakelook at your paste14:40
Serlex1shouldn't network_interface be whatever 172.28.1.x IP14:40
sdakenot at the globals.yml in master14:40
sdakea network interface i slike eth0, enp0s4, etc14:40
sdakeip link show will list your interface14:40
sdakeits the interface names14:40
akwasniesdake: I prepared Elasticsearch patch (Dockerfile+ansible) and discussed with Eric that we will begin its integration with Heka after Elasticsearch merge14:41
sdakeakwasnie ok what do we need to get unblocked then?14:41
sdakea review of your work?14:41
akwasniein the meantime I prepared also Logstash, so we can test both solutions (Heka and Logstash)14:41
akwasniesdake: yes, review14:41
d_codesdake: I switched to registry v2, but I was having issues at another point in the deploy…so I switched to the docker container like you suggested14:42
sdakezomg big patch14:42
sdakedont have time before dr, but i'll review today akwasnie14:42
sdakedocker registery 2.3?14:42
akwasniesdake: ok, thanks14:42
sdakei have seen your error with older docker registries14:43
sdakebut 2.3 should deploy14:43
sdakeyou may have to rebuild unfortunately14:43
sdakedoc fixes are in progress a tm14:43
*** cfarquhar has quit IRC14:43
sdakeyou said you switched to the docker container like i suggested, you mean 2.3?14:44
sdakedcode ^^14:44
sdaked_code ^^14:44
d_codemaybe I should just clear out the whole box and start from scratch14:44
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Add neutron config
sdakelets keep as is14:45
sdakeyo uturned off selinux right?14:45
d_codeI’m keeping a record of what I do…in the hopes I’ll eventually have something that works14:45
sdaketurn off firewalld14:45
*** cfarquhar has joined #kolla14:45
*** cfarquhar has quit IRC14:45
*** cfarquhar has joined #kolla14:45
*** dwalsh has joined #kolla14:45
sdaketurn off iptables14:45
d_codewell…no firewalld, but I have iptables14:45
sdakei thought this was all documented14:45
sdakei think people must have removed it14:45
*** jtriley has joined #kolla14:46
sdakeyour on centos without firewalld?14:46
sdakeminimal install ?14:46
*** _tzn has quit IRC14:46
sdakeok you need to yum udpate your kernel14:46
d_codealready did that14:46
sdakei'd just yum updat everything14:46
Serlex1sdake - what should neutron_external_interface be?14:46
sdakeSerlex1 whatever your 172 (not your normal internet address) is mapped to in your host14:47
sdakeip link show | grep 17214:47
sdakethat should give you a few choices to look at :)14:47
*** akwasnie has quit IRC14:47
sdaked_code cool14:47
d_codesdake: I basically did this script, starting from a minimal install, removed firewalld, replaced with iptables-services, and yum updated, then reboot14:47
d_codegonna add turn off iptables to that for now14:47
sdakei actually leave iptables on14:48
sdakeis it regisstery 2.3.0 or registery 2.3 on line 30?14:48
sdakeI'm not sure :)14:48
*** JoseMello has quit IRC14:48
sdakelet me look at dockerhub14:48
Serlex1ok and then deploy? :D14:48
sdakeserlex paste your globals.yml pls14:48
d_codesdake: I copied and pasted what you put last night14:49
d_codebut it installed 2.3.014:49
Serlex1i must say you are a machine sdake14:49
Serlex1thanks for the hlep14:49
d_codelooks like 2.3.0 is the latest14:50
sdaked_code 2.3 and 2.3.0 are both versions14:51
d_codeah…see that now14:51
sdakein the future, i'd switch that script around to 2.314:51
d_codek…will do14:51
sdakebut  for now14:51
sdakekeep as is14:51
sdakeok clenaup-containers14:51
sdakerun script14:51
sdakerun kolla-ansible deploy14:51
sdakepaste typescript file created14:51
d_codeI don’t need to redo the build, right?  everything seemed like it built from source just fine in the local docker cache14:52
d_codelike I said, I had issues pushing to registry14:52
sdakeSerlex1 need your docker images | fpaste14:52
sdaked_code shouldn't need to14:52
sdakeoh your stuck on ./build --push?14:52
d_codeI figured if I could deploy all in one, then I could figure out the registry thing and get it working on multiple hosts14:52
d_codeI skipped push since I’m doing all in one14:53
d_codedoes that not work14:53
sdakewith master without a registry?14:53
sdakeserlex1 do tools/cleanup-containers then try a deploy14:54
sdakeyour config looks good14:54
d_codesdake: yes…master without registry to do all-in-one14:54
sdaked_code ok there was a recent change to master14:54
d_codehaha…. bleeding edge….love it14:54
sdakeI am nto sure, but someone may ahve removed the ability to run without a registry14:54
sdakeI dont know14:55
sdakeso lets try it14:55
sdakethen I'll know14:55
*** clayton- is now known as clayton14:55
sdakeI wasn't sure - I saw the patch review fly by my email14:55
d_codek…just lost connection to my host…  lemme switch modes to fixing that :p14:55
sdakethe core team is punshing me with vacation this week :)14:55
*** mbound has quit IRC14:55
sdaked_code yu will want to do cleanup-containers tho14:56
sdakejust ot blow away whatever mess you have with the selinux stuff atm14:56
sdakei think selinux was your problem14:56
sdakeyour script is essentially what I do when I setup a host but I do it manually14:56
*** Marga__ has quit IRC14:56
sdakehaving that script in the repo would be great :)14:56
d_codeI’m not so sure, but I’ll do it without and see where it goes14:56
rhalliseyI'll see if I can come up with some policy for us real quick14:56
d_codeI didn’t see any denials14:56
d_codesdake: happy to contribute…I’ve done the same for a couple of other projects14:57
*** Marga_ has joined #kolla14:57
*** dave-mccowan has quit IRC15:01
sdaked_code what I want is that as a playbook but I'll take a script to begin ;)15:02
d_codewell…I can do that too15:04
d_codegenerally how I write my playbooks….do it manually…record it all in a text file….then automate with tool like Ansible15:05
d_codewell…Ansible or Chef…depending on what I’m writing…obviously ansible makes more sense for this :)15:05
*** dwalsh has quit IRC15:08
d_codesdake: cleanup-images too?15:08
Serlex1failed: [localhost] => {"changed": false, "cmd": ["docker", "exec", "haproxy", "/usr/local/bin/kolla_ensure_haproxy_latest_config"], "delta": "0:00:00.643014", "end": "2016-02-12 15:07:59.630687", "rc": 137, "start": "2016-02-12 15:07:58.987673", "stdout_lines": [], "warnings": []}15:09
d_codeso, by default, storage will just go somewhere in /var/lib/docker/ as a file, right?15:09
sdaked_code keep images in shape15:09
sdaked_code right15:09
sdaked_code dont delete images15:09
d_codeoh boy….yeah…so something on my box isn’t quite right15:09
sdakeok i know how to fix15:10
sdakesudo systemctl stop docker15:10
sdakesudo rm -rf /var/lib/docker15:10
d_codeI lost connectivity because got a general protection fault in the kernel when cleaning containers15:10
sdakesudo reboot15:10
sdakeya your docker cache is busted15:10
sdakeyou probably were on docker 1.8.3 and 1.10 at different times15:10
d_codethat….may have happened15:10
sdakedocker blows at version upgrades15:10
sdakebut getting better15:11
sdakeif you had 1.10 and then ran 1.83 with same /var/lib/docker15:11
sdakeyour cache was corrupted15:11
d_codek…makes sense15:11
sdakeyou can just delete /var/lib/docker15:11
d_codek…  rebooting :)15:11
sdakeand it will recreate teh files15:11
d_codethen I’ll clean it up15:11
sdakeand rebuild15:11
sdakeyou just cleaned it up :)15:11
sdakethats the nuclear option with docker dev :)15:12
*** iNeilus has joined #kolla15:12
sdakemuch faste rthen cleanup-c-ontainers too if you dont mind losing your images :)15:12
Serlex1sorry sdake, typo, deploy still running15:12
d_codeno…I had a kernel panic…I’m rebooting so that I can cleanup15:12
sdakeoh roger15:12
sdakeya kernel panic15:13
sdakeuntil runc comes around15:13
sdakenot much kolla can do about kernel and docker misbheaviors15:13
Serlex1I see there is a new docker version just pushed out15:14
sdakea stable version?15:14
Serlex1[stack@localhost kolla]$ docker -v15:14
Serlex1Docker version 1.10.1, build 9e8376515:14
sdakecool should be better15:15
Serlex1ok my deploy failed again on neutron-dhcp-agent15:15
d_codeyeah…that’s what I was having15:15
Serlex1I'm clearly a chump on understanding the IPs15:15
*** tzn has joined #kolla15:15
sdakeSerlex1 paste ip link show and your globals.yml file plz15:16
sdakeserlex1 what type of failure did you get on the neutron-dhcp-agent?15:16
sdakeit is possible neutron-dhcp-agent is broken15:16
Serlex1TASK: [neutron | Starting neutron-dhcp-agent container] ***********************15:16
Serlex1failed: [localhost] => {"changed": true, "failed": true}15:16
Serlex1msg: APIError(HTTPError('500 Server Error: Internal Server Error',),)15:16
*** iNeilus has quit IRC15:16
sdaketwo reproductions15:16
sdakeseems like a legit problem15:16
sdakefolks I can't test any of kolla atm because my lab is bust15:17
sdakeI can test one node vm15:17
*** dave-mccowan has joined #kolla15:17
sdakebut been so busy on the irc channel this last week haven't had an opportunity to build15:17
sdakeserlex1 type getenforce15:17
d_codewe appreciate your presence, sdake15:17
Serlex1disabled sdake15:18
sdakei've got to go to the doc soon, let me start a build15:18
sdakeserlex1 can you ping from your vm?15:19
sdakeor whateveryour deploying from15:19
Serlex1neutron_external_interface: "enp0s3" <--- this interface is a host-only interface with DHCP enabled on virtualbox15:20
Serlex1I hope that helpss15:20
Serlex1172.X range15:20
*** sdake_ has joined #kolla15:22
sdake_serlex1 which images are you building, centos ones?15:23
sdake_rather d_code15:23
*** dwalsh has joined #kolla15:23
d_codek…/var/lib/docker is detonated….rebooting and trying again15:24
Serlex1my images are oraclelinux15:24
*** sdake has quit IRC15:24
Serlex1Where can I find more verbose logs for that error?15:25
d_codeSerlex1: I couldn’t find anything besides what was logged in journalctl15:29
d_codewhich just said what it was doing, then it failed15:29
Serlex1bloody hell15:30
Serlex1which images you using?15:30
sdake_i am building now15:30
sdake_it is possible this is a centos specific  ergression15:30
d_codeI’m not an expert on oracle, but there really aren’t that many difference between the two, afaik15:31
*** absubram has joined #kolla15:31
sdake_our gate is busted for centos so we have been ignroing the deploy of centos a bit until we can fix the gate15:31
d_codek…I built my images with selinux on…I’ve turned it off, blown away docker, I’ll try rebuilding from source again15:31
Serlex1they are both based on rhel right? so yeah I dont think there is much difference15:31
*** daneyon has joined #kolla15:32
*** absubram_ has joined #kolla15:32
elemoine_akwasnie, sdake, it would be good to merge Elasticsearch to be able to start integration, Heka is ready15:32
sdake_d_code building images with selinux on will make no difference15:34
sdake_d_code so you have built from source and built from binary and hae this same regression?15:34
d_codeI thought binary build from CentOS isn’t supported currently15:35
d_codethere’s a missing package15:35
d_codeor couple15:35
sdake_well kolla doesn't really support anything15:35
sdake_organizations provide support for kolla15:35
sdake_we implement15:35
d_codeI thought binary build using CentOS was not currently functional15:35
*** absubram has quit IRC15:36
*** absubram_ is now known as absubram15:36
sdake_it may not be because of a problem in nova_api - the gate fails on deploy15:36
sdake_but that may be a gate problem rather the na nova problem15:36
d_codelemme check the logs…someone said that RDO changed a package, I think15:36
sdake_i'll let you know15:36
*** achanda has joined #kolla15:36
d_codeoh yeah…it’s that the RPM creates a dir that ansible expects to create and errors15:36
rhalliseyalso building..15:36
d_codeI applied the patch that’s in launchpad, but it didn’t fix my issue15:36
sdake_d_code got link to bug ?15:37
d_codeyeah…sec…it’s one of these open tabs :p15:37
*** absubram has quit IRC15:37
sdake_i've got to leave in 12 minutes15:37
*** rhallisey has quit IRC15:38
*** rhallisey has joined #kolla15:38
rhalliseyd_code, I'm building centos binary images now I'll see if I hit the issue15:38
Serlex1sdake do you think I should continue troubleshooting this or ack its an issue on kolla side?15:38
sdake_serlex1 two peopel have same problem15:39
sdake_seems like a bug should be filed and mark confirmed to me15:39
sdake_that is typicallly how it goes ;)15:39
sdake_i suspect the kolla_docker module is busted15:39
Serlex1ok - up for it d_code?15:39
sdake_but it could be a missing bindmount for centos thatis present on ubuntu15:40
sdake_who knows15:40
sdake_it could still be user error15:40
openstackLaunchpad bug 1543417 in kolla "nova-compute bootstrap failing on centos-binary" [Critical,Fix released] - Assigned to Martin André (mandre)15:40
sdake_but i think you both are doing everything i know how to do to fix things up15:41
*** achanda has quit IRC15:41
d_codelooks like that was commited15:42
d_codealso…just want to express….  Launchpad is awful….that is all….I won’t mention it again15:43
*** dmsimard has joined #kolla15:43
elemoine_d_code are you a GitHub user? ;)15:43
d_codeI am15:43
Serlex1ok sdake I will ping back here next week or something15:43
d_codesdake_: I’ll build from source again and post the bug15:44
sdake_d_code thanks15:47
*** JoseMello has joined #kolla15:47
sdake_ok dr appointment15:48
sdake_back in hour or two15:48
sdake_i should have results by then15:48
sdake_d_code when yo u file bug link i nchannel and i'll sort out getting it set rightin the tracker15:48
*** aginwala has joined #kolla15:54
*** tzn has quit IRC15:58
*** aginwala has quit IRC15:58
*** sdake_ has quit IRC15:58
*** salv-orlando has quit IRC16:01
*** tzn has joined #kolla16:04
*** stvnoyes has quit IRC16:07
*** neilus has joined #kolla16:08
*** stvnoyes has joined #kolla16:08
*** blahRus has joined #kolla16:08
*** tzn has quit IRC16:10
*** dwalsh has quit IRC16:11
d_codek. built centos from source, no errors on build…also pushed to registry (version 2.3)16:14
d_codetrying deploy16:14
d_codewell…prechecks first16:14
rhalliseyd_code, k cool16:15
rhalliseyI built with centos binary just fine16:15
rhalliseynot sure what you were hitting16:15
rhalliseywell so far..16:15
d_coderhallisey: it was a bug that ws commited 2 days ago16:15
Serlex1problems were at deploy16:15
rhalliseyoh I thought it was around building16:16
rhalliseygetting there16:16
d_coderhallisey: like I said, it looks like the change was commited…there was a conflict between ansible trying to create a dir and the RPM already having created it16:19
*** neilus has quit IRC16:20
d_codeand #fail16:21
d_codeduring “Creating the admin project, user, and role”16:21
d_codeHTTP 40116:21
d_codetry again w/ -vvv16:22
*** dwalsh has joined #kolla16:24
*** jmccarthy1 has quit IRC16:29
d_codeso…anything I try to do it gives me a HTTP 40116:30
*** tzn has joined #kolla16:32
rhalliseyd_code, which service gives you that?  Sorry I missed your discussion earlier with sdake16:32
d_codeI assume it’s keystone16:33
*** CheKoLyN has joined #kolla16:33
d_codewhen I run with -vvv…  I see that it tries 10 times16:34
d_codehere’s the final output:
rhalliseyso a keystone perms error16:37
rhalliseylet me see if I can get that too..just finsihing build16:37
Serlex1oh i thought we had the same issue d_code  never mind16:43
Serlex1mine is a neutron agent container failing tos tart16:43
d_codeSerlex1: that’s the error that I did have, then I tried again and got…well…I think I got past it16:43
openstackgerritMerged openstack/kolla-mesos: Remove a hanging whitespace
Serlex1can I ask what the solution was?16:48
d_codeuh…I did a couple things. 1. Shutdown docker, 2. rm -rf /var/lib/docker/*16:49
d_codeI set selinux to permissive mode16:49
d_codeyum update -y16:49
*** dwalsh_ has joined #kolla16:50
Serlex1hmm I've disabled selinux, rebooted, cleaned up containers and tried to deploy16:50
Serlex1same error16:50
d_codebuilt from scratch using source and pushed to local repo running using docker registry 2.316:50
Serlex1hmm mine is binary and registry 2.316:51
Serlex1I see what Steve can do16:51
Serlex1I think binary deployment is standard practise16:51
*** dwalsh has quit IRC16:52
d_codeI’m happy to try that…just ran into roadblocks…but since that’s fixed, I’ll give that a go16:52
d_codeso…  kolla-build --base centos --type binary --registry --push --tag=2.0.017:03
d_codehere we go17:03
Serlex1thats what failed or succeeded?17:11
*** fgimenez has quit IRC17:14
*** tzn has quit IRC17:14
*** Jeffrey4l has quit IRC17:15
*** openstackgerrit has quit IRC17:17
*** openstackgerrit has joined #kolla17:17
d_codeit’s building now17:17
*** tzn has joined #kolla17:25
*** aginwala has joined #kolla17:25
*** tzn has quit IRC17:26
*** haplo37 has joined #kolla17:27
openstackgerritRuslan Kamaldinov proposed openstack/kolla: Output image statuses to log instead of return value of main function
*** achanda has joined #kolla17:38
d_codeSerlex1: failed on neutron-dhcp-agent17:39
d_code APIError(HTTPError('500 Server Error: Internal Server Error',),)17:39
*** aginwala has quit IRC17:39
*** achanda has quit IRC17:39
*** achanda has joined #kolla17:40
d_codebbl….  lunch17:41
Serlex1ok thanks for the update d_code17:41
*** achanda has quit IRC17:41
*** achanda has joined #kolla17:42
*** achanda has quit IRC17:43
*** dwalsh_ has quit IRC17:43
*** sdake has joined #kolla17:44
sdaked_code stil lgoing at it?17:44
sdakeback from dr17:44
Serlex1all well?17:47
*** sdake_ has joined #kolla17:48
*** Serlex1 has left #kolla17:49
*** Serlex has joined #kolla17:49
*** sdake has quit IRC17:50
*** gfidente has quit IRC17:51
*** achanda has joined #kolla17:54
*** neilus has joined #kolla17:55
*** jasonsb has quit IRC17:58
*** neilus has quit IRC18:00
openstackgerritDave McCowan proposed openstack/kolla: Use variables to specify http or https when constructing URLs
dave-mccowansamyaple stdake ^^ i was in a hacking mood18:02
sdake_dave-mccowan  big patch reviewing18:04
sdake_is SamYaple in today - the operator cats seem to be uninimous on openstack-operators mailing list they want two vips18:04
sdake_damn i wish I had recognized this earlier18:04
sdake_now its a damn fire drill18:04
*** achanda has quit IRC18:06
*** aginwala has joined #kolla18:12
sdake_dave-mccowan reviewed, errors but looks good18:13
*** salv-orlando has joined #kolla18:16
rhalliseysdake_, how were you able to get your deployment going on a vm?  Having issues with the database18:17
*** athomas has joined #kolla18:18
sdake_i didn't say i did get it going18:18
sdake_still workign on it18:18
sdake_i had it going last week though18:18
sdake_but that was stable18:18
sdake_my suspicion is master is busted.18:18
rhalliseyI think my env is wrong here..18:18
rhalliseysdake_, ya looks like were seeing an issue with neutron18:19
rhalliseymy issues is likely vm related..18:19
* dave-mccowan listening. i'm about to start up my first kolla deployment, but i only have a VM. is that known to work, or should i try to track down some real hardware?18:19
sdake_my build is faling18:19
rhalliseydave-mccowan, It might work.  I'm seeing an issue starting the database18:20
*** neilus has joined #kolla18:20
sdake_vms work18:20
rhalliseysdake_, where is it failing?18:20
sdake_rhallisey i am not sure the whole thing blew up18:20
sdake_but i think my wife unplugged my laptop and it went to sleep so that may have not helped ;)18:20
rhalliseyI've never run into the wife bug yet XD18:21
sdake_dave-mccowan if you want to play around use stable/liberty18:21
sdake_it works well enough18:21
sdake_if you want to see bleeding edge, use ubuntu source on master18:21
sdake_that may work better then centos binary or source18:22
sdake_you wont notice much difference between liberty and mitaka18:23
sdake_the code is totally different and much tidier18:23
*** ssurana has joined #kolla18:23
sdake_but from an interactive perspective the experience is about the same18:23
*** dwalsh_ has joined #kolla18:24
*** aginwala has quit IRC18:24
*** openstack has joined #kolla18:25
*** aginwala has joined #kolla18:28
*** achanda has joined #kolla18:30
openstackgerritDave McCowan proposed openstack/kolla: Use variables to specify http or https when constructing URLs
Serlexhey sdake18:32
Serlexwhat should I do18:32
sdake_can you give me a little more context18:33
sdake_i think your blocked on master until i deploy or dont deploy it18:33
sdake_another option is you can try ubuntu source build18:33
sdake_sam doesn't test centos and he was the last one to touch that container18:34
sdake_it is a possibility he broke it18:34
sdake_our centos gate is broken18:34
sdake_so it wouldn't have been caught by the gate18:34
*** aginwala has quit IRC18:35
sdake_I bought both my kids one of these for christmas and it finally arrived:
sdake_its almost 4 feet tall18:36
sdake_and weighs 250 pounds18:36
*** Marga_ has quit IRC18:37
*** aginwala has joined #kolla18:37
*** Marga_ has joined #kolla18:38
d_codesdake_: I was able to get the source build up until it tries to configure the admin project, but I get 401 errors, like it isn’t authenticated to keystone18:39
d_codeI retried the binary build, but stuck at neutron-dhcp-agent18:40
sdake_source doesn't have the dhcp agent problem in centos?18:41
d_codehere’s the verbose output18:45
d_codeI assume “keystone | Creating the admin project, user and role” happens after neutron-dhcp-agent18:45
d_codeaside from source and binary, the config was the same….allinone, push to registry 2.3, selinux and firewalls off18:46
d_codethough, I don’t think the firewall is relevant18:46
*** openstackgerrit has quit IRC18:47
*** openstackgerrit has joined #kolla18:47
*** Marga_ has quit IRC18:48
*** Serlex has quit IRC18:49
*** Marga_ has joined #kolla19:05
*** iNeilus has joined #kolla19:08
*** neilus has quit IRC19:09
*** Marga_ has quit IRC19:09
*** Serlex has joined #kolla19:10
*** Serlex has left #kolla19:10
*** Serlex1 has joined #kolla19:10
*** dmsimard has quit IRC19:10
Serlex1Ok i will try the source19:11
Serlex1got to go19:11
*** Marga_ has joined #kolla19:15
*** Serlex1 has quit IRC19:15
*** Marga__ has joined #kolla19:16
*** Marga_ has quit IRC19:16
*** aginwala has quit IRC19:22
*** dmsimard has joined #kolla19:25
*** Marga__ has quit IRC19:28
*** Marga_ has joined #kolla19:29
*** sdake_ has quit IRC19:31
*** aginwala has joined #kolla19:36
*** dmsimard has quit IRC19:36
*** mdnadeem has quit IRC19:40
*** mdnadeem has joined #kolla19:40
*** dwalsh_ has quit IRC19:54
*** aginwala has quit IRC20:00
*** aginwala has joined #kolla20:02
*** aginwala has quit IRC20:03
*** achanda has quit IRC20:04
*** aginwala has joined #kolla20:07
*** JoseMello has quit IRC20:08
*** achanda has joined #kolla20:10
*** aginwala has quit IRC20:12
*** aginwala has joined #kolla20:13
*** AJaeger has joined #kolla20:24
AJaegercould somebody review this liberty change, please? it's the second part of moving from linters to pep820:25
*** neilus has joined #kolla20:43
*** iNeilus has quit IRC20:47
*** dave-mccowan has quit IRC20:48
*** opennode has joined #kolla20:50
*** aginwala has quit IRC20:57
*** dwalsh has joined #kolla20:58
*** aginwala has joined #kolla20:59
*** salv-orlando has quit IRC21:03
*** dwalsh has quit IRC21:15
*** haplo37 has quit IRC21:18
*** jtriley_ has joined #kolla21:22
*** jtriley has quit IRC21:26
*** aginwala has quit IRC21:31
*** sdake has joined #kolla21:35
*** jtriley_ has quit IRC21:42
-openstackstatus- NOTICE: The infrastructure team is taking gerrit offline for maintenance this afternoon, beginning at 22:00 utc. We should have it back online around 23:00 utc.
*** achanda has quit IRC21:46
*** alyson_ has quit IRC21:52
openstackgerritRuslan Kamaldinov proposed openstack/kolla: Output image statuses to log instead of return value of main function
*** aginwala has joined #kolla21:54
sdaked_code just got back from dentist21:54
sdakegive me a moment to read your paste21:54
sdakekeystone is one of the first things to happen21:57
sdakeneutron-dhcp-agent happens later21:57
*** achanda has joined #kolla21:57
sdakeit would appear dhcp is broken in centos-binary21:57
sdakesincey ou tried centos-source21:57
sdakewould you paste your globals.yml please?21:58
sdakelets see if we can atleast get you workgin with source builds of centos21:58
sdakeyou ended up with an unauthorized21:58
sdakequick q, are you running ansible deploy as root?21:58
d_codegimme a few…I’ll catch up21:59
sdakesounds good21:59
-openstackstatus- NOTICE: Gerrit is offline for maintenacne until 23:00 utc22:02
*** ChanServ changes topic to "Gerrit is offline for maintenacne until 23:00 utc"22:02
*** jtriley has joined #kolla22:05
*** ssurana has quit IRC22:05
d_codeI haven’t retried since I did the binary attempt22:05
d_codelemme clean those out and deploy source builds22:05
sdaked_code what is the IP of your machine22:05
sdakehang tight a sec22:05
sdakethat internal address22:06
sdakeneeds to be a free empty ip address on your network22:06
sdakemaybe I should make the docs capitalized there :)22:06
d_codeeven for all in one?22:06
sdakeeven for all in one22:06
d_codeno problem22:06
d_codesdake: so, I should turn haproxy back on, then?22:10
sdakeoh ya22:11
d_codek..did that…22:11
sdakepaste again plz22:11
d_codehere we go…running22:11
sdakei misseed that part of yoru config22:12
sdakejust let me double check one mor etime22:12
d_codehaproxy disable was hiding at the bottom22:12
d_codeI commented it out22:12
d_codesdake: back to keystone error22:14
sdakeeno0 is your main interface ip address?22:14
d_codeeno1 =
sdakeeno3 is your neutron interface?22:14
d_codeeno3 is a network that has a router to the interwebs22:15
sdakedoes eno3 have an IP associated with it?22:15
d_codenot an IPv4 address…it has the normal default IPv6 that is auto-assigned for the link22:15
sdakeyou cleaned up your deployment (cleainup-containers) ?22:15
sdakeand still getting the error with kolla's toolbox?22:16
sdakeabout keystone unauthorized22:16
* sdake groans22:16
sdakelet me see if my vm ever built the damn containers just a minute22:16
d_code:)  thanks22:17
sdakebinary didn't build right, building again22:18
sdakeshould take about 15 mins22:18
sdakelaptop went to sleep, that makes build not work ;)22:18
sdakewife keeps unplugging my laptop for whatever reason when i depart the house22:18
sdakei think she is jacking my power22:19
sdakeshe works at ansible22:19
sdakeand we have same power cable but she has her own22:19
sdake"but boy that one is right there..."22:20
d_codehahaha….  my wife does the same22:20
sdakenot many women there22:20
sdakei heard only 222:20
*** bmace has quit IRC22:22
*** bmace has joined #kolla22:22
*** haplo37 has joined #kolla22:24
sdaked_code what does your wife do may I ask?22:25
d_codeshe’s a pediatrician…she just has a macbook air and steals my power cables22:26
sdakeoh i thought you meant she worked at ansible22:27
sdakebut what you meant is she steals your power ;)22:27
*** rhallisey has quit IRC22:30
sdakeopenstack-base is not building for me atm22:33
* sdake growls22:33
sdakemoment let me debug this22:33
sdakethis explains why my whole build is nto working22:33
*** aginwala has quit IRC22:38
sdakeintereesting there is an overlayfs rpm plugin now22:39
sdaked_code I just learned something22:39
sdakeif your vm goes to sleep22:39
sdakedocker has to be stopped and started22:40
sdakegoes to sleep as in you close the lid or something22:40
*** achanda has quit IRC22:41
*** aginwala has joined #kolla22:42
SamYaplesdake: dont freak out about the vip22:46
SamYapleyoure blowing up when this was always teh situation22:46
SamYaplethis is literally nothing new22:46
sdakei haven't blown anything up22:46
SamYapleonce we agreed to the haproxy ssl termintation we _must_ use two vips22:47
SamYaplebefore we said the external address was user managed22:47
SamYaplewe had this conversation before liberty22:47
sdakeSamYaple you may be confused with my frustration with the fact that kolla is afu atm22:47
SamYaplehow so?22:47
SamYapleliberty you mean?22:47
sdakenot sure, dhcp master doesn't work on centos-binary22:47
sdakekeystone master doesn't work on centos source22:47
sdakeand its blocking several people from using kolla22:48
sdakebeen debugging since 3am22:48
SamYaplei never use centos22:48
SamYaplei dont know man22:48
sdakethats what I'm frustrated about :)22:48
SamYapleubuntus working great :(22:48
sdakei understand you dont test centos22:48
sdakei'll sort out centos22:48
sdakebut if we are good with two vips, wfm, that is secondary to this centos concern22:48
SamYaplei mean ubuntu stays sorted because I am on top of it. but i dont know anyone "on top of" centos on a regular basis22:48
sdakewhich I will handle22:48
sdaketypically thats me sam22:49
SamYapletwo vips is something we that we discussed22:49
sdakebut ya, been spotty lately as admitted22:49
SamYaplebasically we cant ensure firewalls are firewalling properly22:49
SamYaplemost people care about that and thats outside our control22:49
sdakeat one point you said you didn't want to do it when i uggested it22:49
SamYaplebecause of the reason above22:49
sdakethe operators seem to care about using two vips on the operator list22:50
SamYapleubt i said if we do haproxy ssl termination (what we agreed to at midcycle) then we MUST do two vips22:50
sdakecool sounds good then22:50
SamYaplesure, they can use two vips22:50
SamYapledo they care about _KOLLA_ managing two vips22:50
sdakei missed hhe two vip requirement on the external ssl22:50
SamYaplethat was always teh questions22:50
sdakethat wasn't in the notes i took22:50
SamYaplenah it was a condition from liberty midcycle22:50
SamYaplei dont think it was brought back up22:50
SamYaplesince i said I would handle it22:51
sdakecould have missed it, we made alot of decisions, I tried to capture everthing in notes22:51
sdakebut you guys talk faster then i type ;)22:51
SamYapleill hit up the mailing list to disperse any confusion22:51
sdakesounds good22:52
sdakei posted to the operator list not dev list22:52
sdakenot sure if you saw that thread22:52
*** tzn has joined #kolla22:52
sdakeall the ops want two vips22:52
sdakewhether we manage your right, i didn't ask unfortunately22:52
sdakec_code i've got a binary build going22:53
sdaked_code i've got a binary build going22:53
sdakeand its finally working22:53
sdakethe docker restart fixed the docker proxy which I htink goes out to lucn hof a vm sleepover22:53
sdakebut that is just speculation22:53
sdakecertain vm sleep causes some kind of docker problem though22:54
SamYaplei see everythign!22:54
SamYaplesdake: ubuntu recovers from vm/laptop sleep22:54
SamYaplejust fyi22:54
sdakeimo take more superman pills22:54
SamYapleim aware of this issue22:54
sdakeyet its not in the docs :)22:55
*** ssurana has joined #kolla22:55
*** daneyon has quit IRC22:56
sdaked_code have 1 hr meeting which is about how long the images will take to build22:56
sdakei'll let you know where its at when they are done22:56
*** tzn has quit IRC22:57
-openstackstatus- NOTICE: Gerrit is offline for maintenance, ETA updated to 23:30 utc23:03
*** ChanServ changes topic to "Gerrit is offline for maintenance, ETA updated to 23:30 utc"23:03
d_codethanks sdake23:06
*** CheKoLyN has quit IRC23:07
*** aginwala has quit IRC23:13
*** blahRus has quit IRC23:15
*** aginwala has joined #kolla23:19
*** neilus has quit IRC23:25
*** neilus has joined #kolla23:28
*** neilus has quit IRC23:32
*** achanda has joined #kolla23:36
-openstackstatus- NOTICE: Gerrit is offline for maintenance, ETA updated to 23:59 utc23:38
*** ChanServ changes topic to "Gerrit is offline for maintenance, ETA updated to 23:59 utc"23:38
*** salv-orlando has joined #kolla23:44
*** achanda has quit IRC23:48
sdaked_code my containers have built23:51
*** salv-orlando has quit IRC23:53
sdaked_code deploying now23:54
*** opennode has quit IRC23:54
*** opennode has joined #kolla23:55
*** opennode has quit IRC23:55
sdaked_code neutron dhcp agent fails to start confirmed23:57
sdakedefinately some type of docker bug or kolla bug23:57
d_codeoh…. oops….  I didn’t open that bug :-/23:57
d_codecooking dinner now, can get to it later, if needed23:58
sdakei'll open it thanks23:58
d_codesdake: one thing that we were having problems with is getting more info…the run doesn’t produce much and not much in journallctl23:58
sdaked_code to get the right info you have to ru ndocker in debug mode23:59

Generated by 2.14.0 by Marius Gedminas - find it at!