Friday, 2016-03-25

*** sbezverk has joined #kolla00:02
*** salv-orlando has joined #kolla00:02
vhosakotSamYaple: can you review https://review.openstack.org/#/q/owner:vhosakot%2540cisco.com+status:open... Eric gave +100:04
*** salv-orl_ has quit IRC00:05
*** ajafo has quit IRC00:07
*** ajafo has joined #kolla00:08
*** tfukushima has joined #kolla00:09
*** sdake_ has joined #kolla00:20
*** achanda has joined #kolla00:20
*** sdake has quit IRC00:22
*** achanda has quit IRC00:23
*** unicell1 has joined #kolla00:34
*** unicell has quit IRC00:35
*** ccesario_ has joined #kolla00:35
*** salv-orlando has quit IRC00:42
*** ccesario_ has quit IRC00:45
*** weiyu has joined #kolla00:49
*** Marga_ has quit IRC01:06
*** Marga_ has joined #kolla01:08
*** alisonh has quit IRC01:09
*** achanda has joined #kolla01:12
*** Marga_ has quit IRC01:13
*** unicell1 has quit IRC01:15
*** hui has joined #kolla01:22
*** hui has quit IRC01:25
*** sdake_ has quit IRC01:27
*** alisonh has joined #kolla01:27
*** ssurana has quit IRC01:37
*** ccesario has joined #kolla01:43
*** vhosakot has left #kolla01:47
*** vhosakot has joined #kolla01:49
*** vhosakot has quit IRC01:49
*** vhosakot has joined #kolla02:07
*** vhosakot has quit IRC02:09
weiyuHello SamYaple02:16
*** ccesario_ has joined #kolla02:18
*** tfukushima has quit IRC02:19
*** unicell has joined #kolla02:20
*** tfukushima has joined #kolla02:21
SamYapleweiyu: hello02:21
weiyuhttps://bugs.launchpad.net/kolla/+bug/154023402:21
openstackLaunchpad bug 1540234 in kolla mitaka "rabbitmq cluster does not work with dashes in the hostname" [Critical,Confirmed] - Assigned to weiyu (weiyu)02:21
weiyuI test this on ubuntu02:22
weiyurabbitmq cluster does not work with dashes in the hostname02:22
weiyuhttps://review.openstack.org/#/c/275111/ This patch not break rabbbmq cluster02:23
patchbotweiyu: patch 275111 - kolla - fix miss rabbitmq config when hostname include spe... (MERGED)02:23
SamYapleweiyu: i have dashes in my hostname02:24
*** unicell1 has joined #kolla02:24
SamYaplethat patch absollutely broke my environment02:24
SamYapletwice02:24
SamYapledifferent versions of rabbitmq i would assume02:24
SamYapleperhaps different regions02:24
weiyurabbitmqctl eval 'rabbit_clusterer:apply_config("/etc/rabbitmq/rabbitmq_cluster.config").'02:25
*** unicell has quit IRC02:25
weiyuyou can exec this command for test your config02:25
weiyurabbitmqctl eval 'rabbit_clusterer:apply_config("/etc/rabbitmq/rabbitmq_clusterer.config").'02:26
weiyurabbitmq version is 3.5.7-102:35
*** vhosakot has joined #kolla02:45
*** cemason has joined #kolla02:46
*** cemason has quit IRC02:48
*** barra204 has joined #kolla02:50
SamYapleweiyu: that bug was filed before we moved to 3.5.702:53
SamYaplei have not tested if it works in 3.5.702:53
weiyuyou mean i must test it on old version rabbitmq?02:56
SamYapleweiyu: no i believe you it fixed an issue for you, but as i post in the comments of the bug it _BROKE_ existing things02:58
SamYapleweiyu: and i use dashes in my hostnames without issue, so we need to find the real causes here02:59
*** salv-orlando has joined #kolla03:01
*** sdake has joined #kolla03:07
*** ccesario_ has quit IRC03:07
*** dave-mccowan has quit IRC03:10
vhosakotcore reviewers,  can  https://review.openstack.org/#/c/296649/  be reviewed and merged... it fixes a bug about incorrect toml libvirtd.log in master.. I have tested this patch set and the gate is green03:11
patchbotvhosakot: patch 296649 - kolla - Revert libvirt change in heka-openstack.toml.j2 in...03:11
vhosakotSamYaple: could you look at this when you get a chance.. thanks!  ^^^03:12
*** dave-mccowan has joined #kolla03:13
*** achanda has quit IRC03:15
sdakeevening folks03:15
sdakeboy that dentist visit was brutal03:15
* sdake hates the dentist03:15
weiyuSamYaple: I agree find the read causes here03:16
*** absubram has quit IRC03:18
*** dave-mccowan has quit IRC03:20
*** salv-orlando has quit IRC03:21
openstackgerritMerged openstack/kolla: Revert "Make heat work"  https://review.openstack.org/29709003:26
*** salv-orlando has joined #kolla03:32
*** yuanying_ has joined #kolla03:35
*** yuanying has quit IRC03:37
*** salv-orlando has quit IRC03:37
*** tfukushima has quit IRC03:38
*** yuanying has joined #kolla03:38
*** vhosakot has quit IRC03:39
weiyuSamYaple: please look http://paste.openstack.org/show/491848/03:40
*** yuanyin__ has joined #kolla03:40
*** yuanying_ has quit IRC03:40
*** jasonsb has joined #kolla03:42
*** salv-orlando has joined #kolla03:43
sdakenote core reviewers, please ack https://review.openstack.org/#/c/297200/1 if you are goin g t ack the revert patch :)03:43
patchbotsdake: patch 297200 - kolla - Change heat domain to heat_user_domain03:43
sdakethat needs to alnd together03:43
sdakeso heat isn't broken03:43
*** yuanying has quit IRC03:43
*** salv-orlando has quit IRC03:47
*** yuanyin__ has quit IRC03:49
*** yuanying has joined #kolla03:51
*** tfukushima has joined #kolla03:56
*** yuanying has quit IRC03:59
*** yuanying has joined #kolla04:02
*** jasonsb has quit IRC04:04
*** yuanying has quit IRC04:07
openstackgerritMerged openstack/kolla: Change heat domain to heat_user_domain  https://review.openstack.org/29720004:10
*** achanda has joined #kolla04:18
*** asalkeld_ has joined #kolla04:19
*** asalkeld_ has quit IRC04:20
*** jasonsb has joined #kolla04:26
*** weiyu has quit IRC04:29
*** iceyao has joined #kolla04:30
*** salv-orlando has joined #kolla04:46
*** Marga_ has joined #kolla04:47
*** Marga_ has quit IRC04:47
*** Marga_ has joined #kolla04:48
*** jasonsb_ has joined #kolla04:50
*** jasonsb has quit IRC04:50
*** salv-orlando has quit IRC04:53
*** Pierre_Carette has quit IRC04:54
*** iceyao has quit IRC05:07
*** weiyu_ has joined #kolla05:07
*** tfukushima has quit IRC05:08
openstackgerritSteven Dake proposed openstack/kolla: Change heat domain to heat_user_domain  https://review.openstack.org/29751705:18
*** weiyu_ has quit IRC05:28
*** weiyu_ has joined #kolla05:38
*** Jeffrey4l has joined #kolla05:39
*** weiyu_ has quit IRC05:40
*** sdake has quit IRC05:50
openstackgerritGerard Braad proposed openstack/kolla: Use packaged pip instead of easy_install  https://review.openstack.org/29752305:52
*** weiyu_ has joined #kolla06:02
*** tfukushima has joined #kolla06:09
*** tfukushima has quit IRC06:10
*** salv-orlando has joined #kolla06:31
*** salv-orlando has quit IRC06:34
*** tfukushima has joined #kolla06:37
*** yuanying has joined #kolla06:46
*** salv-orlando has joined #kolla06:52
*** salv-orlando has quit IRC06:59
*** vishwanathj has joined #kolla07:01
*** vishwana_ has joined #kolla07:06
*** vishwanathj has quit IRC07:08
*** cristicalin has joined #kolla07:22
openstackgerritJeffrey Zhang proposed openstack/kolla: Only copy the necessary config files for Glance  https://review.openstack.org/29698207:50
openstackgerritJeffrey Zhang proposed openstack/kolla: Only copy the needed config files for Cinder  https://review.openstack.org/29461807:53
*** salv-orlando has joined #kolla07:58
*** stvnoyes has quit IRC08:07
*** stvnoyes has joined #kolla08:08
*** salv-orlando has quit IRC08:13
*** Jeffrey4l_ has joined #kolla08:16
*** achanda has quit IRC08:16
*** Jeffrey4l has quit IRC08:17
*** kproskurin has joined #kolla08:22
*** jmccarthy has quit IRC08:35
*** jmccarthy has joined #kolla08:36
*** iceyao has joined #kolla08:54
*** weiyu__ has joined #kolla08:56
*** weiyu_ has quit IRC08:56
*** weiyu__ has quit IRC09:00
*** achanda has joined #kolla09:03
*** openstackstatus has joined #kolla09:07
*** ChanServ sets mode: +v openstackstatus09:07
*** salv-orlando has joined #kolla09:09
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Parse and render yaml files in the proper order  https://review.openstack.org/29583109:16
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Fix OpenStack services registration  https://review.openstack.org/29125109:16
*** kklimonda has joined #kolla09:22
Jeffrey4l_SamYaple, around?09:23
*** Jeffrey4l_ is now known as Jeffrey4l09:23
*** Slower has quit IRC09:39
*** Slower_ has joined #kolla09:39
*** achanda has quit IRC09:41
*** Jeffrey4l has quit IRC09:46
*** Jeffrey4l has joined #kolla09:46
*** Marga_ has quit IRC09:48
*** Marga_ has joined #kolla09:48
*** Marga_ has quit IRC09:51
*** Marga_ has joined #kolla09:53
*** mgoddard has joined #kolla10:02
*** Marga_ has quit IRC10:04
*** Marga_ has joined #kolla10:06
*** mgoddard has quit IRC10:09
*** gfidente has joined #kolla10:10
*** mathlin has quit IRC10:12
*** iceyao has quit IRC10:13
*** mathlin has joined #kolla10:14
*** pbourke has quit IRC10:15
*** pbourke has joined #kolla10:15
*** achanda has joined #kolla10:41
*** achanda has quit IRC10:50
openstackgerritMerged openstack/kolla: Add kibana authentication using HAproxy  https://review.openstack.org/29665010:53
openstackgerritMerged openstack/kolla: Upgrade elasticsearch  https://review.openstack.org/29699310:58
openstackgerritMerged openstack/kolla: Adds genisoimage package dependency to nova-compute  https://review.openstack.org/28559810:59
*** prithiv has joined #kolla11:13
prithivanyone tried multi node setup ?11:13
prithivare you able to see list of hypervisors in multi node setup11:13
prithivHi Sam11:19
*** tfukushima has quit IRC11:23
*** cristicalin has quit IRC11:34
*** iceyao has joined #kolla11:35
openstackgerritsean mooney proposed openstack/kolla: Adds genisoimage package dependency to nova-compute  https://review.openstack.org/29763711:40
*** rhallisey has joined #kolla11:41
*** salv-orlando has quit IRC11:56
*** salv-orlando has joined #kolla11:56
*** prithiv has quit IRC12:01
*** salv-orl_ has joined #kolla12:03
*** salv-orl_ has quit IRC12:04
*** salv-orlando has quit IRC12:05
*** dwalsh has joined #kolla12:13
openstackgerritJeffrey Zhang proposed openstack/kolla: DO NOT MERGE  https://review.openstack.org/29764212:13
*** mbound has joined #kolla12:17
*** unicell1 has quit IRC12:24
*** unicell has joined #kolla12:26
openstackgerritJeffrey Zhang proposed openstack/kolla: DO NOT MERGE  https://review.openstack.org/29764212:29
openstackgerritAlicja Kwasniewska proposed openstack/kolla: Add kibana authentication using HAproxy  https://review.openstack.org/29764912:35
openstackgerritLukasz Oles proposed openstack/kolla-mesos: Stop when marathon framework is found  https://review.openstack.org/29765412:39
*** salv-orlando has joined #kolla12:46
*** achanda has joined #kolla12:47
*** achanda has quit IRC12:53
*** prithiv has joined #kolla13:05
*** jtriley has joined #kolla13:14
*** peterjenkins has joined #kolla13:15
openstackgerritSerguei Bezverkhi proposed openstack/kolla: iscsi container with lvm2 support  https://review.openstack.org/29128513:18
*** jtriley has quit IRC13:19
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Parse and render yaml files in the proper order  https://review.openstack.org/29583113:24
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Fix OpenStack services registration  https://review.openstack.org/29125113:24
*** jtriley has joined #kolla13:32
*** mgoddard has joined #kolla13:42
sbezverkGood morning13:42
sbezverkJeffrey41 ping13:43
*** ayoung has quit IRC13:44
*** kproskurin_ has joined #kolla13:48
*** kproskurin has quit IRC13:49
*** kproskurin_ is now known as kproskurin13:49
*** mgoddard has quit IRC13:51
*** absubram has joined #kolla13:52
*** absubram_ has joined #kolla13:54
*** absubram has quit IRC13:57
*** absubram_ is now known as absubram13:57
SamYaplejpeeler: morning14:00
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Parse and render yaml files in the proper order  https://review.openstack.org/29583114:00
openstackgerritMichal Rostecki proposed openstack/kolla-mesos: Fix OpenStack services registration  https://review.openstack.org/29125114:00
SamYapleJeffrey4l: morning*14:01
SamYaplejpeeler: you too, but i pinged the wrong person first :)14:01
Jeffrey4lsbezverk, pong14:03
Jeffrey4lSamYaple, morning14:03
SamYapleJeffrey4l: you pinted me?14:04
SamYaplepinged*14:04
Jeffrey4lyea. I search the url I want to ask. A small question. wfm14:04
SamYapleprithiv: yes. i run multinode kolla for my home lab. i do kolla dev ontop of it14:04
prithivcan you post your inventory file… i just want to make sure am not missing anything14:05
Jeffrey4lSamYaple, https://github.com/openstack/kolla/blob/master/docker/kolla-toolbox/find_disks.py#L102 here.  re.sub(r'.*[^\d$]', '', dev.device_node)14:05
SamYapleyes?14:06
Jeffrey4ldoes the $ is really needed?14:06
SamYapleJeffrey4l: i think so, but im not sure. i always close out my regex14:06
sbezverkJeffrey41 could you please check my reply to your comments and the conditional solution14:06
SamYaplei do know if you DONT have that it will break if you have a digitin your partition14:07
Jeffrey4lprithiv, SamYaple I set up a multi node env several days before. It works fine. And Kolla is very coll.14:07
Jeffrey4lcool*14:07
Jeffrey4lsbezverk, working on it.14:07
Jeffrey4lsbezverk, I am Jeffrey4l not Jeffrey41.14:07
SamYapleJeffrey4l: to me you are Jef<tab>14:08
prithivJeffrey4l, SamYaple, can any one of you share the inventory file….14:08
openstackgerritMerged openstack/kolla: Change heat domain to heat_user_domain  https://review.openstack.org/29751714:08
SamYapleprithiv: this might go easier if you share yours14:08
sbezverkJeffrey4l Sorry, need better glasses :-)14:08
SamYapleprithiv: my is just three hosts14:09
Jeffrey4lSamYaple, i do not see the needed to add the $, because the device_node never has a $ char.14:09
SamYapleJeffrey4l: thats a regex14:09
SamYaple$ means end-of-line14:09
Jeffrey4l$ in [] means EOL? I doubt that.14:10
prithivmy inventory14:10
prithivhttp://paste.openstack.org/show/491899/14:10
prithivplease advice if anything am missing14:10
Jeffrey4lprithiv, the inventory is meaningless. You env and mine is not different.14:10
SamYapleJeffrey4l: [] does not me any of these characters14:10
SamYaplemore specifically, [^]14:11
prithivwhen i have a 3 node cluster, i dont even see the nova-libvirt container getting created...14:11
Jeffrey4lSamYaple, yea. So [^\d$] mean do not include the \d and $ char? right?14:11
prithivso hypervisor-list doesnt show up… with/without fake libvirt14:11
SamYapleJeffrey4l: yea thats not whats there though14:12
SamYapleor is it14:12
SamYapleJeffrey4l: yea youre right that can be removed14:13
Jeffrey4lSamYaple, :D14:13
Jeffrey4lSamYaple, btw, I may will try to deploy a ceph cluster, which use the external jounal disk. Just want to make sure kolla now support this? right? I saw some logical about this. SamYaple14:14
SamYapleJeffrey4l: yes. though i havent written the kolla docs for it14:14
SamYapleJeffrey4l: https://yaple.net/2016/03/20/deploying-openstack-mitaka-with-kolla-docker-and-ansible/14:15
SamYapleJeffrey4l: there are some references in there about how to do it14:15
Jeffrey4lSamYaple, kolla. your solution(find_disk.py) is  very cool :D14:15
SamYapleJeffrey4l: yea i think its better than the variablie substitition in bash14:16
SamYaplemore reliable14:16
Jeffrey4lI agree. Like this solution. simple and powerful.14:16
*** spisarski has joined #kolla14:17
Jeffrey4lprithiv, could you check the ansible log and check whether the nova-compute task is skipped. And try to find the root cause.14:17
sbezverkquestion for ansible gurus, would this statement be ok: "      - { volume: "iscsi_info:/etc/iscsi", enabled: "{{ enable_iscsi }}" }14:18
sbezverk"14:18
openstackgerritBenedikt Trefzer proposed openstack/kolla: Make kibana container build with debian  https://review.openstack.org/29720614:18
Jeffrey4lsbezverk, do not think so..14:20
sbezverkJeffrey4l thanks, will keep looking then14:23
openstackgerritJeffrey Zhang proposed openstack/kolla: Remove meaningless char in regexp of find_disks.py file  https://review.openstack.org/29770314:26
*** ayoung has joined #kolla14:26
*** iceyao has quit IRC14:36
*** sdake has joined #kolla14:41
sdakemorning14:42
akwasnie_hi sdake14:45
sdakehey akwasnie_14:45
akwasnie_'Add kibana authentication using HAproxy' was merged today  and I cherry picked it to stable/mitaka14:47
SamYaplemorning sdake14:48
sdakehey SamYaple14:48
SamYaplecongrats, saw the results this morning14:48
sdakethanks14:48
sdakei dont know ify ou cuaght message last ight, thanks for running14:48
sdakeits healthy for a project to have multiple viable leaders14:49
sdakein cae I wer et obe hit by a bus for example14:49
sdakeakwasnie nice ob!14:50
pbourkesdake: congrats14:50
*** achanda has joined #kolla14:50
sdakepbourke thanks but lets put the election behind us and focuson the future :)14:50
pbourkehopefully you wont get hit by a bus :)14:50
sdakethe only bummer part is only 50% of people voted hat could have14:51
sdake69 people voted, 42 for me, 24 for sam, and the remainder had no preference14:52
sdaketalk about magic numbers14:52
sdake24 is an anagram of 42, which as e all knwo is the meaning of life :)14:52
sdakei won't get into 69 ;)14:52
SamYapleso first up for newton should be splitting off kolla-ansible, no? so we can stabilize that14:53
sdakei am oepn to that, depends what happens with kolla-mesos i think14:54
SamYaplei dont think it does...14:54
sdakemaye not14:54
SamYaplewe should probably start rolling the balls on that now14:54
sdakei think about workflow with two repos and it seems painful14:54
SamYapleits going to take about 3 weeks to do the split14:54
SamYaple1 we already have two repos, 2 the community overwhelmingly wants it and you know that14:55
*** dwalsh has quit IRC14:55
*** achanda has quit IRC14:55
SamYaplei actually think you are the only one to ever voice dissent on the matter14:55
*** inc0 has joined #kolla14:55
sdakein the past th edissent was for political reasons14:55
sdakenow i dont hae a strong opinion14:55
inc0hello14:56
sdakei was concerned it would block our big tent application process14:56
SamYaplecool so it takes about 3 weeks to split off kolla-ansible, we should probably get that going14:56
SamYaplewith any luck it wont block devel before the summit14:56
sdakeand indeed we were attacked just as i suspected we woudl be during big tent 10 months ago14:56
SamYaplethat says nothing about the mitaka cycle where we were suppose to split this off14:57
SamYaplebut lets focus on the now and get it rolling14:57
inc0SamYaple, don't we have more important stuff to do before summit than repo split?14:57
SamYapleinc0: repo split blocks dev work14:57
inc0how so?14:57
SamYapleinc0: on master, which lets us focus on mitaka14:57
SamYapleinc0: gotta hold patches while infra patches merge14:57
sdakewell we are not spitting repos until atleast mitaka final is cut14:57
SamYapleso first thing after right?14:58
sdakebecausei want an easy cherrypick method14:58
inc0also we keep getting bugs and bugs14:58
inc0I'd rather focus on stabilization14:58
inc0that's wat RCs are for14:58
sdakeagree with inc014:58
SamYapleinc0: splitting repos takes about 3 weeks to go14:58
SamYaplestarting it now means we will be ready for mitaka final to split14:58
SamYapleits alot of upstream patches first14:58
inc0ok, I'd rather use first 3 weeks after summit, which are slow anyway14:58
SamYaplefair enough, as long as we do that14:59
sdakeinc0 i think tha ttiming is good14:59
inc0all I'm saying, wer'e getting bug reports14:59
inc0a lot now14:59
SamYaplei completely agree14:59
sdakepeople go into hibernation after summit for a few weeks14:59
SamYapleas long as we do it14:59
inc0we need to focus on getting kolla super-stable14:59
sdakeSamYaple you got agreement from me, although i dont think i will like the workflow14:59
inc0so let's just start deploying different scenerios and focus on this14:59
sdakeya we wont have to backport a million tihngs if kolla is stable out of the gate15:00
inc0sdake, if we do it first thing after summit, that means we can prep patches in Austin together15:00
inc0and discuss the details15:00
sdakewell i dont hae a strong opinion15:00
sdakeagain, mu oposition was based upon us being  attacked during big tent15:01
sdakeand we were attacked during big tent15:01
sdakeif we had split the repos, kolla-ansible may not hae ended up in big tent and kolla-docker may have15:01
sdakeits not like i suffer undue paranoioa for a reason15:02
SamYaplesdake: that was almost a year ago, what was your position during the mitaka cycle when we were pushing this?15:02
sdakeI just put myself in the shoes of my opponents and paly through what they woudl do:)15:02
sdakei dont recall it relay coming up during mitaka, but i want a stable abi15:02
sdakebefore spitting15:02
sdakei think we have that now15:02
sdakei think we had that before mitaka tbh15:03
SamYapleit came up, you always had an excuse to not want to do it. thats why im asking us to make a firm timetable you stick to now15:03
sdakeanyway splitting first thing after mitaka wfm15:03
SamYaplecool15:03
sdakeduring newton15:03
sdakei stil ldoont think dev will like th workflow ;)15:04
sdakebtw folks ren ois in the repo15:04
sdakeplease use it15:04
sdakeits dead simpel to use15:04
sdakereno lvm-iscsi15:05
sdakeedit the yaml file15:05
sdakeadd a note 'Added ISCIS support to lvm"15:05
sdakeI submittedd a gate job for this and its supposed to be published15:05
sdakewhere I cannot find15:05
sdakerelease notes are here: http://docs.openstack.org/releasenotes/kolla/15:06
*** ssurana has joined #kolla15:11
*** vhosakot has joined #kolla15:15
*** mbound has quit IRC15:16
*** vhosakot has quit IRC15:17
*** vhosakot has joined #kolla15:18
sdakei will work on getting us added here too:15:19
sdakehttp://releases.openstack.org/mitaka/index.html15:19
*** prithiv has quit IRC15:22
openstackgerritPaul Bourke proposed openstack/kolla: Replace dd with fallocate in cinder guide  https://review.openstack.org/29772315:24
openstackgerritBenedikt Trefzer proposed openstack/kolla: make rabbitmq container build  https://review.openstack.org/29772815:30
ccesariomorning!15:35
*** vishwanathj has joined #kolla15:38
sbezverkSamYaple I am trying to see if inside ansible list, one element can have condition, do you know if it is doable?15:38
sdakesbezverk did you sort out reno15:38
sbezverksdake I removed it as I got -115:38
SamYaplesbezverk: you mean only include in list if <conditional>?15:38
sbezverkSamYaple yes, if variable for this service is enabled15:39
sbezverkthen this volume will be in the list, otherwise no15:39
sdakesbezverk got -1 for what15:39
SamYaplesbezverk: oh variable volumes are tough. that was something that was going to be tackled this cycle15:40
SamYaplesbezverk: i would highly suggest a module that returns a dict15:40
sdakeok folks i have to go to doctor15:40
SamYaplebut that spec hasnt been written yet15:40
sdakebe back in 1 hr or so15:40
sbezverkSamYaple thank you, Jeffrey4l I guess what SamYapple settled that volume comment15:41
sbezverkat least for now..15:41
SamYaplesbezverk: yea it needs to be handled project wide15:41
SamYapleso people can do host bind mounts instead of docker volumes15:42
sbezverkSamYaple, I understand, make sense.. I was just trying to address Jefferey4l comment15:42
*** dwalsh has joined #kolla15:44
openstackgerritSerguei Bezverkhi proposed openstack/kolla: iscsi container with lvm2 support  https://review.openstack.org/29128515:45
*** rhallisey has quit IRC15:45
openstackgerritBenedikt Trefzer proposed openstack/kolla: Make heka container build with debian  https://review.openstack.org/29773715:45
pbourkeSamYaple: know any major filesystems that dont support fallocate off the top of your head?15:48
openstackgerritBenedikt Trefzer proposed openstack/kolla: use silent option for curl download in heka container  https://review.openstack.org/29773915:48
SamYaplepbourke: fallocate is also dependant on root15:49
SamYaplepbourke: http://man7.org/linux/man-pages/man2/fallocate.2.html15:49
*** prithiv has joined #kolla15:49
SamYaplethe filesystems sometimes get support late :)15:49
pbourkethe root thing doesn't seem to be true15:50
pbourkeat the end of the day its just documentation, fallocate is quicker for most situations15:51
pbourkeif someones using something esoteric Im sure they can figure out to use dd15:51
SamYaplepbourke: maybe im thinking of truncate15:52
gmmahahi, deploying openstack with kolla, shouldnt the role "_member_" get created by default?16:03
gmmahaI am not able to see it in my setup http://paste.openstack.org/show/491910/16:03
openstackgerritBenedikt Trefzer proposed openstack/kolla: Make container kolla-toolbox build with debian  https://review.openstack.org/29710416:03
gmmahathis in-turn leads to failure in creating new user or projects16:03
openstackgerritPaul Bourke proposed openstack/kolla: Replace dd with fallocate in cinder guide  https://review.openstack.org/29772316:07
inc0gmmaha, well..that's keystone stuff16:14
inc0not sure if they changed anything, we never explicitly added this role16:15
inc0(nor should we I think)16:15
SamYaplepbourke: i was thinking of truncate16:16
SamYaplepbourke: sorry. truncate is special16:16
gmmahainc0: yeah thats what i thought.. the last i heard when you instantiate keystone it creates both admin and member by default16:16
pbourkeok np16:16
gmmahaasking there as well.. Will let you know what i find16:16
SamYaplefallocate and truncate are similiar16:16
inc0ayoung, around?:)16:17
*** mbound has joined #kolla16:17
SamYaplegmmaha: the _member_ role doesnt get created by default16:18
SamYaplegmmaha: you can create it, but we dotn by default since that is configurable whereas the admin role is mostly non-configurable16:18
openstackgerritBenedikt Trefzer proposed openstack/kolla: use silent option for curl download in heka container  https://review.openstack.org/29773916:18
gmmahaSamYaple: aah... for a while i always thought openstack created both the roles.. maybe i was wrong all along16:20
SamYaplegmmaha: kolla creates the admin role and setups the admin user, but thats it16:20
SamYapleits not an unreasonable thing to say kolla should create the member role, but it was discussed at one point and we came up with "we wont make it because its configurable in configs which user can modify"16:21
gmmahaSamYaple: thats fiar. if keystone doesnt use it then there is no point16:21
*** mbound has quit IRC16:23
openstackgerritBenedikt Trefzer proposed openstack/kolla: Use debian repos for debian base docker image  https://review.openstack.org/28893616:25
openstackgerritBenedikt Trefzer proposed openstack/kolla: Make marathon container build with debian  https://review.openstack.org/29737816:36
*** unicell has quit IRC16:41
*** salv-orl_ has joined #kolla16:42
*** salv-orlando has quit IRC16:45
*** achanda has joined #kolla16:53
*** kproskurin has quit IRC16:55
*** achanda has quit IRC16:59
openstackgerritBenedikt Trefzer proposed openstack/kolla: make rabbitmq container build with debian  https://review.openstack.org/29772816:59
*** achanda has joined #kolla17:02
*** gfidente has quit IRC17:03
*** marmat has joined #kolla17:08
*** prithiv has quit IRC17:09
sbezverkSamYaple would you be able to do a review on iscsi in near future?17:12
SamYaplesbezverk: im loaded down on liberty atm17:13
SamYaplesbezverk: but ill do my best17:13
sbezverkSamYaple, understood, thank you17:14
vhosakotgmmaha: you could add member_role_name=_member_ under [DEFAULT] in keystone.conf is needed...https://github.com/openstack/keystone/blob/master/etc/keystone.conf.sample#L51...17:14
vhosakotis==if17:14
SamYaplealso `openstack role add _member_` works17:16
*** unicell has joined #kolla17:16
SamYaplevhosakot: the issue is the member role doesnt exist17:16
vhosakotah ok ok... i know in IceHouse or older, keystone I think created _member_ by itself... dont remember17:17
SamYaplehmmm where is ryan today?17:17
vhosakotor may be it needed a tenant17:17
vhosakotSamYaple: does kolla use tenants ?17:17
vhosakotyes, "openstack role add _member_" is cool17:18
vhosakotSamYaple: does kolla create tenant id ?17:19
vhosakotI see user ID and domain ID for user with no tenant ID...17:20
SamYaplevhosakot: project17:21
vhosakotor project ID... I think project ID ie equivalent for tenant ?17:21
SamYaplethere are not tenatns17:21
vhosakotah ok thanks!17:21
SamYapleno it keystone v317:21
vhosakotno tenants in keystone v3 ?17:21
vhosakotSamYaple: is this right ? no tenants in keystone v3 ?17:22
*** dwalsh has quit IRC17:23
SamYapleyes17:23
SamYaplethat is correct17:23
SamYaplethey are called projects17:23
*** thumpba has joined #kolla17:24
inc0one of reasons I love ks;)17:24
vhosakotcool!17:24
inc0"let's rename this stuff"17:24
vhosakotI see user has domain ID, and that domain ID is under a project.. cool..17:24
*** Marga_ has quit IRC17:33
*** spisarski has left #kolla17:36
gmmahavhosakot: yeah i just added the role with openstack tool17:39
vhosakotah ok17:39
gmmahaand updated a ticket in keystone to see if they want to doit themselves17:39
SamYapleinc0: to be fair it was called "projects" and "tenants" interchangabily and they choose one17:40
SamYaplebut i agree with teh sentiment17:40
vhosakotbut, I have ks v3 working with no _member_.... I can create nova VM and stuff17:40
vhosakotyes, project and tenant are same... keystone10117:40
*** inc0 has quit IRC17:44
*** inc0 has joined #kolla17:46
*** ayoung has quit IRC17:47
*** Marga_ has joined #kolla17:54
*** jasonsb_ has quit IRC18:06
*** sdake_ has joined #kolla18:07
*** sdake has quit IRC18:08
Jeffrey4lanyone consider who nova migrate case? it will copy file using ssh, but the nova compute data is no longer saved at host:/var/lib/nova/instace, it is stored in docker named container.18:16
Jeffrey4lThis should be a critical bug, right?18:17
Jeffrey4lSamYaple, sdake_ ^^18:17
SamYapleJeffrey4l: ye this is why kolla says use ceph18:17
Jeffrey4ldocker named volume**18:17
SamYaplemigrate == broke18:17
SamYaplelive-migrate wokrs18:17
SamYapleshared storage is needed to migrate in kolla18:18
Jeffrey4lSamYaple, no. when using ceph, it will also use ssh to copy files. ssh 172.16.16.14 touch /var/lib/nova/instances/ff5b1cb8-fab7-4721-885a-87e42892d149/c0624ef0e5d24294aa1134f7a3458b10.tmp18:19
SamYapleJeffrey4l: i live-migrate multiple times a day with ceph18:19
SamYapleits not a requirement to ssh18:20
Jeffrey4lmoreover, how about when using nova resize action. It will tigger this logical too. even when using ceph.18:20
SamYapleyea resize is bust18:20
Jeffrey4lSamYaple, I mean nova migrate not nova live-migration18:20
SamYapleim not saying we dont need to solve it18:20
SamYapleJeffrey4l: i know, migrate == broke18:20
Jeffrey4llive-migration works fine.18:20
SamYapleJeffrey4l: im not saying we _dont_ need to solve this, im saying its not solved18:21
Jeffrey4lSamYaple, so do you have any solution in mind.18:21
Jeffrey4lI just raise that issue. no other means. :D18:21
Jeffrey4lhow about map the host folder to the container for nova-compute? rather than use the docker named volume?18:22
SamYapleJeffrey4l: yea the problem is its the container that must ssh to a host, which means the contaienr must have ssh-keys18:23
SamYapleJeffrey4l: i do not have a solution in mind because of this18:23
SamYaplewe _strongly_ recommend shared storage dor these reasons18:23
SamYapleits not ideal18:23
Jeffrey4lOr just create a soft link: ln -s /var/lib/docker/volumes/nova_compute/_data /var/lib/nova ?18:24
Jeffrey4lyea :(18:24
Jeffrey4lhow about: 1. start a sshd container with another port 8022 which map nova_compute volume to /var/lib/nova/    2. add a ssh config file to the nova_compute container to set default ssh port to 802218:30
Jeffrey4lSamYaple, ^18:31
*** ayoung has joined #kolla18:34
sdake_agree with SamYaple - live migration requires ceph18:34
sdake_Jeffrey4l i like sshd o na speicifc port as a solution18:35
Jeffrey4lsdake_, thanks.18:36
sdake_dave sugggeted this for fernet toke nrotation18:37
Jeffrey4lI think this is a very critical issue. It will block the end-user use the kolla in production.18:39
Jeffrey4lsdake_, ^18:44
sdake_moe details18:45
*** achanda has quit IRC18:45
sdake_Jeffrey4l can you add a bug with details18:46
Jeffrey4lnp sdake_18:46
sdake_if it reallly is critical it is probbly something we should fix18:47
sdake_explain in bug whsy its a critical use case18:47
manjeetsinc0: did all the cleanup and rebuild registry and again ended up in same issue18:50
manjeetshttp://paste.openstack.org/show/491936/18:50
*** achanda has joined #kolla18:52
sdake_manjeets its time for sdake debug sesion18:56
sdake_can youo give me background plz18:56
*** sdake_ is now known as sdake18:56
sdakebase distro, contrainers, node count, etc18:56
manjeetsbase distro ubuntu18:57
sdakemanjeets paste your inventor yfile plz18:57
manjeetsnode count 6418:57
manjeetssure18:57
sdakebiggest node count i've heard of anyone deploying with18:57
inc0brb changing vpn's and going back to deploy less nodes...but damn powerful;)18:58
*** inc0 has quit IRC18:58
SamYapleJeffrey4l: id have to see it in action first. im not sure about the ssh-to-container thing18:58
SamYapleJeffrey4l: it might work, but please post a blueprint/review18:58
*** inc0 has joined #kolla18:58
Jeffrey4lSamYaple, i will18:59
inc0manjeets, you're running this on public cloud?18:59
SamYapleinc0: networking setup in my cloud is working pretty well18:59
manjeetsinc0: no19:00
manjeetsvpn19:00
manjeetshttp://paste.openstack.org/show/491939/ sdake19:00
SamYaplestill allows public floating and neutron routers19:00
inc0so first of all, it's better to run odd number of controllers19:01
inc03 should be enough for this scale19:01
inc0for mariadb sace19:01
inc0sake19:01
inc0you have 719:02
manjeetsi guess its already odd 719:02
inc0yeah, just you don't need that much I think19:02
inc0shouldn't matter tho19:02
inc0can you paste docker logs mariadb from all maria nodes please?19:05
manjeetsinc0: sure its not even going to second after it fails on first control19:06
inc0well it should deploy mariadb19:06
inc0manjeets, can you scale-down mariadb?19:08
manjeetsinc0: you mean control node ? or just mariadb ?19:09
inc0try control first19:09
inc0there is really no need for 7 controllers19:09
inc0try it with 319:09
inc0let see if that's cluster scale problem19:09
inc0shouldn't be19:09
inc0tho19:09
inc0but let's see19:10
manjeetssure I am gonna deploy on 10 nodes in total first19:10
manjeetswill move futhur if succeeded19:10
inc0kk19:11
* manjeets started cleanup first19:12
manjeetsi don't think its a scale problem19:20
manjeetsinc0: new mariadb log http://paste.openstack.org/show/491940/19:24
manjeetsmariadb is not even running on other than first node19:25
inc0hm..quorum doesn't seem to agree with each other19:26
inc0SamYaple, around?19:26
*** Jeffrey4l_ has joined #kolla19:33
*** Jeffrey4l has quit IRC19:34
*** SiRiuS_ has joined #kolla19:41
*** thumpba has quit IRC19:41
Jeffrey4l_sdake, see the bug https://bugs.launchpad.net/kolla/+bug/156214119:42
openstackLaunchpad bug 1562141 in kolla "kolla can not work with nova migrate and nova resize" [Undecided,New] - Assigned to Jeffrey Zhang (jeffrey4l)19:42
*** achanda has quit IRC19:47
*** mathlin has quit IRC19:48
*** jmccarthy1 has joined #kolla19:55
*** jmccarthy1 has quit IRC19:56
*** jmccarthy1 has joined #kolla19:56
*** inc0 has quit IRC20:07
*** inc0 has joined #kolla20:07
vhosakotSamYaple: when you have a chance, could you reconsider your -2 for https://review.openstack.org/#/c/296012/... Eric gave +1 and the patch set is tested and works fine... I need it to merge for lua work on libvirtd.log.. thanks!20:13
patchbotvhosakot: patch 296012 - kolla (stable/mitaka) - Make heka collect libvirtd.log in nova_libvirt con...20:13
vhosakot"I need it to be merged"  I meant20:13
*** sdake_ has joined #kolla20:14
*** inc0 has quit IRC20:14
*** sdake has quit IRC20:17
*** achanda has joined #kolla20:17
openstackgerritBenedikt Trefzer proposed openstack/kolla: Make kibana container build with debian  https://review.openstack.org/29720620:17
*** salv-orl_ has quit IRC20:25
*** jmccarthy1 has quit IRC20:26
gmmahavhosakot: have you tried running kolla with linuxbrdige?20:26
gmmahalinuxbridge?20:26
vhosakotyes...20:26
gmmahamy linux bridge container is constnatly on restarting state :(20:26
gmmahaacross all 3 controller nodes20:26
vhosakotworked for me... you have neutron_plugin_agent: "linuxbridge" in ansible/group_vars/all.yml , right ?20:27
gmmahavhosakot: this is the log http://paste.openstack.org/show/491945/20:28
gmmahavhosakot: yes20:28
gmmahavhosakot:  grep linuxbridge globals.yml20:28
gmmaha# Valid options are [ openvswitch, linuxbridge ]20:28
gmmahaneutron_plugin_agent: "linuxbridge"20:28
vhosakotdid you have ovs (and ovs bridges, tap interfaces, qvo* and qvb* interfaces) before you deployed linuxbridge ?20:29
gmmahahmm.. let me check20:29
gmmahai have a feeling i might20:29
vhosakothmm... that may conflict... I have seen stale OVS bridges affect subsequent linux brodge deployemnt20:30
gmmahaaaah20:30
gmmahayeah i see ovs-system20:30
vhosakotgmmaha: not good.. hmm20:30
gmmahaouch20:30
gmmahaso blow them out by hand and then re-deploy?20:30
vhosakotgmmaha: can you post the output of "docker logs linux bridge container"20:30
vhosakotgmmaha: yes, try that... it should work20:30
vhosakotgmmaha: do you have tap interfaces, qvo* and qvb* interfaces as well from ovs ?20:31
gmmahavhosakot: docker logs linux bridge cotnainer gives me nothing20:31
gmmahai only see ovs-system.. dont see qvo and qvb20:31
vhosakotno docker logs for linux brodge container ?20:31
gmmahavhosakot: sorry the logs from the neutron_linuxbridge_agent is this http://paste.openstack.org/show/491945/20:32
*** inc0 has joined #kolla20:34
vhosakotgmmaha: the logs good20:35
gmmahabut just that its always shwoing as restarting20:35
*** achanda has quit IRC20:35
gmmahaLet me try and clean out the interfaces20:36
vhosakotgmmaha:  the logs look good... how do yoou plan to delete the interfaces ? ip link del ?20:55
gmmahavhosakot: that or brctl delbr20:55
gmmahaip link del seems an easier one given that brctl doesnt exist20:56
vhosakotgmmaha: I suggest, delete all the VM created prevuilsy using OVS, stop and remove ovs continaer, redeploy LB20:56
*** Jeffrey4l_ has quit IRC20:56
vhosakotyep, I have seen "brctl show ..." work inside the linux bridge container20:56
gmmahavhosakot: now i am not able to get intot he linuxbridge container at all..20:58
gmmahaas its constantly showing restarting :(20:58
gmmahalet me blow them all20:58
gmmahathere arent any VMs anyway20:58
vhosakotgmmaha: no issues, just recreate it20:58
*** vhosakot has quit IRC21:17
*** achanda has joined #kolla21:17
*** inc0 has quit IRC21:37
*** sdake has joined #kolla21:41
*** sdake_ has quit IRC21:44
*** pbourke-home has joined #kolla21:59
*** inc0 has joined #kolla22:00
*** pbourke-home has quit IRC22:03
*** pbourke-home has joined #kolla22:03
*** jtriley has quit IRC22:04
*** pbourke-home has quit IRC22:10
*** pbourke-home has joined #kolla22:10
*** inc0 has quit IRC22:14
*** Marga_ has quit IRC22:18
*** Marga_ has joined #kolla22:18
*** pbourke-home has quit IRC22:18
*** pbourke-home has joined #kolla22:22
*** jtriley has joined #kolla22:26
sdakemanjeets need the mariadb logs22:36
sdakedocker docker exec heka ls /var/log/mariadb22:36
sdakedocker exec heka cat /var/log/mariadb/mariadb.log iirc22:36
sdakemanjeets yuo have my full attention for several hours if you wnt to deebug22:37
openstackgerritWaldemar Znoinski proposed openstack/kolla: add docutils, pytz to requirements.txt  https://review.openstack.org/29788322:37
manjeetssdake: thanks22:37
manjeetsI appreciate22:37
manjeetslet me grab that22:37
sdakedid deployment happen successfully?22:38
manjeetsno22:38
sdakecan you paste where it failed22:38
manjeetsI tried re doing all the stuff http://paste.openstack.org/show/491940/22:38
manjeetsit fails on mariadb22:38
sdakemanjeets there aer several mariadb steps, which one did it fail on?22:39
sdakeare you on a mac or a windows or linux machine for your irc client?22:39
manjeetslet me re run it jost lost that log22:40
manjeetsit will be here in a minute or so22:40
sdakeif your on mac or windows we can use webex screen share22:40
sdakeit will go much faster22:40
sdakealso need  your inventoryfile22:41
sdakeand your globals.yml file22:41
manjeetssdake: failing step http://paste.openstack.org/show/491936/22:41
manjeetssdake: my inventory22:42
manjeetshttp://paste.openstack.org/show/491939/22:43
*** sdake_ has joined #kolla22:44
manjeetssdake http://paste.openstack.org/show/491952/ global.yml22:44
sdake_manjeets you are lacking an internal address22:45
sdake_where did you get that globals file from?22:46
*** pbourke-home has quit IRC22:47
*** sdake has quit IRC22:47
sbezverksdake last night we debugged majeets setup and no matter what mariadb process was getting stopped without much info..22:47
manjeetssdake_: ihave that paste missed it22:48
manjeetsmy global.yml http://paste.openstack.org/show/491953/22:50
sdake_what is 0.208?22:51
sdake_an empty ip address?22:51
sdake_empty/unassigned22:52
sdake_201 is your registry22:52
*** jtriley has quit IRC22:55
*** salv-orlando has joined #kolla23:17
*** SiRiuS_ has quit IRC23:37
*** eandersson_ has quit IRC23:44
*** vhosakot has joined #kolla23:45
sdake_is multinode kolla broken?23:54

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!