Tuesday, 2018-01-23

alee__arxcruz, weshay|rover good to know.  interesting , because it sometimes passed00:01
alee__arxcruz, weshay|rover lets hope this fixes the problem :)00:01
alee__weshay|rover, I'd like to keep the environment till tommorow to diagnose the logging issue ..00:02
*** itlinux has quit IRC00:03
*** alee__ is now known as alee_dinner00:03
*** anilvenkata has quit IRC00:04
*** ooolpbot has joined #tripleo00:10
*** ooolpbot has quit IRC00:10
openstackLaunchpad bug 1742936 in tripleo "TestBootVolumePattern is failing in multinode jobs" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)00:10
*** Hazelesque has quit IRC00:12
*** rdopiera has quit IRC00:13
*** tbarron has quit IRC00:13
*** tbarron has joined #tripleo00:13
*** rdopiera has joined #tripleo00:13
*** Hazelesque has joined #tripleo00:15
*** tosky has quit IRC00:17
*** noslzzp has joined #tripleo00:23
*** noslzzp has quit IRC00:24
*** noslzzp has joined #tripleo00:24
*** chlong has joined #tripleo00:24
*** thrash is now known as thrash|g0ne00:27
*** jkilpatr has quit IRC00:28
*** tzumainn has quit IRC00:32
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart-extras master: DNM - debug  https://review.openstack.org/53655600:34
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart-extras master: config-download: handle special case for 3nodes job  https://review.openstack.org/53547200:38
*** daidv has quit IRC00:39
*** daidv has joined #tripleo00:39
openstackgerritMerged openstack/diskimage-builder master: Correct link address  https://review.openstack.org/53519900:51
openstackgerritMerged openstack/diskimage-builder master: Add SUSE Mapping  https://review.openstack.org/53292500:51
*** karthiks has quit IRC00:52
*** karthiks has joined #tripleo00:54
*** d0ugal has quit IRC00:55
openstackgerritMerged openstack/tripleo-common-tempest-plugin master: Remove obsolete tempest-lib  https://review.openstack.org/53605000:59
*** d0ugal has joined #tripleo01:00
EmilienMstevebaker: hey, still around?01:04
stevebakerEmilienM: yep01:05
EmilienMstevebaker: I've been trying to switch the multinode job with a 2nodes overcloud into containers01:05
EmilienMhere: https://review.openstack.org/#/c/534977/01:05
EmilienMand it fails on the networking config: http://logs.openstack.org/77/534977/2/check/tripleo-ci-centos-7-3nodes-multinode/b5d7284/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-01-19_07_38_3101:05
EmilienMI'm not sure how to debug NetworkDeployment01:06
EmilienMAFIK it creates the bridges and the interfaces, right?01:06
*** mdnadeem has joined #tripleo01:06
stevebakerEmilienM: eys01:07
*** mdnadeem has quit IRC01:07
*** mdnadeem has joined #tripleo01:07
EmilienMmaybe net-config-multinode-os-net-config.yaml doesn't work with the controllerapi role01:08
stevebakerEmilienM: are there any interesting logging showing a failure?01:09
*** ooolpbot has joined #tripleo01:10
openstackLaunchpad bug 1742936 in tripleo "TestBootVolumePattern is failing in multinode jobs" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)01:10
*** ooolpbot has quit IRC01:10
EmilienMI haven't found anything exiting01:10
EmilienMexciting even01:10
EmilienMstevebaker: I found the trace in Heat but it's the same error message01:11
EmilienMnothing that tells where it fails exactly01:11
EmilienMI'll probably have to add some debug code01:11
EmilienMstevebaker: we probably miss os-net-config logs01:12
stevebakerEmilienM: I'll spend the rest of today looking too. I'm stuck on a unit test which fails in CI but passes locally01:13
EmilienMstevebaker: link?01:13
*** psahoo has joined #tripleo01:14
stevebakerEmilienM: http://logs.openstack.org/76/526576/7/check/openstack-tox-py27/f63fab5/01:14
*** Goneri has quit IRC01:14
openstackgerritMerged openstack/diskimage-builder master: Remove RH1 check OVB jobs from configuration  https://review.openstack.org/53648901:15
EmilienMstevebaker: let me give it a try01:18
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/pike: Containerize multinode-3nodes  https://review.openstack.org/53497701:19
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/pike: Containerize multinode-3nodes  https://review.openstack.org/53497701:19
EmilienMstevebaker: fails for me01:22
*** liverpooler has quit IRC01:23
EmilienMtripleoclient.tests.v1.overcloud_image.test_container_image.TestContainerImagePrepare.test_container_image_prepare fails01:23
stevebakerEmilienM: is tripleo-ci-centos-7-3nodes-multinode working on master? I see it didn't pass when https://review.openstack.org/#/c/532623 landed01:23
EmilienMstevebaker: http://paste.openstack.org/show/650748/01:23
EmilienMstevebaker: yeah it does01:24
EmilienMstevebaker: nevermind 3nodes, I think I used the wrong network template01:24
EmilienMstevebaker: I'm trying with the previous that worked01:24
tonybstevebaker: fails for me too but differently from CI (same as EmilienM) ... Can you rebuild you toxenv(s) ?01:25
stevebakerEmilienM, tonyb: it passes for me in a clean tox on fedora, trying centos now01:26
tonybstevebaker: strange :( I'm on Fedora26 FWIW01:27
EmilienMtried on a fresh tox on fedora01:27
stevebakerweird, its passing on my centos too01:28
*** noslzzp has quit IRC01:53
openstackgerritwes hayutin proposed openstack-infra/tripleo-ci master: Configure RDO cloud jobs to send stats to InfluxDB  https://review.openstack.org/53443401:53
*** ihrachys has quit IRC01:59
*** ihrachys has joined #tripleo02:00
*** yolanda has joined #tripleo02:09
*** ooolpbot has joined #tripleo02:10
openstackLaunchpad bug 1742936 in tripleo "TestBootVolumePattern is failing in multinode jobs" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)02:10
*** ooolpbot has quit IRC02:10
*** rbrady has quit IRC02:18
*** liverpooler has joined #tripleo02:24
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart master: DNM: Test feature sets with downstream  https://review.openstack.org/53645702:29
*** rbrady has joined #tripleo02:33
*** rbrady has joined #tripleo02:33
openstackgerritTony Breeds proposed openstack/instack-undercloud master: Ensure Boolean Strings from the environment are treated correctly  https://review.openstack.org/53460402:40
*** fzdarsky has quit IRC02:42
*** fzdarsky has joined #tripleo02:43
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Don't fstrim vfat partitions  https://review.openstack.org/53496902:50
openstackgerritIan Wienand proposed openstack/diskimage-builder master: GPT partitioning support  https://review.openstack.org/53349002:50
*** bkopilov has quit IRC02:53
openstackgerritIan Wienand proposed openstack/diskimage-builder master: bootloader: handle grub-efi better  https://review.openstack.org/53660002:56
openstackgerritIan Wienand proposed openstack/diskimage-builder master: change default image into GPT with ESP and BSP  https://review.openstack.org/53660102:56
*** jaganathan has quit IRC02:57
openstackgerritIan Wienand proposed openstack/diskimage-builder master: GPT partitioning support  https://review.openstack.org/53349002:59
openstackgerritIan Wienand proposed openstack/diskimage-builder master: bootloader: handle grub-efi better  https://review.openstack.org/53660002:59
openstackgerritIan Wienand proposed openstack/diskimage-builder master: change default image into GPT with ESP and BSP  https://review.openstack.org/53660102:59
openstackgerritMerged openstack/diskimage-builder master: Updated from global requirements  https://review.openstack.org/53395803:03
*** agopi__ has joined #tripleo03:03
*** ooolpbot has joined #tripleo03:10
*** ooolpbot has quit IRC03:10
openstackLaunchpad bug 1742936 in tripleo "TestBootVolumePattern is failing in multinode jobs" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)03:10
openstackgerritlvxianguo proposed openstack/tripleo-quickstart-extras master: assertEquals is deprecated, use assertEqual  https://review.openstack.org/53665903:12
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart-extras master: config-download: handle special case for 3nodes job  https://review.openstack.org/53547203:12
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: fs032: activate config-download  https://review.openstack.org/52876103:13
*** jaganathan has joined #tripleo03:14
*** ramishra has joined #tripleo03:16
*** udesale has joined #tripleo03:20
*** fragatina has quit IRC03:25
*** daidv has quit IRC03:30
*** akane has joined #tripleo03:30
*** akane has quit IRC03:31
*** akane has joined #tripleo03:32
*** numans has joined #tripleo03:37
openstackgerritMerged openstack/tripleo-heat-templates stable/pike: Correct indentation of docker_puppet_apply.sh run  https://review.openstack.org/53654703:37
*** rlandy has quit IRC03:43
*** daidv has joined #tripleo03:44
*** liverpooler has quit IRC03:55
*** yamahata has joined #tripleo03:57
*** fragatina has joined #tripleo04:01
*** links has joined #tripleo04:02
*** skramaja has joined #tripleo04:04
*** fragatina has quit IRC04:05
openstackgerritTony Breeds proposed openstack/python-tripleoclient master: Add --validate-only to openstack overcloud node import  https://review.openstack.org/52860204:06
openstackgerritTony Breeds proposed openstack/python-tripleoclient master: Allow passing an agent-name as a command line argument  https://review.openstack.org/53561504:06
openstackgerritTony Breeds proposed openstack/python-tripleoclient master: Introduce trivial helpers to manage image_files and names  https://review.openstack.org/53561604:06
openstackgerritTony Breeds proposed openstack/python-tripleoclient master: Always define properties for image even if empty  https://review.openstack.org/53561704:06
openstackgerritTony Breeds proposed openstack/python-tripleoclient master: Add --architecture support when uploading images  https://review.openstack.org/53561804:06
openstackgerritTony Breeds proposed openstack/python-tripleoclient master: Add --platform support when uploading images  https://review.openstack.org/53561904:06
openstackgerritTony Breeds proposed openstack/python-tripleoclient master: Actually use the arch and platform to select appropriate images  https://review.openstack.org/53562004:06
openstackgerritTony Breeds proposed openstack/python-tripleoclient master: Switch order of actual and reference args to assertEqual  https://review.openstack.org/53562104:06
*** links has quit IRC04:07
*** ooolpbot has joined #tripleo04:10
*** ooolpbot has quit IRC04:10
openstackLaunchpad bug 1742936 in tripleo "TestBootVolumePattern is failing in multinode jobs" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)04:10
openstackgerritMerged openstack/tripleo-common master: Add healthcheck script for Nova placement API  https://review.openstack.org/53597704:10
openstackgerritMerged openstack/tripleo-quickstart master: Remove RH1 OVB jobs form configuration  https://review.openstack.org/53646604:10
*** fragatina has joined #tripleo04:10
*** moshele has joined #tripleo04:11
*** fragatina has quit IRC04:11
*** skramaja has quit IRC04:12
*** skramaja_ has joined #tripleo04:12
*** skramaja_ is now known as skramaja04:16
*** links has joined #tripleo04:19
*** udesale_ has joined #tripleo04:19
*** ykarel|afk has joined #tripleo04:19
*** udesale has quit IRC04:20
*** moshele has quit IRC04:20
*** bkopilov has joined #tripleo04:24
*** daidv has quit IRC04:28
*** daidv has joined #tripleo04:28
*** karthiks has quit IRC04:28
*** karthiks has joined #tripleo04:30
*** yolanda has quit IRC04:32
*** psachin has joined #tripleo04:41
*** fragatina has joined #tripleo04:45
*** gvrangan has joined #tripleo04:47
*** jaganathan has quit IRC04:48
*** gvrangan has quit IRC04:51
openstackgerritIan Wienand proposed openstack/diskimage-builder master: GPT partitioning support  https://review.openstack.org/53349004:53
openstackgerritIan Wienand proposed openstack/diskimage-builder master: bootloader: handle grub-efi better  https://review.openstack.org/53660004:53
openstackgerritIan Wienand proposed openstack/diskimage-builder master: change default image into GPT with ESP and BSP  https://review.openstack.org/53660104:53
openstackgerritMerged openstack/diskimage-builder master: Don't fstrim vfat partitions  https://review.openstack.org/53496904:55
*** dmacpher has quit IRC04:56
*** dsneddon has quit IRC04:56
*** akane has quit IRC04:57
openstackgerritRajini Karthik proposed openstack/tripleo-heat-templates master: Add support for Dell EMC VNX Cinder Backend  https://review.openstack.org/53461204:57
*** stendulker has joined #tripleo04:59
*** ratailor has joined #tripleo05:01
*** dpawar has joined #tripleo05:05
*** links has quit IRC05:06
*** links has joined #tripleo05:08
*** gvrangan has joined #tripleo05:09
*** ooolpbot has joined #tripleo05:10
*** ooolpbot has quit IRC05:10
openstackLaunchpad bug 1742936 in tripleo "TestBootVolumePattern is failing in multinode jobs" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)05:10
*** gvrangan has quit IRC05:13
*** gvrangan has joined #tripleo05:14
*** moshele has joined #tripleo05:16
*** gvrangan has quit IRC05:16
*** masco has quit IRC05:16
*** ramishra has quit IRC05:20
*** ramishra has joined #tripleo05:22
*** pcaruana has joined #tripleo05:24
*** ratailor has quit IRC05:25
*** akane has joined #tripleo05:27
*** ratailor has joined #tripleo05:27
*** pgadiya has joined #tripleo05:29
*** ramishra has quit IRC05:32
*** ratailor has quit IRC05:34
*** ramishra has joined #tripleo05:36
*** pcaruana has quit IRC05:36
*** ratailor has joined #tripleo05:41
*** ratailor_ has joined #tripleo05:45
*** ratailor has quit IRC05:47
*** artom has quit IRC05:51
*** artom has joined #tripleo05:51
*** anilvenkata has joined #tripleo05:54
*** jaosorior has quit IRC05:57
*** jaosorior has joined #tripleo05:57
*** ooolpbot has joined #tripleo06:10
*** ooolpbot has quit IRC06:10
openstackLaunchpad bug 1742936 in tripleo "TestBootVolumePattern is failing in multinode jobs" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)06:10
*** janki has joined #tripleo06:13
*** numans is now known as numans_afk06:14
*** yamahata has quit IRC06:15
*** mdnadeem has quit IRC06:17
*** waleedm has joined #tripleo06:23
*** jaganathan has joined #tripleo06:25
*** ykarel|afk is now known as ykarel06:27
*** mdnadeem has joined #tripleo06:32
*** masco has joined #tripleo06:34
*** ratailor__ has joined #tripleo06:36
*** ratailor_ has quit IRC06:38
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-ui master: Imported Translations from Zanata  https://review.openstack.org/53563606:39
*** jtomasek has joined #tripleo06:40
*** dsariel has joined #tripleo06:44
*** suuuper has joined #tripleo06:47
*** etingof has quit IRC06:52
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: DNM: Display hostvars  https://review.openstack.org/53632506:52
*** itlinux has joined #tripleo06:56
*** psachin has quit IRC06:56
*** itlinux has quit IRC06:59
*** marios has joined #tripleo07:00
*** numans_afk is now known as numans07:04
*** etingof has joined #tripleo07:05
*** ooolpbot has joined #tripleo07:10
*** ooolpbot has quit IRC07:10
openstackLaunchpad bug 1742936 in tripleo "TestBootVolumePattern is failing in multinode jobs" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)07:10
*** dparkes has joined #tripleo07:10
*** jrist has quit IRC07:15
openstackgerritJanki Chhatbar proposed openstack/tripleo-heat-templates master: Add endpoint for ODL  https://review.openstack.org/53578307:16
*** fragatina has quit IRC07:18
*** itlinux has joined #tripleo07:26
*** agurenko has joined #tripleo07:30
*** pcaruana has joined #tripleo07:33
*** akane has quit IRC07:33
*** SlickNik has joined #tripleo07:34
*** jwb has joined #tripleo07:42
*** tdasilva has quit IRC07:47
openstackgerritMartin Mágr proposed openstack/tripleo-common master: Add healthcheck script for Panko API  https://review.openstack.org/53598707:53
*** jfrancoa has joined #tripleo07:53
*** jrist has joined #tripleo07:53
openstackgerritMartin Mágr proposed openstack/tripleo-common master: Add healthcheck script for Redis  https://review.openstack.org/53640507:53
*** psachin has joined #tripleo07:54
openstackgerritMartin Mágr proposed openstack/tripleo-common master: Add healthcheck script for Tacker  https://review.openstack.org/53636007:54
*** itlinux has quit IRC07:58
*** pcaruana has quit IRC07:58
*** rcernin has quit IRC08:00
openstackgerritMerged openstack/tripleo-quickstart-extras master: Remove RH1 OVB jobs form configuration  https://review.openstack.org/53646808:05
*** paramite has joined #tripleo08:06
*** etingof has quit IRC08:07
*** tdasilva has joined #tripleo08:08
*** tesseract has joined #tripleo08:09
*** ooolpbot has joined #tripleo08:10
*** ooolpbot has quit IRC08:10
openstackLaunchpad bug 1742936 in tripleo "TestBootVolumePattern is failing in multinode jobs" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)08:10
*** nyechiel_ has joined #tripleo08:10
*** pcaruana has joined #tripleo08:13
*** aufi has joined #tripleo08:16
*** cylopez has joined #tripleo08:17
*** yprokule has joined #tripleo08:20
*** cshastri has joined #tripleo08:22
*** akane has joined #tripleo08:29
*** akane__ has joined #tripleo08:32
*** akane has quit IRC08:32
*** agurenko_ has joined #tripleo08:33
*** nyechiel_ has quit IRC08:33
*** bkopilov has quit IRC08:33
*** dsariel has quit IRC08:33
*** nyechiel_ has joined #tripleo08:33
*** agurenko has quit IRC08:34
*** agopi__ has quit IRC08:37
*** daidv has quit IRC08:38
*** daidv has joined #tripleo08:38
*** holser has joined #tripleo08:40
*** mdnadeem has quit IRC08:42
*** bogdando has joined #tripleo08:43
*** amoralej|off is now known as amoralej08:43
*** oidgar has joined #tripleo08:44
*** bkopilov has joined #tripleo08:46
*** itlinux has joined #tripleo08:46
*** dsariel has joined #tripleo08:46
openstackgerritMerged openstack/tripleo-heat-templates stable/pike: Containerize multinode-3nodes  https://review.openstack.org/53497708:49
*** jpena|off is now known as jpena08:49
*** paramite has quit IRC08:51
openstackgerritJanki Chhatbar proposed openstack/tripleo-heat-templates master: Upgrade ODL  https://review.openstack.org/48920108:52
*** oidgar has quit IRC08:52
*** dsneddon has joined #tripleo08:53
*** jbadiapa has joined #tripleo08:53
*** paramite has joined #tripleo08:55
*** shardy has joined #tripleo08:56
*** nyechiel_ has quit IRC09:01
*** nyechiel has joined #tripleo09:01
*** mdnadeem has joined #tripleo09:01
*** agurenko_ has quit IRC09:01
*** eck` is now known as eck`gone09:01
*** dsariel has quit IRC09:01
*** agurenko_ has joined #tripleo09:01
*** bkopilov has quit IRC09:02
*** jpich has joined #tripleo09:03
*** athomas has joined #tripleo09:06
*** nyechiel has quit IRC09:09
*** ooolpbot has joined #tripleo09:10
*** ooolpbot has quit IRC09:10
openstackLaunchpad bug 1742936 in tripleo "TestBootVolumePattern is failing in multinode jobs" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)09:10
*** nyechiel has joined #tripleo09:10
*** itlinux has quit IRC09:12
*** agurenko_ has quit IRC09:13
*** dsariel has joined #tripleo09:14
*** bkopilov has joined #tripleo09:15
*** artom has quit IRC09:20
*** artom has joined #tripleo09:21
*** mmethot_ has joined #tripleo09:21
*** mmethot has quit IRC09:21
*** oidgar has joined #tripleo09:25
*** agurenko has joined #tripleo09:26
jaosoriorshardy: there seems to be something really wrong with the inventory09:27
jaosoriorshardy: I'm merely trying to print the hostvars variable09:28
jaosoriorlike this09:28
jaosoriorand that gives an error too http://logs.openstack.org/25/536325/3/check-tripleo/tripleo-ci-centos-7-ovb-ha-oooq/e09d401/logs/undercloud/var/lib/mistral/72210520-1e22-4e43-b43c-688d0e1ac413/ansible.log.txt.gz#_2018-01-23_09_01_41_31209:28
jaosoriorshardy: I've been trying to reproduce it locally, but haven't been able to do a successful config-download deployment because it gets stuck at: Deploying overcloud configuration09:29
jaosoriorand in mistral the logs output this:09:29
jaosorior2018-01-23 08:40:27.752 17588 ERROR mistral.executors.default_executor Stderr: u''09:29
jaosorior2018-01-23 08:40:27.752 17588 ERROR mistral.executors.default_executor09:30
jaosoriorthis was a fresh deployment I did today09:30
jaosoriorwith tripleo-master-ci09:30
*** stendulker has quit IRC09:30
*** iranzo has joined #tripleo09:31
*** iranzo has joined #tripleo09:31
*** mcornea has joined #tripleo09:32
*** tesseract-RH has joined #tripleo09:33
shardyjaosorior: ack looking - does running config download then ansible-playbook manually work for you locally?09:34
*** tesseract-RH has quit IRC09:34
shardythat's how I do most of my testing and it's easier for debugging than --config-download IME09:34
jaosoriorshardy: that's what I've been doing as well09:35
*** tesseract-RH has joined #tripleo09:35
jaosoriorshardy: that was working, gonna try it again in my new deployment09:35
jaosoriorshardy: how is it working in CI currently?09:35
jaosorior--config-downlaod? or doing a manual download?09:35
shardyjaosorior: what leads you to believe it's an inventory error - that log looks like a task failed?09:36
shardyjaosorior: not sure tbh, I thought the 3nodes job was running --config-download but best to check09:36
*** numans has quit IRC09:37
*** tesseract has quit IRC09:37
jaosoriorshardy: becauase that log fails when outputting hostvars09:37
jaosoriorshardy: I also tried using hostvars[inventory_hostname]  and that lead to an exception as well09:39
jaosoriorso I tried this https://review.openstack.org/#/c/536325/2/extraconfig/services/ipsec.yaml   and this https://review.openstack.org/#/c/536325/3/extraconfig/services/ipsec.yaml09:39
*** derekh has joined #tripleo09:39
shardyUnexpected failure during module execution - not very informative error :(09:40
jaosoriorshardy: anyway, the error I was getting when I added verbosity was this http://logs.openstack.org/34/535234/1/check-tripleo/tripleo-ci-centos-7-ovb-ha-oooq/bc6da56/logs/undercloud/var/lib/mistral/845008d8-840e-4721-85ee-2b5876ed8eb5/ansible.log.txt.gz#_2018-01-19_11_04_36_79309:41
shardyhttps://github.com/ansible/ansible/issues/31679 might be related?09:41
shardyseems like it's a magic variable that you need to access by inventory name and keys and/or via set_fact?09:42
shardymsg: "{{ hostvars[inventory_hostname].keys() }}" may work for the debug task, or have you already tried that?09:43
*** dbecker has joined #tripleo09:43
jaosoriorshardy: I'll give that a try09:44
shardyjaosorior: actually that's not a bug, but my point is I don't think you can dump the entire hostvars map, only per host via the inventory_hostname09:44
jaosoriorshardy: but that's just how I'm trying to debug09:45
jaosoriormy actual error is when trying to access a key09:45
jaosoriorso I do09:45
jaosoriorand that's failing09:45
jaosoriorwith an exception09:45
jaosoriorwhich doesn't fail when I run ansible directly09:45
shardyack maybe dumping the keys will help09:45
jaosoriorgonna try it out09:46
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: DNM: Display hostvars  https://review.openstack.org/53632509:47
jaosoriorlets see09:47
*** etingof has joined #tripleo09:48
shardyjaosorior: so it looks like the --config-download workflow uses --static-yaml-inventory vs the dynamic inventory09:49
shardyso that could be something to try locally09:49
shardyjaosorior: for some reason, ansible json/yaml structure is different for static and dynamic inventory sources09:50
shardyso it's possible I missed something in the format mangling09:50
shardyI have no idea why they did that :(09:50
shardyit's not even very well documented AFAICS09:50
shardymaybe sdoran can shed some light on it later09:51
shardythat's the flag that enables the alternate format for static inventory output, maybe I missed something there wrt hostvars09:52
jaosoriorshardy: I'm starting to think some invalid characters sneaks up in there09:53
jaosoriorgonna try with the dynamic inventory and with the yaml one09:53
jaosoriorlets see how it goes09:53
jaosoriorall I need is the <network>_ip  and <network>_vip variables09:53
*** salmankhan has joined #tripleo09:54
*** ccamacho has joined #tripleo09:55
jaosoriorbe right back09:55
*** jaosorior has quit IRC09:55
*** gfidente has joined #tripleo09:56
*** gfidente has quit IRC09:56
*** gfidente has joined #tripleo09:56
*** wojdec has joined #tripleo09:58
*** dtantsur|afk is now known as dtantsur09:58
wojdecHi. I've been trying to get this patch verified https://review.openstack.org/#/c/533351/ , however it fails at various steps (never the same). Any hints as to what's wrong?09:59
*** stendulker has joined #tripleo09:59
*** apevec has joined #tripleo10:01
shardywojdec: Hi, we have had a few issues in CI but it should be OK now, looking10:02
*** nyechiel_ has joined #tripleo10:03
*** nyechiel has quit IRC10:04
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Add yaml validations for FFU.  https://review.openstack.org/53470910:05
*** agurenko has quit IRC10:05
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: ffu: Introduce Glance fast-forward upgrade tasks  https://review.openstack.org/52154410:05
wojdecshardy: I'll run a recheck, unless you see something that is clearly my fault10:06
*** tosky has joined #tripleo10:06
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Include extra nodes in te-broker config  https://review.openstack.org/53674210:06
*** pgadiya has quit IRC10:09
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: ffu: Introduce Neutron fast-forward upgrade tasks  https://review.openstack.org/52154310:09
*** ooolpbot has joined #tripleo10:10
*** ooolpbot has quit IRC10:10
openstackLaunchpad bug 1742936 in tripleo "TestBootVolumePattern is failing in multinode jobs" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz)10:10
shardywojdec: I added some review comments so please check those first10:10
shardywojdec: I don't think the CI errors are related, so when you push a revision hopefully it will pass10:10
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: DNM: test influxdb sending  https://review.openstack.org/53421810:12
wojdecok. tx10:12
shardywojdec: I guess the point of the new service template is just to enable the neutron_plugin_ml2 tags for the docker-puppet (puppet_config), but I was surprised to see no other changes10:12
shardyif that works then I'm fine to +2 when the passwords are removed/commented10:13
*** itlinux has joined #tripleo10:15
wojdecshardy: so a basic question about the "properties" element in the "resources"… I noticed, by experimentation that if the parent template has the properties already defined, then these properties do not need to be re-assigned at the resources level. That said, several templates seem to do just that. Not sure about the reason10:17
*** ccamacho1 has joined #tripleo10:18
*** ccamacho1 has quit IRC10:19
*** ccamacho1 has joined #tripleo10:19
*** ccamacho has quit IRC10:19
shardywojdec: There are some parameters that are passed from the parent template, e.g EndpointMap, ServiceData, ServiceNetMap, DefaultPasswords etc10:21
shardywojdec: in those cases, you should pass the parameter value via properties as you have done, e.g to NeutronBase10:22
*** ccamacho has joined #tripleo10:22
shardywojdec: if you don't, you end up with weird bugs where the nested service template uses defaults instead of the value generated in the parent10:22
shardywojdec: probably we should remove all the defaults to enforce that10:23
wojdecshardy: Ok, thanks for the explanation. I'll add to the list I have then10:24
*** ccamacho1 has quit IRC10:24
*** pgadiya has joined #tripleo10:25
*** marios has quit IRC10:30
*** ramishra has quit IRC10:31
*** timothyb89 has quit IRC10:32
*** marios has joined #tripleo10:32
*** ramishra has joined #tripleo10:32
*** jfrancoa has quit IRC10:33
*** timothyb89 has joined #tripleo10:33
openstackgerritJanki Chhatbar proposed openstack/tripleo-heat-templates master: Add endpoint for ODL  https://review.openstack.org/53578310:34
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Add ability  to deploy FreeIPA on OVB  https://review.openstack.org/53443510:34
skramajamandre: related tht patch for ovs user - https://review.openstack.org/#/c/529272/10:35
*** lucas-afk is now known as lucasagomes10:35
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart master: Add Featureset 39 -- HA deploy w/ IPA  https://review.openstack.org/53378510:37
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Add ability  to deploy FreeIPA on OVB  https://review.openstack.org/53443510:37
*** mdnadeem has quit IRC10:39
*** jbadiapa has quit IRC10:41
*** psahoo has quit IRC10:42
openstackgerritDerek Higgins proposed openstack/tripleo-heat-templates master: WIP: Add ironic config to scenario 004  https://review.openstack.org/48526110:43
mandreskramaja: ack, I'll do a first pass on it10:44
*** nyechiel_ has quit IRC10:44
*** nyechiel has joined #tripleo10:46
*** itlinux has quit IRC10:47
openstackgerritMoshe Levi proposed openstack/tripleo-heat-templates master: Adding support for odl hw offloading in containers  https://review.openstack.org/53462210:48
openstackgerritTim Flink proposed openstack/diskimage-builder master: Add support for Fedora 27, remove EOL Fedora 25  https://review.openstack.org/53675910:49
*** shardy has quit IRC10:49
*** jfrancoa has joined #tripleo10:51
*** shardy has joined #tripleo10:51
*** ramishra has quit IRC10:52
*** itlinux has joined #tripleo10:53
*** jaosorior has joined #tripleo10:53
*** ramishra has joined #tripleo10:54
*** nyechiel has quit IRC10:57
*** nyechiel_ has joined #tripleo10:57
*** stendulker has quit IRC11:00
*** pgadiya has quit IRC11:02
*** skramaja has quit IRC11:03
*** pgadiya has joined #tripleo11:05
*** itlinux has quit IRC11:07
*** salmankhan has quit IRC11:08
*** jfrancoa has quit IRC11:11
*** bkopilov has quit IRC11:12
*** jbadiapa has joined #tripleo11:13
*** nyechiel has joined #tripleo11:18
*** nyechiel_ has quit IRC11:18
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-quickstart-extras master: Use repos variable for the upgrade jobs.  https://review.openstack.org/53646711:20
*** jfrancoa has joined #tripleo11:25
*** numans has joined #tripleo11:27
*** salmankhan has joined #tripleo11:35
openstackgerritNoam Angel proposed openstack/diskimage-builder master: Add support for Fedora 27, remove EOL Fedora 25  https://review.openstack.org/53675911:37
*** jbadiapa has quit IRC11:38
*** paramite has quit IRC11:39
openstackgerritJiri Stransky proposed openstack/tripleo-upgrade master: More robust check for deployment_with_ha  https://review.openstack.org/53637111:41
openstackgerritJiri Stransky proposed openstack/tripleo-upgrade master: Less opinionated detection of service env files  https://review.openstack.org/53640211:41
*** rfolco|off is now known as rfolco|rover11:41
openstackgerritJiri Stransky proposed openstack/tripleo-upgrade master: More robust check for deployment_with_ha  https://review.openstack.org/53637111:43
openstackgerritJiri Stransky proposed openstack/tripleo-upgrade master: Less opinionated detection of service env files  https://review.openstack.org/53640211:43
*** ratailor__ has quit IRC11:44
*** itlinux has joined #tripleo11:45
*** dtantsur is now known as dtantsur|brb11:50
*** tesseract-RH has quit IRC11:52
openstackgerritBogdan Dobrelya proposed openstack/tripleo-heat-templates master: OpenShift: Don't pre-install Docker  https://review.openstack.org/53282711:52
openstackgerritMerged openstack/python-tripleoclient stable/pike: Push the config container outside of heat stack update  https://review.openstack.org/53432011:53
bogdandohttps://review.openstack.org/#/c/535689/ PTAL :)11:54
jaosoriorshardy: funky, printing the keys as you suggested still returns an exception" https://logs.rdoproject.org/25/536325/4/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/Z34509e3470ea42b4a4b2c9e96bd7aa76/undercloud/var/lib/mistral/4734f77f-608c-4403-b5a4-ee33bdd8ede7/ansible.log.txt.gz#_2018-01-23_11_22_30_03411:55
*** pkovar has joined #tripleo11:56
*** paramite has joined #tripleo11:57
*** dmacpher has joined #tripleo11:59
shardyjaosorior: Oh hey, you dropped off earlier, I tested it locally with your original syntax, it works fine12:01
shardyso I am confused12:01
jaosoriorshardy: did you test it with the static yaml file?12:01
shardyjaosorior: yes ^^12:02
jaosoriorshardy: I'm quite confused too, haven't been able to reproduce it locally12:02
shardyI was hoping that would fail12:02
jaosoriorbut it sure seems to fail in CI12:02
shardyyea, have you tried adding -vvv to get more info out of ansible?12:03
jaosoriorshardy: not with the debug statement. But I did try that with my original issue. And that's the stacktrace I passed you earlier12:03
shardyjaosorior: ack, just trying to work out how to proceed, as that earlier ansible error doesn't really tell us much :(12:04
jaosoriorshardy: so, mistral calls the dynamic ansible inventory, gets the yaml file, and passes in that yaml file? or how does the workflow work?12:04
jaosoriorI'm thinking it has to do with that workflow12:04
shardyjaosorior: I'd maybe try doing a local deploy with HostnameMap overriding the hostname12:04
*** raildo has joined #tripleo12:04
shardyjaosorior: it creates a static yaml inventory the same as --static-yaml-inventory, I linked where earlier12:05
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: DNM: Display hostvars  https://review.openstack.org/53632512:05
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: DNM: Display hostvars  https://review.openstack.org/53632512:06
jaosoriorshardy: I actually don't think we even pass the HostnameMap in that job12:06
shardyjaosorior: you could maybe add a log to tripleo-common/inventory.py so we see the whole inventory, unless that's already saved somewhere?12:06
shardyjaosorior: Ok, that was related to the problem Emilien had but I wasn't sure if it could be related12:07
shardythere must be some key difference between the CI config and what we both have working locally tho12:07
*** aditya_r has joined #tripleo12:10
*** jkilpatr has joined #tripleo12:12
*** atoth has joined #tripleo12:14
*** gfidente has quit IRC12:19
*** anilvenkata has quit IRC12:19
*** gfidente has joined #tripleo12:20
*** gfidente^2nd has joined #tripleo12:20
*** gfidente^2nd has quit IRC12:22
*** gfidente has quit IRC12:22
*** dtantsur|brb is now known as dtantsur12:23
*** panda is now known as panda|lunch12:25
*** lvdombrkr has joined #tripleo12:27
*** gfidente has joined #tripleo12:27
*** gfidente has quit IRC12:27
*** gfidente has joined #tripleo12:27
*** jpena is now known as jpena|lunch12:33
*** ebarrera has joined #tripleo12:34
*** yamahata has joined #tripleo12:34
*** aditya_r has quit IRC12:34
*** masco has quit IRC12:39
*** wojdec has quit IRC12:43
*** ebarrera has quit IRC12:44
*** tesseract-RH has joined #tripleo12:47
*** psachin has quit IRC12:51
jaosoriorshardy: right, I ran the overcloud deploy with IPSEC and the static yaml environment and it's working. It seems it's an issue with --config-download. I guess the next step is to add the debug in tripleo-common/inventory.py as you suggested, and figure out what changes in CI12:53
*** jlabarre has joined #tripleo12:54
*** tesseract-RH has quit IRC12:54
jaosoriorshardy: by the way, could you review this if you have some time? https://review.openstack.org/#/c/533704/12:54
dtantsurhi all, could you review https://review.openstack.org/#/c/533145/ please? it's been without attention for more than a week12:55
*** tesseract has joined #tripleo12:55
itlinuxhello all.. I wonder if there is a way to leverage the UC ironic/dhcp for non stack.. like other baremetal solutions (KVM) that will serve as other project12:58
*** panda|lunch is now known as panda12:58
*** ebarrera has joined #tripleo12:59
*** morazi has joined #tripleo13:02
*** abishop has joined #tripleo13:02
*** dmacpher has quit IRC13:02
openstackgerritDmitry Tantsur proposed openstack/tripleo-common stable/pike: Discover nova hosts in cells right before deployment  https://review.openstack.org/53432813:03
*** yamahata has quit IRC13:04
*** dmacpher has joined #tripleo13:05
*** thrash|g0ne is now known as thrash13:05
*** yolanda has joined #tripleo13:05
*** lblanchard has joined #tripleo13:07
*** bkopilov has joined #tripleo13:08
*** sshnaidm|off is now known as sshnaidm13:09
*** eck`gone is now known as eck`13:09
*** numans has quit IRC13:10
*** numans_ has joined #tripleo13:10
*** aufi_ has joined #tripleo13:13
*** aufi has quit IRC13:15
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-common master: DNM: Debugging ansible setup  https://review.openstack.org/53680513:16
*** jbadiapa has joined #tripleo13:16
*** psachin has joined #tripleo13:16
*** amoralej is now known as amoralej|lunch13:16
*** liverpooler has joined #tripleo13:19
*** numans has joined #tripleo13:19
*** numans_ has quit IRC13:19
*** numans has quit IRC13:19
*** numans has joined #tripleo13:20
*** etingof has quit IRC13:21
*** cdearborn has joined #tripleo13:21
*** ykarel is now known as ykarel|afk13:23
*** pchavva has joined #tripleo13:24
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-common master: DNM dump yaml inventory to log  https://review.openstack.org/53681013:26
*** tzumainn has joined #tripleo13:28
*** rbrady has quit IRC13:29
*** jpena|lunch is now known as jpena13:29
openstackgerritOr Idgar proposed openstack/tripleo-heat-templates master: Ensure packages for octavia only when EnablePackageInstall is True  https://review.openstack.org/53681613:32
*** ykarel|afk has quit IRC13:32
*** jcoufal has joined #tripleo13:32
*** rlandy has joined #tripleo13:34
oidgarjaosorior: thanks for being fast :)13:34
jaosorioroidgar: it was a straight forward patch :)13:35
oidgarjaosorior: lol yeah13:35
mandrehey folks, can we merge https://review.openstack.org/#/c/534326/? it bits me everytime I need to redeploy with quickstart (quite often these days)13:36
itlinuxquick suggestions needed.. I just update the UC, all good but when I try to deploy i get no valid hosts found [req-440230aa-cee5-4e36-97da-2fbe6a6a1e22] Another thread already created13:36
itlinux a resource provider with the UUID eacc6a4c-46bb-40fb-ab73-63135c8fb24f. Grabbing that record from the placement API.13:36
itlinux2018-01-23 05:32:50.637 2353 WARNING nova.scheduler.client.report [req-62c0875c-3eea-4d7e-a3b6-29f058263b9f - - - - -] Unable to refresh my resource provider record13:36
*** alee_dinner has quit IRC13:36
itlinuxany tips? TY13:37
jaosoriormandre: so, neutron fails with IPv6 addresses? or is it missing brackets?13:39
*** pgadiya has quit IRC13:40
*** rbrady has joined #tripleo13:42
sshnaidmchandankumar, hi, do you know something about python-tripleo-common-tests-tempest ? http://logs.openstack.org/95/533795/9/gate/tripleo-ci-centos-7-containers-multinode/d63ad26/job-output.txt.gz#_2018-01-22_19_27_11_82904613:44
chandankumarsshnaidm: https://review.rdoproject.org/r/#/c/11031/ it is still in review13:45
chandankumarsshnaidm: is this one used somewhere?13:45
sshnaidmchandankumar, it tried to build it in this job.. weird13:45
itlinuxI wonder if there is a way to retrieve the password when I do ironic node-show from the IPMI, we are building a script and wanted to grub that .. thanks13:45
openstackgerritCarlos Camacho proposed openstack/python-tripleoclient master: Add Openstack Undercloud backup  https://review.openstack.org/46621313:45
chandankumarsshnaidm: as the spec file is available in dist-git so failure is legimate13:46
*** dprince has joined #tripleo13:46
chandankumar*is not13:46
jaosorioritlinux: shouldn't your instackenv.json file contain the credentials?13:47
itlinuxbut we want to query the ipmi not the instackenv file..13:47
itlinuxin case someone changed it..13:48
itlinuxnever know too many people touching things13:48
chandankumarsshnaidm: do we need tripleo-common tempest plugin for downstream also?13:50
sshnaidmchandankumar, hmm.. no idea tbh, weshay|rover myoung rlandy ^^?13:51
weshay|roverI think so yes13:51
chandankumarshardy: weshay|rover here is the list of plugins https://review.rdoproject.org/etherpad/p/rhos-tempest-plugins getting imported downstream13:52
sshnaidmbnemec, trying to get extra node from te-broker, but it doesn't work, could you please take a look, what do I do wrong? https://logs.rdoproject.org/35/534435/4/openstack-experimental/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039-master/Zac75d678982d476ca37f77c9161a65b0/console.txt.gz#_2018-01-23_11_26_07_47813:53
*** maufart__ has joined #tripleo13:54
chandankumarweshay|rover: you added the list :-)13:54
chandankumarsshnaidm: in which feature set we are planning to run tripleo-common tempest plugin?13:55
*** akane__ has quit IRC13:56
sshnaidmchandankumar, I don't know yet, just don't understand why it was building it in not related patch.. but seems like we need to support its building13:57
*** trown|outtypewww is now known as trown|ruck13:57
*** aufi_ has quit IRC13:57
*** ykarel|afk has joined #tripleo13:57
chandankumarsshnaidm: https://review.rdoproject.org/r/#/c/11031/ once this patch is merged, spec file will be available, then it will build the packahe13:57
*** akane_ has joined #tripleo13:57
openstackgerritJiri Stransky proposed openstack/python-tripleoclient master: Use YAML inventory for minor update  https://review.openstack.org/53683613:58
*** noslzzp has joined #tripleo13:58
*** Goneri has joined #tripleo13:58
EmilienMthe tripleo meeting is about to start shortly, you can propose any topic here: https://etherpad.openstack.org/p/tripleo-meeting-items13:58
*** etingof has joined #tripleo13:59
sshnaidmchandankumar, great14:00
EmilienM#startmeeting tripleo14:00
openstackMeeting started Tue Jan 23 14:00:32 2018 UTC and is due to finish in 60 minutes.  The chair is EmilienM. Information about MeetBot at http://wiki.debian.org/MeetBot.14:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
*** openstack changes topic to " (Meeting topic: tripleo)"14:00
openstackThe meeting name has been set to 'tripleo'14:00
EmilienMhello everyone14:00
dpeacocko/ mornin'14:01
EmilienM#topic agenda14:01
openstackgerritOr Idgar proposed openstack/tripleo-heat-templates master: Ensure packages for octavia only when EnablePackageInstall is True  https://review.openstack.org/53681614:02
*** openstack changes topic to "agenda (Meeting topic: tripleo)"14:02
EmilienM* Review past action items14:02
EmilienM* One off agenda items14:02
EmilienM* Squad status14:02
EmilienM* Bugs & Blueprints14:02
EmilienM* Projects releases or stable backports14:02
EmilienM* Specs14:02
EmilienM* Open discussion14:02
EmilienMAnyone can use the #link, #action and #info commands, not just the moderator!14:02
EmilienMalright let's start14:02
*** atoth has quit IRC14:02
EmilienM#topic review past action items14:02
*** openstack changes topic to "review past action items (Meeting topic: tripleo)"14:02
EmilienMEmilienM to send notes about bug policy & also storyboard evaluation14:02
EmilienMthis one isn't done yet, I'll send it before PTG and propose a session eventually14:03
openstackgerritDerek Higgins proposed openstack/tripleo-heat-templates master: WIP: Add ironic config to scenario 004  https://review.openstack.org/48526114:03
EmilienMEmilienM to contact individual specs owners to move them to Rocky: done14:03
EmilienMI sent 2 notes (one 7 days ago and one last night)14:04
EmilienM#topic one off agenda items14:04
*** openstack changes topic to "one off agenda items (Meeting topic: tripleo)"14:04
EmilienMsshnaidm: hey14:04
sshnaidmEmilienM, yeah, so we need to merge all these patches disabling rh1 jobs14:04
sshnaidmRH1 cloud is going to maintenance14:05
sshnaidmand we will use RDO 3d party CI for OVB jobs14:05
*** links has quit IRC14:05
sshnaidmcores +w is required: https://review.openstack.org/#/q/topic:bug/174476314:05
sshnaidmthat's it14:05
EmilienM#action team to review https://review.openstack.org/#/q/topic:bug/1744763 so we can move forward with RH114:06
EmilienMsshnaidm: thanks14:06
EmilienM#link https://etherpad.openstack.org/p/tripleo-meeting-items14:06
EmilienMhamzy: go ahead14:06
hamzyI've added two patches for supporting multiple architectures in TripleO14:07
hamzywould like people's opinions14:07
hamzyhttps://review.openstack.org/#/c/528000/ and https://review.openstack.org/#/c/528060/14:07
EmilienM#action team to review multiple arch patches https://review.openstack.org/#/c/528000/ and https://review.openstack.org/#/c/528060/14:08
EmilienMhamzy: anything else?14:08
hamzyno... thanks14:08
EmilienMok, so my turn14:08
EmilienMjust wanted to share again https://etherpad.openstack.org/p/tripleo-ptg-rocky14:08
EmilienM#link Rocky PTG https://etherpad.openstack.org/p/tripleo-ptg-rocky14:09
EmilienMI sent a note on the ML last week, it would be great to get some inputs on topics we could discuss during PTG14:09
EmilienMthink about things we couldn't finish this cycle or things we want to do next14:09
*** psachin has quit IRC14:09
EmilienMPTG is a good time to propose blueprints, specs, ideas, changes, and do some planning as well14:10
EmilienMthe schedule will be the usual14:10
EmilienMbetween Wednesday and Friday14:10
EmilienMif any question about it, just ping me14:10
EmilienMlet's move on14:11
EmilienM#topic Squad status14:11
*** openstack changes topic to "Squad status (Meeting topic: tripleo)"14:11
EmilienM#link https://etherpad.openstack.org/p/tripleo-ci-squad-meeting14:11
EmilienM#link https://etherpad.openstack.org/p/tripleo-upgrade-squad-status14:11
EmilienM#link https://etherpad.openstack.org/p/tripleo-containers-squad-status14:11
EmilienM#link https://etherpad.openstack.org/p/tripleo-integration-squad-status14:11
EmilienM#link https://etherpad.openstack.org/p/tripleo-ui-cli-squad-status14:11
EmilienM#link https://etherpad.openstack.org/p/tripleo-validations-squad-status14:11
EmilienM#link https://etherpad.openstack.org/p/tripleo-networking-squad-status14:11
EmilienM#link https://etherpad.openstack.org/p/tripleo-workflows-squad-status14:11
EmilienMI'll take care of updating https://etherpad.openstack.org/p/tripleo-containers-squad-status today14:12
EmilienMunless mandre wants to do it ;-)14:12
*** melwitt has quit IRC14:12
EmilienMor dprince ^14:12
EmilienMbut otherwise I'll take a stab14:12
dpeacockEmilienM: Can that simply be retired at this point?14:13
EmilienMgchamoul, akrivoka: any chance to get an update on https://etherpad.openstack.org/p/tripleo-validations-squad-status ?14:13
EmilienMdpeacock: indeed, it's something we can discuss14:13
EmilienMdpeacock: or rename the squad, not sure. I'll propose something to the ML probablyu14:13
EmilienMthrash, rbrady : any chance to get updates on https://etherpad.openstack.org/p/tripleo-workflows-squad-status ?14:14
*** melwitt has joined #tripleo14:14
gchamoulEmilienM: sure thing14:14
mandreI still think a separate squad for containers make sense, especially considering the goal running the undercloud services in containers14:14
rbradyEmilienM: ack14:15
EmilienMmandre: true, and all the ongoing work around k8s & openshift14:15
EmilienMrbrady, gchamoul: thx, I'll wait for your notes today before sending the Weekly Owl14:15
EmilienMok moving on14:15
EmilienM#topic bugs & blueprints14:15
*** openstack changes topic to "bugs & blueprints (Meeting topic: tripleo)"14:15
*** lucasagomes is now known as lucas-hungry14:15
EmilienMmandre, dpeacock: we can re-discuss it offline btw.14:16
EmilienMdo we have outstanding bugs or blueprints that we want to discuss this week?14:16
EmilienMI don't see any alert14:16
*** lvdombrkr89 has joined #tripleo14:16
*** lvdombrkr has quit IRC14:16
EmilienMweshay|rover, arxcruz : https://bugs.launchpad.net/tripleo/+bug/1742936 isn't close TBH14:17
openstackLaunchpad bug 1742936 in tripleo "TestBootVolumePattern is failing in multinode jobs" [Critical,Fix released] - Assigned to Arx Cruz (arxcruz)14:17
EmilienMit's technical debt14:17
*** moshele has quit IRC14:17
*** etingof has quit IRC14:17
arxcruzEmilienM: well, the issue was the lack of resources in the machine, how shall I close this ?14:17
EmilienMI retriaged it:14:17
EmilienMtags: added: tech14:17
EmilienMremoved: alert promotion-blocker14:17
EmilienMChanged in tripleo:14:17
EmilienMstatus: Fix Released → Triaged14:18
EmilienMimportance: Critical → High14:18
EmilienMit's a regression for sure14:18
EmilienMany other bug we need to discuss this week?14:18
arxcruzEmilienM: nope14:18
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: ffu: Introduce Cinder fast-forward upgrade tasks  https://review.openstack.org/52154514:18
EmilienMmoving on14:19
EmilienM#topic projects releases or stable backports14:19
*** openstack changes topic to "projects releases or stable backports (Meeting topic: tripleo)"14:19
*** jmelvin has joined #tripleo14:19
EmilienMas soon as we have a promotion I'll push the queens-milestone3 tag14:20
EmilienMwe won't have a branch before the latest moment, so no worries14:20
EmilienMwe'll have RCs anyway14:20
EmilienMbut before Thursday we'll have m3 tag14:20
EmilienM#action EmilienM to release tripleo queens milestone 3 this week14:21
*** alee_dinner has joined #tripleo14:21
EmilienM#action EmilienM to release tripleo stable newton/ocata/pike this week14:21
EmilienMany question about release management?14:21
EmilienM#link https://releases.openstack.org/queens/schedule.html14:22
EmilienMgfidente: which reminds me we need to reply to your FFE14:22
gfidentetbarron ^^14:22
EmilienMgfidente: will do today14:23
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-common master: DNM: Debugging ansible setup  https://review.openstack.org/53680514:23
EmilienM#action EmilienM to reply to gfidente's FFE14:23
EmilienM#topic specs14:23
*** openstack changes topic to "specs (Meeting topic: tripleo)"14:23
tbarrongfidente: ty14:23
EmilienM#link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open14:23
EmilienMso I'll repeat what I said earlier14:25
EmilienMI sent 2 emails to specs owners, asking to move the spec to Rocky14:25
EmilienMif it's not done by the PTG (in 1 month), I'll abandon the spec14:25
EmilienMtrying to move things that sometimes stale (bug reports, patches, specs, etc)14:25
EmilienM#topic open discussion14:26
*** openstack changes topic to "open discussion (Meeting topic: tripleo)"14:26
EmilienManything else for this week?14:26
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-specs master: Move logging-stdout spec to Rocky  https://review.openstack.org/53684614:26
EmilienMalright, have a great week everyone, see you online14:27
*** openstack changes topic to "CI status: VERY GREEN | http://tripleo.org/ | https://docs.openstack.org/tripleo-docs/latest/"14:27
openstackMeeting ended Tue Jan 23 14:27:23 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:27
openstackMinutes:        http://eavesdrop.openstack.org/meetings/tripleo/2018/tripleo.2018-01-23-14.00.html14:27
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/tripleo/2018/tripleo.2018-01-23-14.00.txt14:27
openstackLog:            http://eavesdrop.openstack.org/meetings/tripleo/2018/tripleo.2018-01-23-14.00.log.html14:27
rajinirEmilienM: Is there another core to merge this one? https://review.openstack.org/#/c/532638/14:27
fultonjrajinir: yes14:28
jaosoriorshardy: hey, so, I reproduced the error by using updating all my packages and using the --config-download flag.14:28
fultonjrajinir: done14:28
rajinirfultonj>  Great thanks14:28
jaosoriorshardy: after it failed, I copied the inventory to the stack user's directory, and ran it directly with ansible... and it worked...14:28
rbradyEmilienM: we canceled the squad meeting last week because of low turnout.  The squad should be continuing to brainstorm on topics from 10 Jan.14:28
jaosoriorshardy: so there is something funky going on with the way the workflow is ran14:28
EmilienMrbrady: ok so no updates this week?14:29
*** myoung is now known as myoung|taxi14:29
rbradyEmilienM: nope14:29
*** noslzzp has quit IRC14:30
shardyjaosorior: intersting, so I think --config-download will use a different user, as it's run via mistral14:30
EmilienMrbrady: ack14:30
jaosoriorshardy: right, I had to change the user when I copied the inventory14:30
shardyjaosorior: I think slagle has the most knowledge in this area, so you may want to chat with him about it14:30
jaosoriorshardy: thanks, will do14:31
jaosoriorslagle: hey, I'm having some weird issues with --config-download that I've beens struggling with for several days and I'm quite stumped :/, got some time to help me out?14:31
slaglejaosorior: ok14:31
EmilienMsame here14:33
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart master: Add Featureset 39 -- HA deploy w/ IPA  https://review.openstack.org/53378514:33
jaosoriorslagle: so, I've been trying to get the IPSEC service to be tested in CI https://review.openstack.org/#/c/535171/  https://review.openstack.org/#/c/534731/   But it's failing in the config-download workflow with something like this14:33
jaosoriorhttp://logs.openstack.org/34/535234/1/check-tripleo/tripleo-ci-centos-7-ovb-ha-oooq/bc6da56/logs/undercloud/var/lib/mistral/845008d8-840e-4721-85ee-2b5876ed8eb5/ansible.log.txt.gz#_2018-01-19_11_04_36_752   (note that I got that output by enabling verbosity)14:33
EmilienMjaosorior: is it the same problem as me?14:34
jaosoriorEmilienM: no14:34
jaosoriorslagle: That exception seems to happen every time I try to access hostvars14:34
shardyEmilienM: No I think this one is different, it's not related to HostnameMap, the hostvars are broken when run via the mistral user14:34
jaosoriorand it only happens when I use the workflow14:34
jaosoriorif I run it separately (overcloud deploy ... config-download ... ansible-playbook), it works14:34
jaosoriorEmilienM: it's happening in ovb-ha, which doesn't get the HostnameMap14:35
slaglei don't think we should add ipsec to 00114:35
slaglethe job is already slow enough14:35
slaglei'll look at the error though :)14:36
*** mmethot_ has quit IRC14:36
openstackgerritBrent Eagles proposed openstack/tripleo-quickstart master: Add fs038 for Octavia scenario testing  https://review.openstack.org/53116814:36
*** mmethot has joined #tripleo14:36
EmilienMyeah, strong -1 to add anything to OVB14:36
jaosoriorEmilienM: any other suggestions on where to test it?14:37
slaglejaosorior: ok, i saw something similar in some of my testing. i had to switch some escaped double qoutes to single14:37
slaglei got distracted and never submitted the patch14:37
EmilienMyes, create a separated job (multinode maybe?)14:37
slaglelet me do so now and see if it helps at all14:37
EmilienMthat only runs when you touch ipsec services probably14:38
jaosoriorEmilienM: I need network isolation though14:38
jaosoriorslagle: so this is where it breaks https://github.com/JAORMX/tripleo-ipsec/blob/master/tasks/ipsec-conf.yml#L19 , what part should have single quotes?14:39
EmilienMjaosorior: ok, then create another OVB job or re-use another one that exists14:39
*** agopi__ has joined #tripleo14:39
EmilienMbut we shouldn't add things to OVB imho14:39
jaosoriorEmilienM: so... we got to this point when I discussed with with mwhahaha initially, this is why I ended up proposing it for 00114:40
EmilienMjaosorior: keep in mind this job can be run on any gate14:40
EmilienMjaosorior: if you need OVB + netiso, create a new OVB job (with 2 nodes maybe, 1 con, 1 com) and activate IPsec14:41
EmilienMand run it only when you need thanks to zuul layout14:41
jaosoriorEmilienM: ok, once we get this fixed, could you help me out with that? I'm not very acquainted with creating jobs14:41
EmilienMjaosorior: of course I'll help14:42
EmilienMjaosorior: there is a whole CI team willing to help as well14:42
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates master: Use single quotes in deploy-steps.j2  https://review.openstack.org/53684914:43
slaglejaosorior: it might not be there. when I saw it, it was because of that usage ^14:45
slaglejaosorior: since the HostPrepConfig deployment is set as a hostvar via the group_vars mechanism that config-download uses14:45
slaglethen if you tried to access hostvars in any other task, you got the TemplateSyntaxError14:45
ansiwenshardy: who else could approve https://review.openstack.org/#/c/532465/, since you and skramaja are co-authors. I'm happy to bug someone else who might be a good candidate. EmilienM already gave +2) As you can see, I'm *very* interested, that this gets merged very soon ;-)14:46
ansiwenjaosorior: ^^^ you maybe?14:46
*** numans has quit IRC14:47
*** agopi__ has quit IRC14:47
*** fragatina has joined #tripleo14:47
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Add IPSEC service to ovb-ha environment  https://review.openstack.org/53517114:48
slaglejaosorior: try depending on that patch and see if it helps at all.14:48
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Add IPSEC service to ovb-ha environment  https://review.openstack.org/53517114:48
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-quickstart master: Activate ipsec for ovb-ha  https://review.openstack.org/53473114:48
jaosoriorslagle: did so, thanks!14:48
ansiwenEmilienM: shardy also, if you could have a quick look on this simple bug-fix I'm depending on, it would be also great: https://review.openstack.org/#/c/535428/ :-)14:49
*** ykarel|afk has quit IRC14:50
openstackgerritLukas Bezdicka proposed openstack/tripleo-heat-templates master: Always evaluate step first in conditional  https://review.openstack.org/53685114:52
*** numans has joined #tripleo14:52
*** ykarel|afk has joined #tripleo14:53
ansiwenjaosorior: awesome, thanks a lot!14:53
*** noslzzp has joined #tripleo14:53
openstackgerritTim Flink proposed openstack/diskimage-builder master: Don't install dmidecode on Fedora ppc64le  https://review.openstack.org/53685214:54
*** amoralej|lunch is now known as amoralej14:55
*** etingof has joined #tripleo14:55
*** yolanda has quit IRC14:56
*** waleedm has quit IRC14:56
weshay|roveralee_dinner, you are still eating dinner :)?14:56
*** noslzzp_ has joined #tripleo14:56
weshay|roveralee_dinner, https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/b7194c8/tempest.html.gz14:56
*** jfrancoa has quit IRC14:57
itlinuxhello I see an screen in the console which says no proxy offer.. as I am builing the OC14:57
itlinuxproxyDHCP no offer..14:58
*** noslzzp has quit IRC14:59
EmilienMSlower, bogdando: what's the current status on containerized undercloud CI job?15:01
*** jfrancoa has joined #tripleo15:02
*** pblaho has quit IRC15:02
*** anilvenkata has joined #tripleo15:02
*** thrash is now known as thrash|biab15:02
shardyEmilienM: tripleo-ci-centos-7-undercloud-containers seems quite stable to me, IMO we should make it voting15:03
jankimarios, hey15:03
*** dsariel has quit IRC15:03
EmilienMshardy: is it using the correct workflow, etc?15:03
EmilienMshardy: can we deploy an overcloud with the result?15:03
shardyEmilienM: that is not tested in CI atm, it works for me locally though15:04
shardymaybe we should switch the containers-multinode job to containerized undercloud to prove it15:05
shardythat can be the container all-the-things job15:05
shardyHave we decided to formally deprecate baremetal support for Queens?15:05
*** akane_ has quit IRC15:05
EmilienMthrash|biab, rbrady : these mistral errors should be fixed: http://logs.openstack.org/60/534360/9/check/tripleo-ci-centos-7-undercloud-containers/e96445a/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-01-19_17_32_2315:05
bogdandolink: https://review.openstack.org/#/q/status:open+topic:bp/containerized-undercloud15:05
*** dpawar has quit IRC15:05
EmilienMshardy: yeah, we don't support any baremetal thing on the overcloud anymore15:06
EmilienMthere is no job testing on baremetal at this time15:06
EmilienM(in master)15:06
bogdandodo you mean non containerized services?15:06
holserslagle - There are couple of pull requiests in traas... Could you be so kind to have a look?15:06
*** mdnadeem has joined #tripleo15:06
*** atoth has joined #tripleo15:06
EmilienMshardy: I wish someone could have sent an update on this topic15:07
EmilienMshardy: someone who is working on it, like Slower or weshay|rover I know they did some work recently15:08
EmilienMbut I can take a look if nobody else does15:08
EmilienMshardy: we can probably make that change into Rocky15:08
weshay|roverEmilienM, ya.. we need to get reorganized on that work15:09
shardyEmilienM: FWIW I did put a topic about this in the ptg etherpad, that's probably a good time to discuss and prioritize the remaining work?15:09
slagleholser: yes, i will look today15:09
EmilienMshardy: exactly15:10
holserslagle - the problem is ---- too much forks15:10
holserI would like to make your repo official and make gerrithub.io on top15:10
EmilienMarxcruz: quick question: could we easily re-use validate-tempest quickstart role to run against the undercloud?15:10
*** shardy is now known as shardy_afk15:10
holserbogdando, chem, my fork :D we are circulating patches around15:11
EmilienMarxcruz: to me, the role only works for an overcloud now15:11
trown|ruckweshay|rover: the previous run of 035 passed so it might be some transient issue... but I made https://bugs.launchpad.net/tripleo/+bug/1744954 in any case15:11
openstackLaunchpad bug 1744954 in tripleo "[promote] scenario035 failing during mysql_bootstrap container start" [Critical,Triaged]15:11
rbradyEmilienM: looks like version mismatch with openstack clients vs mistral15:12
EmilienMrbrady: could we file a bug and fix that please?15:12
*** nyechiel has quit IRC15:12
EmilienMarxcruz: I think I know how, I'll file a blueprint and assign to myself, so we can run tempest on undercloud15:13
arxcruzEmilienM: I think so... are we going to start test undercloud ?15:13
EmilienMarxcruz: yes.15:13
*** janki has quit IRC15:14
*** numans has quit IRC15:17
EmilienMarxcruz: https://blueprints.launchpad.net/tripleo/+spec/undercloud-tempest15:17
*** numans has joined #tripleo15:18
arxcruzEmilienM: I think instead of using overcloudrc, just use stackrc will make tempest run on undercloud15:18
*** moshele has joined #tripleo15:18
openstackgerritSven Anderson proposed openstack/tripleo-heat-templates master: Add custom role for realtime compute  https://review.openstack.org/53173915:18
EmilienMbogdando, weshay|rover, dprince, shardy_afk ^ fyi - I think a plan could be to make the role working on the undercloud and eventually switch current multinode jobs to deploy containerized undercloud & execute tempest15:18
EmilienMarxcruz: yeah but I think there is a little bit more15:18
EmilienMarxcruz: not much but still15:18
*** itlinux has quit IRC15:19
openstackgerritChristian Schwede proposed openstack/tripleo-heat-templates master: Enable Ansible-based Swift ring deployment by default  https://review.openstack.org/53323515:20
bogdandoEmilienM: yes, that's a great idea15:20
*** links has joined #tripleo15:20
*** lucas-hungry is now known as lucasagomes15:20
arxcruzchandankumar: ^15:20
trown|ruckadarazs: sshnaidm could I get a review on https://review.openstack.org/#/c/536366/ <-- will fix gate failures15:21
*** agopi has joined #tripleo15:21
adarazstrown|ruck: I'll take a look.15:21
EmilienMbogdando: will need some help, we can pair maybe15:21
bogdandoEmilienM: yeah15:22
*** oidgar has quit IRC15:23
openstackgerritDaniel Alvarez proposed openstack/tripleo-heat-templates stable/pike: Add trunk service plugin to OVN environment files  https://review.openstack.org/53685915:24
pabelangersshnaidm: weshay|rover: EmilienM: morning, how does migration of OVB to RDOcloud look this morning?15:24
*** moshele has quit IRC15:24
EmilienMbogdando, shardy_afk, weshay|rover : I found out that we still have some ongoing tasks for the containerized undercloud: https://review.openstack.org/#/q/topic:bp/containerized-undercloud15:25
EmilienMbogdando, shardy_afk, weshay|rover : are they blocking now?15:25
EmilienMto me https://review.openstack.org/#/c/517444/ is critical here15:25
weshay|roverpabelanger, lots of patches to merge15:25
sshnaidmpabelanger, waiting for +w on https://review.openstack.org/#/q/topic:bug/174476315:25
weshay|roverpabelanger, https://review.openstack.org/#/q/topic:bug/1744763+(status:open+OR+status:merged)15:26
openstackgerritMarcin Juszkiewicz proposed openstack/diskimage-builder master: GPT partitioning support  https://review.openstack.org/53349015:26
openstackgerritMarcin Juszkiewicz proposed openstack/diskimage-builder master: bootloader: handle grub-efi better  https://review.openstack.org/53660015:26
openstackgerritMarcin Juszkiewicz proposed openstack/diskimage-builder master: change default image into GPT with ESP and BSP  https://review.openstack.org/53660115:26
EmilienMweshay|rover: why don't you approve it?15:27
EmilienMweshay|rover: it's CI related15:27
sshnaidmEmilienM, can we just merge them?15:27
weshay|roverok.. I'll rock it out15:27
EmilienMbut wes could have done it15:27
weshay|roverhe's a f.. slacker15:27
*** aditya_r has joined #tripleo15:28
EmilienMweshay|rover: yeah and plz review the backports too15:29
openstackgerritMarcin Juszkiewicz proposed openstack/diskimage-builder master: GPT partitioning support  https://review.openstack.org/53349015:29
openstackgerritMarcin Juszkiewicz proposed openstack/diskimage-builder master: bootloader: handle grub-efi better  https://review.openstack.org/53660015:29
openstackgerritMarcin Juszkiewicz proposed openstack/diskimage-builder master: change default image into GPT with ESP and BSP  https://review.openstack.org/53660115:29
*** wojdec has joined #tripleo15:30
*** jfrancoa has quit IRC15:30
weshay|roverwho is core on dib-utils? https://review.openstack.org/#/c/536499/15:30
sshnaidmweshay|rover, https://review.openstack.org/#/admin/groups/115,members15:31
sshnaidmweshay|rover, I think ianw can help15:31
*** wojdec1 has joined #tripleo15:32
dprinceEmilienM: getting the undercloud-containers job in ci (anywhere) was our current plan. Adding tempest is fine too15:33
*** thrash|biab is now known as thrash15:34
EmilienMweshay|rover: ianw15:34
*** aditya_r has quit IRC15:34
*** wojdec has quit IRC15:34
thrashEmilienM: I'll take a look. Do we have an lp for that?15:35
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates stable/pike: Fixes generating public certificates  https://review.openstack.org/53686515:35
pabelangerweshay|rover: +3 dib-utils15:36
weshay|roverthanks.. saved me an email15:36
*** aditya_r has joined #tripleo15:36
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates stable/pike: Move parameter to parameter_defaults section  https://review.openstack.org/53686715:36
weshay|roversshnaidm, ok.. we have them all approved ya? https://review.openstack.org/#/q/topic:bug/1744763+(status:open+OR+status:merged)15:37
sshnaidmweshay|rover, yep15:37
*** rmascena has joined #tripleo15:38
EmilienMthrash: no15:39
thrashEmilienM: I'll create one.15:39
*** raildo has quit IRC15:40
*** rmascena is now known as raildo15:41
*** paramite has quit IRC15:41
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: split multinode playbook  https://review.openstack.org/52641415:42
pabelangerweshay|rover: sshnaidm: EmilienM: Thanks! I'll keep an eye on nodepool / zuul pipelines this morning too15:43
EmilienMthrash: thx15:44
weshay|roverpabelanger, thanks15:45
*** nyechiel has joined #tripleo15:45
*** jfrancoa has joined #tripleo15:46
*** shardy_afk is now known as shardy15:46
*** myoung|taxi is now known as myoung15:47
rfolco|roverdpeacock, I'm looking at https://bugs.launchpad.net/tripleo/+bug/1739620... If I understand it right, oooq deployments are not logging swift, undercloud does not have var/log/swift. So a patch is required to oooq just like you did in https://review.rdoproject.org/r/#/c/11010/4 ?15:48
openstackLaunchpad bug 1739620 in tripleo "quickstart covers up deficiency in swift logging in standard deployment" [Low,Triaged] - Assigned to David Peacock (davidjpeacock)15:48
dpeacockrfolco|rover: hey - yeah basically it was a reminder ticket for myself to look at what caused this cover up in tripleo and whether or not it was incidental15:50
dpeacocktripleo-quickstart, I meant15:50
rfolco|roverdpeacock, I took a look at several logs.... and none had var/log/swift under undercloud logs dir.... eg http://logs.openstack.org/12/535612/1/check/tripleo-ci-centos-7-scenario002-multinode-oooq/fc8772e/logs/undercloud/var/log/15:52
*** pblaho has joined #tripleo15:52
rfolco|roverdpeacock, I am not saying I have already the fix on top of my head, but I can help15:53
*** alee_dinner is now known as alee15:53
dpeacockrfolco|rover: I'd be glad of any help :-)15:53
openstackgerritDavid Vallee Delisle proposed openstack/tripleo-image-elements master: Try to get a METADATA_IP_PING_TIMEOUT and adding -W to the ping command  https://review.openstack.org/53604115:54
rfolco|roverdpeacock, ok cool ;)15:54
dpeacockrfolco|rover: thanks15:56
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras  https://review.openstack.org/47260715:58
*** ykarel|afk is now known as ykarel15:58
*** openstackgerrit has quit IRC16:03
*** aditya_r has quit IRC16:04
*** etingof has quit IRC16:04
*** janki has joined #tripleo16:05
*** lvdombrkr89 has quit IRC16:07
*** openstackgerrit has joined #tripleo16:09
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci master: Wire in extra_nodes as a job param  https://review.openstack.org/53687816:09
bnemecsshnaidm: ^16:09
bnemecI missed a link in the chain when passing --extra-nodes through to the testenv-worker.16:09
thrashEmilienM: Just so you are aware, not all of those will go away. Some will remain because tripleo doesn't install the client (glare for instance)16:09
thrashEmilienM: some were deprecated and have been removed from the client, like the nova stuff.16:09
sshnaidmbnemec, cool, thanks, was searching for this parameter too..16:10
dtantsurcan someone please approve this backport https://review.openstack.org/#/c/534328/ ? the CI is green16:10
*** trown|ruck is now known as trown|brb16:11
*** dparkes has quit IRC16:14
*** gvrangan has joined #tripleo16:15
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Include extra nodes in te-broker config  https://review.openstack.org/53674216:15
openstackgerritOliver Walsh proposed openstack/puppet-tripleo master: WIP: add support for libvirt VNC TLS with dedicated CA  https://review.openstack.org/53640416:16
*** trown|brb is now known as trown|ruck16:17
fultonjdtantsur: LGTM16:19
*** etingof has joined #tripleo16:21
*** oidgar has joined #tripleo16:21
*** tesseract has quit IRC16:22
EmilienMthrash: ok, why not deploying the client then?16:22
thrashEmilienM: Beats me. :D16:22
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates master: Restore disable templates for telemetry for Queens  https://review.openstack.org/53582016:22
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates master: WIP: Add fast_forward_upgrade_tasks for ceilometer services  https://review.openstack.org/53688816:22
honzaEmilienM: any ideas why this isn't merging? https://review.openstack.org/#/c/512266/16:23
thrashEmilienM: my guess is it was never added to the base image.16:23
EmilienMhonza: depends-on didn't merge16:23
honzaEmilienM: ah, thanks!16:23
* honza learns new stuff every day16:24
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates master: WIP: add support for libvirt VNC TLS with dedicated CA  https://review.openstack.org/53646916:24
thrashEmilienM: so far, I see glareclient, tackerclient16:24
thrashEmilienM: there might be one more.16:24
thrashEmilienM: Oh, and keystone.oauth1... keystone.oauth1.request_tokens_build_url: NotImplementedError: To use 'request_tokens' oauthlib must be installed"16:25
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates master: WIP: Add fast_forward_upgrade_tasks for ceilometer services  https://review.openstack.org/53372916:25
*** ramishra has quit IRC16:25
thrashEmilienM: this patch should remove quite a bit of those errors though. https://review.openstack.org/#/c/536886/16:26
EmilienMthrash: cool; thx16:26
thrashEmilienM: I'll take a look at the others separately.16:27
EmilienMthrash: merci16:27
thrashEmilienM: de rein16:27
EmilienMthrash: "de rien"16:30
thrashEmilienM: dammit google16:30
thrashI thought that looked wrong.16:30
thrashMy french is VERY rusty.16:30
*** salmankhan has quit IRC16:31
fultonjwould someone please review https://review.openstack.org/#/c/536359 ? it adds comments that we plan to deprecate some ceph env files in rocky16:32
*** jaganathan has quit IRC16:32
*** gvrangan has quit IRC16:34
EmilienMfultonj: a release note would have been great, maybe in a separated patch?16:35
EmilienMgfidente: ^16:35
EmilienMI approved it anyway16:35
gfidenteEmilienM ah right16:35
gfidenteEmilienM I'll add it16:36
gfidentefultonj thanks as well16:36
gfidentesorry guys :(16:36
EmilienMnp :)16:36
*** salmankhan has joined #tripleo16:36
fultonjgfidente: thanks16:36
gfidenteEmilienM not sure if you remove w+116:37
gfidentecan I still update it?16:37
EmilienMgfidente: you can send a separated patch on top of this one16:37
EmilienMgfidente: no need to waste CI jobs16:37
*** moshele has joined #tripleo16:37
*** jaganathan has joined #tripleo16:39
*** moshele has quit IRC16:41
*** pcaruana has quit IRC16:41
thrashEmilienM: https://bugs.launchpad.net/tripleo/+bug/174497216:42
openstackLaunchpad bug 1744972 in tripleo "Clients need to be added to rid of error messages" [Medium,Triaged] - Assigned to Brad P. Crochet (brad-9)16:42
*** dhill_ has joined #tripleo16:43
openstackgerritAde Lee proposed openstack/puppet-tripleo master: Add variable for keystone topics queue for barbican  https://review.openstack.org/53689616:44
*** gvrangan has joined #tripleo16:46
EmilienMthrash: ack16:46
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: ffu: Introduce Swift fast-forward upgrade tasks  https://review.openstack.org/52664216:47
*** udesale_ has quit IRC16:47
*** marrusl has quit IRC16:49
openstackgerritAde Lee proposed openstack/tripleo-heat-templates master: Add barbican keystone notification topics  https://review.openstack.org/53690016:52
*** jaganathan has quit IRC16:52
*** maufart__ has quit IRC16:54
openstackgerritKeith Schincke proposed openstack/tripleo-heat-templates master: Add rgw_keystone_implicit_tenants to ceph-ansible/ceph-base.yaml  https://review.openstack.org/53690116:55
*** mmethot is now known as mmethot|away16:55
*** mmethot|away is now known as mmethot|brb16:55
*** gvrangan has quit IRC16:56
*** jfrancoa has quit IRC16:56
*** jbadiapa has quit IRC16:56
*** iranzo has quit IRC16:56
openstackgerritMarios Andreou proposed openstack/tripleo-common master: Remove step_tags_to_when from config download and fix sorting  https://review.openstack.org/52936916:57
*** jkilpatr_ has joined #tripleo16:58
weshay|roveralee, have you seen the fresh new error on volumeencryption?16:58
jankiHi. I am trying to install overlcoud via quickstart and am getting permission error to virt-customise overcloud-full.qcow2 image16:58
janki'/home/stack/overcloud-full.qcow2' (as uid:107, gid:107): Permission denied16:59
janki[code=38 int1=13]16:59
*** dparkes has joined #tripleo16:59
aleeweshay|rover, yeah - I just saw there was one .. looking at it16:59
*** jkilpatr has quit IRC17:00
weshay|roveralee, I took down the environment you were working on btw.. it's too old at this point17:00
weshay|roverwe need a new one17:01
*** oidgar has quit IRC17:01
aleeweshay|rover, I noticed that too when my tmux died :)17:01
aleeweshay|rover, EmilienM this config looks weird -- https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/b7194c8/subnode-2/var/log/config-data/barbican/etc/barbican/barbican.conf.txt.gz17:01
aleeweshay|rover, EmilienM look at the barbican db connection setting at the end of the file17:02
aleeis that right?17:02
*** suuuper has quit IRC17:04
aleeit could be -- just confirming17:04
*** arxcruz is now known as arxcruz|brb17:06
weshay|roveralee, worked here http://logs.openstack.org/67/536467/2/check/tripleo-ci-centos-7-scenario002-multinode-oooq-container/553dffc/logs/subnode-2/var/log/config-data/barbican/etc/barbican/barbican.conf.txt.gz17:07
aleeEmilienM, weshay|rover I do see db operations happening so its probably right17:07
weshay|rovertrown|ruck, need anything from me?17:07
openstackgerritBrad P. Crochet proposed openstack/instack-undercloud master: Add oauthlib, tackerclient, and glareclient  https://review.openstack.org/53690517:09
*** jaganathan has joined #tripleo17:10
*** gvrangan has joined #tripleo17:10
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Add release notes for the migration of Ceph env files to containers  https://review.openstack.org/53690617:11
*** jkilpatr_ has quit IRC17:12
apevecweshay|rover, alee - latest barbican error is not the same what was initially reported in https://bugs.launchpad.net/tripleo/+bug/174415117:13
openstackLaunchpad bug 1744151 in tripleo "barbican_tempest_plugin.tests.scenario.test_volume_encryption.VolumeEncryptionTest fails on Invalid Volume" [Critical,In progress] - Assigned to Arx Cruz (arxcruz)17:13
weshay|roverapevec, correct17:13
weshay|rovernew bug?17:13
apevecup to alee17:14
*** moshele has joined #tripleo17:14
*** mmethot|brb is now known as mmethot17:14
*** gvrangan has quit IRC17:15
aleeapevec, weshay|rover its definitely a new bug.  looking at it -- this is just in the promotion CI, right?17:16
weshay|roveralee, ya.. which means some recent merge changed the behavior17:17
*** yolanda has joined #tripleo17:17
aleeweshay|rover, is there a list of recent merges anywhewre?17:17
*** gvrangan has joined #tripleo17:18
openstackgerritJanki Chhatbar proposed openstack/tripleo-heat-templates master: Minor update steps for ODL  https://review.openstack.org/53690717:20
*** gvrangan has quit IRC17:20
aleeweshay|rover, ok - there is a big change that came in yesterday17:20
aleeweshay|rover, do you have an environment I can look at?17:21
weshay|roverI dont atm17:21
weshay|roverwant me to try another to get another going?17:21
aleeweshay|rover, yeah - I think I may need it17:22
aleeweshay|rover, in the meantime , I'll keep looking at it17:22
aleeapevec, its definitley a new bug17:22
aleeapevec, (this one likely a barbican one)17:23
*** panda is now known as panda|off17:26
openstackgerritJanki Chhatbar proposed openstack/tripleo-heat-templates master: Add endpoint for ODL  https://review.openstack.org/53578317:30
*** apevec_ has joined #tripleo17:31
*** apevec_ has quit IRC17:31
*** apevec_ has joined #tripleo17:31
*** dparkes has quit IRC17:32
*** apevec has quit IRC17:33
openstackgerritLokesh Jain proposed openstack/tripleo-heat-templates master: Make heat parameter plugin_dirs configurable  https://review.openstack.org/53691017:33
*** bogdando has quit IRC17:34
*** cshastri has quit IRC17:36
weshay|roveralee, ok.. hitting some issues in rdo-cloud for repro but I'll let you know when I have something17:36
aleeweshay|rover, ok thanks17:37
*** ebarrera has quit IRC17:37
openstackgerritMarc Methot proposed openstack/tripleo-heat-templates master: Add nfs as a cinder backup driver option to CinderBackupBackend  https://review.openstack.org/53577617:37
*** jpich has quit IRC17:39
*** rbowen has quit IRC17:40
openstackgerritDerek Higgins proposed openstack/tripleo-heat-templates master: WIP: Add ironic config to scenario 004  https://review.openstack.org/48526117:45
*** gvrangan has joined #tripleo17:45
*** yolanda has quit IRC17:47
*** trown|ruck is now known as trown|lunch17:48
*** etingof has quit IRC17:48
*** etingof has joined #tripleo17:53
*** oidgar has joined #tripleo17:57
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: Update containers when the overcloud is containerized  https://review.openstack.org/52939917:57
*** gvrangan has quit IRC17:58
*** yprokule has quit IRC17:58
*** derekh has quit IRC18:01
*** alee is now known as alee_lunch18:02
Tenguhello there!18:02
Tengusmall question: would tripleo support a deploy using a local Satellite instead of Delorean repositories? I saw something regarding rhn registration - but I'll need to kind of override the delorean repositories, for both undercloud and overcloud.18:03
*** thrash is now known as thrash|biab18:04
Tengujanki: you probably need to ensure the qemu image is owned by "qemu" or something like that - you might want to `grep 107 /etc/passwd' in order to check the username.18:05
*** dparkes has joined #tripleo18:05
Tengujanki: had that very same issue while playing with ansible/libvirt/qemu-image this week :). Had to ensure the rights were OK on the image.18:06
jankiTengu, the reason was https://review.openstack.org/#/c/535293/. I commented these lines and it worked fine18:07
*** janki has left #tripleo18:08
beaglestrozet: out of curiousity - is there a plan for a scenario test for ODL?18:08
*** janki has joined #tripleo18:08
Tenguhmm. interesting issue.18:08
Tengujanki: thanks for the link. I got it working with the correct rights on the image, at least. but I'm not using the quick-start, having to push other things :)18:09
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-quickstart master: New featureset for fast forward upgrade testing.  https://review.openstack.org/52568618:09
*** fzdarsky is now known as fzdarsky|afk18:10
jankiTengu, I had stack:stack and rw18:10
*** agopi has quit IRC18:12
Tengujanki: in my case (remember, not quick-start) user 107 is "qemu". Or something like that. Anyway, if you have it working, that's good enough ;)18:12
*** mcornea has quit IRC18:13
*** dpawar has joined #tripleo18:14
*** agopi has joined #tripleo18:16
*** apevec_ is now known as apevec18:17
*** oidgar has quit IRC18:18
jankiTengu, ya. thanks for the reply18:19
*** jkilpatr has joined #tripleo18:20
*** shardy has quit IRC18:22
*** gvrangan has joined #tripleo18:22
weshay|roveralee_lunch, ok.. recreate run started18:23
Tenguhmmm. ok, just saw a doc for the satellite part. That should work pretty nicely in fact, especially with the "activation key" that should automagically enable repositories. I love that.18:24
Tenguweshay|rover: hello :). guess what. I'm playing with ansible once again, in order to get my openstack lab + satellite + undercloud-prod on a host, as virtual machines :)18:25
weshay|roverTengu, dam man.. you are awesome18:25
Tengufor now it's working. That's the funny part :]18:26
*** janki has quit IRC18:26
*** links has quit IRC18:26
weshay|roverget this man a fedora hat please18:26
*** karthiks has quit IRC18:26
Tenguyou might give me one at the PTG ;).18:27
*** yolanda has joined #tripleo18:29
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo stable/pike: Prepare 7.4.8 (pike)  https://review.openstack.org/53692018:30
*** fpan has quit IRC18:31
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo stable/ocata: Prepare 6.5.8 (Ocata)  https://review.openstack.org/53692118:31
*** rmascena has joined #tripleo18:31
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo stable/newton: Prepare 5.6.8 (newton)  https://review.openstack.org/53692318:32
*** raildo has quit IRC18:35
*** lucasagomes is now known as lucas-afk18:35
*** thrash|biab is now known as thrash18:36
*** pblaho has quit IRC18:37
*** cylopez has quit IRC18:38
*** mdnadeem has quit IRC18:39
*** links has joined #tripleo18:39
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: FFU: Add support for panko service upgrade  https://review.openstack.org/53376418:47
*** fragatina has quit IRC18:48
*** jpena is now known as jpena|off18:48
*** dtantsur is now known as dtantsur|afk18:48
*** fragatina has joined #tripleo18:49
*** gvrangan has quit IRC18:50
*** dpawar has quit IRC18:51
mfedosinEmilienM: hey! do you have 5 minutes? I want to tell you about one cool optimization we can implement in tripleo...18:51
EmilienMmfedosin: go ahead18:52
*** dsariel has joined #tripleo18:53
mfedosinEmilienM: I've noticed that mistral consumes a lot of cpu18:53
mfedosinfor instance http://paste.openstack.org/show/644910/18:53
*** fpan has joined #tripleo18:54
*** holser has quit IRC18:54
mfedosinanAnd this is its normal behavior18:54
mfedosinit happens because of cron trigger mechanism that hammers the database18:55
mfedosinrecently we added new option that allows to disable cron triggers for mistral18:55
mfedosintake a look: https://github.com/openstack/mistral/blob/763a1317b10eefa49f9419577495af103cd7f638/mistral/config.py#L244-L25418:56
*** gfidente has quit IRC18:57
EmilienMwhat happens when we disable the cron?18:57
EmilienMwhat impact on Mistral workflows?18:57
mfedosinafaik they want to deprecate and remove this feature18:58
EmilienMdoes it impact workflows running periodically?18:58
EmilienMso why providing True as a default value?18:58
*** pkovar has quit IRC18:58
EmilienMit sounds like a bad default for users, isn't?18:58
EmilienMthrash, rbrady ^ fyi18:58
mfedosinyes, it is, but I think they did it for backward compatibility18:59
EmilienMmfedosin: thank you18:59
*** jkilpatr has quit IRC18:59
mfedosinbecause we backported this parameter to pike18:59
mfedosinI tested it, and this allows us to reduce CPU consumption by 20-30%18:59
mfedosinfor the undercloud18:59
EmilienMso iiuc:19:00
EmilienMenabled = false19:00
*** myoung is now known as myoung|biab19:00
EmilienMmfedosin: that's huge19:00
EmilienMthat will make rook happy today19:00
EmilienMand we all want rook to be happy19:00
EmilienMthe option isn't in puppet-mistral, we would have to add it and backport19:01
mfedosinit's not there19:01
mfedosinbut I proposed a patch19:02
EmilienMmfedosin: where?19:02
rbradyEmilienM: the cron feature was enabled to support the UI logging19:02
EmilienMrbrady: so we need it enabled in tripleo?19:02
*** holser has joined #tripleo19:02
EmilienMmfedosin: +219:03
mfedosinEmilienM: thank you :) it was my first puppet19:03
rbradyEmilienM: I believe so.  Honza would be a good person to ask.19:04
EmilienMpretty good for a first patch19:04
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates master: Ensure get-occ-config wait on very slow deployed servers.  https://review.openstack.org/53693219:04
EmilienMrbrady: the performance issue is pretty big19:04
EmilienMrbrady: CPU consumption by 20-30% o_O19:04
EmilienMit sounds weird to me19:04
*** agopi has quit IRC19:05
EmilienMmfedosin: how did you you run benchmarks?19:05
rbradyEmilienM: I don't doubt that, but I think it's good to give the UI squad a short time to think of a workaround before you cut off their logging functionality19:05
*** agopi has joined #tripleo19:05
thrashEmilienM: I believe the UI team uses a cron trigger.19:05
EmilienMsure, let's give them some time to get the info19:05
mfedosinEmilienM: we hit this issue We were faced with this problem in Nokia, and the decision was to turn them off19:06
EmilienMjrist: ^ can you make sure the discussion happens in your squad?19:06
honzaEmilienM: rbrady: why are we turning off the feature instead of fixing the performance issue?19:06
*** holser has quit IRC19:06
mfedosinEmilienM: there was another optimization: https://github.com/openstack/mistral/blob/763a1317b10eefa49f9419577495af103cd7f638/mistral/config.py#L209-L23019:07
EmilienMhonza: we don't turn anything off now19:07
rbradyhonza: mfedosin mentioned the feature is likely intended to go away in mistral19:07
EmilienMhonza: but yeah it's my question, why do we have this performance issue19:07
honzaEmilienM: rbrady: I'm fine with turning it off.  At the moment, and probably for the rest of queens, logging is busted anyway :(19:08
honzasorry, i don't have much context19:08
mfedosinand afair the values we used in Nokia were 2 and 1, respectively. but I think defaults are also fine19:09
mfedosinEmilienM: what I want to try is to disable cron triggers on ci and test the performance19:10
*** moshele has quit IRC19:10
EmilienMI'll defer to honza / rbrady / thrash for this decision19:10
EmilienMbut anything that can save time in CI is welcome19:10
honzamfedosin: fine by me19:11
mfedosinThank you! always happy to help :)19:11
thrashhonza: curious about the logging failing... Is that on your end? Or mistral?19:14
*** alee_lunch is now known as alee19:14
*** rbowen has joined #tripleo19:16
honzathrash: definitely not mistral19:16
*** links has quit IRC19:19
jaosoriorslagle: there still seems to be an issue with the rendering https://logs.rdoproject.org/31/534731/6/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/Z09a94029f2014a55add59886b578cbbe/undercloud/var/lib/mistral/b2aa6168-bc4f-4ff5-8a9d-545da7184997/ansible.log.txt.gz#_2018-01-23_16_49_12_52819:21
jaosoriorEmilienM: could you check these out if you have a chance https://review.openstack.org/536865  https://review.openstack.org/53686719:24
jaosoriorand this https://review.openstack.org/#/c/533704/19:25
jaosoriorthey're backport19:25
EmilienMjaosorior: done19:25
honzaour project's tarballs aren't being synced to tarballs.openstack.org, where can i report that?19:28
aleeweshay|rover, any env so far?19:29
* weshay|rover checks19:29
weshay|roverTASK [overcloud-deploy : Deploy the overcloud] ***********************************************************19:30
slaglejaosorior: different error though. i think the other problem is fixed19:30
jaosoriorslagle: progress it seems19:30
jaosoriorEmilienM: thanks19:30
fultonjslagle: how did your external deployment go with puppet-ceph?19:31
aleeweshay|rover, cool thanks19:31
jaosoriorslagle: still seems like a formatting problem though19:31
slaglefultonj: seemed to work!19:32
fultonjslagle: ok, cool19:32
jaosoriorso there might be some other quotting issue somewhere19:32
fultonjslagle: did you try enable-ssh-admin.sh for using ceph-ansible?19:32
slaglejaosorior: maybe. it says "Undefined...". something is not defined19:32
slaglethat's my guess :)19:32
fultonji set up an env where i am about to do my first deployed-server and will add ceph-ansible external next19:33
slaglefultonj: i would have to convince them to redeploy again. i don't know if that's going to happen. especially to just test if it works19:33
fultonji see, well either way this combination should be tested and docs/code updated if necessary19:34
slaglefultonj: is it a seemless upgrade from external ceph when using puppet-ceph to ceph-ansible?19:35
*** jwb has quit IRC19:35
jaosoriorslagle: right, but if you read the stuff it tried to serialize, it failed with undefine because of:" Unexpected templating type error occurred on ..."19:35
fultonjslagle: yes, it should be19:36
jaosoriorso the json_query returned undefined because there was an exception rendering hostvars again19:36
*** marios has quit IRC19:37
jaosoriormore specifically19:37
jaosoriorfatal: []: FAILED! => {"failed": true, "msg": "Error in jmespath.search in json_query filter plugin:\nUnexpected templating type error occurred on  ....19:37
*** fragatina has quit IRC19:38
*** amoralej is now known as amoralej|off19:38
jaosoriorwhich happened when doing a json_query on hostvars19:39
*** sshnaidm has quit IRC19:39
slaglejaosorior: so you may need to determine where exactly in hostvars is throwing it off19:39
slaglethen trace back how that value gets set19:39
openstackgerritMark Hamzy proposed openstack/python-tripleoclient master: Support multiple architectures  https://review.openstack.org/52806019:39
jaosoriorslagle: any tips?19:39
openstackgerritwes hayutin proposed openstack/tripleo-specs master: introduce a ci team structure policy  https://review.openstack.org/50928019:40
jaosoriorslagle: I've been out of ideas for the past 4 days. So yeah, your patch was really helpful in getting forward, but I'm really not sure where to start tracking this :/19:40
*** trown|lunch is now known as trown19:40
*** trown is now known as trown|ruck19:40
slaglejaosorior: what happens when you use a debug task to print hostvars?19:43
slaglejaosorior: with everything else the same, inventory, group_vars, etc19:43
aleejaosorior, what about this here -- https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-master/b7194c8/subnode-2/var/log/journal.txt.gz#_Jan_23_13_37_42  ?19:43
jaosoriorslagle: it breaks too19:43
jaosoriorslagle: that's the last thing I had checked with shardy19:44
jaosorioralee: seems it did run the db_sync19:45
slaglejaosorior: in that case, I would run with -vvv, see if you can get a traceback from ansible, put a breakpoint in ansible where it's passing hostvars down into the templating to print out the value, then much with hostvars by removing keys, etc until you can figure out the offending part19:45
aleejaosorior, how can you tell ?  there are a whole bunch of error messages here -- not sure if they arew significant or not19:45
slagleperhaps iterate over the keys in hostvars and pass them in to however ansible is rendering the value one by one19:45
slaglesomething in the value is causing it to choke. we can either stare at the value and see if we can manually spot it, or take some approach similar to this debugging method19:46
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates master: WIP: add support for libvirt VNC TLS with dedicated CA  https://review.openstack.org/53646919:46
openstackgerritBrent Eagles proposed openstack/tripleo-heat-templates master: Configure neutron notifier to use internal endpoint type for nova  https://review.openstack.org/53694419:47
*** ykarel is now known as ykarel|afk19:50
aleejaosorior, I'm looking at abnother log where this apparaently worked before -- http://logs.openstack.org/67/536467/2/check/tripleo-ci-centos-7-scenario002-multinode-oooq-container/553dffc/logs/subnode-2/var/log/journal.txt.gz#_Jan_23_13_08_0919:50
aleejaosorior, see the same kinds of error messages - so they are probably not signiificant19:51
*** dparkes has quit IRC19:52
*** dsneddon has quit IRC20:00
EmilienMowalsh: where are we with https://blueprints.launchpad.net/tripleo/+spec/tripleo-realtime ?20:01
owalshEmilienM: need to ask ansiwen20:01
*** dsneddon has joined #tripleo20:01
owalshEmilienM: AFAIK everything has been proposed now but I could be wrong20:02
EmilienMtonyb: https://blueprints.launchpad.net/tripleo/+spec/multiarch-support - should we close it for queens?20:02
*** ykarel|afk has quit IRC20:03
jaosorioralee: not sure where the issue is there :/20:04
jaosoriorneed to reproduce it20:04
*** ykarel|afk has joined #tripleo20:06
*** itlinux has joined #tripleo20:07
*** nyechiel has quit IRC20:07
EmilienMjrist: https://blueprints.launchpad.net/tripleo/+spec/tripleo-ui-undercloud-container20:08
aleejaosorior, yup -- I'm trying to reproduce and weshay|rover almost has an env ..20:08
EmilienMjrist: I'm moving it to Rocky, no updates since December20:08
weshay|roverit's running tempest now20:08
EmilienMjaosorior: hey20:09
EmilienMjaosorior: what did we say about https://blueprints.launchpad.net/tripleo/+spec/logging-stdout-rsyslog again?20:09
weshay|roveralee, paste a pub ke20:09
weshay|roverit just completed20:09
EmilienMjaosorior: close it?20:09
weshay|roverDetails: {"code": 500, "description": "Secret creation failure seen - please contact site administrator.", "title": "Internal Server Error"}20:09
weshay|roverrecreate success \0/20:09
*** tcw has quit IRC20:10
EmilienMjaosorior: same for https://blueprints.launchpad.net/tripleo/+spec/ipsec20:10
aleeweshay|rover, see pm20:11
*** sshnaidm has joined #tripleo20:11
*** ihrachys has quit IRC20:17
*** ihrachys_ has joined #tripleo20:17
larsksslagle: dumb question: why does t-h-t use legacy ansible syntax (module: key=value key=value) rather than YAML syntax? I see you fixed that recently in one place in https://review.openstack.org/#/c/536849.20:21
*** myoung|biab is now known as myoung20:23
*** Jeffrey4l has quit IRC20:25
*** Jeffrey4l has joined #tripleo20:25
slaglelarsks: legacy?20:26
larsksslagle: yeah.20:26
slaglelarsks: i see the difference. likely the author was not aware the former is legacy20:26
larsksFair enough.  I've noticed that all over t-h-t and was just curious.20:27
slaglewhere is this documented. we can quote it as a cluestick20:27
openstackgerritBrent Eagles proposed openstack/tripleo-heat-templates stable/ocata: Remove non-conditional worker count configuration  https://review.openstack.org/53695720:28
larsksslagle: I dunno if I can find it documented, but using yaml dicts often simplifies things a lot.  I wasn't suggesting a change, I was just curious if there was a particular reason...20:29
slaglenone that I'm aware20:29
slaglei prefer the yaml, but still use key=value quite a bit with ad-hoc command line ansible20:30
jaosoriorEmilienM: I want it tested upstream before closing it :D20:30
larsksslagle: well, sure, it's a lot harder to type yaml on the cli :)20:30
jaosoriorEmilienM: and it currently doesn't work with the --config-download flag (I gotta run ansible manually), cause there are some template issues that we were discussing earlier20:31
*** salmankhan has quit IRC20:32
EmilienMjaosorior: and logging?20:39
*** jkilpatr has joined #tripleo20:39
jaosoriorEmilienM: I proposed a commit to the specs repo to pass it to rocky, I had no chance to finish up that work20:41
EmilienMjaosorior: so moving https://blueprints.launchpad.net/tripleo/+spec/ipsec to rocky?20:41
EmilienMnot this one20:41
*** nyechiel has joined #tripleo20:46
EmilienMjaosorior: what about this one ^20:47
*** rmascena has quit IRC20:47
openstackgerritSteve Baker proposed openstack/python-tripleoclient master: DNM Move more prepare logic into kolla_builder  https://review.openstack.org/52657620:48
openstackgerritSteve Baker proposed openstack/python-tripleoclient master: Add --tag-from-label to prepare command  https://review.openstack.org/52944520:48
*** cdearborn_ has joined #tripleo20:50
jristEmilienM: yes it is moved to Rocky20:52
jristEmilienM: thanks20:52
EmilienMjrist: ok, please handle the blueprints with the squad, it would really help me20:53
*** moshele has joined #tripleo20:54
tonybEmilienM: mwhahaha didn't grant me an FFE, but said to keep the patches coming and then evaluate it's impact on queens/x86_6420:57
*** dsneddon has quit IRC20:58
*** dsneddon has joined #tripleo20:58
tonybEmilienM: I agree the code impact is larger than expected but I don't think the functional impact is terrible20:58
jaosoriorEmilienM: same issue20:59
*** ykarel|afk has quit IRC20:59
*** itlinux has quit IRC20:59
*** moshele has quit IRC20:59
*** cdearborn has quit IRC21:01
*** moshele has joined #tripleo21:01
*** dparkes has joined #tripleo21:03
jristEmilienM: will try21:05
EmilienMjrist: quite a bit of a challenge to keep track of ~20 blueprints per cycle tbh21:05
EmilienMjrist: not talking about the hundred of bugs21:05
*** Goneri has quit IRC21:06
*** moshele has quit IRC21:09
*** dsneddon has quit IRC21:12
*** atoth has quit IRC21:12
*** dsneddon has joined #tripleo21:13
*** arxcruz|brb is now known as arxcruz21:13
*** jcoufal has quit IRC21:14
*** tcw has joined #tripleo21:17
*** tcw has quit IRC21:23
*** tcw has joined #tripleo21:23
*** moshele has joined #tripleo21:27
openstackgerritMerged openstack/tripleo-docs master: add reproducer-quickstart.sh documentation  https://review.openstack.org/53462121:27
openstackgerritCarol Bouchard proposed openstack/tripleo-heat-templates stable/pike: Latest Nexus Configuration Variables Updates  https://review.openstack.org/53697121:33
*** dprince has quit IRC21:38
*** abishop has quit IRC21:39
openstackgerritCarol Bouchard proposed openstack/tripleo-heat-templates stable/ocata: Latest Nexus Configuration Variables Updates  https://review.openstack.org/53697721:42
*** tcw has quit IRC21:43
*** salmankhan has joined #tripleo21:46
openstackgerritMark Hamzy proposed openstack/tripleo-common master: Support multiple architectures  https://review.openstack.org/52800021:46
*** fragatina has joined #tripleo21:47
*** agopi has quit IRC21:48
*** salmankhan has quit IRC21:50
*** tcw has joined #tripleo21:52
*** holser has joined #tripleo21:52
*** trown|ruck is now known as trown|outtypewww21:54
*** pchavva has quit IRC21:57
*** tcw has quit IRC21:57
*** vpickard is now known as vpickard_22:00
*** akrivoka has quit IRC22:00
fultonjslagle: while using deployed-server my overcloud's interface to reach the undercloud is modified by os-net-config and it looses its connection to the undercloud causing the stack update to fail22:02
ihrachys_haleyb, I remember you tried the reproducer-script.sh22:02
ihrachys_haleyb, where did you get it from?22:02
ihrachys_from u/s logs?22:02
*** Goneri has joined #tripleo22:03
openstackgerritCarol Bouchard proposed openstack/tripleo-heat-templates stable/newton: Latest Nexus Configuration Variables Updates  https://review.openstack.org/53698822:03
slaglefultonj: don't modify it that way :)22:04
slaglefultonj: you could use a different nic22:04
ihrachys_weshay|rover, maybe you have an idea where I am supposed to get the 'reproducer-script.sh' if it's not committed in git and I don't need any patches included in the environment except those that are committed in corresponding upstream repos.22:04
slaglefultonj: or, express the needed configuration via os-net-config so that it adds the route back22:04
*** noslzzp_ has quit IRC22:05
fultonji used a ctlplane-assignments.yaml as per the doc assuming it would modify that connection (my overcloud's only connection) but re-establish it's IPs by specifying controller-ctlplane: fixed_ips: ip_address22:06
aleeEmilienM, weshay|rover  so I see (as I suspected) that some data which is supposed to be written to the barbican database on service startup has not been written.22:06
slaglefultonj: that is how it works22:07
ansiwenEmilienM: re https://blueprints.launchpad.net/tripleo/+spec/tripleo-realtime -> just two merges missing: https://review.openstack.org/#/q/topic:bp/tripleo-realtime22:07
aleespecifically, its data about what what secret stores are available.22:07
slaglefultonj: check /etc/os-net-config/config.json on the node and see if that looks as you expect22:07
fultonjslagle: it does look as i'd expect22:08
aleeEmilienM, weshay|rover do you guys know how I might attach a debugger to the barbican process so that I could step through it on container startup?22:08
*** anilvenkata has quit IRC22:09
slaglefultonj: so what's the actual problem?22:09
fultonjslagle: my ctlplane-assignemnts.yaml > http://ix.io/EzN22:11
fultonjmy os-net-config/config.json > http://paste.openstack.org/show/651024/22:11
fultonjgiven this and that ens9 is my connection, which was hardcoded to when i started22:11
fultonjis it true that it should update ens9, e.g. add in ovs to the mix, and re-establish ens9?22:12
slaglefultonj: /etc/os-net-config/config.json is not right22:12
slagleyou should have an ip under for the ip_netmask value22:12
*** ccamacho has quit IRC22:13
slaglefultonj: looks like the hostname of the node is "over"?22:13
*** rlandy is now known as rlandy|bbl22:13
slagleyet the port mapping is "controller-ctlplane"22:13
slaglethat should be "over-ctlplane"22:13
*** rcernin has joined #tripleo22:15
*** moshele has quit IRC22:16
fultonjand my ip_address: is wrong in http://ix.io/EzN22:16
* fultonj replaces w/
openstackgerritDerek Higgins proposed openstack/tripleo-heat-templates master: WIP: Add ironic config to scenario 004  https://review.openstack.org/48526122:18
* fultonj updates and tries again22:19
fultonjthanks slagle22:19
weshay|roveralee, sorry man I don't22:20
aleeweshay|rover, I'll keep plugging at it -- at least I know whats messed up - now to just figure out why ..22:21
*** yamahata has joined #tripleo22:28
*** ccamacho has joined #tripleo22:31
*** apevec has quit IRC22:31
*** aputtur_ has quit IRC22:34
openstackgerritSteve Baker proposed openstack/python-tripleoclient master: Move more prepare logic into kolla_builder  https://review.openstack.org/52657622:36
openstackgerritSteve Baker proposed openstack/python-tripleoclient master: Add --tag-from-label to prepare command  https://review.openstack.org/52944522:36
*** wojdec1 has quit IRC22:37
*** lblanchard has quit IRC22:40
openstackgerritMerged openstack/tripleo-specs master: Move logging-stdout spec to Rocky  https://review.openstack.org/53684622:43
*** fragatina has quit IRC22:50
*** fragatina has joined #tripleo22:50
*** anilvenkata has joined #tripleo22:50
*** jmelvin has quit IRC22:54
*** PagliaccisCloud has quit IRC22:55
*** PagliaccisCloud has joined #tripleo22:59
*** anilvenkata has quit IRC23:01
*** alee has quit IRC23:02
*** artom has quit IRC23:03
*** toure is now known as toure|gone23:17
openstackgerritMerged openstack/tripleo-heat-templates master: Set permissions for openstack keyring on CephPools  https://review.openstack.org/53278523:21
openstackgerritMerged openstack/tripleo-heat-templates master: OpenShift: Don't pre-install Docker  https://review.openstack.org/53282723:21
*** tcw has joined #tripleo23:25
*** morazi has quit IRC23:26
*** dhill_ has quit IRC23:29
*** holser has quit IRC23:34
gunixwhere can i find undercloud logs ?23:35
gunixfor example the web gui is returning "Connection to Keystone API could not be established" and i want to debug this23:36
larsksgunix: keystone is probably running as a wsgi service under Apache...23:40
larsks...so logs are wherever the apache config puts them.  For Red Hat, that means /var/log/httpd/...23:40
larsks(where "..." might be something like "keystone_wsgi_admin_access.log")23:40
*** cdearborn_ has quit IRC23:42
*** alee has joined #tripleo23:43
gunixlarsks: i actually checked that path before posting here23:44
gunixlarsks: the only issue was, i checked it on the wrong serevr23:44
openstackgerritIhar Hrachyshka proposed openstack/tripleo-quickstart master: Added a troubleshooting tip to clean up ~/.quickstart  https://review.openstack.org/53700823:46
*** tosky has quit IRC23:46
*** dmacpher has quit IRC23:47
gunixlarsks: [root@undercloud ~]# tail -f /var/log/httpd/tripleo-ui_error.log23:47
gunixthis file is empty23:47
gunixthe push is 4 months old, the code returning the error is http://git.openstack.org/cgit/openstack/tripleo-ui/tree/src/js/services/KeystoneApiService.js?id=24facd7a39229e426f914c6c2050b6f7d7feeb6023:53
gunixi need to find that 401 ^^23:53
*** rlandy|bbl is now known as rlandy23:54
larsksYou said that keystone access from, e.g., the command line was working fine, right?23:55

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!