Wednesday, 2018-03-28

*** dparkes has quit IRC00:00
*** fragatina has quit IRC00:04
*** tbonds has quit IRC00:06
*** fragatina has joined #tripleo00:06
*** ooolpbot has joined #tripleo00:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION00:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/175755600:10
*** ooolpbot has quit IRC00:10
openstackLaunchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged]00:10
*** fragatina has quit IRC00:10
*** rlandy is now known as rlandy|bbl00:11
*** liverpooler has joined #tripleo00:11
*** dmacpher has quit IRC00:13
*** openstack has joined #tripleo00:14
*** ChanServ sets mode: +o openstack00:14
openstackgerritMarius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade  https://review.openstack.org/55668700:19
*** gvrangan has joined #tripleo00:20
openstackgerritSai Ram Peesapati proposed openstack/tripleo-heat-templates master: Resource name correction  https://review.openstack.org/55710100:21
openstackgerritMarius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade  https://review.openstack.org/55668700:21
*** abishop has quit IRC00:30
*** pmannidi has joined #tripleo00:33
*** yamahata has joined #tripleo00:38
*** fragatina has joined #tripleo00:39
*** fragatina has quit IRC00:43
*** yamahata has quit IRC00:49
*** dmacpher has joined #tripleo01:07
*** itlinux has joined #tripleo01:07
*** alee has joined #tripleo01:09
*** ooolpbot has joined #tripleo01:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION01:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/175755601:10
openstackLaunchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged]01:10
*** ooolpbot has quit IRC01:10
*** alee_ has quit IRC01:10
*** psahoo has joined #tripleo01:11
openstackgerritMerged openstack-infra/tripleo-ci master: Increase time for log collection in OVB  https://review.openstack.org/55688401:11
*** gyankum has joined #tripleo01:12
mwhahahaEmilienM: weshay: so i think the httpd count might account for the higher load that was previously mentioned for baremetal vs containers. we'll see what the tempest bits come back with01:13
*** gvrangan has quit IRC01:13
*** itlinux has quit IRC01:15
*** PagliaccisCloud has quit IRC01:19
*** PagliaccisCloud has joined #tripleo01:20
*** cshastri has joined #tripleo01:22
*** gvrangan has joined #tripleo01:26
*** tbonds has joined #tripleo01:32
*** gvrangan has quit IRC01:35
*** gvrangan has joined #tripleo01:44
*** gvrangan has quit IRC01:48
*** atoth has quit IRC01:54
openstackgerritMerged openstack/os-net-config stable/queens: Don't restart ivs/nfvswitch in os-net-config  https://review.openstack.org/55699802:01
*** ykarel|away has joined #tripleo02:02
*** cshastri has quit IRC02:08
*** ooolpbot has joined #tripleo02:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION02:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/175755602:10
openstackLaunchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged]02:10
*** ooolpbot has quit IRC02:10
*** ramishra has joined #tripleo02:11
*** psachin has joined #tripleo02:12
*** masco has joined #tripleo02:14
*** ykarel|away has quit IRC02:14
*** ykarel|away has joined #tripleo02:16
*** cshastri has joined #tripleo02:18
*** skramaja has joined #tripleo02:31
*** ykarel|away has quit IRC02:34
*** ccamacho|PTO has quit IRC02:38
*** jaganathan has quit IRC02:41
*** jaganathan has joined #tripleo02:42
weshaymwhahaha, geez my controller  load average: 22.79, 22.60, 22.4602:42
weshayhttpd and openvswitch02:42
weshayw/ https://review.openstack.org/#/c/557073/02:43
*** fragatina has joined #tripleo02:44
*** fragatin_ has joined #tripleo02:46
*** fragatina has quit IRC02:48
*** anilvenkata has quit IRC02:50
*** rbrady has joined #tripleo02:51
openstackgerritwes hayutin proposed openstack-infra/tripleo-ci master: featureset20 requires larger instances  https://review.openstack.org/55701502:54
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: DO NOT MERGE  https://review.openstack.org/55711602:56
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: allow full tempest fs020 more resources  https://review.openstack.org/55696502:59
*** dpawar has joined #tripleo03:03
*** dpawar has quit IRC03:03
*** dpawar has joined #tripleo03:04
*** ooolpbot has joined #tripleo03:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION03:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/175755603:10
*** ooolpbot has quit IRC03:10
openstackLaunchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged]03:10
*** ccamacho has joined #tripleo03:11
*** itlinux has joined #tripleo03:18
openstackgerritVincent S. Cojot proposed openstack/tripleo-heat-templates master: Santize shell syntax a bit  https://review.openstack.org/55713303:23
*** janki has joined #tripleo03:32
openstackgerritMarius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade  https://review.openstack.org/55668703:37
*** cshastri has quit IRC03:45
*** cshastri has joined #tripleo03:45
*** kbyrne has quit IRC03:51
*** jaosorior has joined #tripleo03:56
*** kbyrne has joined #tripleo03:56
*** udesale has joined #tripleo03:57
*** sanjayu has quit IRC04:06
*** sshnaidm_ has joined #tripleo04:08
*** sshnaidm has quit IRC04:08
*** pdeore has joined #tripleo04:09
*** pdeore has quit IRC04:09
*** ooolpbot has joined #tripleo04:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION04:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/175755604:10
*** ooolpbot has quit IRC04:10
openstackLaunchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged]04:10
*** ykarel|away has joined #tripleo04:15
*** shreshtha has joined #tripleo04:22
jaganathand0ugal, hi, when i execute wrokflow separately working fine after updating the workbook. but seems invoking old workbook when using deploy command.04:24
jaganathand0ugal, any idea?04:24
*** psachin has quit IRC04:25
*** masco has quit IRC04:26
*** sshnaidm_ has quit IRC04:32
*** cshastri has quit IRC04:39
*** psachin has joined #tripleo04:43
*** gkadam has joined #tripleo04:44
openstackgerritSai Ram Peesapati proposed openstack/tripleo-heat-templates master: Add Nuage Docker Service  https://review.openstack.org/55714904:44
*** links has joined #tripleo04:46
*** sshnaidm_ has joined #tripleo04:48
*** ratailor has joined #tripleo04:48
*** ratailor_ has joined #tripleo04:52
*** cshastri has joined #tripleo04:54
*** cshastri has joined #tripleo04:55
*** ratailor has quit IRC04:57
*** dpawar_ has joined #tripleo04:57
*** dpawar_ has quit IRC04:58
*** udesale has quit IRC04:58
*** udesale has joined #tripleo04:59
*** anilvenkata has joined #tripleo05:00
*** dpawar has quit IRC05:00
*** dmacpher_ has joined #tripleo05:02
*** masco has joined #tripleo05:02
*** dmacpher has quit IRC05:04
*** gkadam has quit IRC05:07
*** sshnaidm_ has quit IRC05:09
*** ooolpbot has joined #tripleo05:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION05:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/175755605:10
openstackLaunchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged]05:10
*** ooolpbot has quit IRC05:10
*** pdeore has joined #tripleo05:15
*** agurenko has joined #tripleo05:17
*** dpawar has joined #tripleo05:18
*** pgadiya has joined #tripleo05:19
*** moshele has joined #tripleo05:22
*** sshnaidm has joined #tripleo05:24
openstackgerritMerged openstack/tripleo-quickstart-extras master: Add option to specify the timeout for reproducer script  https://review.openstack.org/55525105:26
*** dmacpher__ has joined #tripleo05:32
*** dmacpher_ has quit IRC05:36
*** jfrancoa has joined #tripleo05:36
*** karthiks has quit IRC05:40
*** rlandy|bbl has quit IRC05:43
*** jtomasek has joined #tripleo05:43
openstackgerritJose Luis Franco proposed openstack/tripleo-quickstart master: Update overcloud-prep-containers script name.  https://review.openstack.org/55686005:48
*** avivgta has quit IRC05:48
itlinuxhello all, in a push for the overcloud, does the cinder-nic template gets to be overwritten by the controller or by the compute? I have setup the compute with two bonds. Thanks for the advice05:52
openstackgerritzenghui.shi proposed openstack/tripleo-heat-templates master: Add condition to ovs run during upgrade.  https://review.openstack.org/55392605:52
*** gfidente has joined #tripleo05:54
*** gfidente has quit IRC05:54
*** gfidente has joined #tripleo05:54
*** marios has joined #tripleo05:57
*** cshastri has quit IRC05:57
*** masco has quit IRC06:00
*** dpawar has quit IRC06:01
*** dpawar has joined #tripleo06:01
*** karthiks has joined #tripleo06:03
*** khyr0n has joined #tripleo06:05
*** dpawar has quit IRC06:05
openstackgerritzenghui.shi proposed openstack/tripleo-heat-templates master: Ensure openvswitch user exist before package update  https://review.openstack.org/55715506:08
*** cshastri has joined #tripleo06:09
*** ooolpbot has joined #tripleo06:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION06:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/175755606:10
*** ooolpbot has quit IRC06:10
openstackLaunchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged]06:10
*** moshele has quit IRC06:14
openstackgerritJuan Antonio Osorio Robles proposed openstack/python-tripleoclient master: Remove repeated code from TLS related bits in undercloud_config  https://review.openstack.org/55715906:17
openstackgerritJuan Antonio Osorio Robles proposed openstack/python-tripleoclient master: Enable TLS by default in the containerized undercloud  https://review.openstack.org/55716006:17
openstackgerritJose Luis Franco proposed openstack-infra/tripleo-ci master: Add multinode-overcloud-update playbook to run list  https://review.openstack.org/54705806:22
*** gfidente has quit IRC06:24
*** florianf has joined #tripleo06:24
*** dpawar has joined #tripleo06:26
*** udesale_ has joined #tripleo06:28
*** jaosorior has quit IRC06:29
*** ykarel_ has joined #tripleo06:30
*** udesale has quit IRC06:30
*** udesale__ has joined #tripleo06:30
*** yprokule has joined #tripleo06:30
*** dbecker has quit IRC06:32
*** jtomasek has quit IRC06:32
*** ykarel|away has quit IRC06:32
*** udesale_ has quit IRC06:33
*** pcaruana has joined #tripleo06:38
openstackgerritDan Macpherson proposed openstack/tripleo-docs master: WIP: Add FFU docs  https://review.openstack.org/54989206:39
*** ccamacho has quit IRC06:41
openstackgerritMerged openstack/tripleo-quickstart-extras master: Switch to tempest run cli for running tempest tests  https://review.openstack.org/55601406:43
*** ihrachys has quit IRC06:43
*** karthiks has quit IRC06:43
*** dbecker has joined #tripleo06:45
*** ccamacho has joined #tripleo06:59
*** karthiks has joined #tripleo07:02
*** ykarel__ has joined #tripleo07:03
openstackgerritzenghui.shi proposed openstack/tripleo-heat-templates stable/newton: Ensure openvswitch user exist before package update  https://review.openstack.org/55716407:04
*** aufi has joined #tripleo07:06
*** ykarel_ has quit IRC07:06
*** jaosorior has joined #tripleo07:10
*** ooolpbot has joined #tripleo07:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION07:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/175755607:10
*** ooolpbot has quit IRC07:10
openstackLaunchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged]07:10
jaosoriorbandini: could you check this out https://review.openstack.org/#/c/554836/07:11
*** avivgta has joined #tripleo07:12
*** ykarel__ is now known as ykarel07:13
*** rcernin has quit IRC07:15
*** ebarrera has joined #tripleo07:15
*** ebarrera has quit IRC07:19
*** ebarrera has joined #tripleo07:19
*** tzumainn has quit IRC07:22
*** mdnadeem has joined #tripleo07:22
*** tosky has joined #tripleo07:23
Tenguhello! small question: does the undercloud machine hostname impact the overcloud heat stack? I.e. if I decide to change the undercloud hostname, does it change anything for tripleo?07:24
*** tesseract has joined #tripleo07:25
openstackgerritJose Luis Franco proposed openstack/tripleo-quickstart-extras master: Create import-role role.  https://review.openstack.org/55349207:30
*** cylopez has joined #tripleo07:33
*** dmacpher__ has quit IRC07:33
*** cylopez1 has joined #tripleo07:34
*** gfidente has joined #tripleo07:36
*** gfidente has quit IRC07:36
*** gfidente has joined #tripleo07:36
*** holser__ has joined #tripleo07:37
*** cylopez has quit IRC07:37
*** shardy has joined #tripleo07:38
*** bogdando has joined #tripleo07:39
*** athomas has joined #tripleo07:43
*** udesale__ is now known as udesale07:43
*** skramaja is now known as skramaja|lunch07:46
*** mdnadeem has quit IRC07:51
*** cylopez1 has left #tripleo07:53
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras  https://review.openstack.org/47260707:55
*** lucas-afk is now known as lucasagomes07:56
*** mrunge has quit IRC07:58
*** mrunge has joined #tripleo08:01
*** pmannidi has quit IRC08:01
*** arxcruz|rover is now known as arxcruz|off08:03
*** jpich has joined #tripleo08:04
*** ktibi has joined #tripleo08:05
bogdandoo/ https://bugs.launchpad.net/tripleo/+bug/1759507 is it something new, or do we know a bz already?08:07
openstackLaunchpad bug 1759507 in tripleo "containerized keystone intermittently fails undercloud: grpc: the connection is unavailable" [High,Triaged]08:07
*** derekh has joined #tripleo08:08
*** ooolpbot has joined #tripleo08:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION08:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/175755608:10
*** ooolpbot has quit IRC08:10
openstackLaunchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged]08:10
*** mdnadeem has joined #tripleo08:11
*** gkadam has joined #tripleo08:19
*** akrivoka has joined #tripleo08:21
*** akane_ has joined #tripleo08:23
*** ccamacho has quit IRC08:23
*** rcernin has joined #tripleo08:24
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: WIP: Migrate Ceph deployment to external_deploy_task  https://review.openstack.org/54696608:36
*** dpawar has quit IRC08:40
*** rcernin has quit IRC08:42
*** masco has joined #tripleo08:43
*** anande has joined #tripleo08:45
*** ccamacho has joined #tripleo08:46
*** chem|eod is now known as chem08:49
*** anande has quit IRC08:53
*** anande_ has joined #tripleo08:53
*** ccamacho has quit IRC08:56
*** anande_ is now known as anande08:56
openstackgerritBogdan Dobrelya proposed openstack/python-tripleoclient master: Hard link http boot contents for Ironic  https://review.openstack.org/55651608:56
*** udesale has quit IRC08:58
bogdandofolks, PTAL https://review.openstack.org/#/q/topic:containers-default+(status:open+OR+status:merged)09:00
*** anande is now known as anande|afk09:01
jaosoriorbogdando: is the undercloud-upgrades job working?09:04
bogdandojaosorior: AFAIK, it should be. holser__^^09:05
bogdandothough not into the containerized UC yet09:05
jaosoriorbogdando: could you check this out https://review.openstack.org/#/c/557159/1 ?09:05
*** udesale has joined #tripleo09:05
bogdandojaosorior, holser__: http://logs.openstack.org/59/557159/1/check/tripleo-ci-centos-7-undercloud-upgrades/e4a1cf3/logs/var/log/undercloud_upgrade.txt.gz09:09
*** chem has quit IRC09:09
*** dpawar has joined #tripleo09:09
bogdandoSergii knows something of those online migrations fixes09:09
*** ooolpbot has joined #tripleo09:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION09:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/175755609:10
openstackLaunchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged]09:10
*** ooolpbot has quit IRC09:10
*** salmankhan has joined #tripleo09:10
*** chem has joined #tripleo09:10
holser__jaosorior, bogdando - it's a known bug I am producing a fix09:10
*** shardy has quit IRC09:10
jaosoriorholser__: thanks09:11
holser__jaosorior, give me 5 min09:11
holser__I am writing commit message09:11
holser__of fix09:11
jaosoriorbogdando: anyway, would appreciate the reviews for these https://review.openstack.org/#/q/status:open+project:openstack/python-tripleoclient+branch:master+topic:public-tls-default09:11
*** shardy has joined #tripleo09:12
bogdandojaosorior: this brings undercloud security into new level! W/o heat installer and templates integration, would never be a thing. Well done, really.09:14
*** skramaja|lunch is now known as skramaja09:14
bogdandojaosorior: is this tested by undercloud-containers job?..09:15
jaosoriorbogdando: that's what I need to figure out next.09:15
bogdando:)09:15
jaosoriorbogdando: any idea what fs configures that?09:15
bogdandofor ovb, this enables TLS for UC AFAICT https://review.openstack.org/#/c/542556/85/config/general_config/featureset001.yml09:17
bogdandoI can see no heat params adjusted, so is it defaults in t-h-t?09:17
bogdandolet me check for that undercloud containers job as well09:17
jaosoriorbogdando: that's awesome09:17
openstackgerritSergii Golovatiuk proposed openstack/instack-undercloud master: Update puppet-nova before running nova-manage  https://review.openstack.org/55726109:18
holser__jaosorior -  https://review.openstack.org/55726109:18
bogdandojaosorior: note, ovb is not yet switched to UC containers (WIP)09:18
*** waleedm has joined #tripleo09:18
*** akane_ has quit IRC09:20
jaosoriorbogdando: I wonder why we're trying to use undercloud deploy instead of using undercloud install --user-heat09:21
jaosoriorthat way it reads undercloud.conf, and it's a more straightforward way to upgrade to using containers, since we would then expect users to deploy with the same command.09:21
jaosoriorit would then be a matter of changing --use-heat to default to true09:23
holser__jaosorior EmilienM is working on that - so answer is WIP09:23
jaosoriorok09:23
jaosoriorthere probably is a reason, I was just wondering why09:23
holser__time & priorities...09:24
holser__but everything you say makes total sense to me09:24
bogdandojaosorior, holser__, EmilienM: is working on integrating --use-heat for instack upgrades09:25
bogdandojaosorior: any ideas how to figure out TLS deployment from the CI logs?..09:25
bogdandovia tht params passed prolly?09:26
bogdandojaosorior: http://git.openstack.org/cgit/openstack/tripleo-quickstart/tree/config/general_config/featureset027.yml here is the fs for that undercloud-containers job btw09:26
bogdandowe need it undercloud_generate_service_certificate: true?09:27
jaosoriorbogdando: yes09:28
openstackgerritMichele Baldessari proposed openstack/puppet-pacemaker master: Fix up error code path  https://review.openstack.org/55727309:28
bogdandojaosorior: commented09:29
bogdandowe prolly want only OVB fs001 testing it09:29
*** kopecmartin has joined #tripleo09:30
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-quickstart master: Test containerized undercloud with TLS  https://review.openstack.org/55727609:31
jaosoriorbogdando: lets give it a try ^^09:31
*** salmankhan has quit IRC09:32
*** anande|afk has quit IRC09:32
*** salmankhan has joined #tripleo09:35
*** itlinux has quit IRC09:36
*** dmacpher has joined #tripleo09:41
*** anande|afk has joined #tripleo09:43
*** anande|afk is now known as anande09:43
*** dtantsur|afk is now known as dtantsur09:46
openstackgerritMerged openstack/tripleo-common master: Refactor setting default CA  https://review.openstack.org/55483509:49
openstackgerritMerged openstack/tripleo-quickstart master: Add ff-upg to featureset table  https://review.openstack.org/55537209:49
*** anande has quit IRC09:50
*** anande has joined #tripleo09:50
*** ykarel is now known as ykarel|afk09:53
*** dpawar has quit IRC09:54
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates master: Add purge to Nova cleanup cron.  https://review.openstack.org/55196609:55
jaosoriorbogdando: could you check this one out as well https://review.openstack.org/#/c/554836/ ?09:55
*** liverpooler has quit IRC09:59
*** ratailor_ is now known as rtailor10:00
*** anande has quit IRC10:00
openstackgerritJiri Stransky proposed openstack/tripleo-quickstart master: Fix Fast Forward featureset documentation.  https://review.openstack.org/55680110:00
openstackgerritJiri Stransky proposed openstack/tripleo-quickstart master: Add missing Featureset037 documentation.  https://review.openstack.org/55681410:00
*** anande has joined #tripleo10:00
*** adarazs_lunch is now known as adarazs10:03
*** panda|off is now known as panda10:04
*** jaganathan has quit IRC10:09
*** ykarel|afk is now known as ykarel10:09
*** ooolpbot has joined #tripleo10:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION10:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/175755610:10
openstackLaunchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged]10:10
*** ooolpbot has quit IRC10:10
*** dpawar has joined #tripleo10:11
*** dmacpher has quit IRC10:20
openstackgerritJose Luis Franco proposed openstack-infra/tripleo-ci master: Default to docker.io instead of trunk.registry.rdoproject.org  https://review.openstack.org/54587310:21
openstackgerritJose Luis Franco proposed openstack-infra/tripleo-ci master: DNM: test P->Q Upgrade in experimental.  https://review.openstack.org/54547710:21
*** cshastri has quit IRC10:28
*** sshnaidm has quit IRC10:29
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Migrate Ceph deployment to external_deploy_task  https://review.openstack.org/54696610:31
jaosoriorslagle: Hey, I took a quick look at the patch you posted for populating the undercloud's /etc/hosts, but haven't figured out what's wrong there. Have you?10:36
*** moshele has joined #tripleo10:36
*** ansmith has quit IRC10:37
*** sshnaidm has joined #tripleo10:38
apetrichyolanda, ping. Do you have a second to talk about https://bugs.launchpad.net/tripleo/+bug/173141310:41
openstackLaunchpad bug 1731413 in tripleo "mistral db-sync error when upgrading undercloud from newton to ocata" [High,Triaged] - Assigned to Adriano Petrich (apetrich)10:41
*** dpawar has quit IRC10:45
*** udesale_ has joined #tripleo10:46
*** khyr0n has quit IRC10:48
*** udesale has quit IRC10:49
*** dpawar has joined #tripleo10:49
*** atoth has joined #tripleo10:52
*** udesale_ has quit IRC10:52
*** sshnaidm has quit IRC10:55
*** psahoo has quit IRC10:56
*** gyankum has quit IRC10:56
*** dpawar has quit IRC10:59
*** ykarel_ has joined #tripleo11:03
*** mdnadeem_ has joined #tripleo11:03
*** pdeore_ has joined #tripleo11:03
*** gkadam_ has joined #tripleo11:03
*** pgadiya_ has joined #tripleo11:03
*** shreshtha_ has joined #tripleo11:03
*** ratailor_ has joined #tripleo11:03
*** chkumar246 has joined #tripleo11:03
*** pdeore has quit IRC11:04
*** rtailor has quit IRC11:04
*** mdnadeem has quit IRC11:04
*** ykarel has quit IRC11:04
*** links has quit IRC11:04
*** chandankumar has quit IRC11:04
*** pgadiya has quit IRC11:04
*** gkadam__ has joined #tripleo11:04
*** shreshtha has quit IRC11:04
*** links has joined #tripleo11:05
*** shrjoshi has joined #tripleo11:05
*** ratailor__ has joined #tripleo11:05
*** ykarel__ has joined #tripleo11:05
*** pdeore__ has joined #tripleo11:06
openstackgerrityogananth subramanian proposed openstack/tripleo-common master: [WIP] Build OVS system container  https://review.openstack.org/55730011:06
*** gkadam has quit IRC11:07
*** sshnaidm has joined #tripleo11:07
*** shreshtha_ has quit IRC11:08
*** gkadam_ has quit IRC11:08
*** mdnadeem_ has quit IRC11:08
*** chkumar246 has quit IRC11:08
*** ratailor_ has quit IRC11:08
*** ykarel_ has quit IRC11:08
*** pgadiya_ has quit IRC11:08
*** pdeore_ has quit IRC11:08
*** jtomasek has joined #tripleo11:10
*** ooolpbot has joined #tripleo11:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION11:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/175755611:10
*** ooolpbot has quit IRC11:10
openstackLaunchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged]11:10
*** dpawar has joined #tripleo11:14
openstackgerritRafael Folco proposed openstack/tripleo-quickstart-extras master: [tempestmail] Add instruction to run unit tests  https://review.openstack.org/53050411:14
*** pkovar has joined #tripleo11:14
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Migrate Ceph deployment to external_deploy_task  https://review.openstack.org/54696611:18
*** panda is now known as panda|lunch11:18
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Migrate Ceph deployment to external_deploy_task  https://review.openstack.org/54696611:19
*** thrash|g0ne is now known as thrash11:19
*** chkumar246 has joined #tripleo11:20
openstackgerritMartin André proposed openstack/tripleo-docs master: Document toggling debug for containerized services  https://review.openstack.org/55730511:20
*** mdnadeem_ has joined #tripleo11:21
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/newton: Work around packaging issue in iptables-services.  https://review.openstack.org/55464711:22
*** pkovar has quit IRC11:24
*** shardy is now known as shardy_lunch11:24
*** pkovar has joined #tripleo11:27
*** jtomasek has quit IRC11:28
*** dtantsur is now known as dtantsur|brb11:31
*** rfolco|off is now known as rfolco|ruck11:34
jankiHi. How can I debug puppet hiera issue in deployment? I have a "log_mechanism" variable in THT. Default is file. I am changing it to "console" via env file. on overcloud node, hiera says the value is console but puppet manifest says "file" and the service is configured according to the manifest.11:34
jankimandre, marios shardy_lunch jaosorior ^ any idea?11:34
*** pdeore__ has quit IRC11:36
*** shrjoshi has quit IRC11:38
jaosoriorjanki: got links to the code? Which manifest and which variable in t-h-t?11:39
jistrmarios / matbu / chem: has anyone tested upgrade converge recently? does it work for you? i'm getting just this:11:40
jistrWaiting for messages on queue 'tripleo' with no timeout.11:40
jistrsocket is already closed.11:40
jistrand then nothing happens...11:40
jistri'll dive in, just wondering if anyone hit this already so that i don't do duplicate work11:40
jaosoriorjistr: I recently had the same issue (on update). I ended up redeploying the whole thing :/11:41
jankijaosorior, https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/opendaylight-api.yaml#L69 I am changing log mechanism at this by https://github.com/openstack/tripleo-heat-templates/blob/master/environments/services-docker/neutron-opendaylight.yaml#L1611:41
mariosjistr: i had that at some point but there was a missing patch (lemme see if i can find)11:41
mariosjistr: but generally converge hasn't passed for me yet. matbu is looking into it (i mean 'final bits for converge' )11:41
jaosoriorjanki: ok, what about the manifest that sets that?11:41
jankijaosorior, puppet will then configure the logging mechanism to console (the code is merged and working) and ODL will log to console. However, after the deployment ODL still logs to file becasue the manifest on controller says it is file11:42
mariosjistr: maybe this https://review.openstack.org/#/c/554237/11:42
jankijaosorior, and wierd thing is this whole thing works fine in non-ha deployment. ODL logs to console. but in HA, ODL Logs to file despite being directed to log to consolde11:42
*** leanderthal has joined #tripleo11:43
mariosjistr: (from my notes: still hanging ping matbu missing https://review.openstack.org/554237 & add to readme & rerun further converge fails for http://pastebin.test.redhat.com/567490  ) fwiw :)11:43
jankijaosorior, on overcloud controller nodes, hiers has the value of console11:43
mariosjistr: but if i recall yeah it was just hanging w no output11:43
openstackgerritLukas Bezdicka proposed openstack/tripleo-heat-templates stable/queens: FFU: Fix gnocchi FFU tasks  https://review.openstack.org/55730911:43
mariosjistr: and the stack never went to update in progress11:43
jaosoriorjanki: understood. Can you pass me the code that configures that in puppet?11:43
jistrmarios: yup that sounds like my issue, thanks a lot!11:43
jaosoriorjanki: I saw the hiera setup in t-h-t. Just need to see the puppet bits.11:43
*** dsariel has quit IRC11:43
mariosjistr: glad useful np11:44
*** gkadam__ has quit IRC11:44
jankijaosorior, https://github.com/opendaylight/integration-packaging-puppet-opendaylight/commit/cc8f1728228fbac279d91aab5837096907a5e82811:44
openstackgerritbinhong.hua proposed openstack/tripleo-quickstart-extras master: move extra_tht_config_args to overcloud_deploy  https://review.openstack.org/55731011:44
fultonjgfidente: fyi, i'm looking at your updates to https://review.openstack.org/#/c/54696611:45
gfidentefultonj believe it or not, it works11:45
fultonjawesome11:46
gfidenteI am sure there are bugs, but ... ps17 stands up things11:46
fultonji'll let you know how it works for me, and we'll see what ci says11:46
gfidentejistr ^^ maybe you want to review https://review.openstack.org/#/c/54696611:46
*** moshele has quit IRC11:47
gfidentejistr maybe you actually don't want to!11:47
gfidentejistr still11:47
gfidentepls :D11:47
jistr:D sure thing i'll review11:48
*** abishop has joined #tripleo11:48
bandinitherve: are you going to do the backport dance to stable/queens for https://review.openstack.org/521204 ?11:48
*** ykarel__ is now known as ykarel11:49
*** anilvenkata has quit IRC11:50
*** anilvenkata has joined #tripleo11:50
*** raildo has joined #tripleo11:51
*** quiquell is now known as quiquell|lunch11:53
*** ktibi has quit IRC11:54
*** anande has quit IRC11:54
*** morazi has joined #tripleo11:54
*** lucasagomes is now known as lucas-hungry11:55
jankijaosorior, any idea?11:56
jaosoriorjanki: I don't find where the ::opendaylight::config class is called11:56
*** jcoufal has joined #tripleo11:56
EmilienMhi11:57
jaosoriorjanki: oh wait, found it11:57
jaosoriorEmilienM: uh... you might be able to help janki better, she's having puppet-related issues.11:57
EmilienMjaosorior: i'm in meeting now11:58
*** jlabarre has joined #tripleo11:58
jankijaosorior, ohk.11:58
jaosoriorjanki: haven't figured out the issue yet11:58
jaosoriorjanki: I'll give it another try after the security squad meeting (which starts in 2 min)11:58
jankijaosorior, the code and implementation is good as non-HA deployment has correct configs and works fine. issue is in HA deployment only. sure. please let me know if you find anything11:59
jaosoriorjanki: will do11:59
jaosoriorhow the hell can it be snowing again.. get it together Finland!11:59
owalshjaosorior, lhinds: do you need me for anything at the security squad meeting? need to step out for a few minutes12:00
moguimarjaosorior: meeting time?12:00
jaosoriorlhinds, alee, moguimar around?12:00
moguimaro/12:00
thervebandini, Yep12:00
jaosoriorowalsh: no worries. There will be logs of the meeting. Just gonna point out the current progress of things.12:00
owalshjaosorior: ack, thanks, I'll review the logs right after12:01
jaosoriorLets wait a bit to see if more folks join12:01
raildoo/12:01
*** holser__ has quit IRC12:01
openstackgerritmathieu bultel proposed openstack/tripleo-heat-templates stable/queens: FFU: Fix gnocchi FFU tasks  https://review.openstack.org/55730912:01
*** holser__ has joined #tripleo12:01
*** tcw has quit IRC12:02
jaosoriorHey raildo, how's it going?12:02
*** udesale has joined #tripleo12:02
raildohey jaosorior, all good, what about you?12:02
jaosoriorall good here. Was hoping for spring to come... but it seems it was a false alarm12:03
moguimarhaha12:03
jaosorior#startmeeting TripleO Security Squad12:03
openstackMeeting started Wed Mar 28 12:03:45 2018 UTC and is due to finish in 60 minutes.  The chair is jaosorior. Information about MeetBot at http://wiki.debian.org/MeetBot.12:03
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.12:03
*** openstack changes topic to " (Meeting topic: TripleO Security Squad)"12:03
openstackThe meeting name has been set to 'tripleo_security_squad'12:03
jaosoriorAlright folks, lets start12:04
jaosorior#topic Work progress update12:04
*** openstack changes topic to "Work progress update (Meeting topic: TripleO Security Squad)"12:04
jaosoriorOut of all the topics, the main effort lately has been in two areas: * TLS by default  * Secret Management12:05
raildo#link https://etherpad.openstack.org/p/tripleo-security-squad12:05
jaosoriorthanks raildo12:05
jaosorior#topic Public TLS by default12:05
*** openstack changes topic to "Public TLS by default (Meeting topic: TripleO Security Squad)"12:05
jaosoriorSo, there has been a bunch of progress on this side.12:05
jaosoriorTLS is now the default in the classic undercloud, and mistral will automatically push the CA certificate to the overcloud nodes.12:06
jaosoriorI also have put up patches for enabling TLS by default for the containerized undercloud12:06
jaosoriornamely12:06
jaosorior#link https://review.openstack.org/55715912:06
jaosoriorand12:06
jaosorior#link https://review.openstack.org/55716012:06
jaosoriorAnd I actually got TLS (public) for the overcloud working.12:07
jaosoriorBut currently I'm facing some package related issues that prevent CI on that job from running properly12:07
jaosorior#link https://review.openstack.org/#/c/554926/12:08
jaosoriorThat one ^^ enables TLS by default for the overcloud. But is currently failing because I introduce a new script to tripleo-common, thus packaging fails.12:08
jaosorior#link https://review.openstack.org/#/c/554926/12:08
jaosoriorthat one ^^ is the commit to the spec file in RDO, that would enable that script. But I haven't gotten that commit to work yet12:09
jaosoriorSo any help on that side is welcome12:09
*** dprince has joined #tripleo12:09
jaosoriorThe rest of the reviews that have been put up are on the etherpad.12:09
jaosoriorAny questions/feedback?12:09
*** ooolpbot has joined #tripleo12:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION12:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/175755612:10
*** ooolpbot has quit IRC12:10
openstackLaunchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged]12:10
bogdandofolks PTAL https://review.openstack.org/#/c/553427/ all green12:10
raildofor now, I believe that we just need to go deeper in those patches and have that reviewed soon12:10
jaosoriorthat would be appreciated12:11
openstackgerritGael Chamoulaud proposed openstack/tripleo-validations stable/queens: Fix overcloud services connectivity validation  https://review.openstack.org/55731512:11
jaosoriorso please, if you have some time, reviews are always welcome12:11
*** ansmith has joined #tripleo12:11
jaosoriorMoving on12:12
jaosorior#topic Secret Management audit12:12
*** openstack changes topic to "Secret Management audit (Meeting topic: TripleO Security Squad)"12:12
jaosorioralee and me went through a deployment and started listing out the files and places where we have sensitive data12:13
openstackgerritChandan Kumar proposed openstack/tripleo-quickstart-extras master: Refactored validate-tempest role for undercloud and containers  https://review.openstack.org/55144112:13
jaosorior#link https://etherpad.openstack.org/p/tripleo-audit-secrets12:13
jaosoriorfortunately, it seems that we're well covered in basic unix permissions for those files.12:14
jaosoriorthe goal of that is, once we have identified everything, we'll go through those secrets and come up with strategies on how to secure them12:14
jaosoriorwe're already throwing around ideas on how to do that, but we'll need to meet up again to formalize some of those approaches, and discuss them further.12:14
jaosoriorSo, if folks have time to review the etherpad I posted above, please check it out; if we missed anything feel free to add that there.12:15
jaosoriorWe could probably use the same Etherpad to post some ideas on how to lock those places down12:16
jaosoriorAny questions/feedback?12:16
*** moshele has joined #tripleo12:18
raildojaosorior, it more fore curiosity, is there anyway to guarantee that we are covering all the secrets in this document?12:18
raildojaosorior, I mean, I'm supposing that you guys listed those secrets bases in your previous knowledge in where the secrets are, right?12:19
*** gkadam has joined #tripleo12:19
jaosoriorraildo: we also did a bunch of grep magic around the nodes :D but of course we might have missed something12:19
raildojaosorior, ok, got it12:20
jaosoriorraildo: this is why it's important to have more folks review that list, and if someone knows of something we missed, it should be posted there.12:20
jaosoriorand once we secure it, it'll be a matter of documenting the approach and writing up some best-practices.12:20
raildojaosorior, yeak, makes sense. thanks!12:20
jaosoriorin k8s, nobody stops you from writing out some passwords in your templates; but of course it's prefered to use k8s secrets instead.12:21
openstackgerritBogdan Dobrelya proposed openstack/python-tripleoclient master: Hard link http boot contents for Ironic  https://review.openstack.org/55651612:21
jaosoriorAnyway12:22
jaosoriorthat's all from my side12:22
*** eck` is now known as eck`gone12:22
jaosorior#topic Any other business12:22
*** openstack changes topic to "Any other business (Meeting topic: TripleO Security Squad)"12:22
jaosoriorDoes someone have something else that you would like to bring up to the meeting?12:22
*** ramishra has quit IRC12:22
*** moshele has quit IRC12:23
holser__bnemec - Do we really need https://github.com/openstack/instack-undercloud/blob/master/instack_undercloud/undercloud.py#L2362-L236612:24
moguimarnothing here12:24
*** pradk has quit IRC12:24
jaosoriorAlright12:24
holser__I guess it was special case for N>M upgrade12:24
jaosoriorRemember that if you're interested in a topic and would like some help getting started out, you're welcome to reach out to me or lhinds and we'll help you out getting started12:24
jaosoriorThanks for attending folks!12:25
jaosorior#endmeeting12:25
*** openstack changes topic to "Welcome to Rocky. CI status - Promotions: RED; check/gate: GREEN; RDO CI jobs: Questionable | http://tripleo.org/ | https://docs.openstack.org/tripleo-docs/latest/"12:25
openstackMeeting ended Wed Mar 28 12:25:12 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)12:25
openstackMinutes:        http://eavesdrop.openstack.org/meetings/tripleo_security_squad/2018/tripleo_security_squad.2018-03-28-12.03.html12:25
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/tripleo_security_squad/2018/tripleo_security_squad.2018-03-28-12.03.txt12:25
openstackLog:            http://eavesdrop.openstack.org/meetings/tripleo_security_squad/2018/tripleo_security_squad.2018-03-28-12.03.log.html12:25
*** ramishra has joined #tripleo12:25
*** masco has quit IRC12:28
jaosoriorbandini: around?12:31
rookI wonder if my deployment didn't pull the patch for the apache worker change12:32
rooki set ZUUL_CHANGES="openstack/tripleo-heat-templates:master:refs/changes/73/557073/1", but i don't see a reduction12:33
jaosoriorjistr, marios: Do the update_tasks also get run on upgrades?12:33
jankiEmilienM, hey, are you free now?12:34
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/queens: Fix fluentd upgrade tasks during FFU.  https://review.openstack.org/55483112:34
*** quiquell|lunch is now known as quiquell12:34
*** pchavva has joined #tripleo12:36
openstackgerritMerged openstack/tripleo-common master: Read bytes for default CA  https://review.openstack.org/55483612:36
hjensasHey, I have a heat templating question. I would like to get_attr the hostname in a puppet service template. Ref: http://paste.openstack.org/show/716297/12:36
bandinijaosorior: sort of12:36
*** rlandy has joined #tripleo12:37
jaosoriorbandini: you had passed me an example from rabbitmq that would be relevant in the SSL issue.12:37
jaosoriorI lost it :(12:37
jaosoriorcould you pass it again?12:37
bogdandofolks, https://review.openstack.org/#/c/553427/ please12:37
bogdandoplease review https://review.openstack.org/#/q/topic:containers-default+(status:open+OR+status:merged)12:39
rookI wonder if my deployment didn't pull the patch for the apache worker change ( weshay )12:39
mariosjaosorior: no only the upgrade_tasks12:39
rooki set ZUUL_CHANGES="openstack/tripleo-heat-templates:master:refs/changes/73/557073/1", but i don't see a reduction12:39
bandinijaosorior: ah sure12:39
mariosjaosorior: (that is the upgrade_tasks_playbook and the deploy_steps_playbook )12:39
therveshardy_lunch, https://review.openstack.org/#/c/530948/ isn't used yet in tripleo, right?12:40
*** masco has joined #tripleo12:40
jaosoriormarios: that's good to know.12:40
bandinijaosorior: https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/pacemaker/rabbitmq.yaml#L309 for example12:40
openstackgerritJose Luis Franco proposed openstack/tripleo-common master: Log ansible and ansible-playbook command.  https://review.openstack.org/55732012:40
jaosoriorbandini: thanks12:40
rookhttps://snapshot.raintank.io/dashboard/snapshot/07Z6jz5SjyHBAw1feiOTxR5prpIur4ft?orgId=2 ( weshay EmilienM ) we can see from yesterday to today # of processes/threads12:41
jaosoriorbandini, marios: So... if I would like some tasks to run both on upgrade and update... I would need to replicate the code then?12:41
bandinijaosorior: hohum good question. dciabrin ^12:42
openstackgerritBogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: Revert "undercloud: workaround for masquerading network in CI/OVB"  https://review.openstack.org/55732212:42
openstackgerritJose Luis Franco proposed openstack-infra/tripleo-ci master: Add multinode-overcloud-update playbook to run list  https://review.openstack.org/54705812:42
*** chkumar246 is now known as chandankumar12:43
EmilienMrook: with alex's patch?12:44
*** dsariel has joined #tripleo12:44
rookEmilienM: I am wondering if I need to do more than set the ZUUL_CHANGES prior to running oooq -- i pinged weshay about it.12:44
rookI set that env var -- but I am doubting it is setup right, i can jump into the container and double check12:44
weshayrook, /me checks my env.. I did the same thing12:46
rookack12:46
*** sshnaidm is now known as sshnaidm|afk12:46
mariosjaosorior: no we do someincludes12:47
mariosbandini: already12:47
mariosjaosorior: sec let me get you pointer12:47
weshayrook, on your undercloud do you have /opt/gating_repo/openstack-tripleo-heat-templates-8.0.0-0.20180327225321.bfadb63.el7.centos.noarch.rpm12:47
EmilienMrook: you can easily check with ps and # of httpd processes12:47
mariosjaosorior: by includes i mean yaml & referencing & and *12:47
EmilienMjanki: what's up?12:47
rookEmilienM: that is what the graph i shared shows12:47
rookEmilienM: which doesn't show much of a difference12:47
rookweshay: ack let me check12:47
mariosjaosorior: e.g. here https://github.com/openstack/tripleo-heat-templates/blob/4ed5eae601ac591696f87db5b7c57eef97bce95c/docker/services/pacemaker/rabbitmq.yaml#L34512:48
EmilienMrook: because I tested mwhahaha's patch and I could reduce the number of process by 312:48
bandinimarios: ah right. damien mentioned that too me once12:48
rookNope weshay12:48
rookdoesn't exist12:48
mariosjaosorior: it is including this EATE_COMPLETE  state changed12:48
*** mcornea has joined #tripleo12:48
mariosjaosorior: this https://github.com/openstack/tripleo-heat-templates/blob/4ed5eae601ac591696f87db5b7c57eef97bce95c/docker/services/pacemaker/rabbitmq.yaml#L23512:48
mariosbandini: yeah i added it first for the upgrade_tasks, trying to re-use what we were already doing for the pike minor update12:49
mariosbandini: i.e. container to container12:49
bandiniright12:49
mariosbandini: wasn't sure it would work but it did :)12:49
bandinieheh w00t12:49
jankiEmilienM, I have a parameter in ODL THT will logs to file. I am chaging it to "console" via THT env file. ODL Puppet module changes are implemented and works fine. In non-HA deployment, ODL Logs to console but in HA, it logs to file even though specified to log to console via THT env file. hieradata on controller node has value set to console but the manifest at /etc/puppet/ has it set to file. I dont know why this is happening anf where to look further12:50
weshayrook, http://pastebin.test.redhat.com/56968112:50
*** salmankhan has quit IRC12:50
jaosoriormarios: thanks!12:51
rookok weshay let me nuke this and try again12:51
* marios nods jaosorior 12:52
jaosoriorbandini: got time for a quick call?12:52
*** lblanchard has joined #tripleo12:53
*** thrash is now known as thrash|biab12:53
rookbeagles: so close!12:54
rookStep312:54
*** lucas-hungry is now known as lucasagomes12:55
*** panda|lunch is now known as panda12:56
EmilienMjanki: show me the code in THT that does this param12:57
jankiEmilienM, https://github.com/openstack/tripleo-heat-templates/blob/master/environments/services-docker/neutron-opendaylight.yaml#L16, https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/opendaylight-api.yaml#L69 https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/opendaylight-api.yaml#L14712:58
openstackgerritMarius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade  https://review.openstack.org/55668712:59
jankiEmilienM, puupet-odl changes https://github.com/opendaylight/integration-packaging-puppet-opendaylight/commit/cc8f1728228fbac279d91aab5837096907a5e82812:59
*** salmankhan has joined #tripleo12:59
jankiEmilienM, I would like to emphasis on the point that in non-HA, ODL logs to console (as is specified and desired). In HA, it logs to file even though specified to log to console.13:00
EmilienMjanki: have you checked that the version of the puppet module deployed in your env is the right one and contains the right code?13:00
*** trown|brb is now known as trown13:00
skramajaEmilienM: could you take a look at https://review.openstack.org/#/c/554846/?13:01
EmilienMskramaja: sure13:01
skramajathanks.13:01
jankiEmilienM, yes version is correct. infact, hiera opendaylight::log_mechanism on controller node returns "console". so I am guessing correct value is being passed from THT to puppet.13:02
bandinijaosorior: sure13:02
trownchem: is ffu supposed to be from newton->queens?13:03
*** eck`gone is now known as eck`13:03
chemtrown: hi, yes13:04
jankiEmilienM, but /etc/puppet/modules/opendaylight has the log_mechanism value set to "file". I think something is wrong here13:04
trownchem: so skips 2 releases?13:04
*** aputtur_ has joined #tripleo13:04
chemtrown: yes13:04
trownok13:05
*** Goneri has joined #tripleo13:06
EmilienMjanki: it's very hard to tell when you can't reproduce easily13:06
trownchem: so for ffu job we deploy newton bits using queens heat templates13:06
trownchem: is that right?13:06
EmilienMjanki: and when there is no CI job in place13:06
openstackgerritMerged openstack/tripleo-common stable/newton: Fix overcloud node delete after an upgrade  https://review.openstack.org/54648713:07
openstackgerritMerged openstack/tripleo-common stable/ocata: Fix overcloud node delete after an upgrade  https://review.openstack.org/54648613:07
weshayEmilienM, so in the latest run fs001 passed, but 35 failed13:07
EmilienMjanki: so if I summarize, hiera returns the right value on the controller but it's not applied in the ODL config?13:07
jankiEmilienM, I am currently facing it in the system. Would you like to access it?13:07
EmilienMweshay: :( reason?13:07
weshaychecking13:07
jankiEmilienM, yes. correct13:07
chemtrown: hum... it's a mixed version so we deploy newton overcloud with checked out newton templates in ~/tripleo-heat-template from a queens undercloud13:08
jankiEmilienM, and this is happening only in HA deployment13:08
weshayEmilienM, same reason fs001 failed the last time https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-master/25885fd/tempest.html.gz13:08
weshayssh timeout13:08
EmilienMweshay: with new ovs?13:09
*** jmelvin has joined #tripleo13:09
weshayEmilienM, that package is in dlrn-deps so they all will have it, but me looks13:09
weshayEmilienM, openvswitch-2.9.0-3.el7.x86_6413:10
*** ooolpbot has joined #tripleo13:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION13:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/175755613:10
*** ooolpbot has quit IRC13:10
openstackLaunchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged]13:10
*** anilvenkata has quit IRC13:10
*** tzumainn has joined #tripleo13:10
*** toure|gone is now known as toure13:10
*** alee has quit IRC13:10
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Lower the number of apache servers  https://review.openstack.org/55707313:11
trownchem: ah i see that now, so we have to backport anything we want to use if ffu to newton (like keystone only)13:11
fultonjHas anyone seen docker.io/tripleomaster/centos-binary-cinder-volume fail to standup because it is trying to delete /etc/hosts?13:11
EmilienMmwhahaha: ^ added bug reference and also with new ovs so we can see if numbers are better13:11
chemtrown: yeap13:11
fultonjhttp://paste.openstack.org/show/716299/13:11
mwhahahaK13:12
*** ykarel_ has joined #tripleo13:12
*** ykarel has quit IRC13:12
*** ykarel__ has joined #tripleo13:14
*** dtantsur|brb is now known as dtantsur13:15
jankiEmilienM, any suggestion on where to look?13:16
jistrgfidente: reviewed. Just a nit + a few things to consider but overall i think it's very nice13:16
EmilienMjanki: not really, is the config file modified by other things at all during the deployment?13:16
*** ykarel_ has quit IRC13:17
jankiEmilienM, no. there is no other file changing the param. and if there had been any, hiera wont have shown "console" right??13:17
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/newton: Work around packaging issue in iptables-services.  https://review.openstack.org/55464713:20
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-upgrade stable/queens: WIP: Add support for per role FFU upgrade  https://review.openstack.org/55732813:22
chemmcornea: backported ^ to queens to have a test run13:23
EmilienMjanki: I guess I'm asking if the config file changed because some other parameters were changed too13:23
EmilienMjanki: i'm wondering if the config file is touched at all13:23
*** lblanchard has quit IRC13:24
mcorneachem: cool, one note about that - I saw that it triggers regular upgrade job while the patches touches only ffu part in tripleo-upgrade, can we limit the jobs based on the portion of code that is changed?13:24
chemmcornea: we can't until software factory has moved to zuulv313:25
mcorneachem: ack13:25
*** masco has quit IRC13:25
openstackgerritSergii Golovatiuk proposed openstack/instack-undercloud master: Remove nova-manage db from pre_upgrade  https://review.openstack.org/55733013:25
*** aputtur_ has quit IRC13:26
jankiEmilienM, I dont think this parameter is dependent on any other parameter. How to confirm it though?13:26
EmilienMjanki: ok let me try again: what other parameters in the config files are controlled by tripleo?13:27
*** psachin has quit IRC13:30
jankiEmilienM, https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/opendaylight-api.yaml#L61, SNAT mechanism. I guess I know where you are going with this. I need to check snat value in puppet manifest if that is changing13:30
EmilienMjanki: yes, check it13:31
EmilienMand see if they are applied.13:31
*** mmethot has quit IRC13:31
jankiEmilienM, well. I just started a stack update. I will check once this is over and get back to you13:31
*** aputtur_ has joined #tripleo13:31
*** ykarel__ is now known as ykarel13:31
*** pradk has joined #tripleo13:32
dtantsurhi folks, an easy patch for review, if you have a minute: https://review.openstack.org/#/c/555336/13:32
*** udesale has quit IRC13:32
*** thrash|biab is now known as thrash13:34
*** tcw has joined #tripleo13:35
skramajai see the RDO ci is failing most of the time. is it mandatory to get RDO ci verified before pushing patches? done few re-tries for https://review.openstack.org/#/c/555166/, but still failing..13:35
skramajaEmilienM: mwhahaha ^13:36
*** mmethot has joined #tripleo13:37
chandankumarweshay: EmilienM tosky arxcruz|off seomthing happened on containers undercloud with tempest http://logs.openstack.org/42/551442/7/check/tripleo-ci-centos-7-undercloud-containers/1310e2f/logs/undercloud/home/zuul/tempest_output.log.txt.gz#_2018-03-28_13_13_2913:38
weshaysomeone ate the cirros image eh13:38
toskyit wasn't me!13:39
EmilienMskramaja: +213:39
*** masco has joined #tripleo13:39
skramajathanks. i will give few more rechecks :)13:39
toskyI don't even remember who was supposed to pre-cache the image13:40
chandankumarweshay: tosky the good thing is that it has started running13:40
chandankumarfew more tweaks we are back on track13:41
*** liverpooler has joined #tripleo13:41
*** haleyb has quit IRC13:41
EmilienMskramaja: no need13:42
skramajaack.13:43
mwhahahachandankumar: it's cause the image isn't in the container, it's on the host. would need to get mounted13:43
chandankumarmwhahaha: yup one more thing, we need to create tempest related resources outside containers13:44
*** skramaja has quit IRC13:45
*** akane_ has joined #tripleo13:46
*** akane_ has quit IRC13:46
mwhahahaowalsh: seeing failures on upgrades for the undercloud in the nova client, http://logs.openstack.org/59/557159/1/check/tripleo-ci-centos-7-undercloud-upgrades/c63ec8a/job-output.txt.gz#_2018-03-28_10_46_04_20602313:46
*** aputtur_ has quit IRC13:50
jaosoriormwhahaha: holser__ was working on that ^^13:51
mwhahahayea13:51
*** ratailor_ has joined #tripleo13:58
*** ratailor__ has quit IRC14:01
bnemecholser__: I think we probably do, but you might want to talk to owalsh about that.14:04
*** udesale has joined #tripleo14:05
*** udesale has quit IRC14:05
*** udesale has joined #tripleo14:05
*** jrist has quit IRC14:07
*** dsariel has quit IRC14:09
*** dsariel has joined #tripleo14:09
openstackgerritMerged openstack/tripleo-quickstart-extras master: Add missing dash in repo-setup task.  https://review.openstack.org/55688614:09
openstackgerritMerged openstack/tripleo-quickstart master: Update overcloud-prep-containers script name.  https://review.openstack.org/55686014:09
*** ooolpbot has joined #tripleo14:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION14:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/175755614:10
*** ooolpbot has quit IRC14:10
openstackLaunchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged]14:10
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates master: get-occ-config.sh: fix handling multiple nodes in a role  https://review.openstack.org/55734614:10
*** ratailor_ has quit IRC14:11
slagleholser__: fyi^^14:11
openstackgerritMerged openstack/tripleo-common master: Fix missing permissions on Swift rebalance playbook  https://review.openstack.org/55692814:12
holser__slagle - thanks14:13
holser__I understand why you changed it14:14
jaosoriorbandini: pacemaker restart didn't work14:17
*** florianf has quit IRC14:17
openstackgerritRafal Szmigiel proposed openstack/tripleo-heat-templates stable/pike: Adding missing space in octavia-worker.yaml causing Pike deployment fail when Octavia and Fluentd are enabled.  https://review.openstack.org/55734714:18
bandinijaosorior: can you drop me and dciabrin a mail with the exact change and command you ran and result?14:18
bandiniI'm doing 5 things at the same time and am easily confused right now14:19
*** ykarel has quit IRC14:20
bogdandomwhahaha: chandankumar: so volumes here https://review.openstack.org/#/c/548325/19/docker/services/tempest.yaml@49 should prolly include, at the very least,  /opt/cache:/opt/cache IIUC14:20
mwhahahabogdando: chandankumar: yea but it probably should be configirable. this is one of the problems with using a container for a cli :/14:21
bogdando...via puppet-tripleo :)14:22
mwhahahafor containers it'd be THT14:22
mwhahahaprobably need a CacheDir option for tempest14:22
bogdandoI understood you wrong then, so let's add some TempestVolumes param there14:23
mwhahahayea14:24
mwhahahaso we need to pass in where we're going to pass in cached content14:24
*** florianf has joined #tripleo14:24
*** mcornea has quit IRC14:25
*** dougbtv has quit IRC14:25
*** masco has quit IRC14:25
*** waleedm has quit IRC14:27
openstackgerritRafal Szmigiel proposed openstack/tripleo-heat-templates stable/ocata: Adding missing space in octavia-worker.yaml causing Ocata deployment fail when Octavia and Fluentd are enabled.  https://review.openstack.org/55735314:32
*** moshele has joined #tripleo14:33
*** jrist has joined #tripleo14:33
*** jrist has joined #tripleo14:33
chandankumarmwhahaha: bogdando I need to mount files only if something is available14:33
chandankumarmwhahaha: bogdando in my first attempt, I want to get things working then we can improve the thingsa14:34
*** mcornea has joined #tripleo14:34
*** kopecmartin has quit IRC14:34
mwhahahachandankumar: if you're launching the container by hand (and specifying -v options) then you could do that14:34
chandankumarmwhahaha: yup I will update the patchset14:35
*** ramishra has quit IRC14:38
*** moshele has quit IRC14:40
*** etingof has quit IRC14:41
*** moshele has joined #tripleo14:43
*** holser__ has quit IRC14:44
openstackgerritRafael Folco proposed openstack/tripleo-quickstart-extras master: Temporarily add tempest failures to skip list  https://review.openstack.org/55735414:44
*** moshele has quit IRC14:45
*** ykarel has joined #tripleo14:46
*** holser__ has joined #tripleo14:50
openstackgerritSteven Hardy proposed openstack/python-tripleoclient master: Don't exit on a RUNNING message for deploy workflow  https://review.openstack.org/55735614:50
*** udesale has quit IRC14:50
therveWho can I bother for quickstart-extra reviews? I have dumb patches that have sitting for a while14:51
openstackgerritSteven Hardy proposed openstack/tripleo-common master: Standardise Workflow messaging and optionally persist messages  https://review.openstack.org/42506014:52
jaosoriortherve: I can help reviewing (tomorrow)14:52
thervejaosorior, Thanks I'll bother you then :)14:52
shardy_lunchmandre: ^^ updated the workflow patch, it doesn't handle the plan state correctly for tripleo.deployment.v1.deploy_plan yet, but my assumption is we'll be moving to the synchronous config_download version as discussed yesterday14:53
*** shardy_lunch is now known as shardy14:53
*** zoli is now known as zoli|gone14:54
*** zoli|gone is now known as zoli14:54
shardytherve: Hey I can help wrt reviews, will check your open reviews now14:55
therveshardy, https://review.openstack.org/#/q/owner:therve%2540redhat.com+status:open+project:openstack/tripleo-quickstart-extras14:55
thervehttps://review.openstack.org/#/c/536385/ if you like deleting code too :)14:56
thrashflorianf: Is it possible to override the variables in a validation when run from the ui?14:57
*** sshnaidm|afk is now known as sshnaidm14:57
thrashflorianf: or from the cli for that matter...14:58
florianfthrash: nope, not currently. IIRC all the ui hands over to mistral is the plan name. are you asking in general or something specific?14:58
thrashflorianf: well, both in general, and specifically to migrate the pre-deploy validations to tripleo-validations.14:58
*** sai_p has joined #tripleo14:59
*** udesale has joined #tripleo14:59
*** gfidente has quit IRC14:59
sai_pHello can someone review this https://review.openstack.org/#/c/546476/2 ? Thank you14:59
sai_pIt;s been open for a long time14:59
thrashflorianf: for instance, the check-boot-images validation that I pushed up... There is no way to change the default kernel and ramdisk name. It's not something that is likely going to differ, but it is possible.15:00
thrash*differ from the default15:00
florianfthrash: from the command line, if you use ansible-playbook, you could use the --extra-vars option15:00
thrashflorianf: I understand that part. But users won't be calling them directly to do so.15:01
florianfthrash: ok, i understand. is this something that would be detectable from the plan files?15:01
florianfthrash: or parameters?15:01
*** gfidente has joined #tripleo15:01
*** gfidente has quit IRC15:01
*** gfidente has joined #tripleo15:01
florianfthrash: or glance maybe?15:01
thrashflorianf: Right now, I don't think so. I think they were strictly cli params.15:01
thrashflorianf: we might want to use a tag on the glance images instead of relying on the name.15:01
openstackgerritChandan Kumar proposed openstack/tripleo-quickstart-extras master: Refactored validate-tempest role for undercloud and containers  https://review.openstack.org/55144115:02
bogdandodsneddon: hi, PTAL https://review.openstack.org/#/c/553427/15:02
florianfthrash: and then get the kernel and ramdisk names based on the tag?15:02
mandreshardy: that was a long lunch, even according to French standards :)15:02
thrashflorianf: at that point, the names would probably not matter. We just want to see that one and only one image has the tag.15:02
florianfthrash: that way you wouldn't need to add parameters, but the validation could just look them up from glance15:02
shardymandre: hehe yeah I forgot to update my nick (honest! :)15:02
florianfthrash: oh right, even better.15:02
*** links has quit IRC15:03
thrashflorianf: ok... Let me see what it would take to change that.15:03
florianfthrash: sounds great15:03
*** udesale has quit IRC15:03
openstackgerritChandan Kumar proposed openstack/tripleo-quickstart-extras master: Refactored validate-tempest role for undercloud and containers  https://review.openstack.org/55144115:04
mandreshardy: I'll review your changes, on my side I've been working on the next patch in the series to run the enable_ssh_admin workflow prior to config_download15:04
thrashrbrady,apetrich,jtomasek workflows squad meeting15:04
shardymandre: ack sounds good, then we can rebase the series and test together15:04
shardymandre: I think I noticed an issue with how tripleoclient handles SUCCESS messages but the workflow changes should be ready for review I think15:05
mandreshardy: alright, I'll give it a try locally15:05
shardycurrently we break on any SUCCESS message, but we're getting them from the nested workflows, so we actually break on tripleo.swift_rings_backup.v1.create_swift_rings_backup_container_plan not tripleo.deployment.v1.deploy_plan15:05
openstackgerritChandan Kumar proposed openstack/tripleo-quickstart master: Run tempest on fs027 using containerized tempest  https://review.openstack.org/55144215:06
openstackgerritChandan Kumar proposed openstack/tripleo-quickstart master: Run tempest on undercloud on fs003  https://review.openstack.org/55452215:06
bandinijaosorior: https://review.openstack.org/#/c/556772/ got a min?15:09
*** pcaruana has quit IRC15:09
*** ooolpbot has joined #tripleo15:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION15:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/175755615:10
*** ooolpbot has quit IRC15:10
openstackLaunchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged]15:10
*** agurenko has quit IRC15:11
bnemechjensas: Just to wrap up my undercloud installation issues from yesterday, it turned out to be a problem related to my proxy.15:11
bnemecApparently you now need to no_proxy the admin vip too.15:11
*** agurenko has joined #tripleo15:11
mwhahahaboo proxies15:11
*** gyankum has joined #tripleo15:11
hjensasbnemec: good to hear you where able to figure it out.15:12
*** chlong has joined #tripleo15:14
bnemechjensas: Thanks for your help with my investigation.  It was very helpful to have someone else check my results.15:14
*** yprokule has quit IRC15:15
bnemecmwhahaha: Yeah, unfortunately some of our users have to use proxies. :-/15:15
* bnemec does it so squid can cache packages15:15
mwhahahayea i know, it royally messes with things :/15:16
bnemecOoh, I probably have a blog post to update.15:16
bnemecI think I had one about using squid with TripleO.15:16
mwhahahabnemec: if you have a specific blog post around it let me know, i think this might be something we should document better15:16
shardysomewhat relatedly, now containers are the default, can we modify overcloud-full to remove all the packages?15:17
shardye.g the openstack stuff, they're all still in there and I'm guessing CI will run faster with a smaller image15:17
mwhahahashardy: not until we figure out the provision bits15:18
mwhahahashardy: it's more dependent on ironic & ovs and stuff15:18
shardymwhahaha: I'm not saying move to a generic centos image, just lets remove all the packages for things we now deploy in containers15:18
mwhahahawe ran into problem where we still are using things from the image build (ie openvswitch)15:18
mwhahahashardy: i think dprince had a test patch15:19
mwhahahawith that15:19
shardymwhahaha: Ok, well something to work towards I guess, it seems kind of a waste to have a huge glance image and also a bunch of huge container images15:19
mwhahahashardy: https://review.openstack.org/#/c/521172/15:19
mwhahahayes15:20
openstackgerritMerged openstack/tripleo-common master: Make some tweaks to swift logging  https://review.openstack.org/49893015:20
*** ukalifon has quit IRC15:21
bnemecmwhahaha: http://blog.nemebean.com/content/taking-advantage-squid-tripleo-development15:21
*** agopi|out has joined #tripleo15:21
EmilienMmwhahaha, slagle: we have OVB job almost ready to deploy a containerized undercloud; what do you think if we move forward now with https://review.openstack.org/#/c/542906/ so we also have a multinode job. So by rocky-m1, we would have one OVB and one multinode with the containerized u/c.15:23
EmilienMnote: the OVB job is here: https://review.openstack.org/#/c/542556/ (waiting on a promotion to have latest openstackclient15:23
*** morazi has quit IRC15:24
openstackgerritSteven Hardy proposed openstack/python-tripleoclient master: Don't exit on a RUNNING message for deploy workflow  https://review.openstack.org/55735615:25
*** jfrancoa has quit IRC15:25
openstackgerritJiri Stransky proposed openstack/python-tripleoclient master: Keep plan environment on upgrade converge  https://review.openstack.org/55737115:27
*** aufi has quit IRC15:29
*** leanderthal has quit IRC15:29
*** etingof has joined #tripleo15:30
jistrchem, marios, matbu ^ here's a possible fix for the converge issue. I just broke my env by not having it :D so need to revert a bit and retest.15:30
jistri got the classic resources.Networks: Resource DELETE failed: Conflict: resources.TenantNetwork.resources.TenantSubnet: Unable to complete operation on subnet 0850a855-48bd-4a9d-ba9e-8488c2618214: One or more ports have an IP allocation from this subnet.15:30
*** morazi has joined #tripleo15:30
mariosjistr: thanks, will check it out tomorrow15:30
jistrack thanks15:30
*** avivgta has quit IRC15:31
openstackgerritChristian Schwede proposed openstack/tripleo-common stable/queens: Fix missing permissions on Swift rebalance playbook  https://review.openstack.org/55737215:31
*** pradk has quit IRC15:32
jistrmarios: btw thanks a lot for the patch link earlier, indeed bumping tripleo-common to latest fixed the issue15:32
mariosjistr: cool thanks for letting me know, i just passed on info from matbu anyway15:33
mariosjistr: have a good evening o//15:33
jistrmarios: thanks you too15:34
slagleEmilienM: undercloud-containers has been voting for a while hasn't it?15:41
slagleEmilienM: if so, i'm ok now with moving containers-multinode if you feel it's stable15:41
openstackgerritMerged openstack/tripleo-heat-templates master: Do not create NetworkVlanID is the value is not defined  https://review.openstack.org/55487215:41
openstackgerritMerged openstack/tripleo-heat-templates master: Remove the lacp bond mode constraints  https://review.openstack.org/55484615:41
EmilienMslagle: yes it's voting for some weeks now and yes, I think the fs010 is stable now, as I'm doing recheck almost every day and keep passing well. OVB isn't stable yet but for other reasons that are under investigation and not related to containerized u/c15:42
jankiEmilienM, Hey. update - snat value is changing in the ODL config.15:42
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: Test zuul layout for RDO CI  https://review.openstack.org/55738015:48
EmilienMjanki: weird, so it's really this log parameter that doesn't work.15:49
jankiEmilienM, yes. and only in HA15:49
openstackgerritMerged openstack/instack-undercloud stable/queens: Use the new dnsmasq PXE filter in ironic-inspector  https://review.openstack.org/55667715:54
openstackgerritMerged openstack/python-tripleoclient master: Updated from global requirements  https://review.openstack.org/55708215:54
*** itlinux has joined #tripleo15:54
*** dpawar has quit IRC15:55
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras  https://review.openstack.org/47260715:55
*** fpan has joined #tripleo15:56
*** etingof has quit IRC16:01
ykarelmwhahaha, i remember now the patch for fs20 containerized was required for queens,16:01
mwhahahaykarel: yea we merged the cherry-pick for the default thing later. but still shouldn't have been required16:02
mwhahahahttps://review.openstack.org/#/c/550965/16:02
mwhahahathat merged 2 weeks ago16:02
ykarelokk if cherry-picked than not required16:02
mwhahahaweshay: to answer you question, yes that included neutron16:02
weshayk.. and the logs are going to kill us here16:03
weshaypinche logs16:03
*** florianf has quit IRC16:03
mwhahahaso i wonder if we're having more of a penalty of the neutron services under containerization because they are heavy on the process launching (ie neutron-rootwrap)16:04
mwhahahaweshay: we could try running an fs020 on baremetal and see if it's timing out16:04
mwhahahaie -e environments/baremetal-services.yaml16:04
ykarelyes that could be as rootwrap was taking time16:04
mwhahahafor giggles16:04
*** fragatin_ has quit IRC16:05
weshaymwhahaha, sure.. I can line that up16:06
EmilienMjanki: are you using the same environment file that load the parameter right?16:06
mwhahahaweshay: or we could just try uncontainerizing neutron16:06
jankiEmilienM, yes16:06
*** ebarrera has quit IRC16:07
*** agurenko has quit IRC16:07
*** bogdando has quit IRC16:07
weshaymwhahaha, let's create a patch that uncontainerizes neutron and run fs20 via a test patch16:07
jankiEmilienM, and I have re-re-rechecked all spellings everywhere16:08
*** florianf has joined #tripleo16:08
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: Define overcloud_docker_registry_namespace for rdocloud  https://review.openstack.org/55703516:08
mwhahahaweshay: k let me try that. what's the bug we're using for all this stuff?16:08
*** marios has quit IRC16:08
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: CI test - never merge  https://review.openstack.org/55742316:09
*** ooolpbot has joined #tripleo16:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION16:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/175755616:10
openstackLaunchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged]16:10
*** ooolpbot has quit IRC16:10
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: DNM: Test uncontainerizing neutron services  https://review.openstack.org/55742416:11
mwhahahaweshay: -^16:11
*** lucasagomes is now known as lucas-afk16:13
mcorneachem: looks like we caught a valid ffu bug in https://logs.rdoproject.org/28/557328/1/openstack-check/gate-tripleo-ci-centos-7-multinode-1ctlr-featureset036-oc-ffu-queens-nv/Zfd50b33301c347f3ba3813f3e824bfdf/undercloud/home/jenkins/ffu_upgrade_steps_playbook.log.txt.gz16:13
*** dparkes has joined #tripleo16:17
ykarelmwhahaha, will this run fs20?16:17
weshayykarel, no16:17
mwhahahai'm attempting a local reproducer with it16:17
mcorneachem: now I am wondering where the fluentd service comes from16:17
weshaywe have to go make that happen now.. I don't think fs20 is a check job16:17
weshayI'm going to cheat16:17
mwhahahait takes too long to be a check job16:18
*** holser__ has quit IRC16:18
chemmcornea: yeah, I've got a valid patch for this16:18
chemmcornea: currently merging https://review.openstack.org/#/c/554831/16:19
chemmcornea: but yeah your question is still valid16:19
openstackgerritChristian Schwede proposed openstack/tripleo-common master: Fix parameter indentation on Swift rebalance playbook  https://review.openstack.org/55742516:20
weshayykarel, ah.. I think we have it in check experimental for rdo sf16:21
mcorneachem: I see there's no ci/environments/scenario001-multinode.yaml in https://github.com/openstack/tripleo-heat-templates/tree/stable/queens/ci/environments16:21
mcorneachem: only environments/scenario001-multinode-containers.yaml which does contain the fluentd service16:21
ykarelweshay, if its there we should run it16:22
rookweshay if it is easier for me to use someones RDO cloud account I can play there...16:22
chemmcornea: yeah it's automatically converted at some point from non-container to container16:22
rookvs trying to reproduce this on my hypervisor.16:22
chemmcornea: and yeah must have the fluentd service in dest but not in source (which may be valide use case)16:23
weshaybah.. it was removed16:23
weshayhttps://review.rdoproject.org/jenkins/job/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/16:23
chemmcornea: and consequently (if my memory about the ansible tasks playbook is correct) would lead to have a lot of upgrade tasks rewriten16:23
mcorneachem: yeah but fluentd is an optional service and we shouldn't enable optional services during ffu16:24
*** ihrachys has joined #tripleo16:25
mcorneachem: unless it's a requirement as it's the case with panko which is enabled by default16:25
*** florianf has quit IRC16:26
*** morazi has quit IRC16:26
chemmcornea: it was faster to fix fluentd16:26
weshayykarel, fyi https://review.rdoproject.org/r/1312616:26
chem"fix"16:26
ykarelweshay, ack16:27
chemmcornea: so new services can appear during upgrade though, making those upgrade task flawed16:27
chemmcornea: so even if not required for ffu it was the right thing to do16:27
*** haleyb has joined #tripleo16:27
mcorneachem: yeah, just thinking out loud why I didn't see this in my test16:28
weshayrook, this box is killing me.. virt-customize works fine by hand16:28
weshayrook, can we reboot this thing16:29
*** jpich has quit IRC16:29
hjensasshardy: Any pointers on this: http://paste.openstack.org/show/716297/ ? Can I get the hostname somehow on L43?16:33
hjensasshardy: or, if there is a better|different approach ...16:33
weshaymwhahaha, ok.. man.. we're queued up16:35
mwhahahak16:35
weshaymwhahaha, https://review.rdoproject.org/zuul/16:35
weshayykarel, you see it ^16:35
weshayrook, where you at?16:36
ykarelweshay, Good16:36
openstackgerritMerged openstack/tripleo-common master: Fails to get DPDK feature status.  https://review.openstack.org/55688716:36
openstackgerritMerged openstack/tripleo-heat-templates master: Added the missing attributes for roles_data of ovs-dpdk-agent docker service  https://review.openstack.org/55516616:36
rookweshay: kick it16:37
weshayrook, k.. rebooting16:37
*** khrystop_ has joined #tripleo16:38
*** ihrachys_ has joined #tripleo16:38
*** khrystoph has quit IRC16:38
*** dsariel has quit IRC16:39
*** morazi has joined #tripleo16:39
*** fragatina has joined #tripleo16:40
*** yamahata has joined #tripleo16:40
*** thrash is now known as thrash|biab16:40
*** ihrachys has quit IRC16:41
itlinuxhello guys. has anyone created a role for NOC? can anyone suggest what's best on doing that... I have some old info where I had to modify the policy.json but that was for ocata.. and my understanding is that the RBAC now is going to be changing16:41
*** dparkes has quit IRC16:42
*** dparkes has joined #tripleo16:43
shardyhjensas: Hmm no the service/*.yaml templates are instantiated once per role, not once per node, so in the heat template there's no way to access the hostname - probably the way I'd do it is pass the entire map into puppet-tripleo and look up the hostname there at runtime?16:44
shardyhjensas: the other approach would be to set the entire map as individual keys in config_settings (with some prefix to avoid colliding names) then use hiera interpolation to look up the hostname16:45
*** trown is now known as trown|lunch16:46
shardyhjensas: e.g similar to what we do in https://review.openstack.org/#/c/526692/ to look up net_ip_map entries for a specific network16:47
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: CI test - never merge  https://review.openstack.org/55742316:47
*** alee has joined #tripleo16:47
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: Test zuul layout for RDO-CI  https://review.openstack.org/55738016:48
openstackgerritMerged openstack/tripleo-heat-templates master: remove OS::TripleO::Services::CephClient from CephAll  https://review.openstack.org/55341616:49
*** dsariel has joined #tripleo16:51
hjensasshardy: thanks, I'll look into those suggestions and see if I can come up with something.16:51
*** rbrady is now known as rbrady-afk16:51
*** chlong has quit IRC16:51
EmilienMweshay: I would appreciate some feedback from the CI squad (at least one ack) https://review.openstack.org/#/c/542906/16:52
EmilienMjanki: sorry I was away a bit. I'll keep looking and let you know if I find something.16:53
*** trozet has quit IRC16:53
weshayack16:54
weshayEmilienM, this is looking awesome man!  We should really get ara in the project our logs have gotten worse due to ansible :)16:56
EmilienMweshay: for the steps run by config-download you mean?16:56
EmilienMyes it would be cool for our CI environment16:57
weshayEmilienM, ya for the new undercloud and config-download.. ya16:57
EmilienMI remember we talked about it, let me pull the strings16:57
weshayEmilienM, unformatted ansible logs SUCK16:57
*** pkovar has quit IRC16:57
weshayansible logs generally suck anyway16:57
EmilienMhttps://etherpad.openstack.org/p/tripleo-config-download16:58
EmilienMARA integration. Pending packaging: https://bugzilla.redhat.com/show_bug.cgi?id=142619316:58
openstackbugzilla.redhat.com bug 1426193 in Package Review "Review Request: ara - Ansible Run Analysis, Record and visualize Ansible Playbook runs" [Unspecified,On_qa] - Assigned to misc16:58
weshaydmsimard, get on that ;)16:58
EmilienMlooks like it's on qa by number8016:58
dmsimardThe package is already in rawhide16:58
dmsimardit's approved and everything16:59
number80yeah, it needs to be closed, usually, it happens when update is pushed to stable16:59
*** tosky has quit IRC16:59
weshayEmilienM, do we want featureset010 to work only for master atm?17:00
weshaythis looks awesome, so that's my only question17:00
weshayEmilienM, I thought chem was using fs010 in some upgrade job.. I'll have to look17:00
dmsimardweshay, EmilienM: I'll try and get the ball rolling for ara on centos, no ETA though because Ansible 2.5 broke some things17:01
dmsimardoh and btw, 2.5 should probably be landing in centos extras sooner rather than later so I hope we're ready :/17:01
EmilienMweshay: for now, only fs010 and very soon 001 (ovb-ha)17:01
weshayrook, this box is fooooked17:02
chemEmilienM: hum pike upgrade is fs011 not fs 01017:02
weshayrook, check it out17:02
rookweshay: I have tried.. not sure what tmux you are in17:02
rookugh17:02
weshayrook, there should only be one session17:02
rookyeah, I am in it17:02
rooklol17:02
rookbut you aren't17:02
weshaywha?17:03
rook[root@c08-h09-r930 ~]# tmux ls17:03
rook0: 1 windows (created Wed Mar 28 16:46:39 2018) [203x64]17:03
*** dsariel has quit IRC17:04
weshayrook, oh shit it worked17:04
weshayhot dam17:04
weshaywe're in business17:04
weshayrook, run "yum update libvirt" w/o -y17:04
weshayrook, there are package dep errors17:05
rookare we on the same host?17:05
rooklol17:05
rook[root@c08-h09-r930 tripleo-quickstart]# yum update libvirt17:05
rookLoaded plugins: product-id, search-disabled-repos, subscription-manager17:05
rookThis system is not registered with an entitlement server. You can use subscription-manager to register.17:05
rookNo packages marked for update17:05
weshay10.12.70.1517:05
rookyarp17:06
*** derekh has quit IRC17:07
*** thrash|biab is now known as thrash17:08
jankiEmilienM, sure. thanks and have a nice day17:08
*** janki has quit IRC17:09
*** ooolpbot has joined #tripleo17:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION17:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/175755617:10
openstackLaunchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged]17:10
*** ooolpbot has quit IRC17:10
*** dtantsur is now known as dtantsur|afk17:10
EmilienMchem: ack, then we're good17:10
EmilienMweshay: nice comment I'll fix hat up17:11
EmilienMthat*17:11
openstackgerritMerged openstack/tripleo-heat-templates master: Add purge to Nova cleanup cron.  https://review.openstack.org/55196617:13
weshayEmilienM, ya.. I'm not like that mwhahaha guy.. +2'ing everything I see17:15
* weshay runs17:15
EmilienMlol17:15
mwhahahasomeone has to actually look at code17:15
mwhahahait doesn't merge itself17:15
sdoranEmilienM: Did some digging on Ansible packaging. Our docs were out of date since there has been a good bit of repo shuffling. The short answer is Upstream Ansible will be in EPEL and releases.ansible.com, and RHEL has the Ansible Engine repo. No more extras.17:16
sdoranI updated the docs, but we'll have to either use EPEL or releases.ansible.com to get Ansible for CentOS. The last Ansible in CentOS Extras was 2.4.2 and there won't be any more release in the repo.17:17
EmilienMsdoran: that looks cool17:17
EmilienM(easier to upgrade I imagine)17:17
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Use the same namespace for all docker registries  https://review.openstack.org/55744417:17
sdoranreleases.ansible.com is the fastest way to get new release since things tend to take a while to get through EPEL Testing then into EPEL.17:17
sdoranPlus it has the least amount of extra baggage.17:18
EmilienMok17:18
*** dsariel has joined #tripleo17:19
sdoranThe long term plan on the RHEL side is for the Ansible Engine repo to be available to every Red Hat sub. And they have a repo for each major release, e.g., `rhel-7-server-ansible-2.4-rpms`, making it easy to pin versions.17:20
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: FFU: Add support for panko service upgrade  https://review.openstack.org/53376417:22
sdoranI'll have to get with slagle and see about how we go about managing repos for CentOS since Ansible won't be "magically available" anymore.17:22
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: Deploy container-multinode (fs010) with a containerized undercloud  https://review.openstack.org/54290617:23
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/queens: CI test on queens  https://review.openstack.org/55744617:24
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/pike: CI test on queens  https://review.openstack.org/55744717:24
*** Goneri has quit IRC17:25
slaglesdoran: EmilienM : i'm guessing that might mean we want to package ansible in the rdo deps repos then.17:26
slaglesdoran: EmilienM : b/c we do not want to be in the business of enabling EPEL17:26
openstackgerritMerged openstack/puppet-pacemaker master: Fix up error code path  https://review.openstack.org/55727317:27
openstackgerritMerged openstack/puppet-pacemaker master: Allow meta_params and op_params on resource::ip  https://review.openstack.org/55320617:27
*** mdnadeem_ has quit IRC17:28
*** moshele has joined #tripleo17:30
EmilienMslagle: exactly, since mitaka and now, it has never been a plan to use EPEL in OpenStack, so indeed if Ansible is taken from EPEL we'll have to duplicate the effort and package it in RDO (and probably gate on every change that is made); if it's taken from releases.ansible.com then we might need consider pulling it directly but again we need to find a way to not break our CI every single day and have proper17:31
EmilienMtesting17:31
mwhahahawe have testing?17:33
*** fpan has quit IRC17:33
slaglewe tests for tests and the former always fail.17:34
mwhahahawe should probably align with how we plan on distributing ansible downstream as well17:34
mwhahahaso we don't do something in RDO that won't actually replicate what we do elsewhere and vice versa17:34
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/ocata: CI test on queens  https://review.openstack.org/55745117:35
mwhahahait seems like it's similar to the ceph bits17:35
mwhahahawhere we need to rely on an external part17:35
*** fragatina has quit IRC17:35
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/newton: CI test on queens  https://review.openstack.org/55745217:35
mwhahahamaybe we could just automate a version bump somewhere?17:35
*** shardy has quit IRC17:35
*** tesseract has quit IRC17:35
mwhahahathey do appear to have a preview and anightly set of rpms17:35
EmilienMmwhahaha: what would be cool imho is 1) import the package in RDO 2) do automatic version bumps 3) for each bump, we run CI jobs in RDO CI before (with some tripleo scenarios)17:36
EmilienM1) probably means to import ansible into delorean17:37
*** dparkes has quit IRC17:38
*** yamahata has quit IRC17:38
*** Goneri has joined #tripleo17:40
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: FFU: Add support for panko service upgrade  https://review.openstack.org/53376417:41
*** trozet has joined #tripleo17:45
*** fpan has joined #tripleo17:48
openstackgerritRafael Folco proposed openstack/tripleo-quickstart-extras master: Collect dstat on overcloud nodes  https://review.openstack.org/55745517:48
EmilienMslagle, mwhahaha, sdoran : I'll keep the discussion on the ML re: ansible packaging, please keep posted, it would be cool to move things forward during Rocky!17:48
*** moshele has quit IRC17:49
*** khyr0n has joined #tripleo17:51
*** psachin has joined #tripleo17:53
openstackgerritMerged openstack/tripleo-quickstart-extras master: Use openstack commands in overcloud-deploy.sh  https://review.openstack.org/52934717:54
*** gkadam has quit IRC17:54
openstackgerritMarius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade  https://review.openstack.org/55668717:54
*** gfidente has quit IRC17:55
slagleEmilienM: what discussion on the ML? where is it?17:55
*** salmankhan has quit IRC17:55
*** trown|lunch is now known as trown17:56
EmilienMslagle: there is none now, but I plan to kick it off today17:56
slagleoh. sorry, was confused17:56
EmilienMnp17:57
EmilienMif we could have a standard path because ansible isn't alone here, like we said there is OVS, Ceph and many other things17:57
*** khyr0n has quit IRC17:58
*** panda is now known as panda|off17:59
openstackgerritMarius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade  https://review.openstack.org/55668718:00
*** itlinux has quit IRC18:00
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: CI test - never merge  https://review.openstack.org/55742318:03
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: Test zuul layout for RDO-CI  https://review.openstack.org/55738018:03
openstackgerritMarius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade  https://review.openstack.org/55668718:04
*** lvdombrkr has joined #tripleo18:05
*** dsariel has quit IRC18:06
openstackgerritMarius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade  https://review.openstack.org/55668718:07
openstackgerritMarius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade  https://review.openstack.org/55668718:09
*** ooolpbot has joined #tripleo18:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION18:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/175755618:10
*** ooolpbot has quit IRC18:10
openstackLaunchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged]18:10
*** liverpooler has quit IRC18:11
openstackgerritHarald Jensås proposed openstack/puppet-tripleo master: Ironic Inspector - disjoint ip range(s) for HA  https://review.openstack.org/55745918:12
*** pblaho has quit IRC18:12
*** mcornea has quit IRC18:14
*** itlinux has joined #tripleo18:15
*** yamahata has joined #tripleo18:16
*** psachin has quit IRC18:16
weshayrook, can you join my bluejeans18:17
*** athomas has quit IRC18:17
weshaytalking perf tools18:17
openstackgerritMarius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade  https://review.openstack.org/55668718:18
rooklink to join the bj party18:18
sshnaidmEmilienM, is there an easy way to run service in overcloud nodes at start of overcloud deployment?18:20
EmilienMsshnaidm: write the service in tripleo-heat-templates/docker/services/18:20
EmilienMwhat service is it?18:20
*** abishop_ has joined #tripleo18:20
mwhahahaEmilienM: he probably wants to run dstat or something18:21
sshnaidmEmilienM, dstat for example.. not sure if it should run in docker, for measuring host performance18:21
mwhahahaso it's not exactly a service18:21
mwhahahait'd be more of a host prep or pre-deploy thing18:21
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: CI test - never merge  https://review.openstack.org/55746018:21
rookweshay: I have a interview in 10 minutes18:21
mwhahahayou could either hack it into the image modification part for CI18:21
rookshoot me the link if ya want me there homie.18:21
mwhahaharook: good luck on your new job? :D18:22
rookthanks mwhahaha18:22
*** lvdombrkr has quit IRC18:22
weshayrook, bah.. k.. don't worry bout it18:22
rookweshay what? too long of a convo18:22
rookI like perf tools18:22
rooklets do this18:22
weshayok18:22
mwhahahai like to listen about perf tools, can i join18:22
EmilienMsshnaidm: is it just a CI thing?18:22
weshayhttps://bluejeans.com/u/whayutin/18:23
*** abishop has quit IRC18:23
sshnaidmEmilienM, hmm.. I think it could be useful for any ops too, but currently for CI, yes18:24
openstackgerritChandan Kumar proposed openstack/tripleo-quickstart-extras master: Refactored validate-tempest role for undercloud and containers  https://review.openstack.org/55144118:24
EmilienMsshnaidm: if it's useful outside of ci then create a service in THT18:24
ykarelmwhahaha, weshay tempest run results on local setup with non containerized neutron services: https://review.rdoproject.org/paste/show/93/18:24
ykarelAnd i checked l3 agent logs and it was taking around 3 seconds for doing router operations18:25
sshnaidmEmilienM, where do we have docs/guidelines how to do it?18:26
EmilienMsshnaidm: you probably want to look at existing services in extraconfig/services/18:26
EmilienMand just create a service that runs host_prep_tasks18:27
EmilienMit's ansible driven18:27
openstackgerritChandan Kumar proposed openstack/tripleo-quickstart master: Run tempest on fs027 using containerized tempest  https://review.openstack.org/55144218:27
*** fragatina has joined #tripleo18:27
*** tbonds has quit IRC18:28
openstackgerritHarald Jensås proposed openstack/tripleo-heat-templates master: Ironic Inspector - disjoint ip range(s) for HA  https://review.openstack.org/55746518:29
*** mcornea has joined #tripleo18:30
openstackgerritTim Rozet proposed openstack/puppet-tripleo master: Removes neutron ownership of certs  https://review.openstack.org/55746918:30
openstackgerritMarius Cornea proposed openstack/tripleo-upgrade master: WIP: Add support for per role FFU upgrade  https://review.openstack.org/55668718:30
openstackgerritTim Rozet proposed openstack/tripleo-heat-templates master: Fixes Neutron certificate and key permissions  https://review.openstack.org/55667318:33
weshayykarel++18:34
weshayykarel, nice man18:34
mwhahahaso it sounds like we could maybe try with neutron containerized but l3 agent not?18:34
*** morazi has quit IRC18:34
mwhahahaand see if it's a similar number18:34
ykarelmwhahaha, need some extra work for that18:35
ykareljust non containerized l3 failed for me18:35
mwhahahaorly18:35
*** rbrady-afk is now known as rbrady18:35
ykarelmwhahaha, https://review.rdoproject.org/paste/show/94/18:35
mwhahahaoh that's interesting18:36
mwhahahabeagles: -^18:36
*** gyankum has quit IRC18:36
*** morazi has joined #tripleo18:36
*** gyankum has joined #tripleo18:40
itlinuxhello all, I am getting close to production, and I wonder if anyone has the policy.json to adopt a role with read only for our NOC!18:41
mwhahahaweshay: atop is the other app i've used in the past for recording process interactions over time, https://www.atoptool.nl/ we might think of that instead of dstat if there's a package available18:44
mwhahahaweshay: that being said I think configuring collectd on the overcloud nodes to send to the undercloud might make more sense for the tempest runs18:44
weshayaye18:45
*** lblanchard has joined #tripleo18:47
*** lblanchard has quit IRC18:47
weshaymwhahaha, do you want access to ykarel deployment w/o containerized neutron?18:47
weshayack re: collectd18:47
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: CI test - DNM  https://review.openstack.org/55747418:48
*** gyankum has quit IRC18:48
mwhahahaweshay: no i'm ok i'm spinning mine up still18:48
weshayk18:48
openstackgerritSagi Shnaidman proposed openstack/tripleo-heat-templates master: WIP: running dstat in overcloud nodes  https://review.openstack.org/55747618:50
sshnaidmEmilienM, something like that? ^^18:50
EmilienMlooking18:51
EmilienMsshnaidm: yeah and you can even live test it by adding the service to the ci environment18:52
EmilienMsshnaidm: you'll need to add the service in the catalog, grep for OS::TripleO::Services::Rhsm and add OS::TripleO::Services::Dstat maybe18:52
*** moshele has joined #tripleo18:54
*** moshele has quit IRC18:56
sshnaidmEmilienM, is catalog here (in environments directory)? https://github.com/openstack/tripleo-heat-templates/blob/6f2204fc245dfebeff986f877daa55c7c8ff8ad1/environments/rhsm.yaml18:57
openstackgerritCarlos Goncalves proposed openstack/tripleo-common master: Move health_manager cfg section to octavia_worker  https://review.openstack.org/55748318:59
EmilienMsshnaidm: no, you'll have to patch overcloud-resource-registry-puppet.j2.yaml and roles_data.yaml probably, look for existing services18:59
EmilienMsshnaidm: also create an environments/dstat.yaml probably19:00
*** sanjayu has joined #tripleo19:02
*** gkadam has joined #tripleo19:06
sshnaidmEmilienM, "rhsm" also appears in all files in roles/*.yaml, should I add it there too?19:07
EmilienMsshnaidm: you can add it on the roles you think the service should be running19:08
EmilienMmwhahaha might have thoughts on that19:09
*** ooolpbot has joined #tripleo19:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION19:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/175755619:10
openstackLaunchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged]19:10
*** ooolpbot has quit IRC19:10
mwhahahaehhh i'm not sure this is the best thing to configure as a service19:10
mwhahahai think i'd rather spend time to get collectd configured correctly19:10
*** atoth has quit IRC19:10
*** yamahata has quit IRC19:11
*** yamahata has joined #tripleo19:11
slagleweshay: is there a place to see ovb jobs here? http://cistatus.tripleo.org/19:13
slagleones from the check queue19:13
sshnaidmslagle, http://cistatus.tripleo.org/promotion/19:13
sshnaidmslagle, and look for "RDO cloud OVB jobs"19:14
sshnaidmmwhahaha, I suspect it will require big puppet config, right?19:14
mwhahahasshnaidm: which will? collectd? we supposedly ship that already19:15
EmilienMslagle, sdoran: I just the email re: packaging/ansible https://lists.rdoproject.org/pipermail/dev/2018-March/008645.html19:15
sdoran<ack>19:15
sshnaidmmwhahaha, what needs to be done to run it then?19:15
mwhahahasshnaidm: ¯\_(ツ)_/¯19:16
mwhahahathe key would be getting the collector going on the undercloud. that i don't think we currently do19:16
slaglesshnaidm: thanks!19:16
mwhahahahttps://github.com/openstack/tripleo-heat-templates/blob/master/environments/collectd-environment.yaml19:16
sshnaidmmwhahaha, sorry, I don't understand you19:17
sshnaidmmwhahaha, we want to run performance tool on overcloud when it's being deployed19:18
mwhahahai'm aware at you what you're doing19:18
sshnaidmespecially in OVB case19:18
mwhahahai'm saying we already ship a tool that could be used19:18
mwhahahaand dstat doesn't give us decent enough data to actually do anything with19:18
mwhahahait's ok for the basics but doesn't really give us info19:19
EmilienMhave we reached out MetMon team?19:19
mwhahahamy personal preference would be atop (but we don't have it in rdo), collectd we already ship so we'll miss some of the early io but i think it can be setup in like step119:20
mwhahahathe issue with collectd is that it ships the info to a central server so we have to do that configuration on the undercloud (or somewhere)19:20
mwhahahaalso it would be a really good idea to actually test the stuff we ship (novel concept)19:21
rookI agree mwhahaha -- I need to look at the shipping version of collectd we have now...19:22
rookdo we have a graphite/carbon server we can send data to right now?19:23
mwhahahai think rdocloud has one19:23
EmilienMmwhahaha: I updated https://storyboard.openstack.org/#!/story/2001546 for tripleo and assigned to me and set In Progress19:25
sshnaidmmwhahaha, what does atop provide that dstat can't provide?19:25
*** pblaho has joined #tripleo19:25
mwhahahasshnaidm: a file that can be used to replay the events19:25
EmilienMimho, we should reach out the metmon folks via ML, they own these features...19:26
mwhahahaso i can pull the atop file down, load it up and get a whole bunch more detail than dstat provides19:26
mwhahahaprocess/network/io/etc19:26
mwhahahaat a configurable interval19:26
mwhahahaso i could go to 5 mins into the deploy and see exactly what was using cpu19:26
mwhahahaand step forward in chunks19:26
mwhahahahttps://www.atoptool.nl/19:27
openstackgerritHarald Jensås proposed openstack/puppet-tripleo master: Ironic Inspector - disjoint ip range(s) for HA  https://review.openstack.org/55745919:27
mwhahahaanyway, might as well reuse the collectd stuff we already have19:27
*** dprince has quit IRC19:27
sshnaidmmwhahaha, I don't see a problem to run atop instead of dstat in my patch19:28
mwhahahaatop isn't available via rdo19:28
mwhahahai think it's in epel19:28
mwhahahaand we don't want that19:28
sshnaidmmwhahaha, didn't you say want it? I'm confused19:29
mwhahahawe don't want epel19:29
mwhahahahttps://centos.pkgs.org/7/epel-x86_64/atop-2.3.0-8.el7.x86_64.rpm.html19:30
*** tbonds has joined #tripleo19:30
sshnaidmwell, we can check with #rdo if it's possible to ship it, but if it's not there it couldn't be used right now19:31
mwhahaharook: there are other outputs than graphite right?19:31
*** waleedm has joined #tripleo19:31
rookmwhahaha: our stack is, collectd -> graphite -> Grafana19:32
rookwell collectd -> graphite -> carbon -> graphite -> grafana19:32
rookto be explicit.19:32
mwhahahaif we want to do it itteratively, can we do something else beside graphite for consumption19:32
mwhahahabecause realistically we just want to output to the logs/ folder somehow19:32
mwhahahanot necessarily to a central place in the beginning19:33
rookI have never used it in that fashion, so I don't know.19:34
rookI don't see why you would want to do that.19:34
sshnaidmweshay, well, configuring collectd in tht seems like something requires a big deep dive and N days of effort, not sure it worth it, I can continue with dstat, at least we'll have something in CI for now19:34
rookwell, we have collectd + ansible playbooks to configure it today...19:35
rookI literally just ran the playbook against my OOOQ cloud19:35
rookyeah, if you want to reinvent like we did, days.. for sure.19:35
weshaysshnaidm, please leave it for now19:36
weshaysshnaidm, focus on upgrades19:36
mwhahahaso interesting note, neutron-rootwrap isn't high on the cpu list with neutron not containerized19:37
* mwhahaha wonders19:37
openstackgerritSagi Shnaidman proposed openstack/tripleo-heat-templates master: WIP: running dstat in overcloud nodes  https://review.openstack.org/55747619:38
rookhttps://snapshot.raintank.io/dashboard/snapshot/t9Y1DJU0l7eluWCDhPS2cCx2B3qUxWID?orgId=2 EmilienM mwhahaha , weshay hooked me up with a working patch of your work. We can see less http processes -- i mean, still a ton, but less . Tempest is running now. I will run some performance tests against neutron once it completes.19:38
EmilienMmwhahaha: you have numbers? link?19:38
mwhahahano i don't have numbers cause it never shows up in any of our stuff19:39
rookmwhahaha what about the agent?19:39
mwhahahai'm watching the tempest run19:39
rookovs-agent19:39
mwhahahaneutron-server is high19:39
mwhahahabut that's it19:39
mwhahahaand i assume that's always been high19:39
mwhahahaso i bet it's the rootwrap stuff in containers takes a huge perf hit19:40
openstackgerritHarald Jensås proposed openstack/puppet-tripleo master: Ironic Inspector - disjoint ip range(s) for HA  https://review.openstack.org/55745919:42
EmilienMon another topic I don't think we use mox3 in tripleo, I checked tripleo-common, tripleoclient and os-*-config19:43
EmilienMbnemec: any thoughts? ^ I think i'm going to close the goal for tripleo19:43
rookhm mwhahaha i wonder if we try switching the rootwrap mode?19:43
rookhttps://specs.openstack.org/openstack/neutron-specs/specs/kilo/rootwrap-daemon-mode.html19:43
*** asbishop has joined #tripleo19:44
mwhahaharook: i think we're using daemon already19:44
mwhahaharook: or maybe we werent' for ovs19:44
mwhahahai see daemons19:44
*** abishop_ has quit IRC19:46
openstackgerritEmilien Macchi proposed openstack/os-net-config master: Remove mox from test-requirements.txt  https://review.openstack.org/55749319:46
rooki wonder if ajo could comment on the rootwrap stuff19:47
ajoRook which stuff? :)19:48
rookajo rootwrap in containers bad... rootwrap outof containers, not bad.19:48
rookhigh CPU utilization in containers.19:48
*** yamahata has quit IRC19:48
ajoHmmm19:50
ajoWhy is that?19:50
mwhahahai believe that is the question19:50
ajoI noticed high cpu utilization in the ovs agent.19:50
ajoCalling epoll like crazy19:50
rookme too19:50
mwhahahathe ovs agent uses rootwrap right? is it not using it in a daemon mode?19:51
eck`ajo: interesting i've been looking at epoll stuff today...19:52
eck`just submitted... https://review.openstack.org/#/c/557487/19:52
eck`which is similar to a thing i fixed a while ago in oslo.messaging19:53
ajoHmmm let me read19:53
eck`where it spams epoll19:53
eck`(that patch is for mistral)19:53
eck`but we probably have similar issues elsewhere19:53
ajomwhahaha: by default uses root daemon afaik19:53
mwhahahak19:53
eck`right now i'm looking at ironic, it seems to have a similar problem19:53
rookyeah, i don't see anything int he config stating other wise19:53
rookso it must be the default, daemon19:53
eck`i am just tired of my poor cpus burning to pieces for no good reason, that is why i am looking into this19:53
eck`i have reached my breaking point19:53
*** ledo_ has joined #tripleo19:55
rookeck`: which ironic service?19:55
*** waleedm has quit IRC19:56
eck`rook: both conductor and inspector19:56
eck`i found one instance of my known issue that i am submitting a fix for in literally 30 seconds...19:57
eck`but it doesn't seem to help the symptom19:57
eck`so something else is going on19:57
ajoeck`: probably related. I also suspected of event let too19:57
*** ykarel is now known as ykarel|away19:57
eck`(but my patch would still be an improvement so i'll throw it up)19:57
eck`eventlet is the devil19:57
ajoI can't find my bugreport19:57
ajoWhere was it...19:57
bnemecEmilienM: The only one I was aware of was os-net-config, and I believe we already merged a patch to remove it from there.19:57
bnemecThat makes sense.  Most of our python code was written after the switch to mock.19:58
bnemecWe weren't even using mox in os-net-config, just the stubout module of mox.19:58
rookwell ajo mwhahaha i take that back... looking at neutron currently the ovs-agent isn't pegging the CPU.. just neutron-server19:58
eck`hm ironic uses threading.Lock19:58
eck`which probably has the same problem under the cover19:58
eck`because it ultimately uses a Condition19:58
*** tosky has joined #tripleo19:58
EmilienMbnemec: good news!20:01
*** ansmith has quit IRC20:03
*** tosky has quit IRC20:05
*** ykarel|away has quit IRC20:06
eck`nevermind i have that backwards, conditions use locks20:07
*** tosky has joined #tripleo20:09
eck`neutron at a glance appears to only use threading.Lock as well...20:10
* eck` wonders if eventlet is doing something silly with that as well20:10
*** ooolpbot has joined #tripleo20:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION20:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/175755620:10
*** ooolpbot has quit IRC20:10
openstackLaunchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged]20:10
*** fpan has quit IRC20:10
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: Collect dstat on overcloud nodes  https://review.openstack.org/55745520:10
*** moshele has joined #tripleo20:11
ajoeck`: saving your review to the future me, :)20:11
EmilienMbnemec: why did we abandon https://review.openstack.org/#/c/532354/ ?20:11
ajoIt's probably related to what I saw20:11
EmilienMI should probably ask Sean20:11
openstackgerritHarald Jensås proposed openstack/puppet-tripleo master: Ironic Inspector - disjoint ip range(s) for HA  https://review.openstack.org/55745920:12
bnemecEmilienM: Because at the time it was still in use.20:12
*** gkadam has quit IRC20:12
eck`ajo: fyi https://review.openstack.org/#/c/557500/ is the ironic version i just posted20:13
eck`same thing basically20:13
bnemecEmilienM: I could have sworn I reviewed a patch that removed it, but I don't see it so maybe I'm nuts. :-/20:13
eck`threading.Event i know for a fact is evil20:13
eck`so those ones are easy to just patch out20:13
ajoThanks eck`20:13
ajoEventlet is evil, I agree20:13
bnemecEmilienM: Oh, I looked but didn't review: https://review.openstack.org/#/c/550988/20:13
ajoSo is the python GIL :P20:14
rooklol20:14
eck`and i see that eventlet patches out _allocate_lock, so i am suspicious of that20:14
eck`hate.20:16
mwhahahaweshay: so i still got a timeout on tempest in my fs020 recreate locally20:21
mwhahahaweshay: with uncontainerized neutron20:21
*** tbonds has quit IRC20:21
weshaymwhahaha, well.. this one didn't time out :) https://review.rdoproject.org/jenkins/job/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/9/console20:23
mwhahahaweshay: maybe recreate script has different timeout values?20:23
* mwhahaha shrugs20:23
mwhahahainteresting20:23
weshaymwhahaha, https://review.openstack.org/#/c/555251/20:24
weshayoh. .that merged20:24
mwhahahaoh well i didn't specify -t20:24
mwhahahaso yea that's the problem20:24
mwhahahai need a -t 30020:24
mwhahahaso yea20:24
weshayI love yatin20:25
mwhahahai'll try containerizing somethings next20:25
weshayI express my love publically20:25
* rook is waiting for his tempest to fail20:26
rookthe thought is that it is something with HA20:27
rookI wonder if spawning keepalived20:27
rookis some of the issue?20:27
*** karthiks has quit IRC20:27
* rook waits to see20:27
*** paramite has joined #tripleo20:28
*** morazi has quit IRC20:32
EmilienMbnemec: thank you20:35
*** eck` is now known as eck`gone20:38
*** karthiks has joined #tripleo20:40
*** dparkes has joined #tripleo20:40
*** pchavva has quit IRC20:41
openstackgerritAlex Schultz proposed openstack/tripleo-quickstart master: Test containerized undercloud with TLS  https://review.openstack.org/55727620:42
weshayhrm... ok mwhahaha shiat.. same results https://logs.rdoproject.org/24/557424/1/openstack-experimental/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/Z5a2df7deb1104f2399ad2daeb489de22/tempest.html.gz20:44
weshaymade no diff20:44
*** morazi has joined #tripleo20:44
weshayrfolco|ruck, you have the tempest bugs logged?20:44
mwhahahaweshay: so thsoe seem legit failures20:44
*** moshele has quit IRC20:44
mwhahahaweshay: but it actually succeeded in 3:4520:45
mwhahahathat's what i thought wasn't currently happening20:45
rfolco|ruckweshay, https://bugs.launchpad.net/tripleo/+bug/175958320:45
openstackLaunchpad bug 1759583 in tripleo "fs020 full tempest failures" [High,Triaged]20:45
weshayrfolco|ruck, ok.. let's work on that bug20:45
weshayrfolco|ruck, join in a bluejeans20:45
rfolco|ruckweshay, https://review.openstack.org/#/c/557354/20:46
*** tosky has quit IRC20:50
*** raildo has quit IRC20:51
mwhahahais it supposed to be concurrency 3 or 220:52
openstackgerritHarald Jensås proposed openstack/tripleo-heat-templates master: Ironic Inspector - disjoint ip range(s) for HA  https://review.openstack.org/55746520:52
mwhahahafor tempest20:52
*** dsariel has joined #tripleo20:52
rookweshay: mwhahaha the task that has been in question is tempest.scenario.test_network_basic_ops.TestNetworkBasicOps20:52
rookmwhahaha: i thought i heard 220:52
mwhahahamy recreate has 3 for some reason20:53
mwhahaharook: so that's the one we get ssh timeouts for in ovb and other CI jobs20:53
mwhahahain fs020, we're getting job timeouts trying to run tempest20:53
mwhahahabut there are some failing tests now20:53
mwhahahalooks like a 50320:54
mwhahahaso i wonder if we maybe need to tune some haproxy configs or something20:54
*** tosky has joined #tripleo20:54
rook503 to the client?20:54
mwhahahayea20:54
mwhahahahttps://logs.rdoproject.org/24/557424/1/openstack-experimental/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/Z5a2df7deb1104f2399ad2daeb489de22/tempest.html.gz20:54
mwhahahathat's w/o containerized neutron20:55
mwhahahaso it runs and completes in a decent time20:55
mwhahahabut we have fails in some api compute items20:55
mwhahahai'm trying to just rerun only one of those failed test groups to see if i can reproduce the problem20:55
rooki wonder if there is anything in the keystone log?20:55
rookack20:56
mwhahaha503 can come from anywhere :D though it's more likely to come from haproxy i think20:56
rookmwhahaha typically i have seen 503s due to not getting a token20:56
rookbut, yeah, anywhere.20:56
rook:/20:56
*** yamahata has joined #tripleo20:58
mwhahahaof course 503s tend not to get logged correctly anyway20:58
mwhahahai wished we logged haproxy too20:59
rooki thought I had a patch for that21:01
rooki am sure i did21:01
mwhahahaand it probably got lost in containers21:01
mwhahahacause redoing the work is always fun21:01
openstackgerritMerged openstack/tripleo-quickstart-extras master: Remove compatibility cruft from overcloud-validate  https://review.openstack.org/53732721:02
openstackgerritMerged openstack/tripleo-common master: Remove PackageUpdateManager  https://review.openstack.org/53638521:02
mwhahahaso yea if i rerun only those failed test groups it works fine21:02
mwhahahaso there's a concurrency thing going on somewhere21:02
rookhttps://review.openstack.org/#/c/497524/21:02
rookshit21:02
mwhahahayea you're logging, but it doesn't look like it's getting outputted (or captured)21:03
*** yamahata has quit IRC21:03
mwhahahaowalsh: nova metadata is http protocol right?21:05
rookcheck the swift log21:05
rooklol21:05
mwhahahapffft swift21:05
owalshmwhahaha: you mean vs https?21:06
mwhahahaowalsh: i mean vs tcp21:07
mwhahahaowalsh: https://logs.rdoproject.org/24/557424/1/openstack-experimental/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/Z5a2df7deb1104f2399ad2daeb489de22/overcloud-controller-0/var/log/extra/docker/containers/haproxy/stdout.log.txt.gz21:07
mwhahahaowalsh: it's missing mode http21:07
mwhahahaso is panko21:07
rooki wish i was kiding21:07
rookbut i think haproxy for some odd reason, logs to the swift logs.21:07
rooki also think they are in /var/log/messages21:07
owalshmwhahaha: yea, it's http21:07
mwhahaharook: oh it's the faciliity mapping21:07
mwhahahaowalsh: ok we're missing the mode http in the haproxy config for it21:08
*** moshele has joined #tripleo21:08
*** trown is now known as trown|outtypewww21:08
*** paramite has quit IRC21:08
mwhahaharook: ok well no 503 from the swift log21:09
mwhahahabut i don't see haproxy in there21:09
mwhahahahttps://logs.rdoproject.org/24/557424/1/openstack-experimental/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/Z5a2df7deb1104f2399ad2daeb489de22/overcloud-controller-0/var/log/swift/swift.log.txt.gz21:10
mwhahahadid you mean that log?21:10
*** ooolpbot has joined #tripleo21:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION21:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/175755621:10
*** ooolpbot has quit IRC21:10
openstackLaunchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged]21:10
*** gunix has quit IRC21:10
owalshmwhahaha: ok, got an LP for it? I'll take a look tomorrow if I can21:12
mwhahahaowalsh: i've got it. thanks for the info21:12
mwhahahai'll add you to the lp21:12
mwhahahaer review21:12
owalshack, thanks21:12
openstackgerritAlex Schultz proposed openstack/puppet-tripleo master: Fix nova metadata and panko mode  https://review.openstack.org/55751121:13
*** gunix has joined #tripleo21:13
*** asbishop has quit IRC21:13
openstackgerritMichele Baldessari proposed openstack/puppet-pacemaker master: WIP DNR initial sketches for proper resource change support  https://review.openstack.org/55751221:13
*** itlinux has quit IRC21:14
mwhahahahmm these failures are getting a 503 in the compute client21:16
* mwhahaha goes looking in the right service logs21:16
mwhahahamaybe we need more nova blaming21:16
owalshmwhahaha: I expect ec2api metadata needs mode http too21:19
mwhahahaowalsh: so i want to say it doesn't cause it's rcp21:19
mwhahaharpc21:19
mwhahahaso mode http screws with it21:19
mwhahahabut that might just be ec2api21:19
openstackgerritMerged openstack/tripleo-heat-templates stable/queens: Fix fluentd upgrade tasks during FFU.  https://review.openstack.org/55483121:19
openstackgerritMerged openstack/tripleo-quickstart-extras master: pass the fine input to subunit2junitxml  https://review.openstack.org/55136421:19
openstackgerritMerged openstack/instack-undercloud stable/queens: Add configuration for the Nova proxy endpoint  https://review.openstack.org/55639621:19
openstackgerritMerged openstack/instack-undercloud stable/queens: Add Nova UI endpoints to the endpoint list  https://review.openstack.org/55639821:19
openstackgerritMerged openstack/instack-undercloud stable/queens: Fix next_hop for metadata service host route on local_subnet  https://review.openstack.org/55667521:19
openstackgerritRafael Folco proposed openstack/tripleo-quickstart-extras master: Temporarily add tempest failures to skip list  https://review.openstack.org/55735421:19
openstackgerritMerged openstack/instack-undercloud master: Updated from global requirements  https://review.openstack.org/55567221:19
openstackgerritMerged openstack/instack-undercloud master: add lower-constraints job  https://review.openstack.org/55556621:19
openstackgerritMerged openstack/instack-undercloud master: Fix help string for subnets option  https://review.openstack.org/55489521:19
openstackgerritMerged openstack/puppet-tripleo master: Add fact to get array of nic alias name  https://review.openstack.org/55311921:19
owalshyea, ec2 metadata is just a hacked nova metadata AFAIK21:20
owalshmwhahaha: ^^^21:20
mwhahahak21:20
*** rlandy is now known as rlandy|brb21:22
mwhahahabandini or jaosorior: why do we not have any haproxy logs anywhere? did we not properly configure syslog for it under containers21:24
*** yamahata has joined #tripleo21:24
*** moshele has quit IRC21:26
*** rfolco|ruck is now known as rfolco|off21:26
mwhahahacause of course it's in journal.txt21:26
mwhahahaMar 28 19:03:40 overcloud-controller-0 haproxy[32495]: 192.168.24.13:49754 [28/Mar/2018:19:03:40.880] nova_osapi nova_osapi/<NOSRV> 0/-1/-1/-1/0 503 212 - - SC-- 17/0/0/0/0 0/0 "GET /v2.1/servers/detail?all_tenant21:27
mwhahahas=True HTTP/1.1"21:27
*** ansmith has joined #tripleo21:27
mwhahahaoh i found it21:29
mwhahahaweshay: nova_osapi crashes or something21:29
*** alee has quit IRC21:29
mwhahahahttps://logs.rdoproject.org/24/557424/1/openstack-experimental/gate-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/Z5a2df7deb1104f2399ad2daeb489de22/overcloud-controller-0/var/log/journal.txt.gz#_Mar_28_18_36_0121:30
openstackgerritMerged openstack/puppet-tripleo master: Enable networking-mlnx ml2 drivers  https://review.openstack.org/55050521:30
*** Goneri has quit IRC21:32
*** jlabarre has quit IRC21:33
openstackgerritSteve Baker proposed openstack/tripleo-common master: Document functions for container prepare  https://review.openstack.org/55470721:35
openstackgerritSteve Baker proposed openstack/tripleo-common master: Represent image prepare defaults in new heat param format  https://review.openstack.org/55470821:35
openstackgerritSteve Baker proposed openstack/tripleo-common master: Move build_service_filter to kolla_builder from tripleoclient  https://review.openstack.org/55473821:35
openstackgerritSteve Baker proposed openstack/tripleo-common master: Perform multiple container image prepares and merge result  https://review.openstack.org/55473921:35
openstackgerritSteve Baker proposed openstack/tripleo-common master: Break out function get_undercloud_registry  https://review.openstack.org/55751921:35
openstackgerritSteve Baker proposed openstack/tripleo-common master: WIP repurpose action container_images_prepare  https://review.openstack.org/55752021:35
sai_pCan someone please give workflow to these reviews https://review.openstack.org/#/c/546476/ https://review.openstack.org/#/c/557101/ ?21:38
sai_pThank you21:38
mwhahahasai_p: do you know if nuage is replacing metadata_ip with metadata_host? i think that's the new one in nova21:39
rookI see the haproxy logs mwhahaha21:39
mwhahaharook: i found them eventually21:39
rookMar 28 18:23:51 overcloud-controller-0 haproxy[32399]: 192.168.24.1:50366 [28/Mar/2018:18:23:50.993] nova_osapi nova_osapi/overcloud-controller-0.internalapi.localdomain 0/0/0/124/124 200 1797 - - ---- 82/0/0/0/0 0/0 "GET /v2.1/servers/d93be453-7a5c-41f4-893c-73a78468a9a9 HTTP/1.1"21:39
* mwhahaha wishes we output them in the haproxy container instead of journal.txt21:39
rookmwhahaha: ack21:39
openstackgerritHarald Jensås proposed openstack/instack-undercloud stable/queens: Fix help string for subnets option  https://review.openstack.org/55752121:40
rookmwhahaha yeah me too -- if you have a example how other services do this, i don't mind fixing it.21:43
mwhahahawell it's cause we're using syslog21:43
rookI think we can update the haproxy conf to log to a specific log.21:44
rookoh21:44
mwhahahaother services get outputed to stdout21:44
rookwell21:44
rookFML21:44
mwhahahayea something like that21:44
rookmaybe we can just scrape the logs as a post task to create a haproxy log?21:44
mwhahahaat least we are logging :D21:44
rookyour welcome?21:44
rooklol21:44
mwhahahaor drop a rsyslog config21:44
mwhahaharook: you know we actually had an option to enable it21:44
*** ledo_ has quit IRC21:44
mwhahahathat we could have just set to true21:44
* mwhahaha failed on that review21:45
mwhahahahttps://github.com/openstack/puppet-tripleo/blob/master/manifests/haproxy.pp#L5221:45
rookwell shitfire.21:47
*** akrivoka has quit IRC21:47
mwhahahaoh well :D21:47
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: Deploy container-multinode (fs010) with a containerized undercloud  https://review.openstack.org/54290621:48
*** pickle has quit IRC21:48
*** rlandy|brb is now known as rlandy21:48
rookmwhahaha yeah, i looked back at my patch, i enabled http log and tcplog21:48
mwhahahayea it says if you set it globally it'll fall back as needed21:49
mwhahahaeh that whole thing needs a rewrite anyway21:49
EmilienMsorry if I missed some context; but have we checked the load of nova-api and nova-api-metadata?21:49
rookEmilienM: https://snapshot.raintank.io/dashboard/snapshot/5fvHrtMq15duO2iQyFS1U7LQ5rq10Y3x?orgId=221:51
EmilienMlooks like nova_api_wsgi is hungry21:52
*** pblaho has quit IRC21:53
-openstackstatus- NOTICE: the zuul web dashboard will experience a short downtime as we roll out some changes - no job execution should be affected21:53
EmilienMmwhahaha: what are we waiting to land your patch?21:54
EmilienMaka https://review.openstack.org/#/c/557073/21:54
EmilienMI just noticed all ovb jobs became red on this one21:55
mwhahahaEmilienM: yea it's not super pressing21:55
EmilienMwell21:55
EmilienMreducing httpd instances can only help no?21:55
mwhahahaevidently no :D21:55
*** jmelvin has quit IRC21:55
EmilienMsshnaidm: thanks for your comment on https://review.rdoproject.org/r/#/c/13130/ - can you merge it?21:55
mwhahahai think it'll help but i don't think now is the time to merge that one21:56
EmilienMsshnaidm: oh you say "files:" won't work at all?21:56
sshnaidmEmilienM, yes, "files:" doesn't work21:56
EmilienMwhere is paul21:57
*** social has quit IRC21:58
EmilienMsshnaidm: I think I know how to make it work21:59
*** pabelanger has joined #tripleo22:03
pabelangerhello22:03
EmilienMpabelanger: hey, sshnaidm has a comment on https://review.rdoproject.org/r/#/c/13130/1/zuul/upstream.yaml about the version of zuul and usage of files:22:04
*** yamahata has quit IRC22:04
EmilienMpabelanger: can you check the syntax and tell if it's supposed to work? I checked old version of project config with examples22:04
EmilienMand files: was used in zuul 2.5 in the project layout22:05
openstackgerritAlex Schultz proposed openstack/puppet-tripleo master: Fix mode for openstack services  https://review.openstack.org/55751122:05
pabelangeryou should be able to see the config output that zuul test generates to validate zuul loaded the regex on that job22:05
*** tbonds has joined #tripleo22:06
pabelangerbut regex is supported on files22:06
EmilienMpabelanger: files is supported in zuul v2.5?22:06
EmilienMsshnaidm: ^ fyi22:06
pabelangeryes22:07
EmilienMok22:07
EmilienMI'll investigate22:07
pabelangerhttp://git.zuul-ci.org/cgit/zuul/tree/doc/source/zuul.rst?h=2.6.0#n75222:07
pabelangerthat is an example from 2.5 docs22:08
pabelangerwell, docs, not example22:08
EmilienMoh doc22:08
EmilienMgood22:08
pabelangerhttp://git.zuul-ci.org/cgit/zuul/tree/tests/fixtures/layout.yaml?h=2.6.0#n11322:09
pabelangerfor unit test example22:09
pabelangeralso, I should have used https urls :)22:09
*** ooolpbot has joined #tripleo22:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION22:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/175755622:10
*** ooolpbot has quit IRC22:10
openstackLaunchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged]22:10
EmilienMpabelanger: like jim said on #zuul it might be an ordering thing22:10
EmilienMpabelanger: the layout below might cancel what I did above22:10
sai_pmwhahaha: Currently, Nuage is not using metadata_host and not sure will be used in future based on TLS value22:11
pabelangerEmilienM: oh, yes. that is like it. Last rule wins IIRC22:11
mwhahahasai_p: ok22:11
*** Goneri has joined #tripleo22:12
*** alee has joined #tripleo22:14
mwhahahaowalsh: tomorrow can you take a peek at https://bugs.launchpad.net/tripleo/+bug/1759583 seems nova-api is timing out22:14
openstackLaunchpad bug 1759583 in tripleo "UnexpectedResponseCode: Unexpected response code received from api.compute tempest tests" [High,Triaged]22:14
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: Temporarily add tempest failures to skip list  https://review.openstack.org/55735422:15
owalshmwhahaha: smells like networking looking at the traceback...22:19
mwhahahaowalsh: it's likely, but any hints as to what part of networking would be nice :D since i can't find that message anywhere22:19
*** morazi has quit IRC22:24
*** fpan has joined #tripleo22:25
*** rcernin has joined #tripleo22:28
openstackgerritMerged openstack/os-net-config master: Remove mox usage  https://review.openstack.org/55098822:29
*** fpan has quit IRC22:32
EmilienMmwhahaha: fyi both community goals for rocky are done now22:34
mwhahahak22:34
EmilienM(sb updated)22:34
mwhahahastevebaker: i'm getting an error when trying to do an image prepate http://paste.openstack.org/show/716658/ thoughts?22:41
stevebakermwhahaha: looking22:42
mwhahahait's not consistent on the image, next run it's the barbican-worker22:43
stevebakermwhahaha: could docker.io be flaking on ping API calls?22:43
mwhahahait's possible i guess22:44
mwhahahado we retry that?22:44
rookEmilienM: just read back - were you referring to my graphs?22:46
EmilienMrook: yes22:46
rookre: nova_api_wsgi being hungry22:46
rookEmilienM ah, 10% isn't bad22:46
stevebakermwhahaha: my immediate emotional reaction is if a REST API can't respond to ping calls then I give up22:46
stevebaker:)22:46
mwhahahahow anti-cloud of you22:46
rookWhen compared to Neutron 50-70% CPU util22:46
stevebakermwhahaha: where are you calling this from? an rdocloud node?22:47
mwhahahastevebaker: no rh222:47
mwhahahamust be something going on over there22:47
* mwhahaha gives up for the day22:47
mwhahahastevebaker: i'll figure it out next week i guess22:47
*** ansiwen[q] has joined #tripleo22:47
stevebakerI've not managed to reproduce yet, will try from a few other places22:48
*** pblaho has joined #tripleo22:48
*** pblaho has quit IRC23:00
EmilienMdsneddon: hey, if you have time this week, please review https://review.openstack.org/#/c/553427/ - it would help for the efforts with containerized undercloud and masquerading23:01
dsneddonEmilienM, I am doing that this moment.23:02
dsneddonEmilienM, I have a question. how do you expect people to set masquerading for routed networks with this patch? Would they create an environment file that overrides environments/services/masquerade-networks.yaml? I wonder if we should have a comment that explains that the hash is <source>:[<destination>, <destination>, ...] ?23:03
dsneddonEmilienM, I was thinking of adding a comment and then merging.23:03
*** fpan has joined #tripleo23:04
EmilienMdsneddon: see the default in https://review.openstack.org/#/c/553427/9/puppet/services/masquerade-networks.yaml23:04
EmilienMdefault: {'192.168.24.0/24': ['192.168.24.0/24', '192.168.25.0/24']}23:05
*** tbonds has quit IRC23:05
EmilienMwhich comes from hjensas's input on how he feed it in tripleoclient23:05
EmilienMdsneddon: I have to go now but I'll check your answer later, thanks for your review!23:09
dsneddonEmilienM, Thanks23:09
*** ooolpbot has joined #tripleo23:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION23:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/175755623:10
*** ooolpbot has quit IRC23:10
openstackLaunchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged]23:10
EmilienMalso if it's just a comment / or doc, I'm happy to push on top of this patch23:10
*** rlandy has quit IRC23:10
dsneddonEmilienM, Good point, I'll merge and figure out the details of documentation with hjensas23:10
EmilienMack thanks23:11
* EmilienM out23:11
*** tosky has quit IRC23:16
openstackgerritMerged openstack/puppet-tripleo master: Removes neutron ownership of certs  https://review.openstack.org/55746923:21
*** Goneri has quit IRC23:22
openstackgerritMarius Cornea proposed openstack/tripleo-upgrade master: Add support for per role FFU upgrade  https://review.openstack.org/55668723:23
*** mcornea has quit IRC23:34
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: Collect dstat on overcloud nodes  https://review.openstack.org/55745523:48
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras  https://review.openstack.org/47260723:55
*** fpan has quit IRC23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!