Thursday, 2020-04-09

openstackgerritMerged openstack/os-net-config stable/train: SRIOV VF: Add min_tx_rate and max_tx_rate  https://review.opendev.org/71767700:12
openstackgerritMerged openstack/puppet-tripleo master: Remove undefined octavia::rabbit_password from default hieradata  https://review.opendev.org/71812600:37
openstackgerritTakashi Kajinami proposed openstack/tripleo-heat-templates master: Remove client for unsupported services from lower-constraint  https://review.opendev.org/71858300:39
openstackgerritTakashi Kajinami proposed openstack/tripleo-heat-templates master: Remove clients for unsupported services from lower-constraints  https://review.opendev.org/71858300:40
*** sanjayu_ has quit IRC00:42
*** cdearborn has quit IRC00:50
openstackgerritFlavio Fernandes proposed openstack/tripleo-heat-templates master: Add Octavia OVN Provider configuration  https://review.opendev.org/70769500:51
*** jmasud has quit IRC00:54
*** jmasud has joined #tripleo00:55
*** raildo has quit IRC01:24
*** ysandeep|away is now known as ysandeep|rover01:52
*** Goneri has quit IRC01:59
*** mrunge_ has joined #tripleo02:02
*** mrunge has quit IRC02:03
openstackgerritRabi Mishra proposed openstack/tripleo-ansible master: WIP escape newline in nested json string  https://review.opendev.org/71795302:06
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: tripleo_container_manage: optimize systemd services/healthcheck bootstrap  https://review.opendev.org/71854502:17
*** jdandrea has quit IRC02:40
*** ysandeep|rover is now known as ysandeep|afk03:09
*** rh-jelabarre has quit IRC03:12
*** social has quit IRC03:16
openstackgerritMerged openstack/puppet-tripleo stable/train: Support for PowerMax Volume Config  https://review.opendev.org/71779303:25
openstackgerritMerged openstack/tripleo-common stable/stein: Wire-in missing concurrency for deploy_instances  https://review.opendev.org/71719603:26
openstackgerritMerged openstack/tripleo-heat-templates master: Remove hieradata for TripleO UI  https://review.opendev.org/71810703:28
openstackgerritTakashi Kajinami proposed openstack/tripleo-heat-templates master: Cleanup remaining items for Trove  https://review.opendev.org/71809503:32
openstackgerritMerged openstack/tripleo-heat-templates stable/train: Workaround for cinder A/A and etcd with TLS-everywhere  https://review.opendev.org/71783703:36
openstackgerritMerged openstack/tripleo-heat-templates master: Enable caching in heat  https://review.opendev.org/71696703:36
openstackgerritMerged openstack/tripleo-heat-templates master: Remove duplicate key  https://review.opendev.org/71710603:36
*** skramaja has joined #tripleo03:58
*** ysandeep|afk is now known as ysandeep04:00
*** evrardjp has quit IRC04:37
*** evrardjp has joined #tripleo04:37
*** udesale has joined #tripleo04:40
*** ykarel|away is now known as ykarel04:47
*** ysandeep is now known as ysandeep|rover04:56
openstackgerritRabi Mishra proposed openstack/tripleo-ansible master: WIP escape newline in nested json string  https://review.opendev.org/71795305:02
*** dmacpher has quit IRC05:03
*** dmacpher has joined #tripleo05:03
ramishraykarel: Hey! I guess we need a promotion for https://review.opendev.org/#/c/718127/? ovb jobs still failing05:05
ramishrahttps://trunk.rdoproject.org/centos8/component/baremetal/40/be/40bed1ea88e1b0e2cddc380f24823390f6e3bba3_50ba4be6/ does not seem to have the latest packages05:06
ykarelramishra, where are u seeing issue ? the issue shouldn't be seen in upstream jobs05:06
ykarelonly promotion jobs should fail due to that05:07
ramishraykarel: https://logserver.rdoproject.org/53/717953/2/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/0305619/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz05:08
ykarelramishra, looking, may be it's false promoted again05:09
*** ratailor has joined #tripleo05:10
*** marios_ has joined #tripleo05:11
ykarelramishra, checked, it's again false promotion, ovb failed and still promoted https://trunk.rdoproject.org/api-centos8-master-uc/api/civotes_agg_detail.html?ref_hash=33c72c65c61b68a92cd1e9ace947ed4c05:13
ykarelso we need another promotion with that ironic fix05:14
ykarelysandeep|rover, fyi ^^05:14
ysandeep|roverykarel, ack05:15
ramishraykarel: Is it someone ignoring those ovb failures and force promoting?05:16
ykarelramishra, yes, it's ignored, yesterdays promotion needed with ignoring ovb, but then it was missed to revert the criteria05:19
ykarelso it false promoted again05:19
*** jbadiapa has quit IRC05:20
ykarelramishra, https://review.rdoproject.org/r/#/c/26311/05:21
ykarelchandankumar, can u get ^^ merged05:21
chandankumarykarel, on it05:21
ramishraykarel: Ah ok.. thanks!05:21
ykarelfor now i am appliying it manually on promoter05:21
ramishrasounds good, thanks for looking05:23
*** fultonj has quit IRC05:25
*** fultonj has joined #tripleo05:26
*** bandini has joined #tripleo05:31
openstackgerritMerged openstack/python-tripleoclient stable/train: Ensure the undercloud output dir constant is consistent  https://review.opendev.org/71841505:35
openstackgerritMerged openstack/puppet-tripleo master: Add check for bootstrap_node for downcase  https://review.opendev.org/71819405:35
openstackgerritMerged openstack/instack-undercloud stable/rocky: Remove trove firewall rule  https://review.opendev.org/69932805:35
*** dwilde has quit IRC05:45
*** sanjayu_ has joined #tripleo06:06
*** dpawlik has joined #tripleo06:07
*** sanjayu__ has joined #tripleo06:09
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates master: Move the haproxy iptables rules creation to host_prep_tasks  https://review.opendev.org/71815906:11
*** sanjayu_ has quit IRC06:12
openstackgerritMerged openstack/tripleo-ansible master: tripleo_all_nodes_data: sort enabled_services (idempotency)  https://review.opendev.org/71847106:30
ramishraykarel: I guess you'll hit  https://bugs.launchpad.net/bugs/187133806:33
openstackLaunchpad bug 1871338 in tripleo "overcloud deployment failing with msg: 'argument parameters is of type <class ''str''> and we were unable to convert to dict: unable to evaluate string as dictionary'." [Critical,In progress] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)06:33
ramishrawith promotion06:33
ykarelramishra, yes06:33
ramishraI was testing https://review.opendev.org/#/c/717953/ to see if it fixes or works around it.. but it seems Depends-On also does not work06:34
ramishrafor the ovb jobs06:34
ykarelramishra, depends-on not working?06:34
ramishralooks like it did not https://logserver.rdoproject.org/53/717953/3/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/3ed6cb1/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz06:35
ykarelramishra, ack /me checks logs06:35
*** slaweq_ has quit IRC06:36
ykarelramishra, depends on do not work on merged patches06:36
*** dpawlik has quit IRC06:37
ykarelramishra, if u see ZUUL_CHANGES=openstack/tripleo-ansible:master:refs/changes/53/717953/3, ironic patch is not there as it's merged06:37
ykarelin job-output.txt06:37
ramishraHmm.. not sure how we can test it..06:38
ykarelramishra, there can be two option06:38
*** bogdando has joined #tripleo06:39
ramishraforce promote the ironic fix?06:39
*** dpawlik has joined #tripleo06:39
ykareleither test with promotion job where we have ironic patch06:39
ykarelor depends-on an ironic patch that is not merged06:39
ramishraykarel: good idea, let me try the second one and see if it works06:40
ykarelramishra, ack06:40
openstackgerritRabi Mishra proposed openstack/tripleo-ansible master: WIP escape newline in nested json string  https://review.opendev.org/71795306:47
*** xek_ has joined #tripleo06:48
*** bogdando has quit IRC06:53
*** bogdando has joined #tripleo06:55
*** sshnaidm|afk is now known as sshnaidm06:55
*** jcapitao has joined #tripleo06:57
*** jtomasek has joined #tripleo06:57
*** jtomasek has quit IRC06:59
*** jtomasek has joined #tripleo06:59
*** slaweq_ has joined #tripleo07:00
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/python-tripleoclient stable/train: Fix misused deployment vs connection timeouts  https://review.opendev.org/71834707:02
openstackgerritFrancesco Pantano proposed openstack/tripleo-docs master: Customizing crushmap using device classes  https://review.opendev.org/71810907:03
*** slaweq_ is now known as slaweq07:03
*** dtantsur|afk is now known as dtantsur07:06
*** maciejjozefczyk has joined #tripleo07:10
*** rfolco has quit IRC07:11
*** iurygregory has joined #tripleo07:11
*** rpittau|afk is now known as rpittau07:18
openstackgerritSagi Shnaidman proposed openstack/ansible-role-collect-logs master: Add timeout for containers logs collection  https://review.opendev.org/71760407:18
openstackgerritTakashi Kajinami proposed openstack/instack-undercloud stable/queens: Remove trove firewall rule  https://review.opendev.org/71860507:20
openstackgerritMerged openstack/tripleo-heat-templates stable/train: Include {{step}} when setting facts  https://review.opendev.org/71752607:25
*** florianf has joined #tripleo07:36
*** psachin has joined #tripleo07:38
*** tosky has joined #tripleo07:42
openstackgerritMarios Andreou proposed openstack/tripleo-operator-ansible master: Adds overcloud roles generate role - creates roles_data.yaml  https://review.opendev.org/71663907:43
*** marios_ is now known as marios07:43
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-ci master: Refactor image build for downstream and image-build role  https://review.opendev.org/71704307:44
*** cylopez has joined #tripleo07:46
*** zbr has quit IRC07:46
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-ci master: Refactor image build for downstream and image-build role  https://review.opendev.org/71704307:46
*** zbr has joined #tripleo07:47
*** suuuper has joined #tripleo07:48
*** jschlueter has quit IRC07:49
*** ratailor has quit IRC07:51
*** jschlueter has joined #tripleo07:51
*** abdysn has joined #tripleo07:54
*** jpich has joined #tripleo07:55
*** ratailor has joined #tripleo07:58
*** pkopec has joined #tripleo07:59
*** threestrands has quit IRC08:00
*** ysandeep|rover is now known as ysandeep|lunch08:10
*** xek_ is now known as xek08:17
*** florianf has quit IRC08:17
*** lucasagomes has joined #tripleo08:18
openstackgerritMark Goddard proposed openstack/tripleo-ci master: Exclude rabbitmq-3.7.24 container image  https://review.opendev.org/71861508:19
*** alexmcleod has joined #tripleo08:28
openstackgerritMerged openstack/tripleo-heat-templates master: Deprecate KeepalivedRestart  https://review.opendev.org/71849208:29
dciabrin_whois jfrancoa08:36
dciabrin_oops sorry guys :)08:37
*** dciabrin_ is now known as dciabrin08:37
ykarelramishra, Depends-On on ironic worked and it moved forward.08:37
ykarelalso it looks like ur patch is working as it's in overcloud deploy for more than 20+ minutes08:38
ykarellooking https://review.rdoproject.org/zuul/stream/587204b34e114382812b529db0484109?logfile=console.log08:38
ykarelsshnaidm, fyi ^^08:40
ykarelpatch is https://review.opendev.org/#/c/717953/08:41
sshnaidmnice08:42
sshnaidmso it's multiline issue08:42
ykarelme not aware why it passes in some jobs and fails in some08:43
sshnaidmykarel, it passes multiline params in ovb, but singleline in multinode08:43
ykarelsshnaidm, seen failures in multionode job too08:44
ykarelfs03008:44
sshnaidmmaybe it has multiline params too?08:44
sshnaidmlike ssh keys08:45
ykarelsshnaidm, failure logs https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset030-master/7097e91/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz08:45
ykarelsshnaidm, where to check multiline?08:45
ramishraykarel: looks like it failed, don't know if the same issue or something else in a later stage08:50
ykarelramishra, possibly different issue08:51
openstackgerritMerged openstack/os-apply-config master: Use unittest.mock instead of third party mock  https://review.opendev.org/71741108:51
ykarellet's wait for logs and confirm08:53
sshnaidmykarel, rendered_parameters08:54
sshnaidmykarel, banner text is multinode08:54
sshnaidms/multinode/multiline08:55
ykarelsshnaidm, hmm got it08:55
ykarelso in featureset why it's not multiline08:55
ykarelfeatureset 1008:55
ykarelme checks logs08:55
*** derekh has joined #tripleo08:56
ykarelhmm that parameter is not there https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_334/717865/6/check/tripleo-ci-centos-8-containers-multinode/334ec7d/logs/undercloud/home/zuul/overcloud_deploy.log08:57
ykarelohhk it's set only for some scenarios http://codesearch.openstack.org/?q=BannerText&i=nope&files=&repos=08:58
*** social has joined #tripleo08:59
ramishraykarel: yeah the patch worked.. it seems to have failed with "2020-04-09 08:44:13 |     <13>Apr  9 08:44:11 puppet-user: Error: /Stage[main]/Pacemaker::Service/Service[pcsd]/ensure: change from 'stopped' to 'running' failed: Systemd start for pcsd failed!"09:00
ykarelramishra, cool09:01
*** tkajinam has quit IRC09:03
*** ykarel is now known as ykarel|lunch09:09
*** dpawlik has quit IRC09:10
*** holser has joined #tripleo09:11
openstackgerritamolkahat proposed openstack/tripleo-ci master: Removed centos7-scenario012 job. Added job tripleo-ci-centos-7-scenario012-standalone in depricated-jobs.yaml file.  https://review.opendev.org/71834609:12
*** holser has joined #tripleo09:12
*** sanjayu__ is now known as saneax_lunch09:14
ramishraHmm...https://logserver.rdoproject.org/53/717953/4/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-vexxhost/587204b/logs/overcloud-controller-0/var/log/pcsd/pcsd.log.txt.gz09:17
ramishrabandini: ^^^ any idea what's wrong09:17
*** ysandeep|lunch is now known as ysandeep|rover09:18
bandiniramishra: we have seen that one one other time, tried to debug it and failed since it reproduces rarely and we did not have the env09:19
ramishrabandini: you mean https://bugs.launchpad.net/tripleo/+bug/186760209:20
openstackLaunchpad bug 1867602 in tripleo "overcloud deploy failed due to Systemd start for pcsd failed" [Critical,Incomplete]09:20
bandiniramishra: yeah, ideally we'd add pacemaker::corosync::pcsd_debug: true' to CI so we can investigate more09:21
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates master: WIP: refactor nova db config  https://review.opendev.org/71855209:21
openstackgerritamolkahat proposed openstack/tripleo-ci master: Port tripleo-ci-centos-8-scenario012-standalone job to os_tempest  https://review.opendev.org/71799509:26
owalshgchamoul: hey, getting issues with tripleo-validations lower constraints on stable/train...09:32
owalshhttps://5e8b4515abd7c7f803b6-b71c90f67122d6f7b38b8442bc3a114a.ssl.cf2.rackcdn.com/718557/1/check/openstack-tox-lower-constraints/b78a02c/job-output.txt09:32
owalshgchamoul: is that using oslo.service from master instead of train?09:33
openstackgerritPiotr Kopec proposed openstack/tripleo-ansible master: Add new role to manage persistent memory devices  https://review.opendev.org/71715809:34
*** udesale_ has joined #tripleo09:34
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-operator-ansible master: Fixed typos  https://review.opendev.org/71863709:35
*** udesale has quit IRC09:38
*** florianf has joined #tripleo09:39
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/tripleo-heat-templates stable/train: Properly place undercloud hosts record upgrade task  https://review.opendev.org/71864009:39
*** mgoddard has joined #tripleo09:45
mgoddardHi09:45
mgoddardI'm trying to add an exclude for a container image that we are adding in kolla09:46
mgoddardhttps://review.opendev.org/#/c/718615/09:46
mgoddardI can see that it has been picked up: https://logserver.rdoproject.org/24/715224/3/openstack-check/tripleo-build-containers-rhel-8-train/0c31d19/logs/build_containers.sh09:46
mgoddardbut tripleo still tries to build it: https://logserver.rdoproject.org/24/715224/3/openstack-check/tripleo-build-containers-rhel-8-train/0c31d19/logs/containers-expected-to-build.log09:47
mgoddardanyone have any ideas?09:47
mgoddardcloudnull: ^09:47
*** dpawlik has joined #tripleo09:49
*** ykarel|lunch is now known as ykarel09:51
*** surpatil has joined #tripleo09:51
openstackgerritOliver Walsh proposed openstack/tripleo-validations stable/train: Drop eventlet from lower constraints  https://review.opendev.org/71864509:53
openstackgerritOliver Walsh proposed openstack/tripleo-validations stable/train: tripleo-validations failing on lower constraints  https://review.opendev.org/71864910:05
*** jpich has quit IRC10:06
bandiniade_lee: I have an environmnet with https://bugs.launchpad.net/tripleo/+bug/1871703/comments/310:06
openstackLaunchpad bug 1871703 in tripleo " $ca_pkcs12 = '/var/lib/certmonger/local/creds' not found in centos-8 ussuri undercloud deployment" [High,Triaged]10:06
*** jpich has joined #tripleo10:07
*** Abdallahyas has joined #tripleo10:07
owalshramishra, gchamoul: hi, is this right? https://review.opendev.org/71864910:08
*** abdysn has quit IRC10:10
openstackgerritOliver Walsh proposed openstack/tripleo-validations stable/train: Fix multistack inventory  https://review.opendev.org/71855710:10
*** gfidente|afk is now known as gfidente10:11
ramishraowalsh: that's not true or stable/train.. eventlet is not bumped in oslo.service https://github.com/openstack/oslo.service/blob/stable/train/requirements.txt#L710:12
owalshramishra: hrmm, so we need to set constraints for oslo.service?10:13
ramishraWhat error do you see.. may be someting wrong with stable jobs for tripleo-validations10:13
owalshramishra: https://5e8b4515abd7c7f803b6-b71c90f67122d6f7b38b8442bc3a114a.ssl.cf2.rackcdn.com/718557/1/check/openstack-tox-lower-constraints/b78a02c/job-output.txt10:13
ramishraowalsh: removing that is fine.. but the reason is not true10:13
*** rpittau is now known as rpittau|bbl10:13
ykarelramishra, wrt to pcsd start error, u posted some fix?10:14
ykareli see u posted a recheck-rdo there10:14
owalshramishra: yea, didn't seem right to me, why I'm asking for sanity checks :-)10:14
*** apetrich has quit IRC10:15
openstackgerritSorin Sbarnea proposed openstack/ansible-role-collect-logs master: Convert flatten_nested_dict filter into a module  https://review.opendev.org/71772310:15
ramishraykarel: no I don't yet know the issue.. Just checked if it's reproducing as https://bugs.launchpad.net/tripleo/+bug/1867602 has been marked incomplete10:16
openstackLaunchpad bug 1867602 in tripleo "overcloud deploy failed due to Systemd start for pcsd failed" [Critical,Incomplete]10:16
ramishrawhich looks the same10:16
ykarelchandankumar, ohhk, two jobs failed with same error so seems a real issue10:16
ykarelme checks logs to see if can find something10:16
chandankumarykarel, which one10:17
ykarelchandankumar, https://review.opendev.org/#/c/717953/ ovb failures10:18
ykareli see E, [2020-04-09T08:49:51.199 #00000]    ERROR -- : Unable to start pcsd daemon, exiting: [Errno 2] No such file or directory: '/var/lib/pcsd/pcsd.crt'  in pcsd logs10:20
chandankumarykarel, above bug then10:20
openstackgerritamolkahat proposed openstack/tripleo-quickstart master: Ported featureset060.yml to os_tempest.  https://review.opendev.org/71809110:20
ykarelchandankumar, yes,10:21
ykarelchandankumar, from bandini suggestion is pacemaker::corosync::pcsd_debug: true tried10:21
ykareland got something?10:21
chandankumarykarel, last time, we failed to reproduce it10:22
chandankumarykarel, https://review.opendev.org/#/c/713219/10:22
bandinii don't think it has ever been tried? or i never got to see any debug logs10:22
owalshramishra: also had to cherry pick the master patch that bumps the keystonauth1/openstacksdk/os-service-types ... more concerned about that10:22
ykarelchandankumar, ohhk, let' see if it reproduces again,10:23
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-heat-templates master: [DNM] pcs debug  https://review.opendev.org/71321910:23
chandankumarykarel, ^^10:23
ykarelchandankumar, ack /me adjusts10:23
openstackgerrityatin proposed openstack/tripleo-heat-templates master: [DNM] pcs debug  https://review.opendev.org/71321910:24
ykarelchandankumar, updated with depends-on on fixes10:25
ykarelbandini, will let u know if it reproduces10:25
bandiniack thanks I cced myself there in the meantime10:25
ykarelokk thanks10:26
ykarelramishra, fyi promoting that ironic fix10:28
ykarelhttps://trunk.rdoproject.org/api-centos8-master-uc/api/civotes_agg_detail.html?ref_hash=e979cbb265693180fcde894768701cda10:28
*** xek has quit IRC10:32
openstackgerritSorin Sbarnea proposed openstack/ansible-role-collect-logs master: Convert flatten_nested_dict filter into a module  https://review.opendev.org/71772310:32
*** saneax_lunch is now known as saneax_10:34
*** surpatil has quit IRC10:44
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-ci master: Refactor image build for downstream and image-build role  https://review.opendev.org/71704310:52
*** jcapitao is now known as jcapitao_lunch10:59
*** apetrich has joined #tripleo11:00
*** mgoddard has quit IRC11:02
ramishraykarel: ack11:07
ramishraowalsh: I don't know why you've to bump minimum version of libraries in stable branches unless it's being bumped in a dep.. May be something wrong with the job11:11
*** rfolco has joined #tripleo11:34
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-operator-ansible master: Make tripleo_shell_script more robust  https://review.opendev.org/71867511:34
owalshramishra: yea, missing upper constraints for train?11:36
*** ratailor has quit IRC11:36
owalshramishra: https://github.com/openstack/tripleo-validations/blob/stable/train/tox.ini#L2511:37
owalshramishra: not used in the lower-constraints job though11:40
ramishraowalsh: that's an issue and should be fixed for stable branches but lower-constraints job uses https://github.com/openstack/tripleo-validations/blob/stable/train/tox.ini#L12011:40
ramishrayeah11:40
*** rh-jelabarre has joined #tripleo11:41
*** ysandeep|rover is now known as ysandeep|coffee11:47
owalshramishra: weird, job just passed for me without bumping any versions, just removing evenlet11:48
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-ci master: Refactor image build for downstream and image-build role  https://review.opendev.org/71704311:48
ykarelramishra, master promoted, ironic fix is available now in current-tripleo11:49
ramishraowalsh: yeah, that's what the error https://5e8b4515abd7c7f803b6-b71c90f67122d6f7b38b8442bc3a114a.ssl.cf2.rackcdn.com/718557/1/check/openstack-tox-lower-constraints/b78a02c/job-output.txt about11:51
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-operator-ansible master: Make tripleo_shell_script more robust  https://review.opendev.org/71867511:52
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-operator-ansible master: Ensure we don't pass empty params  https://review.opendev.org/71867811:52
owalshramishra: yes, earlier when I tested the fix locally I also had to cherry pick the other patch... but now it passes with just the eventlet patch11:52
ramishraI fail to understand where it's pulling that oslo.service that needs eventlet>=0.22.011:52
ramishraowalsh: I can look at it later11:52
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/train: Exclude /var/lib/ironic/* from container-puppet.sh rsync  https://review.opendev.org/71854811:53
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/train: Deprecate KeepalivedRestart  https://review.opendev.org/71867911:53
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible stable/train: tripleo_all_nodes_data: sort enabled_services (idempotency)  https://review.opendev.org/71868011:53
*** dpawlik has quit IRC11:56
*** rpittau|bbl is now known as rpittau11:57
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-operator-ansible master: Ensure we don't pass empty params  https://review.opendev.org/71867811:59
*** ysandeep|coffee is now known as ysandeep|rover12:00
EmilienMchem: sorry I had to -2 https://review.opendev.org/#/c/717719 -- this is clearly not the solution12:00
EmilienMchem: we have had the issue at each cycle and each time it's the same thing, we need to update tripleoclient dependencies versions12:00
EmilienMchem: the path forward is the following :12:01
EmilienMchem: 1) make sure there is a recent tag in tripleo-ansible that contains the new ansible plugin12:01
EmilienMchem: 2) update tripleoclient-distgit to depends on tripleo-ansible newtag12:01
EmilienMthat's it12:01
chemEmilienM: hey, so I have no problem for this review to go away, it's just that I opened it after a discussion (https://bugzilla.redhat.com/show_bug.cgi?id=1817457#c5) where it was said that maybe maintening that dependency inside the pkg was not a good idea12:03
openstackbugzilla.redhat.com bug 1817457 in python-tripleoclient "[osp16] Undercloud update fails, ansible cannot parse the generated playbook: ERROR! no action detected in task" [Medium,On_dev] - Assigned to sathlang12:03
EmilienMchem: discussion with who?12:04
EmilienMchem: that's how we have solved these (common) issues at every release12:04
chemEmilienM: I think jfrancoa mwhahaha and odyssey4me were around12:04
chemEmilienM: butagain no problem going that way if that's the usual solution, but I would need some mentoring/pointer as I never did that before12:05
EmilienMchem: I'm on it12:06
EmilienMchem: I'll show you what I did12:06
chemEmilienM: thanks, appreciated12:06
owalshramishra: tripleo-common in lower-constraints is ancient12:06
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-operator-ansible master: Make tripleo_shell_script more robust  https://review.opendev.org/71867512:07
EmilienMchem: step 1 https://review.opendev.org/#/c/71868312:07
*** Abdallahyas has quit IRC12:07
*** dpawlik has joined #tripleo12:08
openstackgerritTakashi Kajinami proposed openstack/tripleo-heat-templates master: Use absolute path to include puppet classes  https://review.opendev.org/71868412:10
owalshramishra: and that version of tripleo-common requires mistral which requires oslo.service12:10
EmilienMchem: step2: https://review.rdoproject.org/r/2635812:10
EmilienMchem: it was THT in fact, but same result12:10
*** raildo has joined #tripleo12:10
openstackgerritTakashi Kajinami proposed openstack/tripleo-heat-templates master: Use absolute name to include puppet classes  https://review.opendev.org/71868412:10
owalshramishra: can either pin mistral (which t-h-t lc does) or bump tripleo-common to a newer version that doesn't require mistral12:11
*** jcapitao_lunch is now known as jcapitao12:13
*** florianf has quit IRC12:15
chemEmilienM: sorry, just one last question abouth the train entry in the release file12:17
chemEmilienM: https://review.opendev.org/#/c/718683/1/deliverables/train/tripleo-ansible.yaml12:17
chemEmilienM: and then I just sit back and applause :)12:17
EmilienMchem: answered12:20
ramishraowalsh: Looks like tripleo-common 7.1.0 does not have lower constraints ..yeah bumping tripleo-common would probably be the right thing12:20
openstackgerritMerged openstack/os-refresh-config master: Cleanup py27 support  https://review.opendev.org/71756812:20
chemEmilienM: thanks, all clear now.12:20
cloudnullmornings12:20
owalshramishra: ack,  11.0.0?12:21
openstackgerritSagi Shnaidman proposed openstack/tripleo-operator-ansible master: Add missed variables to image upload command  https://review.opendev.org/71868712:23
sshnaidmmwhahaha, marios ^^12:23
ramishraowalsh: may be 11.3.2 https://github.com/openstack/releases/blob/master/deliverables/train/tripleo-common.yaml#L3012:24
owalshramishra: ack, thanks!12:25
EmilienMchem: I looked-up and found a few examples where we did the same thing: https://review.rdoproject.org/r/#/c/20986/ - https://review.rdoproject.org/r/#/c/20245/ - https://review.rdoproject.org/r/#/c/19747/12:26
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/paunch master: Process labels before building container run args  https://review.opendev.org/71868812:26
*** psachin has quit IRC12:26
owalshramishra: too new, metalsmith requires a newer openstacksdk, 11.3.1 is good12:28
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/paunch stable/train: Process labels before building container run args  https://review.opendev.org/71845912:28
ramishraowalsh: ok12:29
openstackgerritOliver Walsh proposed openstack/tripleo-validations stable/train: tripleo-validations failing on lower constraints  https://review.opendev.org/71864912:31
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-ci master: Refactor image build for downstream and image-build role  https://review.opendev.org/71704312:31
openstackgerritOliver Walsh proposed openstack/tripleo-validations stable/train: Fix multistack inventory  https://review.opendev.org/71855712:32
paramiteGreetings guys, I have a question. Does Keystone user creation during overcloud deployment changed recently?12:32
openstackgerritMarios Andreou proposed openstack/tripleo-ci master: WIP: Modify conatiner build role to allow for internal builds  https://review.opendev.org/71789512:34
*** pkopec has quit IRC12:35
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-ci master: Refactor image build for downstream and image-build role  https://review.opendev.org/71704312:43
*** iurygregory has quit IRC12:43
*** iurygregory has joined #tripleo12:44
*** hkominos has joined #tripleo12:49
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates master: WIP: refactor nova db config  https://review.opendev.org/71855212:50
*** lbragstad_ has joined #tripleo12:52
*** lbragstad has quit IRC12:54
*** jamesden_ has joined #tripleo12:55
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates master: WIP: refactor nova db config  https://review.opendev.org/71855212:55
openstackgerritMaciej Józefczyk proposed openstack/tripleo-heat-templates master: WIP: Change default dns domain in neutron  https://review.opendev.org/69940712:56
openstackgerritFlavio Fernandes proposed openstack/tripleo-heat-templates master: Add Octavia OVN Provider configuration  https://review.opendev.org/70769512:56
hkominosHi guys. During an attepted update from Rocky to Stein I get an "ResourceFailure: resources.Networks: Resource UPDATE failed: resources.ManagementNetwork: Updating a stack when it is deleting is not supported." My rocky templates had the default behaviour . (enabled but not in the role).  Did something change in the way rendering happens for the12:57
hkominosnetwork templates ?12:57
*** jamesdenton has quit IRC12:57
ykarelramishra, bandini, it failed and logs with debug enabled patch- https://logserver.rdoproject.org/19/713219/3/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/1b7bf6b/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz12:59
*** cdearborn has joined #tripleo12:59
hkominosnvm13:03
owalshramishra: I hate pip! tripleo-common 11.3.1 works for me but fails in CI13:07
bandiniykarel: ack will look after this call13:07
openstackgerritTakashi Kajinami proposed openstack/tripleo-heat-templates master: Use absolute names to include puppet classes  https://review.opendev.org/71868413:07
owalshtripleo-common requires metalsmith requires newer openstacksdk, but tripleo-common lower-constraints pins a lower openstacksdk but magically works (because pip is dumb I assume)13:08
ykarelbandini, okk13:11
*** ykarel is now known as ykarel|afk13:11
*** mrunge_ is now known as mrunge13:14
openstackgerritOliver Walsh proposed openstack/tripleo-validations stable/train: tripleo-validations failing on lower constraints  https://review.opendev.org/71864913:16
openstackgerritOliver Walsh proposed openstack/tripleo-validations stable/train: Fix multistack inventory  https://review.opendev.org/71855713:17
openstackgerritLuke Short proposed openstack/python-tripleoclient stable/train: Expose --limit, --skip-tags, and --tags on the CLI.  https://review.opendev.org/71665413:19
weshay|ruckmwhahaha, EmilienM as you guys have time.. understanding thoughts / vote on https://review.opendev.org/#/c/718468/13:25
openstackgerritFlavio Fernandes proposed openstack/tripleo-heat-templates stable/train: Add Octavia OVN Provider configuration  https://review.opendev.org/71491613:25
*** pkopec has joined #tripleo13:26
mariosmwhahaha: should we nuke that file? https://review.opendev.org/#/c/716639/15/roles/tripleo_overcloud_roles_generate/tests/test.yml folks were asking about it but as far as we can see isn't used anywhere13:28
mariosmwhahaha: do you want it there for merge or should we remove?13:28
mariosmwhahaha: (its in all the roles though so perhaps you have some plan for those)13:28
mwhahahajust leave it, we have it in all because that's one of the default role things13:29
mariosmwhahaha: k zbr fyi ^^^13:29
sshnaidmmwhahaha, do you know why we have TripleOImagePrepareDefault https://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/container_image.py#L885:L88513:29
sshnaidmmwhahaha, how would I call it in operator with "... default"13:30
mwhahahatripleo_container_image_prepare_default ?13:30
mwhahahaalready exists?13:30
EmilienMdciabrin, bandini : can you guys look at https://review.opendev.org/#/c/718615/ ?13:30
weshay|ruckcloudnull, https://review.opendev.org/#/c/718468/13:31
mwhahahasshnaidm: https://github.com/openstack/tripleo-operator-ansible/blob/master/roles/tripleo_container_image_prepare_default/tasks/main.yml ?13:31
sshnaidmmwhahaha, oh, I didn't notice it's a different role, thanks13:32
bandiniEmilienM: done. (dciabrin has his birthday today so he is probably not in front of the pc, I hope)13:34
zbrmwhahaha: marios: i doubt we need a tests/test.yaml as we we have molecule/default/converge.yml for that. Good idea to propose a wide removal and see what happens.13:35
weshay|ruckade_lee, fyi https://bugs.launchpad.net/tripleo/+bug/1871703 looks like bandini hit it as well13:35
openstackLaunchpad bug 1871703 in tripleo " $ca_pkcs12 = '/var/lib/certmonger/local/creds' not found in centos-8 ussuri undercloud deployment" [High,Triaged]13:35
zbrlikely a leftover carried away from the template.13:35
weshay|ruckade_lee, /me opens new tmate13:35
mwhahahazbr: marios: i don't really care, we can remove if you feel strongly about it13:35
zbrtest/ folder is used by testinfra, but is supposed to have python code inside.13:35
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Run container image prepare with operator  https://review.opendev.org/71870113:35
marioszbr: indeed it isn't used currently...13:36
marioszbr: i guess the main 'problem' is that it is in all the roles... so it will require a cleanup patch to remove all13:36
marioszbr: i can remove from this patch though ... k sec13:36
bandiniweshay|ruck: yeah, I have an env ready to be accessed and debugged (fwiw)13:38
EmilienMbandini: thank you sir13:40
openstackgerritLukas Bezdicka proposed openstack/tripleo-heat-templates stable/train: [Q->T] Introduce Queens to Train Upgrade  https://review.opendev.org/71348213:40
weshay|ruckbandini, aye..  I'll grab you and ade in an internal channel where we can share tmates.. etc13:40
bandiniweshay|ruck: ack let me drop the access info to you and ade13:40
openstackgerritMarios Andreou proposed openstack/tripleo-operator-ansible master: Adds overcloud roles generate role - creates roles_data.yaml  https://review.opendev.org/71663913:40
mariosmwhahaha: zbr: removed test.yaml ^^13:40
mariossshnaidm: please revote when you get chance thanks ^13:41
zbrmarios: can you also open another change that removes all fake tests.yaml files?13:42
openstackgerritFlavio Fernandes proposed openstack/puppet-tripleo master: Add Octavia OVN Provider configuration (2 of 2)  https://review.opendev.org/71371613:42
marioszbr: sure13:42
ade_leeweshay|ruck, I'm curious -- did the creds ever show up  on your system?13:44
weshay|ruckade_lee, bandini can you guys join #tls_help internal13:46
ade_leeEmilienM, mwhahaha hjensas cloudnull https://review.opendev.org/#/c/718120/ and https://review.opendev.org/#/c/718191/ please13:48
*** dwilde has joined #tripleo13:48
openstackgerritFlavio Fernandes proposed openstack/puppet-tripleo stable/train: Add Octavia OVN Provider configuration (2 of 2)  https://review.opendev.org/71596013:52
*** rcernin has quit IRC13:54
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: tripleo_container_manage: optimize systemd services/healthcheck bootstrap  https://review.opendev.org/71854513:57
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: tripleo_container_manage: optimize systemd services/healthcheck bootstrap  https://review.opendev.org/71854513:57
sshnaidmcloudnull, ramishra so we go with this patch https://review.opendev.org/#/c/717953/ and I can abandon mine?13:58
ramishrasshnaidm: though it worked when updating parameters, there is some issue later.. we're still looking at that14:00
ramishradon't know if it's related14:00
sshnaidmramishra, ack14:00
cloudnullsshnaidm ramishra.14:01
cloudnull++14:01
cloudnullthat's a good find14:01
*** TrevorV has joined #tripleo14:01
*** ykarel|afk is now known as ykarel14:02
*** abdysn has joined #tripleo14:04
*** cloudnull has quit IRC14:08
openstackgerritMerged openstack/tripleo-operator-ansible master: Adds overcloud roles generate role - creates roles_data.yaml  https://review.opendev.org/71663914:09
*** tkajinam has joined #tripleo14:13
*** Goneri has joined #tripleo14:14
*** pojadhav is now known as pojadhav|away14:15
openstackgerritMerged openstack/tripleo-ansible master: Don't call sdk.connect() again  https://review.opendev.org/71795214:20
*** lbragstad_ is now known as lbragstad14:20
openstackgerritOliver Walsh proposed openstack/tripleo-validations stable/train: tripleo-validations failing on lower constraints  https://review.opendev.org/71864914:22
openstackgerritOliver Walsh proposed openstack/tripleo-validations stable/train: Fix multistack inventory  https://review.opendev.org/71855714:23
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: Use kolla tools for memcached configuration  https://review.opendev.org/71856514:25
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Update tripleo-repos to match pattern  https://review.opendev.org/71813514:30
*** rascasoft has joined #tripleo14:38
openstackgerritMerged openstack/tripleo-operator-ansible master: Add missed variables to image upload command  https://review.opendev.org/71868714:40
ade_leeEmilienM, mwhahaha https://review.opendev.org/#/c/718120/ please14:42
openstackgerritMerged openstack/tripleo-operator-ansible master: Fixed typos  https://review.opendev.org/71863714:44
openstackgerritMerged openstack/tripleo-operator-ansible master: Ensure we don't pass empty params  https://review.opendev.org/71867814:47
openstackgerritMerged openstack/tripleo-operator-ansible master: Make tripleo_shell_script more robust  https://review.opendev.org/71867514:47
*** bogdando has quit IRC14:49
*** ysandeep|rover is now known as ysandeep|afk14:51
*** abdysn has quit IRC14:54
*** abdysn has joined #tripleo14:54
weshay|ruckchandankumar, bandini let's sync on some overcloud image build issues14:54
weshay|ruckchandankumar, https://bugs.launchpad.net/tripleo/+bug/1867602 https://bugs.launchpad.net/tripleo/+bug/187170314:55
openstackLaunchpad bug 1867602 in tripleo "overcloud deploy failed due to Systemd start for pcsd failed" [Critical,Triaged]14:55
openstackLaunchpad bug 1871703 in tripleo " $ca_pkcs12 = '/var/lib/certmonger/local/creds' not found in centos-8 ussuri undercloud deployment" [High,Triaged]14:55
*** cloudnull has joined #tripleo14:55
weshay|ruckbandini, not sure if I understand yet.. what is wrong w/ the centos-8 build .. yet14:55
bandiniweshay|ruck: yeah trying to narrow down the prob a bit14:55
bandinilet me update the pcsd bz in the meantime14:57
*** abdysn has quit IRC15:00
*** cloudnull has quit IRC15:01
*** cloudnull3 has joined #tripleo15:01
*** cloudnull3 is now known as cloudnull15:02
openstackgerritMarios Andreou proposed openstack/tripleo-ci master: WIP: Modify conatiner build role to allow for internal builds  https://review.opendev.org/71789515:04
*** hkominos has quit IRC15:04
*** rh-jelabarre has quit IRC15:06
*** slaweq has quit IRC15:06
lbragstadEmilienM i ended up testing a stable/train ovb environment last night (sans TLS or using FreeIPA for DNS) and the scale down tasks failed for me15:07
lbragstadis that a relatively new feature? or is not fully baked?15:07
lbragstadi wanted to see if i could successfully scale down a compute instance without any of the TLS changes we introduced15:08
EmilienMlbragstad: yeah me too it failed15:09
EmilienMbecause the scale down tasks only work if podman is enabled instead of docker15:09
EmilienMthe default will switch in ussuri (podman vs docker)15:10
lbragstadbah... so that's why it's failing to find that service15:10
EmilienMyup15:10
lbragstadgot it- that makes sense15:10
EmilienMso you have 2 options :15:10
lbragstaddoesn't explain the connectivity issues we're seeing, but it helps15:10
EmilienM1) deploy without HA (one controller) and enable podman (centos7)15:10
EmilienM2) deploy on centos8, but i don't know if we have packages/containers15:11
openstackgerritMerged openstack/tripleo-heat-templates master: Add missing memcached_servers for undercloud placement  https://review.opendev.org/71523415:11
openstackgerritMerged openstack/tripleo-ci master: Revert "make scenario001 non-voting until packages are sane"  https://review.opendev.org/71847715:11
EmilienMi would do 1)15:11
openstackgerritMerged openstack/tripleo-heat-templates master: Added scale_tasks to handle cleanup on scale down of nodes  https://review.opendev.org/70656415:11
openstackgerritMerged openstack/tripleo-heat-templates master: Remove Dell EMC PS Series Driver support  https://review.opendev.org/71714515:11
EmilienMlbragstad: I'm going to fix the scale_down tasks to be skipped when docker15:11
lbragstadEmilienM yeah - we haven't gotten our automation working with centos8 yet :(15:11
openstackgerritMerged openstack/python-tripleoclient master: Fix ResourceWarning: unclosed file  https://review.opendev.org/71821415:11
lbragstadok - so scale down will only be supported for podman deployments?15:12
EmilienMyup15:12
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: nova-compute: disable scale_tasks when docker is used  https://review.opendev.org/71872715:13
EmilienMlbragstad: ^ that will help with the error15:13
openstackgerritLuke Short proposed openstack/python-tripleoclient master: Expose --limit, --skip-tags, and --tags on the CLI.  https://review.opendev.org/69242115:16
lbragstadEmilienM looking15:17
lbragstadEmilienM also - trying to figure out why docker doesn't find that container on the compute host we're scaling down15:18
lbragstaddo we know why tripleo_nova_compute_healthcheck isn't on the compute host?15:18
EmilienMwhen docker is used, there is no systemd services managing the containers15:18
EmilienMthat's why..15:18
lbragstadahhh...15:18
*** rh-jelabarre has joined #tripleo15:21
*** slaweq has joined #tripleo15:22
lbragstadEmilienM is the reason you're stopping the healthcheck and nova-compute service is so that you can delete it later (at line 1160)?15:24
lbragstadhttps://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deployment/nova/nova-compute-container-puppet.yaml#L116015:24
lbragstadcan you not delete the nova compute service if it's running?15:24
EmilienMlbragstad: it's not clean15:24
EmilienMyes it's the reason15:24
lbragstadok - cool15:24
EmilienMwe followed reqs from nova folks15:24
EmilienMin any case we don't support docker anymore so15:25
lbragstadso - the compute container must be down before disabling the service15:25
EmilienMit's better, yes15:25
EmilienMa proper teardown15:25
lbragstadthat makes sense15:25
EmilienMhttps://docs.openstack.org/operations-guide/ops-maintenance-compute.html#planned-maintenance15:27
EmilienMlbragstad: we follow that doc ^15:27
EmilienMwe disable the service, we stop the containers and delete the service15:27
lbragstad++15:28
lbragstadok - cool15:28
lbragstadi see your point in using option #115:28
lbragstadwe just shouldn't expect to use HA with docker on train15:28
EmilienMunless you don't care about scale down :D15:28
EmilienMbut I'll fix it15:28
EmilienMso nobody else hits this15:29
lbragstadwell - we care about scale down and wanted to test it without any of our patches to see if it was something we were doing15:29
lbragstadwe do notice a different failure on scale down when we implement tls15:29
lbragstadour tls scale down executes successfully, but the very first nova scale down task fails15:30
openstackgerritDave Wilde (d34dh0r53) proposed openstack/tripleo-heat-templates stable/train: Added scale_tasks to handle cleanup on scale down of nodes  https://review.opendev.org/71873115:31
*** ykarel is now known as ykarel|away15:33
lbragstadspeaking of - ^ that's the patch we were testing when we hit the nova scale down issue15:35
openstackgerritDave Wilde (d34dh0r53) proposed openstack/tripleo-heat-templates stable/train: Added scale_tasks to handle cleanup on scale down of nodes  https://review.opendev.org/71873115:35
openstackgerritDave Wilde (d34dh0r53) proposed openstack/tripleo-heat-templates stable/train: Added scale_tasks to handle cleanup on scale down of nodes  https://review.opendev.org/71873115:37
owalshbandini, dciabrin: hey looking at https://bugs.launchpad.net/tripleo/+bug/1871795 ...15:43
openstackLaunchpad bug 1871795 in tripleo "tripleo-upgrade playbook fails in task "store roles and register controller role name"" [High,New]15:43
openstackgerritTakashi Kajinami proposed openstack/puppet-tripleo master: Remove manifests for Trove  https://review.opendev.org/71809715:43
owalshbandini, dciabrin: doesn't look lke https://review.opendev.org/715491 is the culpit, inventory is empty so filters will do nothing15:45
openstackgerritEmilien Macchi proposed openstack/tripleo-docs master: node_config: document ExtraGroupVars  https://review.opendev.org/71874115:46
owalshbandini, dciabrin: could be related to https://review.opendev.org/713919 but nothing obviously broken15:47
bandiniowalsh: yeah not sure, I did not see it on my reviews, damien mentioned it to me before disappearing into PTO today, so I thought I'd bring it up15:47
owalshbandini: happen to know what is generating /home/zuul/tripleo-ansible-inventory.yaml15:48
*** maciejjozefczyk has quit IRC15:49
*** tkajinam has quit IRC15:49
bandiniowalsh: it used to be tripleo-ansible-inventory in tripleo-common repo, but I see now that it is part of the tripleo-validation rpm15:50
owalshbandini: yea, I've landed patches for it too e.g https://review.opendev.org/713919. I mean where that's being run in CI, try to figure out how it resulted in an empty inventory15:51
*** lucasagomes has quit IRC15:53
bandiniowalsh: ops sorry :) not sure where it is being run, sorry15:53
owalshbandini: don't see how I could have broken it... 404 from heat might cause this though15:56
bandiniyou reckon it was a temporary hiccup thing?15:57
bandinior another review maybe?15:57
*** rpittau is now known as rpittau|afk15:59
owalshbandini: recheck to see?16:00
bandiniowalsh: sure thing let me try16:01
EmilienMlbragstad: I don't think we should spend time on docker support for scale down tasks in train16:03
EmilienMimo16:03
*** iurygregory has quit IRC16:03
lbragstadEmilienM yeah - that's fine16:03
lbragstadEmilienM i'll defer to what y'all want to do with it :)16:04
*** marios is now known as marios|out16:05
*** dpawlik has quit IRC16:06
*** Trevor_V has joined #tripleo16:16
*** marios|out has quit IRC16:18
*** TrevorV has quit IRC16:19
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient stable/train: Fix ResourceWarning: unclosed file  https://review.opendev.org/71874416:20
openstackgerritamolkahat proposed openstack/tripleo-quickstart master: Ported featureset060.yml to os_tempest.  https://review.opendev.org/71809116:22
owalshbandini: FWIW looks ok in other jobs e.g https://b2319ea300d930171925-35e07a4cf42cfaed2fcffa4bf0b16f1b.ssl.cf5.rackcdn.com/692421/20/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/94dc35c/logs/undercloud/home/zuul/tripleo-ansible-inventory.yaml16:23
bandiniowalsh: ack got you, let's see what the recheck does tonight16:25
*** jpich has quit IRC16:28
*** cloudnull has quit IRC16:31
weshay|ruckneed eyes on https://bugs.launchpad.net/tripleo/+bug/187133816:32
openstackLaunchpad bug 1871338 in tripleo "overcloud deployment failing with msg: 'argument parameters is of type <class ''str''> and we were unable to convert to dict: unable to evaluate string as dictionary'." [Critical,In progress] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)16:32
*** weshay|ruck changes topic to "CI status: Upstream GREEN, 3rd Party RED lp#1871338 | community irc meeting Tues@1400 UTC - tripleo-ci-community meeting Tues@1330 UTC | https://docs.openstack.org/tripleo-docs"16:33
weshay|ruckthanks for ops!!!16:33
*** cloudnull has joined #tripleo16:34
openstackgerritMerged openstack/tripleo-docs master: node_config: document ExtraGroupVars  https://review.opendev.org/71874116:34
*** bandini has quit IRC16:34
weshay|ruckade_lee, we're going to add rpm -V verification to the image builds.. to try and ensure this does not happen again.. if it is feasible of course16:35
*** udesale_ has quit IRC16:35
*** cloudnull has quit IRC16:35
ade_leeweshay|ruck, cool16:36
openstackgerritMerged openstack/tripleo-heat-templates master: Move the haproxy iptables rules creation to host_prep_tasks  https://review.opendev.org/71815916:36
weshay|ruckmwhahaha, you have a sec to look at a promotion blocker? https://bugs.launchpad.net/tripleo/+bug/187133816:36
openstackLaunchpad bug 1871338 in tripleo "overcloud deployment failing with msg: 'argument parameters is of type <class ''str''> and we were unable to convert to dict: unable to evaluate string as dictionary'." [Critical,In progress] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)16:36
weshay|rucksagi is gone to break unleavened bread16:36
*** cloudnull has joined #tripleo16:36
*** evrardjp has quit IRC16:37
mwhahahaisn't ekultails and cloudnull looking at the one?16:37
mwhahahai can if they aren't16:37
*** evrardjp has joined #tripleo16:37
* weshay|ruck asks now that cloudnull is back in his lounge16:37
* cloudnull missed a bunch of messages16:37
weshay|ruckthanks mwhahaha ..16:37
*** jcapitao has quit IRC16:37
cloudnullmy ISP is having a bad day16:37
weshay|ruckcloudnull, I was going to just give you $1400 of my bonus from gov cheese..16:37
weshay|ruckbut you weren't here16:38
cloudnullhahaha16:38
cloudnullmwhahaha what is ekultails looking at?16:38
weshay|ruckso instead of free $$ and cheese.. https://bugs.launchpad.net/tripleo/+bug/187133816:38
openstackLaunchpad bug 1871338 in tripleo "overcloud deployment failing with msg: 'argument parameters is of type <class ''str''> and we were unable to convert to dict: unable to evaluate string as dictionary'." [Critical,In progress] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)16:38
mwhahaha-^16:38
weshay|ruckany clue on dat one?16:38
cloudnulloh , sshnaidm ramishra and I were looking at that16:38
weshay|ruckcloudnull, sagi abandoned his change16:38
cloudnullthere is a change, let me go find that16:39
cloudnullhttps://review.opendev.org/#/c/717953/16:39
cloudnulllooks like it needs WIP removed.16:39
cloudnullthat should resolve the issue16:40
ekultailsThat's not something I've been working on. Would be happy to lend a hand if needed.16:40
ekultailsIt looks like it Depends-On a Do-Not-Merge patch.16:40
* cloudnull goes to tune up that review 16:40
ekultailsThe DNM patch is no-op so I guess yeah we can remove the "WIP" and "Depends-On" from the commit message and it should be fine.16:41
mwhahahaseems an odd solution16:42
* mwhahaha shrugs16:42
ekultailsIndeed16:42
cloudnullmwhahaha i think we can sanatize the input, instead of trying to deal with the issue in the playbook16:43
cloudnull**sanitize16:43
cloudnullbut that's the problem, stupid escaped newlines16:44
mwhahahaindeed16:45
mwhahahais it because we're pretty printing some json stuff16:45
*** ysandeep|afk is now known as ysandeep|out16:46
*** dtantsur is now known as dtantsur|afk16:46
cloudnulli'm not really sure...  I think it comes out of heat?16:47
cloudnullhttps://github.com/openstack/python-tripleoclient/blob/master/tripleoclient/v1/overcloud_deploy.py#L30216:47
mwhahahano16:47
mwhahahaprobably from files16:47
mwhahahanot certain tho16:48
mwhahahait's passed in via env16:48
mwhahahaand isn't this the thing that updates the plan for heat?16:48
mwhahahaso this is before heat runs16:48
cloudnullits an env, but the key is parameter_defaults and in some cases it seems to be a valid dictionary, but it others its a dictionary like string16:49
mwhahahahttps://github.com/openstack/python-tripleoclient/blob/a54ea2d01a947c77bc6b9fabb474bab90a831598/tripleoclient/v1/overcloud_deploy.py#L46016:49
cloudnullso you're probably right in that its coming from files16:50
mwhahahacomes from my favorit function process_multiple_environments16:50
* cloudnull clicks the "do you want to know more" link16:50
mwhahahacomes from heatclient.common16:51
mwhahahaactually16:51
mwhahahafiles, env = template_utils.process_environment_and_files(16:52
mwhahahahttps://github.com/openstack/python-heatclient/blob/master/heatclient/common/template_utils.py#L28716:53
mwhahahahttps://github.com/openstack/python-heatclient/blob/9dede4eb2b824b21c0031507bd986796f535810c/heatclient/common/template_format.py#L5316:53
mwhahahait's the yaml loader16:53
mwhahahai think16:53
mwhahahaanyway16:53
mwhahahasanitizing it in tripleoclient would probably be a good idea16:53
cloudnull++16:54
mwhahahais it yaml with json in it?16:54
mwhahahais that the problem?16:54
cloudnullit seems private keys are in that parameters defaults object.16:54
mwhahahafun16:55
cloudnullwhich seem to have extra newlines16:55
*** alexmcleod has quit IRC16:55
*** pojadhav|away is now known as pojadhav16:56
*** hjensas has quit IRC17:00
*** derekh has quit IRC17:01
*** sshnaidm is now known as sshnaidm|afk17:12
openstackgerritLukas Bezdicka proposed openstack/tripleo-heat-templates stable/train: [Q->T] Introduce Queens to Train Upgrade  https://review.opendev.org/71348217:18
openstackgerritGrzegorz Grasza proposed openstack/tripleo-heat-templates stable/train: Skip both tenant and management networks when generating certs  https://review.opendev.org/71875617:19
*** pkopec has quit IRC17:20
openstackgerritGrzegorz Grasza proposed openstack/tripleo-heat-templates stable/stein: Skip both tenant and management networks when generating certs  https://review.opendev.org/71875717:20
openstackgerritMerged openstack/tripleo-ansible master: Hide passwords better  https://review.opendev.org/71809617:20
*** panda is now known as panda|off17:27
*** hewbrocca is now known as hewbrocca_afk17:28
*** rascasoft has quit IRC17:33
openstackgerritKevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Add the ability to load vars from files  https://review.opendev.org/71875917:35
openstackgerritKevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Add the ability to load vars from files  https://review.opendev.org/71875917:36
cloudnullramishra sshnaidm|afk ^17:36
cloudnull-cc weshay|ruck17:36
weshay|ruckdanka17:37
cloudnullwe'll need to make sure fs001 is happy, but that should-tm take care of it17:38
*** sshnaidm|afk is now known as sshnaidm|off18:02
*** suuuper has quit IRC18:10
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: tripleo_container_manage: optimize systemd services/healthcheck bootstrap  https://review.opendev.org/71854518:20
*** rfolco is now known as rfolco|bbl18:22
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart-extras master: Revert "Enable ConfigDebug for undercloud deployments"  https://review.opendev.org/71876618:25
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart-extras master: Revert "Enable ConfigDebug for undercloud deployments"  https://review.opendev.org/71876618:25
EmilienMcloudnull: hey, with mwhahaha we're looking at why we get so much logs into overcloud deploys in CI18:26
EmilienMI suspect the --verbose increases Ansible verbosity18:26
EmilienMnormally we would get logs if tripleo_container_manage_debug is set to True18:27
EmilienMI'm trying to look where we set it to true in our CI configs18:27
EmilienMoh I found it18:28
EmilienMDockerPuppetDebug is set to true18:28
EmilienMcloudnull: unping18:28
* cloudnull unpining 18:29
* EmilienM blames mwhahaha18:29
EmilienMhttps://review.opendev.org/#/c/531015/18:29
EmilienMmwhahaha: should we partial revert it? ^18:29
openstackgerritPooja Jadhav proposed openstack/tripleo-quickstart-extras master: DNM: Testing ipa multinode  https://review.opendev.org/70637318:31
EmilienMekultails: btw on https://review.opendev.org/#/c/718545 - I replied. I couldn't find a good name, although I don't think it'll be easy to re-use that filter18:31
mwhahaha2 years ago?18:32
mwhahahabut we don't see the puppet debug bits in the older versions18:33
mwhahahaor maybe it was partially eaten by the old scripts18:41
mwhahahawho knows18:41
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: tripleo_container_manage: optimize systemd services/healthcheck bootstrap  https://review.opendev.org/71854518:45
*** pojadhav is now known as pojadhav|off18:45
*** mgoddard has joined #tripleo18:46
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: tripleo_bootstrap: do not fail if OVS rpm wasn't found  https://review.opendev.org/71787618:52
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: tripleo_bootstrap: do not fail if OVS rpm wasn't found  https://review.opendev.org/71787618:52
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates master: WIP: refactor nova db config  https://review.opendev.org/71855218:56
openstackgerritMerged openstack/tripleo-ansible stable/train: tripleo_all_nodes_data: sort enabled_services (idempotency)  https://review.opendev.org/71868018:58
ekultailsEmilienM: A lot can change in a few minutes. ;-)19:04
ekultails+119:04
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates stable/train: Move the haproxy iptables rules creation to host_prep_tasks  https://review.opendev.org/71820119:07
*** spsurya_ has quit IRC19:08
owalshfultonj: hi, just spotted something with the multistack inventory...19:11
owalshfultonj: since https://review.opendev.org/713919 the inventory is alway a multistack inventory, even if it's a single stack...19:12
weshay|ruckade_lee, it's working now :) stack@undercloud ~]$ sudo ls /var/lib/certmonger/local/creds19:13
weshay|ruckade_lee, another packaging snafu19:13
owalshfultonj: different structure but ultimately the same inventory, but just noticed this https://review.opendev.org/#/c/713750/9/tripleo_common/inventories.py@10619:14
ade_leeweshay|ruck, sweet19:14
ade_leemwhahaha, cloudnull https://review.opendev.org/#/c/718731/  please19:16
owalshfultonj: I guess I should keep it in train at least19:16
fultonjowalsh: i think i'd need an example to be sure i understand what you're saying19:19
fultonjowalsh: i understand "inventory is alway a multistack inventory even if it's a single stack"19:20
fultonjowalsh: but if i had a diff of before/after then i'd understand more exactly what you mean19:20
owalshfultonj: right, before with single stack the undercloud had a 'plan' var, now it has a 'plans' var19:21
owalshwell, it still has a plan var, but it's == ''19:22
fultonjhttps://review.opendev.org/#/c/713919/2/scripts/tripleo-ansible-inventory19:23
fultonjowalsh: when multistack inventory was introduced, it did exactly what was done in the past (triggered the same code) unless >1 stack was passed19:24
fultonjafter this ^ review the new TripleInventories object is always used19:25
owalshfultonj: yea, that's intentional19:25
fultonjsure19:26
owalshand the new inventory is a superset of the old, except for this one var19:26
fultonjyou made "inventories" handle the 1 stack scenario but perhaps these things you're noticing were overlooked19:26
fultonji say this only because it sounds like you were not expecting the change19:27
owalshfultonj: just missed that line19:27
fultonjexcept for this one var19:27
fultonjplan vs plans19:27
fultonjso a 3rd patch ?19:27
fultonjjust to handle the plan vs plans?19:28
fultonjextgra log around line 10619:28
fultonj"extra logic" around line 10619:29
fultonjthen again, but why merge one inventory19:29
owalshmuch better to have a single code path19:31
owalshor they will diverge/rot19:31
fultonjowalsh: ok, i'll accept that19:32
owalshI'll push a fix just in case anybody/thing uses that19:34
fultonjowalsh: ok, thanks. please add me to the review19:35
openstackgerritAlex Schultz proposed openstack/python-tripleoclient master: Use undercloud upgrade to update all required pkgs  https://review.opendev.org/71878419:48
mwhahahachem, odyssey4me -^ fyi19:49
*** gfidente is now known as gfidente|afk19:53
*** rfolco|bbl is now known as rfolco20:15
*** cylopez has quit IRC20:15
openstackgerritOliver Walsh proposed openstack/tripleo-common master: Include uc plan var when TripleoInventories contains a single plan  https://review.opendev.org/71878920:21
*** raildo_ has joined #tripleo20:24
*** raildo has quit IRC20:26
*** jtomasek has quit IRC20:41
*** owalsh is now known as owalsh_afk20:43
weshay|ruckcloudnull++ https://review.opendev.org/#/c/718759/20:53
weshay|ruckthank you!! it passed20:54
weshay|ruckmwhahaha, can we get a review ^20:54
weshay|ruckpasses here https://logserver.rdoproject.org/59/718759/2/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/805bbf2/20:54
cloudnullawesome!20:54
mwhahahak20:56
weshay|ruckcheck is passing too20:59
weshay|ruckwill hit the gate in a few20:59
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: tripleo_container_manage: optimize systemd services/healthcheck bootstrap  https://review.opendev.org/71854521:03
*** Trevor_V has quit IRC21:05
weshay|ruckcheck it out ya'll.. our new cloud is slower than our old one :)21:09
weshay|rucktripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001SUCCESS in 3h 10m 36s21:09
weshay|rucktripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-vexxhostSUCCESS in 3h 30m 02s21:09
weshay|ruckweeeee21:09
mwhahahaso the usual21:09
mwhahahagot it21:09
EmilienMweshay|ruck: is it using local mirror and stuff?21:10
weshay|ruckEmilienM, ya21:12
weshay|ruckEmilienM, only diff is there is a monthly charge21:12
EmilienMlol21:12
*** fultonj has quit IRC21:12
*** fultonj has joined #tripleo21:14
*** fultonj has quit IRC21:16
*** fultonj has joined #tripleo21:17
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: molecule/podman_*: enable check mode testing  https://review.opendev.org/71881121:17
*** fultonj has quit IRC21:18
EmilienMsshnaidm|off: ^ FYI I'm going to push this for all molecule testing, to cover the dry runs aka check mode in our CI21:20
EmilienMit very often breaks in TripleO and these tests will get us coverage in tripleo-ansible.21:20
EmilienMwe'll miss coverage in tasks defined in THT but it's another story ...21:20
sshnaidm|offEmilienM, ack21:20
weshay|rucksshnaidm|off, CHAG21:21
weshay|rucksshnaidm|off, CHAG21:21
weshay|rucksshnaidm|off, CHAG21:21
* sshnaidm|off off21:21
EmilienMa likely story21:21
weshay|ruckgotta go deal w/ kids.. bbl21:23
*** hberaud has quit IRC21:25
*** hberaud has joined #tripleo21:26
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: standalone/overcloud: enable the HA deployment by default  https://review.opendev.org/35906021:26
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: standalone/overcloud: enable the HA deployment by default  https://review.opendev.org/35906021:27
openstackgerritMerged openstack/puppet-tripleo master: Fix incorrect spec file name  https://review.opendev.org/71857521:36
openstackgerritMerged openstack/puppet-tripleo master: Add missing unit tests for placement manifests  https://review.opendev.org/71857421:37
openstackgerritSlawek Kaplonski proposed openstack/tripleo-heat-templates master: Add support for resource provider bandwidth in Neutron config  https://review.opendev.org/71446421:45
*** cloudnull has quit IRC21:51
openstackgerritSlawek Kaplonski proposed openstack/tripleo-heat-templates master: Add support for resource provider bandwidth in Neutron config  https://review.opendev.org/71446421:53
*** ramishra has quit IRC22:03
*** slaweq has quit IRC22:14
openstackgerritLukas Bezdicka proposed openstack/tripleo-heat-templates stable/train: [Q->T] Introduce Queens to Train Upgrade  https://review.opendev.org/71348222:20
*** raildo_ has quit IRC22:21
rajinirhttps://review.opendev.org/#/c/712533/9 - the rdo job keeps failing on this review. The error is not related , any idea?22:27
rajiniranother one - same error - https://review.opendev.org/#/c/716638/22:29
*** cloudnull has joined #tripleo22:35
*** jmasud has quit IRC22:40
openstackgerritMerged openstack/tripleo-repos master: [ussuri][goal] Drop python 2.7 support and testing  https://review.opendev.org/71740522:42
openstackgerritMerged openstack/tripleo-heat-templates master: Add placement to testing matric  https://review.opendev.org/71810822:42
*** jmasud has joined #tripleo22:43
*** tkajinam has joined #tripleo22:53
*** hjensas has joined #tripleo23:00
mwhahahait's not related23:03
*** tosky has quit IRC23:08
openstackgerritMerged openstack/tripleo-ansible stable/train: 'tripleo_ovs_dpdk_lcore_list' should be optional  https://review.opendev.org/71730323:32
*** rfolco has quit IRC23:37

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!