Friday, 2017-04-21

openstackgerritMerged openstack/puppet-tripleo master: Update UI language list  https://review.openstack.org/45565600:06
openstackgerritMerged openstack/tripleo-heat-templates master: Add NeutronDnsDomain heat option, undercloud fix  https://review.openstack.org/45151700:08
openstackgerritMerged openstack-infra/tripleo-ci master: Automatically read both voting and non-voting forms of job  https://review.openstack.org/45380600:08
*** ooolpbot has joined #tripleo00:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION00:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168025900:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168427200:10
*** ooolpbot has quit IRC00:10
openstackLaunchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)00:10
openstackLaunchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress]00:10
*** artom has quit IRC00:10
*** artom has joined #tripleo00:11
*** artom has quit IRC00:12
*** artom has joined #tripleo00:12
*** rlandy has quit IRC00:17
openstackgerritMerged openstack-infra/tripleo-ci master: Add multinode-upgrades scenario jobs to cistatus  https://review.openstack.org/45380700:40
openstackgerritMerged openstack/instack-undercloud master: Disable Telemetry services on undercloud by default  https://review.openstack.org/45815500:41
openstackgerritMerged openstack/tripleo-heat-templates master: Add environment to preselect only VIP IP addresses  https://review.openstack.org/33674900:41
openstackgerritMerged openstack/tripleo-heat-templates master: Add network_data.yaml to encapsulate list of networks for j2  https://review.openstack.org/40992100:41
*** jerrygb has joined #tripleo00:45
*** limao has joined #tripleo00:50
*** jerrygb has quit IRC00:52
*** tobias_fiberdata has joined #tripleo00:56
openstackgerritOliver Walsh proposed openstack/puppet-tripleo stable/ocata: Configure migration SSH tunnel  https://review.openstack.org/45867401:02
*** jerrygb has joined #tripleo01:04
*** fragatina has quit IRC01:08
*** ooolpbot has joined #tripleo01:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION01:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168025901:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168427201:10
*** ooolpbot has quit IRC01:10
openstackLaunchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)01:10
openstackLaunchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress]01:10
*** newmember has quit IRC01:17
*** eck`gone is now known as eck`01:18
*** fzdarsky_ has joined #tripleo01:21
*** fzdarsky has quit IRC01:24
*** japestinho has joined #tripleo01:58
*** yamahata has quit IRC02:03
*** newmember has joined #tripleo02:07
*** ooolpbot has joined #tripleo02:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION02:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168025902:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168427202:10
*** ooolpbot has quit IRC02:10
openstackLaunchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)02:10
openstackLaunchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress]02:10
*** rbowen has joined #tripleo02:13
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/ocata: Test upgrade jobs from newton to ocata  https://review.openstack.org/45871402:25
*** rbowen has quit IRC02:40
*** Erming_ has joined #tripleo03:03
*** Erming__ has quit IRC03:03
*** chem_gone has quit IRC03:03
*** chem_gone has joined #tripleo03:04
*** bugzy has quit IRC03:04
*** bugzy has joined #tripleo03:06
*** dmacpher has quit IRC03:06
openstackgerritOliver Walsh proposed openstack/puppet-tripleo master: Restrict nova migration ssh tunnel  https://review.openstack.org/45807703:09
*** ooolpbot has joined #tripleo03:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION03:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168025903:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168427203:10
*** ooolpbot has quit IRC03:10
openstackLaunchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)03:10
openstackLaunchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress]03:10
*** portdirect has quit IRC03:16
openstackgerritMerged openstack/tripleo-heat-templates master: Use conditionals for neutron and glance worker defaults  https://review.openstack.org/42542303:16
openstackgerritOliver Walsh proposed openstack/puppet-tripleo master: Restrict nova migration ssh tunnel  https://review.openstack.org/45807703:18
*** fragatina has joined #tripleo03:19
*** dmacpher has joined #tripleo03:19
*** portdirect has joined #tripleo03:23
*** pmannidi has quit IRC03:25
*** tzumainn has quit IRC03:26
*** psahoo has joined #tripleo03:31
openstackgerritMerged openstack/tripleo-heat-templates stable/ocata: N->O upgrade, fix wrong parameters to nova placement.  https://review.openstack.org/45841603:39
*** gkadam has joined #tripleo03:42
*** ykarel has joined #tripleo04:00
*** pmannidi has joined #tripleo04:04
*** ooolpbot has joined #tripleo04:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION04:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168025904:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168427204:10
*** ooolpbot has quit IRC04:10
openstackLaunchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)04:10
openstackLaunchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress]04:10
*** atheurer has quit IRC04:22
*** yamahata has joined #tripleo04:24
*** atheurer has joined #tripleo04:33
*** pgadiya has joined #tripleo04:34
*** udesale has joined #tripleo04:42
*** jaganathan has joined #tripleo04:43
openstackgerritMerged openstack/puppet-tripleo master: Refactor SSHD config to allow both SSHD options and banner/motd to be set  https://review.openstack.org/45758505:00
*** atheurer has quit IRC05:01
*** ratailor has joined #tripleo05:05
*** skramaja has joined #tripleo05:06
*** ooolpbot has joined #tripleo05:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION05:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168025905:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168427205:10
*** ooolpbot has quit IRC05:10
openstackLaunchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)05:10
openstackLaunchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress]05:10
*** apetrich has joined #tripleo05:12
*** jaganathan has quit IRC05:25
*** apetrich has quit IRC05:30
*** apetrich has joined #tripleo05:32
*** prateek has joined #tripleo05:35
*** apetrich has quit IRC05:38
*** janki has joined #tripleo05:48
*** apetrich has joined #tripleo05:50
openstackgerritIan Wienand proposed openstack/diskimage-builder master: BlockDevice - pass command line arguments directly  https://review.openstack.org/45872705:52
*** liupe__ has joined #tripleo05:53
*** dsneddon_afk is now known as dsneddon05:54
*** lmiccini has joined #tripleo05:55
*** yolanda has joined #tripleo05:58
*** jbadiapa has joined #tripleo05:59
openstackgerritMerged openstack/puppet-tripleo stable/ocata: Move ceilometer wsgi to step 3  https://review.openstack.org/45725906:00
openstackgerritMerged openstack-infra/tripleo-ci master: Use AFS mirrors for legacy jobs  https://review.openstack.org/45847406:01
*** brault has quit IRC06:01
openstackgerritMerged openstack/tripleo-heat-templates stable/ocata: Replace references to the 192.0.2 network  https://review.openstack.org/45577506:01
*** brault has joined #tripleo06:01
*** brault has quit IRC06:05
*** jaosorior has joined #tripleo06:06
jaosoriormarios: Hey dude, could you +A this commit? https://review.openstack.org/#/c/458596/ it makes the fakeha-caserver gate to pass again06:07
jaosoriorbandini: ^^06:07
*** iranzo has joined #tripleo06:07
bandinilgtm06:09
*** ooolpbot has joined #tripleo06:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION06:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168025906:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168427206:10
*** ooolpbot has quit IRC06:10
openstackLaunchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)06:10
openstackLaunchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress]06:10
jaosoriorbandini: thanks dude06:10
jaosoriorbandini: hey dude, any idea what would cause this http://logs.openstack.org/03/457603/3/check/gate-tripleo-ci-centos-7-scenario002-multinode-upgrades-nv/94626b6/logs/subnode-2/var/log/barbican/main.txt.gz#_2017-04-20_21_25_41_66506:11
*** liupe__ has quit IRC06:14
*** liupe__ has joined #tripleo06:14
*** rcernin has joined #tripleo06:15
marioso/ jaosorior bandini beat me to it06:15
*** pcaruana has joined #tripleo06:18
*** liupe__ has quit IRC06:19
*** liupe__ has joined #tripleo06:19
*** dparkes has quit IRC06:20
*** aufi has joined #tripleo06:25
bandinijaosorior: looking06:25
jaosoriormarios, bandini thanks dudes06:28
*** liupe__ has quit IRC06:29
*** liupe__ has joined #tripleo06:29
openstackgerritSaravanan KR proposed openstack/tripleo-heat-templates master: Add role specific information to the service template  https://review.openstack.org/43795606:29
*** yprokule has joined #tripleo06:32
*** jaganathan has joined #tripleo06:34
bandinijaosorior: so something is shutting down mysql http://logs.openstack.org/03/457603/3/check/gate-tripleo-ci-centos-7-scenario002-multinode-upgrades-nv/94626b6/logs/subnode-2/var/log/mysqld.txt.gz#_2017-04-20_20_24_24 at the time of failure06:35
* bandini still looking06:35
bandinijaosorior: nevermind I might be blind with the time06:36
jaosoriorthat was one minute before06:36
*** chem_gone is now known as chem06:36
*** tobias_fiberdata has quit IRC06:36
*** chem has quit IRC06:36
*** chem has joined #tripleo06:36
bandiniwell but it takes time for galera to come up06:36
bandinibut it's more like an hour before06:37
bandiniunless one log has some timezone skew in it and the other one does not06:37
*** udesale__ has joined #tripleo06:37
*** tobias_fiberdata has joined #tripleo06:38
jaosorioroh shit, that's write06:38
jaosoriorit's like an hour06:38
jaosoriorwell, damn06:38
jaosoriormaybe barbican just sucks at reconnecting06:38
*** udesale has quit IRC06:40
openstackgerritMerged openstack/tripleo-heat-templates stable/ocata: Add composable role support for NetApp Cinder back end  https://review.openstack.org/45626406:41
openstackgerritMerged openstack/tripleo-heat-templates master: SSHD Service extensions  https://review.openstack.org/44462206:42
*** janki has quit IRC06:49
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: Add ovb-fakeha-caserver (TLS everywhere) job to cistatus  https://review.openstack.org/45846806:50
*** janki has joined #tripleo06:50
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates stable/newton: Add special case upgrade from openvswitch 2.5.0-14  https://review.openstack.org/45873706:50
jaosoriorany reviews on this backport https://review.openstack.org/#/c/458005/ ?06:51
marioschem: stable/newton https://review.openstack.org/458737 :/ had an email about this on a bz06:51
*** gbarros has joined #tripleo06:53
*** chlong has quit IRC06:56
bandinijaosorior: looking06:57
*** dparkes has joined #tripleo06:59
*** liupe__ has quit IRC06:59
jaosoriorbandini: hey, thanks dude06:59
*** liupe__ has joined #tripleo06:59
bandinihas anyone hit "Error: Swift temporary URLs require a shared secret to be created. You must provide "swift_temp_url_key" as a config option." on the undercloud when deploying?06:59
bandiniI keep getting this no matter which repos I use06:59
bandinimarios: do you need reviews on that backport?07:00
mariosbandini: yes please07:01
mariosbandini: haven't come across that swift thing fwiw07:01
bandini"the ghastly openvswitch update saga"07:02
mariosbandini: honestly, it completely ruined my mojo of a friday morning to have to do that07:02
bandinimarios: nothing can break us down on a Friday! ;)07:02
mariosrelevant context bandini https://bugzilla.redhat.com/show_bug.cgi?id=1416088#c5 fwiW07:03
openstackbugzilla.redhat.com bug 1416088 in openvswitch "Rebase openvswitch to 2.5.0.23 in OSP 10" [Medium,New] - Assigned to mburns07:03
* bandini reads07:03
*** jlinkes has joined #tripleo07:03
chemmarios: what about special case upgrade from openvswitch 2.5.0-14 ?07:04
chemmarios: hi by the way :)07:04
bandiniit's needed for osp10/newton I believe07:04
bandinichem: morning sir ;)07:04
marioschem: o/ did you hear, ovs hates us07:04
*** ckyriakidou has joined #tripleo07:05
marioschem: nothing major actually, just looks like they are gearing up for ovs 2.6 in 10+ so i went looking. we landed the *removal* of the workaround on newton, but didn't add the 'new and improved' on with --notriggerun07:05
marioschem: assaf emailed me about it and just posted the backport07:05
marioschem: would appreciate eyes at https://review.openstack.org/#/c/458737/ as you know it is a beautiful review ful of bash and heredoc07:06
chemmarios: looking ...07:08
*** liupe__ has quit IRC07:09
*** mhenkel_ has joined #tripleo07:09
*** liupe__ has joined #tripleo07:09
*** ooolpbot has joined #tripleo07:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION07:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168025907:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168427207:10
*** ooolpbot has quit IRC07:10
openstackLaunchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)07:10
openstackLaunchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress]07:10
*** ffiore has joined #tripleo07:10
*** tesseract has joined #tripleo07:11
bandinimarios: what's "declare -f special_case_ovs_upgrade_if_needed > $UPGRADE_SCRIPT" for in the ceph upgrade script?07:11
bandiniit's probably just my lack of bash-fu I guess07:11
*** florianf has joined #tripleo07:13
openstackgerritMerged openstack/tripleo-heat-templates stable/ocata: Update Dell EMC Cinder back end services  https://review.openstack.org/45638207:14
bandinimarios: ah nm. got it07:14
openstackgerritMartin André proposed openstack/tripleo-common master: Make DockerImageUploader work with missing tags  https://review.openstack.org/45874307:14
*** saibarspeis has joined #tripleo07:15
openstackgerritMartin André proposed openstack/tripleo-quickstart-extras master: Generate the list of required containers from t-h-t  https://review.openstack.org/45857507:15
*** masco has joined #tripleo07:17
bandinimarios, chem: I +Aed that backport. if you did not want that yet for some reason, speak up :)07:17
mariosbandini: thanks yeah we need that07:18
marioschem: bandini right now there is *no* workaround for stable/newton ovs (we were told we didn't need one)07:18
*** liupe__ has quit IRC07:18
*** ckyriakidou has quit IRC07:18
bandinigot it let07:19
bandinis let the jenkins gods do their job then07:19
mariosthanks bandini07:19
*** jprovazn has joined #tripleo07:21
*** udesale has joined #tripleo07:24
*** shardy has joined #tripleo07:24
*** udesale__ has quit IRC07:26
marioschem: lets try a recheck @ https://review.openstack.org/#/c/458530/2 ovb-ha failed on 'not enough hosts'07:29
*** cylopez has joined #tripleo07:30
*** hjensas has joined #tripleo07:34
*** hjensas has quit IRC07:34
*** hjensas has joined #tripleo07:34
chemmarios: ack, but I would say that it's unrelated :)07:36
*** paramite has joined #tripleo07:37
*** jpena|off is now known as jpena07:39
*** bogdando has joined #tripleo07:41
*** jpich has joined #tripleo07:43
*** leanderthal|afk is now known as leanderthal07:47
openstackgerritMerged openstack/tripleo-heat-templates master: glance: deploy services with Keystone v3 endpoints  https://review.openstack.org/44279807:48
openstackgerritMerged openstack-infra/tripleo-ci master: Update fakeha net-iso configuration  https://review.openstack.org/45859607:49
*** janki has quit IRC07:53
jaosoriormandre: could you check this out https://review.openstack.org/#/c/458026/1 ?07:53
mandrejaosorior: looking07:54
mandrejaosorior: I wonder if we really need to make a conditional for internal_tls_enabled07:56
mandrejaosorior: wouldn't bind mounting the directories all the time work?07:56
jaosoriormandre: cause those directories don't exist if internal_tls_enabled is false07:57
*** pmannidi has quit IRC07:57
mandrejaosorior: I don't think it's a problem for docker, AFAIK it will create them07:57
jaosorioruhm07:59
*** zzzeek has quit IRC08:00
*** zzzeek has joined #tripleo08:00
jaosoriormandre: just not too keen on bind mounting empty directories08:00
*** apetrich has quit IRC08:01
*** apetrich has joined #tripleo08:02
mandrejaosorior: I understand... in my mind that it would simplify the templates quite a bit and wouldn't necessitate modification for each new service that runs on top of apache08:04
mandreI don't think mounting empty dirs is necessarily bad if it simplifies the code08:05
mandreanyways, I'm fine merging your patch as it08:06
jaosoriormandre: I think it can merge. Lets bring it up with jistr and other folks as well to get more input, and if necessary, I'll submit another patch cleaning those bits up08:06
mandrejaosorior: sounds good08:07
*** lucas-afk is now known as lucasagomes08:07
*** ooolpbot has joined #tripleo08:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION08:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168025908:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168427208:10
*** ooolpbot has quit IRC08:10
openstackLaunchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)08:10
openstackLaunchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress]08:10
*** jerrygb_ has joined #tripleo08:11
*** janki has joined #tripleo08:11
*** jerrygb has quit IRC08:11
*** derekh has joined #tripleo08:12
*** rcernin has quit IRC08:13
*** assassin is now known as ubijtsa08:15
*** mcornea has joined #tripleo08:16
*** jerrygb_ has quit IRC08:21
*** jerrygb has joined #tripleo08:22
*** karimb has joined #tripleo08:23
shardychem: Hey FYI see https://review.openstack.org/#/c/458510/08:24
shardychem: looks like jaosorior beat you to it adding a str_concat function to heat ;)08:24
*** suuuper has joined #tripleo08:24
*** zoli|gone is now known as zoli08:24
chemshardy: well ... as long as it's there08:25
*** cmyster has quit IRC08:26
*** cmyster has joined #tripleo08:26
*** brault has joined #tripleo08:27
shardychem: cool, I just wanted to make you aware so we don't duplicate effort08:27
chemshardy: thanks08:27
shardychem: there are probably a few more functions we could add where we're using yaql atm, if you're interested we could start an etherpad with the yaql examples and try to show native heat alternatives08:27
chemshardy: anyway it was nowhere near the top of my todo list unfortunatly :)08:28
*** rcernin has joined #tripleo08:28
shardychem: ack, Ok no worries08:28
chemshardy: but yes, it would bu super cool to start something like this, like some fresh air08:28
*** jpich has quit IRC08:29
chems/bu/be/08:29
*** jpich has joined #tripleo08:30
*** shardy has quit IRC08:34
*** shardy has joined #tripleo08:35
openstackgerritSteven Hardy proposed openstack/python-tripleoclient master: Save DefaultPasswords values for undercloud deploy  https://review.openstack.org/45840708:36
*** social has joined #tripleo08:42
jistri'm getting this error when trying to do an `openstack overcloud deploy` to update existing overcloud08:46
jistr2017-04-20 18:34:51 | Unable to create plan. A container with the name overcloud already exists.08:46
jistrit rings a bell vaguely, but i thought it was long fixed08:46
jistrif anyone remembers any additional info around this i'd be grateful for any hints08:47
jistrfwiw here's the log08:47
jistrhttp://logs.openstack.org/84/450784/17/experimental/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/4c05d45/logs/undercloud/home/jenkins/overcloud_upgrade_console.log.txt.gz#_2017-04-20_18_34_5108:47
shardyjistr: is the unable to create plan line just a warning?08:49
shardyjistr: it looks to me like the actual error is tripleo.validations.v1.check_pre_deployment_validations?08:49
shardyand that it seems doesn't allow for multinode as it expects the nodes to exist08:50
jistrshardy: that's what i thought too at first, but see a job where this passed through that step fine:08:50
jistrhttp://logs.openstack.org/84/450784/15/experimental/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/823dfd7/logs/undercloud/home/jenkins/overcloud_upgrade_console.log.txt.gz#_2017-04-19_10_26_1308:50
*** afazekas has quit IRC08:50
jistrit seems it's the contrary: the validations errors are non-fatal, but the plan update issue is08:50
shardyhttps://github.com/openstack/tripleo-common/commit/b69b950a0723cda6a812ecef3d12e7b83f45086408:51
shardyjistr: Hmm, yeah but the way we do the validations has changed recently I think08:51
jistrwe run the deploy with --validation-errors-nonfatal08:51
* jistr looks at the link08:51
shardyhttps://review.openstack.org/#/c/445914/ is the patch, but it did pass some multinode jobs08:52
shardyHmm08:52
jistryea... another point of info -- the upgrade job started failing on this just yesterday. On Wednesday 19th the job got further than this, it actually started the overcloud stack update.08:53
*** fzdarsky_ is now known as fzdarsky|afk08:55
jistri'm even wondering where does the error message come from08:55
jistri found https://github.com/openstack/tripleo-common/blob/6dfd35f95375c671af41558e0a74c07cdbb7dc4a/workbooks/plan_management.yaml#L8208:55
jistrbut that's not the same08:55
*** jrist has quit IRC08:57
jistrah here https://github.com/openstack/tripleo-common/blob/6dfd35f95375c671af41558e0a74c07cdbb7dc4a/tripleo_common/actions/plan.py#L11908:59
*** salmankhan has joined #tripleo08:59
*** stendulker has joined #tripleo08:59
*** tobias-fiberdata has joined #tripleo08:59
*** pgadiya has quit IRC08:59
*** hewbrocca_afk is now known as hewbrocca08:59
*** tobias_fiberdata has quit IRC09:01
*** salmankhan has quit IRC09:01
*** salmankhan has joined #tripleo09:02
openstackgerritSteven Hardy proposed openstack/python-tripleoclient master: Save DefaultPasswords values for undercloud deploy  https://review.openstack.org/45840709:03
*** pgadiya has joined #tripleo09:04
openstackgerritMerged openstack/tripleo-heat-templates stable/ocata: Run token flush cron job hourly by default  https://review.openstack.org/45800509:06
*** ooolpbot has joined #tripleo09:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION09:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168025909:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168427209:10
*** ooolpbot has quit IRC09:10
openstackLaunchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)09:10
openstackLaunchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress]09:10
*** tosky has joined #tripleo09:12
*** brault has quit IRC09:14
*** brault has joined #tripleo09:14
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates master: Don't attempt to configure libvirt when not needed  https://review.openstack.org/45841409:16
openstackgerritMerged openstack/tripleo-heat-templates master: containers: TLS in the internal network for telemetry services  https://review.openstack.org/45802609:18
*** brault has quit IRC09:19
*** flepied has quit IRC09:22
*** limao has quit IRC09:23
*** limao has joined #tripleo09:24
*** flepied has joined #tripleo09:24
*** pgadiya has quit IRC09:28
*** limao has quit IRC09:28
jistrshardy: hmm this looks like a very likely candidate https://review.openstack.org/#/c/45688909:31
jistrshardy: it may be that none of our jobs caught it because they don't upgrade undercloud and overcloud at the same time09:31
jaosoriorjistr: ??09:31
jistrjaosorior: what ?? :)09:32
*** anshul has joined #tripleo09:32
openstackgerritMerged openstack/tripleo-heat-templates stable/newton: Add special case upgrade from openvswitch 2.5.0-14  https://review.openstack.org/45873709:32
*** anshul has quit IRC09:33
*** flepied has quit IRC09:34
*** anshul has joined #tripleo09:35
jaosoriorjistr: nevermind, read up now09:35
jaosoriorjistr: hadn't noticed that you were asking about a bug you're seeing09:36
jistrhehe ok :)09:36
jistryea i'm still chasing it09:36
*** brault has joined #tripleo09:37
*** ramishra has quit IRC09:37
openstackgerritJiri Stransky proposed openstack/instack-undercloud master: Revert "Migrate plans from Mistral to Swift"  https://review.openstack.org/45879209:38
openstackgerritJiri Stransky proposed openstack-infra/tripleo-ci master: Support container upgrades in multinode OOOQ CI  https://review.openstack.org/45078409:40
*** pgadiya has joined #tripleo09:41
jistrdamn09:45
jistri just realized that the revert cannot be tested by CI anyway09:45
jistras the undercloud upgrade will pull the latest delorean, which might be already broken09:45
*** ramishra has joined #tripleo09:45
jistri'll also disable undercloud upgrade in the container upgrade CI job then09:47
jistr /o\09:47
*** dixiaoli has joined #tripleo09:50
*** afazekas has joined #tripleo09:50
*** zoli is now known as zoli|lunch09:52
openstackgerritJiri Stransky proposed openstack/tripleo-quickstart master: Add featureset011 - upgrade from BM to containerized  https://review.openstack.org/45029509:54
shardyjistr: Is it failing on the initial deploy or the upgrade?09:55
jistrshardy: just the upgrade09:55
shardyjistr: Depends-On to a revert should work then, as we overlay the tripleo-ci repo?09:55
jistri'm suspecting that the migration to swift does something that makes it impossible to do a stack update09:55
shardythe problem is if you want to test a revert to the stuff we use before upgrade IME09:55
shardycurrently there's no way to do that, because cross-branch Depends-On doesn't work09:56
jistrshardy: there's no cross branch, it's just master->master still09:56
jistrshardy: but i'm running both undercloud and overcloud upgrade09:56
shardyjistr: Ok, then Depends-On should work I think09:57
jistrso what i thought would be an issue is that zuul sets up the correct instack-undercloud with depends-on09:57
jistrand then i re-setup the repos and run undercloud upgrade09:57
jistrso it will pull latest instack-undercloud regardless of what zuul set up earlier09:57
shardyjistr: for the non-container upgrade job, we enable the tripleo-ci repo before doing the upgrade, along with the other repos we want to upgrade to09:58
shardyand ensure tripleo-ci takes priority09:58
shardyjistr: also, for non-container we don't upgrade the undercloud, we just deploy master undercloud, ocata overcloud, then upgrade the overcloud09:58
shardywhich saves some time09:58
*** dixiaoli has quit IRC09:59
jistrshardy: but is that for OOOQ jobs too? re CI taking priority?10:00
openstackgerritJoe Talerico proposed openstack/tripleo-heat-templates master: Adding Tuned Service  https://review.openstack.org/45412310:00
jistrhere's how we exchange the repos currently10:00
jistrhttps://review.openstack.org/#/c/448576/20/roles/overcloud-upgrade/templates/upgrade-repo-tripleo-ci.sh.j210:00
shardyjistr: I don't know re oooq, but that is how it should work IMO10:00
jistrright yea10:00
*** zoli|lunch is now known as zoli10:00
jistrif there's something not named delorean* with higher prio, then yea it would persist and take priority i think10:01
openstackgerritDamien Ciabrini proposed openstack/tripleo-common master: WIP kolla template overrides for pacemaker HA containers  https://review.openstack.org/45617410:01
jistrhere are the undercloud upgrade repos10:02
jistrhttp://logs.openstack.org/84/450784/17/experimental/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/4c05d45/logs/undercloud/etc/yum.repos.d/10:02
openstackgerritGael Chamoulaud proposed openstack/tripleo-quickstart master: Unpin ansible-lint  https://review.openstack.org/45656410:02
jistrso the gating.repo could be it perhaps10:02
openstackgerritGael Chamoulaud proposed openstack/tripleo-quickstart master: Implement fact caching  https://review.openstack.org/44847810:03
openstackgerritGael Chamoulaud proposed openstack/tripleo-quickstart-extras master: Fix ansible-lint.sh to check playbooks  https://review.openstack.org/44652510:03
shardyjistr: Hmm, no idea what that is10:03
jistrshardy: anyway, i can disable the undercloud upgrade in the containers job anyway, to save time as well10:03
shardyit's always been called delorean-ci, so quickstart should just call it that IMO10:03
*** pgadiya has quit IRC10:04
jistrbut the fact that we generally don't test uc + oc upgrade together means that we miss these bugs10:04
shardyjistr: maybe, but this isn't because of the undercloud upgrade is it?10:04
jistrshardy: i think it likely is10:04
shardyjistr: Hmm, OK - well until now we've viewed separating the two upgrades as a reasonable compromise given the infra timeout constraints10:05
jistrshardy: the topic and merge time of this commit both match in terms of what could potentially be the cause https://review.openstack.org/#/c/456889/10:05
*** ccamacho has joined #tripleo10:07
shardyjistr: aha, yeah I see10:07
shardyjistr: FWIW I didn't expect us to do that, I thought we were going to just put new overcloud data in to swift, and rewrite plan-environment when the plan next gets updated10:08
shardyI guess a forcible undercloud migration is OK, but reverting it should be low impact if we think it's broken things10:08
*** ooolpbot has joined #tripleo10:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION10:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168025910:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168427210:10
*** ooolpbot has quit IRC10:10
openstackLaunchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)10:10
openstackLaunchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress]10:10
*** brault has quit IRC10:10
jistrshardy: not sure if we can reliably prove or disprove that via CI right now though (probably depending on how the repository setup and upgrade works there). I'll see how the upgrade goes now that i've disabled undercloud upgrade, and then maybe try re-enabling it.10:11
*** gfidente has joined #tripleo10:11
*** gfidente has quit IRC10:11
*** gfidente has joined #tripleo10:11
jistri mean i'm not sure if we can prove/disprove what caused the issue10:11
openstackgerritSaravanan KR proposed openstack/tripleo-heat-templates master: Add role specific information to the service template  https://review.openstack.org/43795610:14
openstackgerritMartin André proposed openstack-infra/tripleo-ci master: Add undercloud-containers job to cistatus  https://review.openstack.org/44695710:15
jpichjistr: I've reviewed a few of Ana's patches around this work and am taking over them while she's away for a few weeks, looking at the logs you pasted earlier now to see if something rings a bell10:15
jistrjpich: awesome, thanks :)10:15
jpichjistr: I recall a conversation between akrivoka and d0ugal around having the migration happen during the undercloud upgrade to avoid... $some_kind_of_problem... but I don't remember which one so I'm gonna do some IRC log digging too :)10:16
*** zoli is now known as zoli|lunch10:16
*** cylopez has quit IRC10:18
jistrjpich: thanks10:18
*** pgadiya has joined #tripleo10:19
jistrjpich: just for tl;dr context -- currently we don't have much of a lead as to why the job started failing this way aside from the time (somewhere during 19th and 20th). And due to what's the error message, the cause *might* be https://review.openstack.org/#/c/456889/ but we don't know that for sure.10:19
jpichjistr: Thank you10:21
shardyjistr: ack, well FWIW I'd try to prove it with a Depends-On to the instack-undercloud revert, hopefully we can get that to work via oooq10:23
jistryea, for now i've also disabled the UC upgrade completely, just to see if that helps in general, and then we can try to re-enabling it again10:26
jistralso linked good/bad job logs at https://bugs.launchpad.net/tripleo/+bug/168517110:26
openstackLaunchpad bug 1685171 in tripleo "updating overcloud after undercloud upgrade: Unable to create plan. A container with the name overcloud already exists." [High,In progress] - Assigned to Jiří Stránský (jistr)10:26
* jistr lunch, brb10:26
bandinijistr, shardy: since you mentioned some validations being borked, I just filed https://bugs.launchpad.net/tripleo/+bug/1685177 because I see them non working in my deploy10:29
openstackLaunchpad bug 1685177 in tripleo "Since moving some validations to mistral deploy warnings seem to be broken" [Undecided,New]10:29
bandinithrash|g0ne: can you take a look at https://bugs.launchpad.net/tripleo/+bug/1685177 if you have some time?10:30
*** brault has joined #tripleo10:34
*** stendulker has quit IRC10:34
*** brault has quit IRC10:39
*** jtomasek_ has joined #tripleo10:43
yolandahi, hitting this on an ocata deploy using quickstart10:44
yolandaoslo_db.exception.DBError: (pymysql.err.InternalError) (1061, u"Duplicate key name 'uniq_ports0standard_attr_id'") [SQL: u'ALTER TABLE ports ADD CONSTRAINT uniq_ports0standard_attr_id UNIQUE (standard_attr_id)']10:44
yolandais that a known issue?10:44
*** salmankhan has quit IRC10:52
*** salmankhan has joined #tripleo10:52
*** udesale has quit IRC10:53
*** pgadiya has quit IRC10:54
*** atoth has joined #tripleo10:54
*** dbecker has quit IRC10:55
*** yamahata has quit IRC11:00
*** abishop has quit IRC11:01
*** pgadiya has joined #tripleo11:07
*** ooolpbot has joined #tripleo11:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION11:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168025911:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168427211:10
*** ooolpbot has quit IRC11:10
openstackLaunchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)11:10
openstackLaunchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress]11:10
*** zoli|lunch is now known as zoli11:15
*** tvignaud has quit IRC11:15
EmilienMhello11:15
EmilienMcschwede: can you please look https://review.openstack.org/#/c/458714/ and compare the runtime of upgrade jobs with the ones we had yesterday?11:17
openstackgerritJuan Antonio Osorio Robles proposed openstack/instack-undercloud master: Replace mistral command with OpenStack CLI  https://review.openstack.org/45880811:23
*** gkadam has quit IRC11:24
*** lucasagomes is now known as lucas-afk11:25
jaosoriorEmilienM: the barbican issue is pretty funky, I'll need to reproduce it but it's gonna take a bit.11:27
EmilienMjaosorior: cool11:30
*** masco has quit IRC11:32
*** pkovar has joined #tripleo11:32
*** shardy is now known as shardy_lunch11:32
yolandaEmilienM, do you know about that failures deploying undercloud on ocata?11:33
yolandaor jaosorior11:33
jaosoriorI don't :(11:33
EmilienMyolanda: what repos do you use.?11:34
yolandajust a quickstart default rdo deploy11:34
yolandain that case, nothing customized11:34
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs  https://review.openstack.org/35921511:36
owalshjistr: hola11:36
jistrowalsh: heya i'm just reading your comment ;)11:36
owalshjistr: I'm probably missing something crucial :-)11:37
cschwedeEmilienM: i had a look; gate-tripleo-ci-centos-7-scenario002-multinode-upgrades was much faster, but gate-tripleo-ci-centos-7-multinode-upgrades and gate-tripleo-ci-centos-7-scenario001-multinode-upgrades are slow again11:37
cschwedeEmilienM: also, gate-tripleo-ci-centos-7-scenario001-multinode-upgrades would have been cancelled again without the timeout increase11:37
owalshjistr: but I don't understand why tripleo::profile::base::nova::libvirt_enabled would be true in e.g he nova-api service, unless it's picking up the nova-libvirt hiera11:38
jistrowalsh: indeed we discussed this with mandre earlier today11:38
jistrowalsh: we have an issue with the container puppet things11:38
jistrowalsh: the hiera is global on the host, not per-container11:39
*** abishop has joined #tripleo11:39
jistrso hiera settings from across all services get mangled together11:39
owalshahhhh, that's what I was missing ^^^11:39
jistrfurther interesting might be that this is actually beneficial in some cases11:40
jistri think jaosorior had some use case for this with certmonger, IIRC11:40
jistrbut it breaks us in other cases, like this libvirt issue11:41
owalshprobably should move the migration bits out of the base nova profile, it's not common to all nova services11:41
EmilienMcschwede: I think it depends on which clouds it runs11:41
EmilienMcschwede: see my last comment on the bug report https://bugs.launchpad.net/tripleo/+bug/168025911:42
openstackLaunchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)11:42
*** abishop_ has joined #tripleo11:42
jistrowalsh: hmm i'm not sure about this, but doesn't e.g. nova-api need to have some of the migration settings as well? It's also configuring nova.conf, not just libvirt.11:42
*** morazi has joined #tripleo11:43
cschwedeEmilienM: good catch! wondering if that is related to slower network connectivity or slower cpus?11:43
*** tobias-fiberdata has quit IRC11:43
openstackgerritBrad P. Crochet proposed openstack/puppet-tripleo master: Enable mistral to run under mod_wsgi  https://review.openstack.org/45850311:43
*** abishop has quit IRC11:43
*** thrash|g0ne is now known as thrash11:43
thrashbandini: sure11:44
jistrowalsh: either way for the short term immediate fix we could probably just set the libvirt_enabled option to false in docker nova-compute, effectively disabling the migration.11:45
*** ratailor has quit IRC11:45
*** tvignaud has joined #tripleo11:45
jistrjpich, shardy_lunch: FWIW https://bugs.launchpad.net/tripleo/+bug/1685171/comments/311:47
openstackLaunchpad bug 1685171 in tripleo "updating overcloud after undercloud upgrade: Unable to create plan. A container with the name overcloud already exists." [High,In progress] - Assigned to Jiří Stránský (jistr)11:47
owalshjistr: probably nova-libvirt ^^ but yea... I think disable it. Docker support is the next big thing on my list once the other migration patches land11:47
jistrowalsh: ok great, thanks for the involvement on that patch! i wasn't aware of the things you brought up11:48
*** fzdarsky|afk is now known as fzdarsky11:48
owalshjistr: and nova-api doens't need any migration settings, it's just the nova-compute nova.conf and the libvirtd.conf11:49
EmilienMcschwede: probably a little mix of both11:49
EmilienMcschwede: I would continue the discussion on infra channel, when pabelanger is up11:49
jistrowalsh: ack, thanks11:50
*** dougbtv_ has joined #tripleo11:50
*** dougbtv_ is now known as dougbtv|laptop11:50
owalshnp11:51
*** tobias-fiberdata has joined #tripleo11:51
*** prateek has quit IRC11:52
jistrowalsh: any opinion whether i should set to false manage_migration, or just libvirt_enabled?11:55
*** jpena is now known as jpena|lunch11:55
arxcruzEmilienM: Thanks for review my patch. If you don't mind can you take a look at https://review.openstack.org/#/c/455210/ too?11:56
EmilienMarxcruz: done11:59
arxcruzEmilienM: thanks!11:59
yolandamaybe it was a glitch, second try went fine11:59
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/ocata: Test upgrade jobs from newton to ocata  https://review.openstack.org/45871411:59
EmilienMyolanda: good12:00
*** salmankhan has quit IRC12:00
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/ocata: Test upgrade jobs from newton to ocata  https://review.openstack.org/45871412:00
EmilienMcschwede: trying again to see if it's consistent ^12:00
*** brault has joined #tripleo12:01
jistrowalsh: i'll try just libvirt_enabled. I see that manage_migration is set in nova-compute.yaml as well, and if that comes after nova-libvirt.yaml, it would override its setting. We might need to implement hieradata isolation between puppet containers to get rid of these issues...12:01
jistr(and create new ones... :)) )12:01
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-heat-templates master: Updated from global requirements  https://review.openstack.org/45603012:02
*** trown|outtypewww is now known as trown12:03
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates master: Don't attempt to configure live migration  https://review.openstack.org/45841412:05
openstackgerritJames Slagle proposed openstack/tripleo-common stable/ocata: Add MigrationSshKey to generated passwords  https://review.openstack.org/45867112:06
*** rbowen has joined #tripleo12:09
*** ooolpbot has joined #tripleo12:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION12:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168025912:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168427212:10
*** ooolpbot has quit IRC12:10
openstackLaunchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)12:10
openstackLaunchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress]12:10
openstackgerritOliver Walsh proposed openstack/puppet-tripleo master: Restrict nova migration ssh tunnel  https://review.openstack.org/45807712:11
*** jaosorior is now known as jaosorior_away12:12
openstackgerritMerged openstack/tripleo-heat-templates stable/ocata: N->O Manual puppet commands have the right modulepath.  https://review.openstack.org/45853012:14
openstackgerritMerged openstack/tripleo-heat-templates master: Add defaults for docker puppet tasks  https://review.openstack.org/45571812:20
*** lmiccini has quit IRC12:22
jpichjistr: Thank you for the update, been combing the mistral logs but not finding anything obvious12:22
*** hewbrocca has quit IRC12:23
*** markmc` has joined #tripleo12:23
*** shardy_lunch is now known as shardy12:25
*** markmc has quit IRC12:25
*** amoralej has quit IRC12:25
*** jpena|lunch has quit IRC12:25
*** dmanchad has quit IRC12:26
*** lucas-afk is now known as lucasagomes12:26
bandinithrash: thanks ;)12:33
*** lmiccini has joined #tripleo12:33
*** jtomasek_ has quit IRC12:34
*** jchhatbar has joined #tripleo12:34
owalshjistr: just commented - I think we will need to set manage_migration=false12:35
*** hjensas has quit IRC12:37
*** brault has quit IRC12:38
*** dmanchad has joined #tripleo12:38
*** janki has quit IRC12:38
*** jpena has joined #tripleo12:38
*** markmc` has quit IRC12:41
*** amoralej has joined #tripleo12:43
EmilienMzaneb: could we try https://review.openstack.org/#/c/422983/ in triploe-ci? (depends-on or check experimental) so we can see if it actually helps12:44
*** markmc has joined #tripleo12:46
*** yprokule has quit IRC12:48
*** Goneri has joined #tripleo12:49
*** rlandy has joined #tripleo12:50
*** tvignaud has quit IRC12:50
dmsimardEmilienM: I don't know what to do about https://review.openstack.org/#/c/448108/, I've rechecked plenty of times and it keeps failing on unrelated things12:53
EmilienMdmsimard: looking12:54
EmilienMdmsimard: +212:55
*** hewbrocca_afk has joined #tripleo12:55
openstackgerritMerged openstack/puppet-tripleo master: Add ML2 configuration for Bagpipe BGPVPN extension  https://review.openstack.org/45615312:55
*** hewbrocca_afk is now known as hewbrocca12:55
*** lblanchard has joined #tripleo12:55
openstackgerritMerged openstack/tripleo-heat-templates master: Use -net=host for docker-puppet.py config gen  https://review.openstack.org/45579412:56
*** jmelvin has joined #tripleo12:56
*** jrist has joined #tripleo12:59
*** saibarspeis has quit IRC12:59
*** pgadiya has quit IRC13:00
*** lblanchard1 has joined #tripleo13:01
openstackgerritMerged openstack/puppet-tripleo master: Add resource profile for vmware nsx_v3  https://review.openstack.org/45204713:01
*** dprince has joined #tripleo13:02
*** tvignaud has joined #tripleo13:03
*** salmankhan has joined #tripleo13:04
*** lblanchard has quit IRC13:05
openstackgerritOliver Walsh proposed openstack/puppet-tripleo stable/ocata: SSHD Service extensions  https://review.openstack.org/45882713:06
openstackgerritOliver Walsh proposed openstack/puppet-tripleo stable/ocata: Stop SSHD profile clobbering SSH client config  https://review.openstack.org/45882813:07
openstackgerritOliver Walsh proposed openstack/puppet-tripleo stable/ocata: Refactor SSHD config to allow both SSHD options and banner/motd to be set  https://review.openstack.org/45883013:08
openstackgerritDamien Ciabrini proposed openstack/puppet-tripleo master: WIP containerized pacemaker haproxy  https://review.openstack.org/45883213:10
*** ooolpbot has joined #tripleo13:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION13:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168025913:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168427213:10
*** ooolpbot has quit IRC13:10
openstackLaunchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)13:10
openstackLaunchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress]13:10
openstackgerritMerged openstack/puppet-tripleo master: Cover gnocchi api step 4 and 5  https://review.openstack.org/45637613:10
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates stable/ocata: SSHD Service extensions  https://review.openstack.org/45883613:14
*** thrash is now known as thrash|biab13:14
*** rbrady has joined #tripleo13:17
*** Goneri has quit IRC13:17
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-heat-templates master: Updated from global requirements  https://review.openstack.org/45603013:17
openstackgerritSteven Hardy proposed openstack/python-tripleoclient master: Save DefaultPasswords values for undercloud deploy  https://review.openstack.org/45840713:18
jistrowalsh: ok thanks, i'll see how far we get with the current CI run in progress and then i'll update the patch13:19
openstackgerritHarry Rybacki proposed openstack/tripleo-quickstart master: Add supplemental node provisioner  https://review.openstack.org/45152313:22
openstackgerritHarry Rybacki proposed openstack/tripleo-quickstart master: Update quickstart-extras to deploy FreeIPA  https://review.openstack.org/45322313:22
*** dsariel has joined #tripleo13:23
*** dhill__ has joined #tripleo13:23
*** tzumainn has joined #tripleo13:24
openstackgerritHarry Rybacki proposed openstack/tripleo-quickstart-extras master: Add role to deploy FreeIPA  https://review.openstack.org/43619813:24
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates master: Don't attempt to configure live migration  https://review.openstack.org/45841413:25
owalshjistr: should pass CI for now I would think13:25
owalshjistr: but I'm pretty sure I'll break it soon :-)13:26
jistrowalsh: i realized i mentioned libvirt_enabled in commit message but set manage_migration in the code, but only on nova-libvirt so i should set it also in nova-compute (last one wins, and we don't want to take chances which one it will be). So i updated the patch right now ^^ and i'll retrigger CI.13:26
skramajajtomasek: hey, as per the comments, looks like you are ok with the derived params spec as such, barring the implementation details we disucssion on the patch, we can continue discussing on the code review, if agree, can you accept the spec?13:27
EmilienMflaper87, shardy, bnemec: I hope you can join https://etherpad.openstack.org/p/BOS-forum-future-of-configuration-management13:28
owalshjistr: cool :-) LGTM, maybe open an LP for migration support in docker?13:28
rooktrozet: hey - did you want to review the Tuned work?13:29
rooks/docker/moby13:29
*** amoralej is now known as amoralej|lunch13:29
*** liverpooler has joined #tripleo13:31
jistrowalsh: yup https://bugs.launchpad.net/tripleo/+bug/168521913:32
openstackLaunchpad bug 1685219 in tripleo "Nova migration support in dockerized deployments" [Medium,Triaged]13:32
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates master: Combine writing docker-puppet json with host_prep_tasks  https://review.openstack.org/45630713:33
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates master: Use ansible to write the docker json configs  https://review.openstack.org/45481613:33
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates master: Move DockerConfigJsonStartupData to HostPrepConfig  https://review.openstack.org/45631013:33
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates master: Move KollaJsonConfig to HostPrepConfig  https://review.openstack.org/45631113:33
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates master: Move create-config-dir steps to HostPrepConfig ansible tasks  https://review.openstack.org/45630813:33
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates master: Move GenerateConfig to HostPrepConfig  https://review.openstack.org/45630913:33
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates master: Convert DockerPuppetTasks to run via ansible  https://review.openstack.org/45625413:33
*** newmember has quit IRC13:33
*** flepied has joined #tripleo13:34
honzajtomasek: florianf: would you be really sad if I disabled colors in redux-logger?13:36
*** psahoo has quit IRC13:36
rnoriegashardy, ping13:37
shardyrnoriega: Hi13:38
thervedtrainor, You around? I have a question about the undercloud SSL certificate, if you know about it13:38
rnoriegashardy, hi! reading your blogpost about custom roleS: http://hardysteven.blogspot.com.es/2016/10/tripleo-composablecustom-roles.html13:38
rnoriegashardy, I have one small question. Do you have 1 min?13:39
openstackgerritMerged openstack/puppet-tripleo master: Allow to configure haproxy daemon's status  https://review.openstack.org/45802513:39
rnoriegashardy, awesome blog btw, very educative.13:39
shardyrnoriega: sure, and thanks!13:39
rnoriegashardy, how do you assign, for example, the NtpRole to a introspected node??13:40
florianfhonza: Yes, very sad. Can you add animations or emoji instead?13:40
honza<313:41
shardyrnoriega: you tag the nodes with a particular role, which is then selected by the flavor, e.g "controller" "compute" or whatever13:41
*** brault has joined #tripleo13:41
rnoriegashardy, yes, but for instance, the role name is Controller...but the profile (flavor) is called Control13:41
rnoriegashardy, how do you map flavor-role?13:41
shardyrnoriega: we don't, all roles default to the baremetal flavor13:42
shardyrnoriega: but you can pass an environment file with MyRoleFlavor: myroleflavor13:42
*** kjw3 has joined #tripleo13:42
shardyhttps://github.com/openstack/tripleo-heat-templates/blob/master/puppet/role.role.j2.yaml#L613:43
shardyrnoriega: actually OvercloudMyRoleFlavor: myroleflavor13:43
shardythe ^Overcloud thing is a historical inconsistency we couldn't remove13:43
*** chlong has joined #tripleo13:43
*** shardy is now known as shardy_afk13:44
rnoriegashardy, so you say I'd have to add a OvercloudNtpRoleFlavor section and the default would be a new flavor I create in the undercloud.13:45
rnoriegashardy_afk, ok...I think I get it13:45
*** thrash|biab is now known as thrash13:49
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Parameterize OVB stack flavors  https://review.openstack.org/44516313:50
*** jchhatbar has quit IRC13:50
*** gbarros has quit IRC13:54
rnoriegashardy_afk, nope, I don't get it... coz Overcloud{{role}}Flavor: ends up in a OvercloudControlFlavor in the compute-role.yaml13:54
rnoriegashardy_afk, and not a OvercloudControllerFlavor which is the name of the role...13:55
*** ratailor has joined #tripleo13:56
*** prateek has joined #tripleo13:57
openstackgerritMerged openstack/puppet-tripleo stable/ocata: Configure migration SSH tunnel  https://review.openstack.org/45867413:58
*** prateek has quit IRC13:58
openstackgerritMerged openstack-infra/tripleo-ci master: Adding subunit files to artcl_collect_list  https://review.openstack.org/45521913:58
openstackgerritMerged openstack/tripleo-ui master: Add Toolbar component and filters state  https://review.openstack.org/45217513:58
openstackgerritMerged openstack/tripleo-quickstart-extras master: Adding testrepository.subunit to artcl_collect_list  https://review.openstack.org/45521013:58
dtrainortherve, sure, what's up?14:00
jtomasekhonza: ouch colors!14:01
*** ykarel has quit IRC14:01
jtomasekhonza: what is the problem you're hitting? is the output mangled?14:01
honzajtomasek: color control characters look out of place in a server-side log file14:02
jtomasekskramaja: ok, will do14:02
skramajathanks jtomasek14:02
florianfhonza: that sounds like a good enough reason to drop colors IMO.14:03
honzajtomasek: florianf: e.g. "%c prev state,color: #9E9E9E; font-weight: bold,[object Object]"14:03
jpichjistr: I think I understand what is happening, and believe you got the right patch for the revert... However, it looks like it should be solvable by commenting out one line as opposed to reverting the whole thing14:04
thervedtrainor, I'd like to modify the cert, to change subject alt name. Do you have an idea how I can do it?14:04
*** lblanchard1 has quit IRC14:04
*** colonwq has joined #tripleo14:05
jaosorior_awaytherve: which cert?14:05
*** jaosorior_away is now known as jaosorior_mtg14:05
thervejaosorior_mtg, The undercloud pem file14:05
jaosorior_mtgyou would need this commit https://review.openstack.org/#/c/457619/14:06
jistrjpich: sounds great, i'd very much prefer a fix than a revert. I think we're not in an emergency as it looks like for the time being, given that the upgrade in the job i'm implementing is master->master anyway, i should be able to just temporarily disable it (since it shouldn't effectively do anything anyway :) ).14:06
jtomasekhonza: I am fine with disabling colors, it is not a big deal14:06
jaosorior_mtgtherve: and then support would be needed to be added for the undercloud14:06
thervejaosorior_mtg, Yeah I mostly just want to make a local hack for now14:06
jistrjpich: the undercloud upgrade would become important when we start testing actual ocata->master upgrade, if we're testing both uc + oc upgrades in one job14:06
*** dsariel has quit IRC14:07
jistrjpich: but we're not at that point yet :) So we don't have to hurry with any reverting there.14:07
jaosorior_mtgyou could try to resubmit the cert via certmonger14:07
*** amoralej|lunch is now known as amoralej14:08
dmsimardany tripleo-ci core want to look at https://review.openstack.org/#/c/448108/ ? It's a trivial outstanding change for a month now.14:09
thervejaosorior_mtg, Yeah I think I'm looking for the right getcert call, but I can't find it for now14:09
jpichjistr: Cool, thank you! Yes I would prefer to fix it soon if we can too. Would you prefer I submit a separate patch with that small change, or modify the patch you have up?14:09
jaosorior_mtgtherve: getcert list    -> will give you the nickname14:10
jaosorior_mtgtherve: and you're looking for getcert resubmit14:10
*** ooolpbot has joined #tripleo14:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION14:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168025914:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168427214:10
*** ooolpbot has quit IRC14:10
openstackLaunchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)14:10
openstackLaunchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress]14:10
jistrjpich: i think a separate one would be better perhaps?. I'll -2-A the revert for now.14:10
jpichjistr: Considering the difficulties around testing the revert this morning, I guess there's no simple way to test and confirm it does work before the patch gets merged...?14:10
thervejaosorior_mtg, Yeah, I get "State NEED_GUIDANCE, stuck: yes."14:10
jaosorior_mtgah shit14:10
jpichjistr: Ok, will prepare that now14:10
jaosorior_mtgtherve: how did you call it?14:11
thervejaosorior_mtg, getcert resubmit -i "undercloud-haproxy-public-cert" -w -v -U id-kp-clientAuth -A 192.168.24.2 -U id-kp-serverAuth14:11
skramajashardy_afk: EmilienM - all open ends are clarified for the spec - https://review.openstack.org/#/c/423304/, can we merge it? - fyi: fultonj14:11
EmilienMskramaja: done14:12
skramajathanks EmilienM14:12
jaosorior_mtgtherve: funky, that looks about right14:13
jistrjpich: re testing before merging i'm not 100% sure -- it depends on how the zuul patch-in-question repos work with OOOQ, which i haven't dug through yet. We might try testing it then and hope that the repos will be correct. Or we could just merge it too, if it passes all the standard jobs.14:13
jaosorior_mtgtherve: maybe you gotta specify the key and cert file again?14:13
jistrjpich: we can sync about that further. Either way thanks a lot for looking into the bug :)14:14
*** dixiaoli has joined #tripleo14:15
thervejaosorior_mtg, OK I'll try14:15
openstackgerritMerged openstack/tripleo-specs master: Deriving TripleO Parameters  https://review.openstack.org/42330414:16
*** jcoufal has joined #tripleo14:16
*** shardy_afk is now known as shardy14:17
numansit would be really nice if some one could review the OVN patches - https://review.openstack.org/#/c/372274/ and https://review.openstack.org/#/c/436767/14:17
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates master: Test I71f8aa431a60457326167b8c82adc03ca750eda6  https://review.openstack.org/45885814:18
shardyrnoriega: yeah we had some inconsistent naming for historical reasons, and we couldn't just change the parameters to match the role name14:18
jpichjistr: No problems! :) I think the change will be minor enough to just "merge & test" but we'll talk then, I could just be reckless14:19
fultonjThanks EmilienM14:19
shardyrnoriega: for all custom roles we use the role.role.j2.yaml though, which is Overcloud$RoleFlavor14:19
EmilienMfultonj: thanks for pushing it :)14:19
*** dixiaoli has quit IRC14:19
rnoriegashardy, so If I create a flavor named MyRole...it'll work14:19
shardyrnoriega: yes create a flavor called myrole, then add OvercloudMyRoleFlavor: myrole to an environment file14:20
shardyrnoriega: then set the capabilities in the myrole flavor to match whatever you tag the nodes with14:20
*** bnemec is now known as beekneemech14:20
shardye.g myrole or whatever tag/profile you want14:20
*** yamahata has joined #tripleo14:21
rnoriegashardy, good! got it! thanks!!14:21
shardyrnoriega: np!14:21
zanebEmilienM: ControllerServiceChain 101.014:24
zanebhttp://logs.openstack.org/83/422983/20/experimental/gate-tripleo-ci-centos-7-nonha-multinode-oooq/577f861/logs/postci.txt.gz14:24
*** rcernin has quit IRC14:24
*** brault has quit IRC14:25
EmilienMzaneb: sounds better, no?14:25
zanebEmilienM: yeah, down from like 50014:25
EmilienM:)14:26
EmilienMnice14:26
zanebControllerServiceChain 547.0 in https://bugs.launchpad.net/tripleo/+bug/1684272/comments/214:26
openstackLaunchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress]14:26
EmilienMzaneb: should we cancel the revert?14:26
zanebI think it's ok to revert14:26
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates master: WIP THT-part of galera ha container  https://review.openstack.org/45534814:27
*** brault has joined #tripleo14:28
thervejaosorior_mtg, http://paste.openstack.org/show/607472/ does that look correct?14:29
*** atheurer has joined #tripleo14:29
jistrjpich: read your comment with analysis of the problem -- good catch! So the hope is that we'd just be able to stop deleting the mistral environment for now?14:29
jistrso that the original code paths in tripleo-common work as expected?14:29
*** paramite has quit IRC14:29
jpichjistr: Yes!14:29
jistrsounds good14:29
*** trozet has quit IRC14:29
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates master: WIP THT-part of rabbitmq HA docker container  https://review.openstack.org/45288914:30
openstackgerritJulie Pichon proposed openstack/instack-undercloud master: Plan format migration: keep the Mistral env around  https://review.openstack.org/45885914:30
*** skramaja has quit IRC14:30
jaosorior_mtgtherve: well, that looks well, although that's a request and not a resubmit14:30
thervejaosorior_mtg, Yeah I did a stop-tracking beforehand14:30
*** michapma_dsk has joined #tripleo14:30
jaosorior_mtgtherve: oh, alright14:30
thervejaosorior_mtg, What's that supposed to generate though? :)14:31
jaosorior_mtgtherve: you might wanna clean up the cert file as well.14:32
jaosorior_mtgtherve: it generates the cert file that you give with the -f flag14:32
thervejaosorior_mtg, So remove it beforehand?14:32
jaosorior_mtgtherve: yeah14:32
openstackgerritMichele Baldessari proposed openstack/puppet-tripleo master: WIP rabbitmq HA docker container  https://review.openstack.org/45518914:32
*** jaganathan has quit IRC14:34
*** hjensas has joined #tripleo14:36
*** hjensas has quit IRC14:36
*** hjensas has joined #tripleo14:36
openstackgerritMichele Baldessari proposed openstack/puppet-tripleo master: WIP galera HA docker container  https://review.openstack.org/45534514:36
chandankumartrown: ggillies https://review.openstack.org/#/c/458013/ please have a look, thanks!14:41
*** pliu has quit IRC14:41
*** adarazs has quit IRC14:42
*** slagle has quit IRC14:42
*** marios has quit IRC14:43
*** lhinds has quit IRC14:43
*** gchamoul has quit IRC14:43
*** cschwede has quit IRC14:43
*** jschlueter has quit IRC14:43
*** myoung has quit IRC14:43
openstackgerritNuman Siddique proposed openstack/puppet-tripleo master: Pacemaker support for OVN DB servers  https://review.openstack.org/37227414:43
*** stevebaker has quit IRC14:44
*** rnoriega has quit IRC14:44
*** fultonj has quit IRC14:44
*** zoli has quit IRC14:44
*** trozet has joined #tripleo14:44
*** dparkes has quit IRC14:47
pradkcan i get some reviews on https://review.openstack.org/#/c/456385/14:50
openstackgerritPradeep Kilambi proposed openstack/puppet-tripleo master: Move ceilometer upgrade re-run out of collector  https://review.openstack.org/45803614:50
*** atheurer has quit IRC14:50
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates master: Remove no longer used environment files - older upgrade workflows  https://review.openstack.org/45886414:51
dtrainortherve, sorry i got stuck in a call.  was jaosorior_mtg's answer sufficient?14:51
*** atheurer has joined #tripleo14:51
thervedtrainor, I'm progressing :)14:51
dtrainoryou would need to interface with certmonger, looks like https://review.openstack.org/#/c/457619/ is a good start14:51
*** marios has joined #tripleo14:52
thervehttp://paste.openstack.org/show/607473/ stuck with NEED_GUIDANCE again14:52
dtrainorugh i exploded my environment14:53
dtrainorgive me a few minutes here14:53
*** chlong_ has joined #tripleo14:54
*** chlong_ has quit IRC14:54
dtrainoroff the top of my head it looks good - is it not working?14:54
*** atheurer has quit IRC14:55
*** ratailor has quit IRC14:56
*** lmiccini has quit IRC14:56
*** lmiccini has joined #tripleo14:57
jaosorior_mtgtherve: if all else fails, look for ayoung, he's pretty versed with this issues.14:58
*** markmc has quit IRC14:58
*** jaosorior_mtg is now known as jaosorior14:58
jaosorioralright, I'm off. Have a good weekend everyone!14:58
thervejaosorior, Cheers :)14:59
*** jaosorior has quit IRC14:59
*** florianf has quit IRC14:59
thervedtrainor, Yeah it doesn't work14:59
EmilienMouch, a lof of Red Hat servers are down (or networking issue) at this time15:00
*** michapma_dsk has quit IRC15:00
EmilienMit might affect our CI, I would be careful before rechecking15:01
therveEmilienM, What did you do again15:01
EmilienMtherve: bugzilla is down, but also a bunch of other things15:01
beekneemechOh good, so it's not just me.15:01
dtrainorvpn appears to be affected too15:01
EmilienMdtrainor: try rdu215:01
*** markmc has joined #tripleo15:01
dtrainorya a friend said that worked15:01
jristEmilienM: phx15:02
jristis down15:02
* dtrainor peeks out the window15:02
jristprobably the whole data center15:02
jrist:)15:02
dtrainorwell, the dc isn't on fire15:02
*** flepied has quit IRC15:02
jristlol15:02
dtrainori can confirm that much at least15:02
jristdon't let the smoke escape!15:02
dmsimardFYI PHX2 is confirmed down right now, bunch of things affected -- rdoproject.org, bugzilla, etc15:03
dtrainorno smoke til brooklyn15:03
*** yamahata has quit IRC15:03
*** yamahata_ has joined #tripleo15:03
EmilienMhopefully we now use AFS mirrors :D15:04
*** ykarel has joined #tripleo15:04
dmsimardtrunk.rdo is okay actually15:04
EmilienMhttps://trunk.rdoproject.org/centos7-master/consistent/ is not down15:04
*** aufi has quit IRC15:04
EmilienMyeah15:04
dmsimardit's not hosted out of phx215:04
EmilienMI think tripleo CI shouldn't be affected15:04
EmilienMbut just in case15:04
dmsimardand buildlogs is out of RDU15:04
EmilienMif you see an outage in our CI, please let us know here15:04
dmsimardbasically OS1 and RDO Cloud are down15:05
dmsimardamongst other things15:05
*** leanderthal is now known as leanderthal|afk15:05
thervedtrainor, Hum no it does GENERATING_KEY_PAIR in a loop :/15:05
*** brault has quit IRC15:05
*** atheurer has joined #tripleo15:06
dtrainortherve, as soon as i can get to these systems again i'll give it a stab15:06
dtrainorbut that is bizarre15:06
*** brault has joined #tripleo15:07
*** slagle has joined #tripleo15:07
*** newmember has joined #tripleo15:08
*** ooolpbot has joined #tripleo15:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION15:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168025915:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168427215:10
*** ooolpbot has quit IRC15:10
openstackLaunchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)15:10
openstackLaunchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress]15:10
*** brault has quit IRC15:12
*** markmc has quit IRC15:12
dtrainorsorry therve i don't have an environment in which i can risk breaking certmonger on at the moment15:13
*** migi has quit IRC15:13
*** chlong has quit IRC15:13
thervedtrainor, No worries15:13
*** milan has joined #tripleo15:13
*** anshul has quit IRC15:13
*** migi has joined #tripleo15:14
openstackgerritMerged openstack/tripleo-heat-templates stable/ocata: Fix bogus parameters in get_param  https://review.openstack.org/45754115:14
openstackgerritMerged openstack/tripleo-heat-templates master: scenario001/pingtest: enable Gnocchi resource again  https://review.openstack.org/44279515:14
EmilienMpradk: ^15:15
pradknice15:16
*** florianf has joined #tripleo15:16
*** markmc has joined #tripleo15:17
*** paramite has joined #tripleo15:18
*** asbishop has joined #tripleo15:19
openstackgerritPradeep Kilambi proposed openstack/tripleo-common master: Add ceilometer agent images  https://review.openstack.org/45430815:20
*** paramite has quit IRC15:20
jistrmandre: we're past the config generation trouble, into DB sync trouble http://logs.openstack.org/84/450784/18/experimental/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/b3d61f9/logs/undercloud/home/jenkins/overcloud_upgrade_console.log.txt.gz15:21
*** myoung has joined #tripleo15:21
*** abishop_ has quit IRC15:21
flaper87EmilienM: I'll be there15:23
mandrejistr: yay! progress \o/15:23
*** colonwq has quit IRC15:23
flaper87jistr: yaaaaaaaaaaaaaaaaaaaaaaaaahoooo15:25
jistr:D :D15:25
flaper87or rather, yeeeeehaaaaaa 'cause texas15:25
*** kjw3 has quit IRC15:26
mandrejistr: so you have a error with nova and neutron db, is that right?15:28
owalshjistr: http://logs.openstack.org/84/450784/18/experimental/gate-tripleo-ci-centos-7-containers-multinode-upgrades-nv/b3d61f9/logs/subnode-2/var/log/messages.txt.gz#_Apr_21_15_04_0415:28
owalshnova db sync can't access the api db15:29
jistrmandre, owalsh: indeed nova and neutron db sync15:30
*** zoliXXL has joined #tripleo15:31
dtrainoroh and now my internet is throwing fits15:33
*** migi has quit IRC15:33
owalshjistr: is start_order being ignored in nova-api.yaml?15:33
*** migi has joined #tripleo15:33
owalshjistr: or are the tasks running concurrently?15:34
*** atheurer has quit IRC15:35
mandrejistr: I'm also past the libvirt error myself, onto a ceilometer-upgrade issue :)15:35
jistrhahaha15:35
*** myoung has quit IRC15:35
jistrbut yea it's good, at least we're going forward again :)15:36
*** zoliXXL has quit IRC15:36
owalshjrist: ah, hang on. It's an upgrade yea?15:36
openstackgerritBrad P. Crochet proposed openstack/tripleo-common master: Add '_comment' as a valid field in instackenv.json  https://review.openstack.org/45888715:36
thrashrasca: ^^^15:36
jistrowalsh: yea upgrade15:36
*** hewbrocca is now known as hewbrocca_afk15:36
mandrejistr: ceilometer on the host is missing the gnocchiclient module15:36
mandrehttp://logs.openstack.org/52/454152/10/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/933db18/logs/subnode-2/var/log/ceilometer/ceilometer-upgrade.log.txt.gz15:36
mandrepradk: ^^15:36
mandrethis one looks like a packaging issue15:37
*** morazi has quit IRC15:37
EmilienMbeekneemech: do you know out of your fresh mind where can we change permissions of log files before collecting them in tripleo ci (non quickstart) jobs?15:38
EmilienMbeekneemech: /etc/sensu has some files owned by root and it makes CI failing15:38
rascathrash++15:38
*** thrash is now known as thrash|biab15:39
beekneemechEmilienM: We do some of that in postci today: http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/common_functions.sh#n28115:39
*** colonwq has joined #tripleo15:39
EmilienMpabelanger: do you have the logs details of the sensu problem please?15:40
jistrowalsh: nova_api_db_sync seems to have run fine though15:41
*** dprince has quit IRC15:41
pabelangerEmilienM: it is a permission issue, /etc/sensu on subnodes should be 0755, but is 0444 I think15:41
pabelangerno logs15:41
EmilienMpabelanger: weird, we chown the workspace, https://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/common_functions.sh#n28115:42
owalshjistr: don't think upgrades have been implemented for nova-api. It's err, complex: https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/nova-api.yaml#L19415:42
owalsh^^ have not been implemented for nova-api docker service15:43
pabelangerEmilienM: chmod15:43
EmilienMpabelanger: can you tell where in the script it fails?15:44
EmilienMpabelanger: in zuul logs?15:44
pabelangerEmilienM: it doesn't fail, that is the issue. We rsync things to zuul-launcher, and because we preserve the 0440 directory permission, we cannot delete it, then job aborts.15:44
owalshjistr: and for upgrades the order is different, api db sync run as the very end https://github.com/openstack/tripleo-heat-templates/blob/master/puppet/services/nova-api.yaml#L26815:44
pabelangerwe have fixed this today on zuul side, but need to schedule a restart15:44
pabelangerand zuulv3 will make this easy to dealwith15:45
EmilienMpabelanger: ok I see. I'm trying to fix it in tripleo-ci, give me a minut15:45
pabelangerbut today, you could ensure your logs folder is 0755 for all directories15:45
openstackgerritHarry Rybacki proposed openstack/tripleo-quickstart master: Add supplemental node provisioner  https://review.openstack.org/45152315:45
EmilienMpabelanger: it fails during the tarball creation or extraction?15:46
pabelangerEmilienM: find logs -type d -exec chmod 755 {} \;15:46
EmilienMyeah I can do that,I'm just trying where15:46
pabelangerEmilienM: no, when we try to delete the tmp directories15:46
jistrowalsh: alright :D i thought the general nova cell funkiness was specific to N->O upgrade (this upgrade is master->master, so P->P). But yea if the order of db syncs should be reversed on upgrade vs. initial deployment, then that could be the problem.15:46
pabelangeron zuul15:46
EmilienMah... gotcha15:46
*** jpena has quit IRC15:46
mandrepradk: when you have time, https://review.rdoproject.org/r/#/c/6337/15:47
openstackgerritSteven Hardy proposed openstack/python-tripleoclient master: Save DefaultPasswords values for undercloud deploy  https://review.openstack.org/45840715:47
*** mburned is now known as mburned_out15:47
jistrowalsh: also we currently don't have a way to express this order difference in templates easily, but maybe i could use heat conditionals for that15:47
owalshjistr: although this is an ocata -> pike upgrade so we already have cells_v2 setup in ocata. So there should be nothing to do15:48
owalshjistr: yea, like you just said :-)15:48
owalshthe order only matter when upgrading from non-cell_v2 to cell_v215:49
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: logs: chmod 755 all workspace directories  https://review.openstack.org/45889515:49
EmilienMpabelanger: ^^^15:49
EmilienMlarsks: fyi ^15:49
jistrowalsh: ah ok so we should be good hopefully w/r/t order?15:49
openstackgerritMarkos Chandras (hwoarang) proposed openstack/diskimage-builder master: diskimage-builder: Use correct path for element-info  https://review.openstack.org/45889615:50
*** jcoufal_ has joined #tripleo15:51
*** trozet has quit IRC15:51
*** jpena has joined #tripleo15:52
EmilienMpabelanger: anyway we can test this one?15:53
owalshjistr: yea, I'd hazard a guess that /etc/my.cnf.d/tripleo.cnf is the culprit15:54
owalshjistr: probably has the wrong IP address15:54
owalshalthough that should break all DB operations15:55
*** jcoufal has quit IRC15:55
jistrwe're using host mode networking15:55
pabelangerEmilienM: gate-tripleo-ci-centos-7-scenario001-multinode-upgrades-nv is the job that fails15:55
jistrso 192.168.24.9 doesn't look too suspicious15:56
pabelangerEmilienM: so depends on maybe from openstack/tripleo-heat-templates15:56
EmilienMpabelanger: indeed15:56
*** lyarwood is now known as me15:56
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs  https://review.openstack.org/35921515:56
*** me is now known as Guest4138315:57
owalshjistr: yea, don't think it's that. We wouldn't connect to the DB at all15:57
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: DNM - CI testing - Upgrades from Ocata to Pike  https://review.openstack.org/45760315:57
*** Guest41383 is now known as lyarwood15:57
openstackgerritMerged openstack/instack-undercloud master: Make 'reauthentication_auth_method=trusts' the default  https://review.openstack.org/45783715:58
owalshjistr: just looks like the api_db sync hasn't gotten to the expected schema version15:59
jistrowalsh: yea... maybe just bad RPM or wrong version? We had a very similar issue with Glance db sync not finding Pike target in alembic, was fixed by rebuilding the containers with newer glance RPM.16:01
jistrit could simply be that the Nova in container is older in terms of available DB versions than the one on bare metal that we're upgrading from16:02
jistrthat would be quite likely as the RPMs on BM come from delorean known-good snapshots, but the containers images aren't being rebuilt that often (manual process currently AFAIK)16:03
*** bugzy_ has joined #tripleo16:04
*** bugzy has quit IRC16:06
*** ooolpbot has joined #tripleo16:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION16:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168025916:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168427216:10
*** ooolpbot has quit IRC16:10
openstackLaunchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)16:10
openstackLaunchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress]16:10
*** jpich has quit IRC16:10
*** mcornea has quit IRC16:13
zanebEmilienM: apparently I was looking at the wrong job :( it's still 574s in the ovb-nonha job. revert has landed though, so you should have some breathing room while I figure it out16:13
EmilienMzaneb: yeah I'm now trying to promote CI16:14
*** lmiccini has quit IRC16:14
*** jlinkes has quit IRC16:22
mnaserhey folks, i was stuck with tripleo deploys failing pxe boots for the longest time ever till i found out that stp was blocking the port for too long (and setting the port to portfast was a common thing to work around this)16:22
mnaserwould a note about this in the tripleo-docs be in the right place or not?16:23
*** ramishra has quit IRC16:24
*** ramishra has joined #tripleo16:26
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-heat-templates master: Updated from global requirements  https://review.openstack.org/45603016:27
*** jkilpatr has quit IRC16:28
*** jkilpatr_ has joined #tripleo16:28
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates master: Test I71f8aa431a60457326167b8c82adc03ca750eda6  https://review.openstack.org/45885816:29
*** dprince has joined #tripleo16:29
openstackgerritMerged openstack/tripleo-docs master: Add export to deployment document for image builds.  https://review.openstack.org/45778016:29
*** dsneddon has quit IRC16:30
shardymnaser: yes, perhaps a note in the troubleshooting section?16:30
*** trown is now known as trown|lunch16:30
*** beekneemech has quit IRC16:30
openstackgerritMartin André proposed openstack-infra/tripleo-ci master: [WIP] add containerized deployment on multinode  https://review.openstack.org/45415216:31
*** colonwq has quit IRC16:33
*** lucasagomes is now known as lucas-afk16:36
*** jayg is now known as jayg|g0n316:38
*** lblanchard has joined #tripleo16:39
*** lblanchard has quit IRC16:39
*** jprovazn has quit IRC16:41
*** dsneddon has joined #tripleo16:41
*** gfidente is now known as gfidente|afk16:42
*** adarazs has joined #tripleo16:43
*** bnemec has joined #tripleo16:44
*** salmankhan has quit IRC16:45
*** bnemec is now known as beekneemech16:45
*** adarazs has quit IRC16:47
*** shardy has quit IRC16:50
*** rasca has quit IRC16:52
*** mburned_out has quit IRC16:52
*** markmc has quit IRC16:53
*** hewbrocca_afk has quit IRC16:53
*** amoralej has quit IRC16:53
*** akrzos has quit IRC16:53
*** asbishop has quit IRC16:53
*** pabelanger has quit IRC16:54
*** dmanchad has quit IRC16:54
*** jpena has quit IRC16:54
*** weshay has quit IRC16:54
*** migi has quit IRC16:54
*** atheurer has joined #tripleo16:55
*** florianf has quit IRC16:56
*** derekh has quit IRC17:00
*** rasca has joined #tripleo17:00
*** abishop has joined #tripleo17:00
*** hewbrocca_afk has joined #tripleo17:00
*** jpena|off has joined #tripleo17:00
*** migi has joined #tripleo17:00
*** pabelanger has joined #tripleo17:00
*** dmanchad has joined #tripleo17:01
*** akrzos has joined #tripleo17:01
*** markmc has joined #tripleo17:01
*** weshay has joined #tripleo17:01
*** mburned_out has joined #tripleo17:02
*** amoralej has joined #tripleo17:04
*** jpena|off is now known as jpena17:04
*** jpena is now known as jpena|off17:07
EmilienMFYI RH1 is unreachable, don't expect OVB jobs working today17:07
*** tosky has quit IRC17:10
*** ooolpbot has joined #tripleo17:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION17:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168025917:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168427217:10
*** ooolpbot has quit IRC17:10
openstackLaunchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)17:10
openstackLaunchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress]17:10
*** brault has joined #tripleo17:10
*** trown|lunch is now known as trown17:14
*** ffiore has quit IRC17:16
*** brault has quit IRC17:18
*** tesseract has quit IRC17:19
beekneemechEmilienM: I opened https://bugs.launchpad.net/tripleo/+bug/1685318 to track it.17:21
openstackLaunchpad bug 1685318 in tripleo "OVB jobs down due to network outage" [Critical,Triaged]17:21
* beekneemech re-prioritizes the ability to run OVB on an arbitrary public cloud17:21
beekneemechIt would be nice if we could spin up a new ci environment on a random public cloud when this sort of thing happens.17:22
*** marios has quit IRC17:22
pabelanger1000++17:24
pabelangerit would be great to have tripleo not depend on a single cloud17:24
*** bugzy_ is now known as bugzy17:24
pabelangerWas there any progress on getting OVB upstream for nova?17:24
*** jcoufal_ has quit IRC17:25
*** pkovar has quit IRC17:25
beekneemechpabelanger: I think https://blueprints.launchpad.net/nova/+spec/virt-rescue-stable-disk-devices would allow us to do full OVB in vanilla nova.17:25
*** jtomasek has quit IRC17:26
*** jcoufal has joined #tripleo17:26
beekneemechBut for ci we can actually use an ipxe image too, so it's possible today.17:26
beekneemechIntegrated pxe boot support in nova is more of an interactive developer environment issue.17:26
*** thrash|biab is now known as thrash17:27
beekneemechSo theoretically it's possible to do this today, but I haven't had a chance to investigate whether any public clouds actually expose all of the features we need.17:27
*** jcoufal has quit IRC17:28
pabelangersounds like something to add into devstack, if possible. Then we'd know for sure if all our clouds support it17:28
*** ykarel has quit IRC17:31
*** dspano has joined #tripleo17:36
*** jaganathan has joined #tripleo17:36
openstackgerritMerged openstack/tripleo-common master: Workflow to select nodes matching a profile/role  https://review.openstack.org/44105417:37
dspanoGood afternoon. I have a question regarding post-deploy overcloud config. Are there any folks using Ansible to configure their clouds after deploying?17:38
dspanoNot asking for support, more of a am I stupid for thinking of it kind of thing.17:39
*** dprince has quit IRC17:41
*** lblanchard has joined #tripleo17:41
beekneemechdspano: There are people doing that, but you do need to be careful.  TripleO uses puppet to configure things, and if puppet and ansible end up trying to manage the same thing there can be issues.17:41
*** ramishra has quit IRC17:42
*** newmember has quit IRC17:42
dspano@beekneemech, thanks for the feedback. I totally understand.17:43
*** ramishra has joined #tripleo17:44
*** anshul has joined #tripleo17:46
*** ramishra has quit IRC17:49
*** rwsu has quit IRC17:51
*** ramishra has joined #tripleo17:51
*** dprince has joined #tripleo17:53
*** dprince has quit IRC17:53
*** jaganathan has quit IRC17:53
*** dprince has joined #tripleo17:53
*** gfidente|afk has quit IRC17:54
*** dparkes has joined #tripleo18:03
*** yamahata_ has quit IRC18:04
*** owalsh_ has joined #tripleo18:06
*** jmelvin has quit IRC18:09
*** ykarel has joined #tripleo18:09
*** owalsh has quit IRC18:10
*** ooolpbot has joined #tripleo18:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION18:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168025918:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168427218:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168531818:10
*** ooolpbot has quit IRC18:10
openstackLaunchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)18:10
openstackLaunchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress]18:10
openstackLaunchpad bug 1685318 in tripleo "OVB jobs down due to network outage" [Critical,Triaged]18:10
*** ccamacho has quit IRC18:12
*** ykarel has quit IRC18:14
*** yamahata_ has joined #tripleo18:21
*** amoralej is now known as amoralej|off18:23
*** gchamoul has joined #tripleo18:24
*** atheurer has quit IRC18:24
*** cschwede has joined #tripleo18:24
*** rnoriega has joined #tripleo18:25
*** atheurer has joined #tripleo18:25
*** pliu has joined #tripleo18:27
*** dparkes has quit IRC18:27
*** zoli has joined #tripleo18:27
*** stevebaker has joined #tripleo18:27
*** lhinds has joined #tripleo18:27
*** dparkes has joined #tripleo18:28
*** marios has joined #tripleo18:29
*** slagle_ has joined #tripleo18:29
*** tvignaud has quit IRC18:29
*** pkovar has joined #tripleo18:30
*** bogdando has quit IRC18:32
*** paramite has joined #tripleo18:35
*** rwsu has joined #tripleo18:36
*** suuuper has quit IRC18:38
*** ihrachys has joined #tripleo18:40
*** jerrygb has quit IRC18:40
ihrachyspradk, ping re telemetry on overcloud taking all resources18:40
ihrachyshttps://bugs.launchpad.net/tripleo/+bug/162116418:40
openstackLaunchpad bug 1621164 in tripleo "gnocchi statsd consumes all overcloud resources when configured with swift backend" [High,In progress]18:40
ihrachysjust hit something similar, it was ceilo/gnocchi + swift taking all CPU18:41
ihrachysI wonder why we have it on by default in the first place18:41
pradkihrachys, whats the error in statsd log?18:41
ihrachysI would think those interested would opt in18:41
pradkihrachys, there were some optimizations made in swift in ocata i believe.. what version are you using?18:42
ihrachyspradk, no error, just wasted cycles18:42
*** myoung has joined #tripleo18:42
ihrachysit's ocata18:43
pradkihrachys, as jd said in the comment, we saw this before, but the issue turned out to be on swift side18:43
ihrachysfor now I disabled ceilo and gnocchi, bumped swift replication intervals.18:43
ihrachysseems a bit better18:43
pradkok18:43
ihrachysso my question is mostly about why we have it on by default.18:44
ihrachysespecially since we tend to struggle with getting resources to deploy the thing in e.g. dev env18:44
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: Add ssh tunneling for GUI  https://review.openstack.org/40773318:44
pradkihrachys, we want telemetry enabled by default for events and metric processing .. but if you dont care about it, you can disbale by removing from the roles yaml18:45
*** zoli has quit IRC18:45
pradkihrachys, we saw perf issues in newton but most of those have been fixed afaik in ocata18:45
*** rwsu has quit IRC18:45
ihrachyspradk, I deploy with infrared so it's same libvirt handling all nodes18:45
ihrachyspradk, would need to have a look how to use a custom yaml there.18:46
ihrachysit seems to me the least there should be some ready profiles for minimal env18:46
ihrachysif not default18:46
ihrachyssomething that environments like infrared and generally devs could use to get work done with reasonable resources18:47
ihrachysnot e.g. 64gb+18:47
*** tvignaud has joined #tripleo18:47
*** myoung|remote has joined #tripleo18:47
*** myoung|remote is now known as myoung|alt18:48
pradkihrachys, in pike, we're removing ceilo api and collector so should help with load we have 2 less services on by default. But yea good idea to have some pre-canned yamls for minimal deploys18:49
*** jerrygb has joined #tripleo18:49
ihrachyspradk, I have 32gb workstation. imagine the struggle. ;)18:50
*** rwsu has joined #tripleo18:50
pradkihrachys, i can feel your pain :/18:50
*** zoli has joined #tripleo18:50
*** jmelvin has joined #tripleo18:50
*** atheurer has quit IRC18:52
*** atheurer has joined #tripleo18:52
*** gbarros has joined #tripleo18:52
*** gbarros_ has joined #tripleo18:54
*** dparkes_ has joined #tripleo18:54
*** zoli has quit IRC18:55
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: Deprecate ceilometer collector  https://review.openstack.org/45088518:56
*** dparkes has quit IRC18:57
*** gbarros has quit IRC18:58
*** ansmith has quit IRC18:59
*** rwsu has quit IRC19:00
openstackgerritHarald Jensås proposed openstack/instack-undercloud master: Tripleo routed networks ironic inspector, and Undercloud  https://review.openstack.org/43754419:01
*** fragatina has quit IRC19:01
*** zoli has joined #tripleo19:01
*** pcaruana has quit IRC19:02
*** dparkes_ has quit IRC19:03
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: Add ssh tunneling for GUI  https://review.openstack.org/40773319:03
*** fragatina has joined #tripleo19:03
*** kbyrne has quit IRC19:04
*** dparkes has joined #tripleo19:05
*** tvignaud has quit IRC19:06
*** rwsu has joined #tripleo19:07
*** ooolpbot has joined #tripleo19:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION19:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168025919:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168427219:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168531819:10
*** ooolpbot has quit IRC19:10
openstackLaunchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)19:10
openstackLaunchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress]19:10
openstackLaunchpad bug 1685318 in tripleo "OVB jobs down due to network outage" [Critical,Triaged]19:10
*** Slower_ is now known as Slower19:13
*** brault has joined #tripleo19:14
*** newmember has joined #tripleo19:17
*** dparkes has quit IRC19:17
*** brault has quit IRC19:18
*** iranzo has quit IRC19:19
*** fultonj has joined #tripleo19:19
*** zoli has quit IRC19:20
*** tvignaud has joined #tripleo19:20
*** gbarros_ has quit IRC19:20
*** zoli has joined #tripleo19:21
*** gbarros has joined #tripleo19:21
*** fragatina has quit IRC19:22
*** dparkes has joined #tripleo19:24
EmilienMbeekneemech: sounds like it's back19:26
EmilienMI see ovb jobs running19:27
*** dparkes has quit IRC19:28
*** dparkes has joined #tripleo19:30
*** zoli has quit IRC19:30
*** tvignaud has quit IRC19:34
*** dparkes has quit IRC19:34
beekneemechEmilienM: Yep, looks better now.19:35
-openstackstatus- NOTICE: Gerrit will be offline briefly starting at 20:00 for scheduled maintenance http://lists.openstack.org/pipermail/openstack-dev/2017-April/115702.html19:44
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: Add ssh tunneling for GUI  https://review.openstack.org/40773319:46
*** tvignaud has joined #tripleo19:47
*** stevebaker has quit IRC19:48
*** gchamoul has quit IRC19:48
*** pliu has quit IRC19:48
*** marios has quit IRC19:48
*** cschwede has quit IRC19:48
*** rnoriega has quit IRC19:49
*** lhinds has quit IRC19:49
*** myoung has quit IRC19:49
*** fragatina has joined #tripleo19:50
paramiteEmilienM, Hey Emilien, are you here?19:51
openstackgerritMerged openstack/puppet-tripleo master: Move ceilometer upgrade re-run out of collector  https://review.openstack.org/45803619:51
*** anshul has quit IRC19:52
openstackgerritMerged openstack/tripleo-heat-templates master: Add service config settings to agent services  https://review.openstack.org/45638519:52
*** dougbtv|laptop has quit IRC19:57
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: Containerize Redis service  https://review.openstack.org/44215120:01
EmilienMparamite: yes20:05
paramiteEmilienM, I've sent you a mail re sensu. Are you sure correct opstools repo is used in CI jobs?20:05
*** gchamoul has joined #tripleo20:05
paramitebecause it seems that old sensu is installed instead of latest20:06
*** cschwede has joined #tripleo20:06
-openstackstatus- NOTICE: Gerrit is offline briefly for scheduled maintenance http://lists.openstack.org/pipermail/openstack-dev/2017-April/115702.html20:06
*** ChanServ changes topic to "Gerrit is offline briefly for scheduled maintenance http://lists.openstack.org/pipermail/openstack-dev/2017-April/115702.html"20:06
*** ooolpbot has joined #tripleo20:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION20:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168025920:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168427220:10
*** ooolpbot has quit IRC20:10
openstackLaunchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)20:10
openstackLaunchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress]20:10
*** jerrygb has quit IRC20:10
EmilienMparamite: i'm sure no nothing20:11
EmilienMparamite: http://logs.openstack.org/85/450885/9/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq/785654d/logs/subnode-2/etc/yum.repos.d/centos-opstools.repo.txt.gz20:11
EmilienMtell us if it's good20:11
EmilienMit's a mirror from centos20:11
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: DNM - CI testing - Upgrades from Ocata to Pike  https://review.openstack.org/45760320:11
*** gchamoul has quit IRC20:12
*** slagle_ has quit IRC20:12
*** cschwede has quit IRC20:13
paramiteEmilienM, uhm ... 0.27.0 is there ... let me check which sensu version has been installed20:13
*** atheurer has quit IRC20:15
EmilienMsensu-0.27.0-1.el7.noarch20:18
EmilienMhttp://logs.openstack.org/85/450885/9/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq/785654d/logs/subnode-2/rpm-qa.txt.gz20:18
EmilienMparamite: ^20:18
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: Expose metric delay processing metric  https://review.openstack.org/45895920:18
*** atheurer has joined #tripleo20:18
*** ramishra has quit IRC20:20
paramiteEmilienM, ok, now again I don't have an idea what's going on ... is there a possibility that 0.23.2 has been installed before on image and the README.md just did not get cleaned?20:20
EmilienMparamite: we found the issue on upgrade jobs but AFIK opstools are not versioned in RDO, since they are hosted on centos20:21
*** jmelvin has quit IRC20:21
EmilienMso I don't see how 0.23.2 can be installed20:21
EmilienMwe can check in logs btw...20:21
EmilienMbut no, 0.23.2 can't be installed, since it's not the version provided in the mirror20:22
*** jmelvin has joined #tripleo20:22
paramiteEmilienM, there is 0.23.2 in mirror repo: http://mirror.regionone.osic-cloud1.openstack.org/centos/7/opstools/x86_64/sensu/sensu-0.23.2-3.el7.noarch.rpm20:22
EmilienMparamite: unless we pin a version somewhere in tripleo, I think we pick the latest version...20:23
paramiteok20:23
*** jerrygb has joined #tripleo20:25
*** anshul has joined #tripleo20:26
paramiteEmilienM, well, /etc/sensu/conf.d/README.md is located only in 0.23.2 package ... I will try to install it and then update to see if the file is left in the directory20:27
*** atheurer has quit IRC20:27
*** dprince has quit IRC20:29
paramiteEmilienM, well nope, the file is correctly cleaned after update20:29
EmilienMparamite: where do you see that?20:30
paramiteEmilienM, I just tried that in centos VM with the mirror repo20:30
*** anshul has quit IRC20:31
EmilienMparamite: maybe the package didn't get updated and zuul failed to collect logs. Weird, why would it work on stable/ocata20:31
paramiteEmilienM, no idea, but even in 0.23.2 the file is not owned by root: http://paste.openstack.org/show/607523/20:33
EmilienMparamite: weird. Puppet manage the dir or files somehow?20:35
paramiteEmilienM, yes but AFAIR default owner is sensu:sensu ... let me check20:35
*** gbarros has quit IRC20:36
paramiteEmilienM, yup https://github.com/sensu/sensu-puppet/blob/f1ed6eace5ec0a2e9b77ee70516bf89f35b651df/manifests/init.pp#L54720:36
EmilienMah found it20:37
EmilienMhttps://github.com/sensu/sensu-puppet/blob/f1ed6eace5ec0a2e9b77ee70516bf89f35b651df/manifests/init.pp#L55320:37
EmilienMthis is not enough for zuul to get logs20:37
EmilienMthat's why I think20:37
openstackgerritAlan Bishop proposed openstack/tripleo-common master: WIP: Derive Params (part 1): Initial workflow to start  https://review.openstack.org/45646920:37
paramiteEmilienM, ah-hah, so we need o+r?20:37
*** rbowen has quit IRC20:38
*** ramishra has joined #tripleo20:38
openstackgerritBrent Eagles proposed openstack/tripleo-docs master: WIP: Post deployment installation instructions for Octavia  https://review.openstack.org/44749620:38
EmilienMparamite: 755 prob20:38
*** ChanServ changes topic to "TripleO : http://tripleo.org/ | https://wiki.openstack.org/wiki/TripleO | Meetings On Tuesdays at 14:00 UTC in #openstack-meeting-alt"20:38
-openstackstatus- NOTICE: Gerrit is back in service and generally usable, though remote Git replicas (git.openstack.org and github.com) will be stale for the next few hours until online reindexing completes20:38
openstackgerritBrent Eagles proposed openstack/tripleo-docs master: WIP: Post deployment installation instructions for Octavia  https://review.openstack.org/44749620:38
paramiteEmilienM, uhm, on plain files?20:38
*** lblanchard has quit IRC20:40
EmilienMparamite: that's what pabelanger suggested yeah20:40
beekneemechHmm, ovb jobs died again.20:42
EmilienMbeekneemech: network again?20:43
paramiteEmilienM, hmm, ok that's gonna need sensu-puppet patch, because as you can see those variables are hardcoded20:43
beekneemechEmilienM: Not sure.  I still have access through the bastion.20:43
pabelangerwe dumped zuul queues20:47
EmilienMparamite: yeah, sad20:47
pabelangerso jobs would have been killed20:47
beekneemechAhh, that would do it.20:47
*** dparkes has joined #tripleo20:47
pabelangerproject renames per notice^20:47
* beekneemech will worry about it if they're still broken on Monday at this point20:48
*** openstackgerrit has quit IRC20:48
*** ramishra has quit IRC20:49
*** trown is now known as trown|outtypewww20:50
*** atheurer has joined #tripleo20:52
*** dsneddon has quit IRC20:53
*** abishop has quit IRC20:53
paramiteEmilienM, so gate-tripleo-ci-centos-7-scenario001-multinode-upgrades-nv is failing on sensu error?20:54
EmilienMparamite: Zuul fails to collect logs20:55
EmilienMbecause of sensu README in etc20:55
leifmadsenI feel like I've seen this error before, and it was just a mismatch in profile names or something, but I get this on a Newton deploy....20:56
leifmadsenError: Could not find class ::tripleo::profile::base::time::ntp for overcloud-novacompute-0.localdomain on node overcloud-novacompute-0.localdomain20:56
*** tvignaud has quit IRC20:58
*** zaneb has quit IRC20:59
mwhahahaleifmadsen: outdated packages21:03
leifmadsenahhh21:04
mwhahahathe puppet-tripleo is too old21:04
leifmadsenoic21:04
*** tvignaud has joined #tripleo21:04
mwhahahaleifmadsen: or the tripleo-heat-templates package is too n ew21:04
mwhahahawhich ever :D21:04
leifmadsen:D21:04
leifmadsenah guess whatever I get in centos7-newton/current-passed-ci/ is too old :\21:05
mwhahahalet me see21:05
leifmadsen5.5.121:05
leifmadsenhrmmm.... April 10th21:05
leifmadsenseems like it should be new enough21:05
mwhahahaleifmadsen: yea it should be, https://review.openstack.org/#/c/443333/ was the patch that added that class21:06
mwhahahawhich was from march21:06
leifmadsenoh wait, I bet it's because of something I'm doing...21:06
mwhahahathough did you use pre-built images maybe?21:06
leifmadsen(I'm trying to do a Contrail deploy with their instructions, and I bet some of the things they had me do reverted this)21:06
leifmadsennot pre-built, but I bet I'm overriding the modules with something too old21:07
leifmadsenactually huh... well whatever I updated for modules has nothing to do with ntp21:07
mwhahahacheck what package is actually installed on the node21:07
leifmadsenah ha I think21:09
leifmadsen5.5.021:09
leifmadsenmarch 07?21:09
leifmadsenlet me check the modules21:09
*** ooolpbot has joined #tripleo21:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION21:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168025921:10
openstackLaunchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)21:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168427221:10
*** ooolpbot has quit IRC21:10
openstackLaunchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress]21:10
mwhahahaleifmadsen: you can workaround it in your THT by reverting to https://review.openstack.org/#/c/443334/21:10
leifmadsen+121:10
leifmadsenmwhahaha: thanks for the tips, I knew I'd seen this before, but I couldn't remember how I'd resolved it before21:10
mwhahahaleifmadsen: switch back to include ::ntp, but you'll end up with ntp/chronyd fighting on reboots21:10
leifmadsenyea, all good21:11
leifmadsenI know what I can do to work around it / resolve it then21:11
leifmadsenbasically need to make sure I get a newer overcloud image21:11
leifmadsenhave a mismatch it seems21:11
leifmadsenmwhahaha: thanks again! this will move me forward next week21:11
mwhahahaleifmadsen: np21:11
*** zaneb has joined #tripleo21:12
*** brault has joined #tripleo21:15
*** brault has quit IRC21:20
*** fzdarsky has quit IRC21:20
*** karimb has quit IRC21:21
*** dparkes has quit IRC21:22
*** openstackgerrit has joined #tripleo21:22
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: Add ssh tunneling for GUI  https://review.openstack.org/40773321:22
*** mhenkel_ has quit IRC21:26
*** ramishra has joined #tripleo21:34
*** kbyrne has joined #tripleo21:39
*** ramishra has quit IRC21:40
*** fragatin_ has joined #tripleo21:42
*** ramishra has joined #tripleo21:42
*** fragatina has quit IRC21:45
*** dspano has quit IRC21:46
*** dsavineau has quit IRC21:47
*** ramishra has quit IRC21:48
pabelangerparamite: EmilienM: we should be protected on zuul side now, haven't checked the job yet21:50
EmilienMok21:50
paramiteso what does it mean?21:51
*** myoung|alt has quit IRC21:51
*** jcoufal has joined #tripleo21:52
pabelangeryou should still fix your directory permissions, but zuul will no longer abort the job on failure21:55
pabelangerin zuulv3, we'll have better control over this with ansible playbooks21:55
*** rlandy has quit IRC21:57
paramiteack22:01
*** ramishra has joined #tripleo22:05
*** jmelvin has quit IRC22:06
*** newmember has quit IRC22:08
*** fragatin_ has quit IRC22:08
*** fragatina has joined #tripleo22:09
*** ooolpbot has joined #tripleo22:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION22:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168025922:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168427222:10
*** ooolpbot has quit IRC22:10
openstackLaunchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress]22:10
openstackLaunchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress]22:10
*** dsneddon has joined #tripleo22:11
*** ramishra has quit IRC22:13
honzaI'm having trouble enqueuing new messages in zaqar on the undercloud, I'm getting this strange 500 error from swift22:33
honza"Object GET failed: http://192.168.24.1:8080/v1/AUTH_8eabab8673e7431280817d6d8fe59696/zaqar_subscription%3Atripleo%3Ab57034af85334fbea5d650487c2db9bf/42e69b1e-a380-4250-951a-9ce9f523800d 404 Not Found  [first 60 chars of response] <html><h1>Not Found</h1><p>The resource could not be found.<"22:33
honzaIt was working for a while but suddenly stopped :(22:34
honzaWhere could one start looking?22:34
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: Add ssh tunneling for TripleO-UI  https://review.openstack.org/40773322:35
mwhahahahonza: i wonder if that's an old auth url? did you deploy then delete the stack without clearing out the old plan?22:37
mwhahahahonza: https://bugs.launchpad.net/tripleo/+bug/164487522:38
openstackLaunchpad bug 1644875 in tripleo "Plan isn't purged of roles_data on a new create" [High,Triaged]22:38
honzamwhahaha: I haven't touched the plan at all.  All I'm doing is logging into the GUI and sending a bunch of messages via websockets.22:38
* mwhahaha knows nothing about the ui22:38
honzathe issue is between zaqar and swift it seems22:39
*** ramishra has joined #tripleo22:41
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: Add ssh tunneling for TripleO-UI  https://review.openstack.org/40773322:42
honzamessage_post works fine via a Python script (websockets), but not via js :(22:43
*** ramishra has quit IRC22:46
*** ramishra has joined #tripleo22:48
*** ramishra has quit IRC22:53
*** jcoufal has quit IRC22:55
*** ramishra has joined #tripleo23:00
*** ramishra has quit IRC23:05
*** ramishra has joined #tripleo23:06
*** ooolpbot has joined #tripleo23:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION23:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168025923:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/168427223:10
*** ooolpbot has quit IRC23:10
openstackLaunchpad bug 1680259 in tripleo "Upgrades jobs timing out regularly" [Critical,In progress]23:10
openstackLaunchpad bug 1684272 in tripleo "10 minute increase in overcloud deploy/update time" [Critical,In progress]23:10
*** mmethot has joined #tripleo23:13
*** owalsh_ is now known as owalsh23:13
*** brault has joined #tripleo23:16
*** ramishra has quit IRC23:17
*** owalsh_ has joined #tripleo23:19
*** brault has quit IRC23:20
*** owalsh has quit IRC23:22
*** ramishra has joined #tripleo23:24
*** pkovar has quit IRC23:28
*** ramishra has quit IRC23:29
*** dixiaoli has joined #tripleo23:30
*** dixiaoli has quit IRC23:35
*** ramishra has joined #tripleo23:37
*** yamahata_ has quit IRC23:40
*** owalsh has joined #tripleo23:44
*** owalsh_ has quit IRC23:44
*** ramishra has quit IRC23:44

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!