Friday, 2017-11-17

*** itlinux has quit IRC00:07
*** ooolpbot has joined #tripleo00:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION00:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432800:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173103200:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173106300:10
*** ooolpbot has quit IRC00:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)00:10
openstackLaunchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]00:10
*** pmannidi_2 has joined #tripleo00:10
openstackLaunchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged]00:10
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: Don't try to start docker for the UC  https://review.openstack.org/51944100:14
mwhahahaEmilienM: after fixing larks's magical code, it kinda still works http://people.redhat.com/aschultz/foo.svg https://github.com/mwhahaha/dotstack00:19
* mwhahaha wanders off for now00:20
weshayhrm.. not good00:20
weshayI think we're seeing the same ssh failure that we see in 001 in 00300:20
weshayhttps://review.openstack.org/52057400:20
weshayhttp://logs.openstack.org/06/517906/9/gate/legacy-tripleo-ci-centos-7-scenario003-multinode-oooq-container/4e21544/logs/undercloud/home/zuul/tempest_output.log.txt.gz00:20
mwhahahaweshay: yea00:20
mwhahahaweshay: hopefully https://review.openstack.org/#/c/520280/ will help shed some light on the subject00:21
weshayrlandy|rover|afk, fyi.. I updated the query as we discussed earlier00:21
weshayarxcruz, ^00:21
mwhahahaweshay: since it was a different job, the old query didn't seem to work with it00:22
mwhahahasorry different tempest test00:22
*** dxiri_ has joined #tripleo00:22
weshayI ran it locally w/ rlandy|rover|afk it picked up 003 as failing 1% of the time00:22
mwhahahaalso 004 just failed with it00:22
weshayoh f me00:22
*** rlandy|rover|afk is now known as rlandy|rover00:23
weshaywe should talk less often00:23
*** bfournie has joined #tripleo00:23
rlandy|roverweshay: thank you - will review00:23
mwhahahahttp://logs.openstack.org/25/519725/4/gate/legacy-tripleo-ci-centos-7-scenario004-multinode-oooq-puppet/d35082b/logs/undercloud/home/zuul/tempest_output.log.txt.gz#_2017-11-17_00_15_1200:23
mwhahahathough it might be different00:24
mwhahahahaven't looked00:24
mwhahahathat was the top of the gate00:24
*** bfournie has quit IRC00:24
weshayweird.. I didn't get that in my feed00:24
mwhahahait's still running00:24
* mwhahaha uses the status page to spot them early00:25
weshaybah see it00:25
mwhahahahttp://zuulv3.openstack.org/00:25
*** bfournie has joined #tripleo00:25
weshayya.. I do a little bit of both00:25
weshaymwhahaha, hrm.. may want to shoot it all in the head to get https://review.openstack.org/#/c/520280/6/puppet/services/neutron-metadata.yaml in00:26
*** dxiri has quit IRC00:26
mwhahahai gotta go pickup the kid, maybe later tonight00:26
weshaykk no worries00:26
mwhahahai'll see what it looks like in 3 hours. we probably won't have merged anything anyway00:26
weshaythanks mwhahaha00:26
mwhahaha:(00:26
EmilienMback online00:29
*** daidv has quit IRC00:30
EmilienMmwhahaha: do you remember the vxlan setup error when deploying traas in rdo cloud?00:30
EmilienMI have it now, I can't figure what is it again00:30
*** daidv has joined #tripleo00:31
EmilienMhttp://paste.openstack.org/show/626564/00:31
EmilienMovs not started on overcloud node, and I can successfully start it00:33
EmilienMI think it was related to openvswitch-2.800:33
EmilienMok found it, ovs 2.8 update doesn't restart ovs service00:37
*** gbarros has quit IRC00:39
*** jlabarre has quit IRC00:55
*** liverpooler has joined #tripleo00:57
*** dxiri_ has quit IRC01:06
*** dxiri has joined #tripleo01:07
*** ooolpbot has joined #tripleo01:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION01:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432801:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173103201:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173106301:10
*** ooolpbot has quit IRC01:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)01:10
openstackLaunchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]01:10
openstackLaunchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged]01:10
*** dxiri has quit IRC01:11
*** dsneddon has quit IRC01:11
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates master: Check DeployedServerPortMap  https://review.openstack.org/52081601:15
*** ipsecguy has quit IRC01:37
*** chlong_ has joined #tripleo01:38
*** almondjoy has quit IRC01:41
*** chlong has quit IRC01:41
*** cshastri has joined #tripleo01:49
*** dxiri has joined #tripleo01:54
*** dxiri has joined #tripleo01:54
*** dxiri has joined #tripleo01:54
*** ipsecguy has joined #tripleo01:56
*** dxiri has quit IRC01:59
*** chlong__ has joined #tripleo01:59
*** chlong_ has quit IRC02:03
*** ooolpbot has joined #tripleo02:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION02:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432802:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173103202:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173106302:10
*** ooolpbot has quit IRC02:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)02:10
openstackLaunchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]02:10
openstackLaunchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged]02:10
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: Don't try to start docker for the UC  https://review.openstack.org/51944102:11
*** morazi has joined #tripleo02:15
*** zshi has quit IRC02:17
*** fragatina has quit IRC02:30
*** toure is now known as toure_biab02:31
*** zshi has joined #tripleo02:31
*** ramishra has joined #tripleo02:39
*** yamahata has joined #tripleo02:51
openstackgerritMerged openstack/tripleo-quickstart-extras master: Fix tempest html generation file  https://review.openstack.org/51790602:52
*** fzdarsky_ has joined #tripleo02:52
*** itlinux has joined #tripleo02:53
*** fzdarsky has quit IRC02:55
openstackgerritRemo Mattei proposed openstack/tripleo-docs master: Adding custom image steps by adding an example of dependency repo.  https://review.openstack.org/51945602:56
openstackgerritRemo Mattei proposed openstack/tripleo-docs master: Adding custom image steps by adding an example of dependency repo.  https://review.openstack.org/51945602:58
mwhahahalooks like the releasenotes job is broken03:00
mwhahahahttps://bugs.launchpad.net/tripleo/+bug/173281503:02
openstackLaunchpad bug 1732815 in tripleo "CI: releasenotes job broken with Could not import extension oslosphinx (exception: No module named oslosphinx) " [Critical,Triaged]03:02
*** links has joined #tripleo03:03
*** stendulker has joined #tripleo03:05
mwhahahaalso different failure but also releasenotes, https://bugs.launchpad.net/tripleo/+bug/173281603:07
openstackLaunchpad bug 1732816 in tripleo "CI: releasenotes job fails due to ValueError: Unknown reference 'origin/stable/ocata'" [Critical,Triaged] - Assigned to Christian Inzinger (ci)03:07
mwhahahanot sure if completely blocking so i'm not adding alert for now03:08
*** ooolpbot has joined #tripleo03:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION03:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432803:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173103203:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173106303:10
*** ooolpbot has quit IRC03:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)03:10
openstackLaunchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]03:10
openstackLaunchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged]03:10
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: Don't try to start docker for the UC  https://review.openstack.org/51944103:18
*** tzumainn has quit IRC03:19
*** fragatina has joined #tripleo03:21
*** fragatina has quit IRC03:25
*** jrist has quit IRC03:29
*** psachin has joined #tripleo03:31
*** itlinux has quit IRC03:36
*** psahoo has joined #tripleo03:42
*** itlinux has joined #tripleo03:45
*** jrist has joined #tripleo03:53
itlinuxhello has anyone setup regions or cells with OOO?03:54
*** morazi has quit IRC03:55
openstackgerritMerged openstack/tripleo-common master: Log Ansible actions to ansible.log  https://review.openstack.org/51478303:58
openstackgerritMerged openstack/tripleo-common master: Add work_dir for some actions  https://review.openstack.org/51478403:59
*** gbarros has joined #tripleo04:02
*** gbarros has quit IRC04:05
*** ooolpbot has joined #tripleo04:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION04:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432804:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173103204:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173106304:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)04:10
*** ooolpbot has quit IRC04:10
openstackLaunchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]04:10
openstackLaunchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged]04:10
openstackgerritMerged openstack/python-tripleoclient master: Add --config-download  https://review.openstack.org/51470104:11
*** shreshtha_ has joined #tripleo04:18
*** dbecker has quit IRC04:23
openstackgerritMerged openstack/python-tripleoclient master: Add --output-dir  https://review.openstack.org/51478904:23
openstackgerritMerged openstack/tripleo-heat-templates stable/pike: Add missing host_prep_tasks for log dirs creation  https://review.openstack.org/52056804:23
openstackgerritMerged openstack/tripleo-heat-templates stable/pike: Add missing host_prep_tasks for memcached  https://review.openstack.org/52056904:23
openstackgerritMerged openstack/tripleo-common master: Truncate the zaqar message to 512 kbytes  https://review.openstack.org/51832804:24
*** itlinux has quit IRC04:24
*** ykarel|away has joined #tripleo04:25
*** shreshtha_ has quit IRC04:30
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: Don't try to start docker for the UC  https://review.openstack.org/51944104:31
weshaymwhahaha, scen003 is dying04:32
*** itlinux has joined #tripleo04:33
*** itlinux has quit IRC04:33
*** aditya_r has joined #tripleo04:37
*** dbecker has joined #tripleo04:38
EmilienMweshay: how that04:38
EmilienMweshay: timeouts and stuff?04:38
weshaytempest04:38
EmilienMdamn04:38
EmilienMssh thing?04:38
weshayyup04:38
EmilienMdamn04:38
EmilienMwe should make it non voting...04:38
EmilienMthat's why we need in-tree layout asap04:38
EmilienMso we can control these things immediatly04:39
*** shreshtha has joined #tripleo04:39
weshayhttp://logs.openstack.org/57/520057/1/gate/legacy-tripleo-ci-centos-7-scenario003-multinode-oooq-container/8c18a46/logs/undercloud/home/zuul/tempest/tempest.log.txt.gz#_2017-11-17_03_55_51_56504:39
EmilienMif it's still failing on our morning, let's disable voting on 003 as well04:39
EmilienMso we can merge stuffs04:39
EmilienMcarefully I guess04:39
weshayEmilienM, and merge that debug patch on neutron metadata service04:40
EmilienMyea04:40
EmilienMdo we have debug btw? let me look04:40
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: DNM: Testing debug neutron metadata  https://review.openstack.org/51975604:41
EmilienMit didn't fail04:41
EmilienMtrying again ^04:41
EmilienMI'm out04:41
EmilienMhopefully we can merge the tripleo-ci patch with zuul layout04:41
EmilienMso we can quickly disable 003 voting tomorrow04:41
weshayaight04:41
EmilienMotherwise I'll ping infra first thing on the morning04:41
EmilienMand disable it in project config04:41
EmilienMno big deal04:41
EmilienMttyl04:41
*** daidv has quit IRC04:42
*** daidv has joined #tripleo04:42
*** pmannidi has quit IRC04:48
*** itlinux has joined #tripleo04:49
*** udesale has joined #tripleo04:50
*** dpawar has joined #tripleo04:53
*** pgadiya has joined #tripleo05:05
*** ooolpbot has joined #tripleo05:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION05:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432805:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173103205:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173106305:10
*** ooolpbot has quit IRC05:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)05:10
openstackLaunchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]05:10
openstackLaunchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged]05:10
*** abregman|pto is now known as abregman|afk05:12
*** jaosorior has joined #tripleo05:12
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates stable/pike: Enable httpd to request certificates for the "external" network  https://review.openstack.org/52089305:12
*** fragatina has joined #tripleo05:12
*** fragatina has quit IRC05:14
*** fragatina has joined #tripleo05:15
*** janki has joined #tripleo05:22
*** itlinux has quit IRC05:35
*** skramaja has joined #tripleo05:36
*** ratailor has joined #tripleo05:36
*** ratailor has quit IRC05:36
*** ratailor has joined #tripleo05:36
*** itlinux has joined #tripleo05:38
*** itlinux has quit IRC05:51
jaosoriorflaper87: hey, saw you were testing out CI for the ansible roles. If you get some time, can you get me an overview of how that works? Also, have you gotten around packaging the roles for RDO?05:51
flaper87jaosorior: yes and no! Yes I'll happily walk you through it and no I haven't gotten around packaging these roles. I think the packaging will probably start early next year.05:52
flaper87jaosorior: do you need the packages now?05:52
flaper87jaosorior: do you want to have a quick call later today?05:53
jaosoriorsure, lets do that05:53
jaosoriorflaper87: so, I ask because of two things: 1: I want to start trying to stuff out. 2. I need to do the same for the ipsec role (not related to k8s)05:53
flaper87jaosorior: coolio, shoot me an invite, pref between 10am and 12pm (since we are both in Europe)05:53
jaosoriorflaper87: what time is it there?05:53
jaosoriorit's almost 8am here.05:54
flaper87jaosorior: I think we are both in... dafuq?05:54
flaper87you are 1h ahead of me?05:54
jaosoriorflaper87: Finland :D05:54
jaosorioryeah, one hour difference05:54
flaper87I thought you started your day as early as I did05:54
jaosoriorand holy shit you're up early05:54
flaper87;)05:54
jaosoriorat what time do you start?05:54
flaper876am, or earlier05:55
flaper87but I'm weird, I just don't sleep much05:55
flaper87:D05:55
jaosoriorflaper87: haha nah, I take my time having breakfast and only start my workday when my gf goes off to work.05:55
flaper87jaosorior: makes total sense :D05:55
jaosoriorthen I also take my time preparing my coffee :P05:55
flaper87jaosorior: ok ok, so, 10am my time?05:55
flaper87jaosorior: I prepare coffee as soon as I get my feet out of my bed05:56
jaosoriorlets do that05:56
flaper87ok, I start my day at 6:10 'cause I take 10mins to prepare coffee05:56
flaper87:P05:56
jaosoriorhaha same here!05:56
jaosorior(regarding the coffee preparation)05:56
jaosorioranyway, 10 am it is (11 here I guess)05:56
flaper87lol05:56
*** marios has joined #tripleo05:56
flaper87ttyl then, going to finish something05:57
jaosoriorenjoy05:57
flaper87jaosorior: but seriously, shoot me an invite. I don't mean to be all formal and stuff but I'll forget if I don't have something buzzing my brain (and I sometimes ignore IRC)05:59
bandinimorning06:00
*** psachin_ has joined #tripleo06:00
jaosoriorflaper87: got the invite?06:01
jaosoriorbandini: yoooo06:01
*** psachin has quit IRC06:01
openstackgerritMerged openstack/tripleo-common stable/pike: Fail ceph-ansible workflow if there is stale puppet-ceph hieradata  https://review.openstack.org/52057706:02
bandinijaosorior: yo! ;)06:02
flaper87bandini: sssh06:03
flaper87jaosorior: yes06:03
bandiniflaper87: you old fart!06:03
flaper87bandini: *cough* last time I checked, you were older than me *cough*06:04
EmilienMquick reminder (before I go to bed) please do not approve patches today06:05
* flaper87 ties his hands to not start an innappropriate conversation on public channels06:05
EmilienMsorry for that but we're trying to migrate to zuul v3 layout and need the patches to land06:05
flaper87EmilienM: roger that, read your email now, gtfo06:05
EmilienMlol06:05
EmilienMflaper87: thx for your update06:05
EmilienMvery useful06:05
bandiniEmilienM: ack!06:05
* EmilienM kidding. So useless.06:05
bandinilol06:05
EmilienMoh wait it's a public channel06:05
flaper87EmilienM: no problem, as said, I commit to sending more useless electronic paper updates06:05
EmilienMno it's awesome! thx flavio06:05
bandiniflaper87: I don't know what you're talking about06:05
EmilienMflaper87: I go to bed now, in case you cared06:06
flaper87EmilienM: I don't but hey, sleep tight and dream about failing CI jobs06:06
EmilienMcan you sing me a song before?06:06
flaper87EmilienM: there was a time, when there was CI, then EmilienM touched it and it was all broken06:06
flaper87EmilienM: do you like that song?06:06
EmilienMflaper87, bandini: hey btw, how is world cup selections going? ahahah. Good night?06:07
*** rlandy|rover has quit IRC06:07
bandiniEmilienM: ahahahah06:07
EmilienMk, thx, bye06:07
flaper87EmilienM: got the wrong guy, dude. I couldn't care less about the world cup06:07
flaper87:P06:07
bandinisame here, although it was hilarious to watch people freak out about it :)06:07
flaper87bandini: seriously06:07
bandini0 fucks given06:08
flaper87FIRST TIME AFTER 60+ YEARS! ARRGH, suicide, shame hats on, etc06:08
flaper87and I was like eating gelato in front of them... in late november06:08
bandiniahaha yes!06:08
flaper87anyway, stop distracting me, gotta finish something06:08
bandinidude, it's Beverdi. there is nothing to do but, start drinking in the morning early on06:09
flaper87bandini: what do you think the Rum bottle on the table is for?06:10
*** ooolpbot has joined #tripleo06:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION06:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432806:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)06:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173103206:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173106306:10
*** ooolpbot has quit IRC06:10
openstackLaunchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]06:10
openstackLaunchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged]06:10
*** akane_ has joined #tripleo06:11
*** akane has joined #tripleo06:11
*** jprovazn has joined #tripleo06:24
*** jprovazn has quit IRC06:25
openstackgerritMichele Baldessari proposed openstack/puppet-tripleo master: Make sure rabbitmq is fully up before creating any rabbitmq resources  https://review.openstack.org/52011906:28
*** pmannidi_2 has quit IRC06:47
openstackgerritOpenStack Proposal Bot proposed openstack/instack-undercloud master: Updated from global requirements  https://review.openstack.org/52040906:49
openstackgerritMerged openstack/tripleo-quickstart-extras master: update the tripleo-ui when condition  https://review.openstack.org/51475706:50
openstackgerritMerged openstack/tripleo-quickstart-extras master: add a very simple tripleo-ui validation  https://review.openstack.org/51479406:50
openstackgerritOpenStack Proposal Bot proposed openstack/instack-undercloud master: Updated from global requirements  https://review.openstack.org/52040906:51
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-ui stable/pike: Imported Translations from Zanata  https://review.openstack.org/52093506:54
*** suuuper has joined #tripleo06:54
*** aditya_r has quit IRC06:56
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras  https://review.openstack.org/47260707:00
*** pcaruana has joined #tripleo07:04
*** dparkes has joined #tripleo07:06
*** ykarel|away is now known as ykarel|lunch07:06
*** ooolpbot has joined #tripleo07:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION07:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432807:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)07:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173103207:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173106307:10
*** ooolpbot has quit IRC07:10
openstackLaunchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]07:10
openstackLaunchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged]07:10
*** rcernin has quit IRC07:18
*** psachin_ has quit IRC07:20
*** jfrancoa has joined #tripleo07:21
openstackgerritMerged openstack/tripleo-validations master: Add per-network bind ips to inventory  https://review.openstack.org/51705107:24
*** psachin has joined #tripleo07:33
*** psachin has quit IRC07:38
*** psachin has joined #tripleo07:39
*** florianf has joined #tripleo07:53
*** hjensas has quit IRC07:55
jaosoriorbandini: if you have some time can you check this out https://github.com/ClusterLabs/resource-agents/pull/105807:57
openstackgerritOpenStack Proposal Bot proposed openstack/instack-undercloud master: Updated from global requirements  https://review.openstack.org/52040908:01
jaosoriordciabrin_: hey, could you take a look at this? https://github.com/ClusterLabs/resource-agents/pull/105808:06
cschwedeGood morning! Can someone please review https://review.openstack.org/#/c/516436/ ? It's a critical bug, might be even a blocker in some cases. LP for this is: https://bugs.launchpad.net/tripleo/+bug/173267008:08
openstackLaunchpad bug 1732670 in tripleo "Containerized deployment fails when using SwiftRawDisk" [Medium,In progress] - Assigned to Christian Schwede (cschwede)08:08
*** chem`` has joined #tripleo08:10
*** ooolpbot has joined #tripleo08:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION08:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432808:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)08:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173103208:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173106308:10
*** ooolpbot has quit IRC08:10
openstackLaunchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]08:10
openstackLaunchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged]08:10
*** pgadiya has quit IRC08:10
*** chem` has quit IRC08:12
*** ffiore has joined #tripleo08:14
*** dpawar has quit IRC08:15
*** ebarrera has joined #tripleo08:16
openstackgerritMerged openstack/tripleo-heat-templates stable/pike: Allow the configuration of image_member_quota from THT  https://review.openstack.org/51930908:22
*** bogdando has joined #tripleo08:23
*** amoralej|off is now known as amoralej08:27
*** pgadiya has joined #tripleo08:27
*** chem`` has quit IRC08:28
flaper87jistr: bogdando fwiw, I'm rebasing my openshift patch08:32
flaper87shardy ^08:32
jaosorioryay08:33
openstackgerritMerged openstack-infra/tripleo-ci master: Import TripleO legacy zuul jobs  https://review.openstack.org/51479308:33
openstackgerritMerged openstack/tripleo-heat-templates master: Drop step_config as top level docker requirement  https://review.openstack.org/44271608:33
chandankumarflaper87: hello May i pm?08:33
openstackgerritMerged openstack/tripleo-heat-templates master: Dynamically generate FQDN hiera entries  https://review.openstack.org/52008208:33
openstackgerritMerged openstack/tripleo-heat-templates master: Debug configuration for neutron metadata agent  https://review.openstack.org/52028008:33
flaper87chandankumar: yeah, sure08:35
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Add rsyslog-sidecar resource and configuration  https://review.openstack.org/51936108:39
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Add option for HAProxy (non-HA) container to log to stdout/stderr  https://review.openstack.org/51936208:39
*** pgadiya has quit IRC08:40
*** dparkes has quit IRC08:43
openstackgerritFlavio Percoco proposed openstack/tripleo-heat-templates master: Deploy OpenShift using OOO on the overcloud  https://review.openstack.org/49447008:46
flaper87jaosorior: did a tiny change to our call, hope it works for ya'08:47
*** dparkes has joined #tripleo08:48
*** cylopez has joined #tripleo08:48
*** jpena|off is now known as jpena08:49
jaosoriorflaper87: still doesn't reflect in the calendar08:49
*** ccamacho has joined #tripleo08:49
*** ffiore has quit IRC08:49
*** ffiore has joined #tripleo08:50
*** jpich has joined #tripleo08:51
jaosoriorflaper87: what change?08:51
*** pgadiya has joined #tripleo08:52
*** dparkes has quit IRC08:56
*** cylopez has quit IRC08:56
*** dpawar has joined #tripleo08:56
flaper87jaosorior: mde it 30 mins starting at 10:3008:58
flaper87does that work?08:58
flaper87(starting in 30mins)08:59
jaosoriorflaper87: sure. I gotta go at 10:50. But I think 20 min is just fine.08:59
flaper87++08:59
*** hjensas has joined #tripleo09:02
openstackgerritmathieu bultel proposed openstack/tripleo-quickstart-extras master: Add test_mtu_sized_frames to the skip list during investigation  https://review.openstack.org/51977009:06
*** cylopez has joined #tripleo09:10
chandankumard0ugal: feel free to package tripleo-common tempest plugin for RDO, here is the reference review https://review.rdoproject.org/r/#/q/topic:add-zaqar-tempest-plugin09:10
*** ooolpbot has joined #tripleo09:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION09:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432809:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173103209:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173106309:10
*** ooolpbot has quit IRC09:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)09:10
openstackLaunchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]09:10
openstackLaunchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged]09:10
*** dbecker has quit IRC09:10
*** shardy has joined #tripleo09:11
Tengujaosorior: "small" question: how does the public endpoint VIP works with a multi-controller containerized infra? I guess it's attached to the node itself - do the service containers have any idea of the VIP location?09:12
Tengujaosorior: asking for certodia, as it detects if it's running on the master by just checking if the VIP is attached to the node, I'm pretty sure this can't work with containers.09:13
*** dparkes has joined #tripleo09:15
*** dparkes has quit IRC09:16
jaosoriorTengu: the containers use net=host09:17
Tenguah, cool09:17
jaosoriorTengu: so that bit didn't change.09:17
Tenguso they know the node IPs09:17
jaosorioryep09:17
Tenguso certodia will work with some mounted volume for the HAProxy socket09:17
Tenguthere's just the restart of haproxy in order to reload the certificate.09:18
Tenguthat might be worked a bit I guess.09:18
Tengubut basically, it will at least renew the certificate correctly. hmm. another volume should be provided in order to get the "horizon" apache vhost root directory, aka /var/www in the non-container env.09:19
Tenguas it's based on the webroot validation.09:19
openstackgerritmathieu bultel proposed openstack/tripleo-heat-templates master: Add validation task in docker services [Aodh]  https://review.openstack.org/49141109:20
*** derekh has joined #tripleo09:20
*** marios has quit IRC09:23
*** dparkes has joined #tripleo09:24
*** dparkes has quit IRC09:26
*** gkadam has joined #tripleo09:32
*** lucas-afk is now known as lucasagomes09:33
*** mrunge_ has joined #tripleo09:34
*** mrunge_ has quit IRC09:35
*** ykarel|lunch is now known as ykarel09:38
jaosoriorTengu: so the restart/reload will work nicely without containers. But with containers is still a bit problematic09:43
jaosoriorI had the same issue with the CRL09:43
*** holser has joined #tripleo09:43
jaosoriorthough it might work, as dciabrin_ made the container use the systemd wrapper. But I haven't tried it out.09:43
jaosoriorbrb, lunch09:43
*** dparkes has joined #tripleo09:46
*** dparkes has quit IRC09:46
*** dparkes has joined #tripleo09:46
*** chem`` has joined #tripleo09:47
*** holser has quit IRC09:48
*** holser has joined #tripleo09:49
openstackgerritMerged openstack/puppet-pacemaker master: zuul: run ovb-ha-oooq  https://review.openstack.org/52027809:50
*** chem`` is now known as chem09:52
*** chem has quit IRC09:53
*** chem has joined #tripleo09:53
openstackgerritMerged openstack/tripleo-ui stable/pike: Imported Translations from Zanata  https://review.openstack.org/52093509:55
Tengujaosorior: I can cehck how we did that in our case. I think we bypass the systemd stuff all the way in order to restart the haproxy from a sidekick (in a rancher env). that same one has the haproxy configuration in fact.09:56
Tengujaosorior: that way, we have two containers: one with the service, and another one that hold the configuration and a reload script - there's also a third one for the metrics. That way, the service is never down, there's a call we can make that mimic the "graceful" of apache.09:57
Tengujaosorior: both containers (config + service) are basically the same (same base), volumes are cross-mounted, meaning we only have to update the configuration container in case of haproxy config update - and the service is gracefully restarted. no downtime :).09:58
Tengusame can be done for the certificate reload (well, and config update, in fact). though it might be a bit more complicated.09:59
shardySmall docs fix needs another reviewer if anyone has a moment: https://review.openstack.org/#/c/520146/10:02
*** mrch has joined #tripleo10:05
openstackgerritSteven Hardy proposed openstack/puppet-tripleo master: Add kubernetes API to haproxy LB configuration  https://review.openstack.org/51640710:07
openstackgerritJuan Badia Payno proposed openstack/tripleo-heat-templates master: logging: merge fluentd-client and fluentd-base  https://review.openstack.org/50145710:09
*** ooolpbot has joined #tripleo10:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION10:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432810:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173103210:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173106310:10
*** ooolpbot has quit IRC10:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)10:10
openstackLaunchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]10:10
openstackLaunchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged]10:10
Tengushardy: can't do more than a +1 :/. sorry10:15
shardyTengu: np :)10:16
*** marios has joined #tripleo10:23
flaper87shardy: btw, I rebased my openshift patch10:24
flaper87it took me like 10 mins to do so10:24
flaper87will keep an eye on it and move it forward10:24
*** fragatin_ has joined #tripleo10:25
shardyflaper87: ack thanks - I'm still getting the haproxy/vip stuff working via kubespray but then will try the openshift patch10:25
flaper87shardy: awesome, I'll let you know when it's in a state for you to try it. don't want to make you waste time10:26
shardyflaper87: ack sounds good, thanks!10:26
*** daidv has quit IRC10:26
*** fragatina has quit IRC10:27
flaper87jaosorior: question re keystone-fernet keys10:28
flaper87how do we manage those in tht?10:28
flaper87Do we generate them once for all of the controller nodes?10:29
flaper87or do we generate them on every node?10:29
*** cshastri has quit IRC10:31
lyarwoodmarios: seeing the following error running the upgrade tasks directly from my ffu playbook, can you remind me where we are adding the - step|int == x line for these?10:35
lyarwoodmarios: http://paste.openstack.org/show/626595/10:36
openstackgerritJulie Pichon proposed openstack/python-tripleoclient master: Fix truncated error message  https://review.openstack.org/52105710:37
marioslyarwood: looking10:38
*** dpawar has quit IRC10:39
shardyflaper87: AFAICS we treat the keys the same as passwords, e.g they're generated in tripleo-common then passed in via parameters, which means they're common to all nodes10:39
shardy(unless we do something magic in puppet)10:39
flaper87shardy: gotcha, that's what I thought10:40
openstackgerritJulie Pichon proposed openstack/python-tripleoclient master: Fix truncated error message  https://review.openstack.org/52105710:40
marioslyarwood: yeah we are doing them in tripleo-common config download for P leme get you a link10:40
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Add zaqarclient to TripleOAction  https://review.openstack.org/51634710:40
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Stream the Ansible output to a Zaqar queue  https://review.openstack.org/52105810:40
*** athomas has quit IRC10:41
lyarwoodmarios: cool thanks10:41
marioslyarwood: but that doesn't look like it failed for the int cast? https://github.com/openstack/tripleo-common/blob/a8caa24f1b84af1ae2fde7946e543b23175c7343/tripleo_common/utils/config.py#L14110:41
lyarwoodmarios: it's the ordering of the conditionals, for this step os_net_config_need_upgrade hasn't been created yet10:42
lyarwoodmarios: so the step check needs to go above10:42
lyarwoodmarios: the indentation is also off10:42
*** shreshtha has quit IRC10:44
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Stream the Ansible output to a Zaqar queue  https://review.openstack.org/52105810:45
*** paramite has quit IRC10:48
*** akane has quit IRC10:52
*** akane_ has quit IRC10:52
*** salmankhan has joined #tripleo10:53
*** dbecker has joined #tripleo10:54
ansiwenmy change depending on a python-tempestconf change is not using the new python-tempestconf10:54
ansiwenhttp://logs.openstack.org/39/515139/3/check/legacy-tripleo-ci-centos-7-scenario002-multinode-oooq-container/cbeb36f/job-output.txt.gz#_2017-11-16_17_12_25_40306310:55
chandankumaransiwen: because these jobs uses rpm10:55
ansiwenchandankumar: but it pulls the changed version, if I understand the logs correctly10:56
*** udesale has quit IRC10:56
ansiwenchandankumar: or is that log line that I just sent unrelated?10:56
ansiwenchandankumar: you can see before, that the changes are pulled at least10:57
*** dtantsur|afk is now known as dtantsur10:57
chandankumaransiwen: seeing first time10:57
*** salmankhan has quit IRC10:57
ansiwenchandankumar: how else can I test python-tempestconf changes? manually?10:57
chandankumaransiwen: can you add tempest_format: venv in https://review.openstack.org/#/c/515139 this review?10:59
*** shreshtha has joined #tripleo11:02
openstackgerritLee Yarwood proposed openstack/tripleo-common master: config: Always add step conditional first for upgrade_tasks  https://review.openstack.org/52106011:02
shardyjistr: Hey question re the kube*localhost kubespray params, did you test without those, as I need hypercube to bind to the internal API ip not :::6443 or we get a conflict with binding to that port on the vip11:04
shardyI removed them and still see hyperkube bound to all interfaces11:04
shardytrying to figure out the vars needed but wanted to see if you'd already debugged similar11:04
*** ykarel is now known as ykarel|afk11:06
*** salmankhan has joined #tripleo11:06
ansiwenchandankumar: I don't understand, can you explain for stupid people?11:09
*** pgadiya has quit IRC11:09
*** ooolpbot has joined #tripleo11:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION11:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432811:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173103211:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173106311:10
*** ooolpbot has quit IRC11:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)11:10
openstackgerritLee Yarwood proposed openstack/tripleo-common master: config: Always add step conditional first for upgrade_tasks  https://review.openstack.org/52106011:10
openstackLaunchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]11:10
openstackLaunchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged]11:10
*** pgadiya has joined #tripleo11:10
*** dpawar has joined #tripleo11:15
*** cdearborn has joined #tripleo11:15
*** stendulker has quit IRC11:15
*** dpawar has quit IRC11:18
*** dpawar has joined #tripleo11:22
openstackgerritChandan Kumar proposed openstack/tripleo-quickstart master: TEST DON'T MERGE - Enabling EC2-API Tempest tests.  https://review.openstack.org/51513911:22
chandankumaransiwen: ^^ it might help11:22
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Stream the Ansible output to a Zaqar queue  https://review.openstack.org/52105811:23
dpawarin a containerized overcloud deployment, if i want to enable debug on nova_compute container so that i can get debug log of nova, how can i do that ?11:24
dpawarshardy: ^^11:24
*** ykarel|afk is now known as ykarel11:26
yolandahi, question... i'm doing some test on upgrades and i hit https://bugs.launchpad.net/tripleo/+bug/171068511:27
openstackLaunchpad bug 1710685 in tripleo "Ugly output when registering nodes" [High,Triaged]11:27
yolandai say it went to fix released, then on triaged11:27
shardydpawar: the same as pre-containers, e.g set either Debug: true or NovaDebug: true in an environment file, then re-run the deploy command11:27
yolandawhat's the status of it, is that a known bug?11:27
shardydpawar: that should regenerate the nova config with debug enabled and restart the service container11:27
shardydpawar: you could probably also manually hack the generated config under /var/lib/config-data and restart the service container manually, but obviously any subsequent deploy via heat will overwrite that11:29
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Stream the Ansible output to a Zaqar queue  https://review.openstack.org/52105811:30
dpawarshardy: in the second option, with that be a persistent change ? i mean across restarts of containers11:31
shardydpawar: no, as I said the next time you do any kind of update your changes will be erased11:33
shardyso the heat param approach is probably best, even though it'll take slightly longer to apply11:33
*** shreshtha has quit IRC11:35
dpawarright11:35
*** pgadiya has quit IRC11:36
*** pradk has quit IRC11:39
openstackgerritJulie Pichon proposed openstack/python-tripleoclient master: Mock creation of ~/.tripleo directory  https://review.openstack.org/52106511:41
dpawarshardy: also we add veritas entry in nova.conf in nova_compute, eg enabled_filters. is there any way to do that ? like for block-storage/cinder we already have hooks for adding veritas entries in cinder.conf, in pre-containers veritas installer used to add this entry11:42
Tenguduh. I think certodia is almost finished. Just have to finish the "custodia" library, and create a bunch of tests for the libs… once it's done, let's encrypt certificate will be supported in a really nice way for openstack public endpoints \o/11:43
shardydpawar: there is a NovaSchedulerDefaultFilters parameter, which should work exactly the same as pre-containers11:43
shardydpawar: because we still use puppet to generate the config files, all the pre-containers parameters and hiera overrides should still work exactly the same as before11:44
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Stream the Ansible output to a Zaqar queue  https://review.openstack.org/52105811:53
*** ratailor has quit IRC11:54
dpawarshardy: one more - veritas binaries reside on host. where as since now nova volume service is now containerized, 1. veritas extension of nova driver (opensourced) expects a veritas_cli to be present inside container. 2. veritas_cli needs to communicate services running on host (outside container). is there any way to achieve that ?11:54
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Stream the Ansible output to a Zaqar queue  https://review.openstack.org/52105811:54
dpawari mean how to achieve 1 and then 2 ?11:54
*** hjensas has quit IRC11:55
*** yamahata has quit IRC11:55
*** rhallisey has joined #tripleo12:00
shardydpawar: if the tools are packaged and open source we can probably add them to the nova volume container image, if not you may need to rebuild the container image to add those tools12:01
shardydpawar: The normal pattern for bootstrapping tasks isn't to run them inside the service container, but instead we launch temporary containers (by default using the same image) that run either puppet or CLI commands12:02
shardydpawar: what services on the host does the tool connect to?12:02
*** raildo has joined #tripleo12:05
*** tvignaud has quit IRC12:05
*** tvignaud has joined #tripleo12:05
jankiHi. Can I get some reviews on https://review.openstack.org/#/c/520573/12:08
dpawarshardy: 1. tool is not opensourced. the nova driver is opensourced. pre-containers because the veritas_cli was present on host (/usr/bin/) and nova driver was also no host hence there was no issue. 2. the vertas_cli communicates with rabbitmq server and requires certain python libraries12:08
*** ramishra has quit IRC12:08
*** dbecker has quit IRC12:08
jaosoriorflaper87: once for all the controller nodes.12:09
jaosoriorflaper87: so we generate those via tripleo-common; and then inject them to all the controllers. They have to be the same.12:09
dpawarveritas_cli requires mq libraries as well as veritas libraries12:09
shardydpawar: ack, it's kind of weird to have an open source drive which depends on a proprietary tool IMO12:10
shardydriver12:10
*** ooolpbot has joined #tripleo12:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION12:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432812:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)12:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173103212:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173106312:10
*** ooolpbot has quit IRC12:10
openstackLaunchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]12:10
openstackLaunchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged]12:10
jaosoriorshardy: hey, if you get some time could you take another look at this https://review.openstack.org/#/c/520346/ ? and of course, if you think we should do another approach, I would really appreciate the feedback.12:10
shardydpawar: would it be possible to build a container image with only the veritas cli and it's deps?12:10
*** psahoo has quit IRC12:10
shardyjaosorior: ack will do12:11
*** janki has quit IRC12:11
openstackgerritMerged openstack/tripleo-heat-templates master: Set host name explicitly for telemetry  https://review.openstack.org/50919912:11
*** dtantsur is now known as dtantsur|brb12:12
dpawarshardy: do you mean, take existing nova compute image and rebuilt it with required cli and deps and use it at the time of overcloud deploy? will that be acceptable ?12:13
*** udesale has joined #tripleo12:13
shardydpawar: for upstream, no, but we'd have to have a discussion about the distribution/support model for downstream builds12:14
ansiwenchandankumar: I see, thanks!12:14
shardydpawar: thats's why I wondered if it's possible to decouple the driver part from the CLI part12:14
shardydpawar: is the CLI used for setup/bootstrapping or does the driver actually call it?12:14
dpawarshardy: drives actually calls CLI while connecting to volume12:15
dpawarshardy: AFAIK, every third party driver calls CLI for this12:16
shardydpawar: Ok, I guess rebuilding the nova image is the only option then - the TripleO interfaces enable that, you'd just pass a different DockerCinderVolumeImage or whatever to consume the modified image12:17
shardyobviously maintaining that image forever is a maintenance burden tho12:17
shardydpawar: how did you install the proprietary bits pre-containers, virt-customize of the overcloud-full image?12:18
dpawarshardy: what if we patch and commit the running container ?12:18
shardydpawar: thanks kind of a container anti-pattern, and we don't have any interfaces that enable it atm12:19
shardymandre: ^^ any thoughts on this, does kolla have any existing patterns to handle vendor plugins?12:20
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates master: Check DeployedServerPortMap  https://review.openstack.org/52081612:21
mandreshardy, dpawar: I need to take kids to school, will be back in a snap12:21
shardymandre: ack, thanks12:22
*** shardy is now known as shardy_lunch12:22
dpawarshardy: for install of prorietory bits pre-containers we used overcloud deploy command that calls veritas_puppet modules which do install and configure veritas hyperscale.12:23
dpawarshardy_lunch: for install of prorietory bits pre-containers we used overcloud deploy command that calls veritas_puppet modules which do install and configure veritas hyperscale.12:23
*** suuuper has quit IRC12:24
*** dpawar has quit IRC12:30
*** lucasagomes is now known as lucas-hungry12:33
*** abishop has joined #tripleo12:36
*** social has joined #tripleo12:38
*** apetrich has quit IRC12:41
openstackgerritWojciech Dec proposed openstack/puppet-tripleo master: Adding manifest for Cisco VTS ML2 mechanism driver configuration  https://review.openstack.org/51064512:42
*** salmankhan has quit IRC12:45
*** nguyentrihai has joined #tripleo12:45
*** jlabarre has joined #tripleo12:46
*** akane has joined #tripleo12:47
*** akane_ has joined #tripleo12:47
*** thrash|g0ne is now known as thrash12:47
*** dpawar has joined #tripleo12:47
*** jpena is now known as jpena|lunch12:50
*** salmankhan has joined #tripleo12:51
*** pradk has joined #tripleo12:55
*** salmankhan has quit IRC12:56
*** salmankhan has joined #tripleo12:57
*** akane has quit IRC12:59
*** akane_ has quit IRC12:59
*** ramishra has joined #tripleo13:00
*** salmankhan has quit IRC13:01
*** psachin has quit IRC13:01
mandredpawar: so generally, if you need to include 3rd party packages to a kolla image, you do this by using the kolla customization mechanism to build customized images13:03
mandredpawar: https://docs.openstack.org/kolla/latest/admin/image-building.html#dockerfile-customisation13:03
*** dtantsur|brb is now known as dtantsur13:03
jaosoriorflaper87, jistr: At what point in the deployment does the external_deploy_tasks stuff gets ran?13:03
jaosorioris it part of the regular steps?13:03
jaosoriorflaper87, jistr: We're looking, with jkilpatr to use that in order to deploy a containerized rsyslog.13:04
jaosoriorwith ansible.13:04
mandredpawar: however, in the tripleo/osp case were we build the base images, you'll need to add a new layer to the existing images13:05
mandredpawar: https://docs.openstack.org/tripleo-docs/latest/install/containers_deployment/3rd_party.html13:05
*** ffiore has quit IRC13:05
openstackgerritOliver Walsh proposed openstack/puppet-tripleo master: Configure libvirt SASL SCRAM-SHA1 when TLS is enabled  https://review.openstack.org/52025313:05
*** cshastri has joined #tripleo13:05
*** skramaja has quit IRC13:09
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates stable/pike: Set host name explicitly for telemetry  https://review.openstack.org/52108213:09
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates master: Re-enable libvirt TLS with SCRAM SHA-1 auth  https://review.openstack.org/52025413:09
*** ooolpbot has joined #tripleo13:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION13:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432813:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173103213:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173106313:10
*** ooolpbot has quit IRC13:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)13:10
openstackLaunchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]13:10
openstackLaunchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged]13:10
*** dprince has joined #tripleo13:10
*** atoth has quit IRC13:10
*** ecerquei has joined #tripleo13:13
*** leitan has joined #tripleo13:15
*** dhill_ has quit IRC13:15
*** trown|outtypewww is now known as trown13:17
*** ramishra has quit IRC13:17
*** chlong__ has quit IRC13:18
*** atoth has joined #tripleo13:18
*** shardy_lunch is now known as shardy13:18
*** akane has joined #tripleo13:19
*** akane_ has joined #tripleo13:19
*** links has quit IRC13:19
*** holser has quit IRC13:19
*** toure_biab is now known as toure13:20
*** lblanchard has joined #tripleo13:21
jaosoriorjkilpatr: but yeah, more or less it would look very similar to this: https://github.com/openstack/tripleo-heat-templates/blob/master/extraconfig/services/kubernetes-master.yaml#L5813:22
*** ramishra has joined #tripleo13:22
openstackgerritFlavio Percoco proposed openstack/tripleo-quickstart master: Add featureset33 for scenario009  https://review.openstack.org/49441813:22
*** holser has joined #tripleo13:22
dpawarmandre: Thanks! will go through these docs and get back to you if requred.13:22
dpawarshardy: Thanks!13:22
openstackgerritFlavio Percoco proposed openstack/tripleo-heat-templates master: Deploy OpenShift using OOO on the overcloud  https://review.openstack.org/49447013:23
jaosoriorjkilpatr: except that we would let tripleo handle the docker container creation, like this https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/collectd.yaml#L9113:23
*** holser has quit IRC13:24
*** holser has joined #tripleo13:25
mandredpawar: anytime!13:26
*** rlandy has joined #tripleo13:29
*** rlandy is now known as rlandy|rover13:29
*** janki has joined #tripleo13:30
*** lucas-hungry is now known as lucasagomes13:30
*** apetrich has joined #tripleo13:31
*** athomas has joined #tripleo13:34
*** ffiore has joined #tripleo13:34
jaosoriorshardy: could you revisit this https://review.openstack.org/#/c/520333/ ?13:38
*** eck` is now known as eck`gone13:39
shardyjaosorior: done13:39
jaosoriorshardy: thanks13:40
*** cshastri has quit IRC13:42
*** morazi has joined #tripleo13:45
*** aditya_r has joined #tripleo13:46
*** salmankhan has joined #tripleo13:46
*** jpena|lunch is now known as jpena13:52
*** artom has quit IRC13:58
*** jmelvin has joined #tripleo14:02
*** dtantsur is now known as dtantsur|brb14:03
*** jkilpatr has quit IRC14:03
*** zshi has quit IRC14:06
openstackgerritDan Prince proposed openstack/tripleo-quickstart-extras master: Don't try to start docker for the UC  https://review.openstack.org/51944114:08
*** lblanchard has quit IRC14:09
*** dbecker has joined #tripleo14:10
*** ooolpbot has joined #tripleo14:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION14:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432814:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173103214:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173106314:10
*** ooolpbot has quit IRC14:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)14:10
openstackLaunchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]14:10
openstackLaunchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged]14:10
openstackgerritDan Prince proposed openstack/tripleo-heat-templates master: docker: Add unit tests on service_name  https://review.openstack.org/44275514:11
*** almondjoy has joined #tripleo14:11
*** beagles has joined #tripleo14:13
*** chandankumar is now known as chkumar|somewher14:13
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Stream the Ansible output to a Zaqar queue  https://review.openstack.org/52105814:14
*** tzumainn has joined #tripleo14:14
*** ykarel is now known as ykarel|away14:14
*** jkilpatr has joined #tripleo14:16
openstackgerritmathieu bultel proposed openstack/tripleo-common master: Update ceph-ansible playbook path in parameter default for update  https://review.openstack.org/51544214:16
pradkcan i request some review on this backport https://review.openstack.org/#/c/521082/14:17
*** rfolco is now known as rfolco_lunch14:17
*** salmankhan has quit IRC14:18
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Stream the Ansible output to a Zaqar queue  https://review.openstack.org/52105814:18
EmilienMhi14:19
*** salmankhan has joined #tripleo14:20
openstackgerritmathieu bultel proposed openstack/tripleo-common master: Stream the Ansible output to a Zaqar queue  https://review.openstack.org/52105814:20
*** ykarel|away has quit IRC14:20
jaosoriormatbu_: could you check this out https://review.openstack.org/#/c/520333/ ?14:23
matbu_jaosorior: sure14:25
*** dxiri has joined #tripleo14:25
*** eck`gone is now known as eck`14:25
chemmatbu_: when using master-tripleo-ci as a release I've got an consistant problem with openstack-ceilometer-api package during undercloud install is that known ?14:26
*** gbarros has joined #tripleo14:27
*** salmankhan has quit IRC14:28
EmilienMmwhahaha: gate is ok'ish14:29
EmilienMmwhahaha: we could start landing the zuul v3 things14:29
EmilienMwdyt?14:29
EmilienMit's 5h now14:29
d0ugalmatbu_: IIUC doing the queue post after the loop will send the full stdout in one message when the process finishes.14:29
matbu_chem: hmm i wasn't aware of14:29
matbu_d0ugal: yep but is it a pb ?14:29
matbu_d0ugal: if the message is too big maybe14:30
d0ugalmatbu_: but I thought the point was to have realtime output while the process was still running?14:30
matbu_d0ugal: yep true, so sorry about the change, anyway the post() wasn't correct14:31
*** udesale has quit IRC14:31
matbu_d0ugal: but i will still working on it14:31
matbu_folks upgrade squad mtg14:31
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Stream the Ansible output to a Zaqar queue  https://review.openstack.org/52105814:32
d0ugalmatbu_: np, I just fixed an issue I spotted ^ but I'll leave you to work on it now14:32
d0ugalmatbu_: let me know if I can help.14:32
*** dxiri has quit IRC14:33
*** ramishra has quit IRC14:34
*** anilvenkata has quit IRC14:34
*** ramishra has joined #tripleo14:35
weshaypradk, thanks for jumping into that rdo-cloud escalation14:36
weshaypradk++14:36
*** karthiks has quit IRC14:36
jaosoriorchem: do you know how to fix that part?14:38
jaosoriorchem: of the VIP generation?14:38
*** chem is now known as chem_mtg14:42
chem_mtgjaosorior: in the link [2] there is a example on how to do that14:42
chem_mtgjaosorior: basically you check if compat_name is defined or else fall back to name14:43
chem_mtgjaosorior: makes sense ?14:43
jaosoriorkinda14:44
jaosoriorish14:44
mwhahahaEmilienM: I want to see a few of these things merge first. also check to see if there are any important patches we're missing14:47
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Dynamically generate VIP hiera entries  https://review.openstack.org/52033314:47
jaosoriorchem_mtg: is this better https://review.openstack.org/#/c/520333/3/puppet/all-nodes-config.j2.yaml ?14:47
*** salmankhan has joined #tripleo14:48
openstackgerritRyan Brady proposed openstack/tripleo-common master: Validate roles data and network data  https://review.openstack.org/50856714:49
openstackgerritRyan Brady proposed openstack/tripleo-common master: Add a Get Networks workflow  https://review.openstack.org/50941914:49
openstackgerritRyan Brady proposed openstack/tripleo-common master: Add UpdateNetworks workflow  https://review.openstack.org/51346314:49
openstackgerritRyan Brady proposed openstack/tripleo-common master: Adds Workflow to List Available Roles  https://review.openstack.org/50983214:49
openstackgerritRyan Brady proposed openstack/tripleo-common master: [WIP] Adds UpdateRoles Workflow  https://review.openstack.org/51504514:49
EmilienMmwhahaha: ok14:51
EmilienMmatbu_: fyi, we asked to not approve patches on the ML14:51
EmilienMchem_mtg: same ^14:52
trownitching to use those new powers :P14:52
trowncongrats chem_mtg and mwhahaha14:52
chem_mtgEmilienM: ack14:52
trownerr14:52
trown matbu_14:52
mwhahahai see how it is14:52
chem_mtgjaosorior: looking again14:53
EmilienMtrown: "with great powers come..." I forgot the rest :P14:53
mwhahahain the mean time, it looks like ovs problems http://logs.openstack.org/70/519770/3/gate/legacy-tripleo-ci-centos-7-scenario003-multinode-oooq-container/1db4d85/logs/subnode-2/var/log/containers/neutron/neutron-l3-agent.log.txt.gz#_2017-11-17_07_48_14_01114:53
EmilienMmwhahaha: OVS 2.8?14:53
mwhahahano idea14:53
EmilienMnumans: are you around?14:53
EmilienMmwhahaha: is that your debug patch ^?14:53
mwhahahayes 2.8.114:54
mwhahahano that was the test_mtu_size_frames patch https://review.openstack.org/#/c/519770/14:54
EmilienMok14:54
mwhahahabut that's the 2nd one i've seen fail on scenario003 with errors in there14:54
EmilienMmwhahaha: can you confirm the same neutron-l3 error in scenario001 in previous days14:55
EmilienM?14:55
mwhahahait is not14:55
EmilienM:(14:55
mwhahahawell same general failure14:55
mwhahahabut i think different cause14:55
mwhahahametadata agent does have a connection problem to rabbitmq14:55
mwhahahabut it's ealier than the test14:55
mwhahahaso not sure if related14:55
*** akrivoka has joined #tripleo14:56
mwhahahahere's the other ovs one http://logs.openstack.org/56/519756/4/check/legacy-tripleo-ci-centos-7-scenario003-multinode-oooq-container/1b81416/logs/subnode-2/var/log/containers/neutron/neutron-l3-agent.log.txt.gz#_2017-11-17_06_18_16_92314:56
mwhahahathat was my DMN test patch of my debug flag14:56
*** dxiri has joined #tripleo14:57
mwhahahaboth had http://logs.openstack.org/56/519756/4/check/legacy-tripleo-ci-centos-7-scenario003-multinode-oooq-container/1b81416/logs/subnode-2/var/log/containers/neutron/neutron-metadata-agent.log.txt.gz#_2017-11-17_06_15_10_69514:57
mwhahahawhich is also concerning14:57
*** ratailor has joined #tripleo14:58
chem_mtgjaosorior: I've just checked the newton and there was a change only with *InternalNetwork*, not with *InternalApiNetMap* so my comment was irrelevant14:58
jaosoriorchem_mtg: oh14:58
*** yamahata has joined #tripleo14:58
jaosoriorchem_mtg: I'll revert to the previous version then14:58
chem_mtgjaosorior: sorry about that, so yeap revert.14:59
*** psachin has joined #tripleo14:59
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Dynamically generate VIP hiera entries  https://review.openstack.org/52033315:00
mwhahahaEmilienM: speaking of broken things, did you see my note about the release notes jobs being broken last night https://bugs.launchpad.net/tripleo/+bug/173281515:00
openstackLaunchpad bug 1732815 in tripleo "CI: releasenotes job broken with Could not import extension oslosphinx (exception: No module named oslosphinx) " [Critical,Triaged]15:00
jaosoriorchem_mtg: done ^^15:00
*** janki has quit IRC15:00
mwhahahaEmilienM: and https://bugs.launchpad.net/tripleo/+bug/173281615:00
openstackLaunchpad bug 1732816 in tripleo "CI: releasenotes job fails due to ValueError: Unknown reference 'origin/stable/ocata'" [Critical,Triaged]15:00
EmilienMI didn't15:01
EmilienMmwhahaha: have you talked with Doug?15:01
*** chlong__ has joined #tripleo15:01
EmilienMI don't see why it fails now15:01
mwhahahanot yet15:01
EmilienMmaybe something changed outside of our control15:02
*** dtantsur|brb is now known as dtantsur15:02
*** artom has joined #tripleo15:04
ratailorhttps://review.openstack.org/#/c/520273/15:04
ratailorhttps://review.openstack.org/#/c/520274/15:04
ratailorEmilienM, mwhahaha could you please set workflow for these ^^ patches, these needed to be merged before OSP12 GA.15:04
EmilienMratailor: see mailing-list, we don't approve patches today. We'll probably approve it over the weekend.15:05
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: Add S3 storage driver as an option and its config  https://review.openstack.org/52109315:05
ratailorEmilienM, sure, no problem.15:05
openstackgerritBeth Elwell proposed openstack/tripleo-heat-templates stable/pike: Disabled fwaas plugin for horizon  https://review.openstack.org/52109915:08
openstackgerritDmitry Tantsur proposed openstack/instack-undercloud master: [DO NOT MERGE] Just testing, nothing to see here  https://review.openstack.org/52110015:09
*** artom_ has joined #tripleo15:09
openstackgerritSteven Hardy proposed openstack/puppet-tripleo master: Add kubernetes API to haproxy LB configuration  https://review.openstack.org/51640715:10
*** ooolpbot has joined #tripleo15:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION15:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432815:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173103215:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173106315:10
*** ooolpbot has quit IRC15:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)15:10
EmilienMmwhahaha: interesting to see only scenarios that are unstable15:10
openstackLaunchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]15:10
openstackLaunchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged]15:10
EmilienMmwhahaha: and not other classic multinode jobs15:10
*** dbecker has quit IRC15:10
weshayEmilienM, yup15:10
mwhahahaEmilienM: do classic run tempest?15:10
*** dbecker has joined #tripleo15:10
EmilienMmwhahaha: yep, afik15:10
EmilienMunless it changed15:10
EmilienMmwhahaha: I think gate is about to reset15:11
EmilienMmwhahaha: see first job on top, http://zuulv3.openstack.org/static/stream.html?uuid=cae58e56b98c4cab83a9effe287e0f1b&logfile=console.log15:11
*** udesale has joined #tripleo15:11
mwhahahameh15:11
mwhahahai'm approving the v3 patches to get them in now15:11
EmilienMk15:11
weshaymwhahaha, yes.. it runs tempest http://logs.openstack.org/84/514784/10/gate/legacy-tripleo-ci-centos-7-containers-multinode/617922a/logs/stackviz/#/stdin15:12
*** udesale has quit IRC15:12
*** artom has quit IRC15:12
weshayhttp://logs.openstack.org/84/514784/10/gate/legacy-tripleo-ci-centos-7-containers-multinode/617922a/logs/stackviz/#/stdin/timeline?test=tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern15:12
openstackgerritAlan Bishop proposed openstack/tripleo-heat-templates stable/pike: Update Iscsid puppet_config to reset host's IQN  https://review.openstack.org/52110215:13
mwhahahaEmilienM: i hope the project-config patch doesn't hang on https://review.openstack.org/#/q/I34113054a2f9ca4c9aa7e7b1ad3bb8dcc76523f6 since one of those is abanonded15:14
mwhahahaI hope gerrit isn't going to be dumb15:14
mwhahahaanyway all the v3 patches are queued15:14
EmilienMmwhahaha: we're talking about the reno errors on #openstack-meeting15:14
EmilienMmwhahaha: ok, I'll remove one depends-on and make infra re-approve it15:15
*** chem has joined #tripleo15:15
mwhahahaEmilienM: no let's just wait15:15
mwhahahai think it should be ok15:15
EmilienMI don't think so15:15
EmilienMin my experience, all depends-on need to be merged15:16
*** chem_mtg has quit IRC15:16
*** rmascena has joined #tripleo15:16
weshayarxcruz, are we pushing https://review.openstack.org/#/c/518723/ along? I'd like to see some progress on the tempest emails. Do you need help or to discuss?15:16
arxcruzweshay: yes, I'll update the patch15:17
openstackgerritMartin AndrĂ© proposed openstack/tripleo-heat-templates master: Bind mount /etc/fstab to generate glance-api config  https://review.openstack.org/52110315:18
*** aditya_ra has joined #tripleo15:18
jaosoriormandre: commented ^^15:18
*** raildo has quit IRC15:19
*** dxiri has quit IRC15:19
*** jaosorior has quit IRC15:19
*** aditya_r has quit IRC15:21
*** bobh has joined #tripleo15:22
mwhahahaEmilienM: so going back to the classic vs scenarios. It does seem like it's a resource problem15:24
mwhahahaweshay: did we ever fix the missing dstat?15:24
*** aditya_ra has quit IRC15:24
weshaymwhahaha, aye.. rlandy did15:25
openstackgerritOliver Walsh proposed openstack/puppet-tripleo master: Configure libvirt SASL SCRAM-SHA1 when TLS is enabled  https://review.openstack.org/52025315:25
*** ramishra has quit IRC15:28
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: Add MTU adjustment parameters to multinode settings  https://review.openstack.org/52111215:30
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: dns from libvirthost should be off for ovb  https://review.openstack.org/52111315:30
*** ratailor has quit IRC15:31
*** itlinux has joined #tripleo15:31
rlandy|rovermwhahaha: https://review.openstack.org/#/c/519725/ - dstat fix15:35
mwhahaharlandy|rover: k i'll recheck it after the v3 stuff lands. thanks15:35
rlandy|rovermwhahaha: thanks15:36
*** mrch has quit IRC15:36
*** ramishra has joined #tripleo15:38
*** derekh has quit IRC15:40
*** artom_ is now known as artom15:40
openstackgerritBeth Elwell proposed openstack/tripleo-heat-templates master: Disabled fwaas plugin for horizon  https://review.openstack.org/52111615:41
*** itlinux has quit IRC15:42
*** ramishra has quit IRC15:43
*** itlinux has joined #tripleo15:44
*** ramishra has joined #tripleo15:45
jpichakrivoka: Noticed your vote on https://review.openstack.org/#/c/518619/ - it's been over a week since your tripleo-validations core nominations with loadsa +1 and no objections, IMO it's fine to start using your +2 powers :)15:48
*** cylopez has quit IRC15:48
akrivokajpich: I have no +2 power on validations yet15:49
jpichakrivoka: Oh, really! I thought you had generic TripleO +2 powers with the usual "use your judgement for where to use etc", my bad. Hm... Ok let's find out how to fix this15:50
akrivokajpich: I actually have no core status anywhere on tripleo (I have technically on tripleo-ui, just because I created the repo/gerrit, but I'm not supposed to use it)15:52
akrivokajpich: thanks for looking into this!15:52
mwhahahaakrivoka: jpich: fixed15:53
akrivokamwhahaha: jpich: works, thanks!15:53
jpichmwhahaha: Thank you! What steps does this involve? Adding someone to the tripleo-core group on gerrit, or are there other steps as well?15:53
mwhahahajpich: just that15:53
jpichGreat, good to know15:53
mwhahahaplease don't approve anything at the moment15:53
mwhahahasee ML as we're trying to switch to zuul v315:54
akrivokaack15:54
mwhahahaif you need something merged, let me know and i'll add it to the list to approve after the v3 stuff lands15:54
mwhahahatrying to make this as quick and painless as possible :D15:54
akrivokajpich: I amended my vote on the above patch, but it still needs a second +215:54
ccamachohey mwhahaha I remove the +A in a backport to pike https://review.openstack.org/#/c/519680 added you to the review, can you +A it when is the correct time?15:54
mwhahahaccamacho: sure15:55
ccamachothanks man15:55
ccamachobeagles ^15:55
*** thrash is now known as thrash|biab15:55
jpichakrivoka: Yep, but it's one step closer now :-) Cheers! And welcome/congrats \o/15:55
beaglesccamacho, ack15:55
akrivokajpich: yaay!15:55
akrivokathanks :)15:55
openstackgerritJose Luis Franco proposed openstack/tripleo-quickstart master: Set correct docker registry namespace.  https://review.openstack.org/52112115:57
*** artom_ has joined #tripleo16:00
*** artom has quit IRC16:04
*** artom__ has joined #tripleo16:06
*** artom_ has quit IRC16:09
*** ooolpbot has joined #tripleo16:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION16:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432816:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173103216:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173106316:10
*** ooolpbot has quit IRC16:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)16:10
openstackLaunchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]16:10
openstackLaunchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged]16:10
rlandy|roverjfrancoa: FYI regarding your review above, we decided to redo all the defaults on docker registry: https://review.openstack.org/#/c/520639/  and ... https://review.openstack.org/#/c/520201/16:12
jfrancoarlandy|rover: thanks for the info, I observed that upgrades jobs were failing because they could not find the container image. But if it's already implemented I will abandon the patch16:13
*** ffiore has quit IRC16:18
*** akane_ has quit IRC16:19
*** akane has quit IRC16:19
*** ccamacho has quit IRC16:23
*** dpawar has quit IRC16:23
*** ffiore has joined #tripleo16:23
*** thrash|biab is now known as thrash16:26
*** f13o has joined #tripleo16:28
*** artom__ is now known as artom16:29
*** bogdando has quit IRC16:31
*** dxiri has joined #tripleo16:33
*** dxiri has joined #tripleo16:33
*** shardy has quit IRC16:38
*** jistr has quit IRC16:39
*** shardy has joined #tripleo16:39
openstackgerritOliver Walsh proposed openstack/puppet-tripleo master: Configure libvirt SASL SCRAM-SHA1 when TLS is enabled  https://review.openstack.org/52025316:39
*** ebarrera has quit IRC16:40
*** gkadam has quit IRC16:41
*** gkadam has joined #tripleo16:41
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart master: Specify docker.io defaults for upstream config release files.  https://review.openstack.org/52063916:43
*** jistr has joined #tripleo16:43
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Update the Docker registry defaults to rdoproject  https://review.openstack.org/52020116:44
*** yamahata has quit IRC16:47
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: Don't try to start docker for the UC  https://review.openstack.org/51944116:48
*** lucasagomes is now known as lucas-afk16:48
*** holser has quit IRC16:52
*** holser has joined #tripleo16:53
*** f13o has quit IRC17:01
*** psachin has quit IRC17:02
*** holser has quit IRC17:03
*** holser has joined #tripleo17:03
openstackgerritJulien Danjou proposed openstack/tripleo-heat-templates stable/newton: ceilometer: set event dispatcher to Gnocchi by default  https://review.openstack.org/52113217:07
*** gkadam has quit IRC17:08
*** ooolpbot has joined #tripleo17:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION17:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432817:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173103217:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173106317:10
*** ooolpbot has quit IRC17:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)17:10
openstackLaunchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]17:10
openstackLaunchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged]17:10
*** marios has quit IRC17:12
openstackgerritJulien Danjou proposed openstack/tripleo-heat-templates stable/newton: ceilometer: set event dispatcher to Gnocchi by default  https://review.openstack.org/52113217:12
*** ffiore has quit IRC17:13
*** ykarel|away has joined #tripleo17:19
EmilienMweshay: have you found the file in tripleo-ci? i don't see the patch yet17:20
EmilienMweshay: now the tripleo-ci patch has merged, I can re-add the jobs in projects like mistral, etc17:21
EmilienMif you see what I mean17:21
EmilienMthe job / template layout is merged17:21
* mwhahaha waits for the grass to grow17:21
weshayEmilienM, you had it set correctly in tripleo-ci17:21
weshayI shouldn't need to change anything17:21
EmilienMweshay: you're welcome.17:22
openstackgerritMerged openstack/tripleo-common stable/pike: Truncate the zaqar message to 512 kbytes  https://review.openstack.org/52057117:22
weshayEmilienM, You are welcome AMERICA.. thank you17:22
EmilienM:P17:22
*** dougbtv_ has joined #tripleo17:22
*** shardy has quit IRC17:23
openstackgerritAlex Schultz proposed openstack/tripleo-quickstart-extras master: Add -n to lsof collection  https://review.openstack.org/52113617:23
*** shardy has joined #tripleo17:24
*** shardy has quit IRC17:25
*** rfolco_lunch is now known as rfolco17:25
*** dbecker has quit IRC17:26
*** holser has quit IRC17:29
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates master: WIP: Mount etc/my.cnf.d/tripleo.cnf in all db_sync containers  https://review.openstack.org/52113917:30
*** dparkes has quit IRC17:30
openstackgerritMerged openstack/tripleo-heat-templates master: Add validation task in docker services [Mixed-3]  https://review.openstack.org/50558717:32
dprinceweshay: added you as coauthor on https://review.openstack.org/#/c/519441/. With your latest commit msg update do you think we are ready to land it?17:37
*** brault has quit IRC17:39
openstackgerritJulie Pichon proposed openstack/python-tripleoclient master: Add mock to avoid writing missing files to disk  https://review.openstack.org/52114017:40
*** sri__ has quit IRC17:41
*** jfrancoa has quit IRC17:41
EmilienMsshnaidm|off: I sent an email and ask to not approve patches17:45
openstackgerritRonelle Landy proposed openstack-infra/tripleo-ci master: Add MTU adjustment parameters to multinode settings  https://review.openstack.org/52111217:45
EmilienMsshnaidm|off: in fact, it's gchamoul who did a recheck...17:45
EmilienMgchamoul: please no recheck, like requested on tripleo ML17:45
openstackgerritAlex Schultz proposed openstack/tripleo-quickstart-extras master: Use openstack commands for postci  https://review.openstack.org/52114617:46
* mwhahaha optimizes17:46
EmilienMpradk: please no +A, like sent on ML17:46
*** dtantsur is now known as dtantsur|afk17:48
*** jpich has quit IRC17:48
weshaydprince, yes I want to recheck it w/ the job definition that EmilienM has in tripleo-ci now17:49
weshaybut I think we're good..17:49
weshaydprince, the interesting thing was that it was not the duplicate entries in /etc/hosts17:49
dprinceweshay: it was the way hosts was configured for multinode jobs?17:50
weshay\/etc/hosts had to be completely cleared in the same we clear for the overcloud deployments upstream17:50
dprinceweshay: okay. The latest run looks to be failing though. Is that expected?17:50
*** dougbtv_ has quit IRC17:51
dprinceyou removed the Depends-On for the CI patch I think?17:51
weshayhrm.. k will check in a sec17:51
*** ykarel|away has quit IRC17:51
openstackgerritDan Prince proposed openstack/tripleo-heat-templates master: docker: Add unit tests on service_name  https://review.openstack.org/44275517:54
*** gbarros has quit IRC18:02
*** cylopez has joined #tripleo18:02
EmilienMnumans: can we move scenario007-container to gate and stop experimental?18:03
openstackgerritMerged openstack/tripleo-heat-templates master: Add validation task in docker services [Sahara]  https://review.openstack.org/50203918:06
openstackgerritMerged openstack/tripleo-heat-templates master: Add validation task in docker services [Ironic]  https://review.openstack.org/50203718:06
openstackgerritMerged openstack/tripleo-heat-templates master: Add validation task in docker services [Neutron]  https://review.openstack.org/48995718:07
openstackgerritMerged openstack/tripleo-heat-templates master: Add validation task in docker services [Aodh]  https://review.openstack.org/49141118:07
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: Move scenario007-container to the gate  https://review.openstack.org/52114918:08
*** ooolpbot has joined #tripleo18:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION18:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432818:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173103218:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173106318:10
*** ooolpbot has quit IRC18:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)18:10
openstackLaunchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]18:10
openstackLaunchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged]18:10
openstackgerritBrad P. Crochet proposed openstack/tripleo-common master: Validate roles data and network data  https://review.openstack.org/50856718:11
openstackgerritBrad P. Crochet proposed openstack/tripleo-common master: Add a Get Networks workflow  https://review.openstack.org/50941918:11
openstackgerritBrad P. Crochet proposed openstack/tripleo-common master: Add UpdateNetworks workflow  https://review.openstack.org/51346318:11
*** jpena is now known as jpena|off18:14
*** ramishra has quit IRC18:20
EmilienMfor these who are curious how simple it is to add tripleo CI jobs outside of tripleo projects, see https://review.openstack.org/#/q/topic:tripleo/scenario00718:21
*** gbarros has joined #tripleo18:24
*** trown is now known as trown|lunch18:29
*** leitan has quit IRC18:30
*** athomas has quit IRC18:31
*** leitan has joined #tripleo18:38
EmilienMarxcruz: thx for restoring the tempest html file, it works now!18:53
*** dhill_ has joined #tripleo18:56
*** hjensas has joined #tripleo18:59
*** hjensas has quit IRC18:59
*** hjensas has joined #tripleo18:59
weshaydprince, the undercloud-containers job is failing because it's again deploying the overcloud18:59
weshayI blame EmilienM :)18:59
*** dsneddon has joined #tripleo19:00
weshayso not sure if we have everything in place yet for zuul19:00
dprinceweshay: would re-adding the Depends-On fix it again?19:00
fultonjEmilienM, regarding topic:tripleo/scenario007 I just want to confirm my understanding....19:00
weshaydprince, ya.. I abandoned the patch, so we may need to turn that back on19:00
fultonjIn 521149, looks like scenario007 already exists and you're moving it from tripleo experimental to regluar tripleo check and gate jobs.19:00
fultonjIn 521152, looks like ovn already uses zuul so you're adding tripleo's scenario007 to it19:00
fultonjright?19:00
EmilienMfultonj: that's 100% exact19:01
fultonjok, cool19:01
EmilienMweshay: what did I do again :P19:01
weshaydprince, restored patch19:01
EmilienMfultonj: and as you can see the benefit of having that in our repos19:02
EmilienMfultonj: we get control of everything quite easily19:03
EmilienMfultonj: also, projects themselves can decide to disable voting when they want19:03
fultonjyeah, it's nice19:03
openstackgerritRyan Brady proposed openstack/tripleo-common master: Adds Workflow to List Available Roles  https://review.openstack.org/50983219:03
openstackgerritRyan Brady proposed openstack/tripleo-common master: [WIP] Adds UpdateRoles Workflow  https://review.openstack.org/51504519:03
*** salmankhan has quit IRC19:04
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: Don't try to start docker for the UC  https://review.openstack.org/51944119:04
fultonjnow if only other projects used zuul....19:04
weshaydprince, ok.. it's back19:04
*** chlong__ has quit IRC19:04
dprinceweshay: ack, thanks19:04
EmilienMfultonj: they do19:04
weshaybbiab19:04
EmilienMfultonj: everyone does19:04
EmilienMfultonj: you mean outside of openstack?19:05
fultonjyeah19:05
*** thrash is now known as thrash|mtg19:05
EmilienMah..19:05
EmilienMweshay: I'm re-adding tripleo jobs to projects (telemetry, etc) like they were before, but with zuul v3 layout. So no lost in the transition19:08
*** ooolpbot has joined #tripleo19:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION19:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432819:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173103219:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173106319:10
*** ooolpbot has quit IRC19:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)19:10
openstackLaunchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]19:10
openstackLaunchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged]19:10
*** akane has joined #tripleo19:15
*** akane_ has joined #tripleo19:15
*** akane_ has quit IRC19:17
*** akane has quit IRC19:17
EmilienMdprince: do we still actually need t-p-e?19:18
dprinceEmilienM: sorry, t-p-e?19:19
EmilienMdprince: tripleo-puppet-elements19:19
EmilienMin a containerized overcloud, I'm unsure where t-p-e is useful19:19
EmilienMbut I probably missed something19:19
dprincethere are still some baremetal dependencies there. Pacemaker19:19
dprinceI've been testing a leaner version though19:20
dprinceEmilienM: sec and I'll so you what worked for me19:20
EmilienMok19:20
*** chlong__ has joined #tripleo19:20
EmilienMI'm trying to find out which CI jobs do we actually need to run for t-p-e, any thoughts on multinode jobs?19:21
dprincefeel free to try it19:21
EmilienMtry what?19:22
*** fragatin_ has quit IRC19:22
openstackgerritDan Prince proposed openstack/tripleo-puppet-elements master: Try removing OS packages from tripleo-puppet-elements  https://review.openstack.org/52117219:25
dprinceEmilienM: ^^ that has worked for me locally. Can probably clean out a bunch more19:25
dprinceEmilienM: the pacemaker and networking stuff needs to stay though19:26
EmilienMoh cool19:26
EmilienMyeah indeed19:26
dprincemade my image a *lot* smaller and thus speeds up larger deployments just a bit19:26
openstackgerritEmilien Macchi proposed openstack/tripleo-puppet-elements master: Migrate instack-undercloud to zuul v3 jobs  https://review.openstack.org/52117419:28
EmilienMdprince: yeah19:28
EmilienMdprince: same question for tripleo-image-elements - do we use it in multinode jobs? i believe not19:30
*** itlinux has quit IRC19:31
dprinceEmilienM: probably not. os-net-config's element is the only peice that is there for the legacy os-net-config Yaml format19:31
EmilienMok19:32
dprinceEmilienM: if the Yaml has been updated should be okay to drop it.19:32
openstackgerritEmilien Macchi proposed openstack/tripleo-image-elements master: Migrate instack-undercloud to zuul v3 jobs  https://review.openstack.org/52117519:33
dsneddondprince, EmilienM: I know I've run across a number of upgrade scenarios where the YAMLs weren't updated. I'm fine with dropping it, since it was technically deprecated for more than one cycle.19:33
*** itlinux has joined #tripleo19:33
dprinceEmilienM, dsneddon: https://review.openstack.org/#/c/470017/19:34
*** amoralej is now known as amoralej|off19:34
dprinceper the last comment from bnemec-pto we were just waiting on CI19:34
*** chlong__ has quit IRC19:36
EmilienMdprince: sounds like we're blocked by newton support19:37
dsneddondprince, Thanks, do you want me to submit a patch to tripleo-docs, since we still document the old element config? https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/network_isolation.html19:37
EmilienMhttps://review.openstack.org/#/c/483064/119:37
dprincedsneddon: sure, that would be cool man19:38
EmilienMdprince: maybe we should make the switch and maintain newton nic-configs in tripleo-ci19:38
EmilienMso we can move forward19:38
dprinceEmilienM: works for me19:38
EmilienMI'll check with bnemec-pto when he's back19:38
dsneddonI'll write up the docs now so I don't forget19:38
EmilienMdsneddon++19:39
EmilienMslagle: any thoughts on making tripleo-ci-centos-7-3nodes-multinode voting?19:41
EmilienMalso maybe rename to have -oooq in the name19:41
EmilienMslagle: the job is fast and stable19:41
openstackgerritEmilien Macchi proposed openstack/paunch master:  Migrate instack-undercloud to zuul v3 jobs  https://review.openstack.org/52117719:45
*** dprince has quit IRC19:48
openstackgerritEmilien Macchi proposed openstack/tripleo-validations master: Migrate instack-undercloud to zuul v3 jobs  https://review.openstack.org/52117919:48
*** dtrainor has quit IRC19:48
*** chlong__ has joined #tripleo19:49
*** itlinux has quit IRC19:50
EmilienMcan someone approve https://review.openstack.org/#/c/520689/ please19:50
*** leitan has quit IRC19:51
*** trown|lunch is now known as trown19:51
*** vfarias has joined #tripleo19:52
*** fragatina has joined #tripleo19:54
*** itlinux has joined #tripleo19:54
*** haint has joined #tripleo19:55
*** holser has joined #tripleo19:56
openstackgerritMerged openstack/instack-undercloud master: Migrate instack-undercloud to zuul v3 jobs  https://review.openstack.org/52026419:56
openstackgerritMerged openstack/tripleo-common master: Migrate tripleo-common to zuul v3 jobs  https://review.openstack.org/52023819:56
openstackgerritMerged openstack/python-tripleoclient master: Migrate python-tripleoclient to zuul v3 jobs  https://review.openstack.org/52024119:56
*** cylopez has quit IRC19:57
*** dtrainor has joined #tripleo19:57
*** nguyentrihai has quit IRC19:58
slagleEmilienM: maybe not a good idea right now, due to resource concerns raised by infra at the past ptg19:59
EmilienMmwhahaha: please do not backport these patches ^19:59
mwhahahaEmilienM: hadn't planned on it19:59
EmilienMslagle: i'll ask them again, last time I checked, resources were fine19:59
slagleEmilienM: until we're well past that i don't think i would add new voting jobs, especially 3 nodes19:59
slagleEmilienM: ok19:59
EmilienMmwhahaha: good. I'll take care of that this weekend.19:59
mwhahahawell i'd rather drop some of the other ones and use 3 node instead20:00
mwhahahasince that's more of a realistic deployment scenario20:00
mwhahahait runs better anyway20:00
EmilienMyou mean, consolidating services?20:00
EmilienMit's complicated to manage upgrades, from previous tags, etc...20:00
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: Add tls roles for undercloud  https://review.openstack.org/51707920:07
openstackgerritEmilien Macchi proposed openstack/os-apply-config master: Migrate instack-undercloud to zuul v3 jobs  https://review.openstack.org/52118120:08
openstackgerritEmilien Macchi proposed openstack/os-net-config master: Migrate instack-undercloud to zuul v3 jobs  https://review.openstack.org/52118220:09
mwhahahaEmilienM: did you miss the os-* projects?20:09
openstackgerritEmilien Macchi proposed openstack/os-collect-config master: Migrate instack-undercloud to zuul v3 jobs  https://review.openstack.org/52118320:09
mwhahahaEmilienM: also your patch commit message is wrong20:09
*** ooolpbot has joined #tripleo20:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION20:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432820:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)20:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173103220:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173106320:10
*** ooolpbot has quit IRC20:10
openstackLaunchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]20:10
openstackLaunchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged]20:10
openstackgerritEmilien Macchi proposed openstack/os-refresh-config master: Migrate instack-undercloud to zuul v3 jobs  https://review.openstack.org/52118420:10
EmilienMmwhahaha: damn20:10
openstackgerritEmilien Macchi proposed openstack/os-refresh-config master: Migrate os-refresh-config to zuul v3 jobs  https://review.openstack.org/52118420:11
*** rmascena has quit IRC20:11
openstackgerritEmilien Macchi proposed openstack/os-collect-config master: Migrate os-collect-config to zuul v3 jobs  https://review.openstack.org/52118320:11
openstackgerritEmilien Macchi proposed openstack/os-net-config master: Migrate os-net-config to zuul v3 jobs  https://review.openstack.org/52118220:11
openstackgerritEmilien Macchi proposed openstack/os-apply-config master: Migrate os-apply-config to zuul v3 jobs  https://review.openstack.org/52118120:12
openstackgerritEmilien Macchi proposed openstack/tripleo-validations master: Migrate tripleo-validations to zuul v3 jobs  https://review.openstack.org/52117920:12
openstackgerritEmilien Macchi proposed openstack/paunch master: Migrate paunch to zuul v3 jobs  https://review.openstack.org/52117720:13
openstackgerritEmilien Macchi proposed openstack/tripleo-image-elements master: Migrate t-i-e to zuul v3 jobs  https://review.openstack.org/52117520:13
openstackgerritEmilien Macchi proposed openstack/tripleo-puppet-elements master: Migrate t-p-e to zuul v3 jobs  https://review.openstack.org/52117420:13
*** gbarros has quit IRC20:15
*** bobh has quit IRC20:17
mwhahahaweshay, EmilienM: we don't have a bug for teh scenario003 failures yet do we?20:17
mwhahahacause it is different20:18
mwhahahaso i'll create one for those20:18
EmilienMmwhahaha: not afik20:18
*** rfolco has quit IRC20:18
*** thrash|mtg is now known as thrash|biab20:22
*** eck` is now known as eck`gone20:27
*** ipsecguy_ has joined #tripleo20:27
weshaymwhahaha, it's failing on a different tempest test, but is it a different root cause?20:28
mwhahahamaybe?20:29
*** morazi has quit IRC20:29
mwhahahait seems like different reasons but not sure20:29
mwhahahai'm trying to take a look now20:29
*** ipsecguy has quit IRC20:30
weshaymwhahaha, if you see something different please let me know, I'd like to see it for my own neducation20:33
mwhahahai'm kinda concerned about the dockerd-current messages about something becoming an orphan20:33
mwhahahahttp://logs.openstack.org/36/521136/1/check/legacy-tripleo-ci-centos-7-scenario003-multinode-oooq-container/a6708d9/logs/subnode-2/var/log/journal.txt.gz#_Nov_17_18_54_1720:33
*** cylopez has joined #tripleo20:34
mwhahahaand ofcourse docker is not set to debug on the subnode20:35
weshaymwhahaha, hrm..20:35
weshayhttp://logs.openstack.org/36/521136/1/check/legacy-tripleo-ci-centos-7-scenario003-multinode-oooq-container/a6708d9/logs/subnode-2/var/log/extra/errors.txt.gz20:35
mwhahahaweshay: do we do any docker log capture stuff? or status of docker ?20:36
weshayno time stamps but look for  NotConnectedError: Cannot commit transaction20:36
weshaymwhahaha, ya.. sec20:36
mwhahahaso tempest didn't start running until 18:5620:36
mwhahaha18:56:34 to be specific, so anything prior to that is probably not something valid20:37
mwhahahahowever teh ovs connection stuff might be problemmattic20:37
mwhahahawhich is what is saw ealier20:37
mwhahahacause that is right when tempest is running20:37
mwhahahaoh silly glareclient20:38
mwhahahabut not related to this20:39
mwhahahai think the scenario003 problems are related to the ovs failures20:39
mwhahahalet me see if those show up on the scenario001 failures20:39
weshayTimeout in RPC method get_active_networks_info. Waiting for 28 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.: MessagingTimeout: Timed out waiting for a reply to message ID f40afd12b16c4c6abfa0ea604cb6ff6120:40
mwhahahaweshay: are you talking about http://logs.openstack.org/36/521136/1/check/legacy-tripleo-ci-centos-7-scenario003-multinode-oooq-container/a6708d9/logs/subnode-2/var/log/containers/neutron/neutron-metadata-agent.log.txt.gz#_2017-11-17_18_54_58_56520:40
mwhahaha?20:40
mwhahahathat was while the deployment was occuring20:40
mwhahahaprior to tempest20:40
mwhahahahttp://logs.openstack.org/36/521136/1/check/legacy-tripleo-ci-centos-7-scenario003-multinode-oooq-container/a6708d9/logs/subnode-2/var/log/containers/neutron/neutron-l3-agent.log.txt.gz#_2017-11-17_18_57_44_52320:41
weshayI see it after 18:5620:41
mwhahahathat however, is different20:41
mwhahahawhere?20:41
*** fultonj has quit IRC20:41
*** dprince has joined #tripleo20:41
weshaybah.. sorry the errors.txt has the is scrambled a bit20:43
mwhahahayea20:43
*** itlinux has quit IRC20:43
mwhahahait's better to go look at the source20:43
mwhahahai think the scenario003 problem is the ovs errors20:43
mwhahahathat being said the scenario001 logs that i have were all openvswitch-2.7.220:45
mwhahahaso i wonder if we have a new problem and 001 is fine now20:45
mwhahahaneed to find a more recent 001 failure20:45
weshayk.. /me gets20:46
weshayrlandy|rover, ^20:46
weshayhere is the lasted scen001 failure on tempest http://logs.openstack.org/81/520581/1/check/legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-container/128fccc/20:47
mwhahahaok that one is openvswitch-2.8.1-2.1fc28.el7.x86_6420:47
mwhahahahttp://logs.openstack.org/81/520581/1/check/legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-container/128fccc/logs/subnode-2/var/log/containers/neutron/neutron-l3-agent.log.txt.gz#_2017-11-16_15_11_32_81020:48
mwhahahaalso issues talking to ovs20:48
mwhahahahttp://logs.openstack.org/81/520581/1/check/legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-container/128fccc/logs/subnode-2/var/log/containers/neutron/neutron-metadata-agent.log.txt.gz#_2017-11-16_15_21_24_094 metadata service was working20:48
mwhahahasee there's always that messagingtimeout because i think it gets started prior to the neutron service20:48
mwhahahai don't think we have good ordering on container startup, but i'm not sure20:49
mwhahahawe used to with puppet because you could tell the agents to wait, but i'm not sure we have a similar concept with containers anymore20:49
mwhahahadprince: do container starts have ordering associated with them still? ie agent waits until server is started20:50
dprincemwhahaha: there are 2 types of ordering20:50
dprincemwhahaha: the "stepwise" ording of the higher deployment workflow (1-5), where 1 is for the LB, 2 database/rabbit, etc.20:51
dprincemwhahaha: then there is a start_order which can be used to control container startup within each step20:51
mwhahahaok so there's still that concept, cool20:52
dprincemwhahaha: start order is applied via paunch20:52
mwhahahai'll have to remember start_order20:52
dprincemwhahaha: yep, we already rely on it for a few things20:52
mwhahahamight need to check if the neutron agents containers are properly waiting for neutron server20:52
mwhahahaweshay: http://logs.openstack.org/79/516979/2/gate/legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-container/13841da/logs/subnode-2/var/log/containers/neutron/neutron-l3-agent.log.txt.gz#_2017-11-08_21_13_17_387 looks like the ovs backend error was there with the older ovs package as well20:53
dprincemwhahaha: had a nice chat with stevebaker yesterday about some potential improvements to paunch to help us catch (fail-fast) containers in restart, and or unhealthy20:53
mwhahahadprince: are we tracking when docker containers are crashing? I was wondering if this error was a crashing container http://logs.openstack.org/36/521136/1/check/legacy-tripleo-ci-centos-7-scenario003-multinode-oooq-container/a6708d9/logs/subnode-2/var/log/journal.txt.gz#_Nov_17_18_54_1720:54
mwhahahacontainerd: 6c5eb757d223c6711a2ab61dd25e0023bda57dff8ca275154ad00a1372158a7c:e80037679a625f34dfa3e10ae1adb8111fc3dc0fcf2ff21c28748645d9142567 (pid 74940) has become an orphan, killing it20:54
dprincemwhahaha: no, we start it and go20:54
dprincemwhahaha: having paunch detect the failure with a 'status' command or something and then wiring that into the higher level workflow between each step would be useful here20:55
dprincemwhahaha: we aren't catching this at this point20:55
mwhahahaok sounds like something we'd probably like to wire into CI as well to see if containers aren't fully OK at teh end of the run20:56
*** jmelvin has quit IRC20:58
*** holser has quit IRC21:00
*** vfarias has quit IRC21:01
*** holser has joined #tripleo21:01
weshaymwhahaha, are these errors just startup issues as http://logs.openstack.org/79/516979/2/gate/legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-container/13841da/logs/subnode-2/var/log/containers/neutron/neutron-l3-agent.log.txt.gz#_2017-11-08_21_13_30_98521:01
weshayshows the deivce eventually comes up21:02
*** ianw has quit IRC21:02
mwhahahaweshay: i don't think so21:02
mwhahahabut i don't know, i'm not neutron guy21:02
mwhahahabecause it started 5 mins before21:02
mwhahahaso the only think i could think of is that it's some perf related issue since it's so flaky21:02
weshayjust sayin.. I see21:02
weshay record=qg-8d5f417f-66). Not connected21:02
weshayand then finally21:03
weshayDevice qg-8d5f417f-66 already exists21:03
*** Hazelesque_ is now known as Hazelesque21:03
weshaymwhahaha, EmilienM rlandy|rover are we going to move 003 to non-voting?21:04
mwhahahaof course i bet we don't have l3 agent debugging enabled21:04
* mwhahaha sighs and prepares a patch21:04
weshayheh21:04
* mwhahaha throws things at beagles 21:04
weshaymwhahaha, it's a tripleo bug man.. it's ALL YOUR FAULT21:05
mwhahahait was like this when i got here!21:05
weshaylolz21:05
*** dparkes has joined #tripleo21:06
* mwhahaha debugs all the things21:07
weshaydprince, think we're back to green21:07
dprinceweshay: boom :)21:08
*** ianw has joined #tripleo21:09
*** ooolpbot has joined #tripleo21:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION21:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432821:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)21:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173103221:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173106321:10
*** ooolpbot has quit IRC21:10
openstackLaunchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]21:10
openstackLaunchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged]21:10
*** holser has quit IRC21:13
*** eck`gone is now known as eck`21:14
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: Add debug to neutron agents  https://review.openstack.org/52119121:14
mwhahahadebug all the things -^21:15
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: DNM: Testing debug neutron metadata  https://review.openstack.org/51975621:15
mwhahahaweshay: so to answer your question, yea we may have to disable voting on scenario003 if this continues21:16
weshayrlandy|rover, ^21:16
rlandy|roverweshay: ack - with containers and w/o - both are unstable21:17
weshaymwhahaha, doesn't look like a promotion would help either21:17
mwhahahano it's not something that anyone seems to be aware of21:18
weshayalthough21:18
weshayscen003 only started acting flaky in the promotion jobs on the 16th21:18
weshaythat just may be luck though21:18
mwhahahawell it might have been flaky before that but we had told people to not aprove anything last week :D21:19
mwhahahaand 001 was far more noticable21:19
weshayhttp://38.145.34.234/21:19
weshaymaster21:19
mwhahahaweshay: btw with our reduction of log that we collect, it seems that collect-logs is down to <3 mins21:19
weshayfs16 and fs1821:19
weshaymwhahaha, k.. good.. I doc'd the diff I saw in the bug21:20
mwhahahai've got a patch that might drop it another 30secs21:20
* weshay looks21:20
mwhahahahttp://logs.openstack.org/46/521146/1/check/legacy-tripleo-ci-centos-7-scenario002-multinode-oooq-puppet/73977e4/logs/quickstart_collect_logs.txt.gz21:20
mwhahaha2:3221:21
mwhahahahttps://review.openstack.org/#/c/521146/ the openstack command is unfortunately really heavy on startup21:21
mwhahahaalso https://review.openstack.org/#/c/521136/ will help as well21:21
weshayEOC, cool21:22
mwhahahawell EOF was already taken so i figured EOC for end of command :D21:22
*** dtrainor has quit IRC21:23
*** rlandy|rover has quit IRC21:24
weshaythis is how we know nothing merges into elastic-recheck http://status.openstack.org/elastic-recheck/21:25
weshaythe bug w/ the highest fail rate has 1 fail in 24hr21:25
mwhahahadat 43 hour indexing delay21:25
openstackgerritAlan Bishop proposed openstack/puppet-tripleo master: Fix use of deprecated "api_class" key manager option  https://review.openstack.org/52119521:25
openstackgerritDan Sneddon proposed openstack/tripleo-docs master: Update docs to reflect updates to os-net-config mechanism  https://review.openstack.org/52119721:30
openstackgerritAlex Schultz proposed openstack/tripleo-quickstart-extras master: Use openstack commands for postci  https://review.openstack.org/52114621:32
*** thrash|biab is now known as thrash21:39
*** pradk has quit IRC21:39
*** myoung is now known as myoung|bbl21:42
thrashmwhahaha: made a suggestion on those commands... wdyt?21:43
mwhahahathrash: that might be an improvement21:44
thrashmwhahaha: just a thought... I know I use it a lot when I'm just working in general.21:44
mwhahahaunfortunately i'm not sure every one of those commands supports it21:45
mwhahahabut we can give it a shot21:45
mwhahahacause the formatter is on a per client basis21:45
mwhahahaso the mistral one might, but nova might not21:45
thrashmwhahaha: I thought with osc the formatter was in osc, not the clients.21:45
mwhahahait's not returning when you do openstack --help21:46
mwhahahamaybe though21:46
thrashhmmm21:46
mwhahahai think we can just append it to the command21:46
mwhahahai'll give it a shot when my undercloud finishes building21:46
*** florianf has quit IRC21:48
mwhahahayea you can just append it21:48
openstackgerritAlex Schultz proposed openstack/tripleo-quickstart-extras master: Use openstack commands for postci  https://review.openstack.org/52114621:49
*** trown is now known as trown|outtypewww21:51
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: Add debug to neutron agents  https://review.openstack.org/52119121:55
*** jtomasek has quit IRC22:02
*** artom has quit IRC22:03
*** fragatina has quit IRC22:03
*** Goneri has joined #tripleo22:04
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: Fix the undercloud-containers job  https://review.openstack.org/51944122:04
*** fragatina has joined #tripleo22:05
*** abishop has quit IRC22:09
*** ooolpbot has joined #tripleo22:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION22:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432822:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173103222:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173106322:10
*** ooolpbot has quit IRC22:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)22:10
openstackLaunchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]22:10
openstackLaunchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged]22:10
openstackgerritAlex Schultz proposed openstack-infra/tripleo-ci master: Move scenario003 to non-voting  https://review.openstack.org/52120522:16
mwhahahaweshay, EmilienM -^ fyi22:16
* mwhahaha sends a note to the ML22:17
*** fultonj has joined #tripleo22:21
zanebmwhahaha: hopefully this will do the trick on bug 1731032: https://review.openstack.org/#/c/521204/22:22
openstackbug 1731032 in OpenStack Heat "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,In progress] https://launchpad.net/bugs/1731032 - Assigned to Zane Bitter (zaneb)22:22
mwhahahazaneb: cool thanks22:23
*** Goneri has quit IRC22:24
*** gbarros has joined #tripleo22:30
*** dprince has quit IRC22:33
*** fultonj has quit IRC22:34
*** Guest78 has joined #tripleo22:35
*** cdearborn has quit IRC22:39
*** Guest78 has quit IRC22:42
openstackgerritmathieu bultel proposed openstack/tripleo-common master: Stream the Ansible output to a Zaqar queue  https://review.openstack.org/52105822:48
*** lblanchard has joined #tripleo22:56
*** akrivoka has quit IRC22:58
*** chlong__ has quit IRC23:00
* mwhahaha wonders if the top change in the queue is just going to time out and ruin everything23:04
*** ooolpbot has joined #tripleo23:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION23:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/172432823:10
openstackLaunchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)23:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173103223:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/173106323:10
*** ooolpbot has quit IRC23:10
openstackLaunchpad bug 1731032 in OpenStack Heat "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,In progress] - Assigned to Zane Bitter (zaneb)23:10
openstackLaunchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged]23:10
*** gbarros has quit IRC23:53
*** pcaruana has quit IRC23:54

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!