Tuesday, 2016-03-01

*** dcain has joined #tripleo00:11
*** akuznetsov has joined #tripleo00:11
*** tiswanso has joined #tripleo00:14
*** dmacpher-afk has quit IRC00:16
*** rhallisey has joined #tripleo00:16
*** akuznetsov has quit IRC00:16
*** dcain1 has joined #tripleo00:17
*** dcain has quit IRC00:20
*** trozet has quit IRC00:26
*** tiswanso has quit IRC00:28
*** dshulyak has joined #tripleo00:28
*** tiswanso has joined #tripleo00:28
*** dshulyak has quit IRC00:33
*** saneax is now known as saneax_AFK00:40
*** dmacpher has joined #tripleo01:03
*** lblanchard has joined #tripleo01:07
*** penick has quit IRC01:08
jidarshardy: any chance there is work being done to support OS::Heat::SoftwareDeployments with a servers param that takes (or tripleo providing) a Controllers/Computes option? I've alluded to this before but man is it a goofy problem to get around01:09
jidarI really don't like having to do `if [[ $HOSTNAME =~ .*controller.* ]]; then` in my scripts01:09
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo: IPv6: duak-stack support for Keystone  https://review.openstack.org/28634401:12
slagleEmilienM: fyi, looks like almost all (if not all) tripleo ha jobs are failing due to what i suspect is this: https://review.openstack.org/#/c/284876/01:31
slaglethat passed tripleo ci, but we only run the nonha job there01:31
EmilienMslagle: well, TripleO CI passed it01:31
slagleyea, the nonha job01:32
slagleEmilienM: here's how the ha jobs are failing: http://logs.openstack.org/23/235423/44/check-tripleo/gate-tripleo-ci-f22-ha/7465efa/console.html01:32
slaglesearch for keystone-manage bootstrap01:32
EmilienMslagle: let me look01:33
EmilienMthe exec failed, probably because orchestration01:33
EmilienMlet me dig a bit01:33
EmilienMoh but it's undercloud?01:34
EmilienMslagle: I have a patch to THT01:35
slagleEmilienM: feel free to take the bug i just reported, https://bugs.launchpad.net/tripleo/+bug/155150101:36
openstackLaunchpad bug 1551501 in tripleo "CI: HA jobs failing with Error: /Stage[main]/Keystone/Exec[keystone-manage bootstrap]: Failed to call refresh: keystone-manage bootstrap --bootstrap-password <password> returned 1 instead of one of [0]" [Critical,Triaged]01:36
EmilienMagain, this kind of bug will be fixed when we'll use our new pacemaker provider to manage services01:36
EmilienMslagle: https://github.com/openstack/puppet-keystone/blob/master/manifests/init.pp#L98901:36
EmilienMwe need to set that for pacemaker in THT01:36
EmilienMI'm sending a patch now01:36
slagleyea, we only want to run that one time on the bootstrap node i'd guess?01:37
EmilienMI'm taking the bug01:37
EmilienMactually no that's not this dependency01:40
*** trozet has joined #tripleo01:41
EmilienMslagle: do you have the output of the Exec?01:42
EmilienMI suspect Galera is not ready01:42
slagleEmilienM: look at the log i linked, that's all i have01:42
*** dcain1 has quit IRC01:42
slaglethat was all the stdout from puppet01:42
EmilienMgalera should be ready at step 2 and the error is at step 301:42
*** dcain has joined #tripleo01:43
EmilienMoh I know01:44
EmilienMslagle: keystone resources are not managed by puppet-keystone on the overcloud01:44
EmilienMso the bootstrap tries to create them01:44
EmilienMbut can't I think01:45
EmilienMI have a temporary fix I guess01:45
EmilienMslagle: do you manage default domain, etc already?01:45
slagleis it required?01:45
slagleif it's required, then we must create it via os-cloud-config01:45
EmilienMsince a few commits in OpenStack Keystone, yes01:45
slagleoh, possible not then01:46
EmilienMsee 7b7fea7a3fe7677981fbf9bac5121bc1560116301:46
EmilienMslagle: https://github.com/openstack/keystone/commit/f699ca93fc6f2485ec8e76e907572a2f838cd3cb01:47
slaglethe repo we're using is from 2016-02-23, so we likely don't have that01:49
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: controller/ha: disable keystone-manage bootstrap.  https://review.openstack.org/28635201:50
EmilienMslagle: ^ let's see01:50
EmilienMit should fix the CI01:50
*** rhallisey has quit IRC02:04
*** trozet has quit IRC02:08
*** ooolpbot has joined #tripleo02:10
*** ooolpbot has quit IRC02:10
openstackLaunchpad bug 1551501 in tripleo "CI: HA jobs failing with Error: /Stage[main]/Keystone/Exec[keystone-manage bootstrap]: Failed to call refresh: keystone-manage bootstrap --bootstrap-password <password> returned 1 instead of one of [0]" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)02:10
EmilienMyes I know02:10
slaglewould you like me to remove the alert?02:11
EmilienMno please keep it02:11
slagleit's just the "alert" tag02:11
EmilienMit just send notifis to my phone when someone ping me02:11
EmilienMahah I should drop that02:11
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo: IPv6: duak-stack support for Keystone  https://review.openstack.org/28634402:15
*** Marga__ has joined #tripleo02:16
*** akuznetsov has joined #tripleo02:17
*** Marga_ has quit IRC02:19
*** akuznetsov has quit IRC02:21
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates: Make all network ports type to consume FixedIPs  https://review.openstack.org/28636002:40
*** ooolpbot has joined #tripleo03:10
*** ooolpbot has quit IRC03:10
openstackLaunchpad bug 1551501 in tripleo "CI: HA jobs failing with Error: /Stage[main]/Keystone/Exec[keystone-manage bootstrap]: Failed to call refresh: keystone-manage bootstrap --bootstrap-password <password> returned 1 instead of one of [0]" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)03:10
*** yuanying has joined #tripleo03:21
*** yuanying_ has quit IRC03:23
*** xinwu has quit IRC03:23
*** Marga__ has quit IRC03:26
*** yuanying has quit IRC03:27
*** Marga_ has joined #tripleo03:30
*** Nakato has quit IRC03:31
*** Marga_ has quit IRC03:35
*** lblanchard has quit IRC03:38
*** panda has quit IRC03:38
*** panda has joined #tripleo03:38
mandreI get an error installing my undercloud with enable_mistral=true because of missing python-mistralclient dependency: http://paste.openstack.org/show/488683/03:39
mandrewhere is the best place to add the missing dependency?03:39
*** Marga_ has joined #tripleo03:43
*** Nakato has joined #tripleo03:45
*** Marga_ has quit IRC03:47
*** rlandy has quit IRC03:48
*** links has joined #tripleo03:49
*** xinwu has joined #tripleo03:51
*** chlong_ has quit IRC04:03
*** saneax_AFK is now known as saneax04:06
*** yuanying has joined #tripleo04:10
*** ooolpbot has joined #tripleo04:10
*** ooolpbot has quit IRC04:10
openstackLaunchpad bug 1551501 in tripleo "CI: HA jobs failing with Error: /Stage[main]/Keystone/Exec[keystone-manage bootstrap]: Failed to call refresh: keystone-manage bootstrap --bootstrap-password <password> returned 1 instead of one of [0]" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)04:10
*** yamahata has joined #tripleo04:16
*** Marga_ has joined #tripleo04:16
*** Marga_ has quit IRC04:16
*** Marga_ has joined #tripleo04:17
*** anande has joined #tripleo04:35
*** tiswanso has quit IRC04:47
*** anande has quit IRC05:02
*** ooolpbot has joined #tripleo05:10
*** ooolpbot has quit IRC05:10
openstackLaunchpad bug 1551501 in tripleo "CI: HA jobs failing with Error: /Stage[main]/Keystone/Exec[keystone-manage bootstrap]: Failed to call refresh: keystone-manage bootstrap --bootstrap-password <password> returned 1 instead of one of [0]" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)05:10
*** dcain has quit IRC05:15
*** Marga_ has quit IRC05:20
*** masco has joined #tripleo05:23
*** dmacpher has quit IRC05:24
*** Marga_ has joined #tripleo05:30
*** jaosorior has joined #tripleo05:49
*** dmacpher has joined #tripleo06:01
*** lucasagomes has quit IRC06:02
*** lucasagomes has joined #tripleo06:02
*** dmacpher_ has joined #tripleo06:08
*** rcernin has joined #tripleo06:09
*** ooolpbot has joined #tripleo06:10
*** ooolpbot has quit IRC06:10
openstackLaunchpad bug 1551501 in tripleo "CI: HA jobs failing with Error: /Stage[main]/Keystone/Exec[keystone-manage bootstrap]: Failed to call refresh: keystone-manage bootstrap --bootstrap-password <password> returned 1 instead of one of [0]" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)06:10
*** dmacpher has quit IRC06:10
*** dmacpher_ has quit IRC06:16
*** dmacpher_ has joined #tripleo06:23
*** lucasagomes has quit IRC06:24
*** lucasagomes has joined #tripleo06:27
*** thrash|g0ne has quit IRC06:33
*** trown|outtypewww has quit IRC06:34
*** trown has joined #tripleo06:35
*** tzumainn has quit IRC06:40
*** akuznetsov has joined #tripleo06:45
*** akuznetsov has quit IRC06:49
*** ishant has joined #tripleo06:54
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/tripleo-heat-templates: Fix typos  https://review.openstack.org/26512606:55
*** dmacpher_ has quit IRC06:56
*** Marga_ has quit IRC06:57
*** dshulyak has joined #tripleo07:02
*** dmacpher has joined #tripleo07:04
*** jprovazn has joined #tripleo07:05
*** ooolpbot has joined #tripleo07:10
*** ooolpbot has quit IRC07:10
openstackLaunchpad bug 1551501 in tripleo "CI: HA jobs failing with Error: /Stage[main]/Keystone/Exec[keystone-manage bootstrap]: Failed to call refresh: keystone-manage bootstrap --bootstrap-password <password> returned 1 instead of one of [0]" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)07:10
*** liverpooler has joined #tripleo07:13
*** mikelk has joined #tripleo07:16
*** dmacpher has quit IRC07:20
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo: Always override X-Forwarded-Proto header for Heat  https://review.openstack.org/28642007:36
*** bnemec has quit IRC07:36
*** bnemec has joined #tripleo07:39
*** hjensas has joined #tripleo07:48
*** hjensas has quit IRC07:48
*** hjensas has joined #tripleo07:48
openstackgerritMarios Andreou proposed openstack/tripleo-common: Adds a bash script that operators will use during upgrades  https://review.openstack.org/28642607:50
*** jtomasek has joined #tripleo07:50
*** fgimenez has joined #tripleo08:00
*** fgimenez has quit IRC08:00
*** fgimenez has joined #tripleo08:00
*** slagle has quit IRC08:06
*** shadower has quit IRC08:06
*** shadower has joined #tripleo08:07
*** devvesa has joined #tripleo08:08
*** pcaruana has joined #tripleo08:09
*** ooolpbot has joined #tripleo08:10
*** ooolpbot has quit IRC08:10
openstackLaunchpad bug 1551501 in tripleo "CI: HA jobs failing with Error: /Stage[main]/Keystone/Exec[keystone-manage bootstrap]: Failed to call refresh: keystone-manage bootstrap --bootstrap-password <password> returned 1 instead of one of [0]" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)08:10
*** yamahata has quit IRC08:11
*** paramite has joined #tripleo08:18
*** ccamacho has joined #tripleo08:20
*** rdopiera has joined #tripleo08:21
*** aufi has joined #tripleo08:21
*** sshnaidm has quit IRC08:22
*** mcornea has joined #tripleo08:23
*** ifarkas has joined #tripleo08:30
openstackgerritMerged openstack/tripleo-heat-templates: controller/ha: disable keystone-manage bootstrap.  https://review.openstack.org/28635208:30
*** paramite is now known as paramite|afk08:32
*** ohamada has joined #tripleo08:35
openstackgerritMichael Chapman proposed openstack/tripleo-specs: Refactor top level puppet manifests  https://review.openstack.org/28643908:37
*** paramite|afk is now known as paramite08:37
*** xinwu has quit IRC08:45
*** openstackgerrit has quit IRC08:48
*** openstackgerrit has joined #tripleo08:48
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates: Move ordering constraints from httpd to openstack-core  https://review.openstack.org/28644608:48
*** hewbrocca has joined #tripleo08:54
*** paramite is now known as paramite|afk08:55
*** paramite|afk is now known as paramite08:57
*** gfidente has joined #tripleo08:58
*** athomas has joined #tripleo09:01
*** mbound has joined #tripleo09:06
*** jcoufal has joined #tripleo09:10
*** jistr has joined #tripleo09:14
*** sshnaidm has joined #tripleo09:18
*** akrivoka has joined #tripleo09:19
*** olap has quit IRC09:29
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Collect status of all nested stacks in resource-list and event-list  https://review.openstack.org/28606209:29
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Roll up static Heat envs into CI directory  https://review.openstack.org/28043109:29
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: DO NOT MERGE: Print output of brctl show from hosting node  https://review.openstack.org/28627909:29
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Use netiso in the ha job  https://review.openstack.org/27342409:29
openstackgerritAthlan-Guyot sofer proposed openstack/puppet-pacemaker: Basic beaker one node test.  https://review.openstack.org/28137609:31
hewbroccagfidente: good morning!09:32
hewbroccagfidente: I see you are working on the CI...09:32
*** dtantsur|afk is now known as dtantsur09:33
gfidentehewbrocca, morning, yes but it seems we're filtering vlan traffic in upstream env09:34
gfidentehewbrocca, I'm still trying to figure why that is09:34
gfidentehewbrocca, bridges on undercloud and overcloud nodes are good so we're down to the one on the baremetal host09:35
hewbroccaOK, cool09:36
*** derekh has joined #tripleo09:39
gfidentederekh, !09:39
openstackgerritMerged openstack/tripleo-heat-templates: Make all network ports type to consume FixedIPs  https://review.openstack.org/28636009:39
derekhgfidente: howdy09:40
* gfidente needs help on something09:40
gfidentewe have vsctl in hostinfo for undercloud and overcloud and those look okay09:40
gfidenteso I was looking into baremetal and tried https://review.openstack.org/#/c/286279/09:41
gfidentebut neither brctl nor ovs-vsctl is available on HOST_IP ?09:41
gfidentedid I ssh on the right host?09:41
*** mkovacik has joined #tripleo09:42
derekhgfidente: that brctl command is being run on the jenkins node, try it further down in the here doc09:42
gfidenteHOST_IP is the jenkins node?09:43
derekhgfidente: opps sorry I missed that, HOST_IP is the testenv host09:44
gfidenteyeah so I thought one of brctl or ovs-vsctl would work but none did?09:44
mariosgfidente: o/ morning, i made a comment on the netiso review as promised earlier this morning09:45
mariosgfidente: oh you updated https://review.openstack.org/#/c/273424/09:46
derekhgfidente: just ran it manually now on the test env host http://paste.openstack.org/show/488723/09:46
*** mgould has joined #tripleo09:47
gfidentemarios, thanks for that :)09:47
derekhgfidente: brctl isn't installed but ovs-vsctl is09:47
gfidentederekh, in patchset 1 was trying ovs-vsctlfwiw09:47
openstackgerritMerged openstack/tripleo-common: Adds a bash script that operators will use during upgrades  https://review.openstack.org/28642609:48
derekhgfidente: ahhh, ssh is locked down to only allow specific commands, see RE_COMMANDS http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/elements/testenv-worker/bin/ci_commands#n6509:50
gfidentederekh, ah okay that explains it thanks!09:50
gfidentederekh, so this https://jenkins07.openstack.org/job/gate-tripleo-ci-f22-ha/581/console is using ctlplane for corosync09:51
derekhgfidente: ack, lets see what happens09:52
gfidentederekh, cause ctlplane is untagged, I think the tagged traffic is filtered09:52
gfidenteso I expect it to fail anyway, but not on corosync, just to confirm the thesis09:53
hewbroccagfidente, derekh if you guys need any help don't hesitate to reach out10:00
hewbroccathis is our top priority issue right now10:00
derekhhewbrocca: thanks, will do10:01
mariosgerrit really slow or just here?10:19
marios(wasn't too bad earlier)10:20
*** olap has joined #tripleo10:21
shardyalso really slow for me :(10:27
*** ishant has quit IRC10:28
*** lucasagomes has quit IRC10:30
mariosshardy: thanks10:31
*** lucasagomes has joined #tripleo10:31
*** lucasagomes has quit IRC10:32
*** lucasagomes has joined #tripleo10:32
dtantsurslow for everyone, I've reported to #infra already (it's not that they didn't notice..)10:33
*** lucasagomes has quit IRC10:37
*** lucasagomes has joined #tripleo10:37
-openstackstatus- NOTICE: Gerrit is going to be restarted due to poor performance10:39
*** ChanServ changes topic to "Gerrit is going to be restarted due to poor performance"10:39
*** lucasagomes has quit IRC10:41
*** lucasagomes has joined #tripleo10:43
*** ChanServ changes topic to "TripleO | CI status: http://tripleo.org/cistatus.html | Docs: http://tripleo.org/"10:45
-openstackstatus- NOTICE: gerrit finished restartign10:45
*** tosky has joined #tripleo10:48
shardyderekh: are you OK with https://review.openstack.org/#/c/275668 landing now?  lgtm10:55
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates: Move ordering constraints from httpd to openstack-core  https://review.openstack.org/28644610:58
derekhshardy: yup, go for it10:59
shardyderekh: thanks, done11:00
*** chlong_ has joined #tripleo11:01
derekhshardy: the other patch on the series will also need to be approved https://review.openstack.org/#/c/275667/811:01
openstackgerritMerged openstack-infra/tripleo-ci: Source undercloud environment variable from a file  https://review.openstack.org/27566711:05
openstackgerritMerged openstack-infra/tripleo-ci: Split the deploy script into its own file  https://review.openstack.org/27566811:06
openstackgerritMerged openstack/instack-undercloud: mistral: deploy API in WSGI with apache  https://review.openstack.org/28071911:06
derekhshardy: thanks11:07
shardyderekh: Have a question re https://review.openstack.org/#/c/271416/, the Depends-On looks bogus?11:09
derekhshardy: yup, I'll remove it now, i had forgotten about it, it was there so that delorean would build something so I could see if the logs were grabbed11:14
shardyaha, cool11:15
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Archive all of the delorean logs  https://review.openstack.org/27141611:16
derekhshardy: ^^11:16
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Roll up static Heat envs into CI directory  https://review.openstack.org/28043111:36
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Collect status of all nested stacks in resource-list and event-list  https://review.openstack.org/28606211:36
hewbroccagfidente: need any additional help with the net-iso tests?11:37
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Use netiso in the ha job  https://review.openstack.org/27342411:37
*** panda has quit IRC11:37
*** panda has joined #tripleo11:38
gfidentehewbrocca, I think we haven't figured the root issue yet11:41
gfidentehewbrocca, I was thinking to ping dsneddon if/when he is around11:42
gfidentehewbrocca, we're only dealing with the effects for now :(11:42
hewbroccagfidente: dsneddon won't be on for another 6 hours or so11:42
hewbroccahopefully dprince will have some ideas11:44
gfidenteyeah it's that we don't see this on local envs11:46
gfidenteso it's baiscally troubleshooting we're trying to do on the ci infra to see if we can spot the issue11:47
hewbroccaI can't tell from the gerrit page what the issue actually is?11:48
gfidentewell, we can see what goes wrong in the overcloud nodes, but they just don't talk to each other11:52
gfidenteat least not via the vlan tagged interfaces which we create in the overcloud nodes to have isolation11:52
hewbroccahmm is ping working11:53
hewbroccaand if not, are we sure the switch is set up correctly?11:53
gfidenteyeah we're trying just that now11:55
* hewbrocca backs away11:56
*** tosky has quit IRC11:57
*** Hazelesque_ is now known as Hazelesque12:03
openstackgerritImre Farkas proposed openstack/python-tripleoclient: [WIP] Update baremetal ready state command  https://review.openstack.org/28597712:16
jaosoriormarios: Thanks a lot for the review!12:21
jaosorioron the spec12:21
*** leanderthal|afk is now known as leanderthal12:21
mariosjaosorior: np, just a first pass12:21
*** mcornea has quit IRC12:21
*** mcornea has joined #tripleo12:22
*** saneax is now known as saneax_AFK12:22
*** rasca_ has quit IRC12:23
*** rasca has joined #tripleo12:24
*** tosky has joined #tripleo12:27
*** rasca has quit IRC12:28
*** rasca has joined #tripleo12:29
EmilienMtripleo reviewers, please look this backport: https://review.openstack.org/#/c/286031/12:29
jistrand these 2 upgrade fixups https://review.openstack.org/286155 https://review.openstack.org/286156 :)12:30
* jistr looking at the IPv6 pingtest12:30
derekhgfidente: the ci env I've started is ready to run the job,12:32
derekhgfidente: about to run this patch on it "Use netiso in the ha job"12:32
derekhgfidente: are there any others I need to include12:32
gfidentethe two it depends upon12:32
gfidenteor at least https://review.openstack.org/#/c/280431/12:33
*** anande has joined #tripleo12:33
derekhgfidente: ok its running12:36
derekhgfidente: ssh jenkins@
derekhgfidente: when its finished the testenv will stay around for a few hours12:36
*** jaosorior has quit IRC12:36
derekhgfidente: it running in screen tab 312:37
gfidentederekh, I just got on one of the overcloud nodes for the existing run12:37
*** jaosorior has joined #tripleo12:37
EmilienMjistr: that's a backport ... https://review.openstack.org/#/c/286031/12:37
EmilienMjistr: so I won't modify the code except for a very good reason.12:38
jistrEmilienM: ah right, thanks :) +212:39
EmilienMjistr: can you backport https://review.openstack.org/#/c/238097/ too please?12:39
jistrEmilienM: yup i'll do that right now12:39
*** lblanchard has joined #tripleo12:39
mariosjistr: EmilienM: wrt jistr question on https://review.openstack.org/#/c/286031/ see https://review.openstack.org/#/c/285858/1 (is a good point jistr)12:41
EmilienMmarios: so we need to backport https://review.openstack.org/#/c/285858/ ?12:41
mariosjistr: it seems the env it was tested in at the time had it already12:41
jistrmarios: haha you asked the same i see :) thanks for the link12:42
mariosEmilienM: i think so, are we seeing these validations fail here (like 'ipaddr not available?' if so we should)12:42
openstackgerritEmilien Macchi proposed openstack/tripleo-puppet-elements: Add overcloud-base element  https://review.openstack.org/28655312:43
EmilienMmarios, jistr ^12:43
mariosEmilienM: ack12:43
*** rhallisey has joined #tripleo12:43
EmilienMgfidente: can you also review https://github.com/puppetlabs/puppetlabs-inifile/pull/199 please ? It will help to have puppet-ceph working for ipv612:46
*** shardy has quit IRC12:47
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates: OpenContrail heat templates  https://review.openstack.org/28655912:48
EmilienMI also have 3 more review requests for bigswitch integration: https://review.openstack.org/#/c/280496/ (backport) https://review.openstack.org/#/c/274492 and https://review.openstack.org/#/c/279842/12:48
jistrEmilienM: contrail backport ^12:48
jistraha that's where ipaddr comes from :))12:49
*** rbrady has joined #tripleo12:58
*** lblanchard has quit IRC13:06
*** links has quit IRC13:07
*** rlandy has joined #tripleo13:10
*** julim has joined #tripleo13:10
gfidentederekh, so ping actually works13:11
gfidenteand yet corosync is logging those timeouts and never coming up13:11
derekhgfidente: so the problem isn't the net setup?13:11
gfidenteI suspect an corosync/pcmk guy can help here13:12
gfidentederekh, it doesn't look like13:12
gfidentebandini, can you help us?13:12
* gfidente having a quick lunch13:12
*** paramite is now known as paramite|afk13:13
*** pradk has joined #tripleo13:14
hewbroccagfidente: there's no ipv6 in here yet, is there?13:14
gfidenteactually yes13:15
gfidenteI mean, it's netiso on v4 but corosync is using an ipv6 socket13:15
*** lucasagomes is now known as lucas-hungry13:15
hewbroccathat might be your problem?13:15
bandinigfidente: sure, where, what, when?13:16
gfidentethe name resolves to an ipv4 address13:16
gfidentebandini, I think we need to wait for derekh too13:16
bandinigfidente: do we have some logs I can start looking at, to understand the issue?13:17
gfidentebandini, we're only seeing this in the upstream Ci but yes13:17
gfidentein there check the pcsd.log it looks like it's not getting to the other overcloud nodes13:18
gfidenteand, thanks :)13:18
gfidentebandini, fwiw, we haven't landed this yet https://review.openstack.org/#/c/267073/13:18
gfidenteso corosync is not using --ipv613:18
gfidenteand we're using ipv4 networks13:19
gfidentebut lsof is telling me corosync opened an ipv6 socket to get to the other nodes13:19
bandinioh I see, an ipv6-only socket or an ipv6-mapped-ipv4 one?13:19
* bandini downloads the logs13:19
*** aufi has quit IRC13:20
gfidenteruby    5555 root   11u  IPv6              94068       0t0      TCP overcloud-controller-0.localdomain:efi-mg->overcloud-controller-2.localdomain:4752413:21
*** jayg|g0n3 is now known as jayg13:23
*** dprince has joined #tripleo13:25
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: compute: include VIR_MIGRATE_TUNNELLED when doing VM shared storage  https://review.openstack.org/28658413:27
bandinigfidente: mmh weird because there seem to be many connections from controller-0 to controller-{1,2} that actually succeed (I cound 85 timed out and 416 successful ones)13:28
bandinigfidente: let me fetch the logs of the other nodes13:28
gfidentesome are IPv413:28
*** saneax_AFK is now known as saneax13:29
gfidentederekh, I wonder if it isn't worth trying with disable ipv6 sysctl?13:31
bandinigfidente: do we have access to these boxes?13:32
gfidentebandini, derek is setting up an env because this only seems to happen there13:33
gfidenteso what we saw was by logging on the nodes during deployment but then jobs go away when job completes13:33
derekhbandini: send me a key and I'll add it, the undercloud is still being installed13:33
bandiniah I see13:33
bandiniwill do13:33
*** slagle has joined #tripleo13:33
derekhthe env I'm setting up wont vanish afterwards13:33
bandiniperfect, thanks13:35
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: compute: include VIR_MIGRATE_TUNNELLED when doing VM shared storage  https://review.openstack.org/28658413:35
bandinigfidente: sorry I sent the key to you as well. I am drunk13:37
* hewbrocca checks watch13:37
* bandini whistles innocently13:38
* hewbrocca promotes bandini to manager13:38
*** paramite|afk is now known as paramite13:39
derekhbandini: ssh jenkins@
gfidentebandini, you don't need to tell you're drunk everybody13:40
derekhbandini: then ssh root@
derekhbandini:  thats the undercloud, overcloud images are now being build, so go get a coffee and sober up while you wait ;-)13:41
gfidentebuilding images13:41
bandiniderekh: ehehe oki ;)13:41
gfidentederekh, is it using the templates from /usr/share?13:42
gfidentederekh, I wanted to add the sysctl key13:42
derekhgfidente: yes it should be13:43
*** liverpooler has quit IRC13:43
*** jtomasek_ has joined #tripleo13:43
*** liverpooler has joined #tripleo13:43
bandinigfidente: what sysctl are you changing? just so I know13:45
gfidentebandini, yeah there is screen running on the first host13:46
gfidentecan you attach to it?13:46
gfidentederekh, ^^13:46
*** Marga_ has joined #tripleo13:46
gfidenteI'm on window 013:47
bandinion it13:47
derekhgfidente: sorry, im in13:49
gfidentederekh, fine see screen13:50
*** Marga_ has quit IRC13:50
*** shardy has joined #tripleo13:50
*** anande has quit IRC13:53
*** jtomasek_ has quit IRC13:54
openstackgerritBrad P. Crochet proposed openstack/tripleo-common: Add StackUpgradeManager  https://review.openstack.org/28660413:55
* leanderthal sips a bit of bandini 's special wine. offers some to hewbrocca . . .13:56
*** mbound has quit IRC13:56
derekhdprince: 2 of the periodic jobs passed last night for the first time, the ha one failed13:56
derekhdprince: Error: /Stage[main]/Keystone/Exec[keystone-manage bootstrap]: keystone-manage bootstrap --bootstrap-password Z9NgcwkR6rpNU2FRKaJuR7TAH returned 113:56
openstackgerritBrad P. Crochet proposed openstack/python-tripleoclient: Add 'stack upgrade' command  https://review.openstack.org/28660614:00
dprincederekh: any logs we can get from the failed bootstrap?14:00
slaglederekh: that should be fixed, we were hitting that in the check jobs as well14:00
dprincemeeting time14:00
openstackgerrityolanda.robla proposed openstack/diskimage-builder: Create new partitioning element.  https://review.openstack.org/25988114:00
derekhdprince: http://tripleo.org/cistatus-periodic.html14:00
openstackgerritRyan Brady proposed openstack/tripleo-specs: TripleO Deployment Library with Mistral  https://review.openstack.org/28040714:00
derekhdprince: http://logs.openstack.org/periodic/periodic-tripleo-ci-f22-ha/4492ced/14:00
gfidenteshardy, agreed on not doing -n5 for event-list :)14:01
slaglederekh: https://bugs.launchpad.net/tripleo/+bug/155150114:01
openstackLaunchpad bug 1551501 in tripleo "CI: HA jobs failing with Error: /Stage[main]/Keystone/Exec[keystone-manage bootstrap]: Failed to call refresh: keystone-manage bootstrap --bootstrap-password <password> returned 1 instead of one of [0]" [Critical,Fix released] - Assigned to Emilien Macchi (emilienm)14:01
shardygfidente: cool - that's actually a really expensive operation, so best avoided unless really needed :)14:01
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Collect status of all nested stacks in resource-list  https://review.openstack.org/28606214:02
*** derekh is now known as ndipanov_14:04
*** ndipanov_ is now known as derekh14:04
derekhslagle: ack, thanks, so it will hopfuly pass tonight14:06
openstackgerritMerged openstack/instack-undercloud: Unset DIB_YUM_REPO_CONF before running undercloud install  https://review.openstack.org/28444414:07
hewbroccaoh thank god ^^^14:08
*** lucas-hungry is now known as lucasagomes14:09
*** lblanchard has joined #tripleo14:14
*** tzumainn has joined #tripleo14:15
*** mbound has joined #tripleo14:15
dtantsurhey folks, daily request to review a small patch https://review.openstack.org/#/c/277974/ required for using python-hardware in profile matching :)14:17
*** paramite is now known as paramite|afk14:21
*** paramite|afk is now known as paramite14:22
*** eggmaster has joined #tripleo14:22
openstackgerritMerged openstack-infra/tripleo-ci: Convert the container job to a noop  https://review.openstack.org/28532514:22
*** tiswanso has joined #tripleo14:23
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: compute: include VIR_MIGRATE_TUNNELLED when doing VM shared storage  https://review.openstack.org/28658414:28
*** jroll has quit IRC14:29
*** jroll has joined #tripleo14:30
*** jroll has quit IRC14:30
*** jroll has joined #tripleo14:30
*** mbound has quit IRC14:30
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: compute: include VIR_MIGRATE_TUNNELLED when doing VM shared storage  https://review.openstack.org/28658414:31
*** shardy has quit IRC14:33
*** mbound has joined #tripleo14:34
*** shardy has joined #tripleo14:35
*** mgould has quit IRC14:35
*** sshnaidm has quit IRC14:35
*** weshay has joined #tripleo14:36
*** d0ugal has quit IRC14:39
*** d0ugal has joined #tripleo14:40
*** d0ugal has quit IRC14:40
*** trozet has joined #tripleo14:42
*** akuznetsov has joined #tripleo14:49
*** mgould has joined #tripleo14:50
*** akuznetsov has quit IRC14:54
hewbroccagfidente, derekh how's the CI looking? Time is slipping away :)14:59
gfidentehewbrocca, we're looking into it with bandini14:59
hewbroccatick tock :D15:00
*** shardy has quit IRC15:00
* bandini feels the manager pressure15:00
derekhgfidente: that env is going to dissapear at some stage as the ci command was killed15:01
dprincemichchap: hi, so we are (hopefully) getting ready to start work on composable services15:01
derekhgfidente: If it does, I kept the overcloud images so it wont take as long to setup again15:01
dprincemichchap: https://review.openstack.org/#/c/245804/15:01
bandiniderekh: how much time do we have?15:02
dprincemichchap: that work would move around quite a bit of the puppet code. Likely out of t-h-t and into puppet-tripleo15:02
derekhbandini: no way of knowing, whenever the env gets handed out to a new ci job15:02
michchapdprince: yep that's all I wanted to do initially. I'm on the opnfv team so our interest is mostly being able to easily add/modify the network profiles15:02
bnemecgfidente: Back to the net-iso job, have you looked at the most recent failure?15:03
bnemecTrying to ping default gateway\n10.0.0.1 is not pingable.15:03
bandiniderekh: it just did :)15:03
*** dustins has joined #tripleo15:03
bnemecWith the default single-nic-vlans setup the external network is on vlan 10, with a CIDR of
dprincemichchap: yep, and one of main benefits of composable services that teams don't have to deal as much with the larger architecture, just their services15:03
bnemecYou _have_ add that interface on the undercloud, unless we're going to do a completely custom network-isolation configuration.15:03
derekhgfidente: bandini: ok, restarting it, leave tab 3 running, thats what was keeping the env held15:04
dprincemichchap: there is a lot of debate on the spec ATM about how we run puppet though (all in one manifest vs. smaller snippets). I really would like to see the TripleO core team agree on these things before we do any of this work15:04
michchapdprince: I'll have a read through it and leave some feedback there, thanks.15:04
slagledprince: i'm in agreement now on running all one manifest15:05
gfidentederekh, which screen window shall we use?15:05
michchapdprince: smaller snippets leads to pain15:05
dprincemichchap: cool, thanks15:05
derekhgfidente: any one but 3, that were I had run the ./toci_gate_test.sh command that got killed15:06
slagledprince: it's just i'd like to see the concatenation avoided in the top level overcloud.yaml, but if we can't then we can't15:06
dprinceslagle: okay, perhaps it is time I made another pass at fixing some of the other comments.15:06
dprinceslagle: sure, I view that as sort of an implementation detail in that so long as it doesn't effect the "interfaces" (how services can plug into our deployment architecture) I think we could refactor it later15:06
*** yamahata has joined #tripleo15:07
*** Goneri has joined #tripleo15:10
gfidentederekh, is instack the same host?15:11
slaglebnemec: success, https://review.openstack.org/#/c/285509/15:11
gfidenteI wanted to edit the images15:11
slaglebnemec: i say we merge that and the previous one15:11
bnemecslagle: Agreed.15:12
dprincegfidente: where are you with debugging the netiso job?15:12
dprincegfidente,derekh: what can I help w/15:12
bnemecslagle: Do you want to do the honors on https://review.openstack.org/#/c/269827 since I proposed it?15:13
derekhgfidente: images are in /home/jenkins15:13
bnemecThere is no way the current netiso patch will ever pass CI.15:13
bnemecAre you guys working on a different version of it or something?15:13
slaglebnemec: done15:14
bnemecgfidente: derekh: ^15:14
derekhbnemec: this is the patch we're trying, https://review.openstack.org/#/c/273424/15:15
dprincebnemec: it could work if we used ctlplane for public (tenant) traffic I think15:15
bnemecdprince: Which is not what that patch does.15:15
gfidentebnemec, we can add the interface anyway, we're not there yet15:15
openstackgerritMerged openstack/tripleo-heat-templates: Revert "Revert "neutron: enable nova-event-callback by default""  https://review.openstack.org/26982715:15
derekhbnemec: but /me doesn't know the details, I've mainy been helping by getting access to various machines for debugging etc..15:15
bnemecThe CI failure on that patch is clear.  I'm not sure what needs to be debugged at the moment.15:16
dprincebnemec: last I saw puppet was hanging15:16
dprincebnemec: and we don't know exactly why that was yet15:16
bnemecdprince: That was on the previous version.15:16
dprincegfidente: what was the puppet error?15:17
bnemecThe current one is failing on Trying to ping default gateway\n10.0.0.1 is not pingable.15:17
bnemecBecause is not being configured on the undercloud anymore.15:17
*** dmacpher has joined #tripleo15:17
dprincebnemec: oh, yeah, well that is the public vlan then15:17
gfidentebnemec, dprince so we can add a fake to make validation to pass15:18
dprincebnemec: sorry, I missed something that gfidente did to get puppet output15:18
gfidentewe can also change the default gateway to something else so that validation will pass15:18
gfidenteI will update the submission adding back
gfidentebut what we're debugging is why corosync on the nodes can't talk to each other15:18
dprincegfidente: rather than just focus on HA, could we do this instead? https://review.openstack.org/#/c/285674/415:19
gfidenteor why computes can't reach rabbit on the controller15:19
dprincegfidente: in my patch the compute was failing to run puppet. But the controller seemed to complete...15:19
gfidentedprince, in nonha the controller is only one15:19
dprincegfidente: I'd like to focus on the simplest case which is a failure first15:19
gfidentedprince, yes agreed15:20
gfidenteI am only saying that if controller worked in nonha is probably because there is no communication with other nodes involved15:20
gfidenteuntil it gets to compute15:20
dprincegfidente: agree, that is why15:20
gfidentebut it's still simpler case15:20
dprincegfidente: but it is a simpler case so I'd rather start there...15:20
dprincegfidente: did you do something to enable the puppet output?15:20
gfidentedprince, no we were looking at the testenv which derekh brought up15:21
dprincegfidente: I see. So you guys are off in the weeds :)15:21
gfidenteyes! :(15:22
gfidenteI could verify that ping on the vlans work though15:22
*** pradk has quit IRC15:22
gfidentebandini, saw traffic with tcpdump15:22
dprincegfidente: yes, the validation (which is now fixed) tests that15:22
*** pradk has joined #tripleo15:22
dprincegfidente: I didn't question that. I'd like to see more specifically the puppet output from the compute failure. Where it fails and why15:23
gfidentedprince, yes I'll run puppet manually on the new env so we have the puppet output15:23
dprincegfidente: is this iptables on the controller blocking it for example?15:23
gfidentedprince, on the ha scenario iptables was empty cause puppet failed because setting it up15:24
*** dustins_ has joined #tripleo15:28
*** dustins has quit IRC15:29
*** rpothier has joined #tripleo15:29
bandiniderekh: just making sure is the undercloud where I can debug the pcsd/corosync issue. correct?15:33
bandiniand how much time will I have to play with the env once it starts deploying the overcloud?15:33
derekhbandini: yes correct IP, env should stay up a few hours once tab 3 isn't killed15:34
*** sshnaidm has joined #tripleo15:34
derekhbandini: on a call, back in a few15:34
bandiniderekh: ack, thanks15:34
*** aufi has joined #tripleo15:35
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates: Set project name for nova -> neutron notifications  https://review.openstack.org/28550915:37
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Roll up static Heat envs into CI directory  https://review.openstack.org/28043115:37
gfidentedprince, bnemec, ^^ I brought back in the NETISO flag15:37
*** mcornea_ has joined #tripleo15:37
*** pblaho has quit IRC15:37
*** panda has quit IRC15:37
*** mcornea_ has quit IRC15:37
*** mcornea_ has joined #tripleo15:38
*** panda has joined #tripleo15:38
gfidenteexcept I did it in the wrong change15:39
bnemecslagle: Did we just break Liberty CI then?  How could that have merge conflicted? :-/15:39
*** mbound has quit IRC15:40
*** mbound has joined #tripleo15:40
*** mcornea has quit IRC15:40
slaglebnemec: not sure. there was no conflict when i rebased it locally15:40
slaglei +A'd it again15:40
bnemecslagle: Annoying.  I intentionally approved that patch first so they would merge together.  Oh well.15:41
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Roll up static Heat envs into CI directory  https://review.openstack.org/28043115:43
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Use netiso in the ha job  https://review.openstack.org/27342415:43
gfidente^^ better now15:43
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Enable network isolation on all the jobs.  https://review.openstack.org/28567415:45
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Deploy Gnocchi as a Ceilometer metrics storage backend  https://review.openstack.org/25203215:47
*** devvesa has quit IRC15:49
*** masco has quit IRC15:49
*** akuznetsov has joined #tripleo15:50
*** shardy has joined #tripleo15:51
*** yamahata has quit IRC15:51
*** akuznetsov has quit IRC15:55
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo: IPv6: duak-stack support for Keystone  https://review.openstack.org/28634415:58
*** rpothier has left #tripleo15:59
hewbroccapradk: I keep seeing promising-looking reviews16:03
hewbroccapradk: are you also working on AODH?16:03
pradkhewbrocca, yea16:03
hewbroccapradk: do you need help?16:03
pradkhewbrocca, i think its in a pretty decent shape now, if i can get some reviews that would be super helpful. The ci keeps running out of memory on oc nodes though.16:04
pradkhewbrocca, had an email conversation with slagle, derekh and dprince to see if we can bump the memory16:05
bnemechewbrocca: Can you look at https://etherpad.openstack.org/p/tripleo-mitaka-rc-blockers and let us know if there's anything missing that is a drop-dead requirement in Mitaka?16:05
*** masco has joined #tripleo16:05
*** jaosorior is now known as jaosorior_away16:06
openstackgerritMerged openstack/tripleo-heat-templates: Set project name for nova -> neutron notifications  https://review.openstack.org/28550916:07
*** chlong_ has quit IRC16:10
*** dmacpher is now known as dmacpher-afk16:11
*** aufi has quit IRC16:12
openstackgerritAthlan-Guyot sofer proposed openstack/puppet-tripleo: IPv6: duak-stack support for Keystone  https://review.openstack.org/28634416:12
*** paramite has quit IRC16:14
*** liverpooler has quit IRC16:15
gfidentederekh, dprince so we can ping on the vlans but we can't telnet (ssh) the other nodes it seems16:16
*** dcain has joined #tripleo16:17
openstackgerritEmilien Macchi proposed openstack/puppet-tripleo: IPv6: duak-stack support for Keystone  https://review.openstack.org/28634416:17
gfidentederekh, join screen on window 016:17
dprincegfidente: which machine?16:17
dprincegfidente: which testenv?16:17
gfidente ssh jenkins@
gfidentethen screen -x16:18
*** leanderthal is now known as leanderthal|afk16:19
dprincegfidente: add my key? https://launchpad.net/~dan-prince/+sshkeys16:19
gfidentedprince, done16:20
hewbroccabnemec: added upgrades16:21
bnemechewbrocca: Good call, thanks.16:22
hewbroccaOh -- predictable hostnames16:24
openstackgerritAthlan-Guyot sofer proposed openstack/puppet-pacemaker: Basic beaker one node test.  https://review.openstack.org/28137616:24
hewbroccaalso keystone + wsgi (requires pcmk constraint changes)16:24
*** olap has quit IRC16:24
*** yamahata has joined #tripleo16:25
gfidentedprince, with bandini we were thinking about packets being propagate on other testenvs on the same baremetal16:27
bnemechewbrocca: I think that's covered by the HostnameMap change above.  And I thought we had the Keystone change merged, is that just fixing the constraints?16:27
*** xinwu has joined #tripleo16:27
derekhgfidente: ping works but ssh doesn't, sounds like what happens to me at home when I have MTU issues16:27
hewbroccabnemec: yeah, we gotta get the constraints fixed16:27
hewbroccabandini: is working on that16:27
gfidentederekh, MTU issues good hint16:28
*** dustins_ is now known as dustins16:28
bandinidprince: http://acksyn.org/files/tripleo/pcs-debug.pcap here is the network traffic on vlan20 port 2224 (of pcs's traffic during setup). if you look at it there is something rather fishy (truckloads of retransmits on syn, as if other hosts with the same ip were on the same segment)16:28
bandinibut yeah it could be mtu issues as well16:29
dprincederekh, bandini: but we don't change MTU in our CI anywhere that I know of from the defaults16:32
*** absubram has joined #tripleo16:32
bandinidprince: ack. is the hypotesis of other hosts with the same ip addresses being present on the same segment a viable one?16:33
dprincebandini: each test environment has its own bridge16:33
*** absubram_ has joined #tripleo16:33
dprincebandini: which should isolate the traffic from one another...16:34
gfidentedprince, though all underclouds share a nic on the baremetal bridge16:34
*** tiswanso has quit IRC16:34
gfidentebut traffic shouldn't go that far16:34
bandinidprince: I see, I am a bit out of ideas ;) it clearly is something fishy with the network (mtu and duplicate hosts were my only hypothesis so far)16:35
dprincebandini: I guess it is possible16:35
*** tiswanso has joined #tripleo16:35
dprincebandini: we can switch the MTU as a test16:35
EmilienMdprince: wdyt about enabling puppet parser future in tripleo?16:35
gfidentedprince, yeah I was looking into the nic templates to try that16:35
EmilienMthat would be so nice16:35
bandinidprince: aye, giulio is looking into that16:35
EmilienMdprince: https://docs.puppetlabs.com/puppet/3/reference/experiments_future.html16:35
EmilienMdprince: so we could iterate data within manifests16:36
dprincegfidente: it'll be a custom patch I think to t-h-t16:36
*** yamahata has quit IRC16:36
EmilienMdprince: though we might need puppet 4.116:36
EmilienM(at least)16:36
dprinceEmilienM: yeah, I'm mixed16:37
*** absubram has quit IRC16:37
*** absubram_ is now known as absubram16:37
dprinceEmilienM: is there some language feature you really want to use?16:37
EmilienMdprince: but the features are awesome16:37
EmilienMdprince: example: https://review.openstack.org/28634416:37
EmilienMdprince: with future parser I could have done a patch in 4 lines16:37
gfidentederekh, can I retrigger overcloud deploy manually or is the env going away?16:38
derekhgfidente: you can retrigger it, as long as the command in tab 3 isn't kill the env will be kept16:39
derekhgfidente: we'll it will timeout eventually but you should be good for a few hours16:40
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates: endpoint_map: bump nova api from 2.0 to 2.1  https://review.openstack.org/26336616:43
openstackgerritDan Prince proposed openstack/tripleo-heat-templates: WIP: Bump down the MTU for the single-nic-with-vlans  https://review.openstack.org/28670716:44
*** dustins has quit IRC16:49
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Add undercloud idempotency test to periodic job  https://review.openstack.org/27921816:52
EmilienMdprince: what prevents us to use parser?16:52
EmilienMcould we give it a try?16:52
gfidentedprince, ack, we did the edits manually and are testing that already16:52
openstackgerritMerged openstack/tripleo-heat-templates: Let Puppet update all packages on non-controllers  https://review.openstack.org/28448916:53
dprinceEmilienM: it could be fine. but I'd rather put the effort into composable service roles at this point and do the refactoring there16:54
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Enable undercloud ssl on nonha job  https://review.openstack.org/27374316:54
*** rcernin has quit IRC16:55
openstackgerritOpenStack Proposal Bot proposed openstack/python-tripleoclient: Updated from global requirements  https://review.openstack.org/26852816:56
EmilienMdprince: Sure, what can I do?16:56
*** mikelk has quit IRC16:56
*** masco has quit IRC16:58
*** mbound has quit IRC17:01
EmilienMdprince: this is an experiment: https://review.openstack.org/28671417:01
dprincegfidente: can I make the CI patch depend on this? https://review.openstack.org/#/c/286707/17:02
gfidentedprince sure but on the host it's set to 1400 on all vlans but telnet still not going17:02
gfidentecheck screen17:02
gfidentederekh, ^^17:03
derekhgfidente: which address are you ssh'ing to?17:04
gfidentewe're trying ssh over internal_api17:04
gfidentein between controller/compute17:04
*** ifarkas has quit IRC17:05
*** jistr has quit IRC17:05
gfidentederekh, on window 017:06
*** pcaruana has quit IRC17:07
derekhgfidente: ack, just looking around on another tab at the moment17:07
*** mcornea_ has quit IRC17:08
openstackgerritBen Nemec proposed openstack/tripleo-heat-templates: Enable notifications on the overcloud  https://review.openstack.org/28629917:08
EmilienMslagle: do you know if I can run OOO jobs on openstack/heat-templates patches?17:09
EmilienMI don't see the jobs in the zuul layout, neither in experimental17:09
EmilienMslagle: can I run a patch in THT and use depends-on?17:09
hewbroccadsneddon: over here ^^^17:10
derekhgfidente: so I'm running tcpdump on the compute node, it is getting the tcp setup traffic for ssh17:11
gfidenteso I never got the ack to telnet though17:12
*** stendulker has joined #tripleo17:13
derekhgfidente: ack,17:13
*** xinwu has quit IRC17:18
derekhgfidente:  weird  [heat-admin@overcloud-controller-0 ~]$ ssh
derekh Are you sure you want to continue connecting (yes/no)? ^C17:20
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Enable per-role SchedulerHints  https://review.openstack.org/26693017:20
derekhgfidente: ^ that what isn't working isn't it ?17:20
*** athomas has quit IRC17:21
bandiniderekh: exactly, sometimes it does work (although most of the times it will just hang there)17:21
derekhbandini: ack17:21
gfidentederekh, does it work consistently?17:21
derekhgfidente: nope, I just thought it was weird that it worked once17:22
gfidenteah yeah17:22
EmilienMderekh, dprince: do you know how can I test https://review.openstack.org/#/c/286714/ with tripleo jobs? patching THT with a Depends-On would work? do we use zuul-cloner to pull heat-templates?17:25
*** ohamada has quit IRC17:26
*** athomas has joined #tripleo17:26
derekhEmilienM: a depends on will get you the correct patch on the jenkins node, do we actually us it though17:29
*** dtantsur is now known as dtantsur|afk17:29
EmilienMderekh: the question is: do we use zuul cloner to clone openstack/heat-templates17:29
derekhgfidente: I'm going to have to run in about 10 minutes, if that env gets reclaimed you can rerun the command in tab 317:30
gfidentederekh, ok thanks17:30
openstackgerrityolanda.robla proposed openstack/diskimage-builder: Add dib element to generate logical volumes  https://review.openstack.org/25204117:30
derekhgfidente: I can join again later tonight (in about 4 or 5 hour) if you want me to  do anything send me a mail17:30
*** tiswanso has quit IRC17:30
gfidentederekh, ack17:31
derekhEmilienM: we're using the normal devstack-gate scripts, so I believe the answer to your question is yes, as that uses zuul-cloner doesn't it?17:31
gfidentederekh, last thing, can we get on the baremetal node hosting this environment somehow?17:32
EmilienMderekh: fair enough, I'll try17:32
*** tiswanso has joined #tripleo17:32
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Test Puppet Parser Future - Do not merge  https://review.openstack.org/28673217:33
shardybnemec: Hey, good to hear you're trying the scheduler hints stuff, perhaps you can leave feedback re your findings on the docs patch https://review.openstack.org/#/c/274217 :)17:34
shardyI tested it a while back and all seemed OK, then I re-tested when I wrote the docs and saw inconsistent results, e.g the nodes weren't pinned as expected17:35
derekhgfidente: the host you want to get to is testenv6-testenv3-t2oqwxetgmbd  , dprince can you see if you can help gfidente out with that, I did it earlier by letting him use my account but puppet periodically changes the ssh keys back17:35
bnemecshardy: Yeah, I'm finding that the scheduler hints aren't showing up in the ComputeCapabilitiesFilter.17:35
bnemecStill trying to figure out why.17:35
shardybnemec: Hmm, that would explain it17:35
shardybnemec: Are you running master tripleo or an older build?17:36
bnemecshardy: master tripleo17:36
shardyI just rebased the liberty "Enable per-role SchedulerHints" patch, but I've mostly been testing on master17:36
*** trown is now known as trown|lunch17:37
openstackgerritRichard Su proposed openstack/instack-undercloud: Option to store events in Undercloud Ceilometer  https://review.openstack.org/28673417:37
*** derekh has quit IRC17:43
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates: Add HostnameMap to allow granular control of hostnames  https://review.openstack.org/28673917:44
*** stendulker has quit IRC17:45
*** absubram has quit IRC17:48
openstackgerritJames Slagle proposed openstack/instack-undercloud: Set max_resources_per_stack to -1  https://review.openstack.org/28674617:55
openstackgerritJames Slagle proposed openstack/instack-undercloud: Set max_resources_per_stack to -1  https://review.openstack.org/28674617:56
*** fgimenez has quit IRC17:57
*** hjensas has quit IRC17:58
hewbroccadprince: So... what's our fallback if we can't get net-iso CI going today?18:00
openstackgerritMerged openstack/tripleo-heat-templates: Update the path to the Docker file  https://review.openstack.org/26413718:01
*** hjensas has joined #tripleo18:02
openstackgerritPradeep Kilambi proposed openstack/tripleo-puppet-elements: Gnocchi Integration  https://review.openstack.org/23801318:03
openstackgerritMerged openstack/tripleo-heat-templates: Enable heat-manage purge_deleted cron job  https://review.openstack.org/27934218:03
*** penick has joined #tripleo18:05
EmilienMany change to get https://review.openstack.org/#/c/286584/ landed?18:05
*** rdopiera has quit IRC18:05
dprincehewbrocca: manual testing?18:06
EmilienMand https://review.openstack.org/#/c/286031/18:06
openstackgerritMerged openstack/instack-undercloud: Set max_resources_per_stack to -1  https://review.openstack.org/28616518:06
dprincehewbrocca: when people aren't doing AFAIKT. At least not with upstream...18:06
dprincehewbrocca: which people aren't doing18:06
dprincehewbrocca: still working the the CI job in the meantime18:07
hewbroccaEmilienM: ^^^ that's what we're going to have to do then, if we don't get the CI job going18:07
hewbroccawe can't afford to hold off any longer18:07
hewbroccaThanks guys. I have to head out, I will see you all tomorrow18:07
*** xinwu has joined #tripleo18:07
openstackgerritgreghaynes proposed openstack/diskimage-builder: Make debootstrap cache opt-in  https://review.openstack.org/28588618:11
*** openstackgerrit has quit IRC18:18
*** openstackgerrit has joined #tripleo18:18
*** lucasagomes is now known as lucas-dinner18:22
gfidentehewbrocca, dprince so we were trying to do some tcpdump on the baremetal with bandini18:24
gfidentewe do see packets going18:24
bandinibut stupidly slow...we see like 1 second delay to reply to a syn18:25
bandiniand then it increases and and then sometimes it just gets stuck it seems18:25
gfidenteand now we tried again and I don't see acks coming back at all18:26
gfidentethe only difference I could find from a working environment is that ovs, on the baremetal18:26
gfidenteseems to not be forwarding traffic to all ports but only on the selected ports where the sender/receiver are?18:27
gfidenteI am not sure if this is a behaviour we can configure nor if it is relevant, I just see it behaving differently18:27
*** weshay has quit IRC18:27
gfidenteso I am sorry but basically I still don't know what to blame18:28
*** jaosorior_away is now known as jaosorior18:31
dprincegfidente: the latest CI results for non-ha and Ceph show stack completed successfully18:31
dprincegfidente: CI job still fails but the heat stack was created18:32
*** mgould has quit IRC18:32
dprincegfidente: just noticed this:18:33
dprince2016-03-01 17:22:59.688 | | tenant-stack          | 43a543af-7aab-4d76-bb92-3d4f4ffde482 | Resource CREATE failed: BadRequest: resources.router_interface: Bad router request: Cidr of subnet 6278d068-8a92-467b-b9ed-93dac23c2425 overlaps with cidr of subnet 984b559e-d9d5-4ea4-91da-44711764b5c3. | CREATE_FAILED      | 2016-03-01T17:03:42 |18:33
dprincegfidente: we've got a /8 somewhere?18:34
gfidentethe overcloud ping test uses it seems18:35
gfidenteI suspect it's creating it in the wrong "cloud"18:35
*** trown|lunch is now known as trown18:35
dprincegfidente: still the jobs are getting quite far18:35
dprincegfidente: I'm not sure exactly what you and derekh did this morning to create this custom environment18:36
dprincegfidente: could you run me through that? because, I'm wondering if perhaps it is related to the errors you see now?18:36
*** hjensas has quit IRC18:37
gfidentedprince, I think it's a regular CI job where he purposely set a longer timeout18:37
gfidenteit's running on one of the existing testenvs18:38
dprincegfidente: and you are attaching via the jenkins slave?18:38
gfidentedprince, yes18:38
*** dustins has joined #tripleo18:38
dprincegfidente: okay, shouldn't be anything too odd about that18:38
gfidentebut are the CI jobs you see passing from https://review.openstack.org/#/c/285674/ ?18:39
dprincegfidente: if we get either ceph or nonha passing I'm going to say lets just go w/ it18:39
dprincegfidente: no, your most recent patch which adopted mine18:40
dprincegfidente: Now that you enabled all jobs I'll ditch mine18:40
gfidenteno wait no18:40
gfidentethose don't run netiso on nonha and ceph18:40
dprincegfidente: seriously, why?18:41
gfidenteI defaulted to 1 NETISO but it only creates the vlan10 on the underclloud18:41
gfidenteI was not passing the envs to the other jobs too18:41
dprincegfidente: why are we waisting time and resources on this. PLus it is just really confusing not to look at them all at once18:41
dprincegfidente: lets do them all, until we see one passing18:41
dprincegfidente: you're previous patch did that no?18:42
gfidenteso I didn't enable it for all jobs because I thought we want to check non netiso too18:42
dprincegfidente: we do, This is a WIP18:42
gfidentebut at least for the purpose of testing your submission is better as it enables it for all18:42
dprincegfidente: forget about the -1's complaining about enabling it for all jobs and call this a WIP18:43
gfidenteso last thing I can do before going is rebase on the MTU change18:43
dprincegfidente: I can do it18:43
dprincegfidente: if you need to sign off no worries18:43
gfidenteI think the forwarding issue on the baremetal is the real thing18:44
gfidentewe were looking with wireshark at pcap files18:44
gfidentethe syns are retransmitted, the acks too18:44
gfidenteas if things were just extremely slow18:44
dprincegfidente: we could just be hitting an old OVS bug or something18:45
gfidenteso tcp connections time out, yet ping works18:45
dsneddongfidente, That sounds like MTU18:45
gfidentedsneddon, hey!18:46
gfidentethe environment we're looking at now has mtu set to 1400 on the overcloud nodes18:46
dsneddongfidente, I've been reading along for a while, just didn't have any helpful suggestions18:46
dprincegfidente: okay, so rebase to try this https://review.openstack.org/#/c/286707/18:46
gfidentedprince, ack18:48
gfidentethen I'll leave guys18:48
*** akuznetsov has joined #tripleo18:48
dprincedsneddon: could you have a gander at https://review.openstack.org/#/c/286707/ and see if it is missing anything.18:48
gfidenteI am sorry but ... couldn't figure much more18:48
dprincedsneddon: we only need the 3 nodes18:48
gfidentedprince, we tested that identical change and it works18:48
dprincedsneddon: 3 roles18:50
gfidentethe vlan interfaces have mtu 140018:50
dsneddongfidente, You have to set the MTU on the bridge, too18:50
gfidenteit just doesn't change the end result18:50
gfidentedsneddon, ok I will18:50
*** tserong has quit IRC18:52
dsneddongfidente, And the interface, too18:52
dsneddongfidente, It has to be set at all levels.18:52
*** afazekas has quit IRC18:52
*** SpamapS has quit IRC18:54
*** openstack has joined #tripleo19:14
*** SpamapS has joined #tripleo19:15
*** openstackstatus has joined #tripleo19:18
*** ChanServ sets mode: +v openstackstatus19:18
*** openstackgerrit has joined #tripleo19:19
*** hjensas has quit IRC19:25
*** CaptTofu has quit IRC19:25
*** StevenK has quit IRC19:25
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates: Support adding swap to overcloud nodes  https://review.openstack.org/27375219:27
openstackgerritJames Slagle proposed openstack/tripleo-heat-templates: Support adding a swap file to overcloud nodes  https://review.openstack.org/27375219:29
*** CaptTofu has joined #tripleo19:31
*** StevenK has joined #tripleo19:31
*** jrist has quit IRC19:31
*** CaptTofu has quit IRC19:31
*** hjensas has joined #tripleo19:32
*** hjensas has quit IRC19:32
*** hjensas has joined #tripleo19:32
*** CaptTofu has joined #tripleo19:33
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Use swapfile environment in CI  https://review.openstack.org/28679319:38
*** panda has quit IRC19:38
*** panda has joined #tripleo19:39
slaglepradk: this is the patch you can try a Depends-On for the adoh/gnocchi patches, https://review.openstack.org/#/c/28679319:40
slagleit should result in a 4GB swapfile getting added to the oc nodes19:40
pradkslagle, great, will do now19:40
slaglenp. i hope it works19:41
*** akuznetsov has joined #tripleo19:42
*** ccamacho has quit IRC19:43
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates: Deploy Aodh services, replacing Ceilometer Alarm  https://review.openstack.org/24140819:44
*** jprovazn has quit IRC19:48
ayoungMad props to you guys...very happy with the tripleo.sh script and triple-common....been setting up and tearing down all day19:55
ayoungI think we might need to up the minimum sizeof a VM, though19:55
ayoungI had deploys failing before setting the node size to 16 G.  Trying now with 8,  but 4 is too small19:55
*** jrist has joined #tripleo19:55
ayoungI think that was just the controller...19:55
*** david-lyle has quit IRC19:59
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: WIP: Enable network isolation in all CI jobs  https://review.openstack.org/27342420:02
*** xinwu has quit IRC20:05
*** julim has quit IRC20:07
*** julim has joined #tripleo20:08
*** jcoufal_ has quit IRC20:08
*** david-lyle has joined #tripleo20:09
*** julim has quit IRC20:10
*** jaosorior has quit IRC20:12
*** mburned is now known as mburned_out20:17
*** mburned_out is now known as mburned20:19
*** mburned is now known as mburned_out20:20
*** yamahata has quit IRC20:21
*** ccamacho has joined #tripleo20:27
*** akuznetsov has quit IRC20:29
*** penick has quit IRC20:38
*** ccamacho has quit IRC20:38
*** ccamacho has joined #tripleo20:38
*** yamahata has joined #tripleo20:39
*** mburned_out is now known as mburned20:40
*** pcaruana has joined #tripleo20:44
*** xinwu has joined #tripleo20:47
*** yamahata has quit IRC20:47
openstackgerritMerged openstack/instack-undercloud: Increase scheduler_max_attempts on undercloud and make it configurable  https://review.openstack.org/28600820:50
*** julim has joined #tripleo20:55
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: Use swapfile environment in CI  https://review.openstack.org/28679320:58
openstackgerritJohn Trowbridge proposed openstack/instack-undercloud: Deploy Monitoring on the undercloud with Puppet  https://review.openstack.org/27612720:58
*** ccamacho has quit IRC20:58
*** ccamacho has joined #tripleo21:04
*** weshay has joined #tripleo21:04
dprinceThis would be helpful to the UI developers: https://review.openstack.org/#/c/283221/21:04
dsneddonEmilienM, I'm currently developing two alternate solutions to outputting both IPv4 and IPv6 IPs for external network and Public API, not sure which will end up being the best.21:08
dsneddonEmilienM, In one case, I create a second external network, and create new ports everywhere for that network (but they default to noop,yaml, so will output the control plane IP/VIP).21:08
EmilienMdsneddon: cool. I just need an array whatever your alternative :P21:08
*** david-lyle has quit IRC21:08
dsneddonEmilienM, In the other case, I create two subnets on the same network. Neutron then gives the port 2 IPs, one IPv6 and one IPv4. I just haven't figured out how to make the external load balancer work with that, where we preselect the controller IPs.21:09
EmilienMfirst solution seems ok to me21:10
dsneddonEmilienM, Yeah, but then the first solution creates a lot more cruft for the default case (extra ports and parameters in more places)21:11
dsneddonEmilienM, I'll probably end up posting early WIP reviews for both soon.21:11
openstackgerritJohn Trowbridge proposed openstack/instack-undercloud: Deploy Monitoring on the undercloud with Puppet  https://review.openstack.org/27612721:13
trownEmilienM: hopefully its ok if I rebase/recheck/generally catherd  your opstools patches :)21:14
EmilienMtrown: go ahead !21:15
*** david-lyle has joined #tripleo21:17
*** jayg is now known as jayg|g0n321:19
*** mburned is now known as mburned_out21:27
*** mburned_out is now known as mburned21:29
openstackgerritGabriele Cerami proposed openstack/instack-undercloud: mysqld config: set innodb_file_per_table to ON  https://review.openstack.org/28522721:31
*** yamahata has joined #tripleo21:32
*** mburned is now known as mburned_out21:33
openstackgerritGabriele Cerami proposed openstack/tripleo-heat-templates: mysqld config: set innodb_file_per_table to ON  https://review.openstack.org/28522421:34
*** mburned_out is now known as mburned21:36
*** derekh has joined #tripleo21:41
*** dshulyak has quit IRC21:43
*** dshulyak has joined #tripleo21:44
*** r-mibu has quit IRC21:46
*** r-mibu has joined #tripleo21:47
*** dshulyak has quit IRC21:48
*** rhallisey has quit IRC21:55
*** shivrao has joined #tripleo21:55
*** ccamacho has quit IRC21:58
*** ccamacho has joined #tripleo21:59
*** rhallisey has joined #tripleo21:59
derekhbandini: did ye find the problem?21:59
*** trown is now known as trown|outtypewww21:59
*** pcaruana has quit IRC22:01
bandiniderekh: nope. it is definitely a network issue...it's as if everything is slow, or packets dropped and sometimes duplicated22:01
bandiniI will discuss it again with giulio tomorrow22:01
derekhbandini: ok, is anybody poking at the env? If not I'll see if I can figure anything out22:02
shardyderekh: Hey, I just noticed that the undercloud install pulls from https://github.com/derekhiggins/puppet-vlan22:03
shardyis that expected?22:03
shardy(I'm assuming it's an old stray element dep somewhere)22:04
derekhshardy: yes, thats where the puppet module lives is it used? iirc I wrote it for packstack at one stage22:04
shardyderekh: I guess I was surprised to see us pulling from a personal repo, I'm not sure if it's used tbh22:05
* shardy will investigate tomorrow22:05
derekhshardy: I wouldn't be surprised if it isn't used, somebody probably took the list of repos that packstack uses or something like that22:05
shardyyeah, sounds likely, thanks for the info!22:06
* derekh didn't put it there22:06
*** dprince has quit IRC22:08
derekhshardy: its got impressive download stats, https://github.com/derekhiggins/puppet-vlan/graphs/traffic22:09
shardyhehe, I can't actually see the stats, but I can imagine ;)22:12
*** tiswanso has quit IRC22:12
openstackgerritMerged openstack/instack-undercloud: Enable AMT driver  https://review.openstack.org/28625222:13
*** dshulyak has joined #tripleo22:14
*** shardy has quit IRC22:15
*** ccamacho has quit IRC22:15
dmsimardfyi, trown's out so I figured I'd let you know .. it looks like there are some issues around swift backend commits merged in glance_store today22:16
dmsimardwe had to fix stuff in packstack and puppet-openstack, I added a note on the etherpad https://etherpad.openstack.org/p/delorean_master_current_issues22:16
*** penick has joined #tripleo22:16
dmsimardI know you guys are generally pinned but just in case22:17
*** dshulyak has quit IRC22:19
*** shivrao has quit IRC22:19
*** lblanchard has quit IRC22:21
openstackgerritMerged openstack/os-collect-config: Replace assertEqual(None, *) with assertIsNone in tests  https://review.openstack.org/26327322:28
*** david-lyle has quit IRC22:29
*** shivrao has joined #tripleo22:30
*** yamahata has quit IRC22:30
*** rbrady has quit IRC22:34
*** jtomasek has quit IRC22:41
*** rhallisey has quit IRC22:43
*** trozet has quit IRC22:49
*** david-lyle has joined #tripleo23:00
*** ianw has quit IRC23:07
*** chlong_ has joined #tripleo23:13
*** dustins has quit IRC23:15
*** dcain has quit IRC23:22
*** lblanchard has joined #tripleo23:30
pradkslagle, \o/ looks like the swap patch worked, the nonha ci now passes https://review.openstack.org/#/c/241408/23:35
*** panda has quit IRC23:37
*** panda has joined #tripleo23:38
*** david-lyle has quit IRC23:41
EmilienMpradk: nice, still issues with HA? did my suggestion helped?23:59
* EmilienM looks logs23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!