14:01:16 <mwhahaha> #startmeeting tripleo 14:01:17 <mwhahaha> #topic agenda 14:01:17 <mwhahaha> * Review past action items 14:01:17 <mwhahaha> * One off agenda items 14:01:17 <mwhahaha> * Squad status 14:01:17 <mwhahaha> * Bugs & Blueprints 14:01:17 <openstack> Meeting started Tue Nov 14 14:01:16 2017 UTC and is due to finish in 60 minutes. The chair is mwhahaha. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:01:17 <mwhahaha> * Projects releases or stable backports 14:01:18 <mwhahaha> * Specs 14:01:18 <mwhahaha> * open discussion 14:01:18 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:01:19 <mwhahaha> Anyone can use the #link, #action and #info commands, not just the moderatorǃ 14:01:19 <mwhahaha> Hi everyone! who is around today? 14:01:21 <openstack> The meeting name has been set to 'tripleo' 14:01:44 <shardy> o/ 14:01:47 <jtomasek> o/ 14:01:50 <abishop> o/ 14:01:55 <jaosorior> o/ 14:02:01 <weshay> o/ 14:02:03 <beagles> o/ 14:02:25 <jfrancoa> o/ 14:02:28 <fultonj> o/ 14:02:56 <marios> o/ 14:03:14 <chem> o/ 14:03:19 <openstackgerrit> Daniel Alvarez proposed openstack/puppet-tripleo master: Add support for OVN Metadata Agent https://review.openstack.org/502940 14:03:33 <d0ugal_> o/ 14:03:55 <mwhahaha> ok let's start 14:04:00 <matbu_> o/ 14:04:05 <mwhahaha> #topic review past action items 14:04:10 <mwhahaha> team to review upgrades developer docs https://review.openstack.org/#/c/517916/ 14:04:17 <mwhahaha> that review is just the structure 14:04:27 <mwhahaha> we should get that in so we can start filling out the upgrade sections 14:04:43 <mwhahaha> looks like it merged last night 14:04:48 <gfidente^2nd> o/ 14:04:51 <mwhahaha> so I look forward to seeing actual details :D 14:05:00 <mwhahaha> gfidente to send a note requesting feedback on the ML about multiple service instances issues 14:05:09 <jfrancoa> mwhahaha: yes, we'll try to split the work and start filling it in 14:05:14 <ccamacho> sorry for the delay o/ 14:05:34 <oidgar> o/ 14:05:52 <mwhahaha> gfidente^2nd: any chance to compile a list for the ML? 14:06:20 <gfidente^2nd> mwhahaha I did not prepare that email for multiple clusters yet, sorry 14:06:29 <mwhahaha> k 14:06:33 <gfidente^2nd> please keep the item for next wek 14:06:37 <mwhahaha> #action gfidente to send a note requesting feedback on the ML about multiple service instances issues 14:06:45 <mwhahaha> mwhahaha send a note about CI to ML and propsing no more merging of items not specifically critical CI bugs - DONE http://lists.openstack.org/pipermail/openstack-dev/2017-November/124294.html 14:07:01 <openstackgerrit> Daniel Alvarez proposed openstack/tripleo-heat-templates master: Add support for OVN Metadata Agent https://review.openstack.org/502943 14:07:03 <mwhahaha> i've got an item to talk about what to do in the one off agenda 14:07:07 <mwhahaha> so moving on to that 14:07:13 <mwhahaha> #topic one off agenda items 14:07:18 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-meeting-items 14:07:34 <mwhahaha> (mwhahaha) scenario001 to non-voting? 14:07:51 <mwhahaha> so basically https://bugs.launchpad.net/tripleo/+bug/1731063 makes scenario001 a major issue 14:07:51 <openstack> Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] 14:08:14 <mwhahaha> unless we have some movement on figuring out what is happening, i propose we switch scenario001 in master to non-voting so we can unblock everyone 14:08:30 <mwhahaha> any thoughts or objections? 14:08:38 <gfidente> well 14:08:46 <gfidente> yes I have objections 14:08:50 <mwhahaha> i don't like losing the coverage but we've been broken for over a wekk 14:09:03 <gfidente> we're using ceph-ansible in scenario001 and scenario004, it's not clear to me why this should be happening only for one of the two 14:09:19 <mwhahaha> gfidente: most likely because its not ceph related 14:09:25 <openstackgerrit> Yurii Prokulevych proposed openstack/tripleo-heat-templates master: Add HostnameFormatDefault parameter to Block/Object/Ceph roles. https://review.openstack.org/519658 14:09:26 <fultonj> happens in non-containerized too 14:09:40 <fultonj> comments in that bug indicate ceph is ok as per job logs 14:09:54 <mwhahaha> yea so it's most likely a nova/cinder problem but without understanding what is happening we can't keep it in the gate 14:10:00 <fultonj> who on the team knows tempest well? 14:10:09 <gfidente> sshnaidm ^^ ? 14:10:16 <ccamacho> dmellado ^ 14:10:17 <ooolpbot> URGENT TRIPLEO TASKS NEED ATTENTION 14:10:18 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1724328 14:10:18 <openstack> Launchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel) 14:10:18 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1731032 14:10:19 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1731063 14:10:19 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1731456 14:10:19 <weshay> fultonj, arxcruz is our local guru 14:10:20 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1731540 14:10:20 <openstack> Launchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged] 14:10:21 <openstack> Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged] 14:10:22 <openstack> Launchpad bug 1731456 in tripleo "Timed out CI jobs not collecting logs, "FAILED with status: 137"" [Critical,Triaged] 14:10:23 <openstack> Launchpad bug 1731540 in tripleo "CI: Deployment times out because signal back to undercloud fails with a connection timed out" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) 14:10:34 <gfidente> I volunteer to look into the issue 14:10:52 <mwhahaha> gfidente: ok so we'll give it one more day 14:10:54 <fultonj> arxcruz: do you think you could look into my question about adding a wait and retry? 14:10:57 <fultonj> thanks gfidente 14:11:01 <gfidente> mwhahaha ack 14:11:07 <mwhahaha> but unless we can figure out what the cause is today, we need to switch it to non-voting tomorrow 14:11:16 <weshay> gfidente thanks 14:11:30 <adarazs|ruck> o/ 14:11:34 <mwhahaha> anyway moving on 14:11:37 <mwhahaha> (gfidente) help review ceph/luminous submission (passed CI) 14:12:00 <gfidente> yeah this is basically a call for help with reviews as we submitted a few patches to use ceph luminous instead of jewel 14:12:04 <gfidente> upstream CI seems happy 14:12:07 <trown> gfidente fultonj could either of you look at a reproducer env for that job? I cant get deploy to work because an issue with ceph-mon var/run file 14:12:17 <mwhahaha> gfidente: yea but we can't merge due to scenario001 flakyness 14:12:42 <gfidente> mwhahaha fwiw it passed https://review.openstack.org/#/c/510108/ 14:12:45 <mwhahaha> i'm not against the switch as you're right ci is happy but we won't merge until we resolve the previous topic 14:12:55 <mwhahaha> gfidente: yea i know the problem is that it could fail in the gate repeatedly 14:13:02 <gfidente> mwhahaha understood 14:13:03 <mwhahaha> hence previous topic 14:13:07 <gfidente> though reviews still helpful 14:13:11 <mwhahaha> yup 14:13:34 <Tengu> hello ! small question: are there some issues with cinder on the CI for stable/pike, especially for the gate-tripleo-ci-centos-7-containers-multinode-upgrades-pike running on "RDO Third Party CI" ? 14:13:50 <mwhahaha> Tengu: we can look into it after the moeeting 14:14:00 <Tengu> oh, sorry, didn't check the topic 14:14:14 <mwhahaha> gfidente: anything else about the ceph luminous switch? 14:14:54 <gfidente> mwhahaha nah, it should be work on upgrade too 14:15:00 <mwhahaha> ok sounds good 14:15:01 <gfidente> ceph-ansible has a playbook 14:15:02 <mwhahaha> (weshay) CI logging changes 14:15:03 <gfidente> so testabe 14:15:06 <gfidente> *testable 14:15:49 <weshay> mwhahaha, ya.. just a FYI for general awareness and reviews 14:16:09 <mwhahaha> #link https://review.openstack.org/#/c/511526/ 14:16:18 <mwhahaha> weshay: does that improve the log capture time or anything? 14:16:53 <weshay> I need to check, I'll comment in the review 14:16:57 <mwhahaha> k 14:17:13 <mwhahaha> any other one off items folks would like to talk about? 14:17:29 <openstackgerrit> Anastasia Kravets proposed openstack/puppet-tripleo master: Merge from from Juniper repo for opencontrail https://review.openstack.org/516651 14:17:34 <openstackgerrit> Anastasia Kravets proposed openstack/puppet-tripleo master: Merge from from Juniper repo for opencontrail https://review.openstack.org/516651 14:17:49 <mwhahaha> sounds like nope 14:17:52 <mwhahaha> status time 14:18:00 <openstackgerrit> Carlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud backup https://review.openstack.org/466213 14:18:06 <mwhahaha> #topic Squad status 14:18:06 <mwhahaha> ci 14:18:06 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-ci-squad-scrum 14:18:06 <mwhahaha> upgrade 14:18:06 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-upgrade-squad-status 14:18:07 <mwhahaha> containers 14:18:07 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-containers-squad-status 14:18:08 <mwhahaha> integration 14:18:08 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-integration-squad-status 14:18:09 <mwhahaha> ui/cli 14:18:09 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-ui-cli-squad-status 14:18:10 <mwhahaha> validations 14:18:10 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-validations-squad-status 14:18:11 <mwhahaha> networking 14:18:11 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-networking-squad-status 14:18:12 <mwhahaha> workflows 14:19:00 <d0ugal> I'm in the process of setting up the Workflows meeting, finally :) - will send out a email to the list today with the first meeting details. No status otherwise. 14:19:10 <mwhahaha> weshay, ci status seems missing 14:19:10 <d0ugal> but there will be more soon! 14:19:22 <mwhahaha> validations also missing status 14:19:30 <mwhahaha> jrist: ui/cli status missing 14:19:31 <weshay> mwhahaha, we'll have a sprint end, and planning update today 14:19:34 <weshay> arxcruz, ^ 14:19:43 <mwhahaha> k 14:19:57 <jrist> womp womp 14:20:14 <jrist> mwhahaha: I'll chat with jtomasek to see what the update was 14:20:20 <mwhahaha> jrist: thanks 14:20:46 <jtomasek> mwhahaha, jrist: I'll update it 14:21:15 <mwhahaha> moving on to bugs 14:21:19 <mwhahaha> #topic bugs & blueprints 14:21:19 <mwhahaha> #link https://launchpad.net/tripleo/+milestone/queens-2 14:21:19 <mwhahaha> For Queens we currently have 70 (+0) blueprints and about 526 (+5) open bugs. 251 queens-2 and 275 queens-3. 14:21:51 <mwhahaha> please take a look at the critical bug list, i reported some yesterday for some failures in CI that aren't blocking but should get resolved 14:22:16 <mwhahaha> for example, https://bugs.launchpad.net/tripleo/+bug/1732010 14:22:16 <openstack> Launchpad bug 1732010 in tripleo "CI: tripleoclient package build fails because unit tests fail because tmp file exists" [Critical,Triaged] 14:23:07 <mwhahaha> anyway have any other bugs they wish to bring up? 14:23:21 <mwhahaha> s/anyway/anyone 14:24:17 <openstackgerrit> Carlos Camacho proposed openstack/tripleo-common master: WIP add UC backup actions https://review.openstack.org/517610 14:24:37 <mwhahaha> sounds like nope 14:24:38 <adarazs|ruck> nope. just fix the critical ones quickly please, we need promotions :) 14:24:49 <mwhahaha> indeed 14:25:14 <mwhahaha> #topic projects releases or stable backports 14:25:16 <jpich> Maybe also https://bugs.launchpad.net/tripleo/+bug/1732140 - a patch is merging that will fix it, tripleo-common master is broken until then 14:25:16 <openstack> Launchpad bug 1732140 in tripleo "Kolla tests failing for tripleo-common master" [Critical,Invalid] - Assigned to Adriano Petrich (apetrich) 14:26:17 <mwhahaha> jpich: thanks for pointing that out, looks like https://review.openstack.org/#/c/516136/ is in the gate so we'll keep an eye on it 14:26:43 <mwhahaha> as for releases, I think EmilienM was working on getting a pike release cut. 14:26:57 <mwhahaha> reminder that queens-m2 is in ~3 weeks 14:27:30 <weshay> dear lord 14:27:37 <mwhahaha> so please make sure your blueprints are updated with their current status and please raise any issues that might prevent you from getting your feature merged before m2 (other than the known scenario001 problems) 14:28:20 <mwhahaha> also along those lines 14:28:24 <mwhahaha> #topic specs 14:28:24 <mwhahaha> #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open 14:28:35 <mwhahaha> please review specs, they need to be merged before m2 14:29:34 <mwhahaha> #topic open discussion 14:29:38 <mwhahaha> any other business? 14:31:13 <trown> gfidente fultonj could either of you help me with some ceph-ansible questions? 14:31:24 <fultonj> trown: sure 14:31:58 <openstackgerrit> Anastasia Kravets proposed openstack/tripleo-heat-templates master: Merge from from Juniper THT for opencontrail https://review.openstack.org/516630 14:32:02 <openstackgerrit> Liz Blanchard proposed openstack/tripleo-ui master: Add message to spinner on Deployment Config modal https://review.openstack.org/519478 14:32:08 <trown> fultonj: sweet... I have tracked down why multinode ceph jobs do not reproduce on rdocloud to this line https://github.com/ceph/ceph-ansible/blob/master/roles/ceph-mon/tasks/docker/main.yml#L10 14:33:00 <trown> fultonj: the "monitor_name" var there is resolving to "subnode-1", but the var run file is actually created as "*subnode-1.rdocloud.asok" 14:33:13 <mwhahaha> ok well sounds like no further items for the meeting 14:33:15 <mwhahaha> #endmeeting