14:01:16 <mwhahaha> #startmeeting tripleo
14:01:17 <mwhahaha> #topic agenda
14:01:17 <mwhahaha> * Review past action items
14:01:17 <mwhahaha> * One off agenda items
14:01:17 <mwhahaha> * Squad status
14:01:17 <mwhahaha> * Bugs & Blueprints
14:01:17 <openstack> Meeting started Tue Nov 14 14:01:16 2017 UTC and is due to finish in 60 minutes.  The chair is mwhahaha. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:01:17 <mwhahaha> * Projects releases or stable backports
14:01:18 <mwhahaha> * Specs
14:01:18 <mwhahaha> * open discussion
14:01:18 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:01:19 <mwhahaha> Anyone can use the #link, #action and #info commands, not just the moderatorǃ
14:01:19 <mwhahaha> Hi everyone! who is around today?
14:01:21 <openstack> The meeting name has been set to 'tripleo'
14:01:44 <shardy> o/
14:01:47 <jtomasek> o/
14:01:50 <abishop> o/
14:01:55 <jaosorior> o/
14:02:01 <weshay> o/
14:02:03 <beagles> o/
14:02:25 <jfrancoa> o/
14:02:28 <fultonj> o/
14:02:56 <marios> o/
14:03:14 <chem> o/
14:03:19 <openstackgerrit> Daniel Alvarez proposed openstack/puppet-tripleo master: Add support for OVN Metadata Agent  https://review.openstack.org/502940
14:03:33 <d0ugal_> o/
14:03:55 <mwhahaha> ok let's start
14:04:00 <matbu_> o/
14:04:05 <mwhahaha> #topic review past action items
14:04:10 <mwhahaha> team to review upgrades developer docs https://review.openstack.org/#/c/517916/
14:04:17 <mwhahaha> that review is just the structure
14:04:27 <mwhahaha> we should get that in so we can start filling out the upgrade sections
14:04:43 <mwhahaha> looks like it merged last night
14:04:48 <gfidente^2nd> o/
14:04:51 <mwhahaha> so I look forward to seeing actual details :D
14:05:00 <mwhahaha> gfidente to send a note requesting feedback on the ML about multiple service instances issues
14:05:09 <jfrancoa> mwhahaha: yes, we'll try to split the work and start filling it in
14:05:14 <ccamacho> sorry for the delay o/
14:05:34 <oidgar> o/
14:05:52 <mwhahaha> gfidente^2nd: any chance to compile a list for the ML?
14:06:20 <gfidente^2nd> mwhahaha I did not prepare that email for multiple clusters yet, sorry
14:06:29 <mwhahaha> k
14:06:33 <gfidente^2nd> please keep the item for next wek
14:06:37 <mwhahaha> #action gfidente to send a note requesting feedback on the ML about multiple service instances issues
14:06:45 <mwhahaha> mwhahaha send a note about CI to ML and propsing no more merging of items not specifically critical CI bugs - DONE http://lists.openstack.org/pipermail/openstack-dev/2017-November/124294.html
14:07:01 <openstackgerrit> Daniel Alvarez proposed openstack/tripleo-heat-templates master: Add support for OVN Metadata Agent  https://review.openstack.org/502943
14:07:03 <mwhahaha> i've got an item to talk about what to do in the one off agenda
14:07:07 <mwhahaha> so moving on to that
14:07:13 <mwhahaha> #topic one off agenda items
14:07:18 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-meeting-items
14:07:34 <mwhahaha> (mwhahaha) scenario001 to non-voting?
14:07:51 <mwhahaha> so basically https://bugs.launchpad.net/tripleo/+bug/1731063 makes scenario001 a major issue
14:07:51 <openstack> Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged]
14:08:14 <mwhahaha> unless we have some movement on figuring out what is happening, i propose we switch scenario001 in master to non-voting so we can unblock everyone
14:08:30 <mwhahaha> any thoughts or objections?
14:08:38 <gfidente> well
14:08:46 <gfidente> yes I have objections
14:08:50 <mwhahaha> i don't like losing the coverage but we've been broken for over a wekk
14:09:03 <gfidente> we're using ceph-ansible in scenario001 and scenario004, it's not clear to me why this should be happening only for one of the two
14:09:19 <mwhahaha> gfidente: most likely because its not ceph related
14:09:25 <openstackgerrit> Yurii Prokulevych proposed openstack/tripleo-heat-templates master: Add HostnameFormatDefault parameter to Block/Object/Ceph roles.  https://review.openstack.org/519658
14:09:26 <fultonj> happens in non-containerized too
14:09:40 <fultonj> comments in that bug indicate ceph is ok as per job logs
14:09:54 <mwhahaha> yea so it's most likely a nova/cinder problem but without understanding what is happening we can't keep it in the gate
14:10:00 <fultonj> who on the team knows tempest well?
14:10:09 <gfidente> sshnaidm ^^ ?
14:10:16 <ccamacho> dmellado ^
14:10:17 <ooolpbot> URGENT TRIPLEO TASKS NEED ATTENTION
14:10:18 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1724328
14:10:18 <openstack> Launchpad bug 1724328 in tripleo "Netwon to Ocata upgrade failure because of ceilometer-upgrade" [Critical,Triaged] - Assigned to mathieu bultel (mat-bultel)
14:10:18 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1731032
14:10:19 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1731063
14:10:19 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1731456
14:10:19 <weshay> fultonj, arxcruz is our local guru
14:10:20 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1731540
14:10:20 <openstack> Launchpad bug 1731032 in tripleo "CI: Deployment fails on controller timeout with task creation in heat timing out" [Critical,Triaged]
14:10:21 <openstack> Launchpad bug 1731063 in tripleo "CI: tempest TestVolumeBootPattern tests fail due to not being able to ssh to the VM" [Critical,Triaged]
14:10:22 <openstack> Launchpad bug 1731456 in tripleo "Timed out CI jobs not collecting logs, "FAILED with status: 137"" [Critical,Triaged]
14:10:23 <openstack> Launchpad bug 1731540 in tripleo "CI: Deployment times out because signal back to undercloud fails with a connection timed out" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz)
14:10:34 <gfidente> I volunteer to look into the issue
14:10:52 <mwhahaha> gfidente: ok so we'll give it one more day
14:10:54 <fultonj> arxcruz: do you think you could look into my question about adding a wait and retry?
14:10:57 <fultonj> thanks gfidente
14:11:01 <gfidente> mwhahaha ack
14:11:07 <mwhahaha> but unless we can figure out what the cause is today, we need to switch it to non-voting tomorrow
14:11:16 <weshay> gfidente thanks
14:11:30 <adarazs|ruck> o/
14:11:34 <mwhahaha> anyway moving on
14:11:37 <mwhahaha> (gfidente) help review ceph/luminous submission (passed CI)
14:12:00 <gfidente> yeah this is basically a call for help with reviews as we submitted a few patches to use ceph luminous instead of jewel
14:12:04 <gfidente> upstream CI seems happy
14:12:07 <trown> gfidente fultonj could either of you look at a reproducer env for that job? I cant get deploy to work because an issue with ceph-mon var/run file
14:12:17 <mwhahaha> gfidente: yea but we can't merge due to scenario001 flakyness
14:12:42 <gfidente> mwhahaha fwiw it passed https://review.openstack.org/#/c/510108/
14:12:45 <mwhahaha> i'm not against the switch as you're right ci is happy but we won't merge until we resolve the previous topic
14:12:55 <mwhahaha> gfidente: yea i know the problem is that it could fail in the gate repeatedly
14:13:02 <gfidente> mwhahaha understood
14:13:03 <mwhahaha> hence previous topic
14:13:07 <gfidente> though reviews still helpful
14:13:11 <mwhahaha> yup
14:13:34 <Tengu> hello ! small question: are there some issues with cinder on the CI for stable/pike, especially for the gate-tripleo-ci-centos-7-containers-multinode-upgrades-pike running on "RDO Third Party CI" ?
14:13:50 <mwhahaha> Tengu: we can look into it after the moeeting
14:14:00 <Tengu> oh, sorry, didn't check the topic
14:14:14 <mwhahaha> gfidente: anything else about the ceph luminous switch?
14:14:54 <gfidente> mwhahaha nah, it should be work on upgrade too
14:15:00 <mwhahaha> ok sounds good
14:15:01 <gfidente> ceph-ansible has a playbook
14:15:02 <mwhahaha> (weshay) CI logging changes
14:15:03 <gfidente> so testabe
14:15:06 <gfidente> *testable
14:15:49 <weshay> mwhahaha, ya.. just a FYI for general awareness and reviews
14:16:09 <mwhahaha> #link https://review.openstack.org/#/c/511526/
14:16:18 <mwhahaha> weshay: does that improve the log capture time or anything?
14:16:53 <weshay> I need to check, I'll comment in the review
14:16:57 <mwhahaha> k
14:17:13 <mwhahaha> any other one off items folks would like to talk about?
14:17:29 <openstackgerrit> Anastasia Kravets proposed openstack/puppet-tripleo master: Merge from from Juniper repo for opencontrail  https://review.openstack.org/516651
14:17:34 <openstackgerrit> Anastasia Kravets proposed openstack/puppet-tripleo master: Merge from from Juniper repo for opencontrail  https://review.openstack.org/516651
14:17:49 <mwhahaha> sounds like nope
14:17:52 <mwhahaha> status time
14:18:00 <openstackgerrit> Carlos Camacho proposed openstack/python-tripleoclient master: Add openstack undercloud backup  https://review.openstack.org/466213
14:18:06 <mwhahaha> #topic Squad status
14:18:06 <mwhahaha> ci
14:18:06 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-ci-squad-scrum
14:18:06 <mwhahaha> upgrade
14:18:06 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-upgrade-squad-status
14:18:07 <mwhahaha> containers
14:18:07 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-containers-squad-status
14:18:08 <mwhahaha> integration
14:18:08 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-integration-squad-status
14:18:09 <mwhahaha> ui/cli
14:18:09 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-ui-cli-squad-status
14:18:10 <mwhahaha> validations
14:18:10 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-validations-squad-status
14:18:11 <mwhahaha> networking
14:18:11 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-networking-squad-status
14:18:12 <mwhahaha> workflows
14:19:00 <d0ugal> I'm in the process of setting up the Workflows meeting, finally :) - will send out a email to the list today with the first meeting details. No status otherwise.
14:19:10 <mwhahaha> weshay, ci status seems missing
14:19:10 <d0ugal> but there will be more soon!
14:19:22 <mwhahaha> validations also missing status
14:19:30 <mwhahaha> jrist: ui/cli status missing
14:19:31 <weshay> mwhahaha, we'll have a sprint end, and planning update today
14:19:34 <weshay> arxcruz, ^
14:19:43 <mwhahaha> k
14:19:57 <jrist> womp womp
14:20:14 <jrist> mwhahaha: I'll chat with jtomasek to see what the update was
14:20:20 <mwhahaha> jrist: thanks
14:20:46 <jtomasek> mwhahaha, jrist: I'll update it
14:21:15 <mwhahaha> moving on to bugs
14:21:19 <mwhahaha> #topic bugs & blueprints
14:21:19 <mwhahaha> #link https://launchpad.net/tripleo/+milestone/queens-2
14:21:19 <mwhahaha> For Queens we currently have 70 (+0) blueprints and about 526 (+5) open bugs. 251 queens-2 and 275 queens-3.
14:21:51 <mwhahaha> please take a look at the critical bug list, i reported some yesterday for some failures in CI that aren't blocking but should get resolved
14:22:16 <mwhahaha> for example, https://bugs.launchpad.net/tripleo/+bug/1732010
14:22:16 <openstack> Launchpad bug 1732010 in tripleo "CI: tripleoclient package build fails because unit tests fail because tmp file exists" [Critical,Triaged]
14:23:07 <mwhahaha> anyway have any other bugs they wish to bring up?
14:23:21 <mwhahaha> s/anyway/anyone
14:24:17 <openstackgerrit> Carlos Camacho proposed openstack/tripleo-common master: WIP add UC backup actions  https://review.openstack.org/517610
14:24:37 <mwhahaha> sounds like nope
14:24:38 <adarazs|ruck> nope. just fix the critical ones quickly please, we need promotions :)
14:24:49 <mwhahaha> indeed
14:25:14 <mwhahaha> #topic projects releases or stable backports
14:25:16 <jpich> Maybe also https://bugs.launchpad.net/tripleo/+bug/1732140 - a patch is merging that will fix it, tripleo-common master is broken until then
14:25:16 <openstack> Launchpad bug 1732140 in tripleo "Kolla tests failing for tripleo-common master" [Critical,Invalid] - Assigned to Adriano Petrich (apetrich)
14:26:17 <mwhahaha> jpich: thanks for pointing that out, looks like https://review.openstack.org/#/c/516136/ is in the gate so we'll keep an eye on it
14:26:43 <mwhahaha> as for releases, I think EmilienM was working on getting a pike release cut.
14:26:57 <mwhahaha> reminder that queens-m2 is in ~3 weeks
14:27:30 <weshay> dear lord
14:27:37 <mwhahaha> so please make sure your blueprints are updated with their current status and please raise any issues that might prevent you from getting your feature merged before m2 (other than the known scenario001 problems)
14:28:20 <mwhahaha> also along those lines
14:28:24 <mwhahaha> #topic specs
14:28:24 <mwhahaha> #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open
14:28:35 <mwhahaha> please review specs, they need to be merged before m2
14:29:34 <mwhahaha> #topic open discussion
14:29:38 <mwhahaha> any other business?
14:31:13 <trown> gfidente fultonj could either of you help me with some ceph-ansible questions?
14:31:24 <fultonj> trown: sure
14:31:58 <openstackgerrit> Anastasia Kravets proposed openstack/tripleo-heat-templates master: Merge from from Juniper THT for opencontrail  https://review.openstack.org/516630
14:32:02 <openstackgerrit> Liz Blanchard proposed openstack/tripleo-ui master: Add message to spinner on Deployment Config modal  https://review.openstack.org/519478
14:32:08 <trown> fultonj: sweet... I have tracked down why multinode ceph jobs do not reproduce on rdocloud to this line https://github.com/ceph/ceph-ansible/blob/master/roles/ceph-mon/tasks/docker/main.yml#L10
14:33:00 <trown> fultonj: the "monitor_name" var there is resolving to "subnode-1", but the var run file is actually created as "*subnode-1.rdocloud.asok"
14:33:13 <mwhahaha> ok well sounds like no further items for the meeting
14:33:15 <mwhahaha> #endmeeting