14:00:12 <mwhahaha> #startmeeting tripleo
14:00:12 <openstack> Meeting started Tue Sep 24 14:00:12 2019 UTC and is due to finish in 60 minutes.  The chair is mwhahaha. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:12 <mwhahaha> #topic agenda
14:00:12 <mwhahaha> * Review past action items
14:00:13 <mwhahaha> * One off agenda items
14:00:13 <mwhahaha> * Squad status
14:00:14 <mwhahaha> * Bugs & Blueprints
14:00:14 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:14 <mwhahaha> * Projects releases or stable backports
14:00:15 <mwhahaha> * Specs
14:00:15 <mwhahaha> * open discussion
14:00:16 <mwhahaha> Anyone can use the #link, #action and #info commands, not just the moderator«É
14:00:17 <openstack> The meeting name has been set to 'tripleo'
14:00:17 <mwhahaha> Hi everyone! who is around today?
14:00:34 <dpeacock> o/
14:00:43 <weshay> 0/
14:00:51 <ksambor> o/
14:00:51 <Tengu> \o/
14:00:52 <ekultails> o/
14:01:05 <rlandy> o/
14:01:26 <owalsh> o/
14:01:26 <marios> hello
14:01:59 <chandankumar> ykarel: which branch? master ?
14:02:01 <chandankumar> \o/
14:02:56 <rfolco> o/
14:02:59 <mschuppert> o/
14:03:08 <ykarel> chandankumar, yes
14:03:16 <ykarel> o/
14:03:45 <chandankumar> patch coming
14:04:18 <mwhahaha> alright let's get on with the meeting
14:04:23 <mwhahaha> #topic review past action items
14:04:23 <mwhahaha> mwhahaha to close out train-m3 - DONE
14:04:41 <mwhahaha> I went ahead and closed out the launchpad milestone and we release m3 last week
14:04:52 <mwhahaha> any questions on that?
14:05:38 <rh-jelabarre> o/
14:05:59 <mwhahaha> sounds like nope
14:06:02 <mwhahaha> sync with weshay on ptl things - NOT DONE
14:06:03 <mwhahaha> #action mwhahaha sync with weshay on ptl things
14:06:13 <mwhahaha> weshay: do you have time later today so we can get just mark this one done
14:06:15 <weshay> slackers
14:06:18 <weshay> yes
14:06:27 <mwhahaha> k i'll ping you a bit later
14:06:38 <mwhahaha> that's it on the past items
14:06:45 <mwhahaha> moving on to more exciting things
14:06:48 <mwhahaha> #topic one off agenda items
14:06:48 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-meeting-item
14:06:54 <mwhahaha> centos8/pcmk ; clues when it will be available? should ci switch back to pcmk when it becomes available? (gfidente/bandini)
14:07:18 <mwhahaha> IMHO the answer to pcmk would be yes
14:07:42 <mwhahaha> weshay, do we have any thoughts on the turn around time for centos8, is that a question for rdo as well?
14:08:12 <weshay> once we have a upstream nodepool image.. I'm hoping we'll see standalone running with in days.. and i'll provide an estimate then
14:08:19 <mwhahaha> k
14:08:31 <ykarel> RDO would need time to prepare
14:09:04 <bandini> mwhahaha: yes, once we can use centos8 we should switch back ci to pcmk
14:09:10 <bandini> and also work on ha by default
14:09:28 <mwhahaha> we'll also finally be able to switch to podman on ovb as well
14:09:32 <mwhahaha> and remove docker packaging
14:09:39 <mwhahaha> so there's a bunch pending centos8
14:09:51 <mwhahaha> hopefully we can switch early in the U cycle
14:09:51 <mjturek> rfolco: does this look good? ppc container builds are failing due to kolla issue and this is the fix I think https://review.opendev.org/#/c/683997/
14:10:15 <weshay> mjturek, in a mtg fyi
14:10:20 <rfolco> mjturek, lets chat on #oooq
14:10:27 <mjturek> whoops, I'm  sorry
14:10:29 <ykarel> yes hopefully for U cycle we will be able to go with CentOS
14:10:36 <bandini> aye
14:10:38 <ykarel> but more update will be there in coming days
14:10:45 <ykarel> once it's actually out
14:10:49 <mwhahaha> any other pacemaker/centos8 thoughts?
14:10:51 <gfidente> weshay mwhahaha yeah and both scenario004 and scenario001
14:11:26 <cloudnull> o/
14:11:35 <ykarel> also we would need other SIGs to be ready apart from cloud sig
14:11:45 <ykarel> like virt and ceph to be ready with centos8
14:12:32 <mwhahaha> #link http://mirror.centos.org/centos-8/
14:12:46 <mwhahaha> looks like we're still pending qcow cloud images, but at least a repo is available
14:13:46 <ykarel> regarding RDO and CentOS8, couple of info can be found at https://review.rdoproject.org/etherpad/p/moving-rdo-to-centos8
14:14:07 <mwhahaha> k
14:14:15 <mwhahaha> I think that's about it for now. centos8 is still a WIP
14:14:19 <mwhahaha> moving on
14:14:21 <mwhahaha> <weshay> quay registry
14:14:30 <weshay> ya...
14:14:55 <weshay> just a quick FYI.. we're exploring using quay ( key ) lolz.. in upstream.  patches are being tested now
14:15:06 <weshay> expect containers to live in both docker.io and quay for some time
14:15:27 <weshay> https://review.rdoproject.org/r/#/c/22445/ <- being tested
14:15:40 <weshay> that's it
14:16:12 <weshay> mwhahaha, ^
14:16:16 <mwhahaha> k
14:16:34 <mwhahaha> moving on
14:16:35 <weshay> next item is mine as well
14:16:38 <mwhahaha> <weshay> thoughts on a daily periodic job to update master containers w/ update_repo: "delorean-current,quickstart-centos-ceph-nautilus" ?
14:16:38 <mwhahaha> w/ standalone, or other job.. if it passes update containers in docker.io or quay
14:17:07 <weshay> ya.. I have arxcruz|rover and zbr|ruck looking at building a periodic (daily) job to update containers w/
14:17:15 <weshay> "delorean-current,quickstart-centos-ceph-nautilus"
14:17:26 <sshnaidm> it shouldn't even pass, just build and push
14:17:32 <mwhahaha> that doesnt' seem correct
14:17:38 <weshay> and push to docker.io / quay.. to shorten the loop w/ containers prep
14:17:50 <weshay> sshnaidm, k.. we'll consider that
14:17:50 <mwhahaha> shouldn't it match our includepkgs list (tripleo-repos current-tripleo-dev)
14:18:00 <sshnaidm> anyway we update to these repos containers in every job, and it's merged only if it's passed
14:18:02 <weshay> again.. we're seeing very long container prep times in CI
14:18:06 <weshay> causing failures
14:18:07 <mwhahaha> which is the equivalent of CI
14:18:23 <mwhahaha> when was the last promotion?
14:18:23 <weshay> some examples can be found here:  https://snapshot.raintank.io/dashboard/snapshot/SjZ16RtMXZCFbEIOjgZ7N7KKJvOpKBhB?orgId=2
14:18:38 <weshay> master is at 4 days
14:18:44 <bogdando> weshay: yes, it's about ~1h++ for each job
14:18:47 <mwhahaha> hmm k
14:18:58 <weshay> you can see.. we have infra issues w/ ovh and fortnebula
14:18:59 <bogdando> only to update the containers with modify image role
14:19:05 <weshay> most of the infra guys are out atm.. at ansible-fest
14:19:29 <weshay> in rdo periodic jobs w/o yum updates.. containers prep is taking 5-6 minutes
14:19:46 <ykarel> really?
14:19:59 <weshay> so we're still exploring ways to get our job times back down to a reasonable time... updates will come
14:20:13 <weshay> ykarel,  aye
14:20:29 <mwhahaha> k
14:20:31 <ykarel> okk nice, any link to logs
14:20:34 <weshay> http://dashboard-ci.tripleo.org/d/si1tipHZk/jobs-exploration?orgId=1&from=now-12h&to=now&var-influxdb_filter=branch%7C%3D%7Cmaster
14:20:42 <weshay> ykarel,  look for the rdo provider
14:21:10 <weshay> hrm.. let me get you a better link
14:21:11 <sshnaidm> container build takes about 15 minutes? maybe worth just to build them and not to update? :)
14:21:26 <ykarel> i see 25,21, 15 minutes
14:21:43 <mwhahaha> you mean building them in CI?
14:21:48 <sshnaidm> yeah
14:22:02 <mwhahaha> i think it'll probably take longer than that on some of these clouds
14:22:31 <mwhahaha> though maybe we would get better cache results for packages than containers
14:22:34 <mwhahaha> we could try it
14:22:38 <weshay> ykarel, use https://snapshot.raintank.io/dashboard/snapshot/S9m9pvyn4iPSELe0bd566ZlMv4KvVWwO?orgId=2
14:22:45 <mwhahaha> but it requires an undercloud to build
14:22:50 <mwhahaha> because $reasons
14:22:58 <mwhahaha> i'm not sure it'd be a simple switch
14:23:51 <weshay> http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario001-standalone-master/7b77387/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz
14:24:03 <bogdando> ykarel: just checked, http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario001-standalone-master/7b77387/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz is exactly 5 minutes
14:24:21 <ykarel> rhel8
14:24:32 <bogdando> ykarel: 2019-09-24 01:32:26 - 2019-09-24 01:37:21
14:24:33 <weshay> doesn't matter
14:25:03 <mwhahaha> anyway sounds like we need to continue investigating our options around containers
14:25:17 <mwhahaha> anythign else?
14:25:25 <weshay> nope
14:25:30 <weshay> aye
14:25:31 <mwhahaha> k moving on
14:25:34 <mwhahaha> #topic Squad status
14:25:34 <mwhahaha> ci
14:25:34 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-ci-squad-meeting
14:25:34 <mwhahaha> upgrade
14:25:34 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-upgrade-squad-status
14:25:35 <mwhahaha> edge
14:25:35 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-edge-squad-status
14:25:36 <mwhahaha> integration
14:25:36 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-integration-squad-status
14:25:37 <mwhahaha> validations
14:25:37 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-validations-squad-status
14:25:38 <mwhahaha> networking
14:25:38 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-networking-squad-status
14:25:39 <mwhahaha> transformation
14:25:39 <mwhahaha> #link https://etherpad.openstack.org/p/tripleo-ansible-agenda
14:25:40 <mwhahaha> any status highlights?
14:25:43 <ykarel> weshay, i see http://dashboard-ci.tripleo.org/d/si1tipHZk/jobs-exploration?orgId=1&from=now-12h&to=now&var-influxdb_filter=branch%7C%3D%7Cmaster&var-influxdb_filter=provider%7C%3D%7Crdo-cloud and in this i see 25,21,15
14:25:48 <ykarel> the other link seems not working
14:26:03 <weshay> ykarel, k.. let's go out of channel
14:26:08 <ykarel> ack
14:27:00 <bogdando> ykarel: but that was prolly a bad example, there was nothing to update at all :)
14:27:03 * ykarel checked what's with rhel8, there is dnf issue, so it didn't run completely
14:27:17 <weshay> there are never container updates in periodic..
14:28:52 <openstackgerrit> Karthik S proposed openstack/tripleo-heat-templates master: Fixed the path for neutron-base.yaml  https://review.opendev.org/684318
14:30:29 <mwhahaha> ok moving on
14:30:34 <mwhahaha> #topic bugs & blueprints
14:30:34 <mwhahaha> #link https://launchpad.net/tripleo/+milestone/train-rc1
14:30:34 <mwhahaha> For Train we currently have 27 blueprints and 521 (+15) open Launchpad Bugs. 28 train-rc1, 493 ussuri-1.  91 (-75) open Storyboard bugs.
14:30:34 <mwhahaha> #link https://storyboard.openstack.org/#!/project_group/76
14:30:46 <mwhahaha> I cleaned up the tripleo-ui story board items last week. Additionally I closed out milestone 3.  I will review the open rc1 bugs to move them to ussuri-1 if they aren't really blocking the rc1.
14:30:46 <mwhahaha> #action mwhahaha review train-rc1 open bugs
14:31:13 <mwhahaha> any questions/concerns around bugs/blueprints?
14:32:19 <apetrich> amoralej, so I've narrowed down the issue to this https://i.imgflip.com/3bex5p.jpg it is a bit weird. I'm trying to understand what causes that
14:33:02 <mwhahaha> sounds like that's in reference to https://bugs.launchpad.net/tripleo/+bug/1845181
14:33:02 <openstack> Launchpad bug 1845181 in tripleo "Problem with tripleo_common git functionalities and eventlet 0.25.1" [Medium,In progress] - Assigned to Adriano Petrich (apetrich)
14:33:38 <mwhahaha> also an appropriate use of memes
14:33:42 <mwhahaha> anything else?
14:33:42 <apetrich> mwhahaha, it is.
14:34:00 <apetrich> mwhahaha, so tempted to add that to the bug
14:34:49 <openstackgerrit> Merged openstack/tripleo-heat-templates stable/stein: Enable deep_compare by default for stonith resources  https://review.opendev.org/683409
14:34:50 <openstackgerrit> Merged openstack/tripleo-heat-templates stable/rocky: Enable deep_compare by default for stonith resources  https://review.opendev.org/683404
14:34:52 <openstackgerrit> Merged openstack/tripleo-heat-templates stable/queens: Enable deep_compare by default for stonith resources  https://review.opendev.org/683405
14:34:54 <openstackgerrit> Merged openstack/tripleo-validations master: Update the constraints url  https://review.opendev.org/684161
14:34:56 <openstackgerrit> Merged openstack/tripleo-upgrade stable/stein: Fix "unbound" variable  https://review.opendev.org/684191
14:35:00 <weshay> \0/
14:35:10 <mwhahaha> alright moving on
14:35:10 <mwhahaha> #topic projects releases or stable backports
14:35:18 <mwhahaha> any release related topics?
14:37:37 <mwhahaha> moving on
14:37:40 <mwhahaha> #topic specs
14:37:40 <mwhahaha> #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open
14:37:54 <mwhahaha> the U cycle will be starting soon, don't forget to update your specs
14:38:10 <mwhahaha> any spec related items?
14:40:23 <mwhahaha> #topic open discussion
14:40:26 <mwhahaha> anything else?
14:43:01 <mwhahaha> alright thanks everyone
14:43:03 <mwhahaha> #endmeeting