14:00:12 #startmeeting tripleo 14:00:12 Meeting started Tue Sep 24 14:00:12 2019 UTC and is due to finish in 60 minutes. The chair is mwhahaha. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:00:12 #topic agenda 14:00:12 * Review past action items 14:00:13 * One off agenda items 14:00:13 * Squad status 14:00:14 * Bugs & Blueprints 14:00:14 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:00:14 * Projects releases or stable backports 14:00:15 * Specs 14:00:15 * open discussion 14:00:16 Anyone can use the #link, #action and #info commands, not just the moderatorǃ 14:00:17 The meeting name has been set to 'tripleo' 14:00:17 Hi everyone! who is around today? 14:00:34 o/ 14:00:43 0/ 14:00:51 o/ 14:00:51 \o/ 14:00:52 o/ 14:01:05 o/ 14:01:26 o/ 14:01:26 hello 14:01:59 ykarel: which branch? master ? 14:02:01 \o/ 14:02:56 o/ 14:02:59 o/ 14:03:08 chandankumar, yes 14:03:16 o/ 14:03:45 patch coming 14:04:18 alright let's get on with the meeting 14:04:23 #topic review past action items 14:04:23 mwhahaha to close out train-m3 - DONE 14:04:41 I went ahead and closed out the launchpad milestone and we release m3 last week 14:04:52 any questions on that? 14:05:38 o/ 14:05:59 sounds like nope 14:06:02 sync with weshay on ptl things - NOT DONE 14:06:03 #action mwhahaha sync with weshay on ptl things 14:06:13 weshay: do you have time later today so we can get just mark this one done 14:06:15 slackers 14:06:18 yes 14:06:27 k i'll ping you a bit later 14:06:38 that's it on the past items 14:06:45 moving on to more exciting things 14:06:48 #topic one off agenda items 14:06:48 #link https://etherpad.openstack.org/p/tripleo-meeting-item 14:06:54 centos8/pcmk ; clues when it will be available? should ci switch back to pcmk when it becomes available? (gfidente/bandini) 14:07:18 IMHO the answer to pcmk would be yes 14:07:42 weshay, do we have any thoughts on the turn around time for centos8, is that a question for rdo as well? 14:08:12 once we have a upstream nodepool image.. I'm hoping we'll see standalone running with in days.. and i'll provide an estimate then 14:08:19 k 14:08:31 RDO would need time to prepare 14:09:04 mwhahaha: yes, once we can use centos8 we should switch back ci to pcmk 14:09:10 and also work on ha by default 14:09:28 we'll also finally be able to switch to podman on ovb as well 14:09:32 and remove docker packaging 14:09:39 so there's a bunch pending centos8 14:09:51 hopefully we can switch early in the U cycle 14:09:51 rfolco: does this look good? ppc container builds are failing due to kolla issue and this is the fix I think https://review.opendev.org/#/c/683997/ 14:10:15 mjturek, in a mtg fyi 14:10:20 mjturek, lets chat on #oooq 14:10:27 whoops, I'm sorry 14:10:29 yes hopefully for U cycle we will be able to go with CentOS 14:10:36 aye 14:10:38 but more update will be there in coming days 14:10:45 once it's actually out 14:10:49 any other pacemaker/centos8 thoughts? 14:10:51 weshay mwhahaha yeah and both scenario004 and scenario001 14:11:26 o/ 14:11:35 also we would need other SIGs to be ready apart from cloud sig 14:11:45 like virt and ceph to be ready with centos8 14:12:32 #link http://mirror.centos.org/centos-8/ 14:12:46 looks like we're still pending qcow cloud images, but at least a repo is available 14:13:46 regarding RDO and CentOS8, couple of info can be found at https://review.rdoproject.org/etherpad/p/moving-rdo-to-centos8 14:14:07 k 14:14:15 I think that's about it for now. centos8 is still a WIP 14:14:19 moving on 14:14:21 quay registry 14:14:30 ya... 14:14:55 just a quick FYI.. we're exploring using quay ( key ) lolz.. in upstream. patches are being tested now 14:15:06 expect containers to live in both docker.io and quay for some time 14:15:27 https://review.rdoproject.org/r/#/c/22445/ <- being tested 14:15:40 that's it 14:16:12 mwhahaha, ^ 14:16:16 k 14:16:34 moving on 14:16:35 next item is mine as well 14:16:38 thoughts on a daily periodic job to update master containers w/ update_repo: "delorean-current,quickstart-centos-ceph-nautilus" ? 14:16:38 w/ standalone, or other job.. if it passes update containers in docker.io or quay 14:17:07 ya.. I have arxcruz|rover and zbr|ruck looking at building a periodic (daily) job to update containers w/ 14:17:15 "delorean-current,quickstart-centos-ceph-nautilus" 14:17:26 it shouldn't even pass, just build and push 14:17:32 that doesnt' seem correct 14:17:38 and push to docker.io / quay.. to shorten the loop w/ containers prep 14:17:50 sshnaidm, k.. we'll consider that 14:17:50 shouldn't it match our includepkgs list (tripleo-repos current-tripleo-dev) 14:18:00 anyway we update to these repos containers in every job, and it's merged only if it's passed 14:18:02 again.. we're seeing very long container prep times in CI 14:18:06 causing failures 14:18:07 which is the equivalent of CI 14:18:23 when was the last promotion? 14:18:23 some examples can be found here: https://snapshot.raintank.io/dashboard/snapshot/SjZ16RtMXZCFbEIOjgZ7N7KKJvOpKBhB?orgId=2 14:18:38 master is at 4 days 14:18:44 weshay: yes, it's about ~1h++ for each job 14:18:47 hmm k 14:18:58 you can see.. we have infra issues w/ ovh and fortnebula 14:18:59 only to update the containers with modify image role 14:19:05 most of the infra guys are out atm.. at ansible-fest 14:19:29 in rdo periodic jobs w/o yum updates.. containers prep is taking 5-6 minutes 14:19:46 really? 14:19:59 so we're still exploring ways to get our job times back down to a reasonable time... updates will come 14:20:13 ykarel, aye 14:20:29 k 14:20:31 okk nice, any link to logs 14:20:34 http://dashboard-ci.tripleo.org/d/si1tipHZk/jobs-exploration?orgId=1&from=now-12h&to=now&var-influxdb_filter=branch%7C%3D%7Cmaster 14:20:42 ykarel, look for the rdo provider 14:21:10 hrm.. let me get you a better link 14:21:11 container build takes about 15 minutes? maybe worth just to build them and not to update? :) 14:21:26 i see 25,21, 15 minutes 14:21:43 you mean building them in CI? 14:21:48 yeah 14:22:02 i think it'll probably take longer than that on some of these clouds 14:22:31 though maybe we would get better cache results for packages than containers 14:22:34 we could try it 14:22:38 ykarel, use https://snapshot.raintank.io/dashboard/snapshot/S9m9pvyn4iPSELe0bd566ZlMv4KvVWwO?orgId=2 14:22:45 but it requires an undercloud to build 14:22:50 because $reasons 14:22:58 i'm not sure it'd be a simple switch 14:23:51 http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario001-standalone-master/7b77387/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz 14:24:03 ykarel: just checked, http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario001-standalone-master/7b77387/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz is exactly 5 minutes 14:24:21 rhel8 14:24:32 ykarel: 2019-09-24 01:32:26 - 2019-09-24 01:37:21 14:24:33 doesn't matter 14:25:03 anyway sounds like we need to continue investigating our options around containers 14:25:17 anythign else? 14:25:25 nope 14:25:30 aye 14:25:31 k moving on 14:25:34 #topic Squad status 14:25:34 ci 14:25:34 #link https://etherpad.openstack.org/p/tripleo-ci-squad-meeting 14:25:34 upgrade 14:25:34 #link https://etherpad.openstack.org/p/tripleo-upgrade-squad-status 14:25:35 edge 14:25:35 #link https://etherpad.openstack.org/p/tripleo-edge-squad-status 14:25:36 integration 14:25:36 #link https://etherpad.openstack.org/p/tripleo-integration-squad-status 14:25:37 validations 14:25:37 #link https://etherpad.openstack.org/p/tripleo-validations-squad-status 14:25:38 networking 14:25:38 #link https://etherpad.openstack.org/p/tripleo-networking-squad-status 14:25:39 transformation 14:25:39 #link https://etherpad.openstack.org/p/tripleo-ansible-agenda 14:25:40 any status highlights? 14:25:43 weshay, i see http://dashboard-ci.tripleo.org/d/si1tipHZk/jobs-exploration?orgId=1&from=now-12h&to=now&var-influxdb_filter=branch%7C%3D%7Cmaster&var-influxdb_filter=provider%7C%3D%7Crdo-cloud and in this i see 25,21,15 14:25:48 the other link seems not working 14:26:03 ykarel, k.. let's go out of channel 14:26:08 ack 14:27:00 ykarel: but that was prolly a bad example, there was nothing to update at all :) 14:27:03 * ykarel checked what's with rhel8, there is dnf issue, so it didn't run completely 14:27:17 there are never container updates in periodic.. 14:28:52 Karthik S proposed openstack/tripleo-heat-templates master: Fixed the path for neutron-base.yaml https://review.opendev.org/684318 14:30:29 ok moving on 14:30:34 #topic bugs & blueprints 14:30:34 #link https://launchpad.net/tripleo/+milestone/train-rc1 14:30:34 For Train we currently have 27 blueprints and 521 (+15) open Launchpad Bugs. 28 train-rc1, 493 ussuri-1. 91 (-75) open Storyboard bugs. 14:30:34 #link https://storyboard.openstack.org/#!/project_group/76 14:30:46 I cleaned up the tripleo-ui story board items last week. Additionally I closed out milestone 3. I will review the open rc1 bugs to move them to ussuri-1 if they aren't really blocking the rc1. 14:30:46 #action mwhahaha review train-rc1 open bugs 14:31:13 any questions/concerns around bugs/blueprints? 14:32:19 amoralej, so I've narrowed down the issue to this https://i.imgflip.com/3bex5p.jpg it is a bit weird. I'm trying to understand what causes that 14:33:02 sounds like that's in reference to https://bugs.launchpad.net/tripleo/+bug/1845181 14:33:02 Launchpad bug 1845181 in tripleo "Problem with tripleo_common git functionalities and eventlet 0.25.1" [Medium,In progress] - Assigned to Adriano Petrich (apetrich) 14:33:38 also an appropriate use of memes 14:33:42 anything else? 14:33:42 mwhahaha, it is. 14:34:00 mwhahaha, so tempted to add that to the bug 14:34:49 Merged openstack/tripleo-heat-templates stable/stein: Enable deep_compare by default for stonith resources https://review.opendev.org/683409 14:34:50 Merged openstack/tripleo-heat-templates stable/rocky: Enable deep_compare by default for stonith resources https://review.opendev.org/683404 14:34:52 Merged openstack/tripleo-heat-templates stable/queens: Enable deep_compare by default for stonith resources https://review.opendev.org/683405 14:34:54 Merged openstack/tripleo-validations master: Update the constraints url https://review.opendev.org/684161 14:34:56 Merged openstack/tripleo-upgrade stable/stein: Fix "unbound" variable https://review.opendev.org/684191 14:35:00 \0/ 14:35:10 alright moving on 14:35:10 #topic projects releases or stable backports 14:35:18 any release related topics? 14:37:37 moving on 14:37:40 #topic specs 14:37:40 #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open 14:37:54 the U cycle will be starting soon, don't forget to update your specs 14:38:10 any spec related items? 14:40:23 #topic open discussion 14:40:26 anything else? 14:43:01 alright thanks everyone 14:43:03 #endmeeting