14:08:09 <jaosorior> #startmeeting tripleo
14:08:10 <openstack> Meeting started Tue Nov  6 14:08:09 2018 UTC and is due to finish in 60 minutes.  The chair is jaosorior. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:08:11 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:08:11 <jaosorior> #topic agenda
14:08:13 <openstack> The meeting name has been set to 'tripleo'
14:08:13 <jaosorior> * Review past action items
14:08:15 <jaosorior> * One off agenda items
14:08:17 <jaosorior> * Squad status
14:08:19 <jaosorior> * Bugs & Blueprints
14:08:21 <jaosorior> * Projects releases or stable backports
14:08:23 <jaosorior> * Specs
14:08:25 <jaosorior> * open discussion
14:08:27 <jaosorior> Anyone can use the #link, #action and #info commands, not just the moderatorǃ
14:08:43 <jrist> o/
14:08:51 <weshay> \0
14:08:54 <fultonj> o/
14:08:57 <d0ugal> o/
14:09:06 <rlandy> o/
14:09:14 <ksambor> o/
14:09:27 <arxcruz> o/
14:09:29 <shardy> o/
14:09:36 <hjensas> o/
14:09:56 <mwhahaha> hi2u
14:09:57 <marios> o/
14:10:17 <ooolpbot> URGENT TRIPLEO TASKS NEED ATTENTION
14:10:18 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1801769
14:10:18 <openstack> Launchpad bug 1801769 in tripleo "FailedValidation: Hostname is not fully qualified, fails undercloud install" [Critical,Triaged]
14:11:29 <jaosorior> #topic review past action items
14:11:41 <jaosorior> There was one:
14:11:43 <jaosorior> jaosorior to poke the squads and check status. Based on this we might need to update the list of active squads.
14:12:00 <jaosorior> I didn poke all the squads, so, I'll leave this action active until I do.
14:12:05 <jaosorior> at least we did get some updates on some of them
14:12:49 <jaosorior> #topic one off agenda items
14:12:50 <jtomasek> o/
14:13:01 <jaosorior> First of them: (sshnaidm) New workflow for having master containers updated with tripleo packages from delorean-current (consistent) repo
14:13:13 <quiquell> o/
14:13:17 <sshnaidm|rover> o/
14:13:27 <dpeacock> oh crap this is at 09:00 now
14:13:28 <honza> o/
14:13:29 <dpeacock> o/
14:13:32 <matbu> o/
14:13:34 <chem> o/
14:13:44 <jrist> jaosorior: jtomasek or I will update squad for UI. still active, but busy with releasing
14:13:51 <jaosorior> jrist: thanks!
14:14:33 <trozet> weshay: is there anyway you can restore that queens undercloud.qcow2? our opnfv release is Friday
14:14:49 <sshnaidm|rover> jaosorior, so we have delorean-current repos for tripleo-* packages that we use in our CI jobs, it's the newest packages. And I suppose we need to have the same in containers to avoid all these long updates
14:14:58 <gfidente> o/
14:15:10 <trozet> weshay: its fine to not have it for rocky and later, but since undercloud isnt containerized for our queens deployments we kind of rely on that image
14:15:11 <owalsh> o/
14:15:37 <rfolco|ruck> o/
14:15:53 <sshnaidm|rover> jaosorior, so after a time period or after every tripleo-* repo package is built we need to update all our containers with it and to publish on docker.
14:16:48 <mwhahaha> sshnaidm|rover: we might be able to leverage the container CI job I'm working on to do a periodic update
14:17:06 <sshnaidm|rover> mwhahaha, periodic update to all containers?
14:17:54 <mwhahaha> #link https://review.openstack.org/#/c/614392/
14:18:01 <weshay> trozet, see pm #tripleo meeting atm man
14:18:03 <sshnaidm|rover> mwhahaha, as we saw in jobs the most time is wasted on containers updates from delorean-current repo, it's about 40 min - 1 hour, which causes timeouts
14:18:06 <mwhahaha> sshnaidm|rover: yea
14:18:43 <sshnaidm|rover> mwhahaha, so if it's periodic update of all containers - it's actually what I propose :)
14:19:00 <mwhahaha> sshnaidm|rover: it currently isnt but could be adopted to support that
14:19:22 <mwhahaha> sshnaidm|rover: btw container builds are broken due to grafana
14:19:28 <mwhahaha> It looks like
14:19:33 * mwhahaha afk for a bit
14:19:37 <sshnaidm|rover> mwhahaha, not sure I understand
14:19:49 <sshnaidm|rover> mwhahaha, ok, we can take it offline and discuss later
14:21:08 <sshnaidm|rover> that's it for this point from me, if anyone has comments..
14:21:21 <jaosorior> sshnaidm|rover: is there any work you hve done to enable this? or were you looking for input first?
14:21:58 <sshnaidm|rover> jaosorior, not done yet, in planning stage
14:22:08 <sshnaidm|rover> and looking for input, yes
14:22:13 <hjensas> how much time do we spend on updating containers when packages are up-to-date in the containers?
14:22:26 <openstackgerrit> Emilien Macchi proposed openstack/tripleo-heat-templates master: (Partial) Revert "Use container_cli for post_upgrade_tasks & external_upgrade_tasks"  https://review.openstack.org/615891
14:22:34 <hkominos> fultonj: ping
14:22:34 <sshnaidm|rover> hjensas, it will take time only to install a built package from patch
14:22:59 <sshnaidm|rover> hjensas, I think overall it sholdn't take more than 10 mins
14:23:27 <fultonj> hkominos: there's a tripleo meeting happening now; can you ping me after?
14:23:49 <hjensas> sshnaidm|rover: ok, thanks.
14:23:52 <sshnaidm|rover> hjensas, we can see it right after a promotion when all packages are up to date
14:24:15 <hkominos> fultonj: sure :) take care
14:24:28 <jaosorior> sshnaidm|rover: I think it makes sense, though I guess first you wanna sync with mwhahaha to get more input, right?
14:24:39 <sshnaidm|rover> jaosorior, yep
14:24:54 <sshnaidm|rover> jaosorior, need to sync if he's working on this too
14:25:13 <jaosorior> got it
14:25:27 <jaosorior> so, the next topic is:     (sshnaidm) OVB check jobs turning off in Wed
14:25:46 <sshnaidm|rover> ok, so we actually decided to turn them off even today
14:25:52 <sshnaidm|rover> #link https://review.rdoproject.org/r/#/c/17272/
14:26:05 <sshnaidm|rover> we turn off check jobs for patches
14:26:07 <openstackgerrit> Quique Llorente proposed openstack/tripleo-quickstart-extras master: DNM: Squash for fedora28 job  https://review.openstack.org/615532
14:26:15 <sshnaidm|rover> periodic jobs will still run
14:26:22 <dpeacock> :q!
14:26:35 <sshnaidm|rover> and if you want to run ovb jobs on your patch, you can run them by comment "check-rdo"
14:26:38 <openstackgerrit> Quique Llorente proposed openstack-infra/tripleo-ci master: DNM: To test standalone-28  https://review.openstack.org/615479
14:26:45 <sshnaidm|rover> but nothing will run automatically
14:26:54 <jaosorior> sshnaidm|rover: what's the background on that?
14:27:25 <weshay> jaosorior, the background reason why we need to shut off 3rd party check?
14:27:27 <sshnaidm|rover> jaosorior, RDO cloud is not stable for last months
14:27:43 <jaosorior> right
14:28:15 <weshay> jaosorior, the pass rate is roughly 15%, lots of broken glass not providing value. The root cause seems to be networking and the load on rdo-cloud atm
14:28:21 <jaosorior> thought the instability would be addressed, and not that we would be moving away from testing our patches there.
14:28:39 <openstackgerrit> Emilien Macchi proposed openstack/tripleo-heat-templates master: Rework neutron/own agent wrapper tools for podman  https://review.openstack.org/606975
14:28:47 <weshay> so we're hoping to at least get full promotions running, when 50 red / broken check jobs are not currently running
14:29:01 <weshay> jaosorior, yes.. I also thought this would be addressed by now
14:29:26 <openstackgerrit> Emilien Macchi proposed openstack/tripleo-heat-templates master: Rework neutron/own agent wrapper tools for podman  https://review.openstack.org/606975
14:29:27 <jaosorior> when/if these issues in RDO cloud get addressed, is the plan to turn the checks back on?
14:29:40 <weshay> jaosorior, yes, of course..
14:29:50 <weshay> and it could be we use another cloud for some time.. not sure
14:30:15 <weshay> jaosorior, I am trying coordinate a cross team response to the issue here
14:30:16 <jaosorior> alright
14:30:30 <jaosorior> weshay: any way I could help there?
14:30:34 <weshay> https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https
14:30:36 <weshay> ://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tr
14:30:36 <weshay> ee.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.ta
14:30:40 <weshay> iga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.i
14:30:48 <weshay> o/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/pro
14:30:52 <weshay> ject/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/
14:30:57 <weshay> tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripl
14:31:01 <openstackgerrit> Quique Llorente proposed openstack-infra/tripleo-ci master: DNM: To test standalone-28  https://review.openstack.org/615479
14:31:06 <weshay> eo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci
14:31:11 <weshay> -board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-boar
14:31:13 <jaosorior> weshay: none of the links worked for me :/
14:31:18 <weshay> d/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298https://tree.taiga.io/project/tripleo-ci-board/epic/298vvvvvvvvvvv
14:31:23 <weshay> oh wow
14:31:25 <weshay> lolz.. my laptop just had a brain fart
14:31:33 <weshay> https://tree.taiga.io/project/tripleo-ci-board/epic/298
14:31:45 <weshay> jaosorior, ur on the epic
14:31:57 <jaosorior> ohoo, THAT epic :D
14:31:57 <weshay> also escalated https://trello.com/c/2k9DDzpX/778-cixlp1797526tripleociproa-bring-ovb-jobs-back-to-passing-at-75
14:32:14 <weshay> jaosorior, the communication on this has been very odd.. folks were trying to move that issue to preventative action
14:32:46 <sshnaidm|rover> jaosorior, just how looks rdo cloud ci: http://dashboard-ci.tripleo.org/dashboard/snapshot/spgOhTI6f9QG3CyyRu38rzfpIAg48LZE?orgId=1&from=now-90d&to=now
14:32:52 <jaosorior> weshay: lets catch up on this after this meeting
14:32:58 <weshay> jaosorior, mwhahaha and I should coordinate prior to a prod-chain infra meeting
14:33:00 <weshay> jaosorior, k
14:33:11 <jaosorior> thanks for the info
14:33:15 <jaosorior> sshnaidm|rover: ouch
14:33:33 <sshnaidm|rover> yep
14:34:11 <jaosorior> So, the next topic is mine
14:34:16 <jaosorior> * No meeting next week
14:34:32 <jaosorior> So, a bunch of folks will be in Berlin in the Summit next week, so will I
14:34:53 <jaosorior> so, we won't have a meeting next week
14:35:16 <jaosorior> so, if you're going, hope to see you folks there :D
14:35:48 <jaosorior> that's all for the one-off agenda items
14:36:25 <jaosorior> #topic Squad status
14:36:27 <jaosorior> ci
14:36:29 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-ci-squad-meeting
14:36:31 <jaosorior> upgrade
14:36:33 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-upgrade-squad-status
14:36:35 <jaosorior> containers
14:36:37 <jaosorior> #link https://trello.com/b/S8TmOU0u/tripleo-podman
14:36:39 <jaosorior> integration
14:36:41 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-integration-squad-status
14:36:43 <jaosorior> ui/cli
14:36:45 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-ui-cli-squad-status
14:36:47 <jaosorior> validations
14:36:49 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-validations-squad-status
14:36:51 <jaosorior> networking
14:36:53 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-networking-squad-status
14:36:55 <jaosorior> security
14:36:57 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-security-squad
14:36:59 <jaosorior> edge
14:37:01 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-edge-squad-status
14:37:14 <openstackgerrit> Emilien Macchi proposed openstack/tripleo-heat-templates master: Rework neutron/own agent wrapper tools for podman  https://review.openstack.org/606975
14:37:32 <jaosorior> Any squad wanting to bring up their status, or a topic for the general public?
14:38:14 <openstackgerrit> Emilien Macchi proposed openstack/tripleo-heat-templates master: Allow the environment to enable Podman on the overcloud  https://review.openstack.org/612522
14:38:19 <openstackgerrit> Merged openstack-infra/tripleo-ci master: turn on tcpdump debug by default for ovb jobs  https://review.openstack.org/610072
14:38:20 <openstackgerrit> Merged openstack-infra/tripleo-ci master: Collect atop statistics  https://review.openstack.org/615515
14:39:05 <EmilienM> o/
14:39:16 <jaosorior> \o
14:39:54 <jaosorior> #topic bugs & blueprints
14:39:56 <jaosorior> #link https://launchpad.net/tripleo/+milestone/stein-1
14:39:58 <jaosorior> For Stein we currently have 28 (for stein-1) and 4 (for stein-2) blueprints open in Launchpad.
14:40:01 <jaosorior> Bugs: 757 (-4) stein-2, 1 stein-3.  102 (-1) open Storyboard bugs.
14:40:02 <openstack> bug 757 in Launchpad itself "Assignee should be CC'd when assigneed to a private bug" [Medium,Invalid] https://launchpad.net/bugs/757
14:40:02 <jaosorior> #link https://storyboard.openstack.org/#!/project_group/76
14:40:36 <EmilienM> lol the bot
14:41:33 <jaosorior> #topic projects releases or stable backports
14:42:47 <openstackgerrit> Emilien Macchi proposed openstack/tripleo-heat-templates master: Set setype for log and persistant data directories  https://review.openstack.org/611801
14:42:49 <jaosorior> nothing here on my side
14:42:58 <jaosorior> #topic specs
14:43:00 <jaosorior> #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open
14:43:09 <jaosorior> Any specs that folks would like to bring up here?
14:44:12 <hjensas> #link https://review.openstack.org/451584
14:44:39 <jaosorior> I'll check it out, thanks hjensas !
14:44:51 <hjensas> Most of the code to implement is up for review ... so would be good to get that done. :)
14:45:21 <openstackgerrit> Emilien Macchi proposed openstack/tripleo-heat-templates master: Reimplement HA docker tagging  https://review.openstack.org/614827
14:45:22 <openstackgerrit> Emilien Macchi proposed openstack/tripleo-heat-templates master: Support HA container tagging with podman  https://review.openstack.org/614830
14:45:23 <openstackgerrit> Emilien Macchi proposed openstack/tripleo-heat-templates master: Rework neutron/own agent wrapper tools for podman  https://review.openstack.org/606975
14:45:23 <openstackgerrit> Emilien Macchi proposed openstack/tripleo-heat-templates master: Set setype for log and persistant data directories  https://review.openstack.org/611801
14:45:23 <openstackgerrit> Emilien Macchi proposed openstack/tripleo-heat-templates master: Allow the environment to enable Podman on the overcloud  https://review.openstack.org/612522
14:46:32 <openstackgerrit> Emilien Macchi proposed openstack/tripleo-heat-templates master: Reimplement HA docker tagging  https://review.openstack.org/614827
14:46:41 <jaosorior> #topic open discussion
14:46:43 <jaosorior> Anything else that folks want to bring up to the meeting?
14:47:15 <honza> #link https://review.openstack.org/#/c/607323/
14:47:20 <openstackgerrit> Yurii Prokulevych proposed openstack/tripleo-upgrade master: Drop ssh-user from 'upgrade run' template.  https://review.openstack.org/615897
14:47:23 <openstackgerrit> Emilien Macchi proposed openstack/tripleo-quickstart master: WIP / fs010: enable Podman on the overcloud  https://review.openstack.org/612526
14:47:23 <honza> Need some eyes/help on that patch
14:48:42 <jaosorior> ack, added to my queue
14:48:59 <openstackgerrit> Emilien Macchi proposed openstack/tripleo-quickstart master: WIP / fs010: enable Podman on the overcloud  https://review.openstack.org/612526
14:49:16 <honza> <3
14:50:29 <jaosorior> alright! thanks for joining folks!
14:50:31 <jaosorior> #endmeeting