14:01:09 <jaosorior> #startmeeting TripleO
14:01:09 <openstack> Meeting started Tue Jan 22 14:01:09 2019 UTC and is due to finish in 60 minutes.  The chair is jaosorior. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:01:10 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:01:13 <openstack> The meeting name has been set to 'tripleo'
14:01:31 <jaosorior> #topic agenda
14:01:33 <jaosorior> * Review past action items
14:01:35 <jaosorior> * One off agenda items
14:01:37 <jaosorior> * Squad status
14:01:39 <jaosorior> * Bugs & Blueprints
14:01:41 <jaosorior> * Projects releases or stable backports
14:01:43 <jaosorior> * Specs
14:01:44 <EmilienM> o/
14:01:45 <jaosorior> * open discussion
14:01:47 <jaosorior> Anyone can use the #link, #action and #info commands, not just the moderatorǃ
14:01:49 <jaosorior> Hey folks! who's around?
14:01:55 <bandini> jaosorior: I'll try and check that lp bug later
14:01:58 <beagles> o/
14:02:07 <fultonj> o/
14:02:09 <jaosorior> bandini: thanks
14:02:15 <chandankumar> \o/
14:02:16 <bogdando> hi
14:02:18 <marios> o/
14:02:19 <matbu> o/
14:02:28 <chem> o/
14:02:59 <ksambor> o/
14:03:32 <d0ugal> o/
14:04:04 <Tengu> «o/
14:04:08 <jbadiapa> o/
14:04:27 <openstackgerrit> Matthias Runge proposed openstack/tripleo-heat-templates stable/rocky: Enable ovs-stats by default when using ovs  https://review.openstack.org/632471
14:04:40 <jaosorior> #topic review past action items
14:04:41 <jaosorior> None.
14:04:42 <holser_> o/
14:04:51 <jaosorior> #topic one off agenda items
14:04:53 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-meeting-items
14:05:02 <jaosorior> First topic:     (gcerami) make fedora 28 with centos 7 containers job voting
14:05:36 <jaosorior> panda: ^^
14:06:01 <toure> o/
14:07:42 <jaosorior> panda or anybody else from the CI team that can comment on that? ^^
14:07:47 <panda> yeah sorry
14:07:49 <panda> got sidetracked
14:08:14 <panda> so, the job has proven to have valuable informations on the interaction between tripleo  and python 3
14:08:17 <jrist> o/
14:08:29 <panda> we had some legit failure on the periodic runs
14:08:37 <panda> and we are fixing actual bugs,
14:08:59 <ccamacho> jelou!
14:09:07 <panda> so I would like to understand if it's ok for us to start blocking patchesd base on the result of that jo
14:09:18 <panda> b
14:10:01 <fultonj> panda: seems reasonable to me (fwiw)
14:10:04 <panda> we are talking about the fedora 28 job taht uses centos 7 containers
14:10:32 <panda> bugs like this for example https://bugs.launchpad.net/tripleo/+bug/1812837
14:10:33 <openstack> Launchpad bug 1812837 in tripleo "periodic fedora 28 job failing with "/bin/sh: line 1: exit: null: numeric argument required" in Run async deployment StandalonePostDeployment step" [Undecided,Triaged]
14:11:10 <ykarel> panda, but for catching these we need to run fedora jobs in distgit changes also ^^
14:11:47 <panda> ykarel: are we ready to do it ?
14:12:20 <ykarel> panda, we started with testing centos standalone, didn't checked it can cover fedora too yet
14:13:19 <jaosorior> panda: should we enable it for the distgit changes before enabling it for every patch here?
14:13:42 <ykarel> overall for catching other issues that happened recently voting is must in upstream, and for particual issues like 1812837 we need in distgit
14:13:44 <panda> we also hit this one https://bugs.launchpad.net/tripleo/+bug/1812632
14:13:46 <openstack> Launchpad bug 1812632 in tripleo "Overcloud nova compute docker command failing on nova_cellv2_discover_hosts" [Critical,Fix released] - Assigned to Martin Schuppert (mschuppert)
14:13:55 <ykarel> yes mentioning ^^ only
14:15:08 <panda> ykarel: what you think, can this involves distgit too ? ^
14:15:09 <ykarel> breakages in distgit changes are rare as compared to project changes
14:15:50 <ykarel> panda, 181263 is caused by tht change
14:16:01 <ykarel> fedora was non-voting, but no one noticed
14:16:07 <panda> ykarel: ok, so we need the job voting there
14:16:11 <ykarel> even that patch caused fs035 also
14:16:37 <ykarel> panda, yes, also it's good if reviewers not ignore non-voting jobs
14:16:43 <ykarel> results
14:16:48 <jaosorior> tox
14:16:58 <panda> jaosorior: so yeah, not only distgit
14:17:15 <panda> jaosorior: ERROR: toxini file 'tox.ini' not found
14:17:17 <jaosorior> panda: if we have started catching bugs with it. And we have the capacity, then yeah
14:17:22 <jaosorior> panda: lol
14:17:34 <jaosorior> panda: do we have the capacity to add that job?
14:17:49 <panda> jaosorior: it's standalone, we'll make the capacity
14:17:57 <ssbarnea|bkp2> jaosorior: i think that if we make f28 voting we will much better, as we already have to take care of these jobs. it would save time
14:18:25 <jaosorior> understood
14:18:45 <jaosorior> panda: the job is passing right now, right?
14:19:17 <panda> jaosorior: no, but it hits legit bugs.
14:19:46 <jaosorior> panda: the suggestion is still to add it as non-voting initially, right? and then move it to voting?
14:20:40 <panda> jaosorior: the job is already there non-voting, it has been for at least a month. It was quite resilient even if the hashes between fedora and centos repos where not aligned
14:21:10 <panda> jaosorior: no it's in periodic too, and we are basing promotion on that result
14:21:16 <panda> jaosorior: on both fedora and centos side
14:22:44 <jaosorior> alright, if the job has been stable, it's working, and it's legitimally catching bugs; lets block based on it.
14:23:26 <panda> thanks. We are available in the community meeting to discuss eventual details
14:23:39 <jaosorior> ack
14:23:44 <jaosorior> thanks for bringing it up panda
14:24:02 <jaosorior> Next topic:     (chkumar) We are now able to run os_tempest in standalone job (patches in review)
14:24:31 <jaosorior> #link http://logs.openstack.org/00/627500/65/check/tripleo-ci-centos-7-standalone-os-tempest/198ae77/logs/undercloud/var/log/tempest/stestr_results.html.gz
14:24:45 <jaosorior> chandankumar: ^^
14:24:48 <chandankumar> Hey It's me
14:25:12 <chandankumar> As we are working with OSA team to consume os_tempest role in Tripleo CI
14:25:33 <chandankumar> jaosorior: we are able to now run tempest using os_tempest in tripleo CI,
14:26:00 <chandankumar> here is the new job https://review.openstack.org/#/c/627500/ which does the same will be get merged soon
14:26:26 <chandankumar> and here is more updates what happened last week http://lists.openstack.org/pipermail/openstack-discuss/2019-January/001946.html
14:26:33 <Tengu> (19
14:26:36 <chandankumar> we will try to finish integration by upcoming week
14:27:13 <jaosorior> chandankumar: nice work!  I'll stay put if you need reviews.
14:27:24 <chandankumar> jaosorior: sure thanks :-)
14:27:47 <jaosorior> The next topic is from me:     (jaosorior) Reminder: ssbarnea and I are going through Launchpad bugs each week
14:28:04 <jaosorior> So, if anybody is interested. It's one hour before this meeting
14:28:39 <jaosorior> That's all.
14:28:44 <panda> ssbarnea|bkp2 because it's rover I guess
14:31:44 <jaosorior> #topic Squad status
14:31:46 <jaosorior> ci
14:31:48 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-ci-squad-meeting
14:31:50 <jaosorior> upgrade
14:31:52 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-upgrade-squad-status
14:31:54 <jaosorior> containers
14:31:56 <jaosorior> #link https://trello.com/b/S8TmOU0u/tripleo-podman
14:31:58 <jaosorior> integration
14:32:00 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-integration-squad-status
14:32:02 <jaosorior> ui/cli
14:32:04 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-ui-cli-squad-status
14:32:06 <jaosorior> validations
14:32:08 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-validations-squad-status
14:32:10 <jaosorior> networking
14:32:12 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-networking-squad-status
14:32:14 <jaosorior> security
14:32:16 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-security-squad
14:32:18 <jaosorior> edge
14:32:20 <jaosorior> #link https://etherpad.openstack.org/p/tripleo-edge-squad-status
14:32:22 <jaosorior> Any squad wanting to bring up their status, or a topic for the general public?
14:32:34 <Tengu> hmm maybe for validation?
14:32:52 <jaosorior> Tengu: wanna give the update?
14:33:01 <Tengu> The validation framework is back on the bench, and we'll get weekly meeting (hopefully).
14:33:31 <Tengu> among the tasks, it was decided to move the validations as plain ansible roles in order to have a generic way to run and maintain them
14:33:47 <Tengu> more information about the on-going work is located on this public trello board
14:33:51 <Tengu> #link https://trello.com/b/x3h5FrnX/validation-framework
14:34:29 <jaosorior> thanks Tengu
14:35:24 <jaosorior> #topic bugs & blueprints
14:35:25 <jaosorior> #link https://launchpad.net/tripleo/+milestone/stein-3
14:35:27 <jaosorior> For Stein we currently have 21 (for stein-3) blueprints open in Launchpad.
14:35:29 <jaosorior> Bugs: 593 stein-3.  105 (0) open Storyboard bugs.
14:35:31 <jaosorior> #link https://storyboard.openstack.org/#!/project_group/76
14:36:26 <jaosorior> #link https://storyboard.openstack.org/#!/project_group/76
14:36:27 <jaosorior> #topic projects releases or stable backports
14:37:12 <jaosorior> #topic specs
14:37:14 <jaosorior> #link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open
14:37:22 <jaosorior> Any specs someone would like to discuss here?
14:37:27 <openstackgerrit> Merged openstack/tripleo-heat-templates master: mistral-executor: bind mount the docker socket only when needed  https://review.openstack.org/631775
14:39:12 <jaosorior> #topic open discussion
14:39:14 <jaosorior> Anything else that folks want to bring up to the meeting?
14:39:34 <Tengu> firewall management in tripleo maybe?
14:39:42 <fultonj> looks like https://review.openstack.org/#/c/622324 got a lot of feedback. i assume jistr integrated it into it's most recent update.
14:39:45 <Tengu> (sorry to bring that now ;))
14:40:02 <marios> ci community call immediately after tripleo weekly https://bluejeans.com/4113567798
14:40:34 <fultonj> jistr: do you think it's ready ? (i need to read the updated version)
14:40:36 <jistr> fultonj: yea i did. There are some small tweaks to make to the spec still, but i think majority of it is quite close to reality.
14:41:00 <jistr> fultonj: i'll WIP it, one more update to match the currently-being-implemented CLI
14:41:08 <jistr> and then it's good i think
14:41:32 <fultonj> so if anyone provided feedback intially, they should check that it's been added as they think it should.
14:41:36 <fultonj> i'll do that
14:41:38 <fultonj> for ceph
14:41:41 <fultonj> thanks jistr
14:42:56 <jaosorior> fultonj: I'll check it out as well. Thanks for brining it up
14:42:59 <jistr> yup thanks fultonj & and all who provided feedback
14:43:19 <openstackgerrit> Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Use os_tempest for running tempest on standalone  https://review.openstack.org/628415
14:43:32 <jaosorior> Tengu: firewall management sounds good.
14:43:37 <jaosorior> * like a good topic
14:43:53 <jistr> btw there's a good bunch of patches for that spec already posted by me and chem, if you're interested in reviewing those too https://review.openstack.org/#/q/topic:bp/upgrades-with-os+(status:open+OR+status:merged)
14:43:55 <Tengu> hehe
14:44:24 <Tengu> jaosorior: so yeah, basically we have some issues with iptables management. some dangling rules, some default, unmanaged rules and the like.
14:44:35 <openstackgerrit> Chandan Kumar proposed openstack-infra/tripleo-ci master: Run tempest using os_tempest role in standalone job  https://review.openstack.org/627500
14:44:45 <Tengu> I started working on the latter, especially the rules added by the iptables-services package itself: https://review.openstack.org/#/c/632117/
14:45:18 <Tengu> I'd love to get some feedback on the approach, and if anyone has some idea about how to remove those nasty default rules from an already-deployed infra.... :)
14:45:47 <Tengu> fact is, one of them actually open the ssh port for the world - this is actually wanted now, as per https://review.openstack.org/#/q/Ie548f7216610e15af24c96f65a58cc8de603235c - but it is now optional.
14:46:06 <jaosorior> Tengu well, it is wanted for the undercloud only. Not the overcloud.
14:46:16 <Tengu> also, the default rules pushed by iptables-services prevent any logging to happen, as it has a REJECT before the LOG...
14:46:17 <jaosorior> the overcloud should only allow access in the ctlplane network.
14:46:32 <Tengu> jaosorior: yeah, in addition - ssh is wide open on the overcloud nodes, like the controllers.....
14:46:52 <jaosorior> Tengu: not anymore, AFAIK
14:47:22 <Tengu> jaosorior: well, once https://review.openstack.org/#/c/632117/ is merged, we'll be clean for new install.
14:47:36 <Tengu> but older ones will keep the 4-5 rules, unless we find a way to drop them.
14:47:38 <jaosorior> either way, the point remains, we have a mess with iptables rules management
14:47:55 <Tengu> the big issue is, puppet doesn't see those rules as they don't have any comment.
14:48:11 <Tengu> puppetlabs-firewall manages rules using the ressouce name as a comment directly.
14:48:44 <Tengu> so the only way I see is via ansible, in an update/upgrade task, and remove those rules with a state: absent
14:49:39 <openstackgerrit> Carlos Camacho proposed openstack/tripleo-heat-templates master: Include the DB password in a Mistral environment for creating backups and restores  https://review.openstack.org/632438
14:49:50 <Tengu> Also, we have to keep in mind that removing rule from puppet will NOT remove it from the system
14:50:11 <Tengu> we must set the "ensure => absent" in order to actually remove the rule.
14:50:36 <Tengu> we don't have a way to purge unknown rules, since other software are injecting stuff inside iptables, like neutron.
14:50:51 <jaosorior> Tengu: well, purging unkown rules would be problematic for neutron, wouldn't it?
14:51:03 <Tengu> yeah, that's my point.
14:51:06 <Tengu> UNLESS....
14:51:18 <Tengu> we create dedicated chains, and neutron pushes its rules in them only.
14:51:27 <jaosorior> Tengu: might wanna talk to beagles about that
14:51:30 <Tengu> and we manage the "-j neutron-chain-name" within puppet
14:51:38 <Tengu> yeap
14:52:20 <Tengu> so yeah. pretty dense topic, I'll stop here for now, but I'm pretty sure I'll be back on it shortly ;).
14:52:43 <jaosorior> alright, at this point it makes sense to talk to some folks that know neutron and ask what can we do to play well with it
14:52:49 <Tengu> just wanted to draw attention on that.
14:53:00 <jaosorior> sounds to me like it's something we do want to fix
14:53:08 <Tengu> yep.
14:53:15 <Tengu> well, it would be good, at least :).
14:53:21 <jaosorior> having ports randomly open cause we didn't purge them is not a good idea
14:53:29 <Tengu> for the sake of security, clean env and some other consideration
14:53:38 <jaosorior> Tengu: thanks for bringing it up
14:53:44 <Tengu> yw :)
14:53:54 <jaosorior> Tengu: lets start tracking this in the security squad
14:53:58 <jaosorior> I'll bring it up there too.
14:54:17 <jaosorior> Alright! any other topics people wanna bring up?
14:54:22 <fultonj> Reminder: deadline for openstack summit CFP is tomorrow 11:59PM PT
14:54:37 <jaosorior> oh! true!
14:54:57 <beagles> fwiw- I think neutron managed chains are already on things like neutron-openvswi-PREROUTING
14:55:04 <beagles> I'll check to see if this is always the case
14:55:13 <beagles> jaosorior, Tengu
14:55:17 <beagles> ^^^
14:55:23 <jaosorior> beagles: thanks! let us know if that's the case.
14:55:45 <Tengu> beagles: thanks! I'm in another mtg, but I'll be happy to discuss with you :)
14:55:47 <beagles> there may be other rules neutron "sticks in there" as a workaround to another issue - i.e. being able to talk to the openvswitch daemon
14:55:55 <beagles> Tengu,  ack
14:56:11 <jaosorior> #endmeeting