13:02:20 <jaosorior> #startmeeting TripleO Security Squad
13:02:21 <openstack> Meeting started Wed Mar 21 13:02:20 2018 UTC and is due to finish in 60 minutes.  The chair is jaosorior. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:02:22 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:02:24 <openstack> The meeting name has been set to 'tripleo_security_squad'
13:02:32 <gfidente> Tengu can I see your cmdline?
13:02:51 <jaosorior> Hey! d0ugal, lhinds, owalsh
13:02:52 <Tengu> gfidente: 2s
13:03:00 <Tengu> gfidente: basically I did that: http://paste.openstack.org/show/707512/
13:03:02 <jaosorior> So, today should be a shorter meeting than last time :D
13:03:09 <d0ugal> :)
13:03:12 <jaosorior> should I wait a bit more for other folks? or should we start already?
13:03:26 <lhinds> I think we can kick off with d0ugal here now
13:03:30 <jaosorior> Alright!
13:03:35 <lhinds> mistral is the first topic
13:03:41 <Tengu> jaosorior: oh, meeting? here?
13:03:42 <jaosorior> #topic Mistral Secret Storage
13:03:45 <d0ugal> apetrich, thrash, rbrady, toure ^ we are going to chat about mistral and secrets if you want to join.
13:03:59 <Tengu> gfidente: do you take part in the meeting?
13:04:00 <thrash> d0ugal: ack
13:04:01 <jaosorior> Tengu: yes. It's the weekly Security Squad meeting
13:04:04 <apetrich> oh dear
13:04:16 <gfidente> Tengu security squad?
13:04:22 <Tengu> jaosorior: oh. I'll go DM with gfidente then :)
13:04:26 <rbrady> d0ugal: ack
13:04:46 <jaosorior> So, we've been talking a while about needing secret storage for mistral
13:04:54 <jaosorior> This is due to the fact that we store a  LOT of sensitive information there
13:05:06 <jaosorior> the overcloud private keys and passwords namely
13:05:13 <openstackgerrit> Tim Rozet proposed openstack/puppet-tripleo stable/queens: Fixes incorrect ownership of ODL TLS cert/key  https://review.openstack.org/554909
13:05:41 <jaosorior> Being TripleO an active user of mistral, I would like it to "beta" or take into use any solution that we have in mind
13:06:04 <jaosorior> Also, having talked to thrash in the PTG, I also volunteer to help out on the coding side of mistral if more hands are needed.
13:06:22 <jaosorior> But I would like to talk and understand what are the main challenges on this side
13:07:03 <d0ugal> so, first I think we need to clarify exactly what is stored and why.
13:07:09 <jaosorior> sure
13:07:43 <d0ugal> Mistral has a database that is mostly in-flight only. We store all the heat parameters etc. while the workflow is being executed
13:07:52 <d0ugal> They are then stored for 48 hours afterwards
13:08:22 <d0ugal> Mistral does log lots of information, and parameters may be logged at times - but I think this has been reduced (or possibly stopped)
13:08:23 <thrash> I think the more sensitive stuff is stored in a mistral environment, is it not?
13:08:37 <d0ugal> thrash: no, it is stored in Swift now
13:08:41 <thrash> d0ugal: ack
13:08:51 <apetrich> d0ugal, parameters are logged in debug only now
13:09:20 <apetrich> as with most sensitive info AFAIK
13:09:26 <d0ugal> The only information stored in mistral long term is two different "environments" - blobs of json basically
13:09:45 <d0ugal> These are the ssh keys for overcloud nodes, iirc
13:09:51 <d0ugal> and ..
13:10:00 <jaosorior> d0ugal: which environments?
13:10:03 <d0ugal> undercloud_ceilometer_snmpd_password and undercloud_db_password
13:10:12 <d0ugal> tripleo.undercloud-config and "ssh_keys"
13:10:14 <ooolpbot> URGENT TRIPLEO TASKS NEED ATTENTION
13:10:14 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1757111
13:10:14 <ooolpbot> https://bugs.launchpad.net/tripleo/+bug/1757174
13:10:15 <openstack> Launchpad bug 1757111 in tripleo " fs020(both queens/master) tempest tests failing while booting an instance" [Critical,Triaged]
13:10:16 <openstack> Launchpad bug 1757174 in tripleo "tripleo-buildimage-overcloud-full-centos-7 failing with diskimage_builder.element_dependencies.MissingElementException: Element 'size=4096'' not found" [Critical,Triaged]
13:10:23 <d0ugal> They can be viewed with...
13:10:23 <jaosorior> d0ugal: why do we specifically store those passwords in mistral and not swift?
13:10:25 <d0ugal> $ mistral environment-get tripleo.undercloud-config
13:10:35 <d0ugal> $ mistral environment-get ssh_keys
13:11:04 <openstackgerrit> Marius Cornea proposed openstack/tripleo-upgrade master: Include connectivity check prepare scripts during FFU  https://review.openstack.org/554914
13:11:04 <d0ugal> jaosorior: good question. Mostly for legacy reasoning I think. They could be moved to swift
13:11:05 <owalsh> ssh_keys is the heat-admin key?
13:11:20 <d0ugal> owalsh: I believe so, but I am not sure.
13:11:26 <jaosorior> d0ugal: would be great if we would keep all the passwords in one place. So we can secure that one place at some point.
13:11:44 <Tengu> (use gopass + gpg :D)
13:11:46 <d0ugal> jaosorior: The tripleo.undercloud-config environment is related to the undercloud itself, rather than a plan - I think that is why it is in mistral.
13:12:04 <d0ugal> jaosorior: +1
13:12:31 <openstackgerrit> Carlos Camacho proposed openstack/tripleo-quickstart-extras master: Collect installed cron jobs  https://review.openstack.org/554889
13:12:38 <d0ugal> I think the ssh_keys environment was added out of simplicity, we didn't have a better plan at the time.
13:12:59 <jaosorior> thrash, apetrich does anybody know what ssh_keys actually is? is it the keys for heat-admin?
13:13:00 <trozet> can another core help out with reviewing https://review.openstack.org/#/c/553788/1 please?
13:13:16 <dtantsur> folks.. I know it may sound provocative, but is it possible to add configuration steps to a service template that are NOT written in puppet
13:13:18 <dtantsur> ?
13:13:23 <d0ugal> jaosorior: I can find out.
13:13:25 <thrash> jaosorior: I think so. Would need to double check.
13:13:38 <dtantsur> I don't really want to spend half of cycle doing a trivial thing like 'call a command, get its result'
13:13:44 <d0ugal> or shadower and mandre would know if they are around
13:13:53 <openstackgerrit> Harald Jensås proposed openstack/tripleo-heat-templates master: Add ctlplane networking for routed networks  https://review.openstack.org/547326
13:14:04 <jaosorior> either way, there's a private key there, which would be considered sensitive info. So we need to secure it somehow
13:14:17 <d0ugal> jaosorior: +1
13:14:35 <hjensas> derekh: ^^ Can you have a look at the python script there? Make sure I don't mess up the ipv6 stuff again?
13:14:36 <dtantsur> EmilienM: hey, maybe you know (re my question above)
13:14:40 <jaosorior> d0ugal, thrash: One option would be to move all that to swift. And rely on swift encryption (which we don't have right now, but we could enable)
13:14:53 <apetrich> jaosorior, during ping test (and I think tempest as well but not 100% sure) the keys to the created servers are stored in an env in mistral
13:15:18 <thrash> jaosorior: +1000
13:15:34 <d0ugal> jaosorior: I didn't know swift had that option, sounds like a good (and easy?) starting point.
13:15:39 <jaosorior> thrash, d0ugal, apetrich: Would you guys be able to dedicate some time to move those to swift?
13:15:50 <thrash> jaosorior: somebody can, yes. :)
13:16:07 <jaosorior> d0ugal, to be able to do that, we probably need barbican in the undercloud, but that's something alee and me can work on.
13:16:11 <d0ugal> jaosorior: we are going to do some planning soon, so we could open a bug for this and consider it then
13:16:50 <jaosorior> d0ugal, apetrich, thrash: So, having moved those environments to be stored in swift. Would that be the last bits of sensitive info stored in mistral?
13:16:54 <owalsh> if it's only used the the pingtest/tempest key do we care?
13:17:29 <thrash> jaosorior: I think from a tripleo perspective, that's a good bet.
13:17:31 <apetrich> owalsh, not only those keys unfortunately
13:17:40 <owalsh> apetrich: ack
13:17:55 <jaosorior> owalsh: it sure depends on the user that pingtest/tempest uses. If it's heat-admin it's problematic, since it's able to do sudo su.
13:18:02 <d0ugal> jaosorior: do you could storing for 48 hours as storing? :)
13:18:19 <owalsh> jaosorior: runs as stack AFAIK
13:18:29 <d0ugal> jaosorior: we also probably need to do some checking of the logs and/or protection there against future leaks
13:18:32 <jaosorior> d0ugal: I need to double check on that one. lhinds what do you think?
13:18:38 <jaosorior> d0ugal: definitely
13:18:47 <lhinds> jaosorior: just reading..
13:19:41 <lhinds> I guess time could be configurable for now (if that's what you were refering to)
13:20:08 <lhinds> or log integrity?
13:20:38 <jaosorior> Log integrity is something we should cover, so we should report any issues as mistral bugs and get those fixed.
13:20:58 <jaosorior> lhinds: but currently mistral stores the heat environments (which might contain sensitive info) for a limited time (48 hours)
13:21:11 <jaosorior> lhinds: is this something we can live with, or should we also avoid this?
13:21:17 <openstackgerrit> Harald Jensås proposed openstack/tripleo-heat-templates master: Add ctlplane networking for routed networks  https://review.openstack.org/547326
13:21:21 <d0ugal> FWIW, fixing this in Mistral will likely be very hard.
13:21:48 <alee> o/
13:21:51 <lhinds> so it would be difficult to encrypt the heat envs?
13:22:06 <lhinds> (stored in mistal)
13:22:28 <d0ugal> lhinds: I think so, mistral internally duplicates them in a few places to optimize db lookup
13:22:51 <lhinds> d0ugal: ack
13:23:10 <jaosorior> d0ugal: I thought the generated heat environments were all stored in swift.
13:23:21 <lhinds> so i think as far as time periods, any time window is a potential exploit window (although shorted better of course)
13:23:27 <d0ugal> jaosorior: they are - but while the workflow is running and for 48 hours after they are also in Mistral
13:23:45 <jaosorior> d0ugal: is it possible to disable that?
13:24:20 <d0ugal> jaosorior: yes, they could be deleted when the workflow finishes, but it is extremely useful for debugging etc.
13:24:30 <d0ugal> We actually increased the time, the default is 1 hour irrc
13:24:34 <d0ugal> iirc*
13:24:46 <jaosorior> d0ugal: how is it useful for debugging?
13:25:22 <d0ugal> jaosorior: when the execution is stored you can inspect it and find out exactly what happened, what inputs and outputs happened at every point in the workflow
13:25:31 <d0ugal> jaosorior: you can even restart workflows in the middle etc.
13:25:33 <lhinds> has there been any BP / LP for encrypting heat envs stored in mistral (so it's on the radar so to speak). I could take a look at the code, can't promise anything as new to mistral
13:25:55 <d0ugal> it is a bit like having the interactive debugger you have in most programming languages (but via a rest api :))
13:25:59 <jaosorior> d0ugal: What about making that attribute configurable? In the hardening docs we could then tell folks to lower that time, or disable it entirely.
13:26:14 <lhinds> but with a key in barbican, it should be doable.
13:26:27 <d0ugal> jaosorior: it is configured by instack-undercloud, can users change those puppet settings?
13:26:45 <jaosorior> should be possible
13:26:49 <jaosorior> depending on how it's configured
13:27:30 <jaosorior> Need to double-check if the instack-undercloud hieradata takes precedence or the hieradata overrides do. but it should be doable.
13:27:56 <d0ugal> lhinds: there was a blueprint for mistral for securing secrets. I think both rbrady and thrash had a look at doing it. So they know more about that than me.
13:28:00 <jaosorior> #action For now, we will document how to lower the time mistral stores heat environments and add it to the hardening guide.
13:28:39 <d0ugal> jaosorior: FYI, here is the setting: https://github.com/openstack/instack-undercloud/blob/master/elements/puppet-stack-config/puppet-stack-config.yaml.template#L671
13:28:51 <jaosorior> #link https://github.com/openstack/instack-undercloud/blob/master/elements/puppet-stack-config/puppet-stack-config.yaml.template#L671
13:28:51 <lhinds> d0ugal / rbrady / thrash if you manage to dig it out (the BP) please paste if for me.
13:28:58 <d0ugal> lhinds: looking for it.
13:29:02 <lhinds> thanks d0ugal
13:29:12 <d0ugal> lhinds: https://blueprints.launchpad.net/mistral/+spec/mistral-secure-sensitive-data
13:29:28 <lhinds> so configurable as first port of call, and then ideal future functionaility to encrypt
13:29:31 <d0ugal> See the spec linked at the top and there was a patch, but I think that got stuck.
13:30:45 <lhinds> so there is a fair whack of code there, any reason for the abandon by Brad?
13:31:02 <d0ugal> jaosorior: should I open a bug for the mistral environments?
13:31:09 <jaosorior> d0ugal: that would be great
13:31:14 <d0ugal> k, on it
13:32:07 <alee> d0ugal, I'm having trouble finding the actual spec ..
13:32:27 <d0ugal> alee: https://specs.openstack.org/openstack/mistral-specs/specs/pike/approved/secure-sensitive-data.html
13:32:34 <lhinds> alee: spec has gone missing, but some code here:
13:32:36 <lhinds> https://review.openstack.org/#/c/459747/
13:32:44 <jaosorior> #link https://specs.openstack.org/openstack/mistral-specs/specs/pike/approved/secure-sensitive-data.html
13:32:46 <alee> ah cool thanks
13:32:47 <d0ugal> I think the spec was moved because it missed the openstack release
13:32:57 <d0ugal> Which is a bad idea it seems :)
13:33:50 <lhinds> k, found the spec:
13:33:50 <jaosorior> Alright, but at least for the short term we have a plan
13:33:53 <lhinds> #link https://github.com/openstack/mistral-specs/blob/master/specs/pike/approved/secure-sensitive-data.rst
13:34:10 <jaosorior> * Move all sensitive data to swift (to have it all in one place)
13:34:23 <lhinds> ok, brad is thrash, got it now
13:34:27 <jaosorior> * Document how to reduce time mistral stores heat environments)
13:34:28 <thrash> lhinds: :D
13:34:32 <d0ugal> #link https://bugs.launchpad.net/tripleo/+bug/1757430
13:34:33 <openstack> Launchpad bug 1757430 in tripleo "The ssh_keys and tripleo.undercloud-config Mistral environments should be move to swift" [High,Confirmed]
13:34:46 <jaosorior> and then we can focus on securing swift instead, which already can encrypt with barbican.
13:35:00 <jaosorior> d0ugal: awesome
13:35:32 <jaosorior> thanks
13:36:28 <d0ugal> np
13:36:33 <jaosorior> Anything else someone wants to bring up about this topic?
13:37:09 <lhinds> nothing from me this week
13:37:18 <jaosorior> ok
13:37:31 <openstackgerrit> Harald Jensås proposed openstack/python-tripleoclient master: Fix Genconfig - no HOME in environment  https://review.openstack.org/554678
13:37:41 <jaosorior> Thanks d0ugal, thrash and apetrich for joining
13:37:57 <jaosorior> #topic Work progress udpate
13:38:17 <d0ugal> jaosorior: np, thanks for the input!
13:38:50 <jaosorior> Just a heads up for folks in the squad, there are a bunch of reviews for different items in the etherpad https://etherpad.openstack.org/p/tripleo-security-squad (Maybe we need to come up with an easier way to track those)
13:38:55 <jaosorior> so reviews are appreciated
13:39:26 <jaosorior> Right now, most of the work that I've been doing has been on enabling TLS by default (which hopefully almost merges for the undercloud https://review.openstack.org/#/c/552382/ )
13:39:47 <jaosorior> I'm also working on enabling it by default in the overcloud, so if someone is intersted in joining that work or testing, let me know.
13:40:13 <jaosorior> that's all on my side.
13:40:35 <alee> jaosorior, I'll probably ping you about joining that work later today or tomorrow
13:40:44 <jaosorior> alee: awesome
13:41:07 <jaosorior> #topic Any other business
13:41:17 <jaosorior> Anything else someone wants to bring up to the squad?
13:41:33 <alee> jaosorior, I think we wanted to do a quick meeting to identify secrets to be secured/ passwords etc.
13:41:50 <alee> jaosorior, did we want to schedule that?
13:41:58 <jaosorior> alee: that would be good.
13:42:17 <jaosorior> alee: Any day/time preference?
13:42:34 <alee> jaosorior, how about tommorow?
13:42:44 <jaosorior> works for me
13:42:49 <openstackgerrit> Martin André proposed openstack/tripleo-common master: Pass connection info via ansible config file  https://review.openstack.org/554526
13:42:53 <alee> morning my time -- say 10 am EST?
13:43:08 <jaosorior> alee: that works for me. 2pm UTC
13:43:12 <jaosorior> lhinds: does that work for you?
13:43:34 <lhinds> jaosorior: thats fine for me
13:44:25 <lhinds> I have a work shop thing, but might be able to leave a little early
13:44:32 <lhinds> (it's remote)
13:45:13 <jaosorior> lhinds, alee: I'll poke you tomorrow then before the time.
13:45:16 <jaosorior> Anybody else is welcome to join
13:46:12 <jaosorior> Anything else someone would like to bring up?
13:47:11 <jaosorior> Alright
13:47:15 <jaosorior> thanks everyone for joining!
13:47:17 <jaosorior> #endmeeting