14:00:20 <mattmceuen> #startmeeting airship
14:00:21 <openstack> Meeting started Tue Jan 15 14:00:20 2019 UTC and is due to finish in 60 minutes.  The chair is mattmceuen. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:22 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:24 <openstack> The meeting name has been set to 'airship'
14:00:28 <mattmceuen> #topic rollcall
14:00:42 <mattmceuen> gm everyone!
14:00:44 <dwalt> \o/
14:00:50 <dwalt> GM!'
14:00:59 <mattmceuen> Here's our agenda for today: https://etherpad.openstack.org/p/airship-meeting-2019-01-15
14:01:11 <mattmceuen> Please add anything you'd like to discuss
14:01:15 <evgenyl> Good morning!
14:01:26 <mattmceuen> and we'll give it a couple mins for folks to filter in
14:01:30 <sthussey> here
14:01:42 <Kaspars> also here
14:01:49 <mattmceuen> hey guys
14:02:29 <portdirect> o/
14:03:15 <mattmceuen> so the meat of the agenda today is to share/explain a few things that have gotten added in recently
14:03:20 <mattmceuen> #topic podsecuritypolicy chart
14:03:52 <mattmceuen> So you may or may not be familiar with k8s PodSecurityPolicies
14:04:43 <mattmceuen> If you configure your k8s to use them, then the k8s api server will only allow you an actor to schedule pods that meet certain criteria/policy
14:04:56 <mattmceuen> Based on k8s RBAC
14:05:36 <mattmceuen> This is a security feature we wanted to add into airship, but in a way that doesn't break all the privileged actions that are taken across airship and openstack-helm
14:05:46 <mattmceuen> This helm chart was added to openstack-helm-infra: https://github.com/openstack/openstack-helm-infra/tree/master/podsecuritypolicy
14:06:17 <mattmceuen> It specifies by default an incredibly (100%) permissive podsecuritypolity, and sets it up as a default for the cluster
14:06:54 <mattmceuen> It was also added into the Treasuremap reference yamls: https://review.openstack.org/#/c/629686/
14:07:04 <mattmceuen> You can do a couple things through the chart:
14:07:18 <sthussey> Why was the chart put in a different project if it was intended as a part of Airship?
14:07:41 <mattmceuen> because it's not airship-specific
14:07:46 <mattmceuen> pretty broadly applicable
14:08:24 <mattmceuen> You can use the chart to:
14:08:24 <mattmceuen> 1) add whatever additional podsecuritypolicies you want for your cluster, programmatically, letting helm manage lifecycle
14:08:34 <mattmceuen> 2) change the default(s)
14:09:04 <mattmceuen> Over time, we want to tune the defaults in the chart to be a reasonable non-fully-open set of policy, as much as possible.  However, the intent is also that operators fully customize it for their workloads,
14:09:39 <portdirect> Nice work mattmceuen
14:09:59 <mattmceuen> You can set up defaults in the chart individually for serviceacounts, authenticated users, and unauthenticated users via the chart, and/or associate the PSPs to other roles outside of the chart itself
14:10:04 <mattmceuen> that's it - any questions?
14:10:13 <portdirect> Should help make it much easier to manage the security posture of a cluster.
14:10:25 <roman_g> o/
14:10:37 <hogepodge> Hi
14:10:42 <mattmceuen> o/ guys :)
14:10:56 <mattmceuen> One more note, just having to do with how PSP's work in k8s
14:10:59 <sthussey> What is the expectation of coordinating access to PSPs and the service accounts created by other Helm charts?
14:11:48 <mattmceuen> that'll be on the operator to coordinate; in an airship world, via substitution.  That would be a good pattern to construct/demo in treasuremap I'd think
14:12:59 <mattmceuen> so PSPs only affect pods created *after* the PSPs are in place.  So, that's why the psp chart was added first in the treasuremap armada manifest
14:13:26 <mattmceuen> alright, next topic:
14:13:35 <mattmceuen> #topic Mini-mirror integration
14:13:41 <mattmceuen> dwalt take it away!
14:14:00 <dwalt> alrighty!
14:14:16 <roman_g> dwalt: Nice job! I like it. Thank you.
14:14:21 <dwalt> One of the challenges we were having in larger scale deployments was the coordination of packages on the host
14:14:26 <dwalt> Thanks roman_g!
14:15:57 <dwalt> i.e. packages changed between the time that we ran test deployment pipelines to the time we actually deployed Airship
14:17:13 <dwalt> Mini-mirror exists as a way to combat that issue. By mirroring Ubuntu package repositories and deploying them into the cluster, we can control what packages exist on the host for a deployment
14:17:55 <powerds0111> o/
14:18:23 <dwalt> Currently, you can utilize mini-mirror by building an image by specifying the sources/packages you would like to mirror
14:18:25 <dwalt> #link https://github.com/openstack/openstack-helm-images/blob/master/mini-mirror/README.rst
14:19:11 <dwalt> The next step is integrating mini-mirror into the genesis process, with the vision that we may one be able to deploy Airship without internet connectivity
14:19:31 <dwalt> The chart for mini-mirror is also here:
14:19:33 <dwalt> #link https://github.com/openstack/openstack-helm-addons/tree/master/mini-mirror
14:19:37 <dwalt> Any questions?
14:19:57 <Kaspars> this looks really cool, are you thinking to add it to treasuremap any time soon?
14:20:26 <roman_g> At the moment mini-mirror does not contain: docker image archives and k8s binaries; they would need to be baked into some /var/cache inside the image
14:20:30 <dwalt> Kaspars: Thanks! Definitely. As soon as we have a published demo image, we would like to get a full demo
14:21:50 <sthussey> mini-mirror is just a framework. It should contain whatever packages you tell it sourced from whatever source you provide when you build the image.
14:22:05 <dwalt> ++
14:22:19 <mattmceuen> Yup roman_g those would be good enhancements down the road
14:22:24 <mattmceuen> o/ powerds0111
14:22:48 <mattmceuen> anything else on the mini-mirror front guys?
14:22:57 <mattmceuen> also, hooray for clever names.  very catchy
14:23:16 <sthussey> I would suggest if it isn't merged yet that you consider naming it mini-apt or something
14:23:47 <mattmceuen> lol well you got one vote each way there
14:23:52 <mattmceuen> but it's merged
14:24:10 <sthussey> It isn't really part of Airship, so this is just tangential, but there is a merged spec in Airship to support multiple OSes and making mini-mirror Ubuntu specific doesn't really jive with that
14:24:20 <portdirect> There would be nothing stopping adding rpm support
14:24:35 <mattmceuen> totally agree
14:24:35 <sthussey> And I don't think long term adding the complexity to make a single framework support both is worth it
14:24:50 <sthussey> Just pick whether you want mini-apt or mini-rpm and go forward
14:25:02 <portdirect> Or just a value toggel
14:25:18 <portdirect> All it would take is turning on or off, two manifests
14:25:19 <mattmceuen> the *chart* part is really an nginx chart, right dwalt?
14:25:31 <sthussey> Yeah, I don't really care about the chart part
14:25:32 <mattmceuen> and the apt magic is in the docker build stuff
14:25:36 <sthussey> The complexity is in building the image
14:25:44 <dwalt> At it's core, it really is just NGINX
14:25:56 <dwalt> the image itself is just an NGINX image with the mirrored snapshots
14:26:15 <sthussey> Again, just tangential as it isn't part of Airship. But a consideration.
14:26:15 <dwalt> so a similar tool could be used (with a toggle) to mirror rpm repos
14:26:24 <mattmceuen> it might make sense to separate the apt docker build from the rpm docker build, since they'd likely be quite different
14:26:52 <dwalt> This is true. And OpenStack-Helm addons already has the pattern for multiple OS/Dockerfiles
14:27:07 <mattmceuen> I don't think it's tangential, we want to bring multi-os support into Airship
14:27:08 <dwalt> OpenStack-Helm-images*
14:27:10 <mattmceuen> good point
14:27:26 <mattmceuen> anything else on MM?
14:27:55 <portdirect> Maxi-mirror?
14:28:16 <srwilkers> :)
14:28:19 <mattmceuen> Just depends how much stuff you shove in the image portdirect :)
14:28:27 <mattmceuen> #topic Monthly Treasuremap Release
14:28:27 <portdirect> Is there thought toward caching of things like docker images at this point, or that still at design stage?
14:28:41 * mattmceuen jumps the gun
14:28:48 <mattmceuen> #topic maxi-mirror
14:29:23 <mattmceuen> portdirect: I think there's been some discussion around it, but I don't know if any stories are in storyboard or anyone working on it yet
14:29:51 <roman_g> me neither
14:29:52 <sthussey> While possibly branded as mini-mirror, I would expect something like that to basically be completely separate in terms of image build
14:30:32 <portdirect> I was more thinking an adjunt here, to separate the concerns
14:30:45 <sthussey> I think designs right now around image caching are something like squid w/ MITM TLS signing
14:30:48 <portdirect> Looks like mini-mirror allows offline host provisioning
14:31:05 <roman_g> dwalt: would you work on blueprint, or I can take it?
14:31:19 <portdirect> But still leaves the gap of how to get containers into a site without internet or artifactory or summat.
14:31:35 <roman_g> (bake into genesis iso)
14:31:39 <dwalt> roman_g: You can take a stab at it if you'd like!
14:31:45 <roman_g> dwalt: deal
14:31:50 <sthussey> At this point, that wasn't the business value targeted w/ mini-mirror
14:31:50 <portdirect> Nice
14:32:09 <portdirect> sthussey: I understand that, hence my q.
14:32:56 <roman_g> sthussey: portdirect: mattmceuen what is your decision/input?
14:33:23 <sthussey> Decision on what? If you want to draw up a spec for something, go for it.
14:33:40 <mattmceuen> roman_g:  if you'd like to take a stab at a spec PS, I think that gives a good opportunity to prompt discussion, even (especially) if that prompts a change in direction
14:33:46 <portdirect> After 3 mins of percolation time, bit early for me to put my stake in the ground on anything roman_g
14:34:01 <portdirect> ++ to mattmceuen s point
14:34:18 <mattmceuen> thanks roman_g
14:34:34 <mattmceuen> Anything else on the multi-mirror discussino?
14:35:00 <mattmceuen> #topic Monthly Treasuremap Release
14:35:08 <mattmceuen> Kaspars, all yours
14:36:00 <Kaspars> great! so we have I would say almost routine monthly tag of airship-treasuremap manifests found located at https://github.com/openstack/airship-treasuremap/releases
14:36:48 <Kaspars> as before - it includes auto-uplifts of components (with a few exclusions this time that include MAAS HA that Scott will talk about later, and armada due to test pod handling)
14:37:17 <Kaspars> there are a few other notable changes as ceph partition handling and update of Docker to much newer version
14:38:22 <Kaspars> as before - it has ran through full airship deployment pipeline (Airship Seaworthy), here are few sentances about it https://airship-treasuremap.readthedocs.io/en/latest/
14:38:57 <Kaspars> I would say enjoy the latest release candidate for 1.0
14:39:21 <mattmceuen> that's awesome Kaspars
14:39:29 <seaneagan> in-flight fix for Armada test pod handling: https://review.openstack.org/#/c/629313/
14:39:45 <mattmceuen> thanks for keeping the good stuff coming and the pipelines green :)
14:39:59 <mattmceuen> o/ seaneagan
14:40:30 <mattmceuen> any other discussion on the release candidate?
14:41:18 <mattmceuen> #topic HA MaaS overview
14:41:23 <mattmceuen> sthussey all yours!
14:42:12 <sthussey> Recently there a significant refactor of the Airship chart for the Canonical MAAS product.
14:42:41 <sthussey> Previously it was a bit fragile in terms of where it ran and how it responded to moving to other nodes.
14:43:44 <sthussey> It now should support a highly available configuration for the MAAS components (HA Postgres under it is out of scope) and with that HA support comes mobility across nodes in the cluster
14:44:09 <sthussey> There are some caveats and suppositions for the current solution
14:44:17 <portdirect> thats great sthussey
14:44:39 <portdirect> is the required config in treasuremap and airship in a bottle - so we can take it for a spin?
14:44:41 <sthussey> 1) It expects a flat L2 broadcast domain for the PXE network across all nodes in the cluster
14:45:07 <sthussey> re: config - Can't speak to Treasuremap, but the multinode-gate in AIAB should work with it
14:46:03 <Kaspars> there has not been patchsets submitted for treasuremap, yet
14:46:19 <sthussey> 2) You must be able to route between the PXE booting network and whatever network your cluster ingress runs on. If there is a single network in your cluster for all traffic, that is sufficient.
14:47:05 <sthussey> 3) Currently there is no TLS support for accessing the MAAS API/metadata server. That support will hopefully be in place in the near future
14:47:41 <hogepodge> Somewhat related, is Ironic making progress in the RCs?
14:48:08 <mattmceuen> hogepodge:  there was good discussion in rodolfo's zoom call last thursday, if you want to see the recording
14:48:11 <sthussey> 4) If you are using the OSH-infra ingress chart, you must be using nginx-ingress-controller v0.20.0 or better
14:48:29 <hogepodge> (kind of previous topic juggling multiple chat windows)
14:48:30 <sthussey> Otherwise you get false port binding conflicts between cluster ingress and maas ingress
14:48:42 <sthussey> That is all from me.
14:49:01 <mattmceuen> thanks for all the details sthussey - that was a big effort with a lot of tricky bits
14:49:12 <hogepodge> mattmceuen yes please, I’ll coordinate offline thx
14:49:21 <mattmceuen> awesome, sounds good hogepodge
14:49:37 <mattmceuen> sounds like treasuremap is pinned to an old version - I guess the interface to the maas chart isn't backwards compatible, right sthussey?
14:50:34 <sthussey> Not sure, would have to test. The previous overrides certainly won't enable the HA functionality
14:51:28 <Kaspars> the pipeline did fail - so I had to disable auto-uplift for maas/drydock charts..
14:51:33 <mattmceuen> gotcha.  Kaspars, maybe we can coordinate in the chat here to after the meeting to see what the issues you were seeing are, so we can get HA MaaS into TM now that the stable release was cut
14:52:12 <mattmceuen> k we have a few more topics and scant few minutes!
14:52:20 <mattmceuen> #topic PS needing review
14:52:25 <Kaspars> there was a pretty large PS comitted to AIAB - so those changes need to be ported to treasuremap as well
14:52:32 <mattmceuen> I'm just going to paste these with a blurb and then move on!
14:52:44 <Kaspars> to support latest MAAS/drydock - as well as few DNS handling need to be settled
14:52:47 <Kaspars> some work..
14:52:54 <mattmceuen> Blurb:  I want to make sure there's an opportunity for folks to bring up PS weekly that they view as high priority, and/or lost in the shuffle
14:52:58 <mattmceuen> https://review.openstack.org/630749
14:52:59 <mattmceuen> https://review.openstack.org/630752
14:52:59 <mattmceuen> https://review.openstack.org/630755
14:53:15 <mattmceuen> ^^ from dwalt - please give those some eyes soon team
14:53:20 <mattmceuen> any others?
14:54:09 <mattmceuen> One topic not on the agenda:
14:54:33 <mattmceuen> #topic best of luck mark-burnett !
14:54:44 <hogepodge> I have one regarding ATC codes for Denver Summit
14:55:24 <Kaspars> best of luck, Mark!!
14:55:32 <mattmceuen> Mark is taking another opportunity.  He is of course more than welcome to continue participating in the airship community as much as possible!  But I also just wanted to share that hey may not have the same bandwidth
14:56:08 <mattmceuen> Mark has been instrumental in forming the airship community, and definitely with him best of luck in the next endeavor!
14:56:25 <mattmceuen> #topic ATC codes for Denver Summit
14:56:30 <mattmceuen> go for it hogepodge
14:57:00 <hogepodge> Congratulations Mark!
14:57:40 <cbcg> congarts Mark!
14:57:47 <hogepodge> we need to put together a list of active contributors to receieve discount codes for the Denver summit
14:58:23 <hogepodge> To help us with that we need a list of repositories to draw from for contributions.
14:58:54 <hogepodge> I’m assuming all repos prefixed airship-, but if there are others we missed getting them to fungi would be a big help
14:59:11 <mattmceuen> I think that is the right set hogepodge
14:59:31 <mattmceuen> thanks for making sure
14:59:37 <hogepodge> Since Airship is so new, we’re looking at contributions back to the launch of Airship as a pilot project in Vancouver for the contribution timeframe
14:59:48 <hogepodge> cool, thanks :-)
14:59:51 <mattmceuen> Sounds reasonable to me! :)
15:00:36 <mattmceuen> Ok guys we are hard outta time -- please vote in the board election, and help out with the deckhand doc builds:
15:00:36 <mattmceuen> Please, assist: airship-deckhand documentation builds are broken since Oct. 30, 2018
15:00:36 <mattmceuen> Bug: https://storyboard.openstack.org/#!/story/2004747
15:00:36 <mattmceuen> Logs: https://readthedocs.org/api/v2/build/8395458.txt
15:01:06 <roman_g> It's mine, and I don't know how to fix it.
15:01:32 <dwalt> roman_g: I will take a look at that sometime today
15:01:34 <mattmceuen> Cool - let's continue the discussion here post-meeting roman_g and get that working
15:01:38 <mattmceuen> #endmeeting