16:00:03 <mattmceuen> #startmeeting Airship
16:00:05 <openstack> Meeting started Tue May 28 16:00:03 2019 UTC and is due to finish in 60 minutes.  The chair is mattmceuen. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:06 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:08 <mattmceuen> #topic Rollcall
16:00:08 <openstack> The meeting name has been set to 'airship'
16:00:18 <alexanderhughes> \o/
16:00:19 <mattmceuen> Good morning / evening everyone!
16:00:32 <ian-pittwood> o/
16:00:34 <mattmceuen> Our illustrious agenda for today https://etherpad.openstack.org/p/airship-meeting-2019-05-28
16:01:05 <mattmceuen> Let's give folks a couple minutes; I know we have a number of folks still out for the US holiday
16:01:30 <michael-beaver> o/
16:01:36 <levmorgan> o/
16:01:44 <evgenyl> Hi!
16:02:20 <jamesgu> o/
16:02:48 <mattmceuen> thanks all for joining us today!  let's get started:
16:03:03 <mattmceuen> #topic OpenSuse Airskiff Gates
16:03:25 <arunkant> o/
16:04:03 <mattmceuen> So, as discussed at the PTG, there are both technical and roadmap-type challenges associated with doing full-airship-1.0-stack integrated testing using OpenSuse
16:04:27 <mattmceuen> SUSE doesn't plan to use maas, and so it's not really a helpful test, among other reasons
16:05:06 <mattmceuen> So some of the validation that would be most valuable would be what airskiff gives you -- bring your own bare metal and k8s
16:05:43 <mattmceuen> So today I just wanted to chat through the work involved in making Airskiff gates for non-ubuntu images
16:05:50 <mattmceuen> Do we have dwalt here by chance?
16:06:04 <jamesgu> so that will covers shipyard, armada, dechhand and pegleg, right/
16:06:09 <dwalt> o/
16:06:30 <mattmceuen> Yup exactly, with gates on the first three projects (not on pegleg itself) is how we do it today
16:06:32 <mattmceuen> o/ dwalt
16:07:21 <mattmceuen> The airskiff manifests live here, as a site definition in treasuremap: https://opendev.org/airship/treasuremap/src/branch/master/site/airskiff
16:07:55 <mattmceuen> they pull from the global versions.yaml, which defines the image overrides
16:08:13 <evgenyl> Is there a plan on how images overrides should be implemented?
16:08:27 <dwalt> That is going to be tricky
16:09:12 <dwalt> We may need to leverage the cicd layer to override images
16:09:32 <mattmceuen> so I think the two broad approaches to non-ubuntu airskiff would be either "make a new site", e.g. airskiff-opensuse, which would be highly redundant; or, use the existing site definition but somehow solve for it to have multiple sets of overrides (which gets a little non-declarative)
16:10:06 <mattmceuen> there is not evgenyl, I think this is the first we've discussed
16:10:24 <evgenyl> +1 for the second one, even if it would involve a hack with some `cp override.yaml site/airskiff..`
16:10:38 <jamesgu> dwalt: cicd layer you mean use ansible templates?
16:10:40 <dwalt> So, if we used the cicd layer, I think we would only need to override three charts. That would also keep us declarative
16:11:10 <dwalt> no. There is an additional cicd layer in treasuremap
16:11:12 <dwalt> #link https://opendev.org/airship/treasuremap/src/branch/master/global/layering-policy.yaml#L11
16:11:21 <mattmceuen> this would be a nice use case for having another layer in our definition hierarchy, if that existed -- e.g. something in between type and site, or a site definition that could inherit from/override another site.  But we don't have that today.
16:11:55 <dwalt> mattmceuen: I think we do have something like that. I linked above :)
16:12:30 <evgenyl> How do we enable this layer only for specific gates?
16:12:49 <mattmceuen> oh hey - I didn't realize that was working yet dwalt :)
16:13:04 <mattmceuen> that is perfect
16:13:12 <evgenyl> I assume it's a site config change.
16:13:17 <dwalt> great question, I think the best way would be to have a separate manifest at that layer. Then, it could be toggled in the pipeline
16:13:47 <dwalt> I don't know if we can select layers using the site configuration. But that would be sweet
16:14:05 <mattmceuen> where does that cicd layer get used today?
16:14:36 <mattmceuen> e.g. I don't see a `/cicd` in treasuremap -- does it get used dynamically only?
16:14:45 <dwalt> I don't think it is used at all, but MattCoachCarter has been playing around with it for the new Deckhand integration gates
16:15:00 <dwalt> mattmceuen: it's possible that's happening in one of the pipelines
16:15:30 <dwalt> I hope we can walk away with a solution that doesn't do that for airskiff
16:16:13 <mattmceuen> git blame tells me that kaspars added that cicd layer - I don't think we have him here today
16:16:13 <evgenyl> mattmceuen: I think it is used somehow in BM pipeline, but there is some jenkins magic that adds required overrides, so they are not in the repo.
16:17:04 <mattmceuen> Is there any reason not to add a `/cicd` to treasuremap, and create an `airskiff-opensuse` under that, with very minimal document replacement -- versions.yaml alone hopefully?
16:17:35 <dwalt> ++ I like that approach
16:17:57 <dwalt> Except -- it would be the individual charts themselves, not versions.yaml
16:18:13 <evgenyl> Can we override versions.yaml?
16:18:25 <mattmceuen> Why not versions.yaml, to just override the image definitions?
16:18:32 <evgenyl> It would be useful to be able to use an existing updater.py to do uplift.
16:18:33 <dwalt> Not with the way it's setup today, the Airskiff site does the overrides there.
16:18:51 <dwalt> I think we could probably fix that
16:18:54 <jamesgu> mattmceun: I like it... but I think there is more than versions.yaml... some osh charts have distro specific overrides, e.g., https://review.opendev.org/gitweb?p=openstack/openstack-helm.git;a=tree;f=horizon/values_overrides;h=1b85f9e317676e59ca8392aa529cbc0b8edf36ec;hb=refs/heads/master
16:19:20 <mattmceuen> aha, good point jamesgu
16:19:55 <dwalt> Actually, I don't think we can fix that. Someone tell me if this reasoning is off
16:20:19 <jamesgu> would be nice that airship can reference the overrides subdir for both distro and releases in airship/airskiff
16:20:44 <dwalt> We would "layer" to replace the versions.yaml, but the chart would use "substitution" to utilize the image. Since substitution happens before layering, we would be grabbing the unmodified images for the charts.
16:20:45 <evgenyl> jamesgu: Hmm, how these overrides are being used now? Are they just passed to helm to override the defaults (ubuntu)?
16:21:48 <jamesgu> I believe osh gate scripts are just passing in multiple overrides paths at the moment
16:22:10 <mattmceuen> dwalt:  no, it should work fine
16:22:37 <mattmceuen> dwalt: trying to think of an example :)
16:23:06 <dwalt> mattmceuen: no worries! I'll accept that answer for now for timing purposes
16:23:10 <evgenyl> I would assume pegleg resolves the dependencies and does some sort of topological sort for dependencies.
16:23:23 <mattmceuen> evgenyl: yep
16:23:41 <mattmceuen> jamesgu:  I think we'll need to duplicate those configs into the new cicd/site definition
16:24:07 <jamesgu> tht's our experience re. versions and substitution.. +1 to what mattmceun said :-0
16:24:25 <mattmceuen> as they are raw values overrides in OSH, rather than in deckhand-friendly docs
16:24:51 <dwalt> That's good news. We should probably take care of modifying the current Airskiff setup first to avoid any unexpected consequences when implementing the cicd layer overrides.
16:25:04 <mattmceuen> agree, would be good to get that done first
16:25:11 <evgenyl> ++
16:25:22 <mattmceuen> Cool - well I think the work in treasuremap should be the "interesting" part, and then copying / pasting the gates in armada, shipyard, deckhand should be pretty straightforwardf
16:25:36 <mattmceuen> anything else to cover on this topic?
16:26:02 <dwalt> Just that we would need to build the correct image for each component
16:26:17 <dwalt> So, that means environment variables, I believe
16:26:29 <mattmceuen> ahh - good point
16:27:08 <jamesgu> and add the ability to plug in image tags to the shipyard and pegleg tools script
16:27:39 <dwalt> For example, in Deckhand, the command would be `make images DISTRO=opensuse_15`
16:27:59 <mattmceuen> jamesgu: I think that should be taken care of declaratively by using the opensuse-specific definition, right?
16:28:30 <mattmceuen> dwalt: would it be as simple as exporting DISTRO prior to running the airskiff scripts?
16:28:42 <mattmceuen> (i.e. no changes needed to the scripts)
16:28:48 <dwalt> Currently, that happens in this playbook, so we could use Zuul variables https://opendev.org/airship/deckhand/src/branch/master/tools/gate/playbooks/airskiff-deploy.yaml#L26
16:29:35 <dwalt> mattmceuen: correct, no script changes needed. Only zuul.yaml and playbook
16:29:43 <mattmceuen> awesome
16:29:44 <jamesgu> mattmceund: I remember the shipyard tools is hard coded to master
16:31:03 <mattmceuen> oh are you talking about this guy, jamesgu: https://opendev.org/airship/shipyard/src/branch/master/tools/shipyard.sh
16:31:13 <mattmceuen> or are you talking about the airskiff script
16:31:27 <jamesgu> yes, the shipyard.sh
16:31:38 <jamesgu> does Airskiff still invoke it?
16:31:49 <mattmceuen> cool - looks like that can be overridden using SHIPYARD_IMAGE as well
16:31:56 <jamesgu> ah okay
16:32:05 <evgenyl> I think airskiff uses https://opendev.org/airship/treasuremap/src/branch/master/tools/airship now
16:32:05 <mattmceuen> good catch
16:32:26 <dwalt> Airskiff now uses a new utility script in treasuremap:
16:32:28 <dwalt> #link https://opendev.org/airship/treasuremap/src/branch/master/tools/airship
16:32:34 <jamesgu> ah okay!
16:32:40 <jamesgu> nice!
16:33:32 <mattmceuen> dwalt: will the `airship` command use the shipyard image from the rendered software-versions doc, or from the global values.yaml?
16:34:00 <mattmceuen> looks like the latter unfortunately
16:34:22 <kskels> right - reading versions.yaml directly
16:34:24 <dwalt> mattmceuen: Looks like you're right. Why unfortunately?
16:34:44 <dwalt> oh, I see. We need rendered.
16:34:49 <mattmceuen> because the global versions.yaml won't have the distro-specific override for the SY image in it
16:35:39 <kskels> the YAML reading of python I believe was already meant for rendered document.. e.g. it should traverse docs..
16:35:51 <dwalt> Does anyone think that introducing an environment variable override for images to that script would be inappropriate?
16:35:59 <kskels> so it should be possible to add that functionalitty to the script rather straight forward as it already has the pegleg command also
16:36:19 <mattmceuen> kskels: that would be great
16:36:22 <dwalt> ++ that would be preferable to what I suggested
16:36:42 <kskels> I think adding envs is nice also
16:36:49 <kskels> so you can use that script for more generic case..
16:37:04 <mattmceuen> agree - glad we uncovered that here
16:37:06 <dwalt> true. We can still do that but use the declarative option :)
16:37:38 <mattmceuen> Ok - we need to keep moving! :)  a couple more agenda items
16:37:54 <mattmceuen> I think we're in good shape with next steps on the airskiff genericization
16:38:08 <mattmceuen> #topic AIAB -> Treasuremap update
16:38:27 <mattmceuen> evgenyl - this is yours, you've been doing a lot of work here.  Want to give us an update?
16:38:57 <evgenyl> Yep, in the last few days I've been working on moving AIAB into treasuremap repo.
16:39:06 <evgenyl> Reviews are welcome https://review.opendev.org/#/c/656900/
16:39:15 <mattmceuen> \o/
16:39:26 <evgenyl> There is a comment from kskels , I have addressed it in a separate PS.
16:39:34 <evgenyl> https://review.opendev.org/#/c/661363/
16:39:39 <dwalt> this is fantastic evgenyl!
16:40:01 <kskels> I'll review today once more, thank you! this is awesome
16:40:10 <evgenyl> As soon as we get it merged and it is stable, we can start looking into multinode gate.
16:40:12 <evgenyl> kskels: thanks!
16:40:13 <mattmceuen> Yes, this'll be very valuable in helping to keep AIAB up to date in its definitions, which will make it much more helpful to the community
16:41:15 <mattmceuen> Please give that a review - and if anyone out there has had issues with AIAB in the past, please give this a try as well -- it should have the latest and greatest airship definitions
16:41:41 <mattmceuen> Anything else on this one?
16:42:04 <evgenyl> That is it from me, thanks!
16:42:11 <mattmceuen> Alright - thanks evgenyl.  Moving on:
16:42:14 <mattmceuen> #topic airskiff deployment gate is failing on shipyard project, and maybe others
16:42:21 <mattmceuen> MattCoachCarter, what's the damage
16:42:34 <MattCoachCarter> I've been noticing this on recent PSs in the shipyard project.
16:42:47 <MattCoachCarter> The gate seems to fail around 12m in trying to install docker I think.
16:43:02 <MattCoachCarter> I don't know much about these gates, I just wanted to bring it to everyone's attention.
16:43:21 <MattCoachCarter> It's non-voting, so it isn't blocking anything, but I think it's a useful check and I'd like to see it stable again.
16:44:16 <mattmceuen> for context if anyone didn't know:  the airskiff gates ar non-voting, so that cross-project integration against master is not a hard requirement of making per-project changes
16:44:16 <MattCoachCarter> Not sure who the best person to look into this would be. But that's about it for this topic.
16:45:33 <mattmceuen> 2019-05-23 17:11:38.391285 | primary | 2019-05-23 17:11:38.390 1 ERROR armada.handlers.wait [-] [chart=ucp-deckhand]: Timed out waiting for jobs (namespace=ucp, labels=(release_group=airship-ucp-deckhand)). These jobs were not ready=['deckhand-db-init', 'deckhand-db-sync'][00m
16:45:55 <mattmceuen> we'll need to look in the logs to see why armada was timing out on deckhand
16:46:17 <dwalt> `E: Unable to locate package libxtables12` ... hmm. It looks like new dependencies were added
16:46:26 <MattCoachCarter> Yeah I saw that too.
16:46:33 <dwalt> mattmceuen: I think that one may be a read-herring
16:46:37 <dwalt> red*
16:47:10 <MattCoachCarter> Anyway, it's not something we need to debug and solve during this meeting. As long as we know it's something that needs fixing.
16:47:45 <mattmceuen> yep, thanks for raising awareness MattCoachCarter.  Non-voting gates are only as valuable as the people that watch for them to fail :)
16:48:35 <mattmceuen> Ok final topic -
16:48:40 <mattmceuen> #topic Deckhand multi-dstro review pending for core reviewer attention
16:48:47 <mattmceuen> https://review.opendev.org/#/c/638301/
16:49:26 <mattmceuen> this been waiting for review since we worked around the old deckhand gate issue - would appreciate additional review!
16:49:55 <mattmceuen> that's a good transition into
16:49:59 <mattmceuen> #topic Roundtable
16:50:09 <mattmceuen> Anything else that needs some eyeballs today?
16:51:28 <mattmceuen> or any other topics on your minds
16:52:32 <mattmceuen> In that case - I will give everyone a couple minutes rest before the next meeting (or a couple extra minutes of sleep if you're jayahn)
16:52:39 <mattmceuen> Thanks everyone, great meeting!
16:52:44 <evgenyl> Thanks!
16:52:47 <mattmceuen> #endmeeting