16:00:03 #startmeeting airship 16:00:03 Meeting started Tue Jun 18 16:00:03 2019 UTC and is due to finish in 60 minutes. The chair is mattmceuen. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:04 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:06 The meeting name has been set to 'airship' 16:00:09 #topic Rollcall 16:00:19 o/ 16:00:20 Hi everyone! 16:00:27 GM / GE everyone! here's our agenda for today, you know the drill: https://etherpad.openstack.org/p/airship-meeting-2019-06-18 16:00:28 o/ 16:00:29 o/ 16:00:34 we'll give it just a minute 16:00:36 Hello everyone! 16:00:45 o/ obravo welcome 16:01:45 o/ 16:02:05 Alright, let's go: 16:02:12 #topic Election 16:02:54 Just a very brief update on what we discussed last week - it took some time to get the airship/elections project created, so I'm planning on sending out an email on that to kick off TC nominations today 16:03:12 That's all I have on that 16:04:09 PS: if you visit the election repo (https://opendev.org/airship/election) the dates are all wrong - they were the optimistic ones I authored last week, I'll fix them prior to sending out an email 16:04:29 unless any questions, I'll move on: 16:04:53 #topic Meeting Time Voting 16:05:23 I sent out an email on this yesterday, but the instructions are also in the etherpad (no need to find the email if you missed it) 16:06:14 We discussed at the PTG that there's no good time for this IRC meeting that works well for everyone, and we wanted to vote not just on meeting times but also on some potential approaches (e.g. alternating meetings) to try to accomodate more attendees 16:07:00 The ask is for everyone (who cares about timing) to give their top five choices for Meeting Time + Meeting Approach by filling in a template on the bottom of the etherpad: 16:07:01 https://etherpad.openstack.org/p/airship-meeting-vote-2019 16:07:25 More details in the etherpad, and thanks to everyone who's voted so far! 16:07:44 We'll leave voting open till EOD next Tuesday so we can remind folks in next Tuesday's meeting 16:07:45 Kudaka Poorna Rajesh proposed airship/promenade master: [DNM] Test probes for calico-etcd https://review.opendev.org/665900 16:07:58 any questions / discussion on the meeting time stuff? 16:09:08 Cool, moving on: 16:09:14 #topic Tungsten Fabric Integration 16:09:42 obravo has been doing some work to integrate Tungsten Fabric into Airship -- AIAB PS: https://review.opendev.org/#/c/664648/ 16:09:52 obravo, can you please give us some quick overview of this work? 16:11:32 Well , it makes AIAB use TF instead of openvswitch . I did some changes into deploy scripts and configs. 16:12:02 That's awesome 16:12:03 I tried to keep switching between TF and vSwitch as easy , as possible 16:12:28 How is the work progressing / working? 16:13:04 it`s tested with TF running and is working good in singlenode deployment 16:13:19 Should we consider leaving neutron-ovs by default and creating overrides specific to other network providers? 16:13:43 Also you may start looking into an up to date AIAB configuration https://github.com/airshipit/treasuremap/tree/master/tools/deployment/aiab 16:14:01 The one in Airship in a Bottle is outdated, and will be retired in the nearest future. 16:14:27 *The one in Airship in a Bottle repository 16:14:48 Yeah, obravo's patchset prompted me to add that onto the agenda - we'll dig into the "which aiab" more in a minute 16:15:17 Yep, sounds good. 16:15:39 obravo: I know you've been working to get TF support into the upstream OSH Neutron Chart: https://review.opendev.org/#/c/663390/ 16:15:55 Would it make sense to point your patchset for TF AIAB at that patchset? 16:16:22 Yes , this is related review 16:16:40 PRATEEK REDDY DODDA proposed airship/promenade master: [WIP] Coredns: Add pod/container security context https://review.opendev.org/664662 16:17:39 my thought was just that, in `versions.yaml` you could point the neutron chart to that patchset, rather than the github version, to help keep the two in sync. just a suggestion 16:18:37 this is great to see this progressing, and I'll dig in and provide feedback in the PS obravo 16:18:45 Oh , i got it. Thank you , this will he be more convenient 16:19:24 one thing to keep in mind as evenyl said we're moving aiab into treasuremap. Part of the driver for that is to keep all of our "reference manifests" aligned and de-duplicated 16:20:17 Fine , i`ll take a look into it 16:20:19 I'm considering how best to make TF manifests available as a reference -- AIAB is a nice start, but it would be ideal not to constrain them just to the AIAB site 16:20:59 This would be a perfect use case for the "service layer" mix-and-match approach we've wanted for a long time, but don't yet have 16:21:00 +1 ... I was going to make the same request 16:22:16 Using "what we have today", it would be somewhat straightforward to: 16:22:33 well, hmm 16:23:12 there are multiple tools in our toolbox, including overriding at the type or site level, making new manifests at the global level, or actually implementing something like service layers 16:23:21 Can we do this: 16:24:16 obravo: I don't want to derail your work -- perhaps you can keep doing what you're doing, perhaps migrating over to treasuremap 16:25:02 and then let's chew on the best way to integrate TF reference manifests (and other types of use cases in the future) into treasuremap more generally next week 16:25:34 does that sound good to everyone? Ok to take some "think this through" homework? :) 16:26:19 sounds good.. (minus the homework:-)) 16:26:21 Yes, that`s fine 16:26:29 lol jamesgu__ 16:26:40 ok, great 16:26:47 thanks for bringing this up obravo 16:27:09 related topic: 16:27:16 #topic AIAB -> Treasuremap migration status 16:27:57 We've started directing folks (informally) to the new treasuremap single-node AIAB when they hit issues with the single-node demo site in the airship-in-a-bottle repo 16:28:31 Yeah, as mentioned earlier here is link https://github.com/airshipit/treasuremap/tree/master/tools/deployment/aiab 16:28:32 Are we comfortable to make the treasuremap version `the`AIAB single-node deployment yet? 16:28:40 Please try it out. 16:28:51 The builds are kind of stable, the only concern is MaaS. 16:29:05 Sometime it takes more than 20 minutes for maas-import-resource to finish. 16:29:15 So it may fail due to timeout. 16:29:36 how long does maas-import-resource take on the aiab repo version? 16:30:08 mattmceuen: I have not measured on a previous version, but definitely less than 20 minutes. 16:30:32 Also the average time of the build between old and new conifguration increased by 20 mins. 16:30:40 So this looks to be consistent. 16:30:49 why the difference in the import time, do you know - just "recent maas chart" vs "older maas chart"? 16:31:06 Yep, I don't see any other difference. 16:31:20 It waits for rack to sync. 16:31:32 Maybe there is some issue with properly configuring timeouts. 16:31:47 It would great if we can find some airship-maas expert to look into that. 16:31:51 ok. well, it is what it is, and if we need to increase timeouts, then we need to increase timeouts -- I don't think we should pin to an old maas just for timing's sake 16:32:03 and agree, better to improve the new stuff than pin to the old stuff 16:32:16 This change may be of help - https://review.opendev.org/#/c/664045/ 16:32:30 still in review phase 16:33:13 The issue with postgresql crashloop is still present by the way. 16:33:23 But it also happens in the old configuration. 16:33:36 Let's see if switching to newer charts helps. 16:33:52 Since they use patroni now for the clustering and configuration. 16:34:09 evgenyl - you and I have different PS out there to integrate new patroni, and I think we need some of each of our PS :) let's sync later today on this 16:34:35 Sure, let's discuss it. 16:34:48 nishantkr - thanks for that one, I added it to the "please review" list 16:35:27 evgenyl, should we target hardening & tuning this week, with a goal of formally switching our "advertised" AIAB over next week? 16:36:18 I think we can start asking people to use it even now, the deployment should be as stable as the new one. 16:36:48 But AIAB repo will still be there, at least until we move multinode installation over to treasuremap. 16:37:05 Yeah, I was just thinking the same thing. I think I do want to try to harden it for a couple days though - when we ask people to switch over, we want to make a good first impression 16:37:09 *as stable as the old one 16:37:58 i.e., let's switch over when it's a clear and above improvement to switch -- if we all beat up on the TM AIAB a bit I bet we can get there this week 16:38:41 michael-beaver: how is work on migrating the multi-node AIAB over to TM going? Sorry, I've lost line of sight on that 16:39:43 It is going alright, but I am stuck at the barbican test in genesis as I am getting some SSL errors when it tries to contact keystone 16:39:57 michael-beaver: Do you need some help with that? 16:40:02 I wanted to start looking into that. 16:40:11 That would definitely be appreciated, would love to sync up on it 16:40:46 that would be awesome - thanks michael-beaver & evgenyl 16:40:54 Yep, sounds good. 16:41:57 Alright! If nothing else on this one, moving on: 16:42:21 #topic consuming openstack services value overrides added for OS distro and openstack release in openstack-helm 16:42:30 arunkant: this one's yours 16:42:56 yes..so I was wondering if there has been discussion in past about this following topic 16:42:57 and maybe related to the "how to best add TF to treasuremap" conversation from before! 16:43:05 So first thing: 16:43:30 Could you please outline what you see as the full set of overrides that would be deployed in this context 16:43:48 So in openstack-helm service charts, there was value overrides were added specific to a OS distro and openstack release as well. 16:43:58 Basically "all of them" across airship components and openstack components, right? 16:44:25 Its specifically for openstack services .. 16:44:58 so we had talked previously about the more narrowly-scoped "how can we do SUSE Airskiff" 16:45:03 as we store config and other related data in per services values file..which can change across openstack release 16:45:14 I think I know the answer, but is there anything else other than images overrides? 16:45:58 Yes..there are configuration changes as well. We noticed in neutron some of rootwrap configuration does not work with rocky release as its changed 16:46:11 Ok, I see. 16:46:23 from the version we have in neutron values.yml 16:47:35 So that's one of the reason these value overrides were added to allow per release changes and then if anything which is different for specific distro, then have override file for that 16:47:35 So, in treasuremap, we have three layers defined: Global (general defaults for charts), Type (tunings for particular use cases), and Site (site-specific overrides) 16:47:41 speaking about review with TF support - overriding is not needed 16:47:43 * redrobot pokes head in at the sound of barbican 16:48:17 The challenge is that when we're talking about overrides for *versions* of things, that's kind a different question than "use case" or "site" 16:48:37 Would be nice to have a "mixin" of overrides on top of a use case or site 16:48:39 It is not a serious suggestion, however maybe we should start looking into what kustomize can do and if we can start leveraging it for these use-cases? 16:48:48 As a part of transition to Airship 2.0. 16:48:59 yes, was thinking the same thing 16:49:02 Alexander Noskov proposed airship/treasuremap master: Move Airship Seaworthy pipeline to the folder. https://review.opendev.org/666048 16:49:30 Jury is still out on kustomize I believe, but especially if we switch toolsets for 2.0 then we should do so with a plan for mixin / service layer type functionality 16:49:38 o/ redrobot 16:50:21 It may be as pre or post pegleg processing step for now, until we have a complete (?) replacement. 16:50:52 Yes, this can be something to think for 2.0 . In the interim, was there any discussion how those value overrides can be consumed on airship side. 16:50:55 For "SUSE Airskiff", we had talked about applying extra overrides on top of the airskiff site manifests at the little-used fourth layer, "cicd" 16:51:34 we could easily follow the same approach for, e.g. the seaworthy site, with the burden of repetition however 16:51:43 as we have a strict tree-model 16:51:49 for our manifest layers 16:53:07 mattmceuen: So with tree-model or a new layer in-between, are we going to duplicate those override configuration in those layer ? 16:53:12 arunkant, I think the answer today is to either create a different `type` which applies all the overrides, unless airskiff is enough from a treasuremap perspective, in which case that fourth layer (cicd) should fit the bill 16:54:41 mattmceuen: I am not aware of cicd layer.. is it documented or sample available somewhere ? 16:55:05 The only way to avoid duplication will be to 1) do some kind of pre- or post- processing (sidestepping the declarative model) like evgenyl suggested, or, implement something like service layering 16:55:18 the most documentation was when we discussed it a couple weeks ago :) 16:55:32 I will drop you a link to the logs arunkant, will need to dig a smidge 16:56:16 But this does in fact tie into the same general question that obravo's TF integration raised 16:56:33 "orthogonal" - that's the word I was trying to think of 16:56:36 mattmceuen: So if we need to add pre- or post-processing logic to avoid duplication , which airship component is best suited for ? 16:56:43 mattmceuen: So if we need to add pre- or post-processing logic to avoid duplication , which airship component is best suited for that ? 16:57:17 airskiff suse gate could be sufficient until we get to 2.0 16:57:28 choice of software, operating system, openstack version, SDN choice... these things are someone (not completely) orthogonal to one another, and don't fit into a hierarchy well 16:57:46 cool. jamesgu__ I'll add this to rodolfo's design call agenda for next week 16:57:56 to at least make sure it's on the radar 16:58:11 k 16:58:39 arunkant: I don't have a really solid idea on that yet, do you evgenyl? 16:59:47 in any case - I think you're caught up on the conversation now arunkant :) more conversation needs to be had 17:00:09 This should be done somewhere in the gate before or after pegleg collect, but how we can exactly use customize for changing the manifests is not completely clear. 17:00:29 This requires some additional research and thinking. 17:00:33 I've captured in our "homework" for the week: 17:00:37 Beat up on the new AIAB in treasuremap 17:00:37 Give some thought to how to incorporate e.g. TungstenFabric and SUSE manifests in to treasuremap 17:00:54 mattmceuen: I have not even thought through about this, just want to see if there are any ideas discussed in past. 17:00:54 would be awesome if everyone can do some of each! 17:01:12 alright guys, we are out of time 17:01:19 can i pop in with one thing 17:01:19 #topic Review Requests 17:01:20 ? 17:01:26 almost! 17:01:29 https://review.opendev.org/#/c/664648/ - Tungsten Fabric integration in AIAB 17:01:29 https://review.opendev.org/#/c/664993/ - Treasuremap [aiab] Fix a path to scripts directory 17:01:29 https://review.opendev.org/#/c/665007/ - Treasuremap [aiab] Add a site linting gate to Zuul 17:01:29 https://review.opendev.org/#/c/665228/ - Treasuremap Uplift PostgreSQL 17:01:29 https://review.opendev.org/#/c/664045/ - Check sync of only active MaaS rack controllers 17:01:33 Please review :) ^ 17:01:47 #topic Hogepodge gets a word in! 17:01:51 go for it hogepodge 17:01:59 OSF is planning on having a booth at KubeCon NA this November 17:02:32 We're having our first planning meeting for it today, but I wanted to make you all aware that we want to have an Airhip presence and to start thinking about things we might want to do there. 17:02:35 Demos, talks, so on. 17:02:43 awesome! 17:02:54 that's really good to hear 17:03:02 If you know that you're going to be going, please let me know so I can coordinate with the rest of the staff. 17:03:09 I'll be there. 17:03:28 I don't know about me yet - need to submit some talks :) 17:04:07 anything we can do to help at this point (before attempting to show up there) hogepodge? 17:05:21 Right now knowing who's going and how we want to handle our presence would be the biggest help. 17:05:42 sounds like a plan - will work toward that sir 17:05:49 thanks for bringing this up 17:06:12 Thanks for joining, everyone, we're out of time! 17:06:17 have a great week :) 17:06:21 #endmeeting