09:01:03 <jakeyip> #startmeeting magnum
09:01:03 <opendevmeet> Meeting started Wed Apr  3 09:01:03 2024 UTC and is due to finish in 60 minutes.  The chair is jakeyip. Information about MeetBot at http://wiki.debian.org/MeetBot.
09:01:03 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
09:01:03 <opendevmeet> The meeting name has been set to 'magnum'
09:01:09 <jakeyip> #link https://etherpad.opendev.org/p/magnum-weekly-meeting
09:01:21 <jakeyip> #topic Roll Call
09:01:23 <jakeyip> o/
09:01:39 <jakeyip> mnasiadka dalees courtesy ping :)
09:01:53 <dalees> o/
09:03:18 <jakeyip> #topic Abandon old patches
09:03:31 <jakeyip> I've abandoned many old patches as agreed
09:03:44 <jakeyip> some I've starred for further reviews but haven't had time to action all of them
09:03:50 <dalees> Thanks for doing that, jakeyip
09:06:03 <jakeyip> feel free to abandon any earlier than 2022-10-05 as previously agreed
09:06:20 <jakeyip> anything else on this topic?
09:08:04 <jakeyip> I'll go on to the next one
09:08:13 <jakeyip> next topic
09:08:26 <jakeyip> #topic PTG
09:08:51 <jakeyip> Wed, 10 Apr, 06-08 UTC (next week)
09:09:05 <jakeyip> mark your cal, see you all there  :)
09:11:11 <jakeyip> anything on PTG to chat about?
09:11:48 <dalees> nothing from me, but see you there
09:11:48 <mkjpryor> I guess all the Cluster API stuff
09:12:08 <mkjpryor> Just to make sure we all understand the plan, and agree as much as possible
09:12:14 <jakeyip> #topic ClusterAPI
09:12:18 <jakeyip> mkjpryor: go ahead :)
09:12:33 <mkjpryor> So we have moved the Helm driver to opendev now
09:13:04 <mkjpryor> As discussed before, we are still hoping for in to be in-tree in Dalmation
09:13:30 <mkjpryor> But I think that is one part that definitely needs discussing at the PTG
09:14:02 <mkjpryor> There are a few supporting components that we would also like to contribute to the Magnum project, but are currently quite dependent on GitHub Actions for their CI
09:14:10 <mkjpryor> In particular, the Helm charts themselves
09:14:27 <mkjpryor> But also our addon provider and the janitor that we use to do cleanup of OCCM resources
09:14:42 <mkjpryor> They will take more work to move to opendev
09:15:28 <mkjpryor> So, in summary, our current plan is for the Magnum project to own the Helm driver, the charts themselves, the addon provider and the janitor
09:16:07 <mkjpryor> Whether the Helm driver is in-tree or not, or just the Cluster API driver that is actually owned by the Magnum project in a separate repo, needs discussion
09:16:16 <mkjpryor> Our preference is for an in-tree Cluster API driver
09:16:22 <jakeyip> I think we should drop the in-tree discussion this round, let's get it working out-of-tree properly first. being out-of-tree gives us good advantage to iterate without the massive change chain I was grappling with previously.
09:16:47 <jakeyip> we need to get the CI working
09:16:54 <mkjpryor> This is all true
09:17:40 <mkjpryor> Although the "change chain" this time would just be an import of the driver as-is right? We wouldn't be doing it in increments like before.
09:18:28 <mkjpryor> I think mnasiadka has made some progress with the CI
09:19:09 <jakeyip> yeah we can discuss if and how to get it in when we clear the current obstacles
09:19:45 <jakeyip> on the same topic, I had some issues running it, so I was wondering if it's being used in the current iteration anyway
09:20:17 <mkjpryor> We should have somewhere to raise and discuss issues that isn't this meeting
09:20:29 <jakeyip> my issue is a mismatch version of autoscaler + kubernetes. I can fix it, but without CI it's not useful.
09:20:39 <jakeyip> so CI first
09:21:17 <mkjpryor> We have seen basically no issues with mismatched autoscaler and kubernetes versions. There may be something more sinister going on.
09:21:34 <jakeyip> mkjpryor: bugs will be opened in Launchpad as per goverance I think?
09:21:39 <mkjpryor> Yes
09:22:06 <mkjpryor> But in terms of chatting about general usage? Maybe bugs in launchpad that don't actually turn out to be bugs are fine
09:22:27 <jakeyip> we can chat here or in the Kubernetes Slack.
09:24:06 <dalees> mkjpryor: there is a channel #openstack-magnum on Kubernetes Slack. It's not very active, but that is a good persistent chat location perhaps.
09:24:24 <mkjpryor> Sounds good
09:24:34 <mkjpryor> Slack works for me
09:25:58 <jakeyip> it's early days not sure where the majority of the chat will end up. both sides have their advantages. feel free to DM me on Slack if you need me, the notification is a bit better there :)
09:26:46 <jakeyip> if you can stay after the meeting, it'll be great so I can post more details about the autoscaler issue
09:26:56 <jakeyip> mkjpryor ^
09:27:22 <mkjpryor> I can do that
09:27:26 <jakeyip> thanks
09:28:41 <jakeyip> mkjpryor: for your other points, the helm chart already has a repo `openstack/magnum-capi-helm-charts`, but is currently empty
09:29:08 <jakeyip> it's set up for the same governance as `openstack/magnum-capi-helm`
09:29:10 <mkjpryor> Of all the projects, that is the one that is most heavily reliant on GitHub Actions
09:29:18 <mkjpryor> The CI will be difficult to port
09:29:27 <mkjpryor> And will take time, which we do not currently have
09:30:32 <jakeyip> can you elaborate on the difficulties? maybe others can help
09:31:03 <mkjpryor> Just we rely on a lot of actions to do things that would need to be replicated in another way
09:32:15 <jakeyip> ok, anything that's technically not possible in zuul?
09:32:40 <mkjpryor> I don't think so
09:33:09 <mkjpryor> We also use images built for Azimuth in the CI, which might be politically not good :shrugs:
09:33:47 <mkjpryor> I'm not sure whether the Magnum project wants to build and ship images
09:33:58 <mkjpryor> Probably not
09:34:00 <jakeyip> ok, someone has to push that... StackHPC is probably best suited to do it...
09:34:20 <mkjpryor> We are, but we are also small and busy
09:34:38 <jakeyip> don't have to replicate ALL the CI too, a small subset is fine
09:34:39 <mkjpryor> This is the issue with open-source that isn't fully funded by a customer right
09:35:45 <jakeyip> yeah same deal for most of us here... it'll be worse if we can't work together right? :D
09:37:27 <jakeyip> if we can get it working, we can attract more people to help with maintenance. the initial hump is hard to get it up enough for others to see it as viable.
09:37:29 <mkjpryor> Of course, but there is a significant chunk of initial effort that is required from us
09:37:41 <jakeyip> I'm struggling with this being the PTL.
09:38:22 <mkjpryor> We have to offset this effort against customer work as well. It is tricky.
09:38:47 <mkjpryor> Also, we already have a set of things that work for us so there is inertia there as well.
09:39:00 <mkjpryor> But we do want to get there
09:39:37 <jakeyip> yeah can understand. but this is critical get us from the OpenStack world to the Kubernetes world. That's why I'm still doing this...
09:42:06 <jakeyip> anyway for the helm chart, I think we need it in governance and CI so that we can strip it down as a minimal product that can be extended by operators. that's what I see is good about this
09:43:13 <mkjpryor> I agree. That is definitely a strength of the approach for sue.
09:43:16 <mkjpryor> sure*
09:43:39 <jakeyip> FYI it's a point of contention with TC if the driver references the helm chart managed in stackhpc github, so I don't think we have much room on this.
09:44:13 <jakeyip> this driver sure got many people's attention now :)
09:44:39 <mkjpryor> So we talked about setting up a GitHub org for Azimuth the project, because we also want that to exist independently of StackHPC, and the charts could live there
09:44:54 <mkjpryor> There is definitely precedent for using components from other open-source projects
09:45:05 <mkjpryor> So maybe that is more palatable as an interim solution
09:45:34 <mkjpryor> After all, it is just an external component that can be easily replaced
09:46:01 <mkjpryor> I do understand that people have an issue with the default coming from a specific vendor
09:46:05 <jakeyip> I don't understand this approach, is it so that the GH actions can be ported more easily from SHPC to Azimuth org? and it is harder to port it to Zuul ?
09:46:56 <mkjpryor> Well - we want somewhere for Azimuth to live that is independent of StackHPC. No offence to Gerrit, but I wouldn't choose to use it unless I really had to.
09:47:14 <mkjpryor> So the new GitHub org is primarily for that
09:47:51 <mkjpryor> But it would have the side-effect that the CAPI Helm charts would now be an Azimuth sub-project rather than a "StackHPC product" (which we don't consider them to be now, but others clearly do)
09:48:15 <mkjpryor> And yes - the CI would "just work" (modulo setting up some CI vars) in the new org
09:48:24 <mkjpryor> So it is easier than porting to Zuul in that sense
09:50:41 <jakeyip> It's a departure from what was originally proposed, I'm not sure if it will be agreeable to the many parties
09:50:46 <mkjpryor> Just a thought on an alternative way to get to a palatable solution for the short term that doesn't rely on a large time investment from us that we might not be able to commit to
09:51:01 <mkjpryor> Long term, we definitely want the charts under Magnum I think
09:51:54 <mkjpryor> If only we could get a customer to fund "moving the CAPI Helm charts under the governance of OpenInfra"
09:52:03 <mkjpryor> :chuckles:
09:52:53 <mkjpryor> Honestly, the time and availability of the right people to do the work is the major blocker here
09:53:10 <mkjpryor> Especially when something exists that works nicely
09:53:49 <mkjpryor> Anyway - I think we are saying the same thing. We want the charts under Magnum governance.
09:54:17 <mkjpryor> But tensioning the time required to do that against the work that pays the bills is difficult.
09:54:20 <jakeyip> I'm looking at the actions now https://github.com/stackhpc/capi-helm-charts/actions , a bunch of them don't have any runs, is there like a minimal thing we need to get it working? keep in mind we don't need ALL the CI
09:54:42 <jakeyip> I think a big issue is StackHPC shifting to use the charts under Magnum governance but that doesn't have to happen
09:55:41 <mkjpryor> So at the moment, because they use our test infra, all the runs require manual approval which is only given after the changes have been reviewed
09:56:01 <mkjpryor> So there are a lot of runs that stay in the pending state and never get executed, yes
09:56:27 <mkjpryor> That would obviously change with Zuul
09:57:25 <jakeyip> I'm seeing like https://github.com/stackhpc/capi-helm-charts/actions/workflows/ensure-capi-images.yaml doesn't have any run
09:57:31 <jakeyip> https://github.com/stackhpc/capi-helm-charts/actions/workflows/lint.yaml
09:57:32 <mkjpryor> Oh
09:57:49 <mkjpryor> A lot of those workflows are reusable workflows that are called from other workflows
09:57:52 <mkjpryor> Not run directly
09:58:09 <mkjpryor> For example, those two you mention are called from main.yaml and pr.yaml
09:58:18 <jakeyip> I see.
09:58:41 <mkjpryor> This is what I mean by it isn't going to be trivial to port
09:59:29 <opendevreview> Merged openstack/magnum master: CI: Use Calico v3.26.4  https://review.opendev.org/c/openstack/magnum/+/911577
10:00:09 <jakeyip> should I start from main.yaml to get an idea what needs to be ported?
10:01:13 <mkjpryor> Yeah
10:01:25 <mkjpryor> main.yaml runs the most minimal set of tests that we have
10:02:06 <jakeyip> ok I'll take a look
10:02:20 <mkjpryor> Another thing we do is mirror all the required images to https://quay.io/organization/azimuth, and use that as a registry mirror for each of the major registries
10:02:23 <jakeyip> I have a question - how do we get the chart built and hosted on opendev?
10:02:51 <mkjpryor> mnasiadka and I spoke about this before
10:03:06 <mkjpryor> Probably the easiest place to host the chart would be on Artifact Hub
10:03:20 <mkjpryor> https://artifacthub.io/
10:03:43 <mkjpryor> So I guess there would be a Zuul job that ran the Helm packaging and pushed the resulting chart to there
10:04:06 <mkjpryor> But you don't need to package the chart to test it
10:04:16 <mkjpryor> Probably getting the CI working is more pressing
10:05:33 <jakeyip> yeah
10:06:30 <jakeyip> ok I need to end meeting cos overrun
10:06:52 <jakeyip> dalees / mnasiadka / mkjpryor: anything else we need to capture for meeting?
10:09:11 <jakeyip> ok I'll end it
10:09:14 <jakeyip> #endmeeting