09:03:04 <jakeyip> #startmeeting magnum
09:03:04 <opendevmeet> Meeting started Wed Apr 26 09:03:04 2023 UTC and is due to finish in 60 minutes.  The chair is jakeyip. Information about MeetBot at http://wiki.debian.org/MeetBot.
09:03:04 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
09:03:04 <opendevmeet> The meeting name has been set to 'magnum'
09:03:08 <jakeyip> #topic Roll Call
09:03:25 <dalees> o/
09:03:38 <jakeyip> o/
09:04:00 <travisholton> o/
09:05:16 <jakeyip> Thanks for joining the meeting
09:05:31 <jakeyip> Agenda:
09:05:33 <jakeyip> #link https://etherpad.opendev.org/p/magnum-weekly-meeting
09:05:54 <jakeyip> #topic Supported versions for Bobcat
09:06:19 <jakeyip> we need a decision on what we should officially support for Bobcat
09:07:01 <dalees> when does Bobcat release? K8s 1.24 is end of life 2023-07-28
09:07:08 <dalees> ref: https://kubernetes.io/releases/
09:07:45 <jakeyip> https://releases.openstack.org/ Bobcat releases in 2023-10-04
09:08:33 <jakeyip> due to the differences in containerd / podsecuritypolicy / blah blah, I feel like we should not spend any effort on backwards compatibility with k8s < v1.25
09:09:54 <jakeyip> my rationale is that 1.24 will be EOL soon, and by the time someone installs Bobcat on their infra (which may be a year or more after Bobcat releases), no one will bother with k8s < 1.25
09:10:14 <jakeyip> any effort supporting EOL versions is wasted
09:10:28 <dalees> well, that's a good point. 1.24 is alive now, but doesn't need to be in Bobcat in October.
09:11:15 <dalees> at Catalyst Cloud *we* aim for 3 supported k8s versions, but that'd be covered with 1.25+, and we'd not need to break the older **intentionally.**.
09:11:48 <dalees> I think that's reasonable for 2023-10-04 (and 1.25 EOL's in 2023-10-28).
09:12:59 <dalees> So I'd suggest we add 1.26, 1.27 to the supported list along with 1.25 for Bobcat (I have these running locally, but not certified yet)
09:13:57 <jakeyip> yeah that's the plan, it'll be easier if we decided to support k8s >= 1.25 only. the breaking change is PSP
09:14:15 <jakeyip> I want to support newer versions instead of worrying about backwards compatibility
09:15:09 <jakeyip> we will have to make it super clear it is breaking change in Bobcat and operators should not upgrade if they want to still support k8s <1.25
09:15:57 <jakeyip> I think we have to stray from OpenStack's deprecation workflow as it does not make sense for Magnum
09:15:59 <dalees> Yeah, we need to be moving forward. I do maintain a few older versions of calico in our branch with Heat template `if` statements. We could do similar but it'll bloat the HOT code to keep them long term.
09:16:41 <jakeyip> what's the 3 supported versions now for Catalyst ?
09:17:17 <dalees> I was going to say, as long as there's an overlap in versions. But Magnum doesn't have upgrade paths working, so it's blue/green rebuild anyway. Might as well jump to 1.25 if you're older ;)
09:17:30 <travisholton> 1.23.17, 1.24.12, 1.25.8
09:18:12 <dalees> 1.24+ with containerd
09:18:44 <jakeyip> ok similar for us
09:18:49 <jakeyip> we hope to drop 1.23 soon
09:19:16 <dalees> cool
09:19:34 <jakeyip> ok, similar about FCOS, I think we will also support 1 verrsion only
09:20:01 <jakeyip> I have a review up at https://review.opendev.org/c/openstack/magnum/+/880568
09:20:27 <jakeyip> #agreed support k8s >=1.25 only for Bobcat
09:20:29 <dalees> I think I'm okay with that, the older will likely **work** but not be supported/recommended. Theres only one current "stable" FCOS anyway
09:20:57 <jakeyip> the difference in FCOS is really a pain to work with
09:21:19 <jakeyip> I say document it as well as possible so operators know what verrsion to keep to
09:22:01 <dalees> Yeah, anything older than current stable may have CVEs. So running older is risky
09:23:15 <dalees> but we need to keep it functional the stable stream upstream changes, especially the rebases. So documenting known good versions is useful, for those who may be running older Magnum versions
09:24:09 <jakeyip> yeah good point about CVEs
09:24:18 <jakeyip> so hard to get user to upgrade though :D
09:24:48 <jakeyip> #agreed support only 1 version of FCOS and document it
09:25:57 <jakeyip> any other comments on this topic?
09:26:01 <dalees> similar topic - how  about we document "known good" template labels? We're often incrementing calico versions and the manifests to match, in our codebase. Same would apply to other `kube-system` services
09:26:39 <dalees> (such as cloud provider openstack)
09:27:01 <jakeyip> yeah I plan to have another change up in docs for template labels
09:27:18 <jakeyip> I have some that I test with, but it's not public and it really should
09:27:34 <jakeyip> I was hoping to capture that in tests too
09:27:35 <dalees> great. If we can't change defaults due to breaking templates, then we can define them in docs so there are some known good.
09:28:19 <jakeyip> sure. please help me review to docs changes so we can get them in asap
09:28:29 <dalees> will do
09:28:51 <dalees> you'll have flannel versions, and we'll have some calico versions to share.
09:28:56 <dalees> IIRC
09:30:03 <jakeyip> yeah that'll be great, we can iterate since it's just docs. better something than nothing (current state)
09:31:06 <jakeyip> this one needs help too https://review.opendev.org/c/openstack/magnum/+/874092
09:31:30 <jakeyip> along the same vein I'll deprecate lots of thing
09:31:47 <dalees> sure will do after meeting - i missed your update to that one
09:32:59 <jakeyip> any other topic?
09:33:19 <jakeyip> oh I have something else
09:33:32 <jakeyip> #topic backport to Antelope
09:34:17 <jakeyip> we missed the Antelope deadline to get deprecation in
09:36:26 <jakeyip> also, in related news, the first version of 2023.1 has been released, but magnum isn't working because https://review.opendev.org/c/openstack/magnum/+/880820 didn't make the cut
09:37:12 <jakeyip> so since the first version isn't working, I was thinking of working on some more deprecation notice, get them merged to master and backported to antelope, then release a newer version of antelope that is working
09:37:28 <jakeyip> I wonder what everyone thinks
09:37:51 <jakeyip> this will allow us to remove the code in Bobcat
09:38:17 <dalees> ah - we need to declare the deprecations in previous release, so we can remove them in Bobcat?
09:38:47 <jakeyip> yeah
09:39:22 <dalees> is that reasonable for the community, to merge them now? And which code were you thinking of?
09:41:04 <mnasiadka> i think it's reasonable
09:42:29 <jakeyip> it'll be something like this https://review.opendev.org/c/openstack/magnum/+/833949
09:43:20 <dalees> oh, yeah.
09:45:15 <dalees> that's reasonable to backport, i feel.
09:45:20 <jakeyip> I want to deprecate swarm
09:45:47 <jakeyip> since fedora atomic is eol
09:46:08 <jakeyip> that review I posted made it into antelope so we can remove fedora atomic in bobcat
09:46:15 <mnasiadka> makes a lot of sense, I guess nobody would complain if we deprecated and dropped, but since we Antelope is fresh, we can backport the deprecate notice and drop in B
09:46:32 <jakeyip> but I also want to do the same for swarm and backport to Antelope
09:47:05 <mnasiadka> regarding https://review.opendev.org/c/openstack/magnum/+/874092 - so Bobcat will not support 1.24 and earlier, makes sense - how do we go about backporting this to stable branches?
09:47:37 <mnasiadka> jakeyip: from my perspective - we should end up with fedora coreos k8s driver only (and CAPI once it gets merged) - sooner the better
09:48:04 <jakeyip> yes that's why I want to drop as much as possible; it'll make other reviews easier too
09:48:49 <jakeyip> like the RBAC review from Rico is handling things like bay / baymodel that we want to drop in https://review.opendev.org/c/openstack/magnum/+/803780
09:49:23 <dalees> Removing PSP usage wouldn't break older versions, only if users of these versions were using that admission controller. We can decide between backporting that to Antelope, or Antelope never supporting k8s 1.25.
09:49:51 <dalees> mnasiadka: +1, I'm aligned with that driver viewpoint.
09:50:49 <jakeyip> dalees: do you mean Antelope never supporting k8s <1.25 ?
09:51:04 <jakeyip> instead of never supporting 1.25 :D
09:51:34 <dalees> jakeyip: if we don't backport that PSP change to Antelope, 1.25 won't work, will it? (because it'll be trying to create PSPs)
09:51:53 <jakeyip> oh ok sorry from different sides, I get you now
09:52:06 <jakeyip> unfortunately Antelope probably will not support 1.25 and above
09:52:59 <dalees> right - i guess if we backported it, the breaking change moves back too. and we don't want to do that.
09:53:00 <jakeyip> because our release notes says 1.24 and removal of dockershim. https://docs.openstack.org/releasenotes/magnum/2023.1.html
09:53:40 <dalees> ok that's cool. I think that answers mnasiadka's question - we don't backport it.
09:53:48 <jakeyip> mnasiadka: is that ok?
09:55:05 <mnasiadka> well, I guess from our users perspective that's a bit problematic, I'll think of somehow checking the kubernetes version in the image and basic PSP enablement on the version we get (and push that upstream), but for now it's ok ;)
09:55:56 <jakeyip> I was working on that, but it got complicated very quickly due to how tightly coupled things are
09:56:50 <mnasiadka> Understand, I don't know if I'll have time for such mission, but we'll see
09:57:12 <mnasiadka> Another stupid question out of the blue - Heat is deprecating SoftwareDeployment - do we need to do anything?
09:57:36 <dalees> implement ClusterAPI? :)
09:57:40 <jakeyip> I've answered that we still need it ?
09:57:49 <jakeyip> lol when is CAPI coming...
09:58:03 <mnasiadka> And assume that unicorns and rainbows will fix everything? nice
09:58:10 <dalees> haha
09:58:16 <jakeyip> k8s is supposed to fix everything
09:58:25 <mnasiadka> It's only bringing more complexity :)
09:58:43 <jakeyip> I feel conned
09:59:00 <jakeyip> ok we are almost at time, any other thing?
09:59:07 <mnasiadka> Ok, let's see what Heat comes up with - is there any alternative in Heat we could move into?
10:00:18 <dalees> Not this week, but I want to discuss CAPI for Bobcat sometime. We'll be focusing some effort on it
10:00:38 <dalees> do we need it as a goal, or can it go in if it's ready?
10:02:45 <jakeyip> I am not sure, will need updated from John or the stackhpc people.
10:03:26 <jakeyip> I can see them working on it so it's not abandoned. Plus they have to present at Vancouver! :)
10:04:17 <jakeyip> dalees: ping him if you want to help?
10:05:19 <dalees> jakeyip: yep; travisholton and I will do.
10:05:51 <jakeyip> cool, we are out of time.
10:06:00 <dalees> thanks all
10:06:03 <jakeyip> #endmeeting