09:00:34 <jakeyip> #startmeeting magnum
09:00:34 <opendevmeet> Meeting started Wed Feb 15 09:00:34 2023 UTC and is due to finish in 60 minutes.  The chair is jakeyip. Information about MeetBot at http://wiki.debian.org/MeetBot.
09:00:34 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
09:00:34 <opendevmeet> The meeting name has been set to 'magnum'
09:00:45 <jakeyip> #topic Roll Call
09:00:51 <jakeyip> o/
09:01:09 <dalees> o/
09:01:30 <travissoto> o/
09:02:13 <jakeyip> #link https://etherpad.opendev.org/p/magnum-weekly-meeting
09:02:26 <jakeyip> Please feel free to populate the agenda
09:03:21 <jakeyip> hi travissoto
09:03:32 <jakeyip> thanks everyone for coming to the meeting. feel free to join in at anytime.
09:03:39 <jakeyip> #topic PTG
09:04:00 <travissoto> hi all
09:04:24 <jakeyip> There are two PTGs this time, (1) Vitual PTG (March 27-31)   (2) OpenInfra Summit + PTG (June 13-15)
09:05:04 <jakeyip> does everyone have a preference for PTG?
09:05:47 <jakeyip> unfortunately I probably will not be able to make it to OpenInfra
09:06:28 <jakeyip> if there is no hard preference I may book something for March and see if there will be interest closer to the date
09:06:29 <dalees> likewise, virtual is preferred this time for me.
09:06:45 <travissoto> +1
09:07:17 <jakeyip> ok
09:07:27 <jakeyip> #action jakeyip to book Virtual PTG
09:07:48 <jakeyip> #topic Antelope supported versions
09:08:30 <jakeyip> Thanks everyone for the patches to make FCOS 36-37 work
09:08:39 <jakeyip> and also k8s v1.24
09:09:26 <jakeyip> A common issue for user is that they are unsure which versions of FCOS / K8S is supported. For that I have recently fixed up the docs to reflect that
09:09:40 <jakeyip> #link https://docs.openstack.org/magnum/latest/user/#supported-versions
09:10:28 <jakeyip> I would like to propose that we say FCOS 36/37 + k8s v1.24 is supported for this cycle.
09:10:35 <jakeyip> what does everyone think?
09:10:39 <dalees> I've also passed conformance for 1.25, and had 1.26 mostly running (but I have not reviewed kube-system pod versions yet).
09:10:54 <dalees> sounds good, 1.24 is still supported k8s version.
09:11:24 <jakeyip> dalees: does default labels work ?
09:11:59 <dalees> jakeyip: unlikely, that was the other topic I'd like to discuss. We bump a large number of things and the defaults are way out of date now (calico etc)
09:12:18 <jakeyip> dalees: yeah that is a problem. lots of discussion needed there :)
09:12:40 <jakeyip> ok if we are in agreement let's just target 1.24 and 1.25 as stretch goal ;)
09:13:01 <jakeyip> #info Antelope supported version FCOS 36/37 and Kubernetes v1.24
09:13:03 <dalees> maybe we can share (or update) working template labels if we have the 1.24 locked in. it's hard to know to update the default without the version (1.24) set in place.
09:13:48 <jakeyip> we can target tests for these versions, and we can have labels that work possibly out of the box (later discussion)
09:13:59 <jakeyip> great :)
09:14:12 <jakeyip> #topic Deprecation
09:15:12 <jakeyip> As Antelope is about to come to an end, I am in a bit of a hurry to mark things as deprecated this cycle, so to allow them to be removed in 2 cycles' time
09:16:04 <jakeyip> #topic Deprecate Fedora Atomic for Kubernetes
09:16:33 <jakeyip> #link https://review.opendev.org/c/openstack/magnum/+/833949 I see a few +1, I think we can get this in this cycle. Thanks dalees :)
09:16:45 <jakeyip> #topic Deprecate Swarm
09:16:57 <jakeyip> is anybody still using swarm? or not?
09:17:08 <dalees> not us.
09:17:09 <jakeyip> we are not using Swarm at all, and I'm not even sure it works.
09:18:16 <jakeyip> OK if there is someone using Swarm please feel free to email me. If not I will drop a mail on the ML to see who is using and wants to take up maintenance
09:18:35 <jakeyip> #action jakeyip Propose deprecation of Swarm to ML
09:19:00 <dalees> +1 needs a mailing list post, and then propose removal if it's not relevant anymore.
09:19:23 <jakeyip> :)
09:19:50 <jakeyip> if we can get Fedora Atomic out there is lots of code we can remove
09:20:46 <jakeyip> #topic python-magnumclient intermittent failures after tox4
09:21:24 <jakeyip> So I tried updating to tox4 format, but weirdly I am getting intermittent failures running `tox -e py38` with those changes.
09:21:39 <jakeyip> it fails in check too
09:22:13 <jakeyip> we have patches stuck because of this. if someone can help it'll be great
09:23:51 <jakeyip> alright we went through the previous items pretty quickly, are there questions for those items, dalees / travissoto  ?
09:24:27 <dalees> i'll see if i can look into those failures with magnumclient, the "intermittent" part is concerning.
09:24:40 <travissoto> no not from me at this stage :)
09:24:56 <jakeyip> thanks dalees
09:25:27 <jakeyip> dalees: do you want to discuss prometheus helm chart now?
09:26:10 <dalees> yeah, sure.
09:26:13 <jakeyip> #topic Prometheus helm charts
09:26:50 <dalees> so the prometheus/grafana stack is installed into kube-system namespace with helm if monitoring_enabled is set.
09:27:00 <dalees> and this breaks in 1.22
09:27:39 <dalees> we(actually, travissoto ) replaced the helm charts with the newer, completely different ones from kube-prometheus-stack.
09:28:03 <dalees> do others want or use these, or should we keep this patch local and remove the complexity from Magnum?
09:29:53 <jakeyip> I think I tried the default one and gave up :)
09:30:18 <jakeyip> dalees: do you install it for all your users?
09:30:41 <dalees> our templates enable it by default yes, but many turn it off and install their own monitoring stacks.
09:31:39 <jakeyip> yeah ok
09:32:11 <jakeyip> we don't install it by default IIRC and I suspect users might prefer their own
09:32:29 <dalees> as we refactor to CAPI, we're going to consider if and how we keep it. It's a big job to keep it maintained in Magnum codebase.
09:33:03 <jakeyip> of cos having more things work out of the box is great for users, but maintaining them up to date is an issue
09:33:30 <jakeyip> and the more is in the codebase, the more we are responsible for testing
09:34:05 <jakeyip> we don't have good test for that now (?) so merging changes will be difficult without tests
09:35:19 <dalees> yeah, with k8s 1.24 as supported i suspect that won't work out of the box.
09:35:29 <jakeyip> I propose that we should remove it if it is broken. What does everyone think?
09:35:59 <travissoto> agree better to remove it
09:36:44 <dalees> ok by me
09:36:56 <dalees> lets confirm it's broken first.
09:37:24 <dalees> (it is for us, but i want to be sure it's not our local patches)
09:37:36 <jakeyip> OK. can you help to confirm in devstack and send up a change to remove it if it is?
09:37:57 <jakeyip> I may take a look later too
09:38:13 <dalees> ok
09:39:35 <jakeyip> #action dalees / travissoto to confirm prometheus helm chart is broken, and propose patch to remove if it is
09:39:46 <jakeyip> #topic Container Labels
09:39:53 <jakeyip> big topic ;)
09:40:31 <jakeyip> I guess executive summary is: default labels may be broken, but updating them may break existing cluster templates that do not set them
09:40:39 <jakeyip> what are our options?
09:42:42 <jakeyip> oh oh
09:43:50 <dalees> yep, that's pretty difficult. making a template that relies on defaults that may change isn't a great experience.
09:44:46 <dalees> we've a similar issue with manifests, if someone updates the calico manifests for version v1.23 and we're still allowing users to create calico v1.13 their new clusters all break.
09:46:33 <dalees> I've resolved this by copying all templates that change and picking them up with version matches (as seen in https://github.com/openstack/magnum/blob/master/magnum/drivers/k8s_fedora_coreos_v1/templates/kubecluster.yaml#L1424 )
09:47:37 <jakeyip> look on the bright side, new cluster breaking is better than current cluster breaking
09:47:41 <dalees> back to labels topic though - i think defining as many labels as possible in a template helps, which is what we end up doing. it means the defaults don't apply.
09:48:16 <jakeyip> that's what we do too because the defaults are just too old
09:49:46 <dalees> so who has the problem of user templates breaking? do we just run with it and update them to match the current k8s (1.24 right now)? and produce example templates that can be published with little chance of breaking?
09:50:29 <jakeyip> I did have our organisation templates broken before, that's why I learnt to pin as many labels as possible
09:51:04 <dalees> you mentioned breaking existing clusters - how are these labels ever re-applied to a running cluster? the upgrade or scaling process doesn't do it (if we're talking kube-system container images). Existing Heat stacks stay the same.
09:51:52 <jakeyip> yeah sorry I mean existing cluster _templates_, I might have typo
09:52:14 <dalees> ah ok; just checking I understood properly
09:53:14 <dalees> this type of problem may not go away with CAPI. we still need some concept of cluster templates.
09:55:00 <jakeyip> yeah, I feel the least disruptive is to leave the defaults alone and document what works for the current versions
09:55:55 <dalees> but then you sacrifice the "works out of the box" experience, if that is  the goal.
09:56:17 <jakeyip> updating the labels in code is an impossible task. we can push it to latest in Antelope, but by the time an organisation installs / upgrade to Antelope they will be out of date already
09:56:23 <dalees> you could remove all defaults and force them to be specified in template labels :)
09:56:30 <jakeyip> :)
09:56:44 <jakeyip> for CAPI? :)
09:57:16 <dalees> it's worth considering yeah.
09:57:33 <dalees> then you only maintain versions in once place, and they match the k8s version
09:58:08 <jakeyip> yeah agree much nicer
09:58:47 <jakeyip> I guess what we can do better is document it. I've heard complains :)
09:58:59 <jakeyip> we are almost out of time. any other topic?
09:59:10 <dalees> keen to hear others' ideas on the topic, who aren't in the meeting but involved in Magnum.
09:59:43 <dalees> I've got some for another week, but they can wait. thanks for the discussion
10:00:07 <jakeyip> me too. let's hold this regularly and more may join
10:00:38 <jakeyip> Thanks dalees and travissoto for coming
10:01:03 <jakeyip> #endmeeting