14:00:39 <bbezak> #startmeeting kolla
14:00:39 <opendevmeet> Meeting started Wed Mar 27 14:00:39 2024 UTC and is due to finish in 60 minutes.  The chair is bbezak. Information about MeetBot at http://wiki.debian.org/MeetBot.
14:00:39 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
14:00:39 <opendevmeet> The meeting name has been set to 'kolla'
14:00:46 <bbezak> #topic rollcall
14:00:49 <mmalchuk> o/
14:01:09 <kevko> \o/
14:01:10 <SvenKieske> o/
14:01:13 <mattcrees> o/
14:01:22 <r-krcek> o/
14:01:32 <mhiner> o/
14:01:48 <jovial> o/
14:01:49 <frickler> \o
14:02:35 <bbezak> #topic agenda
14:02:37 <bbezak> * Roll-call
14:02:37 <bbezak> * Agenda
14:02:37 <bbezak> * Announcements
14:02:37 <bbezak> * Review action items from the last meeting
14:02:37 <bbezak> * CI status
14:02:38 <bbezak> * Release tasks
14:02:38 <bbezak> * Regular stable releases (first meeting in a month)
14:02:40 <bbezak> * Current cycle planning
14:02:40 <bbezak> * Additional agenda (from whiteboard)
14:02:42 <bbezak> * Open discussion
14:03:26 <bbezak> #topic CI status
14:03:53 <bbezak> watermelon?
14:04:09 <frickler> red on the inside? ;)
14:04:11 <SvenKieske> watermelon? :D
14:04:15 <SvenKieske> ah! lol
14:04:18 <bbezak> :)
14:04:20 <SvenKieske> didn't get that
14:04:37 <mmalchuk> green outside, red inside))) lol
14:04:43 <bbezak> exactly
14:04:52 <bbezak> not so bad nowadays I think
14:05:03 <SvenKieske> well at least kolla build failrate seems flat since 24.03.
14:05:30 <SvenKieske> wasn't the dashboard supposed to be fixed? I see "no data" again: https://grafana.opendev.org/d/c0d59dad13/kolla-failure-rate?orgId=1
14:05:55 <SvenKieske> for k-a source graphs, kolla graphs are fine
14:06:06 <bbezak> I think it was working last time I checked
14:06:14 <frickler> I didn't look at it in ages, no idea
14:06:32 <SvenKieske> I'm pretty sure as well it worked, I'm to blame for not looking often enough I guess. :D
14:06:49 <bbezak> ok, let's look into that
14:07:19 <frickler> I haven't seen too many complaints from kevko recently, so CI can't be too red in general, I'd say ;)
14:07:39 <bbezak> yeah, I think it looks good
14:07:58 <kevko> :D
14:08:13 <kevko> centos build is broken
14:08:21 <kevko> but it's non-voting
14:08:23 <SvenKieske> mhm, can I ask something about CI? I know we plan to extend ARM support downstream, ARM pipeline still seems rather flaky
14:08:45 <SvenKieske> and is non voting, if it's in good shape now, could we make it voting?
14:08:58 <frickler> seems there was an issue with centos mirrors earlier today, jrosser was also complaining, should be better now
14:09:28 <frickler> for ARM builds we only have a single cloud provider, that's mostly why we consider it not too reliable
14:10:29 <bbezak> #topic Release tasks
14:10:43 <bbezak> we're in R-1!
14:11:05 <mnasiadka> People running and screaming
14:11:13 <bbezak> :)
14:11:17 <bbezak> https://docs.openstack.org/kolla/latest/contributor/release-management.html#r-1-prepare-kolla-kolla-ansible-for-rc1-stable-branch-creation
14:11:43 <frickler> bit early for branching, isn't it?
14:11:53 <mnasiadka> It says prepare
14:12:04 <mnasiadka> And we will be prepared for next 3 months
14:12:10 <bbezak> let's start to prepare then
14:12:17 <bbezak> I can take a look
14:12:33 <SvenKieske> well the UCA and RDO stuff got merged afaik
14:12:45 <mnasiadka> So let’s have a look what is missing
14:12:53 <mnasiadka> And we need a volunteer to fix Octavia
14:13:10 <kevko> octavia ? what is with octavia ?
14:13:17 <kevko> *wrong
14:13:19 <SvenKieske> do we have a bugreport for octavia? I have only a fuzzy memory what exactly is broken
14:13:36 <mnasiadka> kevko: CI is failing on cascade delete of LB
14:13:45 <kevko> aaaaa, i saw this
14:13:47 <mmalchuk> maybe forgot anything in A-C-K ? again
14:14:41 <bbezak> any volunteer to take a look into octavia in CI ?
14:14:45 <SvenKieske> now, is this a release task? or just regular bugfix?
14:14:57 <mnasiadka> Exploratory surgery
14:15:00 <bbezak> regular bugfix needed for release
14:15:01 <bbezak> :)
14:15:02 <SvenKieske> :D
14:15:08 <kevko> bbezak: i will probably because i am playing with ci and now everything is almost working.... so i am playing around ...
14:15:21 <bbezak> cool
14:15:29 <bbezak> shout if you need help
14:15:39 <kevko> ok ..
14:15:48 <SvenKieske> I got a question regarding the release tasks posted above, it has this gem: "The release notes for each project should be tidied up" and I have no idea what that means even after following the review examples there.
14:16:17 <mnasiadka> It means read through all renos, fix obvious errors, remove duplicates
14:16:18 <SvenKieske> it seems there are mainly old release notes being deleted? I'm not really sure, is this obsolete?
14:16:44 <mnasiadka> So it looks readable for a random person
14:16:53 <SvenKieske> ah okay. weird. I would think this is part of normal code review, but okay, thanks for the explainer :)
14:17:15 <mnasiadka> We have sort of a rule to not obsess over renos in code review
14:17:21 <mnasiadka> And fix it close to release
14:17:36 <jovial> And mistakes do sneek in :)
14:17:38 <mnasiadka> To improve the overall merge flow or something like that
14:18:03 <frickler> some things are also easier to spot with a global view, like consistent formatting, e.g. use ``code`` where applicable
14:18:07 <SvenKieske> ok, I guess I missed those fixes then the last few releases :) thanks for the explanation, I just didn't know how to interpret "tidied up".
14:18:23 <mnasiadka> Ok, I’d like to discuss the ptg schedule if bbezak will permit
14:18:34 <bbezak> I'll think about it
14:18:50 <mnasiadka> I can’t do any other day that Monday, bbezak is on vacation
14:18:53 <bbezak> ok, granted
14:18:54 <bbezak> #topic PTG planning
14:19:11 <mnasiadka> So my proposal is to do Kolla/Kolla-Ansible on Monday for 4 hours
14:19:44 <mnasiadka> And follow up on not discussed topic on the week after if needed
14:20:31 <mnasiadka> Unless frickler or kevko feel an urgent need to drive the Kolla PTG on Tue
14:20:36 <kevko> hmm, we have state holiday :/
14:20:51 <kevko> but i will probably join ..
14:20:57 <mnasiadka> kevko: on Monday?
14:21:14 <kevko> let me check
14:21:28 <kevko> i am not sure
14:21:29 <SvenKieske> that would be April, 8th
14:21:39 <frickler> I'm fine with skipping Tue
14:22:27 <kevko> okay, we don't :)
14:22:35 <mnasiadka> Ok, so that’s agreed - Monday only
14:22:40 <kevko> +1
14:22:45 <mnasiadka> jovial: want to move Kayobe?
14:22:53 <mnasiadka> It’s currently planned for Thursday
14:23:09 <jovial> I would prefer Wednesday if we did move it. But does that work for others?
14:24:08 <SvenKieske> does somebody adjust the ptg schedule then? so people are informed?
14:24:59 <mnasiadka> I will adjust and send mail to Ml
14:25:18 <mmalchuk> all this because we skip doodle service)
14:25:23 <mnasiadka> jovial: Kayobe is mainly SHPC - just make sure key people will attend and I’ll move
14:25:39 <jovial> I guess if Kayobe was moved to the Tuesday then we'll avoid the normal kolla IRC meeting
14:25:48 <mnasiadka> mmalchuk: doodle made no sense because you all chose the same slots we used for ages
14:26:12 <mmalchuk> maybe or maybe not
14:27:01 <mmalchuk> anyway we change slots
14:27:39 <bbezak> ok, anything else PTG related?
14:28:37 <bbezak> as a reminder a link to etherpad - https://etherpad.opendev.org/p/kolla-dalmatian-ptg
14:30:02 <bbezak> let's move on
14:30:18 <SvenKieske> #Link https://etherpad.opendev.org/p/kolla-dalmatian-ptg
14:30:18 <bbezak> #topic Current cycle planning
14:31:27 <bbezak> does anybody has something to discuss about outstanding tasks?
14:32:09 <bbezak> I was overloaded with customer related tasks recently, so no much update from me I'm affraid
14:32:40 <kevko> I want to ask to merge init-runonce modification to stable branches
14:32:57 <kevko> :)
14:33:28 <SvenKieske> well looking at the caracal tasks not that many are done afaik?
14:34:17 <SvenKieske> anybody has any news on the next ubuntu release? we wanted to build that in for C release
14:34:47 <kevko> 25.4 ? :D
14:34:53 <mmalchuk> nope... silent
14:34:55 <SvenKieske> no, 24.04.
14:35:00 <bbezak> kevko: I've seen a bit discussion the other day. I don't think it is a good idea to modify stable branches without doing the same to master?
14:35:08 <kevko> google saying Ubuntu 24.04 LTS, codenamed Noble Numbat, is scheduled for release on April 25, 2024.
14:35:29 <mmalchuk> SvenKieske ask again afer 25 apr
14:35:39 <mmalchuk> after*
14:35:56 <kevko> bbezak: yes, i want to merge also to master ...same patch ...but it's needed also to merge to stable branches ..as upgrade jobs are checkout ing previous release ...
14:36:02 <SvenKieske> but why was it put on the agenda then? I mean the release dates of ubuntu are pretty well known in advance? weird.
14:36:37 <kevko> bbezak: moreover, it's only runonce script which is only suggested to use as per documentation ...normally it's not used I would say ..with end user
14:37:21 <bbezak> SvenKieske: I think it was just to prepare images building for now. And caracal is a good place for that as it is a SLURP release
14:37:28 <kevko> bbezak: now i hacked ci :D and download script from my github repo to pass upgrade jobs :D
14:37:51 <bbezak> 24.04 feature will be probably backported to caracal anyway at some point
14:38:03 <SvenKieske> ok
14:38:14 <mmalchuk> btw about CI we have an issue with docs site preview. anyone know this?
14:39:06 <SvenKieske> meta: can we maybe clean up the whiteboard? it still has listed e.g. train PTG; afaik a link to some archived document would be enough there, no? the Whiteboard is 880 lines long now and hard to navigate..
14:39:46 <SvenKieske> sorry, maybe off topic for current cycle planning
14:40:14 <frickler> mmalchuk: where do you see that issue?
14:40:28 <mmalchuk> preview didn't work
14:40:39 <mmalchuk> on any last change
14:40:41 <bbezak> SvenKieske: I'll add it to notes
14:40:49 <mmalchuk> I'll show you... moment
14:41:21 <SvenKieske> bbezak: ty, I can also volunteer to do cleanup or create some archive somewhere (different etherpad) as long as I know what can be cleaned up and what should stay.
14:41:28 <frickler> wfm https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_33b/911417/12/check/openstack-tox-docs/33b87a9/docs/
14:42:04 <SvenKieske> mmalchuk: maybe some storage issue? did you get an (HTTP) error? anything?
14:42:20 <mmalchuk> oops... fixed. looks like infra knows and fixed
14:43:07 <mmalchuk> it was an issue with listing 403 forbidden with banner
14:43:26 <mmalchuk> anyway fixed... lets move ahead
14:43:33 <bbezak> #topic Additional agenda (from whiteboard)
14:43:44 <bbezak> I can see mostly a review requests there
14:43:58 <bbezak> anything to add apart of that?
14:44:09 <SvenKieske> yes :) I already asked this week but I guess if you don't highlight usernames nobody has time :P :D
14:44:21 <mhiner> yes, plus Ivan still has that issue with Docker version on Debian
14:44:22 <bbezak> mhiner, r-krkcek
14:44:35 <bbezak> sorry :) r-krcek
14:45:17 <mhiner> other than that, it's just reviews from me
14:45:26 <SvenKieske> so I'll explicitly mention it here: mnasiadka, frickler: https://review.opendev.org/c/openstack/kolla-ansible/+/855498 please come to a solution where to test ovn-exporter, currently I made it work in the ovn scenario and I honestly would prefer if it stays there, because these tests are pretty brittle
14:46:08 <SvenKieske> meaning it didn't work at all in prometheus-opensearch scenario for many reasons.
14:46:20 <r-krcek> For me its the reviews and one question, but wanted to ask that in the open debate.
14:47:00 <bbezak> #topic Open discussion
14:47:17 <SvenKieske> r-krcek: seems you need to rebase the "add sysctl role change" https://review.opendev.org/c/openstack/kolla-ansible/+/912351
14:47:19 <r-krcek> I noticed that we have jinja filter for finding out that service is enabled and mapped to host. However, it looks like it is used only with certain services. Is there a specific reason for not using it everywhere or was is just because of lack of time to do it? The filter: https://review.opendev.org/c/openstack/kolla-ansible/+/665922
14:47:58 <mmalchuk> lets discuss https://bugs.launchpad.net/kolla/+bug/2042954
14:48:14 <r-krcek> SvenKieske: oh, sorry. Ill do that right after the meeting.
14:48:19 <mmalchuk> imho https://review.opendev.org/c/openstack/kolla/+/913584 not needed and https://review.opendev.org/c/openstack/kolla/+/904781 need revert
14:49:26 <mmalchuk> the issue appears in kernel 5.18+ but we have 5.4 and 5.15 maximum
14:50:04 <mmalchuk> the issue not reproduced on Focal and Yoga
14:50:26 <SvenKieske> mmalchuk: case in point: I know people running HWE kernels on jammy, which bumps you up til 6.5 kernel :)
14:50:40 <mmalchuk> HWE is a 5.15
14:50:42 <opendevreview> Roman Krček proposed openstack/kolla-ansible master: Update cell0 database connection  https://review.opendev.org/c/openstack/kolla-ansible/+/910924
14:51:08 <frickler> 22.04-hwe is 6.5
14:52:13 <mmalchuk> should we support HWE on Jammy in Yoga? Jammy only for transition to Zed
14:52:32 <frickler> yoga is unmaintained, we should not care at all
14:52:37 <mmalchuk> so
14:52:41 <SvenKieske> frickler: maybe we don't have this issue because we have newer openstack?
14:52:51 <SvenKieske> didn't look into it what the actual issue is
14:53:03 <mmalchuk> I've dig deeper
14:53:20 <mmalchuk> the issue only on upstream kernel with old libs
14:53:20 <SvenKieske> funnily enough the bugreport doesn't mention what the bug is, just that there is one.
14:54:00 <SvenKieske> ah this is the original bug: https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1995735
14:54:22 <mmalchuk> yep
14:55:00 <mmalchuk> I've tested all the libs/packages for issue and provide paste links
14:55:35 <mmalchuk> even if we planed to merge https://review.opendev.org/c/openstack/kolla/+/913584 we need fix issue with pyroute2.protocols
14:55:55 <mmalchuk> we install it in the https://review.opendev.org/c/openstack/kolla/+/904781 via packages
14:56:08 <mmalchuk> and it can't be uninstalled via pip
14:58:41 <bbezak> r-krcek: let me get back to you on your services/filter question, I need to check this feature
14:58:49 <SvenKieske> lol?
14:59:07 <SvenKieske> according to https://openstack-ci-reports.ubuntu.com/reports/cloud-archive/yoga_versions.html the package is not called "python3-pyroute2" but "pyroute2"
14:59:25 <SvenKieske> but it's installed via "python3-pyroute2" https://review.opendev.org/c/openstack/kolla/+/904781/3/docker/openstack-base/Dockerfile.j2
14:59:40 <SvenKieske> the other versions in the CI also report this package name afaik
14:59:51 <mmalchuk> there is no issues with packages
15:00:07 <kevko> SvenKieske: pyroute2 is source package name
15:00:19 <mmalchuk> in current images we have pyroute2 on both venv and system
15:00:22 <kevko> SvenKieske: python3-pyroute2 is binary package
15:00:42 <SvenKieske> kevko: ah nice, madness..
15:00:58 <mmalchuk> I've publeshed an edit for https://review.opendev.org/c/openstack/kolla/+/913584
15:01:20 <SvenKieske> mmalchuk: understood, seems you are right (beside the HWE kernel version, but those are a moving target anyway)
15:01:32 <kevko> SvenKieske: standard
15:02:02 <SvenKieske> I guess we are in overtime now.
15:02:03 <bbezak> ok guys, I need to jump to some call. However please continue the discussion of course :)
15:02:12 <bbezak> thank you all
15:02:14 <bbezak> #endmeeting