13:00:00 <mnasiadka> #startmeeting kolla
13:00:00 <opendevmeet> Meeting started Wed Apr 24 13:00:00 2024 UTC and is due to finish in 60 minutes.  The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot.
13:00:00 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
13:00:00 <opendevmeet> The meeting name has been set to 'kolla'
13:00:03 <mnasiadka> #topic rollcall
13:00:05 <mnasiadka> o/
13:00:27 <opendevreview> Maksim Malchuk proposed openstack/kolla stable/2023.1: Nova: fix swtpm and swtpm-tools missing from deb installs  https://review.opendev.org/c/openstack/kolla/+/916768
13:00:35 <mmalchuk> \o
13:00:43 <SvenKieske> o/
13:00:50 <r-krcek> o/
13:00:53 <bbezak> o/
13:00:56 <mhiner> o/
13:00:57 <frickler> \o
13:03:15 <mnasiadka> #topic agenda
13:03:15 <mnasiadka> * CI status
13:03:15 <mnasiadka> * Release tasks
13:03:15 <mnasiadka> * Regular stable releases (first meeting in a month)
13:03:15 <mnasiadka> * Current cycle planning
13:03:17 <mnasiadka> * Additional agenda (from whiteboard)
13:03:17 <mnasiadka> * Open discussion
13:03:19 <mnasiadka> #topic CI status
13:03:24 <mnasiadka> Octavia still bleeding red
13:03:32 <mnasiadka> Seems cascade delete is failing
13:03:45 <mnasiadka> on unplugging the VM port, or something similar
13:04:54 <mnasiadka> working on getting to the bottom of it in https://review.opendev.org/c/openstack/kolla-ansible/+/916037
13:04:59 <mnasiadka> #topic Release tasks
13:05:23 <mnasiadka> Soo... I'd like to close down the number of patches we want to have in Kolla Caracal
13:05:41 <mnasiadka> https://review.opendev.org/q/project:openstack/kolla+status:open+NOT+label:Workflow%3C%3D-1+NOT+label:Code-Review%3C%3D-2+label:Review-Priority%3D1
13:05:52 <mnasiadka> If anybody thinks we should add something more - let me know
13:07:30 <SvenKieske> ok, is there some kind of deadline for this?
13:07:32 <mnasiadka> the same comes with kolla-ansible - but I'd like to get Kolla released first
13:07:46 <mnasiadka> deadline? Let's say two weeks
13:07:58 <mnasiadka> (because I'm off next week - 1st of May is public holiday in a lot of places)
13:08:05 <mnasiadka> So I assume we should cancel the meeting next week
13:08:12 <SvenKieske> sounds good
13:08:15 <frickler> same here, ack
13:08:37 <mnasiadka> ok then, let's move on
13:08:49 <mnasiadka> #topic Current cycle planning
13:08:56 <frickler> what about stsble?
13:08:59 <frickler> do you want a final release for stable/zed before it moves to unmaintained?
13:09:33 <mnasiadka> Well, from one perspective it would make sense - but we as SHPC don't really need that
13:09:55 <mnasiadka> So if somebody volunteers for last Zed release - I'm happy to +1
13:10:00 <frickler> I don't need it either
13:10:19 <frickler> so fine without one, just note the deadline for that is next week, too
13:10:20 <SvenKieske> kevko is not here
13:10:31 <SvenKieske> guess he could care ;)
13:11:14 <SvenKieske> well maybe not, afaik he has his downstream mirrors etc.
13:11:42 <mnasiadka> Well then - the absent don't get a vote I'd say - so let's just go with moving zed to unmaintained
13:12:20 <mnasiadka> I'll find the release patches after the meeting and +1 them
13:12:40 <mnasiadka> Current cycle for us is still Caracal - so I think we have enough of a plan.
13:12:41 <mmalchuk> I care
13:12:45 <mmalchuk> but ok
13:12:56 <mnasiadka> mmalchuk: do you want to raise the release patch for Zed?
13:13:26 <mmalchuk> mnasiadka I care to postpone move to unmaintained
13:13:53 <mmalchuk> can we?
13:14:20 <frickler> no, the deadline for that is set
13:15:07 <mnasiadka> we still maintain the unmaintained, however that sounds
13:15:57 <mmalchuk> ok lets move
13:16:12 <mnasiadka> #topic Additional agenda (from whiteboard)
13:16:56 <mnasiadka> (SvenKieske): cores: please review the following:
13:16:56 <mnasiadka> ovn-exporter https://review.opendev.org/c/openstack/kolla-ansible/+/855498 CI is now greenish found bugs, which I need to fix first
13:16:56 <mnasiadka> neutron service limit https://review.opendev.org/c/openstack/kolla-ansible/+/877776 one +2 left to go
13:16:56 <mnasiadka> add new sysctl role: https://review.opendev.org/c/openstack/kolla-ansible/+/912351 one +2 left to go
13:16:56 <mnasiadka> cell0 VIP change fix https://review.opendev.org/c/openstack/kolla-ansible/+/910924 one +2 left to go
13:16:57 <mnasiadka> note from me: feel free to remove stuff from this list if it's merged , thanks! I also think this is just a temporary list, should probably just use the review board instead.
13:17:52 <mnasiadka> frickler/bbezak: willing to have a look in the bottom three?
13:18:31 <SvenKieske> I already crossed out the ovn-exporter, found a myriad of bugs, likely will even need to fix those upstream first I guess.. :-/ thanks for the pointers all.
13:18:31 <bbezak> will do, neutron needs outside SHPC I recon
13:19:08 <SvenKieske> afaik frickler had some good comments on the neutron change which should be addressed
13:19:27 <mnasiadka> ah
13:19:44 <frickler> yes, was just checking whether there had been a response yet
13:20:01 <mnasiadka> now it's more visible with -1 :)
13:20:19 <mnasiadka> ok, so the bottom two pretty please
13:20:21 <mnasiadka> and let's move forward
13:20:25 <frickler> and I'm not too sure about the two others, maybe someone else can care for those
13:20:39 <jovial> Sorry, missed the comments, will try and update the service limit patch next week
13:20:39 <mnasiadka> I think bbezak can have a look in those
13:20:44 <mnasiadka> (r-krcek)
13:20:44 <mnasiadka> please review https://review.opendev.org/c/openstack/kolla-ansible/+/912351
13:20:44 <mnasiadka> https://review.opendev.org/c/openstack/kolla-ansible/+/912378
13:20:44 <mnasiadka> https://review.opendev.org/c/openstack/kolla-ansible/+/910924
13:20:56 <mnasiadka> first one is sysctl role once again
13:21:57 <mnasiadka> second one is trove, looks ok, but I'm no trove expert
13:22:10 <mnasiadka> and the third one is cell0 once again
13:22:40 <frickler> trove is dead, can we deprecate and drop it?
13:22:49 <frickler> no PTL
13:22:54 <mnasiadka> ah
13:22:56 <mnasiadka> then let's drop it
13:22:59 <mnasiadka> any volunteer?
13:23:22 <frickler> I'll add that to my list
13:23:53 <mnasiadka> thanks
13:24:09 <mnasiadka> #agreed to deprecate and drop Trove since the project has no PTL
13:24:22 <mnasiadka> (mhiner): migration patch https://review.opendev.org/c/openstack/kolla-ansible/+/836941
13:24:22 <mnasiadka> nova_conductor tries to contact openvswitch_db container 30 secs before it gets deployed
13:24:22 <mnasiadka> this creates error message in nova logs and nova_conductor is deemed unhealthy and fails my CI tests (kolla-ansible-rocky9-container-engine-migration)
13:24:22 <mnasiadka> any tips on how to remedy this, please?
13:24:33 <mnasiadka> mhiner: my question is - how does that work in Docker land?
13:24:59 <mhiner> you mean Podman to Docker migration?
13:25:59 <mnasiadka> No, I mean when you use Docker - we don't have that problem
13:26:09 <frickler> does the ovs creation take too long or does the migration need a better ordering of actions?
13:26:11 <mnasiadka> And nova_conductor_healthcheck_test: ["CMD-SHELL", "healthcheck_port nova-conductor {{ om_rpc_port }}"] basically means healthchecks only check RMQ connectivity
13:26:37 <mnasiadka> unless it's an ERROR in the logs because of which CI is failing
13:26:57 <mhiner> it is ERROR in the logs
13:27:15 <SvenKieske> it was on the PTG topics for nova to implement better healthchecks upstream :D
13:27:28 <mnasiadka> it's rather unrelated to their PTG topic
13:28:09 <mhiner> and it happens because nova_conductor tries to contact openvswitch_db 30secs before it gets created
13:28:21 <frickler> I also wonder why nova needs to talk to ovs at all, but that's another topic
13:28:54 <frickler> I'll take a look at the logs later
13:29:04 <mnasiadka> well, should we have openvswitch before nova in site.yml?
13:29:30 <mhiner> we can try that
13:29:50 <mhiner> but the strange thing is this does not happen in regular deployment
13:30:01 <mhiner> only in migration "redeployment"
13:30:37 <mnasiadka> because we have VMs running on the redeployment?
13:30:51 <frickler> maybe need to stop nova before migrating ovs?
13:30:56 <mnasiadka> At what stage conductor needs to run OVS commands?
13:31:14 <mnasiadka> Well, if we're aiming for a rolling migration - that might be complicated
13:31:21 <mnasiadka> maybe we need to stop Nova before the migration
13:32:19 <mhiner> by stopping Nova, do you mean all VMs? because otherwise all containers are stopped and removed before redeploying
13:33:24 <SvenKieske> could this be related to the neutron ovs integration bridge which nova can use? But I'm just guessing.
13:33:32 <frickler> I was wondering whether the error might be from the old container, but it is the new one which is unhealthy
13:33:53 <frickler> also just noticing that the patch needs a rebase, can you look into that, please?
13:33:54 <mnasiadka> I think it's still a bit weird, that if Nova requires OVS to run - we deploy Nova first, and then OVS
13:34:28 <mhiner> I compared times of redeployment and the ERROR msg and it should be the new container
13:34:49 <mhiner> frickler: will do the rebase
13:35:00 <jovial> I agree that it seems like we should configure ovs before nova
13:35:13 <SvenKieske> depends on the nova scenario I think. e.g. if you use sriov some scenarios do that with ovs afaik. but I don't think we do use that in CI, do we?
13:35:30 <SvenKieske> but yeah, in general it seems better to setup ovs before nova
13:36:21 <mnasiadka> let's try that, maybe we should basically analyse site.yml a bit and see if there's anything we could improve
13:36:29 <mnasiadka> but that's for D rather
13:36:35 <mnasiadka> ok then, let's move on
13:36:42 <mnasiadka> #topic Open discussion
13:36:49 <mnasiadka> Anybody anything?
13:37:19 <kevko> hi, sorry, another meeting :'(
13:37:51 <mmalchuk> reviews
13:38:05 <mmalchuk> merge of backports
13:38:06 <mmalchuk> https://review.opendev.org/q/Ib6c725880caaa7f39bb269bd8398f3894eb033c5
13:38:12 <mmalchuk> https://review.opendev.org/c/openstack/kolla-ansible/+/915156
13:38:22 <mmalchuk> https://review.opendev.org/c/openstack/kolla-ansible/+/907495
13:38:26 <mmalchuk> kayobe
13:38:27 <mmalchuk> https://review.opendev.org/c/openstack/kayobe/+/916138
13:38:35 <mmalchuk> a-c-k
13:38:35 <mmalchuk> https://review.opendev.org/c/openstack/ansible-collection-kolla/+/916143
13:39:06 <mmalchuk> thats all for now
13:39:28 <SvenKieske> ah right, i also did some backports of the mariadb cluster recovery fix: https://review.opendev.org/#/q/Iea2661c9d5d262cf99edd5f5b567f252607a0003
13:39:51 <jovial> mmalchuk, I've marked 916138 as RP +1
13:39:51 <SvenKieske> those should be trivial I guess :)
13:39:51 <mnasiadka> So basically https://review.opendev.org/q/(project:openstack/kolla-ansible+OR+project:openstack/ansible-collection-kolla)+status:open+NOT+label:Workflow%3C%3D-1+NOT+label:Code-Review%3C%3D-2+branch:%5Estable/.*+status:open+NOT+label:Review-Priority%3D-1
13:40:33 <SvenKieske> LGTM I guess
13:41:07 <SvenKieske> I need to compile for myself a fresh list of patches I have in flight, I tend to lose track of those (at least in my head :D)
13:42:03 <mnasiadka> ok then, let's have a look at those backports mgoddard, bbezak, kevko and frickler
13:42:14 <mnasiadka> And I guess that's it
13:43:03 <mnasiadka> Thanks for coming - and the next meeting is in two weeks time!
13:43:27 <mnasiadka> #endmeeting