13:00:00 <mnasiadka> #startmeeting kolla 13:00:00 <opendevmeet> Meeting started Wed Apr 24 13:00:00 2024 UTC and is due to finish in 60 minutes. The chair is mnasiadka. Information about MeetBot at http://wiki.debian.org/MeetBot. 13:00:00 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 13:00:00 <opendevmeet> The meeting name has been set to 'kolla' 13:00:03 <mnasiadka> #topic rollcall 13:00:05 <mnasiadka> o/ 13:00:27 <opendevreview> Maksim Malchuk proposed openstack/kolla stable/2023.1: Nova: fix swtpm and swtpm-tools missing from deb installs https://review.opendev.org/c/openstack/kolla/+/916768 13:00:35 <mmalchuk> \o 13:00:43 <SvenKieske> o/ 13:00:50 <r-krcek> o/ 13:00:53 <bbezak> o/ 13:00:56 <mhiner> o/ 13:00:57 <frickler> \o 13:03:15 <mnasiadka> #topic agenda 13:03:15 <mnasiadka> * CI status 13:03:15 <mnasiadka> * Release tasks 13:03:15 <mnasiadka> * Regular stable releases (first meeting in a month) 13:03:15 <mnasiadka> * Current cycle planning 13:03:17 <mnasiadka> * Additional agenda (from whiteboard) 13:03:17 <mnasiadka> * Open discussion 13:03:19 <mnasiadka> #topic CI status 13:03:24 <mnasiadka> Octavia still bleeding red 13:03:32 <mnasiadka> Seems cascade delete is failing 13:03:45 <mnasiadka> on unplugging the VM port, or something similar 13:04:54 <mnasiadka> working on getting to the bottom of it in https://review.opendev.org/c/openstack/kolla-ansible/+/916037 13:04:59 <mnasiadka> #topic Release tasks 13:05:23 <mnasiadka> Soo... I'd like to close down the number of patches we want to have in Kolla Caracal 13:05:41 <mnasiadka> https://review.opendev.org/q/project:openstack/kolla+status:open+NOT+label:Workflow%3C%3D-1+NOT+label:Code-Review%3C%3D-2+label:Review-Priority%3D1 13:05:52 <mnasiadka> If anybody thinks we should add something more - let me know 13:07:30 <SvenKieske> ok, is there some kind of deadline for this? 13:07:32 <mnasiadka> the same comes with kolla-ansible - but I'd like to get Kolla released first 13:07:46 <mnasiadka> deadline? Let's say two weeks 13:07:58 <mnasiadka> (because I'm off next week - 1st of May is public holiday in a lot of places) 13:08:05 <mnasiadka> So I assume we should cancel the meeting next week 13:08:12 <SvenKieske> sounds good 13:08:15 <frickler> same here, ack 13:08:37 <mnasiadka> ok then, let's move on 13:08:49 <mnasiadka> #topic Current cycle planning 13:08:56 <frickler> what about stsble? 13:08:59 <frickler> do you want a final release for stable/zed before it moves to unmaintained? 13:09:33 <mnasiadka> Well, from one perspective it would make sense - but we as SHPC don't really need that 13:09:55 <mnasiadka> So if somebody volunteers for last Zed release - I'm happy to +1 13:10:00 <frickler> I don't need it either 13:10:19 <frickler> so fine without one, just note the deadline for that is next week, too 13:10:20 <SvenKieske> kevko is not here 13:10:31 <SvenKieske> guess he could care ;) 13:11:14 <SvenKieske> well maybe not, afaik he has his downstream mirrors etc. 13:11:42 <mnasiadka> Well then - the absent don't get a vote I'd say - so let's just go with moving zed to unmaintained 13:12:20 <mnasiadka> I'll find the release patches after the meeting and +1 them 13:12:40 <mnasiadka> Current cycle for us is still Caracal - so I think we have enough of a plan. 13:12:41 <mmalchuk> I care 13:12:45 <mmalchuk> but ok 13:12:56 <mnasiadka> mmalchuk: do you want to raise the release patch for Zed? 13:13:26 <mmalchuk> mnasiadka I care to postpone move to unmaintained 13:13:53 <mmalchuk> can we? 13:14:20 <frickler> no, the deadline for that is set 13:15:07 <mnasiadka> we still maintain the unmaintained, however that sounds 13:15:57 <mmalchuk> ok lets move 13:16:12 <mnasiadka> #topic Additional agenda (from whiteboard) 13:16:56 <mnasiadka> (SvenKieske): cores: please review the following: 13:16:56 <mnasiadka> ovn-exporter https://review.opendev.org/c/openstack/kolla-ansible/+/855498 CI is now greenish found bugs, which I need to fix first 13:16:56 <mnasiadka> neutron service limit https://review.opendev.org/c/openstack/kolla-ansible/+/877776 one +2 left to go 13:16:56 <mnasiadka> add new sysctl role: https://review.opendev.org/c/openstack/kolla-ansible/+/912351 one +2 left to go 13:16:56 <mnasiadka> cell0 VIP change fix https://review.opendev.org/c/openstack/kolla-ansible/+/910924 one +2 left to go 13:16:57 <mnasiadka> note from me: feel free to remove stuff from this list if it's merged , thanks! I also think this is just a temporary list, should probably just use the review board instead. 13:17:52 <mnasiadka> frickler/bbezak: willing to have a look in the bottom three? 13:18:31 <SvenKieske> I already crossed out the ovn-exporter, found a myriad of bugs, likely will even need to fix those upstream first I guess.. :-/ thanks for the pointers all. 13:18:31 <bbezak> will do, neutron needs outside SHPC I recon 13:19:08 <SvenKieske> afaik frickler had some good comments on the neutron change which should be addressed 13:19:27 <mnasiadka> ah 13:19:44 <frickler> yes, was just checking whether there had been a response yet 13:20:01 <mnasiadka> now it's more visible with -1 :) 13:20:19 <mnasiadka> ok, so the bottom two pretty please 13:20:21 <mnasiadka> and let's move forward 13:20:25 <frickler> and I'm not too sure about the two others, maybe someone else can care for those 13:20:39 <jovial> Sorry, missed the comments, will try and update the service limit patch next week 13:20:39 <mnasiadka> I think bbezak can have a look in those 13:20:44 <mnasiadka> (r-krcek) 13:20:44 <mnasiadka> please review https://review.opendev.org/c/openstack/kolla-ansible/+/912351 13:20:44 <mnasiadka> https://review.opendev.org/c/openstack/kolla-ansible/+/912378 13:20:44 <mnasiadka> https://review.opendev.org/c/openstack/kolla-ansible/+/910924 13:20:56 <mnasiadka> first one is sysctl role once again 13:21:57 <mnasiadka> second one is trove, looks ok, but I'm no trove expert 13:22:10 <mnasiadka> and the third one is cell0 once again 13:22:40 <frickler> trove is dead, can we deprecate and drop it? 13:22:49 <frickler> no PTL 13:22:54 <mnasiadka> ah 13:22:56 <mnasiadka> then let's drop it 13:22:59 <mnasiadka> any volunteer? 13:23:22 <frickler> I'll add that to my list 13:23:53 <mnasiadka> thanks 13:24:09 <mnasiadka> #agreed to deprecate and drop Trove since the project has no PTL 13:24:22 <mnasiadka> (mhiner): migration patch https://review.opendev.org/c/openstack/kolla-ansible/+/836941 13:24:22 <mnasiadka> nova_conductor tries to contact openvswitch_db container 30 secs before it gets deployed 13:24:22 <mnasiadka> this creates error message in nova logs and nova_conductor is deemed unhealthy and fails my CI tests (kolla-ansible-rocky9-container-engine-migration) 13:24:22 <mnasiadka> any tips on how to remedy this, please? 13:24:33 <mnasiadka> mhiner: my question is - how does that work in Docker land? 13:24:59 <mhiner> you mean Podman to Docker migration? 13:25:59 <mnasiadka> No, I mean when you use Docker - we don't have that problem 13:26:09 <frickler> does the ovs creation take too long or does the migration need a better ordering of actions? 13:26:11 <mnasiadka> And nova_conductor_healthcheck_test: ["CMD-SHELL", "healthcheck_port nova-conductor {{ om_rpc_port }}"] basically means healthchecks only check RMQ connectivity 13:26:37 <mnasiadka> unless it's an ERROR in the logs because of which CI is failing 13:26:57 <mhiner> it is ERROR in the logs 13:27:15 <SvenKieske> it was on the PTG topics for nova to implement better healthchecks upstream :D 13:27:28 <mnasiadka> it's rather unrelated to their PTG topic 13:28:09 <mhiner> and it happens because nova_conductor tries to contact openvswitch_db 30secs before it gets created 13:28:21 <frickler> I also wonder why nova needs to talk to ovs at all, but that's another topic 13:28:54 <frickler> I'll take a look at the logs later 13:29:04 <mnasiadka> well, should we have openvswitch before nova in site.yml? 13:29:30 <mhiner> we can try that 13:29:50 <mhiner> but the strange thing is this does not happen in regular deployment 13:30:01 <mhiner> only in migration "redeployment" 13:30:37 <mnasiadka> because we have VMs running on the redeployment? 13:30:51 <frickler> maybe need to stop nova before migrating ovs? 13:30:56 <mnasiadka> At what stage conductor needs to run OVS commands? 13:31:14 <mnasiadka> Well, if we're aiming for a rolling migration - that might be complicated 13:31:21 <mnasiadka> maybe we need to stop Nova before the migration 13:32:19 <mhiner> by stopping Nova, do you mean all VMs? because otherwise all containers are stopped and removed before redeploying 13:33:24 <SvenKieske> could this be related to the neutron ovs integration bridge which nova can use? But I'm just guessing. 13:33:32 <frickler> I was wondering whether the error might be from the old container, but it is the new one which is unhealthy 13:33:53 <frickler> also just noticing that the patch needs a rebase, can you look into that, please? 13:33:54 <mnasiadka> I think it's still a bit weird, that if Nova requires OVS to run - we deploy Nova first, and then OVS 13:34:28 <mhiner> I compared times of redeployment and the ERROR msg and it should be the new container 13:34:49 <mhiner> frickler: will do the rebase 13:35:00 <jovial> I agree that it seems like we should configure ovs before nova 13:35:13 <SvenKieske> depends on the nova scenario I think. e.g. if you use sriov some scenarios do that with ovs afaik. but I don't think we do use that in CI, do we? 13:35:30 <SvenKieske> but yeah, in general it seems better to setup ovs before nova 13:36:21 <mnasiadka> let's try that, maybe we should basically analyse site.yml a bit and see if there's anything we could improve 13:36:29 <mnasiadka> but that's for D rather 13:36:35 <mnasiadka> ok then, let's move on 13:36:42 <mnasiadka> #topic Open discussion 13:36:49 <mnasiadka> Anybody anything? 13:37:19 <kevko> hi, sorry, another meeting :'( 13:37:51 <mmalchuk> reviews 13:38:05 <mmalchuk> merge of backports 13:38:06 <mmalchuk> https://review.opendev.org/q/Ib6c725880caaa7f39bb269bd8398f3894eb033c5 13:38:12 <mmalchuk> https://review.opendev.org/c/openstack/kolla-ansible/+/915156 13:38:22 <mmalchuk> https://review.opendev.org/c/openstack/kolla-ansible/+/907495 13:38:26 <mmalchuk> kayobe 13:38:27 <mmalchuk> https://review.opendev.org/c/openstack/kayobe/+/916138 13:38:35 <mmalchuk> a-c-k 13:38:35 <mmalchuk> https://review.opendev.org/c/openstack/ansible-collection-kolla/+/916143 13:39:06 <mmalchuk> thats all for now 13:39:28 <SvenKieske> ah right, i also did some backports of the mariadb cluster recovery fix: https://review.opendev.org/#/q/Iea2661c9d5d262cf99edd5f5b567f252607a0003 13:39:51 <jovial> mmalchuk, I've marked 916138 as RP +1 13:39:51 <SvenKieske> those should be trivial I guess :) 13:39:51 <mnasiadka> So basically https://review.opendev.org/q/(project:openstack/kolla-ansible+OR+project:openstack/ansible-collection-kolla)+status:open+NOT+label:Workflow%3C%3D-1+NOT+label:Code-Review%3C%3D-2+branch:%5Estable/.*+status:open+NOT+label:Review-Priority%3D-1 13:40:33 <SvenKieske> LGTM I guess 13:41:07 <SvenKieske> I need to compile for myself a fresh list of patches I have in flight, I tend to lose track of those (at least in my head :D) 13:42:03 <mnasiadka> ok then, let's have a look at those backports mgoddard, bbezak, kevko and frickler 13:42:14 <mnasiadka> And I guess that's it 13:43:03 <mnasiadka> Thanks for coming - and the next meeting is in two weeks time! 13:43:27 <mnasiadka> #endmeeting