15:01:02 <ykarel> #startmeeting neutron_ci
15:01:02 <opendevmeet> Meeting started Tue Feb 13 15:01:02 2024 UTC and is due to finish in 60 minutes.  The chair is ykarel. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:02 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:02 <opendevmeet> The meeting name has been set to 'neutron_ci'
15:01:10 <ralonsoh> hello
15:01:10 <ykarel> ping bcafarel, lajoskatona, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva, elvira
15:01:14 <bcafarel> o/
15:01:17 <slaweq> o/
15:01:26 <haleyb> o/
15:01:48 <mtomaska> o/
15:02:15 <ykarel> hi everyone, let's start with topics
15:02:16 <ykarel> #topic Actions from previous meetings
15:02:22 <ykarel> lajoskatona to check failure with test_configurations_are_synced_towards_placement
15:03:05 <mlavalle> \o
15:03:56 <lajoskatona> o/
15:03:59 <lajoskatona> I checked
15:04:34 <lajoskatona> not much result, as I see it happened only once (from opensearch)
15:05:36 <lajoskatona> What I saw in the logs it seems that the placement service plugin changed the state of the agent to be resources_synced=False
15:05:53 <lajoskatona> but the agent_db change is not happened or just later
15:07:17 <lajoskatona> but I can't tell why the update was late (here: https://opendev.org/openstack/neutron/src/branch/master/neutron/services/placement_report/plugin.py#L118 )
15:07:18 <ykarel> ohkk let's keep an eye on it
15:07:23 <lajoskatona> +1
15:07:40 <ykarel> if i see it again will report a lp for futher investigation
15:07:47 <ykarel> thx lajoskatona for looking into it
15:07:58 <lajoskatona> +1 for lp bug if it happens again
15:08:04 <ykarel> ralonsoh to check network interface not found issue in namespace
15:08:35 <ykarel> this is happening quite often now, and most functional job failures are impacted with this only
15:08:36 <ralonsoh> I have a patch ready to retry the pyroute command after creating the interface
15:08:45 <ralonsoh> but I didn't push it yet
15:09:03 <ykarel> thx ralonsoh , hope that will clear these frequent issues
15:09:18 <ykarel> slaweq to check and handle sqlalchemy master issue https://bugs.launchpad.net/neutron/+bug/2052509
15:09:27 <slaweq> this is fixed now
15:09:30 <ykarel> #link https://review.opendev.org/c/openstack/devstack/+/908572
15:09:33 <ykarel> thx slaweq
15:09:40 <slaweq> with https://review.opendev.org/c/openstack/devstack/+/908572
15:09:51 <ykarel> #topic Stable branches
15:10:07 <ykarel> bcafarel, any update ^
15:10:10 <slaweq> and today those periodic jobs were green :)
15:10:42 <ykarel> grenade job in stable/zed and 2023.1 impacted with yoga transitioning to unmaintained
15:10:43 <bcafarel> not a lot beyond https://bugs.launchpad.net/neutron/+bug/2052915 already mentioned in neutron meeting (grenade job issues)
15:11:12 <lajoskatona> a few related aptches: https://review.opendev.org/q/topic:%22bug/2052915%22
15:11:31 <ykarel> okk so just that, rest all good. thx for the update
15:11:47 <ykarel> #topic Stadium projects
15:11:55 <ykarel> all green in periodic-weekly
15:12:05 <lajoskatona> yes
15:12:22 <lajoskatona> And there's a few cosmetic changes for them, please check if you have free minutes:
15:12:42 <lajoskatona> https://review.opendev.org/q/topic:%22tox-envdir%22+owner:katonalala@gmail.com
15:12:58 <lajoskatona> https://review.opendev.org/q/topic:%22bump-hacking%22+owner:katonalala@gmail.com
15:13:10 <lajoskatona> I have some reviews thanks for them
15:13:24 <ykarel> +1
15:13:44 <lajoskatona> they are for some minimal tox change and hacking update to make them closer to what Openstack other projects uses
15:13:48 <lajoskatona> (-s)
15:14:53 <lajoskatona> thats it for stadiums
15:15:02 <ykarel> thx
15:15:44 <ykarel> #topic Rechecks
15:16:46 <ykarel> rechecks increased this week, likely due to those grenade issue, also a day there were some pypi related issue
15:17:01 <ykarel> also frequent functional failures
15:17:22 <ykarel> there were 6 bare rechecks too out of 35
15:17:37 <ykarel> let's try to avoid bare rechecks
15:17:54 <ykarel> let's move to f
15:17:58 <ykarel> #topic fullstack/functional
15:18:07 <ykarel> test_securitygroup(ovs-hybrid)
15:18:18 <ykarel> neutron_lib.exceptions.ProcessExecutionError: Exit code: -6; Cmd: ['iptables-save', '-t', 'raw']; Stdin: ; Stdout: ; Stderr: free(): double free detected in tcache 2
15:18:29 <ykarel> seen once https://fe0d8609d5db0382bdda-1cfb81f46e8d4d51af7baa8d8d19cc9e.ssl.cf2.rackcdn.com/892542/9/gate/neutron-fullstack-with-uwsgi/92a2ac3/testr_results.html
15:18:52 <ykarel> related ubuntu jammy bug for iptables https://bugs.launchpad.net/ubuntu/+source/iptables/+bug/2049318
15:19:12 <ralonsoh> uhhh serious issue
15:19:40 <ralonsoh> but that was solved 10 years ago
15:20:06 <ralonsoh> (sorry, I'm not able to read anymore...)
15:20:18 <ralonsoh> that was solved 1 week ago
15:20:30 <ykarel> as per report, impacted version 1.8.7-1ubuntu5.1, upcoming fix in 1.8.7-1ubuntu5.2
15:20:53 <haleyb> i just saw an update to iptables this morning on ubuntu
15:21:17 <haleyb> but don't think our images would have it
15:21:43 <ykarel> okk i see latest runs have 1.8.7-1ubuntu5.2 included
15:21:45 <ykarel> so we are good
15:22:03 <ykarel> seen in https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a76/908764/2/check/neutron-tempest-plugin-ovn/a76fc31/controller/logs/dpkg-l.txt
15:22:27 <ykarel> k moving to next
15:22:30 <ykarel> Network interface not found in namespace failures in Namespace
15:22:37 <ykarel> https://b2a03fe4b3eeed803979-76ffc4576b5ebbbaec0e39025919d8c0.ssl.cf1.rackcdn.com/908434/2/check/neutron-functional-with-uwsgi/2fc1bb9/testr_results.html
15:22:38 <ykarel> https://4225788823b4d673b32a-970b30994ec26ff19befc99de6943775.ssl.cf1.rackcdn.com/908588/2/check/neutron-functional-with-uwsgi/253bfaf/testr_results.html
15:22:38 <ykarel> https://bf551380c009c1591acf-43d675d9b4d25dd83e8dc401d48f3c75.ssl.cf1.rackcdn.com/907504/4/check/neutron-functional-with-uwsgi/2f4cdb4/testr_results.html
15:22:49 <ykarel> we already talked about it and ralonsoh going to push fix for it
15:23:01 <ralonsoh> tomorrow morning, for sure
15:23:06 <ykarel> thx
15:23:10 <ykarel> neutron_lib.exceptions.agent.AgentNotFoundByTypeHost: Agent with agent_type=L3 agent and host=az0-host-0 could not be found
15:23:18 <ykarel> https://62ba78566d698900aad5-4082203e55009ffc8095c87eb70265ba.ssl.cf1.rackcdn.com/908587/2/check/neutron-functional-with-uwsgi/dc4d027/testr_results.html
15:23:26 <ykarel> seen once in backport patch
15:24:17 <ralonsoh> I don't think this is related at all with this patch
15:24:33 <ralonsoh> actually  there was a legit issue with testing already solved
15:24:46 <ykarel> yes that doesn't look related
15:26:14 <ykarel> will check logs if i find anything for this
15:26:40 <ykarel> #action ykarel to check failure in test_auto_schedule_router
15:26:52 <ykarel> #topic Tempest/Scenario
15:26:59 <ykarel> https://ac3deee033df2f80309a-9b1010a8ed0ed23e4a7e66dfa043a295.ssl.cf5.rackcdn.com/907418/2/check/tempest-slow-py3/6dff044/testr_results.html
15:27:23 <ykarel> some metadata issue slaweq reported https://bugs.launchpad.net/neutron/+bug/2052787
15:27:56 <ralonsoh> I think he has something, related to HA proxy
15:28:01 <slaweq> yeah, I want to add some more debug logs to haproxy service but I didn't had chance yet to look into it
15:28:06 <ralonsoh> (sorry)
15:28:36 <ykarel> ohkk thanks
15:28:53 <ykarel> #topic grenade
15:29:04 <ykarel> https://zuul.openstack.org/builds?job_name=neutron-ovs-grenade-multinode&job_name=neutron-ovn-grenade-multinode&branch=stable%2Fzed
15:29:10 <ykarel> https://zuul.openstack.org/builds?job_name=neutron-ovn-grenade-multinode&job_name=neutron-ovn-grenade-multinode-skip-level&branch=stable%2F2023.1&skip=0
15:29:24 <ykarel> this already discussed, and being tracked in https://bugs.launchpad.net/neutron/+bug/2052915
15:29:32 <ykarel> #topic Periodic
15:29:43 <ykarel> centos 9 stream jobs broken with libvirt-9.10
15:29:52 <ykarel> #link https://zuul.openstack.org/builds?job_name=neutron-ovn-tempest-ovs-master-centos-9-stream&job_name=neutron-ovn-tempest-ovs-release-fips&job_name=neutron-ovs-tempest-fips&branch=master&skip=0
15:30:01 <ykarel> Related Issue https://issues.redhat.com/browse/RHEL-20609
15:30:18 <ykarel> libvirt-10.0 have fix for it, already included in upcoming compose https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/AppStream/x86_64/os/Packages/
15:30:44 <ykarel> job should be back to green with next compose once available in infra mirrors
15:31:19 <ykarel> i recall those composes are normally created on Wednesday
15:31:32 <ykarel> so hopefully this week it should be solved
15:31:46 <ykarel> that's it on failures
15:31:52 <ykarel> #topic Grafana
15:31:52 <ykarel> https://grafana.opendev.org/d/f913631585/neutron-failure-rate
15:32:02 <ykarel> let's have a quick look at grafana too
15:32:54 <slaweq> IMHO all looks good there
15:33:05 <ralonsoh> I think so yes
15:33:52 <ykarel> yes overall looks good. thx all
15:34:02 <ykarel> #topic On Demand
15:34:11 <ykarel> anything you would like to raise here?
15:34:27 <ralonsoh> no thanks
15:35:05 <mlavalle> I have one quick point
15:36:12 <mlavalle> yesterday I saw this functional failure unrelated to my patch: https://e4cf1ab71b6ca5f5dc46-fe390436bababd65005a5c1c9412b532.ssl.cf5.rackcdn.com/901513/12/check/neutron-functional-with-uwsgi/de8b2ef/testr_results.html
15:36:43 <ralonsoh> I talked aboyt this error last week
15:36:43 <mlavalle> oslo_db.exception.DBReferenceError: (sqlite3.IntegrityError) FOREIGN KEY constraint failed
15:36:46 <mlavalle> [SQL: INSERT INTO ml2_port_bindings (port_id, host, vnic_type, profile, vif_type, vif_details, status) VALUES (?, ?, ?, ?, ?, ?, ?)]
15:36:48 <mlavalle> [parameters: ('b3c63e59-7c0c-4c6e-80d2-b52b976b262e', '', 'normal', '', 'unbound', '', 'ACTIVE')]
15:36:50 <mlavalle> (Background on this error at: https://sqlalche.me/e/14/gkpj)Yy
15:36:54 <mlavalle> ah ok
15:36:57 <ralonsoh> this is a sqlite error in the transaction
15:37:11 <ralonsoh> because the reference object is also created in this same tnx
15:37:18 <ralonsoh> I really don't know why this is happening
15:37:25 <ralonsoh> but Neutron code is correct
15:37:35 <mlavalle> ahh, ok, I'll keep an eye on it
15:37:48 <mlavalle> and see if I can take a crack at it
15:37:55 <ralonsoh> perfect
15:37:58 <mlavalle> Thanks!
15:39:15 <ykarel> thx all
15:39:22 <ykarel> let's have everyone 20 minutes back
15:39:26 <ykarel> #endmeeting