15:00:54 <ykarel> #startmeeting neutron_ci
15:00:54 <opendevmeet> Meeting started Tue Jan 16 15:00:54 2024 UTC and is due to finish in 60 minutes.  The chair is ykarel. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:54 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:54 <opendevmeet> The meeting name has been set to 'neutron_ci'
15:00:57 <mlavalle> \o
15:01:00 <mtomaska> hello wncslln. Looking at the neutron bug list there are 22 low hanging fruit bugs https://bugs.launchpad.net/neutron/+bugs?field.tag=low-hanging-fruit
15:01:04 <ykarel> bcafarel, lajoskatona, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva, elvira
15:01:09 <ralonsoh> hi
15:01:15 <mtomaska> o/
15:01:34 <lajoskatona> o/
15:01:38 <bcafarel> o/
15:02:46 <ykarel> let's start with the topics
15:02:47 <ykarel> #topic Actions from previous meetings
15:02:54 <ykarel> ralonsoh to push patch for pyroute command handling
15:03:26 <ralonsoh> sorry, I have the patch but didn't push it
15:03:30 <ralonsoh> not working locally
15:04:07 <ykarel> ok np thx for looking into it
15:04:24 <ykarel> ralonsoh to check if functional test failures are related to the backport
15:04:43 <ralonsoh> yes, the errors are legit
15:04:49 <ralonsoh> I'm going to stop these backports
15:04:57 <ralonsoh> but I still don't now the cause
15:05:15 <ralonsoh> once investigated, if I can solve it, I'll re propose these patches
15:05:35 <ykarel> thx much ralonsoh
15:05:51 <ykarel> slaweq to check with cirros guy to see if this is some known issue related to kernel panic trace
15:06:11 <ykarel> slaweq is not around but he shared is findings already
15:06:46 <ykarel> but checking both chatgpt and a cirros guy :)
15:06:47 <ykarel> The kernel panic message you provided indicates a general protection fault during the boot process of the Cirros operating system. The error occurs in the __kmalloc function, which is a kernel memory allocation function. The stack trace also shows that the issue is related to RSA key handling and signature verification.
15:06:48 <ykarel> Here are a few suggestions to troubleshoot and resolve the issue:
15:06:48 <ykarel> Memory Issues:
15:06:48 <ykarel> Make sure that the virtual machine has sufficient memory allocated. If possible, try increasing the allocated memory and see if the issue persists.
15:07:06 <ykarel> So it seems to me that it was issue with not enough memory given for the guest OS. According to info from the "Cirros guy" who I asked about it also, we should give 256 MB of memory for the flavor used for Cirros now. See https://github.com/cirros-dev/cirros/issues/53 for more details.
15:07:11 <lajoskatona> you mean use larger flavor?
15:07:43 <ykarel> that what the suggesting is
15:08:07 <ykarel> but looking at that issue link i see for bios/x86_64 128 mb is enough and that's what we using in these jobs
15:08:13 <lajoskatona> for that the parallely running tests number should be decreased perhaps also
15:08:22 <lajoskatona> ah, ok
15:09:30 <ykarel> considering this point i am not sure how much worth to update flavor in these jobs, as we seen that issue rarely
15:10:01 <ykarel> there is mention that for uefi it would need 256 mb, but looking logs doesn't seem we use that
15:11:56 <ykarel> so i think we can keep a watch and if we see it more frequently we can attempt that change, wdyt?
15:12:45 <ralonsoh> +1
15:12:48 <lajoskatona> +1
15:12:50 <ralonsoh> do we have a LP bug?
15:13:28 <ykarel> no we don't
15:13:41 <ykarel> seen just once so haven't reported it yet
15:14:03 <ralonsoh> I'll ping slaweq to do this because he has all the information
15:14:10 <ykarel> k thx
15:14:29 <ykarel> #topic Stable branches
15:14:39 <ykarel> bcafarel, any update here?
15:15:22 <bcafarel> all looking good from what I checked! and now as haleyb commented, we have one less branch to worry about with ussuri EOL quite close
15:15:45 <ykarel> that's good
15:16:20 <ykarel> just one thing i noticed in xena some linux bridge job timing out, seen in check and periodic queue, not consistent though
15:16:50 <ykarel> possibly just slow nodes
15:17:42 <ykarel> #topic Stadium projects
15:17:51 <ykarel> all green in periodic-weekly
15:17:58 <ykarel> lajoskatona anything else to share here?
15:18:06 <lajoskatona> all is grren as I checked
15:18:20 <lajoskatona> perhaps I mention here the py311 addition to setup.cfg:
15:18:39 <lajoskatona> https://review.opendev.org/q/topic:%22py3.11%22+status:open,25
15:19:02 <opendevreview> Merged openstack/neutron stable/wallaby: "ebtables-nft" MAC rule deletion failing  https://review.opendev.org/c/openstack/neutron/+/905538
15:19:11 <ykarel> thx lajoskatona
15:19:18 <lajoskatona> I added the extra line to the stadiums, so please check the list when you have few extra minutes and let's add py311 to networking projects coverage :-)
15:19:33 <ykarel> ++
15:19:35 <lajoskatona> that's it for stadiums
15:19:53 <ykarel> #topic Rechecks
15:20:25 <ykarel> quite good number of patches merged, with a few rechecks
15:20:43 <ykarel> 1 bare recheck out of 7, let's keep doing it
15:20:54 <ykarel> #topic fullstack/functional
15:21:03 <ykarel> test_delete_route
15:21:09 <ykarel> https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_060/905105/1/gate/neutron-functional-with-uwsgi/0603eb7/testr_results.html
15:21:25 <ykarel> so an older issue which got noticed again https://bugs.launchpad.net/neutron/+bug/1988037
15:21:29 <ralonsoh> I've added some extra logs
15:21:32 <ykarel> Rodolfo pushed patch to collect more info if it fails https://review.opendev.org/c/openstack/neutron/+/905291
15:21:42 <ykarel> thx ralonsoh
15:22:04 <ykarel> next one
15:22:05 <ykarel> ovsdbapp.exceptions.TimeoutException: Commands ... exceeded timeout 10 seconds, cause: Result queue is empty
15:22:11 <ykarel> https://ce5c304e9b3a7699cf9c-f4e1e757f6e1e9727e757a7fa4b37fa7.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-uwsgi-fips/d9449f2/testr_results.html
15:22:16 <ykarel> https://a08072a22aec7dbb4aca-289b34983b13ece36a1a19c591f4d0ce.ssl.cf1.rackcdn.com/905332/1/check/neutron-functional-with-uwsgi/8770081/testr_results.html
15:22:44 <ykarel> you recall seeing similar issue?
15:23:23 <ykarel> wondering if it just due to slow node
15:23:54 <lajoskatona> I never seens similar to tell the truth
15:24:10 <ykarel> as looking into dstat output around the time of failure i noticed some cpu wait for i/o
15:25:28 <ralonsoh> there is indeed a 10 seconds gap in the server
15:25:41 <ralonsoh> https://ce5c304e9b3a7699cf9c-f4e1e757f6e1e9727e757a7fa4b37fa7.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-uwsgi-fips/d9449f2/controller/logs/openvswitch/ovs-vswitchd_log.txt
15:26:03 <ralonsoh> 2024-01-12T03:29:08.878Z|10296|bridge|INFO|bridge br-test-1fa6b16: using datapath ID 0000ded23192b84f
15:26:03 <ralonsoh> 2024-01-12T03:29:08.878Z|10297|connmgr|INFO|br-test-1fa6b16: added service controller "punix:/var/run/openvswitch/br-test-1fa6b16.mgmt"
15:26:03 <ralonsoh> 2024-01-12T03:29:18.814Z|10298|connmgr|INFO|test-bre94359f5<->tcp:127.0.0.1:16002: 9 flow_mods 10 s ago (9 adds)
15:26:20 <ralonsoh> but most probably because the test was waiting
15:28:44 <ykarel> not sure anything we can do for this issue
15:30:26 <ykarel> for now will just monitor and if i see it again will report lp to track the further investigation
15:30:46 <ykarel> #topic Periodic
15:30:56 <ykarel> ovn source(main branch) jobs broken
15:31:12 <ykarel> #link https://bugs.launchpad.net/neutron/+bug/2049488
15:31:18 <ykarel> Fix https://review.opendev.org/c/openstack/neutron/+/905645
15:31:34 <ykarel> already merged, so jobs should be back to green now
15:31:53 <ykarel> that's it on failures
15:31:58 <ykarel> #topic Grafana
15:32:08 <ykarel> let's hava quick look at #topic Grafana
15:32:16 <ykarel> sorry https://grafana.opendev.org/d/f913631585/neutron-failure-rate
15:33:13 <ralonsoh> looks fine, didn't it?
15:33:30 <ralonsoh> doesn't it**
15:33:50 <ykarel> yes looks good, just check queue has some spikes
15:33:55 <mlavalle> I think so
15:34:23 <ykarel> and when i was checking failures there it were mostly patch related or the ones we already discussed
15:34:26 <ykarel> so all good here
15:34:39 <ykarel> #topic On Demand
15:34:48 <ykarel> anything you would like to raise here?
15:35:12 <ralonsoh> no thanks
15:35:33 <lajoskatona> not from me
15:36:31 <ykarel> k in that case let's close and have everyone 24 minutes back
15:36:32 <ykarel> thx all
15:36:40 <ykarel> #endmeeting