15:00:55 <ykarel> #startmeeting neutron_ci
15:00:55 <opendevmeet> Meeting started Tue Feb 27 15:00:55 2024 UTC and is due to finish in 60 minutes.  The chair is ykarel. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:55 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:55 <opendevmeet> The meeting name has been set to 'neutron_ci'
15:01:03 <slaweq> o/
15:01:08 <ykarel> ping bcafarel, lajoskatona, mlavalle, mtomaska, ralonsoh, ykarel, jlibosva, elvira
15:01:30 <lajoskatona> o/
15:01:44 <lajoskatona> ykarel: is this a video or IRC only meeting today?
15:01:49 <ykarel> irc
15:01:55 <lajoskatona> cool , thanks
15:02:02 <bcafarel> o/
15:02:13 <mlavalle> o/
15:03:44 <ykarel> k let's start with topics
15:03:54 <ykarel> #topic Actions from previous meetings
15:04:00 <ykarel> ykarel to check fullstack failure
15:04:39 <ykarel> i checked logs but didn't find much there
15:04:50 <ykarel> will keep an eye on it if it reproduces will open a bug for it
15:05:01 <ykarel> ykarel to check grenade status
15:05:18 <ykarel> so it's broken on stable/2023.1
15:05:21 <ykarel> https://zuul.openstack.org/builds?job_name=neutron-ovn-grenade-multinode-skip-level&job_name=neutron-ovs-grenade-multinode-skip-level&branch=stable%2F2023.1&skip=0
15:05:27 <ykarel> https://review.opendev.org/q/topic:%22bug/2052915%22
15:05:54 <ykarel> there are some open patches, and there is discussion whether we should test grenade from unmaintained branch or not
15:06:06 <lajoskatona> good that you bring it here
15:06:19 <ykarel> also non skip level job in zed are impacted
15:06:32 <lajoskatona> to tell the truth I am lost what is the direction and I am not a grenade expert anyway :-(
15:06:47 <bcafarel> yes skip-level is non-voting for 2023.1, but zed gate is blocked atm
15:07:10 <ykarel> https://zuul.openstack.org/builds?job_name=neutron-ovs-grenade-multinode&job_name=neutron-ovn-grenade-multinode&branch=stable%2Fzed&skip=0
15:07:24 <lajoskatona> I remember somebody said that we should stop pgrade test from unmaintained to anything else
15:07:26 <ykarel> last run passed, don't know how as not related to patch :)
15:08:09 <ykarel> may be slaweq did some magic in his patch
15:08:42 <bcafarel> hmm interesting
15:09:02 <slaweq> I didn't do any magic for sure :)
15:09:18 <bcafarel> unconscious magic, even more impressive! :)
15:10:04 <ykarel> so i think for this we should join the discussion on those patches https://review.opendev.org/q/topic:%22bug/2052915%22
15:10:24 <ykarel> and decide next steps, i.e drop or pin jobs to unmaintained/yoga or some commit
15:11:38 <lajoskatona> both fine for me, if QA or TC has some common thing for all related project that would be the best
15:12:05 <bcafarel> +1 we can wait a bit and have common fix/way to go from there
15:12:25 <ykarel> to unblock zed we can make jobs non-voting
15:12:59 <lajoskatona> +1
15:13:50 <lajoskatona> I can pus hthe n-vote patches for zed
15:13:56 <ykarel> thx lajoskatona
15:14:07 <ykarel> moving to next
15:14:08 <ykarel> #topic Stable branches
15:14:23 <ykarel> bcafarel, anything else apart from zed grenade issue?
15:14:32 <bcafarel> we already discussed the main topic here :) (grenade in zed/antelope branches)
15:14:50 <bcafarel> apart from that, all good - it helps with less branches to keep an eye on
15:15:06 <ykarel> thx for the update
15:15:12 <ykarel> #topic Stadium projects
15:15:17 <ykarel> all green in periodic
15:15:27 <lajoskatona> yes
15:15:27 <ykarel> lajoskatona, anything else to share for stadiums?
15:15:51 <lajoskatona> the usual: please keep an eye on the open patches on stadiums :-) thanks for it :-)
15:16:24 <ykarel> thx lajoskatona
15:16:57 <ykarel> #topic Rechecks
15:17:15 <ykarel> overall it's good here as not many rechecks in master
15:17:24 <ykarel> and just 1 bare rechecks out of 15
15:17:41 <ykarel> let's keep doing it i.e no bare rechecks
15:17:48 <ykarel> #topic fullstack/functional
15:18:11 <ykarel> we still seeing NetworkInterfaceNotFound in functional jobs
15:18:30 <ykarel> after ralonsoh fixes, seems it helped as not seeing many such failures as before
15:18:56 <ralonsoh> do you have links?
15:18:58 <ykarel> #link https://review.opendev.org/c/openstack/neutron/+/909385
15:19:05 <ralonsoh> to check if something else could be done
15:19:13 <ykarel> https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_326/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-uwsgi-fips/32627fb/testr_results.html
15:19:14 <ykarel> https://f0db0d256d3d4b350b70-3f475577843c6519bda0c8c12c28da61.ssl.cf1.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-sqlalchemy-master/1d7e278/testr_results.html
15:19:14 <ykarel> https://840d36dd41e668b905ff-5ceaba4a59a2578b108bb2f456b59bef.ssl.cf1.rackcdn.com/910208/2/check/neutron-functional-with-uwsgi/fcf02d9/testr_results.html
15:20:09 <ralonsoh> these are actually failing on the methods already decorated
15:20:27 <ykarel> symptom wise in all ^ failed at tenacity retry get/set link_attribute. and from ovs vswitchd logs all had shy port symptom, quickly add/delete/add port
15:23:40 <ykarel> so we can follow up any further action for this on the same lp bug
15:24:02 <ykarel> for now may be we can backport your retry patch to stable too?
15:24:13 <ralonsoh> sure, I'll do it
15:24:18 <ykarel> thx
15:24:35 <ykarel> next one
15:24:37 <ykarel> AttributeError: 'WaitForCreatePortBindingEvent' object has no attribute 'row'
15:24:44 <ykarel> https://e3258dd049a220460aa0-add3baf7d164319a1fbb45a2e58a56e9.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/neutron/stable/2023.1/neutron-functional-with-uwsgi-fips/1d1fbd3/testr_results.html
15:24:50 <ykarel> seen once
15:26:05 <ralonsoh> very weird, self.row is assigned in the run method,
15:26:19 <ralonsoh> if that happens, wait will exit
15:26:55 <ralonsoh> ah ah ah, we don't check that wait result
15:26:59 <ralonsoh> row_event.wait()
15:27:11 <ralonsoh> this should be self.assertTrue(row_event.wait())
15:27:36 <ralonsoh> at least we check that wait failed or not
15:27:42 <ralonsoh> I'll push a patch
15:27:55 <ykarel> thx ralonsoh
15:28:59 <ykarel> #topic Tempest/Scenario
15:29:08 <ykarel> https://b2732a041c3f8a8a2227-9262fa59dca60616978c458f4dd8b484.ssl.cf5.rackcdn.com/908973/1/gate/neutron-ovn-tempest-ipv6-only-ovs-release/e79929e/testr_results.html
15:29:08 <ykarel> https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_11c/908973/1/gate/neutron-ovs-tempest-multinode-full/11c9845/testr_results.html
15:29:47 <ykarel> also similar failure in other grenade job https://22dafa932966946580c1-ab66e702835932941e7e8df246c29428.ssl.cf5.rackcdn.com/909955/4/check/neutron-ovn-grenade-multinode-skip-level/c292b2c/testr_results.html
15:30:32 <ykarel> any volunteer to check if these some real issues?
15:31:02 <ralonsoh> I'll check them tomorrow
15:31:08 <ykarel> thx ralonsoh
15:31:30 <ykarel> #action ralonsoh to check tempest test_server_actions failures
15:31:39 <ykarel> #topic Periodic
15:31:54 <ykarel> all good except some functional test failure that were raised before
15:32:15 <ykarel> #topic Grafana
15:32:15 <ykarel> https://grafana.opendev.org/d/f913631585/neutron-failure-rate
15:32:26 <ykarel> let's have quick look at it too
15:33:31 <ykarel> apart from some spikes in check queue rest looks good
15:33:43 <mlavalle> lgtm as well
15:33:50 <ykarel> when i was checking failures most of those failures were patch related
15:34:22 <ykarel> ok moving on
15:34:30 <ykarel> #topic On Demand
15:34:42 <ykarel> anything else you would like to raise?
15:35:06 <mlavalle> not me
15:35:20 <slaweq> nothing from me
15:35:49 <lajoskatona> nothing from me
15:37:04 <ykarel> k in that case let's close early and have everyone 23 minutes back :)
15:37:09 <ykarel> #endmeeting