15:00:16 <kopecmartin> #startmeeting qa
15:00:16 <opendevmeet> Meeting started Tue Jun  6 15:00:16 2023 UTC and is due to finish in 60 minutes.  The chair is kopecmartin. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:16 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:16 <opendevmeet> The meeting name has been set to 'qa'
15:00:22 <kopecmartin> #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Agenda_for_next_Office_hours
15:00:24 <kopecmartin> agenda ^
15:00:30 <frickler> \o
15:00:40 <kopecmartin> o/
15:01:31 <lpiwowar> o/
15:02:46 <kopecmartin> #topic Announcement and Action Item (Optional)
15:03:12 <kopecmartin> there's gonna be the openinfra summit next week
15:03:24 <lpiwowar> \o/
15:03:42 <kopecmartin> i'm gonna cancel our regular office hour on June 13th
15:04:01 <kopecmartin> one other announcement
15:04:09 <kopecmartin> we're about to release xena-last tempest release
15:04:10 <kopecmartin> #link https://review.opendev.org/q/topic:xena-last
15:05:07 <kopecmartin> #topic Bobcat Priority Items progress
15:05:11 <kopecmartin> #link https://etherpad.opendev.org/p/qa-bobcat-priority
15:05:45 * frickler didn't get to check the venv failures
15:07:20 <frickler> I also wanr to bump cirros, they made new releases with updated kernels
15:08:05 <kopecmartin> me neither :/
15:08:20 <kopecmartin> yeah, sounds good, we could at least propose the change and see how many things will get broken
15:09:07 <frickler> I hope not too many, unless canonical has introduced regressions into the kernel
15:09:50 <kopecmartin> yup , we'll see
15:10:31 <kopecmartin> #topic OpenStack Events Updates and Planning
15:10:59 <kopecmartin> anyone else planning to be in Vancouver except me, gmann and lpiwowar?
15:11:26 <kopecmartin> the link for the etherpad, if there are any topics to be discussed in person:
15:11:27 <kopecmartin> #link https://etherpad.opendev.org/p/vancouver-june2023-qa
15:13:47 <kopecmartin> moving on
15:13:48 <kopecmartin> #topic Gate Status Checks
15:13:53 <kopecmartin> #link https://review.opendev.org/q/label:Review-Priority%253D%252B2+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade)
15:14:03 <kopecmartin> nothing there
15:14:07 <kopecmartin> any urgent patches to review?
15:14:28 <lpiwowar> I updated the comment on this patch: https://review.opendev.org/c/openstack/tempest/+/881575 ... But I do not think it is urgent.
15:14:55 <lpiwowar> It is the patch that fixes cleanup for the object_storate tests.
15:15:01 <kopecmartin> thanks, yeah, right now there are couple of patches before we can merge this one (considering the current unstable behaviour of the CI)
15:15:24 <lpiwowar> Got it:)
15:16:14 <kopecmartin> #topic Bare rechecks
15:16:18 <kopecmartin> #link https://etherpad.opendev.org/p/recheck-weekly-summary
15:16:25 <kopecmartin> this hasn't been updated, so nothing new there
15:16:34 <kopecmartin> #topic Periodic jobs Status Checks
15:16:34 <kopecmartin> periodic stable full
15:16:34 <kopecmartin> #link https://zuul.openstack.org/builds?pipeline=periodic-stable&job_name=tempest-full-yoga&job_name=tempest-full-xena&job_name=tempest-full-zed&job_name=tempest-full-2023-1
15:16:36 <kopecmartin> periodic stable slow
15:16:38 <kopecmartin> #link https://zuul.openstack.org/builds?job_name=tempest-slow-2023-1&job_name=tempest-slow-zed&job_name=tempest-slow-yoga&job_name=tempest-slow-xena
15:16:40 <kopecmartin> periodic extra tests
15:16:42 <kopecmartin> #link https://zuul.openstack.org/builds?job_name=tempest-full-2023-1-extra-tests&job_name=tempest-full-zed-extra-tests&job_name=tempest-full-yoga-extra-tests&job_name=tempest-full-xena-extra-tests
15:16:44 <kopecmartin> periodic master
15:16:46 <kopecmartin> #link https://zuul.openstack.org/builds?project=openstack%2Ftempest&project=openstack%2Fdevstack&pipeline=periodic
15:17:54 <kopecmartin> tempest-slow-2023-1 timeouts in ~50% of all the runs :/
15:19:04 <frickler> could these be related to mysql slowdown?
15:19:24 <kopecmartin> maybe, it took 50 minutes to deploy devstack
15:19:30 <kopecmartin> #link https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6d0/periodic-stable/opendev.org/openstack/tempest/master/tempest-slow-2023-1/6d0d1c2/job-output.txt
15:20:03 <kopecmartin> *40
15:21:05 <kopecmartin> and the tests took 2 hours, what's strange is that tempest was executed with concurrency=2 but i see only one thread running tests o.O
15:22:03 <frickler> tempest run --serial --regex '\[.*\bslow\b.*\]' --concurrency=2
15:22:05 <kopecmartin> in a passing build, the installation of devstack takes only 30 min
15:22:11 <frickler> --serial
15:22:42 <kopecmartin> oh, yeah, so why is --concurrency=2 there?
15:23:03 <frickler> to confuse casual reviewers? ;)
15:23:10 <kopecmartin> :D for sure
15:23:58 <frickler> maybe we need to split this into two jobs, half the tests each
15:24:57 <frickler> or some day rekindle the discussion with opendev about flavor memory
15:25:02 <kopecmartin> right! --serial and --parallel decide whether the --concurrency value will be used , if --serial is passed, the concurrency is ignored ...
15:25:59 <kopecmartin> what's ironic about that is that slow job was created to avoid timeouts :D
15:26:10 <kopecmartin> ... or try to run it in parallel?
15:26:37 <kopecmartin> although it might timeout too but from different reasons
15:26:47 <frickler> we could give it a try, but I fear that it will trigger oom issues
15:27:52 <kopecmartin> gmann: what do you think?
15:28:03 <kopecmartin> i can propose a patch to run the tests in parallel, just as an experiment
15:28:50 <kopecmartin> the other jobs look good, not sure why centos9-stream fails again though
15:28:58 <kopecmartin> but it's just started today, maybe it won't repeat
15:29:26 <kopecmartin> and the tempest-full-test-account-no-admin-py3 job will be fixed once lpiwowar's patch is merged (was already linked above)
15:29:47 <lpiwowar> kopecmartin: +1
15:29:54 <kopecmartin> #topic Distros check
15:29:54 <kopecmartin> cs-9
15:29:54 <kopecmartin> #link https://zuul.openstack.org/builds?job_name=tempest-full-centos-9-stream&job_name=devstack-platform-centos-9-stream&skip=0
15:29:56 <kopecmartin> fedora
15:29:58 <kopecmartin> #link https://zuul.openstack.org/builds?job_name=devstack-platform-fedora-latest&skip=0
15:30:00 <kopecmartin> debian
15:30:02 <kopecmartin> #link https://zuul.openstack.org/builds?job_name=devstack-platform-debian-bullseye&skip=0
15:30:04 <kopecmartin> focal
15:30:07 <kopecmartin> #link https://zuul.opendev.org/t/openstack/builds?job_name=devstack-platform-ubuntu-focal&skip=0
15:30:09 <kopecmartin> rocky
15:30:11 <kopecmartin> #link https://zuul.openstack.org/builds?job_name=devstack-platform-rocky-blue-onyx
15:30:14 <kopecmartin> openEuler
15:30:16 <kopecmartin> #link https://zuul.openstack.org/builds?job_name=devstack-platform-openEuler-22.03-ovn-source&job_name=devstack-platform-openEuler-22.03-ovs&skip=0
15:30:39 <kopecmartin> we're about to delete the devstack-platform-ubuntu-focal job
15:30:45 <frickler> did we talk about dropping fedora already?
15:30:55 <kopecmartin> #link https://review.opendev.org/c/openstack/tempest/+/884952
15:31:03 <kopecmartin> once the CI agrees and finally merges that
15:31:24 <kopecmartin> i think so
15:31:27 <kopecmartin> let me try to find it
15:31:39 <frickler> hmm, I can force merge the above, got some other forcing to do anyway
15:31:56 <frickler> and since it is job removal only, no need to wait for random jobs passing
15:32:12 <kopecmartin> that be great frickler , thanks
15:33:02 <kopecmartin> found it
15:33:04 <kopecmartin> #link https://meetings.opendev.org/meetings/qa/2023/qa.2023-05-23-15.00.log.html#l-85
15:33:09 <frickler> can we also delete the devstack focal platform job as followup?
15:33:21 <kopecmartin> we talked about fedora couple weeks ago
15:33:25 <kopecmartin> hm, one moment
15:34:50 <kopecmartin> seems so
15:34:59 <kopecmartin> it's not used anywhere else
15:35:01 <kopecmartin> #lin https://codesearch.opendev.org/?q=devstack-platform-ubuntu-focal&i=nope&literal=nope&files=&excludeFiles=&repos=
15:35:59 <frickler> oh, so I said I'd do fedora removal patches. guess I need to do that, then :D
15:36:59 <kopecmartin> :D loop me in the reviews please
15:37:28 <frickler> guess I'll combine focal + fedora
15:37:45 <frickler> for the alliteration fun + to avoid conflicts
15:38:08 <kopecmartin> sounds good
15:38:30 <kopecmartin> #topic Sub Teams highlights
15:38:34 <kopecmartin> Changes with Review-Priority == +1
15:38:38 <kopecmartin> #link https://review.opendev.org/q/label:Review-Priority%253D%252B1+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade)
15:39:09 <kopecmartin> we have there the 3 patches to tag tempest and this one
15:39:10 <kopecmartin> #link https://review.opendev.org/c/openstack/tempest/+/884764
15:40:51 <kopecmartin> i'm on those
15:41:23 <kopecmartin> one strange thing in https://review.opendev.org/c/openstack/tempest/+/885089 gmann, otherwise good to go
15:41:35 <kopecmartin> #topic Open Discussion
15:41:38 <kopecmartin> anything for the open discussion?
15:41:58 <frickler> the GIT_DEPTH thing?
15:42:49 <frickler> at first I was wondering whether we should revert the patch until we have a solution. but it also got backported already
15:43:10 <kopecmartin> #link https://bugs.launchpad.net/devstack/+bug/2023020
15:43:19 <kopecmartin> i've reported that so that we have it tracked
15:43:28 <kopecmartin> it's not that urgent
15:43:50 <kopecmartin> the bug we fixed by that patch was a bit more pressing and was blocking some jobs
15:44:20 <kopecmartin> anyway, it's a bug and we need to find a solution working with depth too
15:46:21 <frickler> ok
15:47:15 <kopecmartin> #topic Bug Triage
15:47:20 <kopecmartin> #link https://etherpad.openstack.org/p/qa-bug-triage-bobcat
15:48:26 <kopecmartin> this is basically all from my side
15:49:32 <kopecmartin> thank you everyone
15:49:41 <kopecmartin> see you in Vancouver or in 2 weeks online :)
15:49:51 <kopecmartin> #endmeeting