15:00:07 <slaweq> #startmeeting neutron_ci
15:00:07 <opendevmeet> Meeting started Tue Aug 17 15:00:07 2021 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:07 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:07 <opendevmeet> The meeting name has been set to 'neutron_ci'
15:00:12 <slaweq> hi
15:00:32 <lajoskatona> Hi
15:00:44 <ralonsoh> hi
15:01:08 <obondarev> hi
15:01:15 <slaweq> Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate
15:01:38 <slaweq> and let's move on
15:01:40 <slaweq> #topic Actions from previous meetings
15:01:51 <slaweq> slaweq to report failure in test_ipv6_router_advts_and_fwd_after_router_state_change_backup functional test
15:01:56 <slaweq> Bug reported https://bugs.launchpad.net/neutron/+bug/1939507
15:02:06 <slaweq> I proposed some patch with extra logs for now
15:02:22 <slaweq> as I don't know exactly why it was failing
15:02:37 <slaweq> with extra logs hopefully I will know better what happened there
15:03:12 <slaweq> ok, next one
15:03:14 <slaweq> ralonsoh to send email about memory in CI vms
15:03:31 <ralonsoh> I forgot that, I'll do it right now
15:03:36 <slaweq> thx
15:03:47 <slaweq> but please wait a bit
15:03:52 <ralonsoh> sure
15:03:57 <slaweq> I have this topic to discuss more today
15:04:01 <ralonsoh> ok
15:04:09 <slaweq> so let's wait until end of the meeting at least :)
15:04:15 <slaweq> next one
15:04:17 <slaweq> ralonsoh to check neutron-ovn-tempest-ovs-master-fedora job failures
15:04:32 <ralonsoh> there are several devstack patches ongoing
15:04:42 <ralonsoh> (let me find the bug)
15:04:58 <ralonsoh> https://bugs.launchpad.net/neutron/+bug/1939429
15:05:06 <ralonsoh> patches: https://review.opendev.org/q/topic:%22uncap-pip-url-resolver%22+(status:open%20OR%20status:merged)
15:05:13 <ralonsoh> actually, all of them are merged now
15:05:21 <ralonsoh> the periodic job should work now
15:05:33 <slaweq> it is fine indeed
15:05:35 <slaweq> thx a lot
15:05:52 <slaweq> You can close LP bug :)
15:06:01 <slaweq> next one
15:06:03 <slaweq> obondarev to promote dvr-ha job to be voting
15:06:09 <obondarev> done: https://review.opendev.org/c/openstack/neutron/+/804211
15:06:32 <slaweq> obondarev: can You also update our grafana dashboard with that change?
15:06:54 <slaweq> and great work with stabilizing that job finally
15:06:58 <obondarev> sure, please remind which repo is it?
15:07:01 <slaweq> thx a lot obondarev for all that work
15:07:14 <slaweq> it's project-config
15:07:17 <slaweq> let me find link
15:07:22 <obondarev> ah, got it, thanks
15:07:27 <slaweq> https://opendev.org/openstack/project-config/
15:07:29 <slaweq> :)
15:07:33 <obondarev> please set an AI for me
15:07:52 <slaweq> #action obondarev to update grafana after promoting dvr-ha job to be voting
15:07:55 <slaweq> thx a lot
15:08:03 <obondarev> thanks!
15:08:19 <slaweq> that's all AI from last week
15:08:26 <slaweq> next topic
15:08:28 <slaweq> #topic Stadium projects
15:08:32 <slaweq> lajoskatona: any updates?
15:08:56 <lajoskatona> things are fine, a lot ot patches merged :-)
15:09:04 <slaweq> that's good to hear :)
15:10:14 <slaweq> so let's move on to the next topic
15:10:17 <slaweq> #topic Stable branches
15:10:27 <slaweq> there is no bcafarel today
15:10:41 <slaweq> but I just wanted to ask You for review https://review.opendev.org/c/openstack/networking-bgpvpn/+/796476/ and https://review.opendev.org/c/openstack/networking-bagpipe/+/796477
15:11:27 <lajoskatona> sure, I just seen them :-)
15:11:51 <slaweq> thx
15:12:01 <slaweq> lajoskatona: now You should have +2 power there
15:12:21 <lajoskatona> yes, I used it today for some other patches :P
15:13:02 <slaweq> great :)
15:13:16 <slaweq> anything else regarding stable branches' ci?
15:14:16 <slaweq> ok, so let's move on
15:14:18 <slaweq> #topic Grafana
15:14:21 <slaweq> https://grafana.opendev.org/d/BmiopeEMz/neutron-failure-rate?orgId=1
15:14:36 <slaweq> I don't see anything new what would be urgent there
15:14:47 <slaweq> we still have same issues with functional and scenario jobs
15:14:49 <slaweq> :/
15:15:09 <slaweq> but maybe You see something what You would like to raise regarding our dashboard
15:16:25 <slaweq> if not, let's move on
15:16:49 <slaweq> I will go strictly to the scenario jobs as I don't have anything new to discuss regarding functional/fullstack today
15:16:53 <slaweq> #topic Tempest/Scenario
15:17:18 <slaweq> Seems that we have serious problem with oom-killer in neutron_tempest_plugin.scenario.* jobs
15:17:26 <slaweq> I opened bug today https://bugs.launchpad.net/neutron/+bug/1940243
15:17:37 <slaweq> I found at least 10 examples from last week :/
15:17:41 <slaweq> so it's a lot
15:17:45 <ralonsoh> indeed
15:18:08 <slaweq> after initial look, I think that we have now more tests which are spawning Ubuntu vms
15:18:12 <slaweq> and that's causing problem
15:18:30 <slaweq> in one case for example I saw that mysql was using around 8.8% of memory
15:18:38 <lajoskatona> so it happens with all backends?
15:18:49 <slaweq> but then there were 4 different qemu processes which used around 8% each
15:18:57 <slaweq> lajoskatona: yes, all backends
15:19:35 <slaweq> my proposal is to somehow extract tests which are using advanced image and run them with --concurrency 1 as second "stage" in the job
15:19:48 <slaweq> something similar what we are doing in fullstack job for example
15:19:51 <ralonsoh> right
15:20:03 <slaweq> that way we shouldn't have so many Ubuntu vms spawned in same time
15:20:09 <slaweq> but maybe You have other ideas
15:20:12 <ralonsoh> at least big ones
15:20:40 <ralonsoh> ask for bigger testing VMs... (that's easy)
15:20:58 <slaweq> ralonsoh: but I don't know if that is even possible to get bigger vms
15:21:12 <slaweq> other possibility maybe could be to use different flavor for Ubuntu vms
15:21:27 <slaweq> maybe flavor with slightly smaller memory would be still ok for our needs
15:21:31 <ralonsoh> we are suing cirros256
15:21:32 <lajoskatona> I asked once for ODL, but I got it only temporarily
15:21:37 <ralonsoh> and for advance?
15:22:06 <slaweq> ralonsoh: but for advanced I think we are using d1 (or something like that) which is 512MB
15:22:06 <ralonsoh> the point is that the VM memory is dynamic
15:22:28 <ralonsoh> qemu won't reserve all the flavor requested mem
15:22:37 <ralonsoh> so we are safe on this
15:22:46 <slaweq> but if Ubuntu vm have 512M, it may try to use it
15:23:09 <ralonsoh> well, we can reduce it to 384
15:23:16 <ralonsoh> at least we can try
15:23:27 <slaweq> that should buy us some time at least :)
15:23:31 <ralonsoh> yes
15:23:55 <lajoskatona> and we can tag the advanced tests and run them with concurency 1
15:24:07 <lajoskatona> as slaweq suggested
15:24:29 <ralonsoh> yes. I can push this patch (but I won't send the mail yet top request bigger VMs)
15:25:06 <slaweq> ralonsoh: yes, that's why I wanted You to hold on until end of the meeting :)
15:25:32 <ralonsoh> perfect then
15:25:52 <slaweq> so ralonsoh, will You push patch to use smaller flavor in our jobs? and/or tag advanced tests to run them with concurency 1?
15:26:06 <ralonsoh> both patches
15:26:13 <slaweq> thx a lot
15:26:35 <slaweq> #action ralonsoh to send patches to use smaller flavors and concurency 1 for advanced scenario tests
15:26:56 <slaweq> I hope that this will help with that issue as this is killing us now
15:27:20 <slaweq> anything else regarding scenario jobs?
15:27:30 <ralonsoh> not from me
15:28:05 <slaweq> if not, let's move one
15:28:11 <slaweq> we have one more topic for today
15:28:22 <slaweq> #topic Periodic
15:28:32 <slaweq> I opened today bug https://bugs.launchpad.net/neutron/+bug/1940224
15:28:52 <slaweq> it seems that since few days our UT running with neutron-lib from master branch are broken
15:29:13 <slaweq> so we probably need quick fix for it or we will have broken neutron gate when we will do new release
15:29:27 <ralonsoh> I'll check this tomorrow
15:29:28 <opendevreview> Bodo Petermann proposed openstack/neutron-vpnaas master: VPNaaS support for OVN  https://review.opendev.org/c/openstack/neutron-vpnaas/+/765353
15:29:32 <slaweq> thx ralonsoh
15:29:42 <slaweq> I didn't had time to look into it yet,  sorry
15:30:20 <slaweq> ralonsoh: if You will find that fix needs to be in neutron-lib, please let me know asap and/or please comment in https://review.opendev.org/c/openstack/releases/+/804666
15:30:32 <slaweq> so we will not release broken neutron-lib
15:30:38 <ralonsoh> sure, I'll check it asap
15:30:42 <slaweq> thx a lot
15:30:57 <slaweq> #action ralonsoh to check UT failing with neutron-lib master
15:31:08 <slaweq> and that's all from me for today :)
15:31:16 <slaweq> anything else regarding our CI?
15:31:24 <slaweq> or if not, we can finish earlier today
15:31:36 <ralonsoh> I'm fine
15:31:44 <lajoskatona> nothing from me
15:32:03 <slaweq> ok, thx for attending the meeting
15:32:15 <slaweq> see You online and have a great evening :)
15:32:17 <slaweq> o/
15:32:19 <slaweq> #endmeeting