Wednesday, 2020-05-27

*** ricolin has joined #openstack-meeting-300:57
*** jamesmcarthur has joined #openstack-meeting-302:35
*** jamesmcarthur has quit IRC03:26
*** jamesmcarthur has joined #openstack-meeting-303:27
*** psachin has joined #openstack-meeting-303:30
*** jamesmcarthur has quit IRC03:35
*** jamesmcarthur has joined #openstack-meeting-303:46
*** jamesmcarthur has quit IRC03:51
*** jamesmcarthur has joined #openstack-meeting-305:13
*** psachin has quit IRC05:16
*** jamesmcarthur has quit IRC05:17
*** jamesmcarthur has joined #openstack-meeting-305:18
*** ianychoi has quit IRC05:34
*** links has joined #openstack-meeting-305:41
*** jamesmcarthur has quit IRC05:50
*** jamesmcarthur has joined #openstack-meeting-305:52
*** jamesmcarthur has quit IRC06:22
*** jamesmcarthur has joined #openstack-meeting-306:23
*** jamesmcarthur has quit IRC06:29
*** ralonsoh has joined #openstack-meeting-306:52
*** jamesmcarthur has joined #openstack-meeting-307:00
*** belmoreira has joined #openstack-meeting-307:01
*** jamesmcarthur has quit IRC07:10
*** slaweq has joined #openstack-meeting-307:31
*** jamesmcarthur has joined #openstack-meeting-307:46
*** jamesmcarthur has quit IRC07:55
*** links has quit IRC08:11
*** mugsie_ has joined #openstack-meeting-308:14
*** melwitt has quit IRC08:20
*** mugsie has quit IRC08:20
*** slaweq has quit IRC08:22
*** links has joined #openstack-meeting-308:23
*** slaweq has joined #openstack-meeting-308:50
*** slaweq has quit IRC08:56
*** links has quit IRC09:01
*** links has joined #openstack-meeting-309:19
*** slaweq has joined #openstack-meeting-309:19
*** rubasov has quit IRC09:39
*** rubasov has joined #openstack-meeting-309:41
*** jamesmcarthur has joined #openstack-meeting-309:52
*** jamesmcarthur has quit IRC09:58
*** lpetrut has joined #openstack-meeting-310:02
*** slaweq has quit IRC10:25
*** slaweq has joined #openstack-meeting-310:26
*** slaweq has quit IRC10:30
*** e0ne has joined #openstack-meeting-311:03
*** raildo has joined #openstack-meeting-311:42
*** slaweq has joined #openstack-meeting-311:51
*** jamesmcarthur has joined #openstack-meeting-311:55
*** slaweq has quit IRC11:56
*** jamesmcarthur has quit IRC12:00
*** benj_ has quit IRC12:36
*** e0ne_ has joined #openstack-meeting-313:14
*** e0ne has quit IRC13:15
*** lpetrut has quit IRC13:18
*** links has quit IRC13:26
*** slaweq has joined #openstack-meeting-313:42
*** slaweq has quit IRC13:56
*** jamesmcarthur has joined #openstack-meeting-313:56
*** e0ne_ has quit IRC14:00
*** e0ne has joined #openstack-meeting-314:01
*** jamesmcarthur has quit IRC14:01
*** liuyulong has joined #openstack-meeting-314:02
*** links has joined #openstack-meeting-314:07
*** jamesmcarthur has joined #openstack-meeting-314:09
*** slaweq has joined #openstack-meeting-314:46
slaweq#startmeeting neutron_ci15:00
openstackMeeting started Wed May 27 15:00:11 2020 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: neutron_ci)"15:00
openstackThe meeting name has been set to 'neutron_ci'15:00
slaweqhi15:00
maciejjozefczykhey15:00
njohnstono/15:00
bcafarelo/15:01
*** lajoskatona has joined #openstack-meeting-315:01
lajoskatonao/15:01
slaweqGrafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate15:01
slaweqPlease open now :)15:01
slaweqand I think that we can start15:01
slaweq#topic Actions from previous meetings15:01
*** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)"15:01
slaweqfirst one15:01
slaweqralonsoh to continue checking ovn jobs timeouts15:02
ralonsohsorry15:02
ralonsohI'm late15:02
slaweqralonsoh: no problem :)15:02
ralonsohI spent some time on this again15:02
ralonsohwith ovsdbapp and python-ovn developers15:02
ralonsohand I still don't find a "breach" in the code15:03
ralonsohthis is something permanent in my plate15:03
ralonsohcan we put this task on hold?15:03
slaweqbut it still happens in the ci, right?15:03
ralonsohsometimes15:03
ralonsohbut no so often15:03
slaweqok15:03
ralonsohand this is something that happens in OVS too15:04
slaweqso it's not always related to ovn jobs? also ml2/ovs jobs has got the same issue?15:04
ralonsohyes15:04
slaweqok15:05
ralonsohit's a problem with eventlet, ovsdbapp and pyuthon-ovs15:05
slaweqouch, probably will be hard to find :/15:05
ralonsohpfffff15:05
slaweq:)15:05
slaweqok, I know You will continue this investigation so I think we can move on15:07
slaweqnext one15:07
slaweq    slaweq to check failure in test_ha_router_failover: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6d0/726168/2/check/neutron-functional/6d0b174/testr_results.html15:07
slaweqI have it in my todo list but I didn't have time to get to this one yet15:07
slaweqI will try this week15:07
slaweq#action slaweq to check failure in test_ha_router_failover: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6d0/726168/2/check/neutron-functional/6d0b174/testr_results.html15:07
slaweqnext one15:08
slaweqslaweq to add additional logging for fullstack's firewall tests15:08
*** melwitt has joined #openstack-meeting-315:09
*** e0ne has quit IRC15:09
slaweqand here it is the same: I have it in my todo list but I didn't have time yet to get to this15:09
*** e0ne has joined #openstack-meeting-315:09
slaweqI reopened bug related to this test and marked it as unstable again15:09
slaweqso it will not make our life harder :)15:09
slaweq#action slaweq to add additional logging for fullstack's firewall tests15:10
slaweqnext one15:10
njohnstonthere si so much that goes on in that one test, would it make sense to break it up?15:10
slaweqnjohnston: I was thinking about that, and I even started something https://review.opendev.org/#/c/716773/15:11
slaweqbut it will require some more work15:11
njohnstoncool!  As always, you are ahead of the curve. :-)15:11
slaweqand I agree that this would be good to break it into few smaller tests15:11
slaweqnjohnston: thx :)15:11
slaweqnjohnston: so I will continue this effort in next weeek(s) but as low priority task15:13
njohnstonmakes sense15:13
slaweqok15:14
slaweqthx15:14
slaweqso lets move on15:14
slaweqslaweq to reopen bug related to failing fuillstack firewall tests15:14
slaweqas I said I reopended bug https://bugs.launchpad.net/neutron/+bug/174240115:14
openstackLaunchpad bug 1742401 in neutron "Fullstack tests neutron.tests.fullstack.test_securitygroup.TestSecurityGroupsSameNetwork fails often" [High,Confirmed] - Assigned to Slawek Kaplonski (slaweq)15:14
slaweqand marked test as unstable again15:14
*** links has quit IRC15:15
slaweqok, and the last one15:16
slaweqralonsoh to check  Address already allocated in subnet issue in tempest job15:16
ralonsohI'm on it, sorry15:16
ralonsohdidn't spend too much time on this one15:16
slaweqralonsoh: I can imagine as I know what You were doing for most of the week :)15:17
* slaweq also don't likes conjuntions :P15:17
maciejjozefczyk:15:18
maciejjozefczyk:P15:18
slaweqwith that I think we can move on to the next topic15:18
slaweq#topic Stadium projects15:18
*** openstack changes topic to "Stadium projects (Meeting topic: neutron_ci)"15:18
slaweqI don't have anything related to the stadium for today15:18
slaweqbut maybe You have something to discuss here?15:19
njohnstonnope15:19
lajoskatonanothing special, at least from me15:19
slaweqok, so lets move on to the next topic15:20
slaweq#topic Stable branches15:20
*** openstack changes topic to "Stable branches (Meeting topic: neutron_ci)"15:20
*** links has joined #openstack-meeting-315:20
slaweqTrain dashboard: http://grafana.openstack.org/d/pM54U-Kiz/neutron-failure-rate-previous-stable-release?orgId=115:20
slaweqStein dashboard: http://grafana.openstack.org/d/dCFVU-Kik/neutron-failure-rate-older-stable-release?orgId=115:20
bcafarelUssuri and Train now :)15:20
njohnstonnice!15:20
slaweqbcafarel: are You sure?15:20
slaweqI see in the description train and stein still15:21
slaweqmaybe You forgot change description?15:21
bcafarelargh, checking15:21
bcafarelthe changeset merged for sure15:21
bcafarelnah, names were updated too, I guess 14h ago is too recent?15:22
bcafarel( https://review.opendev.org/#/c/729291/ )15:23
slaweqok, lets wait some more time15:23
slaweqhopefully it will change15:23
slaweqand thx bcafarel for taking care of it15:24
bcafarelsomething to check on next meeting :)15:24
bcafareland apart from that from memory I don't think I have seen many stable failures15:24
bcafarel(also new stable releases for trein/stain)15:24
bcafarel*train/stein15:25
slaweqyes, me too - most of my patches were merged pretty fast recently15:25
slaweqif there is nothing else regarding stable branches, I think we can move on to the next topic15:26
*** e0ne_ has joined #openstack-meeting-315:28
*** e0ne has quit IRC15:28
slaweq#topic Grafana15:29
*** openstack changes topic to "Grafana (Meeting topic: neutron_ci)"15:29
slaweqhttp://grafana.openstack.org/dashboard/db/neutron-failure-rate15:29
slaweqI don't know but our gate queue looks very good since few days - no failures at all :)15:32
slaweqor almost at all15:32
slaweqbut it's worst in the check queue15:32
slaweqe.g. neutron-ovn-tempest-slow is failing 100% times15:33
maciejjozefczykhmm, I may know whats going there15:36
slaweqmaciejjozefczyk: I was hoping that You will know :P15:36
maciejjozefczykI can take a look, this failing test seems similar to me :P (test_port_security_macspoofing_port)15:36
bcafarelhmm yeah the name does ring a bell15:36
slaweqyes, that's the test which is failing every time since around last friday15:36
maciejjozefczykI think we blacklisted that test in other jobs15:36
maciejjozefczykthere is a fix in core-ovn for this but I think its not yet in the OVN version we use in the gates15:37
slaweqmaciejjozefczyk: please check that if You will have some time15:37
maciejjozefczyksure, thats gonan be quick :) slaweq15:37
slaweq#action maciejjozefczyk to check failing test_port_security_macspoofing_port test15:37
maciejjozefczykI left the vivaldi tab open for tomorrow morning to not forget :)15:37
slaweqother than that things looks quite good IMO15:37
slaweqanything else You want to discuss regarding grafana?15:38
slaweqok, so next topic15:39
slaweq#topic fullstack/functional15:39
*** openstack changes topic to "fullstack/functional (Meeting topic: neutron_ci)"15:39
slaweqI have only one thing regarding functional job15:40
slaweq    What to do with https://review.opendev.org/#/c/729588/ ? IMO it's good to go15:40
slaweqthis uwsgi job is even more stable recently than "normal" functional tests job15:40
ralonsohI think so15:40
njohnston+115:40
ralonsoh+115:40
lajoskatona+115:40
slaweqok, so please review and approve it :)15:41
slaweqthx lajoskatona bcafarel and maciejjozefczyk for review of it already :)15:41
maciejjozefczyk:) +115:42
slaweqfullstack tests seems to be better when I marked this one test as unstable15:42
lajoskatonaslaweq: no problem15:42
slaweqany questions/comments or can we move on?15:42
njohnstongo ahead15:43
bcafarelnothing from me15:43
slaweqok15:44
slaweq#topic Tempest/Scenario15:44
*** openstack changes topic to "Tempest/Scenario (Meeting topic: neutron_ci)"15:44
slaweqwe already talked about failing ovn-slow job15:44
slaweqthe only other issue which I have today is yet another error with IpAddressAlreadyAllocated error15:44
slaweqthis time in grenade job:     https://7ba1272f105c99db4826-d9c28f5658476db1e4ca5968d196888d.ssl.cf2.rackcdn.com/729591/1/check/neutron-grenade-dvr-multinode/54a824e/controller/logs/grenade.sh_log.txt15:44
*** e0ne_ has quit IRC15:45
*** e0ne_ has joined #openstack-meeting-315:46
slaweqI'm not sure why it's like that but for me it smells like issue in the test15:48
slaweqok15:49
slaweqI found15:49
slaweqMay 20 14:02:57.487624 ubuntu-bionic-rax-ord-0016690112 neutron-server[5514]: INFO neutron.wsgi [None req-e0e86a16-03d9-4d2c-86fd-db9c1c4133b9 tempest-RoutersTest-1419393395 tempest-RoutersTest-1419393395] 10.209.98.10 "PUT /v2.0/routers/68d7b71d-47e2-4b98-9f87-d972e2f3889d/add_router_interface HTTP/1.1" status: 409  len: 369 time: 38.498257915:49
slaweqsorry15:50
slaweqfirst was:15:50
slaweqMay 20 14:03:00.501061 ubuntu-bionic-rax-ord-0016690112 neutron-server[5514]: INFO neutron.wsgi [None req-a76c0e0a-7bc9-4c57-94e9-fe92c402f768 tempest-RoutersTest-1419393395 tempest-RoutersTest-1419393395] 10.209.98.10 "PUT /v2.0/routers/68d7b71d-47e2-4b98-9f87-d972e2f3889d/add_router_interface HTTP/1.1" status: 200  len: 503 time: 102.889462215:50
slaweqthis took long time, so client ended up with timeout and retried request15:50
ralonsohslaweq, I found the error for trhe IpAddressAlreadyAllocated15:50
ralonsohslaweq, https://bugs.launchpad.net/neutron/+bug/188097615:50
openstackLaunchpad bug 1880976 in neutron "[tempest] Error in "test_reuse_ip_address_with_other_fip_on_other_router" with duplicated floating IP" [Undecided,New]15:50
ralonsohreported 5 mins ago15:50
slaweqbut in the meantime it was allocated already so second request was 40915:51
slaweqso at least in this grenade job it's nothing really new15:51
ralonsohin a nutshell: between the FIP deletion and the creation again with the same IP address, another test requested an FIP15:51
ralonsohand the server gave the same IP15:51
ralonsohjust a coincidence15:51
njohnstonugh15:51
ralonsohI'll try to reduce the time between the deletion and the new creation15:51
ralonsohreusing the IP address15:52
slaweqnice catch ralonsoh15:52
ralonsohgood luck!15:52
njohnstonit's just unfortunate that with our ipam randomization work that it got the same IP15:52
ralonsohyeah...15:52
bcafarellower probability but still not 0%15:52
njohnstonbad roll of the dice15:52
slaweqralonsoh: so is it conflict of FIP or fixed IP?15:53
ralonsohFIP15:53
slaweqhow this test works?15:53
ralonsohsimple15:53
slaweqdo we need to delete it?15:53
ralonsoh2 vms with FIP15:53
ralonsoh1 vm deleted and the FIP15:53
ralonsohthen a VM is created and the FIP is created again, "reusing" the IP address15:54
slaweqmaybe we could just create FIP, attach to the first vm, detach from the vm, attach to the second vm?15:54
slaweqthat way FIP would be "reserved" in the tenant for whole test15:54
slaweqam I right?15:54
ralonsohbut we really need to delete it15:54
ralonsohone sec15:54
ralonsohit's in the test case description15:55
ralonsohhttps://github.com/openstack/neutron-tempest-plugin/blob/7b374486a54456d3c67fd2961c5894fb64ba48ab/neutron_tempest_plugin/scenario/test_floatingip.py#L518-L53615:55
ralonsohstep 615:55
ralonsoh(btw, we SHOULD document always the test cases like this)15:56
njohnston+1 for documenting15:56
slaweqI agree15:56
ralonsohso clear, with those steps15:56
slaweqand according to the test, I still don't see any reason why FIP has to be deleted? if we would detach it from VM it would be just DB record15:57
ralonsohyou are right15:57
slaweqso You don't need to "remember" its IP address but just reuse it later for VM315:57
ralonsohwe really don't need to delete the DB register15:57
slaweqand this should be more stable IMHO15:57
ralonsohokidoki, I'll propose a patch15:58
slaweqralonsoh++ thx15:58
bcafarelnice15:58
slaweqand with that we are almost out of time15:58
slaweqat the end one quick note about periodic jobs15:58
slaweqall seems ok this week15:58
lajoskatonajust a question: isn't it possible to run serially these tests?15:58
lajoskatonatoo much time?15:58
slaweqour job with ovsdbapp from mast is fine now15:58
slaweqlajoskatona: yes, it would take too much time15:59
lajoskatonaok15:59
slaweqin tempest-slow job tests are run in serial but there is only few of them15:59
slaweqthx for attending the meeting, see You all tomorrow :)15:59
slaweqo/15:59
bcafarelo/15:59
slaweq#endmeeting15:59
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"15:59
openstackMeeting ended Wed May 27 15:59:55 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:59
lajoskatonao/15:59
maciejjozefczyk\o15:59
openstackMinutes:        http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-05-27-15.00.html15:59
ralonsohbye16:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-05-27-15.00.txt16:00
njohnstono/16:00
openstackLog:            http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-05-27-15.00.log.html16:00
njohnstonhave a great rest of your day off slaweq16:00
*** lajoskatona has left #openstack-meeting-316:00
slaweqnjohnston: thx16:00
slaweq:)16:00
bcafareloh true :)16:00
*** e0ne_ has quit IRC16:03
*** e0ne has joined #openstack-meeting-316:03
*** e0ne_ has joined #openstack-meeting-316:08
*** e0ne has quit IRC16:08
*** liuyulong has quit IRC16:20
*** e0ne_ has quit IRC16:45
*** links has quit IRC16:55
*** e0ne has joined #openstack-meeting-316:56
*** ricolin has quit IRC16:58
*** bbowen has quit IRC17:24
*** ianychoi has joined #openstack-meeting-317:27
*** ralonsoh has quit IRC18:09
*** jamesmcarthur has quit IRC18:17
*** jamesmcarthur has joined #openstack-meeting-318:18
*** e0ne has quit IRC18:18
*** jamesmcarthur has quit IRC18:19
*** jamesmcarthur has joined #openstack-meeting-318:19
*** jamesmcarthur has quit IRC18:43
*** jamesmcarthur has joined #openstack-meeting-318:54
*** bbowen has joined #openstack-meeting-318:54
*** jamesmcarthur has quit IRC18:55
*** jamesmcarthur has joined #openstack-meeting-318:55
*** jamesmcarthur has quit IRC18:59
*** jamesmcarthur has joined #openstack-meeting-319:06
*** jamesmcarthur has quit IRC19:15
*** jamesmcarthur has joined #openstack-meeting-319:15
*** jamesmcarthur has quit IRC19:33
*** jamesmcarthur has joined #openstack-meeting-319:35
*** apetrich has quit IRC19:38
*** e0ne has joined #openstack-meeting-319:38
*** jamesmcarthur has quit IRC19:53
*** jamesmcarthur has joined #openstack-meeting-319:56
*** jamesmcarthur has quit IRC20:11
*** jamesmcarthur has joined #openstack-meeting-320:16
*** jamesmcarthur has quit IRC20:20
*** jamesmcarthur has joined #openstack-meeting-320:21
*** jamesmcarthur has quit IRC20:28
*** jamesmcarthur has joined #openstack-meeting-320:29
*** jamesmcarthur has quit IRC20:35
*** belmoreira has quit IRC20:36
*** jamesmcarthur has joined #openstack-meeting-320:49
*** e0ne has quit IRC20:55
*** raildo has quit IRC21:06
*** jamesmcarthur_ has joined #openstack-meeting-321:37
*** jamesmcarthur has quit IRC21:41
*** jamesmcarthur_ has quit IRC21:48
*** slaweq has quit IRC21:50
*** slaweq has joined #openstack-meeting-321:51
*** apetrich has joined #openstack-meeting-321:53
*** jamesmcarthur has joined #openstack-meeting-322:03
*** bbowen has quit IRC22:14
*** bbowen has joined #openstack-meeting-322:15
*** jamesmcarthur has quit IRC23:50
*** jamesmcarthur has joined #openstack-meeting-323:54

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!