15:00:37 #startmeeting neutron_l3 15:00:43 Meeting started Thu Oct 18 15:00:37 2018 UTC and is due to finish in 60 minutes. The chair is mlavalle. Information about MeetBot at http://wiki.debian.org/MeetBot. 15:00:44 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 15:00:46 The meeting name has been set to 'neutron_l3' 15:00:49 o/ 15:00:54 o/ 15:00:55 hi 15:00:59 o/ 15:01:07 haleyb: go Sox. almost there 15:01:25 even the bad calls have been going our way :) 15:01:44 * haleyb is very tired this morning 15:01:50 is next game at Fainway? 15:01:58 no, tonight in houston 15:02:05 good luck 15:02:06 hi 15:02:29 #topic Announcements 15:02:47 Just a reminder that next week we reach Stein-1 15:02:59 o/ 15:03:26 and the Summit in Berlin is three weeks away, November 13 - 15 15:03:38 any other annoucements? 15:04:15 ok, let's move on 15:04:24 #topic Bugs 15:04:47 Swami: fire away 15:04:50 mlavalle: thanks, will do 15:05:11 #link https://bugs.launchpad.net/neutron/+bug/1794991 15:05:12 Launchpad bug 1794991 in neutron "Inconsistent flows with DVR l2pop VxLAN on br-tun" [Undecided,New] 15:05:43 I was trying to reproduce this pike, but unfortunately I am not successful in reproducing this bug. 15:06:03 We have also internally seen such problems in newton. 15:06:41 Has anyone else seen similar problems mentioned in this bug, where there is a race between the vxlan tunnel creation and the l2pop adding the flows for the vxlan. 15:06:54 no I haven't 15:07:28 so, according to the bug report, what is the effect when it happens? 15:07:29 mlavalle: OK I will try couple of more tests to see if I can reproduce this. There is definitely a race. 15:08:02 The reason I ask is that we have other bugs related to DVR 15:08:05 When it happens the new VMs that come in on that compute cannot reach the DHCP server on the network node and also VM to VM communication will be blocked. 15:08:39 mlavalle: So just to refresh, the vxlan tunnels are created when the agent first registers with the neutron-server right? 15:09:04 Is there a possibility that the vxlan tunnel interfaces might change down the line. 15:09:05 yes, that should be it 15:09:37 and no, I don't think so 15:09:51 The reason I am asking is that I am debugging a couple of DVR related bugs 15:10:12 mlavalle: sure 15:10:14 I will keep this in mind as a possible cause for the bugs I am working on 15:10:24 If we see this problem, it should be in the gate 15:10:35 where we have concurrency 15:10:39 mlavalle: Yes basically you should have all the flows in place for the communication to be error free. 15:10:47 mlavalle: ok thanks 15:11:04 many of these races happen only with many concurrrent thinks happening 15:12:03 mlavalle:yes that's why reproducing it is a nightmare, unless we have the hardware resources and a script to trigger it. 15:12:09 #link https://bugs.launchpad.net/neutron/+bug/1793529 15:12:09 Launchpad bug 1793529 in neutron "[dvr][ha][dataplane down] router_gateway port binding host goes wrong after the 'master' host down/up" [Undecided,In progress] - Assigned to LIU Yulong (dragon889) 15:12:28 now, if the it hist DHCP, it seems to me that the problem goes beyond DVR, right? 15:12:31 There is a patch up for review. #link https://review.openstack.org/#/c/606384/ 15:13:42 mlavalle: no the reason is there should be a flow rule in there with the MAC address of the dhcp server pointing to the right vxlan tunnel. Otherwise the dhcp request is not sent 15:14:03 ah, that's right 15:14:03 ok? 15:14:04 ok 15:14:42 mlavalle: It is directly a problem with DHCP, but somehow when the l2pop populates the flows, it is either not populated properly when the VM on that network pops up or it is getting removed on restarts etc., 15:15:00 ack 15:15:02 s/it is directly/it is not directly 15:15:06 mlavalle: thanks 15:15:21 If you have not review this, please review this patch. #link https://review.openstack.org/#/c/606384/ 15:15:53 The next one in the list is 15:16:31 #link https://bugs.launchpad.net/neutron/+bug/1796491 15:16:31 Launchpad bug 1796491 in neutron "DVR Floating IP setup in the SNAT namespace of the network node and also in the qrouter namespace in the compute node" [Medium,In progress] - Assigned to Swaminathan Vasudevan (swaminathan-vasudevan) 15:16:59 There is a patch up for review. #link https://review.openstack.org/609924 15:17:20 haleyb: liuyulong: thanks for your reviews on the patch. 15:17:47 haleyb: I will address your comment, I think it is a minor one. So I hope this patch is in a good condition. 15:17:56 thanks 15:18:44 The next one is 15:18:49 #link https://bugs.launchpad.net/neutron/+bug/1796824 15:18:49 Launchpad bug 1796824 in neutron "Port in some type of device_owner should not allow update IP address" [Medium,In progress] - Assigned to LIU Yulong (dragon889) 15:19:11 There is a patch up for review #link https://review.openstack.org/#/c/608909/ 15:19:22 If not review please review it. 15:20:04 I think this bug #link https://bugs.launchpad.net/neutron/+bug/1785227 is a duplicate of the above one. So I am going to mark it as Duplicate. 15:20:04 Launchpad bug 1785227 in neutron "Router port: no dataplane update on change" [Medium,Confirmed] 15:20:13 Does anyone have any concerns on this. 15:20:37 go ahead 15:20:50 mlavalle: ok 15:21:11 #link https://bugs.launchpad.net/neutron/+bug/1797037 15:21:11 Launchpad bug 1797037 in neutron "Extra routes configured on routers are not set in the router namespace and snat namespace with DVR-HA routers" [Medium,In progress] - Assigned to Brian Haley (brian-haley) 15:21:53 #link https://review.openstack.org/609273 - This patch fails zuul. 15:22:30 It seems to be happy now, but let us see if it will merge. 15:22:56 #link https://bugs.launchpad.net/neutron/+bug/1786272 15:22:56 Launchpad bug 1786272 in neutron "Connection between two virtual routers does not work with DVR" [Medium,In progress] - Assigned to Slawek Kaplonski (slaweq) 15:23:15 I just pushed new PS for this one: https://review.openstack.org/#/c/597567/27 15:23:21 #link https://review.openstack.org/#/c/597567/ 15:23:50 liuyulong is doing great review and testing manually different scenarios on this patch 15:23:55 slaweq: yes just noticed. I will review it. I hope everything is fine with this patch. But liuyulong had some questions still. 15:23:59 I hope that now it will be fine 15:24:10 slaweq: Ok thanks. 15:24:31 Swami: if You can, please try it also on Your env - it's really tricky patch IMO and I don't want to break something :) 15:24:37 I'm testing this locally too. 15:24:47 slaweq: Yes I will do one more round of testing today. 15:24:48 You have much more experience with dvr than me 15:24:55 thx liuyulong and Swami 15:25:13 slaweq:I also wanted to test the shared-network scenario with this patch between two tenants. Let me do it. 15:25:29 Swami: ok 15:25:31 thx 15:25:44 thanks everyone, at PS27 this has taken a long time but it's good work :) 15:26:04 haleyb: Yep it is a complex one. 15:26:20 yes, it is 15:26:24 The next one is 15:26:29 #link https://bugs.launchpad.net/neutron/+bug/1774459 15:26:29 Launchpad bug 1774459 in neutron "Update permanent ARP entries for allowed_address_pair IPs in DVR Routers" [High,Confirmed] - Assigned to Swaminathan Vasudevan (swaminathan-vasudevan) 15:27:17 mlavalle: haleyb: can you take a look at this patch . #link https://review.openstack.org/601336 15:27:49 Swami: yes, I promised last week to take a look and couldn't find the time. sorry 15:27:50 This is just a WIP based on our conversation at the PTG, but I am waiting for your inputs on this. I have a question. I also mentioned it in the last meeting. 15:27:55 I'll try again this week 15:27:59 mlavalle: thanks 15:28:18 i'll try to look by tomorrow too 15:28:28 mlavalle: haleyb : thank you 15:28:36 mlavalle: that's all I have for today. 15:28:41 mlavalle: Back to you. 15:29:01 I want to mention two bugs 15:29:19 First one is https://bugs.launchpad.net/neutron/+bug/1795870 15:29:19 Launchpad bug 1795870 in neutron "Trunk scenario test test_trunk_subport_lifecycle fails from time to time" [High,Confirmed] - Assigned to Miguel Lavalle (minsel) 15:29:36 it was reported as a trunk failure (it is a trunk test failure) 15:30:10 however, based on the debugging I've done so far, it seems to be a DVR problem 15:30:22 I left debugging notes a couple of days ago 15:30:37 so for the time being I'm adding it to the DVR backlog 15:30:44 and I'l, continue debugging 15:31:23 mlavalle: thanks 15:31:41 the problem really occurs when trying to ssh to a test instance using a floating ip 15:33:05 the second bug is https://bugs.launchpad.net/neutron/+bug/1787919 15:33:05 Launchpad bug 1787919 in neutron "Upgrade router to L3 HA broke IPv6" [High,Confirmed] - Assigned to Miguel Lavalle (minsel) 15:33:25 At this point I have a deployment reproducing the conditions and am testing 15:33:41 I'll keep pushing forward with this one as well 15:34:16 finally, we have https://bugs.launchpad.net/neutron/+bug/1789434 15:34:16 Launchpad bug 1789434 in neutron "neutron_tempest_plugin.scenario.test_migration.NetworkMigrationFromHA failing 100% times" [High,Confirmed] - Assigned to Manjeet Singh Bhatia (manjeet-s-bhatia) 15:34:27 https://review.openstack.org/#/c/611461/ 15:34:57 I posted a solution ideally notification part should be in l3_hamodedb code 15:35:25 Please reviews it if solution is okay I’ll update and add tests 15:36:18 manjeets_: Yes i did a first pass on it yesterday 15:36:36 Thanks swami totally agree to your comments 15:37:26 I’ll revise it today 15:38:18 manjeets_: thanks 15:38:41 manjeets_: I will also look at it soon 15:40:50 any other bugs we should discuss today? 15:42:01 ok, moving on 15:42:19 #topic On demand agenda 15:43:01 mlavalle, Thank you for looking at the agent refactor, We're working through it atm to make sure the tests are right. 15:43:19 davidsha_: last evening I left Xubo some guidnace as to how to debug the failures 15:43:29 It's mostly mocks pointing to the wrong objects atm. 15:43:50 yeah, I know. I just didn't want to fix it for him 15:44:04 I wanted to have him work through the process 15:44:28 it's part of my job to help new devs 15:45:24 kk, We're having calls as well so I'll try to keep things moving! 15:45:56 tell him that if he has questions or needs guidance, he can leave a comment in the patch and I'll follow up 15:46:37 any other topics to discuss? 15:46:58 ok, thanks for attending 15:47:06 cool, think he's online if you have any questions 15:47:08 thanks! 15:47:11 have a great weekend! 15:47:33 haleyb: go finish those Astros off tonight! 15:47:34 #endmeeting