14:00:59 #startmeeting neutron_l3 14:01:00 Meeting started Wed Jul 31 14:00:59 2019 UTC and is due to finish in 60 minutes. The chair is liuyulong. Information about MeetBot at http://wiki.debian.org/MeetBot. 14:01:01 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 14:01:03 The meeting name has been set to 'neutron_l3' 14:01:23 Hello 14:02:33 hi 14:02:41 Today will be a quick short meeting, IMO. 14:02:44 o/ 14:02:53 #topic Announcements 14:03:20 hi 14:03:21 #link https://etherpad.openstack.org/p/Shanghai-Neutron-Planning 14:03:31 Just added my name here ^ 14:03:50 great, we will finally meet in person liuyulong :) 14:03:51 #chair haleyb 14:03:52 Current chairs: haleyb liuyulong 14:05:18 I have no more announcements. IMO, we have reminded all of them yesterday in team meeting. 14:06:04 OK, let's move on. 14:06:13 #topic Bugs 14:06:24 #link http://lists.openstack.org/pipermail/openstack-discuss/2019-July/008089.html 14:06:30 Boden Russell was our bug deputy last week, thanks. 14:07:24 And again, I will skip all the bugs which were fixed or the related patches are getting merged now. 14:07:37 First one: 14:07:39 #link https://bugs.launchpad.net/neutron/+bug/1837635 14:07:40 Launchpad bug 1837635 in neutron "HA router state change from "standby" to "master" should be delayed" [Undecided,In progress] - Assigned to Rodolfo Alonso (rodolfo-alonso-hernandez) 14:08:16 For the fix, IMO, it looks good to me, but I'd like to see more deep test results. 14:08:42 I just added two scenarios here: https://review.opendev.org/#/c/672533/ 14:10:06 Sometimes, the actual results of the running program may be different from what you expected. 14:10:32 Next: 14:10:37 #link https://bugs.launchpad.net/neutron/+bug/1834308 14:10:39 Launchpad bug 1834308 in neutron "[DVR][DB] too many slow query during agent restart" [Medium,In progress] - Assigned to LIU Yulong (dragon889) 14:11:06 I submitted the patch set yesterday. 14:11:16 It is here: https://review.opendev.org/#/c/673557/ 14:11:30 A pep8 failure... 14:12:49 All the DB query in this patch has the highest frequency of call when restart ovs-agent. 14:13:58 do you have numbers on the improvement? 14:14:00 And it is time-consuming, when your 'ports' table is getting larger and lager. These query will have a worse results. 14:14:52 40 nodes of ovs-agent restart will call these DB query about 300K+ times. 14:17:02 And these query costs about 0.1s+ seconds logged by our mariadb cluster. 14:17:28 .1s per query? 14:18:18 Yes, one of them is about 0.5s+. Let me link it in the gerrit. 14:18:50 https://review.opendev.org/#/c/673557/1/neutron/db/dvr_mac_db.py@145 14:18:56 get_ports_on_host_by_subnet 14:19:02 This one. 14:20:01 haleyb, the results is when the ports table has about 10-20K records. 14:23:09 so _get_ports_query() is really slow 14:25:05 The scale of resource is about: 17000+ VMs, 3000+ DVR routers, 3000+ network, 3000+ subnets and 3000+ security groups; 40 security group rules for each security group. 14:26:08 After this change, the ovs-agent restart time has a very significant improvement, it's about 40-50mins to 15mins. 14:26:29 I wonder if it would be further optimized by adding an index specifically on the Port.device_owner field. I'll comment on the change. 14:26:50 interesting 14:27:08 hi, btw 14:27:22 liuyulong: that's quite an improvement, even if 15mins is still a long time :) 14:27:44 40 - 50 mins, I can't believe it once, but indeed it is. 14:28:23 rpc_loop 1 it will scan the ports and process it. 40-50mins....... 14:30:24 liuyulong: it looks like you have lots of reviewers now 14:31:03 More detail about our test deployment is: 3 neutron-server with about (172 workers), its 3 nodes DB and 3 nodes MQ, are all in dedicated server. 14:31:10 Yes, neutron has its own DB and MQ. 14:35:30 Last one: 14:35:38 #link https://bugs.launchpad.net/neutron/+bug/1838431 14:35:39 Launchpad bug 1838431 in neutron "[scale issue] ovs-agent port processing time increases linearly and eventually timeouts" [Undecided,New] 14:35:51 More like a L2 issue... 14:37:05 liuyulong: this one looks like related to already known problem with "remote_security_group" rules in SG 14:37:19 there was bug reported for it already IIRC 14:37:19 The test have not get successfully yet. 14:38:12 I will test it again today. 14:40:19 One more interesting thing is that we disable the DHCP for this test. No DHCP agent in this test. I can image if DHCP is enabled the vif-plug-timeout may get more... 14:41:13 That's all bugs from me. 14:41:24 any other bugs that need the team to pay attention? 14:41:37 there was one miguel filed yesterday 14:41:39 liuyulong: I can't find any bug reported for Your last issue but please check https://etherpad.openstack.org/p/openstack-networking-train-ptg in L347 14:41:48 njohnston: raised this problem on last PTG 14:41:58 slaweq, OK, great 14:42:00 https://bugs.launchpad.net/neutron/+bug/1838449 14:42:01 Launchpad bug 1838449 in neutron "Router migrations failing in the gate" [Medium,Confirmed] - Assigned to Miguel Lavalle (minsel) 14:42:23 liuyulong: please do Your test without security group rules which reference to remote_group_id 14:42:32 than it should be much, much faster 14:44:20 liuyulong: that was the only other bug i had, was going to try and reproduce locally today for miguel 14:45:06 haleyb: yes, this one hurts us quite lot in CI jobs 14:45:24 slaweq, actually I refactor may test to 27 tenants yesterday. It looks better now. 14:46:09 haleyb, thanks for bring up this, seems Miguel has found the issue code. 14:46:12 liuyulong: because if You have more tenants, there is less IPs (ports) using same security group probably and thus it's faster 14:47:16 slaweq, yes, and I'm trying to add more security group for each tenant, or network. 14:48:19 For one tenant and one default security group, it is a disaster. 14:49:12 IMO, every one try to test this will be very easy to encounter this problem. 14:49:41 njohnston's PTG summary looks very similar to this. 14:49:50 liuyulong: yes, we had this issue too 14:51:12 And maybe some security group DB query also need some optimizing work. 14:51:28 OK, next topic 14:51:36 #topic Routed Networks 14:51:44 mlavalle, tidwellr, wwriverrat: your turn now. 14:53:30 if mlavalle and wwriverrat don't have anything, we can talk briefly floating IP's for routed networks 14:53:55 https://review.opendev.org/#/c/486450/ and the POC code https://review.opendev.org/#/c/669395/ 14:55:04 I don't see mlavalle online 14:55:36 if it isn't obvious by my nagging folks to take a look at these, this has turned into my pet project :) 14:57:50 I've spun up a little lab where I've tested the POC code, it seems to work nicely and it's not terribly invasive. What I'm interested in is feedback about the approach in the spec 15:00:05 tidwellr, thank you for replying to my question in the patch sets. 15:00:12 OK, let's end the meeting. 15:00:17 #endmeeting