15:00:33 <mlavalle> #startmeeting neutron_l3
15:00:35 <openstack> Meeting started Thu Jun 29 15:00:33 2017 UTC and is due to finish in 60 minutes.  The chair is mlavalle. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:36 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:38 <openstack> The meeting name has been set to 'neutron_l3'
15:00:42 <Swami> hi
15:00:47 <mlavalle> hi
15:01:23 <mlavalle> haleyb|afk won't be attending today's meeting
15:01:54 <mlavalle> #topic Announcements
15:02:30 <mlavalle> The only one I have is to remind folks that we are a little less than a month from Pike-3
15:02:38 <Swami> mlavalle: I think you tried to ping couple of times, but I couldn't sync.
15:02:52 <mlavalle> #link https://releases.openstack.org/pike/schedule.html
15:03:24 <mlavalle> Swami: yeah, I just want to make sure you are ok with his patchset: https://review.openstack.org/#/c/474467/
15:03:50 <mlavalle> Last time we touched that inheritance relationship without your consent, we broke DVR ;-)
15:04:36 <mlavalle> any other announcements from the team?
15:04:51 <Swami> mlavalle: ok will check it out today.
15:04:57 <mlavalle> thanks!
15:05:32 <mlavalle> ok, moving on
15:05:39 <mlavalle> #topic Bugs
15:06:03 <mlavalle> Swami: stage is all yours, fire away please
15:06:08 <Swami> mlavalle: thanks
15:06:19 <Swami> this week I have 1 new bug related to DVR.
15:06:52 <Swami> #link https://bugs.launchpad.net/neutron/+bug/1701288
15:06:53 <openstack> Launchpad bug 1701288 in neutron "In scale testing RPC timeout error seen in the ovs_neutron_agent when update_device_list is called with DVR routers" [Undecided,New]
15:07:08 <Swami> mlavalle: I think I briefly discussed about this bug last week.
15:07:18 <mlavalle> I think you did
15:07:23 <Swami> I have more information on this bug after triaging more.
15:07:31 <mlavalle> thanks!
15:07:47 <Swami> It seems that l2pop _create_agent_fdb is taking a lot more time in large scale.
15:08:32 <Swami> Each DB transaction is taking around 200-300ms.
15:09:38 <Swami> So the _create_agent_fdb is taking a long time and rpc timesout.
15:09:54 <mlavalle> makes sense
15:10:16 <Swami> mlavalle: Is there a way that port-update can return and port_update_postcommit actions can run in a separate thread or in parallel.
15:10:54 <mlavalle> No that I know of
15:11:03 <Swami> I see that the update_port_postcommit is blocking and that's the reason for the delay. I am not a DB expert and I need some guidance here.
15:11:35 <Swami> Probably kevinbenton might have some idea.
15:11:35 <mlavalle> I cannot answer that from the top of my head. I will take a look at the code and the bug
15:11:49 <Swami> mlavalle: thanks
15:12:33 <mlavalle> out of curiosity, do you have access to an environment where you can run 3000 vms?
15:13:11 <Swami> mlavalle: I have it in our internal datacenter not external.
15:13:25 <mlavalle> cool!
15:13:37 <Swami> mlavalle: I kind of checkin in the test system when they see such problems.
15:14:05 <Swami> mlavalle: But this issue can be reproduced in the devstack and we can see the time it takes to finish the _create_agent_fdb entries.
15:14:16 <mlavalle> ah, ok
15:14:19 <Swami> mlavalle: I was able to debug it through devstack.
15:14:33 <Swami> Also this might be also associated with this bug.
15:14:37 <Swami> #link https://bugs.launchpad.net/neutron/+bug/1526855
15:14:38 <openstack> Launchpad bug 1526855 in neutron "VMs fail to get metadata in large scale environments" [Medium,Confirmed] - Assigned to Swaminathan Vasudevan (swaminathan-vasudevan)
15:15:49 <Swami> That's all I had for bugs.
15:16:06 <Swami> I need some review on the floatingip centralized patch.
15:16:09 <Swami> #link https://review.openstack.org/#/c/466434/
15:16:28 <Swami> #link https://review.openstack.org/#/c/437986/
15:17:32 <mlavalle> will take a look
15:17:42 <Swami> mlavalle: thanks
15:17:48 <Swami> mlavalle: back to you
15:17:52 <mlavalle> added to my pile
15:17:59 <mlavalle> thanks Swami!
15:18:03 <mlavalle> for the update
15:18:45 <mlavalle> First bug I have today is https://bugs.launchpad.net/neutron/+bug/1655567
15:18:46 <openstack> Launchpad bug 1655567 in neutron "DHCPv6 failures with "address already allocated in subnet" error" [High,Fix released] - Assigned to Miguel Lavalle (minsel)
15:19:02 <mlavalle> We merged the fix for this one about a week ago
15:19:27 <mlavalle> so this morning I spent some time trying to find hits in Kibana
15:19:44 <Swami> mlavalle: are you seeing it again.
15:19:48 <mlavalle> I found some, but a couple of them happend in Ocata
15:20:13 <mlavalle> and the other are related to a buggy patchset
15:20:26 <mlavalle> so I am claiming victory on this one
15:20:48 <Swami> mlavalle: great.
15:20:51 <mlavalle> and removing it from our list in the etherpad
15:21:06 <mlavalle> done ;-)
15:21:33 <mlavalle> Next one is https://bugs.launchpad.net/neutron/+bug/1683227
15:21:34 <openstack> Launchpad bug 1683227 in neutron "test_connection_from_same_address_scope failed with: Cannot find device "qg-74372988-7f"" [High,Incomplete] - Assigned to Miguel Lavalle (minsel)
15:22:01 <mlavalle> I also spent time trying to find hits for this one in Kibana
15:22:10 <mlavalle> Found none over tha past 7 days
15:22:37 <Swami> mlavalle: is this only happening with cvr routers are also with DVR routers, just for curiosity.
15:22:39 <mlavalle> so I decided to port my Kibana query in the bug for everybody to see and then proceeded to mark it invalid
15:22:54 <mlavalle> it's just not happening
15:23:20 <mlavalle> it might have been a one of when Ihar filed the bug
15:23:42 <Swami> mlavalle: ok
15:24:22 <mlavalle> Next one is https://bugs.launchpad.net/neutron/+bug/1694764
15:24:23 <openstack> Launchpad bug 1694764 in neutron "test_metadata_proxy_respawned failed to spawn metadata proxy" [High,Confirmed] - Assigned to Miguel Lavalle (minsel)
15:24:52 <mlavalle> I just assigned it to me and will proceed to triage / debug it later today
15:25:38 <mlavalle> Those are all the bugs I had for today
15:25:45 <Swami> mlavalle: ok
15:25:49 <mlavalle> Any bugs that should be discussed today?
15:26:14 <Swami> mlavalle: I think I have discussed all in my list.
15:26:27 <mlavalle> Perfect, let's move on
15:26:33 <mlavalle> #topic DNS
15:26:59 <mlavalle> I have finished all the codes and related tests for adding dns_domain functionality to ports
15:27:09 <mlavalle> I have 4 patchsets ready for review:
15:27:32 <mlavalle> #link https://review.openstack.org/#/c/459891/
15:27:50 <mlavalle> #link https://review.openstack.org/#/c/457101/
15:28:08 <mlavalle> #link https://review.openstack.org/#/c/457035
15:28:23 <mlavalle> #link https://review.openstack.org/#/c/468697
15:28:42 <mlavalle> That's the update for this week
15:28:52 <mlavalle> #topic Routed Networks
15:29:08 <Swami> mlavalle: thanks will review it.
15:29:29 <mlavalle> Since I have finished DNS, next I am going to focus on adding floating ips to routed networks
15:29:40 <mlavalle> I will be working over the next few days on the spec
15:29:44 <mlavalle> will push it soon
15:30:21 <mlavalle> and that's the update for this week on this subject
15:30:32 <mlavalle> #topic Open Agenda
15:30:35 <Swami> mlavalle: ok thanks.
15:30:47 <mlavalle> Any other topics we should discuss today?
15:30:58 <Swami> mlavalle: do you have any plans for paper presentation in sydney.
15:31:18 <mlavalle> no, but I would love to partner with you
15:31:24 <Swami> mlavalle: any topics of interest.
15:31:44 <mlavalle> mhhh let's think about it over the next few days
15:32:00 <mlavalle> we still have some time
15:32:03 <Swami> mlavalle: if you have any topic of interest let me know and I can right up an abstract. If anything comes in my mind I will let you know.
15:32:13 <Swami> mlavalle: yes it is July 14th.
15:32:14 <mlavalle> great!\
15:32:39 <davidsha> I have a question in relation to the patch I submitted, I noticed that the q-svc seems to be causing the q-l2 to constantly add the internal router ports constantly. have either of you seen this before?
15:32:53 <davidsha> q-l2/q-agt -_-
15:33:16 <mlavalle> no
15:33:18 <Swami> davidsha: can you point us to the bug or the logs that you are seeing.
15:34:03 <davidsha> Swami: They're local, I didn't spot them in the init tests for the patch. I can make a copy and put it in pastebin if thats ok?
15:34:19 <mlavalle> that would work
15:34:24 <Swami> davidsha: yes that would work.
15:34:49 <mlavalle> first impression is that it sounds like a bug. The port should be added once
15:35:14 <Swami> mlavalle: I agree
15:36:43 <davidsha> kk, sorry. trying to figure out where it starts and ends, I'll grab the q-agt first
15:37:10 <mlavalle> davidsha: thanks
15:37:37 <mlavalle> ok, if there are no other topics, lets end the meeting
15:37:46 <mlavalle> Swami: enjoy the long weekend
15:37:47 <Swami> mlavalle: thanks
15:37:55 <Swami> mlavalle: you too.
15:38:00 <Swami> mlavalle: bye
15:38:14 <mlavalle> I found out yesterday that Monday is also holiday for IBM Austin
15:38:30 <mlavalle> so 4 days weekend .... yaay!
15:38:33 <Swami> mlavalle: we don't have leave on Monday, but I am planning to take off on Monday.
15:38:41 <mlavalle> enjoy!
15:38:48 <Swami> mlavalle: Happy 4th of July
15:38:48 <mlavalle> davidsha: thanks for attending
15:39:05 <mlavalle> and sorry for the parochial conversation at the end
15:39:16 <mlavalle> we don't get many holidays in this country ;-)
15:39:36 <mlavalle> #endmeeting