15:00:56 <carl_baldwin> #startmeeting neutron_l3
15:00:56 <openstack> Meeting started Thu Oct 22 15:00:56 2015 UTC and is due to finish in 60 minutes.  The chair is carl_baldwin. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:57 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:57 <obondarev> hi
15:00:59 <openstack> The meeting name has been set to 'neutron_l3'
15:01:13 <carl_baldwin> #chair mlavalle
15:01:14 <openstack> Current chairs: carl_baldwin mlavalle
15:01:24 * carl_baldwin trying to implement HA for the L3 meeting.
15:01:36 <carl_baldwin> #topic Announcements
15:01:45 <carl_baldwin> #link https://wiki.openstack.org/wiki/Meetings/Neutron-L3-Subteam
15:02:08 * regXboi slips into the back of the room
15:02:16 <Swami> hi
15:02:19 <carl_baldwin> If you don’t know that summit is next week then you must be new.  To that, I say “welcome to the L3 team meeting!”
15:03:08 <carl_baldwin> Given that summit is next week and we will all be engaged in exciting, productive and fulfulling conversation, we will not hold this IRC meeting next week.
15:03:46 <fitoduarte> hi
15:03:50 <mlavalle> do we resume the week after summit?
15:04:19 <carl_baldwin> mlavalle: good question, we will resume the week after summit.  So, two weeks from now.
15:04:28 <john-davidge> hi all
15:04:31 <carl_baldwin> Any other announcements?
15:05:09 * carl_baldwin hands out tardy slips.  Especially to regXboi who was trying to be sneaky in the back of the room.
15:05:32 * regXboi takes the slip and puts it in the "firestarter" pile :)
15:05:42 <carl_baldwin> #topic Bugs
15:05:44 <mlavalle> regXboi: lol
15:05:58 <mlavalle> ok, trying to do this quickly
15:06:04 * john-davidge has to try to get internal meetings to stop clashing with IRC
15:06:09 <carl_baldwin> I figured we’d go through bugs today and then go straight to an on-demand agenda.
15:06:29 <carl_baldwin> If we’re lucky, we can have some time back to prepare for summit travel.
15:06:53 <regXboi> oh boy
15:07:05 <regXboi> who did we lose?
15:07:44 <regXboi> so the ones I care about are the ones that aren't in progress *yet*
15:07:49 <mlavalle> first up is https://bugs.launchpad.net/neutron/+bug/1365473
15:07:49 <openstack> Launchpad bug 1365473 in neutron "Unable to create a router that's both HA and distributed" [High,In progress] - Assigned to Adolfo Duarte (adolfo-duarte)
15:08:05 <mlavalle> fix is progressing https://review.openstack.org/#/c/143169/
15:08:07 <jschwarz> That patch is mostly waiting for reviews :)
15:08:07 * carl_baldwin still here.
15:08:21 <mlavalle> last revision was on 10/20
15:08:27 <regXboi> yes, can we get some review love on that patch?
15:08:28 <mlavalle> and yes, waiting for reviews
15:08:41 <fitoduarte> yes. it keeps going into merge conflict
15:09:25 <regXboi> carl_baldwin: can we get a couple of cores to look at that patch?
15:09:40 <carl_baldwin> jschwarz: I started a review.  Will finish today.
15:09:48 <jschwarz> carl_baldwin, excellent, thanks a lot :)
15:09:54 <jschwarz> I'm sure fitoduarte will like it as well
15:10:00 <carl_baldwin> haleyb is travelling but he may see an email when he arrives.
15:10:05 <fitoduarte> tx
15:10:30 <carl_baldwin> fitoduarte: Did you change nicks or am I imagining it?
15:11:10 <fitoduarte> yes. forgot to log off my laptop
15:11:26 <mlavalle> ok next one up is https://bugs.launchpad.net/neutron/+bug/1494351
15:11:26 <openstack> Launchpad bug 1494351 in neutron "Observed StaleDataError in gate-neutron-dsvm-api tests if reference IPAM driver is used" [High,In progress] - Assigned to Pavel Bondar (pasha117)
15:11:52 <pavel_bondar> two patches are ready for review
15:11:53 <mlavalle> there are are two patchsets awating reviews. here's the gerrit topic https://review.openstack.org/#/q/status:open+project:openstack/neutron+branch:master+topic:bug/1494351,n,z
15:11:59 <pavel_bondar> #link https://review.openstack.org/#/c/237677/
15:12:11 <pavel_bondar> and #link https://review.openstack.org/#/c/223123
15:12:46 <pavel_bondar> jenkins passed, so hope to get some feedback on them
15:12:53 <mlavalle> pavel_bondar: they are just needing review, right?
15:13:05 <pavel_bondar> mlavalle: right
15:13:23 <mlavalle> any other comments?
15:13:35 <pavel_bondar> no
15:13:42 <mlavalle> ok, moving on
15:13:50 <mlavalle> https://bugs.launchpad.net/neutron/+bug/1486795
15:13:50 <openstack> Launchpad bug 1486795 in neutron "DVR: create or update port by using notify specific host rather than fanout" [High,In progress] - Assigned to Oleg Bondarev (obondarev)
15:14:17 <mlavalle> there is some confusion as to the correct fix for this one. obondarev do you care to comment?
15:14:27 <obondarev> yep
15:14:35 <carl_baldwin> pavel_bondar: I’ll look.
15:14:43 <obondarev> seems authors of alternative patches are oke with https://review.openstack.org/#/c/231555/
15:14:50 <pavel_bondar> carl_baldwin: thanks
15:14:57 <obondarev> so I updated it today
15:15:12 <mlavalle> obondarev: yeah, they seem to be ok with this fix
15:15:12 <carl_baldwin> obondarev: great, thanks for syncing that up.
15:15:18 <regXboi> have the other patches been abandoned?
15:15:24 <obondarev> not yet
15:15:37 <mlavalle> obondarev: I even asked them to abandon their patchsets yesterday, to avoid confusion
15:15:45 <obondarev> mlavalle: saw that, thanks
15:15:47 <regXboi> ok, can we get that done asap and update LP?
15:15:58 <carl_baldwin> I can abandon them today with a note that we discussed in this meeting.
15:16:08 <regXboi> carl_baldwin: ack and thx
15:16:23 <regXboi> I'll cover LP if the abandonments don't automagically show up
15:17:01 <mlavalle> anything else on this one?
15:17:11 <regXboi> reviews on 231555 :)
15:17:37 <mlavalle> ok, next up is https://bugs.launchpad.net/neutron/+bug/1486828
15:17:37 <openstack> Launchpad bug 1486828 in neutron "DVR: Notify specific agent when dealing with floating ips" [High,In progress] - Assigned to Oleg Bondarev (obondarev)
15:18:01 <obondarev> ok, for this one
15:18:14 <obondarev> one of partial fixes broke Ironic gate jobs
15:18:20 <obondarev> and was reverted
15:18:22 <mlavalle> obondarev has a proposed fix: https://review.openstack.org/#/c/231455/
15:19:01 <mlavalle> revereted one is https://review.openstack.org/#/c/215136
15:19:11 <obondarev> then I uploaded revert of revert which is not breaking Ironic
15:19:24 <obondarev> and that was merged as well
15:19:44 <obondarev> so the final fix is  https://review.openstack.org/#/c/231455, needs reviews
15:20:03 <regXboi> obondarev: have you addressed the -1's on it ?
15:20:07 <carl_baldwin> obondarev: I’m a little confused.  What about this one:  https://review.openstack.org/#/c/237476/
15:20:22 <mlavalle> obondarev: so, 231455 is the final one?
15:20:30 <obondarev> carl_baldwin: https://review.openstack.org/#/c/237476/ is kind of revert of revert
15:20:41 <regXboi> carl_baldwin: that was the revert of the revert to avoid breaking ironic
15:20:50 <obondarev> regXboi: I've replied to the comments, yes
15:20:56 <regXboi> obondarev: ack - thx
15:21:23 <carl_baldwin> I guess I’m going patch blind (like snow blind) from seeing a lot of patches to notify specific agents.
15:21:35 <mlavalle> lol
15:21:41 <obondarev> :-)
15:21:47 * regXboi hears a fourplay song parody coming
15:22:14 <obondarev> carl_baldwin: yeah, we need to reduce the notification flood at scale
15:22:41 <mlavalle> obondarev: thanks for taking care of this
15:22:55 <mlavalle> anything else on this one?
15:22:56 <carl_baldwin> I’ll review https://review.openstack.org/#/c/231455/
15:23:13 <obondarev> mlavalle: carl_baldwin: thanks
15:23:40 <mlavalle> ok, next up is https://bugs.launchpad.net/neutron/+bug/1476097
15:23:40 <openstack> Launchpad bug 1476097 in neutron "[fwaas]Support fwaas to control east-west traffic in dvr router" [High,Triaged] - Assigned to lee jian (leejian0612)
15:23:56 <mlavalle> last status, Swami was defining a solution. any updates?
15:24:01 <Swami> This is still under discussion with the Fwaas team.
15:24:27 <Swami> mlavalle: not yet, will probably update the doc after the summit.
15:24:37 <mlavalle> ok, thanks!
15:24:40 <regXboi> wait
15:24:50 <mlavalle> next up is https://bugs.launchpad.net/neutron/+bug/1505575
15:24:50 <openstack> Launchpad bug 1505575 in neutron "Fatal memory consumption by neutron-server with DVR at scale" [High,In progress] - Assigned to Oleg Bondarev (obondarev)
15:24:54 <regXboi> mlavelle: hold on
15:24:59 <mlavalle> and our hero of the morning appears again, obondarev
15:25:01 <regXboi> I want a little more clarification on that last one
15:25:09 <mlavalle> ok
15:25:16 <mlavalle> holding
15:25:19 <regXboi> Swami: is the plan to do a spec along with or before the code?
15:26:04 <regXboi> because I'd sort of like to see the devref/spec to see the how before I see the code
15:26:08 <Swami> regXboi: right now it is not possible to propose a solution for this problem without a change in DVR functionality. So I would prefer that we propose a spec before modifying any DVR code in this respect if possible.
15:26:23 <regXboi> good - that's where I am as well
15:26:26 <regXboi> thx
15:26:40 <regXboi> thx for the hold mlavalle ... back to you
15:26:59 <mlavalle> ok, going back to https://bugs.launchpad.net/neutron/+bug/1505575, obondarev has the following proposed fix https://review.openstack.org/#/c/234067/
15:26:59 <openstack> Launchpad bug 1505575 in neutron "Fatal memory consumption by neutron-server with DVR at scale" [High,In progress] - Assigned to Oleg Bondarev (obondarev)
15:27:11 <obondarev> this one has a patch on review https://review.openstack.org/#/c/234067/
15:27:13 <mlavalle> the fix is active and getting reviews
15:27:22 <obondarev> but there are some concerns regarding the approach
15:27:29 <obondarev> the discussion is in PS1
15:27:38 <regXboi> can we go with a configuration option for now and get clever later?
15:27:49 <regXboi> I don't want to hold this patch up for bikeshedding
15:28:31 <obondarev> the concern might be if we introduce config option now it will be harder to remove it later
15:28:34 * carl_baldwin looks at patch
15:29:37 * carl_baldwin ’s memory now refreshed
15:29:44 <obondarev> so this might need more thinking to try it without config option
15:30:52 <regXboi> I'm thinking that even if we had some magic automated system for determining batch size, I (as an operator) may still want the knob to override it
15:31:44 <carl_baldwin> obondarev: I don’t have any good feedback for you right now but I will take time tody.
15:31:47 <carl_baldwin> *today
15:31:55 <obondarev> carl_baldwin: thanks
15:32:06 <obondarev> regXboi: please comment in the review ;)
15:32:17 <regXboi> obondarev: I will add that now :)
15:33:55 <regXboi> did we lose mlavalle there?
15:34:17 <carl_baldwin> Exactly why we need to run HA meetings.
15:34:26 <regXboi> is this an HA meeting?
15:34:39 * regXboi hopes it is
15:35:11 <carl_baldwin> regXboi: We are trying, but not quite there.  The blueprint for HA meeting has been approved.
15:35:19 <regXboi> nice
15:35:49 <mlavalle> hi
15:35:50 <regXboi> and he's back :)
15:35:57 <mlavalle> sorry, got disconnected
15:36:13 <mlavalle> had to reset router
15:36:16 <carl_baldwin> mlavalle: welcome back.  We were just discussing the need to make our meeting fully HA.
15:36:34 * regXboi proposes always having 2+ entries in #chair
15:36:36 <mlavalle> ;-(
15:37:09 <mlavalle> carl_baldwin: so where are we
15:37:13 <mlavalle> ?
15:37:19 <regXboi> mlavalle: we were finishing up with 1505575
15:37:29 <carl_baldwin> regXboi: You actually missed the beginning of the meeting where I did add cochair.  But, we still have some SPOFs.
15:37:46 <regXboi> carl_baldwin: cool
15:38:14 <carl_baldwin> regXboi: We’ll get better with experience.
15:38:15 <regXboi> mlavalle: any more items in the bug list?
15:38:59 <regXboi> (if not I have a few)
15:39:11 <mlavalle> regXboi: go ahead
15:39:25 <regXboi> I have three: bug 1462154
15:39:25 <openstack> bug 1462154 in neutron "With DVR Pings to floating IPs replied with fixed-ips" [High,In progress] https://launchpad.net/bugs/1462154 - Assigned to Stephen Ma (stephen-ma)
15:39:50 <mlavalle> ah ok, that is in the agenda
15:40:00 <mlavalle> regXboi: so I didn't really loose anything
15:40:19 <regXboi> the patch set is https://review.openstack.org/233334 - it is in merge conflict and carl_baldwin and I aren't comfortable with what it is proposing
15:40:52 <regXboi> folks, pls take a look and weigh in
15:41:22 <regXboi> second is bug 1504726
15:41:22 <openstack> bug 1504726 in neutron "The vm can not access the vip of load balancer under DVR enviroment" [High,New] https://launchpad.net/bugs/1504726 - Assigned to Swaminathan Vasudevan (swaminathan-vasudevan)
15:41:54 <Swami> regXboi: Yes this bug I am trying to reproduce it right now. I have some trouble in bringing up the Lbaas in devstack.
15:42:03 <regXboi> this is still being triaged, but I'm questioning its severity
15:42:12 <regXboi> Swami: I saw that
15:42:31 <Swami> regXboi: right now it is been tagged as High.
15:42:43 <regXboi> Swami: yes, I'm wondering if it should be less than high
15:42:46 <regXboi> but that's all
15:42:55 <Swami> regXboi: it was mentioned that this is seen in multinode only scenarios.
15:42:56 <regXboi> and that can wait for the triage being finished
15:43:06 <Swami> regXboi: for now let us leave it as such.
15:43:13 * carl_baldwin looking at severity
15:43:37 <regXboi> Last one is bug 1507602
15:43:37 <openstack> bug 1507602 in neutron "_get_router() sometimes raises RouterNotFound when called from under create_floatingip" [High,Confirmed] https://launchpad.net/bugs/1507602 - Assigned to Oleg Bondarev (obondarev)
15:43:47 <regXboi> which brings our hero, obondarev back to the table :)
15:44:08 <obondarev> so this one is yet to be investigated
15:44:15 <carl_baldwin> +1, like what obondarev is doing for us here
15:44:19 <mlavalle> the question I have here is: https://review.openstack.org/#/c/237476/ was merged is this the only fix needed?
15:44:34 <obondarev> mlavalle: not quite
15:44:51 <obondarev> we still need to know the reason for the race condition
15:44:58 <regXboi> exactly
15:45:05 <regXboi> all we are doing now is masking it
15:45:07 <obondarev> 237476 is kind of workaround
15:45:32 <obondarev> we're just preserving the original behavior
15:46:00 <mlavalle> obondarev: ok i'll add a note to the bug so we all know where we stand
15:46:08 <obondarev> I have an idea on this, will check it soon
15:46:14 <regXboi> agreed - I've been trying out the dvr multinode full job locally and am way down the rabbit hole trying to catch race conditions
15:46:27 <obondarev> mlavalle: thanks
15:47:23 <mlavalle> regXboi: you have one more bug in the agenda. de we want to discuss?
15:47:38 <regXboi> mlavalle: number?
15:47:56 <mlavalle> https://bugs.launchpad.net/neutron/+bug/1505571
15:47:56 <openstack> Launchpad bug 1505571 in neutron "VM delete operation fails with 'Connection to neutron failed - Read timeout' error" [Undecided,Incomplete] - Assigned to Sonu (sonu-sudhakaran)
15:48:34 <regXboi> I'm not sure how that made it in
15:48:54 <regXboi> it needs some more import from the reporter
15:48:59 <mlavalle> regXboi: in that case, we have covered all the bugs I wanted to discuss today
15:49:00 <regXboi> er input
15:49:10 <regXboi> mlavalle: ack and mine as well
15:49:11 <mlavalle> any more from the team
15:49:15 <Swami> mlavalle: regXboi: I have another couple of bugs, that I wanted to bring in.
15:49:26 <Swami> #link https://bugs.launchpad.net/neutron/+bug/1501969
15:49:26 <openstack> Launchpad bug 1501969 in neutron "No dhcp IPv6 assigned (slaac/slaac) with interface-add after VM boot" [Medium,In progress] - Assigned to Brian Haley (brian-haley)
15:49:51 <neiljerram> thought that one was done now
15:50:07 <Swami> The patch is ready and carl_baldwin I need your blessings on this.
15:50:46 <Swami> #link https://bugs.launchpad.net/neutron/+bug/1499787
15:50:46 <openstack> Launchpad bug 1499787 in neutron "Static routes are attempted to add to SNAT Namespace of DVR routers without checking for Router Gateway." [Undecided,In progress] - Assigned to Swaminathan Vasudevan (swaminathan-vasudevan)
15:51:01 <carl_baldwin> Swami: Will look
15:51:03 <Swami> I also have a patch to address this issue.
15:51:44 <Swami> carl_baldwin: I just rebased a couple of patches that you already reviewed, need your approval again #link https://review.openstack.org/#/c/230079/
15:52:05 <Swami> carl_baldwin: another one #link https://review.openstack.org/#/c/225319/
15:52:24 <Swami> thanks
15:52:34 <carl_baldwin> Swami: ack
15:53:18 <Swami> mlavalle: are we done with bugs
15:53:48 <regXboi> carl_baldwin, mlavalle: I'm planning on doing a walkthrough to update the undecided items I'm seeing
15:54:13 <mlavalle> done with bugs
15:54:23 <mlavalle> regXboi: thanks!
15:54:42 <mlavalle> regXboi: ping me if you need help
15:54:44 <Swami> carl_baldwin: a general question I do see this test "test_dualnet_dhcp6_stateless_from_os" failing mostly in the gate with both DVR and non-DVR routers. But it is random.
15:55:35 <carl_baldwin> regXboi: obondarev: haleyb: ^
15:56:22 <regXboi> carl_baldwin: right now I consider that to be a race condition
15:56:32 <regXboi> one of many that we see in the gate
15:57:16 <regXboi> as I said earlier, I've been running dvr-multinode-full on a broken out multinode configuration to try and catch these things
15:57:19 <Swami> regXboi: Yes the log message itself reveals that the fip private id is not responding.
15:57:28 <regXboi> and see what's going on with each of them
15:58:32 <regXboi> but that's all I can say for now
15:58:49 <Swami> regXboi: keep me posted.
15:59:22 <carl_baldwin> Swami: Do we have a bug for this failure?
15:59:50 <Swami> carl_baldwin: no I have not created a bug yet, I was looking at various failures, but I will file one.
16:00:14 <elmiko> hi
16:00:32 <carl_baldwin> Swami: thanks
16:00:35 <carl_baldwin> elmiko: ack
16:00:37 <carl_baldwin> #endmeeting