15:00:42 <mlavalle> #startmeeting neutron_l3
15:00:42 <openstack> Meeting started Thu Oct  4 15:00:42 2018 UTC and is due to finish in 60 minutes.  The chair is mlavalle. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:43 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:45 <openstack> The meeting name has been set to 'neutron_l3'
15:00:46 <njohnston> o/
15:00:55 <davidsha> Hi
15:00:59 <mlavalle> Hi everybody
15:01:04 <slaweq> hi
15:01:14 <Swami> mlavalle: I have an internal meeting clashing right now, so will not be able to attend. I will catch up later
15:01:24 <mlavalle> Swami: ack
15:02:06 <mlavalle> #topic Announcements
15:03:02 <mlavalle> Stein-1 is a little more than two weeks away
15:04:00 <mlavalle> Any other annoucements from the team?
15:04:14 <liuyulong> https://blueprints.launchpad.net/neutron/+spec/router-gateway-ip-qos
15:04:31 <liuyulong> I'd like raise this to the team
15:05:38 <mlavalle> liuyulong: is this a request for reviews for the patchsets?
15:06:05 <slaweq> liuyulong: I have it in my pile
15:06:37 <liuyulong> Yap, since it is ready for that. And we have run such code locally for a really long time.
15:07:08 <mlavalle> liuyulong: ack, will review the patches between now and early next week
15:07:14 <mlavalle> Thanks for bringing it up
15:07:30 <mlavalle> any other announcements?
15:07:58 <davidsha> Me and Xubo have a question but we can wait for opens
15:08:14 <mlavalle> davidsha: ack
15:08:19 <mlavalle> #topic Bugs
15:08:37 <mlavalle> First one for today is https://bugs.launchpad.net/neutron/+bug/1774459
15:08:37 <openstack> Launchpad bug 1774459 in neutron "Update permanent ARP entries for allowed_address_pair IPs in DVR Routers" [High,Confirmed] - Assigned to Swaminathan Vasudevan (swaminathan-vasudevan)
15:09:06 <mlavalle> We discussed this bug in a google hang out during the PTG
15:09:14 <mlavalle> and came up with a plan on how to fix it
15:09:48 <mlavalle> I was expecting an update from Swami, but since he is not attebding the meeting, let's wait for an update in the bug
15:09:58 <haleyb> hi
15:10:30 <mlavalle> Next bug is https://bugs.launchpad.net/neutron/+bug/1789434
15:10:30 <openstack> Launchpad bug 1789434 in neutron "neutron_tempest_plugin.scenario.test_migration.NetworkMigrationFromHA failing 100% times" [High,Confirmed] - Assigned to Manjeet Singh Bhatia (manjeet-s-bhatia)
15:10:51 <mlavalle> manjeets has been working on this one since the PTG
15:11:02 <mlavalle> I don't see him in the meeting today
15:11:43 <mlavalle> The only change since then is that the test was marked as unstable
15:12:22 <mlavalle> Next one is https://bugs.launchpad.net/neutron/+bug/1791989
15:12:22 <openstack> Launchpad bug 1791989 in neutron "grenade-dvr-multinode job fails" [High,Confirmed] - Assigned to Slawek Kaplonski (slaweq)
15:12:36 <slaweq> so I'm still working on it
15:12:36 <mlavalle> any updates slaweq?
15:12:58 <slaweq> I recently sent DNM patch which adds more logs to this grenade job
15:13:12 <haleyb> i sent this patch yesterday as well, https://review.openstack.org/#/c/607685/
15:13:20 <mlavalle> is that the one we were re-checking a couple of days ago?
15:13:27 <slaweq> basically it looks like this instance starts pinging after about 4 minutes
15:13:32 <slaweq> mlavalle: yes
15:13:40 <haleyb> it will make the rfp/fpr ARP entries permanent instead of dyanmic
15:13:48 <slaweq> haleyb: and on Your patch there was still same issue
15:13:56 <haleyb> :
15:14:03 <haleyb> :(
15:14:13 <slaweq> but I was today checking logs from Your patch and I found that there is still one STALE entry:
15:14:19 <slaweq> 169.254.125.69 dev rfp-a79c5312-8 lladdr aa:28:12:72:ae:2c STALE
15:14:32 <slaweq> it's in qrouter-XXX namespace on host where instance is
15:14:50 <slaweq> and when ping works, thos becomes REACHABLE
15:15:24 <haleyb> i'll have to see where that's getting set.  so is the gateway IP still STALE as well?
15:15:29 <slaweq> haleyb: I didn't have time to check why this entry is not added as PERMANENT
15:15:50 <slaweq> haleyb: no, this is the only STALE entry now
15:15:55 <slaweq> nothing else like that
15:16:14 <slaweq> ahh, no
15:16:16 <slaweq> wait
15:16:18 <slaweq> there is also:
15:16:28 <slaweq> 172.24.5.1 dev fg-23bc9681-fc lladdr 86:1c:ee:fe:fb:40 STALE
15:16:35 <slaweq> in fip-88d3d067-d1cb-4298-8c46-dfb3cac06d89
15:16:50 <slaweq> so it looks that it's still the same as we were checking yesterday
15:17:13 <slaweq> and also:
15:17:14 <slaweq> 169.254.125.68 dev fpr-a79c5312-8 lladdr b6:2f:ee:39:12:42 STALE
15:18:13 <haleyb> we can at least try and track down the rfp/fpr ones, not sure about the fg- one
15:19:51 <slaweq> if someone else wants to check, here is log from grenade job: http://logs.openstack.org/85/607685/2/check/neutron-grenade-dvr-multinode/48b00e1/logs/grenade.sh.txt.gz
15:20:20 <slaweq> it has added some additional info like iptables-save, openflow rules, ip a, ip n, ip r from each namespace on each host and things like that
15:20:40 <slaweq> those logs are added after failed attempt of pinging FIP
15:20:55 <slaweq> and then pinging is retried until it will be fine
15:21:05 <slaweq> then logs are collected once again
15:21:32 <slaweq> and we were comparing log from moment "before ping starts working" and "after ping works"
15:21:56 <slaweq> and only differences we saw with haleyb was those STALE entries in qrouter- and fip- namespaces
15:22:07 <slaweq> maybe someone will have any idea why it is like that
15:22:09 <mlavalle> in qr-xxx and fip-xxxx?
15:22:34 <slaweq> mlavalle: yes, on compute where instance with attached FIP is
15:24:59 <mlavalle> anything else on this bug?
15:25:05 <slaweq> mlavalle: no :/
15:25:26 <mlavalle> slaweq: thanks for continuing working on it :-)
15:26:08 <mlavalle> Next one is https://bugs.launchpad.net/neutron/+bug/1787919
15:26:08 <openstack> Launchpad bug 1787919 in neutron "Upgrade router to L3 HA broke IPv6" [High,Confirmed] - Assigned to Miguel Lavalle (minsel)
15:26:17 <mlavalle> I've been working on this one
15:26:57 <mlavalle> I was hoping to reproduce / debug it in a DVR-HA environment
15:27:37 <mlavalle> whereas all the entries in the bug seem to refer to pure HA
15:27:53 <mlavalle> I have a question for haleyb:
15:28:04 <haleyb> sure
15:28:12 <mlavalle> I was looking at https://review.openstack.org/#/c/442518
15:28:22 <mlavalle> which is refered to in the bug
15:29:10 <mlavalle> will the set up in that patch in the qr-xxx name space for ipv6 forwarding happen in the fip-xxx namespace if my router is dvr-ha?
15:31:06 <haleyb> mlavalle: i don't think so
15:31:35 <mlavalle> ok, so in my testing / debugging I need to get the fip namespace out of the way
15:31:51 <haleyb> ipv6 n/s should be in qrouter, i can look at the bug too
15:32:12 <mlavalle> if you have time, that would be welcome
15:32:29 <mlavalle> I can also try testing / debugging with a pure HA router
15:32:38 <mlavalle> like the one in the bug
15:33:27 <mlavalle> This seems to be the piece of code in question: https://github.com/openstack/neutron/blob/master/neutron/agent/l3/router_info.py#L687
15:33:56 <mlavalle> ok, I'll continue digging
15:34:00 <mlavalle> thanks haleyb
15:34:09 <haleyb> ack
15:34:39 <mlavalle> Last bug I have for today is https://bugs.launchpad.net/neutron/+bug/1792493
15:34:39 <openstack> Launchpad bug 1792493 in neutron "DVR and floating IPs broken in latest 7.0.0.0rc1?" [High,Triaged]
15:34:50 <mlavalle> This one doesn't have an owner yet
15:34:54 <mlavalle> any takers?
15:35:31 <mlavalle> if not, I'll take it after I'm done with the previous one
15:35:46 <mlavalle> Any other bugs we should discuss today?
15:36:28 <mlavalle> ok....
15:36:37 <slaweq> I wanted to mention one
15:36:37 <mlavalle> #topic open flow DVR
15:36:39 <slaweq> https://bugs.launchpad.net/neutron/+bug/1794809
15:36:39 <openstack> Launchpad bug 1794809 in neutron "Gateway ports are down after reboot of control plane nodes" [Medium,In progress] - Assigned to Slawek Kaplonski (slaweq)
15:36:42 <mlavalle> #undo
15:36:43 <openstack> Removing item from minutes: #link https://bugs.launchpad.net/neutron/+bug/1794809
15:36:58 <mlavalle> #link https://bugs.launchpad.net/neutron/+bug/1794809
15:37:10 <slaweq> and patch for it is ready for review https://review.openstack.org/#/c/606085/ so I just wanted to ask team for review it :)
15:37:17 <slaweq> and that's all :)
15:37:30 <mlavalle> thanks!
15:37:49 <mlavalle> davidsha: you are up
15:38:11 <davidsha> Hey, we're working on the refactor of the agent code and we came across abit of a problem for us. We're trying to move the driver, namespace manager and a few other objects into the router info class
15:38:14 <davidsha> thanks!
15:38:58 <davidsha> Problem is the routerinfo class if inherited by NameSpace object and it causes a circular import
15:39:29 <davidsha> We're just trying to figure out what the best way to go about circuventing this is.
15:40:11 <mlavalle> well, my first reaction is let's keep the separate. Isn't a router a specialization of namespace?
15:41:06 <mlavalle> and the fact that you are hitting the circularity a sumptom that maybe the approach is not right?
15:41:14 <davidsha> Yes, but we wanted to move the NameSpaceManager into RouterInfo as a static variable within the class
15:41:19 <mlavalle> symptom^^^
15:41:43 <davidsha> NameSpaceManager imports NameSpace class to store in a dict
15:42:31 <mlavalle> has any of that re-factoring been pushed to gerrit?
15:42:38 <davidsha> Maybe this discussion might be better on the patch when it's pushed
15:42:53 <mlavalle> yeah, hence my question ^^^^
15:42:59 <mlavalle> I think so
15:43:14 <davidsha> An early version was but this is a bit durther along that the previous patch
15:43:36 <davidsha> further* than*
15:43:55 <davidsha> but ya, I'm meeting Xubo this evening and I'll push what we have so far
15:44:02 <mlavalle> let's take a look when you push the latest version
15:44:12 <davidsha> kk, Thanks!
15:44:22 <mlavalle> please ping me when it is up :-)
15:44:26 <mlavalle> THanks!
15:44:30 <davidsha> will do!
15:44:42 <mlavalle> ok, let's move on
15:44:52 <mlavalle> #topic On demand agenda
15:45:00 <mlavalle> anything else we should sicuss today?
15:45:30 <mlavalle> ok, team... thanks for your time
15:45:35 <davidsha> Thanks!
15:45:36 <mlavalle> have a nice weekend!
15:45:41 <mlavalle> #endmeeting