15:00:48 <carl_baldwin> #startmeeting neutron_l3
15:00:49 <openstack> Meeting started Thu Mar 20 15:00:48 2014 UTC and is due to finish in 60 minutes.  The chair is carl_baldwin. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:50 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:52 <openstack> The meeting name has been set to 'neutron_l3'
15:01:11 <carl_baldwin> #topic Announcements
15:01:24 <carl_baldwin> I never came to a decision about a meeting time change.
15:01:53 <carl_baldwin> ajo: You said that an hour earlier would be better but would two hours later be bad?
15:02:38 <Swami> carl: I too feel the same, because of the time change it just affects our regular schedule, childrens school time.
15:02:43 <ajo> carl_baldwin, hmm, yes for me, two hours later I have another fixed meeting :(
15:02:53 <ajo> I mean, yes, it would be bad for me
15:03:15 <carl_baldwin> Swami: which time would you prefer?
15:03:16 <ajo> -1h, or +0h works for me, other option would be +3h
15:03:19 <xuhanp> carl_baldwin, two hour later will be too late for Asia folks :-(
15:03:25 <safchain_> 2 hours later bad for me too
15:03:43 <ajo> xuhanp, you're right
15:03:47 <Swami> probably 1 hour earlier would be better.
15:04:13 <carl_baldwin> It sounds like the current time is the best.
15:04:33 <Swami> never mind, its ok,
15:04:36 <carl_baldwin> Oh, maybe 1 hour earlier would be better.
15:04:48 <ajo> hi rook & amuller
15:04:52 <amuller> ajo: heya
15:04:53 <rook> ajo hi
15:04:55 <ajo> we were talking about this meeting time
15:05:08 <ajo> may be changing -1h, staying, or +1h
15:05:19 <amuller> ok, no preference here
15:05:22 <rook> same
15:05:39 <carl_baldwin> Well let's take a vote between an hour earlier and keeping the same time.
15:05:48 <ajo> there was +2h too, but that's bad for asia, and collides with our weekly team meeting :)
15:06:03 <carl_baldwin> Keep in mind that the next daylight savings time shift will shift it yet another hour earlier.
15:06:12 <carl_baldwin> +2 hours I think is out.
15:07:24 <Swami> Same time slot will work. +1
15:07:33 <safchain_> +1 for the same time
15:07:35 <ajo> what's the meeting time reference? UTC/GMT ?
15:07:50 <ajo> +1 for the same time (considering day light saving time changes..)
15:08:00 <carl_baldwin> Current meeting time is 1500 UTC with no regard for DST.
15:08:17 <xuhanp> +1 to same time or -1 hour :-)
15:08:36 <ajo> well, sorry, changing my vote for: +1 same time or -1 hour
15:09:15 <carl_baldwin> Thanks.  I'll see what I can do to keep the same time.
15:09:38 <ajo> Thanks carl
15:09:45 <carl_baldwin> #topic l3-high-availability
15:10:04 <carl_baldwin> safchain_: Any update since last week?
15:10:34 <safchain_> Not really, I have to rebase all my patches to update db migration version
15:10:55 <safchain_> I also submitted a patch to show how the conntrackd patch could be used
15:11:20 <safchain_> I'm working on a patch to send a notification to the neutron server when a master is elected
15:11:36 <carl_baldwin> Has anyone had the chance to run through the document about testing the patches?
15:11:44 <amuller> safchain_: I've been meaning to talk to you - I think it would be more productive if you took over the conntrackd patch that I originally created... Fix it up according to your needs
15:12:16 <safchain_> amuller, ok why not
15:12:35 <amuller> cool :)
15:12:41 <safchain_> any suggestion about the notification about the vrrp status ?
15:12:41 <ajo> amuller, safchain_ can you point to your current conntrackd patches ?
15:13:05 <safchain_> https://review.openstack.org/#/c/80332/
15:13:07 <safchain_> ajo ^
15:13:13 <amuller> ajo: https://review.openstack.org/#/c/71586/, https://review.openstack.org/#/c/80332/
15:13:38 <safchain_> ajo, not working today since there are some few things to change in the conntrackd patch
15:14:23 <ajo> thanks amuller , safchain_
15:14:57 <carl_baldwin> I'll have another look at the patches this week and begin going through your test document.
15:15:00 <carl_baldwin> Anything else?
15:15:12 <safchain_> thanks carl_baldwin
15:15:18 <Swami> amuller: you had a blueprint for the active /active solution, are you targeting that Juno
15:15:30 <Swami> Will it be merged with safchain_ vrrp solution.
15:15:51 <safchain_> amuller, it could be a nice improvement
15:15:52 <Swami> Or will be patch that will go after the current vrrp solution
15:16:15 <amuller> Swami: I'm not 100% on the value of it. I think with DVR, the VRRP stuff will only be used for SNAT traffic
15:16:32 <Swami> amuller: yes you are right.
15:16:49 <carl_baldwin> If it could slow down the original patches then it should go in after.
15:17:06 <carl_baldwin> scope creep and all  ;)
15:17:06 <amuller> carl_baldwin: The original plan was to file a follow up blueprint, and add patches later
15:17:17 <Swami> amuller: Have you started the work.
15:17:33 <amuller> Swami: No, I have an early draft for a blueprint, haven't touched it since
15:17:54 <amuller> Assuming DVR is merged for Juno I am not 100% sure about the value of a VRRP optimization
15:18:14 <Swami> As I mentioned in my previous discussions, we need to figure out for Juno how the VRRP solution will complement the DVR solution.
15:18:32 <Swami> amuller: thanks
15:18:38 <carl_baldwin> amuller: We can discuss it at a later time.  There is also a possible bp for distributed SNAT or some other alternative to centralized SNAT.
15:18:56 <carl_baldwin> So, I agree the value of active / active should be in question.
15:19:14 <carl_baldwin> Let's move on.
15:19:15 <amuller> carl_baldwin: I am not keen on the details of DVR yet or why distributed SNAT is an issue, but that would be preferable
15:19:33 <carl_baldwin> #topic neutron-ovs-dvr
15:19:44 <carl_baldwin> Swami: hi
15:19:44 <Swami> Hi carl
15:19:55 <ajo> VRRP also has a value
15:20:03 <Swami> The DVR team has posted the L2 and L3 documents for review.
15:20:18 <ajo> when there is a need to cut down the physical network nodes
15:20:19 <Swami> Teaming is trying to address the comments that come in.
15:20:25 <ajo> or there are many physical networks
15:20:30 <Swami> carl: thanks for yur comments.
15:21:03 <ajo> sorry : when there's a need to cut down public/physical network connections
15:21:18 <Swami> We are currently progressing on the East-West story.
15:21:45 <Swami> The issue that we had with the East-west was with some rules, and we are now trying to sort it out with sylvain.
15:22:08 <Swami> Because there was dup learning in the bridges.
15:22:24 <Swami> That's all I have.
15:22:54 <Swami> Some of the sub team members requested for the WIP code, we are planning to post the WIP code when the East-west is completely working.
15:23:05 <carl_baldwin> I got disconnected momentarily.  I'll go read the log to see if I missed anything.
15:23:07 <amuller> Very cool
15:23:27 <carl_baldwin> I'm anxious to look at the implementation, too.  :)
15:23:45 <Swami> It should be there soon
15:23:54 <carl_baldwin> #topic bgp-dynamic-routing
15:24:06 <carl_baldwin> nextone92: Are you around?
15:24:09 <Swami> That's all from my side
15:24:12 <nextone92> Hi Carl!
15:24:22 <carl_baldwin> Swami: thanks
15:24:52 <nextone92> From the last discussion: made it more clear in the blueprint that primary use case is to exchange the routes for the whole OpenStack system
15:25:19 <nextone92> And also added list of permitted users who can adjust bgp settings on the router
15:25:43 <carl_baldwin> I'll have a look at your updates.
15:25:48 <nextone92> I've looked at MPLS BGP blueprint: https://blueprints.launchpad.net/neutron/+spec/neutron-bgp-mpls-vpn
15:26:31 <nextone92> Based on the available information, I think dynamic routing implementation should benefit VPN deployment that requires dynamic routing
15:27:16 <nextone92> One question that I had, should the dynamic routing document primarily concentrate on dynamic routing? With bgp being the first available implementation?
15:29:10 <carl_baldwin> I'll admit that I don't have much experience with dynamic routing outside of BGP.
15:29:47 <Swami> nextone92: Yes you are right, that should compliment the VPN
15:29:48 <carl_baldwin> I guess to answer that question we would need to understand what other dynamic routing protocols are out there and find out what can be abstracted.
15:30:03 <amuller> OSPF and ISIS stand out
15:30:13 <amuller> open, standardized
15:30:17 <ajo> yes, an abstraction would be interesting, if we can gather the knowledge
15:30:37 <Swami> With respect to protocols, we can start with one and then add others.
15:31:15 <Swami> The most used one and the immediate requirement would be the bgp.
15:31:35 <ajo> sounds good
15:31:41 <carl_baldwin> Swami: That is the plan.  BPG will be the first implementation.  The question is how much can we abstract the interface so that it is applicable to others down the road.
15:32:02 <nextone92> Okay, sounds good!
15:32:05 <nextone92> What do you think about the idea of having a main virtual "provider" router to handle dynamic routing? This entity will exchange routing information but doesn't necessarily have to forward packets
15:32:36 <amuller> Looking at the blueprint, I really don't understand what is the use case / what problem are we trying to solve
15:32:48 <carl_baldwin> nextone92: I think the idea is good.  It coincides with what I was thinking.  My only problem with it is calling it a "router" since it doesn't route.
15:33:49 <nextone92> carl_baldwin: let me think of a good alternate name
15:34:05 <Swami> But if it is abstracted as a dynamic routing service, then can be used with any other variable..
15:34:37 <carl_baldwin> amuller: A use case that I have in mind is to use dynamic routing to announce IPs on an external network.  There are some advantages to using routing over a flat L2 network.
15:35:19 <nextone92> amuller: and another user case would be for external network to provide routing information to OpenStack
15:35:33 <amuller> Does this blueprint have a use outside of VPNaaS?
15:35:39 <nextone92> amuller: this could be useful if you have multiple uplinks
15:35:54 <carl_baldwin> nextone92: Maybe we call it the "main dynamic routing service" instead of a router.
15:36:01 <amuller> I mean, current OpenStack virtual routers only have a single uplink so no routing is needed
15:36:24 <carl_baldwin> amuller: Both use cases just mentioned are outside of VPNaaS.
15:36:49 <nextone92> amuller: virtually there may have a single uplink, but it doesn't have to be a single uplink physically
15:37:31 <amuller> ok, then I don't understand "A use case that I have in mind is to use dynamic routing to announce IPs on an external network" - The end goal is for virtual routers to know how to route to certain IPs outside of the cloud?
15:38:00 <amuller> So virtual routers would be connected to multiple external networks?
15:38:14 <ajo> multiple uplinks
15:39:21 <nextone92> amuller: in the use case that carl has mentioned, when you add an external network and/or floating IPs, the routing information to that network will be announced to the uplink router(s)
15:39:31 <nextone92> amuller: simplifying network administration
15:39:56 <amuller> ahh, so you're talking about routers outside of the cloud to know how to route into the cloud, not the other way around
15:39:57 <carl_baldwin> amuller: In current neutron, floating ips are all on a big flat L2 network.
15:40:23 <carl_baldwin> amuller: Both are use cases that we have discussed.
15:41:31 <amuller> I understand the 2nd one - Instead of configuring your cloud's physical edge router, and adding the external networks so they're advertised via BGP, the openstack virtual routers will automatically generate BGP advertisements whenever an external net is defined
15:42:25 <amuller> Basically saving you a few lines of configuration on your physical routers
15:42:37 <nextone92> amuller: great! Going in another direction, a system administrator may add a route to a segment of a corporate network on one of the uplinks
15:43:06 <nextone92> amuller: or define a load balancing policy based on routes and their weights
15:43:44 <carl_baldwin> amuller: We can do some work on the document to make the use cases more clear and discuss them individually after that.
15:44:00 <amuller> carl_baldwin: That would be fantastic
15:44:11 <ajo> +1 :)
15:44:17 <nextone92> great!
15:44:27 <carl_baldwin> #action carl_baldwin will take a crack at describing the use cases.
15:44:33 <amuller> carl_baldwin: If you could state things explicitly (Such as virtual routers having more than one uplink / connected to more than a single external network at a time)
15:44:55 <amuller> carl_baldwin: + Why the other direction is interesting as well (Virtual routers advertising their own external networks via a routing protocol)
15:45:13 <carl_baldwin> I will try to be as clear as possible.
15:45:35 <carl_baldwin> #topic DNS lookup of instances
15:45:40 <amuller> I don't know how customers set up routing on their clouds atm - Do they have an IGP running, then redistributing out to BGP, or what...
15:45:45 <amuller> That would affect our choice of a routing protocol
15:46:41 <carl_baldwin> amuller: that is an open question.  Maybe we can do some discovery here.
15:46:59 <carl_baldwin> No progress on DNS since my whole week was consumed with unplanned work.
15:47:20 <carl_baldwin> #topic Agent Performance with Wrapper Overhead
15:47:30 <carl_baldwin> #link https://etherpad.openstack.org/p/neutron-agent-exec-performance
15:48:23 <ajo> Yuriy, are you around?
15:48:33 <carl_baldwin> I made some progress on a change to L3 agent to make it more responsive to RPC.
15:48:40 <carl_baldwin> #link https://review.openstack.org/#/c/78819
15:48:48 <YorikSar> ajo: o/
15:48:56 <carl_baldwin> The patch is still a WIP but pretty much works.  I could use some feedback.
15:49:25 <YorikSar> I've pushed first version of rootwrap changes to make it run in agent mode.
15:49:44 <YorikSar> I've also though of another speedup we can get dealing with namespaces (see ML and etherpad)
15:49:45 <carl_baldwin> YorikSar: hi
15:49:48 <carl_baldwin> Do you have a link?
15:49:58 <YorikSar> https://review.openstack.org/81798
15:50:19 <ajo> #link https://review.openstack.org/81798
15:50:46 <ajo> YorikSar, your results look promising
15:50:52 <YorikSar> It shows good performance (~2x speedup compared to plain sudo, ~22x compared to rootwrap)
15:51:16 <carl_baldwin> Great, I'll have a look.
15:51:48 <ajo> YorikSar, yes, I thought about using setns too, instead of using the full blown ip netns exec
15:51:49 <YorikSar> I think I can add some namespace-related stuff there. But I wonder if it should be done in Neutron itself, not in common rootwrap.
15:52:05 <ajo> no mounting or normal network env configuration, but it should do for many cases
15:52:13 <ajo> and we cut out the overhead a lot
15:52:35 <YorikSar> I didn't investigate if any command running in namespace actually needs conf files etc
15:52:48 <ajo> YorikSar,  may be we could add some filter, to do setns
15:52:57 <ajo> setns xxxxxxxxxxxx ip a
15:53:05 <YorikSar> But from what I see in filters it looks like only ip itself runs within ip netns
15:53:10 <ajo> get's handled by the filter, which sets the namespace, and calls ip a
15:53:30 <carl_baldwin> I'm not familiar with setns.
15:53:42 <ajo> ip netns exec does several things
15:53:46 <YorikSar> ajo: Hm... Sideeffects are bad...
15:53:49 <ajo> calls "setns(xxxx) system calls...
15:54:11 <YorikSar> ajo: I think we can add an argument to that RPC call.
15:54:13 <ajo> and mounts several files around to simulate a normal environment on /etc/ or other places for the child process
15:54:24 <carl_baldwin> Oh, I see.  I'm following you.
15:54:28 <YorikSar> ajo: And leave plain rootwrap ip setns exec slow.
15:54:44 <ajo> the mounting itc.. it's very expensive in performance terms, and may processes probably would need the setns only
15:54:57 <ajo> YorikSar,
15:55:16 <ajo> do you think it would be possible to implement in neutron, with a switch to allow the old root_helper method , or the new daemon?
15:55:34 <ajo> this would allow for a testing transition during icehouse, for example
15:56:17 <YorikSar> ajo: Yes, that's the next step I'm thinking about.
15:56:20 <ajo> who's more worried about performance could enable the mechanism, and who's not, can relay on rootwrap, or even plain sudo
15:57:09 <ajo> YorikSar,
15:57:15 <ajo> did you read my concerns about "BaseManager"
15:57:41 <ajo> I yet don't have the details, but it seems that other projects using a similar mechanism for the same purpose have hit race conditions & bugs on the BaseManager implementation
15:57:54 <YorikSar> ajo: Yes. I hope you'll be able to get what issues there was from your people.
15:57:54 <ajo> do you think we could have some other alternatives if we hit those?
15:58:08 <YorikSar> ajo: We can backport any changes necessary.
15:58:12 <ajo> sure, I should get the information sure
15:58:19 <ajo> sure->soon
15:58:35 <YorikSar> ajo: In fact... I think we shouldn't hit races with running only one background process.
15:59:34 <ajo> I have also built a POC on the py->C++ translation
15:59:46 <ajo> it works with very little overhead, I was aiming a solution for havana/icehouse
15:59:55 <ajo> but may be your solution will be soon enough, and backportable
16:00:05 <carl_baldwin> ajo: What have you found out about popen and logging?
16:00:17 <ajo> carl_baldwin, was about to write on this,
16:00:29 <ajo> we would need to implement those modules,
16:00:44 <ajo> I suppose is doable, but I'm more concerned about the auditability of the code
16:01:03 <ajo> everything looks to me like dynamic, std::list<> ... etc...
16:01:09 <ajo> and with bounds checkings everywhere
16:01:32 <ajo> but, of course, it's machine generated C, which is hard to read
16:01:54 <ajo> the performance is good, did my message on performance arrived to the list?
16:02:11 <carl_baldwin> I'm not sure.  I'm a little behind reading the list.
16:02:47 <carl_baldwin> With the need to hand-implement some modules, this looks less attractive but maybe we can keep it in our back pocket.
16:02:59 <ajo> #link http://lists.openstack.org/pipermail/openstack-dev/2014-March/030561.html
16:03:09 <carl_baldwin> I don't know if there is a meeting after this but I'm running out of time.
16:03:22 <ajo> I must leave too,
16:03:29 <carl_baldwin> ajo: Have you written anything on the ether pad about setns?  I'd like to think about it some more.
16:03:30 <YorikSar> ajo: I think given my rootwrap daemon beats sudo itself, running C++ version of rootwrap will be behind anyway.
16:03:40 <ajo> but yes, I agree, it's a solution to keep in the back pocket
16:04:33 <ajo> YorikSar, I agree, if the neutron side implementation gets working fast, and work reliably , we should not need for any C++
16:04:52 <YorikSar> What's the deadline for such change?
16:04:54 <ajo> carl_baldwin, sure I could try to expand the information on ip netns
16:05:09 <ajo> carl_baldwin, man ip-netns explains the logic
16:05:14 <ajo> on the first few lines
16:05:24 <YorikSar> I've added some info on setns to the etherpad
16:05:33 <ajo> ah, good YorikSar  :)
16:05:51 <carl_baldwin> I got disconnected again.  :(
16:05:54 <YorikSar> But I didn't explain how netns exec works there.
16:06:09 <ajo> YorikSar, I'd aim on having something that can work in icehouse.
16:07:03 <carl_baldwin> We may need to be content with potential back port to Icehouse.  But, we can always ask.
16:07:51 <ajo> Sure, if we keep backporting in mind, it should be ok
16:08:16 <YorikSar> Oh... According to schedule, we just a week away from RC...
16:08:24 <carl_baldwin> Thanks, all.  Have a great week.
16:08:26 <ajo> YorikSar, I will spend some time tomorrow morning on review to your rootwrap daemon, it looks very good to me, so far, simple & effective
16:08:40 <carl_baldwin> I will review as well.
16:08:44 <YorikSar> ajo: Thanks, I appreciate that.
16:08:49 <carl_baldwin> #endmeeting