08:00:20 <gsagie> #startmeeting dragonflow
08:00:21 <openstack> Meeting started Mon Mar 28 08:00:20 2016 UTC and is due to finish in 60 minutes.  The chair is gsagie. Information about MeetBot at http://wiki.debian.org/MeetBot.
08:00:22 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
08:00:24 <openstack> The meeting name has been set to 'dragonflow'
08:00:30 <oanson> Hi
08:00:32 <gampel1> hi
08:00:37 <gsagie> Hello all, who is here for Dragonflow meeting?
08:01:21 <gsagie> #info oanson, gampel1, gsagie in meeting
08:01:35 <Shlomo_N> hi
08:01:56 <gsagie> #info Shlomo_N in meeting, yuli_s suppose to be in meeting
08:01:58 <gsagie> ;)
08:02:40 <Shlomo_N> in a min
08:02:44 <gsagie> no one from China is logged in
08:02:47 <gsagie> :(
08:02:52 <gsagie> ok lets do a fast meeting
08:03:00 <gampel1> i just pinged frank
08:03:00 <oanson> Hear hear
08:03:12 <gsagie> ok, so lets wait for him 1-2 minutes
08:04:00 <gsagie> #topic stable branch
08:04:33 <gsagie> gampel1: we need to decide on a time we do code freeze for new features
08:04:42 <gsagie> and start stabling things and tag a release
08:04:56 <gampel1> yes and ideas ?
08:04:56 <gsagie> our release is independent so we can do it regardless of Mitaka release
08:05:07 <gsagie> lets see features we want in:
08:05:09 <gsagie> security groups
08:05:12 <gsagie> dnat
08:05:13 <gampel1> yes
08:05:24 <gsagie> selective proactive, redis
08:05:28 <gsagie> OVS as a service
08:05:32 <gsagie> and soem bug fixes
08:05:37 <gsagie> anything else?
08:05:43 <gsagie> some
08:05:46 <gampel1> what about controller reliability
08:05:56 <gsagie> yeah that too
08:06:14 <oanson> publish subscribe using publisher table?
08:06:26 <gsagie> yes
08:06:59 <gsagie> so lets decide that no other new features are getting in, and we are focusing on bug fixes
08:07:15 <gsagie> and we release before the summit
08:07:19 <gampel1> I think that we could aim for code freeze in mid next mount , what do you think
08:07:27 <gsagie> yes that sounds good
08:07:56 <gsagie> we do need few days for stabilizing current code after all the merges so lets try to merge everything by start of next week (feature wise)
08:08:07 <oanson> When's the release date?
08:08:14 <gampel1> I have the DHCP ODS up for review but we could push it to next release
08:08:20 <gsagie> April 18
08:08:38 <gsagie> gampel: yeah lets postpone it
08:08:51 <oanson> That doesn't give us a lot of time between code freeze and release
08:08:53 <yuli_s> heh, it's my birthday date
08:08:57 <yuli_s> :)
08:09:13 <gsagie> we merge everything by 7/4, sounds reasonable gampel?
08:09:16 <gampel1> of so feature freeze next Wednesday  ?
08:09:36 <gampel1> yes
08:09:39 <gsagie> 7/4 end of next week
08:09:44 <gampel1> sound good
08:09:49 <gsagie> and we call this version "yulison"
08:09:56 <yuli_s> ;)
08:10:00 <gsagie> :)
08:10:00 <gampel1> :)
08:10:03 <Shlomo_N> :)
08:10:05 <oanson> :)
08:10:13 <gsagie> good lots of happy faces
08:10:32 <gsagie> now just need to also let nick-ma, DuanKebo, dingboopt, Raofei know
08:10:39 <gsagie> and all the beijing team
08:11:03 <gsagie> gampel: want to send an email to everyone or should i?
08:11:14 <gampel1> I will no problem
08:11:16 <gsagie> i will make the release patch
08:11:40 <gsagie> #action gampel1 send email about feature freeze (7/4) and stable release tag (18/4)
08:12:25 <gsagie> okie, should we talk about redis/security groups/dnat/reliability or move to bugs ? as no one is here to talk about these features
08:13:06 <gsagie> #topic bugs
08:13:06 * yuli_s I can start with bugs
08:13:06 <gampel1> lets move to the BUGS and hope that they will join
08:13:20 <yuli_s> ok great
08:13:25 <gsagie> yuli_s, oanson please update
08:13:38 <yuli_s> we have 4 bugs
08:13:53 <yuli_s> that needs somebody to take
08:13:59 <yuli_s> https://bugs.launchpad.net/dragonflow/+bug/1548725
08:13:59 <openstack> Launchpad bug 1548725 in DragonFlow "It's not possible to assign a FIP for a VM on the default private network" [High,New]
08:14:00 <yuli_s> https://bugs.launchpad.net/dragonflow/+bug/1557412
08:14:00 <yuli_s> https://bugs.launchpad.net/dragonflow/+bug/1559841
08:14:01 <yuli_s> https://bugs.launchpad.net/dragonflow/+bug/1517049
08:14:01 <openstack> Launchpad bug 1557412 in DragonFlow "VM connected to two different networks" [High,New]
08:14:02 <openstack> Launchpad bug 1559841 in DragonFlow "dhcp rpc is not handled at plugin side" [Medium,New]
08:14:03 <openstack> Launchpad bug 1517049 in DragonFlow "Creating a VM on the public network crashes the DragonFlow" [Low,New]
08:14:33 <yuli_s> In addition, I need help in pushing fix to the degradation problem found in neutron,
08:15:04 <gsagie> yuli_s: triage them to me and i will investigate, 1557412 and 1559841 sounds more urgent
08:15:30 <gsagie> yuli_s: do you know what is the problem?
08:15:41 <yuli_s> a fix was proposed, but somehow it is not progressing
08:15:43 <gampel1> I think that 1517049 is not reproducible
08:16:06 <gsagie> yuli_s: do you have review link?
08:16:21 <yuli_s> sec/
08:16:26 <Shlomo_N> I already published a patch for 1517049
08:16:41 <gampel1> yuli_s:  will you be able to try reproduce 1517049
08:16:45 <Shlomo_N> a long time ago
08:16:55 <oanson> I can investigate 1517049
08:17:05 <yuli_s> https://review.openstack.org/#/c/293976/
08:17:09 <gampel1> Shlomo_N: But the infinite loop is not happening
08:17:14 <gsagie> Shlomo_N: yes i remember, lets see if it reproduce first
08:17:22 <Shlomo_N> ok
08:17:27 <Shlomo_N> sure
08:17:56 <gampel1> I can look at 1548725
08:18:07 <gsagie> #link https://review.openstack.org/#/c/293976/  control plane degradation in Neutron
08:18:36 <gsagie> ok assigned 1548725 to gampel1
08:19:02 <yuli_s> it needs a push
08:19:13 <yuli_s> no progress probably for 10 days already
08:19:40 <gsagie> https://bugs.launchpad.net/dragonflow/+bug/1517049  - assigned to Yuli_s but if you need help oanson volunteered to look at it as well
08:19:40 <openstack> Launchpad bug 1517049 in DragonFlow "Creating a VM on the public network crashes the DragonFlow" [Low,New] - Assigned to Yuli (stremovsky)
08:19:46 <gsagie> i will assign the others to me
08:19:56 <gampel1> which high priority  bugs we have on the list now ?
08:20:02 <gsagie> yuli_s: ok, i will try to help but its out of our hands
08:20:12 <yuli_s> ok
08:20:23 <gsagie> oanson: any status on the bugs you handle? or everything under control?
08:20:33 <oanson> everything under control
08:20:48 <oanson> https://bugs.launchpad.net/dragonflow/+bug/1530516 is no longer happening
08:20:48 <openstack> Launchpad bug 1530516 in DragonFlow "No west-east flows add into table 20 when running the distributed dragonflow mode" [High,In progress] - Assigned to Omer Anson (omer-anson)
08:21:09 <oanson> I added a test to it in https://review.openstack.org/#/c/297316/
08:21:35 <oanson> https://bugs.launchpad.net/dragonflow/+bug/1561517 has a proposed fix in https://review.openstack.org/#/c/297188/
08:21:35 <openstack> Launchpad bug 1561517 in DragonFlow "OVSDB monitor reports events on irrelevant ports" [High,In progress] - Assigned to Omer Anson (omer-anson)
08:21:55 <oanson> and https://bugs.launchpad.net/dragonflow/+bug/1530878 gives me a strange error in CI.
08:21:55 <openstack> Launchpad bug 1530878 in DragonFlow "OVS vswitchd and ovsdb-server start as services" [High,In progress] - Assigned to Omer Anson (omer-anson)
08:21:56 <gampel1> what about https://bugs.launchpad.net/dragonflow/+bug/1523509 can we close it
08:21:57 <openstack> Launchpad bug 1523509 in DragonFlow "Plugin dont raise exceptions if delete failed due to key not found" [High,New] - Assigned to Gal Sagie (gal-sagie)
08:21:59 <gsagie> oanson: ok you can approach the original creator of the bug (in email) and ask him if he can reproduce it, if not you can close the bug
08:22:00 <oanson> I'm investigating it.
08:22:34 <oanson> gsagie, the bug was opened 86 days ago. I think that once the tests merge, I can close the bug anyway
08:22:48 <gampel1> I think 1523509 is invalid
08:23:09 <gsagie> gampel1: i will take a look
08:23:36 <gsagie> I dont think we suppress deletes of all objects in the plugin yet (if they dont exist)
08:23:42 <gsagie> but need to recheck after nick-ma work
08:23:47 <gampel1> we need to make sure https://bugs.launchpad.net/dragonflow/+bug/1558415 is handled
08:23:47 <openstack> Launchpad bug 1558415 in DragonFlow "the problem of modules start sequence for dragonflow" [High,New] - Assigned to hujie (mike-hu)
08:23:47 <gsagie> maybe he handled this case
08:24:09 <gsagie> gampel1: i think he wants to do a meeting about this whole feature
08:24:16 <gampel1> yes
08:24:16 <gsagie> maybe its best to schedule it with him and DuanKebo
08:24:21 <gampel1> Ok
08:25:14 <gsagie> okie, anything else anyone?
08:25:27 <gsagie> #topic performance testing
08:25:33 <gsagie> Shlomo_N: want to update?
08:25:49 <Shlomo_N> After running multiple tests, I have managed to generate the following performance:
08:26:01 <Shlomo_N> 1)	4 VMs running from single host managed to generate 9Gb/s (not line rate)
08:26:09 <Shlomo_N> * 4 vms
08:26:10 <gsagie> #action schedule meeting with hujie and DuanKebo to talk about controller reliability
08:26:29 <Shlomo_N> 2) After increasing VM’s properties, it managed to generate ~5Gb/s (from single VM)
08:26:47 <Shlomo_N> Tested with 2/4/8/16 cores.
08:27:11 <gsagie> we can also test with core affinity as we mentioned
08:27:31 <gsagie> but its less important right now
08:27:39 <Shlomo_N> ok
08:27:47 <Shlomo_N> We need to decide how we want to proceed from here.
08:27:55 <Shlomo_N> We have few options:
08:28:05 <Shlomo_N> - Run 4/5 VMs in parallel.
08:28:10 <gsagie> Shlomo_N: can you run two iperf processes from the same VM when it has multi cores?
08:28:25 <Shlomo_N> I think I can
08:28:26 <gsagie> multi vcpus
08:28:33 <gsagie> another thing worth testing
08:28:33 <Shlomo_N> sure
08:28:37 <Shlomo_N> good idea
08:28:44 <Shlomo_N> - Run this test using containers (the iPerf process will run on the host).
08:28:55 <Shlomo_N> -	Run the tests with strong VMs (max perf. will be 5Gb/s).
08:29:30 <gsagie> and maybe we should start investigating a bit about OVS performance tunning (we are not hitting the bottleneck yet but we do want some prefered deployment configurations
08:29:46 <gampel1> did you do l2 or l3 cross node ?
08:29:52 <gsagie> (we will also want this for redis or other DB and pub/sub mechanism that we will pick <-- this is something Yuli need to do)
08:29:54 <Shlomo_N> l2
08:30:03 <gsagie> so we need L3
08:30:13 <gampel1> we need to test l3 and see whats the diffrence
08:30:22 <gampel1> it should be teh same % as local
08:31:14 <Shlomo_N> sure, but the question is in which scenario
08:31:24 <gsagie> Shlomo_N: 4 VMs scenario
08:31:29 <gsagie> lets concentrate on thios
08:31:41 <gampel1> when we have an RC version of SG we need to add that to the testing
08:31:52 <Shlomo_N> ok
08:32:02 <gsagie> yes, this is why i think maybe at some point start working on automation for this
08:32:11 <gsagie> so we can fetch a specific patch and run the tests
08:32:13 <gampel1> I agree
08:32:21 <gsagie> so we can run this over night on several patches as well
08:32:27 <gsagie> or something like this
08:32:36 <gsagie> that does unstack/stack and etc..
08:33:10 <Shlomo_N> sure
08:33:16 <gampel1> Shlomo_N: I think that you should do the L3 test and start working on automation
08:33:24 <gsagie> yeah, L3 test first
08:33:29 <Shlomo_N> ok
08:34:12 <gsagie> okie
08:34:17 <gsagie> #topic open discussion
08:34:24 <gsagie> anyone want to share anything else?
08:34:28 <gsagie> or ask
08:34:38 <gampel1> Yuli can you update the status of your test
08:35:01 <yuli_s> sec,
08:35:15 <gampel1> I talked with dima and he have a python  test for redis that we could use
08:35:35 <yuli_s> in my last test I created in total 1000 subnets linked to 100 routers
08:35:56 <yuli_s> 100 routers - each one connected to 1 network and each one with 10 subnets
08:36:05 <yuli_s> the results were ok
08:36:14 <yuli_s> sec. will paste the results
08:36:43 <yuli_s> test3: 20 in 34.5030839443 avg: 1.72515419722
08:36:52 <yuli_s> test3: 100 in 172.239629984 avg: 1.72239629984
08:37:00 <yuli_s> test3: 500 in 861.065084934 avg: 1.72213016987
08:37:06 <yuli_s> test3: 1000 in 1721.52497792 avg: 1.72152497792
08:37:10 <Shlomo_N> is it seconds?
08:37:13 <gsagie> yuli_s: but for this we need to know what are the numbers for Neutron alone and how many workers you used
08:37:15 <yuli_s> yes,
08:37:23 <gsagie> whats the setup details, API workers
08:37:35 <yuli_s> it is a single box, default settings
08:37:45 <gsagie> we need to understand Dragonflow overhead, not Neutron
08:38:15 <yuli_s> ok
08:38:51 <gsagie> ok
08:38:55 <gsagie> anything else anyone?
08:40:11 <gsagie> okie, thanks for participating and see you all next week
08:40:17 <gampel1> Just think they are confused with the time change
08:40:30 <gsagie> ohhh, maybe its us that are confused
08:40:30 <gampel1> let me check
08:40:31 <gsagie> heh
08:40:35 <gampel1> yes
08:40:40 <gampel1> thats what i am checking
08:40:47 <gsagie> :) good point gampel1
08:41:16 <gampel1> ops the meeting starts in 20 min
08:41:32 <gampel1> >:o
08:41:38 <gsagie> we need to change the time then
08:41:48 <gsagie> its 12:00PM and oanson is hungry usually at this time
08:41:49 <gsagie> :)
08:42:22 <oanson> Technically, I am also hungry at 11:00
08:42:24 <gsagie> okie, lets update this for next week
08:42:40 <gampel1> what do you want to do, just update them and get the updates about SG, DNAT , redis on the channel
08:42:45 <gsagie> #action gsagie change meeting time to one hour earlier to accommodate oanson hunger needs
08:42:58 <gampel1> :)
08:43:09 <gsagie> gampel1: lets schedule a meeting to talk about controller reliability
08:43:19 <oanson> gsagie, you can also make sure I have a sandwich
08:43:23 <gsagie> and lets ask if anyone has any problem
08:43:26 <gsagie> with any patch
08:43:33 <gsagie> dingboopt still need to work on the exceptions
08:43:46 <gsagie> just maybe update everyone with the feature freeze time and stable release time
08:43:57 <gsagie> and need DuanKebo feedback on this
08:44:15 <gampel1> I will try to set a meeting for 13:00
08:44:22 <gsagie> gampel1: today?
08:44:40 <gsagie> armax :)
08:44:48 <gsagie> joining Dragonflow meeting.. :)
08:45:00 <gampel1> yes if it is good for you and omer
08:45:17 <oanson> Can we make it 13:30?
08:45:21 <gampel1> sure
08:45:25 <gsagie> okie great
08:45:38 <gsagie> if its problematic we can schedule for tommorow morning
08:45:57 <gampel1> Ok I will update DuanKebo on the time change confusion
08:46:01 <gsagie> dont forget the Hangzhou guys (dingboopt, Raofei) want to talk with them as well about DNAT and security groups
08:46:09 <gampel1> O k
08:46:09 <gsagie> and nick-ma
08:46:12 <gampel1> yes
08:46:12 <gsagie> of course
08:46:18 <gsagie> thanks gampel1 :)
08:46:43 <hshan> hi, all, about the DF reliability feature, anyone have suggestions?
08:46:58 <gsagie> May the Dragon sword always serve you well, but not for slaying Dragong
08:47:25 <gsagie> hshan: gampel1 is arranging a meeting because we had a clock change in Israel and we all were confused about the meeting
08:47:49 <hshan> okay, I see
08:48:29 <gsagie> sorry for that!
08:48:35 <gampel1> can we all meet later today in  dragonflow channel for 30 min
08:49:11 <gampel1> we had a time change  confusion
08:51:00 <gampel1> DuanKebo: can we meet at in 90 min or willit be too late
08:51:21 <gampel1> 10.30 UTC
08:51:36 <DuanKebo> It will be super time.
08:51:49 <gampel1> can we meet after super
08:52:18 <gampel1> let us know when it could be today or tomorrow
08:52:24 <DuanKebo> Yes, Is 11:30 UTC ok for you?
08:52:44 <gampel1> Yes for me
08:52:54 <gampel1> omer gal , how about you ?
08:52:55 <oanson> Fine by me
08:53:37 <oanson> Also fine by gsagie
08:53:39 <gampel1> gal: can you update nick-ma
08:54:02 <gampel1> ok so lets all meet in Df channel at 11:30 UTC
08:54:09 <gampel1> thx everybody
08:54:15 <nick-ma> Hi, just noticed the meeting time was changed.
08:55:24 <DuanKebo> OK
08:55:32 <gampel1> it was not it was our problem
08:55:49 <gampel1> nick-ma: can you meet us at 11:30 UTC in the DF channel ?
08:55:56 <nick-ma> Today?
08:55:59 <gampel1> ye s
08:56:14 <nick-ma> Sure.
08:56:20 <gampel1> great
08:56:51 <gsagie> #endmeeting