17:18:07 <numans> #startmeeting ovn_community_development_discussion
17:18:08 <openstack> Meeting started Thu Sep  3 17:18:07 2020 UTC and is due to finish in 60 minutes.  The chair is numans. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:18:09 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
17:18:11 <openstack> The meeting name has been set to 'ovn_community_development_discussion'
17:18:30 <numans> Who wants to go firat
17:18:34 <numans> s/firat/first
17:19:05 <zhouhan> I can go first
17:19:10 <numans> sure
17:19:35 <zhouhan> I improved the nb_cfg mechanism with timestamp which measures end to end latency more accurately
17:20:39 <zhouhan> With that, I retested the incremental flow installation patch series, and can measure the actual latency improvement (in addition to CPU cost).
17:20:44 <_lore_> hi all
17:21:29 <zhouhan> I could also spin up 3k HVs with 30k lports in scale test, with only 15 farm nodes, thanks to all the ovn-controller CPU improvements.
17:21:49 <zhouhan> with this env, the processing time of both ovn-northd and all ovn-controllers, the end to end latency is reduced by 30%.
17:22:05 <zhouhan> If considering ovn-controllers only (the total time spent on processing SB lflow change by all HVs), the latency reduced by 60%.
17:22:40 <numans> zhouhan, that's great.
17:22:45 <flaviof> zhouhan: does that use the private_chassis table? Is that relevant in your test?
17:22:56 <numans> zhouhan, does your testing also include your ofctrl I-P patches ?
17:22:58 <zhouhan> And now the total e2e latency is 1.5 seconds, which is not bad considering the scale 3K HVs and 30k lports
17:23:18 <zhouhan> flaviof: yes, private_chassis table, with timestamp
17:23:32 <flaviof> zhouhan: nice!
17:23:58 <zhouhan> numans: yes, the 30% and 60% improvement I mentioned is for the ofctrl I-P patches.
17:24:14 <numans> ok
17:24:35 <blp> I have a new 3990X desktop that I'm going to use for benchmarking OVN. It should do a good job.
17:24:44 <zhouhan> numans: please take a look when you have time. #link https://patchwork.ozlabs.org/project/openvswitch/list/?series=197009
17:25:03 <numans> zhouhan, ack. If I understand you store the timestamp in the db right ?
17:25:06 <numans> blp, cool.
17:25:36 <zhouhan> For the nb_cfg improvements, it is: #link https://patchwork.ozlabs.org/project/ovn/list/?series=198962
17:26:17 <zhouhan> I also submitted a patch that optionally avoids checking lsp_is_up for programming ARP responder flows
17:27:07 <zhouhan> I think in most cases we don't need this check, and it would largely reduce the cost of the control plane, for port creation and binding - around half of the cost
17:27:50 <numans> zhouhan, ok. So the option is added per ls ? or lsp ? or globally ?
17:28:09 <numans> I didn't see the patch yet
17:28:20 <zhouhan> numans: globally
17:28:42 <numans> zhouhan, ok. sounds good
17:29:01 <zhouhan> numans: the patch is here #link https://patchwork.ozlabs.org/project/ovn/patch/1599099225-113525-1-git-send-email-hzhou@ovn.org/
17:29:07 <numans> ack
17:29:42 <zhouhan> I am also thinking of adding support for directly specifying port-binding chassis from NB, to avoid port-binding updates from chassises. I think this may be useful for some use cases, probably k8s? (at least it is useful for me)
17:30:05 <zhouhan> That's my update :)
17:30:29 <numans> Thanks. Who wants to go next
17:31:05 <flaviof> zhouhan: specifying port-binding chassis from NB instead of .... ?
17:31:21 <Ankur1> Can i go next?
17:31:32 <zhouhan> oh, I forgot one thing. For e2e latency 1.5 sec, 1 sec is in ovn-northd.
17:31:49 <zhouhan> Thanks blp for the update on DDlog progress
17:32:13 <zhouhan> I think it is the next biggest improvement for the e2e latency
17:32:37 <zhouhan> flaviof: instead of updating from chassis to SB.
17:32:51 <zhouhan> sorry Ankur1. Please go ahead.
17:33:04 <flaviof> zhouhan: ack. Ankur1 sorry for the interrupt
17:34:00 <Ankur1> Thanks a lot zhouhan. No problem at all flaviof.
17:34:00 <Ankur1> Just wanted to highlight a couple of problem statements we are working on and wanted to get some opinion on one of the solutions.
17:34:29 <Ankur1> a. We are getting plenty of scenarios where there is a need to support multiple distributed gateway router ports per logical router.
17:34:53 <Ankur1> Does not look like a trivial scenario to solve, but just wanted to highlight we have started some efforts around this.
17:35:43 <blp> zhouhan: I'm embarrassed that I haven't had progress updates in a while.
17:36:36 <Ankur1> b. Flushing of CT zones: As of now the logic is tied with the existence of datapath. However, there are scenarios where datapath is still existent but CT zone should be flushed (for example if just remove all the NAT rules from a router).
17:37:57 <Ankur1> For b. i submitted a patch some time back, however patch was handling the LB case, hence we abandoned it.
17:37:58 <Ankur1> As of now, to make CT FLUSH by ZONE more generic, we are thinking if we can add something like CT_FLUSH_BY <ZONE, CT_MARK>, where CT_MARK indicates if a CT entry is because of NAT or LB (or some other feature).
17:38:37 <Ankur1> Its the CT_FLUSH_BY <ZONE, CT_MARK> where wanted to seek some opinions and thoughts here.
17:38:41 <numans> Ankur1, for LB, we already set ct_label[1]
17:38:51 <numans> ct_label.natted is the logical field
17:39:11 <Ankur1> Cool, let us say CT_FLUSH_BY <ZONE, CT_LABEL>..
17:40:19 <numans> Ankur1, I'm not sure If I'm following what CT_FLUSH_By <..> would do.
17:41:22 <numans> May be you can send an email to the ML about your proposal and we could discuss there.
17:41:43 <numans> thanks for the efforts for all these features.
17:42:20 <Ankur1> Sure, what i mean is that as of now we have CT_FLUSH_BY <ZONE>, and we are thinking if we make is more granular CT_FLUSH_BY <ZONE, LABEL>, where LABEL identifies a feature.
17:42:20 <Ankur1> We want to achieve following:
17:42:21 <Ankur1> a. Trigger CT ZONE flush through config detach/attach rather than datapath delete/add.
17:42:21 <Ankur1> b. Do CT ZONE FLUSH by feature, so that we are flushing the whole zone, but rather just for a feature, like NAT, LB etc.
17:42:41 <Ankur1> Sure, i will send out more details in ML. Just wanted to provide a summary here :)
17:42:49 <Ankur1> Thats all from my side.
17:42:50 <numans> Ankur1, cool.
17:43:30 <zhouhan> thanks Ankur1. a) sounds reasonable. I tried to do it once, but it seemed tricky in the code. So I instead worked around it by creating multiple routers and peering them with static routes. It would be great if the code can be improved to support the use case.
17:43:56 <zhouhan> blp: no worries, as long as it is in progress :) numans and I wondered if we need a temporary I-P for northd before DDlog version is ready, which was why I wanted to know where we are in the DDlog progress first.
17:45:01 <zhouhan> numans: do you want to talk about your idea, since blp is here today
17:45:56 <blp> numans: Please tell me about it.
17:46:13 <numans> My idea was to add a simple I-P engine which would not recompute for unnecessary db changes to start with - like nb_cfg change.
17:46:56 <blp> So, here's what needs to happen for the DDlog OVN:
17:47:01 <blp> 1. Benchmark.
17:47:09 <blp> 2. Update to whatever hasn't made it in.
17:47:23 <blp> That's all, I think.
17:47:41 <blp> It would be easier to maintain if folks were OK with merging it before #1 and #2 happened.
17:47:47 <blp> #2 in particular is a treadmill.
17:48:34 <numans> blp, by merging you mean replacing the c version ? or we have both in parallel ?
17:48:47 <blp> Having both in parallel.
17:48:51 <zhouhan> blp: yes, I feel #2 is hard, too, since ovn-northd is still changing very often.
17:49:01 <blp> That's the way it is in the current fork: both versions get built.
17:49:13 <numans> It sounds good to me.
17:49:32 <numans> And we can switch over to ddlog as default when we have (1) comparable to 'C'.
17:49:32 <blp> OK, then I will plan to post patches in the next few weeks.
17:50:05 <zhouhan> Sounds good to me, too, as long as there are no major issues (except the feature gap)
17:50:16 <blp> The feature gap is really just whatever has been added in the last few months.
17:51:11 <numans> zhouhan, blp when we have feature gap addressed, we can have the patch submitters to update both the versions
17:51:25 <zhouhan> I assume the benchmark should be better than C, according to some very basic testing I did last year (and now a year past)
17:51:28 <numans> otherwise we will always have delta between the two
17:52:10 <blp> I am sure that some tuning and iteration will be needed, because this is the biggest DDlog program that has been written, much bigger than any other.
17:52:19 <zhouhan> "when we have feature gap addressed" - maybe this is the hardest part. How to achieve that/
17:53:03 <numans> zhouhan, may be we can dedicate the next release to ddlog ?
17:53:11 <numans> after branching for 20.09
17:53:27 <zhouhan> blp: last year my test showed it 90% more efficient than the C version. Even without further tuning, I assume it shouldn't degrade too much after adding more features recently, right?
17:53:46 <blp> zhouhan: You are probably right.
17:54:10 <zhouhan> numans: I am ok with that.
17:54:45 <zhouhan> would a feature freeze for ovn-northd helpful?
17:54:58 <blp> zhouhan: At some point, yes.
17:55:22 <numans> zhouhan, I actually meant that. once we branch for 20.09, we can work on closing the gaps and until then no new features.
17:55:59 <zhouhan> numans: sounds good to me, but not sure about the broad community :)
17:56:21 <numans> zhouhan, we can discuss on this in ML.
17:56:36 <zhouhan> sure
17:57:08 <numans> blp, thanks for all the efforts
17:57:15 <flaviof> blp: hi! stupid q on ddlog: rust dependency will be part of ovn, but not needed for ovs codebase at all, right? I hear the compiling will take longer with rust. Is that noticeable to you?
17:57:33 <blp> flaviof: OVS will not need rust.
17:57:52 <blp> flaviof: For now, it will be an optional dependency for OVN. Without Rust, you won't get the DDlog version.
17:58:32 <flaviof> ack. #link http://www.openvswitch.org/support/ovscon2019/day1/1553-OVS-OVN'19%20-%20DDlog%20in%20OVN.pdf ovscon talk on ddlog changes
17:58:41 * zhouhan have to drop off in 1 min, will read your discussion offline
17:58:43 <flaviof> blp: thanks!
17:58:45 <blp> flaviof: Compiling definitely takes noticeably longer. Most of the time, incremental compilation is not much slower, but the first build takes longer.
17:59:15 <flaviof> blp: fair enough. it is one time cost vs runtime gains. a good trade
17:59:47 <numans> zhouhan, bye
18:00:20 <numans> blp, anything else on the ddlog topic ? If not, I can go next real quick
18:00:38 <flaviof> blp on next ovscon I anticipate an update on that talk that Mark+Dumitru gave. ;)
18:00:40 <blp> numans: That is all I have.
18:00:51 <numans> thanks.
18:01:03 <numans> I can go next real quick.
18:01:14 <numans> I submitted the patch series to cache lflow expr matches/expr tree.
18:01:20 <numans> #link https://patchwork.ozlabs.org/project/ovn/list/?series=199144
18:01:41 <numans> I was behind reviews. I started reviewing some of the patches.
18:01:52 <numans> I also submitted a patch to handle cluster db upgrades for run_xx_ovsdb ovn-ctl commands.
18:02:02 <numans> #link https://patchwork.ozlabs.org/project/ovn/patch/20200903130400.1971690-1-numans@ovn.org/
18:02:10 <numans> Request to take a look at these patches.
18:02:15 <numans> That's all from me.
18:03:59 <numans> who wants to go next ?
18:04:06 <_lore_> I can go next?
18:04:11 <_lore_> very quick
18:04:27 <_lore_> I posted some ovs/ovn patch for review
18:04:43 <_lore_> ovs: I added some debug code for raft
18:05:07 <_lore_> ovn:
18:05:14 <_lore_> - dhcp decline msg support for dhcp
18:05:44 <_lore_> - running ovs_db with valgrind/strace
18:06:14 <_lore_> - a fix for IPv6 empty_lp controller_event
18:06:40 <_lore_> now I restarted working on ovn-scale code
18:06:50 <_lore_> that's all from my side, thx
18:09:00 <numans> _lore_, thanks.
18:10:25 <numans> anyone else ?
18:10:33 <numans> If not we can probably end the meeting.
18:11:17 <flaviof> I was on pto last week; not much to report other than small nits on ovn-fake-multinode
18:11:18 <flaviof> #link https://github.com/ovn-org/ovn-fake-multinode/pull/36
18:11:25 <flaviof> that is all from me :)
18:11:47 <numans> flaviof, thanks for the PR.
18:12:31 <numans> Ok. Everyone. Let's end the meeting.
18:12:31 <numans> Bye
18:12:37 <flaviof> bye all!
18:12:41 <numans> #endmeeting