18:15:21 #startmeeting ovn_community_development_discussion 18:15:22 Meeting started Thu Dec 17 18:15:21 2020 UTC and is due to finish in 60 minutes. The chair is mmichelson. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:15:23 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:15:25 The meeting name has been set to 'ovn_community_development_discussion' 18:15:55 Hi everyone 18:16:08 hi! 18:16:10 Tomorrow is the proposed release date for 20.12.0 18:16:12 Hi 18:16:25 So I have sent out the commits just before this meeting started 18:16:37 That way we can get it released early-ish (for the western hemisphere) tomorrow 18:17:20 In other news, I reworked my unit test patches so that rather than trying to introduce a framework that requires instrumenting source files, it just is a small refactor of some IPAM code and addition of tests using ovstest 18:17:22 mmichelson: +1 18:17:56 blp, I'm glad you're here today :) The release of 20.12.0 means we should be in position to merge DDLog to master whenever you think you're ready. 18:18:05 mmichelson: That is good news. 18:18:06 I understand you're currently working on a new version of your patches 18:18:28 I see that Dumitru passed along some more reviews. 18:18:31 Would you want to get the latest version posted or would you rather us merge what currently exists? 18:18:35 I think that my job is then: 18:18:43 - Rebase and deal with whatever changes have happened. 18:18:50 - Address Dumitru's feedback. 18:19:13 If the changes are relatively small based on those two things, then I'd be inclined to merge it. 18:19:22 If they need more work, then I'd be inclined to post a v10. 18:19:29 What do you think? 18:19:54 And post-merge there are at least two jobs left: 18:19:59 - Keep up-to-date with changes. 18:20:13 - Run benchmarks and iterate on making it faster and scale better. 18:20:26 blp: The last patch of the series depends on the IDL split changes (which need a rebase). I'm not sure if there are more changes needed in the IDL patches, but with the current version quite a few OVN tests were failing. 18:20:27 I think it was Numan who passed along a useful benchmark script. 18:20:37 Oh, crap. 18:20:41 blp, for the rebase step, I assume you're not going to aim for feature parity, right? 18:20:56 mmichelson: I've kept up with feature parity so far. 18:21:16 blp, ah, ok. I just know that some of the changes (such as the datapath groups change) could be kind of big. 18:21:41 I'll rebase the IDL split changes then. 18:22:00 Last time I did that I did not see any test failures, but maybe some were introduced aftewrad. 18:22:16 It's hard to keep rebasing a big series against a constantly shifting base. 18:22:21 yes it is 18:22:41 that's why I was wondering if we should just merge what we have and then work directly in master from this point? 18:23:00 That way, devs introducing new features to northd can introduce to C and DDLog simultaneously 18:23:15 If the IDL split changes can get into OVS, then we can do that. 18:23:16 And we only have to chase that gap that currently exists from between your last posting and now 18:23:29 Without the IDL split changes, ovn+ddlog won't compile. 18:23:36 got it 18:23:38 I agree with the philosophy otherwise though. 18:23:44 blp, I'm going to review idl-split patch-set tomorrow. 18:23:53 imaximets: Thanks! That will enable the rest. 18:24:06 imaximets: I'll try to get the rebase of it done today, then. 18:24:17 blp: for the post-merge task "keep up-to-date with changes", is it assumed that the developer who submit the change should keep the DDlog part up-to-date? 18:24:41 zhouhan: I would hope so, but I am also available to help, either to eduate developers or to help doing the porting. 18:24:55 blp: sounds good! 18:25:19 blp, ok. One quick comment: can we re-name reconnect_try_receive to reconnect_receive_attempt ? Current name sounds like we're requesting it to receive something. 18:25:38 imaximets: OK. 18:25:47 blp, thanks! 18:27:01 imaximets: I'll fold that into my rebase then. 18:27:17 anyway, I kind of transitioned the conversation to ddlog, but to be clear I also was done talking about the things I had worked on. 18:27:34 I don't think I have anything else. 18:28:08 9 18:28:12 oops wrong window 18:28:32 I have a topic to discuss. 18:29:27 last time I talked about this patch: https://patchwork.ozlabs.org/project/openvswitch/patch/20201211205447.3874314-1-i.maximets@ovn.org/ for ovsdb-server. 18:30:04 imaximets: yes 18:30:10 basically, it changes the format of file transactions that stored in database file and in raft logs to contain diffs of columns. 18:31:16 recent tests shows decrease by 90% of memory consumption for northbound database for ovn-k8s tests, 40% for southbound datbase and overall 40% reduction in disk IO. 18:31:22 wow! 18:31:33 blp, zhouhan: it'll be great if you can review it. 18:31:47 imaximets: ok! 18:32:06 main problem, I guess, is the upgrade process, since ovd ovsdb-server will not be able to read new database format. 18:32:10 I'll put it on my to-do list. 18:32:40 that is also a problem if we'll have different versions of ovsdb-server within same raft cluster. 18:32:44 blp, thanks! 18:33:11 Why is that a problem for upgrades? I would expect that to be a problem for downgrades, which are still worth considering but not in the same way. 18:34:20 It's because if raft leader will send new append request to old follower that will be a problem. 18:34:29 Oh, OK, I see, we're changing not just the on-disk format but the format that gets sent around the raft cluster. 18:34:40 yep 18:35:17 I also worked on adding memory reports to ovn processes today: https://patchwork.ozlabs.org/project/ovn/patch/20201217181427.639377-1-i.maximets@ovn.org/ 18:35:35 that's it from my side. 18:35:54 Maybe this format parameter should be part of the database definition, so that the cluster has to agree on what format to use from the start. 18:36:19 Then, the format would only be upgraded once all the cluster members can handle it. 18:37:11 blp, problem is that we must be sure that new old server will not be connected to a cluster once format changed. 18:37:33 blp, and also old ovsdb-server will not understand that something is wrong. 18:38:02 can ovsdb raft cluster size change during normal operations ? 18:38:15 imaximets: The format upgrade would have to happen after the software upgrade was finished. 18:38:40 panda: OVSDB supports adding and removing raft cluster members, yes. 18:39:17 panda: And, of course, cluster members can fail and come back at runtime. I don't know whether you consider failures to be normal operations. 18:39:20 blp, I added a cmdline option to disable new format. After upgrade user will be able to enable it via unixctl or be re-starting without the option. 18:39:38 *by 18:40:35 but I'm not sure if that is a best way to handle this. 18:40:37 imaximets: That's kind of an admin-intensive way to deal with it. The admin will have to make sure to use the right command-line options during upgrade, and then turn them off afterward. 18:41:13 imaximets: It might be harder to automate than telling people to upgrade the database format once they have finished their server upgrade. 18:41:17 imaximets: I'm not an admin though. 18:43:39 blp, I can't think of a correct automatic way that will work in all cases. But I'd like to discuss ideas on a mail-list. 18:43:46 imaximets: OK. 18:44:19 cool, y'all. Anybody else? 18:45:36 I just wanted to mention that since yesterday OVN CI runs with AddressSanitizer enabled for clang (memleak detection too) on GitHub. Thanks imaximets for the suggestion and review! 18:45:45 That's it on my side, thanks! 18:47:53 AddressSanitizer is a big help. Glad to see it in use for CI. 18:48:27 Anyone else? 18:49:54 I guess that's it 18:50:09 Thanks everyone! 18:50:21 #endmeeting