18:17:25 <mmichelson> #startmeeting ovn_community_development_discussion 18:17:26 <openstack> Meeting started Thu Nov 5 18:17:25 2020 UTC and is due to finish in 60 minutes. The chair is mmichelson. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:17:27 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 18:17:29 <openstack> The meeting name has been set to 'ovn_community_development_discussion' 18:17:39 <ryzhyk> hi! 18:17:49 <mmichelson> Hi everyone 18:17:52 <numans> Hi 18:17:54 <dceara> Hi 18:17:55 <imaximets> hi 18:18:03 <mmichelson> Looks like we have a lot of people here today! 18:18:08 <mmichelson> Who would like to go first? 18:18:31 <blp> I can go. 18:18:54 <blp> I have posted two versions of the final DDlog series. 18:19:03 <blp> Numan has provided some feedback on it, and I will post v3 soon. 18:19:16 <flaviof> hi all 18:19:32 <blp> I think that's my report, but perhaps there are questions. 18:20:02 <flaviof> blp I got a question/ask... 18:20:09 <blp> flaviof: go ahead 18:20:38 <flaviof> blp would you be ok helping me with this change on northd using ddlog: 18:21:14 <blp> Oh, yes, in general I'm available to help with that kind of question. 18:21:27 <flaviof> https://patchwork.ozlabs.org/project/ovn/patch/20201103221834.25541-2-flavio@flaviof.com/ 18:22:11 <flaviof> I will take tomorrow to learn more about ddlog and use that as an excuse to learn ddlog northd :) 18:22:27 <blp> flaviof: I guess this must have been posted to the list. I'll take a look at it. 18:22:46 <numans> blp, dceara found one problem while compiling ovn with ddlog - http://paste.openstack.org/show/799751/. Can you plz take a look. If we configure ovs disabling avx512, ovn compilation succeeds - http://paste.openstack.org/show/799752/ 18:24:25 <blp> numans: Wow, that is mysterious at first glance. How do I reproduce the original failure? 18:24:44 <numans> blp, its reproducible with ovs master and ovn + ddlog patches. 18:25:26 <numans> dceara and I tried on fedora. I'm not sure if there are any distro/package related stuff. 18:25:26 <blp> numans: That's how I build ovs, so the question is really "what stands out about how dceara is building ovs and/or ovn"? 18:25:28 <dceara> blp: I think it's great that DDlog made it on the mailing list! As numans pointed out, I just tried building OVS + OVN master branch + ddlog on Fedora 32 on a machine with AVX-512. 18:26:02 <imaximets> blp, dceara seems to have avx512 support in compiler/cpu. 18:26:24 <dceara> imaximets: right, only then libopenvswitchavx512 gets compiled 18:26:26 <blp> dceara: So that patch is one that triggers the problem? 18:27:19 <blp> dceara: I mean, building ovs master with http://paste.openstack.org/show/799752/ makes the problem show in ovn? 18:27:40 <numans> blp, no. building ovs with http://paste.openstack.org/show/799752/ resolves the problem. 18:27:49 <dceara> blp: No, that's the only way to make it compile, i.e., to remove AVX support in OVS so that the rust compiler doesn't get libopenvswitchavx512 passed as argument. 18:28:04 <imaximets> blp, looking at line 'process didn't exit successfully:' it seems that '-L' is missing before the path to libopenvswitchavx512.la. 18:28:45 <dceara> imaximets: I'm not sure, I see '-L /mnt/alt_home/numans/workspace_cpp/ovs/lib/.libs' 18:28:46 <blp> dceara: Oh, I see, I didn't realize that avx512 support had been added to ovs. I understand now. 18:29:24 <imaximets> dceara, yep, but -L should be before /mnt/alt_home/numans/workspace_cpp/ovs/lib/libopenvswitchavx512.la too. 18:29:46 <blp> -L is for a directory but a .la file is not a directory. 18:30:33 <blp> OK, I'll have a look at this. I need to pull the latest ovs master, I guess I'm behind. 18:31:06 <imaximets> blp, oh, sorry. Ignore my comments then. 18:31:20 <blp> I'm also using Fedora for my dev machine FWIW. 18:31:38 <dceara> blp: One more thing, on the same system 16 OVN e2e tests were failing: 65 67 73 81 91 101 145 175 189 207 270 275 281 292 326 348 423 18:31:56 <dceara> blp: I can share more details on the mailing list. 18:32:30 <blp> OK, please do. 18:32:56 <dceara> blp: ack. 18:34:06 <blp> OK, so I have my to-do list, then. Who's next? 18:34:15 <mmichelson> I can jump in if it's OK 18:34:21 <flaviof> blp thanks for the testing helpers (4afe409e9) super cleaner .at tests now! 18:34:31 <flaviof> mmichelson: go ahead 18:34:35 <blp> mmichelson: You're in charge of the meeting, of course you can go head. 18:34:58 <numans> blp++ for the testing helpers. 18:35:04 <mmichelson> I checked the calendar, and we're supposed to have soft freeze for ovn 20.12 tomorrow. I think that it kind of snuck up on us. So instead, how would people feel about pushing it out one additional week to the 13th? 18:35:20 <dceara> mmichelson: ++ 18:35:23 <numans> ++ 18:35:40 <mmichelson> To correct this for 2021, I'm adding calendar reminders two weeks in advance of soft freeze dates so I can send announcements out with plenty of warning 18:35:47 <dceara> (although Friday 13th..) 18:35:56 <blp> flaviof: you're welcome for the helpers! 18:36:15 <blp> mmichelson: good idea 18:36:57 <mmichelson> OK cool 18:37:38 <flaviof> dceara: it will not be full moon. Hope that is ok 18:37:50 <flaviof> mmichelson ++ 18:37:54 <mmichelson> In addition to that, I'm working on a patch that will allow for users to choose which CT zone to use for SNAT on gateway routers. I made a simpler version of the patch last week to be tested by some RH people in OpenShift that would allow for the default CT zone (0) to be used 18:38:12 <mmichelson> But this new version is a bit more complicated since it needs to ensure that we don't have CT zone conflicts when the user chooses their zone. 18:38:34 <mmichelson> And I would love to have feedback on v3 of the unit test patch series. 18:38:38 <mmichelson> I think that's about it for m.e 18:39:38 <ihrachys> can I? 18:40:21 <ihrachys> two things. first is I am low key sitting on v11 of multiple controllers / virtual chassis patch so reviews are welcome: https://patchwork.ozlabs.org/project/ovn/patch/20201102231419.539520-1-ihrachys@redhat.com/ probably numans if possible since he already checked it the last time 18:40:41 <numans> ihrachys, ack. 18:41:08 <numans> ihrachys, I'll take a look. thanks for v11 18:41:09 <ihrachys> and the other is, above in the chat I was asking about why CFI flag is forced to 1 and used as a marker for vlan tag present. I suspect there's some back story that I should be aware of. 18:41:30 <blp> ihrachys: This is in OVS? It's because that's how the Linux kernel did it at the time. 18:41:37 <ihrachys> since I am going to touch the flow in another patch 18:42:05 <ihrachys> blp: vlan.present transformed into tci==0x1000/0x1000 mask check, yes. 18:42:21 <ihrachys> so the flag was overloaded with other meaning different from protocol? 18:42:45 * numans waves and disappears. bye 18:43:26 <blp> ihrachys: Yes, because in Ethernet this bit was always required to be zero. OVS runs on Ethernet. Then the Ethernet folks changed their mind. https://en.wikipedia.org/wiki/IEEE_802.1Q#cite_note-8 18:44:18 <ihrachys> oh I was reading the page but missed the bottom note *facepalm* 18:44:31 <ihrachys> thanks for the tip, that's all from me :) 18:46:02 <blp> Anyone else? 18:46:04 <dceara> I can go next if that's OK. 18:46:16 <imaximets> blp, the conversation around following patch might shed some light on why some things around avx512 in ovs done as they are done: https://patchwork.ozlabs.org/project/openvswitch/patch/1594759760-20400-1-git-send-email-gvrose8192@gmail.com/ Just in case. 18:47:02 <blp> imaximets: Thanks for the background. 18:48:00 <imaximets> dceara, sorry, go ahead. 18:48:01 <dceara> We had a customer issue come over this week (from OpenStack) where we were hitting the 4K resubmit limit in OVS (again). We had partially fixed this in the past and now we fixed all remaining cases (I hope). 18:48:41 <blp> I can hardly imagine the performance is great if OVS is doing >4000 resubmits. 18:49:06 <dceara> Well, it's actually not true what I said above, there's still traffic that might hit the multicast group that floods to a lot of ports. 18:50:29 <blp> Anyone else? 18:50:54 <imaximets> I have a small update. 18:51:32 <flaviof> #link https://github.com/ovn-org/ovn/commit/8c6a5bc21847dab8ccbe18ab1e4b563ddca13379 dceara 18:51:43 <flaviof> imaximets: pls go ahead 18:51:50 <imaximets> 2 weeks ago I talked about debugging memory consumption of ovsdb-server. It turned out to be few issues combined. 18:52:04 <imaximets> 1. raft log could be huge in memory. 18:52:31 <imaximets> 2. backlogs on raft connections could grow significantly 18:52:47 <imaximets> 3. libc doesn't return all that memory back to system. 18:53:16 <imaximets> There was a patch set to address all these issues. 3 of 5 patches already merged. 18:53:35 <imaximets> Last 2 patches reviewed and will be merged soon. 18:53:40 <blp> That's all good news! 18:54:09 <imaximets> I'm also backporting all these fixes down to 2.13 as we agreed on a list. 18:55:08 <blp> Anything else? 18:55:15 <blp> I have to go in 5 minutes. 18:55:35 <imaximets> That's all from me. 18:58:52 <blp> OK, thanks a lot everyone, I have my to-do list for today. Talk to you next week! 18:59:09 <flaviof> bye all! 18:59:13 <dceara> bye! 18:59:17 <imaximets> bye. 19:00:26 <mmichelson> #endmeeting