| *** jamesmcarthur has quit IRC | 00:01 | |
| *** tetsuro has joined #openstack-meeting | 00:09 | |
| *** armax has quit IRC | 00:09 | |
| *** jmasud has quit IRC | 00:13 | |
| *** jamesmcarthur has joined #openstack-meeting | 00:13 | |
| *** mlavalle has quit IRC | 00:15 | |
| *** jamesmcarthur has quit IRC | 00:16 | |
| *** jmasud has joined #openstack-meeting | 00:22 | |
| *** jmasud has quit IRC | 00:23 | |
| *** jmasud has joined #openstack-meeting | 00:24 | |
| *** jamesmcarthur has joined #openstack-meeting | 00:29 | |
| *** jamesmcarthur has quit IRC | 00:30 | |
| *** yasufum has joined #openstack-meeting | 00:34 | |
| *** Adri2000 has quit IRC | 00:37 | |
| *** jmasud_ has joined #openstack-meeting | 00:41 | |
| *** jmasud_ has quit IRC | 00:43 | |
| *** jmasud has quit IRC | 00:43 | |
| *** Adri2000 has joined #openstack-meeting | 00:45 | |
| *** jmasud has joined #openstack-meeting | 00:48 | |
| *** jmasud has quit IRC | 00:55 | |
| *** gyee has quit IRC | 01:19 | |
| *** jmasud has joined #openstack-meeting | 01:21 | |
| *** dmacpher_ has quit IRC | 01:27 | |
| *** dmacpher has joined #openstack-meeting | 01:27 | |
| *** armax has joined #openstack-meeting | 01:39 | |
| *** armax has quit IRC | 01:43 | |
| *** Liang__ has joined #openstack-meeting | 01:56 | |
| *** jamesmcarthur has joined #openstack-meeting | 01:59 | |
| *** jamesmcarthur has quit IRC | 02:10 | |
| *** apetrich has quit IRC | 02:10 | |
| *** armax has joined #openstack-meeting | 02:22 | |
| *** jmasud has quit IRC | 02:23 | |
| *** manuvakery has joined #openstack-meeting | 02:30 | |
| *** jamesmcarthur has joined #openstack-meeting | 02:51 | |
| *** ociuhandu has joined #openstack-meeting | 03:05 | |
| *** ociuhandu has quit IRC | 03:10 | |
| *** psachin has joined #openstack-meeting | 03:36 | |
| *** jamesmcarthur has quit IRC | 03:54 | |
| *** jamesmcarthur has joined #openstack-meeting | 03:55 | |
| *** rh-jelabarre has quit IRC | 04:03 | |
| *** jamesmcarthur has quit IRC | 04:12 | |
| *** jamesmcarthur has joined #openstack-meeting | 04:12 | |
| *** number80 has quit IRC | 04:13 | |
| *** jamesmcarthur has quit IRC | 04:16 | |
| *** jamesmcarthur has joined #openstack-meeting | 04:16 | |
| *** lifeless has quit IRC | 04:22 | |
| *** diablo_rojo has quit IRC | 04:30 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-meeting | 04:33 | |
| *** manuvakery has quit IRC | 04:40 | |
| *** jmasud has joined #openstack-meeting | 04:44 | |
| *** lifeless has joined #openstack-meeting | 04:52 | |
| *** psachin has quit IRC | 05:05 | |
| *** jmasud has quit IRC | 05:11 | |
| *** brinzhang_ has joined #openstack-meeting | 05:14 | |
| *** brinzhang has quit IRC | 05:14 | |
| *** dklyle has quit IRC | 05:25 | |
| *** links has joined #openstack-meeting | 05:31 | |
| *** jamesmcarthur has quit IRC | 05:32 | |
| *** jamesmcarthur has joined #openstack-meeting | 05:33 | |
| *** jamesmcarthur has quit IRC | 05:38 | |
| *** jmasud has joined #openstack-meeting | 05:39 | |
| *** lifeless has quit IRC | 05:41 | |
| *** links has quit IRC | 05:43 | |
| *** links has joined #openstack-meeting | 05:45 | |
| *** lifeless has joined #openstack-meeting | 05:50 | |
| *** brinzhang has joined #openstack-meeting | 05:56 | |
| *** brinzhang_ has quit IRC | 05:59 | |
| *** brinzhang_ has joined #openstack-meeting | 06:00 | |
| *** brinzhang_ has quit IRC | 06:02 | |
| *** brinzhang_ has joined #openstack-meeting | 06:02 | |
| *** brinzhang has quit IRC | 06:03 | |
| *** jamesmcarthur has joined #openstack-meeting | 06:09 | |
| *** Liang__ has quit IRC | 06:18 | |
| *** jamesmcarthur has quit IRC | 06:19 | |
| *** jmasud has quit IRC | 06:26 | |
| *** Liang__ has joined #openstack-meeting | 06:28 | |
| *** jawad_axd has joined #openstack-meeting | 06:31 | |
| *** brinzhang has joined #openstack-meeting | 06:40 | |
| *** brinzhang_ has quit IRC | 06:43 | |
| *** slaweq has joined #openstack-meeting | 06:45 | |
| *** belmoreira has joined #openstack-meeting | 06:50 | |
| *** ralonsoh has joined #openstack-meeting | 06:50 | |
| *** jamesmcarthur has joined #openstack-meeting | 06:54 | |
| *** apetrich has joined #openstack-meeting | 06:55 | |
| *** tetsuro_ has joined #openstack-meeting | 07:01 | |
| *** dmellado has quit IRC | 07:02 | |
| *** ociuhandu has joined #openstack-meeting | 07:04 | |
| *** tetsuro has quit IRC | 07:04 | |
| *** jamesmcarthur has quit IRC | 07:05 | |
| *** maciejjozefczyk has joined #openstack-meeting | 07:07 | |
| *** ociuhandu has quit IRC | 07:14 | |
| *** ociuhandu has joined #openstack-meeting | 07:14 | |
| *** ttsiouts has joined #openstack-meeting | 07:15 | |
| *** vishalmanchanda has joined #openstack-meeting | 07:25 | |
| *** ociuhandu has quit IRC | 07:27 | |
| *** ykatabam has quit IRC | 07:28 | |
| *** jmasud has joined #openstack-meeting | 07:34 | |
| *** ttsiouts has quit IRC | 07:37 | |
| *** witek has joined #openstack-meeting | 07:44 | |
| *** ttsiouts has joined #openstack-meeting | 07:54 | |
| *** masahito has joined #openstack-meeting | 07:58 | |
| *** ociuhandu has joined #openstack-meeting | 08:04 | |
| *** ociuhandu has quit IRC | 08:12 | |
| *** ociuhandu has joined #openstack-meeting | 08:15 | |
| *** dmellado has joined #openstack-meeting | 08:34 | |
| *** Lucas_Gray has joined #openstack-meeting | 08:35 | |
| *** jmasud has quit IRC | 08:39 | |
| *** jmasud has joined #openstack-meeting | 08:40 | |
| *** number80 has joined #openstack-meeting | 08:49 | |
| *** jamesmcarthur has joined #openstack-meeting | 09:02 | |
| *** jamesmcarthur has quit IRC | 09:06 | |
| *** ociuhandu has quit IRC | 09:10 | |
| *** ociuhandu has joined #openstack-meeting | 09:11 | |
| *** ociuhandu has quit IRC | 09:14 | |
| *** ociuhandu has joined #openstack-meeting | 09:14 | |
| *** tetsuro_ has quit IRC | 09:21 | |
| *** links has quit IRC | 09:22 | |
| *** links has joined #openstack-meeting | 09:24 | |
| *** jraju__ has joined #openstack-meeting | 09:29 | |
| *** links has quit IRC | 09:30 | |
| *** tetsuro has joined #openstack-meeting | 09:31 | |
| *** masahito has quit IRC | 09:32 | |
| *** brinzhang has quit IRC | 09:35 | |
| *** brinzhang has joined #openstack-meeting | 09:35 | |
| *** yasufum has quit IRC | 09:39 | |
| *** ociuhandu has quit IRC | 09:53 | |
| *** ociuhandu has joined #openstack-meeting | 09:53 | |
| *** ociuhandu has quit IRC | 09:58 | |
| *** ociuhandu has joined #openstack-meeting | 09:59 | |
| *** Liang__ has quit IRC | 10:15 | |
| *** tetsuro has quit IRC | 10:31 | |
| *** tetsuro has joined #openstack-meeting | 10:39 | |
| *** jraju__ has quit IRC | 10:55 | |
| *** janders has joined #openstack-meeting | 11:00 | |
| *** raildo has joined #openstack-meeting | 11:02 | |
| *** ociuhandu has quit IRC | 11:03 | |
| *** jamesmcarthur has joined #openstack-meeting | 11:03 | |
| *** jmasud has quit IRC | 11:04 | |
| *** ociuhandu has joined #openstack-meeting | 11:04 | |
| *** jmasud has joined #openstack-meeting | 11:05 | |
| *** Lucas_Gray has quit IRC | 11:08 | |
| *** ociuhandu has quit IRC | 11:11 | |
| *** brinzhang_ has joined #openstack-meeting | 11:14 | |
| *** andrebeltrami has joined #openstack-meeting | 11:17 | |
| *** brinzhang has quit IRC | 11:17 | |
| *** janders has quit IRC | 11:17 | |
| *** jraju__ has joined #openstack-meeting | 11:18 | |
| *** ociuhandu has joined #openstack-meeting | 11:19 | |
| *** brinzhang_ has quit IRC | 11:20 | |
| *** brinzhang_ has joined #openstack-meeting | 11:20 | |
| *** jamesmcarthur has quit IRC | 11:22 | |
| *** tetsuro has quit IRC | 11:23 | |
| *** ociuhandu has quit IRC | 11:27 | |
| *** jmasud has quit IRC | 11:32 | |
| *** JangwonLee__ has joined #openstack-meeting | 11:43 | |
| *** Luzi has joined #openstack-meeting | 11:44 | |
| *** JangwonLee_ has quit IRC | 11:47 | |
| *** brinzhang has joined #openstack-meeting | 11:55 | |
| *** brinzhang_ has quit IRC | 11:58 | |
| *** raildo has quit IRC | 12:03 | |
| *** rh-jelabarre has joined #openstack-meeting | 12:06 | |
| *** raildo has joined #openstack-meeting | 12:06 | |
| *** vishalmanchanda has quit IRC | 12:17 | |
| *** ttsiouts has quit IRC | 12:27 | |
| *** ociuhandu has joined #openstack-meeting | 12:30 | |
| *** lpetrut has joined #openstack-meeting | 12:32 | |
| *** ociuhandu has quit IRC | 12:40 | |
| *** hemna has quit IRC | 13:02 | |
| *** hemna has joined #openstack-meeting | 13:03 | |
| *** ociuhandu has joined #openstack-meeting | 13:06 | |
| *** njohnston is now known as njohnston|pto | 13:07 | |
| *** jamesdenton has joined #openstack-meeting | 13:15 | |
| *** jamesmcarthur has joined #openstack-meeting | 13:19 | |
| *** ttsiouts has joined #openstack-meeting | 13:19 | |
| *** ttsiouts has quit IRC | 13:24 | |
| *** jamesmcarthur has quit IRC | 13:26 | |
| *** ttsiouts has joined #openstack-meeting | 13:27 | |
| *** andrebeltrami has quit IRC | 13:33 | |
| *** jamesmcarthur has joined #openstack-meeting | 13:35 | |
| *** vishalmanchanda has joined #openstack-meeting | 13:41 | |
| *** TrevorV has joined #openstack-meeting | 13:41 | |
| *** Liang__ has joined #openstack-meeting | 13:50 | |
| *** Liang__ is now known as LiangFang | 13:52 | |
| *** liuyulong has joined #openstack-meeting | 13:52 | |
| *** Luzi has quit IRC | 13:58 | |
| liuyulong | test | 13:59 |
|---|---|---|
| liuyulong | #startmeeting neutron_l3 | 14:00 |
| openstack | Meeting started Wed May 20 14:00:41 2020 UTC and is due to finish in 60 minutes. The chair is liuyulong. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| *** openstack changes topic to " (Meeting topic: neutron_l3)" | 14:00 | |
| openstack | The meeting name has been set to 'neutron_l3' | 14:00 |
| ralonsoh | hi | 14:01 |
| liuyulong | hi | 14:01 |
| maciejjozefczyk | Hey! | 14:01 |
| liuyulong | Hi | 14:02 |
| liuyulong | OK, let's start | 14:02 |
| liuyulong | #topic Announcements | 14:02 |
| *** openstack changes topic to "Announcements (Meeting topic: neutron_l3)" | 14:02 | |
| slaweq | hi | 14:03 |
| liuyulong | #link https://etherpad.opendev.org/p/neutron-victoria-ptg | 14:03 |
| liuyulong | Only a few days, we will begin the V virtual ptg online. | 14:03 |
| liuyulong | Maybe I should add something that was mentioned in Shanghai for a further discussion. | 14:05 |
| liuyulong | #link https://doodle.com/poll/xcd82ea6svs5dben | 14:05 |
| liuyulong | From the time slot here, I should have only 2 hours to meet you guys all at the same time online. | 14:06 |
| liuyulong | Each day | 14:06 |
| liuyulong | Anyway, again topics are welcomed to be added to the list. | 14:07 |
| liuyulong | More announcements please have a look at: | 14:08 |
| liuyulong | #link http://eavesdrop.openstack.org/meetings/networking/2020/networking.2020-05-19-14.00.log.html#l-12 | 14:08 |
| liuyulong | OK, no more from me then. | 14:08 |
| liuyulong | OK, let's move on | 14:10 |
| liuyulong | #topic Bugs | 14:10 |
| *** openstack changes topic to "Bugs (Meeting topic: neutron_l3)" | 14:10 | |
| liuyulong | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-May/014913.html | 14:10 |
| liuyulong | The bug list from our deputy last week. | 14:10 |
| liuyulong | First one | 14:11 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1877977 | 14:11 |
| openstack | Launchpad bug 1877977 in neutron "[DVR] Recovery from openvswitch restart fails when veth are used for bridges interconnection" [Medium,Confirmed] | 14:11 |
| liuyulong | I want to mention one more action | 14:12 |
| liuyulong | change the config option and run "kill -SIGHUP <ovs-agent-pid>", this should help to recovery the dataplane? | 14:13 |
| ralonsoh | I think the idea was to deprecate veth pairs for bridge connectivity | 14:14 |
| ralonsoh | and use only patch ports | 14:14 |
| *** ttsiouts has quit IRC | 14:15 | |
| liuyulong | And typically, when the openvswitch (vswitchd) is restarted, the canary flows will be lose. | 14:15 |
| liuyulong | The ovs-agent monitor this and will consider it should run a full sync. | 14:16 |
| liuyulong | Everything should be added back again. | 14:16 |
| liuyulong | If not, this should be a bug absolutely. | 14:17 |
| ralonsoh | it should, by the ovs agent | 14:17 |
| *** dklyle has joined #openstack-meeting | 14:17 | |
| *** brinzhang_ has joined #openstack-meeting | 14:18 | |
| liuyulong | More information about this is we use the systemd to do the next step from the bug "restart the neutron-ovs-agent." | 14:18 |
| liuyulong | If the openvswitch is restarted, the ovs-agent will run restart action automatically. | 14:19 |
| *** ayoung has joined #openstack-meeting | 14:19 | |
| liuyulong | OK, next one | 14:20 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1878299 | 14:20 |
| openstack | Launchpad bug 1878299 in neutron "[floatingip port_forwarding] changing external port to used value hangs with retriable exception" [Medium,Confirmed] | 14:20 |
| *** brinzhang has quit IRC | 14:21 | |
| liuyulong | I did not reproduce the bug locally, but haleyb has confirmed the bug. | 14:21 |
| *** brinzhang_ has quit IRC | 14:22 | |
| *** brinzhang_ has joined #openstack-meeting | 14:22 | |
| liuyulong | Looks like some DB related issue when the 'port' check is running. | 14:23 |
| ralonsoh | I need to check with haleyb what is really happening and how I can reproduce it | 14:24 |
| ralonsoh | in order to try code a fix | 14:24 |
| liuyulong | ralonsoh, thank you. I just run a quick test locally, not see the such issue. | 14:27 |
| haleyb | ralonsoh: hi, sorry, in another meeting | 14:27 |
| liuyulong | OK, let's try to dig this locally in details. | 14:28 |
| liuyulong | Next | 14:28 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1877301 | 14:29 |
| openstack | Launchpad bug 1877301 in neutron "[RFE] L3 Router support ndp proxy" [Wishlist,Confirmed] | 14:29 |
| liuyulong | We have discussed this in last L3 meeting, and it was approved during last drivers meeting. | 14:30 |
| liuyulong | yangjianfeng has submitted the spec | 14:30 |
| liuyulong | #link https://review.opendev.org/#/c/728628/ | 14:31 |
| liuyulong | So we can continue the discussion in this spec. | 14:31 |
| liuyulong | Next one: | 14:32 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1879215 | 14:32 |
| openstack | Launchpad bug 1879215 in neutron "[L3] Unexcepted HA router scheduled instance shows up after manully scheduling" [Undecided,New] | 14:32 |
| liuyulong | This was reported by me, it is easy to reproduce. | 14:33 |
| liuyulong | These action is useful when the operators want to move the network node offline. | 14:34 |
| maciejjozefczyk | Looks like we're relying on some old data during --enable operation. | 14:34 |
| liuyulong | The existing router resources (scheduled routers) should be moved from one to another. | 14:34 |
| maciejjozefczyk | The priority of this bug isn't to big, right? I could try to reproduce it and triage. | 14:36 |
| liuyulong | maciejjozefczyk, yes, it's not big. But it breaks the config limitation. | 14:38 |
| liuyulong | my colleague found the problem is related to this function. https://github.com/openstack/neutron/blob/master/neutron/db/l3_agentschedulers_db.py#L507 | 14:38 |
| liuyulong | maciejjozefczyk, I will be cool if you can confirm if it is also existing in master branch. | 14:39 |
| liuyulong | d/I | 14:39 |
| liuyulong | I will submit the fix after we check and test it really works locally. | 14:40 |
| maciejjozefczyk | liuyulong, yes I'll try to check if this exists in master. | 14:40 |
| maciejjozefczyk | ok, liuyulong | 14:40 |
| liuyulong | OK, no more bugs from me now. | 14:41 |
| liuyulong | Any updates? | 14:41 |
| * liuyulong scanning the LP bug list... | 14:42 | |
| liuyulong | OK, I see something related to L3. | 14:43 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1878905 | 14:43 |
| openstack | Launchpad bug 1878905 in neutron "when I use dvr mode for neutron,it's not functional" [Undecided,Incomplete] | 14:43 |
| maciejjozefczyk | Yeah, the bug is not clear to me. Looks like the guy deployed the env without DVR and then switched it to DVR? Hard to tell... | 14:44 |
| liuyulong | maciejjozefczyk has marked it as incomplete. Yes, I will do same thing. | 14:44 |
| *** brinzhang_ has quit IRC | 14:45 | |
| liuyulong | maciejjozefczyk, exactly, I have the same question. | 14:45 |
| maciejjozefczyk | yeah, what about the bridge configuration on computes, perhaps its missing, I dunno. | 14:46 |
| liuyulong | And the config option should be pasted to the bug for the team to analyse. | 14:46 |
| liuyulong | ++ | 14:46 |
| maciejjozefczyk | But as per Kolla cores they have the test covering DVR scenario. | 14:46 |
| *** lpetrut has quit IRC | 14:46 | |
| maciejjozefczyk | I'll ask for inputs from computes and neutron config | 14:46 |
| liuyulong | Yes, we need some clue to do the trouble shooting, not guess... : ) | 14:47 |
| maciejjozefczyk | ;) | 14:48 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1877978 | 14:48 |
| openstack | Launchpad bug 1877978 in neutron "SNAT Problem Floating IP" [Undecided,Incomplete] | 14:48 |
| liuyulong | This should be another "unfriendly bug report"... | 14:49 |
| maciejjozefczyk | :) | 14:49 |
| *** mlavalle has joined #openstack-meeting | 14:49 | |
| liuyulong | I have such encounter such phenomenon, you guys just guess... | 14:50 |
| *** ayoung has quit IRC | 14:50 | |
| liuyulong | #link https://docs.openstack.org/neutron/pike/contributor/policies/bugs.html#bug-report-template | 14:51 |
| liuyulong | I guess for the bug deputy, you guys could easily paste this link to the bug. | 14:51 |
| *** priteau has joined #openstack-meeting | 14:52 | |
| maciejjozefczyk | liuyulong++ ok, I'll paste | 14:52 |
| liuyulong | Sorry, it is not the latest. | 14:52 |
| * liuyulong is looking for it now... | 14:52 | |
| *** ayoung has joined #openstack-meeting | 14:52 | |
| liuyulong | #link https://docs.openstack.org/neutron/latest/contributor/policies/bugs.html#bug-report-template | 14:53 |
| *** belmoreira has quit IRC | 14:54 | |
| liuyulong | OK, no more from me. (really this time) | 14:54 |
| liuyulong | Actually no OVN related bugs were reported in last 2 weeks, so let's move to the last topic (we are running out of time). | 14:56 |
| liuyulong | #topic On demand agenda | 14:56 |
| *** openstack changes topic to "On demand agenda (Meeting topic: neutron_l3)" | 14:56 | |
| liuyulong | mlavalle, FYI, yangjianfeng has submitted the spec of the IPv6 ndp proxy: https://review.opendev.org/#/c/728628/ | 14:57 |
| mlavalle | liuyulong: ack. Thanks | 14:58 |
| mlavalle | in my pile | 14:58 |
| *** munimeha1 has joined #openstack-meeting | 14:58 | |
| mlavalle | liuyulong: does he work with you? | 14:58 |
| liuyulong | mlavalle, actually yangjianfeng is my former colleague. He has moved to a new position in another company. | 14:59 |
| *** ociuhandu has quit IRC | 14:59 | |
| mlavalle | liuyulong: ack | 14:59 |
| liuyulong | For the ipv6 ndp_proxy, in china telecom we have implemented something similar to this. | 15:00 |
| liuyulong | But we introduce a new resource, IPv6Address (something like floating IPv6), to routers to enable the ndp_proxy | 15:00 |
| *** ociuhandu has joined #openstack-meeting | 15:00 | |
| liuyulong | It is mainly for productization of our public cloud | 15:01 |
| liuyulong | OK, time is up... | 15:01 |
| liuyulong | Thank you guys. | 15:01 |
| liuyulong | #endmeeting | 15:01 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 15:01 | |
| openstack | Meeting ended Wed May 20 15:01:57 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:01 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-05-20-14.00.html | 15:02 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-05-20-14.00.txt | 15:02 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-05-20-14.00.log.html | 15:02 |
| maciejjozefczyk | \o/ | 15:02 |
| *** ociuhandu has quit IRC | 15:05 | |
| *** jamesmcarthur has quit IRC | 15:11 | |
| *** jamesmcarthur has joined #openstack-meeting | 15:12 | |
| *** LiangFang has quit IRC | 15:15 | |
| *** ttsiouts has joined #openstack-meeting | 15:18 | |
| *** ociuhandu has joined #openstack-meeting | 15:24 | |
| *** priteau has quit IRC | 15:33 | |
| *** yasufum has joined #openstack-meeting | 15:34 | |
| *** yasufum has quit IRC | 15:38 | |
| *** yasufum has joined #openstack-meeting | 15:39 | |
| *** ttsiouts has quit IRC | 15:40 | |
| *** ttsiouts has joined #openstack-meeting | 15:40 | |
| *** gyee has joined #openstack-meeting | 15:40 | |
| *** ttsiouts has quit IRC | 15:40 | |
| *** priteau has joined #openstack-meeting | 15:47 | |
| *** rcernin has quit IRC | 16:04 | |
| *** witek has quit IRC | 16:12 | |
| *** jraju__ has quit IRC | 16:20 | |
| *** ayoung has quit IRC | 16:23 | |
| *** jmasud has joined #openstack-meeting | 16:24 | |
| *** ayoung has joined #openstack-meeting | 16:26 | |
| *** andrebeltrami has joined #openstack-meeting | 16:32 | |
| *** alistarle has joined #openstack-meeting | 16:36 | |
| *** jamesmcarthur has quit IRC | 16:36 | |
| *** ociuhandu has quit IRC | 16:37 | |
| *** jamesmcarthur has joined #openstack-meeting | 16:39 | |
| *** ociuhandu has joined #openstack-meeting | 16:41 | |
| *** ociuhandu has quit IRC | 16:54 | |
| *** liuyulong has quit IRC | 17:00 | |
| *** jmasud has quit IRC | 17:00 | |
| *** yasufum has quit IRC | 17:06 | |
| *** jmasud has joined #openstack-meeting | 17:08 | |
| *** alistarle has quit IRC | 17:08 | |
| *** jamesmcarthur has quit IRC | 17:24 | |
| *** moguimar has quit IRC | 17:24 | |
| *** moguimar has joined #openstack-meeting | 17:24 | |
| *** ociuhandu has joined #openstack-meeting | 17:24 | |
| *** raildo has quit IRC | 17:25 | |
| *** raildo has joined #openstack-meeting | 17:25 | |
| *** priteau has quit IRC | 17:28 | |
| *** ociuhandu has quit IRC | 17:37 | |
| *** ayoung has quit IRC | 17:38 | |
| *** ayoung has joined #openstack-meeting | 17:40 | |
| *** jamesmcarthur has joined #openstack-meeting | 17:45 | |
| *** ayoung has quit IRC | 17:51 | |
| *** jamesmcarthur has quit IRC | 17:51 | |
| *** ayoung has joined #openstack-meeting | 17:53 | |
| *** ralonsoh has quit IRC | 17:54 | |
| *** jamesmcarthur has joined #openstack-meeting | 18:00 | |
| *** jmasud has quit IRC | 18:04 | |
| *** e0ne has quit IRC | 18:06 | |
| *** ociuhandu has joined #openstack-meeting | 18:07 | |
| *** munimeha1 has quit IRC | 18:18 | |
| *** jamesmcarthur has quit IRC | 18:40 | |
| *** jamesmcarthur has joined #openstack-meeting | 18:41 | |
| *** jmasud has joined #openstack-meeting | 19:08 | |
| *** ayoung has quit IRC | 19:15 | |
| *** e0ne has joined #openstack-meeting | 19:17 | |
| *** ociuhandu has quit IRC | 19:23 | |
| *** jmasud has quit IRC | 19:23 | |
| *** jmasud has joined #openstack-meeting | 19:29 | |
| *** e0ne has quit IRC | 19:44 | |
| *** jmasud has quit IRC | 20:04 | |
| *** jmasud has joined #openstack-meeting | 20:08 | |
| *** jmasud has quit IRC | 20:12 | |
| *** jamesmcarthur has quit IRC | 20:19 | |
| *** jamesmcarthur has joined #openstack-meeting | 20:19 | |
| *** maciejjozefczyk has quit IRC | 20:30 | |
| *** vishalmanchanda has quit IRC | 20:37 | |
| *** larainema has quit IRC | 20:38 | |
| *** patchbot has joined #openstack-meeting | 20:51 | |
| timburke | #startmeeting swift | 21:00 |
| openstack | Meeting started Wed May 20 21:00:34 2020 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| *** openstack changes topic to " (Meeting topic: swift)" | 21:00 | |
| openstack | The meeting name has been set to 'swift' | 21:00 |
| timburke | who's here for the swift meeting? | 21:00 |
| mattoliverau | o/ | 21:00 |
| rledisez | hi o/ | 21:00 |
| kota_ | o/ | 21:00 |
| tdasilva | o/ | 21:01 |
| *** gyee has quit IRC | 21:01 | |
| timburke | agenda's at https://wiki.openstack.org/wiki/Meetings/Swift | 21:02 |
| timburke | #topic ptg | 21:02 |
| *** openstack changes topic to "ptg (Meeting topic: swift)" | 21:02 | |
| timburke | it's only a week and a half away! | 21:02 |
| timburke | thanks everybody for adding topics to the etherpad | 21:03 |
| timburke | #link https://etherpad.opendev.org/p/swift-ptg-victoria | 21:03 |
| timburke | i made sure that the link was included on http://ptg.openstack.org/etherpads.html | 21:03 |
| timburke | if you haven't already, please do register -- i expect it'll help with logistics/planning | 21:04 |
| timburke | #link https://www.eventbrite.com/e/virtual-project-teams-gathering-june-2020-tickets-103456996662 | 21:04 |
| *** rfolco|rover is now known as rfolco|rover|off | 21:04 | |
| timburke | mattoliverau did a great job of making sure we had some timeslots booked for video conferencing | 21:05 |
| timburke | the schedule of those is up at http://ptg.openstack.org/ptg.html | 21:05 |
| timburke | as well as the top of the etherpad | 21:05 |
| *** gyee has joined #openstack-meeting | 21:06 | |
| timburke | one thing i think i'd like to try is having everyone come to the ptg with a few patches they'd really like to see some progress on -- pick three or so and add them (and your name) to the priority reviews page! | 21:07 |
| timburke | #link https://wiki.openstack.org/wiki/Swift/PriorityReviews | 21:07 |
| timburke | any questions/comments on the ptg? | 21:08 |
| mattoliverau | Nope, looking forward to it :) | 21:08 |
| timburke | on to new business :-) | 21:09 |
| timburke | #topic ratelimit + s3api | 21:09 |
| *** openstack changes topic to "ratelimit + s3api (Meeting topic: swift)" | 21:09 | |
| timburke | so after digging out from the massive pile of async pendings, i wanted to make sure it didn't happen again, at least not easily. and one easy way to do that is to limit how quickly writes happen in a cluster | 21:10 |
| timburke | fortunately, we have a ratelimit middleware! unfortunately, it could be a little annoying to deploy with s3api | 21:11 |
| *** jamesmcarthur has quit IRC | 21:12 | |
| *** jamesmcarthur has joined #openstack-meeting | 21:12 | |
| timburke | (my understanding is) you usually want to place it left of auth -- that auth decision may be expensive, and you don't want to have auth fall down because of an over-eager swift client | 21:12 |
| timburke | but if it's an s3 request, you won't have the full account/container path until *after* auth | 21:13 |
| timburke | so i'm thinking that having it twice (once before s3api and once after auth) might be a reasonable way to go? how do other people deploy that? | 21:14 |
| timburke | #link https://review.opendev.org/729051 | 21:14 |
| patchbot | patch 729051 - swift - ratelimit: Allow multiple placements - 3 patch sets | 21:14 |
| *** jamesmcarthur has quit IRC | 21:15 | |
| *** jamesmcarthur has joined #openstack-meeting | 21:15 | |
| timburke | rledisez, surely you've got *something* for this, yeah? is it a custom ratelimiter? | 21:15 |
| rledisez | right now (when we enabled it) we put it left of auth and s3api, but we don't have much s3 requests. and by default we do not use it | 21:16 |
| rledisez | timburke: nothing custom. we don't ratelimit, we scale :P | 21:16 |
| rledisez | joke aside, we enable this only when really necessary, it's pretty rare | 21:17 |
| kota_ | excellent | 21:17 |
| timburke | oh, cool! never mind then ;-) | 21:17 |
| rledisez | the usual situation is the "delete storm" | 21:17 |
| timburke | fwiw, following https://review.opendev.org/#/c/697535/ we can have ratelimit right of s3api and auth and still serve "reasonable" responses to s3 clients | 21:18 |
| patchbot | patch 697535 - swift - s3api: Better handle 498/429 responses (MERGED) - 1 patch set | 21:18 |
| timburke | ...but i quickly realized that it'd throw off my metrics, since AWS sends out 503s -- i want a way to easily differentiate between 503 (slow down) and 503 (backend failed) | 21:20 |
| timburke | which led to https://review.opendev.org/729092 | 21:20 |
| patchbot | patch 729092 - swift - s3api: Log ratelimited requests as 498s - 2 patch sets | 21:20 |
| timburke | idk how sane of a thing that is to do though -- it feels a little dirty lying in logs like that | 21:21 |
| timburke | so i also started thinking about returning some other error code -- i don't know of any s3 clients that would retry a 498, but the rfc-compliant 429 seemed to get awscli to retry at least | 21:23 |
| timburke | #link https://review.opendev.org/729093 | 21:23 |
| patchbot | patch 729093 - swift - s3api: Add config option to return 429s on ratelimit - 1 patch set | 21:23 |
| timburke | rledisez, kota_: any opinion on which approach seems better/more reasonable? | 21:23 |
| timburke | (could even do both, i suppose; if configured, log & return 429, otherwise log 498 but return 503) | 21:24 |
| rledisez | we need to log the same code that what is returned to customer (otherwise we can't discuss the SLA: I got 10% 503, I only see 2%) | 21:25 |
| kota_ | sounds reeasonable but what i thought when looking the log change patch a little is we may leave 503 logging for debug due. | 21:25 |
| rledisez | so I would say we need to reeturn something difference | 21:25 |
| kota_ | s/due/for user support/ | 21:26 |
| kota_ | i didn't dig it in detail yet, just quick look. | 21:26 |
| timburke | *nod* makes sense. i suppose i ought to dig in more to see how well other s3 clients support 429 | 21:27 |
| rledisez | I know that I would for sure only use the "429" option you describe, seems the best option (and retry is not that bad I guess, it does not consume much resources) | 21:27 |
| kota_ | user might say, "hey I got 503s" so 503s reported in the swift logs helps us to debug. | 21:28 |
| timburke | all right, on to updates! | 21:29 |
| timburke | #topic lots of small files | 21:29 |
| *** openstack changes topic to "lots of small files (Meeting topic: swift)" | 21:29 | |
| timburke | rledisez, alecuyer how's it going? | 21:30 |
| rledisez | So alecuyer is off, he told me few days ago that some tests were passing. I guess he worked on it this week but to be honest I don't know much. We saw a change in diskfile that need to be backoprted to LOSF (a new parameter to _finalize_durable from a recent patch of you timburke) | 21:31 |
| rledisez | that's it | 21:31 |
| timburke | 👍 sorry for the extra trouble ;-) | 21:32 |
| *** jamesmcarthur has quit IRC | 21:32 | |
| timburke | #topic database reclaim locking | 21:33 |
| *** openstack changes topic to "database reclaim locking (Meeting topic: swift)" | 21:33 | |
| *** jamesmcarthur has joined #openstack-meeting | 21:33 | |
| timburke | i was hoping clayg would be around to discuss his findings on https://review.opendev.org/#/c/727876/ | 21:33 |
| patchbot | patch 727876 - swift - Breakup reclaim into batches - 4 patch sets | 21:33 |
| timburke | but i could take a stab at it :D | 21:34 |
| timburke | the current code breaks up the deleted namespace into batches of ~1k and reaps them in a loop, taking getting a fresh connection/lock each time | 21:35 |
| *** jamesmcarthur has quit IRC | 21:35 | |
| timburke | and it seems to be working well! the reclaim still takes a while, but the server could continue writing deletes at a decent clip while it was happening | 21:36 |
| timburke | i think tests should pass now, and it's ready for review! i might have to ask clay for his testing setup though | 21:37 |
| timburke | has anyone else had a chance to take a look at it yet? | 21:38 |
| rledisez | not yet | 21:38 |
| timburke | no worries. it's ready when you are :-) | 21:40 |
| timburke | that's all i had | 21:40 |
| timburke | #topic open discussion | 21:40 |
| *** openstack changes topic to "open discussion (Meeting topic: swift)" | 21:40 | |
| timburke | anything else we should discuss? | 21:40 |
| *** jamesmcarthur has joined #openstack-meeting | 21:42 | |
| timburke | i wonder whether clayg's reclaim patch will make https://review.opendev.org/#/c/571917/ and https://review.opendev.org/#/c/724943/ unnecessary... | 21:42 |
| patchbot | patch 571917 - swift - Manage async_pendings priority per containers - 5 patch sets | 21:42 |
| patchbot | patch 724943 - swift - WIP: Batched updates for object-updater - 2 patch sets | 21:42 |
| clayg | one of the comments on the patch has a gist for the script i'm using to do reclaim in one thread while inserting tombstones in another | 21:43 |
| timburke | all right, we oughta let mattoliverau and kota_ start their day ;-) | 21:45 |
| rledisez | I'm not sure that it makes them unecessary. they are just other tools in the toolbox to reduce the container-listing lag | 21:45 |
| timburke | thank you all for coming, and thank you for working on siwft! | 21:45 |
| timburke | #endmeeting | 21:45 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 21:45 | |
| openstack | Meeting ended Wed May 20 21:45:45 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:45 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-05-20-21.00.html | 21:45 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-05-20-21.00.txt | 21:45 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-05-20-21.00.log.html | 21:45 |
| *** mlavalle has quit IRC | 21:53 | |
| *** raildo has quit IRC | 21:56 | |
| *** raildo_ has joined #openstack-meeting | 21:56 | |
| *** mlavalle has joined #openstack-meeting | 22:07 | |
| *** klindgren has quit IRC | 22:09 | |
| *** slaweq has quit IRC | 22:13 | |
| *** slaweq has joined #openstack-meeting | 22:21 | |
| *** slaweq has quit IRC | 22:25 | |
| *** rh-jelabarre has quit IRC | 22:26 | |
| *** rh-jelabarre has joined #openstack-meeting | 22:27 | |
| *** jamesmcarthur has quit IRC | 22:29 | |
| *** jamesmcarthur has joined #openstack-meeting | 22:30 | |
| *** TrevorV has quit IRC | 22:35 | |
| *** slaweq has joined #openstack-meeting | 22:39 | |
| *** slaweq has quit IRC | 22:44 | |
| *** jamesmcarthur has quit IRC | 22:52 | |
| *** jmasud has joined #openstack-meeting | 22:52 | |
| *** jamesmcarthur has joined #openstack-meeting | 22:53 | |
| *** jamesmcarthur has quit IRC | 22:56 | |
| *** jamesmcarthur has joined #openstack-meeting | 22:56 | |
| *** jmasud has quit IRC | 23:03 | |
| *** raildo_ has quit IRC | 23:03 | |
| *** jmasud has joined #openstack-meeting | 23:05 | |
| *** jamesmcarthur has quit IRC | 23:07 | |
| *** rcernin has joined #openstack-meeting | 23:08 | |
| *** jmasud has quit IRC | 23:09 | |
| *** jamesmcarthur has joined #openstack-meeting | 23:14 | |
| *** jamesmcarthur has quit IRC | 23:20 | |
| *** jamesmcarthur has joined #openstack-meeting | 23:20 | |
| *** jamesmcarthur has quit IRC | 23:25 | |
| *** ociuhandu has joined #openstack-meeting | 23:38 | |
| *** ociuhandu has quit IRC | 23:43 | |
| *** mlavalle has quit IRC | 23:43 | |
| *** gyee has quit IRC | 23:48 | |
| *** jamesmcarthur has joined #openstack-meeting | 23:59 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!