| *** yasufum has joined #openstack-meeting | 00:04 | |
| *** yasufum_ has joined #openstack-meeting | 00:06 | |
| *** yasufum has quit IRC | 00:08 | |
| *** yasufum_ is now known as yasufum | 00:08 | |
| *** diurnalist has quit IRC | 00:13 | |
| *** jmasud has quit IRC | 00:17 | |
| *** jmasud has joined #openstack-meeting | 00:20 | |
| *** andrebeltrami has quit IRC | 00:38 | |
| *** yamamoto has joined #openstack-meeting | 00:46 | |
| *** jmasud has quit IRC | 00:48 | |
| *** yamamoto has quit IRC | 00:49 | |
| *** yamamoto has joined #openstack-meeting | 00:58 | |
| *** diurnalist has joined #openstack-meeting | 01:08 | |
| *** Liang__ has joined #openstack-meeting | 01:17 | |
| *** markvoelker has joined #openstack-meeting | 01:20 | |
| *** markvoelker has quit IRC | 01:24 | |
| *** jmasud has joined #openstack-meeting | 01:32 | |
| *** gyee has quit IRC | 01:35 | |
| *** yaawang has quit IRC | 01:42 | |
| *** ricolin_ has joined #openstack-meeting | 01:43 | |
| *** yaawang has joined #openstack-meeting | 01:43 | |
| *** Liang__ has quit IRC | 01:46 | |
| *** diurnalist has quit IRC | 02:07 | |
| *** hyunsikyang has joined #openstack-meeting | 02:18 | |
| *** jmasud has quit IRC | 02:45 | |
| *** yaawang has quit IRC | 02:45 | |
| *** yaawang has joined #openstack-meeting | 02:46 | |
| *** yasufum has quit IRC | 02:50 | |
| *** jmasud has joined #openstack-meeting | 02:53 | |
| *** yasufum has joined #openstack-meeting | 02:59 | |
| *** yasufum has quit IRC | 03:13 | |
| *** armax has quit IRC | 03:19 | |
| *** rfolco has quit IRC | 03:20 | |
| *** ricolin_ is now known as ricolin | 03:30 | |
| *** psachin has joined #openstack-meeting | 03:33 | |
| *** Liang__ has joined #openstack-meeting | 03:58 | |
| *** markvoelker has joined #openstack-meeting | 04:03 | |
| *** diurnalist has joined #openstack-meeting | 04:05 | |
| *** yasufum has joined #openstack-meeting | 04:05 | |
| *** markvoelker has quit IRC | 04:08 | |
| *** yasufum has quit IRC | 04:14 | |
| *** Lucas_Gray has quit IRC | 04:30 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-meeting | 04:33 | |
| *** apetrich has quit IRC | 04:43 | |
| *** moguimar has joined #openstack-meeting | 04:44 | |
| *** mugsie has quit IRC | 04:53 | |
| *** mugsie has joined #openstack-meeting | 04:57 | |
| *** markvoelker has joined #openstack-meeting | 05:03 | |
| *** strigazi has quit IRC | 05:04 | |
| *** diurnalist has quit IRC | 05:06 | |
| *** strigazi has joined #openstack-meeting | 05:06 | |
| *** markvoelker has quit IRC | 05:07 | |
| *** psahoo has joined #openstack-meeting | 05:21 | |
| *** vishalmanchanda has joined #openstack-meeting | 05:23 | |
| *** links has joined #openstack-meeting | 05:34 | |
| *** yaawang has quit IRC | 05:45 | |
| *** yaawang has joined #openstack-meeting | 05:45 | |
| *** strigazi has quit IRC | 05:46 | |
| *** njohnston has quit IRC | 06:18 | |
| *** yasufum has joined #openstack-meeting | 06:26 | |
| *** tetsuro has joined #openstack-meeting | 06:33 | |
| *** tetsuro has quit IRC | 06:43 | |
| *** tetsuro has joined #openstack-meeting | 06:43 | |
| *** tetsuro has quit IRC | 06:43 | |
| *** yaawang has quit IRC | 06:43 | |
| *** yaawang has joined #openstack-meeting | 06:51 | |
| *** tetsuro has joined #openstack-meeting | 06:55 | |
| *** yaawang has quit IRC | 06:55 | |
| *** yaawang has joined #openstack-meeting | 06:55 | |
| *** tetsuro has quit IRC | 06:56 | |
| *** psahoo has quit IRC | 07:00 | |
| *** psahoo has joined #openstack-meeting | 07:00 | |
| *** rcernin has quit IRC | 07:02 | |
| *** markvoelker has joined #openstack-meeting | 07:04 | |
| *** rcernin has joined #openstack-meeting | 07:05 | |
| *** markvoelker has quit IRC | 07:09 | |
| *** slaweq has quit IRC | 07:12 | |
| *** slaweq has joined #openstack-meeting | 07:13 | |
| *** markvoelker has joined #openstack-meeting | 07:18 | |
| *** markvoelker has quit IRC | 07:23 | |
| *** rcernin has quit IRC | 07:30 | |
| *** Liang__ has quit IRC | 07:30 | |
| *** Liang__ has joined #openstack-meeting | 07:31 | |
| *** ralonsoh has joined #openstack-meeting | 07:33 | |
| *** ociuhandu has quit IRC | 07:41 | |
| *** yasufum has quit IRC | 07:48 | |
| *** rcernin has joined #openstack-meeting | 07:50 | |
| *** ttsiouts has joined #openstack-meeting | 07:53 | |
| *** ttsiouts has quit IRC | 07:53 | |
| *** ttsiouts has joined #openstack-meeting | 07:54 | |
| *** e0ne has joined #openstack-meeting | 07:58 | |
| *** rcernin has quit IRC | 08:06 | |
| *** ociuhandu has joined #openstack-meeting | 08:20 | |
| *** ttsiouts has quit IRC | 08:37 | |
| *** ttsiouts has joined #openstack-meeting | 08:38 | |
| *** yasufum has joined #openstack-meeting | 08:39 | |
| *** ttsiouts has quit IRC | 08:42 | |
| *** yaawang has quit IRC | 08:46 | |
| *** yaawang has joined #openstack-meeting | 08:46 | |
| *** apetrich has joined #openstack-meeting | 08:48 | |
| *** yasufum has quit IRC | 08:51 | |
| *** yasufum has joined #openstack-meeting | 08:56 | |
| *** ttsiouts has joined #openstack-meeting | 09:27 | |
| *** yaawang has quit IRC | 09:30 | |
| *** yaawang has joined #openstack-meeting | 09:31 | |
| *** Lucas_Gray has joined #openstack-meeting | 09:33 | |
| *** yamamoto has quit IRC | 09:37 | |
| *** moguimar has quit IRC | 09:41 | |
| *** moguimar has joined #openstack-meeting | 09:41 | |
| *** yasufum has quit IRC | 09:42 | |
| *** ricolin has quit IRC | 09:52 | |
| *** yamamoto has joined #openstack-meeting | 09:55 | |
| *** yamamoto has quit IRC | 10:06 | |
| *** yamamoto has joined #openstack-meeting | 10:07 | |
| *** apetrich has quit IRC | 10:12 | |
| *** moguimar has quit IRC | 10:17 | |
| *** moguimar has joined #openstack-meeting | 10:17 | |
| *** moguimar has quit IRC | 10:19 | |
| *** moguimar has joined #openstack-meeting | 10:19 | |
| *** moguimar has joined #openstack-meeting | 10:20 | |
| *** Liang__ has quit IRC | 10:40 | |
| *** janders has joined #openstack-meeting | 10:52 | |
| *** njohnston has joined #openstack-meeting | 11:01 | |
| *** priteau has joined #openstack-meeting | 11:02 | |
| *** janders has quit IRC | 11:03 | |
| *** rajivmucheli has joined #openstack-meeting | 11:04 | |
| *** verdurin has joined #openstack-meeting | 11:04 | |
| *** janders has joined #openstack-meeting | 11:05 | |
| *** kevinz has quit IRC | 11:11 | |
| *** markvoelker has joined #openstack-meeting | 11:15 | |
| *** ykatabam has quit IRC | 11:15 | |
| *** ykatabam has joined #openstack-meeting | 11:16 | |
| *** markvoelker has quit IRC | 11:19 | |
| *** raildo has joined #openstack-meeting | 11:28 | |
| *** janders has quit IRC | 11:30 | |
| *** apetrich has joined #openstack-meeting | 11:45 | |
| *** rh-jelabarre has joined #openstack-meeting | 11:50 | |
| *** csatari has quit IRC | 11:53 | |
| *** csatari has joined #openstack-meeting | 11:56 | |
| *** Lucas_Gray has quit IRC | 11:58 | |
| *** rajivmucheli has quit IRC | 11:59 | |
| *** Lucas_Gray has joined #openstack-meeting | 12:00 | |
| *** rajivmucheli has joined #openstack-meeting | 12:01 | |
| *** andrebeltrami has joined #openstack-meeting | 12:06 | |
| *** alecuyer has quit IRC | 12:12 | |
| *** psahoo_ has joined #openstack-meeting | 12:12 | |
| *** psahoo has quit IRC | 12:16 | |
| *** irclogbot_1 has quit IRC | 12:16 | |
| *** irclogbot_3 has joined #openstack-meeting | 12:17 | |
| *** JangwonLee_ has joined #openstack-meeting | 12:31 | |
| *** rajivmucheli has quit IRC | 12:35 | |
| *** trident has quit IRC | 12:46 | |
| *** trident has joined #openstack-meeting | 12:49 | |
| *** ricolin has joined #openstack-meeting | 12:54 | |
| *** rfolco has joined #openstack-meeting | 13:00 | |
| *** manuvakery has joined #openstack-meeting | 13:03 | |
| *** yasufum has joined #openstack-meeting | 13:10 | |
| *** yasufum has quit IRC | 13:14 | |
| *** yasufum has joined #openstack-meeting | 13:15 | |
| *** abhishekk has quit IRC | 13:22 | |
| *** jhesketh has quit IRC | 13:22 | |
| *** abhishekk has joined #openstack-meeting | 13:23 | |
| *** jhesketh has joined #openstack-meeting | 13:24 | |
| *** Lucas_Gray has quit IRC | 13:27 | |
| *** Lucas_Gray has joined #openstack-meeting | 13:30 | |
| *** psachin has quit IRC | 13:31 | |
| *** TrevorV has joined #openstack-meeting | 13:31 | |
| *** yasufum has quit IRC | 13:32 | |
| *** dklyle has quit IRC | 13:35 | |
| *** rajivmucheli has joined #openstack-meeting | 13:44 | |
| *** liuyulong has joined #openstack-meeting | 13:50 | |
| *** yamamoto has quit IRC | 13:50 | |
| *** yasufum has joined #openstack-meeting | 13:53 | |
| *** mlavalle has joined #openstack-meeting | 13:58 | |
| liuyulong | #startmeeting neutron_l3 | 14:02 |
|---|---|---|
| openstack | Meeting started Wed Jul 1 14:02:54 2020 UTC and is due to finish in 60 minutes. The chair is liuyulong. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:02 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:02 |
| *** openstack changes topic to " (Meeting topic: neutron_l3)" | 14:02 | |
| openstack | The meeting name has been set to 'neutron_l3' | 14:02 |
| *** Liang__ has joined #openstack-meeting | 14:02 | |
| *** Liang__ is now known as LiangFang | 14:02 | |
| liuyulong | #topic Announcements | 14:03 |
| *** openstack changes topic to "Announcements (Meeting topic: neutron_l3)" | 14:03 | |
| slaweq | hi | 14:03 |
| liuyulong | hi | 14:03 |
| liuyulong | The L3 meeting is trending to lack of followers and attendees. | 14:04 |
| liuyulong | Maybe we should change the schedule to every 2 weeks. | 14:05 |
| liuyulong | Or merge the meeting with some others, or directly go back to a section during the team meeting. | 14:06 |
| liuyulong | Mostly the L3 meeting is mainly aiming on the bugs, and mostly were mentioned before L3 meeting in the team meeting. | 14:07 |
| haleyb | hi, sorry, i always have a conflict with a downstream meeting, sometimes forget to join | 14:07 |
| slaweq | I think we can do it biweekly | 14:07 |
| slaweq | but I don't think we will be able to merge it to team meeting | 14:08 |
| slaweq | as there may be not enough time there to discuss all L3 bugs e.g. | 14:08 |
| *** yasufum has quit IRC | 14:09 | |
| *** rajivmucheli has left #openstack-meeting | 14:09 | |
| liuyulong | Sure, thoese are some opinions from me based on my statistics in the last few L3 meetings. | 14:10 |
| liuyulong | So biweekly is ok to me. | 14:10 |
| liuyulong | Alright, thanks, no more announcement from me. | 14:11 |
| *** yamamoto has joined #openstack-meeting | 14:11 | |
| *** yamamoto has quit IRC | 14:11 | |
| liuyulong | Next | 14:11 |
| liuyulong | #topic Bugs | 14:11 |
| *** openstack changes topic to "Bugs (Meeting topic: neutron_l3)" | 14:11 | |
| *** yamamoto has joined #openstack-meeting | 14:12 | |
| liuyulong | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-June/015707.html | 14:12 |
| liuyulong | Slawek Kaplonski slaweq was our bug deputy last week, thank you. | 14:12 |
| liuyulong | First one: | 14:12 |
| *** sluna has quit IRC | 14:12 | |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1884906 | 14:12 |
| openstack | Launchpad bug 1884906 in neutron "L3 agent cannot be manually scheduled" [High,Fix released] - Assigned to Rodolfo Alonso (rodolfo-alonso-hernandez) | 14:12 |
| liuyulong | We have talked about this last week during L3 meeting. | 14:13 |
| *** sluna has joined #openstack-meeting | 14:13 | |
| liuyulong | The bug https://bugs.launchpad.net/neutron/+bug/1786272 is the long history of this. | 14:14 |
| openstack | Launchpad bug 1786272 in neutron "Connection between two virtual routers does not work with DVR" [Medium,Fix released] - Assigned to Slawek Kaplonski (slaweq) | 14:14 |
| liuyulong | Sorry wrong link... | 14:15 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1884527 | 14:15 |
| openstack | Launchpad bug 1884527 in neutron "Related dvr routers aren't created on compute nodes" [Medium,In progress] - Assigned to Slawek Kaplonski (slaweq) | 14:15 |
| slaweq | I need to check why some functional test is failing there and update this patch | 14:15 |
| liuyulong | #link https://review.opendev.org/#/c/737286/ | 14:16 |
| *** yamamoto has quit IRC | 14:16 | |
| liuyulong | slaweq, yes, this is the link of the patch. And unit test cases too. : ) | 14:16 |
| liuyulong | I will try the patch after the zuul pass. | 14:17 |
| *** yasufum has joined #openstack-meeting | 14:19 | |
| liuyulong | Allow me share some experiences from our cloud, we have no such "complex" user define topology. | 14:19 |
| liuyulong | In the production perspective, we define the virtual private cloud (aka VPC) with a router, a external gateway, some networks/subnets and the connections between them. | 14:21 |
| liuyulong | Such hierarchical topology can introduce a very high load of trouble shooting. And in some cases like VIP, load balance, bare mentel, it is no so much available. | 14:23 |
| liuyulong | So we directly disable that. | 14:23 |
| slaweq | liuyulong: yes, but we have customers who are using it for some reason | 14:24 |
| liuyulong | But it is a nice feature of neutron, it is kind of something we called "SDN", software defined network! | 14:24 |
| slaweq | and in fact this is perfectly valid use case | 14:24 |
| slaweq | which works fine for topologies other than dvr | 14:24 |
| liuyulong | slaweq, Definitely, it is the cool point that neutron can define such topology. | 14:26 |
| liuyulong | OK, next | 14:27 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1883321 | 14:27 |
| openstack | Launchpad bug 1883321 in neutron "Neutron OpenvSwitch DVR - connection problem" [High,New] | 14:27 |
| liuyulong | The bug reporter did really nice work. | 14:28 |
| *** maciejjozefczyk has joined #openstack-meeting | 14:28 | |
| liuyulong | We almost have everything we need. | 14:28 |
| liuyulong | And after reading the bug, I noticed that this may be a bug of floating IP migration. | 14:29 |
| liuyulong | change the compute node L3 agent mode from dvr to dvr_no_external, or conversely. | 14:29 |
| liuyulong | The floating IPs will lose the connection. | 14:29 |
| liuyulong | It is not related to the config option "explicity_egress_direct". | 14:30 |
| liuyulong | I will take this one, and test the floating IP migration. | 14:31 |
| liuyulong | For now, I can say, a valid step to do this work is: | 14:32 |
| liuyulong | 1. disassociate the floating IPs | 14:32 |
| liuyulong | 2. disable the routers before change the agent mode | 14:32 |
| liuyulong | 3. change the agent mode | 14:32 |
| liuyulong | 4. enable the routers | 14:32 |
| liuyulong | 5. associate the floating IPs back | 14:33 |
| liuyulong | OK, I will paste this to the LP for the reporter. | 14:33 |
| slaweq | k | 14:33 |
| liuyulong | OK, no more L3 bugs from the deputy list. | 14:34 |
| liuyulong | Let's scan the LP list. | 14:35 |
| liuyulong | Alright, a fresh one: | 14:36 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1885921 | 14:36 |
| openstack | Launchpad bug 1885921 in neutron "[RFE][floatingip port_forwarding] Add port ranges" [Undecided,New] - Assigned to Pedro Henrique Pereira Martins (pedrohpmartins) | 14:36 |
| liuyulong | Looks like a nice request of the L3 port forwarding, something like a bulk creation. | 14:37 |
| liuyulong | Seems there is a large scale uses case behind such request. | 14:38 |
| liuyulong | Let's continue the discussion on the LP bug. | 14:38 |
| liuyulong | Next one: | 14:39 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1885898 | 14:39 |
| openstack | Launchpad bug 1885898 in neutron "test connectivity through 2 routers fails in neutron-ovn-tempest-full-multinode-ovs-master job" [High,Confirmed] | 14:39 |
| liuyulong | This is related to OVN L3, IMO. | 14:40 |
| slaweq | yes | 14:40 |
| slaweq | I opened it today | 14:40 |
| slaweq | and it's only in ovn based scenarios | 14:40 |
| liuyulong | If this test_connectivity_through_2_routers case blocks the gate/CI alot, maybe we should mark it as unstable. | 14:42 |
| slaweq | liuyulong: no, I saw it failing only in non-voting jobs | 14:42 |
| slaweq | that's why it's "high" and not "critical" | 14:43 |
| liuyulong | At least bug 1884527 and 1885898 are all failing on it. | 14:43 |
| openstack | bug 1885898 in neutron "test connectivity through 2 routers fails in neutron-ovn-tempest-full-multinode-ovs-master job" [High,Confirmed] https://launchpad.net/bugs/1885898 | 14:43 |
| openstack | bug 1884527 in neutron "Related dvr routers aren't created on compute nodes" [Medium,In progress] https://launchpad.net/bugs/1884527 - Assigned to Slawek Kaplonski (slaweq) | 14:43 |
| *** diurnalist has joined #openstack-meeting | 14:44 | |
| liuyulong | slaweq, OK | 14:45 |
| liuyulong | Alright, no more bugs from LP list then. | 14:45 |
| liuyulong | Any updates? | 14:45 |
| slaweq | I have one old to talk about | 14:45 |
| liuyulong | Sure | 14:45 |
| slaweq | https://bugs.launchpad.net/neutron/+bug/1774459 | 14:45 |
| openstack | Launchpad bug 1774459 in neutron "Update permanent ARP entries for allowed_address_pair IPs in DVR Routers" [High,In progress] - Assigned to Brian Haley (brian-haley) | 14:45 |
| slaweq | there is patch https://review.opendev.org/#/c/601336/ which I think is last missing bit to solve that | 14:46 |
| slaweq | liuyulong but as You had many comments there, I wanted to ask if You think we should/can move on with this one | 14:47 |
| slaweq | or maybe do You have some other idea about how to solve this issue | 14:47 |
| *** armax has joined #openstack-meeting | 14:47 | |
| * liuyulong opening the link... | 14:47 | |
| *** LiangFang has quit IRC | 14:48 | |
| liuyulong | OK, I see that. | 14:48 |
| liuyulong | This is really a tough one. | 14:48 |
| liuyulong | The current fix is relying on the garp sending out from the VM. | 14:49 |
| slaweq | yes | 14:49 |
| slaweq | when additional IP is configured there | 14:49 |
| *** dklyle has joined #openstack-meeting | 14:49 | |
| liuyulong | This could be the main risk of the fix, neutron is rely on something which is out of control. | 14:50 |
| slaweq | yes but do You have any other idea how to solve this? | 14:51 |
| *** Liang__ has joined #openstack-meeting | 14:51 | |
| slaweq | and also main use case for that is using e.g. keepalived and it sends garps when configures IP on the host | 14:51 |
| liuyulong | What if the garp is not out? Or there are some tools which does not send garp by default? | 14:52 |
| slaweq | so in this use case it would be fine | 14:52 |
| liuyulong | yep, for keepalived for now. | 14:52 |
| slaweq | I see Your point but if tool/vm is not informing us that it is using this IP now actually, how we can know that? | 14:52 |
| liuyulong | I have an potential alternative of the fix which based on arp proxy. | 14:53 |
| liuyulong | It may work. | 14:53 |
| liuyulong | I have not tested it yet. | 14:54 |
| haleyb | i don't think there's a perfect solution, this area has always been "should be good enough for most" unfortunately | 14:54 |
| liuyulong | haleyb, agreed, so maybe each solution can be configurable. | 14:55 |
| *** links has quit IRC | 14:55 | |
| liuyulong | then the end user can choose their best way. | 14:56 |
| liuyulong | slaweq, arp proxy on the qr-device for those VIPs cross the subnets. | 14:56 |
| slaweq | IMHO for now we have use cases with keepalived for which garps should be ok and we should focus to address that | 14:56 |
| haleyb | that might lead to confusion having two ways with a config option | 14:57 |
| slaweq | if we will have other valid use case which isn't addressed with such solution we can think about other and about some config switch | 14:57 |
| haleyb | slaweq: my next question is how would OVN do this? :) | 14:57 |
| slaweq | haleyb: I'm not even sure if that is the issue in case of ovn | 14:58 |
| haleyb | that would be good | 14:58 |
| slaweq | AFAIU this issue is strictly related with how dvr works | 14:58 |
| slaweq | e.g. in L3ha it should works fine probably | 14:58 |
| liuyulong | maybe we should test the VIP for OVN in east-west connections. | 14:58 |
| slaweq | but that would need to be tested probably | 14:58 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1859638 | 14:59 |
| openstack | Launchpad bug 1774459 in neutron "duplicate for #1859638 Update permanent ARP entries for allowed_address_pair IPs in DVR Routers" [High,In progress] - Assigned to Brian Haley (brian-haley) | 14:59 |
| liuyulong | It is marked as duplicated. | 14:59 |
| liuyulong | The title is "VIP between dvr east-west networks does not work at all". | 14:59 |
| slaweq | I think we are over time today | 15:00 |
| liuyulong | OK, time is up. | 15:00 |
| liuyulong | Thank you guys for the discussion. | 15:01 |
| liuyulong | #endmeeting | 15:01 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 15:01 | |
| openstack | Meeting ended Wed Jul 1 15:01:24 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:01 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-07-01-14.02.html | 15:01 |
| slaweq | thx | 15:01 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-07-01-14.02.txt | 15:01 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-07-01-14.02.log.html | 15:01 |
| *** manuvakery has quit IRC | 15:12 | |
| *** _mlavalle_1 has joined #openstack-meeting | 15:19 | |
| *** mlavalle has quit IRC | 15:22 | |
| *** Liang__ has quit IRC | 15:33 | |
| *** jiaopengju1 has quit IRC | 15:45 | |
| *** jiaopengju1 has joined #openstack-meeting | 15:46 | |
| *** yasufum has quit IRC | 15:48 | |
| *** ricolin has quit IRC | 16:01 | |
| *** liuyulong has quit IRC | 16:05 | |
| *** armax has quit IRC | 16:06 | |
| *** ttsiouts has quit IRC | 16:06 | |
| *** ttsiouts has joined #openstack-meeting | 16:07 | |
| *** ttsiouts has quit IRC | 16:11 | |
| *** armax has joined #openstack-meeting | 16:11 | |
| *** etp has quit IRC | 16:16 | |
| *** etp has joined #openstack-meeting | 16:19 | |
| *** etp has quit IRC | 16:24 | |
| *** etp has joined #openstack-meeting | 16:44 | |
| *** ociuhandu_ has joined #openstack-meeting | 16:45 | |
| *** ociuhandu has quit IRC | 16:49 | |
| *** ociuhandu_ has quit IRC | 16:52 | |
| *** Lucas_Gray has quit IRC | 16:53 | |
| *** Lucas_Gray has joined #openstack-meeting | 16:58 | |
| *** ociuhandu has joined #openstack-meeting | 16:59 | |
| *** rh-jelabarre has quit IRC | 16:59 | |
| *** rh-jelabarre has joined #openstack-meeting | 16:59 | |
| *** psahoo_ has quit IRC | 17:01 | |
| *** ociuhandu has quit IRC | 17:07 | |
| *** psahoo_ has joined #openstack-meeting | 17:08 | |
| *** ralonsoh has quit IRC | 17:10 | |
| *** ralonsoh has joined #openstack-meeting | 17:10 | |
| *** psahoo_ has quit IRC | 17:11 | |
| *** corvus has quit IRC | 17:15 | |
| *** bnemec has quit IRC | 17:15 | |
| *** rledisez has quit IRC | 17:15 | |
| *** mattoliverau has quit IRC | 17:15 | |
| *** timburke has quit IRC | 17:15 | |
| *** jamespage has quit IRC | 17:15 | |
| *** ericyoung has quit IRC | 17:15 | |
| *** freefood has quit IRC | 17:15 | |
| *** jamespage has joined #openstack-meeting | 17:18 | |
| *** corvus has joined #openstack-meeting | 17:18 | |
| *** ericyoung has joined #openstack-meeting | 17:18 | |
| *** timburke has joined #openstack-meeting | 17:18 | |
| *** freefood has joined #openstack-meeting | 17:18 | |
| *** bnemec has joined #openstack-meeting | 17:21 | |
| *** johanssone has quit IRC | 17:26 | |
| *** freerunner has quit IRC | 17:26 | |
| *** yoshito-ito has quit IRC | 17:26 | |
| *** hongbin has joined #openstack-meeting | 17:28 | |
| *** irclogbot_3 has quit IRC | 17:28 | |
| *** gyee has joined #openstack-meeting | 17:29 | |
| *** johanssone has joined #openstack-meeting | 17:29 | |
| *** freerunner has joined #openstack-meeting | 17:29 | |
| *** yoshito-ito has joined #openstack-meeting | 17:29 | |
| *** irclogbot_1 has joined #openstack-meeting | 17:30 | |
| *** Lucas_Gray has quit IRC | 17:30 | |
| *** njohnston_ has joined #openstack-meeting | 17:43 | |
| *** njohnston has quit IRC | 17:43 | |
| *** yamamoto has joined #openstack-meeting | 17:44 | |
| *** njohnston_ is now known as njohnston | 17:45 | |
| *** yamamoto has quit IRC | 17:53 | |
| *** e0ne has quit IRC | 17:59 | |
| *** eharney has quit IRC | 18:16 | |
| *** slaweq has quit IRC | 18:25 | |
| *** dasp_ has quit IRC | 18:39 | |
| *** dasp has joined #openstack-meeting | 18:41 | |
| *** hongbin has quit IRC | 18:44 | |
| *** dklyle has quit IRC | 18:54 | |
| *** dklyle has joined #openstack-meeting | 19:02 | |
| *** ociuhandu has joined #openstack-meeting | 19:06 | |
| *** _mlavalle_1 has quit IRC | 19:10 | |
| *** _mlavalle_1 has joined #openstack-meeting | 19:15 | |
| *** hongbin has joined #openstack-meeting | 19:18 | |
| *** yoctozepto7 has joined #openstack-meeting | 19:37 | |
| *** tris has quit IRC | 19:45 | |
| *** yoctozepto has quit IRC | 19:45 | |
| *** yoctozepto7 is now known as yoctozepto | 19:45 | |
| *** tris- has joined #openstack-meeting | 19:45 | |
| *** tris- is now known as tris | 19:45 | |
| *** vishalmanchanda has quit IRC | 19:47 | |
| *** ralonsoh has quit IRC | 19:50 | |
| *** eharney has joined #openstack-meeting | 19:59 | |
| *** hongbin has quit IRC | 20:06 | |
| *** e0ne has joined #openstack-meeting | 20:20 | |
| *** slaweq has joined #openstack-meeting | 20:25 | |
| *** maciejjozefczyk has quit IRC | 20:36 | |
| *** jgriffith has quit IRC | 20:54 | |
| *** hongbin has joined #openstack-meeting | 20:54 | |
| *** alecuyer has joined #openstack-meeting | 20:57 | |
| *** patchbot has joined #openstack-meeting | 20:59 | |
| timburke | #startmeeting swift | 21:00 |
| openstack | Meeting started Wed Jul 1 21:00:06 2020 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| *** openstack changes topic to " (Meeting topic: swift)" | 21:00 | |
| openstack | The meeting name has been set to 'swift' | 21:00 |
| timburke | who's here for the swift meeting? | 21:00 |
| kota_ | o/ | 21:00 |
| alecuyer | o/ | 21:00 |
| seongsoocho | o/ | 21:00 |
| clayg | howdy! | 21:00 |
| timburke | agenda's at https://wiki.openstack.org/wiki/Meetings/Swift | 21:02 |
| timburke | first up | 21:02 |
| timburke | #topic Berlin | 21:02 |
| *** openstack changes topic to "Berlin (Meeting topic: swift)" | 21:02 | |
| timburke | the Request for Presentations recently opened up | 21:02 |
| timburke | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-July/015730.html | 21:02 |
| timburke | deadline's Aug 4 | 21:02 |
| timburke | summit will be Oct 19-23 | 21:03 |
| *** ircuser-1 has quit IRC | 21:03 | |
| timburke | (assuming pandemic is more or less under control, so... who knows?) | 21:03 |
| kota_ | :/ | 21:04 |
| timburke | but i wanted to at least bring the call for presentations to everyone's attention | 21:04 |
| timburke | that's all i've got in terms of announcements | 21:05 |
| timburke | #topic replication servers | 21:05 |
| *** openstack changes topic to "replication servers (Meeting topic: swift)" | 21:05 | |
| timburke | so clayg made a very important observation on p 735751 | 21:05 |
| patchbot | https://review.opendev.org/#/c/735751/ - swift - Allow direct and internal clients to use the repli... - 4 patch sets | 21:05 |
| clayg | go team! | 21:06 |
| timburke | the replication servers currently only respond to SSYNC and REPLICATE requests! | 21:06 |
| timburke | #link https://bugs.launchpad.net/swift/+bug/1446873 | 21:06 |
| openstack | Launchpad bug 1446873 in OpenStack Object Storage (swift) "ssync doesn't work with replication_server = true" [Medium,Confirmed] | 21:06 |
| timburke | it won't even respond to OPTIONS, which is supposed to tell you what request methods *are* allowed | 21:07 |
| *** mattoliverau has joined #openstack-meeting | 21:07 | |
| clayg | ^ that's like @timburke 's favorite joke 🤣 | 21:07 |
| *** e0ne has quit IRC | 21:07 | |
| mattoliverau | Sorry I'm late, woke up this morning and had IRC troubles, but finally connected. | 21:08 |
| timburke | it looks like it won't be too hard to allow replication servers to service all requests, and make `replication_server = False` basically mean "don't do SSYNC or REPLICATE" | 21:08 |
| timburke | does that seem like a reasonable thing to everyone, though? | 21:08 |
| timburke | i feel like the work involved in making it so i can properly *test* with separate replication servers is going to be way bigger than making it work :-/ | 21:10 |
| clayg | I think the conclusion on the lp bug was drop the semantic of `replicatoin_server = None` - which I think is what you HAVE to run on your replication server's now if you want to do a replication netowork and have EC rebuild still work | 21:12 |
| timburke | seems related to https://review.opendev.org/#/c/337861/ 🤔 | 21:13 |
| patchbot | patch 337861 - swift - Permit to bind object-server on replication_port - 7 patch sets | 21:13 |
| clayg | yeah probably 😬 | 21:13 |
| clayg | i feel like if this was easy we wouldn't have put it off so long | 21:13 |
| clayg | it would be helpful to have sanity checks as we try to merge shit | 21:14 |
| timburke | it's probably worth me putting in some work to make probe tests runnable with separate replication servers regardless. seems like a lot of people run that way (we certainly do!) and anything i can do to make my dev environment more representative of prod seems like a good idea | 21:15 |
| timburke | but, it'll probably be a bit | 21:15 |
| timburke | #topic waterfall EC | 21:15 |
| *** openstack changes topic to "waterfall EC (Meeting topic: swift)" | 21:15 | |
| timburke | clayg, how's it going? | 21:16 |
| clayg | timburke: it's a non-zero cost to have to run a second set of backend *-server, but could be nice | 21:16 |
| clayg | timburke: you're the only one who's commented 😠 | 21:16 |
| timburke | hehe | 21:16 |
| timburke | fair enough | 21:16 |
| clayg | i'm really proud of my little feeder system tho - I've started to conceptualize breaking them up as like a poly rhythm sort of situation | 21:17 |
| clayg | where we have on flow of code that's popping ticks predictably (start ndata, then remaining nparity at a predictable pattern) | 21:17 |
| clayg | then there's this *other* beat that's all random just based on when stuff responds | 21:17 |
| clayg | doing that all in one loop was maddness - two loops is better | 21:18 |
| clayg | and then I'm also proud of the logic simplifications that I'd managed to get so far - but it was super helpful to have another brain load it up and WTF at it | 21:18 |
| clayg | some bits are still confusion - but I was desensitized | 21:18 |
| *** slaweq has quit IRC | 21:19 | |
| timburke | so, anyone else have some bandwidth to take a look in the next week or two? | 21:19 |
| clayg | you were talking about good buckets and bad buckets and 416 as success ... there might be more cleanup, but I'd need some sort of "bad behavior" to really motiviate me to "get back in there" | 21:19 |
| clayg | otherwise it's just pushing symbols around for subjective qualities | 21:20 |
| alecuyer | I think I can at least try the patch next week, I should have more time than these past weeks | 21:20 |
| clayg | I'm happy with the tradeoffs for "the non-durable problem" - I know were never did that zoom or w/e - but it's fine (i'm still happy to answer questions as needed) | 21:20 |
| timburke | alecuyer, thank you | 21:20 |
| clayg | the final patch - the "make timeouts configurable per replica" | 21:21 |
| clayg | it's... a little "much" by some estimations. I like the expressiveness; but worry it's the same as the pipeline problem 😞 | 21:21 |
| *** priteau has quit IRC | 21:21 | |
| clayg | the more people who would be willing to put on their operator hat and try to grok what those options even *mean* THE BETTER | 21:22 |
| clayg | if I can explain it to YOU guys trivially there's no hope I'll ever be able to write clear docs | 21:22 |
| alecuyer | ok | 21:22 |
| timburke | would it be worth us documenting them as experimental/subject to change (or even removal)? just as soon as we get a chance to have a better option | 21:22 |
| clayg | Then the last gotcha is the final fate of ECFragGetter - do we trim it down lean and mean and try to pull all remains of EC out of GETorHEADHandler - or is there still some hope we can unify them some way that's sane? | 21:23 |
| clayg | It's really not clear to me; and I guess I'm avoiding trying to code that I'm not confident is a good idea 👎 | 21:24 |
| clayg | timburke: I think that'd be reasonable if after everyone looks at them we have some different ideas about directions we might go; or the best we can come up with seems like a lot of work | 21:24 |
| clayg | if we look at them and say "yeah, per policy; per replica - makes sense" then we just write that down with some common examples/patterns and move on | 21:25 |
| clayg | so I think I really could use some feedback right now - from anyone who has cycles - you don't have to grok all the code; or even check out the change | 21:26 |
| clayg | reading the example config in the final patch (based on the converstations we been having) and providing feed back there is a great start | 21:26 |
| timburke | one thing i'm definitely digging is the fact that it's available as a per-policy option immediately -- i think it'll be really handy for a lab cluster to be able to define, say, five EC policies that are identical except for those tunings | 21:26 |
| mattoliverau | I'll make sure I have a look this week, especially big picture so can comment of config options. If I have time I'll try and go a little deeper. | 21:27 |
| clayg | if you can glance through some of the weird corners of how ECFragGetter is starting to diverge from GETorHEADHandler and give me a gut check "yeah probably different enough" or "these smell like minor differences" would also be helpful | 21:27 |
| timburke | #link https://review.opendev.org/#/c/737096 | 21:27 |
| patchbot | patch 737096 - swift - Make concurrency timeout per policy and replica - 3 patch sets | 21:27 |
| clayg | if you can grok that main ec-waterfall loops - even setting aside weird bucket stuff like shortfall and durable - that'd like above and beyond | 21:28 |
| timburke | (just realized i had a few comments i forgot to publish) | 21:28 |
| clayg | at that point you may as well check it out and try to configure it - it probalby wouldn't take much work to see that it DOES do what it says on the tin | 21:28 |
| clayg | THANK YOU ALL SO MUCH!!! feedback is what I need - couldn't do this without y'all - GO TEAM! | 21:29 |
| timburke | all right | 21:29 |
| timburke | #topic open discussion | 21:29 |
| *** openstack changes topic to "open discussion (Meeting topic: swift)" | 21:29 | |
| timburke | what else do we need to bring up today? | 21:30 |
| clayg | libec !!! | 21:30 |
| clayg | so we definately have the old style checksum frags... | 21:30 |
| clayg | everyone run this on your EC .data files -> https://gist.github.com/clayg/df7c276a43c3618d7897ba50ae87ea9d | 21:30 |
| clayg | if your hash matches zlib you're golden! upgrade to >1.16 and rejoice! | 21:31 |
| clayg | if you've got that stupid stupid stupid libec inline checksums - you're probably gunna be stuck with them for awhile (and also don't upgrade - you'll die) | 21:31 |
| alecuyer | thanks for that snippet clay | 21:32 |
| clayg | well... you might die anyway because 1.15 was like... somehow indeterminate? | 21:32 |
| timburke | well, until everything upgrades ;-) | 21:32 |
| clayg | so we have to do better; but 1.16 isn't quite good enough | 21:32 |
| clayg | alecuyer: timburke wrote it | 21:32 |
| timburke | so, it seems like we need a way to tell libec to continue writing old frags | 21:33 |
| mattoliverau | wow, nice useful bit of coding there! | 21:33 |
| timburke | we also almost certainly need to write a detailed bug ;-) | 21:34 |
| clayg | right, for us to upgrade we need new code to not start writing in the new format yet because old nodes won't know what to do with that shit until the upgrade | 21:34 |
| clayg | I think we could probably just like... "upgrade libec real fast and restart everything!" | 21:34 |
| clayg | but... we'll probably like "be careful" or whatever 🙄 | 21:34 |
| clayg | I'm gunna try to talk timburke into a *build* option that's just like "always write the old stupid inline crc value forever" | 21:35 |
| clayg | that way I can just make it WOMM and then ship it and never thing about how much i hate C again | 21:35 |
| alecuyer | hehe | 21:36 |
| clayg | but... it might not work - timburke is pretty sure we should actually use the zlib version as soon as we can - so an env var with a portable build might be "better" | 21:36 |
| clayg | if by "better" is - you'd rather pay some operational cost to rollout with the latch; then after upgrade remove the latch; and then be on the path of righteousness forever fighting back the forces of kludge and evil | 21:37 |
| timburke | yeah, i'm still fairly nervous that our funky crc doesn't have the same kinds of guarantees that we're expecting from zlib... | 21:38 |
| clayg | timburke: well i'm not sure I can write controller code that can do the upgrade properly! like... we'd have to do a checkpoint release or something 🤮 | 21:39 |
| clayg | I'm sure I could build a packge with an option that's like CLAYG_IS_TOO_LAZY_TO_DO_THIS_RIGHT=True | 21:39 |
| timburke | side note on all of this: thank you alecuyer for noticing this and bringing it up last week! much better to be arguing about how best to navigate this now than mid-upgrade ;-) | 21:39 |
| clayg | and then if at somepoint you're sure all legacy swiftstack customers have upgrade you turn that off ;) | 21:39 |
| clayg | yeah FOR SURE - alecuyer is god send ❤️ | 21:40 |
| *** haleyb has quit IRC | 21:40 | |
| alecuyer | thanks but I wish i'd do more - and be faster.. :) but thanks | 21:40 |
| timburke | clayg, we could totally have a controller that always says "write legacy crcs" -- we've done controller checkpoint releases before; after the next one, we tell it to switch over | 21:40 |
| clayg | yeah, again - if we can couple it to an upstream swift change that makes it just an ini/config var I'm totally down | 21:41 |
| clayg | if I have to update our systemd units in the package to turn it on/off i'm pretty sure I'm too dumb to get it right | 21:42 |
| clayg | alecuyer: kota_: mattoliverau: if you have access to any EC data in any cluster anywhere please see if you can get the crc thing to run on it and report back w/i a couple of weeks | 21:42 |
| clayg | we'll probably kick this can down the road for awhile (1.15 is working fine for me!) | 21:43 |
| clayg | that's all I got on libec | 21:43 |
| alecuyer | yep, will do | 21:43 |
| timburke | anybody think they'll have a chance to look at https://review.opendev.org/#/c/737856/ ? it seems to be working well for ormandj | 21:43 |
| patchbot | patch 737856 - swift - py3: Stop munging RAW_PATH_INFO - 2 patch sets | 21:44 |
| timburke | i'd like to backport it to ussuri and train, then ideally do a round of releases | 21:44 |
| clayg | timburke: oh neat - is the test new? | 21:45 |
| timburke | (probably should have done that earlier, but now that there's an additional known issue...) | 21:45 |
| timburke | yeah | 21:45 |
| clayg | b"GET /oh\xffboy%what$now%E2%80%bd HTTP/1.0\r\n" 🤣 | 21:45 |
| timburke | oh, and as a heads-up: it looks like there might be some movement on an eventlet fix for that py37/ssl bug | 21:46 |
| timburke | #link https://github.com/eventlet/eventlet/pull/621 | 21:46 |
| alecuyer | sounds good! | 21:47 |
| timburke | all right, last call | 21:49 |
| timburke | thank you all for coming, and thank you for working on swift! | 21:50 |
| timburke | #endmeeting | 21:50 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 21:50 | |
| openstack | Meeting ended Wed Jul 1 21:50:33 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:50 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-07-01-21.00.html | 21:50 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-07-01-21.00.txt | 21:50 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-07-01-21.00.log.html | 21:50 |
| *** patchbot has left #openstack-meeting | 21:50 | |
| *** haleyb has joined #openstack-meeting | 21:52 | |
| *** rcernin has joined #openstack-meeting | 22:37 | |
| *** rcernin has quit IRC | 22:47 | |
| *** rcernin has joined #openstack-meeting | 22:47 | |
| *** ykatabam has joined #openstack-meeting | 22:51 | |
| *** TrevorV has quit IRC | 22:54 | |
| *** jmasud has quit IRC | 23:01 | |
| *** jmasud has joined #openstack-meeting | 23:03 | |
| *** _mlavalle_1 has quit IRC | 23:11 | |
| *** hongbin has quit IRC | 23:11 | |
| *** moguimar has quit IRC | 23:14 | |
| *** hongbin has joined #openstack-meeting | 23:16 | |
| *** Lucas_Gray has joined #openstack-meeting | 23:21 | |
| *** rfolco has quit IRC | 23:48 | |
| *** yamamoto has joined #openstack-meeting | 23:52 | |
| *** yamamoto has quit IRC | 23:57 | |
| *** yasufum has joined #openstack-meeting | 23:58 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!