| *** armax has joined #openstack-meeting | 00:01 | |
| *** andrebeltrami has quit IRC | 00:09 | |
| *** armax has quit IRC | 00:18 | |
| *** yamamoto has joined #openstack-meeting | 00:29 | |
| *** yamamoto has quit IRC | 01:03 | |
| *** ayoung has quit IRC | 01:08 | |
| *** ayoung has joined #openstack-meeting | 01:11 | |
| *** Liang__ has joined #openstack-meeting | 01:17 | |
| *** yamamoto has joined #openstack-meeting | 01:33 | |
| *** gyee has quit IRC | 01:49 | |
| *** Liang__ has quit IRC | 01:54 | |
| *** ayoung has quit IRC | 02:32 | |
| *** ayoung has joined #openstack-meeting | 02:34 | |
| *** psahoo has joined #openstack-meeting | 03:07 | |
| *** rcernin has quit IRC | 03:10 | |
| *** rcernin has joined #openstack-meeting | 03:18 | |
| *** ayoung has quit IRC | 03:19 | |
| *** manpreet has joined #openstack-meeting | 03:22 | |
| *** ayoung has joined #openstack-meeting | 03:22 | |
| *** psachin has joined #openstack-meeting | 03:40 | |
| *** ayoung has quit IRC | 03:50 | |
| *** ayoung has joined #openstack-meeting | 03:53 | |
| *** ociuhandu has joined #openstack-meeting | 04:07 | |
| *** ociuhandu has quit IRC | 04:12 | |
| *** psahoo has quit IRC | 04:20 | |
| *** markvoelker has joined #openstack-meeting | 04:25 | |
| *** dklyle has quit IRC | 04:29 | |
| *** dklyle has joined #openstack-meeting | 04:29 | |
| *** markvoelker has quit IRC | 04:30 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** psahoo has joined #openstack-meeting | 04:33 | |
| *** evrardjp has joined #openstack-meeting | 04:33 | |
| *** markvoelker has joined #openstack-meeting | 05:09 | |
| *** markvoelker has quit IRC | 05:21 | |
| *** manpreet has quit IRC | 05:31 | |
| *** sridharg has joined #openstack-meeting | 05:51 | |
| *** kaisers has quit IRC | 05:59 | |
| *** dmacpher_ has joined #openstack-meeting | 06:04 | |
| *** johanssone has quit IRC | 06:04 | |
| *** tobberydberg has quit IRC | 06:05 | |
| *** ricolin has quit IRC | 06:08 | |
| *** dmacpher has quit IRC | 06:08 | |
| *** johanssone has joined #openstack-meeting | 06:11 | |
| *** tobberydberg has joined #openstack-meeting | 06:11 | |
| *** kaisers has joined #openstack-meeting | 06:12 | |
| *** mahatic has quit IRC | 06:14 | |
| *** slaweq has joined #openstack-meeting | 06:28 | |
| *** dklyle has quit IRC | 06:46 | |
| *** ykatabam has quit IRC | 07:00 | |
| *** moguimar has joined #openstack-meeting | 07:03 | |
| *** maciejjozefczyk has joined #openstack-meeting | 07:04 | |
| *** rcernin has quit IRC | 07:07 | |
| *** ssbarnea has joined #openstack-meeting | 07:17 | |
| *** martial has quit IRC | 07:28 | |
| *** zbr has quit IRC | 07:29 | |
| *** ssbarnea has quit IRC | 07:29 | |
| *** zbr9 has joined #openstack-meeting | 07:29 | |
| *** tosky has joined #openstack-meeting | 07:41 | |
| *** e0ne has joined #openstack-meeting | 07:46 | |
| *** Lucas_Gray has joined #openstack-meeting | 07:51 | |
| *** markvoelker has joined #openstack-meeting | 07:55 | |
| *** markvoelker has quit IRC | 08:00 | |
| *** masahito has joined #openstack-meeting | 08:00 | |
| *** masahito has quit IRC | 08:06 | |
| *** masahito has joined #openstack-meeting | 08:06 | |
| *** zbr9 has quit IRC | 08:12 | |
| *** zbr has joined #openstack-meeting | 08:13 | |
| *** evrardjp has quit IRC | 08:14 | |
| *** evrardjp has joined #openstack-meeting | 08:21 | |
| *** evrardjp has quit IRC | 08:26 | |
| *** evrardjp has joined #openstack-meeting | 08:27 | |
| *** Lucas_Gray has quit IRC | 08:38 | |
| *** Lucas_Gray has joined #openstack-meeting | 08:42 | |
| *** masahito has quit IRC | 08:43 | |
| *** ociuhandu has joined #openstack-meeting | 08:51 | |
| *** ociuhandu has quit IRC | 08:56 | |
| *** ykatabam has joined #openstack-meeting | 09:06 | |
| *** ociuhandu has joined #openstack-meeting | 09:28 | |
| *** ykatabam has quit IRC | 09:41 | |
| *** psahoo has quit IRC | 09:47 | |
| *** psahoo has joined #openstack-meeting | 10:03 | |
| *** manpreet has joined #openstack-meeting | 10:13 | |
| *** Lucas_Gray has quit IRC | 10:16 | |
| *** ykatabam has joined #openstack-meeting | 10:18 | |
| *** Lucas_Gray has joined #openstack-meeting | 10:21 | |
| *** ykatabam has quit IRC | 10:25 | |
| *** evrardjp has quit IRC | 10:29 | |
| *** Lucas_Gray has quit IRC | 10:34 | |
| *** evrardjp has joined #openstack-meeting | 10:35 | |
| *** ociuhandu has quit IRC | 10:38 | |
| *** ociuhandu has joined #openstack-meeting | 10:39 | |
| *** yamamoto has quit IRC | 10:41 | |
| *** Lucas_Gray has joined #openstack-meeting | 10:45 | |
| *** lpetrut has joined #openstack-meeting | 10:45 | |
| *** evrardjp has quit IRC | 10:48 | |
| *** yamamoto has joined #openstack-meeting | 10:50 | |
| *** vishalmanchanda has joined #openstack-meeting | 10:57 | |
| *** ociuhandu_ has joined #openstack-meeting | 10:59 | |
| *** ociuhandu has quit IRC | 11:02 | |
| *** ociuhandu_ has quit IRC | 11:03 | |
| *** ykatabam has joined #openstack-meeting | 11:09 | |
| *** ykatabam has quit IRC | 11:09 | |
| *** ociuhandu has joined #openstack-meeting | 11:18 | |
| *** JJH42 has joined #openstack-meeting | 11:22 | |
| *** JJH49 has joined #openstack-meeting | 11:23 | |
| *** ociuhandu has quit IRC | 11:23 | |
| *** JJH42 has quit IRC | 11:26 | |
| *** JJH49 has quit IRC | 11:26 | |
| *** jheyd has joined #openstack-meeting | 11:31 | |
| jheyd | Sorry, too late to the party ;-) | 11:32 |
|---|---|---|
| *** raildo has joined #openstack-meeting | 11:37 | |
| *** yamamoto has quit IRC | 11:41 | |
| *** yamamoto has joined #openstack-meeting | 11:41 | |
| *** carloss has joined #openstack-meeting | 11:42 | |
| *** yamamoto has quit IRC | 11:42 | |
| *** yamamoto has joined #openstack-meeting | 11:57 | |
| *** Lucas_Gray has quit IRC | 12:17 | |
| *** yamamoto has quit IRC | 12:28 | |
| *** masahito has joined #openstack-meeting | 12:35 | |
| *** ianychoi has quit IRC | 12:40 | |
| *** e0ne_ has joined #openstack-meeting | 12:45 | |
| *** masahito has quit IRC | 12:48 | |
| *** e0ne has quit IRC | 12:49 | |
| *** psahoo has quit IRC | 12:50 | |
| *** Lucas_Gray has joined #openstack-meeting | 12:51 | |
| *** masahito has joined #openstack-meeting | 12:52 | |
| *** ociuhandu has joined #openstack-meeting | 12:56 | |
| *** masahito has quit IRC | 12:57 | |
| *** dulek has quit IRC | 13:01 | |
| *** psahoo has joined #openstack-meeting | 13:02 | |
| *** yamamoto has joined #openstack-meeting | 13:07 | |
| *** ZhuXiaoYu has joined #openstack-meeting | 13:17 | |
| *** yamamoto has quit IRC | 13:18 | |
| *** ricolin has joined #openstack-meeting | 13:22 | |
| *** TrevorV has joined #openstack-meeting | 13:23 | |
| *** hongbin has joined #openstack-meeting | 13:40 | |
| *** ianychoi has joined #openstack-meeting | 13:44 | |
| *** jheyd has quit IRC | 13:49 | |
| *** liuyulong has joined #openstack-meeting | 13:55 | |
| *** eharney has quit IRC | 13:59 | |
| liuyulong | #startmeeting neutron_l3 | 14:00 |
| openstack | Meeting started Wed Aug 12 14:00:19 2020 UTC and is due to finish in 60 minutes. The chair is liuyulong. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| *** openstack changes topic to " (Meeting topic: neutron_l3)" | 14:00 | |
| openstack | The meeting name has been set to 'neutron_l3' | 14:00 |
| haleyb | hi, and just an fyi that i'm in a conflicting meeting but will notice a ping with my nick | 14:01 |
| liuyulong | sure | 14:01 |
| liuyulong | Today we will be a short meeting. | 14:01 |
| liuyulong | I have no announcements and only one L3 bug. | 14:01 |
| *** eharney has joined #openstack-meeting | 14:01 | |
| liuyulong | OK, let's directly starts the Bugs topic. | 14:03 |
| liuyulong | #topic Bugs | 14:03 |
| *** openstack changes topic to "Bugs (Meeting topic: neutron_l3)" | 14:03 | |
| liuyulong | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-August/016290.html | 14:03 |
| liuyulong | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-August/016430.html | 14:03 |
| liuyulong | First one | 14:04 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1889388 | 14:04 |
| openstack | Launchpad bug 1889388 in neutron "[OVN] access between Floatings ip and instance with Direct External IP" [Undecided,New] | 14:04 |
| liuyulong | It is taged as OVN backend, but I have never try to run instance in such case for OVS backend. | 14:05 |
| liuyulong | It's a corner case, I will check it in our OVS deployment. | 14:06 |
| slaweq | liuyulong: hi | 14:07 |
| slaweq | I have some urgent d/s issue so I will just be lurking here | 14:07 |
| haleyb | and i'm assuming there's a SG rule for access? there was no answer to rodolfo's question either | 14:07 |
| liuyulong | slaweq, hi | 14:07 |
| *** yamamoto has joined #openstack-meeting | 14:07 | |
| liuyulong | slaweq, sure, go ahead, : ) | 14:08 |
| liuyulong | haleyb, IMO whether in the same host should not be the problem, the instance can be created in different hosts. | 14:09 |
| liuyulong | For OVN, I guess the data pipeline may lack of some metadata/tag/header for packets between this two VMs. | 14:10 |
| haleyb | well, sometimes there are specific issues when both are on the same host, hairpin issue for example | 14:10 |
| *** Liang__ has joined #openstack-meeting | 14:11 | |
| liuyulong | For OVS, this may do not work either since there are also some block flows in the physical bridges. | 14:11 |
| haleyb | right, someone would have to reproduce it | 14:12 |
| liuyulong | Besides, amotoki asked one question about the tag "l3-dvr-backlog". | 14:13 |
| liuyulong | Quote | 14:13 |
| liuyulong | "l3-dvr-backlog tag was originally introduced to identify DVR feature | 14:14 |
| liuyulong | gaps. What should we use for OVN L3 feature gaps?" | 14:14 |
| *** yamamoto has quit IRC | 14:14 | |
| *** hongbin has quit IRC | 14:14 | |
| haleyb | i think we have just an 'ovn' tag | 14:14 |
| haleyb | we could add a new one for the gaps | 14:15 |
| liuyulong | I don't know the tag history, but IMO, a one shot tag may same filter works. | 14:15 |
| liuyulong | s/same/save | 14:15 |
| haleyb | i know we added the ovn one when we merged the code | 14:16 |
| liuyulong | OK, thank you for the information. | 14:17 |
| liuyulong | Next | 14:17 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1890400 | 14:17 |
| openstack | Launchpad bug 1890400 in neutron "Default gateway in HA router namespace not set if using Keepalived 1.x" [High,In progress] - Assigned to Slawek Kaplonski (slaweq) | 14:17 |
| liuyulong | slaweq has uploded the fix: https://review.opendev.org/#/c/745641/ | 14:17 |
| liuyulong | A simple config option, it's fine to me. | 14:18 |
| liuyulong | Users should be informed by the release note. | 14:19 |
| *** Liang__ has quit IRC | 14:19 | |
| haleyb | yes, it's not perfect, but we have no way of specifying a keepalived version afaik | 14:20 |
| liuyulong | yes, that's the main point. | 14:20 |
| liuyulong | Alright, no more L3 bugs from me now. | 14:21 |
| liuyulong | Let's continue to discuss something in next section. | 14:21 |
| liuyulong | #topic On demand agenda | 14:22 |
| *** openstack changes topic to "On demand agenda (Meeting topic: neutron_l3)" | 14:22 | |
| liuyulong | #link https://review.opendev.org/#/c/601336/ | 14:22 |
| liuyulong | I noticed that slaweq uploaded a new patch set for this one. | 14:22 |
| *** armax has joined #openstack-meeting | 14:24 | |
| liuyulong | But no update for the comments of patch set 42, if this will be carried on again, we should fix those. | 14:24 |
| liuyulong | Next should be this: | 14:25 |
| *** Liang__ has joined #openstack-meeting | 14:25 | |
| liuyulong | Floating IP's for routed networks | 14:25 |
| liuyulong | #link https://review.opendev.org/#/c/486450/ | 14:25 |
| liuyulong | #link https://review.opendev.org/#/c/669395/ | 14:25 |
| slaweq | liuyulong: yes, for now I only repaced it and resolved conflicts | 14:26 |
| slaweq | nothing else | 14:26 |
| liuyulong | slaweq, cool | 14:27 |
| liuyulong | For this routed network stuffs, it is perrty good now, we can move forward IMO. | 14:27 |
| *** e0ne has joined #openstack-meeting | 14:28 | |
| *** e0ne_ has quit IRC | 14:29 | |
| liuyulong | OK, no more things from me now. | 14:29 |
| liuyulong | I will hold this section a few minutes open, feel free to start your topic. | 14:30 |
| haleyb | nothing else from me | 14:32 |
| slaweq | nothing from me neighter | 14:36 |
| *** liuyulong has quit IRC | 14:37 | |
| *** hongbin has joined #openstack-meeting | 14:39 | |
| *** mlavalle has joined #openstack-meeting | 14:49 | |
| *** liuyulong has joined #openstack-meeting | 14:49 | |
| liuyulong | Sorry lost the connection | 14:49 |
| liuyulong | #endmeeting | 14:50 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 14:50 | |
| openstack | Meeting ended Wed Aug 12 14:50:11 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:50 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-08-12-14.00.html | 14:50 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-08-12-14.00.txt | 14:50 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-08-12-14.00.log.html | 14:50 |
| *** e0ne_ has joined #openstack-meeting | 14:56 | |
| *** liuyulong has quit IRC | 14:57 | |
| *** e0ne has quit IRC | 14:59 | |
| *** priteau has joined #openstack-meeting | 14:59 | |
| *** dklyle has joined #openstack-meeting | 15:05 | |
| *** Liang__ has quit IRC | 15:07 | |
| *** ykatabam has joined #openstack-meeting | 15:33 | |
| *** ykatabam has quit IRC | 15:38 | |
| *** lpetrut has quit IRC | 15:58 | |
| *** psahoo has quit IRC | 16:12 | |
| *** yamamoto has joined #openstack-meeting | 16:13 | |
| *** ayoung has quit IRC | 16:22 | |
| *** yamamoto has quit IRC | 16:23 | |
| *** ayoung has joined #openstack-meeting | 16:24 | |
| *** psahoo has joined #openstack-meeting | 16:25 | |
| *** maciejjozefczyk has quit IRC | 16:28 | |
| *** ayoung has quit IRC | 16:31 | |
| *** ayoung has joined #openstack-meeting | 16:32 | |
| *** Lucas_Gray has quit IRC | 16:36 | |
| *** ayoung has quit IRC | 16:38 | |
| *** psahoo has quit IRC | 16:39 | |
| *** psachin has quit IRC | 16:39 | |
| *** ayoung has joined #openstack-meeting | 16:40 | |
| *** hongbin has quit IRC | 16:43 | |
| *** hongbin has joined #openstack-meeting | 17:00 | |
| *** sridharg has quit IRC | 17:00 | |
| *** ayoung has quit IRC | 17:07 | |
| *** ayoung has joined #openstack-meeting | 17:08 | |
| *** ayoung has quit IRC | 17:22 | |
| *** ayoung has joined #openstack-meeting | 17:23 | |
| *** priteau has quit IRC | 17:27 | |
| *** ayoung has quit IRC | 17:31 | |
| *** ayoung has joined #openstack-meeting | 17:33 | |
| *** ZhuXiaoYu has quit IRC | 17:36 | |
| *** e0ne_ has quit IRC | 17:41 | |
| *** e0ne has joined #openstack-meeting | 17:42 | |
| *** e0ne has quit IRC | 17:44 | |
| *** e0ne has joined #openstack-meeting | 17:49 | |
| *** maciejjozefczyk has joined #openstack-meeting | 17:52 | |
| *** gyee has joined #openstack-meeting | 17:56 | |
| *** hyunsikyang__ has joined #openstack-meeting | 18:11 | |
| *** knikolla_ has joined #openstack-meeting | 18:13 | |
| *** vkmc_ has joined #openstack-meeting | 18:13 | |
| *** hyunsikyang has quit IRC | 18:20 | |
| *** darvon has quit IRC | 18:20 | |
| *** gouthamr has quit IRC | 18:20 | |
| *** knikolla has quit IRC | 18:20 | |
| *** vkmc has quit IRC | 18:20 | |
| *** knikolla_ is now known as knikolla | 18:20 | |
| *** vkmc_ is now known as vkmc | 18:20 | |
| *** gouthamr has joined #openstack-meeting | 18:22 | |
| *** manpreet has quit IRC | 18:23 | |
| *** e0ne has quit IRC | 18:23 | |
| *** yamamoto has joined #openstack-meeting | 18:23 | |
| *** ricolin has quit IRC | 18:23 | |
| *** e0ne has joined #openstack-meeting | 18:29 | |
| *** yamamoto has quit IRC | 18:32 | |
| *** Lucas_Gray has joined #openstack-meeting | 18:50 | |
| *** jmasud_ has joined #openstack-meeting | 18:59 | |
| *** jmasud_ has quit IRC | 19:00 | |
| *** Lucas_Gray has quit IRC | 19:00 | |
| *** hongbin has quit IRC | 19:26 | |
| *** maciejjozefczyk has quit IRC | 19:39 | |
| *** hongbin has joined #openstack-meeting | 19:44 | |
| *** e0ne has quit IRC | 19:49 | |
| *** mattia has quit IRC | 19:59 | |
| *** TrevorV has quit IRC | 20:24 | |
| *** yamamoto has joined #openstack-meeting | 20:32 | |
| *** yamamoto has quit IRC | 20:39 | |
| *** e0ne has joined #openstack-meeting | 20:40 | |
| *** Lucas_Gray has joined #openstack-meeting | 20:52 | |
| *** rh-jelabarre has quit IRC | 20:52 | |
| *** rh-jelabarre has joined #openstack-meeting | 20:52 | |
| *** ayoung has quit IRC | 20:57 | |
| timburke | #startmeeting swift | 21:00 |
| openstack | Meeting started Wed Aug 12 21:00:40 2020 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| *** openstack changes topic to " (Meeting topic: swift)" | 21:00 | |
| openstack | The meeting name has been set to 'swift' | 21:00 |
| timburke | who's here for the swift meeting? | 21:00 |
| mattoliverau | o/ | 21:00 |
| *** patchbot has joined #openstack-meeting | 21:00 | |
| clayg | o/ | 21:01 |
| kota_ | hi | 21:01 |
| rledisez | o/ | 21:01 |
| *** ayoung has joined #openstack-meeting | 21:02 | |
| *** dklyle has quit IRC | 21:02 | |
| timburke | agenda's at https://wiki.openstack.org/wiki/Meetings/Swift | 21:02 |
| timburke | #topic summit and ptg | 21:03 |
| *** openstack changes topic to "summit and ptg (Meeting topic: swift)" | 21:03 | |
| timburke | looks like the dates are set for the ptg | 21:03 |
| timburke | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-August/016424.html | 21:03 |
| timburke | it will immediately follow the summit | 21:03 |
| timburke | so, summit will be oct 19-23, then ptg oct 26-30 | 21:04 |
| clayg | it's all virtual like last time? that seemed to work ok... | 21:04 |
| timburke | registration's already open, and both virtual events are free | 21:04 |
| timburke | #link https://www.eventbrite.com/e/open-infrastructure-summit-2020-tickets-96967218561 | 21:04 |
| timburke | #link https://www.eventbrite.com/e/project-teams-gathering-october-2020-tickets-116136313841 | 21:04 |
| kota_ | good info. thx. | 21:05 |
| timburke | i'll keep an eye out for what we need to do to sign up for space and such, and try to be more on it than last time | 21:05 |
| clayg | timburke: or just let mattoliverau do it again - things worked great!? | 21:05 |
| timburke | though mattoliverau did a great job getting everything organized! | 21:05 |
| timburke | that's all i had by way of announcements; any questions or comments? | 21:06 |
| mattoliverau | Lol | 21:06 |
| timburke | all right | 21:07 |
| clayg | timburke: great update! especially the part about mattoliverau being awesome 👍 | 21:07 |
| timburke | #topic s3api subrequest logging | 21:08 |
| *** openstack changes topic to "s3api subrequest logging (Meeting topic: swift)" | 21:08 | |
| timburke | clayg, i think you added this, yeah? | 21:08 |
| clayg | oh no 😬 | 21:08 |
| clayg | did I add a patch? | 21:08 |
| timburke | two of them! | 21:08 |
| timburke | and now one's landed :-) | 21:08 |
| clayg | why is zuul so mad at p 735221 😡 | 21:08 |
| patchbot | https://review.opendev.org/#/c/735221/ - swift - s3api: Use swift.backend_path to proxy-log s3api r... - 2 patch sets | 21:08 |
| timburke | er, approved -- i'll keep poking at the gate | 21:09 |
| clayg | so if we CAN get that one landed I'm pretty much fine with p 735220 - in fact I'd like to do some follow-on to update the example configs | 21:09 |
| patchbot | https://review.opendev.org/#/c/735220/ - swift - proxy-logging: Be able to configure log_route - 3 patch sets | 21:09 |
| clayg | rledisez: do you run with two proxy-logging middlewares in your pipeline? we got SUPER confused having statsd metrics for client facing responses mixed in with subrequests | 21:10 |
| clayg | we're splitting everything up right now - internal_client.conf is going to get it's own metrics namespace too - i'll let you know how it works out; but we expect it to be pretty great | 21:11 |
| rledisez | clayg: yes, we use the 2 proxy-logging instance | 21:11 |
| rledisez | clayg: I though statsd metrics were emited by the left instance of proxy-logging, which is not in the path of subrequests | 21:12 |
| rledisez | I guess I assumed wrong :) | 21:13 |
| clayg | timburke: ok, I think i got what I need - the first one will merge, no one is going have much input on "fixing" prefix if you want it... | 21:13 |
| clayg | so if there's any dicussion to have around "best practices" it should happen on the follow-on patch to update example configs | 21:13 |
| timburke | they'll *both* emit statsd metrics | 21:13 |
| clayg | rledisez: so we fixed something that made s3 requests parse a, c, o and NOW they emit metrics | 21:13 |
| clayg | i mean they always both emitted... it's a disastor | 21:14 |
| timburke | but you're probably already getting doubled-up metrics from things like SLO | 21:14 |
| clayg | ^ 👍 | 21:14 |
| clayg | 😬 | 21:14 |
| clayg | like transfer stats... the extra response codes and GET timings don't quite look so weird... | 21:15 |
| clayg | well - depending on what you think they mean | 21:15 |
| *** e0ne has quit IRC | 21:15 | |
| clayg | cause it wasn't too hard for us to get confused - so we're gunna break it up! | 21:15 |
| timburke | all right, sounds like we know what's happening with the two linked patches, and we'll expect more discussion to happen on a follow-up (and possibly at the ptg) | 21:17 |
| timburke | #topic py3 encryption bug and upgrades | 21:18 |
| *** openstack changes topic to "py3 encryption bug and upgrades (Meeting topic: swift)" | 21:18 | |
| timburke | #link https://review.opendev.org/#/c/742033/ | 21:18 |
| patchbot | patch 742033 - swift - py3: Work with proper native string paths in crypt... - 4 patch sets | 21:18 |
| timburke | so that patch now has the config option from p 742756 squashed-in | 21:18 |
| patchbot | https://review.opendev.org/#/c/742756/ - swift - crypto: Add config option to support rolling upgrades (ABANDONED) - 3 patch sets | 21:18 |
| *** Lucas_Gray has quit IRC | 21:18 | |
| timburke | i think the UpgradeImpact has reasonably useful/actionable information | 21:19 |
| timburke | i've still got some concerns about an effective checkpoint release for switching from py2 to py3, and what that'll mean once the fix gets backported to train and ussuri | 21:20 |
| timburke | but maybe those fears are misplaced? | 21:21 |
| clayg | timburke: this one is definitely on my list, i'm onboard with the direction - if anyone wants to merge it for me it'd be a big help! | 21:21 |
| timburke | i was just about to ask who could review, so i could get the backport ball rolling :-) | 21:21 |
| clayg | timburke: oh right; you left a comment about that? | 21:22 |
| clayg | nm, I don't see it - so I don't think I understand your concern | 21:23 |
| clayg | do you want to add it to the review; or what to see if I notice the same thing when reviewing? | 21:23 |
| *** rh-jelabarre has quit IRC | 21:23 | |
| timburke | just in general, i'm worried about the added complexity for operators in needing to upgrade swift while still on py2, then upgrade to py3. if someone's following ubuntu's repos for example, i'm not sure they're gonna have a good time | 21:24 |
| clayg | again; if anyone else besides timburke has this loaded in their head (seongsoocho?) i'd love some help shaking out anything else needed | 21:24 |
| timburke | we might want to divvy up upgrade testing, too, to make sure we're covering all the cases we care about | 21:25 |
| clayg | timburke: so the issue is for py3 rollings to go smoothly py3 by default writes v2 broken? so py2 reads will fail - unless everyone is on v3 before upgrading. | 21:25 |
| clayg | timburke: i only really have bandwidth enough to functionally test for py2 upgrade case - if someone wants to do more than that BEFORE we +A I'd be happy to hold off | 21:27 |
| timburke | yup. and there's no way (currently; i suppose we could add yet another option...) to configure new py3 to write something that old py2 knows how to read | 21:27 |
| clayg | do we have a community deployment that's waiting on this fix? or everyone is happily in the python version island with v2 and working fine? | 21:27 |
| timburke | ormandj first reported the bug; i think he's running on a patched ussuri to pick up the fix | 21:28 |
| clayg | timburke: ok so realistically you can't upgrade from a current version of swift running py2 to a newer version of swift running py3 even with this patch... so... what are we doing? 🤣 | 21:29 |
| timburke | chasing my tail, i think | 21:29 |
| clayg | for US writing v3 is a blocker to migration onto py3 - so we want this patch, the rolling-upgrade behavior is seemless and all we have to do is flip to v3 asap so when we finally upgrade to py3 it's a non issue | 21:30 |
| clayg | and people already on py3 are happy | 21:31 |
| clayg | they also have a similar; easy upgrade; switch config afterwards story | 21:31 |
| clayg | ok, so I think we'll just have to say "When switching from Python 2 to Python 3, first upgrade Swift while on Python 2, then upgrade to Python 3." is reasonable and also the best we can do. | 21:32 |
| timburke | cool -- so i probably *was* overly concerned and we just need to review it, land it, and backport it | 21:32 |
| timburke | i think i've got what i need then. thanks for offering to review, clayg! again, if anyone else has a chance to review/test it, that'd be much appreciated | 21:33 |
| timburke | #topic shrinking and overlapping shard ranges | 21:33 |
| *** openstack changes topic to "shrinking and overlapping shard ranges (Meeting topic: swift)" | 21:33 | |
| clayg | thanks to the awesome mattoliverau for spotting bugs (i haven't written the unittest yet; pls help) | 21:34 |
| timburke | so when i originally added this, i wanted to make sure we had some consensus about which path forward we ought to take | 21:34 |
| timburke | but i think it's settled pretty well onto clayg's p 741721 | 21:34 |
| patchbot | https://review.opendev.org/#/c/741721/ - swift - add swift-manage-shard-ranges shink command - 4 patch sets | 21:34 |
| timburke | and i'm going to continue working with getting a probe test to exercise it in p 744256 | 21:35 |
| patchbot | https://review.opendev.org/#/c/744256/ - swift - sharding: probe test to exercise manual shrinking - 1 patch set | 21:35 |
| timburke | (mostly extracted from p 738149) | 21:35 |
| patchbot | https://review.opendev.org/#/c/738149/ - swift - Have shrinking and sharded shards save all ranges ... - 5 patch sets | 21:35 |
| mattoliverau | thanks guys for looking into it, and breaking things, be it shrinking in general or related to the manual patch, it's good to start excersizing the shrinking code. | 21:36 |
| clayg | 👍 gunna be so great | 21:37 |
| timburke | i'm not actually sure we need much more discussion, just more review bandwidth :-D | 21:37 |
| clayg | REVEWAZ!!! | 21:37 |
| *** dklyle has joined #openstack-meeting | 21:37 | |
| clayg | mattoliverau: did revewaz; mattoliverau is awesome; be like mattoliverau | 21:38 |
| *** slaweq has quit IRC | 21:38 | |
| timburke | so, just one more last-minute topic | 21:38 |
| timburke | #topic libec upgrades | 21:38 |
| *** openstack changes topic to "libec upgrades (Meeting topic: swift)" | 21:38 | |
| clayg | also mattoliverau - do more reviews 😁 | 21:38 |
| mattoliverau | I might try and progress the shard range audit stuff to fix gaps, split brains and overlaps so we can more easily recover from these kinda issues. (even if it starts of manual). | 21:38 |
| mattoliverau | lol, I'll find more reviewing time :) | 21:38 |
| timburke | zaitcev's taken a look already, but i think it's probably worth a second set of eyes | 21:38 |
| timburke | p 738959 and p 739164 | 21:39 |
| patchbot | https://review.opendev.org/#/c/738959/ - liberasurecode - Be willing to write fragments with legacy crc - 2 patch sets | 21:39 |
| patchbot | https://review.opendev.org/#/c/739164/ - swift - ec: Add an option to write fragments with legacy crc - 1 patch set | 21:39 |
| clayg | mattoliverau: I love that - if we can get where we automatically and confidently fix overlapping shard ranges it's gunna be WAY easier to start electing leaders to automatically shard 😍 | 21:39 |
| mattoliverau | +100 | 21:39 |
| timburke | heck, we might even get to the point where we say, yeah, replica 0's probably good enough :-) | 21:40 |
| clayg | i'm still scared of this patch; kota_ has always stronger at libec that I've been - for me ideally this can merge without me thinking about it anymore | 21:40 |
| clayg | it's supposed to be a similar "upgrade nothing changes; then flip the switch" change? | 21:40 |
| kota_ | i may take time to review on ec patch. | 21:41 |
| clayg | only caveat is you also have to repackage a new libec/pyeclib | 21:41 |
| kota_ | is that ok if I will do that in this week? | 21:41 |
| timburke | kota_, that'd be great, thanks! | 21:41 |
| clayg | kota_: I don't have any timeline | 21:41 |
| kota_ | ok. will try it. | 21:42 |
| timburke | clayg, no pyeclib changes necessary, given the current state of the patch | 21:42 |
| clayg | agree; if we have update/progress to show by next meeting that's a huge win | 21:42 |
| clayg | timburke: i'm not sure WE can release a new libec w/o a new pyeclib - but that could easily just be our fault 🤷 | 21:42 |
| clayg | and I could be wrong - i haven't looked at the dependency chain | 21:42 |
| clayg | they may already be independent/orthogonal | 21:43 |
| timburke | there is a bit of a detection problem for upgrades: you *may* need to tell new swift to write legacy fragments... or that may be the exact wrong thing to do :-( | 21:43 |
| clayg | the default isn't rolling-upgrade compatible? | 21:43 |
| timburke | ... it is so long as you aren't touching libec... | 21:44 |
| * clayg tries again to not think about this anymore | 21:44 | |
| timburke | the trouble is, we don't know what the upgrade-compatible behavior will be. there's a chance (depending on library load order) that you were already writing zlib crcs | 21:45 |
| clayg | oh RIGHT! 🤮 | 21:45 |
| timburke | fwiw, i wrote up p 744078 to try to help operators figure out what the right thing to do is (and to try out the audit-watchers interface) | 21:46 |
| clayg | timburke: you go LOOKING for this bugs 😡 you have a nose for trouble | 21:46 |
| patchbot | https://review.opendev.org/#/c/744078/ - swift - watchers: Add EC stat gatherer - 1 patch set | 21:46 |
| clayg | except we don't have audit watchers 🤷♂️ | 21:47 |
| timburke | well, it's all based on the tool i added to https://bugs.launchpad.net/swift/+bug/1886088, so you could still just use that :P | 21:48 |
| openstack | Launchpad bug 1886088 in OpenStack Object Storage (swift) "Mixed versions of liberasurecode leads to quarantined fragments" [Undecided,In progress] | 21:48 |
| *** ykatabam has joined #openstack-meeting | 21:48 | |
| clayg | 👍 and it sounds like "figuring out which version you're on" needs to happen before you upgrade so you can set the default for the new swift version | 21:49 |
| timburke | yup | 21:49 |
| timburke | all right, those are the major things i wanted to cover | 21:49 |
| timburke | #topic open discussion | 21:49 |
| *** openstack changes topic to "open discussion (Meeting topic: swift)" | 21:49 | |
| timburke | what else should we discuss? | 21:49 |
| *** yamamoto has joined #openstack-meeting | 21:49 | |
| clayg | FYI: we're going to push waterfall-ec into the pipeline so we can do more testing in staging and production | 21:52 |
| clayg | thanks for everyone's help with the design; i'm happy with where it ended up | 21:52 |
| clayg | timburke: did a bunch of cleanup I was able to squash in and mattoliverau found a few more splinters | 21:52 |
| timburke | it's gonna be so great | 21:52 |
| clayg | but ultimately I think the results look good and the code is going to be fine | 21:52 |
| mattoliverau | thanks for all your hard work on it clayg! | 21:53 |
| clayg | we'll probably start to push to get some of it landed and I think that's also going to be fine - I hope folks try it out after their next upgrade! | 21:53 |
| clayg | timburke: do you have the socket/accept/worker patch handy? | 21:53 |
| timburke | not sure if anyone else has seen this, but we noticed some lumpy request distribution between proxy-server workers -- resolution seems to be to bind a listen socket per-worker and let the kernel do a better job distributing work | 21:54 |
| timburke | i was just about to mention it! | 21:54 |
| clayg | 😍 | 21:54 |
| timburke | #link https://review.opendev.org/#/c/745603/ | 21:54 |
| patchbot | patch 745603 - swift - Bind a new socket per-worker - 3 patch sets | 21:54 |
| rledisez | timburke: yes, we observed that. it especially important with EC as the process gets CPU bound | 21:54 |
| clayg | ^^^ so much this | 21:54 |
| *** yamamoto has quit IRC | 21:54 | |
| *** vishalmanchanda has quit IRC | 21:55 | |
| timburke | i still haven't gotten around to testing it not-in-my-dev-VM, though -- debating about applying it on just one machine in prod to see that the distribution gets fixed | 21:56 |
| *** raildo has quit IRC | 21:57 | |
| clayg | 🤔 | 21:57 |
| timburke | all right, we're about out of time | 21:58 |
| timburke | thank you all for coming, and thank you for working on swift! | 21:58 |
| clayg | 👋 | 21:58 |
| timburke | #endmeeting | 21:59 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 21:59 | |
| openstack | Meeting ended Wed Aug 12 21:59:02 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:59 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-08-12-21.00.html | 21:59 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-08-12-21.00.txt | 21:59 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-08-12-21.00.log.html | 21:59 |
| *** kopecmartin is now known as kopecmartin|pto | 22:16 | |
| *** markvoelker has joined #openstack-meeting | 22:30 | |
| *** markvoelker has quit IRC | 22:35 | |
| *** rcernin has joined #openstack-meeting | 22:43 | |
| *** haleyb has quit IRC | 22:55 | |
| *** haleyb has joined #openstack-meeting | 22:55 | |
| *** rfolco has quit IRC | 23:04 | |
| *** moguimar has quit IRC | 23:05 | |
| *** carloss has quit IRC | 23:11 | |
| *** tosky has quit IRC | 23:13 | |
| *** ayoung has quit IRC | 23:17 | |
| *** ayoung has joined #openstack-meeting | 23:18 | |
| *** apetrich has quit IRC | 23:55 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!