Wednesday, 2021-03-24

*** mlavalle has quit IRC00:03
*** tosky has quit IRC00:09
*** macz_ has joined #openstack-meeting00:42
*** macz_ has quit IRC00:47
*** timburke__ has quit IRC00:47
*** timburke__ has joined #openstack-meeting00:48
*** jmasud has quit IRC01:01
*** manpreet has quit IRC01:02
*** rcernin has quit IRC01:34
*** baojg has joined #openstack-meeting01:34
*** rcernin has joined #openstack-meeting01:45
*** rcernin has quit IRC01:46
*** rcernin has joined #openstack-meeting01:46
*** rh-jelabarre has quit IRC02:05
*** rcernin has quit IRC02:45
*** rcernin has joined #openstack-meeting02:57
*** rcernin has quit IRC02:57
*** rcernin has joined #openstack-meeting02:58
*** manpreet has joined #openstack-meeting03:14
*** rcernin has quit IRC03:31
*** evrardjp has quit IRC03:33
*** evrardjp has joined #openstack-meeting03:33
*** rcernin has joined #openstack-meeting03:38
*** udesale has joined #openstack-meeting03:42
*** rcernin has quit IRC03:52
*** rcernin has joined #openstack-meeting03:52
*** rcernin has quit IRC03:53
*** rcernin has joined #openstack-meeting03:54
*** jamesdenton has quit IRC04:30
*** jamesden_ has joined #openstack-meeting04:31
*** armax has joined #openstack-meeting04:31
*** armax has quit IRC04:33
*** macz_ has joined #openstack-meeting04:41
*** vishalmanchanda has joined #openstack-meeting04:42
*** macz_ has quit IRC04:46
*** jamesden_ has quit IRC04:55
*** jamesdenton has joined #openstack-meeting04:56
*** macz_ has joined #openstack-meeting05:02
*** macz_ has quit IRC05:06
*** jamesdenton has quit IRC05:32
*** Luzi has joined #openstack-meeting05:40
*** jamesdenton has joined #openstack-meeting05:42
*** jmasud has joined #openstack-meeting05:44
*** jamesdenton has quit IRC05:52
*** psachin has joined #openstack-meeting05:52
*** jamesdenton has joined #openstack-meeting05:53
*** jmasud has quit IRC05:58
*** psachin has quit IRC06:01
*** zbr|rover has quit IRC06:08
*** zbr|rover has joined #openstack-meeting06:11
*** jmasud has joined #openstack-meeting06:27
*** ralonsoh has joined #openstack-meeting06:38
*** baojg has quit IRC06:43
*** baojg has joined #openstack-meeting06:44
*** dklyle has quit IRC06:49
*** psachin has joined #openstack-meeting06:50
*** jmasud has quit IRC06:55
*** jmasud has joined #openstack-meeting07:00
*** lpetrut has joined #openstack-meeting07:06
*** jmasud has quit IRC07:10
*** cgoncalves has quit IRC07:18
*** cgoncalves has joined #openstack-meeting07:20
*** cgoncalves has quit IRC07:20
*** cgoncalves has joined #openstack-meeting07:21
*** csatari has quit IRC07:25
*** csatari has joined #openstack-meeting07:28
*** rcernin has quit IRC08:09
*** rpittau|afk is now known as rpittau08:19
*** dsariel has joined #openstack-meeting08:21
*** ricolin has quit IRC08:27
*** ricolin has joined #openstack-meeting08:32
*** tosky has joined #openstack-meeting08:41
*** rcernin has joined #openstack-meeting08:52
*** rcernin has quit IRC08:57
*** vishalmanchanda has quit IRC09:01
*** abhishekk is now known as abhishekk|away09:03
*** macz_ has joined #openstack-meeting09:04
*** ociuhandu has joined #openstack-meeting09:06
*** rcernin has joined #openstack-meeting09:08
*** macz_ has quit IRC09:09
*** ociuhandu has quit IRC09:10
*** ociuhandu has joined #openstack-meeting09:17
*** e0ne has joined #openstack-meeting09:19
*** e0ne has quit IRC09:22
*** ociuhandu has quit IRC09:22
*** e0ne has joined #openstack-meeting09:24
*** e0ne has quit IRC09:24
*** ociuhandu has joined #openstack-meeting09:27
*** rcernin has quit IRC09:32
*** ociuhandu has quit IRC09:36
*** vishalmanchanda has joined #openstack-meeting09:42
*** ociuhandu has joined #openstack-meeting09:42
*** ociuhandu has quit IRC09:48
*** dsariel has quit IRC09:54
*** e0ne has joined #openstack-meeting09:56
*** rcernin has joined #openstack-meeting09:57
*** strigazi has joined #openstack-meeting09:58
*** ociuhandu has joined #openstack-meeting10:01
*** rcernin has quit IRC10:02
*** ociuhandu has quit IRC10:02
*** ociuhandu has joined #openstack-meeting10:03
*** e0ne has quit IRC10:09
*** rcernin has joined #openstack-meeting10:12
*** baojg has quit IRC10:16
*** rcernin has quit IRC10:22
*** dsariel has joined #openstack-meeting10:42
*** ociuhandu has quit IRC10:46
*** ociuhandu has joined #openstack-meeting10:47
*** ociuhandu has quit IRC10:54
*** oneswig has joined #openstack-meeting10:56
oneswig#startmeeting scientific-sig11:01
openstackMeeting started Wed Mar 24 11:01:18 2021 UTC and is due to finish in 60 minutes.  The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot.11:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.11:01
*** openstack changes topic to " (Meeting topic: scientific-sig)"11:01
openstackThe meeting name has been set to 'scientific_sig'11:01
*** rcernin has joined #openstack-meeting11:01
*** ociuhandu has joined #openstack-meeting11:01
*** rcernin has quit IRC11:01
oneswighi11:03
oneswigNo set agenda for today, alas11:04
*** rcernin has joined #openstack-meeting11:04
*** ociuhandu has quit IRC11:07
eliaswimmerhi11:08
oneswigHi eliaswimmer, welcome11:08
oneswigI didn't set any agenda alas :-(11:08
oneswigSmall update, have a contact about Magic Castle which I'm hoping to involve to present their JupyterHub approach soon11:09
oneswig(nothing set yet)11:09
oneswigAny news from you eliaswimmer?11:13
oneswigOK - lets close for today, and plan ahead for items for next time11:17
eliaswimmerNot really, sorry.11:17
oneswigNice to see you anyway :-) until next time11:17
oneswig#endmeeting11:18
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"11:18
openstackMeeting ended Wed Mar 24 11:18:21 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)11:18
openstackMinutes:        http://eavesdrop.openstack.org/meetings/scientific_sig/2021/scientific_sig.2021-03-24-11.01.html11:18
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/scientific_sig/2021/scientific_sig.2021-03-24-11.01.txt11:18
openstackLog:            http://eavesdrop.openstack.org/meetings/scientific_sig/2021/scientific_sig.2021-03-24-11.01.log.html11:18
*** ociuhandu has joined #openstack-meeting11:18
*** oneswig has quit IRC11:20
*** martial_ has joined #openstack-meeting11:20
*** baojg has joined #openstack-meeting11:29
*** baojg has quit IRC11:29
*** baojg has joined #openstack-meeting11:30
*** baojg has quit IRC11:30
*** baojg has joined #openstack-meeting11:31
*** baojg has quit IRC11:31
*** baojg has joined #openstack-meeting11:31
*** baojg has quit IRC11:32
*** baojg has joined #openstack-meeting11:32
*** baojg has quit IRC11:33
*** baojg has joined #openstack-meeting11:33
*** ociuhandu has quit IRC11:33
*** baojg has quit IRC11:33
*** macz_ has joined #openstack-meeting11:41
*** macz_ has quit IRC11:45
*** verdurin has joined #openstack-meeting11:48
*** rcernin has quit IRC11:58
*** macz_ has joined #openstack-meeting12:01
*** ociuhandu has joined #openstack-meeting12:03
*** ociuhandu has quit IRC12:04
*** ociuhandu has joined #openstack-meeting12:05
*** macz_ has quit IRC12:06
*** rcernin has joined #openstack-meeting12:06
*** ociuhandu has joined #openstack-meeting12:11
*** dsariel has quit IRC12:17
*** rh-jelabarre has joined #openstack-meeting12:20
*** rcernin has quit IRC12:22
*** ociuhandu has quit IRC12:23
*** ociuhandu has joined #openstack-meeting12:32
*** ociuhandu has quit IRC12:39
*** ociuhandu has joined #openstack-meeting12:40
*** psachin has quit IRC12:50
*** ociuhandu has quit IRC12:55
*** ociuhandu has joined #openstack-meeting13:01
*** ociuhandu has quit IRC13:02
*** ociuhandu has joined #openstack-meeting13:02
*** macz_ has joined #openstack-meeting13:03
*** macz_ has quit IRC13:08
*** dsariel has joined #openstack-meeting13:09
*** jamesdenton has quit IRC13:13
*** jamesdenton has joined #openstack-meeting13:13
*** ociuhandu has quit IRC13:16
*** e0ne has joined #openstack-meeting13:18
*** ociuhandu has joined #openstack-meeting13:25
*** belmoreira has joined #openstack-meeting13:27
*** ociuhandu has quit IRC13:30
*** dansmith has quit IRC13:31
*** ociuhandu has joined #openstack-meeting13:35
*** dansmith has joined #openstack-meeting13:38
*** ociuhandu has quit IRC13:41
*** ociuhandu has joined #openstack-meeting13:53
*** martial_ has quit IRC14:00
*** lajoskatona has joined #openstack-meeting14:00
*** liuyulong has joined #openstack-meeting14:01
lajoskatonaliuyulong: Hi, do we have l3 meeting today?14:01
liuyulongHi14:02
liuyulongyes14:02
lajoskatonaliuyulong: cool14:02
liuyulong#startmeeting neutron_l314:02
openstackMeeting started Wed Mar 24 14:02:24 2021 UTC and is due to finish in 60 minutes.  The chair is liuyulong. Information about MeetBot at http://wiki.debian.org/MeetBot.14:02
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:02
*** openstack changes topic to " (Meeting topic: neutron_l3)"14:02
openstackThe meeting name has been set to 'neutron_l3'14:02
liuyulongSorry, a bit late.14:02
lajoskatonaliuyulong: I was late ~8 mins yesterday from all meetings :-)14:03
liuyulonglajoskatona, lol14:03
liuyulongOK, let's start.14:03
liuyulong#topic Announcements14:04
*** openstack changes topic to "Announcements (Meeting topic: neutron_l3)"14:04
liuyulong#link https://etherpad.opendev.org/p/neutron-xena-ptg14:04
*** ociuhandu has quit IRC14:04
*** ociuhandu has joined #openstack-meeting14:04
liuyulongWe are at the end of W dev cycle. Now we hve the X ptg on the way.14:04
liuyulong#link https://doodle.com/poll/cc2ste3emzw7ekrh?utm_source=poll&utm_medium=link14:05
liuyulongThis is the time slot for neutron Xena PTG.14:06
liuyulongSo two things here:14:06
liuyulong1. add your topic to the etherpad14:06
*** armax has joined #openstack-meeting14:09
*** liuyulong_ has joined #openstack-meeting14:10
liuyulong_Sorry, I lost the connection...14:10
*** liuyulong has quit IRC14:10
liuyulong_1. add your topic to the etherpad14:11
liuyulong_2. choose the time available based on your time zone14:11
slaweqhi14:11
slaweqsorry for being late14:11
liuyulong_Two things (repeated...)14:11
liuyulong_It's a virtual PTG again.14:12
liuyulong_Hope to see you guys face to face someday. : )14:12
liuyulong_slaweq, Hi14:12
liuyulong_OK, no more announcement from me now.14:12
liuyulong_Let's move on.14:12
*** liuyulong_ is now known as liuyulong14:12
liuyulong#topic Bugs14:13
*** openstack changes topic to "Bugs (Meeting topic: neutron_l3)"14:13
liuyulong#link http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020903.html14:13
liuyulong#link http://lists.openstack.org/pipermail/openstack-discuss/2021-March/021064.html14:14
* haleyb wanders in late14:14
liuyulongThe first one14:15
liuyulongSorry, wrong link14:15
liuyulong#link http://lists.openstack.org/pipermail/openstack-discuss/2021-March/021230.html14:15
liuyulong#link http://lists.openstack.org/pipermail/openstack-discuss/2021-March/021064.html14:16
liuyulongThese two are the last two weeks bug list.14:16
liuyulong#link https://bugs.launchpad.net/neutron/+bug/191910714:16
openstackLaunchpad bug 1919107 in neutron "[L3][L2 pop] Is it necessary to enforce enabling the option arp_responder and l2_population?" [High,Confirmed]14:16
liuyulongThis is a real long story.14:17
liuyulong#link https://review.opendev.org/c/openstack/neutron/+/77882014:17
liuyulongThese two config are enforced for the patch https://review.opendev.org/c/openstack/neutron/+/60133614:18
liuyulongBut as we can see in the 601336, it will not be a good approch to fix the problem in that way. So it will not be merged.14:18
liuyulongBut, we have done some pre works.14:19
liuyulongSome of them casues the bug of https://bugs.launchpad.net/neutron/+bug/191676114:19
openstackLaunchpad bug 1916761 in Ubuntu Cloud Archive ussuri "[dvr] bound port permanent arp entries never deleted" [High,Fix committed]14:19
liuyulongSo, we need to revert them since the final approch 601336 will not be accepted.14:20
liuyulongIn the revert patch https://review.opendev.org/c/openstack/neutron/+/778820, I put all the work together.14:20
haleybliuyulong: i was just writing a comment in that one, just noticed something with the release note, otherwise fine14:21
liuyulongThe import thing is on this https://review.opendev.org/c/openstack/neutron/+/752248.14:21
liuyulonghaleyb, Sure, I will update14:21
liuyulongIt has some conflicts on that 752248.14:22
liuyulong#link https://review.opendev.org/c/openstack/neutron/+/778820/7/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_dvr_neutron_agent.py14:22
liuyulong#link https://review.opendev.org/c/openstack/neutron/+/752248/14/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_dvr_neutron_agent.py14:23
liuyulongHope my change is in the right way.14:23
liuyulongIMO, we should not rush to merge this patch. I will run "check experimental" for the neutron-tempest-plugin-dvr-multinode-scenario job.14:25
liuyulongTo verify the east-west traffic of DVR.14:26
liuyulongIt is failed now, but it's not the DVR case.14:26
liuyulong#link https://497849cb3a7ea1ef6b00-cfc9a6b6175848be06351d4fb9f59b07.ssl.cf5.rackcdn.com/778820/7/experimental/neutron-tempest-plugin-dvr-multinode-scenario/58d1c94/testr_results.html14:26
lajoskatonait failed now, not sure if related (test_dns_domain_and_name failed )14:26
lajoskatonahttps://497849cb3a7ea1ef6b00-cfc9a6b6175848be06351d4fb9f59b07.ssl.cf5.rackcdn.com/778820/7/experimental/neutron-tempest-plugin-dvr-multinode-scenario/58d1c94/testr_results.html14:26
lajoskatonaok, sorry, seems my irc client delayed things...14:27
liuyulonglajoskatona, np, maybe it was my message just running slowly on the internet...14:29
liuyulongAnother thing related to this is the flows for the l2pop and arp responder.14:30
liuyulong#link https://bugs.launchpad.net/networking-bagpipe/+bug/191910414:30
openstackLaunchpad bug 1919104 in BaGPipe "[bagpipe_bgpvpn] restart ovs agent only will lose flows" [Undecided,In progress] - Assigned to LIU Yulong (dragon889)14:30
liuyulongThis ovs-agent extension bagpipe_bgpvpn will handle the l2pop tunnel direct flow and arp responder flow.14:31
liuyulongBut if arp_responder and l2_population are enforced, then [bagpipe_bgpvpn] actually just do things duplicated to the ovs-agent built-in functions.14:32
liuyulongSo, after the revert, we can disable the arp_responder and l2_population for ovs-agent.14:33
*** ociuhandu has quit IRC14:33
liuyulongBut, one more thing is ovs-agent will clean the flows not installed by it. Another conflict with [bagpipe_bgpvpn].14:34
liuyulong#link https://review.opendev.org/c/openstack/neutron/+/782345/14:34
liuyulongSo I created this patch for the bug 1919104, which will skip the l2pop and arp responder stale flow clean during ovs-agent restart if they are not enabled.14:35
openstackbug 1919104 in BaGPipe "[bagpipe_bgpvpn] restart ovs agent only will lose flows" [Undecided,In progress] https://launchpad.net/bugs/1919104 - Assigned to LIU Yulong (dragon889)14:35
liuyulongOK, the final thing about this is I will take some time to consider the alternative of the patch 601336.14:37
liuyulongMaybe we can discuss it in the X ptg.14:38
*** lpetrut has quit IRC14:39
liuyulongNext14:39
liuyulong#link https://bugs.launchpad.net/neutron/+bug/192097514:39
openstackLaunchpad bug 1920975 in neutron "neutron dvr should lower proxy_delay when using proxy_arp" [Undecided,In progress] - Assigned to Edward Hope-Morley (hopem)14:39
liuyulong#link https://review.opendev.org/c/openstack/neutron/+/78257014:39
liuyulongI see the patch here.14:39
liuyulongWith some comments, maybe the kernel config tunning can workaround this problem. https://review.opendev.org/c/openstack/neutron/+/782570/2/neutron/agent/l3/dvr_fip_ns.py#19614:40
liuyulongNext14:41
liuyulong#link https://bugs.launchpad.net/neutron/+bug/192000114:41
openstackLaunchpad bug 1920001 in neutron "Creation of floating ip with new rbac policies fails" [High,In progress] - Assigned to Slawek Kaplonski (slaweq)14:41
liuyulongIt's in the deputy's list.14:42
liuyulongAnd the patch was merged.14:42
liuyulong#link https://review.opendev.org/c/openstack/neutron-lib/+/78162514:42
liuyulongThis is proper fix we can see in the LP comments from slaweq.14:43
*** Luzi has quit IRC14:43
slaweqyes, it's all related to the new policies14:43
liuyulongLooks simple, but a new version of neutron-lib should be added for neutron to get the fix work. That will take some time. :)14:44
liuyulongAlright, no more bugs from me now.14:45
liuyulongOK, let's move on.14:47
liuyulong#topic On demand agenda14:47
*** openstack changes topic to "On demand agenda (Meeting topic: neutron_l3)"14:47
lajoskatonaIf you have a few minute please check https://review.opendev.org/c/openstack/neutron-specs/+/76733714:48
lajoskatonathe spec for BFD14:48
lajoskatonaI run into several issues with os-ken BFD, and to make it work I had to "move" it to an l2 extension which starts os-ken app14:48
lajoskatonaI tried to summarize the thing in the spec's backend part14:49
liuyulongMaybe this can be a new agent which is not only for "L3 routes", but also for internal connection detect, pr neutron router to physical router.14:51
liuyulongs/pr/or14:51
lajoskatonapossible14:51
*** ociuhandu has joined #openstack-meeting14:51
liuyulongCould the command "ovs-vsctl" work to control the ovsdb for BFD?14:52
lajoskatonathe main issue with os/ken bfd is that it needs to connect to ovsdb to fetch some dp related thing so even in new agent it need a new bridge to add ports dedicated to BFD as can't connect to the same bridge with 2 controller14:53
lajoskatonanot sure if this is os-ken limitation os vsdb limitation14:53
lajoskatonaBFD in os-ken sends packet independently from ovs, but for administration it uses dp information14:54
liuyulong#link https://review.opendev.org/c/openstack/neutron-specs/+/767337/13/specs/wallaby/bfd_support.rst@54714:54
lajoskatonabut as I had now a more or less working PoC I realized that the code has strange conditionals, so another option is that I can totally clean dp related things from it, I think nobody uses BFd in openstack14:55
liuyulongFor the spec here, my idea is to add the ovs command line examples which is equal to os-ken "app" to explain the real usage.14:55
lajoskatonayeah I checked that "manually" and checked ovs' BFD implementation, perhaps that can be simpler14:57
lajoskatonabut even in that way touching ovs from l3-agent can be strange14:58
*** ociuhandu has quit IRC14:58
liuyulongGive you some clue, maybe the BFD implementation in the OVN can help. : )14:58
*** dklyle has joined #openstack-meeting14:58
*** dklyle has quit IRC14:59
liuyulongOK, time is up.14:59
liuyulongLet's end here.14:59
lajoskatonaliuyulong: I can check14:59
lajoskatonathanks14:59
liuyulongThank you guys for attending.14:59
liuyulong#endmeeting14:59
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"14:59
openstackMeeting ended Wed Mar 24 14:59:43 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:59
openstackMinutes:        http://eavesdrop.openstack.org/meetings/neutron_l3/2021/neutron_l3.2021-03-24-14.02.html14:59
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/neutron_l3/2021/neutron_l3.2021-03-24-14.02.txt14:59
openstackLog:            http://eavesdrop.openstack.org/meetings/neutron_l3/2021/neutron_l3.2021-03-24-14.02.log.html14:59
lajoskatonaBye14:59
liuyulongBye14:59
*** liuyulong has quit IRC15:00
*** ociuhandu has joined #openstack-meeting15:04
*** ociuhandu has quit IRC15:08
*** macz_ has joined #openstack-meeting15:09
*** macz_ has quit IRC15:09
*** macz_ has joined #openstack-meeting15:10
*** dklyle has joined #openstack-meeting15:10
*** ociuhandu has joined #openstack-meeting15:22
*** ociuhandu has quit IRC15:26
*** stand has joined #openstack-meeting15:29
*** mlavalle has joined #openstack-meeting15:39
*** jmasud has joined #openstack-meeting15:46
*** markmcclain has joined #openstack-meeting15:53
*** ociuhandu has joined #openstack-meeting16:03
*** gyee has joined #openstack-meeting16:23
*** ociuhandu has quit IRC16:26
*** timburke_ has joined #openstack-meeting16:39
*** timburke__ has quit IRC16:42
*** ociuhandu has joined #openstack-meeting16:52
*** ociuhandu_ has joined #openstack-meeting16:57
*** jmasud has quit IRC16:59
*** ociuhandu has quit IRC17:00
*** ociuhandu_ has quit IRC17:01
*** ociuhandu has joined #openstack-meeting17:07
*** ociuhandu has quit IRC17:11
*** jmasud has joined #openstack-meeting17:13
*** udesale has quit IRC17:19
*** jmasud has quit IRC17:21
*** rpittau is now known as rpittau|afk17:24
*** jmasud has joined #openstack-meeting17:29
*** jmasud has quit IRC17:37
*** jamesdenton has quit IRC18:04
*** jamesdenton has joined #openstack-meeting18:05
*** rcernin has joined #openstack-meeting18:19
*** rcernin has quit IRC18:23
*** timburke_ has quit IRC18:35
*** timburke__ has joined #openstack-meeting18:35
*** ralonsoh has quit IRC19:07
*** jmasud has joined #openstack-meeting19:11
*** jmasud has quit IRC19:18
*** e0ne has quit IRC19:51
*** belmoreira has quit IRC20:14
*** lajoskatona has quit IRC20:29
*** ociuhandu has joined #openstack-meeting20:43
*** ociuhandu has quit IRC20:48
*** jamesdenton has quit IRC20:52
*** jamesden_ has joined #openstack-meeting20:52
timburke__#startmeeting swift21:00
openstackMeeting started Wed Mar 24 21:00:03 2021 UTC and is due to finish in 60 minutes.  The chair is timburke__. Information about MeetBot at http://wiki.debian.org/MeetBot.21:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:00
*** openstack changes topic to " (Meeting topic: swift)"21:00
openstackThe meeting name has been set to 'swift'21:00
timburke__who's here for the swift meeting?21:00
seongsoochoo/21:00
kota_o/21:00
*** acoles has joined #openstack-meeting21:00
rledisezo/21:01
*** rcernin has joined #openstack-meeting21:01
mattoliverauo/21:01
*** rcernin has quit IRC21:01
*** rcernin has joined #openstack-meeting21:02
acoleso/21:03
timburke__as usual, the agenda's at https://wiki.openstack.org/wiki/Meetings/Swift21:03
timburke__first up21:03
timburke__#topic PTG21:03
*** openstack changes topic to "PTG (Meeting topic: swift)"21:03
timburke__if you haven't yet, please register and respond on the doodle poll for meeting times21:04
timburke__#link https://april2021-ptg.eventbrite.com/21:04
timburke__#link https://doodle.com/poll/cnr3qb7y2afiad9v21:04
*** jmasud has joined #openstack-meeting21:05
timburke__and we've got an etherpad for gathering topics21:05
timburke__#link https://etherpad.opendev.org/p/swift-ptg-xena21:05
timburke__i really ought to book (virtual) rooms and such today21:06
mattoliverauthen doodle poll needs to be a priority.. so please do it if you haven't yet!21:06
timburke__any questions or comments on it? less than a month away now!21:08
*** jmasud has quit IRC21:08
*** zaitcev has joined #openstack-meeting21:08
*** jmasud has joined #openstack-meeting21:10
timburke__all right then21:11
timburke__on to some updates!21:11
timburke__#topic relinker21:11
*** openstack changes topic to "relinker (Meeting topic: swift)"21:11
timburke__so we did another part power at nvidia! just finished up this morning21:11
timburke__went better than the last time -- we're definitely getting better at it21:12
timburke__but there are still some rough edges21:12
*** jmasud has quit IRC21:12
timburke__notably, we saw a bunch of extra (empty) handoff partitions getting created because we ran the relinker more than once, and it'd want to do a rehash on a partition that never got any links put in it21:14
*** jmasud has joined #openstack-meeting21:14
timburke__i think https://review.opendev.org/c/openstack/swift/+/781908 should deal with it -- basically, if we call get_hashes for a partition that doesn't exist, we should be able to return early21:14
*** jmasud has quit IRC21:16
timburke__since last week, we had a bunch of great stuff land -- operators can now target specific policies and even partitions, and we've merged acoles's great refactoring to make relink and cleanup reuse a lot of the same code21:16
*** jmasud has joined #openstack-meeting21:18
timburke__still to come, mattoliverau has some more improvements to progress visibility, and i rebased my patch to parallelize the relinker21:18
mattoliveraunice21:18
timburke__if you're interested in part power increases, those are likely the best places to jump in!21:19
seongsoochocool. the relinker is getting better21:19
zaitcevSo how high are you now?21:19
zaitcevAnd I don't mean marijuana21:19
timburke__only 18. we've done it twice, and intend to do at least two more21:19
acoleswe also learnt some stuff doing this increase after a previous one. For example, we discovered we had some tombstones that had not been cleaned up in their original partitions, so during this relink we had *two* locations trying to create a link in the same new location21:19
zaitcevHmm. So even 18 is not enough for you?21:20
acoleswe just love doing it :D21:20
timburke__(speaking of, https://review.opendev.org/c/openstack/swift/+/781591 is a first crack at being able to step by more than one)21:20
zaitcevThat's like 262000 partitions21:20
*** jmasud has quit IRC21:20
timburke__if you assume 10k disks, that's still only 26/disk ;-)21:20
mattoliverauoh cool, jump more then 1, interesting21:21
seongsoochoone question for relink. in my case, I forgot to increase reclaim age before the relink, so there are a lots of cleanup job is failed..21:22
seongsoochoI fi21:22
timburke__as long as the failures are for tombstones (.ts files) i expect it should be mostly fine21:23
seongsoochoI finished the part power job and the object replicator is running again. The failed diskfiles will be delete? Most of diskfiles are .data and it was deleted in other node during relink21:24
*** jamesden_ has quit IRC21:25
*** jamesdenton has joined #openstack-meeting21:25
timburke__the .datas -- are the objects still in listings?21:27
seongsoochonop. It is not in list21:27
timburke__i think those will need to be manually cleaned up :-(21:28
timburke__though zaitcev's dark data audit watcher could come in handy when you upgrade21:28
zaitcevIt does not reattach them though21:30
zaitcevCan only delete21:30
zaitcevBut what the heck happened? Relinker misfired, or they were orphans even before partition power increase?21:30
timburke__fwiw, newer versions also make it so the relinker can respect the cluster's reclaim_age by pointing it at object-server.conf21:30
zaitcevOh, n/m I see the comment about reclaim age.21:31
mattoliverausounds like they were deletes that weren't propergated fully so delete makes sense.21:31
timburke__i'm guessing there was a delete after the increased ring went out but before the cleanup processed the partition. then a reclaim age passed. cleanup finally got to it, went to open the diskfile in the new location -- and cleaned up the tombstone before erroring21:32
timburke__...makes me think the relinker (when not given a config file) should use a reclaim_age that's intentionally longer than anyone would likely have it configured...21:33
zaitcevThat's going to be the first time ever watcher got useful to anyone then.21:33
zaitcevIf seongsoocho uses it.21:33
seongsoochoyes. during the relink , the object was deleted  in other 2 nodes but not deleted in one node (because relinker use that disk too heavy , so object server return error)21:33
timburke__peace of mind is still a valid use-case ;-)21:33
seongsoochozaitcev:  cool.21:33
timburke__all right21:34
timburke__#topic sharding and shrinking21:34
*** openstack changes topic to "sharding and shrinking (Meeting topic: swift)"21:34
*** jmasud has joined #openstack-meeting21:35
timburke__i know mattoliverau and acoles have continued working on this21:35
mattoliverauI've been playing with al's repair and anaylze chain, and it's pretty awesome.21:35
timburke__and mattoliverau, you put together an etherpad about how to tell that a shard is old enough to consider shrinking, yeah?21:36
timburke__#link https://etherpad.opendev.org/p/swift-shrinking-active-age21:36
mattoliverauoh yeah. The current shrinking blocker is ^21:36
acoleszaitcev pulled me up on some __hash__ kludge in that for which I am grateful - made it better21:36
acolesI finally got some ideas about tombstone reporting up https://review.opendev.org/c/openstack/swift/+/782832 - very much WIP though21:37
mattoliverauthere is an edge case that one primary is small and shards before the others, the new shards ranges become active and then being pretty empty will be considered candidates for shrinking. The etherpad explains current progress, problems, pain points.21:38
mattoliverauturns out what we thought should be an easy fix seems to get more and more complicated :P21:39
acolesmattoliverau: thanks for starting the etherpad21:39
mattoliverauSo if your interested, read, ask questions and comment in the etherpad.21:40
acolesmattoliverau: that's ditributed systems for you! ;)21:40
mattoliveraulol yeah :)21:40
mattoliverauin the meantime I wanted to give Als' repair stuff some love, because they'd also be great to have and have been up a long time.21:40
mattoliverauI plugged them into my WIP shard graph visualiser tool and now you can get a green path for the best path. Which is nice when testing als changes.21:41
timburke__sounds good. we'll be able to fix up busted sharding yet!21:41
timburke__#topic open discussion21:41
*** openstack changes topic to "open discussion (Meeting topic: swift)"21:41
timburke__what else should we bring up this week?21:41
kota_ah21:42
kota_one thing, it is not actually related to Swift21:42
kota_before a week from PTG, NVIDIA GTC will be held at online and I have a session to introduce my recent work.21:43
kota_just FYI.21:43
acoleskota_: great!21:43
timburke__cool! congrats!21:43
seongsoochokota_:  cool!!!!21:43
acolesdo you have a link / date/ time ?21:43
mattoliveraunice!21:43
kota_it's free so you can watch the pre-recorded video21:43
kota_the registration is at https://gtc21.event.nvidia.com/21:44
*** jmasud has quit IRC21:44
kota_it looks like the direct link to the video is not work unless someone registered to see.21:45
acoleskota_: thanks for sharing that21:45
kota_:D21:45
zaitcevkota_: thanks21:45
mattoliverau+121:45
zaitcevAlthough I'm a little sick of everything remote.21:45
kota_exactly, I like the face to face meeting we did in the past.21:46
timburke__yeah, me too. with a little luck, maybe we'll be able to actually see each other in person the next time21:46
mattoliverauyeah. but at least it makes it easier to attend conferences you might not be able to convince someone to send you too.. esp if your on the otherside of the world :)21:46
mattoliveraubut +100 for missing going to f2f conferences21:47
timburke__all right, i think i'll call it then21:49
timburke__thank you all for coming, and thank you for working on swift!21:49
timburke__#endmeeting21:49
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"21:49
openstackMeeting ended Wed Mar 24 21:49:29 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)21:49
openstackMinutes:        http://eavesdrop.openstack.org/meetings/swift/2021/swift.2021-03-24-21.00.html21:49
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/swift/2021/swift.2021-03-24-21.00.txt21:49
openstackLog:            http://eavesdrop.openstack.org/meetings/swift/2021/swift.2021-03-24-21.00.log.html21:49
*** zaitcev has left #openstack-meeting21:49
*** acoles has left #openstack-meeting21:49
*** rcernin has quit IRC22:11
*** jmasud has joined #openstack-meeting22:12
*** rcernin has joined #openstack-meeting22:26
*** rcernin has quit IRC22:31
*** jmasud has quit IRC22:32
*** rcernin has joined #openstack-meeting22:32
*** ganso has quit IRC22:41
*** vishalmanchanda has quit IRC22:41
*** ganso has joined #openstack-meeting22:44
*** slaweq has quit IRC23:02
*** jmasud has joined #openstack-meeting23:05
*** macz_ has quit IRC23:15
*** stand has quit IRC23:20
*** jamesdenton has quit IRC23:32
*** jamesdenton has joined #openstack-meeting23:33
*** gyee has quit IRC23:35

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!