Wednesday, 2021-01-27

*** yasufum has joined #openstack-meeting00:25
*** zbr has quit IRC00:27
*** zbr5 has joined #openstack-meeting00:27
*** ircuser-1 has quit IRC00:37
*** mlavalle has quit IRC00:58
*** ircuser-1 has joined #openstack-meeting01:01
*** jamesmcarthur has joined #openstack-meeting01:21
*** rh-jelabarre has quit IRC01:23
*** armax has quit IRC01:37
*** baojg has joined #openstack-meeting01:41
*** jmasud has quit IRC01:49
*** yasufum has quit IRC01:50
*** yasufum has joined #openstack-meeting01:54
*** rcernin has quit IRC01:54
*** jamesmcarthur has quit IRC01:55
*** jamesmcarthur has joined #openstack-meeting01:55
*** dklyle has quit IRC01:55
*** dklyle has joined #openstack-meeting01:57
*** jmasud has joined #openstack-meeting02:04
*** ircuser-1 has quit IRC02:12
*** ircuser-1 has joined #openstack-meeting02:18
*** baojg has quit IRC02:21
*** baojg has joined #openstack-meeting02:22
*** timburke__ has quit IRC02:29
*** timburke__ has joined #openstack-meeting02:30
*** armax has joined #openstack-meeting02:32
*** armax has quit IRC02:35
*** rcernin has joined #openstack-meeting02:38
*** rcernin has quit IRC02:43
*** jamesmcarthur has quit IRC02:44
*** jamesmcarthur has joined #openstack-meeting02:45
*** jamesmcarthur has quit IRC02:50
*** jamesmcarthur has joined #openstack-meeting02:50
*** jamesmcarthur has quit IRC02:50
*** jamesmcarthur has joined #openstack-meeting02:50
*** jamesmcarthur has quit IRC02:56
*** rcernin has joined #openstack-meeting03:05
*** rcernin has quit IRC03:12
*** rcernin has joined #openstack-meeting03:12
*** jmasud has quit IRC03:21
*** jamesmcarthur has joined #openstack-meeting03:21
*** ircuser-1 has quit IRC03:32
*** jamesmcarthur has quit IRC03:35
*** psachin has joined #openstack-meeting03:36
*** jmasud has joined #openstack-meeting03:42
*** ircuser-1 has joined #openstack-meeting03:45
*** jamesmcarthur has joined #openstack-meeting03:50
*** baojg has quit IRC03:54
*** jmasud has quit IRC03:55
*** baojg has joined #openstack-meeting03:56
*** baojg has quit IRC03:56
*** baojg has joined #openstack-meeting03:56
*** psahoo has joined #openstack-meeting03:57
*** baojg has quit IRC03:57
*** baojg has joined #openstack-meeting03:57
*** baojg has quit IRC03:58
*** baojg has joined #openstack-meeting03:58
*** baojg has quit IRC03:58
*** baojg has joined #openstack-meeting03:59
*** baojg has quit IRC03:59
*** baojg has joined #openstack-meeting04:00
*** baojg has quit IRC04:00
*** baojg has joined #openstack-meeting04:00
*** baojg has quit IRC04:01
*** baojg has joined #openstack-meeting04:01
*** baojg has quit IRC04:01
*** baojg has joined #openstack-meeting04:02
*** baojg has quit IRC04:02
*** baojg has joined #openstack-meeting04:03
*** baojg has quit IRC04:03
*** jamesden_ has quit IRC04:08
*** cmart has quit IRC04:08
*** PrinzElvis has quit IRC04:08
*** zigo has quit IRC04:08
*** ricolin has quit IRC04:11
*** PrinzElvis has joined #openstack-meeting04:14
*** zigo has joined #openstack-meeting04:14
*** dklyle has quit IRC04:24
*** baojg has joined #openstack-meeting04:28
*** ircuser-1 has quit IRC04:31
*** jamesden_ has joined #openstack-meeting04:33
*** jamesden_ has quit IRC04:36
*** jamesden_ has joined #openstack-meeting04:38
*** ircuser-1 has joined #openstack-meeting04:38
*** vishalmanchanda has joined #openstack-meeting04:41
*** jamesden_ has quit IRC04:54
*** jamesden_ has joined #openstack-meeting04:57
*** jamesmcarthur has quit IRC05:04
*** jamesmcarthur has joined #openstack-meeting05:04
*** jamesmcarthur has quit IRC05:13
*** jamesmcarthur has joined #openstack-meeting05:13
*** jamesmcarthur has quit IRC05:18
*** ircuser-1 has quit IRC05:31
*** evrardjp has quit IRC05:33
*** evrardjp has joined #openstack-meeting05:33
*** cmart has joined #openstack-meeting05:38
*** jmasud has joined #openstack-meeting05:39
*** jamesmcarthur has joined #openstack-meeting05:43
*** jamesmcarthur has quit IRC05:52
*** jmasud has quit IRC06:02
*** lbragstad has quit IRC06:03
*** lbragstad has joined #openstack-meeting06:03
*** jamesmcarthur has joined #openstack-meeting06:05
*** jamesmcarthur has quit IRC06:16
*** jamesmcarthur has joined #openstack-meeting06:16
*** jmasud has joined #openstack-meeting06:17
*** jmasud has quit IRC06:40
*** jamesmcarthur has quit IRC06:40
*** jamesmcarthur has joined #openstack-meeting06:41
*** gyee has quit IRC06:44
*** psachin has quit IRC06:45
*** jamesmcarthur has quit IRC06:46
*** psachin has joined #openstack-meeting07:00
*** e0ne has joined #openstack-meeting07:03
*** jamesmcarthur has joined #openstack-meeting07:11
*** psahoo has quit IRC07:27
*** e0ne has quit IRC07:30
*** lpetrut has joined #openstack-meeting07:32
*** aprice has quit IRC07:32
*** ralonsoh has joined #openstack-meeting07:34
*** whoami-rajat has joined #openstack-meeting07:34
*** aprice has joined #openstack-meeting07:34
*** psachin has quit IRC07:36
*** psachin has joined #openstack-meeting07:37
*** psahoo has joined #openstack-meeting07:41
*** whoami-rajat has quit IRC07:47
*** jamesmcarthur has quit IRC08:07
*** slaweq has joined #openstack-meeting08:08
*** jamesmcarthur has joined #openstack-meeting08:19
*** e0ne has joined #openstack-meeting08:27
*** jamesmcarthur has quit IRC08:27
*** rpittau|afk is now known as rpittau08:30
*** yamamoto has quit IRC08:33
*** psahoo has quit IRC08:37
*** jamesmcarthur has joined #openstack-meeting08:38
*** tosky has joined #openstack-meeting08:43
*** psahoo has joined #openstack-meeting08:52
*** rcernin has quit IRC08:53
*** ociuhandu has joined #openstack-meeting09:00
*** vishalmanchanda has quit IRC09:00
*** rcernin has joined #openstack-meeting09:04
*** jamesmcarthur has quit IRC09:12
*** yamamoto has joined #openstack-meeting09:17
*** yamamoto has quit IRC09:26
*** e0ne has quit IRC09:56
*** baojg has quit IRC10:04
*** baojg has joined #openstack-meeting10:09
*** jamesmcarthur has joined #openstack-meeting10:16
*** jmasud has joined #openstack-meeting10:21
*** jamesmcarthur has quit IRC10:22
*** rcernin has quit IRC10:28
*** vishalmanchanda has joined #openstack-meeting10:28
*** oneswig has joined #openstack-meeting10:52
oneswig#startmeeting scientific-sig11:00
openstackMeeting started Wed Jan 27 11:00:39 2021 UTC and is due to finish in 60 minutes.  The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot.11:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.11:00
*** openstack changes topic to " (Meeting topic: scientific-sig)"11:00
openstackThe meeting name has been set to 'scientific_sig'11:00
oneswigHello11:01
priteauHi oneswig11:06
oneswigHi priteau :-)11:06
priteauIt's quiet today11:06
oneswigvery - but I didn't set an agenda nor do much to publicise, alas11:06
oneswigOn the sudo issue, I just found an update in Stream - http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/Packages/sudo-1.8.29-6.el8_3.1.x86_64.rpm11:09
*** ianychoi_ has joined #openstack-meeting11:09
*** julianp has joined #openstack-meeting11:09
oneswigAn interesting case where Stream has the update - how long before CentOS 8 gets it?11:10
*** yamamoto has joined #openstack-meeting11:11
*** ianychoi__ has quit IRC11:12
oneswigpriteau: some interesting discussion on Slack about preemptibles.11:14
priteauyes, I pointed out to blair that the CERN code wasn't merged11:15
oneswigand thodoris has moved on, is that true?11:17
*** rcernin has joined #openstack-meeting11:17
priteauI believe so. His LinkedIn says his appointment with CERN ended in September11:18
oneswighmm, too bad11:19
priteauhttps://gitlab.cern.ch/ttsiouts11:19
priteauLast aardvark commit 6 months ago: https://gitlab.cern.ch/ttsiouts/aardvark11:19
priteauOr maybe this is the canonical repo: https://gitlab.cern.ch/cloud-infrastructure/aardvark11:20
*** baojg has quit IRC11:22
*** baojg has joined #openstack-meeting11:24
*** baojg has quit IRC11:24
*** baojg has joined #openstack-meeting11:24
*** baojg has quit IRC11:25
*** baojg has joined #openstack-meeting11:25
*** baojg has quit IRC11:26
*** baojg has joined #openstack-meeting11:26
*** baojg has quit IRC11:26
*** ociuhandu has quit IRC11:27
*** baojg has joined #openstack-meeting11:27
*** baojg has quit IRC11:27
*** ociuhandu has joined #openstack-meeting11:27
*** baojg has joined #openstack-meeting11:27
*** baojg has quit IRC11:28
oneswigI don't think there was anything else to raise.  Shall we close early?11:28
*** baojg has joined #openstack-meeting11:29
*** baojg has quit IRC11:29
*** ociuhandu has quit IRC11:30
*** baojg has joined #openstack-meeting11:30
*** ociuhandu has joined #openstack-meeting11:30
*** baojg has quit IRC11:30
*** baojg has joined #openstack-meeting11:31
*** baojg has quit IRC11:31
*** rcernin has quit IRC11:31
*** baojg has joined #openstack-meeting11:32
*** baojg has quit IRC11:32
*** baojg has joined #openstack-meeting11:32
*** baojg has quit IRC11:33
priteauaye11:34
oneswig#endmeeting11:40
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"11:40
openstackMeeting ended Wed Jan 27 11:40:47 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)11:40
openstackMinutes:        http://eavesdrop.openstack.org/meetings/scientific_sig/2021/scientific_sig.2021-01-27-11.00.html11:40
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/scientific_sig/2021/scientific_sig.2021-01-27-11.00.txt11:40
openstackLog:            http://eavesdrop.openstack.org/meetings/scientific_sig/2021/scientific_sig.2021-01-27-11.00.log.html11:40
oneswigthanks priteau :-)11:40
*** yasufum has quit IRC11:42
julianpSee y'all next time.11:43
*** julianp has quit IRC11:50
*** ricolin has joined #openstack-meeting11:55
*** jmasud has quit IRC12:09
*** ociuhandu_ has joined #openstack-meeting12:15
*** ociuhandu has quit IRC12:17
*** ociuhandu_ has quit IRC12:19
*** jamesmcarthur has joined #openstack-meeting12:20
*** ociuhandu has joined #openstack-meeting12:23
*** yamamoto has quit IRC12:24
*** jamesmcarthur has quit IRC12:27
*** ociuhandu has quit IRC12:28
*** raildo has joined #openstack-meeting12:30
*** gibi has quit IRC12:41
*** ricolin has quit IRC12:48
*** rh-jelabarre has joined #openstack-meeting12:58
*** ricolin has joined #openstack-meeting13:11
*** liuyulong has joined #openstack-meeting13:12
*** yamamoto has joined #openstack-meeting13:13
*** zbr5 has quit IRC13:20
*** zbr has joined #openstack-meeting13:21
*** yamamoto has quit IRC13:25
*** e0ne has joined #openstack-meeting13:35
*** hemna has quit IRC13:36
*** hemna has joined #openstack-meeting13:36
*** liuyulong has quit IRC13:58
*** liuyulong has joined #openstack-meeting13:59
liuyulong#startmeeting neutron_l314:00
openstackMeeting started Wed Jan 27 14:00:01 2021 UTC and is due to finish in 60 minutes.  The chair is liuyulong. Information about MeetBot at http://wiki.debian.org/MeetBot.14:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:00
*** openstack changes topic to " (Meeting topic: neutron_l3)"14:00
openstackThe meeting name has been set to 'neutron_l3'14:00
*** lajoskatona has joined #openstack-meeting14:00
lajoskatonao/14:00
liuyulonghi14:01
liuyulongOK, let's start14:01
liuyulong#topic Bugs14:01
*** openstack changes topic to "Bugs (Meeting topic: neutron_l3)"14:01
liuyulong#link http://lists.openstack.org/pipermail/openstack-discuss/2021-January/020008.html14:02
ralonsohhi14:02
liuyulong#link http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019878.html14:02
haleybhi14:02
liuyulong#link http://lists.openstack.org/pipermail/openstack-discuss/2021-January/019781.html14:02
liuyulongWe have these bug lists.14:02
liuyulongralonsoh, haleyb, Hi14:02
liuyulongFirst one14:02
liuyulong#link https://bugs.launchpad.net/neutron/+bug/191259614:02
openstackLaunchpad bug 1912596 in neutron "neutron-server report 500 error when update floating ip port forwarding" [Medium,In progress]14:02
liuyulongThe error is clear.14:03
liuyulong#link https://review.opendev.org/c/openstack/neutron/+/77177614:03
*** spotz has joined #openstack-meeting14:03
liuyulongBut the fix may have large a range of impacts.14:04
liuyulong#link https://review.opendev.org/c/openstack/neutron/+/771776/1/neutron/objects/base.py14:04
ralonsohexactly14:04
ralonsohthis is not a trivial change14:04
liuyulong#link https://review.opendev.org/c/openstack/neutron/+/771776/1/neutron/objects/base.py#85314:05
liuyulongIMO, the code is trying to copy these lines.14:05
liuyulongBut, maybe it my oversight some callers that have already catched the inner exception "DBDuplicateEntry".14:06
*** liuyulong has quit IRC14:07
ralonsohthere are, as commented in the patch14:07
*** liuyulong has joined #openstack-meeting14:08
*** liuyulong has quit IRC14:09
ralonsohI think out chairman is having some issues with the connection14:09
ralonsohour*14:09
*** liuyulong92 has joined #openstack-meeting14:10
liuyulong92sorry14:10
liuyulong92I lost the connection...14:10
*** liuyulong92 is now known as liuyulong14:11
haleybralonsoh: i would agree with your assessment on the patch, "small but remarkable"14:11
*** yasufum has joined #openstack-meeting14:11
liuyulong#chair haleyb14:11
openstackCurrent chairs: haleyb liuyulong14:11
liuyulongJust in case I lost connection again.14:12
liuyulong"""But, maybe it my oversight some callers that have already catched the inner exception "DBDuplicateEntry".""""14:12
liuyulongContinue the comments..14:12
liuyulong"DBDuplicateEntry" is replaced by NeutronDbObjectDuplicateEntry, so some callers may get a new 500, IMO.14:13
ralonsohyes, as commented in the patch, there are update methods expecting the other exception14:14
ralonsohI found one14:14
ralonsoh(it's in the patch)14:14
liuyulongOK, so let's move these comments to the gerrit.14:15
liuyulongNext one14:16
liuyulong#link https://bugs.launchpad.net/neutron/+bug/191112614:16
openstackLaunchpad bug 1911126 in neutron "[RFE][L3] add ability to control router SNAT more granularly" [Wishlist,New] - Assigned to LIU Yulong (dragon889)14:16
liuyulongThis was approved during last drivers meeting.14:16
liuyulongBecause there was one drivers meeting skipped in 16-01-2021. So I missed last meeting.14:18
liuyulongSo here, I will answer some question.14:18
liuyulongralonsoh it is not a TC or QoS mechanism.14:18
liuyulongIt is a 'datapath', or we can call it a "SNAT data path" for the VMs, IPs, subnets under a router.14:19
ralonsohI know14:19
ralonsohwhat I was proposing was to mark those packets14:19
ralonsohfor example, those ones sent from 10.0.10.0/2414:20
ralonsohand use this conntrack mark in tc-tc14:20
liuyulong#link https://review.opendev.org/c/openstack/neutron-specs/+/770540/5/specs/wallaby/elastic_snat.rst@21714:20
ralonsohtc is what you use to shape traffic in FIP14:20
liuyulongHere I pasted some iptables rules for python L3 agent.14:20
ralonsohI know and I understand your architecture14:21
ralonsohyou are doing a kind of FIP nating14:21
ralonsohsame as in FIP namespace14:21
liuyulongWe mark nothing for the snat rules, it is just a SNAT rule in router namespace.14:21
ralonsohI know you don't mark them14:22
ralonsohthat was my proposal14:22
ralonsohanyway, I'll comment in the spec14:22
liuyulongralonsoh, we can consider it as a opposite rules from "floating IP port forwarding".14:22
liuyulongfloating IP port forwarding is DNAT, while this is SNAT.14:23
*** jamesmcarthur has joined #openstack-meeting14:24
liuyulongsaying the TC, linux TC, the "floating SNAT IPs" will have QoS limitation like other floating IP do.14:24
liuyulongBecause the "floating SNAT IPs" will also be set on the related router external gateway.14:25
ralonsohI'll comment in the spec14:25
liuyulongSo for the implamentation about this, I may say, we did it in queens branch.14:27
liuyulongSo I need some time to refactor the code to master branch.14:27
liuyulongAnd time for testing on master branch.14:27
liuyulongSo IMO, we have lots of times to review the spec at first.14:28
liuyulongOK, no more bugs from me now.14:28
liuyulongAny updates?14:28
liuyulongOK, let's move on.14:29
liuyulong#topic On demand agenda14:29
*** openstack changes topic to "On demand agenda (Meeting topic: neutron_l3)"14:29
*** jamesmcarthur has quit IRC14:29
lajoskatonaIf you have some time please check the bfd spec: https://review.opendev.org/c/openstack/neutron-specs/+/767337 , thanks in advance14:29
liuyulong#link https://bugs.launchpad.net/neutron/+bug/190093414:29
openstackLaunchpad bug 1900934 in neutron "[RFE][DHCP][OVS] flow based DHCP" [Wishlist,New] - Assigned to LIU Yulong (dragon889)14:30
liuyulonglajoskatona, sure, I will do it.14:30
lajoskatonaI removed the bgp / dynamic-routing related parts14:30
ralonsohsure14:30
lajoskatonaliuyulong, ralonsoh: thanks14:30
*** liuyulong62 has joined #openstack-meeting14:33
liuyulong62sorry... I lost the connection.14:33
*** liuyulong has quit IRC14:34
*** liuyulong62 is now known as liuyulong14:34
liuyulongOK, I'm back really.14:34
*** tosky has quit IRC14:35
liuyulonglajoskatona, in your uploaded spec,  we will make L3 agent has ability to control the ovs-db (ovs bfd)?14:35
*** tosky has joined #openstack-meeting14:36
lajoskatonayes, actually I checked os-ken and with that it can be done transparently (I hope)14:36
lajoskatonaI am working on a PoC, but progressing slowly14:36
liuyulongOK, cool, you can link the PoC page to the spec's references.14:39
liuyulong#link https://bugs.launchpad.net/neutron/+bug/190093414:40
openstackLaunchpad bug 1900934 in neutron "[RFE][DHCP][OVS] flow based DHCP" [Wishlist,New] - Assigned to LIU Yulong (dragon889)14:40
lajoskatonaI will do14:40
liuyulongFor this DHCP agent extension for ml2/ovs.14:40
liuyulongI have some updates.14:40
liuyulong#link https://review.opendev.org/c/openstack/neutron-specs/+/768588/8/specs/wallaby/distributed_dhcp.rst14:40
liuyulongThe spec is basically completed.14:41
liuyulongOleg Bondarev14:42
liuyulong and I had some disscussion on that.14:42
liuyulongThe main point about Oleg's comments is "how to do upgrade" and "how to assemble the DHCP reponse".14:43
liuyulongWe can find the answer at:14:44
liuyulonghttps://review.opendev.org/c/openstack/neutron-specs/+/768588/8/specs/wallaby/distributed_dhcp.rst#28214:44
liuyulong#link https://review.opendev.org/c/openstack/neutron-specs/+/768588/8/specs/wallaby/distributed_dhcp.rst#18114:44
liuyulongAnother thing is I've uploaded a patch for this RFE.14:44
liuyulong#link https://review.opendev.org/c/openstack/neutron/+/77225514:45
liuyulongOK, and and, I created a blueprint for it.14:45
liuyulong#link https://blueprints.launchpad.net/neutron/+spec/distributed-dhcp-for-ml2-ovs14:45
*** jamesmcarthur has joined #openstack-meeting14:45
liuyulongThe neutron patch 772255 is for the new config option "disable_traditional_dhcp".14:46
liuyulongLet me quote some lines from the patch:14:46
liuyulongIf this is set to True, '14:46
liuyulong                       'neutron server will disable: '14:46
liuyulong                       '1. DHCP provisioning block '14:46
liuyulong                       '2. DHCP API extension and scheduler '14:46
liuyulong                       '3. Network scheduling mechanism '14:46
liuyulong                       '4. DHCP RPC/notification'14:46
liuyulongAnd Oleg also left me some kind comments about this patch.14:47
liuyulongHis main point about this is "how to use this new option".14:48
liuyulongAn here is my answer: "This can be not only for "distributed DHCP", but has another use case is for some deployment the operators want to offline all DHCP agents when they change the VM IP/metadata provisioning  action from DHCP to config drive."14:48
liuyulongSo, if you guys have time, please take a look at the spec. Comments are always welcomed. : )14:49
liuyulongAnd the patch. : )14:49
liuyulongOK, OK, wordy me.14:49
liuyulongNo more updates from me now.14:50
*** tosky has quit IRC14:50
liuyulongI will leave some mins for the meeting.14:50
liuyulongIf you have something to discuess, go ahead.14:50
*** tosky has joined #openstack-meeting14:50
liuyulongOK, let's end the meeting here.14:54
liuyulongThank you guys for attending.14:54
liuyulongBye14:54
ralonsohbye14:54
lajoskatonaBye14:54
liuyulong#endmeeting14:54
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"14:54
openstackMeeting ended Wed Jan 27 14:54:37 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)14:54
openstackMinutes:        http://eavesdrop.openstack.org/meetings/neutron_l3/2021/neutron_l3.2021-01-27-14.00.html14:54
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/neutron_l3/2021/neutron_l3.2021-01-27-14.00.txt14:54
openstackLog:            http://eavesdrop.openstack.org/meetings/neutron_l3/2021/neutron_l3.2021-01-27-14.00.log.html14:54
*** lajoskatona has left #openstack-meeting14:54
*** liuyulong has quit IRC14:55
*** irclogbot_3 has quit IRC14:55
*** belmoreira has joined #openstack-meeting14:57
*** irclogbot_3 has joined #openstack-meeting14:59
*** ociuhandu has joined #openstack-meeting15:11
*** oneswig has quit IRC15:17
*** ociuhandu has quit IRC15:23
*** ociuhandu has joined #openstack-meeting15:24
*** yamamoto has joined #openstack-meeting15:26
*** mlavalle has joined #openstack-meeting15:26
*** ociuhandu has quit IRC15:29
*** yamamoto has quit IRC15:33
*** ociuhandu has joined #openstack-meeting15:34
*** armax has joined #openstack-meeting15:35
*** rcernin has joined #openstack-meeting15:36
*** rcernin has quit IRC15:41
*** jamesmcarthur has quit IRC15:49
*** lpetrut has quit IRC16:00
*** jamesmcarthur has joined #openstack-meeting16:08
*** macz_ has joined #openstack-meeting16:13
*** dklyle has joined #openstack-meeting16:14
*** rpittau is now known as rpittau|afk16:24
*** e0ne has quit IRC16:27
*** yamamoto has joined #openstack-meeting16:32
*** PrinzElvis has quit IRC16:37
*** clayg has quit IRC16:37
*** erbarr has quit IRC16:37
*** patrickeast has quit IRC16:37
*** gagehugo has quit IRC16:37
*** vkmc has quit IRC16:37
*** zeestrat has quit IRC16:37
*** mnaser has quit IRC16:37
*** hillpd has quit IRC16:37
*** jamespage has quit IRC16:37
*** vishalmanchanda has quit IRC16:37
*** phrobb has quit IRC16:38
*** johnsom has quit IRC16:38
*** walshh_ has quit IRC16:38
*** gmann has quit IRC16:38
*** arne_wiebalck has quit IRC16:38
*** jungleboyj has quit IRC16:38
*** dougwig has quit IRC16:38
*** aprice has quit IRC16:38
*** csatari has quit IRC16:38
*** knikolla has quit IRC16:38
*** rajinir has quit IRC16:38
*** rpittau|afk has quit IRC16:38
*** manpreet has quit IRC16:38
*** nicolasbock has quit IRC16:38
*** sri_ has quit IRC16:38
*** seongsoocho has quit IRC16:38
*** zeestrat has joined #openstack-meeting16:39
*** vkmc has joined #openstack-meeting16:39
*** psahoo has quit IRC16:39
*** jamespage has joined #openstack-meeting16:39
*** johnsom has joined #openstack-meeting16:39
*** clayg has joined #openstack-meeting16:40
*** gouthamr has quit IRC16:40
*** erbarr has joined #openstack-meeting16:40
*** hillpd has joined #openstack-meeting16:40
*** phrobb has joined #openstack-meeting16:40
*** vishalmanchanda has joined #openstack-meeting16:40
*** dougwig has joined #openstack-meeting16:40
*** knikolla has joined #openstack-meeting16:40
*** manpreet has joined #openstack-meeting16:40
*** rajinir has joined #openstack-meeting16:40
*** rpittau|afk has joined #openstack-meeting16:41
*** yamamoto has quit IRC16:42
*** walshh_ has joined #openstack-meeting16:42
*** jungleboyj has joined #openstack-meeting16:42
*** arne_wiebalck has joined #openstack-meeting16:42
*** csatari has joined #openstack-meeting16:42
*** patrickeast has joined #openstack-meeting16:43
*** gagehugo has joined #openstack-meeting16:43
*** aprice has joined #openstack-meeting16:45
*** sri_ has joined #openstack-meeting16:45
*** gmann has joined #openstack-meeting16:45
*** mnaser has joined #openstack-meeting16:46
*** nicolasbock has joined #openstack-meeting16:50
*** nicolasbock has quit IRC16:55
*** seongsoocho has joined #openstack-meeting16:56
*** PrinzElvis has joined #openstack-meeting16:56
*** viks____ has joined #openstack-meeting16:57
*** nicolasbock has joined #openstack-meeting16:57
*** johanssone has quit IRC16:58
*** johanssone has joined #openstack-meeting17:01
*** psachin has quit IRC17:14
*** nicolasbock has quit IRC17:18
*** viks____ has quit IRC17:18
*** nicolasbock has joined #openstack-meeting17:19
*** stand has joined #openstack-meeting17:20
*** zeestrat has quit IRC17:21
*** arne_wiebalck has quit IRC17:21
*** jmasud has joined #openstack-meeting17:24
*** arne_wiebalck has joined #openstack-meeting17:28
*** zeestrat has joined #openstack-meeting17:28
*** ociuhandu_ has joined #openstack-meeting17:30
*** ociuhandu has quit IRC17:34
*** ociuhandu_ has quit IRC17:35
*** yasufum has quit IRC17:43
*** e0ne has joined #openstack-meeting17:46
*** timburke__ has quit IRC17:47
*** jamesmcarthur has quit IRC17:50
*** jamesmcarthur has joined #openstack-meeting17:51
*** e0ne_ has joined #openstack-meeting17:57
*** jamesmcarthur_ has joined #openstack-meeting17:57
*** e0ne has quit IRC17:57
*** tosky has quit IRC17:58
*** jamesmcarthur_ has quit IRC17:58
*** tosky has joined #openstack-meeting17:58
*** jamesmcarthur has quit IRC17:59
*** gagehugo has quit IRC18:00
*** seongsoocho has quit IRC18:00
*** vishalmanchanda has quit IRC18:00
*** gagehugo has joined #openstack-meeting18:00
*** manpreet has quit IRC18:00
*** manpreet has joined #openstack-meeting18:00
*** seongsoocho has joined #openstack-meeting18:01
*** vishalmanchanda has joined #openstack-meeting18:01
*** jamesmcarthur has joined #openstack-meeting18:02
*** patrickeast has quit IRC18:04
*** patrickeast has joined #openstack-meeting18:04
*** vishalmanchanda has quit IRC18:08
*** ralonsoh has quit IRC18:13
*** e0ne_ has quit IRC18:13
*** e0ne has joined #openstack-meeting18:13
*** manpreet has quit IRC18:14
*** seongsoocho has quit IRC18:14
*** patrickeast has quit IRC18:14
*** gagehugo has quit IRC18:14
*** timburke has joined #openstack-meeting18:14
*** ricolin has quit IRC18:16
*** clayg has quit IRC18:17
*** erbarr has quit IRC18:17
*** dougwig has quit IRC18:17
*** nicolasbock has quit IRC18:17
*** mnaser has quit IRC18:17
*** patrickeast has joined #openstack-meeting18:18
*** gagehugo has joined #openstack-meeting18:18
*** clayg has joined #openstack-meeting18:18
*** manpreet has joined #openstack-meeting18:18
*** seongsoocho has joined #openstack-meeting18:18
*** nicolasbock has joined #openstack-meeting18:18
*** mnaser has joined #openstack-meeting18:18
*** erbarr has joined #openstack-meeting18:19
*** dougwig has joined #openstack-meeting18:19
*** gouthamr has joined #openstack-meeting18:20
*** gyee has joined #openstack-meeting18:23
*** dougwig has quit IRC18:25
*** clayg has quit IRC18:25
*** seongsoocho has quit IRC18:25
*** manpreet has quit IRC18:25
*** patrickeast has quit IRC18:25
*** gagehugo has quit IRC18:25
*** patrickeast has joined #openstack-meeting18:27
*** mnaser has quit IRC18:27
*** nicolasbock has quit IRC18:27
*** jamesmcarthur has quit IRC18:27
*** seongsoocho has joined #openstack-meeting18:28
*** gagehugo has joined #openstack-meeting18:31
*** clayg has joined #openstack-meeting18:31
*** mnaser has joined #openstack-meeting18:32
*** manpreet has joined #openstack-meeting18:33
*** dougwig has joined #openstack-meeting18:33
*** nicolasbock has joined #openstack-meeting18:33
*** zbr0 has joined #openstack-meeting18:35
*** zbr has quit IRC18:37
*** zbr0 is now known as zbr18:37
*** jmasud has quit IRC18:42
*** yamamoto has joined #openstack-meeting18:42
*** yamamoto has quit IRC18:53
*** gagehugo has quit IRC18:54
*** manpreet has quit IRC18:54
*** clayg has quit IRC18:54
*** nicolasbock has quit IRC18:55
*** dougwig has quit IRC18:55
*** mnaser has quit IRC18:55
*** manpreet has joined #openstack-meeting18:55
*** markmcclain has quit IRC18:55
*** eharney has quit IRC18:56
*** markmcclain has joined #openstack-meeting18:57
*** e0ne has quit IRC19:01
*** e0ne has joined #openstack-meeting19:06
*** e0ne has quit IRC19:06
*** eharney has joined #openstack-meeting19:09
*** gagehugo has joined #openstack-meeting19:14
*** gouthamr_ has joined #openstack-meeting19:18
*** mnaser has joined #openstack-meeting19:19
*** dougwig has joined #openstack-meeting19:20
*** nicolasbock has joined #openstack-meeting19:20
*** clayg has joined #openstack-meeting19:21
*** jamesmcarthur has joined #openstack-meeting19:24
*** e0ne has joined #openstack-meeting19:35
*** e0ne has quit IRC19:35
*** clayg has quit IRC19:40
*** dougwig has quit IRC19:40
*** gouthamr_ has quit IRC19:40
*** mnaser has quit IRC19:40
*** nicolasbock has quit IRC19:40
*** clayg has joined #openstack-meeting19:41
*** dougwig has joined #openstack-meeting19:41
*** mnaser has joined #openstack-meeting19:42
*** nicolasbock has joined #openstack-meeting19:45
*** rcernin has joined #openstack-meeting20:01
*** rcernin has quit IRC20:41
*** stand has quit IRC20:42
*** e0ne has joined #openstack-meeting20:43
*** stand has joined #openstack-meeting20:46
*** e0ne has quit IRC20:47
*** e0ne has joined #openstack-meeting20:47
*** e0ne has quit IRC20:48
*** yamamoto has joined #openstack-meeting20:52
*** acoles has joined #openstack-meeting20:59
timburke#startmeeting swift21:00
openstackMeeting started Wed Jan 27 21:00:48 2021 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.21:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:00
*** openstack changes topic to " (Meeting topic: swift)"21:00
openstackThe meeting name has been set to 'swift'21:00
timburkewho's here for the swift meeting?21:00
mattoliverauo/21:00
seongsoochoo/21:01
kota_o/21:01
rledisezo/21:01
acoleso/21:01
timburkeas usual, the agenda's at https://wiki.openstack.org/wiki/Meetings/Swift21:02
timburkefirst up21:02
timburke#topic ssync and non-durables21:02
*** openstack changes topic to "ssync and non-durables (Meeting topic: swift)"21:02
claygo/21:03
timburkeacoles, clayg how's it looking?21:03
*** zaitcev has joined #openstack-meeting21:03
acoleswe deployed the fix to ssync this week and it has cleaned up some handoff partitions that were stuck with only non-durable frags :)21:04
acolesthat's https://review.opendev.org/c/openstack/swift/+/77004721:04
claygaka TOTAL SUCCESS!!! :shipit:21:04
acoleszaitcev: thanks for review21:04
timburkei see it's already got one +2; how important do you think it is to get more eyes on it before merging?21:05
claygtimburke: i clicked +A21:05
timburke🎉21:05
acoles*really important* :D21:05
timburkethat was easy :-)21:05
acolesJK21:05
mattoliveraulol21:05
timburke#topic orphaned shard dbs21:06
*** openstack changes topic to "orphaned shard dbs (Meeting topic: swift)"21:06
zaitcevBTW, does it work? Do you have handsoff partitions getting drained?21:06
acoleszaitcev: yes21:06
claygzaitcev: YES!  it's amazing!  you do NOT want to be running swift w/o this patch 💪21:06
acoleszaitcev: we had at least one non-durable frag that we monitored and it was cleaned up21:06
timburkei think i lost track of who's leading the charge on orphaned shards -- mattoliverau, clayg, acoles?21:07
mattoliverauThings are progressing here. We've been working on the shrinking side of things.21:07
mattoliverauthe only reclaim when root has no shards patch has landed21:07
mattoliverauwhich is a stop gap, and stops orphaned shards21:08
mattoliverauacoles has https://review.opendev.org/c/openstack/swift/+/77188521:08
mattoliverauwhich allows us to fully collapse back up with shrinking all driven by root (which is amazing work BTW).21:09
acolesthe root reclaim patch is struggling to get through gate but has +A21:09
mattoliveraunow just filling in the missing middle pieces21:09
mattoliveraunamely more shrinking. acoles compact patch is where things are going21:09
acolesyep, so I'm hoping we can make progress on chain of patches starting with https://review.opendev.org/c/openstack/swift/+/77188521:10
mattoliverauthat will not only make manage-shard-ranges work better, but it's code that lives in the sharder that will improve autosharding21:10
claygthat's where i started!21:10
acoleswhich would give us cli tool to shrink away less populated shards21:10
mattoliverauSo we're getting closer21:11
timburkesounds great. anything about the chain that we should discuss here?21:11
mattoliverauI've been playing with dumping shrinking candidates to recon dump: https://review.opendev.org/c/openstack/swift/+/77262421:11
*** jamesmcarthur has quit IRC21:12
mattoliverauWhich is just a mirror of the top containers to shard. But shrinking works differently and so am thinking of changing the metrics we send to recon for shrinking21:12
mattoliverauanyway I have comments in the commit message and having a play with it.21:12
acolestimburke: nothing particular to flag up on that patch chain, but reviews welcome - there's quite a lot going on what with shrinking and also some overlap repair features21:12
mattoliverau+121:13
mattoliveraureviews welcome :)21:13
timburkei definitely like the direction you're headed in the last paragraph of that commit message, mattoliverau21:13
mattoliverauyeah, wanted to see how ops used the data before I went too cowboy :)21:13
mattoliverau*would use*21:13
acoles+1 left you some comments today mattoliverau21:14
mattoliverauoh cool, thanks acoles21:14
acolesseems we have consensus on the way ahead21:14
acolesi.e. report number of shrinkable ranges not object counts21:14
mattoliverauCool, yeah makes much more sense in shrinking speak21:15
timburkemaybe even "count of all objects in shrinkable shards"?21:15
mattoliverauI'll have a play with the code :)21:16
timburkeall right. as long as we're already talking about sharding/shrinking...21:16
timburke#topic sharding in train21:16
*** openstack changes topic to "sharding in train (Meeting topic: swift)"21:16
timburkezaitcev, am i right to think this was your topic?21:16
*** yamamoto has quit IRC21:16
zaitcevtimburke: yes21:16
zaitcevAt this moment I identified 20 patches and I'm cherry-picking all of them. Not many obvious conflicts. Some are purely stull like "add some probe tests". So let's say 15 patches in a go. Does it sounds reasonable to just file for review and have a shocking flood from bot on IRC?21:17
zaitcevBTW, the total amount of patches in stable/train after the fork point is 54.21:18
zaitcevMaster is 579 ahead. Just to amuse you with the meaningless stats. But gosh, we're committing a ton of patches, guys.21:19
timburkefrom my perspective, it doesn't matter too much -- most all the sharding patches i can think of fit squarely in the "worth backporting" category21:19
*** jamesmcarthur has joined #openstack-meeting21:19
zaitcevOkay21:20
zaitcevI'll run through tests on my cluster and pile it upon you then.21:20
timburkeeven looks like the stable gate is in half-way reasonable shape21:21
timburkeis there any time table on when you'd want a stable tag that i should keep in mind?21:21
timburkei won't worry about it too much ;-)21:23
timburke#topic relinker and part power increase21:24
*** openstack changes topic to "relinker and part power increase (Meeting topic: swift)"21:24
*** jamesmcarthur has quit IRC21:24
claygtimburke: i saw you fighting with probe tests21:24
timburkerledisez, thanks for approving the state-file patch, and talking through suffix hashing with me!21:24
timburkeclayg, yeah, i kicked up https://review.opendev.org/c/openstack/swift/+/772772 -- i'm less worried now, but for a stretch i was seeing occasional failures21:26
timburkei think it was because (1) i added a new file to the probe test, one that is created before relinking then not written to again and (2) we were a little sloppy about what we put in org_locations21:27
timburkeon the disk-unmounted patch, does anyone have thoughts on https://review.opendev.org/c/openstack/swift/+/769632/2/swift/common/utils.py#3262 ? should i back out that part so we raise EPERM?21:28
timburkei think the other option for that patch would be to wrap the logger we pass down to the audit location generator so we can count warning/error calls and use that to inform our exit code...21:30
zaitcevlessee21:30
rledisezraising an error might be the best move. i'm assuming an operator at some scale is not watching the output of every command ran on its clusters…21:31
rlediseza timeserie information? logger.increment('relinker.skipped-device')?21:32
rledisezeh, it's not that different of a log. except I think people use more grafana than elasticsearch (especially these days :D)21:33
timburkerledisez, yeah, that was my concern -- i know our sres will be running this across the fleet with ansible, and i want to make it really easy for them to know when it's safe to continue ;-)21:34
seongsoochoDoes logger.increment send metrics to statsd?21:34
claygseongsoocho: yes21:34
mattoliverauor at least more people might be watching grafana live to see results (or at least closer to live) rather then looking at historic data and might say ops, missed that :P21:35
seongsoochookay. then That will be more helpful and also logging in log file21:35
rledisezFWIW, I know we ran clush on the whole cluster with a specially crafted command that was reading the JSON file to get the status of all partitions before moving on to the next step. if a file was missing we were getting an error21:36
rledisezso my vote would be either raise or log/metrics+a relinker status command21:37
timburkeall right, i might go ahead and back out the utils.py change for now (so the error pops and we get a non-zero exit), and look at counting logger.warning/error/exception calls as a follow-up21:38
claygnoice21:38
timburkethere was one more patch i realized i'd need to have a smooth part-power-increase: https://review.opendev.org/c/openstack/swift/+/77241921:39
mattoliveraurelinker could dump failed/unmounted devices to recon so we have a list for the recon to try again later. But I guess we have the stat files that do pretty much the same.21:39
mattoliverauoh nice, drop priv cool.21:40
timburkeso the relinker needs to read the hash prefix/suffix from /etc/swift/swift.conf -- but we deploy with that solely readable by root (and rely on the swift_user config in daemons to appropriately drop privs)21:41
timburkeso on master, i couldn't run the relinker as swift (EPERM), and running as root ended poorly (wrong owner for all the new partitions)21:41
claygtimburke: it's not super obvious to me why swift:swift 600 isn't good enough, but yeah... we do want a sudo drop priv21:42
claygI feel like it would all look a lot more like everything else if we just had an /etc/swift/relinker.conf21:42
timburkeyeah, i was about to bring that up... 'cause there are some other things i'm thinking about doing, like ratelimiting on the location generator or adding ionice tunings like we've got for other daemons, where it feels a little weird to continue growing options21:44
timburkedo we have much precedent for conf files for kinda one-shot tools?21:45
timburkei guess there's /etc/swift/bench.conf ...21:45
timburkewell, i'll take a look at it at some point21:47
timburkethat's all i had21:47
timburke#topic open discussion21:47
*** openstack changes topic to "open discussion (Meeting topic: swift)"21:47
timburkewhat else should we discuss this week?21:47
zaitcevclayg: it's in case something in swift is broken and wipes swift.conf21:47
claygzaitcev: maybe so!21:47
seongsoochotimburke:  I saw the p769855 merged.  (Thanks for the patch) Is it safe to only relinker use recent code? In my case, object server is running in train and relinker is master21:47
timburkezaitcev, good thought -- i'd been assuming that it was because you might consider the prefix/suffix "secrets"21:48
*** ircuser-1 has joined #openstack-meeting21:49
claygbut sekret from *swift* 🤯21:49
timburkeseongsoocho, yes, i'd expect so. the imports in cli/relinker.py are pretty minimal and unlikely to change. of course, it's always worth testing in a dev/staging environment first ;-)21:50
*** cmart has quit IRC21:50
*** timburke has quit IRC21:51
*** raildo has quit IRC21:51
*** raildo has joined #openstack-meeting21:52
seongsoochotimburke:  okay. thanks!21:52
*** timburke has joined #openstack-meeting21:52
mattoliverauhe's back :)21:52
zaitcevwoops21:52
*** rcernin has joined #openstack-meeting21:52
timburkeidk what that was about *shrug*21:52
timburkeit would probably be sufficient to pull up a saio on train, copy in cli/relinker.py from master, then run probe tests21:53
timburke(and, if you want the drop-privs patch, be sure to grab bin/swift-object-relinker from master, too)21:54
timburkeall right, seems like we're winding down21:55
timburkethank you all for coming, and thank you for working on swift!21:55
*** rcernin has quit IRC21:55
timburke#endmeeting21:55
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"21:55
openstackMeeting ended Wed Jan 27 21:55:47 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)21:55
openstackMinutes:        http://eavesdrop.openstack.org/meetings/swift/2021/swift.2021-01-27-21.00.html21:55
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/swift/2021/swift.2021-01-27-21.00.txt21:55
openstackLog:            http://eavesdrop.openstack.org/meetings/swift/2021/swift.2021-01-27-21.00.log.html21:55
*** acoles has left #openstack-meeting21:55
*** rcernin has joined #openstack-meeting21:56
*** zaitcev has left #openstack-meeting21:56
*** macz_ has quit IRC22:02
*** timburke has quit IRC22:04
*** rcernin has quit IRC22:05
*** rcernin has joined #openstack-meeting22:05
*** yamamoto has joined #openstack-meeting22:20
*** jamesmcarthur has joined #openstack-meeting22:29
*** zigo has quit IRC22:29
*** zigo has joined #openstack-meeting22:30
*** ircuser-1 has quit IRC22:31
*** jamesmcarthur has quit IRC22:34
*** macz_ has joined #openstack-meeting22:45
*** slaweq has quit IRC22:51
*** jmasud has joined #openstack-meeting23:02
*** jmasud has quit IRC23:04
*** jmasud has joined #openstack-meeting23:06
*** jmasud has quit IRC23:12
*** cmart has joined #openstack-meeting23:32
*** jamesmcarthur has joined #openstack-meeting23:38
*** jamesmcarthur has quit IRC23:45
*** yamamoto has quit IRC23:51
*** yamamoto has joined #openstack-meeting23:58
*** jmasud has joined #openstack-meeting23:58

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!