Wednesday, 2020-10-14

*** jmasud has joined #openstack-meeting00:04
*** jamesmcarthur has quit IRC00:09
*** armax has quit IRC00:13
*** yasufum has joined #openstack-meeting00:22
*** yasufum_ has joined #openstack-meeting00:25
*** yasufum has quit IRC00:27
*** yasufum_ is now known as yasufum00:27
*** jamesmcarthur has joined #openstack-meeting00:40
*** gyee has quit IRC00:43
*** jamesmcarthur has quit IRC00:48
*** jamesmcarthur has joined #openstack-meeting00:56
*** rfolco has quit IRC01:01
*** jmasud has quit IRC01:12
*** dklyle has quit IRC01:17
*** david-lyle has joined #openstack-meeting01:17
*** jamesmcarthur has quit IRC01:23
*** jamesmcarthur has joined #openstack-meeting01:24
*** david-lyle has quit IRC01:24
*** Liang__ has joined #openstack-meeting01:31
*** jheyd has quit IRC01:34
*** manpreet has quit IRC01:46
*** manpreet has joined #openstack-meeting02:44
*** rcernin has quit IRC02:48
*** rcernin has joined #openstack-meeting03:05
*** rcernin has quit IRC03:06
*** rcernin has joined #openstack-meeting03:06
*** psachin has joined #openstack-meeting03:36
*** jamesmcarthur has quit IRC03:41
*** jamesmcarthur has joined #openstack-meeting03:48
*** jamesmcarthur has quit IRC03:53
*** jamesmcarthur has joined #openstack-meeting03:58
*** yasufum_ has joined #openstack-meeting04:01
*** yasufum has quit IRC04:02
*** yasufum_ is now known as yasufum04:02
*** dsariel has left #openstack-meeting04:13
*** jmasud has joined #openstack-meeting04:21
*** evrardjp has quit IRC04:33
*** evrardjp has joined #openstack-meeting04:33
*** yasufum_ has joined #openstack-meeting04:34
*** yasufum has quit IRC04:35
*** yasufum_ is now known as yasufum04:35
*** jamesmcarthur has quit IRC04:40
*** jamesmcarthur has joined #openstack-meeting04:41
*** jamesmcarthur has quit IRC04:46
*** psachin has quit IRC04:48
*** dmacpher has quit IRC04:55
*** hongbin has quit IRC04:59
*** hongbin has joined #openstack-meeting05:09
*** martial has quit IRC05:15
*** jamesmcarthur has joined #openstack-meeting05:17
*** hongbin has quit IRC05:19
*** b1airo has quit IRC05:25
*** ralonsoh has joined #openstack-meeting05:33
*** jamesmcarthur has quit IRC05:40
*** ociuhandu has joined #openstack-meeting05:45
*** ociuhandu has quit IRC05:49
*** slaweq has joined #openstack-meeting06:08
*** vishalmanchanda has joined #openstack-meeting06:19
*** Liang has quit IRC06:54
*** Luzi has joined #openstack-meeting06:58
*** tosky has joined #openstack-meeting07:43
*** Liang__ has quit IRC07:47
*** ykatabam has quit IRC07:48
*** Liang__ has joined #openstack-meeting07:48
*** rcernin has quit IRC07:50
*** lpetrut has joined #openstack-meeting07:50
*** rpittau|afk is now known as rpittau07:52
*** e0ne has joined #openstack-meeting08:02
*** e0ne has quit IRC08:04
*** ociuhandu has joined #openstack-meeting08:11
*** jamesmcarthur has joined #openstack-meeting08:16
*** lbragstad_ has joined #openstack-meeting08:18
*** lbragstad has quit IRC08:20
*** jamesmcarthur has quit IRC08:21
*** ykatabam has joined #openstack-meeting08:25
*** rcernin has joined #openstack-meeting08:27
*** ykatabam has quit IRC08:48
*** rcernin has quit IRC08:59
*** ociuhandu_ has joined #openstack-meeting09:01
*** ociuhandu has quit IRC09:04
*** hyunsikyang has joined #openstack-meeting09:14
*** ociuhandu_ has quit IRC09:15
*** ociuhandu has joined #openstack-meeting09:16
*** hyunsikyang__ has quit IRC09:17
*** yasufum has quit IRC09:35
*** yasufum has joined #openstack-meeting09:36
*** armax has joined #openstack-meeting09:39
*** Liang__ has quit IRC09:43
*** Liang__ has joined #openstack-meeting09:44
*** ykatabam has joined #openstack-meeting10:00
*** ykatabam has joined #openstack-meeting10:00
*** yasufum has quit IRC10:05
*** zigo has joined #openstack-meeting10:12
*** Liang__ has quit IRC10:17
*** yasufum has joined #openstack-meeting10:22
*** rcernin has joined #openstack-meeting10:25
*** psachin has joined #openstack-meeting10:56
*** vishalmanchanda has quit IRC11:08
*** yasufum has quit IRC11:10
*** hyunsikyang__ has joined #openstack-meeting11:34
*** hyunsikyang has quit IRC11:38
*** hyunsikyang has joined #openstack-meeting11:38
*** priteau has joined #openstack-meeting11:39
*** hyunsikyang__ has quit IRC11:39
*** raildo has joined #openstack-meeting11:42
*** manuvakery has joined #openstack-meeting11:43
*** hyunsikyang__ has joined #openstack-meeting11:46
*** hyunsikyang has quit IRC11:50
*** hyunsikyang has joined #openstack-meeting11:50
*** hyunsikyang__ has quit IRC11:52
*** armstrong has joined #openstack-meeting11:54
*** yasufum has joined #openstack-meeting12:05
*** yasufum_ has joined #openstack-meeting12:16
*** yasufum has quit IRC12:18
*** yasufum_ is now known as yasufum12:18
*** jamesmcarthur has joined #openstack-meeting12:18
*** jamesmcarthur has quit IRC12:22
*** jamesmcarthur has joined #openstack-meeting12:26
*** rfolco has joined #openstack-meeting12:31
*** jamesmcarthur has quit IRC12:34
*** ykatabam has quit IRC12:34
*** jamesmcarthur has joined #openstack-meeting12:35
*** ykatabam has joined #openstack-meeting12:37
*** yasufum has quit IRC12:44
*** thgcorrea has joined #openstack-meeting12:55
*** lbragstad_ is now known as lbragstad13:02
*** jamesmcarthur has quit IRC13:05
*** jamesmcarthur has joined #openstack-meeting13:07
*** ykatabam has quit IRC13:08
*** ykatabam has joined #openstack-meeting13:09
*** ykatabam has quit IRC13:17
*** Luzi has quit IRC13:23
*** TrevorV has joined #openstack-meeting13:24
*** jamesmcarthur has quit IRC13:28
*** vishalmanchanda has joined #openstack-meeting13:43
*** jamesmcarthur has joined #openstack-meeting13:50
*** Liang__ has joined #openstack-meeting13:57
*** rcernin has quit IRC14:01
*** Liang__ is now known as LiangFang14:01
*** e0ne has joined #openstack-meeting14:04
*** priteau has quit IRC14:09
*** priteau has joined #openstack-meeting14:19
*** macz_ has joined #openstack-meeting14:23
*** macz_ has quit IRC14:28
*** ianychoi_ has quit IRC14:33
*** psachin has quit IRC14:33
*** lpetrut has quit IRC14:40
*** LiangFang has quit IRC14:47
*** ianychoi has joined #openstack-meeting14:51
*** dklyle has joined #openstack-meeting14:54
*** belmoreira has joined #openstack-meeting15:01
*** macz_ has joined #openstack-meeting15:02
*** alistarle has joined #openstack-meeting15:05
*** macz_ has quit IRC15:15
*** dmacpher has joined #openstack-meeting15:17
*** jmasud has quit IRC15:33
*** gyee has joined #openstack-meeting15:41
*** e0ne has quit IRC15:49
*** slaweq has quit IRC15:55
*** slaweq has joined #openstack-meeting15:58
*** rpittau is now known as rpittau|afk16:05
*** alistarle has quit IRC16:07
*** tosky has quit IRC16:21
*** ociuhandu_ has joined #openstack-meeting16:50
*** ociuhandu has quit IRC16:54
*** ociuhandu_ has quit IRC16:55
*** e0ne has joined #openstack-meeting17:00
*** jamesmcarthur has quit IRC17:02
*** Trevor_V has joined #openstack-meeting17:03
*** TrevorV has quit IRC17:03
*** jmasud has joined #openstack-meeting17:04
*** jmasud has quit IRC17:09
*** jamesmcarthur has joined #openstack-meeting17:14
*** jamesmcarthur has quit IRC17:15
*** jamesmcarthur has joined #openstack-meeting17:15
*** jamesmcarthur has quit IRC17:19
*** mlavalle has joined #openstack-meeting17:26
*** jmasud has joined #openstack-meeting17:28
*** ralonsoh has quit IRC17:42
*** e0ne has quit IRC17:50
*** e0ne has joined #openstack-meeting17:52
*** jamesmcarthur has joined #openstack-meeting17:59
*** smcginnis has quit IRC17:59
*** smcginnis has joined #openstack-meeting18:01
*** e0ne has quit IRC18:01
*** jamesmcarthur has quit IRC18:05
*** jamesmcarthur has joined #openstack-meeting18:05
*** jmasud has quit IRC18:09
*** rcernin has joined #openstack-meeting18:22
*** rcernin has quit IRC18:37
*** e0ne has joined #openstack-meeting18:39
*** belmoreira has quit IRC18:51
*** ociuhandu has joined #openstack-meeting19:00
*** belmoreira has joined #openstack-meeting19:03
*** ociuhandu has quit IRC19:05
*** e0ne has quit IRC19:09
*** priteau has quit IRC19:20
*** Trevor_V has quit IRC19:30
*** TrevorV has joined #openstack-meeting19:31
*** jmasud has joined #openstack-meeting19:52
*** tosky has joined #openstack-meeting19:53
*** jmasud has quit IRC20:00
*** vishalmanchanda has quit IRC20:08
*** manuvakery has quit IRC20:32
*** jamesmcarthur has quit IRC20:32
*** belmoreira has quit IRC20:36
*** patchbot has joined #openstack-meeting20:36
*** jamesmcarthur has joined #openstack-meeting20:40
timburke#startmeeting swift21:00
openstackMeeting started Wed Oct 14 21:00:24 2020 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.21:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:00
*** openstack changes topic to " (Meeting topic: swift)"21:00
openstackThe meeting name has been set to 'swift'21:00
*** zaitcev has joined #openstack-meeting21:00
timburkewho's here for the swift meeting?21:00
kota_hello21:00
seongsoochoo/21:00
rledisezhi o/21:01
zaitcev0721:01
timburkeas usual, the agenda's at https://wiki.openstack.org/wiki/Meetings/Swift21:02
timburkefirst up, though, something i forgot to put on there ;-)21:02
timburke#topic releases21:02
*** openstack changes topic to "releases (Meeting topic: swift)"21:02
timburkevictoria was released today!21:02
timburkethanks everyone for making this another great cycle21:03
kota_congrats!21:03
seongsoochoyay ~~~ congrats!!! 🎉21:03
timburkewe also have new stable releases for ussuri (2.25.1) and train (2.23.2)21:03
mattoliverauo/ (sorry I'm late)21:05
timburkethey include a lot of py3 bug fixes, including the recent crypto patch where you'll need to upgrade while still on py2 before transitioning to py321:05
timburke(as much as anything, just making sure people are aware of them)21:05
timburkeand i'll probably try to get a swiftclient release out in the near future, as i think it's blocking my current attempt at getting py3 probe tests in the gate21:06
timburke#topic PTG21:07
*** openstack changes topic to "PTG (Meeting topic: swift)"21:07
timburkejust a week and a half away!21:07
mattoliverau\o/21:08
timburkei added a rough design for ALOs to the etherpad21:08
timburke#link https://etherpad.opendev.org/p/swift-ptg-wallaby21:08
mattoliverauOh nice21:08
kota_actually, Summit will start on the next Monday.21:08
timburkealso true!21:08
timburkei should see what sessions look interesting...21:09
timburkeif anyone has recommendations, i'd love to hear them (now or in -swift)21:09
timburkethe ptg schedule is up at https://object-storage-ca-ymq-1.vexxhost.net/swift/v1/6e4619c416ff4bd19e1c087f27a43eea/www-assets-prod/Uploads/PTG2-Oct26-30-2020-Schedule-1.pdf and all the meeting times on the etherpad are accurate21:10
* kota_ did NOT catch up the session schedule yet :P21:10
timburkesummit schedule's also available, at https://www.openstack.org/summit/2020/summit-schedule/21:11
timburkei can't wait to see you all again and do some hacking! :-)21:13
timburkespeaking of... let's talk patches!21:13
timburke#topic audit watchers21:13
*** openstack changes topic to "audit watchers (Meeting topic: swift)"21:13
timburke#link https://review.opendev.org/#/c/706653/21:13
patchbotpatch 706653 - swift - Let developers/operators add watchers to object au... - 37 patch sets21:13
timburkezaitcev, i saw you added a +2 -- does that mean you and david are ready for the rest of us to do some reviews?21:14
zaitcevYes21:14
timburkecool21:14
timburke!21:15
zaitcevdsariel (not in the meeting) was already looking for another project, this may bet float up.21:15
timburkei'll be sure to make some time for a look, probably rebase https://review.opendev.org/#/c/744078/121:15
patchbotpatch 744078 - swift - watchers: Add EC stat gatherer - 1 patch set21:15
zaitcevOh, nice. I forgot you had that.21:16
timburkeand see what other fun ideas for watchers i can come up with :-)21:16
timburkeanything else you need there besides reviews? anything we should know digging into it?21:17
zaitcevBut I just want a claim post in the ground -- even if the EC plugin ends requiring changes to the API. I resigned myself to APIs that need to be changed. I think this is what Sam meant when he wrote that all arguments are passed by name. Note that it requires implementations to always add **kwargs, but beyond that a flexibility is built in.21:17
timburkeyeah, and we could surely have a few releases where we label the feature as being experimental and subject to change if we really need21:18
zaitcevI'm glad that I talked you all into the reduced isolation, because separate processes produced nasty problems with not keeping up and logging.21:18
zaitcevIf you can accept in-process model, I'm okay with anything. Even "except Exception" which I always -1 elsewhere.21:19
zaitcevThat is all.21:20
timburke#topic replication locking21:20
*** openstack changes topic to "replication locking (Meeting topic: swift)"21:20
zaitcevIf you look at it and like it, we don't really need to waste PTG time on it. Feel free to overstrike after you review.21:20
timburke#link https://review.opendev.org/#/c/754242/21:20
patchbotpatch 754242 - swift - Fix a race condition in case of cross-replication - 5 patch sets21:20
zaitcevTsk21:21
zaitcevI promised to review it but didn't.21:21
timburkerledisez, sorry, i still haven't gotten to rebuilding an env that would be amenable to trying to repro :-(21:21
timburkei don't want to drop it from the agenda, though, because i hope to shame myself into actually doing what i said i'd do ;-)21:22
rledisezSo patchset 5 passed the production test, I was unable to lose datafiles using this patch. I added a comment on timeout at the end. I'm in favor of hardcoding a small value, but you may have different opinions21:22
timburkethat seems perfectly reasonable to me. if we can't get the lock quickly, skip it and try again later; makes sense21:22
rledisezok. so I'll update the review with a hardcoded timeout of 0.2; It's working fine in our deployments21:23
timburkeanything else you need there? would it be useful for us to think some more about how to handle locking for rsync?21:24
rledisezI still have to work on the rsync fix, I have to take some time to think about it. I'm affraid it would incur a major change in SSYNC (splitting "negociation" and "data transfert). But it may be for the best actually21:25
rledisezI still think the best option is to wrap rsync with a ssync call21:25
rledisezthat's all for me on that topic21:27
zaitcevInteresting.21:27
timburke#topic async slo cleanup21:28
*** openstack changes topic to "async slo cleanup (Meeting topic: swift)"21:28
timburke#link https://review.opendev.org/#/c/733026/21:28
patchbotpatch 733026 - swift - Add a new URL parameter to allow for async cleanup... - 12 patch sets21:28
timburkeso i think clayg's getting skittish about turning this on by default21:29
timburkei might end up having it be opt-in, at least for an initial release. i know the main cluster i care about doesn't make heavy use of the expirer yet, and this would presumably change that21:30
timburkemeanwhile, i don't have a great handle on how best to monitor my expirers -- i know clayg has a little script that can check how much is in the queue and how much of it is ready for reaping, though21:31
mattoliverauI guess opt in initially is probably a safe way to go forward.21:32
timburkei'd also had an idea a while back (based on rledisez's https://review.opendev.org/#/c/715580/) to add a "lag" metric for the expirer21:32
patchbotpatch 715580 - swift - obj-updater: add metric on lag of containers listing - 1 patch set21:32
claygyeah SRE just deployed the expirer monitor on like one random node - so we have stats now (as long as that node is up 🙄 )21:32
mattoliverauShould we get expires to emit something, somehow? Have think about it.21:33
timburkewe've got some stats already, at least; iirc, mostly just success/failure counts21:34
zaitcevI looked at that patch and it seemed okay, but then Clay came in and he had some fundamental comments21:34
timburke(fwiw, the expirer patch is https://review.opendev.org/#/c/735271/)21:35
patchbotpatch 735271 - swift - metrics: Add lag metric to expirer - 1 patch set21:35
mattoliverauCan you pull general task queue account stats to get a rough idea on size? With and account head. I should play around with general task queue some more.21:35
mattoliverauOr did we shard the accounts too.21:36
* mattoliverau is just thinking out loud, and isn't at his computer 21:36
timburkeyeah, pretty sure that's the idea with https://gist.github.com/clayg/7f66eab2a61c77869e1e84ac4ed6f1df21:37
*** rfolco has quit IRC21:38
*** rf0lc0 has joined #openstack-meeting21:38
timburkeoh, but with https://review.opendev.org/#/c/517389/ that might get more complicated21:38
patchbotpatch 517389 - swift - Add object-expirer new mode to execute tasks from ... - 46 patch sets21:38
timburkeanyway, mainly just wanted to call attention to that default change -- i should have a fresh patchset up soon21:41
timburke#topic open discussion21:41
*** openstack changes topic to "open discussion (Meeting topic: swift)"21:41
timburkewhat else should we bring up today?21:41
zaitcevI recommended dsariel to look into sharding, help Matt along.21:42
zaitcevNot sure if it's going to work.21:42
*** thgcorrea has quit IRC21:42
mattoliverauThanks!21:43
timburkecool! speaking of, i should get my probe test at https://review.opendev.org/#/c/744256/ to a point that it passes consistently :-/21:45
patchbotpatch 744256 - swift - sharding: probe test to exercise manual shrinking - 3 patch sets21:45
mattoliverauI've been writing some tests for my poc rangescanner that will hopefully fsch root shard ranges21:46
mattoliverau*fsck/scan and attempt to fix21:47
timburkei'm really coming around on the idea that the way to "solve" autosharding is to get an automated recovery from overlapping shard ranges21:47
mattoliverauYeah, I want to do leader election properly, but in any case being able to fix is more important initially21:49
mattoliverauThe scanner currently deals with overlaps and can rebuild fragmented paths.21:49
mattoliverauIt's still a WIP but feel free to have a look. I also have a new doc/braindump21:50
mattoliverauWhich I hope to go over at the ptg.21:50
timburkeoh! there's a patch i keep meaning to bring up during meetings but never quite get around to: https://review.opendev.org/#/c/751966/21:51
patchbotpatch 751966 - swift - replace md5 with swift utils version - 11 patch sets21:51
zaitcevwait, what21:52
timburkesomeone's interested in running swift with FIPS mode enabled, which means we'd need to annotate all uses of md521:52
timburke...which understandably means that there'd be a decent number of conflicts if/when we merge it21:53
zaitcevRussian anecdote: One guys said "What do you know about security through obscurity? On my last job, I replaced MD5 with SHA256 only trimmed to fit."21:53
timburkeheh21:54
mattoliverauLol21:54
timburke(honestly, it makes me think a bit too much of https://tools.ietf.org/html/rfc3514, but w/e...)21:55
zaitcev1 April 2003 - nice tray21:56
*** rf0lc0 has quit IRC21:56
*** slaweq_ has joined #openstack-meeting21:57
timburkedoes anyone have an interest in trying to review this? i'll probably get to it eventually, but it's very much a "when i get around to it" sort of endeavor)21:57
zaitcevI could, I suppose. Just after Romain's race condition thing.21:58
timburkei like that prioritization :-)21:58
*** slaweq has quit IRC21:59
timburkeall right, we're about at time. thank you all for coming, and thank you for working on swift!22:00
timburke#endmeeting22:00
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"22:00
openstackMeeting ended Wed Oct 14 22:00:13 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)22:00
openstackMinutes:        http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-10-14-21.00.html22:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-10-14-21.00.txt22:00
openstackLog:            http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-10-14-21.00.log.html22:00
*** zaitcev has left #openstack-meeting22:00
*** raildo has quit IRC22:01
*** slaweq_ has quit IRC22:02
*** slaweq has joined #openstack-meeting22:05
*** vkmc has quit IRC22:10
*** vkmc has joined #openstack-meeting22:10
*** erbarr has quit IRC22:13
*** erbarr has joined #openstack-meeting22:15
*** luyao has quit IRC22:18
*** luyao has joined #openstack-meeting22:20
*** rcernin has joined #openstack-meeting22:34
*** rcernin has quit IRC22:36
*** slaweq has quit IRC22:36
*** rcernin has joined #openstack-meeting22:36
*** TrevorV has quit IRC22:45
*** yasufum has joined #openstack-meeting22:59
*** jamesmcarthur has quit IRC23:11
*** yasufum has quit IRC23:14
*** jamesmcarthur has joined #openstack-meeting23:17
*** tosky has quit IRC23:27
*** mlavalle has quit IRC23:29
*** yasufum has joined #openstack-meeting23:34
*** yasufum has quit IRC23:39
*** yasufum has joined #openstack-meeting23:39
*** rcernin has quit IRC23:45
*** rcernin has joined #openstack-meeting23:51
*** yasufum has quit IRC23:54
*** yasufum_ has joined #openstack-meeting23:54
*** dklyle has quit IRC23:57

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!