21:00:44 #startmeeting swift 21:00:45 Meeting started Wed Nov 11 21:00:44 2020 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:46 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:48 The meeting name has been set to 'swift' 21:00:54 who's here for the swift meeting? 21:01:25 o/ 21:02:04 hi o/ 21:02:20 hi 21:03:03 \o/ 21:03:46 as usual, the agenda's at https://wiki.openstack.org/wiki/Meetings/Swift 21:04:26 mostly just follow-up on work in progress (which reminds me, it's been a while since i read through the ML...) 21:04:52 #topic audit watchers 21:05:35 i know zaitcev has his +2 on there; i've been eaning to circle back on it since the ptg but haven't really 21:06:31 is anyone else interested in reviewing it? or is it mostly a matter of "tim needs more time"? 21:08:22 well, i'll see what i can do this coming week, then ;-) 21:08:43 #topic cross-replication locking 21:09:14 rledisez, i finally got around to setting up an env where i could repro it! 21:09:45 fix looks great; just had a few follow-up thoughts that i put in https://review.opendev.org/#/c/762393/ 21:09:46 patch 762393 - swift - Cross-SSYNC race follow-up - 1 patch set 21:10:45 did we ever write up a second bug specifically for rsync? 21:10:58 yes, I quickly looked at it. the follow up totally makes sense to me 21:11:33 timburke: I didn't, I created only 1 bugreport for both, but as patch will close it maybe we need a second one for rsync 21:12:10 yeah, that sounds good. do you want to write it or should i? 21:13:12 I will. I think it will mostly be a copy/paste; though, I don't know how to reproduce event if I'm sure it applies for rsync 21:13:21 *even 21:14:09 thanks! i'll see about fiddling with my ssync env to test rsync and plan on throwing some more info on the new bug if/as needed 21:15:07 speaking of replication... 21:15:14 #topic post-sync rehashing 21:16:25 so acoles approved https://review.opendev.org/#/c/758636/ and i know clayg is planning on packaging https://review.opendev.org/#/c/761801/ for our cluster in the near future 21:16:25 patch 758636 - swift - Have REPLICATE with suffixes just append to hashes... - 8 patch sets 21:16:27 patch 761801 - swift - Stop invalidating suffixes post-SSYNC - 4 patch sets 21:17:29 iirc there was a slight concren that on-disk files may take a bit longer to get cleaned up 21:18:17 but on the whole it seems like we've done a lot of good stuff to keep drives from filling up entirely, so it's probably not actually a problem 21:19:21 as much as anything, i just want to call these out as changes that are coming, in case people are running near capacity 21:19:54 #topic open discussion 21:20:07 anything else we should talk about today? 21:23:29 sorry distracted 21:23:43 did we already say we want to merge the SLO async thing? 21:23:53 https://review.opendev.org/#/c/733026/ 21:23:54 patch 733026 - swift - Add a new URL parameter to allow for async cleanup... - 16 patch sets 21:24:51 let's do it! i'm not scared! 21:25:11 https://review.opendev.org/#/c/759183/ is in a similar boat except mattoliverau has some ideas to make it better - i'm sure acoles and timburke will get to those by next week 21:25:11 patch 759183 - swift - memcache: Refuse to error limit the last available... (ABANDONED) - 2 patch sets 21:25:52 but pete is says he's +2 with ?asycn=on 21:26:05 so... that'll probably happen just so everyone knows 21:26:44 clayg: which 'similar' patch did you mean - the link is to an abandoned patch? 21:26:59 ... but yeah, that's all I got unmerged that i'd like to see landed - well the shrinking stuff -> but mattoliverau looked at that one too, and it's still WIP 21:27:00 https://review.opendev.org/#/c/741721/ 21:27:01 patch 741721 - swift - WIP: Fix sharder consistency issues - 11 patch sets 21:27:01 I'm guessing...shrinking 21:27:06 i think we went with https://review.opendev.org/#/c/761029/ for the memcache-error-limiting stuff -- reminds me that i ought to make some changed so our config-pushing to make sure we turn off the error-limiting when only writing out a single memcached server 21:27:06 patch 761029 - swift - memcache: Make error-limiting values configurable (MERGED) - 4 patch sets 21:27:51 oh yeah nix 759183 long live 761029 21:28:26 yeah i want to merge the shriking fixes but like... after they're more baked 21:28:45 the memcache stuff I guess is also not like "will merge any day" because we haven't shipped the new apporach I don't think 21:29:01 so it's really only the SLO that's coming down the pipe! cool! I feel really good about what we're carrying 21:30:08 supposedly we (NVIDIA) will see a bunch of deletes in the near future, so it'll be interesting to see whether we feel the need to turn allow_async_deletes back off ;-) 21:31:37 timburke: ahhaha - or just make the expirer faster ;) 21:33:43 fwiw, i know we're also trying to run down some issues with short reads and/or client disconnects 21:35:44 which has led me down various rabbit-holes like why we set eventlet.wsig.WRITE_TIMEOUT when it seems nothing in eventlet actually looks at it, or second-guessing the safety of eventlet.green.threading's thread-local storage 21:36:43 i'm sure we'll discuss it more as the investigation continues 21:38:19 all right, i think i'll call it 21:38:30 thank you all for coming, and thank you for working on swift! 21:38:36 #endmeeting