21:00:25 #startmeeting swift 21:00:25 Meeting started Wed Oct 21 21:00:25 2020 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:26 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:28 The meeting name has been set to 'swift' 21:00:34 who's here for the swift meeting? 21:00:43 o/ 21:00:45 hi 21:00:51 o/ 21:00:59 🥳 21:01:00 hi o/ 21:01:55 first order of business 21:02:00 #topic acoles is back! 21:02:08 🎉 21:02:26 hi everyone, it's nice to be here again 21:02:27 amazing 21:03:02 if there are no objections, i'd like to give him commit rights again 21:03:51 timburke: +2 21:04:28 i didn't realize we'd retired him 21:05:48 excellent -- 'cause i jumped the gun a little and already did it :-) 21:06:17 i figure, we can use all the help we can get -- thanks for coming back acoles, and so glad to have you back! 21:06:28 #topic ptg 21:06:41 and he has good timing, as we've got the PTG next week! 21:06:55 yey 21:07:02 topics continue to gathre on the etherpad 21:07:05 #link https://etherpad.opendev.org/p/swift-ptg-wallaby 21:07:16 \o/ 21:07:43 ...and i should be sure to set aside sometime for some ops feedback and community feedback 21:09:07 excellent 21:09:09 i don't really have much of a plan for next week, but as long as we're all together, thinking and chatting about swift, i know good things will come outof it :-) 21:10:18 i'll try to order items by level of interest again, probably on friday -- so put yourself down for topics that you care about! it'll give us a good way to prioritize conversations 21:11:26 and since we'll have the PTG together, i figure no irc meeting is necessary next week 21:11:47 timburke: how does it work - is all discussion over irc? 21:12:05 nah all zoom or whatever 21:12:08 video chats over zoom 21:12:18 ah ok 21:12:24 it was not too bad! not as good as whiteboard 😞 21:13:07 though we can also use jitsi via https://meetpad.opendev.org/swift-ptg-wallaby 21:13:21 We probably still need to figure out how to white board things 21:13:49 clayg: maybe you can use jamboard as a white board https://jamboard.google.com/ 21:14:04 worth a shot!!! 💪 21:14:06 fingers crossed that in another six months, things have quieted down enough that we can actually all be in the same physical room again 21:14:09 Nice 21:14:23 timburke: that would be GREAT - miss everyone 😢 21:15:21 i've got just one more "announcement"-type item 21:15:32 #topic review.openstack.org security incident 21:15:44 you all may have noticed that gerrit was down yesterday 21:15:57 * acoles not my fault 21:16:05 there's a nice write-up of what happened at 21:16:07 #link http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html 21:16:15 Lol, acoles comes back and then.. hmm 21:16:29 lol 21:17:18 it wasn't so bad - just a bummer day 21:17:24 i want to applaud the infra team's transparency and quick response; considering the scope of what needed to be checked, losing a day seems remarkably fast 21:17:36 +100 21:18:28 one of the action items was that "We should audit all changes for projects since 2020-10-01." 21:18:45 they already compiled the commits for us at 21:18:48 #link https://static.opendev.org/project/opendev.org/gerrit-diffs/ 21:19:22 and i've already confirmed swift, python-swiftclient, and liberasurecode commits look correct and expected 21:19:54 Nice work, thanks timburke 21:20:07 (pyeclib and swift-bench didn't even have enough activity to show up) 21:20:25 are there other projects that i'm forgetting about that i should check? 21:22:45 if anyone has other projects that they have a vested interest in, that's where you can go for first-hand data; expect most temas to put out statements on the mailing list as well (i certainly intend to, probably shortly after this meeting) 21:23:58 all right, on to updates! 21:24:00 #topic updates 21:24:16 we've got two patches that i think mostly just need reviews 21:24:27 #link https://review.opendev.org/#/c/706653/ 21:24:27 patch 706653 - swift - Let developers/operators add watchers to object au... - 37 patch sets 21:24:34 #link https://review.opendev.org/#/c/754242/ 21:24:34 patch 754242 - swift - Fix a race condition in case of cross-replication - 6 patch sets 21:24:54 i'd meant to look at both and wound up looking at neither :-( 21:25:55 does anyone have anything else they'd like to discuss about them? or is it mostly a matter of us finding time to review them? 21:26:20 i know zaitcev put a +2 on the replication patch -- thanks! 21:26:47 I'm worried that I missed some kind of deadlock there, but I cannot find any. I looked at all instances where lock_path is invoked for various paths. 21:27:00 So, what the heck, seems okay. +2 21:27:31 I owed Romain for looking at Dark Data 21:27:34 did patch 706653 have a predecessor - it rings a bell, and I see torgomatic is an author 21:27:34 https://review.opendev.org/#/c/706653/ - swift - Let developers/operators add watchers to object au... - 37 patch sets 21:27:45 Yes. 21:28:02 patch 212824 21:28:03 https://review.opendev.org/#/c/212824/ - swift - Let developers/operators add watchers to object audit - 21 patch sets 21:28:15 See, this is "alternative" 21:28:38 acoles: I worked on Sam 21:29:00 @zaitcev thanks 21:29:09 's patch first, but ended with a trouble trying to debug things, so I threw my toys down and forked it. 21:30:12 the main change (zaitcev, correct me if i'm wrong) was that sam's patch had a much more adversarial attitude towards the watchers, and so enforced process isolation 21:30:29 I created a logger for Sam's patch that actually worked, before the fork. 21:30:41 where this patch runs it all in the same process -- and if a watcher breaks the audit, you need to fix itor drop it 21:31:57 To an extent, we still isolate, as much as Python allows. But a plugin can run out of file descriptors, for example. 21:32:56 ok, so IIRC Sam's used a queue so watchers got notifications on 'best-effort' basis. Is this more guaranteed to notify watchers? 21:33:40 This is just a method call, no queue. 21:35:02 ...or call os._exit(0), or os.kill(os.getpid(), signal.SIGKILL), or time.sleep(1<<30)... 21:37:27 So, anyway. 21:38:12 If Alistair finds a moment to look, it would be good. But for the extra lock for replicators too. 21:39:08 sounds like we mainly need to get some reviews on them. i'll get the replicator guy squared away -- i've probably been overthinking my dev setup, and promise to leave *some* sort of review by the end of the week 21:40:11 I'll try but can't promise 21:40:16 zaitcev, do you know if david is planning to be at the ptg? i remember seeing audit watchers on the agenda 21:40:38 noice 21:40:38 do we think that'll mostly be a session to discuss and land it? 21:41:26 timburke: he was interested in it, but I don't know if he can do it. 21:42:17 no worries; just figured i'd check 21:42:37 all right 21:42:42 #topic open discussion 21:42:51 what else should we bring up this week? 21:43:05 is there any kind of sign up for ptg or just show up? 21:43:22 acoles, https://october2020ptg.eventbrite.com/ 21:43:35 thanks 21:43:37 though i'm sure you could also just show up and we'd make itwork ;-) 21:43:49 i want my t shirt 21:44:07 lol 21:44:12 ...who wants to tell him? 21:44:25 lol 21:44:49 awww 21:45:09 🤔 i *should* talk to notmyname, investigate doing a small run of shirts... 21:45:38 😬 cutting it close now 21:45:39 also, i wonderif we've still got some swiftstack shirts stashed somewhere -- collector's items, now! 21:45:44 Yeah, even if it's in red bubble or something so we pay for our own shirts, just need the design. 21:45:45 hahah 21:45:58 clayg, oh, almost certainly for a future event 21:46:31 oic 21:47:16 zoom list windows showing guys who wear same shirts should look great on the virtual event, so +1 21:47:30 i think I have a pile of those hpe iron on badges somewhere... 21:48:20 I have my yellow-orange "Swift Justice" shirt. 21:48:24 Justice! 21:48:43 i wore mine for acoles's first day back 21:49:02 oh! one other patch i'd like to call attention to -- https://review.opendev.org/#/c/758636/ 21:49:03 patch 758636 - swift - Add option to REPLICATE to just invalidate hashes - 2 patch sets 21:50:29 i've got this memory of talking about doing something like that as a *pre*-sync check, to try to feel out when there might be a newly-assigned primary 21:53:45 Curious. Does that help performance any, or what does it do? 21:53:49 still needs tests (and some fixups, looks like), but i wanted to have people start thinking about how we could make replication smarter if we had a cheap way to check remote suffix counts 21:55:57 so the idea specific to that patch is that when you're moving data as part of a rebalance (and so, suffixes are probably fairly dense, say 3k or so), the sender doesn't really need to wait for the rehash to clean up data; it just needs to make sure everybody got invalidated 21:57:54 but *in the future* i've got this thought like we could get rid of handoffs-first and instead have partition sync jobs in a priority queue of sorts, weighted by some combination of how many suffixes are in the partition (if it's a handoff) and when it was last successfully synced 21:59:09 timburke: nice! 21:59:40 skipping the re-hash IO on the post-rsync invalidate step seems like a quite reasonable enhancement to me 22:00:57 all right, we're about at time. oughta let kota_ and mattoliverau start their day, and acoles end his :-) 22:01:08 thank you all for coming, and thank you for working on swift! 22:01:14 Kk, so great to have you back acoles 22:01:16 acoles, again, greatto have you back! 22:01:22 #endmeeting