21:02:00 <timburke> #startmeeting swift
21:02:00 <opendevmeet> Meeting started Wed Nov 29 21:02:00 2023 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:02:00 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:02:00 <opendevmeet> The meeting name has been set to 'swift'
21:02:08 <timburke> who's here for the swift team meeting?
21:02:24 <acoles> o/
21:03:29 <mattoliver> o/
21:04:40 <timburke> first off, sorry for the last-minute cancellation a couple weeks ago, and the (not explicitly mentioned) cancellation last week for the US holiday
21:05:00 <timburke> things have been a little hectic for me
21:05:48 <timburke> which also means i don't have much of an agenda for this week -- though i have the sense that there's been a lot of effort concentrated around the SLO/MPU part-number changes
21:05:55 <acoles> no worries
21:06:43 <mattoliver> yeah no worries Tim
21:07:14 <mattoliver> I've been really distracted the last 2 weeks, so unfortuantly I haven't done much myself anyway.
21:07:26 <timburke> is there anything that we'd like to draw extra attention to, or that has questions that need answering that we ought to bring up?
21:07:35 <acoles> we've merged the slo refactor that Clay worked so hard on :)
21:07:47 <timburke> 🎉
21:07:50 <mattoliver> \o/
21:08:04 <acoles> so our review effort is now focussed on the new partnumber support in SLO, and then s3api
21:08:32 <acoles> those are the  indianwhocodes patches
21:09:20 <timburke> oh, and gerrit's down for now... i think the SLO one was https://review.opendev.org/c/openstack/swift/+/894570 tho
21:09:45 <acoles> I think we've all been distracted by holidays and family stuff in last week or so
21:10:38 <timburke> speaking of distractions, there's a chance i'll be out for jury duty next week...
21:11:24 <timburke> i'll set a reminder for myself to get a message out one way or the other my tuesday evening, though
21:11:54 <mattoliver> kk, or just let me know and I'll pass it along
21:13:31 <timburke> so i've just got one other thing i've been thinking about the last few weeks:
21:13:48 <timburke> i noticed that our "seamless reloads" aren't always so seamless
21:14:28 <timburke> it's pretty easy to see: run swift-bench, then reload the proxy part way through -- there'll be a bunch of connection resets
21:15:17 <mattoliver> oh good test
21:16:08 <timburke> i'm pretty sure the issue comes down to incoming connections sitting in the old workers' accept queues; when we close the old listen socket, our accept() call immediately starts returning EBADF and all the queued connections get reset
21:16:53 <timburke> so i'm going down an ebpf rabbit hole to figure out how we might be able to prioritize the new worker sockets over the old ones
21:17:14 <mattoliver> ahh yeah. We could test that by turning the accept queue down to something tiny
21:17:29 <timburke> it'll be an adventure!
21:17:41 <mattoliver> yay
21:17:41 <acoles> so we wait for greenthreads to complete in flight requests but those in the accept queue are dropped?
21:17:59 <timburke> yup, as best i can tell
21:18:47 <mattoliver> the accept queue have been accepted by the OS but we haven't accepted them from our side of the socket. So if we're not pulling requests of the socket anymore, those clients miss out
21:19:26 <timburke> eh, the queue doesn't need to be tiny -- in fact, if we make it *too* small, you'll see the bad behavior regardless, as the kernel doesn't have anywhere to put the connection (so it'll just drop it)
21:19:29 <acoles> right
21:20:38 <mattoliver> yeah, I just meant to test if its the accept queue
21:21:28 <mattoliver> I wonder if there is a way to put the socket into close_wait but still deal with the requests (if that is even possible) not sure of the socket state machine.
21:22:39 <timburke> i think something like https://paste.opendev.org/show/bcMWJVwkAbAlhpt6xQY0/ shows the bad pretty well, and doesn't even need swift or eventlet
21:22:41 <mattoliver> Anyway, I super interesting research spike, good thinking Tim.
21:24:01 <timburke> but i think that's all i've got for this week
21:25:19 <mattoliver> kk, I don't have anything either. A big chuck of downstream work is now completed for me, so hopefully I'll get a bunch of swift time now, so maybe next week I'll have been more productive :)
21:27:19 <timburke> all right, i think i'll call it early
21:27:29 <timburke> thank you all for coming, and thank you for working on swift!
21:27:35 <timburke> #endmeeting