21:00:20 <timburke> #startmeeting swift
21:00:20 <opendevmeet> Meeting started Wed Jul 12 21:00:20 2023 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:20 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:20 <opendevmeet> The meeting name has been set to 'swift'
21:00:29 <timburke> who's here for the swift team meeting?
21:00:36 <kota> hi
21:01:05 <mattoliver> o/
21:01:44 <timburke> i actually update the agenda! though there's surely some stuff i missed :P
21:01:47 <timburke> #link https://wiki.openstack.org/wiki/Meetings/Swift
21:01:54 <timburke> first up
21:02:04 <timburke> #topic ssync metadata corruption
21:02:14 <timburke> #link https://bugs.launchpad.net/swift/+bug/2020667
21:02:25 <timburke> the fix merged!
21:03:01 <mattoliver> yay
21:03:04 <kota> great
21:03:05 <timburke> i figure that's as good a spot as any to cut a new release -- it's been something like 3 months since our last one
21:03:32 <timburke> so i'll work on getting some release notes up this week
21:03:54 <mattoliver> yeah, good idea
21:04:51 <timburke> #topic logging/metrics for backend HEAD requests on cache misses
21:05:00 <timburke> #link https://review.opendev.org/c/openstack/swift/+/884931
21:05:46 <timburke> it's gotten some positive feedback, but it looks like acoles still has some hesitations
21:06:02 <timburke> meanwhile, we've got other patches starting to build on it
21:06:12 <timburke> #link https://review.opendev.org/c/openstack/swift/+/885798
21:07:03 <mattoliver> oh a logged app intersting idea
21:07:11 <timburke> i know i previously suggested it maybe ought to have an upstream bug -- i might still try to write that
21:07:14 <mattoliver> yeah sucks that we lost those metrics cool
21:08:28 <timburke> i suppose one question is whether we want to prioritize getting that landed ahead of the next release, too
21:08:50 <timburke> i think we've only had the one release where we dropped the metrics
21:09:04 <indianwhocodes> o/
21:09:09 <timburke> if so, i *definitely* ought to write up that launchpad bug
21:11:15 <mattoliver> yeah, write up the bug. I'll give this a review and lets try and get it landed before the end of the week
21:11:24 <mattoliver> so it can be in the release
21:11:25 <timburke> thanks
21:12:14 <timburke> #topic partNumber support for s3api
21:12:17 <timburke> #link https://bugs.launchpad.net/swift/+bug/1735284
21:12:40 <timburke> it sounds like indianwhocodes is going to take a look at implementing this soon!
21:12:57 <indianwhocodes> yes its a priority!
21:13:10 <mattoliver> cool
21:14:08 <timburke> for some background: there's a parameter you can provide to GET/HEAD requests against AWS that will fetch individual parts of a multipart object
21:14:41 <timburke> we don't currently support it -- and not in a "we'll just ignore it" sort of a way -- we'll respond with a 405
21:14:44 <kota> nice
21:15:00 <kota> nice to have, i mean
21:15:35 <timburke> for a while it was undocumented, but at least one of AWS's SDKs would use it; a few years ago it got some documentation
21:17:08 <mattoliver> OK, so whats the plan, give partnumber support (or something like it) to SLO and then plumb  that support in s3api
21:17:08 <timburke> but what's really driving the prioritization is that we want to provide some means of verifying MPU downloads to clients. one way we can do that is to let clients download individual parts, calculate the MD5s of each part as it's downloaded, then calculate the MD5-of-MD5s to compare against the ETag for the MPU
21:17:16 <mattoliver> cause it also sounds nice for SLOs
21:19:00 <mattoliver> The bug talks about being able to do it for normal objects too, so how will that work? does it turn it into a byte request? or do we ignore non slos?
21:19:24 <timburke> mattoliver, yeah, exactly. it's a little debatable how much utility you get for SLOs, since the client could also just download the actual manifest, then grab the segments directly -- but it definitely seems like a way to make it easier to write a better client without going all the way to relying on a bunch of swift-isms
21:20:24 <mattoliver> in a normal object would it be 1/1000s of the object (or whatever the max partnumber was)
21:20:36 <mattoliver> so an auto calculated ranged request
21:21:06 <mattoliver> Sorry just thinking out loud now
21:22:06 <timburke> i think the thing to do would be to use a query param, like S3 does -- object-server doesn't even see the thing, so it'd automatically send back everything. then slo can look at the query param, see that it got a manifest as a response, and send back just the one segment. if it *wasn't* a manifest, again, we'll just send what we *did* get back
21:22:40 <mattoliver> kk
21:22:58 <timburke> other things going on...
21:23:06 <timburke> #topic container server performance
21:23:53 <timburke> jian jian has been doing some profiling, and noticed a lot of CPU time being spent on getting shard ranges
21:25:25 <timburke> it looks like he and acoles will be seeing how we can avoid unnecessary work and improve the container server
21:26:18 <timburke> mostly, this has been coming out of observations about much lower than expected performance, partitularly for container with large numbers of shards (think 10-20,000)
21:27:37 <timburke> ok, a more administrative note
21:27:42 <timburke> #topic next meeting
21:27:46 <mattoliver> yeah, when we were writing sharding we thought about large number of shards, but it was so far off I think we punted it. Well I guess were here now
21:27:59 <zaitcev> I don't get that partNumber thing... Why does Range: not work for those guys.
21:28:21 <timburke> #undo
21:28:21 <opendevmeet> Removing item from minutes: #topic next meeting
21:28:25 <mattoliver> probably could start having high water marks when syncing shards now that we're getting > 10k
21:28:26 <zaitcev> Actually n/m. It is what it is and we ought to be compatible.
21:29:25 <timburke> zaitcev, the main trouble is that the client doesn't know what the original sizes were for uploads -- so it doesn't know how to build the MD5-of-MD5s
21:29:40 <mattoliver> currently we always send them all on every container replication. So yeah there is a bunch of more to make sharding work better with alot more shards.
21:30:54 <timburke> (s3's way of doing it is almost but not quite the same as SLO's; it's good enough to think of how you would go about verifying an SLO download if you knew it was an SLO, but didn't have access to the actual manifest)
21:31:24 <zaitcev> ok
21:32:39 <timburke> like, range requests are good enough to get you parallel downloading, but you can't try to recreate the etag you got back unless you know the individual part sizes
21:33:25 <timburke> #topic next meeting
21:33:30 <timburke> i'm going to be out on vacation for a couple of weeks
21:34:05 <timburke> so i'm going to propose that we skip the next couple meetings, and come back Aug 2
21:34:26 <indianwhocodes> sounds good
21:34:27 <timburke> unless anyone else would like to volunteer to chair the meeting
21:35:28 <mattoliver> I'm happy to chair while your gone.. but also only if there is something to talk about.. so happy to skip.
21:35:39 <mattoliver> so skipping is fine.
21:36:11 <timburke> all right, then -- if anyone has anything they'd like to bring up, add it to the agenda and mattoliver will keep an eye out for it ;-)
21:36:30 <timburke> otherwise, we'll next meet up in August
21:36:36 <timburke> that's all i've got
21:36:42 <timburke> #topic open discussion
21:36:49 <timburke> anything else we ought to bring up?
21:36:56 <mattoliver> kk
21:37:05 <zaitcev> I'm going on vacation too, to Wisconsin.
21:37:32 <kota> i saw the nest ptg info in my mailbox...
21:37:34 <timburke> oh, nice! i spent a few years there -- where you going, zaitcev?
21:37:37 <kota> next
21:37:43 <mattoliver> oh yeah, I keep forgetting its summer time in the US.
21:37:55 <zaitcev> timburke: https://www.eaa.org/en/airventure
21:37:59 <timburke> oh, yeah! i heard something about the PTG...
21:38:21 <mattoliver> kota: yeah I saw that too! a virtual one
21:38:25 <timburke> zaitcev, looks exciting :-)
21:38:26 <kota> https://ptg2023.openinfra.dev/
21:38:47 <mattoliver> End of October
21:38:56 <zaitcev> So, next PTG is virtual again.
21:39:21 <mattoliver> looks like it.
21:39:43 <mattoliver> not sure if their alternating or reverting back to it.
21:40:02 <zaitcev> So, is there anything I can do to make acoles to review https://review.opendev.org/c/openstack/swift/+/787656
21:40:26 <zaitcev> Maybe I can review something...? But looks like you guys do well without me.
21:41:02 <mattoliver> acoles was away on vacation last week. I'll poke him about it for you if you like zaitcev
21:41:05 <timburke> we always appreciate more eyes :-) i'll try to take a look -- i'm long overdue to check it out again
21:41:15 <mattoliver> +1
21:41:20 <zaitcev> Technically it's your patch :-)
21:44:48 <timburke> all right, i think i'll call it
21:45:00 <timburke> thank you all for coming, and thank you for working on swift!
21:45:05 <timburke> #endmeeting