21:00:14 <clayg> #startmeeting swift
21:00:15 <openstack> Meeting started Wed Feb 26 21:00:14 2020 UTC and is due to finish in 60 minutes.  The chair is clayg. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:16 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:18 <openstack> The meeting name has been set to 'swift'
21:00:25 <clayg> anyone here for the swift meeting today?
21:00:38 <seongsoocho> o/
21:00:44 <alecuyer> hi
21:01:07 <kota_> hi
21:01:14 <rledisez> hi o/
21:01:41 <mattoliverau> o/
21:01:53 <clayg> tdasilva: if i'm remember where you're at it's getting kinda late for you?
21:03:19 <clayg> Ok, so Tim is busy - let's go ahead and do a quick meeting
21:03:25 <clayg> Agenda is at https://wiki.openstack.org/wiki/Meetings/Swift
21:03:37 <clayg> #topic vancouver
21:03:50 <clayg> I think Tim just wants to reminder everyone this is coming up
21:04:10 <clayg> we're still in planning and I guess we still have some time
21:04:31 <clayg> If anyone has any questions ... probably best to email timburke :D
21:04:41 <mattoliverau> lol
21:04:43 <alecuyer> hehe
21:05:40 <clayg> #topic concurrent gets for EC
21:05:58 <mattoliverau> I assume this is the waterfall topic on gerrit?
21:06:02 <clayg> I peeled this one off our queue - my first attempt didn't work out 😢
21:06:14 <mattoliverau> :(
21:06:18 <clayg> mattoliverau: yeah, waterfall-ec https://review.opendev.org/#/c/709326/
21:07:13 <clayg> I thought I saw a way to life the concurrency context up one level where EC could manage it across all the ResumingGetters... but I don't think it's gunna work 😞
21:08:29 <clayg> basically we DO want to to have ndata responses, but picking which "frag lane" to spawn the extra requests in is going to be ... maybe impossible?
21:09:19 <mattoliverau> I was things do we want that? does it matter if we look at a few hand offs that might actaully have some ndata one them
21:09:35 <mattoliverau> (just thinking out loud)
21:10:11 <alecuyer> you mean we could start a few extra requests, in case some of them fail?
21:10:12 <clayg> So I think it's time to put the GetOrHeadHandler back together and let it work as intended for replicated requests then just extend they way we're already spawning and joining requests in the ECObjectController so it works like replicate concurrent_gets
21:10:35 <clayg> then maybe in the end it's possible to pull out something smaller or simpler that can be re-used - but for now I'm just trying to get both cases working the way we want
21:11:14 <clayg> mattoliverau: I think replicated (and ec) should not open a request to a handoff until at least one primary comes back with a timeout or error
21:12:04 <clayg> alecuyer: yes, the config options should *allow* an operator to spawn up to ec_n_unquie_frags (data+parity) requests concurrently and take the first ndata
21:12:29 <clayg> alecuyer: unfortunately when testing that case I found the original design didn't degrade gracefully into that working well
21:12:30 <alecuyer> clayg:  that sounds good (to me at least? :D)
21:12:36 <alecuyer> ok
21:13:21 <clayg> alecuyer: in the general sense I think we'll open connections to the first ndata sorted by timing - then after a small timeout (500ms) open another connection (or more?) to the other primaries to make up the difference for those yet to come back
21:13:42 <clayg> ... but definately as soon as we have ndata responses we can start to iter out the fragarchives and return a response
21:13:52 <alecuyer> right
21:13:54 <clayg> any questions?
21:14:37 <mattoliverau> what can we do to help? need anything from us at this point. Or should I just review the topic?
21:14:44 <zaitcev> So, reconstruct every time, even if systematic scheme could possibly deliver data in a few more milliseconds?
21:15:21 <clayg> zaitcev: I don't think we have a natural prefernce to node order currently - default sort order is timing - so parity frags might already be coming back
21:15:41 <clayg> zaitcev: in general we assume the rebuild is cheap compared to all the other work of getting bytes off disk and across a network
21:15:53 <clayg> mattoliverau: I was hoping I would need some help - but it seems i'm back to the drawing board
21:16:02 <rledisez> zaitcev: is there an algorithmic difference between only data frag and data+parity frags?
21:16:08 <clayg> I'd like to keep moving - hopefully I'll have some more to sa ynext week
21:16:11 <rledisez> zaitcev: I always assumed it was the same math
21:16:20 <mattoliverau> :(
21:16:33 <clayg> #topic forced client backoff on 503
21:17:23 <clayg> just this morning in a side conversation with Ed on the proxyfs team we ended up thinking about forcing clients to backoff by throwing some sleep on our 503 responses
21:17:34 <mattoliverau> rledisez: data can just be basically appended because it's the data, parity need to be be computed before appendings. So getting ndata should be more effiecent. AFAIK
21:18:00 <mattoliverau> oh interesting
21:18:06 <kota_> mattoliverau: correct
21:18:07 <rledisez> mattoliverau: thx, very instructive
21:18:13 <clayg> the issue came up in a thundering heard scenario where an artificial load generator would do like 200 requests slam a bunch of disks then wait a fixed amount of time before doing it again
21:18:55 <clayg> the "solution" was to make do some random dithering to the retries - but he pointed out we could do that server side
21:19:00 <clayg> I honestly hadn't considered it
21:19:18 <clayg> Has anyone ever done anything like this in middleware or on other services?
21:19:49 <mattoliverau> ok so backing off at while point? from client to proxy for proxy to the storage nodes. proxy is stateless so how do we deem its time to backoff or are we thinking another middleware?
21:19:52 <clayg> i assume it could all be operator configuable... just threw it on the agenda to get some feedback
21:20:16 <clayg> mattoliverau: yeah it might not be a backoff where it gets longer and longer - but it could sleep a random amount
21:20:24 <clayg> mattoliverau: i'm not sure would might be useful or expected
21:20:53 <clayg> most of the time we already hit a timeout which is kinda long, so maybe 0-500ms extra sleep isn't a big deal...
21:21:00 <clayg> but I'm also not sure how useful it would be
21:21:08 <alecuyer> too long would probably hold too many ports open ? (slowloris like issue?)
21:22:09 <clayg> alecuyer: yeah if the client can reliably generate a 5xx it's a vehicle for attack, maybe could be mitigated with some limit or bounds
21:22:18 <mattoliverau> like adding a back off window. potentailly adding it to the rate limitting middleware. But the proxy being stateless might make tracking the data harder.
21:22:54 <zaitcev> Don't we already have a rate limiting middleware?
21:23:03 <zaitcev> Right, that one.
21:23:21 <clayg> mattoliverau: I was thinking more the simple case or just adding some jitter, but I could see...
21:23:41 <clayg> I think if we started doing exponential backoff (via ratelimiting middleware) that could get really akward for well behaved clients
21:24:09 <clayg> I think rate limit currently only effects PUT/DELETE (it was specifically for container update limiting)
21:24:10 <mattoliverau> yeah, true. jitter is a good start.. starting simple is always a good starting point.
21:24:17 <clayg> but it'd probably be a good place to look
21:24:23 <clayg> ok, thanks for the feedback
21:24:39 <clayg> #topic open discussion
21:24:50 <clayg> https://wiki.openstack.org/wiki/Swift/PriorityReviews
21:25:25 <clayg> looks like python-swiftclient went out maybe?  https://github.com/openstack/python-swiftclient/releases
21:27:00 <clayg> I think we did a pretty good job with Tim gone!
21:27:08 <zaitcev> I feel like I promised some improvements to dark data and I forgot about it.
21:27:10 <clayg> Fully expect everything to be back to normal next week
21:27:15 <zaitcev> ok
21:27:41 <clayg> zaitcev: hah
21:28:00 <zaitcev> I only remember that I tried to get Romain's buy-in.
21:28:41 <rledisez> yeah, I remember that, need to go back to that review
21:30:01 <clayg> I might have to keep working on https://bugs.launchpad.net/swift/+bug/1863841
21:30:04 <openstack> Launchpad bug 1863841 in OpenStack Object Storage (swift) "container listing with limit=N is returning additional containers" [High,Confirmed]
21:30:29 <clayg> Tim and I are not really sure what we're gunna do about https://review.opendev.org/#/c/333331/
21:30:43 <clayg> I think both of those have functests now tho, so that's progress
21:31:00 <clayg> the query param thing is for https://bugs.launchpad.net/swift/+bug/1553714
21:31:01 <openstack> Launchpad bug 1553714 in OpenStack Object Storage (swift) "SLO multipart-manifest=put ignored with temp-url" [Medium,In progress]
21:31:52 <kota_> interesting
21:31:52 <zaitcev> That query parameters thing looks awfully familiar in theme to something I liked before.
21:33:27 <clayg> ok, well that's it from me - can we call it early?
21:34:17 <mattoliverau> we can :) then kota_ seongsoocho and I can get breakfast ;)
21:34:20 <clayg> thanks for making swift awesome!
21:34:21 <clayg> #endmeeting