21:00:14 #startmeeting swift 21:00:15 Meeting started Wed Feb 26 21:00:14 2020 UTC and is due to finish in 60 minutes. The chair is clayg. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:16 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:18 The meeting name has been set to 'swift' 21:00:25 anyone here for the swift meeting today? 21:00:38 o/ 21:00:44 hi 21:01:07 hi 21:01:14 hi o/ 21:01:41 o/ 21:01:53 tdasilva: if i'm remember where you're at it's getting kinda late for you? 21:03:19 Ok, so Tim is busy - let's go ahead and do a quick meeting 21:03:25 Agenda is at https://wiki.openstack.org/wiki/Meetings/Swift 21:03:37 #topic vancouver 21:03:50 I think Tim just wants to reminder everyone this is coming up 21:04:10 we're still in planning and I guess we still have some time 21:04:31 If anyone has any questions ... probably best to email timburke :D 21:04:41 lol 21:04:43 hehe 21:05:40 #topic concurrent gets for EC 21:05:58 I assume this is the waterfall topic on gerrit? 21:06:02 I peeled this one off our queue - my first attempt didn't work out 😢 21:06:14 :( 21:06:18 mattoliverau: yeah, waterfall-ec https://review.opendev.org/#/c/709326/ 21:07:13 I thought I saw a way to life the concurrency context up one level where EC could manage it across all the ResumingGetters... but I don't think it's gunna work 😞 21:08:29 basically we DO want to to have ndata responses, but picking which "frag lane" to spawn the extra requests in is going to be ... maybe impossible? 21:09:19 I was things do we want that? does it matter if we look at a few hand offs that might actaully have some ndata one them 21:09:35 (just thinking out loud) 21:10:11 you mean we could start a few extra requests, in case some of them fail? 21:10:12 So I think it's time to put the GetOrHeadHandler back together and let it work as intended for replicated requests then just extend they way we're already spawning and joining requests in the ECObjectController so it works like replicate concurrent_gets 21:10:35 then maybe in the end it's possible to pull out something smaller or simpler that can be re-used - but for now I'm just trying to get both cases working the way we want 21:11:14 mattoliverau: I think replicated (and ec) should not open a request to a handoff until at least one primary comes back with a timeout or error 21:12:04 alecuyer: yes, the config options should *allow* an operator to spawn up to ec_n_unquie_frags (data+parity) requests concurrently and take the first ndata 21:12:29 alecuyer: unfortunately when testing that case I found the original design didn't degrade gracefully into that working well 21:12:30 clayg: that sounds good (to me at least? :D) 21:12:36 ok 21:13:21 alecuyer: in the general sense I think we'll open connections to the first ndata sorted by timing - then after a small timeout (500ms) open another connection (or more?) to the other primaries to make up the difference for those yet to come back 21:13:42 ... but definately as soon as we have ndata responses we can start to iter out the fragarchives and return a response 21:13:52 right 21:13:54 any questions? 21:14:37 what can we do to help? need anything from us at this point. Or should I just review the topic? 21:14:44 So, reconstruct every time, even if systematic scheme could possibly deliver data in a few more milliseconds? 21:15:21 zaitcev: I don't think we have a natural prefernce to node order currently - default sort order is timing - so parity frags might already be coming back 21:15:41 zaitcev: in general we assume the rebuild is cheap compared to all the other work of getting bytes off disk and across a network 21:15:53 mattoliverau: I was hoping I would need some help - but it seems i'm back to the drawing board 21:16:02 zaitcev: is there an algorithmic difference between only data frag and data+parity frags? 21:16:08 I'd like to keep moving - hopefully I'll have some more to sa ynext week 21:16:11 zaitcev: I always assumed it was the same math 21:16:20 :( 21:16:33 #topic forced client backoff on 503 21:17:23 just this morning in a side conversation with Ed on the proxyfs team we ended up thinking about forcing clients to backoff by throwing some sleep on our 503 responses 21:17:34 rledisez: data can just be basically appended because it's the data, parity need to be be computed before appendings. So getting ndata should be more effiecent. AFAIK 21:18:00 oh interesting 21:18:06 mattoliverau: correct 21:18:07 mattoliverau: thx, very instructive 21:18:13 the issue came up in a thundering heard scenario where an artificial load generator would do like 200 requests slam a bunch of disks then wait a fixed amount of time before doing it again 21:18:55 the "solution" was to make do some random dithering to the retries - but he pointed out we could do that server side 21:19:00 I honestly hadn't considered it 21:19:18 Has anyone ever done anything like this in middleware or on other services? 21:19:49 ok so backing off at while point? from client to proxy for proxy to the storage nodes. proxy is stateless so how do we deem its time to backoff or are we thinking another middleware? 21:19:52 i assume it could all be operator configuable... just threw it on the agenda to get some feedback 21:20:16 mattoliverau: yeah it might not be a backoff where it gets longer and longer - but it could sleep a random amount 21:20:24 mattoliverau: i'm not sure would might be useful or expected 21:20:53 most of the time we already hit a timeout which is kinda long, so maybe 0-500ms extra sleep isn't a big deal... 21:21:00 but I'm also not sure how useful it would be 21:21:08 too long would probably hold too many ports open ? (slowloris like issue?) 21:22:09 alecuyer: yeah if the client can reliably generate a 5xx it's a vehicle for attack, maybe could be mitigated with some limit or bounds 21:22:18 like adding a back off window. potentailly adding it to the rate limitting middleware. But the proxy being stateless might make tracking the data harder. 21:22:54 Don't we already have a rate limiting middleware? 21:23:03 Right, that one. 21:23:21 mattoliverau: I was thinking more the simple case or just adding some jitter, but I could see... 21:23:41 I think if we started doing exponential backoff (via ratelimiting middleware) that could get really akward for well behaved clients 21:24:09 I think rate limit currently only effects PUT/DELETE (it was specifically for container update limiting) 21:24:10 yeah, true. jitter is a good start.. starting simple is always a good starting point. 21:24:17 but it'd probably be a good place to look 21:24:23 ok, thanks for the feedback 21:24:39 #topic open discussion 21:24:50 https://wiki.openstack.org/wiki/Swift/PriorityReviews 21:25:25 looks like python-swiftclient went out maybe? https://github.com/openstack/python-swiftclient/releases 21:27:00 I think we did a pretty good job with Tim gone! 21:27:08 I feel like I promised some improvements to dark data and I forgot about it. 21:27:10 Fully expect everything to be back to normal next week 21:27:15 ok 21:27:41 zaitcev: hah 21:28:00 I only remember that I tried to get Romain's buy-in. 21:28:41 yeah, I remember that, need to go back to that review 21:30:01 I might have to keep working on https://bugs.launchpad.net/swift/+bug/1863841 21:30:04 Launchpad bug 1863841 in OpenStack Object Storage (swift) "container listing with limit=N is returning additional containers" [High,Confirmed] 21:30:29 Tim and I are not really sure what we're gunna do about https://review.opendev.org/#/c/333331/ 21:30:43 I think both of those have functests now tho, so that's progress 21:31:00 the query param thing is for https://bugs.launchpad.net/swift/+bug/1553714 21:31:01 Launchpad bug 1553714 in OpenStack Object Storage (swift) "SLO multipart-manifest=put ignored with temp-url" [Medium,In progress] 21:31:52 interesting 21:31:52 That query parameters thing looks awfully familiar in theme to something I liked before. 21:33:27 ok, well that's it from me - can we call it early? 21:34:17 we can :) then kota_ seongsoocho and I can get breakfast ;) 21:34:20 thanks for making swift awesome! 21:34:21 #endmeeting