21:00:09 #startmeeting swift 21:00:10 Meeting started Wed Mar 11 21:00:09 2020 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:11 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:14 The meeting name has been set to 'swift' 21:00:17 who's here for the swift meeting? 21:00:22 o/ 21:00:30 o/ 21:00:31 o/ 21:00:32 o/ 21:00:34 o/ 21:01:00 o/ 21:01:19 agenda's at https://wiki.openstack.org/wiki/Meetings/Swift 21:01:37 i've got a couple things to point out before we get to updates, though 21:02:07 there's an interesting call to "consolidate" on the ML: http://lists.openstack.org/pipermail/openstack-discuss/2020-March/013236.html 21:02:57 i'm not sure what exactly that will mean in practice, or whether we'd even be impacted by it, but i figure it's a thread to be aware of 21:03:56 pretty sure i mentioned it in -swift, but in case you missed it, after Victoria, we'll have Wallaby: http://lists.openstack.org/pipermail/openstack-discuss/2020-March/013006.html 21:04:38 and it looks like they've approved the schedule for Victoria: https://releases.openstack.org/victoria/schedule.html 21:05:35 that's all i've got for general openstack news -- any questions or comments? 21:05:52 so many links to get lost reading :P 21:06:12 hehe sorry -- i can wait a bit :-) 21:08:04 all right, on to updates! 21:08:12 #topic waterfall EC 21:08:35 i don't know that clayg's actually around right now... 21:08:59 Poke him harder. 21:09:10 but he's been hard at work trying to figure out what we'd want out of something similar to concurrent gets but for EC 21:09:14 I often forget about our meeting. 21:09:51 turns out, it's a bit of a mess 21:11:24 part of the trouble is that we've *also* got some logic where if we start hitting read timeouts, the proxy will try to find another node to read from 21:12:46 to make that still work with EC we put each "frag lane" (as clayg's been calling them) in its own ResumingGetter 21:13:57 but then there's *also* logic to have us spin up more than ndata frag lanes in some cases (specifically, when a single node may have more than one frag due to rebalances) 21:14:59 so... like i said, it's a bit of a mess. it might be a bit before there's a patch that'd actually be ready to merge. and we might come asking you guys about trade-offs that we might make 21:15:27 Thanks for explaining these bits already, I'm not familiar with that logic 21:15:52 I don't understand why we cannot request 2 fragments from the same node in the same time. We do exactly that for PUT, don't we? 21:18:10 we can, following https://review.opendev.org/#/c/215276/ 21:18:10 patch 215276 - swift - Enable object server to return non-durable data (MERGED) - 43 patch sets 21:18:57 i'm not sure that we do for PUT, though... oh, except i was saying "node" when i really meant "disk" 21:19:44 so it means each frag-lane needs to get ndata fragments to decode each segment, hmm 21:21:12 i was talking to clay about this a bit yesterday; if i understood him right, the models is more like each frag-lane is responsible for getting one frag ready and reading it, and we need ndata frag-lanes to respond 21:21:15 even the segment that is decoded is just different from other frag-lane's ones like concurrent gets, isn't it? 21:22:08 but since they're all sharing the same node_iter, it gets messy trying to spawn off extra requests 21:22:22 hmm 21:24:09 like, if if we have a 3+2 policy, you might spin up lanes A, B, and C ... all of which are kinda sluggish to respond. so the concurrency timeout pops and you... do what? the cheap thing is to have each lane spawn another request: A', B', C' 21:25:10 but then you still can't respond even if you've got A, B, and A' in hand, since A and A' are in the same "lane" 21:26:04 and meanwhile, C' must be going to a handoff, which we don't expect to have anything useful anyway 21:27:05 otoh, you could spin up new lanes D and E... but then we lose the ResumingGetter behavior of going to another disk when there's a read timeout mid-stream 21:27:52 i see 21:28:01 anyway, clayg's got a couple exploratory patches up: p 709326 and p 711342 21:28:01 https://review.opendev.org/#/c/709326/ - swift - Enable more concurrency for EC GET - 2 patch sets 21:28:03 https://review.opendev.org/#/c/711342/ - swift - wip: asyc concurrent ecfragfetcher - 2 patch sets 21:29:09 and there's probably some more history i could dig up to remind us how we got here and what was informing our design decisions at the time, so we could sanity-check that those assumptions/goals are still valid 21:29:40 but i don't have that all collected right now -- i'll try to get some stuff together for next week or the week after, maybe start an etherpad 21:30:02 #topic lots of small files 21:30:33 rledisez, alecuyer how's it going? should i be getting the feature branch up-to-date? 21:31:14 so, last week alecuyer worked on the new key format to save CPU. it's mostly done. for the comoing week, he will work on the algorithm to select the volume where to write an incoming object 21:31:45 I'll take some time to try to merge master in the feature branch 21:32:03 I think last time I checked (few weeks ago), there were minor conflicts, nothing big IIRW 21:32:13 sounds great! thanks for keeping us updated 21:32:39 anything else you need from us? things to review, or even just to be thinking about? 21:32:54 alecuyer: ^ 21:33:31 I don't think so yet (although of course any advice or comment is helpful!), 21:33:52 👍 21:34:03 #topic CORS 21:34:05 I'll try to push what has been done so far on the feature branch once we get it rebased with romain? 21:34:18 yeah, that sounds great 21:35:21 so i've been getting CORS loaded back into my head -- the last time i gave it much thought was when torgomatic was poking at pulling it out to middleware in https://review.opendev.org/#/c/528106/ 21:35:21 patch 528106 - swift - Move CORS to middleware. - 5 patch sets 21:36:10 "updated: 2years, 2 months ago" oh 21:36:24 at the time, i started trying to write some functional tests -- https://review.opendev.org/#/c/533028/ 21:36:24 patch 533028 - swift - Add some functional CORS tests - 7 patch sets 21:37:36 which have proven really useful in getting myself familiar with the browser model and how these three agents (swift, browser, and some third-party js) interact 21:38:53 in the most-recent patchset, i've even got a test script that will spin up a webserver running on localhost, use selenium to drive a variety of browsers, run the tests, and print out some results 21:39:12 wow 21:39:20 great 21:39:20 next up, i need to figure out how to run that as a gate job 21:40:34 now, the end goal (for me) is to get some app that works with S3 to also work against Swift 21:41:13 i'm not actually 100% sure what this app is, or how it's used, though :-( hopefully i'll be able to get more details on that in the next week or so 21:41:50 but toward that end, i started poking at adding some s3api CORS tests in https://review.opendev.org/#/c/710354/ 21:41:50 patch 710354 - swift - Add CORS func tests for s3api - 3 patch sets 21:42:50 and fixing up s3api so they can actually pass 21:42:59 in https://review.opendev.org/#/c/710330/ and https://review.opendev.org/#/c/710355/ 21:43:00 patch 710330 - swift - s3api: Pass through CORS headers - 2 patch sets 21:43:01 patch 710355 - swift - s3api: Allow CORS preflight requests - 4 patch sets 21:43:22 Maybe some kind of gallery? 21:44:47 well, the pass/fail output looks something like http://paste.openstack.org/show/790568/ 21:45:10 looks nice 21:45:59 i originally wanted to get it working more like/with unittest, but... that went poorly (rife with import side-effects, overall pretty terrible) 21:46:31 i settled on TAP (https://en.wikipedia.org/wiki/Test_Anything_Protocol) mainly for its simplicity 21:47:35 so, i've got a bunch of stuff that i think is ready for review... but my bigger question at the moment is: does anyone else have much interest in CORS? 21:48:08 It used to be more important when Swift was supposed to back websites, like images or geo data. 21:48:09 honestly, if it weren't for this one user, i probably would've continued ignoring it 21:48:30 What does that user do, unless it's a secret? 21:49:03 we rarely got requests/complaints, but some people use it for sure 21:49:04 i don't remember! notmyname's been the main contact -- i need to cut out the middleman ;-) 21:50:01 that said, I like the idea of a middleware for that 21:50:22 basically, i'm trying to figure out whether i should be bugging you all for reviews, or just clayg and tdasilva :P 21:50:36 side question: does anybody already measured the cost of a middleware? (thinking of latency) 21:51:35 i've not -- though i'd expect it will tend to vary a lot with the middleware and its scope of responsibility 21:52:42 timburke: of course. I was more thinking of the basic noop middleware, just the overhead of having a middleware in the pipeline 21:53:01 it was jus crossing my mind, it's probably negligible 21:53:06 * timburke nods 21:53:17 extra function calls add up, for sure 21:53:37 oh yeah, and while investigating all of this CORS stuff, i noticed that it can be used to check whether encryption is enabled or not -- https://review.opendev.org/#/c/712237/ should fix it, though 21:53:38 patch 712237 - swift - encryption: Expose decrypted metadata via CORS - 2 patch sets 21:54:02 ok, want to still have some time for discussion, so 21:54:07 #topic open discussion 21:54:20 anything else to bring up? 21:55:06 just one thing on MD5 "replacement" 21:55:11 I had some strange case reviewing one of rledisez' patches. 21:56:04 well, forget it, it's so unclear in my head right now that I can't even formulate my point. i'll think more :) 21:56:21 Running tox -e py27 has a bad case of tests interfering: it only fails when testing all-up, but not individually. 21:56:28 rledisez, next week :-) 21:56:31 zaitcev: yeah, I saw your comment, thx for trying it out, I spent so much time trying to figure out what could happen 21:56:32 https://review.opendev.org/704892 21:56:32 patch 704892 - swift - proxy: stop sending frags to PyECLib with a Queue - 1 patch set 21:57:31 zaitcev, yeah, rledisez called out the trouble before -- we're suspecting some kind of eventlet shenanigans? gotta dig in some more though; definitely a blocker to merging as-is 21:57:33 It all seems to have something to do with StopIteration leaking somewhere. Seems like those queues isolated callers from it. But with yields all over I cannot track who calls what. Well, not at a glance. 21:58:03 ok 21:58:28 seems like a tactical scale problem, just annoyingly slippery 21:58:44 well, I need to go 21:59:01 yeah, it's getting to be that time 21:59:13 just, one last announcement: today is the last day of SwiftStack! tomorrow, clayg, tdasilva, notmyname and myself will all be nvidians 21:59:43 the king is dead long live the king! 21:59:54 it's been a crazy ride, and i can't wait to see where this next part of the journey takes us :-) 22:00:20 have fun, all of you, new challenges for sure :) 22:00:30 all right 22:00:40 thank you all for coming, and thank you for working on swift! 22:00:45 #endmeeting