| openstackgerrit | Ghanshyam Mann proposed openstack/python-swiftclient master: [goal] Migrate testing to ubuntu focal https://review.opendev.org/744605 | 00:26 |
|---|---|---|
| openstackgerrit | Ghanshyam Mann proposed openstack/pyeclib master: [goal] Migrate testing to ubuntu focal https://review.opendev.org/744623 | 00:37 |
| *** psachin has joined #openstack-swift | 03:38 | |
| *** m75abrams has joined #openstack-swift | 04:11 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-swift | 04:33 | |
| *** mikecmpbll has joined #openstack-swift | 07:50 | |
| *** dsariel has joined #openstack-swift | 08:14 | |
| *** dsariel is now known as dsariel|PTO | 08:14 | |
| *** tdasilva has quit IRC | 09:11 | |
| *** tdasilva has joined #openstack-swift | 09:18 | |
| *** ChanServ sets mode: +v tdasilva | 09:18 | |
| *** mikecmpbll has quit IRC | 09:53 | |
| *** tkajinam has quit IRC | 09:55 | |
| *** mikecmpbll has joined #openstack-swift | 09:55 | |
| *** tkajinam has joined #openstack-swift | 11:27 | |
| *** mikecmpbll has quit IRC | 11:56 | |
| *** mikecmpbll has joined #openstack-swift | 12:04 | |
| *** jv_ has joined #openstack-swift | 14:03 | |
| *** tkajinam has quit IRC | 14:06 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/swift master: [goal] Migrate testing to ubuntu focal https://review.opendev.org/744607 | 14:17 |
| *** cschwede has joined #openstack-swift | 14:30 | |
| *** ChanServ sets mode: +v cschwede | 14:30 | |
| *** psachin has quit IRC | 14:45 | |
| *** m75abrams has quit IRC | 14:50 | |
| *** gyee has joined #openstack-swift | 14:58 | |
| *** dsariel|PTO has quit IRC | 15:44 | |
| *** zaitcev has joined #openstack-swift | 17:01 | |
| *** ChanServ sets mode: +v zaitcev | 17:01 | |
| *** mikecmpbll has quit IRC | 17:04 | |
| *** mikecmpbll has joined #openstack-swift | 17:06 | |
| clayg | timburke: it's possible I never really groked the second half of that EC GET error handling code. But theory is we're adding 404s in more cases than we should, and the is_success(best_bucket.status) is more restrictive... | 17:09 |
| timburke | yeah, that's my thinking -- the idea in https://github.com/openstack/swift/blob/2.25.0/swift/proxy/controllers/obj.py#L2458-L2463 seemed to be *just* to fill out for non-durable frags | 17:31 |
| timburke | in general, i'm not really sure about the use of best_response there when we're *already* doing filtering and making decisions about which responses to consider -- it leads to really funky (and not-uncommonly *useless*) logging when we get to https://github.com/openstack/swift/blob/2.25.0/swift/proxy/controllers/base.py#L1928-L1929 | 17:46 |
| timburke | i've seen lines like "Object returning 503 for []" when the actual responses gathered were 4x timeout, 1 200, 1 404, 2x 404 from handoffs | 17:48 |
| timburke | so it *looks like* there was some massive error limiting going on or something, but in reality we *really tried* to find something and got a *very* mixed result | 17:49 |
| clayg | yeah i'm worried about continuing to extend this "hide it from best_response" strategy 😬 | 18:13 |
| *** hoonetorg has joined #openstack-swift | 18:29 | |
| *** cschwede has quit IRC | 19:16 | |
| clayg | @timburke I feel like https://github.com/openstack/swift/blob/master/test/unit/proxy/controllers/test_obj.py#L2860 should be hitting the case we're worried about | 19:25 |
| clayg | maybe it's significant the test name says "will 503" then it asserts 404? | 19:25 |
| clayg | well, maybe if there's non-durable it *should* 404 🤔 | 19:26 |
| clayg | I guess `and best_bucket and not best_bucket.durable` was mostly a guard on `best_bucket == None` - is it possible the logic is correct in practice despite best_bucket always being true and bad buckets being durable? | 19:34 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: wip: tests for mixed frag 404 vs 503 https://review.opendev.org/746593 | 20:03 |
| clayg | @timburke I could really use some help zero-ing a test in on the failure you said you saw when playing with probes? | 20:03 |
| *** mikecmpbll has quit IRC | 20:09 | |
| *** mikecmpbll has joined #openstack-swift | 20:12 | |
| timburke | ok, i think i found my edge case again; wrote it up on p 746593 | 21:09 |
| patchbot | https://review.opendev.org/#/c/746593/ - swift - wip: tests for mixed frag 404 vs 503 - 1 patch set | 21:09 |
| timburke | basically, there were a *ton* of timeouts, mixed in with one 404 from primaries. the idea with p 744942 was that we should ignore the 404 and assume that there *likely* is data on the other servers -- or at least, it's worth the client trying again to find out | 21:12 |
| patchbot | https://review.opendev.org/#/c/744942/ - swift - Client should retry when there's just one 404 and ... - 5 patch sets | 21:12 |
| timburke | i'm increasingly convinced that the EC controller is going to need to just *entirely replace* best_response | 21:13 |
| timburke | maybe even make it behave more like a coroutine, where we feed responses to it and it either hands you a response or a sentinel that says "i need more responses" | 21:15 |
| clayg | yes! best_response can pound sand | 22:13 |
| clayg | @timburke i'm getting a 503 on EC when I stop my object servers | 22:13 |
| timburke | on which patch? | 22:14 |
| clayg | do you think there's a regression in https://review.opendev.org/#/c/711342/or not? | 22:14 |
| patchbot | patch 711342 - swift - Extend concurrent_gets to EC GET requests - 16 patch sets | 22:14 |
| clayg | oh, sorry, difference between GET and HEAD - I guess there *is* a regression there | 22:16 |
| clayg | well, that sucks | 22:17 |
| timburke | oh, funny... i suppose because it just punts up to the base controller at that point | 22:17 |
| timburke | 🤔 maybe HEAD *should* confirm there's at least ndata frags out there... | 22:18 |
| clayg | well, it doesn't bother me so much that it's different - but I am disappointed there was no test on master for the behavior 😞 | 22:26 |
| *** rcernin has joined #openstack-swift | 22:33 | |
| clayg | so the problem is we don't know if `not bucket.durable` is a good request that wasn't durable or a bad request 😞 | 22:36 |
| timburke | may have been one of those cases where it felt like "oh, well *obviously* if everything's down, we oughta 503" | 22:36 |
| timburke | yup | 22:36 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: wip: tests for mixed frag 404 vs 503 https://review.opendev.org/746593 | 22:57 |
| *** tkajinam has joined #openstack-swift | 23:07 | |
| mattoliverau | morning | 23:20 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!