21:00:35 <timburke> #startmeeting swift
21:00:35 <opendevmeet> Meeting started Wed May  3 21:00:35 2023 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:35 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:35 <opendevmeet> The meeting name has been set to 'swift'
21:00:42 <timburke> who's here for the swift meeting?
21:03:27 <timburke> well, i'll go ahead and call out some things going on for the record, anyway ;-)
21:03:36 <timburke> #topic gate issues
21:03:54 <timburke> the remaining flakey probe test has been resolved
21:04:03 <timburke> #link https://bugs.launchpad.net/swift/+bug/2017021
21:05:02 <timburke> the issue was in how we were determining which reconstructors to run
21:05:52 <timburke> in retrospect, it's amazing it *ever* passed (and probably a testament to how constrained the SAIO environment is; we should consider how we can get more virtual nodes in there)
21:06:05 <timburke> #link https://review.opendev.org/c/openstack/swift/+/881672
21:06:06 <mattoliver> o/
21:06:15 <timburke> ^^^ the fix, if anyone wants more details
21:06:21 <timburke> mattoliver, o/
21:06:49 <timburke> #topic EC "generator already executing" errors
21:06:49 <mattoliver> Nice
21:07:03 <timburke> looks like we have a fix in hand!
21:07:09 <timburke> #link https://review.opendev.org/c/openstack/swift/+/880356
21:08:06 <timburke> good job indianwhocodes and clayg for running that down -- might still be good for us to write up an upstream bug we can point people at later if someone reports this for an older version of swift
21:08:26 <mattoliver> +1
21:09:38 <timburke> fix seems to make sense to me; the test change needed to demonstrate it was pretty subtle and a reminder that it's hard to really be *sure* that tests demonstrate what we *think* they do
21:09:59 <timburke> #topic infinite recursion error in tests
21:10:40 <mattoliver> That topic around ominous
21:11:14 <timburke> for a while now, i (and others) have noticed an infinite recursion error down in ssl handling when trying to run func tests in my SAIO
21:11:59 <timburke> clayg and shreeya (who i don't think is on IRC yet) found a good way to avoid it, though!
21:12:26 <mattoliver> oh cool, nice work Shreeya! we should get her on here
21:13:44 <timburke> the core trouble was that requests-mock (required by some tests for s3token) incudes a pytest plugin which would cause requests to get imported before we get a chance to call eventlet.monkey_patch()
21:14:38 <timburke> there was some debate about explicitly reloading ssl so it'd be properly monkey-patched, or pinning to an older requests-mock that didn't include the plugin...
21:15:10 <timburke> then clayg noticed that we could just disable the thing via config options in tox.ini!
21:15:23 <timburke> #link https://review.opendev.org/c/openstack/swift/+/882105
21:15:34 <mattoliver> oh cool
21:16:24 <timburke> that's currently stacked on another change, though -- i think i'll swap the order if no one gets to it before me
21:16:43 <timburke> last up
21:17:00 <timburke> #topic reading py2 non-ascii metadata from py3
21:17:08 <timburke> #link https://launchpad.net/bugs/2012531
21:18:02 <timburke> we got a bug report about this more than a month ago -- it took longer than i wanted, but i think i've finally got a decent fix now
21:18:09 <timburke> #link https://review.opendev.org/c/openstack/swift/+/878558
21:18:11 <mattoliver> could we squash it down?
21:18:34 <mattoliver> oh yeah, that bug!
21:19:10 <timburke> mattoliver, i was thinking keep 'em separate -- the pytest plugin fix will be really nice all on its own; i'm tired of uninstalling/reinstalling requests-mock depending on which tests i want to run :P
21:19:33 <mattoliver> lol, fair enough!
21:19:53 <mattoliver> ether way lets get a +2/+A on it ;)
21:20:24 <timburke> the metadata fix is a little gross -- and feels brittle, like some future cpython change could come along and throw off our assumptions
21:20:59 <mattoliver> I do have it on my todo list to review.. just got distracted with a downstream release this week.
21:22:10 <timburke> so i might try to do a follow-up to add some third way of writing down metadata that wouldn't require this heuristic to determine whether (some) metadata was written down with wsgi strings or not
21:23:06 <timburke> but i think we could have that as a separate piece of work, and try to get edausq unblocked before then
21:23:29 <timburke> so if you've got time to review, mattoliver, that'd be great -- thanks!
21:23:41 <timburke> that's all i've got by way of updates
21:23:46 <timburke> #topic open discussion
21:23:55 <timburke> anything else you'd like to bring up mattoliver?
21:25:18 <jrosser> hello - i have a question which came up relating to swift deployed using openstack-ansible, and this https://opendev.org/openstack/swift/commit/3053c53ef7e9dd3efea10c535021786f7e6e92f2
21:25:32 <mattoliver> I'm still writing tracing unit tests that confirm the expected spans are present in the middlewares that I've instrumented. Just working through versioned writes. atm. So tedius and boring, but getting through it slowly
21:26:12 <jrosser> for a use case where all things need to be TLS, is the warning about not using TLS in that patch correct?
21:26:26 <timburke> thanks for continuing to push on it mattoliver
21:27:32 <timburke> jrosser, i think i'd still recommend using an external TLS-terminator (haproxy, nginx, hitch, etc.) over eventlet's (which is what you'd get by configuring cert_file/key_file in swift)
21:28:02 <jrosser> the thing is we can run into, for example, regulatory requirements that mandate TLS everywhere
21:28:26 <mattoliver> jrosser: the proxy already does a bunch cpu (we find) so we use a terminator that speaks the proxy protocol
21:28:49 <mattoliver> the terminator can be on the same server. so the last hop is loop back.
21:29:02 <mattoliver> I gues
21:29:14 <mattoliver> I guess that'll still be TLS everywhere
21:29:37 <timburke> and you can couple it with something like the proxy protocol from https://github.com/openstack/swift/commit/661838d968c163cec716cd5c8bcce9f5c69d18d4 if you're worried about preserving client ip info
21:29:44 <jrosser> yes, we would normally do that with uwsgi for other services in OSA
21:30:27 <jrosser> i am just checking if there is a preferred / tested setup for these scenarios
21:32:02 <mattoliver> Well we have a very large (proably the biggest) prod and thats what we do. We use hitch on each proxy node and use the proxy protocol that tim mentions. If that helps.
21:32:54 <jrosser> ok thats useful info - thanks
21:33:20 <timburke> "external terminator using proxy protocol v1" is the best solution i know of. fwiw, i run hitch at work (as mattoliver mentioned) and haproxy at home. i seem to recall OVH doing soemthing similar (though if they told me which terminator they use, i don't recall)
21:34:13 <timburke> the bigger thing to watch out for with "TLS everywhere" is backend communications. might be able to call it good with a wireguard network, though?
21:35:21 <jrosser> tbh we need to look at this in more detail
21:35:21 <mattoliver> good point. and yeah, if you need/want a software solution.
21:36:24 <jrosser> the focus so far has been getting everything between the frontend loadbalancer (haproxy) and the backends covered
21:37:11 <timburke> i seem to recall some interested parties years ago looking at using https connections for proxy <-> backend communication, but don't think it gained much traction (unfortunately)
21:37:54 <jrosser> anyway - this has been insightful and gives a direction to look in, thankyou
21:38:25 <mattoliver> nps, hit us up if you need any more info or anything else
21:38:44 <jrosser> sure, will do
21:38:47 <timburke> happy to help! keep us posted
21:40:00 <timburke> all right, i think i'll wrap it up a little early
21:40:01 <mattoliver> timburke: I think that's all I got, or I can think of atm.. been a bit distracted (as I've said).
21:40:15 <timburke> thank you all for coming, and thank you for working on swift!
21:40:19 <timburke> #endmeeting