21:00:10 <timburke> #startmeeting swift
21:00:10 <opendevmeet> Meeting started Wed Oct 11 21:00:10 2023 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:10 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:10 <opendevmeet> The meeting name has been set to 'swift'
21:00:48 <mattoliver> o/
21:01:05 <timburke> who's here for the swift meeting?
21:01:42 <timburke> (sorry, i'm a little distracted trying to concurrently help my daughter with some homework)
21:02:00 <mattoliver> lol, np
21:02:17 <timburke> which also means i haven't updated the agenda
21:03:26 <timburke> i think the only big thing i wanted to bring up was the PTG
21:04:54 <timburke> i'll i still need to add my own availability, and i should probably nudge clayg, indianwhocodes, and a few others to add theirs
21:05:22 <timburke> but i'll book meeting slots soon
21:05:52 <mattoliver> yeah, I did mine
21:06:37 <timburke> and we'll continue to add topics to the etherpad
21:06:44 <timburke> #link https://etherpad.opendev.org/p/swift-ptg-caracal
21:06:54 <mattoliver> +1
21:07:13 <timburke> i started adding some today, and i see mattoliver's adding, too :-)
21:07:44 <mattoliver> yup, adding a few discussion topics. But will pad out with some more as I think of them.
21:08:23 <timburke> only a week and a half away now!
21:09:00 <timburke> that's the only major thing i wanted to bring up this week
21:09:03 <timburke> #open discussion
21:09:36 <timburke> mattoliver, since it's just you and me -- anything you'd like to bring up? or i can fill you in on some of what i've been doing with labeled metrics
21:10:24 <mattoliver> ahh, I don't have too much, I know I keep saying that, but other work keeps getting in my way. But I'd love to hear about labeled metrics!
21:11:43 <timburke> so at one point i was working toward a big-bang transition -- one patch that would touch absolutely *everywhere* that we emit metrics
21:12:14 <mattoliver> like tracing
21:12:22 <timburke> #link https://review.opendev.org/c/openstack/swift/+/885321
21:13:05 <timburke> this was rather horrible -- more than 1k line diff, and that's before even starting to look at fixing up tests
21:14:05 <timburke> on investigating, it seems like a more gradual transition shouldn't be too bad, operationally -- the statsd_exporter that i'm targeting is happy to eat a mix of labeled and legacy metrics
21:15:09 <mattoliver> kk
21:16:14 <mattoliver> I thought one benefit of the whole hog approach, was we could maybe decouple the logger from metrics. Is that something we can still do easily in a gradual approach? is that still the plan?
21:16:33 <timburke> so i did a spike to break it up as a chain: first, make some assertions about the actual bytes-on-the-wire that we're sending, then add the new can-emit-as-labeled-metrics api to StatsdClient (making sure that those on-the-wire tests still pass), then change the calls to use labels
21:17:06 <timburke> yeah, we probably could still do that... haven't looked hard at it yet, as it doesn't seem to be on the critical path
21:17:18 <mattoliver> kk
21:17:37 <timburke> that chain ends at
21:17:40 <timburke> #link https://review.opendev.org/c/openstack/swift/+/896969
21:18:26 <timburke> though it clearly needs a little more work
21:19:56 <timburke> but once we have the new api from https://review.opendev.org/c/openstack/swift/+/896968 i think we can spin out a bunch of separate patches for all the other places we want to start having labeled metrics
21:20:18 <mattoliver> oh I like the build up. More tests first to get a better picture and then moving up to labeled metrics.
21:22:52 <timburke> it seemed like a good way to de-risk us breaking legacy metrics -- in no small part because i realized that our biggest cluster will probably have to live with them for a while, as graphs get rebuilt to use the new stuff
21:22:57 <mattoliver> Nice, might need to do some research into some of these mode types, some I've heard others not so much
21:23:45 <mattoliver> oh yeah, now that's a great point. There is alot of infrastructure that revolves around metrics.. so there will be alot of work to use the new hotness.
21:24:50 <mattoliver> This is looking pretty great tim
21:25:23 <mattoliver> also thanks for comment links for each build_line func
21:26:06 <timburke> one idea that's come up is providing an option to have us emit *both* labeled and legacy metrics -- i'm torn about it, though -- if an operator wanted to put in the work to ensure metrics continuity, it seems likely to lead to double-counting...
21:27:03 <mattoliver> yeah. But I guess if you can point them to different end points, there could be a clear separation.
21:27:56 <mattoliver> this is a great start. I'll keep this tab open and work my way though it more indepth.
21:28:25 <timburke> thanks, mattoliver
21:28:45 <timburke> all right, that's all i've got
21:28:59 <timburke> i think i'll call it early and let you enjoy your morning :-)
21:29:10 <timburke> thanks for coming, and thanks for working on swift!
21:29:15 <timburke> #endmeeting