21:00:48 <timburke> #startmeeting swift
21:00:48 <openstack> Meeting started Wed Jan 27 21:00:48 2021 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:00:49 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:00:51 <openstack> The meeting name has been set to 'swift'
21:00:55 <timburke> who's here for the swift meeting?
21:00:56 <mattoliverau> o/
21:01:00 <seongsoocho> o/
21:01:02 <kota_> o/
21:01:11 <rledisez> o/
21:01:34 <acoles> o/
21:02:33 <timburke> as usual, the agenda's at https://wiki.openstack.org/wiki/Meetings/Swift
21:02:47 <timburke> first up
21:02:55 <timburke> #topic ssync and non-durables
21:03:14 <clayg> o/
21:03:17 <timburke> acoles, clayg how's it looking?
21:04:01 <acoles> we deployed the fix to ssync this week and it has cleaned up some handoff partitions that were stuck with only non-durable frags :)
21:04:24 <acoles> that's https://review.opendev.org/c/openstack/swift/+/770047
21:04:28 <clayg> aka TOTAL SUCCESS!!! :shipit:
21:04:49 <acoles> zaitcev: thanks for review
21:05:11 <timburke> i see it's already got one +2; how important do you think it is to get more eyes on it before merging?
21:05:23 <clayg> timburke: i clicked +A
21:05:37 <timburke> 🎉
21:05:39 <acoles> *really important* :D
21:05:43 <timburke> that was easy :-)
21:05:48 <acoles> JK
21:05:50 <mattoliverau> lol
21:06:00 <timburke> #topic orphaned shard dbs
21:06:14 <zaitcev> BTW, does it work? Do you have handsoff partitions getting drained?
21:06:30 <acoles> zaitcev: yes
21:06:44 <clayg> zaitcev: YES!  it's amazing!  you do NOT want to be running swift w/o this patch 💪
21:06:56 <acoles> zaitcev: we had at least one non-durable frag that we monitored and it was cleaned up
21:07:32 <timburke> i think i lost track of who's leading the charge on orphaned shards -- mattoliverau, clayg, acoles?
21:07:41 <mattoliverau> Things are progressing here. We've been working on the shrinking side of things.
21:07:56 <mattoliverau> the only reclaim when root has no shards patch has landed
21:08:13 <mattoliverau> which is a stop gap, and stops orphaned shards
21:08:35 <mattoliverau> acoles has https://review.opendev.org/c/openstack/swift/+/771885
21:09:06 <mattoliverau> which allows us to fully collapse back up with shrinking all driven by root (which is amazing work BTW).
21:09:19 <acoles> the root reclaim patch is struggling to get through gate but has +A
21:09:37 <mattoliverau> now just filling in the missing middle pieces
21:09:58 <mattoliverau> namely more shrinking. acoles compact patch is where things are going
21:10:23 <acoles> yep, so I'm hoping we can make progress on chain of patches starting with https://review.opendev.org/c/openstack/swift/+/771885
21:10:30 <mattoliverau> that will not only make manage-shard-ranges work better, but it's code that lives in the sharder that will improve autosharding
21:10:48 <clayg> that's where i started!
21:10:50 <acoles> which would give us cli tool to shrink away less populated shards
21:11:15 <mattoliverau> So we're getting closer
21:11:41 <timburke> sounds great. anything about the chain that we should discuss here?
21:11:57 <mattoliverau> I've been playing with dumping shrinking candidates to recon dump: https://review.opendev.org/c/openstack/swift/+/772624
21:12:40 <mattoliverau> Which is just a mirror of the top containers to shard. But shrinking works differently and so am thinking of changing the metrics we send to recon for shrinking
21:12:55 <mattoliverau> anyway I have comments in the commit message and having a play with it.
21:12:59 <acoles> timburke: nothing particular to flag up on that patch chain, but reviews welcome - there's quite a lot going on what with shrinking and also some overlap repair features
21:13:14 <mattoliverau> +1
21:13:19 <mattoliverau> reviews welcome :)
21:13:26 <timburke> i definitely like the direction you're headed in the last paragraph of that commit message, mattoliverau
21:13:47 <mattoliverau> yeah, wanted to see how ops used the data before I went too cowboy :)
21:13:58 <mattoliverau> *would use*
21:14:01 <acoles> +1 left you some comments today mattoliverau
21:14:11 <mattoliverau> oh cool, thanks acoles
21:14:19 <acoles> seems we have consensus on the way ahead
21:14:38 <acoles> i.e. report number of shrinkable ranges not object counts
21:15:25 <mattoliverau> Cool, yeah makes much more sense in shrinking speak
21:15:31 <timburke> maybe even "count of all objects in shrinkable shards"?
21:16:10 <mattoliverau> I'll have a play with the code :)
21:16:18 <timburke> all right. as long as we're already talking about sharding/shrinking...
21:16:25 <timburke> #topic sharding in train
21:16:37 <timburke> zaitcev, am i right to think this was your topic?
21:16:46 <zaitcev> timburke: yes
21:17:55 <zaitcev> At this moment I identified 20 patches and I'm cherry-picking all of them. Not many obvious conflicts. Some are purely stull like "add some probe tests". So let's say 15 patches in a go. Does it sounds reasonable to just file for review and have a shocking flood from bot on IRC?
21:18:29 <zaitcev> BTW, the total amount of patches in stable/train after the fork point is 54.
21:19:12 <zaitcev> Master is 579 ahead. Just to amuse you with the meaningless stats. But gosh, we're committing a ton of patches, guys.
21:19:48 <timburke> from my perspective, it doesn't matter too much -- most all the sharding patches i can think of fit squarely in the "worth backporting" category
21:20:05 <zaitcev> Okay
21:20:27 <zaitcev> I'll run through tests on my cluster and pile it upon you then.
21:21:01 <timburke> even looks like the stable gate is in half-way reasonable shape
21:21:41 <timburke> is there any time table on when you'd want a stable tag that i should keep in mind?
21:23:52 <timburke> i won't worry about it too much ;-)
21:24:06 <timburke> #topic relinker and part power increase
21:24:33 <clayg> timburke: i saw you fighting with probe tests
21:24:37 <timburke> rledisez, thanks for approving the state-file patch, and talking through suffix hashing with me!
21:26:01 <timburke> clayg, yeah, i kicked up https://review.opendev.org/c/openstack/swift/+/772772 -- i'm less worried now, but for a stretch i was seeing occasional failures
21:27:06 <timburke> i think it was because (1) i added a new file to the probe test, one that is created before relinking then not written to again and (2) we were a little sloppy about what we put in org_locations
21:28:51 <timburke> on the disk-unmounted patch, does anyone have thoughts on https://review.opendev.org/c/openstack/swift/+/769632/2/swift/common/utils.py#3262 ? should i back out that part so we raise EPERM?
21:30:27 <timburke> i think the other option for that patch would be to wrap the logger we pass down to the audit location generator so we can count warning/error calls and use that to inform our exit code...
21:30:55 <zaitcev> lessee
21:31:42 <rledisez> raising an error might be the best move. i'm assuming an operator at some scale is not watching the output of every command ran on its clusters…
21:32:32 <rledisez> a timeserie information? logger.increment('relinker.skipped-device')?
21:33:22 <rledisez> eh, it's not that different of a log. except I think people use more grafana than elasticsearch (especially these days :D)
21:34:16 <timburke> rledisez, yeah, that was my concern -- i know our sres will be running this across the fleet with ansible, and i want to make it really easy for them to know when it's safe to continue ;-)
21:34:26 <seongsoocho> Does logger.increment send metrics to statsd?
21:34:33 <clayg> seongsoocho: yes
21:35:05 <mattoliverau> or at least more people might be watching grafana live to see results (or at least closer to live) rather then looking at historic data and might say ops, missed that :P
21:35:22 <seongsoocho> okay. then That will be more helpful and also logging in log file
21:36:32 <rledisez> FWIW, I know we ran clush on the whole cluster with a specially crafted command that was reading the JSON file to get the status of all partitions before moving on to the next step. if a file was missing we were getting an error
21:37:10 <rledisez> so my vote would be either raise or log/metrics+a relinker status command
21:38:23 <timburke> all right, i might go ahead and back out the utils.py change for now (so the error pops and we get a non-zero exit), and look at counting logger.warning/error/exception calls as a follow-up
21:38:40 <clayg> noice
21:39:03 <timburke> there was one more patch i realized i'd need to have a smooth part-power-increase: https://review.opendev.org/c/openstack/swift/+/772419
21:39:54 <mattoliverau> relinker could dump failed/unmounted devices to recon so we have a list for the recon to try again later. But I guess we have the stat files that do pretty much the same.
21:40:57 <mattoliverau> oh nice, drop priv cool.
21:41:06 <timburke> so the relinker needs to read the hash prefix/suffix from /etc/swift/swift.conf -- but we deploy with that solely readable by root (and rely on the swift_user config in daemons to appropriately drop privs)
21:41:55 <timburke> so on master, i couldn't run the relinker as swift (EPERM), and running as root ended poorly (wrong owner for all the new partitions)
21:42:24 <clayg> timburke: it's not super obvious to me why swift:swift 600 isn't good enough, but yeah... we do want a sudo drop priv
21:42:46 <clayg> I feel like it would all look a lot more like everything else if we just had an /etc/swift/relinker.conf
21:44:22 <timburke> yeah, i was about to bring that up... 'cause there are some other things i'm thinking about doing, like ratelimiting on the location generator or adding ionice tunings like we've got for other daemons, where it feels a little weird to continue growing options
21:45:07 <timburke> do we have much precedent for conf files for kinda one-shot tools?
21:45:43 <timburke> i guess there's /etc/swift/bench.conf ...
21:47:05 <timburke> well, i'll take a look at it at some point
21:47:09 <timburke> that's all i had
21:47:14 <timburke> #topic open discussion
21:47:26 <timburke> what else should we discuss this week?
21:47:31 <zaitcev> clayg: it's in case something in swift is broken and wipes swift.conf
21:47:50 <clayg> zaitcev: maybe so!
21:47:59 <seongsoocho> timburke:  I saw the p769855 merged.  (Thanks for the patch) Is it safe to only relinker use recent code? In my case, object server is running in train and relinker is master
21:48:23 <timburke> zaitcev, good thought -- i'd been assuming that it was because you might consider the prefix/suffix "secrets"
21:49:59 <clayg> but sekret from *swift* 🤯
21:50:01 <timburke> seongsoocho, yes, i'd expect so. the imports in cli/relinker.py are pretty minimal and unlikely to change. of course, it's always worth testing in a dev/staging environment first ;-)
21:52:15 <seongsoocho> timburke:  okay. thanks!
21:52:29 <mattoliverau> he's back :)
21:52:32 <zaitcev> woops
21:52:45 <timburke> idk what that was about *shrug*
21:53:58 <timburke> it would probably be sufficient to pull up a saio on train, copy in cli/relinker.py from master, then run probe tests
21:54:25 <timburke> (and, if you want the drop-privs patch, be sure to grab bin/swift-object-relinker from master, too)
21:55:28 <timburke> all right, seems like we're winding down
21:55:42 <timburke> thank you all for coming, and thank you for working on swift!
21:55:47 <timburke> #endmeeting