| opendevreview | Tim Burke proposed openstack/swift master: Authors/ChangeLog for 2.31.0 https://review.opendev.org/c/openstack/swift/+/871854 | 01:10 |
|---|---|---|
| opendevreview | Jianjian Huo proposed openstack/swift master: Proxy: restructure cached updating shard ranges https://review.opendev.org/c/openstack/swift/+/870886 | 01:17 |
| opendevreview | Jianjian Huo proposed openstack/swift master: Proxy: restructure cached updating shard ranges https://review.opendev.org/c/openstack/swift/+/870886 | 03:30 |
| mcape | hello all... so the rocky sharding story goes like this: | 05:15 |
| mcape | I've had one big container A from the start of production usage of the storage, it had about 120M objects in it | 05:15 |
| mcape | so I went with a sharding, and created ~100 shards of that container. Fast forward 5 years. | 05:15 |
| mcape | I started to see some slowness with container-servers. | 05:15 |
| mcape | Saw that the A container reached 250M objects, and new one - container B - emerged with 30M+ objects. | 05:15 |
| mcape | So I did another round of sharding, sharded the B container, and few of shards of A container. | 05:15 |
| mcape | Only sharded a few out of 100+ because I saw that sharding the B, or maybe increased number of container-server workers helped with the initial problem. | 05:15 |
| mcape | Now, the log is full of warnings from shard-audit, lots of warnings about different DB's, but they all look similar: | 05:15 |
| mcape | https://pastebin.com/tX6UtHjd | 05:15 |
| mcape | I see successful requests for the objects in the 'missing shard' ranges | 05:17 |
| opendevreview | Alistair Coles proposed openstack/swift master: sharder: make misplaced objects lookup faster https://review.opendev.org/c/openstack/swift/+/871843 | 13:42 |
| timburke | mcape, it's strange -- that "missing range(s)" message should only happen when auditing root containers, but it's obviously a shard... | 16:17 |
| timburke | it's not clear to me that there *are* any missing ranges -- my next step would be to find the root container and check its shard ranges | 16:19 |
| timburke | that and run swift-container-info on the shard that swift's complaining about -- maybe it's something with us not properly detecting shards if they've been deleted? that seems kind of familiar... | 16:21 |
| timburke | a couple patches might be handy: | 16:26 |
| timburke | https://review.opendev.org/c/openstack/swift/+/738113 Identify deleted shards as shards | 16:26 |
| timburke | https://review.opendev.org/c/openstack/swift/+/773832 Do not delete root_path on ContainerBroker.delete_db | 16:26 |
| timburke | draft release notes are up for 2.31.0: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_77e/871854/2/check/build-openstack-releasenotes/77e0bf3/docs/current.html | 16:53 |
| timburke | please take a look -- if you spot any typos, think there's something missing, or feel like anything needs clarification, please leave a comment on https://review.opendev.org/c/openstack/swift/+/871854 | 16:54 |
| opendevreview | Alexander Fadeev proposed openstack/swift master: Present `pytest` steps in development guidelines https://review.opendev.org/c/openstack/swift/+/872031 | 17:33 |
| opendevreview | Jianjian Huo proposed openstack/swift master: Test py2 fix. https://review.opendev.org/c/openstack/swift/+/872046 | 19:41 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!