| opendevreview | Merged openstack/swift master: Move base CI job to jammy https://review.opendev.org/c/openstack/swift/+/863545 | 01:32 |
|---|---|---|
| opendevreview | Tim Burke proposed openstack/swift master: CI: Add py310 unit test job https://review.opendev.org/c/openstack/swift/+/850947 | 03:59 |
| opendevreview | Clay Gerrard proposed openstack/swift master: Delete s3api MPU segments when the manifest expires https://review.opendev.org/c/openstack/swift/+/800701 | 04:15 |
| opendevreview | Tim Burke proposed openstack/swift master: Preserve client-provided paths that start '//' https://review.opendev.org/c/openstack/swift/+/863441 | 04:21 |
| opendevreview | Andre Aranha proposed openstack/swift stable/wallaby: CI: Add nslookup_target to FIPS jobs https://review.opendev.org/c/openstack/swift/+/863621 | 11:24 |
| opendevreview | Tim Burke proposed openstack/swift master: tests: Tolerate NoSuchBucket errors when cleaning up https://review.opendev.org/c/openstack/swift/+/863670 | 16:40 |
| opendevreview | Alistair Coles proposed openstack/swift master: WIP: restructure cached shard ranges https://review.opendev.org/c/openstack/swift/+/863562 | 18:35 |
| opendevreview | Jianjian Huo proposed openstack/swift master: Proxy: add metrics related to backend error limiter. https://review.opendev.org/c/openstack/swift/+/863446 | 19:16 |
| opendevreview | Jianjian Huo proposed openstack/swift master: Proxy: add metrics related to backend error limiter. https://review.opendev.org/c/openstack/swift/+/863446 | 19:21 |
| opendevreview | ASHWIN A NAIR proposed openstack/swift master: Add support of Sigv4-streaming https://review.opendev.org/c/openstack/swift/+/836755 | 20:59 |
| paladox | Hi, i'm wondering if anyone can tell me how i can depool a node whilst the data staying available to load? | 21:21 |
| paladox | i found https://mindmajix.com/openstack/removing-nodes-from-a-cluster | 21:23 |
| paladox | which i used but our data isn't showing... | 21:23 |
| timburke_ | paladox, what's the cluster topology like? how many disks, how many replicas? | 21:35 |
| paladox | We have a server per disk (hdd) and it's 5 (although we're removing two because the disk is too slow causing issues. We're going to setup two new servers to replace those two. Also replica is set at 1 because we don't have the disk space to store more then one copy. (Yeh i'm aware that comes with a risk of data loss). But i don't expect that if you drain that your data become inaccessible until it replicates? | 21:37 |
| timburke_ | try setting request_node_count=5 in proxy-server.conf -- as long as the disk still has at least one assignment, it should work | 21:39 |
| paladox | Ah! Thanks, will try | 21:39 |
| paladox | that didn't fix it :/ | 21:41 |
| paladox | I've set the weight on the two servers to 0 to have its data drained | 21:42 |
| timburke_ | i think you'll need to give 'em enough weight to get at least one assignment, unfortunately :-( | 21:42 |
| paladox | oh, what would you suggest? | 21:43 |
| timburke_ | add a little weight and rebalance -- right at the end, you might need to just tolerate the outage for that partition until replication can move the data off | 21:46 |
| paladox | 0.25? | 21:47 |
| timburke_ | FWIW there's a chain of patches that would help with the no-assignments case, but it's not there yet -- idea is to track the previous assignments in the ring and insert them into the handoffs list for GET/HEAD | 21:47 |
| timburke_ | maybe? depends on the other weights in the ring | 21:47 |
| paladox | ah nice | 21:47 |
| paladox | https://www.irccloud.com/pastebin/5SNvzUua/ | 21:48 |
| paladox | is how i have it currently | 21:48 |
| timburke_ | alternatively, if you're willing to hack up some code, you could crib a bit from https://review.opendev.org/c/openstack/swift/+/740866/2/swift/common/ring/ring.py | 21:49 |
| timburke_ | yeah, 0.25 might do it. might need to go a little higher -- could try 0.25, rebalance, then if it wasn't enough bump it up to 0.4 and try again | 21:51 |
| paladox | ok, thanks! And that'll drain the data as well, right? | 21:51 |
| paladox | https://www.irccloud.com/pastebin/mvJF6Max/ | 21:53 |
| timburke_ | yeah, all of it except for that one partition. let replication run for a while, watch the disks drain to just the one part, then drop weight to 0 again and take the outage on 0.1% of the data for a bit | 21:53 |
| timburke_ | you'll want to run `swift-ring-builder object.builder` again to check the number of assignments | 21:54 |
| paladox | thanks! And thanks again for all the help :) | 21:54 |
| paladox | That'll be cool when you can set it to 0 and it won't cause any data to be inaccessible | 21:55 |
| timburke_ | yeah -- the proxy part of the chain ends at https://review.opendev.org/c/openstack/swift/+/857758 -- i need to add some tests, though | 21:58 |
| paladox | data works now! | 22:01 |
| timburke_ | 🎉 | 22:02 |
| timburke_ | fwiw, if/when you get the disk space for more than one replica, `swift-ring-builder object.builder set_replicas <replicas>` will be handy | 22:02 |
| timburke_ | <replicas> can even be something like 1.1 so you can gradually increase it without lighting up everything all at once | 22:03 |
| paladox | thanks! | 22:09 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!