| ozzzo_work | I need to upgrade Swift from queens to train. I'm looking at this document: https://docs.openstack.org/operations-guide/ops-upgrades.html | 16:40 |
|---|---|---|
| ozzzo_work | The first sentence appears to be referring to an easier upgrade process for Swift. Where can I find that document? | 16:41 |
| timburke__ | ozzzo_work, swift has been committed for many, many years to ensuring that upgrades can be done smoothly and with near-zero downtime. unfortunately, i don't think we've documented it very well, though -- the best write-up i recall was by notmyname ages ago: https://web.archive.org/web/20201111163653/https://www.swiftstack.com/blog/2013/12/20/upgrade-openstack-swift-no-downtime/ | 16:53 |
| timburke__ | even though it's from 2013, the process looks mostly the same today. starting in ussuri you'll be able to further reduce client impact with seamless reloads: https://opendev.org/openstack/swift/commit/1107f2417 | 16:58 |
| timburke__ | the tl;dr is: upgrade storage servers first (typically object, then container, then account, if you can do it fine-grained like that), then proxies. proxies require a little more care, since you'll want to take them out of your load-balancer and wait for connections to drain before upgrading. old configs work with new code, though there may be some circumstances where you'll want to effect some config change prior to upgrading. | 17:09 |
| timburke__ | release notes should do a good job of calling out when and under what circumstances that may be desirable | 17:09 |
| ozzzo_work | thank you timburke__ ! | 18:02 |
| ozzzo_work | this is very helpful | 18:02 |
| ozzzo_work | What about rolling back? I'd like to test the upgrade in the lab a few times before starting on dev/qa/prod. How hard is it, to roll back versions? | 18:06 |
| timburke__ | in a lab environment, it's probably fine -- you might need to clear out the hashes.pkl files on your object nodes or some things like that. depending on how much trouble it is to set up, you might be better rebuilding the environment on the old version, though -- there are some things (like schema migrations in the container layer) that don't really roll back | 18:10 |
| timburke__ | definitely recommend testing your upgrade procedure a few times before doing it in prod, big +1 there ;-) | 18:10 |
| opendevreview | Tim Burke proposed openstack/swift master: common: Stop translating log messages https://review.opendev.org/c/openstack/swift/+/823423 | 18:37 |
| opendevreview | Tim Burke proposed openstack/swift master: common: Stop translating a bunch of printed messages and exceptions https://review.opendev.org/c/openstack/swift/+/823424 | 18:37 |
| timburke__ | i tried to split that up between things that are definitely covered by https://bugs.launchpad.net/swift/+bug/1674543 and everything else. i maybe should have split that second one up even better (operator-facing vs client-facing, say) but got impatient. at any rate, i suspect we may want some discussion on the second one | 18:41 |
| ozzzo_work | timburke__: : how can we work around the schema migrations? Can we save the old containers and re-use them when rolling back? | 19:29 |
| timburke__ | ozzzo_work, trouble is you'd lose whatever updates landed between the snapshot and rollback | 19:34 |
| timburke__ | fwiw, old code should mostly deal with new schemas fine, but (1) it's not a well-tested workflow and (2) you won't get to repeatedly test the migration for those containers | 19:34 |
| ozzzo_work | timburke__: : Obviously we can't do that in dev/qa/prod but I think it would be fine in the lab; it is only used for testing. The problem with rebuilding is that it takes a long time, so we only want to do that as a last resort | 20:06 |
| ozzzo_work | looking at the rocky, stein and train release notes, I don't see any talk about database schema changes. Should I be looking elsewhere, or is it safe to assume that schema changes would be mentioned in the release notes? | 20:07 |
| timburke__ | makes sense. fwiw, i'm usually more worried about the client impact from service restarts during an upgrade than schema changes and the like | 20:08 |
| timburke__ | those aren't typically called out in the release notes; it's not generally something that operators need to think about | 20:09 |
| ozzzo_work | I guess I should assume the opposite then; there most likely are db schema changes from queens->train | 20:13 |
| opendevreview | Tim Burke proposed openstack/swift master: squash: Trim tempurl signature with obscuremap in the logs (CVE-2017-8761) https://review.opendev.org/c/openstack/swift/+/823432 | 21:30 |
| timburke__ | ozzzo_work, yeah, looks like we added the shard_range table to container DBs in rocky (2.18.0, specifically) | 22:33 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!