| opendevreview | Tim Burke proposed openstack/swift master: CI: pin tox at the project level https://review.opendev.org/c/openstack/swift/+/869135 | 01:33 |
|---|---|---|
| reid_g | Hello, Looking at this bug's https://bugs.launchpad.net/liberasurecode/+bug/1886088 upgrade impact. "Depending on the order in which shared libraries are loaded, your servers may already be reading and writing zlib CRCs, even with old liberasurecode. In that case, no special action is required and WRITING LEGACY CRCS DURING THE UPGRADE WILL CAUSE AN OUTAGE." How does writing legacy crcs cause an outage if liberasurecode 1.6.2 can read/write both? | 13:47 |
| reid_g | I don't think it affects us but was just curious. | 13:47 |
| reid_g | We are going to add a systemd override to swift-proxy, swift-object, swift-object-reconstructor systemd services to add the env var to enable legacy crcs on servers upgrade to 20.04 before we start swift services. We're on swift 2.25.2 | 13:52 |
| timburke | reid_g, libec 1.6.2 can read both zlib crcs and "legacy" crcs, and write one or the other | 16:56 |
| timburke | old libec (1.5.0 or earlier) can read and write just one of the two, and which one depends on library-load order -- if liberasurecode gets loaded before libz, it only knows how to use legacy crcs; if libz gets loaded before liberasurecode, it only knows how to use zlib crcs | 16:56 |
| timburke | the potential outage comes from rolling upgrades: if an upgraded node computes and writes EC fragments into the cluster and an old node goes to read it, the old node won't be able to. the impact of that varies: | 16:57 |
| timburke | if an upgraded proxy writes (default) zlib crcs to a bunch of older objects-servers that are using legacy crcs, the fragments will later get quarantined when the object-server goes to read them | 16:57 |
| timburke | if an upgraded reconstructor uses the env var to ensure it writes legacy crcs when rebuilding frags but pushes them to older object-servers that are using zlib crcs (because that's how the library load-order worked out), these will similarly be quarantined eventually | 16:57 |
| timburke | there's also situations like an upgraded container-reconciler moving data into an erasure-coded policy -- even if all of the object-servers have already been upgraded, any older proxies wanting to read the data may hit checksum errors and return 503s | 16:57 |
| timburke | the first two start to look like data loss, though with enough operator intervention it should still be recoverable; the last one is at least limited to "just" being an availability issue | 16:59 |
| timburke | my recommendation would be to use the script from https://bugs.launchpad.net/liberasurecode/+bug/1886088/comments/2 on a few fragments from various nodes in the cluster to determine whether you've got legacy or zlib crcs | 17:01 |
| timburke | in practice, i'd expect you to get a single answer on that -- if there are a mix, it'd indicate that library-load order is causing incompatibilities even with the old version | 17:03 |
| timburke | once you have an answer to whether legacy crcs are in use, you know whether you ought to set the env var (legacy crcs are used) or not (zlib crcs are used). once all nodes are upgraded, you can remove the env var and always write zlib crcs | 17:05 |
| reid_g | Ok. That is what I thought but the wording in that post confused me a little. We are currently writing legacy on our Ubuntu 18.04 servers. After upgrading to 20.04 they were writing zlib. | 17:58 |
| timburke | 👍 got it -- so yeah, adding the env var via systemd files makes perfect sense, and as long as all writers get it, the upgrade should go smoothly | 18:43 |
| opendevreview | Tim Burke proposed openstack/swift master: CI: pin tox at the project level https://review.opendev.org/c/openstack/swift/+/869135 | 18:53 |
| opendevreview | Tim Burke proposed openstack/swift stable/zed: CI: pin tox at the project level https://review.opendev.org/c/openstack/swift/+/869253 | 18:56 |
| timburke | almost meeting time! | 20:54 |
| timburke | #startmeeting swift | 21:00 |
| opendevmeet | Meeting started Wed Jan 4 21:00:02 2023 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| opendevmeet | The meeting name has been set to 'swift' | 21:00 |
| timburke | who's here for the swift meeting? | 21:00 |
| clayg | o/ | 21:03 |
| timburke | whoa, it's a clayg! | 21:03 |
| timburke | well, may as well get started | 21:03 |
| timburke | only one topic, really | 21:04 |
| timburke | #topic busted gate | 21:04 |
| clayg | yeah, when the gate is busted there isn't really much else to talk about 🤷 | 21:04 |
| clayg | how'd it go fighting with tox *today* !? 😁 | 21:04 |
| timburke | over the holidays, a new version of tox came out that caused a bunch of failures | 21:04 |
| timburke | i spent a bit of time making our tox.ini on master work with new tox, and things were good | 21:06 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/868732 | 21:06 |
| timburke | ...but then a few days went by, another tox was released, and new failures were seen | 21:06 |
| timburke | #link https://github.com/tox-dev/tox/issues/2811 | 21:06 |
| timburke | so my current approach is to just pin tox<4 | 21:08 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/869135 | 21:08 |
| timburke | that looks most promising, and is currently working its way through the gate | 21:11 |
| timburke | at some point in the future, we can consider trying tox>=4 again, but i'm in no rush | 21:11 |
| timburke | meanwhile, i'll get some similar patches up against stable ('cause i'm currently getting a bunch of emails every night about how our stable branches have failing gates) | 21:12 |
| timburke | that's all i've got | 21:12 |
| timburke | #topic open discussion | 21:12 |
| timburke | anything else we should talk about this week? | 21:12 |
| clayg | all good 👍 | 21:16 |
| timburke | all right, let's call it early | 21:18 |
| timburke | thanks for coming clayg! happy new year! | 21:18 |
| timburke | #endmeeting | 21:18 |
| opendevmeet | Meeting ended Wed Jan 4 21:18:50 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:18 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/swift/2023/swift.2023-01-04-21.00.html | 21:18 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/swift/2023/swift.2023-01-04-21.00.txt | 21:18 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/swift/2023/swift.2023-01-04-21.00.log.html | 21:18 |
| opendevreview | Merged openstack/swift master: CI: pin tox at the project level https://review.opendev.org/c/openstack/swift/+/869135 | 21:49 |
| opendevreview | Tim Burke proposed openstack/liberasurecode master: Replace Flat XOR link with an archived version https://review.opendev.org/c/openstack/liberasurecode/+/869266 | 23:38 |
| opendevreview | Tim Burke proposed openstack/swift stable/yoga: CI: pin tox at the project level https://review.opendev.org/c/openstack/swift/+/869268 | 23:58 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!