| mattoliver | zaitcev: which one is that? cephadm <-- I has to hack on that quite a bit as Suse was using it for SES release. I haven't been following what's been happening post being at Suse. | 03:00 |
|---|---|---|
| mattoliver | timburke: Sorry man, yeah, kinda fell off my radar.. too much stuff happening atm. I see zaitcev gave it a review though, so know it's in great hands. | 03:01 |
| zaitcev | mattoliver: I forgot already. IIRC they mentioned that cephadm is changed to work through that new daemon. | 03:01 |
| mattoliver | ahh ok | 03:02 |
| *** mabrams1 is now known as mabrams | 04:25 | |
| opendevreview | Matthew Oliver proposed openstack/swift master: wip: Cluster assisted Storage Policy Migration https://review.opendev.org/c/openstack/swift/+/173580 | 06:28 |
| opendevreview | Matthew Oliver proposed openstack/swift master: sharder: update shard storage_policy_index if roots changes https://review.opendev.org/c/openstack/swift/+/800748 | 06:28 |
| mattoliver | that first one is just a rebase of clayg 's old change. Basically I used his cli tool and migrate helper method, as it made confirming the issue and testing that the fix seems to work on my SAIO. I might have the change the second to just update the storage_policy_index in the container db directly. But just wanted to get something up. | 06:30 |
| mattoliver | as a first attempt | 06:30 |
| timss | Hi, on Swift 2.25 I'm getting "Error with Object server ip:port/device re: Trying to write to /AUTH_../container/object: ChunkWriteTimeout (10.0s)" on my proxy, and at the object node "PUT /device/... 499" (client closed request). Tried searching for a bit and seems that the scenarios where one is getting HTTP 499 was improved in this version specifically. Haven't been able to find much | 11:04 |
| timss | else in logs, any tips on where to start looking? The error seems to get triggered more easily as the number of requests increase, but we're talking like 4 threads from a single client so not much to begin with. | 11:04 |
| timss | I'm rolling Ubuntu packaged Swift, but found a note about fixing "ChunkWriteTimeout errors due to a race condition" in SwiftStack 7.2.0.1 (jan 30, 2020). That'd be before Swift 2.25.1 was released I believe, but perhaps it's a configuration issue rather than Swift bug? | 11:43 |
| zaitcev | I'd look at service times on drives with sar/iostat first. | 13:22 |
| opendevreview | Clay Gerrard proposed openstack/swift master: Add concurrency to reconciler https://review.opendev.org/c/openstack/swift/+/800802 | 15:34 |
| opendevreview | Tim Burke proposed openstack/swift master: s3api: Allow CORS preflights for pre-signed URLs https://review.opendev.org/c/openstack/swift/+/792490 | 17:52 |
| timburke_ | timss, sounds like the disks may have been overloaded. ballpark, how big is the cluster? do you know if the object disks are SMR? you might consider increasing the node_timeout on your proxy servers | 18:00 |
| timburke_ | i agree, though, 4 threads does *not* seem like it ought to be able to trigger much of anything :-/ might check on some disk health stats | 18:01 |
| *** timburke_ is now known as timburke | 18:12 | |
| zaitcev | maybe some unfortunate hotspotting or something else we don't know | 18:14 |
| timburke | yeah -- the storage policy could make a decent difference, too -- EC means more disks involved in every PUT, and higher risk of hitting a bad/overloaded disk | 18:22 |
| opendevreview | Merged openstack/swift master: reconciler: PPI aware reconciler https://review.opendev.org/c/openstack/swift/+/799561 | 18:53 |
| opendevreview | Tim Burke proposed openstack/swift master: Add concurrency to reconciler https://review.opendev.org/c/openstack/swift/+/800802 | 19:27 |
| opendevreview | Tim Burke proposed openstack/swift master: AUTHORS/CHANGELOG for 2.28.0 https://review.opendev.org/c/openstack/swift/+/799121 | 20:14 |
| kota | good morning | 20:57 |
| timburke | almost meeting time! | 20:58 |
| timburke | #startmeeting swift | 21:01 |
| opendevmeet | Meeting started Wed Jul 14 21:01:50 2021 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:01 |
| opendevmeet | The meeting name has been set to 'swift' | 21:01 |
| timburke | who's here for the swift meeting? | 21:01 |
| zaitcev | o7 | 21:02 |
| kota | o/ | 21:02 |
| zaitcev | But I'll have to bug out at half the hour. | 21:02 |
| timburke | 👍 | 21:02 |
| acoles | o/ | 21:03 |
| timburke | i expect mattoliver's still sleeping; he's been staying up late for a (virtual) conference here in the states | 21:03 |
| timburke | clayg may or may not be around ;-) | 21:03 |
| timburke | as usual, the agenda's at https://wiki.openstack.org/wiki/Meetings/Swift | 21:03 |
| timburke | #topic swift release | 21:04 |
| timburke | just a heads-up that i want to get a 2.28.0 release out -- there's a lot of great stuff that's landed! | 21:04 |
| timburke | current draft of the notes is at https://review.opendev.org/c/openstack/swift/+/799121 | 21:04 |
| timburke | if anyone has a chance to proofread that, or make sure i'm not missing or under-explaining something, i'd appreciate you guys looking at it | 21:05 |
| zaitcev | Dang, we didn't complete that compatibility of watcher with sharding that you made. | 21:05 |
| zaitcev | Sorry | 21:06 |
| timburke | still time! i feel like i'm notoriously slow to put releases together :P | 21:06 |
| zaitcev | ok | 21:06 |
| timburke | and on the topic of releases... | 21:06 |
| timburke | #topic next openstack release name | 21:06 |
| timburke | #link http://lists.openstack.org/pipermail/openstack-discuss/2021-July/023512.html | 21:06 |
| timburke | tl;dr: after Xena will be Yoga | 21:07 |
| timburke | and with a new cycle coming up, there's | 21:08 |
| timburke | #topic PTG | 21:08 |
| timburke | i think i'd mentioned the dates briefly a couple weeks back, but now i've got a doodle poll up to pick meeting slots, and an etherpad to gather topics | 21:09 |
| timburke | #link https://doodle.com/poll/a5fx9zy7s3bu72da | 21:09 |
| timburke | #link https://etherpad.opendev.org/p/swift-ptg-yoga | 21:09 |
| timburke | #link https://openinfra-ptg.eventbrite.com/ | 21:09 |
| zaitcev | Is that availability poll? | 21:09 |
| timburke | zaitcev, yeah, that's the idea. similar to the past virtual PTGs | 21:10 |
| timburke | and like with previous PTGs, the idea is to schedule <=16hrs of dedicated meeting time across the week | 21:12 |
| timburke | any questions about the PTG prep? iirc, they're hoping to have teams signed up for slots next week | 21:13 |
| opendevreview | Merged openstack/swift master: func tests: Better-tolerate deleted containers in account listings https://review.opendev.org/c/openstack/swift/+/799314 | 21:14 |
| timburke | all right, moving on | 21:15 |
| timburke | one last announcement: | 21:15 |
| timburke | #topic OpenInfra Live event tomorrow | 21:16 |
| timburke | #link http://lists.openstack.org/pipermail/openstack-discuss/2021-July/023599.html | 21:16 |
| timburke | titled "How OpenStack Large Clouds Manage their Spare Capacity" | 21:16 |
| timburke | and you can watch it live at | 21:16 |
| timburke | #link https://www.youtube.com/watch?v=G7oN2XdI__k | 21:17 |
| timburke | seemed fairly interesting | 21:17 |
| timburke | all right, a couple more swift-related things: | 21:17 |
| timburke | #topic part-power increases and the reconciler | 21:18 |
| zaitcev | Why does everything have to be a video these days, or even worse, a livestream. I could just read a blog post or whitepaper. | 21:18 |
| timburke | i suspect literal "increased visibility" | 21:19 |
| timburke | so we saw some issues with concurrent writes from the reconciler interacting badly with the relinking going on during a part-power increase | 21:19 |
| timburke | #link https://bugs.launchpad.net/swift/+bug/1934314 | 21:19 |
| acoles | oh yeah, that! :( | 21:20 |
| timburke | and mattoliver put together a patch to have the reconciler skip moving data either into or out of policies that have a ppi in progress | 21:20 |
| zaitcev | Oh, brother. I was wondering just why you guys are focusing on reconciler of all things. | 21:20 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/799561 | 21:21 |
| timburke | it does what it says on the tin, and i definitely think it's the right move for now, but i wanted to point out another line of work that we've put off for a while | 21:22 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/103779 | 21:22 |
| timburke | ^^^ makes the reconciler behave much more like the expirer, taking on `process` and `processes` config opts to divy up work (among some other things) | 21:23 |
| clayg | 👍 | 21:23 |
| timburke | and i want us to think a bit about whether we still need the "defer reconciler work during a ppi" patch if we improve the reconciler's scale-out story | 21:24 |
| clayg | Maybe not if we encode the process number in the offset 🤔 | 21:25 |
| acoles | timburke: it's merged though? | 21:26 |
| timburke | it seems like we could get it so reconcilers on different nodes never (or at any rate, rarely) try to move the same data | 21:26 |
| timburke | it is. and again, i think it's the right move for now | 21:26 |
| acoles | OIC but you're wondering if we could drop it in future | 21:27 |
| timburke | yup | 21:27 |
| timburke | the patch as it stands makes it less likely to run into errors during this rare ops event at the cost of deferring reconciler work for what's likely *days*, which kinda sucks | 21:28 |
| timburke | i'm not looking for any definitive answers today, just wanted to seed the idea as we work on these | 21:29 |
| timburke | #topic relinker --link-check-limit option | 21:29 |
| zaitcev | gtg | 21:29 |
| timburke | o/ | 21:29 |
| timburke | as i was putting the release together, i noticed that this new option was introduced since 2.27.0 | 21:30 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/783467 | 21:30 |
| timburke | and deprecated before we did another release | 21:30 |
| timburke | #link https://review.opendev.org/c/openstack/swift/+/798849 | 21:30 |
| timburke | so i wanted to get opinions on how safe we feel it is to just remove the thing all together | 21:31 |
| opendevreview | Merged openstack/swift master: Add concurrency to reconciler https://review.opendev.org/c/openstack/swift/+/800802 | 21:31 |
| acoles | yeah it'd be weird to have its first release be the one that says its going to be removed in a future release :) | 21:33 |
| timburke | at any rate, i have no intention of calling it out in the release notes ;-) | 21:34 |
| acoles | and IIRC "removing" now juts means not allowing the option on the relinker command line | 21:34 |
| clayg | aww man I was going to do some refactoring on that reconciler concurrency patch | 21:34 |
| acoles | it's ignored as far as behaviour is concerned | 21:35 |
| timburke | yup | 21:35 |
| acoles | clayg: you can refactor on master ;) | 21:35 |
| clayg | RIP IT OUT!!! | 21:35 |
| acoles | timburke: do you want me to put a patch to remove it? or did you already write it? | 21:35 |
| timburke | i haven't already written it, but it'll be quick to write. i'll do that while you go to bed ;-) | 21:36 |
| timburke | all right then | 21:36 |
| timburke | #topic open discussion | 21:36 |
| timburke | what else should we bring up this week? | 21:36 |
| acoles | are there any high priority reviews to merge *before* the release? | 21:38 |
| acoles | looks like https://wiki.openstack.org/wiki/Swift/PriorityReviews could use some love | 21:39 |
| timburke | yeah, sorry :-( | 21:40 |
| timburke | i don't update that nearly often enough | 21:40 |
| acoles | I just culled a few items | 21:41 |
| acoles | I guess as zaitcev mentioned, the other dark data watcher improvements | 21:41 |
| acoles | would be good to have | 21:42 |
| timburke | fwiw we've only got a handful of high-priority bugs: https://bugs.launchpad.net/swift/+bugs?field.importance=High | 21:43 |
| timburke | and a couple of those seem like they should be at "Fix Released" | 21:44 |
| acoles | sorry, I need to drop off o/ | 21:49 |
| timburke | oh, and there's a critical bug (though it's also not in the right state): https://bugs.launchpad.net/swift/+bug/1926648 | 21:50 |
| timburke | that's fine, seems like we're about done anyway | 21:50 |
| timburke | thank you all for coming, and thank you for working on swift! | 21:50 |
| timburke | #endmeeting | 21:50 |
| opendevmeet | Meeting ended Wed Jul 14 21:50:35 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:50 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/swift/2021/swift.2021-07-14-21.01.html | 21:50 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/swift/2021/swift.2021-07-14-21.01.txt | 21:50 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/swift/2021/swift.2021-07-14-21.01.log.html | 21:50 |
| opendevreview | Clay Gerrard proposed openstack/swift master: reconciler: concurreny follow-up https://review.opendev.org/c/openstack/swift/+/800842 | 21:54 |
| opendevreview | Tim Burke proposed openstack/swift master: Remove support for --link-check-limit https://review.opendev.org/c/openstack/swift/+/800844 | 21:58 |
| opendevreview | Tim Burke proposed openstack/swift master: Add some more arm64 non-voting tests https://review.opendev.org/c/openstack/swift/+/793280 | 22:17 |
| opendevreview | Clay Gerrard proposed openstack/swift master: container-reconciler: support multiple processes https://review.opendev.org/c/openstack/swift/+/800850 | 22:35 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!