| acoles | errr, am I going crazy or is zuul reporting success when unit tests are actually failing? dig into https://review.opendev.org/c/openstack/swift/+/824608 swift-tox-py36 job for example | 15:04 |
|---|---|---|
| acoles | I think it maybe because we re-try failed unit tests and the original failure/error was due to test coupling?? :/ | 15:17 |
| acoles | so running the failed test in isolation passes | 15:17 |
| opendevreview | Alistair Coles proposed openstack/swift master: Fix statsd prefix mutation in proxy controllers https://review.opendev.org/c/openstack/swift/+/824608 | 15:38 |
| opendevreview | Alistair Coles proposed openstack/swift master: Make NodeIter require a logger argument https://review.opendev.org/c/openstack/swift/+/825335 | 15:38 |
| opendevreview | Tim Burke proposed openstack/swift master: staticweb: Work with prefix-based tempurls https://review.opendev.org/c/openstack/swift/+/810754 | 17:18 |
| opendevreview | Tim Burke proposed openstack/swift master: s3api: Allow multiple storage domains https://review.opendev.org/c/openstack/swift/+/825399 | 20:54 |
| timburke | almost meeting time! | 20:58 |
| kota | good morning | 20:59 |
| seongsoocho | good morning! | 21:00 |
| timburke | #startmeeting swift | 21:00 |
| opendevmeet | Meeting started Wed Jan 19 21:00:34 2022 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| opendevmeet | The meeting name has been set to 'swift' | 21:00 |
| timburke | who's here for the swift meeting? | 21:00 |
| mattoliver | o/ | 21:00 |
| kota | o/ | 21:00 |
| seongsoocho | o/ hi | 21:00 |
| acoles | \o | 21:00 |
| timburke | as usual, the agenda's at https://wiki.openstack.org/wiki/Meetings/Swift | 21:01 |
| timburke | first up, in case you missed it on the ML | 21:02 |
| timburke | #topic z cycle naming | 21:02 |
| timburke | #link http://lists.openstack.org/pipermail/openstack-discuss/2022-January/026620.html | 21:02 |
| timburke | we can propose names on the wiki at | 21:02 |
| timburke | #link https://wiki.openstack.org/wiki/Release_Naming/Z_Proposals | 21:02 |
| mattoliver | Zorro! | 21:03 |
| mattoliver | 😜 | 21:03 |
| timburke | nominations close monday, then as i recall, the TC votes for the final name | 21:04 |
| timburke | on to swift-y topics! | 21:04 |
| timburke | #topic updater fairness | 21:04 |
| timburke | the first patch in the series (to add per-container ratelimits) landed -- https://review.opendev.org/c/openstack/swift/+/820962 | 21:05 |
| timburke | the second one (to add a queue of deferred updates to chew on when the disk sweep is shorter than the configured interval) is still in review -- https://review.opendev.org/c/openstack/swift/+/821736 | 21:06 |
| timburke | we've started running with it in prod, though, and are definitely seeing some processing from the deferred queue; it seems like a good and useful idea | 21:06 |
| mattoliver | Oh cool. | 21:07 |
| timburke | anyone want to commit to reviewing it for next week? i know it's on my radar | 21:07 |
| mattoliver | I was somewhat involved, but happy to review it again. | 21:08 |
| acoles | I had a thought that it might benefit from more commentary on the design of the deferral queue...let me know if you feel it does | 21:09 |
| acoles | (for the sake of future us) | 21:09 |
| timburke | thanks | 21:09 |
| timburke | #topic memcache periodic refresh | 21:09 |
| timburke | acoles spotted a bug in how i did the skip percentage on a previous patchset of https://review.opendev.org/c/openstack/swift/+/736802 | 21:10 |
| timburke | which gave me an opportunity to look at the stats i was emitting with fresh eyes -- the tl;dr is that this no longer requires any changes to the memcache client, and it's on callers to decide whether to skip or not before calling memcache | 21:12 |
| mattoliver | Oh, yeah I guess that makes sense. I thought thr doing in the client had the benefit of happening in one place. | 21:13 |
| timburke | which also means that the caller can spit out more-detailed stats; having memcache hit/miss/skip counts wasn't near as useful without knowing what type of key we were looking for | 21:13 |
| opendevreview | Ade Lee proposed openstack/swift master: Add FIPS CI jobs https://review.opendev.org/c/openstack/swift/+/796057 | 21:13 |
| timburke | there may be more revisions, but i think that patch is getting close | 21:14 |
| mattoliver | Cool, I'll take a look | 21:14 |
| timburke | next in the chain was removing pipeline_property -- https://review.opendev.org/c/openstack/swift/+/821920 -- there was a fair bit of back-and-forth, but i think it's settling down; it has one +2 on it now | 21:15 |
| acoles | @mattoliver for me, having the more-detailed stats won over having the skip pct applied in just one place | 21:15 |
| mattoliver | Kk | 21:17 |
| timburke | the get info calls have also been reworked to have more-detailed stats (as well as reduce test churn) | 21:17 |
| timburke | it still hasn't seen many eyes yet, but i'd prefer we review & land the first two | 21:18 |
| timburke | #topic tempurl signatures in logs | 21:19 |
| timburke | we've still got the open patch at https://review.opendev.org/c/openstack/swift/+/822585 | 21:19 |
| timburke | acoles, still think you might find time to take a look? | 21:20 |
| acoles | oh, sorry, yes I will try | 21:20 |
| timburke | thanks! | 21:20 |
| timburke | we'll close that bug one of these days ;-) | 21:21 |
| * acoles upgrades the review to bold on todo list ;) | 21:21 | |
| timburke | looks like clayg isn't around again, so i guess i'll skip the mpu-expiration overview again | 21:22 |
| timburke | #topic request tracing | 21:22 |
| zaitcev | What's an MPU? | 21:22 |
| timburke | multipart-upload -- how S3 does large (>5GB) objects | 21:23 |
| timburke | #undo | 21:23 |
| opendevmeet | Removing item from minutes: #topic request tracing | 21:23 |
| timburke | because of how s3api puts segments in this mostly-client-inaccessible namespace, we're feeling pretty comfortable with the idea that expiring an MPU manifest ought to automagically clean up the segments | 21:24 |
| timburke | clayg (and acoles) have been thinking about how to do that (as well as including a backstop daemon that goes around looking for orphaned segments and deleting them) | 21:25 |
| timburke | #topic request tracing | 21:26 |
| timburke | mattoliver gave his talk! | 21:26 |
| timburke | #link http://youtu.be/iP_qy8KKDno | 21:26 |
| mattoliver | \o/ talk complete | 21:26 |
| timburke | i haven't watched it yet, but plan to soon | 21:26 |
| kota | excellent | 21:26 |
| seongsoocho | wow good! thanks mattoliver | 21:27 |
| mattoliver | I've been off looking after kids and that conference, so will be back at work today playing catchup. | 21:27 |
| timburke | 👍 | 21:28 |
| timburke | that's all i've got | 21:28 |
| timburke | #topic open discussion | 21:28 |
| timburke | anything else we ought to bring up this week? | 21:28 |
| mattoliver | I got nothing.. I'm still in holiday mode :) | 21:28 |
| acoles | I've been working on a fix for some strange stasd naming mutation that we spotted https://review.opendev.org/c/openstack/swift/+/824608 | 21:30 |
| acoles | *statsd | 21:30 |
| acoles | it does result in some metric names emitted by the proxy changing, but only because they were unpredictable before | 21:31 |
| acoles | in summary, the proxy would prefix a metric name with [account|container|object] but the prefix mutated if a subrequest was issued, so the prefix was unpredictable. | 21:32 |
| mattoliver | Oh good find | 21:33 |
| acoles | I plan to write a bug report, now the patch is done ;-) | 21:33 |
| timburke | things like needing to fetch shard ranges to find out which shard an object update should go to would trip it, yeah? | 21:34 |
| mattoliver | Lol | 21:34 |
| acoles | we spotted this when studying the outputs of timburke's memcache skipping stats | 21:34 |
| timburke | do you remember if it would be that the container request would get stats that made it look like an object request, or vice-versa? | 21:34 |
| acoles | timburke: yes, or needing to do a container HEAD to populate container_info during an object request | 21:35 |
| acoles | I think object requests mutated to container, or account | 21:35 |
| acoles | sometime the prefix was lost altogether, which I haven't reproduced but could be when there is an error handling a (sub)request before the proxy controller is instantiated | 21:36 |
| timburke | 🤮 | 21:37 |
| timburke | i pushed up a new patchset for staticweb+tempurls: https://review.opendev.org/c/openstack/swift/+/810754 | 21:38 |
| acoles | the prefix would change such as: proxy-server -> proxy-server.object -> proxy-server -> proxy-server.container for example | 21:38 |
| timburke | review it this month and you'll get to browse through pictures from a trip i took back in college ;-) | 21:38 |
| mattoliver | Lol timburke | 21:39 |
| timburke | all right, i'm'a call it | 21:40 |
| timburke | thank you all for coming, and thank you for working on swift! | 21:41 |
| timburke | #endmeeting | 21:41 |
| opendevmeet | Meeting ended Wed Jan 19 21:41:08 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:41 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/swift/2022/swift.2022-01-19-21.00.html | 21:41 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/swift/2022/swift.2022-01-19-21.00.txt | 21:41 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/swift/2022/swift.2022-01-19-21.00.log.html | 21:41 |
| zaitcev | Okay. timburke, did you think about a watcher for S3 MPU segments | 21:59 |
| zaitcev | Sorry, I have too many irons on fire right now. | 21:59 |
| zaitcev | Or in fire. | 21:59 |
| timburke | i did! https://review.opendev.org/c/openstack/swift/+/783476 -- i think clayg's thinking about something at the container-level, though (so you can do things like high-water marks) | 22:03 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!