| openstackgerrit | Tim Burke proposed openstack/swift master: relinker: Rehash the parts actually touched when relinking https://review.opendev.org/c/openstack/swift/+/783731 | 00:11 |
|---|---|---|
| openstackgerrit | Tim Burke proposed openstack/swift master: relinker: Remove replication locks for empty parts https://review.opendev.org/c/openstack/swift/+/790305 | 00:23 |
| openstackgerrit | Merged openstack/swift master: Document why TestReconstructorRebuildUTF8 is skipped on py3 https://review.opendev.org/c/openstack/swift/+/790302 | 00:37 |
| *** dsariel has joined #openstack-swift | 00:41 | |
| *** dsariel has quit IRC | 00:53 | |
| *** gyee has quit IRC | 02:20 | |
| *** rcernin has quit IRC | 02:23 | |
| *** evrardjp has quit IRC | 02:33 | |
| *** evrardjp has joined #openstack-swift | 02:33 | |
| *** rcernin has joined #openstack-swift | 03:09 | |
| *** rcernin has quit IRC | 03:16 | |
| *** rcernin has joined #openstack-swift | 03:25 | |
| *** rcernin has quit IRC | 03:26 | |
| *** rcernin has joined #openstack-swift | 03:27 | |
| *** tkajinam has quit IRC | 03:34 | |
| *** tkajinam has joined #openstack-swift | 03:34 | |
| openstackgerrit | Matthew Oliver proposed openstack/swift master: ring: Keep track of last primary nodes from last rebalance https://review.opendev.org/c/openstack/swift/+/790550 | 05:37 |
| openstackgerrit | Matthew Oliver proposed openstack/swift master: WIP: Single frag iter hack https://review.opendev.org/c/openstack/swift/+/790374 | 05:37 |
| zaitcev | FileNotFoundError: [Errno 2] No such file or directory: 'swift-manage-shard-ranges' | 05:45 |
| zaitcev | But of course | 05:45 |
| zaitcev | [console_scripts] | 05:45 |
| zaitcev | swift-manage-shard-ranges = swift.cli.manage_shard_ranges:main | 05:45 |
| zaitcev | So just PATH=${PATH}:/q/zaitcev/hail/swift-probez/bin PYTHONPATH=/q/zaitcev/hail/swift-probez TMPDIR=/var/tmp ./.probetests no longer cuts it. | 05:46 |
| *** m75abrams has joined #openstack-swift | 06:05 | |
| *** godog has joined #openstack-swift | 07:20 | |
| *** fingo has quit IRC | 07:21 | |
| *** fingo has joined #openstack-swift | 07:22 | |
| *** rpittau|afk is now known as rpittau | 07:26 | |
| *** m75abrams has quit IRC | 08:22 | |
| *** m75abrams has joined #openstack-swift | 08:38 | |
| openstackgerrit | Alistair Coles proposed openstack/swift master: relinker: Rehash the parts actually touched when relinking https://review.opendev.org/c/openstack/swift/+/783731 | 09:03 |
| *** rpittau is now known as rpittau|bbl | 09:07 | |
| *** dsariel has joined #openstack-swift | 09:32 | |
| *** dsariel has quit IRC | 09:50 | |
| *** rpittau|bbl is now known as rpittau | 10:19 | |
| *** dsariel has joined #openstack-swift | 10:53 | |
| openstackgerrit | Merged openstack/swift master: relinker: Rehash the parts actually touched when relinking https://review.opendev.org/c/openstack/swift/+/783731 | 11:14 |
| *** rcernin has quit IRC | 11:23 | |
| *** rcernin has joined #openstack-swift | 11:24 | |
| *** rcernin has quit IRC | 11:35 | |
| openstackgerrit | Alistair Coles proposed openstack/swift master: swift-get-nodes: move --path-as-is to end of curl commands https://review.opendev.org/c/openstack/swift/+/790952 | 11:50 |
| *** timburke has quit IRC | 13:26 | |
| *** timburke has joined #openstack-swift | 13:27 | |
| *** ChanServ sets mode: +v timburke | 13:27 | |
| *** sorrison has quit IRC | 13:46 | |
| *** sorrison has joined #openstack-swift | 13:48 | |
| *** klamath_atx has joined #openstack-swift | 14:30 | |
| klamath_atx | Howdy Folks, running into an odd issue with very old pending asyncs on a cluster. We are trying to resolve over 100,000 asyncs and found that all of these asyncs are from 2018, the object-updater keeps updating the container-server with put/deletes however they are all returning 404's. Wondering if there is a reclaim setting for the updater to discard these old records or is removing the async_pending f | 14:31 |
| klamath_atx | olders the quick and dirty method to resolve the issue? | 14:31 |
| zaitcev | I am pretty sure you'll have to remove by hand. | 14:33 |
| zaitcev | Well, scripted obviously, but I mean rm(1). | 14:33 |
| zaitcev | Seems like the same issue that happens with containers too, except the container server does not crash continuously when being hit by updater with a bogus update. | 14:34 |
| klamath_atx | that is what i figured, thank you for the response | 14:35 |
| zaitcev | Romain's patch for container updater is not applicable to object updater, I don't think. Either way, it's not in the tree. | 14:35 |
| klamath_atx | jumped in to help on this old cluster yesterday due to performance issues, notice the pending asyncs, restart swift across the board and that unleashed the stream of pending asyncs hitting the container tier via the object-updater. stumbled on a landmine here | 14:39 |
| *** gyee has joined #openstack-swift | 15:26 | |
| -openstackstatus- NOTICE: Any builds with POST_FAILURE result and no available logs between 11:41 and 14:41 UTC today were related to an authentication endpoint problem in one of our providers and can be safely rechecked now | 15:33 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Optimize get_own_shard_range https://review.opendev.org/c/openstack/swift/+/791015 | 15:51 |
| clayg | ^ I don't know if that's a good idea or why i'm messing with it really... but something about p 790689 made me think we could go a different way (depending on what we're trying to accomplish I guess) | 15:52 |
| *** m75abrams has quit IRC | 16:09 | |
| *** rpittau is now known as rpittau|afk | 16:12 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: relinker: Remove replication locks for empty parts https://review.opendev.org/c/openstack/swift/+/790305 | 16:35 |
| openstackgerrit | Tim Burke proposed openstack/swift master: lock_path: Prevent aquiring deleted/moved locks https://review.opendev.org/c/openstack/swift/+/791022 | 16:35 |
| openstackgerrit | Merged openstack/swift master: swift-get-nodes: move --path-as-is to end of curl commands https://review.opendev.org/c/openstack/swift/+/790952 | 17:12 |
| timburke | klamath_atx, so wait, some of these asyncs are causing PUTs to the container server? might want to check whether there's still .data files out there for the objects... | 17:26 |
| klamath_atx | There are PUTs and Deletes going to container-server via object-updater @timburke | 17:41 |
| klamath_atx | we have around 130,000 asyncs, cluster has been running now with all services bounced for around 11 hours but it hasnt resolved a single async yet | 17:42 |
| klamath_atx | i see the same object update bubbling up every 10 mins in the logs, all 404's | 17:42 |
| *** fingo has quit IRC | 17:55 | |
| *** fingo has joined #openstack-swift | 17:57 | |
| clayg | so, *all* 130K async_pendings want to update a container that's been deleted? | 18:12 |
| clayg | klamath_atx: you may find it useful to get a better picture of what's in those async pendings: https://gist.github.com/clayg/754646ec7157b3cffae3ca2b53aebb04 | 18:12 |
| clayg | @klamath_atx also I think you may want to let it go a little faster: https://github.com/openstack/swift/blob/master/etc/object-server.conf-sample#L446 | 18:14 |
| openstackgerrit | Tim Burke proposed openstack/swift master: lock_path: Prevent aquiring deleted/moved locks https://review.opendev.org/c/openstack/swift/+/791022 | 18:23 |
| openstackgerrit | Tim Burke proposed openstack/swift master: relinker: Remove replication locks for empty parts https://review.opendev.org/c/openstack/swift/+/790305 | 18:23 |
| klamath_atx | Thank you guys for the help, the container in question was used as a zaqar cache back in 2018, something went down around 2018 that caused 140K worth of asyncs to show up. I checked the timestamps on these files and they all are from 2018, the project switched over to reddis to these async are just ddosing the container server | 18:35 |
| klamath_atx | we nuked the asyncs on all obj drives and everything seems to have calmed down, this cluster setup wasn't ideal and we are migrating off the last little bit of workload | 18:35 |
| *** fingo has quit IRC | 18:54 | |
| *** fingo has joined #openstack-swift | 18:55 | |
| *** hoonetorg has joined #openstack-swift | 19:07 | |
| timburke | 👍 | 19:15 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Optimize get_own_shard_range https://review.opendev.org/c/openstack/swift/+/791015 | 19:15 |
| *** dviroel is now known as dviroel|away | 19:48 | |
| *** Jeffrey4l has quit IRC | 19:49 | |
| *** Jeffrey4l has joined #openstack-swift | 20:00 | |
| kota_ | good morning | 20:57 |
| seongsoocho | kota_: good morning! | 20:58 |
| kota_ | seongsoocho: o/ | 20:58 |
| clayg | i'm about to run; not going to be long for the meeting today | 20:59 |
| tdasilva_ | /join #openstack-meeting | 21:11 |
| *** tdasilva_ is now known as tdasilva | 21:11 | |
| acoles | tdasilva: welcome! | 21:28 |
| tdasilva | acoles: o/ | 21:29 |
| *** dsariel has quit IRC | 21:33 | |
| *** dsariel has joined #openstack-swift | 21:33 | |
| *** openstackgerrit has quit IRC | 21:47 | |
| *** timburke has quit IRC | 22:08 | |
| *** timburke has joined #openstack-swift | 22:09 | |
| *** ChanServ sets mode: +v timburke | 22:09 | |
| *** dviroel|away has quit IRC | 23:01 | |
| *** rcernin has joined #openstack-swift | 23:07 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!