| *** timburke_ has joined #openstack-swift | 00:54 | |
| *** ChanServ sets mode: +v timburke_ | 00:54 | |
| *** timburke_ has quit IRC | 00:58 | |
| *** timburke_ has joined #openstack-swift | 01:13 | |
| *** ChanServ sets mode: +v timburke_ | 01:13 | |
| *** nanzha has joined #openstack-swift | 01:27 | |
| *** nanzha has quit IRC | 02:36 | |
| *** nanzha has joined #openstack-swift | 02:37 | |
| *** timburke_ has quit IRC | 02:55 | |
| *** timburke_ has joined #openstack-swift | 02:55 | |
| *** ChanServ sets mode: +v timburke_ | 02:55 | |
| *** timburke_ has quit IRC | 03:10 | |
| *** timburke_ has joined #openstack-swift | 03:18 | |
| *** ChanServ sets mode: +v timburke_ | 03:18 | |
| *** tkajinam has joined #openstack-swift | 03:18 | |
| *** timburke_ has quit IRC | 03:19 | |
| *** psachin has joined #openstack-swift | 03:21 | |
| *** timburke_ has joined #openstack-swift | 03:22 | |
| *** ChanServ sets mode: +v timburke_ | 03:22 | |
| *** timburke_ has quit IRC | 03:27 | |
| *** timburke_ has joined #openstack-swift | 03:31 | |
| *** ChanServ sets mode: +v timburke_ | 03:31 | |
| *** timburke_ has quit IRC | 04:05 | |
| *** timburke_ has joined #openstack-swift | 04:09 | |
| *** ChanServ sets mode: +v timburke_ | 04:09 | |
| *** timburke_ has quit IRC | 04:14 | |
| *** tkajinam has quit IRC | 04:19 | |
| clayg | Whoa! IRC working. | 04:42 |
|---|---|---|
| *** timburke_ has joined #openstack-swift | 04:55 | |
| *** ChanServ sets mode: +v timburke_ | 04:55 | |
| *** psachin has quit IRC | 04:59 | |
| *** timburke_ has quit IRC | 05:00 | |
| *** timburke_ has joined #openstack-swift | 05:01 | |
| *** ChanServ sets mode: +v timburke_ | 05:01 | |
| *** timburke_ has quit IRC | 05:06 | |
| *** timburke_ has joined #openstack-swift | 05:09 | |
| *** ChanServ sets mode: +v timburke_ | 05:09 | |
| *** timburke_ has quit IRC | 05:10 | |
| *** timburke_ has joined #openstack-swift | 05:16 | |
| *** ChanServ sets mode: +v timburke_ | 05:16 | |
| *** timburke_ has quit IRC | 05:16 | |
| *** timburke_ has joined #openstack-swift | 05:16 | |
| *** ChanServ sets mode: +v timburke_ | 05:16 | |
| *** timburke_ has quit IRC | 05:39 | |
| *** timburke_ has joined #openstack-swift | 06:15 | |
| *** ChanServ sets mode: +v timburke_ | 06:15 | |
| *** timburke_ has quit IRC | 06:20 | |
| *** timburke_ has joined #openstack-swift | 06:20 | |
| *** ChanServ sets mode: +v timburke_ | 06:20 | |
| *** alexlecuyer-ch has joined #openstack-swift | 06:24 | |
| *** timburke_ has quit IRC | 06:31 | |
| *** timburke_ has joined #openstack-swift | 06:40 | |
| *** ChanServ sets mode: +v timburke_ | 06:40 | |
| *** timburke_ has quit IRC | 06:44 | |
| *** alexlecuyer-ch has quit IRC | 06:48 | |
| *** timburke_ has joined #openstack-swift | 06:53 | |
| *** ChanServ sets mode: +v timburke_ | 06:53 | |
| *** nanzha has quit IRC | 07:00 | |
| *** nanzha has joined #openstack-swift | 07:10 | |
| *** timburke_ has quit IRC | 07:30 | |
| *** timburke_ has joined #openstack-swift | 07:31 | |
| *** ChanServ sets mode: +v timburke_ | 07:31 | |
| *** timburke_ has quit IRC | 07:43 | |
| *** timburke_ has joined #openstack-swift | 07:43 | |
| *** ChanServ sets mode: +v timburke_ | 07:43 | |
| *** timburke_ has quit IRC | 07:51 | |
| *** timburke_ has joined #openstack-swift | 08:12 | |
| *** ChanServ sets mode: +v timburke_ | 08:12 | |
| *** alexlecuyer-ch has joined #openstack-swift | 08:15 | |
| *** alexlecuyer-ch has quit IRC | 08:20 | |
| *** alexlecuyer-ch has joined #openstack-swift | 08:20 | |
| *** tesseract has joined #openstack-swift | 08:24 | |
| *** pcaruana has joined #openstack-swift | 08:30 | |
| *** timburke_ has quit IRC | 08:46 | |
| *** timburke_ has joined #openstack-swift | 08:49 | |
| *** ChanServ sets mode: +v timburke_ | 08:49 | |
| *** alexlecuyer-ch has quit IRC | 08:52 | |
| *** alexlecuyer-ch has joined #openstack-swift | 08:59 | |
| *** alexlecuyer-ch has quit IRC | 08:59 | |
| *** alexlecuyer-ch has joined #openstack-swift | 08:59 | |
| *** alexlecuyer-ch has quit IRC | 09:08 | |
| *** alexlecuyer-ch has joined #openstack-swift | 09:19 | |
| *** alexlecuyer-ch has quit IRC | 09:21 | |
| *** timburke_ has quit IRC | 09:27 | |
| *** timburke_ has joined #openstack-swift | 09:33 | |
| *** ChanServ sets mode: +v timburke_ | 09:33 | |
| *** timburke_ has quit IRC | 09:37 | |
| *** timburke_ has joined #openstack-swift | 10:01 | |
| *** ChanServ sets mode: +v timburke_ | 10:01 | |
| *** timburke_ has quit IRC | 10:05 | |
| *** timburke_ has joined #openstack-swift | 10:17 | |
| *** ChanServ sets mode: +v timburke_ | 10:17 | |
| *** timburke_ has quit IRC | 10:18 | |
| *** rdejoux has joined #openstack-swift | 10:51 | |
| *** timburke_ has joined #openstack-swift | 11:26 | |
| *** ChanServ sets mode: +v timburke_ | 11:26 | |
| *** timburk__ has joined #openstack-swift | 11:27 | |
| *** ChanServ sets mode: +v timburk__ | 11:27 | |
| *** timburke_ has quit IRC | 11:30 | |
| *** timburk__ has quit IRC | 11:32 | |
| *** timburke_ has joined #openstack-swift | 11:36 | |
| *** ChanServ sets mode: +v timburke_ | 11:36 | |
| timburke_ | hrm. looks like the lower-constraints py2->py3 transition affects stable branches, too: https://review.opendev.org/#/c/690738/ | 11:43 |
| patchbot | patch 690738 - swift (stable/rocky) - Fix quarantine when object path is not a directory - 1 patch set | 11:43 |
| openstackgerrit | Tim Burke proposed openstack/swift stable/train: pin lower-constraints to run against python 2.7 https://review.opendev.org/692810 | 11:50 |
| openstackgerrit | Tim Burke proposed openstack/swift stable/stein: pin lower-constraints to run against python 2.7 https://review.opendev.org/692811 | 11:50 |
| timburke_ | aaaand merge conflict on rocky :-/ | 11:51 |
| tdasilva | that's weird that it was changed to run py3 on stable branches too | 11:52 |
| tdasilva | i can understand master, stable branches seems like a mistake | 11:52 |
| *** nanzha has quit IRC | 12:02 | |
| *** timburke_ has quit IRC | 12:05 | |
| *** nanzha has joined #openstack-swift | 12:10 | |
| *** timburke_ has joined #openstack-swift | 12:27 | |
| *** ChanServ sets mode: +v timburke_ | 12:27 | |
| *** timburke_ has quit IRC | 12:31 | |
| *** FlorianFa has joined #openstack-swift | 12:36 | |
| *** nanzha has quit IRC | 13:02 | |
| *** nanzha has joined #openstack-swift | 13:02 | |
| *** henriqueof has joined #openstack-swift | 13:26 | |
| openstackgerrit | Merged openstack/swift stable/train: pin lower-constraints to run against python 2.7 https://review.opendev.org/692810 | 14:10 |
| *** nanzha has quit IRC | 14:48 | |
| *** nanzha has joined #openstack-swift | 14:55 | |
| *** mugsie has quit IRC | 15:02 | |
| *** mugsie has joined #openstack-swift | 15:05 | |
| *** dosaboy has quit IRC | 15:10 | |
| *** dosaboy has joined #openstack-swift | 15:50 | |
| *** dosaboy has joined #openstack-swift | 15:50 | |
| *** dosaboy has quit IRC | 15:51 | |
| *** dosaboy has joined #openstack-swift | 15:52 | |
| *** dosaboy has quit IRC | 15:53 | |
| *** dosaboy has joined #openstack-swift | 15:55 | |
| *** gyee has joined #openstack-swift | 16:03 | |
| *** nanzha has quit IRC | 16:33 | |
| *** mvkr has quit IRC | 16:48 | |
| *** mwheckmann has joined #openstack-swift | 16:54 | |
| mwheckmann | hello. Does anyone know under what circumstances a container DB (i.e a container listing) will get out of sync and list objects that were deleted? (we confirmed the object deletions through the logs). This was with 2.13 BUT while we were in the middle of upgrading the storage nodes to 2.21. I'm not entirely sure that the upgrade had anything to do with it though. These were with segment | 17:14 |
| mwheckmann | objects from an application that uses the DLO Direct API (so basically just like regular objects). | 17:15 |
| *** gregwork has joined #openstack-swift | 17:38 | |
| *** pcaruana has quit IRC | 18:08 | |
| *** pcaruana has joined #openstack-swift | 18:08 | |
| *** tesseract has quit IRC | 18:53 | |
| *** mwheckmann has quit IRC | 19:33 | |
| *** mwheckmann has joined #openstack-swift | 19:34 | |
| *** mwheckmann has quit IRC | 19:35 | |
| *** alexlecuyer-ch has joined #openstack-swift | 19:47 | |
| *** alexlecuyer-ch has quit IRC | 20:28 | |
| *** mwheckmann has joined #openstack-swift | 21:10 | |
| *** pcaruana has quit IRC | 21:30 | |
| *** timburke_ has joined #openstack-swift | 21:42 | |
| *** ChanServ sets mode: +v timburke_ | 21:42 | |
| *** mvkr has joined #openstack-swift | 21:56 | |
| mattoliverau | mwheckmann: how out of sync? When an OP sits the object server, ie a PUT or DELETE. Then the object server will attempt to update the container server. If it fails, due to load, or timeout or whatever, it'll write it to async pending and the object-updater will pick it up and update the container servers async. So there can be a lag sometimes depending on what's happening in your cluster. | 22:23 |
| mattoliverau | *hits | 22:23 |
| mattoliverau | also morning | 22:24 |
| mattoliverau | how's the summit/ptg going? | 22:24 |
| timburke_ | rledisez, alecuyer: hmm... https://github.com/docker/distribution/pull/1229 | 22:25 |
| timburke_ | mattoliverau: it's going well! did the project update and ops feedback yesterday, met a new operator or two | 22:25 |
| mattoliverau | nice! | 22:25 |
| timburke_ | https://etherpad.openstack.org/p/PVG-swift-ops-feedback | 22:26 |
| mwheckmann | mattoliverau, that is exaclty my understanding as well. Out of sync in the sense that we have a handful of objects in a very large container DB that did not get marked as "deleted". | 22:26 |
| mwheckmann | Like I said, this started while we were upgrading from 2.13 -> 2.21. | 22:26 |
| mwheckmann | The objects themselves no longer exist in their respective partitions. Only the ts files are there. | 22:27 |
| timburke_ | mwheckmann: just one replica, or all replicas of the container DB? how's your async pending count and object-updater cycle time? | 22:27 |
| mwheckmann | All replicas of the container DB yes. | 22:27 |
| mwheckmann | timburke_: I'll have to check that. | 22:28 |
| mattoliverau | if a delete has made it to one replica of the container, it should propergate. As timburke suggests, the async pending might not have fired or had trouble firing. | 22:30 |
| mattoliverau | if only there was a way to shard up these large container db.. so they'd repond quickly and now slow down | 22:30 |
| mattoliverau | :P | 22:30 |
| mwheckmann | Could be because of not doing the clean "swift-init rest stop" during the upgrade? | 22:31 |
| mwheckmann | It's a relatively busy container. | 22:31 |
| mwheckmann | (the container DBs are on SSD BTW) | 22:32 |
| mattoliverau | the object-updaters will attempt to update the dbs. but if the container is busy may timeout or fail and try again next cycle. So checking the object updater cycle time and stats might give us a clue. | 22:33 |
| mattoliverau | or if your updaters aren't running | 22:34 |
| timburke_ | fwiw, something along the lines of `find /srv/node/*/async_pending/ -type f | wc -l` should get you the current async pending count for a node | 22:36 |
| mwheckmann | The object updaters are running. Our swift-recon stats are broken since upgrade (we also moved to running in containers). I'll try what timeburke_ suggests | 22:37 |
| mwheckmann | I have to go. But at first glance async_pending's are non-existant. Last week during the upgrade there were quite a few of them though. I'll have to dig deeper | 22:40 |
| mwheckmann | I'll checl later on. | 22:40 |
| *** mwheckmann has quit IRC | 22:41 | |
| mattoliverau | timburke: cool, some good feedback from the session. So my take after a brief glance. Teiring sounds like it will be useful feature or maybe at least policy migration. And maybe make sure expiring objects can target versions in the new versioning API :) | 22:41 |
| mattoliverau | consistant directory listings.. well they'll be more consistant in healthy clusters. If everyone used sharding (ie auto sharding was eventual on all the time, on day) then that'll hopefully help somewhat. | 22:43 |
| mattoliverau | maybe one day revisit partpower increase (2.0) where there is less op intervention and downtime to consistency engine. | 22:45 |
| timburke_ | poor guy had a part power of 10 for ~1000 disks :-( | 22:49 |
| mattoliverau | oh wow | 22:57 |
| *** mwheckmann has joined #openstack-swift | 23:09 | |
| *** timburke_ has quit IRC | 23:23 | |
| mwheckmann | timburke_: there is a lot of object expiry in this cluster and we had scale out the expirers during our upgrade, I wonder if that was putting more pressure on the container servers. | 23:45 |
| mwheckmann | async_pendings are non-existant for the storage policy that was affected. | 23:45 |
| mwheckmann | It looks like the expiry has caught and all is calm now. | 23:46 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!