| *** gyee has quit IRC | 00:11 | |
| *** gyee has joined #openstack-swift | 00:25 | |
| *** ChanServ changes topic to "OpenStack Swift object storage | Logs: http://eavesdrop.openstack.org/irclogs/%23openstack-swift/ | Meetings: https://wiki.openstack.org/wiki/Meetings/Swift | Priority Reviews: https://wiki.openstack.org/wiki/Swift/PriorityReviews | Ideas: https://wiki.openstack.org/wiki/Swift/ideas" | 00:39 | |
| -openstackstatus- NOTICE: The Gerrit service at review.opendev.org is back up and running; for outage details see analysis here: http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html | 00:39 | |
| *** gyee has quit IRC | 01:12 | |
| *** mikecmpbll has quit IRC | 01:15 | |
| *** dsariel has quit IRC | 03:31 | |
| *** psachin has joined #openstack-swift | 03:32 | |
| *** dsariel has joined #openstack-swift | 03:39 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-swift | 04:33 | |
| *** m75abrams has joined #openstack-swift | 05:25 | |
| *** dsariel has quit IRC | 06:17 | |
| *** dsariel has joined #openstack-swift | 06:18 | |
| *** renich has quit IRC | 06:26 | |
| *** rpittau|afk is now known as rpittau | 06:42 | |
| *** hoonetorg has joined #openstack-swift | 06:51 | |
| *** openstackgerrit has quit IRC | 07:38 | |
| *** rcernin has quit IRC | 07:39 | |
| *** rcernin has joined #openstack-swift | 08:09 | |
| *** rcernin has quit IRC | 08:14 | |
| *** rcernin has joined #openstack-swift | 08:28 | |
| *** rcernin has quit IRC | 08:56 | |
| *** mikecmpbll has joined #openstack-swift | 09:46 | |
| *** aluria has quit IRC | 10:28 | |
| *** aluria has joined #openstack-swift | 10:30 | |
| *** aluria has quit IRC | 10:38 | |
| *** aluria has joined #openstack-swift | 10:39 | |
| *** mikecmpbll has quit IRC | 10:58 | |
| *** dsariel has quit IRC | 12:30 | |
| *** zamba has quit IRC | 13:03 | |
| *** zamba has joined #openstack-swift | 13:11 | |
| *** tdasilva has quit IRC | 13:15 | |
| *** tdasilva has joined #openstack-swift | 13:15 | |
| *** ChanServ sets mode: +v tdasilva | 13:15 | |
| *** psachin has quit IRC | 13:19 | |
| *** fingo has joined #openstack-swift | 14:19 | |
| *** mikecmpbll has joined #openstack-swift | 14:55 | |
| *** klamath_atx has joined #openstack-swift | 14:58 | |
| *** openstackgerrit has joined #openstack-swift | 15:24 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Be willing to send hashes as JSON https://review.opendev.org/758638 | 15:24 |
|---|---|---|
| openstackgerrit | Tim Burke proposed openstack/swift master: Compress REPLICATE responses https://review.opendev.org/758639 | 15:24 |
| openstackgerrit | Tim Burke proposed openstack/swift master: py3: Fix status comparison https://review.opendev.org/759075 | 15:25 |
| *** tdasilva_ has joined #openstack-swift | 15:27 | |
| *** ChanServ sets mode: +v tdasilva_ | 15:27 | |
| *** tdasilva has quit IRC | 15:30 | |
| *** m75abrams has quit IRC | 15:51 | |
| *** noonedeadpunk has quit IRC | 15:51 | |
| *** rpittau is now known as rpittau|afk | 16:04 | |
| *** noonedeadpunk_ has joined #openstack-swift | 16:12 | |
| timburke | huh. so in update_deleted, we only mark suffixes invalid if the sync was successful: https://github.com/openstack/swift/blob/2.26.0/swift/obj/replicator.py#L515-L524 | 16:23 |
| timburke | but in update, we do it all the time: https://github.com/openstack/swift/blob/2.26.0/swift/obj/replicator.py#L694-L701 | 16:24 |
| *** mikecmpbll has quit IRC | 16:24 | |
| timburke | i wonder *why*. i'm kind of tempted to push the rehash logic into the rsync function, in light of https://bugs.launchpad.net/swift/+bug/1818709 | 16:25 |
| openstack | Launchpad bug 1818709 in OpenStack Object Storage (swift) "object replicator update_deleted post ssync REPLICATE request considered harmful" [Undecided,New] | 16:25 |
| *** mikecmpbll has joined #openstack-swift | 16:26 | |
| *** noonedeadpunk_ has quit IRC | 16:33 | |
| *** noonedeadpunk has joined #openstack-swift | 16:39 | |
| *** mikecmpbll has quit IRC | 17:25 | |
| *** mikecmpbll has joined #openstack-swift | 17:26 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Add option to REPLICATE to just invalidate hashes https://review.opendev.org/758636 | 18:08 |
| *** gyee has joined #openstack-swift | 18:09 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: s3api: Tolerate absolute-form request targets https://review.opendev.org/759119 | 18:11 |
| ormandj | on deletes/bulk deletes, i would expect tombstone files are getting created and the purge is actually happening after the reclaim_age by the replicator and so forth. is there any reason for a bulk delete to be 'slow' as such? we're seeing lots of clients timeout at 120s+ on bulk delete operations (granted, 10k items) | 19:13 |
| timburke | ormandj, a bulk delete turns into a whole bunch of normal deletes on the backend -- there's some concurrency allowed, but the defaults are somewhat low, as it's a potential abuse vector | 20:15 |
| timburke | see https://github.com/openstack/swift/blob/master/etc/proxy-server.conf-sample#L983-L986 for the swift api, and https://github.com/openstack/swift/blob/master/etc/proxy-server.conf-sample#L531-L533 for s3api | 20:16 |
| clayg | @timburke I think on update_deleted we're reverting handoffs - when adding capacity we can get rejected by rsync for concurrency and then if we end up doing a REPLICATE call that's super dumb | 20:24 |
| timburke | clayg, can/should we skip it on failure for update? | 20:26 |
| clayg | primary syncing isn't an area i worry too much about optimizing | 20:26 |
| clayg | the biggest problem with the REPLICATE call is there's no concurrency control - lock N per disk, 1 per part with a timeout/5XX/stats would be ideal | 20:28 |
| clayg | in theory the hash calculation is lockless so we can support hash invalidation while hashing - but I think once we moved to the .invalid file we actually have a lot more options for how we shape and throttle the i/o we use there | 20:28 |
| mattoliverau | Not sure if we're meeting today, but i have to take Zoe in for a day care orientation early (for some reason, I guess so parents can work?) so might not really be around. Ptg next week, looking forward to it! | 20:33 |
| timburke | almost meeting time! | 20:55 |
| kota_ | morning | 20:59 |
| acoles | kota_: hello! | 20:59 |
| seongsoocho | morning o/ | 21:00 |
| kota_ | wow acoles! how's going? | 21:00 |
| acoles | good thanks | 21:00 |
| rledisez | welcome back acoles! :) | 21:00 |
| * kota_ reads the backlog. wow acoles is now at NVIDIA. | 21:01 | |
| acoles | rledisez: thanks | 21:01 |
| kota_ | welcome back! | 21:02 |
| acoles | kota_: thank you | 21:04 |
| ormandj | timburke: yeah, saw those variables, still didn't expect it to be quite _this_ slow, but we'll play with tuning those | 21:07 |
| mattoliverau | What! \o/ acoles! Best news ever! | 21:08 |
| acoles | hi mattoliverau | 21:08 |
| zaitcev | It's like that time when I went to work on Ceph. | 21:08 |
| zaitcev | In a year's time I quit and went back to Swift. | 21:08 |
| acoles | with swift you can go missing but eventually you'll get replicated :) | 21:10 |
| tdasilva_ | join #openstack-meeting | 21:11 |
| *** tdasilva_ is now known as tdasilva | 21:11 | |
| *** mikecmpbll has quit IRC | 21:29 | |
| mattoliverau | Lol, even the devs are eventually consistent | 21:30 |
| *** mikecmpbll has joined #openstack-swift | 21:31 | |
| timburke | swift cores: possibly degraded, but always durable | 21:31 |
| mattoliverau | Lol | 21:31 |
| acoles | but prone to inconsistency | 21:39 |
| timburke | just please tell me you didn't format your brain prior to coming back ;-) | 21:41 |
| tdasilva | maybe just quarantined some data | 21:44 |
| clayg | 😍 | 22:02 |
| acoles | love it ... "thank you all for coming, and thank you for working on swift!" ... some things haven't changed | 22:03 |
| zaitcev | But Chuck is now a VP at a grocery store chain. | 22:03 |
| tdasilva | some things haven't changed.... | 22:07 |
| timburke | it's a good ending to a meeting :-) | 22:15 |
| *** rcernin has joined #openstack-swift | 22:51 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!