| *** ormandj has quit IRC | 00:18 | |
| *** diablo_rojo has joined #openstack-swift | 00:22 | |
| *** ianychoi has joined #openstack-swift | 00:34 | |
| *** ormandj has joined #openstack-swift | 00:41 | |
| *** diablo_rojo has quit IRC | 01:18 | |
| *** e0ne has joined #openstack-swift | 01:48 | |
| *** e0ne has quit IRC | 01:52 | |
| *** psachin has joined #openstack-swift | 03:21 | |
| *** gyee has quit IRC | 03:26 | |
| *** psachin has quit IRC | 03:36 | |
| openstackgerrit | Tim Burke proposed openstack/python-swiftclient master: Delete/overwrite symlinks better https://review.opendev.org/673123 | 03:44 |
|---|---|---|
| *** gkadam has joined #openstack-swift | 03:49 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: s3api: paginate listings when aborting MPUs https://review.opendev.org/652184 | 03:55 |
| openstackgerrit | Tim Burke proposed openstack/swift master: slo_manifest_hook follow-up https://review.opendev.org/629659 | 04:04 |
| openstackgerrit | Tim Burke proposed openstack/python-swiftclient stable/stein: Fix up stable gate https://review.opendev.org/674180 | 04:14 |
| openstackgerrit | Merged openstack/swift master: symlink: Allow symlinks to be created via chunk-encoded PUTs https://review.opendev.org/673121 | 04:17 |
| *** viks___ has joined #openstack-swift | 04:26 | |
| *** tkajinam has quit IRC | 05:00 | |
| *** tkajinam has joined #openstack-swift | 05:02 | |
| openstackgerrit | Merged openstack/python-swiftclient master: Drag forward prettytable in lower-constraints https://review.opendev.org/672830 | 05:48 |
| openstackgerrit | Merged openstack/swift master: Negative test for non-empty chunked put symlink https://review.opendev.org/674052 | 05:48 |
| *** tkajinam_ has joined #openstack-swift | 05:53 | |
| *** tkajinam has quit IRC | 05:55 | |
| *** e0ne has joined #openstack-swift | 06:04 | |
| *** e0ne has quit IRC | 06:42 | |
| *** e0ne has joined #openstack-swift | 06:45 | |
| *** rcernin has quit IRC | 06:58 | |
| *** redrobot has quit IRC | 07:17 | |
| openstackgerrit | Merged openstack/python-swiftclient master: Delete/overwrite symlinks better https://review.opendev.org/673123 | 07:17 |
| *** tesseract has joined #openstack-swift | 07:17 | |
| *** e0ne has quit IRC | 07:31 | |
| openstackgerrit | Merged openstack/python-swiftclient master: Fix up requests so we can send non-RFC-compliant headers on py3 https://review.opendev.org/672808 | 07:55 |
| *** mikecmpbll has joined #openstack-swift | 08:01 | |
| *** threestrands has quit IRC | 08:25 | |
| *** tkajinam_ has quit IRC | 09:01 | |
| *** e0ne has joined #openstack-swift | 09:03 | |
| *** tdasilva has quit IRC | 10:33 | |
| *** ccamacho has joined #openstack-swift | 11:18 | |
| *** tdasilva has joined #openstack-swift | 11:19 | |
| *** ChanServ sets mode: +v tdasilva | 11:19 | |
| *** takamatsu is now known as mauro|call | 11:44 | |
| *** redrobot has joined #openstack-swift | 12:07 | |
| *** gkadam has quit IRC | 12:26 | |
| *** sasregulus has joined #openstack-swift | 12:44 | |
| openstackgerrit | Merged openstack/swift master: sharding: better handle get_shard_ranges failures https://review.opendev.org/655784 | 13:17 |
| openstackgerrit | Merged openstack/swift master: Ignore 404s from handoffs for objects when calculating quorum https://review.opendev.org/672186 | 13:17 |
| *** ccamacho has quit IRC | 13:18 | |
| *** BjoernT has joined #openstack-swift | 13:22 | |
| *** BjoernT_ has joined #openstack-swift | 13:25 | |
| *** BjoernT has quit IRC | 13:27 | |
| sasregulus | My understanding is that when an object is deleted from the ring the row for the object is left in the container database forever. If this is true, is there a way to clean out these old rows eventually? I'm asking because we are running out of space on the SSDs where our container databases live. | 13:43 |
| DHE | I don't think that's correct, but the database is based on sqlite which does have its own space compaction rules independent of swift. | 13:47 |
| sasregulus | Okay, I hope that is true. What we see when we look at the sqlite db though looks like a "tombstone" row is put into the database when an object is deleted from the ring but we haven't seen the tombstone row ever be actually deleted from the sqlite db. | 13:49 |
| tdasilva | sasregulus: IIRC tombstone should be cleaned out `reclaim_age`, no? | 13:52 |
| tdasilva | cleaned out *after* `reclaim_age` | 13:56 |
| tdasilva | rledisez: what version of grpcio are you running with losf? | 14:08 |
| sasregulus | tdasilva, we thought that after the deleted column was set to 1 the row would eventually be removed but we are not seeing that yet. Our reclaim_age is the default which I think is 18 hours. We'll keep a closer eye on it but we are not seeing those rows remove. | 14:09 |
| tdasilva | sasregulus: default reclaim_age is 7 days | 14:10 |
| DHE | reclaim_age needs to be long enough that any host down for that long would most likely be nuked and rebuilt | 14:11 |
| sasregulus | tdasilva: Oh, I did my math wrong, we haven't been watching for 7 days yet. That probably explains it.Thank you! | 14:11 |
| rledisez | tdasilva: grpcio==1.3.3 | 14:17 |
| tdasilva | sasregulus: yw :) | 14:21 |
| tdasilva | rledisez: ok, thanks | 14:21 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Make GreenAsyncPile not hang https://review.opendev.org/623360 | 14:22 |
| openstackgerrit | Alex Schultz proposed openstack/python-swiftclient master: Cleanup session on delete https://review.opendev.org/674320 | 14:22 |
| *** gkadam has joined #openstack-swift | 14:32 | |
| *** gkadam has quit IRC | 14:38 | |
| *** sasregulus has quit IRC | 15:15 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Make GreenAsyncPile not hang https://review.opendev.org/623360 | 15:20 |
| *** tdasilva has quit IRC | 15:49 | |
| *** tdasilva has joined #openstack-swift | 15:49 | |
| *** ChanServ sets mode: +v tdasilva | 15:49 | |
| *** tdasilva has quit IRC | 15:54 | |
| *** mikecmpbll has quit IRC | 16:04 | |
| *** gyee has joined #openstack-swift | 16:06 | |
| *** tdasilva has joined #openstack-swift | 16:07 | |
| *** ChanServ sets mode: +v tdasilva | 16:07 | |
| *** pcaruana has quit IRC | 16:14 | |
| *** altlogbot_3 has quit IRC | 16:29 | |
| *** irclogbot_1 has quit IRC | 16:33 | |
| *** altlogbot_1 has joined #openstack-swift | 16:37 | |
| *** irclogbot_2 has joined #openstack-swift | 16:42 | |
| *** rickflare has joined #openstack-swift | 16:51 | |
| *** tesseract has quit IRC | 16:58 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Allow "harder" symlinks https://review.opendev.org/633094 | 16:59 |
| timburke | crap! i forgot to fix the quote thing, too 😞 | 17:14 |
| timburke | but clayg, i think ^^^ is mutually agreeable? | 17:14 |
| *** openstackgerrit has quit IRC | 17:22 | |
| *** gyee has quit IRC | 17:24 | |
| clayg | yeah, looks solid | 17:28 |
| timburke | except for that part where it doesn't ;-) | 17:31 |
| timburke | i'll fix that up, try to remember to address the quote thing at the same time | 17:32 |
| timburke | i still kinda wonder if hardlink-to-softlink should *drop* symlink_bytes as an indication that the target's a symlink rather than some generic zero-byte object... *shrug* | 17:35 |
| *** gyee has joined #openstack-swift | 17:36 | |
| timburke | and i might still prefer to translate 404 -> 409 during PUT -- i *think* that'd be more clear that validation failed instead of container not existing? | 17:38 |
| clayg | i think it'sll be very rare for a client to write a hardlink to symlink | 17:39 |
| timburke | separate issue. i want to create a hardlink to an object that doesn't actually exist yet -- i get a 404. does that mean the container i'm PUTting to doesn't exist? or my hardlink target? | 17:40 |
| clayg | i recognize there's some ambiguity on the 404 between target missing or container missing... | 17:43 |
| clayg | but it's not obvoius to me thats worse than the ambiguity of 409 on target missing and etag mis-match 🤔 | 17:44 |
| clayg | maybe if you get a 409 the first thing you're going to do is HEAD the object and check the etag - maybe the 404 response clues you in! | 17:44 |
| timburke | i wonder if we oughta use swift_bytes for normal objects, too... that or have s3api swap out symlink_bytes (if present) for bytes (if zero) in listings... it feels weird that via S3, links show zero-bytes in listings for normal objects, but expected bytes for SLOs/MPUs | 17:45 |
| timburke | oh, and possibly symlink_etag... | 17:47 |
| clayg | swift_bytes is a mess - does it get lost on POST? | 17:47 |
| clayg | if you update ctype | 17:47 |
| clayg | i'm not sure I have an opinion on how symlinks should iteract with s3api 😬 seems like oil & water | 17:48 |
| timburke | sometimes? 🤮 | 17:48 |
| timburke | see my comment on patch set 2 of https://review.opendev.org/#/c/334719/ | 17:53 |
| patchbot | patch 334719 - swift - Preserve X-Static-Large-Object from .data file aft... (MERGED) - 2 patch sets | 17:53 |
| clayg | i don't like the idea of differentiating a hardlink to symlink from a hardlink to a zero-byte-object by dropping the symlink_bytes key, seems to subtle | 17:57 |
| timburke | fair enough -- it's definitely subtle... | 17:59 |
| *** diablo_rojo has joined #openstack-swift | 18:06 | |
| *** diablo_rojo has quit IRC | 18:07 | |
| *** diablo_rojo has joined #openstack-swift | 18:07 | |
| clayg | Yeah it’s weird that hard-links to SLO show bytes in the listing as the size of the target SLO. But that’s SLOs fault. | 18:10 |
| *** e0ne has quit IRC | 18:11 | |
| timburke | and i feel bad about propagating swift_bytes 🙁 | 18:16 |
| timburke | so where are we landing on 404 vs 409? got tests fixed up (i think), working on the quoting issue (i think the resolution is just, don't strip?) | 18:19 |
| *** diablo_rojo has quit IRC | 18:25 | |
| *** diablo_rojo has joined #openstack-swift | 18:26 | |
| clayg | yes, just don't strip 🤣 | 18:28 |
| clayg | I guess 409 on missing or mismatch is ok by me! you seem pretty sure it's an improvement to make 404 on PUT always missing container? | 18:28 |
| timburke | *i* think so -- like, that's an established expectation; getting back a 409 when putting a hadrlink, that's all new behavior, we can kinda do what we like there | 18:31 |
| clayg | i'll buy that argument! | 18:33 |
| clayg | can we distinguish the case in the body? 🤔 | 18:33 |
| *** guimaluf has joined #openstack-swift | 18:41 | |
| timburke | yeah, easy enough | 18:45 |
| tdasilva | not to beat a dead horse, but just want to make sure i understand, for slo the target_etag will be the manifest etag, but the tgt_size is the actual slo size? | 18:58 |
| timburke | good question! no, not quite. have a look around https://review.opendev.org/#/c/633094/19/test/functional/test_symlink.py@1459 -- symlink_etag/bytes will match the manifest, but *specifically for links to SLOs*, the listing bytes will match the large object size (instead of being zero) | 19:03 |
| patchbot | patch 633094 - swift - Allow "harder" symlinks - 19 patch sets | 19:03 |
| timburke | at least, as of the most recent patch set | 19:04 |
| *** openstackgerrit has joined #openstack-swift | 19:05 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Allow "harder" symlinks https://review.opendev.org/633094 | 19:05 |
| timburke | so i went ahead and stripped the etag that we use all over, in part because we already wrote a test about it (kinda; confirming 409 behavior, but nothing getting to a 201) | 19:06 |
| tdasilva | timburke: thanks! | 19:07 |
| clayg | ugh, yeah the SLO bytes overwrite thing is really weird... | 19:07 |
| clayg | it'd be too weird for symlink_etag to be the manifest and symlink_bytes to be the swift_bytes/slo_size - RIGHT!? | 19:08 |
| clayg | like looking one patch down the line to versioned_writes we want the listing of a versioned container's hardlinks to have the "bytes" keys reflect the target objects (which will in somecases be SLOs) | 19:09 |
| clayg | what's weird tho is we end up forwarding swift_bytes (like hardlink now does automatically) and then *not* replacing the bytes key with symlink_bytes when we're handling a versioned writes hardlink to a SLO 😞 | 19:10 |
| clayg | if we *always* put the target "bytes" in symlink_bytes we could remove the special case in versioned writes maybe? | 19:10 |
| *** e0ne has joined #openstack-swift | 19:13 | |
| timburke | i agree -- it's weird. makes me wish we'd always had a slo_bytes plumbed through the listing etag :-( | 19:16 |
| clayg | right! | 19:16 |
| timburke | having the *container-server* doing the listing manipulation is just too strange | 19:16 |
| clayg | too strange | 19:16 |
| clayg | 😥 | 19:16 |
| timburke | i've thought a bit before about using sam's audit watchers to try to do data migrations... nothing ever quite seems to rise to the level of warranting that lift, but this swift_bytes business almost feels tempting... | 19:20 |
| timburke | burrito time | 19:23 |
| *** tdasilva has quit IRC | 19:23 | |
| *** irclogbot_2 has quit IRC | 19:39 | |
| *** irclogbot_1 has joined #openstack-swift | 19:44 | |
| *** e0ne has quit IRC | 20:11 | |
| *** diablo_rojo has quit IRC | 20:12 | |
| *** baojg has quit IRC | 20:31 | |
| timburke | rledisez, fyi, corvus is having a bit of trouble with configuring CORS on OVH over in -infra... | 20:40 |
| timburke | i figure you might know a thing or two about that ;-) | 20:40 |
| corvus | rledisez: o/ | 20:41 |
| corvus | rledisez: i'm having trouble with cors in ovh; i've set "X-Container-Meta-Access-Control-Allow-Origin: *" on the container and verified that is returned on a HEAD. because this script also is designed to work with rax, i'm also sending "Access-Control-Allow-Origin: *" when uploading each object. but the objects don't have the allow-origin header when i fetch them. do you happen to know of any | 20:41 |
| corvus | reason that might be? | 20:41 |
| corvus | (i've also repeated the experiment without setting any access control headers on the object uploads) | 20:42 |
| rledisez | hi corvus | 20:43 |
| rledisez | it should work as expected. maybe we can continue this discussion in private | 20:44 |
| corvus | rledisez: sure thing | 20:44 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Allow "harder" symlinks https://review.opendev.org/633094 | 20:46 |
| *** BjoernT_ has quit IRC | 21:09 | |
| openstackgerrit | Merged openstack/swift master: Make GreenAsyncPile not hang https://review.opendev.org/623360 | 21:16 |
| clayg | timburke: yeah using swift_bytes would be more consistent with SLO - but there's two problems with that | 21:20 |
| clayg | 1) it uses content-type instead of etag, which is just *wrong* like not... in some kind of "purity" argument (although bytes *do* "go with" the etag) - by eventual consistency with ctype updates I think makes it proably incorrect | 21:21 |
| clayg | 2) we don't like swift_bytes - it's loosing information *in the container-server* - why purpetuate a lossy pattern that's already causing problems | 21:22 |
| clayg | and yes, I was saying symlink-bytes should "go with" symlink-etag; so either I was wrong, or I was right and my experiment to change hardlinks-to-slo to use symlink_bytes for the slo-size will prove to be a bad idea | 21:23 |
| clayg | alternatively all software is terrible and everything is shit and no matter what we do it will be bad | 21:24 |
| timburke | i'm not sure we're losing information here though -- the presence of symlink_etag tells us that the underlying plain-old-swift object is zero bytes -- so we can use the field for something halfway useful | 21:24 |
| timburke | that last one sounds right :-) | 21:24 |
| timburke | it *would* mean that you need to have a whole lot more context loaded up to understand that you haven't lost information, though... | 21:25 |
| clayg | that sounds like an argument to use symlink_bytes for the slo-size | 21:25 |
| clayg | swift_bytes is lossy in that you can never see the size of the manifest in the container listing | 21:25 |
| clayg | but maybe that's unfairly conflating multiple problems with how SLOs handled forwarding slo-size to the container listing | 21:25 |
| timburke | and that's already true. as the patch stands now, we're in this odd situation of the hardlink knowing more in its listing than the slo, but that's not hardlink's fault | 21:26 |
| clayg | yeah, if we had some clear path to a very explicit non-lossy pattern for propogating information to listings I'd be easier to get behind something that seems messy in the intereum | 21:27 |
| clayg | timburke: I agree the current situation is strange, I think that's part of the reason I'm exploring alternatives | 21:28 |
| timburke | 👍 | 21:28 |
| clayg | since swift_bytes is new it doesn't seem so outrageous that it might mean "content-length of the symlink_path when you do a HEAD" | 21:28 |
| clayg | gah s/swift_bytes/symlink_bytes/ 🤦 | 21:28 |
| timburke | at some point, we'll either find something we like, or at least find something we can live with :P | 21:29 |
| clayg | backtracking on my previous justification: "if symlink_etag is the etag you get with ?manifest=get then symlink_bytes should be the content-length you get with ?manifest=get" 🤷♂️ | 21:29 |
| clayg | it also means symlink_etag is consistent between *LO but symlink_bytes ... not so much for DLO 😞 | 21:30 |
| clayg | timburke: is suicide an option? | 21:31 |
| timburke | but then, dlo's already got bad/weird listings. can't really avoid *that* | 21:32 |
| timburke | (though having it expose manifest_prefix or something in the listing would've been a step in the right direction...) | 21:34 |
| clayg | eventaully we'll have a sys-meta like framework to forward arbitrary extra k/v pairs to listings | 21:36 |
| *** jistr has quit IRC | 22:02 | |
| *** jistr has joined #openstack-swift | 22:09 | |
| *** jistr has quit IRC | 22:17 | |
| *** jistr has joined #openstack-swift | 22:19 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Allow "harder" symlinks https://review.opendev.org/633094 | 22:31 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: symlink-backed versioned_writes https://review.opendev.org/633857 | 22:31 |
| clayg | i'll see what zuul thinks of that over the weekend | 22:32 |
| clayg | and how it looks in the morning | 22:32 |
| clayg | g'night | 22:32 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!