| *** d34dh0r53 has quit IRC | 00:51 | |
| *** NM has joined #openstack-swift | 01:06 | |
| *** nanzha has joined #openstack-swift | 01:08 | |
| *** NM has quit IRC | 01:23 | |
| *** nanzha has quit IRC | 01:37 | |
| *** nanzha has joined #openstack-swift | 01:38 | |
| *** spsurya has joined #openstack-swift | 02:32 | |
| *** nanzha has quit IRC | 02:54 | |
| *** nanzha has joined #openstack-swift | 02:54 | |
| *** manuvakery has joined #openstack-swift | 05:02 | |
| *** hoonetorg has quit IRC | 05:37 | |
| *** hoonetorg has joined #openstack-swift | 05:48 | |
| *** ccamacho has joined #openstack-swift | 06:39 | |
| *** ianychoi has quit IRC | 06:51 | |
| *** ianychoi has joined #openstack-swift | 06:54 | |
| *** pcaruana has joined #openstack-swift | 07:16 | |
| *** tesseract has joined #openstack-swift | 07:22 | |
| *** rcernin has quit IRC | 07:32 | |
| *** rpittau|afk is now known as rpittau | 07:51 | |
| *** mikecmpbll has joined #openstack-swift | 07:52 | |
| *** e0ne has joined #openstack-swift | 08:00 | |
| *** hugokuo has quit IRC | 08:05 | |
| *** hugokuo has joined #openstack-swift | 08:05 | |
| *** nanzha has quit IRC | 08:23 | |
| *** nanzha has joined #openstack-swift | 08:25 | |
| *** nanzha has quit IRC | 08:53 | |
| *** nanzha has joined #openstack-swift | 09:06 | |
| *** openstackgerrit has quit IRC | 09:37 | |
| *** rcernin has joined #openstack-swift | 10:42 | |
| *** rcernin has quit IRC | 10:52 | |
| *** rcernin has joined #openstack-swift | 11:24 | |
| *** rcernin has quit IRC | 11:34 | |
| *** jistr is now known as jistr|mtgs | 11:47 | |
| *** nanzha has quit IRC | 11:48 | |
| *** nanzha has joined #openstack-swift | 11:54 | |
| *** openstackgerrit has joined #openstack-swift | 14:10 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: WIP: Allow internal clients to use reserved namespace https://review.opendev.org/682138 | 14:10 |
|---|---|---|
| clayg | binwiederhier: so in my understanding HTTP PUT doesn't support trailers - only GET | 14:18 |
| clayg | Have you seen differently? curl is pretty good at following the rules and rfcs - if you can get curl to send trailers with a PUT I'd happily re-evaluate my understanding | 14:18 |
| clayg | now HTTP/2 OTOH 😁 | 14:19 |
| *** ianychoi has quit IRC | 14:21 | |
| *** nanzha has quit IRC | 14:23 | |
| *** nanzha has joined #openstack-swift | 14:28 | |
| *** ianychoi has joined #openstack-swift | 14:29 | |
| timburke | i think i'm with clayg -- request needs to include a `TE: trailers` header to indicate that the client supports parsing those extra fields | 15:03 |
| timburke | that said, binwiederhier, you might be interested in our notion of "footers" -- there are some places in swift's code where we won't necessarily have all of the metadata for an object available when we first make the backend connections. this was originally done for EC, then generalized to replicated policies for encryption in https://github.com/openstack/swift/commit/03b762e | 15:07 |
| timburke | that probably has the best example to crib from if you wanted to try it out: https://github.com/openstack/swift/blob/2.23.0/swift/common/middleware/crypto/encrypter.py#L97-L181 | 15:08 |
| timburke | the way it's currently implemented involves multipart MIME documents, which isn't great -- we've wanted to replace it with something better/actually-HTTP-compliant for a while; see https://review.opendev.org/#/c/427911/ -- but i don't expect the middleware API for it to materially change | 15:11 |
| patchbot | patch 427911 - swift - Replace MIME with PUT+POST for EC and Encryption - 37 patch sets | 15:11 |
| *** diablo_rojo has joined #openstack-swift | 15:22 | |
| *** ianychoi has quit IRC | 15:43 | |
| *** ianychoi has joined #openstack-swift | 15:44 | |
| clayg | binwiederhier: Do you have a write up on what you're trying to do? Is this a client packing multiple objects into a single payload (like a many backup software's do)? | 15:45 |
| clayg | binwiederhier: ... or something on the server side of swift that's somehow buffering clint requests to deliver packed payloads to the object layer which can later be dereferenced for individual object name requests? | 15:45 |
| clayg | If it's the later ... I'm so confused ... so many questions. | 15:46 |
| *** e0ne has quit IRC | 15:58 | |
| *** tesseract has quit IRC | 16:03 | |
| *** jistr|mtgs is now known as jistr | 16:08 | |
| *** nanzha has quit IRC | 16:13 | |
| *** mikecmpbll has quit IRC | 16:35 | |
| *** spsurya has quit IRC | 16:40 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: WIP: Allow internal clients to use reserved namespace https://review.opendev.org/682138 | 16:53 |
| ormandj | there any easy way to "undeleted" an account that had the v1/{account} DELETE call made? | 17:20 |
| ormandj | undelete, that is | 17:20 |
| ormandj | POST only allows for updates/removal of custom metadata as far as i can tell | 17:20 |
| ormandj | and DELETE isn't even documented in the API docs: https://docs.openstack.org/api-ref/object-store/#accounts | 17:21 |
| ormandj | but: https://docs.openstack.org/swift/stein/admin/objectstorage-account-reaper.html | 17:21 |
| ormandj | and i've validated for the account you see: X-Account-Status: Deleted | 17:21 |
| openstackgerrit | Thiago da Silva proposed openstack/swift master: WIP: New Object Versioning mode https://review.opendev.org/682382 | 17:21 |
| ormandj | and get a 410 gone when you query the account url w/ a valid token | 17:21 |
| ormandj | we're trying to figure out the best way to reverse this short of hunting down the sqlite dbs and manually editing them | 17:22 |
| tdasilva | ormandj: maybe this is helpful? https://review.opendev.org/#/c/507808/ | 17:27 |
| patchbot | patch 507808 - swift - Add ability to undelete an account. - 24 patch sets | 17:27 |
| *** nanzha has joined #openstack-swift | 17:27 | |
| tdasilva | i've honestly not looked at that patch yet, but your question reminded me of it | 17:27 |
| *** nanzha has quit IRC | 17:33 | |
| *** ccamacho has quit IRC | 17:40 | |
| ormandj | well, we need it today, but the patch may have the answers i need | 17:41 |
| ormandj | let me look now, thank you | 17:41 |
| ormandj | looks like there's a runbook doc https://github.com/openstack/swift/blob/master/doc/source/ops_runbook/procedures.rst#procedure-revive-a-deleted-account | 17:47 |
| timburke | ormandj, if you haven't already, i'd strongly consider increasing delay_reaping (and possibly stopping the account-reapers all together) while you sort this out ;-) https://github.com/openstack/swift/blob/master/etc/account-server.conf-sample#L245-L252 | 17:51 |
| ormandj | timburke: we've already got that set to 14 days | 18:02 |
| ormandj | delay_reaping that is | 18:02 |
| ormandj | the problem is, it sounds like the re-enable process doesn't allow visibility into the old data | 18:02 |
| timburke | 👍 | 18:02 |
| ormandj | so i'm not sure how useful this reaping thing in general is, since even if it's prior to the delay reaping value, you can't gain access to the old data short of raw FS level on the storage nodes | 18:03 |
| ormandj | ie: there's no way to "oops my bad" from the customer give them access back to their data | 18:03 |
| ormandj | the documented process involves deleting the account dbs and recreating them, which orphans all the old data, with no way to recover it | 18:03 |
| timburke | hmm... the run-book sounds like it could cause dark data... | 18:04 |
| timburke | yeah, what you said ;-) | 18:04 |
| ormandj | yeah... and i'm also unclear on how we're supposed to ever provide the data for the customer within that window | 18:04 |
| ormandj | prior to the actual reaping | 18:05 |
| timburke | better would be to keep the account DB intact but fix it up so it's no longer marked deleted -- something much more like the patch tdasilva linked | 18:05 |
| ormandj | go sqlite munge the account dbs manually if possible, perhaps, figuring out how to reverse whatever the delete would be | 18:05 |
| ormandj | hah yeah, gmta | 18:05 |
| timburke | i *think* you should only need to do it on one DB -- replication should get it propagated to the others | 18:06 |
| timburke | the core of it is really this part: https://review.opendev.org/#/c/507808/24/swift/account/backend.py | 18:07 |
| patchbot | patch 507808 - swift - Add ability to undelete an account. - 24 patch sets | 18:07 |
| ormandj | hmm, sounds good. replication work on most recently updated? | 18:08 |
| ormandj | so i can pick any of the replicas, | 18:08 |
| timburke | yup -- replication uses some combination of that put_timestamp and status_changed_at timestamp to figure out which replica should "win" | 18:10 |
| ormandj | sweet, thanks tim | 18:14 |
| *** e0ne has joined #openstack-swift | 18:15 | |
| ormandj | now i need to figure out how to +1 the priority for our sanity on that undelete function haha | 18:16 |
| ormandj | anyone in/around the SAT area, free burgers and beer for helping to get that merged ;) | 18:17 |
| *** nanzha has joined #openstack-swift | 18:32 | |
| ormandj | alright, so went through the whole dance, it did replicate, but the account still shows as deleted - wondering if i have to bounce swift on storage nodes after that db file is updated? | 18:33 |
| ormandj | manual select show status is now "" in the account db | 18:35 |
| ormandj | nm - looks like it just took some time | 18:36 |
| *** nanzha has quit IRC | 18:37 | |
| *** e0ne has quit IRC | 18:38 | |
| clayg | ormandj: nice! | 18:40 |
| ormandj | yeah, didn't replicate like i expected | 18:41 |
| ormandj | the db files got updated, timestamp matches on all | 18:41 |
| ormandj | (filesystem level) | 18:41 |
| ormandj | but the 02/03 nodes still have DELETED in the account_stat | 18:41 |
| clayg | oic, so it's... not working 😞 | 18:42 |
| ormandj | sorta. not really sure how the replication works, might be an interval i need to wait | 18:42 |
| ormandj | (for account dbs) | 18:42 |
| ormandj | i just sqlite3'd one of the db files on one of the storage nodes | 18:43 |
| clayg | I think most latest swifts would allow swift-account-replicator to be called in the forground with a -d and -p flags to "aim" it at the db you want to hit | 18:43 |
| ormandj | we're stein, so i can try | 18:43 |
| ormandj | we haven't quite yet gone to train, but we'll do so once we have time to validate things | 18:44 |
| clayg | oh if you're just looking at the fields in the status table that may not really mean it's still "deleted" - like tim said - really it comes down to the timestamps | 18:44 |
| ormandj | well, it 404s when i head the account and gives deleted status | 18:44 |
| ormandj | and looking at the backend.py code | 18:44 |
| ormandj | that field is what it uses to check for deleted status | 18:44 |
| ormandj | if i HEAD the account on the server i did the sqlite3 munge, it works fine | 18:44 |
| *** e0ne has joined #openstack-swift | 18:45 | |
| ormandj | the other two that should replicate from it, 404/deleted status in the HEAD results | 18:45 |
| ormandj | comparing the two sqlite3 dbs, one the one i updated and the one i did not, put_timestamp is the same | 18:47 |
| ormandj | the only difference is status = "" on the one i munged myself, and it's DELETED on the other | 18:47 |
| ormandj | so it replicated... but not the status | 18:47 |
| ormandj | clayg: did another test, re-updated with new timestamps, it replicated the timestamps across almost immediately, but not status | 19:01 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: WIP: Allow internal clients to use reserved namespace https://review.opendev.org/682138 | 19:01 |
| clayg | ok, so maybe that's one of the issues addressed in the patch - to repliate the status field properly | 19:02 |
| clayg | so it absolutely really would be useful to get it merged | 19:03 |
| clayg | do you have a swift-all-in-one environment - or would you be willing to try something like vsaio to pull down the purposed change, exercise it functionally, and provide operational feedback? | 19:03 |
| clayg | it would go a long way to moving it closer to merged! | 19:03 |
| clayg | maybe we should do TX based hack-a-thon or meetup to focus on this or some other patches we want to get merged ? | 19:04 |
| timburke | needs to get fixed first ;-) the patch *does not* seem to do anything about replicating status.... | 19:08 |
| timburke | looking at it a bit, there are a few other issues, too -- i'll try to get another patchset up today, but there may be more work required than what i can get to right now | 19:09 |
| timburke | "maybe we should do TX based hack-a-thon" as long as there's a dinner at Salt Lick, i'm in :D | 19:10 |
| ormandj | we do have a development environment we can test with, if needed | 19:14 |
| ormandj | but we need to fix this for someone today, so i'm going to go the manual route | 19:14 |
| ormandj | i'll go sqlite update all three locations | 19:14 |
| ormandj | looks like i just need to get status in sync and then we're ok | 19:15 |
| *** gyee has joined #openstack-swift | 19:29 | |
| ormandj | yeah, manually updating all status fields in the three account dbs seemed to work | 19:37 |
| *** diablo_rojo has quit IRC | 20:00 | |
| clayg | ormandj: awesome! sounds like you have a better idea of how to build swift functionality that works than that patch authors - i'd love to capture your findings in a a review of that patch | 20:06 |
| clayg | they've been pretty responsive to re-spin - if you could help them make it work that would get 90% of the way to merged! | 20:07 |
| *** pcaruana has quit IRC | 20:16 | |
| timburke | hmm... https://github.com/openstack/swift/blob/2.23.0/swift/account/backend.py#L269-L277 vs https://github.com/openstack/swift/blob/2.23.0/swift/container/backend.py#L781-L792 | 20:16 |
| *** diablo_rojo has joined #openstack-swift | 20:16 | |
| timburke | i kinda feel like status ought to be an emergent behavior based on the relative values of put_timestamp and delete_timestamp ... | 20:16 |
| clayg | flake8's W503 and W504 seem a bit ... at odds 😡 | 20:19 |
| ormandj | clayg: i can sanitize my crib-notes on what i had to do to make this work for you if that's what you're looking for? | 20:22 |
| ormandj | (manually, in my case) | 20:22 |
| clayg | that would be amazingly helpful! | 20:22 |
| clayg | the "extra mile" would be referencing them as a comment on that review in gerrit - if you were feeling like... being a superhero today | 20:23 |
| clayg | but no pressure! | 20:23 |
| ormandj | i'm 100% a super villain every day, but i'm happy to help my selfish desire for this to work on my path to world domination with a comment. :) just might not be today... have other worlds to conquer first | 20:24 |
| openstackgerrit | Tim Burke proposed openstack/swift master: WIP: Allow internal clients to use reserved namespace https://review.opendev.org/682138 | 20:25 |
| clayg | timburke: so technically at this point that should probably have been get_reserved_name('abc', 'def') | 20:26 |
| timburke | i feel strongly that `'%s' % b'x'` should produce a TypeError, not "b'x'" 😡 | 20:27 |
| clayg | tdasilva: FWIW the rebase is looking super easy! the way you developed that context class makes it *so* easy to plugin the refactor to use the reserved namespace | 20:27 |
| timburke | if i *wanted* a repr(), i would'a used a %r! | 20:27 |
| clayg | @timburke srly, come ON python3 like '%r' % b'x' - SURE - but str? no... just no | 20:28 |
| clayg | hahha, exactly 👍 | 20:28 |
| timburke | at *least* gimme a warning that i can treat as an error -- new python complains 'SyntaxWarning: "is" with a literal. Did you mean "=="?' when you type `1 is 1` -- this should be easy | 20:30 |
| timburke | huh. we never use that status column in the container, do we? | 20:36 |
| timburke | if anybody's got time to spot-check me on https://review.opendev.org/#/c/689567/ i'd much appreciate it. i'm pretty sure it's the sort of thing that ought to get backported (the container can become unusable!) but the fix and its follow-ups get a little hairy -- i maybe shoulda pushed to have them all land as a unit :-/ | 20:41 |
| patchbot | patch 689567 - swift (stable/stein) - Sharding: Clean up old CleaveConext's during audit - 1 patch set | 20:41 |
| timburke | then i hope to push up a 2.21.1 changelog patch shortly, and start some backports for rocky... | 20:43 |
| ormandj | clayg: to be clear, i was doing everything manually - i had not pulled down just that patchset and tested. hopefully that part was not missed | 20:43 |
| ormandj | i was using sqlite3 to do the db edits by hand | 20:43 |
| timburke | (ultimate goal is to get a 2.17.2 for queens before it goes extended-maintenance and we can't tag any more) | 20:44 |
| *** e0ne has quit IRC | 21:13 | |
| openstackgerrit | Tim Burke proposed openstack/swift stable/rocky: Verify client input for v4 signatures https://review.opendev.org/689883 | 21:59 |
| *** e0ne has joined #openstack-swift | 22:01 | |
| openstackgerrit | Tim Burke proposed openstack/swift stable/rocky: s3api: Block ETag header from CompleteMultipartUpload requests https://review.opendev.org/689885 | 22:06 |
| *** openstackgerrit has quit IRC | 22:07 | |
| *** openstackgerrit has joined #openstack-swift | 22:09 | |
| openstackgerrit | Tim Burke proposed openstack/swift stable/rocky: s3api: Increase max body size for Delete Multiple Objects requests https://review.opendev.org/689887 | 22:09 |
| openstackgerrit | Tim Burke proposed openstack/swift stable/rocky: Unquote URL before using splited parts. https://review.opendev.org/689888 | 22:11 |
| *** e0ne has quit IRC | 22:16 | |
| *** nanzha has joined #openstack-swift | 22:18 | |
| *** e0ne has joined #openstack-swift | 22:18 | |
| openstackgerrit | Tim Burke proposed openstack/swift stable/rocky: fix documentation of default https://review.opendev.org/689889 | 22:21 |
| mattoliverau | morning | 22:22 |
| *** nanzha has quit IRC | 22:23 | |
| *** ianychoi has quit IRC | 23:00 | |
| *** ianychoi has joined #openstack-swift | 23:00 | |
| *** rcernin has joined #openstack-swift | 23:19 | |
| *** rcernin has quit IRC | 23:19 | |
| *** rcernin has joined #openstack-swift | 23:20 | |
| *** ianychoi has quit IRC | 23:24 | |
| *** ianychoi has joined #openstack-swift | 23:26 | |
| *** e0ne has quit IRC | 23:58 | |
| *** e0ne has joined #openstack-swift | 23:59 | |
| *** e0ne has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!