| *** rickyrem has quit IRC | 00:03 | |
| *** rickyrem has joined #openstack-swift | 00:07 | |
| *** ouchkernel has quit IRC | 00:12 | |
| *** vinsh has quit IRC | 00:16 | |
| *** vinsh has joined #openstack-swift | 00:17 | |
| *** ouchkernel has joined #openstack-swift | 00:19 | |
| *** furlongm has joined #openstack-swift | 00:48 | |
| *** vinsh has quit IRC | 00:53 | |
| kota_ | morning | 01:15 |
|---|---|---|
| mattoliverau | kota_: morning | 01:17 |
| kota_ | mattoliverau: o/ | 01:18 |
| zaitcev | I ended running hummingbird under strace to figure out what's wrong. | 01:47 |
| zaitcev | I succeeded, but man, that userland is _weird_. All the openat's for no reason... | 01:47 |
| *** amit213 has joined #openstack-swift | 02:39 | |
| *** Jeffrey4l_ has quit IRC | 03:23 | |
| *** Jeffrey4l_ has joined #openstack-swift | 03:36 | |
| *** SkyRocknRoll has joined #openstack-swift | 03:40 | |
| *** sk_ has joined #openstack-swift | 03:53 | |
| sk_ | is there any way to display applied config like swift show config... | 03:54 |
| *** klrmn has quit IRC | 03:55 | |
| *** Jeffrey4l_ has quit IRC | 04:00 | |
| *** kei_yama has quit IRC | 04:06 | |
| *** Jeffrey4l_ has joined #openstack-swift | 04:09 | |
| jrichli | timburke acoles: see latest comments on https://trello.com/c/6kiiS8KZ/47-consider-deriving-the-nonce-for-user-metadata | 04:14 |
| *** links has joined #openstack-swift | 04:15 | |
| jrichli | I am gonna be driving Monday morning and Tuesday afternoon, so my responsiveness will be spotty the next couple days :-) | 04:15 |
| *** Jeffrey4l_ has quit IRC | 04:21 | |
| *** psachin has joined #openstack-swift | 04:39 | |
| *** Jeffrey4l_ has joined #openstack-swift | 04:39 | |
| *** amit213 has quit IRC | 04:46 | |
| *** zaitcev has quit IRC | 04:57 | |
| *** ChubYann has quit IRC | 05:13 | |
| *** rickyrem1 has joined #openstack-swift | 05:15 | |
| *** rickyrem has quit IRC | 05:18 | |
| *** kei_yama has joined #openstack-swift | 05:36 | |
| *** mariusv has quit IRC | 05:36 | |
| *** ouchkernel has quit IRC | 05:44 | |
| *** mariusv has joined #openstack-swift | 05:50 | |
| *** rickyrem1 has quit IRC | 05:50 | |
| *** ouchkernel has joined #openstack-swift | 05:51 | |
| *** mariusv has quit IRC | 05:51 | |
| rfeusi | who | 05:57 |
| rfeusi | can help me in an architectic question? | 05:58 |
| *** rcernin has joined #openstack-swift | 06:27 | |
| *** janonymous has joined #openstack-swift | 06:32 | |
| *** pcaruana has joined #openstack-swift | 06:41 | |
| *** geaaru has joined #openstack-swift | 06:42 | |
| mahatic_ | acoles_: thanks for the doc. I think timburke already has a patch for your proposal. (I'm pulling in those changes to take a look) | 06:43 |
| *** sk_ has quit IRC | 06:53 | |
| *** d0ugal has joined #openstack-swift | 06:58 | |
| *** d0ugal has quit IRC | 06:59 | |
| *** tesseract- has joined #openstack-swift | 06:59 | |
| *** d0ugal has joined #openstack-swift | 07:00 | |
| *** jamielennox is now known as jamielennox|away | 07:01 | |
| *** sheel has joined #openstack-swift | 07:12 | |
| *** admin6 has joined #openstack-swift | 07:13 | |
| *** rledisez has joined #openstack-swift | 07:16 | |
| *** dmk0202 has joined #openstack-swift | 07:31 | |
| *** _JZ_ has joined #openstack-swift | 07:51 | |
| *** permalac has joined #openstack-swift | 07:56 | |
| openstackgerrit | Cheng Li proposed openstack/python-swiftclient: Add python version constraint python>=2.7 https://review.openstack.org/334333 | 08:11 |
| *** haypo has joined #openstack-swift | 08:14 | |
| *** joeljwright has joined #openstack-swift | 08:20 | |
| *** ChanServ sets mode: +v joeljwright | 08:20 | |
| *** janonymous has quit IRC | 08:34 | |
| *** acoles_ is now known as acoles | 09:11 | |
| *** asettle has joined #openstack-swift | 09:12 | |
| *** SkyRocknRoll has quit IRC | 09:13 | |
| *** sheel has quit IRC | 09:15 | |
| *** SkyRocknRoll has joined #openstack-swift | 09:15 | |
| *** jamielennox|away is now known as jamielennox | 09:26 | |
| *** janonymous has joined #openstack-swift | 09:39 | |
| *** asettle has quit IRC | 10:10 | |
| *** asettle has joined #openstack-swift | 10:11 | |
| *** NaveeNeo has joined #openstack-swift | 10:22 | |
| NaveeNeo | hello! | 10:22 |
| NaveeNeo | I want to work on Swift related project | 10:23 |
| NaveeNeo | what is the mechanism involved in writing data into the cloud using swift and nova | 10:24 |
| NaveeNeo | should i first learn things about swift or nova? | 10:24 |
| psachin | NaveeNeo, You can start with SAIO | 10:24 |
| psachin | http://docs.openstack.org/developer/swift/development_saio.html | 10:25 |
| NaveeNeo | tq psachin! :) | 10:25 |
| NaveeNeo | in order to know the data flow involved in openstack should i need to read source code?? | 10:26 |
| admin6 | Hi there, could someone please help me understand the object-reconstructor logs, especially concerning time remaining values. I used to have last week a full reconctruction done in about one hour with around 30 partitions per second treated, but the more data are injected into my erasure coded ring, the more time it takes, and the values are today : 70 hours for a full reconstruction with only 5 partitions per seconds for | 10:27 |
| admin6 | about 5TB of stored data | 10:27 |
| admin6 | Jun 27 06:33:08 STACO1 object-reconstructor: 818326/818326 (100.00%) partitions of 7/7 (100.00%) devices reconstructed in 159563.78s (5.13/sec, 0s remaining) | 10:27 |
| admin6 | Jun 27 06:38:39 STACO1 object-reconstructor: 955/116211 (0.82%) partitions of 1/7 (14.29%) devices reconstructed in 300.03s (3.18/sec, 70h remaining) | 10:27 |
| *** manous has joined #openstack-swift | 10:31 | |
| admin6 | Also what means the "partitions" values given ? Last week I had only 113992 partitions, and now I have 818326? However, my ring only contains 26244 partitions. | 10:32 |
| admin6 | 262144 partitions, 12.000000 replicas, 1 regions, 4 zones, 28 devices, 0.82 balance, 31.02 dispersion | 10:33 |
| admin6 | an extract of last week object-reconstructor log : | 10:34 |
| admin6 | Jun 18 06:26:51 STACO1 object-reconstructor: 102414/113992 (89.84%) partitions of 7/7 (100.00%) devices reconstructed in 3600.01s (28.45/sec, 6m remaining) | 10:34 |
| *** NaveeNeo has quit IRC | 10:39 | |
| *** NaveeNeo has joined #openstack-swift | 10:40 | |
| *** links has quit IRC | 10:43 | |
| *** silor has joined #openstack-swift | 10:44 | |
| NaveeNeo | ?? | 10:45 |
| *** daemontool has joined #openstack-swift | 10:50 | |
| *** natarej has quit IRC | 10:53 | |
| *** links has joined #openstack-swift | 10:55 | |
| *** NaveeNeo has quit IRC | 10:57 | |
| *** Jeffrey4l_ has quit IRC | 11:18 | |
| *** silor1 has joined #openstack-swift | 11:21 | |
| *** silor has quit IRC | 11:22 | |
| *** silor1 is now known as silor | 11:22 | |
| *** mariusv has joined #openstack-swift | 11:24 | |
| *** mariusv has quit IRC | 11:24 | |
| *** mariusv has joined #openstack-swift | 11:24 | |
| *** mariusv has quit IRC | 11:24 | |
| *** Jeffrey4l_ has joined #openstack-swift | 11:30 | |
| *** Jeffrey4l_ has quit IRC | 11:38 | |
| mahatic_ | acoles: hi, I was looking at derived iv on trello, I have a question - "for each v, use N+L as a nonce where L is greater than the maximum number of blocks allowed in a user metadata header value" | 11:44 |
| mahatic_ | acoles: why can't we simply choose L? (without appending N) as a random nonce and increase the counter for each value? | 11:47 |
| mahatic_ | I think because of iv needs to be 32 bytes and L may or may not adhere to that? | 11:49 |
| acoles | mahatic_: the idea as it was originally described to me was to guarantee that the iv used for each item of metadata was unique from any counter value used while encrypting any other item of metadata, and adding L provides that guarantee. | 11:49 |
| acoles | If N is 32bytes then N+L can be | 11:50 |
| acoles | mahatic_: however, it could be that we simply let the encrypter counter increment from a single starting value N, since we will always encrypt all the headers sequentially, we are never trying to encrypt an individual header independent of the others | 11:51 |
| acoles | I think timburke maybe raised that point too ^^ | 11:51 |
| acoles | So it gets simpler than I wrote on trello... | 11:52 |
| acoles | sort all the headers, then encrypt their values in sorted order using a single encryptor context with randomly chose IV. Do the inverse when decrypting. | 11:53 |
| * acoles afk back later | 11:53 | |
| mahatic_ | acoles: right. but I'm not sure if that doesn't conflict with a future case of independently updatable metdata | 11:53 |
| *** raildo-a` is now known as raildo | 11:56 | |
| *** Jeffrey4l_ has joined #openstack-swift | 11:58 | |
| *** asettle has quit IRC | 11:59 | |
| *** asettle has joined #openstack-swift | 12:03 | |
| *** daemontool has quit IRC | 12:05 | |
| *** daemontool has joined #openstack-swift | 12:05 | |
| *** manous has quit IRC | 12:08 | |
| mahatic_ | for future case, what you suggested on trello ought to do | 12:11 |
| openstackgerrit | Christian Schwede proposed openstack/swift: Add swiftbackmeup to associated projects https://review.openstack.org/334446 | 12:19 |
| *** manous has joined #openstack-swift | 12:21 | |
| openstackgerrit | Christian Schwede proposed openstack/swift: Add swiftbackmeup to associated projects https://review.openstack.org/334446 | 12:38 |
| *** kei_yama has quit IRC | 12:50 | |
| haypo | hello. as you may have noticed, i sent new patches for port swift to python 3: https://review.openstack.org/#/c/333297/ -- my new patches are simpler than my previous patch serie ;) | 12:57 |
| patchbot | haypo: patch 333297 - swift - Python 3: Fix basestring, long and StringIO | 12:57 |
| haypo | come to me if you need more information ;) | 12:58 |
| *** klamath has joined #openstack-swift | 13:01 | |
| *** klamath has quit IRC | 13:01 | |
| *** klamath has joined #openstack-swift | 13:02 | |
| *** wanghua has joined #openstack-swift | 13:05 | |
| tdasilva | good morning | 13:16 |
| *** psachin has quit IRC | 13:19 | |
| *** psachin has joined #openstack-swift | 13:21 | |
| *** SkyRocknRoll has quit IRC | 13:22 | |
| *** ManojK has joined #openstack-swift | 13:24 | |
| *** diogogmt has quit IRC | 13:29 | |
| *** vinsh has joined #openstack-swift | 13:33 | |
| *** baojg has joined #openstack-swift | 13:36 | |
| *** tongli has joined #openstack-swift | 13:47 | |
| *** _JZ_ has quit IRC | 13:48 | |
| tdasilva | haypo: thanks for the patches, just as a heads up, master branch is currently in a soft-freeze in order to get encryption middleware merged: http://lists.openstack.org/pipermail/openstack-dev/2016-June/097102.html | 13:50 |
| *** ManojK has quit IRC | 13:54 | |
| *** ManojK has joined #openstack-swift | 13:55 | |
| *** ametts has joined #openstack-swift | 14:00 | |
| haypo | tdasilva: ok | 14:05 |
| *** siva_krish has joined #openstack-swift | 14:07 | |
| *** siva_krish has quit IRC | 14:09 | |
| *** diogogmt has joined #openstack-swift | 14:12 | |
| *** cdelatte has joined #openstack-swift | 14:13 | |
| *** tmoreira has quit IRC | 14:27 | |
| *** tmoreira has joined #openstack-swift | 14:33 | |
| *** cdelatte has quit IRC | 14:34 | |
| *** psachin has quit IRC | 14:35 | |
| *** manous has quit IRC | 14:36 | |
| *** siva_krish has joined #openstack-swift | 14:40 | |
| *** amit213 has joined #openstack-swift | 14:43 | |
| *** tmoreira has quit IRC | 14:47 | |
| *** arch-nemesis has joined #openstack-swift | 14:48 | |
| *** manous has joined #openstack-swift | 14:48 | |
| acoles | timburke: thanks for the hmac patch! | 14:55 |
| *** Suyash has joined #openstack-swift | 14:55 | |
| *** tmoreira has joined #openstack-swift | 14:56 | |
| *** psachin has joined #openstack-swift | 14:58 | |
| *** psachin has quit IRC | 15:01 | |
| *** psachin has joined #openstack-swift | 15:01 | |
| *** jistr is now known as jistr|mtg | 15:01 | |
| *** mvk_ has quit IRC | 15:02 | |
| *** baojg has quit IRC | 15:04 | |
| *** pgbridge has joined #openstack-swift | 15:09 | |
| *** ManojK has quit IRC | 15:11 | |
| *** jistr|mtg is now known as jistr | 15:16 | |
| *** ManojK has joined #openstack-swift | 15:18 | |
| *** klrmn has joined #openstack-swift | 15:21 | |
| *** psachin has quit IRC | 15:29 | |
| *** dmk0202 has quit IRC | 15:30 | |
| *** cdelatte has joined #openstack-swift | 15:34 | |
| *** pcaruana has quit IRC | 15:38 | |
| *** tesseract- has quit IRC | 15:40 | |
| *** jmccarthy has quit IRC | 15:43 | |
| *** jmccarthy has joined #openstack-swift | 15:44 | |
| *** daemontool has quit IRC | 15:44 | |
| *** cdelatte has quit IRC | 15:45 | |
| *** cdelatte has joined #openstack-swift | 15:51 | |
| notmyname | good morning | 15:51 |
| *** cdelatte has quit IRC | 15:52 | |
| *** links has quit IRC | 15:52 | |
| *** dmorita has joined #openstack-swift | 15:54 | |
| *** dmorita has quit IRC | 15:58 | |
| notmyname | I have a new "intro to swift" slide, I think | 15:58 |
| notmyname | Swift is for Highly Resilient HyperScale HyperConvergence Web-Scale/Enterprise/Carrier-Grade Big Data Cloud (both Edge/IoT & Core/Data Center) Automation | 15:58 |
| notmyname | I mean, who wouldn't want that? | 15:58 |
| *** dmorita has joined #openstack-swift | 15:59 | |
| acoles | notmyname: good morning | 15:59 |
| notmyname | hi! | 16:00 |
| acoles | timburke: I' using your dashboard to hopefully stop me forgetting to publish comments! | 16:00 |
| notmyname | are there new patch sets to downloan? | 16:00 |
| notmyname | or even to download? | 16:00 |
| notmyname | seemingly this monday am my fingers arent' ready for typing | 16:00 |
| acoles | notmyname: real soon now, local testing just completing | 16:00 |
| notmyname | great! | 16:00 |
| acoles | notmyname: we need to discuss what action (if any) to take re constraints checks. see my reply to comments on patch 328207 | 16:01 |
| patchbot | acoles: https://review.openstack.org/#/c/328207/ - swift (feature/crypto-review) - Allow middleware to override metadata header checking | 16:01 |
| *** haypo has left #openstack-swift | 16:02 | |
| notmyname | hmmmmm | 16:03 |
| openstackgerrit | Alistair Coles proposed openstack/swift: Enable middleware to set metadata on object POST https://review.openstack.org/328206 | 16:03 |
| openstackgerrit | Alistair Coles proposed openstack/swift: Allow middleware to override metadata header checking https://review.openstack.org/328207 | 16:03 |
| openstackgerrit | Alistair Coles proposed openstack/swift: Enable object body and metadata encryption https://review.openstack.org/328208 | 16:03 |
| openstackgerrit | Alistair Coles proposed openstack/swift: Add encryption overview doc https://review.openstack.org/328209 | 16:03 |
| notmyname | on the one hand I totally get how "but middleware could do something bad" is not a very strong argument. it's about as good as "but the admin with root access can erase the computer" or "the storage system could delete my data". I mean, you've got to trust the code that's running and the people running it. so if there's some middleware that does Bad Things, then don't install it | 16:04 |
| notmyname | like I said, it's just something that seems like one of those places where some bug will creep in because of some unknown interaction between different modules. that's what made me nervous about it | 16:05 |
| *** tqtran has joined #openstack-swift | 16:05 | |
| acoles | notmyname: on the other hand... if it could happen, it will | 16:05 |
| notmyname | but I totally understand why it's needed and how it fits into the larger whole | 16:06 |
| notmyname | yeah | 16:06 |
| acoles | especially in a HyperScale HyperConvergence Web-Scale system ;) | 16:06 |
| notmyname | you should check out the rest of that statement ;-) https://twitter.com/notmyname/status/747456995242872833 | 16:07 |
| notmyname | what is patch 318441 and why does it keep getting rechecked? | 16:07 |
| patchbot | notmyname: https://review.openstack.org/#/c/318441/ - swift - [WIP] Testing latest u-c | 16:07 |
| notmyname | with now 37 patch sets | 16:07 |
| *** lyrrad has joined #openstack-swift | 16:08 | |
| notmyname | tdasilva: just land it ;-) https://review.openstack.org/#/c/334446/ | 16:08 |
| patchbot | notmyname: patch 334446 - swift - Add swiftbackmeup to associated projects | 16:08 |
| notmyname | tdasilva: cschwede: you're not in SF for the red hat summit, are you? | 16:10 |
| cschwede | notmyname: Hi! unfortunately not, i would have told you :) | 16:10 |
| notmyname | :-) | 16:11 |
| notmyname | turns out that portante is :-) | 16:11 |
| acoles | notmyname: I can't remember why we did not consider simply moving the user metadata to the other namespace that allowed longer values. maybe it didn't occur to us. | 16:12 |
| acoles | ah, perhaps because that ^^ doesn't work out with container/account metadata, where you could end up with a mix. | 16:14 |
| *** pauloewerton has joined #openstack-swift | 16:17 | |
| *** Suyash has quit IRC | 16:19 | |
| tdasilva | notmyname: I'm not either, but yeah, portante and luis will be there | 16:19 |
| *** rledisez has quit IRC | 16:23 | |
| *** klrmn has quit IRC | 16:25 | |
| *** dmorita has quit IRC | 16:27 | |
| *** dmorita has joined #openstack-swift | 16:28 | |
| notmyname | acoles: do the new patch sets include timburke's patch for the iv/hmac? | 16:33 |
| timburke | notmyname: yes, with modifications (at least, that's my understanding) | 16:34 |
| notmyname | ah, good | 16:34 |
| notmyname | just pulling it all down now | 16:34 |
| *** vinsh has quit IRC | 16:37 | |
| *** vinsh has joined #openstack-swift | 16:38 | |
| acoles | timburke: notmyname : yes, hmac for conditional gets is included. single iv for user metadata is not, yet - I like that approach but jrichli has concerns so I have left it for a future version (or not). | 16:43 |
| acoles | timburke: the mods are cosmetic, the algorithm is as etherpad and as you implemented. | 16:43 |
| *** asettle has quit IRC | 16:44 | |
| acoles | notmyname: http://www.xe.com/currencycharts/?from=GBP&to=USD&view=1W :( | 16:44 |
| notmyname | ouch | 16:45 |
| acoles | uhuh | 16:45 |
| timburke | acoles: yeah, i wasn't expecting the single-user-meta-crypto-meta unless i'm willing to code it. i'd be hesitant to do it as future work, though, as we'd still need to have code to handle existing objects with per-user-meta-crypto-meta | 16:46 |
| timburke | we'll see what today/tonight brings | 16:46 |
| *** amit213 has quit IRC | 16:46 | |
| *** _JZ_ has joined #openstack-swift | 16:46 | |
| *** manous has quit IRC | 16:46 | |
| notmyname | timburke: still have you're google shortlink for the crypto dash? | 16:50 |
| acoles | timburke: jrichli : i'll watch for opinions on https://trello.com/c/6kiiS8KZ/47-consider-deriving-the-nonce-for-user-metadata overnight | 16:50 |
| acoles | mahatic_: ^^ | 16:50 |
| *** gyee has joined #openstack-swift | 16:51 | |
| timburke | notmyname: https://goo.gl/f9gMj4 | 16:52 |
| notmyname | timburke: ah, found it https://goo.gl/f9gMj4 | 16:53 |
| notmyname | :-) | 16:53 |
| notmyname | thanks | 16:53 |
| notmyname | added it to http://not.mn/swift/swift_community_dashboard.html | 16:55 |
| clayg | acoles: timburke: is the derived iv thing fixed for etags? | 16:55 |
| acoles | clayg: yes | 16:56 |
| acoles | fixed == no more derived iv's, use an hmac | 16:56 |
| clayg | acoles: nice, i'll get back to work then | 16:56 |
| acoles | clayg: k, thanks | 16:56 |
| clayg | man, was anyone online when pasachin NaveeNeo and admin6 were asking questsions? | 16:58 |
| acoles | timburke: btw, I'm not sure we need to track the IV offset for each metadata header | 16:58 |
| timburke | acoles: why? | 16:58 |
| *** manous has joined #openstack-swift | 17:00 | |
| timburke | for the rest of the channel, i had an idea that (i think) would address jrichli's concern: similar to us tacking on ';swift_meta=...' for the container-update etag, we tack ';swift_crypto_offset=N' onto the end of the encrypted value | 17:00 |
| acoles | timburke: say we pick a random IV for all user metadata, then create an encryptor context. Then, sort all the headers and update the encrypter with their values in sorted order. The counter will tick up from the starting IV, we never re-use a counter value. And its all repeatable in the decryption path. | 17:01 |
| *** joeljwright has quit IRC | 17:01 | |
| timburke | acoles: right; we certainly don't need it given the way that things currently work. but if the concern is a future implementation of individually-updatable object metadata, this would let us discard items that were overwritten | 17:01 |
| acoles | timburke: how would we know what offset to choose for a new piece of individually updateable metadata - we wouldn't know what offsets were already in use with existing metadata, and we'd no longer be able to rely on all existing metadata being replaced. | 17:04 |
| *** joeljwright has joined #openstack-swift | 17:05 | |
| *** ChanServ sets mode: +v joeljwright | 17:05 | |
| *** ManojK has quit IRC | 17:05 | |
| acoles | timburke: we could tack individual IVs onto the end of the metadata, or indeed a crypto_meta dict, but there is a reason we didn't already do that. | 17:05 |
| *** joeljwright has quit IRC | 17:05 | |
| * acoles tries to remember why | 17:05 | |
| timburke | acoles: i was thinking each update would need to have its own encryption context and random iv. basically, each ts associated with metadata would be independent. not sure how it would be stored yet (since we don't want to overwrite the existing crypto meta), but if i knew that, i'd just code something up :-) | 17:07 |
| *** ukaynar has joined #openstack-swift | 17:10 | |
| acoles | timburke: OIC, so each item of metadata would be somehow marked with its ts (to map it back to the iv)? | 17:10 |
| timburke | i can't think of how else one would reconcile them | 17:11 |
| acoles | timburke: I'm wondering if we can ever make x-object-meta- individually updateable ... clients expect a POST to replace everything, can we change that semantic in the same namespace? IDK. | 17:14 |
| timburke | although i kind like the idea of just putting an iv on it instead, and making sure that each meta encryption starts at a fresh block | 17:14 |
| timburke | v2! v2! v2! | 17:14 |
| timburke | ;-) | 17:15 |
| acoles | hmmm, which takes me back to why have we not done that i.e. appended the crypto-meta | 17:15 |
| acoles | ^^ jrichli will have it noted somewhere :) | 17:16 |
| timburke | i suppose *all* of the crypto meta would make it considerably longer, while just an iv is fairly cheep...but i'm not sure how it's any worse than splitting it over two headers... | 17:16 |
| *** klrmn has joined #openstack-swift | 17:17 | |
| *** vinsh_ has joined #openstack-swift | 17:18 | |
| acoles | hehe, I already argued that we didn't need the cipher stored N times but got over-ruled on that | 17:18 |
| *** cdelatte has joined #openstack-swift | 17:19 | |
| *** vinsh has quit IRC | 17:21 | |
| *** Suyash has joined #openstack-swift | 17:23 | |
| timburke | acoles: well, if we ever need to change the cipher... | 17:29 |
| acoles | timburke: I'll be retired ;) | 17:29 |
| acoles | timburke: we'll only use one cipher with each POST, so we only need one record of the cipher somewhere in the metadata (imho) | 17:30 |
| acoles | unless its individually updateable, of course, but then we're back to your idea. | 17:31 |
| acoles | timburke: So, the concern with tacking crypto meta to the header was, IIRC, that when decrypting we must know that the header was in fact encrypted, otherwise we might just have user plaintext meta that happened to contain "...;iv=...". But, we could have a single piece of transient sysmeta that tells us that metadata (at ts in future) *is* encrypted, then tack iv onto end of each value. | 17:32 |
| acoles | jrichli: what have I forgotten? why did we not do this ^^ | 17:33 |
| *** dmorita has quit IRC | 17:35 | |
| timburke | acoles: the "swift_" param protection only applies to content-type currently, right? | 17:37 |
| acoles | timburke: yes. there is no restriction on user meta content (other than length) | 17:39 |
| timburke | there's our solution! unnecessary padding! :P | 17:40 |
| acoles | timburke: hehe, it was considered! | 17:40 |
| acoles | timburke: but I do think we can tack ;iv= onto the end so long as we have some piece of meta that tells us we've encrypted stuff, and in your future updateable meta world, then you need one piece of such meta for each ts. | 17:42 |
| timburke | acoles: yup. which asymptotically approaches the current two-headers-per-meta implementation... | 17:43 |
| acoles | timburke: updateable is hard! | 17:44 |
| timburke | maybe *that's* the really compelling reason. it would look like this anyway, and it will look like this for account/container once we add support there, so let's just have one way | 17:45 |
| acoles | those two headers are analogous to the (val, ts) tuple that goes into the container/account db | 17:45 |
| acoles | timburke: but why build in support for an unknown future implementation that will break existing meta anyway, or at least require a way to detect and differentiate old vs new? | 17:53 |
| * acoles being devil's advocate ^ | 17:54 | |
| acoles | timburke: plus there's other gotcha's with container and accounts :/ like what happens when you update previously encrypted x-account-meta-foo with an unencrypted value, you have stale x-account-sysmeta-crypto-meta-foo lying around, and to do the right thing we'd need to start shipping the per item meta timestamps back to the middleware. | 17:56 |
| timburke | you would still need to have the encrypter in the pipeline, though, right? why wouldn't it clear x-account-sysmeta-crypto-meta-foo when it sees that it shouldn't encrypt x-account-meta-foo? | 17:58 |
| acoles | but you may not have it in the pipeline, even if you are *supposed to* - the current object implementation will cope with that, as test_encrypter_decrypter.py should verify. | 18:00 |
| acoles | maybe we can reason that its ok or acceptable, I was just mentioning it. | 18:01 |
| acoles | timburke: BTW in case you aren't aware, this is best I came up with for updateable object sysmeta, and "best" does not necessarily imply "good", https://review.openstack.org/#/c/109314/ | 18:01 |
| patchbot | acoles: patch 109314 - swift-specs - Updateable Object Sysmeta (MERGED) | 18:01 |
| *** ManojK has joined #openstack-swift | 18:01 | |
| acoles | notmyname: I just noticed that this ^^ does not appear on the archive list here http://specs.openstack.org/openstack/swift-specs/ | 18:01 |
| acoles | notmyname: maybe it was so old it was in the wrong dir for the indexing?? | 18:02 |
| acoles | timburke: I have to leave. Sounds like we should leave things as they are?? pending further debate?? | 18:03 |
| timburke | acoles: i think i'm coming around on it, yeah | 18:03 |
| *** manous has quit IRC | 18:04 | |
| acoles | I did convince myself we could change strategy later without breaking what we are doing now. Obviously it means having legacy handling code around, but no unsolvable compatibiliy issue. | 18:04 |
| timburke | yeah, i'm not too worried there. although i think i'd still argue in favor of not changing it later | 18:06 |
| notmyname | acoles: that's unfortunate? | 18:08 |
| notmyname | ;-) | 18:08 |
| timburke | acoles: now i'm wondering whether we ought to require the encrypter. i don't like allowing operators to have a bad config like that. maybe only automatically add it if decrypter is present? | 18:09 |
| *** diogogmt has quit IRC | 18:09 | |
| *** diogogmt has joined #openstack-swift | 18:10 | |
| openstackgerrit | Merged openstack/swift: Add swiftbackmeup to associated projects https://review.openstack.org/334446 | 18:15 |
| *** manous has joined #openstack-swift | 18:16 | |
| *** siva_krish has quit IRC | 18:16 | |
| acoles | timburke: not sure how to enforce encrypter. will think more on that. | 18:21 |
| acoles | timburke: on meta, I'd still prefer to reduce the overhead of all the redundant crypto meta, if we can, so please keep pondering that. good night. | 18:23 |
| timburke | acoles: i'd be tempted to always auto-insert, and pass disable_encryption=True when auto-inserting | 18:23 |
| *** acoles is now known as acoles_ | 18:24 | |
| timburke | acoles: will do. getting it properly sorted may require that i go implement individually-updatable metadata first, though :P | 18:24 |
| *** rcernin has quit IRC | 18:34 | |
| *** mwheckmann has joined #openstack-swift | 18:36 | |
| notmyname | you could save some typing by renaming test_create_decryption_context_non_zero_offset() to test_magic() in test_crypto_utils.py | 18:44 |
| *** ChubYann has joined #openstack-swift | 18:51 | |
| *** rcernin has joined #openstack-swift | 19:03 | |
| *** amit213 has joined #openstack-swift | 19:04 | |
| *** arch-nemesis has quit IRC | 19:10 | |
| *** arch-nemesis has joined #openstack-swift | 19:11 | |
| *** adu has joined #openstack-swift | 19:12 | |
| *** ukaynar has quit IRC | 19:14 | |
| *** siva_krish has joined #openstack-swift | 19:22 | |
| tdasilva | acoles_: 3 goals in 18 minutes, my expectations were completely wrong | 19:29 |
| *** Jeffrey4l__ has joined #openstack-swift | 19:32 | |
| *** silor has quit IRC | 19:32 | |
| *** Jeffrey4l_ has quit IRC | 19:34 | |
| jrichli | timburke acoles: yes, timburke said exactly what i was about to say - "it will look like this for account/container once we add support there, so let's just have one way" | 19:40 |
| jrichli | I am pretty sure that is why we did it that way, because we had container and account meta already implemented | 19:40 |
| *** ManojK has quit IRC | 19:59 | |
| pdardeau | kudos to the author(s) of overview_encryption.rst -- very well written :-) | 20:00 |
| *** ManojK has joined #openstack-swift | 20:01 | |
| notmyname | pdardeau: +1 | 20:02 |
| *** tongli has quit IRC | 20:04 | |
| pdardeau | in looking at encryption stuff (just beginning), one of the things that jumped out at me is encryption_root_secret. i was kinda hoping it would not require the secret to be explicitly set in conf file. instead maybe allow for a URI-ish setting of something like "file:/path/to/file/containing/encryption/root/secret" | 20:16 |
| pdardeau | having that indirection would be nice since it wouldn't require additional secrecy of the conf file (which could be helpful when the conf files need to be shared, put in version control, etc.) | 20:17 |
| notmyname | pdardeau: yeah, that's a common topic | 20:18 |
| notmyname | pdardeau: we've chosen to do it this way initially for a few reasons | 20:18 |
| notmyname | first, there's other sensitive stuff in the config files anyway. maybe this is "more sensitive", but maybe not. point is, you still should have config files relatively protected anyway | 20:19 |
| notmyname | ...and the use case we're designing crypto for (initially) is summarized as the "you can RMA a drive" use case. not "prevent operators from reading your data" | 20:19 |
| notmyname | second, no matter where the key is stored or what level of indirection is used, it all comes down to get_crypto_key() and all the rest of the patch chain is really the hard part that has to be done anyway | 20:20 |
| notmyname | ...so we figured let's restrict the scope and focus on the baseline, especially since the baseline in the patch now is actually useful to users today | 20:20 |
| notmyname | and finally, it comes down to the keymaster middleware. I'm positive that there will be other keymaster implementations that do something different (including storing the master key in some crypto vault thing) | 20:21 |
| notmyname | so...baby steps. it's hard enough already :-) | 20:21 |
| pdardeau | notmyname: thx for explanation. makes sense, although i didn't think anyone RMA drives anymore (maybe they will after this!) | 20:23 |
| notmyname | pdardeau: yeah, now they might start. but probably more likely that the data could be at risk if a drive moves from a swift cluster to some other server in the data center without being wiped. now all they'd see is encrypted stuff | 20:25 |
| openstackgerrit | Michael Barton proposed openstack/swift: go: time out flock differently https://review.openstack.org/334646 | 20:26 |
| *** adu has quit IRC | 20:32 | |
| *** manous has quit IRC | 20:37 | |
| *** asettle has joined #openstack-swift | 20:42 | |
| openstackgerrit | Michael Barton proposed openstack/swift: go: get rid of BackendError https://review.openstack.org/334655 | 20:44 |
| notmyname | pdardeau: your question reminded me of something I wanted to add to that doc. thanks :-) | 20:48 |
| *** manous has joined #openstack-swift | 20:50 | |
| *** catintheroof has joined #openstack-swift | 20:55 | |
| *** asettle has quit IRC | 20:58 | |
| *** manous has quit IRC | 20:58 | |
| *** mvk_ has joined #openstack-swift | 21:01 | |
| *** cdelatte has quit IRC | 21:03 | |
| *** cdelatte has joined #openstack-swift | 21:03 | |
| *** mkrcmari__ has joined #openstack-swift | 21:05 | |
| *** raildo is now known as raildo-afk | 21:07 | |
| *** mvk has joined #openstack-swift | 21:07 | |
| *** mvk_ has quit IRC | 21:09 | |
| *** mkrcmari__ has quit IRC | 21:10 | |
| timburke | all this stuff about metadata has me wondering whether we should be migrating from X-Static-Large-Object to X-Object-Sysmeta-Static-Large-Object, and why we haven't already. i think there's the possibility for bad behavior during failures | 21:20 |
| notmyname | timburke: bad behavior if we migrate or if we don't migrate? | 21:20 |
| timburke | if we don't | 21:20 |
| *** asettle has joined #openstack-swift | 21:24 | |
| *** asettle has quit IRC | 21:26 | |
| clayg | timburke: historically i've not worried much about the system meta that manually plumbs it's business down - the ones that freak me out are the system meta that hi-jacks the user meta space? | 21:34 |
| *** mvk_ has joined #openstack-swift | 21:35 | |
| *** geaaru has quit IRC | 21:35 | |
| *** permalac_ has joined #openstack-swift | 21:35 | |
| *** pauloewerton has quit IRC | 21:35 | |
| *** permalac has quit IRC | 21:36 | |
| *** mvk has quit IRC | 21:38 | |
| timburke | clayg: i think this (combined with your observations about unparseable SLOs) might lead to objects that are only *sometimes* available. if an SLO is overwritten but one of those copies has to go to a handoff, then the object is POSTed to after things recover but before replication, you'll have a split brain where one or more object servers have the X-Static-Large-Object header despite no longer storing a manifest | 21:39 |
| *** mkrcmari__ has joined #openstack-swift | 21:41 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift: Fix swift-get-nodes arg parsing for missing ring https://review.openstack.org/334238 | 21:42 |
| *** mkrcmari__ has quit IRC | 21:43 | |
| *** mvk has joined #openstack-swift | 21:44 | |
| *** mvk_ has quit IRC | 21:44 | |
| *** catintheroof has quit IRC | 21:55 | |
| *** ManojK has quit IRC | 22:00 | |
| *** ManojK has joined #openstack-swift | 22:01 | |
| *** zul_ has joined #openstack-swift | 22:09 | |
| *** zul has quit IRC | 22:12 | |
| *** peterlisak has quit IRC | 22:13 | |
| *** onovy has quit IRC | 22:14 | |
| *** kaleta has quit IRC | 22:14 | |
| *** jamielennox is now known as jamielennox|away | 22:21 | |
| mattoliverau | morning | 22:24 |
| *** ametts has quit IRC | 22:28 | |
| *** ManojK has quit IRC | 22:30 | |
| *** siva_krish has quit IRC | 22:34 | |
| *** siva_krish has joined #openstack-swift | 22:35 | |
| *** siva_krish has quit IRC | 22:38 | |
| *** ametts has joined #openstack-swift | 22:41 | |
| *** ouchkernel has quit IRC | 22:43 | |
| *** ouchkernel has joined #openstack-swift | 22:48 | |
| *** ametts has quit IRC | 22:48 | |
| *** ManojK has joined #openstack-swift | 22:57 | |
| *** mvk_ has joined #openstack-swift | 22:58 | |
| *** mvk has quit IRC | 23:02 | |
| *** ouchkernel has quit IRC | 23:08 | |
| *** _JZ_ has quit IRC | 23:12 | |
| *** ouchkernel has joined #openstack-swift | 23:15 | |
| *** kei_yama has joined #openstack-swift | 23:23 | |
| clayg | timburke: that sounds annoying terrible - like X-Static-Large-Object should already be a protected header that has the life-cycle of sysmeta (i.e. lives with and is persisted only in the .data) | 23:26 |
| timburke | clayg: this call seems to indicate otherwise :-/ https://github.com/openstack/swift/blob/3944d82/swift/obj/server.py#L500 | 23:29 |
| timburke | i still haven't writtena probe test to check it or anything, though | 23:29 |
| *** _JZ_ has joined #openstack-swift | 23:31 | |
| *** pgbridge has quit IRC | 23:35 | |
| *** mwheckmann has quit IRC | 23:37 | |
| clayg | timburke: yup, one of good reasons for classifying metadata was so that all the complicated lifecycle stuff could get applied consistently - i'm sure x-static-large-object works fine with POST in the old post_as_copy world... | 23:39 |
| *** onovy has joined #openstack-swift | 23:39 | |
| *** peterlisak has joined #openstack-swift | 23:39 | |
| *** kaleta has joined #openstack-swift | 23:40 | |
| clayg | i guess DLO's manifest header/metadata are updable with POST, but a SLO has a body and should probably be disallowed in .meta files :\ | 23:40 |
| *** lyrrad has quit IRC | 23:41 | |
| clayg | somewhere around https://github.com/openstack/swift/blob/3944d82/swift/obj/diskfile.py#L2220 is were we pull out key system metadata to play over the .meta metadata :\ | 23:41 |
| *** lyrrad has joined #openstack-swift | 23:42 | |
| *** jamielennox|away is now known as jamielennox | 23:43 | |
| *** _JZ_ has quit IRC | 23:46 | |
| clayg | timburke: yeah it's going to be pretty annoying to expose in a probe test - but it's almost definately a bug - we should have just continued to ignore x-static-large-object in metadata on post and make it so that metadata item on a data file could bleed through a .meta update | 23:48 |
| clayg | timburke: basically revert patch 182564 and make the change in diskfile | 23:49 |
| patchbot | clayg: https://review.openstack.org/#/c/182564/ - swift - Fix the missing SLO state on fast-post (MERGED) | 23:49 |
| clayg | timburke: I think it's mostly safe to ignore any x-static-large-object in a .meta if the key is not there in a data because that's nonsense too | 23:50 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!