| *** saschpe has quit IRC | 00:00 | |
| openstackgerrit | A change was merged to openstack/swift: Fix unit tests. https://review.openstack.org/82207 | 00:06 |
|---|---|---|
| openstackgerrit | A change was merged to openstack/swift: Put X-Timestamp in object 404 responses https://review.openstack.org/80729 | 00:06 |
| *** saschpe has joined #openstack-swift | 00:07 | |
| openstackgerrit | A change was merged to openstack/swift: Block X-Backend* in gatekeeper https://review.openstack.org/81964 | 00:17 |
| *** gyee has quit IRC | 00:19 | |
| *** sungju_ has joined #openstack-swift | 00:26 | |
| openstackgerrit | Jenkins proposed a change to openstack/swift: Updated from global requirements https://review.openstack.org/81207 | 00:27 |
| *** shri has quit IRC | 00:36 | |
| *** piousbox has quit IRC | 00:42 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack/swift: Add sitepackages flag to tox.ini https://review.openstack.org/82228 | 00:51 |
| *** piousbox has joined #openstack-swift | 00:54 | |
| *** jasondotstar has joined #openstack-swift | 00:57 | |
| *** sungju_ has quit IRC | 01:01 | |
| zaitcev | portante: https://gist.github.com/zaitcev/9699561 | 01:02 |
| *** piousbox has quit IRC | 01:07 | |
| * portante looks | 01:20 | |
| portante | nice, clayg, notmyname, others ^^^ | 01:22 |
| *** sungju_ has joined #openstack-swift | 01:23 | |
| *** sungju_ has quit IRC | 01:25 | |
| zaitcev | maybe make it narrower | 01:42 |
| *** sungju_ has joined #openstack-swift | 01:45 | |
| *** sungju_ has quit IRC | 01:45 | |
| clayg | zaitcev: portante: yes - so nice | 01:56 |
| zaitcev | clayg: Not sure if it makes apparent how the surface of the API expands into the body of the server, sweeping some of the sordid details of checking if this exists and that exists. | 01:57 |
| clayg | well... how does GET for example raise 404? | 01:58 |
| zaitcev | clayg: In the same time, since auditors/replicators require the old Broker to be kept, the implementation currently is stratified | 01:58 |
| zaitcev | clayg: I can tell without looking at the code - get_info returns None (I hope) | 01:59 |
| clayg | well either way this gist is very helpful - did portante do that or did you put it together | 02:00 |
| zaitcev | I did | 02:00 |
| clayg | well thank you | 02:00 |
| zaitcev | Peter pointed out that most likely we still have a bunch of races like the one you fixed. If we divorce the Backend from assumption that DB is a file which can be checked for existence etc., then we should be able to proceed with some sort of transactional bracketing | 02:02 |
| zaitcev | without changing API further | 02:02 |
| zaitcev | hmm. Looking at the text file anew, it seems unobvious why we had to un-roll _deleted_response. | 02:04 |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Container Confict on obj update with wrong Storage-Policy-Index https://review.openstack.org/79731 | 02:08 |
| *** fifieldt has joined #openstack-swift | 02:16 | |
| creiht | https://leastauthority.com/blog/BLAKE2-harder-better-faster-stronger-than-MD5.html | 02:21 |
| Alex_Gaynor | BLAKE2 is pretty awesome | 02:22 |
| creiht | Alex_Gaynor: yeah I know, zooko has been pushing it for a while | 02:24 |
| creiht | and I have a lot of respect for zooko | 02:24 |
| creiht | just something to keep in the back of our minds if we decide to move from the default md5 that we do | 02:24 |
| creiht | if/when | 02:25 |
| portante | nice | 02:25 |
| Alex_Gaynor | creiht: Do you know to what extent the hashing of the account/container/objectname is a perforamnce issue? I know the marconi folks found md5 to be too slow for their needs, but I think they were aiming for much tighter performance | 02:26 |
| clayg | Alex_Gaynor: do you know off-hand what they went with? | 02:27 |
| creiht | Alex_Gaynor: in the grand scheme of things not a huge deal | 02:27 |
| creiht | we did a lot of benchmarking at the beginning | 02:27 |
| creiht | md5 won out mostly because it was well known and ubiquitous | 02:28 |
| Alex_Gaynor | clayg: Nope, I could have sworn they went with something less conventional, but I don't see any cool things in their requirements.txt | 02:28 |
| creiht | I *think* all of the faster options we looked at were not cryptographic hashes, so easy to get collisions | 02:29 |
| creiht | been a while | 02:29 |
| * Alex_Gaynor goes to search the abanndoned changes to requirements | 02:30 | |
| hugokuo | clayg: https://github.com/swiftstack/keystone_install ... You may want to change the reseller_prefix in the script from SWIFTSTACK_ to KEY_ or something else | 02:30 |
| clayg | hugokuo: werd | 02:30 |
| *** saschpe has quit IRC | 02:30 | |
| Alex_Gaynor | waaa, looks like they're using crc32 | 02:30 |
| Alex_Gaynor | There was a proposal to use murmurhash | 02:31 |
| creiht | we played with murmur | 02:31 |
| Alex_Gaynor | tbh, if I was doing this from scratch I'd probably use siphash | 02:31 |
| Alex_Gaynor | You're mapping to way fewer slots than 2**64, so the extra output bits aren't valuable | 02:31 |
| *** saschpe has joined #openstack-swift | 02:31 | |
| Alex_Gaynor | (https://131002.net/siphash/) | 02:32 |
| creiht | Alex_Gaynor: I think we have had requests to do ring sizes that are greater than 32bits | 02:32 |
| Alex_Gaynor | siphash is a 64-bit output | 02:32 |
| creiht | ahh | 02:32 |
| creiht | well another one to put in the back of the mind | 02:33 |
| creiht | :) | 02:33 |
| Alex_Gaynor | yeah, blake2 for file contents and siphash would be the things to think about I guess | 02:33 |
| creiht | Implementations of SipHash are available in C, C#, Cryptol, D, Erlang, Go, Haskell, Java, Javascript, Lisp, Ruby, Rust, and PHP. | 02:33 |
| Alex_Gaynor | For some future wher we all have tons of free time | 02:33 |
| creiht | there is not python on that list | 02:33 |
| creiht | :( | 02:33 |
| Alex_Gaynor | There's python bindings to the C implementation | 02:33 |
| creiht | ahh | 02:33 |
| creiht | Alex_Gaynor: From a UX perspective though, it is hard to go with something besides md5 | 02:35 |
| creiht | for file contents | 02:35 |
| Alex_Gaynor | You mean because it requires extra packages? | 02:35 |
| Alex_Gaynor | Specificalyl on clients as well | 02:35 |
| creiht | yeah | 02:35 |
| Alex_Gaynor | Yeah. | 02:35 |
| creiht | and you can md5sum on the commandline | 02:36 |
| * Alex_Gaynor looks forward to a future where we all have free time and access to neat crypto | 02:37 | |
| creiht | heh | 02:37 |
| clarkb | sha1? | 02:37 |
| creiht | I have free time, but that is mostly occupied by kids and a 3dprinter | 02:37 |
| creiht | clarkb: sha1 has similar issues as md5 | 02:38 |
| clarkb | creiht: not entirely | 02:38 |
| clarkb | there are no arbitrary collision methods | 02:38 |
| Alex_Gaynor | It's also slower. | 02:38 |
| creiht | clarkb: well I should specify, it is assumed in most circles it is only a short matter of time before sha1 is in the same boat as md5 | 02:38 |
| Alex_Gaynor | Plus, SHA has the problem of naming conceit. With a name like "Secure Hash Algorithm", youv'e tempted the fates. | 02:39 |
| creiht | If we change to something, I would like to make sure it is a bit more future proof, and has other advantages (like being faster) | 02:39 |
| clarkb | Alex_Gaynor: heh | 02:39 |
| creiht | haha | 02:40 |
| clarkb | creiht: thats fair | 02:40 |
| clarkb | but on the ubuquitous bit sha1 is the next best bet from md5 | 02:40 |
| clarkb | and it doesn't have the md5 problems | 02:40 |
| clarkb | not yet at least | 02:40 |
| portante | has anybody compared an md5 python implementation to siphash or blake1? | 02:40 |
| creiht | when we first implemented swift, md5 was in a very similar boat as sha1 is now | 02:40 |
| creiht | portante: no | 02:40 |
| Alex_Gaynor | portante: As in, pure python, or bindings to some C impl? | 02:41 |
| creiht | I think md5 is still fine as we use it now, but that window continues to close | 02:41 |
| portante | any comparison to get some more sense of what ballparks they play in other than somebody says blake1 is faster | 02:41 |
| clarkb | creiht: Alex_Gaynor if you want command line tools and cryptographic usefulness today sha1 isn't terrible | 02:42 |
| clarkb | md5 is | 02:42 |
| portante | do you mean somebody can construct an attack on a swift cluster? | 02:42 |
| creiht | portante: not currently | 02:42 |
| creiht | it would require a preimage attack on a source that is not in your control | 02:44 |
| creiht | in a very small space of bytes | 02:46 |
| creiht | but anyways, at some point users are going to start requesting to use a stronger hash as time goes on | 02:48 |
| * portante has nothing in his control anyways ;) | 02:49 | |
| creiht | heh | 02:49 |
| creiht | the worst attack vector would be for someone to craft a valid url that hashes to another user's object | 02:50 |
| portante | but that just sounds like a collision attack | 02:50 |
| Alex_Gaynor | Preimage attack, they can't control what the other person's object is. | 02:50 |
| Alex_Gaynor | There's also the potential for a DoS against the service provider, that really does just require collisions. | 02:50 |
| creiht | yeah that's where the idea of a preimage attack comes in | 02:51 |
| creiht | even if someone was able to do that, we have a check that validates the account with the account in the metadata of the object | 02:52 |
| creiht | just in case that were to happen some day | 02:52 |
| Alex_Gaynor | It's not clear that impersonating another object is a possible thing. DoS definitely is. Including DoS against a particular object (with preimage attack) | 02:52 |
| creiht | and yes the other potential is that an attack is able to create collisions that say put a ton of files on the same partition to fill up a hard drive | 02:53 |
| portante | but wouldn't the attacker have to have additional information about the swift cluster to do that? | 02:53 |
| creiht | yes | 02:54 |
| creiht | but that info can leak | 02:54 |
| portante | but we have gatekeeper! | 02:54 |
| creiht | lol | 02:54 |
| portante | ;) | 02:54 |
| creiht | as in a bad employee takes it with them | 02:54 |
| portante | ah yes | 02:54 |
| creiht | the upside is that bandwidth is limited | 02:55 |
| *** sriram has joined #openstack-swift | 02:55 | |
| portante | yes | 02:55 |
| creiht | it would also be very apparent | 02:56 |
| creiht | why is this one drive so statistically out of bounds in size? | 02:56 |
| portante | so for large object case that might be easier to see just by considering the logs | 02:57 |
| portante | but for small files, does that become much harder to detect? | 02:57 |
| Alex_Gaynor | Well there's DoS against teh disk space, and DoS against machine itself. | 02:58 |
| portante | true | 02:59 |
| creiht | portante: either way it should be detectable | 03:00 |
| *** sriram has quit IRC | 03:00 | |
| creiht | start looking at the partition | 03:00 |
| creiht | noticing it has a statistically larger number of objects than it is supposed to | 03:00 |
| Alex_Gaynor | Are people actually monitoring for this? | 03:01 |
| portante | perhaps the object auditor should construct reports on what it finds oriented towards that | 03:01 |
| portante | if it does not already | 03:01 |
| * portante does not remember | 03:01 | |
| creiht | Alex_Gaynor: we certainly watch statistics for stuff | 03:01 |
| creiht | if things are are statistically out of whack we start investigating | 03:01 |
| creiht | it would likely be easier to DOS an entire cluster than try stuff like that | 03:03 |
| clayg | we track drive fullness as proxy - i don't think we watch number of objects | 03:04 |
| creiht | we don't directly watch number of objects | 03:04 |
| clayg | portante: the auditor could definately throw something in recon | 03:04 |
| clayg | pandemicsyn: ^ | 03:04 |
| portante | since it is already crawling in the attic ... | 03:05 |
| creiht | I should do a summit talk sometimes about the interesting ways that a user can use your swift cluster | 03:08 |
| creiht | :) | 03:08 |
| portante | great, but can you charge a few thousand to attend? | 03:09 |
| portante | ;) | 03:09 |
| creiht | haha | 03:09 |
| creiht | well the talk would quickly turn into things like | 03:09 |
| creiht | and this is why you turn on ratelimiting | 03:09 |
| creiht | etc. | 03:09 |
| creiht | :) | 03:09 |
| portante | yes | 03:10 |
| creiht | or why we implemented certain features | 03:10 |
| creiht | or why you *can't* do some things in swift :) | 03:10 |
| *** Longgeek_ has joined #openstack-swift | 03:16 | |
| *** Longgeek_ has quit IRC | 03:38 | |
| *** RockKuo has joined #openstack-swift | 04:12 | |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Add Storage Policy Support to Account HEAD https://review.openstack.org/73747 | 04:31 |
| madhuri_ | clayg: there | 04:37 |
| madhuri_ | ? | 04:37 |
| clayg | bbiab - fwiw I didn't get a change to look at your branch today - but it's on my radar! If you have any questions fire away and I'll answer when I get a chance... | 04:38 |
| madhuri_ | I was just asking you fo that only. | 04:43 |
| *** jasondotstar has quit IRC | 04:53 | |
| clayg | madhuri_: I'm not worried about the ring thing - but you could go ahead and import optparse and make it an option if you're worried about gerry's comment. | 05:05 |
| clayg | madhuri_: why do you have that guard on the value for the container sync stuff - why not just display the -1 | 05:05 |
| madhuri_ | clayg: for the container sync stuff, I thought printing default value doesn't worth | 05:08 |
| clayg | hrmm... maybe | 05:09 |
| clayg | I guess it depends on if you're running container-sync or not | 05:09 |
| madhuri_ | clayg: And about gerry's comment, last night I tried releasing the fix | 05:09 |
| clayg | I appreciated the reported_put_timestamp stuff - and the 1970 "default" is useful to know that the container-updater hasn't run yet | 05:10 |
| madhuri_ | clayg: But hang up beacuse of some git error :) | 05:10 |
| clayg | lol | 05:10 |
| madhuri_ | Ok. I will remove the guard on the value for the container sync | 05:10 |
| clayg | oh idk, i was just asking | 05:11 |
| madhuri_ | Gerry's comment was only for the path i.e. /etc/swift. Right? | 05:11 |
| clayg | yeah in the server's configs they just specify "swift_dir" | 05:12 |
| clayg | /etc/swift is the default - you can't *really* change it because of /etc/swift/swift.conf and the hash_prefix biz | 05:12 |
| clayg | i don't think anyone really cares | 05:12 |
| madhuri_ | Yeah I saw it but I thought better option will be passing command line argument | 05:12 |
| clayg | sure | 05:12 |
| clayg | optional | 05:12 |
| clayg | like i said you can't really have a swift that doesn't home /etc/swift/swift.conf in /etc/swift so I think most people go ahead and put their rings in there too | 05:13 |
| clayg | plus if the ring is missing all you do is don't print the ring locations - right? | 05:14 |
| clayg | no bigz... | 05:14 |
| clayg | I'm sorta thinking the 'hash' value returned from get_info should really call itself chexor - most pople know it by the name of the function that calculates it | 05:15 |
| clayg | I honestly didn't recognize it cause I'm used to seeing "Object Hash" come out of swift-object-info and I thought it was telling me the salted md5 of the container path - so I freaked out it was all zeros - then I was like... oh yeah - that. | 05:15 |
| madhuri_ | Yeah. Better not printing the ring location | 05:16 |
| clayg | I had forgotten that swift-object-info even did that - I always just use swift-get-nodes for that job | 05:16 |
| clayg | anyway - i like it! | 05:16 |
| madhuri_ | Ok. Should I rename it? | 05:16 |
| clayg | how come it doesn't print X-Container-Read/Write - they should be metadata somewhere? But I don't think they start with x-container-meta | 05:17 |
| clayg | I would bet other folks would like it... I'm just talking about in the display name - there's not a good case in renaming it in the db | 05:17 |
| madhuri_ | get_info method doesnot return it | 05:17 |
| madhuri_ | Yeah me too talking about the display name | 05:18 |
| clayg | but to the layman or ops or someone like me who just forgot what that key was really called in the db - having it display as the 'hash' key as "Chexor" on the commandline would prevent anyone from confusing it with the hashed container path | 05:18 |
| clayg | I'll comment on the diff - it looks great - thanks for doing that. | 05:18 |
| madhuri_ | Ok. I will do that | 05:18 |
| *** zaitcev has quit IRC | 05:18 | |
| madhuri_ | My pleasure :) | 05:18 |
| madhuri_ | Right now writing swift-account-info :) | 05:19 |
| madhuri_ | X-Container-Read/Write, I will look for it and add it if I get something | 05:19 |
| *** saju_m has joined #openstack-swift | 06:08 | |
| openstackgerrit | Madhuri Kumari proposed a change to openstack/swift: Added swift-container-info tool. https://review.openstack.org/81968 | 07:55 |
| *** mlipchuk has joined #openstack-swift | 08:03 | |
| openstackgerrit | Clay Gerrard proposed a change to openstack/swift: Add Storage Policy Support to Account HEAD https://review.openstack.org/73747 | 08:21 |
| *** jasondotstar has joined #openstack-swift | 08:56 | |
| *** mmcardle has joined #openstack-swift | 09:33 | |
| *** d89 has joined #openstack-swift | 09:56 | |
| *** jamie_h has joined #openstack-swift | 10:38 | |
| *** jasondotstar has quit IRC | 10:44 | |
| *** mkollaro has joined #openstack-swift | 11:16 | |
| *** Trixboxer has joined #openstack-swift | 11:26 | |
| *** mkollaro has quit IRC | 11:31 | |
| *** jasondotstar has joined #openstack-swift | 11:48 | |
| openstackgerrit | Madhuri Kumari proposed a change to openstack/swift: Added swift-container-info tool. https://review.openstack.org/81968 | 12:00 |
| *** saju_m has quit IRC | 12:03 | |
| *** jamie_h has quit IRC | 12:04 | |
| *** mmcardle has quit IRC | 12:49 | |
| *** Longgeek_ has joined #openstack-swift | 12:55 | |
| *** Longgeek_ has quit IRC | 12:56 | |
| *** Midnightmyth has joined #openstack-swift | 12:57 | |
| *** mmcardle has joined #openstack-swift | 12:59 | |
| openstackgerrit | Madhuri Kumari proposed a change to openstack/swift: Added swift-account-info tool. https://review.openstack.org/82296 | 13:04 |
| openstackgerrit | paul luse proposed a change to openstack/swift: Add Storage Policy Support to Account HEAD https://review.openstack.org/73747 | 13:05 |
| openstackgerrit | paul luse proposed a change to openstack/swift: Container Confict on obj update with wrong Storage-Policy-Index https://review.openstack.org/79731 | 13:05 |
| *** mkollaro has joined #openstack-swift | 13:12 | |
| *** Longgeek_ has joined #openstack-swift | 13:29 | |
| *** Longgeek_ has quit IRC | 13:37 | |
| *** mkollaro has quit IRC | 14:00 | |
| *** taras_ has quit IRC | 14:01 | |
| *** mkerrin has quit IRC | 14:01 | |
| *** Alex_Gaynor has quit IRC | 14:01 | |
| *** wayneeseguin has quit IRC | 14:01 | |
| *** fbo_away has quit IRC | 14:01 | |
| *** jeblair has quit IRC | 14:01 | |
| *** acorwin has quit IRC | 14:01 | |
| *** redbo has quit IRC | 14:01 | |
| *** notmyname has quit IRC | 14:01 | |
| *** MooingLemur has quit IRC | 14:01 | |
| *** rahmu has quit IRC | 14:01 | |
| *** torgomatic has quit IRC | 14:01 | |
| *** minnear has quit IRC | 14:01 | |
| *** glange has quit IRC | 14:01 | |
| *** flashgordon has quit IRC | 14:01 | |
| *** hugokuo has quit IRC | 14:01 | |
| *** ctennis has quit IRC | 14:01 | |
| *** anderstj has quit IRC | 14:01 | |
| *** alpha_ori has quit IRC | 14:01 | |
| *** ryao has quit IRC | 14:01 | |
| *** anticw has quit IRC | 14:01 | |
| *** dfg has quit IRC | 14:01 | |
| *** mordred has quit IRC | 14:01 | |
| *** kragniz has quit IRC | 14:01 | |
| *** flashgordon has joined #openstack-swift | 14:01 | |
| *** anticw has joined #openstack-swift | 14:01 | |
| *** torgomatic has joined #openstack-swift | 14:01 | |
| *** ChanServ sets mode: +v torgomatic | 14:01 | |
| *** redbo has joined #openstack-swift | 14:01 | |
| *** ChanServ sets mode: +v redbo | 14:01 | |
| *** wayneeseguin has joined #openstack-swift | 14:01 | |
| *** notmyname has joined #openstack-swift | 14:01 | |
| *** MooingLemur has joined #openstack-swift | 14:01 | |
| *** minnear has joined #openstack-swift | 14:01 | |
| *** glange has joined #openstack-swift | 14:01 | |
| *** ChanServ sets mode: +v glange | 14:01 | |
| *** rahmu has joined #openstack-swift | 14:01 | |
| *** Alex_Gaynor has joined #openstack-swift | 14:02 | |
| *** dfg has joined #openstack-swift | 14:02 | |
| *** ryao has joined #openstack-swift | 14:02 | |
| *** jeblair has joined #openstack-swift | 14:02 | |
| *** mordred has joined #openstack-swift | 14:02 | |
| *** mkerrin has joined #openstack-swift | 14:02 | |
| *** ctennis has joined #openstack-swift | 14:02 | |
| *** fbo_away has joined #openstack-swift | 14:02 | |
| *** fbo_away is now known as fbo | 14:02 | |
| *** anderstj has joined #openstack-swift | 14:03 | |
| *** alpha_ori has joined #openstack-swift | 14:04 | |
| *** hugokuo has joined #openstack-swift | 14:04 | |
| *** acorwin has joined #openstack-swift | 14:05 | |
| *** taras_ has joined #openstack-swift | 14:06 | |
| *** kragniz has joined #openstack-swift | 14:06 | |
| *** saurabh_ has quit IRC | 14:16 | |
| *** saurabh_ has joined #openstack-swift | 14:16 | |
| *** saurabh_ has joined #openstack-swift | 14:16 | |
| *** seandrummond has joined #openstack-swift | 14:37 | |
| *** mmcardle has quit IRC | 15:37 | |
| *** mmcardle has joined #openstack-swift | 15:37 | |
| *** seandrummond has quit IRC | 15:39 | |
| *** mmcardle has quit IRC | 16:10 | |
| *** RockKuo has quit IRC | 16:37 | |
| *** haomaiw__ has quit IRC | 16:45 | |
| *** haomaiwang has joined #openstack-swift | 16:45 | |
| *** jasondotstar has quit IRC | 16:52 | |
| *** torgomatic has quit IRC | 17:01 | |
| *** seandrummond has joined #openstack-swift | 17:50 | |
| *** seandrummond has quit IRC | 17:52 | |
| *** mkollaro has joined #openstack-swift | 18:03 | |
| *** PradeepChandani_ has quit IRC | 18:19 | |
| *** krtaylor has quit IRC | 18:20 | |
| *** Trixboxer has quit IRC | 20:28 | |
| *** seandrummond has joined #openstack-swift | 20:29 | |
| *** anticw has quit IRC | 20:34 | |
| *** seandrummond has quit IRC | 20:40 | |
| *** ctennis has quit IRC | 20:52 | |
| *** ctennis has joined #openstack-swift | 20:52 | |
| *** seandrummond has joined #openstack-swift | 21:41 | |
| *** seandrummond has quit IRC | 22:10 | |
| *** krtaylor has joined #openstack-swift | 23:27 | |
| *** mkollaro has quit IRC | 23:28 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!