| *** gyee has quit IRC | 00:45 | |
| *** nanzha has joined #openstack-swift | 01:21 | |
| *** spsurya has joined #openstack-swift | 01:59 | |
| *** baojg has quit IRC | 02:11 | |
| *** zaitcev has quit IRC | 02:50 | |
| *** psachin has joined #openstack-swift | 04:02 | |
| openstackgerrit | Merged openstack/swift master: py38: Provide readinto() interface for RingReader https://review.opendev.org/688591 | 04:03 |
|---|---|---|
| openstackgerrit | Merged openstack/swift master: swift-account-audit: clean up some error formatting https://review.opendev.org/688529 | 04:03 |
| *** baojg has joined #openstack-swift | 04:27 | |
| *** pcaruana has joined #openstack-swift | 04:37 | |
| openstackgerrit | Tim Burke proposed openstack/swift stable/train: CHANGELOG for 2.23.1 https://review.opendev.org/688848 | 04:52 |
| *** tkajinam has quit IRC | 05:04 | |
| *** pcaruana has quit IRC | 05:14 | |
| *** hoonetorg has quit IRC | 05:35 | |
| *** tkajinam has joined #openstack-swift | 05:42 | |
| *** hoonetorg has joined #openstack-swift | 05:48 | |
| *** pcaruana has joined #openstack-swift | 06:13 | |
| *** nanzha has quit IRC | 06:34 | |
| *** nanzha has joined #openstack-swift | 06:43 | |
| ktsuyuzaki | hello | 06:46 |
| ktsuyuzaki | oops, my nick was changed... | 06:46 |
| *** ktsuyuzaki is now known as kota_ | 06:46 | |
| kota_ | ok | 06:46 |
| *** takamatsu has joined #openstack-swift | 07:08 | |
| *** tesseract has joined #openstack-swift | 07:12 | |
| *** tesseract has quit IRC | 07:14 | |
| openstackgerrit | Merged openstack/swift master: tests: Stop invoking python just to get the real source directory https://review.opendev.org/688826 | 07:14 |
| *** tesseract has joined #openstack-swift | 07:14 | |
| *** e0ne has joined #openstack-swift | 07:19 | |
| *** rdejoux has joined #openstack-swift | 07:29 | |
| *** rpittau|afk is now known as rpittau | 08:12 | |
| *** tkajinam has quit IRC | 08:16 | |
| *** takamatsu has quit IRC | 08:21 | |
| *** takamatsu has joined #openstack-swift | 08:36 | |
| *** nanzha has quit IRC | 08:40 | |
| *** nanzha has joined #openstack-swift | 08:43 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift master: Imported Translations from Zanata https://review.opendev.org/688883 | 09:02 |
| *** nanzha has quit IRC | 09:14 | |
| *** nanzha has joined #openstack-swift | 09:21 | |
| *** rcernin has quit IRC | 09:54 | |
| *** NM has joined #openstack-swift | 10:41 | |
| *** NM has quit IRC | 10:59 | |
| *** rcernin has joined #openstack-swift | 11:30 | |
| *** nanzha has quit IRC | 11:55 | |
| *** tomha has joined #openstack-swift | 12:50 | |
| *** tomha has quit IRC | 12:53 | |
| openstackgerrit | Alexandre Lécuyer proposed openstack/swift feature/losf: EC frags are the ones with the # https://review.opendev.org/686846 | 13:00 |
| openstackgerrit | Alexandre Lécuyer proposed openstack/swift feature/losf: In _get_vfile(), handle the case when a referenced object has been punched https://review.opendev.org/659254 | 13:06 |
| *** NM has joined #openstack-swift | 13:13 | |
| ormandj | is there any support for immutability? i didn't see any in docs, but figured i'd ask | 13:21 |
| ormandj | (WORM support) | 13:22 |
| *** NM has quit IRC | 13:24 | |
| rledisez | ormandj: it's currently not possible in Swift. The eventually consistent model of Swift makes it really difficult to implement | 13:24 |
| ormandj | fair enough, thank you for the direct answer | 13:25 |
| *** NM has joined #openstack-swift | 13:25 | |
| rledisez | ormandj: what are you trying to achieve. do you need immutability like you need guarantee. because if your looking to publish an object, you can play with ACL or tempurl so only you can write, and "others" can read | 13:28 |
| ormandj | immutability, so there is no way possible to tamper with data after write as defined by policy | 13:28 |
| ormandj | to include the original writer (in this case, it would be software) | 13:29 |
| rledisez | ormandj: ok, so it's what I was saying, not possible | 13:29 |
| ormandj | we had looked at ACLs as an option but hadn't found a way to make it happen. this is unfortunate, as it is becoming more and more required by certain types of customers | 13:30 |
| ormandj | thank you again for the info | 13:30 |
| tdasilva | ormandj: would versioning help? | 13:30 |
| ormandj | tdasilva: as long as old versions cannot be deleted :) a typical scenario would be ransomware concerns | 13:30 |
| ormandj | backups are no good if the software creating them is on a compromised machine and the backups can be mutated, for example | 13:31 |
| ormandj | SEC/FINRA requires this kind of functionality | 13:31 |
| tdasilva | ormandj: would your users be able to modify the container metadata itself or just be given write/read acls to it? | 13:35 |
| tdasilva | ormandj: checkout https://github.com/openstack/swift/blob/master/test/functional/test_versioned_writes.py#L508 maybe something like that could help | 13:35 |
| ormandj | well, in this case, let's say you have backup software X. X would need write access to actually commit the backup to the object store | 13:35 |
| ormandj | would there by any way to restrict X to write only, with no ability to delete/mutate/etc? | 13:36 |
| ormandj | (looking through that test file now) | 13:37 |
| ormandj | hm - is the idea you'd create a versions container that X software would have no access to, then create a container that it would have access to but no permissions to update version container location/feature, then as writes happen they end up being written to both the container as well as the second versions container | 13:40 |
| ormandj | so X could delete from the original container it was uploaded to, but not the versions container? | 13:40 |
| tdasilva | Correct, but while using history mode. New writes or deletes just cause new writes into versions container. | 13:42 |
| ormandj | does the data physically exist in two containers? ie: storage consumption is doubled? | 13:44 |
| ormandj | and if the original container is deleted, does the data persist in the versions container? | 13:44 |
| ormandj | (thanks for all of the help, as always) | 13:45 |
| *** rcernin has quit IRC | 13:57 | |
| tdasilva | ormandj: data is not duplicated. latest version is stored on versioned (primary) container while older versions are on versions (secondary) container | 14:07 |
| tdasilva | if using history mode, if object is deleted, we actually store a delete marker | 14:08 |
| tdasilva | and keep the object in the versions container | 14:08 |
| ormandj | hm, ok | 14:22 |
| ormandj | if you delete the primary container, does the data persist in secondary? | 14:23 |
| ormandj | i'm assuming so based on your description - we'll just test all this - thanks for all the tips | 14:26 |
| tdasilva | ormandj: yes, you would have to delete each container separately | 14:29 |
| tdasilva | and to delete the primary container, you would need to issue a delete to all objects in that container first, which would ensure the data is moved to the versions container | 14:29 |
| tdasilva | ormandj: Now here comes the curve ball....having said all that, we are currently working on a new version of object versioning and I just realized we don't have such "functionality" in this new version. so I'm writing that down as something we need to look into | 14:31 |
| ormandj | hah, well, look into the SEC/FINRA requirements, amazon implements this as 'object lock' | 14:45 |
| ormandj | if you can cover those bases while implementing your versioning | 14:45 |
| ormandj | you'd remove a barrier for a lot of clients | 14:45 |
| ormandj | https://aws.amazon.com/blogs/storage/protecting-data-with-amazon-s3-object-lock/ | 14:48 |
| tdasilva | yeah, I saw the aws 'object lock' feature and that looked very interesting. It does however put as back into the issue of eventual consistency within swift | 14:48 |
| ormandj | yeah | 14:49 |
| ormandj | but even with eventual consistency, if you enforce the WORM model, i'm not fully clear on how that causes a problem | 14:49 |
| ormandj | 1) create container with policy in place that disallows any data mutation/deletes, only new commits 2) ? | 14:50 |
| tdasilva | heh, number 1 is the real challenge | 14:52 |
| ormandj | couldn't the existing ACL stuff just be updated to support a new restriction that disallows any of the operations that would mutate/delete/etc data? | 14:53 |
| tdasilva | swift could write data in any handoff device | 14:53 |
| tdasilva | ormandj: but a write can be used as a "delete" | 14:53 |
| ormandj | can't the ACL system be updated to prevent that? or perhaps i'm missing something. | 14:56 |
| ormandj | or is the eventual consistency thing you are describing the whole PUT to replace not actually checking when you submit the PUT if the object already exists | 14:58 |
| ormandj | thus no ACL check can be done at that point that would block the PUT? | 14:58 |
| tdasilva | exactly, a PUT has no way of checking if something already exists | 14:58 |
| tdasilva | and being 100% certain | 14:59 |
| ormandj | maybe add a policy that can be applied to containers that turns on a consistency model for this? | 15:00 |
| ormandj | that'd be nice to have for other reasons, too | 15:12 |
| ormandj | i suppose there's some performance implications there with blocking processes (aside from the obvious performance hit with it enabled) | 15:14 |
| *** NM has quit IRC | 15:22 | |
| openstackgerrit | Merged openstack/swift feature/losf: EC frags are the ones with the # https://review.opendev.org/686846 | 15:25 |
| *** gyee has joined #openstack-swift | 15:28 | |
| *** diablo_rojo has joined #openstack-swift | 15:48 | |
| *** rdejoux has quit IRC | 15:57 | |
| *** NM has joined #openstack-swift | 15:59 | |
| *** tesseract has quit IRC | 16:08 | |
| *** diablo_rojo has quit IRC | 16:20 | |
| *** diablo_rojo has joined #openstack-swift | 16:20 | |
| *** psachin has quit IRC | 16:29 | |
| *** NM has quit IRC | 16:29 | |
| rledisez | ormandj: timburke: would it be acceptable to have an "eventually immutable" possibility? i imagine setting a metadata on object that flag it immutable, then during replication, if the flag is set on an older version, the new version is cleaned up instead of the older | 16:34 |
| tdasilva | rledisez, ormandj sorry for not getting back, just been dealing with multiple things today. anyway, i think a feature like 'object lock' would require either changes in swift in how we write to handoffs or change in the backend of where we write swift data to. for example we could have a storage policy with 1 replica and the backing "device" be a strong consistent layer that is doing replication | 16:37 |
| tdasilva | the fact that aws require object-lock to be turned on to new buckets only makes me thing they are in a similar situation | 16:38 |
| tdasilva | basically requires a new storage policy | 16:38 |
| timburke | rledisez, interesting idea... i think it'd have to be set on the initial PUT -- but what about when all old "immutable" replicas are unavailable and a new "immutable" write comes in? presumably we'd go out to handoffs, but how should replication resolve it? no matter what we do, we're going to end up removing data that was PUT with the "immutable" flag and for which we sent back a 201... | 16:43 |
| rledisez | timburke: i would say the older wins (opposite of current policy) | 16:47 |
| rledisez | it could event be a different policy with different diskfile with reverted replication behavior (keep older timestramp) | 16:47 |
| rledisez | but i guess there would be many corner cases | 16:48 |
| *** rpittau is now known as rpittau|afk | 16:50 | |
| *** e0ne has quit IRC | 17:01 | |
| *** baojg has quit IRC | 17:02 | |
| rledisez | timburke: just so you know, at OVH we regularly get this request of "immutable" object | 17:07 |
| timburke | i'm sure | 17:10 |
| *** NM has joined #openstack-swift | 17:12 | |
| *** spsurya has quit IRC | 17:58 | |
| *** NM has quit IRC | 17:59 | |
| *** NM has joined #openstack-swift | 18:05 | |
| *** e0ne has joined #openstack-swift | 18:18 | |
| ormandj | yeah, it's a huge need for certain industries, unfortunately it's a hard problem with a non-consistent backing store hah | 18:22 |
| *** diablo_rojo has quit IRC | 18:32 | |
| *** diablo_rojo has joined #openstack-swift | 18:35 | |
| *** e0ne has quit IRC | 18:57 | |
| *** NM has quit IRC | 18:58 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: WIP: Allow internal clients to use null namespace https://review.opendev.org/682138 | 18:58 |
| *** NM has joined #openstack-swift | 19:10 | |
| *** NM has quit IRC | 19:47 | |
| *** NM has joined #openstack-swift | 19:50 | |
| *** rdejoux has joined #openstack-swift | 20:03 | |
| *** pcaruana has quit IRC | 20:28 | |
| *** NM has quit IRC | 20:28 | |
| *** NM has joined #openstack-swift | 20:32 | |
| clayg | so... I need to bounce 😞 | 20:54 |
| clayg | but for the meeting... I'd guess @timburke and @tdasilva will be able to say whatever neeeds to be said for versioning | 20:55 |
| clayg | if there's a review I'm putting off please ping a few at me and I'll be sure to put in some time | 20:55 |
| clayg | ya'll have a good afternood | 20:55 |
| kota_ | morning | 20:57 |
| timburke | kota_! glad you're doing ok -- family and everyone ok, too? | 21:00 |
| kota_ | timburke: sure. thx! | 21:00 |
| *** ormandj has quit IRC | 21:09 | |
| *** ormandj has joined #openstack-swift | 21:10 | |
| *** NM has quit IRC | 21:27 | |
| tdasilva | forgot to mention, if anyone has a chance, it would be good to present swift at fosdem again next year: https://lists.fosdem.org/pipermail/fosdem/2019q4/002891.html | 21:46 |
| tdasilva | alecuyer, rledisez, cschwede ^^^ | 21:46 |
| tdasilva | unfortunately i won't be able to make it this itme | 21:47 |
| alecuyer | tdasilva: thanks for the heads-up! we could try with romain, maybe talk about versioning and other things you are all working on | 21:49 |
| timburke | tdasilva, oh wow... i hadn't realized how much larger the py3 docker image is compared to the py2 one... funny | 21:51 |
| timburke | still way smaller than what i made for myself for keystone and barbican, though; those were like 165-180MB | 21:52 |
| *** diablo_rojo has quit IRC | 23:10 | |
| *** gyee has quit IRC | 23:16 | |
| *** tkajinam has joined #openstack-swift | 23:31 | |
| *** diablo_rojo has joined #openstack-swift | 23:45 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!