| *** gyee has quit IRC | 01:41 | |
| * kota_ is back to online | 02:07 | |
| kota_ | thx timburke for a lot of info (vdisk and storlets work)! | 02:08 |
|---|---|---|
| kota_ | hmm... the vdsik repo explains like `mounting virtual CD-ROM images backed by object storage`, that means it's for read-only? | 02:09 |
| DHE | I presume so. object storage (at least swift) doesn't really lend itself to in-place modification, and .iso isn't a write-friendly filesystem format | 02:11 |
| kota_ | DHE: make sense | 02:15 |
| *** ccamacho has quit IRC | 03:13 | |
| *** tkajinam has quit IRC | 04:26 | |
| *** tkajinam has joined #openstack-swift | 04:33 | |
| *** tkajinam_ has joined #openstack-swift | 05:08 | |
| *** tkajinam has quit IRC | 05:11 | |
| viks___ | Hi, I'm testing https://docs.aws.amazon.com/cli/latest/reference/s3api/list-buckets.html against my local swift s3, and i see that it is working | 05:19 |
| viks___ | i.e. | 05:19 |
| viks___ | ``` | 05:19 |
| viks___ | aws s3api list-buckets --endpoint-url https://10.255.252.22 --query "Buckets[].Name" | 05:19 |
| viks___ | [ | 05:19 |
| viks___ | "container1" | 05:19 |
| viks___ | ] | 05:19 |
| viks___ | ``` | 05:19 |
| viks___ | But in our swift doc, i.e. https://docs.openstack.org/swift/latest/s3_compat.html, it is mentioned as not supported i.e. `GET Bucket service Advanced Feature No`. | 05:19 |
| viks___ | Just wanted to know if it is documentation bug or am i missing something? | 05:20 |
| *** tkajinam_ has quit IRC | 05:34 | |
| *** tkajinam has joined #openstack-swift | 05:36 | |
| *** tkajinam has quit IRC | 06:03 | |
| *** tkajinam has joined #openstack-swift | 06:05 | |
| *** tkajinam has quit IRC | 06:30 | |
| *** tkajinam has joined #openstack-swift | 06:31 | |
| *** tkajinam has quit IRC | 06:31 | |
| kota_ | viks___: IIRC, GET service (i.e. list buckets) should work so I assume it's docs bug. | 06:38 |
| kota_ | but perhaps, there would have some ristrictions. | 06:38 |
| kota_ | there would be | 06:38 |
| kota_ | it seems like swift s3api is able to return the bucket list but the creation date seems to be fixed value so it's not complete feature. | 06:41 |
| viks___ | kota_: Ok thanks | 06:53 |
| *** tkajinam has joined #openstack-swift | 07:01 | |
| *** rdejoux_ has joined #openstack-swift | 07:14 | |
| viks___ | kota_: Any idea why do i get the below error: | 07:33 |
| viks___ | ``` | 07:33 |
| viks___ | aws s3api put-object --bucket my-bucket --key 1mb_file.txt --body 1mb_file.txt --endpoint-url https://10.255.252.22 | 07:33 |
| viks___ | An error occurred (BadDigest) when calling the PutObject operation (reached max retries: 4): The Content-MD5 you specified did not match what we received. | 07:33 |
| viks___ | ``` | 07:33 |
| viks___ | It works if i use signature v2 for awscli | 07:34 |
| viks___ | but not for v4 | 07:34 |
| kota_ | curious | 07:35 |
| kota_ | swift s3api supports v4 staff so it should work. | 07:35 |
| kota_ | i'm not an expert of awscli but the error looks to be caused by etag mismatch. | 07:37 |
| kota_ | ah... perhaps, something different from your swift setting and awscli? | 07:43 |
| kota_ | e.g. IIRC signature v4 requires region information to verify your credential so it should match between your local client setting and s3api config. | 07:45 |
| *** tesseract has joined #openstack-swift | 08:00 | |
| viks___ | kota_: Ok.. will check and get back | 08:02 |
| *** rdejoux has joined #openstack-swift | 08:09 | |
| viks___ | kota_: i have set `location = us-east-1` in `[filter:s3api]` and using `region = us-east-1` in `.aws/config`. Not sure why this is happening | 08:13 |
| viks___ | other commands like get object. list buckets etc work.. | 08:14 |
| kota_ | viks___: oh, you have the permission to look at swift configuration? | 08:14 |
| viks___ | kota_: yes | 08:14 |
| kota_ | perhaps, the swift log would be helpful to figure out what exactly happens. | 08:14 |
| kota_ | 201, 503, 4xx or some error trace... | 08:15 |
| *** rpittau|afk is now known as rpittau | 08:17 | |
| viks___ | it's 400 | 08:27 |
| *** tkajinam has quit IRC | 08:35 | |
| kota_ | viks___: ok. it seems like not md5 related but authentication error. | 09:01 |
| kota_ | sorry, I'm running out of my time to stay my office so you would check the configuration between the cli and service | 09:03 |
| kota_ | or possibly, the response body will be helpful to know what's exactly cause of the authentication error. | 09:04 |
| kota_ | e.g. `AWS authentication requires a valid Date or x-amz-date header` is an example error response body of 400. | 09:05 |
| *** mikecmpbll has joined #openstack-swift | 09:06 | |
| *** ccamacho has joined #openstack-swift | 09:12 | |
| *** jistr has quit IRC | 09:19 | |
| *** jistr has joined #openstack-swift | 09:20 | |
| viks___ | kota_: i see awscli is sending below headers when run with `--debug`: | 09:27 |
| viks___ | ``` | 09:27 |
| viks___ | {'Content-MD5': b'45G2LrOHl+D8Vd9FfunXVg==', 'Content-Length': '125389', 'Authorization': b'AWS4-HMAC-SHA256 Credential=1ed62d4b25aafeb5c1e2f969541405a7/20191113/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date, Signature=ae756ea7058504cbaeee78cbf87dc03f85e65b1a3292a2ed4733488402f26275', 'User-Agent': b'aws-cli/1.16.266 Python/3.5.2 Linux/4.15.0-66-generic | 09:27 |
| viks___ | botocore/1.13.2', 'Expect': b'100-continue', 'X-Amz-Content-SHA256': b'UNSIGNED-PAYLOAD', 'X-Amz-Date': b'20191113T085545Z'} | 09:27 |
| viks___ | ``` | 09:27 |
| *** jistr has quit IRC | 09:29 | |
| *** ccamacho has quit IRC | 09:32 | |
| *** jistr has joined #openstack-swift | 09:36 | |
| *** ccamacho has joined #openstack-swift | 09:36 | |
| *** ianychoi has quit IRC | 10:24 | |
| *** henriqueof1 has quit IRC | 12:04 | |
| *** rcernin has quit IRC | 12:54 | |
| *** ianychoi has joined #openstack-swift | 13:00 | |
| baffle | Has there been any discussion with regards to implementing a special/magic separator for the storage_domain/domain_remap middleware? I.e. to have working SSL with virtual host-style access. Since S3 has unique bucket names, they can issue a valid wildcard certificate for "*.s3.amazonaws.com", so that requests to https://bucket.s3.amazonaws.com works great. But in swift, since bucket/container names are not unique, we | 14:26 |
| baffle | also need the project-id (resellerprefix) in the requests, so we need to remap two levels; https://bucketname.AUTH_resellerprefix.objecstorage.com. Wildcard SSL certfs only supports one level of wildcards. Could we introduce something like "__" as a separator? I.e. use bucketname__resellerprefix.objecstorage.com instead. And make domain remap split on __.... Or something better/smarter. :) | 14:26 |
| DHE | I can only speak for myself. This isn't specific to S3 support either, but we're going to put nginx in front of swift and have it do SSL and path rewrites. | 14:28 |
| *** tesseract has quit IRC | 14:52 | |
| *** tesseract has joined #openstack-swift | 14:55 | |
| timburke | viks___, what version of swift are you running? i wonder if what you're seeing may have been fixed by https://review.opendev.org/#/c/660196/ | 15:15 |
| patchbot | patch 660196 - swift - s3api: Allow clients to upload with UNSIGNED-PAYLOAD (MERGED) - 1 patch set | 15:15 |
| clayg | baffle: I'd like to see something more like a "bucket registar" that allows projects to map their /AUTH_project/container to an unclaimed "myproject.mydomain.org" bucket | 15:15 |
| clayg | then requests to "myproject.mydomain.org.objectstorage.com" could lookup "myproject.mydomain.org" and rewrite to "objectstorage.com/AUTH_project/container" | 15:16 |
| DHE | wasn't there a cname or other DNS based middleware to do something like htat? | 15:17 |
| timburke | DHE, i was just thinking the same :-) | 15:17 |
| timburke | yeah, cname lookup + domain remap works pretty well | 15:17 |
| clayg | timburke: does it? for s3api style requests? | 15:18 |
| timburke | not s3api. but i don't think that's what baffle was going for... was it? | 15:19 |
| DHE | it's just a domain rewrite middleware. looks like you could just put it south of the s3api middleware. | 15:19 |
| baffle | timburke: I'm looking to use it for S3 middleware, but it is relevant for normal Swift API access as well, esp. for public objects. But for S3 is is more important, as AWS are planning on deprecating path-style access. I.e. they are deprecating endpoints like s3.amazonaws.com/bucket/object in favor of bucket.s3.amazonaws.com. Ref: https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/ | 15:29 |
| baffle | * timburke: I'm looking to use it for S3 middleware, but it is relevant for normal Swift API access as well, esp. for public objects. But for S3 it is more important, as AWS are planning on deprecating path-style access. I.e. they are deprecating endpoints like s3.amazonaws.com/bucket/object in favor of bucket.s3.amazonaws.com. Ref: | 15:32 |
| baffle | https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/ | 15:32 |
| baffle | So with CNAME middleware, we could have uniquecontainername.object.com resolve to nonuniquename.AUTH_uuid.objectstorage.com then. Well, that works, but it requires an extra step, registring the unique name. Having support for a magic separator in domain remap would work out of the box. | 15:36 |
| DHE | you're free to write your own middleware, and this CNAME middleware might make a good starting point. | 15:39 |
| DHE | but users might also appreciate the shorter names offered by cnames | 15:39 |
| timburke | s3api pulls the account info out of the auth info -- bucket.objectstorage.com should work (today, even!) for multiple tenants' containers named "bucket" | 15:40 |
| baffle | Is "_" actually valid in a hostname per. RFCs? It would seem only a to z, numbers and "-" is valid.. | 15:41 |
| timburke | (maybe "pulls out" isn't quite right -- but the auth info gets used to figure out which account this request should map to) | 15:41 |
| timburke | baffle, yeah, i don't think _ is good... part of why AWS started requiring that bucket names not include _ | 15:42 |
| timburke | if you want virtual-host style s3 access, you'll need to set the storage_domain option in proxy-server.conf: https://github.com/openstack/swift/blob/2.23.0/etc/proxy-server.conf-sample#L536-L538 | 15:43 |
| baffle | @s3api: That would only apply to authenticated requests. So one could even put "notvalid.objecstorage.com" and still reach "objecstorage.com/v1/AUTH_uuid/mybucket"? | 15:44 |
| baffle | timburke: I have virtual-host style enabled, but it will only work with http:// as wildcard certs are only one level. | 15:45 |
| baffle | * @timburkei: That would only apply to authenticated requests. So one could even put "notvalid.objecstorage.com" and still reach "objecstorage.com/v1/AUTH_uuid/mybucket"? | 15:49 |
| baffle | * timburke: That would only apply to authenticated requests. So one could even put "notvalid.objecstorage.com" and still reach "objecstorage.com/v1/AUTH_uuid/mybucket"? | 15:49 |
| timburke | on wildcard certs: s3api only needs the one level. i've got aws-cli making requests like https://bucket.saio/?list-type=2&prefix=&delimiter=%2F&encoding-type=url that will work just fine | 15:59 |
| timburke | presumably "notvalid.objecstorage.com" would get mapped to "objecstorage.com/v1/AUTH_uuid/notvalid" -- we just take the leading part of the domain and use that as the bucket | 16:00 |
| timburke | not that you can't do anonymous access, but presigned URLs will work just fine | 16:00 |
| timburke | note* that ... | 16:00 |
| *** mikecmpbll has quit IRC | 16:06 | |
| *** ccamacho has quit IRC | 16:06 | |
| *** mikecmpbll has joined #openstack-swift | 16:10 | |
| *** gyee has joined #openstack-swift | 16:18 | |
| *** rpittau is now known as rpittau|afk | 17:08 | |
| *** mikecmpbll has quit IRC | 17:32 | |
| *** rdejoux has quit IRC | 17:33 | |
| *** rdejoux_ is now known as rdejoux | 17:33 | |
| viks___ | timburke: What i see in my set up is, if i make call via my haproxy gateway where ssl gets terminated, i get the above error as mentioned. But if i make call directly to swift node(i.e. without ssl) it seems to be working.. I'm bit puzzled with this behaviour :( | 17:34 |
| viks___ | b/n i use stein.. | 17:35 |
| timburke | viks___, that was part of why i didn't notice the regression sooner -- per https://docs.aws.amazon.com/cli/latest/topic/s3-config.html#payload-signing-enabled "By default, this is disabled ..., but only if ... the endpoint uses HTTPS." | 17:37 |
| timburke | fwiw, i backported the fix to stein -- need to tag a stable release though | 17:38 |
| *** diablo_rojo__ has joined #openstack-swift | 17:42 | |
| viks___ | so it's the problem with s3api middleware? am i right? | 17:42 |
| timburke | yep. fixed for stein in https://github.com/openstack/swift/commit/6ed165c | 17:43 |
| timburke | will get you a 2.21.1 tag later this week, hopefully | 17:44 |
| viks___ | timburke: Thanks... will try to test this tomorrow | 17:45 |
| openstackgerrit | Tim Burke proposed openstack/swift stable/rocky: s3api: Allow clients to upload with UNSIGNED-PAYLOAD https://review.opendev.org/694121 | 17:50 |
| openstackgerrit | Tim Burke proposed openstack/swift stable/rocky: Authors/changelog for 2.19.2 https://review.opendev.org/694136 | 18:28 |
| *** tesseract has quit IRC | 18:35 | |
| *** diablo_rojo__ has quit IRC | 20:05 | |
| *** mwheckmann has quit IRC | 20:46 | |
| clayg | oh neat, i'll be able to catch most of the meeting with the time change | 20:50 |
| openstackgerrit | Thiago da Silva proposed openstack/swift master: WIP: Allow internal clients to use reserved namespace https://review.opendev.org/682138 | 20:52 |
| openstackgerrit | Thiago da Silva proposed openstack/swift master: New Object Versioning mode https://review.opendev.org/682382 | 20:52 |
| clayg | tdasilva: oh wow! did you make symlink_path cooler? | 20:58 |
| kota_ | mornin | 20:59 |
| mattoliverau | Morning | 20:59 |
| clayg | hey guys! | 21:00 |
| *** zaitcev has joined #openstack-swift | 21:00 | |
| *** ChanServ sets mode: +v zaitcev | 21:00 | |
| tdasilva | clayg: yeah, still struggling with string handling between py2 and py3. I just added some utf8 func tests, passed on my envs, hoping it passes on gate | 21:01 |
| clayg | i hate strings! | 21:01 |
| clayg | or bytes... i'm not sure | 21:01 |
| tdasilva | heh | 21:02 |
| zaitcev | timburke: are we having a meeting today? | 21:03 |
| zaitcev | Or is everyone still dead from Chinese food | 21:03 |
| * tdasilva wants some Chinese food | 21:03 | |
| timburke | lousy time to go installing the new fedora... | 21:04 |
| timburke | meeting time! | 21:04 |
| clayg | zaitcev: so many noodles | 21:05 |
| zaitcev | clayg: carbohydrates are hazardous if you're not used to them. | 21:06 |
| *** patchbot has quit IRC | 21:11 | |
| clayg | seongsoocho: we're all talking about you in the #openstack-meeting for Swift - we all gab every Wednesday around this time, but we noticed it's kind of early for you | 21:15 |
| clayg | ... maybe another time would work better!? | 21:15 |
| mattoliverau | or thursday morning in our side of the world ;) | 21:16 |
| *** patchbot has joined #openstack-swift | 21:17 | |
| *** rdejoux has quit IRC | 21:26 | |
| seongsoocho | clayg: hi ! i'm in utc+9. but It is ok. I can join a meeting in early morning. | 22:03 |
| timburke | seongsoocho, good to hear from you again! very good to meet you last week :-D | 22:04 |
| rledisez | seongsoocho: the one for this week is just done but you should totally join us next week. you can find some link here for the archive and co :https://wiki.openstack.org/wiki/Meetings/Swift | 22:04 |
| seongsoocho | timburke: Hi ! I also very good to meet you last week :) | 22:06 |
| seongsoocho | rledisez: Wow, thanks I will read today meeting log. | 22:06 |
| mattoliverau | seongsoocho: morning o/ | 22:13 |
| seongsoocho | mattoliverau: morning :-) | 22:14 |
| *** rcernin has joined #openstack-swift | 22:32 | |
| *** tkajinam has joined #openstack-swift | 23:06 | |
| *** diablo_rojo__ has joined #openstack-swift | 23:33 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!