Friday, 2018-11-09

*** mvkr has joined #openstack-swift00:12
*** gyee has quit IRC00:16
openstackgerritMerged openstack/swift master: s3api: Add basic support for ?versions bucket listings  https://review.openstack.org/57583800:19
*** pcaruana has quit IRC00:25
kota_good morning01:06
kota_thanks timburke and tdasilva for addressing issues on p 57583801:07
patchbothttps://review.openstack.org/#/c/575838/ - swift - s3api: Add basic support for ?versions bucket list... (MERGED) - 4 patch sets01:07
timburkekota_: thanks for looking at it! it definitely improved from where i started with it01:07
mattoliveraukota_: morning01:30
openstackgerritMatthew Oliver proposed openstack/python-swiftclient master: Clean up work dir before running func tests  https://review.openstack.org/61147101:58
openstackgerritMatthew Oliver proposed openstack/python-swiftclient master: Stop leaking quite so many connections  https://review.openstack.org/57758401:58
mattoliverau^ Just a rebase01:58
*** nguyenhai_ has joined #openstack-swift04:16
*** nguyenhai has quit IRC04:19
*** openstackstatus has quit IRC04:59
*** openstack has joined #openstack-swift07:09
*** ChanServ sets mode: +o openstack07:09
*** pcaruana has joined #openstack-swift07:21
*** e0ne has quit IRC07:31
*** admin6 has joined #openstack-swift07:33
*** admin6 has quit IRC07:36
*** e0ne has joined #openstack-swift09:39
*** e0ne has quit IRC10:56
*** e0ne has joined #openstack-swift11:44
openstackgerritchenxiangui proposed openstack/swift master: Fix typo  https://review.openstack.org/61689012:10
openstackgerritchenxiangui proposed openstack/swift master: Fix typo  https://review.openstack.org/61689512:32
*** mikecmpbll has joined #openstack-swift14:53
*** jistr is now known as jistr|call16:00
clayghi folks16:11
rledisezgood morning clayg16:12
*** mikecmpbll has quit IRC16:15
*** mikecmpb_ has joined #openstack-swift16:15
*** jistr|call is now known as jistr16:17
*** gyee has joined #openstack-swift16:22
*** mikecmpb_ has quit IRC16:32
*** mikecmpbll has joined #openstack-swift16:34
*** e0ne has quit IRC16:35
*** portante has joined #openstack-swift16:38
portantecschwede, notmyname, tdasilva, clayg: hi folks16:39
portanteperhaps a quick question ... do you know of a web server technology that will serve the contents of a tar file just like a file system hierarchy could be served?16:40
portanteso say I had a tar ball in an object store, but wanted to browse it and look at a file or two in it?16:40
DHEthe problem with a tarball is there's no index. building a file list would require fetching the whole file. that's rather suboptimal16:55
*** mikecmpbll has quit IRC16:57
timburkemight be able to do something with range requests... but it'll be slow; each range will depend on having read the previous header...17:41
timburkecould be an interesting further application for joeljwright's https://review.openstack.org/#/c/499260/ though...17:42
patchbotpatch 499260 - swift - WIP: Add TLO middleware - 7 patch sets17:42
timburkewould still require have the constituent files already splayed out, though :-/17:43
*** d34dh0r53 has joined #openstack-swift17:46
openstackgerritTim Burke proposed openstack/python-swiftclient master: Stop leaking quite so many connections  https://review.openstack.org/57758417:55
*** e0ne has joined #openstack-swift18:09
openstackgerritTim Burke proposed openstack/python-swiftclient master: Update reno for stable/rocky  https://review.openstack.org/58582318:25
*** e0ne has quit IRC18:33
portanteDHE, timburke, thanks!18:33
*** e0ne has joined #openstack-swift18:36
*** bigdogstl has joined #openstack-swift18:50
*** MooingLe1ur is now known as MooingLemur18:53
*** e0ne has quit IRC18:58
*** bigdogstl has quit IRC19:08
*** bigdogstl has joined #openstack-swift19:12
*** zigo has quit IRC19:25
*** bigdogstl has quit IRC19:26
*** bigdogstl has joined #openstack-swift19:30
*** bigdogstl has quit IRC19:35
openstackgerritMerged openstack/swift master: py3: port account/container replicators  https://review.openstack.org/61465619:43
*** bigdogstl has joined #openstack-swift19:53
*** bigdogstl has quit IRC19:57
*** e0ne has joined #openstack-swift20:01
*** e0ne has quit IRC20:04
*** anovaes has joined #openstack-swift20:11
anovaesHi people, I'm having a problem using swift with swift3 on the openstack pike.20:16
anovaesI can use swift with swift api normaly, but with s3 api a get problem.20:16
anovaesThe short message is: SignatureDoesNotMatch20:16
anovaesCan anybody help me?20:16
*** edmilsonssa has joined #openstack-swift20:17
notmynameanovaes: what s3 client are you using?20:26
*** edmilsonssa has quit IRC20:26
anovaesnotmyname: using aws, s3curl and s3cmd20:27
anovaesawscli*20:27
timburkedo you know what region your client is using? does it match the `location` configured for swift3?20:28
timburkeadding a --debug flag to the command line is probably going to be helpful, too20:30
notmynameah, good call. didn't older versions of swift3 default to a non-standard aws region name?20:30
timburkenotmyname: still do: https://github.com/openstack/swift/blob/2.19.0/swift/common/middleware/s3api/s3api.py#L10920:30
notmynamebah! I thought we changed that20:31
notmynamewe only changed it in our product, didn't we?20:31
timburkeiirc we had some reason to think it would work OK for s3cmd, though? fwiw, one of the things i pretty much always do is switch that to us-east-120:31
*** edmilsonssa_ has joined #openstack-swift20:32
anovaestimburke: my region is RegionOne, I have keystone and swift, only. The region was configured on filter s3token in proxy-server.conf20:33
timburkeswift3 has its own idea of region, configured with the option `location` :-/20:34
timburkeit gets used as part of the v4 signing process20:34
edmilsonssa_@timburke I thought Keystone region and Swift location should match, doesn't?20:35
timburkefwiw, after the transition to s3api we landed some patches to improve SignatureDoesNotMatch messages, and provide separate errors if the region doesn't match -- but that's newer swift20:35
timburkeedmilsonssa_: maybe? they certainly don't have to, though. if you don't explicitly configure swift3's location, it defaults to "US"20:37
edmilsonssa_timburke: ok, and what else could lead to SignatureDoesNotMatch ?20:39
timburkeswift in general (as opposed to the swift3/s3api middlewares) doesn't really expose "region" to the client... it's just a layer in the failure-domain hierarchy, and not necessarily tied to keystone's notions of regions20:40
edmilsonssa_ok, got it. but we are geting SignatureDoesNotMatch and we've have checked the config many times and didn't find anything suspicious. we're going nuts :)20:41
timburkebasically, either the user can't be found (so we don't have a secret to use in signing), the secrets don't match (so of course the signatures don't), or the string that's getting signed doesn't match (which can be a bug in our code, a bug in the client's code, or a misconfiguration like the S3-notion of region not matching)20:42
edmilsonssa_we've seen some people configuring the client with a combined string of project:user_id. does it make sense?20:42
timburkemakes sense for something like tempauth, but as i recall keystone's ec2 credentials (which we piggy-back off of) don't usually follow that form...20:43
timburkewhat do your swift logs say? maybe turn on debug logging and restart the proxy server first. we should be able to see what we're sending to keystone and what response we're getting back at debug level20:44
edmilsonssa_we are passing access and secret, we don't use tempauth in our pipeline config20:45
edmilsonssa_good, how do we enable debug for this?20:46
timburkein the [DEFAULT] section of proxy-server.conf, set log_level=DEBUG20:46
anovaesOK20:47
edmilsonssa_let's try20:47
anovaeswe go configure20:47
anovaes2018-11-09 17:51:25.732 4903 INFO keystone.common.wsgi [req-85897ae5-1b74-4478-b701-4d78a3efe505 f35a85b996a54e17af1a305695340fa3 f8b8cba91c654277832b432ddf55ddcb - default default] GET http://keystone.ip:35357/v3/users/f35a85b996a54e17af1a305695340fa3/credentials/OS-EC220:55
anovaes2018-11-09 17:52:01.160 4904 INFO keystone.common.wsgi [req-1372fcb3-5e66-426d-b3ed-d37741b1c97b - - - - -] POST http://my.keystone.ip::35357/v2.0/s3tokens20:55
anovaes2018-11-09 17:52:01.169 4904 WARNING keystone.common.wsgi [req-1372fcb3-5e66-426d-b3ed-d37741b1c97b - - - - -] Authorization failed. Credential signature mismatch (Disable insecure_debug mode to suppress these details.) from my.swift-proxy.ip: Unauthorized: Credential signature mismatch (Disable insecure_debug mode to suppress these details.)20:55
*** bigdogstl has joined #openstack-swift20:59
edmilsonssa_timburke: Nov  9 18:05:56 swfpxd001 proxy-server: {'access_key': 'c564e802a5d849b99b180593e5520ea0', 'account': None, '_timestamp': 1541797556.00000_0000000000000000, 'user_id': None, 'slo_enabled': True, 'container_name': '', 'string_to_sign': 'GET\n\napplication/x-www-form-urlencoded\nSex, 09 Nov 2018 21:05:56 +0000\n/', '_params_cache': {}, 'headers': <swift.common.swob.HeaderEnvironProxy object at 0x7f64f9d62e90>,21:08
*** bigdogstl has quit IRC21:09
*** bigdogstl has joined #openstack-swift21:13
anovaesCan charset is a problem?21:14
timburkehmm... access key looks like a keystone ec2 cred... string_to_sign looks sane for a v2 request, which wouldn't need to think much about regions...21:18
*** bigdogstl has quit IRC21:18
timburkethe secret key the clent's using is the one you got back when creating the credential, yeah? not the keystone user's password?21:19
edmilsonssa_yes, it's the ec2 secret21:20
edmilsonssa_I can see it using "openstack ec2 credentials list"21:20
timburkedo the swift logs have something like https://github.com/openstack/swift3/blob/master/swift3/s3_token_middleware.py#L270-L271 ? "Connecting to Keystone sending ..."21:22
edmilsonssa_yes21:24
edmilsonssa_Nov  9 18:05:56 swfpxd001 proxy-server: Connecting to Keystone sending this JSON: {"credentials": {"access": "c564e802a5d849b99b180593e5520ea0", "token": "R0VUCgphcHBsaWNhdGlvbi94LXd3dy1mb3JtLXVybGVuY29kZWQKU2V4LCAwOSBOb3YgMjAxOCAyMTowNTo1NiArMDAwMAov", "signature": "thJHlKHic/wKX4ECBxYp9/AE51I="}} (txn: tx3e2a6c83f2604eccabb72-005be5f6b4)21:24
timburkeso you can try manually doing what keystone would be doing, with something like `import base64, hashlib, hmac; print base64.b64encode(hmac.new('your key', base64.b64decode('R0VUCgphcHBsaWNhdGlvbi94LXd3dy1mb3JtLXVybGVuY29kZWQKU2V4LCAwOSBOb3YgMjAxOCAyMTowNTo1NiArMDAwMAov'), hashlib.sha1).digest())`21:28
timburke(substituting "your key" for your actual key, of course)21:28
timburkeand you can also try to get the debug output for the client, and checking that the client's notion of the "string to sign" matches what's being received21:29
timburkeone thing that seems odd to me is the date: "Sex, 09 Nov 2018 21:05:56 +0000"21:30
timburkebut if that's what the client's using too, it shouldn't be a problem...21:30
edmilsonssa_date match between proxy, keystone and client21:32
edmilsonssa_let me test manualy21:32
timburkecool; i was just expecting something more like "Fri, 09 Nov 2018 21:05:56 +0000"21:33
edmilsonssa_that's why our server is installed with brazilian language settings ...21:35
edmilsonssa_may this be a problem?21:35
timburkecould be... though i thought we just took what the client provided when building the string to sign... so if the client sent a portuguese date header, i'd expect it to have signed with a portuguese date header in the string-to-sign...21:37
edmilsonssa_I've made it manually and the signatures doesn't match ..21:38
timburkemight be worth trying with something like `base64.b64encode(hmac.new('your key', base64.b64decode('R0VUCgphcHBsaWNhdGlvbi94LXd3dy1mb3JtLXVybGVuY29kZWQKU2V4LCAwOSBOb3YgMjAxOCAyMTowNTo1NiArMDAwMAov').replace('Sex,', 'Fri,'), hashlib.sha1).digest())`21:39
edmilsonssa_yes, client sends: string_to_sign': 'GET\n\n\nSex, 09Nov201820: 55: 01+0000\n/21:39
timburkenote the replace(...)21:39
edmilsonssa_replace(...) where?21:40
edmilsonssa_got it....21:40
edmilsonssa_desn't match either ...21:42
edmilsonssa_:(21:42
edmilsonssa_signature sent by client is: "signature": "sMVKbDrZckblWs9l6CUQMYDJ3EE="21:42
edmilsonssa_calculated is wPys9vuUdv25tS6IcqkrMc2z0J0=21:42
edmilsonssa_or RPxckYM7Xb1oTbKUNeCeCSMd05M= w/o replace21:43
edmilsonssa_keystone log says "CredentialNotFound: Could not find credential"21:45
timburkefwiw, when i use awscli on my machine with a --debug flag, i get output like http://paste.openstack.org/show/734501/21:45
timburkeburied in there is the StringToSign, toward the end21:46
timburkeyou could compare the sts you see in the cli against the sts you see in the log, try to figure out what was different (if anything)21:47
timburkei forget whether keystone honestly can't find the credential at all or just can't find a credential *that can produce a valid signature* when it logs that...21:49
edmilsonssa_keystone log is:21:50
edmilsonssa_2018-11-09 18:50:04.196 4905 WARNING keystone.common.wsgi [req-bc6a8b94-b7a5-476b-be30-7f0852f265c4 - - - - -] Authorization failed. Credential signature mismatch (Disable insecure_debug mode to suppress these details.) from 10.10.1.158: Unauthorized: Credential signature mismatch (Disable insecure_debug mode to suppress these details.)21:50
timburkeyeah, that makes it seem more like it could find the credential, but got a mismatch (which we manually verified is in some sense "expected")21:52
edmilsonssa_yeah, but can't figure out why they doesn't match. do the time in both servers need to be equal?21:55
edmilsonssa_we have 8 sec of diff21:56
timburkethey should be similar, but don't have to be exact. 8s should be fine; iirc if it hits 5min you'll start seeing RequestTimeTooSkewed errors21:56
timburkecan we get a client debug log, along with the proxy-server logs for the same time? i think the next step is to compare the string-to-sign from each side manually21:58
edmilsonssa_yes21:59
edmilsonssa_tim, anovaes sent to you in pvt...22:09
edmilsonssa_we are going to put in the paste....22:10
edmilsonssa_better22:10
timburkeedmilsonssa_: i was just going to recommend that :-)22:10
anovaesHi22:11
anovaeshttp://paste.openstack.org/show/734502/22:11
anovaeslog client and server proxy22:11
timburkehttpt? should just be http, most likely?22:12
anovaessorry, one minute22:14
timburkeand yeah, looks like swift3 either doesn't have location set, or has it set to it's default of "US" -- you could try changing the client to use that too, or configure location=RegionOne in the [filter:swift3] section of proxy-server.conf22:14
timburke(based on the 403 that showed up in the proxy log)22:15
timburkejust after "Calling Swift3 Middleware" there's a long line that includes `'string_to_sign': 'AWS4-HMAC-SHA256\n20181109T220825Z\n20181109/US/s3/aws4_request\n641722ee6c367ce8eb5af0ad3501e581ad7a0e0bb997bd3719384ec426f6e279'`22:17
timburkethat 20181109/US/s3/aws4_request is the giveaway22:17
timburkestill not sure why the v2 signature wasn't working though :-/22:18
timburkefwiw, you can force different signature versions via config, as at https://github.com/tipabu/swift3-demo/blob/master/.aws/config#L1022:19
timburkeand with awscli-plugin-endpoint, you can avoid needing to put --endpoint-url on the command line all the time22:20
anovaessorry, we need change de desk22:25
timburkeno worries :-)22:26
anovaesthis new log http://paste.openstack.org/show/734503/22:30
*** edmilsonssa has joined #openstack-swift22:31
timburkelooks like we need a bit more of the server logs -- maybe try grepping for txf2d005b1812442e0b88f3 ?22:32
timburkeserver log got cut off at 19:22:03, while the client request was ~19:26:2522:32
anovaesgrep: http://paste.openstack.org/show/734505/22:39
timburke:-/ StringToSign looks right...22:41
timburkeoh, hey, yeah! "Keystone Reply: Status: 200"22:42
timburkeso why the 403...?22:42
timburkeis s3_acl enabled? what's your [filter:keystoneauth] section look like?22:44
edmilsonssa_maybe because we've put location in right place ...22:46
edmilsonssa_it was in s3token before ...22:46
anovaesendpoint do swift on the keystone is http://swif-proxy.ip:8080/v122:47
anovaeshave problem?22:47
timburkeis there a reseller prefix configured for s3token? does it match the one (if any) for keystoneauth?22:47
timburkenope, that should be fine -- at this point, i think we've eliminated the true mismatch problem (keystone responds 200, yay!) but we have some other authorization problem that still returns a SignatureDoesNotMatch error to the client22:48
timburkeoh, and it looks like reseller_prefix in s3token may need to have a trailing underscore... i think we've got some niceties in keystoneauth and tempauth to take things like AUTH and automatically add an underscore, but not so with s3token...22:51
*** bigdogstl has joined #openstack-swift22:51
edmilsonssa_[filter:keystoneauth] has no reseller, but [filter:s3token] has reseller = AUTH22:54
timburketry adding an underscore to the end22:58
openstackgerritTim Burke proposed openstack/swift master: s3token: Add an underscore to the end of reseller_prefix if not present  https://review.openstack.org/61704323:00
*** bigdogstl has quit IRC23:03
edmilsonssa_we've added. anovaes gonna send paste23:03
anovaesLog client: http://paste.openstack.org/show/734507/23:04
*** bigdogstl has joined #openstack-swift23:05
timburkenext bit of debugging i'd add: in [filter:swift3], set `force_swift_request_proxy_log = true` and we can dig into what backend requests we're making a bit, make sure we're going to the account that we were expecting23:07
*** mikecmpbll has joined #openstack-swift23:08
*** bigdogstl has quit IRC23:10
*** bigdogstl has joined #openstack-swift23:11
edmilsonssa_set it and got http://paste.openstack.org/show/734509/23:16
openstackgerritMerged openstack/python-swiftclient master: Update reno for stable/rocky  https://review.openstack.org/58582323:18
openstackgerritMerged openstack/python-swiftclient master: Stop leaking quite so many connections  https://review.openstack.org/57758423:18
timburke:-/ so why the 401 on GET /v1/AUTH_f8b8cba91c654277832b432ddf55ddcb ? hmm...23:19
edmilsonssa_f8b8cba91c654277832b432ddf55ddcb is Project ID, is it right?23:20
timburkeyeah23:20
edmilsonssa_shoudn't be Access Key?23:21
timburkenope23:21
edmilsonssa_ok23:21
timburkedoes the user work via the Swift API?23:21
edmilsonssa_yes23:23
timburkeie, keystoneauth includes "admin" in the list of operator_roles, and knows that user with id f35a85b996a54e17af1a305695340fa3 can access AUTH_f8b8cba91c654277832b432ddf55ddcb?23:23
timburkegood...23:23
timburkehmm23:23
*** bigdogstl has quit IRC23:24
edmilsonssa_[filter:keystoneauth] has operator_roles = admin, user23:24
timburkeah! proxy-server pipeline needs adjusting23:24
timburkei saw New pipeline is "catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk ratelimit authtoken keystoneauth copy swift3 s3token container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server" in one of the pastes23:25
edmilsonssa_it is set as pipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk ratelimit authtoken keystoneauth swift3 s3token container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server23:25
timburkewe want keystoneauth after (to right of) s3token23:25
edmilsonssa_done, testing23:26
*** bigdogstl has joined #openstack-swift23:27
edmilsonssa_seens like it's working http://paste.openstack.org/show/734510/23:30
timburke\o/23:30
edmilsonssa_and I own you a botle of beer ...23:30
edmilsonssa_\o/23:30
timburke:-)23:30
edmilsonssa_Thkank you a lot!!!23:30
timburkehappy to help!23:31
edmilsonssa_we where about to reinstall everything ...23:31
timburkeedmilsonssa_: anovaes: out of curiosity, what are you guys using swift for?23:31
edmilsonssa_thank you in name of ma colegue anovaes (who had an emergency with is dog broken leg).23:32
*** bigdogstl has quit IRC23:32
timburkeeep! hope his dog's doing better23:32
edmilsonssa_we are starting a poc for Bahia Court House here in Brazil ...23:32
edmilsonssa_if it goes right, we will use it for one of our systems ...23:33
timburkecool! good luck with Swift, let us know if you need more help!23:34
edmilsonssa_and if it performs well maybe other court houses in Brazil will start unsing something similar or even Swift ....23:34
edmilsonssa_Ok, thank you, We'll let you know ...23:35
edmilsonssa_bye, it's late here ...23:35
timburkegood night23:36
*** mikecmpbll has quit IRC23:36
*** edmilsonssa has quit IRC23:39
*** edmilsonssa_ has quit IRC23:40
*** bigdogstl has joined #openstack-swift23:43
openstackgerritTim Burke proposed openstack/swift master: s3api: Change default location to us-east-1  https://review.openstack.org/61704623:52
*** bigdogstl has quit IRC23:54

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!