| *** tdasilva has joined #openstack-swift | 00:26 | |
| *** ChanServ sets mode: +v tdasilva | 00:26 | |
| *** m75abrams has quit IRC | 00:37 | |
| *** openstackgerrit has quit IRC | 00:58 | |
| *** m75abrams has joined #openstack-swift | 01:06 | |
| *** openstackgerrit has joined #openstack-swift | 02:05 | |
| openstackgerrit | Merged openstack/swift master: Pin Bandit on py2 https://review.opendev.org/c/openstack/swift/+/765883 | 02:05 |
|---|---|---|
| *** paladox has quit IRC | 02:10 | |
| *** paladox has joined #openstack-swift | 02:15 | |
| *** psachin has joined #openstack-swift | 03:16 | |
| *** rcernin has quit IRC | 03:46 | |
| *** rcernin has joined #openstack-swift | 03:50 | |
| *** m75abrams has quit IRC | 04:06 | |
| *** m75abrams has joined #openstack-swift | 04:08 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-swift | 05:33 | |
| *** rcernin has quit IRC | 05:39 | |
| *** benj_ has quit IRC | 06:04 | |
| *** benj_ has joined #openstack-swift | 06:05 | |
| *** rcernin has joined #openstack-swift | 06:08 | |
| *** rcernin has quit IRC | 06:45 | |
| *** timburke has quit IRC | 06:55 | |
| *** rcernin has joined #openstack-swift | 07:01 | |
| *** lxkong has quit IRC | 07:09 | |
| *** rcernin has quit IRC | 07:25 | |
| *** gyee has quit IRC | 07:37 | |
| *** rpittau|afk is now known as rpittau | 08:10 | |
| openstackgerrit | Merged openstack/swift master: Give unit tests a second chance to pass https://review.opendev.org/c/openstack/swift/+/765589 | 08:28 |
| *** baojg has quit IRC | 08:44 | |
| *** lxkong has joined #openstack-swift | 09:00 | |
| *** baojg has joined #openstack-swift | 09:05 | |
| *** baojg has quit IRC | 09:08 | |
| *** baojg has joined #openstack-swift | 09:09 | |
| *** mvalsecc has quit IRC | 09:13 | |
| *** dsariel has quit IRC | 09:22 | |
| *** baojg has quit IRC | 09:57 | |
| *** baojg has joined #openstack-swift | 09:58 | |
| *** dsariel has joined #openstack-swift | 10:46 | |
| *** dsariel has quit IRC | 10:54 | |
| *** dsariel has joined #openstack-swift | 10:54 | |
| *** baojg has quit IRC | 12:07 | |
| *** baojg has joined #openstack-swift | 12:07 | |
| *** dsariel has quit IRC | 12:40 | |
| *** dsariel has joined #openstack-swift | 12:40 | |
| *** dsariel has quit IRC | 13:27 | |
| *** dsariel has joined #openstack-swift | 13:28 | |
| *** tdasilva has quit IRC | 13:48 | |
| *** tdasilva_ has joined #openstack-swift | 13:48 | |
| *** ChanServ sets mode: +v tdasilva_ | 13:48 | |
| *** tkajinam has quit IRC | 14:47 | |
| *** tkajinam has joined #openstack-swift | 14:47 | |
| *** tdasilva_ is now known as tdasilva | 14:56 | |
| *** tkajinam has quit IRC | 15:18 | |
| *** gyee has joined #openstack-swift | 15:58 | |
| *** klamath_atx has joined #openstack-swift | 16:21 | |
| klamath_atx | I took a break from the tempauth for a few days and picked it back up today. Wanted to test it out cluster wide and running into issues trying to use tempauth + keystone + swift3. If someone has the time can you take a look at these errors? https://pastebin.com/iyWHQ2Ha | 16:23 |
| klamath_atx | I can get either keystone working, or tempauth, but not both at the same time, very odd error on the proxy head when authing with keystone via swift cli | 16:24 |
| klamath_atx | Dec 8 16:19:21 overcloud-controller-0 proxy-server: ERROR Unhandled exception in request: #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/swift/proxy/server.py", line 511, in handle_request#012 resp = req.environ['swift.authorize'](req)#012 File "/usr/lib/python2.7/site-packages/swift/common/middleware/tempauth.py", line 559, in authorize#012 user_groups = (req. | 16:24 |
| klamath_atx | remote_user or '').split(',')#012AttributeError: 'tuple' object has no attribute 'split' (txn: tx89bdc00667704870bccd0-005fcfa789) (client_ip: 172.16.1.7) | 16:24 |
| klamath_atx | Dec 8 16:19:21 overcloud-controller-0 proxy-server: 172.16.1.7 172.16.1.7 08/Dec/2020/16/19/21 GET /v1/AUTH_b8556435276141c6a6370b1abace2068%3Fformat%3Djson HTTP/1.0 500 - python-swiftclient-3.6.1 gAAAAABfz6eHnwE9... - 125 - tx89bdc00667704870bccd0-005fcfa789 - 0.0050 - - 1607444361.791141987 1607444361.796123981 - | 16:24 |
| *** timburke has joined #openstack-swift | 16:31 | |
| *** ChanServ sets mode: +v timburke | 16:31 | |
| *** psachin has quit IRC | 16:38 | |
| *** m75abrams has quit IRC | 16:57 | |
| *** rpittau is now known as rpittau|afk | 17:43 | |
| *** tdasilva_ has joined #openstack-swift | 18:13 | |
| *** ChanServ sets mode: +v tdasilva_ | 18:13 | |
| *** tdasilva has quit IRC | 18:16 | |
| *** tdasilva_ has quit IRC | 18:19 | |
| *** tdasilva_ has joined #openstack-swift | 18:19 | |
| *** ChanServ sets mode: +v tdasilva_ | 18:19 | |
| timburke | klamath_atx, sorry for the delay -- are both keystone and tempauth using the same reseller_prefix? i think you should be able to get them to work in the same cluster as long as they each have their own account namespace to work under | 18:32 |
| timburke | reminds me that i ought to rebase https://review.opendev.org/c/openstack/swift/+/630415 ... | 18:34 |
| klamath_atx | gotcha, so i should create another auth prefix for tempauth like KEY_ vs AUTH_ for keystone? | 18:39 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Fix the handling of keystone groups in tempauth. https://review.opendev.org/c/openstack/swift/+/630415 | 18:50 |
| timburke | klamath_atx, yep! then each authmiddleware will handle requests just within its own prefix (iirc) | 18:51 |
| klamath_atx | perfect, thank you, will try that | 18:59 |
| *** ianychoi__ has quit IRC | 19:44 | |
| *** renich has joined #openstack-swift | 19:46 | |
| renich | Good time_of_day! o/ | 19:47 |
| renich | I am trying to configure my minio client to be used with a test openstack swift + keystone deployment. I've configured keystonemiddleware and added s3api and s3token to my configuration. I've, also, configured auth_uri and www_authenticate_uri to my keysotne server. Still, this doesn't seem to work for some reason. | 19:49 |
| renich | I get this in the logs: https://paste.centos.org/view/387cc6e9 | 19:50 |
| renich | here're my relevant configs: https://gitlab.com/-/snippets/2047914 | 19:56 |
| timburke | renich, i'm surprised you just get the one log line for the request with log_level=DEBUG -- or maybe proxy.log is apache's logging and not swift's? i wonder where swift's logs are going... | 20:21 |
| timburke | do the backend servers have any requests logged? | 20:22 |
| openstackgerrit | Alistair Coles proposed openstack/swift master: swift-manage-shard-ranges: add 'compact' command https://review.opendev.org/c/openstack/swift/+/765623 | 20:56 |
| renich | timburke: Yeah, the proxy.log is just the access log for the requests. I've renamed it.... | 20:57 |
| renich | timburke: what do you mean backend servers? apache2? | 20:58 |
| timburke | the account/container servers | 20:58 |
| renich | ah, I haven't checked those. Those log to journald | 20:58 |
| renich | let me check | 20:58 |
| renich | timburke: I'm getting this: https://paste.centos.org/view/c465baff | 20:59 |
| renich | odd... it's authorizing as anonymous? | 21:00 |
| renich | And it's trying to find some file that isn't there... | 21:00 |
| renich | ... man, I had forgotten all about journald... :S | 21:01 |
| timburke | interesting... the "Not a path query" message seems to be coming from https://opendev.org/openstack/swift/src/tag/2.26.0/swift/common/middleware/s3api/s3token.py#L255 ... which would seem to mean that s3api didn't parse out any auth info (it would look at the Authorization header and query params) | 21:03 |
| timburke | (normally, there'd be an access key identifier that s3api then stuffs into the request path as though it were the swift account, and then s3token would re-write it to be the correct account) | 21:05 |
| renich | I'm using minio's client: mc. I've generated the access and secret keys with openstack ec2 credentials create | 21:06 |
| renich | And I'm pretty sure they're correct... double-checking now | 21:06 |
| timburke | even if they were wrong, i'd expect a 403, not 404 | 21:07 |
| renich | OK | 21:08 |
| renich | They're correct, though. | 21:08 |
| timburke | if you can get some verbose client logs, that'd probably be the next thing i check. or find a way to separate the TLS-terminating from the WSGI-serving in apache so i could peek at the on-the-wire request | 21:09 |
| renich | OK | 21:09 |
| renich | timburke: https://paste.centos.org/view/0848c856 | 21:09 |
| timburke | great! so there *should be* an `Authorization: AWS4-HMAC-SHA256 Credential=...` header coming across -- why isn't s3api finding it and parsing it? maybe try adding some debug logging around https://github.com/openstack/swift/blob/2.26.0/swift/common/middleware/s3api/s3request.py#L500-L517 to make sure it's getting detected as a v4 sigature? | 21:16 |
| renich | timburke: OK, any tips on how to add the debug logging? how do I add the logging? | 21:17 |
| renich | My current mc config is this: https://paste.centos.org/view/81ee48d5 | 21:21 |
| timburke | in a pinch, print statements should work; a better solution would probably be to add a new arg to the function and pass self.logger across from https://github.com/openstack/swift/blob/master/swift/common/middleware/s3api/s3api.py#L286 -- that's the only caller | 21:21 |
| renich | OK | 21:24 |
| renich | let me try that | 21:24 |
| timburke | weird... i can get mc to work in my dev env... looks like i need "path" set to "auto" or "on" though; "off" causes it to hang while trying to list :-/ | 21:30 |
| timburke | you might try forcing it to "on" rather than "auto" | 21:31 |
| renich | OK, I'll force it | 21:42 |
| renich | was trying out the print statements but I made a mess, hehe. In some cases, it loops through the print many times and returns a 500 error | 21:42 |
| renich | ``` | 21:42 |
| renich | def get_request_class(env, s3_acl): | 21:42 |
| renich | """ | 21:42 |
| renich | Helper function to find a request class to use from Map | 21:42 |
| renich | """ | 21:42 |
| renich | if s3_acl: | 21:42 |
| renich | request_classes = (S3AclRequest, SigV4S3AclRequest) | 21:42 |
| renich | else: | 21:42 |
| renich | request_classes = (S3Request, SigV4Request) | 21:42 |
| renich | print(request_classes) | 21:42 |
| renich | req = swob.Request(env) | 21:42 |
| renich | print(req) | 21:42 |
| renich | Same result when setting path to "on" | 21:42 |
| renich | So, it's working for you timburke, so we know I have an issue somewhere... | 21:43 |
| renich | The auth isn't taking place, it seems. Also, for some reason, I am not getting any logs from keystone | 21:44 |
| renich | I mean, if I use openstack container list, stuff works | 21:45 |
| timburke | yeah, it seems to be an issue somewhere in the s3api middleware in particular -- seems like it's failing to parse the authorization header so it passes the request on verbatim; s3token then doesn't find what it needs to make it go contact keystone, and *somewhere* the proxy ends up deciding that "GET /" should be served a 404 | 21:50 |
| *** thiago__ has joined #openstack-swift | 21:50 | |
| *** ChanServ sets mode: +v thiago__ | 21:50 | |
| renich | timburke: could it be that the issue is in apache? or the the proxy's wsgi? | 21:50 |
| *** sorrison has joined #openstack-swift | 21:51 | |
| timburke | ...potentially? i'd be surprised, but i'm also running out of other ideas. it's part of why i want to see what's making its way into the request env. fwiw, i added something like http://paste.openstack.org/show/800870/ and got some useful info in my logs (and no 500s) | 21:52 |
| *** dsariel has quit IRC | 21:52 | |
| renich | timburke: right on. Let me try that. | 21:53 |
| *** tdasilva_ has quit IRC | 21:53 | |
| timburke | (basically, http://paste.openstack.org/show/800871/. note that signatures etc will not be redacted) | 21:54 |
| renich | http://paste.openstack.org/show/800872/ | 21:55 |
| renich | I think tokien is throttling me, hehe. | 21:58 |
| renich | due to the flooding earlier | 21:58 |
| *** rcernin has joined #openstack-swift | 21:59 | |
| renich | I see the region is part of the credentials... I have a different region setup | 22:03 |
| timburke | https://stackoverflow.com/questions/26475885/authorization-header-missing-in-php-post-request makes me think apache may be stripping the Authorization header... | 22:06 |
| timburke | maybe s3api could also check env['REDIRECT_HTTP_AUTHORIZATION']? idk that there's any sort of standard for that, though. certainly, apache seems to be deviating from WSGI here... :-/ | 22:10 |
| *** thiago__ has quit IRC | 22:13 | |
| *** thiago__ has joined #openstack-swift | 22:14 | |
| *** ChanServ sets mode: +v thiago__ | 22:14 | |
| timburke | fwiw, i seem to see a bunch of rewrite rules like https://opendev.org/osf/openstackid/src/branch/master/public/.htaccess#L17-L19 about | 22:16 |
| *** tdasilva_ has joined #openstack-swift | 22:18 | |
| *** ChanServ sets mode: +v tdasilva_ | 22:18 | |
| *** thiago__ has quit IRC | 22:21 | |
| renich | timburke: yeah, I'll see if that fixes it. | 22:26 |
| renich | Oh, man. I am considering on migrating to Nginx... | 22:27 |
| renich | But, I'll give it a try anyway | 22:27 |
| renich | timburke: hah! that worked! | 22:31 |
| renich | man!!! | 22:31 |
| renich | I owe you a box of beers! | 22:31 |
| timburke | 🎉 | 22:31 |
| renich | oh man... I've been too long looking for the cause... | 22:33 |
| renich | I need to better my knowledge of apache/switch to nginx|caddy|unit... | 22:34 |
| renich | timburke: man, you have my gratitude and my best wishes for you! Thank you, very much. | 22:45 |
| timburke | happy to help! | 22:45 |
| renich | BTW, the fix was adding this to both apache configurations (keystone and swift-proxy): SetEnvIf Authorization "(.*)" HTTP_AUTHORIZATION=$1 | 22:47 |
| renich | It looks like this: https://paste.centos.org/view/5c8d86dc | 22:48 |
| renich | Here's it again; to prevent eye burning caused by bad indentation (mixed spaces and tabs): https://paste.centos.org/view/d205696f | 22:49 |
| timburke | nice! i should make sure our docs have that in there somewhere... | 22:59 |
| renich | timburke: Awesome. I can help out if you need. | 22:59 |
| timburke | fwiw, i don't think keystone will need the rewrite rule; we only need it because that's the header S3 uses; the swift <-> keystone protocol we more or less make up as we see fit | 22:59 |
| renich | Right on. I'll test and confirm | 23:00 |
| timburke | sure, that'd be great! i think we'd just want to update https://github.com/openstack/swift/blob/master/examples/apache2/account-server.template ? | 23:00 |
| timburke | er, *proxy*, not account | 23:01 |
| *** tkajinam has joined #openstack-swift | 23:01 | |
| renich | timburke: OK, so, can I do this using github or should we go through the review and stuff? | 23:01 |
| timburke | we only do patches through gerrit (https://review.opendev.org/) | 23:02 |
| renich | Right, I remember. I will submit it through it. | 23:03 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Give probe tests a second chance to pass https://review.opendev.org/c/openstack/swift/+/766104 | 23:24 |
| *** mvalsecc has joined #openstack-swift | 23:27 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Increase gate timeouts from 1hr to 1.5hr https://review.opendev.org/c/openstack/swift/+/766105 | 23:31 |
| *** hoonetorg has quit IRC | 23:36 | |
| *** hoonetorg has joined #openstack-swift | 23:40 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!