| *** thurloat_isgone is now known as thurloat | 00:00 | |
| ho | good morning! | 00:01 |
|---|---|---|
| mattoliverau | ho morning | 00:03 |
| ho | mattoliverau: morning! | 00:03 |
| ho | Brian, around? | 00:04 |
| *** shakamunyi has joined #openstack-swift | 00:12 | |
| *** barra204 has joined #openstack-swift | 00:12 | |
| *** jkugel has joined #openstack-swift | 00:14 | |
| ho | all, do you know Brian Cline's IRC name and time zone? | 00:19 |
| *** remix_tj has quit IRC | 00:21 | |
| *** remix_tj has joined #openstack-swift | 00:23 | |
| *** thurloat is now known as thurloat_isgone | 00:28 | |
| *** asettle has quit IRC | 00:42 | |
| *** breitz has quit IRC | 00:45 | |
| *** breitz1 has joined #openstack-swift | 00:45 | |
| *** bkopilov has quit IRC | 00:58 | |
| *** bkopilov has joined #openstack-swift | 01:08 | |
| *** superfly_ has joined #openstack-swift | 01:12 | |
| *** superfly_ is now known as superflyy | 01:14 | |
| ho | ping bcline | 01:21 |
| tdasilva | ho: I think it's briancline | 01:21 |
| ho | tdasilva: briancline is right one? | 01:22 |
| tdasilva | ho: I believe so, but I think he is in the US | 01:23 |
| tdasilva | so he might not be available around this time | 01:23 |
| ho | tdasilva: thanks! it's difficult to get in touch with him because of our time zone ... | 01:23 |
| *** barra204 has quit IRC | 01:24 | |
| *** shakamunyi has quit IRC | 01:24 | |
| *** superflyy has quit IRC | 01:24 | |
| *** blmartin has quit IRC | 01:34 | |
| *** nadeem has joined #openstack-swift | 01:50 | |
| *** kota_ has joined #openstack-swift | 01:52 | |
| kota_ | morning | 01:53 |
| ho | kota_: morning! | 01:53 |
| kota_ | ho: :) | 01:56 |
| *** nadeem has quit IRC | 01:59 | |
| *** kota_ has quit IRC | 02:02 | |
| *** wbhuber has quit IRC | 02:15 | |
| *** jrichli has joined #openstack-swift | 02:22 | |
| *** haigang has quit IRC | 02:26 | |
| *** haigang has joined #openstack-swift | 02:27 | |
| *** annegentle has joined #openstack-swift | 02:28 | |
| *** haigang has quit IRC | 02:30 | |
| *** haigang has joined #openstack-swift | 02:32 | |
| *** haigang has quit IRC | 02:33 | |
| *** haigang has joined #openstack-swift | 02:34 | |
| *** haigang has quit IRC | 02:38 | |
| *** annegentle has quit IRC | 02:44 | |
| *** haigang has joined #openstack-swift | 02:46 | |
| *** annegentle has joined #openstack-swift | 02:51 | |
| *** annegentle has quit IRC | 02:51 | |
| *** Ciipher has quit IRC | 02:55 | |
| *** proteusguy has joined #openstack-swift | 02:58 | |
| *** gyee is now known as operator99 | 03:00 | |
| *** jkugel has quit IRC | 03:11 | |
| *** remix_tj has quit IRC | 03:35 | |
| *** remix_tj has joined #openstack-swift | 03:38 | |
| *** jlk has joined #openstack-swift | 03:41 | |
| jlk | halp! I'm stuck in a world of invalid keystone tokens and I can't get out! | 03:42 |
| jlk | I can't figure out the magic to get my swift proxy to auth with keystone. I can't figure out if it's the password that's bad or the user's token that's bad, or something else entirely | 03:42 |
| *** dimasot has joined #openstack-swift | 03:44 | |
| *** lcurtis_ has joined #openstack-swift | 03:48 | |
| ho | jlk: I try to solve your problem. but i have 1H meeting from now. could you tell me more detail of it? | 03:54 |
| jlk | ho: so I am running swift client, with debug, and I can see it hit keystone for a token | 03:54 |
| jlk | it takes that token and attempts to use it against the swift end point, but gets a 401 from that | 03:54 |
| jlk | INFO:swiftclient:REQ: curl -i https://tardis-swift.openstack.blueboxgrid.com:8090/v2/9d60928c9cde4b2ea488c8296a | 03:54 |
| jlk | 5707b9 -I -H "X-Auth-Token: 091a2dca7a054b6ca2accc48d78d9fac" | 03:54 |
| jlk | INFO:swiftclient:RESP STATUS: 401 Unauthorized | 03:54 |
| jlk | that transactoin ID shows this in the swift log | 03:56 |
| jlk | Jun 3 03:53:22 localhost.localdomain proxy-server: 173.247.104.24 10.230.6.16 03/Jun/2015/03/53/22 HEAD /v2/9d60928c9cde4b2ea488c8296a5707b9 HTTP/1.0 401 - - 091a2dca7a054b6c... - - - txb9cdeb3923e04a248f7d8-00556e7a32 - 0.0008 - - 1433303602.956839085 1433303602.957592010 | 03:56 |
| jlk | it /seems/ to me that while my client is able to get a token, it's swift-proxy unable to auth to keystone in order to validate tokens | 03:57 |
| *** jrichli has quit IRC | 04:03 | |
| ho | jlk: is it correct endpoint? it seems to use v2 api. | 04:04 |
| jlk | yes | 04:04 |
| jlk | endpoint is correct | 04:04 |
| jlk | I specified -V 2 in my swift command | 04:04 |
| ho | jlk: v2 fro | 04:06 |
| jlk | hrm? | 04:06 |
| ho | jlk: v2 for keystone works but not for swift | 04:06 |
| jlk | did it work at one point? | 04:07 |
| ho | jlk: i have to go to meeting now. | 04:07 |
| jlk | ho: but my point is, the error happens when swift client talks to swift proxy and gives over the token. Swift proxy ties to do something with it, to validate the token and that's when the 401 comes back | 04:09 |
| *** sorrison_laptop has joined #openstack-swift | 04:29 | |
| *** ppai has joined #openstack-swift | 04:40 | |
| *** lcurtis_ has quit IRC | 04:52 | |
| jlk | definitely appears to be a problem in the ssl stuff. If I point it directly at keystone without ssl things work. | 05:01 |
| ho | jlk: I'm back. If you don't use https for the communication, I think that the endpoint will be "http://tardis-swift.openstack.blueboxgrid.com:8090/v1/AUTH_%(tenant_id)s". | 05:07 |
| jlk | sorry, let me rephrase | 05:07 |
| jlk | whenI configure the authtoken middleware in the swift-proxy server to connect to keystone via http then everything works. | 05:07 |
| jlk | swift-proxy is able to validate the tokens. | 05:07 |
| jlk | but if I have swift-proxy connect to keystone via https (through our haproxy) then it gets 401s | 05:08 |
| *** Triveni has joined #openstack-swift | 05:08 | |
| ho | jlk: i see. in proxy-server log you wrote, there is "HEAD /v2/9d60928c9cde4b2ea488c8296a5707b9" message. So I think your endpoint registration to keystone seems to be wrong. | 05:16 |
| jlk | I switched it back to v1 (even though v2 was working in our havana version) | 05:16 |
| jlk | and even with v1 it is acting as I describe. http fine, https not fine | 05:16 |
| jlk | it's in how the proxy server is talking to keystone and ssl | 05:16 |
| *** kota_ has joined #openstack-swift | 05:22 | |
| *** proteusguy has quit IRC | 05:28 | |
| ho | jlk: is it possible to execute curl command for getting token from keystone? http://paste.openstack.org/show/258504/ | 05:35 |
| ho | jlk: then I would like to know authtoken info in proxy-server.conf (lines of identity_uri and auth_uri) | 05:37 |
| jlk | that token is kind of long, a big blob of json | 05:38 |
| jlk | so, auth_uri doesn't appear to actually be used, only identity_uri | 05:39 |
| *** SkyRocknRoll has joined #openstack-swift | 05:39 | |
| *** SkyRocknRoll has joined #openstack-swift | 05:39 | |
| ho | jlk: both commands work? could you paste the uri of the commands here? | 05:45 |
| jlk | well, I'm not sure what all from this token response I need to cpy. It's like 30 lines | 05:45 |
| ho | jlk: I need only your execution line. not output. | 05:45 |
| ho | jlk: s/execution line/executed command/ | 05:46 |
| jlk | if what ou want is to make sure I can see the endpoints with the user that swift proxy is configured to use, yes, I can. When I create a stackrc with those credentials, I can do a keystone catalog just fine | 05:47 |
| jlk | DEBUG:keystoneclient.auth.identity.v2:Making authentication request to https://tardis.openstack.blueboxgrid.com | 05:48 |
| jlk | :5001/v2.0/token | 05:48 |
| jlk | it gets the token, gets the catalog | 05:48 |
| jlk | curl -g -i -X GET https://tardis.openstack.blueboxgrid.com:35358/v2.0/endpoints -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}9d9ae562e6289b6 | 05:49 |
| jlk | 810e77836f5c304f25b6286cf" | 05:49 |
| jlk | endpoints come across fine | 05:49 |
| ho | jlk: can I see the endpoit of object-store and identity_uri in proxy-server.conf ? just make sure. | 05:50 |
| jlk | object-store endpiont is | 05:51 |
| jlk | | 0eb4fce9d14941b5b7d26de329a6dcad | RegionOne | https://tardis-swift.openstack.blueboxgrid.com:8090/v1/%(tenant_id)s | https://tardis-swift.openstack.blueboxgrid.com:8090/v1/%(tenant_id)s | https://tardis-swift.openstack.blueboxgrid.com:8090/v1/%(tenant_id)s | 9695a549a04044838fde970a31e3a4e9 | | 05:51 |
| jlk | right now, I have identity_uri as identity_uri = http://10.230.6.1:5000 which is the private IP of the keystone server, because it works. Normally it would be https://tardis.openstack.blueboxgrid.com:35358/v2.0 | 05:52 |
| openstackgerrit | Merged openstack/swift: Don't quarantine on read_metadata ENOENT https://review.openstack.org/187116 | 05:58 |
| ho | jlk: thanks! is it possible to try this: identity_uri = https://10.230.6.1:35358 | 05:58 |
| jlk | I can try, but I did try that. That port will hit the haproxy running on that host (we run haproxy on both our keystones). that will break. SSL is terminated at the haproxy. | 06:02 |
| jlk | Account GET failed: https://tardis-swift.openstack.blueboxgrid.com:8090/v1/5da25cc3853f4c54850898f9614c20bb?format=json 401 Unauthorized [first 60 chars of response] <html><h1>Unauthorized</h1><p>This server could not verify t | 06:03 |
| jlk | proxy log shows | 06:04 |
| jlk | Jun 3 06:03:20 localhost.localdomain proxy-server: Authorization failed for token | 06:04 |
| ho | jlk: i see. is it possible to change endpoint using AUTH_? your endpoint is not using AUTH_ for tenant name. https://tardis-swift.openstack.blueboxgrid.com:8090/v1/AUTH_%(tenant_id)s is there any reason? | 06:06 |
| jlk | Because it's not necessary? | 06:07 |
| jlk | We don't use it in other swift endpoints | 06:07 |
| ho | jlk: Does it work? | 06:07 |
| jlk | yes | 06:07 |
| ho | jlk: you use habana right? I will check the code of keystoneauth. | 06:08 |
| jlk | We were on havana, now we're on juno | 06:08 |
| jlk | I have another swift stack on juno as well with seemingly identical configuration and it's functional. I can't find the difference. | 06:09 |
| jlk | I'm giving up for the night, it's late and I have to be up early. Thank you for your help so far | 06:12 |
| ho | jlk: have a good night! Sorry I can not solve it... | 06:13 |
| *** shakamunyi has joined #openstack-swift | 06:16 | |
| openstackgerrit | Christian Schwede proposed openstack/swift: Fix testing issues https://review.openstack.org/187754 | 06:17 |
| *** barra204 has joined #openstack-swift | 06:17 | |
| ppai | cschwede, I noticed in my saio vm that "tox -e func" when none of the swift services are running returns a success. Does 187754 fix that ? | 06:19 |
| *** proteusguy has joined #openstack-swift | 06:20 | |
| *** proteusguy has quit IRC | 06:20 | |
| cschwede | ppai: yes, exactly - there are two issues here: 1. exception is catched and tox reports a success 2. missing python-keystoneclient, raising the exception | 06:20 |
| cschwede | ppai: i’m still investigating why openstack.nose_plugin catches the exception atm | 06:20 |
| ho | jlk: I assume you don't use user which has reseller_admin_role. If your endpoint is not started with AUTH_, you meet L394 in keystoneauth. https://github.com/openstack/swift/blob/stable/juno/swift/common/middleware/keystoneauth.py#L394 | 06:22 |
| ppai | cschwede, cool! | 06:22 |
| ho | jlk: So you need to put reseller_prefix for your endpoint at least in Juno. But if you use an user which is reseller_admin_role, it works. | 06:26 |
| *** sorrison_laptop has quit IRC | 06:27 | |
| openstackgerrit | Prashanth Pai proposed openstack/swift: Make object creation more atomic in Linux https://review.openstack.org/162243 | 06:29 |
| *** Trixboxer has quit IRC | 06:32 | |
| *** silor has joined #openstack-swift | 06:35 | |
| openstackgerrit | Christian Schwede proposed openstack/swift: Fix testing issues https://review.openstack.org/187754 | 06:35 |
| ho | kota_: around? | 06:38 |
| ppai | cschwede, your patch fixes it http://paste.openstack.org/show/258658/ | 06:38 |
| cschwede | ppai: cool, thanks for checking this! | 06:39 |
| kota_ | ho: hi | 06:44 |
| kota_ | I'm now reading scroll back recent irc channel logs... | 06:45 |
| ho | kota_: thanks for the review. as for the patch 187489, I think your understanding is not correct. | 06:46 |
| patchbot | ho: https://review.openstack.org/#/c/187489/ | 06:46 |
| kota_ | ho: ok, let me know... | 06:46 |
| *** asettle has joined #openstack-swift | 06:49 | |
| kota_ | ho: swift has constraints about max_meta_name_length, max_meta_value_length and max_meta_overall_size, right/ | 06:49 |
| kota_ | ? | 06:49 |
| ho | kota_: wait a minute | 06:50 |
| kota_ | ho: i think existing tests makes some kinds of metadata under constraint max_meta_overall_size and then filling up the rest of size by X-Account-Meta-k | 06:51 |
| kota_ | ho: ok, waiting | 06:51 |
| ho | kota_: sorry, I was caught by my boss... | 06:54 |
| kota_ | ho: no worries :) | 06:55 |
| ho | kota_: the post at L827 is first one with the header the size is 4096 | 06:55 |
| kota_ | ho: test/functional/test_account.py? | 06:56 |
| ho | kota_: the post at 840 is only update X-Account-Meta-k without any size change. | 06:56 |
| ho | kota_: yes | 06:57 |
| kota_ | ho: I still didn't get the purpose. | 06:58 |
| kota_ | ho: you mean we need an assertion for metadata updates? | 06:59 |
| kota_ | ho: but the first post means metadata updates because "POST" is used for metadata updates. | 06:59 |
| ho | kota_: now i'm looking for acoles' patch. wait a minute. | 06:59 |
| kota_ | ho: which one? related to 187489? | 07:01 |
| *** admin6 has joined #openstack-swift | 07:01 | |
| ho | kota_: found it the patch #187264 | 07:02 |
| kota_ | ho: thx | 07:03 |
| ho | kota_: in this patch, acoles added a test "this last POST would be ok by itself but takes the aggregate backend metadata size over limit" | 07:05 |
| ho | kota_: I thought it's better to have border one but my review for it didn't meet the dead line. So I submitted by myself. | 07:06 |
| kota_ | ho: sounds these are just sanity for successful request... | 07:08 |
| kota_ | ho: now reading the acoles's commit message. | 07:09 |
| ho | kota_: the purpose is "tighten up the metadata size tests" | 07:10 |
| kota_ | ho: I'm assuming acoles intended that we should assert the constraint should be checked from the backend aggregation. | 07:12 |
| kota_ | ho: and it has done on 187264 basically. | 07:13 |
| kota_ | ho: now I'm realising we might have your patch but maybe more description seems needed to land. | 07:13 |
| kota_ | ho: because the constraint test has existed by 187264 and I don't want to make other reviewers to misunderstand, if it's only me to do so... | 07:15 |
| *** hugespoon has quit IRC | 07:16 | |
| *** sudorandom has quit IRC | 07:16 | |
| *** hugespoon has joined #openstack-swift | 07:16 | |
| *** sudorandom has joined #openstack-swift | 07:18 | |
| *** hseipp has joined #openstack-swift | 07:18 | |
| *** hseipp has quit IRC | 07:18 | |
| *** hseipp has joined #openstack-swift | 07:18 | |
| kota_ | ho: so maybe commit message should be updated because the border case is existing. | 07:18 |
| *** haigang has quit IRC | 07:19 | |
| ho | kota_: i think constraint test needs to cover it's border such as line829 and line835. | 07:20 |
| kota_ | ho: I guess you want to add the border case of the constraint on the aggregated size between incomming and existing metadata size, right? | 07:20 |
| kota_ | ho: does it increase test corverage on nosetests? | 07:21 |
| ho | kota_: yes. | 07:22 |
| ho | kota_: as for the coverage, i didn't check it. | 07:23 |
| ho | kota_: do i need to check the coverage? do you know how to do it? | 07:24 |
| kota_ | ho: not sure but I guess tox shows the coverage or use an option to show it (please see nose manual) | 07:25 |
| kota_ | ho: if it increase the coverage, it will be good reason to land. | 07:25 |
| kota_ | s/increase/increases/ | 07:26 |
| kota_ | ho: right now I'm realising not to oppose your opinion to add the sanity but I'd like to reccomend you to write the reason why we need this and what's difference from exisiting *the border case* in commit message. | 07:28 |
| kota_ | ho: like as acoles's merged patch. | 07:29 |
| kota_ | ho: *because* the border case looks duplicated... | 07:30 |
| kota_ | ho: and I'd like to appologize my misundrstanding (2nd) on gerrit comment. This change should assert the border case. | 07:33 |
| *** silor has quit IRC | 07:33 | |
| ho | kota_: I was caught by my boss again.. sorry... thanks for your comment. I will update the commit message and I'm happy to have your understanding of the code. | 07:34 |
| *** chlong has quit IRC | 07:37 | |
| *** zhill_ has joined #openstack-swift | 07:38 | |
| kota_ | ho: logged the summary on the discussion, here :) | 07:40 |
| *** jistr has joined #openstack-swift | 07:51 | |
| *** acoles_away is now known as acoles | 07:58 | |
| acoles | good morning | 07:59 |
| kota_ | acoles: morning | 08:00 |
| cschwede | Good Morning! | 08:02 |
| cschwede | acoles: found the bug: https://github.com/cschwede/nose-example-bug | 08:02 |
| acoles | cschwede: so running nose with --with-openstack fixes all bugs! thats the power of openstack :D | 08:05 |
| *** admin6 has quit IRC | 08:05 | |
| *** admin6_ has joined #openstack-swift | 08:05 | |
| cschwede | haha, yeah! :D | 08:05 |
| acoles | cschwede: seriously, good work, thx. | 08:06 |
| cschwede | i’m sending a mail to -infra and -dev soon, might be that other projects are affected too | 08:06 |
| cschwede | thx | 08:06 |
| acoles | so is the issue that errors raised *during package setup* are ignored? | 08:07 |
| cschwede | acoles: yes | 08:09 |
| cschwede | and then the whole package isn’t executed | 08:09 |
| cschwede | which basically means - skip the tests | 08:10 |
| *** sudorandom has quit IRC | 08:11 | |
| *** jordanP has joined #openstack-swift | 08:12 | |
| *** hugespoon has quit IRC | 08:13 | |
| *** CrackerJackMack has quit IRC | 08:13 | |
| *** hugespoon has joined #openstack-swift | 08:13 | |
| acoles | cschwede: my saio is in a mess after last night, i will restore and then try out your patch 187754 locally. | 08:13 |
| patchbot | acoles: https://review.openstack.org/#/c/187754/ | 08:13 |
| acoles | i'm curious to see if jenkins passes func tests with keystone now you have added keystoneclient requirement | 08:14 |
| acoles | kota_: hi! | 08:14 |
| kota_ | acoles: hi :) | 08:15 |
| *** CrackerJackMack has joined #openstack-swift | 08:18 | |
| *** sudorandom has joined #openstack-swift | 08:19 | |
| cschwede | acoles: notmyname: bug report for the -infra team: https://bugs.launchpad.net/swift/+bug/1461440 | 08:19 |
| openstack | Launchpad bug 1461440 in OpenStack Object Storage (swift) "Tests are not run, but reported as success" [Undecided,In progress] - Assigned to Christian Schwede (cschwede) | 08:19 |
| *** admin6 has joined #openstack-swift | 08:19 | |
| *** admin6_ has quit IRC | 08:19 | |
| *** zhill_ has quit IRC | 08:19 | |
| kota_ | cschwede: hi | 08:27 |
| kota_ | cschwede: I pushed a pull request to your durability calculater repository for EC case :) | 08:28 |
| kota_ | cschwede: just now | 08:28 |
| kota_ | cschwede: please let me know if you have requests to fix my patch set ;) | 08:30 |
| openstackgerrit | Christian Schwede proposed openstack/swift: Fix testing issues https://review.openstack.org/187754 | 08:31 |
| cschwede | kota_: great, just saw that! thanks for the PR, i will review it later and merge it then | 08:32 |
| kota_ | cschwede: :) | 08:34 |
| *** ho has quit IRC | 08:36 | |
| jordanP | cschwede, thanks for fixing the func tests in the gate :) | 08:42 |
| cschwede | jordanP: you’re welcome - glad that you brought this up! might be that other projects are affected as well? | 08:43 |
| jordanP | I don't know, in fact, we run a 3rd party CI to test swift again our diskfile and we kept failing the "test_tempurl_keys_visible_to_account_owner" test. So I wondered whether this test passed 'upstream'. That's how I noticed it was not run. | 08:44 |
| cschwede | jordanP: it didn’t pass upstream on the gate, but locally. i just updated my patch, and hopefully the last version fixes the issue with the tempurl tests as well | 08:49 |
| *** tellesnobrega has quit IRC | 09:07 | |
| *** geaaru has joined #openstack-swift | 09:30 | |
| *** silor has joined #openstack-swift | 09:31 | |
| cschwede | yeah, tests are running & passing again! https://jenkins05.openstack.org/job/check-swift-dsvm-functional/226/console | 09:57 |
| cschwede | acoles: ^^ | 09:57 |
| acoles | cschwede: nice | 09:59 |
| *** tellesnobrega has joined #openstack-swift | 10:02 | |
| *** hseipp has quit IRC | 10:03 | |
| *** mmcardle has joined #openstack-swift | 10:12 | |
| *** mmcardle has quit IRC | 10:14 | |
| *** [1]dimasot has joined #openstack-swift | 10:24 | |
| *** dimasot has quit IRC | 10:26 | |
| *** [1]dimasot is now known as dimasot | 10:26 | |
| *** rvasilets has left #openstack-swift | 10:27 | |
| acoles | cschwede: i am so confused :( | 10:30 |
| cschwede | acoles: ? maybe i can help? | 10:30 |
| acoles | the func tests ran but both runs were with keystone config i think (tests requiring account acls skipped both times) | 10:30 |
| acoles | which i thought i had fixed back in february | 10:31 |
| *** samueldmq has joined #openstack-swift | 10:31 | |
| samueldmq | notmyname, morning | 10:31 |
| samueldmq | notmyname, thanks for updating the oslo.policy bug on swift side | 10:31 |
| acoles | cschwede: https://review.openstack.org/#/c/158671/ ... tests after that merged were running against tempauth | 10:33 |
| *** mmcardle has joined #openstack-swift | 10:39 | |
| cschwede | acoles: hmm, that’s strange | 10:44 |
| *** mmcardle has quit IRC | 10:45 | |
| cschwede | acoles: found it - top of the changelog, 2.0.0: http://tox.readthedocs.org/en/latest/changelog.html | 10:45 |
| cschwede | "introduce environment variable isolation" | 10:45 |
| acoles | cschwede: yes. i wondered if the test config file was not being found but that would cause IN_PROCESS mode to be used and that would be clearly logged | 10:45 |
| acoles | cschwede: heh. that would explain it! | 10:46 |
| acoles | so looks like we need to patch infra with the passenv setting | 10:47 |
| acoles | patch tox.ini i mean | 10:48 |
| cschwede | yes, doing that right now | 10:49 |
| cschwede | SWIFT_TEST_AUTH,SWIFT_TEST_USER,SWIFT_TEST_KEY,SWIFT_TEST_POLICY,SWIFT_TEST_IN_PROCESS_CONF_DIR,SWIFT_TEST_IN_PROCESS,SWIFT_TEST_IN_MEMORY_OBJ - anything i forgot? | 10:49 |
| acoles | cschwede: good man! | 10:50 |
| acoles | just the important one: SWIFT_TEST_CONFIG_FILE :D | 10:51 |
| acoles | looks like passenv = SWIFT_* might work | 10:51 |
| acoles | cschwede: https://bugs.launchpad.net/openstack-gate/+bug/1455102 <- be worth linking from your patch | 10:54 |
| openstack | Launchpad bug 1455102 in python-novaclient "some test jobs broken by tox 2.0 not passing env variables" [High,In progress] - Assigned to Sean Dague (sdague) | 10:54 |
| openstackgerrit | Christian Schwede proposed openstack/swift: Fix testing issues https://review.openstack.org/187754 | 10:54 |
| cschwede | ok, trying with SWIFT_* - looks better than the long line, and using that is is more likely to catch all possible future env variables as well | 10:55 |
| openstackgerrit | Christian Schwede proposed openstack/swift: Fix testing issues https://review.openstack.org/187754 | 10:55 |
| acoles | BTW the swiftclient bug I found that was causing my func tests to not run came in after 2.4.0 so not yet affecting jenkins | 10:56 |
| cschwede | but it is a bug and will break things if a new release is published? | 10:57 |
| acoles | oh yes, i have a fix just running tests... | 10:58 |
| cschwede | acoles: patch 187749 ? | 10:59 |
| patchbot | cschwede: https://review.openstack.org/#/c/187749/ | 10:59 |
| acoles | cschwede: i updated bug 1455102 with a link to your patch | 10:59 |
| openstack | bug 1455102 in python-novaclient "some test jobs broken by tox 2.0 not passing env variables" [High,In progress] https://launchpad.net/bugs/1455102 - Assigned to Sean Dague (sdague) | 10:59 |
| acoles | yes 187749 | 11:01 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift: Fix missing container update https://review.openstack.org/186735 | 11:02 |
| *** lcurtis_ has joined #openstack-swift | 11:03 | |
| openstackgerrit | Alistair Coles proposed openstack/python-swiftclient: Make default get_auth timeout be None https://review.openstack.org/187749 | 11:04 |
| acoles | cschwede: zaitcev: ^^ | 11:08 |
| cschwede | acoles: thx, just saw it - now with tests :) | 11:08 |
| *** kota_ has quit IRC | 11:09 | |
| *** lcurtis_ has quit IRC | 11:10 | |
| acoles | ok time for a coffee... | 11:11 |
| *** Triveni has quit IRC | 11:12 | |
| *** zul has quit IRC | 11:22 | |
| *** cdelatte has joined #openstack-swift | 11:25 | |
| *** aix has quit IRC | 11:25 | |
| *** delattec has joined #openstack-swift | 11:25 | |
| *** zul has joined #openstack-swift | 11:28 | |
| *** [1]dimasot has joined #openstack-swift | 11:30 | |
| *** dimasot has quit IRC | 11:33 | |
| *** [1]dimasot is now known as dimasot | 11:33 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift: Updated from global requirements https://review.openstack.org/88736 | 11:35 |
| *** dimasot has quit IRC | 11:46 | |
| *** distroguy has quit IRC | 11:55 | |
| *** aix has joined #openstack-swift | 12:00 | |
| *** lpabon has joined #openstack-swift | 12:02 | |
| *** distroguy has joined #openstack-swift | 12:02 | |
| *** ppai has quit IRC | 12:04 | |
| *** distroguy has quit IRC | 12:07 | |
| *** SkyRocknRoll has quit IRC | 12:16 | |
| openstackgerrit | Mark Seger proposed openstack/swift: Change usage help and Attention messages to warnings https://review.openstack.org/187974 | 12:17 |
| *** distroguy has joined #openstack-swift | 12:17 | |
| *** ppai has joined #openstack-swift | 12:17 | |
| acoles | cschwede: so it seems the reason i could not get tox 2.0.1 to install a venv last night is because of the change you spotted - my http_proxy env vars are no longer passed so pip cannot reach pypi :/ and there seems to be no tox command line override to force the var through to venv | 12:20 |
| acoles | cschwede: do you think it is acceptable to add http_proxy (and https_proxy and no_proxy) to passenv in tox.ini? seems ugly but i can't see an alternative | 12:22 |
| cschwede | acoles: oh wow, now that is bad luck! i’m wondering if adding a *_proxy would work too. trying | 12:23 |
| acoles | cschwede: the other answer is that i work at home everyday ;) | 12:24 |
| *** mmcardle has joined #openstack-swift | 12:24 | |
| acoles | actually we have a pypi mirror inside but that brings another set of issues... | 12:25 |
| cschwede | acoles: what do you prefer? shall i search for a good technical reason to avoid that env variable, and force you to stay home? ;) | 12:25 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift: Fix missing container update https://review.openstack.org/186735 | 12:25 |
| *** mmcardle has quit IRC | 12:26 | |
| acoles | cschwede: heh | 12:26 |
| acoles | i can try *_proxy | 12:26 |
| acoles | cschwede: yes *_proxy works. so do you think it could be added with your patch or shall i do a follow on patch? | 12:31 |
| cschwede | acoles: thx for checking, i’ll add it | 12:32 |
| acoles | bt i ran into this because i am verifying your patch with tox2.0.1 locally | 12:32 |
| acoles | cschwede: maybe there is someone else that will benefit from passenv = *_proxy | 12:33 |
| openstackgerrit | Christian Schwede proposed openstack/swift: Fix testing issues https://review.openstack.org/187754 | 12:33 |
| cschwede | acoles: yes, i would assume so. i think an update to the bugreport is also a good idea to notify others | 12:34 |
| acoles | yup i will do that | 12:34 |
| *** mmcardle has joined #openstack-swift | 12:36 | |
| *** mwheckmann has joined #openstack-swift | 12:45 | |
| *** bi_fa_fu has joined #openstack-swift | 12:51 | |
| *** jistr is now known as jistr|mtg | 12:57 | |
| *** kei_yama has quit IRC | 13:05 | |
| *** mmcardle has quit IRC | 13:06 | |
| *** km has quit IRC | 13:12 | |
| *** jkugel has joined #openstack-swift | 13:14 | |
| openstackgerrit | Alistair Coles proposed openstack/swift: Fix testing issues https://review.openstack.org/187754 | 13:14 |
| acoles | cschwede: ^^ passenv needed to be space separated | 13:15 |
| acoles | cschwede: i added a note in the docs about needing tox -r | 13:15 |
| cschwede | acoles: thx for the update, looks good! | 13:16 |
| acoles | cschwede: just realised i have the same issue with tox and http_proxy on all projects! | 13:22 |
| cschwede | acoles: ouch! that hurts | 13:24 |
| *** petertr7_away is now known as petertr7 | 13:25 | |
| acoles | cschwede: yeah. i think i will need to write a tox.ini-morphing script to add the passenv option locally | 13:26 |
| *** ppai has quit IRC | 13:28 | |
| *** hseipp has joined #openstack-swift | 13:31 | |
| *** hseipp has quit IRC | 13:31 | |
| *** hseipp has joined #openstack-swift | 13:32 | |
| petertr7 | Hi cschwede, I was wondering, what are your general plans are for django-swiftbrowser? | 13:37 |
| *** ahale has quit IRC | 13:38 | |
| *** blmartin has joined #openstack-swift | 13:40 | |
| *** blmartin_ has joined #openstack-swift | 13:42 | |
| *** SkyRocknRoll has joined #openstack-swift | 13:44 | |
| cschwede | petertr7: Hello Peter! Well, I want to add a few more key features, for example pagination, metadata and a better way to integrate 3rd party addons/plugins | 13:45 |
| cschwede | petertr7: I’m also looking into making it usable (again) without Javascript, by using some more sane defaults | 13:45 |
| petertr7 | cschwede, Hello =) Ahh I see | 13:46 |
| cschwede | petertr7: what are you looking for? | 13:46 |
| cschwede | anything specific that you’d like to see? | 13:46 |
| petertr7 | cschwede, I got started on pagination actually. I created a stack that tracks container names. It works but it's a little messy | 13:46 |
| petertr7 | Thanks for asking. I just wanted to get an idea of what direction you were moving in so I can potentially push back upstream | 13:47 |
| cschwede | petertr7: yes, please do so :) that said, i think there is already a PR from you, adding pagination, or not? | 13:49 |
| petertr7 | Yes that's me | 13:50 |
| petertr7 | I think I need to build some test cases for it | 13:50 |
| cschwede | that would be great | 13:51 |
| petertr7 | Great | 13:51 |
| petertr7 | I am a little stumped. How would I replicate queries to a swift cluster while doing unit testing? | 13:52 |
| petertr7 | For example, I have a function that retrieves the next 5 containers starting from container "alpha". While testing locally I just created the containers on my cluster manually | 13:52 |
| petertr7 | How do I automate it so that the unit test can run independent of connecting to a swift cluster? | 13:53 |
| *** zaitcev has joined #openstack-swift | 13:53 | |
| *** ChanServ sets mode: +v zaitcev | 13:53 | |
| cschwede | petertr7: you need to mock the functionality in the tests. for example: https://github.com/cschwede/django-swiftbrowser/blob/master/tests/test_swiftbrowser.py#L83-L109 | 13:56 |
| cschwede | petertr7: line 94 (https://github.com/cschwede/django-swiftbrowser/blob/master/tests/test_swiftbrowser.py#L94) contains the dummy data (container listing), that is then used by swiftbrowser in the tests | 13:57 |
| petertr7 | Ohhh I see | 13:57 |
| petertr7 | That's convenient | 13:57 |
| petertr7 | Thank you! | 13:57 |
| *** jistr|mtg is now known as jistr | 13:59 | |
| cschwede | you’re welcome! | 13:59 |
| openstackgerrit | Mark Seger proposed openstack/swift: Change usage help and Attention messages to warnings https://review.openstack.org/187974 | 14:01 |
| *** distroguy has quit IRC | 14:07 | |
| openstackgerrit | Alistair Coles proposed openstack/python-swiftclient: Run functional tests using keystone auth options https://review.openstack.org/123514 | 14:09 |
| *** csmart has joined #openstack-swift | 14:10 | |
| *** silor has quit IRC | 14:11 | |
| *** silor1 has joined #openstack-swift | 14:11 | |
| *** bkopilov has quit IRC | 14:12 | |
| *** petertr7 is now known as petertr7_away | 14:15 | |
| *** petertr7_away is now known as petertr7 | 14:16 | |
| *** acampbell has joined #openstack-swift | 14:18 | |
| *** acampbel11 has joined #openstack-swift | 14:19 | |
| *** wbhuber has joined #openstack-swift | 14:20 | |
| *** david-lyle has quit IRC | 14:25 | |
| *** bsdkurt has joined #openstack-swift | 14:27 | |
| *** bsdkurt has quit IRC | 14:28 | |
| *** bsdkurt has joined #openstack-swift | 14:28 | |
| *** thurloat_isgone is now known as thurloat | 14:29 | |
| *** petertr7 is now known as petertr7_away | 14:30 | |
| *** petertr7_away is now known as petertr7 | 14:31 | |
| *** barra204 has quit IRC | 14:36 | |
| *** shakamunyi has quit IRC | 14:36 | |
| *** minwoob has joined #openstack-swift | 14:37 | |
| *** acampbel11 has quit IRC | 14:39 | |
| *** acampbell has quit IRC | 14:39 | |
| openstackgerrit | Mark Seger proposed openstack/swift: Change usage help and Attention messages to warnings https://review.openstack.org/187974 | 14:49 |
| *** petertr7 is now known as petertr7_away | 14:55 | |
| *** jordanP has quit IRC | 15:17 | |
| openstackgerrit | Mark Seger proposed openstack/swift: Change usage help and Attention messages to warnings https://review.openstack.org/187974 | 15:19 |
| *** petertr7_away is now known as petertr7 | 15:26 | |
| *** mmcardle has joined #openstack-swift | 15:33 | |
| *** david-lyle has joined #openstack-swift | 15:39 | |
| *** jrichli has joined #openstack-swift | 15:41 | |
| *** mmcardle has quit IRC | 15:41 | |
| *** david-lyle has quit IRC | 15:47 | |
| thurloat | does the swift-ring-builder do anything differently if it has access to the services listed in the rings? | 15:49 |
| thurloat | I'm getting drastically different results running a ring-builder add & rebalance between a local server, and a proxy node | 15:50 |
| thurloat | here's the different outputs: https://gist.githubusercontent.com/thurloat/fdd8f33db357c0451518/raw/1a48890617c5462905dc46fed07d28faa2c4e9d6/output | 15:53 |
| *** david-lyle has joined #openstack-swift | 15:53 | |
| *** acampbell has joined #openstack-swift | 15:54 | |
| *** acampbel11 has joined #openstack-swift | 15:54 | |
| thurloat | builders are the same both local and on proxy server (checksummed to make sure) | 15:54 |
| *** lpabon has quit IRC | 15:54 | |
| notmyname | good morning | 15:55 |
| acoles | notmyname: good morning | 15:58 |
| thurloat | notmyname, word | 16:00 |
| notmyname | I'm down at http://storageconference.us today | 16:01 |
| notmyname | unfortunately, I'm on a panel that looks like it will overlap with today's swift team meeting | 16:01 |
| *** mmcardle has joined #openstack-swift | 16:02 | |
| *** mmcardle has quit IRC | 16:03 | |
| *** jistr has quit IRC | 16:11 | |
| *** nadeem has joined #openstack-swift | 16:15 | |
| notmyname | which raises the question of having today's meeting or not | 16:16 |
| notmyname | any thoughts? | 16:16 |
| *** nadeem has quit IRC | 16:16 | |
| *** pksingh__ has joined #openstack-swift | 16:17 | |
| notmyname | current agenda is at https://wiki.openstack.org/wiki/Meetings/Swift | 16:18 |
| notmyname | not much there except a question of what to do with py3/six patches | 16:18 |
| pksingh__ | Hi, could you guys please review : SwiftClient object upload beginning with / or "./" https://review.openstack.org/#/c/185269/2 | 16:19 |
| acoles | notmyname: maybe py3/six discussion can wait a week? i see plenty of other reviewing to keep us busy | 16:20 |
| *** admin6 has quit IRC | 16:20 | |
| notmyname | acoles: yeah | 16:20 |
| pksingh__ | Hi could you guys please review : Read the response body, if response status is greater than 300 https://review.openstack.org/#/c/183899/ | 16:20 |
| notmyname | and every mornign when I wake up I'm seeing a ton of swift emails from gerrit. very nice to see a lot of reviews going on | 16:20 |
| *** jamielennox is now known as jamielennox|away | 16:21 | |
| *** barra204 has joined #openstack-swift | 16:21 | |
| *** david-lyle has quit IRC | 16:22 | |
| acoles | torgomatic: ping | 16:22 |
| *** bi_fa_fu has quit IRC | 16:22 | |
| *** gyee has joined #openstack-swift | 16:23 | |
| notmyname | ok, let's skip today's meeting | 16:23 |
| notmyname | sorry for the last minute notice. | 16:24 |
| *** barra204 has quit IRC | 16:28 | |
| *** pksingh__ has quit IRC | 16:29 | |
| *** proteusguy has joined #openstack-swift | 16:29 | |
| thurloat | I think i found out why I had different output: https://review.openstack.org/#/c/115441/ | 16:29 |
| thurloat | looks like I was running a newer version on the local server -- and it appears as if it works 'better' than the prod version in rebuilding. | 16:29 |
| ctennis | thurloat: reassigning 100% partitions in the 2nd example seems really odd (bad?). Essentially every partition has moved, which is likely something you really do not want to have happen. | 16:32 |
| ctennis | i'm sure there's some strange nuance to that, but I don't know why it would do that | 16:33 |
| thurloat | ctennis, i thought it was strange behaviour for adding two drives to want to redistribute everything | 16:36 |
| ctennis | yeah that sounds wrong. | 16:36 |
| *** SkyRocknRoll has quit IRC | 16:37 | |
| thurloat | thanks for the sounding board ctennis. going to move forward /w my upgrades now :) | 16:38 |
| *** dimasot has joined #openstack-swift | 16:38 | |
| notmyname | I'm starting to go through some LP bugs while I'm in the back of the room at this conference | 16:38 |
| *** petertr7 is now known as petertr7_away | 16:38 | |
| notmyname | tag for erasure-code related work is "ec". please tag things there as needed, and look there for stuff that we need to get done | 16:39 |
| notmyname | peluse: torgomatic: clayg: ^ | 16:39 |
| *** marzif_ has joined #openstack-swift | 16:39 | |
| notmyname | https://bugs.launchpad.net/swift/+bugs?field.tag=ec | 16:40 |
| peluse | cool | 16:46 |
| peluse | so remind me again how I tag something? :) | 16:46 |
| *** RackerShagz has joined #openstack-swift | 16:47 | |
| *** bsdkurt has quit IRC | 16:48 | |
| *** bsdkurt has joined #openstack-swift | 16:49 | |
| *** dimasot has quit IRC | 16:50 | |
| notmyname | peluse: heh, I had to relearn that myself | 16:50 |
| notmyname | peluse: right under the description there's a "tags" field | 16:51 |
| *** SkyRocknRoll has joined #openstack-swift | 16:51 | |
| *** hseipp has left #openstack-swift | 16:51 | |
| notmyname | peluse: if none are assigned, then it says "add tags" | 16:51 |
| notmyname | if there are some, it will list them | 16:51 |
| peluse | OK, is it just for bugs? | 16:51 |
| notmyname | tags are space-separated | 16:51 |
| notmyname | and seemingly don't allow _ | 16:51 |
| notmyname | but - is fine | 16:51 |
| notmyname | yeah. or rather, it's for launchpad | 16:51 |
| peluse | OK thanks, For patches should we use topic EC or not worry about trying to all be consistent there? | 16:52 |
| openstackgerrit | Alistair Coles proposed openstack/python-swiftclient: Add passenv to tox.ini to make functests run with tempauth https://review.openstack.org/188080 | 16:52 |
| notmyname | peluse: so my idea is that I want to get a good portion of the LP bugs for swift/swiftclient triaged and tagged and then have something similar to the gerrit dash or the priority reviews page to track them and know what to work on | 16:52 |
| acoles | cschwede: ^^ same issue on swiftclient func tests | 16:52 |
| peluse | gotcha | 16:52 |
| notmyname | peluse: oh yeah. whatever you want. I don't really care about what "topic" you use for patches | 16:52 |
| peluse | that opens up all sorts of doors :) | 16:53 |
| acoles | peluse: be good ;) | 16:53 |
| peluse | FYI low hanging review fruit policy/EC related fix: https://review.openstack.org/#/c/185459/ | 16:55 |
| *** kutija has joined #openstack-swift | 16:57 | |
| *** david-lyle has joined #openstack-swift | 17:13 | |
| *** david-lyle has quit IRC | 17:13 | |
| *** david-lyle has joined #openstack-swift | 17:15 | |
| *** geaaru has quit IRC | 17:19 | |
| *** david-lyle has quit IRC | 17:28 | |
| *** petertr7_away is now known as petertr7 | 17:29 | |
| notmyname | timburke: seems like your patch fixes this? https://bugs.launchpad.net/swift/+bug/1181146 | 17:30 |
| openstack | Launchpad bug 1181146 in OpenStack Object Storage (swift) "TypeError: exceptions must be old-style classes or derived from BaseException, not NoneType" [Medium,Fix committed] - Assigned to Samuel Merritt (torgomatic) | 17:30 |
| notmyname | timburke: you tagged the patch as "related bug", not "closes bug". is there more work needed for that bug? | 17:31 |
| notmyname | (I marked it as "fix committed" for now | 17:31 |
| *** fthiagogv has joined #openstack-swift | 17:34 | |
| jlk | Hey all, so I need some help in debugging swift <-> keystone interaction, particularly over SSL. If I point my swift at my haproxy that terminates SSL for keystone I get failures authing tokens from swift-proxy. But if I point it directly at keystone non-ssl things work. I need some help. | 17:44 |
| clayg | jlk: what are you using to validate haproxy is terminating ssl and forwarding to keystone correctly (outside of swift) | 17:52 |
| jlk | clayg: my other services all do this, nova, glance, neutron, etc.. and they all work fine. My client software can get tokens from that keystone endpoint correctly. | 17:53 |
| acoles | jlk: someone had a similar issue in here recently, turned out be to do with setting insecure=true in authtoken middleware config section of proxy-server.conf iirc. ymmv | 17:53 |
| jlk | swift client code gets the token successfully from the same auth point that swift-proxy points to. | 17:53 |
| acoles | jlk: https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token/__init__.py#L270 | 17:53 |
| jlk | I would hope we don't have to do that. We are pointing to a ca file that has our cert in it inside the swift-proxy config | 17:53 |
| clayg | jlk: excellent - can you reproduce the results with keystone client *on* one of the proxy nodes? | 17:54 |
| jlk | yes, that's what I've been doing, using swift client on the proxy node | 17:54 |
| acoles | torgomatic: you around? | 17:54 |
| jlk | the cert is cafile = "/etc/ssl/certs/ca-certificates.crt" | 17:54 |
| clayg | notmyname: you're going to town on some bug triage - nice! | 17:56 |
| jlk | I've also constructed a stackrc file with swift's credentials and with that stackrc I can still get a catalog from keystone | 17:56 |
| acoles | torgomatic: nm. i'm +2 on your mult-range ec get patch but the ocd in me wants to fix the rfc ref in the commit message. | 17:56 |
| acoles | torgomatic: i'll change it and risk incurring your wrath | 17:56 |
| timburke | notmyname: mainly marked it as related since it only addressed the logging, rather than whatever was triggering exceptions under load. i suppose that'll have to wait for another, more specific ticket | 17:57 |
| clayg | jlk: yeah so that's swiftclient talking to the public keystone uri - what about the admin uri that's used for token validation? Can you validate that path - from a proxy node - outside of swift (e.g. using the keystone-client command line utility and swift's service account creds pointed to the keystone admin uri you have configured in the proxy's authtoken middleware section) | 17:58 |
| openstackgerrit | Alistair Coles proposed openstack/swift: EC: support multiple ranges for GET requests https://review.openstack.org/173497 | 17:58 |
| jlk | clayg: yes | 17:59 |
| jlk | clayg: our swift runs out if a venv, I'm wondering if it's some problem with ssl validation inside the venv, even with the config settings pointing to our ca file. I just don't know how to properly debug that bit of interaction inside swift-proxy | 18:00 |
| clayg | well - it's hard - the keystone authtoken middleware doesn't log that great :\ | 18:02 |
| jlk | so I've noticed. | 18:02 |
| jlk | it has log.DEBUG in the code, but how does one turn that on? | 18:02 |
| clayg | you could ativate the virtualenv and repeat the keystone admin with swift creds from the commandline test | 18:03 |
| clayg | log_level = DEBUG in the [DEFAULT] section in proxy conf + restart proxy | 18:03 |
| jlk | yeah I have that | 18:03 |
| jlk | oh | 18:03 |
| clayg | s/restart/reload/g | 18:03 |
| *** acoles is now known as acoles_away | 18:03 | |
| torgomatic | acoles: sup? | 18:03 |
| torgomatic | pfft :D | 18:03 |
| *** proteusguy has quit IRC | 18:04 | |
| jlk | okay got more now. | 18:05 |
| jlk | raise exceptions.SSLError(msg)#012SSLError: SSL exception connecting to https://tardis.openstack.blueboxgrid.com:35358: [Errno 185090050] _ssl.c:340: error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib | 18:05 |
| jlk | so looks like authtoken isn't loading the CA file correctly | 18:06 |
| clayg | heh - tardis | 18:13 |
| jlk | lovely, how the heck. | 18:13 |
| clayg | jlk: oh yeah - i was going to ask if you tried what acoles_away suggested about testing with insecure = true | 18:13 |
| jlk | I'm sure that'll work, but that's not a solution. | 18:13 |
| clayg | jlk: did you test it - or you're just "sure" | 18:14 |
| clayg | jlk: just curious - it's your time :P | 18:14 |
| jlk | sure, I can test it. | 18:15 |
| jlk | but I'm 99.9% it'll work | 18:15 |
| jlk | yeah that totally "works" | 18:15 |
| clayg | ok, well still - it would move the problem from "ssl doesn't work" to "ssl certificate verification doesn't work" - which seems a fair bit more specific | 18:15 |
| clayg | jlk: nice! | 18:16 |
| clayg | so - i wonder how it's loading the certificate - maybe a different ssl lib on this box that other services? | 18:16 |
| clayg | maybe from the commandline client you have to --secure to get the certificate validation to play? | 18:16 |
| *** RackerShagz has quit IRC | 18:16 | |
| jlk | I think the command line client, it's outside the venv, and gets to load the standard ca paths | 18:17 |
| jlk | but that's what the cafile option for authtoken is supposed to fix | 18:17 |
| notmyname | I _really_ wish that LP bugs has some other categories for confirming or closing a reported bug | 18:17 |
| notmyname | eg https://bugs.launchpad.net/swift/+bug/634358 | 18:17 |
| openstack | Launchpad bug 634358 in OpenStack Object Storage (swift) "remote address ACLs" [Wishlist,Confirmed] | 18:17 |
| *** acampbel11 has quit IRC | 18:18 | |
| *** acampbell has quit IRC | 18:18 | |
| *** acampbell has joined #openstack-swift | 18:18 | |
| *** acampbel11 has joined #openstack-swift | 18:18 | |
| *** Kr4zy has joined #openstack-swift | 18:19 | |
| *** marzif_ has quit IRC | 18:19 | |
| clayg | jlk: yeah idk custom certifcates have always confused me - you have to tell the ssllib to use them - and i'm not sure how the keystone middleware plumbs that - can you just install your cert on the box and then update the ca databases or whatever? what distro is this on? | 18:19 |
| *** marzif_ has joined #openstack-swift | 18:19 | |
| jlk | it's on Ubuntu. We DO have our cert installed in /etc/ssl/certs/ca-certificates.crt | 18:20 |
| jlk | but apparently python httplib / requests inside the venv isn't loading it | 18:20 |
| *** RackerShagz has joined #openstack-swift | 18:21 | |
| clayg | jlk: so.... if you activate the venv and use the commandline client it reproduces the certificate failure? | 18:21 |
| Kr4zy | I am seeing slow replication speed for the object-replicator relative to other nodes in a Swift cluster. Anyone know the root cause for this? | 18:22 |
| *** RackerShagz has quit IRC | 18:22 | |
| clayg | jlk: i'm just trying to understand why you think the venv has anything to do with it - i would think the ssl stuff punts to a shared library installed on the system somewhere | 18:22 |
| jlk | clayg: let me try that. I don't think we have the client installed inside the venv. | 18:22 |
| clayg | jlk: if not you could create a new venv and install the same deps see if it's reproduceable there | 18:23 |
| notmyname | love this conference. "so we've got 40TB of DRAM in this cluster...." | 18:23 |
| *** annegentle has joined #openstack-swift | 18:23 | |
| jlk | hrm, there is keystone in the venv | 18:24 |
| jlk | and it can get a catalog when called from the venv | 18:24 |
| clayg | notmyname: hrmm... http://yourdatafitsinram.com/ says that's not likly | 18:24 |
| *** RackerShagz has joined #openstack-swift | 18:24 | |
| jlk | clayg: I mention the venv because we've had similar problems before. python httplib ships its own ca certificate pile and loads it, rather than the system's. | 18:25 |
| jlk | but I can't find that inside the venv | 18:25 |
| notmyname | clayg: well this is the "massive storage systems and technology" conference ;-) | 18:27 |
| notmyname | these are the people that have been dealing with hundreds to thousands of PB for many years now | 18:27 |
| clayg | jlk: weird, i've never run into that - well for one because stdlib doesn't normally *do* certificate validation - so idk | 18:27 |
| peluse | that's a good conf BTW | 18:28 |
| clayg | jlk: and with requests it's always seems to work if it works for curl - even if i use requests from w/i a venv it's still the same comipled shared object from openssl under the hood | 18:29 |
| clayg | notmyname: yeah, i was just joking around - my way of saying "that's a rather impressive amount of ram" | 18:29 |
| notmyname | heh :-) | 18:30 |
| notmyname | ...another bug who's last activity was 3 years ago... | 18:30 |
| clayg | notmyname: but I guess it's really only a few hundered boxes with 128 GiB :D | 18:30 |
| jlk | well, then I don't know why keystoneauth isn't able to find it | 18:30 |
| jlk | keystone client from the venv can | 18:31 |
| clayg | are you sure keystone client from the venv is *doing* certificate validation? Can you move the custom ca out of the way and see if it still "works" | 18:31 |
| jlk | hrm. | 18:32 |
| jlk | I'll try, but there is no insecure setting in my stackrc. and with --debug I see it hitting the https spot. | 18:32 |
| torgomatic | jlk: strace the proxy to see where it's looking on the filesystem? | 18:32 |
| *** david-lyle has joined #openstack-swift | 18:33 | |
| jlk | hrm, client may not actually be doing the ssl verify :/ | 18:35 |
| Kr4zy | I am seeing slow replication speed for the object-replicator relative to other nodes in a Swift cluster. Anyone know the root cause for this? | 18:37 |
| Kr4zy | notmyname: do you know? | 18:37 |
| notmyname | no | 18:37 |
| notmyname | do you have a failed/failing drive on that box? | 18:38 |
| clayg | jlk: so all of the connecting to keystone is handled by keystone's authtoken middleware - if you have other servcies that use it and are workig you might spend a few sorta checking for differences between those boxes and here? | 18:39 |
| notmyname | bah. laptop power is nearly gone. | 18:39 |
| *** aix has quit IRC | 18:39 | |
| clayg | jlk: e.g. version of python packages in venvs, version of openssl libraries, md5 of ca, etc. | 18:40 |
| notmyname | be back later when/if I can find power | 18:40 |
| openstackgerrit | Samuel Merritt proposed openstack/swift: EC: support multiple ranges for GET requests https://review.openstack.org/173497 | 18:42 |
| *** flaper87 has quit IRC | 18:44 | |
| *** nadeem has joined #openstack-swift | 18:45 | |
| swifterdarrell | portante: just how high have you bumped workers (for object-server) and seen improvement? | 18:45 |
| openstackgerrit | Merged openstack/python-swiftclient: Make default get_auth timeout be None https://review.openstack.org/187749 | 18:47 |
| openstackgerrit | Merged openstack/swift: Fix testing issues https://review.openstack.org/187754 | 18:47 |
| clayg | swifterdarrell: how high have *you* bumped workers for object-server!? </spoiler-alert> | 18:49 |
| ctennis | notmyname: I've hand patched your policy stuff into ssbench on these nodes, btu I'm getting errors with zmq.green import in worker.py not being found | 18:50 |
| ctennis | notmyname: I assume there's a python lib change/requirement needed but I don't see it in the pull req | 18:51 |
| *** silor has joined #openstack-swift | 18:53 | |
| *** mordred has quit IRC | 18:55 | |
| *** mordred has joined #openstack-swift | 18:55 | |
| *** silor1 has quit IRC | 18:55 | |
| *** marzif_ has quit IRC | 18:57 | |
| *** barra204 has joined #openstack-swift | 18:57 | |
| *** barra204 has quit IRC | 19:02 | |
| *** mjseger has joined #openstack-swift | 19:08 | |
| *** flaper87 has joined #openstack-swift | 19:08 | |
| *** flaper87 has quit IRC | 19:09 | |
| *** flaper87 has joined #openstack-swift | 19:09 | |
| mjseger | jenkins finally ran on https://review.openstack.org/#/c/187974/ and it's generating errors with | 19:16 |
| mjseger | gate-swift-python27 complaining about some modules associated with erasure code, not part of any of the code I patched | 19:16 |
| mjseger | anyone have any clue how to track down the problem? given that it took may hours for jenkins to run I'd like to fix whatever is causing the problem 1st try and and wondering what the possibility is of something else unrelated to my patch causing this and if I should just upload again? | 19:18 |
| *** SkyRocknRoll has quit IRC | 19:20 | |
| tdasilva | mjseger: I don't think those errors are related to your patch with the exception of this one: FAIL: test_print_obj_policy_name_mismatch (test.unit.cli.test_info.TestPrintObjFullMeta) | 19:26 |
| tdasilva | I get the same failure on my saio | 19:26 |
| mjseger | tdasilva: thanks, I'll look more closely at that one, but I guess my issue is even if the others aren't my problem won't they prevent my patch from going forward? | 19:27 |
| mjseger | tdasilva: now I see the problem, I changed the text of an error message and the test didn't match. easy fix, but should I do that and re-upload? | 19:30 |
| tdasilva | mjseger: yes | 19:30 |
| mjseger | thx | 19:30 |
| *** RackerShagz has quit IRC | 19:31 | |
| *** RackerShagz has joined #openstack-swift | 19:31 | |
| peluse | tdasilva, congrats man! | 19:31 |
| tdasilva | peluse: thanks! :) | 19:31 |
| peluse | did notmyname tell you about the monthly dues? | 19:32 |
| tdasilva | he mentioned you take beer and he takes tequilla | 19:32 |
| openstackgerrit | Mark Seger proposed openstack/swift: Change usage help and Attention messages to warnings https://review.openstack.org/187974 | 19:32 |
| peluse | sounds good to me! | 19:33 |
| tdasilva | peluse: lol...not sure about everyone else... | 19:33 |
| mjseger | peluse: did you see my question about swift-get-nodes and storage policies? | 19:33 |
| peluse | clayg works for whiskey | 19:34 |
| tdasilva | peluse: oh yeah, i've heard that somewhere before | 19:34 |
| peluse | mjseger, I didn't. I'm on a tiny screen and pretty tied up for the rest of the week - can you repeat real quick? | 19:34 |
| mjseger | sure - in swift-get-nodes it looks like it pulls the policy index from a ring named object-idx | 19:35 |
| mjseger | but my question is what if the ring is named foo-idx? I don't think it's going to work, or am I missing somethign? | 19:35 |
| peluse | the ring has to be named object-X where X is the index | 19:36 |
| peluse | or no -X for policy 0 of course | 19:36 |
| mjseger | thats what I thought, but I had a very long discussion with acoles: beciase submitting the patch and he felt pretty strongly that there is no need for explicit ring names, they can be anything | 19:37 |
| mjseger | I think if the naming really isn't a convention but more of a rule, tools like get-nodes should verify the container names match account/container/object and throw an error if they don't | 19:38 |
| mjseger | then again it may be too late | 19:38 |
| peluse | I know there's multiple places in the code where 'object' is expected - remember from adding policy support but acoles_away is generally correct so maybe something has changed | 19:38 |
| mjseger | right | 19:38 |
| mjseger | while I'm in the midst of the patch, I could see if somoene specifies an account container object and a ring named xxx-yyy to tell them it looks like a policy ring but must be named object-yyy to work | 19:39 |
| mjseger | may be a longer discussion for another day especially if you need to run | 19:40 |
| peluse | yeah, it wouldn't be hard to check the code but yeah I'm off to a meeting... | 19:40 |
| mjseger | I'm thinking this may be the case where the code had changed multiple times over multiple years and while there may once have been a reason for doing something that reason may have gone away and everyone if afraid (including me) to change anything | 19:41 |
| portante | swifterdarrell: we run 2 object-servers per CPU on 24 core systems | 19:42 |
| mjseger | it also begs the question that with all the scrutiny policies got, if nobody complained about explicitly naming the object ring object-policy,perhaps the name are more of a convention and should be a rule | 19:42 |
| swifterdarrell | portante: do you count HT cores in that? | 19:43 |
| portante | I don't think I did at the time, I had HT turned off | 19:45 |
| swifterdarrell | portante: k, I'm seeing normal object-servers doing better all the way up to 3x disk count (in my case, I've got ~30 disks, ~12 cores not counting HT); significant improvement between workers=30 and workers=90, though workers=90 is still not quite as good as servers_per_port=3 (w/about 90 processes, 3 per disk) | 19:45 |
| swifterdarrell | portante: so you had 24 actual cores? like 4 sockets and 6 cores each? | 19:46 |
| *** blmartin_ has quit IRC | 19:47 | |
| *** blmartin has quit IRC | 19:47 | |
| thurloat | fun question: is there a way to make the rebalanced objects go faster? currently transferring across nodes at around ~50Mbps, have already uppped the rsync concurrency but that's no longer a bottleneck | 19:47 |
| portante | swifterdarrell: what is the difference between the workers=90 and servers_per_port=3? | 19:48 |
| swifterdarrell | portante: still quantifying that (have one more workers=90 run to do to get two samples on that one) | 19:49 |
| *** petertr7 is now known as petertr7_away | 19:49 | |
| swifterdarrell | portante: I think it'll be single-digit percentage difference | 19:49 |
| swifterdarrell | portante: after this, I'll emulate a clobbered disk and normal object-auditor and expect to see the benefit of the I/O isolation kick in | 19:50 |
| *** petertr7_away is now known as petertr7 | 19:50 | |
| portante | I would expect that with an normal distribution for objects across the set of disks that servers_per_port would beat the default always because a greedy object server accepting connections won't have as bad an effect for servers_per_port | 19:50 |
| portante | swifterdarrell: are you lowering the greenthread count when using the higher object server counts? | 19:52 |
| swifterdarrell | portante: no; always the def. 1024 | 19:53 |
| portante | try lowering it, it should help with the higher worker counts | 19:53 |
| swifterdarrell | portante: well, it's extremely time-expensive for each thing I check; not sure I'll get to new ideas not already in my plan ;) | 19:53 |
| portante | ;) | 19:54 |
| portante | always happy to heap more work on! | 19:54 |
| clayg | lol | 19:56 |
| clayg | portante: what are you doing hanging about - we missed you! | 19:56 |
| clayg | portante: you should come sabatocal on some swift stuff - ec is kind of interesting - we'll offer you a temporary-core-visa. | 19:56 |
| clayg | s/what-ever-the-heck-that-was/sabbatical/g | 19:57 |
| *** flaper87 has quit IRC | 19:57 | |
| portante | clayg: thanks, now that thiago is a core dev I can just delegate to him! | 19:58 |
| portante | ;) | 19:58 |
| *** barra204 has joined #openstack-swift | 19:58 | |
| *** flaper87 has joined #openstack-swift | 19:59 | |
| portante | and I have never left, just lurking. ;) | 20:00 |
| *** RackerShagz has quit IRC | 20:02 | |
| *** barra204 has quit IRC | 20:03 | |
| *** flaper87 has quit IRC | 20:03 | |
| *** nadeem has quit IRC | 20:03 | |
| clayg | :) | 20:08 |
| *** thurloat is now known as thurloat_isgone | 20:10 | |
| *** Kr4zy has quit IRC | 20:16 | |
| *** flaper87 has joined #openstack-swift | 20:16 | |
| *** RackerShagz has joined #openstack-swift | 20:24 | |
| mattoliverau | Morning all.. So we move the meeting, so I should be able to sleep in, but the wife needs to travel today so up early #sleepfail | 20:28 |
| *** acampbell has quit IRC | 20:33 | |
| *** acampbel11 has quit IRC | 20:33 | |
| mattoliverau | On the plus side, I've got a new version of best candidate sub trees algorithm (used in the container sharder) that can scan even a giant containers in one iteration while using minimal ram. (Well finished it last night, but scarily working in testing this morn) | 20:34 |
| *** ho has joined #openstack-swift | 20:37 | |
| ho | Good morning! | 20:38 |
| swifterdarrell | portante: haha, so what I'm hearing is that tdasilva can do your benchmarking for you! | 20:38 |
| swifterdarrell | portante: oh, one other thing, my current round of tests is 2hr runs, but the workload is "realistic" which also means "random variance", so there's a fair bit of noise in the measurements; OTOH, it's less synthetic | 20:39 |
| swifterdarrell | portante: I may do some more shorter GET/PUT-only runs, as well as use swift-bench to make sure ssbench isn't biasing or otherwise doing something stupid | 20:40 |
| ho | ping briancline | 20:41 |
| mattoliverau | ho: morning | 20:44 |
| ho | mattoliverau: morning! | 20:45 |
| clayg | notmyname reminds us "no meeting today!" | 20:52 |
| *** kota_ has joined #openstack-swift | 20:52 | |
| kota_ | good morning | 20:53 |
| zaitcev | what, really no meeting? | 20:53 |
| hurricanerix_ | clayg: thanks, i would have been sitting in there for a while wondering where everybody was. =P | 20:53 |
| zaitcev | It's the first time I remembered about the new time. | 20:53 |
| clayg | ho: mattoliverau: zaitcev: kota_: sorry! | 20:53 |
| clayg | zaitcev: well let's do it then! | 20:54 |
| * clayg is not it for agenda | 20:54 | |
| kota_ | no meeting? | 20:54 |
| ho | clayg: np! | 20:55 |
| kota_ | oh, just finding at wiki | 20:55 |
| mattoliverau | Ahh the agenda is dated for next week.. date recognition fail :P | 20:55 |
| clayg | ho: kota_: zaitcev: mattoliverau: so the issue is that notmyname couldn't make it and no one wanted to setup and cover for him because none of us really had anything we wanted to go over today | 20:56 |
| mattoliverau | clayg: fair enough.. too big shoes to fill anyway :) | 20:56 |
| mattoliverau | I might go a rage sleep then :P | 20:56 |
| kota_ | clayg: I see, no problem | 20:57 |
| clayg | :) still - GOOD MORNING! | 20:57 |
| kota_ | mattoliverau: +1! | 20:57 |
| clayg | lol | 20:57 |
| mattoliverau | Like rage sleep == rage quit, but really means simply going back to bed :P | 20:57 |
| clayg | perfect | 20:57 |
| mattoliverau | clayg: oh and good morning clayg :) | 20:57 |
| clayg | from what i understand we should be back to normal next week - so - GOOD NIGHT! | 20:58 |
| *** nadeem has joined #openstack-swift | 20:59 | |
| kota_ | ok, have a good day :) | 20:59 |
| *** kota_ has quit IRC | 20:59 | |
| mattoliverau | cools, no stress :) | 21:00 |
| *** silor has quit IRC | 21:01 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-swiftclient: Updated from global requirements https://review.openstack.org/89250 | 21:01 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift: Updated from global requirements https://review.openstack.org/88736 | 21:01 |
| *** bsdkurt has quit IRC | 21:01 | |
| *** bsdkurt has joined #openstack-swift | 21:03 | |
| *** annegentle has quit IRC | 21:06 | |
| *** samueldmq has quit IRC | 21:06 | |
| *** petertr7 is now known as petertr7_away | 21:09 | |
| portante | swifterdarrell: sorry, back | 21:12 |
| portante | swifterdarrell: so ... can't resist, one more thing to consider ... we saw small requests starve out large requests | 21:13 |
| clayg | yay nova thread from garbutt about going semver - and infra's all like "on the plus side we already support this because Swift" | 21:13 |
| clayg | "Swift doing it right until you catch up" | 21:13 |
| portante | basically, 5 clients, one at 10 K, one pulling 100K, one pull 1 M, one pulling 10 M, one pull 100M, the 10M and 100M go "slow" until the 10 K clients clear out | 21:14 |
| portante | so I would expect to NOT see that behavior with servers_per_port, because you have enough object servers to cover the cases (assuming small and large requests don't land on the same disk) | 21:15 |
| portante | using normal object servers, we eliminated that effect by raising work count and lowering greenlets to 1 (maxclients, I believe) | 21:16 |
| portante | the graphs were published a while, I can dig them up if you'd like | 21:17 |
| *** RackerShagz has quit IRC | 21:24 | |
| *** mwheckmann has quit IRC | 21:25 | |
| *** bsdkurt has quit IRC | 21:35 | |
| *** bsdkurt has joined #openstack-swift | 21:37 | |
| *** dontalton has joined #openstack-swift | 21:38 | |
| *** mmcardle has joined #openstack-swift | 21:41 | |
| *** fthiagogv has quit IRC | 21:45 | |
| *** mmcardle has quit IRC | 21:52 | |
| *** nadeem has quit IRC | 21:53 | |
| *** nadeem has joined #openstack-swift | 21:54 | |
| *** jkugel has quit IRC | 21:54 | |
| *** thurloat_isgone is now known as thurloat | 22:05 | |
| *** nadeem has quit IRC | 22:19 | |
| *** jamielennox|away is now known as jamielennox | 22:20 | |
| *** bill_az has joined #openstack-swift | 22:23 | |
| swifterdarrell | portante: interesting | 22:29 |
| swifterdarrell | portante: I don't have time to mess with too much characterization atm; I'm mostly after A) not worse; and B) demonstrably better when 1 or 2 of your disks really suck | 22:30 |
| swifterdarrell | portante: I'm pretty happy with A so far; and I'm about to target B | 22:30 |
| swifterdarrell | portante: for these 2 hr runs, between initial file setup, the run itself, deletion, and letting object replication catch up, I get maybe 1 run every 4 hrs or so | 22:31 |
| swifterdarrell | portante: in the last 32 hours, I will gotten nearly 6 runs (the evenness of the results in that period says something about my sleep schedule and prioritization of this work); so that's ~5.3 hrs per run | 22:34 |
| *** minwoob has quit IRC | 22:35 | |
| *** bsdkurt has quit IRC | 22:40 | |
| *** bsdkurt has joined #openstack-swift | 22:41 | |
| *** barra204 has joined #openstack-swift | 22:48 | |
| *** barra204 has quit IRC | 22:53 | |
| *** wbhuber has quit IRC | 22:57 | |
| *** km has joined #openstack-swift | 23:01 | |
| *** blmartin has joined #openstack-swift | 23:04 | |
| *** blmartin_ has joined #openstack-swift | 23:04 | |
| *** jrichli has quit IRC | 23:08 | |
| *** kei_yama has joined #openstack-swift | 23:11 | |
| *** chlong has joined #openstack-swift | 23:19 | |
| *** thurloat is now known as thurloat_isgone | 23:22 | |
| MooingLemur | clayg: were you able to backread to yesterday, etc? | 23:31 |
| *** dontalton has quit IRC | 23:37 | |
| *** asettle has quit IRC | 23:38 | |
| *** tellesnobrega has quit IRC | 23:40 | |
| *** wbhuber has joined #openstack-swift | 23:43 | |
| *** asettle has joined #openstack-swift | 23:45 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!