| *** dpak has left #openstack-swift | 00:00 | |
| *** kei_yama has quit IRC | 00:04 | |
| *** sams-gleb has joined #openstack-swift | 00:07 | |
| *** sams-gleb has quit IRC | 00:13 | |
| *** kei_yama has joined #openstack-swift | 00:15 | |
| clayg | so I'm super close to understading how patch 402043 works now! At this point I'm pretty sure I know why it works - I couldn't explain it yet - but i understand now how Pavel's fix is also a correct solution for https://review.openstack.org/#/c/418690/ | 00:16 |
|---|---|---|
| patchbot | https://review.openstack.org/#/c/402043/ - swift - Optimize hash calculation when suffix hash invalid... | 00:16 |
| patchbot | patch 418690 - swift - Fix race in new partitions detecting new/invalid s... | 00:16 |
| *** kei_yama has quit IRC | 00:18 | |
| *** kei_yama has joined #openstack-swift | 00:21 | |
| *** asettle has joined #openstack-swift | 00:35 | |
| *** asettle has quit IRC | 00:38 | |
| *** asettle has joined #openstack-swift | 00:38 | |
| *** asettle has quit IRC | 00:44 | |
| *** jamielennox is now known as jamielennox|away | 00:50 | |
| clayg | ok, I definately understand it now - and I could even explain why it's correct :\ | 00:55 |
| *** jamielennox|away is now known as jamielennox | 01:21 | |
| clayg | timburke: I've got to sign off soon | 01:24 |
| clayg | but I went back and got my list of those patches ... umm what gives with patch 299225 | 01:24 |
| patchbot | https://review.openstack.org/#/c/299225/ - swift - Treat invalid limit parameters as errors | 01:24 |
| clayg | you want me to change my mind? or +1 so there's some sort of officail "if you guys thing this is for the best"? Or do you need me to play the bad guy and -2 that wishlist needs-next-api thing! | 01:25 |
| *** vint_bra has quit IRC | 01:28 | |
| *** tqtran has quit IRC | 01:30 | |
| *** varunsomani has joined #openstack-swift | 01:45 | |
| varunsomani | Hi I am seeing an issue in the swift proxy screen, can anyone help ? | 01:46 |
| *** ukaynar has joined #openstack-swift | 01:51 | |
| *** _JZ_ has quit IRC | 01:51 | |
| varunsomani | Hi I am seeing an issue in the swift proxy server, can anyone help ? | 01:54 |
| mattoliverau | varunsomani: maybe, whats up? | 01:56 |
| varunsomani | I am getting the following message on screen | 01:56 |
| varunsomani | proxy-server: Authenticating user token | 01:56 |
| varunsomani | proxy-server: Invalid user token | 01:56 |
| varunsomani | proxy-server: Deferring reject downstream | 01:56 |
| varunsomani | proxy-server: Received request from user: user_id f2e6b61346b14946b389353b38016b87, project_id 37e383b047fa4f8196b7c13141f33916, roles admin | 01:56 |
| varunsomani | proxy-server: Authorizing as anonymous (txn: tx0c59249a2e0c45bcb5c4a-005876ab43) | 01:56 |
| varunsomani | this happens in the HEAD query | 01:57 |
| varunsomani | In GET query there is no issue. | 01:57 |
| mattoliverau | varunsomani: what are you using for auth? keystone. Because I think that sounds like a problem with keystone. | 01:58 |
| varunsomani | yes keystone | 01:58 |
| varunsomani | but then why is GET working, also i am able to create container and things | 01:58 |
| mattoliverau | varunsomani: has it ever worked or is this having trouble getting it set up? | 01:59 |
| varunsomani | set up | 01:59 |
| mattoliverau | varunsomani: your users is a memeber of one of the operator roles you've set up for Swift in keystone and defined what the operator roles are in the swift proxy keystoneauth config? | 02:02 |
| mattoliverau | *user | 02:02 |
| mattoliverau | and you have the reseller | 02:02 |
| mattoliverau | *and you have the reseller_prefix set up correctly? | 02:03 |
| * mattoliverau is just trying to think of the gotchas I always fall for | 02:03 | |
| varunsomani | i think we are using the default reseller prefix | 02:04 |
| varunsomani | AUTH | 02:04 |
| mattoliverau | varunsomani: and the storage url your using's account starts with AUTH_? | 02:05 |
| varunsomani | should be, I haven't changed that setting. should get the default only. | 02:07 |
| *** sams-gleb has joined #openstack-swift | 02:10 | |
| *** klrmn has quit IRC | 02:12 | |
| mattoliverau | varunsomani: I mean when your hitting the swift endpoint if the account part starting with AUTH_ ie: http://<ip>/v1/AUTH_<account>/? just want to make sure, because it's how the keystone middleware knows what auth plugin to use. | 02:14 |
| *** sams-gleb has quit IRC | 02:14 | |
| varunsomani | yes i think so with the same auth get command is working fine | 02:15 |
| varunsomani | but HEAD is failing | 02:15 |
| mattoliverau | oh right, so it's just the HEAD failing, sorry, failed to read :P | 02:15 |
| varunsomani | no worries | 02:15 |
| varunsomani | it doesn't need any other authentication right ? | 02:16 |
| mattoliverau | nope, if GET wotks, HEAD should too, from an auth point of view | 02:16 |
| varunsomani | yeah then keystone is working as expected i assume. | 02:18 |
| varunsomani | any idea ? | 02:25 |
| mattoliverau | hmm, not sure what could be causing it.. need to think about this. | 02:44 |
| *** ppai has joined #openstack-swift | 02:49 | |
| *** ppai has quit IRC | 02:55 | |
| *** JimCheung has quit IRC | 02:59 | |
| si1ver | I'm the glitch inhalator, herb conjugator, verb modulator.... teleport massive... | 03:06 |
| * si1ver notes that this is the wrong room | 03:06 | |
| *** varunsomani has quit IRC | 03:09 | |
| si1ver | Well hey since I'm in here anyway... Do the tests on the swift3 middleware include aws4 request signing? I read through them but was not sure. | 03:09 |
| *** bkopilov has quit IRC | 03:13 | |
| *** JimCheung has joined #openstack-swift | 03:14 | |
| *** JimCheung has quit IRC | 03:19 | |
| *** links has joined #openstack-swift | 03:39 | |
| *** dmorita has quit IRC | 04:01 | |
| *** psachin has joined #openstack-swift | 04:05 | |
| *** Jeffrey4l__ has joined #openstack-swift | 04:11 | |
| *** sams-gleb has joined #openstack-swift | 04:12 | |
| *** Jeffrey4l_ has quit IRC | 04:14 | |
| *** sams-gleb has quit IRC | 04:16 | |
| *** SkyRocknRoll has joined #openstack-swift | 04:27 | |
| *** bkopilov has joined #openstack-swift | 04:38 | |
| *** ppai has joined #openstack-swift | 04:55 | |
| *** dschultz has quit IRC | 04:56 | |
| *** manij has joined #openstack-swift | 04:58 | |
| *** dmorita has joined #openstack-swift | 05:02 | |
| *** manij has quit IRC | 05:03 | |
| *** dmorita has quit IRC | 05:06 | |
| *** ukaynar has quit IRC | 05:21 | |
| *** dmorita has joined #openstack-swift | 05:27 | |
| *** dmorita has quit IRC | 05:31 | |
| openstackgerrit | Janie Richling proposed openstack/swift: Symlink implementation. https://review.openstack.org/232162 | 05:51 |
| openstackgerrit | Janie Richling proposed openstack/swift: Symlink API documentation https://review.openstack.org/415943 | 05:51 |
| jrichli | just rebasing before squash ^^ | 05:51 |
| openstackgerrit | Janie Richling proposed openstack/swift: Symlink implementation. https://review.openstack.org/232162 | 05:55 |
| jrichli | ugh, the same thing happened and version 43 was used instead of 44 for symlinks.py and others. sorry tdasilva. | 05:59 |
| * jrichli is calling it a night | 06:00 | |
| *** _JZ_ has joined #openstack-swift | 06:03 | |
| *** dmorita has joined #openstack-swift | 06:14 | |
| *** sams-gleb has joined #openstack-swift | 06:14 | |
| *** dmorita has quit IRC | 06:18 | |
| *** sams-gleb has quit IRC | 06:18 | |
| *** tesseract has joined #openstack-swift | 07:14 | |
| *** psachin has quit IRC | 07:27 | |
| *** _JZ_ has quit IRC | 07:29 | |
| *** psachin has joined #openstack-swift | 07:29 | |
| *** winggundamth has joined #openstack-swift | 07:30 | |
| *** ChubYann has quit IRC | 07:32 | |
| *** manous has joined #openstack-swift | 07:44 | |
| *** sams-gleb has joined #openstack-swift | 07:46 | |
| *** logan- has quit IRC | 07:47 | |
| *** hseipp has joined #openstack-swift | 07:55 | |
| *** logan- has joined #openstack-swift | 07:56 | |
| *** dmorita has joined #openstack-swift | 08:02 | |
| *** dmorita has quit IRC | 08:06 | |
| *** rledisez has joined #openstack-swift | 08:10 | |
| *** manij has joined #openstack-swift | 08:21 | |
| *** manij has quit IRC | 08:25 | |
| *** kei_yama has quit IRC | 08:31 | |
| *** winggundamth has quit IRC | 08:33 | |
| *** glb has joined #openstack-swift | 08:37 | |
| *** jordanP has joined #openstack-swift | 08:37 | |
| *** jordanP has quit IRC | 08:37 | |
| *** hseipp has quit IRC | 08:39 | |
| *** geaaru has joined #openstack-swift | 08:49 | |
| *** david-lyle has quit IRC | 09:01 | |
| *** manous has quit IRC | 09:09 | |
| *** mvk has quit IRC | 09:18 | |
| *** david-lyle has joined #openstack-swift | 09:24 | |
| mahatic | timburke: clayg : thank you for the update/review on patch 374419! | 09:30 |
| patchbot | https://review.openstack.org/#/c/374419/ - swift - Move documented reclaim_age option to correct loca... | 09:30 |
| *** JimCheun_ has joined #openstack-swift | 09:31 | |
| *** jordanP has joined #openstack-swift | 09:33 | |
| *** JimCheun_ has quit IRC | 09:35 | |
| *** Shashikant86 has joined #openstack-swift | 09:39 | |
| *** asettle has joined #openstack-swift | 09:41 | |
| *** mvk has joined #openstack-swift | 09:51 | |
| *** cbartz has joined #openstack-swift | 10:22 | |
| *** sams-gleb has quit IRC | 10:31 | |
| *** sams-gleb has joined #openstack-swift | 10:32 | |
| *** mvk has quit IRC | 10:36 | |
| *** sams-gleb has quit IRC | 10:36 | |
| *** acoles_ is now known as acoles | 10:43 | |
| *** furlongm_ has quit IRC | 10:46 | |
| *** furlongm has joined #openstack-swift | 10:47 | |
| *** mvk has joined #openstack-swift | 10:48 | |
| *** sams-gleb has joined #openstack-swift | 10:51 | |
| *** Jeffrey4l__ is now known as Jeffrey4l | 10:58 | |
| *** Shashikant86 has quit IRC | 10:58 | |
| *** dmorita has joined #openstack-swift | 11:18 | |
| *** dmorita has quit IRC | 11:23 | |
| *** ganders has joined #openstack-swift | 11:35 | |
| *** vint_bra has joined #openstack-swift | 11:52 | |
| *** dmorita has joined #openstack-swift | 11:55 | |
| *** dmorita has quit IRC | 11:59 | |
| *** bkopilov has quit IRC | 11:59 | |
| *** jordanP has quit IRC | 12:07 | |
| *** jordanP has joined #openstack-swift | 12:08 | |
| *** links has quit IRC | 12:13 | |
| *** tmoreira has quit IRC | 12:15 | |
| *** sheel has joined #openstack-swift | 12:19 | |
| *** tmoreira has joined #openstack-swift | 12:19 | |
| *** ppai has quit IRC | 12:26 | |
| *** SkyRocknRoll has quit IRC | 12:38 | |
| tdasilva | good morning | 13:17 |
| tdasilva | acoles: you around? | 13:17 |
| acoles | tdasilva: yes | 13:17 |
| tdasilva | acoles: hi! so I'm trying to understand the use of 'X-Object-Sysmeta-Container-Update-Override-Etag'. I see how in the encrypter we check to see if it has been set, so that it can be added to the footer | 13:18 |
| tdasilva | but i'm still trying to understand who currently sets that header in the first place? | 13:19 |
| acoles | tdasilva: no one IIRC, unless it got added in SLO - timburke had that in mind SLO using it | 13:20 |
| acoles | so it is just used by the encrypter | 13:20 |
| tdasilva | acoles: is that the header used by EC proxy controller to send the "correct" etag to the container listing thou? | 13:21 |
| acoles | tdasilva: no (I was just refreshing my memory on that) | 13:22 |
| acoles | tdasilva: https://github.com/openstack/swift/blob/e8a80e874a5086e94c5ae93e3a9191cb813d1631/swift/proxy/controllers/obj.py#L1822-L1829 | 13:22 |
| acoles | for b/wards compatibility we left EC using the older Backend- style override headers | 13:23 |
| acoles | The object server handles both and applies correct precedence | 13:23 |
| acoles | But for new use cases X-Object-Sysmeta-Container-Update-Override-Etag is the correct one to use | 13:24 |
| acoles | tdasilva: Is this related to sending the symlink target to the container server? | 13:24 |
| tdasilva | acoles: yes, i guess i'm trying to understand if another middleware could override whatever value is put in there | 13:25 |
| tdasilva | ? | 13:25 |
| *** links has joined #openstack-swift | 13:25 | |
| acoles | It can but IIRC the left most middleware (closest to client) takes precedence, so if symlink was very last in pipeline then it could be smart and append to the existing etag, but otherwise the symlink supplied value might get replaced by a later middeware | 13:27 |
| acoles | sorry, that's wrong | 13:27 |
| acoles | I mean if symlink was before encryption and supplied a value then it would override the etag value that encryption wished to set | 13:28 |
| acoles | It's assumed that the left most middleware is the authority for the etag value that should go to container server update | 13:29 |
| psachin | clayg: Can you please review 1) https://review.openstack.org/406012 2) https://review.openstack.org/415473 | 13:29 |
| patchbot | patch 406012 - swift - Fix swift-get-nodes arg parsing for missing ring | 13:29 |
| patchbot | patch 415473 - swift - Proxy server controller tests should be reorganized | 13:29 |
| tdasilva | acoles: are you referring to this: https://github.com/openstack/swift/blob/master/swift/common/middleware/crypto/encrypter.py#L144,L152 | 13:32 |
| acoles | tdasilva: yes, so an override header or footer would take precedence over the usual etag header when the encrypter decides what to encryt and then pass in the override footer | 13:38 |
| tdasilva | acoles: and in the case of symlink, the expectation is that the etag being set is always the MD5_OF_EMPTY_STRING so, crypto should not try to set (encrypt) anything anyway, right? | 13:38 |
| tdasilva | acoles: see line 155 just below... | 13:38 |
| *** bkopilov has joined #openstack-swift | 13:39 | |
| tdasilva | acoles: will just need to modify that to account for 'etag;param=value' | 13:39 |
| acoles | right, the encrypter would need to first parse the etag for possible params | 13:40 |
| acoles | tdasilva: what I'd like to suggest for consideration is that symlink uses a separate header to pass the target to the container server, and not abuse etag. I *think* we have support for that. | 13:41 |
| acoles | see this test https://github.com/openstack/swift/blob/3bf011b0e313b174e98ca59774420c1527cceb47/test/unit/obj/test_server.py#L5044-L5044 | 13:41 |
| tdasilva | acoles: well..that was going to be one of my next questions...why can't we just comeup with another header to pass to the container server? I'm guessing we would want to limit that | 13:42 |
| acoles | IIRC I considered doing that to handle swift_bytes during fast post container updates, but in the end we always have to support swift_bytes being appended to content-type so there was no point. But for this new use case I think it is worth consideration. | 13:43 |
| acoles | So if symlink adds X-Object-Sysmeta-Container-Update-Override-Symlink-Target=a/c/o' then I think the container update will get and X-Symlink-Target=a/c/o header | 13:44 |
| acoles | that's assuming we made it plumb all the way through, but the object server seems to support that | 13:45 |
| acoles | See https://github.com/openstack/swift/blob/2bdf61fadda655323a36185f921f8a3e183bcdf9/swift/obj/server.py#L459-L495 | 13:46 |
| acoles | So then the container server needs to be made to handle that new header, and it might do that by appending to etag in the db, but that choice is confined to the container server impl and we don't need to handle etags with possible params everywhere. | 13:48 |
| tdasilva | acoles: ok, trying to take a look at container code atm to see if i understand it all | 13:54 |
| * acoles afk | 14:02 | |
| openstackgerrit | Janie Richling proposed openstack/swift: Symlink implementation. https://review.openstack.org/232162 | 14:07 |
| *** klamath has joined #openstack-swift | 14:10 | |
| *** klamath has quit IRC | 14:10 | |
| *** klamath has joined #openstack-swift | 14:11 | |
| rledisez | hello all | 14:13 |
| rledisez | does anyone have tips to run the swift functional tests on a specific cluster? i tried to update sample.conf, but it seems it was ignored. i can't find anything except ops guide telling me i should already be doing it :) | 14:13 |
| rledisez | http://docs.openstack.org/developer/swift/ops_runbook/diagnose.html#functional-tests-usage | 14:13 |
| tdasilva | rledisez: you need to have a test.conf file | 14:14 |
| tdasilva | and there's a env. variable to update that points to that test.conf file | 14:15 |
| tdasilva | one sec | 14:15 |
| tdasilva | rledisez: basically something similar to this: https://thiagodasilvablog.wordpress.com/2016/11/02/running-functional-test-against-an-openstack-swift-cluster-deployed-with-tripleo/ | 14:16 |
| rledisez | tdasilva: great, thank you! it does things now :) | 14:19 |
| *** psachin has quit IRC | 14:28 | |
| tdasilva | acoles: would there be any reason to *not* add X-Symlink-Target to save headers? https://github.com/openstack/swift/blob/master/swift/container/server.py#L83 | 14:28 |
| tdasilva | acoles: instead of appending to the etag header | 14:28 |
| openstackgerrit | Janie Richling proposed openstack/swift: Symlink implementation. https://review.openstack.org/232162 | 14:36 |
| jrichli | finally worked ^^ | 14:39 |
| tdasilva | jrichli: :) | 14:40 |
| acoles | tdasilva: not sure what you are thinking. save_headers only applies to requests to container path, not object path updates? | 14:40 |
| acoles | tdasilva: you'd probably be looking to modify the object update PUT handler around here https://github.com/openstack/swift/blob/9d29ca1c766f6e846dc7a8b8730fd76d304acfe4/swift/container/server.py#L370-L370 | 14:42 |
| acoles | to consume the x-symlink-target header and append to etag internally to the container server | 14:42 |
| tdasilva | oh i see, sorry, was looking at line 393 below | 14:43 |
| tdasilva | acoles: i think i'm caught up now, thanks | 14:50 |
| acoles | tdasilva: and then in GET listing path maybe extract the symlink value from the etag value in update_record https://github.com/openstack/swift/blob/9d29ca1c766f6e846dc7a8b8730fd76d304acfe4/swift/container/server.py#L441-L441 | 14:50 |
| acoles | tdasilva: if we went this way it might be smart to make the container server handling generic i.e. any extra X-* headers get mapped to a key value param stored in the db (appended to etag) and then in GET listing path they are extraced and added as items to the listing. | 14:53 |
| acoles | tdasilva: the disadvantage of this approach is modifying the container server. the advantage is not having to consider any effects of having extra params tacked onto the etag throughout the request paths. | 14:54 |
| tdasilva | acoles: i guess i'm just wondering if we want to have a generic handling like that if it would make sense to extend the db to add a "obj metadata" column? so we don't have to keep appending to etag?? i don't know.... | 14:55 |
| tdasilva | acoles: plus i honestly don't know how "easy" it is to just add a column to the database in terms of upgrades and all that | 14:56 |
| ganders | i'm trying to upgrade version from 2.7.0 to 2.12.1, and Im getting the following error msg while trying to start the account-server deamons: http://pastebin.com/raw/1ZpyRTLA any ideas? | 14:56 |
| tdasilva | ganders: what version of pyeclib do you have installed? | 14:58 |
| tdasilva | oh ImportError: No module named middleware.recon | 14:59 |
| ganders | tdasilva: 1.2.0-1~cloud0 | 14:59 |
| acoles | tdasilva: I have been wondering the same, and have the same uncertainty re how "easy" that is. | 15:00 |
| tdasilva | acoles: and then there's also the question of POST updates, right? | 15:01 |
| acoles | tdasilva: yeah, this should only be for object metadata that is immutable i.e. sysmeta as it currently is. | 15:02 |
| *** chlong has joined #openstack-swift | 15:02 | |
| acoles | tdasilva: that was the challenge with content-type, that it can be modified by a post but also is in container db | 15:03 |
| tdasilva | acoles: right | 15:03 |
| acoles | tdasilva: we should avoid needing to handle that scenario again | 15:03 |
| ganders | ok, got the middleware.recon issue.. now getting: http://pastebin.com/raw/CF0CW6z6 | 15:04 |
| tdasilva | ganders: healthcheck? | 15:05 |
| ganders | add the lines on account-server.conf resolved | 15:05 |
| ganders | liberasurecode[13611]: liberasurecode_backend_open: dynamic linking error libisal.so.2: cannot open shared object file: No such file or directory | 15:05 |
| ganders | liberasurecode[13611]: liberasurecode_backend_open: dynamic linking error libshss.so.1: cannot open shared object file: No such file or directory | 15:05 |
| ganders | only those errors now | 15:06 |
| tdasilva | ganders: for that i believe you should just need to upgrade to the latest pyeclib/libec. | 15:06 |
| *** sheel has quit IRC | 15:07 | |
| tdasilva | acoles: in terms of symlink that should be ok because we require a PUT for setting new target path, no POSTs, but any new metadata being added there would basically need to follow the same pattern | 15:07 |
| acoles | tdasilva: another thing to bear in mind is that etags may also have a param appended from encrypter (swift_meta) so any parsing of etag for params needs to be aware of that. | 15:08 |
| tdasilva | acoles: hmm...that's the case today?? | 15:08 |
| acoles | tdasilva: yep, timburke reminded me of that (only PUT) yesterday | 15:09 |
| tdasilva | so in a listing, is that returned to the user? or removed by the middleare on the way back? | 15:09 |
| acoles | it is removed in the decrypter | 15:09 |
| tdasilva | oh i see | 15:09 |
| acoles | hmmm, gets me thinking of another approach (maybe this is what timburke is already thinking)... | 15:09 |
| tdasilva | heh | 15:10 |
| acoles | tdasilva: if we generalised the encrypter's use of swift-meta in proxy middleware, so any middleware can put a key-val into a swift-meta dict that gets appended to the override etag in proxy server | 15:11 |
| acoles | and encrypter uses that mechanism itself | 15:11 |
| acoles | so symlink would add target=a/c/o to a dict in request environ and proxy takes care of appending that to the override etag, and extracting the dict in GET/HEAD path. | 15:14 |
| acoles | Now I have remembered that encrypter adds a param to etag I realise we already have dealt with any possible side effects of that in request path. (presumably!) | 15:16 |
| acoles | But I also know that we made the update override header mechanism generic for precisely this kind of use case :) | 15:17 |
| *** manous has joined #openstack-swift | 15:21 | |
| tdasilva | acoles: sorry, did we build two mechanism to accomplish the same thing? | 15:24 |
| acoles | tdasilva: hehe | 15:24 |
| acoles | tdasilva: not quite but I think we may have laid the foundations for two | 15:24 |
| acoles | or there is third way, doing something bespoke for symlink targets! | 15:25 |
| tdasilva | hehehehehe | 15:25 |
| *** ukaynar has joined #openstack-swift | 15:27 | |
| *** sams-gleb has quit IRC | 15:28 | |
| *** sams-gleb has joined #openstack-swift | 15:28 | |
| tdasilva | acoles: reading up the code in encrypter....in fact, swift-meta gets appended to every user metadata that gets encrypted, right? | 15:30 |
| acoles | yes | 15:30 |
| tdasilva | it contains the information necessary to decrypt it later | 15:31 |
| acoles | exactly | 15:31 |
| acoles | different content for each item though | 15:31 |
| acoles | different swift_meta content I mean, at least soem of it differes | 15:31 |
| *** _JZ_ has joined #openstack-swift | 15:32 | |
| tdasilva | understood | 15:32 |
| acoles | So the "generic" idea I decribed would need to use more precise names than just swift_meta for whatever accumulated the specific meta to be appended to an etag. | 15:32 |
| tdasilva | acoles: in the case of X-Object-Sysmeta-Container-Update-Override-Etag, the etag that is sent to the container is actually an encrypted value, but it contains the swift_meta necessary to decrypt on the way out | 15:32 |
| *** sams-gleb has quit IRC | 15:33 | |
| acoles | tdasilva: yes, to be precise the header value is of form <encrypted etag>;swift_meta=... | 15:34 |
| acoles | i.e. the swift_meta is not encrypted | 15:34 |
| acoles | tdasilva: this conversation has helped me get up to speed on the topic. I'm sure timburke will roll his eyes when he gets in and tell me how it needs to work ;) | 15:34 |
| tdasilva | acoles: hehehe, thank you for taking the time to walk me through | 15:35 |
| ganders | tdasilva: got the pyeclib issue resolved.. all services are now up and running... is there any way to see if the services are running with the 'new' version of swift? | 15:37 |
| tdasilva | ganders: /info will return the version you are running, but i think that's just what the proxy is running | 15:40 |
| tdasilva | does recon return version info? | 15:41 |
| *** hseipp has joined #openstack-swift | 15:43 | |
| *** mvk has quit IRC | 15:44 | |
| *** sams-gleb has joined #openstack-swift | 15:47 | |
| *** klrmn has joined #openstack-swift | 15:50 | |
| tdasilva | ganders: this worked for me: curl -i http://localhost:6012/recon/version | 15:54 |
| tdasilva | it is not clear to me if swift-recon cli tool returns this info | 15:54 |
| acoles | tdasilva: you're welcome | 15:55 |
| ganders | tdasilva: works great thanks :) {"version": "2.12.1.dev19"} | 15:56 |
| *** bkopilov has quit IRC | 15:56 | |
| *** pcaruana has joined #openstack-swift | 15:58 | |
| *** oshritf has quit IRC | 16:04 | |
| ganders | tdasilva: after performing the upg version and tuning some opts, the disks remain at 100%, logically if i stop the -replicator and -auditor daemons, the pct drops to 10-12% of usage, the prob is that i cannot leave stoped those daemons for a long time, right? is there any rule out there? i mean, is possible to set a script in order to start the daemons (repl and auditor) at certain time (for ex when there's less load) and then at the most he | 16:09 |
| ganders | avily loaded schedule, stoped? | 16:09 |
| *** silor has joined #openstack-swift | 16:26 | |
| *** bkopilov has joined #openstack-swift | 16:33 | |
| *** links has quit IRC | 16:37 | |
| *** chsc has joined #openstack-swift | 16:38 | |
| *** chsc has joined #openstack-swift | 16:38 | |
| *** chlong has quit IRC | 16:52 | |
| tdasilva | ganders: i don't think you should leave them stopped for a long time as you risk losing data, but rledisez did mention tuning them, i guess things like how often they run... | 16:53 |
| ganders | what would be the 'best' tuning for that? | 16:58 |
| *** adu has quit IRC | 16:59 | |
| rledisez | ganders: are you using replica or erasure code policy? | 17:00 |
| *** arch-nemesis has joined #openstack-swift | 17:00 | |
| *** chsc has quit IRC | 17:03 | |
| *** peterlisak has quit IRC | 17:08 | |
| *** peterlisak has joined #openstack-swift | 17:09 | |
| *** jarbod_ has quit IRC | 17:10 | |
| *** sheel has joined #openstack-swift | 17:10 | |
| *** jarbod_ has joined #openstack-swift | 17:10 | |
| *** ukaynar has quit IRC | 17:11 | |
| *** ganders has quit IRC | 17:13 | |
| *** adu has joined #openstack-swift | 17:16 | |
| *** cbartz has quit IRC | 17:17 | |
| notmyname | good morning | 17:17 |
| *** oshritf has joined #openstack-swift | 17:17 | |
| *** oshritf has quit IRC | 17:18 | |
| *** ukaynar has joined #openstack-swift | 17:18 | |
| *** chosafine has joined #openstack-swift | 17:23 | |
| *** hseipp has quit IRC | 17:26 | |
| *** chlong has joined #openstack-swift | 17:26 | |
| *** silor has quit IRC | 17:29 | |
| *** chsc has joined #openstack-swift | 17:29 | |
| *** chsc has joined #openstack-swift | 17:29 | |
| *** ganders has joined #openstack-swift | 17:31 | |
| ganders | rledisez: replica | 17:33 |
| *** ukaynar has quit IRC | 17:36 | |
| rledisez | ganders: in object-auditor, reduce files_per_second, set zero_byte_files_per_second to 0, increase disk_chunk_size, in object-replicator use SSYNC, increase interval, in account/container auditor play reduce account/container per second, in account/container replicator increase interval | 17:38 |
| rledisez | ganders: just set reasonable values, don't set an interval of one week ;) | 17:38 |
| *** chsc has quit IRC | 17:39 | |
| *** silor has joined #openstack-swift | 17:39 | |
| ganders | rledisez: thanks a lot :) will try that out | 17:40 |
| *** dmorita has joined #openstack-swift | 17:42 | |
| *** chsc has joined #openstack-swift | 17:48 | |
| *** chsc has quit IRC | 17:48 | |
| *** chsc has joined #openstack-swift | 17:48 | |
| *** manous has quit IRC | 17:51 | |
| *** rledisez has quit IRC | 17:53 | |
| *** silor has quit IRC | 17:53 | |
| *** sams-gleb has quit IRC | 17:53 | |
| *** JimCheung has joined #openstack-swift | 17:58 | |
| *** JimCheung has quit IRC | 17:58 | |
| *** JimCheung has joined #openstack-swift | 17:59 | |
| *** jordanP has quit IRC | 18:03 | |
| *** bkeller` has joined #openstack-swift | 18:08 | |
| *** mvk has joined #openstack-swift | 18:09 | |
| openstackgerrit | Alistair Coles proposed openstack/swift: Support last modified on listing containers https://review.openstack.org/198634 | 18:09 |
| openstackgerrit | Alistair Coles proposed openstack/swift: Trivial follow up to addition of last modified in container listings https://review.openstack.org/419579 | 18:09 |
| *** chlong has quit IRC | 18:19 | |
| *** ukaynar_ has joined #openstack-swift | 18:26 | |
| clayg | acoles: mahatic: Is there really a behavior change in patch 374419? It's not documented in tests if there is one lurking? | 18:27 |
| patchbot | https://review.openstack.org/#/c/374419/ - swift - Move documented reclaim_age option to correct loca... | 18:27 |
| clayg | we don't... "ignore" anything now or after the patch? we use the value that gets read into conf and give that to the DiskFile? | 18:27 |
| clayg | ... if you are already on master setting reclaim_age in default you are golden - you don't need my stupid cleanup/plumbing + doc fix patch | 18:28 |
| clayg | if you have different services configured with different relcaim ages (like anyone following the current docs would) then you're getting some undefined behavior - I don't really want to detail all of that because it's not what anyone wants | 18:29 |
| clayg | the only sane thing we support is one reclaim_age everywhere | 18:29 |
| clayg | if you set it in DEFAULT - you're done | 18:29 |
| clayg | if you want to set it every individual config section - by all means | 18:29 |
| acoles | clayg: look at the call from finalize_put to cleanup_ondisk_files https://github.com/openstack/swift/blob/8737ebe519110cbe8fa5ba5fba945de0d627142f/swift/obj/diskfile.py#L1497-L1497 | 18:29 |
| clayg | if you want to set it in DEFAULT and also in the config section... ummm ok | 18:29 |
| acoles | in that case cleanup_ondisk_files on master does not use any configured value, it always uses one week. | 18:30 |
| clayg | acoles: well - see that's a whole different bug! | 18:30 |
| acoles | lol | 18:30 |
| clayg | acoles: your beef isn't docs/commitmessage/upgradeimpact | 18:30 |
| clayg | it's "you can't set reclaim_age for object servers" | 18:30 |
| clayg | I fixed it on accident (knew cleanup of passing around that stupid internal state was a good idea) - and I should write a damn test to prove it! | 18:31 |
| acoles | well the patch fixes that right, so every call to cleanup_ondisk_files uses the conf value of reclaim_age. that's good, but it is a behavioural change | 18:31 |
| clayg | I really thought I was good2go when I moved my reclaim_age setting to DEFAULT | 18:31 |
| *** chlong has joined #openstack-swift | 18:32 | |
| clayg | i even bumped my default deploy reclaim age to 6 wks to try and make it harder to get dark data | 18:32 |
| *** chosafine has quit IRC | 18:32 | |
| clayg | with the "always be reclaiming tombstones" fix that mahatic did I've seen tombstones go down overall | 18:33 |
| acoles | clayg: I think mahatic is arguing a different point to mine (and i'm not sure I agree with her) | 18:33 |
| *** openstackgerrit has quit IRC | 18:33 | |
| clayg | acoles: well... i'm just saying your point would be more clear if you stated it "this patch introduces a behavior change and a behavior change should have a test" | 18:33 |
| *** chosafine has joined #openstack-swift | 18:33 | |
| clayg | I was reading tim's comments and started thinking "yeah, he's right, there *can't* be a behavior change in this patch i haven't looked at in months - if there was a behavior change there would be a test for it!" | 18:34 |
| *** pcaruana has quit IRC | 18:34 | |
| clayg | acoles: thanks as always for catching my goofs ;) | 18:34 |
| acoles | clayg: "I think there is a change in behaviour..." https://review.openstack.org/#/c/374419/14//COMMIT_MSG@33 | 18:35 |
| patchbot | patch 374419 - swift - Move documented reclaim_age option to correct loca... | 18:35 |
| *** chosafine has quit IRC | 18:36 | |
| acoles | clayg: the test would be weird cos we'd uimmediately say well that's a bug thats being asserted i.e. set reclaim_age to zero in DEFAULT and you'll get no tombstones | 18:36 |
| clayg | yeah i read that and was like "oh, so there *is* a behavior change" - but then I still couldn't spot it - i think i was looking at the diff on my phone instead of the current source code (your github link is more clear; i'm adding it to my -1) | 18:37 |
| *** oshritf has joined #openstack-swift | 18:37 | |
| clayg | sorry to make a fuss about "you shoud have said" this or that - that's silly - the point is you spotted it - well done | 18:37 |
| acoles | clayg: funnt thing is i was just thining, ummm maybe i should knock up a test to prove it :) | 18:37 |
| acoles | thinking* | 18:38 |
| clayg | acoles: wait... so you're saying the only way to prove that we don't pass reclaim age into _finalize_put is if we use a reclaim_age of 0? | 18:38 |
| clayg | what if you do a delete with an x-timestamp or something and then it reaps the tombstone anyway ahead of the configured reclaim age? | 18:38 |
| acoles | or put a sleep into the cleanup | 18:39 |
| clayg | ... well you could mock the return value of time; but that makes it feel less strong - I don't think anyone is going to set reclaim_age to 0 in any config section anywhere on a production system - sorry but no ops I know would do that "just to prove blah balh" | 18:40 |
| clayg | all the ops I know are way more paranoid than I am | 18:40 |
| clayg | I'll be like "it's fine; it won't hurt anything; that's stupid there's no way the code would ever do that" and they'll be like "well better safe than sorry" - they know better than to trust me | 18:41 |
| clayg | not passig reclaim_age into finalize_put still feels like a bug tho | 18:41 |
| clayg | but if it's benign maybe I'm less worried about it overall | 18:42 |
| acoles | the patch fixes it and on master IDK if it was a bug or just a useful feature, on master it prevents ever cleaning up a tombstone you just wrote. | 18:42 |
| clayg | now that you mention it... I feel like I've set reclaim_age to 0 and had delete's not write tombstones? was it while testing this patch/ | 18:43 |
| acoles | yes!!! | 18:44 |
| acoles | I did same this morning, which proves that devs are insane in a way that ops aren't | 18:44 |
| acoles | Ideally we'd have the cleanup code be smart and never delete the tombstone it just wrote | 18:45 |
| clayg | why? it's gunna happen next cleanup_hash_dir anyway? | 18:45 |
| clayg | acoles: will it stil delete the .data file? | 18:46 |
| clayg | cause it sounds sorta like exactly what you'd expect reclaim_age of 0 to do? :P | 18:46 |
| clayg | well that or just not write the tombstone file at all | 18:46 |
| acoles | thats what I saw, empty dir after the delete | 18:46 |
| clayg | wfm | 18:46 |
| clayg | and your sure that doesn't happen on master (i'm going to test it anyway - so you don't have to answer) | 18:47 |
| acoles | but it would not get reclaimed in next cleanup call if the replicator is config'd as spec'd on master today so the replicator would pass in a sensible reclaim_age to the cleanup code | 18:47 |
| clayg | having different reclaim_age for different object-* services is terrible sin I'm trying to paper over by suggestion configuration of the value once in DEFAULT | 18:48 |
| acoles | clayg: the patch is good. I just get all worried about an op out there who did this once upon a time http://paste.openstack.org/show/594755/ , that is the extent of my concern, nothing wrong with the patch, just me worrying about crazy stuff in the wild. | 18:49 |
| clayg | object server with 1 wk replicator with 2 wk is silly/bad - the zero case is just a pathologicial version of that - and not super explicitly worried about reclaim_age 0 having a particuarlly odd (dangerous?) beahvior | 18:49 |
| clayg | we're talking past each other | 18:50 |
| clayg | I hadn't realized that config would do one thing on master and a different thing on this patch | 18:51 |
| *** geaaru has quit IRC | 18:51 | |
| clayg | i understand now that you believe it would | 18:51 |
| clayg | we will prove it with a test | 18:51 |
| clayg | once all behavoir changes are documented with a test - it should be more clear what the upgrade impact will be | 18:51 |
| *** tesseract has quit IRC | 18:51 | |
| *** bkeller` has quit IRC | 18:51 | |
| *** tqtran has joined #openstack-swift | 18:51 | |
| clayg | I'm more worried about my current configs of reclaim_age 6wks in [DEFAULT] not doing the right thing in finalize_put | 18:52 |
| clayg | but I guess you're saying that unless x-timestamp is very very close to reclaimage expiration anyway there's no behavior change - it only effects the finalize_put call on the object-server not the get_hashes/REPLICATE calls? | 18:52 |
| *** sams-gleb has joined #openstack-swift | 18:53 | |
| clayg | acapgcoah LookupError: Entry point 'symlink' not found in egg 'swift' | 18:54 |
| clayg | ^ that's not merged on master yet! ;) | 18:54 |
| tdasilva | ? | 18:55 |
| clayg | tdasilva: i'd ust switched branches and my proxies wouldn't start because I had configured symlink middleware to test the conditional get's stuff | 18:56 |
| tdasilva | oh lol | 18:58 |
| *** sams-gleb has quit IRC | 18:59 | |
| *** vern has quit IRC | 18:59 | |
| clayg | acoles: ok, yeah I agree the behavior on master and this patch are different :D | 19:01 |
| *** oshritf has quit IRC | 19:03 | |
| acoles | clayg: https://gist.github.com/46a67c833d959fcb55cec785c06e20e1 | 19:09 |
| *** oshritf has joined #openstack-swift | 19:14 | |
| *** oshritf has quit IRC | 19:18 | |
| timburke | clayg: on https://review.openstack.org/#/c/299225/ -- did you review it previously? i didn't see anything about it. i also don't know why we'd say needs-next-api -- 400 on a bad param instead of silently ignoring it seems like a reasonable change | 19:21 |
| patchbot | patch 299225 - swift - Treat invalid limit parameters as errors | 19:21 |
| clayg | timburke: sorry i was just reading the history on the associated bug report | 19:22 |
| timburke | there were definitely problems with it before i re-wrote most of it back in october, but now it seems like it could merge? | 19:22 |
| clayg | torgomatic: also had a recomendation that some pre-existing lack of validation may lead to clients of past doing all kinds of unknown weird things | 19:23 |
| *** acoles is now known as acoles_ | 19:23 | |
| timburke | oh, yeah, that bug report seems rather vague | 19:23 |
| clayg | acoles_: thanks for the test! | 19:24 |
| clayg | timburke: the bug is tagged needs-new-api ? | 19:25 |
| clayg | timburke: does the patch add a new api version1? | 19:25 |
| timburke | ...i disagree? seems reasonable to me that when we document that we expect an integer value, we can change it to error out if you *don't* provide one. | 19:27 |
| clayg | timburke: i *know* you disagree - and better yet - I *know* I might be wrong - i recognize the whole thing is scale and I've just got too much personal experience that in-action has the least negative consequences | 19:28 |
| clayg | sure some dev might stare a screen for 10 mins only to realize they were sending garbage to limit - but they'll curse me; they'll curse themselves; they'll get over it | 19:29 |
| clayg | OTOH if I start returning 400 and some *user* has their client break because BigCorp upgraded their cluster - we're jersk "breaking API" they'll screen | 19:29 |
| clayg | *scream | 19:29 |
| clayg | you can point at the spec all day long; but behavior trumps spec every time | 19:30 |
| clayg | so.. I'd -2 the patch - but totally recognize that's just my judgement | 19:30 |
| clayg | so... do what can I do to help you with that patch? | 19:30 |
| *** oshritf has joined #openstack-swift | 19:31 | |
| *** ChubYann has joined #openstack-swift | 19:31 | |
| clayg | in particular ''.isdigit() was pointed out by sam as being particuarlly scary from the breaking POV | 19:32 |
| clayg | maybe something in wsgi makes it so `/a/c?limit=` doens't result in a 400 - but I don't see a test to prove it | 19:32 |
| clayg | some client somewhere sent `/a/c?limit=` and expected it to be ignored for sure | 19:33 |
| clayg | but why do I care? I can just sit here and not change it and let the bug be tagged next-api until the cows come home | 19:33 |
| clayg | it's easy... watch | 19:33 |
| clayg | ;) | 19:33 |
| *** oshritf has quit IRC | 19:36 | |
| timburke | clayg: i guess go -1 it while asking for a test with a blank limit? look a the code, that will continue to be ignored | 19:36 |
| timburke | it's not wsgi, it's us, and it's in the diff | 19:36 |
| clayg | oic, the __nonzero__ if | 19:37 |
| *** oshritf has joined #openstack-swift | 19:39 | |
| clayg | timburke: ok, so if we want to remove the needs-new-api tag from the bug we should bring it up at the swift meeting - i'll try not to be there ;) | 19:39 |
| clayg | timburke: i can't help but imagine some clint sending '?limit=None' without realizing it :\ sorry | 19:40 |
| *** chlong has quit IRC | 19:41 | |
| clayg | *my* code has but None in X-Backend-Frag-Index before when I didn't mean too :\ | 19:41 |
| clayg | it would have been better if it had been ignored - isntead we stuffed it a string 'None' and it sucked :\ | 19:41 |
| clayg | "liberal in what you accept" | 19:41 |
| timburke | si1ver: yes, swift3 includes support (and tests) for v4 signatures. it currently requires keystone, however. i've got plans to add support to tempauth and swauth, but haven't gotten around to it | 19:42 |
| clayg | there's lots of stuff in rfc 2616 that used to say like "if a server doesn't not recognize it may ignore..." kinda stuff | 19:42 |
| *** ukaynar_ has quit IRC | 19:45 | |
| *** JimCheung has quit IRC | 19:46 | |
| *** oshritf has quit IRC | 19:48 | |
| timburke | "liberal"...like accepting (and writing down!) 'None' when it totally should have been an int? i am *way* more interesting in validating user input early & often than saying "well, they probably didn't mean to send that" | 19:49 |
| timburke | at least now i've got an explanation for https://review.openstack.org/#/c/219165/43/swift/obj/reconstructor.py@273 ... | 19:49 |
| patchbot | patch 219165 - swift - EC Fragment Duplication - Foundational Global EC C... | 19:49 |
| *** arch-nemesis has quit IRC | 19:50 | |
| *** ganders has quit IRC | 19:52 | |
| timburke | acoles_: even if we use a separate header to relay the symlink target to the container server, we've still gotta *store* it somewhere. stuffing it in etag seems lower-friction than a db migration, and it means there's some window of compatibility in case your proxies upgrade before your storage nodes | 19:53 |
| *** chlong has joined #openstack-swift | 19:56 | |
| -openstackstatus- NOTICE: Gerrit will be offline between now and 20:30 for scheduled maintenance: http://lists.openstack.org/pipermail/openstack-dev/2017-January/109910.html | 20:10 | |
| *** ChanServ changes topic to "Gerrit will be offline between now and 20:30 for scheduled maintenance: http://lists.openstack.org/pipermail/openstack-dev/2017-January/109910.html" | 20:10 | |
| clayg | notmyname: how do I work https://wiki.openstack.org/wiki/Meetings/Swift - timburke wants to untag 'needs-new-api' on lp bug #1254638 - I think he's gotta a pretty strong case that the associated fix is reasonable and I want others to chime in so that folks can feel comfortable in telling me I worry too much | 20:13 |
| openstack | Launchpad bug 1254638 in OpenStack Object Storage (swift) "Some bad parameters do not return 4xx" [Wishlist,In progress] https://launchpad.net/bugs/1254638 - Assigned to Ankur Jain (j-ankur) | 20:13 |
| notmyname | clayg: top right corner, log in. then return to the page and top left corner to edit. typey typey and hit publish | 20:13 |
| clayg | oic - "next meeting" it already has an empty bullet point! | 20:14 |
| clayg | notmyname: thanks! | 20:14 |
| clayg | timburke: https://wiki.openstack.org/wiki/Meetings/Swift boom! you got this bro! | 20:15 |
| *** oshritf has joined #openstack-swift | 20:26 | |
| *** adu has quit IRC | 20:27 | |
| *** oshritf has quit IRC | 20:30 | |
| clayg | hey!? | 20:32 |
| clayg | gerrit is down? | 20:32 |
| clayg | did i miss a notification? | 20:32 |
| zaitcev | "Gerrit will be offline until 20:45 for scheduled maintenance (running longer than anticipated): http://lists.openstack.org/pipermail/openstack-dev/2017-January/109910.html" | 20:32 |
| clayg | bah, unread in my inbox :'( | 20:33 |
| clayg | I've been behind! :P | 20:33 |
| clayg | zaitcev: thanks | 20:33 |
| zaitcev | clayg: np. BTW, did you see this https://groups.google.com/forum/#!topic/golang-nuts/cR5bWfLroiQ | 20:34 |
| clayg | zaitcev: how you doing? what's the deal with the golang something binding somewhere with 6200? | 20:34 |
| clayg | zaitcev: omg we are *so* going to get dinged for abusing http like this - did anyone respond? | 20:34 |
| -openstackstatus- NOTICE: Updated: Gerrit will be offline until 20:45 for scheduled maintenance (running longer than anticipated): http://lists.openstack.org/pipermail/openstack-dev/2017-January/109910.html | 20:35 | |
| *** ChanServ changes topic to "Updated: Gerrit will be offline until 20:45 for scheduled maintenance (running longer than anticipated): http://lists.openstack.org/pipermail/openstack-dev/2017-January/109910.html" | 20:35 | |
| zaitcev | clayg: nobody seems to care. I asked on #go-nuts and they suggest to follow the contribution procedure. File the item, outline the need, propose the fix... | 20:35 |
| zaitcev | clayg: I'm just letting you know in case you fancy Hummingbird work... I know you had your hands full with all the hashes updates and so on. | 20:36 |
| clayg | zaitcev: so I *do* still fancy some hummingbird(ish) work - have you seen monty's oaktree? https://github.com/openstack/oaktree | 20:37 |
| clayg | the gRPC stuff is *pretty* nice - protocol buffers are designed to make api's between client/servers written in different languages a breeze | 20:37 |
| clayg | I've had some less than great expereiences using them for data transfer before - but there's some stuff you can do to make serialization of datablobs less bad | 20:38 |
| zaitcev | Hmm, so this is some kind of a better Aeolus, it appears | 20:39 |
| clayg | that plus the builtin support for http2 multiplexing and the framing/termination/resume/footer stuff make it a really intersting protocol to me | 20:39 |
| zaitcev | Although, no... Aeolus is across AWS/Azure/OpenStack and this is only OpenStack. | 20:39 |
| clayg | oh oh oh - yeah the oaktree thing - I just pointed it out as an example of gRPC that already exists in OpenStack | 20:40 |
| zaitcev | Plus gRPC as you mention | 20:40 |
| clayg | what I want to do with gRPC has nothing to do with monty's goals of oaktree/porcilain/shade/etc | 20:41 |
| *** openstackgerrit has joined #openstack-swift | 20:43 | |
| openstackgerrit | Tim Burke proposed openstack/swift: Treat invalid limit parameters as errors https://review.openstack.org/299225 | 20:43 |
| clayg | zaitcev: SO if at somepoint you get annoyed enough with lp bug #1496636 you may want to consider gRPC - there's some non-zero chance something like it will be part of the golang object-replication-server anyway - and python parts of swift the like reconstructor that need to do pyeclib stuff but still want to talk SSYNCv2 could very easily be doing gRPC protocol stuff anyway | 20:44 |
| openstack | Launchpad bug 1496636 in OpenStack Object Storage (swift) "EC: Chunked transfer/commit protocol is *not* HTTP" [Undecided,New] https://launchpad.net/bugs/1496636 | 20:44 |
| *** chlong has quit IRC | 20:44 | |
| clayg | but you know... I might have stronger thoughts about it in ATL - you going to the PTG? | 20:45 |
| zaitcev | yes | 20:45 |
| *** JimCheung has joined #openstack-swift | 20:46 | |
| clayg | w0000! | 20:46 |
| clayg | god, i re-read that bug about how jank our ec put protocol is and I sort of forgot :\ | 20:46 |
| clayg | I figure there is zero chance you can write this protcol to anything other server in the world except eventlet.wsgi :\ | 20:47 |
| clayg | i man you could *try* to port the bug - but *hopefully* someone would stop you ;) | 20:48 |
| zaitcev | I'm still reading https://bugs.launchpad.net/swift/+bug/1496636, sorry. | 20:49 |
| openstack | Launchpad bug 1496636 in OpenStack Object Storage (swift) "EC: Chunked transfer/commit protocol is *not* HTTP" [Undecided,New] | 20:49 |
| clayg | oh it's a h00t! | 20:49 |
| zaitcev | yeah, bfitz isn't going to like us | 20:50 |
| *** JimCheung has quit IRC | 20:51 | |
| *** d0ugal has quit IRC | 20:52 | |
| *** ChanServ changes topic to "Let's talk, we're nice. | Ideas: https://wiki.openstack.org/wiki/Swift/ideas | Logs: http://eavesdrop.openstack.org/irclogs/%23openstack-swift/ | Meetings: https://wiki.openstack.org/wiki/Meetings/Swift | Priority Reviews: https://wiki.openstack.org/wiki/Swift/PriorityReviews" | 20:54 | |
| *** d0ugal has joined #openstack-swift | 20:54 | |
| *** d0ugal has quit IRC | 20:54 | |
| *** d0ugal has joined #openstack-swift | 20:54 | |
| *** manous has joined #openstack-swift | 20:56 | |
| *** sams-gleb has joined #openstack-swift | 20:57 | |
| *** chlong has joined #openstack-swift | 20:58 | |
| *** david-lyle has quit IRC | 21:01 | |
| *** david-lyle has joined #openstack-swift | 21:02 | |
| *** sams-gleb has quit IRC | 21:02 | |
| *** d0ugal has quit IRC | 21:03 | |
| *** chlong has quit IRC | 21:04 | |
| notmyname | pdardeau: mmotiani: I'm sorry for your loss http://tpr.org/post/san-antonio-pushes-pause-google-fiber-deployment | 21:05 |
| si1ver | timburke: ok I'll have to look it over again. Was this in the openstack/swift3 repo or the fujita one? (may have spelled that wrong) I was looking in the openstack one. | 21:05 |
| mattoliverau | Morning | 21:05 |
| si1ver | I am very close to having this working but I'm having another weird issue. If I specify the reseller_prefix=KEY line in [filter:keystoneauth], I get different results in test and stage. In stage with the line present it fails with the debug error "tenant mismatch: AUTH_c3ca2b0eecc94c2bb2d2528b5d0af2df != c3ca2b0eecc94c2bb2d2528b5d0af2df", but it works fine without the line. In test WITHOUT the line preset I get a failure "tenant | 21:09 |
| patchbot | Error: No closing quotation | 21:09 |
| si1ver | There's no other mention of AUTH or KEY in either swift proxy config file. Is there somewhere else I can check for some kind of default value? | 21:10 |
| clayg | yay I think gerrit is back-ish? | 21:10 |
| mattoliverau | si1ver have you got the reseller prefix being added to the end point in Keystone? And is KEY_ included in your storage URL your using when talking to swift? | 21:13 |
| mattoliverau | If not then it's equiv of having and empty reseller set | 21:13 |
| si1ver | the two service catalogs looks the same except for the hostname. The two sets of openstack env vars look the same except for the hostname and the passwords. | 21:14 |
| si1ver | the two keystone service catalogs were setup using the same mysql and openstack commands. | 21:14 |
| si1ver | The staging cluster was deployed as liberty and upgraded to mitaka before installing keystone. the test one is deployed fresh with mitaka each run. | 21:15 |
| *** chlong has joined #openstack-swift | 21:18 | |
| *** JimCheung has joined #openstack-swift | 21:18 | |
| *** openstackgerrit has quit IRC | 21:18 | |
| zaitcev | clayg: Do you remember by any chance why the commit phase is not a separate POST? Because of unacceptable overhead, right? | 21:18 |
| *** silor has joined #openstack-swift | 21:20 | |
| clayg | zaitcev: there is *no* good reason it shouldn't be a pipelined http request - that *exactly* what it should have been IMHO | 21:21 |
| *** JimCheung has quit IRC | 21:22 | |
| *** ukaynar has joined #openstack-swift | 21:22 | |
| *** ukaynar has quit IRC | 21:22 | |
| clayg | but we need some way for the second POST to mean "mae this fragment at this timestamp durable" and not like "the most recent fragment" or w/e | 21:23 |
| clayg | I think with the current implementation we do the whole flush/fsync/mkdir/rename dance so the object *is* in the hashdir - not in /tmp | 21:24 |
| clayg | I don't think the proxy is really setup to make pipelined reqeusts tho - you try to spike on it - I think you could even make a pipelined POST work for footers and drop all the mime stuff if you're really smart/careful | 21:24 |
| clayg | you might run it by torgomatic | 21:25 |
| zaitcev | so, a new EC put in order to avoid justifying myself in front of Go core | 21:26 |
| zaitcev | hmm | 21:26 |
| zaitcev | and resolve 1496636 | 21:26 |
| *** silor1 has joined #openstack-swift | 21:30 | |
| *** silor has quit IRC | 21:31 | |
| *** silor1 is now known as silor | 21:31 | |
| timburke | si1ver: the tests generally look something like https://github.com/openstack/swift3/blob/master/swift3/test/functional/test_object.py#L778-L816 -- you'll notice, though, that there are pieces that boto doesn't handle properly :-/ not sure what client/lib you're using | 21:33 |
| timburke | on the reseller_prefix issue, i think you might need to specify it in the s3token config as well? i forget exactly; might need to look at it again | 21:34 |
| *** Jeffrey4l has quit IRC | 21:34 | |
| si1ver | well I guess my puzzle is how the default could be different between the two environments. | 21:35 |
| *** manous has quit IRC | 21:35 | |
| si1ver | So I need a way to see that | 21:35 |
| si1ver | looks like swift tenants and openstack tenants are not the same thing, so it's difficult to google for. | 21:35 |
| pdardeau | notmyname: ya, we just chalk it up to that cosmic injustice thing ;) | 21:35 |
| *** JimCheung has joined #openstack-swift | 21:36 | |
| notmyname | si1ver: not sure I follow that. from what I've seen, the openstack tenant id is used to construct the account in swift (and that's what goes into the service catalog). what two things are you seeing that are different? | 21:40 |
| si1ver | I only see a difference in the swift proxy logs for failed requests. They fail as KEY_ not matching on one, and AUTH_ not matching on the other. Specifying the reseller prefix makes one work and makes the other fail. | 21:42 |
| si1ver | So I am assuming one must have a different default than the other but I don't know how to see that. | 21:42 |
| notmyname | oh, wait. I'm jsut catching up on your topic. you're running two keystones at once? | 21:42 |
| si1ver | yeah separate clusters. | 21:43 |
| si1ver | One is a chef cookbook test environment, the other is bare metal. | 21:43 |
| notmyname | oh, ok. not in the same cluster | 21:43 |
| notmyname | definitely specify the reseller prefix. better to be explicit than implicit. then you've got one of your clusters working and can look in to what the other issue is | 21:44 |
| si1ver | right. The same config file works on one and fails in the other. | 21:44 |
| *** openstackgerrit has joined #openstack-swift | 21:45 | |
| openstackgerrit | Samuel Merritt proposed openstack/swift: Fix download resumption for new SLOs. https://review.openstack.org/419664 | 21:45 |
| mattoliverau | si1ver: yeah so looks like the s3token also has a reseller_prefix as well you need to set | 21:45 |
| si1ver | reseller prefix in the cookbook test env must be in the keystoneauth section or keystone fails to work. It doesn't matter if it is there or not for the bare metal cluster, keystone works either way, but with it present AWS4 auth fails with the same message that keystone fails with on the cookbook test. | 21:46 |
| si1ver | I had to whiteboard all this out this morning to get it straight... | 21:46 |
| si1ver | specifying a reseller prefix in the swift3 section didn't seem to do anything. I'll try again. | 21:46 |
| *** sheel has quit IRC | 21:47 | |
| mattoliverau | in the s3token section, and it defaults to AUTH_ so that might be where the AUTH_ is coming from | 21:47 |
| mattoliverau | si1ver: https://github.com/openstack/swift3/blob/master/etc/proxy-server.conf-sample#L147 | 21:48 |
| mattoliverau | si1ver: it goes in the s3token section not the swift3 | 21:49 |
| si1ver | roger | 21:49 |
| si1ver | Should keystoneauth and s3token both have the reseller prefix? It is not working on the bare metal nodes. | 21:51 |
| si1ver | Either or both cause aws4 to fail on the bare metal cluster. | 21:53 |
| mattoliverau | si1ver: You need a reseller prefix for keystone, to identify it as the auth, the s3token one is so it can translate the s3api into the valid/required authenicatable storage url for swift (or in this case ketstone), so yeah you need both.. that's my limited understanding of swift3 anyway. | 21:55 |
| si1ver | With it in either section, I get this debug error in the proxy log: tenant mismatch: KEYc3ca2b0eecc94c2bb2d2528b5d0af2df != c3ca2b0eecc94c2bb2d2528b5d0af2df | 21:55 |
| mattoliverau | si1ver: you probably need it as KEY_ | 21:55 |
| mattoliverau | not KEY | 21:55 |
| mattoliverau | in s3token | 21:56 |
| si1ver | tenant mismatch: KEY_c3ca2b0eecc94c2bb2d2528b5d0af2df != c3ca2b0eecc94c2bb2d2528b5d0af2df | 21:56 |
| si1ver | heeey now we are getting somewhere. I added it to the keystoneauth section again but WITH the _, and it's working on the metal. | 21:58 |
| si1ver | both present but without the _ was failing. | 21:58 |
| si1ver | I may have spent a week hunting a single char error... ha | 21:59 |
| mattoliverau | si1ver: swift allows you to have more then one auth middleware, the reseller prefix tells swift which one to use. You left it out from the keystone so it was expecting things without the KEY_ | 21:59 |
| mattoliverau | now you've told swift keystone auth with the auth for everyone (account starting in KEY_) | 21:59 |
| mattoliverau | *is the auth | 22:00 |
| si1ver | Yeah but it was infering the KEY on one and infering an AUTH on the other, which was my confusion. | 22:00 |
| mattoliverau | si1ver: yeah, sorry didn't realise swift3 defaulted to AUTH_ | 22:00 |
| mattoliverau | so we all have learnt something :) | 22:00 |
| si1ver | beerdebt++ | 22:00 |
| si1ver | it works in test kitchen as well. Thank you very much. The different behavior is not explained, but eliminating it is good enough. | 22:02 |
| clayg | kota_: I really *want* to start reviewing global ec support - but for some reason instead I'm going to do another revision of patch 374419 :\ | 22:03 |
| patchbot | https://review.openstack.org/#/c/374419/ - swift - Move documented reclaim_age option to correct loca... | 22:03 |
| clayg | hopefully it won't take long! | 22:03 |
| mattoliverau | ^ famous last words. | 22:04 |
| *** d0ugal has joined #openstack-swift | 22:07 | |
| *** darrenc is now known as darrenc_afk | 22:09 | |
| *** gabor_antal has quit IRC | 22:11 | |
| *** gabor_antal_ has joined #openstack-swift | 22:11 | |
| clayg | onovy: did you get ahold of Pavel - I hate to bother him - but if he's not going to be working for awhile (good for him!) we should try to make progress without him | 22:16 |
| clayg | if he's available I really don't want to do something without his valuable input! | 22:16 |
| *** silor has quit IRC | 22:19 | |
| *** gabor_antal_ has quit IRC | 22:23 | |
| *** gabor_antal has joined #openstack-swift | 22:23 | |
| *** chlong has quit IRC | 22:29 | |
| *** sams-gleb has joined #openstack-swift | 22:30 | |
| *** ukaynar has joined #openstack-swift | 22:31 | |
| *** chlong has joined #openstack-swift | 22:37 | |
| *** darrenc_afk is now known as darrenc | 22:41 | |
| *** sams-gleb has quit IRC | 22:49 | |
| *** ukaynar has quit IRC | 22:55 | |
| openstackgerrit | Samuel Merritt proposed openstack/swift: Fix download resumption for new SLOs. https://review.openstack.org/419664 | 23:17 |
| *** kei_yama has joined #openstack-swift | 23:23 | |
| *** tqtran has quit IRC | 23:29 | |
| *** vint_bra has quit IRC | 23:31 | |
| *** chsc has quit IRC | 23:36 | |
| *** Jeffrey4l has joined #openstack-swift | 23:37 | |
| *** klamath has quit IRC | 23:46 | |
| *** sams-gleb has joined #openstack-swift | 23:50 | |
| *** sams-gleb has quit IRC | 23:54 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!