| openstackgerrit | A change was merged to openstack/swift: Handle COPY verb in container quota middleware https://review.openstack.org/61181 | 00:10 |
|---|---|---|
| *** zigo has quit IRC | 00:25 | |
| *** zigo has joined #openstack-swift | 00:26 | |
| *** matsuhashi has joined #openstack-swift | 00:28 | |
| *** joeljwright has joined #openstack-swift | 00:28 | |
| *** joeljwright has quit IRC | 00:32 | |
| *** zigo has quit IRC | 00:42 | |
| bsdkurt | in a three copy cluster, for object GET, how does the proxy server decide which copy to retrieve? | 00:49 |
| tburnes | For a production cluster, is there any documentation or guidance from the community on best practice for security of information? | 00:49 |
| *** MirandaZhang has joined #openstack-swift | 00:55 | |
| *** zigo has joined #openstack-swift | 00:59 | |
| *** zigo has quit IRC | 01:08 | |
| *** zigo has joined #openstack-swift | 01:13 | |
| *** shri has quit IRC | 01:14 | |
| MirandaZhang | could anyone help me with the swift client problem https://ask.openstack.org/en/question/12155/swiftclient-202-importerror/ | 01:14 |
| *** zigo has quit IRC | 01:20 | |
| *** zigo has joined #openstack-swift | 01:22 | |
| notmyname | mordred: I haven't been online today. I just sat down briefly (ie can't stay now) and saw you mentioned some "known problem with swift and grenade". something I need to take a look at later? | 01:25 |
| *** zigo has quit IRC | 01:27 | |
| mordred | notmyname: I think mtreinish has it | 01:27 |
| notmyname | mordred: k, thanks. I'll be able to check in later if anything's needed | 01:27 |
| mordred | notmyname: swiftclient release removed a call that stable/grizzly swift tests were using, aiui | 01:27 |
| notmyname | hmmm | 01:27 |
| mordred | notmyname: which means that grenade had an issue | 01:27 |
| * mordred waves hands in air | 01:28 | |
| notmyname | well that's what you get when you play with grenades! | 01:28 |
| *** zigo has joined #openstack-swift | 01:28 | |
| notmyname | ok, I'm out again. I'll check in this evening later | 01:28 |
| *** joeljwright has joined #openstack-swift | 01:29 | |
| *** nosnos has joined #openstack-swift | 01:33 | |
| *** joeljwright has quit IRC | 01:34 | |
| openstackgerrit | A change was merged to openstack/python-swiftclient: Add option to skip downloading/uploading identical files https://review.openstack.org/65470 | 01:45 |
| *** Diddi has quit IRC | 01:50 | |
| *** zigo has quit IRC | 01:51 | |
| *** Diddi has joined #openstack-swift | 01:52 | |
| *** zigo has joined #openstack-swift | 01:53 | |
| *** zigo has quit IRC | 02:04 | |
| *** Diddi has quit IRC | 02:05 | |
| *** zigo has joined #openstack-swift | 02:06 | |
| *** Diddi has joined #openstack-swift | 02:07 | |
| *** joeljwright has joined #openstack-swift | 02:31 | |
| *** zigo has quit IRC | 02:35 | |
| *** joeljwright has quit IRC | 02:35 | |
| *** zigo has joined #openstack-swift | 02:48 | |
| *** zackf has quit IRC | 03:06 | |
| *** ppai has joined #openstack-swift | 03:18 | |
| *** matsuhashi has quit IRC | 03:23 | |
| notmyname | hola | 03:26 |
| *** joeljwright has joined #openstack-swift | 03:32 | |
| *** joeljwright has quit IRC | 03:39 | |
| openstackgerrit | Thomas Goirand proposed a change to openstack/python-swiftclient: swift.1 manpage fix for groff warnings https://review.openstack.org/74249 | 03:49 |
| luisbg | notmyname, hola | 04:01 |
| notmyname | luisbg: good evening | 04:01 |
| luisbg | no en español? | 04:02 |
| notmyname | heh. my son is in spanish immersion in kindergarten. but I know basically no spanish (other than the little you pick up growing up in Texas) | 04:03 |
| luisbg | so your kid can use it as a secret language between his friends | 04:05 |
| notmyname | ya, no kidding. he and my wife already do :-) | 04:07 |
| luisbg | notmyname, muy bien por ellos! :) | 04:15 |
| openstackgerrit | John Dickinson proposed a change to openstack/swift: Make PBR based setup completely optional https://review.openstack.org/73738 | 04:17 |
| luisbg | notmyname, copyright said Hewlett Packard? | 04:18 |
| notmyname | the old version of that file what contributed by HP | 04:19 |
| notmyname | err..current | 04:19 |
| luisbg | soon to be old version | 04:21 |
| luisbg | notmyname, I feel a bit like a fool tempted to +1 your review :P | 04:22 |
| notmyname | luisbg: well, it's actually a very contentious patch. | 04:22 |
| *** pberis has quit IRC | 04:24 | |
| luisbg | notmyname, ahhhh I see (about copyright dates) | 04:28 |
| notmyname | ya | 04:28 |
| luisbg | thanks for clarifying | 04:30 |
| *** Dharmit has joined #openstack-swift | 04:34 | |
| *** joeljwright has joined #openstack-swift | 04:35 | |
| *** joeljwright has quit IRC | 04:40 | |
| luisbg | notmyname, I am having some tests pass on my machine using tox, but not pass on Jenkins, is there a way to avoid spamming the review with changes/rechecks trying to figure out why? | 04:42 |
| notmyname | luisbg: depends on what's failing in jenkins | 04:42 |
| notmyname | is it swift jobs that are failing? | 04:42 |
| *** matsuhashi has joined #openstack-swift | 04:43 | |
| luisbg | notmyname, no | 04:44 |
| luisbg | this is a patch for a small bug in oslo.config | 04:44 |
| notmyname | ah | 04:44 |
| *** Dharmit has quit IRC | 04:44 | |
| notmyname | luisbg: so in that case, you're seeing all the tests pass locally in devstack? | 04:45 |
| * notmyname assumes that oslo is using devstack as a dev environment | 04:45 | |
| luisbg | notmyname, well, machine where I have devstack, running tox in the oslo.config git clone | 04:45 |
| notmyname | you have now reached the end of my knowledge about testing patches for oslo | 04:46 |
| luisbg | notmyname, hahahahahaa | 04:46 |
| luisbg | sorry about that | 04:46 |
| luisbg | should I just mention this in the gerrit review and see if somebody who knows oslo.config suggests something? | 04:46 |
| notmyname | ya. or find someone on IRC | 04:48 |
| *** matsuhashi has quit IRC | 05:09 | |
| *** matsuhashi has joined #openstack-swift | 05:09 | |
| *** matsuhashi has quit IRC | 05:10 | |
| *** matsuhashi has joined #openstack-swift | 05:10 | |
| openstackgerrit | John Dickinson proposed a change to openstack/swift: Make PBR based setup completely optional https://review.openstack.org/73738 | 05:13 |
| notmyname | wow pep8 thanks! | 05:13 |
| * notmyname only likes pep8 when it doesn't catch issues in his own code | 05:13 | |
| *** chandan_kumar has joined #openstack-swift | 05:20 | |
| *** gvernik_ has joined #openstack-swift | 05:26 | |
| *** nosnos has quit IRC | 05:33 | |
| *** nosnos has joined #openstack-swift | 05:34 | |
| *** joeljwright has joined #openstack-swift | 05:36 | |
| *** matsuhashi has quit IRC | 05:36 | |
| *** matsuhashi has joined #openstack-swift | 05:37 | |
| *** matsuhas_ has joined #openstack-swift | 05:38 | |
| *** matsuhashi has quit IRC | 05:39 | |
| *** joeljwright has quit IRC | 05:40 | |
| *** rturk-away has quit IRC | 05:43 | |
| *** rturk-away has joined #openstack-swift | 05:48 | |
| *** nshaikh has joined #openstack-swift | 05:57 | |
| *** rturk-away has quit IRC | 05:59 | |
| *** rturk-away has joined #openstack-swift | 06:02 | |
| *** bsdkurt has quit IRC | 06:07 | |
| *** nosnos_ has joined #openstack-swift | 06:10 | |
| *** nosnos has quit IRC | 06:10 | |
| openstackgerrit | A change was merged to openstack/python-swiftclient: Only run flake8 on swiftclient code https://review.openstack.org/73507 | 06:22 |
| openstackgerrit | A change was merged to openstack/python-swiftclient: Add missing backslash. https://review.openstack.org/71224 | 06:25 |
| openstackgerrit | A change was merged to openstack/swift: Added docs about the swift_source log field https://review.openstack.org/71163 | 06:29 |
| *** joeljwright has joined #openstack-swift | 06:37 | |
| *** joeljwright has quit IRC | 06:42 | |
| *** zaitcev has quit IRC | 06:46 | |
| *** saju_m has joined #openstack-swift | 06:48 | |
| *** ppai has quit IRC | 06:56 | |
| *** Dharmit has joined #openstack-swift | 06:59 | |
| *** gvernik_ has quit IRC | 06:59 | |
| *** foexle has joined #openstack-swift | 07:04 | |
| *** ppai has joined #openstack-swift | 07:13 | |
| *** Dharmit has quit IRC | 07:14 | |
| *** nosnos_ has quit IRC | 07:19 | |
| *** nosnos has joined #openstack-swift | 07:19 | |
| *** bsdkurt has joined #openstack-swift | 07:35 | |
| *** joeljwright has joined #openstack-swift | 07:38 | |
| *** joeljwright has quit IRC | 07:42 | |
| *** mlipchuk has joined #openstack-swift | 07:47 | |
| *** kris_h has quit IRC | 07:53 | |
| *** MirandaZhang has left #openstack-swift | 08:06 | |
| *** joeljwright has joined #openstack-swift | 08:10 | |
| *** Dharmit has joined #openstack-swift | 08:33 | |
| *** mmcardle has joined #openstack-swift | 08:41 | |
| *** nacim has joined #openstack-swift | 08:47 | |
| *** nshaikh has left #openstack-swift | 09:00 | |
| *** chandankumar_ has joined #openstack-swift | 09:02 | |
| *** chandan_kumar has quit IRC | 09:03 | |
| *** mkollaro has joined #openstack-swift | 09:13 | |
| *** fbo_away is now known as fbo | 09:23 | |
| *** saju_m has quit IRC | 09:37 | |
| *** saju_m has joined #openstack-swift | 09:37 | |
| *** nosnos has quit IRC | 09:41 | |
| *** nosnos_ has joined #openstack-swift | 09:41 | |
| *** saju_m has quit IRC | 09:46 | |
| *** kris_h has joined #openstack-swift | 09:48 | |
| *** saju_m has joined #openstack-swift | 10:05 | |
| *** kris_h has quit IRC | 10:27 | |
| *** chandan_kumar has joined #openstack-swift | 10:30 | |
| *** chandankumar_ has quit IRC | 10:34 | |
| openstackgerrit | A change was merged to openstack/swift: Allow log disable in account/container server https://review.openstack.org/74128 | 11:02 |
| openstackgerrit | Tristan Cacqueray proposed a change to openstack/python-swiftclient: Add requests related unit-tests https://review.openstack.org/74328 | 11:02 |
| *** nosnos_ has quit IRC | 11:43 | |
| *** nosnos has joined #openstack-swift | 11:44 | |
| openstackgerrit | Donagh McCabe proposed a change to openstack/swift: Improve StaticWeb 404 on web-listings/index https://review.openstack.org/74336 | 11:46 |
| *** nosnos_ has joined #openstack-swift | 12:43 | |
| *** nosnos has quit IRC | 12:43 | |
| *** bingbu has joined #openstack-swift | 13:01 | |
| *** matsuhas_ has quit IRC | 13:04 | |
| *** mlipchuk has quit IRC | 13:04 | |
| *** mkollaro has quit IRC | 13:06 | |
| *** matsuhashi has joined #openstack-swift | 13:06 | |
| *** mkollaro has joined #openstack-swift | 13:06 | |
| *** ppai has quit IRC | 13:14 | |
| *** mlipchuk has joined #openstack-swift | 13:23 | |
| *** rando has joined #openstack-swift | 13:25 | |
| *** mdonohoe has quit IRC | 13:26 | |
| *** mmcardle has quit IRC | 13:37 | |
| *** bingbu has quit IRC | 13:44 | |
| *** saju_m has quit IRC | 13:50 | |
| *** mmcardle has joined #openstack-swift | 13:53 | |
| *** mkollaro has quit IRC | 13:56 | |
| *** mkollaro1 has joined #openstack-swift | 13:56 | |
| *** Trixboxer has joined #openstack-swift | 13:57 | |
| *** pconstantine has quit IRC | 14:05 | |
| *** pconstantine has joined #openstack-swift | 14:07 | |
| *** tdasilva has joined #openstack-swift | 14:07 | |
| *** tongli has joined #openstack-swift | 14:08 | |
| *** mlipchuk has quit IRC | 14:08 | |
| *** zul has quit IRC | 14:09 | |
| *** zul has joined #openstack-swift | 14:11 | |
| *** mlipchuk has joined #openstack-swift | 14:14 | |
| *** kris_h has joined #openstack-swift | 14:16 | |
| *** pberis has joined #openstack-swift | 14:44 | |
| *** foexle has quit IRC | 14:46 | |
| *** jergerber has joined #openstack-swift | 14:52 | |
| *** nosnos_ has quit IRC | 14:57 | |
| *** dmsimard has joined #openstack-swift | 14:57 | |
| *** mkollaro has joined #openstack-swift | 15:02 | |
| *** mkollaro1 has quit IRC | 15:02 | |
| *** donagh has joined #openstack-swift | 15:04 | |
| mtreinish | notmyname: so the issue wasn't actually swift but the devstack exercises which use swiftclient, it's a python install/requirements issue because grizzly has a version cap < 2 but we install from master during grenade | 15:15 |
| notmyname | mtreinish: ah. gotcha. so it was the major version bump then? | 15:16 |
| mtreinish | notmyname: yeah | 15:16 |
| notmyname | mtreinish: anything I should have done differently in the 2.X release? | 15:16 |
| mtreinish | there is a swift issue too but only if you guys ever want to backport something to the stable/grizzly branch (one of the functional tests that uses swiftclient) | 15:16 |
| notmyname | mtreinish: ok. can you give me a link? | 15:17 |
| mtreinish | notmyname: no, I think it's more the state of global reqs in grizzly, I pushed an uncap but it didn't fix it | 15:17 |
| mtreinish | notmyname: sure let me dig it up one sec | 15:17 |
| *** zackf has joined #openstack-swift | 15:17 | |
| mtreinish | notmyname: https://review.openstack.org/#/c/74150/ | 15:18 |
| mtreinish | the requests move dropped the HTTPException class from swiftclient | 15:19 |
| notmyname | right. got it. so aside from a porr commit message, do you know the reason it was abandoned? | 15:20 |
| mtreinish | well I think he jumped the gun because the cells job still failed which is the install issue | 15:20 |
| mtreinish | which I still have to figure out how to unwedge | 15:21 |
| *** mkollaro1 has joined #openstack-swift | 15:21 | |
| *** mkollaro has quit IRC | 15:21 | |
| notmyname | mtreinish: seems like it would be reasonable to keep grizzly (and other stable releases for that matter) at <2. | 15:23 |
| mtreinish | notmyname: the issue with that is we run all the tests with master branches of the clients | 15:23 |
| mtreinish | so when you push a major release it will cause issues | 15:24 |
| notmyname | perhaps that's not a good idea with historic stable releases | 15:24 |
| *** chandan_kumar has quit IRC | 15:43 | |
| *** byeager has joined #openstack-swift | 15:45 | |
| openstackgerrit | Constantine Peresypkin proposed a change to openstack/swift: account to account copy implementation https://review.openstack.org/72157 | 15:48 |
| mjseger | notmyname: I've been doing some testing over the weekend and today with swiftclient V2 and am finding what appears to be a significant drop in performance and am wondering if you or others have seen that as well | 15:51 |
| tristanC | mjseger: was it 2.0.2 ? | 15:55 |
| openstackgerrit | Alistair Coles proposed a change to openstack/swift: Fix invalid account acl generating 500 response. https://review.openstack.org/74417 | 15:56 |
| creiht | mjseger: heh... I was actually planning on doing some testing with that | 15:56 |
| creiht | but haven't done that yet | 15:56 |
| mjseger | it was the one announced on the openstack list. hang on a sec | 15:56 |
| tristanC | 2.0 and 2.0.1 have a memory hog bug fixed in 2.0.2 | 15:57 |
| mjseger | yes, 2.0.2 | 15:57 |
| mjseger | ssegerm@az1-nv-compute-0003:~$ swift --version swift 2.0.2 | 15:57 |
| mjseger | if anyone would like to try my getput tool, which makes it pretty easy to run test with, I'll be happy to send you a copy OR just use whatever mechanisms you normally use | 15:58 |
| mjseger | what I also found is it's not just limited to small objects and even large PUTs look like they suffer | 15:58 |
| *** Dharmit has quit IRC | 15:59 | |
| mjseger | I even tried timing 'swift upload' commands for large objects and found them to be slowed with 2.0.2 than 1.9 | 15:59 |
| acoles | otherjon: could you cast an eye over https://review.openstack.org/#/c/74417 . thanks | 16:00 |
| mjseger | one other thing I found is if I install 2.0.2 from the tarball and then try to downgrade to 1.9 I can't! it doesn't replace client.py with the older one. shouldn't it? | 16:00 |
| *** Dharmit has joined #openstack-swift | 16:03 | |
| tristanC | mjseger: I notice a 4.2% drop in performance between 2.0.2 and 1.9 (time swift upload) | 16:08 |
| mjseger | tristanC: i | 16:08 |
| mjseger | tristanC: I've actually found measuring single operations not to be that accurate. for my tests I ran 1k and 2k puts for 2 minutes and found almost a 2:1 difference between the IOPS for 1.9 and 2.0 | 16:10 |
| mjseger | have you done any testing like that? | 16:10 |
| tristanC | hum interesting, no I didn't perform such tests | 16:14 |
| *** Cotes has joined #openstack-swift | 16:14 | |
| tristanC | I'd like to give your "getput tool" a try | 16:14 |
| notmyname | mjseger: thanks for bringing it up | 16:15 |
| *** Eelis has left #openstack-swift | 16:16 | |
| mjseger | notmyname: is would be good to hear if others can confirm my findings. also, would you consider if a bug if you can't install 1.9 on top of 2.0.2? I would... | 16:17 |
| notmyname | mjseger: how are you trying to install it? | 16:18 |
| *** zackf has quit IRC | 16:18 | |
| *** zackf has joined #openstack-swift | 16:18 | |
| mjseger | I'm using "python setup.py install" | 16:18 |
| cschwede | mjseger: you might need a "python setup.py develop" as well | 16:20 |
| *** kris_h has quit IRC | 16:20 | |
| mjseger | hmm, so I do that before the install? | 16:20 |
| creiht | mjseger: well, you are just going to have headaches trying to install an older version over a newer version | 16:21 |
| creiht | you might have to go delete the older version first | 16:21 |
| creiht | well whatever version you are trying to replace | 16:21 |
| notmyname | looks like creiht and cschwede are up (thanks!). I'm biking to work now. be back online in ~30 min | 16:22 |
| mjseger | from a purely functional perspective I'd think users should be able to move back and forth between versions, especially if you're trying to benchmark things | 16:22 |
| creiht | hah | 16:22 |
| creiht | mjseger: yeah, unfortunately python doesn't make that very easy | 16:22 |
| cschwede | mjseger: not that easy | 16:22 |
| *** zackf has quit IRC | 16:22 | |
| mjseger | bummer... | 16:22 |
| cschwede | i sometimes use python -c "import swiftclient; print swiftclient" to get the location where a lib has been installed, and then i just delete that path (not the cleanest way i know, but works for me) | 16:24 |
| mjseger | so that means when install the new swiftcleint and see the performance drop by a factor of 2 I can't go back to the previous version? do I need to reimage the whole system then? | 16:24 |
| cschwede | mjseger: no, i don't think that this is required | 16:25 |
| mjseger | ahh, I was thinking of doing something like that but didn't want to make things worse. I'll give that a go | 16:25 |
| creiht | mjseger: well the easist way is probably to clean out any pip installed or packger installed swiftclients | 16:25 |
| mjseger | so that gets me back to my original question about performance and seeing the drop. I am testing on different physical boxes but have tried on enough machines that I'm feeling pretty confident about what I'm seeing | 16:25 |
| creiht | by doins something similar to what cschwede is saying | 16:25 |
| mjseger | I'd be more than happy to share any results | 16:26 |
| creiht | you can also print swiftclient.__file__ | 16:26 |
| creiht | once you have done that, you could check out the swiftclient versions in different repos | 16:26 |
| creiht | and then I would recommend doing python setup.py develop in whichever one you want to use | 16:26 |
| creiht | that *should* then overwrite the current version each time | 16:26 |
| creiht | I think | 16:27 |
| creiht | :) | 16:27 |
| creiht | mjseger: and yeah it would be interesting to see your results | 16:27 |
| cschwede | creiht: sounds like a good plan :) | 16:27 |
| creiht | I was going to run some swift-benchs on a couple of environs with both versions to see what that looks like | 16:27 |
| mjseger | are there any specific types of tests you'd like to see? anyhow hang on a sec and I'll rerun and post the results. should be pretty quick | 16:28 |
| creiht | mjseger: I think the best data is test runs that represent your use cases | 16:29 |
| mjseger | since I'm doing general purpose benchmarking I have have any specific use cases. | 16:30 |
| *** matsuhashi has quit IRC | 16:32 | |
| mjseger | creight: wow, I just ran get tests for the first time and those numbers are even worse! here's my data for 1k and 1M puts/gets: http://paste.openstack.org/show/66941/ | 16:32 |
| cschwede | how is your tool working? is it using client.put_object direct (ie from within a python tool)? | 16:32 |
| cschwede | mjseger: are you deleting all data from your containers before testing? | 16:33 |
| mjseger | good question, current I'm not but I can rerun the tests with unique container names ot make sure they're brand new. hang on | 16:33 |
| cschwede | mjseger: uhh! | 16:33 |
| cschwede | mjseger: uhh: i mean the results | 16:34 |
| cschwede | 130 vs 30? thats a 4:1 drop :( | 16:34 |
| cschwede | and looks like cpu usage is significantly higher | 16:34 |
| mjseger | yes, 4:1. I just reran with UTC appended to each container name and the overall perforamnce for both dropped by 2.0.2 is still pretty poor | 16:35 |
| tristanC | mjseger: can you share your getput tool ? | 16:35 |
| cschwede | mjseger: can you share that tool? | 16:36 |
| cschwede | := | 16:36 |
| cschwede | :) | 16:36 |
| tristanC | mjseger: also, it might be worth mentioning the python-requests version you are using | 16:36 |
| mjseger | re tool - absolutely. where shoudl I send it? | 16:36 |
| mjseger | here's the results with uynique container names: http://paste.openstack.org/show/66942/ | 16:36 |
| tristanC | mjseger: as you wish, github, paste.openstack.org, ... | 16:37 |
| cschwede | if it is only getput.py, than paste.openstack.org should be fine | 16:37 |
| mjseger | I'm using requests 2.2.1 I think since that's the one that disables nagel | 16:37 |
| *** IRTermite has left #openstack-swift | 16:49 | |
| *** byeager has quit IRC | 16:52 | |
| mjseger | cschwede: still trying to find the best way to share. how big a script can you sick into paste.com? I have a whole tarball with help, man pages, etc. OR I can paste in getput.py which is about 1700 lines long | 16:53 |
| *** kris_h has joined #openstack-swift | 16:55 | |
| cschwede | mjseger: i discussed the performance issue with tristanC and two questions popped up: 1. are you using ssl cert verification in your tests and 2. are you reusing the http connection? | 16:55 |
| creiht | mjseger: put it up on github? :) | 16:56 |
| mjseger | yesh, I really do need to get it there. unfortunately I have a concall in 5 mins but will try to set things up when I'm done | 16:57 |
| tristanC | mjseger: thank you! | 16:57 |
| cschwede | mjseger: great, that would be cool. I'm also leaving in a few minutes for dinner, but will be around later | 16:57 |
| mjseger | cschwede: re ssl and reusing connections, I'm doing what I think is vanilla swftclient call, creating a connection and then doing a bunch of puts, gets, etc. so i assume that means I'm resusing. as for ssl, I'm taking the defaults | 16:58 |
| mjseger | gotta run but like the terminator, I'll be back ;) | 16:58 |
| cschwede | mjseger: ok that means a ssl cert check which might slow things down. that behavior changed | 16:58 |
| * cschwede afk for dinner, will be online later | 17:00 | |
| *** jergerber has left #openstack-swift | 17:00 | |
| mjseger | cschwede: but I'd assume the cert check is only done once, so on a large object put wouldn't the time be negligible? | 17:00 |
| *** zackf has joined #openstack-swift | 17:02 | |
| *** tanee has joined #openstack-swift | 17:03 | |
| tristanC | maybe the requests implementation could use the requests.Session thing, might speed up things. I'll give it a try now | 17:03 |
| *** lpabon has joined #openstack-swift | 17:07 | |
| *** Midnightmyth has joined #openstack-swift | 17:08 | |
| *** krtaylor has quit IRC | 17:09 | |
| *** lpabon has quit IRC | 17:12 | |
| *** rando has quit IRC | 17:15 | |
| *** wer has joined #openstack-swift | 17:16 | |
| *** mlipchuk has quit IRC | 17:20 | |
| *** kris_h has quit IRC | 17:22 | |
| *** tanee is now known as tane-away | 17:32 | |
| *** krtaylor has joined #openstack-swift | 17:33 | |
| *** gyee has joined #openstack-swift | 17:45 | |
| *** Dharmit has quit IRC | 17:46 | |
| openstackgerrit | Constantine Peresypkin proposed a change to openstack/swift: account to account copy implementation https://review.openstack.org/72157 | 17:49 |
| *** nacim has quit IRC | 17:51 | |
| *** mvenesio has joined #openstack-swift | 17:54 | |
| mjseger | cschwede: I think I have something in github, though never having created a repository no promises it's completely correct ;) but here it is: https://github.com/markseger/getput (I think) | 17:57 |
| mjseger | cschwede: if you indeed can see it and want to try something real simple, just pull down getput.py and I can tell you ina few seconds what commands you need run | 17:58 |
| *** joeljwright has quit IRC | 18:06 | |
| mjseger | tristanC: notmyname: in case you didn't see my message to cschwede, you can pull a copy of getput from github now. I'd really like to get some independent confirmation of what I'm seeing | 18:06 |
| *** mkollaro1 has quit IRC | 18:09 | |
| openstackgerrit | Jon Snitow proposed a change to openstack/swift: Return a 400 when a non-dict JSON ACL is posted. https://review.openstack.org/74459 | 18:13 |
| openstackgerrit | Jon Snitow proposed a change to openstack/swift: Return a 400 when a non-dict JSON ACL is posted. https://review.openstack.org/74459 | 18:19 |
| gholt | torgomatic: dfg mentioned that the tempurl isn't a big deal (right now) because you can only make tempurls if you have account admin access anyway. | 18:19 |
| torgomatic | gholt: true, but I wonder if they work at all | 18:20 |
| torgomatic | if it's my account, my manifest, and my segments, I should be able to make a large-object tempurl | 18:20 |
| gholt | If we ever change that so non-account-admins can make tempurls, we'll have to fix that. Er, I mean, whoever does that work. :) | 18:20 |
| *** mkollaro has joined #openstack-swift | 18:21 | |
| * torgomatic ∉ "we" :) | 18:21 | |
| *** mkollaro has quit IRC | 18:21 | |
| *** joeljwright has joined #openstack-swift | 18:55 | |
| *** shri has joined #openstack-swift | 18:55 | |
| *** byeager has joined #openstack-swift | 18:57 | |
| mjseger | notmyname: just wondering if you made it to the office yet as I doubt it's snowing like mad there like it is here ;) | 19:03 |
| *** joeljwright has quit IRC | 19:05 | |
| *** shri has quit IRC | 19:06 | |
| *** shri1 has joined #openstack-swift | 19:07 | |
| *** byeager_ has joined #openstack-swift | 19:07 | |
| *** shri1 has quit IRC | 19:08 | |
| *** mmcardle has quit IRC | 19:09 | |
| *** byeager has quit IRC | 19:10 | |
| cschwede | mjseger: I'm back, thanks for putting getput to github, I'm now looking into it | 19:21 |
| mjseger | ok, great. and thanks. some real quick tips to get this going: | 19:21 |
| mjseger | I always build a file that exports my credentials to ST_variable | 19:22 |
| mjseger | once exported, a simple command like this will write 10 1k objects to a container named c and objects named o | 19:22 |
| mjseger | ./getput.py -cc -oo -n1 -s1k -tp | 19:22 |
| mjseger | you can vary things quite a lot and even run lots of parallel processes | 19:23 |
| mjseger | my favorite swift is -s1k,2k which first does 1k operations than that does them again with 2k objects | 19:23 |
| mjseger | just give a holler if it doesn't work straight away | 19:24 |
| cschwede | mjseger: works out of the box, thanks! | 19:25 |
| * cschwede now trying different swiftclient versions | 19:25 | |
| mjseger | cschwede: good to hear. lots more options when you're ready for them. such as spitting out individual transaction IDs so you can use it to dig through the logs if you need to | 19:26 |
| mjseger | so for the record, if you run with 1.9 you should see 2k objects anywhere from 2 to 4 times slows. if you run with 2.0.2 and request < 2.1.1, you should see similar ratios but slower. if you run with requests 2.1.1 the 1k and 2k numbers are the same since nagel disabled, but they're still slower than with 1.9 | 19:29 |
| mjseger | at least that's what I'm seeing | 19:29 |
| *** mmcardle has joined #openstack-swift | 19:37 | |
| *** fbo is now known as fbo_away | 19:37 | |
| *** mmcardle1 has joined #openstack-swift | 19:39 | |
| *** mmcardle has quit IRC | 19:41 | |
| cschwede | mjseger: what kind of test environment are you using? is that a production swift cluster or running in a local VM? | 19:42 |
| mjseger | actually this is in a testing environment and I'm running on bare metal but I'm hitting our production servers | 19:43 |
| mjseger | have you gotten any results worth sharing yet? | 19:44 |
| cschwede | mjseger: http://paste.openstack.org/show/66991/ -> 2.0.2 much better for PUT, 1.9.0 a little bit better for GET | 19:44 |
| mjseger | really!?! that is a surpise as I saw 2.0.2 slower for both small and large objects | 19:45 |
| cschwede | running in a local SAIO VM, cleared before both runs (ie no container/objects on the disks) | 19:45 |
| cschwede | repeated it three times, just to be sure | 19:45 |
| mjseger | I wonder if ssl compression at your place is enabled by default. try running again and add --nocomp because I think I head 2.0.2 disables compression by default and so if 1.9 doesn't that could explain some times | 19:47 |
| cschwede | mjseger: my local env is hitting the proxy server directly, ie no load balancer/ssl terminator in between and the proxy itself is running without SSL. Maybe that is the reason | 19:47 |
| cschwede | mjseger: there is no ssl at all for my local VM | 19:47 |
| mjseger | there are a lot of variables | 19:47 |
| mjseger | the other piece is the compression | 19:47 |
| mjseger | you're also using a LOT of cpu compared to me. | 19:48 |
| cschwede | mjseger: most likely because I only assigned 1 VCPU to the VM | 19:48 |
| cschwede | and everything is run in that VM | 19:48 |
| mjseger | the other thing that is interesting is your 1k and 2k PUT IOPS are about the same for both whereas mine if over 3 time faster with 1.9 | 19:48 |
| mjseger | as someone asked my earlier, or was it you?, whether or not I was starting each test cleanly, if you add --utc getput will append the current time to the name of the container assuring an empty/unique container | 19:50 |
| cschwede | mjseger: yepp, i was asking that | 19:52 |
| mjseger | ;) | 19:52 |
| mjseger | cschwede: well this is certainly unsettling if you're getting that much better performance than me. ;( | 19:55 |
| mjseger | I'd've expected things to be similar | 19:55 |
| cschwede | mjseger: yes, but i think we need to find the reason for these differences. I'm adding a pound SSL terminator now... | 19:55 |
| mjseger | yes, we do use poind | 19:56 |
| *** joeljwright has joined #openstack-swift | 20:02 | |
| cschwede | i needed to disable ssl cert verification (I'm only using a self-generated cert), but the results are nearly the same | 20:04 |
| cschwede | mjseger: maybe you could test also without cert verification? | 20:05 |
| *** gyee has quit IRC | 20:05 | |
| cschwede | mjseger: i changed line 541 and 550 in getput.py to "os_options=opts, ssl_compression=comp, insecure=True)" | 20:05 |
| mjseger | cschwede: how do I do that? | 20:05 |
| mjseger | ;) | 20:06 |
| mjseger | I think I can handle that | 20:06 |
| *** joeljwright has quit IRC | 20:06 | |
| mjseger | cschede: a little faster but still not as fast as 1.9. btw - did you really mean line 550? that's the connection for older versions of swiftclient that don't support connection.close() | 20:09 |
| mjseger | I added that code in both connection calls | 20:09 |
| mjseger | btw, sounds like I need a new switch ;) --insecure | 20:10 |
| *** joeljwright has joined #openstack-swift | 20:11 | |
| mjseger | cschwede: here's a new switch for you (I love switches), "--ldist 2" it will report a latency histogram to 2 decimal places showing how many options of different durations there were | 20:13 |
| mjseger | when I do 100 PUTs I see 11 in the 0.02 bucket, 71 in the 0.03 bucket, 14 in the 0.04 bucket and 4 in the 0.05 one | 20:14 |
| cschwede | i changed line 541 and 550 yes, because i was only looking where the connection is created ;-) | 20:14 |
| mjseger | since you're going faster than me I'd expect higher counts in lower numbered buckets | 20:14 |
| cschwede | lets see | 20:16 |
| *** joeljwright has quit IRC | 20:16 | |
| cschwede | mjseger: http://paste.openstack.org/show/67006/ | 20:23 |
| mjseger | cschwede: bingo - see how most of your operations are in the 0.01/0.02 seconds in the 2.0 case compare to 1.9? | 20:25 |
| *** mmcardle1 has quit IRC | 20:25 | |
| *** mmcardle has joined #openstack-swift | 20:27 | |
| cschwede | mjseger: yes, i see it, but 1.9.0 is still slower? put for 1.9.0 is in 0.04/0.05 range, and some get in 0.01 instead of 0.00 | 20:27 |
| mjseger | right, your puts are slower than mine and I don;t know why | 20:28 |
| mjseger | maybe I shoudl try this out on a VM tjat is closer to the production swift servers. hang on... | 20:29 |
| *** byeager_ has quit IRC | 20:30 | |
| mjseger | cschewe: might be making progress, both the 1.9 and 2.0 times are the same, wiht the 1k PUTs being 3 times slower. now I'll upgrade to request 2.2.1 and see if they equal out wiht nagel disabled | 20:43 |
| mjseger | cschede: hmm, that didn't help. don't you installed reuests via a command like: sudo pip install requests-2.2.1.tar.gz ? | 20:45 |
| mjseger | it says it successfully installed but I'm seeing 1K puts with iops of 20 and 2K puts with iops of 68! | 20:46 |
| mjseger | oh wait, I didn't install python-swiftclient! ;) | 20:46 |
| mjseger | it was still at v1.5, talk abotu ancient ;) | 20:47 |
| mjseger | damn, both 1k and 2k down to 12 IOPS with 2.0 | 20:48 |
| mjseger | something clearly wrong here | 20:48 |
| mjseger | cschwede: you're in europe, right? | 20:50 |
| cschwede | mjseger: yepp, in hamburg/germany | 20:51 |
| *** shri has joined #openstack-swift | 20:52 | |
| mjseger | I'm wondering if we can table this until the morning (though it's only 4PM here in new england) and wait until my irish colleagues are awake and perhaps they can join in too as they know more about the configuration details | 20:52 |
| mjseger | clearly this is something we want to get to the bottom of | 20:52 |
| cschwede | mjseger: sure, actually it is high time for me to close my laptop soon ;-) | 20:54 |
| mjseger | as an aside if you still want to play a little with getput just to see what it can do, if you include the switch --latexc number, it will report any individual operations that exceed that latency ( specified as a float) along with the transaction IDs and container/object names | 20:54 |
| mjseger | ok great, and thanks for your attention. we'll talk again tomorrow | 20:54 |
| cschwede | mjseger: looks like that tool is really sophisticated. I will play with it more tomorrow. thanks and see you tomorrow! | 20:55 |
| cschwede | mjseger: ... and btw: tomorrow at 7pm utc is the swift meeting, if you like to join :) | 20:56 |
| mjseger | actually as they say, "you ain't see nothing yet". what until you check out gpsuite! I've has as many as 2048 processes accessing swift in parallel! | 20:56 |
| *** byeager has joined #openstack-swift | 21:00 | |
| *** mvenesio has quit IRC | 21:06 | |
| *** gholt has quit IRC | 21:07 | |
| *** gholt has joined #openstack-swift | 21:08 | |
| *** ChanServ sets mode: +v gholt | 21:08 | |
| *** byeager has quit IRC | 21:08 | |
| *** joeljwright has joined #openstack-swift | 21:12 | |
| *** joeljwright has quit IRC | 21:17 | |
| *** gyee has joined #openstack-swift | 21:49 | |
| *** gyee has quit IRC | 21:51 | |
| *** gyee has joined #openstack-swift | 21:51 | |
| *** byeager has joined #openstack-swift | 21:53 | |
| *** shri has quit IRC | 21:53 | |
| *** mmcardle has quit IRC | 22:09 | |
| *** Trixboxer has quit IRC | 22:12 | |
| *** joeljwright has joined #openstack-swift | 22:13 | |
| *** joeljwright has quit IRC | 22:18 | |
| *** joeljwright has joined #openstack-swift | 22:31 | |
| openstackgerrit | paul luse proposed a change to openstack/swift: Add Storage Policy Support to Account HEAD https://review.openstack.org/73747 | 22:48 |
| creiht | so I have done some initial testing | 22:49 |
| *** shri has joined #openstack-swift | 22:49 | |
| creiht | and swiftclient 2.0 is about 5-10% slower than 1.9 for smaller objects | 22:49 |
| creiht | and that is all I have tested so far | 22:49 |
| creiht | oh and that's without ssl | 22:52 |
| creiht | but that also begs to question, at what point do we really consider too slow? | 22:52 |
| *** zaitcev has joined #openstack-swift | 22:55 | |
| *** ChanServ sets mode: +v zaitcev | 22:55 | |
| creiht | for larger files there is little to no difference between the two | 22:56 |
| portante | and do we consider python-swiftclient a tool that has a performance requirement? | 22:59 |
| notmyname | to me swiftclient has never been the thing to use for performance. not to say that it should be slow, but you'd probably go faster if you write your own wrapper | 23:01 |
| creiht | notmyname: well yeah, but if it were say %50 slower, then I might consider that a problem :) | 23:02 |
| notmyname | agreed :-) | 23:03 |
| notmyname | of course, a 5-10% drop in each release adds up ;-) | 23:03 |
| creiht | yes | 23:03 |
| *** rturk-aw` has joined #openstack-swift | 23:09 | |
| *** krtaylor has quit IRC | 23:16 | |
| *** bsdkurt has quit IRC | 23:16 | |
| *** rturk-away has quit IRC | 23:16 | |
| *** openstackgerrit has quit IRC | 23:16 | |
| *** portante has quit IRC | 23:16 | |
| *** Anju has quit IRC | 23:16 | |
| *** jokke_ has quit IRC | 23:16 | |
| *** dmsimard has quit IRC | 23:17 | |
| *** joeljwright has quit IRC | 23:18 | |
| *** krtaylor has joined #openstack-swift | 23:18 | |
| *** bsdkurt has joined #openstack-swift | 23:18 | |
| *** openstackgerrit has joined #openstack-swift | 23:18 | |
| *** Anju has joined #openstack-swift | 23:18 | |
| *** jokke_ has joined #openstack-swift | 23:18 | |
| *** portante has joined #openstack-swift | 23:18 | |
| *** ChanServ sets mode: +v portante | 23:18 | |
| *** Midnightmyth has quit IRC | 23:20 | |
| *** byeager has quit IRC | 23:20 | |
| *** shri1 has joined #openstack-swift | 23:22 | |
| *** shri has quit IRC | 23:25 | |
| mjseger | notmyname: sorry, I've been clearing my driveway of a lot of snow. from my perspective, I though if I wanted to talk to swift via python the recommended method is using swiftclient, and so that's why I used it with getput. are people backing away from that statement? | 23:27 |
| mjseger | as for the performance difference, I saw a much bigger drop than 10% for small object and also a drop that might've been around 10% for large ones. but since I'm the only one who saw that perhaps it's my configuration and not swiftclient | 23:28 |
| mjseger | the plan it to continue the discussion with cschwede in the morning and see if we can identify the problem | 23:29 |
| *** tongli has quit IRC | 23:33 | |
| *** mmcardle has joined #openstack-swift | 23:39 | |
| *** byeager has joined #openstack-swift | 23:41 | |
| *** byeager has quit IRC | 23:44 | |
| *** mmcardle has quit IRC | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!