| ho | mattoliverau: morning! | 00:02 |
|---|---|---|
| *** m_kazuhiro has joined #openstack-swift | 00:29 | |
| *** haomaiwang has quit IRC | 00:35 | |
| *** nakagawamsa has joined #openstack-swift | 00:48 | |
| *** kota_ has joined #openstack-swift | 00:52 | |
| *** ChanServ sets mode: +v kota_ | 00:52 | |
| kota_ | good morning | 00:53 |
| *** chlong has joined #openstack-swift | 00:55 | |
| mattoliverau | kota_: morning | 00:59 |
| kota_ | mattoliverau: morning :) | 00:59 |
| *** wbhuber_ has joined #openstack-swift | 01:20 | |
| ho | kota_: morning! | 01:21 |
| *** wbhuber has quit IRC | 01:22 | |
| *** marzif__ has joined #openstack-swift | 01:33 | |
| *** marzif_ has quit IRC | 01:37 | |
| *** haomaiwang has joined #openstack-swift | 01:40 | |
| *** aix_ has joined #openstack-swift | 01:41 | |
| *** aix has quit IRC | 01:42 | |
| *** chlong has quit IRC | 01:49 | |
| *** kota_ has quit IRC | 02:04 | |
| *** Shihta has joined #openstack-swift | 02:15 | |
| *** Shihta has left #openstack-swift | 02:15 | |
| *** kota_ has joined #openstack-swift | 02:24 | |
| *** ChanServ sets mode: +v kota_ | 02:24 | |
| openstackgerrit | paul luse proposed openstack/swift: Better error handling for EC PUT path when client goes away https://review.openstack.org/211338 | 02:42 |
| peluse | morning guys | 02:46 |
| peluse | kota_, mattoliverau if you have a chance during my evening here could you take a look at patch 211338 | 02:46 |
| patchbot | peluse: https://review.openstack.org/#/c/211338/ | 02:46 |
| kota_ | peluse: morning and thnaks for approving my patch. | 02:47 |
| peluse | it keeps a lot of ugly 503s out of the logs during perf runs (heavy loads) which causes our benchmark team a lot of heartburn :) | 02:47 |
| peluse | kota_, welcome! I've been waiting for that one, good stuff | 02:47 |
| mattoliverau | peluse: I'll try at take a look beofre I end for the day :) | 02:47 |
| peluse | thanks! its pretty small/simple | 02:47 |
| peluse | famous last words.... | 02:48 |
| kota_ | mattoliverau, peluse: me too. | 02:48 |
| peluse | thanks!! I'm outta here.... | 02:48 |
| mattoliverau | peluse: enjoy the rest of your weekend | 02:48 |
| *** jrichli has quit IRC | 02:57 | |
| *** haomaiwang has quit IRC | 03:03 | |
| *** haomaiw__ has joined #openstack-swift | 03:03 | |
| *** jroll has quit IRC | 03:26 | |
| *** jroll has joined #openstack-swift | 03:31 | |
| *** I has joined #openstack-swift | 03:35 | |
| *** I is now known as Guest25059 | 03:36 | |
| *** ppai has joined #openstack-swift | 03:36 | |
| *** links has joined #openstack-swift | 03:40 | |
| *** Guest25059 has quit IRC | 03:49 | |
| *** haigang has joined #openstack-swift | 03:49 | |
| openstackgerrit | John Dickinson proposed openstack/swift: Updated CHANGELOG, AUTHORS, and .mailmap for 2.4.0 release. https://review.openstack.org/218681 | 03:58 |
| notmyname | kota_: thanks. fixed ^^ | 03:58 |
| kota_ | notmyname: you're welocome^^ | 04:00 |
| *** hrou has joined #openstack-swift | 04:06 | |
| *** kei_yama has quit IRC | 04:07 | |
| *** kei_yama_ has joined #openstack-swift | 04:07 | |
| *** km_ has joined #openstack-swift | 04:09 | |
| *** km has quit IRC | 04:09 | |
| *** _hrou_ has joined #openstack-swift | 04:40 | |
| *** hrou has quit IRC | 04:41 | |
| *** _hrou_ has quit IRC | 04:47 | |
| *** SkyRocknRoll has joined #openstack-swift | 04:47 | |
| *** ig0r__ has joined #openstack-swift | 04:58 | |
| *** ig0r_ has quit IRC | 04:59 | |
| *** haigang has quit IRC | 05:23 | |
| *** dimasot has joined #openstack-swift | 05:25 | |
| *** proteusguy_ has quit IRC | 05:34 | |
| *** haigang has joined #openstack-swift | 05:47 | |
| *** proteusguy_ has joined #openstack-swift | 05:47 | |
| openstackgerrit | Matthew Oliver proposed openstack/swift-specs: Large containers (Sharding) - Pivot/Split Tree https://review.openstack.org/218738 | 06:12 |
| *** pushkarajthorat has joined #openstack-swift | 06:21 | |
| *** trifon has joined #openstack-swift | 06:23 | |
| *** haigang has quit IRC | 06:27 | |
| *** haigang has joined #openstack-swift | 06:28 | |
| *** haigang has quit IRC | 06:29 | |
| *** mahatic has joined #openstack-swift | 06:36 | |
| *** haigang has joined #openstack-swift | 06:36 | |
| *** SkyRocknRoll has quit IRC | 06:54 | |
| *** trifon has quit IRC | 07:03 | |
| *** pushkarajthorat has quit IRC | 07:04 | |
| *** kei_yama_ has quit IRC | 07:13 | |
| *** kei_yama has joined #openstack-swift | 07:13 | |
| *** rledisez has joined #openstack-swift | 07:14 | |
| *** pushkarajthorat has joined #openstack-swift | 07:14 | |
| *** SkyRocknRoll has joined #openstack-swift | 07:19 | |
| *** marcusvrn_ has quit IRC | 07:25 | |
| *** pushkarajthorat has quit IRC | 07:31 | |
| *** pushkarajthorat has joined #openstack-swift | 07:32 | |
| *** geaaru has joined #openstack-swift | 07:40 | |
| *** baojg has joined #openstack-swift | 07:40 | |
| *** pushkarajthorat has left #openstack-swift | 07:41 | |
| *** SkyRocknRoll_ has joined #openstack-swift | 08:13 | |
| *** jistr has joined #openstack-swift | 08:15 | |
| *** aix_ has quit IRC | 08:32 | |
| *** aix has joined #openstack-swift | 08:33 | |
| *** haigang has quit IRC | 08:44 | |
| *** haigang has joined #openstack-swift | 08:44 | |
| *** haigang has quit IRC | 08:50 | |
| *** haigang has joined #openstack-swift | 08:52 | |
| dimasot | hello all | 08:58 |
| dimasot | I found at proxy the next ccomment "import swiftclient | 08:59 |
| dimasot | import time | 08:59 |
| dimasot | conn = swiftclient.client.Connection(authurl='http://159.8.75.222:5000/v2.0',user="swift", key="passw0rd",tenant_name="service",auth_version='2') | 08:59 |
| dimasot | http_conn = conn.http_connection() | 08:59 |
| dimasot | print conn.put_object(container='test', obj='test.test6', contents="test6") | 08:59 |
| dimasot | print http_conn.getresponse() | 08:59 |
| dimasot | time.sleep(5) | 08:59 |
| dimasot | print conn.put_object(container='test', obj='test.test6', contents="test6") | 08:59 |
| dimasot | print http_conn.getresponse() | 08:59 |
| dimasot | conn.close() | 08:59 |
| dimasot | sorry wrong copy/past | 08:59 |
| *** sanchitmalhotra has joined #openstack-swift | 09:02 | |
| dimasot | I fount teh following comment # Sending an etag with if-none-match isn't currently supported | 09:02 |
| dimasot | at swift/proxy/controllers/obj.py | 09:03 |
| dimasot | does it means that 100 continue does supported? | 09:03 |
| *** sanchitmalhotra1 has quit IRC | 09:04 | |
| dimasot | is there an open bug on this issue? | 09:05 |
| *** chenhuayi has joined #openstack-swift | 09:13 | |
| *** haigang has quit IRC | 09:20 | |
| *** baojg has quit IRC | 09:23 | |
| *** sanchitmalhotra1 has joined #openstack-swift | 09:30 | |
| *** sanchitmalhotra has quit IRC | 09:33 | |
| *** haigang has joined #openstack-swift | 09:33 | |
| *** ig0r_ has joined #openstack-swift | 09:37 | |
| *** marzif__ has quit IRC | 09:38 | |
| *** baojg has joined #openstack-swift | 09:38 | |
| *** marzif__ has joined #openstack-swift | 09:38 | |
| *** ig0r__ has quit IRC | 09:39 | |
| *** aix has quit IRC | 09:49 | |
| *** baojg has quit IRC | 10:00 | |
| *** mahatic has quit IRC | 10:00 | |
| *** marzif__ has quit IRC | 10:12 | |
| *** haigang has quit IRC | 10:14 | |
| *** haigang has joined #openstack-swift | 10:15 | |
| *** aix has joined #openstack-swift | 10:18 | |
| *** sanchitmalhotra has joined #openstack-swift | 10:30 | |
| *** sanchitmalhotra1 has quit IRC | 10:32 | |
| *** m_kazuhiro has quit IRC | 10:35 | |
| *** m_kazuhiro has joined #openstack-swift | 10:36 | |
| *** marzif__ has joined #openstack-swift | 10:47 | |
| *** sanchitmalhotra1 has joined #openstack-swift | 10:56 | |
| *** sanchitmalhotra has quit IRC | 10:58 | |
| *** mahatic has joined #openstack-swift | 11:09 | |
| *** sanchitmalhotra has joined #openstack-swift | 11:12 | |
| *** sanchitmalhotra1 has quit IRC | 11:14 | |
| *** haomaiw__ has quit IRC | 11:15 | |
| *** sanchitmalhotra has quit IRC | 11:18 | |
| *** nakagawamsa has quit IRC | 11:20 | |
| *** sanchitmalhotra has joined #openstack-swift | 11:22 | |
| *** _hrou_ has joined #openstack-swift | 11:27 | |
| *** _hrou_ has quit IRC | 11:36 | |
| *** sanchitmalhotra has quit IRC | 11:38 | |
| *** marzif__ has quit IRC | 11:42 | |
| *** openstackgerrit has quit IRC | 11:46 | |
| *** openstackgerrit has joined #openstack-swift | 11:47 | |
| *** kairo has joined #openstack-swift | 11:58 | |
| *** kairo has quit IRC | 12:00 | |
| *** kairo has joined #openstack-swift | 12:01 | |
| *** km_ has quit IRC | 12:02 | |
| *** ppai_ has joined #openstack-swift | 12:02 | |
| *** ppai has quit IRC | 12:04 | |
| openstackgerrit | Merged openstack/swift: Fix EC GET backend stream iteration state https://review.openstack.org/199043 | 12:05 |
| *** m_kazuhi_ has joined #openstack-swift | 12:05 | |
| *** m_kazuhiro has quit IRC | 12:05 | |
| *** petertr7_away is now known as petertr7 | 12:09 | |
| *** kei_yama has quit IRC | 12:21 | |
| *** wbhuber_ has quit IRC | 12:27 | |
| *** haomaiwa_ has joined #openstack-swift | 12:37 | |
| *** ccavanna_ has quit IRC | 12:43 | |
| *** m_kazuhi_ has quit IRC | 12:43 | |
| *** annegentle has joined #openstack-swift | 12:44 | |
| *** ppai_ has quit IRC | 12:50 | |
| *** kota_ has quit IRC | 12:53 | |
| *** dimasot has quit IRC | 12:55 | |
| *** SkyRocknRoll_ has quit IRC | 13:01 | |
| *** AndreiaKumpera has joined #openstack-swift | 13:04 | |
| *** dustins has joined #openstack-swift | 13:04 | |
| *** CR7_ has joined #openstack-swift | 13:05 | |
| *** wbhuber has joined #openstack-swift | 13:21 | |
| *** SkyRocknRoll_ has joined #openstack-swift | 13:23 | |
| *** _hrou_ has joined #openstack-swift | 13:28 | |
| *** trifon has joined #openstack-swift | 13:30 | |
| *** links has quit IRC | 13:30 | |
| *** SkyRocknRoll_ has quit IRC | 13:32 | |
| *** esker has quit IRC | 13:47 | |
| *** esker has joined #openstack-swift | 13:48 | |
| *** bill_az has joined #openstack-swift | 13:49 | |
| *** annegentle has quit IRC | 13:49 | |
| *** breitz has quit IRC | 13:50 | |
| *** breitz has joined #openstack-swift | 13:50 | |
| *** jkugel has joined #openstack-swift | 13:51 | |
| *** janonymous has joined #openstack-swift | 13:58 | |
| *** trifon has quit IRC | 14:03 | |
| *** Kennan has quit IRC | 14:03 | |
| *** Kennan2 has joined #openstack-swift | 14:03 | |
| *** jroll has quit IRC | 14:09 | |
| *** jroll has joined #openstack-swift | 14:09 | |
| *** annegentle has joined #openstack-swift | 14:19 | |
| *** CR7_ has quit IRC | 14:24 | |
| *** ccavanna has joined #openstack-swift | 14:24 | |
| *** jlhinson has joined #openstack-swift | 14:30 | |
| *** jrichli has joined #openstack-swift | 14:37 | |
| *** jkremer has joined #openstack-swift | 14:48 | |
| *** petertr7 is now known as petertr7_away | 14:51 | |
| *** jistr is now known as jistr|call | 14:57 | |
| *** nexusz99 has joined #openstack-swift | 14:57 | |
| *** petertr7_away is now known as petertr7 | 14:59 | |
| *** lcurtis has joined #openstack-swift | 15:00 | |
| *** gyee has joined #openstack-swift | 15:07 | |
| *** dimasot has joined #openstack-swift | 15:10 | |
| *** vinsh has joined #openstack-swift | 15:16 | |
| *** nexusz99 has quit IRC | 15:21 | |
| mahatic | good evening | 15:23 |
| jrichli | mahatic: good evening. How are you? | 15:24 |
| *** wshao_ has joined #openstack-swift | 15:24 | |
| mahatic | jrichli: Morning! good now :) thanks | 15:25 |
| *** [1]dimasot has joined #openstack-swift | 15:26 | |
| jrichli | mahatic: good to hear you are feeling better. | 15:28 |
| *** dimasot has quit IRC | 15:28 | |
| *** [1]dimasot is now known as dimasot | 15:28 | |
| *** minwoob has joined #openstack-swift | 15:29 | |
| janonymous | jrichli mahatic : Texas cowboys :) | 15:29 |
| *** haomaiwa_ has quit IRC | 15:29 | |
| mahatic | jrichli: :) how is it going? Saw your progress on COPY (on trello), cool! | 15:31 |
| mahatic | jamielennox: I'm no texas cowboy :D | 15:31 |
| *** david-ly_ has joined #openstack-swift | 15:31 | |
| mahatic | janonymous: ^ | 15:31 |
| janonymous | hehe :) | 15:31 |
| janonymous | you gud ? | 15:32 |
| mahatic | yup, thanks | 15:32 |
| *** david-ly_ is now known as david-lyle_ | 15:33 | |
| *** [1]dimasot has joined #openstack-swift | 15:33 | |
| jrichli | mahatic: yes, I was happy to get a little farther. but there are still things to test and figure out. | 15:33 |
| *** david-lyle has quit IRC | 15:33 | |
| mahatic | yeah, true | 15:33 |
| *** david-lyle_ is now known as david-lyle | 15:35 | |
| *** hseipp has joined #openstack-swift | 15:36 | |
| *** dimasot has quit IRC | 15:36 | |
| *** [1]dimasot is now known as dimasot | 15:36 | |
| *** hseipp has quit IRC | 15:36 | |
| *** hseipp has joined #openstack-swift | 15:37 | |
| *** trifon has joined #openstack-swift | 15:41 | |
| openstackgerrit | janonymous proposed openstack/swift: pep8 fix: assertEquals -> assertEqual https://review.openstack.org/209645 | 15:43 |
| *** zaitcev has joined #openstack-swift | 15:48 | |
| *** ChanServ sets mode: +v zaitcev | 15:48 | |
| *** jistr|call is now known as jistr | 15:53 | |
| *** annegentle has quit IRC | 15:59 | |
| *** wshao_ has quit IRC | 15:59 | |
| *** chsc has joined #openstack-swift | 16:03 | |
| dimasot | hi all | 16:06 |
| dimasot | I have some problems with 100-continue | 16:07 |
| dimasot | I saw that there was some old bug that should be resolved that relates to my issue https://bugs.launchpad.net/openstack-api-site/+bug/1301677 | 16:07 |
| openstack | Launchpad bug 1301677 in openstack-api-site " Add "If-None-Match: *" support to PUT" [Medium,Fix released] - Assigned to Diane Fleming (diane-fleming) | 16:07 |
| *** rledisez has quit IRC | 16:08 | |
| *** mfalatic has joined #openstack-swift | 16:09 | |
| dimasot | but it looks like that feature still does not works for me | 16:10 |
| *** jistr has quit IRC | 16:13 | |
| zaitcev | I don't know if I have the time to look properly, but what is the actual symptom? Do you get 412 or not? | 16:17 |
| zaitcev | If not, then what happens? It hangs waiting for the body, sends 100-continue instead, or what? | 16:18 |
| zaitcev | (presuming the object already exists) | 16:18 |
| *** lyrrad has joined #openstack-swift | 16:20 | |
| *** jlhinson_ has joined #openstack-swift | 16:22 | |
| *** jlhinson has quit IRC | 16:22 | |
| openstackgerrit | janonymous proposed openstack/swift: pep8 fix: assertNotEquals -> assertNotEqual https://review.openstack.org/218937 | 16:23 |
| *** mahatic has quit IRC | 16:29 | |
| notmyname | good morning | 16:33 |
| *** lifeless has quit IRC | 16:34 | |
| *** annegentle has joined #openstack-swift | 16:35 | |
| notmyname | hmm... interesting https://review.openstack.org/#/c/217710/ | 16:36 |
| *** mahatic has joined #openstack-swift | 16:37 | |
| openstackgerrit | John Dickinson proposed openstack/swift: Updated CHANGELOG, AUTHORS, and .mailmap for 2.4.0 release. https://review.openstack.org/218681 | 16:40 |
| notmyname | peluse: around? | 16:40 |
| *** rjaiswal has joined #openstack-swift | 16:41 | |
| *** lifeless has joined #openstack-swift | 16:41 | |
| *** _hrou_ has quit IRC | 16:47 | |
| *** mahatic has quit IRC | 16:56 | |
| openstackgerrit | John Dickinson proposed openstack/swift: Updated CHANGELOG, AUTHORS, and .mailmap for 2.4.0 release. https://review.openstack.org/218681 | 16:57 |
| *** geaaru has quit IRC | 17:16 | |
| *** aix has quit IRC | 17:21 | |
| *** jlhinson_ has quit IRC | 17:32 | |
| *** hseipp has quit IRC | 17:34 | |
| *** SkyRocknRoll has quit IRC | 17:34 | |
| *** [1]dimasot has joined #openstack-swift | 17:35 | |
| *** dimasot has quit IRC | 17:37 | |
| *** [1]dimasot is now known as dimasot | 17:37 | |
| *** janonymous has quit IRC | 17:39 | |
| torgomatic | mattoliverau: interesting; what are the RPC ops folks going to do with the ring analyzer? | 17:42 |
| *** ig0r__ has joined #openstack-swift | 17:43 | |
| *** CaioBrentano has joined #openstack-swift | 17:44 | |
| * torgomatic wrote it primarily for use as a code-review-assistance tool | 17:45 | |
| *** ig0r_ has quit IRC | 17:46 | |
| dimasot | zaitcev: I uploaded the same file twice by swiftclient command: print conn.put_object(container='test', obj='100.test', contents=f, headers = {'if-none-match':"*","Expect": "100-continue"}) | 17:47 |
| dimasot | where f is the file heandler | 17:47 |
| dimasot | the first uploaded succed and the second failed with [Errno 32] Broken pipe | 17:48 |
| dimasot | the file is big >900MB | 17:48 |
| *** aix has joined #openstack-swift | 17:48 | |
| dimasot | when instead the file I use some short srting as a content the first succed the second failed as expected with 412 and others failed with 405 even it I try to upload new objects | 17:50 |
| dimasot | I tried it for 4 uploads in a line | 17:50 |
| openstackgerrit | John Dickinson proposed openstack/swift: Updated CHANGELOG, AUTHORS, and .mailmap for 2.4.0 release. https://review.openstack.org/218681 | 17:53 |
| clayg | ohai | 17:57 |
| clayg | jrichli: yeah the footer encryption sounds like maybe the wonky memcache footer stuff? Like maybe in the copy the name change is causing the memcace lookup for the footers to not do the right thing? | 17:58 |
| jrichli | clayg: im in a call, but will rely in a min | 18:01 |
| *** panbalag has joined #openstack-swift | 18:02 | |
| *** panbalag has left #openstack-swift | 18:02 | |
| *** chenhuayi has quit IRC | 18:08 | |
| *** dimasot has quit IRC | 18:14 | |
| *** dimasot has joined #openstack-swift | 18:15 | |
| *** petertr7 is now known as petertr7_away | 18:23 | |
| *** petertr7_away is now known as petertr7 | 18:25 | |
| *** mvandijk has joined #openstack-swift | 18:29 | |
| *** [1]dimasot has joined #openstack-swift | 18:29 | |
| *** dimasot has quit IRC | 18:33 | |
| *** gyee has quit IRC | 18:33 | |
| *** [1]dimasot is now known as dimasot | 18:33 | |
| jrichli | clayg: I wasn't aware of a name change. I know the object path will change for versioning, but I was thinking the PATH_INFO was the same in this case. But I will do some debugging in fake_footers to see whats happening. | 18:40 |
| jrichli | clayg: I was still curious what part of the system is failing to set user-metadata because the etag is binary data | 18:40 |
| *** panbalag has joined #openstack-swift | 18:41 | |
| *** ig0r__ has quit IRC | 18:48 | |
| openstackgerrit | Michael Barton proposed openstack/swift: go: replicator fix https://review.openstack.org/218984 | 18:55 |
| *** jlhinson has joined #openstack-swift | 18:59 | |
| peluse | hey I'm switching from windows to os/x (finally), anyone have a favorite IRC client they swear by? Or any other tools I would be stupid not to check out? | 19:07 |
| *** eranrom has joined #openstack-swift | 19:11 | |
| *** hrou has joined #openstack-swift | 19:12 | |
| *** aerwin has joined #openstack-swift | 19:17 | |
| vinsh | peluse: I settled on LimeChat. | 19:21 |
| vinsh | This was when I went ubuntu->mac os for primary laptop. | 19:21 |
| vinsh | 100% of the day is spent working ssh'd into all linux nodes :) | 19:22 |
| peluse | vinsh, thanks I'll check it out. I've been on a mac for years but stuck w/windows spending most of the day ssh'd into linux so go figure... spilled water on my system and had to get a new one so figured I'd ditch one OS :) | 19:27 |
| vinsh | :) | 19:27 |
| vinsh | the pluses about my mac outweigh the minuses. But thats for a whole nother channel ;) | 19:28 |
| *** petertr7 is now known as petertr7_away | 19:28 | |
| *** eranrom has quit IRC | 19:45 | |
| *** petertr7_away is now known as petertr7 | 19:48 | |
| dimasot | hi, when I am trying to use if-none-match and continue-100 headers for a big enought objects I got [Errno 32] Broken pipe instead responce 412 | 19:51 |
| dimasot | big enought is bigger then 1 MB | 19:51 |
| *** DericHorn-HP has joined #openstack-swift | 19:52 | |
| dimasot | in addition for small objects I got 412 for the first repetition and then I got 405 for a while - even for new objects | 19:52 |
| dimasot | is it a known issue? or should I open a bug related to this issue? should it be a single bug that describes both behaviours or one bug per issue? | 19:54 |
| dimasot | in addition to use it for contaiiner_sync purpose we need to extend it to support eTag also because 'if-none-math:*' would not be enought | 19:55 |
| clayg | isn't 405 method not allowed? | 19:56 |
| clayg | 409 would be timestamp conflict - but I think that's translated to a 202 | 19:56 |
| dimasot | yep, this wa svery strange to me too to get such a responce | 19:56 |
| clayg | really the x-if-match isn't quite as good as the timestamp match - just because the body matches doesn't mean the metadata does - i'm not sure if the symantics should keep a request from syncing "mearly" for the same etag - only timestamp will do (I believe) | 19:57 |
| dimasot | yes, today when I am suncing the same object by container_sync I see 409 at object server and 202 at proxy | 19:57 |
| clayg | well that part sounds right | 19:57 |
| dimasot | but at that case I move the smae object again over th ewire | 19:57 |
| clayg | dimasot: I think the issue may come down to pipelining | 19:58 |
| clayg | dimasot: torgomatic and I were discussing this just this morning (related to a ssync change) - I think if .getexpect doesn't give you back 100 you have to tear down the connection and make a new one before you can send another request | 19:58 |
| dimasot | I see | 19:59 |
| clayg | it sucks... but I'm not sure rfc 2616 allows another option - and almost cirtainly eventlet.wsgi.server is not going to accept a piplined request behind a 4XX on a request that was supposed to include a body | 19:59 |
| dimasot | so it resolves teh 405 issue | 19:59 |
| dimasot | but what about [Errno 32] Broken pipe instead 4112? | 20:00 |
| clayg | dimasot: yeah I suppose if it's reads some random amount of content-length off the wire and the next line is the middle of something else... it would be some kind "wtf is this request" response? | 20:00 |
| clayg | dimasot: which end is finding the closed socket? the client maybe? some sort of timeout waiting at getexpect perhaps? | 20:00 |
| *** david-lyle has quit IRC | 20:01 | |
| dimasot | I uploaded the same object twice with 60 seconds delay between the uploads | 20:02 |
| dimasot | if the seconf one was without if-none-match header both of then was successful | 20:02 |
| clayg | dimasot: seperate connections? | 20:02 |
| dimasot | but if I add the if-non-match header to the second one I got [Errno 32] Broken pipe | 20:02 |
| clayg | dimasot: well you definately need to send the x-timestamp if you're not going to send if-match | 20:03 |
| dimasot | the same one, I can try the separate connection also | 20:03 |
| openstackgerrit | Joel Wright proposed openstack/python-swiftclient: Log and report trace on service operation fails https://review.openstack.org/171692 | 20:03 |
| clayg | dimasot: yeah that doesn't make much sense - i don't think I see why if-none-match would cause the server to shutdown the connection... | 20:04 |
| clayg | dimasot: I do think the server should not process the request - but it might should eat the bytes if the client tries to send them | 20:04 |
| dimasot | so to summarize you suggest to use a saparate connection for each upload and then you expect I will get 412 regardless to the object size | 20:04 |
| clayg | dimasot: well - I just think that if getexpect returns anything besides 100 you need to close the connection before you can try sending the next object - as long as getexpect is returning 100 i think you can (should?) keep using the same connection | 20:05 |
| dimasot | the good thing was that I did not see network trafic for the second request (when I use if-none-match) | 20:05 |
| *** david-lyle has joined #openstack-swift | 20:06 | |
| dimasot | ok, I see | 20:06 |
| notmyname | peluse: I also use LimeChat (and I like it) | 20:06 |
| *** petertr7 is now known as petertr7_away | 20:06 | |
| dimasot | and now related to extendig if-none-match by eTag | 20:06 |
| *** biyiklioglu has joined #openstack-swift | 20:07 | |
| dimasot | it looks like proxy kills all the request where if-none-match != * | 20:07 |
| dimasot | but the object server supports both * and eTag | 20:08 |
| dimasot | so it looks like allowing of eTag bypassing at proxy will resolve this eTag related issue, or I miss something? | 20:09 |
| clayg | dimasot: I don't think i follow your explination - nor do I think the proxy should be closing the connection - I think if you're pipling with expect-100 continue things will break | 20:10 |
| clayg | "but the object server supports both * and eTag" < I don't know what that means | 20:10 |
| clayg | "allowing of eTag bypassing at proxy" < I don't know what that means | 20:10 |
| dimasot | object server returns 412 for both cases: if-none-match == * or if-none-match == eTag of stored object | 20:11 |
| dimasot | https://github.com/openstack/swift/blob/master/swift/obj/server.py#L551 | 20:12 |
| clayg | dimasot: if you're doing container_sync stuff don't use if-none-match - x-timestamp is the right way to get a expect-100 request to bail early if the object has already been sent | 20:12 |
| dimasot | but proxy failes all the request with if-none-match != * | 20:13 |
| dimasot | https://github.com/openstack/swift/blob/master/swift/proxy/controllers/obj.py#L677 | 20:13 |
| dimasot | so today if I set if-none-math to any value except * I will get 400 responce | 20:14 |
| clayg | dimasot: that's true | 20:14 |
| clayg | dimasot: but don't use if-none-match for contaienr_sync - use x-timestamp | 20:14 |
| dimasot | but for container_sync we want to avoid movig the same data over wire multiple times on one hand, but on another we want to sync between the containers and copy the latest version of object | 20:15 |
| dimasot | but at today's implementation I already use x-timestamp | 20:16 |
| clayg | dimasot: well I don't think the existing if-match support for PUT will be sufficient to avoid the former - but to avoid syncing an object twice x-timestamp would be near ideal | 20:16 |
| dimasot | but it does not protect me from moving object data between container server to remote proxy server | 20:16 |
| clayg | dimasot: I'm not sure if "todays" version supports 100-continue correctly tho | 20:17 |
| dimasot | it just saves teh proxy to object network (at remote cluster) | 20:17 |
| dimasot | I see | 20:17 |
| clayg | dimasot: yeah but I think that's mostly in how the client is handling 100-continue - not so much if-match vs. x-timestamp | 20:17 |
| dimasot | so you suggest to use 100-continue with x-timestamp | 20:17 |
| clayg | dimasot: I think that is the best plan | 20:18 |
| dimasot | now, I see | 20:18 |
| dimasot | yep, sound like a plan | 20:18 |
| dimasot | on, now I need to dig into th ecode to see what is required to support such an option | 20:19 |
| dimasot | should I open a bug or something on this issue to notify that I working on it? | 20:19 |
| *** rjaiswal has quit IRC | 20:20 | |
| dimasot | clayg: thank you for you help | 20:22 |
| clayg | dimasot: np, I don't think "support arbitrary etag on if-none-match" makes much sense right now - I think container-sync not doing the right thing WRT pipelined 100-Continue is a bug - but I don't think filing it is as important as digging in understanding what's broke getting a patch up and helping people get it tested/reviewed/merged | 20:33 |
| clayg | ... but maybe a bug (and failing test case) are helpful in *that* process? | 20:33 |
| dimasot | I see, so I will start from adding 100-continue to the x-timestamp and then will try to find/resolve the pipelined 100-Continue issue | 20:38 |
| dimasot | thansk again | 20:38 |
| *** oddsam91 has joined #openstack-swift | 20:39 | |
| *** gyee has joined #openstack-swift | 20:43 | |
| *** dimasot has quit IRC | 20:44 | |
| openstackgerrit | Joel Wright proposed openstack/python-swiftclient: Log and report trace on service operation fails https://review.openstack.org/171692 | 20:48 |
| *** bkopilov has quit IRC | 20:48 | |
| *** bkopilov has joined #openstack-swift | 20:53 | |
| oddsam91 | Hi there, Is there someone in here who can help me out. I'm looking to verify if my swift cluster is actually consuming more data or if there's some failure in the clean up processes. | 20:54 |
| *** bkopilov has quit IRC | 21:00 | |
| *** esker has quit IRC | 21:01 | |
| openstackgerrit | Joel Wright proposed openstack/python-swiftclient: Log and report trace on service operation fails https://review.openstack.org/171692 | 21:03 |
| *** annegentle has quit IRC | 21:04 | |
| *** joeljwright has joined #openstack-swift | 21:07 | |
| *** ChanServ sets mode: +v joeljwright | 21:07 | |
| notmyname | oddsam91: I'm not sure I follow what you're asking | 21:09 |
| oddsam91 | Well I'm noticing that over the past few months the average amount of disk space used is trending upwards at a rate that doesn't seem to be consistent with natural growth/higher usage of the cluster. | 21:10 |
| oddsam91 | So I'm trying to find if there's something swift isn't cleaning up. | 21:11 |
| notmyname | oddsam91: how are you tracking the disk space? | 21:11 |
| oddsam91 | df -h | 21:11 |
| oddsam91 | well, I mean I'm using collectd | 21:11 |
| notmyname | and what are you comparing that to? I mean, how do you know that `df -h` is something that is out of line? | 21:12 |
| oddsam91 | http://imgur.com/EZ9WS8m | 21:17 |
| clayg | oddsam91: looks like a fairly liner growth curve? are your users not expected to have been adding new data? | 21:18 |
| clayg | nightly backups maybe? new vm images? new assets? what makes you think the growth isn't organic and somehow disassociated with client usage? | 21:19 |
| *** esker has joined #openstack-swift | 21:19 | |
| clayg | oddsam91: GREAT graph BTW! up and to the left == good | 21:19 |
| clayg | er... to the right - one of those is good | 21:19 |
| oddsam91 | Haha, thanks. I'm not convinced this is actual growth though. But no, my swift cluster is supposed to cycle out all data older than 36 hours. | 21:19 |
| joeljwright | :) | 21:20 |
| *** bkopilov has joined #openstack-swift | 21:20 | |
| clayg | oddsam91: ah... so lots of expiring objects? | 21:20 |
| oddsam91 | right | 21:20 |
| joeljwright | are you using X_EXPIRE_AFTER? | 21:20 |
| oddsam91 | X-Delete-At | 21:20 |
| joeljwright | and large objects? | 21:20 |
| clayg | oddsam91: k, well you better check on the expirer then | 21:20 |
| joeljwright | because you could be leaving the segments on disk | 21:20 |
| oddsam91 | We have a max file size of 50Mb enforced at a different level | 21:20 |
| clayg | redbo: dfg_: glange: hurricanerix: what is alex's handle? | 21:21 |
| notmyname | clayg: allen (alan)? | 21:21 |
| hurricanerix | clayg: alan? it's aerwin | 21:21 |
| oddsam91 | I did locate tombstone files that have never been cleaned up circa Dec 2013 when the cluster was stood up. | 21:21 |
| clayg | oddsam91: grrrr... lp bug #1301728 | 21:23 |
| openstack | Launchpad bug 1301728 in OpenStack Object Storage (swift) "tombstone (.ts) object will never be deleted if the hash_suffix exists in the hashes.pkl " [Undecided,In progress] https://launchpad.net/bugs/1301728 - Assigned to Brian Cline (briancline) | 21:23 |
| oddsam91 | But I'm fairly certain the expirer is able to keep up. I did some spot checking around the different containers using the swift api and I didn't find a single file whose X-Delete-At header was in the past | 21:23 |
| oddsam91 | yeah I did run into that one in my 100+ google searches | 21:23 |
| clayg | hurricanerix: notmyname: yes *allan* - oddsam91 aerwin knows everything there is to know about dealing with an expirer that's backed up | 21:23 |
| clayg | oddsam91: oh | 21:23 |
| oddsam91 | I'm using an xfs file system would those 0 byte .ts file actually consume disk space | 21:23 |
| clayg | oddsam91: sure inodes are a thing | 21:24 |
| *** esker has quit IRC | 21:24 | |
| redbo | and dirents | 21:24 |
| oddsam91 | df -i reports only ~36% of the inodes are consumed | 21:24 |
| clayg | redbo: yes even more so the wasted hashdirs :'( | 21:24 |
| redbo | yeah | 21:24 |
| *** esker has joined #openstack-swift | 21:24 | |
| clayg | oddsam91: yeah not os much that you would run out - just that yeah having tombstones not getting cleaned up is sort of stupid and annoying | 21:24 |
| oddsam91 | Agreed. Just not sure if I can point to that and say "ah that's the cause for increased disk usage" | 21:25 |
| clayg | I think the latest plan on that bug was to make the auditer just use reclaim age to make the tombstones & hashdirs go away - leaving the hashes.pkl entry alone - maybe - I acctually forget what the plan is | 21:25 |
| *** dustins has quit IRC | 21:26 | |
| oddsam91 | (Of course it probably doesn't help if I'm running 1.4.0) | 21:26 |
| oddsam91 | latest swift is like 10+ minor versions ahead of my install | 21:26 |
| clayg | oddsam91: ok, so if it's not expired data files - is it the unreclaimed tombstones? Like have you tried deleting a hashes.pkl in a part here and there and watching the tombstones get reaped - is that freeing up disk space? | 21:27 |
| clayg | oddsam91: we're 2.0 baby! | 21:27 |
| oddsam91 | I haven't tried anything destructive yet. However, I can say at least some disk space is being reclaimed if you notice in the graph I posted a while back, there significant dips on the weekends (which system utilization is much, much lower). | 21:29 |
| oddsam91 | the resolution is too low to see the daily cleanup but there's still about 8% points of disk space reclaimed every night | 21:29 |
| oddsam91 | ~119GB per store | 21:30 |
| clayg | oddsam91: "daily cleanup" - like something besdies the expirer? | 21:30 |
| *** wbhuber has quit IRC | 21:31 | |
| *** jkremer has quit IRC | 21:31 | |
| oddsam91 | No it's just the expirer. But I say daily cleanup because we've set the expire time to be 36 hours after upload. Which is approximately 9pm the next day. | 21:31 |
| oddsam91 | So it sorta all happens at once | 21:32 |
| clayg | oic | 21:33 |
| joeljwright | oddsam91: I have seen behaviour like this caused by uploading SLO/DLO with the python-swiftclient, because the expiry headers are only set on the manifest files | 21:33 |
| clayg | joeljwright: ^ good idea! | 21:34 |
| oddsam91 | Uploads are from a php client. | 21:34 |
| oddsam91 | which, arguably, could be worse lol | 21:34 |
| joeljwright | it's worth checking if you're creating any large objects | 21:34 |
| oddsam91 | what do you define as large? | 21:35 |
| joeljwright | and if so, whether the x-delete-at header is being set on the segments | 21:35 |
| oddsam91 | Max *should* be 50mb | 21:35 |
| joeljwright | anything using SLO/DLO | 21:35 |
| *** annegentle has joined #openstack-swift | 21:35 | |
| oddsam91 | I'm not sure what SLO/DLO is | 21:35 |
| joeljwright | http://docs.openstack.org/developer/swift/overview_large_objects.html | 21:35 |
| oddsam91 | Ah I see | 21:36 |
| oddsam91 | I can't imagine we're using this. | 21:36 |
| joeljwright | ok, that was my best guess :D | 21:36 |
| oddsam91 | I now understand what you mean by segments | 21:36 |
| joeljwright | you can find out by looking for objects with either 'x-object-manifest' or 'x-static-large-object' headers | 21:37 |
| joeljwright | but if you're enforcing <50Mb before upload it shouldn't be that | 21:37 |
| *** annegent_ has joined #openstack-swift | 21:38 | |
| oddsam91 | Ah gotcha. right | 21:39 |
| oddsam91 | http://pastebin.com/GsbqGnH6 | 21:39 |
| oddsam91 | here's an object's metadata that was just uploaded | 21:39 |
| *** annegentle has quit IRC | 21:40 | |
| joeljwright | well that looks like a simple 26M zip file | 21:40 |
| joeljwright | if you never see those two headers, then unfortunately it can't be the thing I suggested | 21:41 |
| oddsam91 | Right. | 21:41 |
| oddsam91 | So the 0 byte .ts files (while annoying) aren't a cause for increased disk usage (just increased inode usage) correcT? | 21:42 |
| joeljwright | that one I can't answer confidently | 21:43 |
| clayg | oddsam91: well no - the inodes and dirents do take up space on the disk (by way of filesystem metadata) | 21:44 |
| clayg | oddsam91: it's not *much* space - but it can add up | 21:44 |
| clayg | oddsam91: you could see how much (if any) it's impacting your deployment by cleaning some up | 21:44 |
| oddsam91 | besides doing an rm -rf /mnt/path/to/objects/*/*/*/*.ts what's the best way of doing this? | 21:45 |
| clayg | oddsam91: the easiest way I can think of to clean them up is to delete a hashes.pkl in a part some where - this will trigger some load (by way of REPLICATE requests, which will do listdirs and such) - but it's going to be fairly nominal | 21:45 |
| clayg | ^ redbo you got a better way to trigger a rehash/reclaim of a bunch of tombstones? | 21:45 |
| oddsam91 | Oh and in a single day, (one storage node out of three): | 21:46 |
| oddsam91 | 350152 DELETE | 21:46 |
| oddsam91 | 580504 GET | 21:46 |
| oddsam91 | 420575 PUT | 21:46 |
| clayg | oddsam91: well there's *some* growth there if PUT > DELETE :) | 21:46 |
| oddsam91 | If I had what is used to be I'd be able to know for sure. | 21:46 |
| clayg | *assuming more or less consistent object size :P | 21:46 |
| oddsam91 | I'd say on avg no increase in object size | 21:46 |
| oddsam91 | clayg: I just processed what you said about growth, that does make some sense. But most of our deletes happen in the pm so it'll go up later tonight. I'll check yesterday's log | 21:49 |
| oddsam91 | Well thursdays log | 21:50 |
| oddsam91 | 1628305 DELETE | 21:51 |
| oddsam91 | 776958 GET | 21:51 |
| oddsam91 | 528974 PUT | 21:51 |
| *** biyiklioglu has quit IRC | 21:51 | |
| *** CaioBrentano has quit IRC | 21:55 | |
| clayg | oddsam91: well *that's* going in the wrong direction :P | 21:55 |
| oddsam91 | Definitely. Especially when the graph is going the other way. | 21:56 |
| clayg | i'm just sayin' you need to add moar data! | 21:57 |
| clayg | notmyname always says the DELETE verb was a mis-feature | 21:57 |
| *** joeljwright has quit IRC | 21:59 | |
| oddsam91 | clayg: well sounds like the collective next best move would be to delete a few hashes.pkl and see what happens to disk usage? | 22:00 |
| oddsam91 | It also sounds like aerwin may be able to help further? | 22:00 |
| *** ccavanna has quit IRC | 22:01 | |
| clayg | oddsam91: well idk, not if the expirer is keeping up already - then that's not really the issue | 22:01 |
| oddsam91 | Maybe I suck at verifying if the expirer is truly keeping. How might I verify that properly? | 22:02 |
| aerwin | oddsam91: one thing you can do is go through the .expiring_objects account and see if there are any objects in containers with timestamp before now. | 22:07 |
| aerwin | That will tell you how far behind you are, if you are behind | 22:07 |
| oddsam91 | Ah! Let me try that | 22:08 |
| oddsam91 | should I do that via the api or the file system/ | 22:08 |
| aerwin | I use the Swiftly DirectClient to handle the requests. | 22:09 |
| oddsam91 | I've just been doing curl requests directly | 22:10 |
| *** chsc has quit IRC | 22:11 | |
| oddsam91 | I've g2g but I'll check into the .expiring_objects account tomorrow. Is there something documenting this account besides source code? | 22:12 |
| aerwin | Not as far as I can tell. This is all I have been able to find. http://docs.openstack.org/developer/swift/overview_expiring_objects.html | 22:13 |
| oddsam91 | That's what I just found. I might hit you guys up tomorrow with more questions. Thanks for all the help/support | 22:13 |
| aerwin | But, I am working on submitting a spec to change the way the expiring of objects works as a whole. | 22:14 |
| aerwin | Your welcome. | 22:14 |
| oddsam91 | Considering I'm still on 1.4.0, it maybe more effort to upgrade to a new version of swift than just wiping out all the data in the cluster. | 22:14 |
| oddsam91 | But I'd prefer not to go nuclear. | 22:14 |
| aerwin | I can understand that idea. | 22:15 |
| oddsam91 | Thanks again! | 22:15 |
| *** oddsam91 has quit IRC | 22:15 | |
| *** jkugel has quit IRC | 22:16 | |
| *** wbhuber has joined #openstack-swift | 22:17 | |
| *** garthb has joined #openstack-swift | 22:19 | |
| mattoliverau | Morning | 22:22 |
| peluse | yo | 22:22 |
| *** jrichli has quit IRC | 22:23 | |
| aerwin | mattoliverau: o/ | 22:23 |
| mattoliverau | aerwin: hey man! yes please spec us with you expiring objects fix, then if you don't have time to implement we'll find someone (even if its me) :) | 22:28 |
| *** jlhinson has quit IRC | 22:29 | |
| *** annegent_ has quit IRC | 22:32 | |
| *** annegentle has joined #openstack-swift | 22:32 | |
| aerwin | Well I am back to CFile so I am focused on getting this fixed. :) | 22:33 |
| *** flwang1 has joined #openstack-swift | 22:38 | |
| flwang1 | cschwede: ping | 22:41 |
| *** resker has joined #openstack-swift | 22:43 | |
| *** esker has quit IRC | 22:47 | |
| *** resker has quit IRC | 22:48 | |
| *** esker has joined #openstack-swift | 22:48 | |
| garthb | Hello looking for some help on fixing a devstack swift issue and uploading cirros images... | 22:49 |
| garthb | Here's one of the uploads that's failing: openstack --os-token 374ce3ff7672466aa485e21934aedba0 --os-url http://10.50.177.1:9292 image create cirros-0.3.4-x86_64-uec-kernel --public --container-format aki --disk-format aki | 22:50 |
| garthb | Here's a pointer to 2 paste files for s-proxy.log and g-api.log: http://paste.openstack.org/show/435921/ and http://paste.openstack.org/show/435922/ | 22:51 |
| garthb | The upload fails with ERROR: openstack 500 Internal Server Error: Failed to upload image 3936ee83-7195-43e1-ad04-869b733259c2 (HTTP 500) | 22:51 |
| garthb | It works when run from the command line immediately after devstack bails... | 22:52 |
| garthb | any suggestions / pointers would be greatly appreciated...spent several hrs exploring google, launchpad, etc. no luck | 22:53 |
| *** annegentle has quit IRC | 23:04 | |
| flwang1 | garthb: i'm not a swift guy but glance | 23:08 |
| flwang1 | so, does your glance service and swift service works fine separately? | 23:09 |
| *** hrou has quit IRC | 23:09 | |
| garthb | flwang1 yes I believe so...I'm able to manually run the upload after stack.sh fails... | 23:11 |
| garthb | flwang1 i.e. openstack --os-token 374ce3ff7672466aa485e21934aedba0 --os-url http://10.50.177.1:9292 image create cirros-0.3.4-x86_64-uec-kernel --public --container-format aki --disk-format aki | 23:11 |
| *** lcurtis has quit IRC | 23:13 | |
| *** tanee has quit IRC | 23:16 | |
| *** tanee has joined #openstack-swift | 23:16 | |
| *** km has joined #openstack-swift | 23:17 | |
| flwang1 | just reviewed the 2 logs, seems it's swift error. can you run those swift command manually? | 23:18 |
| garthb | flwang1 I just unstacked and stacked...one moment, I'll give those swift commands a manual try... | 23:22 |
| *** kei_yama has joined #openstack-swift | 23:31 | |
| notmyname | FYI I'll be turning the crank on the release machinery tonight. swift 2.4.0 | 23:32 |
| mattoliverau | notmyname: \o/ | 23:32 |
| garthb | flwang1 curl -i http://10.50.177.1:8080/v1/AUTH_9a92a5007f8a4fd3892feea8147816ab/glance -I -H "X-Auth-Token: 0921030cfcab48f8a1ae6c8a91f220b1"... | 23:39 |
| garthb | flwang1 fails with authorization error manually as well... | 23:39 |
| flwang1 | garthb: you need some help from the swift guys :) | 23:40 |
| garthb | flwang1: yeah, I suspected the same...thanks for taking a look though | 23:42 |
| *** kota_ has joined #openstack-swift | 23:45 | |
| *** ChanServ sets mode: +v kota_ | 23:45 | |
| kota_ | morning | 23:46 |
| mattoliverau | kota_: morning | 23:48 |
| mattoliverau | garthb: hmm, looks like the account server isn't working properly, it isn't just not running otherwise I'd expect a Connection Refused. Is the account server running or something else running on the account servers port? | 23:49 |
| ho | good morning! | 23:51 |
| *** DericHorn-HP has quit IRC | 23:54 | |
| garthb | mattoliverau: hmmm...I don't see it in the ps list...I see a bunch of other swift servers though... | 23:55 |
| garthb | mattoliverau: can I try manually starting it with /usr/bin/python /usr/local/bin/swift-account-server /etc/swift/account-server/1.conf? | 23:56 |
| mattoliverau | garthb: I don't use devstack (well not in a while), can you run: swift-init account start? | 23:57 |
| mattoliverau | that is if swift has been installed to the system properly | 23:57 |
| mattoliverau | garthb: does it specify a port in /etc/swift/account-server/1.conf? | 23:58 |
| garthb | mattoliverau: That fails with Exception: Could not bind to 0.0.0.0:6012 after trying for 30 seconds | 23:58 |
| mattoliverau | garthb: right somethign else is using the port | 23:58 |
| mattoliverau | garthb: netstat -ntlp | 23:59 |
| garthb | mattoliverau: /etc/swift/account-server/1.conf specifies port 6012 | 23:59 |
| mattoliverau | garthb: does netstat show something else using the port? | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!