| *** dmsimard is now known as dmsimard_away | 00:10 | |
| *** acoles has quit IRC | 00:13 | |
| *** jwang_ has joined #openstack-swift | 00:14 | |
| *** acoles_away has joined #openstack-swift | 00:15 | |
| *** acoles_away is now known as acoles | 00:15 | |
| *** ChanServ sets mode: +v acoles | 00:15 | |
| *** dmorita has joined #openstack-swift | 00:25 | |
| *** tellesnobrega has quit IRC | 00:37 | |
| *** shri1 has joined #openstack-swift | 00:38 | |
| *** shri1 has quit IRC | 00:38 | |
| *** shri has quit IRC | 00:40 | |
| *** tellesnobrega has joined #openstack-swift | 00:44 | |
| *** tellesnobrega_ has quit IRC | 00:46 | |
| *** Masahiro has joined #openstack-swift | 00:46 | |
| *** Masahiro has quit IRC | 00:50 | |
| *** addnull has joined #openstack-swift | 01:00 | |
| *** Masahiro has joined #openstack-swift | 01:08 | |
| *** addnull has quit IRC | 01:31 | |
| *** nellysmitt has joined #openstack-swift | 01:32 | |
| *** nellysmitt has quit IRC | 01:36 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift: Efficient Replication for Distributed Regions https://review.openstack.org/99824 | 01:44 |
|---|---|---|
| *** kota_ has joined #openstack-swift | 01:46 | |
| *** rmcall has quit IRC | 01:53 | |
| *** nosnos has joined #openstack-swift | 01:56 | |
| *** tellesnobrega_ has joined #openstack-swift | 02:04 | |
| *** addnull has joined #openstack-swift | 02:14 | |
| *** haomaiwang has joined #openstack-swift | 02:14 | |
| *** zackmdavis has left #openstack-swift | 02:14 | |
| *** Masahiro has quit IRC | 02:18 | |
| *** Masahiro has joined #openstack-swift | 02:21 | |
| *** jwang_ has quit IRC | 02:34 | |
| *** Masahiro has quit IRC | 02:52 | |
| *** Masahiro has joined #openstack-swift | 02:53 | |
| *** addnull has quit IRC | 02:55 | |
| *** Masahiro has quit IRC | 02:59 | |
| *** bill_az has quit IRC | 03:09 | |
| *** Masahiro has joined #openstack-swift | 03:22 | |
| *** madhuri has joined #openstack-swift | 03:28 | |
| madhuri | Hi, I am facing an issue while object PUT. I pressed control + C from the terminal which closes the connection between client and proxy. Proxy returned 499 | 03:30 |
| madhuri | But connection between proxy and object was still connected. And object service returned 408 after client_timeout of 60 seconds. | 03:31 |
| madhuri | I think the connection between the proxy and object must also get closed in above. | 03:31 |
| madhuri | I am using Icehouse stable release | 03:31 |
| *** nellysmitt has joined #openstack-swift | 03:32 | |
| *** nellysmitt has quit IRC | 03:37 | |
| *** nosnos has quit IRC | 03:39 | |
| mattoliverau | madhuri: that's expected, when you put, the proxy opens a connection to the storage nodes and streams the data to them. When you cancelled the client connection (499), the storage server connections are still open but now stop receiving data, that is until they time out (408). | 03:47 |
| madhuri | But we can explicitly close the connection. | 03:56 |
| madhuri | In proxy/controller/obj.py in exception cases | 03:57 |
| mattoliverau | I don't think a control + c does a friendly close | 03:57 |
| *** addnull has joined #openstack-swift | 03:59 | |
| madhuri | Yes. I agree | 03:59 |
| madhuri | But my concern is we can close it like this | 04:00 |
| madhuri | conn.close() | 04:00 |
| madhuri | http://paste.openstack.org/show/143905/ | 04:01 |
| madhuri | mattoliverau: Can you suggest would it be acceptable to do this change | 04:01 |
| madhuri | ? | 04:01 |
| mattoliverau | madhuri: where are you putting this code? When you return the HTTPClientDisconnect, the client will get its response so close the wsgi connection. | 04:04 |
| madhuri | http://paste.openstack.org/show/143911/ | 04:05 |
| madhuri | Please find the diff above | 04:06 |
| madhuri | mattoliverau: Can I ping you after half an hour> Got to go. | 04:09 |
| mattoliverau | madhuri: of course, I'll be around :) | 04:10 |
| madhuri | Thanks mattoliverau :) | 04:10 |
| *** nosnos has joined #openstack-swift | 04:19 | |
| *** abhirc_ has joined #openstack-swift | 04:24 | |
| *** abhirc has quit IRC | 04:27 | |
| *** addnull has quit IRC | 04:28 | |
| *** bill_az has joined #openstack-swift | 04:34 | |
| madhuri | mattoliverau: Hi | 05:03 |
| mattoliverau | madhuri: welcome back :) | 05:04 |
| madhuri | Thank you | 05:04 |
| madhuri | Have a look on the patch | 05:05 |
| *** kopparam has joined #openstack-swift | 05:06 | |
| mattoliverau | madhuri: I've looked at the patch, but let me refamilarise myself if the code in question :) | 05:07 |
| madhuri | ok | 05:09 |
| *** ppai has joined #openstack-swift | 05:10 | |
| *** alexiz has quit IRC | 05:15 | |
| *** sungju has quit IRC | 05:17 | |
| *** kopparam has quit IRC | 05:20 | |
| *** kopparam has joined #openstack-swift | 05:21 | |
| *** addnull has joined #openstack-swift | 05:24 | |
| mattoliverau | madhuri: is the 408 and the 499 occuring at the same time? Or does the 408 happen 60 seconds after the other? Just making checking to see if the exception catching of the client disconnect is cleaning up (closing the connection) in which case nothing has to be done.. I might have to test it myself. But one thing you;ll have a problem with is that the conn object doesn't exist. | 05:27 |
| madhuri | 408 from object service is raised after 60 seconds after proxy returned 499 | 05:28 |
| *** dvorkbjel has quit IRC | 05:28 | |
| *** tellesnobrega_ has quit IRC | 05:28 | |
| madhuri | I have checked closing connection explicitly and it worked. | 05:28 |
| madhuri | But not in one case. | 05:29 |
| *** zaitcev has quit IRC | 05:30 | |
| madhuri | That is once object service has started reading the chunks, even after closing the connection explicitly(the 3rd conn.close in the patch). The object returned 408 after 60 second. | 05:30 |
| *** dvorkbjel has joined #openstack-swift | 05:31 | |
| madhuri | That is when the object server failed to read the next chunk in 60 seconds. | 05:31 |
| *** SkyRocknRoll has joined #openstack-swift | 05:31 | |
| *** SkyRocknRoll has joined #openstack-swift | 05:31 | |
| madhuri | How is object service able to read from a closed connection? Does conn.close() not closing the socket? | 05:32 |
| *** nellysmitt has joined #openstack-swift | 05:33 | |
| *** abhirc_ has quit IRC | 05:34 | |
| mattoliverau | Well looking at the icehouse code.. unless I'm missing something, which I could be, there isn't a conn object, there is a list of connections called conns | 05:35 |
| madhuri | Yes, you are right. | 05:36 |
| madhuri | But each conn object is iterated from conns list to send file to each storage server | 05:36 |
| *** nellysmitt has quit IRC | 05:38 | |
| mattoliverau | which is being run in the background via a greenlet pool, the and peices of data are being placed in the conn objects queue to be pushed. So your dealing with x conn objects at once. | 05:39 |
| mattoliverau | sorry a contextpool | 05:39 |
| madhuri | Yes mattoliverau :) | 05:39 |
| mattoliverau | madhuri: so you'd need to loop through all the conn objects in conns and call close, if you wanted to be sure to close them all. | 05:40 |
| madhuri | Yes. But the one for which exception has raised | 05:42 |
| mattoliverau | madhuri: the exception is being raised on the client connection isn't it? not on the conns | 05:43 |
| madhuri | Yes | 05:44 |
| madhuri | You are right. | 05:44 |
| madhuri | And for the client connections, if any exception raised so we need to close the object service connection also. | 05:44 |
| mattoliverau | When the ContectPool leaves the 'with' scope each coroutine is killed.. I wonder why this isn't closing the connections. | 05:45 |
| mattoliverau | *contextpool | 05:45 |
| mattoliverau | madhuri: and the object server connections are inside the conns object. | 05:46 |
| mattoliverau | sorry conns list (I'm still a little jetlagged :P) | 05:46 |
| madhuri | Yes | 05:46 |
| mattoliverau | Firstly, I'm wondering why the connection aren't automatically closed when the pool object (contextpool) object isn't garbage collected, seeing as there is an __exit__ method defined which should kill all the coroutines (conns) running inside it. | 05:49 |
| mattoliverau | *is garbage collected | 05:50 |
| mattoliverau | wow, I fail at making sentenses today | 05:50 |
| *** rmcall has joined #openstack-swift | 05:52 | |
| *** rmcall has quit IRC | 05:52 | |
| *** Masahiro has quit IRC | 05:52 | |
| madhuri | Yes. I am also wondering. The connections were alive :( | 05:52 |
| *** Masahiro has joined #openstack-swift | 05:53 | |
| *** rmcall has joined #openstack-swift | 05:54 | |
| *** rmcall_ has joined #openstack-swift | 05:59 | |
| *** kopparam has quit IRC | 06:00 | |
| *** rmcall has quit IRC | 06:02 | |
| madhuri | mattoliverau: Let me know if you get the answer. I am looking into it. | 06:03 |
| mattoliverau | madhuri: I wonder if this will work. In _send_file (the function being run in contextpool) if we get it to close the connection on cleanup, something like: | 06:03 |
| mattoliverau | with contextlib.closing(conn): | 06:04 |
| *** rmcall_ has quit IRC | 06:04 | |
| mattoliverau | <do the rest> | 06:04 |
| mattoliverau | So when it is killed (garbage collected) the connection is also closed at garbage collection. | 06:04 |
| *** rmcall has joined #openstack-swift | 06:04 | |
| mattoliverau | I don't know if that would work, but might be cleaner | 06:04 |
| mattoliverau | As it means it will clean itself up | 06:05 |
| mattoliverau | * with contextlib.closing(conn) as conn: | 06:05 |
| *** kota_ has quit IRC | 06:07 | |
| *** kopparam has joined #openstack-swift | 06:08 | |
| *** addnull has quit IRC | 06:08 | |
| *** Masahiro has quit IRC | 06:09 | |
| *** Masahiro has joined #openstack-swift | 06:10 | |
| *** kopparam has quit IRC | 06:11 | |
| *** kopparam has joined #openstack-swift | 06:11 | |
| *** k4n0 has joined #openstack-swift | 06:20 | |
| madhuri | Yes. Even after closing the connection explicitly, the connection object is not destroyed because of cyclic reference from _send_file for an example | 06:23 |
| madhuri | with contextlib.closing(conn) as conn should work | 06:24 |
| *** wer has quit IRC | 06:28 | |
| *** wer has joined #openstack-swift | 06:28 | |
| mattoliverau | madhuri: http://paste.openstack.org/show/143922/ | 06:33 |
| mattoliverau | in quick testing that works.. though I did that in master rather then icehouse. | 06:33 |
| mattoliverau | well looks like it works.. I didn't really test too hard. | 06:34 |
| madhuri | Thanks mattoliverau. I am also testing it | 06:35 |
| openstackgerrit | Daisuke Morita proposed openstack/swift: Output logs of policy index https://review.openstack.org/136995 | 06:35 |
| *** gyee_ has quit IRC | 06:36 | |
| mattoliverau | madhuri: cools, well happy hacking :) I'm calling it a day. | 06:37 |
| mattoliverau | night swift land. | 06:38 |
| madhuri | Thanks mattoliverau :) | 06:38 |
| *** nosnos has quit IRC | 06:38 | |
| *** nosnos has joined #openstack-swift | 06:40 | |
| *** wer has quit IRC | 06:41 | |
| *** wer has joined #openstack-swift | 06:43 | |
| *** sandywalsh_ has joined #openstack-swift | 06:46 | |
| *** addnull has joined #openstack-swift | 06:47 | |
| *** sandywalsh has quit IRC | 06:47 | |
| *** SkyRocknRoll has quit IRC | 06:52 | |
| *** SkyRocknRoll has joined #openstack-swift | 06:53 | |
| *** nshaikh has joined #openstack-swift | 06:56 | |
| *** kopparam has quit IRC | 06:56 | |
| *** kopparam has joined #openstack-swift | 06:57 | |
| *** rmcall_ has joined #openstack-swift | 07:05 | |
| *** rmcall has quit IRC | 07:08 | |
| *** rmcall_ is now known as rmcall | 07:08 | |
| *** rmcall has quit IRC | 07:19 | |
| *** bill_az has quit IRC | 07:20 | |
| *** Masahiro has quit IRC | 07:27 | |
| *** Masahiro has joined #openstack-swift | 07:31 | |
| *** nellysmitt has joined #openstack-swift | 07:34 | |
| *** nellysmitt has quit IRC | 07:39 | |
| *** openstackgerrit has quit IRC | 07:50 | |
| *** openstackgerrit has joined #openstack-swift | 07:50 | |
| *** Masahiro has quit IRC | 07:59 | |
| *** Masahiro has joined #openstack-swift | 08:02 | |
| *** rledisez has joined #openstack-swift | 08:09 | |
| *** geaaru has joined #openstack-swift | 08:45 | |
| *** ahonda has joined #openstack-swift | 08:48 | |
| *** jordanP has joined #openstack-swift | 08:53 | |
| *** sgowda has joined #openstack-swift | 09:03 | |
| *** nellysmitt has joined #openstack-swift | 09:23 | |
| *** kopparam has quit IRC | 09:35 | |
| *** jordanP has quit IRC | 09:39 | |
| *** addnull has quit IRC | 09:44 | |
| *** jordanP has joined #openstack-swift | 09:53 | |
| *** jordanP has quit IRC | 09:59 | |
| *** tellesnobrega_ has joined #openstack-swift | 10:18 | |
| *** Masahiro has quit IRC | 10:29 | |
| *** kopparam has joined #openstack-swift | 10:30 | |
| *** jistr has joined #openstack-swift | 10:37 | |
| *** jistr is now known as jistr|trng | 10:38 | |
| *** erlon has quit IRC | 10:43 | |
| *** erlon has joined #openstack-swift | 10:48 | |
| *** addnull has joined #openstack-swift | 10:55 | |
| *** haomaiwang has quit IRC | 10:57 | |
| *** addnull has quit IRC | 10:59 | |
| *** tellesnobrega_ has quit IRC | 11:02 | |
| openstackgerrit | Merged openstack/swift: Removing unused method: _remaining_items https://review.openstack.org/138559 | 11:14 |
| *** aix has joined #openstack-swift | 11:22 | |
| *** k4n0 has quit IRC | 11:23 | |
| *** sgowda has quit IRC | 11:28 | |
| *** Masahiro has joined #openstack-swift | 11:30 | |
| *** nshaikh has quit IRC | 11:31 | |
| *** mahatic has joined #openstack-swift | 11:34 | |
| *** Masahiro has quit IRC | 11:34 | |
| *** CaioBrentano has joined #openstack-swift | 12:03 | |
| *** kopparam has quit IRC | 12:06 | |
| *** kopparam has joined #openstack-swift | 12:09 | |
| *** addnull has joined #openstack-swift | 12:23 | |
| openstackgerrit | Daisuke Morita proposed openstack/swift: Show each policy's information on audited results in recon https://review.openstack.org/138697 | 12:25 |
| *** kopparam has quit IRC | 12:27 | |
| *** addnull has quit IRC | 12:28 | |
| *** jistr|trng has quit IRC | 12:28 | |
| *** dmorita has quit IRC | 12:31 | |
| *** lpabon has joined #openstack-swift | 12:33 | |
| *** dmsimard_away is now known as dmsimard | 12:34 | |
| *** jistr has joined #openstack-swift | 12:34 | |
| *** jistr is now known as jistr|trng | 12:35 | |
| *** aix has quit IRC | 12:38 | |
| CaioBrentano | hi guys | 13:06 |
| CaioBrentano | does anyone can help me with some issue with swiftclient cli > | 13:06 |
| CaioBrentano | ? | 13:06 |
| CaioBrentano | I have a non-admin user that have write permission on a specific container… When I try to create an object in this container, the swift client is returning "Account not found" | 13:07 |
| *** Masahiro has joined #openstack-swift | 13:08 | |
| CaioBrentano | This happens on v2.3.1… On v2.2.0, swficlient returns a error about trying to create container, but creates the object successfully | 13:11 |
| *** exploreshaifali has joined #openstack-swift | 13:12 | |
| *** kopparam has joined #openstack-swift | 13:12 | |
| *** Masahiro has quit IRC | 13:13 | |
| *** ppai has quit IRC | 13:13 | |
| *** ppai has joined #openstack-swift | 13:15 | |
| *** nellysmitt has quit IRC | 13:17 | |
| *** aix has joined #openstack-swift | 13:19 | |
| *** kopparam has quit IRC | 13:33 | |
| *** kopparam has joined #openstack-swift | 13:36 | |
| *** oomichi has quit IRC | 13:40 | |
| *** X019 has joined #openstack-swift | 13:42 | |
| *** rdaly2 has joined #openstack-swift | 13:46 | |
| *** nellysmitt has joined #openstack-swift | 13:49 | |
| acoles | CaioBrentano: sounds like a known bug https://bugs.launchpad.net/python-swiftclient/+bug/1371650 | 13:54 |
| acoles | CaioBrentano: I have a fix up for review https://review.openstack.org/#/c/125759/ | 13:54 |
| *** ppai has quit IRC | 13:56 | |
| CaioBrentano | thanks acoles! | 13:56 |
| CaioBrentano | I'll check your review | 13:57 |
| acoles | CaioBrentano: btw the error re trying to create a container can be ignored - swift cli attempts to create the container before putting the object which will fail if the user does not have permission to create a container, but the cli should then proceed to attempt the object PUT | 13:57 |
| acoles | CaioBrentano: ^^ on 2.2.0 that is | 13:57 |
| *** exploreshaifali has quit IRC | 13:57 | |
| *** rdaly2 has quit IRC | 13:58 | |
| CaioBrentano | acoles: exactly… on 2.2.0 | 13:58 |
| CaioBrentano | acoles: anyway, the return status is error on "$?"… so my buid process breaks | 13:59 |
| *** tdasilva has joined #openstack-swift | 13:59 | |
| CaioBrentano | acoles: that's why I can't simply ignore this on v2.2.0 | 13:59 |
| acoles | CaioBrentano: yeah, i think my patch fixes that too, it really should not cause a 1 exit status unless the object PUT fails :( | 14:00 |
| CaioBrentano | acoles: great! | 14:02 |
| *** jordanP has joined #openstack-swift | 14:04 | |
| *** tdasilva has quit IRC | 14:15 | |
| *** annegent_ has joined #openstack-swift | 14:19 | |
| *** X019 has quit IRC | 14:20 | |
| *** kopparam has quit IRC | 14:27 | |
| *** nosnos has quit IRC | 14:28 | |
| *** tdasilva has joined #openstack-swift | 14:30 | |
| *** bill_az has joined #openstack-swift | 14:32 | |
| *** kopparam has joined #openstack-swift | 14:44 | |
| *** annegent_ has quit IRC | 14:54 | |
| *** silor has joined #openstack-swift | 14:54 | |
| *** Masahiro has joined #openstack-swift | 14:57 | |
| *** CaioBrentano has quit IRC | 15:00 | |
| *** Masahiro has quit IRC | 15:02 | |
| *** tdasilva has quit IRC | 15:04 | |
| *** tellesnobrega_ has joined #openstack-swift | 15:12 | |
| *** tdasilva has joined #openstack-swift | 15:17 | |
| *** madhuri has quit IRC | 15:23 | |
| *** kopparam has quit IRC | 15:34 | |
| *** kopparam has joined #openstack-swift | 15:39 | |
| *** kopparam has quit IRC | 15:51 | |
| *** rmcall has joined #openstack-swift | 15:53 | |
| *** CaioBrentano has joined #openstack-swift | 16:00 | |
| *** SkyRocknRoll has quit IRC | 16:03 | |
| openstackgerrit | Donagh McCabe proposed openstack/swift-specs: Minor updates to composite token spec https://review.openstack.org/138771 | 16:03 |
| notmyname | good morning | 16:04 |
| *** zaitcev has joined #openstack-swift | 16:05 | |
| *** ChanServ sets mode: +v zaitcev | 16:05 | |
| notmyname | reminder that we do not have a team meeting today | 16:13 |
| zaitcev | ok | 16:13 |
| tdasilva | notmyname: ok, thanks for the heads up | 16:14 |
| notmyname | we need to talk about the next release, though | 16:16 |
| notmyname | releasing 2.2.1 with the various bug fixes we've merged since 2.2.0 | 16:17 |
| notmyname | I added a 2.2.1 section to https://wiki.openstack.org/wiki/Swift/PriorityReviews | 16:17 |
| notmyname | and generally targeting the week of the 15th for a release | 16:18 |
| notmyname | that's actually about all the detail I have right now. I think it should be a pretty minor event. good stuff has landed. let's make it available. no big coordination effort needed for this one | 16:19 |
| openstackgerrit | Donagh McCabe proposed openstack/swift: Add multiple reseller prefixes and composite tokens https://review.openstack.org/137086 | 16:19 |
| notmyname | if someone thinks any of that is a terrible idea, let me know :-) | 16:19 |
| * notmyname hopes mattoliverau saw my note in channel yesterday about no meeting | 16:20 | |
| acoles | notmyname: ack | 16:26 |
| notmyname | I've got to drive to a meeting soon. I'll be online later today | 16:27 |
| *** tellesnobrega_ has quit IRC | 16:28 | |
| *** rdaly2 has joined #openstack-swift | 16:28 | |
| *** Nadeem has joined #openstack-swift | 16:28 | |
| *** echevemaster has joined #openstack-swift | 16:29 | |
| *** Nadeem has quit IRC | 16:29 | |
| tdasilva | notmyname: ok, planning to go through the priority reviews today and see where i can help | 16:30 |
| *** annegent_ has joined #openstack-swift | 16:43 | |
| *** Masahiro has joined #openstack-swift | 16:46 | |
| *** Masahiro has quit IRC | 16:50 | |
| *** annegent_ has quit IRC | 16:55 | |
| bsdkurt | I'm running a small cluster and one object-server is acting up. I'm frequently getting ChunkWriteTimeout (60s) on the proxy and http 408's after 60 seconds on the one storage node. | 17:00 |
| bsdkurt | Does anyone have pointers for me to figure out what's going wrong? | 17:00 |
| bsdkurt | oh, I'm running 2.2.0 | 17:00 |
| *** jistr|trng has quit IRC | 17:11 | |
| *** gyee_ has joined #openstack-swift | 17:12 | |
| *** annegent_ has joined #openstack-swift | 17:15 | |
| *** zackmdavis has joined #openstack-swift | 17:25 | |
| *** atan8 has joined #openstack-swift | 17:26 | |
| *** rledisez has quit IRC | 17:28 | |
| *** rmcall has quit IRC | 17:39 | |
| clayg | acoles: yeah I think now that capture output merged i was supposed to rebase something... | 17:44 |
| *** rmcall has joined #openstack-swift | 17:44 | |
| acoles | clayg: say more ? | 17:45 |
| *** nshaikh has joined #openstack-swift | 17:46 | |
| *** aix has quit IRC | 17:48 | |
| clayg | you had linked earlier the os-storage-url branch, which I think was maybe back on the review ticket with capture output merged | 17:50 |
| clayg | i know i had some other outstanding swift client change (something about auth retries?) that needed to be rebased | 17:50 |
| acoles | clayg: ah, right, with you now :) yeah that was dependent on capture output. idk if i need to rebase or if its good still | 17:52 |
| gyee_ | question for Swift team, once the account is marked as deleted and before the reaper takes it, can it be reversed? | 17:53 |
| acoles | clayg: btw i asked dan to look at applying capture output uniformly across the client tests (if its not already) | 17:53 |
| *** annegent_ has quit IRC | 17:56 | |
| clayg | acoles: that's cool | 17:58 |
| clayg | acoles: how'd you manage to cram enough bits into your Timewrap that you're not gunna loose info? | 17:58 |
| *** gvernik has joined #openstack-swift | 17:59 | |
| clayg | it looked like you were doing a bit mask of the least significant bits - but i feel like that would fall apart if you had some updates that were 100's of hours apart | 17:59 |
| acoles | clayg: at the moment i'm only packing 2.5 timestamps in there: the original data time, the content-type time, and two digits for reconciler bumping the time | 18:00 |
| acoles | clayg: mask two rightmost digits, then shift content-type time into upper 14 digits | 18:01 |
| clayg | acoles: so two posts based on the same content-type at 0x1ff and 0x0ff look like the same timestamp to the container? now row_id bump? | 18:02 |
| *** jwang_ has joined #openstack-swift | 18:02 | |
| *** simonj has joined #openstack-swift | 18:02 | |
| bsdkurt | There were no clues as to why that object server was behaving badly for me; nothing in syslog. rebooting the server seems to have cleared the problem. very strange. | 18:04 |
| *** tkay has joined #openstack-swift | 18:05 | |
| *** tkay has left #openstack-swift | 18:05 | |
| clayg | bsdkurt: can you see if the timeouts were hitting the same disk? were you running with threads per disk option or port per disk setup? | 18:05 |
| *** rmcall has quit IRC | 18:06 | |
| clayg | do you have smartmontools installed on the node, can you run a quick -i on all the devices? | 18:06 |
| bsdkurt | clayg: it was different disks. I'm running with 8 threads per disk | 18:06 |
| bsdkurt | ah good idea, will do | 18:06 |
| *** silor has quit IRC | 18:07 | |
| clayg | bsdkurt: how many disks/workers with those 8 threads per disk? | 18:07 |
| *** rmcall has joined #openstack-swift | 18:07 | |
| bsdkurt | clayg: 16 workers (node has 16 cpus), 5 disks | 18:08 |
| clayg | k, so 8 threads * 5 disks * 16 workers is 640 threads - seems fine | 18:09 |
| *** rmcall_ has joined #openstack-swift | 18:11 | |
| *** rmcall has quit IRC | 18:12 | |
| *** rmcall_ is now known as rmcall | 18:12 | |
| clayg | torgomatic: bsdkurt got some chunkwritetimeouts with threads per disk ^ ??? | 18:12 |
| *** nellysmitt has quit IRC | 18:13 | |
| clayg | torgomatic: i'm trying to imagine something that could get screwed up besides just the obvious (fsyncs blocking like a boss) | 18:13 |
| *** jordanP has quit IRC | 18:13 | |
| clayg | torgomatic: I guess if we don't have a timeout on those you could get some hung fysncs eating up your disk threads, and then nothing gets done? | 18:14 |
| clayg | torgomatic: or maybe memory presure somehow casing a write to block trying to get space free'd in the page cache to accept the buffer? | 18:14 |
| bsdkurt | clayg: bbl, lunch is here. thanks for looking into it. I'll be back in 30min. | 18:15 |
| clayg | later | 18:15 |
| *** atan8 has quit IRC | 18:20 | |
| *** silor has joined #openstack-swift | 18:23 | |
| *** exploreshaifali has joined #openstack-swift | 18:26 | |
| *** aix has joined #openstack-swift | 18:33 | |
| *** simonj has quit IRC | 18:34 | |
| bsdkurt | clayg: hmm, the problem happened again. this time on a different storage server. same hardware setup: 5 disks, 16 workers, 8 threads per disk | 18:34 |
| *** Masahiro has joined #openstack-swift | 18:35 | |
| *** cutforth has joined #openstack-swift | 18:36 | |
| bsdkurt | btw, the http 408 makes its way out to the caller. if two of the three copies succeed, shouldn't it return success out? | 18:38 |
| *** nellysmitt has joined #openstack-swift | 18:39 | |
| *** Masahiro has quit IRC | 18:40 | |
| *** annegent_ has joined #openstack-swift | 18:40 | |
| *** annegent_ has quit IRC | 18:48 | |
| bsdkurt | the object-server was cruising along and then the problem tripped and about 9 out of 10 PUTs fail with 408 for several minutes. | 18:56 |
| *** annegent_ has joined #openstack-swift | 18:57 | |
| bsdkurt | if I stop adding data to the cluster for a while and start up again. seems like all is well for a few minutes then it trips again on the same server although it looks like my success rate is higher about 50/50 | 18:58 |
| *** acoles is now known as acoles_away | 18:58 | |
| peluse | clayg, I have a proto of what we talked about last wrt the reconstructor kinda working. There are of course a few options before continuing, would you have any time today to chat? | 18:58 |
| *** geaaru has quit IRC | 18:59 | |
| clayg | heh | 19:01 |
| clayg | i just got off the skype with acoles :P | 19:01 |
| bsdkurt | cpu use looks ok (~80%) idle, memory doesn't look too bad: 5.5gig in cached mem, no swaping, but the rest in use 16 gig total | 19:01 |
| clayg | peluse: i was about to head into the office, you wanna do it after lunch PST? | 19:01 |
| peluse | sure, ping me when ready. will see if I can get my skype working by then :)( | 19:02 |
| clayg | peluse: nice | 19:04 |
| clayg | bsdkurt: can you turn the object server log level to debug and maybe even eventlet_debug = true? | 19:04 |
| clayg | bsdkurt: also check your messages/syslog for network or disk related stuff - "con" "write" "blk" "err" "fail" "drop" anything like that... | 19:05 |
| clayg | bsdkurt: it's weird that it moved servers, makes it much less likly to be disk related in mind... but maybe if your requests per disk is overwhelming the threads * worker per disk things could get starved and manifest as timeouts - are you sending lots of requests or lots of bytes? | 19:06 |
| clayg | peluse: k, i'll ping ya - later | 19:07 |
| bsdkurt | yea. the smartmon short test was good on the first server.. the syslog does have something in it... just a sec | 19:07 |
| bsdkurt | looks like kernel is complaining about memory allocation failures | 19:09 |
| *** cutforth has quit IRC | 19:10 | |
| bsdkurt | stack traces look like networking buffers are an issue | 19:10 |
| bsdkurt | all the kernel allocation errors are in the network stack. I have 10gig nics in these servers. | 19:13 |
| *** cloudnull has left #openstack-swift | 19:15 | |
| *** gvernik has quit IRC | 19:15 | |
| bsdkurt | ahh yea, the first server also has these. I miss them when I looked (log rotation) | 19:15 |
| bsdkurt | I'm running ubuntu 14.04.1. | 19:16 |
| bsdkurt | google tells me that increasing vm.min_free_kbytes may fix this. I'll give it go and see | 19:27 |
| bsdkurt | clayg: thanks for the help. sorry I missed those kernel messages when I first looked. It seems that after the messages appear, the server is hosed until a reboot and the messages don't reoccur. | 19:34 |
| *** mahatic has quit IRC | 19:38 | |
| *** imkarrer has quit IRC | 19:50 | |
| *** aix has quit IRC | 19:51 | |
| openstackgerrit | Caleb Tennis proposed openstack/swift: Catch OSError in replicator if directory is gone. https://review.openstack.org/138837 | 19:51 |
| *** aix has joined #openstack-swift | 20:05 | |
| *** nellysmitt has quit IRC | 20:07 | |
| *** nshaikh has left #openstack-swift | 20:09 | |
| *** annegent_ has quit IRC | 20:09 | |
| *** annegent_ has joined #openstack-swift | 20:13 | |
| *** Masahiro has joined #openstack-swift | 20:23 | |
| *** nellysmitt has joined #openstack-swift | 20:26 | |
| *** nellysmitt has quit IRC | 20:28 | |
| *** Masahiro has quit IRC | 20:28 | |
| *** annegent_ has quit IRC | 20:32 | |
| *** CaioBrentano has quit IRC | 20:40 | |
| *** CaioBrentano has joined #openstack-swift | 20:43 | |
| *** CaioBrentano has quit IRC | 20:57 | |
| *** silor has quit IRC | 20:59 | |
| *** silor1 has joined #openstack-swift | 20:59 | |
| mattoliverau | Morning all | 21:01 |
| mattoliverau | notmyname thanks for the meeting reminder but lucky I tend to read channel scroll back so I heard about no meeting yest ;) the swift meeting agenda also had the date for next week which was a give away! | 21:01 |
| *** silor1 has quit IRC | 21:15 | |
| *** exploreshaifali has quit IRC | 21:23 | |
| *** lpabon has quit IRC | 21:24 | |
| *** dmsimard is now known as dmsimard_away | 21:32 | |
| clayg | poor mattoliverau :\ | 21:39 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-swiftclient: Updated from global requirements https://review.openstack.org/89250 | 21:41 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift: Updated from global requirements https://review.openstack.org/88736 | 21:42 |
| peluse | clayg, say "when" | 21:45 |
| clayg | 15 mins i'll say when | 21:45 |
| peluse | that works... I assume you got the movie quote right? https://www.youtube.com/watch?v=3G2_KZ7_hxc | 21:48 |
| mattoliverau | clayg: yeah it's hard being me :P Everyone take pity! | 21:52 |
| clayg | peluse: when i 'spose | 22:01 |
| clayg | peluse: hrmm... i don't have headphones handy... | 22:02 |
| peluse | OK, tried skype - what's a better way? | 22:02 |
| *** tellesnobrega_ has joined #openstack-swift | 22:05 | |
| mattoliverau | peluse: you guys could try google hangouts, seeing as you don't have to worry about a client. | 22:11 |
| *** Masahiro has joined #openstack-swift | 22:12 | |
| notmyname | back online. hello world | 22:15 |
| *** Masahiro has quit IRC | 22:17 | |
| mattoliverau | notmyname: yo | 22:17 |
| notmyname | yo | 22:18 |
| *** nellysmitt has joined #openstack-swift | 22:28 | |
| openstackgerrit | paul luse proposed openstack/swift: Reconstructor helpers for handoff revert case (and other misc) https://review.openstack.org/131872 | 22:30 |
| *** nellysmitt has quit IRC | 22:33 | |
| *** jwang_ has quit IRC | 22:40 | |
| *** jwang_ has joined #openstack-swift | 23:02 | |
| *** sungju has joined #openstack-swift | 23:30 | |
| *** tdasilva has quit IRC | 23:38 | |
| *** dmsimard_away is now known as dmsimard | 23:51 | |
| *** jwang__ has joined #openstack-swift | 23:55 | |
| *** jwang_ has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!