| *** mkollaro has quit IRC | 00:08 | |
| *** StevenK_ is now known as StevenK | 00:25 | |
| *** StevenK is now known as Guest99695 | 00:26 | |
| *** joeljwright has joined #openstack-swift | 00:26 | |
| *** Guest99695 is now known as StevenK | 00:26 | |
| *** joeljwright has quit IRC | 00:31 | |
| *** matsuhashi has joined #openstack-swift | 00:31 | |
| *** Midnightmyth has quit IRC | 00:38 | |
| *** mmcardle has joined #openstack-swift | 00:44 | |
| *** matsuhas_ has joined #openstack-swift | 00:48 | |
| *** shri1 has joined #openstack-swift | 00:48 | |
| *** mmcardle has quit IRC | 00:48 | |
| *** minnear_ has joined #openstack-swift | 00:49 | |
| *** ahale has joined #openstack-swift | 00:50 | |
| *** yuan has joined #openstack-swift | 00:50 | |
| *** dmorita has joined #openstack-swift | 00:50 | |
| *** notmyname_ has joined #openstack-swift | 00:50 | |
| *** mlanner_ has joined #openstack-swift | 00:50 | |
| *** alpha_ori_ has joined #openstack-swift | 00:51 | |
| *** matsuhashi has quit IRC | 00:55 | |
| *** yuanz has quit IRC | 00:55 | |
| *** shri has quit IRC | 00:55 | |
| *** minnear has quit IRC | 00:55 | |
| *** alpha_ori has quit IRC | 00:55 | |
| *** ahale_ has quit IRC | 00:55 | |
| *** pconstantine has quit IRC | 00:55 | |
| *** mlanner has quit IRC | 00:55 | |
| *** notmyname has quit IRC | 00:55 | |
| *** mlanner_ is now known as mlanner | 00:55 | |
| *** notmyname_ is now known as notmyname | 00:55 | |
| *** alpha_ori_ is now known as alpha_ori | 00:55 | |
| *** ChanServ sets mode: +v dfg | 00:56 | |
| *** pconstantine has joined #openstack-swift | 01:02 | |
| *** yuanz has joined #openstack-swift | 01:03 | |
| *** yuan has quit IRC | 01:06 | |
| *** ChanServ sets mode: +v notmyname | 01:06 | |
| *** yuan has joined #openstack-swift | 01:07 | |
| *** yuanz has quit IRC | 01:08 | |
| *** openstackgerrit has quit IRC | 01:18 | |
| *** openstackgerrit has joined #openstack-swift | 01:18 | |
| *** mmcardle has joined #openstack-swift | 01:20 | |
| *** mmcardle has quit IRC | 01:24 | |
| *** joeljwright has joined #openstack-swift | 01:27 | |
| *** joeljwright has quit IRC | 01:31 | |
| *** ryao has quit IRC | 01:33 | |
| *** fifieldt has joined #openstack-swift | 01:33 | |
| *** nosnos has joined #openstack-swift | 01:34 | |
| *** ryao has joined #openstack-swift | 01:34 | |
| openstackgerrit | A change was merged to openstack/swift: Fix misspellings in swift https://review.openstack.org/71800 | 01:36 |
|---|---|---|
| luisbg | nice to see merges :) | 01:40 |
| *** shri1 has quit IRC | 01:43 | |
| *** haomaiwa_ has quit IRC | 01:55 | |
| *** haomaiwang has joined #openstack-swift | 02:08 | |
| *** yuanz has joined #openstack-swift | 02:13 | |
| *** yuan has quit IRC | 02:19 | |
| clayg | portante: nope | 02:19 |
| *** mmcardle has joined #openstack-swift | 02:21 | |
| openstackgerrit | Matthew Kassawara proposed a change to openstack/swift: Alphabetized items under "Misc" in source documentation https://review.openstack.org/75282 | 02:24 |
| *** mmcardle has quit IRC | 02:26 | |
| *** joeljwright has joined #openstack-swift | 02:28 | |
| *** keving has joined #openstack-swift | 02:32 | |
| *** joeljwright has quit IRC | 02:33 | |
| *** yuanz has quit IRC | 02:33 | |
| *** matsuhas_ has quit IRC | 02:34 | |
| *** yuanz has joined #openstack-swift | 02:34 | |
| *** matsuhas_ has joined #openstack-swift | 02:36 | |
| *** mmcardle has joined #openstack-swift | 02:43 | |
| *** mmcardle has quit IRC | 02:48 | |
| *** matsuhas_ has quit IRC | 03:03 | |
| *** matsuhas_ has joined #openstack-swift | 03:06 | |
| openstackgerrit | Matthew Kassawara proposed a change to openstack/swift: Alphabetized items under "Middleware" in source documentation https://review.openstack.org/75294 | 03:29 |
| *** joeljwright has joined #openstack-swift | 03:29 | |
| *** matsuhas_ has quit IRC | 03:29 | |
| *** joeljwright has quit IRC | 03:33 | |
| *** justIn_ has joined #openstack-swift | 03:34 | |
| *** gyee has quit IRC | 03:40 | |
| *** mmcardle has joined #openstack-swift | 03:45 | |
| *** mmcardle has quit IRC | 03:49 | |
| *** keving has quit IRC | 03:53 | |
| *** keving has joined #openstack-swift | 03:53 | |
| *** madhuri has joined #openstack-swift | 04:01 | |
| *** matsuhashi has joined #openstack-swift | 04:25 | |
| *** joeljwright has joined #openstack-swift | 04:31 | |
| *** joeljwright has quit IRC | 04:35 | |
| *** ppai has joined #openstack-swift | 04:39 | |
| *** mmcardle has joined #openstack-swift | 04:46 | |
| *** mmcardle has quit IRC | 04:50 | |
| openstackgerrit | A change was merged to openstack/swift: Support If-[None-]Match for object HEAD, SLO, and DLO https://review.openstack.org/73162 | 05:02 |
| openstackgerrit | Samuel Merritt proposed a change to openstack/swift: Raise error on long or short DLO https://review.openstack.org/75311 | 05:04 |
| *** madhuri has quit IRC | 05:05 | |
| *** madhuri has joined #openstack-swift | 05:16 | |
| *** zaitcev has quit IRC | 05:19 | |
| *** redbo_ is now known as redbo | 05:23 | |
| *** ChanServ sets mode: +v redbo | 05:23 | |
| *** matsuhashi has quit IRC | 05:23 | |
| *** joeljwright has joined #openstack-swift | 05:32 | |
| *** matsuhashi has joined #openstack-swift | 05:33 | |
| *** joeljwright has quit IRC | 05:37 | |
| *** tdasilva has quit IRC | 05:39 | |
| *** mmcardle has joined #openstack-swift | 05:47 | |
| *** mmcardle has quit IRC | 05:52 | |
| *** sasi has joined #openstack-swift | 06:09 | |
| sasi | Hi All, I am trying out Multi-node swift (RHEL) installation and facing error (503) while uploading an object or container. Can any one in the forum have bit time to help me | 06:12 |
| *** cschwede_ has joined #openstack-swift | 06:20 | |
| *** cschwede_ has quit IRC | 06:24 | |
| *** joeljwright has joined #openstack-swift | 06:32 | |
| *** saju_m has joined #openstack-swift | 06:36 | |
| *** joeljwright has quit IRC | 06:37 | |
| *** mmcardle has joined #openstack-swift | 06:48 | |
| *** mmcardle has quit IRC | 06:52 | |
| *** anju_ has joined #openstack-swift | 06:52 | |
| *** cschwede_ has joined #openstack-swift | 07:04 | |
| openstackgerrit | Samuel Merritt proposed a change to openstack/swift: Functional tests for DLO If-Match/If-None-Match https://review.openstack.org/75327 | 07:13 |
| *** basha has joined #openstack-swift | 07:13 | |
| openstackgerrit | Samuel Merritt proposed a change to openstack/swift: Functional tests for SLO If-Match/If-None-Match https://review.openstack.org/75328 | 07:13 |
| *** foexle has joined #openstack-swift | 07:17 | |
| *** justIn_ has quit IRC | 07:21 | |
| *** basha has quit IRC | 07:28 | |
| *** joeljwright has joined #openstack-swift | 07:33 | |
| *** joeljwright has quit IRC | 07:38 | |
| *** mmcardle has joined #openstack-swift | 07:49 | |
| *** mmcardle has quit IRC | 07:54 | |
| *** cschwede_ has quit IRC | 07:58 | |
| *** basha has joined #openstack-swift | 08:02 | |
| *** sasi has quit IRC | 08:11 | |
| *** mmcardle has joined #openstack-swift | 08:13 | |
| *** cschwede_ has joined #openstack-swift | 08:20 | |
| *** saju_m has quit IRC | 08:31 | |
| *** cschwede_ has left #openstack-swift | 08:32 | |
| *** joeljwright has joined #openstack-swift | 08:34 | |
| *** joeljwright has quit IRC | 08:39 | |
| *** nacim has joined #openstack-swift | 08:50 | |
| *** joeljwright has joined #openstack-swift | 08:58 | |
| *** saju_m has joined #openstack-swift | 09:01 | |
| *** joeljwright has quit IRC | 09:01 | |
| *** saju_m has quit IRC | 09:02 | |
| *** fbo_away is now known as fbo | 09:09 | |
| *** otoolee has left #openstack-swift | 09:09 | |
| *** bvandenh has quit IRC | 09:19 | |
| *** basha has quit IRC | 09:19 | |
| *** saju_m has joined #openstack-swift | 09:21 | |
| *** mkollaro has joined #openstack-swift | 09:21 | |
| *** dmorita has quit IRC | 09:23 | |
| *** chandan_kumar has joined #openstack-swift | 09:25 | |
| *** bvandenh has joined #openstack-swift | 09:27 | |
| *** chandan_kumar has quit IRC | 09:51 | |
| *** chandan_kumar has joined #openstack-swift | 10:05 | |
| *** chandan_kumar has quit IRC | 10:28 | |
| *** [BNC]otoolee has joined #openstack-swift | 10:41 | |
| *** mkollaro has quit IRC | 10:42 | |
| *** [BNC]otoolee has left #openstack-swift | 10:44 | |
| *** [BNC]otoolee has joined #openstack-swift | 10:47 | |
| *** mkollaro has joined #openstack-swift | 10:51 | |
| *** [BNC]otoolee has left #openstack-swift | 10:51 | |
| *** nosnos has quit IRC | 10:58 | |
| *** gdrudy has joined #openstack-swift | 11:00 | |
| *** matsuhashi has quit IRC | 11:02 | |
| *** gdrudy1 has joined #openstack-swift | 11:12 | |
| *** gdrudy1 has left #openstack-swift | 11:13 | |
| *** [BNC]otoolee has joined #openstack-swift | 11:31 | |
| *** [BNC]otoolee has left #openstack-swift | 11:33 | |
| *** [BNC]otoolee has joined #openstack-swift | 11:35 | |
| *** Trixboxer has joined #openstack-swift | 11:42 | |
| *** [BNC]otoolee has left #openstack-swift | 11:42 | |
| *** [BNC]otoolee has joined #openstack-swift | 11:44 | |
| *** [BNC]otoolee has left #openstack-swift | 11:44 | |
| *** Midnightmyth has joined #openstack-swift | 11:46 | |
| *** [BNC]otoolee has joined #openstack-swift | 11:47 | |
| *** [BNC]otoolee has left #openstack-swift | 11:48 | |
| *** [BNC]otoolee has joined #openstack-swift | 12:01 | |
| *** [BNC]otoolee has left #openstack-swift | 12:02 | |
| *** [BNC]otoolee has joined #openstack-swift | 12:38 | |
| *** [BNC]otoolee has left #openstack-swift | 12:38 | |
| *** [BNC]otoolee has joined #openstack-swift | 12:48 | |
| *** [BNC]otoolee has left #openstack-swift | 12:48 | |
| *** ppai has quit IRC | 12:53 | |
| *** [BNC]otoolee has joined #openstack-swift | 12:56 | |
| *** zul has quit IRC | 13:09 | |
| *** zul has joined #openstack-swift | 13:09 | |
| *** [BNC]otoolee has left #openstack-swift | 13:39 | |
| *** tongli has joined #openstack-swift | 13:40 | |
| *** russellb is now known as rustlebee | 13:44 | |
| *** otoolee has joined #openstack-swift | 13:52 | |
| *** saju_m has quit IRC | 14:10 | |
| *** otoolee has left #openstack-swift | 14:18 | |
| *** zul has quit IRC | 14:20 | |
| *** zul has joined #openstack-swift | 14:21 | |
| *** judd7_ has quit IRC | 14:34 | |
| *** otoolee has joined #openstack-swift | 14:34 | |
| *** judd7 has joined #openstack-swift | 14:37 | |
| *** mmcardle has quit IRC | 14:39 | |
| *** otoolee has left #openstack-swift | 14:44 | |
| *** otoolee has joined #openstack-swift | 14:47 | |
| *** otoolee has quit IRC | 14:47 | |
| *** otoolee has joined #openstack-swift | 14:49 | |
| *** mmcardle has joined #openstack-swift | 14:51 | |
| *** dmsimard has joined #openstack-swift | 14:54 | |
| *** mhu has joined #openstack-swift | 14:57 | |
| *** otoolee has quit IRC | 14:59 | |
| mhu | Hello, is it possible to revoke a scheduled expiry on an object once it has been set with X-Delete-xxx ? Provided the object is still there of course | 15:13 |
| *** mlipchuk has joined #openstack-swift | 15:18 | |
| *** mlipchuk has quit IRC | 15:19 | |
| *** nacim has quit IRC | 15:20 | |
| *** nacim has joined #openstack-swift | 15:32 | |
| *** minnear_ is now known as minnear | 15:35 | |
| *** gholt has quit IRC | 15:37 | |
| *** SkyRocknRoll__ has quit IRC | 15:37 | |
| *** marcusvrn has quit IRC | 15:37 | |
| *** gholt has joined #openstack-swift | 15:38 | |
| *** ChanServ sets mode: +v gholt | 15:38 | |
| *** erlon has quit IRC | 15:42 | |
| *** lnxnut has joined #openstack-swift | 15:43 | |
| *** Midnightmyth has quit IRC | 16:01 | |
| *** otoolee has joined #openstack-swift | 16:07 | |
| *** therve_ has joined #openstack-swift | 16:08 | |
| *** bsdkurt has quit IRC | 16:10 | |
| *** bsdkurt has joined #openstack-swift | 16:13 | |
| *** jairo has joined #openstack-swift | 16:19 | |
| jairo | guys I need some guidance here | 16:21 |
| jairo | I am getting a bit worry, since I upgraded to the latest release of swift my nodes are going nuts doing this: http://paste.openstack.org/show/68047/ | 16:21 |
| creiht | jairo: well the first thing you can do is stop the object-auditor | 16:23 |
| creiht | jairo: what version were you upgrading from? | 16:23 |
| jairo | I was on 1.10 upgraded to 1.12 | 16:24 |
| creiht | jairo: did you verify that your swift_hash_path_* didn't change in /etc/swift/swift.conf ? | 16:28 |
| jairo | creiht: I did not change that, it is like: swift_hash_path_suffix = ba3xxxxxxxx | 16:30 |
| jairo | creiht: and it is in the same location as it was /etc/swift/swift.conf | 16:30 |
| creiht | ok if that got accidently changed, that could cause the issue you are seeing | 16:33 |
| notmyname | jairo: can you paste the output of `swift-object-info` from one of those objects? (scrub anything sensitive, of course) | 16:34 |
| creiht | yeah | 16:34 |
| jairo | okay | 16:36 |
| jairo | here it is http://paste.openstack.org/show/68051/ | 16:43 |
| *** therve_ has quit IRC | 16:45 | |
| *** gdrudy has left #openstack-swift | 16:46 | |
| creiht | jairo: ok, what path is that object currently on? | 16:47 |
| jairo | I executed this: swift-object-info /srv/node/sdc1/objects/99679/dad/4073e17bed536685ad7ff87714711dad/1391859167.53165.data | 16:48 |
| creiht | ok | 16:48 |
| creiht | jairo: have you changed rings recently? | 16:48 |
| jairo | no I have not | 16:49 |
| creiht | the location of the object is in a different partition than what the ring is saying it should be | 16:49 |
| creiht | 99679 vs 65999 | 16:49 |
| creiht | and that's why the auditor is trying to quarantine them all | 16:50 |
| creiht | jairo: can you validate that all of your servers have the same ring? for example running md5sum on the /etc/swift/object.ring.gz on each server | 16:51 |
| jairo | creiht: yes I will check that | 16:52 |
| *** otoolee has left #openstack-swift | 16:53 | |
| *** otoolee has joined #openstack-swift | 16:53 | |
| *** otoolee has left #openstack-swift | 16:55 | |
| *** otoolee has joined #openstack-swift | 16:55 | |
| jairo | I checked they all match | 16:56 |
| *** basha has joined #openstack-swift | 16:57 | |
| creiht | hrm | 16:57 |
| openstackgerrit | paul luse proposed a change to openstack/swift: Add Storage Policy Support to ssync https://review.openstack.org/65347 | 16:58 |
| jairo | actually after I upgraded to 1.10 I started seeing this: http://paste.openstack.org/show/68062/ popping up slowly on the log files | 16:59 |
| creiht | jairo: it sounds like at some point either the rings got accidently rebuilt from scratch rather than with the original build files | 17:00 |
| creiht | or the hash secrets changed | 17:01 |
| creiht | jairo: do you have both a swift_hash_path_prefix and swift_hash_path_suffix in /etc/swift/swift.conf? | 17:02 |
| jairo | no I just have swift_hash_path_suffix | 17:03 |
| jairo | I don't have swift_hash_path_prefix | 17:03 |
| creiht | ok that should be ok | 17:06 |
| creiht | notmyname: I'm out of ideas and gotta run... thoughts? | 17:07 |
| creiht | jairo: sorry I can't help much more at the moment | 17:07 |
| notmyname | nothing is springing to mind, immediately. | 17:07 |
| gholt | How does one get 99679 to match the first part of that hash 4073e17bed536685ad7ff87714711dad? | 17:07 |
| jairo | creiht: that's okay thanks for your help | 17:07 |
| notmyname | jairo: do you have no swift_hash_path prefix defined at all (ie not in swift.conf or commented out), or is it set to empty? | 17:08 |
| jairo | notmyname: prefix is not defined | 17:09 |
| notmyname | ok | 17:09 |
| *** nacim has quit IRC | 17:10 | |
| jairo | notmyname: this is how my swift.conf looks like http://paste.openstack.org/show/68063/ | 17:10 |
| notmyname | jairo: ok | 17:10 |
| notmyname | jairo: do you have any other programs besides swift accessing the storage volumes? | 17:11 |
| jairo | no | 17:11 |
| *** tdasilva has joined #openstack-swift | 17:13 | |
| *** basha has quit IRC | 17:13 | |
| jairo | notmyname: will the object-auditor fix it if I leave it running?.. | 17:14 |
| gholt | I really don't get how that hash could ever end up at that disk location indicating partition 99679. | 17:17 |
| jairo | oh wait.. one of the nodes have the wrong suffix | 17:17 |
| notmyname | jairo: good catch | 17:18 |
| creiht | jairo: there you go | 17:18 |
| jairo | so I will fix that now, after that the auditor will fix the incorrect, right? | 17:19 |
| notmyname | jairo: the auditor will move the local stuff out and the replicator will repair it from another copy in the cluster | 17:19 |
| gholt | Hmm, as an aside then, what am I missing? http://paste.openstack.org/show/68065/ | 17:19 |
| *** hurricanerix has joined #openstack-swift | 17:22 | |
| gholt | Ah well, must be some maths I'm missing. I'll move on. ;) | 17:23 |
| *** mkollaro has quit IRC | 17:24 | |
| notmyname | jairo: I'd like confirmation that you've still got access to the data. ie you can do a GET or HEAD and still access it | 17:26 |
| *** otoolee has left #openstack-swift | 17:29 | |
| *** tongli has quit IRC | 17:31 | |
| *** mkollaro has joined #openstack-swift | 17:32 | |
| jairo | yes I can still get data access | 17:35 |
| *** gyee has joined #openstack-swift | 17:38 | |
| *** dtalton has joined #openstack-swift | 17:39 | |
| *** dtalton has left #openstack-swift | 17:40 | |
| jairo | so I guess it is all good.. and the auditor will fix it.. you guys are awesome, I think you guys should have something like zookeeper, to make sure swift.conf and rings are identical | 17:40 |
| notmyname | jairo: swift-recon can do some of that. looks like there could be a few more sanity checks there. glad you're all set again :-) | 17:41 |
| jairo | notmyname: thanks again guys... | 17:42 |
| *** Midnightmyth has joined #openstack-swift | 17:45 | |
| notmyname | I've got a customer meeting to go to today. if you need me, leave a message in here or send an email please | 17:47 |
| *** keving has joined #openstack-swift | 17:50 | |
| *** csd has joined #openstack-swift | 17:52 | |
| *** bsdkurt has quit IRC | 17:52 | |
| *** bsdkurt has joined #openstack-swift | 17:53 | |
| *** vito-ordaz has joined #openstack-swift | 17:53 | |
| *** shri has joined #openstack-swift | 17:56 | |
| openstackgerrit | Matthew Kassawara proposed a change to openstack/swift: Alphabetized items under other sections in source documentation https://review.openstack.org/75475 | 18:02 |
| *** vito-ord_ has joined #openstack-swift | 18:06 | |
| *** vito-ordaz has quit IRC | 18:10 | |
| *** mmcardle has quit IRC | 18:49 | |
| *** foexle has quit IRC | 19:01 | |
| *** dmsimard1 has joined #openstack-swift | 19:09 | |
| *** jairo has quit IRC | 19:10 | |
| *** dmsimard has quit IRC | 19:12 | |
| *** vito-ord_ has quit IRC | 19:16 | |
| *** zaitcev has joined #openstack-swift | 19:17 | |
| *** ChanServ sets mode: +v zaitcev | 19:17 | |
| *** occupant has quit IRC | 19:21 | |
| zaitcev | Guys | 19:22 |
| zaitcev | I looked closer into the /info code and found that it uses its own authentication. | 19:22 |
| zaitcev | Why? | 19:22 |
| *** dmsimard1 is now known as dmsimard | 19:23 | |
| *** jairo has joined #openstack-swift | 19:26 | |
| *** jergerber has joined #openstack-swift | 19:26 | |
| clayg | zaitcev: as opposed to like reseller admin? | 19:33 |
| zaitcev | clayg: yes, that. Or if not reseller admin, maybe invent new role. | 19:34 |
| clayg | ... maybe invent a new role - in the case of multiple auth (reseller_prefix) it's not clear which auth system you'd ask to validate the request? | 19:36 |
| *** jairo has left #openstack-swift | 19:36 | |
| *** Trixboxer has quit IRC | 19:37 | |
| zaitcev | I do not understand that question. How does any controller code decided "which auth system [it] would ask"? It doesn't! It just invokes the swift.authorize hook that middleware left over, if any. | 19:39 |
| clayg | yeah, I could see it would be more flexible if the /info authorization had a hook back to middleware | 19:44 |
| *** dmsimard has quit IRC | 19:44 | |
| *** bada has quit IRC | 19:49 | |
| *** hurrican_ has joined #openstack-swift | 19:49 | |
| *** hurrican_ has quit IRC | 19:50 | |
| *** dmsimard has joined #openstack-swift | 19:50 | |
| *** bada has joined #openstack-swift | 19:50 | |
| *** hurrican_ has joined #openstack-swift | 19:51 | |
| *** dmsimard1 has joined #openstack-swift | 19:52 | |
| *** hurricanerix has quit IRC | 19:53 | |
| *** otoolee has joined #openstack-swift | 19:55 | |
| *** dmsimard has quit IRC | 19:55 | |
| *** lnxnut has quit IRC | 19:56 | |
| *** lnxnut has joined #openstack-swift | 19:57 | |
| *** hurrican_ has quit IRC | 19:57 | |
| *** hurricanerix has joined #openstack-swift | 19:58 | |
| *** lnxnut has quit IRC | 20:01 | |
| *** otoolee has quit IRC | 20:02 | |
| *** otoolee has joined #openstack-swift | 20:18 | |
| *** lnxnut has joined #openstack-swift | 20:21 | |
| *** bsdkurt has quit IRC | 20:25 | |
| *** bsdkurt has joined #openstack-swift | 20:26 | |
| *** bsdkurt has quit IRC | 20:28 | |
| *** bsdkurt has joined #openstack-swift | 20:30 | |
| *** judd7 has quit IRC | 20:36 | |
| *** bada_ has joined #openstack-swift | 20:48 | |
| *** bada has quit IRC | 20:48 | |
| *** csd has quit IRC | 20:49 | |
| *** ekarlso has quit IRC | 20:50 | |
| *** ekarlso has joined #openstack-swift | 20:50 | |
| *** bradsturtevant has joined #openstack-swift | 21:00 | |
| *** shri has quit IRC | 21:03 | |
| *** shri has joined #openstack-swift | 21:08 | |
| *** csd has joined #openstack-swift | 21:15 | |
| *** foexle has joined #openstack-swift | 21:20 | |
| *** dtalton has joined #openstack-swift | 21:21 | |
| *** dtalton2 has joined #openstack-swift | 21:22 | |
| *** dtalton has quit IRC | 21:26 | |
| *** jeblair has quit IRC | 21:45 | |
| *** dtalton2 has quit IRC | 21:46 | |
| *** jeblair has joined #openstack-swift | 21:50 | |
| *** dtalton has joined #openstack-swift | 21:52 | |
| *** dtalton2 has joined #openstack-swift | 21:53 | |
| *** dtalton3 has joined #openstack-swift | 21:55 | |
| *** dtalton3 has left #openstack-swift | 21:56 | |
| *** dtalton has quit IRC | 21:58 | |
| *** dtalton2 has quit IRC | 21:59 | |
| wer | so tempauth is just being weird. I created a new user in proxy-server.conf and it totally doesn't work. gets 401 unauthorized when trying to create or list anything. That's all using the swift cli and ENV variables. meanwhile if I auth with curl a get a valid X-Auth-Token and X-Storage-Url :/ I'm so confused. | 22:11 |
| luisbg | wer, are the ENV variables correct? | 22:17 |
| wer | luisbg: yeah I think so :) | 22:17 |
| luisbg | wer, also... how do you pass the token in the cli? | 22:17 |
| luisbg | 401 is usually wrong path or token | 22:18 |
| wer | I have tryed with -U and -K adn specifying the auth url too. | 22:18 |
| wer | luisbg: not sure? | 22:18 |
| luisbg | -A for auth_url, -U for user, -K for token | 22:18 |
| wer | yup. tried that too. | 22:18 |
| luisbg | what is the curl that works? let's start from there | 22:19 |
| wer | k hold for curl. | 22:19 |
| wer | curl -i -H "X-Auth-User:user:wer:" -H "X-Auth-Key:testpass" http://localhost:8080/auth/v1.0 Something like that yields good things. | 22:20 |
| luisbg | try: | 22:20 |
| luisbg | swift -A http://localhost:8080/auth/v1.0 -U user:wer -K tespass stat | 22:21 |
| *** dmsimard1 has quit IRC | 22:21 | |
| wer | Account HEAD failed: http://127.0.0.1:8080/v1/AUTH_user 403 Forbidden | 22:22 |
| luisbg | wer, what does curl get? | 22:23 |
| wer | it gets back the keys and storage urls and things :) | 22:23 |
| *** Anticime1 is now known as Anticimex | 22:24 | |
| *** fbo is now known as fbo_away | 22:24 | |
| zaitcev | So memcached is not running yet again | 22:24 |
| zaitcev | wer must be the 1000th guy who hit it | 22:25 |
| luisbg | oh! | 22:25 |
| wer | really? So this is a thing? Cause it felt like a thing. | 22:25 |
| *** dmsimard has joined #openstack-swift | 22:26 | |
| portante | 024922 | 22:29 |
| portante | sorry, ignore that! | 22:29 |
| luisbg | portante, nope, 024922 is the new way of lie | 22:30 |
| luisbg | life* | 22:30 |
| luisbg | it can't be ignored! | 22:30 |
| wer | luisbg: k so since I was fibbing to protect my user/auth, I created that test user we were using and go the same results. did someone say I should kick memcache? | 22:37 |
| luisbg | wer, zaitcev said thatr | 22:37 |
| luisbg | s/thatr/that | 22:37 |
| *** csd has quit IRC | 22:38 | |
| wer | hrm. That didn't help me :/ | 22:38 |
| zaitcev | wer: So, old users work? | 22:38 |
| wer | Well. I only ever had two. One was system:root and in .admin. The other was a dispersion one and they both work yes. | 22:39 |
| zaitcev | Okay, let's get serious, then. That means none of "Something like that yields". Please paste a terminal dump for curl auth command somewhere like paste.openstack.org. | 22:43 |
| wer | sure. Thats why I setup the user :) one sec | 22:43 |
| *** hurricanerix has quit IRC | 22:48 | |
| wer | http://pastebin.com/wDXWDsfa stuffs I kind of miffed the logs a bit :/ | 22:48 |
| wer | the garbage looking stuff from the swift cli concerns me a bit user%2CAUTH_tk10cbdebd73b84c7a88798064a35ca5dc | 22:49 |
| luisbg | wer, echo $ST_USER | 22:49 |
| wer | user:wer | 22:50 |
| luisbg | in one you use localhost in the other you use 127.0.0.1 | 22:51 |
| wer | oh interesting. lemmie poke there. | 22:51 |
| *** basha has joined #openstack-swift | 22:51 | |
| wer | I swapped in both directions and curl and swift cli acted the same. I don't think that is it. | 22:54 |
| luisbg | wer, this is running swift from a distribution package or from git? | 22:54 |
| wer | it's a package. 1.8.0 | 22:55 |
| luisbg | maybe zaitcev has more ideas, I would recommend an update | 22:56 |
| luisbg | trying it from git master's source | 22:56 |
| wer | looks like it came from deb http://ubuntu.lindenlab.com/ubuntu-cloud-mirror/ precise-updates/grizzly main. I'm running on ubuntu 12.04. yeah, I could attempt an update :/ I have several hundred TB of data on this thing :) | 22:57 |
| luisbg | AFAIK memcache is used in the swift/common and auth side, not in the client side | 22:57 |
| zaitcev | no, don't have an idea. I'm looking at wer's paste and thinking what it could be | 22:57 |
| luisbg | zaitcev, same here | 22:57 |
| luisbg | wer, after doing git clone of python-swifclient | 22:58 |
| luisbg | you can do ./python-switclient/bin/swift | 22:58 |
| luisbg | to run from source | 22:58 |
| wer | oh! ok. Let me pull that down real quick. have a link handy by chance? | 22:58 |
| luisbg | no. I meant ./python-swiftclient/bin/swift | 22:58 |
| luisbg | https://github.com/openstack/python-swiftclient.git | 22:59 |
| wer | thanks! let me get that on there and I'll try it. | 22:59 |
| zaitcev | I would rather use curl first | 23:01 |
| wer | zaitcev: curl return tokens and storage urls just fine. | 23:02 |
| zaitcev | curl -v -H "X-Auth-Token: AUTH_tk1e82afb0aad14c2e8cc6501bc13562e1" http://localhost:8080/v1/AUTH_user | 23:02 |
| wer | forbidden | 23:03 |
| zaitcev | right... so don't bother with client our of git | 23:03 |
| wer | yeah, I was about to tcpdump with the client before you guys showed up, cause I suspected it may be getting the token and storage url :/ | 23:03 |
| zaitcev | typically this sort of thing is memcached and rarely anything else | 23:04 |
| zaitcev | but since old user is working... | 23:04 |
| wer | well. So there is one thing that is atypical involving my memcache. | 23:04 |
| luisbg | zaitcev, good one | 23:04 |
| *** basha has quit IRC | 23:05 | |
| zaitcev | wer: what was in the log at the moment you tried the curl command that I pasted? Good token start or garbage? | 23:06 |
| wer | checking | 23:06 |
| wer | not garbage. | 23:07 |
| zaitcev | in my case | 23:09 |
| zaitcev | 08/26 GET /v1/AUTH_test HTTP/1.0 200 - curl/7.32.0 AUTH_tk3f089... - 26 - tx253d1d3636b64bc3bc0d3-005307dc6a - 0.0374 - - | 23:09 |
| wer | proxy-server STDOUT: ERROR:root:Error talking to memcached: localhost:11211#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/dist-packages/swift/coproxy-server STDOUT: ERROR:root:Error talking to memcached: localhost:11211#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/dist-packages/swift/common/memcached.py", line 202, in get#012 while line[0].upper() != 'END':#012IndexError: list index | 23:09 |
| wer | out of rangemmon/memcached.py", line 202, in get#012 while line[0].upper() != 'END':#012IndexError: list index out of range | 23:09 |
| zaitcev | ooh | 23:09 |
| wer | maybe that is something :) | 23:09 |
| luisbg | ding ding ding | 23:09 |
| zaitcev | but why | 23:10 |
| wer | cause it hates me :) | 23:10 |
| zaitcev | WTF is "coproxy-server" | 23:10 |
| wer | That was after I kicked memcache. | 23:11 |
| zaitcev | I don't what "kicked" is. Is it running right now or not? | 23:11 |
| wer | yeah it's running right now. | 23:11 |
| zaitcev | Okay. So, two things. I'd like to know what a "coproxy-server" is, because it's not something that comes in stock Swift. Could be anything broken. 2) try to find it and restart, see if it fixes the problem. Normal Swift reconnects after memcached is killed (of course users have to reauthorize, but your 1st curl command does that). | 23:15 |
| zaitcev | oh wait | 23:15 |
| wer | "coproxy-server" is lies. And I have no idea what it is :/ | 23:15 |
| zaitcev | right. Maybe log line length for swift/common/memcached.py, truncated and smashed with "proxy-server STDOUT..." | 23:16 |
| *** jergerber has quit IRC | 23:16 | |
| wer | Each time I auth I get that same token right now. so I think it is still valid. I have one middleware in my pipeline I could disable. | 23:17 |
| zaitcev | that index out of range thing means that readline got EOF | 23:17 |
| wer | is that while reading the cache? | 23:17 |
| zaitcev | while reading from memcached socket | 23:17 |
| wer | ok. hrm. | 23:18 |
| zaitcev | I would not "kick" or restart memcached to begin with, but since it happened, perhaps we have a bug that prevents automatic reconnect, or, rather, detection of disconnect so that reconnect can happen. | 23:20 |
| wer | restarting the proxy-server would get around that problem if it existed right? | 23:20 |
| zaitcev | it should fix it, yes. | 23:21 |
| wer | hrm. I am at a loss here. And the fact that I keep getting the same X-Auth-Token makes me thing memcache is working. I saw that error twice. But have not seen it since. It was a few minutes after the restart of memcache so perhaps it was reconnecting on my first attempt? | 23:23 |
| zaitcev | 0x2C is a comma, so user%2CAUTH_tk1e82a is user,AUTH_tk1e.... | 23:23 |
| wer | ok :) I always wondered. | 23:23 |
| zaitcev | we'll get back to it, but curl first | 23:24 |
| zaitcev | there was some odd way to specify reseller prefix, so I'll need to look at proxy-server.conf. I forgot the exact syntax | 23:25 |
| wer | yeah .reseller or something I thought. | 23:25 |
| zaitcev | reseller prefix goes right there. Technically the default reseller prefix is "AUTH_", which is what we see normally. | 23:26 |
| wer | I may not follow what you are suggesting. I'm not using a reseller right? | 23:33 |
| *** rustlebee is now known as russellb | 23:36 | |
| zaitcev | could you post proxy-server.conf somewhere? When you redact passwords, be careful. Whitespace may be significant. | 23:50 |
| wer | http://pastebin.com/dk33JST4 | 23:52 |
| wer | It's not a very complicated one :) | 23:53 |
| wer | I added wer to .admin just recently. | 23:54 |
| zaitcev | but that addition changed nothing? | 23:54 |
| wer | nothing changed. | 23:54 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!