| *** mwstorer has quit IRC | 00:15 | |
| *** kevinc_ has quit IRC | 00:18 | |
| *** matsuhashi has joined #openstack-swift | 00:23 | |
| *** sungju has joined #openstack-swift | 00:24 | |
| *** yuan has joined #openstack-swift | 00:32 | |
| *** shri1 has joined #openstack-swift | 00:40 | |
| *** shri has quit IRC | 00:41 | |
| *** shri1 has quit IRC | 00:42 | |
| *** changbl has joined #openstack-swift | 00:44 | |
| *** mariusv has quit IRC | 01:11 | |
| *** mariusv has joined #openstack-swift | 01:12 | |
| *** mariusv has quit IRC | 01:12 | |
| *** mariusv has joined #openstack-swift | 01:12 | |
| *** nosnos has joined #openstack-swift | 01:24 | |
| *** baojg has joined #openstack-swift | 01:36 | |
| *** saschpe has quit IRC | 01:36 | |
| *** mariusv has quit IRC | 01:37 | |
| *** mariusv has joined #openstack-swift | 01:37 | |
| *** mariusv has quit IRC | 01:37 | |
| *** mariusv has joined #openstack-swift | 01:37 | |
| *** saschpe has joined #openstack-swift | 01:38 | |
| *** mariusv has quit IRC | 01:41 | |
| *** mariusv has joined #openstack-swift | 01:43 | |
| *** mariusv has quit IRC | 01:43 | |
| *** mariusv has joined #openstack-swift | 01:43 | |
| *** mariusv has quit IRC | 01:46 | |
| *** foexle_ has joined #openstack-swift | 01:47 | |
| *** mariusv has joined #openstack-swift | 01:48 | |
| *** mariusv has quit IRC | 01:48 | |
| *** mariusv has joined #openstack-swift | 01:48 | |
| *** foexle has quit IRC | 01:50 | |
| *** mariusv has quit IRC | 01:54 | |
| *** mariusv has joined #openstack-swift | 01:54 | |
| *** mariusv has quit IRC | 01:54 | |
| *** mariusv has joined #openstack-swift | 01:54 | |
| *** mariusv has quit IRC | 01:55 | |
| *** mariusv has joined #openstack-swift | 01:56 | |
| *** mariusv has quit IRC | 01:56 | |
| *** mariusv has joined #openstack-swift | 01:56 | |
| *** baojg has quit IRC | 01:57 | |
| *** baojg has joined #openstack-swift | 01:57 | |
| *** haomaiw__ has joined #openstack-swift | 02:01 | |
| *** mariusv has quit IRC | 02:05 | |
| *** haomaiwang has quit IRC | 02:06 | |
| *** mariusv has joined #openstack-swift | 02:06 | |
| *** mariusv has quit IRC | 02:06 | |
| *** mariusv has joined #openstack-swift | 02:06 | |
| openstackgerrit | A change was merged to openstack/swift: Bring Object Auditor Coverage To 99% https://review.openstack.org/87101 | 02:06 |
|---|---|---|
| *** mariusv has quit IRC | 02:07 | |
| *** mariusv has joined #openstack-swift | 02:07 | |
| *** mariusv has quit IRC | 02:07 | |
| *** mariusv has joined #openstack-swift | 02:07 | |
| *** haomaiw__ has quit IRC | 02:18 | |
| *** haomaiwa_ has joined #openstack-swift | 02:19 | |
| openstackgerrit | A change was merged to openstack/python-swiftclient: Add tests for bin/swift https://review.openstack.org/73710 | 02:28 |
| *** gyee has quit IRC | 02:59 | |
| *** praveenkumar has joined #openstack-swift | 03:03 | |
| *** matsuhashi has quit IRC | 03:09 | |
| *** matsuhashi has joined #openstack-swift | 03:09 | |
| *** matsuhashi has quit IRC | 03:09 | |
| *** nosnos has quit IRC | 03:19 | |
| *** chandan_kumar has joined #openstack-swift | 03:20 | |
| *** matsuhashi has joined #openstack-swift | 03:25 | |
| *** matsuhashi has quit IRC | 03:27 | |
| *** matsuhashi has joined #openstack-swift | 03:28 | |
| *** matsuhashi has quit IRC | 03:31 | |
| *** matsuhashi has joined #openstack-swift | 03:31 | |
| *** matsuhas_ has joined #openstack-swift | 03:34 | |
| *** matsuhashi has quit IRC | 03:35 | |
| *** lnxnut has quit IRC | 03:37 | |
| *** lnxnut has joined #openstack-swift | 03:39 | |
| *** matsuhas_ has quit IRC | 03:54 | |
| *** matsuhashi has joined #openstack-swift | 03:57 | |
| *** zhiyan_ is now known as zhiyan | 04:04 | |
| *** nosnos has joined #openstack-swift | 04:10 | |
| *** baojg_ has joined #openstack-swift | 04:14 | |
| *** baojg has quit IRC | 04:17 | |
| *** lnxnut has quit IRC | 04:22 | |
| *** baojg_ has quit IRC | 04:30 | |
| *** baojg has joined #openstack-swift | 04:30 | |
| *** zhiyan is now known as zhiyan_ | 04:41 | |
| *** d89 has joined #openstack-swift | 05:00 | |
| *** perryh has joined #openstack-swift | 05:09 | |
| *** zaitcev has quit IRC | 05:17 | |
| *** zhiyan_ is now known as zhiyan | 05:42 | |
| *** cheri has joined #openstack-swift | 06:07 | |
| *** cheri has quit IRC | 06:13 | |
| *** sungju has quit IRC | 06:28 | |
| *** haomaiwa_ has quit IRC | 06:44 | |
| *** haomaiwang has joined #openstack-swift | 06:45 | |
| *** chandan_kumar has quit IRC | 06:57 | |
| *** chandan_kumar has joined #openstack-swift | 07:05 | |
| *** matsuhashi has quit IRC | 07:06 | |
| *** matsuhashi has joined #openstack-swift | 07:08 | |
| *** judd7 has joined #openstack-swift | 07:12 | |
| *** nshaikh has joined #openstack-swift | 07:22 | |
| *** haomaiw__ has joined #openstack-swift | 07:26 | |
| openstackgerrit | Eamonn O'Toole proposed a change to openstack/swift: Parallel object auditor https://review.openstack.org/59778 | 07:27 |
| *** haomaiwang has quit IRC | 07:29 | |
| *** psharma has joined #openstack-swift | 07:45 | |
| *** nacim has joined #openstack-swift | 07:55 | |
| *** nacim has quit IRC | 08:02 | |
| *** nacim has joined #openstack-swift | 08:05 | |
| openstackgerrit | Madhuri Kumari proposed a change to openstack/swift: Added missing direct backend APIs and unit test for it https://review.openstack.org/84819 | 08:20 |
| *** foexle_ has quit IRC | 09:02 | |
| *** AndrewPark has joined #openstack-swift | 09:11 | |
| AndrewPark | I made the s3 compatible api with swift3. So I test the apis using ceph/s3-tests(https://github.com/ceph/s3-tests) for the s3 compatibiilty. Is there anyone who has experience with s3-tests? | 09:19 |
| *** mkollaro has joined #openstack-swift | 09:23 | |
| *** haomaiw__ has quit IRC | 10:06 | |
| *** foexle has joined #openstack-swift | 10:06 | |
| *** haomaiwang has joined #openstack-swift | 10:07 | |
| *** haomaiw__ has joined #openstack-swift | 10:11 | |
| *** haomaiwang has quit IRC | 10:14 | |
| *** mlipchuk has joined #openstack-swift | 10:17 | |
| *** AndrewPark has quit IRC | 10:35 | |
| *** haomaiw__ has quit IRC | 10:37 | |
| *** haomaiwa_ has joined #openstack-swift | 10:38 | |
| *** haomai___ has joined #openstack-swift | 10:44 | |
| *** haomai___ has quit IRC | 10:45 | |
| *** haomaiw__ has joined #openstack-swift | 10:46 | |
| *** haomaiwa_ has quit IRC | 10:46 | |
| *** nosnos has quit IRC | 11:01 | |
| *** matsuhashi has quit IRC | 11:12 | |
| *** matsuhashi has joined #openstack-swift | 11:14 | |
| *** waterkinfe has joined #openstack-swift | 11:16 | |
| *** waterkinfe has quit IRC | 11:26 | |
| *** nosnos has joined #openstack-swift | 11:26 | |
| *** baojg has quit IRC | 11:28 | |
| *** baojg has joined #openstack-swift | 11:28 | |
| *** nosnos has quit IRC | 11:29 | |
| *** waterkinfe has joined #openstack-swift | 11:29 | |
| *** matsuhashi has quit IRC | 11:32 | |
| *** matsuhashi has joined #openstack-swift | 11:32 | |
| *** baojg has quit IRC | 11:33 | |
| *** matsuhashi has quit IRC | 11:38 | |
| *** waterkinfe has quit IRC | 12:00 | |
| *** waterkinfe has joined #openstack-swift | 12:01 | |
| *** foexle has quit IRC | 12:01 | |
| *** lnxnut has joined #openstack-swift | 12:08 | |
| *** Anju_ has joined #openstack-swift | 12:38 | |
| *** bvandenh has joined #openstack-swift | 12:42 | |
| *** foexle has joined #openstack-swift | 12:44 | |
| *** bach has joined #openstack-swift | 12:56 | |
| *** nshaikh has quit IRC | 13:11 | |
| *** psharma has quit IRC | 13:12 | |
| *** bach has quit IRC | 13:13 | |
| *** Trixboxer has joined #openstack-swift | 13:13 | |
| *** waterkinfe has quit IRC | 13:19 | |
| *** bach has joined #openstack-swift | 13:32 | |
| *** foexle has quit IRC | 13:33 | |
| *** chandan_kumar has quit IRC | 13:42 | |
| *** foexle has joined #openstack-swift | 13:45 | |
| *** mlipchuk has quit IRC | 13:46 | |
| *** d89 has quit IRC | 13:48 | |
| *** bvandenh has quit IRC | 14:01 | |
| *** kevinc_ has joined #openstack-swift | 14:23 | |
| *** byeager has joined #openstack-swift | 14:29 | |
| *** foexle has quit IRC | 14:37 | |
| *** baojg has joined #openstack-swift | 14:37 | |
| *** baojg has quit IRC | 14:45 | |
| *** baojg has joined #openstack-swift | 14:46 | |
| *** RockKuo_TW has joined #openstack-swift | 14:46 | |
| *** kevinc_ has quit IRC | 14:52 | |
| *** Anju_ has quit IRC | 15:01 | |
| *** byeager has quit IRC | 15:10 | |
| *** byeager has joined #openstack-swift | 15:11 | |
| *** baojg has quit IRC | 15:20 | |
| *** baojg has joined #openstack-swift | 15:23 | |
| *** d89 has joined #openstack-swift | 15:41 | |
| *** shakamunyi has joined #openstack-swift | 15:44 | |
| *** shakamunyi has quit IRC | 15:44 | |
| *** RockKuo_TW has quit IRC | 15:46 | |
| *** zhiyan is now known as zhiyan_ | 15:50 | |
| *** chandan_kumar has joined #openstack-swift | 15:51 | |
| *** zaitcev has joined #openstack-swift | 15:53 | |
| *** ChanServ sets mode: +v zaitcev | 15:53 | |
| *** d89 has quit IRC | 15:56 | |
| *** Diddi has quit IRC | 15:57 | |
| *** nacim has quit IRC | 15:58 | |
| *** mwstorer has joined #openstack-swift | 16:03 | |
| *** tnewsome has joined #openstack-swift | 16:08 | |
| tnewsome | Running 1.12.0, all of a sudden I have several swift processes chewing 100% CPU. | 16:12 |
| tnewsome | 'swift-init restart all' doesn't touch them. | 16:13 |
| tnewsome | strace shows they're endlessly: sendto(8, "<131>object-server: STDOUT: Traceback (most recent call last): (txn: tx17b63337c9cd419f99420-00534ec976)\0", 105, 0, NULL, 0) = -1 ENOTCONN (Transport endpoint is not connected) | 16:13 |
| tnewsome | Manually killing the processes helps. Is this a known bug? | 16:14 |
| swills | so for someone who isn't that fluent in python, what is this exception trying to tell me: http://paste.mouf.net/pastes/9ai55od3f4bjdv0kjktdn0lu ? | 16:16 |
| tnewsome | swills: That seems to be missing the actual exception itself? Is there more text after what you pasted? | 16:18 |
| zaitcev | tnewsome: not to me... File it to preserve the traceback and circumstances. | 16:18 |
| *** shakamunyi has joined #openstack-swift | 16:18 | |
| swills | tnewsome: that's all I get in syslog | 16:19 |
| swills | i can turn debugging up if that helps | 16:19 |
| tnewsome | swills: Well, it doesn't tell me anything specific. But I only know python, not really swift. | 16:20 |
| tnewsome | zaitcev: OK. This problem occurs if I restart syslogd while swift is running. | 16:24 |
| *** gyee has joined #openstack-swift | 16:33 | |
| *** praveenkumar has quit IRC | 16:37 | |
| *** bach has quit IRC | 16:38 | |
| notmyname | good morning world | 16:47 |
| peluse_ | good morning | 16:47 |
| notmyname | tnewsome: check out the swift scripts "swift-oldies" and "swift-orphans" | 16:47 |
| tnewsome | notmyname: Looks good, but I'm going to just not restart rsyslogd instead. :-) | 17:03 |
| *** shri has joined #openstack-swift | 17:04 | |
| openstackgerrit | paul luse proposed a change to openstack/swift: Add Ratelimit parameters to /info https://review.openstack.org/88633 | 17:08 |
| swills | i've set log_level to DEBUG but i sitll don't get a more useful traceback, what am I missing? | 17:18 |
| *** judd7 has quit IRC | 17:22 | |
| *** byeager has quit IRC | 17:24 | |
| zaitcev | looks length limited. By this time I usually give up and start doing fp=open("/tmp/dump",'a') fp.write(traceback.format_exc()) | 17:26 |
| *** Anju_ has joined #openstack-swift | 17:31 | |
| *** EmilienM has quit IRC | 17:50 | |
| *** EmilienM has joined #openstack-swift | 17:50 | |
| *** baojg has quit IRC | 17:54 | |
| *** byeager has joined #openstack-swift | 17:55 | |
| *** foexle has joined #openstack-swift | 17:56 | |
| *** byeager_ has joined #openstack-swift | 18:00 | |
| *** byeager has quit IRC | 18:00 | |
| *** foexle has quit IRC | 18:01 | |
| *** mlipchuk has joined #openstack-swift | 18:08 | |
| swills | oh, hmm, ok | 18:09 |
| swills | so i just put some debugging in and get that bytes_used == 0 | 18:10 |
| *** foexle has joined #openstack-swift | 18:13 | |
| *** bach has joined #openstack-swift | 18:15 | |
| *** foexle has quit IRC | 18:23 | |
| *** tnewsome has quit IRC | 18:26 | |
| peluse_ | notmyname: did you see my trello note wrt acct reaper and internal client? | 18:26 |
| notmyname | no | 18:26 |
| swills | this is so weird, bytes_used = 0, object_count = 0 and yet i'm getting this trace | 18:28 |
| *** mlipchuk has quit IRC | 18:28 | |
| peluse_ | notmyname: OK, take a look when you get a chance... | 18:30 |
| notmyname | https://wiki.openstack.org/wiki/Governance/TechnicalCommittee/candidates | 18:30 |
| * peluse_ already voted | 18:30 | |
| notmyname | just saw some people on twitter looking for info on who was running, so I put that together | 18:31 |
| peluse_ | cool... I got an email this morn I think to rank the 17 - of course put you in first place :) | 18:31 |
| notmyname | wow. that merge master to ec patch is really making jenkins angry | 18:36 |
| peluse_ | yeah, what's up with that? | 18:38 |
| peluse_ | 0% pass rate, not too shabby :) | 18:38 |
| notmyname | looks like -infra changed some configs | 18:38 |
| peluse_ | sweet | 18:38 |
| notmyname | and there isn't a feature/ec branch for the other projects | 18:38 |
| *** perryh has left #openstack-swift | 18:43 | |
| *** mkollaro has quit IRC | 18:48 | |
| creiht | aww tnewsome is gone | 18:49 |
| creiht | there is certainly an issue when syslog restarts | 18:49 |
| creiht | we have to restart everything when we do | 18:49 |
| notmyname | oh, yeah. we do too. | 18:50 |
| creiht | otherwise some of the processess seem to go bonkers | 18:50 |
| pandemicsyn | heh i just left a note on his bug report | 18:50 |
| creiht | pandemicsyn: oh cool | 18:50 |
| creiht | thx | 18:50 |
| notmyname | and I just now remembered that when you said something | 18:50 |
| notmyname | pandemicsyn: thanks | 18:50 |
| notmyname | spacex live webcast just started http://www.spacex.com/webcast/ | 18:50 |
| creiht | cool | 18:50 |
| notmyname | I got some good news about the "Swift pod" at the summit | 18:50 |
| creiht | notmyname: j's satelite is on that launch | 18:50 |
| notmyname | awesome! | 18:51 |
| notmyname | swift in space? | 18:51 |
| notmyname | what does his sat do? | 18:51 |
| pandemicsyn | i think it transmits his initials and some basic telemetry or something | 18:52 |
| notmyname | swift pod: | 18:54 |
| notmyname | http://d.not.mn/b202.jpg | 18:54 |
| notmyname | http://d.not.mn/pod.png | 18:54 |
| notmyname | we're in pod 1, in the corner | 18:54 |
| notmyname | pandemicsyn: do you have the webcast up on the big tv? :-) | 18:55 |
| pandemicsyn | na, the tv's are too far away, and im feeling kinda lazy today | 18:55 |
| creiht | haha | 18:55 |
| pandemicsyn | don't think i can be bothered to roll into range ;) | 18:55 |
| zaitcev | Let's just hope a leg doesn't unlatch at max-q or something | 19:01 |
| *** mlipchuk has joined #openstack-swift | 19:12 | |
| *** Trixboxer has quit IRC | 19:21 | |
| *** foexle has joined #openstack-swift | 19:26 | |
| *** foexle has quit IRC | 19:26 | |
| *** shakayumi has joined #openstack-swift | 19:26 | |
| *** shakamunyi has quit IRC | 19:27 | |
| *** dmsimard_ has joined #openstack-swift | 19:30 | |
| *** dmsimard has quit IRC | 19:33 | |
| *** dmsimard_ is now known as dmsimard | 19:33 | |
| *** tongli has joined #openstack-swift | 19:33 | |
| *** ashish_ has joined #openstack-swift | 19:37 | |
| ashish_ | Hey everyone I am running ssbench benchmarks on a single node swift which uploads various objects into swift.I am using the admin as user.But when I list the containers in my admin account there is no mention of the containers uploaded by ssbench. | 19:41 |
| zaitcev | Forgot to start updaters again? | 19:43 |
| clayg | lol @ "again" | 19:46 |
| ashish_ | Hey zaitcev clayg . I am running ssbench benchmarks on a single node swift which uploads various objects into swift.I am using the admin as user.But when I list the containers in my admin account there is no mention of the containers uploaded by ssbench. | 19:48 |
| clayg | ashish_: yeah i don't know container put should be a syncronous update to your account databases - unless those services are busted or something? | 19:56 |
| clayg | ashish_: did you check the logs for errors? | 19:56 |
| ashish_ | clayg the containers are uploaded in the storage nodes but when I am trying to access them through curl or swift CLI there is no mention of the ssbench containers | 19:58 |
| clayg | ashish_: k, are you sure ssbench and you tests from the command line are using the same account/credentials? If so - check your logs and find out what's broken. | 20:00 |
| ashish_ | clayg I'm sure that I'm using the admin user for both swift CLI and ssbench. | 20:01 |
| ashish_ | clayg there's nothing in the error log | 20:06 |
| swills | so I'm getting a "disk I/O error" for some reason | 20:08 |
| swills | https://github.com/openstack/swift/blob/master/swift/container/updater.py#L236-L238 | 20:09 |
| swills | there | 20:09 |
| swills | what would cause a disk I/O error there I wonder... | 20:09 |
| clayg | ashish_: both of those statements can't be true, and you probably know this intuitively - can you find one of the log lines from the ssbench run, from that one of the PUT's for an ssbech container? | 20:11 |
| clayg | swills: permissions most of the time | 20:11 |
| clayg | well... i guess that's just a generic ioerror - maybe I should look at your paste ;) | 20:11 |
| swills | clayg: ah! | 20:12 |
| swills | -rw------- 1 swift swift 16384 Apr 18 19:13 /srv/node/1/containers/241/496/3c53a249ea13e846cb847981f0aed496/3c53a249ea13e846cb847981f0aed496.db | 20:12 |
| swills | looks OK I think? | 20:12 |
| clayg | swills: as long as the updaters are running as swift that should do it then yeah | 20:13 |
| clayg | one time I saw uid/gid change - but that had to do with drives like changings hosts or the os getting restinalled... | 20:13 |
| clayg | and then I think the user was ??? or some such... | 20:13 |
| ashish_ | clayg I have checked the log files of swift and could not find anything there. | 20:13 |
| clayg | ashish_: you can find any log lines from your benchmark? and you can't find any containers from your benchmark? what evidence do you have that the benchmark ran exactly? | 20:14 |
| clayg | swills: do you have a stack trace? | 20:14 |
| swills | clayg: sorta http://paste.mouf.net/pastes/9ai55od3f4bjdv0kjktdn0lu/raw | 20:15 |
| swills | been working on getting something more detailed | 20:15 |
| ashish_ | clayg the files created during the benchmark are there on my storage nodes but not being displayed when I use the swift cli. | 20:16 |
| clayg | ashish_: use swift-object-info to inspect one of the files, make sure it matches your expectation - then try to find the container db (with swift-get-nodes) or the log line that created the object. | 20:18 |
| clayg | too bad it's cut off - is the container-updater really missing a catch all exception handler? it'd be nice to get some logging around process container that included at least the disk path :\ | 20:19 |
| *** mlipchuk has quit IRC | 20:20 | |
| *** chandan_kumar has quit IRC | 20:21 | |
| *** Anju_ has quit IRC | 20:21 | |
| *** mlipchuk has joined #openstack-swift | 20:25 | |
| swills | clayg: so i put an exception catcher in... and all I get is "disk I/O error" out of it... | 20:25 |
| clayg | swifterdarrell: but is /srv/node/1/containers/241/496/3c53a249ea13e846cb847981f0aed496/3c53a249ea13e846cb847981f0aed496.db the file that's giving the error? | 20:26 |
| swills | so here, https://github.com/openstack/swift/blob/master/swift/container/backend.py#L350-L356 | 20:26 |
| swills | I added: | 20:26 |
| swills | 356 except Exception as e: | 20:26 |
| swills | 357 swills=open("/tmp/dump",'a') | 20:26 |
| swills | 358 swills.write(str(e)) | 20:26 |
| swills | all I get there is "disk I/O error" over and over... but i do think that's the file, let me check... | 20:26 |
| clayg | swills: ok, can you also write self.db_file so we know which db is blowing up? | 20:26 |
| swills | (after adding try: above...) | 20:26 |
| clayg | do you have swift-container-info in your installed code? it's sorta newish, but it'd be interesting to see if it can select from that table... you can do the same thing with sqlite3 | 20:27 |
| swills | i dont, i'm running 1.13.0 | 20:28 |
| clayg | sqlite3 --line /path/to/db "select * from container_stat;" | 20:29 |
| ashish_ | clayg When I ran /usr/local/bin/swift-get-nodes /etc/swift/account.ring.gz admin it shows 0 containers and objects.Could you tell me in which account is my ssbench data getting uploaded ? | 20:29 |
| swills | clayg: http://paste.mouf.net/pastes/9mqnm66lkq0roafef2gl2j4b | 20:30 |
| swills | container = Ω嘈㜅댌嘅ऀ᠁鐉蔋 | 20:31 |
| swills | heh | 20:31 |
| clayg | swills: interesting that it's DELETED but never reported | 20:32 |
| clayg | i guess it wasn't around very long... | 20:33 |
| swills | probably not | 20:34 |
| ashish_ | clayg i dont know the names of the objects as ssbench creates its own objects during benchmarking which are in encrypted form in the swift db. | 20:34 |
| swills | clayg: i'm OK with whiping out all my data and starting from scratch, but how? | 20:35 |
| clayg | how many accounts do you have? can you "find /srv/node*/sd*/account* -name \*.db" | 20:35 |
| swills | /srv/node/1/accounts/705/253/b06bd0bdd0bbd3c03f42fd74c7ad3253/b06bd0bdd0bbd3c03f42fd74c7ad3253.db | 20:35 |
| clayg | swills: well i don't rightly know that will help - it's weird that the update is getting disk io and the select is working - mounted r/o? | 20:35 |
| swills | just the one | 20:35 |
| swills | clayg: definitely not mounted read only | 20:36 |
| clayg | or sorry swills that was for ashish_ :P | 20:36 |
| swills | np | 20:36 |
| clayg | swills: i guess sqlite /path/to/db "update container_stat set reported_object_count=1" | 20:37 |
| clayg | swills: as far as wipe everything out :\ - is this a saio? | 20:38 |
| swills | clayg: thing is, i'm seeing similar failures on lots of files | 20:38 |
| swills | clayg: not exactly | 20:38 |
| swills | but sorta | 20:38 |
| swills | it's a dev testing setup | 20:38 |
| clayg | swills: well i'm just curious if you can update that row outside of the updater or if the error only appears there? | 20:38 |
| clayg | cause if it works from the commandline it's not like the damn db is *really* messed up - it's just the updater being stupid | 20:38 |
| swills | wait a sec | 20:38 |
| swills | http://paste.mouf.net/pastes/a5hpnrjpfgk343sr3gcg6vpn | 20:39 |
| clayg | ... waiting | 20:39 |
| swills | http://paste.mouf.net/pastes/vca1hl6ph9vh4h4ra21drsss | 20:40 |
| swills | see that first paste there? | 20:40 |
| swills | lots of errors | 20:40 |
| clayg | ok, cool - well can you read *those* containers? (do the sqlite select trick) | 20:40 |
| swills | yep | 20:42 |
| swills | they are all fine | 20:42 |
| swills | they are all containers generated by running the test suite, i think | 20:42 |
| swills | there are 413 .db files in the containers dir | 20:44 |
| ashish_ | clayg yes I can find /srv/1/node/sdb1/accounts/802/178/c8bcccab3ddbfdc34b08e9223f4f5178.db. | 20:44 |
| swills | running "stat" only shows 2 active containers tho | 20:44 |
| swills | which makes sense | 20:45 |
| swills | i created 2 test containers then ran the test suite | 20:45 |
| swills | multiple times | 20:45 |
| swills | well, i have to run now, ttyl | 20:47 |
| ashish_ | swills clayg my containers are being created during benchmark but on stat I see none of the containers created during the benchmark. | 20:48 |
| *** tongli has quit IRC | 20:49 | |
| clayg | ashish_: can you run swift-account-info on that account database? | 20:59 |
| ashish_ | clayg Please give me a minute. | 21:10 |
| *** shri has left #openstack-swift | 21:14 | |
| *** shri has joined #openstack-swift | 21:14 | |
| ashish_ | clayg this file is not there in my swift-account-info file in my /usr/local/bin directory | 21:17 |
| clayg | ashish_: k, it's new - just sqlite3 --line /path/to/db "select * from account_stat" | 21:17 |
| *** Diddi has joined #openstack-swift | 21:19 | |
| ashish_ | clayg what is the cmd here? | 21:19 |
| clayg | "sqlite3 --line /srv/1/node/sdb1/accounts/802/178/c8bcccab3ddbfdc34b08e9223f4f5178.db 'select * from account_stat'" | 21:20 |
| ashish_ | clayg thanks | 21:20 |
| ashish_ | clayg I got a error :"Error: no such table: account_stat" | 21:21 |
| clayg | ashish_: typo? | 21:24 |
| clayg | sqlite3 --line /srv/node1/sdb1/accounts/802/178/c8bcccab3ddbfdc34b08e9223f4f5178/c8bcccab3ddbfdc34b08e9223f4f5178.db 'select * from account_stat' | 21:24 |
| clayg | where is your hashh path? | 21:25 |
| clayg | what are you doing? | 21:25 |
| ashish_ | clayg whats a hashh path. | 21:26 |
| clayg | the part of the path between the hash_suffix and the db file name - your path looks different than the path on my machine | 21:27 |
| ashish_ | clayg thanks. /srv/1/node/sdb1/accounts/802/178/c8bcccab3ddbfdc34b08e9223f4f5178/c8bcccab3ddbfdc34b08e9223f4f5178.db is this my hash path. | 21:32 |
| clayg | yeah so that should have an account_stat table - it has to? | 21:32 |
| ashish_ | clayg got it thanks >Now I get this error:"Error: unable to open database "/srv/1/node/sdb1/accounts/802/178/c8bcccab3ddbfdc34b08e9223f4f5178.db/c8bcccab3ddbfdc34b08e9223f4f5178.db": unable to open database file | 21:34 |
| ashish_ | " | 21:34 |
| clayg | path still looks wrong to many .db's | 21:35 |
| ashish_ | clayg Got it.This is the output.http://paste.openstack.org/show/76313/ | 21:37 |
| ashish_ | clayg are you there. | 21:49 |
| ashish_ | clayg are you there? | 21:49 |
| clayg | k, is that the only account db you have on the file system - no other hashes? | 21:50 |
| *** byeager_ has quit IRC | 21:51 | |
| ashish_ | I do have other accounts | 21:51 |
| ashish_ | should I check all the accounts in my swift | 21:52 |
| *** sungju has joined #openstack-swift | 21:56 | |
| ashish_ | clayg are you there? | 22:04 |
| clayg | ashish_: well maybe you'll find the one that your've configured ssbench to talk to? | 22:09 |
| ashish_ | clayg.But ssbench uses the account I provide in the ssbench-report-scenario.Do I need to configure an account with ssbench? | 22:10 |
| clayg | idk, i thought it was a command line option - did you say you found one of the objects from your ssbench run? | 22:11 |
| clayg | what is the account listed there if you use swift-object-info on it? | 22:11 |
| ashish_ | clayg I.yes I found an object from my test runs.I will check the account for it .please give me a minute | 22:12 |
| ashish_ | clayg the object here shows to be of the admin account http://paste.openstack.org/show/76314/. | 22:19 |
| clayg | stupid paste.openstack.org is slow | 22:20 |
| clayg | ashish_: well ok then good! that solves that mystery! | 22:20 |
| clayg | glad to help | 22:20 |
| ashish_ | clayg thanks | 22:22 |
| ashish_ | clayg but when I swift -A http://127.0.0.1:8080/auth/v1.0 -U admin:admin -K admin \stat -v there is no mention of the containers created by ssbench.That is the main problem I am facing. | 22:23 |
| clayg | ashish_: you sure? what about list? maybe try and find ssbench_000000 with swift-get-nodes | 22:28 |
| ashish_ | clayg I am sure about it.Please give me a minute to try swift-get-nodes | 22:30 |
| ashish_ | clayg I have posted the output of both the cmds. http://paste.openstack.org/show/76318/ | 22:36 |
| clayg | swift-get-nodes /etc/swift/container.ring.gz AUTH_admin ssbench_000000 | 22:37 |
| clayg | then you want to check those paths for the db's | 22:37 |
| *** jeblair is now known as corvus | 22:37 | |
| ashish_ | clayg I have tried the swift-get-node I have attached the output in thae paste. | 22:39 |
| *** corvus is now known as jeblair | 22:40 | |
| clayg | ashish_: you used it wrong | 22:40 |
| clayg | run it like i said -> swift-get-nodes /etc/swift/container.ring.gz AUTH_admin ssbench_000000 | 22:40 |
| clayg | then go and find those containers | 22:40 |
| ashish_ | clayg okay Thanks a lot.I will try. | 22:41 |
| *** bach has quit IRC | 22:41 | |
| ashish_ | clayg I still get the same output. | 22:41 |
| ashish_ | http://paste.openstack.org/show/76319/ | 22:44 |
| ashish_ | clayg I am able to find the containers on the storage nodes.But why cant I find them when I try to"list containers in the admin account via swift CLI."? | 22:48 |
| ashish_ | clayg are you there? | 22:57 |
| ashish_ | clayg Thanks a lot clayg you have cleared several of my doubts.Would ask the remaining doubts tommorrow. | 23:00 |
| *** ashish_ has quit IRC | 23:02 | |
| occupant | so we've been putting account, container, and object data across all drives, but we're thinking it would be better to put the account and container data on an SSD and just use the HDDs for objects. I'm thinking I should try doing this just on one node first, but is this going to cause some kind of rebalancing nightmare? | 23:11 |
| pandemicsyn | occupant: should be fine we made a similar transition when we first started, you can always add the new drives to the ring and slowly increment their weight | 23:16 |
| *** bach has joined #openstack-swift | 23:24 | |
| notmyname | occupant: here's an overview of what pandemicsyn is talking about, but with the perspective of adding capacity. works the same way for readjusting to SSDs https://swiftstack.com/blog/2012/04/09/swift-capacity-management/ | 23:53 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
