| *** lnxnut has quit IRC | 00:12 | |
| *** zhiyan_ is now known as zhiyan | 00:23 | |
| *** matsuhashi has joined #openstack-swift | 00:30 | |
| *** matsuhashi has quit IRC | 00:52 | |
| portante | pandemicsyn: how long have you been using go and what is your view of it compared to python? | 00:57 |
|---|---|---|
| *** matsuhashi has joined #openstack-swift | 00:58 | |
| *** nosnos has joined #openstack-swift | 01:00 | |
| pandemicsyn | portante: just a little while, i started using it while starting the Matasano crypto challenges back in december, Go's kinda more my style... I like the strictness of it i think, but I've never been awesome with python so I'm probably not the best one to make comparisons | 01:31 |
| pandemicsyn | I will stay that i can churn out python code a lot faster but i think i spend a lot less time debugging my Go code when i go to use it | 01:32 |
| pandemicsyn | and after wrestling with multiprocess in python goroutines and GOMAXPROCS() is kinda awesome magic | 01:33 |
| *** saschpe has quit IRC | 01:33 | |
| *** saschpe has joined #openstack-swift | 01:35 | |
| portante | pandemicsyn: thanks for the observations, I have heard the multiple process is really cool | 01:53 |
| *** haomaiwang has quit IRC | 02:07 | |
| *** haomaiwang has joined #openstack-swift | 02:07 | |
| *** fifieldt_ has joined #openstack-swift | 02:11 | |
| *** praveenkumar has joined #openstack-swift | 02:59 | |
| *** matsuhashi has quit IRC | 03:32 | |
| *** matsuhashi has joined #openstack-swift | 03:32 | |
| *** matsuhashi has quit IRC | 03:37 | |
| *** nosnos has quit IRC | 03:46 | |
| *** fifieldt_ has quit IRC | 03:47 | |
| *** zhiyan is now known as zhiyan_ | 03:50 | |
| *** zhiyan_ is now known as zhiyan | 03:52 | |
| *** nosnos has joined #openstack-swift | 04:28 | |
| *** matsuhashi has joined #openstack-swift | 04:29 | |
| *** Anju1 has quit IRC | 04:31 | |
| *** ppai has joined #openstack-swift | 04:32 | |
| *** chandan_kumar has joined #openstack-swift | 04:32 | |
| *** Anju has joined #openstack-swift | 04:34 | |
| *** nshaikh has joined #openstack-swift | 05:31 | |
| *** nosnos_ has joined #openstack-swift | 05:51 | |
| *** nosnos has quit IRC | 05:54 | |
| *** matsuhashi has quit IRC | 05:59 | |
| *** matsuhashi has joined #openstack-swift | 06:00 | |
| *** sungju has joined #openstack-swift | 06:13 | |
| *** sungju has quit IRC | 06:14 | |
| *** nosnos has joined #openstack-swift | 06:32 | |
| *** matsuhashi has quit IRC | 06:32 | |
| *** nosnos_ has quit IRC | 06:32 | |
| *** psharma has joined #openstack-swift | 06:33 | |
| *** matsuhashi has joined #openstack-swift | 06:34 | |
| *** haomai___ has joined #openstack-swift | 06:44 | |
| *** haomaiwang has quit IRC | 06:48 | |
| *** haomaiwa_ has joined #openstack-swift | 07:21 | |
| *** haomai___ has quit IRC | 07:21 | |
| openstackgerrit | Kota Tsuyuzaki proposed a change to openstack/swift: fix DELETE request for versioned object https://review.openstack.org/89312 | 07:31 |
| openstackgerrit | Kota Tsuyuzaki proposed a change to openstack/swift: fix DELETE request for versioned object https://review.openstack.org/89312 | 07:36 |
| *** ekarlso- has joined #openstack-swift | 08:11 | |
| *** d89 has joined #openstack-swift | 08:32 | |
| openstackgerrit | Kota Tsuyuzaki proposed a change to openstack/swift: fix DELETE request for versioned object https://review.openstack.org/89312 | 08:40 |
| *** nshaikh has quit IRC | 08:45 | |
| *** praveenkumar has quit IRC | 08:49 | |
| *** chandan_kumar has quit IRC | 08:50 | |
| *** praveenkumar has joined #openstack-swift | 09:07 | |
| *** chandan_kumar has joined #openstack-swift | 09:08 | |
| *** bvandenh has joined #openstack-swift | 09:16 | |
| *** zhiyan is now known as zhiyan_ | 09:32 | |
| *** matsuhashi has quit IRC | 09:48 | |
| *** cheri has joined #openstack-swift | 09:50 | |
| *** matsuhashi has joined #openstack-swift | 09:54 | |
| *** haomaiwa_ has quit IRC | 09:55 | |
| *** haomaiwang has joined #openstack-swift | 09:56 | |
| *** baojg_ has joined #openstack-swift | 09:57 | |
| *** baojg_ has quit IRC | 10:18 | |
| *** baojg has joined #openstack-swift | 10:18 | |
| *** dmorita has joined #openstack-swift | 10:35 | |
| *** dmsimard has quit IRC | 10:39 | |
| *** chandan_kumar has quit IRC | 10:43 | |
| *** nshaikh has joined #openstack-swift | 10:45 | |
| *** dmsimard has joined #openstack-swift | 10:45 | |
| *** baojg_ has joined #openstack-swift | 10:46 | |
| *** baojg has quit IRC | 10:50 | |
| ekarlso- | hmm | 10:50 |
| *** chandan_kumar has joined #openstack-swift | 10:51 | |
| ekarlso- | http://paste.openstack.org/show/76500/ < anyone got a clue on this? | 10:51 |
| *** Trixboxer has joined #openstack-swift | 10:54 | |
| *** ppai has quit IRC | 11:00 | |
| *** cheri has quit IRC | 11:00 | |
| *** baojg_ has quit IRC | 11:03 | |
| *** baojg has joined #openstack-swift | 11:03 | |
| *** baojg has quit IRC | 11:07 | |
| *** cheri has joined #openstack-swift | 11:14 | |
| ekarlso- | noone knows ? | 11:16 |
| *** ppai has joined #openstack-swift | 11:16 | |
| *** matsuhashi has quit IRC | 11:22 | |
| *** matsuhashi has joined #openstack-swift | 11:30 | |
| ctennis | ekarlso-: http://172.16.10.5:8080:8080 is not a valid url | 11:40 |
| ekarlso- | I dunno where swift proxy get's that from | 11:42 |
| ekarlso- | the keystone endpoint is port :8080 | 11:44 |
| ekarlso- | ctennis: any way I can find out ? | 11:44 |
| ctennis | I assume in your proxy-server.conf file | 11:44 |
| ekarlso- | ctennis: what about that ? | 11:45 |
| ctennis | where in your config is 172.16.10.5 set? | 11:45 |
| ctennis | my guess is you have the port specified as part of the IP address | 11:45 |
| ekarlso- | ctennis: i'm using keystone | 11:46 |
| ctennis | ok, then keystone is returning the wrong auth url | 11:46 |
| ekarlso- | public endpoint is: http://172.16.10.5:8080/v1/AUTH_%(tenant_id)s | 11:46 |
| ctennis | you're going outside my realm of expertise, but it seems like something there is misconfigured | 11:48 |
| ctennis | you can use curl to do auth requests directly and get a better sense of what is being returned during authentication | 11:48 |
| *** matsuhashi has quit IRC | 11:48 | |
| *** matsuhashi has joined #openstack-swift | 11:49 | |
| ekarlso- | sooo | 11:49 |
| ekarlso- | http://paste.openstack.org/show/76507/ < this is what happens | 11:54 |
| *** tdasilva has joined #openstack-swift | 11:56 | |
| *** dmorita has quit IRC | 11:56 | |
| *** ppai has quit IRC | 11:59 | |
| ctennis | do the devices in your account server ring match the disk implementation? | 12:00 |
| ctennis | My guess is your ring file isn't correct | 12:00 |
| ekarlso- | i' | 12:01 |
| ekarlso- | i'll try to re-create | 12:01 |
| ekarlso- | http://paste.openstack.org/show/76508/ < this is the ring builder commands | 12:04 |
| ctennis | the port number should be different for the different servers | 12:06 |
| ctennis | and you need to make sure oyu have disks mounted under the directories corresponding to the names you provided 01 02 03 etc | 12:07 |
| ekarlso- | so for a all in one you can't run it on the same port ? | 12:08 |
| ekarlso- | account servers.. | 12:08 |
| ctennis | no each runs on its own port | 12:09 |
| *** ppai has joined #openstack-swift | 12:12 | |
| ekarlso- | oh, see I had a wrong thing in my script | 12:19 |
| *** Trixboxer has quit IRC | 12:21 | |
| *** cheri has quit IRC | 12:28 | |
| *** cheri has joined #openstack-swift | 12:30 | |
| *** matsuhashi has quit IRC | 12:42 | |
| *** matsuhashi has joined #openstack-swift | 12:44 | |
| *** zhiyan_ is now known as zhiyan | 12:49 | |
| *** matsuhashi has quit IRC | 12:51 | |
| *** cheri has quit IRC | 12:52 | |
| *** cheri has joined #openstack-swift | 13:04 | |
| *** matsuhashi has joined #openstack-swift | 13:06 | |
| *** ppai has quit IRC | 13:07 | |
| *** matsuhashi has quit IRC | 13:12 | |
| *** matsuhashi has joined #openstack-swift | 13:21 | |
| swills | clayg: sorry, missed your comment. but yes, i'm using the same spindle for all of it actually... background daemons, yes, everything should be running. | 13:23 |
| *** tongli has joined #openstack-swift | 13:24 | |
| *** cheri has quit IRC | 13:36 | |
| *** nshaikh has quit IRC | 13:52 | |
| *** nosnos has quit IRC | 13:53 | |
| *** tongli has quit IRC | 14:03 | |
| openstackgerrit | Yuan Zhou proposed a change to openstack/swift: Clean up swift-object-info/swift-get-nodes bin files https://review.openstack.org/86593 | 14:03 |
| *** chandan_kumar has quit IRC | 14:14 | |
| *** matsuhashi has quit IRC | 14:17 | |
| *** chandan_kumar has joined #openstack-swift | 14:35 | |
| *** Trixboxer has joined #openstack-swift | 14:50 | |
| *** mwstorer has joined #openstack-swift | 14:50 | |
| *** lpabon has joined #openstack-swift | 14:57 | |
| *** kevinc_ has joined #openstack-swift | 15:01 | |
| *** jergerber has joined #openstack-swift | 15:08 | |
| *** psharma has quit IRC | 15:15 | |
| *** chandan_kumar has quit IRC | 15:29 | |
| *** gyee has joined #openstack-swift | 15:30 | |
| *** kevinc_ has quit IRC | 15:30 | |
| *** kevinc_ has joined #openstack-swift | 15:33 | |
| notmyname | good morning world | 15:34 |
| Dieterbe | hi notmyname | 15:34 |
| creiht | for some measure of world | 15:42 |
| creiht | :) | 15:42 |
| *** d89 has quit IRC | 15:45 | |
| *** Dharmit has joined #openstack-swift | 15:46 | |
| *** piyush has joined #openstack-swift | 15:55 | |
| *** piyush has quit IRC | 15:56 | |
| *** piyush has joined #openstack-swift | 15:56 | |
| *** zhiyan is now known as zhiyan_ | 15:57 | |
| *** piyush has left #openstack-swift | 15:57 | |
| *** piyush has joined #openstack-swift | 15:58 | |
| *** jamie_h has joined #openstack-swift | 16:10 | |
| openstackgerrit | Joel Wright proposed a change to openstack/python-swiftclient: Add importable SwiftService incorporating shell.py logic https://review.openstack.org/85453 | 16:13 |
| *** jamie_h has quit IRC | 16:16 | |
| notmyname | I'm going to start reviewing and scheduling the 22 proposed swift sessions at the summit today. we have 8 slots, so if you submitted something, it's better than 50/50 that it will get rejected. but we have the swift pod for the week to use as well | 16:32 |
| *** piyush has quit IRC | 16:33 | |
| *** jamie_h has joined #openstack-swift | 16:38 | |
| *** creiht has quit IRC | 16:39 | |
| *** creiht has joined #openstack-swift | 16:40 | |
| *** ChanServ sets mode: +v creiht | 16:40 | |
| *** piyush has joined #openstack-swift | 16:41 | |
| *** piyush1 has joined #openstack-swift | 16:42 | |
| *** piyush has quit IRC | 16:45 | |
| *** shri has joined #openstack-swift | 16:56 | |
| swills | creiht: ping? | 17:00 |
| swills | creiht: only bothering you because we had chatted about my usage of zfs... i'm running into test issues | 17:00 |
| swills | http://meatwad.mouf.net/~swills/swift_tests.txt | 17:00 |
| swills | can't figure out what could be causing this, but it's consistent | 17:00 |
| swills | and I get stuff like | 17:01 |
| swills | Apr 19 03:04:01 10.0.1.143 proxy-server: Object PUT returning 503, 0/2 required connections (txn: txb063725b1e81425bb1272-005351e78c) (client_ip: 10.0.1.143) | 17:01 |
| *** catintheroof has joined #openstack-swift | 17:01 | |
| *** catintheroof has left #openstack-swift | 17:02 | |
| *** Dharmit has quit IRC | 17:14 | |
| *** praveenkumar has quit IRC | 17:15 | |
| *** esmute has left #openstack-swift | 17:20 | |
| *** chandan_kumar has joined #openstack-swift | 17:21 | |
| *** tongli has joined #openstack-swift | 17:58 | |
| *** kevinc_ has quit IRC | 18:02 | |
| *** kevinc_ has joined #openstack-swift | 18:04 | |
| *** zaitcev has joined #openstack-swift | 18:11 | |
| *** ChanServ sets mode: +v zaitcev | 18:11 | |
| creiht | swills: sounds like either the proxy can't connect to some of your object servers, or the ring is configured wrong (like wrong ip or port, etc.) | 18:20 |
| swills | creiht: hmm, well, the thing is, i think some of the object servers choked | 18:21 |
| creiht | could be | 18:21 |
| swills | creiht: with a disk IO exception | 18:21 |
| swills | but i can't figure out why | 18:21 |
| creiht | if they become unresponsive then you could get a similar error | 18:21 |
| swills | because there's no disk issue | 18:21 |
| swills | i mean, i see uncaught exceptions and tracked it down, but all i see is disk io exception writing to the bdb file | 18:22 |
| creiht | hrm | 18:22 |
| swills | made me wonder if i had a issue with my bdb stuff | 18:22 |
| swills | https://github.com/openstack/swift/blob/master/swift/container/backend.py#L350-L356 | 18:23 |
| swills | so basically what seems to happen is there's an uncaught exception there, a disk IO error exception | 18:24 |
| swills | that kills the container updater | 18:24 |
| swills | then the tests go off the rails | 18:24 |
| creiht | weird | 18:24 |
| swills | yeah | 18:24 |
| swills | not sure how to proceed past that | 18:24 |
| swills | or how to look into it, really | 18:24 |
| swills | http://paste.mouf.net/pastes/a5hpnrjpfgk343sr3gcg6vpn | 18:24 |
| swills | i logged the exceptions and the file it was having an issue with and that's what i got | 18:25 |
| swills | of course, i've moved all that out of the way and started over from scratch since logging that | 18:25 |
| creiht | maybe double check file permissions? | 18:25 |
| swills | but i do still have the files around and could test them | 18:25 |
| swills | file permissions are fine | 18:25 |
| creiht | so looks like that error is from the sqlite libraries | 18:27 |
| swills | ok | 18:27 |
| zaitcev | reproduce and strace | 18:27 |
| swills | strace: Command not found. | 18:27 |
| zaitcev | reproduce, isolate, strace | 18:27 |
| zaitcev | oh, right. BSD | 18:27 |
| creiht | lol | 18:27 |
| zaitcev | Maybe just "trace". On Solaris it was called "truss". | 18:28 |
| creiht | swills: remind me again how you set up the storage for each node | 18:28 |
| swills | truss, ktrace, dtrace | 18:28 |
| creiht | you put each one in a jail right? | 18:28 |
| swills | yep | 18:28 |
| creiht | could jail perms be messing with things? | 18:28 |
| swills | they're all in separate ZFS datasets | 18:28 |
| swills | nope | 18:28 |
| zaitcev | dtrace is probably too much | 18:28 |
| swills | zaitcev: i agree | 18:29 |
| creiht | swills: some of my googling indicates that the error can come up when read/write lock that it is trying to get isn't supported | 18:29 |
| swills | creiht: oh? | 18:29 |
| creiht | there are a lot of things that can cause that error actually | 18:29 |
| creiht | but that's one that has popped up so far | 18:29 |
| swills | so i can reproduce easily... | 18:30 |
| swills | and i can ktrace it, see what the syscalls are, etc. | 18:30 |
| creiht | cool | 18:30 |
| swills | which part do i want to ktrace? just the container updater, i think? | 18:30 |
| swills | container server | 18:31 |
| creiht | I would start with the container server | 18:31 |
| swills | k | 18:31 |
| swills | damnit, meeting | 18:31 |
| swills | brb | 18:31 |
| zaitcev | there's going to be a HUEG trace, but you just set string size to 1000 and search for "UNCAUGHT", then look at a few syscalls prior | 18:32 |
| *** byeager has joined #openstack-swift | 18:33 | |
| creiht | swills: it would also be interestnig to try a similar query through the sqlite3 command line on the same erroring db to see if you get the same error | 18:35 |
| creiht | swills: and the disks are local right? | 18:36 |
| *** kevinc_ has quit IRC | 18:38 | |
| *** jamie_h has quit IRC | 18:41 | |
| swills | creiht: yeah | 18:43 |
| swills | zaitcev: yes... | 18:44 |
| swills | creiht: let me see if i can reproduce the error with the exception catching in place... | 18:47 |
| *** piyush1 has quit IRC | 18:53 | |
| *** byeager has quit IRC | 18:54 | |
| *** byeager has joined #openstack-swift | 18:54 | |
| *** chandan_kumar has quit IRC | 18:55 | |
| *** chandan_kumar has joined #openstack-swift | 18:56 | |
| swills | interesting, it's happening even without running the test | 18:57 |
| anticw_ | wrt to conditional requests, i can do if-none-match, if-match and if-modified-since on PUT | 19:05 |
| anticw_ | are there any others? | 19:05 |
| anticw_ | doesn't look like there is explicit support for if-range (not sure if would be useful) | 19:06 |
| anticw_ | +if-unmodified-since | 19:09 |
| *** piyush has joined #openstack-swift | 19:17 | |
| *** piyush1 has joined #openstack-swift | 19:18 | |
| *** piyush has quit IRC | 19:21 | |
| zaitcev | swills: run auditors in "once" mode to isolate | 19:32 |
| *** chandan_kumar has quit IRC | 19:34 | |
| swills | zaitcev: oh, thanks | 19:36 |
| swills | heh | 19:40 |
| swills | drive audit immediately returns: | 19:40 |
| swills | IOError: [Errno 2] No such file or directory: '/proc/mounts' | 19:40 |
| swills | You are correct, sir! | 19:40 |
| *** kevinc_ has joined #openstack-swift | 19:47 | |
| swills | disk I/O error | 19:49 |
| swills | /srv/node/1/containers/748/da9/bb0c829be2ab0e572dcebfbd360b6da9/bb0c829be2ab0e572dcebfbd360b6da9.db | 19:49 |
| swills | put_timestamp: 1398108433.57862 | 19:49 |
| swills | delete_timestamp: 1398108434.85569 | 19:49 |
| swills | object_count: 0 | 19:49 |
| swills | bytes_used: 0 | 19:49 |
| swills | seems legit... | 19:52 |
| swills | i think? | 19:52 |
| swills | oh wow, this thing is creating and deleting lots of files in /tmp | 19:57 |
| *** mwstorer has quit IRC | 19:59 | |
| swills | creiht: those values look OK? like something that should be rejected or no? | 20:00 |
| *** tongli has quit IRC | 20:09 | |
| *** Trixboxer has quit IRC | 20:11 | |
| creiht | swills: hey sorry, I'm a bit distracted with other things today | 20:20 |
| creiht | the info seems legit, but weird that you are getting the I/O error | 20:20 |
| swills | creiht: no worries | 20:21 |
| swills | thx | 20:21 |
| swills | i'm running the ktrace now | 20:21 |
| swills | it seemed like the error was in the container updater | 20:21 |
| swills | but ktracing just that hasn't found anytihng wrong | 20:21 |
| swills | so now i'm just tracing everything | 20:21 |
| swills | so it's extra huge | 20:21 |
| *** byeager has quit IRC | 20:34 | |
| *** byeager has joined #openstack-swift | 20:40 | |
| swills | ok, getting closer | 20:54 |
| swills | 21 #012 File "/usr/local/lib/python2.7/site-packages/swift/common/db.py", line 127, in <lambda> | 20:54 |
| swills | 22 #012 self, *args, **kwargs)) | 20:54 |
| swills | 23 #012OperationalError: disk I/O error\0" | 20:54 |
| swills | hmm, i see some references to 127.0.0.1 in here, which doesn't work in a jail... wonder if that's causing the issue... | 20:57 |
| glange | whatcha talkin' 'bout swillis? | 20:58 |
| swills | -_- | 20:59 |
| swills | zaitcev: http://paste.mouf.net/pastes/flpjii4usaa79kiruktovcj5 this give you any hints? | 21:01 |
| *** mwstorer has joined #openstack-swift | 21:03 | |
| zaitcev | swills: it doesn't. all the connects to localhost look inexplicable, like attempts to determine presence of IPv6 gone horribly wrong. Anything above that segment? | 21:12 |
| swills | zaitcev: tons! :) | 21:17 |
| swills | zaitcev: i think the localhost stuff is related to bdb... | 21:17 |
| zaitcev | you keep saying that word but we actually use SQLite, not BDB | 21:18 |
| swills | duh, right | 21:18 |
| *** openstackstatus has quit IRC | 21:35 | |
| *** openstackstatus has joined #openstack-swift | 21:37 | |
| *** lpabon has quit IRC | 21:41 | |
| *** Diddi has quit IRC | 21:42 | |
| *** anticw_ is now known as anticw | 21:43 | |
| *** anticw has quit IRC | 21:43 | |
| *** anticw has joined #openstack-swift | 21:44 | |
| *** anticw has quit IRC | 21:49 | |
| *** anticw has joined #openstack-swift | 21:54 | |
| *** piyush1 has quit IRC | 22:00 | |
| *** jergerber has quit IRC | 22:08 | |
| *** kevinc_ has quit IRC | 22:11 | |
| openstackgerrit | David Goetz proposed a change to openstack/swift: setting request_node_count doesn't work https://review.openstack.org/89433 | 22:20 |
| *** kevinc_ has joined #openstack-swift | 22:21 | |
| *** byeager has quit IRC | 22:32 | |
| *** wer has quit IRC | 22:34 | |
| *** wer has joined #openstack-swift | 22:35 | |
| *** Diddi has joined #openstack-swift | 22:37 | |
| notmyname | hmm...somewhat late in the day to bring this up.. | 22:49 |
| notmyname | I'm not going to be available to run the swift team meeting on wednesday. options are to not have it or for someone else to run it | 22:50 |
| *** h6w has joined #openstack-swift | 23:28 | |
| zaitcev | maybe just skip it | 23:30 |
| zaitcev | we're relaxing past the release, so it's just going to be like "so how are storage policies coming up" "we're working on it, but reconciler yada yada yada" "okay" | 23:31 |
| *** occupant has joined #openstack-swift | 23:32 | |
| zaitcev | I'm bitter that https://review.openstack.org/85909 is stuck even with +68,-25 diffstat. Paul looked at it, which is great, but not enough. | 23:34 |
| *** openstackgerrit has quit IRC | 23:50 | |
| *** openstackgerrit has joined #openstack-swift | 23:50 | |
| *** kevinc_ has quit IRC | 23:53 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!