| *** mwstorer has quit IRC | 00:09 | |
| *** matsuhashi has joined #openstack-swift | 00:19 | |
| *** shri has quit IRC | 00:44 | |
| *** h6w has quit IRC | 01:18 | |
| *** byeager has joined #openstack-swift | 01:23 | |
| *** saschpe has quit IRC | 01:32 | |
| *** nosnos has joined #openstack-swift | 01:36 | |
| *** saschpe has joined #openstack-swift | 01:37 | |
| *** byeager has quit IRC | 01:40 | |
| swills | zaitcev: would you be willing to look at my ktrace output to see if you can find what's wrong? | 01:43 |
|---|---|---|
| swills | (ktrace output is basically the same as strace) | 01:44 |
| swills | if so, see http://meatwad.mouf.net/~swills/swift.ktrace.txt.gz | 01:47 |
| zaitcev | swills: if my editor can survive 200 meg | 01:50 |
| swills | zaitcev: yeah... use less or something... | 01:52 |
| swills | all the googling i have done on the exception talk about disk errors | 01:55 |
| swills | but given i'm using zfs and there are no errors reported there and using sqlite from commadn line has no problem with the file, i'm finding that hard to believe | 01:56 |
| swills | oh | 01:56 |
| swills | def execute(self, *args, **kwargs): | 01:56 |
| swills | return _db_timeout( | 01:56 |
| swills | self.timeout, self.db_file, lambda: sqlite3.Cursor.execute( | 01:56 |
| swills | self, *args, **kwargs)) | 01:56 |
| swills | self.timeout is mentioned in the excep;tion | 01:57 |
| swills | i wonder if it's simply timing out because things are too slow? | 01:57 |
| swills | http://paste.mouf.net/pastes/pi1dakl33decekf2eoe29ioh | 01:57 |
| swills | is the full trace | 01:57 |
| swills | http://paste.mouf.net/pastes/l47l9ipeetqf2a7lvusadm99 | 01:59 |
| swills | easier to read | 01:59 |
| swills | #012 File "/usr/local/lib/python2.7/site-packages/swift/common/db.py", line 64, in _db_timeout | 01:59 |
| *** baojg has joined #openstack-swift | 02:00 | |
| *** dmorita has joined #openstack-swift | 02:05 | |
| swills | just ran the tests that come with my python sqlite stuff and it seems mostly OK except something related to hooks | 02:05 |
| swills | i'm going to rebuild my python and sqlite and python-sqlite stuff and see if that changes anything... | 02:13 |
| occupant | hmph, my proxy is grabbing way more data from the object servers than it's actually serving | 02:15 |
| zaitcev | swills: okay, it's too much to grasp so quickly, but it appears that one thread, 60074, tries to lock the db (at line 8091316), but some other tread closes the file descriptor it was using. | 02:25 |
| zaitcev | swills: if would be grand if you found who closes it and why | 02:25 |
| zaitcev | swills: And no, it's not timing out. Your traceback reported OperationalError, so the exception was raised by sqlite3, and not the eventlet.Timeout that wraps it with with: op. | 02:27 |
| *** zhiyan_ is now known as zhiyan | 02:30 | |
| *** praveenkumar has joined #openstack-swift | 02:33 | |
| swills | zaitcev: oh! wow, ok | 02:34 |
| swills | zaitcev: what file is that? | 02:34 |
| zaitcev | swills: swift.ktrace.txt that I thought you asked me to examine. | 02:35 |
| swills | i mean, what file is closed | 02:35 |
| swills | there are multiple db files, right? | 02:35 |
| swills | i did see that bit you were talking about but didn't understand it so well | 02:35 |
| zaitcev | I am not knowledgeable about threading model in BSD. In Linux, threads do not share fd space unless they pass special flags to clone(). | 02:36 |
| zaitcev | Perhaps we somehow baked that assumption in | 02:36 |
| swills | http://stackoverflow.com/questions/6296055/serializing-sqlite3-in-python | 02:36 |
| swills | "The Python SQLite module is not threadsafe." | 02:36 |
| swills | or so it claims... | 02:37 |
| zaitcev | yeah | 02:37 |
| swills | i wonder if it's this file /srv/node/1/containers/776/107/c20e998d06390c9dc660790b12c4b107/c20e998d06390c9dc660790b12c4b107.db | 02:37 |
| zaitcev | But in our case threads are greenthreads that eventlet creates. | 02:37 |
| zaitcev | I'm thinking line 8089675 is the nearest same-pid open | 02:38 |
| swills | i wonder about line 8089750, does the lack of wal cause the issue? | 02:40 |
| *** jbs123 has joined #openstack-swift | 02:41 | |
| zaitcev | umm no | 02:41 |
| zaitcev | it's EBADF | 02:41 |
| zaitcev | obviously something was there and then it was closed, no matter what's the name | 02:41 |
| swills | oh, ok, i thought maybe i didn't have wal enabled in my build | 02:42 |
| *** jbs123 has left #openstack-swift | 02:44 | |
| swills | ok, rebuild and reinstalled python and sqlite and python-sqlite stuff | 02:53 |
| swills | nope, still happening | 02:55 |
| swills | so i have thread support enabled in both sqlite and python | 02:56 |
| swills | got another trace | 03:02 |
| swills | oh, i think i found the process that closed it | 03:04 |
| swills | 08d2ef729088d86e8612c1ce2b2323be is the file... | 03:05 |
| swills | so i think it's the container server that's closing the file | 03:06 |
| swills | out from under the container updater | 03:06 |
| swills | zaitcev: http://meatwad.mouf.net/~swills/swift_1.ktrace.txt.gz | 03:09 |
| zaitcev | how can updater and server share the pid space? | 03:10 |
| zaitcev | I don't think it's possible. | 03:10 |
| zaitcev | er, fd space | 03:10 |
| swills | look in that file for 08d2ef729088d86e8612c1ce2b2323be | 03:11 |
| swills | don't worry, i'm sure i'm misunderstanding. :) | 03:11 |
| openstackgerrit | Pete Zaitcev proposed a change to openstack/swift: Plumb insecure flag to functests https://review.openstack.org/88438 | 03:13 |
| *** zaitcev has quit IRC | 03:19 | |
| *** matsuhashi has quit IRC | 03:20 | |
| *** nosnos has quit IRC | 03:32 | |
| *** chandan_kumar has joined #openstack-swift | 04:02 | |
| *** chandan_kumar has quit IRC | 04:10 | |
| *** ppai has joined #openstack-swift | 04:11 | |
| *** mrsnivvel has joined #openstack-swift | 04:18 | |
| *** matsuhashi has joined #openstack-swift | 04:28 | |
| *** nosnos has joined #openstack-swift | 04:37 | |
| *** baojg has quit IRC | 04:48 | |
| *** baojg has joined #openstack-swift | 04:48 | |
| *** haomaiw__ has joined #openstack-swift | 04:49 | |
| *** haomaiwang has quit IRC | 04:52 | |
| *** matsuhas_ has joined #openstack-swift | 05:02 | |
| *** matsuhashi has quit IRC | 05:05 | |
| *** baojg_ has joined #openstack-swift | 05:10 | |
| *** baojg has quit IRC | 05:14 | |
| *** nshaikh has joined #openstack-swift | 05:55 | |
| *** haomaiw__ has quit IRC | 05:56 | |
| *** haomaiwa_ has joined #openstack-swift | 05:56 | |
| *** gyee has quit IRC | 05:56 | |
| *** haomaiw__ has joined #openstack-swift | 05:57 | |
| *** taras___ has quit IRC | 05:58 | |
| *** haomaiwa_ has quit IRC | 06:01 | |
| *** chandan_kumar has joined #openstack-swift | 06:02 | |
| *** matsuhas_ has quit IRC | 06:02 | |
| *** madhuri has quit IRC | 06:04 | |
| *** matsuhashi has joined #openstack-swift | 06:10 | |
| *** early has quit IRC | 06:14 | |
| *** early has joined #openstack-swift | 06:17 | |
| *** psharma has joined #openstack-swift | 06:18 | |
| openstackgerrit | Takashi Kajinami proposed a change to openstack/swift: Fix error log of proxy-server when cache middleware is disabled https://review.openstack.org/89488 | 06:43 |
| *** chandan_kumar has quit IRC | 06:49 | |
| *** haomaiw__ has quit IRC | 06:50 | |
| *** haomaiwang has joined #openstack-swift | 06:51 | |
| *** chandan_kumar has joined #openstack-swift | 07:06 | |
| *** cheri has joined #openstack-swift | 07:09 | |
| *** Honghui has joined #openstack-swift | 07:16 | |
| *** SkyRocknRoll has joined #openstack-swift | 07:41 | |
| *** SkyRocknRoll has joined #openstack-swift | 07:41 | |
| *** mmcardle has joined #openstack-swift | 07:42 | |
| *** haomai___ has joined #openstack-swift | 07:50 | |
| *** nacim has joined #openstack-swift | 07:50 | |
| *** haomaiwang has quit IRC | 07:53 | |
| *** joeljwright has joined #openstack-swift | 08:04 | |
| *** jamie_h has joined #openstack-swift | 08:28 | |
| *** Dharmit has joined #openstack-swift | 08:30 | |
| *** haomai___ has quit IRC | 08:34 | |
| *** haomaiwang has joined #openstack-swift | 08:35 | |
| *** haomaiw__ has joined #openstack-swift | 08:45 | |
| *** haomaiwang has quit IRC | 08:45 | |
| *** jamie_h has quit IRC | 09:11 | |
| *** jamie_h has joined #openstack-swift | 09:12 | |
| *** chandan_kumar has quit IRC | 09:15 | |
| *** foexle has joined #openstack-swift | 09:28 | |
| *** chandan_kumar has joined #openstack-swift | 09:28 | |
| *** mkollaro has joined #openstack-swift | 09:31 | |
| *** jamie_h has quit IRC | 09:36 | |
| *** jamie_h has joined #openstack-swift | 09:36 | |
| *** jamie_h has quit IRC | 09:42 | |
| *** jamie_h has joined #openstack-swift | 09:45 | |
| *** jamie_h has quit IRC | 09:52 | |
| *** jamie_h has joined #openstack-swift | 09:53 | |
| *** chandan_kumar is now known as chandankumar | 09:55 | |
| *** chandankumar is now known as chandan_kumar | 09:56 | |
| *** dmorita has quit IRC | 10:35 | |
| *** Honghui has quit IRC | 10:38 | |
| openstackgerrit | Yuan Zhou proposed a change to openstack/swift: Fix delete versioning objects when previous is expired https://review.openstack.org/88204 | 10:48 |
| *** Trixboxer has joined #openstack-swift | 10:56 | |
| *** jamie_h has quit IRC | 10:56 | |
| *** praveenkumar has quit IRC | 10:59 | |
| *** jamie_h has joined #openstack-swift | 11:00 | |
| *** lpabon has joined #openstack-swift | 11:04 | |
| *** baojg_ has quit IRC | 11:10 | |
| *** baojg has joined #openstack-swift | 11:11 | |
| *** baojg has quit IRC | 11:16 | |
| *** lpabon has quit IRC | 11:27 | |
| *** mlipchuk has joined #openstack-swift | 11:30 | |
| *** praveenkumar has joined #openstack-swift | 11:39 | |
| *** mkollaro has quit IRC | 12:02 | |
| *** mkollaro has joined #openstack-swift | 12:03 | |
| *** mmcardle has quit IRC | 12:04 | |
| *** mkollaro1 has joined #openstack-swift | 12:05 | |
| *** mkollaro has quit IRC | 12:05 | |
| *** acorwin has quit IRC | 12:07 | |
| *** acorwin has joined #openstack-swift | 12:09 | |
| *** lpabon has joined #openstack-swift | 12:29 | |
| openstackgerrit | paul luse proposed a change to openstack/swift: Add Storage Policy Support to Container Sync https://review.openstack.org/86469 | 12:30 |
| *** bada has joined #openstack-swift | 12:32 | |
| *** mmcardle has joined #openstack-swift | 12:33 | |
| *** ppai has quit IRC | 12:39 | |
| *** matsuhashi has quit IRC | 12:43 | |
| *** matsuhashi has joined #openstack-swift | 12:43 | |
| *** nshaikh has quit IRC | 12:44 | |
| *** SkyRocknRoll has quit IRC | 12:45 | |
| *** matsuhashi has quit IRC | 12:48 | |
| *** nosnos has quit IRC | 12:48 | |
| *** nosnos has joined #openstack-swift | 12:51 | |
| openstackgerrit | paul luse proposed a change to openstack/swift: Add Storage Policy Support to Recon Middleware https://review.openstack.org/87387 | 12:52 |
| *** nosnos has quit IRC | 12:52 | |
| *** mkollaro has joined #openstack-swift | 13:00 | |
| *** mkollaro1 has quit IRC | 13:03 | |
| *** mrsnivvel has quit IRC | 13:17 | |
| openstackgerrit | Nassim Babaci proposed a change to openstack/swift: Add policy engine to keystoneauth https://review.openstack.org/89568 | 13:21 |
| openstackgerrit | Nassim Babaci proposed a change to openstack/swift: Add policy engine to keystoneauth https://review.openstack.org/89568 | 13:26 |
| openstackgerrit | paul luse proposed a change to openstack/swift: Add Storage Policy Documentation https://review.openstack.org/85824 | 13:26 |
| openstackgerrit | Nassim Babaci proposed a change to openstack/swift: Add policy engine to keystoneauth https://review.openstack.org/89568 | 13:28 |
| openstackgerrit | Nassim Babaci proposed a change to openstack/swift: Add policy engine to keystoneauth https://review.openstack.org/89568 | 13:29 |
| *** nacim_ has joined #openstack-swift | 13:40 | |
| *** nacim has quit IRC | 13:40 | |
| *** Honghui has joined #openstack-swift | 13:57 | |
| *** byeager has joined #openstack-swift | 14:02 | |
| *** psharma has quit IRC | 14:08 | |
| *** cheri has quit IRC | 14:08 | |
| *** chandan_kumar has quit IRC | 14:17 | |
| *** changbl has quit IRC | 14:18 | |
| *** mlipchuk has quit IRC | 14:39 | |
| *** jergerber has joined #openstack-swift | 14:49 | |
| *** mlipchuk has joined #openstack-swift | 14:54 | |
| *** Honghui has quit IRC | 14:57 | |
| openstackgerrit | gholt proposed a change to openstack/swift: TempURL: Fixed bug with \r or \n in disposition. https://review.openstack.org/86691 | 15:00 |
| *** byeager has quit IRC | 15:03 | |
| *** byeager has joined #openstack-swift | 15:04 | |
| *** lpabon has quit IRC | 15:07 | |
| *** nacim_ has quit IRC | 15:10 | |
| openstackgerrit | paul luse proposed a change to openstack/swift: Add Storage Policy Support to Recon Middleware https://review.openstack.org/87387 | 15:10 |
| swills | creiht: mind if i bother you? :) | 15:11 |
| *** kevinc_ has joined #openstack-swift | 15:15 | |
| creiht | swills: what's up? | 15:15 |
| swills | creiht: still trying to track down this issue... did the syscall trace and it seems like it's closing the sqlite file then trying to write to it or something | 15:16 |
| swills | http://paste.mouf.net/pastes/l47l9ipeetqf2a7lvusadm99 is the exception | 15:16 |
| swills | http://paste.mouf.net/pastes/l47l9ipeetqf2a7lvusadm99/raw | 15:16 |
| creiht | hrm | 15:24 |
| swills | upgrading to 1.13.1 now | 15:25 |
| swills | the trace is pretty big: http://meatwad.mouf.net/~swills/swift_1.ktrace.txt.gz | 15:25 |
| swills | if you download it and look for references to 08d2ef729088d86e8612c1ce2b2323be you'll see what's going on I think | 15:27 |
| swills | i don't fully understand what's going on personally | 15:27 |
| *** tdasilva has left #openstack-swift | 15:28 | |
| swills | creiht: thoughts? i know it's hard to decipher... does it help to know i have threading enabled in both sqlite and python? | 15:30 |
| creiht | I don't think that should matter | 15:31 |
| swills | k | 15:31 |
| *** mwstorer has joined #openstack-swift | 15:33 | |
| *** pberis has joined #openstack-swift | 15:36 | |
| swills | creiht: seen anything like this before? | 15:36 |
| creiht | no | 15:36 |
| swills | think the idea of it being closed when it's trying to write to it makes any sense? | 15:37 |
| swills | what version of sqlite do you guys typically use? | 15:37 |
| *** piyush has joined #openstack-swift | 15:37 | |
| swills | i'm using 3.8.4.2 | 15:38 |
| creiht | ubuntu precies has 3.7.9 | 15:38 |
| creiht | I wouldn't think version differences would matter | 15:38 |
| *** nacim_ has joined #openstack-swift | 15:38 | |
| creiht | sqlite is usually pretty stable | 15:38 |
| creiht | I wish I was better at reading traces | 15:39 |
| swills | it's not easy | 15:40 |
| *** bach has joined #openstack-swift | 15:43 | |
| creiht | swills: ok this is weird | 15:43 |
| creiht | swills: I found the I/O error traceback | 15:44 |
| creiht | at 99573 | 15:44 |
| swills | ok | 15:44 |
| creiht | right above it there are quite a few | 15:44 |
| creiht | 98874 python2.7 RET accept -1 errno 35 Resource temporarily unavailable | 15:44 |
| creiht | not sure if that is related or not but seems suspect | 15:45 |
| swills | hmm, i do see some of this logged: | 15:45 |
| swills | sonewconn: pcb 0xfffff801006be7a8: Listen queue overflow: 193 already in queue awaiting acceptance (36 occurrences) | 15:45 |
| *** pberis has quit IRC | 15:45 | |
| *** pberis has joined #openstack-swift | 15:46 | |
| swills | kern.ipc.somaxconn: 128 | 15:46 |
| swills | what's the linux version again? | 15:46 |
| creiht | which linux version? | 15:47 |
| swills | whatever you guys are using? | 15:47 |
| *** kevinc_ has quit IRC | 15:49 | |
| creiht | oh | 15:49 |
| creiht | well I'm testing on ubuntu precise right now | 15:49 |
| creiht | using kernel 3.2.0 | 15:50 |
| *** zhiyan is now known as zhiyan_ | 15:50 | |
| swills | i wonder what resource it is that's unavailable | 15:52 |
| creiht | hehe disk IO error pops up a lot in that trace | 15:53 |
| swills | yes | 15:54 |
| *** Honghui has joined #openstack-swift | 15:54 | |
| swills | but that's just sqlite's way of saying *something* went wrong, I think | 15:54 |
| swills | upgrading to 1.13.1 didn't change it, it seems, nor did setting kern.ipc.somaxconn=65535 | 15:54 |
| creiht | yeah I found the first occurence of the error, and I'm not seeing the errno 35 around there | 15:55 |
| *** kevinc_ has joined #openstack-swift | 15:56 | |
| swills | hmmm, /etc/swift seems to be hard coded in a lot of places... | 15:57 |
| *** joeljwright has quit IRC | 15:57 | |
| swills | interestingly, i'm running the functests again and i'm seeing those exceptions but it's not causing test failures | 16:02 |
| creiht | heh | 16:02 |
| creiht | swills: are you still using a multiserver config? or did you ever switch to an saio? | 16:02 |
| swills | multiserver | 16:03 |
| creiht | I wonder if it would be better to try to set up just an saio to test with first | 16:03 |
| *** mlipchuk has quit IRC | 16:03 | |
| *** mlipchuk1 has joined #openstack-swift | 16:03 | |
| creiht | it might remove a lot of variables | 16:03 |
| swills | ok, but if i do that and i don't see the error, what does that prove? how does that help find the issue in this setup which is more like what the real production setup will be? | 16:04 |
| openstackgerrit | yummy.bian proposed a change to openstack/swift: Fix redundant add operation of set in ring https://review.openstack.org/89630 | 16:04 |
| creiht | well it would remove the jail variable | 16:05 |
| creiht | just to make sure it isn't limiting something | 16:05 |
| creiht | I'm just trying to start simpler | 16:05 |
| creiht | because otherwise, I'm kind of at a loss | 16:05 |
| creiht | sorry | 16:05 |
| swills | *nod* no worries | 16:06 |
| *** mlipchuk1 has quit IRC | 16:06 | |
| swills | perhaps i'll just give up and use Linux for this | 16:06 |
| creiht | hah | 16:06 |
| creiht | well that would make certain things easier :) | 16:06 |
| swills | thing is | 16:06 |
| swills | i don't trust xfs | 16:06 |
| swills | i really wanted to use zfs | 16:06 |
| swills | using the jails was just an easy way to test that everything worked OK on zfs | 16:07 |
| Honghui | swills, you means zfsonlinux or zfs-fuse? | 16:07 |
| swills | Honghui: definitely not | 16:08 |
| Honghui | zfs on BSD or Solaris? | 16:08 |
| swills | FreeBSD, yes | 16:08 |
| creiht | swills: oh I understand | 16:09 |
| creiht | I would like for everything to work on freebsd | 16:09 |
| swills | and don't take this the wrong way, but the failure of swift on FreeBSD says more about swift than FreeBSD, for me. :) | 16:09 |
| creiht | just most of us aren't really freebsd experts | 16:09 |
| creiht | awww | 16:09 |
| creiht | swills: well for what its worth, we wrote swift to work with the tools we had | 16:10 |
| swills | i wonder if a different version of python would help... is 2.7 what you guys use? | 16:10 |
| creiht | I doubt that would change anything | 16:11 |
| creiht | and yes we use both 2.6 and 2.7 | 16:11 |
| creiht | most use 2.7 | 16:11 |
| Honghui | We run python 2.7, swift 1.10. | 16:11 |
| swills | k | 16:11 |
| *** chandan_kumar has joined #openstack-swift | 16:11 | |
| creiht | I wish the softlayer devs were around to share their experiences | 16:11 |
| creiht | their the only ones that have had freebsd experience | 16:11 |
| swills | that would be cool | 16:12 |
| creiht | but they ended up going with linux just because it was easier | 16:12 |
| swills | perhaps they've written something up and posted it somewhere? | 16:12 |
| creiht | I don't recall anything | 16:12 |
| swills | also, fwiw, https://www.freebsd.org/news/status/report-2014-01-2014-03.html#FreeBSD-Host-Support-for-OpenStack-and-OpenContrail | 16:13 |
| swills | but that seems more focused on nova and such | 16:13 |
| creiht | yeah | 16:13 |
| *** Honghui has quit IRC | 16:14 | |
| creiht | notmyname: you need to add freebsd to your test cluster ;) | 16:14 |
| swills | i'd be happy to help with that | 16:15 |
| creiht | swills: but back to the saio idea, the main reason for that is that it is about the simplest way to setup swift | 16:16 |
| creiht | and test it | 16:16 |
| creiht | if it works fine there, then we can start expanding | 16:16 |
| creiht | to isolate where problems get introduced | 16:16 |
| swills | ok, i'm not 100% sure what you mean by saio | 16:16 |
| creiht | oh sorry | 16:16 |
| creiht | swift all in one | 16:16 |
| swills | right | 16:16 |
| swills | but what does that mean exactly? | 16:16 |
| creiht | http://docs.openstack.org/developer/swift/development_saio.html | 16:16 |
| swills | does that mean running it in the ubuntu VM? | 16:17 |
| creiht | no | 16:17 |
| swills | but that page you just linked says that | 16:17 |
| creiht | you would have to adapt the instruction for freebsd | 16:17 |
| creiht | the idea being that you run all the services locally on different ports | 16:17 |
| swills | so that would mean, what? just taking the 6 jails i have now and reconfiguring 1 so it only knows about itself? | 16:17 |
| creiht | well Ideally, I would like to start with running outside of a jail to start with | 16:18 |
| swills | running multiple copies of swift in the 1 jail? how is that different from 6 jails? | 16:18 |
| swills | i can't imagine running outside the jail would change much | 16:19 |
| *** changbl has joined #openstack-swift | 16:19 | |
| swills | usually if jails cause any issues you see it immediately, like if you try to run postgresql in a jail and forgot to deal with the shared memory stuff | 16:20 |
| *** chandan_kumar has quit IRC | 16:20 | |
| creiht | Yeah the thing is, I'm kinda out of ideas | 16:22 |
| *** mlipchuk has joined #openstack-swift | 16:22 | |
| creiht | and when that happens, all I can think of is trying to remove unknowns | 16:22 |
| creiht | in this situation there are a lot of unknowns for me | 16:22 |
| creiht | maybe someone else will chime in with a better idea | 16:23 |
| creiht | it is pass time for me to go to lunch now... will be back later | 16:24 |
| *** nacim_ has quit IRC | 16:28 | |
| *** joeljwright has joined #openstack-swift | 16:37 | |
| *** joeljwright has quit IRC | 16:38 | |
| *** mlipchuk has quit IRC | 16:39 | |
| *** jamie_h has quit IRC | 16:42 | |
| *** shakamunyi has joined #openstack-swift | 16:43 | |
| *** mmcardle has quit IRC | 16:46 | |
| *** mkollaro has quit IRC | 16:50 | |
| *** bach has quit IRC | 16:51 | |
| *** mkollaro has joined #openstack-swift | 16:54 | |
| *** elambert has joined #openstack-swift | 16:58 | |
| *** shri has joined #openstack-swift | 17:08 | |
| *** joeljwright has joined #openstack-swift | 17:15 | |
| *** byeager has quit IRC | 17:17 | |
| *** mjfork has joined #openstack-swift | 17:17 | |
| swills | FAIL: testContainerListing (test.functional.tests.TestAccount) | 17:21 |
| *** piyush has quit IRC | 17:28 | |
| *** Dharmit has quit IRC | 17:29 | |
| *** gyee has joined #openstack-swift | 17:31 | |
| *** jamie_h has joined #openstack-swift | 17:32 | |
| *** byeager has joined #openstack-swift | 17:33 | |
| *** byeager_ has joined #openstack-swift | 17:34 | |
| *** mnaser has joined #openstack-swift | 17:36 | |
| mnaser | https://review.openstack.org/#/c/45874/ -- how can I remove the -2 here? | 17:36 |
| *** byeager has quit IRC | 17:37 | |
| *** gyee has quit IRC | 17:39 | |
| *** elambert has quit IRC | 17:43 | |
| *** piyush has joined #openstack-swift | 17:44 | |
| *** piyush1 has joined #openstack-swift | 17:46 | |
| *** zaitcev has joined #openstack-swift | 17:47 | |
| *** ChanServ sets mode: +v zaitcev | 17:47 | |
| briancline | are there any drawbacks to moving account and/or container server processes out to their own hosts? | 17:48 |
| *** piyush has quit IRC | 17:48 | |
| zaitcev | you have to pay for the hosts | 17:49 |
| *** elambert has joined #openstack-swift | 17:49 | |
| *** chandan_kumar has joined #openstack-swift | 17:51 | |
| *** joeljwright has quit IRC | 17:51 | |
| *** gyee has joined #openstack-swift | 17:56 | |
| briancline | ha.. so other than that, and the generation of additional heat | 17:57 |
| ctennis | briancline, we have some customers who do that very thing, no problem | 17:58 |
| briancline | another question unrelated to that - | 18:01 |
| briancline | in a scenario where a sequence of object PUTs where some take a noticeable amount longer to perform than others, what's a good place to start digging? | 18:02 |
| briancline | I would assume swift-get-nodes on the object(s), if known, might be key in tracking down object server specific issues - but are there other/better places to be looking? | 18:04 |
| ctennis | the swift logs on each node will contain timing information about how long the operation took, you can see if one is perhaps taking longer than others. | 18:04 |
| swills | creiht: well, thanks for the help, i'm still looking at things | 18:04 |
| swills | trying to understand the exception right now | 18:04 |
| swills | got a more focused trace http://meatwad.mouf.net/~swills/swift_2.ktrace.68029.txt | 18:08 |
| swills | http://paste.mouf.net/pastes/52riits78lats05n44kb5su4/raw is the exception | 18:09 |
| *** acoles is now known as acoles_away | 18:10 | |
| *** piyush1 has quit IRC | 18:12 | |
| swills | and having upgraded to 1.13.1 I can use swift-container-info now: http://paste.mouf.net/pastes/9dr7llglm4dba181o5feaf3q/raw | 18:13 |
| *** mkollaro has quit IRC | 18:15 | |
| zaitcev | briancline: Talk to someone with experience, like Chuck or Greg. But my understanding is, they rarely try to track separate requests by txid. Instead they look at macro stats for all hosts and find duff nodes that way. | 18:16 |
| *** tongli has joined #openstack-swift | 18:19 | |
| *** krtaylor has quit IRC | 18:26 | |
| *** chandan_kumar has quit IRC | 18:29 | |
| *** rcleere has joined #openstack-swift | 18:30 | |
| zaitcev | guys, what's the opinion on Peter's https://review.openstack.org/86710 ? It reduces the amount of skipped tests for me, because I cannot be bothered to keep constraints test.conf. | 18:30 |
| zaitcev | Looks like we have a significant backlog in https://review.openstack.org/#/q/status:open+project:openstack/swift,p,002c6b2c00014f95 | 18:31 |
| zaitcev | granted, half are WIP | 18:32 |
| *** gyee has quit IRC | 18:35 | |
| creiht | briancline: it works pretty well | 18:39 |
| creiht | but the downside is | 18:39 |
| creiht | the container processes can be a bit cpu heavy | 18:39 |
| creiht | so when you bring those to a handfull of container specific nodes, your cpus could be a bottle neck | 18:39 |
| creiht | when you spread those out accross more machines, then you don't notice the cpu load as much | 18:40 |
| creiht | mnaser: I think you will have to ask the glance team :) | 18:40 |
| mnaser | creiht: woah, that was a big fail on my part, head was tuned to swift | 18:40 |
| mnaser | awkward... | 18:40 |
| creiht | zaitcev: yeah it seems most of us have been quite busy with our day jobs | 18:40 |
| creiht | mnaser: no worries man | 18:40 |
| *** kevinc_ has quit IRC | 18:44 | |
| *** gyee has joined #openstack-swift | 18:46 | |
| *** openstackgerrit has quit IRC | 18:49 | |
| *** openstackgerrit has joined #openstack-swift | 18:49 | |
| dfg | briancline: a pretty cool tool to find slow object nodes is using swiftly ping -o object.ring.gz | 18:52 |
| *** chandan_kumar has joined #openstack-swift | 18:53 | |
| briancline | creiht: thanks for the insight -- my guess is account specific nodes likely don't suffer the same bottleneck? | 18:57 |
| briancline | dfg: nice, have not seen this yet. will give it a go | 18:58 |
| creiht | well we put account and containers on the same servers | 18:58 |
| creiht | the problem is that the actual space required for them is so small, all if it gets concentrated to a much smaller set of servers | 18:58 |
| briancline | makes sense | 19:01 |
| *** bach has joined #openstack-swift | 19:02 | |
| *** bach has quit IRC | 19:02 | |
| *** foexle has quit IRC | 19:05 | |
| *** bach has joined #openstack-swift | 19:05 | |
| *** krtaylor has joined #openstack-swift | 19:07 | |
| notmyname | hello | 19:08 |
| *** gyee has quit IRC | 19:11 | |
| *** bvandenh has quit IRC | 19:12 | |
| creiht | swills: I've looked through the trace and nothing is really standing out to me | 19:13 |
| creiht | sorry | 19:14 |
| swills | creiht: k, thanks for looking | 19:15 |
| swills | creiht: i'm starting to suspect sqlite or pysqlite at this point | 19:15 |
| *** chandan_kumar has quit IRC | 19:17 | |
| *** kevinc_ has joined #openstack-swift | 19:23 | |
| *** elambert has quit IRC | 19:31 | |
| *** shakamunyi has quit IRC | 19:33 | |
| notmyname | "Saw a quote for a Netapp backend for OpenStack, it was eye-bulging. No wonder people love Swift so much." <-- zaitcev, I love it :-) | 19:35 |
| *** bach has quit IRC | 19:35 | |
| *** shakamunyi has joined #openstack-swift | 19:35 | |
| *** bach has joined #openstack-swift | 19:36 | |
| zaitcev | notmyname: Apparently a part of the problem is that HA is required for Netapp that supports Glance somehow... Which is odd because you'd think a filer is a filer, so what does Glance care if it's HA or not. But anyhow Swift is better than HA anyway (symmetric). | 19:37 |
| *** byeager_ has quit IRC | 19:37 | |
| *** byeager has joined #openstack-swift | 19:38 | |
| notmyname | zaitcev: that does sound weird | 19:39 |
| notmyname | that and something that was being tlaked about last week with glance make me think that my assumption of what glance is actually doing is incorrect | 19:39 |
| *** byeager has quit IRC | 19:42 | |
| swills | creiht: do you know if the sqlite you use has ICU enabled? | 19:54 |
| *** elambert has joined #openstack-swift | 19:56 | |
| creiht | swills: doesn't look like it | 19:59 |
| swills | creiht: k, thanks | 20:00 |
| *** byeager has joined #openstack-swift | 20:07 | |
| *** lpabon has joined #openstack-swift | 20:09 | |
| *** bach_ has joined #openstack-swift | 20:15 | |
| *** bach has quit IRC | 20:15 | |
| *** shri1 has joined #openstack-swift | 20:18 | |
| *** shri has quit IRC | 20:21 | |
| *** bach_ has quit IRC | 20:22 | |
| *** bach has joined #openstack-swift | 20:23 | |
| *** Trixboxer has quit IRC | 20:25 | |
| swills | found it | 20:26 |
| swills | http://sqlite.1065341.n5.nabble.com/I-O-errors-with-WAL-on-ZFS-td68390.html | 20:26 |
| swills | F_ALLOCSP isn't support on ZFS | 20:26 |
| *** bach has quit IRC | 20:27 | |
| swills | creiht: ^^ | 20:27 |
| creiht | ahh | 20:27 |
| creiht | there you go :) | 20:28 |
| *** tongli has quit IRC | 20:30 | |
| swills | now to see if disable_fallocate = true actually works | 20:33 |
| swills | crap, it's still there | 20:35 |
| *** shri has joined #openstack-swift | 20:39 | |
| glange | someday this will be made into a movie :) | 20:43 |
| *** shri1 has quit IRC | 20:43 | |
| swills | heh | 20:47 |
| glange | Val Kilmer will play creiht, obviously | 20:49 |
| creiht | lol | 20:50 |
| swills | dude, that's cold | 20:50 |
| clayg | umm.... Keanu Reeves | 20:50 |
| redbo | glange always thinks he'll be played by edward norton. I think edward norton is too likeable. | 20:56 |
| creiht | noam chomsky | 20:56 |
| zaitcev | The Sad Keanu | 20:57 |
| notmyname | openstack project meeting starting now, and then I'll be doing summit scheduling stuff | 21:00 |
| notmyname | 26% voter turnout so far for the TC election | 21:01 |
| dfg | there's a TC election? who's going to buy my vote? | 21:01 |
| dfg | or do i get a vote? | 21:02 |
| notmyname | dfg: vote for me, and I'll give you a free download of swift | 21:02 |
| notmyname | dfg: http://lists.openstack.org/pipermail/openstack-dev/2014-April/033173.html | 21:03 |
| dfg | tempting... | 21:03 |
| notmyname | dfg: just for you I'll throw in a copy of keystone. any neutron if you're lucky | 21:03 |
| notmyname | *and | 21:03 |
| dfg | this deal is getting less and less appealing | 21:04 |
| glange | notmyname: can you make dfg's part in the swift movie bigger? Michael Richards is playing dfg | 21:04 |
| notmyname | glange: he's already got a whole song and dance number about CORS | 21:05 |
| dfg | i was promised lines | 21:05 |
| glange | and no nudity | 21:05 |
| dfg | who in here can /kick glange? | 21:06 |
| notmyname | dfg: actually, you can :-) | 21:06 |
| dfg | dammit | 21:07 |
| dfg | nope | 21:07 |
| creiht | nosexcover is a very unfortunate name for a python package | 21:07 |
| notmyname | dfg: /msg chanserv op #openstack-swift dfg | 21:07 |
| notmyname | dfg: then you can kick whoever | 21:07 |
| creiht | and holy cow when tid test-requirements.txt get so many requirements? | 21:08 |
| *** ChanServ sets mode: +o dfg | 21:08 | |
| *** glange was kicked by dfg (glange) | 21:08 | |
| dfg | nice :) | 21:08 |
| creiht | lol | 21:08 |
| *** glange has joined #openstack-swift | 21:08 | |
| *** ChanServ sets mode: +v glange | 21:08 | |
| dfg | notmyname: ok you got my vote :) | 21:08 |
| notmyname | lol | 21:08 |
| notmyname | yikes. Ironic is oversubscribed at the summit 18:4 | 21:09 |
| notmyname | swift is 22:8 | 21:10 |
| notmyname | (merely) | 21:10 |
| glange | you can reduce those ratios :) | 21:10 |
| glange | 9:2 and 11:4 | 21:10 |
| creiht | dfg: you could just kick him in real life :) since you sit right across from him | 21:10 |
| clayg | I think all openstack positions should be *appointed* by dfg | 21:11 |
| creiht | annointed | 21:11 |
| notmyname | heh | 21:11 |
| dfg | clayg: are you trying to get my vote? | 21:12 |
| notmyname | hmm..maybe some sessions can be merged and just take half the time. would be tricky, I think | 21:12 |
| * clayg serves at the plesure of the freaking goetz | 21:13 | |
| dfg | haha | 21:13 |
| *** bach has joined #openstack-swift | 21:14 | |
| notmyname | graphs: http://summit.openstack.org/cfp/topicstatus | 21:14 |
| clayg | notmyname: there's nothing to click on there... | 21:15 |
| notmyname | clayg: why's it always got to be about clicking things? | 21:16 |
| clayg | notmyname: sometimes it's voting on things | 21:16 |
| notmyname | clayg: but only if dfg approves | 21:17 |
| clayg | i just want to see the list of talks proposed so I can lobby if I need to | 21:17 |
| notmyname | can you see http://summit.openstack.org/cfp/topic/12 | 21:17 |
| glange | Forbidden! | 21:17 |
| notmyname | ah | 21:18 |
| notmyname | http://d.not.mn/swift_topics.png | 21:18 |
| swills | so setting disable_fallocate = true seems to not work | 21:19 |
| swills | hacking sqlite not to support fallocate then running the tests results in a clean test run | 21:19 |
| clayg | it's seems strange I can't filter the full list at summit.openstack.org | 21:19 |
| creiht | swills: I think disable_fallocate prevents swift from fallocating, not sqlite | 21:19 |
| swills | creiht: oh, ok | 21:20 |
| *** gyee has joined #openstack-swift | 21:20 | |
| creiht | clayg: click on Topic | 21:20 |
| creiht | it will order them all, then just scroll to the swift ones | 21:20 |
| swills | http://paste.mouf.net/pastes/qqckk5e07vflkt10ufgf8o8d/raw | 21:20 |
| swills | are the skips normal? | 21:20 |
| creiht | \o/ | 21:20 |
| creiht | swills: yeah it will skip stuff if you don't have it enabled | 21:21 |
| clayg | creiht: hmph - not my fault they didn't put little arrows next to it | 21:21 |
| creiht | clayg: yeah obvious right? :) | 21:21 |
| swills | creiht: ok, should it? | 21:21 |
| swills | should i enable something? | 21:21 |
| clayg | a lot of those look interesting... but some of them seem like they'd be more interesting as gerrit reviews :\ | 21:21 |
| notmyname | clayg: agreed | 21:21 |
| torgomatic | i think db_preallocate = false is the one that turns off fallocate in sqlite | 21:21 |
| swills | torgomatic: oh! interesting | 21:22 |
| torgomatic | er, db_preallocation | 21:22 |
| torgomatic | or however it's spelled; it's in the sample configs | 21:22 |
| notmyname | creiht: weren't you thinking that http://summit.openstack.org/cfp/details/192 maybe should be for cross-project or somthing? | 21:22 |
| creiht | notmyname: yes | 21:22 |
| swills | torgomatic: it was off by default? | 21:22 |
| swills | torgomatic: and I was seeing it when it was commented out | 21:23 |
| clayg | i thought db preallocation was on by default? | 21:23 |
| creiht | I think those are separate things in swills' case | 21:23 |
| clayg | i don't think's "allocate ahead of the db" and... ^ what he said | 21:23 |
| torgomatic | swills: yeah, looks that way... swift.common.db.DB_PREALLOCATION = config_true_value(conf.get('db_preallocation', 'f')) | 21:23 |
| creiht | swills: db preallocation will preallocate space on disk for the db | 21:23 |
| torgomatic | creiht: oh, maybe that's something else then | 21:23 |
| torgomatic | please return to your regularly scheduled ignoring me :) | 21:24 |
| creiht | not change sqlite behavior | 21:24 |
| creiht | haha | 21:24 |
| swills | k, thanks tho | 21:24 |
| swills | so about the skips... should i enable something? | 21:24 |
| clayg | creiht: but like... sqlite doesn't work on zfs "out of the box" | 21:24 |
| swills | clayg: it can't and won't ever | 21:24 |
| clayg | swills: we don't do anything that crazy with the sqlite | 21:24 |
| creiht | clayg: yeah | 21:24 |
| * clayg had his mind blown | 21:24 | |
| swills | the idea of fallocate on a COW FS is non-sensical | 21:25 |
| creiht | swills: yeah sounds like you will have to do something special with sqlite | 21:25 |
| swills | nonsensical even | 21:25 |
| swills | creiht: i did, i hacked it's configure script to just hard define FALLOCATE off... | 21:25 |
| creiht | ahh | 21:25 |
| creiht | swills: you can run .functests -v to show the names of the tests that are skipped | 21:26 |
| swills | creiht: k, thanks | 21:27 |
| notmyname | cinder is taking that session, in order to have a cinder/glance cross-project one | 21:27 |
| creiht | swills: and look at swift/etc/proxy-server.conf-sample | 21:28 |
| creiht | that has all the possible things that can be configured for swift | 21:28 |
| swills | creiht: ah, thanks | 21:28 |
| swills | so things seem to be working really well now. :) | 21:29 |
| creiht | swills: http://docs.openstack.org/developer/swift/deployment_guide.html | 21:29 |
| swills | creiht: yep, good idea. :) | 21:30 |
| creiht | that has some further docs on deployment options | 21:30 |
| swills | now i can actually move on to planning my deployment | 21:30 |
| creiht | swills: cool | 21:30 |
| swills | thing is, we won't have a ton of locations or servers | 21:30 |
| swills | or really even a ton of files | 21:30 |
| swills | so having the server be redundant, using a resiliant FS like ZFS, etc. makes more sense than trying to use something like xfs and just let the sheer quantity of boxes help me | 21:31 |
| creiht | sure | 21:34 |
| zaitcev | right until the moment when the mega-node fails to assemble a pool on boot and pukes into your shoes | 21:35 |
| swills | zaitcev: don't get me wrong, i'm still going to have enough nodes that i have redundancy | 21:36 |
| zaitcev | swills: in that case you don't need ZFS to provide said redundancy | 21:37 |
| swills | need? no. want? yes. | 21:37 |
| swills | plus, i don't think business would be confortable with the idea that some data on disk might be corrupted until the auditor comes along and fixes it, and we serve bad data to users for a period of time... or am i misunderstanding something? | 21:38 |
| notmyname | swills: that's why we have the etag. which also protects against transit errors | 21:39 |
| zaitcev | yes, the way proxy verifies etag | 21:39 |
| zaitcev | although I'm not sure it actually does | 21:39 |
| *** lpabon has quit IRC | 21:39 | |
| notmyname | still it's possible to read a bad copy before it's quarantined. but then it will be quarantined because the GET verifies the integrity | 21:40 |
| notmyname | but that's where the etag comes in (which needs to be checked anyway) | 21:40 |
| zaitcev | This discussion happens every time when someone tries to graft Swift on top of a magical vendor solution. Or SAN. Or RAID. People never tire of this kind of thing. | 21:42 |
| swills | interesting. well, i'll have plenty of time to read and understand it all now that i've got this solved. | 21:42 |
| swills | zaitcev: it's all new to me personally. | 21:43 |
| kevinc_ | We used RAID and ZFS for our swift implementation and it was a disaster, I have spent the last 6 months trying to convert everything to JBOD and XFS... | 21:43 |
| notmyname | kevinc_: yay :-) | 21:44 |
| swills | kevinc_: oh, do tell? | 21:44 |
| swills | also, why would you use both RAID and ZFS? | 21:44 |
| kevinc_ | some servers had raid and some had zfs | 21:44 |
| swills | ah, ok | 21:44 |
| swills | i thought you meant both at the same time, which is insane. :) | 21:45 |
| swills | so what gave you trouble? | 21:45 |
| kevinc_ | when a raid set got degraded it killed the performance for weeks while it rebuilt. ZFS on linux wasn't very stable at that point and we lost whole sets of data because it would take 1-2 weeks to mount | 21:46 |
| kevinc_ | those were the worst problems | 21:46 |
| swills | hmm, ok | 21:47 |
| swills | what else? | 21:47 |
| kevinc_ | the recommendations are there for a reason, don't try to be smarter than the devs ;) | 21:47 |
| *** physcx has joined #openstack-swift | 21:48 | |
| creiht | lol | 21:48 |
| notmyname | kevinc_: no, please be smarter and tell us what's wrong :-) | 21:48 |
| creiht | well you can't believe everything you read on the internet :) | 21:48 |
| creiht | heh, and we are often wrong :) | 21:48 |
| creiht | well some more than others :) | 21:49 |
| swills | not trying to outsmart anyone, just have an affinity for zfs... | 21:49 |
| kevinc_ | :P | 21:49 |
| zaitcev | I do not have ZFS in use myself, but I'm enjoying CKS's gripes about it: http://utcc.utoronto.ca/~cks/space/blog/linux/ | 21:49 |
| physcx | Is there a known problem with using chunked encoding when uploading to swift? | 21:50 |
| notmyname | physcx: should work. what are you seeing? | 21:50 |
| swills | zaitcev: err, isn't that mostly complaining about btrfs? | 21:51 |
| zaitcev | chunked requires 100-continue, doesn't it? | 21:51 |
| physcx | more specifically a performance problem - almost 8x faster having content-length known up front as opposed to using chunked encoding on the same files | 21:51 |
| notmyname | physcx: ah, that's interesting. can you write up a simple repro so I can try it on my own swift cluster(s)? | 21:53 |
| creiht | looks like saio install on ubuntu trusty works fine | 21:53 |
| notmyname | creiht: cool | 21:53 |
| creiht | though there is a weird issue with code coverage | 21:53 |
| kevinc_ | well we used raidz2 zpools with v0.6.0 of zfsonlinux, i don't know about using single drive zpools with the latest version of zfs | 21:53 |
| notmyname | physcx: have you changed any of the network buffer settings? if not, maybe the defaults are set poorly | 21:54 |
| swills | kevinc_: i plan to use SSDs for ZIL and limit ARC as well, so performance should be fine | 21:54 |
| swills | kevinc_: the only place i can see a potential issue is scrubbing causing performance slowdown | 21:54 |
| kevinc_ | the performance on zfs went down when it exceeded 95% on a drive | 21:54 |
| swills | kevinc_: that's documented | 21:55 |
| notmyname | creiht: any dependency issues? out-of-the box versions work ok? | 21:55 |
| swills | kevinc_: that's how ZFS works, usage over 80% slows things down, usage over 90% slows things down even more, >95% and things get crazy... | 21:55 |
| creiht | notmyname: everything worked out of the box | 21:55 |
| creiht | the coverage module must be newer, becuase code coverage report on unit tests isn't working correctly | 21:55 |
| creiht | that's the only thing I've seen so far | 21:55 |
| creiht | otherwise all tests work fine | 21:56 |
| swills | test_GET (test.functional.tests.TestSloTempurl) ... SKIP: TempURL and SLO not both enabled | 21:56 |
| swills | ah ha! | 21:56 |
| kevinc_ | yes, i am aware of it, I am 3PB of data on zfs volumes.. | 21:56 |
| notmyname | creiht: does coverage go up or down? ;-) | 21:56 |
| creiht | lol | 21:56 |
| creiht | notmyname: http://paste.openstack.org/show/76698/ | 21:58 |
| physcx | notmyname: http://pastebin.com/5GbmxwhA file shown being uploaded is 205MB and in both cases completes successfully - just a performance issue | 21:59 |
| clayg | GAWD paste.openstack.org is slow - please stop using that Drizzle crap | 21:59 |
| creiht | lol | 21:59 |
| zaitcev | physcx, notmyname: also specify which LB is used to terminate, or if this goes to naked Swift | 21:59 |
| notmyname | https://gist.github.com/notmyname/81fe36d3b4a7102994ca | 21:59 |
| notmyname | clayg: ^ | 21:59 |
| * clayg ain't got no time for that! | 21:59 | |
| clayg | creiht: that coverage is awesome btw | 22:00 |
| creiht | hehe | 22:00 |
| creiht | if you change swift. to swift it works correctly | 22:00 |
| notmyname | ah, interesting | 22:00 |
| * notmyname plans on using http://paste.ronin.io by pandemicsyn from now on | 22:01 | |
| creiht | hehe | 22:01 |
| creiht | yeah I should just switch to using that | 22:02 |
| zaitcev | I just scp into people.redhat.com | 22:04 |
| physcx | zaitcev, notmyname it is using /r/n LBs libcurl4, also 16k object hunk size and client chunksize and looking manually at the syscall reads they are both performing similar number and size of reads as the default curl chunk send is 16k for us | 22:04 |
| zaitcev | In a text file with UNIX newlines | 22:04 |
| pandemicsyn | hrm i just gained 2 users....time to go raise a round of funding for my Swift Pastebin As a Web Scale Software Defined Open Service Stack..thing | 22:05 |
| creiht | lol | 22:05 |
| notmyname | pandemicsyn: percentage growth is all that matters! | 22:05 |
| pandemicsyn | its already using Go and uses a Flat design so i got the important things covered | 22:06 |
| zaitcev | I prefer a House design | 22:06 |
| swifterdarrell | pandemicsyn: lol | 22:06 |
| notmyname | pandemicsyn: swifterdarrell: but is it dockerized? | 22:06 |
| pandemicsyn | hrm, no | 22:07 |
| swifterdarrell | DOCKER DOCKER! | 22:07 |
| pandemicsyn | im also still missing out on the social part | 22:07 |
| creiht | wow.. saio on our new performance flavors runs pretty fast :) | 22:07 |
| pandemicsyn | i need to send you all push notifications when someone one you kinda know pastes something | 22:07 |
| creiht | pandemicsyn: needs keybase.io integration :) | 22:08 |
| swifterdarrell | pandemicsyn: SOCIAL | 22:08 |
| swifterdarrell | pandemicsyn: now you just need "mobile" | 22:08 |
| swifterdarrell | pandemicsyn: and maybe "big data" | 22:08 |
| pandemicsyn | "Oh you'd like to paste some python...thats an in app purchase" | 22:08 |
| notmyname | `keybase prove ronin_paste` | 22:08 |
| creiht | lol | 22:08 |
| notmyname | $ echo $? >1 # pandemicsyn doesn't like you | 22:09 |
| *** joeljwright has joined #openstack-swift | 22:09 | |
| pandemicsyn | heh | 22:09 |
| notmyname | 402 Payment Required | 22:09 |
| notmyname | ^^ more like pandemicsyn ;-) | 22:10 |
| pandemicsyn | lol | 22:10 |
| pandemicsyn | thats the Jay Payne status code | 22:10 |
| notmyname | lol | 22:10 |
| glange | man, it's inside joke day on #openstack-swift :) | 22:14 |
| creiht | hehe | 22:14 |
| *** jamie_h has quit IRC | 22:14 | |
| *** joeljwright has quit IRC | 22:14 | |
| zaitcev | I am left out, but it's all right. | 22:15 |
| shri | Hey guys… I have a question about the swift-ring-builder | 22:16 |
| shri | Can I use a hostname instead of an IP address when building the rings? | 22:17 |
| physcx | notmyname: i'm headed out for the day but if anyone with a test cluster can try curl uploading a file with and without chunked encoding and see if it is horribly slower I'd appreciate it (http://pastebin.com/5GbmxwhA), i'll check back tomorrow | 22:18 |
| notmyname | physcx: I'll try to leave a comment here | 22:18 |
| physcx | ty | 22:18 |
| notmyname | I'm doing summit scheduling and then will be out for the rest of the week | 22:19 |
| zaitcev | shri: you mean like this https://review.openstack.org/80421 | 22:19 |
| dfg | takes a lot out of you huh | 22:19 |
| shri | so..its work in progress. thanks! | 22:20 |
| notmyname | physcx: FWIW, no significant difference on my SAIO: https://gist.github.com/notmyname/17971ee386e7b1960eb4 | 22:25 |
| pandemicsyn | so much for gaining 2 users ;) | 22:27 |
| *** kevinc___ has joined #openstack-swift | 22:27 | |
| notmyname | LimeChat has the very nice feature of multiline pastes into the input box automatically being pasted to gist. | 22:28 |
| *** bach has quit IRC | 22:29 | |
| *** kevinc_ has quit IRC | 22:29 | |
| *** byeager has quit IRC | 22:30 | |
| *** byeager has joined #openstack-swift | 22:30 | |
| dfg | physcx: notmyname : i didn't really see a difference in time with Transfer-Encoding: chunked uploading a 160MB file to our staging either | 22:32 |
| zaitcev | And I don't understand what "/r/n" is supposed to mean. | 22:34 |
| *** byeager has quit IRC | 22:34 | |
| zaitcev | Sounds like a board name at 4chan. | 22:34 |
| notmyname | creiht: are you going to the june hackathon? | 22:37 |
| creiht | notmyname: unsure | 22:38 |
| creiht | but likely not | 22:38 |
| creiht | since it was so close to the summit, we will likely have to split between the two | 22:38 |
| notmyname | ah | 22:38 |
| notmyname | checking since you have the most sessions proposed ;-) | 22:39 |
| creiht | hah | 22:39 |
| notmyname | (which of course are all very important) | 22:39 |
| creiht | lol | 22:39 |
| creiht | I don't expect them all to get accepted | 22:39 |
| creiht | was just throwing some stuff out there that I thought was important | 22:39 |
| creiht | some of them might be able to be combined with others | 22:40 |
| creiht | or may not take up the whole time | 22:40 |
| notmyname | I'll try to sort them by things that deal with the community and things that will benefit more from in-person discussion. some of it, like clayg said, would be really nice to see in gerrit | 22:42 |
| notmyname | and we've got the "swift pod" all week, so that will be good for more BoF style things | 22:42 |
| notmyname | it's interesting that there were at least 4 conference sessions proposed on benchmarking swift and there are also a few summit sessions on that topic too | 22:44 |
| notmyname | (which is nice IMO because I think that's one big area of improvement we can work on in the coming months) | 22:44 |
| *** rcleere has quit IRC | 22:45 | |
| *** rcleere has joined #openstack-swift | 22:46 | |
| notmyname | note that tomorrow's team meeting is only tentative at this point. if you volunteer to lead it, add yourself here: https://wiki.openstack.org/wiki/Meetings/Swift. If nobody does, then enjoy the week off | 22:46 |
| notmyname | note also that gerrit is changing next week: http://lists.openstack.org/pipermail/openstack-dev/2014-April/033414.html | 22:48 |
| *** rcleere has quit IRC | 22:50 | |
| notmyname | dfg: scheduling is hard work! https://pbs.twimg.com/media/Bl3FlnvCUAA-Phg.jpg:large | 22:52 |
| *** elambert has quit IRC | 22:57 | |
| notmyname | pandemicsyn: if you have a chance, can you look over this patch? https://review.openstack.org/#/c/86114/ It adds swift.conf checking to recon | 22:58 |
| *** elambert has joined #openstack-swift | 22:58 | |
| pandemicsyn | heh thats handy as hell, i've done that a few times | 23:01 |
| clayg | creiht: hallway track too - i'm looking forward to seeing you! | 23:04 |
| notmyname | creiht: you say you'll be doing an audit of test coverage in swift http://summit.openstack.org/cfp/details/133 | 23:11 |
| notmyname | creiht: still happening? or only if that session is selected? | 23:11 |
| openstackgerrit | Pete Zaitcev proposed a change to openstack/swift: Return an error exit code if config is missing on start https://review.openstack.org/89715 | 23:11 |
| notmyname | seems pretty useful to have referenced somewhere | 23:12 |
| *** jergerber has quit IRC | 23:15 | |
| notmyname | zaitcev: are you going in june? | 23:15 |
| zaitcev | notmyname: yes | 23:15 |
| notmyname | great! | 23:15 |
| zaitcev | the management changed their mind and said it's okay to go to 2 conferences in a year | 23:15 |
| notmyname | portante: zaitcev: seems like the PBE discussion might be more fruitful in june than in atlanta. do you agree or disagree | 23:15 |
| zaitcev | notmyname: I agree but honestly I was hoping it did not need a discussion | 23:16 |
| notmyname | zaitcev: well blame portante for submitting it then :-) | 23:16 |
| zaitcev | Look, half of it is already in. Unelss we want to back it out, what's there to discuss? I think we should just approve https://review.openstack.org/85909 and be done. | 23:17 |
| zaitcev | Well, we won't be because then I'll file about a dozen little shards on top of 85909, but those are technical details. | 23:17 |
| *** changbl has quit IRC | 23:18 | |
| notmyname | creiht: how do you feel about combining "swift core principles" with "swift growing pains"? | 23:19 |
| *** praveenkumar has quit IRC | 23:36 | |
| *** krtaylor has quit IRC | 23:43 | |
| occupant | so I'm running grizzly (yeah, I know, we should upgrade) and I think I'm getting hit by this bug that was fixed in havana - https://bugs.launchpad.net/swift/+bug/1174660 | 23:54 |
| occupant | if I run iftop on my proxy server, I'm fetching like 1Gb/s from my object servers, but I'm only putting out like 100Mb/s of traffic. | 23:55 |
| occupant | https://bugs.launchpad.net/swift/+bug/1166198 got marked a dupe but it seems to describe what I'm seeing a bit better - lots of extra traffic that seemingly shouldn't be there. | 23:56 |
| occupant | the final comment on that bug, someone says they used a patch against the havana branch on their grizzly install and it fixed things. pondering trying it myself. | 23:56 |
| occupant | Because I'm getting lots of timeouts from backend connections and I'm wondering if it's not just getting overloaded. | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
