| *** asettle has quit IRC | 00:01 | |
| *** mmoti____ has quit IRC | 00:02 | |
| *** mmotia___ has quit IRC | 00:02 | |
| *** mmcardle has quit IRC | 00:02 | |
| *** mmotian__ has quit IRC | 00:02 | |
| *** NM has quit IRC | 00:07 | |
| *** mahatic has quit IRC | 00:08 | |
| *** mahatic has joined #openstack-swift | 00:09 | |
| timburke | tdasilva: is there any reason that copy *has* to be after staticweb? or are we just using that as a convenient marker for "after auth" in case deployers aren't using keystoneauth? | 00:12 |
|---|---|---|
| timburke | i'd just noticed that our vsaio had slo and dlo before staticweb, which broke a bunch of func tests. easy enough to fix (https://github.com/swiftstack/vagrant-swift-all-in-one/commit/22601b3) but i wonder if it might bite others... | 00:13 |
| tdasilva | timburke: in sample files staticweb shows up right after auth: https://github.com/openstack/swift/blob/master/doc/saio/swift/proxy-server.conf#L12 | 00:15 |
| tdasilva | so basically I just kept moving left in that sample until there was no more copy request | 00:16 |
| tdasilva | last one was quotas middleware that you had found | 00:16 |
| *** sams-gleb has joined #openstack-swift | 00:16 | |
| *** mingdang1 has joined #openstack-swift | 00:19 | |
| timburke | yep. i'm aware (now) that the config was busted. one more reason a solver would be nice, i suppose... | 00:20 |
| *** mmotian__ has joined #openstack-swift | 00:21 | |
| *** suyash has quit IRC | 00:22 | |
| *** Jeffrey4l has joined #openstack-swift | 00:22 | |
| *** tqtran has quit IRC | 00:22 | |
| *** mahatic has quit IRC | 00:22 | |
| *** NM has joined #openstack-swift | 00:24 | |
| *** rcernin has quit IRC | 00:24 | |
| *** mahatic has joined #openstack-swift | 00:24 | |
| *** mingdang1 has quit IRC | 00:24 | |
| *** sams-gleb has quit IRC | 00:24 | |
| *** catintheroof has quit IRC | 00:27 | |
| *** mmotia___ has joined #openstack-swift | 00:34 | |
| *** mmoti____ has joined #openstack-swift | 00:35 | |
| *** mmotian__ has quit IRC | 00:38 | |
| *** mmotia___ has quit IRC | 00:39 | |
| *** diogogmt has quit IRC | 00:44 | |
| *** mmotian__ has joined #openstack-swift | 00:45 | |
| *** mmoti____ has quit IRC | 00:48 | |
| *** mmotia___ has joined #openstack-swift | 00:50 | |
| *** mmotian__ has quit IRC | 00:53 | |
| kota_ | good morning | 00:54 |
| openstackgerrit | Merged openstack/swift: Fix download resumption after getting no data. https://review.openstack.org/306227 | 00:55 |
| openstackgerrit | Merged openstack/swift: Tighten up ResumingGetter.fast_forward https://review.openstack.org/306661 | 00:55 |
| openstackgerrit | Merged openstack/swift: List system dependencies for running common tests https://review.openstack.org/298313 | 00:55 |
| *** mmotian__ has joined #openstack-swift | 00:57 | |
| notmyname | git st | 00:58 |
| notmyname | well, that wasn't the right window | 00:58 |
| timburke | good morning kota_ | 00:58 |
| kota_ | good morning timburk. | 00:58 |
| kota_ | ah yeah, the ResumingGetter fix looks great (though my quick look) | 01:00 |
| *** mmotia___ has quit IRC | 01:00 | |
| kota_ | love it | 01:00 |
| *** openstackgerrit has quit IRC | 01:02 | |
| *** openstackgerrit has joined #openstack-swift | 01:03 | |
| *** mmotia___ has joined #openstack-swift | 01:10 | |
| *** chlong has joined #openstack-swift | 01:13 | |
| *** mmoti____ has joined #openstack-swift | 01:13 | |
| *** mmotian__ has quit IRC | 01:14 | |
| *** mmotia___ has quit IRC | 01:15 | |
| *** mmotian__ has joined #openstack-swift | 01:17 | |
| *** mmoti____ has quit IRC | 01:17 | |
| *** mmotia___ has joined #openstack-swift | 01:19 | |
| *** sams-gleb has joined #openstack-swift | 01:22 | |
| *** mmoti____ has joined #openstack-swift | 01:22 | |
| *** mmotian__ has quit IRC | 01:22 | |
| *** mmotia___ has quit IRC | 01:26 | |
| *** mmotian__ has joined #openstack-swift | 01:26 | |
| *** mmotian__ has quit IRC | 01:26 | |
| *** mmotian__ has joined #openstack-swift | 01:27 | |
| *** sams-gleb has quit IRC | 01:29 | |
| *** mmoti____ has quit IRC | 01:30 | |
| *** bill_az has quit IRC | 01:31 | |
| *** hezhiqiang has joined #openstack-swift | 01:31 | |
| *** chlong has quit IRC | 01:31 | |
| *** lyrrad_ has joined #openstack-swift | 01:32 | |
| *** diogogmt has joined #openstack-swift | 01:33 | |
| *** hezhiqiang has quit IRC | 01:33 | |
| *** lyrrad has quit IRC | 01:33 | |
| *** lyrrad_ is now known as lyrrad | 01:33 | |
| *** hezhiqiang has joined #openstack-swift | 01:33 | |
| *** lyrrad_ has joined #openstack-swift | 01:37 | |
| *** lyrrad has quit IRC | 01:38 | |
| *** lyrrad_ is now known as lyrrad | 01:38 | |
| *** mmotia___ has joined #openstack-swift | 01:40 | |
| *** mmoti____ has joined #openstack-swift | 01:42 | |
| *** NM has quit IRC | 01:43 | |
| *** mmotian__ has quit IRC | 01:43 | |
| *** mmotia___ has quit IRC | 01:45 | |
| *** NM has joined #openstack-swift | 01:48 | |
| *** rickyrem has quit IRC | 01:55 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift: Import BrainSpliter directly in expirer probe https://review.openstack.org/315836 | 01:58 |
| *** NM has quit IRC | 01:58 | |
| *** mmotian__ has joined #openstack-swift | 02:11 | |
| *** mmoti____ has quit IRC | 02:11 | |
| tdasilva | just saw the link on hckr news, seems like a good read: http://the-paper-trail.org/blog/distributed-systems-theory-for-the-distributed-systems-engineer/ | 02:21 |
| pdardeau | kota_: started digging into composite ring patch today. interesting stuff and a lot to digest | 02:22 |
| *** mmotian__ has quit IRC | 02:22 | |
| pdardeau | kota_: i'll probably end up with a bunch of questions before i can do a meaningful review | 02:23 |
| *** mmotian__ has joined #openstack-swift | 02:24 | |
| kota_ | padardeau: thanks a lot!!!! please catch me anytime :) | 02:25 |
| kota_ | padardeau: right now, that patch conflicts current newest master, i'll fix the merge conflict around this (or the next) week. | 02:26 |
| *** sams-gleb has joined #openstack-swift | 02:26 | |
| *** mmotian__ has quit IRC | 02:26 | |
| *** mmotian__ has joined #openstack-swift | 02:28 | |
| *** mmotian__ has quit IRC | 02:29 | |
| pdardeau | kota_: i saw that, but i thought i'd start looking now anyway since i figured it would take me a while to understand | 02:32 |
| kota_ | pdardeau: yeah, exactly. | 02:33 |
| *** sams-gleb has quit IRC | 02:34 | |
| pdardeau | kota_: patch 271920 is dependent on patch 268993? (i'm not sure how to interpret what gerrit shows as 'Related Changes') | 02:37 |
| patchbot | pdardeau: https://review.openstack.org/#/c/271920/ - swift - CompositeRing Support | 02:37 |
| patchbot | pdardeau: https://review.openstack.org/#/c/268993/ - swift - Fix EC ring validation at ring reload | 02:37 |
| *** tanee has quit IRC | 02:38 | |
| openstackgerrit | Merged openstack/swift: Fix bug expirer unexpectedly deletes object created after x-delete-at https://review.openstack.org/283351 | 02:39 |
| *** tanee has joined #openstack-swift | 02:40 | |
| *** mmotian__ has joined #openstack-swift | 02:44 | |
| kota_ | pdardeau: yes, that depends on patch 268993 which I got +2 just yesterday from Pete. | 02:48 |
| patchbot | kota_: https://review.openstack.org/#/c/268993/ - swift - Fix EC ring validation at ring reload | 02:48 |
| kota_ | that is because patch 268993 enables Swift to validate the ring correctness in runtime (while reloading the ring) | 02:51 |
| patchbot | kota_: https://review.openstack.org/#/c/268993/ - swift - Fix EC ring validation at ring reload | 02:51 |
| *** tanee has quit IRC | 02:51 | |
| kota_ | and then, composite ring patch expand the logic to validate the state of 2 or more than rings in runtime. | 02:52 |
| *** tanee has joined #openstack-swift | 02:52 | |
| pdardeau | kota_: ok, thx. i thought that might be the case but was a little thrown off trying to understand if there was a connection between EC and composite rings | 02:53 |
| *** mmotia___ has joined #openstack-swift | 02:56 | |
| kota_ | pdardeau: correct. composite ring is not only for EC, it can be used generally for all ring related things. | 02:58 |
| *** mingdang1 has joined #openstack-swift | 02:59 | |
| *** mmotian__ has quit IRC | 02:59 | |
| *** mmotian__ has joined #openstack-swift | 03:00 | |
| *** mmotia___ has quit IRC | 03:00 | |
| openstackgerrit | Merged openstack/swift: Put back missing sysmeta unit tests https://review.openstack.org/315613 | 03:06 |
| *** chlong has joined #openstack-swift | 03:08 | |
| *** mmotia___ has joined #openstack-swift | 03:09 | |
| *** mmoti____ has joined #openstack-swift | 03:10 | |
| *** mmot_____ has joined #openstack-swift | 03:11 | |
| *** mmotian__ has quit IRC | 03:13 | |
| pdardeau | kota_: the one thing that caught my attention between the 2 is that the EC patch adds validation_hook (lambda) to Ring, but then the composite ring patch changes it to validator | 03:14 |
| *** mmotia___ has quit IRC | 03:14 | |
| *** mmoti____ has quit IRC | 03:15 | |
| pdardeau | kota_: my thinking is that since the EC patch is introducing the concept, it should be the one to have the change from validation_hook to validator | 03:16 |
| pdardeau | otherwise there would be a span of time where there's a temporary solution (more room for confusion) | 03:17 |
| *** mmotian__ has joined #openstack-swift | 03:23 | |
| *** dmorita has quit IRC | 03:23 | |
| *** mmot_____ has quit IRC | 03:26 | |
| *** dmorita has joined #openstack-swift | 03:28 | |
| *** dmorita has quit IRC | 03:28 | |
| *** dmorita has joined #openstack-swift | 03:28 | |
| *** dmorita has quit IRC | 03:29 | |
| *** dmorita has joined #openstack-swift | 03:29 | |
| *** sams-gleb has joined #openstack-swift | 03:31 | |
| *** dmorita has quit IRC | 03:32 | |
| *** mmotian__ has quit IRC | 03:32 | |
| *** rickyrem has joined #openstack-swift | 03:34 | |
| *** dmorita has joined #openstack-swift | 03:34 | |
| *** mmotian__ has joined #openstack-swift | 03:35 | |
| *** mmotian__ has quit IRC | 03:36 | |
| *** mmotian__ has joined #openstack-swift | 03:36 | |
| *** mmotia___ has joined #openstack-swift | 03:38 | |
| *** mmotia___ has quit IRC | 03:38 | |
| *** rickyrem has quit IRC | 03:38 | |
| *** mmotia___ has joined #openstack-swift | 03:38 | |
| *** sams-gleb has quit IRC | 03:39 | |
| *** dmorita has quit IRC | 03:39 | |
| *** mmotian__ has quit IRC | 03:42 | |
| *** mmotian__ has joined #openstack-swift | 03:44 | |
| *** mmoti____ has joined #openstack-swift | 03:46 | |
| *** tqtran has joined #openstack-swift | 03:47 | |
| *** mmot_____ has joined #openstack-swift | 03:47 | |
| *** tqtran has quit IRC | 03:48 | |
| *** mmotia___ has quit IRC | 03:49 | |
| *** mmotian__ has quit IRC | 03:49 | |
| *** mmoti____ has quit IRC | 03:50 | |
| *** jamielennox is now known as jamielennox|away | 03:58 | |
| *** mmot_____ has quit IRC | 04:00 | |
| *** asettle has joined #openstack-swift | 04:01 | |
| *** jamielennox|away is now known as jamielennox | 04:03 | |
| *** mmotian__ has joined #openstack-swift | 04:06 | |
| *** asettle has quit IRC | 04:07 | |
| mahatic | good morning | 04:10 |
| *** jith_ has joined #openstack-swift | 04:11 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift: Skip SLO-reconciling probe test when SLO is off https://review.openstack.org/312698 | 04:15 |
| jith_ | hi all.. i installed swift in debian... installed using python setup.py develop.. scripts got copied to /usr/local/bin.. but when i try to execute any script, for example "swift-ring-builder --h". I got one assertion error " error "AssertionError: version mismatch, 0.8.6 != 1.6.0" .. please guide me in this | 04:17 |
| jith_ | the complete error is http://paste.openstack.org/show/497006/ | 04:18 |
| *** psachin has joined #openstack-swift | 04:20 | |
| kota_ | jith_: looks like that error occurred at ffi package | 04:21 |
| *** mmotian__ has quit IRC | 04:21 | |
| kota_ | jith_: could you update libffi-dev? | 04:21 |
| kota_ | jith_: here is docs for dependencies, http://docs.openstack.org/developer/swift/development_saio.html#installing-dependencies | 04:22 |
| *** links has joined #openstack-swift | 04:23 | |
| jith_ | kota_: thanks | 04:26 |
| jith_ | will try | 04:26 |
| kota_ | jith_: :-) | 04:26 |
| kota_ | mahatic: morning | 04:26 |
| jith_ | kota_: i already installed all the dependencies | 04:26 |
| *** hezhiqiang has quit IRC | 04:27 | |
| *** hezhiqiang has joined #openstack-swift | 04:27 | |
| jith_ | i will try to update the libffi-dev | 04:28 |
| jith_ | any specific version?? | 04:28 |
| kota_ | i think no, | 04:29 |
| kota_ | no specific version AFAIK. | 04:29 |
| jith_ | kota_ : its already updated one i think.. installed version is 3.1-2+b2 | 04:31 |
| kota_ | hmm... curious. | 04:32 |
| kota_ | jith_: sorry thus probably I don't have right answer for that now. | 04:33 |
| *** mingdang1 has quit IRC | 04:34 | |
| *** sams-gleb has joined #openstack-swift | 04:36 | |
| jith_ | kota_: thanks | 04:37 |
| jith_ | i cleared | 04:37 |
| kota_ | oh, what is actually problem? | 04:38 |
| *** klrmn has quit IRC | 04:40 | |
| jith_ | cffi package should be 0.8.6 version | 04:41 |
| kota_ | interesting | 04:43 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/python-swiftclient: Correct doc string for http_conn argument https://review.openstack.org/315878 | 04:44 |
| *** sams-gleb has quit IRC | 04:44 | |
| *** hezhiqiang has quit IRC | 05:11 | |
| *** hezhiqiang has joined #openstack-swift | 05:11 | |
| *** asettle has joined #openstack-swift | 05:12 | |
| *** asettle has quit IRC | 05:17 | |
| *** ChubYann has quit IRC | 05:17 | |
| *** silor has joined #openstack-swift | 05:26 | |
| *** chlong has quit IRC | 05:29 | |
| kota_ | oh...what? Was I dreaming? the_object_expirer probe became failing at the master. While my reviewing on the patch set, it got success.... | 05:36 |
| *** zaitcev has quit IRC | 05:41 | |
| *** sams-gleb has joined #openstack-swift | 05:41 | |
| *** sams-gleb has quit IRC | 05:46 | |
| *** sams-gleb has joined #openstack-swift | 05:46 | |
| *** rcernin has joined #openstack-swift | 05:50 | |
| *** jamielennox is now known as jamielennox|away | 05:56 | |
| *** jamielennox|away is now known as jamielennox | 06:10 | |
| *** geaaru has joined #openstack-swift | 06:14 | |
| *** SkyRocknRoll has joined #openstack-swift | 06:25 | |
| *** mmcardle has joined #openstack-swift | 06:25 | |
| *** asettle has joined #openstack-swift | 06:31 | |
| *** asettle has quit IRC | 06:33 | |
| *** sams-gleb has quit IRC | 06:34 | |
| *** sams-gleb has joined #openstack-swift | 06:35 | |
| *** sams-gleb has joined #openstack-swift | 06:35 | |
| *** sams-gleb has quit IRC | 06:35 | |
| *** sams-gleb has joined #openstack-swift | 06:36 | |
| *** mmcardle has quit IRC | 06:38 | |
| *** sams-gleb has quit IRC | 06:39 | |
| kota_ | hmm..... the expirer test is still failing even if recreating my saio env, why it is. | 06:43 |
| *** tesseract has joined #openstack-swift | 06:56 | |
| *** sams-gleb has joined #openstack-swift | 07:00 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift: WIP: fix samall things as follow up https://review.openstack.org/315918 | 07:05 |
| *** eranrom has joined #openstack-swift | 07:05 | |
| *** jeblair has quit IRC | 07:11 | |
| openstackgerrit | Merged openstack/swift: Skip SLO-reconciling probe test when SLO is off https://review.openstack.org/312698 | 07:12 |
| *** rledisez has joined #openstack-swift | 07:14 | |
| *** jeblair has joined #openstack-swift | 07:18 | |
| *** jmccarthy has quit IRC | 07:37 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift: Updated from global requirements https://review.openstack.org/88736 | 07:37 |
| *** jmccarthy has joined #openstack-swift | 07:38 | |
| *** sanchitmalhotra has joined #openstack-swift | 07:40 | |
| *** sanchitmalhotra1 has quit IRC | 07:41 | |
| *** hezhiqiang has quit IRC | 07:48 | |
| *** msno has joined #openstack-swift | 07:48 | |
| *** hezhiqiang has joined #openstack-swift | 07:48 | |
| msno | hi | 07:49 |
| msno | using swift as glance's storage | 07:49 |
| msno | image create fails with 503 error | 07:49 |
| *** jmccarthy1 has joined #openstack-swift | 07:49 | |
| msno | in swift.log i see the following timeout error | 07:49 |
| msno | http://paste.openstack.org/show/497009/ | 07:49 |
| msno | any pointers please | 07:50 |
| msno | it seems proxy server is not getting response from object server .. can i increase this 10 s value timeut parameter somewhere | 07:51 |
| *** tdasilva has quit IRC | 07:54 | |
| ahale | msno: yeah i think thats node_timeout in the proxy conf | 08:02 |
| ahale | or you might want to look at mb_per_sync in the object conf | 08:03 |
| *** sc has quit IRC | 08:05 | |
| *** tdasilva has joined #openstack-swift | 08:09 | |
| *** ChanServ sets mode: +v tdasilva | 08:09 | |
| kota_ | ok, I got a success for the expirer test and on the other time, got failure so definately it's flaky. | 08:12 |
| *** acoles_ is now known as acoles | 08:15 | |
| *** mmcardle has joined #openstack-swift | 08:16 | |
| *** ozialien10 has joined #openstack-swift | 08:20 | |
| kota_ | perhaps, that fails because disallow x-timestamp change???? | 08:27 |
| kota_ | when I checked all diskfiles in the test, the timestmps look different from the test expected. | 08:28 |
| *** hseipp has joined #openstack-swift | 08:33 | |
| openstackgerrit | Alistair Coles proposed openstack/swift: crypto - don't break conditional GETs on unencrypted objects https://review.openstack.org/314740 | 08:33 |
| openstackgerrit | Alistair Coles proposed openstack/swift: crypto - put update_content_type back into obj.py https://review.openstack.org/315955 | 08:34 |
| kota_ | got it! change Timestamp control from X-Timestamp to sleep, that worked. | 08:37 |
| kota_ | next question I am wondering, in probe tests could we set shunt_inbound_x_timestamp = true to save the running time? | 08:38 |
| kota_ | not sure. | 08:38 |
| kota_ | anyways, i can make a patch to fix the probe. | 08:38 |
| *** duobei has joined #openstack-swift | 08:41 | |
| *** jistr has joined #openstack-swift | 08:46 | |
| *** duobei has quit IRC | 08:51 | |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift: Fix probe failure and samall things as follow up https://review.openstack.org/315918 | 08:56 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift: Fix probe failure and samall things https://review.openstack.org/315918 | 08:57 |
| *** janonymous has quit IRC | 09:07 | |
| *** dmk0202 has joined #openstack-swift | 09:09 | |
| acoles | mahatic: i could swear I put the refs [1] and [2] in that commit message ! | 09:10 |
| mahatic | acoles: lol. gerrit ate them! | 09:10 |
| acoles | yeah - gerrit hates me | 09:10 |
| mahatic | heh | 09:11 |
| acoles | when I start making too many mistakes, someone should gently show me to the door... | 09:11 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift: Fix probe failure and small things https://review.openstack.org/315918 | 09:12 |
| acoles | now if gerrit was smart it would have read my mind and inserted the refs | 09:12 |
| acoles | kota_: hi! | 09:13 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/swift: Fix probe failure and small things https://review.openstack.org/315918 | 09:14 |
| mahatic | acoles: if commit msg mistakes are called "too many mistakes", i don't know what to say | 09:15 |
| openstackgerrit | Alistair Coles proposed openstack/swift: crypto - put update_content_type back into obj.py https://review.openstack.org/315955 | 09:16 |
| acoles | mahatic: ^^ :) | 09:16 |
| *** jith_ has quit IRC | 09:17 | |
| mahatic | acoles: :) stupid gerrit! | 09:18 |
| mahatic | acoles: nice that you remember to put that change back | 09:19 |
| acoles | mahatic: i noticed it during the merge from master because it was in a merge conflict | 09:20 |
| mahatic | acoles: oic, which reminds me, need to test removing the crpto-id stuff and see what happens | 09:21 |
| mahatic | I'll check that | 09:21 |
| *** mvk has joined #openstack-swift | 09:25 | |
| *** joeljwright has joined #openstack-swift | 09:30 | |
| *** ChanServ sets mode: +v joeljwright | 09:30 | |
| *** asettle has joined #openstack-swift | 09:40 | |
| acoles | mahatic: worth checking, thanks, but I am still pondering the discussion from yesterday about objects moving between containers etc. | 09:43 |
| mahatic | acoles: yeah, if we're confident enough on the change (using account key), it'll be great for auto-tiering and other related features. Figuring out is the hard part. on high level, it sounds good, but i'm sure i'm not considering all the details | 09:56 |
| *** moonracer has joined #openstack-swift | 10:04 | |
| moonracer | Good morning | 10:04 |
| kota_ | acoles: hi! i noticed just now in the train heading for home. | 10:10 |
| admin6 | Hello. I’m trying to check deeper the health of my erasure coded ring. and I’ve a few logs that I’m trying to understand. | 10:11 |
| admin6 | First one is : object-reconstructor: Unable to get enough responses (1/9) to reconstruct 10.10.2.51:6000/s01z1ecd01/200914/AUTH_XXXX/20160131_O0/test1 policy#1 frag#11 with ETag 31d00d842caf20bbfac92617eed068f2 | 10:11 |
| admin6 | Does that mean that the reconstructor is enable to reconstruct this object ? | 10:12 |
| admin6 | unable | 10:12 |
| kota_ | admin6: it looks so. only a fragment is available for the object | 10:15 |
| kota_ | It needs ec_k fragments to reconstruct | 10:16 |
| admin6 | kota_: ok. :-( | 10:17 |
| admin6 | I also have another type of logs that most looks like a bug : | 10:18 |
| admin6 | object-reconstructor: Exception in top-levelreconstruction loop: #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/dist-packages/swift/obj/reconstructor.py", line 912, in reconstruct#012 jobs = self.build_reconstruction_jobs(part_info)#012 File "/usr/lib/python2.7/dist-packages/swift/obj/reconstructor.py", line 871, in build_reconstruction_jobs#012 jobs = self._get_part_jobs(**part_info)#012 File | 10:18 |
| admin6 | "/usr/lib/python2.7/dist-packages/swift/obj/reconstructor.py", line 761, in _get_part_jobs#012 sync_to=[part_nodes[fi]],#012IndexError: list index out of range | 10:18 |
| kota_ | Possibly, it cam be recovered? (If the other frags in handoff nodes)? | 10:20 |
| *** d0ugal has quit IRC | 10:20 | |
| *** mingdang1 has joined #openstack-swift | 10:21 | |
| *** d0ugal has joined #openstack-swift | 10:21 | |
| *** sams-gleb has quit IRC | 10:23 | |
| kota_ | Looks like your ring has less reprica number than ec_k + ec_m? | 10:23 |
| kota_ | Or floating number... i thought | 10:24 |
| *** moonracer has quit IRC | 10:25 | |
| admin6 | No the ring has 12 replicas with a 9+3 EC | 10:26 |
| admin6 | object-1.builder, build version 44 262144 partitions, 12.000000 replicas, 1 regions, 4 zones, 27 devices, 0.49 balance, 42.95 dispersion | 10:28 |
| kota_ | Curious, will look at. Sorry I'm in moving train and joining via mobile | 10:29 |
| admin6 | [storage-policy:1] name = EC policy_type = erasure_coding ec_type = jerasure_rs_vand ec_num_data_fragments = 9 ec_num_parity_fragments = 3 ec_object_segment_size = 1048576 | 10:30 |
| admin6 | kota_: ok no pb, thanks | 10:30 |
| *** hezhiqiang has quit IRC | 10:47 | |
| mahatic | acoles: looks like versioning will have problems without crypto-id saving off; for instance test.functional.tests.TestObjectVersioning.test_overwriting fails if i remove "rederive" using crypto-id | 11:16 |
| *** sams-gleb has joined #openstack-swift | 11:17 | |
| mahatic | i see about 7 tests fail (3 related to test_overwriting) | 11:17 |
| acoles | mahatic: ack | 11:17 |
| *** mvk has quit IRC | 11:19 | |
| acoles | mahatic: do you have versioned_writes to *left* of crypto in the proxy pipeline? | 11:20 |
| mahatic | acoles: ah, nope. it's on the right | 11:21 |
| mahatic | acoles: was that a change that was always required? I don't recall. I will test again with that change | 11:22 |
| acoles | mahatic: we do say in proxy-server.conf-sample that crypto should be at end of pipeline | 11:23 |
| mahatic | acoles: oic. with that, for new middlewares (say auto-tiering), we shouldn't be too worried about moving objects across containers, no? | 11:27 |
| acoles | mahatic: the potential problem with auto-tiering is if the objects are moved directly between containers by a process that does not use the proxy. or specifically does not use crypto middlewares. the tiering proposal in austin used a daemon to move objects | 11:33 |
| mahatic | acoles: with crypto at end of pipeline and removing crypto-id, all func test pass | 11:33 |
| mahatic | jrichli: ^ | 11:34 |
| *** msno has quit IRC | 11:37 | |
| mahatic | acoles: oh okay | 11:38 |
| * mahatic is afk for a while | 11:41 | |
| ZZelle | joeljwright, hi | 11:45 |
| *** NM has joined #openstack-swift | 11:47 | |
| *** mvk has joined #openstack-swift | 11:49 | |
| *** SkyRocknRoll has quit IRC | 11:52 | |
| *** admin6 has left #openstack-swift | 12:08 | |
| *** links has quit IRC | 12:14 | |
| *** darrenc_ has joined #openstack-swift | 12:17 | |
| *** philipw has quit IRC | 12:17 | |
| *** jroll has quit IRC | 12:17 | |
| *** philipw has joined #openstack-swift | 12:17 | |
| *** darrenc has quit IRC | 12:17 | |
| *** raildo-afk is now known as raildo | 12:22 | |
| *** jroll has joined #openstack-swift | 12:22 | |
| *** jroll has quit IRC | 12:22 | |
| *** jroll has joined #openstack-swift | 12:22 | |
| *** darrenc_ is now known as darrenc | 12:22 | |
| *** okdas_ has quit IRC | 12:23 | |
| *** okdas has joined #openstack-swift | 12:24 | |
| *** okdas has joined #openstack-swift | 12:24 | |
| *** diogogmt has quit IRC | 12:37 | |
| *** mmcardle has quit IRC | 12:38 | |
| *** diogogmt has joined #openstack-swift | 12:39 | |
| *** rcernin_ has joined #openstack-swift | 12:42 | |
| *** rcernin has quit IRC | 12:42 | |
| openstackgerrit | Alistair Coles proposed openstack/swift: crypto - wrap random body key https://review.openstack.org/313640 | 12:49 |
| pdardeau | good morning | 12:50 |
| acoles | pdardeau: good morning | 12:54 |
| pdardeau | acoles: o/ | 12:56 |
| *** sams-gleb has quit IRC | 12:59 | |
| *** sams-gleb has joined #openstack-swift | 13:05 | |
| *** diogogmt has quit IRC | 13:06 | |
| *** kei_yama has quit IRC | 13:09 | |
| *** cbartz has joined #openstack-swift | 13:15 | |
| *** diogogmt has joined #openstack-swift | 13:15 | |
| *** rcernin_ has quit IRC | 13:16 | |
| *** mmcardle has joined #openstack-swift | 13:23 | |
| *** bill_az has joined #openstack-swift | 13:25 | |
| *** rcernin has joined #openstack-swift | 13:27 | |
| acoles | jrichli: i was just looking at devstack and notice there is an env var SWIFT_EXTRAS_MIDDLEWARE_LAST which looks like it could be used to easily drop crypto middlewares into a devstack swift install. | 13:34 |
| acoles | jrichli: then instead of using in process tests, define a new devstack job for zuul. | 13:35 |
| acoles | jrichli: there is some work in progress to add a swift func test job that only uses keystone v3, and that might provide a template to follow, patch 313659 | 13:36 |
| patchbot | acoles: https://review.openstack.org/#/c/313659/ - openstack-infra/project-config - Run Swift functional tests in Identity v3-only | 13:36 |
| *** vint_bra has joined #openstack-swift | 13:37 | |
| acoles | jrichli: here - https://github.com/openstack-dev/devstack/blob/ae4578bed986ebaaf3e09e1ecc9d4d7ae9f022ab/lib/swift#L428-L428 | 13:38 |
| *** dmorita has joined #openstack-swift | 13:38 | |
| *** dmorita has quit IRC | 13:43 | |
| *** openstack has joined #openstack-swift | 13:46 | |
| *** openstack has joined #openstack-swift | 13:54 | |
| *** ametts has joined #openstack-swift | 14:04 | |
| jrichli | acoles: thanks, I will go in that direction. mahatic: thanks for testing without cyrpto-id | 14:07 |
| acoles | jrichli: of course, we could (in fact maybe we should) put the crypto middlewares in the proxy-server.conf-sample pipeline. That way devstack will install them by default. It's debatable whether we'd propose that to master but at least we get jenkins testing it while we're on the feature branch. | 14:10 |
| acoles | ...now that we pass functional tests we can do that ;) | 14:10 |
| acoles | jrichli: would you like to put together a patch for that ^^ I tend to always run func tests locally with crypto installed but jenkins is more reliable than me. | 14:11 |
| *** links has joined #openstack-swift | 14:19 | |
| *** sams-gleb has joined #openstack-swift | 14:23 | |
| bkeller` | 'morning | 14:28 |
| *** d0ugal has quit IRC | 14:29 | |
| *** links has quit IRC | 14:29 | |
| *** d0ugal has joined #openstack-swift | 14:30 | |
| *** sams-gleb has quit IRC | 14:32 | |
| jrichli | acoles: true, I can upload a patch for that. I will still work on the other - as it will be needed. I'll keep in mind that if we do end up with a patch that affects non-crypto processing, then I'll run a regression manually. | 14:33 |
| *** sams-gleb has joined #openstack-swift | 14:36 | |
| acoles | jrichli: we'd still have two lots of in process func tests to guard against any such regression | 14:36 |
| acoles | jrichli: and yes, still work on the other | 14:36 |
| *** can8dnSix has joined #openstack-swift | 14:38 | |
| *** d0ugal has quit IRC | 14:38 | |
| *** gyee_ has joined #openstack-swift | 14:38 | |
| *** gyee has quit IRC | 14:39 | |
| *** _JZ_ has joined #openstack-swift | 14:48 | |
| *** mingdang1 has quit IRC | 14:52 | |
| *** sams-gleb has quit IRC | 14:56 | |
| *** esker has joined #openstack-swift | 14:56 | |
| *** esker has quit IRC | 14:57 | |
| *** esker has joined #openstack-swift | 14:58 | |
| *** awelleck has joined #openstack-swift | 14:58 | |
| *** BAKfr has quit IRC | 14:59 | |
| *** arch-nemesis has joined #openstack-swift | 15:01 | |
| *** BAKfr has joined #openstack-swift | 15:02 | |
| *** arch-nemesis has quit IRC | 15:04 | |
| *** d0ugal has joined #openstack-swift | 15:05 | |
| *** dmk0202 has quit IRC | 15:13 | |
| *** klrmn has joined #openstack-swift | 15:14 | |
| *** openstackgerrit has quit IRC | 15:17 | |
| *** openstackgerrit has joined #openstack-swift | 15:18 | |
| *** dmorita has joined #openstack-swift | 15:33 | |
| *** dmorita has quit IRC | 15:37 | |
| *** cbartz has left #openstack-swift | 15:41 | |
| *** mmotiani has quit IRC | 15:42 | |
| *** mmotiani_ is now known as mmotiani | 15:42 | |
| *** sgundur- has quit IRC | 15:43 | |
| *** pdardeau has quit IRC | 15:43 | |
| *** yarkot has quit IRC | 15:43 | |
| *** nadeem has joined #openstack-swift | 15:43 | |
| *** ntata- has quit IRC | 15:44 | |
| *** zaitcev has joined #openstack-swift | 15:46 | |
| *** ChanServ sets mode: +v zaitcev | 15:46 | |
| *** lyrrad has quit IRC | 15:48 | |
| *** ntata- has joined #openstack-swift | 15:49 | |
| *** psachin has quit IRC | 15:50 | |
| *** esker has quit IRC | 15:54 | |
| *** sgundur1 has joined #openstack-swift | 15:54 | |
| *** tesseract has quit IRC | 15:56 | |
| notmyname | good morning | 15:56 |
| *** sams-gleb has joined #openstack-swift | 15:56 | |
| *** gyee_ has quit IRC | 15:56 | |
| *** mmotiani_ has joined #openstack-swift | 15:57 | |
| *** sams-gleb has quit IRC | 16:01 | |
| *** gyee has joined #openstack-swift | 16:02 | |
| *** rcernin has quit IRC | 16:02 | |
| notmyname | 10 days and 144 messages on the epic ML thread. we're slacking! down to only 14 per day. but maybe since it's morning in NA we'll get some more activity in the next few hours ;-) | 16:04 |
| *** esker has joined #openstack-swift | 16:08 | |
| *** lyrrad has joined #openstack-swift | 16:10 | |
| *** bill_az has joined #openstack-swift | 16:18 | |
| *** dmorita has joined #openstack-swift | 16:19 | |
| *** rledisez has quit IRC | 16:21 | |
| *** rledisez has joined #openstack-swift | 16:22 | |
| *** rledisez has quit IRC | 16:22 | |
| *** jmccarthy1 has quit IRC | 16:24 | |
| *** pdardeau has joined #openstack-swift | 16:25 | |
| *** esker has quit IRC | 16:25 | |
| *** nadeem has quit IRC | 16:26 | |
| *** klrmn has quit IRC | 16:29 | |
| *** lakshmiS has joined #openstack-swift | 16:31 | |
| *** asettle has quit IRC | 16:33 | |
| *** asettle has joined #openstack-swift | 16:33 | |
| *** asettle has quit IRC | 16:33 | |
| *** natarej has joined #openstack-swift | 16:38 | |
| *** pdardeau has quit IRC | 16:39 | |
| *** arch-nemesis has joined #openstack-swift | 16:40 | |
| *** suyash has joined #openstack-swift | 16:40 | |
| *** mvk has quit IRC | 16:41 | |
| *** jistr has quit IRC | 16:42 | |
| *** daemontool_ has joined #openstack-swift | 16:45 | |
| *** daemontool has quit IRC | 16:48 | |
| *** kozhukalov has quit IRC | 16:56 | |
| *** catintheroof has joined #openstack-swift | 16:59 | |
| *** catintheroof has quit IRC | 17:00 | |
| *** catintheroof has joined #openstack-swift | 17:00 | |
| *** CaioBrentano has joined #openstack-swift | 17:01 | |
| tdasilva | timburke: re patch 307929, why does x-auth-token help trace the "activities". what about transaction_id? | 17:02 |
| patchbot | tdasilva: https://review.openstack.org/#/c/307929/ - swift - Include X-Auth-Token in all versioned_writes subre... | 17:02 |
| *** natarej has quit IRC | 17:03 | |
| notmyname | FYI, next tuesday (May 17) is Swift's 6th birthday | 17:03 |
| notmyname | (measured from when it was first in production) | 17:04 |
| *** natarej has joined #openstack-swift | 17:04 | |
| openstackgerrit | Alistair Coles proposed openstack/swift: crypto - wrap random body key https://review.openstack.org/313640 | 17:06 |
| *** acoles is now known as acoles_ | 17:08 | |
| *** nadeem has joined #openstack-swift | 17:08 | |
| *** hezhiqiang has joined #openstack-swift | 17:14 | |
| *** kozhukalov has joined #openstack-swift | 17:14 | |
| *** can8dnSix has quit IRC | 17:19 | |
| *** Jeffrey4l has quit IRC | 17:25 | |
| *** pdardeau has joined #openstack-swift | 17:37 | |
| openstackgerrit | Samuel Merritt proposed openstack/swift: Make info caching work across subrequests https://review.openstack.org/280977 | 17:37 |
| *** hseipp has quit IRC | 17:39 | |
| openstackgerrit | Samuel Merritt proposed openstack/swift: Fix up get_account_info and get_container_info https://review.openstack.org/280978 | 17:41 |
| *** mvk has joined #openstack-swift | 17:43 | |
| *** mvk_ has joined #openstack-swift | 17:48 | |
| *** dmorita has quit IRC | 17:48 | |
| *** mkrcmari__ has joined #openstack-swift | 17:50 | |
| *** dmorita has joined #openstack-swift | 17:50 | |
| *** mvk has quit IRC | 17:52 | |
| *** dmorita has quit IRC | 17:52 | |
| *** dmorita has joined #openstack-swift | 17:52 | |
| *** mingdang1 has joined #openstack-swift | 17:53 | |
| *** mvk has joined #openstack-swift | 17:53 | |
| *** mvk_ has quit IRC | 17:53 | |
| *** mkrcmari__ has quit IRC | 17:55 | |
| *** klrmn has joined #openstack-swift | 17:55 | |
| *** mingdang1 has quit IRC | 17:57 | |
| *** sams-gleb has joined #openstack-swift | 17:58 | |
| *** mvk_ has joined #openstack-swift | 17:59 | |
| *** sams-gleb has quit IRC | 18:02 | |
| *** mvk has quit IRC | 18:02 | |
| *** nadeem has quit IRC | 18:08 | |
| *** dmorita has quit IRC | 18:09 | |
| *** dmorita has joined #openstack-swift | 18:10 | |
| *** klamath has joined #openstack-swift | 18:14 | |
| *** mkrcmari__ has joined #openstack-swift | 18:15 | |
| *** pdardeau has quit IRC | 18:19 | |
| *** mvk_ has quit IRC | 18:19 | |
| *** klamath has quit IRC | 18:20 | |
| *** naire has joined #openstack-swift | 18:34 | |
| *** esker has joined #openstack-swift | 18:35 | |
| timburke | tdasilva: suppose you're investigating a 500 but you don't have much context. if you grep for the single transaction id, you'll see info about that one client request, but you may not know much about the state of the backend. if you grep for the auth token, you'll have much more context for what the user was doing and whether there were compounding failures | 18:36 |
| timburke | at least, that's the reasoning i could come up with for setting it at https://github.com/openstack/swift/blob/master/swift/common/middleware/versioned_writes.py#L261 and https://github.com/openstack/swift/blob/master/swift/common/middleware/versioned_writes.py#L285 | 18:36 |
| timburke | (since those are pre-authed requests, the token isn't necessary nor used for anything but logging AFAICT) | 18:37 |
| notmyname | timburke: isn't the same trans_id used on those sub-requests? | 18:38 |
| timburke | notmyname: yes, better be | 18:38 |
| *** esker has quit IRC | 18:39 | |
| notmyname | so why does grep'ing for the transid not give enough info? | 18:39 |
| *** mkrcmari__ has quit IRC | 18:39 | |
| *** esker has joined #openstack-swift | 18:39 | |
| *** mkrcmari__ has joined #openstack-swift | 18:39 | |
| tdasilva | if I understood correctly, he wants to be able to group infor on multiple requests, more like a session | 18:42 |
| timburke | so, in the context of patch 307929: (1) it seems silly that sometimes we include auth-token in subrequests and sometimes we don't and (2) since we were already including it in some of them with no ill affect, i figured it'd be preferable to include it in all of them | 18:42 |
| patchbot | timburke: https://review.openstack.org/#/c/307929/ - swift - Include X-Auth-Token in all versioned_writes subre... | 18:42 |
| timburke | i don't really care whether we include it in all or none of the subrequests, but i was searching for a reason that it was *ever* included and that was the best i could come up with | 18:42 |
| notmyname | ok, yeah, I can buy the consistency argument more. (although I may have default to consistently not putting them in subrequests) | 18:42 |
| tdasilva | timburke: just wondering if based on your reasoning it then would make sense to add to this list? https://github.com/openstack/swift/blob/master/swift/common/wsgi.py#L1095 | 18:43 |
| *** ChubYann has joined #openstack-swift | 18:45 | |
| timburke | that may be better...we've already got things like HTTP_X_USER_ID and swift.authorize in there... | 18:45 |
| timburke | would there ever be a reason to create a subrequest that (1) isn't pre-authed and (2) *doesn't* use the current user's creds? | 18:46 |
| *** ntata- has quit IRC | 18:47 | |
| notmyname | (1) if the subrequest originates before auth in the pipeline and needs to be gated on the authorization of the user | 18:47 |
| notmyname | (2) can't imagine one--or rather, not that's come up (I *could* imagine one, I guess) | 18:48 |
| *** pdardeau has joined #openstack-swift | 18:49 | |
| *** naire has quit IRC | 18:54 | |
| *** naire has joined #openstack-swift | 18:57 | |
| *** sams-gleb has joined #openstack-swift | 18:59 | |
| *** mvk_ has joined #openstack-swift | 19:03 | |
| *** sams-gleb has quit IRC | 19:04 | |
| *** mkrcmari__ has quit IRC | 19:06 | |
| *** daemontool_ has quit IRC | 19:06 | |
| openstackgerrit | Richard Hawkins proposed openstack/swift: a well-commented skeleton middleware https://review.openstack.org/281885 | 19:11 |
| mmotiani | I pulled the code from master and getting the errors while running functional tests. This is error: http://paste.openstack.org/show/497124/ | 19:17 |
| *** yarkot has joined #openstack-swift | 19:17 | |
| mmotiani | I am not sure if I am the only one who is receiving this. | 19:17 |
| mmotiani | I ran functional test on SAIO as well as on vsaio | 19:18 |
| timburke | mmotiani: the 503s make me think a server didn't start up properly. the copy failure *may* have something to do with the recent copy-as-middleware patch; if so, i think it may be a pipeline issue | 19:20 |
| timburke | i'd tackle the 503s first, though. did all the services start up properly? what do logs for those transactions say? | 19:21 |
| mmotiani | Yes, all the services are started properly | 19:23 |
| mmotiani | timburke: logs are clear and the services are started properly without any issue | 19:25 |
| timburke | ...so what caused the 503? typically i'll see that when the proxy can't contact enough storage nodes to give a sensible response to the client. are you seeing timeouts? | 19:26 |
| *** naire has left #openstack-swift | 19:27 | |
| *** pdardeau has quit IRC | 19:35 | |
| *** nadeem has joined #openstack-swift | 19:49 | |
| *** asettle has joined #openstack-swift | 19:59 | |
| *** asettle has quit IRC | 19:59 | |
| *** dmk0202 has joined #openstack-swift | 20:03 | |
| *** raildo is now known as raildo-afk | 20:13 | |
| *** dmk0202 has quit IRC | 20:22 | |
| *** mvk_ has quit IRC | 20:22 | |
| *** nadeem has quit IRC | 20:23 | |
| *** nadeem has joined #openstack-swift | 20:24 | |
| *** hezhiqiang has quit IRC | 20:24 | |
| *** lcurtis has joined #openstack-swift | 20:26 | |
| *** dmorita has quit IRC | 20:29 | |
| *** dmorita has joined #openstack-swift | 20:30 | |
| openstackgerrit | Tim Burke proposed openstack/swift: Use the same key for memcache and env['swift.infocache'] https://review.openstack.org/310566 | 20:33 |
| *** bill_az has quit IRC | 20:35 | |
| *** sheel has quit IRC | 20:35 | |
| *** ametts has quit IRC | 20:36 | |
| *** bill_az has joined #openstack-swift | 20:45 | |
| *** daemontool has joined #openstack-swift | 20:46 | |
| *** silor has quit IRC | 20:48 | |
| *** daemontool has quit IRC | 20:52 | |
| *** sams-gleb has joined #openstack-swift | 21:00 | |
| *** geaaru has quit IRC | 21:01 | |
| *** sams-gleb has quit IRC | 21:06 | |
| jrichli | mmotiani: I have reproduced your errors and will take a look as well | 21:09 |
| mmotiani | timburke: I don't know yet.. I am still looking into it and trying to figure that out. | 21:09 |
| mmotiani | jrichli, thats great, thanks! | 21:10 |
| *** awelleck has quit IRC | 21:12 | |
| *** dmorita has quit IRC | 21:26 | |
| *** lakshmiS has quit IRC | 21:26 | |
| *** resker has joined #openstack-swift | 21:43 | |
| *** mmcardle has quit IRC | 21:44 | |
| *** esker has quit IRC | 21:46 | |
| *** esker has joined #openstack-swift | 21:47 | |
| *** resker has quit IRC | 21:48 | |
| *** dmorita has joined #openstack-swift | 21:50 | |
| *** hrou has quit IRC | 21:50 | |
| *** pdardeau has joined #openstack-swift | 21:52 | |
| *** resker has joined #openstack-swift | 21:52 | |
| *** dmk0202 has joined #openstack-swift | 21:53 | |
| *** esker has quit IRC | 21:53 | |
| *** daemontool has joined #openstack-swift | 21:59 | |
| *** sambhare has joined #openstack-swift | 22:01 | |
| openstackgerrit | Brian Ober proposed openstack/swift: Bypass Account Mgmt Functional Cases When Disabled https://review.openstack.org/316303 | 22:03 |
| *** suyash has quit IRC | 22:05 | |
| *** CaioBrentano has quit IRC | 22:06 | |
| *** dmk0202 has quit IRC | 22:06 | |
| *** joeljwright has quit IRC | 22:12 | |
| *** NM has quit IRC | 22:14 | |
| *** natarej_ has joined #openstack-swift | 22:14 | |
| *** natarej has quit IRC | 22:17 | |
| *** mingdang1 has joined #openstack-swift | 22:22 | |
| *** _JZ_ has quit IRC | 22:22 | |
| *** pdardeau has left #openstack-swift | 22:23 | |
| *** sheel has joined #openstack-swift | 22:29 | |
| *** daemontool has quit IRC | 22:31 | |
| *** dmorita has quit IRC | 22:35 | |
| *** dmorita has joined #openstack-swift | 22:36 | |
| *** vint_bra has quit IRC | 22:39 | |
| jrichli | mmotiani : from my testing, I know that my SAIO functests run successfully after copy-as-mw landed, but fails after patch 288011. But the gate functests passed for that and all the following patches, so I am going to try updating my dependencies and see if that helps. | 22:39 |
| patchbot | jrichli: https://review.openstack.org/#/c/288011/ - swift - Allow fallocate_reserve to be a percentage (MERGED) | 22:39 |
| *** dmorita has quit IRC | 22:40 | |
| *** dmorita has joined #openstack-swift | 22:40 | |
| *** nadeem has quit IRC | 22:41 | |
| *** mingdang1 has quit IRC | 22:43 | |
| *** mingdang1 has joined #openstack-swift | 22:44 | |
| *** catintheroof has quit IRC | 22:49 | |
| timburke | hrm. i've got them passing on fc7c515f, after that patch... i wonder if the default fallocate_reserve changing from 0 to 1% is the cause? how does an object server respond when we raise ENOSPC there? | 22:55 |
| *** lcurtis has quit IRC | 22:56 | |
| jrichli | timbruke : are you saying you do see the same failures after the patch you referenced? | 22:58 |
| jrichli | timburke ^^ | 22:59 |
| timburke | jrichli: i don't, but i also tore down and rebuilt the machine recently. i'm thinking that the disks somewhat but not entirely full, so the object servers are responding with a 507 (according to notmyname) and the proxy collects all of these 5xx series responses and returns a 503 | 23:00 |
| timburke | disks are* somewhat... | 23:01 |
| timburke | jrichli: mmotiani: what does your df output look like? | 23:01 |
| jrichli | yes ... I saw the "ERROR Insufficient Storage" error | 23:01 |
| jrichli | 4& | 23:04 |
| jrichli | one sec | 23:04 |
| *** jrichli has quit IRC | 23:05 | |
| *** jlhinson has quit IRC | 23:05 | |
| *** mingdang_ has joined #openstack-swift | 23:08 | |
| *** mingdang1 has quit IRC | 23:11 | |
| *** jrichli has joined #openstack-swift | 23:11 | |
| jrichli | I think my bouncer died. | 23:11 |
| jrichli | so .. I have 4% usage on /mnt/sdb1. But when i run test/functional/tests.py:TestFile.testFileSizeLimit, I get 503 | 23:13 |
| jrichli | oops, wrong channel | 23:14 |
| *** jrichli has quit IRC | 23:14 | |
| *** jrichli has joined #openstack-swift | 23:14 | |
| timburke | jrichli: hrm. weird... i'm not sure then... | 23:14 |
| *** dmorita_ has joined #openstack-swift | 23:17 | |
| jrichli | well, im gonna go have dinner. will look later tonight. i'm interested to know if there are others who see these errors. i still have some updating of dependencies i can try. | 23:18 |
| *** jrichli has quit IRC | 23:18 | |
| *** mingdang_ has quit IRC | 23:19 | |
| *** dmorita has quit IRC | 23:20 | |
| *** mingdang_ has joined #openstack-swift | 23:22 | |
| *** ekarlso has quit IRC | 23:23 | |
| *** mmcardle has joined #openstack-swift | 23:23 | |
| *** arch-nemesis has quit IRC | 23:26 | |
| *** ekarlso has joined #openstack-swift | 23:32 | |
| *** nadeem has joined #openstack-swift | 23:34 | |
| *** mingdang_ has quit IRC | 23:36 | |
| *** mingdang1 has joined #openstack-swift | 23:40 | |
| *** nadeem has quit IRC | 23:43 | |
| *** mmcardle has quit IRC | 23:46 | |
| *** gyee has quit IRC | 23:49 | |
| *** mingdang1 has quit IRC | 23:51 | |
| *** mingdang1 has joined #openstack-swift | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!