| *** suyash has quit IRC | 00:05 | |
| *** arch-nemesis has quit IRC | 00:28 | |
| *** zul has joined #openstack-swift | 00:28 | |
| *** lyrrad has quit IRC | 00:29 | |
| *** dmorita has quit IRC | 00:46 | |
| *** diogogmt has joined #openstack-swift | 00:52 | |
| *** klamath_ has quit IRC | 00:58 | |
| *** klamath has joined #openstack-swift | 00:58 | |
| *** diogogmt has quit IRC | 01:06 | |
| *** nadeem has joined #openstack-swift | 01:14 | |
| *** Jeffrey4l has quit IRC | 01:24 | |
| *** dmorita has joined #openstack-swift | 01:26 | |
| *** dmorita has quit IRC | 01:28 | |
| *** dmorita_ has joined #openstack-swift | 01:28 | |
| *** dmorita_ has quit IRC | 01:30 | |
| *** nadeem has quit IRC | 01:31 | |
| *** dmorita has joined #openstack-swift | 01:32 | |
| *** Jeffrey4l has joined #openstack-swift | 01:38 | |
| *** dmorita has quit IRC | 02:01 | |
| *** dmorita has joined #openstack-swift | 02:03 | |
| *** dmorita has quit IRC | 02:06 | |
| *** dmorita has joined #openstack-swift | 02:08 | |
| *** garthb has joined #openstack-swift | 02:10 | |
| *** _JZ_ has joined #openstack-swift | 02:24 | |
| *** Suyash has joined #openstack-swift | 02:25 | |
| *** dmorita has quit IRC | 02:33 | |
| *** dmorita has joined #openstack-swift | 02:47 | |
| *** dmorita has quit IRC | 02:47 | |
| *** garthb has quit IRC | 02:52 | |
| *** dmorita has joined #openstack-swift | 02:54 | |
| *** dmorita has quit IRC | 02:54 | |
| *** dmorita has joined #openstack-swift | 02:54 | |
| *** dmorita has quit IRC | 02:55 | |
| *** _JZ_ has quit IRC | 03:02 | |
| *** jamielennox is now known as jamielennox|away | 03:03 | |
| *** gyee has quit IRC | 03:07 | |
| *** dmorita has joined #openstack-swift | 03:08 | |
| *** dmorita has quit IRC | 03:08 | |
| *** jamielennox|away is now known as jamielennox | 03:19 | |
| *** dmorita has joined #openstack-swift | 03:32 | |
| *** dmorita has quit IRC | 03:37 | |
| *** furlongm has quit IRC | 03:41 | |
| *** furlongm has joined #openstack-swift | 03:41 | |
| *** dmorita has joined #openstack-swift | 04:17 | |
| *** dmorita has quit IRC | 04:17 | |
| *** links has joined #openstack-swift | 04:20 | |
| *** psachin has joined #openstack-swift | 04:35 | |
| *** dmorita has joined #openstack-swift | 04:36 | |
| *** tqtran has quit IRC | 04:36 | |
| *** dmorita has quit IRC | 04:37 | |
| *** Suyash has quit IRC | 04:41 | |
| *** dmorita has joined #openstack-swift | 04:41 | |
| *** dmorita has quit IRC | 04:44 | |
| *** garthb has joined #openstack-swift | 04:57 | |
| *** garthb has quit IRC | 04:58 | |
| *** garthb has joined #openstack-swift | 04:58 | |
| *** garthb has quit IRC | 05:07 | |
| *** SkyRocknRoll has joined #openstack-swift | 05:07 | |
| *** ChubYann has quit IRC | 05:11 | |
| *** nadeem has joined #openstack-swift | 05:18 | |
| *** ppai has joined #openstack-swift | 05:24 | |
| *** links has quit IRC | 05:27 | |
| *** links has joined #openstack-swift | 05:28 | |
| *** dmorita has joined #openstack-swift | 05:45 | |
| *** dmorita has quit IRC | 05:49 | |
| *** dmorita has joined #openstack-swift | 06:06 | |
| *** dmorita has quit IRC | 06:06 | |
| *** rcernin has joined #openstack-swift | 06:23 | |
| *** pcaruana has joined #openstack-swift | 06:29 | |
| *** klrmn has quit IRC | 06:36 | |
| *** SkyRocknRoll has quit IRC | 06:42 | |
| *** sheel has joined #openstack-swift | 06:50 | |
| *** SkyRocknRoll has joined #openstack-swift | 06:56 | |
| *** silor has joined #openstack-swift | 07:06 | |
| *** geaaru has joined #openstack-swift | 07:09 | |
| *** hseipp has joined #openstack-swift | 07:10 | |
| *** tesseract has joined #openstack-swift | 07:10 | |
| *** rledisez has joined #openstack-swift | 07:12 | |
| *** hogepodge has quit IRC | 07:12 | |
| *** SkyRocknRoll has quit IRC | 07:26 | |
| *** silor has quit IRC | 07:27 | |
| *** SkyRocknRoll has joined #openstack-swift | 07:29 | |
| *** dmk0202 has joined #openstack-swift | 07:32 | |
| *** dmk0202 has quit IRC | 07:33 | |
| *** nadeem has quit IRC | 07:41 | |
| *** rcernin has quit IRC | 07:45 | |
| *** SkyRocknRoll has quit IRC | 07:52 | |
| *** dmorita has joined #openstack-swift | 08:07 | |
| *** rcernin has joined #openstack-swift | 08:10 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/swift: [WIP] Testing latest u-c https://review.openstack.org/318441 | 08:11 |
|---|---|---|
| *** dmorita has quit IRC | 08:11 | |
| *** thurloat has quit IRC | 08:21 | |
| *** ndk_ has quit IRC | 08:21 | |
| *** jlvillal has quit IRC | 08:22 | |
| *** jlvillal has joined #openstack-swift | 08:22 | |
| *** thurloat has joined #openstack-swift | 08:23 | |
| *** ndk_ has joined #openstack-swift | 08:23 | |
| *** dmk0202 has joined #openstack-swift | 08:27 | |
| *** daemontool has joined #openstack-swift | 08:28 | |
| *** jmccarthy1 has quit IRC | 08:36 | |
| *** mkrcmari__ has joined #openstack-swift | 08:36 | |
| *** Jeffrey4l has quit IRC | 08:37 | |
| *** jlvillal has quit IRC | 08:37 | |
| *** jmccarthy has joined #openstack-swift | 08:38 | |
| *** tamizh_geek_ has joined #openstack-swift | 08:39 | |
| *** kmARC has quit IRC | 08:39 | |
| *** andymccr_ has joined #openstack-swift | 08:40 | |
| *** kmARC has joined #openstack-swift | 08:40 | |
| *** jlvillal has joined #openstack-swift | 08:42 | |
| *** daemontool_ has joined #openstack-swift | 08:42 | |
| *** mvk_ has quit IRC | 08:42 | |
| *** ekarlso has quit IRC | 08:42 | |
| *** tamizh_geek has quit IRC | 08:42 | |
| *** m3m0 has quit IRC | 08:42 | |
| *** furlongm has quit IRC | 08:42 | |
| *** andymccr has quit IRC | 08:43 | |
| *** furlongm has joined #openstack-swift | 08:43 | |
| *** mtreinish has quit IRC | 08:44 | |
| *** daemontool has quit IRC | 08:44 | |
| *** m3m0 has joined #openstack-swift | 08:45 | |
| *** nottrobin has quit IRC | 08:45 | |
| *** mtreinish has joined #openstack-swift | 08:46 | |
| *** nottrobin has joined #openstack-swift | 08:48 | |
| *** Jeffrey4l has joined #openstack-swift | 08:48 | |
| *** ekarlso has joined #openstack-swift | 08:51 | |
| *** ouchkernel has quit IRC | 09:03 | |
| *** ouchkernel has joined #openstack-swift | 09:04 | |
| *** openstackgerrit has quit IRC | 09:18 | |
| *** openstackgerrit has joined #openstack-swift | 09:18 | |
| *** acoles_ is now known as acoles | 09:22 | |
| *** acoles is now known as acoles_ | 09:23 | |
| *** SkyRocknRoll has joined #openstack-swift | 09:26 | |
| *** acoles_ is now known as acoles | 09:29 | |
| *** SkyRocknRoll has quit IRC | 09:34 | |
| *** geaaru has quit IRC | 09:39 | |
| acoles | kota_: hi. what is Kazuhiro's nick? | 09:49 |
| kota_ | acoles: m_kazuhiro | 09:50 |
| *** SkyRocknRoll has joined #openstack-swift | 09:50 | |
| kota_ | hello | 09:50 |
| acoles | thanks | 09:50 |
| acoles | oh, he's not in the channel, NM | 09:50 |
| kota_ | and he was back home today, probably he is back to online around Monday. | 09:50 |
| kota_ | acoles: He is not ready for irc bouncer yet :/ | 09:51 |
| acoles | ok. I just wanted to thank him for reviews on crypto doc :) | 09:51 |
| *** geaaru has joined #openstack-swift | 09:52 | |
| kota_ | acoles: :) | 09:54 |
| kota_ | acoles: quick question | 09:56 |
| kota_ | acoles: I'm not sure though, do we need http://paste.openstack.org/show/516801/ for patch 328204? | 09:57 |
| patchbot | kota_: https://review.openstack.org/#/c/328204/ - swift (feature/crypto-review) - Support for http footers - Replication and EC | 09:57 |
| acoles | kota_: looking... | 09:58 |
| kota_ | since notmyname is wondering why he got failed for functests, I've taken time to research around the diff of Putters. | 09:58 |
| *** jmccarthy has quit IRC | 09:58 | |
| kota_ | and I found the BaseObjectController returns conn.resp | 09:58 |
| kota_ | at master | 09:58 |
| kota_ | but at encryption code, we could return self.resp (i.e. Putter.resp instead) | 09:59 |
| *** jmccarthy has joined #openstack-swift | 10:00 | |
| kota_ | looking at the EC code, it sems well https://github.com/openstack/swift/blob/master/swift/proxy/controllers/obj.py#L1411-L1432 | 10:00 |
| kota_ | ec putter did it. | 10:00 |
| kota_ | I'm not sure, when the HTTPBufferedConnection.resp and Putter.resp updated while requeting. | 10:01 |
| kota_ | I think, Putter.resp is updated as getexpect() response for the first connection. | 10:01 |
| *** openstackgerrit has quit IRC | 10:03 | |
| *** openstackgerrit has joined #openstack-swift | 10:03 | |
| *** rcernin has quit IRC | 10:05 | |
| kota_ | ah... sorry, I didn't get (summrize) my thought. | 10:06 |
| acoles | kota_: thanks for digging into that, I'm not sure yet either, looks like we have not changed what is on master for EC, but we have possibly changed the replication case? can you put your paste link on gerrit as a comment and i will dig into it some more? | 10:08 |
| acoles | kota_: I need to work throught the final_phase (or not final_phase) condition carefully! | 10:09 |
| kota_ | acoles: ok, will do | 10:09 |
| acoles | kota_: and yeah, need to fugure out why notmyname sees the functional test problems - we have been running the gate functional test job with encryption and replication policy and I don't recall it ever failing. | 10:10 |
| acoles | s/fugure/figure/ | 10:10 |
| acoles | kota_: thanks | 10:11 |
| kota_ | acoles: quick answer, my thought might be way off targt. | 10:11 |
| kota_ | acoles: I added a assertion like "assert self.resp == self.conn.resp" but no unit tests failed. | 10:12 |
| kota_ | like this, http://paste.openstack.org/show/516804/ | 10:13 |
| acoles | kota_: is that good? - that means that the putter always gets a resp | 10:13 |
| kota_ | maybe? not sure. still need more deep dive. | 10:13 |
| acoles | heh. | 10:14 |
| acoles | oic, no the assertion proves that the putter.resp is in same state as conn but both could be None | 10:14 |
| kota_ | no? | 10:18 |
| kota_ | if both of them is None, putter will do either of conn.getexect() or conn getresponse() | 10:19 |
| kota_ | s/is/are/ | 10:19 |
| kota_ | but I'm not sure HTTPBufferedResponse override its __eq__ method? | 10:21 |
| kota_ | hmm...it doesn't look to override there, idk about parent class yet though. | 10:22 |
| kota_ | probably, I'm on wrong way. | 10:28 |
| kota_ | ah.. we might be missing HTTP_UNPROCESSABLE_ENTITY? at Putter._make_connetion? https://github.com/openstack/swift/blob/master/swift/proxy/controllers/obj.py#L688 | 10:39 |
| kota_ | but it seems another thing than what we are looking for... | 10:39 |
| *** permalac has joined #openstack-swift | 10:47 | |
| *** hogepodge has joined #openstack-swift | 10:54 | |
| kota_ | anyways, we need the HTTP_UNPROCESSABLE_ENTITY handling? | 11:06 |
| kota_ | I'm looking at the original bug it resolved, https://bugs.launchpad.net/swift/+bug/1516579 | 11:06 |
| openstack | Launchpad bug 1516579 in OpenStack Object Storage (swift) "PUTing a 0-byte object and supplying an Etag results in a 503 response" [High,Fix committed] - Assigned to Samuel Merritt (torgomatic) | 11:06 |
| kota_ | acoles: it might be good info, I found a bug between encryption-0 and encryption-1 (I'm not sure I hit actually notmyname's one though) | 11:19 |
| kota_ | acloes: probably we need http://paste.openstack.org/show/516819/ | 11:19 |
| kota_ | between them, we restored the bug 1516579 again. | 11:20 |
| openstack | bug 1516579 in OpenStack Object Storage (swift) "PUTing a 0-byte object and supplying an Etag results in a 503 response" [High,Fix committed] https://launchpad.net/bugs/1516579 - Assigned to Samuel Merritt (torgomatic) | 11:20 |
| kota_ | acoles: so if we tried to upload an empty object with invalid etag, it will make 500 at encryption-1. | 11:21 |
| kota_ | but 422 at encryption-0. | 11:21 |
| kota_ | I'm now trying to change a unit test to fail with current encryption-1 | 11:23 |
| kota_ | it looks weird, proxy log said like as "File "/usr/local/lib/python2.7/dist-packages/swift/proxy/controllers/obj.py", line 376, in _handle_response#012 etags.add(response.getheader('etag').strip('"'))#012AttributeError: 'NoneType' object has no attribute 'strip' (txn: txa1125c912fa2474cbd1c1-005763dc07) (client_ip: 127.0.0.1)" | 11:31 |
| kota_ | that means, conn.getresponse() after 422 for conn.getexpect() will return 2xx !??? | 11:31 |
| kota_ | interesting... | 11:32 |
| acoles | kota_: ok, I can see where we may have lost the HTTP_UNPROCESSABLE_ENTITY handling. How does the get returned in make_connection though, since no data has been sent yet? | 11:33 |
| kota_ | not sure, right now for the zero-byte PUT conversation of proxy -> eventlet (?) -> object-server | 11:35 |
| acoles | hmmm, guess something shortcuts when its a zero byte object maybe? | 11:35 |
| kota_ | but in fact, it's broken, right now. | 11:35 |
| kota_ | acoles: you can try via 'swift upload replica_container empty_file --headers 'Etag: foo'' | 11:36 |
| kota_ | with encryption-1 commit. | 11:36 |
| kota_ | it shows us 500 InternalError. | 11:36 |
| kota_ | acoles: I think so (about the shortcuts) | 11:37 |
| kota_ | I'm not sure if it hits notmyname's issue though. | 11:37 |
| kota_ | it seems when zero-bytes PUT comming in, object-server seems to return 422 immediately for getexpect() | 11:38 |
| kota_ | is it right? my answer is 'I'm not sure' | 11:39 |
| acoles | kota_: right. agree its broke, just trying to understand how it breaks, and why this test does not fail https://review.openstack.org/#/c/246010/2/test/unit/proxy/controllers/test_obj.py | 11:39 |
| patchbot | acoles: patch 246010 - swift - Fix 503 on zero-byte replicated PUT with incorrect... (MERGED) | 11:39 |
| acoles | kota_: the comment on that test ^^ suggests exactly that - the getexpect returns 422 | 11:39 |
| kota_ | acoles: http://paste.openstack.org/show/516825/ | 11:40 |
| kota_ | acoles: that one can heads up the issue, probably | 11:40 |
| kota_ | i don't know the None value for getresponse() is correct (or not). | 11:41 |
| kota_ | probably not correct, though. | 11:41 |
| kota_ | we might ask torgomatic for that, I think he has the knowlege around there. | 11:43 |
| kota_ | acoles:^^ | 11:43 |
| acoles | kota_: yes. I'm going to have to stare more at the code. good work kota_ , we definitely need to fix that. | 11:45 |
| * acoles away for lunch | 11:45 | |
| acoles | kota_: have a good weekend! | 11:45 |
| kota_ | acoles: ;) | 11:46 |
| *** rcernin has joined #openstack-swift | 11:47 | |
| *** Jeffrey4l has quit IRC | 11:48 | |
| *** Jeffrey4l has joined #openstack-swift | 11:48 | |
| kota_ | acoles: I added small note we ware tallking here to gerrit :D | 11:52 |
| kota_ | is heading back home | 11:52 |
| *** cdelatte has quit IRC | 12:03 | |
| *** dmk0202 has quit IRC | 12:05 | |
| *** jmccarthy has quit IRC | 12:10 | |
| *** jmccarthy has joined #openstack-swift | 12:19 | |
| *** cdelatte has joined #openstack-swift | 12:19 | |
| *** SkyRocknRoll has quit IRC | 12:22 | |
| *** silor has joined #openstack-swift | 12:24 | |
| *** cdelatte has quit IRC | 12:25 | |
| *** cdelatte has joined #openstack-swift | 12:30 | |
| *** silor1 has joined #openstack-swift | 12:32 | |
| *** pauloewerton has joined #openstack-swift | 12:34 | |
| *** silor has quit IRC | 12:35 | |
| *** silor1 is now known as silor | 12:35 | |
| *** silor has quit IRC | 12:38 | |
| *** silor has joined #openstack-swift | 12:39 | |
| *** cdelatte has quit IRC | 12:39 | |
| *** cdelatte has joined #openstack-swift | 12:41 | |
| *** silor1 has joined #openstack-swift | 12:44 | |
| *** silor has quit IRC | 12:45 | |
| *** silor1 is now known as silor | 12:45 | |
| *** klamath has quit IRC | 12:58 | |
| *** klamath has joined #openstack-swift | 12:58 | |
| *** psachin has quit IRC | 13:00 | |
| *** cdelatte has quit IRC | 13:03 | |
| *** mwheckmann has joined #openstack-swift | 13:14 | |
| *** lcurtis has joined #openstack-swift | 13:32 | |
| *** lcurtis has quit IRC | 13:32 | |
| *** _JZ_ has joined #openstack-swift | 13:35 | |
| *** cdelatte has joined #openstack-swift | 13:42 | |
| *** links has quit IRC | 13:46 | |
| *** cdelatte has quit IRC | 13:49 | |
| *** ametts has joined #openstack-swift | 13:50 | |
| *** dmorita has joined #openstack-swift | 13:52 | |
| *** cdelatte has joined #openstack-swift | 14:06 | |
| *** McMurlock1 has joined #openstack-swift | 14:12 | |
| *** cdelatte has quit IRC | 14:14 | |
| *** Jeffrey4l has quit IRC | 14:15 | |
| *** zul has quit IRC | 14:16 | |
| *** zul has joined #openstack-swift | 14:16 | |
| *** zul_ has joined #openstack-swift | 14:21 | |
| *** zul has quit IRC | 14:23 | |
| *** Jeffrey4l has joined #openstack-swift | 14:28 | |
| *** cdelatte has joined #openstack-swift | 14:31 | |
| *** cdelatte has quit IRC | 14:37 | |
| *** dmorita has quit IRC | 14:38 | |
| *** garthb has joined #openstack-swift | 14:46 | |
| *** cdelatte has joined #openstack-swift | 14:48 | |
| *** pcaruana has quit IRC | 14:54 | |
| *** diogogmt has joined #openstack-swift | 15:02 | |
| *** diogogmt has quit IRC | 15:04 | |
| *** urth has quit IRC | 15:05 | |
| *** tesseract has quit IRC | 15:05 | |
| *** rcernin has quit IRC | 15:07 | |
| *** urth has joined #openstack-swift | 15:07 | |
| *** ukaynar has joined #openstack-swift | 15:12 | |
| *** nadeem has joined #openstack-swift | 15:24 | |
| *** cdelatte has quit IRC | 15:24 | |
| *** cdelatte has joined #openstack-swift | 15:28 | |
| *** klrmn has joined #openstack-swift | 15:29 | |
| *** ppai has quit IRC | 15:31 | |
| *** nadeem has quit IRC | 15:38 | |
| *** nadeem has joined #openstack-swift | 15:38 | |
| *** arch-nemesis has joined #openstack-swift | 15:42 | |
| timburke | acoles: kota_: ah, that makes sense for why we care about HTTP_UNPROCESSABLE_ENTITY -- if we've included in the headers the fact that zero bytes will be sent, then we don't need to send the (non-existent) body for the object server to give a final response. i was wondering about that back around https://review.openstack.org/#/c/328204/2/swift/proxy/controllers/obj.py@1563 ... | 16:02 |
| patchbot | timburke: patch 328204 - swift (feature/crypto-review) - Support for http footers - Replication and EC | 16:02 |
| *** permalac has quit IRC | 16:03 | |
| acoles | timburke: yes, I remember your comment a few days back. I think what happens is that for zero byte body the proxy doesn't send expect 100 continue, so the obj server does not send a 100 continue response, goes ahead and reads (empty) body and the first response back is the 422. | 16:06 |
| acoles | kota ^^ that is the "shortcut" we were wondering about | 16:07 |
| acoles | maybe torgomatic_ will put me right | 16:07 |
| timburke | even if we *did* send the Expect header, i wouldn't be at all surprised if the object-server *ignored* it -- it had everything it needed to know | 16:07 |
| *** dmk0202 has joined #openstack-swift | 16:09 | |
| acoles | timburke: afaict it is eventlet wsgi that decides to send the response when input is first read https://github.com/eventlet/eventlet/blob/master/eventlet/wsgi.py#L122-L125 | 16:10 |
| acoles | good news is I think I figured out why the unit test did not fail due to that missing check on HTTP_UNPROCESSABLE_ENTITY | 16:11 |
| acoles | now I gotta make it fail, then fix the bug | 16:11 |
| acoles | notmyname: I have been running func tests with repl policy on commit 612ee2 for several hours and not yet seen a test fail :( | 16:14 |
| openstackgerrit | Petr Kovar proposed openstack/swift: Add install-guide for swift https://review.openstack.org/330070 | 16:16 |
| *** arcimboldo has joined #openstack-swift | 16:17 | |
| notmyname | good morning | 16:18 |
| notmyname | I'll be diving into the functests errors I was getting | 16:19 |
| notmyname | I'm using real drives in my SAIO (instead of loopback), so maybe that's a difference that's showing up | 16:19 |
| *** ukaynar has quit IRC | 16:21 | |
| jrichli | notmyname: I know you saw this without crypto in pipeline. have you seen this with crypto in pipeline? | 16:28 |
| acoles | notmyname: I don't use loopback, but a virtual disk. would be great to figure out what's causing your failures. | 16:28 |
| *** klrmn has quit IRC | 16:28 | |
| *** SkyRocknRoll has joined #openstack-swift | 16:29 | |
| notmyname | jrichli: yeah, that's when I started going through each patch set individually. I started with the last patch, saw the errors, then went back. but on the 2nd patch there's not even any crypto middleware to add to the pipeline | 16:30 |
| acoles | notmyname: I'm going to work through that patch again in case anything is awry - Kota has spotted one regression that I'll have fixed in next version. | 16:30 |
| notmyname | acoles: ok, well then maybe that's not it. do you have fallocate or mount check set? (I have both turned on) | 16:31 |
| notmyname | not that I think that's the problem necessarily since everything passes on master. very confusing | 16:31 |
| jrichli | oh, right. of course | 16:31 |
| acoles | notmyname: mount_check = false | 16:32 |
| acoles | disable_fallocate = true | 16:32 |
| acoles | I'll flip them | 16:32 |
| *** dmorita has joined #openstack-swift | 16:33 | |
| -openstackstatus- NOTICE: zuul was restarted for a software upgrade; events between 16:08 and 16:30 were missed, please recheck any changes uploaded during that time | 16:34 | |
| *** dmorita has quit IRC | 16:34 | |
| *** dmorita has joined #openstack-swift | 16:34 | |
| *** lyrrad has joined #openstack-swift | 16:34 | |
| acoles | notmyname: testFileSizeLimit -> 503 ?? | 16:39 |
| notmyname | yes. often that's the first thing that fails, and starts nodes being error limited, which causes subsequent tests to fail | 16:39 |
| acoles | i saw some teardown errors too | 16:40 |
| notmyname | ah? interesting. I hadn't been able to find those, yet | 16:40 |
| acoles | ok, well that's progress of a sort i.e. reproduced | 16:41 |
| notmyname | and you saw that when you turned on mount check and fallocate? | 16:42 |
| *** McMurlock1 has quit IRC | 16:42 | |
| acoles | notmyname: i was running with nosetests -x so the package teardown would be attempted immediately after that test failed | 16:42 |
| notmyname | oh ok | 16:42 |
| acoles | just enabled fallocate (one change at a time!) | 16:42 |
| *** gyee has joined #openstack-swift | 16:42 | |
| acoles | I'll try without -x and see what accumulates | 16:42 |
| notmyname | I have disable_fallocate = false and fallocate_reserve = 0 | 16:43 |
| notmyname | the teardown errors are because the nodes are error limited | 16:46 |
| arcimboldo | I have a question: is there a way to give read access to a swift container based on the *source IP*? | 16:48 |
| notmyname | arcimboldo: no. although that wouldn't be too complicated to add (nor is it simple, either). what are you wanting to do? | 16:51 |
| acoles | notmyname: I don't have reserve set so default. full run I see 18 fail/4 errors no teardown errors | 16:51 |
| arcimboldo | it's not a concrete use case yet, but we have a private cloud (university) but we have users that might upload datasets to swift, and give access within the university premises to anyone | 16:52 |
| arcimboldo | however, the data cannot be public (for "reasons") | 16:52 |
| *** SkyRocknRoll has quit IRC | 16:52 | |
| *** SkyRocknRoll has joined #openstack-swift | 16:52 | |
| *** daemontool_ has quit IRC | 16:52 | |
| arcimboldo | If I could use the IP, I could avoid creating a tenant for each user who doesn't really need to use openstack and doesn't know how to use the swift cli | 16:53 |
| arcimboldo | we can just give them the URL | 16:53 |
| *** rledisez has quit IRC | 16:53 | |
| arcimboldo | I imagine this should be implemented with a middleware and a new ACL entry like ".ip:x.x.x.x/n"? | 16:55 |
| notmyname | yeah, I'd imagine some ACL format that looks like that | 16:57 |
| notmyname | but it would likely go in existing auth middleware (or acl.py) instead of something new | 16:58 |
| notmyname | and the other thing to consider is which source IP to use. swift tracks 2 of them. the last hop and the actual client IP (if known) | 16:58 |
| notmyname | (think of a load balancer forwarding requests and being able to have some LB ports tracked differently. also most LBs will add x-forwaded-for or similar) | 16:59 |
| arcimboldo | of course you have to use the client ip, our haproxy needed a configuration option for that, but i can see swift has it in the request | 17:00 |
| *** sheel has quit IRC | 17:05 | |
| *** chsc has joined #openstack-swift | 17:09 | |
| *** mvk_ has joined #openstack-swift | 17:11 | |
| *** mkrcmari__ has quit IRC | 17:15 | |
| notmyname | acoles: FWIW, if I disable fallocate, everything works | 17:16 |
| acoles | notmyname: ack | 17:17 |
| notmyname | my next step: why does fallocate work n master and fail on 612ee2e65c44b71e539e978ccde93ea81bb9135a | 17:19 |
| klamath | what are some best practices for swift proxy server in a multi-region swift environment. Ive noticed a lot of errors when the remote region is offline on the proxy tier, wondering if lowering connection timeouts will help | 17:21 |
| *** silor has quit IRC | 17:22 | |
| notmyname | use one memcache pool for each region. that will help | 17:30 |
| *** hseipp has quit IRC | 17:39 | |
| *** arcimboldo has quit IRC | 17:41 | |
| *** pauloewerton has quit IRC | 17:43 | |
| *** pauloewerton has joined #openstack-swift | 17:50 | |
| *** Suyash has joined #openstack-swift | 17:50 | |
| *** gyee has quit IRC | 17:55 | |
| *** garthb_ has joined #openstack-swift | 18:03 | |
| *** garthb has quit IRC | 18:05 | |
| *** klrmn has joined #openstack-swift | 18:06 | |
| *** nadeem has quit IRC | 18:10 | |
| openstackgerrit | Alistair Coles proposed openstack/swift: Support for http footers - Replication and EC https://review.openstack.org/328204 | 18:10 |
| openstackgerrit | Alistair Coles proposed openstack/swift: Make container update override headers persistent https://review.openstack.org/328205 | 18:10 |
| openstackgerrit | Alistair Coles proposed openstack/swift: Enable middleware to set metadata on object POST https://review.openstack.org/328206 | 18:11 |
| openstackgerrit | Alistair Coles proposed openstack/swift: Allow middleware to override metadata header checking https://review.openstack.org/328207 | 18:11 |
| openstackgerrit | Alistair Coles proposed openstack/swift: Enable object body and metadata encryption https://review.openstack.org/328208 | 18:11 |
| openstackgerrit | Alistair Coles proposed openstack/swift: Add encryption overview doc https://review.openstack.org/328209 | 18:11 |
| acoles | ^^ updated with today's reviews comments, still working on why func tests fail with fallocate enabled put pushing other changes before weekend. | 18:12 |
| notmyname | thanks | 18:12 |
| notmyname | acoles: I think I see why the file size limits are failing. however, I don't know why they're passing on master | 18:13 |
| notmyname | is there any more metadata stored with replicated objects after patch 328204? | 18:14 |
| patchbot | notmyname: https://review.openstack.org/#/c/328204/ - swift (feature/crypto-review) - Support for http footers - Replication and EC | 18:14 |
| acoles | notmyname: looking... | 18:15 |
| notmyname | if that patch is adding 111 bytes of metadata over-and-above master, I can explain it all (and know how to fix it) | 18:18 |
| notmyname | 111 or more | 18:19 |
| acoles | notmyname: there should not be afaict | 18:19 |
| acoles | notmyname: I'm curious... ;) | 18:20 |
| notmyname | https://www.youtube.com/watch?v=WWaLxFIVX1s | 18:20 |
| notmyname | (that was my reaction to "no extra metadata") | 18:20 |
| acoles | notmyname: 111 is a bad number https://en.wikipedia.org/wiki/Nelson_%28cricket%29 | 18:21 |
| notmyname | in testFileSizeLimit, we do 4(!) writes of nearly max file size that are supposed to succeed. it means that you need max_file_size * 4 - 111 bytes available | 18:21 |
| notmyname | turns out my drives on my SAIO are 8GB. and my max_file_size is 2GB | 18:21 |
| notmyname | guess what. if I set my max_file_size to 1GB, everything works! | 18:22 |
| notmyname | the bug IMO is that we don't delete the files before we test the next one | 18:22 |
| notmyname | so you need 4*max_file_size to actually pass the test, if you have fallocate enabled | 18:23 |
| notmyname | so my guess is that if the patch is adding some small amount of metadata, then that may have pushed it over the 111 buffer that I had on my drive | 18:23 |
| *** mkrcmari__ has joined #openstack-swift | 18:25 | |
| *** silor has joined #openstack-swift | 18:25 | |
| *** ChubYann has joined #openstack-swift | 18:27 | |
| *** mvk_ has quit IRC | 18:28 | |
| *** dmorita has quit IRC | 18:28 | |
| *** clayg has joined #openstack-swift | 18:28 | |
| *** ChanServ sets mode: +v clayg | 18:28 | |
| clayg | this is great, my cloud vm got rebooted while I was on vacation - so there's no scrollback - it's like you all just waited for me to come back from vacation! | 18:29 |
| *** dmorita has joined #openstack-swift | 18:29 | |
| notmyname | welcome back :-) | 18:29 |
| jrichli | clayg! welcome back. | 18:30 |
| klrmn | clayg: one of the questions i was planning on addressing later on was whether my VMs were likely to conflict with your VMs | 18:30 |
| clayg | klrmn: ohai! | 18:30 |
| clayg | ... wait - which vms? | 18:30 |
| *** dmorita has quit IRC | 18:30 | |
| klrmn | clayg: ones on RAX | 18:30 |
| clayg | klrmn: oh, no this is like a personal aws vm i use for irc - no conflict | 18:30 |
| klrmn | clayg: not that they exist yet....ah | 18:30 |
| clayg | unrelated - it was probably just unscheduled maintience at ec2 | 18:30 |
| clayg | for the other thing there's a "prefix" thing that allowed me at least to spin up to two indepenent clusters | 18:31 |
| notmyname | acoles: I can't see anything extra added either | 18:31 |
| notmyname | acoles: for now, I think I'll file a bug about the test and move on with the more general crypto testing | 18:31 |
| notmyname | (with a smaller max obj size and fallocate turned on) | 18:32 |
| jrichli | notmyname: glad you figured it out :-) | 18:33 |
| notmyname | oh no | 18:33 |
| notmyname | https://bugs.launchpad.net/swift/+bug/1062969 <--- IT'S FROM 2012!! HOW IS THIS STILL A THING?! | 18:34 |
| openstack | Launchpad bug 1062969 in OpenStack Object Storage (swift) "fallocate fails in test environment" [Undecided,Invalid] | 18:34 |
| acoles | notmyname: I am trying to reproduce your success | 18:34 |
| notmyname | I shall reopen and suggest fixing TestFile.testFileSizeLimit | 18:36 |
| *** SkyRocknRoll has quit IRC | 18:41 | |
| notmyname | wait | 18:41 |
| notmyname | acoles: this might be because of evil PRNG gods. don't we have tests that randomly generate metadata? | 18:42 |
| notmyname | I had a weird test run with this crypto patch where I ran just the testFileSizeLimit test and it worked. but some (most) of the time it fails for me on this patch | 18:43 |
| notmyname | 111 bytes is not much. it could be metadata from some other test (when running full tests.py, I always see a failure. it could be something really crazy like some extra dirs or something | 18:43 |
| acoles | notmyname: when i run only testFileSizeLimit it passes, which does suggest that some other cruft on the disk is causing the error to be induced. | 18:46 |
| acoles | if I run tests.py:TestFile then it fails | 18:46 |
| notmyname | acoles: do you have a similar setup to me where you have drives exactly 4x your max_file_size? | 18:47 |
| acoles | notmyname: no, that is what i am confused about | 18:47 |
| notmyname | ok | 18:47 |
| notmyname | I'm also confused by the consistent failing on this patch while passing on the first one in the chain | 18:47 |
| *** garthb_ has quit IRC | 18:48 | |
| *** garthb has joined #openstack-swift | 18:48 | |
| acoles | I have a single 32GB partition and max file size is default 5GB | 18:49 |
| acoles | so how did that ever pass? | 18:51 |
| notmyname | it would only fallocate 20GB | 18:53 |
| notmyname | or you mean that it would do that 3x and that would be 60GB? | 18:54 |
| acoles | yeah, 3x5x4=60 | 18:55 |
| notmyname | what do you mean by single partition? did you have that carved up into different drives? | 18:57 |
| notmyname | I dont' think the test is actually writing all that data. I think it's just reserving it via fallocate. checking on that... | 18:57 |
| *** geaaru has quit IRC | 18:58 | |
| acoles | notmyname: yep, just discovered that myself, no data sent just content-length set too limit +/- whatever | 19:02 |
| notmyname | yeah, but I'm stilly trying to follow the func client code. it *does* look like it's checking a response, and I don't know how the response would be 2xx when sending no data with a huge content-length header | 19:03 |
| acoles | ditto :) | 19:03 |
| notmyname | swift. how does it work? | 19:03 |
| notmyname | oh! | 19:04 |
| acoles | my proxy logs are showing 499 responses | 19:04 |
| acoles | swift. does it work? | 19:04 |
| notmyname | the timeout wrapper doesn't check the response of the .write() at all | 19:04 |
| acoles | :'( | 19:04 |
| notmyname | so the 499 is expected, but the timeout wrapper really just checks that the Timeout didn't fire | 19:05 |
| notmyname | so that test is only validating that we can send a reqeust with differnet content-length headers and get a response within 3 seconds | 19:06 |
| notmyname | yay! | 19:06 |
| notmyname | :-( | 19:06 |
| acoles | notmyname: well, it does assert that you get a response error in <3secs if content-length exceeds the constrainty | 19:07 |
| notmyname | right. and that's something, I guess | 19:07 |
| acoles | and maybe its deliberately simply checking the opposite i.e. allowed lengths don't get an outright rejection | 19:08 |
| acoles | otherwise we'd all be filling our SAIO drives, and helping people understand why func tests fail with default constraints ;) | 19:08 |
| acoles | but I'd +2A a comment in the test if nothing else | 19:09 |
| *** silor has quit IRC | 19:13 | |
| *** shakamunyi has joined #openstack-swift | 19:13 | |
| *** dmorita has joined #openstack-swift | 19:15 | |
| acoles | clayg is back! | 19:16 |
| acoles | notmyname: I meant my saio setup is like this http://docs.openstack.org/developer/swift/development_saio.html#using-a-partition-for-storage | 19:16 |
| acoles | notmyname: so we still don't know why this fails on crypto-review but not master, but your 111 theory is less confusing now I understand how the test works | 19:17 |
| notmyname | but you only have one of those? | 19:17 |
| acoles | yes | 19:17 |
| *** gyee has joined #openstack-swift | 19:17 | |
| notmyname | do you make loopback on that one partition? | 19:18 |
| acoles | no, symlink to node dirs, like the docs say. which is also why I didn't set mount_check=true in a rush | 19:21 |
| notmyname | ah, hmm | 19:22 |
| acoles | bad? | 19:22 |
| notmyname | no. except you can't test mount_check with it :-) | 19:22 |
| notmyname | tests pass in that config, so ship it! | 19:22 |
| acoles | well I can test mount_check and it works!! :) | 19:22 |
| notmyname | lol | 19:22 |
| *** garthb has quit IRC | 19:22 | |
| *** garthb has joined #openstack-swift | 19:23 | |
| * acoles adds 'build saio with multiple vdisks' to todo list | 19:23 | |
| notmyname | I've got virtualbox attaching 8 8GB drives to the VM, so my SAIO sees "real" drives. then I have normal single partition xfs on them | 19:23 |
| acoles | its on my todo list | 19:24 |
| openstackgerrit | John Dickinson proposed openstack/swift: added note to testFileSizeLimit functional test https://review.openstack.org/331318 | 19:25 |
| notmyname | ^ proposed to master. can land after crypto | 19:25 |
| openstackgerrit | John Dickinson proposed openstack/swift: added note to testFileSizeLimit functional test https://review.openstack.org/331318 | 19:25 |
| acoles | yay I got to do a review today :) | 19:27 |
| *** kmArc_ has joined #openstack-swift | 19:32 | |
| *** thumpba has joined #openstack-swift | 19:32 | |
| *** Jeffrey4l has quit IRC | 19:34 | |
| acoles | notmyname: so help me understand what is going on with this 111 theory...assume you start with an empty 8GB disk, 2GB maxfilesize, fallocate is enabled, the first PUT causes 2GB fallocate, but nothing is actually written. Ditto for second and third PUTs, then you see the fourth fail presumably because an extra 111 bytes crept in somewhere. But when do those fallocated bytes get free'd? cos on master that implies there 111 | 19:34 |
| acoles | bytes free and then there's a bunch of other tests that pass before anything is deleted. | 19:34 |
| acoles | notmyname: so assuming the fallocated space is free'd when the diskfile is closed (nothing was written) then I'm wondering if we're failing to close a connection and that delays the diskfile being closed??? | 19:35 |
| notmyname | they'd get freed when ...yeah | 19:35 |
| *** cdelatte has quit IRC | 19:40 | |
| *** kmArc_ has quit IRC | 19:40 | |
| notmyname | why did I just run all of TestFile and it passed? it's gotta be RNG issues | 19:42 |
| notmyname | oh, I lowered my max_file size | 19:42 |
| *** nadeem has joined #openstack-swift | 19:45 | |
| *** nadeem has quit IRC | 19:45 | |
| *** nadeem has joined #openstack-swift | 19:46 | |
| *** Jeffrey4l has joined #openstack-swift | 19:47 | |
| *** thumpba has quit IRC | 19:48 | |
| *** pauloewerton has quit IRC | 19:49 | |
| acoles | notmyname: oh oh oh... | 19:50 |
| acoles | think I can reproduce on master | 19:50 |
| notmyname | oh? | 19:51 |
| acoles | EC policy (4+2), fallocate enabled, set my maxfilesize bigger (cos the diskfiles are smaller with EC), boom! | 19:52 |
| acoles | oh hang on, I should disable fallocate and repeat | 19:52 |
| *** garthb_ has joined #openstack-swift | 19:52 | |
| *** garthb has quit IRC | 19:53 | |
| acoles | yep, fallocate disabled and tests.py:TestFile pass | 19:54 |
| notmyname | should be the same for replication, then, right? | 19:56 |
| notmyname | the issue is the test doing too much fallocate and some timing issue with when the tmp file is actually relaimed | 19:57 |
| *** raildo is now known as raildo-afk | 19:57 | |
| notmyname | the good news, I guess, is that I dropped my max_file_size and functests pass for me with all of crypto_review. (which is where this whole thing started) | 19:58 |
| acoles | agree, could be a timing thing ...or is it something we are not doing on the EC PUT path. My hunch was that the crypto patch migrates the replica PUT path to use the EC Putter code, so have we just stumbled across something that was lurking in the EC code path all along but only revealed by the tests using replica policy. | 19:59 |
| notmyname | hmm..that's an interesting idea. I had thought it was this patch that was breaking repl, since I saw EC working. but yeah, it could be something in the multipart handling | 20:03 |
| notmyname | so why do we have Putter and MIMEPutter now? why not just one that knows when to do the multipart? | 20:04 |
| acoles | notmyname: first, these tests should be using the Putter for replica policy. The MIMEPutter is only used if middleware has indicated that it has footers to send. So at this commit, replica policy PUTs should not be using mutlipart. | 20:06 |
| *** _JZ_ has quit IRC | 20:06 | |
| notmyname | right | 20:07 |
| acoles | why two classes? one knows how to do the multipart, the other doesn't. | 20:07 |
| *** thumpba has joined #openstack-swift | 20:07 | |
| notmyname | I was just wondering about the original decision about having two classes with minor differences instead of one that's slightly smarter | 20:07 |
| acoles | the obj controllers choose which they need. | 20:07 |
| acoles | OIC. TBH at one point I had 3 Putter classes and was really glad to get to 2 :) | 20:08 |
| notmyname | lol | 20:08 |
| notmyname | so I should just be grateful with what I'm given and be happy there's only 2! ;-) | 20:08 |
| acoles | its a good question and I'll look at it again, it was a while ago now | 20:08 |
| acoles | right! :) | 20:09 |
| acoles | so we could just use the MIMEPutter for all PUTs, but I thought that might be a step too far - this way replica PUTs stay non-multipart until you plugin crypto | 20:10 |
| acoles | the object server can handle multipart for any policy type | 20:11 |
| *** thumpba has quit IRC | 20:19 | |
| *** thumpba has joined #openstack-swift | 20:20 | |
| *** thumpba_ has joined #openstack-swift | 20:26 | |
| acoles | notmyname: on master, I tried with replica policy and increased maxfilesize and fallocate on, see tests in TestFile fail, not repeatbly. I suspect it is to do with the maxfilesize vs available filesystem space and as you said maybe some timing. need to understand what happens to the fallocated tmp file. | 20:26 |
| acoles | notmyname: I need to go now and eat/sleep. thanks for you help. I'd like to get to the bottom of this, clearly it happens on master too but I'd like to be sure we haven't made something worse on crypto | 20:27 |
| acoles | good night | 20:27 |
| notmyname | acoles: thanks for your help | 20:28 |
| *** thumpba has quit IRC | 20:29 | |
| *** mwheckmann has quit IRC | 20:30 | |
| *** acoles is now known as acoles_ | 20:31 | |
| *** McMurlock1 has joined #openstack-swift | 20:33 | |
| *** mvk_ has joined #openstack-swift | 20:34 | |
| *** mkrcmari__ has quit IRC | 20:38 | |
| *** McMurlock1 has quit IRC | 20:45 | |
| *** mkrcmari__ has joined #openstack-swift | 20:54 | |
| *** mvk_ has quit IRC | 20:57 | |
| *** mvk has joined #openstack-swift | 21:01 | |
| *** mkrcmari__ has quit IRC | 21:02 | |
| timburke | notmyname: catching up. re: "two classes with minor differences instead of one that's slightly smarter" -- always remember, "we should only do dumb things" is one option :P | 21:04 |
| notmyname | of course! | 21:05 |
| notmyname | timburke: you're going to make me sad on friday afternoon ;-) | 21:05 |
| timburke | nah, you're smiling. it's right there on the previous line | 21:06 |
| timburke | ;-( | 21:06 |
| *** daemontool has joined #openstack-swift | 21:12 | |
| *** thumpba_ has quit IRC | 21:12 | |
| *** ametts has quit IRC | 21:13 | |
| *** McMurlock1 has joined #openstack-swift | 21:17 | |
| *** McMurlock1 has quit IRC | 21:28 | |
| *** dmk0202 has quit IRC | 21:33 | |
| *** dmk0202 has joined #openstack-swift | 21:34 | |
| *** dmorita has quit IRC | 21:39 | |
| *** dmorita has joined #openstack-swift | 21:47 | |
| *** dmk0202 has quit IRC | 21:50 | |
| *** daemontool has quit IRC | 21:53 | |
| notmyname | FYI http://lists.openstack.org/pipermail/openstack-dev/2016-June/097668.html <-- newly-proposed Architecture working group "...charge would not be design by committee, but a place for architects to share their designs and gain support across projects to move forward with and ratify architectural decisions." | 22:06 |
| zaitcev | My opinion of architects is not particularly high, but I reserve the right to be persuaded otherwise. | 22:12 |
| *** nadeem has quit IRC | 22:22 | |
| openstackgerrit | Tim Burke proposed openstack/swift: Always set swift processes to use UTC https://review.openstack.org/331369 | 22:33 |
| timburke | i have no idea what that^ may break, or what may have been broken before that :-( | 22:33 |
| timburke | actually, that's a lie. i know of at least one thing that was broken before that, and referenced it in the commit (though not the fact that it bit swift3) | 22:37 |
| *** mvk_ has joined #openstack-swift | 22:39 | |
| *** mvk has quit IRC | 22:42 | |
| *** dmorita has quit IRC | 23:14 | |
| *** dmorita has joined #openstack-swift | 23:15 | |
| *** chsc has quit IRC | 23:28 | |
| *** gyee has quit IRC | 23:46 | |
| *** nadeem has joined #openstack-swift | 23:47 | |
| *** nadeem has quit IRC | 23:47 | |
| *** Suyash has quit IRC | 23:53 | |
| *** nadeem has joined #openstack-swift | 23:53 | |
| *** lyrrad has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!