| *** nikivi has quit IRC | 00:01 | |
| *** dfflanders has joined #openstack-swift | 00:02 | |
| *** ediardo has quit IRC | 00:23 | |
| *** ediardo has joined #openstack-swift | 00:23 | |
| *** fungi has quit IRC | 00:26 | |
| *** gyee has quit IRC | 00:29 | |
| *** fungi has joined #openstack-swift | 00:30 | |
| *** chsc has quit IRC | 00:45 | |
| *** sudorandom has quit IRC | 00:47 | |
| *** mvk has quit IRC | 00:50 | |
| *** sudorandom has joined #openstack-swift | 00:53 | |
| *** m_kazuhiro has joined #openstack-swift | 00:55 | |
| *** jamielennox is now known as jamielennox|away | 00:56 | |
| *** jamielennox|away is now known as jamielennox | 01:02 | |
| kota_ | good morning | 01:06 |
|---|---|---|
| *** dfflanders has quit IRC | 01:07 | |
| kota_ | timburke: are you working for fixing pyeclib.get_metadata memory fix? ;-) | 01:08 |
| kota_ | s/memory/memory leak/ | 01:08 |
| openstackgerrit | Clay Gerrard proposed openstack/swift: Fix signal handling for daemons with InternalClient https://review.openstack.org/259347 | 01:13 |
| openstackgerrit | Jamie Lennox proposed openstack/python-swiftclient: Use requests_mock and oslotest https://review.openstack.org/360298 | 01:15 |
| mattoliverau | kota_: morning | 01:16 |
| kota_ | mattolivearu: good morning | 01:20 |
| *** trananhkma has quit IRC | 01:25 | |
| *** kei_yama has quit IRC | 01:25 | |
| *** kei_yama has joined #openstack-swift | 01:26 | |
| zhengyin | kota_, mattoliverau: morning | 01:28 |
| *** trananhkma has joined #openstack-swift | 01:30 | |
| charz | kota_ mattoliverau zhengyin morning | 01:32 |
| zhengyin | charz:morning | 01:33 |
| *** dfflanders has joined #openstack-swift | 01:37 | |
| *** dmorita has joined #openstack-swift | 01:44 | |
| *** zacksh has quit IRC | 01:50 | |
| *** zacksh has joined #openstack-swift | 01:52 | |
| *** Guest50630 has quit IRC | 02:03 | |
| *** zigo has joined #openstack-swift | 02:05 | |
| *** zigo is now known as Guest59427 | 02:06 | |
| *** dfflanders has quit IRC | 02:09 | |
| mattoliverau | charz, zhengyin o/ | 02:12 |
| mattoliverau | kota_: you might be proud of me, last night I made curry katsu don for dinner :) | 02:12 |
| *** tqtran has quit IRC | 02:15 | |
| kota_ | mattoliverau: curry katsu don!? It's my first time to hear such a food :P | 02:23 |
| kota_ | zhengyin, charz: morning | 02:23 |
| mattoliverau | kota_: lol, well you should try it some time.. I hear they may make it somewhere in Japan :P | 02:24 |
| kota_ | mattoliverau: interesting, sounds good anyway. ah, perhaps it's katsu carry in Japanse word??? | 02:27 |
| kota_ | mattolivearu: https://goo.gl/TyPjTx | 02:28 |
| mattoliverau | kota_: yeah, that :) | 02:30 |
| kota_ | mattolivearu: i like it ;-) | 02:33 |
| openstackgerrit | Kota Tsuyuzaki proposed openstack/liberasurecode: Fix reconstruct to return an error when memory allocation failed https://review.openstack.org/385240 | 02:39 |
| *** dmorita has quit IRC | 02:41 | |
| *** dmorita has joined #openstack-swift | 02:41 | |
| kota_ | acoes: i'm not sure if the patch 385240 addresses admin6's issue but it seems happen to make broken fragment via reconstruct function in liberasurecode | 02:42 |
| patchbot | https://review.openstack.org/#/c/385240/ - liberasurecode - Fix reconstruct to return an error when memory all... | 02:42 |
| *** dmorita has quit IRC | 02:42 | |
| kota_ | To figure out that it would happen, I tried via https://gist.github.com/bloodeagle40234/6af7f2d3e67e9cddabbea7e84c453c90 | 02:47 |
| kota_ | acoles: the gist is a diff for current liberasurecode master to break the reconstruct function as similer with the case when memory allocation failed. | 02:48 |
| *** amit213 has quit IRC | 02:48 | |
| *** amit213 has joined #openstack-swift | 02:49 | |
| kota_ | acoles: after compiling that code, and I tried to call https://gist.github.com/bloodeagle40234/bb18ef695c65785910bf4a846c93e0e9 | 02:49 |
| kota_ | that one is for trying to reconstruct | 02:49 |
| kota_ | via pyeclib | 02:49 |
| kota_ | acoles: amazingly, the reconstruct call is succesed (i.e. no error/exception occurred at the time) but it returns broken fragments with same length of expected fragments. | 02:51 |
| kota_ | acoles: if it occurred, swift cannot detect the broken frag anymore. | 02:51 |
| kota_ | acoles: I'll try to ask admin6 he can find some syslog messages to suggest such a memory allocation error from liberasurecode | 02:52 |
| *** dmorita has joined #openstack-swift | 03:00 | |
| *** dmorita has quit IRC | 03:04 | |
| *** dmorita has joined #openstack-swift | 03:09 | |
| jrichli | mattoliverau: yum! did it turn out well? | 03:11 |
| mattoliverau | yeah, it did actually, I think it'll be a keeper :) | 03:12 |
| *** dmorita has quit IRC | 03:14 | |
| *** pdardeau has quit IRC | 03:17 | |
| *** jamielennox is now known as jamielennox|away | 03:20 | |
| *** dmorita has joined #openstack-swift | 03:28 | |
| *** dmorita has quit IRC | 03:32 | |
| *** m_kazuhiro has quit IRC | 03:36 | |
| *** jamielennox|away is now known as jamielennox | 03:38 | |
| *** m_kazuhiro_ has joined #openstack-swift | 03:38 | |
| *** dmorita has joined #openstack-swift | 03:40 | |
| *** dmorita has quit IRC | 03:41 | |
| *** dmorita has joined #openstack-swift | 03:42 | |
| *** m_kazuhiro_ has quit IRC | 03:43 | |
| mahatic_ | good morning | 03:44 |
| * mahatic_ is back after a local holiday in India | 03:44 | |
| *** m_kazuhiro_ has joined #openstack-swift | 03:45 | |
| *** psachin has joined #openstack-swift | 03:57 | |
| *** Jeffrey4l_ has quit IRC | 04:05 | |
| kota_ | mahatic: good morning and welcome back | 04:06 |
| *** chsc has joined #openstack-swift | 04:07 | |
| kota_ | hmm... my though might be wrong. if something like memory allocation failure happen in the liberasurecode, the reconstracted fragment should consist of 0 | 04:07 |
| *** chsc has quit IRC | 04:09 | |
| *** links has joined #openstack-swift | 04:10 | |
| *** jistr has quit IRC | 04:15 | |
| *** jistr has joined #openstack-swift | 04:16 | |
| *** klrmn has quit IRC | 04:23 | |
| *** wasmum has quit IRC | 04:23 | |
| mattoliverau | mahatic_: o/ | 04:25 |
| *** Jeffrey4l has joined #openstack-swift | 04:26 | |
| *** wasmum has joined #openstack-swift | 04:35 | |
| mahatic_ | kota_: mattoliverau : hello | 04:40 |
| *** dmorita has quit IRC | 05:03 | |
| kota_ | acole: i found *really* interesting thing when trying rebuilding the corrupted fragment, please ping me if you get online. | 05:07 |
| *** SkyRocknRoll has joined #openstack-swift | 05:07 | |
| kota_ | acoles:^^ | 05:07 |
| *** SkyRocknRoll has quit IRC | 05:07 | |
| *** SkyRocknRoll has joined #openstack-swift | 05:07 | |
| *** wasmum has quit IRC | 05:09 | |
| *** Jeffrey4l has quit IRC | 05:09 | |
| *** Jeffrey4l has joined #openstack-swift | 05:11 | |
| *** Jeffrey4l has quit IRC | 05:12 | |
| *** Jeffrey4l has joined #openstack-swift | 05:13 | |
| *** Jeffrey4l has quit IRC | 05:13 | |
| *** Jeffrey4l_ has joined #openstack-swift | 05:13 | |
| *** tanee_away is now known as tanee | 05:19 | |
| *** wasmum has joined #openstack-swift | 05:19 | |
| openstackgerrit | zheng yin proposed openstack/swift: Simplify container_updater test code https://review.openstack.org/355858 | 05:21 |
| *** ChubYann has quit IRC | 05:25 | |
| *** dmorita has joined #openstack-swift | 05:34 | |
| *** gabor_antal_ has joined #openstack-swift | 05:35 | |
| *** gabor_antal has quit IRC | 05:37 | |
| *** Jeffrey4l_ has quit IRC | 05:41 | |
| openstackgerrit | Jamie Lennox proposed openstack/python-swiftclient: Use requests_mock and testtools https://review.openstack.org/360298 | 05:43 |
| openstackgerrit | Tuan Luong-Anh proposed openstack/swift: Remove xrange for run both Python 2 and Python 3 https://review.openstack.org/385285 | 05:46 |
| *** Jeffrey4l_ has joined #openstack-swift | 05:52 | |
| *** links has quit IRC | 05:54 | |
| *** dmorita has quit IRC | 05:55 | |
| *** Jeffrey4l_ has quit IRC | 05:57 | |
| *** Jeffrey4l_ has joined #openstack-swift | 05:58 | |
| *** dmorita has joined #openstack-swift | 05:58 | |
| *** Jeffrey4l_ has quit IRC | 06:06 | |
| *** links has joined #openstack-swift | 06:10 | |
| *** dmorita has quit IRC | 06:10 | |
| *** dmorita has joined #openstack-swift | 06:11 | |
| *** Jeffrey4l_ has joined #openstack-swift | 06:19 | |
| *** dmorita has quit IRC | 06:21 | |
| *** Jeffrey4l_ has quit IRC | 06:23 | |
| *** Jeffrey4l_ has joined #openstack-swift | 06:23 | |
| * mattoliverau is calling it a day, have great night swift land o/ | 06:29 | |
| *** m_kazuhiro_ has quit IRC | 06:35 | |
| *** rcernin has joined #openstack-swift | 06:36 | |
| *** dmorita has joined #openstack-swift | 06:39 | |
| *** takashi has joined #openstack-swift | 06:40 | |
| *** qwertyco has joined #openstack-swift | 06:55 | |
| *** tesseract has joined #openstack-swift | 07:06 | |
| *** tesseract is now known as Guest24890 | 07:06 | |
| *** ppai has joined #openstack-swift | 07:09 | |
| *** wasmum has quit IRC | 07:10 | |
| *** rledisez has joined #openstack-swift | 07:13 | |
| openstackgerrit | Christian Schwede proposed openstack/swift: Throttle update_auditor_status calls https://review.openstack.org/383707 | 07:14 |
| *** wasmum has joined #openstack-swift | 07:15 | |
| *** dmorita has quit IRC | 07:26 | |
| *** acoles_ is now known as acoles | 07:28 | |
| *** acoles is now known as acoles_ | 07:30 | |
| *** acoles_ is now known as acoles | 07:31 | |
| acoles | kota_: ping | 07:31 |
| kota_ | hi acoles | 07:33 |
| *** Guest59427 is now known as zigo | 07:33 | |
| *** dmorita has joined #openstack-swift | 07:33 | |
| acoles | kota_: looks like you made some progress | 07:34 |
| kota_ | acoles: yeah, i updated the status on https://bugs.launchpad.net/swift/+bug/1631144 | 07:35 |
| openstack | Launchpad bug 1631144 in OpenStack Object Storage (swift) "Swift Erasure Coding : Error when decoding fragment" [High,Confirmed] | 07:35 |
| kota_ | acoles: summrizing that, i think reconstructor or ssync protocol seems broken. | 07:35 |
| kota_ | acoles: i didn't get how it was broken yet | 07:36 |
| *** dmorita has quit IRC | 07:37 | |
| acoles | kota_: ok I'm reading the bug report updates | 07:37 |
| kota_ | acoles: thanks | 07:37 |
| *** geaaru has joined #openstack-swift | 07:40 | |
| kota_ | acoles: i think, you can see the actual corrupted fragment archive body sent from admin6 | 07:41 |
| kota_ | too | 07:41 |
| *** dmorita has joined #openstack-swift | 07:41 | |
| acoles | kota_: oh, wow, so the bad frag has some http headers in it :/ ? | 07:42 |
| kota_ | acoles: yeah, unbelievable but that's fact | 07:43 |
| acoles | and of course the reconstructor does not send etag so the receiving object server does not know, and writes the etag of whatever it received. | 07:43 |
| kota_ | acoles: i wonder if you could double check that. | 07:43 |
| acoles | kota_: double :/ and :( | 07:43 |
| kota_ | that's assumption what i am going to tell you since now, though | 07:44 |
| *** rcernin has quit IRC | 07:45 | |
| kota_ | if a http client reuse the tcp connection, it could happen the receiver regard the header as the body, maybe? | 07:45 |
| kota_ | in my experience, i had gotten the case http sever regards the http body as another request header. | 07:46 |
| kota_ | not sure right now. | 07:46 |
| kota_ | i think sort of that may occurred in the ssync protocol. | 07:47 |
| acoles | kota_: yes I see the headers in the bad fragment | 07:47 |
| kota_ | acoles: ok, that's fact we're now. | 07:47 |
| *** rcernin has joined #openstack-swift | 07:49 | |
| kota_ | and i'm thinking of how we could test/reproduce that | 07:49 |
| kota_ | acoles: that's current status to me | 07:50 |
| acoles | kota_: the headers do not appear to be for that fragment - the timestamp is different | 07:50 |
| kota_ | acoles: what do you mean "that fragment"? | 07:51 |
| kota_ | acoles: corrupted one? normal one? | 07:51 |
| *** bkeller` has quit IRC | 07:52 | |
| *** bkeller^ has joined #openstack-swift | 07:52 | |
| acoles | kota_: the bad fragment. well the bad object, the timestamp in the headers is newer than in the fragment .data filename | 07:53 |
| *** bkeller^ is now known as bkeller` | 07:53 | |
| kota_ | acoles: ah, good catch | 07:54 |
| *** dmorita_ has joined #openstack-swift | 07:54 | |
| kota_ | acoles: i didn't notice that, exactly it looks... older? | 07:54 |
| *** dmorita has quit IRC | 07:55 | |
| kota_ | header timestamp < .data timestamp? | 07:55 |
| kota_ | let me check both bad frags | 07:55 |
| *** cshastri has joined #openstack-swift | 07:55 | |
| acoles | newer, header timestamp > filename timestamp | 07:56 |
| admin6 | acoles, kota_: Hi | 08:01 |
| kota_ | admin6: hi | 08:01 |
| acoles | kota_: one hypothesis - if an error occurs in ssync protocol that causes a frag body to not be sent, then the receiver may interpret the next set of *headers and body* as the missing body | 08:01 |
| acoles | admin6: hi | 08:01 |
| acoles | kota_: which would mean that the content of the corrupt frag file (a) has headers! and (b) is frag data for a different object! | 08:02 |
| kota_ | acoles: ah, that might be true | 08:02 |
| acoles | kota_: I am not sure how we could verify that (b) is true | 08:03 |
| kota_ | acoles: to tell more, i noticed one more header in the mid of the corrupted file. | 08:03 |
| *** dmorita_ has quit IRC | 08:03 | |
| admin6 | II’m currently checking deeper but so far (on 10 different files tested), the corrupted fragment always is a reconstructed fragment. | 08:03 |
| *** dmorita has joined #openstack-swift | 08:04 | |
| kota_ | acoles: could we confirm that to check the path? | 08:04 |
| acoles | admin6: interesting. thanks. | 08:07 |
| kota_ | admin6: thanks for your good info. i think we are going to be close the reason | 08:10 |
| *** qwertyco has quit IRC | 08:12 | |
| cnf | morning | 08:36 |
| cnf | kota_ i got swift3 working btw | 08:36 |
| cnf | eventually :P | 08:37 |
| kota_ | cnf: morning and congrats! | 08:37 |
| cnf | kota_ if you care about the back story, with keystone v3 port 5000 and 35357 are the same | 08:38 |
| cnf | so the rest of my setup was all on port 5000, reverse proxied behind an ssl terminator | 08:39 |
| cnf | of course, swift3 was ahrdcoded to v2, which doesn't work on port 5000 | 08:39 |
| cnf | timburke submitted a patch to make that configurable | 08:40 |
| cnf | thanks to him for that, and thanks to you for swift3 :P | 08:40 |
| *** wasmum has quit IRC | 08:54 | |
| *** wasmum has joined #openstack-swift | 08:58 | |
| *** mvk has joined #openstack-swift | 09:17 | |
| *** dmorita has quit IRC | 09:34 | |
| *** mvk has quit IRC | 09:50 | |
| *** silor has joined #openstack-swift | 09:53 | |
| *** silor has quit IRC | 09:56 | |
| *** silor has joined #openstack-swift | 09:57 | |
| *** silor1 has joined #openstack-swift | 10:12 | |
| *** silor has quit IRC | 10:13 | |
| *** silor1 is now known as silor | 10:13 | |
| openstackgerrit | Charles Hsu proposed openstack/swift: Make Content-Disposition support `inline; filename=...` format. https://review.openstack.org/385382 | 10:24 |
| *** mvk has joined #openstack-swift | 10:25 | |
| *** takashi has quit IRC | 10:27 | |
| *** ppai_ has joined #openstack-swift | 10:27 | |
| *** dosaboy_ is now known as dosaboy | 10:28 | |
| *** ppai has quit IRC | 10:29 | |
| charz | kota_: acoles Just found a minor issue in tempurl ^^ | 10:29 |
| charz | if you have time, please help to review it. :) | 10:29 |
| acoles | charz: hi. I'm going to be busy on this https://bugs.launchpad.net/swift/+bug/1631144 today, sorry | 10:32 |
| openstack | Launchpad bug 1631144 in OpenStack Object Storage (swift) "Swift Erasure Coding : Error when decoding fragment" [Critical,Confirmed] | 10:32 |
| charz | acoles: kk, no rush. | 10:32 |
| *** SkyRocknRoll has quit IRC | 10:36 | |
| openstackgerrit | Charles Hsu proposed openstack/swift: Make Content-Disposition support `inline; filename=...` format. https://review.openstack.org/385382 | 10:36 |
| *** kei_yama has quit IRC | 10:39 | |
| *** dmorita has joined #openstack-swift | 11:01 | |
| *** dmorita has quit IRC | 11:01 | |
| *** dmorita has joined #openstack-swift | 11:02 | |
| *** Praveen__ has joined #openstack-swift | 11:09 | |
| Praveen__ | Hi clayg, Can you please give me pointers/suggestions for the bug 1632171 !! | 11:15 |
| openstack | bug 1632171 in OpenStack Object Storage (swift) "stub account response output missing policy information" [Low,New] https://launchpad.net/bugs/1632171 - Assigned to Praveen N (praveenn) | 11:15 |
| acoles | kota_: I managed to get a test to reproduce the issue on bug 1631144, need some more time to clean it up | 11:18 |
| openstack | bug 1631144 in OpenStack Object Storage (swift) "Swift Erasure Coding : Error when decoding fragment" [Critical,Confirmed] https://launchpad.net/bugs/1631144 | 11:18 |
| openstackgerrit | Kazuhiro MIYAHARA proposed openstack/swift: Remove 'X-Static-Large-Object' from .meta files https://review.openstack.org/385412 | 11:45 |
| *** ppai_ has quit IRC | 11:50 | |
| *** nottrobin has quit IRC | 11:58 | |
| *** nottrobin has joined #openstack-swift | 11:59 | |
| *** dmorita has quit IRC | 12:03 | |
| *** dmorita has joined #openstack-swift | 12:19 | |
| *** dmorita has quit IRC | 12:23 | |
| *** dmorita has joined #openstack-swift | 12:27 | |
| *** Praveen__ has quit IRC | 12:29 | |
| *** links has quit IRC | 12:42 | |
| *** cshastri has quit IRC | 12:44 | |
| *** klamath has joined #openstack-swift | 13:05 | |
| *** klamath has quit IRC | 13:05 | |
| *** klamath has joined #openstack-swift | 13:06 | |
| *** cdelatte has joined #openstack-swift | 13:13 | |
| *** rcernin has quit IRC | 13:13 | |
| *** rcernin has joined #openstack-swift | 13:13 | |
| *** tongli has joined #openstack-swift | 13:24 | |
| *** dmorita has quit IRC | 13:32 | |
| kota_ | back to online | 13:33 |
| kota_ | thanks acoles for updating https://launchpad.net/bugs/1631144 | 13:34 |
| openstack | Launchpad bug 1631144 in OpenStack Object Storage (swift) "Swift Erasure Coding : Error when decoding fragment" [Critical,Confirmed] | 13:34 |
| kota_ | cnf: yeah, it looks patch 384659 which timburke is working? | 13:36 |
| patchbot | https://review.openstack.org/#/c/384659/ - swift3 - Make s3token work in a Keystone-V3-only world | 13:36 |
| kota_ | it seems good stuff | 13:37 |
| openstackgerrit | Merged openstack/swift: Update test-requirements.txt https://review.openstack.org/365271 | 13:38 |
| openstackgerrit | Merged openstack/swift: [api-ref] Remove temporary block in conf.py https://review.openstack.org/366538 | 13:39 |
| kota_ | charz: thanks for pinging me, will look at in this week. probably we got a priority for the bug acoles suggested in the irc log in my opinion. | 13:39 |
| acoles | kota_: np. I may even have a fix, at least a fix for the test I wrote. It's kinda ugly though. | 13:40 |
| kota_ | acoles: yeah, if i undestand correctly, we have a bunch of broken parts to fix for the bug, broken ssync protocol, broken reconstruction, etc... | 13:42 |
| kota_ | but some of them can be pend as improvement, the most priority thing is to prevent the broken frag anyway. | 13:43 |
| acoles | kota_: exactly, I'm just looking for a pragmatic quick fix first. | 13:44 |
| kota_ | acoles: great | 13:44 |
| acoles | kota_: get some sleep !) | 13:45 |
| kota_ | acoles: thanks! though I'm back to online, I'm getting feeling i might be tired a bit ;-) | 13:46 |
| * kota_ is going to be offline again to take a rest for tommorrow meeting. | 13:46 | |
| acoles | kota_: see you in 7 hours or so :) | 13:47 |
| *** dmorita has joined #openstack-swift | 13:48 | |
| *** dmorita has quit IRC | 13:52 | |
| *** pcaruana has joined #openstack-swift | 13:54 | |
| *** tqtran has joined #openstack-swift | 14:03 | |
| *** tqtran has quit IRC | 14:07 | |
| *** kmARC has quit IRC | 14:09 | |
| cnf | kota_ that's the one | 14:15 |
| *** kmARC has joined #openstack-swift | 14:16 | |
| *** dmorita has joined #openstack-swift | 14:20 | |
| *** SkyRocknRoll has joined #openstack-swift | 14:22 | |
| *** SkyRocknRoll has quit IRC | 14:22 | |
| *** SkyRocknRoll has joined #openstack-swift | 14:22 | |
| *** dmorita has quit IRC | 14:24 | |
| *** pcaruana has quit IRC | 14:27 | |
| *** _JZ_ has joined #openstack-swift | 14:30 | |
| *** sgundur has joined #openstack-swift | 14:34 | |
| *** sgundur has quit IRC | 14:40 | |
| *** _JZ_ has quit IRC | 14:47 | |
| *** dmorita has joined #openstack-swift | 14:51 | |
| *** vint_bra has joined #openstack-swift | 14:53 | |
| *** dmorita has quit IRC | 14:56 | |
| *** psachin has quit IRC | 15:05 | |
| *** urth has quit IRC | 15:08 | |
| *** vint_bra has quit IRC | 15:08 | |
| *** urth has joined #openstack-swift | 15:10 | |
| *** vint_bra has joined #openstack-swift | 15:10 | |
| *** klrmn has joined #openstack-swift | 15:10 | |
| *** SkyRocknRoll has quit IRC | 15:12 | |
| *** rcernin has quit IRC | 15:19 | |
| *** joeljwright has joined #openstack-swift | 15:21 | |
| *** ChanServ sets mode: +v joeljwright | 15:21 | |
| *** dmorita has joined #openstack-swift | 15:23 | |
| *** dmorita has quit IRC | 15:27 | |
| *** vint_bra has quit IRC | 15:28 | |
| *** vern has joined #openstack-swift | 15:38 | |
| *** links has joined #openstack-swift | 15:39 | |
| *** pdardeau has joined #openstack-swift | 15:42 | |
| *** sgundur has joined #openstack-swift | 15:43 | |
| *** david-lyle has joined #openstack-swift | 15:46 | |
| *** david-lyle has quit IRC | 15:52 | |
| *** dmorita has joined #openstack-swift | 15:55 | |
| *** dmorita has quit IRC | 15:59 | |
| *** jordanP has joined #openstack-swift | 16:02 | |
| *** david-lyle has joined #openstack-swift | 16:06 | |
| *** klrmn has quit IRC | 16:19 | |
| *** dmorita has joined #openstack-swift | 16:26 | |
| *** dmorita has quit IRC | 16:31 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift: Updated from global requirements https://review.openstack.org/88736 | 16:34 |
| notmyname | good morning | 16:39 |
| *** links has quit IRC | 16:45 | |
| *** rledisez has quit IRC | 16:46 | |
| *** nikivi has joined #openstack-swift | 16:47 | |
| *** rcernin has joined #openstack-swift | 16:57 | |
| notmyname | wow. interesting ML thread on the PTG. seems to be a bit of concern that this will make communication with users harder and limit available travel | 16:57 |
| *** dmorita has joined #openstack-swift | 16:58 | |
| *** Guest24890 has quit IRC | 17:00 | |
| *** jeblair_ is now known as jeblair | 17:03 | |
| *** dmorita has quit IRC | 17:05 | |
| *** zul has quit IRC | 17:11 | |
| *** jordanP has quit IRC | 17:12 | |
| *** zul has joined #openstack-swift | 17:12 | |
| *** cshastri has joined #openstack-swift | 17:15 | |
| *** klrmn has joined #openstack-swift | 17:16 | |
| *** ChubYann has joined #openstack-swift | 17:22 | |
| clayg | yeah, but I mean realisticly some orgs were already sturggling to get ops to international summit events - i didn't expect PTG's to make it any better | 17:22 |
| *** sgundur has quit IRC | 17:23 | |
| clayg | I think for teams that didn't have a culture of self organizing mid-cycles the reality of traveling 4 times a year is setting in | 17:23 |
| notmyname | FYI http://lists.openstack.org/pipermail/openstack-dev/2016-October/105600.html | 17:31 |
| *** sgundur has joined #openstack-swift | 17:34 | |
| *** rcernin has quit IRC | 17:56 | |
| *** thebloggu has joined #openstack-swift | 18:00 | |
| *** tqtran has joined #openstack-swift | 18:04 | |
| *** tqtran has quit IRC | 18:08 | |
| *** lcurtis has joined #openstack-swift | 18:11 | |
| *** seventy has joined #openstack-swift | 18:14 | |
| *** geaaru has quit IRC | 18:23 | |
| *** mvk has quit IRC | 18:28 | |
| tdasilva | notmyname: cool email! | 18:43 |
| *** sgundur has quit IRC | 18:46 | |
| notmyname | thanks | 18:46 |
| ahale | i liked yours in the tripleo thread tdasilva | 18:46 |
| *** tqtran has joined #openstack-swift | 18:47 | |
| tdasilva | ahale: yeah, i was trying to understand what they were trying to achieve, it's too bad they are going away from swift, it would be nice to understand better what the issues are | 18:48 |
| tdasilva | ahale: i was just learning today about the differences between the gnocchi ceph driver and the swift driver | 18:49 |
| ahale | i havent dared look that closely at it | 18:50 |
| tdasilva | ahale: so I guess there are certain work loads where it is not very suitable for swift, at least not in the way it is written | 18:50 |
| tdasilva | but i'd have to play more with it | 18:50 |
| *** thebloggu has quit IRC | 18:57 | |
| *** sgundur has joined #openstack-swift | 19:02 | |
| *** mvk has joined #openstack-swift | 19:28 | |
| clayg | when is the meeting? | 19:35 |
| acoles | clayg in 1hour 17mins | 19:43 |
| openstackgerrit | Clay Gerrard proposed openstack/swift: Fix signal handling for daemons with InternalClient https://review.openstack.org/259347 | 19:44 |
| acoles | tdasilva: thanks for your comments on patch 355958, I have been busy on the EC corrupt fragments bug today but will respond | 19:47 |
| patchbot | https://review.openstack.org/#/c/355958/ - swift - EC - eliminate .durable files | 19:47 |
| *** cshastri has quit IRC | 19:49 | |
| tdasilva | acoles: so i was about to update that comment, i have verified that the issue I found is fixed by patch 376630 | 19:50 |
| patchbot | https://review.openstack.org/#/c/376630/ - swift - EC: reconstruct using non-durable fragments | 19:50 |
| acoles | tdasilva: cool. thanks! | 19:50 |
| acoles | I just started seein "liberasurecode_backend_open: dynamic linking error libisal.so: cannot open shared object file: No such file or directory" when running probe tests, what magic do I need to do? | 19:52 |
| acoles | since I upgraded to pyeclib 1.3.1 | 19:52 |
| tdasilva | upgrade liberasurecode | 19:52 |
| acoles | k | 19:53 |
| tdasilva | i'm assuming you are running 1.1.1 | 19:53 |
| *** vern has quit IRC | 19:53 | |
| *** _JZ_ has joined #openstack-swift | 19:54 | |
| acoles | I was on pyeclib 1.2.0 | 19:54 |
| tdasilva | mmmmm :/ | 19:54 |
| tdasilva | oh yeah...sorry...pyeclib | 19:55 |
| tdasilva | so yeah, you want to upgrade liberasurecode from 1.1.1 to 1.2.0 | 19:55 |
| acoles | sorry yes I have liberasurecode 1.1.0 so I need to upgrade. so pyeclib does not update liberasurecode as a dependency? | 19:57 |
| * acoles vaguely remembers that changing at soem point | 19:57 | |
| tdasilva | acoles: i don't think so | 20:02 |
| *** vern has joined #openstack-swift | 20:02 | |
| tdasilva | plus i don't think there are distro builds available for 1.2.0 yet ???? maybe ??? | 20:03 |
| tdasilva | i think there's isn't for rh* based systems and remember there not being avaiable for ubuntu either | 20:03 |
| *** sgundur has quit IRC | 20:06 | |
| *** sgundur has joined #openstack-swift | 20:11 | |
| openstackgerrit | Alistair Coles proposed openstack/swift: Prevent ssync writing bad fragment data to diskfile https://review.openstack.org/385609 | 20:12 |
| acoles | kota_: ^^ | 20:13 |
| acoles | tdasilva: k, thanks | 20:13 |
| *** silor has quit IRC | 20:14 | |
| *** tqtran has quit IRC | 20:19 | |
| *** acoles is now known as acoles_ | 20:20 | |
| *** cdelatte has quit IRC | 20:27 | |
| openstackgerrit | Merged openstack/swift: Simplify container_updater test code https://review.openstack.org/355858 | 20:30 |
| *** mtreinish has quit IRC | 20:31 | |
| *** mtreinish has joined #openstack-swift | 20:32 | |
| *** sgundur has quit IRC | 20:39 | |
| clayg | gd lp bug #1632839 | 20:42 |
| openstack | Launchpad bug 1632839 in OpenStack Object Storage (swift) "Reconstructor does not send an ETag with a reconstructed fragment" [Undecided,New] https://launchpad.net/bugs/1632839 | 20:42 |
| clayg | is that the root cause of lp bug #1631144 | 20:42 |
| openstack | Launchpad bug 1631144 in OpenStack Object Storage (swift) "Swift Erasure Coding : Error when decoding fragment" [Critical,Confirmed] https://launchpad.net/bugs/1631144 | 20:42 |
| clayg | cause I think I sorta remember someone pointing out that rebuilt fragments don't do etag validation (we had to pop the etag from the metadata of ondisk frag to avoid ssync throwing a conflict) | 20:43 |
| *** dmorita has joined #openstack-swift | 20:43 | |
| clayg | ... but I probably said something stupid like "that should rarely be an issue" | 20:43 |
| mattoliverau | morning | 20:49 |
| *** seventy has quit IRC | 20:49 | |
| *** m_kazuhiro has joined #openstack-swift | 20:50 | |
| clayg | the woliver! | 20:52 |
| mattoliverau | yeah its me! | 20:53 |
| clayg | :D | 20:53 |
| clayg | makes me so happy | 20:53 |
| *** kei_yama has joined #openstack-swift | 20:56 | |
| kota_ | good morning | 20:57 |
| notmyname | clayg: you know, mattoliverau keeps saying the "au" is for australia, but I think it's really for "gold" | 20:58 |
| mattoliverau | kota_: morning | 20:58 |
| *** acoles_ is now known as acoles | 20:58 | |
| mattoliverau | notmyname: well of course it is, I'm the golden woliver :P | 20:58 |
| clayg | ahh... the golden woliver is the rarest of the wiolivers | 20:59 |
| mattoliverau | yeah we are only found on the Australian NSW coast near beaches ;P | 20:59 |
| kota_ | mattoliverau: o/ | 20:59 |
| notmyname | meeting time in #openstack-meeting | 21:00 |
| kota_ | acoles thanks for the patch, will look at in daytime today | 21:00 |
| acoles | clayg: bug 1632839 isn't the cause of bug 1631144 but it contributes to 1631144 not being detected | 21:00 |
| openstack | bug 1632839 in OpenStack Object Storage (swift) "Reconstructor does not send an ETag with a reconstructed fragment" [Undecided,New] https://launchpad.net/bugs/1632839 | 21:00 |
| openstack | bug 1631144 in OpenStack Object Storage (swift) "Swift Erasure Coding : Error when decoding fragment" [Critical,Confirmed] https://launchpad.net/bugs/1631144 | 21:00 |
| clayg | acoles: i thought i read somewhere that the fragment data was corrupt/bad despite matching the etag that was written on it? so... some sort of error in transmission? | 21:01 |
| *** sgundur has joined #openstack-swift | 21:02 | |
| acoles | clayg: yes, bad data is sent with no ETag (cos reconstructor can't send an ETag header for a sync frag job) so receiver just computes md5 of garbage and adds it to metadata | 21:03 |
| acoles | there is no validation | 21:03 |
| acoles | in this case we don't suspect on the wire corruption, just ssync protocol doing the wrong thing | 21:03 |
| clayg | oh jesus - yeah i'm reading the commit message on patch 385609 | 21:03 |
| patchbot | https://review.openstack.org/#/c/385609/ - swift - Prevent ssync writing bad fragment data to diskfile | 21:03 |
| acoles | clayg: it's...a...mess | 21:04 |
| clayg | good | 21:04 |
| acoles | clayg: meeting time, I was late but you're later ;) | 21:04 |
| clayg | glad you're all over it then | 21:04 |
| *** mtreinish has quit IRC | 21:05 | |
| acoles | clayg: worse thing is the reporter has seen this for real, more than once :/ | 21:06 |
| *** joeljwright has quit IRC | 21:06 | |
| clayg | yup but there's some hope repairing the bad frags will allow them to rebuild successfully | 21:06 |
| clayg | go go ec durability! | 21:07 |
| acoles | yes, just they will have to be manually removed cos the auditor thinks they are fine and dandy | 21:08 |
| *** sgundur1 has joined #openstack-swift | 21:10 | |
| *** joeljwright has joined #openstack-swift | 21:11 | |
| *** ChanServ sets mode: +v joeljwright | 21:11 | |
| *** mtreinish has joined #openstack-swift | 21:12 | |
| *** cebruns_ has quit IRC | 21:15 | |
| *** zigo_ has joined #openstack-swift | 21:17 | |
| *** kota_ has quit IRC | 21:17 | |
| *** kota_ has joined #openstack-swift | 21:18 | |
| *** ChanServ sets mode: +v kota_ | 21:18 | |
| kota_ | did irc halt? | 21:18 |
| acoles | kota_: I didn't work out an nice way to test the read timeout case but the other cases cover same outcome. | 21:19 |
| *** kmARC has quit IRC | 21:19 | |
| *** wasmum has quit IRC | 21:19 | |
| *** zigo has quit IRC | 21:19 | |
| *** cebruns_ has joined #openstack-swift | 21:20 | |
| clayg | can people try and raise a hand in #openstack-meeting? | 21:20 |
| kota_ | acoles: ok, maybe i can try to cover that case | 21:20 |
| *** lcurtis has quit IRC | 21:20 | |
| *** lcurtis has joined #openstack-swift | 21:21 | |
| acoles | kota_: one thought I had was to mock the Timeout context manager to raise an exception. not great. but I didn't want to go putting timeouts into unit tests. | 21:21 |
| kota_ | acoles: ah, you mean it's overkill for *unit*? | 21:22 |
| *** philipw has quit IRC | 21:23 | |
| *** ujjain has quit IRC | 21:23 | |
| *** StevenK has quit IRC | 21:23 | |
| *** CrackerJackMack has quit IRC | 21:23 | |
| *** kmARC has joined #openstack-swift | 21:23 | |
| *** ujjain has joined #openstack-swift | 21:23 | |
| *** ujjain has joined #openstack-swift | 21:23 | |
| *** klrmn has quit IRC | 21:24 | |
| acoles | kota_: I mean its tricky because if we reduce the timeout times we risk the tests being flakey, and if we leave them high then the test is slow | 21:24 |
| acoles | kota_: i.e. if I mock a response that just sleeps on a read, then how long should it sleep? | 21:25 |
| kota_ | acoles: gotcha, if we do that, probably need to patch the timeout value or sort of mock the read method to raise Timeout exception? | 21:26 |
| *** philipw has joined #openstack-swift | 21:28 | |
| *** StevenK has joined #openstack-swift | 21:28 | |
| *** CrackerJackMack has joined #openstack-swift | 21:28 | |
| *** wasmum has joined #openstack-swift | 21:28 | |
| *** wasmum has quit IRC | 21:28 | |
| *** wasmum has joined #openstack-swift | 21:28 | |
| *** zigo_ is now known as zigo | 21:28 | |
| *** vern has quit IRC | 21:34 | |
| *** Jeffrey4l__ has joined #openstack-swift | 21:34 | |
| *** vern has joined #openstack-swift | 21:35 | |
| *** Jeffrey4l_ has quit IRC | 21:35 | |
| *** mtreinish has quit IRC | 21:36 | |
| *** mtreinish has joined #openstack-swift | 21:42 | |
| *** tongli has quit IRC | 21:45 | |
| jrichli | torgomatic: dont leave us hangin | 22:01 |
| clayg | torgomatic: wait what was the bunch line!? | 22:01 |
| tdasilva | lol | 22:01 |
| torgomatic | you get drunk | 22:01 |
| notmyname | I was really hoping to get that in the meeting notes :-) | 22:01 |
| torgomatic | because √beer * √beer is just beer | 22:02 |
| jrichli | :p | 22:02 |
| acoles | kota_: problem I saw is the timeout is on a wait for the threads in the pile, and I am not sure what happens if a mocked method raises a timeout exception in that thread? I always get confused by that. | 22:03 |
| clayg | notmyname: he was looking up the ascii codes for square root symbols :\ | 22:03 |
| *** m_kazuhiro has quit IRC | 22:03 | |
| notmyname | heh | 22:03 |
| cschwede | clayg: let’s chat on your Thursday morning? i have a few ideas to add tomorrow | 22:04 |
| kota_ | acoles: IIRC, we need to catch the exception in the thread and have to wrap or... | 22:04 |
| kota_ | acoles: hmm... probably I have to play with code to clarify the feasibility. | 22:04 |
| acoles | kota_: yeah but IDK how you get the Timeout to pop without making the thread sleep | 22:05 |
| *** mrda has joined #openstack-swift | 22:05 | |
| kota_ | acoles: k, will try (it might fail though) and let you know if better way found. | 22:06 |
| kota_ | acoles: thanks for calling it anyway. | 22:06 |
| clayg | acoles: is patch 385609 wip? | 22:07 |
| patchbot | https://review.openstack.org/#/c/385609/ - swift - Prevent ssync writing bad fragment data to diskfile | 22:07 |
| clayg | cschwede: for me tomorrow is thursday - ah yeah so that would be after your work day | 22:08 |
| acoles | clayg: wip == written-in-panic ? ;) | 22:08 |
| acoles | clayg: no but since you ask I am now wondering if you think it should be | 22:08 |
| clayg | rofl | 22:08 |
| *** joeljwright has quit IRC | 22:08 | |
| *** mtreinish has quit IRC | 22:08 | |
| cschwede | clayg: yep, in my evening. i’m available from ~ 11am-2pm your time - just let me know what works for you | 22:08 |
| clayg | RebuildingECDiskFileStream grew some initialized state that seems unused? | 22:09 |
| acoles | clayg: it could probably use some more tests, but I think what is there is probably sufficient to get a fix out. | 22:09 |
| clayg | cschwede: np, same time as last time except tomorrow wfm | 22:09 |
| acoles | clayg: :/ self.bytes_read ?? I guess that without even looking | 22:09 |
| acoles | clayg: it was a long day, I'll rev it tomorrow | 22:09 |
| cschwede | clayg: ok, does Friday work better for you? | 22:10 |
| clayg | acoles: maybe *I* should rev it while i'm reviewing and if I htink it's good enough I should +2 it!? | 22:10 |
| clayg | cschwede: no tomorrow - maybe on Friday again if we need it - it's getting close :\ | 22:10 |
| * kota_ is going breakfast | 22:11 | |
| clayg | acoles: well before you go could you at lest give me a hint why yout hink it's only a partial fix? | 22:11 |
| clayg | ... fi you're still here | 22:11 |
| cschwede | clayg: i think my brain sleeps already, sorry - when do we meet? | 22:11 |
| acoles | clayg: fine with me | 22:11 |
| clayg | :D | 22:11 |
| notmyname | FYI public transport pass in barca https://www.tmb.cat/en/barcelona/fares-metro-bus/card-transport-barcelona | 22:12 |
| clayg | cschwede: 6:00:00 pm UTC | Thursday, October 13, 2016 | 22:12 |
| cschwede | clayg: awesome! alright. tty tomorrow! | 22:13 |
| acoles | clayg: I went for partial because ...hmm, well I guess that's me being cautious again. Having stared at it all day there's other ways the reconstructor might sync garbage. e.g. there is no check that frags fetched are consistent with their etags. maybe pyeclib would blow up in that case. | 22:14 |
| acoles | clayg: IDK, on reflection I could live with closes-bug | 22:14 |
| clayg | well I think this guys bug based on kota's examination - all signs point the issue you fixing | 22:16 |
| jrichli | for tourist stuff, acoles had mentioned this place earlier: http://www.sagradafamilia.org/en/ | 22:16 |
| clayg | there may be other issues - but let's just open other bugs (failure to send etag is an obvious belts and braces thing that allowed this to happen) | 22:17 |
| acoles | clayg: OK. well since you'll be editing the commit message to add yourself as co-author youcan also s/Partial/Closes/ :) | 22:18 |
| clayg | nice :P | 22:18 |
| clayg | g'night | 22:18 |
| *** mtreinish has joined #openstack-swift | 22:19 | |
| clayg | and a context switching we will go; and a context ... wait what was I doing? | 22:19 |
| acoles | good night | 22:20 |
| clayg | joel<tab><tab> :'( | 22:20 |
| *** sgundur has quit IRC | 22:23 | |
| *** nikivi has quit IRC | 22:24 | |
| *** acoles is now known as acoles_ | 22:25 | |
| clayg | i'm getting liberasurecode_backend_open: dynamic linking error when restarting with swift-init now? | 22:28 |
| clayg | ... on saio | 22:28 |
| clayg | well - i should rephrase - i *always* get that in syslog restarting - but it only shows up at the console with pyeclib 1.3.1 | 22:33 |
| *** mtreinish has quit IRC | 22:41 | |
| kota_ | clayg: please check your liberasurecode version? | 22:46 |
| *** mtreinish has joined #openstack-swift | 22:47 | |
| *** klrmn has joined #openstack-swift | 22:49 | |
| *** lcurtis has quit IRC | 22:52 | |
| *** vern has quit IRC | 22:55 | |
| *** vern has joined #openstack-swift | 22:57 | |
| *** klrmn has quit IRC | 23:00 | |
| clayg | kota_: it's the one that comes with the distro - 1.1 on xenail | 23:05 |
| clayg | is there instructions for installing over distro package versions? | 23:05 |
| clayg | is there anywhere to get pre-built binaries or is it essentailly "compile by hand and put the .so some where in your ld path" | 23:06 |
| clayg | just git clone - configure; make; make install ???? | 23:06 |
| *** klrmn has joined #openstack-swift | 23:15 | |
| kota_ | clayg: clone, bindep, apt, autogen.sh, configure, make, make install | 23:29 |
| kota_ | Sorry, now typing from mobile | 23:29 |
| kota_ | clayg: That's procedure to install, maybe you can see that in sort of README | 23:31 |
| kota_ | Or repo after xenail (forgot the name) looks to include liberasurecode 1.2.1 | 23:33 |
| kota_ | anyway liberasurecode 1.1 + Pyeclib 1.3.1 will show the message, liberasurecode>=1.2.0 is needed | 23:36 |
| *** klamath has quit IRC | 23:40 | |
| clayg | yakkety | 23:41 |
| clayg | 16.10 - comes out tomorrow | 23:42 |
| kota_ | Ah, tomorrow :/ | 23:50 |
| kota_ | Maybe i saw it for rc staff | 23:50 |
| *** dmorita_ has joined #openstack-swift | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!