| *** abhitechie has joined #openstack-swift | 00:05 | |
| thurloat | in the example proxy-server.conf, the keystone pipeline is missing "listing_formats" middleware, is this omission on purpose, or naw? | 00:08 |
|---|---|---|
| timburke | it's an omission, but it doesn't really matter; the proxy server will auto-insert it if it isn't present | 00:10 |
| timburke | we've gotta be able to still work with an old config, after all | 00:11 |
| notmyname | oversight | 00:11 |
| notmyname | almost certainly | 00:11 |
| *** wes_dillingham has quit IRC | 00:31 | |
| *** wes_dillingham has joined #openstack-swift | 00:35 | |
| thurloat | i'll include it anyway :) | 00:36 |
| thurloat | going through our confs with a comb before the upgrade from 1.13.hack to 2.16.0 | 00:37 |
| timburke | whooo! it's gonna be a great upgrade :-) | 00:42 |
| notmyname | no meeting this week, neither 0700 nor 2100 | 00:52 |
| notmyname | acoles: kota_: mattoliverau: rledisez: mahatic: jungleboyj: ^ | 00:52 |
| mattoliverau | notmyname: kk | 00:53 |
| mattoliverau | rledisez is still in Australia somewhere. So assume he's on leave anyway :) | 00:54 |
| notmyname | cool | 00:54 |
| jungleboyj | notmyname: cool. I just got back. Trying to recover and catch up. | 00:59 |
| jungleboyj | mattoliverau: I have no idea how you have done this repeatedly in the past. | 00:59 |
| mattoliverau | jungleboyj: lol, do it 4 times a year for about 4 years and it gets easier.. or rather, it becomes the new normal, not really easier ;p | 01:01 |
| mattoliverau | just imbrance the jetlag.. let it become one with you :P | 01:02 |
| jungleboyj | I have all respect for you. Can't concentrate today too save my life. | 01:04 |
| *** cshastri has joined #openstack-swift | 01:13 | |
| *** tovin07_ has joined #openstack-swift | 01:16 | |
| kota_ | good morning. | 01:22 |
| kota_ | notmyname: ack for no meeting. thanks for sharing | 01:22 |
| *** m_kazuhiro has joined #openstack-swift | 01:23 | |
| *** gyee_ has quit IRC | 01:23 | |
| *** tovin07__ has joined #openstack-swift | 01:24 | |
| *** tovin07_ has quit IRC | 01:28 | |
| *** tovin07__ has quit IRC | 01:28 | |
| *** tovin07_ has joined #openstack-swift | 01:30 | |
| *** wes_dillingham has quit IRC | 01:34 | |
| *** links has joined #openstack-swift | 02:03 | |
| *** threestrands has joined #openstack-swift | 02:09 | |
| *** itlinux has joined #openstack-swift | 02:11 | |
| *** itlinux has quit IRC | 02:17 | |
| *** m_kazuhiro_ has joined #openstack-swift | 02:24 | |
| m_kazuhiro_ | good morning | 02:25 |
| m_kazuhiro | oops I logged in irc twice. | 02:27 |
| *** m_kazuhiro_ has quit IRC | 02:27 | |
| *** JimCheung has quit IRC | 02:35 | |
| *** JimCheung has joined #openstack-swift | 02:35 | |
| *** JimCheung has quit IRC | 02:39 | |
| *** tovin07__ has joined #openstack-swift | 02:42 | |
| *** tovin07_ has quit IRC | 02:44 | |
| *** caiobrentano has quit IRC | 02:48 | |
| *** Sukhdev has quit IRC | 02:51 | |
| *** tovin07 has joined #openstack-swift | 02:53 | |
| *** tovin07__ has quit IRC | 02:56 | |
| -openstackstatus- NOTICE: Due to an unexpected outage with Zuul (1 hour), you'll need to recheck any jobs that were in progress. Sorry for the inconvenience. | 02:57 | |
| *** tovin07 has quit IRC | 02:58 | |
| *** itlinux has joined #openstack-swift | 03:15 | |
| *** two_tired has joined #openstack-swift | 03:16 | |
| kong | timburke: are you still around? | 03:27 |
| kong | i found the root cause of the issue, our nginx cache is filled up occa | 03:27 |
| kong | occasionally, which cause 206 received in client side | 03:28 |
| kong | and swift client is trying to using 'accept-ranges' to fetch all the bytes, finally, the file size is correct, but md5 check failed | 03:28 |
| kong | i did md5sum on the downloaded object, it is different with the object etag in swift | 03:29 |
| kong | i am confused | 03:29 |
| kong | or anyone else can help? | 03:31 |
| kong | the swift client debug log: http://dpaste.com/0CD9FQZ | 03:32 |
| *** rcernin has quit IRC | 03:34 | |
| *** tovin07_ has joined #openstack-swift | 03:35 | |
| *** rcernin has joined #openstack-swift | 03:36 | |
| *** abhitechie has quit IRC | 03:38 | |
| *** kei_yama has quit IRC | 03:38 | |
| kong | i'm not sure https://bugs.launchpad.net/swift/+bug/1717401 is related | 03:46 |
| openstack | Launchpad bug 1717401 in OpenStack Object Storage (swift) "Client(curl) gets partial content while proxy-server got ChunkRead timeout " [High,Fix released] | 03:46 |
| *** ^andrea^ has quit IRC | 03:47 | |
| kong | i will be offline in a few minutes, please if anyone has some suggestion or hints, leave message to me, thanks! | 03:48 |
| *** janonymous has joined #openstack-swift | 03:49 | |
| *** ukaynar has joined #openstack-swift | 03:56 | |
| *** kei_yama has joined #openstack-swift | 04:00 | |
| *** abhitechie has joined #openstack-swift | 04:15 | |
| *** m_kazuhiro has quit IRC | 04:26 | |
| *** psachin has joined #openstack-swift | 04:28 | |
| *** m_kazuhiro has joined #openstack-swift | 04:35 | |
| mahatic | notmyname: ack on the no meeting, thanks | 04:35 |
| jungleboyj | mattoliverau: Also, the fact that I became useful at 6 pm after being useless all day! Yikes! | 05:12 |
| *** itlinux has quit IRC | 05:27 | |
| *** abhitechie has quit IRC | 05:47 | |
| *** two_tired has quit IRC | 05:55 | |
| *** ukaynar has quit IRC | 06:33 | |
| *** ukaynar has joined #openstack-swift | 06:34 | |
| *** abhitechie has joined #openstack-swift | 06:34 | |
| *** threestrands has quit IRC | 06:34 | |
| *** ukaynar has quit IRC | 06:38 | |
| *** ukaynar has joined #openstack-swift | 06:38 | |
| *** gabor_antal_ has joined #openstack-swift | 06:40 | |
| *** ukaynar_ has joined #openstack-swift | 06:43 | |
| *** ukaynar has quit IRC | 06:43 | |
| *** gabor_antal has quit IRC | 06:43 | |
| *** ukaynar_ has quit IRC | 06:47 | |
| *** ukaynar has joined #openstack-swift | 06:47 | |
| *** ukaynar has quit IRC | 06:52 | |
| *** ukaynar has joined #openstack-swift | 06:52 | |
| *** ukaynar has quit IRC | 06:56 | |
| *** ukaynar has joined #openstack-swift | 06:57 | |
| *** abhitechie has quit IRC | 06:59 | |
| *** ukaynar has quit IRC | 07:01 | |
| *** ukaynar has joined #openstack-swift | 07:01 | |
| *** ukaynar has quit IRC | 07:06 | |
| *** ukaynar_ has joined #openstack-swift | 07:06 | |
| *** ukaynar_ has quit IRC | 07:10 | |
| *** ukaynar has joined #openstack-swift | 07:10 | |
| *** spectr has joined #openstack-swift | 07:13 | |
| *** SkyRocknRoll has quit IRC | 07:15 | |
| *** ukaynar_ has joined #openstack-swift | 07:15 | |
| *** ukaynar has quit IRC | 07:15 | |
| *** ukaynar_ has quit IRC | 07:19 | |
| *** rcernin has quit IRC | 07:20 | |
| *** ukaynar has joined #openstack-swift | 07:20 | |
| *** ukaynar has quit IRC | 07:24 | |
| *** ukaynar has joined #openstack-swift | 07:24 | |
| *** armaan has joined #openstack-swift | 07:28 | |
| *** ukaynar_ has joined #openstack-swift | 07:29 | |
| *** ukaynar has quit IRC | 07:29 | |
| *** hoonetorg has quit IRC | 07:30 | |
| *** cshastri has quit IRC | 07:32 | |
| *** ukaynar_ has quit IRC | 07:33 | |
| *** ukaynar has joined #openstack-swift | 07:34 | |
| *** m_kazuhiro has quit IRC | 07:35 | |
| *** cshastri has joined #openstack-swift | 07:38 | |
| *** ukaynar has quit IRC | 07:38 | |
| *** spectr has quit IRC | 07:38 | |
| *** ukaynar has joined #openstack-swift | 07:38 | |
| *** ukaynar has quit IRC | 07:43 | |
| *** hoonetorg has joined #openstack-swift | 07:43 | |
| *** ukaynar has joined #openstack-swift | 07:43 | |
| *** ukaynar has quit IRC | 07:47 | |
| *** ukaynar has joined #openstack-swift | 07:48 | |
| *** pcaruana has joined #openstack-swift | 07:51 | |
| *** ukaynar has quit IRC | 07:52 | |
| *** ukaynar has joined #openstack-swift | 07:52 | |
| *** ukaynar has quit IRC | 07:57 | |
| *** ukaynar_ has joined #openstack-swift | 07:57 | |
| *** PagliaccisCloud has quit IRC | 08:01 | |
| *** ukaynar has joined #openstack-swift | 08:01 | |
| *** ukaynar_ has quit IRC | 08:02 | |
| *** abhitechie has joined #openstack-swift | 08:03 | |
| *** PagliaccisCloud has joined #openstack-swift | 08:05 | |
| *** ukaynar has quit IRC | 08:06 | |
| *** ukaynar has joined #openstack-swift | 08:06 | |
| *** ukaynar_ has joined #openstack-swift | 08:11 | |
| *** ukaynar has quit IRC | 08:11 | |
| *** ukaynar_ has quit IRC | 08:15 | |
| *** ukaynar has joined #openstack-swift | 08:15 | |
| *** tesseract has joined #openstack-swift | 08:17 | |
| *** ukaynar has quit IRC | 08:20 | |
| *** ukaynar has joined #openstack-swift | 08:20 | |
| *** ukaynar has quit IRC | 08:24 | |
| *** ukaynar has joined #openstack-swift | 08:25 | |
| *** tovin07_ has quit IRC | 08:25 | |
| *** oshritf has joined #openstack-swift | 08:28 | |
| *** ukaynar_ has joined #openstack-swift | 08:29 | |
| *** ukaynar has quit IRC | 08:29 | |
| *** SkyRocknRoll has joined #openstack-swift | 08:32 | |
| *** oshritf has quit IRC | 08:34 | |
| *** ukaynar has joined #openstack-swift | 08:34 | |
| *** ukaynar_ has quit IRC | 08:34 | |
| *** oshritf has joined #openstack-swift | 08:36 | |
| *** kei_yama has quit IRC | 08:36 | |
| *** oshritf has quit IRC | 08:38 | |
| *** ukaynar has quit IRC | 08:38 | |
| *** ukaynar has joined #openstack-swift | 08:39 | |
| *** oshritf has joined #openstack-swift | 08:40 | |
| *** geaaru has joined #openstack-swift | 08:43 | |
| *** ukaynar_ has joined #openstack-swift | 08:43 | |
| *** ukaynar has quit IRC | 08:44 | |
| *** ukaynar_ has quit IRC | 08:47 | |
| *** ukaynar has joined #openstack-swift | 08:48 | |
| *** oshritf has quit IRC | 08:52 | |
| *** ukaynar_ has joined #openstack-swift | 08:52 | |
| *** ukaynar has quit IRC | 08:52 | |
| *** ukaynar has joined #openstack-swift | 08:57 | |
| *** ukaynar_ has quit IRC | 08:57 | |
| *** ukaynar has quit IRC | 09:01 | |
| *** ukaynar has joined #openstack-swift | 09:02 | |
| *** gkadam has joined #openstack-swift | 09:02 | |
| *** ukaynar has quit IRC | 09:06 | |
| *** ukaynar has joined #openstack-swift | 09:06 | |
| *** oshritf has joined #openstack-swift | 09:07 | |
| *** zenpwner has joined #openstack-swift | 09:08 | |
| *** klrmn has quit IRC | 09:09 | |
| *** ukaynar_ has joined #openstack-swift | 09:11 | |
| *** ukaynar has quit IRC | 09:11 | |
| *** cbartz has joined #openstack-swift | 09:13 | |
| *** abhitechie has quit IRC | 09:13 | |
| *** ukaynar_ has quit IRC | 09:15 | |
| *** ukaynar has joined #openstack-swift | 09:16 | |
| acoles | good morning | 09:17 |
| acoles | notmyname: ack no meeting, thanks | 09:17 |
| *** ukaynar has quit IRC | 09:20 | |
| *** ukaynar has joined #openstack-swift | 09:20 | |
| *** ukaynar_ has joined #openstack-swift | 09:25 | |
| *** ukaynar has quit IRC | 09:25 | |
| *** ukaynar has joined #openstack-swift | 09:29 | |
| *** ukaynar_ has quit IRC | 09:30 | |
| *** ukaynar has quit IRC | 09:34 | |
| *** ukaynar has joined #openstack-swift | 09:34 | |
| *** ukaynar has quit IRC | 09:39 | |
| *** ukaynar has joined #openstack-swift | 09:39 | |
| *** ukaynar has quit IRC | 09:43 | |
| *** ukaynar has joined #openstack-swift | 09:43 | |
| *** oshritf has quit IRC | 09:48 | |
| *** ukaynar_ has joined #openstack-swift | 09:48 | |
| *** ukaynar has quit IRC | 09:48 | |
| *** ukaynar_ has quit IRC | 09:53 | |
| *** ukaynar has joined #openstack-swift | 09:53 | |
| *** jarbod_ has joined #openstack-swift | 09:54 | |
| *** abhitechie has joined #openstack-swift | 09:54 | |
| *** ukaynar_ has joined #openstack-swift | 09:57 | |
| *** ukaynar has quit IRC | 09:57 | |
| *** oshritf has joined #openstack-swift | 09:58 | |
| *** ukaynar_ has quit IRC | 10:02 | |
| *** ukaynar has joined #openstack-swift | 10:02 | |
| *** ukaynar has quit IRC | 10:06 | |
| *** ukaynar has joined #openstack-swift | 10:06 | |
| *** ukaynar_ has joined #openstack-swift | 10:11 | |
| *** ukaynar has quit IRC | 10:11 | |
| *** zenpwner has quit IRC | 10:15 | |
| *** ukaynar_ has quit IRC | 10:15 | |
| *** ukaynar has joined #openstack-swift | 10:16 | |
| *** mat128 has joined #openstack-swift | 10:20 | |
| *** ukaynar has quit IRC | 10:20 | |
| *** ukaynar has joined #openstack-swift | 10:20 | |
| *** ukaynar_ has joined #openstack-swift | 10:25 | |
| *** ukaynar has quit IRC | 10:25 | |
| bob_cheesey | morning all, i'm currently looking at reworking our SSL termination in front of the proxy servers to use haproxy and I was looking fora definitive list of verbs I need to allow through. at the moment i've got GET, PUT, POST, DELETE and HEAD | 10:27 |
| bob_cheesey | have i missed any? | 10:27 |
| *** ukaynar_ has quit IRC | 10:29 | |
| *** ukaynar has joined #openstack-swift | 10:30 | |
| *** ukaynar has quit IRC | 10:34 | |
| *** ukaynar has joined #openstack-swift | 10:34 | |
| * kota_ is playing with Proxy-FS, trying to install that into my SAIO | 10:37 | |
| kota_ | good morning acoles | 10:37 |
| *** ukaynar_ has joined #openstack-swift | 10:39 | |
| *** ukaynar has quit IRC | 10:39 | |
| kota_ | hmm... it looks the build succeeded but not working yet. though a bunch of warnings appeared in the log :/ | 10:40 |
| acoles | bob_cheesey: COPY https://developer.openstack.org/api-ref/object-store/ | 10:40 |
| bob_cheesey | ah wonderful, thanks acoles! | 10:41 |
| *** ukaynar_ has quit IRC | 10:44 | |
| *** ukaynar has joined #openstack-swift | 10:44 | |
| acoles | bob_cheesey: also, OPTIONS | 10:46 |
| acoles | hmmm, that doesn't seem to be documented in API docs. mentioned here https://docs.openstack.org/swift/latest/cors.html | 10:47 |
| *** ukaynar has quit IRC | 10:48 | |
| *** ukaynar has joined #openstack-swift | 10:48 | |
| acoles | kota_: o/ | 10:49 |
| bob_cheesey | acoles: thanks for that! | 10:51 |
| *** ukaynar has quit IRC | 10:52 | |
| *** ukaynar has joined #openstack-swift | 10:53 | |
| kota_ | ah, maybe `make install` in regression_test.py requires super user privilege. | 10:55 |
| *** baruna has joined #openstack-swift | 10:55 | |
| kota_ | ... no? hmm... interesting | 10:56 |
| *** ukaynar has quit IRC | 10:57 | |
| *** ukaynar has joined #openstack-swift | 10:57 | |
| *** ukaynar has quit IRC | 11:02 | |
| *** ukaynar has joined #openstack-swift | 11:02 | |
| *** armaan has quit IRC | 11:04 | |
| *** armaan has joined #openstack-swift | 11:04 | |
| *** mvk has quit IRC | 11:05 | |
| *** ukaynar has quit IRC | 11:06 | |
| *** ukaynar has joined #openstack-swift | 11:07 | |
| *** ukaynar_ has joined #openstack-swift | 11:11 | |
| *** ukaynar has quit IRC | 11:11 | |
| *** ukaynar_ has quit IRC | 11:16 | |
| *** ukaynar has joined #openstack-swift | 11:16 | |
| *** ukaynar has quit IRC | 11:20 | |
| *** ukaynar has joined #openstack-swift | 11:21 | |
| *** cshastri has quit IRC | 11:22 | |
| kota_ | hmm... it looks making vfs is missing SAMBA_PATH and the VERSION file to include. | 11:23 |
| *** ukaynar has quit IRC | 11:25 | |
| *** ukaynar has joined #openstack-swift | 11:25 | |
| kong | hi guys, is this the expected 'Range' request behavior: http://dpaste.com/3HQXPRR | 11:28 |
| kong | i specified the range bytes: 0-1159721451, but the downloaded file size is 1159721877 | 11:28 |
| *** ukaynar has quit IRC | 11:30 | |
| *** ukaynar has joined #openstack-swift | 11:30 | |
| *** mvk has joined #openstack-swift | 11:32 | |
| *** ukaynar has quit IRC | 11:34 | |
| kota_ | OMG, another regression_test.py found in jrpcclient dir | 11:34 |
| *** ukaynar has joined #openstack-swift | 11:35 | |
| kota_ | and something missing (e.g. conf) for the script. | 11:36 |
| *** ukaynar_ has joined #openstack-swift | 11:39 | |
| *** ukaynar has quit IRC | 11:39 | |
| kota_ | k, let's deep dive more tomorrow. | 11:40 |
| *** ukaynar has joined #openstack-swift | 11:44 | |
| *** ukaynar_ has quit IRC | 11:44 | |
| *** ukaynar has quit IRC | 11:48 | |
| *** ukaynar has joined #openstack-swift | 11:49 | |
| *** ukaynar has quit IRC | 11:53 | |
| *** ukaynar has joined #openstack-swift | 11:53 | |
| mattoliverau | kong: have you looked at the file? It might be coming back as a muli-part mime. So we extra data for mime boundaries etc.. I'd go check but am in bed :) | 11:57 |
| *** ukaynar has quit IRC | 11:57 | |
| *** ukaynar has joined #openstack-swift | 11:58 | |
| *** ukaynar has quit IRC | 12:02 | |
| *** ukaynar has joined #openstack-swift | 12:02 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift master: Updated from global requirements https://review.openstack.org/88736 | 12:05 |
| kong | mattoliverau: thanks for the response, the actual problem we are suffering is http://dpaste.com/0CD9FQZ | 12:05 |
| kong | after multi-range download, the md5 check failed :-( | 12:06 |
| *** ukaynar has quit IRC | 12:07 | |
| *** ukaynar_ has joined #openstack-swift | 12:07 | |
| *** ianychoi has quit IRC | 12:07 | |
| *** ukaynar_ has quit IRC | 12:12 | |
| *** ukaynar has joined #openstack-swift | 12:12 | |
| *** ukaynar has quit IRC | 12:16 | |
| *** ukaynar has joined #openstack-swift | 12:16 | |
| *** abhitechie has quit IRC | 12:19 | |
| *** ukaynar has quit IRC | 12:21 | |
| *** ukaynar has joined #openstack-swift | 12:21 | |
| acoles | kong: that looks like this bug which should be fixed in latest release https://bugs.launchpad.net/python-swiftclient/+bug/1642021 | 12:22 |
| openstack | Launchpad bug 1642021 in python-swiftclient "download with Range header yields error message" [Medium,Fix released] | 12:22 |
| *** ukaynar_ has joined #openstack-swift | 12:26 | |
| *** ukaynar has quit IRC | 12:26 | |
| *** ukaynar has joined #openstack-swift | 12:30 | |
| *** ukaynar_ has quit IRC | 12:30 | |
| *** ukaynar has quit IRC | 12:35 | |
| *** ukaynar has joined #openstack-swift | 12:35 | |
| *** armaan has quit IRC | 12:39 | |
| *** ukaynar_ has joined #openstack-swift | 12:39 | |
| *** ukaynar has quit IRC | 12:40 | |
| *** ukaynar_ has quit IRC | 12:44 | |
| *** ukaynar has joined #openstack-swift | 12:44 | |
| *** ukaynar has quit IRC | 12:49 | |
| *** ukaynar has joined #openstack-swift | 12:49 | |
| *** ukaynar has quit IRC | 12:53 | |
| *** ukaynar has joined #openstack-swift | 12:53 | |
| *** ukaynar has quit IRC | 12:58 | |
| *** ukaynar has joined #openstack-swift | 12:58 | |
| *** ukaynar has quit IRC | 13:02 | |
| *** ukaynar has joined #openstack-swift | 13:03 | |
| *** ianychoi has joined #openstack-swift | 13:05 | |
| *** ukaynar has quit IRC | 13:07 | |
| *** ukaynar has joined #openstack-swift | 13:07 | |
| *** wes_dillingham has joined #openstack-swift | 13:09 | |
| *** ukaynar has quit IRC | 13:12 | |
| *** ukaynar has joined #openstack-swift | 13:12 | |
| *** janonymous has quit IRC | 13:12 | |
| *** ukaynar has quit IRC | 13:16 | |
| *** ukaynar_ has joined #openstack-swift | 13:17 | |
| *** mat128 has quit IRC | 13:17 | |
| *** guimaluf has quit IRC | 13:18 | |
| *** ukaynar_ has quit IRC | 13:21 | |
| *** ukaynar has joined #openstack-swift | 13:21 | |
| *** links has quit IRC | 13:22 | |
| *** links has joined #openstack-swift | 13:24 | |
| *** ukaynar has quit IRC | 13:26 | |
| *** ukaynar has joined #openstack-swift | 13:26 | |
| *** ukaynar has quit IRC | 13:30 | |
| *** ukaynar has joined #openstack-swift | 13:30 | |
| *** ukaynar has quit IRC | 13:35 | |
| *** ukaynar_ has joined #openstack-swift | 13:35 | |
| *** armaan has joined #openstack-swift | 13:38 | |
| *** ukaynar_ has quit IRC | 13:40 | |
| *** ukaynar has joined #openstack-swift | 13:40 | |
| *** oshritf has quit IRC | 13:44 | |
| *** oshritf has joined #openstack-swift | 13:44 | |
| *** ukaynar_ has joined #openstack-swift | 13:44 | |
| *** ukaynar has quit IRC | 13:44 | |
| *** ukaynar_ has quit IRC | 13:49 | |
| *** ukaynar has joined #openstack-swift | 13:49 | |
| *** links has quit IRC | 13:53 | |
| *** ukaynar has quit IRC | 13:53 | |
| *** ukaynar has joined #openstack-swift | 13:54 | |
| *** mabrams has joined #openstack-swift | 13:55 | |
| *** ukaynar has quit IRC | 13:58 | |
| *** ukaynar has joined #openstack-swift | 13:58 | |
| *** ukaynar has quit IRC | 14:03 | |
| *** ukaynar_ has joined #openstack-swift | 14:03 | |
| *** ukaynar_ has quit IRC | 14:07 | |
| *** ukaynar has joined #openstack-swift | 14:07 | |
| *** ukaynar has quit IRC | 14:12 | |
| *** ukaynar has joined #openstack-swift | 14:12 | |
| *** ukaynar has quit IRC | 14:16 | |
| *** ukaynar has joined #openstack-swift | 14:17 | |
| *** ukaynar_ has joined #openstack-swift | 14:21 | |
| *** ukaynar has quit IRC | 14:21 | |
| *** ukaynar_ has quit IRC | 14:26 | |
| *** ukaynar has joined #openstack-swift | 14:26 | |
| *** ukaynar has quit IRC | 14:30 | |
| *** ukaynar has joined #openstack-swift | 14:31 | |
| *** ukaynar has quit IRC | 14:35 | |
| *** ukaynar has joined #openstack-swift | 14:35 | |
| *** armaan has quit IRC | 14:37 | |
| *** ukaynar_ has joined #openstack-swift | 14:40 | |
| *** ukaynar has quit IRC | 14:40 | |
| *** ukaynar_ has quit IRC | 14:45 | |
| *** ukaynar has joined #openstack-swift | 14:45 | |
| *** ukaynar_ has joined #openstack-swift | 14:49 | |
| *** ukaynar has quit IRC | 14:49 | |
| *** ukaynar_ has quit IRC | 14:53 | |
| *** ukaynar has joined #openstack-swift | 14:54 | |
| *** mat128 has joined #openstack-swift | 14:55 | |
| *** ukaynar has quit IRC | 14:58 | |
| *** ukaynar has joined #openstack-swift | 14:58 | |
| *** armaan has joined #openstack-swift | 15:01 | |
| *** armaan has quit IRC | 15:03 | |
| *** ukaynar_ has joined #openstack-swift | 15:03 | |
| *** armaan has joined #openstack-swift | 15:03 | |
| *** ukaynar has quit IRC | 15:04 | |
| *** mabrams has quit IRC | 15:07 | |
| *** ukaynar_ has quit IRC | 15:08 | |
| *** ukaynar has joined #openstack-swift | 15:08 | |
| *** links has joined #openstack-swift | 15:11 | |
| *** ukaynar has quit IRC | 15:12 | |
| *** ukaynar has joined #openstack-swift | 15:12 | |
| *** baruna has quit IRC | 15:14 | |
| *** ukaynar has quit IRC | 15:17 | |
| *** ukaynar has joined #openstack-swift | 15:17 | |
| *** vint_bra has joined #openstack-swift | 15:19 | |
| *** ukaynar has quit IRC | 15:22 | |
| *** ukaynar_ has joined #openstack-swift | 15:22 | |
| *** ukaynar_ has quit IRC | 15:26 | |
| *** ukaynar has joined #openstack-swift | 15:26 | |
| *** ukaynar has quit IRC | 15:30 | |
| *** ukaynar has joined #openstack-swift | 15:31 | |
| *** ukaynar_ has joined #openstack-swift | 15:35 | |
| *** ukaynar has quit IRC | 15:36 | |
| *** klrmn has joined #openstack-swift | 15:39 | |
| *** ukaynar_ has quit IRC | 15:40 | |
| *** itlinux has joined #openstack-swift | 16:01 | |
| *** chsc has joined #openstack-swift | 16:37 | |
| *** chsc has joined #openstack-swift | 16:37 | |
| *** vint_bra has quit IRC | 16:39 | |
| *** ukaynar has joined #openstack-swift | 16:47 | |
| *** oshritf has quit IRC | 16:49 | |
| *** links has quit IRC | 17:00 | |
| *** psachin has quit IRC | 17:02 | |
| *** tesseract has quit IRC | 17:03 | |
| *** cbartz has quit IRC | 17:07 | |
| *** d0ugal has quit IRC | 17:12 | |
| *** SkyRocknRoll_ has joined #openstack-swift | 17:14 | |
| *** JimCheung has joined #openstack-swift | 17:16 | |
| *** vint_bra has joined #openstack-swift | 17:16 | |
| *** klrmn has quit IRC | 17:17 | |
| *** mat128 has quit IRC | 17:17 | |
| *** vint_bra has quit IRC | 17:24 | |
| *** gyee_ has joined #openstack-swift | 17:34 | |
| *** gkadam has quit IRC | 17:37 | |
| *** SkyRocknRoll_ has quit IRC | 17:40 | |
| timburke | idk, that swiftclient bug's mainly talking about if you include a `-H Range: bytes=X-Y` header on the cli | 17:51 |
| *** SkyRocknRoll_ has joined #openstack-swift | 17:53 | |
| *** SkyRocknRoll_ has quit IRC | 18:01 | |
| timburke | kong: fwiw, that swiftclient log tells us we made a request for the whole object, got a 200 (which told us it should be 4205802364 bytes), read 1096938176 bytes, then the connection dropped | 18:04 |
| timburke | did another request for just the bytes starting from 1096938176, was told to expect 3108864188 (which together with the bytes we have gets us back to the original total size), but we can only read 1091629760 bytes before the connection dies | 18:04 |
| timburke | we retry again, and again, and each time the invariant <range lower bound> + <response content length> == <original content length>, which is good | 18:05 |
| timburke | i suppose there's a chance we're not calculating the range lower-bound correctly? i'd be curious about the size of the downloaded file -- if that didn't match the expected size, it may point to a bug in swiftclient. if it *does*... maybe whatever caused the dropped connection also corrupted some in-flight data? | 18:05 |
| timburke | on the curl result, i'm curious what extra output you'd get with a --verbose flag -- i'm guessing curl would give some sort of warning about not receiving as many bytes as it was expecting | 18:06 |
| timburke | i suppose that's what the -S was about... hadn't used that before... i might have to look into it some more... | 18:09 |
| *** klrmn has joined #openstack-swift | 18:11 | |
| *** pcaruana has quit IRC | 18:11 | |
| *** JimCheung has quit IRC | 18:12 | |
| *** JimCheung has joined #openstack-swift | 18:13 | |
| *** mat128 has joined #openstack-swift | 18:13 | |
| *** SkyRocknRoll_ has joined #openstack-swift | 18:17 | |
| *** mvk has quit IRC | 18:17 | |
| *** samueldmq has quit IRC | 18:19 | |
| *** samueldmq has joined #openstack-swift | 18:19 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-swiftclient master: Updated from global requirements https://review.openstack.org/89250 | 18:30 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/swift master: Updated from global requirements https://review.openstack.org/88736 | 18:33 |
| *** geaaru has quit IRC | 18:41 | |
| *** SkyRocknRoll_ has quit IRC | 18:46 | |
| *** itlinux has quit IRC | 18:54 | |
| *** SkyRocknRoll_ has joined #openstack-swift | 19:03 | |
| *** mvk has joined #openstack-swift | 19:06 | |
| *** d0ugal has joined #openstack-swift | 19:21 | |
| *** Sukhdev has joined #openstack-swift | 19:28 | |
| *** itlinux has joined #openstack-swift | 19:30 | |
| *** JimCheung has quit IRC | 19:51 | |
| *** JimCheung has joined #openstack-swift | 19:52 | |
| *** JimCheung has quit IRC | 19:56 | |
| *** flwang has quit IRC | 20:12 | |
| *** itlinux has quit IRC | 20:19 | |
| *** d0ugal_ has joined #openstack-swift | 20:25 | |
| *** flwang has joined #openstack-swift | 20:25 | |
| *** d0ugal has quit IRC | 20:26 | |
| *** JimCheung has joined #openstack-swift | 20:38 | |
| *** Sukhdev has quit IRC | 20:39 | |
| *** itlinux has joined #openstack-swift | 20:40 | |
| *** d0ugal_ has quit IRC | 20:40 | |
| *** d0ugal has joined #openstack-swift | 20:40 | |
| *** d0ugal has quit IRC | 20:40 | |
| *** d0ugal has joined #openstack-swift | 20:40 | |
| *** armaan has quit IRC | 20:40 | |
| *** JimCheung has quit IRC | 20:42 | |
| tdasilva | no meeting today, right? | 20:50 |
| *** wes_dillingham has quit IRC | 20:51 | |
| *** SkyRocknRoll_ has quit IRC | 20:52 | |
| *** d0ugal has quit IRC | 20:52 | |
| *** armaan has joined #openstack-swift | 20:55 | |
| *** itlinux has quit IRC | 21:00 | |
| *** oshritf has joined #openstack-swift | 21:01 | |
| *** spetersen has joined #openstack-swift | 21:04 | |
| *** JimCheung has joined #openstack-swift | 21:18 | |
| *** itlinux has joined #openstack-swift | 21:19 | |
| *** armaan has quit IRC | 21:31 | |
| *** threestrands has joined #openstack-swift | 21:32 | |
| *** threestrands has quit IRC | 21:32 | |
| *** threestrands has joined #openstack-swift | 21:32 | |
| *** oshritf has quit IRC | 21:33 | |
| spetersen | We replaced full drives with larger ones in just one zone in order to help alleviate a situation with full disks. After initiating the rebalance we have not been able to find certain objects. We know they exist and we can find the "missing" objects using the old object.ring.gz and swift-get-nodes. QUESTION: Will replication eventually move the "missing" objects into the proper place or do we need to do | 21:33 |
| spetersen | something manually? | 21:33 |
| clayg | the other replicas should be able to service the READ requests... unless I guess if the other replica(s?) was on the devices you pulled out to make room for the larger ones? | 21:37 |
| clayg | spetersen: the replicators should be working to repair all assigned partition replicas to full health - you just distribute the new rings and the replicators will do the work - but depending on the tuning and available io it could take a while... you might want to monitor it closely. | 21:39 |
| *** JimCheung has quit IRC | 21:40 | |
| clayg | something like https://gist.github.com/clayg/5d3b3ca13dea15f6a48efaa3cea92d7d might give you some additional insight to whats going on with the partition movement | 21:41 |
| *** itlinux has quit IRC | 21:42 | |
| kong | timburke: the downloaded file size is correct, no error shown for curl :-( | 21:45 |
| kong | acoles: thanks for the bug link, but i don't think it's the same issue with us. We are downloading the whole object, and swift client retries for several times (because of a known issue of our nginx) using multi-range request. The md5 check for the downloaded file failed although the file size is expected. | 21:48 |
| kong | can anyone help me please? | 21:49 |
| spetersen | Thanks Clayg! To clarify, the other replicas do exist on different zones and a different partition number, we can download them using the old ring file, the object exists in zone 2 and 3 but not 1 because we replaced those drives with new larger ones (which is expected). If we use the current ring the locations provided by "swift-get-nodes -a" do not contain the object. Do you think they will "magically" | 21:59 |
| spetersen | appear in the new locations in the future? | 21:59 |
| spetersen | Thanks Clayg! To clarify, the other replicas do exist on different zones and a different partition number, we can download them using the old ring file, the object exists in zone 2 and 3 but not 1 because we replaced those drives with new larger ones (which is expected). If we use the current ring, the locations provided by "swift-get-nodes -a" do not contain the object. Do you think they will "magically" | 22:00 |
| spetersen | appear in the new locations in the future? | 22:00 |
| spetersen | I think this may be similar to this issue you commented on back in 2013, https://answers.launchpad.net/swift/+question/231972 | 22:01 |
| spetersen | That old article is regarding missing accounts where as ours is regarding objects. | 22:02 |
| clayg | sounds like it - that's what rebalance does... it takes one of the replicas of a partition and assigns it to a different device that has more room. From the proxy the request to the object will find the other object from the other replicas (in zone 2 & 3) while the one in zone 1 is being replicated to the new host... https://www.youtube.com/watch?v=ger20cqOypE&feature=youtu.be&t=1140 | 22:02 |
| spetersen | basically what we are noticing is that Swift is not seeing the replicas (even though they are there) We were able to find them using the old ring, however using the new ring if you try to get the file it will 404 because it thinks the copies should be in partition 'a' when they are in partition 'b' we looked through all of the handoff locations as well and where the file is actually located isn't in there | 22:11 |
| spetersen | either | 22:11 |
| *** itlinux has joined #openstack-swift | 22:11 | |
| clayg | so a rebalance shouldn't *change* the partition of a given object - replication won't move an object from one partition to different one - replication/rebalance only moves partitions from node to another | 22:17 |
| spetersen | ok, well I guess we have a problem then because the newly generated ring file is stating the objects live on a different partition number than the old ring file. | 22:18 |
| clayg | spetersen: yeah that's pretty weird | 22:19 |
| clayg | that can happen if you change your hash_path_suffix/prefix in swift.conf - or maybe also if you change the part power | 22:19 |
| clayg | but you *should* be using the same old builder you used originally, adding the new devices - and then rebalance | 22:19 |
| spetersen | I made a mistake, the partition number is the same using the old ring and the current ring. | 22:20 |
| clayg | oh that's good news! | 22:20 |
| clayg | so maybe swift-get-nodes -p <part#> from old and new ring - try to figure out which replica moved and make sure the other replicas are still on the same nodes | 22:21 |
| *** SkyRocknRoll has quit IRC | 22:22 | |
| *** itlinux has quit IRC | 22:23 | |
| spetersen | My mistake, the only difference is which drive it thinks the partition lives on. | 22:23 |
| *** rcernin has joined #openstack-swift | 22:23 | |
| clayg | ok, that sounds good - but again - only one of the replicas of the partition should have moved | 22:23 |
| *** SkyRocknRoll has joined #openstack-swift | 22:25 | |
| spetersen | Does it make a difference that we also changed the weight on some drives on the other nodes? | 22:28 |
| clayg | no - that's great - a single rebalance would still only move one replica | 22:32 |
| clayg | have you been able to verify if *some* of the devices a given partition is assigned were the same in the old ring? | 22:32 |
| spetersen | Can I paste the output of swift-get-nodes -p for the old and the current ring? | 22:34 |
| clayg | sure! | 22:34 |
| *** mvk has quit IRC | 22:35 | |
| spetersen | [root@swift-r1z2n1 swift]# swift-get-nodes /tmp/object.ring.gz -p 225 | 22:43 |
| spetersen | Account None | 22:43 |
| spetersen | ContainerNone | 22:43 |
| spetersen | Object None | 22:43 |
| spetersen | Partition225 | 22:43 |
| spetersen | Hash None | 22:43 |
| spetersen | Server:Port Device192.168.99.106:6000 r1z4n1-d3 | 22:43 |
| spetersen | Server:Port Device192.168.99.105:6000 swift-r1z3n1-d9 | 22:43 |
| spetersen | Server:Port Device192.168.99.103:6000 swift-r1z2n1-d19 | 22:43 |
| spetersen | Server:Port Device192.168.99.102:6000 r1z1n1-d26 [Handoff] | 22:44 |
| *** itlinux has joined #openstack-swift | 22:44 | |
| spetersen | Server:Port Device192.168.99.106:6000 r1z4n1-d6 [Handoff] | 22:44 |
| spetersen | Server:Port Device192.168.99.103:6000 r1z2n1-d5 [Handoff] | 22:44 |
| spetersen | curl -g -I -XHEAD "http://192.168.99.106:6000/r1z4n1-d3/225/None" | 22:44 |
| spetersen | curl -g -I -XHEAD "http://192.168.99.105:6000/swift-r1z3n1-d9/225/None" | 22:44 |
| spetersen | curl -g -I -XHEAD "http://192.168.99.103:6000/swift-r1z2n1-d19/225/None" | 22:44 |
| spetersen | curl -g -I -XHEAD "http://192.168.99.102:6000/r1z1n1-d26/225/None" # [Handoff] | 22:44 |
| spetersen | curl -g -I -XHEAD "http://192.168.99.106:6000/r1z4n1-d6/225/None" # [Handoff] | 22:44 |
| spetersen | curl -g -I -XHEAD "http://192.168.99.103:6000/r1z2n1-d5/225/None" # [Handoff] | 22:44 |
| spetersen | Use your own device location of servers: | 22:44 |
| spetersen | such as "export DEVICE=/srv/node" | 22:44 |
| spetersen | ssh 192.168.99.106 "ls -lah ${DEVICE:-/srv/node*}/r1z4n1-d3/objects/225" | 22:44 |
| spetersen | ssh 192.168.99.105 "ls -lah ${DEVICE:-/srv/node*}/swift-r1z3n1-d9/objects/225" | 22:44 |
| spetersen | ssh 192.168.99.103 "ls -lah ${DEVICE:-/srv/node*}/swift-r1z2n1-d19/objects/225" | 22:44 |
| spetersen | ssh 192.168.99.102 "ls -lah ${DEVICE:-/srv/node*}/r1z1n1-d26/objects/225" # [Handoff] | 22:44 |
| spetersen | ssh 192.168.99.106 "ls -lah ${DEVICE:-/srv/node*}/r1z4n1-d6/objects/225" # [Handoff] | 22:44 |
| spetersen | ssh 192.168.99.103 "ls -lah ${DEVICE:-/srv/node*}/r1z2n1-d5/objects/225" # [Handoff] | 22:44 |
| spetersen | note: `/srv/node*` is used as default value of `devices`, the real value is set in the config file on each storage node. | 22:44 |
| spetersen | [root@swift-r1z2n1 swift]# | 22:44 |
| spetersen | And the new ring | 22:44 |
| spetersen | [root@swift-r1z2n1 swift]# swift-get-nodes object.ring.gz -p 225 | 22:44 |
| spetersen | Account None | 22:44 |
| spetersen | ContainerNone | 22:44 |
| spetersen | Object None | 22:44 |
| spetersen | Partition225 | 22:44 |
| spetersen | Hash None | 22:44 |
| spetersen | Server:Port Device192.168.99.106:6000 r1z4n1-d3 | 22:44 |
| spetersen | Server:Port Device192.168.99.105:6000 r1z3n1-d22 | 22:44 |
| spetersen | Server:Port Device192.168.99.102:6000 r1z1n1-d28 | 22:45 |
| spetersen | Server:Port Device192.168.99.103:6000 r1z2n1-d26 [Handoff] | 22:45 |
| spetersen | Server:Port Device192.168.99.106:6000 r1z4n1-d6 [Handoff] | 22:45 |
| spetersen | Server:Port Device192.168.99.102:6000 r1z1n1-d26 [Handoff] | 22:45 |
| spetersen | curl -g -I -XHEAD "http://192.168.99.106:6000/r1z4n1-d3/225/None" | 22:45 |
| spetersen | curl -g -I -XHEAD "http://192.168.99.105:6000/r1z3n1-d22/225/None" | 22:45 |
| spetersen | curl -g -I -XHEAD "http://192.168.99.102:6000/r1z1n1-d28/225/None" | 22:45 |
| spetersen | curl -g -I -XHEAD "http://192.168.99.103:6000/r1z2n1-d26/225/None" # [Handoff] | 22:45 |
| spetersen | curl -g -I -XHEAD "http://192.168.99.106:6000/r1z4n1-d6/225/None" # [Handoff] | 22:45 |
| spetersen | curl -g -I -XHEAD "http://192.168.99.102:6000/r1z1n1-d26/225/None" # [Handoff] | 22:45 |
| spetersen | Use your own device location of servers: | 22:45 |
| spetersen | such as "export DEVICE=/srv/node" | 22:45 |
| spetersen | ssh 192.168.99.106 "ls -lah ${DEVICE:-/srv/node*}/r1z4n1-d3/objects/225" | 22:45 |
| spetersen | ssh 192.168.99.105 "ls -lah ${DEVICE:-/srv/node*}/r1z3n1-d22/objects/225" | 22:45 |
| spetersen | ssh 192.168.99.102 "ls -lah ${DEVICE:-/srv/node*}/r1z1n1-d28/objects/225" | 22:45 |
| spetersen | ssh 192.168.99.103 "ls -lah ${DEVICE:-/srv/node*}/r1z2n1-d26/objects/225" # [Handoff] | 22:45 |
| spetersen | ssh 192.168.99.106 "ls -lah ${DEVICE:-/srv/node*}/r1z4n1-d6/objects/225" # [Handoff] | 22:45 |
| spetersen | ssh 192.168.99.102 "ls -lah ${DEVICE:-/srv/node*}/r1z1n1-d26/objects/225" # [Handoff] | 22:45 |
| spetersen | note: `/srv/node*` is used as default value of `devices`, the real value is set in the config file on each storage node. | 22:45 |
| spetersen | [root@swift-r1z2n1 swift]# | 22:45 |
| *** d0ugal has joined #openstack-swift | 22:51 | |
| *** mat128 has quit IRC | 22:54 | |
| clayg | spetersen: use a gist or pastebin next time please :D | 23:01 |
| clayg | spetersen: it looks like both rings think the partition was/in on r1z4n1-d3 | 23:02 |
| clayg | you might confirm that - if it's true the proxy should be able to find any objects from that device | 23:02 |
| clayg | but it also looks like 2 of the 3 replicas of the part changed between old and new ring - that shouldn't happen with a single rebalance - or even multiple rebalanace in a short period if you have a sane min_part_hours | 23:03 |
| *** wes_dillingham has joined #openstack-swift | 23:12 | |
| spetersen | well, I saw that as well, you would think the object would exist in r1z4n1-d3 but for whatever reason it does not live on that device. | 23:15 |
| *** itlinux has quit IRC | 23:15 | |
| spetersen | We did find the objects with the old ring, here swift-r1z3n1-d9 and here swift-r1z2n1-d19 | 23:16 |
| *** glyn has joined #openstack-swift | 23:29 | |
| *** spetersen has quit IRC | 23:30 | |
| clayg | interesting... | 23:30 |
| clayg | yeah so the replicators on those nodes should be wanting to replicate out to the correct primary locations | 23:31 |
| clayg | you can turn up the replicator concurrency - make sure you rsync has enough max connections - then maybe turn on handoffs_first mode | 23:31 |
| clayg | tuning is a thing | 23:31 |
| glyn | hi folks, just wondering if anyone can provide any input on whether there is a point at which the number of objects in a single container starts to have negative impacts on performance or reliability? | 23:31 |
| clayg | it's not so much the number as how fast you're doing updates - a container that adds 10K and removes 10K objects every day is way worse than a container with 10M objects that isn't having any more new data written to it | 23:33 |
| *** kei_yama has joined #openstack-swift | 23:34 | |
| clayg | that says - shoot for 4M - 10M is ok, 100M is bad - but it mostly effects how quickly the consistency engine can cycle - which can lead to annoying problems with like... some object getting added but not showing up in the container listing for a long while depending... it can get to a tipping point where it never catches up if you just poor 1K/s updates into it without ever stopping. That's where the rate | 23:35 |
| clayg | limit middleware came from | 23:35 |
| glyn | clayg: thanks for that. Is this included in any of the official documentation? | 23:36 |
| clayg | nah, it depends a lot on the hardware and the usage patterns and expectations... I think everyone that uses swift has their own opinions about what's reasonable. | 23:37 |
| *** ianychoi has quit IRC | 23:37 | |
| glyn | ah, ok | 23:37 |
| clayg | mattoliverau: will tell you just to wait for the new deep containers feature he's working on - it's all pivot tables and magic | 23:38 |
| clayg | i guess timburke and acoles are helping ;) | 23:38 |
| *** chsc has quit IRC | 23:38 | |
| *** abhitechie has joined #openstack-swift | 23:40 | |
| mattoliverau | yeah, soon it wont be an issue cause we'll shard up the container databases so they never get too big for the backend. And the world will be a better place | 23:40 |
| *** JimCheung has joined #openstack-swift | 23:40 | |
| mattoliverau | and clayg's right, it's all pivots and ranges all the way down :P | 23:41 |
| *** JimCheung has quit IRC | 23:45 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!