Monday, 2016-03-28

*** haomaiwang has quit IRC00:01
*** haomaiwang has joined #openstack-swift00:01
*** asettle has quit IRC00:27
*** haomaiwang has quit IRC01:01
*** haomaiwang has joined #openstack-swift01:01
*** haomaiwang has quit IRC01:46
*** baojg has joined #openstack-swift01:47
*** chuck_ has joined #openstack-swift01:50
*** chuck_ has quit IRC01:50
kota_hello02:01
*** kei_yama has quit IRC02:03
hosanaikota: hello02:06
*** kei_yama has joined #openstack-swift02:06
kota_hosanai: \o/02:06
*** hosanai has quit IRC02:08
*** hosanai has joined #openstack-swift02:10
*** ChanServ sets mode: +v hosanai02:10
*** jamielennox is now known as jamielennox|away02:13
*** haomaiwang has joined #openstack-swift02:41
charz_morning02:48
kota_charz_: morning :)02:52
openstackgerritJanie Richling proposed openstack/swift: Encrypt/decrypt etag in container listing  https://review.openstack.org/27549203:00
*** haomaiwang has quit IRC03:01
*** haomaiwang has joined #openstack-swift03:01
*** kei_yama has quit IRC03:03
*** nadeem has joined #openstack-swift03:11
*** tech004 has joined #openstack-swift03:19
tech004hi anyone please explain why getting "proxy: Handoff requested"  on my test node I am using two replicas only Is that an issue I have only two nodes with two proxy and 10 hdd on each03:22
tech004Also I am using chunked upload03:22
*** takashi_ has joined #openstack-swift03:24
*** takashi_ has quit IRC03:24
*** takashi_ has joined #openstack-swift03:25
takashi_good afternoon :-)03:25
*** dmorita has joined #openstack-swift03:30
*** klrmn has quit IRC03:31
*** links has joined #openstack-swift03:33
*** dmorita has quit IRC03:36
kota_tech004: I don't think that it is a reason (2 replica ,10 devices) for the message.03:47
*** kei_yama has joined #openstack-swift03:48
kota_tech004: The message will appear when proxy-server failing to connect (or getting error response from object-servers).03:49
kota_s/failing to connect/failing to connect to objecte-server/03:49
kota_tech004: Did you find any other massage like 5xx, 4xx from object-server in your log?03:50
*** nadeem has quit IRC03:58
*** haomaiwang has quit IRC04:01
*** haomaiwang has joined #openstack-swift04:01
tech004<kota> Let me check for 4xx and 5xx04:01
*** asettle has joined #openstack-swift04:02
*** ppai has joined #openstack-swift04:02
*** Jeffrey4l has joined #openstack-swift04:02
*** SkyRocknRoll has quit IRC04:03
tech004<kota> Yes I am getting 503  Also ChunkWriteTimeout (10s)04:04
*** nadeem has joined #openstack-swift04:10
*** mingdang1 has joined #openstack-swift04:14
kota_tech004: ok, that means something happend during data transfer between proxy-server and object-server...but a bit odd because AFAIK, proxy doesn't try to handoff when failing in transfer.04:15
kota_it only happen when proxy notice the error at the begening of connection.04:16
tech004As I said I am testing the system I am using chunked upload my current script split large files into 256mb chunks Is that a problem large chunk size?04:17
tech004I was using 16mb chunks but it was slowing down entire system04:18
kota_tech004: needn't. the chunk means the bufffer at proxy-server/object-server internal use./04:20
kota_tech004: probably you can see what happend in object-server log.04:21
kota_(if object-server doesn't return response actually)04:21
tech004No thats not what I mean I created a script to automate upload I am watching a folder with i notify and upload new files to swift with my script04:22
tech004so I upload files into chunks04:22
kota_tech004: and create SLO manifest for the chunks, right?04:22
tech004<kota_> Yes04:23
*** SkyRocknRoll has joined #openstack-swift04:23
kota_alright, I meant "the chunk" for "ChunkWriteTimeout" in your log so i don't think your chunk in client side related to *Swift chunk size for internal use*.04:24
tech004I had no issues with 16mb chunk when uploading but large chunk size giving me problem and this error So I am trying to understand if it is a script issue or proxy issue04:24
kota_ah, you mean smaller file was successful but bigger file was not.04:25
kota_got it.04:25
tech004Yeah oh man my english sucks :)04:25
kota_no worries, I am not native too :-)04:26
*** cdelatte has joined #openstack-swift04:26
tech004<kota_> bigger files uploading but I am getting errors and my script hangs a lot so I have to restart my script again every time04:26
kota_circle back to the issue. it sounds like object-server problem when writing files.04:26
tech004Yeah I also think the same some time out values not correct right?04:27
kota_tech004: sorry, what does it means?04:28
*** delattec has joined #openstack-swift04:28
tech004Well I am getting this  in my logs  "ChunkWriteTimeout (10s)"04:29
kota_tech004: probably it's in proxy-server's log.04:29
tech004<kota_> Yes it is04:29
kota_and can you see the log *in object-server*?04:30
*** cdelatte has quit IRC04:31
tech004No i dont see the same parameter in object server logs04:33
kota_ah, ok. we don' t find ChunkWriteTimeout in object-server log anyway. I meant another error happend in object-server at that time.04:36
tech004No errors but i can see 404 status on many HEAD request04:36
kota_e.g. error trace back, connection disconnected, or.04:36
kota_tech004: currious.04:37
tech004<kota_> I am sure there is no network issue coz Its not in internet Its my private network and all other systems are working fine It must be some kind of conf issue on my side :P04:38
kota_the point I'm wondring is the ChunkWfiteTimeout will be raised *after the connection accepted* but no log means the connection wasn't accepted.04:38
kota_tech004: coud you try to use the command "swift-get-nodes <ring file> <account> <container> <object> "?04:40
tech004sure04:40
kota_that shows the direct URL for the object-server via curl04:41
kota_so you can try the reachability with it.04:41
kota_reacchability for object-server.04:41
kota_tech004: let me know your Swift version? I'll try to look at codes at the version later and am going to be slow to return response due to attend another meeting in my office.04:44
tech004kota_ Let me learn and test it I will let you know04:45
tech004Well I have installed swift from ubuntu repo using kilo version04:46
tech004add-apt-repository cloud-archive:kilo04:47
tech004<kota_>Have a look into it I am not sure the command is correct or not http://pastebin.com/qbF0fpsh04:56
*** haomaiwang has quit IRC05:01
*** haomaiwang has joined #openstack-swift05:01
*** mingdang1 has quit IRC05:11
kota_tech004: you need 2 more args for <container> <object>05:11
tech004<kota_> Let me check again05:12
kota_if your script attempt to upload files /v1/PLAY_cldcents/container/object, the args will be "swift-get-nodes -a /etc/swift/object.ring.gz PLAY_cldcents container object "05:12
tech004Ok  let me try05:13
tech004kota_  http://pastebin.com/Vc3RS6gk there you go05:14
kota_tech004: thanks. then.. i noticed a thing looks odd you have a couple of network in your ring, Is it expected?05:16
kota_I am looking at the lines since L3505:16
tech004<kota_> Yes I have two servers with 11 hdd each on two private network05:17
kota_tech004: ok, could you try to L35 and L36 at your proxy-server?05:18
*** nadeem has quit IRC05:18
kota_I think you can get 404s for each line.05:18
*** nadeem has joined #openstack-swift05:19
tech004<kota_> I am sorry I dont get it L35 and L36 part could you explain05:20
kota_try to do the command "curl -I -XHEAD "http://10.112.188.224:9000/lona02/3061/PLAY_cldcents/container/object""05:20
kota_and "curl -I -XHEAD "http://10.90.2.200:9000/sja06/3061/PLAY_cldcents/container/object""05:20
kota_if the curl success to get HTTP response even if it's 404, the reachability from proxy to object-server seems healty.05:21
kota_s/success/succeeds/05:21
*** asettle has quit IRC05:23
tech004Yes Its 40405:24
kota_and each object-server got the 404 log, right?05:24
tech004lol no I tried to do it from one server to other and i get 5xx05:25
tech004HTTP/1.1 507 Insufficient Storage05:26
tech004How is that possible05:26
kota_bingo, that is the reason you can see the Handoff requested.05:26
kota_a couple of reasons...05:26
tech004Please explain :)05:27
kota_no free space to write the object05:27
*** SkyRocknRoll has quit IRC05:27
kota_or the device not mounted.05:27
tech004No way I will list my hdd05:27
kota_tech004: why is no way?05:28
tech004No way because Its writing to hdd :D05:28
tech004and each of them are 3TB05:28
tech004Its crazy server I am testing05:28
kota_I think the reason is the second one.05:29
kota_it's not mounted.05:29
tech004If its mounting issue maybe I can accept but no storage no way man05:29
kota_because HEAD doesn't require the free space but you got still 50705:30
tech004kota_ cool then my next question if it is not why I am able to write into it05:30
kota_tech004: if you use the env only for the test, you can set mount_check=False at object-server.conf to avoid the mounting issue.05:31
tech004kota_  but if there is a mounting issue it should be reported to object log right?05:32
kota_but note that object-server will write the files into default root device instead of the HDD you intended at your ring.05:32
*** delatte has joined #openstack-swift05:32
kota_tech004: right, I'm checking the code right now05:32
tech004kota_ and I have checked it I have unmounted a device or say I have changed perm of a folder /srv/node/device I am getting the device as not mounted05:34
*** asettle has joined #openstack-swift05:35
*** delattec has quit IRC05:36
kota_tech004: supprisingly, it looks like not logged for now, https://github.com/openstack/swift/blob/master/swift/obj/diskfile.py#L1067-L106805:36
tech004kota_ Can you give  me a solution05:38
kota_tech004: what I suggest is to mount the actuall hdd device to the dir written in your ring.05:39
kota_it's healty way for swift.05:39
*** SkyRocknRoll has joined #openstack-swift05:39
tech004I think I my ring exactly like that05:40
tech004and now I am getting 404 from both server05:40
*** asettle has quit IRC05:40
tech004hmm I dont get it05:40
tech004/dev/sdl1 /srv/node/sja11 xfs noatime,nodiratime,nobarrier,logbufs=8 0 005:41
kota_when you get 404 from both servers, maybe you don't see Handoff requested anymore while uploading the files.05:41
tech004and I have created my ring with sja** I think you meant that right05:42
kota_yup05:42
tech004cool so my mounting is in proper way05:42
tech004Let me try with mount check false05:43
kota_ /dev/sdl1 is actuall drive path and /srv/node/sja11 is the mount point which Swift can see.05:43
tech004Yeah05:43
kota_the issue resolved?05:43
kota_(i mean the issue you saw Handoff Requested)05:44
tech004no I have just reastrted my worker Its running node.js script hmm I think I must add it to cron so it will restart automatically in every 15 minutes or  so :P05:44
kota_hehe :/05:45
tech004kota_  we are back to square one I think Its chunk size or my script :D05:46
tech004kota_  anyway I will check time outs variables in proxy conf and try again05:49
kota_tech004: ok, if you notice something else, ping me to know what happens in your Swift.05:50
tech004kota_ sure I have to check all logs including syslog to see what is going on  and I will let you know05:51
*** haomaiwang has quit IRC06:01
*** haomaiwang has joined #openstack-swift06:01
*** trifon has joined #openstack-swift06:13
*** sanchitmalhotra has joined #openstack-swift06:40
*** pcaruana has joined #openstack-swift06:49
*** sheel has joined #openstack-swift06:54
*** haomaiwang has quit IRC07:01
*** haomaiwang has joined #openstack-swift07:01
*** huayi has joined #openstack-swift07:05
*** hosanai has quit IRC07:09
*** huayi is now known as chenhuayi07:11
*** asettle has joined #openstack-swift07:24
*** huayi_ has joined #openstack-swift07:25
*** chenhuayi has quit IRC07:27
*** asettle has quit IRC07:29
*** huayi_ has quit IRC07:31
*** haomaiwang has quit IRC08:01
*** haomaiwa_ has joined #openstack-swift08:01
*** mingdang1 has joined #openstack-swift08:09
*** trifon has quit IRC08:09
*** baojg has quit IRC08:11
*** sanchitmalhotra has quit IRC08:14
*** baojg has joined #openstack-swift08:14
*** nadeem has quit IRC08:22
*** trifon has joined #openstack-swift08:33
*** tech004 has quit IRC08:35
*** mingdang1 has quit IRC08:37
*** jmccarthy has quit IRC08:37
*** jmccarthy has joined #openstack-swift08:38
*** haomaiwa_ has quit IRC09:01
*** haomaiwang has joined #openstack-swift09:01
*** takashi_ has quit IRC09:03
*** ppai has quit IRC09:14
*** ppai has joined #openstack-swift09:22
*** karthi has joined #openstack-swift09:39
*** pcaruana has quit IRC09:42
*** kei_yama has quit IRC09:44
*** pcaruana has joined #openstack-swift09:55
*** haomaiwang has quit IRC10:01
*** 77CAAJ17L has joined #openstack-swift10:01
*** 77CAAJ17L has quit IRC10:19
*** haomaiwa_ has joined #openstack-swift10:21
*** silor has joined #openstack-swift10:31
*** SkyRocknRoll has quit IRC10:40
*** mingdang1 has joined #openstack-swift10:46
*** baojg has quit IRC10:56
*** asettle has joined #openstack-swift10:56
*** divya___ has joined #openstack-swift10:56
divya___hi10:56
divya___i am installting mitaka10:57
divya___devstak and fails in openstack object store account set --property Temp-URL-Key=password 2016-03-28 10:23:53.380 | Not Found (HTTP 404)10:57
divya___can anybody help10:57
*** haomaiwa_ has quit IRC11:01
*** haomaiwang has joined #openstack-swift11:01
*** pcaruana has quit IRC11:09
*** divya___ has quit IRC11:14
*** pcaruana has joined #openstack-swift11:23
*** silor1 has joined #openstack-swift11:28
*** silor has quit IRC11:30
*** silor1 is now known as silor11:30
*** flaper87 has quit IRC11:48
*** flaper87 has joined #openstack-swift11:51
*** km has quit IRC11:58
*** haomaiwang has quit IRC12:01
*** haomaiwang has joined #openstack-swift12:01
*** asettle has quit IRC12:05
*** ppai has quit IRC12:10
*** janonymous has joined #openstack-swift12:19
*** pcaruana has quit IRC12:39
*** pcaruana has joined #openstack-swift12:52
*** haomaiwang has quit IRC13:01
*** haomaiwang has joined #openstack-swift13:01
*** klamath has joined #openstack-swift13:13
*** mingdang1 has quit IRC13:19
*** vinsh has quit IRC13:28
tdasilvagood morning13:29
*** vinsh has joined #openstack-swift13:29
*** vinsh_ has joined #openstack-swift13:30
*** vinsh has quit IRC13:30
*** StraubTW has joined #openstack-swift13:32
*** links has quit IRC13:36
*** openstackgerrit has quit IRC13:48
*** openstackgerrit has joined #openstack-swift13:48
*** ametts has joined #openstack-swift13:59
*** haomaiwang has quit IRC14:01
*** sgundur has joined #openstack-swift14:01
*** haomaiwang has joined #openstack-swift14:01
janonymoustdasilva: o/14:03
*** haomaiwang has quit IRC14:06
*** karthi has quit IRC14:20
mahaticfyi, I'm on vacation this week and the next, until 8th of April14:28
*** _JZ_ has joined #openstack-swift14:29
*** _JZ_ has quit IRC14:29
*** _JZ_ has joined #openstack-swift14:30
*** StraubTW_ has joined #openstack-swift14:37
*** StraubTW has quit IRC14:38
*** NM has joined #openstack-swift14:38
tdasilvamahatic: enjoy!14:39
*** haomaiwa_ has joined #openstack-swift14:41
*** zaitcev has joined #openstack-swift14:45
*** ChanServ sets mode: +v zaitcev14:45
klamathCan someone provide some pointers on increasing replication speed on very small object files?  what are some values to start adjusting? object-server rsync workers, object-replicator concurrency?14:52
*** hurricanerix has quit IRC14:58
*** Guest95722 has quit IRC14:58
*** links has joined #openstack-swift14:58
*** hurricanerix has joined #openstack-swift14:58
*** links has quit IRC14:58
*** Guest88568 has joined #openstack-swift14:59
*** haomaiwa_ has quit IRC15:01
*** haomaiwa_ has joined #openstack-swift15:01
*** StraubTW_ has quit IRC15:05
*** StraubTW has joined #openstack-swift15:06
*** garthb has joined #openstack-swift15:06
*** esker has quit IRC15:12
*** esker has joined #openstack-swift15:13
-openstackstatus- NOTICE: Gerrit is restarting on review.openstack.org in an attempt to address an issue reading an object from the ec2-api repository15:22
jrichlimahatic: have fun!15:22
*** dmellado is now known as dmellado|off15:24
*** klrmn has joined #openstack-swift15:25
*** StraubTW_ has joined #openstack-swift15:31
*** StraubTW has quit IRC15:33
*** arch-nemesis has joined #openstack-swift15:37
notmynamegood morning15:46
pdardeaugood morning15:48
notmynamecschwede: as far as I can tell, not much happened on the rolling upgrade test mechanics since you looked at it. any updates there?15:54
*** garthb_ has joined #openstack-swift15:56
*** garthb has quit IRC15:57
*** haomaiwa_ has quit IRC16:01
*** haomaiwang has joined #openstack-swift16:01
*** garthb_ has quit IRC16:06
*** garthb has joined #openstack-swift16:07
*** StraubTW_ has quit IRC16:12
*** lyrrad has joined #openstack-swift16:19
*** mingdang1 has joined #openstack-swift16:20
*** mingdang1 has quit IRC16:25
*** klrmn has quit IRC16:25
claygheyoh!16:26
openstackgerritAndreas Jaeger proposed openstack/swift: List system dependencies for running common tests  https://review.openstack.org/29831316:27
*** gyee has joined #openstack-swift16:28
*** StraubTW has joined #openstack-swift16:30
*** david-lyle_ has joined #openstack-swift16:44
*** david-lyle has quit IRC16:44
*** david-lyle has joined #openstack-swift16:48
*** david-lyle_ has quit IRC16:49
*** dmorita has joined #openstack-swift16:53
openstackgerritAndreas Jaeger proposed openstack/swift: List system dependencies for running common tests  https://review.openstack.org/29831316:57
*** haomaiwang has quit IRC17:01
*** haomaiwang has joined #openstack-swift17:01
*** StraubTW has quit IRC17:06
*** nadeem has joined #openstack-swift17:15
*** nadeem has quit IRC17:15
*** nadeem has joined #openstack-swift17:16
*** david-lyle has quit IRC17:24
*** david-lyle has joined #openstack-swift17:25
*** klrmn has joined #openstack-swift17:31
*** BigWillie has joined #openstack-swift17:36
*** esker has quit IRC17:40
*** tongli has joined #openstack-swift17:47
jrichliclayg: I applied your feedback for transient-sysmeta helper functions in patch 29806317:50
patchbotjrichli: https://review.openstack.org/#/c/298063/ - swift (feature/crypto) - Resolve todos relying on real footers and transien...17:50
claygjrichli: cool, there's still a function that returns a string literal - is it used?  I see in the module they just grab the global constant17:52
jrichliclayg: oops, i meant to change that.  i had to bring back that method because of how gatekeeper uses the prefixes17:54
jrichlibut i meant to switch out the literal17:54
claygwell, i mean now it's a function that returns a constant - but that seems about the same - you don't need to call a function to get at a global17:55
claygjrichli: i'm sure it's fine - if you're happy with don't worry about it17:55
*** haomaiwang has quit IRC18:01
*** haomaiwang has joined #openstack-swift18:01
*** Jeffrey4l has quit IRC18:04
*** david-lyle_ has joined #openstack-swift18:12
*** david-lyle has quit IRC18:14
openstackgerritJanie Richling proposed openstack/swift: Resolve todos relying on real footers and transient-sysmeta cleanup  https://review.openstack.org/29806318:14
*** pcaruana has quit IRC18:16
*** pcaruana has joined #openstack-swift18:30
*** silor has quit IRC18:38
*** pauloewerton has joined #openstack-swift18:47
*** haomaiwang has quit IRC19:01
*** haomaiwang has joined #openstack-swift19:01
openstackgerritOpenStack Proposal Bot proposed openstack/python-swiftclient: Updated from global requirements  https://review.openstack.org/8925019:32
openstackgerritOpenStack Proposal Bot proposed openstack/swift: Updated from global requirements  https://review.openstack.org/8873619:32
*** esker has joined #openstack-swift19:35
*** BigWillie has quit IRC19:50
jrichlitimburke: thanks for the +2 on patch 268723 ... just needs one more!19:52
patchbotjrichli: https://review.openstack.org/#/c/268723/ - swift - swift-[account|container]-info when disk is full19:52
*** sheel has quit IRC19:57
*** esker has quit IRC19:59
*** haomaiwang has quit IRC20:01
*** haomaiwang has joined #openstack-swift20:01
*** BigWillie has joined #openstack-swift20:17
*** asettle has joined #openstack-swift20:40
*** BigWillie has quit IRC20:52
*** pauloewerton has quit IRC21:00
*** haomaiwang has quit IRC21:01
*** haomaiwang has joined #openstack-swift21:01
*** rickyrem has joined #openstack-swift21:20
*** NM has quit IRC21:35
*** trifon has quit IRC21:40
*** _JZ_ has quit IRC21:55
*** haomaiwang has quit IRC22:01
*** haomaiwa_ has joined #openstack-swift22:01
*** klamath has quit IRC22:40
*** bill_az has joined #openstack-swift22:47
*** nadeem has quit IRC22:48
*** esker has joined #openstack-swift22:49
*** tongli has quit IRC22:56
*** david-lyle_ is now known as david-lyle22:57
*** haomaiwa_ has quit IRC23:01
*** haomaiwa_ has joined #openstack-swift23:01
*** esker has quit IRC23:02
*** esker has joined #openstack-swift23:03
*** esker has quit IRC23:10
*** esker has joined #openstack-swift23:16
*** esker has quit IRC23:20
*** ametts has quit IRC23:22
*** asettle has quit IRC23:24
*** kei_yama has joined #openstack-swift23:30
*** shakamunyi has joined #openstack-swift23:30
*** barra204 has quit IRC23:32
*** mingdang1 has joined #openstack-swift23:33
*** esker has joined #openstack-swift23:36
*** km has joined #openstack-swift23:38
*** diogogmt has joined #openstack-swift23:48
*** hosanai has joined #openstack-swift23:49
*** ChanServ sets mode: +v hosanai23:49
*** mingdang1 has quit IRC23:51
hosanaigood morning!23:55

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!