Tuesday, 2014-12-16

*** X019 has joined #openstack-swift00:08
*** Masahiro has joined #openstack-swift00:12
*** lcurtis has quit IRC00:14
*** fandi has joined #openstack-swift00:15
*** Masahiro has quit IRC00:17
*** miurahr has joined #openstack-swift00:17
*** daddyjoseph97 has quit IRC00:22
*** fandi has quit IRC00:28
*** annegent_ has joined #openstack-swift00:29
*** ho has quit IRC00:32
*** mvenesio_ has quit IRC00:34
*** annegent_ has quit IRC00:34
*** occup4nt is now known as occupant00:35
*** david-lyle is now known as david-lyle_afk00:35
*** fandi has joined #openstack-swift00:35
*** dmorita has joined #openstack-swift00:35
*** miurahr has quit IRC00:37
*** ho has joined #openstack-swift00:38
*** oomichi has joined #openstack-swift00:41
*** andrews has joined #openstack-swift00:48
*** Masahiro has joined #openstack-swift00:51
andrews@here Don't you know, guys, if a possibility to delete a container with objects is a new feature in swift? I use python-swiftclient==2.3.1. All docs say the I can delete only empty containers.00:57
*** addnull has joined #openstack-swift00:57
swifterdarrellandrews: python-swiftclient just lists the objects in the container, deletes them, and then deletes the container, all on your behalf; no change was made to the Swift API re: deleting containers00:58
*** panbalag has quit IRC01:05
andrewsswifterdarrell: before, when i tried to delete a container with children I got 409 error and it didn't allow me to do this until I kill all the objects. it is a bit confusing as i expect to get 409 again according to docs. Don't you know it is documented somewhere?01:05
*** swifterdarrell has left #openstack-swift01:05
*** swifterdarrell has joined #openstack-swift01:05
*** ChanServ sets mode: +v swifterdarrell01:05
swifterdarrellandrews: this was with the python-swiftclient vs. the "swift" CLI program?  I dunno, maybe that did change...01:06
swifterdarrellandrews: but the Swift API itself has not changed in this respect, though01:06
swifterdarrellandrews: pretty sure Swift doesn't let you DELETE containers that still have objects.01:06
*** lpabon has quit IRC01:09
andrewsswifterdarrell: I have noticed this working  through the CLI.   and I was pretty surprised being able to delete container with objects.   here the code: https://github.com/openstack/python-swiftclient/blob/202bbab2d06b9ef751fba306effd09f3bae8f383/swiftclient/shell.py#L11401:10
andrewsswifterdarrell: it deleted all objects first and then container itself with no notification that container has objects, no expected 409 errors01:13
*** shri has quit IRC01:13
swifterdarrellandrews: my understanding is that the "swift" CLI has had that behavior for as long as I can remember01:16
swifterdarrellandrews: I think there was a /bug/ at one point where there was a race condition between the last few object deletes and the DELETE of the Container, preventing the "swift" CLI from accomplishing the user's goal (of deleting the container and all objects within it)01:16
andrewsswifterdarrell: under "that behavior" you mean possibility to delete container with objects?01:22
swifterdarrellandrews: I believe "swift delete container_name" has always semantically meant "delete all objects in container_name and then delete container_name itself", and if that was broken and only deleted some objects before saying something about "409", that was a bug that my memory vaguely says we had for a while01:23
*** X019 has quit IRC01:27
andrewsswifterdarrell: ok, if I try to delete a container with objects through the API call, can I expect error message?01:28
*** annegent_ has joined #openstack-swift01:30
*** addnull has quit IRC01:34
*** annegent_ has quit IRC01:35
*** X019 has joined #openstack-swift01:42
*** panbalag has joined #openstack-swift01:47
swifterdarrellandrews: definitely01:47
*** ho has quit IRC01:49
*** ho has joined #openstack-swift01:49
*** haomaiwang has joined #openstack-swift01:57
*** panbalag has quit IRC01:57
andrewsswifterdarrell: thanks a lot for explanations, I just checked it, indeed, API call gives me 409 error, CLI allows me to delete container with objects - it is just different behavior for API and CLI ways. Thanks!02:07
*** gyee has quit IRC02:09
*** lcurtis has joined #openstack-swift02:13
*** oomichi has quit IRC02:16
*** annegent_ has joined #openstack-swift02:29
*** andrews has quit IRC02:32
*** annegent_ has quit IRC02:34
*** rdaly2 has quit IRC02:48
*** addnull has joined #openstack-swift02:51
openstackgerritHisashi Osanai proposed openstack/swift: Fix the GET's response code when there is a missing segment in LO  https://review.openstack.org/13625802:51
*** masonhsiung has joined #openstack-swift02:54
*** addnull has quit IRC02:56
*** lcurtis has quit IRC03:01
*** Masahiro has quit IRC03:02
*** Masahiro has joined #openstack-swift03:02
*** annegent_ has joined #openstack-swift03:03
*** annegent_ has quit IRC03:05
*** lcurtis has joined #openstack-swift03:14
*** fandi has quit IRC03:25
*** lcurtis has quit IRC03:41
*** rdaly2 has joined #openstack-swift03:48
*** rdaly2 has quit IRC03:52
*** oomichi has joined #openstack-swift03:53
*** oomichi has quit IRC03:54
openstackgerritYuan Zhou proposed openstack/swift: Add node/pair index patch back into feature/EC  https://review.openstack.org/13406503:57
*** ho has quit IRC04:12
*** ho has joined #openstack-swift04:18
*** ppai has joined #openstack-swift04:21
*** haomaiwang has quit IRC04:22
*** haomaiwang has joined #openstack-swift04:23
*** miurahr has joined #openstack-swift04:27
*** miurahr has left #openstack-swift04:27
*** vishy has quit IRC04:35
*** sileht has quit IRC04:36
*** zacksh has quit IRC04:36
*** zacksh has joined #openstack-swift04:38
*** vishy has joined #openstack-swift04:39
*** sileht has joined #openstack-swift04:40
*** miurahr has joined #openstack-swift04:42
*** addnull has joined #openstack-swift04:47
*** rdaly2 has joined #openstack-swift04:50
*** rdaly2 has quit IRC04:54
*** miurahr has quit IRC04:56
*** zaitcev has quit IRC05:05
*** addnull has quit IRC05:13
*** addnull has joined #openstack-swift05:20
*** masonhsi_ has joined #openstack-swift05:24
*** masonhsiung has quit IRC05:27
*** addnull has quit IRC05:31
*** masonhsiung has joined #openstack-swift05:31
*** masonhsi_ has quit IRC05:35
*** oomichi has joined #openstack-swift05:38
*** SkyRocknRoll has joined #openstack-swift05:47
*** rdaly2 has joined #openstack-swift05:50
*** rdaly2 has quit IRC05:55
*** oomichi has quit IRC06:02
*** masonhsiung has quit IRC06:05
*** masonhsiung has joined #openstack-swift06:05
*** nshaikh has joined #openstack-swift06:13
*** addnull has joined #openstack-swift06:15
*** rmcall has joined #openstack-swift06:19
*** rmcall has quit IRC06:24
*** rmcall has joined #openstack-swift06:25
mattoliverauI'm, calling it a day, night swift land06:32
*** kopparam has joined #openstack-swift06:42
*** kopparam has quit IRC07:00
*** rmcall has quit IRC07:01
*** kopparam has joined #openstack-swift07:03
*** kopparam has quit IRC07:03
*** ttrumm_ has joined #openstack-swift07:08
*** nshaikh has quit IRC07:10
*** pcaruana has joined #openstack-swift07:19
*** nshaikh has joined #openstack-swift07:20
*** jyoti-ranjan has joined #openstack-swift07:22
*** rdaly2 has joined #openstack-swift07:52
*** rdaly2 has quit IRC07:56
*** addnull has quit IRC08:03
*** kopparam has joined #openstack-swift08:04
homattoliverau: what is "night swift land"?08:05
mattoliverauho: just saying good evening/good night to the swift channel :)08:06
homattoliverau: thanks!08:07
*** rledisez has joined #openstack-swift08:09
ondergetekendeNight swift land sounds like some realm in a fantasy series... ;-)08:10
*** kopparam has quit IRC08:11
*** nellysmitt has joined #openstack-swift08:16
mattoliverauLol, yes yes it does :)08:29
*** geaaru has joined #openstack-swift08:36
*** Krast has joined #openstack-swift08:42
*** k4n0 has joined #openstack-swift08:47
*** kopparam has joined #openstack-swift08:50
*** ppai has quit IRC08:54
*** jordanP has joined #openstack-swift08:56
*** kopparam has quit IRC08:57
*** kopparam has joined #openstack-swift08:59
*** fandi has joined #openstack-swift09:03
*** kopparam has quit IRC09:07
*** ppai has joined #openstack-swift09:08
*** kopparam has joined #openstack-swift09:09
*** kopparam_ has joined #openstack-swift09:28
*** kopparam has quit IRC09:29
*** addnull has joined #openstack-swift09:33
*** sandywalsh has joined #openstack-swift09:34
*** sandywalsh_ has quit IRC09:36
*** acoles_away is now known as acoles09:37
*** addnull has quit IRC09:42
*** nellysmitt has quit IRC09:51
*** rdaly2 has joined #openstack-swift09:53
*** rdaly2 has quit IRC09:58
*** kopparam_ has quit IRC10:00
*** jasondotstar has joined #openstack-swift10:05
*** ekarlso- has quit IRC10:07
*** ekarlso- has joined #openstack-swift10:07
*** nellysmitt has joined #openstack-swift10:25
*** fandi has quit IRC10:27
*** addnull has joined #openstack-swift10:50
*** kopparam has joined #openstack-swift11:00
openstackgerritKoert van der Veer proposed openstack/swift: Add option to insert a default reseller prefix  https://review.openstack.org/14206711:06
*** haomaiwang has quit IRC11:14
*** kopparam has quit IRC11:16
*** kopparam has joined #openstack-swift11:16
*** dmorita has quit IRC11:17
*** Masahiro has quit IRC11:23
*** mahatic has joined #openstack-swift11:30
*** masonhsiung has quit IRC11:33
*** Masahiro has joined #openstack-swift11:39
*** infotection has quit IRC11:53
*** rdaly2 has joined #openstack-swift11:54
openstackgerritKoert van der Veer proposed openstack/swift: Add option to insert a default reseller prefix  https://review.openstack.org/14206711:57
*** infotection has joined #openstack-swift11:59
*** rdaly2 has quit IRC11:59
*** kopparam has quit IRC12:00
*** jasondotstar is now known as jasondotstar|afk12:01
*** ppai has quit IRC12:03
*** lpabon has joined #openstack-swift12:05
*** kopparam has joined #openstack-swift12:05
*** Masahiro has quit IRC12:07
*** infotection has quit IRC12:09
*** infotection has joined #openstack-swift12:11
*** otoolee has quit IRC12:14
*** sileht has quit IRC12:14
*** ipolyzos has quit IRC12:14
*** kopparam has quit IRC12:15
*** sileht has joined #openstack-swift12:16
*** ipolyzos has joined #openstack-swift12:16
*** EmilienM is now known as EmilienM|afk12:17
*** ppai has joined #openstack-swift12:18
*** nexusz99 has quit IRC12:18
*** ppai has quit IRC12:18
*** ppai has joined #openstack-swift12:18
*** nottrobin__ has joined #openstack-swift12:19
*** nottrobin_ has quit IRC12:20
*** nottrobin__ is now known as nottrobin_12:20
*** nottrobin_ is now known as nottrobin12:20
*** otoolee has joined #openstack-swift12:21
*** acoles has quit IRC12:22
*** ondergetekende has quit IRC12:22
*** kopparam has joined #openstack-swift12:25
*** zackmdavis_ has joined #openstack-swift12:26
*** ppai has quit IRC12:31
*** ho has quit IRC12:31
*** ujjain has quit IRC12:31
*** zackmdavis has quit IRC12:31
openstackgerritDaniel Wakefield proposed openstack/python-swiftclient: Verify MD5 of uploaded objects.  https://review.openstack.org/12925412:32
*** ujjain has joined #openstack-swift12:33
*** miurahr has joined #openstack-swift12:33
*** lcurtis has joined #openstack-swift12:33
*** ondergetekende has joined #openstack-swift12:34
*** acoles_away has joined #openstack-swift12:34
*** acoles_away is now known as acoles12:35
*** ChanServ sets mode: +v acoles12:35
*** ppai has joined #openstack-swift12:36
*** miurahr has quit IRC12:38
*** addnull has quit IRC12:40
*** lcurtis has quit IRC12:44
*** panbalag has joined #openstack-swift12:50
*** miurahr has joined #openstack-swift12:50
*** panbalag has quit IRC12:51
*** miurahr has quit IRC12:51
*** tellesnobrega has quit IRC12:56
*** linuxgeek_ has joined #openstack-swift12:56
linuxgeek_hello, with swift as the glance back end it takes 30 minutes for a 30gb file to be uploaded to glance. is there any changes which can be made to better this?12:58
linuxgeek_in the swift or glance conf files12:58
*** rdaly2 has joined #openstack-swift12:58
*** Masahiro has joined #openstack-swift13:07
openstackgerritYuan Zhou proposed openstack/swift: Add node/pair index patch back into feature/EC  https://review.openstack.org/13406513:08
rledisezls -lh13:08
*** tellesnobrega has joined #openstack-swift13:08
rledisezoups, sorry :)13:09
*** Masahiro has quit IRC13:12
*** kopparam has quit IRC13:16
*** kopparam has joined #openstack-swift13:19
mahatichi, I'm looking for tests for OPTIONS verb written in proxy. I can't locate it in the code. Can someone point me to it?13:19
*** kopparam has quit IRC13:20
openstackgerritDoug Hellmann proposed openstack/swift: Remove unneeded condition  https://review.openstack.org/14181813:26
*** EmilienM|afk is now known as EmilienM13:28
*** aix has joined #openstack-swift13:31
*** ppai has quit IRC13:41
*** ppai has joined #openstack-swift13:55
*** ppai has quit IRC14:02
*** nshaikh has quit IRC14:11
*** mvenesio has joined #openstack-swift14:17
openstackgerritDaniel Wakefield proposed openstack/python-swiftclient: Fix misplaced check for None in SwiftUploadObject.  https://review.openstack.org/13310714:18
*** jwang__ has joined #openstack-swift14:20
*** sfineberg_ has joined #openstack-swift14:22
*** mtreinish_ has joined #openstack-swift14:22
*** acorwin_ has joined #openstack-swift14:23
*** otherjon_ has joined #openstack-swift14:24
*** rdaly2 has quit IRC14:24
*** jeblair_ has joined #openstack-swift14:25
*** HenryG_ has joined #openstack-swift14:28
*** portante_ has joined #openstack-swift14:30
*** k4n0 has quit IRC14:30
*** SkyRocknRoll has quit IRC14:30
*** HenryG has quit IRC14:30
*** mtreinish has quit IRC14:30
*** jeblair has quit IRC14:30
*** jwang_ has quit IRC14:30
*** sfineberg has quit IRC14:30
*** otherjon has quit IRC14:30
*** acorwin has quit IRC14:30
*** IRTermite has quit IRC14:30
*** portante has quit IRC14:30
*** otherjon_ is now known as otherjon14:30
*** mtreinish_ is now known as mtreinish14:30
*** k4n0 has joined #openstack-swift14:31
*** aix has quit IRC14:31
*** IRTermite has joined #openstack-swift14:31
*** aix has joined #openstack-swift14:32
*** SkyRocknRoll has joined #openstack-swift14:33
openstackgerritMerged openstack/swift: Adjust MAX_FILE_SIZE during test on 32 bit systems  https://review.openstack.org/12703014:37
*** Masahiro has joined #openstack-swift14:40
*** tellesnobrega has quit IRC14:40
*** tellesnobrega has joined #openstack-swift14:40
*** lihkin has joined #openstack-swift14:44
*** Masahiro has quit IRC14:44
*** mvenesio has quit IRC14:49
*** ttrumm_ has quit IRC14:52
*** X019 has quit IRC14:55
*** dmsimard_away is now known as dmsimard15:03
*** lihkin has quit IRC15:05
*** tdasilva has joined #openstack-swift15:06
*** lihkin has joined #openstack-swift15:07
*** X019 has joined #openstack-swift15:08
*** jasondotstar|afk is now known as jasondotstar15:15
*** jyoti-ranjan has quit IRC15:17
*** SkyRocknRoll has quit IRC15:18
*** HenryG_ has quit IRC15:25
*** HenryG_ has joined #openstack-swift15:25
*** portante_ has quit IRC15:25
*** portante_ has joined #openstack-swift15:25
*** k4n0 has quit IRC15:25
*** k4n0 has joined #openstack-swift15:25
*** aix has quit IRC15:25
*** aix has joined #openstack-swift15:25
*** rdaly2 has joined #openstack-swift15:32
*** erlon has joined #openstack-swift15:32
*** pcaruana has quit IRC15:42
*** daddyjoseph97 has joined #openstack-swift15:44
*** miurahr has joined #openstack-swift15:48
*** rdaly2 has quit IRC15:51
*** rmcall has joined #openstack-swift16:08
imkarrerHey guys.  I have a question about deploying multi-region swift.  This question reguards the memcached servers.  Should the list of memcached servers in proxy-server.conf and object-expirer.conf be Region specific? In other words, should Region1 only have memcached servers listed which reside in region1?16:19
*** daddyjoseph97 has quit IRC16:19
remix_tjimkarrer: in my plan there is one memcached on each region16:24
*** jhfeng has joined #openstack-swift16:24
remix_tjimkarrer: in case of client connecting to proxy on other region will need to reauth16:24
openstackgerritpaul luse proposed openstack/swift-specs: Updates to the reconstructor section  https://review.openstack.org/14214616:25
imkarrerremix_tj: Thanks, that makes since.  I also think that since swift is eventually consistent that region specific memcached makes since.  But the auth seals the deal for me.  Thanks again.16:26
*** kragniz is now known as kragwhale16:26
*** lihkin has quit IRC16:27
*** Masahiro has joined #openstack-swift16:29
*** SkyRocknRoll has joined #openstack-swift16:29
*** Masahiro has quit IRC16:34
*** EmilienM is now known as EmilienM|afk16:34
*** rmcall has quit IRC16:42
*** david-lyle_afk is now known as david-lyle16:44
*** ajiang has joined #openstack-swift16:48
*** kani has joined #openstack-swift16:48
kaniWhile uploading image to glance which has swift backend , the image is going to killed state - swift log shows the error as ChunkWriteTimeOut(10s).16:51
kaniCan anyone tell what is this error and how to fix it?16:51
*** ajiang38740 has joined #openstack-swift16:58
*** zackmdavis_ is now known as zackmdavis17:00
*** lpabon has quit IRC17:01
*** X019 has left #openstack-swift17:01
*** jhfeng has quit IRC17:04
kaniWhile uploading image to glance which has swift backend , the image is going to killed state - swift log shows the error as ChunkWriteTimeOut(10s). Can anyone tell what is this error and how to fix it?17:06
*** nellysmitt has quit IRC17:10
notmynamegood morning17:13
mahaticgood morning!17:14
*** rmcall has joined #openstack-swift17:14
pelusemorning17:16
acolesmorning17:19
*** mvenesio has joined #openstack-swift17:19
mahaticnotmyname, I'm unable to locate tests for OPTIONS verb in proxy, can you point me to it?17:22
notmynameyup17:22
notmynamelet me find them17:22
mahaticokay17:22
torgomaticmahatic: if nothing else, comment out the OPTIONS method, run tests, and see which ones fail17:23
notmynameheh17:23
*** dmsimard is now known as dmsimard_away17:23
mahatictorgomatic, oh. that's a way too :D thanks!17:23
torgomaticI do that all the time when I can't find tests for something.17:23
notmynamemahatic: test/unit/proxy/test_server.py:485517:23
notmynamemahatic: there are a few test methods in that file17:24
mahatictorgomatic, your timezone is PST too?17:24
kaninotmyname: While uploading image to glance which has swift backend , the image is going to killed state - swift log shows the error as ChunkWriteTimeOut(10s). Could you help me understand what is this error and how to fix it?17:24
torgomaticmahatic: yes, I'm near San Francisco17:24
mahaticokay17:24
*** dmsimard_away is now known as dmsimard17:25
notmynamemahatic: when he's not working from home, he sits about 3 feet in front of me :-)17:25
mahaticnotmyname, okay. I was digging through test_base.py all this time17:25
mahaticnotmyname, ah yes. He works for swiftstack too i believe (I don't know how i remember, probably from a discussion here :D)17:26
*** rledisez has quit IRC17:26
notmynamekani: that error is when the proxy server times out taking to an object server. there are a few possible causes17:29
torgomatickani: sounds like glance isn't sending data to swift in a timely fashion17:29
torgomaticnotmyname: it's also in the proxy, FWIW; happens when the client fails to send data for $node_timeout seconds17:29
notmynamekani: importantly, though, what is "killed state". and what message are you getting from swift? ie what response code? swift works around a lot of that17:30
kani503 Service Unavailable . this is in glance-api.log17:30
notmynamekani: yup. both ways17:31
notmynamekani: there's a few next steps17:31
notmynamekani: first, get the transaction id for that failed request. then grep the swift logs for that id so you have all the logs relating to that request17:32
notmynamekani: you'll see the timeout, but maybe some other stuff17:32
*** mkerrin has quit IRC17:32
kanithis does not happen always17:32
notmynamekani: what does the full error message say?17:33
kaniWhile trying to upload 4 images in parallel, I saw this error17:33
notmynamekani: that will tell you if it's a client read issue or a object server write issue17:33
*** kbee has joined #openstack-swift17:33
kaniGot the error like this: proxy-server ERROR with Object server 127.0.0.1:6010/sda8 re: Trying to write to /v1/AUTH_8b0342f9b71f4f3db7272452ef31fa1c/myfiles/testCD.iso: ChunkWriteTimeout (10s)17:34
notmynamekani: ok. so the server listening on port 6010 is timing out. could be over-busy. maybe the drive is having issues (contention or failures).17:36
notmynamekani: interesting that you're only seeing one timeout error and getting a 503. are you using 3 replicas?17:36
kaniyeah... 3 replicas17:37
notmynamekani: can you find all the logs with that trans id and pastebin them?17:38
*** NM has joined #openstack-swift17:39
kaninotmyname: Yes.. I ll do paste17:40
*** SkyRocknRoll has quit IRC17:40
*** kragwhale is now known as kragniz17:43
kbeeHello notmyname , can you pls give your opinion on https://review.openstack.org/#/c/140972/17:44
notmynamekbee: if you have multiple instances running on the same box and need to demux the logs, you can do that with different syslog facilities, right? logs still go to syslog, but you can configure syslog to put the logs where you need them.17:50
notmynamekbee: I'm not saying you don't have a valid use case (far from it!). but I think what you're looking for is already possible without adding new config options17:51
*** lcurtis has joined #openstack-swift17:53
kbeenotmyname: the angle is for isolated drive-audit runs, its easier to mine 3-4 lines of data than go through heap of syslog. this sounds more prudent, and we can achieve with minimal code..17:54
kbeenotmyname: and most important usecase is : if there are simultaneous runs of swift drive audit, the first run logs to syslog and unmounts and second one does nothing since is already unmounted, mining through latest syslog message will be incorrect17:58
kbeenotmyname: one should correctly find out the error state for a particular run of drive-audit. and that was the intention17:59
notmynamekbee: so your proposal is to log the first one to syslog and the second one to the console. then you look in the respective place for the log messages associated with whatever one you're checking. right?18:00
notmynamekbee: and you'd tell the second run to log to the console with a config option18:00
kbeenotmyname: The proposal is to just add a console handler for logger. so when logging is done, it logs to syslog as well as console.18:01
kbeenotmyname: if log_to_console is turned on(default: off), it logs to both syslog and console, else logs only to syslog as all these days18:02
*** rmcall has quit IRC18:02
*** jordanP has quit IRC18:03
*** jeblair_ is now known as jeblair18:04
notmynamekbee: and what do the configs look like? both log to console? or one logs to syslog and the other logs to syslog+console?18:04
*** rmcall has joined #openstack-swift18:05
kbeenotmyname: just one extra config option - log_to_console (default: False). if set to true, it logs to console.18:05
kbeenotmyname: By default, anyways it always gets logged to syslog18:06
*** k4n0 has quit IRC18:06
*** rmcall has quit IRC18:06
kbeenotmyname: no change in that functionality. this is just extra ability to log to console depending on config18:06
notmynamekbee: right. I get that. so how are you running it? both log to console+syslog? or one logs to console and one logs to console+syslog?18:06
*** fandi has joined #openstack-swift18:06
*** rmcall has joined #openstack-swift18:08
kbeenotmyname: sorry. i'm not getting it -> 'both log', do you mean logs or something else ?18:08
*** shri has joined #openstack-swift18:08
notmynamekbee: how are you running each instance of swift-drive-audit? do they have the same config file or different config files?18:08
kbeesame config file18:09
kbeenotmyname: same config file18:09
kaniThe following is the error from swift log.18:10
kaniJun  7 08:51:48 cs-swift proxy-server: ERROR with Object server 192.168.0.11:6000/d1 re: Trying to get final status of PUT to /v1/AUTH_4a6f81f6d3a94b469053da84fc16a8f9/glance/3116d48f-1bc6-43aa-8f45-61d31599ad84: Timeout (10s)18:10
kaniJun  7 08:51:48 cs-swift proxy-server: ERROR with Object server 192.168.0.10:6000/d1 re: Trying to get final status of PUT to /v1/AUTH_4a6f81f6d3a94b469053da84fc16a8f9/glance/3116d48f-1bc6-43aa-8f45-61d31599ad84: Timeout (10s)18:10
kaniJun  7 08:51:48 cs-swift proxy-server: ERROR with Object server 192.168.0.12:6000/d1 re: Trying to get final status of PUT to /v1/AUTH_4a6f81f6d3a94b469053da84fc16a8f9/glance/3116d48f-1bc6-43aa-8f45-61d31599ad84: Timeout (10s)18:10
kaniJun  7 08:51:48 cs-swift proxy-server: 192.168.0.4 192.168.0.4 07/Jun/2014/08/51/48 PUT /v1/AUTH_4a6f81f6d3a94b469053da84fc16a8f9/glance/3116d48f-1bc6-43aa-8f45-61d31599ad84 HTTP/1.0 503 - python-swiftclient-2.3.1.45.g892be34 25d50bf1b8814886... 2147483648 118 - tx2ac01b9ab3b24f71849b4-005392d259 - 75.0299 - - 1402131033.757571936 1402131108.78744411518:10
kaniJun  7 08:52:13 cs-swift container-server: 192.168.0.10 - - [07/Jun/2014:08:52:13 +0000] "PUT /d1/990/AUTH_4a6f81f6d3a94b469053da84fc16a8f9/glance/3116d48f-1bc6-43aa-8f45-61d31599ad84" 201 - "PUT http://192.168.0.10:8080/d1/400/AUTH_4a6f81f6d3a94b469053da84fc16a8f9/glance/3116d48f-1bc6-43aa-8f45-61d31599ad84" "tx2ac01b9ab3b24f71849b4-005392d259" "object-server 4519" 0.0005 "-" 438818:11
kaniJun  7 08:52:13 cs-swift object-server: 192.168.0.10 - - [07/Jun/2014:08:52:13 +0000] "PUT /d1/400/AUTH_4a6f81f6d3a94b469053da84fc16a8f9/glance/3116d48f-1bc6-43aa-8f45-61d31599ad84" 201 - "PUT http://192.168.0.10:8080/v1/AUTH_4a6f81f6d3a94b469053da84fc16a8f9/glance/3116d48f-1bc6-43aa-8f45-61d31599ad84" "tx2ac01b9ab3b24f71849b4-005392d259" "proxy-server 4176" 99.5822 "-" 451918:11
kaniSwift has 3 replicas. And I am trying to upload 4 images in parallel to glance18:12
*** wer_ is now known as wer18:13
*** NM1 has joined #openstack-swift18:15
*** acoles is now known as acoles_away18:15
*** NM has quit IRC18:17
notmynamekbee: beyond the logging, let's back up. you aren't having any issues with running multiple instances of swift-drive-audit at the same time? there isn't any sort of protection built in to the script, and all instances of it will read from the same log source and write the same output file. why are you running more than one at a time?18:17
*** Masahiro has joined #openstack-swift18:17
kbeenotmyname: one can be run from admin to check, one can be run from mointoring tools such as icinga, other can be a cron.. thats the reason18:18
kbeenotmyname: ^ the above may not happen all the time, but there is a window of opportunity18:20
notmynamekani: interesting. seems like 2 of those object server puts succeeded, but the response code didn't get back to the proxy server. so the issue is either that the proxy is overloaded and couldn't get to the response before the timer fired. or that the object server is overloaded. what else is going on in this cluster? what sort of hardware is it running on?18:20
*** NM1 has quit IRC18:20
notmynamekbee: ok, so you normally have it running one way. but sometimes you want to also run it from another tool, and you want to make sure you can cleanly demux the logs to find the right output. right?18:21
*** Masahiro has quit IRC18:22
notmynamekbee: still seems like a good way to corrupt your fstab file (ie it writes to fstab.new every time. not a tempfile)18:22
*** NM has joined #openstack-swift18:22
kbeenotmyname: correct.. for every run, one should be able to get the output for that run predictably18:22
notmynamekbee: ok. that makes sense to me. I still think you could probably get that with some syslog config, but that's secondary to the other issue of multiple writers to fstab.new. that's not really going to work out well for you18:23
*** NM has quit IRC18:23
kbeenotmyname: totally agree.. drive-audit needs to change for that. but all i had in the patch for ability to log to console in addition to syslog :) .. and thats useful in certian scenarios18:24
notmynamegranted, there's also the new config option there so you don't unmount failed drives...18:24
kbeenotmyname: true..18:24
*** rmcall has quit IRC18:25
notmynamebut in that case you'd be running with different config files anyway. ie one for the normal runner that unmounts. and a different one that is used when you run "by hand" and that doesn't unmount18:25
notmynamebut in the second case, the reason for running it would be just as a checker. not to actually unmount. do you agree there?18:26
notmynameis that how you're using it?18:26
*** rmcall has joined #openstack-swift18:27
*** rmcall has quit IRC18:29
mahaticnotmyname, OPTIONS is in base.py and ObjectController class (in obj.py) does not refer to OPTIONS. And tests are in test_server.py but not test_base.py?18:29
mahaticnotmyname, I mentioned ObjectController because the tests are sending the request for OPTIONS to ObjectController18:30
notmynamemahatic: fun! makes perfect sense! ;-)18:30
kbeenotmyname: agree.. , i am trying to use with unmount set to true. i am using the same config file. once i unmount from one session, i will not see those errors again since drive-audit does not read unmounted devices.18:31
kaniSAIO is running on 3 vms on KVM18:31
mahaticnotmyname, :D sorry, Let me do it in a paste.openstack maybe18:31
*** gyee has joined #openstack-swift18:31
kaninotmyname: SAIO is running on 3 vms on KVM18:31
notmynamemahatic: that's because the existing OPTIONS also tests for CORS functionality18:31
*** gordc has joined #openstack-swift18:31
notmynamemahatic: so that's why it's using the ObjectController18:32
notmynamemahatic: as to test_server vs test_base. I don't know18:32
kbeenotmyname: here's the code snippet -> https://gist.github.com/keshavab/f34bb63d93025bdff2b718:32
mahaticnotmyname, true. But there is no OPTIONS in the ObjectController, but it's passing that request as well18:32
mahaticand it's passing*18:33
notmynamemahatic: will ObjectController has OPTIONS because it's inherited from Controller in base.py18:33
mahaticnotmyname, oh I missed that18:33
notmynamemahatic: so I'm not sure if there's a good reason for the way it is. or a reason at all. maybe it should move to test_base.py. I don't know18:34
mahatichmm18:34
*** EmilienM|afk is now known as EmilienM18:35
kbeenotmyname: i was giving the prev example of multiple simultaneous invocations as just an usecase to explain why we need to have ability to log to console for predictable outputs18:35
*** abhirc has joined #openstack-swift18:37
kbeenotmyname: so what do you say ?18:38
*** rmcall has joined #openstack-swift18:38
notmynamekani: I don't know the answer. but here's (basically) what's going on. either the proxy is overloaded and not getting to the backend requests fast enough. this would happen if you have a lot of concurrent requests. the solution is to upgrade the proxy or add more proxy servers. or the object servers are getting slow when doing the fsync to disk and not sending a response soon enough18:39
notmynamekani: because you're running storage nodes on VMs (never recommended), I suspect it's the latter. but again, that's just a guess. those are the areas to investigate though18:39
notmynamemahatic: for now, don't worry about moving the tests around. that is, that's a separate issue from "there is a test, and it is run"18:40
notmynamemahatic: you're looking at writing a test for your current patch?18:40
mahaticnotmyname, yup, I'm not :)18:40
mahaticnotmyname, yeah, test for current patch18:40
mahaticnotmyname, and it comes off quite small (10-15 lines). I'm mentioning because you were saying "tests" the other day18:42
kaninotmyname: Thanks for the response.18:44
notmynamemahatic: ya. as you pointed out, your patch will need tests. so I'm glad you're looking in to that. do you know what needs to be tested and why?18:45
*** kani has quit IRC18:46
mahaticnotmyname, I guess - testing if the OPTIONS is indeed responding with the allowed methods (basically it's functionality). right?18:46
notmynamemahatic: right. this is a unit test to make sure the OPTIONS method is actually responding with what you expect it to respond with. that way we don't break it in the future.18:48
*** NM has joined #openstack-swift18:48
*** rmcall has quit IRC18:48
*** HenryG_ has quit IRC18:48
notmynamemahatic: since you're adding this to a storage node process, there won't be a functional test for it. clients can't talk to object servers, so there's no way to run the test18:48
mahaticnotmyname, and correct me if i'm wrong - right now, I'm only writing test for the patch I've submitted. We're yet to decide where to put the OPTIONS (for all the nodes), consolidate it into one patch and add more tests18:49
*** HenryG has joined #openstack-swift18:49
mahaticnotmyname, yeah18:49
kbeenotmyname: So do you find a value in the patch to have ability to log to console additionally and optionally based on a config  ? :)18:49
notmynamemahatic: correct. but I'd expect that the consolidation of OPTIONS will need to happen before your patch lands.18:49
*** rmcall has joined #openstack-swift18:50
mahaticnotmyname, yeah of course (i'm assuming landing as in merging?)18:51
*** NM has quit IRC18:51
notmynamemahatic: right18:51
notmynamekbee: not sure yet. I'm not convinced either way yet. I think the end functionality might be possible with syslog configs locally with no need for changes in swift. (might). but also I'm concerned about multiple processes at the same time. drive-audit already works with swift-recon to report what's going on...18:51
mahaticnotmyname, so what about where to put OPTIONS (for all the nodes to access), is that for a lil later? after the test?18:52
*** aix has quit IRC18:53
notmynamemahatic: well, I don't think the patch will land without both, so whichever you want to work on first is fine :-)18:54
mahaticnotmyname, yeah, But I don't know where to put after yesterday's inputs from the others.18:55
*** NM has joined #openstack-swift18:55
kbeenotmyname: hmmm...can you pls explain how its possible with local syslog configs..18:55
kbeenotmyname: if i want to isolate drive-audit-logs only ?18:56
notmynamemahatic: my first guess is a common base class for each storage node. that new class would live in swift.common. And for now all it would have is the common OPTIONS method. maybe later (much later) we could figure out if there's any other code consolidation that could be done18:58
mahaticnotmyname, if I were to take a call, I would just go ahead and put it in swift.common coz that's the part i understand ;)18:58
mahaticnotmyname, okay sure!18:58
notmynamemahatic: right. some new file there. maybe a BaseStorageServer() class. naming is hard18:59
*** fandi has quit IRC18:59
mahaticnotmyname, yeah. okay19:00
*** gordc has left #openstack-swift19:01
notmynamekbee: I don't know. but maybe the ErrorMessagesToStderr setting for rsyslog. maybe syslod-ng has something too. combined with the syslog facility, you could redirect the output elsewhere. I've not tried any of that. only just did a quick google search19:02
torgomaticswift.common.uhh_stuff_and_things19:03
notmynametorgomatic: perfect!19:03
mahatic:D sure, I'll also put torgomatic as reviewer ;)19:04
torgomaticmahatic: be careful what you ask for; you may get it ;)19:04
mahatic:D sure19:06
tdasilvanotmyname: hi, just saw this: https://www.brightbox.com/blog/2014/12/16/announcing-orbit-cloud-storage/ thought you might be interested19:08
*** shri1 has joined #openstack-swift19:09
notmynametdasilva: cool! thanks19:09
*** shri1 has quit IRC19:09
kbeenotmyname: hmm.. thnx..19:10
*** nellysmitt has joined #openstack-swift19:11
*** shri has quit IRC19:11
*** kbee has quit IRC19:15
*** nellysmitt has quit IRC19:16
peluseanyone know what's up with recheck no bug apparently not doing anything?19:17
*** acorwin_ is now known as acorwin19:18
*** lpabon has joined #openstack-swift19:20
*** zaitcev has joined #openstack-swift19:20
*** ChanServ sets mode: +v zaitcev19:20
notmynamepeluse: link?19:20
pelusenotmyname, I see a bunch of them but this is the one I was trying https://review.openstack.org/#/c/132389/19:21
notmynamepeluse: looks like it reran (check the "toggle ci" button). but still has an error. I think that's a known thing that people are looking in to. maybe something with ceilometer causing the failure19:24
notmynameI'm looking into that issue. sortof. at least I was asked in -qa about it. to look in to it.19:24
notmynamethere might be a workaround19:25
pelusenotmyname, ahh, thanks.  Man, it fails fast....19:26
linuxgeek_with swift as the glance back end it takes 30 minutes for a 30gb file to be uploaded to glance, is this expected? how do i track what is causing the delay?19:31
*** mahatic has quit IRC19:34
openstackgerritOpenStack Proposal Bot proposed openstack/python-swiftclient: Updated from global requirements  https://review.openstack.org/8925019:36
openstackgerritOpenStack Proposal Bot proposed openstack/swift: Updated from global requirements  https://review.openstack.org/8873619:37
*** mahatic has joined #openstack-swift19:41
*** linuxgeek_ has left #openstack-swift19:48
*** rmcall has quit IRC19:58
*** rmcall has joined #openstack-swift19:59
*** exploreshaifali has joined #openstack-swift20:03
*** Masahiro has joined #openstack-swift20:06
*** Masahiro has quit IRC20:11
mahaticnotmyname, this is more python related question (didn't know how to form this for google search) - when a class A imports class B, the instance self will be of class A even for class B's functions?20:25
mahatici.e. when I call class B's function in class A20:25
mahatictorgomatic,^^20:27
torgomaticmahatic: you mean "class Stuff(Thing)"?20:27
mahatictorgomatic, :) yes20:27
torgomaticmahatic: yes, "self" will be the object on which the method is invoked, so you can have a derived-class self in a base-class method.20:28
mahaticclass A(self) { B.stuff(self) }20:29
torgomaticmahatic: are you inheriting or composing? perhaps some pastebinned code would help.20:29
mahatictorgomatic, now class B will have class A "self" object, correct?20:29
mahatictorgomatic, inheriting20:30
mahaticokay, just a min20:30
mahatictorgomatic, http://paste.openstack.org/show/152005/20:35
mahatictorgomatic, now class B will have the object "self" of A when invoked from A right?20:35
zaitcevmahatic: but the two in your example are not related20:36
mahaticlet me know if the pastebin doesn't make sense :)20:36
mahaticzaitcev, sorry! let me correct20:36
zaitcevwell, I guess you meant   class A(B):20:38
mahatichttp://paste.openstack.org/show/152007/20:38
mahaticzaitcev, makes sense?20:38
torgomaticmahatic: http://paste.openstack.org/show/152008/ is more like what you're after, I think20:39
torgomaticand yes, in the invocation of Base.anothermethod(), "self" will be an instance of Derived20:39
zaitcevthere are exmaples in db.py and backend.py that one could pilfer easily20:40
mahaticzaitcev, yes class A(B)20:41
torgomaticalso, http://learnpythonthehardway.org/book/ex44.html is one of many examples online of Python inheritance, but it has some amusing words at the start that make it better than most.20:41
mahatictorgomatic, you're right. but OPTIONS not needed? when it is invoked in a node, and if that method is inherited, it need not be declared?20:42
mahatictorgomatic, thank you, will look at it.20:42
torgomaticmahatic: yes. if foo() is declared in the base class, and the base-class implementation is what you want the derived class to have too, then you don't need to define foo() in the derived class20:43
mahatictorgomatic, yes, right. I got confused with a similar code elsewhere. Thanks for the clarification!20:45
mahaticzaitcev, will look at them - db.py and backend.py - thanks20:49
*** dmsimard is now known as dmsimard_away20:50
imkarrerGood afternoon!  I am curious.  Which service is responsible for creating and setting permissions on the tmp directory on the devices holding the account and container DB's?  The db_replicator depends tmp being available but I cannot find who is responsible for creating and setting permissions on the tmp directory20:56
imkarrerIs that something which needs to be created when swift is deployed?20:57
notmynameimkarrer: it's done in diskfile.py:159621:03
notmynameimkarrer: ie in the object server21:03
imkarrernotmyname: hmmm, its seems that tmp did not inherit the permissions of its parent directory.  It does not appear that mkdirs assigns permissions after creating tmp21:06
imkarrernotmyname: it is probably more likely that the parent's permissions were not set properly, not allowing /tmp to be created.  I need to further debug my issue.  Thanks for the info!21:08
notmynameimkarrer: np21:08
notmynameimkarrer: it would be created by the user running the swift process. the default is to drop permissions to the "swift" user. so ya, maybe the parent dir wasn't set right21:09
notmynamedoes anyone (swift devs) consider keeping the sample config files in etc/ up to date to be hard or burdensome?21:10
swifterdarrellnotmyname: yes, but I think it's totally worth it21:10
notmynameswifterdarrell: the current cross-project meeting is talking about some standard way to generate them. because recently they were removed from the source tree in other projects and operators complained21:11
notmynameie no static file in the source tree. but also available to packagers21:11
*** nellysmitt has joined #openstack-swift21:11
swifterdarrellnotmyname: doesn't seem hard or burdensome to maintain some wacky code to generate good examples files at all </scathing-sarcasm>21:12
tdasilvalol21:13
mattoliverauMorning21:14
tdasilvanotmyname: wouldn't you have to generate a config to generate the config? or is it going to look in the code to find the default values for options?21:15
*** tellesnobrega has quit IRC21:15
swifterdarrellnotmyname: perhaps it can achieve a similar level of quality as "git shortlog" for Changelog...21:15
tdasilvatorgomatic: "Inheritance is the evil forest", pretty funny stuff21:16
notmynametdasilva: I think it all comes from oslo config21:16
*** nellysmitt has quit IRC21:17
*** tellesnobrega has joined #openstack-swift21:17
torgomaticbesides, sometimes the action of typing in the sample config shows me that my config option is stupidly named or has dumb semantics21:18
mattoliverauI like having the configs in /etc in most cases its better then the doc cause its something concrete with default values and explanations. Putting on my sysadmin hat, I found sample configs very useful. But that could just be me :)21:19
*** mvenesio has quit IRC21:19
*** lpabon has quit IRC21:22
claygmattoliverau: it's definately not just you21:23
claygnotmyname: get out the sanity stick - THE BEATINGS WILL CONTINUE!!!21:23
mattoliverauLol21:23
notmynamewatching this conversation in the meeting is fascinating. I had no idea these are problems elsewhere in openstack21:25
claygnotmyname: conf.d with example configs from oslo messaging that get copied in when you do the oslo update dance?21:26
notmynameclayg: except that oslo is actually better about libraries now. so it's that an external library (real module!) getting updated can change config options in your app21:27
claygi guess if you have a truly external depends that you configure pass-through the *only* correct answer is to say "see docs in this other project, because we *really* have no idea what's the union of options it's supports that work with our app"21:27
notmynameya that21:27
notmynameand since ranges of versions for libraries are supported, then basically and statically generated file is always out of date unless installed versions match exactly21:28
mattoliverauI thought de stack generated the nova comfig but for other projects it grabs the sample config and then seds... But I guess that must've changed now21:28
*** geaaru has quit IRC21:28
mattoliverau*devstack21:28
mattoliverauWow my typing is great this morning (I blame not enough coffee)21:28
claygmattoliverau: i'm not so sure that's changed, we still have to merge default changes for keystone stuff into our /etc configs to "make devstack better"21:29
* notmyname doesn't know if they are joking about an openstack-specs specs repo21:29
mattoliverauclayg: how could that ever go wrong :p21:30
mattoliveraunotmyname: can they call it openstack-specs^2 :p21:31
claygthis is how I imagine specs work for oslo -> https://www.youtube.com/watch?v=w3E1frdYzuo#t=1621:31
mattoliverauLol21:32
*** NM has quit IRC21:41
*** dmsimard_away is now known as dmsimard21:55
*** Masahiro has joined #openstack-swift21:55
notmynameoh my. they aren't joking https://review.openstack.org/#/q/status:open+project:openstack/openstack-specs+branch:master,n,z21:58
*** Masahiro has quit IRC22:00
*** rmcall has quit IRC22:01
*** rmcall has joined #openstack-swift22:03
*** cdelatte has joined #openstack-swift22:06
openstackgerritMerged openstack/swift: Splice: Use `operator.or_` instead of a lambda-expression  https://review.openstack.org/13730722:11
zaitcev"we introduce a tiny but powerful library"22:11
notmynamezaitcev: ya, I liked that too :-)22:13
*** bkopilov has quit IRC22:31
*** bkopilov has joined #openstack-swift22:36
*** rmcall has quit IRC23:00
*** miurahr has quit IRC23:07
*** rmcall has joined #openstack-swift23:07
*** rmcall has quit IRC23:10
*** nellysmitt has joined #openstack-swift23:12
*** rmcall has joined #openstack-swift23:14
*** nellysmitt has quit IRC23:17
*** rmcall has quit IRC23:18
*** rmcall has joined #openstack-swift23:20
*** dmsimard is now known as dmsimard_away23:23
*** ajiang38740 has quit IRC23:27
*** ajiang has quit IRC23:27
*** fandi has joined #openstack-swift23:38
*** fandi has quit IRC23:38
*** Masahiro has joined #openstack-swift23:44
*** Masahiro has quit IRC23:48
*** rmcall has quit IRC23:52

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!