Wednesday, 2015-05-13

*** vinsh has joined #openstack-swift00:12
*** vinsh has quit IRC00:15
*** esker has joined #openstack-swift00:16
*** zhill has quit IRC00:16
*** annegentle has joined #openstack-swift00:17
*** chlong_ has joined #openstack-swift00:20
*** annegentle has quit IRC00:23
*** david-lyle has quit IRC00:28
*** km__ has joined #openstack-swift00:29
*** km has quit IRC00:32
mattoliverauho: morning00:32
*** annegentle has joined #openstack-swift00:33
*** kei_yama has quit IRC00:33
*** esker has quit IRC00:34
homattoliverau: morning!00:34
*** kei_yama has joined #openstack-swift00:35
*** esker has joined #openstack-swift01:01
*** km has joined #openstack-swift01:02
*** km__ has quit IRC01:04
*** esker has quit IRC01:18
*** tobe43 has joined #openstack-swift01:38
*** esker has joined #openstack-swift01:44
*** vjujjuri has quit IRC01:53
*** esker has quit IRC02:03
*** annegentle has quit IRC02:07
*** wbhuber has joined #openstack-swift02:13
*** asettle is now known as asettle-afk02:25
*** esker has joined #openstack-swift02:29
*** annegentle has joined #openstack-swift02:40
*** annegentle has quit IRC02:41
*** esker has quit IRC02:47
*** gyee has quit IRC02:47
*** tobe43 has quit IRC02:58
*** chlong has quit IRC03:05
*** annegentle has joined #openstack-swift03:10
*** esker has joined #openstack-swift03:13
*** wbhuber has quit IRC03:15
*** annegentle has quit IRC03:20
*** esker has quit IRC03:24
*** esker has joined #openstack-swift03:24
*** links has joined #openstack-swift03:30
*** esker has quit IRC03:32
*** vinsh has joined #openstack-swift03:47
*** david-lyle has joined #openstack-swift03:48
*** asettle-afk has quit IRC03:48
*** vjujjuri has joined #openstack-swift03:49
*** vinsh has quit IRC03:51
*** chlong_ has quit IRC03:52
*** km_ has joined #openstack-swift04:02
*** ho_ has joined #openstack-swift04:03
*** ho has quit IRC04:04
*** km has quit IRC04:05
*** occupant has quit IRC04:09
*** occupant has joined #openstack-swift04:11
*** vjujjuri_ has joined #openstack-swift04:19
*** vjujjuri has quit IRC04:20
*** vjujjuri_ is now known as vjujjuri04:20
*** occupant has quit IRC04:33
*** minwoob has quit IRC04:40
*** zaitcev has quit IRC04:57
*** km has joined #openstack-swift05:01
*** ho has joined #openstack-swift05:01
*** ho has quit IRC05:01
*** ho_ has quit IRC05:02
*** km_ has quit IRC05:03
*** ho has joined #openstack-swift05:03
*** bill_az has joined #openstack-swift05:13
*** gsilvis_ has joined #openstack-swift05:14
*** ir2ivps8 has quit IRC05:15
*** patchbot` has joined #openstack-swift05:15
*** patchbot has quit IRC05:15
*** patchbot` is now known as patchbot05:17
*** dosaboy_ has joined #openstack-swift05:18
*** guitarza1 has joined #openstack-swift05:19
*** chlong has joined #openstack-swift05:23
*** bkopilov is now known as bkopilov_wfh05:24
*** ndk_ has joined #openstack-swift05:26
*** ndk has quit IRC05:26
*** thurloat has quit IRC05:26
*** gsilvis has quit IRC05:26
*** guitarzan has quit IRC05:26
*** dosaboy has quit IRC05:27
*** SkyRocknRoll has joined #openstack-swift05:31
*** SkyRocknRoll has joined #openstack-swift05:31
*** thurloat has joined #openstack-swift05:33
*** chlong has quit IRC05:34
*** ir2ivps8 has joined #openstack-swift05:34
*** ppai has joined #openstack-swift05:38
*** chlong has joined #openstack-swift05:47
*** zhill has joined #openstack-swift05:51
*** ppai has quit IRC05:53
*** ppai has joined #openstack-swift05:56
*** silor has joined #openstack-swift06:00
*** ppai has quit IRC06:05
*** proteusguy has joined #openstack-swift06:12
*** ppai has joined #openstack-swift06:12
*** chlong has quit IRC06:24
*** chlong has joined #openstack-swift06:27
*** bill_az has quit IRC06:34
openstackgerritKota Tsuyuzaki proposed openstack/swift: Remove confusable query string on post as copy  https://review.openstack.org/17892706:49
*** fifieldt has joined #openstack-swift06:56
*** fifieldt has quit IRC06:56
*** vjujjuri has quit IRC06:56
*** kota_ has joined #openstack-swift07:00
*** jordanP has joined #openstack-swift07:08
openstackgerritMatthew Oliver proposed openstack/swift: Add concurrent reads option to proxy  https://review.openstack.org/11771007:08
*** zhill has quit IRC07:22
*** geaaru has joined #openstack-swift07:25
*** jistr has joined #openstack-swift07:49
openstackgerritKota Tsuyuzaki proposed openstack/swift: Fix the missing SLO state on fast-post  https://review.openstack.org/18256408:00
*** acoles_away is now known as acoles08:03
kota_acoles, I just posted a fix for bug 1453807!08:05
openstackbug 1453807 in OpenStack Object Storage (swift) "Post (not as copy) to SLO manifest destroys its state as a manifest" [Undecided,New] https://launchpad.net/bugs/145380708:05
acoleskota_: hi, cool, thx08:11
openstackgerritAlistair Coles proposed openstack/swift: Filter Etag key from ssync replication-headers  https://review.openstack.org/17397308:40
tab___Is TempURL generated per container or per account for public folder?08:45
tab___sorry , for private folder08:46
*** ho has quit IRC08:49
ahalethe tempurl secret is per account, but the generated hmac are normally for specific objects08:59
ahalei think thers a patch for per-container secrets, either added recently or coming09:00
*** kei_yama has quit IRC09:25
*** km has quit IRC09:28
openstackgerritKota Tsuyuzaki proposed openstack/swift: Fix the missing SLO state on fast-post  https://review.openstack.org/18256409:32
*** 5EXAA60VI has quit IRC09:40
*** kota_ has quit IRC09:45
openstackgerritAlistair Coles proposed openstack/swift: Cleanup and extend end to end ssync tests  https://review.openstack.org/17937910:05
acoleskota_: thanks for your reviews10:05
*** jamielennox is now known as jamielennox|away10:08
*** mattoliverau has quit IRC10:25
*** early has quit IRC10:26
*** mattoliverau has joined #openstack-swift10:26
*** classicsnail has quit IRC10:27
*** acoles has quit IRC10:27
*** early has joined #openstack-swift10:28
*** jd__ has quit IRC10:28
*** jd__ has joined #openstack-swift10:32
*** EmilienM has quit IRC10:33
*** ekarlso has quit IRC10:33
*** EmilienM has joined #openstack-swift10:35
*** classicsnail has joined #openstack-swift10:36
*** acoles has joined #openstack-swift10:38
*** ChanServ sets mode: +v acoles10:38
tab___creating temp url over some picture:    swift-temp-url GET 300 /v1/AUTH_d2d817e93ad644439ec29fc7bce52eba/privat/narezek.jpg swiftpass10:40
tab___i get url with which i go to browser and it enables me to download a file10:40
*** ekarlso has joined #openstack-swift10:41
tab___wich other method to use for making picture not immediately downloadable , but shown within browser10:41
tab___i tired other methods ( PUT POST) but each time i get than "401 Unauthorized: Temp URL invalid"10:42
ahaleyou can add a &inline to the GET to make it change the content-disposition thats inserted10:43
*** ekarlso has quit IRC10:45
*** ekarlso has joined #openstack-swift10:45
tab___ahale: &inline to the end of generated URL?10:46
ahaleyah, theres a bit of blog about it at https://swiftstack.com/blog/2015/01/29/swift-feature-highlight-tempurls/10:46
ahalein the advanced usage part10:46
tab___ahale: aha great. thank you!10:47
ahaleno probs10:47
tab___ahale: do you have also some great online knowledge resource link for testing object-expiration?10:48
ahalehm not really.. I dont think theres much about that :( do you mean from inside or outside the cluster?10:49
*** thurloat is now known as thurloat_isgone10:51
tab___inside10:51
ahalemainly just checking out the .expiring_objects account and its structure10:52
ahalei use swiftly for that since its direct client makes it really easy to do `swiftly --direct=/v1/.expiring_objects get -l 10`, to see the oldest unexpired hours, then check the pending work inside10:53
tab___ahale: one more question for you maybe. I am setting rings (ring builder files). Let's say i am creating multi node ring, so different IP for object store an disks are used. is it better to use hostnames or IPs withing add to the ring command?10:53
tab___aha ok i will check swiftly. thx10:54
ahaleand theres checking swift-get-nodes ssh output to see if the obj is really 404'ing or just object server is faking it10:54
tab___aha. great to know10:54
ahalewe always use IPs, im not even sure hostnames works?10:54
tab___aha ok. and what happens10:54
tab___let's say. i have multi nodes that are using ring file with ips10:55
tab___and than for examples, ips are changed10:55
tab___how does that influence on data etc10:55
ahale(and the unexpired object will still show in a container listing for the user as well until its cleaned up by the expirer process)10:55
tab___do i just update ring files and replicate them to all machines10:55
tab___what happens with data rebalancing, i do not want that etc...10:55
ahalethat would be bad, you wouldn't want to re-ip a box10:56
tab___yes, but what if :)10:56
tab___that's why i was asking if it would be better to use hostnames on longer period...10:56
ahalei guess, if you did, replication should eventually sort it out, but it would be really horrible user experience while that happened10:57
tab___maybe not life case, but anyways...10:57
tab___ok10:57
ahalehostnames might work, i remember there was some talk about ipv6 as well , i didnt really pay attention so I wouldnt trust me too much on that part10:57
tab___ok. i will check it. ipv6 i guess woudl come into play with kinetic drives :)10:58
openstackgerritMerged openstack/swift: Allow rsync to use compression  https://review.openstack.org/14845410:58
*** Trozz has quit IRC10:58
*** Trozz has joined #openstack-swift11:05
tab___ahale: which tool would you recommend for check on disk health, possible some web GUI?11:06
ahalewe use a homegrown tool for that mostly11:07
ahaleand some hacked up copies of swift-recon -u that feed into graphite for dash / stats11:08
tab___ok thx11:11
ahaleim not sure what other people use for monitoring and health11:12
tab___ahale: regarding swiftly. is that project on github?11:16
tab___i am trying to find the right tool11:16
ahaleyeah should be, gholt/swiftly i think11:17
*** tobe4333 has joined #openstack-swift11:35
openstackgerritMerged openstack/swift: Remove 1 line of dead code from EC reconstructor  https://review.openstack.org/18247911:40
*** delattec has quit IRC11:50
*** cdelatte has quit IRC11:50
*** fthiagogv has joined #openstack-swift11:54
*** tobe4333 has quit IRC11:57
*** lpabon_ has joined #openstack-swift12:01
*** dmorita has quit IRC12:01
*** lpabon_ has quit IRC12:01
*** ppai has quit IRC12:02
tab___ahole: ok thx, i'll check12:06
*** cdelatte has joined #openstack-swift12:07
*** delattec has joined #openstack-swift12:09
*** cdelatte has quit IRC12:12
*** aix has joined #openstack-swift12:13
openstackgerritStuart McLaren proposed openstack/python-swiftclient: Add minimal working service token support.  https://review.openstack.org/18264012:14
*** lpabon has quit IRC12:14
*** proteusguy has quit IRC12:15
*** ppai has joined #openstack-swift12:16
*** lpabon has joined #openstack-swift12:19
*** annegentle has joined #openstack-swift12:34
*** annegentle has quit IRC12:45
*** annegentle has joined #openstack-swift12:46
*** ppai has quit IRC12:50
*** dencaval has joined #openstack-swift12:55
*** fthiagogv has quit IRC12:58
*** bill_az has joined #openstack-swift13:00
*** bkopilov_wfh has quit IRC13:04
*** annegentle has quit IRC13:06
*** bkopilov_wfh has joined #openstack-swift13:07
*** annegentle has joined #openstack-swift13:07
*** wbhuber has joined #openstack-swift13:20
*** wbhuber has quit IRC13:25
*** guitarza1 is now known as guitarzan13:26
*** fthiagogv has joined #openstack-swift13:31
*** wbhuber has joined #openstack-swift13:33
*** SkyRocknRoll has quit IRC13:33
*** vjujjuri has joined #openstack-swift13:34
*** vjujjuri_ has joined #openstack-swift13:37
*** vjujjuri has quit IRC13:39
*** vjujjuri_ is now known as vjujjuri13:39
openstackgerritKamil Rykowski proposed openstack/swift: Replaced setting run_pause with standard interval  https://review.openstack.org/17117013:40
*** vjujjuri has quit IRC13:48
*** links has quit IRC13:48
*** NM has joined #openstack-swift13:59
*** haomaiwang has joined #openstack-swift14:15
*** jrichli has joined #openstack-swift14:21
*** breitz has quit IRC14:23
*** breitz has joined #openstack-swift14:24
*** haomaiwang has quit IRC14:25
*** bkopilov has joined #openstack-swift14:26
*** bkopilov_wfh has quit IRC14:27
*** vinsh has joined #openstack-swift14:30
*** chlong has quit IRC14:30
*** fanyaohong has joined #openstack-swift14:37
*** fthiagogv has quit IRC14:48
*** minwoob has joined #openstack-swift14:52
*** NM has quit IRC14:53
*** bkopilov has quit IRC14:57
*** fthiagogv has joined #openstack-swift15:01
*** marcusvrn_ has joined #openstack-swift15:02
*** bkopilov has joined #openstack-swift15:11
*** vjujjuri has joined #openstack-swift15:18
*** jistr is now known as jistr|mtg15:19
*** nadeem has joined #openstack-swift15:26
*** nadeem has quit IRC15:28
*** nadeem has joined #openstack-swift15:29
*** jistr|mtg is now known as jistr15:29
*** proteusguy has joined #openstack-swift15:35
*** mahatic has joined #openstack-swift15:38
*** geaaru has quit IRC15:39
*** gyee has joined #openstack-swift15:44
*** bkopilov has quit IRC15:44
*** bkopilov has joined #openstack-swift15:45
openstackgerritpradeep kumar singh proposed openstack/swift: Closes-Bug:1169189  https://review.openstack.org/18273415:50
*** shakamunyi has quit IRC15:51
*** barra204 has quit IRC15:51
*** esker has joined #openstack-swift15:53
*** wbhuber_ has joined #openstack-swift16:00
*** wbhuber has quit IRC16:04
*** silor has quit IRC16:12
*** erlon has joined #openstack-swift16:12
*** NM has joined #openstack-swift16:18
notmynamegood morning16:23
*** jistr has quit IRC16:25
tdasilvamorning16:25
acolesnotmyname: good morning. are we meeting today?16:27
*** jordanP has quit IRC16:29
notmynameyes we are meeting today16:31
notmynamewe need to choose the last working session topic (or remind me what it is) and I want to make sure I'm available for any summit questions16:32
*** wbhuber__ has joined #openstack-swift16:36
openstackgerritOpenStack Proposal Bot proposed openstack/swift: Updated from global requirements  https://review.openstack.org/8873616:39
*** wbhuber_ has quit IRC16:40
*** thurloat_isgone is now known as thurloat16:48
*** fanyaohong has quit IRC16:49
*** acoles is now known as acoles_away16:54
*** thurloat is now known as thurloat_isgone17:07
*** nadeem has quit IRC17:09
*** harlowja has quit IRC17:20
*** harlowja has joined #openstack-swift17:20
*** nadeem has joined #openstack-swift17:23
notmynameacoles_away: jrichli: how much have you looked at barbican in the context of the encryption work?17:35
notmynameone thing I've learned recently is that barbican doesn't support inline secret key generation17:39
notmynameyou put in an "order" for generating a secret and it's created async and you can get it later17:39
*** marcusvrn_ is now known as marcusvrn17:41
*** anirup has joined #openstack-swift17:43
*** fthiagogv has quit IRC17:47
*** wbhuber__ has quit IRC17:49
*** wbhuber has joined #openstack-swift17:49
*** NM has quit IRC17:49
vjujjuriHi All I am able to progress from python issue. but running into auth issue. can someone help me?17:54
vjujjurish-4.2# sudo curl -v -H 'X-Storage-User: test:tester' -H 'X-Storage-Pass: testing' http://127.0.0.1:8080/auth/v1.017:56
vjujjuri* About to connect() to 127.0.0.1 port 8080 (#0)17:57
vjujjuri*   Trying 127.0.0.1...17:57
vjujjuri* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)17:57
*** occupant has joined #openstack-swift17:57
vjujjuri> GET /auth/v1.0 HTTP/1.117:57
vjujjuri> User-Agent: curl/7.29.017:57
vjujjuri> Host: 127.0.0.1:808017:57
vjujjuri> Accept: */*17:57
vjujjuri> X-Storage-User: test:tester17:57
vjujjuri> X-Storage-Pass: testing17:57
vjujjuri>17:57
vjujjuri< HTTP/1.1 200 OK17:57
vjujjuri< X-Storage-Url: http://127.0.0.1:8080/v1/AUTH_test17:57
vjujjuri< X-Auth-Token: AUTH_tkb0c7573a84004b1e9bb9e3018b5ed46b17:57
vjujjuri< Content-Type: text/html; charset=UTF-817:57
vjujjuri< X-Storage-Token: AUTH_tkb0c7573a84004b1e9bb9e3018b5ed46b17:57
vjujjuri< Content-Length: 017:57
vjujjuri< X-Trans-Id: tx64f7af4495db438c8285c-0055538fbd17:57
vjujjuri< Date: Wed, 13 May 2015 17:54:05 GMT17:57
vjujjuri<17:57
vjujjuri* Connection #0 to host 127.0.0.1 left intact17:57
vjujjurish-4.2# curl -v -H 'X-Auth-Token: AUTH_tkb0c7573a84004b1e9bb9e3018b5ed46b' http://127.0.0.1:8080/v1/AUTH_test17:57
vjujjuri* About to connect() to 127.0.0.1 port 8080 (#0)17:57
vjujjuri*   Trying 127.0.0.1...17:57
vjujjuri* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)17:57
vjujjuri> GET /v1/AUTH_test HTTP/1.117:57
vjujjuri> User-Agent: curl/7.29.017:57
vjujjuri> Host: 127.0.0.1:808017:57
vjujjuri> Accept: */*17:57
vjujjuri> X-Auth-Token: AUTH_tkb0c7573a84004b1e9bb9e3018b5ed46b17:57
vjujjuri>17:57
vjujjuri< HTTP/1.1 401 Unauthorized17:57
vjujjuri< Content-Length: 13117:57
*** haomaiwa_ has joined #openstack-swift17:57
vjujjuri< Content-Type: text/html; charset=UTF-817:57
vjujjuri< Www-Authenticate: Swift realm="AUTH_test"17:57
vjujjuri< X-Trans-Id: txa52cf687f2cc4e6d950f6-005553905617:57
vjujjuri< Date: Wed, 13 May 2015 17:56:38 GMT17:57
vjujjuri<17:57
vjujjuri* Connection #0 to host 127.0.0.1 left intact17:57
vjujjuri<html><h1>Unauthorized</h1><p>This server could not verify that you are authorized to access the document you requested.</p></html>sh-4.2#17:57
notmynamevjujjuri: please use paste.openstack.org17:58
vjujjurinotmyname: Sure thanks for the reminder.17:58
vjujjurianyone here know how to disable keystone17:59
*** marcusvrn has quit IRC17:59
vjujjuriand to operate swift no auth is needed17:59
vjujjuri?18:00
claygMooingLemur: isn't part of the ring in pickle?  is there a cpan module for that?18:02
claygvjujjuri: it's easy, just take auth out of the pipeline18:03
claygvjujjuri: problem is most clients except some sort of auth endpoint to get from which to get the storage url - but you can curl and put with no auth token just fine18:03
vjujjurii see18:04
vjujjurijust curl put to the util?18:04
*** annegentle has quit IRC18:04
claygyeah that works fine18:04
jrichlinotmyname: we are still in the early stages of exploration with barbican.  I know that we have been focusing on the interaction where we supply a key.18:07
vjujjuriclayg:18:08
vjujjurish-4.2# curl -X PUT /tmp/ha http://127.0.0.1:8080/v1/ha18:08
vjujjuricurl: (3) <url> malformed18:08
vjujjuri<html><h1>Unauthorized</h1><p>This server could not verify that you are authorized to access the document you requested.</p></html>sh-4.218:08
claygvjujjuri: yeah but you have to take the auth middleware out of the pipeline:main section in the proxy config and restart the proxy18:15
vjujjuriclayg: you mean this section?18:16
vjujjuri[filter:tempauth]18:17
vjujjuriuse = egg:swift#tempauth18:17
vjujjuriuser_admin_admin = admin .admin .reseller_admin18:17
vjujjuriuser_test_tester = testing .admin18:17
vjujjuriuser_test2_tester2 = testing2 .admin18:17
vjujjuriuser_test_tester3 = testing318:17
claygno, [pipeline:main] section18:17
vjujjuriclayg: sorry new to swift. :)18:17
vjujjuriRight in that secion18:17
claygyeah np, we love to help - but bro - you have got to stop pasing multiline code into this channel18:17
vjujjurilet me upload my config file to pastebin18:18
claygvjujjuri: why don't you paste up the entire proxy-config somewhere - I can tell you the line number or give you a dif18:18
clayg;)18:18
vjujjurihttp://pastebin.com/P9M4gKTb18:19
vjujjuriclayg: ^18:19
claygso line 13 - tempauth is in the middle of the long list of names - between crossdomain and staticweb18:21
claygjust delete the word tempauth - and you will have a noauth cluster18:22
claygvjujjuri: but didn't you say you were trying to make keystone work?18:22
*** gvernik has joined #openstack-swift18:23
vjujjuriclayg: yes ; if we can make that work it is great.18:23
vjujjuribasically I am trying to get some perf numbers18:23
vjujjurisince getting keystone running is taking time, wondering if we can do this way - by disabling keystone18:24
claygoic18:24
clayg... so did pulling tempauth out of th pipeline work?  what benchmark tool are you using - will it support noauth - or would tempauth be better?18:25
vjujjuriclayg I pulled it out - Stil giving curl: (3) <url> malformed18:26
vjujjuriam I doing the url right? how do I get the list of accounts and container rings?18:26
*** silor has joined #openstack-swift18:27
vjujjuriThis is what I tried:18:27
vjujjurivjujjuri@twinstan-wsl6:~$ curl -X PUT /tmp/ha http://127.0.0.1:8080/v1/abc18:27
claygvjujjuri: oh, sorrt i missed the first part and only saw Unauthorized18:27
claygthe curl syntax to upload a file is -T18:27
claygbut yeah - you need to create a container first18:27
claygyou can list the account with a GET - just curl http://127.0.0.1:8080/v1/abc18:27
clayg^ normally it'd be /v1/AUTH_<account> - but with no-auth and autocreate account you can just hit any auth url18:28
vjujjurisurely I am doing somethig wrong18:29
vjujjurihttp://pastebin.com/jYsztAUP18:30
vjujjuriclayg: ^18:30
claygheh, the last one "curl  http://127.0.0.1:8080/v1/abc" was the only one that would have worked - but without a -v you wouldn't have realized it returned an empty body18:31
claygvjujjuri: the suggestion to "try 'curl --help' or 'curl --manual' for more information" is probably good avice18:32
*** NM has joined #openstack-swift18:35
*** aix has quit IRC18:47
*** legionarius has joined #openstack-swift18:48
*** ho has joined #openstack-swift18:49
hogood morning!18:53
*** aerwin has joined #openstack-swift18:53
*** cutforth has joined #openstack-swift18:54
*** acoles_away is now known as acoles18:57
InAnimaTesigh...18:57
InAnimaTeanyone ever dealt with the majority of their disks being at 100%18:58
mattoliveraumorning18:59
*** dionkalo has joined #openstack-swift18:59
claygInAnimaTe: :'(  sorry bro18:59
notmynamemeeting time18:59
*** annegentle has joined #openstack-swift19:00
*** kota_ has joined #openstack-swift19:02
glangeInAnimaTe: how did that happen?19:11
*** legionarius has quit IRC19:14
*** zaitcev has joined #openstack-swift19:15
*** ChanServ sets mode: +v zaitcev19:15
*** thurloat_isgone is now known as thurloat19:17
*** B4rker has joined #openstack-swift19:18
InAnimaTeglange: mistreatment by other people19:22
InAnimaTeso i have this idea19:22
ahaleare you out of container/account or just object ?19:25
*** dionkalo has left #openstack-swift19:25
InAnimaTeall of them19:26
InAnimaTethe logs are full of database errors19:26
InAnimaTersync disk is full errors, etc..19:26
*** dk2 has joined #openstack-swift19:26
InAnimaTebasically, im thinking if i can remove some containers across the drives, it might free up enough space for things to work again19:27
InAnimaTe(we have 3x replication)19:27
ahaleouch !19:27
*** vjujjuri has quit IRC19:28
InAnimaTeheres the fun part, mgmt has told me i cant change the replication factor19:28
InAnimaTeso yeah...having loads of fun with this19:28
torgomatichttp://nadreck.me/trunk/wp-content/uploads/2015/02/fine.png19:28
torgomaticcan you get more disks?19:28
InAnimaTeso ive already incorporated in another partition (that is a teeny bit smaller) to each box in each zone19:30
*** tdasilva has quit IRC19:30
InAnimaTeits very very slowly being incorporated in19:32
InAnimaTeyeah the way it stores objects raw on each disk is a little weird19:33
InAnimaTenot sure what the "100000" or "9993" directories are exactly for19:33
*** lpabon has quit IRC19:34
glangethose are partition numbers19:34
glangethe ring assigns partitions to devices19:35
glangethat determines where objects go19:35
glangewhich partition they hash to19:35
*** mahatic has quit IRC19:38
*** mahatic has joined #openstack-swift19:38
*** blmartin has joined #openstack-swift19:43
claygInAnimaTe: https://answers.launchpad.net/swift/+question/150077 has some info - there's a bug too19:43
notmynameacoles: seems the answer is "not really".19:44
notmynamehttp://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/gerrit.pp#n142 is what's configured19:44
notmynameso it might be possible19:44
notmynamealso, "isn't that what blueprints are for" is one answer I got19:45
acolesnotmyname: using blueprints as a 'hub' is one way but...yet another place to go manage info :/19:46
notmynameone of the main reasons I don't like it19:47
acolesand blueprints don't issue notifications afaik. lets not go there.19:48
notmynameoh I know :-)19:48
*** B4rker has quit IRC19:49
acolesi wonder if the dashboard url can select on pattern match in a commit message?19:49
notmynamenot that I've seen19:50
notmynamebut the dashboard can be built from anything that works in the search field19:50
notmyname(which is a whole 'nother ball of fun)19:50
*** dk2 has quit IRC19:50
acoleshmm. even manually including a changeid link to the spec review isn't great because the gate doc build on the review tends to expire, so you still don't get to the html19:51
claygInAnimaTe: this was it!  https://bugs.launchpad.net/swift/+bug/135916019:51
openstackLaunchpad bug 1359160 in OpenStack Object Storage (swift) "Problem with disk overflowing" [Undecided,Expired]19:51
*** erlon has quit IRC19:51
*** gvernik has quit IRC19:52
*** tdasilva has joined #openstack-swift19:53
*** dencaval has quit IRC19:54
*** acoles is now known as acoles_away19:58
*** tdasilva has quit IRC20:00
*** delattec has quit IRC20:08
openstackgerritKota Tsuyuzaki proposed openstack/swift: Remove confusable query string on post as copy  https://review.openstack.org/17892720:10
*** esker has quit IRC20:16
*** tdasilva has joined #openstack-swift20:17
*** tdasilva is now known as tdasilva_20:17
InAnimaTehmm interesting20:18
InAnimaTeglange: ok i see20:18
*** tdasilva has joined #openstack-swift20:19
claygInAnimaTe: are your containers and objects on the the same devices?20:19
InAnimaTeyes20:20
claygInAnimaTe: haven't you been through something like this before?20:20
InAnimaTenope20:20
InAnimaTewell20:20
InAnimaTekind of20:20
InAnimaTewe never actually hit 100%20:20
openstackgerritThiago da Silva proposed openstack/swift: move replication code to ReplicatedObjectController  https://review.openstack.org/18282620:20
claygoh20:20
InAnimaTewe were at like 96% etc..20:20
claygso you're like paving new ground here20:21
claygyou're going to be able to write the book on unscrewing full swift clusters20:21
glangewhat would be the title of that book? :)20:21
glange"Not So Swift"? :)20:21
claygglange: ummm  "unscrewing full swift clusters for fun and terror"?20:22
glangedude, that is terrible :)20:22
clayglol20:22
claygglange: you sure you're not coming to vancouver?20:22
glangeyeah, we have it covered20:22
glange"My Big Fat Swift Cluster"20:22
InAnimaTerofl20:23
InAnimaTeso much inspiration in this chan20:23
glangehaha, sorry, we feel bad for you20:23
claygInAnimaTe: so step 1 is normally find some disks - you got that part covered?20:23
glangethe only way to fix things is more capacity or deleting stuff20:23
glangeyou cat put more in the bag than it will hold20:23
glangecan't20:23
*** kota_ has quit IRC20:24
claygglange: deleting stuff is a write operation20:24
glangenot if he does it from the command line :)20:24
claygso.... the only way to fix things is more capacity is good enough20:24
InAnimaTeyeah as i was about to say, deletes are failing hard20:24
claygglange: i have no idea the joke you're trying to spin - if you're just blowing aways stuff on the file system swift is going to fight you every step of the way20:25
glangeit will replicate back for sure20:25
claygInAnimaTe: Step 1 - DISKS!  go find them - the fire will wait20:25
glangeso, yeah, I guess I see what you mean20:25
claygor at least it can't get any bigger20:25
glangeyou could turn off replication, carefully delete some different partitions from each node, then you should be able to do deletes through swift?20:26
*** thurloat is now known as thurloat_isgone20:26
InAnimaTe^so yeah, thats the basically the idea i had20:27
InAnimaTemanually go in and remove objects i know are replicated in other zones20:27
InAnimaTeclayg: can't spend money :(20:28
claygglange: so like reduce the replication factor on some partitions by hand long enough to get some tombstones to free up space - then when you turn the replicators on bring the parts back to full durability will have enough room.... I've never tried that20:28
claygmaybe as long as you're really careful not to delete more than two replicas of a part - it could work - untested20:29
glangeyeah, there is some chance of data loss20:30
clayger... more than one - don't delete two - if the cluster is full you may not have 3 of everything - you may not have 2 of everything :\20:30
glangeyeah20:30
glangeclayg: you don't have a script for this? :)20:32
claygidk, if you could swap in a spare device (keep the full one around for later) at the same time you reduce the replica count in the ring - I think it work out to the same thing - with less change of stuff getting lost20:33
*** annegentle has quit IRC20:39
*** annegentle has joined #openstack-swift20:39
*** vjujjuri_ has joined #openstack-swift20:40
ahaleyou;ll need to be careful not to delete asncs if you're deleting stuff or wiping drives20:40
*** vinsh_ has joined #openstack-swift20:43
*** vjujjuri_ is now known as vjujjuri20:44
*** vinsh has quit IRC20:46
dfg_all of this sounds crazy20:48
*** B4rker has joined #openstack-swift20:48
claygahale: a's and c's - nice20:50
ahaleah yeah asyncs even20:51
claygahale: oh - i thought it was some cute abbreviation for accounts and containers - yeah asyncs - also a good point!20:51
ahalei like that my terrible typing can be mistaken for coolness20:53
claygInAnimaTe: you should just add more disks - maybe you could start a kickstarter or something "overstressed underfunded ops - needs hard disks"20:53
claygahale: IME ahale is basically a synonym for cool - everyone's going to be calling them asncs now20:53
*** silor has quit IRC20:54
*** vinsh has joined #openstack-swift20:59
*** vinsh_ has quit IRC21:03
*** B4rker has quit IRC21:06
*** Dieterbe_ has joined #openstack-swift21:07
MooingLemureh syncs21:07
*** anderstj has quit IRC21:09
*** ekarlso has quit IRC21:09
*** ndk_ has quit IRC21:09
*** wolsen has quit IRC21:09
*** tanee has quit IRC21:09
*** dmsimard has quit IRC21:09
*** ctennis has quit IRC21:09
*** Dieterbe has quit IRC21:09
*** ctennis has joined #openstack-swift21:10
*** wolsen has joined #openstack-swift21:10
*** ekarlso has joined #openstack-swift21:10
*** B4rker has joined #openstack-swift21:10
*** tanee has joined #openstack-swift21:11
*** dmsimard has joined #openstack-swift21:11
*** ndk has joined #openstack-swift21:11
*** anderstj has joined #openstack-swift21:11
*** ctennis has quit IRC21:12
*** ctennis has joined #openstack-swift21:12
*** haomaiwa_ has quit IRC21:21
claygis "invalid" reall the best status for dupe in launchpad?21:22
notmynameclayg: unfortunately :-(21:22
*** B4rker has quit IRC21:23
*** aerwin has quit IRC21:32
*** annegentle has quit IRC21:32
*** vjujjuri has quit IRC21:34
clayghey mattoliverau and peluse - looking at rsync compression - i see a for node in get_more_nodes loop that I think is exahusting every db21:34
openstackgerritSamuel Merritt proposed openstack/swift-specs: Symlinks in Swift  https://review.openstack.org/17360921:39
*** vjujjuri has joined #openstack-swift21:39
claygI think instead of all the plumbing on "different_region" kwarg just to get the optional compression flag down there - we should just get a _local_region(s) attribute on the DBReplicator instance right next to where we build up the _local_device_ids set21:39
*** thurloat_isgone is now known as thurloat21:52
*** jrichli has quit IRC21:52
*** proteusguy has quit IRC22:05
*** vjujjuri has quit IRC22:05
*** NM1 has joined #openstack-swift22:08
*** vinsh has quit IRC22:08
*** nadeem has quit IRC22:08
*** NM has quit IRC22:08
*** NM1 has quit IRC22:11
mattoliverauMorning again22:13
*** omame has quit IRC22:18
*** jamielennox|away is now known as jamielennox22:19
*** wbhuber has quit IRC22:22
homattoliverau: morning!22:23
*** asettle has joined #openstack-swift22:32
*** blmartin has quit IRC22:33
*** ekarlso has quit IRC22:34
*** ekarlso has joined #openstack-swift22:34
openstackgerritHisashi Osanai proposed openstack/swift: Enable Object Replicator's failure count in recon  https://review.openstack.org/13834222:34
*** ChanServ sets mode: +v mattoliverau22:40
*** omame has joined #openstack-swift22:40
-openstackstatus- NOTICE: Gerrit and Zuul are going offline for reboots to fix a security vulnerability.22:41
*** ChanServ changes topic to "Gerrit and Zuul are going offline for reboots to fix a security vulnerability."22:41
notmynamehttps://www.surveymonkey.com/s/77H6SSL <-- a small survey about openstack that will be used in a talk next week (not by me)22:42
claygnotmyname: i can think back that far - how many months is "all of them"22:43
claygnotmyname: nm - it lets me type in "all of them"22:45
*** openstackgerrit has quit IRC22:47
*** openstackgerrit has joined #openstack-swift22:49
claygI'm not going to submit this - "throw away" != "re-architect"22:53
claygbad tech orgs will often fail trying to 'throw away' a successful system - bad tech orgs will often fail trying 'to keep the wheels on' a successful system that's outgrown it's design -  obviously continuous re-architecture of the parts of the system that need it the only path to success; but I'm not going to help some one argue some portion of OpenStack needs to be throw out and redone from scratch?22:56
*** ChanServ changes topic to "Review Dashboard: http://goo.gl/8IUcKl | Summary Dashboard: http://goo.gl/qHus5v | Summit scheduling: https://etherpad.openstack.org/p/liberty-swift-summit-topics | Logs: http://eavesdrop.openstack.org/irclogs/%23openstack-swift/"22:59
-openstackstatus- NOTICE: Gerrit and Zuul are back online.22:59
*** km has joined #openstack-swift22:59
notmynameclayg: I first did "all of them" then it did an int validation23:03
*** cutforth has quit IRC23:03
*** sc has quit IRC23:20
zackmdavisclayg, can you protest by submitting "Other" for all the throw-out/rearchitecture question?23:20
claygno23:21
*** sc has joined #openstack-swift23:22
*** ho has quit IRC23:28
*** kei_yama has joined #openstack-swift23:28
*** zhill has joined #openstack-swift23:33
*** mahatic has quit IRC23:42
*** ho has joined #openstack-swift23:44
openstackgerritJohn Dickinson proposed openstack/python-swiftclient: be more explicit in the --version CLI option  https://review.openstack.org/18289223:46
*** vjujjuri has joined #openstack-swift23:49
notmynamehttp://ibmresearchnews.blogspot.com/2015/05/deep-dive-insights-into-swift.html  <-- teaser/preview of a swift talk IBM is giving next week23:49

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!