Thursday, 2015-05-14

*** gyee has quit IRC00:14
zaitcevcurious00:19
*** zhill has quit IRC00:26
*** dmorita has joined #openstack-swift00:35
*** jrichli has joined #openstack-swift00:37
*** tobe4333 has joined #openstack-swift00:45
*** thurloat is now known as thurloat_isgone00:51
*** vjujjuri has quit IRC01:03
*** bill_az has quit IRC01:30
*** asettle has quit IRC01:38
*** asettle has joined #openstack-swift01:50
*** wbhuber has joined #openstack-swift01:52
*** B4rker has joined #openstack-swift01:59
*** B4rker has quit IRC02:03
*** tobe4333 has quit IRC02:11
openstackgerritJanie Richling proposed openstack/swift-specs: Updates to encryption spec  https://review.openstack.org/15431802:17
*** minwoob has quit IRC02:22
*** minwoob has joined #openstack-swift02:22
*** bkopilov has quit IRC02:34
*** jeblair has quit IRC02:36
*** jeblair has joined #openstack-swift02:36
*** jrichli has quit IRC03:13
*** wbhuber has quit IRC03:23
*** minwoob has quit IRC03:36
*** ho has quit IRC03:41
*** bkopilov has joined #openstack-swift04:04
*** ppai has joined #openstack-swift04:36
*** remix_tj has quit IRC04:57
*** remix_tj has joined #openstack-swift04:57
*** zhill has joined #openstack-swift05:01
*** zhill has quit IRC05:04
*** SkyRocknRoll has joined #openstack-swift05:26
*** zaitcev has quit IRC05:33
*** silor has joined #openstack-swift07:10
*** jistr has joined #openstack-swift07:34
*** links has joined #openstack-swift08:02
*** acoles_away is now known as acoles08:27
*** mkerrin has joined #openstack-swift09:05
*** silor has quit IRC09:40
*** dmorita has quit IRC09:57
*** aix has joined #openstack-swift10:20
*** ppai has quit IRC10:39
*** tdasilva_ has quit IRC10:41
*** aix has quit IRC10:42
*** haomaiwang has joined #openstack-swift10:42
*** haomaiwa_ has joined #openstack-swift10:45
*** haomaiwang has quit IRC10:45
*** ppai has joined #openstack-swift10:52
*** tanee has quit IRC10:54
*** tanee has joined #openstack-swift10:54
*** jamielennox is now known as jamielennox|away10:58
*** ppai has quit IRC10:58
*** ppai has joined #openstack-swift11:11
*** haomaiwa_ has quit IRC11:15
*** cdelatte has joined #openstack-swift11:15
*** delattec has joined #openstack-swift11:15
*** chlong has joined #openstack-swift11:48
openstackgerritAlistair Coles proposed openstack/swift-specs: Add keymaster discussion to encryption spec  https://review.openstack.org/18301411:54
*** aix has joined #openstack-swift11:58
*** bkopilov has quit IRC12:03
*** kei_yama has quit IRC12:08
*** km has quit IRC12:14
*** silor has joined #openstack-swift12:16
*** erlon has joined #openstack-swift12:38
acolestdasilva: thanks for reviewing that ^^ , i thought it might be a good idea to start a separate thread for keymaster discussion ;)12:45
tdasilvaacoles: yeah, I like it how you stated that the keymaster details is out of the scope of this doc, but it still provides some guidance on the expectation of how things would work out12:47
acolestdasilva: well, folks like you and ho were raising good questions about it so i thought sharing some thinking was useful12:48
openstackgerritAlistair Coles proposed openstack/swift-specs: Updates to encryption spec  https://review.openstack.org/15431812:50
*** lpabon has joined #openstack-swift12:53
*** ppai has quit IRC13:05
openstackgerritStuart McLaren proposed openstack/python-swiftclient: Add minimal working service token support.  https://review.openstack.org/18264013:07
*** B4rker has joined #openstack-swift13:09
*** annegentle has joined #openstack-swift13:10
*** SkyRocknRoll has quit IRC13:13
*** ppai has joined #openstack-swift13:16
openstackgerritpradeep kumar singh proposed openstack/swift: Swift account auditor fails to quarantine corrupt db due to Disk IO error. This patch fix that by handling Disk IO Error Exception.  https://review.openstack.org/18273413:23
*** ppai has quit IRC13:23
*** fthiagogv has joined #openstack-swift13:25
*** bkopilov has joined #openstack-swift13:33
*** lastops has joined #openstack-swift13:33
*** jrichli has joined #openstack-swift13:35
*** acoles is now known as acoles_away13:35
lastopsgood morning. I’m having troulbes fidning any information on metadata search api (https://wiki.openstack.org/wiki/MetadataSearchAPI) development status . Did anyone have to solve this problem (indexing, searh) with middleware? Thanks in advance13:38
tdasilvalastops: hi, there's a spec for review here: https://review.openstack.org/#/c/180918/13:39
tdasilvalastops: I don't think there's been any public code up for review yet13:40
lastopstdasilva: thanks.13:43
lastopsso the way to go about this for now is to intercept reqests and send them to external search engine13:44
glangeyou'd probably have to intercept incoming requests to update the indexes of the search engine too13:45
glangeprobably in middleware for both13:46
lastopsglange: yes. thanks. will give it a try.13:50
*** wbhuber has joined #openstack-swift13:51
glangelastops: hang around channel, there might be something working on this already -- it's still kind of early for most people -- you might get more answers/information later13:51
glanges/something/somebody/13:51
lastopsglange: ok. Thanks.13:52
*** mahatic has joined #openstack-swift13:53
*** wbhuber has quit IRC13:59
*** wbhuber has joined #openstack-swift14:14
wbhubergood morning14:17
glangeit is a fine morning!14:17
openstackgerritpradeep kumar singh proposed openstack/swift: Swift account auditor fails to quarantine corrupt db due to Disk IO error. This patch fix that by handling Disk IO Error Exception.  https://review.openstack.org/18273414:17
wbhubernotmyname: have you sketched performance test framework for EC yet?  anything documented for it?14:17
openstackgerritpradeep kumar singh proposed openstack/swift: Swift account auditor fails to quarantine corrupt db due to Disk IO error. This patch fix that by handling Disk IO Error Exception.  https://review.openstack.org/18273414:21
*** breitz has quit IRC14:24
*** breitz has joined #openstack-swift14:24
*** mkerrin has quit IRC14:27
*** acoles_away is now known as acoles14:28
*** shakamunyi has joined #openstack-swift14:29
*** jodah has quit IRC14:37
*** jodah has joined #openstack-swift14:39
*** mordred has quit IRC14:40
*** silor has quit IRC14:40
*** silor has joined #openstack-swift14:40
*** mordred has joined #openstack-swift14:42
*** barra204 has joined #openstack-swift14:43
openstackgerritpradeep kumar singh proposed openstack/swift: Swift account auditor fails to quarantine corrupt db due to Disk IO error. This patch fix that by handling Disk IO Error Exception.  https://review.openstack.org/18273414:46
openstackgerritpradeep kumar singh proposed openstack/swift: Swift account auditor fails to quarantine corrupt db due to Disk IO error. This patch fix that by handling Disk IO Error Exception.  https://review.openstack.org/18273414:49
*** minwoob has joined #openstack-swift14:52
*** links has quit IRC15:24
*** mahatic has quit IRC15:25
*** ajiang has joined #openstack-swift15:26
*** gsilvis_ is now known as gsilvis15:27
InAnimaTegah this sucks15:28
*** haomaiwang has joined #openstack-swift15:28
*** dencaval has joined #openstack-swift15:29
*** haomaiwang has quit IRC15:29
*** haomaiwang has joined #openstack-swift15:29
*** mahatic has joined #openstack-swift15:30
*** nadeem has joined #openstack-swift15:38
*** vjujjuri has joined #openstack-swift15:45
*** gyee has joined #openstack-swift15:51
openstackgerritDenis Cavalcante proposed openstack/swift-specs: Internal communication measurement for swift spec  https://review.openstack.org/18080116:01
*** B4rker has quit IRC16:09
claygpython3 has only been out five years?16:17
clayg... oh it's more like 8 or 916:19
zackmdavisclayg, (well, six years and five months)16:19
claygPython 3.0.0 2008-12-0316:20
claygthat's from python.org/downloads16:20
zackmdavisclayg, right http://www.timeanddate.com/date/durationresult.html?m1=12&d1=03&y1=2008&m2=5&d2=14&y2=201516:21
claygi'm trying to find out when six was released16:23
claygI want to say it wasn't until like python 3.3 when you could acctually run a python code base that would work on either 2 or 316:23
claygeverything before that was possible was really just an early release preview16:23
zackmdavisclayg, looking at the six source repo, the 1.0.0 tag is dated March 2011 https://bitbucket.org/gutworth/six/commits/tag/1.0.016:24
zackmdavisbut maybe people were using it before then, version numbers being meaningless16:26
*** shakamunyi has quit IRC16:27
*** barra204 has quit IRC16:28
*** zaitcev has joined #openstack-swift16:29
*** ChanServ sets mode: +v zaitcev16:29
clayghttp://python-future.org/faq.html claims to be a better six and requires 3.3+16:30
claygsomething must have come in with 3.316:30
claygPython 3.3.0 2012-09-2916:30
zackmdavisclayg, (re something coming in with 3.3) maybe the no-op unicode literals? https://www.python.org/dev/peps/pep-0414/#proposal16:33
claygyup gotta be16:35
claygI wonder if python decided to do a python 2.8 with python-futures built in and a bunch of deprecation warnings if you get a smooth transistion from python 2.7 code works, and runs unmodified on 2.8, then after fixing all the deprecation warnings runs unmodified on 3.4 (or 3.5)16:38
notmynamegood morning16:39
*** jistr has quit IRC16:42
*** vinsh has joined #openstack-swift16:51
MooingLemurso an EC ring just has data+parity chunk "replicas", right?17:08
MooingLemurso if I want 8+3, I create a ring with 11 replicas?17:08
torgomaticMooingLemur: correct17:14
MooingLemurneat, seems straightforward :D17:15
*** gyee has quit IRC17:16
MooingLemurhmm, swift-ring-builder in 2.3.0 doesn't like adding devices with a weight of zero.  I can set it to something else and then change it to zero... how silly17:17
torgomaticMooingLemur: why would you want to add a device with weight 0? what purpose would it serve?17:26
MooingLemurtorgomatic: one instance I can think of there was a device that was mistakenly removed from the ring, but then you want to add it back with weight 0 so that replication will push data off of it17:27
torgomaticMooingLemur: ah, that would do it, yes17:28
*** zhill has joined #openstack-swift17:28
claygtorgomatic: lol17:28
claygdid we *used* to support adding devices with zero weight?17:28
MooingLemurtorgomatic: but as of now, on the prod cluster, if a new disk is found, the automation adds it with weight 0 on the first pass, then bumped up slowly toward the target weight on subsequent passes17:28
torgomaticclayg: well, it seemed dumb to me, but that's legit :D17:28
MooingLemursure :)17:28
torgomaticclayg: I have no idea17:29
MooingLemurmost definitely it works.17:29
MooingLemurthen again, I'm still using a really old swift-ring-builder on the machine that does the automation17:32
lastopsHello. As mentioned earlier today (a bit too early), I need to solve metadata indexing/search problem and while it looks like Metadata Search API (https://wiki.openstack.org/wiki/MetadataSearchAPI) is still far from implementation, I'm certain that the problem is common and had been solved with home-made middleware. Could anyone give me some hints? Thanks in advance.17:32
zaitcevI suspect people who need search and indexes select an architecture with a database for it. Right tool for the job and all that.17:35
notmynamelastops: yeah, softlayer has something like that. you might take a look at their object storage docs to see what their API is17:38
MooingLemurwhen adding a new storage policy, do the daemons need restarting because of a new swift.conf?17:39
lastopszaitcev, notmyname: thanks17:40
MooingLemurI guess that's not too bad of a penalty, since it doesn't happen every day17:41
*** harlowja has quit IRC17:41
*** harlowja has joined #openstack-swift17:42
lastopsHP folks also worked on this https://wiki.openstack.org/w/images/a/ab/Object_Storage_Metadata_Search_Design_Session_v4.pdf17:44
*** thurloat_isgone is now known as thurloat17:49
*** thurloat is now known as thurloat_isgone17:50
redboI had a proof of concept pushing metadata to the container with the rest of the object data and using sqlite's fts to index it.  It works okay, but bloats the container dbs.17:56
annegentleanyone know of a public cloud that implements the endpoints middleware? (Checked Rackspace and TryStack, no go)17:57
annegentleI'm working on API reference for https://blueprints.launchpad.net/swift/+spec/list-endpoints17:58
annegentle(I can spin up a devstack with it configured if need be)17:58
torgomaticannegentle: I'd be really surprised if you found any. It's only useful for people who fully trust their clients, so much so that the clients get to bypass auth and talk directly to the storage nodes18:02
annegentletorgomatic: hm, perhaps it's purposefully underdocumented18:02
torgomaticannegentle: could be; it's really quite a minimal API. It's pretty much just one GET request that responds with some JSON.18:03
swifterdarrellannegentle: torgomatic: yup (other than several "SECURITY WARNING: probably do not deploy this" type things, hopefully); it was added so trusted Hadoop nodes could locate and access objects directly against the object-servers, bypassing /all/ authentication18:03
annegentletorgomatic: sure, but I can't "see" the JSON returned unless I run the call18:03
annegentleunless there's something in the code that can help me glue together the json returned?18:04
redbohttps://github.com/openstack/swift/blob/master/swift/common/middleware/list_endpoints.py#L5518:07
swifterdarrellannegentle: here ya go https://gist.github.com/dbishop/32c6ff6bfb74cbf8c4f118:08
annegentlethanks redbo swifterdarrell! redbo wonder if that string should be updated with the headers:?18:08
annegentleredbo: or is that optional (to apply headers)18:08
redboIt's not optional, it needs that storage policy to know where the object lives on disk.18:10
annegentleredbo: huh, swifterdarrell's gist doesn't have headers (oh wait, the headers are on the request, got it)18:11
annegentleherpderp gotit18:11
swifterdarrellannegentle: one sec18:11
*** lpabon has quit IRC18:11
swifterdarrellannegentle: reload the gist18:12
redbothere's a v1 and v2 of that API18:12
swifterdarrellannegentle: now includes v2 examples18:12
annegentleswifterdarrell: Sweet18:12
swifterdarrellredbo: thx for the pointer! I hadn't seen the v2 biz w/the endpoints middleware18:12
redboI hadn't either!  Shows how much attention I pay.18:13
swifterdarrellredbo: heh18:13
wbhubernotmyname: have you extracted net/cpu/mem util for the latest EC performance run on 5/11?  are the logs, configs and rings accessible for the run on github??18:14
notmynamenot yet18:14
*** gyee has joined #openstack-swift18:19
wbhuberin the average latency column, i assume we're speaking seconds, right?18:21
wbhubernotmyname: ^^18:22
notmynameyes18:22
notmynamescotticus: FYI, I just got this email "...you will present from your own computer so please come prepared with your VGA or DVI compatible laptop with your 16:9 presentations..."18:23
*** vinsh has quit IRC18:28
wbhubernotmyname: care to elaborate a bit more on reqs_sec.  requests per seconds on storing an object?18:30
notmynameyes18:30
notmynameor more generally, just <total requests> / <wall clock time>18:30
wbhuberSo, in the miniscule test for EC isolation, we have 11800 objects to store and it takes 97 seconds to make a request?  am i thinking unclear?18:31
notmynameI'd need to find that gist again ;-)18:32
wbhuberand how different is ec-shared from replica-shared?  both has 10 EC drives and 10 Replica drives?18:32
notmyname"shared" means that the drives used (20 total) are used for both replica and EC policies18:33
notmynamewbhuber: can you give me the link again please?18:33
wbhuberhttps://gist.githubusercontent.com/charz/77dd1c3f73b228bdd6a1/raw/393e72ce70b24f960f61bfce4608b578d1767136/gistfile1.txt18:33
notmynameso the first row?18:34
wbhuberyes18:34
notmyname120 seconds to do operations on 11800 objects, averages of 97 req/sec18:34
notmynamethat test has a mix of read and writes. the read_only and write_only are what you'd expect18:35
*** vinsh has joined #openstack-swift18:36
wbhuberok, makes sense now.18:36
wbhuberi see that there is no delete test scenario yet implemented.  we can do that here when we have configuration/hardware established18:36
notmynameand "minuscule" is a objects between 10 bytes and 2048 bytes. note that the EC strip size is 1MB. so totally not what you should use for EC ;-)18:37
wbhuberi see that, definitely.18:37
notmynamewbhuber: yeah, that would be ok, but deletes for EC objects in swift aren't really implemented any differently than for replicas. so because of that, and because I wanted to fill the cluster over time, I didn't do any deletes18:37
*** aix has quit IRC18:38
wbhuberi better head back and read the configuration and understand the definition of ec_shared and replica_shared more18:38
wbhuberisolation ones are well defined18:38
openstackgerritEran Rom proposed openstack/swift-specs: Swift-Storlets - allowing dynamic compute functions  https://review.openstack.org/16453718:39
*** vinsh has quit IRC18:40
*** atan8 has joined #openstack-swift18:43
*** fthiagogv has quit IRC18:47
*** ajiang_ has joined #openstack-swift18:50
*** ajiang has quit IRC18:50
*** ajiang_ has quit IRC18:53
*** ajiang has joined #openstack-swift18:54
*** silor has quit IRC18:54
*** ajiang has quit IRC18:57
*** ajiang has joined #openstack-swift18:58
*** ajiang has left #openstack-swift19:01
*** ajiang has joined #openstack-swift19:03
*** harlowja has quit IRC19:03
*** harlowja has joined #openstack-swift19:05
*** ajiang has left #openstack-swift19:06
*** ajiang has joined #openstack-swift19:06
*** acoles is now known as acoles_away19:09
*** annegentle has quit IRC19:17
*** harlowja has quit IRC19:25
*** harlowja has joined #openstack-swift19:26
*** lastops has quit IRC19:31
openstackgerritMerged openstack/swift: go: clean up and add to obj tests  https://review.openstack.org/18206519:35
*** vinsh has joined #openstack-swift19:43
*** mwheckmann has quit IRC19:45
*** openstackgerrit has quit IRC19:52
*** openstackgerrit has joined #openstack-swift19:52
*** vinsh has quit IRC19:54
*** annegentle has joined #openstack-swift20:00
*** dencaval has quit IRC20:03
*** wbhuber has quit IRC20:08
*** wasmum has quit IRC20:09
*** cdelatte has quit IRC20:10
*** delattec has quit IRC20:10
*** lcurtis has joined #openstack-swift20:41
lcurtishello all...I just added another container node to swift cluster and rsyncs seem to be taking a very long time20:42
lcurtisor rather..not much throughput20:43
lcurtisany tips or tricks?20:43
*** annegentle has quit IRC20:47
ctenniscan you quantify it a bit?20:49
*** wbhuber has joined #openstack-swift20:52
*** zhill_ has joined #openstack-swift20:59
lcurtisctennis averaging 2-3Mbps21:00
ctennisthe replicators output some statistics in the logs every 5 minutes, might look at that to see partitions21:01
ctennisrsync will run as fast as the network, cpu, and disk allow21:01
lcurtishmmm...interesting21:01
lcurtisany keywords i can look for in logs?21:01
ctennisthis is specifcally containers?21:02
ctennisprobably grep for container-replicator, should be able to narrow it down from there21:02
ctennisotherwise grep for object-replicator and then grep again for "partitions"21:03
lcurtisyes containers21:05
*** mwheckmann has joined #openstack-swift21:08
lcurtisokay....thank you as always for your sage advice ctennis21:08
*** haomaiwang has quit IRC21:08
*** chlong has quit IRC21:08
*** haomaiwa_ has joined #openstack-swift21:09
*** openstackgerrit has quit IRC21:22
*** openstackgerrit has joined #openstack-swift21:22
*** jrichli has quit IRC21:22
*** wasmum has joined #openstack-swift21:28
*** annegentle has joined #openstack-swift21:28
*** erlon has quit IRC21:41
*** mahatic has quit IRC21:57
*** wasmum has quit IRC22:07
*** cdelatte has joined #openstack-swift22:17
mattoliverauMorning22:19
*** annegentle has quit IRC22:20
*** bhakta has joined #openstack-swift22:20
*** bhakta has left #openstack-swift22:21
*** bhakta has joined #openstack-swift22:21
*** zhill_ has quit IRC22:38
*** nadeem has quit IRC22:38
openstackgerritMichael Barton proposed openstack/swift: go: lower-case incoming ETag  https://review.openstack.org/18334022:39
*** rsFF has quit IRC22:39
*** rsFF has joined #openstack-swift22:40
*** annegentle has joined #openstack-swift22:45
*** annegentle has quit IRC22:47
*** shakamunyi has joined #openstack-swift22:47
*** barra204 has joined #openstack-swift22:47
*** wasmum has joined #openstack-swift22:50
*** vinsh has joined #openstack-swift22:55
*** vinsh has quit IRC22:59
*** barra204 has quit IRC23:04
*** shakamunyi has quit IRC23:04
*** Trozz has quit IRC23:04
*** chlong has joined #openstack-swift23:10
*** mwheckmann has quit IRC23:11
*** km has joined #openstack-swift23:14
*** atan8 has quit IRC23:16
*** shakamunyi has joined #openstack-swift23:17
*** barra204 has joined #openstack-swift23:17
*** atan8 has joined #openstack-swift23:26
*** lcurtis has quit IRC23:26
*** atan8 has quit IRC23:26
*** kei_yama has joined #openstack-swift23:36
*** ho has joined #openstack-swift23:49

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!