Tuesday, 2014-09-16

torgomaticnotmyname: PUT calls broker.put_record(), which only sometimes calls _commit_puts() depending on file size00:00
torgomaticGET calls list_objects_iter calls _commit_puts_stale_ok, which tries to flush but doesn't complain if it can't get the lock00:01
notmynametorgomatic: I don't know about your fancy "this is what the code does"; I'm just going off of what I saw on my SAIO ;-)00:02
torgomaticlol00:03
mattoliveraukeep it up guys, I'm learning heaps without having to read the code :)00:03
*** occupant has quit IRC00:04
*** occupant has joined #openstack-swift00:09
zaitcevsphinx.errors.ExtensionError: Could not import extension oslosphinx (exception: No module named oslosphinx) -- god dammit00:21
zaitcevSwift succumbed too00:21
zaitcevdoc/source/conf.py:              'sphinx.ext.ifconfig', 'oslosphinx']00:22
*** echevemaster has joined #openstack-swift00:28
*** dmorita has joined #openstack-swift00:29
*** rmcall has quit IRC00:30
*** kyles_ne has quit IRC00:44
*** tdasilva has joined #openstack-swift00:46
*** mitz_ has joined #openstack-swift00:46
*** StevenK_ is now known as StevenK00:51
*** conor has joined #openstack-swift00:51
claygzaitcev: mfw oslo -> http://memegenerator.net/instance/5437044000:56
openstackgerritClay Gerrard proposed a change to openstack/swift: wip: make /info configurable  https://review.openstack.org/12172301:01
claygumm... how do you mark changes wip now?01:02
claygnm, review, workflow, wip -101:04
*** addnull has joined #openstack-swift01:04
*** conor has quit IRC01:21
openstackgerritClay Gerrard proposed a change to openstack/swift: Let eventlet.wsgi.server log tracebacks when eventlet_debug is enabled  https://review.openstack.org/10591801:23
claygman I'm loving the abandon change alert01:23
*** gyee has quit IRC01:27
*** miqui has quit IRC01:29
mattoliverauclayg: glad your finding it helpful :)01:32
*** occupant has quit IRC01:37
*** occupant has joined #openstack-swift01:37
openstackgerritThiago da Silva proposed a change to openstack/swift: adding same cluster container migration driver  https://review.openstack.org/12172801:39
*** tdasilva has quit IRC01:59
*** haomaiw__ has quit IRC02:16
*** haomaiwang has joined #openstack-swift02:17
*** IgnacioCorderi has quit IRC02:26
*** haomaiw__ has joined #openstack-swift02:33
*** haomaiw__ has quit IRC02:33
*** haomaiwang has quit IRC02:34
*** haomaiw__ has joined #openstack-swift02:34
*** haomaiwang has joined #openstack-swift02:34
*** bkopilov has quit IRC02:37
*** haomaiw__ has quit IRC02:38
*** vu has joined #openstack-swift02:40
*** vu has quit IRC02:41
*** vu has joined #openstack-swift02:41
brianclinenotmyname: torgomatic: thanks for the discussion earlier - a human being oddly found its way to my desk, so I had to tend to that03:08
brianclineI think I found part of the issue. One of the 3 nodes storing that db had an extended network partition, during which there were numerous puts, gets, and deletes03:09
brianclinethe kicker, however, is that it was that node's replication network connection that suffered the partition, not the proxy-facing network03:10
brianclineeach of the three .pending's are empty, but it seems each .db reflects a different state03:12
brianclineso I guess my next step is figuring out how to get what appears to be a three-way split to reconcile, if that's even possible03:14
briancline(I'd hate to cheat and replay the official sequence of requests)03:15
*** mrsnivvel has joined #openstack-swift03:29
*** mahatic has joined #openstack-swift03:31
*** haomaiw__ has joined #openstack-swift03:39
*** haomaiwang has quit IRC03:40
*** haomaiw__ has quit IRC03:41
*** haomaiwa_ has joined #openstack-swift03:41
*** bkopilov has joined #openstack-swift03:43
*** bkopilov has quit IRC03:48
*** haomaiwa_ has quit IRC03:50
*** haomaiwang has joined #openstack-swift03:51
*** haomaiwang has quit IRC03:58
*** bkopilov has joined #openstack-swift04:00
*** haomaiwang has joined #openstack-swift04:09
notmynamebriancline: if gholt were still around, he's who I'd ask. he's done that sort of thing for cloud files before04:14
*** oomichi has joined #openstack-swift04:16
brianclinenotmyname: does he still frequent the channel?04:22
notmynamebriancline: doubt it04:23
notmynamebriancline: maybe dfg would be able to provide some pointers on getting DBs back in sync04:24
brianclineany pointers would be great - the container and its contents aren't important as the correct state is its contents should be fully deleted, it's more that I'd like to dive as deep as necessary to at least get it to a consistent state04:32
brianclineone of those things my brain can't let go :)04:32
notmynameyup. I understand04:35
*** kyles_ne has joined #openstack-swift04:43
*** kyles_ne has quit IRC04:46
*** kyles_ne has joined #openstack-swift04:46
*** kyles_ne has quit IRC04:51
*** haomaiwang has quit IRC05:09
*** zhiyan_ is now known as zhiyan05:10
*** zaitcev has quit IRC05:22
*** echevemaster has quit IRC05:23
*** kopparam has joined #openstack-swift05:30
*** ppai has joined #openstack-swift05:35
*** RockKuo_Office has joined #openstack-swift05:46
*** bkopilov has quit IRC05:47
*** bkopilov has joined #openstack-swift06:03
*** nshaikh has joined #openstack-swift06:30
*** kopparam has quit IRC07:27
*** geaaru has joined #openstack-swift07:33
*** yuan has joined #openstack-swift07:40
mattoliverauI'm calling it a night, need to go cook some dinner. Have a great one all!07:45
*** homegrown has joined #openstack-swift08:02
*** Dafna has joined #openstack-swift08:05
*** ledeveloper has quit IRC08:16
*** cschwede has joined #openstack-swift08:17
*** aix has joined #openstack-swift08:19
*** mkollaro has joined #openstack-swift09:07
*** haomaiwang has joined #openstack-swift09:10
*** RockKuo_Office has quit IRC09:27
*** jasondotstar has quit IRC09:44
hugokuonotmyname: morning !!!10:32
*** addnull_ has joined #openstack-swift10:43
*** cschwede has quit IRC10:45
*** addnull has quit IRC10:46
*** addnull_ has quit IRC10:47
*** addnull has joined #openstack-swift10:48
*** anticw has quit IRC10:50
*** anticw has joined #openstack-swift10:50
*** addnull has quit IRC11:00
*** AbyssOne has quit IRC11:14
*** AbyssOne has joined #openstack-swift11:16
*** cschwede has joined #openstack-swift11:18
*** anticw has quit IRC11:20
*** zul has quit IRC11:21
*** zul has joined #openstack-swift11:22
*** smart_developer has quit IRC11:29
*** smart_developer has joined #openstack-swift11:30
*** mahatic has quit IRC11:35
*** dmorita has quit IRC12:08
*** nshaikh has quit IRC12:33
*** vr1 has joined #openstack-swift12:44
openstackgerritChristian Schwede proposed a change to openstack/swift: Return correct number of changed partitions  https://review.openstack.org/12071312:59
openstackgerritAlistair Coles proposed a change to openstack/swift: Early checks for data migration config and headers  https://review.openstack.org/12183113:24
acolestdasilva: ^^13:29
*** tdasilva has joined #openstack-swift13:37
*** judd7 has joined #openstack-swift13:37
*** nshaikh has joined #openstack-swift13:39
*** miqui has joined #openstack-swift13:42
*** tdasilva has quit IRC13:42
*** mkollaro has quit IRC13:43
*** r-daneel__ has joined #openstack-swift13:48
*** dmsimard_away is now known as dmsimard13:55
*** oomichi has quit IRC13:59
*** bkopilov has quit IRC14:00
*** mkollaro has joined #openstack-swift14:00
*** bkopilov has joined #openstack-swift14:14
openstackgerritDavid Goetz proposed a change to openstack/swift: Small bug with targetted reload.  https://review.openstack.org/12185114:22
*** bkopilov has quit IRC14:22
*** ZBhatti has joined #openstack-swift14:29
openstackgerritA change was merged to openstack/swift: Treat 404s as 204 on object delete in proxy  https://review.openstack.org/11412014:29
openstackgerritA change was merged to openstack/python-swiftclient: Work toward Python 3.4 support and testing  https://review.openstack.org/11880714:30
openstackgerritA change was merged to openstack/swift: Fix delete versioning objects when previous is expired  https://review.openstack.org/8820414:35
smart_developerWhat are some suggestions for tweaking the MSR register for performance optimization?14:38
*** ppai has quit IRC14:39
smart_developeror in general, what are some ways that you can tweak the kernel to optimize Swift performance?14:42
*** sandywalsh has joined #openstack-swift14:46
*** Dafna has quit IRC14:57
*** Dafna has joined #openstack-swift14:58
openstackgerritDavid Goetz proposed a change to openstack/swift: Small bug with swift-init and targetted configs  https://review.openstack.org/12185114:58
openstackgerritChristian Schwede proposed a change to openstack/swift: Fix bug in swift-ring-builder list_parts  https://review.openstack.org/12189315:01
*** homegrown has quit IRC15:04
*** ZBhatti has quit IRC15:18
smart_developerOr rather, does anyone know how to optimize the Swift configurations themselves, for performance?15:21
*** nshaikh has quit IRC15:26
*** vu has quit IRC15:37
openstackgerritChristian Schwede proposed a change to openstack/swift: Add containeralias middleware  https://review.openstack.org/6249415:38
*** ZBhatti has joined #openstack-swift15:48
*** bkopilov has joined #openstack-swift15:50
*** vr1 has quit IRC15:56
*** Trixboxer has joined #openstack-swift15:59
*** mahatic has joined #openstack-swift15:59
*** cschwede has quit IRC16:00
*** nshaikh has joined #openstack-swift16:14
*** kyles_ne has joined #openstack-swift16:15
*** mwstorer has joined #openstack-swift16:22
*** kyles_ne has quit IRC16:31
*** kyles_ne has joined #openstack-swift16:32
*** bsdkurt has quit IRC16:33
*** kyles_ne has quit IRC16:36
*** kyles_ne has joined #openstack-swift16:36
*** geaaru has quit IRC16:40
*** tdasilva has joined #openstack-swift16:46
*** zaitcev has joined #openstack-swift16:54
*** ChanServ sets mode: +v zaitcev16:54
openstackgerritSamuel Merritt proposed a change to openstack/swift: Move multipart MIME parser into utils  https://review.openstack.org/12065217:11
*** tongli has joined #openstack-swift17:14
*** nshaikh has quit IRC17:14
openstackgerritAlistair Coles proposed a change to openstack/swift: Early checks for data migration config and headers  https://review.openstack.org/12183117:21
acolestorgomatic: was that just rebase/conflict resolution to 120652 ?17:24
torgomaticacoles: yeah, there was a really easy conflict with a recent formpost fix. no other code changes.17:25
acolestorgomatic: cool17:25
*** kevinbenton has quit IRC17:27
*** jdaggett has quit IRC17:28
*** jdaggett has joined #openstack-swift17:28
tdasilvaacoles: checking 121831 now17:29
tdasilvaacoles, thanks for your comments on 12172817:29
acolestdasilva: ok i hope i didn't conflict with anything you did17:31
tdasilvaacoles: not at all...I like the direction it is going17:32
*** kevinbenton has joined #openstack-swift17:32
acolestdasilva: good :) i'm done for today but will try to look more closely at your driver tomorrow17:33
tdasilvaacoles: ok, have a good night. I hope I can clean up some stuff there and add some comments in Gil's patch, but you explained things very well17:34
*** judd7 has quit IRC17:38
*** bsdkurt has joined #openstack-swift17:41
*** aix has quit IRC17:48
openstackgerritA change was merged to openstack/swift: Allow filtering by region in swift-recon  https://review.openstack.org/12163418:02
*** nshaikh has joined #openstack-swift18:10
smart_developerIf your middleware configurations are showing up in whenever you do " curl http://swift-proxy-ip:8080/info " does that mean that the list of middleware you've added to your proxy-server.conf pipeline is working?18:28
smart_developeror do you need to further test each middleware and actually try to use each one of them?18:29
claygsmart_developer: I'd recommend testing them ;)18:44
smart_developerIs there a guide on how to test each of the Swift middleware that is available?18:45
*** NM has joined #openstack-swift18:47
*** echevemaster has joined #openstack-swift18:48
*** kevinc has joined #openstack-swift18:55
*** Trixboxer has quit IRC18:57
*** ZBhatti has quit IRC18:58
*** ZBhatti has joined #openstack-swift18:59
openstackgerritChristian Schwede proposed a change to openstack/swift: Fix bug in swift-ring-builder list_parts  https://review.openstack.org/12189319:02
*** kevinc has quit IRC19:07
*** kyles_ne has quit IRC19:14
*** kyles_ne has joined #openstack-swift19:14
*** kyles_ne has quit IRC19:19
openstackgerritThiago da Silva proposed a change to openstack/swift: adding same cluster container migration driver  https://review.openstack.org/12172819:28
*** Victor__ has joined #openstack-swift19:30
Victor__Hi everyone. I'm studying how to deploy a reliable and fault tolerant swift cluster. How could I add a extra proxy node in a already deployed swift cluster? Thanks in advance.19:34
glangeVictor__: just put it behnind the load balancer in front of swift and when it is ready have the load balancer start sending it requests19:36
NMVictor__:  1 - install swift 2 - copy the ring files 3 - have fun :)19:37
*** kyles_ne has joined #openstack-swift19:42
Victor__Glange: Nice! Should I replace the endpoint of swift on Keystone?19:42
Victor__NM: Good. Should I add the new proxy nodes address on keystone catalog?19:43
NMVictor__: usually all of your proxies should be behind a Load Balancer19:45
NMIn my case I have: http://myswift.mydomain.com pointing to all my proxy servers. And my keystone points to myswift.mydomain.com19:46
*** kevinc has joined #openstack-swift19:46
glangeVictor__: typically the load balancer is your end point19:48
*** lpabon has joined #openstack-swift19:57
Victor__NM, glange: Nice! Do I need to configure something in the proxies?19:59
*** fbo_ has quit IRC19:59
NMVictor__: Copy the ring files from other clusters.20:04
NMVictor__: what about your memcached? Are you running then at the same machine as the proxies? If yes you should pay attention to its configuration.20:04
*** fbo has joined #openstack-swift20:05
*** kevinc has quit IRC20:06
*** kevinc has joined #openstack-swift20:07
smart_developerIs anyone able to share a performance-optimized proxy-server.conf, account-server.conf, container-server.conf, object-server.conf for a small, 3-5 node cluster?20:08
*** smart_developer has quit IRC20:08
Victor__NM: Right now, my proxies are in different machines. Should I use only one memcache for both?20:09
*** smart_developer has joined #openstack-swift20:09
*** fbo has quit IRC20:10
glangeVictor__: typically you the proxies don't need different code or confs -- they should use the same memcache server pool20:10
smart_developerOr for any cluster, if small is not available.20:12
Victor__glange: What is the memcached used for?20:13
glangeVictor__: caching certain kinds of information to lighten the req load on swift and sometimes things like auth20:14
glangelike some container information is cached20:14
*** nshaikh has quit IRC20:14
glangeit speeds up requests and lightens the load20:15
Victor__glange, NM: Allright! Thank you so very much guys. You've helped me a lot.20:15
glangeVictor__: good luck!20:16
NMVictor__: Anytime :)20:16
NMsmart_developer: I think one of the most importants tunning you can do you to set the number of workers correctly. And this number depends on the number of processors that you have.20:30
smart_developerso, # of workers = # of processors ?20:38
smart_developer( NM )20:39
*** fbo has joined #openstack-swift20:39
smart_developerThanks in advance.20:40
openstackgerritThiago da Silva proposed a change to openstack/swift: adding same cluster container migration driver  https://review.openstack.org/12172820:41
mahatictorgomatic, Does the patch now look ok? (https://review.openstack.org/#/c/119193/)20:42
*** dencaval has joined #openstack-swift20:48
NMsmart_developer: It's a good start. But if you can, always run a benchmark test.20:50
smart_developerNM : All right, so for my 3-node cluster I am running all the services (proxy, account, container, object) on each node. So node1, node2, node3, each have their own proxy-server, account-server, container-server, and object-server.20:52
smart_developerNM : So for the number of workers that I specify in proxy-server.conf, account-server.conf, container-server.conf, and object-server.conf should add up (at least initially before benchmarking) to the number of processers that I have on that particular node?20:53
smart_developerThanks.20:53
smart_developerThanks in advance20:54
*** kyles_ne has quit IRC20:58
*** kyles_ne has joined #openstack-swift20:59
*** kyles_ne has quit IRC21:01
*** kyles_ne has joined #openstack-swift21:01
*** lpabon has quit IRC21:09
*** smart_developer1 has joined #openstack-swift21:11
*** smart_developer has quit IRC21:11
*** marcusvrn has quit IRC21:11
*** miqui has quit IRC21:23
NMsmart_developer1: I'd use that approach21:24
smart_developer1NM : Okay.21:28
smart_developer1NM : But how would you distribute the # of processors to each service on that node?21:28
smart_developer1NM : Let's say that you have about ~30 processors.21:30
smart_developer1By processor, we mean individual cores as well, right?21:31
smart_developer1NM : So I guess technically you mean # of workers = roughly the # of cores ?21:31
NMsmart_developer1: Yes. I mean it. Is there anyway that you can separate the proxies from the ACO (Account/Container/Object) ?21:32
*** ZBhatti has quit IRC21:33
smart_developer1NM : Unfortunately, that's not an option here.21:37
NMsmart_developer1: Ok. Usually proxies and object servers are more busy than account and container servers.  I'd start with something like 10 Proxy, 10 object, 5 account and 5 container.21:37
NMsmart_developer1:  As a smart developer that you are, you should now that there is no silver bullet :)21:38
smart_developer1NM : Thanks. But don't the account and container servers have to traverse through their DBs?21:38
smart_developer1NM : Which seems it would relatively need a lot of compute power?21:38
NMWhat do you mean by that? Accounts and containers response might be cached on memcached.21:38
smart_developer1NM : I thought searching through a lot of database entries was a very computational intensive process ?21:39
*** morganfainberg is now known as CaptainMorgan21:39
smart_developer1NM : Thus some people also use SSDs for their account and container services, while just HDDs for object services...21:39
*** CaptainMorgan is now known as morganfainberg21:39
NMsmart_developer1: Yes, ssd are great solution. But they speed up the disk I/O. I'm not sure if more workers would be a good solution because you would have more concurrency.21:42
smart_developer1NM : and concurrency doesn't necessarily help with account/container DB reads, is what you're saying?21:46
smart_developer1NM : (in terms of reads-performance)21:47
smart_developer1searching/reads.21:47
NMsmart_developer1: Yes. I think that more process trying to read the same file wouldn't help the performance. And I'd apply the same thought to write.21:48
smart_developer1NM : Would that actually be more likely to *degrade* the performance?21:54
smart_developer1NM : if more processes are trying to read, or to write, to the same file21:55
NMsmart_developer1: I think so.21:56
*** NM has quit IRC22:01
mattoliverauMorning all22:06
*** judd7 has joined #openstack-swift22:07
*** r-daneel__ has quit IRC22:09
*** bnelson has quit IRC22:11
*** dmsimard is now known as dmsimard_away22:12
*** peluse_ has joined #openstack-swift22:18
*** smart_developer1 has quit IRC22:20
*** peluse_away has quit IRC22:21
*** smart_developer has joined #openstack-swift22:21
*** mahatic has quit IRC22:36
*** r-daneel__ has joined #openstack-swift22:39
*** r-daneel__ has quit IRC22:44
*** tdasilva has quit IRC22:52
*** r-daneel__ has joined #openstack-swift22:56
*** acoles has quit IRC23:08
*** tongli has quit IRC23:09
*** acoles has joined #openstack-swift23:19
*** ChanServ sets mode: +v acoles23:19
openstackgerritBilly Olsen proposed a change to openstack/swift: Fix getaddrinfo if dnspython is installed.  https://review.openstack.org/11661823:20
*** kevinc has quit IRC23:28
*** dmsimard_away is now known as dmsimard23:32
*** Dafna has quit IRC23:33
*** mkollaro1 has joined #openstack-swift23:33
*** mkollaro has quit IRC23:33
*** bill_az has joined #openstack-swift23:45
*** ekarlso has quit IRC23:48
*** ekarlso has joined #openstack-swift23:48
*** yuan has quit IRC23:55
*** yuan has joined #openstack-swift23:56
*** cebruns_ has quit IRC23:57
*** cebruns has joined #openstack-swift23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!