| donnyd | DHE: Thank you for helping today :) | 00:20 |
|---|---|---|
| mattoliverau | morning | 00:26 |
| *** gyee has quit IRC | 00:40 | |
| *** spsurya has joined #openstack-swift | 02:12 | |
| openstackgerrit | Merged openstack/swift master: Consolidate Container-Update-Override headers https://review.opendev.org/675451 | 02:52 |
| *** zaitcev_ has quit IRC | 03:38 | |
| *** gkadam has joined #openstack-swift | 03:39 | |
| *** gkadam has quit IRC | 03:40 | |
| *** gkadam has joined #openstack-swift | 05:30 | |
| *** pcaruana has joined #openstack-swift | 05:40 | |
| *** pcaruana has quit IRC | 05:49 | |
| viks___ | DHE: | 07:09 |
| viks___ | DHE: Thanks. Currently i do not have any of the below in my object-server.conf under `[app:object-server]`: | 07:10 |
| viks___ | ``` | 07:10 |
| viks___ | # replication_server = false | 07:10 |
| viks___ | # replication_concurrency_per_device = 1 | 07:10 |
| viks___ | # replication_lock_timeout = 15 | 07:10 |
| viks___ | # replication_failure_threshold = 100 | 07:10 |
| viks___ | # replication_failure_ratio = 1.0 | 07:10 |
| viks___ | ``` | 07:10 |
| viks___ | when is this useful? | 07:10 |
| *** psachin has joined #openstack-swift | 07:16 | |
| *** tesseract has joined #openstack-swift | 07:17 | |
| *** rcernin has quit IRC | 07:21 | |
| *** pcaruana has joined #openstack-swift | 07:45 | |
| *** ccamacho has joined #openstack-swift | 08:03 | |
| *** ccamacho has quit IRC | 08:03 | |
| *** ccamacho has joined #openstack-swift | 08:03 | |
| *** klamath has quit IRC | 08:13 | |
| *** klamath has joined #openstack-swift | 08:14 | |
| *** tdasilva has joined #openstack-swift | 09:00 | |
| *** ChanServ sets mode: +v tdasilva | 09:00 | |
| *** ccamacho has quit IRC | 09:11 | |
| *** psachin has quit IRC | 09:18 | |
| *** takamatsu has joined #openstack-swift | 09:24 | |
| *** e0ne has joined #openstack-swift | 09:34 | |
| *** rcernin has joined #openstack-swift | 10:09 | |
| DHE | viks___: since everything there has a '#' in front of it, none of it matters since it's all commented out | 10:24 |
| viks___ | DHE: Ok... but when these options are useful... the above is taken from sample conf which i do not have in my setup. what situation i need to set these? | 10:33 |
| viks___ | what i'm trying to find out is the high cpu usage of object server processes. Earlier in rocky setup, the CPU usage was not high for these processes... so after upgrade to stein, my overall cpu usage is 80% under no load which was earlier around 45%.. so when i checked, object server processes are taking around 45% cpu... so i'm trying to findout what is causing this high cpu usage.. | 10:45 |
| viks___ | even with a fresh vagrant vm's with no data, i see this object server processes taking high cpu.. Not sure if i'm missing some configuration.. | 10:51 |
| *** rcernin has quit IRC | 11:31 | |
| *** henriqueof has joined #openstack-swift | 12:32 | |
| *** henriqueof has quit IRC | 12:37 | |
| viks___ | clayg: DHE : I think i found out the reason... Actually i have set sometime `replicator_workers = 9` before moving to stein from rocky which is based on number of object disks on the node.. earlier in my rocky setup conf, i did not have this configuration at all.. so only one replicator process was running and overall cpu usage of system was around 45%... Since now replicator workers is 9, the overall cpu | 12:39 |
| viks___ | usage showing 80% and there is increased pressure on object server process. So should i be setting `replicator_workers` based on no.of cpus? What is the general suggestion to set these worker counts? | 12:39 |
| *** cwright has joined #openstack-swift | 13:01 | |
| *** cwright has quit IRC | 13:01 | |
| *** cwright has joined #openstack-swift | 13:03 | |
| *** tdasilva_ has joined #openstack-swift | 13:15 | |
| *** ChanServ sets mode: +v tdasilva_ | 13:15 | |
| *** tdasilva has quit IRC | 13:16 | |
| *** gkadam has quit IRC | 13:17 | |
| *** pcaruana has quit IRC | 13:23 | |
| clayg | I work with a lot of clusters that are adding a lot of new capacity on a regular basis - so I like to have a replication worker per disk | 13:27 |
| clayg | But that option "replicator_workers" interacts with your [object-replicator] "concurrency" setting like workers x concurrency == total number of concurrent outgoing REPLICATE requests per node | 13:28 |
| clayg | the default concurrency is only like 1 tho, 9 should be fine - that's about how many disks you have? | 13:29 |
| clayg | viks___: so when you talk about 45% => 80% CPU usage on "object server proccesses" do you mean overall CPU usage on the your object nodes or *just* the swift-object-server workers? | 13:31 |
| clayg | I may have misunderstood | 13:31 |
| *** BjoernT has joined #openstack-swift | 13:36 | |
| *** BjoernT_ has joined #openstack-swift | 13:39 | |
| *** BjoernT has quit IRC | 13:39 | |
| *** BjoernT has joined #openstack-swift | 13:44 | |
| *** BjoernT_ has quit IRC | 13:46 | |
| *** pcaruana has joined #openstack-swift | 14:00 | |
| *** tdasilva has joined #openstack-swift | 15:29 | |
| *** ChanServ sets mode: +v tdasilva | 15:29 | |
| *** tdasilva_ has quit IRC | 15:30 | |
| *** gyee has joined #openstack-swift | 15:44 | |
| *** spsurya has quit IRC | 16:04 | |
| *** tdasilva has quit IRC | 16:12 | |
| *** BjoernT has quit IRC | 16:17 | |
| *** tdasilva has joined #openstack-swift | 16:32 | |
| *** ChanServ sets mode: +v tdasilva | 16:32 | |
| *** tesseract has quit IRC | 16:43 | |
| *** e0ne has quit IRC | 16:51 | |
| viks___ | clayg: 80% is overall cpu usage per storage node and 45% is each of 4 object server workers cpu usage... i have 9 disks in 2 nodes for object storage.. replicator concurrency is 2 and replicator_workers = 9.. total object workers is 4 because of 4 cpus.. i think i need to reduce either replicator workers or replicator concurrency right? | 16:56 |
| *** zaitcev has joined #openstack-swift | 17:43 | |
| *** ChanServ sets mode: +v zaitcev | 17:43 | |
| *** mvkr has quit IRC | 17:45 | |
| clayg | viks___: yes you can safely reduce both, if you add new devices to the cluster you'll want to monitor your replication cycle times (form logs or recon drops) and maybe re-evaluate your worker/concurrency settings. | 17:53 |
| *** henriqueof has joined #openstack-swift | 18:04 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Allow "static symlinks" https://review.opendev.org/633094 | 18:04 |
| *** BjoernT has joined #openstack-swift | 18:15 | |
| *** onovy has quit IRC | 18:16 | |
| *** onovy has joined #openstack-swift | 18:21 | |
| *** e0ne has joined #openstack-swift | 18:23 | |
| *** manuvakery has quit IRC | 18:28 | |
| *** diablo_rojo has joined #openstack-swift | 18:37 | |
| *** diablo_rojo has quit IRC | 18:47 | |
| *** zaitcev has quit IRC | 19:20 | |
| *** mvkr has joined #openstack-swift | 19:24 | |
| *** diablo_rojo has joined #openstack-swift | 19:29 | |
| *** mvkr has quit IRC | 19:30 | |
| *** zaitcev has joined #openstack-swift | 19:34 | |
| *** ChanServ sets mode: +v zaitcev | 19:34 | |
| *** diablo_rojo has quit IRC | 20:00 | |
| *** diablo_rojo has joined #openstack-swift | 20:19 | |
| *** tdasilva has quit IRC | 20:32 | |
| *** e0ne has quit IRC | 20:38 | |
| *** henriqueof has quit IRC | 22:07 | |
| *** zaitcev has quit IRC | 22:09 | |
| *** zaitcev has joined #openstack-swift | 22:22 | |
| *** ChanServ sets mode: +v zaitcev | 22:22 | |
| *** diablo_rojo has quit IRC | 22:41 | |
| *** tkajinam has joined #openstack-swift | 22:59 | |
| *** diablo_rojo has joined #openstack-swift | 23:35 | |
| *** diablo_rojo has quit IRC | 23:41 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Allow bulk delete of big SLO manifests https://review.opendev.org/540122 | 23:51 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!