| *** rcernin has quit IRC | 01:19 | |
| *** rcernin has joined #openstack-swift | 01:28 | |
| *** gyee has quit IRC | 02:03 | |
| *** rcernin has quit IRC | 02:28 | |
| *** rcernin has joined #openstack-swift | 02:30 | |
| *** jv_ has joined #openstack-swift | 02:36 | |
| *** jv_ has quit IRC | 03:03 | |
| *** rcernin has quit IRC | 03:24 | |
| *** psachin has joined #openstack-swift | 03:35 | |
| *** rcernin has joined #openstack-swift | 03:47 | |
| *** m75abrams has joined #openstack-swift | 03:52 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-swift | 04:33 | |
| openstackgerrit | Merged openstack/swift master: proxy-logging: Be able to configure log_route https://review.opendev.org/735220 | 04:59 |
|---|---|---|
| *** coreycb has quit IRC | 05:55 | |
| *** coreycb has joined #openstack-swift | 05:57 | |
| *** dsariel has joined #openstack-swift | 07:34 | |
| *** gregwork has quit IRC | 07:42 | |
| *** rpittau has quit IRC | 07:47 | |
| *** gregwork has joined #openstack-swift | 07:55 | |
| *** rpittau has joined #openstack-swift | 07:56 | |
| *** rcernin has quit IRC | 07:58 | |
| *** coreycb has quit IRC | 08:30 | |
| *** coreycb has joined #openstack-swift | 08:30 | |
| *** gregwork has quit IRC | 08:30 | |
| *** gregwork has joined #openstack-swift | 08:30 | |
| *** rpittau has quit IRC | 08:30 | |
| *** rpittau has joined #openstack-swift | 08:30 | |
| *** jv_ has joined #openstack-swift | 11:34 | |
| *** zaitcev has quit IRC | 12:43 | |
| *** m75abrams has quit IRC | 12:59 | |
| *** jv_ has quit IRC | 13:25 | |
| *** jv_ has joined #openstack-swift | 13:26 | |
| *** zaitcev has joined #openstack-swift | 13:29 | |
| *** ChanServ sets mode: +v zaitcev | 13:29 | |
| *** psachin has quit IRC | 14:02 | |
| *** jv_ has quit IRC | 15:55 | |
| *** gyee has joined #openstack-swift | 16:08 | |
| *** manuvakery has joined #openstack-swift | 16:21 | |
| timburke | good morning | 16:56 |
| *** gregwork has quit IRC | 18:04 | |
| *** josephillips has joined #openstack-swift | 18:35 | |
| josephillips | hey | 18:35 |
| josephillips | question | 18:35 |
| josephillips | is normal that swift when is getting a lot of data the disk stuck on around 20% of waiting for i/o | 18:35 |
| josephillips | on top | 18:35 |
| josephillips | or i can do something to decrease the waiting for IO | 18:35 |
| josephillips | are 7200rpm disks | 18:36 |
| *** manuvakery has quit IRC | 19:00 | |
| timburke | josephillips, what kind of workload is this? reads or writes? do these disks have just object data, or account/container databases, too? if doing writes, how large are the objects being written? if doing reads, how big are the reads (noting that clients may be doing ranged reads for something less than the whole object) | 19:55 |
| josephillips | is 4MB objects | 19:56 |
| josephillips | is during writes and and some reads | 19:56 |
| timburke | do you know if they're SMR drives? 4MB shouldn't be *so bad*... if they were sub-1MB, i'd chalk it up to the generally lower performance with small files. is this EC or replicated? | 20:08 |
| *** jv_ has joined #openstack-swift | 20:09 | |
| timburke | when you say you've got disks "stuck on around 20% of waiting for i/o", how is that measured? i could see what numbers i see in my cluster... | 20:11 |
| josephillips | on top | 20:13 |
| josephillips | timburke | 20:13 |
| josephillips | im monitor the storage node | 20:14 |
| josephillips | and when i have a heavy load of loading data | 20:14 |
| josephillips | triggers alerts about the nodes are on Waiting for IO | 20:14 |
| josephillips | if i check on top command | 20:14 |
| josephillips | the WA field | 20:14 |
| josephillips | keeps like 20 25 | 20:14 |
| josephillips | until the loads complete | 20:14 |
| josephillips | checking on IOTOP | 20:15 |
| josephillips | i see that write and read is on 100MB/S thats make sense because the server have 1gbit interface | 20:15 |
| timburke | :-/ yeah, so i see <1%, but these are fairly beefy servers -- like 24 cores, probably more than we need for 10 disk chassis. how many disks per node? my best bet is that it's driven by drive contention -- if you watch `iostat -x 2` for a bit, is the load fairly well-distributed across disks, or a bit lopsided? | 20:25 |
| josephillips | let me check | 21:04 |
| josephillips | right now | 21:04 |
| josephillips | each node have 6 disks | 21:04 |
| josephillips | i have 3 storage nodes | 21:04 |
| *** rcernin has joined #openstack-swift | 22:49 | |
| *** rcernin has quit IRC | 23:06 | |
| *** rcernin has joined #openstack-swift | 23:08 | |
| openstackgerrit | Tim Burke proposed openstack/swift master: Client should retry when there's just one 404 and a bunch of errors https://review.opendev.org/744942 | 23:14 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Add sorting_method=none to get a consistent node order https://review.opendev.org/747310 | 23:14 |
| openstackgerrit | Tim Burke proposed openstack/swift master: ec: Close down unused responses quickly https://review.opendev.org/747311 | 23:14 |
| *** dsariel has quit IRC | 23:53 | |
| *** abelur has quit IRC | 23:56 | |
| *** lxkong has quit IRC | 23:56 | |
| *** lxkong has joined #openstack-swift | 23:57 | |
| *** abelur has joined #openstack-swift | 23:58 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!