easystack中山证券私有云平台ceph抖动研究

1、1环境说明集群环境SAS+SSD journal ceph 集群180 个 osd，共 18 台，每台 10 个 10k SAS 盘，2 个 SSD 做 journal 加速，SSD 和 SAS 比例是 1:5，每个 SSD journal 分区大小设置为 100G。纯 SAS ceph 集群纯 SSD ceph 集群1.2 测试工具测试工具： fio-2.19测试内容： 随机写、混合读写(读写比例：7:3)、随机读、网络 IO 带宽测试条件：1.2.3.4.1 numjobs + 1 iodepth(4k)101616numjobs numjobsnumjobs+323216iodepth(

2、4k) iodepth(4k)iodepth(4k)测试时间：10 分钟，取平均值。1.3 测试目的研究 SAS+SSD 做 journal 的场景下，ceph 的写抖动落差很大的原因，并给予中山储最有解决方案。私有云ceph 存2SAS 集群性能测试2.1 没有 SSD 做 journal1 个并发+1 深度(随机写)测试环境global direct=1 time_based group_reportingperftest numjobs=1 rw=randwrite bs=4k iodepth=1r_time=5 #rwmixwrite=30runtime=6002.3.磁盘资源利用率p

3、erftest: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=1 fio-2.19Starting 1 prosCbs: 1 (f=1): w(1)43.0%r=0KiB/s,w=1248KiB/sr=0,w=312 IOPSeta 05m:45sfio: terminating on signal 2perftest: (grou=0, jobs=1): err= 0:=1341291: Sat Jul8 00:25:43 2017 write

4、: IOPS=298, BW=1196KiB/s (1224kB/s)(298MiB/255202msec)slat (usec): min=1, max=1534, avg= 5.33, stdev=12.73 clat (msec): min=2, max=4925, avg= 3.34, stdev=18.24 lat (msec): min=2, max=4925, avg= 3.34, stdev=18.24 clat percentiles (usec):|1.00th= 2640,5.00th= 2800, 10.00th= 2896, 20.00th= 2992,| 30.00

5、th= 3056, 40.00th= 3152, 50.00th= 3216, 60.00th= 3280,| 70.00th= 3376, 80.00th= 3504, 90.00th= 3664, 95.00th= 3792,| 99.00th= 4192, 99.50th= 4512, 99.90th= 6880, 99.95th= 8512,| 99.99th=42240lat (msec) : 4=97.98%, 10=1.98%, 20=0.02%, 50=0.01%, 2000=0.01%lat (msec) : =2000=0.01%cpu: usr=24.17%, sys=7

6、4.92%, ctx=90537, majf=0, minf=12782IO depths: 1=101.5%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0%submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0%complete: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0%ied rwt: total=0,76276,0, short=0,0,0, dropped=0,0,0

7、latency:=0, window=0, percentile=100.00%, depth=1Run sus group 0 (all jobs):WRITE: bw=1196KiB/s (1224kB/s), 1196KiB/s-1196KiB/s (1224kB/s-1224kB/s), io=298MiB (312MB),run=255202-255202mseciodepth_batch_submit=1plete=1ioengine=rbdcntname=adminpute rbdname=test4.计算资源利用率结论：IOPS 很稳定，没有抖动10 个并发 32 深度(随机写

8、)测试环境global direct=1 time_based group_reportingperftest numjobs=10 rw=randwrite bs=4k iodepth=32r_time=5#rwmixwrite=302.3.磁盘资源利用率perftest: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=32.fio-2.19Starting 10 prosesJobs: 10 (f=0): f(10)100.0%r=0KiB/s

9、,w=114MiB/sr=0,w=29.3k IOPSeta 00m:00sperftest: (grou=0, jobs=10): err= 0:=1279697: Fri Jul7 23:43:59 2017 write: IOPS=45.1k, BW=180MiB/s (188MB/s)(105GiB/600017msec)slat (usec): min=1, max=11376, avg= 4.07, stdev=14.93clat (usec): min=128, max=2933.4k, avg=6953.13, stdev=28115.04 lat (msec): min=1,

10、 max=2933, avg= 6.96, stdev=28.12clat percentiles (usec):|1.00th= 1464,5.00th= 1688, 10.00th= 1864, 20.00th= 2128,| 30.00th= 2416, 40.00th= 2672, 50.00th= 2992, 60.00th= 3376,| 70.00th= 3856, 80.00th= 4640, 90.00th= 6624, 95.00th=12224,| 99.00th=108032, 99.50th=189440, 99.90th=382976, 99.95th=464896

11、,| 99.99th=749568lat (usec) : 250=0.01%, 1000=0.01%lat (msec) : 2=14.82%, 4=57.42%, 10=21.87%, 20=2.20%, 50=1.77%lat (msec) : 100=0.84%, 250=0.78%, 500=0.27%, 750=0.03%, 1000=0.01%lat (msec) : 2000=0.01%, =2000=0.01%cpu: usr=25.05%, sys=71.71%, ctx=21858536, majf=0, minf=77308IO depths: 1=0.1%, 2=0.

12、1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.7%, =64=0.0%submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0%complete: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, =64=0.0%ied rwt: total=0,27593271,0, short=0,0,0, dropped=0,0,0 latency:=0, window=0, percentile=100.00%, depth=32Run su

13、s group 0 (all jobs):WRITE: bw=180MiB/s (188MB/s), 180MiB/s-180MiB/s (188MB/s-188MB/s), io=105GiB (113GB),run=600017-600017msecruntime=600 iodepth_batch_submit=1plete=1ioengine=rbdcntname=adminpute rbdname=test4.计算资源利用率结论：没有 SSD 做 journal 的环境下，以 10 个并发 4k 的随机做测试，iops 为：45.1k，延迟为：6953.13，相对比较平稳，IOPS

14、稳定在 30k-40k 之间。16 个并发 32 深度(随机写)测试环境global direct=1 time_based group_reportingperftest numjobs=16 rw=randwrite bs=4k iodepth=32r_time=52.3.磁盘利用率perftest: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=32.fio-2.19Starting 16 prosesJobs: 16 (f=0): f(16

15、)100.0%r=0KiB/s,w=138MiB/sr=0,w=35.4k IOPSeta 00m:00sperftest: (grou=0, jobs=16): err= 0:=1261258: Fri Jul7 23:31:10 2017 write: IOPS=46.5k, BW=182MiB/s (190MB/s)(106GiB/600214msec)slat (usec): min=1, max=19307, avg= 4.87, stdev=23.66clat (usec): min=44, max=1412.2k, avg=11009.84, stdev=41313.43 lat

16、 (msec): min=1, max=1412, avg=11.01, stdev=41.31clat percentiles (usec):|1.00th= 1400,5.00th= 1624, 10.00th= 1800, 20.00th= 2096,| 30.00th= 2416, 40.00th= 2800, 50.00th= 3216, 60.00th= 3824,| 70.00th= 4768, 80.00th= 6368, 90.00th=11456, 95.00th=31104,| 99.00th=207872, 99.50th=313344, 99.90th=536576,

17、 99.95th=626688,| 99.99th=831488lat (usec) : 50=0.01%, 250=0.01%, 750=0.01%lat (msec) : 2=16.43%, 4=45.85%, 10=26.25%, 20=4.93%, 50=2.86%lat (msec) : 100=1.48%, 250=1.43%, 500=0.63%, 750=0.11%, 1000=0.02%lat (msec) : 2000=0.01%cpu: usr=27.71%, sys=67.13%, ctx=23105273, majf=0, minf=60692IO depths: 1

18、=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.1%, =64=0.0%submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0%complete: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, =64=0.0%ied rwt: total=0,27895320,0, short=0,0,0, dropped=0,0,0 latency:=0, window=0, percentile=100.00%, dep

19、th=32#rwmixwrite=30 runtime=600 iodepth_batch_submit=1plete=1ioengine=rbdcntname=adminpute rbdname=test4.计算资源利用率结论：没有 SSD 做 journald 的情况下，以 16 个并发测试 4k 随机写，平均 iops 为 46.5k：，延迟为 11009.84：，iops 抖动较小，普遍在 30k-40k 之间。20 个并发 32 并发(随机写)测试环境：global direct=1 time_based group_reportingperftest numjobs=20 rw=r

20、andwrite bs=4k iodepth=32r_time=52.：3.磁盘利用率：perftest: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=32.fio-2.19Starting 20 prosesJobs: 19 (f=0): f(12),_(1),f(7)100.0%r=0KiB/s,w=82.7MiB/sr=0,w=21.2k IOPSeta 00m:00sperftest: (grou=0, jobs=20): err= 0:

21、=1229703: Fri Jul7 23:13:25 2017 write: IOPS=46.1k, BW=183MiB/s (192MB/s)(107GiB/600180msec)slat (usec): min=1, max=19177, avg= 5.71, stdev=28.09clat (usec): min=19, max=1841.4k, avg=13633.62, stdev=53052.45 lat (usec): min=988, max=1841.4k, avg=13639.33, stdev=53052.45 clat percentiles (usec):|1.00

22、th= 1336,5.00th= 1512, 10.00th= 1656, 20.00th= 1928,| 30.00th= 2192, 40.00th= 2544, 50.00th= 2960, 60.00th= 3568,| 70.00th= 4576, 80.00th= 6624, 90.00th=14272, 95.00th=47360,| 99.00th=268288, 99.50th=395264, 99.90th=667648, 99.95th=774144,| 99.99th=1056768lat (usec) : 20=0.01%, 50=0.01%, 500=0.01%,

23、750=0.01%, 1000=0.01%lat (msec) : 2=22.77%, 4=42.25%, 10=21.94%, 20=4.89%, 50=3.33%lat (msec) : 100=1.85%, 250=1.86%, 500=0.85%, 750=0.21%, 1000=0.05%lat (msec) : 2000=0.01%cpu: usr=23.94%, sys=69.66%, ctx=25339250, majf=0, minf=81342IO depths: 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, =64

24、=0.0%submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0%complete: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, =64=0.0%ied rwt: total=0,28160568,0, short=0,0,0, dropped=0,0,0 latency:=0, window=0, percentile=100.00%, depth=32Run sus group 0 (all jobs):WRITE: bw=183MiB/s (19

25、2MB/s), 183MiB/s-183MiB/s (192MB/s-192MB/s), io=107GiB (115GB),run=600180-600180msec#rwmixwrite=30 runtime=600 iodepth_batch_submit=1plete=1ioengine=rbdcntname=adminpute rbdname=test4.计算资源情况：结论：没有 SSD 做 journal 的情况下，iops 为 46k，延时平均延时为 avg=13639.33ms，抖动相对没那么大，普遍在 30k 到 50k 之间。16 个并发+16 深度(随机写)测试环境glo

26、bal direct=1 time_based group_reportingperftest numjobs=16 rw=randwrite bs=4k2.perftest: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=16.fio-2.19Starting 16 prosesJobs: 16 (f=0): f(16)100.0%r=0KiB/s,w=87.3MiB/sr=0,w=22.4k IOPSeta 00m:00sperftest: (

27、grou=0, jobs=16): err= 0:=1322892: Sat Jul8 00:18:21 2017 write: IOPS=44.4k, BW=173MiB/s (182MB/s)(102GiB/600080msec)slat (usec): min=1, max=15827, avg= 4.90, stdev=20.27clat (usec): min=94, max=1163.9k, avg=5764.85, stdev=21657.33 lat (msec): min=1, max=1163, avg= 5.77, stdev=21.66clat percentiles

28、(usec):|1.00th= 1464,5.00th= 1672, 10.00th= 1832, 20.00th= 2064,| 30.00th= 2288, 40.00th= 2544, 50.00th= 2800, 60.00th= 3088,| 70.00th= 3504, 80.00th= 4128, 90.00th= 5600, 95.00th=10176,| 99.00th=76288, 99.50th=142336, 99.90th=309248, 99.95th=382976,| 99.99th=610304lat (usec) : 100=0.01%, 250=0.01%,

29、 1000=0.01%lat (msec) : 2=17.11%, 4=61.56%, 10=16.28%, 20=1.95%, 50=1.66%lat (msec) : 100=0.69%, 250=0.58%, 500=0.16%, 750=0.01%, 1000=0.01%lat (msec) : 2000=0.01%cpu: usr=24.67%, sys=70.62%, ctx=21810257, majf=0, minf=92493IO depths: 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, =64=0.0%submi

30、t: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0%complete: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0%ied rwt: total=0,26625149,0, short=0,0,0, dropped=0,0,0 latency:=0, window=0, percentile=100.00%, depth=16Run sus group 0 (all jobs):WRITE: bw=173MiB/s (182MB/s), 17

31、3MiB/s-173MiB/s (182MB/s-182MB/s), io=102GiB (109GB),run=600080-600080mseciodepth=16r_time=5 #rwmixwrite=30 runtime=600 iodepth_batch_submit=1plete=1ioengine=rbdcntname=adminpute rbdname=test3.磁盘资源利用率4.计算资源利用率结论：没有 SSD 做 journal 的情况下，iops 为 44k，延时平均延时为 avg=5764.85，抖动相对没那么大，普遍在 30k 到 50k 之间。16 并发 16

32、深度(混合读写)测试环境global direct=1 time_based group_reportingperftest numjobs=16 rw=randrw bs=4k2.perftest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=16.fio-2.19Starting 16 prosesbs: 16 (f=16): m(16)73.9%r=119MiB/s,w=51.4MiB/sr=30.6k,w=13.2k IOPSeta 02m:3

33、8sfio: terminating on signal 2Jobs: 11 (f=0): f(1),_(1),f(4),_(1),f(1),_(1),f(5),_(2)100.0%r=67.2MiB/s,w=28.7MiB/sr=17.2k,w=7328 IOPSeta 00m:00sperftest: (grou=0, jobs=16): err= 0:=1358469: Sat Jul8 00:39:03 2017 read: IOPS=58.2k, BW=227MiB/s (238MB/s)(98.1GiB/442376msec)slat (usec): min=1, max=2205

34、2, avg= 4.65, stdev=20.15clat (usec): min=23, max=2013.5k, avg=2444.25, stdev=15343.19 lat (usec): min=308, max=2013.5k, avg=2448.90, stdev=15343.26 clat percentiles (usec):|1.00th=524,5.00th=596, 10.00th=636, 20.00th=708,| 30.00th=780, 40.00th=868, 50.00th=996, 60.00th= 1176,| 70.00th= 1448, 80.00t

35、h= 1816, 90.00th= 2384, 95.00th= 3120,| 99.00th=17280, 99.50th=69120, 99.90th=244736, 99.95th=325632,| 99.99th=509952write: IOPS=24.1k, BW=97.4MiB/s (102MB/s)(42.5GiB/442376msec)slat (usec): min=1, max=11095, avg= 4.73, stdev=19.00clat (usec): min=96, max=1863.2k, avg=4548.81, stdev=20049.43 lat (ms

36、ec): min=1, max=1863, avg= 4.55, stdev=20.05clat percentiles (usec):|1.00th= 1576,5.00th= 1768, 10.00th= 1912, 20.00th= 2128,| 30.00th= 2288, 40.00th= 2416, 50.00th= 2544, 60.00th= 2704,| 70.00th= 2896, 80.00th= 3152, 90.00th= 3696, 95.00th= 4448,| 99.00th=57600, 99.50th=132096, 99.90th=313344, 99.9

37、5th=387072,| 99.99th=585728lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.37%, 750=17.99%lat (usec) : 1000=16.93%iodepth=16r_time=5 rwmixwrite=30 runtime=600 iodepth_batch_submit=1plete=1ioengine=rbdcntname=adminpute rbdname=test3. 磁盘资源利用率4. 计算资源利用率lat (msec) : 2=27.67%, 4=32.53%, 10=2.75%, 20=0

38、.66%, 50=0.35%lat (msec) : 100=0.28%, 250=0.35%, 500=0.10%, 750=0.01%, 1000=0.01%lat (msec) : 2000=0.01%, =2000=0.01%cpu: usr=24.08%, sys=66.56%, ctx=29827731, majf=0, minf=49615IO depths: 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, =64=0.0%submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%,

39、 64=0.0%, =64=0.0%complete: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, =64=0.0%ied rwt: total=25715612,11021875,0, short=0,0,0, dropped=0,0,0 latency:=0, window=0, percentile=100.00%, depth=16Run sus group 0 (all jobs):READ: bw=227MiB/s (238MB/s), 227MiB/s-227MiB/s (238MB/s-238MB/s), io=98

40、.1GiB (105GB),run=442376-442376msecWRITE: bw=97.4MiB/s (102MB/s), 97.4MiB/s-97.4MiB/s (102MB/s-102MB/s), io=42.5GiB (45.2GB),run=442376-442376msec结论：压测到 40%的之前，读的 iops 在 90k 左右，写在 40k 左右，之后开始下降，读的 iops 维持在 30k，写的 iops 在 10k 左右。2.1.7 16 并发 32 深度(混合读写)1.测试环境2.perftest: (g=0): rw=randrw, bs=(R) 4096B-4

41、096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=32.fio-2.19Starting 16 prosesJobs: 8 (f=0): _(5),f(2),_(1),f(1),_(1),f(1),_(1),f(4)100.0%r=113MiB/s,w=50.2MiB/sr=29.4k,w=12.9k IOPSeta 00m:00sperftest: (grou=0, jobs=16): err= 0:=1372567: Sat Jul8 00:49:48 2017 read: IOPS=65.6k, BW=256MiB

42、/s (268MB/s)(150GiB/600415msec)slat (usec): min=1, max=17677, avg= 4.34, stdev=19.45clat (usec): min=33, max=1904.4k, avg=4413.17, stdev=22161.47 lat (usec): min=317, max=1904.5k, avg=4417.51, stdev=22161.53 clat percentiles (usec):|1.00th=548,5.00th=612, 10.00th=660, 20.00th=756,| 30.00th=940, 40.0

43、0th= 1496, 50.00th= 1896, 60.00th= 2288,| 70.00th= 2896, 80.00th= 3728, 90.00th= 4768, 95.00th= 6048,vim fio.conf global direct=1 time_based group_reportingperftest numjobs=16 rw=randrw bs=4k iodepth=32r_time=5 rwmixwrite=30 runtime=600 iodepth_batch_submit=1plete=1ioengine=rbdcntname=adminpute rbdn

44、ame=test3. 磁盘资源利用率4. 计算资源利用率| 99.00th=71168, 99.50th=146432, 99.90th=333824, 99.95th=423936,| 99.99th=643072write: IOPS=28.9k, BW=110MiB/s (115MB/s)(64.4GiB/600415msec)slat (usec): min=0, max=19708, avg= 4.43, stdev=19.38clat (usec): min=160, max=1971.9k, avg=7907.10, stdev=30745.70 lat (msec): min=

45、1, max=1971, avg= 7.91, stdev=30.75clat percentiles (usec):|1.00th= 1912,5.00th= 2192, 10.00th= 2352, 20.00th= 2576,| 30.00th= 2768, 40.00th= 2992, 50.00th= 3280, 60.00th= 3664,| 70.00th= 4256, 80.00th= 4960, 90.00th= 6240, 95.00th= 8896,| 99.00th=144384, 99.50th=222208, 99.90th=423936, 99.95th=5140

46、48,| 99.99th=733184lat (usec) : 50=0.01%, 250=0.01%, 500=0.22%, 750=13.28%, 1000=8.76%lat (msec) : 2=15.33%, 4=40.49%, 10=18.82%, 20=0.83%, 50=0.70%lat (msec) : 100=0.59%, 250=0.73%, 500=0.22%, 750=0.03%, 1000=0.01%lat (msec) : 2000=0.01%cpu: usr=24.86%, sys=66.74%, ctx=44741130, majf=0, minf=90023I

47、O depths: 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, =64=0.0%submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0%complete: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, =64=0.0%ied rwt: total=39351801,16866211,0, short=0,0,0, dropped=0,0,0 latency:=0, window=0, perce

48、ntile=100.00%, depth=32Run sus group 0 (all jobs):READ: bw=256MiB/s (268MB/s), 256MiB/s-256MiB/s (268MB/s-268MB/s), io=150GiB (161GB),run=600415-600415msecWRITE: bw=110MiB/s (115MB/s), 110MiB/s-110MiB/s (115MB/s-115MB/s), io=64.4GiB (69.9GB),run=600415-600415msec结论：60%之前，读写 iops 在 100k，写 iops 在 40k

49、左右，当执行到 60%时，读 iops 在 50k，写 iops 在 20k左右。2.2 有 SSD 做 journal1 个并发 1 深度(随机写)测试环境global direct=1 time_based group_reportingperftest numjobs=1 rw=randrw bs=4k iodepth=1r_time=5 rwmixwrite=30 runtime=600 iodepth_batch_submit=1plete=1ioengine=rbdcntname=adminpute rbdname=test2.3.磁盘资源利用情况4.计算资源利用情况perftes

50、t: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=1 fio-2.19Starting 1 prosJobs: 1 (f=1): w(1)100.0%r=0KiB/s,w=1276KiB/sr=0,w=319 IOPSeta 00m:00sperftest: (grou=0, jobs=1): err= 0:=1617889: Sat Jul8 03:47:59 2017write: IOPS=319, BW=1278KiB/s (1309kB/

51、s)(749MiB/600004msec) slat (usec): min=1, max=1584, avg= 4.89, stdev= 8.56clat (msec): min=2, max=579, avg= 3.12, stdev= 1.43 lat (msec): min=2, max=579, avg= 3.13, stdev= 1.43 clat percentiles (usec):|1.00th= 2544,5.00th= 2672, 10.00th= 2768, 20.00th= 2864,| 30.00th= 2928, 40.00th= 3024, 50.00th= 3

52、088, 60.00th= 3152,| 70.00th= 3248, 80.00th= 3344, 90.00th= 3504, 95.00th= 3664,| 99.00th= 4016, 99.50th= 4192, 99.90th= 4704, 99.95th= 5536,| 99.99th=20864lat (msec) : 4=98.90%, 10=1.08%, 20=0.01%, 50=0.01%, 100=0.01%lat (msec) : 250=0.01%, 750=0.01%cpu: usr=23.63%, sys=75.61%, ctx=221808, majf=0,

53、minf=18894IO depths: 1=100.6%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, =64=0.0%submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0%complete: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0%ied rwt: total=0,191686,0, short=0,0,0, dropped=0,0,0 latency:=0, window=0, per

54、centile=100.00%, depth=1Run sus group 0 (all jobs):WRITE: bw=1278KiB/s (1309kB/s), 1278KiB/s-1278KiB/s (1309kB/s-1309kB/s), io=749MiB (785MB),run=600004-600004msec2.2.2 10 个并发 32 深度(随机写)1.测试环境2.测试过程perftest: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iod

55、epth=32.global direct=1 time_based group_reportingperftest numjobs=10 rw=randwrite bs=4k iodepth=32r_time=5 #rwmixwrite=30 runtime=600 iodepth_batch_submit=1plete=1ioengine=rbdcntname=adminpute rbdname=test3. 磁盘资源利用率4. 计算资源利用率fio-2.19Starting 10 prosesJobs: 10 (f=0): f(10)100.0%r=0KiB/s,w=227MiB/sr=

56、0,w=58.9k IOPSeta 00m:00sperftest: (grou=0, jobs=10): err= 0:=1525626: Sat Jul8 02:46:30 2017 write: IOPS=51.4k, BW=199MiB/s (209MB/s)(117GiB/600012msec)slat (usec): min=1, max=14066, avg= 3.95, stdev=15.47clat (usec): min=606, max=1183.6k, avg=6268.75, stdev=25190.33 lat (usec): min=984, max=1183.6

57、k, avg=6272.70, stdev=25190.35 clat percentiles (usec):|1.00th= 1352,5.00th= 1512, 10.00th= 1640, 20.00th= 1848,| 30.00th= 2064, 40.00th= 2320, 50.00th= 2608, 60.00th= 2960,| 70.00th= 3440, 80.00th= 4256, 90.00th= 6176, 95.00th=10688,| 99.00th=99840, 99.50th=181248, 99.90th=374784, 99.95th=444416,|

58、99.99th=602112lat (usec) : 750=0.01%, 1000=0.01%lat (msec) : 2=26.77%, 4=50.84%, 10=17.08%, 20=2.17%, 50=1.38%lat (msec) : 100=0.76%, 250=0.71%, 500=0.26%, 750=0.03%, 1000=0.01%lat (msec) : 2000=0.01%cpu: usr=25.04%, sys=71.40%, ctx=23715689, majf=0, minf=77535IO depths: 1=0.1%, 2=0.1%, 4=0.1%, 8=0.

59、1%, 16=0.1%, 32=100.7%, =64=0.0%submit: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, =64=0.0%complete: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, =64=0.0%ied rwt: total=0,30603501,0, short=0,0,0, dropped=0,0,0 latency:=0, window=0, percentile=100.00%, depth=32Run sus group 0 (all j

60、obs):WRITE: bw=199MiB/s (209MB/s), 199MiB/s-199MiB/s (209MB/s-209MB/s), io=117GiB (125GB),run=600012-600012msec总结：IOPS 大部分在 30k-60k 之间，抖动差为 30k 左右，抖动较大。2.2.3 16 个并发 32 深度(随机写)1.测试环境测试过程：global direct=1 time_based group_reportingperftest numjobs=16 rw=randwrite bs=4k iodepth=32r_time=5 #rwmixwrite=30