ganesha内存盘 libaio ##X86 2 [root@node-191 ~]# ./stream

STREAM version $Revision: 5.10 $

This system uses 8 bytes per array element.

***** WARNING: ****** It appears that you set the preprocessor variable N when compiling this code. This version of the code uses the preprocesor variable STREAM_ARRAY_SIZE to control the array size Reverting to default value of STREAM_ARRAY_SIZE=10000000 ***** WARNING: ****** Array size = 10000000 (elements), Offset = 0 (elements) Memory per array = 76.3 MiB (= 0.1 GiB). Total memory required = 228.9 MiB (= 0.2 GiB). Each kernel will be executed 10 times. The best time for each kernel (excluding the first iteration) will be used to compute the reported bandwidth.

Number of Threads requested = 56 Number of Threads counted = 56

Your clock granularity/precision appears to be 1 microseconds. Each test below will take on the order of 3473 microseconds. (= 3473 clock ticks) Increase the size of the arrays if this shows that you are not getting at least 20 clock ticks per test.

WARNING -- The above is only a rough guideline. For best results, please be sure you know the precision of your system timer.

Function Best Rate MB/s Avg time Min time Max time Copy: 77519.8 0.002207 0.002064 0.002785 Scale: 67425.8 0.002690 0.002373 0.003889 Add: 70670.7 0.003597 0.003396 0.003898 Triad: 74049.8 0.003420 0.003241 0.003723

Solution Validates: avg error less than 1.000000e-13 on all three arrays

[root@ceph192 ~]# ./stream

STREAM version $Revision: 5.10 $

This system uses 8 bytes per array element.

***** WARNING: ****** It appears that you set the preprocessor variable N when compiling this code. This version of the code uses the preprocesor variable STREAM_ARRAY_SIZE to control the array size Reverting to default value of STREAM_ARRAY_SIZE=10000000 ***** WARNING: ****** Array size = 10000000 (elements), Offset = 0 (elements) Memory per array = 76.3 MiB (= 0.1 GiB). Total memory required = 228.9 MiB (= 0.2 GiB). Each kernel will be executed 10 times. The best time for each kernel (excluding the first iteration) will be used to compute the reported bandwidth.

Number of Threads requested = 16 Number of Threads counted = 16

Your clock granularity/precision appears to be 1 microseconds. Each test below will take on the order of 4297 microseconds. (= 4297 clock ticks) Increase the size of the arrays if this shows that you are not getting at least 20 clock ticks per test.

WARNING -- The above is only a rough guideline. For best results, please be sure you know the precision of your system timer.

Function Best Rate MB/s Avg time Min time Max time Copy: 35964.0 0.004598 0.004449 0.004810 Scale: 37985.4 0.004316 0.004212 0.004794 Add: 30766.0 0.007990 0.007801 0.008304 Triad: 38308.5 0.006377 0.006265 0.006529

Solution Validates: avg error less than 1.000000e-13 on all three arrays

16k

##16k Seq写测试 fio -filename=/mnt/mem1/16k -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file16kwrite_seq

LA

[root@ceph192 mnt]# fio -filename=/mnt/mem1/16k -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file16kwrite_seq file16kwrite_seq: (g=0): rw=write, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1 ... fio-3.22 Starting 10 threads Jobs: 7 (f=7): [W(6),(2),W(1),(1)][94.3%][w=400MiB/s][w=25.6k IOPS][eta 00m:03Jobs: 3 (f=3): [(4),W(2),(2),W(1),_(1)][96.2%][w=350MiB/s][w=22.4k IOPS][eta 00m:02s] file16kwrite_seq: (groupid=0, jobs=10): err= 0: pid=50533: Thu Aug 31 03:07:39 2023 write: IOPS=25.4k, BW=397MiB/s (416MB/s)(20.0GiB/51617msec); 0 zone resets clat (usec): min=105, max=5320, avg=381.05, stdev=93.92 lat (usec): min=105, max=5321, avg=381.80, stdev=93.93 clat percentiles (usec): | 1.00th=[ 176], 5.00th=[ 245], 10.00th=[ 281], 20.00th=[ 314], | 30.00th=[ 338], 40.00th=[ 359], 50.00th=[ 375], 60.00th=[ 396], | 70.00th=[ 416], 80.00th=[ 445], 90.00th=[ 486], 95.00th=[ 529], | 99.00th=[ 619], 99.50th=[ 652], 99.90th=[ 807], 99.95th=[ 1156], | 99.99th=[ 2278] bw ( KiB/s): min=388512, max=564103, per=100.00%, avg=413969.89, stdev=2435.10, samples=1004 iops : min=24282, max=35256, avg=25873.07, stdev=152.17, samples=1004 lat (usec) : 250=5.55%, 500=86.41%, 750=7.90%, 1000=0.08% lat (msec) : 2=0.04%, 4=0.02%, 10=0.01% cpu : usr=0.95%, sys=2.75%, ctx=1327195, majf=0, minf=0 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,1310720,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): WRITE: bw=397MiB/s (416MB/s), 397MiB/s-397MiB/s (416MB/s-416MB/s), io=20.0GiB (21.5GB), run=51617-51617msec

X86 1

xuhuai@loongsonlab-56:~$ fio -filename=/mnt/mem1/16k -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=60 -group_reporting -name=file16kwrite_rand file16kwrite_rand: (g=0): rw=write, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1 ... fio-3.33 Starting 10 threads Jobs: 10 (f=10): [W(10)][100.0%][w=16.6MiB/s][w=1065 IOPS][eta 00m:00s] file16kwrite_rand: (groupid=0, jobs=10): err= 0: pid=18704: Thu Aug 31 15:07:40 2023 write: IOPS=1052, BW=16.5MiB/s (17.3MB/s)(987MiB/60005msec); 0 zone resets clat (usec): min=4539, max=26747, avg=9482.53, stdev=1710.02 lat (usec): min=4539, max=26749, avg=9483.64, stdev=1710.02 clat percentiles (usec): | 1.00th=[ 6390], 5.00th=[ 7177], 10.00th=[ 7635], 20.00th=[ 8160], | 30.00th=[ 8586], 40.00th=[ 8979], 50.00th=[ 9241], 60.00th=[ 9634], | 70.00th=[10028], 80.00th=[10552], 90.00th=[11600], 95.00th=[12649], | 99.00th=[15139], 99.50th=[16188], 99.90th=[18482], 99.95th=[20055], | 99.99th=[22152] bw ( KiB/s): min=14912, max=18048, per=100.00%, avg=16856.47, stdev=47.15, samples=1190 iops : min= 932, max= 1128, avg=1053.53, stdev= 2.95, samples=1190 lat (msec) : 10=69.59%, 20=30.36%, 50=0.05% cpu : usr=0.14%, sys=0.43%, ctx=63247, majf=0, minf=0 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,63185,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): WRITE: bw=16.5MiB/s (17.3MB/s), 16.5MiB/s-16.5MiB/s (17.3MB/s-17.3MB/s), io=987MiB (1035MB), run=60005-60005msec

X86 2

[root@node-191 ~]# fio -filename=/mnt/mem1/16k -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file16kwrite_seq file16kwrite_seq: (g=0): rw=write, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1 ... fio-3.22 Starting 10 threads Jobs: 10 (f=10): [W(10)][100.0%][w=871MiB/s][w=55.7k IOPS][eta 00m:00s] file16kwrite_seq: (groupid=0, jobs=10): err= 0: pid=2744788: Fri Sep 1 04:01:11 2023 write: IOPS=55.5k, BW=866MiB/s (909MB/s)(20.0GiB/23636msec); 0 zone resets clat (usec): min=59, max=687, avg=179.59, stdev=33.09 lat (usec): min=60, max=687, avg=179.84, stdev=33.09 clat percentiles (usec): | 1.00th=[ 127], 5.00th=[ 143], 10.00th=[ 149], 20.00th=[ 159], | 30.00th=[ 165], 40.00th=[ 169], 50.00th=[ 176], 60.00th=[ 180], | 70.00th=[ 186], 80.00th=[ 196], 90.00th=[ 212], 95.00th=[ 237], | 99.00th=[ 310], 99.50th=[ 347], 99.90th=[ 433], 99.95th=[ 465], | 99.99th=[ 529] bw ( KiB/s): min=847872, max=915232, per=100.00%, avg=888209.02, stdev=1376.49, samples=470 iops : min=52992, max=57202, avg=55513.06, stdev=86.03, samples=470 lat (usec) : 100=0.06%, 250=96.34%, 500=3.58%, 750=0.02% cpu : usr=0.95%, sys=3.37%, ctx=1310781, majf=0, minf=1 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,1310720,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): WRITE: bw=866MiB/s (909MB/s), 866MiB/s-866MiB/s (909MB/s-909MB/s), io=20.0GiB (21.5GB), run=23636-23636msec

16k Rand写测试

fio -filename=/mnt/mem1/16k -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file16kwrite_rand

LA

[root@ceph192 mnt]# fio -filename=/mnt/mem1/16k -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file16kwrite_rand file16kwrite_rand: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1 ... fio-3.22 Starting 10 threads Jobs: 7 (f=7): [w(4),(2),w(2),(1),w(1)][96.1%][w=407MiB/s][w=26.1k IOPS][eta 0Jobs: 1 (f=1): [_(9),w(1)][100.0%][w=266MiB/s][w=17.0k IOPS][eta 00m:00s] file16kwrite_rand: (groupid=0, jobs=10): err= 0: pid=50596: Thu Aug 31 03:10:49 2023 write: IOPS=26.2k, BW=409MiB/s (429MB/s)(20.0GiB/50027msec); 0 zone resets clat (usec): min=109, max=2751, avg=369.77, stdev=78.95 lat (usec): min=110, max=2752, avg=370.51, stdev=78.96 clat percentiles (usec): | 1.00th=[ 192], 5.00th=[ 253], 10.00th=[ 281], 20.00th=[ 314], | 30.00th=[ 334], 40.00th=[ 351], 50.00th=[ 367], 60.00th=[ 383], | 70.00th=[ 400], 80.00th=[ 424], 90.00th=[ 457], 95.00th=[ 494], | 99.00th=[ 578], 99.50th=[ 627], 99.90th=[ 766], 99.95th=[ 881], | 99.99th=[ 2114] bw ( KiB/s): min=399456, max=570432, per=100.00%, avg=426501.75, stdev=2107.36, samples=974 iops : min=24966, max=35652, avg=26656.38, stdev=131.71, samples=974 lat (usec) : 250=4.65%, 500=91.08%, 750=4.16%, 1000=0.08% lat (msec) : 2=0.02%, 4=0.01% cpu : usr=1.13%, sys=2.86%, ctx=1310998, majf=0, minf=0 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,1310720,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): WRITE: bw=409MiB/s (429MB/s), 409MiB/s-409MiB/s (429MB/s-429MB/s), io=20.0GiB (21.5GB), run=50027-50027msec

X86

xuhuai@loongsonlab-56:~$ fio -filename=/mnt/mem1/16k -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=60 -group_reporting -name=file16kwrite_rand file16kwrite_rand: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1 ... fio-3.33 Starting 10 threads Jobs: 10 (f=10): [w(10)][100.0%][w=16.2MiB/s][w=1039 IOPS][eta 00m:00s] file16kwrite_rand: (groupid=0, jobs=10): err= 0: pid=18790: Thu Aug 31 15:10:46 2023 write: IOPS=1124, BW=17.6MiB/s (18.4MB/s)(1054MiB/60005msec); 0 zone resets clat (usec): min=3851, max=56140, avg=8884.05, stdev=1771.57 lat (usec): min=3852, max=56142, avg=8885.16, stdev=1771.57 clat percentiles (usec): | 1.00th=[ 5473], 5.00th=[ 6521], 10.00th=[ 6980], 20.00th=[ 7570], | 30.00th=[ 8029], 40.00th=[ 8356], 50.00th=[ 8717], 60.00th=[ 9110], | 70.00th=[ 9503], 80.00th=[10028], 90.00th=[10945], 95.00th=[11863], | 99.00th=[14353], 99.50th=[15270], 99.90th=[17695], 99.95th=[19268], | 99.99th=[53740] bw ( KiB/s): min=15264, max=25024, per=100.00%, avg=18006.05, stdev=128.67, samples=1190 iops : min= 954, max= 1564, avg=1125.38, stdev= 8.04, samples=1190 lat (msec) : 4=0.01%, 10=79.93%, 20=20.02%, 50=0.03%, 100=0.01% cpu : usr=0.16%, sys=0.47%, ctx=67536, majf=0, minf=0 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,67450,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): WRITE: bw=17.6MiB/s (18.4MB/s), 17.6MiB/s-17.6MiB/s (18.4MB/s-18.4MB/s), io=1054MiB (1105MB), run=60005-60005msec

X86 2

[root@node-191 ~]# fio -filename=/mnt/mem1/16k -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file16kwrite_rand file16kwrite_rand: (g=0): rw=randwrite, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1 ... fio-3.22 Starting 10 threads Jobs: 10 (f=10): [w(10)][100.0%][w=872MiB/s][w=55.8k IOPS][eta 00m:00s] file16kwrite_rand: (groupid=0, jobs=10): err= 0: pid=2744893: Fri Sep 1 04:02:04 2023 write: IOPS=55.3k, BW=865MiB/s (907MB/s)(20.0GiB/23682msec); 0 zone resets clat (usec): min=63, max=742, avg=179.81, stdev=33.14 lat (usec): min=64, max=743, avg=180.06, stdev=33.15 clat percentiles (usec): | 1.00th=[ 127], 5.00th=[ 143], 10.00th=[ 151], 20.00th=[ 159], | 30.00th=[ 165], 40.00th=[ 169], 50.00th=[ 176], 60.00th=[ 180], | 70.00th=[ 188], 80.00th=[ 196], 90.00th=[ 212], 95.00th=[ 237], | 99.00th=[ 306], 99.50th=[ 347], 99.90th=[ 445], 99.95th=[ 474], | 99.99th=[ 537] bw ( KiB/s): min=853824, max=914720, per=100.00%, avg=886436.77, stdev=1343.79, samples=470 iops : min=53364, max=57170, avg=55402.26, stdev=83.99, samples=470 lat (usec) : 100=0.05%, 250=96.44%, 500=3.48%, 750=0.03% cpu : usr=1.00%, sys=3.40%, ctx=1310768, majf=0, minf=1 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,1310720,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): WRITE: bw=865MiB/s (907MB/s), 865MiB/s-865MiB/s (907MB/s-907MB/s), io=20.0GiB (21.5GB), run=23682-23682msec

16k Seq读测试

fio -filename=/mnt/mem1/16k -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file16kread_seq

LA

[root@ceph192 mnt]# fio -filename=/mnt/mem1/16k -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file16kread_seq file16kread_seq: (g=0): rw=read, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1 ... fio-3.22 Starting 10 threads Jobs: 1 (f=1): [(7),R(1),(2)][100.0%][r=267MiB/s][r=17.1k IOPS][eta 00m:00s] file16kread_seq: (groupid=0, jobs=10): err= 0: pid=50648: Thu Aug 31 03:13:08 2023 read: IOPS=27.3k, BW=427MiB/s (448MB/s)(20.0GiB/47935msec) clat (usec): min=96, max=5412, avg=354.63, stdev=91.13 lat (usec): min=96, max=5413, avg=355.14, stdev=91.14 clat percentiles (usec): | 1.00th=[ 169], 5.00th=[ 231], 10.00th=[ 258], 20.00th=[ 289], | 30.00th=[ 310], 40.00th=[ 330], 50.00th=[ 347], 60.00th=[ 367], | 70.00th=[ 392], 80.00th=[ 416], 90.00th=[ 461], 95.00th=[ 502], | 99.00th=[ 586], 99.50th=[ 627], 99.90th=[ 783], 99.95th=[ 1139], | 99.99th=[ 2147] bw ( KiB/s): min=412864, max=621707, per=100.00%, avg=445635.61, stdev=3256.56, samples=933 iops : min=25804, max=38855, avg=27852.21, stdev=203.50, samples=933 lat (usec) : 100=0.01%, 250=8.49%, 500=86.48%, 750=4.91%, 1000=0.06% lat (msec) : 2=0.04%, 4=0.02%, 10=0.01% cpu : usr=0.74%, sys=2.96%, ctx=1319816, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1310720,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): READ: bw=427MiB/s (448MB/s), 427MiB/s-427MiB/s (448MB/s-448MB/s), io=20.0GiB (21.5GB), run=47935-47935msec

X86

xuhuai@loongsonlab-56:~$ fio -filename=/mnt/mem1/16k -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=60 -group_reporting -name=file16kread_seq file16kread_seq: (g=0): rw=read, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1 ... fio-3.33 Starting 10 threads Jobs: 10 (f=10): [R(10)][100.0%][r=14.8MiB/s][r=949 IOPS][eta 00m:00s] file16kread_seq: (groupid=0, jobs=10): err= 0: pid=19246: Thu Aug 31 15:19:26 2023 read: IOPS=985, BW=15.4MiB/s (16.1MB/s)(924MiB/60005msec) clat (usec): min=5132, max=86115, avg=10138.65, stdev=1518.30 lat (usec): min=5132, max=86116, avg=10139.35, stdev=1518.31 clat percentiles (usec): | 1.00th=[ 7373], 5.00th=[ 8356], 10.00th=[ 8848], 20.00th=[ 9241], | 30.00th=[ 9634], 40.00th=[ 9896], 50.00th=[10159], 60.00th=[10290], | 70.00th=[10552], 80.00th=[10945], 90.00th=[11469], 95.00th=[11994], | 99.00th=[13304], 99.50th=[13960], 99.90th=[16319], 99.95th=[19268], | 99.99th=[85459] bw ( KiB/s): min=13312, max=18208, per=100.00%, avg=15774.52, stdev=73.14, samples=1190 iops : min= 832, max= 1138, avg=985.90, stdev= 4.57, samples=1190 lat (msec) : 10=45.66%, 20=54.30%, 50=0.03%, 100=0.02% cpu : usr=0.11%, sys=0.38%, ctx=59155, majf=0, minf=40 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=59105,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): READ: bw=15.4MiB/s (16.1MB/s), 15.4MiB/s-15.4MiB/s (16.1MB/s-16.1MB/s), io=924MiB (968MB), run=60005-60005msec

X86 2

[root@node-191 ~]# fio -filename=/mnt/mem1/16k -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file16kread_seq file16kread_seq: (g=0): rw=read, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1 ... fio-3.22 Starting 10 threads Jobs: 10 (f=10): [R(10)][100.0%][r=878MiB/s][r=56.2k IOPS][eta 00m:00s] file16kread_seq: (groupid=0, jobs=10): err= 0: pid=2744967: Fri Sep 1 04:02:52 2023 read: IOPS=56.8k, BW=887MiB/s (930MB/s)(20.0GiB/23082msec) clat (usec): min=55, max=696, avg=175.57, stdev=30.35 lat (usec): min=55, max=696, avg=175.63, stdev=30.35 clat percentiles (usec): | 1.00th=[ 129], 5.00th=[ 143], 10.00th=[ 149], 20.00th=[ 155], | 30.00th=[ 161], 40.00th=[ 165], 50.00th=[ 169], 60.00th=[ 176], | 70.00th=[ 182], 80.00th=[ 190], 90.00th=[ 208], 95.00th=[ 233], | 99.00th=[ 293], 99.50th=[ 310], 99.90th=[ 359], 99.95th=[ 379], | 99.99th=[ 449] bw ( KiB/s): min=863584, max=940320, per=100.00%, avg=909645.91, stdev=1909.57, samples=460 iops : min=53974, max=58770, avg=56852.87, stdev=119.35, samples=460 lat (usec) : 100=0.03%, 250=96.61%, 500=3.36%, 750=0.01% cpu : usr=0.75%, sys=3.49%, ctx=1310763, majf=0, minf=41 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1310720,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): READ: bw=887MiB/s (930MB/s), 887MiB/s-887MiB/s (930MB/s-930MB/s), io=20.0GiB (21.5GB), run=23082-23082msec

16k Rand读测试

fio -filename=/mnt/mem1/16k -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file16kread_rand

LA

[root@ceph192 ~]# fio -filename=/mnt/mem1/16k -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file16kread_rand file16kread_rand: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1 ... fio-3.22 Starting 10 threads Jobs: 2 (f=2): [r(1),(6),r(1),(2)][97.9%][r=438MiB/s][r=28.0k IOPS][eta 00m:01s] file16kread_rand: (groupid=0, jobs=10): err= 0: pid=50814: Thu Aug 31 03:23:05 2023 read: IOPS=28.8k, BW=450MiB/s (472MB/s)(20.0GiB/45508msec) clat (usec): min=102, max=2510, avg=337.28, stdev=79.92 lat (usec): min=103, max=2510, avg=337.85, stdev=79.94 clat percentiles (usec): | 1.00th=[ 178], 5.00th=[ 229], 10.00th=[ 253], 20.00th=[ 281], | 30.00th=[ 302], 40.00th=[ 314], 50.00th=[ 330], 60.00th=[ 347], | 70.00th=[ 367], 80.00th=[ 392], 90.00th=[ 429], 95.00th=[ 461], | 99.00th=[ 537], 99.50th=[ 570], 99.90th=[ 709], 99.95th=[ 1090], | 99.99th=[ 2212] bw ( KiB/s): min=433902, max=581343, per=100.00%, avg=466946.77, stdev=2252.79, samples=890 iops : min=27114, max=36333, avg=29184.12, stdev=140.79, samples=890 lat (usec) : 250=9.20%, 500=88.63%, 750=2.09%, 1000=0.03% lat (msec) : 2=0.03%, 4=0.02% cpu : usr=0.87%, sys=3.29%, ctx=1396628, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1310720,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): READ: bw=450MiB/s (472MB/s), 450MiB/s-450MiB/s (472MB/s-472MB/s), io=20.0GiB (21.5GB), run=45508-45508msec

X86

xuhuai@loongsonlab-56:~$ fio -filename=/mnt/mem1/16k -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=60 -group_reporting -name=file16kread_rand file16kread_rand: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1 ... fio-3.33 Starting 10 threads Jobs: 10 (f=10): [r(10)][100.0%][r=15.8MiB/s][r=1012 IOPS][eta 00m:00s] file16kread_rand: (groupid=0, jobs=10): err= 0: pid=19490: Thu Aug 31 15:23:24 2023 read: IOPS=1032, BW=16.1MiB/s (16.9MB/s)(968MiB/60004msec) clat (usec): min=4412, max=42299, avg=9670.80, stdev=1362.93 lat (usec): min=4413, max=42299, avg=9671.53, stdev=1362.93 clat percentiles (usec): | 1.00th=[ 6063], 5.00th=[ 7504], 10.00th=[ 8094], 20.00th=[ 8717], | 30.00th=[ 9110], 40.00th=[ 9372], 50.00th=[ 9765], 60.00th=[10028], | 70.00th=[10290], 80.00th=[10683], 90.00th=[11207], 95.00th=[11600], | 99.00th=[12911], 99.50th=[13566], 99.90th=[15795], 99.95th=[16909], | 99.99th=[39584] bw ( KiB/s): min=13856, max=22816, per=100.00%, avg=16529.75, stdev=129.98, samples=1190 iops : min= 866, max= 1426, avg=1033.11, stdev= 8.12, samples=1190 lat (msec) : 10=59.95%, 20=40.04%, 50=0.02% cpu : usr=0.12%, sys=0.41%, ctx=61992, majf=0, minf=40 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=61956,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): READ: bw=16.1MiB/s (16.9MB/s), 16.1MiB/s-16.1MiB/s (16.9MB/s-16.9MB/s), io=968MiB (1015MB), run=60004-60004msec

X86 2

[root@node-191 ~]# fio -filename=/mnt/mem1/16k -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file16kread_rand file16kread_rand: (g=0): rw=randread, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=psync, iodepth=1 ... fio-3.22 Starting 10 threads Jobs: 10 (f=10): [r(10)][100.0%][r=870MiB/s][r=55.7k IOPS][eta 00m:00s] file16kread_rand: (groupid=0, jobs=10): err= 0: pid=2745284: Fri Sep 1 04:10:25 2023 read: IOPS=56.6k, BW=884MiB/s (927MB/s)(20.0GiB/23172msec) clat (usec): min=54, max=643, avg=176.13, stdev=28.80 lat (usec): min=54, max=643, avg=176.20, stdev=28.80 clat percentiles (usec): | 1.00th=[ 129], 5.00th=[ 143], 10.00th=[ 149], 20.00th=[ 157], | 30.00th=[ 163], 40.00th=[ 167], 50.00th=[ 172], 60.00th=[ 178], | 70.00th=[ 182], 80.00th=[ 192], 90.00th=[ 206], 95.00th=[ 227], | 99.00th=[ 293], 99.50th=[ 310], 99.90th=[ 359], 99.95th=[ 379], | 99.99th=[ 433] bw ( KiB/s): min=862208, max=940480, per=100.00%, avg=905840.15, stdev=1534.39, samples=460 iops : min=53888, max=58780, avg=56614.96, stdev=95.90, samples=460 lat (usec) : 100=0.03%, 250=97.22%, 500=2.75%, 750=0.01% cpu : usr=0.85%, sys=3.42%, ctx=1310767, majf=0, minf=41 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=1310720,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): READ: bw=884MiB/s (927MB/s), 884MiB/s-884MiB/s (927MB/s-927MB/s), io=20.0GiB (21.5GB), run=23172-23172msec

64k

64k Seq写测试

fio -filename=/mnt/mem1/64k -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=64k -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file64kwrite_seq

LA

[root@ceph192 ~]# fio -filename=/mnt/mem1/64k -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=64k -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file64kwrite_seq file64kwrite_seq: (g=0): rw=write, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=1 ... fio-3.22 Starting 10 threads file64kwrite_seq: Laying out IO file (1 file / 2048MiB) Jobs: 9 (f=9): [W(8),_(1),W(1)][95.0%][w=1104MiB/s][w=17.7k IOPS][eta 00m:01s] file64kwrite_seq: (groupid=0, jobs=10): err= 0: pid=50870: Thu Aug 31 03:24:23 2023 write: IOPS=17.6k, BW=1101MiB/s (1154MB/s)(20.0GiB/18608msec); 0 zone resets clat (usec): min=160, max=4740, avg=554.37, stdev=121.13 lat (usec): min=162, max=4741, avg=555.81, stdev=121.13 clat percentiles (usec): | 1.00th=[ 269], 5.00th=[ 359], 10.00th=[ 408], 20.00th=[ 469], | 30.00th=[ 502], 40.00th=[ 529], 50.00th=[ 562], 60.00th=[ 586], | 70.00th=[ 611], 80.00th=[ 644], 90.00th=[ 693], 95.00th=[ 734], | 99.00th=[ 824], 99.50th=[ 865], 99.90th=[ 1123], 99.95th=[ 1876], | 99.99th=[ 2507] bw ( MiB/s): min= 1079, max= 1137, per=100.00%, avg=1109.57, stdev= 1.49, samples=358 iops : min=17270, max=18206, avg=17752.92, stdev=23.86, samples=358 lat (usec) : 250=0.64%, 500=28.24%, 750=67.33%, 1000=3.66% lat (msec) : 2=0.09%, 4=0.04%, 10=0.01% cpu : usr=0.95%, sys=2.35%, ctx=337781, majf=0, minf=0 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,327680,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): WRITE: bw=1101MiB/s (1154MB/s), 1101MiB/s-1101MiB/s (1154MB/s-1154MB/s), io=20.0GiB (21.5GB), run=18608-18608msec

X86

xuhuai@loongsonlab-56:~$ fio -filename=/mnt/mem1/64k -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=64k -size=2G -numjobs=10 -runtime=60 -group_reporting -name=file64kwrite_seq file64kwrite_seq: (g=0): rw=write, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=1 ... fio-3.33 Starting 10 threads file64kwrite_seq: Laying out IO file (1 file / 2048MiB) Jobs: 10 (f=10): [W(10)][100.0%][w=64.8MiB/s][w=1036 IOPS][eta 00m:00s] file64kwrite_seq: (groupid=0, jobs=10): err= 0: pid=19598: Thu Aug 31 15:25:08 2023 write: IOPS=1048, BW=65.5MiB/s (68.7MB/s)(3932MiB/60007msec); 0 zone resets clat (usec): min=3606, max=26680, avg=9524.68, stdev=1720.54 lat (usec): min=3608, max=26683, avg=9527.44, stdev=1720.53 clat percentiles (usec): | 1.00th=[ 6456], 5.00th=[ 7177], 10.00th=[ 7635], 20.00th=[ 8225], | 30.00th=[ 8586], 40.00th=[ 8979], 50.00th=[ 9372], 60.00th=[ 9634], | 70.00th=[10028], 80.00th=[10552], 90.00th=[11600], 95.00th=[12649], | 99.00th=[15401], 99.50th=[16450], 99.90th=[18744], 99.95th=[20055], | 99.99th=[22152] bw ( KiB/s): min=62464, max=71168, per=100.00%, avg=67127.93, stdev=177.28, samples=1190 iops : min= 976, max= 1112, avg=1048.87, stdev= 2.77, samples=1190 lat (msec) : 4=0.01%, 10=69.12%, 20=30.83%, 50=0.05% cpu : usr=0.16%, sys=0.50%, ctx=62963, majf=0, minf=0 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,62905,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): WRITE: bw=65.5MiB/s (68.7MB/s), 65.5MiB/s-65.5MiB/s (68.7MB/s-68.7MB/s), io=3932MiB (4123MB), run=60007-60007msec

X86 2

[root@node-191 ~]# fio -filename=/mnt/mem1/64k -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=64k -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file64kwrite_seq file64kwrite_seq: (g=0): rw=write, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=1 ... fio-3.22 Starting 10 threads file64kwrite_seq: Laying out IO file (1 file / 2048MiB) Jobs: 10 (f=10): [W(10)][100.0%][w=2368MiB/s][w=37.9k IOPS][eta 00m:00s] file64kwrite_seq: (groupid=0, jobs=10): err= 0: pid=2745367: Fri Sep 1 04:10:50 2023 write: IOPS=38.2k, BW=2388MiB/s (2504MB/s)(20.0GiB/8576msec); 0 zone resets clat (usec): min=79, max=1248, avg=259.16, stdev=59.62 lat (usec): min=80, max=1253, avg=260.87, stdev=59.66 clat percentiles (usec): | 1.00th=[ 143], 5.00th=[ 176], 10.00th=[ 194], 20.00th=[ 212], | 30.00th=[ 227], 40.00th=[ 239], 50.00th=[ 253], 60.00th=[ 265], | 70.00th=[ 281], 80.00th=[ 302], 90.00th=[ 334], 95.00th=[ 367], | 99.00th=[ 437], 99.50th=[ 474], 99.90th=[ 570], 99.95th=[ 619], | 99.99th=[ 734] bw ( MiB/s): min= 2286, max= 2457, per=100.00%, avg=2392.35, stdev= 4.66, samples=170 iops : min=36584, max=39326, avg=38277.65, stdev=74.51, samples=170 lat (usec) : 100=0.03%, 250=48.33%, 500=51.34%, 750=0.30%, 1000=0.01% lat (msec) : 2=0.01% cpu : usr=1.26%, sys=3.59%, ctx=327729, majf=0, minf=1 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,327680,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): WRITE: bw=2388MiB/s (2504MB/s), 2388MiB/s-2388MiB/s (2504MB/s-2504MB/s), io=20.0GiB (21.5GB), run=8576-8576msec

64k Rand写测试

fio -filename=/mnt/mem1/64k -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=64k -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file64kwrite_rand

LA

[root@ceph192 ~]# fio -filename=/mnt/mem1/64k -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=64k -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file64kwrite_rand file64kwrite_rand: (g=0): rw=randwrite, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=1 ... fio-3.22 Starting 10 threads Jobs: 10 (f=10) Jobs: 7 (f=7): [(2),w(5),(1),w(2)][100.0%][w=1066MiB/s][w=17.1k IOPS][eta 00m:00s] file64kwrite_rand: (groupid=0, jobs=10): err= 0: pid=50936: Thu Aug 31 03:26:30 2023 write: IOPS=17.1k, BW=1070MiB/s (1122MB/s)(20.0GiB/19147msec); 0 zone resets clat (usec): min=156, max=2794, avg=570.86, stdev=109.08 lat (usec): min=157, max=2795, avg=572.27, stdev=109.08 clat percentiles (usec): | 1.00th=[ 285], 5.00th=[ 400], 10.00th=[ 449], 20.00th=[ 498], | 30.00th=[ 529], 40.00th=[ 553], 50.00th=[ 570], 60.00th=[ 594], | 70.00th=[ 619], 80.00th=[ 644], 90.00th=[ 685], 95.00th=[ 734], | 99.00th=[ 824], 99.50th=[ 865], 99.90th=[ 1123], 99.95th=[ 1876], | 99.99th=[ 2474] bw ( MiB/s): min= 1058, max= 1190, per=100.00%, avg=1079.17, stdev= 3.63, samples=372 iops : min=16938, max=19044, avg=17266.75, stdev=58.01, samples=372 lat (usec) : 250=0.59%, 500=19.99%, 750=75.92%, 1000=3.37% lat (msec) : 2=0.09%, 4=0.05% cpu : usr=1.03%, sys=2.36%, ctx=330152, majf=0, minf=0 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,327680,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): WRITE: bw=1070MiB/s (1122MB/s), 1070MiB/s-1070MiB/s (1122MB/s-1122MB/s), io=20.0GiB (21.5GB), run=19147-19147msec

X86

xuhuai@loongsonlab-56:~$ fio -filename=/mnt/mem1/64k -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=64k -size=2G -numjobs=10 -runtime=60 -group_reporting -name=file64kwrite_rand file64kwrite_rand: (g=0): rw=randwrite, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=1 ... fio-3.33 Starting 10 threads Jobs: 10 (f=10): [w(10)][100.0%][w=66.4MiB/s][w=1063 IOPS][eta 00m:00s] file64kwrite_rand: (groupid=0, jobs=10): err= 0: pid=19718: Thu Aug 31 15:27:14 2023 write: IOPS=1072, BW=67.0MiB/s (70.3MB/s)(4021MiB/60005msec); 0 zone resets clat (usec): min=4085, max=77223, avg=9312.18, stdev=1834.08 lat (usec): min=4088, max=77225, avg=9314.96, stdev=1834.09 clat percentiles (usec): | 1.00th=[ 6194], 5.00th=[ 6980], 10.00th=[ 7439], 20.00th=[ 8029], | 30.00th=[ 8455], 40.00th=[ 8717], 50.00th=[ 9110], 60.00th=[ 9372], | 70.00th=[ 9896], 80.00th=[10421], 90.00th=[11338], 95.00th=[12387], | 99.00th=[15008], 99.50th=[16188], 99.90th=[20055], 99.95th=[25035], | 99.99th=[45876] bw ( KiB/s): min=56960, max=81152, per=100.00%, avg=68657.48, stdev=281.43, samples=1190 iops : min= 890, max= 1268, avg=1072.77, stdev= 4.40, samples=1190 lat (msec) : 10=73.45%, 20=26.45%, 50=0.10%, 100=0.01% cpu : usr=0.15%, sys=0.54%, ctx=64397, majf=0, minf=0 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,64329,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): WRITE: bw=67.0MiB/s (70.3MB/s), 67.0MiB/s-67.0MiB/s (70.3MB/s-70.3MB/s), io=4021MiB (4216MB), run=60005-60005msec

X86 2

[root@node-191 ~]# fio -filename=/mnt/mem1/64k -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=64k -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file64kwrite_rand file64kwrite_rand: (g=0): rw=randwrite, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=1 ... fio-3.22 Starting 10 threads Jobs: 10 (f=10): [w(10)][100.0%][w=2389MiB/s][w=38.2k IOPS][eta 00m:00s] file64kwrite_rand: (groupid=0, jobs=10): err= 0: pid=2745440: Fri Sep 1 04:11:12 2023 write: IOPS=38.4k, BW=2402MiB/s (2519MB/s)(20.0GiB/8525msec); 0 zone resets clat (usec): min=83, max=1032, avg=257.64, stdev=58.49 lat (usec): min=84, max=1034, avg=259.30, stdev=58.53 clat percentiles (usec): | 1.00th=[ 145], 5.00th=[ 176], 10.00th=[ 192], 20.00th=[ 212], | 30.00th=[ 227], 40.00th=[ 239], 50.00th=[ 251], 60.00th=[ 265], | 70.00th=[ 281], 80.00th=[ 297], 90.00th=[ 330], 95.00th=[ 363], | 99.00th=[ 437], 99.50th=[ 469], 99.90th=[ 562], 99.95th=[ 611], | 99.99th=[ 791] bw ( MiB/s): min= 2280, max= 2474, per=100.00%, avg=2404.72, stdev= 4.26, samples=169 iops : min=36484, max=39588, avg=38475.66, stdev=68.16, samples=169 lat (usec) : 100=0.01%, 250=49.40%, 500=50.31%, 750=0.27%, 1000=0.01% lat (msec) : 2=0.01% cpu : usr=1.36%, sys=3.56%, ctx=327702, majf=0, minf=1 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,327680,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): WRITE: bw=2402MiB/s (2519MB/s), 2402MiB/s-2402MiB/s (2519MB/s-2519MB/s), io=20.0GiB (21.5GB), run=8525-8525msec

64k Seq读测试

fio -filename=/mnt/mem1/64k -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=64k -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file64kread_seq

LA

[root@ceph192 ~]# fio -filename=/mnt/mem1/64k -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=64k -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file64kread_seq file64kread_seq: (g=0): rw=read, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=1 ... fio-3.22 Starting 10 threads Jobs: 10 (f=10): [R(10)][100.0%][r=1081MiB/s][r=17.3k IOPS][eta 00m:00s] file64kread_seq: (groupid=0, jobs=10): err= 0: pid=50982: Thu Aug 31 03:28:09 2023 read: IOPS=17.9k, BW=1121MiB/s (1176MB/s)(20.0GiB/18265msec) clat (usec): min=146, max=5059, avg=550.46, stdev=117.20 lat (usec): min=146, max=5059, avg=551.09, stdev=117.19 clat percentiles (usec): | 1.00th=[ 318], 5.00th=[ 392], 10.00th=[ 424], 20.00th=[ 465], | 30.00th=[ 494], 40.00th=[ 519], 50.00th=[ 545], 60.00th=[ 570], | 70.00th=[ 594], 80.00th=[ 627], 90.00th=[ 676], 95.00th=[ 734], | 99.00th=[ 881], 99.50th=[ 955], 99.90th=[ 1172], 99.95th=[ 1909], | 99.99th=[ 2442] bw ( MiB/s): min= 1066, max= 1158, per=100.00%, avg=1125.07, stdev= 2.27, samples=360 iops : min=17064, max=18530, avg=18001.06, stdev=36.39, samples=360 lat (usec) : 250=0.37%, 500=31.82%, 750=63.77%, 1000=3.71% lat (msec) : 2=0.29%, 4=0.04%, 10=0.01% cpu : usr=0.39%, sys=2.68%, ctx=366626, majf=0, minf=40 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=327680,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): READ: bw=1121MiB/s (1176MB/s), 1121MiB/s-1121MiB/s (1176MB/s-1176MB/s), io=20.0GiB (21.5GB), run=18265-18265msec

X86

xuhuai@loongsonlab-56:~$ fio -filename=/mnt/mem1/64k -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=64k -size=2G -numjobs=10 -runtime=60 -group_reporting -name=file64kread_seq file64kread_seq: (g=0): rw=read, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=1 ... fio-3.33 Starting 10 threads Jobs: 10 (f=10): [R(10)][100.0%][r=58.4MiB/s][r=933 IOPS][eta 00m:00s] file64kread_seq: (groupid=0, jobs=10): err= 0: pid=19837: Thu Aug 31 15:28:55 2023 read: IOPS=970, BW=60.7MiB/s (63.6MB/s)(3640MiB/60008msec) clat (usec): min=3640, max=41188, avg=10291.90, stdev=1100.27 lat (usec): min=3641, max=41189, avg=10292.58, stdev=1100.27 clat percentiles (usec): | 1.00th=[ 7898], 5.00th=[ 8717], 10.00th=[ 9110], 20.00th=[ 9503], | 30.00th=[ 9765], 40.00th=[10028], 50.00th=[10290], 60.00th=[10421], | 70.00th=[10683], 80.00th=[11076], 90.00th=[11469], 95.00th=[11994], | 99.00th=[13173], 99.50th=[13829], 99.90th=[16057], 99.95th=[18220], | 99.99th=[38536] bw ( KiB/s): min=55680, max=68480, per=100.00%, avg=62153.14, stdev=187.66, samples=1190 iops : min= 870, max= 1070, avg=971.14, stdev= 2.93, samples=1190 lat (msec) : 4=0.01%, 10=39.02%, 20=60.96%, 50=0.03% cpu : usr=0.10%, sys=0.47%, ctx=58297, majf=0, minf=160 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=58239,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): READ: bw=60.7MiB/s (63.6MB/s), 60.7MiB/s-60.7MiB/s (63.6MB/s-63.6MB/s), io=3640MiB (3817MB), run=60008-60008msec

X86 2

[root@node-191 ~]# fio -filename=/mnt/mem1/64k -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=64k -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file64kread_seq file64kread_seq: (g=0): rw=read, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=1 ... fio-3.22 Starting 10 threads Jobs: 10 (f=10): [R(10)][90.0%][r=2123MiB/s][r=33.0k IOPS][eta 00m:01s] file64kread_seq: (groupid=0, jobs=10): err= 0: pid=2745510: Fri Sep 1 04:11:34 2023 read: IOPS=33.4k, BW=2087MiB/s (2189MB/s)(20.0GiB/9812msec) clat (usec): min=73, max=881, avg=298.74, stdev=74.16 lat (usec): min=73, max=882, avg=298.82, stdev=74.16 clat percentiles (usec): | 1.00th=[ 225], 5.00th=[ 241], 10.00th=[ 247], 20.00th=[ 253], | 30.00th=[ 262], 40.00th=[ 269], 50.00th=[ 273], 60.00th=[ 281], | 70.00th=[ 297], 80.00th=[ 322], 90.00th=[ 392], 95.00th=[ 457], | 99.00th=[ 611], 99.50th=[ 619], 99.90th=[ 660], 99.95th=[ 685], | 99.99th=[ 775] bw ( MiB/s): min= 1956, max= 2212, per=99.99%, avg=2087.01, stdev= 5.94, samples=190 iops : min=31300, max=35402, avg=33392.05, stdev=95.00, samples=190 lat (usec) : 100=0.01%, 250=14.41%, 500=81.87%, 750=3.71%, 1000=0.01% cpu : usr=0.55%, sys=3.44%, ctx=327760, majf=0, minf=161 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=327680,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): READ: bw=2087MiB/s (2189MB/s), 2087MiB/s-2087MiB/s (2189MB/s-2189MB/s), io=20.0GiB (21.5GB), run=9812-9812msec

64k Rand读测试

fio -filename=/mnt/mem1/64k -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=64k -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file64kread_rand

LA

[root@ceph192 ~]# fio -filename=/mnt/mem1/64k -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=64k -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file64kread_rand file64kread_rand: (g=0): rw=randread, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=1 ... fio-3.22 Starting 10 threads file64kread_rand: Laying out IO file (1 file / 2048MiB) Jobs: 10 (f=10): [r(10)][95.0%][r=1108MiB/s][r=17.7k IOPS][eta 00m:01s] file64kread_rand: (groupid=0, jobs=10): err= 0: pid=51710: Thu Aug 31 03:35:44 2023 read: IOPS=17.4k, BW=1091MiB/s (1143MB/s)(20.0GiB/18780msec) clat (usec): min=153, max=2680, avg=566.33, stdev=99.22 lat (usec): min=153, max=2680, avg=566.91, stdev=99.20 clat percentiles (usec): | 1.00th=[ 351], 5.00th=[ 412], 10.00th=[ 445], 20.00th=[ 494], | 30.00th=[ 529], 40.00th=[ 562], 50.00th=[ 578], 60.00th=[ 594], | 70.00th=[ 603], 80.00th=[ 619], 90.00th=[ 652], 95.00th=[ 701], | 99.00th=[ 840], 99.50th=[ 922], 99.90th=[ 1221], 99.95th=[ 1713], | 99.99th=[ 2180] bw ( MiB/s): min= 1054, max= 1137, per=100.00%, avg=1093.54, stdev= 1.94, samples=370 iops : min=16870, max=18204, avg=17496.59, stdev=31.02, samples=370 lat (usec) : 250=0.22%, 500=21.83%, 750=75.44%, 1000=2.22% lat (msec) : 2=0.27%, 4=0.02% cpu : usr=0.30%, sys=2.84%, ctx=341493, majf=0, minf=40 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=327680,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): READ: bw=1091MiB/s (1143MB/s), 1091MiB/s-1091MiB/s (1143MB/s-1143MB/s), io=20.0GiB (21.5GB), run=18780-18780msec

X86

xuhuai@loongsonlab-56:~$ fio -filename=/mnt/mem1/64k -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=64k -size=2G -numjobs=10 -runtime=60 -group_reporting -name=file64kread_rand file64kread_rand: (g=0): rw=randread, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=1 ... fio-3.33 Starting 10 threads Jobs: 10 (f=10): [r(10)][100.0%][r=58.7MiB/s][r=938 IOPS][eta 00m:00s] file64kread_rand: (groupid=0, jobs=10): err= 0: pid=20086: Thu Aug 31 15:35:06 2023 read: IOPS=961, BW=60.1MiB/s (63.0MB/s)(3606MiB/60006msec) clat (usec): min=5258, max=29652, avg=10386.14, stdev=1149.28 lat (usec): min=5259, max=29653, avg=10386.85, stdev=1149.28 clat percentiles (usec):大韩民国 | 1.00th=[ 7635], 5.00th=[ 8717], 10.00th=[ 9110], 20.00th=[ 9634], | 30.00th=[ 9896], 40.00th=[10159], 50.00th=[10290], 60.00th=[10552], | 70.00th=[10814], 80.00th=[11207], 90.00th=[11600], 95.00th=[12125], | 99.00th=[13435], 99.50th=[14222], 99.90th=[18482], 99.95th=[20579], | 99.99th=[27919] bw ( KiB/s): min=57600, max=71680, per=100.00%, avg=61581.98, stdev=227.81, samples=1190 iops : min= 900, max= 1120, avg=962.22, stdev= 3.56, samples=1190 lat (msec) : 10=35.13%, 20=64.80%, 50=0.06% cpu : usr=0.12%, sys=0.46%, ctx=57743, majf=0, minf=160 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=57695,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): READ: bw=60.1MiB/s (63.0MB/s), 60.1MiB/s-60.1MiB/s (63.0MB/s-63.0MB/s), io=3606MiB (3781MB), run=60006-60006msec

X86 2

[root@node-191 ~]# fio -filename=/mnt/mem1/64k -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=64k -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file64kread_rand file64kread_rand: (g=0): rw=randread, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=psync, iodepth=1 ... fio-3.22 Starting 10 threads Jobs: 10 (f=10): [r(10)][100.0%][r=2040MiB/s][r=32.6k IOPS][eta 00m:00s] file64kread_rand: (groupid=0, jobs=10): err= 0: pid=2745589: Fri Sep 1 04:11:57 2023 read: IOPS=32.7k, BW=2045MiB/s (2144MB/s)(20.0GiB/10016msec) clat (usec): min=76, max=982, avg=304.76, stdev=83.29 lat (usec): min=76, max=982, avg=304.85, stdev=83.29 clat percentiles (usec): | 1.00th=[ 225], 5.00th=[ 235], 10.00th=[ 243], 20.00th=[ 253], | 30.00th=[ 262], 40.00th=[ 269], 50.00th=[ 277], 60.00th=[ 285], | 70.00th=[ 302], 80.00th=[ 338], 90.00th=[ 412], 95.00th=[ 502], | 99.00th=[ 611], 99.50th=[ 627], 99.90th=[ 685], 99.95th=[ 709], | 99.99th=[ 791] bw ( MiB/s): min= 1840, max= 2200, per=100.00%, avg=2047.46, stdev= 9.60, samples=199 iops : min=29452, max=35212, avg=32759.38, stdev=153.56, samples=199 lat (usec) : 100=0.01%, 250=16.85%, 500=78.05%, 750=5.08%, 1000=0.02% cpu : usr=0.54%, sys=3.45%, ctx=327716, majf=0, minf=161 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=327680,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): READ: bw=2045MiB/s (2144MB/s), 2045MiB/s-2045MiB/s (2144MB/s-2144MB/s), io=20.0GiB (21.5GB), run=10016-10016msec

1m

1m Seq写测试

fio -filename=/mnt/mem1/1m -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=1m -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file1mwrite_seq

LA

[root@ceph192 ~]# fio -filename=/mnt/mem1/1m -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=1m -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file1mwrite_seq file1mwrite_seq: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1 ... fio-3.22 Starting 10 threads file1mwrite_seq: Laying out IO file (1 file / 2048MiB) Jobs: 10 (f=10): [W(10)][100.0%][w=2332MiB/s][w=2332 IOPS][eta 00m:00s] file1mwrite_seq: (groupid=0, jobs=10): err= 0: pid=51759: Thu Aug 31 03:36:41 2023 write: IOPS=2314, BW=2315MiB/s (2427MB/s)(20.0GiB/8848msec); 0 zone resets clat (usec): min=1433, max=5813, avg=4298.85, stdev=222.98 lat (usec): min=1446, max=5820, avg=4311.51, stdev=222.90 clat percentiles (usec): | 1.00th=[ 3687], 5.00th=[ 3884], 10.00th=[ 4047], 20.00th=[ 4228], | 30.00th=[ 4228], 40.00th=[ 4293], 50.00th=[ 4293], 60.00th=[ 4293], | 70.00th=[ 4359], 80.00th=[ 4424], 90.00th=[ 4555], 95.00th=[ 4686], | 99.00th=[ 4883], 99.50th=[ 4948], 99.90th=[ 5211], 99.95th=[ 5342], | 99.99th=[ 5669] bw ( MiB/s): min= 2299, max= 2340, per=100.00%, avg=2317.88, stdev= 1.27, samples=170 iops : min= 2299, max= 2340, avg=2317.47, stdev= 1.28, samples=170 lat (msec) : 2=0.02%, 4=8.66%, 10=91.32% cpu : usr=0.56%, sys=1.17%, ctx=20536, majf=0, minf=0 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,20480,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): WRITE: bw=2315MiB/s (2427MB/s), 2315MiB/s-2315MiB/s (2427MB/s-2427MB/s), io=20.0GiB (21.5GB), run=8848-8848msec

X86

xuhuai@loongsonlab-56:~$ fio -filename=/mnt/mem1/1m -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=1m -size=2G -numjobs=10 -runtime=60 -group_reporting -name=file1mwrite_seq file1mwrite_seq: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1 ... fio-3.33 Starting 10 threads file1mwrite_seq: Laying out IO file (1 file / 2048MiB) Jobs: 6 (f=6): [W(5),(1),W(1),(3)][100.0%][w=1045MiB/s][w=1045 IOPS][eta 00m:00s] file1mwrite_seq: (groupid=0, jobs=10): err= 0: pid=20199: Thu Aug 31 15:36:48 2023 write: IOPS=1209, BW=1210MiB/s (1269MB/s)(20.0GiB/16928msec); 0 zone resets clat (usec): min=2550, max=70080, avg=8131.78, stdev=2023.22 lat (usec): min=2627, max=70120, avg=8194.63, stdev=2028.35 clat percentiles (usec): | 1.00th=[ 4686], 5.00th=[ 5669], 10.00th=[ 6194], 20.00th=[ 6783], | 30.00th=[ 7242], 40.00th=[ 7635], 50.00th=[ 8029], 60.00th=[ 8455], | 70.00th=[ 8848], 80.00th=[ 9372], 90.00th=[10159], 95.00th=[10683], | 99.00th=[11994], 99.50th=[12649], 99.90th=[22414], 99.95th=[25560], | 99.99th=[68682] bw ( MiB/s): min= 1020, max= 1388, per=100.00%, avg=1218.18, stdev=10.14, samples=330 iops : min= 1020, max= 1388, avg=1218.18, stdev=10.14, samples=330 lat (msec) : 4=0.21%, 10=88.53%, 20=11.15%, 50=0.07%, 100=0.05% cpu : usr=0.95%, sys=2.15%, ctx=20577, majf=0, minf=0 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,20480,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): WRITE: bw=1210MiB/s (1269MB/s), 1210MiB/s-1210MiB/s (1269MB/s-1269MB/s), io=20.0GiB (21.5GB), run=16928-16928msec

X86 2

[root@node-191 ~]# fio -filename=/mnt/mem1/1m -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=1m -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file1mwrite_seq file1mwrite_seq: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1 ... fio-3.22 Starting 10 threads file1mwrite_seq: Laying out IO file (1 file / 2048MiB) Jobs: 10 (f=10): [W(10)][100.0%][w=3934MiB/s][w=3933 IOPS][eta 00m:00s] file1mwrite_seq: (groupid=0, jobs=10): err= 0: pid=2745659: Fri Sep 1 04:12:17 2023 write: IOPS=3947, BW=3948MiB/s (4139MB/s)(20.0GiB/5188msec); 0 zone resets clat (usec): min=608, max=9803, avg=2490.31, stdev=596.21 lat (usec): min=634, max=9840, avg=2521.00, stdev=597.24 clat percentiles (usec): | 1.00th=[ 1385], 5.00th=[ 1696], 10.00th=[ 1844], 20.00th=[ 2024], | 30.00th=[ 2147], 40.00th=[ 2278], 50.00th=[ 2409], 60.00th=[ 2540], | 70.00th=[ 2704], 80.00th=[ 2933], 90.00th=[ 3228], 95.00th=[ 3523], | 99.00th=[ 4293], 99.50th=[ 4621], 99.90th=[ 5604], 99.95th=[ 6194], | 99.99th=[ 9634] bw ( MiB/s): min= 3842, max= 4100, per=100.00%, avg=3962.80, stdev= 8.03, samples=100 iops : min= 3842, max= 4100, avg=3962.80, stdev= 8.03, samples=100 lat (usec) : 750=0.14%, 1000=0.05% lat (msec) : 2=18.52%, 4=79.63%, 10=1.65% cpu : usr=1.29%, sys=3.62%, ctx=20510, majf=0, minf=2 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,20480,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): WRITE: bw=3948MiB/s (4139MB/s), 3948MiB/s-3948MiB/s (4139MB/s-4139MB/s), io=20.0GiB (21.5GB), run=5188-5188msec

1m Rand写测试

fio -filename=/mnt/mem1/1m -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=1m -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file1mwrite_rand

LA

[root@ceph192 ~]# fio -filename=/mnt/mem1/1m -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=1m -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file1mwrite_rand file1mwrite_rand: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1 ... fio-3.22 Starting 10 threads Jobs: 10 (f=10): [w(10)][100.0%][w=2037MiB/s][w=2037 IOPS][eta 00m:00s] file1mwrite_rand: (groupid=0, jobs=10): err= 0: pid=51803: Thu Aug 31 03:37:50 2023 write: IOPS=2042, BW=2042MiB/s (2142MB/s)(20.0GiB/10027msec); 0 zone resets clat (usec): min=1511, max=6207, avg=4872.97, stdev=214.34 lat (usec): min=1524, max=6220, avg=4885.30, stdev=214.20 clat percentiles (usec): | 1.00th=[ 4293], 5.00th=[ 4490], 10.00th=[ 4686], 20.00th=[ 4752], | 30.00th=[ 4817], 40.00th=[ 4817], 50.00th=[ 4883], 60.00th=[ 4883], | 70.00th=[ 4948], 80.00th=[ 5014], 90.00th=[ 5080], 95.00th=[ 5211], | 99.00th=[ 5407], 99.50th=[ 5538], 99.90th=[ 5735], 99.95th=[ 5800], | 99.99th=[ 5997] bw ( MiB/s): min= 2021, max= 2060, per=100.00%, avg=2045.29, stdev= 1.13, samples=200 iops : min= 2021, max= 2060, avg=2044.40, stdev= 1.17, samples=200 lat (msec) : 2=0.03%, 4=0.19%, 10=99.79% cpu : usr=0.39%, sys=1.21%, ctx=20534, majf=0, minf=0 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,20480,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): WRITE: bw=2042MiB/s (2142MB/s), 2042MiB/s-2042MiB/s (2142MB/s-2142MB/s), io=20.0GiB (21.5GB), run=10027-10027msec

X86

xuhuai@loongsonlab-56:~$ fio -filename=/mnt/mem1/1m -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=1m -size=2G -numjobs=10 -runtime=60 -group_reporting -name=file1mwrite_rand file1mwrite_rand: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1 ... fio-3.33 Starting 10 threads Jobs: 10 (f=10): [w(10)][94.7%][w=1111MiB/s][w=1111 IOPS][eta 00m:01s] file1mwrite_rand: (groupid=0, jobs=10): err= 0: pid=20402: Thu Aug 31 15:37:51 2023 write: IOPS=1086, BW=1087MiB/s (1140MB/s)(20.0GiB/18843msec); 0 zone resets clat (usec): min=3102, max=16826, avg=9066.25, stdev=1533.30 lat (usec): min=3138, max=16851, avg=9147.58, stdev=1538.06 clat percentiles (usec): | 1.00th=[ 5407], 5.00th=[ 6587], 10.00th=[ 7111], 20.00th=[ 7767], | 30.00th=[ 8291], 40.00th=[ 8717], 50.00th=[ 9110], 60.00th=[ 9372], | 70.00th=[ 9896], 80.00th=[10290], 90.00th=[10945], 95.00th=[11600], | 99.00th=[12780], 99.50th=[13435], 99.90th=[14746], 99.95th=[15664], | 99.99th=[16188] bw ( MiB/s): min= 991, max= 1286, per=100.00%, avg=1088.32, stdev= 6.83, samples=370 iops : min= 991, max= 1286, avg=1088.30, stdev= 6.83, samples=370 lat (msec) : 4=0.05%, 10=73.41%, 20=26.54% cpu : usr=1.05%, sys=1.92%, ctx=20567, majf=0, minf=0 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,20480,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): WRITE: bw=1087MiB/s (1140MB/s), 1087MiB/s-1087MiB/s (1140MB/s-1140MB/s), io=20.0GiB (21.5GB), run=18843-18843msec

X86 2

[root@node-191 ~]# fio -filename=/mnt/mem1/1m -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=1m -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file1mwrite_rand file1mwrite_rand: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1 ... fio-3.22 Starting 10 threads Jobs: 10 (f=10): [w(10)][100.0%][w=3995MiB/s][w=3995 IOPS][eta 00m:00s] file1mwrite_rand: (groupid=0, jobs=10): err= 0: pid=2745732: Fri Sep 1 04:12:37 2023 write: IOPS=3980, BW=3981MiB/s (4174MB/s)(20.0GiB/5145msec); 0 zone resets clat (usec): min=626, max=7195, avg=2468.42, stdev=591.19 lat (usec): min=646, max=7227, avg=2498.69, stdev=592.28 clat percentiles (usec): | 1.00th=[ 1385], 5.00th=[ 1663], 10.00th=[ 1811], 20.00th=[ 1991], | 30.00th=[ 2147], 40.00th=[ 2245], 50.00th=[ 2376], 60.00th=[ 2540], | 70.00th=[ 2704], 80.00th=[ 2900], 90.00th=[ 3228], 95.00th=[ 3556], | 99.00th=[ 4228], 99.50th=[ 4490], 99.90th=[ 5342], 99.95th=[ 5735], | 99.99th=[ 6849] bw ( MiB/s): min= 3866, max= 4130, per=100.00%, avg=3990.92, stdev= 8.26, samples=100 iops : min= 3866, max= 4130, avg=3990.90, stdev= 8.26, samples=100 lat (usec) : 750=0.05%, 1000=0.06% lat (msec) : 2=20.56%, 4=77.53%, 10=1.79% cpu : usr=1.38%, sys=3.53%, ctx=20509, majf=0, minf=1 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,20480,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): WRITE: bw=3981MiB/s (4174MB/s), 3981MiB/s-3981MiB/s (4174MB/s-4174MB/s), io=20.0GiB (21.5GB), run=5145-5145msec

1m Seq读测试

fio -filename=/mnt/mem1/1m -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=1m -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file1mread_seq

LA

[root@ceph192 ~]# fio -filename=/mnt/mem1/1m -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=1m -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file1mread_seq file1mread_seq: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1 ... fio-3.22 Starting 10 threads Jobs: 10 (f=10): [R(10)][100.0%][r=2516MiB/s][r=2515 IOPS][eta 00m:00s] file1mread_seq: (groupid=0, jobs=10): err= 0: pid=51829: Thu Aug 31 03:38:18 2023 read: IOPS=2575, BW=2575MiB/s (2700MB/s)(20.0GiB/7953msec) clat (usec): min=736, max=8135, avg=3868.31, stdev=636.91 lat (usec): min=736, max=8136, avg=3868.96, stdev=636.89 clat percentiles (usec): | 1.00th=[ 2540], 5.00th=[ 3163], 10.00th=[ 3359], 20.00th=[ 3556], | 30.00th=[ 3621], 40.00th=[ 3687], 50.00th=[ 3752], 60.00th=[ 3818], | 70.00th=[ 3916], 80.00th=[ 4080], 90.00th=[ 4424], 95.00th=[ 5669], | 99.00th=[ 5932], 99.50th=[ 6063], 99.90th=[ 6980], 99.95th=[ 7177], | 99.99th=[ 7898] bw ( MiB/s): min= 2278, max= 2698, per=100.00%, avg=2579.02, stdev=12.74, samples=150 iops : min= 2278, max= 2698, avg=2578.40, stdev=12.69, samples=150 lat (usec) : 750=0.01%, 1000=0.03% lat (msec) : 2=0.24%, 4=74.79%, 10=24.93% cpu : usr=0.12%, sys=1.30%, ctx=20774, majf=0, minf=640 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=20480,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): READ: bw=2575MiB/s (2700MB/s), 2575MiB/s-2575MiB/s (2700MB/s-2700MB/s), io=20.0GiB (21.5GB), run=7953-7953msec

X86

xuhuai@loongsonlab-56:~$ fio -filename=/mnt/mem1/1m -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=1m -size=2G -numjobs=10 -runtime=60 -group_reporting -name=file1mread_seq file1mread_seq: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1 ... fio-3.33 Starting 10 threads Jobs: 10 (f=10): [R(10)][100.0%][r=959MiB/s][r=959 IOPS][eta 00m:00s] file1mread_seq: (groupid=0, jobs=10): err= 0: pid=20483: Thu Aug 31 15:38:42 2023 read: IOPS=961, BW=962MiB/s (1008MB/s)(20.0GiB/21294msec) clat (usec): min=3009, max=58833, avg=10364.03, stdev=1447.52 lat (usec): min=3010, max=58834, avg=10364.79, stdev=1447.53 clat percentiles (usec): | 1.00th=[ 8717], 5.00th=[ 9110], 10.00th=[ 9372], 20.00th=[ 9634], | 30.00th=[ 9765], 40.00th=[ 9896], 50.00th=[10159], 60.00th=[10290], | 70.00th=[10552], 80.00th=[10945], 90.00th=[11600], 95.00th=[12387], | 99.00th=[13829], 99.50th=[14484], 99.90th=[16450], 99.95th=[17957], | 99.99th=[56886] bw ( KiB/s): min=860848, max=1028096, per=100.00%, avg=986380.48, stdev=3305.89, samples=420 iops : min= 840, max= 1004, avg=963.24, stdev= 3.24, samples=420 lat (msec) : 4=0.01%, 10=43.72%, 20=56.22%, 100=0.05% cpu : usr=0.11%, sys=1.64%, ctx=20533, majf=0, minf=1042 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=20480,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): READ: bw=962MiB/s (1008MB/s), 962MiB/s-962MiB/s (1008MB/s-1008MB/s), io=20.0GiB (21.5GB), run=21294-21294msec

X86 2

[root@node-191 ~]# fio -filename=/mnt/mem1/1m -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=1m -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file1mread_seq file1mread_seq: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1 ... fio-3.22 Starting 10 threads Jobs: 10 (f=10): [R(10)][100.0%][r=3826MiB/s][r=3825 IOPS][eta 00m:00s] file1mread_seq: (groupid=0, jobs=10): err= 0: pid=2745810: Fri Sep 1 04:12:55 2023 read: IOPS=3872, BW=3872MiB/s (4060MB/s)(20.0GiB/5289msec) clat (usec): min=446, max=7990, avg=2577.26, stdev=253.44 lat (usec): min=446, max=7991, avg=2577.50, stdev=253.45 clat percentiles (usec): | 1.00th=[ 2040], 5.00th=[ 2245], 10.00th=[ 2376], 20.00th=[ 2474], | 30.00th=[ 2507], 40.00th=[ 2540], 50.00th=[ 2573], 60.00th=[ 2606], | 70.00th=[ 2638], 80.00th=[ 2671], 90.00th=[ 2769], 95.00th=[ 2868], | 99.00th=[ 3458], 99.50th=[ 3851], 99.90th=[ 5276], 99.95th=[ 5800], | 99.99th=[ 6849] bw ( MiB/s): min= 3668, max= 4018, per=100.00%, avg=3877.05, stdev=10.08, samples=100 iops : min= 3668, max= 4018, avg=3877.00, stdev=10.09, samples=100 lat (usec) : 500=0.02%, 750=0.01% lat (msec) : 2=0.66%, 4=98.92%, 10=0.39% cpu : usr=0.17%, sys=4.50%, ctx=20538, majf=0, minf=2055 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=20480,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): READ: bw=3872MiB/s (4060MB/s), 3872MiB/s-3872MiB/s (4060MB/s-4060MB/s), io=20.0GiB (21.5GB), run=5289-5289msec

1m Rand读测试

fio -filename=/mnt/mem1/1m -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=1m -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file1mread_rand

LA

[root@ceph192 ~]# fio -filename=/mnt/mem1/1m -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=1m -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file1mread_rand file1mread_rand: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1 ... fio-3.22 Starting 10 threads Jobs: 10 (f=10): [r(10)][100.0%][r=2416MiB/s][r=2416 IOPS][eta 00m:00s] file1mread_rand: (groupid=0, jobs=10): err= 0: pid=51846: Thu Aug 31 03:39:05 2023 read: IOPS=2465, BW=2465MiB/s (2585MB/s)(20.0GiB/8308msec) clat (usec): min=800, max=8163, avg=4041.99, stdev=766.98 lat (usec): min=801, max=8164, avg=4042.67, stdev=766.96 clat percentiles (usec): | 1.00th=[ 2573], 5.00th=[ 3195], 10.00th=[ 3359], 20.00th=[ 3589], | 30.00th=[ 3687], 40.00th=[ 3752], 50.00th=[ 3884], 60.00th=[ 4015], | 70.00th=[ 4178], 80.00th=[ 4359], 90.00th=[ 4817], 95.00th=[ 5932], | 99.00th=[ 6652], 99.50th=[ 6718], 99.90th=[ 7046], 99.95th=[ 7242], | 99.99th=[ 7898] bw ( MiB/s): min= 2141, max= 2625, per=100.00%, avg=2465.18, stdev=14.19, samples=160 iops : min= 2136, max= 2622, avg=2463.25, stdev=14.26, samples=160 lat (usec) : 1000=0.03% lat (msec) : 2=0.22%, 4=58.43%, 10=41.31% cpu : usr=0.19%, sys=1.27%, ctx=20599, majf=0, minf=640 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=20480,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): READ: bw=2465MiB/s (2585MB/s), 2465MiB/s-2465MiB/s (2585MB/s-2585MB/s), io=20.0GiB (21.5GB), run=8308-8308msec

X86

xuhuai@loongsonlab-56:~$ fio -filename=/mnt/mem1/1m -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=1m -size=2G -numjobs=10 -runtime=60 -group_reporting -name=file1mread_rand file1mread_rand: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1 ... fio-3.33 Starting 10 threads Jobs: 10 (f=10): [r(10)][100.0%][r=938MiB/s][r=937 IOPS][eta 00m:00s] file1mread_rand: (groupid=0, jobs=10): err= 0: pid=20561: Thu Aug 31 15:39:23 2023 read: IOPS=965, BW=966MiB/s (1012MB/s)(20.0GiB/21210msec) clat (usec): min=2080, max=25804, avg=10312.17, stdev=1421.04 lat (usec): min=2080, max=25805, avg=10312.86, stdev=1421.05 clat percentiles (usec): | 1.00th=[ 7242], 5.00th=[ 8029], 10.00th=[ 8586], 20.00th=[ 9241], | 30.00th=[ 9634], 40.00th=[10028], 50.00th=[10290], 60.00th=[10552], | 70.00th=[10945], 80.00th=[11338], 90.00th=[11994], 95.00th=[12649], | 99.00th=[13960], 99.50th=[14484], 99.90th=[16188], 99.95th=[18744], | 99.99th=[25560] bw ( KiB/s): min=919552, max=1132544, per=100.00%, avg=989418.19, stdev=5190.91, samples=420 iops : min= 898, max= 1106, avg=966.19, stdev= 5.07, samples=420 lat (msec) : 4=0.06%, 10=39.97%, 20=59.92%, 50=0.05% cpu : usr=0.13%, sys=1.80%, ctx=20534, majf=0, minf=2054 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=20480,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): READ: bw=966MiB/s (1012MB/s), 966MiB/s-966MiB/s (1012MB/s-1012MB/s), io=20.0GiB (21.5GB), run=21210-21210msec

X86 2

[root@node-191 ~]# fio -filename=/mnt/mem1/1m -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=1m -size=2G -numjobs=10 -runtime=600 -group_reporting -name=file1mread_rand file1mread_rand: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=psync, iodepth=1 ... fio-3.22 Starting 10 threads Jobs: 10 (f=10): [r(10)][100.0%][r=3657MiB/s][r=3656 IOPS][eta 00m:00s] file1mread_rand: (groupid=0, jobs=10): err= 0: pid=2745878: Fri Sep 1 04:13:13 2023 read: IOPS=3772, BW=3772MiB/s (3956MB/s)(20.0GiB/5429msec) clat (usec): min=542, max=6264, avg=2645.44, stdev=232.60 lat (usec): min=542, max=6264, avg=2645.64, stdev=232.59 clat percentiles (usec): | 1.00th=[ 2089], 5.00th=[ 2311], 10.00th=[ 2442], 20.00th=[ 2507], | 30.00th=[ 2573], 40.00th=[ 2606], 50.00th=[ 2638], 60.00th=[ 2671], | 70.00th=[ 2704], 80.00th=[ 2769], 90.00th=[ 2835], 95.00th=[ 2900], | 99.00th=[ 3261], 99.50th=[ 3523], 99.90th=[ 5407], 99.95th=[ 5932], | 99.99th=[ 6259] bw ( MiB/s): min= 3610, max= 3892, per=100.00%, avg=3774.80, stdev= 8.40, samples=100 iops : min= 3610, max= 3892, avg=3774.80, stdev= 8.40, samples=100 lat (usec) : 750=0.01%, 1000=0.02% lat (msec) : 2=0.43%, 4=99.27%, 10=0.27% cpu : usr=0.13%, sys=3.71%, ctx=20521, majf=0, minf=1043 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=20480,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs): READ: bw=3772MiB/s (3956MB/s), 3772MiB/s-3772MiB/s (3956MB/s-3956MB/s), io=20.0GiB (21.5GB), run=5429-5429msec

Anyway

16k write libaio No_rand iodepth_128 numjobs_1

fio -filename=/mnt/mem1/16k -direct=1 -iodepth 128 -thread -rw=write -ioengine=libaio -bs=16k -size=2G -numjobs=1 -runtime=600 -group_reporting -name=file16kwrite_seq

LA

[root@ceph192 ~]# fio -filename=/mnt/mem1/16k -direct=1 -iodepth 128 -thread -rw=write -ioengine=libaio -bs=16k -size=2G -numjobs=1 -runtime=600 -group_reporting -name=file16kwrite_seq file16kwrite_seq: (g=0): rw=write, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=128 fio-3.22 Starting 1 thread file16kwrite_seq: Laying out IO file (1 file / 2048MiB) Jobs: 1 (f=1): [W(1)][100.0%][w=436MiB/s][w=27.9k IOPS][eta 00m:00s] file16kwrite_seq: (groupid=0, jobs=1): err= 0: pid=51900: Thu Aug 31 03:40:33 2023 write: IOPS=27.9k, BW=436MiB/s (458MB/s)(2048MiB/4693msec); 0 zone resets slat (nsec): min=3070, max=42160, avg=5104.98, stdev=743.49 clat (usec): min=795, max=8566, avg=4574.56, stdev=349.47 lat (usec): min=802, max=8572, avg=4580.11, stdev=349.54 clat percentiles (usec): | 1.00th=[ 3359], 5.00th=[ 4178], 10.00th=[ 4293], 20.00th=[ 4424], | 30.00th=[ 4490], 40.00th=[ 4555], 50.00th=[ 4555], 60.00th=[ 4621], | 70.00th=[ 4686], 80.00th=[ 4752], 90.00th=[ 4817], 95.00th=[ 4948], | 99.00th=[ 5735], 99.50th=[ 6194], 99.90th=[ 6915], 99.95th=[ 7046], | 99.99th=[ 8094] bw ( KiB/s): min=444128, max=459712, per=100.00%, avg=447505.78, stdev=4906.74, samples=9 iops : min=27758, max=28732, avg=27969.11, stdev=306.67, samples=9 lat (usec) : 1000=0.01% lat (msec) : 2=0.02%, 4=3.40%, 10=96.58% cpu : usr=2.45%, sys=26.60%, ctx=102406, majf=0, minf=0 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,131072,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs): WRITE: bw=436MiB/s (458MB/s), 436MiB/s-436MiB/s (458MB/s-458MB/s), io=2048MiB (2147MB), run=4693-4693msec

X86

xuhuai@loongsonlab-56:~$ fio -filename=/mnt/mem1/16k -direct=1 -iodepth 128 -thread -rw=write -ioengine=libaio -bs=16k -size=2G -numjobs=1 -runtime=60 -group_reporting -name=file16kwrite_seq file16kwrite_seq: (g=0): rw=write, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=128 fio-3.33 Starting 1 thread Jobs: 1 (f=1): [W(1)][100.0%][w=16.4MiB/s][w=1051 IOPS][eta 00m:00s] file16kwrite_seq: (groupid=0, jobs=1): err= 0: pid=20772: Thu Aug 31 15:43:08 2023 write: IOPS=1052, BW=16.5MiB/s (17.3MB/s)(988MiB/60075msec); 0 zone resets slat (usec): min=3, max=323, avg=23.73, stdev= 8.95 clat (msec): min=7, max=194, avg=121.51, stdev= 6.02 lat (msec): min=7, max=194, avg=121.54, stdev= 6.02 clat percentiles (msec): | 1.00th=[ 109], 5.00th=[ 114], 10.00th=[ 116], 20.00th=[ 118], | 30.00th=[ 120], 40.00th=[ 121], 50.00th=[ 122], 60.00th=[ 123], | 70.00th=[ 124], 80.00th=[ 125], 90.00th=[ 127], 95.00th=[ 129], | 99.00th=[ 140], 99.50th=[ 144], 99.90th=[ 180], 99.95th=[ 182], | 99.99th=[ 188] bw ( KiB/s): min=14304, max=18304, per=99.92%, avg=16834.93, stdev=484.40, samples=120 iops : min= 894, max= 1144, avg=1052.18, stdev=30.27, samples=120 lat (msec) : 10=0.01%, 20=0.01%, 50=0.02%, 100=0.12%, 250=99.85% cpu : usr=1.12%, sys=4.00%, ctx=63239, majf=0, minf=63 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,63258,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs): WRITE: bw=16.5MiB/s (17.3MB/s), 16.5MiB/s-16.5MiB/s (17.3MB/s-17.3MB/s), io=988MiB (1036MB), run=60075-60075msec

X86 2

[root@node-191 ~]# fio -filename=/mnt/mem1/16k -direct=1 -iodepth 128 -thread -rw=write -ioengine=libaio -bs=16k -size=2G -numjobs=1 -runtime=600 -group_reporting -name=file16kwrite_seq file16kwrite_seq: (g=0): rw=write, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=128 fio-3.22 Starting 1 thread Jobs: 1 (f=1) file16kwrite_seq: (groupid=0, jobs=1): err= 0: pid=2745950: Fri Sep 1 04:13:28 2023 write: IOPS=54.8k, BW=856MiB/s (897MB/s)(2048MiB/2393msec); 0 zone resets slat (nsec): min=2190, max=76940, avg=4543.53, stdev=1154.25 clat (usec): min=635, max=5527, avg=2330.78, stdev=250.71 lat (usec): min=640, max=5532, avg=2335.36, stdev=250.81 clat percentiles (usec): | 1.00th=[ 2040], 5.00th=[ 2089], 10.00th=[ 2114], 20.00th=[ 2147], | 30.00th=[ 2180], 40.00th=[ 2245], 50.00th=[ 2278], 60.00th=[ 2311], | 70.00th=[ 2376], 80.00th=[ 2442], 90.00th=[ 2606], 95.00th=[ 2769], | 99.00th=[ 3294], 99.50th=[ 3621], 99.90th=[ 4424], 99.95th=[ 5014], | 99.99th=[ 5473] bw ( KiB/s): min=862976, max=895872, per=100.00%, avg=880456.00, stdev=14473.96, samples=4 iops : min=53936, max=55992, avg=55028.50, stdev=904.62, samples=4 lat (usec) : 750=0.01%, 1000=0.01% lat (msec) : 2=0.53%, 4=99.32%, 10=0.14% cpu : usr=9.28%, sys=30.43%, ctx=99717, majf=0, minf=2 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,131072,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs): WRITE: bw=856MiB/s (897MB/s), 856MiB/s-856MiB/s (897MB/s-897MB/s), io=2048MiB (2147MB), run=2393-2393msec

16k read libaio No_rand iodepth_128 numjobs_1

fio -filename=/mnt/mem1/16k -direct=1 -iodepth 128 -thread -rw=read -ioengine=libaio -bs=16k -size=2G -numjobs=1 -runtime=600 -group_reporting -name=file16kwrite_seq

LA

[root@ceph192 ~]# fio -filename=/mnt/mem1/16k -direct=1 -iodepth 128 -thread -rw=read -ioengine=libaio -bs=16k -size=2G -numjobs=1 -runtime=600 -group_reporting -name=file16kwrite_seq file16kwrite_seq: (g=0): rw=read, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=128 fio-3.22 Starting 1 thread Jobs: 1 (f=1): [R(1)][100.0%][r=484MiB/s][r=30.0k IOPS][eta 00m:00s] file16kwrite_seq: (groupid=0, jobs=1): err= 0: pid=51946: Thu Aug 31 03:43:41 2023 read: IOPS=30.8k, BW=481MiB/s (504MB/s)(2048MiB/4260msec) slat (nsec): min=2580, max=49830, avg=4812.34, stdev=1096.46 clat (usec): min=1053, max=9127, avg=4150.72, stdev=474.74 lat (usec): min=1061, max=9133, avg=4155.98, stdev=474.78 clat percentiles (usec): | 1.00th=[ 2868], 5.00th=[ 3490], 10.00th=[ 3654], 20.00th=[ 3818], | 30.00th=[ 3916], 40.00th=[ 4015], 50.00th=[ 4113], 60.00th=[ 4228], | 70.00th=[ 4359], 80.00th=[ 4490], 90.00th=[ 4686], 95.00th=[ 4883], | 99.00th=[ 5538], 99.50th=[ 5800], 99.90th=[ 6259], 99.95th=[ 6587], | 99.99th=[ 8717] bw ( KiB/s): min=470944, max=518912, per=100.00%, avg=494028.00, stdev=18219.10, samples=8 iops : min=29434, max=32432, avg=30876.75, stdev=1138.69, samples=8 lat (msec) : 2=0.03%, 4=38.31%, 10=61.66% cpu : usr=3.87%, sys=25.80%, ctx=81782, majf=0, minf=128 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=131072,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs): READ: bw=481MiB/s (504MB/s), 481MiB/s-481MiB/s (504MB/s-504MB/s), io=2048MiB (2147MB), run=4260-4260msec

X86

xuhuai@loongsonlab-56:~$ fio -filename=/mnt/mem1/16k -direct=1 -iodepth 128 -thread -rw=read -ioengine=libaio -bs=16k -size=2G -numjobs=1 -runtime=60 -group_reporting -name=file16kwrite_seq file16kwrite_seq: (g=0): rw=read, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=128 fio-3.33 Starting 1 thread Jobs: 1 (f=1): [R(1)][100.0%][r=15.1MiB/s][r=966 IOPS][eta 00m:00s] file16kwrite_seq: (groupid=0, jobs=1): err= 0: pid=20855: Thu Aug 31 15:44:45 2023 read: IOPS=959, BW=15.0MiB/s (15.7MB/s)(901MiB/60078msec) slat (usec): min=4, max=531, avg=23.89, stdev= 7.57 clat (msec): min=8, max=211, avg=133.31, stdev= 7.16 lat (msec): min=8, max=211, avg=133.34, stdev= 7.16 clat percentiles (msec): | 1.00th=[ 115], 5.00th=[ 124], 10.00th=[ 127], 20.00th=[ 131], | 30.00th=[ 132], 40.00th=[ 134], 50.00th=[ 134], 60.00th=[ 136], | 70.00th=[ 136], 80.00th=[ 138], 90.00th=[ 138], 95.00th=[ 140], | 99.00th=[ 150], 99.50th=[ 171], 99.90th=[ 207], 99.95th=[ 209], | 99.99th=[ 211] bw ( KiB/s): min=13920, max=17280, per=99.91%, avg=15342.67, stdev=500.46, samples=120 iops : min= 870, max= 1080, avg=958.92, stdev=31.28, samples=120 lat (msec) : 10=0.01%, 20=0.01%, 50=0.02%, 100=0.12%, 250=99.85% cpu : usr=0.98%, sys=3.90%, ctx=57678, majf=0, minf=571 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=57662,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs): READ: bw=15.0MiB/s (15.7MB/s), 15.0MiB/s-15.0MiB/s (15.7MB/s-15.7MB/s), io=901MiB (945MB), run=60078-60078msec

X86 2

[root@node-191 ~]# fio -filename=/mnt/mem1/16k -direct=1 -iodepth 128 -thread -rw=read -ioengine=libaio -bs=16k -size=2G -numjobs=1 -runtime=600 -group_reporting -name=file16kwrite_seq file16kwrite_seq: (g=0): rw=read, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=libaio, iodepth=128 fio-3.22 Starting 1 thread Jobs: 1 (f=1) file16kwrite_seq: (groupid=0, jobs=1): err= 0: pid=2746024: Fri Sep 1 04:13:55 2023 read: IOPS=55.6k, BW=869MiB/s (911MB/s)(2048MiB/2356msec) slat (nsec): min=1749, max=39694, avg=3998.16, stdev=1184.86 clat (usec): min=1461, max=4455, avg=2294.29, stdev=338.58 lat (usec): min=1465, max=4460, avg=2298.32, stdev=338.61 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1975], 10.00th=[ 2008], 20.00th=[ 2057], | 30.00th=[ 2114], 40.00th=[ 2147], 50.00th=[ 2212], 60.00th=[ 2278], | 70.00th=[ 2343], 80.00th=[ 2474], 90.00th=[ 2737], 95.00th=[ 3064], | 99.00th=[ 3556], 99.50th=[ 3654], 99.90th=[ 4113], 99.95th=[ 4228], | 99.99th=[ 4359] bw ( KiB/s): min=866491, max=909504, per=99.48%, avg=885534.75, stdev=22194.85, samples=4 iops : min=54155, max=56844, avg=55345.75, stdev=1387.38, samples=4 lat (msec) : 2=7.75%, 4=92.11%, 10=0.15% cpu : usr=7.60%, sys=30.83%, ctx=108990, majf=0, minf=514 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=131072,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs): READ: bw=869MiB/s (911MB/s), 869MiB/s-869MiB/s (911MB/s-911MB/s), io=2048MiB (2147MB), run=2356-2356msec

64k write libaio No_rand iodepth_128 numjobs_1

fio -filename=/mnt/mem1/64k -direct=1 -iodepth 128 -thread -rw=write -ioengine=libaio -bs=64k -size=4G -numjobs=1 -runtime=600 -group_reporting -name=file16kwrite_seq

LA

[root@ceph192 ~]# fio -filename=/mnt/mem1/64k -direct=1 -iodepth 128 -thread -rw=write -ioengine=libaio -bs=64k -size=4G -numjobs=1 -runtime=600 -group_reporting -name=file16kwrite_seq file16kwrite_seq: (g=0): rw=write, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=128 fio-3.22 Starting 1 thread file16kwrite_seq: Laying out IO file (1 file / 4096MiB) Jobs: 1 (f=1): [W(1)][100.0%][w=1104MiB/s][w=17.7k IOPS][eta 00m:00s] file16kwrite_seq: (groupid=0, jobs=1): err= 0: pid=51993: Thu Aug 31 03:45:21 2023 write: IOPS=17.9k, BW=1118MiB/s (1172MB/s)(4096MiB/3664msec); 0 zone resets slat (nsec): min=4620, max=53160, avg=7778.05, stdev=1146.85 clat (usec): min=1073, max=13136, avg=7143.14, stdev=466.51 lat (usec): min=1080, max=13143, avg=7151.38, stdev=466.52 clat percentiles (usec): | 1.00th=[ 5800], 5.00th=[ 6521], 10.00th=[ 6783], 20.00th=[ 6915], | 30.00th=[ 7046], 40.00th=[ 7111], 50.00th=[ 7177], 60.00th=[ 7242], | 70.00th=[ 7308], 80.00th=[ 7373], 90.00th=[ 7439], 95.00th=[ 7635], | 99.00th=[ 8848], 99.50th=[ 9241], 99.90th=[10159], 99.95th=[11207], | 99.99th=[12780] bw ( MiB/s): min= 1098, max= 1143, per=100.00%, avg=1119.59, stdev=17.62, samples=7 iops : min=17580, max=18300, avg=17913.43, stdev=281.91, samples=7 lat (msec) : 2=0.02%, 4=0.12%, 10=99.74%, 20=0.12% cpu : usr=6.47%, sys=17.85%, ctx=55880, majf=0, minf=0 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs): WRITE: bw=1118MiB/s (1172MB/s), 1118MiB/s-1118MiB/s (1172MB/s-1172MB/s), io=4096MiB (4295MB), run=3664-3664msec

X86

xuhuai@loongsonlab-56:~$ fio -filename=/mnt/mem1/64k -direct=1 -iodepth 128 -thread -rw=write -ioengine=libaio -bs=64k -size=4G -numjobs=1 -runtime=60 -group_reporting -name=file16kwrite_seq file16kwrite_seq: (g=0): rw=write, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=128 fio-3.33 Starting 1 thread file16kwrite_seq: Laying out IO file (1 file / 4096MiB) Jobs: 1 (f=1): [W(1)][100.0%][w=65.4MiB/s][w=1046 IOPS][eta 00m:00s] file16kwrite_seq: (groupid=0, jobs=1): err= 0: pid=20955: Thu Aug 31 15:46:24 2023 write: IOPS=1088, BW=68.0MiB/s (71.3MB/s)(4085MiB/60075msec); 0 zone resets slat (usec): min=5, max=533, avg=32.33, stdev= 9.84 clat (msec): min=2, max=197, avg=117.58, stdev=16.71 lat (msec): min=2, max=197, avg=117.61, stdev=16.71 clat percentiles (msec): | 1.00th=[ 55], 5.00th=[ 70], 10.00th=[ 99], 20.00th=[ 120], | 30.00th=[ 121], 40.00th=[ 122], 50.00th=[ 123], 60.00th=[ 124], | 70.00th=[ 125], 80.00th=[ 126], 90.00th=[ 128], 95.00th=[ 129], | 99.00th=[ 136], 99.50th=[ 142], 99.90th=[ 161], 99.95th=[ 165], | 99.99th=[ 190] bw ( KiB/s): min=61819, max=139776, per=99.96%, avg=69613.56, stdev=11807.92, samples=120 iops : min= 965, max= 2184, avg=1087.69, stdev=184.51, samples=120 lat (msec) : 4=0.01%, 10=0.01%, 20=0.02%, 50=0.26%, 100=9.96% lat (msec) : 250=89.75% cpu : usr=1.63%, sys=4.50%, ctx=65349, majf=0, minf=113 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,65367,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs): WRITE: bw=68.0MiB/s (71.3MB/s), 68.0MiB/s-68.0MiB/s (71.3MB/s-71.3MB/s), io=4085MiB (4284MB), run=60075-60075msec

X86 2

[root@node-191 ~]# fio -filename=/mnt/mem1/64k -direct=1 -iodepth 128 -thread -rw=write -ioengine=libaio -bs=64k -size=4G -numjobs=1 -runtime=600 -group_reporting -name=file16kwrite_seq file16kwrite_seq: (g=0): rw=write, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=128 fio-3.22 Starting 1 thread file16kwrite_seq: Laying out IO file (1 file / 4096MiB) Jobs: 1 (f=1) file16kwrite_seq: (groupid=0, jobs=1): err= 0: pid=2746084: Fri Sep 1 04:14:13 2023 write: IOPS=37.1k, BW=2319MiB/s (2432MB/s)(4096MiB/1766msec); 0 zone resets slat (nsec): min=3811, max=67897, avg=8061.79, stdev=2885.44 clat (usec): min=1301, max=6983, avg=3436.88, stdev=607.29 lat (usec): min=1316, max=6999, avg=3444.99, stdev=607.34 clat percentiles (usec): | 1.00th=[ 2474], 5.00th=[ 2737], 10.00th=[ 2835], 20.00th=[ 2966], | 30.00th=[ 3064], 40.00th=[ 3163], 50.00th=[ 3294], 60.00th=[ 3425], | 70.00th=[ 3589], 80.00th=[ 3851], 90.00th=[ 4293], 95.00th=[ 4621], | 99.00th=[ 5407], 99.50th=[ 5735], 99.90th=[ 6521], 99.95th=[ 6652], | 99.99th=[ 6915] bw ( MiB/s): min= 2305, max= 2336, per=99.87%, avg=2316.33, stdev=17.26, samples=3 iops : min=36892, max=37380, avg=37061.33, stdev=276.15, samples=3 lat (msec) : 2=0.06%, 4=83.81%, 10=16.13% cpu : usr=9.97%, sys=28.50%, ctx=36311, majf=0, minf=2 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,65536,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs): WRITE: bw=2319MiB/s (2432MB/s), 2319MiB/s-2319MiB/s (2432MB/s-2432MB/s), io=4096MiB (4295MB), run=1766-1766msec

64k read libaio No_rand iodepth_128 numjobs_1

fio -filename=/mnt/mem1/64k -direct=1 -iodepth 128 -thread -rw=read -ioengine=libaio -bs=64k -size=2G -numjobs=1 -group_reporting -name=file16kwrite_seq

LA

[root@ceph192 ~]# fio -filename=/mnt/mem1/64k -direct=1 -iodepth 128 -thread -rw=read -ioengine=libaio -bs=64k -size=10G -numjobs=1 -group_reporting -name=file16kwrite_seq file16kwrite_seq: (g=0): rw=read, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=128 fio-3.22 Starting 1 thread file16kwrite_seq: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [R(1)][100.0%][r=1122MiB/s][r=17.0k IOPS][eta 00m:00s] file16kwrite_seq: (groupid=0, jobs=1): err= 0: pid=52041: Thu Aug 31 03:47:30 2023 read: IOPS=18.1k, BW=1133MiB/s (1188MB/s)(10.0GiB/9037msec) slat (usec): min=3, max=466, avg= 8.08, stdev= 9.10 clat (usec): min=1891, max=12842, avg=6925.51, stdev=595.04 lat (usec): min=1924, max=12850, avg=6934.20, stdev=594.30 clat percentiles (usec): | 1.00th=[ 5145], 5.00th=[ 6194], 10.00th=[ 6390], 20.00th=[ 6521], | 30.00th=[ 6652], 40.00th=[ 6783], 50.00th=[ 6915], 60.00th=[ 7046], | 70.00th=[ 7242], 80.00th=[ 7308], 90.00th=[ 7504], 95.00th=[ 7701], | 99.00th=[ 8717], 99.50th=[ 9372], 99.90th=[10421], 99.95th=[10945], | 99.99th=[11994] bw ( MiB/s): min= 766, max= 1195, per=100.00%, avg=1134.56, stdev=96.75, samples=18 iops : min=12267, max=19120, avg=18152.89, stdev=1548.08, samples=18 lat (msec) : 2=0.01%, 4=0.19%, 10=99.63%, 20=0.18% cpu : usr=4.69%, sys=20.00%, ctx=150099, majf=0, minf=512 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=163840,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs): READ: bw=1133MiB/s (1188MB/s), 1133MiB/s-1133MiB/s (1188MB/s-1188MB/s), io=10.0GiB (10.7GB), run=9037-9037msec

X86

xuhuai@loongsonlab-56:~$ fio -filename=/mnt/mem1/64k -direct=1 -iodepth 128 -thread -rw=read -ioengine=libaio -bs=64k -size=2G -numjobs=1 -group_reporting -name=file16kwrite_seq file16kwrite_seq: (g=0): rw=read, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=128 fio-3.33 Starting 1 thread Jobs: 1 (f=0): [f(1)][100.0%][r=60.2MiB/s][r=962 IOPS][eta 00m:00s] file16kwrite_seq: (groupid=0, jobs=1): err= 0: pid=21166: Thu Aug 31 15:48:23 2023 read: IOPS=997, BW=62.4MiB/s (65.4MB/s)(2048MiB/32838msec) slat (usec): min=6, max=2764, avg=30.73, stdev=24.70 clat (msec): min=7, max=216, avg=128.19, stdev= 8.17 lat (msec): min=7, max=216, avg=128.22, stdev= 8.17 clat percentiles (msec): | 1.00th=[ 110], 5.00th=[ 117], 10.00th=[ 121], 20.00th=[ 124], | 30.00th=[ 126], 40.00th=[ 128], 50.00th=[ 130], 60.00th=[ 131], | 70.00th=[ 132], 80.00th=[ 133], 90.00th=[ 136], 95.00th=[ 136], | 99.00th=[ 140], 99.50th=[ 178], 99.90th=[ 188], 99.95th=[ 201], | 99.99th=[ 213] bw ( KiB/s): min=53888, max=72576, per=99.96%, avg=63840.49, stdev=2838.64, samples=65 iops : min= 842, max= 1134, avg=997.51, stdev=44.35, samples=65 lat (msec) : 10=0.01%, 20=0.02%, 50=0.05%, 100=0.12%, 250=99.80% cpu : usr=1.03%, sys=4.80%, ctx=32772, majf=0, minf=547 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=32768,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs): READ: bw=62.4MiB/s (65.4MB/s), 62.4MiB/s-62.4MiB/s (65.4MB/s-65.4MB/s), io=2048MiB (2147MB), run=32838-32838msec

X86 2

[root@node-191 ~]# fio -filename=/mnt/mem1/64k -direct=1 -iodepth 128 -thread -rw=read -ioengine=libaio -bs=64k -size=2G -numjobs=1 -group_reporting -name=file16kwrite_seq file16kwrite_seq: (g=0): rw=read, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=128 fio-3.22 Starting 1 thread Jobs: 1 (f=1) file16kwrite_seq: (groupid=0, jobs=1): err= 0: pid=2746151: Fri Sep 1 04:14:31 2023 read: IOPS=31.0k, BW=1939MiB/s (2034MB/s)(2048MiB/1056msec) slat (usec): min=2, max=323, avg= 6.46, stdev= 4.03 clat (usec): min=2118, max=9392, avg=4106.42, stdev=967.53 lat (usec): min=2137, max=9398, avg=4112.92, stdev=967.36 clat percentiles (usec): | 1.00th=[ 2933], 5.00th=[ 3294], 10.00th=[ 3326], 20.00th=[ 3425], | 30.00th=[ 3490], 40.00th=[ 3556], 50.00th=[ 3687], 60.00th=[ 3851], | 70.00th=[ 4293], 80.00th=[ 4817], 90.00th=[ 5604], 95.00th=[ 6194], | 99.00th=[ 7177], 99.50th=[ 7439], 99.90th=[ 8225], 99.95th=[ 8848], | 99.99th=[ 9241] bw ( MiB/s): min= 1918, max= 1984, per=100.00%, avg=1951.56, stdev=46.40, samples=2 iops : min=30700, max=31750, avg=31225.00, stdev=742.46, samples=2 lat (msec) : 4=64.21%, 10=35.79% cpu : usr=4.55%, sys=25.59%, ctx=30651, majf=0, minf=517 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=32768,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs): READ: bw=1939MiB/s (2034MB/s), 1939MiB/s-1939MiB/s (2034MB/s-2034MB/s), io=2048MiB (2147MB), run=1056-1056msec

1m write libaio No_rand iodepth_128 numjobs_1

fio -filename=/mnt/mem1/64k -direct=1 -iodepth 128 -thread -rw=write -ioengine=libaio -bs=1m -size=10G -numjobs=1 -group_reporting -name=file16kwrite_seq -runtime 600

LA

[root@ceph192 ~]# fio -filename=/mnt/mem1/64k -direct=1 -iodepth 128 -thread -rw=write -ioengine=libaio -bs=1m -size=10G -numjobs=1 -group_reporting -name=file16kwrite_seq -runtime 600 file16kwrite_seq: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=128 fio-3.22 Starting 1 thread Jobs: 1 (f=1): [W(1)][100.0%][w=2054MiB/s][w=2054 IOPS][eta 00m:00s] file16kwrite_seq: (groupid=0, jobs=1): err= 0: pid=52245: Thu Aug 31 03:57:16 2023 write: IOPS=2059, BW=2060MiB/s (2160MB/s)(10.0GiB/4972msec); 0 zone resets slat (usec): min=31, max=189, avg=54.00, stdev=10.09 clat (msec): min=13, max=119, avg=61.98, stdev= 5.21 lat (msec): min=13, max=119, avg=62.04, stdev= 5.21 clat percentiles (msec): | 1.00th=[ 47], 5.00th=[ 62], 10.00th=[ 62], 20.00th=[ 62], | 30.00th=[ 62], 40.00th=[ 62], 50.00th=[ 62], 60.00th=[ 63], | 70.00th=[ 63], 80.00th=[ 63], 90.00th=[ 64], 95.00th=[ 65], | 99.00th=[ 69], 99.50th=[ 94], 99.90th=[ 114], 99.95th=[ 117], | 99.99th=[ 120] bw ( MiB/s): min= 1998, max= 2092, per=100.00%, avg=2062.44, stdev=29.32, samples=9 iops : min= 1998, max= 2092, avg=2062.44, stdev=29.32, samples=9 lat (msec) : 20=0.35%, 50=0.71%, 100=98.54%, 250=0.39% cpu : usr=5.45%, sys=8.69%, ctx=10130, majf=0, minf=0 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.3%, >=64=99.4% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,10240,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs): WRITE: bw=2060MiB/s (2160MB/s), 2060MiB/s-2060MiB/s (2160MB/s-2160MB/s), io=10.0GiB (10.7GB), run=4972-4972msec

X86

xuhuai@loongsonlab-56:~$ fio -filename=/mnt/mem1/64k -direct=1 -iodepth 128 -thread -rw=write -ioengine=libaio -bs=1m -size=10G -numjobs=1 -group_reporting -name=file16kwrite_seq -runtime 60 file16kwrite_seq: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=128 fio-3.33 Starting 1 thread Jobs: 1 (f=1): [W(1)][100.0%][w=1035MiB/s][w=1035 IOPS][eta 00m:00s] file16kwrite_seq: (groupid=0, jobs=1): err= 0: pid=21981: Thu Aug 31 15:57:36 2023 write: IOPS=1019, BW=1020MiB/s (1069MB/s)(10.0GiB/10040msec); 0 zone resets slat (usec): min=85, max=3240, avg=229.29, stdev=63.46 clat (msec): min=19, max=208, avg=125.06, stdev=12.88 lat (msec): min=19, max=208, avg=125.29, stdev=12.88 clat percentiles (msec): | 1.00th=[ 91], 5.00th=[ 114], 10.00th=[ 117], 20.00th=[ 121], | 30.00th=[ 122], 40.00th=[ 124], 50.00th=[ 125], 60.00th=[ 127], | 70.00th=[ 128], 80.00th=[ 131], 90.00th=[ 136], 95.00th=[ 140], | 99.00th=[ 180], 99.50th=[ 188], 99.90th=[ 203], 99.95th=[ 207], | 99.99th=[ 209] bw ( KiB/s): min=866304, max=1107968, per=99.15%, avg=1035571.20, stdev=53472.54, samples=20 iops : min= 846, max= 1082, avg=1011.30, stdev=52.22, samples=20 lat (msec) : 20=0.04%, 50=0.40%, 100=0.81%, 250=98.75% cpu : usr=12.83%, sys=12.45%, ctx=8264, majf=0, minf=93 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.3%, >=64=99.4% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,10240,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs): WRITE: bw=1020MiB/s (1069MB/s), 1020MiB/s-1020MiB/s (1069MB/s-1069MB/s), io=10.0GiB (10.7GB), run=10040-10040msec

X86 2

[root@node-191 ~]# fio -filename=/mnt/mem1/64k -direct=1 -iodepth 128 -thread -rw=write -ioengine=libaio -bs=1m -size=10G -numjobs=1 -group_reporting -name=file16kwrite_seq -runtime 600 file16kwrite_seq: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=128 fio-3.22 Starting 1 thread file16kwrite_seq: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [W(1)][-.-%][w=3664MiB/s][w=3663 IOPS][eta 00m:00s] file16kwrite_seq: (groupid=0, jobs=1): err= 0: pid=2746220: Fri Sep 1 04:14:58 2023 write: IOPS=3637, BW=3638MiB/s (3814MB/s)(10.0GiB/2815msec); 0 zone resets slat (usec): min=48, max=2123, avg=95.99, stdev=81.35 clat (usec): min=10955, max=57193, avg=35007.29, stdev=7409.51 lat (usec): min=11040, max=57296, avg=35103.33, stdev=7393.77 clat percentiles (usec): | 1.00th=[19006], 5.00th=[22414], 10.00th=[24511], 20.00th=[28181], | 30.00th=[31327], 40.00th=[33817], 50.00th=[35914], 60.00th=[37487], | 70.00th=[39584], 80.00th=[41681], 90.00th=[43779], 95.00th=[45876], | 99.00th=[50594], 99.50th=[52167], 99.90th=[56361], 99.95th=[56886], | 99.99th=[56886] bw ( MiB/s): min= 3478, max= 3856, per=100.00%, avg=3647.60, stdev=142.16, samples=5 iops : min= 3478, max= 3856, avg=3647.60, stdev=142.16, samples=5 lat (msec) : 20=1.64%, 50=97.20%, 100=1.16% cpu : usr=13.75%, sys=22.35%, ctx=4282, majf=0, minf=2 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.3%, >=64=99.4% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=0,10240,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs): WRITE: bw=3638MiB/s (3814MB/s), 3638MiB/s-3638MiB/s (3814MB/s-3814MB/s), io=10.0GiB (10.7GB), run=2815-2815msec

1m read libaio No_rand iodepth_128 numjobs_1

fio -filename=/mnt/mem1/64k -direct=1 -iodepth 128 -thread -rw=read -ioengine=libaio -bs=1m -size=10G -numjobs=1 -group_reporting -name=file16kwrite_seq -runtime 600

LA

[root@ceph192 ~]# fio -filename=/mnt/mem1/64k -direct=1 -iodepth 128 -thread -rw=read -ioengine=libaio -bs=1m -size=10G -numjobs=1 -group_reporting -name=file16kwrite_seq -runtime 600 file16kwrite_seq: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=128 fio-3.22 Starting 1 thread Jobs: 1 (f=1): [R(1)][100.0%][r=2231MiB/s][r=2231 IOPS][eta 00m:00s] file16kwrite_seq: (groupid=0, jobs=1): err= 0: pid=52267: Thu Aug 31 03:58:25 2023 read: IOPS=2206, BW=2207MiB/s (2314MB/s)(10.0GiB/4640msec) slat (usec): min=21, max=2917, avg=39.48, stdev=54.37 clat (msec): min=13, max=101, avg=57.81, stdev= 7.07 lat (msec): min=13, max=101, avg=57.85, stdev= 7.06 clat percentiles (msec): | 1.00th=[ 37], 5.00th=[ 50], 10.00th=[ 51], 20.00th=[ 52], | 30.00th=[ 54], 40.00th=[ 56], 50.00th=[ 59], 60.00th=[ 63], | 70.00th=[ 63], 80.00th=[ 64], 90.00th=[ 65], 95.00th=[ 65], | 99.00th=[ 70], 99.50th=[ 77], 99.90th=[ 97], 99.95th=[ 100], | 99.99th=[ 102] bw ( MiB/s): min= 2076, max= 2396, per=99.85%, avg=2203.62, stdev=123.61, samples=9 iops : min= 2076, max= 2396, avg=2203.44, stdev=123.63, samples=9 lat (msec) : 20=0.43%, 50=8.86%, 100=90.68%, 250=0.03% cpu : usr=1.16%, sys=9.89%, ctx=9615, majf=0, minf=6 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.3%, >=64=99.4% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs): READ: bw=2207MiB/s (2314MB/s), 2207MiB/s-2207MiB/s (2314MB/s-2314MB/s), io=10.0GiB (10.7GB), run=4640-4640msec

X86

xuhuai@loongsonlab-56:~$ fio -filename=/mnt/mem1/64k -direct=1 -iodepth 128 -thread -rw=read -ioengine=libaio -bs=1m -size=10G -numjobs=1 -group_reporting -name=file16kwrite_seq -runtime 60 file16kwrite_seq: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=128 fio-3.33 Starting 1 thread Jobs: 1 (f=1): [R(1)][100.0%][r=848MiB/s][r=847 IOPS][eta 00m:00s] file16kwrite_seq: (groupid=0, jobs=1): err= 0: pid=22070: Thu Aug 31 15:58:35 2023 read: IOPS=913, BW=914MiB/s (958MB/s)(10.0GiB/11209msec) slat (usec): min=46, max=1073, avg=125.84, stdev=57.49 clat (msec): min=43, max=259, avg=139.52, stdev=18.82 lat (msec): min=44, max=259, avg=139.65, stdev=18.81 clat percentiles (msec): | 1.00th=[ 82], 5.00th=[ 108], 10.00th=[ 113], 20.00th=[ 138], | 30.00th=[ 140], 40.00th=[ 142], 50.00th=[ 142], 60.00th=[ 144], | 70.00th=[ 146], 80.00th=[ 148], 90.00th=[ 153], 95.00th=[ 159], | 99.00th=[ 199], 99.50th=[ 203], 99.90th=[ 247], 99.95th=[ 253], | 99.99th=[ 257] bw ( KiB/s): min=813056, max=1196032, per=99.88%, avg=934353.45, stdev=83942.86, samples=22 iops : min= 794, max= 1168, avg=912.45, stdev=81.98, samples=22 lat (msec) : 50=0.18%, 100=1.19%, 250=98.55%, 500=0.08% cpu : usr=1.20%, sys=12.50%, ctx=10238, majf=0, minf=159 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.3%, >=64=99.4% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs): READ: bw=914MiB/s (958MB/s), 914MiB/s-914MiB/s (958MB/s-958MB/s), io=10.0GiB (10.7GB), run=11209-11209msec

X86 2

[root@node-191 ~]# fio -filename=/mnt/mem1/64k -direct=1 -iodepth 128 -thread -rw=read -ioengine=libaio -bs=1m -size=10G -numjobs=1 -group_reporting -name=file16kwrite_seq -runtime 600 file16kwrite_seq: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=128 fio-3.22 Starting 1 thread Jobs: 1 (f=1): [R(1)][-.-%][r=3083MiB/s][r=3083 IOPS][eta 00m:00s] file16kwrite_seq: (groupid=0, jobs=1): err= 0: pid=2746285: Fri Sep 1 04:15:18 2023 read: IOPS=3155, BW=3156MiB/s (3309MB/s)(10.0GiB/3245msec) slat (usec): min=34, max=644, avg=47.72, stdev=23.46 clat (usec): min=13876, max=80617, avg=40332.98, stdev=3894.56 lat (usec): min=13964, max=80654, avg=40380.77, stdev=3883.77 clat percentiles (usec): | 1.00th=[21627], 5.00th=[38011], 10.00th=[38536], 20.00th=[38536], | 30.00th=[39060], 40.00th=[39584], 50.00th=[40633], 60.00th=[41681], | 70.00th=[41681], 80.00th=[41681], 90.00th=[42206], 95.00th=[42206], | 99.00th=[47973], 99.50th=[64226], 99.90th=[77071], 99.95th=[79168], | 99.99th=[80217] bw ( MiB/s): min= 3058, max= 3314, per=100.00%, avg=3160.67, stdev=102.45, samples=6 iops : min= 3058, max= 3314, avg=3160.67, stdev=102.45, samples=6 lat (msec) : 20=0.78%, 50=98.27%, 100=0.95% cpu : usr=0.49%, sys=15.84%, ctx=10181, majf=0, minf=68 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.3%, >=64=99.4% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs): READ: bw=3156MiB/s (3309MB/s), 3156MiB/s-3156MiB/s (3309MB/s-3309MB/s), io=10.0GiB (10.7GB), run=3245-3245msec