[LU-8280] Bad Lustre Read Performance with Master Build 3371

Sat, 23 Jul 2016 05:13:00 +0000

*Important notes: Please do not rebuild Lustre on wolf-[33-45], thanks!

The read performance of Lustre master build 3371 is bad. Following are the detailed configurations for my evaluation over Lustre file system using IOR benchmark:

[Lustre Configuration]

MDS/MDT: wolf-37
OST/OSS: wolf-[33-36] (Each OST: MD RAID 0 striped [Chunk = 1M] over 5 SATA Drives)
Clients: wolf-[38-45]

Stripe Size: 4194304
Stripe Count: 4

Lustre Build Version: 3371
Lustre Build Command:

loadjenkinsbuild -b 3371 -a x86_64 -j lustre-master --jenkinsuri https://build.hpdd.intel.com -p test-el6-x86_64 -t server -d el6.7 -n wolf-45 -r -v

[IOR Benchmark]

IOR Command:

mpirun -np 4 -iface ib0 -f /home/congxu/host-ib /home/congxu/Software/ior-master/src/ior -a POSIX -N 4 -d 5 -i 1 -s 32768 -b 4MiB -t 4MiB -w -r -o /mnt/lustre/cong/testfile

IOR Results:

Share File case
IOR-3.0.1: MPI Coordinated Test of Parallel I/O

Began: Tue Jun 14 20:16:18 2016
Command line used: /home/congxu/Software/ior-master/src/ior -a POSIX -N 4 -d 5 -i 1 -s 32768 -b 4MiB -t 4MiB -w -r -o /mnt/lustre/cong/testfile
Machine: Linux wolf-38.wolf.hpdd.intel.com

Test 0 started: Tue Jun 14 20:16:18 2016
Summary:
api = POSIX
test filename = /mnt/lustre/cong/testfile
access = single-shared-file
ordering in a file = sequential offsets
ordering inter file= no tasks offsets
clients = 4 (1 per node)
repetitions = 1
xfersize = 4 MiB
blocksize = 4 MiB
aggregate filesize = 512 GiB

access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter
------ --------- ---------- --------- -------- -------- -------- -------- ----
write 2546.35 4096 4096 0.001613 205.90 10.62 205.90 0
read 1076.82 4096 4096 0.000565 486.89 74.16 486.89 0
remove - - - - - - 0.001506 0

Max Write: 2546.35 MiB/sec (2670.05 MB/sec)
Max Read: 1076.82 MiB/sec (1129.12 MB/sec)

Summary of all tests:
Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum
write 2546.35 2546.35 2546.35 0.00 205.89756 0 4 1 1 0 0 1 0 0 32768 4194304 4194304 549755813888 POSIX 0
read 1076.82 1076.82 1076.82 0.00 486.88718 0 4 1 1 0 0 1 0 0 32768 4194304 4194304 549755813888 POSIX 0

Finished: Tue Jun 14 20:28:01 2016

*************************************On the contrary, "lustre-b2_8 -b 12" performs well**********************************

[Lustre Configuration]

MDS/MDT: wolf-37
OST/OSS: wolf-[33-36] (Each OST: MD RAID 0 striped [Chunk = 1M] over 5 SATA Drives)
Clients: wolf-[38-45]

Stripe Size: 4194304
Stripe Count: 4

Lustre Build Version: 12
Lustre Build Command:

loadjenkinsbuild -b 12 -a x86_64 -j lustre-b2_8 --jenkinsuri https://build.hpdd.intel.com -p test-el6-x86_64 -t server -d el6.7 -n wolf-45 -r -v

[IOR Benchmark]

IOR Command:

mpirun -np 4 -iface ib0 -f /home/congxu/host-ib /home/congxu/Software/ior-master/src/ior -a POSIX -N 4 -d 5 -i 1 -s 32768 -b 4MiB -t 4MiB -w -r -o /mnt/lustre/cong/testfile

IOR Results:

Share File case
IOR-3.0.1: MPI Coordinated Test of Parallel I/O

Began: Wed Jun 15 01:17:36 2016
Command line used: /home/congxu/Software/ior-master/src/ior -a POSIX -N 4 -d 5 -i 1 -s 32768 -b 4MiB -t 4MiB -w -r -o /mnt/lustre/cong/testfile
Machine: Linux wolf-38.wolf.hpdd.intel.com

Test 0 started: Wed Jun 15 01:17:36 2016
Summary:
api = POSIX
test filename = /mnt/lustre/cong/testfile
access = single-shared-file
ordering in a file = sequential offsets
ordering inter file= no tasks offsets
clients = 4 (1 per node)
repetitions = 1
xfersize = 4 MiB
blocksize = 4 MiB
aggregate filesize = 512 GiB

access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter
------ --------- ---------- --------- -------- -------- -------- -------- ----
write 2559.64 4096 4096 0.001187 204.83 5.84 204.83 0
read 2840.79 4096 4096 0.000808 184.56 24.11 184.56 0
remove - - - - - - 0.001111 0

Max Write: 2559.64 MiB/sec (2683.98 MB/sec)
Max Read: 2840.79 MiB/sec (2978.78 MB/sec)

Summary of all tests:
Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum
write 2559.64 2559.64 2559.64 0.00 204.82859 0 4 1 1 0 0 1 0 0 32768 4194304 4194304 549755813888 POSIX 0
read 2840.79 2840.79 2840.79 0.00 184.55737 0 4 1 1 0 0 1 0 0 32768 4194304 4194304 549755813888 POSIX 0

Finished: Wed Jun 15 01:24:15 2016

Whamcloud Community JIRA

[LU-8280] Bad Lustre Read Performance with Master Build 3371