Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-8280

Bad Lustre Read Performance with Master Build 3371

    XMLWordPrintable

Details

    • Bug
    • Resolution: Not a Bug
    • Critical
    • None
    • None
    • None
    • 3
    • 9223372036854775807

    Description

      *Important notes: Please do not rebuild Lustre on wolf-[33-45], thanks!

      The read performance of Lustre master build 3371 is bad. Following are the detailed configurations for my evaluation over Lustre file system using IOR benchmark:

      [Lustre Configuration]

      MDS/MDT: wolf-37
      OST/OSS: wolf-[33-36] (Each OST: MD RAID 0 striped [Chunk = 1M] over 5 SATA Drives)
      Clients: wolf-[38-45]

      Stripe Size: 4194304
      Stripe Count: 4

      Lustre Build Version: 3371
      Lustre Build Command:

      1. loadjenkinsbuild -b 3371 -a x86_64 -j lustre-master --jenkinsuri https://build.hpdd.intel.com -p test-el6-x86_64 -t server -d el6.7 -n wolf-45 -r -v

      [IOR Benchmark]

      IOR Command:

      1. mpirun -np 4 -iface ib0 -f /home/congxu/host-ib /home/congxu/Software/ior-master/src/ior -a POSIX -N 4 -d 5 -i 1 -s 32768 -b 4MiB -t 4MiB -w -r -o /mnt/lustre/cong/testfile

      IOR Results:

      Share File case
      IOR-3.0.1: MPI Coordinated Test of Parallel I/O

      Began: Tue Jun 14 20:16:18 2016
      Command line used: /home/congxu/Software/ior-master/src/ior -a POSIX -N 4 -d 5 -i 1 -s 32768 -b 4MiB -t 4MiB -w -r -o /mnt/lustre/cong/testfile
      Machine: Linux wolf-38.wolf.hpdd.intel.com

      Test 0 started: Tue Jun 14 20:16:18 2016
      Summary:
      api = POSIX
      test filename = /mnt/lustre/cong/testfile
      access = single-shared-file
      ordering in a file = sequential offsets
      ordering inter file= no tasks offsets
      clients = 4 (1 per node)
      repetitions = 1
      xfersize = 4 MiB
      blocksize = 4 MiB
      aggregate filesize = 512 GiB

      access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter
      ------ --------- ---------- --------- -------- -------- -------- -------- ----
      write 2546.35 4096 4096 0.001613 205.90 10.62 205.90 0
      read 1076.82 4096 4096 0.000565 486.89 74.16 486.89 0
      remove - - - - - - 0.001506 0

      Max Write: 2546.35 MiB/sec (2670.05 MB/sec)
      Max Read: 1076.82 MiB/sec (1129.12 MB/sec)

      Summary of all tests:
      Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum
      write 2546.35 2546.35 2546.35 0.00 205.89756 0 4 1 1 0 0 1 0 0 32768 4194304 4194304 549755813888 POSIX 0
      read 1076.82 1076.82 1076.82 0.00 486.88718 0 4 1 1 0 0 1 0 0 32768 4194304 4194304 549755813888 POSIX 0

      Finished: Tue Jun 14 20:28:01 2016

      *************************************On the contrary, "lustre-b2_8 -b 12" performs well**********************************

      [Lustre Configuration]

      MDS/MDT: wolf-37
      OST/OSS: wolf-[33-36] (Each OST: MD RAID 0 striped [Chunk = 1M] over 5 SATA Drives)
      Clients: wolf-[38-45]

      Stripe Size: 4194304
      Stripe Count: 4

      Lustre Build Version: 12
      Lustre Build Command:

      1. loadjenkinsbuild -b 12 -a x86_64 -j lustre-b2_8 --jenkinsuri https://build.hpdd.intel.com -p test-el6-x86_64 -t server -d el6.7 -n wolf-45 -r -v

      [IOR Benchmark]

      IOR Command:

      1. mpirun -np 4 -iface ib0 -f /home/congxu/host-ib /home/congxu/Software/ior-master/src/ior -a POSIX -N 4 -d 5 -i 1 -s 32768 -b 4MiB -t 4MiB -w -r -o /mnt/lustre/cong/testfile

      IOR Results:

      Share File case
      IOR-3.0.1: MPI Coordinated Test of Parallel I/O

      Began: Wed Jun 15 01:17:36 2016
      Command line used: /home/congxu/Software/ior-master/src/ior -a POSIX -N 4 -d 5 -i 1 -s 32768 -b 4MiB -t 4MiB -w -r -o /mnt/lustre/cong/testfile
      Machine: Linux wolf-38.wolf.hpdd.intel.com

      Test 0 started: Wed Jun 15 01:17:36 2016
      Summary:
      api = POSIX
      test filename = /mnt/lustre/cong/testfile
      access = single-shared-file
      ordering in a file = sequential offsets
      ordering inter file= no tasks offsets
      clients = 4 (1 per node)
      repetitions = 1
      xfersize = 4 MiB
      blocksize = 4 MiB
      aggregate filesize = 512 GiB

      access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) total(s) iter
      ------ --------- ---------- --------- -------- -------- -------- -------- ----
      write 2559.64 4096 4096 0.001187 204.83 5.84 204.83 0
      read 2840.79 4096 4096 0.000808 184.56 24.11 184.56 0
      remove - - - - - - 0.001111 0

      Max Write: 2559.64 MiB/sec (2683.98 MB/sec)
      Max Read: 2840.79 MiB/sec (2978.78 MB/sec)

      Summary of all tests:
      Operation Max(MiB) Min(MiB) Mean(MiB) StdDev Mean(s) Test# #Tasks tPN reps fPP reord reordoff reordrand seed segcnt blksiz xsize aggsize API RefNum
      write 2559.64 2559.64 2559.64 0.00 204.82859 0 4 1 1 0 0 1 0 0 32768 4194304 4194304 549755813888 POSIX 0
      read 2840.79 2840.79 2840.79 0.00 184.55737 0 4 1 1 0 0 1 0 0 32768 4194304 4194304 549755813888 POSIX 0

      Finished: Wed Jun 15 01:24:15 2016

      Attachments

        Activity

          People

            cliffw Cliff White (Inactive)
            czx0003 Cong Xu (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: