Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-12429

Single client buffered SSF write is slower than O_DIRECT

Details

    • Bug
    • Resolution: Unresolved
    • Minor
    • None
    • Lustre 2.13.0
    • None
    • 3
    • 9223372036854775807

    Description

      Single client's SSF doesn't scale by nubmer of process

      # mpirun --allow-run-as-root -np X /work/tools/bin/ior -w -t 16m -b $((32/X))g -e -o file
      
      NP     Write(MB/s)
        1     1594
        2     2525
        4     1892
        8     2032
       16     1812
      

      A framegraph output at ior with NP=16 pointed out huge amount cost of spin_lock in add_to_page_cache_lru() and set_page_dirty(). At the resutls, Buffered SSF write on single client is slower than SSF with O_DIRECT. Here is my quick test resutls of single client SSF with/without O_DIRECT.

      # mpirun -np 16 --allow-run-as-root /work/tools/bin/ior -w -t 16m -b 4g -e -o /scratch0/stripe/file 
      Max Write: 1806.31 MiB/sec (1894.06 MB/sec)
      
      # mpirun -np 16 --allow-run-as-root /work/tools/bin/ior -w -t 16m -b 4g -e -o /scratch0/stripe/file -B
      Max Write: 5547.13 MiB/sec (5816.58 MB/sec)
      

      Attachments

        Issue Links

          Activity

            [LU-12429] Single client buffered SSF write is slower than O_DIRECT
            adilger Andreas Dilger made changes -
            Link New: This issue is related to LU-247 [ LU-247 ]
            sihara Shuichi Ihara made changes -
            Attachment New: single-thread.svg [ 33712 ]
            pfarrell Patrick Farrell (Inactive) made changes -
            Assignee Original: Patrick Farrell [ pfarrell ] New: Dongyang Li [ dongyang ]
            adilger Andreas Dilger made changes -
            Link New: This issue is related to LU-9920 [ LU-9920 ]
            pjones Peter Jones made changes -
            Assignee Original: WC Triage [ wc-triage ] New: Patrick Farrell [ pfarrell ]
            pfarrell Patrick Farrell (Inactive) made changes -
            Summary Original: Signle client buffered SSF write is slower than O_DIRECT New: Single client buffered SSF write is slower than O_DIRECT
            adilger Andreas Dilger made changes -
            Link New: This issue is related to DDN-753 [ DDN-753 ]
            sihara Shuichi Ihara made changes -
            Summary Original: Signle node's Buffered SSF write is slower than O_DIRECT New: Signle client buffered SSF write is slower than O_DIRECT
            sihara Shuichi Ihara created issue -

            People

              dongyang Dongyang Li
              sihara Shuichi Ihara
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated: