Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-1167

Poor mdtest unlink performance with multiple processes per node

Details

    • Bug
    • Resolution: Unresolved
    • Minor
    • None
    • Lustre 2.1.1
    • None
    • SL6.1,
      2.6.32-131.12.1.el6.lustre.20.x86_64

    Description

      We have noticed in testing that running multiple mdtest processes per node severely degrades unlink performance.

      • Lustre mounted once per client; not multimount.
      • shared directory case
        This can be seen at a wide range of node counts (5-128) and backends, to varying degrees.
        Interestingly scaling the client count up does not seem to have nearly the same negative performance impact; only the ppn seems to matter.
      	total
nodes	jobs	unlink kops
8	8	11.4
8	16	9.7
8	32	8.6
8	64	7

      We see the same issue with 1.8.6 clients against the server; we do not see it with 1.8.6 servers.

      Attachments

        1. license.txt
          2 kB
        2. metabench.tar
          4.39 MB
        3. metabench-compare.xlsx
          10 kB
        4. metabench-comparison.txt
          24 kB

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. LU-3308 large readdir chunk size slows unlink/"rm -r" performance

          • Major - Major loss of function.
          • Reopened

          Bug - A problem which impairs or prevents the functions of the product. LU-1695 Demonstrate MDS performance with increasing client load for SMP Affinity

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Activity

            [LU-1167] Poor mdtest unlink performance with multiple processes per node
            adilger Andreas Dilger made changes -
            Link New: This issue is related to LU-3308 [ LU-3308 ]
            spitzcor Cory Spitz added a comment -

            It looks as though the comments that were made on 04/Apr/12 were about right. LU-3308 was opened to look into the regression aspect (which is seemingly unrelated to multiple processes per node).

            spitzcor Cory Spitz added a comment - It looks as though the comments that were made on 04/Apr/12 were about right. LU-3308 was opened to look into the regression aspect (which is seemingly unrelated to multiple processes per node).
            mmansk Mark Mansk made changes -
            Attachment New: metabench.tar [ 12650 ]
            mmansk Mark Mansk added a comment -

            metabench source

            mmansk Mark Mansk added a comment - metabench source
            mmansk Mark Mansk made changes -
            Attachment New: license.txt [ 12649 ]
            spitzcor Cory Spitz added a comment -

            Bug should be marked as affects all 2.x versions.

            spitzcor Cory Spitz added a comment - Bug should be marked as affects all 2.x versions.
            mmansk Mark Mansk made changes -
            Attachment New: metabench-comparison.txt [ 12510 ]
            mmansk Mark Mansk added a comment -

            data from metabench runs for 1.8.6,2.3&2.4

            mmansk Mark Mansk added a comment - data from metabench runs for 1.8.6,2.3&2.4
            mmansk Mark Mansk made changes -
            Attachment New: metabench-compare.xlsx [ 12509 ]
            mmansk Mark Mansk added a comment -

            excel sheet with comparisons of 1.8.6 to 2.3 & 2.4 using metabench. show performance drop from 1.8.6 in deletes in same directory for 2.x.

            mmansk Mark Mansk added a comment - excel sheet with comparisons of 1.8.6 to 2.3 & 2.4 using metabench. show performance drop from 1.8.6 in deletes in same directory for 2.x.

            People

              wc-triage WC Triage
              nrutman Nathan Rutman
              Votes:
              1 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated: