Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-11199

mdsrate open() performance degradation

Details

    • Bug
    • Resolution: Fixed
    • Major
    • Lustre 2.12.0
    • None
    • None
    • 3
    • 9223372036854775807

    Description

      A change in https://review.whamcloud.com/#/c/30246/

      (LU-10269 ldlm: fix the issues introduced by try bits)

      introduced a major regression in open() performance in at least an mdsrate open() benchmark:

      On a real system, this mdsrate open benchmark drops from 35K opens/second to around 11K.
      This is 10K opens per process, 64 processes, opens are random files from among 300000 existing files (created by mdsrate earlier): 
      aprun -n 64 /usr/lib64/lustre/tests/mdsrate -d /mnt/lustre/mdsrate --open --iters 10000 --nfile=300000

      On a much smaller VM, I see a drop from 8K opens/second to 4K with this benchmark.
      This is 8K opens per process, 4 processes, opens randomly selected from among 30000 existing files:
      mpirun -n 4 /usr/lib64/lustre/tests/mdsrate -d /mnt/lustre/mdsrate --open --iters 8000 --nfile=30000

      The specific change is no longer attempting to grant the LOOKUP lock on opens.

      Oleg has a patch in flight which reverses this change, but also includes a number of other attempted optimizations:
      https://review.whamcloud.com/#/c/32156/

      Andreas suggested I push the simpler patch to revert, with the idea that that patch can land quickly and Oleg's further optimizations can be considered separately.

      Attachments

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. LU-10948 client cache open lock after N opens

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open
          is related to

          Bug - A problem which impairs or prevents the functions of the product. LU-10269 Fixes for selective trybits

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LU-10957 Return more lock bits from MDS for open-create

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Activity

            [LU-11199] mdsrate open() performance degradation
            pjones Peter Jones added a comment -

            Landed for 2.12

            pjones Peter Jones added a comment - Landed for 2.12
            gerrit Gerrit Updater added a comment -

            Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/32929/
            Subject: LU-11199 mdt: Attempt lookup lock on open
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: 8b9105d8281f10b500d47a00458631a586c7f1d4

            gerrit Gerrit Updater added a comment - Oleg Drokin (green@whamcloud.com) merged in patch https://review.whamcloud.com/32929/ Subject: LU-11199 mdt: Attempt lookup lock on open Project: fs/lustre-release Branch: master Current Patch Set: Commit: 8b9105d8281f10b500d47a00458631a586c7f1d4
            gerrit Gerrit Updater added a comment -

            Patrick Farrell (paf@cray.com) uploaded a new patch: https://review.whamcloud.com/32929
            Subject: LU-11199 mdt: Attempt lookup lock on open
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: 3061c2836db7f917a0dc394fba9cc3fe32b0d4c9

            gerrit Gerrit Updater added a comment - Patrick Farrell (paf@cray.com) uploaded a new patch: https://review.whamcloud.com/32929 Subject: LU-11199 mdt: Attempt lookup lock on open Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: 3061c2836db7f917a0dc394fba9cc3fe32b0d4c9

            People

              paf Patrick Farrell (Inactive)
              paf Patrick Farrell (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: