Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-14139

batched statahead processing

Details

    • New Feature
    • Resolution: Fixed
    • Minor
    • Lustre 2.16.0
    • None
    • None
    • 9223372036854775807

    Description

      Batched metadata processing may get a big performance boost.
      Batched statahead can also increase the performance for directory listing operations such as ls.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. LU-16365 cached 'ls -l' is slow

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. LU-17783 Wrong buffer for field 'batch_update_reply' in format 'MDS_BATCH' from statahead for old server

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. LU-8582 Interop: master<->b2_8 - sanity test_255a: test failed to respond and timed out

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. LU-10334 Ubuntu1604 client sanity-103a: FAIL: ACL misc test failed

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. LU-14868 sanity: all subtests pass but test suite fails with TIMEOUT

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. LU-16247 do open()+readahead for DoM files as batched operations

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. LU-16355 batch dirty buffered write of small files

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. LU-23 Improve performance for traversing large directory with readdir+

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Technical task - A technical task. LU-10280 DoM cross-file open+readahead via statahead

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. LU-14380 Make statahead better support Breadth First Search (BFS) or Depth First Search (DFS)

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. LU-14138 Move more members in PTLRPC request into pill

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LU-14361 Add support for statahead pattern with regularized file name

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. LU-15975 Statahead_V1 Features

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. LU-16139 statahead: avoid new RPC and long wait in the statahead interpret callback

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. LU-10938 Metadata writeback cache support

          • Major - Major loss of function.
          • Open

          Technical task - A technical task. LU-13045 WBC2: batch metadata update

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. LU-16096 recovery: handle compatibility during upgrade for new replay data format

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LU-14361 Add support for statahead pattern with regularized file name

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LU-14393 Batching Processing Framework for Lustre

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LU-15550 WBC: retry the batched RPC when the reply buffer is overflowed

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LU-15562 statahead: using try lock for batched statahead to avoid deadlock with rename and migrate operations

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            [LU-14139] batched statahead processing
            pjones Peter Jones added a comment -

            Landed for 2.16

            pjones Peter Jones added a comment - Landed for 2.16
            gerrit Gerrit Updater added a comment -

            "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/41220/
            Subject: LU-14139 statahead: add test for batch statahead processing
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: cd1e7bdd903ada4467064eaf8613ec62a358fa1c

            gerrit Gerrit Updater added a comment - "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/41220/ Subject: LU-14139 statahead: add test for batch statahead processing Project: fs/lustre-release Branch: master Current Patch Set: Commit: cd1e7bdd903ada4467064eaf8613ec62a358fa1c
            gerrit Gerrit Updater added a comment -

            "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/43707/
            Subject: LU-14139 ptlrpc: grow PtlRPC properly when prepare sub request
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: 5a2dfd36f9c2b6c10ab7ba44b0e9e86372623fde

            gerrit Gerrit Updater added a comment - "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/43707/ Subject: LU-14139 ptlrpc: grow PtlRPC properly when prepare sub request Project: fs/lustre-release Branch: master Current Patch Set: Commit: 5a2dfd36f9c2b6c10ab7ba44b0e9e86372623fde
            gerrit Gerrit Updater added a comment -

            "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/40945/
            Subject: LU-14139 ptlrpc: grow reply buffer properly for batch request
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: 5edb883b44ac707528ce2c0bc812d65b9ffb4a50

            gerrit Gerrit Updater added a comment - "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/40945/ Subject: LU-14139 ptlrpc: grow reply buffer properly for batch request Project: fs/lustre-release Branch: master Current Patch Set: Commit: 5edb883b44ac707528ce2c0bc812d65b9ffb4a50
            gerrit Gerrit Updater added a comment -

            "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/40943/
            Subject: LU-14139 statahead: add stats for batch RPC requests
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: a20f25d24b5f0ce7b5e77f7c596bffd0450cbdae

            gerrit Gerrit Updater added a comment - "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/40943/ Subject: LU-14139 statahead: add stats for batch RPC requests Project: fs/lustre-release Branch: master Current Patch Set: Commit: a20f25d24b5f0ce7b5e77f7c596bffd0450cbdae
            gerrit Gerrit Updater added a comment -

            "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/40720/
            Subject: LU-14139 statahead: batched statahead processing
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: 4435d0121f72aac3ad01c98a33b265a496359890

            gerrit Gerrit Updater added a comment - "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/40720/ Subject: LU-14139 statahead: batched statahead processing Project: fs/lustre-release Branch: master Current Patch Set: Commit: 4435d0121f72aac3ad01c98a33b265a496359890

            People

              qian_wc Qian Yingjin
              qian_wc Qian Yingjin
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: