Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-15058

replace critical vmalloc allocations with kernel APIs to improve performance

Details

    • Improvement
    • Resolution: Unresolved
    • Major
    • None
    • Lustre 2.15.0
    • None
    • Any client or server running Lustre.
    • 9223372036854775807

    Description

      When memory allocation grow top large the method of allocation moves from kmalloc to vmalloc. Using vmalloc() uses global locking which greatly degrades performance. For ORNL we ended up preventing users form creating files stripped greater than 512. This problem impact very large RPC and other components of Lustre. The solution is to use the generic_radix work present in newer kernel developed for this very issue.

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. LU-15606 sanity-quota test_8: osc_brw_prep_request()) ASSERTION( i == 0 || pg->off > pg_prev->off ) failed

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. LU-17916 Enable folio allocation support on the buffed io read/write path (BIO)

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open
          is related to

          Bug - A problem which impairs or prevents the functions of the product. LU-14469 lmv_rmfid() does 128K kmalloc()

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. LU-13212 Lustre client hangs machine under memory pressure

          • Major - Major loss of function.
          • Resolved

          Activity

            [LU-15058] replace critical vmalloc allocations with kernel APIs to improve performance
            gerrit Gerrit Updater added a comment -

            "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/45890/
            Subject: LU-15058 libcfs: introduce genradix support
            Project: fs/lustre-release
            Branch: master
            Current Patch Set:
            Commit: 841843f44cd16e2bfb9b31195f7398cff02f9088

            gerrit Gerrit Updater added a comment - "Oleg Drokin <green@whamcloud.com>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/45890/ Subject: LU-15058 libcfs: introduce genradix support Project: fs/lustre-release Branch: master Current Patch Set: Commit: 841843f44cd16e2bfb9b31195f7398cff02f9088
            gerrit Gerrit Updater added a comment -

            "James Simmons <jsimmons@infradead.org>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/48883
            Subject: LU-15058 osc: replace large RPC allocations with genradix
            Project: fs/lustre-release
            Branch: master
            Current Patch Set: 1
            Commit: da4bf845b7e9a5de9ef44e874d434854994b41f7

            gerrit Gerrit Updater added a comment - "James Simmons <jsimmons@infradead.org>" uploaded a new patch: https://review.whamcloud.com/c/fs/lustre-release/+/48883 Subject: LU-15058 osc: replace large RPC allocations with genradix Project: fs/lustre-release Branch: master Current Patch Set: 1 Commit: da4bf845b7e9a5de9ef44e874d434854994b41f7
            paf0186 Patrick Farrell added a comment -

            Ah, OK - thanks.

            paf0186 Patrick Farrell added a comment - Ah, OK - thanks.
            simmonsja James A Simmons added a comment - - edited

            See Linux commit ba20ba2e3743bac786dff777954c11930256075e. Reading the commit you will see generic radix trees is a direct replacement of flex_array. The link - https://www.kernel.org/doc/html/v4.14/core-api/flexible-arrays.html states the purpose of flexible arrays was to handle the described issue. Currently the only user is procfs. flexible arrays never had that many users.

            simmonsja James A Simmons added a comment - - edited See Linux commit ba20ba2e3743bac786dff777954c11930256075e. Reading the commit you will see generic radix trees is a direct replacement of flex_array. The link - https://www.kernel.org/doc/html/v4.14/core-api/flexible-arrays.html  states the purpose of flexible arrays was to handle the described issue. Currently the only user is procfs. flexible arrays never had that many users.
            paf0186 Patrick Farrell added a comment -

            Can you provide more references on how genradix trees are used to solve this memory allocation problem?

            The description in the kernel docs doesn't suggest they're for this sort of problem:
            "Generic radix trees/sparse arrays

            Very simple and minimalistic, supporting arbitrary size entries up to PAGE_SIZE."[
            https://www.kernel.org/doc/html/latest/core-api/generic-radix-tree.html]

            paf0186 Patrick Farrell added a comment - Can you provide more references on how genradix trees are used to solve this memory allocation problem? The description in the kernel docs doesn't suggest they're for this sort of problem: " Generic radix trees/sparse arrays Very simple and minimalistic, supporting arbitrary size entries up to PAGE_SIZE."[ https://www.kernel.org/doc/html/latest/core-api/generic-radix-tree.html ]

            People

              simmonsja James A Simmons
              simmonsja James A Simmons
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: