Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-10329

DNE3: REMOTE_PARENT_DIR scalability

    XMLWordPrintable

Details

    • Improvement
    • Resolution: Unresolved
    • Critical
    • Lustre 2.17.0
    • None

    Description

      For DNE filesystems where there are large numbers of remote entries, for example once LU-4684 is landed to restripe directories, or DNE2 striped directories with many renames, the size of the REMOTE_PARENT_DIR may become very large.

      In order to limit contention and scaling issues in REMOTE_PARENT_DIR it makes sense to have multiple such directories. As a starting point, one REMOTE_PARENT_DIR_MDTxxxx for each remote MDT would be useful, but it may be necessary to have a tree of directories similar to the O/<seq>/dN object directories. Having a separate REMOTE_PARENT_DIR_MDTxxxx per MDT would also allow LFSCK to efficiently scan remote entries for a given MDT, if there was a problem (e.g. MDT was marked offline and returned into the namespace later).

      Attachments

        Issue Links

          has to be done before

          New Feature - A new feature of the product, which has yet to be developed. LU-11025 DNE3: directory restripe

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LU-10784 DNE3: mkdir() automatically create remote directory on MDS which has more space

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. LU-15383 DNE directories not connected to REMOTE_PARENT_DIR

          • Critical - Crashes, loss of data, severe memory leak.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. LU-4876 LFSCK remove entry from /REMOTE_PARENT_DIR if MDT-object name reside on the same MDT after migration

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. LU-11446 ldiskfs inodes nlink mismatch with DNE

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. LU-11502 add striped directory migration support in racer test

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. LU-15330 ext2fs_get_pathname() very slow for large directory

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. LU-12310 MDT Device-level Replication/Mirroring

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. LU-15307 Option to set max directory stripe count per filesystem

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LU-13440 DNE3: limit directory default layout inheritance

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. LU-12892 Large directory feature is not enabled (REMOTE_PARENT_DIR)

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. LU-11546 enable large_dir support for MDTs

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Technical task - A technical task. LU-10192 Agent entry for cross-MDTs reference

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. LU-4684 DNE3: allow migrating DNE striped directory

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              laisiyao Lai Siyao
              adilger Andreas Dilger
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated: