Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-1365

Implement ldiskfs LARGEDIR support for e2fsprogs

    Details

    • Story Points:
      3
    • Rank (Obsolete):
      10210

      Description

      This INCOMPAT_LARGEDIR feature allows larger directories to be created in ldiskfs, both with directory sizes over 2GB and and a maximum htree depth of 3 instead of the current limit of 2. These features are needed in order to exceed the current limit of approximately 10M entries in a single directory. The INCOMPAT_LARGEDIR feature was added to ldiskfs as part of the pdirops LU-50 coding, but was not part of that project. As there is currently no mke2fs, e2fsck, or tune2fs support for INCOMPAT_LARGEDIR, this feature is disabled by default when creating a new ldiskfs filesystem as it would otherwise make the filesystem unrecoverable in the case that e2fsck needs to be run on it.

      Tasks that need to be completed before INCOMPAT_LARGEDIR can be used:

      • add support for the INCOMPAT_LARGEDIR and "large_dir" features to mke2fs/tune2fs
      • add conf-sanity.sh test LARGEDIR and 3-level htree for local ldiskfs with 1kB blocksize up to 100k entries with 255-byte names (3-level exceeded at 48k entries). This might be done using a smaller number of hard-linked inodes (nlink_max = 65000), to avoid overhead of accessing and caching a large number of different inodes.
      • add parallel-scale.sh test LARGEDIR and >2GB directories with Lustre using 255-byte names and 10M entries (2GB exceeded at 4M entries, 4GB exceeded at 8M entries). This might be done using a smaller number of hard-linked inodes (nlink_max = 65000), to avoid overhead of accessing and caching a large number of different inodes.
      • e2fsck support for 3-level htree
      • e2fsck support for directories larger than 2GB (using i_size_hi consistently for S_IFDIR() inodes)
      • e2fsprogs regression test test for 3-level/2GB+ htree e2fsck, corruptions
      • e2fsprogs add LARGEDIR feature to "tests/f_random_corruption"
      • port the ext4-large-dir.patch with the INCOMPAT_LARGEDIR features (>2GB, 3-level htree) to the upstream kernel and submit to linux-ext4 list for review
      • submit e2fsprogs patches to linux-ext4 list for review
      • solicit testing of feature from community
      • some time after e2fsprogs is released and available for download, a patch is needed to enable large_dir on new filesystems with mkfs.lustre
      • updates to the user manual and release notes to describe how to enable the large_dir feature with tune2fs

        Attachments

        1. conf-san-125-4.txt.tar.gzip
          22 kB
        2. conf-sanity-124-125.tar.bz2
          16 kB
        3. image.png
          image.png
          64 kB
        4. image-2.png
          image-2.png
          11 kB

          Issue Links

            Activity

              People

              • Assignee:
                artem_blagodarenko Artem Blagodarenko
                Reporter:
                adilger Andreas Dilger
              • Votes:
                0 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: