[LU-1365] Implement ldiskfs LARGEDIR support for e2fsprogs

• Type: New Feature
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.12.4
• Affects Version/s: Lustre 2.3.0
• Labels: LTS12 e2fsprogs ldiskfs patch
• Story Points: 3
• Assignee: Artem Blagodarenko (Inactive)
• Reporter: Andreas Dilger

Description

This INCOMPAT_LARGEDIR feature allows larger directories to be created in ldiskfs, both with directory sizes over 2GB and and a maximum htree depth of 3 instead of the current limit of 2. These features are needed in order to exceed the current limit of approximately 10M entries in a single directory. The INCOMPAT_LARGEDIR feature was added to ldiskfs as part of the pdirops LU-50 coding, but was not part of that project. As there is currently no mke2fs, e2fsck, or tune2fs support for INCOMPAT_LARGEDIR, this feature is disabled by default when creating a new ldiskfs filesystem as it would otherwise make the filesystem unrecoverable in the case that e2fsck needs to be run on it.

Tasks that need to be completed before INCOMPAT_LARGEDIR can be used:

• add support for the INCOMPAT_LARGEDIR and "large_dir" features to mke2fs/tune2fs
• add conf-sanity.sh test LARGEDIR and 3-level htree for local ldiskfs with 1kB blocksize up to 100k entries with 255-byte names (3-level exceeded at 48k entries). This might be done using a smaller number of hard-linked inodes (nlink_max = 65000), to avoid overhead of accessing and caching a large number of different inodes.
• add parallel-scale.sh test LARGEDIR and >2GB directories with Lustre using 255-byte names and 10M entries (2GB exceeded at 4M entries, 4GB exceeded at 8M entries). This might be done using a smaller number of hard-linked inodes (nlink_max = 65000), to avoid overhead of accessing and caching a large number of different inodes.
• e2fsck support for 3-level htree
• e2fsck support for directories larger than 2GB (using i_size_hi consistently for S_IFDIR() inodes)
• e2fsprogs regression test test for 3-level/2GB+ htree e2fsck, corruptions
• e2fsprogs add LARGEDIR feature to "tests/f_random_corruption"
• port the ext4-large-dir.patch with the INCOMPAT_LARGEDIR features (>2GB, 3-level htree) to the upstream kernel and submit to linux-ext4 list for review
• submit e2fsprogs patches to linux-ext4 list for review
• solicit testing of feature from community
• some time after e2fsprogs is released and available for download, a patch is needed to enable large_dir on new filesystems with mkfs.lustre
• updates to the user manual and release notes to describe how to enable the large_dir feature with tune2fs

Issue Links

• is duplicated by: LU-896 change e2fsprogs to make it allow directories over 2GB
• is duplicated by: LU-7932 Patch e2fsprogs to allow set special flag to FS increase a hash tree levels
• is related to: LU-11915 conf-sanity test 115 is skipped or hangs
• is related to: LU-14345 e2fsck of very large directories is broken
• is related to: LU-14734 enable large_dir on existing MDTs
• is related to: LU-11546 enable large_dir support for MDTs
• is related to: LU-8974 Сhange force_over_256tb lustre mount option to force_over_512tb for ldiskfs
• is related to: LU-50 pdirops patch for ldiskfs