[LU-12892] Large directory feature is not enabled (REMOTE_PARENT_DIR)

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.12.3
• Type: Bug
• Priority: Critical
• Reporter: Stephane Thiell
• Assignee: WC Triage
• Resolution: Duplicate

Environment

CentOS 7.6

• Severity: 2

Description

We have a problem that might be related to DNE. One of our MDT on Fir (2.12.3 RC1) started generating these errors:

fir-md1-s3: Oct 21 14:21:10 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_003: index 2: reach max htree level 2
fir-md1-s3: Oct 21 14:21:10 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem
fir-md1-s3: Oct 21 14:21:10 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2
fir-md1-s3: Oct 21 14:21:10 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem
fir-md1-s3: Oct 21 14:21:11 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_003: index 2: reach max htree level 2
fir-md1-s3: Oct 21 14:21:11 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem
fir-md1-s3: Oct 21 14:21:27 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2
fir-md1-s3: Oct 21 14:21:27 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem
fir-md1-s3: Oct 21 14:21:27 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2
fir-md1-s3: Oct 21 14:21:27 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem
fir-md1-s3: Oct 21 14:21:30 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2
fir-md1-s3: Oct 21 14:21:30 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem
fir-md1-s3: Oct 21 14:21:30 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_004: index 2: reach max htree level 2
fir-md1-s3: Oct 21 14:21:30 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem
fir-md1-s3: Oct 21 14:21:34 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_004: index 2: reach max htree level 2
fir-md1-s3: Oct 21 14:21:34 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem
fir-md1-s3: Oct 21 14:21:39 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2
fir-md1-s3: Oct 21 14:21:39 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem
fir-md1-s3: Oct 21 14:21:43 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2
fir-md1-s3: Oct 21 14:21:43 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem
fir-md1-s3: Oct 21 14:21:47 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_003: index 2: reach max htree level 2
fir-md1-s3: Oct 21 14:21:47 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem
fir-md1-s3: Oct 21 14:21:51 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2
fir-md1-s3: Oct 21 14:21:51 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem
fir-md1-s3: Oct 21 14:21:52 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_003: index 2: reach max htree level 2
fir-md1-s3: Oct 21 14:21:52 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem
fir-md1-s3: Oct 21 14:21:53 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_004: index 2: reach max htree level 2
fir-md1-s3: Oct 21 14:21:53 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem
fir-md1-s3: Oct 21 14:21:53 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2
fir-md1-s3: Oct 21 14:21:53 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem
fir-md1-s3: Oct 21 14:21:59 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2
fir-md1-s3: Oct 21 14:21:59 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem
fir-md1-s3: Oct 21 14:21:59 fir-md1-s3 kernel: Lustre: 18617:0:(osd_handler.c:501:osd_ldiskfs_add_entry()) fir-MDT0002: directory (inode: 10999297, FID: [0x200000001:0x101b:0x0]) has reached maximum entry
fir-md1-s3: Oct 21 14:21:59 fir-md1-s3 kernel: Lustre: 18617:0:(osd_handler.c:501:osd_ldiskfs_add_entry()) Skipped 16 previous similar messages
fir-md1-s3: Oct 21 14:21:59 fir-md1-s3 kernel: LustreError: 18617:0:(osd_handler.c:4535:osd_xattr_handle_linkea()) fir-MDT0002: failed to create agent entry for [0x2c00013ae:0x6afc:0x0]: rc = -28
fir-md1-s3: Oct 21 14:21:59 fir-md1-s3 kernel: LustreError: 18617:0:(osd_handler.c:4535:osd_xattr_handle_linkea()) Skipped 16 previous similar messages
fir-md1-s3: Oct 21 14:22:08 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2
fir-md1-s3: Oct 21 14:22:08 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem
fir-md1-s3: Oct 21 14:22:09 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_003: index 2: reach max htree level 2
fir-md1-s3: Oct 21 14:22:09 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem
fir-md1-s3: Oct 21 14:22:11 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_000: index 2: reach max htree level 2
fir-md1-s3: Oct 21 14:22:11 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem
fir-md1-s3: Oct 21 14:22:21 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_003: index 2: reach max htree level 2
fir-md1-s3: Oct 21 14:22:21 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem
fir-md1-s3: Oct 21 14:22:23 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_000: index 2: reach max htree level 2
fir-md1-s3: Oct 21 14:22:23 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem

[root@fir-md1-s3 ~]# debugfs -c -R "ncheck 10999297" /dev/mapper/md1-rbod2-mdt2
debugfs 1.44.5.wc1 (15-Dec-2018)
/dev/mapper/md1-rbod2-mdt2: catastrophic mode - not reading inode or group bitmaps
Inode   Pathname
10999297        //REMOTE_PARENT_DIR

The FID mentioned above:

# lfs fid2path /scratch 0x2c00013ae:0x6afc:0x0
/scratch/users/amtseng/tf_dna/results/chipseq_runs/motifs_1/SRX482859/homerResults/motif3RV.logo.svg

# lfs getdirstripe /scratch/users/amtseng/tf_dna/results/chipseq_runs/motifs_1/SRX482859/homerResults/
lmv_stripe_count: 0 lmv_stripe_offset: 2 lmv_hash_type: none

# stat /scratch/users/amtseng/tf_dna/results/chipseq_runs/motifs_1/SRX2789670/knownResults/known68.logo.svg
stat: cannot stat ‘/scratch/users/amtseng/tf_dna/results/chipseq_runs/motifs_1/SRX2789670/knownResults/known68.logo.svg’: No such file or directory

Issue Links

• is related to: LU-11546 enable large_dir support for MDTs
• is related to: LU-14734 enable large_dir on existing MDTs
• is related to: LU-13610 ldiskfs_dx_add_entry large directory
• is related to: LU-10329 DNE3: REMOTE_PARENT_DIR scalability

Comments

[Stephane Thiell]

Additional notes

This MDT was formatted with a pre-release of Lustre 2.12 (and associated e2fsprogs) if I remember correctly.
We also used resize2fs one time to increase its size, followed by a successful e2fsck, but that was weeks ago and everything has been fine until now.

 

[root@fir-md1-s3 ~]# dumpe2fs -h /dev/mapper/md1-rbod2-mdt2
dumpe2fs 1.44.5.wc1 (15-Dec-2018)
Filesystem volume name:   fir-MDT0002
Last mounted on:          /
Filesystem UUID:          7cbc59d2-0567-4bef-bccc-89e81ff6372f
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr dir_index filetype needs_recovery extent 64bit mmp flex_bg dirdata sparse_super large_file huge_file uninit_bg dir_nlink quota project
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              288005760
Block count:              4681213440
Reserved block count:     234060672
Free blocks:              2679153106
Free inodes:              155240256
First block:              0
Block size:               4096
Fragment size:            4096
Group descriptor size:    64
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         2016
Inode blocks per group:   504
Flex block group size:    16
Filesystem created:       Thu Jan 24 14:00:46 2019
Last mount time:          Wed Oct 16 13:59:18 2019
Last write time:          Wed Oct 16 13:59:18 2019
Mount count:              2
Maximum mount count:      -1
Last checked:             Wed Oct 16 12:24:55 2019
Check interval:           0 (<none>)
Lifetime writes:          96 TB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          1024
Required extra isize:     32
Desired extra isize:      32
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      e4e06054-3eb2-461d-92ee-96e46d72af63
Journal backup:           inode blocks
MMP block number:         10335
MMP update interval:      5
User quota inode:         3
Group quota inode:        4
Project quota inode:      151
Journal features:         journal_incompat_revoke journal_64bit
Journal size:             4096M
Journal length:           1048576
Journal sequence:         0x0116aefa
Journal start:            769027
MMP_block:
    mmp_magic: 0x4d4d50
    mmp_check_interval: 10
    mmp_sequence: 0x015363
    mmp_update_date: Mon Oct 21 14:39:39 2019
    mmp_update_time: 1571693979
    mmp_node_name: fir-md1-s3
    mmp_device_name: dm-1

[Stephane Thiell]

Actually there are several of these errors:

Oct 21 14:19:49 fir-md1-s3 kernel: LustreError: 18616:0:(osd_handler.c:4535:osd_xattr_handle_linkea()) fir-MDT0002: failed to create agent entry for [0x2c00013ab:0x8857:0x0]: rc = -28
Oct 21 14:19:59 fir-md1-s3 kernel: LustreError: 18617:0:(osd_handler.c:4535:osd_xattr_handle_linkea()) fir-MDT0002: failed to create agent entry for [0x2c00013ac:0x21f2:0x0]: rc = -28
Oct 21 14:20:22 fir-md1-s3 kernel: LustreError: 18616:0:(osd_handler.c:4535:osd_xattr_handle_linkea()) fir-MDT0002: failed to create agent entry for [0x2c00013ac:0x68d4:0x0]: rc = -28
Oct 21 14:20:54 fir-md1-s3 kernel: LustreError: 18616:0:(osd_handler.c:4535:osd_xattr_handle_linkea()) fir-MDT0002: failed to create agent entry for [0x2c00013ad:0x3ba9:0x0]: rc = -28
Oct 21 14:21:59 fir-md1-s3 kernel: LustreError: 18617:0:(osd_handler.c:4535:osd_xattr_handle_linkea()) fir-MDT0002: failed to create agent entry for [0x2c00013ae:0x6afc:0x0]: rc = -28

[Andreas Dilger]

It looks like you don't have the "large_dir" feature enabled. This can be set with tune2fs on the MDT block device. The safest would be to do this while the MDT is unmounted, though in theory it would also work while the MDT is mounted. It is recommended to upgrade to e2fsprogs-1.45.3-wc1 to get the latest fixes.

[Stephane Thiell]

Hi Andreas,

Thanks! Wow, this is probably because we formatted our MDTs with some old e2fsprogs that didn't set the large_dir feature by default. In any case, we just had a short, early morning maintenance on the MDSes for another issue and I was able to add the flag on all of them while unmounted. Example with MDT0:

# dumpe2fs -h /dev/mapper/md1-rbod1-mdt0 | grep feat
dumpe2fs 1.45.2.wc1 (27-May-2019)
Filesystem features:      has_journal ext_attr dir_index filetype needs_recovery extent 64bit mmp flex_bg dirdata large_dir sparse_super large_file huge_file uninit_bg dir_nlink quota project

I think there is a kernel thread "ext4-rsv-conver" now running probably doing the change in background.

I also updated e2fsprogs to the latest version found in the Whamcloud repo (1.45.2-wc1), thanks!

This issue was likely triggered by some IO-500 benchmarks I have been running using DNEv2. We are pausing this effort to avoid further disruption (for now) but very happy to have found the cause of the issue so quickly. Thanks again.

[Peter Jones]

So do I understand correctly that this issue is resolved from a Stanford point of view and we can resolve this ticket as a duplicate of LU-11546?

[Stephane Thiell]

Feel free to close this ticket if the large_dir feature is now added automatically by mkfs.lustre on MDTs (it's not clear to me at this point when reading LU-11546). Thanks!

[Peter Jones]

Yes- that's the ticket Andreas identified for that purpose.