https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12892 Lustre <p>We have a problem that might be related to DNE. One of our MDT on Fir (2.12.3 RC1) started generating these errors:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>fir-md1-s3: Oct 21 14:21:10 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_003: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:10 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:10 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:10 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:11 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_003: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:11 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:27 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:27 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:27 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:27 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:30 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:30 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:30 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_004: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:30 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:34 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_004: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:34 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:39 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:39 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:43 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:43 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:47 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_003: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:47 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:51 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:51 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:52 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_003: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:52 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:53 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_004: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:53 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:53 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:53 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:59 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:59 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:59 fir-md1-s3 kernel: Lustre: 18617:0:(osd_handler.c:501:osd_ldiskfs_add_entry()) fir-MDT0002: directory (inode: 10999297, FID: [0x200000001:0x101b:0x0]) has reached maximum entry fir-md1-s3: Oct 21 14:21:59 fir-md1-s3 kernel: Lustre: 18617:0:(osd_handler.c:501:osd_ldiskfs_add_entry()) Skipped 16 previous similar messages fir-md1-s3: Oct 21 14:21:59 fir-md1-s3 kernel: LustreError: 18617:0:(osd_handler.c:4535:osd_xattr_handle_linkea()) fir-MDT0002: failed to create agent entry for [0x2c00013ae:0x6afc:0x0]: rc = -28 fir-md1-s3: Oct 21 14:21:59 fir-md1-s3 kernel: LustreError: 18617:0:(osd_handler.c:4535:osd_xattr_handle_linkea()) Skipped 16 previous similar messages fir-md1-s3: Oct 21 14:22:08 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:22:08 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:22:09 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_003: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:22:09 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:22:11 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_000: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:22:11 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:22:21 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_003: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:22:21 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:22:23 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_000: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:22:23 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@fir-md1-s3 ~]# debugfs -c -R "ncheck 10999297" /dev/mapper/md1-rbod2-mdt2 debugfs 1.44.5.wc1 (15-Dec-2018) /dev/mapper/md1-rbod2-mdt2: catastrophic mode - not reading inode or group bitmaps Inode Pathname 10999297 //REMOTE_PARENT_DIR </pre> </div></div> <p>The FID mentioned above:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lfs fid2path /scratch 0x2c00013ae:0x6afc:0x0 /scratch/users/amtseng/tf_dna/results/chipseq_runs/motifs_1/SRX482859/homerResults/motif3RV.logo.svg # lfs getdirstripe /scratch/users/amtseng/tf_dna/results/chipseq_runs/motifs_1/SRX482859/homerResults/ lmv_stripe_count: 0 lmv_stripe_offset: 2 lmv_hash_type: none # stat /scratch/users/amtseng/tf_dna/results/chipseq_runs/motifs_1/SRX2789670/knownResults/known68.logo.svg stat: cannot stat ‘/scratch/users/amtseng/tf_dna/results/chipseq_runs/motifs_1/SRX2789670/knownResults/known68.logo.svg’: No such file or directory </pre> </div></div> CentOS 7.6 LU-12892

Large directory feature is not enabled (REMOTE_PARENT_DIR)

Bug Critical Resolved Duplicate WC Triage Stephane Thiell Mon, 21 Oct 2019 21:37:29 +0000 Fri, 4 Jun 2021 21:25:48 +0000 Tue, 22 Oct 2019 20:42:13 +0000 Lustre 2.12.3 0 3 <p>Additional notes</p> <p>This MDT was formatted with a pre-release of Lustre 2.12 (and associated e2fsprogs) if I remember correctly.<br/> We also used resize2fs one time to increase its size, followed by a successful e2fsck, but that was weeks ago and everything has been fine until now.</p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@fir-md1-s3 ~]# dumpe2fs -h /dev/mapper/md1-rbod2-mdt2 dumpe2fs 1.44.5.wc1 (15-Dec-2018) Filesystem volume name: fir-MDT0002 Last mounted on: / Filesystem UUID: 7cbc59d2-0567-4bef-bccc-89e81ff6372f Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr dir_index filetype needs_recovery extent 64bit mmp flex_bg dirdata sparse_super large_file huge_file uninit_bg dir_nlink quota project Filesystem flags: signed_directory_hash Default mount options: user_xattr acl Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 288005760 Block count: 4681213440 Reserved block count: 234060672 Free blocks: 2679153106 Free inodes: 155240256 First block: 0 Block size: 4096 Fragment size: 4096 Group descriptor size: 64 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 2016 Inode blocks per group: 504 Flex block group size: 16 Filesystem created: Thu Jan 24 14:00:46 2019 Last mount time: Wed Oct 16 13:59:18 2019 Last write time: Wed Oct 16 13:59:18 2019 Mount count: 2 Maximum mount count: -1 Last checked: Wed Oct 16 12:24:55 2019 Check interval: 0 (&lt;none&gt;) Lifetime writes: 96 TB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 1024 Required extra isize: 32 Desired extra isize: 32 Journal inode: 8 Default directory hash: half_md4 Directory Hash Seed: e4e06054-3eb2-461d-92ee-96e46d72af63 Journal backup: inode blocks MMP block number: 10335 MMP update interval: 5 User quota inode: 3 Group quota inode: 4 Project quota inode: 151 Journal features: journal_incompat_revoke journal_64bit Journal size: 4096M Journal length: 1048576 Journal sequence: 0x0116aefa Journal start: 769027 MMP_block: mmp_magic: 0x4d4d50 mmp_check_interval: 10 mmp_sequence: 0x015363 mmp_update_date: Mon Oct 21 14:39:39 2019 mmp_update_time: 1571693979 mmp_node_name: fir-md1-s3 mmp_device_name: dm-1 </pre> </div></div> <p>Actually there are several of these errors:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Oct 21 14:19:49 fir-md1-s3 kernel: LustreError: 18616:0:(osd_handler.c:4535:osd_xattr_handle_linkea()) fir-MDT0002: failed to create agent entry for [0x2c00013ab:0x8857:0x0]: rc = -28 Oct 21 14:19:59 fir-md1-s3 kernel: LustreError: 18617:0:(osd_handler.c:4535:osd_xattr_handle_linkea()) fir-MDT0002: failed to create agent entry for [0x2c00013ac:0x21f2:0x0]: rc = -28 Oct 21 14:20:22 fir-md1-s3 kernel: LustreError: 18616:0:(osd_handler.c:4535:osd_xattr_handle_linkea()) fir-MDT0002: failed to create agent entry for [0x2c00013ac:0x68d4:0x0]: rc = -28 Oct 21 14:20:54 fir-md1-s3 kernel: LustreError: 18616:0:(osd_handler.c:4535:osd_xattr_handle_linkea()) fir-MDT0002: failed to create agent entry for [0x2c00013ad:0x3ba9:0x0]: rc = -28 Oct 21 14:21:59 fir-md1-s3 kernel: LustreError: 18617:0:(osd_handler.c:4535:osd_xattr_handle_linkea()) fir-MDT0002: failed to create agent entry for [0x2c00013ae:0x6afc:0x0]: rc = -28 </pre> </div></div> <p>It looks like you don't have the "<tt>large_dir</tt>" feature enabled. This can be set with <tt>tune2fs</tt> on the MDT block device. The safest would be to do this while the MDT is unmounted, though in theory it would also work while the MDT is mounted. It is recommended to upgrade to <tt>e2fsprogs-1.45.3-wc1</tt> to get the latest fixes.</p> <p>Hi Andreas,</p> <p>Thanks! Wow, this is probably because we formatted our MDTs with some old <tt>e2fsprogs</tt> that didn't set the <tt>large_dir</tt> feature by default. In any case, we just had a short, early morning maintenance on the MDSes for another issue and I was able to add the flag on all of them while unmounted. Example with MDT0:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># dumpe2fs -h /dev/mapper/md1-rbod1-mdt0 | grep feat dumpe2fs 1.45.2.wc1 (27-May-2019) Filesystem features: has_journal ext_attr dir_index filetype needs_recovery extent 64bit mmp flex_bg dirdata large_dir sparse_super large_file huge_file uninit_bg dir_nlink quota project </pre> </div></div> <p>I think there is a kernel thread "ext4-rsv-conver" now running probably doing the change in background.</p> <p>I also updated e2fsprogs to the latest version found in the Whamcloud repo (1.45.2-wc1), thanks!</p> <p>This issue was likely triggered by some IO-500 benchmarks I have been running using DNEv2. We are pausing this effort to avoid further disruption (for now) but very happy to have found the cause of the issue so quickly. Thanks again.</p> <p>So do I understand correctly that this issue is resolved from a Stanford point of view and we can resolve this ticket as a duplicate of <a href="https://jira.whamcloud.com/browse/LU-11546" title="enable large_dir support for MDTs" class="issue-link" data-issue-key="LU-11546"><del>LU-11546</del></a>?</p> <p>Feel free to close this ticket if the <tt>large_dir</tt> feature is now added automatically by mkfs.lustre on MDTs (it's not clear to me at this point when reading <a href="https://jira.whamcloud.com/browse/LU-11546" title="enable large_dir support for MDTs" class="issue-link" data-issue-key="LU-11546"><del>LU-11546</del></a>). Thanks!</p> <p>Yes- that's the ticket Andreas identified for that purpose.</p> Related LU-11546 LU-14734 LU-13610 LU-10329 Development Rank 1|i00odr: Rank (Obsolete) 9223372036854775807 Severity