Details
-
Bug
-
Resolution: Duplicate
-
Critical
-
None
-
Lustre 2.12.3
-
None
-
CentOS 7.6
-
2
-
9223372036854775807
Description
We have a problem that might be related to DNE. One of our MDT on Fir (2.12.3 RC1) started generating these errors:
fir-md1-s3: Oct 21 14:21:10 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_003: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:10 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:10 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:10 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:11 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_003: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:11 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:27 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:27 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:27 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:27 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:30 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:30 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:30 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_004: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:30 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:34 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_004: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:34 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:39 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:39 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:43 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:43 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:47 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_003: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:47 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:51 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:51 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:52 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_003: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:52 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:53 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_004: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:53 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:53 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:53 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:59 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:21:59 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:21:59 fir-md1-s3 kernel: Lustre: 18617:0:(osd_handler.c:501:osd_ldiskfs_add_entry()) fir-MDT0002: directory (inode: 10999297, FID: [0x200000001:0x101b:0x0]) has reached maximum entry fir-md1-s3: Oct 21 14:21:59 fir-md1-s3 kernel: Lustre: 18617:0:(osd_handler.c:501:osd_ldiskfs_add_entry()) Skipped 16 previous similar messages fir-md1-s3: Oct 21 14:21:59 fir-md1-s3 kernel: LustreError: 18617:0:(osd_handler.c:4535:osd_xattr_handle_linkea()) fir-MDT0002: failed to create agent entry for [0x2c00013ae:0x6afc:0x0]: rc = -28 fir-md1-s3: Oct 21 14:21:59 fir-md1-s3 kernel: LustreError: 18617:0:(osd_handler.c:4535:osd_xattr_handle_linkea()) Skipped 16 previous similar messages fir-md1-s3: Oct 21 14:22:08 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_001: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:22:08 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:22:09 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_003: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:22:09 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:22:11 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_000: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:22:11 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:22:21 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_003: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:22:21 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem fir-md1-s3: Oct 21 14:22:23 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2618: inode 10999297: comm mdt_out02_000: index 2: reach max htree level 2 fir-md1-s3: Oct 21 14:22:23 fir-md1-s3 kernel: LDISKFS-fs warning (device dm-1): ldiskfs_dx_add_entry:2622: Large directory feature is not enabled on this filesystem
[root@fir-md1-s3 ~]# debugfs -c -R "ncheck 10999297" /dev/mapper/md1-rbod2-mdt2 debugfs 1.44.5.wc1 (15-Dec-2018) /dev/mapper/md1-rbod2-mdt2: catastrophic mode - not reading inode or group bitmaps Inode Pathname 10999297 //REMOTE_PARENT_DIR
The FID mentioned above:
# lfs fid2path /scratch 0x2c00013ae:0x6afc:0x0 /scratch/users/amtseng/tf_dna/results/chipseq_runs/motifs_1/SRX482859/homerResults/motif3RV.logo.svg # lfs getdirstripe /scratch/users/amtseng/tf_dna/results/chipseq_runs/motifs_1/SRX482859/homerResults/ lmv_stripe_count: 0 lmv_stripe_offset: 2 lmv_hash_type: none # stat /scratch/users/amtseng/tf_dna/results/chipseq_runs/motifs_1/SRX2789670/knownResults/known68.logo.svg stat: cannot stat ‘/scratch/users/amtseng/tf_dna/results/chipseq_runs/motifs_1/SRX2789670/knownResults/known68.logo.svg’: No such file or directory
