[LU-1366] getting "dirdata length set incorrectly" running e2fsck

• Type: Bug
• Resolution: Won't Fix
• Priority: Minor
• Fix Version/s: Lustre 2.1.2
• Affects Version/s: Lustre 2.1.1
• Labels: llnl
• Environment: DDN SFA10k - Dell R710 - TOSS2.0 OS release
• Severity: 3
• Assignee: Zhenyu Xu
• Reporter: Joe Mervini

Description

After adding a network to the file system and adding the IP for the failover node to the MDS it wouldn't mount. (I later found that --param failnode= is no longer valid - much to my chagrin) I attempted to run fsck against the file system but it responded that the e2fsprogs was out of date for the file system so I ran fsck.ldiskfs. The fsck.ldiskfs found some bad inodes and corrected them but on a subsequent run with the -n option (done to make sure it was clean) I started seeing a flood of "dirdata length set incorrectly" messages. I stopped it and was able to mount the FS but later the FS spontaneously unmounted.

What does this mean? Fortunately this file system is in pre-production and can be recreated (which is intended) but I'd like to know if this was caused by running fsck.ldiskfs since I did not see these messages on the first pass. The version of e2fsprogs (non-Redhat) is ldiskfsprogs-1.41.90.3chaos.wc3-0.ch5.x86_64. I have downloaded the wc4 version from the WC repo and installed it into a test image where I have rebooted the node into. I was able to use e2fsck to check the FS and I am using -fDy options but the "dirdata length set incorrectly" message continues to stream and has been going for more that an hour.

Any help would be appreciated.

Issue Links

• is related to: LU-1774 fsck -fD corrupts filesystem
• is related to: LU-1540 e2fsck remove too many symlinks
• Trackbacks: Changelog 2.1 Changes from version 2.1.1 to version 2.1.2 Server support for kernels: 2.6.18308.4.1.el5 (RHEL5) 2.6.32220.17.1.el6 (RHEL6) Client support for unpatched kernels: 2.6.18308.4.1.el5 (RHEL5) 2.6.32220.17.1....

Comments

[James A Simmons]

Old ticket for unsupported version

[Andreas Dilger]

The fix for e2fsck breaking dirdata with "-fD" is fixed in 1.42.3.wc1. The mkfs_lustre.c code now also explicitly disables extents (in b2_1 and master), which will avoid this problem for new filesystems in the future.

What still appears to need fixing is the use of the EXT4_EXTENTS_FL on short symlinks in the osd-ldiskfs code. This would need a special conf-sanity.sh test that tries to format the MDT with extents enabled, since we don't do that by default (specifying '--mkfsoptions="-O extents"' would override the "^extents" option specified internal to mkfs_lustre.c).

[Christopher Morrone (Inactive)]

Ah, I see what happened, the v1.42.3.wc1 tag is actually a different commit than the commit on master-lustre.

* 9a5ba10 (tag: v1.42.3.wc1) e2fsck: allow checking on mounted root filesystem
| * f7a92f9 (wc/master-lustre) e2fsck: allow checking on mounted root filesystem
|/  

You might want to just force-update master-lustre to be the commit that v1.42.3.wc1 tags. It looks like the only difference is the addition of the gerrit commit ID in the commit message in the tagged one.

So where does this leave us? Do we still think that something in osd-ldiskfs or somewhere else in lustre needs fixing, or do we no believe that e2fsck is entirely to blame?

[Christopher Morrone (Inactive)]

Whoops, I needed an explicit "fetch --tags". Must have that remote configured wrong.

[Andreas Dilger]

The v1.42.3.wc1 tag is on the master-lustre branch.

[Christopher Morrone (Inactive)]

I wee the v1.42.3-lustre branch, but not the 1.42.3.wc1 tag.

[Andreas Dilger]

The e2fsck fix for this is included into the rebased e2fsprogs-1.42.3.wc1 build, currently undergoing testing.

[Andreas Dilger]

The "Flags: 0x80000" line maps to EXT4_EXTENTS_FL, so in fact it seems this is being set/inherited incorrectly on the MDT fast symlinks. Note "Fast_link_dest: ../bin/passwd" indicates that the symlink is indeed stored inside the inode.

My first guess is a defect in the osd-ldiskfs code that is unconditionally setting LDISKFS_EXTENTS_FL on all inodes, when this should only be set on regular files.

[Joe Mervini]

I ran the test (mostly out of curiosity and for my own understanding). When I ran lsattr against a linked file I got operation not supported:

root@cmds1 bin2]# lsattr /mnt/ROOT/jamervi/bin/passwd
------------e /mnt/ROOT/jamervi/bin/passwd
[root@cmds1 bin2]# lsattr /mnt/ROOT/jamervi/bin2/passwd
lsattr: Operation not supported While reading flags on /mnt/ROOT/jamervi/bin2/passwd

But when I ran debugfs (and I really don't know how to interpret the output) it appears to me that there are not extents associated with the symlink. At least none are explicitly called out. Am I interpreting this correctly?

[root@cmds1 bin2]# debugfs /dev/mapper/3600c0ff00011bdb4b12c0b4f01000000
debugfs 1.41.12 (17-May-2010)
debugfs: cd /mnt/ROOT
/mnt/ROOT: File not found by ext2_lookup
debugfs: cd /mnt
/mnt: File not found by ext2_lookup
debugfs: cd ROOT
debugfs: cd jamervi
debugfs: cd bin
debugfs: stat passwd
Inode: 405079025 Type: regular Mode: 0755 Flags: 0x80000
Generation: 3906041213 Version: 0x00000001:000011dd
User: 0 Group: 0 Size: 0
File ACL: 0 Directory ACL: 0
Links: 1 Blockcount: 0
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x4fb68353:00000000 – Fri May 18 11:13:55 2012
atime: 0x4fb68353:00000000 – Fri May 18 11:13:55 2012
mtime: 0x4fb68353:00000000 – Fri May 18 11:13:55 2012
crtime: 0x4fb68353:ad6a06dc – Fri May 18 11:13:55 2012
Size of extra inode fields: 28
Extended attributes stored in inode body:
lma = "00 00 00 00 00 00 00 00 00 04 00 00 02 00 00 00 f0 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0
0 00 00 00 00 00 00 00 00 00 00 " (64)
link = "df f1 ea 11 01 00 00 00 30 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 18 00 00 00 02 00 00 04 00 00 00 00 02 00 00 00 00 70 61 73 73 77 64 " (48)
lov = "d0 0b d1 0b 01 00 00 00 f0 08 00 00 00 00 00 00 00 04 00 00 02 00 00 00 00 00 10 00 01 00 00 00 1d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 2e 0
0 00 00 " (56)
EXTENTS:
debugfs: cd ../bin2
debugfs: stat passwd
Inode: 405082862 Type: symlink Mode: 0777 Flags: 0x80000
Generation: 3906045050 Version: 0x00000001:000026ee
User: 0 Group: 0 Size: 13
File ACL: 0 Directory ACL: 0
Links: 1 Blockcount: 0
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x4fb68377:00000000 – Fri May 18 11:14:31 2012
atime: 0x4fb683cb:1cb0d110 – Fri May 18 11:15:55 2012
mtime: 0x4fb68377:00000000 – Fri May 18 11:14:31 2012
crtime: 0x4fb68377:6c55b22c – Fri May 18 11:14:31 2012
Size of extra inode fields: 28
Extended attributes stored in inode body:
lma = "00 00 00 00 00 00 00 00 00 04 00 00 02 00 00 00 ed 17 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0
0 00 00 00 00 00 00 00 00 00 00 " (64)
link = "df f1 ea 11 01 00 00 00 30 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 18 00 00 00 02 00 00 04 00 00 00 00 04 00 00 00 00 70 61 73 73 77 64 " (48)
Fast_link_dest: ../bin/passwd

[Andreas Dilger]

Sorry, I wasn't really using my terms consistently. The fast symlinks are those stored directly in the inode, while slow symlinks are stored in an external block. These correspond to short and long symlinks (the boundary being at 60 bytes).

I think the issue may be that if the symlink is stored in the inode (fast symlink) but the EXTENTS flag is set, that this may incorrectly be interpreting the symlink text as extent data, and e2fsck considers this a corrupt inode.

To test this theory, an MDT filesystem with extents enabled should get some symlinks created, then mounted as ldiskfs and lsattr run on the symlinks to see if the extent flag is set. Alternately, debugfs "stat" can be used ok the inodes to print the flags.