Details
-
Bug
-
Resolution: Won't Fix
-
Minor
-
Lustre 2.1.1
-
DDN SFA10k - Dell R710 - TOSS2.0 OS release
-
3
-
4619
Description
After adding a network to the file system and adding the IP for the failover node to the MDS it wouldn't mount. (I later found that --param failnode= is no longer valid - much to my chagrin) I attempted to run fsck against the file system but it responded that the e2fsprogs was out of date for the file system so I ran fsck.ldiskfs. The fsck.ldiskfs found some bad inodes and corrected them but on a subsequent run with the -n option (done to make sure it was clean) I started seeing a flood of "dirdata length set incorrectly" messages. I stopped it and was able to mount the FS but later the FS spontaneously unmounted.
What does this mean? Fortunately this file system is in pre-production and can be recreated (which is intended) but I'd like to know if this was caused by running fsck.ldiskfs since I did not see these messages on the first pass. The version of e2fsprogs (non-Redhat) is ldiskfsprogs-1.41.90.3chaos.wc3-0.ch5.x86_64. I have downloaded the wc4 version from the WC repo and installed it into a test image where I have rebooted the node into. I was able to use e2fsck to check the FS and I am using -fDy options but the "dirdata length set incorrectly" message continues to stream and has been going for more that an hour.
Any help would be appreciated.
Attachments
Issue Links
- is related to
-
-
- Resolved
-
-
-
- Resolved
-
- Trackbacks
-
![[Confluence: Whamcloud Community Space]](/images/icons/generic_link_16.png "[Confluence: Whamcloud Community Space]")
Changelog 2.1
Changes from version 2.1.1 to version 2.1.2 Server support for kernels: 2.6.18308.4.1.el5 (RHEL5) 2.6.32220.17.1.el6 (RHEL6) Client support for unpatched kernels: 2.6.18308.4.1.el5 (RHEL5) 2.6.32220.17.1....
Activity
Ah, I see what happened, the v1.42.3.wc1 tag is actually a different commit than the commit on master-lustre.
* 9a5ba10 (tag: v1.42.3.wc1) e2fsck: allow checking on mounted root filesystem | * f7a92f9 (wc/master-lustre) e2fsck: allow checking on mounted root filesystem |/
You might want to just force-update master-lustre to be the commit that v1.42.3.wc1 tags. It looks like the only difference is the addition of the gerrit commit ID in the commit message in the tagged one.
So where does this leave us? Do we still think that something in osd-ldiskfs or somewhere else in lustre needs fixing, or do we no believe that e2fsck is entirely to blame?
Whoops, I needed an explicit "fetch --tags". Must have that remote configured wrong.
I wee the v1.42.3-lustre branch, but not the 1.42.3.wc1 tag.
The e2fsck fix for this is included into the rebased e2fsprogs-1.42.3.wc1 build, currently undergoing testing.
The "Flags: 0x80000" line maps to EXT4_EXTENTS_FL, so in fact it seems this is being set/inherited incorrectly on the MDT fast symlinks. Note "Fast_link_dest: ../bin/passwd" indicates that the symlink is indeed stored inside the inode.
My first guess is a defect in the osd-ldiskfs code that is unconditionally setting LDISKFS_EXTENTS_FL on all inodes, when this should only be set on regular files.
I ran the test (mostly out of curiosity and for my own understanding). When I ran lsattr against a linked file I got operation not supported:
root@cmds1 bin2]# lsattr /mnt/ROOT/jamervi/bin/passwd
------------e /mnt/ROOT/jamervi/bin/passwd
[root@cmds1 bin2]# lsattr /mnt/ROOT/jamervi/bin2/passwd
lsattr: Operation not supported While reading flags on /mnt/ROOT/jamervi/bin2/passwd
But when I ran debugfs (and I really don't know how to interpret the output) it appears to me that there are not extents associated with the symlink. At least none are explicitly called out. Am I interpreting this correctly?
[root@cmds1 bin2]# debugfs /dev/mapper/3600c0ff00011bdb4b12c0b4f01000000
debugfs 1.41.12 (17-May-2010)
debugfs: cd /mnt/ROOT
/mnt/ROOT: File not found by ext2_lookup
debugfs: cd /mnt
/mnt: File not found by ext2_lookup
debugfs: cd ROOT
debugfs: cd jamervi
debugfs: cd bin
debugfs: stat passwd
Inode: 405079025 Type: regular Mode: 0755 Flags: 0x80000
Generation: 3906041213 Version: 0x00000001:000011dd
User: 0 Group: 0 Size: 0
File ACL: 0 Directory ACL: 0
Links: 1 Blockcount: 0
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x4fb68353:00000000 – Fri May 18 11:13:55 2012
atime: 0x4fb68353:00000000 – Fri May 18 11:13:55 2012
mtime: 0x4fb68353:00000000 – Fri May 18 11:13:55 2012
crtime: 0x4fb68353:ad6a06dc – Fri May 18 11:13:55 2012
Size of extra inode fields: 28
Extended attributes stored in inode body:
lma = "00 00 00 00 00 00 00 00 00 04 00 00 02 00 00 00 f0 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0
0 00 00 00 00 00 00 00 00 00 00 " (64)
link = "df f1 ea 11 01 00 00 00 30 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 18 00 00 00 02 00 00 04 00 00 00 00 02 00 00 00 00 70 61 73 73 77 64 " (48)
lov = "d0 0b d1 0b 01 00 00 00 f0 08 00 00 00 00 00 00 00 04 00 00 02 00 00 00 00 00 10 00 01 00 00 00 1d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 2e 0
0 00 00 " (56)
EXTENTS:
debugfs: cd ../bin2
debugfs: stat passwd
Inode: 405082862 Type: symlink Mode: 0777 Flags: 0x80000
Generation: 3906045050 Version: 0x00000001:000026ee
User: 0 Group: 0 Size: 13
File ACL: 0 Directory ACL: 0
Links: 1 Blockcount: 0
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x4fb68377:00000000 – Fri May 18 11:14:31 2012
atime: 0x4fb683cb:1cb0d110 – Fri May 18 11:15:55 2012
mtime: 0x4fb68377:00000000 – Fri May 18 11:14:31 2012
crtime: 0x4fb68377:6c55b22c – Fri May 18 11:14:31 2012
Size of extra inode fields: 28
Extended attributes stored in inode body:
lma = "00 00 00 00 00 00 00 00 00 04 00 00 02 00 00 00 ed 17 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0
0 00 00 00 00 00 00 00 00 00 00 " (64)
link = "df f1 ea 11 01 00 00 00 30 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 18 00 00 00 02 00 00 04 00 00 00 00 04 00 00 00 00 70 61 73 73 77 64 " (48)
Fast_link_dest: ../bin/passwd
Sorry, I wasn't really using my terms consistently. The fast symlinks are those stored directly in the inode, while slow symlinks are stored in an external block. These correspond to short and long symlinks (the boundary being at 60 bytes).
I think the issue may be that if the symlink is stored in the inode (fast symlink) but the EXTENTS flag is set, that this may incorrectly be interpreting the symlink text as extent data, and e2fsck considers this a corrupt inode.
To test this theory, an MDT filesystem with extents enabled should get some symlinks created, then mounted as ldiskfs and lsattr run on the symlinks to see if the extent flag is set. Alternately, debugfs "stat" can be used ok the inodes to print the flags.
Not to detour from the subject of this ticket, but could you explain the difference between fast, short and long symlinks? I wanted to keep my ignorance on the down-low by checking the web and with several people here, but no one seems to know.
So that explains why the "extent" option was set for the MDT filesystem. That said, with the patch in http://review.whamcloud.com/2798 it will explicitly unset the extents feature for the MDT filesystem to avoid this problem for new filesystems.
We still need to understand/address the extents symlink problem. I see commits related to symlinks with extents (below), but it isn't clear whether the problem only applies to short symlinks, or long symlinks as well? Given that there are reports of many symlinks being deleted, I would suspect that the problem is with fast symlinks, and somehow the MDT is setting the "EXTENTS_FL" for symlinks, when it shouldn't be doing that.
Author: Theodore Ts'o <tytso@mit.edu>
Date: Thu Mar 13 23:13:18 2008 -0400
e2fsck: Check for fast symlinks that have EXTENTS_FL set
These shouldn't show up in the wild, but if they do, e2fsck will offer
to clear them.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
commit 7cadc57780f3e3e8e644e8976e11a336902d4a25
Author: Theodore Ts'o <tytso@mit.edu>
Date: Thu Mar 13 23:05:00 2008 -0400
e2fsck: Support long symlinks which use extents
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
The fix for e2fsck breaking dirdata with "-fD" is fixed in 1.42.3.wc1. The mkfs_lustre.c code now also explicitly disables extents (in b2_1 and master), which will avoid this problem for new filesystems in the future.
What still appears to need fixing is the use of the EXT4_EXTENTS_FL on short symlinks in the osd-ldiskfs code. This would need a special conf-sanity.sh test that tries to format the MDT with extents enabled, since we don't do that by default (specifying '--mkfsoptions="-O extents"' would override the "^extents" option specified internal to mkfs_lustre.c).