[LU-5780] Corrupted OST and very long running e2fsck

• Type: Bug
• Resolution: Cannot Reproduce
• Priority: Major
• Affects Version/s: Lustre 2.5.2
• Environment: CentOS 6.5, kernel 2.6.32-431.17.1.el6_lustre.x86_64, e2fsprogs 1.42.12.wc1-7
• Severity: 3
• Assignee: Niu Yawei (Inactive)
• Reporter: Tommy Minyard

Description

We had an OST on Stampede experience corruption and e2fsck does not appear to be making much progress in the repairs.

The situation is that a drive failed and the mdadm software raid (RAID-6) detected the drive had failed and automatically added the hot spare into the raid-6 device (8+2 3TB drives). The rebuild of the device started, then paused shortly after the resync began at 0.3% progress and it started to issue handle_stripe errors as shown in the attached system log. The next morning, the system administrator came in and saw that the drive had failed and that the rebuild was not making progress (it was still stuck at 0.3% of rebuild). The admin then removed the failed drive from the array using the mdadm -r command and the array resumed the rebuild process again. At that point, everything looked fine with the filesystem, the rebuild looked like it was doing the right thing and filesystem was operating normally. Then at 16:49 ldiskfs detected corruption and remounted the filesystem as read-only. At this point, the rebuild of the array was still in process at about 50% and appeared to have been working correctly.

The main issue now is that e2fsck on the device is taking far too long and requiring too much memory. Right now we have been running e2fsck for more than 5 days and keeps spewing errors similar to:

Inode 76812 is too big. Truncate? yes

Block #1 (3115984703) causes symlink to be too big. CLEARED.
Block #2 (1071886003) causes symlink to be too big. CLEARED.
Block #3 (318283451) causes symlink to be too big. CLEARED.
Block #4 (1071886789) causes symlink to be too big. CLEARED.
Block #5 (3173575420) causes symlink to be too big. CLEARED.
Block #6 (1071898601) causes symlink to be too big. CLEARED.
Block #7 (742927844) causes symlink to be too big. CLEARED.
Block #8 (1071900730) causes symlink to be too big. CLEARED.
Block #9 (2126297851) causes symlink to be too big. CLEARED.
Block #10 (1071885849) causes symlink to be too big. CLEARED.
Block #11 (3395849213) causes symlink to be too big. CLEARED.
Too many illegal blocks in inode 76812.
Clear inode? yes

Inode 76865 has INDEX_FL flag set but is not a directory.
Clear HTree index? yes

Inode 76865 has INDEX_FL flag set on filesystem without htree support.
Clear HTree index? yes

Inode 76865, i_size is 2450935918319985952, should be 0. Fix? yes

Inode 76865, i_blocks is 144761049163443, should be 0. Fix? yes

Inode 76906 has compression flag set on filesystem without compression support. Clear? yes

Inode 76906 is too big. Truncate? yes

Block #797810728 (827805746) causes file to be too big. CLEARED.
Block #797810729 (1482179393) causes file to be too big. CLEARED.
Block #797810730 (842676282) causes file to be too big. CLEARED.
Block #797810731 (976236851) causes file to be too big. CLEARED.
Block #797810732 (842084665) causes file to be too big. CLEARED.
Block #797810733 (959657786) causes file to be too big. CLEARED.
Block #797810734 (828322868) causes file to be too big. CLEARED.
Block #797810735 (809127482) causes file to be too big. CLEARED.
Block #797810736 (1413567546) causes file to be too big. CLEARED.
Block #797810737 (156516673) causes file to be too big. CLEARED.
Block #797810738 (1917258029) causes file to be too big. CLEARED.
Too many illegal blocks in inode 76906.
Clear inode? yes

At this point we suspect that the array experienced too severe a corruption to recover. Any suggestions on how to get e2fsck to complete in a timely manner or somehow get the OST into a state that we could mount it and try to copy what files we can off of the device?

It would be helpful to have the e2fsck process show some type of progress like it does for ext3/4 filesystems to give us some indication as to how much longer it might take. The last e2fsck message indicates that it is on about inode 76906 and is using up about 70GB of memory. debugfs reports the following about the filesystem:

debugfs: stats
Filesystem volume name: scratch-OST0010
Last mounted on: /
Filesystem UUID: 2989173c-9d2b-4fe5-8f45-99c0eebfda57
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype extent 64bit flex_bg sparse_su
per large_file huge_file uninit_bg dir_nlink quota
Filesystem flags: signed_directory_hash
Default mount options: user_xattr acl
Filesystem state: not clean with errors
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 22892800
Block count: 5860530816
Reserved block count: 0
Free blocks: 1588593434
Free inodes: 18496224
First block: 0
Block size: 4096
Fragment size: 4096
Group descriptor size: 64
Reserved GDT blocks: 1024
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 128
Inode blocks per group: 8
RAID stride: 8
RAID stripe width: 64
Flex block group size: 256
Filesystem created: Tue Dec 11 08:50:05 2012
Last mount time: Tue Jun 17 18:29:25 2014
Last write time: Wed Oct 8 16:57:09 2014

Issue Links

• is related to: LU-5778 MDS not creating files on OSTs properly