[LU-497] DDN failure - Now can't find a valid superblock

• Type: Task
• Resolution: Fixed
• Priority: Critical
• Affects Version/s: Lustre 1.8.6
• Environment: chaos version 4.4-2 on dell R710 servers connection via IB to DDN S2A9900.
• Assignee: Andreas Dilger
• Reporter: Joe Mervini

Description

A tray on one of our 99K enclosures biffed last night causing the OSS to panic. When we got things more or less back in order we attempted fscks on all the LUNs associated with that server and succeeded on all but one.

When I attempt to run the fsck the system complains about fsck.ext4 not being found. When I run fsck.ldiskfs on the trouble LUN I get the following:

[root@aoss11 ~]# fsck.ldiskfs /dev/sdg
fsck-sdg[7235]: running (null)
fsck-sdg[7235]: fsck.ldiskfs 1.41.10.sun2-4chaos (23-Jun-2010)
fsck-sdg[7235]: fsck.ldiskfs: MMP: fsck being run while trying to open /dev/sdg
fsck-sdg[7235]:
fsck-sdg[7235]: The superblock could not be read or does not describe a correct ext2
fsck-sdg[7235]: filesystem. If the device is valid and it really contains an ext2
fsck-sdg[7235]: filesystem (and not swap or ufs or something else), then the superblock
fsck-sdg[7235]: is corrupt, and you might try running e2fsck with an alternate superblock:
fsck-sdg[7235]: e2fsck -b 32768 <device>
fsck-sdg[7235]:
fsck-sdg[7235]: exit code 8 (operational error)

When I go to the alternate superblocks (only three get listed) I get the same error.

The odd thing is if I do a tunefs.lustre on the device I gives me all the information on the OST.

If I try to run dumpe2fs it spits out some of the disk info then just waits. I can break out of the command but even if I run the command on one of the good LUNs I get the same results. I don't know how to try to find any additional superblocks.

This is a production file system so we are obviously down and critical. Any assistance would be greaty appreciated.

Issue Links

• Trackbacks: Lustre 1.8.x known issues tracker While testing against Lustre b18 branch, we would hit known bugs which were already reported in Lustre Bugzilla https://bugzilla.lustre.org/. In order to move away from relying on Bugzilla, we would create a JIRA