https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-497 Lustre <p>A tray on one of our 99K enclosures biffed last night causing the OSS to panic. When we got things more or less back in order we attempted fscks on all the LUNs associated with that server and succeeded on all but one. </p> <p>When I attempt to run the fsck the system complains about fsck.ext4 not being found. When I run fsck.ldiskfs on the trouble LUN I get the following:</p> <p><span class="error">&#91;root@aoss11 ~&#93;</span># fsck.ldiskfs /dev/sdg<br/> fsck-sdg<span class="error">&#91;7235&#93;</span>: running (null)<br/> fsck-sdg<span class="error">&#91;7235&#93;</span>: fsck.ldiskfs 1.41.10.sun2-4chaos (23-Jun-2010)<br/> fsck-sdg<span class="error">&#91;7235&#93;</span>: fsck.ldiskfs: MMP: fsck being run while trying to open /dev/sdg<br/> fsck-sdg<span class="error">&#91;7235&#93;</span>: <br/> fsck-sdg<span class="error">&#91;7235&#93;</span>: The superblock could not be read or does not describe a correct ext2<br/> fsck-sdg<span class="error">&#91;7235&#93;</span>: filesystem. If the device is valid and it really contains an ext2<br/> fsck-sdg<span class="error">&#91;7235&#93;</span>: filesystem (and not swap or ufs or something else), then the superblock<br/> fsck-sdg<span class="error">&#91;7235&#93;</span>: is corrupt, and you might try running e2fsck with an alternate superblock:<br/> fsck-sdg<span class="error">&#91;7235&#93;</span>: e2fsck -b 32768 &lt;device&gt;<br/> fsck-sdg<span class="error">&#91;7235&#93;</span>: <br/> fsck-sdg<span class="error">&#91;7235&#93;</span>: exit code 8 (operational error)</p> <p>When I go to the alternate superblocks (only three get listed) I get the same error.</p> <p>The odd thing is if I do a tunefs.lustre on the device I gives me all the information on the OST. </p> <p>If I try to run dumpe2fs it spits out some of the disk info then just waits. I can break out of the command but even if I run the command on one of the good LUNs I get the same results. I don't know how to try to find any additional superblocks. </p> <p>This is a production file system so we are obviously down and critical. Any assistance would be greaty appreciated.</p> chaos version 4.4-2 on dell R710 servers connection via IB to DDN S2A9900. LU-497

DDN failure - Now can't find a valid superblock

Task Critical Closed Fixed Andreas Dilger Joe Mervini Fri, 8 Jul 2011 15:08:06 +0000 Wed, 26 Oct 2011 19:52:56 +0000 Fri, 8 Jul 2011 23:39:33 +0000 Lustre 1.8.6 0 6 <p>Quick follow-up: so there's no confusion, we used fsck.ldiskfs on all the other devices successfully. </p> <p>Joe</p> <p>I am looking for an engineer to help you with this issue. Can I just confirm on the version of Lustre code that you are running. Is it really Lustre 1.8.6-wc1 or is it the Lustre 1.8.5 + patches bundled with the latest Chaos releae?</p> <p>Thanks</p> <p>Peter</p> <p>It is the Lustre 1.8.5 + patches bundled with the latest Chaos releas.e</p> <p>Older versions of e2fsck have some issues like this with the MMP block being left in a state where it reports e2fsck is still being run. Those problems have been fixed with newer e2fsck releases.</p> <p>In order to clear this flag in the MMP block you need to run:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> tune2fs -f -E clear_mmp /dev/sdg </pre> </div></div> <p>and then run e2fsck as normal. I would separately recommend upgrading e2fsprogs to 1.41.12.ora2, which contains several MMP fixes.</p> <p>Andreas,</p> <p>Thank you so much for the help. tun2fs -f -e clear_mmp /dev/sdg indeed cleared the way to run fsck. The delay in feedback was because we allowed one of the failed drives in the array to complete its rebuild before doing anything else. After that, we got real anal and shutdown the servers attached to the controller pair and restarted them.</p> <p>We then ran fsck will -n to see what got reported, then with -yDf. Both checks took more that 2 hours to complete combined. We then ran ll_recover_lost_found_objs on the LUN mounted ldiskfs and which restored all objects in lost+found.</p> <p>We then brought the whole file system back online and everything is back to normal.</p> <p>Thanks again for the quick response.</p> <p>Joe</p> <p>Glad to hear that normal service has been resumed</p> <p>Peter</p> Development Rank 1|hzw2kn: Rank (Obsolete) 10499