https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1281 Lustre <p>During some maintenance today we did run read-only fsck on all our OSTs in one of our file systems after shutting down the file system. (All OSTs are unmounted.)</p> <p>On some OSTs we see these errors:</p> <p>sudo e2fsck -v -f -n /dev/mapper/ost_lustre01_1<br/> e2fsck 1.41.90.wc4 (01-Sep-2011)<br/> MMP interval is 13 seconds and total wait time is 54 seconds. Please wait...<br/> Pass 1: Checking inodes, blocks, and sizes<br/> Inodes that were part of a corrupted orphan linked list found. Fix? no</p> <p>Inode 393896873 was part of the orphaned inode list. IGNORED.<br/> Inode 393896873 is in use, but has dtime set. Fix? no</p> <p>Inode 393896873, i_size is 0, should be 12288. Fix? no</p> <p>Pass 2: Checking directory structure<br/> Pass 3: Checking directory connectivity<br/> Pass 4: Checking reference counts<br/> Pass 5: Checking group summary information</p> <p>lustre01-OST0001: ********** WARNING: Filesystem still has errors **********</p> <p> 890452 inodes used (0.18%)<br/> 257084 non-contiguous files (28.9%)<br/> 32 non-contiguous directories (0.0%)</p> <ol> <li>of inodes with ind/dind/tind blocks: 32/0/0<br/> Extent depth histogram: 778159/111004/1240<br/> 1437482870 blocks used (73.59%)<br/> 0 bad blocks<br/> 55 large files</li> </ol> <p> 890404 regular files<br/> 39 directories<br/> 0 character device files<br/> 0 block device files<br/> 0 fifos<br/> 0 links<br/> 0 symbolic links (0 fast symbolic links)<br/> 0 sockets<br/> --------<br/> 890443 files</p> <p>We have not yet attempted to mount these OSTs. Can these errors be ignored? Or should we just let fsck fix them?</p> RHEL5 OSSes, DDN 9900 controller. LU-1281

e2fsck: Inodes that were part of a corrupted orphan linked list found.

Task Minor Resolved Fixed Andreas Dilger Frederik Ferner Tue, 3 Apr 2012 15:04:43 +0000 Mon, 16 Jul 2012 12:10:00 +0000 Mon, 16 Jul 2012 12:10:00 +0000 Lustre 1.8.x (1.8.0 - 1.8.5) 0 2 <p>This is a relatively benign error message. It means some object was being truncated, the OST crashed, and the truncate didn't complete correctly on recovery. Running e2fsck will repair the object to a "safe" size (12kB in this case), which might result in NUL padding at the end of the file.</p> <p>You can locate which MDS file this belongs to with the following commands (this is much easier on 2.x). On the OST run:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ost&gt; debugfs -c -R "stat &lt;393896873&gt;" /dev/mapper/ost_lustre01_1 debugfs 1.41.90.wc3 (28-May-2011) /dev/vgmyth/lvmythost0: catastrophic mode - not reading inode or group bitmaps Inode: 63681 Type: regular Mode: 0666 Flags: 0x80000 Generation: 2393149953 Version: 0x0000002a:00005f81 User: 1000 Group: 1000 Size: 25165824 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 49152 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x4bad91f0:00000000 -- Fri Mar 26 23:04:48 2010 atime: 0x4bad91f0:00000000 -- Fri Mar 26 23:04:48 2010 mtime: 0x4bad91d1:00000000 -- Fri Mar 26 23:04:17 2010 crtime: 0x4bad87f0:d2d42b84 -- Fri Mar 26 22:22:08 2010 Size of extra inode fields: 28 Extended attributes stored in inode body: fid = "b9 da 24 00 00 00 00 00 6a fa 0d 3f 01 00 00 00 eb 5b 0b 00 00 00 00 00 00 00 00 00 00 00 00 00 " (32) fid: objid=744427 seq=0 parent=[0x24dab9:0x3f0dfa6a:0x0] stripe=1 EXTENTS: (0-255):4620544-4620799, (256-6143):4621312-4627199 </pre> </div></div> <p>Of interest is the "fid:" line, where it reports the parent inode number (the first number inside the square brackets, 0x24dab9 in this example).</p> <p>On the MDT run:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>mdt&gt; debugfs -c -R "ncheck 0x24dab9" /dev/{MDT} debugfs 1.41.90.wc3 (28-May-2011) /dev/vgmyth/lvmythmdt0.ssd: catastrophic mode - not reading inode or group bitmaps Inode Pathname 2415289 /ROOT/tmp/4stripe </pre> </div></div> <p>If your filesystem is large, ncheck may take some time. If there are multiple inodes affected from multiple OSTs, you can list them all on the same "ncheck" line to avoid scanning the filesystem multiple times. In Lustre 2.x this "ncheck" step can be avoided, and the pathname can be resolved directly from the FID (e.g. "lfs fid2path <span class="error">&#91;0x24dab9:0x3f0dfa6a:0x0&#93;</span>").</p> <p>Andreas,</p> <p>thanks for your comment. Based on this, I have managed to identify 5 (out of 6) files, for on FID ncheck did not return any file on the MDT. I suspect lfsck would report that as orphaned object?</p> <p>In addition to this, I ran e2fsck -v -f -p on the affected OSTs, e2fsck managed to repair only one off them. In all other cases it stopped without repairing the corrupted orphan link list:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[bnh65367@cs04r-sc-oss01-05 ~]$ sudo e2fsck -v -f -p /dev/mapper/ost_lustre01_1 lustre01-OST0001: Inodes that were part of a corrupted orphan linked list found. lustre01-OST0001: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY. (i.e., without -a or -p options) [bnh65367@cs04r-sc-oss01-05 ~]$ [bnh65367@cs04r-sc-oss01-05 ~]$ sudo e2fsck -v -f -n /dev/mapper/ost_lustre01_1 e2fsck 1.41.90.wc4 (01-Sep-2011) Pass 1: Checking inodes, blocks, and sizes Inodes that were part of a corrupted orphan linked list found. Fix? no Inode 393896873 was part of the orphaned inode list. IGNORED. Inode 393896873 is in use, but has dtime set. Fix? no Inode 393896873, i_size is 0, should be 12288. Fix? no Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information lustre01-OST0001: ********** WARNING: Filesystem still has errors ********** 890452 inodes used (0.18%) 257084 non-contiguous files (28.9%) 32 non-contiguous directories (0.0%) # of inodes with ind/dind/tind blocks: 32/0/0 Extent depth histogram: 778159/111004/1240 1437482870 blocks used (73.59%) 0 bad blocks 55 large files 890404 regular files 39 directories 0 character device files 0 block device files 0 fifos 0 links 0 symbolic links (0 fast symbolic links) 0 sockets -------- 890443 files [bnh65367@cs04r-sc-oss01-05 ~]$ </pre> </div></div> <p>After checking the file names of the affected files, I'm not too concerned as they are either no longer important or should be relatively easy to recreate. I would still prefer to fix the OSTs.</p> <p>(Sorry forgot to add this<img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>I'm also usually very cautious when it comes to running fsck manually without knowing what it does. </p> <p>One one OST e2fsck manage to fix it:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">[bnh65367@cs04r-sc-oss01-05 ~]$ sudo e2fsck -v -f -n /dev/mapper/ost_lustre01_0e2fsck 1.41.90.wc4 (01-Sep-2011) Pass 1: Checking inodes, blocks, and sizes Inode 2 creation time (Thu Jan 1 02:08:16 1970) invalid. Clear? no Inodes that were part of a corrupted orphan linked list found. Fix? no Inode 22242677 was part of the orphaned inode list. IGNORED. Inode 22242677 is in use, but has dtime set. Fix? no Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information lustre01-OST0000: ********** WARNING: Filesystem still has errors ********** 930294 inodes used (0.19%) 271214 non-contiguous files (29.2%) 32 non-contiguous directories (0.0%) # of inodes with ind/dind/tind blocks: 32/0/0 Extent depth histogram: 813541/115276/1428 1398216469 blocks used (71.58%) 0 bad blocks 36 large files 930246 regular files 39 directories 0 character device files 0 block device files 0 fifos 0 links 0 symbolic links (0 fast symbolic links) 0 sockets -------- 930285 files [bnh65367@cs04r-sc-oss01-05 ~]$ sudo e2fsck -v -f -p /dev/mapper/ost_lustre01_0 lustre01-OST0000: Truncating orphaned inode 22242677 (uid=499, gid=499, mode=0100666, size=0) lustre01-OST0000: Truncating orphaned inode 55623793 (uid=499, gid=101902, mode=0100666, size=0) lustre01-OST0000: Inode 2 creation time (Thu Jan 1 02:08:16 1970) invalid. CLEARED. 930294 inodes used (0.19%) 271214 non-contiguous files (29.2%) 32 non-contiguous directories (0.0%) # of inodes with ind/dind/tind blocks: 32/0/0 Extent depth histogram: 813541/115276/1428 1398216467 blocks used (71.58%) 0 bad blocks 36 large files 930246 regular files 39 directories 0 character device files 0 block device files 0 fifos 0 links 0 symbolic links (0 fast symbolic links) 0 sockets -------- 930285 files </pre> </div></div> <p>Running e2fsck -fp will repair only straight forward errors. That is a reasonable approach until one knows what the error is. In this case, you should run e2fsck -fy do that it answers "yes" to repairing these errors. </p> <p>Thanks for your reply. Unfortunately I had to mount the file system before I received the reply. I've identified all affected files and will schedule another maintenance window to fix it properly.</p> <p>Are there any questions related to this issue, or can this bug be closed?</p> <p>Apologies for the delay I missed the update.</p> <p>Yes, the ticket can be closed. Thanks.</p> <p>Thanks Frederik!</p> Development Rank 1|hzvzy7: Rank (Obsolete) 10070