https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9670 Lustre <p>Recently our main production file system experienced a outage which has<br/> required a run of e2fsck on the back end OSTs. During the run we encountered<br/> some issues which might lead to data lost. We would like to ask Intel engineers<br/> who have a better understanding of the ext4 filesystem to look at the logs<br/> and report back what will be lost and how safe is it to repair. Their are<br/> two logs attached to this ticket. One is for raw data and the other, bad_luns.out,<br/> is the one we are most concern about.</p> Lustre 2.8 servers using ldiskfs in a RHEL6.9 environment. LU-9670

Advise on e2fsck fixing for OST backend

Bug Critical Resolved Fixed nasf James A Simmons Thu, 15 Jun 2017 15:14:45 +0000 Fri, 25 Aug 2017 20:38:10 +0000 Wed, 23 Aug 2017 15:52:56 +0000 Lustre 2.8.0 0 8 <p>Fan Yong</p> <p>Could you please advise on this one?</p> <p>Thanks</p> <p>Peter</p> <p>James, there isn't necessarily anything in the e2fsck that looks like it is very unusual, if one assumes that the RAID controllers lost their cache. There may be more or less things to be fixed by e2fsck once the journal is replayed (assuming these e2fsck runs were done on a live filesystem). </p> <p>I have checked the hundreds of logs, although there are a lot of inconsistency, most of them are Quota accounting inconsistency, that are not fatal and will not cause data lose. In further, you can recheck the quota by force via reset enable quota after the Lustre mounted up:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>lctl conf_param $FSNAME.quota.{mdt,ost}=none lctl conf_param $FSNAME.quota.{mdt,ost}=ug </pre> </div></div> <p>If you are afraid of losing any data, you can make some check in advance before e2fsck repairing. For example:</p> <p>1. Deleted inode XXX has zero dtime.<br/> That means the inode XXX has zero nlink count, but its dtime is not set. You can get the FID-in-LMA via:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>debugfs -c -R "stat &lt;XXX&gt;" $OST_dev </pre> </div></div> <p>With the FID, we can calculate its original namepath on the OST. It is expected that the original namepath should has been removed (because the target OST object was destroyed).</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>debugfs -c -R "stat $namepath" $OST_dev </pre> </div></div> <p>But if unfortunately the original namepath is still there, then we can use "debugfs sif &lt;XXX&gt; inodes_count 1", then e2fsck will either link it back to the original OST namespace. In fact, it is NOT recommended to fix the inconsistency like that manually, because if the original inode was really destroyed and its blocks have been reassigned to other inodes, then recovering it by force may cause double reference blocks and then cause data crash or lose. So it only can be used for some rare unfortunate corners.</p> <p>2. Inode bitmap differences: -NNN<br/> That means inode is not used, but marked in inode bitmap. If the indoe NNN was in using before the crash, then it is lost.</p> <p>3. Block bitmap differences: <del>(NNN</del>-MMM)<br/> That means blocks are not used, but marked in block bitmap. Since no inode reference the blocks, just let e2fsck to fix the bitmap.</p> <p>4. Inode XXX, i_blocks is NNN, should be MMM.<br/> That means the i_blocks does not match the real blocks usage. Under such case, we have to trust the real block usage even if the i_blocks was right. Just let e2fsck to fix the i_blocks.</p> <p>5. Inode XXX, i_size is NNN, should be MMM. <br/> That the means the i_size is smaller than the real blocks usage. Trust the real blocks usage and let e2fsck to fix the I_size. If the inconsistency was caused by crashed truncated (shrink) operation, then some stale data may be recovered at the tail.</p> <p>...</p> <p>Anyway, as Andreas commented, most of the inconsistency may be disappear after journal replayed.</p> <p>Anything can I do for this ticket?</p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=simmonsja" class="user-hover" rel="simmonsja">simmonsja</a>,</p> <p>Is there anything more you'd like us to do for this ticket?</p> <p>Regards,</p> <p>Brad</p> <p>Brad,</p> <p>We can close this out. Thanks for the help.</p> <p>Thanks, Jesse</p> Development Rank 1|hzzf73: Rank (Obsolete) 9223372036854775807 Severity