https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8706 Lustre <p>We had several OSSes crash yesterday that appear to be related to quotas. It was exposed when I ran e2fsck against one of the OSTs and on a subsequent run after it was repaired, it started complaining about the htree and has been spewing "Unattached inode &lt;num&gt; connect to lost+found (similar to what was reported in <a href="https://jira.whamcloud.com/browse/LU-3542" title="deleted/unused inodes not actually cleared by e2fsck" class="issue-link" data-issue-key="LU-3542"><del>LU-3542</del></a>) for the past couple of hours. All the other OSTs checked fine when I just ran e2fsck without the -D option.</p> <p>Do I just let this run to completion or do I have alternatives? </p> toss-2.4-2 LU-8706

e2fsck -fDy running forever

Bug Minor Resolved Fixed Andreas Dilger Joe Mervini Thu, 13 Oct 2016 18:08:59 +0000 Fri, 14 Oct 2016 17:22:58 +0000 Fri, 14 Oct 2016 17:22:58 +0000 Lustre 2.5.3 0 3 <p>What version of e2fsck are you running? There was a bug in e2fsck (<a href="https://jira.whamcloud.com/browse/LU-7368" title="e2fsck unsafe to interrupt with quota enabled" class="issue-link" data-issue-key="LU-7368"><del>LU-7368</del></a>) that if you interrupted it when quotas was enabled that it could corrupt the filesystem. There was a second bug in the <tt>e2fsck -fD</tt> option (<a href="https://jira.whamcloud.com/browse/LU-7381" title="&quot;e2fsck -fD&quot; on directory may cause extent tree corruption" class="issue-link" data-issue-key="LU-7381"><del>LU-7381</del></a>) that was also fixed in e2fsprogs-1.42.13.wc5, which is the latest release.</p> <p>I suspect that there isn't anything to be done at this stage, and you need to let the e2fsck run to completion. If you are getting entries moved into lost+found, then you will need to run <tt>ll_recover_lost_found_objs</tt> on the OSTs to move the OST objects from <tt>lost+found</tt> back into the <tt>O/0/d&#42;</tt> directories.</p> <p>Andres - On this particular stack we're at 1.42.9-wc1-7 so I'll let it run out. Thank God for ll_recover_lost_found_objs! It's definitely saved me in the past...</p> <p>The fsck has been running now for 6.5 hours and the inode count that has been moved to lost+found is ~2.4M. I'm concerned about that directory space. Should I be?</p> <p>Another point - this file system is relatively new (just over a year old and I believe that it was formatted against 2.5.3). So I'm hoping that <a href="https://jira.whamcloud.com/browse/LU-7381" title="&quot;e2fsck -fD&quot; on directory may cause extent tree corruption" class="issue-link" data-issue-key="LU-7381"><del>LU-7381</del></a> is not relevant.</p> <p>Depending on the amount of corruption, you may have a substantial fraction of all OST objects moved into lost+found. The directory size itself is unlikely to be a problem, since it is grown as more entries are added. However, I believe the insertion is a linear search for each new object, so this has the potential to take a long time. How many objects are on your other OSTs?</p> <p>The fsck completed sometime over and ll_recover_lost_found_objs was successful and our file system is back online. I was surprised that the object recovery took only seconds to perform, especially since the fsck to SO long to connect what appeared to be every object to lost+found. I anticipated it would have taken hours. </p> <p>In any event, this ticket can be closed. Thanks for your support.</p> <p>Glad to hear that this problem was resolved.</p> <p>The ll_recover_lost_found_objs code runs much faster because it is using the hashed directory code in the kernel to insert objects into the <tt>O/0/d*</tt> directories, and since there are 32 of them they are 1/32 as large. The e2fsck code is doing a linear search and insertion for each entry in O(n^2) time, and this is a single directory that is 32x as large.</p> Related LU-7368 LU-7381 Development Rank 1|hzyrrr: Rank (Obsolete) 9223372036854775807 Severity