https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-860 Lustre <p>Hi,</p> <p>Some users at CEA site complain about inconsistencies between "lfs quota -u" vs "du -s" report.</p> <p>After long investigations, on site support finally found that the lost file system space is consumed by orphaned objids on OSTs, and is a consequence of <a href="https://jira.whamcloud.com/browse/LU-601" title="kernel BUG at fs/jbd2/transaction.c:1030" class="issue-link" data-issue-key="LU-601"><del>LU-601</del></a> work-around.<br/> When it was impossible to restart the MDS (systematically asserting in "tgt_recov"), the only solution was to mount the volume in ldiskfs mode and rename the PENDING subdirectory.<br/> Now, there are several old "PENDING* directories", and a lot of orphaned objids belonging to FIDs in these directories.</p> <p>In order to recover all this lost space, the support is asking if it is safe to run "lfsck", or if they have to build their own tool to offline parse all OSTs and remove all objids that belongs to FIDs in PENDING* directories ?</p> <p>Perhaps the PENDING directory was sometimes removed instead of renamed. In this case, is the recovery identical, or is there something else to do?</p> <p>TIA<br/> Patrick</p> <p>Below is the support report, and I have also attached the files containing the traces of the commands executed on Client, MDT and OST. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> #context: Some times ago, a few users started to report Lustre quotas inconsistencies regarding to the "lfs quota -u" report vs "du -s" over their full hierachy/sub-tree. "lfs quotacheck" did not fix inconsistencies. #consequences: Quotas are unusable and inaccurate for these users and (a lot ??) filesystem space is consumed by orphaned objids on OSTs. #details: 1st check made was to identify that the inconsistencies are due to real (and orphaned) filesystem space/blocks consumption and not only/just a bad Quota value !!... 2nd thing has been to identify that the orphaned objids belong to FIDs in the MDS multiple PENDING* directories that have been moved as part of LU-601 work-around !!! See [Client,MDT,OST]_side files showing the details. So what can we do now to recover all the space/blocks used by the orphaned objids ??? Can we safelly run "lfsck" or do we need to build our own tool to offline parse all OSTs and remove all objids that belongs to FIDs in PENDING* directories ??? </pre> </div></div> linux-2.6.32-71.24.1 LU-860

Lustre quota inconsistencies after multiple usages of LU-601 work-around

Bug Minor Resolved Fixed Zhenyu Xu Patrick Valentin Wed, 16 Nov 2011 13:11:24 +0000 Thu, 5 Jan 2012 09:41:36 +0000 Thu, 5 Jan 2012 09:41:36 +0000 Lustre 2.0.0 0 4 <p>I would not recommend to run lfsck which is time consuming. What about moving the files of all the PENDING* directories back to the namespace and unlinking them again (with "unlink" command instead of rm to avoid the stat) from a lustre client?</p> <p>Bobi</p> <p>Could you please comment on this one?</p> <p>Thanks</p> <p>Peter</p> <p>I think manually unlink the file is a doable makeshift. While we will try to fix <a href="https://jira.whamcloud.com/browse/LU-601" title="kernel BUG at fs/jbd2/transaction.c:1030" class="issue-link" data-issue-key="LU-601"><del>LU-601</del></a> issue to make things right, lest this issue happens again.</p> <p>Hello all,</p> <p>Just a quick comment on the "moving the files of all the PENDING* directories back to the namespace and unlinking them again" proposal from Johann, having a look to the MDT inodes in PENDING* directories I have found they don't have any "lov" EA !!... So how will the Unlink process be able to find the associated+orphaned objids we want to remove on the OSTs ???</p> <p>In the case it will fail, what do you think on mounting all OSTs and remove all objids referring to any PENDING*/&lt;FID&gt; in their "fid" EA ??</p> <p>And last, ok about "lfsck" heavy time consuming, but you did not answer on my original question "Can we safelly run lfsck ??" and I will precise it with "Is lfsck THE tool pushed/supported by WhamCloud to repair Lustre inconsistencies or not ???".</p> <p>Bruno.</p> <p>Yes, it is safe to run lfsck.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> lfsck is used to check and repair the distributed coherency of a Lustre filesystem. OPTIONS -c Create (empty) missing OST objects referenced by MDS inodes. -d Delete orphaned objects from the filesystem. Since objects on the OST are often only one of several stripes of a file it can be difficult to put multiple objects back together into a sin- gle usable file. -h Print a brief help message. -l Put orphaned objects into a lost+found directory in the root of the filesystem. -n Do not repair the filesystem, just perform a read-only check (default). -v Verbose operation - more verbosity by specifing option multiple times. </pre> </div></div> <p>BTW, it would be tedious work if you want mount all OSTs and finding all objects which have intended fid, while the fid info lies in the "fid" EA of OST objects. </p> <p>Note that it is possible to generate the lfsck mdsdb and ostdb while the filesystem is mounted, by running:</p> <p> e2fsck -fn --mdsdb </p> {mdsdb_file} /dev/{mdsdev}<br/> <br/> Note "-n" option here. The database file may be slightly inconsistent (e.g. contain files that were deleted during the run) but the code should handle this internally.<br/> <br/> When the databases are created, please run:<br/> <br/> lfsck -nv --mdsdb {mdsdb_file} <p> --ostdb </p> {ostdb_file ...} <p> </p> {lustre mountpoint} <p>(note again -n here) to ensure that this is doing what you expect it to (e.g. the number of orphaned objects is reasonable compared to the amount of missing space). The -n option will prevent lfsck from actually making any changes to the filesystem. I would recommend to check several of the OST objects that lfsck thinks should be deleted to get their MDS FID (this should be possible with 'debugfs -c -R "stat &lt;O/0/d$((objid % 32))/$objid" /dev/</p> {ostdev} <p>' which should print out the filter_fid xattr with the parent MDS inode number, and then run 'debugfs -c -R "ncheck $ino1 $ino2 $ino3 ..." /dev/</p> {mdsdev} <p>' to check whether any MDS inodes reference those objects.</p> <p>Yeah !! The "back to namespace"+unlink method has been applied to all "PENDING*/*" files/FIDs and 40TB have been "magically" recovered/freed with associated Quotas corrected !!!</p> <p>BTW, I am still puzzled on how the FID&lt;-&gt;ObjID<span class="error">&#91;s&#93;</span> relation has been reconstructed ??</p> <p>That's the magic of Christmas <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/><br/> More seriously, are you sure that the files under PENDING*/* had no LOV EA? Do you still have the output of debugfs against one of those file?</p> <p>I will try to find it in my logs and attach it ...</p> <p>In fact it takes more time/work finally to gat to Christmas !!...</p> <p>I need to tell the whole story now ... :</p> <p> _ instead of moving the "PENDING.old*" content back in the namespace, I moved the directories, uninked all their content/files, and finally "rmdir" all dirs !!.. </p> <p> _ it took quite some time to understand the following "live" ... This leaded to a situation where the original/1st PENDING directory was no longer present to satisfy to the multiple conditions/controls/attributes/... (link EA, inode/generation number in OI database, ...) occuring during FS/MDT start/mount !!!!</p> <p>But now FS is started finally and seems that we begin to ear some Christmas songs ...</p> <p>Bull confirm that this issue is now resolved</p> Development Rank 1|hzvhq7: Rank (Obsolete) 6521 Severity