https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4553 Lustre <p>We had a discussion during the 2.6 testing concall about what the behaviour of LFSCK is, or should be, if there an OST that is permanently unavailable? </p> <p>If the administrator has permanently marked the OST inactive, will LFSCK ever be able to complete?</p> <p>Will it be able to detect and report objects that are on the failed OST? Is there an option to delete such files?</p> <p>Will it create new objects on a different OST?</p> <p>If there are no files referencing objects on the inactive OST (e.g. if administrator has previously run "lfs getstripe -O {OST}" or LFSCK to find and delete such files) will it be possible for LFSCK to complete?</p> LU-4553

LFSCK 5: LFSCK behaviour if an OST is permanently unavailable

Improvement Major Open Unresolved WC Triage Andreas Dilger Tue, 28 Jan 2014 18:26:04 +0000 Wed, 13 Feb 2019 07:17:41 +0000 Lustre 2.6.0 0 4 <p>According to current implementation, it allows to run lfsck_layout on part of OSTs, means as long as at least 1 OST joined the lfsck_layout, the scanning can start. If the LFSCK found that some OST-object exists on the OST which does not join the lfsck_layout (may be permanently unavailable, or crashed during the LFSCK scanning), it will add some flags, and continue the scanning, but the last status will be "incomplete" because some OST-objects have not been verified.</p> <p>Currently, the LFSCK can detect the case of missed some OST(s), but it will neither delete related file(s) nor related OST-object(s), it also cannot create new OST-object on different OST, because it does not know the data of the original OST-object.</p> <p>&gt; We had a discussion during the 2.6 testing concall about what the behaviour of LFSCK is, or should be, if there an OST that is permanently unavailable?<br/> The LFSCK should go ahead in spite of whether some OST(s) not join the LFSCK or not. It is necessary for a larger cluster. It is current implementation.</p> <p>&gt; If the administrator has permanently marked the OST inactive, will LFSCK ever be able to complete?<br/> The LFSCK still can complete even if some OST(s) not join the LFSCK or failed during the LFSCK. It is current implementation.</p> <p>&gt; Will it be able to detect and report objects that are on the failed OST? Is there an option to delete such files?<br/> Current implementation can detect the object which references failed OST, but it does not support to delete such file/object. It is not trouble to add option to support that. There are some dangerous to delete such file/objet, because it is not easy for the LFSCK to distinguish whether the OST is really permanently removed or just failed temporarily. So should we support that?</p> <p>&gt; Will it create new objects on a different OST?<br/> Is it enough to crate new object on a different OST? how to process the data stored on original OST?</p> <p>&gt; If there are no files referencing objects on the inactive OST (e.g. if administrator has previously run "lfs getstripe -O </p> {OST} <p>" or LFSCK to find and delete such files) will it be possible for LFSCK to complete?<br/> Yes.</p> <p>Andreas, what do you want me to do next step for this ticket?</p> <p>Make sure that the process in the Lustre manual is correct for deleting files with stripes on the removed OST using "lfs find". This partly relates to <a href="https://jira.whamcloud.com/browse/LU-1738" title="lfs find cannot find files in OST that died, whilst lfs getstripe can" class="issue-link" data-issue-key="LU-1738"><del>LU-1738</del></a> so that users/admins can find all files on the failed OST(s) and delete them.</p> <p>Preferred solution is to add an option to LFSCK to delete all files on the specified OST index, which it will verify is currently marked inactive and/or not present in the filesystem configuration.</p> <p>To make the LFSCK robust enough, we allow the LFSCK to run with some OSTs unavailable (or not join the LFSCK). It is normal that some OSTs may mount up after the LFSCK started. So from the LFSCK view, even if some stipe resides on some OST which is not up yet, the LFSCK cannot know whether the target OST is really permanently unavailable or not. To be safe, I prefer to make another "lfs destroy" tool to delete the files on specified OST, in spite of it is permanently unavailable or not. It seems more clean for me.</p> Related LU-1738 LU-1267 Development Rank 1|hzwds7: Rank (Obsolete) 12441