https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10349 Lustre <p>This ticket is created to handle NASA-specific debugging of corrupted PFIDs discussed in <a href="https://jira.whamcloud.com/browse/LU-10248" title="Need to update PFID of OST objects after layout change" class="issue-link" data-issue-key="LU-10248"><del>LU-10248</del></a>, as well as ensure ports of a fix for lfsck handling of repaired_unmatched_pair, and any other related questions to ensure proper running of lfsck and cleanup of the filesystem.</p> <p>servers: lustre-2.7.3-1nasS_mofed33v3g_2.6.32_642.15.1.el6.20170609.x86_64.lustre273.x86_64 (basically the old FE branch, plus several cherry-picked patches)</p> <p>clients: lustre-client-2.9.0-2.3nasC_mofed34v1_4.4.74_92.32.1.20170808_nasa.x86_64</p> <p>We hope to upgrade both to a 2.10.2-based build in the near future.</p> servers: lustre-2.7.3-1nasS_mofed33v3g_2.6.32_642.15.1.el6.20170609.x86_64.lustre273.x86_64<br/> clients: lustre-client-2.9.0-2.3nasC_mofed34v1_4.4.74_92.32.1.20170808_nasa.x86_64 LU-10349

debug and cleanup of corrupted PFID, unmatched MDT-object and OST-object pairs

Bug Critical Resolved Duplicate nasf Nathan Dauchy Thu, 7 Dec 2017 16:53:31 +0000 Mon, 26 Mar 2018 12:34:40 +0000 Mon, 26 Mar 2018 12:34:40 +0000 Lustre 2.7.0 Lustre 2.9.0 0 4 <p>At this time, it sounds like we should go with "choice 2" in the comment from nasf:<br/> <a href="https://jira.hpdd.intel.com/browse/LU-10248?focusedCommentId=215528&amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-215528" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-10248?focusedCommentId=215528&amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-215528</a></p> <p>We will leave the inconsistencies the way they are until the lfsck patch is available and we can upgrade the servers to include it.</p> <p>The only caveat is that we will be continuing to drain and completely remove multiple OSTs from this file system (to free up the hardware for spares) and I want to make sure that won't combine with the PFID issues and confuse lfsck. One OST is already gone, and the last dry-run lfsck did not appear to die on it so hopefully we are OK on that front... it's just the debug logs getting overwritten with the repaired_inconsistent_owner errors might have masked other problems. Please let me know ASAP if I should <b>not</b> remove additional OSTs.</p> <p> </p> <p>Fan Yong is already assisting here</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=ndauchy" class="user-hover" rel="ndauchy">ndauchy</a>, would you please to check whether or not your branch has the following two patches:<br/> <a href="https://review.whamcloud.com/#/c/16135/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/16135/</a><br/> <a href="https://review.whamcloud.com/#/c/30447/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/30447/</a></p> <p>If not, please try to apply them and re-run the dryun mode layout LFSCK? Thanks!</p> <p>Since we are based on 2.7.3 FE branch, I'm pretty confident we have the first of those patches, but not the second since it looks like you just created the backport (and it failed a build test). Will confirm though.</p> <p>Regardless, it will be a while before we can complete the server rebuild and take a downtime to apply it. In the meantime I need to move forward with removing additional OSTs from the file system to free up the hardware for spares. Do you recommend we run the current lfsck in non-dry-run mode, or just go ahead and remove the OST now and wait for the lfsck updates?</p> <p>Thanks!</p> <p>We do not have either.</p> <p>Review #16135 is in 'Need Code-Review' state for more than 1 year. Is it OK to cherry-pick as it is, Fan?</p> <blockquote> <p>Regardless, it will be a while before we can complete the server rebuild and take a downtime to apply it. In the meantime I need to move forward with removing additional OSTs from the file system to free up the hardware for spares. Do you recommend we run the current lfsck in non-dry-run mode, or just go ahead and remove the OST now and wait for the lfsck updates?</p></blockquote> <p>There are two kinds inconsistency reported by the layout LFSCK, one is inconsistent owner information that cause by some known layout LFSCK issue, can be ignored. Another is the unmatched MDT-object and OST-object pairs. Such inconsistency will NOT affect normal system access unless enabling I/O verification (disable by default) explicitly. So I would suggest to keep the system unchanged since it is without influence now.</p> <blockquote> <p>Review #16135 is in 'Need Code-Review' state for more than 1 year. Is it OK to cherry-pick as it is, Fan?</p></blockquote> <p>Yes, I think so. Such patch has already been landed to b2_8_fe and master. It is not on b2_7_fe may because b2_7 was some old at that time and such issue is not very serious.</p> <blockquote> <p>So I would suggest to keep the system unchanged since it is without influence now.</p></blockquote> <p>OK, we will proceed with removing the additional OSTs from the file system, and wait for install of the patched lfsck on the 2.7.3 servers and perform the cleanup later.  Thanks!</p> <p>The patch for repair unexpected inconsistent owner on b2_7_fe:<br/> <a href="https://review.whamcloud.com/30613" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30613</a></p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=ndauchy" class="user-hover" rel="ndauchy">ndauchy</a>,</p> <p>The patch <a href="https://review.whamcloud.com/#/c/30612/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/30612/</a> for fixing the issue of unexpected inconsistent owner has already been landed to master. And it has been ported to b2_7_fe branch via the patch <a href="https://review.whamcloud.com/30613" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30613</a>. You can use related patch to resolve your system trouble. Please let me you what else you need.</p> <p>The above patch #30612 for b2_10 is at<br/> <a href="https://review.whamcloud.com/#/c/30628/1" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/30628/1</a><br/> Could you land this?</p> <p>Jay</p> <p>That is being tracked under <a href="https://jira.whamcloud.com/browse/LU-10422" title="layout LFSCK try to fix consistent owner unexpectedly" class="issue-link" data-issue-key="LU-10422"><del>LU-10422</del></a> and it will land as soon as the reviews have completed</p> <p>Peter</p> <p>AFAICT this is now resolved with the <a href="https://jira.whamcloud.com/browse/LU-10422" title="layout LFSCK try to fix consistent owner unexpectedly" class="issue-link" data-issue-key="LU-10422"><del>LU-10422</del></a> fix landed to b2_10</p> Duplicate LU-10422 LU-6420 Development Rank 1|hzzoxr: Rank (Obsolete) 9223372036854775807 Severity