[LU-10349] debug and cleanup of corrupted PFID, unmatched MDT-object and OST-object pairs

• Type: Bug
• Resolution: Duplicate
• Priority: Critical
• Affects Version/s: Lustre 2.7.0, Lustre 2.9.0
• Environment: servers: lustre-2.7.3-1nasS_mofed33v3g_2.6.32_642.15.1.el6.20170609.x86_64.lustre273.x86_64; clients: lustre-client-2.9.0-2.3nasC_mofed34v1_4.4.74_92.32.1.20170808_nasa.x86_64
• Severity: 3
• Assignee: nasf (Inactive)
• Reporter: Nathan Dauchy (Inactive)

Description

This ticket is created to handle NASA-specific debugging of corrupted PFIDs discussed in LU-10248, as well as ensure ports of a fix for lfsck handling of repaired_unmatched_pair, and any other related questions to ensure proper running of lfsck and cleanup of the filesystem.

servers: lustre-2.7.3-1nasS_mofed33v3g_2.6.32_642.15.1.el6.20170609.x86_64.lustre273.x86_64 (basically the old FE branch, plus several cherry-picked patches)

clients: lustre-client-2.9.0-2.3nasC_mofed34v1_4.4.74_92.32.1.20170808_nasa.x86_64

We hope to upgrade both to a 2.10.2-based build in the near future.

Issue Links

• duplicates: LU-10422 layout LFSCK try to fix consistent owner unexpectedly
• duplicates: LU-6420 layout LFSCK fixing dangling/unmatched_pair during soak tests unexpectedly

Comments

[Peter Jones]

AFAICT this is now resolved with the LU-10422 fix landed to b2_10

[Peter Jones]

Jay

That is being tracked under LU-10422 and it will land as soon as the reviews have completed

Peter

[Jay Lan (Inactive)]

The above patch #30612 for b2_10 is at
https://review.whamcloud.com/#/c/30628/1
Could you land this?

[nasf (Inactive)]

ndauchy,

The patch https://review.whamcloud.com/#/c/30612/ for fixing the issue of unexpected inconsistent owner has already been landed to master. And it has been ported to b2_7_fe branch via the patch https://review.whamcloud.com/30613. You can use related patch to resolve your system trouble. Please let me you what else you need.

[nasf (Inactive)]

The patch for repair unexpected inconsistent owner on b2_7_fe:
https://review.whamcloud.com/30613

[Nathan Dauchy (Inactive)]

So I would suggest to keep the system unchanged since it is without influence now.

OK, we will proceed with removing the additional OSTs from the file system, and wait for install of the patched lfsck on the 2.7.3 servers and perform the cleanup later.  Thanks!

[nasf (Inactive)]

Regardless, it will be a while before we can complete the server rebuild and take a downtime to apply it. In the meantime I need to move forward with removing additional OSTs from the file system to free up the hardware for spares. Do you recommend we run the current lfsck in non-dry-run mode, or just go ahead and remove the OST now and wait for the lfsck updates?

There are two kinds inconsistency reported by the layout LFSCK, one is inconsistent owner information that cause by some known layout LFSCK issue, can be ignored. Another is the unmatched MDT-object and OST-object pairs. Such inconsistency will NOT affect normal system access unless enabling I/O verification (disable by default) explicitly. So I would suggest to keep the system unchanged since it is without influence now.

Review #16135 is in 'Need Code-Review' state for more than 1 year. Is it OK to cherry-pick as it is, Fan?

Yes, I think so. Such patch has already been landed to b2_8_fe and master. It is not on b2_7_fe may because b2_7 was some old at that time and such issue is not very serious.

[Jay Lan (Inactive)]

We do not have either.

Review #16135 is in 'Need Code-Review' state for more than 1 year. Is it OK to cherry-pick as it is, Fan?