[LU-6420] layout LFSCK fixing dangling/unmatched_pair during soak tests unexpectedly

• Type: Bug
• Resolution: Duplicate
• Priority: Critical
• Fix Version/s: Lustre 2.8.0
• Severity: 3
• Assignee: nasf (Inactive)
• Reporter: nasf (Inactive)

Description

We have noticed that lfsck is sometimes fixing dangling/unmatched_pair on the soak cluster which is pretty unexpected:
repaired_dangling: 2108
repaired_unmatched_pair: 7527414
We are running metadata tests with striped directories and MDT failover. We are investigating the root cause ...

Issue Links

• duplicates: LU-10422 layout LFSCK try to fix consistent owner unexpectedly
• is duplicated by: LU-10349 debug and cleanup of corrupted PFID, unmatched MDT-object and OST-object pairs

Comments

[nasf (Inactive)]

We will resolve the unexpected inconsistent owner issue via LU-10422.

[Gerrit Updater]

Fan Yong ([e-mail]) uploaded a new patch: https://review.whamcloud.com/30612
Subject: LU-6420 lfsck: misc fixes to avoid unexpected repairing
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: bf0448c16e6d311b4773a80ec02ed9c48e8c11f4

[nasf (Inactive)]

We need more work to resolve the unexpected inconsistent owner issue

[Peter Jones]

Landed for 2.8

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch http://review.whamcloud.com/14332/
Subject: LU-6420 lfsck: misc patch to avoid unexpected repairing
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: cb7da2d4c0241f52dec5e05c57847ad6cd0dbb83

[nasf (Inactive)]

The unexpected "repaired_unmatched_pair" disappeared, we still more work about the unexpected "repaired_inconsistent_owner" issues.

[Cliff White (Inactive)]

We have tested now with the 14332 patch and all dependent patches, the errors continue.

2015-04-22 18:33:12,077:fsmgmt.fsmgmt:ERROR    lfsck found errors lola-8/soaked-MDT0000: repaired_multiple_referenced: 0
2015-04-22 18:33:12,077:fsmgmt.fsmgmt:ERROR    lfsck found errors lola-8/soaked-MDT0000: repaired_orphan: 0
2015-04-22 18:33:12,077:fsmgmt.fsmgmt:ERROR    lfsck found errors lola-8/soaked-MDT0000: repaired_inconsistent_owner: 2670573
2015-04-22 18:33:12,077:fsmgmt.fsmgmt:ERROR    lfsck found errors lola-8/soaked-MDT0000: repaired_others: 0
2015-04-22 18:33:27,360:fsmgmt.fsmgmt:ERROR    lfsck found errors lola-8/soaked-MDT0000: repaired_dangling: 0
2015-04-22 18:33:27,360:fsmgmt.fsmgmt:ERROR    lfsck found errors lola-8/soaked-MDT0000: repaired_unmatched_pair: 0
2015-04-22 18:33:27,360:fsmgmt.fsmgmt:ERROR    lfsck found errors lola-8/soaked-MDT0000: repaired_multiple_referenced: 0
2015-04-22 18:33:27,361:fsmgmt.fsmgmt:ERROR    lfsck found errors lola-8/soaked-MDT0000: repaired_orphan: 0
2015-04-22 18:33:27,361:fsmgmt.fsmgmt:ERROR    lfsck found errors lola-8/soaked-MDT0000: repaired_inconsistent_owner: 2670573
2015-04-22 18:33:27,361:fsmgmt.fsmgmt:ERROR    lfsck found errors lola-8/soaked-MDT0000: repaired_others: 0
2015-04-22 18:33:42,649:fsmgmt.fsmgmt:ERROR    lfsck found errors lola-8/soaked-MDT0000: repaired_dangling: 0
2015-04-22 18:33:42,649:fsmgmt.fsmgmt:ERROR    lfsck found errors lola-8/soaked-MDT0000: repaired_unmatched_pair: 0
2015-04-22 18:33:42,649:fsmgmt.fsmgmt:ERROR    lfsck found errors lola-8/soaked-MDT0000: repaired_multiple_referenced: 0
2015-04-22 18:33:42,650:fsmgmt.fsmgmt:ERROR    lfsck found errors lola-8/soaked-MDT0000: repaired_orphan: 0
2015-04-22 18:33:42,650:fsmgmt.fsmgmt:ERROR    lfsck found errors lola-8/soaked-MDT0000: repaired_inconsistent_owner: 2670573
2015-04-22 18:33:42,650:fsmgmt.fsmgmt:ERROR    lfsck found errors lola-8/soaked-MDT0000: repaired_others: 0
2015-04-22 18:33:57,933:fsmgmt.fsmgmt:ERROR    lfsck found errors lola-8/soaked-MDT0000: repaired_dangling: 0
2015-04-22 18:33:57,934:fsmgmt.fsmgmt:ERROR    lfsck found errors lola-8/soaked-MDT0000: repaired_unmatched_pair: 0
2015-04-22 18:33:57,934:fsmgmt.fsmgmt:ERROR    lfsck found errors lola-8/soaked-MDT0000: repaired_multiple_referenced: 0
2015-04-22 18:33:57,934:fsmgmt.fsmgmt:ERROR    lfsck found errors lola-8/soaked-MDT0000: repaired_orphan: 0
2015-04-22 18:33:57,934:fsmgmt.fsmgmt:ERROR    lfsck found errors lola-8/soaked-MDT0000: repaired_inconsistent_owner: 2670573
2015-04-22 18:33:57,935:fsmgmt.fsmgmt:ERROR    lfsck found errors lola-8/soaked-MDT0000: repaired_others: 0
2015-04-22 18:34:13,220:fsmgmt.fsmgmt:ERROR    lfsck found errors lola-8/soaked-MDT0000: repaired_dangling: 0
2015-04-22 18:34:13,220:fsmgmt.fsmgmt:ERROR    lfsck found errors lola-8/soaked-MDT0000: repaired_unmatched_pair: 0
2015-04-22 18:34:13,221:fsmgmt.fsmgmt:ERROR    lfsck found errors lola-8/soaked-MDT0000: repaired_multiple_referenced: 0
2015-04-22 18:34:13,221:fsmgmt.fsmgmt:ERROR    lfsck found errors lola-8/soaked-MDT0000: repaired_orphan: 0
2015-04-22 18:34:13,221:fsmgmt.fsmgmt:ERROR    lfsck found errors lola-8/soaked-MDT0000: repaired_inconsistent_owner: 2670573

[nasf (Inactive)]

Have you tried the single patch http://review.whamcloud.com/#/c/14332/ or all its dependent patches? We need the later case.

[Cliff White (Inactive)]

This patch does not apply cleanly to current master, we have been unable to test it - can we have an updated version, please?

[nasf (Inactive)]

Here is the patch:
http://review.whamcloud.com/#/c/14332/