[LU-10888] 'lctl abort_recovery' allow aborting recovery between MDTs

• Type: New Feature
• Resolution: Not a Bug
• Priority: Major
• Affects Version/s: Lustre 2.10.0, Lustre 2.11.0

Epic/Theme

DNE DNE2 dne

• Assignee: Hongchao Zhang
• Reporter: Lai Siyao

Description

'lctl abort_recovery' doesn't abort recovery between MDTs, because unlike abort recovery on single MDT system which only fail unfinished operations, this may break system consistency, so as a tradeoff, Lustre chose consistency over availability. But there are two major causes if recovery between MDTs doesn't finish, the first is network issue, for this type, we can wait indefinitely for network to recover, while the second is software bug, which is difficult for user to fix manually on backend filesystem.

Now lfsck is ready, which can fix inconsistency in the system. So we should provide an option to allow user to abort recovery between MDTs, and then fix inconsistencies.

Issue Links

• is related to: LU-11111 crash doing LFSCK: orph_index_insert()) ASSERTION( !(obj->mod_flags & ORPHAN_OBJ)
• is related to: LU-11419 lfsck does not complete phase2
• is related to: LU-12546 add option to abort recovery between MDTs but not between client/MDT

Comments

[Hongchao Zhang]

abort_recovery has been enabled between MDTS, the new requirement of aborting
the recovery between MDTs but not aborting between client/MDT will be implemented
in a new ticket

[Patrick Farrell (Inactive)]

I think it's also possible for this lack of MDS-MDS abort_recovery to cause hangs in certain situations.  During some testing at Cray, we had an MDS LBUG that happened during replay on MDS restart, so we tried abort_recovery.  That hung in some complex scenario related to cross MDT communication.  We weren't aware at the time that cross-MDT ops weren't handled by abort recovery, but it seems likely to be related.