[LU-19529] handle RAFT storage failures

• Type: Task
• Resolution: Unresolved
• Priority: Medium
• Severity: 3
• Epic Link: Fault Tolerant MGS
• Assignee: WC Triage
• Reporter: Lai Siyao

Description

Lustre RAFT implementation needs to handle RAFT storage failures: RAFT log write, 'vote'/'term' update and log application. The RAFT node that fails to write RAFT data should be removed from the RAFT cluster, and a non-voting RAFT node should be promoted to voting mode for system health.

One thing to consider: should this change be permanent? Since the RAFT nodes are chosen manually, and voting mode is also set manually, how to resolve the conflict between the change and setting?