[LU-16451] kfilnd: Enhance TN state machine to handle peer in "failed" state

• Type: Improvement
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.16.0
• Assignee: Chris Horn
• Reporter: Chris Horn

Description

If a send request (immediate or bulk) fails with the EHOSTUNREACH error number means the cxi retry handler failed to send a message indicating an issue with the peer or the fabric.

If a kfilnd transaction (TN) fails with the EHOSTUNREACH error number, update the peer to a new "failed" state.

When a peer is in this failed state, require a completed HELLO before sending any more packets to that peer.

The idea is to minimize the amount of outstanding messages (which consume cxi resources) until either the peer recovers or the timeouts expire.