[LU-14979] LNet: add tunable parameter to control max recovery interval duration

• Status: Resolved
• Project: Lustre
• Fix Version/s: Lustre 2.16.0
• Type: Improvement
• Priority: Minor
• Reporter: Serguei Smirnov
• Assignee: Cyril Bordage
• Resolution: Fixed
• Labels: lnet, lnet-health

Description

Currently implemented recovery ping mechanism increases the next scheduled recovery ping attempt timeout exponentially (base 2) and limits the timeout at 900 seconds. This hard-coded value appears to be too high in many cases. Introduce a tunable parameter that can be used to limit the recovery ping timeout and come up with a reasonable default.

Comments

[Chris Horn]

Can you add some detail about the cases where the value is too high? My hope was that resetting the interval when we received a message from an NI would be sufficient. Is that not working for some reason?

[Serguei Smirnov]

Chris,

I set up a test for LU-14978: Node A with one NI, Node B with two NIs, all on the same net. I was using lnetctl ping to create traffic from A to B. Then I executed "ifdown" on the interface corresponding to one of the B's NIs. (This was to simulate a hardware failure on node B.) Some lnetctl pings failed and the "failed" peer NI's health got decremented as seen by A. I left it alone for a few minutes, then brought the "failed" interface on B back up. A didn't realize that B had both NIs healthy until it got around to sending the next recovery ping. In my opinion, the delay was too long. Unless I initiate a ping from B to A and it uses the recovered interface, A has no idea that B has both NIs back until 900 second timeout expires.

[Chris Horn]

Okay, that makes sense and is working like I would expect. It might be interesting to see whether this is an issue in an environment where Node A is a Lustre server and B is a Lustre client (and vice versa) and there is actual i/o going on (or maybe even just idle client traffic). I think if there was i/o going on then things might recover more quickly, but the idle client case might also take a while to recover the NI (but, if the client is idle maybe it doesn't really matter. Once I/O was started we may again recover quickly).

[Gerrit Updater]

"Cyril Bordage <[e-mail]>" uploaded a new patch: https://review.whamcloud.com/44927
Subject: LU-14979 lnet: set max recovery interval duration
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: f3bea849cd255b6bcd8c379904795e3d8d6ffde8

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/44927/
Subject: LU-14979 lnet: set max recovery interval duration
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 4027395fe463b6ea11084ff2af43ba0732ad0ddb

[Peter Jones]

Landed for 2.16