https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14979 Lustre <p>Currently implemented recovery ping mechanism increases the next scheduled recovery ping attempt timeout exponentially (base 2) and limits the timeout at 900 seconds. This hard-coded value appears to be too high in many cases. Introduce a tunable parameter that can be used to limit the recovery ping timeout and come up with a reasonable default.</p> LU-14979

LNet: add tunable parameter to control max recovery interval duration

Improvement Minor Resolved Fixed Cyril Bordage Serguei Smirnov lnet lnet-health Wed, 1 Sep 2021 20:02:16 +0000 Sat, 11 Jun 2022 15:30:22 +0000 Sat, 11 Jun 2022 15:30:22 +0000 Lustre 2.16.0 0 4 <p>Can you add some detail about the cases where the value is too high? My hope was that resetting the interval when we received a message from an NI would be sufficient. Is that not working for some reason?</p> <p>Chris,</p> <p>I set up a test for <a href="https://jira.whamcloud.com/browse/LU-14978" title="LNet: balance peer NI selection if peer NI is added late" class="issue-link" data-issue-key="LU-14978"><del>LU-14978</del></a>: Node A with one NI, Node B with two NIs, all on the same net. I was using lnetctl ping to create traffic from A to B. Then I executed "ifdown" on the interface corresponding to one of the B's NIs. (This was to simulate a hardware failure on node B.) Some lnetctl pings failed and the "failed" peer NI's health got decremented as seen by A. I left it alone for a few minutes, then brought the "failed" interface on B back up. A didn't realize that B had both NIs healthy until it got around to sending the next recovery ping. In my opinion, the delay was too long. Unless I initiate a ping from B to A and it uses the recovered interface, A has no idea that B has both NIs back until 900 second timeout expires.</p> <p>Okay, that makes sense and is working like I would expect. It might be interesting to see whether this is an issue in an environment where Node A is a Lustre server and B is a Lustre client (and vice versa) and there is actual i/o going on (or maybe even just idle client traffic). I think if there was i/o going on then things might recover more quickly, but the idle client case might also take a while to recover the NI (but, if the client is idle maybe it doesn't really matter. Once I/O was started we may again recover quickly).</p> <p>"Cyril Bordage &lt;cbordage@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/44927" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44927</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14979" title="LNet: add tunable parameter to control max recovery interval duration" class="issue-link" data-issue-key="LU-14979"><del>LU-14979</del></a> lnet: set max recovery interval duration<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f3bea849cd255b6bcd8c379904795e3d8d6ffde8</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/44927/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44927/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14979" title="LNet: add tunable parameter to control max recovery interval duration" class="issue-link" data-issue-key="LU-14979"><del>LU-14979</del></a> lnet: set max recovery interval duration<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 4027395fe463b6ea11084ff2af43ba0732ad0ddb</p> <p>Landed for 2.16</p> Development Rank 1|i023dj: Rank (Obsolete) 9223372036854775807