https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13569 Lustre <p>We should not recover interfaces, remote or local, indefinitely. A remote interface may never come back (remote peer rebooted with new LNet config). A local interface may be down for extended periods of time when, say, a NIC needs to be swapped or cable replaced.</p> <p>Let's define criteria for when we should stop trying to recover an interface.</p> <ul class="alternate" type="square"> <li>After X recovery attempts?</li> <li>After Y amount of time?</li> </ul> <p>We also need to decide how to stop doing recover.</p> <ul class="alternate" type="square"> <li>Flag the ni/lpni so that it doesn't get placed back onto the recovery queue?</li> <li>Delete the ni/lpni outright?</li> <li>Rather than stop completely, use an exponential backoff algorithm so that recovery doesn't add any real load to the system.</li> </ul> LU-13569

LNet Health should not recover interfaces indefinitely

Improvement Minor Resolved Fixed Chris Horn Chris Horn Fri, 15 May 2020 20:28:43 +0000 Fri, 27 Jan 2023 17:41:44 +0000 Mon, 14 Jun 2021 19:34:45 +0000 Lustre 2.15.0 0 4 <p>What is the expected timeframe for giving up on the interface? I can definitely see that interfaces might be down for many hours/days because of bad cables/switches/etc. Typically what we do in such cases is have exponential backoff of the retry so that they do not add any real load to the system (on the order of one message every 5-10 minutes.</p> <p>The alternative would be to disable local-side recovery and wait until the peer starts using the interface to send messages to this node again. The drawback here would be if e.g. a switch goes down for an hour and the nodes all stop using their interfaces and never restart.</p> <p>What I'm thinking is that:<br/> 1. We can put a simple hard limit on remote NI recovery. Try to recover remote NIs, perhaps with exponential backoff, until the hard limit is reached. When the hard limit is reached we no longer attempt to recover the NI until we receive a message from that NI. At that point the algorithm resets and the remote NI becomes eligible for recovery again.<br/> 1.1 We might be able to leverage the existing lpni_last_alive field of the lnet_peer_ni struct for this purpose.<br/> 2. We recover local NIs indefinitely, but with an exponential backoff. This ensures that when a local NI becomes healthy again then we'll eventually use it for a send and this will cause other peers to mark it back up.<br/> 3. Maybe successful recovery messages cause us to send subsequent ones without delay. Failed recovery ping -&gt; increases recovery interval; successful recovery ping -&gt; decreases recovery interval.</p> <p>Chris Horn (chris.horn@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/39716" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39716</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13569" title="LNet Health should not recover interfaces indefinitely" class="issue-link" data-issue-key="LU-13569"><del>LU-13569</del></a> lnet: Introduce lnet_recovery_limit parameter<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: b8b728355cbaead6cba862f5cec9703e77bfced7</p> <p>Chris Horn (chris.horn@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/39717" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39717</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13569" title="LNet Health should not recover interfaces indefinitely" class="issue-link" data-issue-key="LU-13569"><del>LU-13569</del></a> lnet: Add lnet_recovery_limit to lnetctl<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 21ba92081131f86de4a6ee0bd1211a51b6f8ff32</p> <p>Chris Horn (chris.horn@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/39718" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39718</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13569" title="LNet Health should not recover interfaces indefinitely" class="issue-link" data-issue-key="LU-13569"><del>LU-13569</del></a> lnet: Age peer NI out of recovery<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 0aa87980f624ceb0e8c96553becb953a5c7b95cb</p> <p>Chris Horn (chris.horn@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/39719" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39719</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13569" title="LNet Health should not recover interfaces indefinitely" class="issue-link" data-issue-key="LU-13569"><del>LU-13569</del></a> lnet: Only recover known good peer NIs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 043ae30076f397cb1ac2f5a0bdd2c6ac4e171fd1</p> <p>Chris Horn (chris.horn@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/39720" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39720</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13569" title="LNet Health should not recover interfaces indefinitely" class="issue-link" data-issue-key="LU-13569"><del>LU-13569</del></a> lnet: Recover peer NI w/exponential backoff interval<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 35017495ca09ad86c6f414d1d3c1f7f61394ee65</p> <p>Chris Horn (chris.horn@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/39721" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39721</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13569" title="LNet Health should not recover interfaces indefinitely" class="issue-link" data-issue-key="LU-13569"><del>LU-13569</del></a> lnet: Recover local NI w/exponential backoff interval<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: e9ba241290326846f9796b9652d58e80b2796f92</p> <p>Chris Horn (chris.horn@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/39722" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39722</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13569" title="LNet Health should not recover interfaces indefinitely" class="issue-link" data-issue-key="LU-13569"><del>LU-13569</del></a> lnet: Deprecate lnet_recovery_interval<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f7e142362ad03d6c43af1dee6dd3fdd15fe26fc0</p> <p>Chris Horn (chris.horn@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/39723" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39723</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13569" title="LNet Health should not recover interfaces indefinitely" class="issue-link" data-issue-key="LU-13569"><del>LU-13569</del></a> tests: Check LNet Health recovery logic<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 2f0cf735e93c9defb7ac911289a65ff5cccd0f5e</p> <p>Chris Horn (chris.horn@hpe.com) uploaded a new patch: <a href="https://review.whamcloud.com/40314" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40314</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13569" title="LNet Health should not recover interfaces indefinitely" class="issue-link" data-issue-key="LU-13569"><del>LU-13569</del></a> lnet: Add health ping stats<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: bc7d9963c08ced81fca69f3f735e477583ba0bf4</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39716/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39716/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13569" title="LNet Health should not recover interfaces indefinitely" class="issue-link" data-issue-key="LU-13569"><del>LU-13569</del></a> lnet: Introduce lnet_recovery_limit parameter<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a2e61838f8de89e0f7c80c3bf288cbeb1b358baa</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39717/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39717/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13569" title="LNet Health should not recover interfaces indefinitely" class="issue-link" data-issue-key="LU-13569"><del>LU-13569</del></a> lnet: Add lnet_recovery_limit to lnetctl<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 3e5c6620fd0b0511498d14d38e8610d08f6da7b3</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39718/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39718/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13569" title="LNet Health should not recover interfaces indefinitely" class="issue-link" data-issue-key="LU-13569"><del>LU-13569</del></a> lnet: Age peer NI out of recovery<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: cc27201a76574b51dc3ffb37f039b3364cab386d</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39719/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39719/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13569" title="LNet Health should not recover interfaces indefinitely" class="issue-link" data-issue-key="LU-13569"><del>LU-13569</del></a> lnet: Only recover known good peer NIs<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 39a169cd02738a13866f3b88fbe3304dc20565d6</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39720/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39720/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13569" title="LNet Health should not recover interfaces indefinitely" class="issue-link" data-issue-key="LU-13569"><del>LU-13569</del></a> lnet: Recover peer NI w/exponential backoff interval<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 917553c537a8860f57a50dc9752e3ac69d06c11c</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39721/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39721/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13569" title="LNet Health should not recover interfaces indefinitely" class="issue-link" data-issue-key="LU-13569"><del>LU-13569</del></a> lnet: Recover local NI w/exponential backoff interval<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8fdf2bc62ac9c418bd8e326112da9a2835f09ccb</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39722/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39722/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13569" title="LNet Health should not recover interfaces indefinitely" class="issue-link" data-issue-key="LU-13569"><del>LU-13569</del></a> lnet: Deprecate lnet_recovery_interval<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 79ab0535622782c82636cee47918dc4b19983144</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/40314/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/40314/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13569" title="LNet Health should not recover interfaces indefinitely" class="issue-link" data-issue-key="LU-13569"><del>LU-13569</del></a> lnet: Add health ping stats<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 4c7e4aa57629660386ae2849151a0639b6177200</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/39723/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/39723/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13569" title="LNet Health should not recover interfaces indefinitely" class="issue-link" data-issue-key="LU-13569"><del>LU-13569</del></a> tests: Check LNet Health recovery logic<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: aa7391445519b46752b5b0adcbe5baa368750e70</p> <p>Landed for 2.15</p> Duplicate LU-13572 Development Epic Link LU-13422 Rank 1|i010j3: Rank (Obsolete) 9223372036854775807