https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12292 Lustre <p>Health value is used to determine route.<br/> In case of device failure, the value is periodically decremented by recovery process and<br/> after restration of the device, the value is incremented periodically.<br/> But, normal route is not selected until the value will be restored.<br/> We think the value should not be decremented by recovery process,<br/> because it takes time to be restored to its original value.</p> <p>We stopped health value decrement at recovery processing after a device failure is detected </p> <p> </p> LU-12292

Decrement Health Value even if recovery processing fault

Bug Minor Resolved Fixed Tatsushi Takamura Tatsushi Takamura LTS12 Mon, 13 May 2019 09:44:45 +0000 Fri, 16 Oct 2020 06:21:54 +0000 Tue, 31 Mar 2020 11:38:36 +0000 Lustre 2.12.1 Lustre 2.14.0 0 8 <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=simmonsja" class="user-hover" rel="simmonsja">simmonsja</a> had a good suggestion in this case. We should increment the health value by the same amount we decrement it with. In this way the interface recovers faster. For example if you want to fail the interface on the first failure and recover it on the first success you can set the health value to 1000. This will basically bring down the interface on the first failure and once we get one successful recovery the interface will be back to the full health. </p> <p>thoughts?</p> <p>I like that idea.</p> <p>I think Andreas had a valid point in <a href="https://jira.whamcloud.com/browse/LU-12303" title="Use lnet_health_sensitivity for restoring health for each lnet_recovery_internal" class="issue-link" data-issue-key="LU-12303"><del>LU-12303</del></a> where this behavior could cause issues with flapping hardware. Could a better approach be a more weighted recovery where it would start with the 1 per success but increase the health recovery amount on consecutive pings? This could allow faster recovery after the hardware issue is resolved, but not simply trust it is resolved right away.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=curtispb" class="user-hover" rel="curtispb">curtispb</a>, yes that sounds like a good idea. Do you have suggestions on the ratio it should increase by on consecutive successes?</p> <p>My first pass at this would be an exponential growth pattern such as health value + ((1 + consecutive successes)^2) with a max bound of 1000 which would recover in ~15 consecutive successes if the health value hit the floor. If there is a failure during this recovery period decrement normally and the consecutive counter is reset. Thoughts?</p> <p>Recent IB is stable and high quality, so we thought it is enough for us to stop health value decrement at recovery processing (1000sec  is too much).<br/> I think your idea is good, because it can handle flapping hardware and recovers in 15 seconds normal case.</p> <p>Amir Shehata (ashehata@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/36921" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36921</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12292" title="Decrement Health Value even if recovery processing fault" class="issue-link" data-issue-key="LU-12292"><del>LU-12292</del></a> lnet: keep health even if recovery failed<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f29e84d3ea2f7a16c98619489b63b42571774003</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/36921/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36921/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12292" title="Decrement Health Value even if recovery processing fault" class="issue-link" data-issue-key="LU-12292"><del>LU-12292</del></a> lnet: keep health even if recovery failed<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 610a7542107d5a8ab0a12dc8bda7a4f44f9f0b60</p> <p>Landed for 2.14</p> Related LU-12303 Development Rank 1|i00g5z: Rank (Obsolete) 9223372036854775807 Severity