https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14518 Lustre <p>If a request is not being processed in a timely manner, it will mark the service unhealthy, which can lead to STONITH. However, in some cases when the server is very heavily loaded, requests may take longer than <tt>at_max</tt> to be processed and this shouldn't cause the server to be killed, since that will slow down request processing even further and put extra load on the backup server(s), slowing down their processing and possibly causing them to fail in a similar manner.</p> LU-14518

allow slow request processing to be removed from health check

Improvement Minor Open Unresolved Andreas Dilger Andreas Dilger Fri, 12 Mar 2021 22:27:29 +0000 Tue, 23 Jan 2024 01:14:27 +0000 Lustre 2.14.0 0 9 <p>I am duplicating here my comment in review: we could consider history of timeouts, e.g. <tt>ost.OSS.ost_io.timeouts</tt> to decide about service health taking into account amount of timeouts per time unit</p> <p>Another note about that health checking based on services, it stops whole node if failed, so all severs will fail over which looks as overreaction in some cases, e.g. thread is stuck waiting for some event like another server recovery or so, or even worse if storage becomes overloaded for a moment in which case server movement to another node will not help, since storage remains the same, that would cause constant ping-pong failovers. The only case when that is helpful - deadlock situation or any other when node requires reboot. Detecting that by single thread timeout is too aggressive</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53225" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53225</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14518" title="allow slow request processing to be removed from health check" class="issue-link" data-issue-key="LU-14518">LU-14518</a> ptlrpc: WIP avoid server STONITH for slow requests<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 515100fb45dda4587c9d60b73a44258c7aba5bdc</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/53451" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/53451</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-14518" title="allow slow request processing to be removed from health check" class="issue-link" data-issue-key="LU-14518">LU-14518</a> libcfs: print CFS_FAIL_CHECK() location<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3f1231a9deb9cde5f2b81d8899d80359ff73dbe7</p> Related Development Rank 1|i01pdz: Rank (Obsolete) 9223372036854775807