https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16002 Lustre <p>Ping evictor adds 3 ping interval to eviction time(6*ping interval) PING_EVICT_TIMEOUT. For obd_timeout 300 the result eviction time became 670 instead of 450. It confuses and delays all conflicting requests on server side.</p> LU-16002

Ping evictor delayed client eviction for 3 ping interval more than defined

Bug Major Reopened Unresolved Alexander Boyko Alexander Boyko patch Sun, 10 Jul 2022 10:47:00 +0000 Wed, 13 Sep 2023 06:11:24 +0000 Lustre 2.16.0 Lustre 2.16.0 0 6 <p>"Alexander Boyko &lt;alexander.boyko@hpe.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/47928" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47928</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16002" title="Ping evictor delayed client eviction for 3 ping interval more than defined" class="issue-link" data-issue-key="LU-16002">LU-16002</a> ptlrpc: reduce pinger eviction time<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 747f6d6f7dfad19a9340275e905e79152978cf35</p> <p>"Alexander Boyko &lt;alexander.boyko@hpe.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/47982" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47982</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16002" title="Ping evictor delayed client eviction for 3 ping interval more than defined" class="issue-link" data-issue-key="LU-16002">LU-16002</a> ptlrpc: adds configurable ping interval<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: f53e72d9b8ad713f5bb509d6e3ff3765eef8f587</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/47982/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/47982/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16002" title="Ping evictor delayed client eviction for 3 ping interval more than defined" class="issue-link" data-issue-key="LU-16002">LU-16002</a> ptlrpc: adds configurable ping interval<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8e66f061c01e53cda84ce80af3860f488e927210</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/47928/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/47928/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16002" title="Ping evictor delayed client eviction for 3 ping interval more than defined" class="issue-link" data-issue-key="LU-16002">LU-16002</a> ptlrpc: reduce pinger eviction time<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 6bdeda7afe92d61db56367875774fa074aaac0fd</p> <p>Landed for 2.16</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=aboyko" class="user-hover" rel="aboyko">aboyko</a>, could you please provide some more background on why a tunable <tt>ping_interval</tt> is needed? I'm concerned that allowing <tt>ping_interval</tt> to be tuned separately from <tt>obd_timeout</tt> can lead to random client eviction when clients are not sending RPCs or <tt>OBD_PING</tt> in a timely manner. This might be hard to notice if it works out like e.g. <tt>ping_interval = obd_timeout - 10</tt> and this is OK while an import is active and sending RPCs or <tt>OBD_PING</tt>, but fails intermittently if the import becomes idle and a ping is also lost.</p> <p>I'd much prefer to have a per-device <tt>obd_timeout</tt> value as implemented in patch <a href="https://review.whamcloud.com/50519" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/50519</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-9912" title="fix multiple client mounts with different server timeouts" class="issue-link" data-issue-key="LU-9912">LU-9912</a> ptlrpc: make obd timeout a per-device param</tt>", and then the <tt>ping_interval</tt> for each import is controlled by <tt>obd-&gt;obd_timeout / 4</tt>. This would work properly for clients that mount multiple filesystems, unlike having a global <tt>obd_timeout</tt> (and now global <tt>ping_interval</tt>).</p> <p>However, before we change anything with the global <tt>ping_interval</tt> that was aadded in 2.16, I'd like to understand why it was added and what problem it was solving. I'd prefer to avoid having a tunable <tt>ping_interval</tt> entirely, just because it can go badly. If this was needed to solve some specific problem, would a per-device <tt>obd_timeout</tt> also solve this same issue? Also, <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=hornc" class="user-hover" rel="hornc">hornc</a> landed patch <a href="https://review.whamcloud.com/49807" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/49807</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-16483" title="Loss of idle ping causes reconnect even if subsequent ping succeeds" class="issue-link" data-issue-key="LU-16483"><del>LU-16483</del></a> ptlrpc: Track highest reply XID</tt>" that also solves a longstanding problem where clients reconnect on a ping failure, even though they have successfully sent other pings in the meantime.</p> <p>I'm thinking we should remove the global <tt>ping_interval</tt> tunable completely (so that pings are always tied to <tt>obd_timeout</tt>), and use something like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> #define PING_INTERVAL(obd) (obd_timeout(obd) / 4) </pre> </div></div> <p>It would still be possible to keep <tt>evict_multiplier</tt> if that is important, something like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> #define PING_INTERVAL(obd) (obd_timeout(obd) * 3 / (evict_multiplier * 2)) </pre> </div></div> <p>but before we add complexity I'd like to understand what this was needed for.</p> <p>We had an issue there cascading failures bring timeouts to ~1700s, blocking callback timeout. Something like - one client node with LDLM lock crashed, server waited it, increased AT. Crash and eviction was not a problem to a whole system but it highly increased AT and shared lock for a root directory. We detected 3-6 problems during it, bl timeouts, eviction logic, etc.  The one way to prevent such case is to detect crashed client early and evict it by pinger_evictor, we can reduce ping_interval and evictor multiplier for this. By default eviction time is 6 ping interval, server would not evict client if 5 pings are lost. For a perfect network it is overhead, and could be reduced (eviction multiplier). Similar things relate to a ping_interval. If obd_timeout is 300s(it is used in real), ping interval is 75s. To detect client fail faster ping interval should be reduced.</p> <p>I've made a comment about obd_timeout at LU. From my point of view obd_timeout is especially recovery timeout, but recovery and pinger don't have any relations. Only some historical.</p> <p>I agree that it is useful in such cases to be able to tune <tt>ping_interval</tt> and/or <tt>evict_multiplier</tt>, but it would make sense to ensure that <tt>ping_interval &lt; obd_timeout / 2</tt> and <tt>evict_multiplier &gt;= 2</tt> so that it cannot be set to a value where the client will be evicted easily. Even so, it still makes sense to use <tt>obd_timeout(exp)</tt> directly by default, unless <tt>ping_interval</tt> is explicitly set:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> #define PING_INTERVAL(obd) (ping_interval ?: (obd_timeout(obd) * 3 / (evict_multiplier * 2))) </pre> </div></div> <p>Also, for future reference, it is possible to evict specific clients from the MDS with "<tt>lctl set_param mdt.&#42;.evict_client=UUID</tt>" or "<tt>lctl set_param mdt.&#42;.evict_client=nid:NID</tt>". It will evict the client UUID/NID from all the targets if "<tt>mdt.&#42;.evict_tgt_nids</tt>" is set.</p> Related LU-9912 LU-16271 Development Rank 1|i02u7j: Rank (Obsolete) 9223372036854775807 Severity