https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12064 Lustre <p>The adaptive timeout code currently works on a granularity of full seconds, and ignores timeouts of "0".  This means the MDS adaptive timeout code doesn't really adjust the timeouts there.</p> <p>This means, for example, the bl_ast timeout stays at the default value of 100 seconds * 1.5 (ldlm_bl_timeout), so, 150 seconds.</p> <p>This is a <b>very</b> long time to wait, and the AT code is supposed to shorten this.</p> <p>There are two obvious approaches here.</p> <ol> <li>Stop ignoring "0" values in the adaptive timeout code, and set a default non-zero at_min (setting it to 1 second should mean no behavioral change, as that's the current minimum real value).  This solution should be simple and shouldn't affect existing installs too much.  (configuring at_min is pretty common anyway)</li> <li>Update the adaptive timeout code to use more precise time intervals than 1 second.</li> </ol> <p> </p> <p>I'm inclined to #1.  But in real configs, at_min is generally recommended to be something like 40 seconds.  So perhaps we should default to that instead.</p> <p> </p> <p>Note specifically in the ldlm_bl_timeout we use the max() of this and ldlm_enqueue_min (default is OBD_TIMEOUT_DEFAULT, 100 seconds), so we'll only get down to that value there.</p> <p> </p> <p>A few open questions here.</p> LU-12064

Adaptive timeout at_min adjustment & granularity

Improvement Major Reopened Unresolved WC Triage Patrick Farrell Tue, 12 Mar 2019 21:23:20 +0000 Wed, 7 Feb 2024 22:28:46 +0000 0 10 <p>It seems patch <a href="http://review.whamcloud.com/9336" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/9336</a> "<a href="https://jira.whamcloud.com/browse/LU-4942" title="lock callback timeout is not per-export" class="issue-link" data-issue-key="LU-4942"><del>LU-4942</del></a> at: per-export lock callback timeout" changed <tt>prolog_timeout()</tt> significantly:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> - <span class="code-keyword">return</span> max(at_est2timeout(at_get(&amp;svcpt-&gt;scp_at_estimate)), ldlm_timeout); + /* We are in the middle of the process - BL AST is sent, CANCEL + * is ahead. Take half of AT + IO process time. / + <span class="code-keyword">return</span> at_est2timeout(at_get(&amp;svcpt-&gt;scp_at_estimate)) + + (ldlm_bl_timeout(lock) &gt;&gt; 1); </pre> </div></div> <p>which I'm not sure I agree with. Definitely if a client is responsive and sending IO it should be allowed to complete, but there should be a shorter timeout for the initial AST if the client is not responsive.</p> <p>It looks like there was an incremental smearing of logic over several patches. Initially, <tt>ldlm&#95;get&#95;rq&#95;timeout()</tt> returned <tt>min(ldlm&#95;timeout, obd&#95;timeout / 3)</tt>, which seems reasonable - we want to allow a timeout and a retry before evicting a client. then AT came in and disabled this if AT_OFF, in favor of <tt>ptlrpc&#95;at&#95;set&#95;req&#95;timeout()</tt> set when the request is allocated, which uses only <tt>obd_timeout</tt> internally. We currently don't use <tt>ldlm&#95;timeout</tt> anywhere in the code when AT is enabled.</p> <p>The <tt>ldlm&#95;server&#95;blocking&#95;ast()</tt> code should not use <tt>ldlm&#95;bl&#95;timeout()</tt> for the initial BL AST reply timeout, since we don't know at this point if the client is responsive or not, but rather something like <tt>max(ldlm&#95;timeout, at&#95;est2timeout(at&#95;get(&amp;lock&#45;&gt;l&#95;export&#45;&gt;exp&#95;bl&#95;lock&#95;at)))</tt>. This is OK for later bulk IO timeouts when doing <tt>prolong&#95;timeout()</tt> after we know the client has replied to the initial blocking AST and is busy doing writes under the lock.</p> <p>In any case, Patrick I agree that setting at_min = 1 by default and allowing zero elapsed time replies makes sense and is relatively easy and low risk to implement. </p> <p>What else would be useful here is to tune <tt>at_min</tt> as a function of the number of clients connected to the servers. For systems with ~200 clients, having <tt>at_min=15</tt> is typical, and with ~1500 clients <tt>at_min=30</tt> is typical, so a function like the following seems reasonable:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> at_min = ilog(num_clients) * 3; </pre> </div></div> <p>though any explicitly-specified <tt>at_min</tt> value should take precedence. That probably means having a separate flag that indicates whether at_min has been explicitly set or not, and otherwise recalculating it when clients connect and disconnect.</p> <p>Presentation from CEA set <tt>at_min=55</tt> for a cluster with 20,000 clients, so this would also match approximately the <tt>ilog(num_clients) &#42; 3</tt> formula (<tt>16 &#42; 3 = 48</tt>).</p> <p>The main concern about at_min is that if you specify a value inferior to your lnet transaction timeout (modulo the number of lnet retry if it is setup...) the propabilty to flood your lnet networks increases drastically because an rpc not acknowledge by server will be retransmitted by client (not directly but through an high priority rpc) regarding the at_min value. I don't remember by heart the piece of code on client side but it is what we have observed. Anyway in lustre others timeouts like ldlm_timeout, obd_timeout are generally initialized in the code with constant value higher than default transaction timeout. On what we have seen on large scale we could set a default at_min value to <tt>lnet_transaction_timeout+1</tt> (for non routing configuration) or <tt>max((lnet_transaction_timeout+1),ilog(num_clients) * 3)</tt>. I think we have to rely on under layers. Does it make sense?</p> <p>I think one of the current issues is that <tt>at_min=0</tt> allows the ping flood to happen. If there is <tt>at_min &gt; 0</tt> it would significantly reduce the flood. Having a reasonable <tt>at_min</tt> for the cluster size will help significantly.</p> <p>With ever-increasing core counts on the client, it makes sense to scale the number of "clients" by <tt>max_mod_rpcs_in_flight</tt> when computing <tt>at_min</tt> so that a multi-threaded workload on a smaller number of clients is handled similarly to a larger number of clients with <tt>max_mod_rpcs_in_flight=1</tt>. With the default <tt>max_mod_rpcs_in_flight=8</tt> this would be a multiplier of 3 to the calculated <tt>at_min</tt> value.</p> <p>"Andreas Dilger &lt;adilger@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50609" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50609</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12064" title="Adaptive timeout at_min adjustment &amp; granularity" class="issue-link" data-issue-key="LU-12064">LU-12064</a> ptlrpc: set at_min=5 by default<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 1805e4dfa8fcba712b5fa3868f26988e7635dbcb</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50609/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50609/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-12064" title="Adaptive timeout at_min adjustment &amp; granularity" class="issue-link" data-issue-key="LU-12064">LU-12064</a> ptlrpc: set at_min=5 by default<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 46804c2230cc0f72d4472cddd5a37456e1f2fb00</p> <p>Landed for 2.16</p> <p>The patch that landed is only increasing at_min to a reasonable minimum value. The work to implement a dynamic at_min/at_max based on the number of connected clients has not been done.</p> <p>One proposal that might help here (and in other places) is for the servers to persistently track the maximum number of connected clients, so that the MDS/OSS knows after a restart how many clients might connect and can set at_min to an appropriate value right from the start.</p> Related LU-4942 LU-15246 LU-11989 LU-17514 Development Rank 1|i00d8v: Rank (Obsolete) 9223372036854775807