https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5183 Lustre <p>Hi </p> <p>I'd like an explanation of which timeout values are being exceeded that are resulting in these evictions, so what does that "227 seconds" reffers to, like which timeout it's considering. Is that "ldlm_timeout, obd_timeout, /proc/sys/lustre/timeout, at_min or at_max. </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>May 14 05:37:59 dc2oss15 kernel: : Lustre: dc2-OST009c: haven't heard from client ac9ef944-83f6-a453-c821-f0067101d2ca (at 149.165.229.28@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bb8ff2800, cur 1400060279 expire 1400060129 last 1400060052 May 14 05:37:59 dc2oss15 kernel: : Lustre: Skipped 9 previous similar messages May 14 05:38:02 dc2oss12 kernel: : Lustre: dc2-OST007b: haven't heard from client ac9ef944-83f6-a453-c821-f0067101d2ca (at 149.165.229.28@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88169eecb400, cur 1400060282 expire 1400060132 last 1400060055 May 14 05:38:02 dc2oss12 kernel: : Lustre: Skipped 8 previous similar messages May 14 05:37:53 dc2oss04 kernel: : Lustre: dc2-OST0021: haven't heard from client ac9ef944-83f6-a453-c821-f0067101d2ca (at 149.165.229.28@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bd683c400, cur 1400060273 expire 1400060123 last 1400060046 May 14 05:37:53 dc2oss04 kernel: : Lustre: Skipped 8 previous similar messages May 14 05:37:58 dc2oss05 kernel: : Lustre: dc2-OST002c: haven't heard from client ac9ef944-83f6-a453-c821-f0067101d2ca (at 149.165.229.28@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88154cc64000, cur 1400060278 expire 1400060128 last 1400060051 May 14 05:37:58 dc2oss05 kernel: : Lustre: Skipped 9 previous similar messages </pre> </div></div> <p> Particularly, I'm interested in knowing whether ldlm_timeouts, which is 20s for OSTs and 6s for MDT, are in play given that we've adaptive timeouts enabled(at_max = 600) and /proc/sys/lustre/timeout=100.</p> <p>Should we consider increasing the ldlm_timeouts if they are in fact being used? Should we consider setting at_min to 60-70s to allow time for slow client responses?</p> <p>If yes then how does that settings helps and makes difference.</p> <p>See sections 2.2.2 and 2.2.8 in Cory Spitz's paper here:<br/> <a href="https://cug.org/5-publications/proceedings_attendee_lists/CUG11CD/page" class="external-link" target="_blank" rel="nofollow noopener">https://cug.org/5-publications/proceedings_attendee_lists/CUG11CD/page</a><br/> s/1-program/final_program/Wednesday/12A-Spitz-Paper.pdf</p> <p>Thank You,<br/> Manish</p> Lustre Server 2.1.6 <br/> Lustre Client 1.8.9 LU-5183

If Adaptive Timeout is set for at_max = 600 then id ldlm_timeouts gets affective or it becomes over ruled

Question/Request Minor Resolved Duplicate Emoly Liu Manish Patel Thu, 12 Jun 2014 15:53:45 +0000 Mon, 21 Jul 2014 13:48:40 +0000 Mon, 21 Jul 2014 13:48:40 +0000 0 3 <p>Emoly</p> <p>Could you please help on this one?</p> <p>Thanks</p> <p>Peter</p> <p>Hi Manish,</p> <p>The first question about "227 seconds" eviction is related to obd_timeout. The code is here:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> expire_time = cfs_time_current_sec() - PING_EVICT_TIMEOUT; ... <span class="code-keyword">if</span> (expire_time &gt; exp-&gt;exp_last_request_time) { class_export_get(exp); cfs_spin_unlock(&amp;obd-&gt;obd_dev_lock); LCONSOLE_WARN(<span class="code-quote">"%s: haven't heard from client %s"</span> <span class="code-quote">" (at %s) in %ld seconds. I think"</span> <span class="code-quote">" it's dead, and I am evicting"</span> <span class="code-quote">" it. exp %p, cur %ld expire %ld"</span> <span class="code-quote">" last %ld\n"</span>, obd-&gt;obd_name, obd_uuid2str(&amp;exp-&gt;exp_client_uuid), obd_export_nid2str(exp), (<span class="code-object">long</span>)(cfs_time_current_sec() - exp-&gt;exp_last_request_time), exp, (<span class="code-object">long</span>)cfs_time_current_sec(), (<span class="code-object">long</span>)expire_time, (<span class="code-object">long</span>)exp-&gt;exp_last_request_time); </pre> </div></div> <p>and</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">#define PING_INTERVAL max(obd_timeout / 4, 1U) /* Client may skip 1 ping; we must wait at least 2.5. But <span class="code-keyword">for</span> multiple * failover targets the client only pings one server at a time, and pings * can be lost on a loaded network. Since eviction has serious consequences, * and there<span class="code-quote">'s no urgent need to evict a client just because it'</span>s idle, we * should be very conservative here. */ #define PING_EVICT_TIMEOUT (PING_INTERVAL * 6) </pre> </div></div> <p>From the log above we can see that PING_EVICT_TIMEOUT is 150 seconds, and the time difference between the last request sent by client and the OST ping eviction check is 227 seconds.<br/> For example,</p> <blockquote> <p>May 14 05:37:59 dc2oss15 kernel: : Lustre: dc2-OST009c: haven't heard from client ac9ef944-83f6-a453-c821-f0067101d2ca (at 149.165.229.28@tcp) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880bb8ff2800, cur 1400060279 expire 1400060129 last 1400060052</p></blockquote> <p>PING_EVICT_TIMEOUT = 1400060279 - 1400060129 = 150</p> <p>As for the other question about adaptive timeout and ldlm_timeout, I need to check the code and the document, and then give a reply.</p> <p>Hi Emoly,</p> <p>If that 227 is the related to "obd_timeout" then which setting need to be tweaked so that it can hold that limit till 300 seconds and what is the tweak setting options for increasing the PING_EVICT_TIMEOUT to 300 sec.</p> <p>About the second let me know if you have any new updates for the "ldlm_timeout and adaptive_timeout", by looking at the codes.</p> <p>Thank you,<br/> Manish</p> <p>Hi Manish,<br/> According to the following code, obd_timeout is /proc/sys/lustre/timeout. </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> { .ctl_name = OBD_TIMEOUT, .procname = <span class="code-quote">"timeout"</span>, .data = &amp;obd_timeout, .maxlen = sizeof(<span class="code-object">int</span>), .mode = 0644, .proc_handler = &amp;proc_set_timeout }, </pre> </div></div> <p>So you can increase the PING_EVICT_TIMEOUT by increasing /proc/fs/lustre/timeout should work.</p> <p>About the second question, ldlm_timeout is a static timeout, that is a server waits for a client to reply to an initial lock cancellation request, and it should be smaller than obd_timeout.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (ldlm_timeout &gt;= obd_timeout) ldlm_timeout = max(obd_timeout / 3, 1U); </pre> </div></div> <p>Somehow, ldlm_timeout is in play given that you have adaptive timeouts enabled.<br/> In ost code,</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">static</span> inline <span class="code-object">int</span> prolong_timeout(struct ptlrpc_request *req) { struct ptlrpc_service *svc = req-&gt;rq_rqbd-&gt;rqbd_service; <span class="code-keyword">if</span> (AT_OFF) <span class="code-keyword">return</span> obd_timeout / 2; <span class="code-keyword">return</span> max(at_est2timeout(at_get(&amp;svc-&gt;srv_at_estimate)), ldlm_timeout); } </pre> </div></div> <p>In mdt_init0() code,</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> /* Reduce the initial timeout on an MDS because it doesn't need such * a <span class="code-object">long</span> timeout as an OST does. Adaptive timeouts will adjust <span class="code-keyword">this</span> * value appropriately. */ <span class="code-keyword">if</span> (ldlm_timeout == LDLM_TIMEOUT_DEFAULT) ldlm_timeout = MDS_LDLM_TIMEOUT_DEFAULT; </pre> </div></div> <p>We can see in Cory's pdf, Cray configures both the minimum Adaptive Timeout, at_min, and the ldlm_timeout to 70 seconds to allow Lustre to “ride through” the re-route for the Gemini feature. And yes, you can set at_min to 60-70s to allow time for slow client responses.</p> <p>Emoly</p> <p>DDN have suggested that we include this material in the Lustre manual. Could you please create an LUDOC ticket to track that?</p> <p>Thanks</p> <p>Peter</p> <p><a href="https://jira.whamcloud.com/browse/LUDOC-250" title="More explanation of several kinds of timeout setting" class="issue-link" data-issue-key="LUDOC-250">LUDOC-250</a> is created to track the lustre manual update.</p> <p>Closing ticket as the remaining doc work will be handled under <a href="https://jira.whamcloud.com/browse/LUDOC-250" title="More explanation of several kinds of timeout setting" class="issue-link" data-issue-key="LUDOC-250">LUDOC-250</a></p> Related LUDOC-250 Development Rank 1|hzwoav: Rank (Obsolete) 14384