https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13020 Lustre <p>We have been setting ko2iblnd timeout = 150 (default of 50) for our cluster. From reading the code this is no longer being used and instead lnet_lnd_timeout is used.</p> <p>For example in kiblnd_queue_tx_locked</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> timeout_ns = lnet_get_lnd_timeout() * NSEC_PER_SEC; tx-&gt;tx_queued = 1; tx-&gt;tx_deadline = ktime_add_ns(ktime_get(), timeout_ns); </pre> </div></div> <p>and <br/> lnet_get_lnd_timeout() returns the new default of 5. Does this mean we went from 150 to 5! </p> <p>In the documentation it says that lnet_lnd_timeout derived from lnet_transaction_timeout and retry_count. But that is not getting set for tx-&gt;tx_deadline.</p> <p>Am I reading the code correctly?</p> LU-13020

ko2iblnd tuning

Bug Critical Resolved Cannot Reproduce Amir Shehata Mahmoud Hanafi Tue, 26 Nov 2019 20:34:29 +0000 Wed, 6 Jan 2021 13:34:54 +0000 Wed, 6 Jan 2021 13:34:54 +0000 Lustre 2.12.2 0 8 <p>If you look at retry_count_set() and transaction_to_set() you'll see that it's setting the lnet_lnd_timeout based on the following calculation</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> retry_count_set() 482 »·······<span class="code-keyword">if</span> (value == 0) 483 »·······»·······lnet_lnd_timeout = lnet_transaction_timeout; 484 »·······<span class="code-keyword">else</span> 485 »·······»·······lnet_lnd_timeout = lnet_transaction_timeout / value; transaction_to_set() 436 »·······*transaction_to = value; 437 »·······<span class="code-keyword">if</span> (lnet_retry_count == 0) 438 »·······»·······lnet_lnd_timeout = value; 439 »·······<span class="code-keyword">else</span> 440 »·······»·······lnet_lnd_timeout = value / lnet_retry_count;</pre> </div></div> <p>I checked master and if you specify</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> options lnet lnet_transaction_timeout=150</pre> </div></div> <p>transaction_to_set() gets called:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> (api-ni.c:442:transaction_to_set()) lnet_lnd_timeout = 50 # it's 50 because retry_count=3 # however <span class="code-keyword">if</span> you <span class="code-keyword">do</span> <span class="code-keyword">this</span>: options lnet lnet_retry_count=0 options lnet lnet_transaction_timeout=150 # you should see: (api-ni.c:442:transaction_to_set()) lnet_lnd_timeout = 50 (api-ni.c:489:retry_count_set()) lnet_lnd_timeout = 150 # note the above is debug code I added.</pre> </div></div> <p>Is this not working for you?</p> <p>So </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> #define LNET_LND_DEFAULT_TIMEOUT 5 </pre> </div></div> <p>is not used? Because when lnet module loads it sets it to 50?</p> <p>Our cluster running 2.12.2 every where we still get client evictions. And enabling 'debug=+net' make the issue go away (ref: <a href="https://jira.whamcloud.com/browse/LU-11644" title="LNet: Service thread inactive for 300 causes client evictions " class="issue-link" data-issue-key="LU-11644">LU-11644</a>)</p> <p>The other issue, on our remote HDR clustre we are getting RDMA timeout to the lustre routers. Running lnet_selftest with-in compute nodes we get timeouts. This may be a fabric issue. But something is quite not right.... <br/> I'll run some tests changing lnet_transaction_timeout to see if I can see some difference.</p> <p>The ko2iblnd timeout should be removed and Docs should be updated. This would eliminate some of the confusion.</p> <p>did increasing the timeout resolve the client evictions issue (ref: <a href="https://jira.whamcloud.com/browse/LU-11644" title="LNet: Service thread inactive for 300 causes client evictions " class="issue-link" data-issue-key="LU-11644">LU-11644</a>)?</p> <p>I ended up setting </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> transaction_timeout: 200 at_min=275 </pre> </div></div> <p>Things look ok for now. I'll know more in a few days. <br/> I tried setting retry_count=2 with transaction_timeout=200 it causes a lot of timeouts. Worse than setting transaction_timeout=100 and retry_count=0.</p> <p>Would it be possible to get some net logging around the timeouts that occur with:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lnet_transaction_timeout=200 lnet_retry_count=2 </pre> </div></div> <p>It would be very useful for me to understand the cause of these timeouts, so I can fine tune the feature if required.</p> <p>The idea with this config, is that the LND timeout will be set to 100. We'll attempt 2 retries within that 200s window.</p> <p>I'd like to see if we do attempt the retry and for what reason and the implications of the retries.</p> <p>Amir Shehata (ashehata@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/36944" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36944</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13020" title="ko2iblnd tuning" class="issue-link" data-issue-key="LU-13020"><del>LU-13020</del></a> o2iblnd: timeout is now obsolete<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d6cbd02ff5d17b0cf53683e1c8db8364850ac4cc</p> <p>I will try to reproduce the issue with retry_count=2 on our test filesystem and gather logs.</p> <p>We can close this</p> Related LU-13145 Development Rank 1|i00q13: Rank (Obsolete) 9223372036854775807 Severity