https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13145 Lustre <p>On larger clusters it appears like the current transaction timeout default of 10 seconds is too short and it causes RDMA timeouts.</p> <p>The proposal is to increase the timeout to 150s. With a retry count of 3 that would bring the LND timeout to 50s, which was the initial value before health.</p> LU-13145

LNet Health: increase transaction timeout

Bug Minor Resolved Fixed Amir Shehata Amir Shehata Thu, 16 Jan 2020 00:56:32 +0000 Mon, 19 Dec 2022 19:38:02 +0000 Sat, 8 Feb 2020 05:14:19 +0000 Lustre 2.14.0 Lustre 2.12.4 0 4 <p><del>Amir Shehata (ashehata@whamcloud.com) uploaded a new patch:</del> <a href="https://review.whamcloud.com/37253" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37253</a><br/> <del>Subject: <a href="https://jira.whamcloud.com/browse/LU-13145" title="LNet Health: increase transaction timeout" class="issue-link" data-issue-key="LU-13145"><del>LU-13145</del></a> lnet: increase transaction timeout</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: master</del><br/> <del>Current Patch Set: 1</del><br/> <del>Commit: 07ef6f9f0c142240a4586674fe9c556482f323c4</del></p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37390" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37390</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13145" title="LNet Health: increase transaction timeout" class="issue-link" data-issue-key="LU-13145"><del>LU-13145</del></a> lnet: use conservative health timeouts<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: f308fc8fc393d939f34e4d7587c0848667084a8c</p> <p>Andreas Dilger (adilger@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/37430" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37430</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13145" title="LNet Health: increase transaction timeout" class="issue-link" data-issue-key="LU-13145"><del>LU-13145</del></a> lnet: use conservative health timeouts<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 188b4cb2f782e348cfa4225745341ac8e3e267c3</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37430/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37430/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13145" title="LNet Health: increase transaction timeout" class="issue-link" data-issue-key="LU-13145"><del>LU-13145</del></a> lnet: use conservative health timeouts<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 361e9eaef13c0f472ad45388d3e147dabc32b737</p> <p>Landed for 2.14</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37390/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37390/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-13145" title="LNet Health: increase transaction timeout" class="issue-link" data-issue-key="LU-13145"><del>LU-13145</del></a> lnet: use conservative health timeouts<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 3c86a9361862d34a4efde73c4f1cb7603ec6b2f9</p> <p>From <a href="https://jira.whamcloud.com/browse/LU-13020" title="ko2iblnd tuning" class="issue-link" data-issue-key="LU-13020"><del>LU-13020</del></a> the workaround to get equivalent behavior for systems without this patch is to run the following commands on all of the 2.12.3 nodes in the shown order:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>echo 150 &gt; /sys/module/lnet/parameters/lnet_transaction_timeout echo 2 &gt; /sys/module/lnet/parameters/lnet_retry_count </pre> </div></div> <p>This only temporarily changes these values, but they can be set permanently by adding the following line in <tt>/etc/modprobe.d/lnet.conf</tt>:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>options lnet lnet_retry_count=2 lnet_transaction_timeout=150 </pre> </div></div> Blocker LU-15288 Related LU-13020 Development Rank 1|i00s47: Rank (Obsolete) 9223372036854775807 Severity