https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1742 Lustre <p>Misleading error message from kiblnd_check_txs_locked(). The value reported in the error message is how many seconds we exceeded the deadline by. What I (and everyone else here) would have expected before reading the source is that the value would be the total time outstanding before timing out the RDMA.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>LNetError: 3073:0:(o2iblnd_cb.c:2988:kiblnd_check_txs_locked()) Timed out tx: active_txs, 10 seconds </pre> </div></div> LU-1742

Fix 'Timed out tx' error message

Bug Trivial Resolved Fixed Cyril Bordage Brian Behlendorf easy llnl Mon, 13 Aug 2012 12:58:03 +0000 Thu, 29 Oct 2020 12:04:10 +0000 Thu, 29 Oct 2020 12:04:10 +0000 Lustre 2.14.0 0 9 <p><a href="http://review.whamcloud.com/3622" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/3622</a></p> <p>Hi Brian, did you really intend to file this bug as a severity 1?</p> <p>Another problem with this error message is that it doesn't tell us how long the tx has been actually on the wire, e.g. the error message above told us a tx expired 60 seconds (10 + default ko2iblnd timeout) after it was queued BUT:</p> <ol> <li>It could have been sitting in the TX queue waiting for a credit for the most of the 60 seconds and just barely got onto the active queue, which indicates problems with flow control protocol. Or:</li> <li>It could have spent most of the 60 seconds on the wire without completion, which usually indicates problems in the IB fabric.</li> </ol> <p>It'd be very useful to be able to distinguish the two cases.</p> <p>Brian B, I have dropped the severity because I assume that LLNL is not down as a result of this issue. Please speak up if I am mistaken</p> <p>Isaac please can you take care of this one.</p> <p>Sorry, this was accidentally filed as high priority. The fix to update the error message is of course not critical.</p> <p>However, ORI-735 is a big deal for us since it's currently preventing us from running IOR on Sequoia. So we need to absolutely get to the bottom of why that is happening , we're just starting to investigate in the context of ORI-735. I'll probably update the patch based on Isaac's suggestion so we can get some more visibility in to actually what's going wrong.</p> <p>Sonia Sharma (sharmaso@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33235" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33235</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-1742" title="Fix &#39;Timed out tx&#39; error message" class="issue-link" data-issue-key="LU-1742"><del>LU-1742</del></a> o2iblnd: 'Timed out tx' error message<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 244d556e8784bbfdafcc13937ec73d53ef416b1a</p> <p><del>Oleg Drokin (green@whamcloud.com) merged in patch</del> <a href="https://review.whamcloud.com/33235/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33235/</a><br/> <del>Subject: <a href="https://jira.whamcloud.com/browse/LU-1742" title="Fix &#39;Timed out tx&#39; error message" class="issue-link" data-issue-key="LU-1742"><del>LU-1742</del></a> o2iblnd: 'Timed out tx' error message</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: master</del><br/> <del>Current Patch Set:</del><br/> <del>Commit: 7308662efc02fde077216f54728ecf278f31311b</del></p> <p>Patch has been reverted due to <a href="https://jira.whamcloud.com/browse/LU-13675" title="LNetError: 14769:0:(o2iblnd.h:1003:kiblnd_queue2str()) LBUG" class="issue-link" data-issue-key="LU-13675"><del>LU-13675</del></a>.</p> <p>Is this still a live issue for LLNL?</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/3622/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/3622/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-1742" title="Fix &#39;Timed out tx&#39; error message" class="issue-link" data-issue-key="LU-1742"><del>LU-1742</del></a> o2iblnd: 'Timed out tx' error message<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 2be289a2b1f12bddce06e9eee65b5581b0b9fb5d</p> <p>Landed for 2.14</p> Related LU-13675 Development End date Thu, 26 Jun 2014 12:58:03 +0000 Rank 1|hzvyef: Rank (Obsolete) 9757 Severity Start date Mon, 13 Aug 2012 12:58:03 +0000