https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2187 Lustre <p>According to the analysis of <a href="https://jira.whamcloud.com/browse/LU-1717" title="mdt_recovery.c:611:mdt_steal_ack_locks()) Resent req xid XXX has mismatched opc: new 101 old 0" class="issue-link" data-issue-key="LU-1717"><del>LU-1717</del></a> we are frequently losing Lustre messages on Sequoia's IB network. We have no LNet routers, and IB is a reliable network. We are not seeing any timeouts or lnet errors that would suggest that we are seeing IB transmission problems.</p> <p>Why are messages being lost that can result in <a href="https://jira.whamcloud.com/browse/LU-1717" title="mdt_recovery.c:611:mdt_steal_ack_locks()) Resent req xid XXX has mismatched opc: new 101 old 0" class="issue-link" data-issue-key="LU-1717"><del>LU-1717</del></a> error messages? I'm worried that we're papering over a larger problem by silencing those errors.</p> LU-2187

Why are we losing messages?

Bug Minor Resolved Cannot Reproduce Doug Oucharek Christopher Morrone sequoia Mon, 15 Oct 2012 22:56:12 +0000 Thu, 13 Feb 2014 01:54:58 +0000 Thu, 13 Feb 2014 01:54:58 +0000 Lustre 2.4.0 0 9 <p>Question: Do we know if the Lustre messages are being "lost" or are just delayed to the point that they are considered lost by the Lustre code in question?</p> <p>The IB network is reliable, but can delay messages to the point we take action assuming the message is lost.</p> <p>I can't say definitively, but I strongly suspect that the messages aren't really lost, but this is more bad Lustre behaviour making poor AT assumptions or delaying send of messages or something.</p> <p>There's a few things to do to diagnose it:</p> <ul class="alternate" type="square"> <li>/proc/sys/lnet/* would be very useful. Please gather them once such errors happen again.</li> <li>"lctl --net o2ib0 conn_list (or list_conn)" gives useful data at o2iblnd layer. Please run it once such errors happen again.</li> <li>Many LNet/LND errors don't go to dmesg by default, please on a node where such errors have occurred, do a 'echo +neterror &gt; /proc/sys/lnet/printk' after each reboot.</li> <li>Please also check IB errors by running 'ibcheckerrors'.</li> </ul> <p>Isaac, I think it is very unlikely that we'll see anything useful in there. I'll certainly keep an eye on it, but I do not believe that we're seeing real network problems here. We've investigated the network pretty thoroughly and it all checks out.</p> <p>I think we need to look higher in the stack. I don't think we've really lost the message. But my suspicion is that the server sent the reply too slowly, and the client sent the retry before the server every replied once.</p> <p>So how do we investigate that problem?</p> <p>When this happens, are there any ptlrpc timeout logs in the console? Theoretically, if there is a resend, there should be a timeout log.</p> <p>I remember a set of patches from llnl that disabled all that "noise" like printing about resent RPCs. I wonder if this i applied by default on their deployments?</p> <p>Chris: Have you been able to "turn up" the RPC logging to see if there are resend logs?</p> <p>Chris, could you please advise whether you quiet the message in ptlrpc_expire_one_request() displayed when a timeout happens?</p> <p>We do change it from D_WARNING to D_NETERROR. Our code is in git on github.com/chaos/lustre. Our most recent branch is <a href="https://github.com/chaos/lustre/tree/2.3.54-llnl" title="2.3.54-llnl" class="external-link" target="_blank" rel="nofollow noopener">2.3.54-llnl</a>.</p> <p>Chris, D_NETERROR messages only go to the Lustre debug log without an explicit 'echo +neterror &gt; /proc/sys/lnet/printk'.</p> <p>We will never get back to this one.</p> Related LU-1717 Development Rank 1|hzvagn: Rank (Obsolete) 5230 Severity