https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5629 Lustre <p>One of our MDS nodes crashed to day with the following assertion:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>client.c:304:ptlrpc_at_adj_net_latency()) Reported service time 548 &gt; total measured time 165 osp_sync.c:355:osp_sync_interpret()) ASSERTION( rc || req-&gt;rq_transno ) failed </pre> </div></div> <p>Note that the two messages above were printed in the same second (as reported by syslog) and by the same kernel thread. I don't know if the ptlrpc_at_adj_net_latency() message is actually related to the assertion or not, but the proximity makes it worth noting. </p> <p>There were a few OST to which the MDS lost and reestablished a connection a couple of minutes earlier in the log.</p> <p>The backtrace was:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>panic lbug_with_loc osp_sync_interpret ptlrpc_check_set ptlrpcd_check ptlrpcd kernel_thread </pre> </div></div> <p>It was running lustre version 2.4.2-14chaos (see github.com/chaos/lustre).</p> <p>We cannot provide logs or crash dumps for this machine.</p> Lustre 2.4.2-14chaos (see github.com/chaos/lustre) LU-5629

osp_sync_interpret() ASSERTION( rc || req->rq_transno ) failed

Bug Critical Resolved Duplicate Dmitry Eremin Christopher Morrone llnl Tue, 16 Sep 2014 00:47:34 +0000 Fri, 12 Jan 2018 13:20:23 +0000 Fri, 12 Jan 2018 13:20:23 +0000 Lustre 2.6.0 Lustre 2.4.2 Lustre 2.5.3 0 15 <p>In <a href="https://jira.whamcloud.com/browse/LU-5193" title="2.6 DNE stress testing: osp_sync_interpret()) ASSERTION( rc || req-&gt;rq_transno ) failed:" class="issue-link" data-issue-key="LU-5193"><del>LU-5193</del></a> Cray reports hitting the same assertion under lustre 2.6.</p> <p>FYI, I think a possible reason of ptlrpc_at_adj_net_latency() warning is because early reply is lost so RPC is expired, then client(it is OSP of MDS at here) resends the request, because reply of original request (on server) is still using the same rq_xid as match-bits of reply, so it can fit into the reposted reply buffer.</p> <p>If this happened, service time returned by original reply can be longer than execution time of the resent RPC. I'm not sure if this is relevant to the assertion, but we at least should remove this warning and only put it in debug info.</p> <p>Dmitry is looking into this one</p> <p>Commit e12b89a9e7d8409c2b624162760c2e7e3481d7be with fix was landed in 2.4.2.</p> <p>duplicate of lu-3892</p> <p>I am reopening this ticket because it does not appear that the issue was resolved as previously believed. We are still seeing the same assertion with Lustre 2.5.3, which contains the patch from <a href="https://jira.whamcloud.com/browse/LU-3892" title="osp_sync.c:356:osp_sync_interpret()) ASSERTION( req-&gt;rq_transno == 0 ) failed" class="issue-link" data-issue-key="LU-3892"><del>LU-3892</del></a> (commit 7f4a635, which landed well before 2.5.0).</p> <p>Looks like we just hit this on our 2.5.3+ production file system</p> <p>Wanted to report we hit this over the weekend on our 2.5.5 production file system here at LLNL.</p> <p>Add one more at snl last night, also running 2.5.5. </p> <p>any logs/dumps?</p> <p>Attached server syslogs, the stack dumps were not captured unfortunately, but the location for that collection is mounted now in case it happens again. </p> <p>There had been network issues earlier in the day, reportedly resolved by 4pm. </p> <p>fyi the number of clients on the fs is currently 6395. And the exact version of the software is <br/> lustre: 2.5.5<br/> kernel: patchless_client<br/> build: -6chaos-CHANGED-2.6.32-573.26.1.1chaos.ch5.4.x86_64</p> <p>Saw the same crash on 8/03/16. <br/> lustre-2.5.5-6chaos_2.6.32_573.26.1.1chaos.ch5.4.x86_64.x86_64</p> <p>We have a 16GB dump available if need be.</p> <p>Not sure how related it is, but an OSS node suffered major hardware problems (MCEs) throughout the 30 minutes before the LBUG on the MDS. The MDS console log messages directly (~2 min) before the assertion were an evict/reconnect to that OST.</p> <p>Exact same issue @ LANL, Lustre 2.5.5.</p> <p>Probably the patch <a href="https://review.whamcloud.com/30129/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30129/</a> should resolve this.</p> <p>Closing as a duplicate of <a href="https://jira.whamcloud.com/browse/LU-9135" title="sanity test_313: osp_sync.c:571:osp_sync_interpret()) LBUG" class="issue-link" data-issue-key="LU-9135"><del>LU-9135</del></a></p> Related LU-3892 LU-7453 LU-9135 LU-5193 Development End date Mon, 23 May 2016 00:47:34 +0000 Rank 1|hzwwcf: Rank (Obsolete) 15744 Severity Start date Tue, 16 Sep 2014 00:47:34 +0000