https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9765 Lustre <p>We have now two routers, from IB to OPA clients, soak-14/15<br/> After several hours of running, soak-15 crashed hard. Multiple NMI on multiple CPU.<br/> Output is somewhat messy<br/> Crash dump is available on the node. vmcore-dmesg and console log attached</p> Soak performance cluster LU-9765

NMI watchdog - OPA <-> IB LNET router

Bug Major Open Unresolved Amir Shehata Cliff White soak Wed, 12 Jul 2017 15:07:19 +0000 Thu, 13 Jul 2017 21:19:45 +0000 Lustre 2.10.0 0 7 <p>Amir is investigating</p> <p>What version of IFS is installed? It looks very similar to issue of IFS which is call schedule() with lock acquired. This is fixed in IFS 10.4 version.</p> <p>How do I find the IFS version installed?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"># cat /etc/opa/version_wrapper </pre> </div></div> <p>From the core it appears that cpt 1 net lock has been unlocked one too many times:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">crash&gt; p* the_lnet.ln_net_lock-&gt;pcl_locks[0] $27 = { { rlock = { raw_lock = { { head_tail = 171313694, tickets = { head = 2590, tail = 2614 } } } } } } crash&gt; p* the_lnet.ln_net_lock-&gt;pcl_locks[1] $28 = { { rlock = { raw_lock = { { head_tail = 2492241038, tickets = { head = 38030, tail = 38028 } } } } } } </pre> </div></div> <p>We're currently suspecting an issue on the routing path, although a code inspection didn't reveal anything obvious. We're continuing to investigate.</p> <p>In the mean time Dmitry installed IFS 10.4 on the routers (soak-14 and soak-15), to avoid running into the HFI bug which leads to a deadlock.</p> <p>Next time when we run the soak tests using the routers can we turn on net error logging:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">lctl set_param debug=+neterror </pre> </div></div> <p>Then turn on the debug daemon to capture any relevant logs during the test run.</p> <p>Maybe <a href="https://jira.whamcloud.com/browse/LU-9769" title="Exit from function with acquired lock (lost lock)." class="issue-link" data-issue-key="LU-9769"><del>LU-9769</del></a> relate to this.</p> <p>After looking at the core, it appears that all the CPUs are stuck on CPT 0. <a href="https://jira.whamcloud.com/browse/LU-9769" title="Exit from function with acquired lock (lost lock)." class="issue-link" data-issue-key="LU-9769"><del>LU-9769</del></a> would have an impact if an older lnetctl was used to delete a net, but provided the wrong net_id. So it could potentially be an issue in that scenario. It would be good to run with that patch just in case..</p> Related LU-9769 Development Rank 1|hzzgj3: Rank (Obsolete) 9223372036854775807 Severity