[LU-9765] NMI watchdog - OPA <-> IB LNET router

• Status: Open
• Project: Lustre
• Affects Version/s: Lustre 2.10.0
• Type: Bug
• Priority: Major
• Reporter: Cliff White (Inactive)
• Assignee: Amir Shehata (Inactive)
• Resolution: Unresolved
• Labels: soak

Environment

Soak performance cluster

• Severity: 3

Description

We have now two routers, from IB to OPA clients, soak-14/15
After several hours of running, soak-15 crashed hard. Multiple NMI on multiple CPU.
Output is somewhat messy
Crash dump is available on the node. vmcore-dmesg and console log attached

Issue Links

• is related to: LU-9769 Exit from function with acquired lock...

Comments

[Peter Jones]

Amir is investigating

[Dmitry Eremin (Inactive)]

What version of IFS is installed? It looks very similar to issue of IFS which is call schedule() with lock acquired. This is fixed in IFS 10.4 version.

[Amir Shehata (Inactive)]

How do I find the IFS version installed?

[Dmitry Eremin (Inactive)]

# cat /etc/opa/version_wrapper

[Amir Shehata (Inactive)]

From the core it appears that cpt 1 net lock has been unlocked one too many times:

crash> p* the_lnet.ln_net_lock->pcl_locks[0]
$27 = {
  {
    rlock = {
      raw_lock = {
        {
          head_tail = 171313694, 
          tickets = {
            head = 2590, 
            tail = 2614
          }
        }
      }
    }
  }
}
crash> p* the_lnet.ln_net_lock->pcl_locks[1]
$28 = {
  {
    rlock = {
      raw_lock = {
        {
          head_tail = 2492241038, 
          tickets = {
            head = 38030, 
            tail = 38028
          }
        }
      }
    }
  }
}

We're currently suspecting an issue on the routing path, although a code inspection didn't reveal anything obvious. We're continuing to investigate.

In the mean time Dmitry installed IFS 10.4 on the routers (soak-14 and soak-15), to avoid running into the HFI bug which leads to a deadlock.

Next time when we run the soak tests using the routers can we turn on net error logging:

lctl set_param debug=+neterror

Then turn on the debug daemon to capture any relevant logs during the test run.

[Dmitry Eremin (Inactive)]

Maybe LU-9769 relate to this.

[Amir Shehata (Inactive)]

After looking at the core, it appears that all the CPUs are stuck on CPT 0. LU-9769 would have an impact if an older lnetctl was used to delete a net, but provided the wrong net_id. So it could potentially be an issue in that scenario. It would be good to run with that patch just in case..