[LU-2187] Why are we losing messages?

• Type: Bug
• Resolution: Cannot Reproduce
• Priority: Minor
• Affects Version/s: Lustre 2.4.0
• Labels: sequoia
• Severity: 3
• Assignee: Doug Oucharek (Inactive)
• Reporter: Christopher Morrone (Inactive)

Description

According to the analysis of LU-1717 we are frequently losing Lustre messages on Sequoia's IB network. We have no LNet routers, and IB is a reliable network. We are not seeing any timeouts or lnet errors that would suggest that we are seeing IB transmission problems.

Why are messages being lost that can result in LU-1717 error messages? I'm worried that we're papering over a larger problem by silencing those errors.

Issue Links

• is related to: LU-1717 mdt_recovery.c:611:mdt_steal_ack_locks()) Resent req xid XXX has mismatched opc: new 101 old 0

Comments

[Christopher Morrone (Inactive)]

We will never get back to this one.

[Isaac Huang (Inactive)]

Chris, D_NETERROR messages only go to the Lustre debug log without an explicit 'echo +neterror > /proc/sys/lnet/printk'.

[Christopher Morrone (Inactive)]

We do change it from D_WARNING to D_NETERROR. Our code is in git on github.com/chaos/lustre. Our most recent branch is 2.3.54-llnl.

[Johann Lombardi (Inactive)]

Chris, could you please advise whether you quiet the message in ptlrpc_expire_one_request() displayed when a timeout happens?

[Doug Oucharek (Inactive)]

Chris: Have you been able to "turn up" the RPC logging to see if there are resend logs?

[Oleg Drokin]

I remember a set of patches from llnl that disabled all that "noise" like printing about resent RPCs. I wonder if this i applied by default on their deployments?

[Doug Oucharek (Inactive)]

When this happens, are there any ptlrpc timeout logs in the console? Theoretically, if there is a resend, there should be a timeout log.