[LU-5629] osp_sync_interpret() ASSERTION( rc || req->rq_transno ) failed

• Type: Bug
• Resolution: Duplicate
• Priority: Critical
• Affects Version/s: Lustre 2.6.0, Lustre 2.4.2, Lustre 2.5.3
• Labels: llnl
• Environment: Lustre 2.4.2-14chaos (see github.com/chaos/lustre)
• Severity: 3
• Assignee: Dmitry Eremin (Inactive)
• Reporter: Christopher Morrone (Inactive)

Description

One of our MDS nodes crashed to day with the following assertion:

client.c:304:ptlrpc_at_adj_net_latency()) Reported service time 548 > total measured time 165
osp_sync.c:355:osp_sync_interpret())  ASSERTION( rc || req->rq_transno ) failed

Note that the two messages above were printed in the same second (as reported by syslog) and by the same kernel thread. I don't know if the ptlrpc_at_adj_net_latency() message is actually related to the assertion or not, but the proximity makes it worth noting.

There were a few OST to which the MDS lost and reestablished a connection a couple of minutes earlier in the log.

The backtrace was:

panic
lbug_with_loc
osp_sync_interpret
ptlrpc_check_set
ptlrpcd_check
ptlrpcd
kernel_thread

It was running lustre version 2.4.2-14chaos (see github.com/chaos/lustre).

We cannot provide logs or crash dumps for this machine.

Issue Links

• is related to: LU-3892 osp_sync.c:356:osp_sync_interpret()) ASSERTION( req->rq_transno == 0 ) failed
• is related to: LU-7453 osp_sync_interpret assertion
• is related to: LU-9135 sanity test_313: osp_sync.c:571:osp_sync_interpret()) LBUG
• is related to: LU-5193 2.6 DNE stress testing: osp_sync_interpret()) ASSERTION( rc || req->rq_transno ) failed:

Comments

[Peter Jones]

Closing as a duplicate of LU-9135

[Dmitry Eremin (Inactive)]

Probably the patch https://review.whamcloud.com/30129/ should resolve this.

[Scott Kirvan (Inactive)]

Exact same issue @ LANL, Lustre 2.5.5.

[Cameron Harr]

Saw the same crash on 8/03/16.
lustre-2.5.5-6chaos_2.6.32_573.26.1.1chaos.ch5.4.x86_64.x86_64

We have a 16GB dump available if need be.

Not sure how related it is, but an OSS node suffered major hardware problems (MCEs) throughout the 30 minutes before the LBUG on the MDS. The MDS console log messages directly (~2 min) before the assertion were an evict/reconnect to that OST.

[Ruth Klundt (Inactive)]

Attached server syslogs, the stack dumps were not captured unfortunately, but the location for that collection is mounted now in case it happens again.

There had been network issues earlier in the day, reportedly resolved by 4pm.

fyi the number of clients on the fs is currently 6395. And the exact version of the software is
lustre: 2.5.5
kernel: patchless_client
build: -6chaos-CHANGED-2.6.32-573.26.1.1chaos.ch5.4.x86_64

[Alex Zhuravlev]

any logs/dumps?

[Ruth Klundt (Inactive)]

Add one more at snl last night, also running 2.5.5.

[Lance Weems]

Wanted to report we hit this over the weekend on our 2.5.5 production file system here at LLNL.

[James A Simmons]

Looks like we just hit this on our 2.5.3+ production file system

[Christopher Morrone (Inactive)]

I am reopening this ticket because it does not appear that the issue was resolved as previously believed. We are still seeing the same assertion with Lustre 2.5.3, which contains the patch from LU-3892 (commit 7f4a635, which landed well before 2.5.0).