[LU-5266] LBUG on Failover -ldlm_process_extent_lock()) ASSERTION( lock->l_granted_mode != lock->l_req_mode )

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 2.6.0
• Fix Version/s: Lustre 2.6.0, Lustre 2.5.4
• Type: Bug
• Priority: Blocker
• Reporter: Cliff White (Inactive)
• Assignee: Hongchao Zhang
• Resolution: Fixed
• Labels: HB

Environment

Hyperion - 2.5.60 build 2538

• Severity: 3

Description

After hard failover of devices to server iws19, server wedged, then hit LBUG.
Services never complete recovery, and appear to either restart the timer or something:

Jun 27 10:08:53 iws19 kernel: Lustre: lustre-OST000c: Will be in recovery for at least 2:30, or until 316 clients reconnect
Jun 27 10:11:53 iws19 kernel: Lustre: lustre-OST000c: recovery is timed out, evict stale exports
Jun 27 10:18:30 iws19 kernel: Lustre: lustre-OST000c: Client c52d4856-d1df-b87b-911c-f1bfbc23a24d (at 192.168.124.182@o2ib) reconnecting, waiting for 316 clients in recovery for 2:27

The server reports being cpu-bound prior to the failure
Console log attached - unfortunately the dump after the LBUG failed.

Issue Links

• is related to: LU-2827 mdt_intent_fixup_resent() cannot find...
• is related to: LU-5496 fix for LU-5266

Comments

[Oleg Drokin]

We need the backtrace for the crash please.

[Andreas Dilger]

Cliff,
Did you try to reboot the server again and/or try more failovers after this one?

[Cliff White (Inactive)]

Yes, we had multiple failures

[Jodi Levi (Inactive)]

HongChao,
could you please look into this one?
Thank you!

[Vitaly Fertman]

not sure if this failure is the same as the fixed one, but as caught by the same assertion, to not create another ticket, I put it here: http://review.whamcloud.com/10903

[Hongchao Zhang]

Yes, the issue could be triggered by the resent lock request.

[Andreas Dilger]

Was this introduced by http://review.whamcloud.com/5978 ?

[Cliff White (Inactive)]

I tested the patch on Hyperion - no more LBUG, but did have a few evictions. lustre-logs and console logs from one run attached.

[Peter Jones]

Landed for 2.6

[Cliff White (Inactive)]

While testing this patch, still have some client evictions and log dumps. Message log and lustre log from latest attached.

[Andreas Dilger]

This should probably go into a new bug, since the LASSERT is fixed.

[Vitaly Fertman]

the previous fix was not exactly correct: http://review.whamcloud.com/11469

[Peter Jones]

Vitaly
Could you please open a new JIRA ticket to track this additional change. The original fix was in the already GA 2.6 release
Thanks
Peter