[LU-4179] LBUG ASSERTION( !lustre_handle_is_used(&lhc->mlh_reg_lh) ) failed:

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.6.0, Lustre 2.4.2, Lustre 2.5.1
• Affects Version/s: Lustre 2.4.0, Lustre 2.4.1
• Labels: server
• Environment: our source is at git://github.com/jlan/lustre-nas.git
• Severity: 3

Epic

server

• Assignee: Di Wang
• Reporter: Mahmoud Hanafi

Description

We have the crash dumps. But requires analysis by US personal only.

LustreError: 6425:0:(mdt_open.c:1690:mdt_reint_open()) ASSERTION( !lustre_handle_is_used(&lhc->mlh_reg_lh) ) failed: ^M
LustreError: 6425:0:(mdt_open.c:1690:mdt_reint_open()) LBUG^M
4>Pid: 6425, comm: mdt01_002^M
<4>^M
<4>Call Trace:^M
<4> [<ffffffffa041f895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]^M
<4> [<ffffffffa041fe97>] lbug_with_loc+0x47/0xb0 [libcfs]^M
<4> [<ffffffffa0ca7553>] mdt_reint_open+0x1973/0x20c0 [mdt]^M
<4> [<ffffffffa0ca832c>] mdt_reconstruct_open+0x68c/0xc30 [mdt]^M
<4> [<ffffffffa072d6a6>] ? __req_capsule_get+0x166/0x700 [ptlrpc]^M
<4> [<ffffffffa07061ae>] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc]^M
<4> [<ffffffffa0c9b195>] mdt_reconstruct+0x45/0x120 [mdt]^M
<4> [<ffffffffa0c76cfb>] mdt_reint_internal+0x6bb/0x780 [mdt]^M
<4> [<ffffffffa0c7708d>] mdt_intent_reint+0x1ed/0x520 [mdt]^M
<4> [<ffffffffa0c74f3e>] mdt_intent_policy+0x39e/0x720 [mdt]^M
<4> [<ffffffffa06bd7e1>] ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc]^M
<4> [<ffffffffa06e424f>] ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc]^M
<4> [<ffffffffa0c753c6>] mdt_enqueue+0x46/0xe0 [mdt]^M
<4> [<ffffffffa0c7bab7>] mdt_handle_common+0x647/0x16d0 [mdt]^M
<4> [<ffffffffa0706c0c>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc]^M
<4> [<ffffffffa0cb5295>] mds_regular_handle+0x15/0x20 [mdt]^M
<4> [<ffffffffa0716428>] ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]^M
<4> [<ffffffffa04205de>] ? cfs_timer_arm+0xe/0x10 [libcfs]^M
<4> [<ffffffffa0431dbf>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]^M
<4> [<ffffffffa070d789>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]^M
<4> [<ffffffff810557f3>] ? __wake_up+0x53/0x70^M
<4> [<ffffffffa07177be>] ptlrpc_main+0xace/0x1700 [ptlrpc]^M
<4> [<ffffffffa0716cf0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]^M
<4> [<ffffffff8100c0ca>] child_rip+0xa/0x20^M
<4> [<ffffffffa0716cf0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]^M
<4> [<ffffffffa0716cf0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]^M
<4> [<ffffffff8100c0c0>] ? child_rip+0x0/0x20^M
<4>^M
<0>Kernel panic - not syncing: LBUG^M
<4>Pid: 6425, comm: mdt01_002 Tainted: G --------------- T 2.6.32-358.6.2.el6.20130607.x86_64.lustre240 #1^M
<4>Call Trace:^M
<4> [<ffffffff8153e8da>] ? panic+0xa7/0x190^M
<4> [<ffffffffa041feeb>] ? lbug_with_loc+0x9b/0xb0 [libcfs]^M
<4> [<ffffffffa0ca7553>] ? mdt_reint_open+0x1973/0x20c0 [mdt]^M
<4> [<ffffffffa0ca832c>] ? mdt_reconstruct_open+0x68c/0xc30 [mdt]^M
<4> [<ffffffffa072d6a6>] ? __req_capsule_get+0x166/0x700 [ptlrpc]^M
<4> [<ffffffffa07061ae>] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc]^M
<4> [<ffffffffa0c9b195>] ? mdt_reconstruct+0x45/0x120 [mdt]^M
<4> [<ffffffffa0c76cfb>] ? mdt_reint_internal+0x6bb/0x780 [mdt]^M
<4> [<ffffffffa0c7708d>] ? mdt_intent_reint+0x1ed/0x520 [mdt]^M
<4> [<ffffffffa0c74f3e>] ? mdt_intent_policy+0x39e/0x720 [mdt]^M
<4> [<ffffffffa06bd7e1>] ? ldlm_lock_enqueue+0x361/0x8d0 [ptlrpc]^M
<4> [<ffffffffa06e424f>] ? ldlm_handle_enqueue0+0x4ef/0x10b0 [ptlrpc]^M
<4> [<ffffffffa0c753c6>] ? mdt_enqueue+0x46/0xe0 [mdt]^M
<4> [<ffffffffa0c7bab7>] ? mdt_handle_common+0x647/0x16d0 [mdt]^M
<4> [<ffffffffa0706c0c>] ? lustre_msg_get_transno+0x8c/0x100 [ptlrpc]^M
<4> [<ffffffffa0cb5295>] ? mds_regular_handle+0x15/0x20 [mdt]^M
<4> [<ffffffffa0716428>] ? ptlrpc_server_handle_request+0x398/0xc60 [ptlrpc]^M
<4> [<ffffffffa04205de>] ? cfs_timer_arm+0xe/0x10 [libcfs]^M
<4> [<ffffffffa0431dbf>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]^M
<4> [<ffffffffa070d789>] ? ptlrpc_wait_event+0xa9/0x290 [ptlrpc]^M
<4> [<ffffffff810557f3>] ? __wake_up+0x53/0x70^M
<4> [<ffffffffa07177be>] ? ptlrpc_main+0xace/0x1700 [ptlrpc]^M
<4> [<ffffffffa0716cf0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]^M
<4> [<ffffffff8100c0ca>] ? child_rip+0xa/0x20^M
<4> [<ffffffffa0716cf0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]^M
<4> [<ffffffffa0716cf0>] ? ptlrpc_main+0x0/0x1700 [ptlrpc]^M
<4> [<ffffffff8100c0c0>] ? child_rip+0x0/0x20^M

Issue Links

• is related to: LU-3607 Interop 2.3.0<->2.5 failure on test suite racer test_1: ASSERTION( !lustre_handle_is_used(&lhc->mlh_reg_lh) ) failed

Comments

[Peter Jones]

Great - thanks Mahmoud!

[Mahmoud Hanafi]

Patched applied and haven't see this bug.
Note: NASA BUILD 2.4.0.4-1nasS

Please close

[Jian Yu]

Lustre client: http://build.whamcloud.com/job/lustre-b2_4/57/
Lustre server: http://build.whamcloud.com/job/lustre-b2_3/41/ (2.3.0)

racer test also hit this LBUG:
https://maloo.whamcloud.com/test_sets/d1007e84-54b0-11e3-9029-52540035b04c

[Di Wang]

Yes, they are same, but just some tag issue. If you just pull out the patch and build yourself, you can ignore it.

[Jay Lan (Inactive)]

Hmm, I had no problem patching and building based on
http://review.whamcloud.com/8145 in my b2_4 build environment.

I compared one in 8145 with one in 8173, the patches seemed to be identical to me?

[Di Wang]

http://review.whamcloud.com/8145 has the wrong tag "based on master), actually that patch is based on b2_4, so I resubmit a new one http://review.whamcloud.com/8173 please track this one.

[Di Wang]

IMHO, the patch itself is fine, but we can not say it is "safe', until it went through all of our validation processes, i.e. reviewing, passing internal tests and landing, as peter mentioned.

[Mahmoud Hanafi]

What is the risk with this patch. We can't fully test the patch using test env because we didn't isolate trigger. We would need to deploy it on production to fully test it.

[Peter Jones]

Jay

Just to be clear - please only use the patch in a test environment until we have completed our validation

Thanks

Peter

[Di Wang]

http://review.whamcloud.com/8145 Here is the one for b2_4, and the patch should be able to applied to 2.4.0 as well, please try.