[LU-16596] Lustre client crashed with ASSERTION( ldlm_is_granted(lock) ) failed

Related — Thu, 30 Nov 2023 17:09:27 +0000

Yesterday one of our Lustre clients built on top of the 2.15.2 release (from the b2_15 branch) crashed with the following LBUG:

[1361556.476660] LustreError: 882654:0:(ldlm_lock.c:1097:ldlm_grant_lock_with_skiplist()) ASSERTION( ldlm_is_granted(lock) ) failed:
[1361556.478018] LustreError: 882654:0:(ldlm_lock.c:1097:ldlm_grant_lock_with_skiplist()) LBUG
[1361556.478681] Pid: 882654, comm: ldlm_bl_47 4.18.0-425.10.1.el8_7.x86_64 #1 SMP Thu Jan 12 16:32:13 UTC 2023
[1361556.479401] Call Trace TBD:
[1361556.479664] [<0>] libcfs_call_trace+0x6f/0xa0 [libcfs]
[1361556.480109] [<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[1361556.480552] [<0>] ldlm_grant_lock_with_skiplist+0x642/0x780 [ptlrpc]
[1361556.481236] [<0>] ldlm_inodebits_drop+0xba/0x160 [ptlrpc]
[1361556.481706] [<0>] ldlm_cli_inodebits_convert+0x426/0x6c0 [ptlrpc]
[1361556.482242] [<0>] ldlm_cli_convert+0x68/0x2a0 [ptlrpc]
[1361556.482673] [<0>] ll_md_blocking_ast+0x131/0x2f0 [lustre]
[1361556.483137] [<0>] ldlm_handle_bl_callback+0xbc/0x3f0 [ptlrpc]
[1361556.483646] [<0>] ldlm_bl_thread_main+0x633/0x930 [ptlrpc]
[1361556.484103] [<0>] kthread+0x10b/0x130
[1361556.484440] [<0>] ret_from_fork+0x1f/0x40
[1361556.484767] Kernel panic - not syncing: LBUG
[1361556.485112] CPU: 9 PID: 882654 Comm: ldlm_bl_47 Kdump: loaded Tainted: G OE --------- - - 4.18.0-425.10.1.el8_7.x86_64 #1
[1361556.486068] Hardware name: Red Hat KVM/RHEL-AV, BIOS 1.13.0-2.module_el8.3.0+555+a55c8938 04/01/2014
[1361556.486785] Call Trace:
[1361556.487006] dump_stack+0x41/0x60
[1361556.487297] panic+0xe7/0x2ac
[1361556.487542] ? ret_from_fork+0x1f/0x40
[1361556.487849] lbug_with_loc.cold.8+0x18/0x18 [libcfs]
[1361556.488254] ldlm_grant_lock_with_skiplist+0x642/0x780 [ptlrpc]
[1361556.488751] ldlm_inodebits_drop+0xba/0x160 [ptlrpc]
[1361556.489183] ldlm_cli_inodebits_convert+0x426/0x6c0 [ptlrpc]
[1361556.489679] ? ll_have_md_lock+0x169/0x3f0 [lustre]
[1361556.490087] ldlm_cli_convert+0x68/0x2a0 [ptlrpc]
[1361556.490492] ll_md_blocking_ast+0x131/0x2f0 [lustre]
[1361556.490906] ? obd_stale_export_get+0x75/0x190 [obdclass]
[1361556.491438] ldlm_handle_bl_callback+0xbc/0x3f0 [ptlrpc]
[1361556.491950] ldlm_bl_thread_main+0x633/0x930 [ptlrpc]
[1361556.492385] ? finish_wait+0x80/0x80
[1361556.492675] ? ldlm_handle_bl_callback+0x3f0/0x3f0 [ptlrpc]
[1361556.493150] kthread+0x10b/0x130
[1361556.493413] ? set_kthread_struct+0x50/0x50
[1361556.493803] ret_from_fork+0x1f/0x40

I found a similar LU: https://jira.whamcloud.com/browse/LU-13927 but with the Lustre 2.12.5 on MDS, patches linked in this LU are already there.

Whamcloud Community JIRA

[LU-16596] Lustre client crashed with ASSERTION( ldlm_is_granted(lock) ) failed