[LU-14780] LustreError: 4936:0:(file.c:4985:ll_layout_lock_set()) LBUG

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.15.0
• Labels: LTS12
• Severity: 3
• Assignee: Zhenyu Xu
• Reporter: Zhenyu Xu

Description

A case happened on client nodes which LBUG on "using ls -l", however, if "lfs getstripe" it first, I can then "ls -l" without triggering the LBUG.

2021-06-03 10:50:33 [ 1584.601278] LustreError: 4936:0:(file.c:4985:ll_layout_lock_set()) ASSERTION( ldlm_has_layout(lock) ) failed: 
2021-06-03 10:50:33 [ 1584.606428] LustreError: 4938:0:(file.c:4985:ll_layout_lock_set()) ASSERTION( ldlm_has_layout(lock) ) failed: 
2021-06-03 10:50:33 [ 1584.612486] LustreError: 4936:0:(file.c:4985:ll_layout_lock_set()) LBUG
2021-06-03 10:50:33 [ 1584.612489] Pid: 4936, comm: dsync 4.18.0-240.15.1.el8.x86_64 #1 SMP Wed Mar 17 23:53:03 UTC 2021
2021-06-03 10:50:33 [ 1584.623732] LustreError: 4938:0:(file.c:4985:ll_layout_lock_set()) LBUG
2021-06-03 10:50:33 [ 1584.631129] Call Trace:
2021-06-03 10:50:33 [ 1584.631157]  libcfs_call_trace+0x86/0xc0 [libcfs]
2021-06-03 10:50:33 [ 1584.656942]  lbug_with_loc+0x43/0x80 [libcfs]
2021-06-03 10:50:33 [ 1584.661880]  ll_layout_lock_set+0xac/0x610 [lustre]
2021-06-03 10:50:33 [ 1584.665010] LustreError: 4935:0:(file.c:4985:ll_layout_lock_set()) ASSERTION( ldlm_has_layout(lock) ) failed: 
2021-06-03 10:50:33 [ 1584.667385]  ll_layout_refresh+0x1b7/0x310 [lustre]
2021-06-03 10:50:33 [ 1584.678675] LustreError: 4935:0:(file.c:4985:ll_layout_lock_set()) LBUG
2021-06-03 10:50:33 [ 1584.684169]  vvp_io_init+0x20c/0x340 [lustre]
2021-06-03 10:50:33 [ 1584.695274] LustreError: 4931:0:(file.c:4985:ll_layout_lock_set()) ASSERTION( ldlm_has_layout(lock) ) failed: 
2021-06-03 10:50:33 [ 1584.696524]  cl_io_init0.isra.16+0x83/0x130 [obdclass]
2021-06-03 10:50:33 [ 1584.707788] LustreError: 4931:0:(file.c:4985:ll_layout_lock_set()) LBUG
2021-06-03 10:50:33 [ 1584.713559]  cl_glimpse_size0+0x97/0x240 [lustre]
2021-06-03 10:50:33 [ 1584.726287]  ll_getattr+0x1d6/0x460 [lustre]
2021-06-03 10:50:33 [ 1584.726847] LustreError: 4939:0:(file.c:4985:ll_layout_lock_set()) ASSERTION( ldlm_has_layout(lock) ) failed: 
2021-06-03 10:50:33 [ 1584.731123]  vfs_statx+0x8a/0xd0
2021-06-03 10:50:33 [ 1584.731125]  __do_sys_newlstat+0x39/0x70
2021-06-03 10:50:33 [ 1584.731129]  do_syscall_64+0x5b/0x1a0
2021-06-03 10:50:33 [ 1584.731132]  entry_SYSCALL_64_after_hwframe+0x65/0xca
2021-06-03 10:50:33 [ 1584.731135]  0xffffffffffffffff
2021-06-03 10:50:33 [ 1584.731138] Kernel panic - not syncing: LBUG
2021-06-03 10:50:33 [ 1584.731139] Pid: 4938, comm: dsync 4.18.0-240.15.1.el8.x86_64 #1 SMP Wed Mar 17 23:53:03 UTC 2021
2021-06-03 10:50:33 [ 1584.731140] Call Trace:
2021-06-03 10:50:33 [ 1584.731170]  libcfs_call_trace+0x86/0xc0 [libcfs]
2021-06-03 10:50:33 [ 1584.731175]  lbug_with_loc+0x43/0x80 [libcfs]
2021-06-03 10:50:33 [ 1584.731201]  ll_layout_lock_set+0xac/0x610 [lustre]
2021-06-03 10:50:34 [ 1584.731209]  ll_layout_refresh+0x1b7/0x310 [lustre]
2021-06-03 10:50:34 [ 1584.731220]  vvp_io_init+0x20c/0x340 [lustre]
2021-06-03 10:50:34 [ 1584.731257]  cl_io_init0.isra.16+0x83/0x130 [obdclass]
2021-06-03 10:50:34 [ 1584.731267]  cl_glimpse_size0+0x97/0x240 [lustre]
2021-06-03 10:50:34 [ 1584.731275]  ll_getattr+0x1d6/0x460 [lustre]
2021-06-03 10:50:34 [ 1584.731280]  vfs_statx+0x8a/0xd0
2021-06-03 10:50:34 [ 1584.731281]  __do_sys_newlstat+0x39/0x70
2021-06-03 10:50:34 [ 1584.731285]  do_syscall_64+0x5b/0x1a0
2021-06-03 10:50:34 [ 1584.731288]  entry_SYSCALL_64_after_hwframe+0x65/0xca
2021-06-03 10:50:34 [ 1584.731290]  0xffffffffffffffff
2021-06-03 10:50:34 [ 1584.731292] Pid: 4935, comm: dsync 4.18.0-240.15.1.el8.x86_64 #1 SMP Wed Mar 17 23:53:03 UTC 2021
2021-06-03 10:50:34 [ 1584.731293] Call Trace:
2021-06-03 10:50:34 [ 1584.731314]  libcfs_call_trace+0x86/0xc0 [libcfs]
2021-06-03 10:50:34 [ 1584.731318]  lbug_with_loc+0x43/0x80 [libcfs]
2021-06-03 10:50:34 [ 1584.731338]  ll_layout_lock_set+0xac/0x610 [lustre]
2021-06-03 10:50:34 [ 1584.731347]  ll_layout_refresh+0x1b7/0x310 [lustre]
2021-06-03 10:50:34 [ 1584.731357]  vvp_io_init+0x20c/0x340 [lustre]
2021-06-03 10:50:34 [ 1584.731386]  cl_io_init0.isra.16+0x83/0x130 [obdclass]
2021-06-03 10:50:34 [ 1584.731396]  cl_glimpse_size0+0x97/0x240 [lustre]
2021-06-03 10:50:34 [ 1584.731405]  ll_getattr+0x1d6/0x460 [lustre]
2021-06-03 10:50:34 [ 1584.731408]  vfs_statx+0x8a/0xd0
2021-06-03 10:50:34 [ 1584.731410]  __do_sys_newlstat+0x39/0x70
2021-06-03 10:50:34 [ 1584.731413]  do_syscall_64+0x5b/0x1a0
2021-06-03 10:50:34 [ 1584.731415]  entry_SYSCALL_64_after_hwframe+0x65/0xca
2021-06-03 10:50:34 [ 1584.731417]  0xffffffffffffffff
2021-06-03 10:50:34 [ 1584.731418] Pid: 4931, comm: dsync 4.18.0-240.15.1.el8.x86_64 #1 SMP Wed Mar 17 23:53:03 UTC 2021
2021-06-03 10:50:34 [ 1584.731420] Call Trace:
2021-06-03 10:50:34 [ 1584.731434]  libcfs_call_trace+0x86/0xc0 [libcfs]
2021-06-03 10:50:34 [ 1584.731439]  lbug_with_loc+0x43/0x80 [libcfs]
2021-06-03 10:50:34 [ 1584.731451]  ll_layout_lock_set+0xac/0x610 [lustre]
2021-06-03 10:50:34 [ 1584.731462]  ll_layout_refresh+0x1b7/0x310 [lustre]
2021-06-03 10:50:34 [ 1584.731474]  vvp_io_init+0x20c/0x340 [lustre]
2021-06-03 10:50:34 [ 1584.731492]  cl_io_init0.isra.16+0x83/0x130 [obdclass]
2021-06-03 10:50:34 [ 1584.731503]  cl_glimpse_size0+0x97/0x240 [lustre]
2021-06-03 10:50:34 [ 1584.731513]  ll_getattr+0x1d6/0x460 [lustre]
2021-06-03 10:50:34 [ 1584.731516]  vfs_statx+0x8a/0xd0
2021-06-03 10:50:34 [ 1584.731518]  __do_sys_newlstat+0x39/0x70
2021-06-03 10:50:34 [ 1584.731520]  do_syscall_64+0x5b/0x1a0
2021-06-03 10:50:34 [ 1584.731522]  entry_SYSCALL_64_after_hwframe+0x65/0xca
2021-06-03 10:50:34 [ 1584.731523]  0xffffffffffffffff
2021-06-03 10:50:34 [ 1584.742387] LustreError: 4939:0:(file.c:4985:ll_layout_lock_set()) LBUG
2021-06-03 10:50:34 [ 1584.746013] CPU: 29 PID: 4936 Comm: dsync Tainted: G           OE    --------- -  - 4.18.0-240.15.1.el8.x86_64 #1
2021-06-03 10:50:34 [ 1584.750438] Pid: 4939, comm: dsync 4.18.0-240.15.1.el8.x86_64 #1 SMP Wed Mar 17 23:53:03 UTC 2021
2021-06-03 10:50:34 [ 1584.754554] Call Trace:
2021-06-03 10:50:34 [ 1584.760241] Call Trace:
2021-06-03 10:50:34 [ 1584.763786]  dump_stack+0x5c/0x80
2021-06-03 10:50:34 [ 1584.768589]  libcfs_call_trace+0x86/0xc0 [libcfs]
2021-06-03 10:50:34 [ 1584.778951]  panic+0xe7/0x2a9
2021-06-03 10:50:34 [ 1584.781733]  lbug_with_loc+0x43/0x80 [libcfs]
2021-06-03 10:50:34 [ 1584.787014]  lbug_with_loc.cold.3+0x18/0x18 [libcfs]
2021-06-03 10:50:34 [ 1584.791932]  ll_layout_lock_set+0xac/0x610 [lustre]
2021-06-03 10:50:34 [ 1584.797413]  ll_layout_lock_set+0xac/0x610 [lustre]
2021-06-03 10:50:34 [ 1584.802900]  ll_layout_refresh+0x1b7/0x310 [lustre]
2021-06-03 10:50:34 [ 1584.807797]  ll_layout_refresh+0x1b7/0x310 [lustre]
2021-06-03 10:50:34 [ 1584.813573]  vvp_io_init+0x20c/0x340 [lustre]
2021-06-03 10:50:34 [ 1584.818854]  vvp_io_init+0x20c/0x340 [lustre]
2021-06-03 10:50:34 [ 1584.823667]  cl_io_init0.isra.16+0x83/0x130 [obdclass]
2021-06-03 10:50:34 [ 1584.827821]  cl_io_init0.isra.16+0x83/0x130 [obdclass]
2021-06-03 10:50:34 [ 1584.832766]  cl_glimpse_size0+0x97/0x240 [lustre]
2021-06-03 10:50:34 [ 1584.837401]  cl_glimpse_size0+0x97/0x240 [lustre]
2021-06-03 10:50:34 [ 1584.843609]  ll_getattr+0x1d6/0x460 [lustre]
2021-06-03 10:50:34 [ 1584.847642]  ll_getattr+0x1d6/0x460 [lustre]
2021-06-03 10:50:34 [ 1584.858503]  vfs_statx+0x8a/0xd0
2021-06-03 10:50:34 [ 1584.861769]  vfs_statx+0x8a/0xd0
2021-06-03 10:50:34 [ 1584.867591]  __do_sys_newlstat+0x39/0x70
2021-06-03 10:50:34 [ 1584.872992]  __do_sys_newlstat+0x39/0x70
2021-06-03 10:50:34 [ 1584.878985]  do_syscall_64+0x5b/0x1a0
2021-06-03 10:50:34 [ 1584.884952]  do_syscall_64+0x5b/0x1a0
2021-06-03 10:50:34 [ 1584.890314]  entry_SYSCALL_64_after_hwframe+0x65/0xca
2021-06-03 10:50:34 [ 1584.896536]  entry_SYSCALL_64_after_hwframe+0x65/0xca
2021-06-03 10:50:34 [ 1584.902384]  0xffffffffffffffff
2021-06-03 10:50:34 [ 1584.907682] RIP: 0033:0x7f1686fc6d89
2021-06-03 10:50:34 [ 1585.205327] Code: 64 c7 00 16 00 00 00 b8 ff ff ff ff c3 0f 1f 40 00 f3 0f 1e fa 48 89 f0 83 ff 01 77 34 48 89 c7 48 89 d6 b8 06 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 07 c3 66 0f 1f 44 00 00 48 8b 15 c9 00 2d 00
2021-06-03 10:50:34 [ 1585.226999] RSP: 002b:00007ffdf03e1458 EFLAGS: 00000246 ORIG_RAX: 0000000000000006
2021-06-03 10:50:34 [ 1585.235853] RAX: ffffffffffffffda RBX: 00007ffdf03e2750 RCX: 00007f1686fc6d89
2021-06-03 10:50:34 [ 1585.244257] RDX: 00007ffdf03e15f0 RSI: 00007ffdf03e15f0 RDI: 00007ffdf03e1680
2021-06-03 10:50:34 [ 1585.252633] RBP: 00007ffdf03e1480 R08: 0000000000000000 R09: ffffff00000fffff
2021-06-03 10:50:34 [ 1585.260999] R10: 00262f4a53a31a46 R11: 0000000000000246 R12: 00000000023e72f0
2021-06-03 10:50:34 [ 1585.269375] R13: 00007f1687dda308 R14: 00007f1687dda380 R15: 00007f1687dda2e8
2021-06-03 10:50:35 [ 1586.310396] Shutting down cpus with NMI
2021-06-03 10:50:35 [ 1586.438268] Kernel Offset: 0x37a00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
2021-06-03 10:50:35 [ 1586.499673] ---[ end Kernel panic - not syncing: LBUG ]---

While after rebooting one of our MDSs several times (in relation to other Lustre issues) this is no longer reproducible. So it may have been induced by one of the MDTs being in a bad state.

Issue Links

• duplicates: LU-7538 file.c:3891:ll_layout_lock_set()) LBUG

Comments

[Gerrit Updater]

"Oleg Drokin <[e-mail]>" merged in patch https://review.whamcloud.com/c/fs/lustre-release/+/44214/
Subject: LU-14780 llite: failed ASSERTION(ldlm_has_layout(lock))
Project: fs/lustre-release
Branch: b2_12
Current Patch Set:
Commit: cea60c64c4540c3ca29bc64a92707410149ecabf

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/44214
Subject: LU-14780 llite: failed ASSERTION(ldlm_has_layout(lock))
Project: fs/lustre-release
Branch: b2_12
Current Patch Set: 1
Commit: 864c86978d7a267046e08f3cee8d0bd1da74126a

[Peter Jones]

Landed for 2.15

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/44054/
Subject: LU-14780 llite: failed ASSERTION(ldlm_has_layout(lock))
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 1b166d6dd6a2f39dfe35b60be169b288665d0283

[Gerrit Updater]

Bobi Jam ([e-mail]) uploaded a new patch: https://review.whamcloud.com/44054
Subject: LU-14780 llite: failed ASSERTION(ldlm_has_layout(lock))
Project: fs/lustre-release
Branch: master
Current Patch Set: 1
Commit: eac0f10c2adaddc61d73a13cae114f05afe4861c