[LU-14752] LustreError: 3594004:0:(lu_object.c:2472:lu_object_assign_fid()) ASSERTION( rc == 0 ) failed: failed hashtable insertion: rc = -16

• Type: Bug
• Resolution: Fixed
• Priority: Major
• Fix Version/s: Lustre 2.15.0
• Affects Version/s: Lustre 2.14.0
• Labels: llnll
• Environment: lustre-2.14.0_2.llnl-2.t4.x86_64 ; zfs-2.0.52_2llnl-2.t4.x86_6 ; kernel 4.18.0-240.22.1.1toss.t4.x86_64
• Severity: 3
• Assignee: Lai Siyao
• Reporter: Gian-Carlo Defazio

Description

While running a performance test that uses mdtest, the error

LustreError: 3594004:0:(lu_object.c:2472:lu_object_assign_fid()) LBUG

 has been see several times. Here's an example:

2021-05-21 15:07:06 [1990667.397573] LustreError: 3594004:0:(lu_object.c:2472:lu_object_assign_fid()) ASSERTION( rc == 0 ) failed: failed hashtable insertion: rc = -16
2021-05-21 15:07:06 [1990667.410521] LustreError: 3594004:0:(lu_object.c:2472:lu_object_assign_fid()) LBUG
2021-05-21 15:07:06 [1990667.418250] Pid: 3594004, comm: mdt01_020 4.18.0-240.22.1.1toss.t4.x86_64 #1 SMP Tue Apr 13 17:18:40 PDT 2021
2021-05-21 15:07:06 [1990667.428331] Call Trace TBD:
2021-05-21 15:07:06 [1990667.431303] Kernel panic - not syncing: LBUG
2021-05-21 15:07:06 [1990667.435744] CPU: 49 PID: 3594004 Comm: mdt01_020 Kdump: loaded Tainted: P        W  OE    --------- -  - 4.18.0-240.22.1.1toss.t4.x86_64 #1
2021-05-21 15:07:06 [1990667.448421] Hardware name: Supermicro SSG-2029P-DN2R24N-LL013/X11DSN-TS, BIOS 3.4 11/04/2020
2021-05-21 15:07:06 [1990667.457027] Call Trace:
2021-05-21 15:07:06 [1990667.459662]  dump_stack+0x5c/0x80
2021-05-21 15:07:06 [1990667.463150]  panic+0xe7/0x2a9
2021-05-21 15:07:06 [1990667.466306]  lbug_with_loc.cold.10+0x18/0x18 [libcfs]
2021-05-21 15:07:06 [1990667.471566]  lu_object_assign_fid+0x3b8/0x3c0 [obdclass]
2021-05-21 15:07:06 [1990667.477056]  osp_create+0x471/0xba0 [osp]
2021-05-21 15:07:06 [1990667.481257]  ? lod_sub_create+0x244/0x4a0 [lod]
2021-05-21 15:07:06 [1990667.485963]  lod_sub_create+0x244/0x4a0 [lod]
2021-05-21 15:07:06 [1990667.490497]  lod_striped_create+0x174/0x4c0 [lod]
2021-05-21 15:07:06 [1990667.495377]  lod_create+0x218/0x330 [lod]
2021-05-21 15:07:06 [1990667.499559]  ? spl_kmem_alloc+0xd5/0x130 [spl]
2021-05-21 15:07:06 [1990667.504186]  mdd_create_object_internal+0xa9/0x300 [mdd]
2021-05-21 15:07:06 [1990667.509673]  mdd_create_object+0x92/0x870 [mdd]
2021-05-21 15:07:06 [1990667.514380]  mdd_create+0xd45/0x1740 [mdd]
2021-05-21 15:07:06 [1990667.518653]  ? mdd_links_rename+0x440/0x440 [mdd]
2021-05-21 15:07:06 [1990667.523549]  mdt_reint_open+0x225e/0x2f20 [mdt]
2021-05-21 15:07:06 [1990667.528276]  ? check_unlink_entry+0x14/0xb0 [obdclass]
2021-05-21 15:07:06 [1990667.533589]  ? mdt_root_squash+0x1e/0x3e0 [mdt]
2021-05-21 15:07:06 [1990667.538307]  mdt_reint_rec+0x11f/0x250 [mdt]
2021-05-21 15:07:06 [1990667.542759]  mdt_reint_internal+0x498/0x780 [mdt]
2021-05-21 15:07:06 [1990667.547649]  mdt_intent_open+0x124/0x3c0 [mdt]
2021-05-21 15:07:06 [1990667.552272]  ? cfs_hash_bd_from_key.isra.41+0x36/0xb0 [libcfs]
2021-05-21 15:07:06 [1990667.558279]  ? mdt_intent_fixup_resent+0x1f0/0x1f0 [mdt]
2021-05-21 15:07:06 [1990667.563766]  mdt_intent_opc+0x44d/0xa80 [mdt]
2021-05-21 15:07:06 [1990667.568300]  mdt_intent_policy+0x1f6/0x380 [mdt]
2021-05-21 15:07:06 [1990667.573153]  ldlm_lock_enqueue+0x4c1/0x9f0 [ptlrpc]
2021-05-21 15:07:06 [1990667.578235]  ldlm_handle_enqueue0+0x61a/0x16d0 [ptlrpc]
2021-05-21 15:07:06 [1990667.583659]  tgt_enqueue+0xa4/0x1f0 [ptlrpc]
2021-05-21 15:07:06 [1990667.588131]  tgt_request_handle+0xc8b/0x1930 [ptlrpc]
2021-05-21 15:07:06 [1990667.593381]  ? ptlrpc_nrs_req_get_nolock0+0xfb/0x1f0 [ptlrpc]
2021-05-21 15:07:06 [1990667.599328]  ptlrpc_server_handle_request+0x31a/0xba0 [ptlrpc]
2021-05-21 15:07:06 [1990667.605358]  ptlrpc_main+0xba9/0x14a0 [ptlrpc]
2021-05-21 15:07:06 [1990667.609974]  ? __schedule+0x2cf/0x720
2021-05-21 15:07:06 [1990667.613833]  ? ptlrpc_register_service+0xfc0/0xfc0 [ptlrpc]
2021-05-21 15:07:06 [1990667.619574]  kthread+0x11d/0x140
2021-05-21 15:07:06 [1990667.622977]  ? kthread_flush_work_fn+0x10/0x10
2021-05-21 15:07:06 [1990667.627588]  ret_from_fork+0x1f/0x40 

The version of lustre used is at https://github.com/llnl/lustre/tree/2.14.0_2.llnl

 

Issue Links

• is related to: LU-8130 Migrate from libcfs hash to rhashtable