[LU-4499] NRS ORR cfs_hash_find_or_add() LBUG

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.8.0
• Affects Version/s: Lustre 2.5.0, Lustre 2.6.0, Lustre 2.8.0
• Labels: cea
• Environment: Lustre-2.5.52 (server), Lustre-2.5.53(Client)
• Severity: 3
• Assignee: Jinshan Xiong (Inactive)
• Reporter: Shuichi Ihara (Inactive)

Description

Hit LBUG and crach on OSS during IOR SSF(Single shared file) test with striping setting on all OSTs(lfs setstripe -c -1).

<0>LustreError: 28757:0:(hash.c:1252:cfs_hash_find_or_add()) ASSERTION( hlist_unhashed(hnode) ) failed: 
<0>LustreError: 28757:0:(hash.c:1252:cfs_hash_find_or_add()) LBUG
<0>Kernel panic - not syncing: LBUG in interrupt.
<0>
<4>Pid: 28757, comm: ll_ost01_008 Not tainted 2.6.32-358.23.2.el6_lustre.ge975b1c.x86_64 #1
<4>Call Trace:
<4> [<ffffffff8150deec>] ? panic+0xa7/0x16f
<4> [<ffffffffa065aedd>] ? lbug_with_loc+0x8d/0xb0 [libcfs]
<4> [<ffffffffa0672d80>] ? cfs_hash_findadd_unique+0x0/0x30 [libcfs]
<4> [<ffffffffa0672d98>] ? cfs_hash_findadd_unique+0x18/0x30 [libcfs]
<4> [<ffffffffa0c83c76>] ? nrs_orr_res_get+0x696/0xb90 [ptlrpc]
<4> [<ffffffff81055ad3>] ? __wake_up+0x53/0x70
<4> [<ffffffffa0c79e36>] ? nrs_resource_get+0x56/0x110 [ptlrpc]
<4> [<ffffffffa0c37d95>] ? lustre_msg_buf+0x55/0x60 [ptlrpc]
<4> [<ffffffffa0c7a7fb>] ? nrs_resource_get_safe+0x8b/0x100 [ptlrpc]
<4> [<ffffffffa0c7ce38>] ? ptlrpc_nrs_req_hp_move+0x68/0x210 [ptlrpc]
<4> [<ffffffffa0c5f845>] ? req_capsule_client_get+0x15/0x20 [ptlrpc]
<4> [<ffffffffa0c1a158>] ? ldlm_server_blocking_ast+0x228/0x880 [ptlrpc]
<4> [<ffffffffa0c8f65b>] ? tgt_blocking_ast+0x7b/0x5e0 [ptlrpc]
<4> [<ffffffffa0beb1ba>] ? ldlm_add_bl_work_item+0x8a/0x1e0 [ptlrpc]
<4> [<ffffffffa0bee405>] ? ldlm_add_ast_work_item+0x55/0x180 [ptlrpc]
<4> [<ffffffffa0bed38d>] ? ldlm_work_bl_ast_lock+0xdd/0x290 [ptlrpc]
<4> [<ffffffffa0c2e3bc>] ? ptlrpc_set_wait+0x6c/0x860 [ptlrpc]
<4> [<ffffffff811685ac>] ? __kmalloc+0x20c/0x220
<4> [<ffffffffa0c2b06a>] ? ptlrpc_prep_set+0xfa/0x2f0 [ptlrpc]
<4> [<ffffffffa0bed2b0>] ? ldlm_work_bl_ast_lock+0x0/0x290 [ptlrpc]
<4> [<ffffffffa0bf006b>] ? ldlm_run_ast_work+0x1bb/0x470 [ptlrpc]
<4> [<ffffffffa0c070ad>] ? ldlm_process_extent_lock+0x13d/0xa90 [ptlrpc]
<4> [<ffffffffa0bef5ab>] ? ldlm_lock_enqueue+0x3fb/0x920 [ptlrpc]
<4> [<ffffffffa0c18c4f>] ? ldlm_handle_enqueue0+0x4ef/0x10a0 [ptlrpc]
<4> [<ffffffffa0c92562>] ? tgt_enqueue+0x62/0x1d0 [ptlrpc]
<4> [<ffffffffa0c94f5a>] ? tgt_handle_request0+0x2ea/0x1490 [ptlrpc]
<4> [<ffffffffa065b4ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
<4> [<ffffffffa066c3af>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
<4> [<ffffffffa0c3792c>] ? lustre_msg_get_opc+0x9c/0x110 [ptlrpc]
<4> [<ffffffffa0c9653a>] ? tgt_request_handle+0x43a/0x980 [ptlrpc]
<4> [<ffffffffa0c4a295>] ? ptlrpc_main+0xd25/0x1970 [ptlrpc]
<4> [<ffffffff810096f0>] ? __switch_to+0xd0/0x320
<4> [<ffffffff8150e600>] ? thread_return+0x4e/0x76e
<4> [<ffffffffa0c49570>] ? ptlrpc_main+0x0/0x1970 [ptlrpc]
<4> [<ffffffff81096a36>] ? kthread+0x96/0xa0
<4> [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
<4> [<ffffffff810969a0>] ? kthread+0x0/0xa0
<4> [<ffffffff8100c0c0>] ? child_rip+0x0/0x20

Issue Links

• is duplicated by: LU-6633 sanityn test_77c: ASSERTION( hlist_unhashed(hnode) ) failed:
• is related to: LU-7084 sanityn test_77c: cfs_hash_find_or_add()) ASSERTION( hlist_unhashed(hnode) ) failed
• is related to: LU-6688 sanityn 77a and 77b fail to set the NRS policy