[LU-488] LustreError: 4564:0:(connection.c:100:ptlrpc_connection_put()) ASSERTION(!cfs_hlist_unhashed(&conn->c_hash)) failed

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.2.0, Lustre 2.1.1
• Affects Version/s: Lustre 2.1.0
• Environment: RHEL6/CHAOS5, Lustre 2.0.63-1chaos
• Severity: 3
• Assignee: Lai Siyao
• Reporter: Prakash Surya (Inactive)

Description

This LBUG was hit on an OSS node running Lustre version 2.0.63-1chaos with an additional patch adding some printk statements.

The command that triggered the assertion:

$ /etc/init.d/lustre stop
Unmounting /mnt/lustre/local/lc2-OST0000
Unmounting /mnt/lustre/local/lc2-OST0004
Unmounting /mnt/lustre/local/lc2-OST0008
Unmounting /mnt/lustre/local/lc2-OST000c
Unmounting /mnt/lustre/local/lc2-OST0010
Unmounting /mnt/lustre/local/lc2-OST0014
Unmounting /mnt/lustre/local/lc2-OST0018
Unmounting /mnt/lustre/local/lc2-OST001c
Unmounting /mnt/lustre/local/lc2-OST0020
Unmounting /mnt/lustre/local/lc2-OST0024
Unmounting /mnt/lustre/local/lc2-OST0028

Console output:

2011-07-01 15:37:41 LustreError: 4564:0:(connection.c:100:ptlrpc_connection_put()) ASSERTION(!cfs_hlist_unhashed(&conn->c_hash)) failed
2011-07-01 15:37:41 LustreError: 4564:0:(connection.c:100:ptlrpc_connection_put()) LBUG
2011-07-01 15:37:41 Pid: 4564, comm: ll_ost_110
2011-07-01 15:37:41 Jul  1 15:37:40 
2011-07-01 15:37:41 zwicky47 kernel:Call Trace:
2011-07-01 15:37:41  LustreError: 45 [<ffffffffa03218b5>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
2011-07-01 15:37:41 64:0:(connection [<ffffffffa0321ef5>] lbug_with_loc+0x75/0xe0 [libcfs]
2011-07-01 15:37:41 .c:100:ptlrpc_co [<ffffffffa032ccf6>] libcfs_assertion_failed+0x66/0x70 [libcfs]
2011-07-01 15:37:41 nnection_put())  [<ffffffffa04c3f3e>] ptlrpc_connection_put+0x22e/0x240 [ptlrpc]
2011-07-01 15:37:41 ASSERTION(!cfs_h [<ffffffffa04cd87c>] ? lustre_msg_set_timeout+0x9c/0x110 [ptlrpc]
2011-07-01 15:37:41 list_unhashed(&c [<ffffffffa04c4cd6>] ptlrpc_send_reply+0x286/0x6f0 [ptlrpc]
2011-07-01 15:37:41 onn->c_hash)) fa [<ffffffff812627f9>] ? cpumask_next_and+0x29/0x50
2011-07-01 15:37:41 iled
2011-07-01 15:37:41 Jul  1 15:3 [<ffffffffa049640a>] target_send_reply_msg+0x4a/0x160 [ptlrpc]
2011-07-01 15:37:41 7:40 zwicky47 ke [<ffffffffa04cca5c>] ? lustre_msg_get_version+0x7c/0xe0 [ptlrpc]
2011-07-01 15:37:41 rnel: LustreErro [<ffffffffa04965de>] target_send_reply+0xbe/0x600 [ptlrpc]
2011-07-01 15:37:41 r: 4564:0:(conne [<ffffffffa04caabc>] ? lustre_msg_set_versions+0x9c/0x130 [ptlrpc]
2011-07-01 15:37:41 ction.c:100:ptlr [<ffffffffa089c625>] ? oti_to_request+0x75/0xc0 [ost]
2011-07-01 15:37:41 pc_connection_pu [<ffffffffa08a5639>] ost_handle+0x279/0x4b90 [ost]
2011-07-01 15:37:41 t()) LBUG
2011-07-01 15:37:41  [<ffffffff8105dc32>] ? default_wake_function+0x12/0x20
2011-07-01 15:37:41  [<ffffffff8104aee9>] ? __wake_up_common+0x59/0x90
2011-07-01 15:37:41  [<ffffffffa04ca7f4>] ? lustre_msg_get_opc+0x94/0x100 [ptlrpc]
2011-07-01 15:37:41  [<ffffffffa04db3ce>] ptlrpc_main+0xb8e/0x1900 [ptlrpc]
2011-07-01 15:37:41  [<ffffffff8100c1ca>] child_rip+0xa/0x20
2011-07-01 15:37:41  [<ffffffffa04da840>] ? ptlrpc_main+0x0/0x1900 [ptlrpc]
2011-07-01 15:37:41  [<ffffffff8100c1c0>] ? child_rip+0x0/0x20
2011-07-01 15:37:41 
2011-07-01 15:37:41 Kernel panic - not syncing: LBUG
2011-07-01 15:37:41 Pid: 4564, comm: ll_ost_110 Tainted: P           ---------------- T 2.6.32-131.2.1.1chaos.ch5.x86_64 #1
2011-07-01 15:37:41 Jul  1 15:37:40 Call Trace:
2011-07-01 15:37:41 zwicky47 kernel: [<ffffffff814da4fe>] ? panic+0x78/0x143
2011-07-01 15:37:41  Kernel panic -  [<ffffffffa0321f4b>] ? lbug_with_loc+0xcb/0xe0 [libcfs]
2011-07-01 15:37:41 not syncing: LBU [<ffffffffa032ccf6>] ? libcfs_assertion_failed+0x66/0x70 [libcfs]
2011-07-01 15:37:41 G
2011-07-01 15:37:41  [<ffffffffa04c3f3e>] ? ptlrpc_connection_put+0x22e/0x240 [ptlrpc]
2011-07-01 15:37:41  [<ffffffffa04cd87c>] ? lustre_msg_set_timeout+0x9c/0x110 [ptlrpc]
2011-07-01 15:37:41  [<ffffffffa04c4cd6>] ? ptlrpc_send_reply+0x286/0x6f0 [ptlrpc]
2011-07-01 15:37:41  [<ffffffff812627f9>] ? cpumask_next_and+0x29/0x50
2011-07-01 15:37:41  [<ffffffffa049640a>] ? target_send_reply_msg+0x4a/0x160 [ptlrpc]
2011-07-01 15:37:41  [<ffffffffa04cca5c>] ? lustre_msg_get_version+0x7c/0xe0 [ptlrpc]
2011-07-01 15:37:41  [<ffffffffa04965de>] ? target_send_reply+0xbe/0x600 [ptlrpc]
2011-07-01 15:37:41  [<ffffffffa04caabc>] ? lustre_msg_set_versions+0x9c/0x130 [ptlrpc]
2011-07-01 15:37:41  [<ffffffffa089c625>] ? oti_to_request+0x75/0xc0 [ost]
2011-07-01 15:37:41  [<ffffffffa08a5639>] ? ost_handle+0x279/0x4b90 [ost]
2011-07-01 15:37:41  [<ffffffff8105dc32>] ? default_wake_function+0x12/0x20
2011-07-01 15:37:41  [<ffffffff8104aee9>] ? __wake_up_common+0x59/0x90
2011-07-01 15:37:41  [<ffffffffa04ca7f4>] ? lustre_msg_get_opc+0x94/0x100 [ptlrpc]
2011-07-01 15:37:41  [<ffffffffa04db3ce>] ? ptlrpc_main+0xb8e/0x1900 [ptlrpc]
2011-07-01 15:37:41  [<ffffffff8100c1ca>] ? child_rip+0xa/0x20
2011-07-01 15:37:41  [<ffffffffa04da840>] ? ptlrpc_main+0x0/0x1900 [ptlrpc]
2011-07-01 15:37:41  [<ffffffff8100c1c0>] ? child_rip+0x0/0x20

Issue Links

• Trackbacks: Changelog 2.1 Changes from version 2.1.0 to version 2.1.1 Server support for kernels: 2.6.18274.12.1.el5 (RHEL5) 2.6.32220.el6 (RHEL6) Client support for unpatched kernels: 2.6.18274.12.1.el5 (RHEL5) 2.6.32220.el6 (RHEL6) 2.6.32.360....
• Trackbacks: Changelog 2.2 version 2.2.0 Support for networks: o2iblnd OFED 1.5.4 Server support for kernels: 2.6.32220.4.2.el6 (RHEL6) Client support for unpatched kernels: 2.6.18274.18.1.el5 (RHEL5) 2.6.32220.4.2.el6 (RHEL6) 2.6.32.360....