[LU-547] 1.8<->2.1 / 2.3<->2.4 interop: dbench kept doing cleanup

• Type: Bug
• Resolution: Won't Fix
• Priority: Major
• Fix Version/s: Lustre 2.1.0, Lustre 1.8.7
• Affects Version/s: Lustre 2.1.0, Lustre 2.4.0, Lustre 1.8.6

Environment

Lustre Clients:
Tag: 1.8.6-wc1
Distro/Arch: RHEL6/x86_64 (kernel version: 2.6.32_131.2.1.el6)
Build: http://newbuild.whamcloud.com/job/lustre-b1_8/100/arch=x86_64,build_type=client,distro=el6,ib_stack=inkernel/
Network: IB (inkernel OFED)
ENABLE_QUOTA=yes

Lustre Servers:
Tag: v2_0_66_0
Distro/Arch: RHEL6/x86_64 (kernel version: 2.6.32-131.2.1.el6_lustre)
Build: http://newbuild.whamcloud.com/job/lustre-master/228/arch=x86_64,build_type=server,distro=el6,ib_stack=inkernel/
Network: IB (inkernel OFED)

• Severity: 3
• Assignee: Oleg Drokin
• Reporter: Jian Yu

Description

While running sanity-benchmark test suite, the dbench test kept doing cleanup and could not stop:

<~snip~>
   1    737819     5.51 MB/sec  execute 598 sec  latency 38819.862 ms
   1    737819     5.50 MB/sec  execute 599 sec  latency 39820.034 ms
   1  cleanup 600 sec
   1  cleanup 601 sec
   ......
   1  cleanup 2897 sec
   1  cleanup 2898 sec
   ......

On client node fat-amd-3-ib:

# ps auxww
<~snip~>
500      18628  0.0  0.0 107264  2552 pts/0    S+   00:50   0:00 bash rundbench -D /mnt/lustre/d0.fat-amd-3-ib 1
500      18636  0.0  0.0   6092   596 pts/0    S+   00:50   0:00 dbench -c client.txt 1
500      18637  2.2  0.0   8140  2540 pts/0    S+   00:50   2:00 dbench -c client.txt 1

# echo t > /proc/sysrq-trigger
<~snip~>
dbench        S 0000000000000009     0 18636  18628 0x00000080
 ffff880219239e78 0000000000000086 0000000000000000 ffffffff812688c5
 ffff880323c50f98 ffff880218872068 0000000000000000 0000000100b91979
 ffff880218af3a78 ffff880219239fd8 000000000000f598 ffff880218af3a78
Call Trace:
 [<ffffffff812688c5>] ? rb_insert_color+0x125/0x160
 [<ffffffff8106b325>] do_wait+0x1c5/0x250
 [<ffffffff8106b453>] sys_wait4+0xa3/0x100
 [<ffffffff810699b0>] ? child_wait_callback+0x0/0x70
 [<ffffffff8100b425>] ? int_check_syscall_exit_work+0x34/0x3d
 [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b
dbench        S 000000000000000c     0 18637  18636 0x00000080
 ffff880419933a98 0000000000000082 0000000000000000 000000004e31176c
 ffff880419933a78 ffffffffa04a2cac ffff880419933a78 ffff8803191cb000
 ffff8804197b05f8 ffff880419933fd8 000000000000f598 ffff8804197b05f8
Call Trace:
 [<ffffffffa04a2cac>] ? ptlrpc_at_adj_net_latency+0x7c/0x230 [ptlrpc]
 [<ffffffffa049c00d>] ldlm_flock_completion_ast+0x61d/0x9f0 [ptlrpc]
 [<ffffffffa046c15f>] ? lock_res_and_lock+0x5f/0xe0 [ptlrpc]
 [<ffffffff8105dc20>] ? default_wake_function+0x0/0x20
 [<ffffffffa048a565>] ldlm_cli_enqueue_fini+0x6c5/0xba0 [ptlrpc]
 [<ffffffff8105dc20>] ? default_wake_function+0x0/0x20
 [<ffffffffa048e074>] ldlm_cli_enqueue+0x344/0x7a0 [ptlrpc]
 [<ffffffffa0677edd>] ll_file_flock+0x47d/0x6b0 [lustre]
 [<ffffffffa049b9f0>] ? ldlm_flock_completion_ast+0x0/0x9f0 [ptlrpc]
 [<ffffffff811bc243>] vfs_lock_file+0x23/0x40
 [<ffffffff811bc497>] fcntl_setlk+0x177/0x320
 [<ffffffff8107ff06>] ? group_send_sig_info+0x26/0x70
 [<ffffffff81080212>] ? sys_kill+0x92/0x190
 [<ffffffff811845f7>] sys_fcntl+0x197/0x530
 [<ffffffff8100b172>] system_call_fastpath+0x16/0x1b

Maloo report: https://maloo.whamcloud.com/test_sets/4f04bb24-b8f5-11e0-8bdf-52540025f9af

Please find the attached dbench-debug_log.tar.bz2 for debug logs.