Jul 16 14:23:44 cs04r-sc-oss01-08 kernel: LustreError: 0:0:(ldlm_lockd.c:313:waiting_locks_callback()) ### lock callback timer expired after 101s: evicting client at 172.23.142.199@tcp  ns: filter-play01-OST0003_UUID lock: ffff81041b944a00/0x8036ad3b0f3d8168 lrc: 3/0,0 mode: PW/PW res: 22839846/0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x80000020 remote: 0xd577a930b0412c96 expref: 306 pid: 8124 timeout 4297209606
Jul 16 14:23:46 cs04r-sc-oss01-08 kernel: LustreError: 4598:0:(filter.c:2577:filter_grant_sanity_check()) filter_statfs: tot_granted 93327360 != fo_tot_granted 118464512
Jul 16 14:23:46 cs04r-sc-oss01-08 kernel: LustreError: 4598:0:(filter.c:2580:filter_grant_sanity_check()) filter_statfs: tot_pending 0 != fo_tot_pending 25137152
Jul 16 14:23:51 cs04r-sc-oss01-08 kernel: LustreError: 4598:0:(filter.c:2577:filter_grant_sanity_check()) filter_statfs: tot_granted 93327360 != fo_tot_granted 118464512
Jul 16 14:23:51 cs04r-sc-oss01-08 kernel: LustreError: 4598:0:(filter.c:2580:filter_grant_sanity_check()) filter_statfs: tot_pending 0 != fo_tot_pending 25137152
Jul 16 14:23:56 cs04r-sc-oss01-08 kernel: LustreError: 4598:0:(filter.c:2577:filter_grant_sanity_check()) filter_statfs: tot_granted 95424512 != fo_tot_granted 120561664
Jul 16 14:23:56 cs04r-sc-oss01-08 kernel: LustreError: 4598:0:(filter.c:2580:filter_grant_sanity_check()) filter_statfs: tot_pending 0 != fo_tot_pending 25137152
Jul 16 14:24:01 cs04r-sc-oss01-08 kernel: LustreError: 4598:0:(filter.c:2577:filter_grant_sanity_check()) filter_statfs: tot_granted 95424512 != fo_tot_granted 120561664
Jul 16 14:24:01 cs04r-sc-oss01-08 kernel: LustreError: 4598:0:(filter.c:2580:filter_grant_sanity_check()) filter_statfs: tot_pending 0 != fo_tot_pending 25137152
Jul 16 14:24:06 cs04r-sc-oss01-08 kernel: LustreError: 4598:0:(filter.c:2577:filter_grant_sanity_check()) filter_statfs: tot_granted 95424512 != fo_tot_granted 120561664
Jul 16 14:24:06 cs04r-sc-oss01-08 kernel: LustreError: 4598:0:(filter.c:2580:filter_grant_sanity_check()) filter_statfs: tot_pending 0 != fo_tot_pending 25137152
Jul 16 14:24:07 cs04r-sc-oss01-08 kernel: Lustre: 8025:0:(ldlm_lib.c:803:target_handle_connect()) play01-OST0003: exp ffff81019b3ac000 already connecting
Jul 16 14:24:07 cs04r-sc-oss01-08 kernel: LustreError: 8025:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-114)  req@ffff8102dc663800 x1407112324567209/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1342445147 ref 1 fl Interpret:/0/0 rc -114/0
Jul 16 14:24:07 cs04r-sc-oss01-08 kernel: LustreError: 8025:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1 previous similar message
Jul 16 14:24:11 cs04r-sc-oss01-08 kernel: LustreError: 4598:0:(filter.c:2577:filter_grant_sanity_check()) filter_statfs: tot_granted 95424512 != fo_tot_granted 120561664
Jul 16 14:24:11 cs04r-sc-oss01-08 kernel: LustreError: 4598:0:(filter.c:2580:filter_grant_sanity_check()) filter_statfs: tot_pending 0 != fo_tot_pending 25137152
Jul 16 14:24:21 cs04r-sc-oss01-08 kernel: LustreError: 4598:0:(filter.c:2577:filter_grant_sanity_check()) filter_statfs: tot_granted 95424512 != fo_tot_granted 120561664
Jul 16 14:24:21 cs04r-sc-oss01-08 kernel: LustreError: 4598:0:(filter.c:2577:filter_grant_sanity_check()) Skipped 1 previous similar message
Jul 16 14:24:21 cs04r-sc-oss01-08 kernel: LustreError: 4598:0:(filter.c:2580:filter_grant_sanity_check()) filter_statfs: tot_pending 0 != fo_tot_pending 25137152
Jul 16 14:24:21 cs04r-sc-oss01-08 kernel: LustreError: 4598:0:(filter.c:2580:filter_grant_sanity_check()) Skipped 1 previous similar message
Jul 16 14:24:23 cs04r-sc-oss01-08 kernel: Lustre: 8112:0:(ldlm_lib.c:803:target_handle_connect()) play01-OST0003: exp ffff81019b3ac000 already connecting
Jul 16 14:24:23 cs04r-sc-oss01-08 kernel: LustreError: 8112:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-114)  req@ffff81024e9f2000 x1407112324567270/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1342445163 ref 1 fl Interpret:/0/0 rc -114/0
Jul 16 14:24:41 cs04r-sc-oss01-08 kernel: Lustre: 8113:0:(ldlm_lib.c:803:target_handle_connect()) play01-OST0003: exp ffff81019b3ac000 already connecting
Jul 16 14:24:41 cs04r-sc-oss01-08 kernel: LustreError: 8113:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-114)  req@ffff810108583400 x1407112324567423/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1342445181 ref 1 fl Interpret:/0/0 rc -114/0
Jul 16 14:24:41 cs04r-sc-oss01-08 kernel: LustreError: 4598:0:(filter.c:2577:filter_grant_sanity_check()) filter_statfs: tot_granted 95424512 != fo_tot_granted 120561664
Jul 16 14:24:41 cs04r-sc-oss01-08 kernel: LustreError: 4598:0:(filter.c:2577:filter_grant_sanity_check()) Skipped 4 previous similar messages
Jul 16 14:24:41 cs04r-sc-oss01-08 kernel: LustreError: 4598:0:(filter.c:2580:filter_grant_sanity_check()) filter_statfs: tot_pending 0 != fo_tot_pending 25137152
Jul 16 14:24:41 cs04r-sc-oss01-08 kernel: LustreError: 4598:0:(filter.c:2580:filter_grant_sanity_check()) Skipped 4 previous similar messages
Jul 16 14:25:01 cs04r-sc-oss01-08 kernel: Lustre: 8262:0:(ldlm_lib.c:803:target_handle_connect()) play01-OST0003: exp ffff81019b3ac000 already connecting
Jul 16 14:25:01 cs04r-sc-oss01-08 kernel: LustreError: 8262:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-114)  req@ffff8103a3aa1400 x1407112324567555/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1342445201 ref 1 fl Interpret:/0/0 rc -114/0
Jul 16 14:25:15 cs04r-sc-oss01-08 kernel: Lustre: Service thread pid 8689 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Jul 16 14:25:15 cs04r-sc-oss01-08 kernel: Pid: 8689, comm: ll_ost_io_297
Jul 16 14:25:15 cs04r-sc-oss01-08 kernel: 
Jul 16 14:25:15 cs04r-sc-oss01-08 kernel: Call Trace:
Jul 16 14:25:15 cs04r-sc-oss01-08 kernel:  [<ffffffff88ba4828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Jul 16 14:25:15 cs04r-sc-oss01-08 kernel:  [<ffffffff800a34a7>] autoremove_wake_function+0x0/0x2e
Jul 16 14:25:15 cs04r-sc-oss01-08 kernel:  [<ffffffff88c5090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Jul 16 14:25:15 cs04r-sc-oss01-08 kernel:  [<ffffffff88c91184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Jul 16 14:25:15 cs04r-sc-oss01-08 kernel:  [<ffffffff80063002>] thread_return+0x62/0xfe
Jul 16 14:25:15 cs04r-sc-oss01-08 kernel:  [<ffffffff88c2be1a>] ost_checksum_bulk+0x2aa/0x5a0 [ost]
Jul 16 14:25:15 cs04r-sc-oss01-08 kernel:  [<ffffffff88c32d09>] ost_brw_write+0x1c99/0x2480 [ost]
Jul 16 14:25:15 cs04r-sc-oss01-08 kernel:  [<ffffffff88969ac8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Jul 16 14:25:15 cs04r-sc-oss01-08 kernel:  [<ffffffff889348b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Jul 16 14:25:15 cs04r-sc-oss01-08 kernel:  [<ffffffff8008ee74>] default_wake_function+0x0/0xe
Jul 16 14:25:15 cs04r-sc-oss01-08 kernel:  [<ffffffff8896e0b8>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Jul 16 14:25:15 cs04r-sc-oss01-08 kernel:  [<ffffffff88c360be>] ost_handle+0x2bce/0x55c0 [ost]
Jul 16 14:25:15 cs04r-sc-oss01-08 kernel:  [<ffffffff888becf0>] class_handle2object+0xe0/0x170 [obdclass]
Jul 16 14:25:15 cs04r-sc-oss01-08 kernel:  [<ffffffff8892819a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Jul 16 14:25:15 cs04r-sc-oss01-08 kernel:  [<ffffffff8892d168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Jul 16 14:25:15 cs04r-sc-oss01-08 kernel:  [<ffffffff8897d6d9>] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc]
Jul 16 14:25:15 cs04r-sc-oss01-08 kernel:  [<ffffffff8897de35>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Jul 16 14:25:15 cs04r-sc-oss01-08 kernel:  [<ffffffff8008d299>] __wake_up_common+0x3e/0x68
Jul 16 14:25:15 cs04r-sc-oss01-08 kernel:  [<ffffffff8897edc6>] ptlrpc_main+0xf66/0x1120 [ptlrpc]
Jul 16 14:25:15 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Jul 16 14:25:15 cs04r-sc-oss01-08 kernel:  [<ffffffff8897de60>] ptlrpc_main+0x0/0x1120 [ptlrpc]
Jul 16 14:25:15 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Jul 16 14:25:15 cs04r-sc-oss01-08 kernel: 
Jul 16 14:25:15 cs04r-sc-oss01-08 kernel: Lustre: Service thread pid 8533 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Jul 16 14:25:15 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445115.8689
Jul 16 14:25:16 cs04r-sc-oss01-08 kernel: Pid: 8533, comm: ll_ost_io_141
Jul 16 14:25:16 cs04r-sc-oss01-08 kernel: 
Jul 16 14:25:16 cs04r-sc-oss01-08 kernel: Call Trace:
Jul 16 14:25:16 cs04r-sc-oss01-08 kernel:  [<ffffffff88bc71fa>] ldiskfs_get_blocks+0xaa/0x210 [ldiskfs]
Jul 16 14:25:16 cs04r-sc-oss01-08 kernel:  [<ffffffff80063c63>] __mutex_lock_slowpath+0x60/0x9b
Jul 16 14:25:16 cs04r-sc-oss01-08 kernel:  [<ffffffff80063cad>] .text.lock.mutex+0xf/0x14
Jul 16 14:25:16 cs04r-sc-oss01-08 kernel:  [<ffffffff88c8fead>] filter_commitrw_write+0xb3d/0x2dd0 [obdfilter]
Jul 16 14:25:16 cs04r-sc-oss01-08 kernel:  [<ffffffff888563a6>] lnet_ni_send+0x96/0xe0 [lnet]
Jul 16 14:25:16 cs04r-sc-oss01-08 kernel:  [<ffffffff88c32d09>] ost_brw_write+0x1c99/0x2480 [ost]
Jul 16 14:25:16 cs04r-sc-oss01-08 kernel:  [<ffffffff88969ac8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Jul 16 14:25:16 cs04r-sc-oss01-08 kernel:  [<ffffffff889348b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Jul 16 14:25:16 cs04r-sc-oss01-08 kernel:  [<ffffffff8008ee74>] default_wake_function+0x0/0xe
Jul 16 14:25:16 cs04r-sc-oss01-08 kernel:  [<ffffffff8896e0b8>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Jul 16 14:25:16 cs04r-sc-oss01-08 kernel:  [<ffffffff88c360be>] ost_handle+0x2bce/0x55c0 [ost]
Jul 16 14:25:16 cs04r-sc-oss01-08 kernel:  [<ffffffff888becf0>] class_handle2object+0xe0/0x170 [obdclass]
Jul 16 14:25:16 cs04r-sc-oss01-08 kernel:  [<ffffffff8892819a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Jul 16 14:25:16 cs04r-sc-oss01-08 kernel:  [<ffffffff8892d168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Jul 16 14:25:16 cs04r-sc-oss01-08 kernel:  [<ffffffff8897d6d9>] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8897de35>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8008d299>] __wake_up_common+0x3e/0x68
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8897edc6>] ptlrpc_main+0xf66/0x1120 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8897de60>] ptlrpc_main+0x0/0x1120 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: 
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: Pid: 8450, comm: ll_ost_io_58
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: 
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: Call Trace:
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff888a9ab8>] lustre_hash_lookup+0x228/0x2b0 [obdclass]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff88b9fd2d>] start_this_handle+0x305/0x3cf [jbd2]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff800a34a7>] autoremove_wake_function+0x0/0x2e
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff88b9fea4>] jbd2_journal_start+0xad/0xe1 [jbd2]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff88c53d8c>] fsfilt_ldiskfs_brw_start+0x35c/0x490 [fsfilt_ldiskfs]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff88c900cb>] filter_commitrw_write+0xd5b/0x2dd0 [obdfilter]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff88c2bee8>] ost_checksum_bulk+0x378/0x5a0 [ost]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff88c32d09>] ost_brw_write+0x1c99/0x2480 [ost]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff88969ac8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff889348b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8008ee74>] default_wake_function+0x0/0xe
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8896e0b8>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff88c360be>] ost_handle+0x2bce/0x55c0 [ost]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff888becf0>] class_handle2object+0xe0/0x170 [obdclass]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8892819a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8007790d>] smp_send_reschedule+0x4e/0x53
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8897d6d9>] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8897de35>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8008d299>] __wake_up_common+0x3e/0x68
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8897edc6>] ptlrpc_main+0xf66/0x1120 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8897de60>] ptlrpc_main+0x0/0x1120 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: 
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: Pid: 8427, comm: ll_ost_io_35
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: 
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: Call Trace:
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff88ba4828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff800a34a7>] autoremove_wake_function+0x0/0x2e
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff88c5090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff88c91184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8005bd83>] cache_alloc_refill+0x108/0x188
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff88c2bead>] ost_checksum_bulk+0x33d/0x5a0 [ost]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff88c32d09>] ost_brw_write+0x1c99/0x2480 [ost]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff88969ac8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff889348b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8008ee74>] default_wake_function+0x0/0xe
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8896e0b8>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff88c360be>] ost_handle+0x2bce/0x55c0 [ost]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff888becf0>] class_handle2object+0xe0/0x170 [obdclass]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8892819a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8892d168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8897d6d9>] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8897de35>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8008d299>] __wake_up_common+0x3e/0x68
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8897edc6>] ptlrpc_main+0xf66/0x1120 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8897de60>] ptlrpc_main+0x0/0x1120 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: 
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: Lustre: Service thread pid 8730 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: Lustre: Skipped 2 previous similar messages
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: Pid: 8730, comm: ll_ost_io_338
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: 
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: Call Trace:
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff88bc71fa>] ldiskfs_get_blocks+0xaa/0x210 [ldiskfs]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff80063c63>] __mutex_lock_slowpath+0x60/0x9b
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff80063cad>] .text.lock.mutex+0xf/0x14
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff88c8fead>] filter_commitrw_write+0xb3d/0x2dd0 [obdfilter]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff88c2beed>] ost_checksum_bulk+0x37d/0x5a0 [ost]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff88c32d09>] ost_brw_write+0x1c99/0x2480 [ost]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff88969ac8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff889348b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8008ee74>] default_wake_function+0x0/0xe
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8896e0b8>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff88c360be>] ost_handle+0x2bce/0x55c0 [ost]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8008f84d>] move_tasks+0x22c/0x2ec
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff888becf0>] class_handle2object+0xe0/0x170 [obdclass]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8892819a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8892d168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8897d6d9>] ptlrpc_server_handle_request+0x989/0xe00 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8897de35>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8008d299>] __wake_up_common+0x3e/0x68
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8897edc6>] ptlrpc_main+0xf66/0x1120 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8897de60>] ptlrpc_main+0x0/0x1120 [ptlrpc]
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: 
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: Lustre: Service thread pid 8896 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: Lustre: Service thread pid 8627 was inactive for 200.52s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445116.8896
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445116.8415
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445116.8687
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445116.8494
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445116.8647
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445117.8627
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445117.8730
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445117.8427
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: 4598:0:(filter.c:2577:filter_grant_sanity_check()) filter_statfs: tot_granted 95424512 != fo_tot_granted 120561664
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: 4598:0:(filter.c:2577:filter_grant_sanity_check()) Skipped 6 previous similar messages
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: 4598:0:(filter.c:2580:filter_grant_sanity_check()) filter_statfs: tot_pending 0 != fo_tot_pending 25137152
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: 4598:0:(filter.c:2580:filter_grant_sanity_check()) Skipped 6 previous similar messages
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445117.8598
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445117.8727
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445117.8801
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445117.8489
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445117.8866
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445117.8808
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445117.8795
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445117.8447
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445117.8774
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445117.8500
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445117.8524
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445117.8733
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445117.8741
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445117.8672
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445117.8778
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445117.8822
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445117.8550
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445117.8481
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445117.8509
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445117.8400
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445117.8505
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445117.8450
Jul 16 14:25:17 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445117.8533
Jul 16 14:25:20 cs04r-sc-oss01-08 kernel: Lustre: Service thread pid 8903 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Jul 16 14:25:20 cs04r-sc-oss01-08 kernel: Lustre: Skipped 25 previous similar messages
Jul 16 14:25:20 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445120.8903
Jul 16 14:25:20 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445120.8820
Jul 16 14:25:20 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445120.8553
Jul 16 14:25:20 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445120.8633
Jul 16 14:25:22 cs04r-sc-oss01-08 kernel: Lustre: Service thread pid 8397 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Jul 16 14:25:22 cs04r-sc-oss01-08 kernel: Lustre: Skipped 3 previous similar messages
Jul 16 14:25:22 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445122.8397
Jul 16 14:25:22 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445122.8764
Jul 16 14:25:22 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445122.8580
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445123.8855
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445123.8493
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel: INFO: task jbd2/dm-5-8:7800 blocked for more than 120 seconds.
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445123.8712
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel: jbd2/dm-5-8   D ffff81035ef62dc0     0  7800    171          7801  7799 (L-TLB)
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  ffff81042b30fd60 0000000000000046 0000002100002e34 ffff810425665820
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  0000000000000001 000000000000000a ffff81023c5167e0 ffff8104263c27a0
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  0000025c44f42292 0000000000000b7f ffff81023c5169c8 000000018008d299
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel: Call Trace:
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff8002dee8>] __wake_up+0x38/0x4f
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff88ba10e0>] :jbd2:jbd2_journal_commit_transaction+0x198/0x1120
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff800a34a7>] autoremove_wake_function+0x0/0x2e
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff8003d7e6>] lock_timer_base+0x1b/0x3c
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff8004ad54>] try_to_del_timer_sync+0x7f/0x88
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff88ba523e>] :jbd2:kjournald2+0x9a/0x1ec
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff800a34a7>] autoremove_wake_function+0x0/0x2e
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff88ba51a4>] :jbd2:kjournald2+0x0/0x1ec
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff800a328f>] keventd_create_kthread+0x0/0xc4
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff80032652>] kthread+0xfe/0x132
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff800a328f>] keventd_create_kthread+0x0/0xc4
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff80032554>] kthread+0x0/0x132
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel: 
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel: INFO: task ll_ost_io_05:8397 blocked for more than 120 seconds.
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel: ll_ost_io_05  D ffff810425504860     0  8397      1          8398  8396 (L-TLB)
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  ffff81042587d810 0000000000000046 ffff8103770e57cc 0000000000000008
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  ffff8104239458b0 000000000000000a ffff81042ab4e7e0 ffff810425504860
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  0000025d6737a131 0000000000004c7d ffff81042ab4e9c8 000000012a302e28
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel: Call Trace:
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff88b9fd2d>] :jbd2:start_this_handle+0x305/0x3cf
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff800a34a7>] autoremove_wake_function+0x0/0x2e
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff88b9fea4>] :jbd2:jbd2_journal_start+0xad/0xe1
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff88c53962>] :fsfilt_ldiskfs:fsfilt_ldiskfs_start+0x4c2/0x590
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff800c931b>] add_to_page_cache_lru+0x1c/0x23
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff80025cbf>] find_or_create_page+0x4b/0x72
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff88c7ce9a>] :obdfilter:filter_setattr_internal+0x59a/0x1d60
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff88c6ccf3>] :obdfilter:filter_fmd_expire_nolock+0x1d3/0x1f0
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff88c70dac>] :obdfilter:filter_fmd_get+0x1bc/0x320
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff8002ff8f>] __up_write+0x27/0xf2
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff88c7e821>] :obdfilter:filter_setattr+0x1c1/0x3b0
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff8896ef64>] :ptlrpc:lustre_msg_add_version+0x34/0x110
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff88971c6a>] :ptlrpc:lustre_pack_reply_flags+0x86a/0x950
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff88969ac8>] :ptlrpc:ptlrpc_send_reply+0x5e8/0x600
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff88c7ebe9>] :obdfilter:filter_truncate+0x1d9/0x270
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff88803378>] :libcfs:cfs_alloc+0x68/0xc0
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff8883a103>] :lvfs:lprocfs_counter_add+0x33/0x100
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff88c2cc78>] :ost:ost_punch+0x9c8/0xce0
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445123.8786
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff8896dff5>] :ptlrpc:lustre_msg_get_version+0x35/0xf0
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff8896df05>] :ptlrpc:lustre_msg_get_opc+0x35/0xf0
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff88c36634>] :ost:ost_handle+0x3144/0x55c0
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff8008f84d>] move_tasks+0x22c/0x2ec
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff8897d6d9>] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff8897de35>] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff8008ee74>] default_wake_function+0x0/0xe
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff8897edc6>] :ptlrpc:ptlrpc_main+0xf66/0x1120
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff8897de60>] :ptlrpc:ptlrpc_main+0x0/0x1120
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel: 
Jul 16 14:25:23 cs04r-sc-oss01-08 kernel: INFO: task ll_ost_io_08:8400 blocked for more than 120 seconds.
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel: ll_ost_io_08  D ffff810429960040     0  8400      1          8401  8399 (L-TLB)
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  ffff81042638f770 0000000000000046 ffff810222460d80 ffffffff888a9ab8
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  ffff81042638f860 000000000000000a ffff81042ab500c0 ffff810429960040
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  0000025c457f59c3 0000000000137e24 ffff81042ab502a8 0000000400000001
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel: Call Trace:
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff888a9ab8>] :obdclass:lustre_hash_lookup+0x228/0x2b0
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88a6ca61>] :lquota:quota_search_lqs+0x51/0x750
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff80047250>] sprintf+0x51/0x59
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff800cb3db>] __rmqueue+0x44/0xc6
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88b9fd2d>] :jbd2:start_this_handle+0x305/0x3cf
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff800a34a7>] autoremove_wake_function+0x0/0x2e
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88b9fea4>] :jbd2:jbd2_journal_start+0xad/0xe1
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88c53d8c>] :fsfilt_ldiskfs:fsfilt_ldiskfs_brw_start+0x35c/0x490
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88a5ed40>] :lquota:filter_quota_acquire+0x0/0x120
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88c900cb>] :obdfilter:filter_commitrw_write+0xd5b/0x2dd0
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88c2be94>] :ost:ost_checksum_bulk+0x324/0x5a0
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88c32d09>] :ost:ost_brw_write+0x1c99/0x2480
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88969ac8>] :ptlrpc:ptlrpc_send_reply+0x5e8/0x600
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff889348b0>] :ptlrpc:target_committed_to_req+0x40/0x120
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff8008ee74>] default_wake_function+0x0/0xe
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff8896e0b8>] :ptlrpc:lustre_msg_check_version_v2+0x8/0x20
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88c360be>] :ost:ost_handle+0x2bce/0x55c0
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff888becf0>] :obdclass:class_handle2object+0xe0/0x170
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff8892819a>] :ptlrpc:lock_res_and_lock+0xba/0xd0
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff8007790d>] smp_send_reschedule+0x4e/0x53
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff8897d6d9>] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff8897de35>] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff8008d299>] __wake_up_common+0x3e/0x68
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff8897edc6>] :ptlrpc:ptlrpc_main+0xf66/0x1120
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff8897de60>] :ptlrpc:ptlrpc_main+0x0/0x1120
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel: 
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel: INFO: task ll_ost_io_15:8407 blocked for more than 120 seconds.
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel: ll_ost_io_15  D ffff81024707a7a0     0  8407      1          8408  8406 (L-TLB)
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  ffff8104263e1810 0000000000000046 ffff8102c6bc585c 0000000000000008
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  ffff8102960e4190 000000000000000a ffff8104263aa820 ffff8102472d7100
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  00000264848b873c 0000000000010990 ffff8104263aaa08 0000000729f704e0
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel: Call Trace:
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88b9fd2d>] :jbd2:start_this_handle+0x305/0x3cf
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff800a34a7>] autoremove_wake_function+0x0/0x2e
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88b9fea4>] :jbd2:jbd2_journal_start+0xad/0xe1
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88c53962>] :fsfilt_ldiskfs:fsfilt_ldiskfs_start+0x4c2/0x590
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff800c931b>] add_to_page_cache_lru+0x1c/0x23
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff80025cbf>] find_or_create_page+0x4b/0x72
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88c7ce9a>] :obdfilter:filter_setattr_internal+0x59a/0x1d60
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88c6ccf3>] :obdfilter:filter_fmd_expire_nolock+0x1d3/0x1f0
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88c70dac>] :obdfilter:filter_fmd_get+0x1bc/0x320
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff8002ff8f>] __up_write+0x27/0xf2
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88c7e821>] :obdfilter:filter_setattr+0x1c1/0x3b0
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff8896ef64>] :ptlrpc:lustre_msg_add_version+0x34/0x110
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88971c6a>] :ptlrpc:lustre_pack_reply_flags+0x86a/0x950
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88969ac8>] :ptlrpc:ptlrpc_send_reply+0x5e8/0x600
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88c7ebe9>] :obdfilter:filter_truncate+0x1d9/0x270
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88803378>] :libcfs:cfs_alloc+0x68/0xc0
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff8883a103>] :lvfs:lprocfs_counter_add+0x33/0x100
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88c2cc78>] :ost:ost_punch+0x9c8/0xce0
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff8896dff5>] :ptlrpc:lustre_msg_get_version+0x35/0xf0
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff8896df05>] :ptlrpc:lustre_msg_get_opc+0x35/0xf0
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88c36634>] :ost:ost_handle+0x3144/0x55c0
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88932b99>] :ptlrpc:ldlm_resource_get+0x1d9/0xa60
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff8007790d>] smp_send_reschedule+0x4e/0x53
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff8897d6d9>] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff8897de35>] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff8008d299>] __wake_up_common+0x3e/0x68
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff8897edc6>] :ptlrpc:ptlrpc_main+0xf66/0x1120
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff8897de60>] :ptlrpc:ptlrpc_main+0x0/0x1120
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel: 
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel: INFO: task ll_ost_io_17:8409 blocked for more than 120 seconds.
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel: ll_ost_io_17  D ffff81042506e820     0  8409      1          8410  8408 (L-TLB)
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  ffff8104263ef810 0000000000000046 ffff81041a9df900 0000000000000008
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  ffff81041a860e50 000000000000000a ffff8104263ac860 ffff81042506e820
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  000002619039c119 000000000000e668 ffff8104263aca48 00000001bd719ba0
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel: Call Trace:
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88b9fd2d>] :jbd2:start_this_handle+0x305/0x3cf
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff800a34a7>] autoremove_wake_function+0x0/0x2e
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88b9fea4>] :jbd2:jbd2_journal_start+0xad/0xe1
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88c53962>] :fsfilt_ldiskfs:fsfilt_ldiskfs_start+0x4c2/0x590
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff800c931b>] add_to_page_cache_lru+0x1c/0x23
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff80025cbf>] find_or_create_page+0x4b/0x72
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88c7ce9a>] :obdfilter:filter_setattr_internal+0x59a/0x1d60
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88c6ccf3>] :obdfilter:filter_fmd_expire_nolock+0x1d3/0x1f0
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff88c70dac>] :obdfilter:filter_fmd_get+0x1bc/0x320
Jul 16 14:25:24 cs04r-sc-oss01-08 kernel:  [<ffffffff8002ff8f>] __up_write+0x27/0xf2
Jul 16 14:25:25 cs04r-sc-oss01-08 kernel:  [<ffffffff88c7e821>] :obdfilter:filter_setattr+0x1c1/0x3b0
Jul 16 14:25:25 cs04r-sc-oss01-08 kernel:  [<ffffffff8896ef64>] :ptlrpc:lustre_msg_add_version+0x34/0x110
Jul 16 14:25:25 cs04r-sc-oss01-08 kernel:  [<ffffffff88971c6a>] :ptlrpc:lustre_pack_reply_flags+0x86a/0x950
Jul 16 14:25:25 cs04r-sc-oss01-08 kernel:  [<ffffffff88969ac8>] :ptlrpc:ptlrpc_send_reply+0x5e8/0x600
Jul 16 14:25:25 cs04r-sc-oss01-08 kernel:  [<ffffffff88c7ebe9>] :obdfilter:filter_truncate+0x1d9/0x270
Jul 16 14:25:25 cs04r-sc-oss01-08 kernel:  [<ffffffff88803378>] :libcfs:cfs_alloc+0x68/0xc0
Jul 16 14:25:25 cs04r-sc-oss01-08 kernel:  [<ffffffff8883a103>] :lvfs:lprocfs_counter_add+0x33/0x100
Jul 16 14:25:25 cs04r-sc-oss01-08 kernel:  [<ffffffff88c2cc78>] :ost:ost_punch+0x9c8/0xce0
Jul 16 14:25:25 cs04r-sc-oss01-08 kernel:  [<ffffffff8896dff5>] :ptlrpc:lustre_msg_get_version+0x35/0xf0
Jul 16 14:25:25 cs04r-sc-oss01-08 kernel:  [<ffffffff8896df05>] :ptlrpc:lustre_msg_get_opc+0x35/0xf0
Jul 16 14:25:25 cs04r-sc-oss01-08 kernel:  [<ffffffff88c36634>] :ost:ost_handle+0x3144/0x55c0
Jul 16 14:25:25 cs04r-sc-oss01-08 kernel:  [<ffffffff88932b99>] :ptlrpc:ldlm_resource_get+0x1d9/0xa60
Jul 16 14:25:25 cs04r-sc-oss01-08 kernel:  [<ffffffff8897d6d9>] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00
Jul 16 14:25:25 cs04r-sc-oss01-08 kernel:  [<ffffffff8897de35>] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310
Jul 16 14:25:25 cs04r-sc-oss01-08 kernel:  [<ffffffff8008d299>] __wake_up_common+0x3e/0x68
Jul 16 14:25:25 cs04r-sc-oss01-08 kernel:  [<ffffffff8897edc6>] :ptlrpc:ptlrpc_main+0xf66/0x1120
Jul 16 14:25:25 cs04r-sc-oss01-08 kernel: Lustre: Service thread pid 8703 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Jul 16 14:25:25 cs04r-sc-oss01-08 kernel: Lustre: Skipped 6 previous similar messages
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445125.8603
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8897de60>] :ptlrpc:ptlrpc_main+0x0/0x1120
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel: 
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel: INFO: task ll_ost_io_23:8415 blocked for more than 120 seconds.
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel: ll_ost_io_23  D ffff810247069aa0     0  8415      1          8416  8414 (L-TLB)
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  ffff810425997770 0000000000000046 ffff810425997810 00000003888a9ab8
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  ffff81024706af20 000000000000000a ffff8104263b4820 ffff8102471ec100
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  0000025c49247788 0000000000001c56 ffff8104263b4a08 0000000300000000
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel: Call Trace:
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88b9fd2d>] :jbd2:start_this_handle+0x305/0x3cf
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff800a34a7>] autoremove_wake_function+0x0/0x2e
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88b9fea4>] :jbd2:jbd2_journal_start+0xad/0xe1
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88c53d8c>] :fsfilt_ldiskfs:fsfilt_ldiskfs_brw_start+0x35c/0x490
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88c900cb>] :obdfilter:filter_commitrw_write+0xd5b/0x2dd0
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff888563a6>] :lnet:lnet_ni_send+0x96/0xe0
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88c2be1a>] :ost:ost_checksum_bulk+0x2aa/0x5a0
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88c32d09>] :ost:ost_brw_write+0x1c99/0x2480
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88969ac8>] :ptlrpc:ptlrpc_send_reply+0x5e8/0x600
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff889348b0>] :ptlrpc:target_committed_to_req+0x40/0x120
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8008ee74>] default_wake_function+0x0/0xe
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8896e0b8>] :ptlrpc:lustre_msg_check_version_v2+0x8/0x20
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88c360be>] :ost:ost_handle+0x2bce/0x55c0
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8008f84d>] move_tasks+0x22c/0x2ec
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8897d6d9>] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8897de35>] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8008ee74>] default_wake_function+0x0/0xe
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8897edc6>] :ptlrpc:ptlrpc_main+0xf66/0x1120
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8897de60>] :ptlrpc:ptlrpc_main+0x0/0x1120
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel: 
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel: INFO: task ll_ost_io_35:8427 blocked for more than 120 seconds.
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel: ll_ost_io_35  D ffffffff8006272c     0  8427      1          8428  8426 (L-TLB)
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  ffff8104259eb810 0000000000000046 ffff81041ddfe9c0 ffff810400afba40
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  ffff810423ceaf40 000000000000000a ffff8104263c27a0 ffff810425225820
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  0000025c4556f84e 00000000000046d6 ffff8104263c2988 00000005ffffffff
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel: Call Trace:
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88ba4828>] :jbd2:jbd2_log_wait_commit+0xa3/0xf5
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff800a34a7>] autoremove_wake_function+0x0/0x2e
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88c5090b>] :fsfilt_ldiskfs:fsfilt_ldiskfs_commit_wait+0xab/0xd0
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88c91184>] :obdfilter:filter_commitrw_write+0x1e14/0x2dd0
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8005bd83>] cache_alloc_refill+0x108/0x188
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88c2bead>] :ost:ost_checksum_bulk+0x33d/0x5a0
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88c32d09>] :ost:ost_brw_write+0x1c99/0x2480
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88969ac8>] :ptlrpc:ptlrpc_send_reply+0x5e8/0x600
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff889348b0>] :ptlrpc:target_committed_to_req+0x40/0x120
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8008ee74>] default_wake_function+0x0/0xe
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8896e0b8>] :ptlrpc:lustre_msg_check_version_v2+0x8/0x20
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88c360be>] :ost:ost_handle+0x2bce/0x55c0
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff888becf0>] :obdclass:class_handle2object+0xe0/0x170
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8892819a>] :ptlrpc:lock_res_and_lock+0xba/0xd0
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8892d168>] :ptlrpc:__ldlm_handle2lock+0x2f8/0x360
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8897d6d9>] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8897de35>] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8008d299>] __wake_up_common+0x3e/0x68
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8897edc6>] :ptlrpc:ptlrpc_main+0xf66/0x1120
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8897de60>] :ptlrpc:ptlrpc_main+0x0/0x1120
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel: 
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel: INFO: task ll_ost_io_42:8434 blocked for more than 120 seconds.
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel: ll_ost_io_42  D ffff810001004420     0  8434      1          8435  8433 (L-TLB)
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  ffff810425a37810 0000000000000046 0000000000000000 ffffffff88c714d2
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  3232383339383535 000000000000000a ffff810425a08100 ffffffff80319b60
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  0000025f548e609c 0000000000008bed ffff810425a082e8 00000000889fae2f
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel: Call Trace:
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88c714d2>] :obdfilter:filter_fid2dentry+0x512/0x740
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88b9fd2d>] :jbd2:start_this_handle+0x305/0x3cf
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff800a34a7>] autoremove_wake_function+0x0/0x2e
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88b9fea4>] :jbd2:jbd2_journal_start+0xad/0xe1
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88c53962>] :fsfilt_ldiskfs:fsfilt_ldiskfs_start+0x4c2/0x590
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff80025c96>] find_or_create_page+0x22/0x72
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88c7ce9a>] :obdfilter:filter_setattr_internal+0x59a/0x1d60
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88c6ccf3>] :obdfilter:filter_fmd_expire_nolock+0x1d3/0x1f0
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88c70dac>] :obdfilter:filter_fmd_get+0x1bc/0x320
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8002ff8f>] __up_write+0x27/0xf2
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88c7e821>] :obdfilter:filter_setattr+0x1c1/0x3b0
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8896ef64>] :ptlrpc:lustre_msg_add_version+0x34/0x110
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88971c6a>] :ptlrpc:lustre_pack_reply_flags+0x86a/0x950
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88969ac8>] :ptlrpc:ptlrpc_send_reply+0x5e8/0x600
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88c7ebe9>] :obdfilter:filter_truncate+0x1d9/0x270
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88803378>] :libcfs:cfs_alloc+0x68/0xc0
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8883a103>] :lvfs:lprocfs_counter_add+0x33/0x100
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88c2cc78>] :ost:ost_punch+0x9c8/0xce0
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8896dff5>] :ptlrpc:lustre_msg_get_version+0x35/0xf0
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8896df05>] :ptlrpc:lustre_msg_get_opc+0x35/0xf0
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88c36634>] :ost:ost_handle+0x3144/0x55c0
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88932b99>] :ptlrpc:ldlm_resource_get+0x1d9/0xa60
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8007790d>] smp_send_reschedule+0x4e/0x53
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8897d6d9>] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8897de35>] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8008d299>] __wake_up_common+0x3e/0x68
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8897edc6>] :ptlrpc:ptlrpc_main+0xf66/0x1120
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8897de60>] :ptlrpc:ptlrpc_main+0x0/0x1120
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel: 
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel: INFO: task ll_ost_io_44:8436 blocked for more than 120 seconds.
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel: ll_ost_io_44  D ffff810001015120     0  8436      1          8437  8435 (L-TLB)
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  ffff810425a4d810 0000000000000046 ffff810107e3c108 ffffffff80063947
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  3232383339383931 000000000000000a ffff810425a0a040 ffff810247202040
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  00000265feb3130a 0000000000009ee7 ffff810425a0a228 00000004889fae2f
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel: Call Trace:
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff80063947>] __wait_on_bit_lock+0x5b/0x66
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88b9fd2d>] :jbd2:start_this_handle+0x305/0x3cf
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff800a34a7>] autoremove_wake_function+0x0/0x2e
Jul 16 14:25:26 cs04r-sc-oss01-08 kernel:  [<ffffffff88b9fea4>] :jbd2:jbd2_journal_start+0xad/0xe1
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff88c53962>] :fsfilt_ldiskfs:fsfilt_ldiskfs_start+0x4c2/0x590
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff80025c96>] find_or_create_page+0x22/0x72
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff88c7ce9a>] :obdfilter:filter_setattr_internal+0x59a/0x1d60
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff88c6ccf3>] :obdfilter:filter_fmd_expire_nolock+0x1d3/0x1f0
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff88c70dac>] :obdfilter:filter_fmd_get+0x1bc/0x320
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff8002ff8f>] __up_write+0x27/0xf2
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff88c7e821>] :obdfilter:filter_setattr+0x1c1/0x3b0
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff8896ef64>] :ptlrpc:lustre_msg_add_version+0x34/0x110
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff88971c6a>] :ptlrpc:lustre_pack_reply_flags+0x86a/0x950
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff88969ac8>] :ptlrpc:ptlrpc_send_reply+0x5e8/0x600
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff88c7ebe9>] :obdfilter:filter_truncate+0x1d9/0x270
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff88803378>] :libcfs:cfs_alloc+0x68/0xc0
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff8883a103>] :lvfs:lprocfs_counter_add+0x33/0x100
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff88c2cc78>] :ost:ost_punch+0x9c8/0xce0
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff8896dff5>] :ptlrpc:lustre_msg_get_version+0x35/0xf0
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff8896df05>] :ptlrpc:lustre_msg_get_opc+0x35/0xf0
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff88c36634>] :ost:ost_handle+0x3144/0x55c0
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff88932b99>] :ptlrpc:ldlm_resource_get+0x1d9/0xa60
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff8008e66d>] dequeue_task+0x18/0x37
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff8007790d>] smp_send_reschedule+0x4e/0x53
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff8897d6d9>] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff8897de35>] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff8008d299>] __wake_up_common+0x3e/0x68
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff8897edc6>] :ptlrpc:ptlrpc_main+0xf66/0x1120
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff8897de60>] :ptlrpc:ptlrpc_main+0x0/0x1120
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel: 
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel: INFO: task ll_ost_io_55:8447 blocked for more than 120 seconds.
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel: ll_ost_io_55  D ffff810247069aa0     0  8447      1          8448  8446 (L-TLB)
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  ffff810425a95850 0000000000000046 0000000000000282 ffffffff88bc71fa
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  00000000000000d8 000000000000000a ffff810425a18820 ffff8102471ec100
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  0000025c44c4cff5 000000000000c720 ffff810425a18a08 0000000388a5d648
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel: Call Trace:
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff88bc71fa>] :ldiskfs:ldiskfs_get_blocks+0xaa/0x210
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff80063c63>] __mutex_lock_slowpath+0x60/0x9b
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff80063cad>] .text.lock.mutex+0xf/0x14
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff88c8fead>] :obdfilter:filter_commitrw_write+0xb3d/0x2dd0
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff88c2be4d>] :ost:ost_checksum_bulk+0x2dd/0x5a0
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff88c32d09>] :ost:ost_brw_write+0x1c99/0x2480
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff88969ac8>] :ptlrpc:ptlrpc_send_reply+0x5e8/0x600
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff889348b0>] :ptlrpc:target_committed_to_req+0x40/0x120
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff8008ee74>] default_wake_function+0x0/0xe
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff8896e0b8>] :ptlrpc:lustre_msg_check_version_v2+0x8/0x20
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff88c360be>] :ost:ost_handle+0x2bce/0x55c0
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff888becf0>] :obdclass:class_handle2object+0xe0/0x170
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff8892819a>] :ptlrpc:lock_res_and_lock+0xba/0xd0
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff8008e66d>] dequeue_task+0x18/0x37
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff8897d6d9>] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff8897de35>] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff8008ee74>] default_wake_function+0x0/0xe
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff8897edc6>] :ptlrpc:ptlrpc_main+0xf66/0x1120
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff8897de60>] :ptlrpc:ptlrpc_main+0x0/0x1120
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel: 
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel: Lustre: 8101:0:(ldlm_lib.c:803:target_handle_connect()) play01-OST0003: exp ffff81019b3ac000 already connecting
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel: LustreError: 8101:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-114)  req@ffff81036f34c400 x1407112324567661/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1342445225 ref 1 fl Interpret:/0/0 rc -114/0
Jul 16 14:25:27 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445126.8703
Jul 16 14:25:28 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445128.8719
Jul 16 14:25:28 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445128.8747
Jul 16 14:25:29 cs04r-sc-oss01-08 kernel: Lustre: Service thread pid 8784 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Jul 16 14:25:29 cs04r-sc-oss01-08 kernel: Lustre: Skipped 3 previous similar messages
Jul 16 14:25:29 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445129.8784
Jul 16 14:25:29 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445129.8690
Jul 16 14:25:30 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445130.8839
Jul 16 14:25:30 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445130.8538
Jul 16 14:25:30 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445130.8434
Jul 16 14:25:35 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445135.8635
Jul 16 14:25:36 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445136.8517
Jul 16 14:25:39 cs04r-sc-oss01-08 kernel: Lustre: Service thread pid 8409 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Jul 16 14:25:39 cs04r-sc-oss01-08 kernel: Lustre: Skipped 6 previous similar messages
Jul 16 14:25:40 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445140.8409
Jul 16 14:25:41 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445141.8831
Jul 16 14:25:43 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445143.8555
Jul 16 14:25:43 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445143.8832
Jul 16 14:25:44 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445144.8669
Jul 16 14:25:44 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445144.8781
Jul 16 14:25:45 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445145.8898
Jul 16 14:25:46 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445146.8706
Jul 16 14:25:46 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445146.8566
Jul 16 14:25:46 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445146.8495
Jul 16 14:25:47 cs04r-sc-oss01-08 kernel: Lustre: 7936:0:(ldlm_lib.c:803:target_handle_connect()) play01-OST0003: exp ffff81019b3ac000 already connecting
Jul 16 14:25:47 cs04r-sc-oss01-08 kernel: LustreError: 7936:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-114)  req@ffff81022085fc50 x1407112324567761/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1342445247 ref 1 fl Interpret:/0/0 rc -114/0
Jul 16 14:25:47 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445147.8894
Jul 16 14:25:49 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445149.8845
Jul 16 14:25:49 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445149.8724
Jul 16 14:25:49 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445149.8883
Jul 16 14:25:51 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445151.8901
Jul 16 14:25:52 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445152.8407
Jul 16 14:25:54 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445154.8544
Jul 16 14:25:54 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445154.8807
Jul 16 14:25:54 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445154.8600
Jul 16 14:25:54 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445154.8661
Jul 16 14:25:54 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445154.8487
Jul 16 14:25:54 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445154.8708
Jul 16 14:25:55 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445155.8682
Jul 16 14:25:55 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445155.8475
Jul 16 14:25:57 cs04r-sc-oss01-08 kernel: Lustre: Service thread pid 8479 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Jul 16 14:25:57 cs04r-sc-oss01-08 kernel: Lustre: Skipped 23 previous similar messages
Jul 16 14:25:57 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445157.8479
Jul 16 14:25:58 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445158.8862
Jul 16 14:25:58 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445158.8671
Jul 16 14:25:58 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445158.8877
Jul 16 14:25:58 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445158.8560
Jul 16 14:25:58 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445158.8639
Jul 16 14:25:58 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445158.8821
Jul 16 14:25:58 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445158.8648
Jul 16 14:25:58 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445158.8766
Jul 16 14:25:59 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445159.8436
Jul 16 14:25:59 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445159.8584
Jul 16 14:26:00 cs04r-sc-oss01-08 kernel: LustreError: dumping log to /tmp/lustre-log.1342445160.8823
Jul 16 14:26:13 cs04r-sc-oss01-08 kernel: Lustre: 8145:0:(ldlm_lib.c:803:target_handle_connect()) play01-OST0003: exp ffff81019b3ac000 already connecting
Jul 16 14:26:13 cs04r-sc-oss01-08 kernel: LustreError: 8145:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-114)  req@ffff8102fb7fb400 x1407112324567861/t0 o8-><?>@<?>:0/0 lens 368/264 e 0 to 0 dl 1342445273 ref 1 fl Interpret:/0/0 rc -114/0
Jul 16 14:26:22 cs04r-sc-oss01-08 kernel: LustreError: 4598:0:(filter.c:2577:filter_grant_sanity_check()) filter_statfs: tot_granted 95424512 != fo_tot_granted 120561664
Jul 16 14:26:22 cs04r-sc-oss01-08 kernel: LustreError: 4598:0:(filter.c:2577:filter_grant_sanity_check()) Skipped 12 previous similar messages
Jul 16 14:26:22 cs04r-sc-oss01-08 kernel: LustreError: 4598:0:(filter.c:2580:filter_grant_sanity_check()) filter_statfs: tot_pending 0 != fo_tot_pending 25137152
Jul 16 14:26:22 cs04r-sc-oss01-08 kernel: LustreError: 4598:0:(filter.c:2580:filter_grant_sanity_check()) Skipped 12 previous similar messages
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0003: slow i_mutex 273s due to heavy IO load
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0003: slow journal start 155s due to heavy IO load
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: Skipped 2 previous similar messages
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: 8224:0:(service.c:1434:ptlrpc_server_handle_request()) @@@ Request x1407112324567028 took longer than estimated (100+56s); client may timeout.  req@ffff810283acac00 x1407112324567028/t0 o8->7189d78c-6463-a455-0a8c-a61a7453c968@NET_0x20000ac178ec7_UUID:0/0 lens 368/264 e 0 to 0 dl 1342445133 ref 1 fl Complete:/0/0 rc 0/0
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0003: slow journal start 271s due to heavy IO load
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0003: slow brw_start 271s due to heavy IO load
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0003: slow journal start 271s due to heavy IO load
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0003: slow brw_start 271s due to heavy IO load
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0003: slow journal start 260s due to heavy IO load
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: Skipped 48 previous similar messages
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0003: slow direct_io 271s due to heavy IO load
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0003: slow journal start 271s due to heavy IO load
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0003: slow commitrw commit 271s due to heavy IO load
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: 8801:0:(service.c:1434:ptlrpc_server_handle_request()) @@@ Request x1407112324560528 took longer than estimated (212+60s); client may timeout.  req@ffff8104086fd800 x1407112324560528/t141733932601 o4->7189d78c-6463-a455-0a8c-a61a7453c968@NET_0x20000ac178ec7_UUID:0/0 lens 448/416 e 2 to 0 dl 1342445129 ref 1 fl Complete:/0/0 rc 0/0
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: Service thread pid 8801 completed after 272.07s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0003: slow start_page_read 74s due to heavy IO load
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0003: slow start_page_read 137s due to heavy IO load
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: Skipped 2 previous similar messages
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: Skipped 2 previous similar messages
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0003: slow journal start 272s due to heavy IO load
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: Skipped 2 previous similar messages
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0003: slow commitrw commit 272s due to heavy IO load
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: Skipped 2 previous similar messages
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: Service thread pid 8427 completed after 273.82s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: Skipped 3 previous similar messages
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: 8786:0:(service.c:1434:ptlrpc_server_handle_request()) @@@ Request x1407112324561649 took longer than estimated (212+54s); client may timeout.  req@ffff81042aa91850 x1407112324561649/t141733932606 o10->7189d78c-6463-a455-0a8c-a61a7453c968@NET_0x20000ac178ec7_UUID:0/0 lens 400/400 e 2 to 0 dl 1342445135 ref 1 fl Complete:H/0/0 rc 0/0
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: 8786:0:(service.c:1434:ptlrpc_server_handle_request()) Skipped 4 previous similar messages
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0003: slow i_mutex 273s due to heavy IO load
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0003: slow journal start 231s due to heavy IO load
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: Service thread pid 8450 completed after 274.32s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: Skipped 53 previous similar messages
Jul 16 14:26:29 cs04r-sc-oss01-08 kernel: Lustre: Skipped 5 previous similar messages
Jul 16 14:26:30 cs04r-sc-oss01-08 kernel: Lustre: play01-OST0003: slow start_page_read 111s due to heavy IO load
Jul 16 14:26:30 cs04r-sc-oss01-08 kernel: Lustre: Skipped 1 previous similar message
Jul 16 14:26:41 cs04r-sc-oss01-08 kernel: Lustre: 7959:0:(ldlm_lib.c:574:target_handle_reconnect()) play01-OST0003: 7189d78c-6463-a455-0a8c-a61a7453c968 reconnecting
Jul 16 14:34:29 cs04r-sc-oss01-08 kernel: Lustre: 8072:0:(ldlm_lib.c:574:target_handle_reconnect()) play01-OST0004: 69d87395-1fce-0c9e-2505-0acc7fd7164e reconnecting
Jul 16 14:34:29 cs04r-sc-oss01-08 kernel: Lustre: 8072:0:(ldlm_lib.c:874:target_handle_connect()) play01-OST0004: refuse reconnection from 69d87395-1fce-0c9e-2505-0acc7fd7164e@172.23.138.5@tcp to 0xffff810424b3d200; still busy with 16 active RPCs
Jul 16 14:34:29 cs04r-sc-oss01-08 kernel: LustreError: 8072:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16)  req@ffff81014c1a5800 x1405109994215362/t0 o8->69d87395-1fce-0c9e-2505-0acc7fd7164e@NET_0x20000ac178a05_UUID:0/0 lens 368/264 e 0 to 0 dl 1342445769 ref 1 fl Interpret:/0/0 rc -16/0
Jul 16 14:34:30 cs04r-sc-oss01-08 kernel: LustreError: 8548:0:(ost_handler.c:829:ost_brw_read()) @@@ Reconnect on bulk PUT  req@ffff8100b8389400 x1405109994215112/t0 o3->69d87395-1fce-0c9e-2505-0acc7fd7164e@NET_0x20000ac178a05_UUID:0/0 lens 448/400 e 0 to 0 dl 1342446003 ref 1 fl Interpret:/0/0 rc 0/0
Jul 16 14:34:30 cs04r-sc-oss01-08 kernel: LustreError: 8518:0:(ost_handler.c:829:ost_brw_read()) @@@ Reconnect on bulk PUT  req@ffff810222430400 x1405109994215248/t0 o3->69d87395-1fce-0c9e-2505-0acc7fd7164e@NET_0x20000ac178a05_UUID:0/0 lens 448/400 e 0 to 0 dl 1342446003 ref 1 fl Interpret:/0/0 rc 0/0
Jul 16 14:34:30 cs04r-sc-oss01-08 kernel: LustreError: 8518:0:(ost_handler.c:829:ost_brw_read()) Skipped 2 previous similar messages
Jul 16 14:34:35 cs04r-sc-oss01-08 kernel: Lustre: 8389:0:(ldlm_lib.c:574:target_handle_reconnect()) play01-OST0000: 69d87395-1fce-0c9e-2505-0acc7fd7164e reconnecting
Jul 16 14:34:35 cs04r-sc-oss01-08 kernel: Lustre: 8389:0:(ldlm_lib.c:874:target_handle_connect()) play01-OST0001: refuse reconnection from 69d87395-1fce-0c9e-2505-0acc7fd7164e@172.23.138.5@tcp to 0xffff810225442200; still busy with 2 active RPCs
Jul 16 14:34:35 cs04r-sc-oss01-08 kernel: Lustre: 8389:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 1 previous similar message
Jul 16 14:34:36 cs04r-sc-oss01-08 kernel: LustreError: 8823:0:(ost_handler.c:829:ost_brw_read()) @@@ Reconnect on bulk PUT  req@ffff810234b45800 x1405109994215245/t0 o3->69d87395-1fce-0c9e-2505-0acc7fd7164e@NET_0x20000ac178a05_UUID:0/0 lens 448/400 e 0 to 0 dl 1342446003 ref 1 fl Interpret:/0/0 rc 0/0
Jul 16 14:34:36 cs04r-sc-oss01-08 kernel: LustreError: 8823:0:(ost_handler.c:829:ost_brw_read()) Skipped 13 previous similar messages
Jul 16 14:34:41 cs04r-sc-oss01-08 kernel: Lustre: 8548:0:(ost_handler.c:887:ost_brw_read()) play01-OST0004: ignoring bulk IO comm error with 69d87395-1fce-0c9e-2505-0acc7fd7164e@NET_0x20000ac178a05_UUID id 12345-172.23.138.5@tcp - client will retry
Jul 16 14:34:41 cs04r-sc-oss01-08 kernel: Lustre: 8548:0:(ost_handler.c:887:ost_brw_read()) Skipped 25 previous similar messages
Jul 16 14:34:41 cs04r-sc-oss01-08 kernel: Lustre: 8832:0:(ost_handler.c:887:ost_brw_read()) play01-OST0004: ignoring bulk IO comm error with 69d87395-1fce-0c9e-2505-0acc7fd7164e@NET_0x20000ac178a05_UUID id 12345-172.23.138.5@tcp - client will retry
Jul 16 14:34:41 cs04r-sc-oss01-08 kernel: Lustre: 8832:0:(ost_handler.c:887:ost_brw_read()) Skipped 25 previous similar messages
Jul 16 14:34:42 cs04r-sc-oss01-08 kernel: Lustre: 7967:0:(ldlm_lib.c:574:target_handle_reconnect()) play01-OST0001: 69d87395-1fce-0c9e-2505-0acc7fd7164e reconnecting
Jul 16 14:34:42 cs04r-sc-oss01-08 kernel: Lustre: 7967:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 4 previous similar messages
Jul 16 14:34:47 cs04r-sc-oss01-08 kernel: Lustre: 8321:0:(ldlm_lib.c:574:target_handle_reconnect()) play01-OST0005: 69d87395-1fce-0c9e-2505-0acc7fd7164e reconnecting
Jul 16 14:35:00 cs04r-sc-oss01-08 kernel: Lustre: 8185:0:(ldlm_lib.c:574:target_handle_reconnect()) play01-OST0004: 69d87395-1fce-0c9e-2505-0acc7fd7164e reconnecting
Jul 16 14:36:28 cs04r-sc-oss01-08 kernel: Lustre: 8190:0:(ldlm_lib.c:574:target_handle_reconnect()) play01-OST0001: 69d87395-1fce-0c9e-2505-0acc7fd7164e reconnecting
Jul 16 14:36:28 cs04r-sc-oss01-08 kernel: Lustre: 8190:0:(ldlm_lib.c:874:target_handle_connect()) play01-OST0001: refuse reconnection from 69d87395-1fce-0c9e-2505-0acc7fd7164e@172.23.138.5@tcp to 0xffff810225442200; still busy with 3 active RPCs
Jul 16 14:36:28 cs04r-sc-oss01-08 kernel: LustreError: 8190:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16)  req@ffff81013e947400 x1405109994264411/t0 o8->69d87395-1fce-0c9e-2505-0acc7fd7164e@NET_0x20000ac178a05_UUID:0/0 lens 368/264 e 0 to 0 dl 1342445888 ref 1 fl Interpret:/0/0 rc -16/0
Jul 16 14:36:28 cs04r-sc-oss01-08 kernel: LustreError: 8190:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 2 previous similar messages
Jul 16 14:36:29 cs04r-sc-oss01-08 kernel: LustreError: 8582:0:(ost_handler.c:829:ost_brw_read()) @@@ Reconnect on bulk PUT  req@ffff81042ac2ec50 x1405109994264283/t0 o3->69d87395-1fce-0c9e-2505-0acc7fd7164e@NET_0x20000ac178a05_UUID:0/0 lens 448/400 e 0 to 0 dl 1342445815 ref 1 fl Interpret:/0/0 rc 0/0
Jul 16 14:36:29 cs04r-sc-oss01-08 kernel: LustreError: 8582:0:(ost_handler.c:829:ost_brw_read()) Skipped 16 previous similar messages
Jul 16 14:36:40 cs04r-sc-oss01-08 kernel: Lustre: 8037:0:(ldlm_lib.c:874:target_handle_connect()) play01-OST0005: refuse reconnection from 69d87395-1fce-0c9e-2505-0acc7fd7164e@172.23.138.5@tcp to 0xffff81022be1b200; still busy with 4 active RPCs
Jul 16 14:36:41 cs04r-sc-oss01-08 kernel: LustreError: 8623:0:(ost_handler.c:829:ost_brw_read()) @@@ Reconnect on bulk PUT  req@ffff8101ee4e4c00 x1405109994264281/t0 o3->69d87395-1fce-0c9e-2505-0acc7fd7164e@NET_0x20000ac178a05_UUID:0/0 lens 448/400 e 0 to 0 dl 1342445815 ref 1 fl Interpret:/0/0 rc 0/0
Jul 16 14:36:41 cs04r-sc-oss01-08 kernel: Lustre: 8646:0:(ost_handler.c:887:ost_brw_read()) play01-OST0005: ignoring bulk IO comm error with 69d87395-1fce-0c9e-2505-0acc7fd7164e@NET_0x20000ac178a05_UUID id 12345-172.23.138.5@tcp - client will retry
Jul 16 14:36:41 cs04r-sc-oss01-08 kernel: LustreError: 8623:0:(ost_handler.c:829:ost_brw_read()) Skipped 2 previous similar messages
Jul 16 14:36:52 cs04r-sc-oss01-08 kernel: Lustre: 8331:0:(ldlm_lib.c:574:target_handle_reconnect()) play01-OST0001: 69d87395-1fce-0c9e-2505-0acc7fd7164e reconnecting
Jul 16 14:36:52 cs04r-sc-oss01-08 kernel: Lustre: 8331:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 2 previous similar messages
