Sep 11 02:26:13 cs04r-sc-mds03-01 kernel: INFO: task mdt_rdpg00_000:12042 blocked for more than 120 seconds.
Sep 11 02:26:13 cs04r-sc-mds03-01 kernel:      Not tainted 2.6.32-431.17.1.el6_lustre.x86_64 #1
Sep 11 02:26:13 cs04r-sc-mds03-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 11 02:26:13 cs04r-sc-mds03-01 kernel: mdt_rdpg00_00 D 0000000000000014     0 12042      2 0x00000080
Sep 11 02:26:13 cs04r-sc-mds03-01 kernel: ffff881e413358b0 0000000000000046 ffffffffffffffff ffff880f9d78e600
Sep 11 02:26:13 cs04r-sc-mds03-01 kernel: ffff881e413358c0 ffffffffa176331f ffff8810ffffffff 0000000000bb2a30
Sep 11 02:26:13 cs04r-sc-mds03-01 kernel: ffff881fd20e5ab8 ffff881e41335fd8 000000000000fbc8 ffff881fd20e5ab8
Sep 11 02:26:13 cs04r-sc-mds03-01 kernel: Call Trace:
Sep 11 02:26:13 cs04r-sc-mds03-01 kernel: [<ffffffffa176331f>] ? qsd_op_begin+0x5f/0xb40 [lquota]
Sep 11 02:26:13 cs04r-sc-mds03-01 kernel: [<ffffffff815297ce>] __mutex_lock_slowpath+0x13e/0x180
Sep 11 02:26:13 cs04r-sc-mds03-01 kernel: [<ffffffff8152966b>] mutex_lock+0x2b/0x50
Sep 11 02:26:13 cs04r-sc-mds03-01 kernel: [<ffffffffa013f688>] __jbd2_log_wait_for_space+0xc8/0x1b0 [jbd2]
Sep 11 02:26:13 cs04r-sc-mds03-01 kernel: [<ffffffffa013b01d>] start_this_handle+0x10d/0x4a0 [jbd2]
Sep 11 02:26:13 cs04r-sc-mds03-01 kernel: [<ffffffffa17d5232>] ? osd_declare_write+0x2a2/0x500 [osd_ldiskfs]
Sep 11 02:26:13 cs04r-sc-mds03-01 kernel: [<ffffffffa013b5b0>] jbd2_journal_start+0xd0/0x110 [jbd2]
Sep 11 02:26:13 cs04r-sc-mds03-01 kernel: [<ffffffffa18cabe2>] ? mdt_txn_start_cb+0xe2/0x290 [mdt]
Sep 11 02:26:13 cs04r-sc-mds03-01 kernel: [<ffffffffa16fe916>] ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs]
Sep 11 02:26:13 cs04r-sc-mds03-01 kernel: [<ffffffffa17b661f>] osd_trans_start+0x1df/0x660 [osd_ldiskfs]
Sep 11 02:26:13 cs04r-sc-mds03-01 kernel: [<ffffffffa197c619>] lod_trans_start+0x1b9/0x250 [lod]
Sep 11 02:26:13 cs04r-sc-mds03-01 kernel: [<ffffffffa19f2387>] mdd_trans_start+0x17/0x20 [mdd]
Sep 11 02:26:13 cs04r-sc-mds03-01 kernel: [<ffffffffa19d8633>] mdd_attr_set+0x4a3/0x1470 [mdd]
Sep 11 02:26:13 cs04r-sc-mds03-01 kernel: [<ffffffffa18ceddc>] mdt_mfd_close+0x7ac/0x1bc0 [mdt]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa123ad65>] ? lustre_msg_buf+0x55/0x60 [ptlrpc]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa1261d26>] ? __req_capsule_get+0x166/0x710 [ptlrpc]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa123bd7e>] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa1090125>] ? class_handle2object+0x95/0x190 [obdclass]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa18d1582>] mdt_close+0x642/0xa80 [mdt]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa18a658a>] mdt_handle_common+0x52a/0x1470 [mdt]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa18e30c5>] mds_readpage_handle+0x15/0x20 [mdt]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa124bbc5>] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa0f9b4ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa0fac3cf>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa12432a9>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff810546b9>] ? __wake_up_common+0x59/0x90
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa124cf2d>] ptlrpc_main+0xaed/0x1740 [ptlrpc]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa124c440>] ? ptlrpc_main+0x0/0x1740 [ptlrpc]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff8109ab56>] kthread+0x96/0xa0
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff8109aac0>] ? kthread+0x0/0xa0
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: INFO: task mdt_rdpg01_001:12045 blocked for more than 120 seconds.
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel:      Not tainted 2.6.32-431.17.1.el6_lustre.x86_64 #1
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: mdt_rdpg01_00 D 000000000000001e     0 12045      2 0x00000080
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: ffff881e4133b8b0 0000000000000046 0000000000000000 ffff880f9b94f9c0
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: ffff881e4133b8c0 ffffffffa176331f ffff8809ffffffff 000000000026ad94
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: ffff8820282dc638 ffff881e4133bfd8 000000000000fbc8 ffff8820282dc638
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: Call Trace:
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa176331f>] ? qsd_op_begin+0x5f/0xb40 [lquota]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff815297ce>] __mutex_lock_slowpath+0x13e/0x180
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff8152966b>] mutex_lock+0x2b/0x50
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa013f688>] __jbd2_log_wait_for_space+0xc8/0x1b0 [jbd2]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa013b01d>] start_this_handle+0x10d/0x4a0 [jbd2]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa17d5232>] ? osd_declare_write+0x2a2/0x500 [osd_ldiskfs]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa013b5b0>] jbd2_journal_start+0xd0/0x110 [jbd2]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa18cabe2>] ? mdt_txn_start_cb+0xe2/0x290 [mdt]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa16fe916>] ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa17b661f>] osd_trans_start+0x1df/0x660 [osd_ldiskfs]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa197c619>] lod_trans_start+0x1b9/0x250 [lod]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa19f2387>] mdd_trans_start+0x17/0x20 [mdd]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa19d8633>] mdd_attr_set+0x4a3/0x1470 [mdd]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa18ceddc>] mdt_mfd_close+0x7ac/0x1bc0 [mdt]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa123ad65>] ? lustre_msg_buf+0x55/0x60 [ptlrpc]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa1261d26>] ? __req_capsule_get+0x166/0x710 [ptlrpc]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa123bd7e>] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa1090125>] ? class_handle2object+0x95/0x190 [obdclass]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa18d1582>] mdt_close+0x642/0xa80 [mdt]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa18a658a>] mdt_handle_common+0x52a/0x1470 [mdt]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa18e30c5>] mds_readpage_handle+0x15/0x20 [mdt]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa124bbc5>] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa0f9b4ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa0fac3cf>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa12432a9>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff810546b9>] ? __wake_up_common+0x59/0x90
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa124cf2d>] ptlrpc_main+0xaed/0x1740 [ptlrpc]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa124c440>] ? ptlrpc_main+0x0/0x1740 [ptlrpc]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff8109ab56>] kthread+0x96/0xa0
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff8109aac0>] ? kthread+0x0/0xa0
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: INFO: task mdt_rdpg03_001:12049 blocked for more than 120 seconds.
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel:      Not tainted 2.6.32-431.17.1.el6_lustre.x86_64 #1
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: mdt_rdpg03_00 D 0000000000000019     0 12049      2 0x00000080
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: ffff88200913d8b0 0000000000000046 0000000000000000 ffff881f98420a40
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: ffff88200913d8c0 ffffffffa176331f ffff881effffffff 0000000000a5e101
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: ffff8820091225f8 ffff88200913dfd8 000000000000fbc8 ffff8820091225f8
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: Call Trace:
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa176331f>] ? qsd_op_begin+0x5f/0xb40 [lquota]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff815297ce>] __mutex_lock_slowpath+0x13e/0x180
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff8152966b>] mutex_lock+0x2b/0x50
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa013f688>] __jbd2_log_wait_for_space+0xc8/0x1b0 [jbd2]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa013b01d>] start_this_handle+0x10d/0x4a0 [jbd2]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa17d5232>] ? osd_declare_write+0x2a2/0x500 [osd_ldiskfs]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa013b5b0>] jbd2_journal_start+0xd0/0x110 [jbd2]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa18cabe2>] ? mdt_txn_start_cb+0xe2/0x290 [mdt]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa16fe916>] ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa17b661f>] osd_trans_start+0x1df/0x660 [osd_ldiskfs]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa197c619>] lod_trans_start+0x1b9/0x250 [lod]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa19f2387>] mdd_trans_start+0x17/0x20 [mdd]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa19d8633>] mdd_attr_set+0x4a3/0x1470 [mdd]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa18ceddc>] mdt_mfd_close+0x7ac/0x1bc0 [mdt]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa123ad65>] ? lustre_msg_buf+0x55/0x60 [ptlrpc]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa1261d26>] ? __req_capsule_get+0x166/0x710 [ptlrpc]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa123bd7e>] ? lustre_pack_reply_flags+0xae/0x1f0 [ptlrpc]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa1090125>] ? class_handle2object+0x95/0x190 [obdclass]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa18d1582>] mdt_close+0x642/0xa80 [mdt]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa18a658a>] mdt_handle_common+0x52a/0x1470 [mdt]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa18e30c5>] mds_readpage_handle+0x15/0x20 [mdt]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa124bbc5>] ptlrpc_server_handle_request+0x385/0xc00 [ptlrpc]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa0f9b4ce>] ? cfs_timer_arm+0xe/0x10 [libcfs]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa0fac3cf>] ? lc_watchdog_touch+0x6f/0x170 [libcfs]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa12432a9>] ? ptlrpc_wait_event+0xa9/0x2d0 [ptlrpc]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff810546b9>] ? __wake_up_common+0x59/0x90
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa124cf2d>] ptlrpc_main+0xaed/0x1740 [ptlrpc]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa124c440>] ? ptlrpc_main+0x0/0x1740 [ptlrpc]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff8109ab56>] kthread+0x96/0xa0
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff8109aac0>] ? kthread+0x0/0xa0
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: INFO: task osp-syn-3-0:12076 blocked for more than 120 seconds.
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel:      Not tainted 2.6.32-431.17.1.el6_lustre.x86_64 #1
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: osp-syn-3-0   D 0000000000000000     0 12076      2 0x00000080
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: ffff881fe9f9b5c0 0000000000000046 6235393566323030 6666666638383134
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: ffffffffa10f2e72 ffff880731e26438 ffff880ca4c4e468 00000000a21db7a7
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: ffff88201a75baf8 ffff881fe9f9bfd8 000000000000fbc8 ffff88201a75baf8
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: Call Trace:
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff815297ce>] __mutex_lock_slowpath+0x13e/0x180
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff8152966b>] mutex_lock+0x2b/0x50
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa013f688>] __jbd2_log_wait_for_space+0xc8/0x1b0 [jbd2]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa17e71f6>] ? osd_declare_qid+0xd6/0x3f0 [osd_ldiskfs]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa013b01d>] start_this_handle+0x10d/0x4a0 [jbd2]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa17e76b2>] ? osd_declare_inode_qid+0x1a2/0x270 [osd_ldiskfs]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa17d5232>] ? osd_declare_write+0x2a2/0x500 [osd_ldiskfs]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa013b5b0>] jbd2_journal_start+0xd0/0x110 [jbd2]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa16fe916>] ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa17b661f>] osd_trans_start+0x1df/0x660 [osd_ldiskfs]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa106ab1c>] llog_write+0x22c/0x420 [obdclass]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff81060aa3>] ? perf_event_task_sched_out+0x33/0x70
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa106adfc>] llog_cancel_rec+0xbc/0x7c0 [obdclass]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa10710a7>] llog_cat_cancel_records+0x107/0x340 [obdclass]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa1a3cb91>] osp_sync_process_committed+0x231/0x770 [osp]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa1a3d164>] osp_sync_process_queues+0x94/0x1610 [osp]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff81061d00>] ? default_wake_function+0x0/0x20
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa106bd77>] llog_process_thread+0x877/0xcf0 [obdclass]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa1a3d0d0>] ? osp_sync_process_queues+0x0/0x1610 [osp]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa106d9d7>] llog_process_or_fork+0x127/0x550 [obdclass]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa1070eea>] llog_cat_process_cb+0x55a/0x610 [obdclass]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa106bd77>] llog_process_thread+0x877/0xcf0 [obdclass]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa1070990>] ? llog_cat_process_cb+0x0/0x610 [obdclass]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa106d9d7>] llog_process_or_fork+0x127/0x550 [obdclass]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa106ee79>] llog_cat_process_or_fork+0x89/0x350 [obdclass]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff81061d12>] ? default_wake_function+0x12/0x20
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff810546b9>] ? __wake_up_common+0x59/0x90
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa1a3d0d0>] ? osp_sync_process_queues+0x0/0x1610 [osp]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa106f159>] llog_cat_process+0x19/0x20 [obdclass]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa1a3ecf3>] osp_sync_thread+0x243/0x7d0 [osp]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff81527de6>] ? schedule+0x176/0x3b0
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa1a3eab0>] ? osp_sync_thread+0x0/0x7d0 [osp]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff8109ab56>] kthread+0x96/0xa0
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff8109aac0>] ? kthread+0x0/0xa0
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: INFO: task osp-syn-25-0:12078 blocked for more than 120 seconds.
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel:      Not tainted 2.6.32-431.17.1.el6_lustre.x86_64 #1
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: osp-syn-25-0  D 0000000000000000     0 12078      2 0x00000080
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: ffff881fe9f9f5c0 0000000000000046 0000000000000000 6666666638383065
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: ffffffffa10f2e72 ffff880212b0a0b8 ffff880515861798 000000001f75e7d6
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: ffff881fed185af8 ffff881fe9f9ffd8 000000000000fbc8 ffff881fed185af8
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: Call Trace:
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff815297ce>] __mutex_lock_slowpath+0x13e/0x180
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff8152966b>] mutex_lock+0x2b/0x50
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa013f688>] __jbd2_log_wait_for_space+0xc8/0x1b0 [jbd2]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa17e71f6>] ? osd_declare_qid+0xd6/0x3f0 [osd_ldiskfs]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa013b01d>] start_this_handle+0x10d/0x4a0 [jbd2]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa17e76b2>] ? osd_declare_inode_qid+0x1a2/0x270 [osd_ldiskfs]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa17d5232>] ? osd_declare_write+0x2a2/0x500 [osd_ldiskfs]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa013b5b0>] jbd2_journal_start+0xd0/0x110 [jbd2]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa16fe916>] ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa17b661f>] osd_trans_start+0x1df/0x660 [osd_ldiskfs]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa106ab1c>] llog_write+0x22c/0x420 [obdclass]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff81060aa3>] ? perf_event_task_sched_out+0x33/0x70
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa106adfc>] llog_cancel_rec+0xbc/0x7c0 [obdclass]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa10710a7>] llog_cat_cancel_records+0x107/0x340 [obdclass]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa1a3cb91>] osp_sync_process_committed+0x231/0x770 [osp]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa1a3d164>] osp_sync_process_queues+0x94/0x1610 [osp]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffff81061d00>] ? default_wake_function+0x0/0x20
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa106bd77>] llog_process_thread+0x877/0xcf0 [obdclass]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa1a3d0d0>] ? osp_sync_process_queues+0x0/0x1610 [osp]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa106d9d7>] llog_process_or_fork+0x127/0x550 [obdclass]
Sep 11 02:26:14 cs04r-sc-mds03-01 kernel: [<ffffffffa1070eea>] llog_cat_process_cb+0x55a/0x610 [obdclass]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa106bd77>] llog_process_thread+0x877/0xcf0 [obdclass]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa1070990>] ? llog_cat_process_cb+0x0/0x610 [obdclass]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa106d9d7>] llog_process_or_fork+0x127/0x550 [obdclass]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa106ee79>] llog_cat_process_or_fork+0x89/0x350 [obdclass]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffff81061d12>] ? default_wake_function+0x12/0x20
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffff810546b9>] ? __wake_up_common+0x59/0x90
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa1a3d0d0>] ? osp_sync_process_queues+0x0/0x1610 [osp]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa106f159>] llog_cat_process+0x19/0x20 [obdclass]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa1a3ecf3>] osp_sync_thread+0x243/0x7d0 [osp]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffff8152806e>] ? thread_return+0x4e/0x760
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa1a3eab0>] ? osp_sync_thread+0x0/0x7d0 [osp]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffff8109ab56>] kthread+0x96/0xa0
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffff8109aac0>] ? kthread+0x0/0xa0
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: INFO: task osp-syn-11-0:12080 blocked for more than 120 seconds.
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel:      Not tainted 2.6.32-431.17.1.el6_lustre.x86_64 #1
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: osp-syn-11-0  D 0000000000000000     0 12080      2 0x00000080
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: ffff881fdf6635c0 0000000000000046 0000000000000000 6666666638383032
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: ffffffffa10f2e72 ffff880ca86469e8 ffff880104bc8318 00000000a2ffab58
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: ffff881f9a4645f8 ffff881fdf663fd8 000000000000fbc8 ffff881f9a4645f8
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: Call Trace:
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffff815297ce>] __mutex_lock_slowpath+0x13e/0x180
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffff8152966b>] mutex_lock+0x2b/0x50
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa013f688>] __jbd2_log_wait_for_space+0xc8/0x1b0 [jbd2]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa17e71f6>] ? osd_declare_qid+0xd6/0x3f0 [osd_ldiskfs]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa013b01d>] start_this_handle+0x10d/0x4a0 [jbd2]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa17e76b2>] ? osd_declare_inode_qid+0x1a2/0x270 [osd_ldiskfs]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa17d5232>] ? osd_declare_write+0x2a2/0x500 [osd_ldiskfs]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa013b5b0>] jbd2_journal_start+0xd0/0x110 [jbd2]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa16fe916>] ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa17b661f>] osd_trans_start+0x1df/0x660 [osd_ldiskfs]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa106ab1c>] llog_write+0x22c/0x420 [obdclass]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffff81060aa3>] ? perf_event_task_sched_out+0x33/0x70
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa106adfc>] llog_cancel_rec+0xbc/0x7c0 [obdclass]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa10710a7>] llog_cat_cancel_records+0x107/0x340 [obdclass]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa1a3cb91>] osp_sync_process_committed+0x231/0x770 [osp]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa1a3d164>] osp_sync_process_queues+0x94/0x1610 [osp]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffff81061d00>] ? default_wake_function+0x0/0x20
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa106bd77>] llog_process_thread+0x877/0xcf0 [obdclass]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa1a3d0d0>] ? osp_sync_process_queues+0x0/0x1610 [osp]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa106d9d7>] llog_process_or_fork+0x127/0x550 [obdclass]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa1070eea>] llog_cat_process_cb+0x55a/0x610 [obdclass]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa106bd77>] llog_process_thread+0x877/0xcf0 [obdclass]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa1070990>] ? llog_cat_process_cb+0x0/0x610 [obdclass]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa106d9d7>] llog_process_or_fork+0x127/0x550 [obdclass]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa106ee79>] llog_cat_process_or_fork+0x89/0x350 [obdclass]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffff81061d12>] ? default_wake_function+0x12/0x20
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffff810546b9>] ? __wake_up_common+0x59/0x90
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa1a3d0d0>] ? osp_sync_process_queues+0x0/0x1610 [osp]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa106f159>] llog_cat_process+0x19/0x20 [obdclass]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa1a3ecf3>] osp_sync_thread+0x243/0x7d0 [osp]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffff8152806e>] ? thread_return+0x4e/0x760
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa1a3eab0>] ? osp_sync_thread+0x0/0x7d0 [osp]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffff8109ab56>] kthread+0x96/0xa0
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffff8109aac0>] ? kthread+0x0/0xa0
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: INFO: task osp-syn-15-0:12082 blocked for more than 120 seconds.
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel:      Not tainted 2.6.32-431.17.1.el6_lustre.x86_64 #1
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: osp-syn-15-0  D 0000000000000000     0 12082      2 0x00000080
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: ffff881fdf66b5c0 0000000000000046 6430333033323030 6666666638383034
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: ffffffffa10f2e72 ffff88072dd32dd8 ffff880104bc8318 00000000ec9dd7d3
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: ffff88201a611098 ffff881fdf66bfd8 000000000000fbc8 ffff88201a611098
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: Call Trace:
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffff815297ce>] __mutex_lock_slowpath+0x13e/0x180
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffff8152966b>] mutex_lock+0x2b/0x50
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa013f688>] __jbd2_log_wait_for_space+0xc8/0x1b0 [jbd2]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa17e71f6>] ? osd_declare_qid+0xd6/0x3f0 [osd_ldiskfs]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa013b01d>] start_this_handle+0x10d/0x4a0 [jbd2]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa17e76b2>] ? osd_declare_inode_qid+0x1a2/0x270 [osd_ldiskfs]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa17d5232>] ? osd_declare_write+0x2a2/0x500 [osd_ldiskfs]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa013b5b0>] jbd2_journal_start+0xd0/0x110 [jbd2]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa16fe916>] ldiskfs_journal_start_sb+0x56/0xe0 [ldiskfs]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa17b661f>] osd_trans_start+0x1df/0x660 [osd_ldiskfs]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa106ab1c>] llog_write+0x22c/0x420 [obdclass]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffff81060aa3>] ? perf_event_task_sched_out+0x33/0x70
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa106adfc>] llog_cancel_rec+0xbc/0x7c0 [obdclass]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa10710a7>] llog_cat_cancel_records+0x107/0x340 [obdclass]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa1a3cb91>] osp_sync_process_committed+0x231/0x770 [osp]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa1a3d164>] osp_sync_process_queues+0x94/0x1610 [osp]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffff81061d00>] ? default_wake_function+0x0/0x20
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa106bd77>] llog_process_thread+0x877/0xcf0 [obdclass]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa1a3d0d0>] ? osp_sync_process_queues+0x0/0x1610 [osp]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa106d9d7>] llog_process_or_fork+0x127/0x550 [obdclass]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa1070eea>] llog_cat_process_cb+0x55a/0x610 [obdclass]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa106bd77>] llog_process_thread+0x877/0xcf0 [obdclass]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa1070990>] ? llog_cat_process_cb+0x0/0x610 [obdclass]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa106d9d7>] llog_process_or_fork+0x127/0x550 [obdclass]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa106ee79>] llog_cat_process_or_fork+0x89/0x350 [obdclass]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffff81061d12>] ? default_wake_function+0x12/0x20
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffff810546b9>] ? __wake_up_common+0x59/0x90
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa1a3d0d0>] ? osp_sync_process_queues+0x0/0x1610 [osp]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa106f159>] llog_cat_process+0x19/0x20 [obdclass]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa1a3ecf3>] osp_sync_thread+0x243/0x7d0 [osp]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffff8152806e>] ? thread_return+0x4e/0x760
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffffa1a3eab0>] ? osp_sync_thread+0x0/0x7d0 [osp]
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffff8109ab56>] kthread+0x96/0xa0
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffff8100c20a>] child_rip+0xa/0x20
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffff8109aac0>] ? kthread+0x0/0xa0
Sep 11 02:26:15 cs04r-sc-mds03-01 kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20
Sep 11 04:00:10 cs04r-sc-mds03-01 kernel: LNet: Service thread pid 22232 completed after 217.24s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Sep 11 04:00:10 cs04r-sc-mds03-01 kernel: LNet: Skipped 7 previous similar messages
Sep 11 04:47:43 cs04r-sc-mds03-01 kernel: LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) ### lock callback timer expired after 151s: evicting client at 10.144.140.46@o2ib  ns: mdt-lustre03-MDT0000_UUID lock: ffff880d7abef3c0/0x4a9a61dbe320f47a lrc: 3/0,0 mode: PR/PR res: [0x4a40692:0xb304ffb3:0x0].0 bits 0x13 rrc: 3 type: IBT flags: 0x60200000000020 nid: 10.144.140.46@o2ib remote: 0xc6d2a2809bd5a9f1 expref: 84365 pid: 20014 timeout: 4398798140 lvb_type: 0
Sep 11 04:47:43 cs04r-sc-mds03-01 kernel: LustreError: 28875:0:(client.c:1079:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff8809f7420400 x1478844569897188/t0(0) o104->lustre03-MDT0000@10.144.140.46@o2ib:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:N/0/ffffffff rc 0/-1
Sep 11 04:47:43 cs04r-sc-mds03-01 kernel: LustreError: 28875:0:(ldlm_lockd.c:662:ldlm_handle_ast_error()) ### client (nid 10.144.140.46@o2ib) returned 0 from blocking AST ns: mdt-lustre03-MDT0000_UUID lock: ffff880168665880/0x4a9a61dbe320f9dd lrc: 1/0,0 mode: --/CR res: [0x4a40695:0xb304ffb6:0x0].0 bits 0x5 rrc: 2 type: IBT flags: 0x64a01000000020 nid: 10.144.140.46@o2ib remote: 0xc6d2a2809bd5aa06 expref: 60513 pid: 12032 timeout: 4398949080 lvb_type: 0
Sep 11 04:49:10 cs04r-sc-mds03-01 kernel: Lustre: lustre03-MDT0000: Client b4d423ad-3219-f806-0fd2-5a2845b5faad (at 10.144.140.46@o2ib) reconnecting
Sep 11 04:49:10 cs04r-sc-mds03-01 kernel: Lustre: Skipped 67 previous similar messages
Sep 11 05:08:41 cs04r-sc-mds03-01 kernel: LNet: Service thread pid 12048 completed after 321.32s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Sep 11 05:08:41 cs04r-sc-mds03-01 kernel: LNet: Skipped 22 previous similar messages
Sep 11 06:09:12 cs04r-sc-mds03-01 kernel: LNet: Service thread pid 12036 completed after 329.22s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Sep 11 06:09:12 cs04r-sc-mds03-01 kernel: LNet: Skipped 1 previous similar message
Sep 11 06:40:46 cs04r-sc-mds03-01 kernel: Lustre: lock timed out (enqueued at 1410413746, 300s ago)
Sep 11 06:40:46 cs04r-sc-mds03-01 kernel: LustreError: dumping log to /tmp/lustre-log.1410414046.22423
Sep 11 07:38:16 cs04r-sc-mds03-01 kernel: LNet: Service thread pid 16759 completed after 371.90s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Sep 11 07:53:25 cs04r-sc-mds03-01 kernel: LNet: Service thread pid 22791 completed after 357.56s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Sep 11 08:48:25 cs04r-sc-mds03-01 kernel: LNet: Service thread pid 28894 completed after 255.32s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Sep 11 09:20:16 cs04r-sc-mds03-01 kernel: LNet: Service thread pid 32160 completed after 499.50s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Sep 11 09:20:16 cs04r-sc-mds03-01 kernel: LNet: Skipped 36 previous similar messages
Sep 11 09:30:21 cs04r-sc-mds03-01 kernel: LNet: Service thread pid 16787 completed after 268.14s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Sep 11 09:54:58 cs04r-sc-mds03-01 kernel: LNet: Service thread pid 20990 completed after 221.11s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Sep 11 10:19:06 cs04r-sc-mds03-01 kernel: Lustre: lock timed out (enqueued at 1410426846, 300s ago)
Sep 11 10:21:31 cs04r-sc-mds03-01 kernel: LNet: Service thread pid 7551 completed after 522.42s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Sep 11 10:21:31 cs04r-sc-mds03-01 kernel: LNet: Skipped 39 previous similar messages
Sep 11 10:41:15 cs04r-sc-mds03-01 kernel: LNet: Service thread pid 16756 completed after 322.11s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Sep 11 11:01:41 cs04r-sc-mds03-01 kernel: LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) ### lock callback timer expired after 677s: evicting client at 10.144.140.45@o2ib  ns: mdt-lustre03-MDT0000_UUID lock: ffff880253d7e9c0/0x4a9a61dc1bb8b68f lrc: 3/0,0 mode: PR/PR res: [0xdc2e82d:0x59c5e92d:0x0].0 bits 0x13 rrc: 4 type: IBT flags: 0x60200000000020 nid: 10.144.140.45@o2ib remote: 0x297233518b380da9 expref: 239038 pid: 28884 timeout: 4421236585 lvb_type: 0
Sep 11 11:03:00 cs04r-sc-mds03-01 kernel: Lustre: lustre03-MDT0000: Client 3dddf195-6c14-0382-3584-f23c10ca3089 (at 10.144.140.45@o2ib) reconnecting
Sep 11 11:54:45 cs04r-sc-mds03-01 kernel: LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) ### lock callback timer expired after 153s: evicting client at 10.144.140.47@o2ib  ns: mdt-lustre03-MDT0000_UUID lock: ffff881c74a85080/0x4a9a61dc23b3735b lrc: 3/0,0 mode: PR/PR res: [0x200007567:0x7:0x0].0 bits 0x13 rrc: 18 type: IBT flags: 0x60200000000020 nid: 10.144.140.47@o2ib remote: 0x228827ab099911b5 expref: 437268 pid: 19461 timeout: 4424420142 lvb_type: 0
Sep 11 11:55:32 cs04r-sc-mds03-01 kernel: Lustre: lock timed out (enqueued at 1410432732, 200s ago)
Sep 11 11:55:32 cs04r-sc-mds03-01 kernel: Lustre: Skipped 1 previous similar message
Sep 11 11:55:33 cs04r-sc-mds03-01 kernel: Lustre: lock timed out (enqueued at 1410432733, 200s ago)
Sep 11 11:55:33 cs04r-sc-mds03-01 kernel: Lustre: Skipped 2 previous similar messages
Sep 11 11:55:35 cs04r-sc-mds03-01 kernel: Lustre: lock timed out (enqueued at 1410432735, 200s ago)
Sep 11 11:55:51 cs04r-sc-mds03-01 kernel: LNet: Service thread pid 28897 completed after 215.77s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Sep 11 11:56:24 cs04r-sc-mds03-01 kernel: Lustre: lustre03-MDT0000: Client c40661aa-f9cf-734e-0ae4-f4fceaa150fe (at 10.144.140.59@o2ib) reconnecting
Sep 11 12:21:18 cs04r-sc-mds03-01 kernel: LNet: Service thread pid 15463 completed after 488.08s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Sep 11 12:21:18 cs04r-sc-mds03-01 kernel: LNet: Skipped 32 previous similar messages
Sep 11 14:25:11 cs04r-sc-mds03-01 kernel: LNet: Service thread pid 18158 completed after 453.22s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Sep 11 14:32:30 cs04r-sc-mds03-01 kernel: LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) ### lock callback timer expired after 9192s: evicting client at 10.144.140.56@o2ib  ns: mdt-lustre03-MDT0000_UUID lock: ffff8801e33bd4c0/0x4a9a61dc24d232a9 lrc: 3/0,0 mode: PR/PR res: [0x200004dca:0xd6:0x0].0 bits 0x13 rrc: 7 type: IBT flags: 0x60200000000020 nid: 10.144.140.56@o2ib remote: 0x543aaf6dd887195e expref: 116430 pid: 28874 timeout: 4433885427 lvb_type: 0
Sep 11 14:32:30 cs04r-sc-mds03-01 kernel: LustreError: 0:0:(ldlm_lockd.c:344:waiting_locks_callback()) Skipped 3 previous similar messages
Sep 11 14:34:15 cs04r-sc-mds03-01 kernel: Lustre: lustre03-MDT0000: Client 0a637150-6b35-eafb-1fa5-9f9c57f67447 (at 10.144.140.41@o2ib) reconnecting
Sep 11 14:34:59 cs04r-sc-mds03-01 kernel: Lustre: lock timed out (enqueued at 1410442199, 300s ago)
Sep 11 14:34:59 cs04r-sc-mds03-01 kernel: Lustre: Skipped 5 previous similar messages
Sep 11 14:35:04 cs04r-sc-mds03-01 kernel: Lustre: lock timed out (enqueued at 1410442204, 300s ago)
Sep 11 14:36:20 cs04r-sc-mds03-01 kernel: Lustre: lock timed out (enqueued at 1410442380, 200s ago)
Sep 11 14:36:20 cs04r-sc-mds03-01 kernel: Lustre: Skipped 1 previous similar message

Sep 11 14:56:44 cs04r-sc-mds03-01 kernel: LNet: Service thread pid 18155 completed after 292.29s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Sep 11 14:56:44 cs04r-sc-mds03-01 kernel: LNet: Skipped 3 previous similar messages