(sorry about the formatting, this was pasted into an email -- mjmac) Jan 7 15:28:13 chroma-mds0 kernel: Lustre: 6261:0:(client.c:1780:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jan 7 15:28:58 chroma-mds0 kernel: INFO: task tgt_recov:6363 blocked for more than 120 seconds. Jan 7 15:28:58 chroma-mds0 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 7 15:28:58 chroma-mds0 kernel: tgt_recov D 0000000000000005 0 6363 2 0x00000080 Jan 7 15:28:58 chroma-mds0 kernel: ffff8806108d1e20 0000000000000046 0000000000000000 0000000000000003 Jan 7 15:28:58 chroma-mds0 kernel: ffff8806108d1db0 ffffffff810538b6 ffff8806108d1dc0 ffff880c215e5500 Jan 7 15:28:58 chroma-mds0 kernel: ffff88062ea17058 ffff8806108d1fd8 000000000000fb88 ffff88062ea17058 Jan 7 15:28:58 chroma-mds0 kernel: Call Trace: Jan 7 15:28:58 chroma-mds0 kernel: [<ffffffff810538b6>] ? enqueue_task+0x66/0x80 Jan 7 15:28:58 chroma-mds0 kernel: [<ffffffffa044af70>] ? check_for_clients+0x0/0x90 [ptlrpc] Jan 7 15:28:58 chroma-mds0 kernel: [<ffffffffa044dccd>] target_recovery_overseer+0x9d/0x230 [ptlrpc] Jan 7 15:28:58 chroma-mds0 kernel: [<ffffffffa044ad70>] ? exp_connect_healthy+0x0/0x20 [ptlrpc] Jan 7 15:28:58 chroma-mds0 kernel: [<ffffffff810920d0>] ? autoremove_wake_function+0x0/0x40 Jan 7 15:28:58 chroma-mds0 kernel: [<ffffffffa0453796>] target_recovery_thread+0x566/0x1880 [ptlrpc] Jan 7 15:28:58 chroma-mds0 kernel: [<ffffffffa0453230>] ? target_recovery_thread+0x0/0x1880 [ptlrpc] Jan 7 15:28:58 chroma-mds0 kernel: [<ffffffff8100c14a>] child_rip+0xa/0x20 Jan 7 15:28:58 chroma-mds0 kernel: [<ffffffffa0453230>] ? target_recovery_thread+0x0/0x1880 [ptlrpc] Jan 7 15:28:58 chroma-mds0 kernel: [<ffffffffa0453230>] ? target_recovery_thread+0x0/0x1880 [ptlrpc] Jan 7 15:28:58 chroma-mds0 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20 Jan 7 15:29:33 chroma-mds0 kernel: Lustre: 6261:0:(client.c:1780:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1357594148/real 1357594148] req@ffff880c1959f400 x1423540373946449/t0(0) o8->chroma-OST0000-osc-MDT0000@10.0.1.3@o2ib:28/4 lens 368/512 e 0 to 1 dl 1357594173 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jan 7 15:29:33 chroma-mds0 kernel: Lustre: 6261:0:(client.c:1780:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jan 7 15:30:58 chroma-mds0 kernel: INFO: task tgt_recov:6363 blocked for more than 120 seconds. Jan 7 15:30:58 chroma-mds0 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 7 15:30:58 chroma-mds0 kernel: tgt_recov D 0000000000000005 0 6363 2 0x00000080 Jan 7 15:30:58 chroma-mds0 kernel: ffff8806108d1e20 0000000000000046 0000000000000000 0000000000000003 Jan 7 15:30:58 chroma-mds0 kernel: ffff8806108d1db0 ffffffff810538b6 ffff8806108d1dc0 ffff880c215e5500 Jan 7 15:30:58 chroma-mds0 kernel: ffff88062ea17058 ffff8806108d1fd8 000000000000fb88 ffff88062ea17058 Jan 7 15:30:58 chroma-mds0 kernel: Call Trace: Jan 7 15:30:58 chroma-mds0 kernel: [<ffffffff810538b6>] ? enqueue_task+0x66/0x80 Jan 7 15:30:58 chroma-mds0 kernel: [<ffffffffa044af70>] ? check_for_clients+0x0/0x90 [ptlrpc] Jan 7 15:30:58 chroma-mds0 kernel: [<ffffffffa044dccd>] target_recovery_overseer+0x9d/0x230 [ptlrpc] Jan 7 15:30:58 chroma-mds0 kernel: [<ffffffffa044ad70>] ? exp_connect_healthy+0x0/0x20 [ptlrpc] Jan 7 15:30:58 chroma-mds0 kernel: [<ffffffff810920d0>] ? autoremove_wake_function+0x0/0x40 Jan 7 15:30:58 chroma-mds0 kernel: [<ffffffffa0453796>] target_recovery_thread+0x566/0x1880 [ptlrpc] Jan 7 15:30:58 chroma-mds0 kernel: [<ffffffffa0453230>] ? target_recovery_thread+0x0/0x1880 [ptlrpc] Jan 7 15:30:58 chroma-mds0 kernel: [<ffffffff8100c14a>] child_rip+0xa/0x20 Jan 7 15:30:58 chroma-mds0 kernel: [<ffffffffa0453230>] ? target_recovery_thread+0x0/0x1880 [ptlrpc] Jan 7 15:30:58 chroma-mds0 kernel: [<ffffffffa0453230>] ? target_recovery_thread+0x0/0x1880 [ptlrpc] Jan 7 15:30:58 chroma-mds0 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20 Jan 7 15:31:18 chroma-mds0 kernel: Lustre: 6261:0:(client.c:1780:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1357594248/real 1357594248] req@ffff880c1855e400 x1423540373946465/t0(0) o8->chroma-OST0000-osc-MDT0000@10.0.1.3@o2ib:28/4 lens 368/512 e 0 to 1 dl 1357594278 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jan 7 15:31:18 chroma-mds0 kernel: Lustre: 6261:0:(client.c:1780:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Jan 7 15:32:58 chroma-mds0 kernel: INFO: task tgt_recov:6363 blocked for more than 120 seconds. Jan 7 15:32:58 chroma-mds0 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 7 15:32:58 chroma-mds0 kernel: tgt_recov D 0000000000000005 0 6363 2 0x00000080 Jan 7 15:32:58 chroma-mds0 kernel: ffff8806108d1e20 0000000000000046 0000000000000000 0000000000000003 Jan 7 15:32:58 chroma-mds0 kernel: ffff8806108d1db0 ffffffff810538b6 ffff8806108d1dc0 ffff880c215e5500 Jan 7 15:32:58 chroma-mds0 kernel: ffff88062ea17058 ffff8806108d1fd8 000000000000fb88 ffff88062ea17058 Jan 7 15:32:58 chroma-mds0 kernel: Call Trace: Jan 7 15:32:58 chroma-mds0 kernel: [<ffffffff810538b6>] ? enqueue_task+0x66/0x80 Jan 7 15:32:58 chroma-mds0 kernel: [<ffffffffa044af70>] ? check_for_clients+0x0/0x90 [ptlrpc] Jan 7 15:32:58 chroma-mds0 kernel: [<ffffffffa044dccd>] target_recovery_overseer+0x9d/0x230 [ptlrpc] Jan 7 15:32:58 chroma-mds0 kernel: [<ffffffffa044ad70>] ? exp_connect_healthy+0x0/0x20 [ptlrpc] Jan 7 15:32:58 chroma-mds0 kernel: [<ffffffff810920d0>] ? autoremove_wake_function+0x0/0x40 Jan 7 15:32:58 chroma-mds0 kernel: [<ffffffffa0453796>] target_recovery_thread+0x566/0x1880 [ptlrpc] Jan 7 15:32:58 chroma-mds0 kernel: [<ffffffffa0453230>] ? target_recovery_thread+0x0/0x1880 [ptlrpc] Jan 7 15:32:58 chroma-mds0 kernel: [<ffffffff8100c14a>] child_rip+0xa/0x20 Jan 7 15:32:58 chroma-mds0 kernel: [<ffffffffa0453230>] ? target_recovery_thread+0x0/0x1880 [ptlrpc] Jan 7 15:32:58 chroma-mds0 kernel: [<ffffffffa0453230>] ? target_recovery_thread+0x0/0x1880 [ptlrpc] Jan 7 15:32:58 chroma-mds0 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20 Jan 7 15:33:58 chroma-mds0 kernel: Lustre: 6261:0:(client.c:1780:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1357594398/real 1357594398] req@ffff880628980c00 x1423540373946489/t0(0) o8->chroma-OST0000-osc-MDT0000@10.0.1.2@o2ib:28/4 lens 368/512 e 0 to 1 dl 1357594438 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jan 7 15:33:58 chroma-mds0 kernel: Lustre: 6261:0:(client.c:1780:ptlrpc_expire_one_request()) Skipped 17 previous similar messages Jan 7 15:34:58 chroma-mds0 kernel: INFO: task tgt_recov:6363 blocked for more than 120 seconds. Jan 7 15:34:58 chroma-mds0 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 7 15:34:58 chroma-mds0 kernel: tgt_recov D 0000000000000005 0 6363 2 0x00000080 Jan 7 15:34:58 chroma-mds0 kernel: ffff8806108d1e20 0000000000000046 0000000000000000 0000000000000003 Jan 7 15:34:58 chroma-mds0 kernel: ffff8806108d1db0 ffffffff810538b6 ffff8806108d1dc0 ffff880c215e5500 Jan 7 15:34:58 chroma-mds0 kernel: ffff88062ea17058 ffff8806108d1fd8 000000000000fb88 ffff88062ea17058 Jan 7 15:34:58 chroma-mds0 kernel: Call Trace: Jan 7 15:34:58 chroma-mds0 kernel: [<ffffffff810538b6>] ? enqueue_task+0x66/0x80 Jan 7 15:34:58 chroma-mds0 kernel: [<ffffffffa044af70>] ? check_for_clients+0x0/0x90 [ptlrpc] Jan 7 15:34:58 chroma-mds0 kernel: [<ffffffffa044dccd>] target_recovery_overseer+0x9d/0x230 [ptlrpc] Jan 7 15:34:58 chroma-mds0 kernel: [<ffffffffa044ad70>] ? exp_connect_healthy+0x0/0x20 [ptlrpc] Jan 7 15:34:58 chroma-mds0 kernel: [<ffffffff810920d0>] ? autoremove_wake_function+0x0/0x40 Jan 7 15:34:58 chroma-mds0 kernel: [<ffffffffa0453796>] target_recovery_thread+0x566/0x1880 [ptlrpc] Jan 7 15:34:58 chroma-mds0 kernel: [<ffffffffa0453230>] ? target_recovery_thread+0x0/0x1880 [ptlrpc] Jan 7 15:34:58 chroma-mds0 kernel: [<ffffffff8100c14a>] child_rip+0xa/0x20 Jan 7 15:34:58 chroma-mds0 kernel: [<ffffffffa0453230>] ? target_recovery_thread+0x0/0x1880 [ptlrpc] Jan 7 15:34:58 chroma-mds0 kernel: [<ffffffffa0453230>] ? target_recovery_thread+0x0/0x1880 [ptlrpc] Jan 7 15:34:58 chroma-mds0 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20 Jan 7 15:36:58 chroma-mds0 kernel: INFO: task tgt_recov:6363 blocked for more than 120 seconds. Jan 7 15:36:58 chroma-mds0 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 7 15:36:58 chroma-mds0 kernel: tgt_recov D 0000000000000005 0 6363 2 0x00000080 Jan 7 15:36:58 chroma-mds0 kernel: ffff8806108d1e20 0000000000000046 0000000000000000 0000000000000003 Jan 7 15:36:58 chroma-mds0 kernel: ffff8806108d1db0 ffffffff810538b6 ffff8806108d1dc0 ffff880c215e5500 Jan 7 15:36:58 chroma-mds0 kernel: ffff88062ea17058 ffff8806108d1fd8 000000000000fb88 ffff88062ea17058 Jan 7 15:36:58 chroma-mds0 kernel: Call Trace: Jan 7 15:36:58 chroma-mds0 kernel: [<ffffffff810538b6>] ? enqueue_task+0x66/0x80 Jan 7 15:36:58 chroma-mds0 kernel: [<ffffffffa044af70>] ? check_for_clients+0x0/0x90 [ptlrpc] Jan 7 15:36:58 chroma-mds0 kernel: [<ffffffffa044dccd>] target_recovery_overseer+0x9d/0x230 [ptlrpc] Jan 7 15:36:58 chroma-mds0 kernel: [<ffffffffa044ad70>] ? exp_connect_healthy+0x0/0x20 [ptlrpc] Jan 7 15:36:58 chroma-mds0 kernel: [<ffffffff810920d0>] ? autoremove_wake_function+0x0/0x40 Jan 7 15:36:58 chroma-mds0 kernel: [<ffffffffa0453796>] target_recovery_thread+0x566/0x1880 [ptlrpc] Jan 7 15:36:58 chroma-mds0 kernel: [<ffffffffa0453230>] ? target_recovery_thread+0x0/0x1880 [ptlrpc] Jan 7 15:36:58 chroma-mds0 kernel: [<ffffffff8100c14a>] child_rip+0xa/0x20 Jan 7 15:36:58 chroma-mds0 kernel: [<ffffffffa0453230>] ? target_recovery_thread+0x0/0x1880 [ptlrpc] Jan 7 15:36:58 chroma-mds0 kernel: [<ffffffffa0453230>] ? target_recovery_thread+0x0/0x1880 [ptlrpc] Jan 7 15:36:58 chroma-mds0 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20 Jan 7 15:38:43 chroma-mds0 kernel: Lustre: 6261:0:(client.c:1780:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1357594673/real 1357594673] req@ffff8806283e0800 x1423540373946530/t0(0) o8->chroma-OST0000-osc-MDT0000@10.0.1.3@o2ib:28/4 lens 368/512 e 0 to 1 dl 1357594723 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Jan 7 15:38:43 chroma-mds0 kernel: Lustre: 6261:0:(client.c:1780:ptlrpc_expire_one_request()) Skipped 29 previous similar messages Jan 7 15:38:58 chroma-mds0 kernel: INFO: task tgt_recov:6363 blocked for more than 120 seconds. Jan 7 15:38:58 chroma-mds0 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 7 15:38:58 chroma-mds0 kernel: tgt_recov D 0000000000000005 0 6363 2 0x00000080 Jan 7 15:38:58 chroma-mds0 kernel: ffff8806108d1e20 0000000000000046 0000000000000000 0000000000000003 Jan 7 15:38:58 chroma-mds0 kernel: ffff8806108d1db0 ffffffff810538b6 ffff8806108d1dc0 ffff880c215e5500 Jan 7 15:38:58 chroma-mds0 kernel: ffff88062ea17058 ffff8806108d1fd8 000000000000fb88 ffff88062ea17058 Jan 7 15:38:58 chroma-mds0 kernel: Call Trace: Jan 7 15:38:58 chroma-mds0 kernel: [<ffffffff810538b6>] ? enqueue_task+0x66/0x80 Jan 7 15:38:58 chroma-mds0 kernel: [<ffffffffa044af70>] ? check_for_clients+0x0/0x90 [ptlrpc] Jan 7 15:38:58 chroma-mds0 kernel: [<ffffffffa044dccd>] target_recovery_overseer+0x9d/0x230 [ptlrpc] Jan 7 15:38:58 chroma-mds0 kernel: [<ffffffffa044ad70>] ? exp_connect_healthy+0x0/0x20 [ptlrpc] Jan 7 15:38:58 chroma-mds0 kernel: [<ffffffff810920d0>] ? autoremove_wake_function+0x0/0x40 Jan 7 15:38:58 chroma-mds0 kernel: [<ffffffffa0453796>] target_recovery_thread+0x566/0x1880 [ptlrpc] Jan 7 15:38:58 chroma-mds0 kernel: [<ffffffffa0453230>] ? target_recovery_thread+0x0/0x1880 [ptlrpc] Jan 7 15:38:58 chroma-mds0 kernel: [<ffffffff8100c14a>] child_rip+0xa/0x20 Jan 7 15:38:58 chroma-mds0 kernel: [<ffffffffa0453230>] ? target_recovery_thread+0x0/0x1880 [ptlrpc] Jan 7 15:38:58 chroma-mds0 kernel: [<ffffffffa0453230>] ? target_recovery_thread+0x0/0x1880 [ptlrpc] Jan 7 15:38:58 chroma-mds0 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20 Jan 7 15:40:58 chroma-mds0 kernel: INFO: task tgt_recov:6363 blocked for more than 120 seconds. Jan 7 15:40:58 chroma-mds0 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 7 15:40:58 chroma-mds0 kernel: tgt_recov D 0000000000000005 0 6363 2 0x00000080 Jan 7 15:40:58 chroma-mds0 kernel: ffff8806108d1e20 0000000000000046 0000000000000000 0000000000000003 Jan 7 15:40:58 chroma-mds0 kernel: ffff8806108d1db0 ffffffff810538b6 ffff8806108d1dc0 ffff880c215e5500 Jan 7 15:40:58 chroma-mds0 kernel: ffff88062ea17058 ffff8806108d1fd8 000000000000fb88 ffff88062ea17058 Jan 7 15:40:58 chroma-mds0 kernel: Call Trace: Jan 7 15:40:58 chroma-mds0 kernel: [<ffffffff810538b6>] ? enqueue_task+0x66/0x80 Jan 7 15:40:58 chroma-mds0 kernel: [<ffffffffa044af70>] ? check_for_clients+0x0/0x90 [ptlr