Dec  9 05:22:57 t2s007037 kernel: Lustre: gscr0-OST0000: haven't heard from client 1ec5a42a-9886-781d-e50f-cbea9460fa87 (at 192.168.18.29@tcp1) in 227 seconds. I think it's dead, and I am evicting it.
Dec  9 05:22:57 t2s007037 kernel: Lustre: Skipped 12 previous similar messages
Dec  9 05:59:47 t2s007037 kernel: Lustre: Service thread pid 30036 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Dec  9 05:59:47 t2s007037 kernel: Lustre: Skipped 2 previous similar messages
Dec  9 05:59:47 t2s007037 kernel: Pid: 30036, comm: ll_ost_io_457
Dec  9 05:59:47 t2s007037 kernel: 
Dec  9 05:59:47 t2s007037 kernel: Call Trace:
Dec  9 05:59:47 t2s007037 kernel:  [<ffffffff886ee121>] LNetMDBind+0x301/0x450 [lnet]
Dec  9 05:59:47 t2s007037 kernel:  [<ffffffff8006389f>] schedule_timeout+0x8a/0xad
Dec  9 05:59:47 t2s007037 kernel:  [<ffffffff8009a41d>] process_timeout+0x0/0x5
Dec  9 05:59:47 t2s007037 kernel:  [<ffffffff88aee22c>] ost_brw_read+0x127c/0x1a70 [ost]
Dec  9 05:59:47 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec  9 05:59:47 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec  9 05:59:47 t2s007037 kernel:  [<ffffffff88af6383>] ost_handle+0x2e73/0x55b0 [ost]
Dec  9 05:59:47 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec  9 05:59:47 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec  9 05:59:47 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec  9 05:59:47 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec  9 05:59:47 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec  9 05:59:47 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec  9 05:59:47 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec  9 05:59:47 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec  9 05:59:47 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec  9 05:59:47 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec  9 05:59:47 t2s007037 kernel: 
Dec  9 05:59:47 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355000387.30036
Dec  9 06:00:06 t2s007037 kernel: Lustre: gscr0-OST0000: haven't heard from client 9ec2a75c-8c4d-e6b8-c42f-e7e05f555bb7 (at 192.168.20.57@tcp1) in 227 seconds. I think it's dead, and I am evicting it.
Dec  9 06:00:06 t2s007037 kernel: Lustre: Skipped 13 previous similar messages
Dec  9 06:00:13 t2s007037 kernel: Lustre: work1-OST000a: haven't heard from client 10f80e74-0034-dbeb-3952-d023dba1f60e (at 192.168.20.57@tcp1) in 227 seconds. I think it's dead, and I am evicting it.
Dec  9 06:00:13 t2s007037 kernel: Lustre: Skipped 12 previous similar messages
Dec  9 06:00:13 t2s007037 kernel: LustreError: 30036:0:(ost_handler.c:825:ost_brw_read()) @@@ Eviction on bulk PUT  req@ffff8106194de050 x1419033709845658/t0 o3->10f80e74-0034-dbeb-3952-d023dba1f60e@NET_0x20001c0a81439_UUID:0/0 lens 448/400 e 2 to 0 dl 1355000419 ref 1 fl Interpret:/0/0 rc 0/0
Dec  9 06:00:13 t2s007037 kernel: Lustre: 30036:0:(ost_handler.c:887:ost_brw_read()) work1-OST000a: ignoring bulk IO comm error with 10f80e74-0034-dbeb-3952-d023dba1f60e@NET_0x20001c0a81439_UUID id 12345-192.168.20.57@tcp1 - client will retry
Dec  9 06:00:13 t2s007037 kernel: Lustre: Service thread pid 30036 completed after 226.02s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
Dec  9 06:07:22 t2s007037 kernel: LustreError: 20376:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-107)  req@ffff810604a32c00 x1416585474323478/t0 o101-><?>@<?>:0/0 lens 296/0 e 0 to 0 dl 1355000848 ref 1 fl Interpret:/0/0 rc -107/0
Dec  9 06:07:22 t2s007037 kernel: LustreError: 20376:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 32 previous similar messages
Dec  9 06:11:09 t2s007037 kernel: Lustre: gscr0-OST0000: haven't heard from client 1ec5a42a-9886-781d-e50f-cbea9460fa87 (at 192.168.18.29@tcp1) in 227 seconds. I think it's dead, and I am evicting it.
Dec  9 13:41:39 t2s007037 kernel: Lustre: 16490:0:(ldlm_lib.c:574:target_handle_reconnect()) gscr0-OST0000: 4970ba4c-1299-11dc-229d-dedbb0fec629 reconnecting
Dec  9 13:41:39 t2s007037 kernel: Lustre: 16490:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 13 previous similar messages
Dec  9 21:42:57 t2s007037 kernel: Lustre: work1-OST0003: haven't heard from client b6ce1e93-8b0a-3e7f-1470-f3b4757b5a76 (at 192.168.18.49@tcp1) in 227 seconds. I think it's dead, and I am evicting it.
Dec  9 21:42:57 t2s007037 kernel: Lustre: gscr0-OST0000: haven't heard from client 922f5cc9-8e17-f60a-a622-a6d0e3c0df76 (at 192.168.18.49@tcp1) in 227 seconds. I think it's dead, and I am evicting it.
Dec  9 21:42:57 t2s007037 kernel: Lustre: Skipped 12 previous similar messages
Dec 10 00:57:47 t2s007037 kernel: Lustre: gscr0-OST0000: haven't heard from client 2a8830fb-9441-76bb-d454-7972f5d130d8 (at 192.168.18.59@tcp1) in 227 seconds. I think it's dead, and I am evicting it.
Dec 10 02:06:55 t2s007037 kernel: LustreError: 137-5: UUID 'gscr0-OST0001_UUID' is not available for connect (no target)
Dec 10 02:06:55 t2s007037 kernel: LustreError: 5524:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19)  req@ffff8104f582d400 x1420409212233398/t0 o8-><?>@<?>:0/0 lens 368/0 e 0 to 0 dl 1355072915 ref 1 fl Interpret:/0/0 rc -19/0
Dec 10 02:06:55 t2s007037 kernel: LustreError: Skipped 13 previous similar messages
Dec 10 03:38:34 t2s007037 kernel: Lustre: gscr0-OST0000: haven't heard from client ba433d74-55f1-0b1e-55db-329476da70a9 (at 10.1.4.111@o2ib) in 227 seconds. I think it's dead, and I am evicting it.
Dec 10 03:38:34 t2s007037 kernel: Lustre: Skipped 13 previous similar messages
Dec 10 03:49:01 t2s007037 kernel: Lustre: gscr0-OST0000: haven't heard from client 28b2ff91-5e41-2f71-a16e-3b8a1205389d (at 192.168.19.121@tcp1) in 227 seconds. I think it's dead, and I am evicting it.
Dec 10 03:49:01 t2s007037 kernel: Lustre: Skipped 13 previous similar messages
Dec 10 03:49:01 t2s007037 kernel: Lustre: work1-OST000c: haven't heard from client bba60d74-6f86-f2d5-a606-1eef566602cc (at 192.168.19.121@tcp1) in 227 seconds. I think it's dead, and I am evicting it.
Dec 10 03:59:40 t2s007037 kernel: Lustre: work1-OST0001: haven't heard from client a0c21911-52b1-cb9e-f090-611055649827 (at 10.1.4.168@o2ib) in 227 seconds. I think it's dead, and I am evicting it.
Dec 10 03:59:40 t2s007037 kernel: Lustre: Skipped 12 previous similar messages
Dec 10 08:29:34 t2s007037 kernel: Lustre: 5620:0:(ldlm_lib.c:574:target_handle_reconnect()) gscr0-OST0000: e3275cf5-7ccb-cf66-bd05-88cccd7f7e95 reconnecting
Dec 10 08:29:34 t2s007037 kernel: LustreError: 137-5: UUID 'gscr0-OST0001_UUID' is not available for connect (no target)
Dec 10 08:29:34 t2s007037 kernel: LustreError: 20292:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19)  req@ffff8102a2265400 x1414587164613946/t0 o8-><?>@<?>:0/0 lens 368/0 e 0 to 0 dl 1355095874 ref 1 fl Interpret:/0/0 rc -19/0
Dec 10 08:29:34 t2s007037 kernel: LustreError: 20292:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 13 previous similar messages
Dec 10 08:29:34 t2s007037 kernel: LustreError: 137-5: UUID 'work1-OST0017_UUID' is not available for connect (no target)
Dec 10 08:29:34 t2s007037 kernel: LustreError: Skipped 6 previous similar messages
Dec 10 08:29:34 t2s007037 kernel: LustreError: 20375:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19)  req@ffff8101cb9ad000 x1414587164613958/t0 o8-><?>@<?>:0/0 lens 368/0 e 0 to 0 dl 1355095874 ref 1 fl Interpret:/0/0 rc -19/0
Dec 10 08:29:34 t2s007037 kernel: LustreError: 20375:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6 previous similar messages
Dec 10 08:29:34 t2s007037 kernel: Lustre: 20341:0:(ldlm_lib.c:803:target_handle_connect()) gscr0-OST0000: exp ffff810257b78c00 already connecting
Dec 10 08:29:34 t2s007037 kernel: Lustre: 5620:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 13 previous similar messages
Dec 10 08:29:58 t2s007037 kernel: Lustre: 20305:0:(ldlm_lib.c:574:target_handle_reconnect()) work1-OST0000: 4227c5f4-3143-5514-773d-7fb46bb6501e reconnecting
Dec 10 08:29:58 t2s007037 kernel: Lustre: 20302:0:(ldlm_lib.c:574:target_handle_reconnect()) work1-OST0002: 4227c5f4-3143-5514-773d-7fb46bb6501e reconnecting
Dec 10 08:29:58 t2s007037 kernel: Lustre: 20302:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 2 previous similar messages
Dec 10 08:29:58 t2s007037 kernel: Lustre: 20305:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 8 previous similar messages
Dec 10 09:52:12 t2s007037 kernel: Lustre: work1-OST0001: haven't heard from client 4573aef8-73aa-7d70-77bf-ede4662c87ad (at 192.168.17.129@tcp1) in 227 seconds. I think it's dead, and I am evicting it.
Dec 10 09:52:12 t2s007037 kernel: Lustre: Skipped 13 previous similar messages
Dec 10 12:06:20 t2s007037 kernel: Lustre: gscr0-OST0000: haven't heard from client 214b93b9-0766-25f9-7f8f-29211658bb21 (at 192.168.17.133@tcp1) in 227 seconds. I think it's dead, and I am evicting it.
Dec 10 12:06:20 t2s007037 kernel: Lustre: Skipped 13 previous similar messages
Dec 10 12:06:20 t2s007037 kernel: Lustre: gscr0-OST0000: haven't heard from client 99fc8811-796f-1e5f-a3cd-f106208b69c5 (at 10.1.1.133@o2ib) in 216 seconds. I think it's dead, and I am evicting it.
Dec 10 12:38:47 t2s007037 kernel: Lustre: gscr0-OST0000: haven't heard from client 83bf7d87-caf5-99a8-8cc6-712b143264c3 (at 192.168.18.111@tcp1) in 227 seconds. I think it's dead, and I am evicting it.
Dec 10 12:38:47 t2s007037 kernel: Lustre: Skipped 26 previous similar messages
Dec 10 13:28:44 t2s007037 kernel: Lustre: gscr0-OST0000: haven't heard from client 78014845-9dc5-fc68-3713-f7d216321c77 (at 10.1.3.121@o2ib) in 227 seconds. I think it's dead, and I am evicting it.
Dec 10 13:28:44 t2s007037 kernel: Lustre: Skipped 13 previous similar messages
Dec 10 13:49:42 t2s007037 kernel: LustreError: 137-5: UUID 'gscr0-OST0001_UUID' is not available for connect (no target)
Dec 10 13:49:42 t2s007037 kernel: LustreError: 5588:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-19)  req@ffff8105e2f18800 x1419033709847052/t0 o8-><?>@<?>:0/0 lens 368/0 e 0 to 0 dl 1355115082 ref 1 fl Interpret:/0/0 rc -19/0
Dec 10 13:49:42 t2s007037 kernel: LustreError: 5588:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 7 previous similar messages
Dec 10 13:49:42 t2s007037 kernel: LustreError: Skipped 19 previous similar messages
Dec 10 13:53:29 t2s007037 kernel: Lustre: gscr0-OST0000: haven't heard from client 9ec2a75c-8c4d-e6b8-c42f-e7e05f555bb7 (at 192.168.20.57@tcp1) in 227 seconds. I think it's dead, and I am evicting it.
Dec 10 13:53:29 t2s007037 kernel: Lustre: Skipped 13 previous similar messages
Dec 10 14:35:41 t2s007037 kernel: Lustre: gscr0-OST0000: haven't heard from client 4cd8b8ea-e22f-5c52-5c29-4adf4b3fe609 (at 192.168.17.122@tcp1) in 227 seconds. I think it's dead, and I am evicting it.
Dec 10 14:35:41 t2s007037 kernel: Lustre: work1-OST000c: haven't heard from client 495d62b2-4e43-bdfe-c2ca-7a91ee83fc70 (at 192.168.17.122@tcp1) in 227 seconds. I think it's dead, and I am evicting it.
Dec 10 18:05:22 t2s007037 kernel: Lustre: 29885:0:(quota_interface.c:491:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Dec 10 18:09:06 t2s007037 kernel: Lustre: 29848:0:(quota_interface.c:491:quota_chk_acq_common()) still haven't managed to acquire quota space from the quota master after 10 retries (err=0, rc=0)
Dec 10 18:11:06 t2s007037 kernel: Lustre: work1-OST0007: slow quota init 31s due to heavy IO load
Dec 10 18:11:06 t2s007037 kernel: Lustre: Skipped 4 previous similar messages
Dec 10 18:46:46 t2s007037 kernel: Lustre: gscr0-OST0000: haven't heard from client 2f2b99f3-ebff-3fc5-b9f8-0104cfcf5a49 (at 192.168.17.137@tcp1) in 227 seconds. I think it's dead, and I am evicting it.
Dec 10 18:46:46 t2s007037 kernel: Lustre: Skipped 12 previous similar messages
Dec 10 18:57:25 t2s007037 kernel: Lustre: Service thread pid 29804 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Dec 10 18:57:25 t2s007037 kernel: Pid: 29804, comm: ll_ost_io_225
Dec 10 18:57:25 t2s007037 kernel: 
Dec 10 18:57:25 t2s007037 kernel: Call Trace:
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff886ef3a6>] lnet_ni_send+0x96/0xe0 [lnet]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 18:57:25 t2s007037 kernel: 
Dec 10 18:57:25 t2s007037 kernel: Lustre: Service thread pid 29747 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Dec 10 18:57:25 t2s007037 kernel: Pid: 29747, comm: ll_ost_io_168
Dec 10 18:57:25 t2s007037 kernel: 
Dec 10 18:57:25 t2s007037 kernel: Call Trace:
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 18:57:25 t2s007037 kernel: 
Dec 10 18:57:25 t2s007037 kernel: Pid: 29792, comm: ll_ost_io_213
Dec 10 18:57:25 t2s007037 kernel: 
Dec 10 18:57:25 t2s007037 kernel: Call Trace:
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff886ef3a6>] lnet_ni_send+0x96/0xe0 [lnet]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff80156c03>] __next_cpu+0x19/0x28
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff8007791b>] smp_send_reschedule+0x4e/0x53
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 18:57:25 t2s007037 kernel: 
Dec 10 18:57:25 t2s007037 kernel: Pid: 29743, comm: ll_ost_io_164
Dec 10 18:57:25 t2s007037 kernel: 
Dec 10 18:57:25 t2s007037 kernel: Call Trace:
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88aebef0>] ost_checksum_bulk+0x360/0x5a0 [ost]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 18:57:25 t2s007037 kernel: 
Dec 10 18:57:25 t2s007037 kernel: Pid: 16727, comm: ll_ost_io_124
Dec 10 18:57:25 t2s007037 kernel: 
Dec 10 18:57:25 t2s007037 kernel: Call Trace:
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff8002deea>] __wake_up+0x38/0x4f
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff886ef3a6>] lnet_ni_send+0x96/0xe0 [lnet]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 18:57:25 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 18:57:25 t2s007037 kernel: 
Dec 10 18:57:25 t2s007037 kernel: Lustre: Service thread pid 29962 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Dec 10 18:57:25 t2s007037 kernel: Lustre: Service thread pid 16607 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Dec 10 18:57:25 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133445.16688
Dec 10 18:57:25 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133445.16722
Dec 10 18:57:25 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133445.16728
Dec 10 18:57:25 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133445.16616
Dec 10 18:57:25 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133445.29876
Dec 10 18:57:25 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133445.16607
Dec 10 18:57:25 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133445.29962
Dec 10 18:57:25 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133445.16727
Dec 10 18:57:25 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133445.29743
Dec 10 18:57:25 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133445.29792
Dec 10 18:57:25 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133445.29747
Dec 10 18:57:25 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133445.29804
Dec 10 18:57:26 t2s007037 kernel: Lustre: Service thread pid 30027 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Dec 10 18:57:26 t2s007037 kernel: Lustre: Skipped 5 previous similar messages
Dec 10 18:57:26 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133446.30027
Dec 10 18:57:26 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133446.16671
Dec 10 18:57:26 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133446.30077
Dec 10 18:57:26 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133446.29987
Dec 10 18:57:26 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133446.29947
Dec 10 18:57:26 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133446.29916
Dec 10 18:57:26 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133446.29787
Dec 10 18:57:26 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133446.29741
Dec 10 18:57:28 t2s007037 kernel: Lustre: Service thread pid 29879 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Dec 10 18:57:28 t2s007037 kernel: Lustre: Skipped 7 previous similar messages
Dec 10 18:57:28 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133448.29879
Dec 10 18:57:28 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133448.30030
Dec 10 18:57:28 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133448.29758
Dec 10 18:57:28 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133448.29827
Dec 10 18:57:28 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133448.16697
Dec 10 18:57:28 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133448.29961
Dec 10 18:57:28 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133448.29755
Dec 10 18:57:28 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133448.16664
Dec 10 18:57:28 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133448.16707
Dec 10 18:57:28 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133448.30059
Dec 10 18:57:28 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133448.16720
Dec 10 18:57:29 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133449.30057
Dec 10 18:57:29 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133449.16676
Dec 10 18:57:29 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133449.29738
Dec 10 18:57:29 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133449.29725
Dec 10 18:57:29 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133449.16645
Dec 10 18:57:29 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133449.16704
Dec 10 18:57:29 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133449.30025
Dec 10 18:57:29 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133449.16623
Dec 10 18:57:29 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133449.29966
Dec 10 18:57:29 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133449.29984
Dec 10 18:57:29 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133449.29767
Dec 10 18:57:29 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133449.29772
Dec 10 18:57:29 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133449.29996
Dec 10 18:57:30 t2s007037 kernel: Lustre: Service thread pid 29730 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Dec 10 18:57:30 t2s007037 kernel: Lustre: Skipped 23 previous similar messages
Dec 10 18:57:30 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133450.29730
Dec 10 18:57:30 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133450.30008
Dec 10 18:57:31 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133450.16679
Dec 10 18:57:31 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133451.16716
Dec 10 18:57:31 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133451.29781
Dec 10 18:57:31 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133451.29819
Dec 10 18:57:31 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133451.29882
Dec 10 18:57:31 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133451.29993
Dec 10 18:57:31 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133451.29751
Dec 10 18:57:31 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133451.29769
Dec 10 18:57:31 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133451.30042
Dec 10 18:57:31 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133451.29944
Dec 10 18:57:31 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133451.29759
Dec 10 18:57:31 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133451.29791
Dec 10 18:57:31 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133451.29866
Dec 10 18:57:31 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133451.29720
Dec 10 18:57:31 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133451.29921
Dec 10 18:57:31 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133451.30021
Dec 10 18:57:31 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133451.16613
Dec 10 18:57:31 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133451.16661
Dec 10 18:57:31 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133451.29934
Dec 10 18:57:31 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133451.29902
Dec 10 18:57:31 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133451.29890
Dec 10 18:57:31 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133451.16693
Dec 10 18:57:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133453.16682
Dec 10 18:57:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133453.30049
Dec 10 18:57:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133453.29891
Dec 10 18:57:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133453.29713
Dec 10 18:57:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133453.29808
Dec 10 18:57:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133453.29830
Dec 10 18:57:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133453.29892
Dec 10 18:57:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133453.29850
Dec 10 18:57:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133453.30078
Dec 10 18:57:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133453.29985
Dec 10 18:57:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133453.29963
Dec 10 18:57:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133453.29982
Dec 10 18:57:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133453.29770
Dec 10 18:57:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133453.16668
Dec 10 18:57:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133453.16729
Dec 10 18:57:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133453.29945
Dec 10 18:57:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133453.16672
Dec 10 18:57:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133453.29886
Dec 10 18:57:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133453.29812
Dec 10 18:57:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133453.16684
Dec 10 18:57:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133453.29974
Dec 10 18:57:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133453.29858
Dec 10 18:57:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133453.29958
Dec 10 18:57:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133453.16630
Dec 10 18:57:35 t2s007037 kernel: Lustre: Service thread pid 16647 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Dec 10 18:57:35 t2s007037 kernel: Lustre: Skipped 47 previous similar messages
Dec 10 18:57:35 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133455.16647
Dec 10 18:57:35 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133455.29989
Dec 10 18:57:35 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133455.29839
Dec 10 18:57:35 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133455.29740
Dec 10 18:57:35 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133455.30037
Dec 10 18:57:35 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133455.16622
Dec 10 18:57:35 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133455.29885
Dec 10 18:57:35 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133455.16702
Dec 10 18:57:36 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133456.16665
Dec 10 18:57:38 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133458.29855
Dec 10 18:57:38 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133458.16657
Dec 10 18:57:38 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133458.29907
Dec 10 18:57:38 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133458.30069
Dec 10 18:57:38 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133458.16603
Dec 10 18:57:38 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133458.29798
Dec 10 18:57:38 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133458.29823
Dec 10 18:57:38 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133458.16618
Dec 10 18:57:39 t2s007037 kernel: INFO: task jbd2/dm-0-8:16413 blocked for more than 120 seconds.
Dec 10 18:57:39 t2s007037 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 10 18:57:39 t2s007037 kernel: jbd2/dm-0-8   D ffff8101d86aa860     0 16413    247         16414 16412 (L-TLB)
Dec 10 18:57:39 t2s007037 kernel:  ffff8102dc6edb90 0000000000000046 0000000000000282 0000000000000008
Dec 10 18:57:39 t2s007037 kernel:  ffff8101b3c483c0 000000000000000a ffff81060d6a1860 ffff8101d86aa860
Dec 10 18:57:39 t2s007037 kernel:  0017bf15f0c46f86 0000000000000be8 ffff81060d6a1a48 0000000a0afb26b8
Dec 10 18:57:39 t2s007037 kernel: Call Trace:
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff8006ece7>] do_gettimeofday+0x40/0x90
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff8005a40e>] getnstimeofday+0x10/0x29
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff80028bd3>] sync_page+0x0/0x42
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff800637de>] io_schedule+0x3f/0x67
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff80028c11>] sync_page+0x3e/0x42
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff80063922>] __wait_on_bit_lock+0x36/0x66
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff8003f9ab>] __lock_page+0x5e/0x64
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff800a34e5>] wake_bit_function+0x0/0x23
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff80047c5b>] pagevec_lookup_tag+0x1a/0x21
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff8001d035>] mpage_writepages+0x14f/0x37d
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88a87bc0>] :ldiskfs:ldiskfs_writepage+0x0/0x3a0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff800a34c0>] autoremove_wake_function+0x9/0x2e
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88a622b4>] :jbd2:jbd2_journal_commit_transaction+0x36c/0x1120
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff8004ad55>] try_to_del_timer_sync+0x7f/0x88
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88a6623e>] :jbd2:kjournald2+0x9a/0x1ec
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88a661a4>] :jbd2:kjournald2+0x0/0x1ec
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff800a329f>] keventd_create_kthread+0x0/0xc4
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff80032654>] kthread+0xfe/0x132
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff800a329f>] keventd_create_kthread+0x0/0xc4
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff80032556>] kthread+0x0/0x132
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 18:57:39 t2s007037 kernel: 
Dec 10 18:57:39 t2s007037 kernel: INFO: task ll_ost_io_01:16603 blocked for more than 120 seconds.
Dec 10 18:57:39 t2s007037 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 10 18:57:39 t2s007037 kernel: ll_ost_io_01  D ffff81032a892120     0 16603      1         16604 16602 (L-TLB)
Dec 10 18:57:39 t2s007037 kernel:  ffff81060f275800 0000000000000046 0000000131746be8 ffff8105fbaa12c0
Dec 10 18:57:39 t2s007037 kernel:  0000000100000000 000000000000000a ffff81061bd0b0c0 ffff81032a9c9080
Dec 10 18:57:39 t2s007037 kernel:  0017bf16bfd53743 0000000000000c20 ffff81061bd0b2a8 00000005ffffffff
Dec 10 18:57:39 t2s007037 kernel: Call Trace:
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88a65828>] :jbd2:jbd2_log_wait_commit+0xa3/0xf5
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88b1090b>] :fsfilt_ldiskfs:fsfilt_ldiskfs_commit_wait+0xab/0xd0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88b51184>] :obdfilter:filter_commitrw_write+0x1e14/0x2dd0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88af2d29>] :ost:ost_brw_write+0x1c99/0x2480
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88800aa8>] :ptlrpc:ptlrpc_send_reply+0x5e8/0x600
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff887cb8b0>] :ptlrpc:target_committed_to_req+0x40/0x120
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88805098>] :ptlrpc:lustre_msg_check_version_v2+0x8/0x20
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88af60be>] :ost:ost_handle+0x2bae/0x55b0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88756cd0>] :obdclass:class_handle2object+0xe0/0x170
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff887bf19a>] :ptlrpc:lock_res_and_lock+0xba/0xd0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff887c4168>] :ptlrpc:__ldlm_handle2lock+0x2f8/0x360
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff888146c1>] :ptlrpc:ptlrpc_server_handle_request+0x981/0xe00
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88814e25>] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88815d66>] :ptlrpc:ptlrpc_main+0xf16/0x10e0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88814e50>] :ptlrpc:ptlrpc_main+0x0/0x10e0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 18:57:39 t2s007037 kernel: 
Dec 10 18:57:39 t2s007037 kernel: INFO: task ll_ost_io_02:16604 blocked for more than 120 seconds.
Dec 10 18:57:39 t2s007037 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 10 18:57:39 t2s007037 kernel: ll_ost_io_02  D ffff81000902e4a0     0 16604      1         16605 16603 (L-TLB)
Dec 10 18:57:39 t2s007037 kernel:  ffff810613adf800 0000000000000046 00000100e50f58b8 ffff8101954bbec0
Dec 10 18:57:39 t2s007037 kernel:  0000000200000000 000000000000000a ffff810619d00860 ffff81061fedf0c0
Dec 10 18:57:39 t2s007037 kernel:  0017bf222ba66d92 00000000000009eb ffff810619d00a48 0000000affffffff
Dec 10 18:57:39 t2s007037 kernel: Call Trace:
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88a65828>] :jbd2:jbd2_log_wait_commit+0xa3/0xf5
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88b1090b>] :fsfilt_ldiskfs:fsfilt_ldiskfs_commit_wait+0xab/0xd0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88b51184>] :obdfilter:filter_commitrw_write+0x1e14/0x2dd0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff886ef3a6>] :lnet:lnet_ni_send+0x96/0xe0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88aebe92>] :ost:ost_checksum_bulk+0x302/0x5a0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88af2d29>] :ost:ost_brw_write+0x1c99/0x2480
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88800aa8>] :ptlrpc:ptlrpc_send_reply+0x5e8/0x600
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff887cb8b0>] :ptlrpc:target_committed_to_req+0x40/0x120
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88805098>] :ptlrpc:lustre_msg_check_version_v2+0x8/0x20
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88af60be>] :ost:ost_handle+0x2bae/0x55b0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88756cd0>] :obdclass:class_handle2object+0xe0/0x170
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff887bf19a>] :ptlrpc:lock_res_and_lock+0xba/0xd0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff887c4168>] :ptlrpc:__ldlm_handle2lock+0x2f8/0x360
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff888146c1>] :ptlrpc:ptlrpc_server_handle_request+0x981/0xe00
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88814e25>] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88815d66>] :ptlrpc:ptlrpc_main+0xf16/0x10e0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88814e50>] :ptlrpc:ptlrpc_main+0x0/0x10e0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 18:57:39 t2s007037 kernel: 
Dec 10 18:57:39 t2s007037 kernel: INFO: task ll_ost_io_05:16607 blocked for more than 120 seconds.
Dec 10 18:57:39 t2s007037 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 10 18:57:39 t2s007037 kernel: ll_ost_io_05  D ffff81032a8a2e20     0 16607      1         16609 16606 (L-TLB)
Dec 10 18:57:39 t2s007037 kernel:  ffff81061a753800 0000000000000046 0000002a1c465b58 ffff8105bcec03c0
Dec 10 18:57:39 t2s007037 kernel:  0000000d00000000 000000000000000a ffff81061ba03040 ffff81061fe86080
Dec 10 18:57:39 t2s007037 kernel:  0017bf14cafd6119 0000000000000ac3 ffff81061ba03228 00000009ffffffff
Dec 10 18:57:39 t2s007037 kernel: Call Trace:
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88a65828>] :jbd2:jbd2_log_wait_commit+0xa3/0xf5
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88b1090b>] :fsfilt_ldiskfs:fsfilt_ldiskfs_commit_wait+0xab/0xd0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88b51184>] :obdfilter:filter_commitrw_write+0x1e14/0x2dd0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88af2d29>] :ost:ost_brw_write+0x1c99/0x2480
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88800aa8>] :ptlrpc:ptlrpc_send_reply+0x5e8/0x600
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff887cb8b0>] :ptlrpc:target_committed_to_req+0x40/0x120
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88805098>] :ptlrpc:lustre_msg_check_version_v2+0x8/0x20
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88af60be>] :ost:ost_handle+0x2bae/0x55b0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88756cd0>] :obdclass:class_handle2object+0xe0/0x170
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff887bf19a>] :ptlrpc:lock_res_and_lock+0xba/0xd0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff887c4168>] :ptlrpc:__ldlm_handle2lock+0x2f8/0x360
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff888146c1>] :ptlrpc:ptlrpc_server_handle_request+0x981/0xe00
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88814e25>] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88815d66>] :ptlrpc:ptlrpc_main+0xf16/0x10e0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88814e50>] :ptlrpc:ptlrpc_main+0x0/0x10e0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 18:57:39 t2s007037 kernel: 
Dec 10 18:57:39 t2s007037 kernel: INFO: task ll_ost_io_06:16609 blocked for more than 120 seconds.
Dec 10 18:57:39 t2s007037 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 10 18:57:39 t2s007037 kernel: ll_ost_io_06  D ffff81000902e4a0     0 16609      1         16610 16607 (L-TLB)
Dec 10 18:57:39 t2s007037 kernel:  ffff81061830f800 0000000000000046 00000079e50f5948 ffff8102649016c0
Dec 10 18:57:39 t2s007037 kernel:  0000000100000000 000000000000000a ffff81061be2d7e0 ffff81061fedf0c0
Dec 10 18:57:39 t2s007037 kernel:  0017bf184c09f2fd 0000000000000798 ffff81061be2d9c8 0000000affffffff
Dec 10 18:57:39 t2s007037 kernel: Call Trace:
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88a65828>] :jbd2:jbd2_log_wait_commit+0xa3/0xf5
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88b1090b>] :fsfilt_ldiskfs:fsfilt_ldiskfs_commit_wait+0xab/0xd0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88b51184>] :obdfilter:filter_commitrw_write+0x1e14/0x2dd0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff886ef3a6>] :lnet:lnet_ni_send+0x96/0xe0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88af2d29>] :ost:ost_brw_write+0x1c99/0x2480
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88800aa8>] :ptlrpc:ptlrpc_send_reply+0x5e8/0x600
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff887cb8b0>] :ptlrpc:target_committed_to_req+0x40/0x120
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88805098>] :ptlrpc:lustre_msg_check_version_v2+0x8/0x20
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88af60be>] :ost:ost_handle+0x2bae/0x55b0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88756cd0>] :obdclass:class_handle2object+0xe0/0x170
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff887bf19a>] :ptlrpc:lock_res_and_lock+0xba/0xd0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff887c4168>] :ptlrpc:__ldlm_handle2lock+0x2f8/0x360
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff888146c1>] :ptlrpc:ptlrpc_server_handle_request+0x981/0xe00
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88814e25>] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88815d66>] :ptlrpc:ptlrpc_main+0xf16/0x10e0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88814e50>] :ptlrpc:ptlrpc_main+0x0/0x10e0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 18:57:39 t2s007037 kernel: 
Dec 10 18:57:39 t2s007037 kernel: INFO: task ll_ost_io_07:16610 blocked for more than 120 seconds.
Dec 10 18:57:39 t2s007037 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 10 18:57:39 t2s007037 kernel: ll_ost_io_07  D ffff81032a8a2e20     0 16610      1         16611 16609 (L-TLB)
Dec 10 18:57:39 t2s007037 kernel:  ffff810615761800 0000000000000046 000001001c465d98 ffff8104a0954bc0
Dec 10 18:57:39 t2s007037 kernel:  0000000200000000 000000000000000a ffff81061be2d080 ffff81061fe86080
Dec 10 18:57:39 t2s007037 kernel:  0017bf28fefda58b 00000000000013a2 ffff81061be2d268 00000009ffffffff
Dec 10 18:57:39 t2s007037 kernel: Call Trace:
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88a65828>] :jbd2:jbd2_log_wait_commit+0xa3/0xf5
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88b1090b>] :fsfilt_ldiskfs:fsfilt_ldiskfs_commit_wait+0xab/0xd0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88b51184>] :obdfilter:filter_commitrw_write+0x1e14/0x2dd0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff886ef3a6>] :lnet:lnet_ni_send+0x96/0xe0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88aebe72>] :ost:ost_checksum_bulk+0x2e2/0x5a0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88af2d29>] :ost:ost_brw_write+0x1c99/0x2480
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88800aa8>] :ptlrpc:ptlrpc_send_reply+0x5e8/0x600
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff887cb8b0>] :ptlrpc:target_committed_to_req+0x40/0x120
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88805098>] :ptlrpc:lustre_msg_check_version_v2+0x8/0x20
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88af60be>] :ost:ost_handle+0x2bae/0x55b0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88756cd0>] :obdclass:class_handle2object+0xe0/0x170
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff887bf19a>] :ptlrpc:lock_res_and_lock+0xba/0xd0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff887c4168>] :ptlrpc:__ldlm_handle2lock+0x2f8/0x360
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff888146c1>] :ptlrpc:ptlrpc_server_handle_request+0x981/0xe00
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88814e25>] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff88815d66>] :ptlrpc:ptlrpc_main+0xf16/0x10e0
Dec 10 18:57:39 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 18:57:40 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133459.30060
Dec 10 18:57:40 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133459.29863
Dec 10 18:57:40 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133459.29865
Dec 10 18:57:40 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133459.29992
Dec 10 18:57:40 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133459.29778
Dec 10 18:57:40 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133459.29870
Dec 10 18:57:40 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133459.29986
Dec 10 18:57:40 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133460.29946
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88814e50>] :ptlrpc:ptlrpc_main+0x0/0x10e0
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 18:57:40 t2s007037 kernel: 
Dec 10 18:57:40 t2s007037 kernel: INFO: task ll_ost_io_09:16612 blocked for more than 120 seconds.
Dec 10 18:57:40 t2s007037 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 10 18:57:40 t2s007037 kernel: ll_ost_io_09  D ffff81032a8a2e20     0 16612      1         16613 16611 (L-TLB)
Dec 10 18:57:40 t2s007037 kernel:  ffff8106176af800 0000000000000046 000000011c465d98 ffff8105f32798c0
Dec 10 18:57:40 t2s007037 kernel:  0000000100000000 000000000000000a ffff81061a5c80c0 ffff81061fe86080
Dec 10 18:57:40 t2s007037 kernel:  0017bf184728a8e1 00000000000008b2 ffff81061a5c82a8 00000009ffffffff
Dec 10 18:57:40 t2s007037 kernel: Call Trace:
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88a65828>] :jbd2:jbd2_log_wait_commit+0xa3/0xf5
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88b1090b>] :fsfilt_ldiskfs:fsfilt_ldiskfs_commit_wait+0xab/0xd0
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88b51184>] :obdfilter:filter_commitrw_write+0x1e14/0x2dd0
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88af2d29>] :ost:ost_brw_write+0x1c99/0x2480
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88800aa8>] :ptlrpc:ptlrpc_send_reply+0x5e8/0x600
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff887cb8b0>] :ptlrpc:target_committed_to_req+0x40/0x120
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88805098>] :ptlrpc:lustre_msg_check_version_v2+0x8/0x20
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88af60be>] :ost:ost_handle+0x2bae/0x55b0
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff888146c1>] :ptlrpc:ptlrpc_server_handle_request+0x981/0xe00
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88814e25>] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88815d66>] :ptlrpc:ptlrpc_main+0xf16/0x10e0
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88814e50>] :ptlrpc:ptlrpc_main+0x0/0x10e0
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 18:57:40 t2s007037 kernel: 
Dec 10 18:57:40 t2s007037 kernel: INFO: task ll_ost_io_10:16613 blocked for more than 120 seconds.
Dec 10 18:57:40 t2s007037 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 10 18:57:40 t2s007037 kernel: ll_ost_io_10  D 0000000000000000     0 16613      1         16614 16612 (L-TLB)
Dec 10 18:57:40 t2s007037 kernel:  ffff810616129800 0000000000000046 00000001e50f5e58 ffff810054946e40
Dec 10 18:57:40 t2s007037 kernel:  0000000100000000 000000000000000a ffff81061b8ee860 ffff81019fa3d860
Dec 10 18:57:40 t2s007037 kernel:  0017bf153b30230e 00000000000009df ffff81061b8eea48 0000000affffffff
Dec 10 18:57:40 t2s007037 kernel: Call Trace:
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff8002deea>] __wake_up+0x38/0x4f
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88a65828>] :jbd2:jbd2_log_wait_commit+0xa3/0xf5
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88b1090b>] :fsfilt_ldiskfs:fsfilt_ldiskfs_commit_wait+0xab/0xd0
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88b51184>] :obdfilter:filter_commitrw_write+0x1e14/0x2dd0
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff886ef3a6>] :lnet:lnet_ni_send+0x96/0xe0
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88af2d29>] :ost:ost_brw_write+0x1c99/0x2480
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88800aa8>] :ptlrpc:ptlrpc_send_reply+0x5e8/0x600
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff887cb8b0>] :ptlrpc:target_committed_to_req+0x40/0x120
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88805098>] :ptlrpc:lustre_msg_check_version_v2+0x8/0x20
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88af60be>] :ost:ost_handle+0x2bae/0x55b0
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88756cd0>] :obdclass:class_handle2object+0xe0/0x170
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff887bf19a>] :ptlrpc:lock_res_and_lock+0xba/0xd0
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff887c4168>] :ptlrpc:__ldlm_handle2lock+0x2f8/0x360
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff888146c1>] :ptlrpc:ptlrpc_server_handle_request+0x981/0xe00
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88814e25>] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88815d66>] :ptlrpc:ptlrpc_main+0xf16/0x10e0
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88814e50>] :ptlrpc:ptlrpc_main+0x0/0x10e0
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 18:57:40 t2s007037 kernel: 
Dec 10 18:57:40 t2s007037 kernel: INFO: task ll_ost_io_13:16616 blocked for more than 120 seconds.
Dec 10 18:57:40 t2s007037 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 10 18:57:40 t2s007037 kernel: ll_ost_io_13  D 0000000000000000     0 16616      1         16617 16615 (L-TLB)
Dec 10 18:57:40 t2s007037 kernel:  ffff81061886d800 0000000000000046 0000008b562b6a08 ffff8103acc8bd40
Dec 10 18:57:40 t2s007037 kernel:  0000001500000000 000000000000000a ffff81061a544040 ffff81019dac40c0
Dec 10 18:57:40 t2s007037 kernel:  0017bf14c64ef7b7 0000000000000ef0 ffff81061a544228 00000005ffffffff
Dec 10 18:57:40 t2s007037 kernel: Call Trace:
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88a65828>] :jbd2:jbd2_log_wait_commit+0xa3/0xf5
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88b1090b>] :fsfilt_ldiskfs:fsfilt_ldiskfs_commit_wait+0xab/0xd0
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88b51184>] :obdfilter:filter_commitrw_write+0x1e14/0x2dd0
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff886ef3a6>] :lnet:lnet_ni_send+0x96/0xe0
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88aebec0>] :ost:ost_checksum_bulk+0x330/0x5a0
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88af2d29>] :ost:ost_brw_write+0x1c99/0x2480
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88800aa8>] :ptlrpc:ptlrpc_send_reply+0x5e8/0x600
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff887cb8b0>] :ptlrpc:target_committed_to_req+0x40/0x120
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88805098>] :ptlrpc:lustre_msg_check_version_v2+0x8/0x20
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88af60be>] :ost:ost_handle+0x2bae/0x55b0
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88756cd0>] :obdclass:class_handle2object+0xe0/0x170
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff887bf19a>] :ptlrpc:lock_res_and_lock+0xba/0xd0
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff887c4168>] :ptlrpc:__ldlm_handle2lock+0x2f8/0x360
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff888146c1>] :ptlrpc:ptlrpc_server_handle_request+0x981/0xe00
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88814e25>] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88815d66>] :ptlrpc:ptlrpc_main+0xf16/0x10e0
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88814e50>] :ptlrpc:ptlrpc_main+0x0/0x10e0
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 18:57:40 t2s007037 kernel: 
Dec 10 18:57:40 t2s007037 kernel: INFO: task ll_ost_io_15:16618 blocked for more than 120 seconds.
Dec 10 18:57:40 t2s007037 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 10 18:57:40 t2s007037 kernel: ll_ost_io_15  D ffff81032a889aa0     0 16618      1         16619 16617 (L-TLB)
Dec 10 18:57:40 t2s007037 kernel:  ffff81061970d800 0000000000000046 00000001562b6c48 ffff810402b358c0
Dec 10 18:57:40 t2s007037 kernel:  0000000100000000 000000000000000a ffff81061b315080 ffff81061ff2f100
Dec 10 18:57:40 t2s007037 kernel:  0017bf16bf7ee560 0000000000001166 ffff81061b315268 00000003ffffffff
Dec 10 18:57:40 t2s007037 kernel: Call Trace:
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88a65828>] :jbd2:jbd2_log_wait_commit+0xa3/0xf5
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88b1090b>] :fsfilt_ldiskfs:fsfilt_ldiskfs_commit_wait+0xab/0xd0
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88b51184>] :obdfilter:filter_commitrw_write+0x1e14/0x2dd0
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88af2d29>] :ost:ost_brw_write+0x1c99/0x2480
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88800aa8>] :ptlrpc:ptlrpc_send_reply+0x5e8/0x600
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff887cb8b0>] :ptlrpc:target_committed_to_req+0x40/0x120
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88805098>] :ptlrpc:lustre_msg_check_version_v2+0x8/0x20
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88af60be>] :ost:ost_handle+0x2bae/0x55b0
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88756cd0>] :obdclass:class_handle2object+0xe0/0x170
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff887bf19a>] :ptlrpc:lock_res_and_lock+0xba/0xd0
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff887c4168>] :ptlrpc:__ldlm_handle2lock+0x2f8/0x360
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff888146c1>] :ptlrpc:ptlrpc_server_handle_request+0x981/0xe00
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88814e25>] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88815d66>] :ptlrpc:ptlrpc_main+0xf16/0x10e0
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff88814e50>] :ptlrpc:ptlrpc_main+0x0/0x10e0
Dec 10 18:57:40 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 18:57:40 t2s007037 kernel: 
Dec 10 18:57:42 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133462.30033
Dec 10 18:57:45 t2s007037 kernel: Lustre: Service thread pid 29811 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Dec 10 18:57:45 t2s007037 kernel: Lustre: Skipped 25 previous similar messages
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.29811
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.30081
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.16713
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.30018
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.29707
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.29952
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.29775
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.16612
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.29925
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.16640
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.30017
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.29953
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.29797
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.30026
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.30083
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.16609
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.30002
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.29917
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.29851
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.16620
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.30050
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.16654
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.29862
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.29950
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.16715
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.16633
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.29817
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.30070
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.29742
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.29978
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.29968
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.29846
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.30032
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.29768
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.16726
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.29895
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.29975
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.29790
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.30075
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.29727
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.30010
Dec 10 18:57:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.16696
Dec 10 18:57:46 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133465.29714
Dec 10 18:57:46 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133466.30090
Dec 10 18:57:46 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133466.16666
Dec 10 18:57:46 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133466.16650
Dec 10 18:57:46 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133466.30087
Dec 10 18:57:47 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133467.30035
Dec 10 18:57:47 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133467.30079
Dec 10 18:57:47 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133467.16698
Dec 10 18:57:47 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133467.29939
Dec 10 18:57:47 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133467.29935
Dec 10 18:57:47 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133467.29928
Dec 10 18:57:47 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133467.16719
Dec 10 18:57:47 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133467.16678
Dec 10 18:57:48 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133468.30014
Dec 10 18:57:48 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133468.29971
Dec 10 18:57:48 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133468.30061
Dec 10 18:57:48 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133468.30056
Dec 10 18:57:48 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133468.29783
Dec 10 18:57:48 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133468.29736
Dec 10 18:57:48 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133468.30036
Dec 10 18:57:48 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133468.29880
Dec 10 18:57:49 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133469.30085
Dec 10 18:57:49 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133469.30053
Dec 10 18:57:49 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133469.29983
Dec 10 18:57:49 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133469.29998
Dec 10 18:57:49 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133469.29737
Dec 10 18:57:49 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133469.29920
Dec 10 18:57:49 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133469.30074
Dec 10 18:57:49 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133469.29857
Dec 10 18:57:50 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133470.29887
Dec 10 18:57:50 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133470.29719
Dec 10 18:57:50 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133470.16689
Dec 10 18:57:50 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133470.29765
Dec 10 18:57:50 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133470.29906
Dec 10 18:57:50 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133470.29832
Dec 10 18:57:50 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133470.16634
Dec 10 18:57:50 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133470.30020
Dec 10 18:57:52 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133472.16701
Dec 10 18:57:52 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133472.29795
Dec 10 18:57:52 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133472.29731
Dec 10 18:57:52 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133472.29815
Dec 10 18:57:52 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133472.29766
Dec 10 18:57:52 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133472.16712
Dec 10 18:57:52 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133472.16638
Dec 10 18:57:52 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133472.30022
Dec 10 18:57:52 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133472.30076
Dec 10 18:57:52 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133472.29965
Dec 10 18:57:52 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133472.16652
Dec 10 18:57:52 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133472.30005
Dec 10 18:57:52 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133472.29873
Dec 10 18:57:52 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133472.29834
Dec 10 18:57:52 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133472.16710
Dec 10 18:57:52 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133472.29762
Dec 10 18:57:56 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133476.29859
Dec 10 18:57:56 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133476.29753
Dec 10 18:57:56 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133476.16663
Dec 10 18:57:56 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133476.30045
Dec 10 18:57:56 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133476.29896
Dec 10 18:57:56 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133476.29800
Dec 10 18:57:56 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133476.29744
Dec 10 18:57:56 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133476.29712
Dec 10 18:57:57 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133477.16692
Dec 10 18:57:57 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133477.29898
Dec 10 18:57:57 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133477.16687
Dec 10 18:57:57 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133477.16691
Dec 10 18:57:57 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133477.29829
Dec 10 18:57:57 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133477.29754
Dec 10 18:57:57 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133477.16656
Dec 10 18:57:57 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133477.29802
Dec 10 18:57:58 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133478.16628
Dec 10 18:57:58 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133478.16635
Dec 10 18:57:58 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133478.30041
Dec 10 18:57:58 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133478.29729
Dec 10 18:57:58 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133478.29912
Dec 10 18:57:58 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133478.29856
Dec 10 18:57:58 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133478.16636
Dec 10 18:57:58 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133478.29806
Dec 10 18:57:58 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133478.29709
Dec 10 18:57:58 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133478.30062
Dec 10 18:57:58 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133478.16637
Dec 10 18:57:58 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133478.16655
Dec 10 18:57:58 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133478.30040
Dec 10 18:57:58 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133478.29942
Dec 10 18:57:58 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133478.29724
Dec 10 18:57:58 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133478.29969
Dec 10 18:57:58 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133478.29910
Dec 10 18:57:59 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133479.29981
Dec 10 18:57:59 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133479.30001
Dec 10 18:57:59 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133479.29999
Dec 10 18:57:59 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133479.16641
Dec 10 18:57:59 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133479.29849
Dec 10 18:57:59 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133479.29732
Dec 10 18:57:59 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133479.29875
Dec 10 18:57:59 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133479.29728
Dec 10 18:58:03 t2s007037 kernel: Lustre: Service thread pid 30043 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Dec 10 18:58:03 t2s007037 kernel: Lustre: Skipped 135 previous similar messages
Dec 10 18:58:03 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133483.30043
Dec 10 18:58:10 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133490.16717
Dec 10 18:58:10 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133490.16669
Dec 10 18:58:10 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133490.29943
Dec 10 18:58:10 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133490.29979
Dec 10 18:58:10 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133490.29903
Dec 10 18:58:10 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133490.29760
Dec 10 18:58:10 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133490.30019
Dec 10 18:58:10 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133490.30054
Dec 10 18:58:14 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133494.29967
Dec 10 18:58:14 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133494.29918
Dec 10 18:58:14 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133494.30023
Dec 10 18:58:14 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133494.16690
Dec 10 18:58:14 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133494.29911
Dec 10 18:58:14 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133494.16670
Dec 10 18:58:14 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133494.30080
Dec 10 18:58:14 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133494.30048
Dec 10 18:58:18 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133498.30034
Dec 10 18:58:18 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133498.29835
Dec 10 18:58:18 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133498.29809
Dec 10 18:58:18 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133498.29836
Dec 10 18:58:18 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133498.30086
Dec 10 18:58:18 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133498.29844
Dec 10 18:58:18 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133498.16631
Dec 10 18:58:18 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133498.29936
Dec 10 18:58:18 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133498.29878
Dec 10 18:58:18 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133498.29868
Dec 10 18:58:18 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133498.30047
Dec 10 18:58:18 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133498.30071
Dec 10 18:58:24 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133504.30009
Dec 10 18:58:27 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133507.29803
Dec 10 18:58:27 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133507.16604
Dec 10 18:58:27 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133507.29776
Dec 10 18:58:29 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133509.29948
Dec 10 18:58:29 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133509.29784
Dec 10 18:58:29 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133509.29774
Dec 10 18:58:29 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133509.16662
Dec 10 18:58:29 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133509.30038
Dec 10 18:58:29 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133509.29764
Dec 10 18:58:29 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133509.29837
Dec 10 18:58:29 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133509.29860
Dec 10 18:58:30 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133510.29976
Dec 10 18:58:30 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133510.29905
Dec 10 18:58:41 t2s007037 kernel: Lustre: Service thread pid 29828 was inactive for 200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Dec 10 18:58:41 t2s007037 kernel: Lustre: Skipped 42 previous similar messages
Dec 10 18:58:41 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133521.29828
Dec 10 18:58:42 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133522.29782
Dec 10 18:58:54 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133534.29711
Dec 10 18:58:54 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133534.16610
Dec 10 18:58:54 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133534.29954
Dec 10 18:58:54 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133534.29716
Dec 10 18:59:02 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133542.16642
Dec 10 18:59:09 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133549.16611
Dec 10 18:59:12 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133552.29960
Dec 10 18:59:14 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133554.16681
Dec 10 18:59:22 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133562.16703
Dec 10 18:59:23 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133563.29749
Dec 10 18:59:23 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133563.16673
Dec 10 18:59:23 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133563.16730
Dec 10 18:59:23 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133563.30015
Dec 10 19:01:09 t2s007037 kernel: Lustre: Service thread pid 16649 was inactive for 298.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Dec 10 19:01:09 t2s007037 kernel: Lustre: Skipped 14 previous similar messages
Dec 10 19:01:09 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133669.16649
Dec 10 19:01:09 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133669.29995
Dec 10 19:01:20 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133680.29734
Dec 10 19:01:20 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133680.30089
Dec 10 19:01:22 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133682.29826
Dec 10 19:01:42 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133702.29908
Dec 10 19:01:42 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133702.29763
Dec 10 19:01:54 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133714.16718
Dec 10 19:01:54 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133714.16653
Dec 10 19:02:03 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133723.16680
Dec 10 19:02:08 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133728.29718
Dec 10 19:02:08 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133728.30031
Dec 10 19:02:08 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133728.29771
Dec 10 19:02:08 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133728.30068
Dec 10 19:02:08 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133728.29881
Dec 10 19:02:08 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133728.29990
Dec 10 19:02:12 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133732.29825
Dec 10 19:02:12 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133732.16660
Dec 10 19:02:12 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133732.29788
Dec 10 19:02:12 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133732.30065
Dec 10 19:02:12 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133732.29861
Dec 10 19:02:12 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133732.16709
Dec 10 19:02:12 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133732.29899
Dec 10 19:02:49 t2s007037 kernel: Lustre: Service thread pid 16617 was inactive for 298.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Dec 10 19:02:49 t2s007037 kernel: Lustre: Skipped 3 previous similar messages
Dec 10 19:02:49 t2s007037 kernel: Pid: 16617, comm: ll_ost_io_14
Dec 10 19:02:49 t2s007037 kernel: 
Dec 10 19:02:49 t2s007037 kernel: Call Trace:
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:02:49 t2s007037 kernel: 
Dec 10 19:02:49 t2s007037 kernel: Pid: 29820, comm: ll_ost_io_241
Dec 10 19:02:49 t2s007037 kernel: 
Dec 10 19:02:49 t2s007037 kernel: Call Trace:
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:02:49 t2s007037 kernel: 
Dec 10 19:02:49 t2s007037 kernel: Pid: 16725, comm: ll_ost_io_122
Dec 10 19:02:49 t2s007037 kernel: 
Dec 10 19:02:49 t2s007037 kernel: Call Trace:
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff887d9cdd>] ldlm_resource_iterate+0x1ad/0x240 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:02:49 t2s007037 kernel: 
Dec 10 19:02:49 t2s007037 kernel: Pid: 30016, comm: ll_ost_io_437
Dec 10 19:02:49 t2s007037 kernel: 
Dec 10 19:02:49 t2s007037 kernel: Call Trace:
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:02:49 t2s007037 kernel: 
Dec 10 19:02:49 t2s007037 kernel: Pid: 29845, comm: ll_ost_io_266
Dec 10 19:02:49 t2s007037 kernel: 
Dec 10 19:02:49 t2s007037 kernel: Call Trace:
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff8002deea>] __wake_up+0x38/0x4f
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88aebe3a>] ost_checksum_bulk+0x2aa/0x5a0 [ost]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:02:49 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:02:49 t2s007037 kernel: 
Dec 10 19:02:49 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133769.16614
Dec 10 19:02:49 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133769.30055
Dec 10 19:02:49 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133769.16632
Dec 10 19:02:49 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133769.29845
Dec 10 19:02:49 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133769.30016
Dec 10 19:02:49 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133769.16725
Dec 10 19:02:49 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133769.29820
Dec 10 19:02:49 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355133769.16617
Dec 10 19:08:15 t2s007037 kernel: Lustre: Service thread pid 30072 was inactive for 586.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Dec 10 19:08:15 t2s007037 kernel: Lustre: Skipped 4 previous similar messages
Dec 10 19:08:15 t2s007037 kernel: Pid: 30072, comm: ll_ost_io_493
Dec 10 19:08:15 t2s007037 kernel: 
Dec 10 19:08:15 t2s007037 kernel: Call Trace:
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff8002deea>] __wake_up+0x38/0x4f
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:08:15 t2s007037 kernel: 
Dec 10 19:08:15 t2s007037 kernel: Pid: 29988, comm: ll_ost_io_409
Dec 10 19:08:15 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134095.30072
Dec 10 19:08:15 t2s007037 kernel: 
Dec 10 19:08:15 t2s007037 kernel: Call Trace:
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:08:15 t2s007037 kernel: 
Dec 10 19:08:15 t2s007037 kernel: Pid: 29909, comm: ll_ost_io_330
Dec 10 19:08:15 t2s007037 kernel: 
Dec 10 19:08:15 t2s007037 kernel: Call Trace:
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88aebe43>] ost_checksum_bulk+0x2b3/0x5a0 [ost]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:08:15 t2s007037 kernel: 
Dec 10 19:08:15 t2s007037 kernel: Pid: 16706, comm: ll_ost_io_103
Dec 10 19:08:15 t2s007037 kernel: 
Dec 10 19:08:15 t2s007037 kernel: Call Trace:
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:08:15 t2s007037 kernel: 
Dec 10 19:08:15 t2s007037 kernel: Pid: 29889, comm: ll_ost_io_310
Dec 10 19:08:15 t2s007037 kernel: 
Dec 10 19:08:15 t2s007037 kernel: Call Trace:
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88aebf05>] ost_checksum_bulk+0x375/0x5a0 [ost]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:08:15 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:08:15 t2s007037 kernel: 
Dec 10 19:08:15 t2s007037 kernel: Lustre: Service thread pid 29888 was inactive for 586.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Dec 10 19:08:15 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134095.29889
Dec 10 19:08:15 t2s007037 kernel: Lustre: Skipped 25 previous similar messages
Dec 10 19:08:15 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134095.29757
Dec 10 19:08:15 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134095.29843
Dec 10 19:08:15 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134095.29888
Dec 10 19:08:15 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134095.16706
Dec 10 19:08:15 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134095.29909
Dec 10 19:08:15 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134095.29988
Dec 10 19:09:28 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134168.29977
Dec 10 19:09:28 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134168.29973
Dec 10 19:09:28 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134168.29871
Dec 10 19:09:28 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134168.29847
Dec 10 19:09:28 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134168.29938
Dec 10 19:09:28 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134168.29750
Dec 10 19:09:28 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134168.16667
Dec 10 19:09:28 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134168.29932
Dec 10 19:09:29 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134169.16627
Dec 10 19:09:40 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134180.30004
Dec 10 19:10:10 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134210.30063
Dec 10 19:10:10 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134210.29931
Dec 10 19:10:54 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134254.29853
Dec 10 19:11:28 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134288.29970
Dec 10 19:11:28 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134288.29956
Dec 10 19:11:28 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134288.29723
Dec 10 19:11:28 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134288.16605
Dec 10 19:11:28 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134288.29872
Dec 10 19:11:28 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134288.16686
Dec 10 19:11:28 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134288.29822
Dec 10 19:11:28 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134288.16724
Dec 10 19:11:28 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134288.29869
Dec 10 19:11:28 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134288.29949
Dec 10 19:11:45 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134305.30029
Dec 10 19:12:16 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134336.29810
Dec 10 19:12:36 t2s007037 kernel: Lustre: Service thread pid 30067 was inactive for 586.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Dec 10 19:12:36 t2s007037 kernel: Lustre: Skipped 27 previous similar messages
Dec 10 19:12:36 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134356.30067
Dec 10 19:12:46 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134366.16705
Dec 10 19:12:52 t2s007037 kernel: Lustre: 16644:0:(service.c:810:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-527), not sending early reply
Dec 10 19:12:52 t2s007037 kernel:   req@ffff8105d72ec400 x1414129514644714/t0 o4->979feb27-4b84-7d83-5fbc-928d620860f7@NET_0x500000a010106_UUID:0/0 lens 752/488 e 4 to 0 dl 1355134377 ref 2 fl Interpret:/0/0 rc 0/0
Dec 10 19:12:52 t2s007037 kernel: Lustre: 16644:0:(service.c:810:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-527), not sending early reply
Dec 10 19:12:52 t2s007037 kernel:   req@ffff8105ec4de800 x1414129514644713/t0 o4->979feb27-4b84-7d83-5fbc-928d620860f7@NET_0x500000a010106_UUID:0/0 lens 480/424 e 4 to 0 dl 1355134377 ref 2 fl Interpret:/0/0 rc 0/0
Dec 10 19:12:53 t2s007037 kernel: Lustre: 16675:0:(service.c:810:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-527), not sending early reply
Dec 10 19:12:53 t2s007037 kernel:   req@ffff810515c63c00 x1414129282461663/t0 o4->c2064668-8adb-bd5e-e3a5-f1d1c8fd7745@NET_0x500000a010067_UUID:0/0 lens 448/416 e 4 to 0 dl 1355134378 ref 2 fl Interpret:/0/0 rc 0/0
Dec 10 19:12:53 t2s007037 kernel: Lustre: 16675:0:(service.c:810:ptlrpc_at_send_early_reply()) Skipped 10 previous similar messages
Dec 10 19:12:55 t2s007037 kernel: Lustre: 29972:0:(service.c:810:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-527), not sending early reply
Dec 10 19:12:55 t2s007037 kernel:   req@ffff81059df6cc00 x1416143860191304/t0 o4->8e966893-d9e9-3508-a406-c2132095af5f@NET_0x500000a010a54_UUID:0/0 lens 448/416 e 4 to 0 dl 1355134380 ref 2 fl Interpret:/0/0 rc 0/0
Dec 10 19:12:55 t2s007037 kernel: Lustre: 29972:0:(service.c:810:ptlrpc_at_send_early_reply()) Skipped 8 previous similar messages
Dec 10 19:12:58 t2s007037 kernel: Lustre: 30066:0:(service.c:810:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-527), not sending early reply
Dec 10 19:12:58 t2s007037 kernel:   req@ffff8105e6dd2850 x1420041865723750/t0 o4->74aba844-3992-815e-f202-31dd876e10c1@NET_0x20001c0a81436_UUID:0/0 lens 448/416 e 4 to 0 dl 1355134383 ref 2 fl Interpret:/0/0 rc 0/0
Dec 10 19:12:58 t2s007037 kernel: Lustre: 30066:0:(service.c:810:ptlrpc_at_send_early_reply()) Skipped 32 previous similar messages
Dec 10 19:13:02 t2s007037 kernel: Lustre: 29733:0:(service.c:810:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-527), not sending early reply
Dec 10 19:13:02 t2s007037 kernel:   req@ffff810448cc5c00 x1414587154967931/t0 o4->423aea63-3cce-ead8-5c83-0ef4475c96a5@NET_0x20001c0a8133d_UUID:0/0 lens 448/416 e 4 to 0 dl 1355134387 ref 2 fl Interpret:/0/0 rc 0/0
Dec 10 19:13:02 t2s007037 kernel: Lustre: 29733:0:(service.c:810:ptlrpc_at_send_early_reply()) Skipped 46 previous similar messages
Dec 10 19:13:12 t2s007037 kernel: Lustre: 29773:0:(service.c:810:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-527), not sending early reply
Dec 10 19:13:12 t2s007037 kernel:   req@ffff810199dc7000 x1417127812275486/t0 o4->3b4f7930-2d13-e777-c916-8e6af2d7bab0@NET_0x20001c0a8134a_UUID:0/0 lens 448/416 e 4 to 0 dl 1355134397 ref 2 fl Interpret:/0/0 rc 0/0
Dec 10 19:13:12 t2s007037 kernel: Lustre: 29773:0:(service.c:810:ptlrpc_at_send_early_reply()) Skipped 23 previous similar messages
Dec 10 19:13:30 t2s007037 kernel: Lustre: 29951:0:(service.c:810:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-527), not sending early reply
Dec 10 19:13:30 t2s007037 kernel:   req@ffff8100734d1c00 x1414587165688090/t0 o4->3b03e1f8-cd18-8fe7-5ea9-efed1d9ceba7@NET_0x20001c0a81407_UUID:0/0 lens 448/416 e 4 to 0 dl 1355134415 ref 2 fl Interpret:/0/0 rc 0/0
Dec 10 19:13:30 t2s007037 kernel: Lustre: 29951:0:(service.c:810:ptlrpc_at_send_early_reply()) Skipped 137 previous similar messages
Dec 10 19:14:11 t2s007037 kernel: Lustre: 29927:0:(service.c:810:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-342), not sending early reply
Dec 10 19:14:11 t2s007037 kernel:   req@ffff81023da34400 x1414132026348750/t0 o4->68f8ae83-cfd7-a3ff-49d5-04ed7818c536@NET_0x20001c0a81192_UUID:0/0 lens 448/416 e 1 to 0 dl 1355134456 ref 2 fl Interpret:/0/0 rc 0/0
Dec 10 19:14:11 t2s007037 kernel: Lustre: 29927:0:(service.c:810:ptlrpc_at_send_early_reply()) Skipped 51 previous similar messages
Dec 10 19:15:20 t2s007037 kernel: Lustre: 29964:0:(service.c:810:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-450), not sending early reply
Dec 10 19:15:20 t2s007037 kernel:   req@ffff8103bd2c0400 x1414351267114512/t0 o4->e07d0474-1930-2c0d-25f9-a2cba18ca924@NET_0x500000a010472_UUID:0/0 lens 448/416 e 2 to 0 dl 1355134525 ref 2 fl Interpret:/0/0 rc 0/0
Dec 10 19:15:20 t2s007037 kernel: Lustre: 29964:0:(service.c:810:ptlrpc_at_send_early_reply()) Skipped 47 previous similar messages
Dec 10 19:15:28 t2s007037 kernel: Lustre: 16504:0:(ldlm_lib.c:574:target_handle_reconnect()) gscr0-OST0000: 448ded8b-6867-b4e3-b095-24a1194a0311 reconnecting
Dec 10 19:15:28 t2s007037 kernel: Lustre: 16504:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 1 previous similar message
Dec 10 19:15:28 t2s007037 kernel: Lustre: 16504:0:(ldlm_lib.c:874:target_handle_connect()) gscr0-OST0000: refuse reconnection from 448ded8b-6867-b4e3-b095-24a1194a0311@192.168.20.53@tcp1 to 0xffff81060f828e00; still busy with 4 active RPCs
Dec 10 19:15:28 t2s007037 kernel: LustreError: 16504:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16)  req@ffff81050be86c00 x1417217812744450/t0 o8->448ded8b-6867-b4e3-b095-24a1194a0311@NET_0x20001c0a81435_UUID:0/0 lens 368/264 e 0 to 0 dl 1355134628 ref 1 fl Interpret:/0/0 rc -16/0
Dec 10 19:15:28 t2s007037 kernel: LustreError: 16504:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 13 previous similar messages
Dec 10 19:15:28 t2s007037 kernel: Lustre: 20370:0:(ldlm_lib.c:874:target_handle_connect()) gscr0-OST0000: refuse reconnection from 98aabcbb-79bf-0dd8-3a0e-f869054aa095@192.168.19.31@tcp1 to 0xffff81028b12ba00; still busy with 4 active RPCs
Dec 10 19:15:28 t2s007037 kernel: LustreError: 20370:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16)  req@ffff810557260400 x1414587155179057/t0 o8->98aabcbb-79bf-0dd8-3a0e-f869054aa095@NET_0x20001c0a8131f_UUID:0/0 lens 368/264 e 0 to 0 dl 1355134628 ref 1 fl Interpret:/0/0 rc -16/0
Dec 10 19:15:28 t2s007037 kernel: Lustre: 20373:0:(ldlm_lib.c:574:target_handle_reconnect()) gscr0-OST0000: 4ba8bf09-dc6f-b140-8039-c63b190f6496 reconnecting
Dec 10 19:15:28 t2s007037 kernel: Lustre: 20373:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 1 previous similar message
Dec 10 19:15:28 t2s007037 kernel: Lustre: 5499:0:(ldlm_lib.c:874:target_handle_connect()) gscr0-OST0000: refuse reconnection from 1e0c4bbc-b2a9-1268-afaf-811307e85c34@192.168.19.80@tcp1 to 0xffff81006d77b600; still busy with 3 active RPCs
Dec 10 19:15:28 t2s007037 kernel: Lustre: 5499:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 1 previous similar message
Dec 10 19:15:28 t2s007037 kernel: LustreError: 5499:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16)  req@ffff810380d9a000 x1414587160731478/t0 o8->1e0c4bbc-b2a9-1268-afaf-811307e85c34@NET_0x20001c0a81350_UUID:0/0 lens 368/264 e 0 to 0 dl 1355134628 ref 1 fl Interpret:/0/0 rc -16/0
Dec 10 19:15:28 t2s007037 kernel: LustreError: 5499:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1 previous similar message
Dec 10 19:15:28 t2s007037 kernel: Lustre: 5625:0:(ldlm_lib.c:574:target_handle_reconnect()) gscr0-OST0000: 979feb27-4b84-7d83-5fbc-928d620860f7 reconnecting
Dec 10 19:15:28 t2s007037 kernel: Lustre: 5625:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 1 previous similar message
Dec 10 19:15:31 t2s007037 kernel: Lustre: 5534:0:(ldlm_lib.c:574:target_handle_reconnect()) gscr0-OST0000: 8e966893-d9e9-3508-a406-c2132095af5f reconnecting
Dec 10 19:15:31 t2s007037 kernel: Lustre: 5534:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 2 previous similar messages
Dec 10 19:15:31 t2s007037 kernel: Lustre: 5534:0:(ldlm_lib.c:874:target_handle_connect()) gscr0-OST0000: refuse reconnection from 8e966893-d9e9-3508-a406-c2132095af5f@10.1.10.84@o2ib to 0xffff81018682a200; still busy with 8 active RPCs
Dec 10 19:15:31 t2s007037 kernel: Lustre: 5534:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 3 previous similar messages
Dec 10 19:15:31 t2s007037 kernel: LustreError: 5534:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16)  req@ffff81052ab13000 x1416143860214663/t0 o8->8e966893-d9e9-3508-a406-c2132095af5f@NET_0x500000a010a54_UUID:0/0 lens 368/264 e 0 to 0 dl 1355134631 ref 1 fl Interpret:/0/0 rc -16/0
Dec 10 19:15:31 t2s007037 kernel: LustreError: 5534:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 3 previous similar messages
Dec 10 19:15:33 t2s007037 kernel: Lustre: 16481:0:(ldlm_lib.c:574:target_handle_reconnect()) gscr0-OST0000: c2b698fa-a4d9-ff0c-6dc5-298134339777 reconnecting
Dec 10 19:15:33 t2s007037 kernel: Lustre: 16481:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 3 previous similar messages
Dec 10 19:15:33 t2s007037 kernel: Lustre: 16481:0:(ldlm_lib.c:874:target_handle_connect()) gscr0-OST0000: refuse reconnection from c2b698fa-a4d9-ff0c-6dc5-298134339777@192.168.19.50@tcp1 to 0xffff8100d2fb0400; still busy with 8 active RPCs
Dec 10 19:15:33 t2s007037 kernel: Lustre: 16481:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 3 previous similar messages
Dec 10 19:15:33 t2s007037 kernel: LustreError: 16481:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16)  req@ffff8104a8b3a800 x1414587156339977/t0 o8->c2b698fa-a4d9-ff0c-6dc5-298134339777@NET_0x20001c0a81332_UUID:0/0 lens 368/264 e 0 to 0 dl 1355134633 ref 1 fl Interpret:/0/0 rc -16/0
Dec 10 19:15:33 t2s007037 kernel: LustreError: 16481:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 3 previous similar messages
Dec 10 19:15:37 t2s007037 kernel: Lustre: 5623:0:(ldlm_lib.c:574:target_handle_reconnect()) gscr0-OST0000: 787b7491-30e8-5a36-f5f1-867506b1a959 reconnecting
Dec 10 19:15:37 t2s007037 kernel: Lustre: 5623:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 12 previous similar messages
Dec 10 19:15:37 t2s007037 kernel: Lustre: 5623:0:(ldlm_lib.c:874:target_handle_connect()) gscr0-OST0000: refuse reconnection from 787b7491-30e8-5a36-f5f1-867506b1a959@192.168.19.140@tcp1 to 0xffff8103afb20a00; still busy with 8 active RPCs
Dec 10 19:15:37 t2s007037 kernel: Lustre: 5623:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 12 previous similar messages
Dec 10 19:15:37 t2s007037 kernel: LustreError: 5623:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16)  req@ffff810552950400 x1416760845671162/t0 o8->787b7491-30e8-5a36-f5f1-867506b1a959@NET_0x20001c0a8138c_UUID:0/0 lens 368/264 e 0 to 0 dl 1355134637 ref 1 fl Interpret:/0/0 rc -16/0
Dec 10 19:15:37 t2s007037 kernel: LustreError: 5623:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 12 previous similar messages
Dec 10 19:15:46 t2s007037 kernel: Lustre: 5564:0:(ldlm_lib.c:574:target_handle_reconnect()) gscr0-OST0000: 788f1faa-691b-fc38-68e1-073c20845376 reconnecting
Dec 10 19:15:46 t2s007037 kernel: Lustre: 5564:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 28 previous similar messages
Dec 10 19:15:46 t2s007037 kernel: Lustre: 5564:0:(ldlm_lib.c:874:target_handle_connect()) gscr0-OST0000: refuse reconnection from 788f1faa-691b-fc38-68e1-073c20845376@10.1.0.122@o2ib to 0xffff81029dd42000; still busy with 8 active RPCs
Dec 10 19:15:46 t2s007037 kernel: Lustre: 5564:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 28 previous similar messages
Dec 10 19:15:46 t2s007037 kernel: LustreError: 5564:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16)  req@ffff8100cbf1f400 x1414129272668717/t0 o8->788f1faa-691b-fc38-68e1-073c20845376@NET_0x500000a01007a_UUID:0/0 lens 368/264 e 0 to 0 dl 1355134646 ref 1 fl Interpret:/0/0 rc -16/0
Dec 10 19:15:46 t2s007037 kernel: LustreError: 5564:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 28 previous similar messages
Dec 10 19:16:02 t2s007037 kernel: Lustre: 20378:0:(ldlm_lib.c:574:target_handle_reconnect()) gscr0-OST0000: 74aba844-3992-815e-f202-31dd876e10c1 reconnecting
Dec 10 19:16:02 t2s007037 kernel: Lustre: 20378:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 83 previous similar messages
Dec 10 19:16:02 t2s007037 kernel: Lustre: 20378:0:(ldlm_lib.c:874:target_handle_connect()) gscr0-OST0000: refuse reconnection from 74aba844-3992-815e-f202-31dd876e10c1@192.168.20.54@tcp1 to 0xffff8106077b1800; still busy with 8 active RPCs
Dec 10 19:16:02 t2s007037 kernel: Lustre: 20378:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 83 previous similar messages
Dec 10 19:16:02 t2s007037 kernel: LustreError: 20378:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16)  req@ffff81041bc89000 x1420041865732923/t0 o8->74aba844-3992-815e-f202-31dd876e10c1@NET_0x20001c0a81436_UUID:0/0 lens 368/264 e 0 to 0 dl 1355134662 ref 1 fl Interpret:/0/0 rc -16/0
Dec 10 19:16:02 t2s007037 kernel: LustreError: 20378:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 83 previous similar messages
Dec 10 19:16:34 t2s007037 kernel: Lustre: 5574:0:(ldlm_lib.c:574:target_handle_reconnect()) gscr0-OST0000: aa158da6-9007-6b6d-3baf-c8a3f04557a3 reconnecting
Dec 10 19:16:34 t2s007037 kernel: Lustre: 5574:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 218 previous similar messages
Dec 10 19:16:34 t2s007037 kernel: Lustre: 5574:0:(ldlm_lib.c:874:target_handle_connect()) gscr0-OST0000: refuse reconnection from aa158da6-9007-6b6d-3baf-c8a3f04557a3@10.1.1.100@o2ib to 0xffff810606f64200; still busy with 8 active RPCs
Dec 10 19:16:34 t2s007037 kernel: Lustre: 5574:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 218 previous similar messages
Dec 10 19:16:34 t2s007037 kernel: LustreError: 5574:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16)  req@ffff8103c8e20400 x1414159786402702/t0 o8->aa158da6-9007-6b6d-3baf-c8a3f04557a3@NET_0x500000a010164_UUID:0/0 lens 368/264 e 0 to 0 dl 1355134694 ref 1 fl Interpret:/0/0 rc -16/0
Dec 10 19:16:34 t2s007037 kernel: LustreError: 5574:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 218 previous similar messages
Dec 10 19:17:38 t2s007037 kernel: Lustre: 16578:0:(ldlm_lib.c:574:target_handle_reconnect()) gscr0-OST0000: 2a17f35c-8700-2d15-a1d2-6e0bbe6b3de0 reconnecting
Dec 10 19:17:38 t2s007037 kernel: Lustre: 16578:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 524 previous similar messages
Dec 10 19:17:38 t2s007037 kernel: Lustre: 16578:0:(ldlm_lib.c:874:target_handle_connect()) gscr0-OST0000: refuse reconnection from 2a17f35c-8700-2d15-a1d2-6e0bbe6b3de0@10.1.1.84@o2ib to 0xffff810612fcce00; still busy with 1 active RPCs
Dec 10 19:17:38 t2s007037 kernel: Lustre: 16578:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 524 previous similar messages
Dec 10 19:17:38 t2s007037 kernel: LustreError: 16578:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16)  req@ffff8105ab777c00 x1414129672482999/t0 o8->2a17f35c-8700-2d15-a1d2-6e0bbe6b3de0@NET_0x500000a010154_UUID:0/0 lens 368/264 e 0 to 0 dl 1355134758 ref 1 fl Interpret:/0/0 rc -16/0
Dec 10 19:17:38 t2s007037 kernel: LustreError: 16578:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 524 previous similar messages
Dec 10 19:18:18 t2s007037 kernel: Lustre: 29833:0:(service.c:810:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-126), not sending early reply
Dec 10 19:18:18 t2s007037 kernel:   req@ffff810440400400 x1416143757921665/t0 o4->2b0e7f2c-1cf0-9f72-c976-d8522351a368@NET_0x500000a010a58_UUID:0/0 lens 464/416 e 0 to 0 dl 1355134703 ref 2 fl Interpret:/0/0 rc 0/0
Dec 10 19:18:18 t2s007037 kernel: Lustre: 29833:0:(service.c:810:ptlrpc_at_send_early_reply()) Skipped 22 previous similar messages
Dec 10 19:19:46 t2s007037 kernel: Lustre: 20296:0:(ldlm_lib.c:574:target_handle_reconnect()) gscr0-OST0000: 3372f99e-7fe3-ae4e-7842-73dd74dc4280 reconnecting
Dec 10 19:19:46 t2s007037 kernel: Lustre: 20296:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 1159 previous similar messages
Dec 10 19:19:46 t2s007037 kernel: Lustre: 20296:0:(ldlm_lib.c:874:target_handle_connect()) gscr0-OST0000: refuse reconnection from 3372f99e-7fe3-ae4e-7842-73dd74dc4280@192.168.19.84@tcp1 to 0xffff81058ed78e00; still busy with 8 active RPCs
Dec 10 19:19:46 t2s007037 kernel: Lustre: 20296:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 1159 previous similar messages
Dec 10 19:19:46 t2s007037 kernel: LustreError: 20296:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16)  req@ffff81061b0bdc50 x1417212233980498/t0 o8->3372f99e-7fe3-ae4e-7842-73dd74dc4280@NET_0x20001c0a81354_UUID:0/0 lens 368/264 e 0 to 0 dl 1355134886 ref 1 fl Interpret:/0/0 rc -16/0
Dec 10 19:19:46 t2s007037 kernel: LustreError: 20296:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 1159 previous similar messages
Dec 10 19:22:38 t2s007037 kernel: Lustre: 29940:0:(service.c:810:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-126), not sending early reply
Dec 10 19:22:38 t2s007037 kernel:   req@ffff8106132c4400 x1415949406012442/t0 o4->9ffa0887-a81f-c271-a9a8-eb67bb9ffab2@NET_0x20001c0a81197_UUID:0/0 lens 448/416 e 0 to 0 dl 1355134963 ref 2 fl Interpret:/0/0 rc 0/0
Dec 10 19:22:38 t2s007037 kernel: Lustre: 29940:0:(service.c:810:ptlrpc_at_send_early_reply()) Skipped 12 previous similar messages
Dec 10 19:22:57 t2s007037 kernel: Lustre: Service thread pid 29721 was inactive for 1162.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Dec 10 19:22:57 t2s007037 kernel: Lustre: Skipped 4 previous similar messages
Dec 10 19:22:57 t2s007037 kernel: Pid: 29721, comm: ll_ost_io_142
Dec 10 19:22:57 t2s007037 kernel: 
Dec 10 19:22:57 t2s007037 kernel: Call Trace:
Dec 10 19:22:57 t2s007037 kernel:  [<ffffffff8002deea>] __wake_up+0x38/0x4f
Dec 10 19:22:57 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:22:57 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:22:57 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:22:57 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:22:57 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:22:57 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:22:57 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:22:57 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:22:57 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:22:57 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:22:57 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:22:57 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 19:22:57 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 19:22:57 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:22:57 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:22:57 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:22:57 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:22:57 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:22:57 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:22:57 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:22:57 t2s007037 kernel: 
Dec 10 19:22:57 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355134977.29721
Dec 10 19:24:02 t2s007037 kernel: Lustre: 16492:0:(ldlm_lib.c:574:target_handle_reconnect()) gscr0-OST0000: b333d4d5-d606-ae09-795c-07722a3f0388 reconnecting
Dec 10 19:24:02 t2s007037 kernel: Lustre: 16492:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 2555 previous similar messages
Dec 10 19:24:02 t2s007037 kernel: Lustre: 16492:0:(ldlm_lib.c:874:target_handle_connect()) gscr0-OST0000: refuse reconnection from b333d4d5-d606-ae09-795c-07722a3f0388@192.168.19.92@tcp1 to 0xffff8105893d3c00; still busy with 8 active RPCs
Dec 10 19:24:02 t2s007037 kernel: Lustre: 16492:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 2555 previous similar messages
Dec 10 19:24:02 t2s007037 kernel: LustreError: 16492:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16)  req@ffff8103fa1fb000 x1414587167519067/t0 o8->b333d4d5-d606-ae09-795c-07722a3f0388@NET_0x20001c0a8135c_UUID:0/0 lens 368/264 e 0 to 0 dl 1355135142 ref 1 fl Interpret:/0/0 rc -16/0
Dec 10 19:24:02 t2s007037 kernel: LustreError: 16492:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 2555 previous similar messages
Dec 10 19:24:10 t2s007037 kernel: Lustre: Service thread pid 29794 was inactive for 1162.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Dec 10 19:24:10 t2s007037 kernel: Pid: 29794, comm: ll_ost_io_215
Dec 10 19:24:10 t2s007037 kernel: 
Dec 10 19:24:10 t2s007037 kernel: Call Trace:
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff886ef3a6>] lnet_ni_send+0x96/0xe0 [lnet]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:24:10 t2s007037 kernel: 
Dec 10 19:24:10 t2s007037 kernel: Pid: 29997, comm: ll_ost_io_418
Dec 10 19:24:10 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135050.29794
Dec 10 19:24:10 t2s007037 kernel: 
Dec 10 19:24:10 t2s007037 kernel: Call Trace:
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff886ef3a6>] lnet_ni_send+0x96/0xe0 [lnet]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88aebe85>] ost_checksum_bulk+0x2f5/0x5a0 [ost]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:24:10 t2s007037 kernel: 
Dec 10 19:24:10 t2s007037 kernel: Pid: 30084, comm: ll_ost_io_505
Dec 10 19:24:10 t2s007037 kernel: 
Dec 10 19:24:10 t2s007037 kernel: Call Trace:
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff886ef3a6>] lnet_ni_send+0x96/0xe0 [lnet]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:24:10 t2s007037 kernel: 
Dec 10 19:24:10 t2s007037 kernel: Pid: 29864, comm: ll_ost_io_285
Dec 10 19:24:10 t2s007037 kernel: 
Dec 10 19:24:10 t2s007037 kernel: Call Trace:
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88aebe59>] ost_checksum_bulk+0x2c9/0x5a0 [ost]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:24:10 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:24:10 t2s007037 kernel: 
Dec 10 19:24:10 t2s007037 kernel: Lustre: Service thread pid 16683 was inactive for 1162.04s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Dec 10 19:24:10 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135050.29864
Dec 10 19:24:10 t2s007037 kernel: Lustre: Skipped 1 previous similar message
Dec 10 19:24:10 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135050.29924
Dec 10 19:24:10 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135050.16695
Dec 10 19:24:10 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135050.29852
Dec 10 19:24:10 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135050.16683
Dec 10 19:24:10 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135050.30084
Dec 10 19:24:10 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135050.29997
Dec 10 19:25:34 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135134.29746
Dec 10 19:26:05 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135165.29818
Dec 10 19:26:05 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135165.16619
Dec 10 19:26:50 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135210.29761
Dec 10 19:27:06 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135226.16602
Dec 10 19:27:06 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135226.16723
Dec 10 19:27:06 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135226.30073
Dec 10 19:27:06 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135226.30013
Dec 10 19:27:06 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135226.29854
Dec 10 19:27:06 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135226.16639
Dec 10 19:27:06 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135226.29904
Dec 10 19:27:06 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135226.16694
Dec 10 19:28:57 t2s007037 kernel: Lustre: Service thread pid 16643 was inactive for 1162.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Dec 10 19:28:57 t2s007037 kernel: Lustre: Skipped 3 previous similar messages
Dec 10 19:28:57 t2s007037 kernel: Pid: 16643, comm: ll_ost_io_40
Dec 10 19:28:57 t2s007037 kernel: 
Dec 10 19:28:57 t2s007037 kernel: Call Trace:
Dec 10 19:28:57 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:28:57 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:28:57 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:28:57 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:28:57 t2s007037 kernel:  [<ffffffff886ef3a6>] lnet_ni_send+0x96/0xe0 [lnet]
Dec 10 19:28:57 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:28:57 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:28:57 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:28:57 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:28:57 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:28:57 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:28:57 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:28:57 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 19:28:57 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 19:28:57 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:28:57 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:28:57 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:28:57 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:28:57 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:28:57 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:28:57 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:28:57 t2s007037 kernel: 
Dec 10 19:28:57 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135337.16643
Dec 10 19:29:54 t2s007037 kernel: Lustre: Service thread pid 29841 was inactive for 1162.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Dec 10 19:29:54 t2s007037 kernel: Pid: 29841, comm: ll_ost_io_262
Dec 10 19:29:54 t2s007037 kernel: 
Dec 10 19:29:54 t2s007037 kernel: Call Trace:
Dec 10 19:29:54 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:29:54 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:29:54 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:29:54 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:29:54 t2s007037 kernel:  [<ffffffff886ef3a6>] lnet_ni_send+0x96/0xe0 [lnet]
Dec 10 19:29:54 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:29:54 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:29:54 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:29:54 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:29:54 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:29:54 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:29:54 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:29:54 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 19:29:54 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 19:29:54 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:29:54 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:29:54 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:29:54 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:29:54 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:29:54 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:29:54 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:29:54 t2s007037 kernel: 
Dec 10 19:29:54 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135394.29841
Dec 10 19:31:09 t2s007037 kernel: Lustre: Service thread pid 29941 was inactive for 1162.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Dec 10 19:31:09 t2s007037 kernel: Pid: 29941, comm: ll_ost_io_362
Dec 10 19:31:09 t2s007037 kernel: 
Dec 10 19:31:09 t2s007037 kernel: Call Trace:
Dec 10 19:31:09 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:31:09 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:31:09 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:31:09 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:31:09 t2s007037 kernel:  [<ffffffff88aebee2>] ost_checksum_bulk+0x352/0x5a0 [ost]
Dec 10 19:31:09 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:31:09 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:31:09 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:31:09 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:31:09 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:31:09 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:31:09 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:31:09 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 19:31:09 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 19:31:09 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:31:09 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:31:09 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:31:09 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:31:09 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:31:09 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:31:09 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:31:09 t2s007037 kernel: 
Dec 10 19:31:09 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135469.29941
Dec 10 19:31:41 t2s007037 kernel: Lustre: Service thread pid 29799 was inactive for 1162.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Dec 10 19:31:41 t2s007037 kernel: Pid: 29799, comm: ll_ost_io_220
Dec 10 19:31:41 t2s007037 kernel: 
Dec 10 19:31:41 t2s007037 kernel: Call Trace:
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff8002deea>] __wake_up+0x38/0x4f
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff886ef3a6>] lnet_ni_send+0x96/0xe0 [lnet]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:31:41 t2s007037 kernel: 
Dec 10 19:31:41 t2s007037 kernel: Pid: 29929, comm: ll_ost_io_350
Dec 10 19:31:41 t2s007037 kernel: 
Dec 10 19:31:41 t2s007037 kernel: Call Trace:
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:31:41 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135501.29799
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff886ef3a6>] lnet_ni_send+0x96/0xe0 [lnet]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:31:41 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:31:41 t2s007037 kernel: 
Dec 10 19:31:41 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135501.29929
Dec 10 19:32:34 t2s007037 kernel: Lustre: 16598:0:(ldlm_lib.c:574:target_handle_reconnect()) gscr0-OST0000: 8b86339c-e4fc-4ef6-4914-a888f5042702 reconnecting
Dec 10 19:32:34 t2s007037 kernel: Lustre: 16598:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 5441 previous similar messages
Dec 10 19:32:34 t2s007037 kernel: Lustre: 16598:0:(ldlm_lib.c:874:target_handle_connect()) gscr0-OST0000: refuse reconnection from 8b86339c-e4fc-4ef6-4914-a888f5042702@192.168.20.52@tcp1 to 0xffff8105ba45d600; still busy with 8 active RPCs
Dec 10 19:32:34 t2s007037 kernel: Lustre: 16598:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 5441 previous similar messages
Dec 10 19:32:34 t2s007037 kernel: LustreError: 16598:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16)  req@ffff81060dae6050 x1416855867486478/t0 o8->8b86339c-e4fc-4ef6-4914-a888f5042702@NET_0x20001c0a81434_UUID:0/0 lens 368/264 e 0 to 0 dl 1355135654 ref 1 fl Interpret:/0/0 rc -16/0
Dec 10 19:32:34 t2s007037 kernel: LustreError: 16598:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 5441 previous similar messages
Dec 10 19:32:44 t2s007037 kernel: Lustre: 29807:0:(service.c:810:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply
Dec 10 19:32:44 t2s007037 kernel:   req@ffff8105da53c000 x1414587164105238/t0 o4->b2b48f8c-06f1-b077-c4dd-058e5db935e9@NET_0x20001c0a8141b_UUID:0/0 lens 448/416 e 0 to 0 dl 1355135569 ref 2 fl Interpret:/0/0 rc 0/0
Dec 10 19:32:44 t2s007037 kernel: Lustre: 29807:0:(service.c:810:ptlrpc_at_send_early_reply()) Skipped 10 previous similar messages
Dec 10 19:33:06 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135586.16646
Dec 10 19:33:48 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135628.16606
Dec 10 19:34:18 t2s007037 kernel: Lustre: Service thread pid 16644 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Dec 10 19:34:18 t2s007037 kernel: Lustre: Skipped 1 previous similar message
Dec 10 19:34:18 t2s007037 kernel: Pid: 16644, comm: ll_ost_io_41
Dec 10 19:34:18 t2s007037 kernel: 
Dec 10 19:34:18 t2s007037 kernel: Call Trace:
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff886ef3a6>] lnet_ni_send+0x96/0xe0 [lnet]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:34:18 t2s007037 kernel: 
Dec 10 19:34:18 t2s007037 kernel: Pid: 29933, comm: ll_ost_io_354
Dec 10 19:34:18 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135658.16644
Dec 10 19:34:18 t2s007037 kernel: 
Dec 10 19:34:18 t2s007037 kernel: Call Trace:
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff886ef3a6>] lnet_ni_send+0x96/0xe0 [lnet]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:34:18 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:34:18 t2s007037 kernel: 
Dec 10 19:34:18 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135658.29933
Dec 10 19:35:16 t2s007037 kernel: Lustre: Service thread pid 30011 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Dec 10 19:35:16 t2s007037 kernel: Lustre: Skipped 1 previous similar message
Dec 10 19:35:16 t2s007037 kernel: Pid: 30011, comm: ll_ost_io_432
Dec 10 19:35:16 t2s007037 kernel: 
Dec 10 19:35:16 t2s007037 kernel: Call Trace:
Dec 10 19:35:16 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:35:16 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:35:16 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:35:16 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:35:16 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:35:16 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:35:16 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:35:16 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:35:16 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:35:16 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:35:16 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:35:16 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 19:35:16 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 19:35:16 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:35:16 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:35:16 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:35:16 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:35:16 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:35:16 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:35:16 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:35:16 t2s007037 kernel: 
Dec 10 19:35:16 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135716.30011
Dec 10 19:37:57 t2s007037 kernel: Lustre: Service thread pid 29708 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Dec 10 19:37:57 t2s007037 kernel: Pid: 29708, comm: ll_ost_io_129
Dec 10 19:37:57 t2s007037 kernel: 
Dec 10 19:37:57 t2s007037 kernel: Call Trace:
Dec 10 19:37:57 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:37:57 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:37:57 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:37:57 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:37:57 t2s007037 kernel:  [<ffffffff88aebc82>] ost_checksum_bulk+0xf2/0x5a0 [ost]
Dec 10 19:37:57 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:37:57 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:37:57 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:37:57 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:37:57 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:37:57 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:37:57 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:37:57 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 19:37:57 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 19:37:57 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:37:57 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:37:57 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:37:57 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:37:57 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:37:57 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:37:57 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:37:57 t2s007037 kernel: 
Dec 10 19:37:57 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355135877.29708
Dec 10 19:40:14 t2s007037 kernel: Lustre: Service thread pid 29789 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Dec 10 19:40:14 t2s007037 kernel: Pid: 29789, comm: ll_ost_io_210
Dec 10 19:40:14 t2s007037 kernel: 
Dec 10 19:40:14 t2s007037 kernel: Call Trace:
Dec 10 19:40:14 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:40:14 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:40:14 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:40:14 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:40:14 t2s007037 kernel:  [<ffffffff886ef3a6>] lnet_ni_send+0x96/0xe0 [lnet]
Dec 10 19:40:14 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:40:14 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:40:14 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:40:14 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:40:14 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:40:14 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:40:14 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:40:14 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 19:40:14 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 19:40:14 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:40:14 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:40:14 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:40:14 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:40:14 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:40:14 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:40:14 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:40:14 t2s007037 kernel: 
Dec 10 19:40:14 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136014.29789
Dec 10 19:40:22 t2s007037 kernel: Pid: 29752, comm: ll_ost_io_173
Dec 10 19:40:22 t2s007037 kernel: 
Dec 10 19:40:22 t2s007037 kernel: Call Trace:
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff886ef3a6>] lnet_ni_send+0x96/0xe0 [lnet]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:40:22 t2s007037 kernel: 
Dec 10 19:40:22 t2s007037 kernel: Pid: 29801, comm: ll_ost_io_222
Dec 10 19:40:22 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136022.29752
Dec 10 19:40:22 t2s007037 kernel: 
Dec 10 19:40:22 t2s007037 kernel: Call Trace:
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff8002deea>] __wake_up+0x38/0x4f
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff886ef3a6>] lnet_ni_send+0x96/0xe0 [lnet]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:40:22 t2s007037 kernel: 
Dec 10 19:40:22 t2s007037 kernel: Pid: 30006, comm: ll_ost_io_427
Dec 10 19:40:22 t2s007037 kernel: 
Dec 10 19:40:22 t2s007037 kernel: Call Trace:
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff886ef3a6>] lnet_ni_send+0x96/0xe0 [lnet]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff8008e67d>] dequeue_task+0x18/0x37
Dec 10 19:40:22 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:40:23 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:40:23 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:40:23 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:40:23 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:40:23 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:40:23 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:40:23 t2s007037 kernel: 
Dec 10 19:40:23 t2s007037 kernel: Pid: 29715, comm: ll_ost_io_136
Dec 10 19:40:23 t2s007037 kernel: 
Dec 10 19:40:23 t2s007037 kernel: Call Trace:
Dec 10 19:40:23 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:40:23 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:40:23 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:40:23 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:40:23 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:40:23 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:40:23 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:40:23 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:40:23 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:40:23 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:40:23 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:40:23 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:40:23 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:40:23 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:40:23 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:40:23 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:40:23 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:40:23 t2s007037 kernel: 
Dec 10 19:40:23 t2s007037 kernel: Lustre: Service thread pid 29816 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Dec 10 19:40:23 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136022.29715
Dec 10 19:40:23 t2s007037 kernel: Lustre: Skipped 17 previous similar messages
Dec 10 19:40:23 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136022.16629
Dec 10 19:40:23 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136022.29914
Dec 10 19:40:23 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136022.29877
Dec 10 19:40:23 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136022.29816
Dec 10 19:40:23 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136022.30006
Dec 10 19:40:23 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136022.29801
Dec 10 19:42:34 t2s007037 kernel: Lustre: 16479:0:(ldlm_lib.c:574:target_handle_reconnect()) gscr0-OST0000: 4abd2263-110b-d471-616d-3c5a2c361de4 reconnecting
Dec 10 19:42:34 t2s007037 kernel: Lustre: 16479:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 6897 previous similar messages
Dec 10 19:42:34 t2s007037 kernel: Lustre: 16479:0:(ldlm_lib.c:874:target_handle_connect()) gscr0-OST0000: refuse reconnection from 4abd2263-110b-d471-616d-3c5a2c361de4@192.168.20.32@tcp1 to 0xffff810082589c00; still busy with 1 active RPCs
Dec 10 19:42:34 t2s007037 kernel: Lustre: 16479:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 6897 previous similar messages
Dec 10 19:42:34 t2s007037 kernel: LustreError: 16479:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16)  req@ffff810610fbf800 x1414587159840882/t0 o8->4abd2263-110b-d471-616d-3c5a2c361de4@NET_0x20001c0a81420_UUID:0/0 lens 368/264 e 0 to 0 dl 1355136254 ref 1 fl Interpret:/0/0 rc -16/0
Dec 10 19:42:34 t2s007037 kernel: LustreError: 16479:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 6897 previous similar messages
Dec 10 19:43:56 t2s007037 kernel: Lustre: 16615:0:(service.c:810:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply
Dec 10 19:43:56 t2s007037 kernel:   req@ffff8104314fbc00 x1418393284786102/t0 o4->7362542d-178b-b128-704b-0ef37e481a63@NET_0x20001c0a81404_UUID:0/0 lens 448/416 e 0 to 0 dl 1355136241 ref 2 fl Interpret:/0/0 rc 0/0
Dec 10 19:43:56 t2s007037 kernel: Lustre: 16615:0:(service.c:810:ptlrpc_at_send_early_reply()) Skipped 47 previous similar messages
Dec 10 19:44:27 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136267.16700
Dec 10 19:44:58 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136298.29773
Dec 10 19:44:58 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136298.29814
Dec 10 19:48:47 t2s007037 kernel: Lustre: Service thread pid 29824 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Dec 10 19:48:47 t2s007037 kernel: Lustre: Skipped 4 previous similar messages
Dec 10 19:48:47 t2s007037 kernel: Pid: 29824, comm: ll_ost_io_245
Dec 10 19:48:47 t2s007037 kernel: 
Dec 10 19:48:47 t2s007037 kernel: Call Trace:
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:48:47 t2s007037 kernel: 
Dec 10 19:48:47 t2s007037 kernel: Pid: 16674, comm: ll_ost_io_71
Dec 10 19:48:47 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136527.29824
Dec 10 19:48:47 t2s007037 kernel: 
Dec 10 19:48:47 t2s007037 kernel: Call Trace:
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:48:47 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:48:47 t2s007037 kernel: 
Dec 10 19:48:47 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136527.16674
Dec 10 19:48:49 t2s007037 kernel: Pid: 30051, comm: ll_ost_io_472
Dec 10 19:48:49 t2s007037 kernel: 
Dec 10 19:48:49 t2s007037 kernel: Call Trace:
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff886ef3a6>] lnet_ni_send+0x96/0xe0 [lnet]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:48:49 t2s007037 kernel: 
Dec 10 19:48:49 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136529.30051
Dec 10 19:48:49 t2s007037 kernel: Pid: 16651, comm: ll_ost_io_48
Dec 10 19:48:49 t2s007037 kernel: 
Dec 10 19:48:49 t2s007037 kernel: Call Trace:
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:48:49 t2s007037 kernel: 
Dec 10 19:48:49 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136529.16651
Dec 10 19:48:49 t2s007037 kernel: Pid: 16624, comm: ll_ost_io_21
Dec 10 19:48:49 t2s007037 kernel: 
Dec 10 19:48:49 t2s007037 kernel: Call Trace:
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:48:49 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:48:49 t2s007037 kernel: 
Dec 10 19:48:49 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136529.16624
Dec 10 19:48:52 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136532.16685
Dec 10 19:49:00 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136540.30007
Dec 10 19:49:00 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136540.30052
Dec 10 19:49:00 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136540.29756
Dec 10 19:49:01 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136541.29838
Dec 10 19:49:01 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136541.29717
Dec 10 19:49:01 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136541.29922
Dec 10 19:49:02 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136542.30058
Dec 10 19:49:02 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136542.29833
Dec 10 19:49:02 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136542.29940
Dec 10 19:49:02 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136542.29785
Dec 10 19:49:02 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136542.29735
Dec 10 19:49:02 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136542.16677
Dec 10 19:49:02 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136542.29739
Dec 10 19:49:27 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136567.30046
Dec 10 19:49:48 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136588.29972
Dec 10 19:49:48 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136588.29955
Dec 10 19:49:48 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136588.29937
Dec 10 19:49:48 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136588.30064
Dec 10 19:49:48 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136588.29805
Dec 10 19:49:48 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136588.29884
Dec 10 19:49:48 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136588.29894
Dec 10 19:49:48 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136588.29780
Dec 10 19:50:00 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136600.29957
Dec 10 19:50:00 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136600.29777
Dec 10 19:50:00 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136600.16699
Dec 10 19:50:00 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136600.29991
Dec 10 19:50:00 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136600.16714
Dec 10 19:50:00 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136600.30066
Dec 10 19:50:00 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136600.29915
Dec 10 19:50:35 t2s007037 kernel: Lustre: Service thread pid 16658 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Dec 10 19:50:35 t2s007037 kernel: Lustre: Skipped 36 previous similar messages
Dec 10 19:50:35 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136635.16658
Dec 10 19:51:26 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136686.29813
Dec 10 19:52:34 t2s007037 kernel: Lustre: 20349:0:(ldlm_lib.c:574:target_handle_reconnect()) gscr0-OST0000: fa321c09-4142-57f7-bd24-fb8bb669144f reconnecting
Dec 10 19:52:34 t2s007037 kernel: Lustre: 20349:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 8150 previous similar messages
Dec 10 19:52:34 t2s007037 kernel: Lustre: 20349:0:(ldlm_lib.c:874:target_handle_connect()) gscr0-OST0000: refuse reconnection from fa321c09-4142-57f7-bd24-fb8bb669144f@10.1.4.115@o2ib to 0xffff81027da69a00; still busy with 8 active RPCs
Dec 10 19:52:34 t2s007037 kernel: Lustre: 20349:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 8150 previous similar messages
Dec 10 19:52:34 t2s007037 kernel: LustreError: 20349:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16)  req@ffff8105a4880800 x1419946514043359/t0 o8->fa321c09-4142-57f7-bd24-fb8bb669144f@NET_0x500000a010473_UUID:0/0 lens 368/264 e 0 to 0 dl 1355136854 ref 1 fl Interpret:/0/0 rc -16/0
Dec 10 19:52:34 t2s007037 kernel: LustreError: 20349:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 8150 previous similar messages
Dec 10 19:55:23 t2s007037 kernel: Lustre: 29883:0:(service.c:810:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply
Dec 10 19:55:23 t2s007037 kernel:   req@ffff8104956bec00 x1414351449588242/t0 o4->bd0482f5-598c-8354-33a6-24b6bde8aa9f@NET_0x500000a0104a9_UUID:0/0 lens 464/416 e 0 to 0 dl 1355136928 ref 2 fl Interpret:/0/0 rc 0/0
Dec 10 19:55:23 t2s007037 kernel: Lustre: 29883:0:(service.c:810:ptlrpc_at_send_early_reply()) Skipped 22 previous similar messages
Dec 10 19:56:22 t2s007037 kernel: Lustre: Service thread pid 29831 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Dec 10 19:56:22 t2s007037 kernel: Lustre: Skipped 4 previous similar messages
Dec 10 19:56:22 t2s007037 kernel: Pid: 29831, comm: ll_ost_io_252
Dec 10 19:56:22 t2s007037 kernel: 
Dec 10 19:56:22 t2s007037 kernel: Call Trace:
Dec 10 19:56:22 t2s007037 kernel:  [<ffffffff8002deea>] __wake_up+0x38/0x4f
Dec 10 19:56:22 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:56:22 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:56:22 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:56:22 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:56:22 t2s007037 kernel:  [<ffffffff886ef3a6>] lnet_ni_send+0x96/0xe0 [lnet]
Dec 10 19:56:22 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:56:22 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:56:22 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:56:22 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:56:22 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:56:22 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:56:22 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:56:22 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 19:56:22 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 19:56:22 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:56:22 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:56:22 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:56:22 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:56:22 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:56:22 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:56:22 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:56:22 t2s007037 kernel: 
Dec 10 19:56:22 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136982.29831
Dec 10 19:56:23 t2s007037 kernel: Pid: 29867, comm: ll_ost_io_288
Dec 10 19:56:23 t2s007037 kernel: 
Dec 10 19:56:23 t2s007037 kernel: Call Trace:
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:56:23 t2s007037 kernel: 
Dec 10 19:56:23 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136983.29867
Dec 10 19:56:23 t2s007037 kernel: Pid: 29926, comm: ll_ost_io_347
Dec 10 19:56:23 t2s007037 kernel: 
Dec 10 19:56:23 t2s007037 kernel: Call Trace:
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff886ef3a6>] lnet_ni_send+0x96/0xe0 [lnet]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:56:23 t2s007037 kernel: 
Dec 10 19:56:23 t2s007037 kernel: Pid: 29893, comm: ll_ost_io_314
Dec 10 19:56:23 t2s007037 kernel: 
Dec 10 19:56:23 t2s007037 kernel: Call Trace:
Dec 10 19:56:23 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136983.29926
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff886ef3a6>] lnet_ni_send+0x96/0xe0 [lnet]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff8008e67d>] dequeue_task+0x18/0x37
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:56:23 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:56:23 t2s007037 kernel: 
Dec 10 19:56:23 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136983.29893
Dec 10 19:56:24 t2s007037 kernel: Pid: 29807, comm: ll_ost_io_228
Dec 10 19:56:24 t2s007037 kernel: 
Dec 10 19:56:24 t2s007037 kernel: Call Trace:
Dec 10 19:56:24 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 19:56:24 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 19:56:24 t2s007037 kernel:  [<ffffffff88728e61>] obdo_from_inode+0x81/0x1c0 [obdclass]
Dec 10 19:56:24 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 19:56:24 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 19:56:24 t2s007037 kernel:  [<ffffffff886ef3a6>] lnet_ni_send+0x96/0xe0 [lnet]
Dec 10 19:56:24 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 19:56:24 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 19:56:24 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 19:56:24 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 19:56:24 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 19:56:24 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 19:56:24 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 19:56:24 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 19:56:24 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 19:56:24 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 19:56:24 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 19:56:24 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 19:56:24 t2s007037 kernel: 
Dec 10 19:56:24 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136984.29807
Dec 10 19:56:24 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136984.16721
Dec 10 19:56:26 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136986.29901
Dec 10 19:56:31 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136991.30024
Dec 10 19:56:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136993.29710
Dec 10 19:56:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136993.30088
Dec 10 19:56:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136993.30012
Dec 10 19:56:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136993.30003
Dec 10 19:56:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136993.29848
Dec 10 19:56:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136993.29959
Dec 10 19:56:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136993.16711
Dec 10 19:56:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355136993.29927
Dec 10 19:56:52 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355137012.29796
Dec 10 19:57:26 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355137046.29930
Dec 10 19:57:26 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355137046.29745
Dec 10 19:59:03 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355137143.29994
Dec 10 19:59:06 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355137146.29874
Dec 10 19:59:08 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355137148.16626
Dec 10 20:02:34 t2s007037 kernel: Lustre: 20369:0:(ldlm_lib.c:574:target_handle_reconnect()) gscr0-OST0000: 28e9feaa-2f80-fb8c-ff62-951677e05138 reconnecting
Dec 10 20:02:34 t2s007037 kernel: Lustre: 20369:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 9057 previous similar messages
Dec 10 20:02:34 t2s007037 kernel: Lustre: 20369:0:(ldlm_lib.c:874:target_handle_connect()) gscr0-OST0000: refuse reconnection from 28e9feaa-2f80-fb8c-ff62-951677e05138@192.168.18.137@tcp1 to 0xffff810285a63e00; still busy with 8 active RPCs
Dec 10 20:02:34 t2s007037 kernel: Lustre: 20369:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 9057 previous similar messages
Dec 10 20:02:34 t2s007037 kernel: LustreError: 20369:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16)  req@ffff81046b031400 x1414249556603848/t0 o8->28e9feaa-2f80-fb8c-ff62-951677e05138@NET_0x20001c0a81289_UUID:0/0 lens 368/264 e 0 to 0 dl 1355137454 ref 1 fl Interpret:/0/0 rc -16/0
Dec 10 20:02:34 t2s007037 kernel: LustreError: 20369:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 9057 previous similar messages
Dec 10 20:02:53 t2s007037 kernel: Pid: 30039, comm: ll_ost_io_460
Dec 10 20:02:53 t2s007037 kernel: 
Dec 10 20:02:53 t2s007037 kernel: Call Trace:
Dec 10 20:02:53 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 20:02:53 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 20:02:53 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 20:02:53 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 20:02:53 t2s007037 kernel:  [<ffffffff886ef3a6>] lnet_ni_send+0x96/0xe0 [lnet]
Dec 10 20:02:53 t2s007037 kernel:  [<ffffffff88aebe59>] ost_checksum_bulk+0x2c9/0x5a0 [ost]
Dec 10 20:02:53 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 20:02:53 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 20:02:53 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 20:02:53 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 20:02:53 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 20:02:53 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 20:02:53 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 20:02:53 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 20:02:53 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 20:02:53 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 20:02:53 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 20:02:53 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 20:02:53 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 20:02:53 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 20:02:53 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 20:02:53 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 20:02:53 t2s007037 kernel: 
Dec 10 20:02:53 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355137373.30039
Dec 10 20:03:27 t2s007037 kernel: Pid: 29897, comm: ll_ost_io_318
Dec 10 20:03:27 t2s007037 kernel: 
Dec 10 20:03:27 t2s007037 kernel: Call Trace:
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff8002deea>] __wake_up+0x38/0x4f
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff886ef3a6>] lnet_ni_send+0x96/0xe0 [lnet]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 20:03:27 t2s007037 kernel: 
Dec 10 20:03:27 t2s007037 kernel: Pid: 29793, comm: ll_ost_io_214
Dec 10 20:03:27 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355137407.29897
Dec 10 20:03:27 t2s007037 kernel: 
Dec 10 20:03:27 t2s007037 kernel: Call Trace:
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff8002deea>] __wake_up+0x38/0x4f
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff8005bd04>] cache_alloc_refill+0x88/0x188
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 20:03:27 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 20:03:27 t2s007037 kernel: 
Dec 10 20:03:27 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355137407.29793
Dec 10 20:06:06 t2s007037 kernel: Lustre: Service thread pid 30044 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Dec 10 20:06:06 t2s007037 kernel: Lustre: Skipped 7 previous similar messages
Dec 10 20:06:06 t2s007037 kernel: Pid: 30044, comm: ll_ost_io_465
Dec 10 20:06:06 t2s007037 kernel: 
Dec 10 20:06:06 t2s007037 kernel: Call Trace:
Dec 10 20:06:06 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 20:06:06 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 20:06:06 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 20:06:06 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 20:06:06 t2s007037 kernel:  [<ffffffff886ef3a6>] lnet_ni_send+0x96/0xe0 [lnet]
Dec 10 20:06:06 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 20:06:06 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 20:06:06 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 20:06:06 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 20:06:06 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 20:06:06 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 20:06:06 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 20:06:06 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 20:06:06 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 20:06:06 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 20:06:06 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 20:06:06 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 20:06:06 t2s007037 kernel: 
Dec 10 20:06:06 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355137566.30044
Dec 10 20:06:07 t2s007037 kernel: Pid: 29842, comm: ll_ost_io_263
Dec 10 20:06:07 t2s007037 kernel: 
Dec 10 20:06:07 t2s007037 kernel: Call Trace:
Dec 10 20:06:07 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 20:06:07 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 20:06:07 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 20:06:07 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 20:06:07 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 20:06:07 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 20:06:07 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 20:06:07 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 20:06:07 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 20:06:07 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 20:06:07 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 20:06:07 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 20:06:07 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 20:06:07 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 20:06:07 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 20:06:07 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 20:06:07 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 20:06:07 t2s007037 kernel: 
Dec 10 20:06:07 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355137567.29842
Dec 10 20:06:07 t2s007037 kernel: Lustre: Service thread pid 29748 was inactive for 1200.00s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Dec 10 20:06:07 t2s007037 kernel: Lustre: Skipped 18 previous similar messages
Dec 10 20:06:07 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355137567.29748
Dec 10 20:06:42 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355137602.16708
Dec 10 20:06:42 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355137602.16615
Dec 10 20:06:42 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355137602.29733
Dec 10 20:07:00 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355137620.29722
Dec 10 20:07:35 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355137655.29786
Dec 10 20:08:51 t2s007037 kernel: Lustre: 29951:0:(service.c:810:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply
Dec 10 20:08:51 t2s007037 kernel:   req@ffff81018934a400 x1416855865715677/t0 o4->f689bd16-3038-2928-598d-6bc5e65d31d7@NET_0x20001c0a81381_UUID:0/0 lens 464/416 e 0 to 0 dl 1355137736 ref 2 fl Interpret:/0/0 rc 0/0
Dec 10 20:08:51 t2s007037 kernel: Lustre: 29951:0:(service.c:810:ptlrpc_at_send_early_reply()) Skipped 22 previous similar messages
Dec 10 20:09:33 t2s007037 kernel: Pid: 29840, comm: ll_ost_io_261
Dec 10 20:09:33 t2s007037 kernel: 
Dec 10 20:09:33 t2s007037 kernel: Call Trace:
Dec 10 20:09:33 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 20:09:33 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 20:09:33 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 20:09:33 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 20:09:33 t2s007037 kernel:  [<ffffffff88aebedf>] ost_checksum_bulk+0x34f/0x5a0 [ost]
Dec 10 20:09:33 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 20:09:33 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 20:09:33 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 20:09:33 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 20:09:33 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 20:09:33 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 20:09:33 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 20:09:33 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 20:09:33 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 20:09:33 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 20:09:33 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 20:09:33 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 20:09:33 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 20:09:33 t2s007037 kernel: 
Dec 10 20:09:33 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355137773.29840
Dec 10 20:10:07 t2s007037 kernel: Pid: 16659, comm: ll_ost_io_56
Dec 10 20:10:07 t2s007037 kernel: 
Dec 10 20:10:07 t2s007037 kernel: Call Trace:
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff8002deea>] __wake_up+0x38/0x4f
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff886ef3a6>] lnet_ni_send+0x96/0xe0 [lnet]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 20:10:07 t2s007037 kernel: 
Dec 10 20:10:07 t2s007037 kernel: Pid: 29779, comm: ll_ost_io_200
Dec 10 20:10:07 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355137807.16659
Dec 10 20:10:07 t2s007037 kernel: 
Dec 10 20:10:07 t2s007037 kernel: Call Trace:
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff886ef3a6>] lnet_ni_send+0x96/0xe0 [lnet]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 20:10:07 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 20:10:07 t2s007037 kernel: 
Dec 10 20:10:07 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355137807.29779
Dec 10 20:10:10 t2s007037 kernel: Pid: 29913, comm: ll_ost_io_334
Dec 10 20:10:10 t2s007037 kernel: 
Dec 10 20:10:10 t2s007037 kernel: Call Trace:
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff887bf19a>] lock_res_and_lock+0xba/0xd0 [ptlrpc]
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff887c4168>] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc]
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff8008d2a9>] __wake_up_common+0x3e/0x68
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 20:10:10 t2s007037 kernel: 
Dec 10 20:10:10 t2s007037 kernel: Pid: 16648, comm: ll_ost_io_45
Dec 10 20:10:10 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355137810.29913
Dec 10 20:10:10 t2s007037 kernel: 
Dec 10 20:10:10 t2s007037 kernel: Call Trace:
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff886ef3a6>] lnet_ni_send+0x96/0xe0 [lnet]
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 20:10:10 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 20:10:10 t2s007037 kernel: 
Dec 10 20:10:10 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355137810.29980
Dec 10 20:10:10 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355137810.30028
Dec 10 20:10:10 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355137810.30000
Dec 10 20:10:10 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355137810.29821
Dec 10 20:10:10 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355137810.16648
Dec 10 20:11:08 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355137868.30082
Dec 10 20:11:35 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355137895.29900
Dec 10 20:11:35 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355137895.29964
Dec 10 20:12:34 t2s007037 kernel: Lustre: 16498:0:(ldlm_lib.c:574:target_handle_reconnect()) gscr0-OST0000: 7af5e50b-64b4-599f-015c-b06fb40248e6 reconnecting
Dec 10 20:12:34 t2s007037 kernel: Lustre: 16498:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 9734 previous similar messages
Dec 10 20:12:34 t2s007037 kernel: Lustre: 16498:0:(ldlm_lib.c:874:target_handle_connect()) gscr0-OST0000: refuse reconnection from 7af5e50b-64b4-599f-015c-b06fb40248e6@192.168.19.76@tcp1 to 0xffff81003bccb000; still busy with 8 active RPCs
Dec 10 20:12:34 t2s007037 kernel: Lustre: 16498:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 9734 previous similar messages
Dec 10 20:12:34 t2s007037 kernel: LustreError: 16498:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16)  req@ffff810564a51400 x1414587161535658/t0 o8->7af5e50b-64b4-599f-015c-b06fb40248e6@NET_0x20001c0a8134c_UUID:0/0 lens 368/264 e 0 to 0 dl 1355138054 ref 1 fl Interpret:/0/0 rc -16/0
Dec 10 20:12:34 t2s007037 kernel: LustreError: 16498:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 9734 previous similar messages
Dec 10 20:16:21 t2s007037 kernel: Lustre: Service thread pid 29726 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Dec 10 20:16:21 t2s007037 kernel: Lustre: Skipped 6 previous similar messages
Dec 10 20:16:21 t2s007037 kernel: Pid: 29726, comm: ll_ost_io_147
Dec 10 20:16:21 t2s007037 kernel: 
Dec 10 20:16:21 t2s007037 kernel: Call Trace:
Dec 10 20:16:21 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 20:16:21 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 20:16:21 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 20:16:21 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 20:16:21 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 20:16:21 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 20:16:21 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 20:16:21 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 20:16:21 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 20:16:21 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 20:16:21 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 20:16:21 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 20:16:21 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 20:16:21 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 20:16:21 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 20:16:21 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 20:16:21 t2s007037 kernel: 
Dec 10 20:16:21 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355138181.29726
Dec 10 20:16:37 t2s007037 kernel: Pid: 29883, comm: ll_ost_io_304
Dec 10 20:16:37 t2s007037 kernel: 
Dec 10 20:16:37 t2s007037 kernel: Call Trace:
Dec 10 20:16:37 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 20:16:37 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 20:16:37 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 20:16:37 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 20:16:37 t2s007037 kernel:  [<ffffffff886ef3a6>] lnet_ni_send+0x96/0xe0 [lnet]
Dec 10 20:16:37 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 20:16:37 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 20:16:37 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 20:16:37 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 20:16:37 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 20:16:37 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 20:16:37 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 20:16:37 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 20:16:37 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 20:16:37 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 20:16:37 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 20:16:37 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 20:16:37 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 20:16:37 t2s007037 kernel: 
Dec 10 20:16:37 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355138197.29883
Dec 10 20:20:18 t2s007037 kernel: Lustre: 29919:0:(service.c:810:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/0), not sending early reply
Dec 10 20:20:18 t2s007037 kernel:   req@ffff8103ff467800 x1414129472885250/t0 o10->c21a5efd-4148-d7fd-0608-b8510d9c0d7c@NET_0x500000a01006a_UUID:0/0 lens 400/0 e 1 to 0 dl 1355138423 ref 2 fl New:/0/0 rc 0/0
Dec 10 20:20:18 t2s007037 kernel: Lustre: 29919:0:(service.c:810:ptlrpc_at_send_early_reply()) Skipped 4 previous similar messages
Dec 10 20:20:55 t2s007037 kernel: Pid: 16625, comm: ll_ost_io_22
Dec 10 20:20:55 t2s007037 kernel: 
Dec 10 20:20:55 t2s007037 kernel: Call Trace:
Dec 10 20:20:55 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 20:20:55 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 20:20:55 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 20:20:55 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 20:20:55 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 20:20:55 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 20:20:55 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 20:20:55 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 20:20:55 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 20:20:55 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 20:20:55 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 20:20:55 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 20:20:55 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 20:20:55 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 20:20:55 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 20:20:55 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 20:20:55 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 20:20:55 t2s007037 kernel: 
Dec 10 20:20:55 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355138455.16625
Dec 10 20:22:34 t2s007037 kernel: Lustre: 5591:0:(ldlm_lib.c:574:target_handle_reconnect()) gscr0-OST0000: 448ded8b-6867-b4e3-b095-24a1194a0311 reconnecting
Dec 10 20:22:34 t2s007037 kernel: Lustre: 5591:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 9969 previous similar messages
Dec 10 20:22:34 t2s007037 kernel: Lustre: 5591:0:(ldlm_lib.c:874:target_handle_connect()) gscr0-OST0000: refuse reconnection from 448ded8b-6867-b4e3-b095-24a1194a0311@192.168.20.53@tcp1 to 0xffff81060f828e00; still busy with 4 active RPCs
Dec 10 20:22:34 t2s007037 kernel: Lustre: 5591:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 9969 previous similar messages
Dec 10 20:22:34 t2s007037 kernel: LustreError: 5591:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16)  req@ffff81049a83b800 x1417217812772718/t0 o8->448ded8b-6867-b4e3-b095-24a1194a0311@NET_0x20001c0a81435_UUID:0/0 lens 368/264 e 0 to 0 dl 1355138654 ref 1 fl Interpret:/0/0 rc -16/0
Dec 10 20:22:34 t2s007037 kernel: LustreError: 5591:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 9969 previous similar messages
Dec 10 20:25:24 t2s007037 kernel: Pid: 29923, comm: ll_ost_io_344
Dec 10 20:25:24 t2s007037 kernel: 
Dec 10 20:25:24 t2s007037 kernel: Call Trace:
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 20:25:24 t2s007037 kernel: 
Dec 10 20:25:24 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355138724.29923
Dec 10 20:25:24 t2s007037 kernel: Pid: 16621, comm: ll_ost_io_18
Dec 10 20:25:24 t2s007037 kernel: 
Dec 10 20:25:24 t2s007037 kernel: Call Trace:
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff88aebe43>] ost_checksum_bulk+0x2b3/0x5a0 [ost]
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff88756cd0>] class_handle2object+0xe0/0x170 [obdclass]
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 20:25:24 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 20:25:24 t2s007037 kernel: 
Dec 10 20:25:24 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355138724.16621
Dec 10 20:30:18 t2s007037 kernel: Lustre: Service thread pid 29951 was inactive for 1200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Dec 10 20:30:18 t2s007037 kernel: Lustre: Skipped 4 previous similar messages
Dec 10 20:30:18 t2s007037 kernel: Pid: 29951, comm: ll_ost_io_372
Dec 10 20:30:18 t2s007037 kernel: 
Dec 10 20:30:18 t2s007037 kernel: Call Trace:
Dec 10 20:30:18 t2s007037 kernel:  [<ffffffff88a65828>] jbd2_log_wait_commit+0xa3/0xf5 [jbd2]
Dec 10 20:30:18 t2s007037 kernel:  [<ffffffff800a34b7>] autoremove_wake_function+0x0/0x2e
Dec 10 20:30:18 t2s007037 kernel:  [<ffffffff88b1090b>] fsfilt_ldiskfs_commit_wait+0xab/0xd0 [fsfilt_ldiskfs]
Dec 10 20:30:18 t2s007037 kernel:  [<ffffffff88b51184>] filter_commitrw_write+0x1e14/0x2dd0 [obdfilter]
Dec 10 20:30:18 t2s007037 kernel:  [<ffffffff88aebe4b>] ost_checksum_bulk+0x2bb/0x5a0 [ost]
Dec 10 20:30:18 t2s007037 kernel:  [<ffffffff88af2d29>] ost_brw_write+0x1c99/0x2480 [ost]
Dec 10 20:30:18 t2s007037 kernel:  [<ffffffff88800aa8>] ptlrpc_send_reply+0x5e8/0x600 [ptlrpc]
Dec 10 20:30:18 t2s007037 kernel:  [<ffffffff887cb8b0>] target_committed_to_req+0x40/0x120 [ptlrpc]
Dec 10 20:30:18 t2s007037 kernel:  [<ffffffff8008ee84>] default_wake_function+0x0/0xe
Dec 10 20:30:18 t2s007037 kernel:  [<ffffffff88805098>] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc]
Dec 10 20:30:18 t2s007037 kernel:  [<ffffffff88af60be>] ost_handle+0x2bae/0x55b0 [ost]
Dec 10 20:30:18 t2s007037 kernel:  [<ffffffff8005bd04>] cache_alloc_refill+0x88/0x188
Dec 10 20:30:18 t2s007037 kernel:  [<ffffffff888146c1>] ptlrpc_server_handle_request+0x981/0xe00 [ptlrpc]
Dec 10 20:30:18 t2s007037 kernel:  [<ffffffff88814e25>] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc]
Dec 10 20:30:18 t2s007037 kernel:  [<ffffffff88815d66>] ptlrpc_main+0xf16/0x10e0 [ptlrpc]
Dec 10 20:30:18 t2s007037 kernel:  [<ffffffff8005dfb1>] child_rip+0xa/0x11
Dec 10 20:30:18 t2s007037 kernel:  [<ffffffff88814e50>] ptlrpc_main+0x0/0x10e0 [ptlrpc]
Dec 10 20:30:18 t2s007037 kernel:  [<ffffffff8005dfa7>] child_rip+0x0/0x11
Dec 10 20:30:18 t2s007037 kernel: 
Dec 10 20:30:18 t2s007037 kernel: LustreError: dumping log to /tmp/lustre-log.1355139018.29951
Dec 10 20:30:31 t2s007037 kernel: Lustre: 29919:0:(service.c:810:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply
Dec 10 20:30:31 t2s007037 kernel:   req@ffff8105433e0c00 x1397844590933886/t0 o10->63431cc0-58f0-982e-0934-16a0063a4451@NET_0x500000a010705_UUID:0/0 lens 400/0 e 0 to 0 dl 1355139036 ref 2 fl New:/0/0 rc 0/0
Dec 10 20:30:31 t2s007037 kernel: Lustre: 29919:0:(service.c:810:ptlrpc_at_send_early_reply()) Skipped 263 previous similar messages
Dec 10 20:32:34 t2s007037 kernel: Lustre: 16559:0:(ldlm_lib.c:574:target_handle_reconnect()) gscr0-OST0000: 3cf9095b-7be5-0776-246f-18632e1e73eb reconnecting
Dec 10 20:32:34 t2s007037 kernel: Lustre: 16559:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 10266 previous similar messages
Dec 10 20:32:34 t2s007037 kernel: Lustre: 16559:0:(ldlm_lib.c:874:target_handle_connect()) gscr0-OST0000: refuse reconnection from 3cf9095b-7be5-0776-246f-18632e1e73eb@10.1.10.81@o2ib to 0xffff8102b4411000; still busy with 1 active RPCs
Dec 10 20:32:34 t2s007037 kernel: Lustre: 16559:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 10114 previous similar messages
Dec 10 20:32:35 t2s007037 kernel: LustreError: 16518:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16)  req@ffff81047eb17c00 x1415867070824585/t0 o8->10a41932-59c5-59e6-b8f1-e40a38e00e9f@NET_0x500000a010499_UUID:0/0 lens 368/264 e 0 to 0 dl 1355139255 ref 1 fl Interpret:/0/0 rc -16/0
Dec 10 20:32:35 t2s007037 kernel: LustreError: 16518:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 10115 previous similar messages
Dec 10 20:40:35 t2s007037 kernel: Lustre: 16675:0:(service.c:810:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply
Dec 10 20:40:35 t2s007037 kernel:   req@ffff810437a86c00 x1414132018774480/t0 o4->e44932d3-3a26-9fdc-fe13-e7a322267668@NET_0x20001c0a8117c_UUID:0/0 lens 448/0 e 0 to 0 dl 1355139640 ref 2 fl New:/0/0 rc 0/0
Dec 10 20:40:35 t2s007037 kernel: Lustre: 16675:0:(service.c:810:ptlrpc_at_send_early_reply()) Skipped 140 previous similar messages
Dec 10 20:42:34 t2s007037 kernel: Lustre: 16548:0:(ldlm_lib.c:574:target_handle_reconnect()) gscr0-OST0000: 448ded8b-6867-b4e3-b095-24a1194a0311 reconnecting
Dec 10 20:42:34 t2s007037 kernel: Lustre: 16548:0:(ldlm_lib.c:574:target_handle_reconnect()) Skipped 10264 previous similar messages
Dec 10 20:42:34 t2s007037 kernel: Lustre: 16548:0:(ldlm_lib.c:874:target_handle_connect()) gscr0-OST0000: refuse reconnection from 448ded8b-6867-b4e3-b095-24a1194a0311@192.168.20.53@tcp1 to 0xffff81060f828e00; still busy with 4 active RPCs
Dec 10 20:42:34 t2s007037 kernel: Lustre: 16548:0:(ldlm_lib.c:874:target_handle_connect()) Skipped 10103 previous similar messages
Dec 10 20:42:35 t2s007037 kernel: LustreError: 5538:0:(ldlm_lib.c:1919:target_send_reply_msg()) @@@ processing error (-16)  req@ffff81049018f400 x1414587156376500/t0 o8->c2b698fa-a4d9-ff0c-6dc5-298134339777@NET_0x20001c0a81332_UUID:0/0 lens 368/264 e 0 to 0 dl 1355139855 ref 1 fl Interpret:/0/0 rc -16/0
Dec 10 20:42:35 t2s007037 kernel: LustreError: 5538:0:(ldlm_lib.c:1919:target_send_reply_msg()) Skipped 10104 previous similar messages