Jan 5 05:09:44 oss7 kernel: Lustre: scratch-OST0010: haven't heard from client 801deff6-ca73-48ba-d677-aa4323cd413c (at 10.191.131.9@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 5 05:09:44 oss7 kernel: Lustre: Skipped 5 previous similar messages Jan 5 05:09:44 oss7 kernel: Lustre: scratch-OST003e: haven't heard from client 801deff6-ca73-48ba-d677-aa4323cd413c (at 10.191.131.9@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 5 05:09:44 oss7 kernel: Lustre: Skipped 2 previous similar messages Jan 5 14:05:34 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 5208a914-6e8e-0ada-aeff-3144e5b1db92 (at 10.191.131.9@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 5 14:05:34 oss7 kernel: Lustre: Skipped 2 previous similar messages Jan 5 23:01:12 oss7 kernel: Lustre: scratch-OST0010: haven't heard from client d3817ffd-5cad-8dcd-3953-fab08a71eeb7 (at 10.191.131.9@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 5 23:01:12 oss7 kernel: Lustre: Skipped 5 previous similar messages Jan 6 07:56:57 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 45890523-a5b5-295b-1e29-3e12b7830001 (at 10.191.131.9@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 6 07:56:57 oss7 kernel: Lustre: Skipped 5 previous similar messages Jan 6 11:08:35 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client ed8eda3f-eb1c-e181-5cfb-80532ec3bf78 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 6 11:08:35 oss7 kernel: Lustre: Skipped 5 previous similar messages Jan 6 11:56:57 oss7 kernel: Lustre: scratch-OST0028: haven't heard from client 73c6cf86-5772-8ab0-c3d0-6c239e16763c (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 6 11:56:57 oss7 kernel: Lustre: Skipped 5 previous similar messages Jan 6 11:56:58 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 73c6cf86-5772-8ab0-c3d0-6c239e16763c (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 6 12:44:07 oss7 kernel: Lustre: scratch-OST001c: haven't heard from client 16b0588f-9af8-fd5b-715d-9405e15b8efb (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 6 12:44:07 oss7 kernel: Lustre: Skipped 4 previous similar messages Jan 6 12:44:08 oss7 kernel: Lustre: scratch-OST0028: haven't heard from client 16b0588f-9af8-fd5b-715d-9405e15b8efb (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 6 12:44:10 oss7 kernel: Lustre: scratch-OST0010: haven't heard from client 16b0588f-9af8-fd5b-715d-9405e15b8efb (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 6 12:44:14 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 16b0588f-9af8-fd5b-715d-9405e15b8efb (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 6 13:26:43 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 23a17301-788c-2e0b-b68b-db756a7b967b (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 6 13:26:43 oss7 kernel: Lustre: Skipped 2 previous similar messages Jan 6 13:26:45 oss7 kernel: Lustre: scratch-OST001c: haven't heard from client 23a17301-788c-2e0b-b68b-db756a7b967b (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 6 13:26:45 oss7 kernel: Lustre: Skipped 3 previous similar messages Jan 6 13:26:49 oss7 kernel: Lustre: scratch-OST0028: haven't heard from client 23a17301-788c-2e0b-b68b-db756a7b967b (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 6 14:14:42 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 34cbeb95-193c-8869-93f6-88ce914bb43b (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 6 14:14:49 oss7 kernel: Lustre: scratch-OST003e: haven't heard from client 34cbeb95-193c-8869-93f6-88ce914bb43b (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 6 14:14:49 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 6 14:14:51 oss7 kernel: Lustre: scratch-OST001c: haven't heard from client 34cbeb95-193c-8869-93f6-88ce914bb43b (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 6 14:14:52 oss7 kernel: Lustre: scratch-OST0034: haven't heard from client 34cbeb95-193c-8869-93f6-88ce914bb43b (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 6 16:52:41 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 2085a7a3-082a-597a-a9d5-c399ff920fba (at 10.191.131.9@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 6 16:52:41 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 7 01:48:25 oss7 kernel: Lustre: scratch-OST0028: haven't heard from client a2c71cfd-9b89-f3c9-9a81-b4a950747307 (at 10.191.131.9@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 7 01:48:25 oss7 kernel: Lustre: Skipped 5 previous similar messages Jan 7 10:44:09 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 06001de8-c7ab-6745-753b-8c4aa28d5749 (at 10.191.131.9@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 7 10:44:09 oss7 kernel: Lustre: Skipped 5 previous similar messages Jan 7 14:02:09 oss7 kernel: Lustre: 22731:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1444444961728212 sent from scratch-OST0028 to NID 10.191.130.12@o2ib 3s ago has failed due to network error (7s prior to deadline). Jan 7 14:02:09 oss7 kernel: req@ffff8102b502ec00 x1444444961728212/t0 o106->@:15/16 lens 296/424 e 0 to 1 dl 1389121333 ref 1 fl Rpc:/0/0 rc 0/0 Jan 7 14:02:30 oss7 kernel: Lustre: 22731:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1444444961728212 sent from scratch-OST0028 to NID 10.191.130.12@o2ib 7s ago has timed out (7s prior to deadline). Jan 7 14:02:30 oss7 kernel: req@ffff8102b502ec00 x1444444961728212/t0 o106->@:15/16 lens 296/424 e 0 to 1 dl 1389121350 ref 4 fl Rpc:/2/0 rc 0/0 Jan 7 14:02:30 oss7 kernel: Lustre: 22731:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Jan 7 14:03:06 oss7 kernel: Lustre: 22731:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1444444961728212 sent from scratch-OST0028 to NID 10.191.130.12@o2ib 7s ago has timed out (7s prior to deadline). Jan 7 14:03:06 oss7 kernel: req@ffff8102b502ec00 x1444444961728212/t0 o106->@:15/16 lens 296/424 e 0 to 1 dl 1389121386 ref 2 fl Rpc:/2/0 rc 0/0 Jan 7 14:03:06 oss7 kernel: Lustre: 22731:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Jan 7 14:04:17 oss7 kernel: Lustre: 22731:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1444444961728212 sent from scratch-OST0028 to NID 10.191.130.12@o2ib 7s ago has timed out (7s prior to deadline). Jan 7 14:04:17 oss7 kernel: req@ffff8102b502ec00 x1444444961728212/t0 o106->@:15/16 lens 296/424 e 0 to 1 dl 1389121457 ref 5 fl Rpc:/2/0 rc 0/0 Jan 7 14:04:17 oss7 kernel: Lustre: 22731:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 10 previous similar messages Jan 7 14:05:23 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 601ca33d-909c-99c4-62ad-0a70895e67d3 (at 10.191.130.12@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 7 14:05:23 oss7 kernel: Lustre: Skipped 5 previous similar messages Jan 7 14:05:23 oss7 kernel: Lustre: scratch-OST001c: haven't heard from client 601ca33d-909c-99c4-62ad-0a70895e67d3 (at 10.191.130.12@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 7 14:05:26 oss7 kernel: Lustre: Service thread pid 22731 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 7 14:05:26 oss7 kernel: Pid: 22731, comm: ll_ost_221 Jan 7 14:05:26 oss7 kernel: Jan 7 14:05:26 oss7 kernel: Call Trace: Jan 7 14:05:26 oss7 kernel: [] schedule_timeout+0x8a/0xad Jan 7 14:05:26 oss7 kernel: [] process_timeout+0x0/0x5 Jan 7 14:05:26 oss7 kernel: [] ptlrpc_queue_wait+0xad8/0x1700 [ptlrpc] Jan 7 14:05:26 oss7 kernel: [] lustre_msg_set_opc+0x45/0x120 [ptlrpc] Jan 7 14:05:26 oss7 kernel: [] ptlrpc_at_set_req_timeout+0x85/0xe0 [ptlrpc] Jan 7 14:05:26 oss7 kernel: [] ptlrpc_prep_req_pool+0x625/0x6c0 [ptlrpc] Jan 7 14:05:26 oss7 kernel: [] default_wake_function+0x0/0xe Jan 7 14:05:26 oss7 kernel: [] ldlm_server_glimpse_ast+0x266/0x3b0 [ptlrpc] Jan 7 14:05:26 oss7 kernel: [] interval_iterate_reverse+0x73/0x240 [ptlrpc] Jan 7 14:05:26 oss7 kernel: [] ldlm_process_extent_lock+0x0/0xad0 [ptlrpc] Jan 7 14:05:26 oss7 kernel: [] filter_intent_policy+0x65a/0x760 [obdfilter] Jan 7 14:05:26 oss7 kernel: [] LNetPut+0x720/0x830 [lnet] Jan 7 14:05:26 oss7 kernel: [] ldlm_lock_enqueue+0x186/0xb20 [ptlrpc] Jan 7 14:05:26 oss7 kernel: [] ldlm_lock_create+0x540/0x9f0 [ptlrpc] Jan 7 14:05:26 oss7 kernel: [] ost_blocking_ast+0x0/0x9b0 [ost] Jan 7 14:05:26 oss7 kernel: [] ldlm_handle_enqueue+0xbf9/0x1210 [ptlrpc] Jan 7 14:05:26 oss7 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 7 14:05:26 oss7 kernel: [] ost_handle+0x5003/0x55c0 [ost] Jan 7 14:05:26 oss7 kernel: [] ptlrpc_server_handle_request+0x984/0xe00 [ptlrpc] Jan 7 14:05:26 oss7 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 7 14:05:26 oss7 kernel: [] __wake_up_common+0x3e/0x68 Jan 7 14:05:26 oss7 kernel: [] ptlrpc_main+0xf16/0x10e0 [ptlrpc] Jan 7 14:05:26 oss7 kernel: [] child_rip+0xa/0x11 Jan 7 14:05:26 oss7 kernel: [] ptlrpc_main+0x0/0x10e0 [ptlrpc] Jan 7 14:05:26 oss7 kernel: [] child_rip+0x0/0x11 Jan 7 14:05:26 oss7 kernel: Jan 7 14:05:26 oss7 kernel: LustreError: dumping log to /tmp/lustre-log.1389121526.22731 Jan 7 14:05:43 oss7 kernel: Lustre: scratch-OST0028: haven't heard from client 601ca33d-909c-99c4-62ad-0a70895e67d3 (at 10.191.130.12@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 7 14:05:43 oss7 kernel: Lustre: Skipped 3 previous similar messages Jan 7 14:05:43 oss7 kernel: Lustre: Service thread pid 22731 completed after 216.83s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Jan 7 16:00:22 oss7 kernel: Lustre: scratch-OST0034: haven't heard from client 2f9bf6c5-a05a-c2d8-f562-37e2dfaa6571 (at 10.191.132.3@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 7 16:00:47 oss7 kernel: Lustre: scratch-OST001c: haven't heard from client 2f9bf6c5-a05a-c2d8-f562-37e2dfaa6571 (at 10.191.132.3@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 7 16:00:47 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 7 16:01:16 oss7 kernel: LustreError: 22546:0:(ldlm_lib.c:1921:target_send_reply_msg()) @@@ processing error (-107) req@ffff810411c7ec00 x1446032283821768/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1389128496 ref 1 fl Interpret:H/0/0 rc -107/0 Jan 7 16:01:17 oss7 kernel: LustreError: 22680:0:(ldlm_lib.c:1921:target_send_reply_msg()) @@@ processing error (-107) req@ffff81060f777800 x1446032283821817/t0 o400->@:0/0 lens 192/0 e 0 to 0 dl 1389128495 ref 1 fl Interpret:H/0/0 rc -107/0 Jan 7 16:01:17 oss7 kernel: LustreError: 22680:0:(ldlm_lib.c:1921:target_send_reply_msg()) Skipped 1 previous similar message Jan 7 16:11:12 oss7 kernel: Lustre: scratch-OST0034: haven't heard from client 2f9bf6c5-a05a-c2d8-f562-37e2dfaa6571 (at 10.191.132.3@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 7 16:11:12 oss7 kernel: Lustre: Skipped 3 previous similar messages Jan 7 16:14:39 oss7 kernel: Lustre: scratch-OST0010: haven't heard from client 2f9bf6c5-a05a-c2d8-f562-37e2dfaa6571 (at 10.191.132.3@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 7 16:14:39 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 7 19:39:53 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client d52b818e-5563-1f29-5002-4a07424173dd (at 10.191.131.9@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 7 19:39:53 oss7 kernel: Lustre: Skipped 3 previous similar messages Jan 8 01:29:31 oss7 kernel: Lustre: 22528:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0034: 7e842302-e845-8a4e-2c49-ee9e63ec7065 reconnecting Jan 8 01:29:31 oss7 kernel: Lustre: 22528:0:(ldlm_lib.c:576:target_handle_reconnect()) Skipped 5 previous similar messages Jan 8 01:43:47 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 7e842302-e845-8a4e-2c49-ee9e63ec7065 (at 10.191.132.164@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 8 01:43:47 oss7 kernel: Lustre: Skipped 5 previous similar messages Jan 8 04:35:37 oss7 kernel: Lustre: scratch-OST003e: haven't heard from client 29d45b38-36fc-6f10-91d4-2791c61d78fb (at 10.191.131.9@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 8 04:35:37 oss7 kernel: Lustre: Skipped 5 previous similar messages Jan 8 13:31:22 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 63d4cdae-8e81-7be0-5367-18f69b00f201 (at 10.191.131.9@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 8 13:31:22 oss7 kernel: Lustre: Skipped 5 previous similar messages Jan 8 13:31:22 oss7 kernel: Lustre: scratch-OST0034: haven't heard from client 63d4cdae-8e81-7be0-5367-18f69b00f201 (at 10.191.131.9@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 8 13:31:22 oss7 kernel: Lustre: Skipped 3 previous similar messages Jan 8 17:59:59 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 6e5a4e28-6fd4-50a1-56ea-bf190fd3aeb0 (at 10.191.131.9@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 8 17:59:59 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 8 18:00:08 oss7 kernel: Lustre: scratch-OST0034: haven't heard from client 6e5a4e28-6fd4-50a1-56ea-bf190fd3aeb0 (at 10.191.131.9@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 8 18:00:08 oss7 kernel: Lustre: Skipped 4 previous similar messages Jan 9 00:40:39 oss7 kernel: Lustre: scratch-OST0034: haven't heard from client 0958a5e1-1657-a999-35ae-8bbceb0c0ca0 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 00:40:40 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 0958a5e1-1657-a999-35ae-8bbceb0c0ca0 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 00:40:40 oss7 kernel: Lustre: Skipped 2 previous similar messages Jan 9 01:00:22 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client fbc901cd-b779-7180-1e3b-0a1dc243a98a (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 01:00:22 oss7 kernel: Lustre: Skipped 2 previous similar messages Jan 9 01:00:23 oss7 kernel: Lustre: scratch-OST003e: haven't heard from client fbc901cd-b779-7180-1e3b-0a1dc243a98a (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 01:00:24 oss7 kernel: Lustre: scratch-OST0010: haven't heard from client fbc901cd-b779-7180-1e3b-0a1dc243a98a (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 01:00:24 oss7 kernel: Lustre: Skipped 3 previous similar messages Jan 9 01:19:07 oss7 kernel: Lustre: scratch-OST0028: haven't heard from client fc70ce7e-f7bc-9a4d-6a00-3747822a3396 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 01:19:11 oss7 kernel: Lustre: scratch-OST0010: haven't heard from client fc70ce7e-f7bc-9a4d-6a00-3747822a3396 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 01:19:12 oss7 kernel: Lustre: scratch-OST0034: haven't heard from client fc70ce7e-f7bc-9a4d-6a00-3747822a3396 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 01:19:12 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 9 01:36:17 oss7 kernel: LustreError: 22307:0:(events.c:387:server_bulk_callback()) event type 4, status -5, desc ffff810503ae8000 Jan 9 01:39:34 oss7 kernel: Lustre: Service thread pid 22882 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 9 01:39:34 oss7 kernel: Pid: 22882, comm: ll_ost_io_114 Jan 9 01:39:34 oss7 kernel: Jan 9 01:39:34 oss7 kernel: Call Trace: Jan 9 01:39:34 oss7 kernel: [] LNetMDBind+0x301/0x450 [lnet] Jan 9 01:39:34 oss7 kernel: [] schedule_timeout+0x8a/0xad Jan 9 01:39:34 oss7 kernel: [] process_timeout+0x0/0x5 Jan 9 01:39:34 oss7 kernel: [] ost_brw_read+0x127c/0x1a70 [ost] Jan 9 01:39:34 oss7 kernel: [] default_wake_function+0x0/0xe Jan 9 01:39:34 oss7 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 9 01:39:34 oss7 kernel: [] ost_handle+0x2e87/0x55c0 [ost] Jan 9 01:39:34 oss7 kernel: [] class_handle2object+0xe0/0x170 [obdclass] Jan 9 01:39:34 oss7 kernel: [] lock_res_and_lock+0xba/0xd0 [ptlrpc] Jan 9 01:39:34 oss7 kernel: [] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc] Jan 9 01:39:34 oss7 kernel: [] ptlrpc_server_handle_request+0x984/0xe00 [ptlrpc] Jan 9 01:39:34 oss7 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 9 01:39:34 oss7 kernel: [] __wake_up_common+0x3e/0x68 Jan 9 01:39:34 oss7 kernel: [] ptlrpc_main+0xf16/0x10e0 [ptlrpc] Jan 9 01:39:34 oss7 kernel: [] child_rip+0xa/0x11 Jan 9 01:39:34 oss7 kernel: [] ptlrpc_main+0x0/0x10e0 [ptlrpc] Jan 9 01:39:34 oss7 kernel: [] child_rip+0x0/0x11 Jan 9 01:39:34 oss7 kernel: Jan 9 01:39:34 oss7 kernel: LustreError: dumping log to /tmp/lustre-log.1389249574.22882 Jan 9 01:39:57 oss7 kernel: Lustre: scratch-OST0028: haven't heard from client a5c38550-c4fb-9e74-7805-200a20785342 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 01:39:57 oss7 kernel: Lustre: Skipped 2 previous similar messages Jan 9 01:39:58 oss7 kernel: Lustre: scratch-OST001c: haven't heard from client a5c38550-c4fb-9e74-7805-200a20785342 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 01:40:00 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client a5c38550-c4fb-9e74-7805-200a20785342 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 01:40:00 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 9 01:40:00 oss7 kernel: LustreError: 22882:0:(ost_handler.c:830:ost_brw_read()) @@@ Eviction on bulk PUT req@ffff81058694fc00 x1456732645705593/t0 o3->a5c38550-c4fb-9e74-7805-200a20785342@:0/0 lens 448/400 e 2 to 0 dl 1389249606 ref 1 fl Interpret:/0/0 rc 0/0 Jan 9 01:40:00 oss7 kernel: Lustre: 22882:0:(ost_handler.c:892:ost_brw_read()) scratch-OST0004: ignoring bulk IO comm error with a5c38550-c4fb-9e74-7805-200a20785342@ id 12345-10.191.130.49@o2ib - client will retry Jan 9 01:40:00 oss7 kernel: Lustre: 22882:0:(ost_handler.c:892:ost_brw_read()) Skipped 4 previous similar messages Jan 9 01:40:00 oss7 kernel: Lustre: Service thread pid 22882 completed after 226.59s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Jan 9 02:00:19 oss7 kernel: Lustre: scratch-OST001c: haven't heard from client 30742280-9759-7e35-be8d-77d4b34d5465 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 02:00:19 oss7 kernel: Lustre: Skipped 2 previous similar messages Jan 9 02:00:20 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 30742280-9759-7e35-be8d-77d4b34d5465 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 02:00:21 oss7 kernel: Lustre: scratch-OST0010: haven't heard from client 30742280-9759-7e35-be8d-77d4b34d5465 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 02:00:21 oss7 kernel: Lustre: Skipped 2 previous similar messages Jan 9 02:21:07 oss7 kernel: Lustre: scratch-OST0010: haven't heard from client 7e61ba21-5c19-bab4-81cc-b58ab0041f23 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 02:21:07 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 9 02:21:08 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 7e61ba21-5c19-bab4-81cc-b58ab0041f23 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 02:21:08 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 9 02:42:24 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 4b0b48a1-2385-d880-3641-d28ec33b199b (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 02:42:24 oss7 kernel: Lustre: Skipped 3 previous similar messages Jan 9 02:42:26 oss7 kernel: Lustre: scratch-OST001c: haven't heard from client 4b0b48a1-2385-d880-3641-d28ec33b199b (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 02:42:26 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 9 02:42:27 oss7 kernel: Lustre: scratch-OST0010: haven't heard from client 4b0b48a1-2385-d880-3641-d28ec33b199b (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 02:42:27 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 9 02:55:47 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 825db6ca-bf94-aca0-fef3-58db85f05e7c (at 10.191.131.9@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 02:55:47 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 9 03:03:55 oss7 kernel: Lustre: scratch-OST003e: haven't heard from client a4d28940-48cd-7240-ffe7-998cc65b1ebe (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 03:03:55 oss7 kernel: Lustre: Skipped 5 previous similar messages Jan 9 03:03:56 oss7 kernel: Lustre: scratch-OST0034: haven't heard from client a4d28940-48cd-7240-ffe7-998cc65b1ebe (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 03:03:57 oss7 kernel: Lustre: scratch-OST0010: haven't heard from client a4d28940-48cd-7240-ffe7-998cc65b1ebe (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 03:25:22 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 4a7a2814-b68e-6386-860d-553e3dfe7b74 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 03:25:22 oss7 kernel: Lustre: Skipped 3 previous similar messages Jan 9 03:25:23 oss7 kernel: Lustre: scratch-OST0028: haven't heard from client 4a7a2814-b68e-6386-860d-553e3dfe7b74 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 03:25:23 oss7 kernel: Lustre: Skipped 3 previous similar messages Jan 9 03:46:29 oss7 kernel: Lustre: scratch-OST0028: haven't heard from client a8c5560f-9dff-a198-aa05-556799026439 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 03:46:29 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 9 03:46:31 oss7 kernel: Lustre: scratch-OST001c: haven't heard from client a8c5560f-9dff-a198-aa05-556799026439 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 03:46:32 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client a8c5560f-9dff-a198-aa05-556799026439 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 03:46:33 oss7 kernel: Lustre: scratch-OST0010: haven't heard from client a8c5560f-9dff-a198-aa05-556799026439 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 03:46:33 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 9 04:07:22 oss7 kernel: Lustre: scratch-OST001c: haven't heard from client 23557cb3-4b39-0a8e-ca61-6863f721e692 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 04:07:22 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 9 04:07:25 oss7 kernel: Lustre: scratch-OST0034: haven't heard from client 23557cb3-4b39-0a8e-ca61-6863f721e692 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 04:07:26 oss7 kernel: Lustre: scratch-OST0010: haven't heard from client 23557cb3-4b39-0a8e-ca61-6863f721e692 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 04:07:27 oss7 kernel: Lustre: scratch-OST003e: haven't heard from client 23557cb3-4b39-0a8e-ca61-6863f721e692 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 04:07:27 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 9 04:24:39 oss7 kernel: LustreError: 22311:0:(events.c:387:server_bulk_callback()) event type 4, status -5, desc ffff81033dae2a80 Jan 9 04:24:39 oss7 kernel: LustreError: 22311:0:(events.c:387:server_bulk_callback()) event type 4, status -5, desc ffff8103dd02e000 Jan 9 04:27:56 oss7 kernel: Lustre: Service thread pid 22936 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 9 04:27:56 oss7 kernel: Pid: 22936, comm: ll_ost_io_168 Jan 9 04:27:56 oss7 kernel: Jan 9 04:27:56 oss7 kernel: Call Trace: Jan 9 04:27:56 oss7 kernel: [] schedule_timeout+0x8a/0xad Jan 9 04:27:56 oss7 kernel: [] process_timeout+0x0/0x5 Jan 9 04:27:56 oss7 kernel: [] ost_brw_read+0x127c/0x1a70 [ost] Jan 9 04:27:56 oss7 kernel: [] default_wake_function+0x0/0xe Jan 9 04:27:56 oss7 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 9 04:27:56 oss7 kernel: [] ost_handle+0x2e87/0x55c0 [ost] Jan 9 04:27:56 oss7 kernel: [] class_handle2object+0xe0/0x170 [obdclass] Jan 9 04:27:56 oss7 kernel: [] lock_res_and_lock+0xba/0xd0 [ptlrpc] Jan 9 04:27:56 oss7 kernel: [] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc] Jan 9 04:27:56 oss7 kernel: [] ptlrpc_server_handle_request+0x984/0xe00 [ptlrpc] Jan 9 04:27:56 oss7 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 9 04:27:56 oss7 kernel: [] __wake_up_common+0x3e/0x68 Jan 9 04:27:56 oss7 kernel: [] ptlrpc_main+0xf16/0x10e0 [ptlrpc] Jan 9 04:27:56 oss7 kernel: [] child_rip+0xa/0x11 Jan 9 04:27:56 oss7 kernel: [] ptlrpc_main+0x0/0x10e0 [ptlrpc] Jan 9 04:27:56 oss7 kernel: [] child_rip+0x0/0x11 Jan 9 04:27:56 oss7 kernel: Jan 9 04:27:56 oss7 kernel: LustreError: dumping log to /tmp/lustre-log.1389259676.22936 Jan 9 04:27:56 oss7 kernel: Pid: 22967, comm: ll_ost_io_199 Jan 9 04:27:56 oss7 kernel: Jan 9 04:27:56 oss7 kernel: Call Trace: Jan 9 04:27:56 oss7 kernel: [] LNetMDBind+0x301/0x450 [lnet] Jan 9 04:27:56 oss7 kernel: [] lock_timer_base+0x1b/0x3c Jan 9 04:27:56 oss7 kernel: [] __mod_timer+0xff/0x10e Jan 9 04:27:56 oss7 kernel: [] schedule_timeout+0x8a/0xad Jan 9 04:27:56 oss7 kernel: [] process_timeout+0x0/0x5 Jan 9 04:27:56 oss7 kernel: [] ost_brw_read+0x127c/0x1a70 [ost] Jan 9 04:27:56 oss7 kernel: [] default_wake_function+0x0/0xe Jan 9 04:27:56 oss7 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 9 04:27:56 oss7 kernel: [] ost_handle+0x2e87/0x55c0 [ost] Jan 9 04:27:56 oss7 kernel: [] class_handle2object+0xe0/0x170 [obdclass] Jan 9 04:27:56 oss7 kernel: [] lock_res_and_lock+0xba/0xd0 [ptlrpc] Jan 9 04:27:56 oss7 kernel: [] __ldlm_handle2lock+0x2f8/0x360 [ptlrpc] Jan 9 04:27:56 oss7 kernel: [] ptlrpc_server_handle_request+0x984/0xe00 [ptlrpc] Jan 9 04:27:56 oss7 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 9 04:27:56 oss7 kernel: [] __wake_up_common+0x3e/0x68 Jan 9 04:27:56 oss7 kernel: [] ptlrpc_main+0xf16/0x10e0 [ptlrpc] Jan 9 04:27:56 oss7 kernel: [] child_rip+0xa/0x11 Jan 9 04:27:56 oss7 kernel: [] ptlrpc_main+0x0/0x10e0 [ptlrpc] Jan 9 04:27:56 oss7 kernel: [] child_rip+0x0/0x11 Jan 9 04:27:56 oss7 kernel: Jan 9 04:27:56 oss7 kernel: LustreError: dumping log to /tmp/lustre-log.1389259676.22967 Jan 9 04:28:21 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 55faf674-1825-6f88-e485-f0e13979ec47 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 04:28:21 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 9 04:28:22 oss7 kernel: Lustre: scratch-OST0010: haven't heard from client 55faf674-1825-6f88-e485-f0e13979ec47 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 04:28:22 oss7 kernel: Lustre: Skipped 2 previous similar messages Jan 9 04:28:22 oss7 kernel: LustreError: 22967:0:(ost_handler.c:830:ost_brw_read()) @@@ Eviction on bulk PUT req@ffff81047734d800 x1456743233183719/t0 o3->55faf674-1825-6f88-e485-f0e13979ec47@:0/0 lens 448/400 e 4 to 0 dl 1389259872 ref 1 fl Interpret:/0/0 rc 0/0 Jan 9 04:28:22 oss7 kernel: Lustre: 22967:0:(ost_handler.c:892:ost_brw_read()) scratch-OST0010: ignoring bulk IO comm error with 55faf674-1825-6f88-e485-f0e13979ec47@ id 12345-10.191.130.49@o2ib - client will retry Jan 9 04:28:22 oss7 kernel: Lustre: Service thread pid 22967 completed after 226.38s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Jan 9 04:28:22 oss7 kernel: LustreError: 22936:0:(ost_handler.c:830:ost_brw_read()) @@@ Eviction on bulk PUT req@ffff810610b9b450 x1456743233183720/t0 o3->55faf674-1825-6f88-e485-f0e13979ec47@:0/0 lens 448/400 e 4 to 0 dl 1389259872 ref 1 fl Interpret:/0/0 rc 0/0 Jan 9 04:28:22 oss7 kernel: Lustre: Service thread pid 22936 completed after 226.39s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Jan 9 04:49:58 oss7 kernel: Lustre: scratch-OST0028: haven't heard from client 69bf6813-1548-a459-46bd-5656f0a20c57 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 04:49:58 oss7 kernel: Lustre: Skipped 2 previous similar messages Jan 9 04:49:59 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 69bf6813-1548-a459-46bd-5656f0a20c57 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 04:49:59 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 9 05:06:30 oss7 kernel: LustreError: 22309:0:(events.c:387:server_bulk_callback()) event type 4, status -5, desc ffff810603f6c000 Jan 9 05:09:47 oss7 kernel: Lustre: Service thread pid 22879 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 9 05:09:47 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 9 05:09:47 oss7 kernel: Pid: 22879, comm: ll_ost_io_111 Jan 9 05:09:47 oss7 kernel: Jan 9 05:09:47 oss7 kernel: Call Trace: Jan 9 05:09:47 oss7 kernel: [] LNetMDBind+0x301/0x450 [lnet] Jan 9 05:09:47 oss7 kernel: [] schedule_timeout+0x8a/0xad Jan 9 05:09:47 oss7 kernel: [] process_timeout+0x0/0x5 Jan 9 05:09:47 oss7 kernel: [] ost_brw_read+0x127c/0x1a70 [ost] Jan 9 05:09:47 oss7 kernel: [] default_wake_function+0x0/0xe Jan 9 05:09:47 oss7 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 9 05:09:47 oss7 kernel: [] ost_handle+0x2e87/0x55c0 [ost] Jan 9 05:09:47 oss7 kernel: [] ptlrpc_server_handle_request+0x984/0xe00 [ptlrpc] Jan 9 05:09:47 oss7 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 9 05:09:47 oss7 kernel: [] default_wake_function+0x0/0xe Jan 9 05:09:47 oss7 kernel: [] ptlrpc_main+0xf16/0x10e0 [ptlrpc] Jan 9 05:09:47 oss7 kernel: [] child_rip+0xa/0x11 Jan 9 05:09:47 oss7 kernel: [] ptlrpc_main+0x0/0x10e0 [ptlrpc] Jan 9 05:09:47 oss7 kernel: [] child_rip+0x0/0x11 Jan 9 05:09:47 oss7 kernel: Jan 9 05:09:47 oss7 kernel: LustreError: dumping log to /tmp/lustre-log.1389262187.22879 Jan 9 05:10:12 oss7 kernel: Lustre: scratch-OST0028: haven't heard from client 3b9b4f64-4e8c-0025-6a92-d62977e81a88 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 05:10:12 oss7 kernel: Lustre: Skipped 3 previous similar messages Jan 9 05:10:13 oss7 kernel: Lustre: scratch-OST0010: haven't heard from client 3b9b4f64-4e8c-0025-6a92-d62977e81a88 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 05:10:13 oss7 kernel: Lustre: Skipped 2 previous similar messages Jan 9 05:10:13 oss7 kernel: LustreError: 22879:0:(ost_handler.c:830:ost_brw_read()) @@@ Eviction on bulk PUT req@ffff81045bd33400 x1456745911249277/t0 o3->3b9b4f64-4e8c-0025-6a92-d62977e81a88@:0/0 lens 448/400 e 2 to 0 dl 1389262219 ref 1 fl Interpret:/0/0 rc 0/0 Jan 9 05:10:13 oss7 kernel: Lustre: 22879:0:(ost_handler.c:892:ost_brw_read()) scratch-OST0010: ignoring bulk IO comm error with 3b9b4f64-4e8c-0025-6a92-d62977e81a88@ id 12345-10.191.130.49@o2ib - client will retry Jan 9 05:10:13 oss7 kernel: Lustre: 22879:0:(ost_handler.c:892:ost_brw_read()) Skipped 1 previous similar message Jan 9 05:10:13 oss7 kernel: Lustre: Service thread pid 22879 completed after 226.02s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Jan 9 05:10:14 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 3b9b4f64-4e8c-0025-6a92-d62977e81a88 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 05:29:50 oss7 kernel: Lustre: scratch-OST0028: haven't heard from client 9e9251c6-b8bb-4005-d616-103eef98d21b (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 05:29:50 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 9 05:29:51 oss7 kernel: Lustre: scratch-OST0010: haven't heard from client 9e9251c6-b8bb-4005-d616-103eef98d21b (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 05:29:52 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 9e9251c6-b8bb-4005-d616-103eef98d21b (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 05:29:52 oss7 kernel: Lustre: Skipped 2 previous similar messages Jan 9 05:59:48 oss7 kernel: Lustre: scratch-OST0010: haven't heard from client ed82190a-2cb6-634d-ac4b-b94f34e20606 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 05:59:48 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 9 05:59:49 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client ed82190a-2cb6-634d-ac4b-b94f34e20606 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 06:19:42 oss7 kernel: Lustre: scratch-OST0010: haven't heard from client 61ff77ce-6d6a-2dbd-cf29-c4c2fc3ac875 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 06:19:42 oss7 kernel: Lustre: Skipped 4 previous similar messages Jan 9 06:19:42 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 61ff77ce-6d6a-2dbd-cf29-c4c2fc3ac875 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 06:19:44 oss7 kernel: Lustre: scratch-OST001c: haven't heard from client 61ff77ce-6d6a-2dbd-cf29-c4c2fc3ac875 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 06:19:44 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 9 06:36:07 oss7 kernel: LustreError: 22308:0:(events.c:387:server_bulk_callback()) event type 4, status -5, desc ffff81051584e000 Jan 9 06:39:24 oss7 kernel: Lustre: Service thread pid 22815 was inactive for 200.00s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: Jan 9 06:39:24 oss7 kernel: Pid: 22815, comm: ll_ost_io_47 Jan 9 06:39:24 oss7 kernel: Jan 9 06:39:24 oss7 kernel: Call Trace: Jan 9 06:39:24 oss7 kernel: [] LNetMDBind+0x301/0x450 [lnet] Jan 9 06:39:24 oss7 kernel: [] schedule_timeout+0x8a/0xad Jan 9 06:39:24 oss7 kernel: [] process_timeout+0x0/0x5 Jan 9 06:39:24 oss7 kernel: [] ost_brw_read+0x127c/0x1a70 [ost] Jan 9 06:39:24 oss7 kernel: [] default_wake_function+0x0/0xe Jan 9 06:39:24 oss7 kernel: [] lustre_msg_check_version_v2+0x8/0x20 [ptlrpc] Jan 9 06:39:24 oss7 kernel: [] ost_handle+0x2e87/0x55c0 [ost] Jan 9 06:39:24 oss7 kernel: [] ptlrpc_server_handle_request+0x984/0xe00 [ptlrpc] Jan 9 06:39:24 oss7 kernel: [] ptlrpc_wait_event+0x2e5/0x310 [ptlrpc] Jan 9 06:39:24 oss7 kernel: [] default_wake_function+0x0/0xe Jan 9 06:39:24 oss7 kernel: [] ptlrpc_main+0xf16/0x10e0 [ptlrpc] Jan 9 06:39:24 oss7 kernel: [] child_rip+0xa/0x11 Jan 9 06:39:24 oss7 kernel: [] ptlrpc_main+0x0/0x10e0 [ptlrpc] Jan 9 06:39:24 oss7 kernel: [] child_rip+0x0/0x11 Jan 9 06:39:24 oss7 kernel: Jan 9 06:39:24 oss7 kernel: LustreError: dumping log to /tmp/lustre-log.1389267564.22815 Jan 9 06:39:43 oss7 kernel: Lustre: scratch-OST003e: haven't heard from client ddfabaa9-23fb-33ac-704e-1afdd8897948 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 06:39:43 oss7 kernel: Lustre: Skipped 2 previous similar messages Jan 9 06:39:47 oss7 kernel: Lustre: scratch-OST0010: haven't heard from client ddfabaa9-23fb-33ac-704e-1afdd8897948 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 06:39:50 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client ddfabaa9-23fb-33ac-704e-1afdd8897948 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 06:39:50 oss7 kernel: LustreError: 22815:0:(ost_handler.c:830:ost_brw_read()) @@@ Eviction on bulk PUT req@ffff8106143b0000 x1456751557831697/t0 o3->ddfabaa9-23fb-33ac-704e-1afdd8897948@:0/0 lens 448/400 e 2 to 0 dl 1389267596 ref 1 fl Interpret:/0/0 rc 0/0 Jan 9 06:39:50 oss7 kernel: Lustre: 22815:0:(ost_handler.c:892:ost_brw_read()) scratch-OST0028: ignoring bulk IO comm error with ddfabaa9-23fb-33ac-704e-1afdd8897948@ id 12345-10.191.130.49@o2ib - client will retry Jan 9 06:39:50 oss7 kernel: Lustre: Service thread pid 22815 completed after 226.03s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). Jan 9 06:59:47 oss7 kernel: Lustre: scratch-OST001c: haven't heard from client 3903c93d-903c-8a6b-dcc8-0757505c3a54 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 06:59:47 oss7 kernel: Lustre: Skipped 3 previous similar messages Jan 9 06:59:48 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 3903c93d-903c-8a6b-dcc8-0757505c3a54 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 06:59:49 oss7 kernel: Lustre: scratch-OST0028: haven't heard from client 3903c93d-903c-8a6b-dcc8-0757505c3a54 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 06:59:50 oss7 kernel: Lustre: scratch-OST0010: haven't heard from client 3903c93d-903c-8a6b-dcc8-0757505c3a54 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 06:59:50 oss7 kernel: Lustre: Skipped 2 previous similar messages Jan 9 07:20:52 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 73c2dc7e-a467-6f57-6a04-15165f0050f9 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 07:20:53 oss7 kernel: Lustre: scratch-OST0034: haven't heard from client 73c2dc7e-a467-6f57-6a04-15165f0050f9 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 07:20:53 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 9 07:20:54 oss7 kernel: Lustre: scratch-OST0010: haven't heard from client 73c2dc7e-a467-6f57-6a04-15165f0050f9 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 07:50:51 oss7 kernel: Lustre: scratch-OST0028: haven't heard from client 2ce3eab9-e739-7421-cdb0-63e4ae94bba4 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 07:50:51 oss7 kernel: Lustre: Skipped 2 previous similar messages Jan 9 07:50:52 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 2ce3eab9-e739-7421-cdb0-63e4ae94bba4 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 08:10:51 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client e9e3ac66-03ed-ab8f-c0b3-32ab04eac20d (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 08:10:51 oss7 kernel: Lustre: Skipped 4 previous similar messages Jan 9 08:10:52 oss7 kernel: Lustre: scratch-OST001c: haven't heard from client e9e3ac66-03ed-ab8f-c0b3-32ab04eac20d (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 08:10:52 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 9 08:10:53 oss7 kernel: Lustre: scratch-OST0010: haven't heard from client e9e3ac66-03ed-ab8f-c0b3-32ab04eac20d (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 08:10:53 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 9 08:21:07 oss7 kernel: Lustre: scratch-OST0034: haven't heard from client 78e15bc1-7afd-5fa6-2aa4-0d4ab2084671 (at 10.191.131.9@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 08:21:07 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 9 08:30:36 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client ef9e87a0-803c-fb9e-77ab-6220051814fa (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 08:30:36 oss7 kernel: Lustre: Skipped 5 previous similar messages Jan 9 08:30:40 oss7 kernel: Lustre: scratch-OST003e: haven't heard from client ef9e87a0-803c-fb9e-77ab-6220051814fa (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 08:50:39 oss7 kernel: Lustre: scratch-OST001c: haven't heard from client 45a2bab0-560a-5cdc-6087-46b0ce777525 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 08:50:39 oss7 kernel: Lustre: Skipped 4 previous similar messages Jan 9 08:50:41 oss7 kernel: Lustre: scratch-OST0034: haven't heard from client 45a2bab0-560a-5cdc-6087-46b0ce777525 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 08:50:41 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 9 08:50:42 oss7 kernel: Lustre: scratch-OST0010: haven't heard from client 45a2bab0-560a-5cdc-6087-46b0ce777525 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 08:50:45 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 45a2bab0-560a-5cdc-6087-46b0ce777525 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 09:21:13 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client b06d6062-a833-63d7-c498-84489f63d40a (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 09:21:13 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 9 09:21:14 oss7 kernel: Lustre: scratch-OST001c: haven't heard from client b06d6062-a833-63d7-c498-84489f63d40a (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 09:21:15 oss7 kernel: Lustre: scratch-OST0010: haven't heard from client b06d6062-a833-63d7-c498-84489f63d40a (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 09:21:15 oss7 kernel: Lustre: Skipped 2 previous similar messages Jan 9 09:40:38 oss7 kernel: Lustre: scratch-OST0034: haven't heard from client 4110ce9d-5fc7-68df-4f80-ff1113659008 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 09:40:38 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 9 09:40:41 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 4110ce9d-5fc7-68df-4f80-ff1113659008 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 09:40:43 oss7 kernel: Lustre: scratch-OST001c: haven't heard from client 4110ce9d-5fc7-68df-4f80-ff1113659008 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 09:40:45 oss7 kernel: Lustre: scratch-OST0010: haven't heard from client 4110ce9d-5fc7-68df-4f80-ff1113659008 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 09:40:45 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 9 09:59:40 oss7 kernel: Lustre: scratch-OST0028: haven't heard from client 9d2b9d80-8b70-ac24-9153-5e5c19fd9898 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 09:59:40 oss7 kernel: Lustre: Skipped 1 previous similar message Jan 9 09:59:46 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 9d2b9d80-8b70-ac24-9153-5e5c19fd9898 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 09:59:47 oss7 kernel: Lustre: scratch-OST003e: haven't heard from client 9d2b9d80-8b70-ac24-9153-5e5c19fd9898 (at 10.191.130.49@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 09:59:47 oss7 kernel: Lustre: Skipped 3 previous similar messages Jan 9 12:44:23 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 00b75c13-c966-c0de-1aef-42fbd93dfd29 (at 10.191.131.9@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 15:05:39 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 15:05:39 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 40 previous similar messages Jan 9 15:06:56 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 15:06:56 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 6 previous similar messages Jan 9 15:07:55 oss7 kernel: LustreError: 22309:0:(events.c:387:server_bulk_callback()) event type 4, status -61, desc ffff8105ed09e000 Jan 9 15:07:55 oss7 kernel: LustreError: 22309:0:(events.c:387:server_bulk_callback()) event type 2, status -61, desc ffff8105ed09e000 Jan 9 15:07:55 oss7 kernel: LustreError: 23003:0:(ost_handler.c:1078:ost_brw_write()) @@@ network error on bulk GET 0(1048576) req@ffff8105d8ec2400 x1449826069692869/t0 o4->d12174ef-c164-5018-d7f7-03674325c534@:0/0 lens 448/416 e 0 to 0 dl 1389298120 ref 1 fl Interpret:/0/0 rc 0/0 Jan 9 15:07:55 oss7 kernel: LustreError: 22309:0:(events.c:387:server_bulk_callback()) event type 4, status -61, desc ffff8104d04ac000 Jan 9 15:07:55 oss7 kernel: LustreError: 22309:0:(events.c:387:server_bulk_callback()) event type 2, status -61, desc ffff8104d04ac000 Jan 9 15:07:55 oss7 kernel: Lustre: 23003:0:(ost_handler.c:1229:ost_brw_write()) scratch-OST0034: ignoring bulk IO comm error with d12174ef-c164-5018-d7f7-03674325c534@ id 12345-10.191.133.6@o2ib - client will retry Jan 9 15:07:55 oss7 kernel: Lustre: 23003:0:(ost_handler.c:1229:ost_brw_write()) Skipped 3 previous similar messages Jan 9 15:07:55 oss7 kernel: LustreError: 22309:0:(events.c:387:server_bulk_callback()) event type 4, status -61, desc ffff8105cb728000 Jan 9 15:07:55 oss7 kernel: LustreError: 22309:0:(events.c:387:server_bulk_callback()) event type 2, status -61, desc ffff8105cb728000 Jan 9 15:08:14 oss7 kernel: Lustre: 22652:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0034: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 9 15:09:27 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 15:09:27 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 26 previous similar messages Jan 9 15:14:33 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 15:14:33 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 47 previous similar messages Jan 9 15:24:42 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 15:24:42 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 96 previous similar messages Jan 9 15:32:43 oss7 kernel: Lustre: 22677:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0004: 2a6442ed-8070-0e0b-4158-e116c83221a5 reconnecting Jan 9 15:32:43 oss7 kernel: Lustre: 22764:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST001c: 2a6442ed-8070-0e0b-4158-e116c83221a5 reconnecting Jan 9 15:32:43 oss7 kernel: Lustre: 22764:0:(ldlm_lib.c:576:target_handle_reconnect()) Skipped 2 previous similar messages Jan 9 15:32:43 oss7 kernel: Lustre: 22677:0:(ldlm_lib.c:576:target_handle_reconnect()) Skipped 2 previous similar messages Jan 9 15:32:43 oss7 kernel: LustreError: 137-5: UUID 'scratch-OST0005_UUID' is not available for connect (no target) Jan 9 15:32:43 oss7 kernel: LustreError: Skipped 14 previous similar messages Jan 9 15:32:43 oss7 kernel: LustreError: 22637:0:(ldlm_lib.c:1921:target_send_reply_msg()) @@@ processing error (-19) req@ffff81007db7c800 x1456784475697405/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1389299663 ref 1 fl Interpret:/0/0 rc -19/0 Jan 9 15:32:43 oss7 kernel: LustreError: 22637:0:(ldlm_lib.c:1921:target_send_reply_msg()) Skipped 1 previous similar message Jan 9 15:32:43 oss7 kernel: Lustre: 22637:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST003e: 2a6442ed-8070-0e0b-4158-e116c83221a5 reconnecting Jan 9 15:32:43 oss7 kernel: Lustre: 22637:0:(ldlm_lib.c:576:target_handle_reconnect()) Skipped 13 previous similar messages Jan 9 15:34:09 oss7 kernel: LustreError: 137-5: UUID 'scratch-OST0005_UUID' is not available for connect (no target) Jan 9 15:34:09 oss7 kernel: LustreError: Skipped 11 previous similar messages Jan 9 15:34:09 oss7 kernel: LustreError: 22539:0:(ldlm_lib.c:1921:target_send_reply_msg()) @@@ processing error (-19) req@ffff810605d00800 x1456784475697988/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1389299749 ref 1 fl Interpret:/0/0 rc -19/0 Jan 9 15:34:09 oss7 kernel: LustreError: 22539:0:(ldlm_lib.c:1921:target_send_reply_msg()) Skipped 11 previous similar messages Jan 9 15:34:09 oss7 kernel: Lustre: 22642:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0004: 2a6442ed-8070-0e0b-4158-e116c83221a5 reconnecting Jan 9 15:34:09 oss7 kernel: Lustre: 22642:0:(ldlm_lib.c:576:target_handle_reconnect()) Skipped 1 previous similar message Jan 9 15:34:17 oss7 kernel: Lustre: 22524:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0004: 2a6442ed-8070-0e0b-4158-e116c83221a5 reconnecting Jan 9 15:34:17 oss7 kernel: Lustre: 22524:0:(ldlm_lib.c:576:target_handle_reconnect()) Skipped 4 previous similar messages Jan 9 15:34:55 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 15:34:55 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 76 previous similar messages Jan 9 15:36:08 oss7 kernel: LustreError: 137-5: UUID 'scratch-OST0005_UUID' is not available for connect (no target) Jan 9 15:36:08 oss7 kernel: LustreError: Skipped 11 previous similar messages Jan 9 15:36:08 oss7 kernel: LustreError: 22605:0:(ldlm_lib.c:1921:target_send_reply_msg()) @@@ processing error (-19) req@ffff8103f27f4c00 x1456784475698593/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1389299868 ref 1 fl Interpret:/0/0 rc -19/0 Jan 9 15:36:08 oss7 kernel: LustreError: 22605:0:(ldlm_lib.c:1921:target_send_reply_msg()) Skipped 11 previous similar messages Jan 9 15:36:08 oss7 kernel: Lustre: 22566:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0034: 2a6442ed-8070-0e0b-4158-e116c83221a5 reconnecting Jan 9 15:36:08 oss7 kernel: Lustre: 22566:0:(ldlm_lib.c:576:target_handle_reconnect()) Skipped 5 previous similar messages Jan 9 15:39:09 oss7 kernel: LustreError: 22305:0:(events.c:387:server_bulk_callback()) event type 4, status -61, desc ffff8105ed727000 Jan 9 15:39:09 oss7 kernel: LustreError: 22305:0:(events.c:387:server_bulk_callback()) event type 2, status -61, desc ffff8105ed727000 Jan 9 15:39:09 oss7 kernel: LustreError: 22821:0:(ost_handler.c:1078:ost_brw_write()) @@@ network error on bulk GET 0(528080) req@ffff810528d0c000 x1449826069708629/t0 o4->d12174ef-c164-5018-d7f7-03674325c534@:0/0 lens 448/416 e 0 to 0 dl 1389299956 ref 1 fl Interpret:/0/0 rc 0/0 Jan 9 15:39:09 oss7 kernel: LustreError: 22821:0:(ost_handler.c:1078:ost_brw_write()) Skipped 2 previous similar messages Jan 9 15:39:09 oss7 kernel: Lustre: 22821:0:(ost_handler.c:1229:ost_brw_write()) scratch-OST0034: ignoring bulk IO comm error with d12174ef-c164-5018-d7f7-03674325c534@ id 12345-10.191.133.6@o2ib - client will retry Jan 9 15:39:09 oss7 kernel: Lustre: 22821:0:(ost_handler.c:1229:ost_brw_write()) Skipped 2 previous similar messages Jan 9 15:39:17 oss7 kernel: Lustre: 22568:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0034: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 9 15:39:17 oss7 kernel: Lustre: 22568:0:(ldlm_lib.c:576:target_handle_reconnect()) Skipped 4 previous similar messages Jan 9 15:44:58 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 15:44:58 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 90 previous similar messages Jan 9 15:48:34 oss7 kernel: LustreError: 22308:0:(events.c:387:server_bulk_callback()) event type 4, status -61, desc ffff8105583b2000 Jan 9 15:48:34 oss7 kernel: LustreError: 22308:0:(events.c:387:server_bulk_callback()) event type 2, status -61, desc ffff8105583b2000 Jan 9 15:48:34 oss7 kernel: LustreError: 22791:0:(ost_handler.c:1078:ost_brw_write()) @@@ network error on bulk GET 0(1048576) req@ffff8104fedd5800 x1449826069713454/t0 o4->d12174ef-c164-5018-d7f7-03674325c534@:0/0 lens 448/416 e 0 to 0 dl 1389300559 ref 1 fl Interpret:/0/0 rc 0/0 Jan 9 15:48:34 oss7 kernel: Lustre: 22791:0:(ost_handler.c:1229:ost_brw_write()) scratch-OST0034: ignoring bulk IO comm error with d12174ef-c164-5018-d7f7-03674325c534@ id 12345-10.191.133.6@o2ib - client will retry Jan 9 15:49:20 oss7 kernel: Lustre: 22521:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0034: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 9 15:55:07 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 15:55:07 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 115 previous similar messages Jan 9 16:05:17 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 16:05:17 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 101 previous similar messages Jan 9 16:15:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 16:15:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 94 previous similar messages Jan 9 16:25:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 16:25:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 93 previous similar messages Jan 9 16:32:23 oss7 kernel: LustreError: 22308:0:(events.c:387:server_bulk_callback()) event type 4, status -61, desc ffff8105f5d6e000 Jan 9 16:32:23 oss7 kernel: LustreError: 22308:0:(events.c:387:server_bulk_callback()) event type 2, status -61, desc ffff8105f5d6e000 Jan 9 16:32:23 oss7 kernel: LustreError: 22909:0:(ost_handler.c:1078:ost_brw_write()) @@@ network error on bulk GET 0(1048576) req@ffff810610548c50 x1449826069736889/t0 o4->d12174ef-c164-5018-d7f7-03674325c534@:0/0 lens 448/416 e 0 to 0 dl 1389303149 ref 1 fl Interpret:/0/0 rc 0/0 Jan 9 16:32:23 oss7 kernel: Lustre: 22909:0:(ost_handler.c:1229:ost_brw_write()) scratch-OST0034: ignoring bulk IO comm error with d12174ef-c164-5018-d7f7-03674325c534@ id 12345-10.191.133.6@o2ib - client will retry Jan 9 16:32:31 oss7 kernel: Lustre: 22665:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0034: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 9 16:35:23 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 16:35:23 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 93 previous similar messages Jan 9 16:45:42 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 16:45:42 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 98 previous similar messages Jan 9 16:55:42 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 16:55:42 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 101 previous similar messages Jan 9 17:05:46 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 17:05:46 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 91 previous similar messages Jan 9 17:15:48 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 17:15:48 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 98 previous similar messages Jan 9 17:16:12 oss7 kernel: LustreError: 22307:0:(events.c:387:server_bulk_callback()) event type 4, status -61, desc ffff81050e4f4000 Jan 9 17:16:12 oss7 kernel: LustreError: 22307:0:(events.c:387:server_bulk_callback()) event type 2, status -61, desc ffff81050e4f4000 Jan 9 17:16:12 oss7 kernel: LustreError: 22863:0:(ost_handler.c:1078:ost_brw_write()) @@@ network error on bulk GET 0(1048576) req@ffff8104246d2000 x1449826069760513/t0 o4->d12174ef-c164-5018-d7f7-03674325c534@:0/0 lens 448/416 e 0 to 0 dl 1389305817 ref 1 fl Interpret:/0/0 rc 0/0 Jan 9 17:16:12 oss7 kernel: Lustre: 22863:0:(ost_handler.c:1229:ost_brw_write()) scratch-OST0034: ignoring bulk IO comm error with d12174ef-c164-5018-d7f7-03674325c534@ id 12345-10.191.133.6@o2ib - client will retry Jan 9 17:16:58 oss7 kernel: Lustre: 22561:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0034: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 9 17:25:56 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 17:25:56 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 96 previous similar messages Jan 9 17:35:59 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 17:35:59 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 90 previous similar messages Jan 9 17:46:03 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 17:46:03 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 98 previous similar messages Jan 9 17:52:42 oss7 kernel: Lustre: 22749:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0034: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 9 17:56:09 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 17:56:09 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 99 previous similar messages Jan 9 17:56:59 oss7 kernel: Lustre: 22737:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0034: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 9 18:06:10 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 18:06:10 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 92 previous similar messages Jan 9 18:16:13 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 18:16:13 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 93 previous similar messages Jan 9 18:26:13 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 18:26:13 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 95 previous similar messages Jan 9 18:36:14 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 18:36:14 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 98 previous similar messages Jan 9 18:46:28 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 18:46:28 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 96 previous similar messages Jan 9 18:56:30 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 18:56:30 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 99 previous similar messages Jan 9 19:04:16 oss7 kernel: Lustre: 22739:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0034: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 9 19:06:34 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 19:06:34 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 96 previous similar messages Jan 9 19:16:34 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 19:16:34 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 93 previous similar messages Jan 9 19:26:38 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 19:26:38 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 90 previous similar messages Jan 9 19:36:41 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 19:36:41 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 89 previous similar messages Jan 9 19:42:36 oss7 kernel: Lustre: 22728:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0034: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 9 19:45:47 oss7 kernel: LustreError: 22305:0:(events.c:387:server_bulk_callback()) event type 4, status -61, desc ffff8105c34b8000 Jan 9 19:45:47 oss7 kernel: LustreError: 22305:0:(events.c:387:server_bulk_callback()) event type 2, status -61, desc ffff8105c34b8000 Jan 9 19:45:47 oss7 kernel: LustreError: 22976:0:(ost_handler.c:1078:ost_brw_write()) @@@ network error on bulk GET 0(528080) req@ffff8100be31e400 x1449826069838702/t0 o4->d12174ef-c164-5018-d7f7-03674325c534@:0/0 lens 448/416 e 0 to 0 dl 1389314791 ref 1 fl Interpret:/0/0 rc 0/0 Jan 9 19:45:47 oss7 kernel: Lustre: 22976:0:(ost_handler.c:1229:ost_brw_write()) scratch-OST0034: ignoring bulk IO comm error with d12174ef-c164-5018-d7f7-03674325c534@ id 12345-10.191.133.6@o2ib - client will retry Jan 9 19:46:32 oss7 kernel: Lustre: 22701:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0034: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 9 19:46:32 oss7 kernel: LustreError: 22304:0:(events.c:387:server_bulk_callback()) event type 4, status -61, desc ffff810599faf000 Jan 9 19:46:32 oss7 kernel: LustreError: 22304:0:(events.c:387:server_bulk_callback()) event type 2, status -61, desc ffff810599faf000 Jan 9 19:46:32 oss7 kernel: LustreError: 22866:0:(ost_handler.c:1078:ost_brw_write()) @@@ network error on bulk GET 0(528080) req@ffff81047734d000 x1449826069838952/t0 o4->d12174ef-c164-5018-d7f7-03674325c534@:0/0 lens 448/416 e 0 to 0 dl 1389314837 ref 1 fl Interpret:/2/0 rc 0/0 Jan 9 19:46:32 oss7 kernel: Lustre: 22866:0:(ost_handler.c:1229:ost_brw_write()) scratch-OST0034: ignoring bulk IO comm error with d12174ef-c164-5018-d7f7-03674325c534@ id 12345-10.191.133.6@o2ib - client will retry Jan 9 19:46:44 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 19:46:44 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 92 previous similar messages Jan 9 19:47:18 oss7 kernel: LustreError: 22311:0:(events.c:387:server_bulk_callback()) event type 4, status -61, desc ffff81044b6da000 Jan 9 19:47:18 oss7 kernel: LustreError: 22311:0:(events.c:387:server_bulk_callback()) event type 2, status -61, desc ffff81044b6da000 Jan 9 19:47:18 oss7 kernel: LustreError: 22968:0:(ost_handler.c:1078:ost_brw_write()) @@@ network error on bulk GET 0(1048576) req@ffff81056deae000 x1449826069839334/t0 o4->d12174ef-c164-5018-d7f7-03674325c534@:0/0 lens 448/416 e 0 to 0 dl 1389314883 ref 1 fl Interpret:/0/0 rc 0/0 Jan 9 19:47:18 oss7 kernel: Lustre: 22968:0:(ost_handler.c:1229:ost_brw_write()) scratch-OST0034: ignoring bulk IO comm error with d12174ef-c164-5018-d7f7-03674325c534@ id 12345-10.191.133.6@o2ib - client will retry Jan 9 19:47:19 oss7 kernel: Lustre: 22751:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0034: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 9 19:56:52 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 19:56:52 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 96 previous similar messages Jan 9 20:06:55 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 20:06:55 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 90 previous similar messages Jan 9 20:17:03 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 20:17:03 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 94 previous similar messages Jan 9 20:27:06 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 20:27:06 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 92 previous similar messages Jan 9 20:36:44 oss7 kernel: LustreError: 22307:0:(events.c:387:server_bulk_callback()) event type 4, status -61, desc ffff8105ac0c2000 Jan 9 20:36:44 oss7 kernel: LustreError: 22308:0:(events.c:387:server_bulk_callback()) event type 4, status -61, desc ffff8104402e2000 Jan 9 20:36:44 oss7 kernel: LustreError: 22308:0:(events.c:387:server_bulk_callback()) event type 2, status -61, desc ffff8104402e2000 Jan 9 20:36:44 oss7 kernel: LustreError: 22307:0:(events.c:387:server_bulk_callback()) event type 2, status -61, desc ffff8105ac0c2000 Jan 9 20:36:44 oss7 kernel: LustreError: 22958:0:(ost_handler.c:1078:ost_brw_write()) @@@ network error on bulk GET 0(1048576) req@ffff8104d6a40400 x1449826069865022/t0 o4->d12174ef-c164-5018-d7f7-03674325c534@:0/0 lens 448/416 e 0 to 0 dl 1389317810 ref 1 fl Interpret:/0/0 rc 0/0 Jan 9 20:36:44 oss7 kernel: Lustre: 22958:0:(ost_handler.c:1229:ost_brw_write()) scratch-OST0034: ignoring bulk IO comm error with d12174ef-c164-5018-d7f7-03674325c534@ id 12345-10.191.133.6@o2ib - client will retry Jan 9 20:36:44 oss7 kernel: LustreError: 22308:0:(events.c:387:server_bulk_callback()) event type 4, status -61, desc ffff8104307cc000 Jan 9 20:36:44 oss7 kernel: LustreError: 22308:0:(events.c:387:server_bulk_callback()) event type 2, status -61, desc ffff8104307cc000 Jan 9 20:36:44 oss7 kernel: LustreError: 22308:0:(events.c:387:server_bulk_callback()) event type 4, status -61, desc ffff8104607c2000 Jan 9 20:36:44 oss7 kernel: LustreError: 22308:0:(events.c:387:server_bulk_callback()) event type 2, status -61, desc ffff8104607c2000 Jan 9 20:36:44 oss7 kernel: LustreError: 22307:0:(events.c:387:server_bulk_callback()) event type 4, status -61, desc ffff81047c752000 Jan 9 20:36:44 oss7 kernel: LustreError: 22307:0:(events.c:387:server_bulk_callback()) event type 2, status -61, desc ffff81047c752000 Jan 9 20:36:52 oss7 kernel: Lustre: 22751:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0034: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 9 20:37:14 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 20:37:14 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 91 previous similar messages Jan 9 20:47:14 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 20:47:14 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 88 previous similar messages Jan 9 20:57:22 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 20:57:22 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 89 previous similar messages Jan 9 21:07:24 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 21:07:24 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 92 previous similar messages Jan 9 21:17:28 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 21:17:28 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 92 previous similar messages Jan 9 21:24:08 oss7 kernel: LustreError: 22311:0:(events.c:387:server_bulk_callback()) event type 4, status -61, desc ffff8102a97e5000 Jan 9 21:24:08 oss7 kernel: LustreError: 22311:0:(events.c:387:server_bulk_callback()) event type 2, status -61, desc ffff8102a97e5000 Jan 9 21:24:08 oss7 kernel: LustreError: 23022:0:(ost_handler.c:1078:ost_brw_write()) @@@ network error on bulk GET 0(58000) req@ffff8103d3c69c00 x1449826069892121/t0 o4->d12174ef-c164-5018-d7f7-03674325c534@:0/0 lens 448/416 e 0 to 0 dl 1389320655 ref 1 fl Interpret:/0/0 rc 0/0 Jan 9 21:24:08 oss7 kernel: LustreError: 23022:0:(ost_handler.c:1078:ost_brw_write()) Skipped 4 previous similar messages Jan 9 21:24:08 oss7 kernel: Lustre: 23022:0:(ost_handler.c:1229:ost_brw_write()) scratch-OST003e: ignoring bulk IO comm error with d12174ef-c164-5018-d7f7-03674325c534@ id 12345-10.191.133.6@o2ib - client will retry Jan 9 21:24:08 oss7 kernel: Lustre: 23022:0:(ost_handler.c:1229:ost_brw_write()) Skipped 4 previous similar messages Jan 9 21:24:16 oss7 kernel: Lustre: 22606:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST003e: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 9 21:27:43 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 21:27:43 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 90 previous similar messages Jan 9 21:37:52 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 21:37:52 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 78 previous similar messages Jan 9 21:40:05 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 96292e34-3bb2-8d65-e4ad-042bb8aae323 (at 10.191.131.9@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 9 21:40:05 oss7 kernel: Lustre: Skipped 5 previous similar messages Jan 9 21:47:53 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 21:47:53 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 84 previous similar messages Jan 9 21:57:53 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 21:57:53 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 90 previous similar messages Jan 9 22:08:05 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 22:08:05 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 92 previous similar messages Jan 9 22:18:08 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 22:18:08 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 97 previous similar messages Jan 9 22:28:14 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 22:28:14 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 100 previous similar messages Jan 9 22:38:18 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 22:38:18 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 100 previous similar messages Jan 9 22:48:27 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 22:48:27 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 103 previous similar messages Jan 9 22:58:29 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 22:58:29 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 102 previous similar messages Jan 9 23:05:20 oss7 kernel: Lustre: 22558:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST001c: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 9 23:05:20 oss7 kernel: LustreError: 137-5: UUID 'scratch-OST0011_UUID' is not available for connect (no target) Jan 9 23:05:20 oss7 kernel: LustreError: Skipped 11 previous similar messages Jan 9 23:05:20 oss7 kernel: LustreError: 22519:0:(ldlm_lib.c:1921:target_send_reply_msg()) @@@ processing error (-19) req@ffff8102ed3db000 x1449826069939439/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1389326820 ref 1 fl Interpret:/0/0 rc -19/0 Jan 9 23:05:20 oss7 kernel: LustreError: 22519:0:(ldlm_lib.c:1921:target_send_reply_msg()) Skipped 11 previous similar messages Jan 9 23:08:41 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 23:08:41 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 91 previous similar messages Jan 9 23:18:42 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 23:18:42 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 97 previous similar messages Jan 9 23:28:51 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 23:28:51 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 98 previous similar messages Jan 9 23:38:52 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 23:38:52 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 98 previous similar messages Jan 9 23:49:01 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 23:49:01 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 103 previous similar messages Jan 9 23:59:03 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 9 23:59:03 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 100 previous similar messages Jan 10 00:09:04 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 00:09:04 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 100 previous similar messages Jan 10 00:19:08 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 00:19:08 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 100 previous similar messages Jan 10 00:29:08 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 00:29:08 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 101 previous similar messages Jan 10 00:39:08 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 00:39:08 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 101 previous similar messages Jan 10 00:49:18 oss7 kernel: Lustre: 2918:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 00:49:18 oss7 kernel: Lustre: 2918:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 102 previous similar messages Jan 10 00:59:20 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 00:59:20 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 103 previous similar messages Jan 10 01:09:29 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 01:09:29 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 104 previous similar messages Jan 10 01:19:31 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 01:19:31 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 100 previous similar messages Jan 10 01:29:33 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 01:29:33 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 100 previous similar messages Jan 10 01:39:33 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 01:39:33 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 98 previous similar messages Jan 10 01:49:33 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 01:49:33 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 97 previous similar messages Jan 10 01:59:33 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 01:59:33 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 97 previous similar messages Jan 10 02:09:46 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 02:09:46 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 101 previous similar messages Jan 10 02:19:47 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 02:19:47 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 100 previous similar messages Jan 10 02:29:54 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 02:29:54 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 102 previous similar messages Jan 10 02:39:58 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 02:39:58 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 100 previous similar messages Jan 10 02:50:07 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 02:50:07 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 102 previous similar messages Jan 10 03:00:08 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 03:00:08 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 102 previous similar messages Jan 10 03:10:18 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 03:10:18 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 100 previous similar messages Jan 10 03:20:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 03:20:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 103 previous similar messages Jan 10 03:30:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 03:30:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 100 previous similar messages Jan 10 03:40:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 03:40:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 101 previous similar messages Jan 10 03:50:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 03:50:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 99 previous similar messages Jan 10 04:00:23 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 04:00:23 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 100 previous similar messages Jan 10 04:10:35 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 04:10:35 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 100 previous similar messages Jan 10 04:20:36 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 04:20:36 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 103 previous similar messages Jan 10 04:30:44 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 04:30:44 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 102 previous similar messages Jan 10 04:40:47 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 04:40:47 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 102 previous similar messages Jan 10 04:50:48 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 04:50:48 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 86 previous similar messages Jan 10 05:00:48 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 05:00:48 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 86 previous similar messages Jan 10 05:10:48 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 05:10:48 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 87 previous similar messages Jan 10 05:20:57 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 05:20:57 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 96 previous similar messages Jan 10 05:31:00 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 05:31:00 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 98 previous similar messages Jan 10 05:41:08 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 05:41:08 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 98 previous similar messages Jan 10 05:51:09 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 05:51:09 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 95 previous similar messages Jan 10 06:01:09 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 06:01:09 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 100 previous similar messages Jan 10 06:11:13 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 06:11:13 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 102 previous similar messages Jan 10 06:21:13 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 06:21:13 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 98 previous similar messages Jan 10 06:31:13 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 06:31:13 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 107 previous similar messages Jan 10 06:35:49 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client c2c9ba5e-aeed-f099-3867-9aa403d3ca80 (at 10.191.131.9@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 10 06:35:49 oss7 kernel: Lustre: Skipped 5 previous similar messages Jan 10 06:41:25 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 06:41:25 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 104 previous similar messages Jan 10 06:51:26 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 06:51:26 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 104 previous similar messages Jan 10 07:01:34 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 07:01:34 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 107 previous similar messages Jan 10 07:11:37 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 07:11:37 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 102 previous similar messages Jan 10 07:21:47 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 07:21:47 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 102 previous similar messages Jan 10 07:31:49 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 07:31:49 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 102 previous similar messages Jan 10 07:41:58 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 07:41:58 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 100 previous similar messages Jan 10 07:51:59 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 07:51:59 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 102 previous similar messages Jan 10 08:01:59 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 08:01:59 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 102 previous similar messages Jan 10 08:11:59 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 08:11:59 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 103 previous similar messages Jan 10 08:21:59 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 08:21:59 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 100 previous similar messages Jan 10 08:31:59 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 08:31:59 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 101 previous similar messages Jan 10 08:42:02 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 08:42:02 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 97 previous similar messages Jan 10 08:48:26 oss7 kernel: Lustre: scratch-OST0034: haven't heard from client 510792b1-0375-8574-bd0f-581d8682262b (at 10.191.131.76@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 10 08:48:26 oss7 kernel: Lustre: Skipped 5 previous similar messages Jan 10 08:48:26 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 510792b1-0375-8574-bd0f-581d8682262b (at 10.191.131.76@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 10 08:52:03 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 08:52:03 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 100 previous similar messages Jan 10 09:02:03 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 09:02:03 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 99 previous similar messages Jan 10 09:08:44 oss7 kernel: Lustre: 22732:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST001c: 2a6442ed-8070-0e0b-4158-e116c83221a5 reconnecting Jan 10 09:08:44 oss7 kernel: Lustre: 22732:0:(ldlm_lib.c:576:target_handle_reconnect()) Skipped 15 previous similar messages Jan 10 09:08:44 oss7 kernel: LustreError: 137-5: UUID 'scratch-OST0005_UUID' is not available for connect (no target) Jan 10 09:08:44 oss7 kernel: LustreError: Skipped 11 previous similar messages Jan 10 09:08:44 oss7 kernel: LustreError: 22741:0:(ldlm_lib.c:1921:target_send_reply_msg()) @@@ processing error (-19) req@ffff81054c238c00 x1456784476194023/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1389363024 ref 1 fl Interpret:/0/0 rc -19/0 Jan 10 09:08:44 oss7 kernel: LustreError: 22741:0:(ldlm_lib.c:1921:target_send_reply_msg()) Skipped 11 previous similar messages Jan 10 09:12:06 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 09:12:06 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 83 previous similar messages Jan 10 09:22:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 09:22:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 81 previous similar messages Jan 10 09:32:24 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 09:32:24 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 86 previous similar messages Jan 10 09:42:24 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 09:42:24 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 91 previous similar messages Jan 10 09:52:28 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 09:52:28 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 96 previous similar messages Jan 10 10:02:28 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 10:02:28 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 95 previous similar messages Jan 10 10:12:37 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 10:12:37 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 96 previous similar messages Jan 10 10:22:40 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 10:22:40 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 98 previous similar messages Jan 10 10:32:48 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 10:32:48 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 98 previous similar messages Jan 10 10:42:50 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 10:42:50 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 100 previous similar messages Jan 10 10:52:53 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 10:52:53 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 98 previous similar messages Jan 10 11:02:53 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 11:02:53 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 101 previous similar messages Jan 10 11:12:53 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 11:12:53 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 101 previous similar messages Jan 10 11:22:53 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 11:22:53 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 101 previous similar messages Jan 10 11:32:53 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 11:32:53 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 103 previous similar messages Jan 10 11:43:03 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 11:43:03 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 100 previous similar messages Jan 10 11:47:36 oss7 kernel: Lustre: 22682:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0034: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 10 11:47:36 oss7 kernel: Lustre: 22682:0:(ldlm_lib.c:576:target_handle_reconnect()) Skipped 15 previous similar messages Jan 10 11:50:59 oss7 kernel: Lustre: 22742:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST003e: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 10 11:51:39 oss7 kernel: LustreError: 22305:0:(events.c:387:server_bulk_callback()) event type 4, status -61, desc ffff81038688a000 Jan 10 11:51:39 oss7 kernel: LustreError: 22305:0:(events.c:387:server_bulk_callback()) event type 2, status -61, desc ffff81038688a000 Jan 10 11:51:39 oss7 kernel: LustreError: 22309:0:(events.c:387:server_bulk_callback()) event type 4, status -61, desc ffff810446516000 Jan 10 11:51:39 oss7 kernel: LustreError: 22309:0:(events.c:387:server_bulk_callback()) event type 2, status -61, desc ffff810446516000 Jan 10 11:51:39 oss7 kernel: LustreError: 22795:0:(ost_handler.c:1078:ost_brw_write()) @@@ network error on bulk GET 0(1048576) req@ffff81039aa19400 x1449826070341396/t0 o4->d12174ef-c164-5018-d7f7-03674325c534@:0/0 lens 448/416 e 0 to 0 dl 1389372709 ref 1 fl Interpret:/0/0 rc 0/0 Jan 10 11:51:39 oss7 kernel: LustreError: 22305:0:(events.c:387:server_bulk_callback()) event type 4, status -61, desc ffff81046789c000 Jan 10 11:51:39 oss7 kernel: LustreError: 22305:0:(events.c:387:server_bulk_callback()) event type 2, status -61, desc ffff81046789c000 Jan 10 11:51:39 oss7 kernel: Lustre: 22795:0:(ost_handler.c:1229:ost_brw_write()) scratch-OST0010: ignoring bulk IO comm error with d12174ef-c164-5018-d7f7-03674325c534@ id 12345-10.191.133.6@o2ib - client will retry Jan 10 11:51:50 oss7 kernel: Lustre: 22720:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0010: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 10 11:53:05 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 11:53:05 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 111 previous similar messages Jan 10 11:54:26 oss7 kernel: Lustre: 22695:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0010: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 10 11:56:38 oss7 kernel: LustreError: 22312:0:(events.c:387:server_bulk_callback()) event type 4, status -103, desc ffff810434982000 Jan 10 11:56:38 oss7 kernel: LustreError: 22312:0:(events.c:387:server_bulk_callback()) event type 2, status -103, desc ffff810434982000 Jan 10 11:56:38 oss7 kernel: LustreError: 22792:0:(ost_handler.c:1078:ost_brw_write()) @@@ network error on bulk GET 0(1048576) req@ffff8103da1ce000 x1449826070344437/t0 o4->d12174ef-c164-5018-d7f7-03674325c534@:0/0 lens 448/416 e 0 to 0 dl 1389373048 ref 1 fl Interpret:/0/0 rc 0/0 Jan 10 11:56:38 oss7 kernel: LustreError: 22792:0:(ost_handler.c:1078:ost_brw_write()) Skipped 2 previous similar messages Jan 10 11:56:38 oss7 kernel: Lustre: 22792:0:(ost_handler.c:1229:ost_brw_write()) scratch-OST0010: ignoring bulk IO comm error with d12174ef-c164-5018-d7f7-03674325c534@ id 12345-10.191.133.6@o2ib - client will retry Jan 10 11:56:38 oss7 kernel: Lustre: 22792:0:(ost_handler.c:1229:ost_brw_write()) Skipped 2 previous similar messages Jan 10 11:56:39 oss7 kernel: Lustre: 22627:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0010: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 10 12:03:16 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 12:03:16 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 137 previous similar messages Jan 10 12:13:18 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 12:13:18 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 91 previous similar messages Jan 10 12:23:18 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 12:23:18 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 86 previous similar messages Jan 10 12:28:08 oss7 kernel: Lustre: 22522:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0010: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 10 12:33:25 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 12:33:25 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 88 previous similar messages Jan 10 12:43:28 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 12:43:28 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 99 previous similar messages Jan 10 12:45:58 oss7 kernel: Lustre: 22693:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0034: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 10 12:45:58 oss7 kernel: Lustre: 22693:0:(ldlm_lib.c:876:target_handle_connect()) scratch-OST0034: refuse reconnection from d12174ef-c164-5018-d7f7-03674325c534@10.191.133.6@o2ib to 0xffff8105251dfc00; still busy with 1 active RPCs Jan 10 12:45:58 oss7 kernel: LustreError: 22693:0:(ldlm_lib.c:1921:target_send_reply_msg()) @@@ processing error (-16) req@ffff81056deae000 x1449826070370455/t0 o8->d12174ef-c164-5018-d7f7-03674325c534@:0/0 lens 368/264 e 0 to 0 dl 1389376058 ref 1 fl Interpret:/0/0 rc -16/0 Jan 10 12:45:58 oss7 kernel: LustreError: 22693:0:(ldlm_lib.c:1921:target_send_reply_msg()) Skipped 11 previous similar messages Jan 10 12:45:59 oss7 kernel: LustreError: 22866:0:(ost_handler.c:1069:ost_brw_write()) @@@ Reconnect on bulk GET req@ffff8104b5e32c00 x1449826070369382/t0 o4->d12174ef-c164-5018-d7f7-03674325c534@:0/0 lens 448/416 e 2 to 0 dl 1389376067 ref 1 fl Interpret:/0/0 rc 0/0 Jan 10 12:45:59 oss7 kernel: LustreError: 22866:0:(ost_handler.c:1069:ost_brw_write()) Skipped 2 previous similar messages Jan 10 12:45:59 oss7 kernel: Lustre: 22866:0:(ost_handler.c:1229:ost_brw_write()) scratch-OST0034: ignoring bulk IO comm error with d12174ef-c164-5018-d7f7-03674325c534@ id 12345-10.191.133.6@o2ib - client will retry Jan 10 12:46:05 oss7 kernel: LustreError: 137-5: UUID 'scratch-OST0005_UUID' is not available for connect (no target) Jan 10 12:46:05 oss7 kernel: LustreError: Skipped 11 previous similar messages Jan 10 12:46:05 oss7 kernel: LustreError: 22684:0:(ldlm_lib.c:1921:target_send_reply_msg()) @@@ processing error (-19) req@ffff81027a678050 x1456784476296883/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1389376065 ref 1 fl Interpret:/0/0 rc -19/0 Jan 10 12:46:05 oss7 kernel: Lustre: 22577:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0034: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 10 12:46:20 oss7 kernel: LustreError: 137-5: UUID 'scratch-OST0005_UUID' is not available for connect (no target) Jan 10 12:46:20 oss7 kernel: LustreError: Skipped 5 previous similar messages Jan 10 12:46:20 oss7 kernel: LustreError: 22529:0:(ldlm_lib.c:1921:target_send_reply_msg()) @@@ processing error (-19) req@ffff810535303000 x1456784476296976/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1389376080 ref 1 fl Interpret:/0/0 rc -19/0 Jan 10 12:46:20 oss7 kernel: LustreError: 22529:0:(ldlm_lib.c:1921:target_send_reply_msg()) Skipped 5 previous similar messages Jan 10 12:53:38 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 12:53:38 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 88 previous similar messages Jan 10 13:03:39 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 13:03:39 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 86 previous similar messages Jan 10 13:13:39 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 13:13:39 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 91 previous similar messages Jan 10 13:14:31 oss7 kernel: LustreError: 22305:0:(events.c:387:server_bulk_callback()) event type 4, status -61, desc ffff81038a532000 Jan 10 13:14:31 oss7 kernel: LustreError: 22305:0:(events.c:387:server_bulk_callback()) event type 2, status -61, desc ffff81038a532000 Jan 10 13:14:31 oss7 kernel: LustreError: 22978:0:(ost_handler.c:1078:ost_brw_write()) @@@ network error on bulk GET 0(1048576) req@ffff8103dc08bc00 x1449826070384236/t0 o4->d12174ef-c164-5018-d7f7-03674325c534@:0/0 lens 448/416 e 0 to 0 dl 1389377716 ref 1 fl Interpret:/0/0 rc 0/0 Jan 10 13:14:31 oss7 kernel: Lustre: 22978:0:(ost_handler.c:1229:ost_brw_write()) scratch-OST0010: ignoring bulk IO comm error with d12174ef-c164-5018-d7f7-03674325c534@ id 12345-10.191.133.6@o2ib - client will retry Jan 10 13:15:17 oss7 kernel: Lustre: 22764:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0010: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 10 13:16:03 oss7 kernel: Lustre: 22594:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0010: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 10 13:22:54 oss7 kernel: LustreError: 22307:0:(events.c:387:server_bulk_callback()) event type 4, status -61, desc ffff810504b5e000 Jan 10 13:22:54 oss7 kernel: LustreError: 22307:0:(events.c:387:server_bulk_callback()) event type 2, status -61, desc ffff810504b5e000 Jan 10 13:22:54 oss7 kernel: LustreError: 22893:0:(ost_handler.c:1078:ost_brw_write()) @@@ network error on bulk GET 0(1048576) req@ffff81007524d400 x1449826070388449/t0 o4->d12174ef-c164-5018-d7f7-03674325c534@:0/0 lens 448/416 e 0 to 0 dl 1389378216 ref 1 fl Interpret:/0/0 rc 0/0 Jan 10 13:22:54 oss7 kernel: Lustre: 22893:0:(ost_handler.c:1229:ost_brw_write()) scratch-OST0010: ignoring bulk IO comm error with d12174ef-c164-5018-d7f7-03674325c534@ id 12345-10.191.133.6@o2ib - client will retry Jan 10 13:23:37 oss7 kernel: Lustre: 22583:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0010: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 10 13:23:42 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 13:23:42 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 97 previous similar messages Jan 10 13:28:17 oss7 kernel: LustreError: 22306:0:(events.c:387:server_bulk_callback()) event type 4, status -61, desc ffff8104c38fb000 Jan 10 13:28:17 oss7 kernel: LustreError: 22306:0:(events.c:387:server_bulk_callback()) event type 2, status -61, desc ffff8104c38fb000 Jan 10 13:28:17 oss7 kernel: LustreError: 22918:0:(ost_handler.c:1078:ost_brw_write()) @@@ network error on bulk GET 0(767688) req@ffff8105e7fed400 x1449826070391054/t0 o4->d12174ef-c164-5018-d7f7-03674325c534@:0/0 lens 448/416 e 0 to 0 dl 1389378504 ref 1 fl Interpret:/0/0 rc 0/0 Jan 10 13:28:17 oss7 kernel: Lustre: 22918:0:(ost_handler.c:1229:ost_brw_write()) scratch-OST0010: ignoring bulk IO comm error with d12174ef-c164-5018-d7f7-03674325c534@ id 12345-10.191.133.6@o2ib - client will retry Jan 10 13:28:25 oss7 kernel: Lustre: 22602:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0010: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 10 13:33:43 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 13:33:43 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 93 previous similar messages Jan 10 13:43:43 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 13:43:43 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 94 previous similar messages Jan 10 13:53:47 oss7 kernel: LustreError: 22308:0:(events.c:387:server_bulk_callback()) event type 4, status -61, desc ffff810452186000 Jan 10 13:53:47 oss7 kernel: LustreError: 22308:0:(events.c:387:server_bulk_callback()) event type 2, status -61, desc ffff810452186000 Jan 10 13:53:47 oss7 kernel: LustreError: 22809:0:(ost_handler.c:1078:ost_brw_write()) @@@ network error on bulk GET 0(1048576) req@ffff810043c7f400 x1449826070403291/t0 o4->d12174ef-c164-5018-d7f7-03674325c534@:0/0 lens 448/416 e 0 to 0 dl 1389380063 ref 1 fl Interpret:/0/0 rc 0/0 Jan 10 13:53:47 oss7 kernel: Lustre: 22809:0:(ost_handler.c:1229:ost_brw_write()) scratch-OST0010: ignoring bulk IO comm error with d12174ef-c164-5018-d7f7-03674325c534@ id 12345-10.191.133.6@o2ib - client will retry Jan 10 13:53:48 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 13:53:48 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 88 previous similar messages Jan 10 13:54:24 oss7 kernel: Lustre: 22591:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0010: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 10 14:03:58 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 14:03:58 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 96 previous similar messages Jan 10 14:14:04 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 14:14:04 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 95 previous similar messages Jan 10 14:24:04 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 14:24:04 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 85 previous similar messages Jan 10 14:34:08 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 14:34:08 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 93 previous similar messages Jan 10 14:44:08 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 14:44:08 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 82 previous similar messages Jan 10 14:54:20 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 14:54:20 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 90 previous similar messages Jan 10 15:04:26 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 15:04:26 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 95 previous similar messages Jan 10 15:09:09 oss7 kernel: Lustre: 22674:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST001c: 2a6442ed-8070-0e0b-4158-e116c83221a5 reconnecting Jan 10 15:09:09 oss7 kernel: LustreError: 137-5: UUID 'scratch-OST0005_UUID' is not available for connect (no target) Jan 10 15:09:09 oss7 kernel: LustreError: Skipped 5 previous similar messages Jan 10 15:09:09 oss7 kernel: LustreError: 22747:0:(ldlm_lib.c:1921:target_send_reply_msg()) @@@ processing error (-19) req@ffff810224e10c00 x1456784476364670/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1389384649 ref 1 fl Interpret:/0/0 rc -19/0 Jan 10 15:09:09 oss7 kernel: LustreError: 22747:0:(ldlm_lib.c:1921:target_send_reply_msg()) Skipped 5 previous similar messages Jan 10 15:10:18 oss7 kernel: LustreError: 137-5: UUID 'scratch-OST001d_UUID' is not available for connect (no target) Jan 10 15:10:18 oss7 kernel: LustreError: Skipped 5 previous similar messages Jan 10 15:10:18 oss7 kernel: LustreError: 22659:0:(ldlm_lib.c:1921:target_send_reply_msg()) @@@ processing error (-19) req@ffff8102c2c8d050 x1449826070475466/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1389384718 ref 1 fl Interpret:/0/0 rc -19/0 Jan 10 15:10:18 oss7 kernel: LustreError: 22659:0:(ldlm_lib.c:1921:target_send_reply_msg()) Skipped 5 previous similar messages Jan 10 15:14:29 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 15:14:29 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 79 previous similar messages Jan 10 15:24:31 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 15:24:31 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 80 previous similar messages Jan 10 15:31:33 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client 09e78bb9-3c62-8cd6-cc57-6e794883966c (at 10.191.131.9@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 10 15:31:33 oss7 kernel: Lustre: Skipped 4 previous similar messages Jan 10 15:32:23 oss7 kernel: LustreError: 137-5: UUID 'scratch-OST0029_UUID' is not available for connect (no target) Jan 10 15:32:23 oss7 kernel: LustreError: Skipped 5 previous similar messages Jan 10 15:32:23 oss7 kernel: LustreError: 22630:0:(ldlm_lib.c:1921:target_send_reply_msg()) @@@ processing error (-19) req@ffff81061fe47400 x1449826070486477/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1389386043 ref 1 fl Interpret:/0/0 rc -19/0 Jan 10 15:32:23 oss7 kernel: LustreError: 22630:0:(ldlm_lib.c:1921:target_send_reply_msg()) Skipped 5 previous similar messages Jan 10 15:32:38 oss7 kernel: LustreError: 137-5: UUID 'scratch-OST0005_UUID' is not available for connect (no target) Jan 10 15:32:38 oss7 kernel: LustreError: Skipped 5 previous similar messages Jan 10 15:32:38 oss7 kernel: LustreError: 22645:0:(ldlm_lib.c:1921:target_send_reply_msg()) @@@ processing error (-19) req@ffff810333d75c00 x1449826070486561/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1389386058 ref 1 fl Interpret:/0/0 rc -19/0 Jan 10 15:32:38 oss7 kernel: LustreError: 22645:0:(ldlm_lib.c:1921:target_send_reply_msg()) Skipped 5 previous similar messages Jan 10 15:34:33 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 15:34:33 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 86 previous similar messages Jan 10 15:44:33 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 15:44:33 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 85 previous similar messages Jan 10 15:54:33 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 15:54:33 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 81 previous similar messages Jan 10 16:04:46 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 16:04:46 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 94 previous similar messages Jan 10 16:14:50 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 16:14:50 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 85 previous similar messages Jan 10 16:24:54 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 16:24:54 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 89 previous similar messages Jan 10 16:34:54 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 16:34:54 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 81 previous similar messages Jan 10 16:44:58 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 16:44:58 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 86 previous similar messages Jan 10 16:55:03 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 16:55:03 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 82 previous similar messages Jan 10 17:05:07 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 17:05:07 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 88 previous similar messages Jan 10 17:15:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 17:15:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 85 previous similar messages Jan 10 17:25:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 17:25:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 86 previous similar messages Jan 10 17:35:23 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 17:35:23 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 85 previous similar messages Jan 10 17:45:34 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 17:45:34 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 86 previous similar messages Jan 10 17:55:37 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 17:55:37 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 85 previous similar messages Jan 10 18:05:44 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 18:05:44 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 85 previous similar messages Jan 10 18:15:44 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 18:15:44 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 87 previous similar messages Jan 10 18:25:48 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 18:25:48 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 94 previous similar messages Jan 10 18:36:02 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 18:36:02 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 90 previous similar messages Jan 10 18:46:07 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 18:46:07 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 87 previous similar messages Jan 10 18:56:09 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 18:56:09 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 93 previous similar messages Jan 10 19:06:09 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 19:06:09 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 82 previous similar messages Jan 10 19:16:13 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 19:16:13 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 90 previous similar messages Jan 10 19:26:13 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 19:26:13 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 89 previous similar messages Jan 10 19:36:28 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 19:36:28 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 87 previous similar messages Jan 10 19:46:33 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 19:46:33 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 85 previous similar messages Jan 10 19:56:33 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 19:56:33 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 85 previous similar messages Jan 10 20:06:34 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 20:06:34 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 85 previous similar messages Jan 10 20:16:38 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 20:16:38 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 88 previous similar messages Jan 10 20:26:45 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 20:26:45 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 86 previous similar messages Jan 10 20:36:50 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 20:36:50 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 89 previous similar messages Jan 10 20:46:54 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 20:46:54 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 86 previous similar messages Jan 10 20:56:55 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 20:56:55 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 88 previous similar messages Jan 10 21:06:59 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 21:06:59 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 90 previous similar messages Jan 10 21:17:03 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 21:17:03 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 85 previous similar messages Jan 10 21:27:08 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 21:27:08 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 88 previous similar messages Jan 10 21:37:09 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 21:37:09 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 85 previous similar messages Jan 10 21:47:13 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 21:47:13 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 85 previous similar messages Jan 10 21:57:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 21:57:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 88 previous similar messages Jan 10 22:07:24 oss7 kernel: Lustre: 2918:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 22:07:24 oss7 kernel: Lustre: 2918:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 88 previous similar messages Jan 10 22:17:25 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 22:17:25 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 86 previous similar messages Jan 10 22:20:49 oss7 kernel: Lustre: 22743:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST001c: 2a6442ed-8070-0e0b-4158-e116c83221a5 reconnecting Jan 10 22:20:49 oss7 kernel: Lustre: 22743:0:(ldlm_lib.c:576:target_handle_reconnect()) Skipped 16 previous similar messages Jan 10 22:20:49 oss7 kernel: LustreError: 137-5: UUID 'scratch-OST0005_UUID' is not available for connect (no target) Jan 10 22:20:49 oss7 kernel: LustreError: Skipped 5 previous similar messages Jan 10 22:20:49 oss7 kernel: LustreError: 22747:0:(ldlm_lib.c:1921:target_send_reply_msg()) @@@ processing error (-19) req@ffff810396410800 x1456784476567606/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1389410549 ref 1 fl Interpret:/0/0 rc -19/0 Jan 10 22:20:49 oss7 kernel: LustreError: 22747:0:(ldlm_lib.c:1921:target_send_reply_msg()) Skipped 5 previous similar messages Jan 10 22:27:28 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 22:27:28 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 79 previous similar messages Jan 10 22:37:28 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 22:37:28 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 85 previous similar messages Jan 10 22:47:43 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 22:47:43 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 90 previous similar messages Jan 10 22:51:35 oss7 kernel: Lustre: 22570:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST001c: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 10 22:51:35 oss7 kernel: Lustre: 22570:0:(ldlm_lib.c:576:target_handle_reconnect()) Skipped 16 previous similar messages Jan 10 22:51:35 oss7 kernel: LustreError: 137-5: UUID 'scratch-OST0005_UUID' is not available for connect (no target) Jan 10 22:51:35 oss7 kernel: LustreError: Skipped 11 previous similar messages Jan 10 22:51:35 oss7 kernel: LustreError: 22699:0:(ldlm_lib.c:1921:target_send_reply_msg()) @@@ processing error (-19) req@ffff8100414ae000 x1449826070741623/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1389412395 ref 1 fl Interpret:/0/0 rc -19/0 Jan 10 22:51:35 oss7 kernel: LustreError: 22699:0:(ldlm_lib.c:1921:target_send_reply_msg()) Skipped 11 previous similar messages Jan 10 22:57:49 oss7 kernel: Lustre: 2918:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 22:57:49 oss7 kernel: Lustre: 2918:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 74 previous similar messages Jan 10 23:02:24 oss7 kernel: Lustre: 22629:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0010: 2a6442ed-8070-0e0b-4158-e116c83221a5 reconnecting Jan 10 23:02:24 oss7 kernel: Lustre: 22629:0:(ldlm_lib.c:576:target_handle_reconnect()) Skipped 16 previous similar messages Jan 10 23:02:24 oss7 kernel: LustreError: 137-5: UUID 'scratch-OST0005_UUID' is not available for connect (no target) Jan 10 23:02:24 oss7 kernel: LustreError: Skipped 11 previous similar messages Jan 10 23:02:24 oss7 kernel: LustreError: 22588:0:(ldlm_lib.c:1921:target_send_reply_msg()) @@@ processing error (-19) req@ffff8106177a4400 x1456784476586816/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1389413044 ref 1 fl Interpret:/0/0 rc -19/0 Jan 10 23:02:24 oss7 kernel: LustreError: 22588:0:(ldlm_lib.c:1921:target_send_reply_msg()) Skipped 11 previous similar messages Jan 10 23:02:31 oss7 kernel: Lustre: 22756:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0004: 2a6442ed-8070-0e0b-4158-e116c83221a5 reconnecting Jan 10 23:02:31 oss7 kernel: Lustre: 22756:0:(ldlm_lib.c:576:target_handle_reconnect()) Skipped 9 previous similar messages Jan 10 23:07:53 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 23:07:53 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 82 previous similar messages Jan 10 23:17:55 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 23:17:55 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 80 previous similar messages Jan 10 23:17:55 oss7 kernel: Lustre: 22629:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0004: 2a6442ed-8070-0e0b-4158-e116c83221a5 reconnecting Jan 10 23:17:55 oss7 kernel: Lustre: 22629:0:(ldlm_lib.c:576:target_handle_reconnect()) Skipped 6 previous similar messages Jan 10 23:17:55 oss7 kernel: LustreError: 137-5: UUID 'scratch-OST0005_UUID' is not available for connect (no target) Jan 10 23:17:55 oss7 kernel: LustreError: Skipped 11 previous similar messages Jan 10 23:17:55 oss7 kernel: LustreError: 22674:0:(ldlm_lib.c:1921:target_send_reply_msg()) @@@ processing error (-19) req@ffff810391645000 x1456784476593740/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1389413975 ref 1 fl Interpret:/0/0 rc -19/0 Jan 10 23:17:55 oss7 kernel: LustreError: 22674:0:(ldlm_lib.c:1921:target_send_reply_msg()) Skipped 11 previous similar messages Jan 10 23:17:55 oss7 kernel: LustreError: 137-5: UUID 'scratch-OST0011_UUID' is not available for connect (no target) Jan 10 23:17:55 oss7 kernel: LustreError: Skipped 7 previous similar messages Jan 10 23:17:55 oss7 kernel: LustreError: 22551:0:(ldlm_lib.c:1921:target_send_reply_msg()) @@@ processing error (-19) req@ffff8103ac95f000 x1456784476593843/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1389413975 ref 1 fl Interpret:/0/0 rc -19/0 Jan 10 23:17:55 oss7 kernel: LustreError: 22551:0:(ldlm_lib.c:1921:target_send_reply_msg()) Skipped 7 previous similar messages Jan 10 23:28:00 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 23:28:00 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 82 previous similar messages Jan 10 23:38:02 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 23:38:02 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 87 previous similar messages Jan 10 23:48:39 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 23:48:39 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 85 previous similar messages Jan 10 23:48:39 oss7 kernel: Lustre: 22615:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0004: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 10 23:48:39 oss7 kernel: Lustre: 22615:0:(ldlm_lib.c:576:target_handle_reconnect()) Skipped 15 previous similar messages Jan 10 23:48:45 oss7 kernel: Lustre: 22634:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0004: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 10 23:48:45 oss7 kernel: Lustre: 22634:0:(ldlm_lib.c:576:target_handle_reconnect()) Skipped 17 previous similar messages Jan 10 23:58:40 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 10 23:58:40 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 90 previous similar messages Jan 11 00:08:43 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 00:08:43 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 85 previous similar messages Jan 11 00:18:43 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 00:18:43 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 85 previous similar messages Jan 11 00:27:17 oss7 kernel: Lustre: scratch-OST0004: haven't heard from client b88055d9-4017-9a10-ccbd-fd943076a800 (at 10.191.131.9@o2ib) in 227 seconds. I think it's dead, and I am evicting it. Jan 11 00:27:17 oss7 kernel: Lustre: Skipped 5 previous similar messages Jan 11 00:28:56 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 00:28:56 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 87 previous similar messages Jan 11 00:39:04 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 00:39:04 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 95 previous similar messages Jan 11 00:49:06 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 00:49:06 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 82 previous similar messages Jan 11 00:59:08 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 00:59:08 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 89 previous similar messages Jan 11 01:09:08 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 01:09:08 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 84 previous similar messages Jan 11 01:19:13 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 01:19:13 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 87 previous similar messages Jan 11 01:29:20 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 01:29:20 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 86 previous similar messages Jan 11 01:39:27 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 01:39:27 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 92 previous similar messages Jan 11 01:49:28 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 01:49:28 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 89 previous similar messages Jan 11 01:59:33 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 01:59:33 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 94 previous similar messages Jan 11 02:09:46 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 02:09:46 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 91 previous similar messages Jan 11 02:19:49 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 02:19:49 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 86 previous similar messages Jan 11 02:29:51 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 02:29:51 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 85 previous similar messages Jan 11 02:39:57 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 02:39:57 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 91 previous similar messages Jan 11 02:49:58 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 02:49:58 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 89 previous similar messages Jan 11 02:59:58 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 02:59:58 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 88 previous similar messages Jan 11 03:10:14 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 03:10:14 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 83 previous similar messages Jan 11 03:20:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 03:20:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 89 previous similar messages Jan 11 03:30:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 03:30:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 84 previous similar messages Jan 11 03:40:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 03:40:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 85 previous similar messages Jan 11 03:50:23 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 03:50:23 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 86 previous similar messages Jan 11 04:00:23 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 04:00:23 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 89 previous similar messages Jan 11 04:10:33 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 04:10:33 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 85 previous similar messages Jan 11 04:20:38 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 04:20:38 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 91 previous similar messages Jan 11 04:30:44 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 04:30:44 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 91 previous similar messages Jan 11 04:40:44 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 04:40:44 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 85 previous similar messages Jan 11 04:50:44 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 04:50:44 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 85 previous similar messages Jan 11 05:00:48 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 05:00:48 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 90 previous similar messages Jan 11 05:11:00 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 05:11:00 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 88 previous similar messages Jan 11 05:21:05 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 05:21:05 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 89 previous similar messages Jan 11 05:31:07 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 05:31:07 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 86 previous similar messages Jan 11 05:41:09 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 05:41:09 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 89 previous similar messages Jan 11 05:51:13 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 05:51:13 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 82 previous similar messages Jan 11 06:01:13 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 06:01:13 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 90 previous similar messages Jan 11 06:11:13 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 06:11:13 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 87 previous similar messages Jan 11 06:21:31 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 06:21:31 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 91 previous similar messages Jan 11 06:31:34 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 06:31:34 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 89 previous similar messages Jan 11 06:41:34 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 06:41:34 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 83 previous similar messages Jan 11 06:51:34 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 06:51:34 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 81 previous similar messages Jan 11 07:01:38 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 07:01:38 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 80 previous similar messages Jan 11 07:11:48 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 07:11:48 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 86 previous similar messages Jan 11 07:21:51 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 07:21:51 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 93 previous similar messages Jan 11 07:31:54 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 07:31:54 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 83 previous similar messages Jan 11 07:41:56 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 07:41:56 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 88 previous similar messages Jan 11 07:51:59 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 07:51:59 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 92 previous similar messages Jan 11 08:01:59 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 08:01:59 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 81 previous similar messages Jan 11 08:12:03 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 08:12:03 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 86 previous similar messages Jan 11 08:22:15 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 08:22:15 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 88 previous similar messages Jan 11 08:32:16 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 08:32:16 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 84 previous similar messages Jan 11 08:42:22 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 08:42:22 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 92 previous similar messages Jan 11 08:52:25 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 08:52:25 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 90 previous similar messages Jan 11 09:02:28 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 09:02:28 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 89 previous similar messages Jan 11 09:12:28 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 09:12:28 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 81 previous similar messages Jan 11 09:22:28 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 09:22:28 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 91 previous similar messages Jan 11 09:32:43 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 09:32:43 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 94 previous similar messages Jan 11 09:40:20 oss7 kernel: Lustre: 22707:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST001c: d12174ef-c164-5018-d7f7-03674325c534 reconnecting Jan 11 09:40:20 oss7 kernel: Lustre: 22707:0:(ldlm_lib.c:576:target_handle_reconnect()) Skipped 5 previous similar messages Jan 11 09:40:20 oss7 kernel: LustreError: 137-5: UUID 'scratch-OST0005_UUID' is not available for connect (no target) Jan 11 09:40:20 oss7 kernel: LustreError: Skipped 3 previous similar messages Jan 11 09:40:20 oss7 kernel: LustreError: 22672:0:(ldlm_lib.c:1921:target_send_reply_msg()) @@@ processing error (-19) req@ffff8103b6ad6000 x1449826071080911/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1389451320 ref 1 fl Interpret:/0/0 rc -19/0 Jan 11 09:40:20 oss7 kernel: LustreError: 22672:0:(ldlm_lib.c:1921:target_send_reply_msg()) Skipped 3 previous similar messages Jan 11 09:41:26 oss7 kernel: LustreError: 137-5: UUID 'scratch-OST0029_UUID' is not available for connect (no target) Jan 11 09:41:26 oss7 kernel: LustreError: Skipped 11 previous similar messages Jan 11 09:41:26 oss7 kernel: LustreError: 22735:0:(ldlm_lib.c:1921:target_send_reply_msg()) @@@ processing error (-19) req@ffff8102cd14c400 x1456945060905102/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1389451386 ref 1 fl Interpret:/0/0 rc -19/0 Jan 11 09:41:26 oss7 kernel: LustreError: 22735:0:(ldlm_lib.c:1921:target_send_reply_msg()) Skipped 11 previous similar messages Jan 11 09:41:26 oss7 kernel: Lustre: 22738:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0004: 6a84de1f-ee5c-cae2-3a7a-30cda4be54ac reconnecting Jan 11 09:41:26 oss7 kernel: Lustre: 22738:0:(ldlm_lib.c:576:target_handle_reconnect()) Skipped 14 previous similar messages Jan 11 09:41:29 oss7 kernel: Lustre: 22743:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0004: 6a84de1f-ee5c-cae2-3a7a-30cda4be54ac reconnecting Jan 11 09:41:29 oss7 kernel: Lustre: 22743:0:(ldlm_lib.c:576:target_handle_reconnect()) Skipped 5 previous similar messages Jan 11 09:41:35 oss7 kernel: LustreError: 137-5: UUID 'scratch-OST001d_UUID' is not available for connect (no target) Jan 11 09:41:35 oss7 kernel: LustreError: Skipped 15 previous similar messages Jan 11 09:41:35 oss7 kernel: LustreError: 22753:0:(ldlm_lib.c:1921:target_send_reply_msg()) @@@ processing error (-19) req@ffff81038f94e400 x1456945070342274/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1389451395 ref 1 fl Interpret:/0/0 rc -19/0 Jan 11 09:41:35 oss7 kernel: LustreError: 22753:0:(ldlm_lib.c:1921:target_send_reply_msg()) Skipped 15 previous similar messages Jan 11 09:41:35 oss7 kernel: Lustre: 22613:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0004: 7cee358c-1a32-59e4-2fae-23c2d9194d98 reconnecting Jan 11 09:41:35 oss7 kernel: Lustre: 22613:0:(ldlm_lib.c:576:target_handle_reconnect()) Skipped 5 previous similar messages Jan 11 09:41:37 oss7 kernel: Lustre: 22703:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0004: 7cee358c-1a32-59e4-2fae-23c2d9194d98 reconnecting Jan 11 09:41:37 oss7 kernel: Lustre: 22703:0:(ldlm_lib.c:576:target_handle_reconnect()) Skipped 5 previous similar messages Jan 11 09:42:23 oss7 kernel: LustreError: 137-5: UUID 'scratch-OST001d_UUID' is not available for connect (no target) Jan 11 09:42:23 oss7 kernel: LustreError: Skipped 15 previous similar messages Jan 11 09:42:23 oss7 kernel: LustreError: 22604:0:(ldlm_lib.c:1921:target_send_reply_msg()) @@@ processing error (-19) req@ffff8100637c5800 x1456945121722495/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1389451443 ref 1 fl Interpret:/0/0 rc -19/0 Jan 11 09:42:23 oss7 kernel: LustreError: 22604:0:(ldlm_lib.c:1921:target_send_reply_msg()) Skipped 15 previous similar messages Jan 11 09:42:23 oss7 kernel: Lustre: 22538:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0004: d303de41-1ace-777a-ea3d-5611375bff43 reconnecting Jan 11 09:42:23 oss7 kernel: Lustre: 22538:0:(ldlm_lib.c:576:target_handle_reconnect()) Skipped 7 previous similar messages Jan 11 09:42:26 oss7 kernel: LustreError: 137-5: UUID 'scratch-OST0035_UUID' is not available for connect (no target) Jan 11 09:42:26 oss7 kernel: LustreError: Skipped 15 previous similar messages Jan 11 09:42:26 oss7 kernel: LustreError: 22549:0:(ldlm_lib.c:1921:target_send_reply_msg()) @@@ processing error (-19) req@ffff810131e99400 x1456945123819674/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1389451446 ref 1 fl Interpret:/0/0 rc -19/0 Jan 11 09:42:26 oss7 kernel: LustreError: 22549:0:(ldlm_lib.c:1921:target_send_reply_msg()) Skipped 15 previous similar messages Jan 11 09:43:20 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 09:43:20 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 70 previous similar messages Jan 11 09:44:10 oss7 kernel: LustreError: 137-5: UUID 'scratch-OST0035_UUID' is not available for connect (no target) Jan 11 09:44:10 oss7 kernel: LustreError: Skipped 15 previous similar messages Jan 11 09:44:10 oss7 kernel: LustreError: 22746:0:(ldlm_lib.c:1921:target_send_reply_msg()) @@@ processing error (-19) req@ffff8104bf38c800 x1456784476888717/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1389451550 ref 1 fl Interpret:/0/0 rc -19/0 Jan 11 09:44:10 oss7 kernel: LustreError: 22746:0:(ldlm_lib.c:1921:target_send_reply_msg()) Skipped 15 previous similar messages Jan 11 09:44:10 oss7 kernel: Lustre: 22746:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0034: 2a6442ed-8070-0e0b-4158-e116c83221a5 reconnecting Jan 11 09:44:10 oss7 kernel: Lustre: 22746:0:(ldlm_lib.c:576:target_handle_reconnect()) Skipped 25 previous similar messages Jan 11 09:44:29 oss7 kernel: LustreError: 137-5: UUID 'scratch-OST0035_UUID' is not available for connect (no target) Jan 11 09:44:29 oss7 kernel: LustreError: Skipped 7 previous similar messages Jan 11 09:44:29 oss7 kernel: LustreError: 22522:0:(ldlm_lib.c:1921:target_send_reply_msg()) @@@ processing error (-19) req@ffff8105ef99a400 x1456945252794522/t0 o8->@:0/0 lens 368/0 e 0 to 0 dl 1389451569 ref 1 fl Interpret:/0/0 rc -19/0 Jan 11 09:44:29 oss7 kernel: LustreError: 22522:0:(ldlm_lib.c:1921:target_send_reply_msg()) Skipped 7 previous similar messages Jan 11 09:44:29 oss7 kernel: Lustre: 22569:0:(ldlm_lib.c:576:target_handle_reconnect()) scratch-OST0004: e3a8d2b1-a0ba-28ea-0e06-c9e00f5af8b9 reconnecting Jan 11 09:44:29 oss7 kernel: Lustre: 22569:0:(ldlm_lib.c:576:target_handle_reconnect()) Skipped 19 previous similar messages Jan 11 09:53:35 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 09:53:35 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 77 previous similar messages Jan 11 10:03:39 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 10:03:39 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 83 previous similar messages Jan 11 10:13:43 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 10:13:43 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 86 previous similar messages Jan 11 10:23:43 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 10:23:43 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 86 previous similar messages Jan 11 10:33:43 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 10:33:43 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 87 previous similar messages Jan 11 10:43:55 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 10:43:55 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 92 previous similar messages Jan 11 10:54:04 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 10:54:04 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 111 previous similar messages Jan 11 11:04:08 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 11:04:08 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 106 previous similar messages Jan 11 11:14:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 11:14:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 104 previous similar messages Jan 11 11:24:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 11:24:19 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 106 previous similar messages Jan 11 11:34:29 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 11:34:29 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 104 previous similar messages Jan 11 11:44:31 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 11:44:31 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 100 previous similar messages Jan 11 11:54:33 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib [old ver: 12, new ver: 12] Jan 11 11:54:33 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 101 previous similar messages Jan 11 11:59:52 oss7 kernel: LDISKFS-fs error (device dm-20): ldiskfs_mb_free_metadata: Double free of blocks 30208 (30208 148) Jan 11 11:59:52 oss7 kernel: Aborting journal on device dm-20-8. Jan 11 11:59:52 oss7 kernel: LDISKFS-fs (dm-20): Remounting filesystem read-only Jan 11 11:59:52 oss7 kernel: LDISKFS-fs error (device dm-20) in ldiskfs_reserve_inode_write: Journal has aborted Jan 11 11:59:52 oss7 kernel: LDISKFS-fs error (device dm-20) in ldiskfs_ext_remove_space: Journal has aborted Jan 11 11:59:52 oss7 kernel: LDISKFS-fs error (device dm-20) in ldiskfs_reserve_inode_write: Journal has aborted Jan 11 11:59:52 oss7 kernel: LDISKFS-fs error (device dm-20) in ldiskfs_orphan_del: Journal has aborted Jan 11 11:59:52 oss7 kernel: LDISKFS-fs error (device dm-20) in ldiskfs_reserve_inode_write: Journal has aborted Jan 11 11:59:52 oss7 kernel: LDISKFS-fs error (device dm-20) in ldiskfs_ext_truncate: Journal has aborted Jan 11 11:59:52 oss7 kernel: LustreError: 22850:0:(fsfilt-ldiskfs.c:369:fsfilt_ldiskfs_start()) error starting handle for op 8 (106 credits): rc -30 Jan 11 11:59:52 oss7 kernel: LDISKFS-fs error (device dm-20) in ldiskfs_free_blocks: Journal has aborted Jan 11 11:59:52 oss7 kernel: LDISKFS-fs error (device dm-20) in ldiskfs_reserve_inode_write: Journal has aborted Jan 11 11:59:52 oss7 kernel: LDISKFS-fs error (device dm-20) in ldiskfs_ext_remove_space: Journal has aborted Jan 11 11:59:52 oss7 kernel: LDISKFS-fs error (device dm-20) in ldiskfs_reserve_inode_write: Journal has aborted Jan 11 11:59:52 oss7 kernel: LDISKFS-fs error (device dm-20) in ldiskfs_orphan_del: Journal has aborted Jan 11 11:59:52 oss7 kernel: LDISKFS-fs error (device dm-20) in ldiskfs_reserve_inode_write: Journal has aborted Jan 11 11:59:52 oss7 kernel: LDISKFS-fs error (device dm-20) in ldiskfs_ext_truncate: Journal has aborted Jan 11 11:59:52 oss7 kernel: LustreError: 22463:0:(obd.h:1401:obd_transno_commit_cb()) scratch-OST0028: transno 167595232167 commit error: 2 Jan 11 11:59:52 oss7 kernel: LustreError: 22907:0:(fsfilt-ldiskfs.c:369:fsfilt_ldiskfs_start()) error starting handle for op 8 (106 credits): rc -30 Jan 11 11:59:53 oss7 kernel: LustreError: 23000:0:(fsfilt-ldiskfs.c:503:fsfilt_ldiskfs_brw_start()) can't get handle for 593 credits: rc = -30 Jan 11 11:59:53 oss7 kernel: LustreError: 23000:0:(filter_io_26.c:710:filter_commitrw_write()) error starting transaction: rc = -30 Jan 11 11:59:53 oss7 kernel: LustreError: 23017:0:(fsfilt-ldiskfs.c:503:fsfilt_ldiskfs_brw_start()) can't get handle for 609 credits: rc = -30 Jan 11 11:59:53 oss7 kernel: LustreError: 23017:0:(filter_io_26.c:710:filter_commitrw_write()) error starting transaction: rc = -30 Jan 11 11:59:53 oss7 kernel: LustreError: 22910:0:(filter_io_26.c:710:filter_commitrw_write()) error starting transaction: rc = -30 Jan 11 11:59:53 oss7 kernel: LustreError: 22821:0:(filter_io_26.c:710:filter_commitrw_write()) error starting transaction: rc = -30 Jan 11 11:59:53 oss7 kernel: LustreError: 22826:0:(filter_io_26.c:710:filter_commitrw_write()) error starting transaction: rc = -30 Jan 11 11:59:53 oss7 kernel: LustreError: 22984:0:(filter_io_26.c:710:filter_commitrw_write()) error starting transaction: rc = -30 Jan 11 11:59:53 oss7 kernel: LustreError: 22973:0:(filter_io_26.c:710:filter_commitrw_write()) error starting transaction: rc = -30 Jan 11 11:59:53 oss7 kernel: LustreError: 22797:0:(fsfilt-ldiskfs.c:369:fsfilt_ldiskfs_start()) error starting handle for op 8 (106 credits): rc -30 Jan 11 11:59:53 oss7 kernel: LustreError: 22797:0:(fsfilt-ldiskfs.c:369:fsfilt_ldiskfs_start()) Skipped 16 previous similar messages Jan 11 11:59:53 oss7 kernel: LustreError: 22898:0:(filter_io_26.c:710:filter_commitrw_write()) error starting transaction: rc = -30 Jan 11 11:59:53 oss7 kernel: LustreError: 22847:0:(filter_io_26.c:710:filter_commitrw_write()) error starting transaction: rc = -30 Jan 11 11:59:53 oss7 kernel: LustreError: 22852:0:(fsfilt-ldiskfs.c:503:fsfilt_ldiskfs_brw_start()) can't get handle for 187 credits: rc = -30 Jan 11 11:59:53 oss7 kernel: LustreError: 22852:0:(fsfilt-ldiskfs.c:503:fsfilt_ldiskfs_brw_start()) Skipped 7 previous similar messages Jan 11 11:59:53 oss7 kernel: LustreError: 22852:0:(filter_io_26.c:710:filter_commitrw_write()) error starting transaction: rc = -30 Jan 11 11:59:53 oss7 kernel: LustreError: 22879:0:(filter_io_26.c:710:filter_commitrw_write()) error starting transaction: rc = -30 Jan 11 11:59:54 oss7 kernel: LustreError: 23002:0:(filter_io_26.c:710:filter_commitrw_write()) error starting transaction: rc = -30 Jan 11 11:59:54 oss7 kernel: LustreError: 22812:0:(filter_io_26.c:710:filter_commitrw_write()) error starting transaction: rc = -30 Jan 11 11:59:54 oss7 kernel: LustreError: 23013:0:(filter_io_26.c:710:filter_commitrw_write()) error starting transaction: rc = -30 Jan 11 11:59:54 oss7 kernel: LustreError: 22884:0:(filter_io_26.c:710:filter_commitrw_write()) error starting transaction: rc = -30 Jan 11 11:59:54 oss7 kernel: LustreError: 22968:0:(filter_io_26.c:710:filter_commitrw_write()) error starting transaction: rc = -30 Jan 11 11:59:54 oss7 kernel: LustreError: 23020:0:(filter_io_26.c:710:filter_commitrw_write()) error starting transaction: rc = -30 Jan 11 11:59:54 oss7 kernel: LustreError: 22982:0:(filter_io_26.c:710:filter_commitrw_write()) error starting transaction: rc = -30 Jan 11 11:59:54 oss7 kernel: LustreError: 22800:0:(filter_io_26.c:710:filter_commitrw_write()) error starting transaction: rc = -30 Jan 11 11:59:54 oss7 kernel: LustreError: 22918:0:(filter_io_26.c:710:filter_commitrw_write()) error starting transaction: rc = -30 Jan 11 11:59:54 oss7 kernel: LustreError: 22943:0:(filter_io_26.c:710:filter_commitrw_write()) error starting transaction: rc = -30 Jan 11 11:59:54 oss7 kernel: LustreError: 22905:0:(filter_io_26.c:710:filter_commitrw_write()) error starting transaction: rc = -30 Jan 11 11:59:54 oss7 kernel: LustreError: 22910:0:(filter_io_26.c:710:filter_commitrw_write()) error starting transaction: rc = -30 Jan 11 11:59:54 oss7 kernel: LustreError: 22869:0:(filter_io_26.c:710:filter_commitrw_write()) error starting transaction: rc = -30 Jan 11 11:59:54 oss7 kernel: LustreError: 22864:0:(filter_io_26.c:710:filter_commitrw_write()) error starting transaction: rc = -30 Jan 11 11:59:54 oss7 kernel: LDISKFS-fs warning (device dm-20): kmmpd: kmmpd being stopped since filesystem has been remounted as readonly.