Jul 16 18:03:23 cs04r-sc-serv-48 kernel: Lustre: 4521:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1407112325877984 sent from play01-OST0000-osc-ffff8101af83dc00 to NID 172.23.66.21@tcp 10s ago has timed out (10s prior to deadline).
Jul 16 18:03:23 cs04r-sc-serv-48 kernel:   req@ffff81010c519000 x1407112325877984/t0 o8->play01-OST0000_UUID@172.23.66.21@tcp:28/4 lens 368/584 e 0 to 1 dl 1342458203 ref 1 fl Rpc:N/0/0 rc 0/0
Jul 16 18:03:23 cs04r-sc-serv-48 kernel: Lustre: 4521:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
Jul 16 18:03:35 cs04r-sc-serv-48 kernel: Lustre: 4522:0:(import.c:517:import_select_connection()) play01-OST0000-osc-ffff8101af83dc00: tried all connections, increasing latency to 6s
Jul 16 18:03:35 cs04r-sc-serv-48 kernel: Lustre: 4522:0:(import.c:517:import_select_connection()) Skipped 2 previous similar messages
Jul 16 18:03:35 cs04r-sc-serv-48 kernel: LustreError: 11-0: an error occurred while communicating with 172.23.66.21@tcp. The ost_connect operation failed with -114
Jul 16 18:03:59 cs04r-sc-serv-48 kernel: LustreError: 11-0: an error occurred while communicating with 172.23.66.21@tcp. The ost_connect operation failed with -114
Jul 16 18:04:00 cs04r-sc-serv-48 kernel: LustreError: 28774:0:(ldlm_request.c:1039:ldlm_cli_cancel_req()) Got rc -11 from cancel RPC: canceling anyway
Jul 16 18:04:00 cs04r-sc-serv-48 kernel: LustreError: 28778:0:(ldlm_request.c:1597:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -11
Jul 16 18:04:00 cs04r-sc-serv-48 kernel: LustreError: 28774:0:(ldlm_request.c:1039:ldlm_cli_cancel_req()) Skipped 8 previous similar messages
Jul 16 18:04:25 cs04r-sc-serv-48 kernel: Lustre: 4521:0:(import.c:855:ptlrpc_connect_interpret()) play01-OST0000_UUID@172.23.66.21@tcp changed server handle from 0x8036ad3b0f68c5b4 to 0x8036ad3b0f6ae2f2
Jul 16 18:04:25 cs04r-sc-serv-48 kernel: LustreError: 167-0: This client was evicted by play01-OST0000; in progress operations using this service will fail.
Jul 16 18:04:26 cs04r-sc-serv-48 kernel: LustreError: 2178:0:(llite_mmap.c:210:ll_tree_unlock()) couldn't unlock -5
Jul 16 18:04:26 cs04r-sc-serv-48 kernel: LustreError: 2187:0:(llite_mmap.c:210:ll_tree_unlock()) couldn't unlock -5
Jul 16 18:04:26 cs04r-sc-serv-48 kernel: LustreError: 9879:0:(file.c:727:ll_page_removal_cb()) writepage inode 6204684(ffff810234f6fd60) of page ffff8101b1fbc198 failed: -5
Jul 16 18:04:26 cs04r-sc-serv-48 kernel: LustreError: 9879:0:(file.c:727:ll_page_removal_cb()) Skipped 663 previous similar messages
Jul 16 18:04:26 cs04r-sc-serv-48 kernel: LustreError: 4520:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff81015658f800 x1407112325878244/t0 o4->play01-OST0000_UUID@172.23.66.21@tcp:6/4 lens 448/608 e 0 to 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/0
Jul 16 18:04:26 cs04r-sc-serv-48 kernel: LustreError: 4520:0:(client.c:858:ptlrpc_import_delay_req()) Skipped 12 previous similar messages
Jul 16 18:04:26 cs04r-sc-serv-48 kernel: LustreError: 2331:0:(import.c:1220:ptlrpc_invalidate_import_thread()) dump the log upon eviction
Jul 16 18:04:26 cs04r-sc-serv-48 kernel: LustreError: dumping log to /tmp/lustre-log.1342458266.2331
Jul 16 18:04:26 cs04r-sc-serv-48 kernel: Lustre: play01-OST0000-osc-ffff8101af83dc00: Connection restored to service play01-OST0000 using nid 172.23.66.21@tcp.
Jul 16 18:04:26 cs04r-sc-serv-48 kernel: Lustre: Skipped 2 previous similar messages
Jul 16 18:04:26 cs04r-sc-serv-48 kernel: LustreError: 2178:0:(llite_mmap.c:210:ll_tree_unlock()) Skipped 7 previous similar messages
Jul 16 18:06:05 cs04r-sc-serv-48 automount[4795]: key "data" not found in map source(s).
Jul 16 18:08:31 cs04r-sc-serv-48 kernel: LustreError: 11-0: an error occurred while communicating with 172.23.66.21@tcp. The ldlm_enqueue operation failed with -107
Jul 16 18:08:31 cs04r-sc-serv-48 kernel: Lustre: play01-OST0003-osc-ffff8101af83dc00: Connection to service play01-OST0003 via nid 172.23.66.21@tcp was lost; in progress operations using this service will wait for recovery to complete.
Jul 16 18:08:31 cs04r-sc-serv-48 kernel: LustreError: 28781:0:(ldlm_request.c:1039:ldlm_cli_cancel_req()) Got rc -107 from cancel RPC: canceling anyway
Jul 16 18:08:31 cs04r-sc-serv-48 kernel: LustreError: 28781:0:(ldlm_request.c:1039:ldlm_cli_cancel_req()) Skipped 1 previous similar message
Jul 16 18:08:31 cs04r-sc-serv-48 kernel: LustreError: 28781:0:(ldlm_request.c:1597:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -107
Jul 16 18:08:31 cs04r-sc-serv-48 kernel: LustreError: 28781:0:(ldlm_request.c:1597:ldlm_cli_cancel_list()) Skipped 11 previous similar messages
Jul 16 18:08:31 cs04r-sc-serv-48 kernel: LustreError: 167-0: This client was evicted by play01-OST0003; in progress operations using this service will fail.
Jul 16 18:08:32 cs04r-sc-serv-48 kernel: LustreError: 4341:0:(llite_mmap.c:210:ll_tree_unlock()) couldn't unlock -5
Jul 16 18:08:32 cs04r-sc-serv-48 kernel: LustreError: 11715:0:(file.c:727:ll_page_removal_cb()) writepage inode 6208954(ffff810112199220) of page ffff8101056cb2d0 failed: -5
Jul 16 18:08:32 cs04r-sc-serv-48 kernel: LustreError: 11715:0:(file.c:727:ll_page_removal_cb()) Skipped 34 previous similar messages
Jul 16 18:08:32 cs04r-sc-serv-48 kernel: LustreError: 4520:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff8101a9e27000 x1407112325899571/t0 o4->play01-OST0003_UUID@172.23.66.21@tcp:6/4 lens 448/608 e 0 to 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/0
Jul 16 18:08:32 cs04r-sc-serv-48 kernel: LustreError: 4520:0:(client.c:858:ptlrpc_import_delay_req()) Skipped 37 previous similar messages
Jul 16 18:08:32 cs04r-sc-serv-48 kernel: LustreError: 4462:0:(ldlm_resource.c:519:ldlm_namespace_cleanup()) Namespace play01-OST0003-osc-ffff8101af83dc00 resource refcount nonzero (1) after lock cleanup; forcing cleanup.
Jul 16 18:08:32 cs04r-sc-serv-48 kernel: LustreError: 4462:0:(ldlm_resource.c:524:ldlm_namespace_cleanup()) Resource: ffff8102514ddb40 (22843740/0/0/0) (rc: 1)
Jul 16 18:08:32 cs04r-sc-serv-48 kernel: LustreError: 4462:0:(import.c:1220:ptlrpc_invalidate_import_thread()) dump the log upon eviction
Jul 16 18:08:32 cs04r-sc-serv-48 kernel: LustreError: dumping log to /tmp/lustre-log.1342458512.4462
Jul 16 18:08:32 cs04r-sc-serv-48 kernel: Lustre: play01-OST0003-osc-ffff8101af83dc00: Connection restored to service play01-OST0003 using nid 172.23.66.21@tcp.
Jul 16 18:08:32 cs04r-sc-serv-48 kernel: LustreError: 4341:0:(llite_mmap.c:210:ll_tree_unlock()) Skipped 1 previous similar message
