Jul 17 10:29:49 cs04r-sc-serv-48 kernel: Lustre: Skipped 5 previous similar messages
Jul 17 10:35:58 cs04r-sc-serv-48 kernel: Lustre: 8751:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1407112326361717 sent from play01-OST0005-osc-ffff8101af83dc00 to NID 172.23.66.21@tcp 111s ago has timed out (111s prior to deadline).
Jul 17 10:35:58 cs04r-sc-serv-48 kernel:   req@ffff8102e48b6400 x1407112326361717/t0 o10->play01-OST0005_UUID@172.23.66.21@tcp:6/4 lens 400/592 e 0 to 1 dl 1342517758 ref 1 fl Rpc:/0/0 rc 0/0
Jul 17 10:35:58 cs04r-sc-serv-48 kernel: Lustre: play01-OST0005-osc-ffff8101af83dc00: Connection to service play01-OST0005 via nid 172.23.66.21@tcp was lost; in progress operations using this service will wait for recovery to complete.
Jul 17 10:36:20 cs04r-sc-serv-48 kernel: Lustre: 4522:0:(import.c:517:import_select_connection()) play01-OST0005-osc-ffff8101af83dc00: tried all connections, increasing latency to 6s
Jul 17 10:36:20 cs04r-sc-serv-48 kernel: LustreError: 11-0: an error occurred while communicating with 172.23.66.21@tcp. The ost_connect operation failed with -114
Jul 17 10:36:36 cs04r-sc-serv-48 kernel: LustreError: 32309:0:(ldlm_request.c:1039:ldlm_cli_cancel_req()) Got rc -11 from cancel RPC: canceling anyway
Jul 17 10:36:36 cs04r-sc-serv-48 kernel: LustreError: 32309:0:(ldlm_request.c:1039:ldlm_cli_cancel_req()) Skipped 8 previous similar messages
Jul 17 10:36:36 cs04r-sc-serv-48 kernel: LustreError: 32309:0:(ldlm_request.c:1597:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -11
Jul 17 10:36:36 cs04r-sc-serv-48 kernel: LustreError: 32309:0:(ldlm_request.c:1597:ldlm_cli_cancel_list()) Skipped 8 previous similar messages
Jul 17 10:36:36 cs04r-sc-serv-48 kernel: LustreError: 28795:0:(ldlm_request.c:1039:ldlm_cli_cancel_req()) Got rc -11 from cancel RPC: canceling anyway
Jul 17 10:36:36 cs04r-sc-serv-48 kernel: LustreError: 11724:0:(ldlm_request.c:1597:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -11
Jul 17 10:36:36 cs04r-sc-serv-48 kernel: LustreError: 28795:0:(ldlm_request.c:1039:ldlm_cli_cancel_req()) Skipped 4 previous similar messages
Jul 17 10:36:44 cs04r-sc-serv-48 kernel: Lustre: 4521:0:(import.c:855:ptlrpc_connect_interpret()) play01-OST0005_UUID@172.23.66.21@tcp changed server handle from 0x8036ad3b1021df5b to 0x8036ad3b1023aa6e
Jul 17 10:36:44 cs04r-sc-serv-48 kernel: LustreError: 167-0: This client was evicted by play01-OST0005; in progress operations using this service will fail.
Jul 17 10:36:44 cs04r-sc-serv-48 kernel: LustreError: 8789:0:(rw.c:198:ll_file_punch()) obd_truncate fails (-4) ino 6205275
Jul 17 10:36:44 cs04r-sc-serv-48 kernel: LustreError: 8695:0:(llite_mmap.c:210:ll_tree_unlock()) couldn't unlock -5
Jul 17 10:36:44 cs04r-sc-serv-48 kernel: LustreError: 4520:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff8100558a3c00 x1407112326363953/t0 o4->play01-OST0005_UUID@172.23.66.21@tcp:6/4 lens 448/608 e 0 to 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/0
Jul 17 10:36:44 cs04r-sc-serv-48 kernel: LustreError: 4520:0:(client.c:858:ptlrpc_import_delay_req()) Skipped 36 previous similar messages
Jul 17 10:36:44 cs04r-sc-serv-48 kernel: LustreError: 8861:0:(ldlm_resource.c:519:ldlm_namespace_cleanup()) Namespace play01-OST0005-osc-ffff8101af83dc00 resource refcount nonzero (1) after lock cleanup; forcing cleanup.
Jul 17 10:36:44 cs04r-sc-serv-48 kernel: LustreError: 8861:0:(ldlm_resource.c:524:ldlm_namespace_cleanup()) Resource: ffff8101d07bae40 (25050393/0/0/0) (rc: 1)
Jul 17 10:36:44 cs04r-sc-serv-48 kernel: LustreError: 8861:0:(import.c:1220:ptlrpc_invalidate_import_thread()) dump the log upon eviction
Jul 17 10:36:44 cs04r-sc-serv-48 kernel: LustreError: dumping log to /tmp/lustre-log.1342517804.8861
Jul 17 10:36:44 cs04r-sc-serv-48 kernel: Lustre: play01-OST0005-osc-ffff8101af83dc00: Connection restored to service play01-OST0005 using nid 172.23.66.21@tcp.
Jul 17 10:36:44 cs04r-sc-serv-48 kernel: LustreError: 8789:0:(rw.c:198:ll_file_punch()) Skipped 17 previous similar messages
