Jul 16 14:23:53 cs04r-sc-serv-48 kernel: Lustre: 7315:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1407112324564365 sent from play01-OST0003-osc-ffff8101af83dc00 to NID 172.23.66.21@tcp 84s ago has timed out (84s prior to deadline).
Jul 16 14:23:53 cs04r-sc-serv-48 kernel:   req@ffff81014027f400 x1407112324564365/t0 o10->play01-OST0003_UUID@172.23.66.21@tcp:6/4 lens 400/592 e 0 to 1 dl 1342445033 ref 1 fl Rpc:/0/0 rc 0/0
Jul 16 14:23:53 cs04r-sc-serv-48 kernel: Lustre: 7315:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 37 previous similar messages
Jul 16 14:23:53 cs04r-sc-serv-48 kernel: Lustre: play01-OST0003-osc-ffff8101af83dc00: Connection to service play01-OST0003 via nid 172.23.66.21@tcp was lost; in progress operations using this service will wait for recovery to complete.
Jul 16 14:23:53 cs04r-sc-serv-48 kernel: Lustre: Skipped 2 previous similar messages
Jul 16 14:24:07 cs04r-sc-serv-48 kernel: Lustre: 4522:0:(import.c:517:import_select_connection()) play01-OST0003-osc-ffff8101af83dc00: tried all connections, increasing latency to 2s
Jul 16 14:24:07 cs04r-sc-serv-48 kernel: Lustre: 4522:0:(import.c:517:import_select_connection()) Skipped 2 previous similar messages
Jul 16 14:24:07 cs04r-sc-serv-48 kernel: LustreError: 11-0: an error occurred while communicating with 172.23.66.21@tcp. The ost_connect operation failed with -114
Jul 16 14:24:07 cs04r-sc-serv-48 kernel: LustreError: Skipped 1 previous similar message
Jul 16 14:24:12 cs04r-sc-serv-48 kernel: Lustre: 7230:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1407112324562867 sent from play01-OST0003-osc-ffff8101af83dc00 to NID 172.23.66.21@tcp 117s ago has timed out (117s prior to deadline).
Jul 16 14:24:12 cs04r-sc-serv-48 kernel:   req@ffff81010cdb7c00 x1407112324562867/t0 o10->play01-OST0003_UUID@172.23.66.21@tcp:6/4 lens 400/592 e 1 to 1 dl 1342445052 ref 1 fl Rpc:/0/0 rc 0/0
Jul 16 14:24:12 cs04r-sc-serv-48 kernel: Lustre: 7230:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 26 previous similar messages
Jul 16 14:24:23 cs04r-sc-serv-48 kernel: Lustre: 4522:0:(import.c:517:import_select_connection()) play01-OST0003-osc-ffff8101af83dc00: tried all connections, increasing latency to 3s
Jul 16 14:24:23 cs04r-sc-serv-48 kernel: LustreError: 11-0: an error occurred while communicating with 172.23.66.21@tcp. The ost_connect operation failed with -114
Jul 16 14:24:41 cs04r-sc-serv-48 kernel: LustreError: 11-0: an error occurred while communicating with 172.23.66.21@tcp. The ost_connect operation failed with -114
Jul 16 14:25:00 cs04r-sc-serv-48 kernel: Lustre: 4521:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1407112324567445 sent from play01-OST0003-osc-ffff8101af83dc00 to NID 172.23.68.17@tcp 9s ago has timed out (9s prior to deadline).
Jul 16 14:25:00 cs04r-sc-serv-48 kernel:   req@ffff81029e10e800 x1407112324567445/t0 o8->play01-OST0003_UUID@172.23.68.17@tcp:28/4 lens 368/584 e 0 to 1 dl 1342445100 ref 1 fl Rpc:N/0/0 rc 0/0
Jul 16 14:25:00 cs04r-sc-serv-48 kernel: Lustre: 4521:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 14 previous similar messages
Jul 16 14:25:01 cs04r-sc-serv-48 kernel: Lustre: 4522:0:(import.c:517:import_select_connection()) play01-OST0003-osc-ffff8101af83dc00: tried all connections, increasing latency to 5s
Jul 16 14:25:01 cs04r-sc-serv-48 kernel: Lustre: 4522:0:(import.c:517:import_select_connection()) Skipped 1 previous similar message
Jul 16 14:25:01 cs04r-sc-serv-48 kernel: LustreError: 11-0: an error occurred while communicating with 172.23.66.21@tcp. The ost_connect operation failed with -114
Jul 16 14:25:26 cs04r-sc-serv-48 kernel: LustreError: 11-0: an error occurred while communicating with 172.23.66.21@tcp. The ost_connect operation failed with -114
Jul 16 14:25:47 cs04r-sc-serv-48 kernel: Lustre: 4522:0:(import.c:517:import_select_connection()) play01-OST0003-osc-ffff8101af83dc00: tried all connections, increasing latency to 7s
Jul 16 14:25:47 cs04r-sc-serv-48 kernel: Lustre: 4522:0:(import.c:517:import_select_connection()) Skipped 1 previous similar message
Jul 16 14:25:47 cs04r-sc-serv-48 kernel: LustreError: 11-0: an error occurred while communicating with 172.23.66.21@tcp. The ost_connect operation failed with -114
Jul 16 14:26:05 cs04r-sc-serv-48 kernel: Lustre: 4520:0:(client.c:1487:ptlrpc_expire_one_request()) @@@ Request x1407112324560521 sent from play01-OST0003-osc-ffff8101af83dc00 to NID 172.23.68.17@tcp 248s ago has timed out (248s prior to deadline).
Jul 16 14:26:05 cs04r-sc-serv-48 kernel:   req@ffff81004c2a9c00 x1407112324560521/t0 o4->play01-OST0003_UUID@172.23.68.17@tcp:6/4 lens 448/608 e 2 to 1 dl 1342445165 ref 2 fl Rpc:/0/0 rc 0/0
Jul 16 14:26:05 cs04r-sc-serv-48 kernel: Lustre: 4520:0:(client.c:1487:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
Jul 16 14:26:13 cs04r-sc-serv-48 kernel: LustreError: 11-0: an error occurred while communicating with 172.23.66.21@tcp. The ost_connect operation failed with -114
Jul 16 14:26:41 cs04r-sc-serv-48 kernel: Lustre: 4521:0:(import.c:855:ptlrpc_connect_interpret()) play01-OST0003_UUID@172.23.66.21@tcp changed server handle from 0x8036ad3b0f3866c2 to 0x8036ad3b0f3dbc39
Jul 16 14:26:41 cs04r-sc-serv-48 kernel: LustreError: 167-0: This client was evicted by play01-OST0003; in progress operations using this service will fail.
Jul 16 14:26:42 cs04r-sc-serv-48 kernel: LustreError: 7373:0:(rw.c:198:ll_file_punch()) obd_truncate fails (-4) ino 6208746
Jul 16 14:26:42 cs04r-sc-serv-48 kernel: LustreError: 7377:0:(rw.c:198:ll_file_punch()) obd_truncate fails (-5) ino 6208751
Jul 16 14:26:42 cs04r-sc-serv-48 kernel: LustreError: 7377:0:(rw.c:198:ll_file_punch()) Skipped 9 previous similar messages
Jul 16 14:26:42 cs04r-sc-serv-48 kernel: LustreError: 4520:0:(client.c:858:ptlrpc_import_delay_req()) @@@ IMP_INVALID  req@ffff8101a9e26800 x1407112324567962/t0 o4->play01-OST0003_UUID@172.23.66.21@tcp:6/4 lens 448/608 e 0 to 1 dl 0 ref 2 fl Rpc:/0/0 rc 0/0
Jul 16 14:26:42 cs04r-sc-serv-48 kernel: LustreError: 4520:0:(client.c:858:ptlrpc_import_delay_req()) Skipped 19 previous similar messages
Jul 16 14:26:42 cs04r-sc-serv-48 kernel: LustreError: 7490:0:(ldlm_resource.c:519:ldlm_namespace_cleanup()) Namespace play01-OST0003-osc-ffff8101af83dc00 resource refcount nonzero (1) after lock cleanup; forcing cleanup.
Jul 16 14:26:42 cs04r-sc-serv-48 kernel: LustreError: 7490:0:(ldlm_resource.c:524:ldlm_namespace_cleanup()) Resource: ffff8102df597e00 (22839877/0/0/0) (rc: 1)
Jul 16 14:26:42 cs04r-sc-serv-48 kernel: Lustre: play01-OST0003-osc-ffff8101af83dc00: Connection restored to service play01-OST0003 using nid 172.23.66.21@tcp.
Jul 16 14:26:42 cs04r-sc-serv-48 kernel: Lustre: Skipped 2 previous similar messages
Jul 16 14:26:42 cs04r-sc-serv-48 kernel: LustreError: 7373:0:(rw.c:198:ll_file_punch()) Skipped 41 previous similar messages
