ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 LustreError: 137-5: scratch-OST0004_UUID: not available for connect from 10.20.30.72@o2ib (no target) LustreError: Skipped 206 previous similar messages Lustre: scratch-OST0001: haven't heard from client 51b6f8ba-0d6a-0cfd-1b31-bef2e50d9802 (at 10.20.30.1@o2ib) in 228 seconds. I think it's dead, and I am evicting it. exp ffff8808580b8000, cur 1433843183 expire 1433843033 last 1433842955 Lustre: scratch-OST0000: haven't heard from client 51b6f8ba-0d6a-0cfd-1b31-bef2e50d9802 (at 10.20.30.1@o2ib) in 229 seconds. I think it's dead, and I am evicting it. exp ffff88081affb400, cur 1433843184 expire 1433843034 last 1433842955 Lustre: Skipped 11 previous similar messages Lustre: scratch-OST0001: haven't heard from client c87da71c-4e3e-93cc-59d4-22b12b4bae76 (at 10.20.30.13@o2ib) in 228 seconds. I think it's dead, and I am evicting it. exp ffff881041af7800, cur 1433843283 expire 1433843133 last 1433843055 Lustre: Skipped 47 previous similar messages Lustre: scratch-MDT0000: haven't heard from client f33f2984-e74c-9f84-4fa5-663d1a1ef177 (at 10.20.30.34@o2ib) in 223 seconds. I think it's dead, and I am evicting it. exp ffff881041a4b000, cur 1433843378 expire 1433843228 last 1433843155 Lustre: Skipped 109 previous similar messages Lustre: scratch-OST0001: haven't heard from client f33f2984-e74c-9f84-4fa5-663d1a1ef177 (at 10.20.30.34@o2ib) in 233 seconds. I think it's dead, and I am evicting it. exp ffff8810545d4c00, cur 1433843388 expire 1433843238 last 1433843155 Lustre: Skipped 17 previous similar messages Lustre: MGS: haven't heard from client 33ad222f-14d2-919d-d1ab-e6389eeeb81e (at 10.20.30.34@o2ib) in 248 seconds. I think it's dead, and I am evicting it. exp ffff88081abfec00, cur 1433843403 expire 1433843253 last 1433843155 Lustre: Skipped 74 previous similar messages Lustre: scratch-MDT0000: haven't heard from client 4d9b7b2b-dbfe-206b-74cc-32f111f606be (at 10.20.30.51@o2ib) in 228 seconds. I think it's dead, and I am evicting it. exp ffff880819d21400, cur 1433843459 expire 1433843309 last 1433843231 Lustre: Skipped 24 previous similar messages Lustre: scratch-OST0001: haven't heard from client 9225e48b-3c9e-b2b2-87aa-a9b17c9e9523 (at 10.20.30.59@o2ib) in 232 seconds. I think it's dead, and I am evicting it. exp ffff880816535c00, cur 1433843488 expire 1433843338 last 1433843256 Lustre: Skipped 16 previous similar messages ib_send_cm_mra: cm_id_priv->id.state: 0x6 LustreError: 137-5: scratch-OST0003_UUID: not available for connect from 10.20.30.71@o2ib (no target) LustreError: Skipped 2 previous similar messages ib_send_cm_mra: cm_id_priv->id.state: 0x6 LustreError: 137-5: scratch-OST0003_UUID: not available for connect from 10.20.30.1@o2ib (no target) LustreError: Skipped 2 previous similar messages ib_send_cm_mra: cm_id_priv->id.state: 0x6 LustreError: 137-5: scratch-OST0004_UUID: not available for connect from 10.20.30.2@o2ib (no target) LustreError: Skipped 2 previous similar messages ib_send_cm_mra: cm_id_priv->id.state: 0x6 LustreError: 137-5: scratch-OST0004_UUID: not available for connect from 10.20.30.3@o2ib (no target) LustreError: Skipped 2 previous similar messages ib_send_cm_mra: cm_id_priv->id.state: 0x6 LustreError: 137-5: scratch-OST0003_UUID: not available for connect from 10.20.30.4@o2ib (no target) LustreError: Skipped 2 previous similar messages ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 LustreError: 137-5: scratch-OST0004_UUID: not available for connect from 10.20.30.9@o2ib (no target) LustreError: Skipped 5 previous similar messages ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 LustreError: 137-5: scratch-OST0003_UUID: not available for connect from 10.20.30.10@o2ib (no target) LustreError: Skipped 5 previous similar messages ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 LustreError: 137-5: scratch-OST0003_UUID: not available for connect from 10.20.30.13@o2ib (no target) LustreError: Skipped 8 previous similar messages ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 LustreError: 137-5: scratch-OST0003_UUID: not available for connect from 10.20.30.12@o2ib (no target) LustreError: Skipped 20 previous similar messages ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 LustreError: 137-5: scratch-OST0003_UUID: not available for connect from 10.20.30.36@o2ib (no target) LustreError: Skipped 41 previous similar messages ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 LustreError: 137-5: scratch-OST0003_UUID: not available for connect from 10.20.30.60@o2ib (no target) LustreError: Skipped 80 previous similar messages ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 ib_send_cm_mra: cm_id_priv->id.state: 0x6 Lustre: scratch-OST0001: haven't heard from client a96ebeb0-a111-96e6-06c5-a514a12d0065 (at 10.20.30.41@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88105460a000, cur 1433887340 expire 1433887190 last 1433887113 Lustre: Skipped 39 previous similar messages ib_send_cm_mra: cm_id_priv->id.state: 0x6 LustreError: 137-5: scratch-OST0003_UUID: not available for connect from 10.20.30.41@o2ib (no target) LustreError: Skipped 32 previous similar messages Lustre: MGS: haven't heard from client 209ff3fd-3d1e-e05c-8c3c-c2e5b7610510 (at 10.20.30.34@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880859d4e400, cur 1435436641 expire 1435436491 last 1435436414 Lustre: Skipped 4 previous similar messages ib_send_cm_mra: cm_id_priv->id.state: 0x6 LustreError: 137-5: scratch-OST0003_UUID: not available for connect from 10.20.30.34@o2ib (no target) LustreError: Skipped 2 previous similar messages Lustre: 10769:0:(client.c:1869:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1436549455/real 0] req@ffff880c05f17400 x1503486730411124/t0(0) o104->scratch-OST0000@10.20.30.41@o2ib:15/16 lens 296/224 e 0 to 1 dl 1436549462 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 Lustre: 10769:0:(client.c:1869:ptlrpc_expire_one_request()) Skipped 5 previous similar messages LustreError: 138-a: scratch-OST0000: A client on nid 10.20.30.41@o2ib was evicted due to a lock blocking callback time out: rc -107 Lustre: scratch-OST0001: haven't heard from client cbb135df-3373-9291-449d-9a511b2fe23c (at 10.20.30.41@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88103f8ba800, cur 1436549495 expire 1436549345 last 1436549268 Lustre: Skipped 4 previous similar messages Lustre: MGS: haven't heard from client 68f6fe03-d0cb-1626-c837-1ee47e8c1dae (at 10.20.30.41@o2ib) in 228 seconds. I think it's dead, and I am evicting it. exp ffff88103fe22c00, cur 1436549496 expire 1436549346 last 1436549268 Lustre: Skipped 2 previous similar messages ib_send_cm_mra: cm_id_priv->id.state: 0x6 LustreError: 137-5: scratch-OST0004_UUID: not available for connect from 10.20.30.41@o2ib (no target) LustreError: Skipped 2 previous similar messages ib_send_cm_mra: cm_id_priv->id.state: 0x6 LustreError: 137-5: scratch-OST0003_UUID: not available for connect from 10.20.30.69@o2ib (no target) LustreError: Skipped 2 previous similar messages ib_send_cm_mra: cm_id_priv->id.state: 0x6 LustreError: 137-5: scratch-OST0004_UUID: not available for connect from 10.20.30.69@o2ib (no target) LustreError: Skipped 2 previous similar messages Lustre: scratch-MDT0000: haven't heard from client f549c5b5-60c4-e42a-6ff1-83794d025b03 (at 10.20.30.41@o2ib) in 228 seconds. I think it's dead, and I am evicting it. exp ffff880e785a8400, cur 1439184543 expire 1439184393 last 1439184315 Lustre: scratch-OST0000: haven't heard from client f549c5b5-60c4-e42a-6ff1-83794d025b03 (at 10.20.30.41@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88104161f400, cur 1439184549 expire 1439184399 last 1439184322 ib_send_cm_mra: cm_id_priv->id.state: 0x6 LustreError: 137-5: scratch-OST0004_UUID: not available for connect from 10.20.30.41@o2ib (no target) LustreError: Skipped 2 previous similar messages Lustre: scratch-MDT0000: haven't heard from client 4e2c004b-610e-7f62-9f63-12c47f58485c (at 10.20.30.34@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880f0e14e000, cur 1439229008 expire 1439228858 last 1439228781 Lustre: Skipped 3 previous similar messages ib_send_cm_mra: cm_id_priv->id.state: 0x6 LustreError: 137-5: scratch-OST0003_UUID: not available for connect from 10.20.30.34@o2ib (no target) LustreError: Skipped 2 previous similar messages ib_send_cm_mra: cm_id_priv->id.state: 0x6 LustreError: 137-5: scratch-OST0003_UUID: not available for connect from 10.20.30.69@o2ib (no target) LustreError: Skipped 2 previous similar messages Lustre: scratch-OST0002: haven't heard from client 657c9d22-2f12-2e18-ddd9-50bcc79a7233 (at 10.20.30.41@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880cfb30b000, cur 1442958379 expire 1442958229 last 1442958152 Lustre: Skipped 4 previous similar messages Lustre: MGS: haven't heard from client 1c376068-4d3a-ac38-fc57-bed245a2880f (at 10.20.30.41@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881041635400, cur 1442958385 expire 1442958235 last 1442958158 Lustre: Skipped 2 previous similar messages Lustre: scratch-MDT0000: haven't heard from client 657c9d22-2f12-2e18-ddd9-50bcc79a7233 (at 10.20.30.41@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88107026e000, cur 1442958391 expire 1442958241 last 1442958164 ib_send_cm_mra: cm_id_priv->id.state: 0x6 LustreError: 137-5: scratch-OST0003_UUID: not available for connect from 10.20.30.41@o2ib (no target) LustreError: Skipped 2 previous similar messages ib_send_cm_mra: cm_id_priv->id.state: 0x6 LustreError: 137-5: scratch-OST0003_UUID: not available for connect from 10.20.30.69@o2ib (no target) LustreError: Skipped 2 previous similar messages ib_send_cm_mra: cm_id_priv->id.state: 0x6 LustreError: 137-5: scratch-OST0003_UUID: not available for connect from 10.20.30.69@o2ib (no target) LustreError: Skipped 2 previous similar messages LustreError: 16956:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel LustreError: 16953:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel LustreError: 16951:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel LustreError: 16951:0:(ost_handler.c:1764:ost_blocking_ast()) Skipped 4 previous similar messages LustreError: 24127:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel LustreError: 16428:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel LustreError: 20574:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel LustreError: 20574:0:(ost_handler.c:1764:ost_blocking_ast()) Skipped 2 previous similar messages LustreError: 24127:0:(ost_handler.c:1764:ost_blocking_ast()) Error -2 syncing data on lock cancel Lustre: scratch-OST0001: haven't heard from client b8344be8-3d24-b225-9d20-5667496f777b (at 10.20.30.41@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880cd7f97800, cur 1443755908 expire 1443755758 last 1443755681 Lustre: scratch-OST0000: haven't heard from client b8344be8-3d24-b225-9d20-5667496f777b (at 10.20.30.41@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808dfcb1c00, cur 1443755925 expire 1443755775 last 1443755698 Lustre: Skipped 1 previous similar message Lustre: scratch-MDT0000: haven't heard from client b8344be8-3d24-b225-9d20-5667496f777b (at 10.20.30.41@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880e1eb38800, cur 1443755927 expire 1443755777 last 1443755700 ib_send_cm_mra: cm_id_priv->id.state: 0x6 LustreError: 137-5: scratch-OST0003_UUID: not available for connect from 10.20.30.41@o2ib (no target) LustreError: Skipped 2 previous similar messages ib_send_cm_mra: cm_id_priv->id.state: 0x6 LustreError: 137-5: scratch-OST0003_UUID: not available for connect from 10.20.30.69@o2ib (no target) LustreError: Skipped 2 previous similar messages Lustre: MGS: haven't heard from client b83a674c-b833-279a-ee2e-5455d5227755 (at 10.20.30.34@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8810413b7400, cur 1444579824 expire 1444579674 last 1444579597 Lustre: Skipped 1 previous similar message Lustre: scratch-OST0001: haven't heard from client 619f3599-1965-5d09-60d9-764f6ea8712f (at 10.20.30.34@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880be30e8400, cur 1444579824 expire 1444579674 last 1444579597 Lustre: scratch-OST0002: haven't heard from client 619f3599-1965-5d09-60d9-764f6ea8712f (at 10.20.30.34@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff88103fe22c00, cur 1444579831 expire 1444579681 last 1444579604 Lustre: scratch-MDT0000: haven't heard from client 619f3599-1965-5d09-60d9-764f6ea8712f (at 10.20.30.34@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff880c255df400, cur 1444579834 expire 1444579684 last 1444579607 Lustre: Skipped 1 previous similar message ib_send_cm_mra: cm_id_priv->id.state: 0x6 LustreError: 137-5: scratch-OST0003_UUID: not available for connect from 10.20.30.34@o2ib (no target) LustreError: Skipped 2 previous similar messages Lustre: 5659:0:(client.c:1869:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1444712451/real 0] req@ffff880b3f6ecc00 x1503486975158716/t0(0) o104->scratch-MDT0000@10.20.30.41@o2ib:15/16 lens 296/224 e 0 to 1 dl 1444712458 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 LustreError: 138-a: scratch-MDT0000: A client on nid 10.20.30.41@o2ib was evicted due to a lock blocking callback time out: rc -107 Lustre: MGS: haven't heard from client da533c50-a280-d8b4-6b52-f5f2f81e5550 (at 10.20.30.41@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8808627bac00, cur 1444712656 expire 1444712506 last 1444712429 ib_send_cm_mra: cm_id_priv->id.state: 0x6 LustreError: 137-5: scratch-OST0005_UUID: not available for connect from 10.20.30.41@o2ib (no target) LustreError: Skipped 2 previous similar messages Lustre: scratch-OST0001: haven't heard from client 49628f1f-dc9d-f93b-1178-45e0fc7cc42f (at 10.20.30.41@o2ib) in 227 seconds. I think it's dead, and I am evicting it. exp ffff881041698000, cur 1444712673 expire 1444712523 last 1444712446 [root@mds1 log]#