Dec 8 09:23:27 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13187:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453430541423115 sent from atlas2-OST007d-osc-ffff881039813c00 to NID 10.36.226.45@o2ib 515s ago has timed out (515s prior to deadline). Dec 8 09:23:27 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13187:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 31 previous similar messages Dec 8 09:26:50 dtn-sch01.ccs.ornl.gov kernel: Lustre: atlas2-OST03e0-osc-ffff881039813c00: Connection to service atlas2-OST03e0 via nid 10.36.226.48@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 09:26:50 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 39 previous similar messages Dec 8 09:26:50 dtn-sch01.ccs.ornl.gov kernel: Lustre: atlas2-OST03e0-osc-ffff881039813c00: Connection restored to service atlas2-OST03e0 using nid 10.36.226.48@o2ib. Dec 8 09:26:50 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 41 previous similar messages Dec 8 09:26:50 dtn-sch01.ccs.ornl.gov kernel: Lustre: Server atlas2-OST03e0_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 09:26:50 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 41 previous similar messages Dec 8 09:36:17 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13187:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453430541439221 sent from atlas2-OST03e0-osc-ffff881039813c00 to NID 10.36.226.48@o2ib 567s ago has timed out (567s prior to deadline). Dec 8 09:36:17 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13187:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 48 previous similar messages Dec 8 09:38:27 dtn-sch01.ccs.ornl.gov kernel: Lustre: atlas2-OST0083-osc-ffff881039813c00: Connection to service atlas2-OST0083 via nid 10.36.226.51@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 09:38:27 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 10 previous similar messages Dec 8 09:38:27 dtn-sch01.ccs.ornl.gov kernel: Lustre: atlas2-OST0113-osc-ffff881039813c00: Connection restored to service atlas2-OST0113 using nid 10.36.226.51@o2ib. Dec 8 09:38:27 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 09:38:27 dtn-sch01.ccs.ornl.gov kernel: Lustre: Server atlas2-OST0113_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 09:38:27 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 09:42:17 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13189:0:(import.c:517:import_select_connection()) atlas2-OST0230-osc-ffff881039813c00: tried all connections, increasing latency to 250s Dec 8 09:42:17 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13189:0:(import.c:517:import_select_connection()) Skipped 1 previous similar message Dec 8 09:47:54 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13187:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453430541459246 sent from atlas2-OST01a3-osc-ffff881039813c00 to NID 10.36.226.51@o2ib 567s ago has timed out (567s prior to deadline). Dec 8 09:47:54 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13187:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 34 previous similar messages Dec 8 09:50:57 dtn-sch01.ccs.ornl.gov kernel: Lustre: atlas2-OST007d-osc-ffff881039813c00: Connection to service atlas2-OST007d via nid 10.36.226.45@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 09:50:57 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 12 previous similar messages Dec 8 09:50:57 dtn-sch01.ccs.ornl.gov kernel: Lustre: atlas2-OST0080-osc-ffff881039813c00: Connection restored to service atlas2-OST0080 using nid 10.36.226.48@o2ib. Dec 8 09:50:57 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 15 previous similar messages Dec 8 09:50:57 dtn-sch01.ccs.ornl.gov kernel: Lustre: Server atlas2-OST0080_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 09:50:57 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 15 previous similar messages Dec 8 10:00:19 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13189:0:(import.c:517:import_select_connection()) atlas2-OST007f-osc-ffff881039813c00: tried all connections, increasing latency to 250s Dec 8 10:00:19 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13189:0:(import.c:517:import_select_connection()) Skipped 2 previous similar messages Dec 8 10:00:24 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13187:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453430541477494 sent from atlas2-OST01a0-osc-ffff881039813c00 to NID 10.36.226.48@o2ib 567s ago has timed out (567s prior to deadline). Dec 8 10:00:24 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13187:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 42 previous similar messages Dec 8 10:04:38 dtn-sch01.ccs.ornl.gov kernel: Lustre: atlas2-OST03e0-osc-ffff881039813c00: Connection to service atlas2-OST03e0 via nid 10.36.226.48@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 10:04:38 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 41 previous similar messages Dec 8 10:04:38 dtn-sch01.ccs.ornl.gov kernel: Lustre: atlas2-OST03e0-osc-ffff881039813c00: Connection restored to service atlas2-OST03e0 using nid 10.36.226.48@o2ib. Dec 8 10:04:38 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 39 previous similar messages Dec 8 10:04:38 dtn-sch01.ccs.ornl.gov kernel: Lustre: Server atlas2-OST03e0_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 10:04:38 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 39 previous similar messages Dec 8 10:09:46 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13189:0:(import.c:517:import_select_connection()) atlas2-OST02c0-osc-ffff881039813c00: tried all connections, increasing latency to 250s Dec 8 10:13:27 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13187:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453430541498788 sent from atlas2-OST0080-osc-ffff881039813c00 to NID 10.36.226.48@o2ib 515s ago has timed out (515s prior to deadline). Dec 8 10:13:27 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13187:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 40 previous similar messages Dec 8 10:15:24 dtn-sch01.ccs.ornl.gov kernel: Lustre: atlas2-OST01a3-osc-ffff881039813c00: Connection to service atlas2-OST01a3 via nid 10.36.226.51@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 10:15:24 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 15 previous similar messages Dec 8 10:15:24 dtn-sch01.ccs.ornl.gov kernel: Lustre: atlas2-OST01a3-osc-ffff881039813c00: Connection restored to service atlas2-OST01a3 using nid 10.36.226.51@o2ib. Dec 8 10:15:24 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 17 previous similar messages Dec 8 10:15:24 dtn-sch01.ccs.ornl.gov kernel: Lustre: Server atlas2-OST01a3_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 10:15:24 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 17 previous similar messages Dec 8 10:24:51 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13187:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453430541515876 sent from atlas2-OST01a3-osc-ffff881039813c00 to NID 10.36.226.51@o2ib 567s ago has timed out (567s prior to deadline). Dec 8 10:24:51 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13187:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 43 previous similar messages Dec 8 10:27:54 dtn-sch01.ccs.ornl.gov kernel: Lustre: atlas2-OST01a0-osc-ffff881039813c00: Connection to service atlas2-OST01a0 via nid 10.36.226.48@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 10:27:54 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 32 previous similar messages Dec 8 10:27:54 dtn-sch01.ccs.ornl.gov kernel: Lustre: atlas2-OST01a0-osc-ffff881039813c00: Connection restored to service atlas2-OST01a0 using nid 10.36.226.48@o2ib. Dec 8 10:27:54 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 32 previous similar messages Dec 8 10:27:54 dtn-sch01.ccs.ornl.gov kernel: Lustre: Server atlas2-OST01a0_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 10:27:54 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 32 previous similar messages Dec 8 10:37:17 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13189:0:(import.c:517:import_select_connection()) atlas2-OST02c0-osc-ffff881039813c00: tried all connections, increasing latency to 250s Dec 8 10:37:17 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13189:0:(import.c:517:import_select_connection()) Skipped 1 previous similar message Dec 8 10:37:21 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13187:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453430541536164 sent from atlas2-OST01a0-osc-ffff881039813c00 to NID 10.36.226.48@o2ib 567s ago has timed out (567s prior to deadline). Dec 8 10:37:21 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13187:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 35 previous similar messages Dec 8 10:40:57 dtn-sch01.ccs.ornl.gov kernel: Lustre: atlas2-OST0080-osc-ffff881039813c00: Connection to service atlas2-OST0080 via nid 10.36.226.48@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 10:40:57 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 18 previous similar messages Dec 8 10:40:57 dtn-sch01.ccs.ornl.gov kernel: Lustre: atlas2-OST0080-osc-ffff881039813c00: Connection restored to service atlas2-OST0080 using nid 10.36.226.48@o2ib. Dec 8 10:40:57 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 15 previous similar messages Dec 8 10:40:57 dtn-sch01.ccs.ornl.gov kernel: Lustre: Server atlas2-OST0080_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 10:40:57 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 15 previous similar messages Dec 8 10:42:49 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13189:0:(import.c:517:import_select_connection()) atlas2-OST0113-osc-ffff881039813c00: tried all connections, increasing latency to 250s Dec 8 10:42:49 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13189:0:(import.c:517:import_select_connection()) Skipped 1 previous similar message Dec 8 10:50:24 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13187:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453430541554400 sent from atlas2-OST03e0-osc-ffff881039813c00 to NID 10.36.226.48@o2ib 567s ago has timed out (567s prior to deadline). Dec 8 10:50:24 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13187:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 51 previous similar messages Dec 8 10:52:21 dtn-sch01.ccs.ornl.gov kernel: Lustre: atlas2-OST01a3-osc-ffff881039813c00: Connection to service atlas2-OST01a3 via nid 10.36.226.51@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 10:52:21 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 32 previous similar messages Dec 8 10:52:21 dtn-sch01.ccs.ornl.gov kernel: Lustre: atlas2-OST01a3-osc-ffff881039813c00: Connection restored to service atlas2-OST01a3 using nid 10.36.226.51@o2ib. Dec 8 10:52:21 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 32 previous similar messages Dec 8 10:52:21 dtn-sch01.ccs.ornl.gov kernel: Lustre: Server atlas2-OST01a3_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 10:52:21 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 32 previous similar messages Dec 8 10:52:57 dtn-sch01.ccs.ornl.gov kernel: LustreError: 13161:0:(events.c:199:client_bulk_callback()) event type 0, status -5, desc ffff8809a8c2b400 Dec 8 10:52:57 dtn-sch01.ccs.ornl.gov kernel: LustreError: 11-0: an error occurred while communicating with 10.36.225.85@o2ib. The ost_connect operation failed with -16 Dec 8 10:55:19 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13189:0:(import.c:517:import_select_connection()) atlas2-OST007f-osc-ffff881039813c00: tried all connections, increasing latency to 250s Dec 8 10:55:19 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13189:0:(import.c:517:import_select_connection()) Skipped 2 previous similar messages Dec 8 10:57:59 dtn-sch01.ccs.ornl.gov kernel: LustreError: 13158:0:(events.c:199:client_bulk_callback()) event type 0, status -5, desc ffff8809cc631000 Dec 8 10:57:59 dtn-sch01.ccs.ornl.gov kernel: LustreError: 11-0: an error occurred while communicating with 10.36.225.43@o2ib. The ost_connect operation failed with -16 Dec 8 11:00:57 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13187:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453430541713158 sent from atlas2-OST0083-osc-ffff881039813c00 to NID 10.36.226.51@o2ib 515s ago has timed out (515s prior to deadline). Dec 8 11:00:57 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13187:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 32 previous similar messages Dec 8 11:02:40 dtn-sch01.ccs.ornl.gov kernel: Lustre: atlas1-OST01bd-osc-ffff880fe6c5c000: Connection restored to service atlas1-OST01bd using nid 10.36.225.43@o2ib. Dec 8 11:02:40 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 25 previous similar messages Dec 8 11:02:40 dtn-sch01.ccs.ornl.gov kernel: Lustre: Server atlas1-OST01bd_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 11:02:40 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 25 previous similar messages Dec 8 11:04:51 dtn-sch01.ccs.ornl.gov kernel: Lustre: atlas2-OST01a0-osc-ffff881039813c00: Connection to service atlas2-OST01a0 via nid 10.36.226.48@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 11:04:51 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 25 previous similar messages Dec 8 11:10:21 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13189:0:(import.c:517:import_select_connection()) atlas2-OST02c3-osc-ffff881039813c00: tried all connections, increasing latency to 250s Dec 8 11:10:21 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13189:0:(import.c:517:import_select_connection()) Skipped 2 previous similar messages Dec 8 11:10:21 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13189:0:(import.c:517:import_select_connection()) atlas2-OST0353-osc-ffff881039813c00: tried all connections, increasing latency to 250s Dec 8 11:13:27 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13187:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453430541821312 sent from atlas2-OST007d-osc-ffff881039813c00 to NID 10.36.226.45@o2ib 515s ago has timed out (515s prior to deadline). Dec 8 11:13:27 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13187:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 38 previous similar messages Dec 8 11:13:27 dtn-sch01.ccs.ornl.gov kernel: Lustre: atlas2-OST007d-osc-ffff881039813c00: Connection restored to service atlas2-OST007d using nid 10.36.226.45@o2ib. Dec 8 11:13:27 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 14 previous similar messages Dec 8 11:13:27 dtn-sch01.ccs.ornl.gov kernel: Lustre: Server atlas2-OST007d_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 11:13:27 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 14 previous similar messages Dec 8 11:17:54 dtn-sch01.ccs.ornl.gov kernel: Lustre: atlas2-OST03e0-osc-ffff881039813c00: Connection to service atlas2-OST03e0 via nid 10.36.226.48@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 11:17:54 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 37 previous similar messages Dec 8 11:22:00 dtn-sch01.ccs.ornl.gov kernel: LustreError: 16284:0:(file.c:3348:ll_inode_revalidate_fini()) failure -116 inode 144117425485470173 Dec 8 11:25:13 dtn-sch01.ccs.ornl.gov kernel: LustreError: 19985:0:(ldlm_request.c:1039:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway Dec 8 11:25:13 dtn-sch01.ccs.ornl.gov kernel: LustreError: 19985:0:(ldlm_request.c:1597:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 Dec 8 11:25:13 dtn-sch01.ccs.ornl.gov kernel: Lustre: client widow3-client(ffff880720275c00) umount complete Dec 8 11:25:19 dtn-sch01.ccs.ornl.gov kernel: LustreError: 20186:0:(file.c:3348:ll_inode_revalidate_fini()) failure -116 inode 144117425485470173 Dec 8 11:25:33 dtn-sch01.ccs.ornl.gov kernel: LustreError: 20629:0:(file.c:3348:ll_inode_revalidate_fini()) failure -116 inode 144117425485470173 Dec 8 11:25:40 dtn-sch01.ccs.ornl.gov kernel: LustreError: 20631:0:(file.c:3348:ll_inode_revalidate_fini()) failure -116 inode 144117425485470173 Dec 8 11:25:54 dtn-sch01.ccs.ornl.gov kernel: LustreError: 20636:0:(file.c:3348:ll_inode_revalidate_fini()) failure -116 inode 144117425485470173 Dec 8 11:25:54 dtn-sch01.ccs.ornl.gov kernel: LustreError: 20636:0:(file.c:3348:ll_inode_revalidate_fini()) Skipped 1 previous similar message Dec 8 11:25:59 dtn-sch01.ccs.ornl.gov kernel: LustreError: 20642:0:(file.c:3348:ll_inode_revalidate_fini()) failure -116 inode 144117425485470173 Dec 8 11:25:59 dtn-sch01.ccs.ornl.gov kernel: LustreError: 20642:0:(file.c:3348:ll_inode_revalidate_fini()) Skipped 2 previous similar messages Dec 8 11:26:09 dtn-sch01.ccs.ornl.gov kernel: LustreError: 20823:0:(file.c:3348:ll_inode_revalidate_fini()) failure -116 inode 144117425485470173 Dec 8 11:26:09 dtn-sch01.ccs.ornl.gov kernel: LustreError: 20823:0:(file.c:3348:ll_inode_revalidate_fini()) Skipped 5 previous similar messages Dec 8 11:26:18 dtn-sch01.ccs.ornl.gov kernel: LustreError: 21130:0:(ldlm_request.c:1039:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway Dec 8 11:26:18 dtn-sch01.ccs.ornl.gov kernel: LustreError: 21130:0:(ldlm_request.c:1597:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 Dec 8 11:26:18 dtn-sch01.ccs.ornl.gov kernel: Lustre: client linkfarm-client(ffff880c3a77d800) umount complete Dec 8 11:26:29 dtn-sch01.ccs.ornl.gov kernel: LustreError: 21516:0:(file.c:3348:ll_inode_revalidate_fini()) failure -116 inode 144117425485470173 Dec 8 11:26:29 dtn-sch01.ccs.ornl.gov kernel: LustreError: 21516:0:(file.c:3348:ll_inode_revalidate_fini()) Skipped 3 previous similar messages Dec 8 11:27:03 dtn-sch01.ccs.ornl.gov kernel: LustreError: 21656:0:(file.c:3348:ll_inode_revalidate_fini()) failure -116 inode 144117425485470173 Dec 8 11:27:03 dtn-sch01.ccs.ornl.gov kernel: LustreError: 21656:0:(file.c:3348:ll_inode_revalidate_fini()) Skipped 3 previous similar messages Dec 8 11:27:21 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13187:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453430541840453 sent from atlas2-OST03e0-osc-ffff881039813c00 to NID 10.36.226.48@o2ib 567s ago has timed out (567s prior to deadline). Dec 8 11:27:21 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13187:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 43 previous similar messages Dec 8 11:27:21 dtn-sch01.ccs.ornl.gov kernel: Lustre: atlas2-OST03e0-osc-ffff881039813c00: Connection restored to service atlas2-OST03e0 using nid 10.36.226.48@o2ib. Dec 8 11:27:21 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 32 previous similar messages Dec 8 11:27:21 dtn-sch01.ccs.ornl.gov kernel: Lustre: Server atlas2-OST03e0_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 11:27:21 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 32 previous similar messages Dec 8 11:28:55 dtn-sch01.ccs.ornl.gov kernel: LustreError: 25131:0:(file.c:3348:ll_inode_revalidate_fini()) failure -116 inode 144117425485470173 Dec 8 11:28:55 dtn-sch01.ccs.ornl.gov kernel: LustreError: 25131:0:(file.c:3348:ll_inode_revalidate_fini()) Skipped 2 previous similar messages Dec 8 11:28:55 dtn-sch01.ccs.ornl.gov kernel: Lustre: 25145:0:(obd_config.c:1130:class_config_llog_handler()) skipping 'lmv' config: cmd=cf001,clilmv:lmv Dec 8 11:28:55 dtn-sch01.ccs.ornl.gov kernel: Lustre: 25145:0:(obd_config.c:1130:class_config_llog_handler()) Skipped 2 previous similar messages Dec 8 11:28:55 dtn-sch01.ccs.ornl.gov kernel: Lustre: client supports 64-bits dir hash/offset! Dec 8 11:28:55 dtn-sch01.ccs.ornl.gov kernel: Lustre: Client linkfarm-client(ffff8803f9845800) mount complete Dec 8 11:28:55 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 2 previous similar messages Dec 8 11:29:02 dtn-sch01.ccs.ornl.gov kernel: Lustre: Client widow3-client(ffff8803fa42d800) mount complete Dec 8 11:29:20 dtn-sch01.ccs.ornl.gov kernel: Lustre: atlas2-OST01a3-osc-ffff881039813c00: Connection to service atlas2-OST01a3 via nid 10.36.226.51@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 11:29:20 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 11:38:47 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13187:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453430541860500 sent from atlas2-OST01a3-osc-ffff881039813c00 to NID 10.36.226.51@o2ib 567s ago has timed out (567s prior to deadline). Dec 8 11:38:47 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13187:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 87 previous similar messages Dec 8 11:38:47 dtn-sch01.ccs.ornl.gov kernel: Lustre: atlas2-OST01a3-osc-ffff881039813c00: Connection restored to service atlas2-OST01a3 using nid 10.36.226.51@o2ib. Dec 8 11:38:47 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 11:38:47 dtn-sch01.ccs.ornl.gov kernel: Lustre: Server atlas2-OST01a3_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 11:38:47 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 9 previous similar messages Dec 8 11:40:57 dtn-sch01.ccs.ornl.gov kernel: Lustre: atlas2-OST01a0-osc-ffff881039813c00: Connection to service atlas2-OST01a0 via nid 10.36.226.48@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 11:40:57 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 35 previous similar messages Dec 8 11:41:59 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13189:0:(import.c:517:import_select_connection()) atlas2-OST007d-osc-ffff881039813c00: tried all connections, increasing latency to 250s Dec 8 11:42:00 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13189:0:(import.c:517:import_select_connection()) atlas2-OST02bd-osc-ffff881039813c00: tried all connections, increasing latency to 250s Dec 8 11:42:00 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13189:0:(import.c:517:import_select_connection()) Skipped 17 previous similar messages Dec 8 11:50:24 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13187:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453430541875674 sent from atlas2-OST01a0-osc-ffff881039813c00 to NID 10.36.226.48@o2ib 567s ago has timed out (567s prior to deadline). Dec 8 11:50:24 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13187:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 40 previous similar messages Dec 8 11:50:24 dtn-sch01.ccs.ornl.gov kernel: Lustre: atlas2-OST01a0-osc-ffff881039813c00: Connection restored to service atlas2-OST01a0 using nid 10.36.226.48@o2ib. Dec 8 11:50:24 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 35 previous similar messages Dec 8 11:50:24 dtn-sch01.ccs.ornl.gov kernel: Lustre: Server atlas2-OST01a0_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 11:50:24 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 35 previous similar messages Dec 8 11:52:15 dtn-sch01.ccs.ornl.gov kernel: LustreError: 19287:0:(file.c:3348:ll_inode_revalidate_fini()) failure -116 inode 144117425485470173 Dec 8 11:52:15 dtn-sch01.ccs.ornl.gov kernel: LustreError: 19287:0:(file.c:3348:ll_inode_revalidate_fini()) Skipped 3 previous similar messages Dec 8 11:52:34 dtn-sch01.ccs.ornl.gov kernel: LustreError: 19295:0:(file.c:3348:ll_inode_revalidate_fini()) failure -116 inode 144117425485470173 Dec 8 11:54:51 dtn-sch01.ccs.ornl.gov kernel: Lustre: atlas2-OST03e0-osc-ffff881039813c00: Connection to service atlas2-OST03e0 via nid 10.36.226.48@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 11:54:51 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 11 previous similar messages Dec 8 12:03:27 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13187:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453430541916386 sent from atlas2-OST0080-osc-ffff881039813c00 to NID 10.36.226.48@o2ib 515s ago has timed out (515s prior to deadline). Dec 8 12:03:27 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13187:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 45 previous similar messages Dec 8 12:03:27 dtn-sch01.ccs.ornl.gov kernel: Lustre: atlas2-OST03e0-osc-ffff881039813c00: Connection restored to service atlas2-OST03e0 using nid 10.36.226.48@o2ib. Dec 8 12:03:27 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 11 previous similar messages Dec 8 12:03:27 dtn-sch01.ccs.ornl.gov kernel: Lustre: Server atlas2-OST03e0_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 12:03:27 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 11 previous similar messages Dec 8 12:07:08 dtn-sch01.ccs.ornl.gov kernel: Lustre: atlas2-OST01a3-osc-ffff881039813c00: Connection to service atlas2-OST01a3 via nid 10.36.226.51@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 12:07:08 dtn-sch01.ccs.ornl.gov kernel: Lustre: Skipped 35 previous similar messages Dec 8 12:09:30 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13189:0:(import.c:517:import_select_connection()) atlas2-OST007d-osc-ffff881039813c00: tried all connections, increasing latency to 250s Dec 8 12:09:30 dtn-sch01.ccs.ornl.gov kernel: Lustre: 13189:0:(import.c:517:import_select_connection()) Skipped 9 previous similar messages