Dec 8 07:56:07 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 08:00:48 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection restored to service atlas2-OST009b using nid 10.36.225.185@o2ib. Dec 8 08:00:48 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 08:00:48 dtn04.ccs.ornl.gov kernel: Lustre: Server atlas2-OST009b_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 08:00:48 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 08:01:08 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 08:01:08 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453425687447959 sent from atlas2-OST009b-osc-ffff880c392f9400 to NID 10.36.225.185@o2ib 20s ago has failed due to network error (567s prior to deadline). Dec 8 08:01:08 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Dec 8 08:01:08 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection to service atlas2-OST009b via nid 10.36.225.185@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 08:01:08 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 08:01:08 dtn04.ccs.ornl.gov kernel: LustreError: 11-0: an error occurred while communicating with 10.36.225.185@o2ib. The ost_connect operation failed with -16 Dec 8 08:01:08 dtn04.ccs.ornl.gov kernel: LustreError: Skipped 1 previous similar message Dec 8 08:06:08 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 08:10:49 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection restored to service atlas2-OST009b using nid 10.36.225.185@o2ib. Dec 8 08:10:49 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 08:10:49 dtn04.ccs.ornl.gov kernel: Lustre: Server atlas2-OST009b_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 08:10:49 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 08:11:09 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 08:11:09 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453425687470743 sent from atlas2-OST009b-osc-ffff880c392f9400 to NID 10.36.225.185@o2ib 20s ago has failed due to network error (567s prior to deadline). Dec 8 08:11:09 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Dec 8 08:11:09 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection to service atlas2-OST009b via nid 10.36.225.185@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 08:11:09 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 08:11:09 dtn04.ccs.ornl.gov kernel: LustreError: 11-0: an error occurred while communicating with 10.36.225.185@o2ib. The ost_connect operation failed with -16 Dec 8 08:11:09 dtn04.ccs.ornl.gov kernel: LustreError: Skipped 1 previous similar message Dec 8 08:16:10 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 08:20:51 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection restored to service atlas2-OST009b using nid 10.36.225.185@o2ib. Dec 8 08:20:51 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 08:20:51 dtn04.ccs.ornl.gov kernel: Lustre: Server atlas2-OST009b_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 08:20:51 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 08:21:11 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 08:21:11 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453425687493347 sent from atlas2-OST009b-osc-ffff880c392f9400 to NID 10.36.225.185@o2ib 20s ago has failed due to network error (567s prior to deadline). Dec 8 08:21:11 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Dec 8 08:21:11 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection to service atlas2-OST009b via nid 10.36.225.185@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 08:21:11 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 08:21:11 dtn04.ccs.ornl.gov kernel: LustreError: 11-0: an error occurred while communicating with 10.36.225.185@o2ib. The ost_connect operation failed with -16 Dec 8 08:21:11 dtn04.ccs.ornl.gov kernel: LustreError: Skipped 1 previous similar message Dec 8 08:26:12 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 08:30:53 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection restored to service atlas2-OST009b using nid 10.36.225.185@o2ib. Dec 8 08:30:53 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 08:30:53 dtn04.ccs.ornl.gov kernel: Lustre: Server atlas2-OST009b_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 08:30:53 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 08:31:13 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 08:31:13 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453425687516402 sent from atlas2-OST009b-osc-ffff880c392f9400 to NID 10.36.225.185@o2ib 20s ago has failed due to network error (567s prior to deadline). Dec 8 08:31:13 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Dec 8 08:31:13 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection to service atlas2-OST009b via nid 10.36.225.185@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 08:31:13 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 08:31:13 dtn04.ccs.ornl.gov kernel: LustreError: 11-0: an error occurred while communicating with 10.36.225.185@o2ib. The ost_connect operation failed with -16 Dec 8 08:31:13 dtn04.ccs.ornl.gov kernel: LustreError: Skipped 1 previous similar message Dec 8 08:36:13 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 08:40:54 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection restored to service atlas2-OST009b using nid 10.36.225.185@o2ib. Dec 8 08:40:54 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 08:40:54 dtn04.ccs.ornl.gov kernel: Lustre: Server atlas2-OST009b_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 08:40:54 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 08:41:14 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 08:41:14 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453425687538813 sent from atlas2-OST009b-osc-ffff880c392f9400 to NID 10.36.225.185@o2ib 20s ago has failed due to network error (567s prior to deadline). Dec 8 08:41:14 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Dec 8 08:41:14 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection to service atlas2-OST009b via nid 10.36.225.185@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 08:41:14 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 08:41:14 dtn04.ccs.ornl.gov kernel: LustreError: 11-0: an error occurred while communicating with 10.36.225.185@o2ib. The ost_connect operation failed with -16 Dec 8 08:41:14 dtn04.ccs.ornl.gov kernel: LustreError: Skipped 1 previous similar message Dec 8 08:46:15 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 08:50:56 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection restored to service atlas2-OST009b using nid 10.36.225.185@o2ib. Dec 8 08:50:56 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 08:50:56 dtn04.ccs.ornl.gov kernel: Lustre: Server atlas2-OST009b_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 08:50:56 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 08:51:16 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 08:51:16 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453425687561971 sent from atlas2-OST009b-osc-ffff880c392f9400 to NID 10.36.225.185@o2ib 20s ago has failed due to network error (567s prior to deadline). Dec 8 08:51:16 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Dec 8 08:51:16 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection to service atlas2-OST009b via nid 10.36.225.185@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 08:51:16 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 08:51:16 dtn04.ccs.ornl.gov kernel: LustreError: 11-0: an error occurred while communicating with 10.36.225.185@o2ib. The ost_connect operation failed with -16 Dec 8 08:51:16 dtn04.ccs.ornl.gov kernel: LustreError: Skipped 1 previous similar message Dec 8 08:56:17 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 09:00:58 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection restored to service atlas2-OST009b using nid 10.36.225.185@o2ib. Dec 8 09:00:58 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 09:00:58 dtn04.ccs.ornl.gov kernel: Lustre: Server atlas2-OST009b_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 09:00:58 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 09:01:18 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 09:01:18 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453425687585146 sent from atlas2-OST009b-osc-ffff880c392f9400 to NID 10.36.225.185@o2ib 20s ago has failed due to network error (567s prior to deadline). Dec 8 09:01:18 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Dec 8 09:01:18 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection to service atlas2-OST009b via nid 10.36.225.185@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 09:01:18 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 09:01:18 dtn04.ccs.ornl.gov kernel: LustreError: 11-0: an error occurred while communicating with 10.36.225.185@o2ib. The ost_connect operation failed with -16 Dec 8 09:01:18 dtn04.ccs.ornl.gov kernel: LustreError: Skipped 1 previous similar message Dec 8 09:06:18 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 09:10:59 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection restored to service atlas2-OST009b using nid 10.36.225.185@o2ib. Dec 8 09:10:59 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 09:10:59 dtn04.ccs.ornl.gov kernel: Lustre: Server atlas2-OST009b_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 09:10:59 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 09:11:19 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 09:11:19 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453425687606810 sent from atlas2-OST009b-osc-ffff880c392f9400 to NID 10.36.225.185@o2ib 20s ago has failed due to network error (567s prior to deadline). Dec 8 09:11:19 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Dec 8 09:11:19 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection to service atlas2-OST009b via nid 10.36.225.185@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 09:11:19 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 09:11:19 dtn04.ccs.ornl.gov kernel: LustreError: 11-0: an error occurred while communicating with 10.36.225.185@o2ib. The ost_connect operation failed with -16 Dec 8 09:11:19 dtn04.ccs.ornl.gov kernel: LustreError: Skipped 1 previous similar message Dec 8 09:16:20 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 09:21:01 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection restored to service atlas2-OST009b using nid 10.36.225.185@o2ib. Dec 8 09:21:01 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 09:21:01 dtn04.ccs.ornl.gov kernel: Lustre: Server atlas2-OST009b_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 09:21:01 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 09:21:21 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 09:21:21 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453425687627617 sent from atlas2-OST009b-osc-ffff880c392f9400 to NID 10.36.225.185@o2ib 20s ago has failed due to network error (567s prior to deadline). Dec 8 09:21:21 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Dec 8 09:21:21 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection to service atlas2-OST009b via nid 10.36.225.185@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 09:21:21 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 09:21:21 dtn04.ccs.ornl.gov kernel: LustreError: 11-0: an error occurred while communicating with 10.36.225.185@o2ib. The ost_connect operation failed with -16 Dec 8 09:21:21 dtn04.ccs.ornl.gov kernel: LustreError: Skipped 1 previous similar message Dec 8 09:26:22 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 09:31:03 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection restored to service atlas2-OST009b using nid 10.36.225.185@o2ib. Dec 8 09:31:03 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 09:31:03 dtn04.ccs.ornl.gov kernel: Lustre: Server atlas2-OST009b_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 09:31:03 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 09:31:23 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 09:31:23 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453425687648412 sent from atlas2-OST009b-osc-ffff880c392f9400 to NID 10.36.225.185@o2ib 20s ago has failed due to network error (567s prior to deadline). Dec 8 09:31:23 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Dec 8 09:31:23 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection to service atlas2-OST009b via nid 10.36.225.185@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 09:31:23 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 09:31:23 dtn04.ccs.ornl.gov kernel: LustreError: 11-0: an error occurred while communicating with 10.36.225.185@o2ib. The ost_connect operation failed with -16 Dec 8 09:31:23 dtn04.ccs.ornl.gov kernel: LustreError: Skipped 1 previous similar message Dec 8 09:36:23 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 09:41:04 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection restored to service atlas2-OST009b using nid 10.36.225.185@o2ib. Dec 8 09:41:04 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 09:41:04 dtn04.ccs.ornl.gov kernel: Lustre: Server atlas2-OST009b_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 09:41:04 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 09:41:24 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 09:41:24 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453425687669041 sent from atlas2-OST009b-osc-ffff880c392f9400 to NID 10.36.225.185@o2ib 20s ago has failed due to network error (567s prior to deadline). Dec 8 09:41:24 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Dec 8 09:41:24 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection to service atlas2-OST009b via nid 10.36.225.185@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 09:41:24 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 09:41:24 dtn04.ccs.ornl.gov kernel: LustreError: 11-0: an error occurred while communicating with 10.36.225.185@o2ib. The ost_connect operation failed with -16 Dec 8 09:41:24 dtn04.ccs.ornl.gov kernel: LustreError: Skipped 1 previous similar message Dec 8 09:46:25 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 09:51:06 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection restored to service atlas2-OST009b using nid 10.36.225.185@o2ib. Dec 8 09:51:06 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 09:51:06 dtn04.ccs.ornl.gov kernel: Lustre: Server atlas2-OST009b_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 09:51:06 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 09:51:26 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 09:51:26 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453425687689764 sent from atlas2-OST009b-osc-ffff880c392f9400 to NID 10.36.225.185@o2ib 20s ago has failed due to network error (567s prior to deadline). Dec 8 09:51:26 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Dec 8 09:51:26 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection to service atlas2-OST009b via nid 10.36.225.185@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 09:51:26 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 09:51:26 dtn04.ccs.ornl.gov kernel: LustreError: 11-0: an error occurred while communicating with 10.36.225.185@o2ib. The ost_connect operation failed with -16 Dec 8 09:51:26 dtn04.ccs.ornl.gov kernel: LustreError: Skipped 1 previous similar message Dec 8 09:56:27 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 10:01:08 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection restored to service atlas2-OST009b using nid 10.36.225.185@o2ib. Dec 8 10:01:08 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 10:01:08 dtn04.ccs.ornl.gov kernel: Lustre: Server atlas2-OST009b_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 10:01:08 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 10:01:28 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 10:01:28 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453425687710452 sent from atlas2-OST009b-osc-ffff880c392f9400 to NID 10.36.225.185@o2ib 20s ago has failed due to network error (567s prior to deadline). Dec 8 10:01:28 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Dec 8 10:01:28 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection to service atlas2-OST009b via nid 10.36.225.185@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 10:01:28 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 10:01:28 dtn04.ccs.ornl.gov kernel: LustreError: 11-0: an error occurred while communicating with 10.36.225.185@o2ib. The ost_connect operation failed with -16 Dec 8 10:01:28 dtn04.ccs.ornl.gov kernel: LustreError: Skipped 1 previous similar message Dec 8 10:06:28 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 10:11:09 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection restored to service atlas2-OST009b using nid 10.36.225.185@o2ib. Dec 8 10:11:09 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 10:11:09 dtn04.ccs.ornl.gov kernel: Lustre: Server atlas2-OST009b_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 10:11:09 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 10:11:29 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 10:11:29 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453425687731165 sent from atlas2-OST009b-osc-ffff880c392f9400 to NID 10.36.225.185@o2ib 20s ago has failed due to network error (567s prior to deadline). Dec 8 10:11:29 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Dec 8 10:11:29 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection to service atlas2-OST009b via nid 10.36.225.185@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 10:11:29 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 10:11:29 dtn04.ccs.ornl.gov kernel: LustreError: 11-0: an error occurred while communicating with 10.36.225.185@o2ib. The ost_connect operation failed with -16 Dec 8 10:11:29 dtn04.ccs.ornl.gov kernel: LustreError: Skipped 1 previous similar message Dec 8 10:16:30 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 10:21:11 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection restored to service atlas2-OST009b using nid 10.36.225.185@o2ib. Dec 8 10:21:11 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 10:21:11 dtn04.ccs.ornl.gov kernel: Lustre: Server atlas2-OST009b_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 10:21:11 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 10:21:31 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 10:21:31 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453425687751837 sent from atlas2-OST009b-osc-ffff880c392f9400 to NID 10.36.225.185@o2ib 20s ago has failed due to network error (567s prior to deadline). Dec 8 10:21:31 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 49 previous similar messages Dec 8 10:21:31 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection to service atlas2-OST009b via nid 10.36.225.185@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 10:21:31 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 10:21:31 dtn04.ccs.ornl.gov kernel: LustreError: 11-0: an error occurred while communicating with 10.36.225.185@o2ib. The ost_connect operation failed with -16 Dec 8 10:21:31 dtn04.ccs.ornl.gov kernel: LustreError: Skipped 1 previous similar message Dec 8 10:26:32 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 10:31:13 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection restored to service atlas2-OST009b using nid 10.36.225.185@o2ib. Dec 8 10:31:13 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 10:31:13 dtn04.ccs.ornl.gov kernel: Lustre: Server atlas2-OST009b_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 10:31:13 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 10:31:33 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 10:31:33 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453425687772564 sent from atlas2-OST009b-osc-ffff880c392f9400 to NID 10.36.225.185@o2ib 20s ago has failed due to network error (567s prior to deadline). Dec 8 10:31:33 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Dec 8 10:31:33 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection to service atlas2-OST009b via nid 10.36.225.185@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 10:31:33 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 10:31:33 dtn04.ccs.ornl.gov kernel: LustreError: 11-0: an error occurred while communicating with 10.36.225.185@o2ib. The ost_connect operation failed with -16 Dec 8 10:31:33 dtn04.ccs.ornl.gov kernel: LustreError: Skipped 1 previous similar message Dec 8 10:36:33 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 10:41:14 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection restored to service atlas2-OST009b using nid 10.36.225.185@o2ib. Dec 8 10:41:14 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 10:41:14 dtn04.ccs.ornl.gov kernel: Lustre: Server atlas2-OST009b_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 10:41:14 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 10:41:34 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 10:41:34 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453425687793460 sent from atlas2-OST009b-osc-ffff880c392f9400 to NID 10.36.225.185@o2ib 20s ago has failed due to network error (567s prior to deadline). Dec 8 10:41:34 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 21 previous similar messages Dec 8 10:41:34 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection to service atlas2-OST009b via nid 10.36.225.185@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 10:41:34 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 10:41:34 dtn04.ccs.ornl.gov kernel: LustreError: 11-0: an error occurred while communicating with 10.36.225.185@o2ib. The ost_connect operation failed with -16 Dec 8 10:41:34 dtn04.ccs.ornl.gov kernel: LustreError: Skipped 1 previous similar message Dec 8 10:46:35 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 10:51:16 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection restored to service atlas2-OST009b using nid 10.36.225.185@o2ib. Dec 8 10:51:16 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 10:51:16 dtn04.ccs.ornl.gov kernel: Lustre: Server atlas2-OST009b_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 10:51:16 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 10:51:36 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 10:51:36 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453425687814261 sent from atlas2-OST009b-osc-ffff880c392f9400 to NID 10.36.225.185@o2ib 20s ago has failed due to network error (567s prior to deadline). Dec 8 10:51:36 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Dec 8 10:51:36 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection to service atlas2-OST009b via nid 10.36.225.185@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 10:51:36 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 10:51:36 dtn04.ccs.ornl.gov kernel: LustreError: 11-0: an error occurred while communicating with 10.36.225.185@o2ib. The ost_connect operation failed with -16 Dec 8 10:51:36 dtn04.ccs.ornl.gov kernel: LustreError: Skipped 1 previous similar message Dec 8 10:56:37 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 11:01:18 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection restored to service atlas2-OST009b using nid 10.36.225.185@o2ib. Dec 8 11:01:18 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 11:01:18 dtn04.ccs.ornl.gov kernel: Lustre: Server atlas2-OST009b_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 11:01:18 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 11:01:38 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 11:01:38 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453425688080193 sent from atlas2-OST009b-osc-ffff880c392f9400 to NID 10.36.225.185@o2ib 20s ago has failed due to network error (567s prior to deadline). Dec 8 11:01:38 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 35 previous similar messages Dec 8 11:01:38 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection to service atlas2-OST009b via nid 10.36.225.185@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 11:01:38 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 11:01:38 dtn04.ccs.ornl.gov kernel: LustreError: 11-0: an error occurred while communicating with 10.36.225.185@o2ib. The ost_connect operation failed with -16 Dec 8 11:01:38 dtn04.ccs.ornl.gov kernel: LustreError: Skipped 1 previous similar message Dec 8 11:06:38 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 11:11:19 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection restored to service atlas2-OST009b using nid 10.36.225.185@o2ib. Dec 8 11:11:19 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 11:11:19 dtn04.ccs.ornl.gov kernel: Lustre: Server atlas2-OST009b_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 11:11:19 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 11:11:39 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 11:11:39 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453425688369946 sent from atlas2-OST009b-osc-ffff880c392f9400 to NID 10.36.225.185@o2ib 20s ago has failed due to network error (567s prior to deadline). Dec 8 11:11:39 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 15 previous similar messages Dec 8 11:11:39 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection to service atlas2-OST009b via nid 10.36.225.185@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 11:11:39 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 11:11:39 dtn04.ccs.ornl.gov kernel: LustreError: 11-0: an error occurred while communicating with 10.36.225.185@o2ib. The ost_connect operation failed with -16 Dec 8 11:11:39 dtn04.ccs.ornl.gov kernel: LustreError: Skipped 1 previous similar message Dec 8 11:16:40 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 11:21:21 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection restored to service atlas2-OST009b using nid 10.36.225.185@o2ib. Dec 8 11:21:21 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 11:21:21 dtn04.ccs.ornl.gov kernel: Lustre: Server atlas2-OST009b_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 11:21:21 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 11:21:41 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 11:21:41 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453425688632845 sent from atlas2-OST009b-osc-ffff880c392f9400 to NID 10.36.225.185@o2ib 20s ago has failed due to network error (567s prior to deadline). Dec 8 11:21:41 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 14 previous similar messages Dec 8 11:21:41 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection to service atlas2-OST009b via nid 10.36.225.185@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 11:21:41 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 11:21:41 dtn04.ccs.ornl.gov kernel: LustreError: 11-0: an error occurred while communicating with 10.36.225.185@o2ib. The ost_connect operation failed with -16 Dec 8 11:21:41 dtn04.ccs.ornl.gov kernel: LustreError: Skipped 1 previous similar message Dec 8 11:26:42 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 11:31:23 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection restored to service atlas2-OST009b using nid 10.36.225.185@o2ib. Dec 8 11:31:23 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 11:31:23 dtn04.ccs.ornl.gov kernel: Lustre: Server atlas2-OST009b_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 11:31:23 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 11:31:43 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 11:31:43 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453425688931359 sent from atlas2-OST009b-osc-ffff880c392f9400 to NID 10.36.225.185@o2ib 20s ago has failed due to network error (567s prior to deadline). Dec 8 11:31:43 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 42 previous similar messages Dec 8 11:31:43 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection to service atlas2-OST009b via nid 10.36.225.185@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 11:31:43 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 11:31:43 dtn04.ccs.ornl.gov kernel: LustreError: 11-0: an error occurred while communicating with 10.36.225.185@o2ib. The ost_connect operation failed with -16 Dec 8 11:31:43 dtn04.ccs.ornl.gov kernel: LustreError: Skipped 1 previous similar message Dec 8 11:36:46 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 11:41:27 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection restored to service atlas2-OST009b using nid 10.36.225.185@o2ib. Dec 8 11:41:27 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 11:41:27 dtn04.ccs.ornl.gov kernel: Lustre: Server atlas2-OST009b_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 11:41:27 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 11:41:46 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 11:41:46 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453425689203206 sent from atlas2-OST009b-osc-ffff880c392f9400 to NID 10.36.225.185@o2ib 19s ago has failed due to network error (567s prior to deadline). Dec 8 11:41:47 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Dec 8 11:41:47 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection to service atlas2-OST009b via nid 10.36.225.185@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 11:41:47 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 11:41:47 dtn04.ccs.ornl.gov kernel: LustreError: 11-0: an error occurred while communicating with 10.36.225.185@o2ib. The ost_connect operation failed with -16 Dec 8 11:41:47 dtn04.ccs.ornl.gov kernel: LustreError: Skipped 1 previous similar message Dec 8 11:46:47 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 11:51:28 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection restored to service atlas2-OST009b using nid 10.36.225.185@o2ib. Dec 8 11:51:28 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 11:51:28 dtn04.ccs.ornl.gov kernel: Lustre: Server atlas2-OST009b_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 11:51:28 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 11:51:48 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 11:51:48 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453425689456747 sent from atlas2-OST009b-osc-ffff880c392f9400 to NID 10.36.225.185@o2ib 20s ago has failed due to network error (567s prior to deadline). Dec 8 11:51:48 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Dec 8 11:51:48 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection to service atlas2-OST009b via nid 10.36.225.185@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 11:51:48 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 11:51:48 dtn04.ccs.ornl.gov kernel: LustreError: 11-0: an error occurred while communicating with 10.36.225.185@o2ib. The ost_connect operation failed with -16 Dec 8 11:51:48 dtn04.ccs.ornl.gov kernel: LustreError: Skipped 1 previous similar message Dec 8 11:56:49 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 12:01:30 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection restored to service atlas2-OST009b using nid 10.36.225.185@o2ib. Dec 8 12:01:30 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 12:01:30 dtn04.ccs.ornl.gov kernel: Lustre: Server atlas2-OST009b_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 12:01:30 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 12:01:50 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 12:01:50 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453425689698153 sent from atlas2-OST009b-osc-ffff880c392f9400 to NID 10.36.225.185@o2ib 20s ago has failed due to network error (567s prior to deadline). Dec 8 12:01:50 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 67 previous similar messages Dec 8 12:01:50 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection to service atlas2-OST009b via nid 10.36.225.185@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 12:01:50 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 12:01:50 dtn04.ccs.ornl.gov kernel: LustreError: 11-0: an error occurred while communicating with 10.36.225.185@o2ib. The ost_connect operation failed with -16 Dec 8 12:01:50 dtn04.ccs.ornl.gov kernel: LustreError: Skipped 1 previous similar message Dec 8 12:06:51 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 12:11:32 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection restored to service atlas2-OST009b using nid 10.36.225.185@o2ib. Dec 8 12:11:32 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 12:11:32 dtn04.ccs.ornl.gov kernel: Lustre: Server atlas2-OST009b_UUID version (2.4.1.0) is much newer than client version (1.8.9) Dec 8 12:11:32 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 12:11:51 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000 Dec 8 12:11:52 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) @@@ Request x1453425689933409 sent from atlas2-OST009b-osc-ffff880c392f9400 to NID 10.36.225.185@o2ib 19s ago has failed due to network error (567s prior to deadline). Dec 8 12:11:52 dtn04.ccs.ornl.gov kernel: Lustre: 24622:0:(client.c:1529:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Dec 8 12:11:52 dtn04.ccs.ornl.gov kernel: Lustre: atlas2-OST009b-osc-ffff880c392f9400: Connection to service atlas2-OST009b via nid 10.36.225.185@o2ib was lost; in progress operations using this service will wait for recovery to complete. Dec 8 12:11:52 dtn04.ccs.ornl.gov kernel: Lustre: Skipped 7 previous similar messages Dec 8 12:11:52 dtn04.ccs.ornl.gov kernel: LustreError: 11-0: an error occurred while communicating with 10.36.225.185@o2ib. The ost_connect operation failed with -16 Dec 8 12:11:52 dtn04.ccs.ornl.gov kernel: LustreError: Skipped 1 previous similar message Dec 8 12:16:52 dtn04.ccs.ornl.gov kernel: LustreError: 24615:0:(events.c:199:client_bulk_callback()) event type 1, status -103, desc ffff880499288000