HOSTS ------------------------------------------------------------------------- cpu-e-836 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-08-20 19:48:44 BST, end at Wed 2019-08-21 11:46:27 BST. -- Aug 20 23:16:54 cpu-e-836 kernel: Adding 15999996k swap on /dev/sda2. Priority:-2 extents:1 across:15999996k SSFS Aug 20 23:19:07 cpu-e-836 kernel: Lustre: Unmounted fs1-client HOSTS ------------------------------------------------------------------------- cpu-e-1055 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-08-20 19:50:34 BST, end at Wed 2019-08-21 11:46:27 BST. -- Aug 20 22:50:00 cpu-e-1055 kernel: Adding 15999996k swap on /dev/sda2. Priority:-2 extents:1 across:15999996k SSFS Aug 20 23:00:33 cpu-e-1055 kernel: perf: interrupt took too long (2504 > 2500), lowering kernel.perf_event_max_sample_rate to 79000 Aug 20 23:17:29 cpu-e-1055 kernel: Adding 15999996k swap on /dev/sda2. Priority:-2 extents:1 across:15999996k SSFS Aug 20 23:19:07 cpu-e-1055 kernel: Lustre: Unmounted fs1-client HOSTS ------------------------------------------------------------------------- cpu-e-1056 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-08-20 19:50:00 BST, end at Wed 2019-08-21 11:46:27 BST. -- Aug 20 22:50:00 cpu-e-1056 kernel: Adding 15999996k swap on /dev/sda2. Priority:-2 extents:1 across:15999996k SSFS Aug 20 23:17:29 cpu-e-1056 kernel: Adding 15999996k swap on /dev/sda2. Priority:-2 extents:1 across:15999996k SSFS Aug 20 23:19:07 cpu-e-1056 kernel: Lustre: Unmounted fs1-client HOSTS ------------------------------------------------------------------------- cpu-e-1058 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-08-20 19:50:13 BST, end at Wed 2019-08-21 11:46:27 BST. -- Aug 20 22:50:00 cpu-e-1058 kernel: Adding 15999996k swap on /dev/sda2. Priority:-2 extents:1 across:15999996k SSFS Aug 20 22:55:53 cpu-e-1058 kernel: LustreError: 66177:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff93d74a5a3000 Aug 20 22:55:53 cpu-e-1058 kernel: LustreError: 66170:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff93bfc5640a00 Aug 20 22:55:53 cpu-e-1058 kernel: LustreError: 66170:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff93bfc5640a00 Aug 20 22:55:53 cpu-e-1058 kernel: LustreError: 66170:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff93bfc5640a00 Aug 20 22:55:53 cpu-e-1058 kernel: LustreError: 66170:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff93bfc5640a00 Aug 20 22:55:53 cpu-e-1058 kernel: LustreError: 66175:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff93be7af1f000 Aug 20 22:55:53 cpu-e-1058 kernel: Lustre: 66677:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566338151/real 1566338153] req@ffff93d7db44ec00 x1642422175275328/t0(0) o3->fs1-OST004c-osc-ffff93d711340000@10.47.18.7@o2ib1:6/4 lens 488/440 e 0 to 1 dl 1566338195 ref 2 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 20 22:55:53 cpu-e-1058 kernel: LustreError: 66170:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff93d74a5a2c00 Aug 20 22:55:53 cpu-e-1058 kernel: Lustre: fs1-OST010a-osc-ffff93d711340000: Connection to fs1-OST010a (at 10.47.18.23@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 22:55:53 cpu-e-1058 kernel: Lustre: Skipped 1 previous similar message Aug 20 22:55:53 cpu-e-1058 kernel: LustreError: 66170:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff93be6cb0e200 Aug 20 22:55:53 cpu-e-1058 kernel: LustreError: 66170:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff93be6cb0e200 Aug 20 22:55:53 cpu-e-1058 kernel: LustreError: 66170:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff93d702e2fc00 Aug 20 22:55:53 cpu-e-1058 kernel: LustreError: 66170:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff93d702e2fc00 Aug 20 22:55:53 cpu-e-1058 kernel: LustreError: 66170:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff93d702e2fc00 Aug 20 22:55:53 cpu-e-1058 kernel: Lustre: 66673:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566338151/real 1566338151] req@ffff93bfd4b7f080 x1642422175273520/t0(0) o3->fs1-OST0042-osc-ffff93d711340000@10.47.18.6@o2ib1:6/4 lens 488/440 e 0 to 1 dl 1566338195 ref 2 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 20 22:55:53 cpu-e-1058 kernel: Lustre: 66673:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Aug 20 22:55:53 cpu-e-1058 kernel: Lustre: fs1-OST0042-osc-ffff93d711340000: Connection to fs1-OST0042 (at 10.47.18.6@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 22:55:53 cpu-e-1058 kernel: Lustre: Skipped 3 previous similar messages Aug 20 22:55:54 cpu-e-1058 kernel: LustreError: 66170:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff93bdea240200 Aug 20 22:55:54 cpu-e-1058 kernel: Lustre: fs1-OST00b7-osc-ffff93d711340000: Connection restored to 10.47.18.16@o2ib1 (at 10.47.18.16@o2ib1) Aug 20 22:55:54 cpu-e-1058 kernel: LustreError: 66170:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff93bdea240200 Aug 20 22:55:54 cpu-e-1058 kernel: LustreError: 66170:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff93bdea240200 Aug 20 22:55:54 cpu-e-1058 kernel: LustreError: 66170:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff93bdea240200 Aug 20 22:55:54 cpu-e-1058 kernel: Lustre: fs1-OST0118-osc-ffff93d711340000: Connection restored to 10.47.18.24@o2ib1 (at 10.47.18.24@o2ib1) Aug 20 22:55:54 cpu-e-1058 kernel: Lustre: Skipped 6 previous similar messages Aug 20 22:55:55 cpu-e-1058 kernel: Lustre: 66682:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566338153/real 1566338155] req@ffff93d7d90f4800 x1642422175277072/t0(0) o400->fs1-OST00db-osc-ffff93d711340000@10.47.18.19@o2ib1:28/4 lens 224/224 e 0 to 1 dl 1566338160 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Aug 20 22:55:55 cpu-e-1058 kernel: Lustre: 66682:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Aug 20 22:55:55 cpu-e-1058 kernel: Lustre: fs1-OST00db-osc-ffff93d711340000: Connection to fs1-OST00db (at 10.47.18.19@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 22:55:55 cpu-e-1058 kernel: Lustre: Skipped 9 previous similar messages Aug 20 22:55:55 cpu-e-1058 kernel: LustreError: 66177:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff93bf61bece00 Aug 20 22:55:55 cpu-e-1058 kernel: LustreError: 66175:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff93bf61bece00 Aug 20 22:55:55 cpu-e-1058 kernel: LustreError: 66178:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff93bf61bece00 Aug 20 22:55:55 cpu-e-1058 kernel: LustreError: 66176:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff93bf61bece00 Aug 20 22:55:55 cpu-e-1058 kernel: LustreError: 66175:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff93bde9755000 Aug 20 22:55:55 cpu-e-1058 kernel: LustreError: 66176:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff93bde9755000 Aug 20 22:55:55 cpu-e-1058 kernel: LustreError: 66177:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff93bde9755000 Aug 20 22:55:55 cpu-e-1058 kernel: LustreError: 66178:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff93bde9755000 Aug 20 22:55:55 cpu-e-1058 kernel: LustreError: 66170:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff93bfc5640a00 Aug 20 22:55:55 cpu-e-1058 kernel: LustreError: 66170:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff93be7af15e00 Aug 20 22:55:55 cpu-e-1058 kernel: LustreError: 66170:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff93be7af15e00 Aug 20 22:55:55 cpu-e-1058 kernel: LustreError: 66170:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff93be7af15e00 Aug 20 22:55:55 cpu-e-1058 kernel: LustreError: 66170:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff93be7af15e00 Aug 20 22:55:55 cpu-e-1058 kernel: Lustre: fs1-OST00f5-osc-ffff93d711340000: Connection restored to 10.47.18.21@o2ib1 (at 10.47.18.21@o2ib1) Aug 20 22:55:55 cpu-e-1058 kernel: Lustre: Skipped 10 previous similar messages Aug 20 22:55:56 cpu-e-1058 kernel: LustreError: 66170:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff93bde9754c00 Aug 20 22:55:58 cpu-e-1058 kernel: Lustre: 71200:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566338151/real 1566338151] req@ffff93d7da27ba80 x1642422175274336/t0(0) o101->fs1-OST0046-osc-ffff93d711340000@10.47.18.6@o2ib1:28/4 lens 328/400 e 0 to 1 dl 1566338158 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Aug 20 22:55:58 cpu-e-1058 kernel: Lustre: 71200:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 20 previous similar messages Aug 20 22:55:58 cpu-e-1058 kernel: Lustre: fs1-OST0046-osc-ffff93d711340000: Connection to fs1-OST0046 (at 10.47.18.6@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 22:55:58 cpu-e-1058 kernel: Lustre: Skipped 19 previous similar messages Aug 20 22:55:58 cpu-e-1058 kernel: Lustre: fs1-OST0046-osc-ffff93d711340000: Connection restored to 10.47.18.6@o2ib1 (at 10.47.18.6@o2ib1) Aug 20 22:55:58 cpu-e-1058 kernel: Lustre: Skipped 11 previous similar messages Aug 20 22:56:23 cpu-e-1058 kernel: Lustre: fs1-OST0043-osc-ffff93d711340000: Connection restored to 10.47.18.6@o2ib1 (at 10.47.18.6@o2ib1) Aug 20 22:56:23 cpu-e-1058 kernel: Lustre: Skipped 1 previous similar message Aug 20 23:01:14 cpu-e-1058 kernel: Lustre: 66661:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566338151/real 1566338151] req@ffff93bfd4b7cc80 x1642422175274912/t0(0) o3->fs1-OST009d-osc-ffff93d711340000@10.47.18.14@o2ib1:6/4 lens 488/440 e 0 to 1 dl 1566338163 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Aug 20 23:01:14 cpu-e-1058 kernel: Lustre: 66661:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 1 previous similar message Aug 20 23:01:14 cpu-e-1058 kernel: Lustre: fs1-OST009d-osc-ffff93d711340000: Connection to fs1-OST009d (at 10.47.18.14@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 23:01:14 cpu-e-1058 kernel: Lustre: Skipped 1 previous similar message Aug 20 23:01:14 cpu-e-1058 kernel: Lustre: fs1-OST009d-osc-ffff93d711340000: Connection restored to 10.47.18.14@o2ib1 (at 10.47.18.14@o2ib1) Aug 20 23:01:14 cpu-e-1058 kernel: Lustre: Skipped 4 previous similar messages Aug 20 23:01:41 cpu-e-1058 kernel: Lustre: 66690:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566338153/real 1566338153] req@ffff93d7d6175580 x1642422175276416/t0(0) o400->fs1-OST0065-osc-ffff93d711340000@10.47.18.9@o2ib1:28/4 lens 224/224 e 0 to 1 dl 1566338160 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Aug 20 23:01:41 cpu-e-1058 kernel: Lustre: fs1-OST0065-osc-ffff93d711340000: Connection to fs1-OST0065 (at 10.47.18.9@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 23:01:41 cpu-e-1058 kernel: Lustre: fs1-OST0065-osc-ffff93d711340000: Connection restored to 10.47.18.9@o2ib1 (at 10.47.18.9@o2ib1) Aug 20 23:03:07 cpu-e-1058 kernel: Lustre: 66670:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566338151/real 1566338151] req@ffff93bfd4813a80 x1642422175275248/t0(0) o3->fs1-OST0024-osc-ffff93d711340000@10.47.18.4@o2ib1:6/4 lens 488/440 e 1 to 1 dl 1566338220 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Aug 20 23:03:07 cpu-e-1058 kernel: Lustre: 66670:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Aug 20 23:03:07 cpu-e-1058 kernel: Lustre: fs1-OST0024-osc-ffff93d711340000: Connection to fs1-OST0024 (at 10.47.18.4@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 23:03:07 cpu-e-1058 kernel: Lustre: Skipped 6 previous similar messages Aug 20 23:03:07 cpu-e-1058 kernel: Lustre: fs1-OST0024-osc-ffff93d711340000: Connection restored to 10.47.18.4@o2ib1 (at 10.47.18.4@o2ib1) Aug 20 23:03:07 cpu-e-1058 kernel: Lustre: Skipped 6 previous similar messages Aug 20 23:04:03 cpu-e-1058 kernel: Lustre: 66666:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566338151/real 1566338151] req@ffff93d7da27ec00 x1642422175274560/t0(0) o103->fs1-OST009f-osc-ffff93d711340000@10.47.18.14@o2ib1:17/18 lens 328/224 e 0 to 1 dl 1566338158 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Aug 20 23:04:03 cpu-e-1058 kernel: Lustre: 66666:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 1 previous similar message Aug 20 23:04:03 cpu-e-1058 kernel: Lustre: fs1-OST009f-osc-ffff93d711340000: Connection to fs1-OST009f (at 10.47.18.14@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 23:04:03 cpu-e-1058 kernel: Lustre: Skipped 1 previous similar message Aug 20 23:17:29 cpu-e-1058 kernel: Adding 15999996k swap on /dev/sda2. Priority:-2 extents:1 across:15999996k SSFS Aug 20 23:19:07 cpu-e-1058 kernel: Lustre: Unmounted fs1-client HOSTS ------------------------------------------------------------------------- cpu-e-837 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-08-20 19:49:07 BST, end at Wed 2019-08-21 11:46:27 BST. -- Aug 20 23:16:54 cpu-e-837 kernel: Adding 15999996k swap on /dev/sda2. Priority:-2 extents:1 across:15999996k SSFS Aug 20 23:19:07 cpu-e-837 kernel: Lustre: Unmounted fs1-client HOSTS ------------------------------------------------------------------------- cpu-e-1054 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-08-20 19:50:04 BST, end at Wed 2019-08-21 11:46:27 BST. -- Aug 20 22:55:53 cpu-e-1054 kernel: LustreError: 66415:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8bce7e7e4a00 Aug 20 22:55:53 cpu-e-1054 kernel: Lustre: 66931:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566338151/real 1566338153] req@ffff8be815725e80 x1642422175277152/t0(0) o3->fs1-OST0056-osc-ffff8be7b5fe7800@10.47.18.8@o2ib1:6/4 lens 488/440 e 0 to 1 dl 1566338163 ref 2 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 20 22:55:53 cpu-e-1054 kernel: Lustre: fs1-OST0056-osc-ffff8be7b5fe7800: Connection to fs1-OST0056 (at 10.47.18.8@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 22:55:53 cpu-e-1054 kernel: LustreError: 66415:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8bce7e7e4a00 Aug 20 22:55:53 cpu-e-1054 kernel: LustreError: 66415:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8bce88393600 Aug 20 22:55:53 cpu-e-1054 kernel: Lustre: fs1-OST0056-osc-ffff8be7b5fe7800: Connection restored to 10.47.18.8@o2ib1 (at 10.47.18.8@o2ib1) Aug 20 22:55:53 cpu-e-1054 kernel: LustreError: 66415:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8bce88393600 Aug 20 22:55:53 cpu-e-1054 kernel: LustreError: 66415:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8bcfc3241e00 Aug 20 22:55:53 cpu-e-1054 kernel: LustreError: 66415:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8bcfc3241e00 Aug 20 22:55:53 cpu-e-1054 kernel: Lustre: fs1-OST00be-osc-ffff8be7b5fe7800: Connection restored to 10.47.18.16@o2ib1 (at 10.47.18.16@o2ib1) Aug 20 22:55:53 cpu-e-1054 kernel: Lustre: Skipped 5 previous similar messages Aug 20 22:55:54 cpu-e-1054 kernel: Lustre: 66905:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566338151/real 1566338151] req@ffff8bd015016c00 x1642422175276368/t0(0) o3->fs1-OST00f6-osc-ffff8be7b5fe7800@10.47.18.21@o2ib1:6/4 lens 488/440 e 0 to 1 dl 1566338163 ref 2 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 20 22:55:54 cpu-e-1054 kernel: Lustre: 66905:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 11 previous similar messages Aug 20 22:55:54 cpu-e-1054 kernel: Lustre: fs1-OST00f6-osc-ffff8be7b5fe7800: Connection to fs1-OST00f6 (at 10.47.18.21@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 22:55:54 cpu-e-1054 kernel: Lustre: Skipped 11 previous similar messages Aug 20 22:55:55 cpu-e-1054 kernel: Lustre: fs1-OST00a5-osc-ffff8be7b5fe7800: Connection restored to 10.47.18.14@o2ib1 (at 10.47.18.14@o2ib1) Aug 20 22:55:58 cpu-e-1054 kernel: Lustre: 71532:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566338151/real 1566338151] req@ffff8bceb7a07080 x1642422175277440/t0(0) o101->fs1-OST0015-osc-ffff8be7b5fe7800@10.47.18.2@o2ib1:28/4 lens 328/400 e 0 to 1 dl 1566338158 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Aug 20 22:55:58 cpu-e-1054 kernel: Lustre: fs1-OST0015-osc-ffff8be7b5fe7800: Connection to fs1-OST0015 (at 10.47.18.2@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 22:55:58 cpu-e-1054 kernel: Lustre: fs1-OST0015-osc-ffff8be7b5fe7800: Connection restored to 10.47.18.2@o2ib1 (at 10.47.18.2@o2ib1) Aug 20 22:55:58 cpu-e-1054 kernel: Lustre: Skipped 5 previous similar messages Aug 20 23:01:08 cpu-e-1054 kernel: Lustre: 66919:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566338151/real 1566338151] req@ffff8be817001680 x1642422175278304/t0(0) o103->fs1-OST00bd-osc-ffff8be7b5fe7800@10.47.18.16@o2ib1:17/18 lens 328/224 e 0 to 1 dl 1566338163 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Aug 20 23:01:08 cpu-e-1054 kernel: Lustre: fs1-OST00bd-osc-ffff8be7b5fe7800: Connection to fs1-OST00bd (at 10.47.18.16@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 23:01:08 cpu-e-1054 kernel: Lustre: fs1-OST00bd-osc-ffff8be7b5fe7800: Connection restored to 10.47.18.16@o2ib1 (at 10.47.18.16@o2ib1) Aug 20 23:01:29 cpu-e-1054 kernel: Lustre: 66922:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566338151/real 1566338151] req@ffff8be8142e3180 x1642422175277776/t0(0) o3->fs1-OST0018-osc-ffff8be7b5fe7800@10.47.18.3@o2ib1:6/4 lens 488/440 e 1 to 1 dl 1566338220 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Aug 20 23:01:29 cpu-e-1054 kernel: Lustre: 66922:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Aug 20 23:01:29 cpu-e-1054 kernel: Lustre: fs1-OST0018-osc-ffff8be7b5fe7800: Connection to fs1-OST0018 (at 10.47.18.3@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 23:01:29 cpu-e-1054 kernel: Lustre: Skipped 2 previous similar messages Aug 20 23:01:29 cpu-e-1054 kernel: Lustre: fs1-OST0018-osc-ffff8be7b5fe7800: Connection restored to 10.47.18.3@o2ib1 (at 10.47.18.3@o2ib1) Aug 20 23:01:29 cpu-e-1054 kernel: Lustre: Skipped 2 previous similar messages Aug 20 23:02:10 cpu-e-1054 kernel: perf: interrupt took too long (3142 > 3132), lowering kernel.perf_event_max_sample_rate to 63000 Aug 20 23:03:01 cpu-e-1054 kernel: Lustre: 66915:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566338151/real 1566338151] req@ffff8bb88fa51680 x1642422175277536/t0(0) o103->fs1-OST0075-osc-ffff8be7b5fe7800@10.47.18.10@o2ib1:17/18 lens 328/224 e 0 to 1 dl 1566338158 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Aug 20 23:03:01 cpu-e-1054 kernel: Lustre: fs1-OST0075-osc-ffff8be7b5fe7800: Connection to fs1-OST0075 (at 10.47.18.10@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 23:03:01 cpu-e-1054 kernel: Lustre: fs1-OST0075-osc-ffff8be7b5fe7800: Connection restored to 10.47.18.10@o2ib1 (at 10.47.18.10@o2ib1) Aug 20 23:03:24 cpu-e-1054 kernel: Lustre: 66927:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566338153/real 1566338153] req@ffff8be817c5ba80 x1642422175278976/t0(0) o400->fs1-OST0004-osc-ffff8be7b5fe7800@10.47.18.1@o2ib1:28/4 lens 224/224 e 0 to 1 dl 1566338160 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Aug 20 23:03:24 cpu-e-1054 kernel: Lustre: 66927:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 1 previous similar message Aug 20 23:03:24 cpu-e-1054 kernel: Lustre: fs1-OST0004-osc-ffff8be7b5fe7800: Connection to fs1-OST0004 (at 10.47.18.1@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 23:03:24 cpu-e-1054 kernel: Lustre: Skipped 1 previous similar message Aug 20 23:17:28 cpu-e-1054 kernel: Adding 15999996k swap on /dev/sda2. Priority:-2 extents:1 across:15999996k SSFS Aug 20 23:19:07 cpu-e-1054 kernel: Lustre: Unmounted fs1-client HOSTS ------------------------------------------------------------------------- cpu-e-1061 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-08-20 19:49:58 BST, end at Wed 2019-08-21 11:46:27 BST. -- Aug 20 22:50:00 cpu-e-1061 kernel: Adding 15999996k swap on /dev/sda2. Priority:-2 extents:1 across:15999996k SSFS Aug 20 22:55:53 cpu-e-1061 kernel: LustreError: 66205:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff9d4758b16400 Aug 20 22:55:53 cpu-e-1061 kernel: LustreError: 66208:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff9d4758b16400 Aug 20 22:55:53 cpu-e-1061 kernel: LustreError: 66206:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff9d4758b16400 Aug 20 22:55:53 cpu-e-1061 kernel: LustreError: 66207:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff9d4758b16400 Aug 20 22:55:54 cpu-e-1061 kernel: Lustre: 66701:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566338151/real 1566338151] req@ffff9d329977b180 x1642422175275568/t0(0) o3->fs1-OST002b-osc-ffff9d606ce59000@10.47.18.4@o2ib1:6/4 lens 488/440 e 0 to 1 dl 1566338163 ref 2 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 20 22:55:54 cpu-e-1061 kernel: Lustre: fs1-OST002b-osc-ffff9d606ce59000: Connection to fs1-OST002b (at 10.47.18.4@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 22:55:54 cpu-e-1061 kernel: Lustre: fs1-OST002b-osc-ffff9d606ce59000: Connection restored to 10.47.18.4@o2ib1 (at 10.47.18.4@o2ib1) Aug 20 22:55:54 cpu-e-1061 kernel: Lustre: 71146:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566338153/real 1566338154] req@ffff9d6118d37980 x1642422175276160/t0(0) o101->fs1-OST0108-osc-ffff9d606ce59000@10.47.18.23@o2ib1:28/4 lens 328/400 e 0 to 1 dl 1566338160 ref 2 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 20 22:55:54 cpu-e-1061 kernel: Lustre: fs1-OST0108-osc-ffff9d606ce59000: Connection to fs1-OST0108 (at 10.47.18.23@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 22:55:55 cpu-e-1061 kernel: LustreError: 66204:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff9d470aa62600 Aug 20 22:55:55 cpu-e-1061 kernel: LustreError: 66206:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff9d470aa62600 Aug 20 22:55:55 cpu-e-1061 kernel: LustreError: 66204:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff9d470aa62600 Aug 20 22:55:55 cpu-e-1061 kernel: LustreError: 66204:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff9d470aa62600 Aug 20 22:55:55 cpu-e-1061 kernel: Lustre: 66715:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566338155/real 1566338155] req@ffff9d6102fbd100 x1642422175277968/t0(0) o103->fs1-OST00fd-osc-ffff9d606ce59000@10.47.18.22@o2ib1:17/18 lens 328/224 e 0 to 1 dl 1566338167 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 20 22:55:55 cpu-e-1061 kernel: Lustre: 66715:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 1 previous similar message Aug 20 22:55:55 cpu-e-1061 kernel: Lustre: fs1-OST00fc-osc-ffff9d606ce59000: Connection to fs1-OST00fc (at 10.47.18.22@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 22:55:55 cpu-e-1061 kernel: Lustre: Skipped 1 previous similar message Aug 20 22:55:55 cpu-e-1061 kernel: LustreError: 66204:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff9d60a2e9c000 Aug 20 22:55:56 cpu-e-1061 kernel: LustreError: 66204:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff9d60a2e9c000 Aug 20 22:55:56 cpu-e-1061 kernel: Lustre: fs1-OST00fc-osc-ffff9d606ce59000: Connection restored to 10.47.18.22@o2ib1 (at 10.47.18.22@o2ib1) Aug 20 22:55:56 cpu-e-1061 kernel: LustreError: 66204:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff9d60a2e9c000 Aug 20 22:55:56 cpu-e-1061 kernel: LustreError: 66204:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff9d60a2e9c000 Aug 20 22:55:56 cpu-e-1061 kernel: LustreError: 66204:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff9d470aa4b600 Aug 20 22:55:57 cpu-e-1061 kernel: Lustre: fs1-OST0062-osc-ffff9d606ce59000: Connection restored to 10.47.18.9@o2ib1 (at 10.47.18.9@o2ib1) Aug 20 22:55:57 cpu-e-1061 kernel: Lustre: Skipped 6 previous similar messages Aug 20 23:00:48 cpu-e-1061 kernel: Lustre: 66702:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566338156/real 1566338156] req@ffff9d6115572880 x1642422175292064/t0(0) o3->fs1-OST0068-osc-ffff9d606ce59000@10.47.18.9@o2ib1:6/4 lens 488/440 e 1 to 1 dl 1566338225 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Aug 20 23:00:48 cpu-e-1061 kernel: Lustre: 66702:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Aug 20 23:00:48 cpu-e-1061 kernel: Lustre: fs1-OST0068-osc-ffff9d606ce59000: Connection to fs1-OST0068 (at 10.47.18.9@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 23:00:48 cpu-e-1061 kernel: Lustre: Skipped 5 previous similar messages Aug 20 23:00:48 cpu-e-1061 kernel: Lustre: fs1-OST0068-osc-ffff9d606ce59000: Connection restored to 10.47.18.9@o2ib1 (at 10.47.18.9@o2ib1) Aug 20 23:01:10 cpu-e-1061 kernel: Lustre: 66724:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566338156/real 1566338156] req@ffff9d60a3b1da00 x1642422175279168/t0(0) o3->fs1-OST0110-osc-ffff9d606ce59000@10.47.18.23@o2ib1:6/4 lens 488/440 e 2 to 1 dl 1566338220 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Aug 20 23:01:10 cpu-e-1061 kernel: Lustre: 66724:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 1 previous similar message Aug 20 23:01:10 cpu-e-1061 kernel: Lustre: fs1-OST0110-osc-ffff9d606ce59000: Connection to fs1-OST0110 (at 10.47.18.23@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 23:01:10 cpu-e-1061 kernel: Lustre: Skipped 1 previous similar message Aug 20 23:01:10 cpu-e-1061 kernel: Lustre: fs1-OST0110-osc-ffff9d606ce59000: Connection restored to 10.47.18.23@o2ib1 (at 10.47.18.23@o2ib1) Aug 20 23:01:10 cpu-e-1061 kernel: Lustre: Skipped 1 previous similar message Aug 20 23:03:07 cpu-e-1061 kernel: Lustre: 66712:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566338151/real 1566338151] req@ffff9d61155f1f80 x1642422175275600/t0(0) o3->fs1-OST0069-osc-ffff9d606ce59000@10.47.18.9@o2ib1:6/4 lens 488/440 e 1 to 1 dl 1566338220 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Aug 20 23:03:07 cpu-e-1061 kernel: Lustre: fs1-OST0069-osc-ffff9d606ce59000: Connection to fs1-OST0069 (at 10.47.18.9@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 23:03:07 cpu-e-1061 kernel: Lustre: fs1-OST0069-osc-ffff9d606ce59000: Connection restored to 10.47.18.9@o2ib1 (at 10.47.18.9@o2ib1) Aug 20 23:04:32 cpu-e-1061 kernel: perf: interrupt took too long (3138 > 3137), lowering kernel.perf_event_max_sample_rate to 63000 Aug 20 23:17:29 cpu-e-1061 kernel: Adding 15999996k swap on /dev/sda2. Priority:-2 extents:1 across:15999996k SSFS Aug 20 23:19:07 cpu-e-1061 kernel: Lustre: Unmounted fs1-client HOSTS ------------------------------------------------------------------------- cpu-e-1057 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-08-20 19:50:20 BST, end at Wed 2019-08-21 11:46:27 BST. -- Aug 20 22:50:00 cpu-e-1057 kernel: Adding 15999996k swap on /dev/sda2. Priority:-2 extents:1 across:15999996k SSFS Aug 20 22:55:53 cpu-e-1057 kernel: Lustre: 71132:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566338151/real 1566338153] req@ffff8889ae38e300 x1642422175275312/t0(0) o101->fs1-OST0119-osc-ffff8889e1c27800@10.47.18.24@o2ib1:28/4 lens 328/400 e 0 to 1 dl 1566338158 ref 2 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 20 22:55:53 cpu-e-1057 kernel: Lustre: 71132:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Aug 20 22:55:53 cpu-e-1057 kernel: Lustre: fs1-OST0119-osc-ffff8889e1c27800: Connection to fs1-OST0119 (at 10.47.18.24@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 22:55:53 cpu-e-1057 kernel: Lustre: Skipped 2 previous similar messages Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff887052f1aa00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff887052f1aa00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff887052f1aa00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66177:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff88896e228e00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66176:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff88896e228e00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66175:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff8889ff272800 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66174:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff8889ff272800 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66176:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff88896e228e00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66177:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff88896e228e00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66174:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff88896e228e00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66174:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff88896e228e00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66177:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff88896e228e00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66176:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff88896e228e00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66176:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff8889ff272800 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66176:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff88896e228e00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66176:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff88896e228e00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66176:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff8889ff272800 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66176:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff88896e228e00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66176:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff88896e228e00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66179:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff888a09121c00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66174:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff88896e228e00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66175:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff88896e228e00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8889ff272800 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8889ff272800 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8889ff272800 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8889ff272800 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff88896e228e00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff88896e228e00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888a09121c00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888a09121c00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888a09121c00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888a09121c00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888a09121c00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888a09121c00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888a09121c00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888a09121c00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888a09121c00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888a09121c00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888a09121c00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888a09121c00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888a09121c00 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff88705fe76600 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff88705fe76600 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8870866fe400 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8870866fe400 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8870866fe400 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8870866fe400 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8870866fe400 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8870866fe400 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8870866fe400 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff88705fe76600 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff88705fe76600 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff88705e6cc000 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff88705e6cc000 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff88705e6cb600 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff88705e6cb600 Aug 20 22:55:53 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff88705e6cb600 Aug 20 22:55:54 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff88705e6cc000 Aug 20 22:55:54 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff88705e6cc000 Aug 20 22:55:54 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff88705e6cc000 Aug 20 22:55:54 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff88705e6cc000 Aug 20 22:55:54 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888a09125600 Aug 20 22:55:54 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8889b224c400 Aug 20 22:55:55 cpu-e-1057 kernel: Lustre: fs1-OST0066-osc-ffff8889e1c27800: Connection to fs1-OST0066 (at 10.47.18.9@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 22:55:55 cpu-e-1057 kernel: Lustre: Skipped 12 previous similar messages Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8870866fe400 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8870866fe400 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8870866fe400 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff88705eb8e800 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff88705eb8e800 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8870866fe400 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8870866fe400 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8870866fe400 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8870866fe400 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff88705e6cb600 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888942791a00 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888942791a00 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888942791a00 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888942791a00 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff88705e6cc000 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff88705e6cc000 Aug 20 22:55:55 cpu-e-1057 kernel: Lustre: 66669:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566338151/real 1566338151] req@ffff887296a16c00 x1642422175274752/t0(0) o3->fs1-OST0100-osc-ffff8889e1c27800@10.47.18.22@o2ib1:6/4 lens 488/440 e 0 to 1 dl 1566338163 ref 2 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 20 22:55:55 cpu-e-1057 kernel: Lustre: 66669:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 79 previous similar messages Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8889b224c400 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8889b224c400 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff887052f1aa00 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff887052f1aa00 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff887052f1aa00 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff887052f65200 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff887052f65200 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff887052f65200 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff887052f65200 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff887052f65200 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff887052f65200 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff887052f65200 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff887052f65200 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff887052f65200 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff887052f65200 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff887052f65200 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff887052f65200 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff887052f65200 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff88705fe9a400 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff887052f1aa00 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff887052f1aa00 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff887052f1aa00 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff887052f1aa00 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff887052f1aa00 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff88705fe9a400 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff88705eb8e800 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff88705eb8e800 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff88705fe76600 Aug 20 22:55:55 cpu-e-1057 kernel: Lustre: fs1-OST00be-osc-ffff8889e1c27800: Connection restored to 10.47.18.16@o2ib1 (at 10.47.18.16@o2ib1) Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff88705fe76600 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8889ff272800 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8889ff272800 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8889ff272800 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8889ff272800 Aug 20 22:55:55 cpu-e-1057 kernel: Lustre: Skipped 2 previous similar messages Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8889ff272800 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8889ff272800 Aug 20 22:55:55 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8889ff272800 Aug 20 22:55:56 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888977774000 Aug 20 22:55:56 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888977774000 Aug 20 22:55:56 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888977774000 Aug 20 22:55:56 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888977774000 Aug 20 22:55:56 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888977774000 Aug 20 22:55:56 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888977774000 Aug 20 22:55:56 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888977774000 Aug 20 22:55:56 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888977774000 Aug 20 22:55:56 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888977774000 Aug 20 22:55:56 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888977774000 Aug 20 22:55:56 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888977774000 Aug 20 22:55:56 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888977774000 Aug 20 22:55:56 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888977774000 Aug 20 22:55:56 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888977774000 Aug 20 22:55:56 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888977774000 Aug 20 22:55:56 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888977774000 Aug 20 22:55:57 cpu-e-1057 kernel: LNet: 66173:0:(o2iblnd_cb.c:3381:kiblnd_check_conns()) Timed out tx for 10.47.18.44@o2ib1: 0 seconds Aug 20 22:55:57 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888a23398e00 Aug 20 22:55:57 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888a23398e00 Aug 20 22:55:57 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8889b224c400 Aug 20 22:55:57 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff88705fe9a400 Aug 20 22:55:57 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8870866fea00 Aug 20 22:55:57 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8870866fea00 Aug 20 22:55:57 cpu-e-1057 kernel: Lustre: fs1-OST0067-osc-ffff8889e1c27800: Connection to fs1-OST0067 (at 10.47.18.9@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 22:55:57 cpu-e-1057 kernel: Lustre: Skipped 81 previous similar messages Aug 20 22:55:57 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888a23398e00 Aug 20 22:55:57 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888a23398e00 Aug 20 22:55:57 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888a23398e00 Aug 20 22:55:57 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888a09122c00 Aug 20 22:55:57 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888a09122c00 Aug 20 22:55:57 cpu-e-1057 kernel: Lustre: fs1-OST00b4-osc-ffff8889e1c27800: Connection restored to 10.47.18.16@o2ib1 (at 10.47.18.16@o2ib1) Aug 20 22:55:57 cpu-e-1057 kernel: Lustre: Skipped 30 previous similar messages Aug 20 22:55:58 cpu-e-1057 kernel: LNet: 66173:0:(o2iblnd_cb.c:3381:kiblnd_check_conns()) Timed out tx for 10.47.18.38@o2ib1: 1 seconds Aug 20 22:55:58 cpu-e-1057 kernel: LNet: 66173:0:(o2iblnd_cb.c:3381:kiblnd_check_conns()) Skipped 9 previous similar messages Aug 20 22:55:58 cpu-e-1057 kernel: LNetError: 71127:0:(o2iblnd_cb.c:1434:kiblnd_connect_peer()) Can't resolve addr for 10.47.18.3@o2ib1: -13 Aug 20 22:56:00 cpu-e-1057 kernel: Lustre: 71147:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566338156/real 1566338160] req@ffff8870f277f980 x1642422175279248/t0(0) o101->fs1-OST0083-osc-ffff8889e1c27800@10.47.18.11@o2ib1:28/4 lens 328/400 e 0 to 1 dl 1566338163 ref 2 fl Rpc:eXS/0/ffffffff rc -11/-1 Aug 20 22:56:00 cpu-e-1057 kernel: Lustre: 71147:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 233 previous similar messages Aug 20 22:56:00 cpu-e-1057 kernel: LNetError: 71147:0:(o2iblnd_cb.c:1434:kiblnd_connect_peer()) Can't resolve addr for 10.47.18.11@o2ib1: -13 Aug 20 22:56:00 cpu-e-1057 kernel: LNetError: 71147:0:(o2iblnd_cb.c:1434:kiblnd_connect_peer()) Skipped 823 previous similar messages Aug 20 22:56:16 cpu-e-1057 kernel: Lustre: 66669:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566338152/real 1566338152] req@ffff88723d7c3600 x1642422175275408/t0(0) o103->fs1-OST010e-osc-ffff8889e1c27800@10.47.18.23@o2ib1:17/18 lens 328/224 e 0 to 1 dl 1566338159 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Aug 20 22:56:16 cpu-e-1057 kernel: Lustre: 66669:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Aug 20 22:56:16 cpu-e-1057 kernel: Lustre: fs1-OST010e-osc-ffff8889e1c27800: Connection to fs1-OST010e (at 10.47.18.23@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 22:56:16 cpu-e-1057 kernel: Lustre: Skipped 223 previous similar messages Aug 20 22:56:16 cpu-e-1057 kernel: Lustre: fs1-OST002e-osc-ffff8889e1c27800: Connection restored to 10.47.18.4@o2ib1 (at 10.47.18.4@o2ib1) Aug 20 22:56:16 cpu-e-1057 kernel: Lustre: Skipped 220 previous similar messages Aug 20 22:56:41 cpu-e-1057 kernel: Lustre: fs1-OST0062-osc-ffff8889e1c27800: Connection restored to 10.47.18.9@o2ib1 (at 10.47.18.9@o2ib1) Aug 20 22:56:41 cpu-e-1057 kernel: Lustre: Skipped 63 previous similar messages Aug 20 23:01:37 cpu-e-1057 kernel: Lustre: 66666:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566338156/real 1566338156] req@ffff887299aee300 x1642422175279200/t0(0) o3->fs1-OST0072-osc-ffff8889e1c27800@10.47.18.10@o2ib1:6/4 lens 488/440 e 1 to 1 dl 1566338225 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Aug 20 23:01:37 cpu-e-1057 kernel: Lustre: fs1-OST0072-osc-ffff8889e1c27800: Connection to fs1-OST0072 (at 10.47.18.10@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 23:01:37 cpu-e-1057 kernel: Lustre: fs1-OST0072-osc-ffff8889e1c27800: Connection restored to 10.47.18.10@o2ib1 (at 10.47.18.10@o2ib1) Aug 20 23:01:37 cpu-e-1057 kernel: Lustre: Skipped 3 previous similar messages Aug 20 23:01:59 cpu-e-1057 kernel: Lustre: fs1-OST00ae-osc-ffff8889e1c27800: Connection to fs1-OST00ae (at 10.47.18.15@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 23:01:59 cpu-e-1057 kernel: Lustre: Skipped 1 previous similar message Aug 20 23:03:33 cpu-e-1057 kernel: Lustre: 66679:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566338156/real 1566338156] req@ffff8870f277a880 x1642422175279040/t0(0) o3->fs1-OST00ca-osc-ffff8889e1c27800@10.47.18.17@o2ib1:6/4 lens 488/440 e 3 to 1 dl 1566338239 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Aug 20 23:03:33 cpu-e-1057 kernel: Lustre: 66679:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Aug 20 23:03:33 cpu-e-1057 kernel: Lustre: fs1-OST00ca-osc-ffff8889e1c27800: Connection to fs1-OST00ca (at 10.47.18.17@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 23:03:33 cpu-e-1057 kernel: Lustre: fs1-OST0067-osc-ffff8889e1c27800: Connection restored to 10.47.18.9@o2ib1 (at 10.47.18.9@o2ib1) Aug 20 23:03:33 cpu-e-1057 kernel: Lustre: Skipped 2 previous similar messages Aug 20 23:06:06 cpu-e-1057 kernel: Lustre: 66693:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566338156/real 1566338156] req@ffff8889ae2e3f00 x1642422175279344/t0(0) o3->fs1-OST005f-osc-ffff8889e1c27800@10.47.18.8@o2ib1:6/4 lens 488/440 e 2 to 1 dl 1566338218 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Aug 20 23:06:06 cpu-e-1057 kernel: Lustre: 66693:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Aug 20 23:06:06 cpu-e-1057 kernel: Lustre: fs1-OST005f-osc-ffff8889e1c27800: Connection to fs1-OST005f (at 10.47.18.8@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 23:06:06 cpu-e-1057 kernel: Lustre: Skipped 6 previous similar messages Aug 20 23:06:06 cpu-e-1057 kernel: Lustre: fs1-OST005f-osc-ffff8889e1c27800: Connection restored to 10.47.18.8@o2ib1 (at 10.47.18.8@o2ib1) Aug 20 23:06:06 cpu-e-1057 kernel: Lustre: Skipped 6 previous similar messages Aug 20 23:17:29 cpu-e-1057 kernel: Adding 15999996k swap on /dev/sda2. Priority:-2 extents:1 across:15999996k SSFS Aug 20 23:19:07 cpu-e-1057 kernel: Lustre: Unmounted fs1-client HOSTS ------------------------------------------------------------------------- cpu-e-1059 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-08-20 19:50:03 BST, end at Wed 2019-08-21 11:46:27 BST. -- Aug 20 22:50:00 cpu-e-1059 kernel: Adding 15999996k swap on /dev/sda2. Priority:-2 extents:1 across:15999996k SSFS Aug 20 22:55:53 cpu-e-1059 kernel: Lustre: 71120:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566338151/real 1566338153] req@ffff905d53f94800 x1642422175275744/t0(0) o101->fs1-OST005e-osc-ffff905d4daed000@10.47.18.8@o2ib1:28/4 lens 328/400 e 0 to 1 dl 1566338158 ref 2 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 20 22:55:53 cpu-e-1059 kernel: Lustre: fs1-OST005e-osc-ffff905d4daed000: Connection to fs1-OST005e (at 10.47.18.8@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 22:55:53 cpu-e-1059 kernel: LustreError: 66144:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff9043a422dc00 Aug 20 22:55:53 cpu-e-1059 kernel: LustreError: 66150:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff904369708600 Aug 20 22:55:53 cpu-e-1059 kernel: LustreError: 66152:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff904369708600 Aug 20 22:55:53 cpu-e-1059 kernel: LustreError: 66149:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff904369708600 Aug 20 22:55:53 cpu-e-1059 kernel: LustreError: 66151:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff904369708600 Aug 20 22:55:53 cpu-e-1059 kernel: LustreError: 66144:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff905beb72a800 Aug 20 22:55:53 cpu-e-1059 kernel: Lustre: 71104:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566338152/real 1566338153] req@ffff905c95af2880 x1642422175276096/t0(0) o101->fs1-OST008f-osc-ffff905d4daed000@10.47.18.12@o2ib1:28/4 lens 328/400 e 0 to 1 dl 1566338159 ref 2 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 20 22:55:53 cpu-e-1059 kernel: Lustre: 71104:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Aug 20 22:55:53 cpu-e-1059 kernel: Lustre: fs1-OST008f-osc-ffff905d4daed000: Connection to fs1-OST008f (at 10.47.18.12@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 22:55:53 cpu-e-1059 kernel: Lustre: Skipped 3 previous similar messages Aug 20 22:55:53 cpu-e-1059 kernel: LustreError: 66144:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff904378653800 Aug 20 22:55:53 cpu-e-1059 kernel: LustreError: 66144:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff904378653800 Aug 20 22:55:53 cpu-e-1059 kernel: LustreError: 66144:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff904378653800 Aug 20 22:55:54 cpu-e-1059 kernel: Lustre: fs1-OST011a-osc-ffff905d4daed000: Connection restored to 10.47.18.24@o2ib1 (at 10.47.18.24@o2ib1) Aug 20 22:55:54 cpu-e-1059 kernel: LustreError: 66144:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff905beba11800 Aug 20 22:55:54 cpu-e-1059 kernel: Lustre: 66634:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566338153/real 1566338154] req@ffff905d53f97980 x1642422175276256/t0(0) o400->fs1-MDT0006-mdc-ffff905d4daed000@10.47.18.7@o2ib1:12/10 lens 224/224 e 0 to 1 dl 1566338160 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Aug 20 22:55:54 cpu-e-1059 kernel: Lustre: 66634:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 1 previous similar message Aug 20 22:55:54 cpu-e-1059 kernel: Lustre: fs1-MDT0006-mdc-ffff905d4daed000: Connection to fs1-MDT0006 (at 10.47.18.7@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 22:55:54 cpu-e-1059 kernel: Lustre: Skipped 1 previous similar message Aug 20 22:55:55 cpu-e-1059 kernel: LustreError: 66144:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff905beba11800 Aug 20 22:55:55 cpu-e-1059 kernel: LustreError: 66144:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff904379e09000 Aug 20 22:55:55 cpu-e-1059 kernel: LustreError: 66144:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff904379e09000 Aug 20 22:55:55 cpu-e-1059 kernel: LustreError: 66144:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff904379e09000 Aug 20 22:55:55 cpu-e-1059 kernel: LustreError: 66144:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff904379e09000 Aug 20 22:55:56 cpu-e-1059 kernel: Lustre: fs1-MDT0006-mdc-ffff905d4daed000: Connection restored to 10.47.18.7@o2ib1 (at 10.47.18.7@o2ib1) Aug 20 22:55:56 cpu-e-1059 kernel: LustreError: 66144:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff905beb72a800 Aug 20 22:56:21 cpu-e-1059 kernel: Lustre: fs1-OST005e-osc-ffff905d4daed000: Connection restored to 10.47.18.8@o2ib1 (at 10.47.18.8@o2ib1) Aug 20 22:56:21 cpu-e-1059 kernel: Lustre: Skipped 6 previous similar messages Aug 20 23:01:11 cpu-e-1059 kernel: Lustre: 66650:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566338156/real 1566338156] req@ffff9045576b8900 x1642422175283520/t0(0) o3->fs1-OST00dd-osc-ffff905d4daed000@10.47.18.19@o2ib1:6/4 lens 488/440 e 1 to 1 dl 1566338227 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Aug 20 23:01:11 cpu-e-1059 kernel: Lustre: 66650:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Aug 20 23:01:11 cpu-e-1059 kernel: Lustre: fs1-OST00dd-osc-ffff905d4daed000: Connection to fs1-OST00dd (at 10.47.18.19@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 23:01:11 cpu-e-1059 kernel: Lustre: Skipped 3 previous similar messages Aug 20 23:01:11 cpu-e-1059 kernel: Lustre: fs1-OST00dd-osc-ffff905d4daed000: Connection restored to 10.47.18.19@o2ib1 (at 10.47.18.19@o2ib1) Aug 20 23:01:11 cpu-e-1059 kernel: Lustre: Skipped 1 previous similar message Aug 20 23:01:25 cpu-e-1059 kernel: Lustre: 66665:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566338156/real 1566338156] req@ffff905d57852d00 x1642422175279952/t0(0) o103->fs1-OST00df-osc-ffff905d4daed000@10.47.18.19@o2ib1:17/18 lens 328/224 e 0 to 1 dl 1566338165 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Aug 20 23:01:25 cpu-e-1059 kernel: Lustre: fs1-OST00df-osc-ffff905d4daed000: Connection to fs1-OST00df (at 10.47.18.19@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 23:01:25 cpu-e-1059 kernel: Lustre: fs1-OST00df-osc-ffff905d4daed000: Connection restored to 10.47.18.19@o2ib1 (at 10.47.18.19@o2ib1) Aug 20 23:01:31 cpu-e-1059 kernel: perf: interrupt took too long (3155 > 3145), lowering kernel.perf_event_max_sample_rate to 63000 Aug 20 23:17:29 cpu-e-1059 kernel: Adding 15999996k swap on /dev/sda2. Priority:-2 extents:1 across:15999996k SSFS Aug 20 23:19:07 cpu-e-1059 kernel: Lustre: Unmounted fs1-client HOSTS ------------------------------------------------------------------------- cpu-e-1060 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-08-20 19:49:46 BST, end at Wed 2019-08-21 11:46:27 BST. -- Aug 20 22:50:00 cpu-e-1060 kernel: Adding 15999996k swap on /dev/sda2. Priority:-2 extents:1 across:15999996k SSFS Aug 20 22:55:53 cpu-e-1060 kernel: LustreError: 66218:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff884f84688600 Aug 20 22:55:56 cpu-e-1060 kernel: Lustre: 66730:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566338151/real 1566338151] req@ffff8838dfbb0900 x1642422175276912/t0(0) o3->fs1-OST0093-osc-ffff8838d7b49000@10.47.18.13@o2ib1:6/4 lens 488/440 e 0 to 1 dl 1566338195 ref 2 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 20 22:55:56 cpu-e-1060 kernel: Lustre: fs1-OST0093-osc-ffff8838d7b49000: Connection to fs1-OST0093 (at 10.47.18.13@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 22:55:57 cpu-e-1060 kernel: LNetError: 66211:0:(o2iblnd_cb.c:3335:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds Aug 20 22:55:57 cpu-e-1060 kernel: LNetError: 66211:0:(o2iblnd_cb.c:3410:kiblnd_check_conns()) Timed out RDMA with 10.47.18.18@o2ib1 (1): c: 6, oc: 0, rc: 16 Aug 20 22:55:57 cpu-e-1060 kernel: Lustre: 66706:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566338155/real 1566338157] req@ffff8838da04a880 x1642422175278752/t0(0) o400->fs1-OST0096-osc-ffff8838d7b49000@10.47.18.13@o2ib1:28/4 lens 224/224 e 0 to 1 dl 1566338162 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1 Aug 20 22:55:57 cpu-e-1060 kernel: Lustre: fs1-OST0096-osc-ffff8838d7b49000: Connection to fs1-OST0096 (at 10.47.18.13@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 22:55:57 cpu-e-1060 kernel: Lustre: Skipped 1 previous similar message Aug 20 22:55:57 cpu-e-1060 kernel: LustreError: 66211:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8838ba1fb800 Aug 20 22:55:57 cpu-e-1060 kernel: Lustre: fs1-OST00d1-osc-ffff8838d7b49000: Connection restored to 10.47.18.18@o2ib1 (at 10.47.18.18@o2ib1) Aug 20 22:56:22 cpu-e-1060 kernel: Lustre: fs1-OST0093-osc-ffff8838d7b49000: Connection restored to 10.47.18.13@o2ib1 (at 10.47.18.13@o2ib1) Aug 20 22:56:22 cpu-e-1060 kernel: Lustre: Skipped 3 previous similar messages Aug 20 23:00:46 cpu-e-1060 kernel: Lustre: 66705:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566338156/real 1566338156] req@ffff8838da901b00 x1642422175280784/t0(0) o3->fs1-OST00cc-osc-ffff8838d7b49000@10.47.18.18@o2ib1:6/4 lens 488/440 e 1 to 1 dl 1566338225 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Aug 20 23:00:46 cpu-e-1060 kernel: Lustre: 66705:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Aug 20 23:00:46 cpu-e-1060 kernel: Lustre: fs1-OST00cc-osc-ffff8838d7b49000: Connection to fs1-OST00cc (at 10.47.18.18@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 23:00:46 cpu-e-1060 kernel: Lustre: Skipped 2 previous similar messages Aug 20 23:00:46 cpu-e-1060 kernel: Lustre: fs1-OST00cc-osc-ffff8838d7b49000: Connection restored to 10.47.18.18@o2ib1 (at 10.47.18.18@o2ib1) Aug 20 23:02:54 cpu-e-1060 kernel: Lustre: 66713:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566338151/real 1566338151] req@ffff883754b03600 x1642422175276688/t0(0) o3->fs1-OST00d0-osc-ffff8838d7b49000@10.47.18.18@o2ib1:6/4 lens 488/440 e 1 to 1 dl 1566338220 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Aug 20 23:02:54 cpu-e-1060 kernel: Lustre: 66713:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Aug 20 23:02:54 cpu-e-1060 kernel: Lustre: fs1-OST00d0-osc-ffff8838d7b49000: Connection to fs1-OST00d0 (at 10.47.18.18@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 23:02:54 cpu-e-1060 kernel: Lustre: Skipped 2 previous similar messages Aug 20 23:02:54 cpu-e-1060 kernel: Lustre: fs1-OST00d0-osc-ffff8838d7b49000: Connection restored to 10.47.18.18@o2ib1 (at 10.47.18.18@o2ib1) Aug 20 23:02:54 cpu-e-1060 kernel: Lustre: Skipped 2 previous similar messages Aug 20 23:03:43 cpu-e-1060 kernel: perf: interrupt took too long (3155 > 3148), lowering kernel.perf_event_max_sample_rate to 63000 Aug 20 23:17:29 cpu-e-1060 kernel: Adding 15999996k swap on /dev/sda2. Priority:-2 extents:1 across:15999996k SSFS Aug 20 23:19:07 cpu-e-1060 kernel: Lustre: Unmounted fs1-client