HOSTS ------------------------------------------------------------------------- cpu-e-1061 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-08-20 19:49:58 BST, end at Wed 2019-08-21 11:30:06 BST. -- Aug 20 22:31:13 cpu-e-1061 kernel: perf: interrupt took too long (2510 > 2500), lowering kernel.perf_event_max_sample_rate to 79000 HOSTS ------------------------------------------------------------------------- cpu-e-1055 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-08-20 19:50:34 BST, end at Wed 2019-08-21 11:30:06 BST. -- HOSTS ------------------------------------------------------------------------- cpu-e-1059 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-08-20 19:50:03 BST, end at Wed 2019-08-21 11:30:06 BST. -- Aug 20 22:33:14 cpu-e-1059 kernel: perf: interrupt took too long (2516 > 2500), lowering kernel.perf_event_max_sample_rate to 79000 HOSTS ------------------------------------------------------------------------- cpu-e-1054 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-08-20 19:50:04 BST, end at Wed 2019-08-21 11:30:06 BST. -- HOSTS ------------------------------------------------------------------------- cpu-e-1057 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-08-20 19:50:20 BST, end at Wed 2019-08-21 11:30:06 BST. -- Aug 20 22:27:04 cpu-e-1057 kernel: Lustre: 66688:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566336422/real 1566336424] req@ffff888a98e59200 x1642422159069536/t0(0) o3->fs1-OST00a4-osc-ffff8889e1c27800@10.47.18.14@o2ib1:6/4 lens 488/440 e 0 to 1 dl 1566336429 ref 2 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 20 22:27:04 cpu-e-1057 kernel: Lustre: fs1-OST00a4-osc-ffff8889e1c27800: Connection to fs1-OST00a4 (at 10.47.18.14@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 22:27:04 cpu-e-1057 kernel: LustreError: 66174:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff887244649400 Aug 20 22:27:04 cpu-e-1057 kernel: LustreError: 66175:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff887244649400 Aug 20 22:27:04 cpu-e-1057 kernel: LustreError: 66177:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff887289f50800 Aug 20 22:27:04 cpu-e-1057 kernel: LustreError: 66179:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff887289f50c00 Aug 20 22:27:04 cpu-e-1057 kernel: LustreError: 66181:0:(events.c:200:client_bulk_callback()) event type 2, status -5, desc ffff887289f50c00 Aug 20 22:27:04 cpu-e-1057 kernel: Lustre: fs1-OST008f-osc-ffff8889e1c27800: Connection restored to 10.47.18.12@o2ib1 (at 10.47.18.12@o2ib1) Aug 20 22:27:04 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8889e5bf4000 Aug 20 22:27:04 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888a16619000 Aug 20 22:27:04 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888a16619000 Aug 20 22:27:04 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff888a16619000 Aug 20 22:27:04 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff887289f50800 Aug 20 22:27:04 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8889e5bf4000 Aug 20 22:27:04 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff8889e5bf4000 Aug 20 22:27:04 cpu-e-1057 kernel: Lustre: 66684:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566336422/real 1566336422] req@ffff88729a772d00 x1642422159068832/t0(0) o3->fs1-OST0110-osc-ffff8889e1c27800@10.47.18.23@o2ib1:6/4 lens 488/440 e 0 to 1 dl 1566336466 ref 2 fl Rpc:eXS/0/ffffffff rc -11/-1 Aug 20 22:27:04 cpu-e-1057 kernel: Lustre: 66684:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Aug 20 22:27:04 cpu-e-1057 kernel: LustreError: 66173:0:(events.c:200:client_bulk_callback()) event type 2, status -103, desc ffff887289f50800 Aug 20 22:27:29 cpu-e-1057 kernel: Lustre: fs1-OST0048-osc-ffff8889e1c27800: Connection restored to 10.47.18.7@o2ib1 (at 10.47.18.7@o2ib1) Aug 20 22:27:29 cpu-e-1057 kernel: Lustre: Skipped 6 previous similar messages Aug 20 22:35:20 cpu-e-1057 kernel: perf: interrupt took too long (2504 > 2500), lowering kernel.perf_event_max_sample_rate to 79000 Aug 20 22:36:51 cpu-e-1057 kernel: Lustre: 66673:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566336422/real 1566336422] req@ffff8870f68a1200 x1642422159069408/t0(0) o3->fs1-OST007d-osc-ffff8889e1c27800@10.47.18.11@o2ib1:6/4 lens 488/440 e 1 to 1 dl 1566336491 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Aug 20 22:36:51 cpu-e-1057 kernel: Lustre: fs1-OST007d-osc-ffff8889e1c27800: Connection to fs1-OST007d (at 10.47.18.11@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 22:36:51 cpu-e-1057 kernel: Lustre: Skipped 7 previous similar messages Aug 20 22:36:51 cpu-e-1057 kernel: Lustre: fs1-OST007d-osc-ffff8889e1c27800: Connection restored to 10.47.18.11@o2ib1 (at 10.47.18.11@o2ib1) Aug 20 22:37:04 cpu-e-1057 kernel: Lustre: 66687:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566336424/real 1566336424] req@ffff888a96207500 x1642422159070288/t0(0) o3->fs1-OST0055-osc-ffff8889e1c27800@10.47.18.8@o2ib1:6/4 lens 488/440 e 1 to 1 dl 1566336493 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Aug 20 22:37:04 cpu-e-1057 kernel: Lustre: 66687:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 1 previous similar message Aug 20 22:37:04 cpu-e-1057 kernel: Lustre: fs1-OST0055-osc-ffff8889e1c27800: Connection to fs1-OST0055 (at 10.47.18.8@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 22:37:04 cpu-e-1057 kernel: Lustre: Skipped 1 previous similar message Aug 20 22:37:04 cpu-e-1057 kernel: Lustre: fs1-OST0055-osc-ffff8889e1c27800: Connection restored to 10.47.18.8@o2ib1 (at 10.47.18.8@o2ib1) Aug 20 22:37:04 cpu-e-1057 kernel: Lustre: Skipped 1 previous similar message Aug 20 22:38:12 cpu-e-1057 kernel: Lustre: 66682:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566336424/real 1566336424] req@ffff888a8feeba80 x1642422159070096/t0(0) o400->fs1-MDT0016-mdc-ffff8889e1c27800@10.47.18.23@o2ib1:12/10 lens 224/224 e 0 to 1 dl 1566336431 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 Aug 20 22:38:12 cpu-e-1057 kernel: Lustre: fs1-MDT0016-mdc-ffff8889e1c27800: Connection to fs1-MDT0016 (at 10.47.18.23@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 22:38:12 cpu-e-1057 kernel: Lustre: fs1-MDT0016-mdc-ffff8889e1c27800: Connection restored to 10.47.18.23@o2ib1 (at 10.47.18.23@o2ib1) Aug 20 22:40:37 cpu-e-1057 kernel: Lustre: 66676:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566336422/real 1566336422] req@ffff887293238900 x1642422159069504/t0(0) o3->fs1-OST0038-osc-ffff8889e1c27800@10.47.18.5@o2ib1:6/4 lens 488/440 e 1 to 1 dl 1566336491 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 Aug 20 22:40:37 cpu-e-1057 kernel: Lustre: fs1-OST0038-osc-ffff8889e1c27800: Connection to fs1-OST0038 (at 10.47.18.5@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 22:40:37 cpu-e-1057 kernel: Lustre: fs1-OST0038-osc-ffff8889e1c27800: Connection restored to 10.47.18.5@o2ib1 (at 10.47.18.5@o2ib1) Aug 20 22:42:19 cpu-e-1057 kernel: perf: interrupt took too long (3137 > 3130), lowering kernel.perf_event_max_sample_rate to 63000 HOSTS ------------------------------------------------------------------------- cpu-e-1056 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-08-20 19:50:00 BST, end at Wed 2019-08-21 11:30:06 BST. -- Aug 20 22:35:45 cpu-e-1056 kernel: perf: interrupt took too long (2501 > 2500), lowering kernel.perf_event_max_sample_rate to 79000 HOSTS ------------------------------------------------------------------------- cpu-e-837 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-08-20 19:49:07 BST, end at Wed 2019-08-21 11:30:06 BST. -- Aug 20 22:34:21 cpu-e-837 kernel: perf: interrupt took too long (2508 > 2500), lowering kernel.perf_event_max_sample_rate to 79000 HOSTS ------------------------------------------------------------------------- cpu-e-1058 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-08-20 19:50:13 BST, end at Wed 2019-08-21 11:30:06 BST. -- Aug 20 22:35:50 cpu-e-1058 kernel: perf: interrupt took too long (2501 > 2500), lowering kernel.perf_event_max_sample_rate to 79000 HOSTS ------------------------------------------------------------------------- cpu-e-836 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-08-20 19:48:44 BST, end at Wed 2019-08-21 11:30:06 BST. -- Aug 20 22:31:46 cpu-e-836 kernel: perf: interrupt took too long (2516 > 2500), lowering kernel.perf_event_max_sample_rate to 79000 HOSTS ------------------------------------------------------------------------- cpu-e-1060 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-08-20 19:49:46 BST, end at Wed 2019-08-21 11:30:06 BST. -- Aug 20 22:33:10 cpu-e-1060 kernel: perf: interrupt took too long (2519 > 2500), lowering kernel.perf_event_max_sample_rate to 79000