-- Logs begin at Sat 2021-05-08 22:30:01 PDT, end at Mon 2021-05-10 10:29:44 PDT. --
May 08 22:30:02 oak-io2-s1 kernel: LustreError: 217703:0:(ldlm_lib.c:3338:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8be9c51e8050 x1695405377218560/t0(0) o4->25ca5655-1c3b-4@10.51.15.8@o2ib3:260/0 lens 488/448 e 0 to 0 dl 1620538300 ref 1 fl Interpret:/0/0 rc 0/0
May 08 22:30:02 oak-io2-s1 kernel: Lustre: oak-OST003e: Bulk IO write error with 25ca5655-1c3b-4 (at 10.51.15.8@o2ib3), client will retry: rc = -110
May 08 22:31:17 oak-io2-s1 kernel: LustreError: 137-5: oak-OST005f_UUID: not available for connect from 10.51.1.49@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server.
May 08 22:31:42 oak-io2-s1 kernel: Lustre: oak-OST0054: Client 7a92055d-404d-2432-fe71-6231da9f74d2 (at 10.51.1.49@o2ib3) reconnecting
May 08 22:31:42 oak-io2-s1 kernel: Lustre: Skipped 30 previous similar messages
May 08 22:32:23 oak-io2-s1 kernel: Lustre: oak-OST0050: Connection restored to a0588488-3bcf-4 (at 10.51.3.33@o2ib3)
May 08 22:32:23 oak-io2-s1 kernel: Lustre: Skipped 534 previous similar messages
May 08 22:42:23 oak-io2-s1 kernel: Lustre: oak-OST004a: Connection restored to  (at 10.51.5.67@o2ib3)
May 08 22:42:23 oak-io2-s1 kernel: Lustre: Skipped 499 previous similar messages
May 08 22:52:26 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to  (at 10.51.3.65@o2ib3)
May 08 22:52:26 oak-io2-s1 kernel: Lustre: Skipped 511 previous similar messages
May 08 23:02:26 oak-io2-s1 kernel: Lustre: oak-OST004a: Connection restored to 09e5c5f8-98ce-4 (at 10.50.9.66@o2ib2)
May 08 23:02:26 oak-io2-s1 kernel: Lustre: Skipped 408 previous similar messages
May 08 23:09:36 oak-io2-s1 kernel: Lustre: oak-OST0044: Client 143a932c-b77e-167c-7fcd-b1cee9ff1e61 (at 10.51.5.2@o2ib3) reconnecting
May 08 23:09:36 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 08 23:12:27 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to 83e05d96-ba49-4 (at 10.51.5.35@o2ib3)
May 08 23:12:27 oak-io2-s1 kernel: Lustre: Skipped 331 previous similar messages
May 08 23:22:27 oak-io2-s1 kernel: Lustre: oak-OST003a: Connection restored to 930eb99f-4c41-4 (at 10.50.8.28@o2ib2)
May 08 23:22:27 oak-io2-s1 kernel: Lustre: Skipped 287 previous similar messages
May 08 23:28:41 oak-io2-s1 kernel: Lustre: oak-OST003e: Client a4bada1b-5f7f-dd47-5c02-ece74e11c8fc (at 10.51.15.16@o2ib3) reconnecting
May 08 23:28:41 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
May 08 23:28:51 oak-io2-s1 kernel: Lustre: oak-OST0042: Client 143a932c-b77e-167c-7fcd-b1cee9ff1e61 (at 10.51.5.2@o2ib3) reconnecting
May 08 23:32:31 oak-io2-s1 kernel: Lustre: oak-OST0046: Connection restored to 6488a11f-7a2b-4 (at 10.50.6.63@o2ib2)
May 08 23:32:31 oak-io2-s1 kernel: Lustre: Skipped 321 previous similar messages
May 08 23:39:42 oak-io2-s1 kernel: Lustre: oak-OST0038: Client eab9e2cf-af4c-4 (at 10.51.15.1@o2ib3) reconnecting
May 08 23:39:42 oak-io2-s1 kernel: Lustre: oak-OST003a: Client eab9e2cf-af4c-4 (at 10.51.15.1@o2ib3) reconnecting
May 08 23:39:42 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 08 23:39:42 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 08 23:42:34 oak-io2-s1 kernel: Lustre: oak-OST0040: Connection restored to 72ad3992-e887-4 (at 10.50.6.64@o2ib2)
May 08 23:42:34 oak-io2-s1 kernel: Lustre: Skipped 596 previous similar messages
May 08 23:50:32 oak-io2-s1 kernel: Lustre: oak-OST0050: Client fce68cff-df67-0383-fa34-7c6a60e3c393 (at 10.50.2.62@o2ib2) reconnecting
May 08 23:50:32 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 08 23:51:49 oak-io2-s1 kernel: Lustre: oak-OST003c: Client 1ff83831-40e8-4119-92ba-0d974d2ff022 (at 10.51.5.4@o2ib3) reconnecting
May 08 23:51:50 oak-io2-s1 kernel: LustreError: 218825:0:(ldlm_lib.c:3338:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8be9a86a1850 x1696234687416384/t0(0) o4->afb0548c-9287-4@10.51.15.2@o2ib3:635/0 lens 488/448 e 0 to 0 dl 1620543205 ref 1 fl Interpret:/0/0 rc 0/0
May 08 23:51:50 oak-io2-s1 kernel: Lustre: oak-OST004e: Bulk IO write error with afb0548c-9287-4 (at 10.51.15.2@o2ib3), client will retry: rc = -110
May 08 23:51:50 oak-io2-s1 kernel: LustreError: 218825:0:(ldlm_lib.c:3338:target_bulk_io()) Skipped 1 previous similar message
May 08 23:51:52 oak-io2-s1 kernel: Lustre: oak-OST005e: Client 33016c70-be50-803e-0431-7a51cafca4a6 (at 10.51.14.19@o2ib3) reconnecting
May 08 23:51:52 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 08 23:52:04 oak-io2-s1 kernel: Lustre: oak-OST003e: Client f71d1188-d01c-9fa0-935c-a63ad652da8a (at 10.51.12.21@o2ib3) reconnecting
May 08 23:52:37 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to 2336974a-ecec-4 (at 10.51.13.22@o2ib3)
May 08 23:52:37 oak-io2-s1 kernel: Lustre: Skipped 360 previous similar messages
May 08 23:53:52 oak-io2-s1 kernel: Lustre: oak-OST004c: Client 9488f889-3d86-6397-835c-2d56c219fc30 (at 10.50.17.11@o2ib2) reconnecting
May 08 23:53:52 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
May 09 00:02:37 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to ebf95241-55a9-4 (at 10.50.7.50@o2ib2)
May 09 00:02:37 oak-io2-s1 kernel: Lustre: Skipped 402 previous similar messages
May 09 00:04:55 oak-io2-s1 kernel: Lustre: oak-OST004a: Client 7e3f34f1-e0b1-6de7-4a42-e4ccca504809 (at 10.51.2.47@o2ib3) reconnecting
May 09 00:04:55 oak-io2-s1 kernel: Lustre: Skipped 12 previous similar messages
May 09 00:07:44 oak-io2-s1 kernel: Lustre: oak-OST0036: Client b6f5f41d-ab62-4469-75f6-3ce01ae123b6 (at 10.51.0.17@o2ib3) reconnecting
May 09 00:07:44 oak-io2-s1 kernel: Lustre: oak-OST0032: Client b6f5f41d-ab62-4469-75f6-3ce01ae123b6 (at 10.51.0.17@o2ib3) reconnecting
May 09 00:07:44 oak-io2-s1 kernel: Lustre: oak-OST003a: Client b6f5f41d-ab62-4469-75f6-3ce01ae123b6 (at 10.51.0.17@o2ib3) reconnecting
May 09 00:07:44 oak-io2-s1 kernel: Lustre: Skipped 4 previous similar messages
May 09 00:12:37 oak-io2-s1 kernel: Lustre: oak-OST0050: Connection restored to b77576b5-26ca-4 (at 10.50.14.5@o2ib2)
May 09 00:12:37 oak-io2-s1 kernel: Lustre: Skipped 383 previous similar messages
May 09 00:22:39 oak-io2-s1 kernel: Lustre: oak-OST005c: Connection restored to d71e2875-d424-4 (at 10.49.18.17@o2ib1)
May 09 00:22:39 oak-io2-s1 kernel: Lustre: Skipped 362 previous similar messages
May 09 00:25:24 oak-io2-s1 kernel: Lustre: oak-OST0056: Client 7f5a03b0-a887-10eb-7386-d75d82cdd92b (at 10.51.13.20@o2ib3) reconnecting
May 09 00:25:25 oak-io2-s1 kernel: Lustre: oak-OST0054: Client 9013d991-28ee-172b-7f94-1611b1f321a6 (at 10.51.1.14@o2ib3) reconnecting
May 09 00:26:11 oak-io2-s1 kernel: Lustre: oak-OST0036: Client 156315a7-a82d-b4fe-847a-396165636f38 (at 10.51.14.3@o2ib3) reconnecting
May 09 00:26:13 oak-io2-s1 kernel: Lustre: oak-OST0046: Client 4d1f30e9-057b-184d-d37b-7a8aa184f0ba (at 10.51.12.10@o2ib3) reconnecting
May 09 00:26:13 oak-io2-s1 kernel: Lustre: Skipped 9 previous similar messages
May 09 00:26:21 oak-io2-s1 kernel: Lustre: oak-OST0044: Client 975c5baf-a929-c88d-37e2-6223f3e0c770 (at 10.51.15.10@o2ib3) reconnecting
May 09 00:26:21 oak-io2-s1 kernel: Lustre: oak-OST0034: Client 975c5baf-a929-c88d-37e2-6223f3e0c770 (at 10.51.15.10@o2ib3) reconnecting
May 09 00:27:43 oak-io2-s1 kernel: Lustre: oak-OST0054: Client 6aa7a50a-0f21-52fd-5fab-0fc412d84d45 (at 10.50.5.21@o2ib2) reconnecting
May 09 00:30:42 oak-io2-s1 kernel: Lustre: oak-OST0044: Client 09749127-876e-3e7f-b6a0-7c54e61383b6 (at 10.51.3.11@o2ib3) reconnecting
May 09 00:30:42 oak-io2-s1 kernel: Lustre: Skipped 3 previous similar messages
May 09 00:31:15 oak-io2-s1 kernel: Lustre: oak-OST0046: Client a7ebb784-0f8b-4 (at 10.51.6.23@o2ib3) reconnecting
May 09 00:31:15 oak-io2-s1 kernel: Lustre: Skipped 7 previous similar messages
May 09 00:32:39 oak-io2-s1 kernel: Lustre: oak-OST0052: Connection restored to 36b00265-a377-4 (at 10.49.18.27@o2ib1)
May 09 00:32:39 oak-io2-s1 kernel: Lustre: Skipped 431 previous similar messages
May 09 00:36:39 oak-io2-s1 kernel: Lustre: oak-OST003c: Client e20da35b-29be-1b37-2be1-552ec3845bfe (at 10.50.5.63@o2ib2) reconnecting
May 09 00:36:39 oak-io2-s1 kernel: Lustre: Skipped 4 previous similar messages
May 09 00:42:40 oak-io2-s1 kernel: Lustre: oak-OST0054: Connection restored to b77576b5-26ca-4 (at 10.50.14.5@o2ib2)
May 09 00:42:40 oak-io2-s1 kernel: Lustre: Skipped 871 previous similar messages
May 09 00:48:38 oak-io2-s1 kernel: Lustre: oak-OST003e: Client 2cb5b80e-8bdb-e5df-69d7-53749fbd86ff (at 10.51.12.8@o2ib3) reconnecting
May 09 00:48:38 oak-io2-s1 kernel: Lustre: Skipped 3 previous similar messages
May 09 00:48:39 oak-io2-s1 kernel: LustreError: 218386:0:(ldlm_lib.c:3338:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8bf730fdc050 x1698973532068160/t0(0) o4->2cb5b80e-8bdb-e5df-69d7-53749fbd86ff@10.51.12.8@o2ib3:266/0 lens 488/448 e 0 to 0 dl 1620546611 ref 1 fl Interpret:/0/0 rc 0/0
May 09 00:48:39 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 3, status -103, desc ffff8bca7a062400
May 09 00:48:39 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8be7b76a6800
May 09 00:48:39 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8be7b76a6800
May 09 00:48:39 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8beaca5b2800
May 09 00:48:39 oak-io2-s1 kernel: Lustre: oak-OST003e: Bulk IO write error with 2cb5b80e-8bdb-e5df-69d7-53749fbd86ff (at 10.51.12.8@o2ib3), client will retry: rc = -110
May 09 00:48:39 oak-io2-s1 kernel: Lustre: Skipped 3 previous similar messages
May 09 00:48:39 oak-io2-s1 kernel: LustreError: 218386:0:(ldlm_lib.c:3338:target_bulk_io()) Skipped 6 previous similar messages
May 09 00:49:43 oak-io2-s1 kernel: LustreError: 218829:0:(ldlm_lib.c:3353:target_bulk_io()) @@@ truncated bulk READ 3145728(4194304)  req@ffff8bf714243850 x1696884695566592/t0(0) o3->615f6034-750a-f27b-66dc-7c591b98c535@10.51.4.20@o2ib3:266/0 lens 488/440 e 0 to 0 dl 1620546611 ref 1 fl Interpret:/0/0 rc 0/0
May 09 00:49:43 oak-io2-s1 kernel: LustreError: 218072:0:(ldlm_lib.c:3344:target_bulk_io()) @@@ network error on bulk READ  req@ffff8bf7369e0050 x1696884695566656/t0(0) o3->615f6034-750a-f27b-66dc-7c591b98c535@10.51.4.20@o2ib3:266/0 lens 488/440 e 0 to 0 dl 1620546611 ref 1 fl Interpret:/0/0 rc 0/0
May 09 00:49:43 oak-io2-s1 kernel: Lustre: oak-OST004c: Bulk IO read error with a2c5442d-4a42-4 (at 10.51.2.45@o2ib3), client will retry: rc -110
May 09 00:49:43 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
May 09 00:49:43 oak-io2-s1 kernel: LustreError: 217700:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 1048576(2006016)  req@ffff8bf73b8fa850 x1696234721399296/t0(0) o4->afb0548c-9287-4@10.51.15.2@o2ib3:266/0 lens 488/448 e 0 to 0 dl 1620546611 ref 1 fl Interpret:/0/0 rc 0/0
May 09 00:49:43 oak-io2-s1 kernel: LustreError: 217700:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) Skipped 3 previous similar messages
May 09 00:49:43 oak-io2-s1 kernel: Lustre: oak-OST003a: Bulk IO write error with 413272a1-a696-4 (at 10.51.2.13@o2ib3), client will retry: rc = -110
May 09 00:49:43 oak-io2-s1 kernel: Lustre: Skipped 4 previous similar messages
May 09 00:49:43 oak-io2-s1 kernel: LustreError: 218829:0:(ldlm_lib.c:3353:target_bulk_io()) Skipped 8 previous similar messages
May 09 00:51:44 oak-io2-s1 kernel: Lustre: oak-OST003c: haven't heard from client 7e0a9a61-2b20-4 (at 10.51.2.19@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8bf5e476dc00, cur 1620546704 expire 1620546554 last 1620546477
May 09 00:51:44 oak-io2-s1 kernel: Lustre: Skipped 575 previous similar messages
May 09 00:52:45 oak-io2-s1 kernel: Lustre: oak-OST0056: Connection restored to 2336974a-ecec-4 (at 10.51.13.22@o2ib3)
May 09 00:52:45 oak-io2-s1 kernel: Lustre: Skipped 1337 previous similar messages
May 09 00:53:02 oak-io2-s1 kernel: ses 12:0:61:0: attempting task abort! scmd(ffff8be6ac372bc0)
May 09 00:53:02 oak-io2-s1 kernel: ses 12:0:61:0: [sg551] tag#9 CDB: Receive Diagnostic 1c 01 0f ff ff 00
May 09 00:53:02 oak-io2-s1 kernel: scsi target12:0:61: _scsih_tm_display_info: handle(0x0050), sas_address(0x5001636001bd167d), phy(76)
May 09 00:53:02 oak-io2-s1 kernel: scsi target12:0:61: enclosurelogical id(0x5001636001bd167d), slot(60) 
May 09 00:53:02 oak-io2-s1 kernel: scsi target12:0:61: enclosure level(0x0001), connector name(     )
May 09 00:53:02 oak-io2-s1 kernel: ses 12:0:61:0: task abort: SUCCESS scmd(ffff8be6ac372bc0)
May 09 01:00:01 oak-io2-s1 kernel: md: data-check of RAID array md26
May 09 01:00:07 oak-io2-s1 kernel: md: data-check of RAID array md42
May 09 01:02:46 oak-io2-s1 kernel: Lustre: oak-OST0044: Connection restored to 76d7694c-c22d-4 (at 10.49.18.13@o2ib1)
May 09 01:02:46 oak-io2-s1 kernel: Lustre: Skipped 473 previous similar messages
May 09 01:09:36 oak-io2-s1 kernel: Lustre: oak-OST0050: Client 457668ab-9620-dd73-14d7-b567e81e6a47 (at 10.51.2.11@o2ib3) reconnecting
May 09 01:09:36 oak-io2-s1 kernel: Lustre: Skipped 819 previous similar messages
May 09 01:09:38 oak-io2-s1 kernel: LustreError: 218419:0:(ldlm_lib.c:3338:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff8bf263c7b050 x1699061131617664/t0(0) o3->7e3f34f1-e0b1-6de7-4a42-e4ccca504809@10.51.2.47@o2ib3:20/0 lens 488/440 e 0 to 0 dl 1620547875 ref 1 fl Interpret:/0/0 rc 0/0
May 09 01:09:38 oak-io2-s1 kernel: Lustre: oak-OST0058: Bulk IO read error with 7e3f34f1-e0b1-6de7-4a42-e4ccca504809 (at 10.51.2.47@o2ib3), client will retry: rc -110
May 09 01:09:38 oak-io2-s1 kernel: Lustre: Skipped 9 previous similar messages
May 09 01:09:38 oak-io2-s1 kernel: LustreError: 218419:0:(ldlm_lib.c:3338:target_bulk_io()) Skipped 3 previous similar messages
May 09 01:12:49 oak-io2-s1 kernel: Lustre: oak-OST0054: Connection restored to 25cf3830-668c-4 (at 10.51.5.22@o2ib3)
May 09 01:12:49 oak-io2-s1 kernel: Lustre: Skipped 488 previous similar messages
May 09 01:22:51 oak-io2-s1 kernel: Lustre: oak-OST0052: Connection restored to 0524aa72-f9f7-4 (at 10.51.4.47@o2ib3)
May 09 01:22:51 oak-io2-s1 kernel: Lustre: Skipped 385 previous similar messages
May 09 01:25:52 oak-io2-s1 kernel: Lustre: oak-OST003e: Client 2cb5b80e-8bdb-e5df-69d7-53749fbd86ff (at 10.51.12.8@o2ib3) reconnecting
May 09 01:25:52 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 09 01:25:52 oak-io2-s1 kernel: LustreError: 217852:0:(ldlm_lib.c:3338:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8be7b70e3050 x1698973569388800/t0(0) o4->2cb5b80e-8bdb-e5df-69d7-53749fbd86ff@10.51.12.8@o2ib3:239/0 lens 488/448 e 0 to 0 dl 1620548849 ref 1 fl Interpret:/0/0 rc 0/0
May 09 01:25:52 oak-io2-s1 kernel: Lustre: oak-OST003e: Bulk IO write error with 2cb5b80e-8bdb-e5df-69d7-53749fbd86ff (at 10.51.12.8@o2ib3), client will retry: rc = -110
May 09 01:25:52 oak-io2-s1 kernel: Lustre: Skipped 6 previous similar messages
May 09 01:25:52 oak-io2-s1 kernel: LustreError: 217852:0:(ldlm_lib.c:3338:target_bulk_io()) Skipped 2 previous similar messages
May 09 01:25:53 oak-io2-s1 kernel: LustreError: 217702:0:(ldlm_lib.c:3338:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8bf667070850 x1698973569361664/t0(0) o4->2cb5b80e-8bdb-e5df-69d7-53749fbd86ff@10.51.12.8@o2ib3:235/0 lens 488/448 e 0 to 0 dl 1620548845 ref 1 fl Interpret:/0/0 rc 0/0
May 09 01:25:53 oak-io2-s1 kernel: Lustre: oak-OST003e: Bulk IO write error with 2cb5b80e-8bdb-e5df-69d7-53749fbd86ff (at 10.51.12.8@o2ib3), client will retry: rc = -110
May 09 01:25:53 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
May 09 01:25:53 oak-io2-s1 kernel: LustreError: 217702:0:(ldlm_lib.c:3338:target_bulk_io()) Skipped 3 previous similar messages
May 09 01:25:53 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 3, status -103, desc ffff8be5cb5be400
May 09 01:25:53 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 3, status -103, desc ffff8be5cb5be400
May 09 01:26:48 oak-io2-s1 kernel: Lustre: oak-OST005a: Client ff2a7001-aaa4-874f-d2c8-422732a620f2 (at 10.50.1.54@o2ib2) reconnecting
May 09 01:26:48 oak-io2-s1 kernel: Lustre: oak-OST0050: Client ff2a7001-aaa4-874f-d2c8-422732a620f2 (at 10.50.1.54@o2ib2) reconnecting
May 09 01:26:48 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
May 09 01:26:48 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 09 01:28:38 oak-io2-s1 kernel: Lustre: oak-OST0034: Client bfd19c73-fed3-06f5-ec55-d90121312358 (at 10.51.14.5@o2ib3) reconnecting
May 09 01:28:38 oak-io2-s1 kernel: Lustre: Skipped 39 previous similar messages
May 09 01:32:51 oak-io2-s1 kernel: Lustre: oak-OST004c: Connection restored to 0d2b66dc-1b19-4 (at 10.50.8.48@o2ib2)
May 09 01:32:51 oak-io2-s1 kernel: Lustre: Skipped 400 previous similar messages
May 09 01:42:52 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to f0643f46-4acf-4 (at 10.51.2.41@o2ib3)
May 09 01:42:52 oak-io2-s1 kernel: Lustre: Skipped 306 previous similar messages
May 09 01:52:52 oak-io2-s1 kernel: Lustre: oak-OST005a: Connection restored to  (at 10.51.14.15@o2ib3)
May 09 01:52:52 oak-io2-s1 kernel: Lustre: Skipped 305 previous similar messages
May 09 01:55:16 oak-io2-s1 kernel: Lustre: oak-OST0050: Client 131f375f-e775-170c-9045-4c4e4584c905 (at 10.50.1.37@o2ib2) reconnecting
May 09 01:55:16 oak-io2-s1 kernel: Lustre: Skipped 12 previous similar messages
May 09 02:01:22 oak-io2-s1 kernel: Lustre: oak-OST005a: Client 943bc612-9b50-c21a-63fc-f5bffc57a9a3 (at 10.50.2.49@o2ib2) reconnecting
May 09 02:01:22 oak-io2-s1 kernel: Lustre: Skipped 5 previous similar messages
May 09 02:02:53 oak-io2-s1 kernel: Lustre: oak-OST003a: Connection restored to  (at 10.51.14.15@o2ib3)
May 09 02:02:53 oak-io2-s1 kernel: Lustre: Skipped 283 previous similar messages
May 09 02:12:54 oak-io2-s1 kernel: Lustre: oak-OST0052: Connection restored to 012f4751-9ad6-4 (at 10.49.18.22@o2ib1)
May 09 02:12:54 oak-io2-s1 kernel: Lustre: Skipped 378 previous similar messages
May 09 02:20:36 oak-io2-s1 kernel: Lustre: oak-OST0040: Client c68a0227-b978-b7b6-2a11-b64857fe7d6e (at 10.50.4.12@o2ib2) reconnecting
May 09 02:22:56 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to 1a571093-7cfe-4 (at 10.50.10.9@o2ib2)
May 09 02:22:56 oak-io2-s1 kernel: Lustre: Skipped 432 previous similar messages
May 09 02:24:38 oak-io2-s1 kernel: Lustre: oak-OST0034: Client 947bbef2-3dca-4145-090e-2ded0027f322 (at 10.51.15.4@o2ib3) reconnecting
May 09 02:24:38 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 09 02:25:38 oak-io2-s1 kernel: Lustre: oak-OST004c: Client 69e8d4d6-d273-dee1-a085-8f7f5adcd394 (at 10.50.2.52@o2ib2) reconnecting
May 09 02:25:38 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
May 09 02:28:26 oak-io2-s1 kernel: Lustre: oak-OST0046: Client 2fd8a4be-fd07-4 (at 10.50.3.53@o2ib2) reconnecting
May 09 02:28:26 oak-io2-s1 kernel: Lustre: Skipped 19 previous similar messages
May 09 02:33:01 oak-io2-s1 kernel: Lustre: oak-OST004e: Connection restored to 2baba63a-d33e-4 (at 10.49.8.33@o2ib1)
May 09 02:33:01 oak-io2-s1 kernel: Lustre: Skipped 288 previous similar messages
May 09 02:40:39 oak-io2-s1 kernel: Lustre: oak-OST0058: Client 40f689f2-9586-b260-63a4-b54b795485cb (at 10.50.5.70@o2ib2) reconnecting
May 09 02:43:05 oak-io2-s1 kernel: Lustre: oak-OST003a: Connection restored to 9eec699c-b959-4 (at 10.50.17.29@o2ib2)
May 09 02:43:05 oak-io2-s1 kernel: Lustre: Skipped 334 previous similar messages
May 09 02:45:54 oak-io2-s1 kernel: Lustre: oak-OST003c: Client 25ca5655-1c3b-4 (at 10.51.15.8@o2ib3) reconnecting
May 09 02:45:54 oak-io2-s1 kernel: Lustre: Skipped 6 previous similar messages
May 09 02:53:06 oak-io2-s1 kernel: Lustre: oak-OST005e: Connection restored to 473ec31f-2480-4 (at 10.50.14.9@o2ib2)
May 09 02:53:06 oak-io2-s1 kernel: Lustre: Skipped 282 previous similar messages
May 09 02:55:28 oak-io2-s1 kernel: Lustre: oak-OST0044: Client ef1be05b-d76e-a01a-4132-86eb2bcbe0b3 (at 10.50.2.65@o2ib2) reconnecting
May 09 03:03:07 oak-io2-s1 kernel: Lustre: oak-OST0046: Connection restored to 765dbb74-f3af-4 (at 10.50.9.52@o2ib2)
May 09 03:03:07 oak-io2-s1 kernel: Lustre: Skipped 200 previous similar messages
May 09 03:13:09 oak-io2-s1 kernel: Lustre: oak-OST004a: Connection restored to  (at 10.51.3.60@o2ib3)
May 09 03:13:09 oak-io2-s1 kernel: Lustre: Skipped 225 previous similar messages
May 09 03:19:11 oak-io2-s1 kernel: Lustre: oak-OST0054: Client c1712232-844c-9a41-1c20-e408d7e53c1f (at 10.50.8.64@o2ib2) reconnecting
May 09 03:19:11 oak-io2-s1 kernel: Lustre: Skipped 3 previous similar messages
May 09 03:19:12 oak-io2-s1 kernel: Lustre: oak-OST0052: Client 41c1d926-ee25-4 (at 10.50.6.15@o2ib2) reconnecting
May 09 03:19:12 oak-io2-s1 kernel: Lustre: oak-OST005e: Client 41c1d926-ee25-4 (at 10.50.6.15@o2ib2) reconnecting
May 09 03:19:12 oak-io2-s1 kernel: Lustre: Skipped 7 previous similar messages
May 09 03:20:52 oak-io2-s1 kernel: LustreError: 137-5: oak-OST005f_UUID: not available for connect from 10.50.1.23@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
May 09 03:23:09 oak-io2-s1 kernel: Lustre: oak-OST004a: Connection restored to 2212eb77-5034-4 (at 10.50.7.56@o2ib2)
May 09 03:23:09 oak-io2-s1 kernel: Lustre: Skipped 376 previous similar messages
May 09 03:33:14 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to  (at 10.51.4.16@o2ib3)
May 09 03:33:14 oak-io2-s1 kernel: Lustre: Skipped 280 previous similar messages
May 09 03:39:04 oak-io2-s1 kernel: Lustre: oak-OST003c: Client 33e06152-c9bd-ac77-8fdb-b49a0d1c77b7 (at 10.50.2.42@o2ib2) reconnecting
May 09 03:39:04 oak-io2-s1 kernel: Lustre: oak-OST0050: Client 9488f889-3d86-6397-835c-2d56c219fc30 (at 10.50.17.11@o2ib2) reconnecting
May 09 03:39:04 oak-io2-s1 kernel: Lustre: Skipped 3 previous similar messages
May 09 03:43:16 oak-io2-s1 kernel: Lustre: oak-OST0056: Connection restored to f3b73773-79ba-4 (at 10.51.14.13@o2ib3)
May 09 03:43:16 oak-io2-s1 kernel: Lustre: Skipped 264 previous similar messages
May 09 03:44:45 oak-io2-s1 kernel: Lustre: oak-OST005e: Client fd184b36-945e-a3ba-9ba0-305c350ba963 (at 10.50.6.21@o2ib2) reconnecting
May 09 03:44:45 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 09 03:47:33 oak-io2-s1 kernel: Lustre: oak-OST0052: Client 84578764-fb76-48a7-9216-8bbc65a8d431 (at 10.50.7.9@o2ib2) reconnecting
May 09 03:53:18 oak-io2-s1 kernel: Lustre: oak-OST0056: Connection restored to 1fbd93da-ab0e-4 (at 10.51.2.40@o2ib3)
May 09 03:53:18 oak-io2-s1 kernel: Lustre: Skipped 280 previous similar messages
May 09 04:03:18 oak-io2-s1 kernel: Lustre: oak-OST003e: Connection restored to e1d4c1dc-2a26-4 (at 10.50.6.60@o2ib2)
May 09 04:03:18 oak-io2-s1 kernel: Lustre: Skipped 213 previous similar messages
May 09 04:13:22 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to bea21439-f3ea-4 (at 10.51.5.8@o2ib3)
May 09 04:13:22 oak-io2-s1 kernel: Lustre: Skipped 207 previous similar messages
May 09 04:23:24 oak-io2-s1 kernel: Lustre: oak-OST005c: Connection restored to b7dbc60f-1178-4 (at 10.49.26.32@o2ib1)
May 09 04:23:24 oak-io2-s1 kernel: Lustre: Skipped 237 previous similar messages
May 09 04:33:28 oak-io2-s1 kernel: Lustre: oak-OST0058: Connection restored to c07d0863-218d-4 (at 10.51.4.30@o2ib3)
May 09 04:33:28 oak-io2-s1 kernel: Lustre: Skipped 298 previous similar messages
May 09 04:43:28 oak-io2-s1 kernel: Lustre: oak-OST0056: Connection restored to bea21439-f3ea-4 (at 10.51.5.8@o2ib3)
May 09 04:43:28 oak-io2-s1 kernel: Lustre: Skipped 241 previous similar messages
May 09 04:53:28 oak-io2-s1 kernel: Lustre: oak-OST0046: Connection restored to 19d08a37-93d2-4 (at 10.51.5.30@o2ib3)
May 09 04:53:28 oak-io2-s1 kernel: Lustre: Skipped 334 previous similar messages
May 09 05:03:31 oak-io2-s1 kernel: Lustre: oak-OST0050: Connection restored to 668a2172-2080-4 (at 10.51.3.70@o2ib3)
May 09 05:03:31 oak-io2-s1 kernel: Lustre: Skipped 278 previous similar messages
May 09 05:13:36 oak-io2-s1 kernel: Lustre: oak-OST0046: Connection restored to a6cef850-c36c-4 (at 10.50.17.35@o2ib2)
May 09 05:13:36 oak-io2-s1 kernel: Lustre: Skipped 285 previous similar messages
May 09 05:23:37 oak-io2-s1 kernel: Lustre: oak-OST004c: Connection restored to 6488a11f-7a2b-4 (at 10.50.6.63@o2ib2)
May 09 05:23:37 oak-io2-s1 kernel: Lustre: Skipped 375 previous similar messages
May 09 05:33:39 oak-io2-s1 kernel: Lustre: oak-OST003a: Connection restored to 4d091fba-fed3-4 (at 10.50.17.10@o2ib2)
May 09 05:33:39 oak-io2-s1 kernel: Lustre: Skipped 320 previous similar messages
May 09 05:43:40 oak-io2-s1 kernel: Lustre: oak-OST003c: Connection restored to 4d091fba-fed3-4 (at 10.50.17.10@o2ib2)
May 09 05:43:40 oak-io2-s1 kernel: Lustre: Skipped 204 previous similar messages
May 09 05:53:44 oak-io2-s1 kernel: Lustre: oak-OST0038: Connection restored to  (at 10.51.5.66@o2ib3)
May 09 05:53:44 oak-io2-s1 kernel: Lustre: Skipped 330 previous similar messages
May 09 06:03:46 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to 0871860c-cfdf-4 (at 10.51.3.30@o2ib3)
May 09 06:03:46 oak-io2-s1 kernel: Lustre: Skipped 306 previous similar messages
May 09 06:13:49 oak-io2-s1 kernel: Lustre: oak-OST003a: Connection restored to cfbdda79-1410-4 (at 10.49.28.7@o2ib1)
May 09 06:13:49 oak-io2-s1 kernel: Lustre: Skipped 431 previous similar messages
May 09 06:23:50 oak-io2-s1 kernel: Lustre: oak-OST0038: Connection restored to 71bdfe57-cd01-4 (at 10.49.28.8@o2ib1)
May 09 06:23:50 oak-io2-s1 kernel: Lustre: Skipped 466 previous similar messages
May 09 06:33:52 oak-io2-s1 kernel: Lustre: oak-OST0054: Connection restored to b3dcd716-6ba2-4 (at 10.50.9.29@o2ib2)
May 09 06:33:52 oak-io2-s1 kernel: Lustre: Skipped 487 previous similar messages
May 09 06:39:17 oak-io2-s1 kernel: Lustre: oak-OST0036: Client 4e65e11b-a59b-af77-3cd8-331f8c77cfc3 (at 10.50.2.16@o2ib2) reconnecting
May 09 06:39:17 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
May 09 06:39:18 oak-io2-s1 kernel: Lustre: oak-OST004a: Client 4e65e11b-a59b-af77-3cd8-331f8c77cfc3 (at 10.50.2.16@o2ib2) reconnecting
May 09 06:39:18 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
May 09 06:43:52 oak-io2-s1 kernel: Lustre: oak-OST004a: Connection restored to a304c9f8-34ad-4 (at 10.50.7.57@o2ib2)
May 09 06:43:52 oak-io2-s1 kernel: Lustre: Skipped 521 previous similar messages
May 09 06:53:52 oak-io2-s1 kernel: Lustre: oak-OST003e: Connection restored to f0643f46-4acf-4 (at 10.51.2.41@o2ib3)
May 09 06:53:52 oak-io2-s1 kernel: Lustre: Skipped 473 previous similar messages
May 09 07:03:53 oak-io2-s1 kernel: Lustre: oak-OST003a: Connection restored to 6bbc0d63-02d6-4 (at 10.50.10.71@o2ib2)
May 09 07:03:53 oak-io2-s1 kernel: Lustre: Skipped 391 previous similar messages
May 09 07:13:55 oak-io2-s1 kernel: Lustre: oak-OST0050: Connection restored to  (at 10.51.14.15@o2ib3)
May 09 07:13:55 oak-io2-s1 kernel: Lustre: Skipped 358 previous similar messages
May 09 07:23:57 oak-io2-s1 kernel: Lustre: oak-OST005e: Connection restored to 485e19fa-600a-4 (at 10.49.27.2@o2ib1)
May 09 07:23:57 oak-io2-s1 kernel: Lustre: Skipped 326 previous similar messages
May 09 07:33:58 oak-io2-s1 kernel: Lustre: oak-OST0050: Connection restored to 558e15a2-2e40-4 (at 10.50.9.26@o2ib2)
May 09 07:33:58 oak-io2-s1 kernel: Lustre: Skipped 814 previous similar messages
May 09 07:40:28 oak-io2-s1 kernel: Lustre: oak-OST0038: Client 44941934-ac39-2b19-c695-ab36186f52dc (at 10.50.10.57@o2ib2) reconnecting
May 09 07:40:28 oak-io2-s1 kernel: Lustre: oak-OST004c: Client 44941934-ac39-2b19-c695-ab36186f52dc (at 10.50.10.57@o2ib2) reconnecting
May 09 07:40:28 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 09 07:40:28 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 09 07:40:29 oak-io2-s1 kernel: Lustre: oak-OST005a: Client da57d56f-27d0-6de2-d934-adc33f336bff (at 10.50.2.28@o2ib2) reconnecting
May 09 07:40:29 oak-io2-s1 kernel: Lustre: Skipped 16 previous similar messages
May 09 07:44:00 oak-io2-s1 kernel: Lustre: oak-OST0048: Connection restored to 31357676-cbf3-4 (at 10.50.9.5@o2ib2)
May 09 07:44:00 oak-io2-s1 kernel: Lustre: Skipped 612 previous similar messages
May 09 07:45:32 oak-io2-s1 kernel: Lustre: oak-OST0052: Client 6cbff61b-9898-4 (at 10.51.13.9@o2ib3) reconnecting
May 09 07:45:32 oak-io2-s1 kernel: Lustre: Skipped 3 previous similar messages
May 09 07:54:01 oak-io2-s1 kernel: Lustre: oak-OST005a: Connection restored to bc8be7f7-9a50-4 (at 10.49.18.14@o2ib1)
May 09 07:54:01 oak-io2-s1 kernel: Lustre: Skipped 328 previous similar messages
May 09 08:04:01 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to 0524aa72-f9f7-4 (at 10.51.4.47@o2ib3)
May 09 08:04:01 oak-io2-s1 kernel: Lustre: Skipped 297 previous similar messages
May 09 08:14:05 oak-io2-s1 kernel: Lustre: oak-OST003e: Connection restored to 44893ef6-f0ab-4 (at 10.50.9.23@o2ib2)
May 09 08:14:05 oak-io2-s1 kernel: Lustre: Skipped 210 previous similar messages
May 09 08:24:08 oak-io2-s1 kernel: Lustre: oak-OST004e: Connection restored to 36b00265-a377-4 (at 10.49.18.27@o2ib1)
May 09 08:24:08 oak-io2-s1 kernel: Lustre: Skipped 310 previous similar messages
May 09 08:34:13 oak-io2-s1 kernel: Lustre: oak-OST004e: Connection restored to b7dbc60f-1178-4 (at 10.49.26.32@o2ib1)
May 09 08:34:13 oak-io2-s1 kernel: Lustre: Skipped 356 previous similar messages
May 09 08:44:15 oak-io2-s1 kernel: Lustre: oak-OST004e: Connection restored to 6f1aab12-ec2f-4 (at 10.49.27.8@o2ib1)
May 09 08:44:15 oak-io2-s1 kernel: Lustre: Skipped 332 previous similar messages
May 09 08:54:19 oak-io2-s1 kernel: Lustre: oak-OST0052: Connection restored to 7b740d36-20ba-4 (at 10.50.15.4@o2ib2)
May 09 08:54:19 oak-io2-s1 kernel: Lustre: Skipped 354 previous similar messages
May 09 09:04:19 oak-io2-s1 kernel: Lustre: oak-OST0046: Connection restored to 9b1109f5-bdcd-4 (at 10.49.27.10@o2ib1)
May 09 09:04:19 oak-io2-s1 kernel: Lustre: Skipped 498 previous similar messages
May 09 09:14:21 oak-io2-s1 kernel: Lustre: oak-OST003a: Connection restored to d2b765d6-c390-4 (at 10.51.5.1@o2ib3)
May 09 09:14:21 oak-io2-s1 kernel: Lustre: Skipped 427 previous similar messages
May 09 09:23:41 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8bf4f7853800
May 09 09:23:41 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8bf1cde22c00
May 09 09:23:41 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8be8954cec00
May 09 09:23:42 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 3, status -103, desc ffff8be8954c8800
May 09 09:23:42 oak-io2-s1 kernel: LustreError: 217832:0:(ldlm_lib.c:3344:target_bulk_io()) @@@ network error on bulk WRITE  req@ffff8bf9018da050 x1697861694183296/t0(0) o4->82effa21-5729-1828-1fbf-0c6e6269342f@10.51.0.63@o2ib3:214/0 lens 488/448 e 0 to 0 dl 1620577514 ref 1 fl Interpret:/0/0 rc 0/0
May 09 09:23:42 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8be5e8eac000
May 09 09:23:42 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8be5e8eac000
May 09 09:23:42 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8be0356f8400
May 09 09:23:42 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8be7957ae800
May 09 09:23:42 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8be8954cec00
May 09 09:23:42 oak-io2-s1 kernel: LustreError: 217832:0:(ldlm_lib.c:3344:target_bulk_io()) Skipped 1 previous similar message
May 09 09:23:42 oak-io2-s1 kernel: Lustre: oak-OST0032: Bulk IO write error with 82effa21-5729-1828-1fbf-0c6e6269342f (at 10.51.0.63@o2ib3), client will retry: rc = -110
May 09 09:23:42 oak-io2-s1 kernel: Lustre: Skipped 3 previous similar messages
May 09 09:24:22 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to 9078cd2f-a6ea-4 (at 10.51.6.2@o2ib3)
May 09 09:24:22 oak-io2-s1 kernel: Lustre: Skipped 612 previous similar messages
May 09 09:24:43 oak-io2-s1 kernel: LustreError: 218417:0:(ldlm_lib.c:3344:target_bulk_io()) @@@ network error on bulk READ  req@ffff8bf748ca1050 x1695190656896256/t0(0) o3->dbbeec82-59c5-4@10.51.14.11@o2ib3:213/0 lens 488/440 e 0 to 0 dl 1620577513 ref 1 fl Interpret:/0/0 rc 0/0
May 09 09:24:43 oak-io2-s1 kernel: LustreError: 217845:0:(ldlm_lib.c:3353:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff8bf73687e050 x1697134052321536/t0(0) o3->56a5a766-0782-0626-7e81-90dde2e2789a@10.51.2.28@o2ib3:213/0 lens 488/440 e 0 to 0 dl 1620577513 ref 1 fl Interpret:/0/0 rc 0/0
May 09 09:24:43 oak-io2-s1 kernel: Lustre: oak-OST0054: Bulk IO read error with a703c65e-c48f-97d7-efaf-c377b3ded349 (at 10.51.15.3@o2ib3), client will retry: rc -110
May 09 09:24:43 oak-io2-s1 kernel: Lustre: oak-OST0030: Bulk IO read error with 33016c70-be50-803e-0431-7a51cafca4a6 (at 10.51.14.19@o2ib3), client will retry: rc -110
May 09 09:24:43 oak-io2-s1 kernel: Lustre: Skipped 3 previous similar messages
May 09 09:24:43 oak-io2-s1 kernel: LustreError: 217857:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(1892)  req@ffff8bf7373b4850 x1696869650179200/t0(0) o4->c1ada0a4-de2a-c24a-4d28-c593030bd6bb@10.51.6.21@o2ib3:216/0 lens 488/448 e 0 to 0 dl 1620577516 ref 1 fl Interpret:/0/0 rc 0/0
May 09 09:24:43 oak-io2-s1 kernel: LustreError: 217857:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) Skipped 6 previous similar messages
May 09 09:24:43 oak-io2-s1 kernel: Lustre: oak-OST005e: Bulk IO write error with c1ada0a4-de2a-c24a-4d28-c593030bd6bb (at 10.51.6.21@o2ib3), client will retry: rc = -110
May 09 09:24:43 oak-io2-s1 kernel: LustreError: 218417:0:(ldlm_lib.c:3344:target_bulk_io()) Skipped 5 previous similar messages
May 09 09:26:28 oak-io2-s1 kernel: Lustre: oak-OST0030: Client 33016c70-be50-803e-0431-7a51cafca4a6 (at 10.51.14.19@o2ib3) reconnecting
May 09 09:26:28 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 09 09:26:28 oak-io2-s1 kernel: Lustre: oak-OST0050: Client 41687892-76b8-4 (at 10.51.3.23@o2ib3) reconnecting
May 09 09:26:28 oak-io2-s1 kernel: Lustre: Skipped 4 previous similar messages
May 09 09:26:29 oak-io2-s1 kernel: Lustre: oak-OST0040: Client 82effa21-5729-1828-1fbf-0c6e6269342f (at 10.51.0.63@o2ib3) reconnecting
May 09 09:26:29 oak-io2-s1 kernel: Lustre: oak-OST004e: Client 82effa21-5729-1828-1fbf-0c6e6269342f (at 10.51.0.63@o2ib3) reconnecting
May 09 09:26:29 oak-io2-s1 kernel: Lustre: oak-OST0044: Client 82effa21-5729-1828-1fbf-0c6e6269342f (at 10.51.0.63@o2ib3) reconnecting
May 09 09:26:29 oak-io2-s1 kernel: Lustre: Skipped 34 previous similar messages
May 09 09:26:29 oak-io2-s1 kernel: Lustre: Skipped 34 previous similar messages
May 09 09:26:29 oak-io2-s1 kernel: Lustre: Skipped 7 previous similar messages
May 09 09:26:33 oak-io2-s1 kernel: Lustre: oak-OST0032: Client b83c983f-3cc8-4 (at 10.51.5.46@o2ib3) reconnecting
May 09 09:26:33 oak-io2-s1 kernel: Lustre: Skipped 446 previous similar messages
May 09 09:26:42 oak-io2-s1 kernel: Lustre: oak-OST003e: Client eab9e2cf-af4c-4 (at 10.51.15.1@o2ib3) reconnecting
May 09 09:26:42 oak-io2-s1 kernel: Lustre: Skipped 422 previous similar messages
May 09 09:26:58 oak-io2-s1 kernel: Lustre: oak-OST0054: Client a703c65e-c48f-97d7-efaf-c377b3ded349 (at 10.51.15.3@o2ib3) reconnecting
May 09 09:26:58 oak-io2-s1 kernel: Lustre: Skipped 8 previous similar messages
May 09 09:28:14 oak-io2-s1 kernel: Lustre: oak-OST005c: Client ea3c0abc-03bf-9064-4b06-a2deb2e4ec48 (at 10.51.4.35@o2ib3) reconnecting
May 09 09:28:14 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 09 09:29:32 oak-io2-s1 kernel: Lustre: oak-OST0046: Client 4cee94e6-025c-589a-13ab-6c9ed337de31 (at 10.51.2.36@o2ib3) reconnecting
May 09 09:29:32 oak-io2-s1 kernel: Lustre: Skipped 40 previous similar messages
May 09 09:34:23 oak-io2-s1 kernel: Lustre: oak-OST0036: Connection restored to 68838616-d51a-4 (at 10.50.6.39@o2ib2)
May 09 09:34:23 oak-io2-s1 kernel: Lustre: Skipped 1481 previous similar messages
May 09 09:37:05 oak-io2-s1 kernel: Lustre: oak-OST005a: Client 2aacab37-4c74-421f-2f72-7f028b6417a2 (at 10.50.2.67@o2ib2) reconnecting
May 09 09:37:05 oak-io2-s1 kernel: Lustre: oak-OST0046: Client 2aacab37-4c74-421f-2f72-7f028b6417a2 (at 10.50.2.67@o2ib2) reconnecting
May 09 09:37:05 oak-io2-s1 kernel: Lustre: Skipped 7 previous similar messages
May 09 09:44:24 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to 6ab66e87-13d6-4 (at 10.50.17.26@o2ib2)
May 09 09:44:24 oak-io2-s1 kernel: Lustre: Skipped 407 previous similar messages
May 09 09:54:28 oak-io2-s1 kernel: Lustre: oak-OST0056: Connection restored to 68838616-d51a-4 (at 10.50.6.39@o2ib2)
May 09 09:54:28 oak-io2-s1 kernel: Lustre: Skipped 300 previous similar messages
May 09 10:04:32 oak-io2-s1 kernel: Lustre: oak-OST003e: Connection restored to bdc2eff6-717d-4 (at 10.51.2.42@o2ib3)
May 09 10:04:32 oak-io2-s1 kernel: Lustre: Skipped 281 previous similar messages
May 09 10:14:35 oak-io2-s1 kernel: Lustre: oak-OST0058: Connection restored to 0ee65775-62ad-4 (at 10.51.4.49@o2ib3)
May 09 10:14:35 oak-io2-s1 kernel: Lustre: Skipped 195 previous similar messages
May 09 10:24:49 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to 8d7d9634-680e-4 (at 10.50.7.15@o2ib2)
May 09 10:24:49 oak-io2-s1 kernel: Lustre: Skipped 199 previous similar messages
May 09 10:34:49 oak-io2-s1 kernel: Lustre: oak-OST004a: Connection restored to 69613e4f-9589-4 (at 10.50.14.12@o2ib2)
May 09 10:34:49 oak-io2-s1 kernel: Lustre: Skipped 176 previous similar messages
May 09 10:42:17 oak-io2-s1 kernel: Lustre: oak-OST0056: Client 2660ed0a-cc8b-ea86-ea7f-af6581531611 (at 10.50.5.41@o2ib2) reconnecting
May 09 10:42:17 oak-io2-s1 kernel: Lustre: Skipped 8 previous similar messages
May 09 10:44:53 oak-io2-s1 kernel: Lustre: oak-OST004c: Connection restored to 76836182-c1cf-4 (at 10.50.3.61@o2ib2)
May 09 10:44:53 oak-io2-s1 kernel: Lustre: Skipped 227 previous similar messages
May 09 10:54:55 oak-io2-s1 kernel: Lustre: oak-OST0056: Connection restored to 473ec31f-2480-4 (at 10.50.14.9@o2ib2)
May 09 10:54:55 oak-io2-s1 kernel: Lustre: Skipped 271 previous similar messages
May 09 11:05:04 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to 2ccb4e55-34e6-4 (at 10.50.9.50@o2ib2)
May 09 11:05:04 oak-io2-s1 kernel: Lustre: Skipped 233 previous similar messages
May 09 11:15:05 oak-io2-s1 kernel: Lustre: oak-OST005c: Connection restored to 76c973ce-bbbb-4 (at 10.49.28.5@o2ib1)
May 09 11:15:05 oak-io2-s1 kernel: Lustre: Skipped 203 previous similar messages
May 09 11:25:11 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to 602c1a8f-6064-4 (at 10.49.18.23@o2ib1)
May 09 11:25:11 oak-io2-s1 kernel: Lustre: Skipped 176 previous similar messages
May 09 11:35:13 oak-io2-s1 kernel: Lustre: oak-OST003c: Connection restored to bac61fc8-3c4a-4 (at 10.50.6.57@o2ib2)
May 09 11:35:13 oak-io2-s1 kernel: Lustre: Skipped 227 previous similar messages
May 09 11:45:14 oak-io2-s1 kernel: Lustre: oak-OST0038: Connection restored to cd9e9665-3ade-4 (at 10.50.17.23@o2ib2)
May 09 11:45:14 oak-io2-s1 kernel: Lustre: Skipped 212 previous similar messages
May 09 11:55:18 oak-io2-s1 kernel: Lustre: oak-OST0040: Connection restored to 6f1aab12-ec2f-4 (at 10.49.27.8@o2ib1)
May 09 11:55:18 oak-io2-s1 kernel: Lustre: Skipped 205 previous similar messages
May 09 11:55:36 oak-io2-s1 kernel: Lustre: oak-OST0052: Client 039ca6cd-2a8c-0a45-406c-a3de7c7ade6f (at 10.50.2.71@o2ib2) reconnecting
May 09 11:55:36 oak-io2-s1 kernel: Lustre: Skipped 12 previous similar messages
May 09 12:05:26 oak-io2-s1 kernel: Lustre: oak-OST0054: Connection restored to 36b00265-a377-4 (at 10.49.18.27@o2ib1)
May 09 12:05:26 oak-io2-s1 kernel: Lustre: Skipped 187 previous similar messages
May 09 12:15:29 oak-io2-s1 kernel: Lustre: oak-OST0056: Connection restored to a90e2d37-eb68-4 (at 10.50.7.19@o2ib2)
May 09 12:15:29 oak-io2-s1 kernel: Lustre: Skipped 140 previous similar messages
May 09 12:25:38 oak-io2-s1 kernel: Lustre: oak-OST0052: Connection restored to  (at 10.50.8.25@o2ib2)
May 09 12:25:38 oak-io2-s1 kernel: Lustre: Skipped 317 previous similar messages
May 09 12:35:41 oak-io2-s1 kernel: Lustre: oak-OST0058: Connection restored to 4054322f-6182-4 (at 10.50.17.30@o2ib2)
May 09 12:35:41 oak-io2-s1 kernel: Lustre: Skipped 194 previous similar messages
May 09 12:45:42 oak-io2-s1 kernel: Lustre: oak-OST005e: Connection restored to c07d0863-218d-4 (at 10.51.4.30@o2ib3)
May 09 12:45:42 oak-io2-s1 kernel: Lustre: Skipped 226 previous similar messages
May 09 12:55:42 oak-io2-s1 kernel: Lustre: oak-OST0036: Connection restored to 63186c86-9795-4 (at 10.50.12.16@o2ib2)
May 09 12:55:42 oak-io2-s1 kernel: Lustre: Skipped 245 previous similar messages
May 09 13:05:42 oak-io2-s1 kernel: Lustre: oak-OST005c: Connection restored to 8a251957-e6c2-4 (at 10.50.9.62@o2ib2)
May 09 13:05:42 oak-io2-s1 kernel: Lustre: Skipped 321 previous similar messages
May 09 13:15:43 oak-io2-s1 kernel: Lustre: oak-OST0044: Connection restored to d5017e03-4385-4 (at 10.50.6.67@o2ib2)
May 09 13:15:43 oak-io2-s1 kernel: Lustre: Skipped 441 previous similar messages
May 09 13:25:46 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to bc8be7f7-9a50-4 (at 10.49.18.14@o2ib1)
May 09 13:25:46 oak-io2-s1 kernel: Lustre: Skipped 340 previous similar messages
May 09 13:35:51 oak-io2-s1 kernel: Lustre: oak-OST0040: Connection restored to 8a251957-e6c2-4 (at 10.50.9.62@o2ib2)
May 09 13:35:51 oak-io2-s1 kernel: Lustre: Skipped 263 previous similar messages
May 09 13:45:54 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to 6d42de8f-4aab-4 (at 10.50.7.52@o2ib2)
May 09 13:45:54 oak-io2-s1 kernel: Lustre: Skipped 239 previous similar messages
May 09 13:56:03 oak-io2-s1 kernel: Lustre: oak-OST004e: Connection restored to bc22dc9e-c3bf-4 (at 10.51.4.44@o2ib3)
May 09 13:56:03 oak-io2-s1 kernel: Lustre: Skipped 238 previous similar messages
May 09 14:06:03 oak-io2-s1 kernel: Lustre: oak-OST003e: Connection restored to d7e00b58-c1b4-4 (at 10.50.7.32@o2ib2)
May 09 14:06:03 oak-io2-s1 kernel: Lustre: Skipped 167 previous similar messages
May 09 14:16:09 oak-io2-s1 kernel: Lustre: oak-OST005c: Connection restored to 6f1aab12-ec2f-4 (at 10.49.27.8@o2ib1)
May 09 14:16:09 oak-io2-s1 kernel: Lustre: Skipped 106 previous similar messages
May 09 14:26:11 oak-io2-s1 kernel: Lustre: oak-OST0046: Connection restored to baef2c84-1e07-4 (at 10.50.17.3@o2ib2)
May 09 14:26:11 oak-io2-s1 kernel: Lustre: Skipped 189 previous similar messages
May 09 14:36:11 oak-io2-s1 kernel: Lustre: oak-OST0036: Connection restored to 0726856a-9987-4 (at 10.50.7.14@o2ib2)
May 09 14:36:11 oak-io2-s1 kernel: Lustre: Skipped 208 previous similar messages
May 09 14:46:14 oak-io2-s1 kernel: Lustre: oak-OST004c: Connection restored to 64d080ce-52f0-4 (at 10.50.9.31@o2ib2)
May 09 14:46:14 oak-io2-s1 kernel: Lustre: Skipped 244 previous similar messages
May 09 14:56:14 oak-io2-s1 kernel: Lustre: oak-OST0056: Connection restored to  (at 10.50.7.34@o2ib2)
May 09 14:56:14 oak-io2-s1 kernel: Lustre: Skipped 261 previous similar messages
May 09 15:06:15 oak-io2-s1 kernel: Lustre: oak-OST003c: Connection restored to b1686e50-92e8-4 (at 10.49.26.33@o2ib1)
May 09 15:06:15 oak-io2-s1 kernel: Lustre: Skipped 258 previous similar messages
May 09 15:16:23 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to cfbdda79-1410-4 (at 10.49.28.7@o2ib1)
May 09 15:16:23 oak-io2-s1 kernel: Lustre: Skipped 243 previous similar messages
May 09 15:26:33 oak-io2-s1 kernel: Lustre: oak-OST0054: Connection restored to  (at 10.51.5.13@o2ib3)
May 09 15:26:33 oak-io2-s1 kernel: Lustre: Skipped 130 previous similar messages
May 09 15:29:01 oak-io2-s1 kernel: Lustre: oak-OST004a: Client 9488f889-3d86-6397-835c-2d56c219fc30 (at 10.50.17.11@o2ib2) reconnecting
May 09 15:29:01 oak-io2-s1 kernel: Lustre: Skipped 7 previous similar messages
May 09 15:36:34 oak-io2-s1 kernel: Lustre: oak-OST004c: Connection restored to 73359ebf-3387-4 (at 10.50.3.66@o2ib2)
May 09 15:36:34 oak-io2-s1 kernel: Lustre: Skipped 115 previous similar messages
May 09 15:46:37 oak-io2-s1 kernel: Lustre: oak-OST0048: Connection restored to 4eea7dc0-9e6d-4 (at 10.49.18.18@o2ib1)
May 09 15:46:37 oak-io2-s1 kernel: Lustre: Skipped 260 previous similar messages
May 09 15:56:38 oak-io2-s1 kernel: Lustre: oak-OST003e: Connection restored to 5a1327cd-5da5-4 (at 10.50.13.3@o2ib2)
May 09 15:56:38 oak-io2-s1 kernel: Lustre: Skipped 152 previous similar messages
May 09 16:06:42 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to 44847e14-ddb1-4 (at 10.50.17.36@o2ib2)
May 09 16:06:42 oak-io2-s1 kernel: Lustre: Skipped 231 previous similar messages
May 09 16:12:41 oak-io2-s1 kernel: Lustre: oak-OST0034: Client 266154da-cbea-4 (at 10.51.15.19@o2ib3) reconnecting
May 09 16:12:41 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
May 09 16:12:43 oak-io2-s1 kernel: Lustre: oak-OST005c: Client 62b89a52-cd86-4 (at 10.51.6.17@o2ib3) reconnecting
May 09 16:12:43 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 09 16:16:56 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to 18d47612-74b2-4 (at 10.50.6.58@o2ib2)
May 09 16:16:56 oak-io2-s1 kernel: Lustre: Skipped 122 previous similar messages
May 09 16:27:06 oak-io2-s1 kernel: Lustre: oak-OST003e: Connection restored to  (at 10.50.3.71@o2ib2)
May 09 16:27:06 oak-io2-s1 kernel: Lustre: Skipped 134 previous similar messages
May 09 16:37:08 oak-io2-s1 kernel: Lustre: oak-OST0054: Connection restored to 791e6f5b-af4c-4 (at 10.49.27.22@o2ib1)
May 09 16:37:08 oak-io2-s1 kernel: Lustre: Skipped 252 previous similar messages
May 09 16:40:40 oak-io2-s1 kernel: Lustre: oak-OST003a: Client e44e1572-e5d3-83e6-d20d-7af7ffb9c967 (at 10.50.4.4@o2ib2) reconnecting
May 09 16:40:41 oak-io2-s1 kernel: Lustre: oak-OST0052: Client da3befd2-46e4-c270-255b-72a37aa976cf (at 10.50.1.49@o2ib2) reconnecting
May 09 16:40:42 oak-io2-s1 kernel: Lustre: oak-OST0048: Client 4fe39366-d539-b2d4-b932-3c940eec4381 (at 10.50.3.49@o2ib2) reconnecting
May 09 16:40:42 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
May 09 16:40:44 oak-io2-s1 kernel: Lustre: oak-OST005a: Client d688f313-d136-dbef-857a-ea1e7f28fc6b (at 10.50.1.70@o2ib2) reconnecting
May 09 16:40:44 oak-io2-s1 kernel: Lustre: Skipped 11 previous similar messages
May 09 16:47:10 oak-io2-s1 kernel: Lustre: oak-OST005e: Connection restored to 36b00265-a377-4 (at 10.49.18.27@o2ib1)
May 09 16:47:10 oak-io2-s1 kernel: Lustre: Skipped 177 previous similar messages
May 09 16:51:43 oak-io2-s1 kernel: Lustre: oak-OST004c: Client 66bf6793-ef6e-d6d2-96ad-3432d26adbce (at 10.51.13.5@o2ib3) reconnecting
May 09 16:51:43 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 09 16:51:43 oak-io2-s1 kernel: Lustre: oak-OST0032: Client 7e3f34f1-e0b1-6de7-4a42-e4ccca504809 (at 10.51.2.47@o2ib3) reconnecting
May 09 16:51:43 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 09 16:52:38 oak-io2-s1 kernel: Lustre: oak-OST004a: Client 2cb5b80e-8bdb-e5df-69d7-53749fbd86ff (at 10.51.12.8@o2ib3) reconnecting
May 09 16:52:38 oak-io2-s1 kernel: Lustre: Skipped 3 previous similar messages
May 09 16:52:41 oak-io2-s1 kernel: Lustre: oak-OST005c: Client 3472eb9f-bad9-4a8c-dea2-0106c2dd3fee (at 10.51.1.58@o2ib3) reconnecting
May 09 16:52:41 oak-io2-s1 kernel: Lustre: Skipped 3 previous similar messages
May 09 16:54:30 oak-io2-s1 kernel: Lustre: oak-OST0032: Client d507c462-9f0f-5857-aa2a-29a15c595cfc (at 10.51.6.7@o2ib3) reconnecting
May 09 16:54:30 oak-io2-s1 kernel: Lustre: oak-OST0050: Client d507c462-9f0f-5857-aa2a-29a15c595cfc (at 10.51.6.7@o2ib3) reconnecting
May 09 16:54:30 oak-io2-s1 kernel: Lustre: oak-OST0040: Client d507c462-9f0f-5857-aa2a-29a15c595cfc (at 10.51.6.7@o2ib3) reconnecting
May 09 16:54:30 oak-io2-s1 kernel: Lustre: Skipped 6 previous similar messages
May 09 16:54:30 oak-io2-s1 kernel: Lustre: Skipped 6 previous similar messages
May 09 16:54:30 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 09 16:57:14 oak-io2-s1 kernel: Lustre: oak-OST0050: Connection restored to  (at 10.51.13.7@o2ib3)
May 09 16:57:14 oak-io2-s1 kernel: Lustre: Skipped 265 previous similar messages
May 09 17:07:15 oak-io2-s1 kernel: Lustre: oak-OST003e: Connection restored to bc8be7f7-9a50-4 (at 10.49.18.14@o2ib1)
May 09 17:07:15 oak-io2-s1 kernel: Lustre: Skipped 156 previous similar messages
May 09 17:14:28 oak-io2-s1 kernel: Lustre: oak-OST0052: Client f71d1188-d01c-9fa0-935c-a63ad652da8a (at 10.51.12.21@o2ib3) reconnecting
May 09 17:15:04 oak-io2-s1 kernel: Lustre: oak-OST003e: Client 6cbff61b-9898-4 (at 10.51.13.9@o2ib3) reconnecting
May 09 17:15:04 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 09 17:17:21 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to 40fafb1b-2372-4 (at 10.50.9.71@o2ib2)
May 09 17:17:21 oak-io2-s1 kernel: Lustre: Skipped 152 previous similar messages
May 09 17:17:52 oak-io2-s1 kernel: Lustre: oak-OST0042: Client b77266a5-0ea9-ac34-d92d-29be9e2ccdd7 (at 10.51.5.3@o2ib3) reconnecting
May 09 17:20:33 oak-io2-s1 kernel: Lustre: oak-OST004e: Client 60fdba1c-732a-b99c-63c8-be091395f5f0 (at 10.51.12.5@o2ib3) reconnecting
May 09 17:27:21 oak-io2-s1 kernel: Lustre: oak-OST0050: Connection restored to bc8be7f7-9a50-4 (at 10.49.18.14@o2ib1)
May 09 17:27:21 oak-io2-s1 kernel: Lustre: Skipped 230 previous similar messages
May 09 17:37:28 oak-io2-s1 kernel: Lustre: oak-OST005a: Connection restored to 0726856a-9987-4 (at 10.50.7.14@o2ib2)
May 09 17:37:28 oak-io2-s1 kernel: Lustre: Skipped 319 previous similar messages
May 09 17:47:30 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to e1d4c1dc-2a26-4 (at 10.50.6.60@o2ib2)
May 09 17:47:30 oak-io2-s1 kernel: Lustre: Skipped 221 previous similar messages
May 09 17:48:38 oak-io2-s1 kernel: Lustre: oak-OST005a: Client fab123b0-1ddb-1087-562d-2215869e7c07 (at 10.210.15.140@tcp1) reconnecting
May 09 17:48:38 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 09 17:49:03 oak-io2-s1 kernel: Lustre: oak-OST0030: Client fab123b0-1ddb-1087-562d-2215869e7c07 (at 10.210.15.140@tcp1) reconnecting
May 09 17:49:03 oak-io2-s1 kernel: Lustre: Skipped 11 previous similar messages
May 09 17:57:31 oak-io2-s1 kernel: Lustre: oak-OST0038: Connection restored to 96dd8de7-f5eb-4 (at 10.49.27.24@o2ib1)
May 09 17:57:31 oak-io2-s1 kernel: Lustre: Skipped 176 previous similar messages
May 09 18:07:32 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to 0726856a-9987-4 (at 10.50.7.14@o2ib2)
May 09 18:07:32 oak-io2-s1 kernel: Lustre: Skipped 292 previous similar messages
May 09 18:17:41 oak-io2-s1 kernel: Lustre: oak-OST0036: Connection restored to 76c973ce-bbbb-4 (at 10.49.28.5@o2ib1)
May 09 18:17:41 oak-io2-s1 kernel: Lustre: Skipped 137 previous similar messages
May 09 18:27:47 oak-io2-s1 kernel: Lustre: oak-OST0056: Connection restored to 44847e14-ddb1-4 (at 10.50.17.36@o2ib2)
May 09 18:27:47 oak-io2-s1 kernel: Lustre: Skipped 94 previous similar messages
May 09 18:38:00 oak-io2-s1 kernel: Lustre: oak-OST0040: Connection restored to 012f4751-9ad6-4 (at 10.49.18.22@o2ib1)
May 09 18:38:00 oak-io2-s1 kernel: Lustre: Skipped 86 previous similar messages
May 09 18:48:13 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to 1794bb67-724a-4 (at 10.49.27.7@o2ib1)
May 09 18:48:13 oak-io2-s1 kernel: Lustre: Skipped 141 previous similar messages
May 09 18:58:25 oak-io2-s1 kernel: Lustre: oak-OST0040: Connection restored to bc8be7f7-9a50-4 (at 10.49.18.14@o2ib1)
May 09 18:58:25 oak-io2-s1 kernel: Lustre: Skipped 77 previous similar messages
May 09 19:03:11 oak-io2-s1 kernel: Lustre: oak-OST0030: Client afee7f25-5fbd-3c3c-3fd2-3eaa33534147 (at 10.210.15.136@tcp1) reconnecting
May 09 19:03:11 oak-io2-s1 kernel: Lustre: Skipped 9 previous similar messages
May 09 19:03:12 oak-io2-s1 kernel: Lustre: oak-OST005e: Client afee7f25-5fbd-3c3c-3fd2-3eaa33534147 (at 10.210.15.136@tcp1) reconnecting
May 09 19:03:12 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 09 19:08:26 oak-io2-s1 kernel: Lustre: oak-OST005c: Connection restored to 791e6f5b-af4c-4 (at 10.49.27.22@o2ib1)
May 09 19:08:26 oak-io2-s1 kernel: Lustre: Skipped 174 previous similar messages
May 09 19:09:46 oak-io2-s1 kernel: Lustre: oak-OST0030: Client 682755c0-8d5e-7596-b8fc-e7fdcce119e9 (at 10.210.15.138@tcp1) reconnecting
May 09 19:09:46 oak-io2-s1 kernel: Lustre: Skipped 5 previous similar messages
May 09 19:09:51 oak-io2-s1 kernel: Lustre: oak-OST003c: Client 682755c0-8d5e-7596-b8fc-e7fdcce119e9 (at 10.210.15.138@tcp1) reconnecting
May 09 19:09:51 oak-io2-s1 kernel: Lustre: Skipped 5 previous similar messages
May 09 19:12:24 oak-io2-s1 kernel: Lustre: oak-OST0040: Client afee7f25-5fbd-3c3c-3fd2-3eaa33534147 (at 10.210.15.136@tcp1) reconnecting
May 09 19:12:24 oak-io2-s1 kernel: Lustre: Skipped 11 previous similar messages
May 09 19:12:25 oak-io2-s1 kernel: LustreError: 137-5: oak-OST0045_UUID: not available for connect from 10.210.15.138@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
May 09 19:12:25 oak-io2-s1 kernel: LustreError: Skipped 3 previous similar messages
May 09 19:13:34 oak-io2-s1 kernel: Lustre: oak-OST0040: Client 682755c0-8d5e-7596-b8fc-e7fdcce119e9 (at 10.210.15.138@tcp1) reconnecting
May 09 19:13:34 oak-io2-s1 kernel: Lustre: Skipped 5 previous similar messages
May 09 19:13:59 oak-io2-s1 kernel: Lustre: oak-OST003a: Client 3a334b6c-a17d-55a7-b5dc-37821133b96e (at 10.210.15.139@tcp1) reconnecting
May 09 19:13:59 oak-io2-s1 kernel: Lustre: Skipped 4 previous similar messages
May 09 19:17:37 oak-io2-s1 kernel: Lustre: oak-OST0030: Client a1953a9b-2c21-8a85-b7f5-99f164a19569 (at 10.210.15.133@tcp1) reconnecting
May 09 19:17:37 oak-io2-s1 kernel: Lustre: Skipped 19 previous similar messages
May 09 19:18:31 oak-io2-s1 kernel: LustreError: 137-5: oak-OST0043_UUID: not available for connect from 10.210.15.135@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
May 09 19:18:32 oak-io2-s1 kernel: Lustre: oak-OST0050: Connection restored to 9b1109f5-bdcd-4 (at 10.49.27.10@o2ib1)
May 09 19:18:32 oak-io2-s1 kernel: Lustre: Skipped 154 previous similar messages
May 09 19:18:56 oak-io2-s1 kernel: Lustre: oak-OST003c: Client a678c91c-d9e5-577a-b48b-e82be95b85aa (at 10.210.15.135@tcp1) reconnecting
May 09 19:18:56 oak-io2-s1 kernel: Lustre: Skipped 17 previous similar messages
May 09 19:21:39 oak-io2-s1 kernel: Lustre: oak-OST005a: Client 0ccc6a02-5901-c129-3cba-2d5230a4c9ad (at 10.51.12.22@o2ib3) reconnecting
May 09 19:21:39 oak-io2-s1 kernel: Lustre: Skipped 63 previous similar messages
May 09 19:27:16 oak-io2-s1 kernel: Lustre: oak-OST0030: Client e759629a-ca06-c63e-e05d-bdc65491d8bb (at 10.210.15.137@tcp1) reconnecting
May 09 19:27:16 oak-io2-s1 kernel: Lustre: Skipped 32 previous similar messages
May 09 19:28:38 oak-io2-s1 kernel: Lustre: oak-OST0054: Connection restored to 602c1a8f-6064-4 (at 10.49.18.23@o2ib1)
May 09 19:28:38 oak-io2-s1 kernel: Lustre: Skipped 222 previous similar messages
May 09 19:38:38 oak-io2-s1 kernel: Lustre: oak-OST0040: Connection restored to  (at 10.50.3.68@o2ib2)
May 09 19:38:38 oak-io2-s1 kernel: Lustre: Skipped 150 previous similar messages
May 09 19:39:19 oak-io2-s1 kernel: Lustre: oak-OST0034: Client de6d1df3-e5d8-837d-1f58-ceda43babf5c (at 10.210.15.134@tcp1) reconnecting
May 09 19:39:19 oak-io2-s1 kernel: Lustre: Skipped 65 previous similar messages
May 09 19:48:44 oak-io2-s1 kernel: Lustre: oak-OST0046: Connection restored to f35b4cd6-c6c1-4 (at 10.50.13.15@o2ib2)
May 09 19:48:44 oak-io2-s1 kernel: Lustre: Skipped 202 previous similar messages
May 09 19:58:29 oak-io2-s1 kernel: Lustre: oak-OST0048: Client afee7f25-5fbd-3c3c-3fd2-3eaa33534147 (at 10.210.15.136@tcp1) reconnecting
May 09 19:58:29 oak-io2-s1 kernel: Lustre: Skipped 56 previous similar messages
May 09 19:58:54 oak-io2-s1 kernel: Lustre: oak-OST0038: Connection restored to afee7f25-5fbd-3c3c-3fd2-3eaa33534147 (at 10.210.15.136@tcp1)
May 09 19:58:54 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to afee7f25-5fbd-3c3c-3fd2-3eaa33534147 (at 10.210.15.136@tcp1)
May 09 19:58:54 oak-io2-s1 kernel: Lustre: Skipped 186 previous similar messages
May 09 20:08:55 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to 485e19fa-600a-4 (at 10.49.27.2@o2ib1)
May 09 20:08:55 oak-io2-s1 kernel: Lustre: Skipped 138 previous similar messages
May 09 20:18:19 oak-io2-s1 kernel: Lustre: oak-OST0030: Client a1953a9b-2c21-8a85-b7f5-99f164a19569 (at 10.210.15.133@tcp1) reconnecting
May 09 20:18:19 oak-io2-s1 kernel: Lustre: Skipped 30 previous similar messages
May 09 20:18:44 oak-io2-s1 kernel: LustreError: 137-5: oak-OST0033_UUID: not available for connect from 10.210.15.137@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
May 09 20:18:44 oak-io2-s1 kernel: LustreError: Skipped 4 previous similar messages
May 09 20:18:56 oak-io2-s1 kernel: Lustre: oak-OST0050: Connection restored to  (at 10.50.10.19@o2ib2)
May 09 20:18:56 oak-io2-s1 kernel: Lustre: Skipped 177 previous similar messages
May 09 20:28:58 oak-io2-s1 kernel: Lustre: oak-OST003e: Connection restored to 485e19fa-600a-4 (at 10.49.27.2@o2ib1)
May 09 20:28:58 oak-io2-s1 kernel: Lustre: Skipped 164 previous similar messages
May 09 20:31:07 oak-io2-s1 kernel: Lustre: oak-OST003a: Client de6d1df3-e5d8-837d-1f58-ceda43babf5c (at 10.210.15.134@tcp1) reconnecting
May 09 20:31:07 oak-io2-s1 kernel: Lustre: Skipped 61 previous similar messages
May 09 20:32:02 oak-io2-s1 kernel: ses 1:0:183:0: attempting task abort! scmd(ffff8bc9ad889880)
May 09 20:32:02 oak-io2-s1 kernel: ses 1:0:183:0: [sg184] tag#1 CDB: Receive Diagnostic 1c 01 07 ff ff 00
May 09 20:32:02 oak-io2-s1 kernel: scsi target1:0:183: _scsih_tm_display_info: handle(0x00ca), sas_address(0x5001636001ab917d), phy(76)
May 09 20:32:02 oak-io2-s1 kernel: scsi target1:0:183: enclosurelogical id(0x5001636001ab917d), slot(60) 
May 09 20:32:02 oak-io2-s1 kernel: scsi target1:0:183: enclosure level(0x0001), connector name(     )
May 09 20:32:02 oak-io2-s1 kernel: ses 1:0:183:0: task abort: SUCCESS scmd(ffff8bc9ad889880)
May 09 20:39:08 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to 2baba63a-d33e-4 (at 10.49.8.33@o2ib1)
May 09 20:39:08 oak-io2-s1 kernel: Lustre: Skipped 97 previous similar messages
May 09 20:49:09 oak-io2-s1 kernel: Lustre: oak-OST005c: Connection restored to 68838616-d51a-4 (at 10.50.6.39@o2ib2)
May 09 20:49:09 oak-io2-s1 kernel: Lustre: Skipped 77 previous similar messages
May 09 20:49:28 oak-io2-s1 kernel: Lustre: oak-OST0046: Client de6d1df3-e5d8-837d-1f58-ceda43babf5c (at 10.210.15.134@tcp1) reconnecting
May 09 20:49:28 oak-io2-s1 kernel: Lustre: oak-OST0042: Client de6d1df3-e5d8-837d-1f58-ceda43babf5c (at 10.210.15.134@tcp1) reconnecting
May 09 20:49:28 oak-io2-s1 kernel: Lustre: Skipped 14 previous similar messages
May 09 20:49:28 oak-io2-s1 kernel: Lustre: Skipped 10 previous similar messages
May 09 20:59:09 oak-io2-s1 kernel: Lustre: oak-OST0058: Connection restored to 76ecdc90-9d7b-4 (at 10.50.10.72@o2ib2)
May 09 20:59:09 oak-io2-s1 kernel: Lustre: Skipped 122 previous similar messages
May 09 21:05:12 oak-io2-s1 kernel: Lustre: oak-OST0046: Client a678c91c-d9e5-577a-b48b-e82be95b85aa (at 10.210.15.135@tcp1) reconnecting
May 09 21:05:12 oak-io2-s1 kernel: Lustre: Skipped 11 previous similar messages
May 09 21:09:23 oak-io2-s1 kernel: Lustre: oak-OST0038: Connection restored to 9b1109f5-bdcd-4 (at 10.49.27.10@o2ib1)
May 09 21:09:23 oak-io2-s1 kernel: Lustre: Skipped 223 previous similar messages
May 09 21:15:36 oak-io2-s1 kernel: Lustre: oak-OST0048: Client e759629a-ca06-c63e-e05d-bdc65491d8bb (at 10.210.15.137@tcp1) reconnecting
May 09 21:15:36 oak-io2-s1 kernel: Lustre: Skipped 36 previous similar messages
May 09 21:19:28 oak-io2-s1 kernel: Lustre: oak-OST0052: Connection restored to be5087a8-f6b2-4 (at 10.50.3.48@o2ib2)
May 09 21:19:28 oak-io2-s1 kernel: Lustre: Skipped 128 previous similar messages
May 09 21:29:29 oak-io2-s1 kernel: Lustre: oak-OST0046: Connection restored to 71e1dc67-e312-4 (at 10.51.4.5@o2ib3)
May 09 21:29:29 oak-io2-s1 kernel: Lustre: Skipped 124 previous similar messages
May 09 21:33:32 oak-io2-s1 kernel: Lustre: oak-OST0030: Client fab123b0-1ddb-1087-562d-2215869e7c07 (at 10.210.15.140@tcp1) reconnecting
May 09 21:33:32 oak-io2-s1 kernel: Lustre: Skipped 28 previous similar messages
May 09 21:39:44 oak-io2-s1 kernel: Lustre: oak-OST004c: Connection restored to 1794bb67-724a-4 (at 10.49.27.7@o2ib1)
May 09 21:39:44 oak-io2-s1 kernel: Lustre: Skipped 192 previous similar messages
May 09 21:49:46 oak-io2-s1 kernel: Lustre: oak-OST0058: Connection restored to  (at 10.51.3.65@o2ib3)
May 09 21:49:46 oak-io2-s1 kernel: Lustre: Skipped 147 previous similar messages
May 09 21:59:46 oak-io2-s1 kernel: Lustre: oak-OST0058: Connection restored to 73359ebf-3387-4 (at 10.50.3.66@o2ib2)
May 09 21:59:46 oak-io2-s1 kernel: Lustre: Skipped 246 previous similar messages
May 09 22:00:55 oak-io2-s1 kernel: Lustre: oak-OST004a: Client 0ccc6a02-5901-c129-3cba-2d5230a4c9ad (at 10.51.12.22@o2ib3) reconnecting
May 09 22:00:55 oak-io2-s1 kernel: Lustre: Skipped 18 previous similar messages
May 09 22:02:21 oak-io2-s1 kernel: Lustre: oak-OST004c: Client afee7f25-5fbd-3c3c-3fd2-3eaa33534147 (at 10.210.15.136@tcp1) reconnecting
May 09 22:02:21 oak-io2-s1 kernel: Lustre: Skipped 10 previous similar messages
May 09 22:09:48 oak-io2-s1 kernel: Lustre: oak-OST005c: Connection restored to b290c20f-399c-4 (at 10.51.5.68@o2ib3)
May 09 22:09:48 oak-io2-s1 kernel: Lustre: Skipped 1472 previous similar messages
May 09 22:19:50 oak-io2-s1 kernel: Lustre: oak-OST005a: Connection restored to  (at 10.51.2.4@o2ib3)
May 09 22:19:50 oak-io2-s1 kernel: Lustre: Skipped 741 previous similar messages
May 09 22:29:50 oak-io2-s1 kernel: Lustre: oak-OST005a: Connection restored to 934d532f-1b5d-4 (at 10.51.4.50@o2ib3)
May 09 22:29:50 oak-io2-s1 kernel: Lustre: Skipped 1624 previous similar messages
May 09 22:39:50 oak-io2-s1 kernel: Lustre: oak-OST004a: Connection restored to 99090ff6-5c4e-4 (at 10.51.5.40@o2ib3)
May 09 22:39:50 oak-io2-s1 kernel: Lustre: Skipped 2514 previous similar messages
May 09 22:49:51 oak-io2-s1 kernel: Lustre: oak-OST0040: Connection restored to 934d532f-1b5d-4 (at 10.51.4.50@o2ib3)
May 09 22:49:51 oak-io2-s1 kernel: Lustre: Skipped 2520 previous similar messages
May 09 22:59:52 oak-io2-s1 kernel: Lustre: oak-OST004c: Connection restored to 4f6b7bfb-8e36-4 (at 10.51.14.18@o2ib3)
May 09 22:59:52 oak-io2-s1 kernel: Lustre: Skipped 2489 previous similar messages
May 09 23:09:53 oak-io2-s1 kernel: Lustre: oak-OST0036: Connection restored to 4f6b7bfb-8e36-4 (at 10.51.14.18@o2ib3)
May 09 23:09:53 oak-io2-s1 kernel: Lustre: Skipped 2615 previous similar messages
May 09 23:19:53 oak-io2-s1 kernel: Lustre: oak-OST0054: Connection restored to 489ee6f9-e0d9-4 (at 10.50.14.15@o2ib2)
May 09 23:19:53 oak-io2-s1 kernel: Lustre: Skipped 771 previous similar messages
May 09 23:29:58 oak-io2-s1 kernel: Lustre: oak-OST003a: Connection restored to 794b32ac-6683-4 (at 10.50.10.70@o2ib2)
May 09 23:29:58 oak-io2-s1 kernel: Lustre: Skipped 276 previous similar messages
May 09 23:39:59 oak-io2-s1 kernel: Lustre: oak-OST0056: Connection restored to ed9e5a4f-72f7-4 (at 10.50.8.58@o2ib2)
May 09 23:39:59 oak-io2-s1 kernel: Lustre: Skipped 561 previous similar messages
May 09 23:42:02 oak-io2-s1 kernel: Lustre: oak-OST0044: Client a678c91c-d9e5-577a-b48b-e82be95b85aa (at 10.210.15.135@tcp1) reconnecting
May 09 23:42:02 oak-io2-s1 kernel: Lustre: Skipped 25 previous similar messages
May 09 23:42:30 oak-io2-s1 kernel: Lustre: oak-OST0036: Client 3a334b6c-a17d-55a7-b5dc-37821133b96e (at 10.210.15.139@tcp1) reconnecting
May 09 23:42:30 oak-io2-s1 kernel: Lustre: Skipped 4 previous similar messages
May 09 23:49:59 oak-io2-s1 kernel: Lustre: oak-OST0054: Connection restored to  (at 10.50.7.63@o2ib2)
May 09 23:49:59 oak-io2-s1 kernel: Lustre: Skipped 293 previous similar messages
May 10 00:00:00 oak-io2-s1 kernel: Lustre: oak-OST0048: Connection restored to 794b32ac-6683-4 (at 10.50.10.70@o2ib2)
May 10 00:00:00 oak-io2-s1 kernel: Lustre: Skipped 204 previous similar messages
May 10 00:07:48 oak-io2-s1 kernel: LustreError: 54968:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8be0d58dcc00
May 10 00:07:48 oak-io2-s1 kernel: LustreError: 54967:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8bd84d303c00
May 10 00:07:48 oak-io2-s1 kernel: LustreError: 54966:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8bd84d303c00
May 10 00:07:48 oak-io2-s1 kernel: LustreError: 54969:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8be0d58dcc00
May 10 00:07:48 oak-io2-s1 kernel: LustreError: 54968:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8bf5a32dec00
May 10 00:08:53 oak-io2-s1 kernel: LustreError: 218398:0:(ldlm_lib.c:3344:target_bulk_io()) @@@ network error on bulk READ  req@ffff8bdff554c850 x1698172291403584/t0(0) o3->a703c65e-c48f-97d7-efaf-c377b3ded349@10.51.15.3@o2ib3:410/0 lens 488/440 e 0 to 0 dl 1620630560 ref 1 fl Interpret:/0/0 rc 0/0
May 10 00:08:53 oak-io2-s1 kernel: Lustre: oak-OST0052: Bulk IO read error with a703c65e-c48f-97d7-efaf-c377b3ded349 (at 10.51.15.3@o2ib3), client will retry: rc -110
May 10 00:08:53 oak-io2-s1 kernel: Lustre: Skipped 73 previous similar messages
May 10 00:08:53 oak-io2-s1 kernel: LustreError: 218398:0:(ldlm_lib.c:3344:target_bulk_io()) Skipped 2 previous similar messages
May 10 00:10:01 oak-io2-s1 kernel: Lustre: oak-OST0042: Connection restored to 9b1109f5-bdcd-4 (at 10.49.27.10@o2ib1)
May 10 00:10:01 oak-io2-s1 kernel: Lustre: Skipped 162 previous similar messages
May 10 00:10:33 oak-io2-s1 kernel: Lustre: oak-OST005e: Client 341ff9d8-ce51-a34b-3b59-737651e19da4 (at 10.51.2.29@o2ib3) reconnecting
May 10 00:10:33 oak-io2-s1 kernel: Lustre: Skipped 7 previous similar messages
May 10 00:10:44 oak-io2-s1 kernel: Lustre: oak-OST0030: Client 25ca5655-1c3b-4 (at 10.51.15.8@o2ib3) reconnecting
May 10 00:10:44 oak-io2-s1 kernel: Lustre: Skipped 8 previous similar messages
May 10 00:20:04 oak-io2-s1 kernel: Lustre: oak-OST0036: Connection restored to 12d63d61-be36-4 (at 10.50.9.16@o2ib2)
May 10 00:20:04 oak-io2-s1 kernel: Lustre: Skipped 173 previous similar messages
May 10 00:30:05 oak-io2-s1 kernel: Lustre: oak-OST0052: Connection restored to 58f7955b-7510-4 (at 10.50.10.13@o2ib2)
May 10 00:30:05 oak-io2-s1 kernel: Lustre: Skipped 350 previous similar messages
May 10 00:35:53 oak-io2-s1 kernel: Lustre: oak-OST004e: Client fab123b0-1ddb-1087-562d-2215869e7c07 (at 10.210.15.140@tcp1) reconnecting
May 10 00:35:53 oak-io2-s1 kernel: Lustre: Skipped 4 previous similar messages
May 10 00:38:52 oak-io2-s1 kernel: Lustre: oak-OST0032: Client a678c91c-d9e5-577a-b48b-e82be95b85aa (at 10.210.15.135@tcp1) reconnecting
May 10 00:38:52 oak-io2-s1 kernel: Lustre: Skipped 7 previous similar messages
May 10 00:40:05 oak-io2-s1 kernel: Lustre: oak-OST0054: Connection restored to 12d63d61-be36-4 (at 10.50.9.16@o2ib2)
May 10 00:40:05 oak-io2-s1 kernel: Lustre: Skipped 443 previous similar messages
May 10 00:43:08 oak-io2-s1 kernel: Lustre: oak-OST003c: Client 32915785-19a9-8b9d-6d19-5ca86f5e9a64 (at 10.50.7.3@o2ib2) reconnecting
May 10 00:43:08 oak-io2-s1 kernel: Lustre: Skipped 3 previous similar messages
May 10 00:50:05 oak-io2-s1 kernel: Lustre: oak-OST003c: Connection restored to 71bdfe57-cd01-4 (at 10.49.28.8@o2ib1)
May 10 00:50:05 oak-io2-s1 kernel: Lustre: Skipped 464 previous similar messages
May 10 00:50:29 oak-io2-s1 kernel: Lustre: oak-OST0058: Client f2735854-71c0-ac54-7890-7cb7922301f0 (at 10.50.10.37@o2ib2) reconnecting
May 10 00:50:29 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 10 00:56:43 oak-io2-s1 kernel: Lustre: oak-OST004c: Client 6df17ae1-271d-55da-49d3-73f806e47db4 (at 10.50.5.49@o2ib2) reconnecting
May 10 00:56:43 oak-io2-s1 kernel: Lustre: Skipped 10 previous similar messages
May 10 01:00:09 oak-io2-s1 kernel: Lustre: oak-OST0036: Connection restored to cda32b8c-6875-4 (at 10.50.16.18@o2ib2)
May 10 01:00:09 oak-io2-s1 kernel: Lustre: Skipped 492 previous similar messages
May 10 01:10:08 oak-io2-s1 kernel: Lustre: oak-OST005c: Client eb8cea22-3545-c4b8-6cb6-b3e875ecfb11 (at 10.51.1.23@o2ib3) reconnecting
May 10 01:10:08 oak-io2-s1 kernel: Lustre: Skipped 4 previous similar messages
May 10 01:10:10 oak-io2-s1 kernel: Lustre: oak-OST0036: Connection restored to 22f75865-19b9-71a9-aa3b-548e62a806e3 (at 10.51.0.16@o2ib3)
May 10 01:10:10 oak-io2-s1 kernel: Lustre: Skipped 447 previous similar messages
May 10 01:20:13 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to 80873df8-db2a-4 (at 10.50.10.62@o2ib2)
May 10 01:20:13 oak-io2-s1 kernel: Lustre: Skipped 313 previous similar messages
May 10 01:20:48 oak-io2-s1 kernel: Lustre: oak-OST0034: Client de6d1df3-e5d8-837d-1f58-ceda43babf5c (at 10.210.15.134@tcp1) reconnecting
May 10 01:20:48 oak-io2-s1 kernel: Lustre: Skipped 13 previous similar messages
May 10 01:22:15 oak-io2-s1 kernel: Lustre: oak-OST0032: Client f634b433-71b1-a8da-58e1-d59537ef9a2c (at 10.50.4.57@o2ib2) reconnecting
May 10 01:30:14 oak-io2-s1 kernel: Lustre: oak-OST0058: Connection restored to 9f536579-bb94-4 (at 10.50.6.47@o2ib2)
May 10 01:30:14 oak-io2-s1 kernel: Lustre: Skipped 404 previous similar messages
May 10 01:40:15 oak-io2-s1 kernel: Lustre: oak-OST003a: Connection restored to  (at 10.50.15.6@o2ib2)
May 10 01:40:15 oak-io2-s1 kernel: Lustre: Skipped 277 previous similar messages
May 10 01:45:29 oak-io2-s1 kernel: Lustre: oak-OST005a: Client 07c2edba-254b-34b8-a31c-924990eaebcf (at 10.50.1.17@o2ib2) reconnecting
May 10 01:45:29 oak-io2-s1 kernel: Lustre: oak-OST0050: Client 9401cc4c-d0f4-90d4-6708-e5b279f85f92 (at 10.50.4.22@o2ib2) reconnecting
May 10 01:45:29 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 10 01:49:42 oak-io2-s1 kernel: Lustre: oak-OST0042: Client f4f31fbb-c316-9d0e-dea6-2a23d0a9a983 (at 10.51.1.12@o2ib3) reconnecting
May 10 01:50:17 oak-io2-s1 kernel: Lustre: oak-OST005e: Connection restored to 500d2488-869a-4 (at 10.50.9.3@o2ib2)
May 10 01:50:17 oak-io2-s1 kernel: Lustre: Skipped 277 previous similar messages
May 10 02:00:17 oak-io2-s1 kernel: Lustre: oak-OST0050: Connection restored to 6cb94e48-1445-4 (at 10.50.16.15@o2ib2)
May 10 02:00:17 oak-io2-s1 kernel: Lustre: Skipped 198 previous similar messages
May 10 02:03:05 oak-io2-s1 kernel: Lustre: oak-OST0030: Client d7a86ed6-5d65-4 (at 10.50.9.57@o2ib2) reconnecting
May 10 02:03:05 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 10 02:10:18 oak-io2-s1 kernel: Lustre: oak-OST0054: Connection restored to 9a280014-aa1b-4 (at 10.50.13.2@o2ib2)
May 10 02:10:18 oak-io2-s1 kernel: Lustre: Skipped 204 previous similar messages
May 10 02:13:05 oak-io2-s1 kernel: Lustre: oak-OST0032: Client 23393310-2844-97b0-9a60-ab9cde689ddb (at 10.51.2.17@o2ib3) reconnecting
May 10 02:17:21 oak-io2-s1 kernel: Lustre: oak-OST0052: Client afee7f25-5fbd-3c3c-3fd2-3eaa33534147 (at 10.210.15.136@tcp1) reconnecting
May 10 02:17:21 oak-io2-s1 kernel: Lustre: Skipped 3 previous similar messages
May 10 02:20:19 oak-io2-s1 kernel: Lustre: oak-OST0034: Connection restored to 76d7694c-c22d-4 (at 10.49.18.13@o2ib1)
May 10 02:20:19 oak-io2-s1 kernel: Lustre: Skipped 272 previous similar messages
May 10 02:30:23 oak-io2-s1 kernel: Lustre: oak-OST004e: Connection restored to e5015edd-09df-4 (at 10.50.7.7@o2ib2)
May 10 02:30:23 oak-io2-s1 kernel: Lustre: Skipped 192 previous similar messages
May 10 02:40:24 oak-io2-s1 kernel: Lustre: oak-OST0048: Connection restored to 36b00265-a377-4 (at 10.49.18.27@o2ib1)
May 10 02:40:24 oak-io2-s1 kernel: Lustre: Skipped 241 previous similar messages
May 10 02:50:25 oak-io2-s1 kernel: Lustre: oak-OST003a: Connection restored to 7eefd404-b3ca-4 (at 10.50.6.62@o2ib2)
May 10 02:50:25 oak-io2-s1 kernel: Lustre: Skipped 371 previous similar messages
May 10 03:00:26 oak-io2-s1 kernel: Lustre: oak-OST0046: Connection restored to 3aa00c0e-4c49-4 (at 10.50.9.38@o2ib2)
May 10 03:00:26 oak-io2-s1 kernel: Lustre: Skipped 311 previous similar messages
May 10 03:10:31 oak-io2-s1 kernel: Lustre: oak-OST0052: Connection restored to a178498d-1f86-4 (at 10.50.10.66@o2ib2)
May 10 03:10:31 oak-io2-s1 kernel: Lustre: Skipped 612 previous similar messages
May 10 03:20:33 oak-io2-s1 kernel: Lustre: oak-OST0038: Connection restored to e29795f8-8bfe-4 (at 10.50.10.26@o2ib2)
May 10 03:20:33 oak-io2-s1 kernel: Lustre: Skipped 345 previous similar messages
May 10 03:23:16 oak-io2-s1 kernel: Lustre: oak-OST0056: Client a780244d-b3c6-4d7e-ae30-3e39b56d6962 (at 10.49.27.1@o2ib1) reconnecting
May 10 03:23:18 oak-io2-s1 kernel: Lustre: oak-OST004c: Client a780244d-b3c6-4d7e-ae30-3e39b56d6962 (at 10.49.27.1@o2ib1) reconnecting
May 10 03:30:33 oak-io2-s1 kernel: Lustre: oak-OST004e: Connection restored to  (at 10.51.2.9@o2ib3)
May 10 03:30:33 oak-io2-s1 kernel: Lustre: Skipped 579 previous similar messages
May 10 03:40:34 oak-io2-s1 kernel: Lustre: oak-OST0050: Connection restored to dc5e2566-c126-4 (at 10.50.10.51@o2ib2)
May 10 03:40:34 oak-io2-s1 kernel: Lustre: Skipped 420 previous similar messages
May 10 03:50:35 oak-io2-s1 kernel: Lustre: oak-OST0050: Connection restored to 3ac62d68-a7e6-4 (at 10.50.7.42@o2ib2)
May 10 03:50:35 oak-io2-s1 kernel: Lustre: Skipped 401 previous similar messages
May 10 04:00:36 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to 012f4751-9ad6-4 (at 10.49.18.22@o2ib1)
May 10 04:00:36 oak-io2-s1 kernel: Lustre: Skipped 405 previous similar messages
May 10 04:01:59 oak-io2-s1 kernel: Lustre: oak-OST0034: Client a1953a9b-2c21-8a85-b7f5-99f164a19569 (at 10.210.15.133@tcp1) reconnecting
May 10 04:01:59 oak-io2-s1 kernel: Lustre: Skipped 12 previous similar messages
May 10 04:02:25 oak-io2-s1 kernel: Lustre: oak-OST0030: Client a1953a9b-2c21-8a85-b7f5-99f164a19569 (at 10.210.15.133@tcp1) reconnecting
May 10 04:02:25 oak-io2-s1 kernel: Lustre: Skipped 5 previous similar messages
May 10 04:05:06 oak-io2-s1 kernel: Lustre: oak-OST0056: Client 3a334b6c-a17d-55a7-b5dc-37821133b96e (at 10.210.15.139@tcp1) reconnecting
May 10 04:05:06 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 10 04:10:40 oak-io2-s1 kernel: Lustre: oak-OST0050: Connection restored to 9e006fbc-e4ba-4 (at 10.50.17.27@o2ib2)
May 10 04:10:40 oak-io2-s1 kernel: Lustre: Skipped 354 previous similar messages
May 10 04:14:09 oak-io2-s1 kernel: Lustre: oak-OST003c: Client eab9e2cf-af4c-4 (at 10.51.15.1@o2ib3) reconnecting
May 10 04:15:16 oak-io2-s1 kernel: Lustre: oak-OST0056: Client 6a9de4e6-b748-6da3-5c58-3c2a8da7b38d (at 10.51.3.24@o2ib3) reconnecting
May 10 04:20:41 oak-io2-s1 kernel: Lustre: oak-OST004e: Connection restored to 0b13f2ae-d6e1-4 (at 10.51.2.70@o2ib3)
May 10 04:20:41 oak-io2-s1 kernel: Lustre: Skipped 490 previous similar messages
May 10 04:23:12 oak-io2-s1 kernel: Lustre: oak-OST004a: Client 8b66e4db-fcc9-6215-2cf9-86ed141f42d8 (at 10.51.2.26@o2ib3) reconnecting
May 10 04:23:12 oak-io2-s1 kernel: Lustre: Skipped 3 previous similar messages
May 10 04:30:41 oak-io2-s1 kernel: Lustre: oak-OST005c: Connection restored to  (at 10.50.10.19@o2ib2)
May 10 04:30:41 oak-io2-s1 kernel: Lustre: Skipped 374 previous similar messages
May 10 04:33:51 oak-io2-s1 kernel: Lustre: oak-OST0052: Client ba9adc4e-b7ab-586c-741b-469808595652 (at 10.50.4.68@o2ib2) reconnecting
May 10 04:40:45 oak-io2-s1 kernel: Lustre: oak-OST005c: Connection restored to eed83b08-729a-4 (at 10.49.18.19@o2ib1)
May 10 04:40:45 oak-io2-s1 kernel: Lustre: Skipped 467 previous similar messages
May 10 04:50:56 oak-io2-s1 kernel: Lustre: oak-OST0042: Connection restored to  (at 10.51.4.15@o2ib3)
May 10 04:50:56 oak-io2-s1 kernel: Lustre: Skipped 272 previous similar messages
May 10 04:56:14 oak-io2-s1 kernel: Lustre: oak-OST0048: Client 84772643-3e0c-a25c-a9b0-965c7b792170 (at 10.51.13.15@o2ib3) reconnecting
May 10 04:56:14 oak-io2-s1 kernel: Lustre: Skipped 3 previous similar messages
May 10 04:59:32 oak-io2-s1 kernel: Lustre: oak-OST0038: Client a703c65e-c48f-97d7-efaf-c377b3ded349 (at 10.51.15.3@o2ib3) reconnecting
May 10 04:59:32 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 10 05:00:22 oak-io2-s1 kernel: Lustre: oak-OST0058: Client f8da5f03-699b-e8b9-19dd-de6ca7bd30e7 (at 10.51.1.13@o2ib3) reconnecting
May 10 05:00:59 oak-io2-s1 kernel: Lustre: oak-OST003a: Connection restored to 76d7694c-c22d-4 (at 10.49.18.13@o2ib1)
May 10 05:00:59 oak-io2-s1 kernel: Lustre: Skipped 174 previous similar messages
May 10 05:10:59 oak-io2-s1 kernel: Lustre: oak-OST004c: Connection restored to 485e19fa-600a-4 (at 10.49.27.2@o2ib1)
May 10 05:10:59 oak-io2-s1 kernel: Lustre: Skipped 89 previous similar messages
May 10 05:11:39 oak-io2-s1 kernel: Lustre: oak-OST005e: Client 4cee94e6-025c-589a-13ab-6c9ed337de31 (at 10.51.2.36@o2ib3) reconnecting
May 10 05:11:39 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 10 05:18:44 oak-io2-s1 kernel: Lustre: oak-OST005c: Client 66bf6793-ef6e-d6d2-96ad-3432d26adbce (at 10.51.13.5@o2ib3) reconnecting
May 10 05:18:44 oak-io2-s1 kernel: Lustre: oak-OST003a: Client 66bf6793-ef6e-d6d2-96ad-3432d26adbce (at 10.51.13.5@o2ib3) reconnecting
May 10 05:18:44 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 10 05:20:06 oak-io2-s1 kernel: Lustre: oak-OST003c: Client 057d7d47-6e0c-f38f-eddf-48feb04705f1 (at 10.51.13.12@o2ib3) reconnecting
May 10 05:21:14 oak-io2-s1 kernel: Lustre: oak-OST005a: Connection restored to e0f69ef9-1d42-4 (at 10.49.8.30@o2ib1)
May 10 05:21:14 oak-io2-s1 kernel: Lustre: Skipped 171 previous similar messages
May 10 05:31:19 oak-io2-s1 kernel: Lustre: oak-OST0046: Connection restored to e0f69ef9-1d42-4 (at 10.49.8.30@o2ib1)
May 10 05:31:19 oak-io2-s1 kernel: Lustre: Skipped 118 previous similar messages
May 10 05:41:21 oak-io2-s1 kernel: Lustre: oak-OST0038: Connection restored to eed83b08-729a-4 (at 10.49.18.19@o2ib1)
May 10 05:41:21 oak-io2-s1 kernel: Lustre: Skipped 205 previous similar messages
May 10 05:44:52 oak-io2-s1 kernel: Lustre: oak-OST005e: Client 19dc2fec-d596-6864-12c4-9a2cd00f6cfa (at 10.51.15.7@o2ib3) reconnecting
May 10 05:44:52 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
May 10 05:46:32 oak-io2-s1 kernel: LustreError: 137-5: oak-OST0049_UUID: not available for connect from 10.51.15.7@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server.
May 10 05:46:57 oak-io2-s1 kernel: Lustre: oak-OST005e: Client 19dc2fec-d596-6864-12c4-9a2cd00f6cfa (at 10.51.15.7@o2ib3) reconnecting
May 10 05:46:57 oak-io2-s1 kernel: Lustre: Skipped 3 previous similar messages
May 10 05:49:33 oak-io2-s1 kernel: Lustre: oak-OST0030: haven't heard from client 68ef67a6-4c12-c839-550b-8737b418c4ec (at 10.49.18.29@o2ib1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff8bbce6c42c00, cur 1620650973 expire 1620650823 last 1620650746
May 10 05:51:25 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to 602c1a8f-6064-4 (at 10.49.18.23@o2ib1)
May 10 05:51:25 oak-io2-s1 kernel: Lustre: Skipped 235 previous similar messages
May 10 06:01:26 oak-io2-s1 kernel: Lustre: oak-OST0044: Connection restored to 9b1109f5-bdcd-4 (at 10.49.27.10@o2ib1)
May 10 06:01:26 oak-io2-s1 kernel: Lustre: Skipped 120 previous similar messages
May 10 06:03:07 oak-io2-s1 kernel: Lustre: oak-OST005e: Client f7461f98-ea9f-0924-372b-c5c0bdb871f7 (at 10.51.1.31@o2ib3) reconnecting
May 10 06:05:13 oak-io2-s1 kernel: Lustre: oak-OST005e: Client d63f6918-7cc4-821b-d43e-904b891ff51e (at 10.51.0.62@o2ib3) reconnecting
May 10 06:05:13 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
May 10 06:11:28 oak-io2-s1 kernel: Lustre: oak-OST0038: Connection restored to  (at 10.50.9.40@o2ib2)
May 10 06:11:28 oak-io2-s1 kernel: Lustre: Skipped 107 previous similar messages
May 10 06:21:29 oak-io2-s1 kernel: Lustre: oak-OST0058: Connection restored to 9b1109f5-bdcd-4 (at 10.49.27.10@o2ib1)
May 10 06:21:29 oak-io2-s1 kernel: Lustre: Skipped 81 previous similar messages
May 10 06:31:31 oak-io2-s1 kernel: Lustre: oak-OST0040: Connection restored to 485e19fa-600a-4 (at 10.49.27.2@o2ib1)
May 10 06:31:31 oak-io2-s1 kernel: Lustre: Skipped 146 previous similar messages
May 10 06:38:05 oak-io2-s1 kernel: Lustre: oak-OST0030: Client de6d1df3-e5d8-837d-1f58-ceda43babf5c (at 10.210.15.134@tcp1) reconnecting
May 10 06:38:05 oak-io2-s1 kernel: Lustre: oak-OST0032: Client de6d1df3-e5d8-837d-1f58-ceda43babf5c (at 10.210.15.134@tcp1) reconnecting
May 10 06:38:05 oak-io2-s1 kernel: Lustre: Skipped 10 previous similar messages
May 10 06:41:33 oak-io2-s1 kernel: Lustre: oak-OST003a: Connection restored to c374cba1-8633-4 (at 10.51.4.66@o2ib3)
May 10 06:41:33 oak-io2-s1 kernel: Lustre: Skipped 152 previous similar messages
May 10 06:51:34 oak-io2-s1 kernel: Lustre: oak-OST0040: Connection restored to 485e19fa-600a-4 (at 10.49.27.2@o2ib1)
May 10 06:51:34 oak-io2-s1 kernel: Lustre: Skipped 140 previous similar messages
May 10 07:01:34 oak-io2-s1 kernel: Lustre: oak-OST0040: Connection restored to bc8be7f7-9a50-4 (at 10.49.18.14@o2ib1)
May 10 07:01:34 oak-io2-s1 kernel: Lustre: Skipped 124 previous similar messages
May 10 07:11:44 oak-io2-s1 kernel: Lustre: oak-OST005e: Connection restored to 034cb74c-4ae2-4 (at 10.50.6.71@o2ib2)
May 10 07:11:44 oak-io2-s1 kernel: Lustre: Skipped 134 previous similar messages
May 10 07:21:49 oak-io2-s1 kernel: Lustre: oak-OST0058: Connection restored to 76c973ce-bbbb-4 (at 10.49.28.5@o2ib1)
May 10 07:21:49 oak-io2-s1 kernel: Lustre: Skipped 128 previous similar messages
May 10 07:25:16 oak-io2-s1 kernel: Lustre: oak-OST0030: Client e759629a-ca06-c63e-e05d-bdc65491d8bb (at 10.210.15.137@tcp1) reconnecting
May 10 07:25:16 oak-io2-s1 kernel: Lustre: Skipped 11 previous similar messages
May 10 07:31:54 oak-io2-s1 kernel: Lustre: oak-OST004e: Connection restored to 64d080ce-52f0-4 (at 10.50.9.31@o2ib2)
May 10 07:31:54 oak-io2-s1 kernel: Lustre: Skipped 180 previous similar messages
May 10 07:42:26 oak-io2-s1 kernel: Lustre: oak-OST005e: Connection restored to 2baba63a-d33e-4 (at 10.49.8.33@o2ib1)
May 10 07:42:26 oak-io2-s1 kernel: Lustre: Skipped 147 previous similar messages
May 10 07:52:26 oak-io2-s1 kernel: Lustre: oak-OST0044: Connection restored to 034cb74c-4ae2-4 (at 10.50.6.71@o2ib2)
May 10 07:52:26 oak-io2-s1 kernel: Lustre: Skipped 160 previous similar messages
May 10 08:02:26 oak-io2-s1 kernel: Lustre: oak-OST005a: Connection restored to 36b00265-a377-4 (at 10.49.18.27@o2ib1)
May 10 08:02:26 oak-io2-s1 kernel: Lustre: Skipped 176 previous similar messages
May 10 08:04:20 oak-io2-s1 kernel: Lustre: oak-OST005c: Client 62935bb7-73e1-04f7-3b52-0957a312c1ed (at 10.50.17.32@o2ib2) reconnecting
May 10 08:04:21 oak-io2-s1 kernel: Lustre: oak-OST005c: Client 956b66aa-9080-c813-1ca8-f1de8712f6b6 (at 10.50.5.3@o2ib2) reconnecting
May 10 08:04:21 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
May 10 08:06:08 oak-io2-s1 kernel: Lustre: oak-OST005e: Client 8d0a36b4-8848-1fae-670d-b656a1e97135 (at 10.51.0.67@o2ib3) reconnecting
May 10 08:07:07 oak-io2-s1 kernel: Lustre: oak-OST0052: Client fc15c6ac-f3bb-69b1-6aaa-7b0c5692b793 (at 10.50.2.10@o2ib2) reconnecting
May 10 08:07:07 oak-io2-s1 kernel: Lustre: oak-OST003e: Client fc15c6ac-f3bb-69b1-6aaa-7b0c5692b793 (at 10.50.2.10@o2ib2) reconnecting
May 10 08:07:07 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 10 08:12:38 oak-io2-s1 kernel: Lustre: oak-OST0046: Connection restored to 14eaca5b-4bd5-4 (at 10.49.18.30@o2ib1)
May 10 08:12:38 oak-io2-s1 kernel: Lustre: Skipped 153 previous similar messages
May 10 08:22:39 oak-io2-s1 kernel: Lustre: oak-OST003a: Connection restored to 4bf6b5eb-9e3f-4 (at 10.50.10.32@o2ib2)
May 10 08:22:39 oak-io2-s1 kernel: Lustre: Skipped 95 previous similar messages
May 10 08:32:40 oak-io2-s1 kernel: Lustre: oak-OST003a: Connection restored to 485e19fa-600a-4 (at 10.49.27.2@o2ib1)
May 10 08:32:40 oak-io2-s1 kernel: Lustre: Skipped 77 previous similar messages
May 10 08:38:27 oak-io2-s1 kernel: Lustre: oak-OST0032: Client cb2b26a5-88b2-1497-68ef-f7bec37e3c2d (at 10.50.2.21@o2ib2) reconnecting
May 10 08:42:41 oak-io2-s1 kernel: Lustre: oak-OST005c: Connection restored to  (at 10.51.4.13@o2ib3)
May 10 08:42:41 oak-io2-s1 kernel: Lustre: Skipped 113 previous similar messages
May 10 08:52:34 oak-io2-s1 kernel: Lustre: oak-OST0032: Client 4e7846b4-d3af-3eb0-574a-20943de34eaf (at 10.50.10.10@o2ib2) reconnecting
May 10 08:52:34 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
May 10 08:52:35 oak-io2-s1 kernel: Lustre: oak-OST004e: Client f92cc94b-ae2c-c095-d5a4-50b5a6ac55b5 (at 10.50.2.11@o2ib2) reconnecting
May 10 08:52:35 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 10 08:52:42 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to 034cb74c-4ae2-4 (at 10.50.6.71@o2ib2)
May 10 08:52:42 oak-io2-s1 kernel: Lustre: Skipped 146 previous similar messages
May 10 08:54:30 oak-io2-s1 kernel: LustreError: 137-5: oak-OST0031_UUID: not available for connect from 10.50.3.21@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
May 10 08:54:30 oak-io2-s1 kernel: LustreError: Skipped 23 previous similar messages
May 10 08:54:33 oak-io2-s1 kernel: LustreError: 137-5: oak-OST0033_UUID: not available for connect from 10.50.3.14@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
May 10 08:54:33 oak-io2-s1 kernel: LustreError: 137-5: oak-OST0031_UUID: not available for connect from 10.50.3.14@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
May 10 08:54:33 oak-io2-s1 kernel: LustreError: Skipped 1 previous similar message
May 10 08:54:33 oak-io2-s1 kernel: LustreError: Skipped 21 previous similar messages
May 10 08:54:42 oak-io2-s1 kernel: LustreError: 137-5: oak-OST0033_UUID: not available for connect from 10.50.3.23@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
May 10 08:54:42 oak-io2-s1 kernel: LustreError: 137-5: oak-OST0037_UUID: not available for connect from 10.50.3.23@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
May 10 08:54:42 oak-io2-s1 kernel: LustreError: 137-5: oak-OST0031_UUID: not available for connect from 10.50.3.23@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
May 10 08:54:42 oak-io2-s1 kernel: LustreError: Skipped 22 previous similar messages
May 10 08:54:42 oak-io2-s1 kernel: LustreError: Skipped 22 previous similar messages
May 10 08:54:42 oak-io2-s1 kernel: LustreError: Skipped 21 previous similar messages
May 10 08:54:50 oak-io2-s1 kernel: LustreError: 137-5: oak-OST0037_UUID: not available for connect from 10.50.3.1@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
May 10 08:54:50 oak-io2-s1 kernel: LustreError: 137-5: oak-OST0035_UUID: not available for connect from 10.50.3.1@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
May 10 08:54:50 oak-io2-s1 kernel: LustreError: Skipped 71 previous similar messages
May 10 08:54:50 oak-io2-s1 kernel: LustreError: Skipped 21 previous similar messages
May 10 08:55:14 oak-io2-s1 kernel: LustreError: 137-5: oak-OST0033_UUID: not available for connect from 10.50.3.5@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
May 10 08:55:14 oak-io2-s1 kernel: LustreError: Skipped 46 previous similar messages
May 10 09:02:44 oak-io2-s1 kernel: Lustre: oak-OST0054: Connection restored to eed83b08-729a-4 (at 10.49.18.19@o2ib1)
May 10 09:02:44 oak-io2-s1 kernel: Lustre: Skipped 691 previous similar messages
May 10 09:05:34 oak-io2-s1 kernel: Lustre: oak-OST0034: Client b8292f7b-b9c6-80c7-9466-f724c8589f36 (at 10.50.1.14@o2ib2) reconnecting
May 10 09:05:34 oak-io2-s1 kernel: Lustre: Skipped 5 previous similar messages
May 10 09:05:35 oak-io2-s1 kernel: Lustre: oak-OST004e: Client 8c18a895-4dbf-4411-c4a0-1f3cda182928 (at 10.50.4.6@o2ib2) reconnecting
May 10 09:05:35 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 10 09:12:51 oak-io2-s1 kernel: Lustre: oak-OST004e: Connection restored to 4eea7dc0-9e6d-4 (at 10.49.18.18@o2ib1)
May 10 09:12:51 oak-io2-s1 kernel: Lustre: Skipped 117 previous similar messages
May 10 09:22:51 oak-io2-s1 kernel: Lustre: oak-OST0046: Connection restored to 73e2a698-8548-4 (at 10.50.17.24@o2ib2)
May 10 09:22:51 oak-io2-s1 kernel: Lustre: Skipped 440 previous similar messages
May 10 09:31:10 oak-io2-s1 kernel: Lustre: oak-OST0058: Client d4be9a17-4e81-b3d1-f53d-21ecf82a80e0 (at 10.51.1.51@o2ib3) reconnecting
May 10 09:31:11 oak-io2-s1 kernel: Lustre: oak-OST005c: Client 22f75865-19b9-71a9-aa3b-548e62a806e3 (at 10.51.0.16@o2ib3) reconnecting
May 10 09:31:11 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 10 09:31:13 oak-io2-s1 kernel: Lustre: oak-OST0048: Client 5382169d-059c-4 (at 10.51.2.52@o2ib3) reconnecting
May 10 09:31:13 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 10 09:31:53 oak-io2-s1 kernel: Lustre: oak-OST004e: Client 9fb98453-c3f6-048b-74b8-00159087ac4b (at 10.50.8.34@o2ib2) reconnecting
May 10 09:32:51 oak-io2-s1 kernel: Lustre: oak-OST0052: Connection restored to 6bbc0d63-02d6-4 (at 10.50.10.71@o2ib2)
May 10 09:32:51 oak-io2-s1 kernel: Lustre: Skipped 2025 previous similar messages
May 10 09:33:35 oak-io2-s1 kernel: Lustre: oak-OST004e: Client be21abca-4d92-bbd5-3ccb-1ba191006e06 (at 10.50.6.17@o2ib2) reconnecting
May 10 09:33:35 oak-io2-s1 kernel: Lustre: Skipped 5 previous similar messages
May 10 09:34:41 oak-io2-s1 kernel: Lustre: oak-OST0056: Client 9c0fe87d-3aff-3c0f-a012-39c9311fc356 (at 10.50.2.64@o2ib2) reconnecting
May 10 09:34:41 oak-io2-s1 kernel: Lustre: Skipped 5 previous similar messages
May 10 09:36:08 oak-io2-s1 kernel: Lustre: oak-OST0056: Client 4e7846b4-d3af-3eb0-574a-20943de34eaf (at 10.50.10.10@o2ib2) reconnecting
May 10 09:36:08 oak-io2-s1 kernel: Lustre: Skipped 12 previous similar messages
May 10 09:36:23 oak-io2-s1 kernel: LustreError: 54964:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8bdb17207800
May 10 09:36:23 oak-io2-s1 kernel: LustreError: 54962:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8bbcf9534000
May 10 09:36:23 oak-io2-s1 kernel: LustreError: 54963:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8bbcf9534000
May 10 09:36:23 oak-io2-s1 kernel: LustreError: 54963:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8bbcf9534000
May 10 09:36:23 oak-io2-s1 kernel: LustreError: 54963:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8bbcf9534000
May 10 09:36:23 oak-io2-s1 kernel: LustreError: 54963:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8becbd5f3000
May 10 09:36:23 oak-io2-s1 kernel: LustreError: 54965:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8bdb17207800
May 10 09:37:11 oak-io2-s1 kernel: Lustre: oak-OST0030: Client af10ecff-434b-8ba9-f825-153886913b67 (at 10.50.10.18@o2ib2) reconnecting
May 10 09:37:11 oak-io2-s1 kernel: Lustre: Skipped 7 previous similar messages
May 10 09:37:13 oak-io2-s1 kernel: LustreError: 230917:0:(ldlm_lib.c:3344:target_bulk_io()) @@@ network error on bulk READ  req@ffff8bc94d488050 x1688209507385216/t0(0) o3->a4991fff-ead5-4@10.50.3.44@o2ib2:547/0 lens 488/440 e 0 to 0 dl 1620664672 ref 1 fl Interpret:/0/0 rc 0/0
May 10 09:37:13 oak-io2-s1 kernel: Lustre: oak-OST0052: Bulk IO read error with 96c56161-5d1b-6443-ec8a-caebdf9364d1 (at 10.50.6.53@o2ib2), client will retry: rc -110
May 10 09:37:13 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
May 10 09:37:13 oak-io2-s1 kernel: LustreError: 230917:0:(ldlm_lib.c:3344:target_bulk_io()) Skipped 2 previous similar messages
May 10 09:39:00 oak-io2-s1 kernel: Lustre: oak-OST0038: Client be0bfa3e-8c9d-2ed5-f297-38fed47c7d5c (at 10.50.10.17@o2ib2) reconnecting
May 10 09:39:00 oak-io2-s1 kernel: Lustre: Skipped 6 previous similar messages
May 10 09:41:50 oak-io2-s1 kernel: Lustre: oak-OST0040: Client 0ccc6a02-5901-c129-3cba-2d5230a4c9ad (at 10.51.12.22@o2ib3) reconnecting
May 10 09:41:50 oak-io2-s1 kernel: Lustre: oak-OST0054: Client 0ccc6a02-5901-c129-3cba-2d5230a4c9ad (at 10.51.12.22@o2ib3) reconnecting
May 10 09:41:50 oak-io2-s1 kernel: Lustre: Skipped 58 previous similar messages
May 10 09:41:50 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 10 09:42:52 oak-io2-s1 kernel: Lustre: oak-OST0032: Connection restored to a06ff198-d843-4 (at 10.51.13.17@o2ib3)
May 10 09:42:52 oak-io2-s1 kernel: Lustre: Skipped 1248 previous similar messages
May 10 09:43:28 oak-io2-s1 kernel: LustreError: 218828:0:(ldlm_lib.c:3353:target_bulk_io()) @@@ truncated bulk READ 3145728(4194304)  req@ffff8be39b61f050 x1698513026562816/t0(0) o3->f3c3cb5d-4fd1-5584-5b4c-093db428aef8@10.50.2.29@o2ib2:173/0 lens 488/440 e 0 to 0 dl 1620665053 ref 1 fl Interpret:/0/0 rc 0/0
May 10 09:43:28 oak-io2-s1 kernel: LustreError: 218828:0:(ldlm_lib.c:3353:target_bulk_io()) Skipped 68 previous similar messages
May 10 09:43:28 oak-io2-s1 kernel: Lustre: oak-OST005e: Bulk IO read error with f3c3cb5d-4fd1-5584-5b4c-093db428aef8 (at 10.50.2.29@o2ib2), client will retry: rc -110
May 10 09:43:28 oak-io2-s1 kernel: Lustre: Skipped 2 previous similar messages
May 10 09:51:24 oak-io2-s1 kernel: Lustre: oak-OST0044: Client 12b1f738-54c0-d451-a785-aa3d710fd8d0 (at 10.50.4.15@o2ib2) reconnecting
May 10 09:51:24 oak-io2-s1 kernel: Lustre: Skipped 90 previous similar messages
May 10 09:52:01 oak-io2-s1 kernel: LustreError: 232135:0:(ldlm_lib.c:3338:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8bdb050fb850 x1698515085721664/t0(0) o4->9450e990-6ab7-b003-8cd7-d28ca10ecce5@10.50.5.31@o2ib2:733/0 lens 488/448 e 0 to 0 dl 1620665613 ref 1 fl Interpret:/0/0 rc 0/0
May 10 09:52:01 oak-io2-s1 kernel: LustreError: 232135:0:(ldlm_lib.c:3338:target_bulk_io()) Skipped 1 previous similar message
May 10 09:52:01 oak-io2-s1 kernel: Lustre: oak-OST0038: Bulk IO write error with 9450e990-6ab7-b003-8cd7-d28ca10ecce5 (at 10.50.5.31@o2ib2), client will retry: rc = -110
May 10 09:52:01 oak-io2-s1 kernel: Lustre: Skipped 4 previous similar messages
May 10 09:52:54 oak-io2-s1 kernel: Lustre: oak-OST003e: Connection restored to 76c973ce-bbbb-4 (at 10.49.28.5@o2ib1)
May 10 09:52:54 oak-io2-s1 kernel: Lustre: Skipped 607 previous similar messages
May 10 09:53:03 oak-io2-s1 kernel: LustreError: 230927:0:(ldlm_lib.c:3353:target_bulk_io()) @@@ truncated bulk READ 3145728(4194304)  req@ffff8bdb08f74850 x1696623373843584/t0(0) o3->bdfa38aa-0b4b-9ae8-7177-71f369bf7d36@10.50.16.10@o2ib2:743/0 lens 488/440 e 0 to 0 dl 1620665623 ref 1 fl Interpret:/0/0 rc 0/0
May 10 09:53:03 oak-io2-s1 kernel: Lustre: oak-OST0058: Bulk IO read error with bdfa38aa-0b4b-9ae8-7177-71f369bf7d36 (at 10.50.16.10@o2ib2), client will retry: rc -110
May 10 10:00:08 oak-io2-s1 kernel: LustreError: 218132:0:(ldlm_lib.c:3353:target_bulk_io()) @@@ truncated bulk READ 3145728(4194304)  req@ffff8bfb67f00850 x1695899554090624/t0(0) o3->eab9e2cf-af4c-4@10.51.15.1@o2ib3:417/0 lens 488/440 e 0 to 0 dl 1620666052 ref 1 fl Interpret:/0/0 rc 0/0
May 10 10:00:08 oak-io2-s1 kernel: Lustre: oak-OST003e: Bulk IO read error with eab9e2cf-af4c-4 (at 10.51.15.1@o2ib3), client will retry: rc -110
May 10 10:00:08 oak-io2-s1 kernel: LustreError: 218132:0:(ldlm_lib.c:3353:target_bulk_io()) Skipped 1 previous similar message
May 10 10:02:07 oak-io2-s1 kernel: Lustre: oak-OST003e: Client eab9e2cf-af4c-4 (at 10.51.15.1@o2ib3) reconnecting
May 10 10:02:07 oak-io2-s1 kernel: Lustre: Skipped 156 previous similar messages
May 10 10:02:18 oak-io2-s1 kernel: LustreError: 137-5: oak-OST0053_UUID: not available for connect from 10.50.17.30@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
May 10 10:02:18 oak-io2-s1 kernel: LustreError: Skipped 23 previous similar messages
May 10 10:02:55 oak-io2-s1 kernel: Lustre: oak-OST0030: Connection restored to 7624d599-2451-4 (at 10.50.16.8@o2ib2)
May 10 10:02:55 oak-io2-s1 kernel: Lustre: Skipped 388 previous similar messages
May 10 10:03:02 oak-io2-s1 kernel: LustreError: 137-5: oak-OST0053_UUID: not available for connect from 10.50.8.20@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
May 10 10:03:11 oak-io2-s1 kernel: LustreError: 137-5: oak-OST0053_UUID: not available for connect from 10.50.6.40@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
May 10 10:07:13 oak-io2-s1 kernel: LustreError: 232137:0:(ldlm_lib.c:3353:target_bulk_io()) @@@ truncated bulk READ 0(4194304)  req@ffff8bc825a61850 x1698525547669888/t0(0) o3->4b74a5ff-d5b1-5838-1867-556233007c60@10.50.5.67@o2ib2:71/0 lens 488/440 e 0 to 0 dl 1620666461 ref 1 fl Interpret:/0/0 rc 0/0
May 10 10:07:13 oak-io2-s1 kernel: Lustre: oak-OST005c: Bulk IO read error with 9cccf31d-4a90-3071-0cc8-0a846f4a01d6 (at 10.50.4.29@o2ib2), client will retry: rc -110
May 10 10:07:13 oak-io2-s1 kernel: Lustre: Skipped 1 previous similar message
May 10 10:07:13 oak-io2-s1 kernel: LustreError: 232137:0:(ldlm_lib.c:3353:target_bulk_io()) Skipped 5 previous similar messages
May 10 10:08:11 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8be5342c9c00
May 10 10:08:11 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8be5342c9c00
May 10 10:08:11 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 3, status -103, desc ffff8bf2ce738400
May 10 10:08:11 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8be8978a0c00
May 10 10:08:11 oak-io2-s1 kernel: LustreError: 218404:0:(ldlm_lib.c:3344:target_bulk_io()) @@@ network error on bulk WRITE  req@ffff8be6d9255850 x1685134129569984/t0(0) o4->6f8a86f0-c2d6-4@10.51.1.42@o2ib3:194/0 lens 488/448 e 0 to 0 dl 1620666584 ref 1 fl Interpret:/0/0 rc 0/0
May 10 10:08:11 oak-io2-s1 kernel: Lustre: oak-OST0058: Bulk IO write error with 6f8a86f0-c2d6-4 (at 10.51.1.42@o2ib3), client will retry: rc = -110
May 10 10:08:11 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8be8978a0c00
May 10 10:08:11 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8bd34e44a400
May 10 10:08:11 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8bf2c64d0c00
May 10 10:09:19 oak-io2-s1 kernel: LustreError: 218129:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(2912274)  req@ffff8bf22ef57050 x1697709250158336/t0(0) o4->9013d991-28ee-172b-7f94-1611b1f321a6@10.51.1.14@o2ib3:193/0 lens 488/448 e 0 to 0 dl 1620666583 ref 1 fl Interpret:/0/0 rc 0/0
May 10 10:09:19 oak-io2-s1 kernel: LustreError: 218128:0:(ldlm_lib.c:3344:target_bulk_io()) @@@ network error on bulk READ  req@ffff8bf744e36050 x1688689737573312/t0(0) o3->8121ff1d-2ca1-4@10.51.4.22@o2ib3:193/0 lens 488/440 e 0 to 0 dl 1620666583 ref 1 fl Interpret:/0/0 rc 0/0
May 10 10:09:19 oak-io2-s1 kernel: LustreError: 218833:0:(ldlm_lib.c:3353:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff8bf744f26850 x1689657990922752/t0(0) o3->99c7dd7c-3db7-4@10.51.5.58@o2ib3:193/0 lens 488/440 e 0 to 0 dl 1620666583 ref 1 fl Interpret:/0/0 rc 0/0
May 10 10:09:19 oak-io2-s1 kernel: Lustre: oak-OST0040: Bulk IO read error with 4c32b5fb-e821-4 (at 10.51.2.65@o2ib3), client will retry: rc -110
May 10 10:09:19 oak-io2-s1 kernel: Lustre: Skipped 5 previous similar messages
May 10 10:09:19 oak-io2-s1 kernel: Lustre: oak-OST0058: Bulk IO write error with 6f8a86f0-c2d6-4 (at 10.51.1.42@o2ib3), client will retry: rc = -110
May 10 10:09:19 oak-io2-s1 kernel: LustreError: 218129:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) Skipped 13 previous similar messages
May 10 10:11:19 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8bddae79a800
May 10 10:11:19 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8bddae79a800
May 10 10:11:19 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8bdf2bc89400
May 10 10:11:20 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8bdf2bc89400
May 10 10:11:20 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8bdf298adc00
May 10 10:11:20 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8bddb0fee400
May 10 10:11:20 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8bddb0fee400
May 10 10:11:20 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8bf428016800
May 10 10:12:12 oak-io2-s1 kernel: Lustre: oak-OST005c: Client f36a27e2-f202-12fb-4235-55f4a8db5980 (at 10.50.7.2@o2ib2) reconnecting
May 10 10:12:12 oak-io2-s1 kernel: Lustre: Skipped 420 previous similar messages
May 10 10:12:14 oak-io2-s1 kernel: LustreError: 218824:0:(ldlm_lib.c:3353:target_bulk_io()) @@@ truncated bulk READ 3145728(4194304)  req@ffff8bf734fdd850 x1694716603167360/t0(0) o3->7d4930b0-1dcd-4@10.51.13.8@o2ib3:381/0 lens 488/440 e 0 to 0 dl 1620666771 ref 1 fl Interpret:/0/0 rc 0/0
May 10 10:12:14 oak-io2-s1 kernel: LustreError: 217622:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff8bf5a9286850 x1685134129848320/t0(0) o4->6f8a86f0-c2d6-4@10.51.1.42@o2ib3:382/0 lens 488/448 e 0 to 0 dl 1620666772 ref 1 fl Interpret:/0/0 rc 0/0
May 10 10:12:14 oak-io2-s1 kernel: LustreError: 217867:0:(ldlm_lib.c:3344:target_bulk_io()) @@@ network error on bulk READ  req@ffff8bf5646c9050 x1694716603167488/t0(0) o3->7d4930b0-1dcd-4@10.51.13.8@o2ib3:381/0 lens 488/440 e 0 to 0 dl 1620666771 ref 1 fl Interpret:/0/0 rc 0/0
May 10 10:12:14 oak-io2-s1 kernel: LustreError: 217867:0:(ldlm_lib.c:3344:target_bulk_io()) Skipped 3 previous similar messages
May 10 10:12:14 oak-io2-s1 kernel: Lustre: oak-OST004a: Bulk IO read error with f04c53ee-80b0-4 (at 10.51.4.23@o2ib3), client will retry: rc -110
May 10 10:12:14 oak-io2-s1 kernel: Lustre: Skipped 22 previous similar messages
May 10 10:12:14 oak-io2-s1 kernel: Lustre: oak-OST005c: Bulk IO write error with 6f8a86f0-c2d6-4 (at 10.51.1.42@o2ib3), client will retry: rc = -110
May 10 10:12:14 oak-io2-s1 kernel: Lustre: Skipped 9 previous similar messages
May 10 10:12:14 oak-io2-s1 kernel: LustreError: 218824:0:(ldlm_lib.c:3353:target_bulk_io()) Skipped 78 previous similar messages
May 10 10:12:14 oak-io2-s1 kernel: LustreError: 217829:0:(ldlm_lib.c:3338:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8bdb08f73850 x1698745605010304/t0(0) o4->00e98dec-0593-aca4-e0f7-37d65f7ef6d3@10.50.5.30@o2ib2:434/0 lens 488/448 e 0 to 0 dl 1620666824 ref 1 fl Interpret:/0/0 rc 0/0
May 10 10:13:15 oak-io2-s1 kernel: Lustre: oak-OST0036: Connection restored to  (at 10.50.16.1@o2ib2)
May 10 10:13:15 oak-io2-s1 kernel: Lustre: Skipped 686 previous similar messages
May 10 10:15:48 oak-io2-s1 kernel: Lustre: DEBUG MARKER: Mon May 10 10:15:48 2021
May 10 10:16:36 oak-io2-s1 kernel: LustreError: 137-5: oak-OST005f_UUID: not available for connect from 10.51.12.2@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server.
May 10 10:18:25 oak-io2-s1 kernel: LustreError: 137-5: oak-OST003f_UUID: not available for connect from 10.50.2.36@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
May 10 10:18:53 oak-io2-s1 kernel: LustreError: 217900:0:(ldlm_lib.c:3353:target_bulk_io()) @@@ truncated bulk READ 45056(1093632)  req@ffff8bf664d25850 x1698173686265664/t0(0) o3->a703c65e-c48f-97d7-efaf-c377b3ded349@10.51.15.3@o2ib3:28/0 lens 488/440 e 0 to 0 dl 1620667173 ref 1 fl Interpret:/0/0 rc 0/0
May 10 10:18:53 oak-io2-s1 kernel: Lustre: oak-OST0034: Bulk IO read error with a703c65e-c48f-97d7-efaf-c377b3ded349 (at 10.51.15.3@o2ib3), client will retry: rc -110
May 10 10:18:53 oak-io2-s1 kernel: Lustre: Skipped 65 previous similar messages
May 10 10:18:58 oak-io2-s1 kernel: LustreError: 217904:0:(ldlm_lib.c:3338:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8bec7770c850 x1697709260350656/t0(0) o4->9013d991-28ee-172b-7f94-1611b1f321a6@10.51.1.14@o2ib3:86/0 lens 488/448 e 0 to 0 dl 1620667231 ref 1 fl Interpret:/0/0 rc 0/0
May 10 10:18:58 oak-io2-s1 kernel: Lustre: oak-OST003e: Bulk IO write error with 9013d991-28ee-172b-7f94-1611b1f321a6 (at 10.51.1.14@o2ib3), client will retry: rc = -110
May 10 10:18:58 oak-io2-s1 kernel: Lustre: Skipped 6 previous similar messages
May 10 10:19:46 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 3, status -103, desc ffff8be73ec37400
May 10 10:19:46 oak-io2-s1 kernel: LustreError: 217702:0:(ldlm_lib.c:3344:target_bulk_io()) @@@ network error on bulk WRITE  req@ffff8bf235818850 x1685134131677248/t0(0) o4->6f8a86f0-c2d6-4@10.51.1.42@o2ib3:133/0 lens 488/448 e 0 to 0 dl 1620667278 ref 1 fl Interpret:/0/0 rc 0/0
May 10 10:19:46 oak-io2-s1 kernel: LustreError: 217702:0:(ldlm_lib.c:3344:target_bulk_io()) Skipped 4 previous similar messages
May 10 10:19:46 oak-io2-s1 kernel: Lustre: oak-OST0058: Bulk IO write error with 6f8a86f0-c2d6-4 (at 10.51.1.42@o2ib3), client will retry: rc = -110
May 10 10:19:46 oak-io2-s1 kernel: Lustre: Skipped 8 previous similar messages
May 10 10:20:33 oak-io2-s1 kernel: LustreError: 217870:0:(ldlm_lib.c:3353:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff8be044deb050 x1683988257377280/t0(0) o3->f04c53ee-80b0-4@10.51.4.23@o2ib3:133/0 lens 488/440 e 0 to 0 dl 1620667278 ref 1 fl Interpret:/0/0 rc 0/0
May 10 10:20:33 oak-io2-s1 kernel: Lustre: oak-OST0038: Bulk IO read error with f04c53ee-80b0-4 (at 10.51.4.23@o2ib3), client will retry: rc -110
May 10 10:20:33 oak-io2-s1 kernel: LustreError: 217870:0:(ldlm_lib.c:3353:target_bulk_io()) Skipped 2 previous similar messages
May 10 10:20:50 oak-io2-s1 kernel: LustreError: 218403:0:(ldlm_lib.c:3338:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff8bfcb8624850 x1698173689371072/t0(0) o3->a703c65e-c48f-97d7-efaf-c377b3ded349@10.51.15.3@o2ib3:136/0 lens 488/440 e 0 to 0 dl 1620667281 ref 1 fl Interpret:/0/0 rc 0/0
May 10 10:20:50 oak-io2-s1 kernel: LustreError: 218403:0:(ldlm_lib.c:3338:target_bulk_io()) Skipped 10 previous similar messages
May 10 10:20:58 oak-io2-s1 kernel: LustreError: 217645:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff8bf9b92aa850 x1685134131676544/t0(0) o4->6f8a86f0-c2d6-4@10.51.1.42@o2ib3:136/0 lens 488/448 e 0 to 0 dl 1620667281 ref 1 fl Interpret:/0/0 rc 0/0
May 10 10:20:58 oak-io2-s1 kernel: Lustre: oak-OST0058: Bulk IO write error with 6f8a86f0-c2d6-4 (at 10.51.1.42@o2ib3), client will retry: rc = -110
May 10 10:20:58 oak-io2-s1 kernel: LustreError: 217645:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) Skipped 9 previous similar messages
May 10 10:22:17 oak-io2-s1 kernel: Lustre: oak-OST0044: Client 671f1f99-e7d0-ca3d-b878-9ab1292a7a3d (at 10.50.17.39@o2ib2) reconnecting
May 10 10:22:17 oak-io2-s1 kernel: Lustre: Skipped 1013 previous similar messages
May 10 10:23:15 oak-io2-s1 kernel: Lustre: oak-OST0056: Connection restored to ae42a492-45b3-41a5-b54a-2ba5a4359c63 (at 10.50.4.28@o2ib2)
May 10 10:23:15 oak-io2-s1 kernel: Lustre: Skipped 1297 previous similar messages
May 10 10:23:34 oak-io2-s1 kernel: LustreError: 217640:0:(ldlm_lib.c:3338:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff8be37f5a8050 x1698745607116992/t0(0) o4->00e98dec-0593-aca4-e0f7-37d65f7ef6d3@10.50.5.30@o2ib2:353/0 lens 488/448 e 0 to 0 dl 1620667498 ref 1 fl Interpret:/0/0 rc 0/0
May 10 10:23:34 oak-io2-s1 kernel: Lustre: oak-OST0046: Bulk IO write error with 00e98dec-0593-aca4-e0f7-37d65f7ef6d3 (at 10.50.5.30@o2ib2), client will retry: rc = -110
May 10 10:23:34 oak-io2-s1 kernel: Lustre: Skipped 4 previous similar messages
May 10 10:23:54 oak-io2-s1 kernel: LustreError: 137-5: oak-OST0059_UUID: not available for connect from 10.50.5.70@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
May 10 10:24:18 oak-io2-s1 kernel: LustreError: 20690:0:(ldlm_lib.c:3353:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff8bdb01b8d050 x1698512547429568/t0(0) o3->8c617009-95d8-661d-4ed9-edc3824b123c@10.50.2.32@o2ib2:349/0 lens 488/440 e 0 to 0 dl 1620667494 ref 1 fl Interpret:/0/0 rc 0/0
May 10 10:24:18 oak-io2-s1 kernel: Lustre: oak-OST0048: Bulk IO read error with 8c617009-95d8-661d-4ed9-edc3824b123c (at 10.50.2.32@o2ib2), client will retry: rc -110
May 10 10:24:18 oak-io2-s1 kernel: Lustre: Skipped 18 previous similar messages
May 10 10:24:18 oak-io2-s1 kernel: LustreError: 20690:0:(ldlm_lib.c:3353:target_bulk_io()) Skipped 16 previous similar messages
May 10 10:24:51 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8be2d1754000
May 10 10:24:51 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8be2d1754000
May 10 10:24:51 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8bef4af00800
May 10 10:24:51 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8bef4af00800
May 10 10:24:51 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8bf40bee1000
May 10 10:24:51 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8be73ec33800
May 10 10:24:51 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff8be73ec33800
May 10 10:24:51 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 3, status -103, desc ffff8bddb43f0c00
May 10 10:24:51 oak-io2-s1 kernel: LustreError: 54947:0:(events.c:450:server_bulk_callback()) event type 3, status -103, desc ffff8bddb43f0c00
May 10 10:24:51 oak-io2-s1 kernel: LustreError: 218139:0:(ldlm_lib.c:3344:target_bulk_io()) @@@ network error on bulk WRITE  req@ffff8bec6d9be850 x1685134132326784/t0(0) o4->6f8a86f0-c2d6-4@10.51.1.42@o2ib3:438/0 lens 488/448 e 0 to 0 dl 1620667583 ref 1 fl Interpret:/0/0 rc 0/0
May 10 10:24:51 oak-io2-s1 kernel: Lustre: oak-OST0058: Bulk IO write error with 6f8a86f0-c2d6-4 (at 10.51.1.42@o2ib3), client will retry: rc = -110
May 10 10:24:55 oak-io2-s1 kernel: LustreError: 137-5: oak-OST004b_UUID: not available for connect from 10.50.16.10@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
May 10 10:25:58 oak-io2-s1 kernel: LustreError: 218404:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff8be3b1f85850 x1685134132327232/t0(0) o4->6f8a86f0-c2d6-4@10.51.1.42@o2ib3:438/0 lens 488/448 e 0 to 0 dl 1620667583 ref 1 fl Interpret:/0/0 rc 0/0
May 10 10:25:58 oak-io2-s1 kernel: LustreError: 218087:0:(ldlm_lib.c:3344:target_bulk_io()) @@@ network error on bulk READ  req@ffff8bf7409df850 x1699125515693056/t0(0) o3->825593db-0cf5-eb9f-468c-fdaee8cc73e4@10.51.13.6@o2ib3:438/0 lens 488/440 e 0 to 0 dl 1620667583 ref 1 fl Interpret:/0/0 rc 0/0
May 10 10:25:58 oak-io2-s1 kernel: Lustre: oak-OST0058: Bulk IO write error with 6f8a86f0-c2d6-4 (at 10.51.1.42@o2ib3), client will retry: rc = -110
May 10 10:25:58 oak-io2-s1 kernel: LustreError: 218404:0:(sec.c:2485:sptlrpc_svc_unwrap_bulk()) Skipped 5 previous similar messages
May 10 10:27:48 oak-io2-s1 kernel: LustreError: 218836:0:(ldlm_lib.c:3338:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff8be230e15050 x1698173703565504/t0(0) o3->a703c65e-c48f-97d7-efaf-c377b3ded349@10.51.15.3@o2ib3:617/0 lens 488/440 e 0 to 0 dl 1620667762 ref 1 fl Interpret:/0/0 rc 0/0
May 10 10:27:48 oak-io2-s1 kernel: LustreError: 218836:0:(ldlm_lib.c:3338:target_bulk_io()) Skipped 1 previous similar message
May 10 10:27:49 oak-io2-s1 kernel: LustreError: 54969:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff8bf40c757c00
May 10 10:29:18 oak-io2-s1 kernel: LustreError: 217693:0:(ldlm_lib.c:3353:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff8bf737814850 x1696623289904960/t0(0) o3->5fbc6dd0-2ef9-b4a9-8121-d2dcfb28fdfb@10.50.10.8@o2ib2:645/0 lens 488/440 e 0 to 0 dl 1620667790 ref 1 fl Interpret:/0/0 rc 0/0
May 10 10:29:18 oak-io2-s1 kernel: Lustre: oak-OST004c: Bulk IO read error with 5fbc6dd0-2ef9-b4a9-8121-d2dcfb28fdfb (at 10.50.10.8@o2ib2), client will retry: rc -110
May 10 10:29:18 oak-io2-s1 kernel: Lustre: Skipped 26 previous similar messages
May 10 10:29:18 oak-io2-s1 kernel: LustreError: 217693:0:(ldlm_lib.c:3353:target_bulk_io()) Skipped 17 previous similar messages