HOSTS ------------------------------------------------------------------------- dac-e-10 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-05-07 14:07:27 BST, end at Wed 2019-08-21 11:25:26 BST. -- Aug 20 20:14:37 dac-e-10 kernel: Lustre: fs1-OST006c: Connection restored to ea42bfba-6613-4dee-aa4b-19c1f0d400fe (at 10.47.21.38@o2ib1) Aug 20 20:14:37 dac-e-10 kernel: Lustre: Skipped 127 previous similar messages Aug 20 20:22:43 dac-e-10 kernel: Lustre: fs1-OST006e: Connection restored to 2a21b08a-665f-68a5-f629-a4c31d7cff10 (at 10.47.21.36@o2ib1) Aug 20 20:22:43 dac-e-10 kernel: Lustre: fs1-OST0071: Connection restored to a142f57f-3986-5d9a-f4ac-ea41013ab5b4 (at 10.47.21.34@o2ib1) Aug 20 20:22:43 dac-e-10 kernel: Lustre: fs1-OST006d: Connection restored to 448bebe0-3beb-da9f-3d47-dbc50659268e (at 10.47.21.33@o2ib1) Aug 20 20:22:43 dac-e-10 kernel: Lustre: Skipped 13 previous similar messages Aug 20 20:22:43 dac-e-10 kernel: Lustre: Skipped 13 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-13 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:45 BST, end at Wed 2019-08-21 11:25:26 BST. -- Aug 20 20:14:37 dac-e-13 kernel: Lustre: fs1-OST0097: Connection restored to 21ae9b96-1e86-cd59-3acd-9878c229c355 (at 10.47.21.35@o2ib1) Aug 20 20:14:37 dac-e-13 kernel: Lustre: Skipped 129 previous similar messages Aug 20 20:22:43 dac-e-13 kernel: Lustre: fs1-OST009b: Connection restored to a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) Aug 20 20:22:43 dac-e-13 kernel: Lustre: Skipped 12 previous similar messages Aug 20 20:22:45 dac-e-13 kernel: LustreError: 364505:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9a66c1ee8e00 Aug 20 20:22:45 dac-e-13 kernel: LustreError: 364497:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9a66c1ee8e00 Aug 20 20:22:45 dac-e-13 kernel: LustreError: 364499:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9a66c1ee8e00 Aug 20 20:22:45 dac-e-13 kernel: LustreError: 364498:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9a66c1ee8e00 Aug 20 20:22:45 dac-e-13 kernel: Lustre: fs1-OST009b: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:22:45 dac-e-13 kernel: LustreError: 393103:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff9a65f7f5f850 x1642414042283488/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:721/0 lens 488/440 e 0 to 0 dl 1566329006 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:22:45 dac-e-13 kernel: LustreError: 364500:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9a66c1ee8e00 Aug 20 20:22:45 dac-e-13 kernel: LustreError: 364497:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9a66c1ee8e00 Aug 20 20:22:45 dac-e-13 kernel: LustreError: 364499:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9a66c1ee8e00 Aug 20 20:22:45 dac-e-13 kernel: Lustre: fs1-OST009b: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:23:15 dac-e-13 kernel: Lustre: fs1-OST009b: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:23:15 dac-e-13 kernel: Lustre: fs1-OST009b: Connection restored to a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) Aug 20 20:23:15 dac-e-13 kernel: Lustre: Skipped 12 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-14 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:46 BST, end at Wed 2019-08-21 11:25:26 BST. -- Aug 20 20:14:37 dac-e-14 kernel: Lustre: fs1-OST00a7: Connection restored to a142f57f-3986-5d9a-f4ac-ea41013ab5b4 (at 10.47.21.34@o2ib1) Aug 20 20:14:37 dac-e-14 kernel: Lustre: Skipped 126 previous similar messages Aug 20 20:22:43 dac-e-14 kernel: Lustre: fs1-OST00a2: Connection restored to 2a21b08a-665f-68a5-f629-a4c31d7cff10 (at 10.47.21.36@o2ib1) Aug 20 20:22:43 dac-e-14 kernel: Lustre: Skipped 12 previous similar messages Aug 20 20:22:45 dac-e-14 kernel: LustreError: 83271:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9776d91ace00 Aug 20 20:22:45 dac-e-14 kernel: LustreError: 83270:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9776d91ace00 Aug 20 20:22:45 dac-e-14 kernel: LustreError: 83273:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9776d91ace00 Aug 20 20:22:45 dac-e-14 kernel: LustreError: 83272:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9776d91ace00 Aug 20 20:22:45 dac-e-14 kernel: LustreError: 83273:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9776d91ace00 Aug 20 20:22:45 dac-e-14 kernel: LustreError: 83278:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9776d91ace00 Aug 20 20:22:45 dac-e-14 kernel: LustreError: 83272:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9776d91ace00 Aug 20 20:22:45 dac-e-14 kernel: LustreError: 83271:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9776d91ace00 Aug 20 20:22:45 dac-e-14 kernel: LustreError: 83273:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9776d7e1d800 Aug 20 20:22:45 dac-e-14 kernel: LustreError: 83278:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9776d91ace00 Aug 20 20:22:45 dac-e-14 kernel: LustreError: 83272:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9776d91ace00 Aug 20 20:22:45 dac-e-14 kernel: LustreError: 83271:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9776d91ace00 Aug 20 20:22:45 dac-e-14 kernel: LustreError: 83273:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9776d91ace00 Aug 20 20:22:45 dac-e-14 kernel: LustreError: 83270:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9776d91ace00 Aug 20 20:22:45 dac-e-14 kernel: LustreError: 83272:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9776d91ace00 Aug 20 20:22:45 dac-e-14 kernel: LustreError: 83273:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9776d91ace00 Aug 20 20:22:45 dac-e-14 kernel: LustreError: 83271:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9776d91ace00 Aug 20 20:22:48 dac-e-14 kernel: Lustre: fs1-OST00a4: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:22:49 dac-e-14 kernel: LustreError: 440049:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff97764d257050 x1642414042282416/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:689/0 lens 488/440 e 0 to 0 dl 1566328974 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:22:49 dac-e-14 kernel: Lustre: fs1-OST00a4: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:22:49 dac-e-14 kernel: LNet: 16653:0:(o2iblnd.c:941:kiblnd_create_conn()) peer 10.47.21.31@o2ib1 - queue depth reduced from 128 to 63 to allow for qp creation Aug 20 20:22:49 dac-e-14 kernel: LNet: 16653:0:(o2iblnd.c:941:kiblnd_create_conn()) Skipped 19 previous similar messages Aug 20 20:22:49 dac-e-14 kernel: Lustre: fs1-OST009d: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:22:50 dac-e-14 kernel: LustreError: 440053:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff97764d254850 x1642414042283808/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:721/0 lens 488/440 e 0 to 0 dl 1566329006 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:22:50 dac-e-14 kernel: Lustre: fs1-OST009d: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:23:40 dac-e-14 kernel: LustreError: 440039:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 13631488(16777216) req@ffff97764d259850 x1642414042282416/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:744/0 lens 488/440 e 2 to 0 dl 1566329029 ref 1 fl Interpret:/2/0 rc 0/0 Aug 20 20:23:40 dac-e-14 kernel: Lustre: fs1-OST009d: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:23:40 dac-e-14 kernel: LustreError: 440039:0:(ldlm_lib.c:3268:target_bulk_io()) Skipped 1 previous similar message Aug 20 20:33:53 dac-e-14 kernel: Lustre: fs1-OST00a4: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:33:53 dac-e-14 kernel: Lustre: fs1-OST00a4: Connection restored to a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) Aug 20 20:33:53 dac-e-14 kernel: Lustre: Skipped 11 previous similar messages Aug 20 20:36:44 dac-e-14 kernel: Lustre: fs1-OST009d: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:36:44 dac-e-14 kernel: Lustre: fs1-OST009d: Connection restored to a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) Aug 20 20:38:02 dac-e-14 kernel: Lustre: fs1-OST00a4: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:38:02 dac-e-14 kernel: Lustre: fs1-OST00a4: Connection restored to a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) Aug 20 20:38:40 dac-e-14 kernel: LustreError: 440039:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 13631488(16777216) req@ffff97764d256050 x1642414050374416/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:133/0 lens 488/440 e 3 to 0 dl 1566329928 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:38:40 dac-e-14 kernel: Lustre: fs1-OST009d: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:38:40 dac-e-14 kernel: Lustre: Skipped 1 previous similar message Aug 20 20:38:49 dac-e-14 kernel: Lustre: fs1-OST009d: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:38:49 dac-e-14 kernel: Lustre: Skipped 1 previous similar message Aug 20 20:38:49 dac-e-14 kernel: Lustre: fs1-OST009d: Connection restored to a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) Aug 20 20:38:49 dac-e-14 kernel: Lustre: Skipped 1 previous similar message HOSTS ------------------------------------------------------------------------- dac-e-1 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-04-16 21:34:08 BST, end at Wed 2019-08-21 11:25:26 BST. -- Aug 20 20:14:37 dac-e-1 kernel: Lustre: fs1-OST0007: Connection restored to c77f4037-5413-c6a2-471e-396e3d2ca371 (at 10.47.21.36@o2ib1) Aug 20 20:14:37 dac-e-1 kernel: Lustre: Skipped 138 previous similar messages Aug 20 20:22:43 dac-e-1 kernel: Lustre: fs1-OST0004: Connection restored to 8fa1bd84-2b85-379b-0497-bc36974308e6 (at 10.47.21.31@o2ib1) Aug 20 20:22:43 dac-e-1 kernel: Lustre: Skipped 12 previous similar messages Aug 20 20:22:45 dac-e-1 kernel: LustreError: 186365:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8ad8e97db400 Aug 20 20:22:45 dac-e-1 kernel: LustreError: 186363:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8ad8e97db400 Aug 20 20:22:45 dac-e-1 kernel: LustreError: 186362:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8ad8e97db400 Aug 20 20:22:45 dac-e-1 kernel: LustreError: 186364:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8ad8e97db400 Aug 20 20:23:47 dac-e-1 kernel: LustreError: 146798:0:(ldlm_lib.c:3259:target_bulk_io()) @@@ network error on bulk READ req@ffff8ad8c3b90050 x1642414042283456/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:16/0 lens 488/440 e 2 to 0 dl 1566329056 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:23:47 dac-e-1 kernel: Lustre: fs1-OST0005: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:33:53 dac-e-1 kernel: Lustre: fs1-OST0005: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:33:53 dac-e-1 kernel: Lustre: fs1-OST0005: Connection restored to 8fa1bd84-2b85-379b-0497-bc36974308e6 (at 10.47.21.31@o2ib1) Aug 20 20:33:53 dac-e-1 kernel: Lustre: Skipped 12 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-15 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:46 BST, end at Wed 2019-08-21 11:25:26 BST. -- Aug 20 20:14:37 dac-e-15 kernel: Lustre: fs1-OST00b1: Connection restored to ea42bfba-6613-4dee-aa4b-19c1f0d400fe (at 10.47.21.38@o2ib1) Aug 20 20:14:37 dac-e-15 kernel: Lustre: fs1-OST00ae: Connection restored to 1ea7fed8-2cb0-2b1b-4bfc-d6d667225e5e (at 10.47.20.68@o2ib1) Aug 20 20:14:37 dac-e-15 kernel: Lustre: fs1-OST00af: Connection restored to ea42bfba-6613-4dee-aa4b-19c1f0d400fe (at 10.47.21.38@o2ib1) Aug 20 20:14:37 dac-e-15 kernel: Lustre: fs1-OST00ab: Connection restored to 0ebebeeb-b828-0337-48a2-2bd97ab9fa21 (at 10.47.21.37@o2ib1) Aug 20 20:14:37 dac-e-15 kernel: Lustre: fs1-OST00a8: Connection restored to ea42bfba-6613-4dee-aa4b-19c1f0d400fe (at 10.47.21.38@o2ib1) Aug 20 20:14:37 dac-e-15 kernel: Lustre: Skipped 131 previous similar messages Aug 20 20:14:37 dac-e-15 kernel: Lustre: Skipped 131 previous similar messages Aug 20 20:14:37 dac-e-15 kernel: Lustre: Skipped 131 previous similar messages Aug 20 20:14:37 dac-e-15 kernel: Lustre: Skipped 131 previous similar messages Aug 20 20:22:43 dac-e-15 kernel: Lustre: fs1-OST00b0: Connection restored to 3f94c4e7-d409-8966-0a4b-959655ee93b1 (at 10.47.21.32@o2ib1) Aug 20 20:22:43 dac-e-15 kernel: Lustre: fs1-OST00ae: Connection restored to ea42bfba-6613-4dee-aa4b-19c1f0d400fe (at 10.47.21.38@o2ib1) Aug 20 20:22:43 dac-e-15 kernel: Lustre: Skipped 8 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-12 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:45 BST, end at Wed 2019-08-21 11:25:26 BST. -- Aug 20 20:14:37 dac-e-12 kernel: Lustre: fs1-OST0089: Connection restored to a142f57f-3986-5d9a-f4ac-ea41013ab5b4 (at 10.47.21.34@o2ib1) Aug 20 20:14:37 dac-e-12 kernel: Lustre: Skipped 130 previous similar messages Aug 20 20:22:43 dac-e-12 kernel: Lustre: fs1-OST008f: Connection restored to a142f57f-3986-5d9a-f4ac-ea41013ab5b4 (at 10.47.21.34@o2ib1) Aug 20 20:22:43 dac-e-12 kernel: Lustre: Skipped 12 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-11 ------------------------------------------------------------------------------- -- Logs begin at Thu 2019-03-21 15:42:22 GMT, end at Wed 2019-08-21 11:25:26 BST. -- Aug 20 20:14:37 dac-e-11 kernel: Lustre: fs1-OST0083: Connection restored to 0ebebeeb-b828-0337-48a2-2bd97ab9fa21 (at 10.47.21.37@o2ib1) Aug 20 20:14:37 dac-e-11 kernel: Lustre: Skipped 130 previous similar messages Aug 20 20:22:35 dac-e-11 kernel: Lustre: fs1-OST007d: Connection restored to 1ea7fed8-2cb0-2b1b-4bfc-d6d667225e5e (at 10.47.20.68@o2ib1) Aug 20 20:22:35 dac-e-11 kernel: Lustre: Skipped 10 previous similar messages Aug 20 20:22:45 dac-e-11 kernel: LustreError: 363941:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9848f47cc400 Aug 20 20:22:45 dac-e-11 kernel: LustreError: 363939:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9848f47cc400 Aug 20 20:22:45 dac-e-11 kernel: Lustre: fs1-OST0081: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:22:45 dac-e-11 kernel: Lustre: fs1-OST0081: Connection restored to a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) Aug 20 20:22:45 dac-e-11 kernel: Lustre: Skipped 12 previous similar messages Aug 20 20:22:45 dac-e-11 kernel: LustreError: 393537:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff9848a086e050 x1642414042282656/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:689/0 lens 488/440 e 0 to 0 dl 1566328974 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:22:45 dac-e-11 kernel: Lustre: fs1-OST0081: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 HOSTS ------------------------------------------------------------------------- dac-e-16 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:46 BST, end at Wed 2019-08-21 11:25:26 BST. -- Aug 20 20:14:37 dac-e-16 kernel: Lustre: fs1-OST00ba: Connection restored to a142f57f-3986-5d9a-f4ac-ea41013ab5b4 (at 10.47.21.34@o2ib1) Aug 20 20:14:37 dac-e-16 kernel: Lustre: Skipped 129 previous similar messages Aug 20 20:22:43 dac-e-16 kernel: Lustre: fs1-OST00b7: Connection restored to a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) Aug 20 20:22:43 dac-e-16 kernel: Lustre: Skipped 12 previous similar messages Aug 20 20:22:45 dac-e-16 kernel: LustreError: 369887:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fa7af9dfa00 Aug 20 20:22:45 dac-e-16 kernel: LustreError: 369886:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fa7af9dfa00 Aug 20 20:22:45 dac-e-16 kernel: LustreError: 369888:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fa7af9dfa00 Aug 20 20:22:45 dac-e-16 kernel: LustreError: 369894:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fa7af9dfa00 Aug 20 20:22:45 dac-e-16 kernel: LustreError: 369889:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fa7a4d2d800 Aug 20 20:22:45 dac-e-16 kernel: LustreError: 369887:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fa7a4d2d800 Aug 20 20:22:45 dac-e-16 kernel: LustreError: 369886:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fa7a4d2d800 Aug 20 20:22:45 dac-e-16 kernel: LustreError: 369888:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fa7a4d2d800 Aug 20 20:22:45 dac-e-16 kernel: LustreError: 369886:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fa7a4d2d800 Aug 20 20:22:45 dac-e-16 kernel: LustreError: 369888:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fa7a4d2d800 Aug 20 20:22:45 dac-e-16 kernel: LustreError: 369894:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fa7a4d2d800 Aug 20 20:22:45 dac-e-16 kernel: LustreError: 369887:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fa7a4d2d800 Aug 20 20:22:45 dac-e-16 kernel: LustreError: 369889:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fa7a4d2d800 Aug 20 20:22:45 dac-e-16 kernel: LustreError: 369887:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fa7a4d2d800 Aug 20 20:23:15 dac-e-16 kernel: Lustre: fs1-OST00b6: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:23:15 dac-e-16 kernel: Lustre: fs1-OST00b6: Connection restored to a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) Aug 20 20:23:15 dac-e-16 kernel: Lustre: Skipped 12 previous similar messages Aug 20 20:23:16 dac-e-16 kernel: LustreError: 64812:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8fa7d19fb850 x1642414042283376/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:721/0 lens 488/440 e 0 to 0 dl 1566329006 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:23:16 dac-e-16 kernel: Lustre: fs1-OST00b6: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:23:42 dac-e-16 kernel: LustreError: 64811:0:(ldlm_lib.c:3259:target_bulk_io()) @@@ network error on bulk READ req@ffff8fa7d19e3050 x1642414042283136/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:746/0 lens 488/440 e 1 to 0 dl 1566329031 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:23:42 dac-e-16 kernel: LustreError: 403499:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 14680064(16777216) req@ffff8fa7d19f1850 x1642414042283872/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:746/0 lens 488/440 e 1 to 0 dl 1566329031 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:23:42 dac-e-16 kernel: Lustre: fs1-OST00b7: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:30:58 dac-e-16 kernel: Lustre: fs1-OST00b7: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:30:58 dac-e-16 kernel: Lustre: fs1-OST00b7: Connection restored to a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) Aug 20 20:31:21 dac-e-16 kernel: Lustre: fs1-OST00b6: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:32:02 dac-e-16 kernel: LustreError: 64856:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 14680064(16777216) req@ffff8fa7d19fc050 x1642414047878144/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:509/0 lens 488/440 e 0 to 0 dl 1566329549 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:32:02 dac-e-16 kernel: Lustre: fs1-OST00b7: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:32:02 dac-e-16 kernel: Lustre: Skipped 1 previous similar message Aug 20 20:33:35 dac-e-16 kernel: Lustre: fs1-OST00b7: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:33:35 dac-e-16 kernel: Lustre: fs1-OST00b7: Connection restored to a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) Aug 20 20:33:35 dac-e-16 kernel: Lustre: Skipped 1 previous similar message Aug 20 20:33:59 dac-e-16 kernel: Lustre: fs1-OST00b6: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:34:32 dac-e-16 kernel: LustreError: 64856:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 13631488(16777216) req@ffff8fa7d19fa050 x1642414048466720/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:657/0 lens 488/440 e 3 to 0 dl 1566329697 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:34:32 dac-e-16 kernel: Lustre: fs1-OST00b7: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:36:44 dac-e-16 kernel: Lustre: fs1-OST00b7: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:36:44 dac-e-16 kernel: Lustre: fs1-OST00b7: Connection restored to a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) Aug 20 20:36:44 dac-e-16 kernel: Lustre: Skipped 1 previous similar message Aug 20 20:37:31 dac-e-16 kernel: Lustre: fs1-OST00b8: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:37:52 dac-e-16 kernel: LustreError: 64856:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 11534336(16777216) req@ffff8fa7d19fc850 x1642414050083632/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:115/0 lens 488/440 e 0 to 0 dl 1566329910 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:37:52 dac-e-16 kernel: Lustre: fs1-OST00b7: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:38:03 dac-e-16 kernel: LustreError: 6080:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8fa7d19fa050 x1642414050387552/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:163/0 lens 488/440 e 0 to 0 dl 1566329958 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:38:03 dac-e-16 kernel: Lustre: fs1-OST00b8: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:38:31 dac-e-16 kernel: Lustre: fs1-OST00b7: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:38:31 dac-e-16 kernel: Lustre: Skipped 1 previous similar message Aug 20 20:38:56 dac-e-16 kernel: LustreError: 64856:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8fa7d19f6050 x1642414050387552/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:194/0 lens 488/440 e 0 to 0 dl 1566329989 ref 1 fl Interpret:/2/0 rc 0/0 Aug 20 20:38:56 dac-e-16 kernel: Lustre: fs1-OST00b8: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:39:27 dac-e-16 kernel: LustreError: 64856:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8fa7d19f0850 x1642414050404384/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:247/0 lens 488/440 e 0 to 0 dl 1566330042 ref 1 fl Interpret:/0/0 rc 0/0 HOSTS ------------------------------------------------------------------------- dac-e-24 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:47 BST, end at Wed 2019-08-21 11:25:26 BST. -- Aug 20 20:14:37 dac-e-24 kernel: Lustre: fs1-OST0116: Connection restored to 2a21b08a-665f-68a5-f629-a4c31d7cff10 (at 10.47.21.36@o2ib1) Aug 20 20:14:37 dac-e-24 kernel: Lustre: Skipped 128 previous similar messages Aug 20 20:22:43 dac-e-24 kernel: Lustre: fs1-OST011c: Connection restored to 3f94c4e7-d409-8966-0a4b-959655ee93b1 (at 10.47.21.32@o2ib1) Aug 20 20:22:43 dac-e-24 kernel: Lustre: Skipped 14 previous similar messages Aug 20 20:22:45 dac-e-24 kernel: LustreError: 27707:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88957e3b3e00 Aug 20 20:22:45 dac-e-24 kernel: LustreError: 27705:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88957e3b3e00 Aug 20 20:22:45 dac-e-24 kernel: LustreError: 27753:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88957e3b3e00 Aug 20 20:22:45 dac-e-24 kernel: LustreError: 27706:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88957e3b3e00 Aug 20 20:22:45 dac-e-24 kernel: LustreError: 27708:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8895abfc3800 Aug 20 20:22:45 dac-e-24 kernel: LustreError: 27707:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8895abfc3800 Aug 20 20:22:45 dac-e-24 kernel: LustreError: 27706:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8895abfc3800 Aug 20 20:22:45 dac-e-24 kernel: LustreError: 27708:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8895abfc3800 Aug 20 20:22:45 dac-e-24 kernel: LustreError: 27753:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8895aaab4200 Aug 20 20:22:45 dac-e-24 kernel: LustreError: 27753:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8895abfc3800 Aug 20 20:22:45 dac-e-24 kernel: LustreError: 27707:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8895abfc3800 Aug 20 20:22:45 dac-e-24 kernel: LustreError: 27706:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8895abfc3800 Aug 20 20:22:45 dac-e-24 kernel: LustreError: 27707:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8895aaab4200 Aug 20 20:22:45 dac-e-24 kernel: LustreError: 27753:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8895aaab4200 Aug 20 20:22:45 dac-e-24 kernel: LustreError: 27705:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8895aaab4200 Aug 20 20:22:49 dac-e-24 kernel: Lustre: fs1-OST011a: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:22:50 dac-e-24 kernel: Lustre: fs1-OST0114: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:22:50 dac-e-24 kernel: LustreError: 419065:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8895222ed050 x1642414042283280/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:721/0 lens 488/440 e 0 to 0 dl 1566329006 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:22:50 dac-e-24 kernel: Lustre: fs1-OST011a: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:22:50 dac-e-24 kernel: LustreError: 419065:0:(ldlm_lib.c:3253:target_bulk_io()) Skipped 1 previous similar message Aug 20 20:23:38 dac-e-24 kernel: LustreError: 419036:0:(ldlm_lib.c:3259:target_bulk_io()) @@@ network error on bulk READ req@ffff8895222ec050 x1642414042283232/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:746/0 lens 488/440 e 1 to 0 dl 1566329031 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:23:38 dac-e-24 kernel: Lustre: fs1-OST011f: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:23:38 dac-e-24 kernel: Lustre: Skipped 1 previous similar message Aug 20 20:24:03 dac-e-24 kernel: LustreError: 419213:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 8388608(16777216) req@ffff88951ab1c850 x1642414042283440/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:22/0 lens 488/440 e 2 to 0 dl 1566329062 ref 1 fl Interpret:/2/0 rc 0/0 Aug 20 20:24:03 dac-e-24 kernel: Lustre: fs1-OST011a: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:31:16 dac-e-24 kernel: Lustre: fs1-OST011a: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:31:16 dac-e-24 kernel: Lustre: fs1-OST011a: Connection restored to a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) Aug 20 20:31:16 dac-e-24 kernel: Lustre: Skipped 12 previous similar messages Aug 20 20:33:53 dac-e-24 kernel: Lustre: fs1-OST011f: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:33:53 dac-e-24 kernel: Lustre: fs1-OST011f: Connection restored to a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) HOSTS ------------------------------------------------------------------------- dac-e-17 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:47 BST, end at Wed 2019-08-21 11:25:26 BST. -- Aug 20 20:14:37 dac-e-17 kernel: Lustre: fs1-OST00c7: Connection restored to 6d4b3b17-aa0e-1cb4-a610-cacce42ddc94 (at 10.47.20.69@o2ib1) Aug 20 20:14:37 dac-e-17 kernel: Lustre: Skipped 130 previous similar messages Aug 20 20:22:43 dac-e-17 kernel: Lustre: fs1-OST00c0: Connection restored to 448bebe0-3beb-da9f-3d47-dbc50659268e (at 10.47.21.33@o2ib1) Aug 20 20:22:43 dac-e-17 kernel: Lustre: Skipped 10 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-18 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:46 BST, end at Wed 2019-08-21 11:25:26 BST. -- Aug 20 20:14:37 dac-e-18 kernel: Lustre: fs1-OST00d0: Connection restored to 448bebe0-3beb-da9f-3d47-dbc50659268e (at 10.47.21.33@o2ib1) Aug 20 20:14:37 dac-e-18 kernel: Lustre: fs1-OST00d4: Connection restored to a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) Aug 20 20:14:37 dac-e-18 kernel: Lustre: Skipped 131 previous similar messages Aug 20 20:22:43 dac-e-18 kernel: Lustre: fs1-OST00cc: Connection restored to 6d4b3b17-aa0e-1cb4-a610-cacce42ddc94 (at 10.47.20.69@o2ib1) Aug 20 20:22:43 dac-e-18 kernel: Lustre: fs1-OST00d0: Connection restored to 3f94c4e7-d409-8966-0a4b-959655ee93b1 (at 10.47.21.32@o2ib1) Aug 20 20:22:43 dac-e-18 kernel: Lustre: Skipped 9 previous similar messages Aug 20 20:22:45 dac-e-18 kernel: Lustre: fs1-OST00d1: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:22:45 dac-e-18 kernel: LustreError: 432127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff96d35ad16c00 Aug 20 20:22:45 dac-e-18 kernel: LustreError: 432124:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff96d35ad16c00 Aug 20 20:22:45 dac-e-18 kernel: LustreError: 432132:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff96d35ad16c00 Aug 20 20:22:45 dac-e-18 kernel: LustreError: 432126:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff96d35ad16c00 Aug 20 20:22:45 dac-e-18 kernel: LustreError: 22197:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff96d44604d850 x1642414042283920/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:721/0 lens 488/440 e 0 to 0 dl 1566329006 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:22:46 dac-e-18 kernel: Lustre: fs1-OST00d1: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 HOSTS ------------------------------------------------------------------------- dac-e-19 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:47 BST, end at Wed 2019-08-21 11:25:26 BST. -- Aug 20 20:14:37 dac-e-19 kernel: Lustre: fs1-OST00e1: Connection restored to 2a21b08a-665f-68a5-f629-a4c31d7cff10 (at 10.47.21.36@o2ib1) Aug 20 20:14:37 dac-e-19 kernel: Lustre: Skipped 128 previous similar messages Aug 20 20:22:43 dac-e-19 kernel: Lustre: fs1-OST00de: Connection restored to a142f57f-3986-5d9a-f4ac-ea41013ab5b4 (at 10.47.21.34@o2ib1) Aug 20 20:22:43 dac-e-19 kernel: Lustre: Skipped 15 previous similar messages Aug 20 20:22:45 dac-e-19 kernel: LustreError: 129273:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9caafd7f5800 Aug 20 20:22:45 dac-e-19 kernel: LustreError: 129275:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9caafd7f5800 Aug 20 20:22:45 dac-e-19 kernel: LustreError: 129274:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9caafd7f5800 Aug 20 20:22:45 dac-e-19 kernel: LustreError: 129280:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9caafd7f5800 Aug 20 20:22:46 dac-e-19 kernel: LustreError: 129272:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9caafd7f5800 Aug 20 20:22:46 dac-e-19 kernel: LustreError: 129274:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9caafd7f5800 Aug 20 20:22:46 dac-e-19 kernel: LustreError: 129273:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9caafd7f5800 Aug 20 20:22:46 dac-e-19 kernel: LustreError: 129280:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9caafd7f5800 Aug 20 20:22:46 dac-e-19 kernel: LustreError: 129275:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9caafd7f5800 Aug 20 20:22:46 dac-e-19 kernel: LustreError: 129272:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9caafd7f5800 Aug 20 20:22:46 dac-e-19 kernel: LustreError: 129273:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9caafd7f5800 Aug 20 20:22:46 dac-e-19 kernel: LustreError: 129273:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9caafd7f5800 Aug 20 20:22:46 dac-e-19 kernel: LustreError: 129274:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9caafd7f5800 Aug 20 20:22:46 dac-e-19 kernel: LustreError: 129280:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9caafd7f5800 Aug 20 20:22:46 dac-e-19 kernel: LustreError: 129275:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9caafd7f5800 Aug 20 20:22:46 dac-e-19 kernel: LustreError: 129272:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9caafd7f5800 Aug 20 20:22:46 dac-e-19 kernel: Lustre: fs1-OST00e0: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:22:47 dac-e-19 kernel: LustreError: 82875:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff9cc0d97a6850 x1642414042283936/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:721/0 lens 488/440 e 0 to 0 dl 1566329006 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:22:47 dac-e-19 kernel: Lustre: fs1-OST00e0: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 HOSTS ------------------------------------------------------------------------- dac-e-21 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:47 BST, end at Wed 2019-08-21 11:25:26 BST. -- Aug 20 20:14:37 dac-e-21 kernel: Lustre: fs1-OST00f4: Connection restored to a142f57f-3986-5d9a-f4ac-ea41013ab5b4 (at 10.47.21.34@o2ib1) Aug 20 20:14:37 dac-e-21 kernel: Lustre: Skipped 129 previous similar messages Aug 20 20:22:43 dac-e-21 kernel: Lustre: fs1-OST00f9: Connection restored to 0ebebeeb-b828-0337-48a2-2bd97ab9fa21 (at 10.47.21.37@o2ib1) Aug 20 20:22:43 dac-e-21 kernel: Lustre: Skipped 10 previous similar messages Aug 20 20:22:45 dac-e-21 kernel: LustreError: 87864:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff959312c53000 Aug 20 20:22:45 dac-e-21 kernel: LustreError: 87926:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9592d7237800 Aug 20 20:22:45 dac-e-21 kernel: LustreError: 87862:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9592d7237800 Aug 20 20:22:45 dac-e-21 kernel: LustreError: 87865:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9592d7237800 Aug 20 20:22:45 dac-e-21 kernel: LustreError: 87863:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9592d7237800 Aug 20 20:22:45 dac-e-21 kernel: LustreError: 87864:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9592d7237800 Aug 20 20:22:45 dac-e-21 kernel: LustreError: 87862:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9592d7237800 Aug 20 20:22:45 dac-e-21 kernel: LustreError: 87865:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff959312c53000 Aug 20 20:22:45 dac-e-21 kernel: LustreError: 87926:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff959312c53000 Aug 20 20:22:45 dac-e-21 kernel: LustreError: 87863:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9592d7237800 Aug 20 20:22:45 dac-e-21 kernel: LustreError: 87864:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9592d7237800 Aug 20 20:22:45 dac-e-21 kernel: LustreError: 87862:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9592d7237800 Aug 20 20:22:45 dac-e-21 kernel: LustreError: 87863:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9592d7237800 Aug 20 20:22:45 dac-e-21 kernel: LustreError: 87865:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9592d7237800 Aug 20 20:22:45 dac-e-21 kernel: LustreError: 87862:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9592d7237800 Aug 20 20:22:45 dac-e-21 kernel: LustreError: 87864:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff959312c53000 Aug 20 20:22:45 dac-e-21 kernel: LustreError: 87862:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9592d7237800 Aug 20 20:22:45 dac-e-21 kernel: LustreError: 87865:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9592d7237800 Aug 20 20:22:45 dac-e-21 kernel: LustreError: 87863:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9592d7237800 Aug 20 20:22:45 dac-e-21 kernel: LustreError: 87926:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9592d7237800 Aug 20 20:22:49 dac-e-21 kernel: Lustre: fs1-OST00f7: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:22:50 dac-e-21 kernel: LustreError: 56729:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff959345736850 x1642414042283328/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:721/0 lens 488/440 e 0 to 0 dl 1566329006 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:22:50 dac-e-21 kernel: Lustre: fs1-OST00f7: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:23:34 dac-e-21 kernel: LustreError: 68853:0:(ldlm_lib.c:3259:target_bulk_io()) @@@ network error on bulk READ req@ffff9593453c7850 x1642414042283712/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:746/0 lens 488/440 e 1 to 0 dl 1566329031 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:23:34 dac-e-21 kernel: Lustre: fs1-OST00f1: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:30:58 dac-e-21 kernel: Lustre: fs1-OST00f1: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:30:58 dac-e-21 kernel: Lustre: fs1-OST00f1: Connection restored to a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) Aug 20 20:30:58 dac-e-21 kernel: Lustre: Skipped 13 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-23 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:47 BST, end at Wed 2019-08-21 11:25:26 BST. -- Aug 20 20:14:37 dac-e-23 kernel: Lustre: fs1-OST0110: Connection restored to a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) Aug 20 20:14:37 dac-e-23 kernel: Lustre: Skipped 128 previous similar messages Aug 20 20:22:43 dac-e-23 kernel: Lustre: fs1-OST010e: Connection restored to ea42bfba-6613-4dee-aa4b-19c1f0d400fe (at 10.47.21.38@o2ib1) Aug 20 20:22:43 dac-e-23 kernel: Lustre: Skipped 12 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-2 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:46 BST, end at Wed 2019-08-21 11:25:26 BST. -- Aug 20 20:14:37 dac-e-2 kernel: Lustre: fs1-OST0017: Connection restored to 448bebe0-3beb-da9f-3d47-dbc50659268e (at 10.47.21.33@o2ib1) Aug 20 20:14:37 dac-e-2 kernel: Lustre: fs1-OST000e: Connection restored to a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) Aug 20 20:14:37 dac-e-2 kernel: Lustre: Skipped 129 previous similar messages Aug 20 20:22:43 dac-e-2 kernel: Lustre: fs1-OST0012: Connection restored to 21ae9b96-1e86-cd59-3acd-9878c229c355 (at 10.47.21.35@o2ib1) Aug 20 20:22:43 dac-e-2 kernel: Lustre: fs1-OST000f: Connection restored to 0ebebeeb-b828-0337-48a2-2bd97ab9fa21 (at 10.47.21.37@o2ib1) Aug 20 20:22:43 dac-e-2 kernel: Lustre: Skipped 11 previous similar messages Aug 20 20:22:45 dac-e-2 kernel: LustreError: 206908:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8e0f4fb08000 Aug 20 20:22:45 dac-e-2 kernel: LustreError: 206914:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8e0f4fb08000 Aug 20 20:22:45 dac-e-2 kernel: LustreError: 206909:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8e0f4fb08000 Aug 20 20:22:45 dac-e-2 kernel: LustreError: 206907:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8e0f4fb08000 Aug 20 20:22:45 dac-e-2 kernel: LustreError: 206914:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8e0f4fb08000 Aug 20 20:22:45 dac-e-2 kernel: LustreError: 206908:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8e0f4fb08000 Aug 20 20:22:45 dac-e-2 kernel: LustreError: 206909:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8e0f4fb08000 Aug 20 20:22:45 dac-e-2 kernel: LustreError: 206906:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8e0f4fb08000 Aug 20 20:22:45 dac-e-2 kernel: LustreError: 206907:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8e0f4fb08000 Aug 20 20:22:45 dac-e-2 kernel: LustreError: 206908:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8e0f4fb08000 Aug 20 20:22:50 dac-e-2 kernel: Lustre: fs1-OST0016: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:22:51 dac-e-2 kernel: LustreError: 281922:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8e280ba44050 x1642414042283616/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:721/0 lens 488/440 e 0 to 0 dl 1566329006 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:22:51 dac-e-2 kernel: Lustre: fs1-OST0016: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 HOSTS ------------------------------------------------------------------------- dac-e-22 ------------------------------------------------------------------------------- -- Logs begin at Thu 2019-05-09 10:24:44 BST, end at Wed 2019-08-21 11:25:26 BST. -- Aug 20 20:14:37 dac-e-22 kernel: Lustre: fs1-OST0104: Connection restored to 6d4b3b17-aa0e-1cb4-a610-cacce42ddc94 (at 10.47.20.69@o2ib1) Aug 20 20:14:37 dac-e-22 kernel: Lustre: fs1-OST0107: Connection restored to 21ae9b96-1e86-cd59-3acd-9878c229c355 (at 10.47.21.35@o2ib1) Aug 20 20:14:37 dac-e-22 kernel: Lustre: fs1-OST00fc: Connection restored to ea42bfba-6613-4dee-aa4b-19c1f0d400fe (at 10.47.21.38@o2ib1) Aug 20 20:14:37 dac-e-22 kernel: Lustre: fs1-OST0100: Connection restored to 21ae9b96-1e86-cd59-3acd-9878c229c355 (at 10.47.21.35@o2ib1) Aug 20 20:14:37 dac-e-22 kernel: Lustre: Skipped 128 previous similar messages Aug 20 20:14:37 dac-e-22 kernel: Lustre: Skipped 128 previous similar messages Aug 20 20:14:37 dac-e-22 kernel: Lustre: Skipped 128 previous similar messages Aug 20 20:14:37 dac-e-22 kernel: Lustre: Skipped 1 previous similar message Aug 20 20:22:43 dac-e-22 kernel: Lustre: fs1-OST00fd: Connection restored to a142f57f-3986-5d9a-f4ac-ea41013ab5b4 (at 10.47.21.34@o2ib1) Aug 20 20:22:43 dac-e-22 kernel: Lustre: Skipped 6 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-3 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:47 BST, end at Wed 2019-08-21 11:25:26 BST. -- Aug 20 20:14:37 dac-e-3 kernel: Lustre: fs1-OST0020: Connection restored to a142f57f-3986-5d9a-f4ac-ea41013ab5b4 (at 10.47.21.34@o2ib1) Aug 20 20:14:37 dac-e-3 kernel: Lustre: fs1-OST001b: Connection restored to 2a21b08a-665f-68a5-f629-a4c31d7cff10 (at 10.47.21.36@o2ib1) Aug 20 20:14:37 dac-e-3 kernel: Lustre: Skipped 130 previous similar messages Aug 20 20:22:43 dac-e-3 kernel: Lustre: fs1-OST0020: Connection restored to 448bebe0-3beb-da9f-3d47-dbc50659268e (at 10.47.21.33@o2ib1) Aug 20 20:22:43 dac-e-3 kernel: Lustre: Skipped 10 previous similar messages Aug 20 20:22:45 dac-e-3 kernel: LustreError: 79243:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff993f70f30800 Aug 20 20:22:45 dac-e-3 kernel: LustreError: 79246:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff993f70f30800 Aug 20 20:22:45 dac-e-3 kernel: LustreError: 79244:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff993f70f30800 Aug 20 20:22:45 dac-e-3 kernel: LustreError: 79251:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff993f70f30800 Aug 20 20:22:49 dac-e-3 kernel: Lustre: fs1-OST001c: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:22:50 dac-e-3 kernel: LustreError: 58883:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff993f808c5850 x1642414042283776/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:721/0 lens 488/440 e 0 to 0 dl 1566329006 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:22:50 dac-e-3 kernel: Lustre: fs1-OST001c: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 HOSTS ------------------------------------------------------------------------- dac-e-6 ------------------------------------------------------------------------------- -- Logs begin at Thu 2019-05-09 08:54:36 BST, end at Wed 2019-08-21 11:25:26 BST. -- Aug 20 20:14:37 dac-e-6 kernel: Lustre: fs1-OST0046: Connection restored to 2a21b08a-665f-68a5-f629-a4c31d7cff10 (at 10.47.21.36@o2ib1) Aug 20 20:14:37 dac-e-6 kernel: Lustre: Skipped 128 previous similar messages Aug 20 20:22:43 dac-e-6 kernel: Lustre: fs1-OST003d: Connection restored to 3f94c4e7-d409-8966-0a4b-959655ee93b1 (at 10.47.21.32@o2ib1) Aug 20 20:22:43 dac-e-6 kernel: Lustre: Skipped 12 previous similar messages Aug 20 20:32:27 dac-e-6 kernel: LustreError: 398049:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 12582912(16777216) req@ffff8fdaffdb7050 x1642414051656448/t0(0) o3->21ae9b96-1e86-cd59-3acd-9878c229c355@10.47.21.35@o2ib1:520/0 lens 488/440 e 1 to 0 dl 1566329560 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:32:27 dac-e-6 kernel: Lustre: fs1-OST0044: Bulk IO read error with 21ae9b96-1e86-cd59-3acd-9878c229c355 (at 10.47.21.35@o2ib1), client will retry: rc -110 Aug 20 20:32:27 dac-e-6 kernel: LustreError: 398049:0:(ldlm_lib.c:3268:target_bulk_io()) Skipped 1 previous similar message Aug 20 20:32:41 dac-e-6 kernel: Lustre: fs1-OST0043: Client 21ae9b96-1e86-cd59-3acd-9878c229c355 (at 10.47.21.35@o2ib1) reconnecting Aug 20 20:32:41 dac-e-6 kernel: Lustre: fs1-OST0043: Connection restored to 21ae9b96-1e86-cd59-3acd-9878c229c355 (at 10.47.21.35@o2ib1) Aug 20 20:32:41 dac-e-6 kernel: Lustre: Skipped 9 previous similar messages Aug 20 20:32:43 dac-e-6 kernel: Lustre: fs1-OST0044: Client 21ae9b96-1e86-cd59-3acd-9878c229c355 (at 10.47.21.35@o2ib1) reconnecting HOSTS ------------------------------------------------------------------------- dac-e-20 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:47 BST, end at Wed 2019-08-21 11:25:26 BST. -- Aug 20 20:13:25 dac-e-20 kernel: perf: interrupt took too long (2509 > 2500), lowering kernel.perf_event_max_sample_rate to 79000 Aug 20 20:14:37 dac-e-20 kernel: Lustre: fs1-OST00ef: Connection restored to 0ebebeeb-b828-0337-48a2-2bd97ab9fa21 (at 10.47.21.37@o2ib1) Aug 20 20:14:37 dac-e-20 kernel: Lustre: Skipped 129 previous similar messages Aug 20 20:22:43 dac-e-20 kernel: Lustre: fs1-OST00ef: Connection restored to 2a21b08a-665f-68a5-f629-a4c31d7cff10 (at 10.47.21.36@o2ib1) Aug 20 20:22:43 dac-e-20 kernel: Lustre: Skipped 10 previous similar messages Aug 20 20:22:45 dac-e-20 kernel: Lustre: fs1-OST00e6: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:22:45 dac-e-20 kernel: LustreError: 102284:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff91d890b8ee00 Aug 20 20:22:45 dac-e-20 kernel: LustreError: 102349:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff91d890b8ee00 Aug 20 20:22:45 dac-e-20 kernel: LustreError: 102286:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff91d890b8ee00 Aug 20 20:22:45 dac-e-20 kernel: LustreError: 102283:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff91d890b8ee00 Aug 20 20:22:45 dac-e-20 kernel: LustreError: 122653:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff91d8e6ae1850 x1642414042283760/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:721/0 lens 488/440 e 0 to 0 dl 1566329006 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:22:46 dac-e-20 kernel: Lustre: fs1-OST00e6: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:31:51 dac-e-20 kernel: LustreError: 122649:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 15728640(16777216) req@ffff91d8e6af4050 x1642414047845264/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:491/0 lens 488/440 e 3 to 0 dl 1566329531 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:31:51 dac-e-20 kernel: Lustre: fs1-OST00e6: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:33:35 dac-e-20 kernel: Lustre: fs1-OST00e6: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:33:35 dac-e-20 kernel: Lustre: fs1-OST00e6: Connection restored to a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) Aug 20 20:33:35 dac-e-20 kernel: Lustre: Skipped 8 previous similar messages Aug 20 20:33:53 dac-e-20 kernel: Lustre: fs1-OST00e7: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:33:53 dac-e-20 kernel: Lustre: fs1-OST00e7: Connection restored to a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) HOSTS ------------------------------------------------------------------------- dac-e-5 ------------------------------------------------------------------------------- -- Logs begin at Thu 2019-05-09 08:54:36 BST, end at Wed 2019-08-21 11:25:26 BST. -- Aug 20 20:14:37 dac-e-5 kernel: Lustre: fs1-OST0030: Connection restored to ea42bfba-6613-4dee-aa4b-19c1f0d400fe (at 10.47.21.38@o2ib1) Aug 20 20:14:37 dac-e-5 kernel: Lustre: Skipped 129 previous similar messages Aug 20 20:22:43 dac-e-5 kernel: Lustre: fs1-OST0039: Connection restored to 0ebebeeb-b828-0337-48a2-2bd97ab9fa21 (at 10.47.21.37@o2ib1) Aug 20 20:22:43 dac-e-5 kernel: Lustre: Skipped 11 previous similar messages Aug 20 20:22:45 dac-e-5 kernel: LustreError: 173746:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883cd87f2000 Aug 20 20:22:45 dac-e-5 kernel: LustreError: 173747:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883cd87f2000 Aug 20 20:22:45 dac-e-5 kernel: LustreError: 173744:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883cd87f2000 Aug 20 20:22:45 dac-e-5 kernel: LustreError: 173760:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883cd87f2000 Aug 20 20:22:45 dac-e-5 kernel: LustreError: 173745:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883cd87f2000 Aug 20 20:22:45 dac-e-5 kernel: LustreError: 173744:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883cd87f2000 Aug 20 20:22:45 dac-e-5 kernel: LustreError: 173744:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883cd87f2000 Aug 20 20:22:45 dac-e-5 kernel: LustreError: 173746:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883cd87f2000 Aug 20 20:22:45 dac-e-5 kernel: LustreError: 173760:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883cd87f2000 Aug 20 20:22:45 dac-e-5 kernel: LustreError: 173745:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883cd87f2000 Aug 20 20:22:46 dac-e-5 kernel: Lustre: fs1-OST0036: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:22:46 dac-e-5 kernel: LustreError: 243159:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff883c4e2e6850 x1642414042283056/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:709/0 lens 488/440 e 1 to 0 dl 1566328994 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:22:46 dac-e-5 kernel: Lustre: fs1-OST0036: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 HOSTS ------------------------------------------------------------------------- dac-e-4 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-05-07 14:07:28 BST, end at Wed 2019-08-21 11:25:26 BST. -- Aug 20 20:14:37 dac-e-4 kernel: Lustre: fs1-OST0024: Connection restored to 3f94c4e7-d409-8966-0a4b-959655ee93b1 (at 10.47.21.32@o2ib1) Aug 20 20:14:37 dac-e-4 kernel: Lustre: Skipped 127 previous similar messages Aug 20 20:22:43 dac-e-4 kernel: Lustre: fs1-OST0024: Connection restored to a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) Aug 20 20:22:43 dac-e-4 kernel: Lustre: fs1-OST0027: Connection restored to 448bebe0-3beb-da9f-3d47-dbc50659268e (at 10.47.21.33@o2ib1) Aug 20 20:22:43 dac-e-4 kernel: Lustre: Skipped 16 previous similar messages Aug 20 20:22:45 dac-e-4 kernel: Lustre: fs1-OST0029: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:22:45 dac-e-4 kernel: LustreError: 353704:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff981534933400 Aug 20 20:22:45 dac-e-4 kernel: LustreError: 353697:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff981534933400 Aug 20 20:22:45 dac-e-4 kernel: LustreError: 353698:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff981534933400 Aug 20 20:22:45 dac-e-4 kernel: LustreError: 353696:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff981534933400 Aug 20 20:22:45 dac-e-4 kernel: LustreError: 40661:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff98154d35e050 x1642414042283104/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:721/0 lens 488/440 e 0 to 0 dl 1566329006 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:22:45 dac-e-4 kernel: LustreError: 353698:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff981534933400 Aug 20 20:22:45 dac-e-4 kernel: LustreError: 353704:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff981534933400 Aug 20 20:22:45 dac-e-4 kernel: LustreError: 353699:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff981534933400 Aug 20 20:22:45 dac-e-4 kernel: LustreError: 353697:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff981534933400 Aug 20 20:22:45 dac-e-4 kernel: LustreError: 353698:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff981437924a00 Aug 20 20:22:45 dac-e-4 kernel: LustreError: 353696:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff981437924a00 Aug 20 20:22:45 dac-e-4 kernel: LustreError: 353699:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9813baee4600 Aug 20 20:22:45 dac-e-4 kernel: Lustre: fs1-OST0029: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:22:45 dac-e-4 kernel: LustreError: 353697:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9813baee4600 Aug 20 20:22:45 dac-e-4 kernel: LustreError: 353704:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9813baee4600 Aug 20 20:22:45 dac-e-4 kernel: LustreError: 353696:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9813baee4600 Aug 20 20:22:45 dac-e-4 kernel: LustreError: 353699:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff981534933400 Aug 20 20:22:45 dac-e-4 kernel: LustreError: 353698:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff981534933400 Aug 20 20:22:45 dac-e-4 kernel: LustreError: 353696:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff981534933400 Aug 20 20:22:45 dac-e-4 kernel: LustreError: 353697:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff981534933400 Aug 20 20:22:46 dac-e-4 kernel: LustreError: 353698:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff981534933400 Aug 20 20:22:46 dac-e-4 kernel: LustreError: 353696:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff981534933400 Aug 20 20:22:46 dac-e-4 kernel: LustreError: 353704:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff981534933400 Aug 20 20:22:46 dac-e-4 kernel: LustreError: 353697:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff981534933400 Aug 20 20:23:15 dac-e-4 kernel: Lustre: fs1-OST002c: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:23:15 dac-e-4 kernel: Lustre: fs1-OST002c: Connection restored to a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) Aug 20 20:23:15 dac-e-4 kernel: Lustre: Skipped 8 previous similar messages Aug 20 20:23:16 dac-e-4 kernel: LustreError: 377791:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff98154d345850 x1642414042284000/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:721/0 lens 488/440 e 0 to 0 dl 1566329006 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:23:16 dac-e-4 kernel: Lustre: fs1-OST002c: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:23:44 dac-e-4 kernel: LustreError: 377965:0:(ldlm_lib.c:3259:target_bulk_io()) @@@ network error on bulk READ req@ffff98154d347050 x1642414042283296/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:746/0 lens 488/440 e 1 to 0 dl 1566329031 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:23:44 dac-e-4 kernel: LustreError: 442475:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 14680064(16777216) req@ffff98154d35b050 x1642414042283104/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:749/0 lens 488/440 e 1 to 0 dl 1566329034 ref 1 fl Interpret:/2/0 rc 0/0 Aug 20 20:23:44 dac-e-4 kernel: Lustre: fs1-OST0029: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:30:58 dac-e-4 kernel: Lustre: fs1-OST0024: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:30:58 dac-e-4 kernel: Lustre: fs1-OST0024: Connection restored to a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) Aug 20 20:32:04 dac-e-4 kernel: LustreError: 377811:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 15728640(16777216) req@ffff98154d359050 x1642414047877440/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:509/0 lens 488/440 e 0 to 0 dl 1566329549 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:32:04 dac-e-4 kernel: Lustre: fs1-OST0024: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:32:04 dac-e-4 kernel: Lustre: Skipped 1 previous similar message Aug 20 20:33:35 dac-e-4 kernel: Lustre: fs1-OST0029: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:33:35 dac-e-4 kernel: Lustre: fs1-OST0029: Connection restored to a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) Aug 20 20:34:59 dac-e-4 kernel: LustreError: 43771:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 14680064(16777216) req@ffff98154d34a050 x1642414048548848/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:676/0 lens 488/440 e 3 to 0 dl 1566329716 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:34:59 dac-e-4 kernel: Lustre: fs1-OST0029: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:36:44 dac-e-4 kernel: Lustre: fs1-OST0029: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:37:16 dac-e-4 kernel: Lustre: fs1-OST002c: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:37:54 dac-e-4 kernel: LustreError: 43771:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 13631488(16777216) req@ffff98154d34c050 x1642414050082896/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:115/0 lens 488/440 e 0 to 0 dl 1566329910 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:37:54 dac-e-4 kernel: Lustre: fs1-OST0029: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:38:19 dac-e-4 kernel: LustreError: 377811:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 15728640(16777216) req@ffff98154d35e050 x1642414050379776/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:160/0 lens 488/440 e 0 to 0 dl 1566329955 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:38:19 dac-e-4 kernel: Lustre: fs1-OST0024: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:38:32 dac-e-4 kernel: Lustre: fs1-OST0029: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:38:32 dac-e-4 kernel: Lustre: Skipped 1 previous similar message Aug 20 20:38:32 dac-e-4 kernel: Lustre: fs1-OST0029: Connection restored to a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) Aug 20 20:38:32 dac-e-4 kernel: Lustre: Skipped 3 previous similar messages Aug 20 20:39:34 dac-e-4 kernel: LustreError: 377811:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 12582912(16777216) req@ffff98154d34a050 x1642414050390816/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:223/0 lens 488/440 e 0 to 0 dl 1566330018 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:39:34 dac-e-4 kernel: Lustre: fs1-OST0029: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 HOSTS ------------------------------------------------------------------------- dac-e-9 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-05-07 20:43:31 BST, end at Wed 2019-08-21 11:25:26 BST. -- Aug 20 20:14:37 dac-e-9 kernel: Lustre: fs1-OST0066: Connection restored to 6d4b3b17-aa0e-1cb4-a610-cacce42ddc94 (at 10.47.20.69@o2ib1) Aug 20 20:14:37 dac-e-9 kernel: Lustre: Skipped 127 previous similar messages Aug 20 20:22:43 dac-e-9 kernel: Lustre: fs1-OST0065: Connection restored to 3f94c4e7-d409-8966-0a4b-959655ee93b1 (at 10.47.21.32@o2ib1) Aug 20 20:22:43 dac-e-9 kernel: Lustre: Skipped 138 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-7 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-05-07 14:07:28 BST, end at Wed 2019-08-21 11:25:26 BST. -- Aug 20 20:14:37 dac-e-7 kernel: Lustre: fs1-OST0050: Connection restored to a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) Aug 20 20:14:37 dac-e-7 kernel: Lustre: Skipped 128 previous similar messages Aug 20 20:22:43 dac-e-7 kernel: Lustre: fs1-OST004d: Connection restored to 6d4b3b17-aa0e-1cb4-a610-cacce42ddc94 (at 10.47.20.69@o2ib1) Aug 20 20:22:43 dac-e-7 kernel: Lustre: fs1-OST004d: Connection restored to 3f94c4e7-d409-8966-0a4b-959655ee93b1 (at 10.47.21.32@o2ib1) Aug 20 20:22:43 dac-e-7 kernel: Lustre: Skipped 12 previous similar messages Aug 20 20:22:45 dac-e-7 kernel: Lustre: fs1-OST0053: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:22:45 dac-e-7 kernel: LustreError: 365928:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff968632630800 Aug 20 20:22:45 dac-e-7 kernel: LustreError: 365927:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff968632630800 Aug 20 20:22:45 dac-e-7 kernel: LustreError: 365971:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff968632630800 Aug 20 20:22:45 dac-e-7 kernel: LustreError: 365925:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff968632630800 Aug 20 20:22:45 dac-e-7 kernel: LustreError: 365925:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff968632630800 Aug 20 20:22:45 dac-e-7 kernel: LustreError: 394086:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff968748e51050 x1642414042283968/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:721/0 lens 488/440 e 0 to 0 dl 1566329006 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:22:45 dac-e-7 kernel: LustreError: 365926:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff968632630800 Aug 20 20:22:45 dac-e-7 kernel: LustreError: 365971:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff968632630800 Aug 20 20:22:45 dac-e-7 kernel: LustreError: 365928:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff968632630800 Aug 20 20:22:45 dac-e-7 kernel: LustreError: 365927:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff968632630800 Aug 20 20:22:45 dac-e-7 kernel: LustreError: 365925:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff968632630800 Aug 20 20:22:45 dac-e-7 kernel: LustreError: 365926:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff968632630800 Aug 20 20:22:45 dac-e-7 kernel: LustreError: 365926:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff968632630800 Aug 20 20:22:45 dac-e-7 kernel: LustreError: 365971:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff968632630800 Aug 20 20:22:45 dac-e-7 kernel: LustreError: 365928:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff968632630800 Aug 20 20:22:45 dac-e-7 kernel: LustreError: 365925:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff968632630800 Aug 20 20:22:45 dac-e-7 kernel: LustreError: 365927:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff968632630800 Aug 20 20:22:45 dac-e-7 kernel: Lustre: fs1-OST0053: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:23:48 dac-e-7 kernel: LustreError: 394086:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 13631488(16777216) req@ffff968748e23050 x1642414042283968/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:749/0 lens 488/440 e 1 to 0 dl 1566329034 ref 1 fl Interpret:/2/0 rc 0/0 Aug 20 20:23:48 dac-e-7 kernel: Lustre: fs1-OST0053: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:33:53 dac-e-7 kernel: Lustre: fs1-OST0053: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:33:53 dac-e-7 kernel: Lustre: fs1-OST0053: Connection restored to a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) Aug 20 20:33:53 dac-e-7 kernel: Lustre: Skipped 12 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-8 ------------------------------------------------------------------------------- -- Logs begin at Thu 2019-05-09 08:54:37 BST, end at Wed 2019-08-21 11:25:26 BST. -- Aug 20 20:14:37 dac-e-8 kernel: Lustre: fs1-OST0054: Connection restored to 0ebebeeb-b828-0337-48a2-2bd97ab9fa21 (at 10.47.21.37@o2ib1) Aug 20 20:14:37 dac-e-8 kernel: Lustre: Skipped 128 previous similar messages Aug 20 20:22:43 dac-e-8 kernel: Lustre: fs1-OST005b: Connection restored to 448bebe0-3beb-da9f-3d47-dbc50659268e (at 10.47.21.33@o2ib1) Aug 20 20:22:43 dac-e-8 kernel: Lustre: fs1-OST0056: Connection restored to 2a21b08a-665f-68a5-f629-a4c31d7cff10 (at 10.47.21.36@o2ib1) Aug 20 20:22:43 dac-e-8 kernel: Lustre: Skipped 12 previous similar messages Aug 20 20:22:43 dac-e-8 kernel: Lustre: Skipped 1 previous similar message Aug 20 20:22:45 dac-e-8 kernel: LustreError: 354783:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fc1a029ea00 Aug 20 20:22:45 dac-e-8 kernel: LustreError: 354775:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fc1a029ea00 Aug 20 20:22:45 dac-e-8 kernel: LustreError: 354776:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fc1a029ea00 Aug 20 20:22:45 dac-e-8 kernel: LustreError: 354778:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fc1a029ea00 Aug 20 20:22:45 dac-e-8 kernel: LustreError: 354776:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fc1c8fea600 Aug 20 20:22:45 dac-e-8 kernel: LustreError: 354783:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fc1c8fea600 Aug 20 20:22:45 dac-e-8 kernel: LustreError: 354776:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fc1c8fea600 Aug 20 20:22:45 dac-e-8 kernel: LustreError: 354775:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fc1c8fea600 Aug 20 20:22:49 dac-e-8 kernel: Lustre: fs1-OST005e: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:22:50 dac-e-8 kernel: LustreError: 46422:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8fc25b632050 x1642414042283680/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:709/0 lens 488/440 e 1 to 0 dl 1566328994 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:22:50 dac-e-8 kernel: Lustre: fs1-OST005e: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:23:47 dac-e-8 kernel: LustreError: 46417:0:(ldlm_lib.c:3259:target_bulk_io()) @@@ network error on bulk READ req@ffff8fc25b636850 x1642414042283520/t0(0) o3->a6bdc688-d4aa-fa71-980c-a80daac4bfde@10.47.21.31@o2ib1:4/0 lens 488/440 e 3 to 0 dl 1566329044 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 20:23:47 dac-e-8 kernel: Lustre: fs1-OST005d: Bulk IO read error with a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1), client will retry: rc -110 Aug 20 20:33:53 dac-e-8 kernel: Lustre: fs1-OST005d: Client a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) reconnecting Aug 20 20:33:53 dac-e-8 kernel: Lustre: fs1-OST005d: Connection restored to a6bdc688-d4aa-fa71-980c-a80daac4bfde (at 10.47.21.31@o2ib1) Aug 20 20:33:53 dac-e-8 kernel: Lustre: Skipped 8 previous similar messages