HOSTS ------------------------------------------------------------------------- dac-e-10 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-05-07 14:07:27 BST, end at Wed 2019-08-21 11:51:58 BST. -- Aug 21 01:22:41 dac-e-10 kernel: Lustre: fs1-MDT0009: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:22:41 dac-e-10 kernel: Lustre: Skipped 598 previous similar messages Aug 21 01:23:42 dac-e-10 kernel: Lustre: fs1-OST0076: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:23:42 dac-e-10 kernel: Lustre: Skipped 127 previous similar messages Aug 21 01:29:14 dac-e-10 kernel: Lustre: fs1-OST0076: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 21 01:29:14 dac-e-10 kernel: Lustre: Skipped 72 previous similar messages Aug 21 01:29:23 dac-e-10 kernel: Lustre: fs1-OST0072: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:29:23 dac-e-10 kernel: LustreError: 5417:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff905d49f65e00 Aug 21 01:29:23 dac-e-10 kernel: LustreError: 5428:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff905d49f65e00 Aug 21 01:29:23 dac-e-10 kernel: LustreError: 381135:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff90454ed5a850 x1642422187202208/t0(0) o3->4e14a970-7ca0-d5d3-629d-52842291c983@10.47.21.37@o2ib1:212/0 lens 488/440 e 0 to 0 dl 1566347372 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:29:23 dac-e-10 kernel: Lustre: 448148:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347362/real 1566347363] req@ffff905d23a59680 x1642422111953184/t0(0) o105->fs1-OST006f@10.47.20.69@o2ib1:15/16 lens 360/224 e 0 to 1 dl 1566347373 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 21 01:29:23 dac-e-10 kernel: Lustre: 448148:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Aug 21 01:29:23 dac-e-10 kernel: LustreError: 5428:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff90451f8ef200 Aug 21 01:29:23 dac-e-10 kernel: LustreError: 5417:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff90451f8ef200 Aug 21 01:29:23 dac-e-10 kernel: Lustre: fs1-OST006c: Bulk IO read error with 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1), client will retry: rc -110 Aug 21 01:29:23 dac-e-10 kernel: Lustre: Skipped 2 previous similar messages Aug 21 01:29:23 dac-e-10 kernel: LustreError: 381135:0:(ldlm_lib.c:3253:target_bulk_io()) Skipped 1 previous similar message Aug 21 01:29:24 dac-e-10 kernel: LustreError: 5417:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff90451f8ef200 Aug 21 01:29:24 dac-e-10 kernel: LustreError: 5419:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff90451f8ef200 Aug 21 01:29:24 dac-e-10 kernel: Lustre: 448137:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347362/real 1566347364] req@ffff905d49f21f80 x1642422111952752/t0(0) o105->fs1-OST006c@10.47.20.69@o2ib1:15/16 lens 360/224 e 0 to 1 dl 1566347373 ref 1 fl Rpc:eXS/0/ffffffff rc -11/-1 Aug 21 01:29:24 dac-e-10 kernel: LustreError: 5420:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff90451f8ef200 Aug 21 01:29:24 dac-e-10 kernel: LustreError: 5418:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff90451f8ef200 Aug 21 01:29:24 dac-e-10 kernel: LustreError: 5417:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff90451f8ef200 Aug 21 01:29:24 dac-e-10 kernel: LustreError: 5419:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff90451f8ef200 Aug 21 01:29:24 dac-e-10 kernel: LustreError: 373548:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff904654ca1050 x1642422187202496/t0(0) o3->ca24e700-cef1-0f8f-792b-2c007834cfd0@10.47.20.69@o2ib1:244/0 lens 488/440 e 0 to 0 dl 1566347404 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:29:24 dac-e-10 kernel: Lustre: fs1-OST0074: Bulk IO read error with ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1), client will retry: rc -110 Aug 21 01:29:24 dac-e-10 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:29:28 dac-e-10 kernel: Lustre: fs1-OST006f: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:29:28 dac-e-10 kernel: Lustre: Skipped 5 previous similar messages Aug 21 01:29:36 dac-e-10 kernel: Lustre: 308495:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566347365/real 1566347365] req@ffff9044a8e8a880 x1642422111959600/t0(0) o104->fs1-OST006c@10.47.20.69@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1566347376 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Aug 21 01:29:36 dac-e-10 kernel: Lustre: 308495:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 1 previous similar message Aug 21 01:29:54 dac-e-10 kernel: Lustre: fs1-OST0077: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:29:54 dac-e-10 kernel: Lustre: Skipped 36 previous similar messages Aug 21 01:30:02 dac-e-10 kernel: Lustre: fs1-OST006c: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:30:02 dac-e-10 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:30:34 dac-e-10 kernel: LustreError: 373977:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 15728640(16777216) req@ffff904547f54050 x1642422187204960/t0(0) o3->4e14a970-7ca0-d5d3-629d-52842291c983@10.47.21.37@o2ib1:295/0 lens 488/440 e 2 to 0 dl 1566347455 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:30:34 dac-e-10 kernel: LustreError: 373977:0:(ldlm_lib.c:3268:target_bulk_io()) Skipped 1 previous similar message Aug 21 01:30:34 dac-e-10 kernel: Lustre: fs1-OST0074: Bulk IO read error with 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1), client will retry: rc -110 Aug 21 01:34:22 dac-e-10 kernel: Lustre: fs1-OST0072: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:34:22 dac-e-10 kernel: Lustre: Skipped 7 previous similar messages Aug 21 01:35:21 dac-e-10 kernel: Lustre: fs1-OST0073: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:36:37 dac-e-10 kernel: Lustre: fs1-OST0074: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:36:37 dac-e-10 kernel: Lustre: fs1-OST0074: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:36:37 dac-e-10 kernel: Lustre: Skipped 2 previous similar messages Aug 21 01:39:04 dac-e-10 kernel: Lustre: fs1-OST006d: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:39:04 dac-e-10 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:41:07 dac-e-10 kernel: Lustre: fs1-OST0071: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:41:07 dac-e-10 kernel: Lustre: fs1-OST0071: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:41:07 dac-e-10 kernel: Lustre: Skipped 2 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-1 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-04-16 21:34:08 BST, end at Wed 2019-08-21 11:51:58 BST. -- Aug 21 01:22:41 dac-e-1 kernel: Lustre: MGS: Connection restored to 387aa84c-fd4e-4bfd-4ca8-9fd17d62dec7 (at 10.47.21.37@o2ib1) Aug 21 01:22:41 dac-e-1 kernel: Lustre: Skipped 647 previous similar messages Aug 21 01:23:42 dac-e-1 kernel: Lustre: fs1-OST000b: Connection restored to bfa3419e-5f60-2112-6155-b28d2a4d1c73 (at 10.47.21.38@o2ib1) Aug 21 01:23:42 dac-e-1 kernel: Lustre: Skipped 138 previous similar messages Aug 21 01:29:21 dac-e-1 kernel: Lustre: fs1-OST0009: Connection restored to e4e19149-72fb-2eb1-0340-69c3579d93fe (at 10.47.21.31@o2ib1) Aug 21 01:29:21 dac-e-1 kernel: Lustre: Skipped 68 previous similar messages Aug 21 01:29:23 dac-e-1 kernel: LustreError: 176173:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8ad80f6ffe00 Aug 21 01:29:23 dac-e-1 kernel: LustreError: 176180:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8ad80f6ffe00 Aug 21 01:29:23 dac-e-1 kernel: Lustre: fs1-OST0005: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:29:23 dac-e-1 kernel: LustreError: 22021:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8ad7ffffd050 x1642422187212304/t0(0) o3->77aabb7c-29d1-0881-b291-653fb977b8da@10.47.21.36@o2ib1:212/0 lens 488/440 e 0 to 0 dl 1566347372 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:29:23 dac-e-1 kernel: Lustre: fs1-OST0005: Bulk IO read error with 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1), client will retry: rc -110 Aug 21 01:29:23 dac-e-1 kernel: Lustre: 404105:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347362/real 1566347363] req@ffff8ac10f809f80 x1642422111040992/t0(0) o105->fs1-OST0005@10.47.21.37@o2ib1:15/16 lens 360/224 e 0 to 1 dl 1566347369 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 21 01:29:23 dac-e-1 kernel: LustreError: 176168:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8ac02ca90e00 Aug 21 01:29:23 dac-e-1 kernel: LustreError: 176171:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8ac02ca90e00 Aug 21 01:29:23 dac-e-1 kernel: LustreError: 176170:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8ac02ca90e00 Aug 21 01:29:23 dac-e-1 kernel: LustreError: 176179:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8ac02ca90e00 Aug 21 01:29:24 dac-e-1 kernel: LustreError: 176170:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8abfaab18a00 Aug 21 01:29:24 dac-e-1 kernel: LustreError: 176171:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8abfaab18a00 Aug 21 01:29:25 dac-e-1 kernel: Lustre: fs1-OST0002: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:29:25 dac-e-1 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:29:25 dac-e-1 kernel: Lustre: MGS: Received new LWP connection from 10.47.20.69@o2ib1, removing former export from same NID Aug 21 01:29:25 dac-e-1 kernel: LustreError: 7704:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8ac0230b0050 x1642422187202368/t0(0) o3->ca24e700-cef1-0f8f-792b-2c007834cfd0@10.47.20.69@o2ib1:212/0 lens 488/440 e 0 to 0 dl 1566347372 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:29:25 dac-e-1 kernel: Lustre: fs1-OST000a: Bulk IO read error with ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1), client will retry: rc -110 Aug 21 01:29:26 dac-e-1 kernel: Lustre: 406537:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347366/real 1566347366] req@ffff8ac10f809f80 x1642422111047552/t0(0) o104->fs1-OST0000@10.47.20.69@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1566347373 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 21 01:29:26 dac-e-1 kernel: Lustre: 406537:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Aug 21 01:29:33 dac-e-1 kernel: Lustre: fs1-OST0000: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:29:33 dac-e-1 kernel: Lustre: Skipped 7 previous similar messages Aug 21 01:29:54 dac-e-1 kernel: Lustre: fs1-OST000b: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:29:54 dac-e-1 kernel: Lustre: fs1-OST000b: Connection restored to 8bef0c8e-594b-512d-b17f-d485c878ff27 (at 10.47.21.36@o2ib1) Aug 21 01:29:54 dac-e-1 kernel: Lustre: Skipped 45 previous similar messages Aug 21 01:30:02 dac-e-1 kernel: Lustre: fs1-MDT0000: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:30:35 dac-e-1 kernel: LustreError: 5364:0:(ldlm_lib.c:3259:target_bulk_io()) @@@ network error on bulk READ req@ffff8ac07b745850 x1642422187203744/t0(0) o3->4e14a970-7ca0-d5d3-629d-52842291c983@10.47.21.37@o2ib1:288/0 lens 488/440 e 3 to 0 dl 1566347448 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:30:35 dac-e-1 kernel: LustreError: 5357:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 15728640(16777216) req@ffff8ad84f365050 x1642422187214320/t0(0) o3->77aabb7c-29d1-0881-b291-653fb977b8da@10.47.21.36@o2ib1:288/0 lens 488/440 e 3 to 0 dl 1566347448 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:30:35 dac-e-1 kernel: Lustre: fs1-OST0006: Bulk IO read error with 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1), client will retry: rc -110 Aug 21 01:36:23 dac-e-1 kernel: Lustre: fs1-OST0007: Connection restored to 387aa84c-fd4e-4bfd-4ca8-9fd17d62dec7 (at 10.47.21.37@o2ib1) Aug 21 01:36:23 dac-e-1 kernel: Lustre: Skipped 10 previous similar messages Aug 21 01:36:40 dac-e-1 kernel: Lustre: fs1-OST0006: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:36:40 dac-e-1 kernel: Lustre: Skipped 5 previous similar messages Aug 21 01:38:04 dac-e-1 kernel: Lustre: fs1-OST0009: Connection restored to 8bef0c8e-594b-512d-b17f-d485c878ff27 (at 10.47.21.36@o2ib1) Aug 21 01:38:04 dac-e-1 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:42:06 dac-e-1 kernel: Lustre: fs1-OST0003: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:42:06 dac-e-1 kernel: Lustre: fs1-OST0003: Connection restored to 387aa84c-fd4e-4bfd-4ca8-9fd17d62dec7 (at 10.47.21.37@o2ib1) HOSTS ------------------------------------------------------------------------- dac-e-13 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:45 BST, end at Wed 2019-08-21 11:51:58 BST. -- Aug 21 01:22:41 dac-e-13 kernel: Lustre: fs1-MDT000c: Connection restored to (at 10.47.21.37@o2ib1) Aug 21 01:22:41 dac-e-13 kernel: Lustre: Skipped 598 previous similar messages Aug 21 01:23:42 dac-e-13 kernel: Lustre: fs1-OST0092: Connection restored to (at 10.47.21.34@o2ib1) Aug 21 01:23:42 dac-e-13 kernel: Lustre: fs1-OST0093: Connection restored to (at 10.47.21.34@o2ib1) Aug 21 01:23:42 dac-e-13 kernel: Lustre: Skipped 128 previous similar messages Aug 21 01:29:21 dac-e-13 kernel: Lustre: fs1-OST0094: Connection restored to (at 10.47.21.37@o2ib1) Aug 21 01:29:21 dac-e-13 kernel: Lustre: fs1-OST0091: Connection restored to (at 10.47.21.38@o2ib1) Aug 21 01:29:21 dac-e-13 kernel: Lustre: Skipped 69 previous similar messages Aug 21 01:29:24 dac-e-13 kernel: Lustre: 16006:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347362/real 1566347364] req@ffff9a4e12c51200 x1642422111956576/t0(0) o104->fs1-OST009a@10.47.21.36@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1566347369 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 21 01:29:24 dac-e-13 kernel: Lustre: 16006:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Aug 21 01:29:25 dac-e-13 kernel: Lustre: fs1-OST0091: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:29:25 dac-e-13 kernel: Lustre: 409653:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347365/real 1566347365] req@ffff9a654ca1a880 x1642422111959488/t0(0) o104->fs1-OST0090@10.47.20.69@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1566347372 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 21 01:29:25 dac-e-13 kernel: Lustre: 409653:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 1 previous similar message Aug 21 01:30:02 dac-e-13 kernel: Lustre: fs1-MDT000c: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:30:02 dac-e-13 kernel: Lustre: Skipped 6 previous similar messages Aug 21 01:34:22 dac-e-13 kernel: Lustre: fs1-OST0096: Connection restored to (at 10.47.21.36@o2ib1) Aug 21 01:34:22 dac-e-13 kernel: Lustre: Skipped 45 previous similar messages Aug 21 01:36:36 dac-e-13 kernel: Lustre: fs1-OST0092: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:36:36 dac-e-13 kernel: Lustre: Skipped 3 previous similar messages Aug 21 01:39:35 dac-e-13 kernel: Lustre: fs1-OST0093: Connection restored to (at 10.47.21.37@o2ib1) Aug 21 01:39:35 dac-e-13 kernel: Lustre: Skipped 2 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-14 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:46 BST, end at Wed 2019-08-21 11:51:58 BST. -- Aug 21 01:22:41 dac-e-14 kernel: Lustre: fs1-MDT000d: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:22:41 dac-e-14 kernel: Lustre: Skipped 598 previous similar messages Aug 21 01:23:42 dac-e-14 kernel: Lustre: fs1-OST00a4: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 21 01:23:42 dac-e-14 kernel: Lustre: Skipped 129 previous similar messages Aug 21 01:29:21 dac-e-14 kernel: Lustre: fs1-OST009e: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:29:21 dac-e-14 kernel: Lustre: Skipped 72 previous similar messages Aug 21 01:29:23 dac-e-14 kernel: Lustre: fs1-OST009d: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:29:23 dac-e-14 kernel: LustreError: 409878:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9776092b1000 Aug 21 01:29:23 dac-e-14 kernel: LustreError: 409822:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9776092b1000 Aug 21 01:29:23 dac-e-14 kernel: LustreError: 409821:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9776092b1000 Aug 21 01:29:23 dac-e-14 kernel: LustreError: 409823:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9776092b1000 Aug 21 01:29:24 dac-e-14 kernel: LustreError: 301103:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff9775419df850 x1642422187214112/t0(0) o3->77aabb7c-29d1-0881-b291-653fb977b8da@10.47.21.36@o2ib1:245/0 lens 488/440 e 0 to 0 dl 1566347405 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:29:24 dac-e-14 kernel: Lustre: fs1-OST009d: Bulk IO read error with 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1), client will retry: rc -110 Aug 21 01:29:24 dac-e-14 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:29:53 dac-e-14 kernel: Lustre: fs1-OST00a0: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:29:53 dac-e-14 kernel: Lustre: Skipped 3 previous similar messages Aug 21 01:29:53 dac-e-14 kernel: Lustre: fs1-OST00a0: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:29:53 dac-e-14 kernel: Lustre: Skipped 28 previous similar messages Aug 21 01:30:34 dac-e-14 kernel: LustreError: 300446:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 15728640(16777216) req@ffff97754d9a4050 x1642422187216464/t0(0) o3->77aabb7c-29d1-0881-b291-653fb977b8da@10.47.21.36@o2ib1:295/0 lens 488/440 e 2 to 0 dl 1566347455 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:30:34 dac-e-14 kernel: Lustre: fs1-OST00a1: Bulk IO read error with 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1), client will retry: rc -110 Aug 21 01:30:34 dac-e-14 kernel: LustreError: 300446:0:(ldlm_lib.c:3268:target_bulk_io()) Skipped 1 previous similar message Aug 21 01:33:47 dac-e-14 kernel: Lustre: fs1-OST00a1: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:33:47 dac-e-14 kernel: Lustre: Skipped 2 previous similar messages Aug 21 01:33:47 dac-e-14 kernel: Lustre: fs1-OST00a1: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:33:47 dac-e-14 kernel: Lustre: Skipped 8 previous similar messages Aug 21 01:36:37 dac-e-14 kernel: Lustre: fs1-OST009f: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:36:37 dac-e-14 kernel: Lustre: fs1-OST009f: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:36:37 dac-e-14 kernel: Lustre: Skipped 2 previous similar messages Aug 21 01:38:39 dac-e-14 kernel: Lustre: fs1-OST009e: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:42:06 dac-e-14 kernel: Lustre: fs1-OST009f: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:42:06 dac-e-14 kernel: Lustre: Skipped 1 previous similar message HOSTS ------------------------------------------------------------------------- dac-e-12 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:45 BST, end at Wed 2019-08-21 11:51:58 BST. -- Aug 21 01:22:41 dac-e-12 kernel: Lustre: fs1-MDT000b: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:22:41 dac-e-12 kernel: Lustre: Skipped 598 previous similar messages Aug 21 01:23:42 dac-e-12 kernel: Lustre: fs1-OST0087: Connection restored to efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) Aug 21 01:23:42 dac-e-12 kernel: Lustre: Skipped 127 previous similar messages Aug 21 01:29:14 dac-e-12 kernel: Lustre: fs1-OST008e: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 21 01:29:14 dac-e-12 kernel: Lustre: Skipped 72 previous similar messages Aug 21 01:29:23 dac-e-12 kernel: Lustre: fs1-OST008c: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:29:23 dac-e-12 kernel: Lustre: Skipped 2 previous similar messages Aug 21 01:29:23 dac-e-12 kernel: LustreError: 50957:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9d2ef74e8800 Aug 21 01:29:23 dac-e-12 kernel: LustreError: 50960:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9d2f73d17600 Aug 21 01:29:23 dac-e-12 kernel: LustreError: 50962:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9d2f73d17600 Aug 21 01:29:23 dac-e-12 kernel: LustreError: 50961:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9d2f73d17600 Aug 21 01:29:23 dac-e-12 kernel: LustreError: 50968:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9d2f73d17600 Aug 21 01:29:23 dac-e-12 kernel: LustreError: 50963:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9d2f73d17600 Aug 21 01:29:23 dac-e-12 kernel: LustreError: 50958:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9d2ef74e8800 Aug 21 01:29:23 dac-e-12 kernel: LustreError: 50967:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9d2ef74e8800 Aug 21 01:29:23 dac-e-12 kernel: LustreError: 50956:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9d2ef74e8800 Aug 21 01:29:23 dac-e-12 kernel: LustreError: 50959:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9d300fa2b000 Aug 21 01:29:23 dac-e-12 kernel: LustreError: 50957:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9d300fa2b000 Aug 21 01:29:23 dac-e-12 kernel: LustreError: 50957:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9d300fa2b000 Aug 21 01:29:23 dac-e-12 kernel: LustreError: 50967:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9d2ef74e8800 Aug 21 01:29:23 dac-e-12 kernel: LustreError: 50958:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9d300fa2b000 Aug 21 01:29:23 dac-e-12 kernel: LustreError: 50959:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9d300fa2b000 Aug 21 01:29:23 dac-e-12 kernel: LustreError: 50956:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9d300fa2b000 Aug 21 01:29:23 dac-e-12 kernel: LustreError: 50957:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9d300fa2b000 Aug 21 01:29:23 dac-e-12 kernel: LustreError: 50957:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9d300fa2b000 Aug 21 01:29:23 dac-e-12 kernel: LustreError: 50967:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9d300fa2b000 Aug 21 01:29:23 dac-e-12 kernel: LustreError: 50959:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9d2ef74e8800 Aug 21 01:29:23 dac-e-12 kernel: LustreError: 50956:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9d300fa2b000 Aug 21 01:29:24 dac-e-12 kernel: LustreError: 50956:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9d2ef74e8800 Aug 21 01:29:24 dac-e-12 kernel: LustreError: 50958:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9d300fa2b000 Aug 21 01:29:24 dac-e-12 kernel: LustreError: 50957:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9d300fa2b000 Aug 21 01:29:24 dac-e-12 kernel: LustreError: 50967:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9d300fa2b000 Aug 21 01:29:24 dac-e-12 kernel: LustreError: 50959:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9d300fa2b000 Aug 21 01:29:24 dac-e-12 kernel: LustreError: 50956:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9d300fa2b000 Aug 21 01:29:24 dac-e-12 kernel: LustreError: 50957:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9d300fa2b000 Aug 21 01:29:24 dac-e-12 kernel: Lustre: 400816:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347361/real 1566347364] req@ffff9d16d3c9f980 x1642422111952736/t0(0) o105->fs1-OST0084@10.47.21.37@o2ib1:15/16 lens 360/224 e 0 to 1 dl 1566347372 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 21 01:29:24 dac-e-12 kernel: Lustre: 400816:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 273 previous similar messages Aug 21 01:29:24 dac-e-12 kernel: Lustre: fs1-OST0089: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:29:24 dac-e-12 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:29:24 dac-e-12 kernel: LustreError: 22666:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff9d17c06c7850 x1642422187202224/t0(0) o3->ca24e700-cef1-0f8f-792b-2c007834cfd0@10.47.20.69@o2ib1:212/0 lens 488/440 e 0 to 0 dl 1566347372 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:29:24 dac-e-12 kernel: Lustre: fs1-OST0089: Bulk IO read error with ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1), client will retry: rc -110 Aug 21 01:29:24 dac-e-12 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:29:25 dac-e-12 kernel: LustreError: 22661:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff9d172a0cf050 x1642422187202352/t0(0) o3->ca24e700-cef1-0f8f-792b-2c007834cfd0@10.47.20.69@o2ib1:232/0 lens 488/440 e 1 to 0 dl 1566347392 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:29:25 dac-e-12 kernel: Lustre: fs1-OST0088: Bulk IO read error with ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1), client will retry: rc -110 Aug 21 01:30:02 dac-e-12 kernel: Lustre: fs1-OST008c: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:30:02 dac-e-12 kernel: Lustre: Skipped 2 previous similar messages Aug 21 01:30:02 dac-e-12 kernel: Lustre: fs1-OST008c: Connection restored to 9a3f8291-8d90-e819-b6ab-9b2c8e66825a (at 10.47.20.69@o2ib1) Aug 21 01:30:02 dac-e-12 kernel: Lustre: Skipped 33 previous similar messages Aug 21 01:30:34 dac-e-12 kernel: LustreError: 100173:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 15728640(16777216) req@ffff9d17ac3a9850 x1642422187207840/t0(0) o3->4e14a970-7ca0-d5d3-629d-52842291c983@10.47.21.37@o2ib1:284/0 lens 488/440 e 3 to 0 dl 1566347444 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:30:34 dac-e-12 kernel: Lustre: fs1-OST0084: Bulk IO read error with 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1), client will retry: rc -110 Aug 21 01:30:34 dac-e-12 kernel: LustreError: 44415:0:(ldlm_lib.c:3259:target_bulk_io()) @@@ network error on bulk READ req@ffff9d2f3ca4b050 x1642422187213408/t0(0) o3->77aabb7c-29d1-0881-b291-653fb977b8da@10.47.21.36@o2ib1:294/0 lens 488/440 e 2 to 0 dl 1566347454 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:30:34 dac-e-12 kernel: LustreError: 100173:0:(ldlm_lib.c:3268:target_bulk_io()) Skipped 1 previous similar message Aug 21 01:33:37 dac-e-12 kernel: Lustre: fs1-OST0087: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:33:37 dac-e-12 kernel: Lustre: fs1-OST0087: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:33:37 dac-e-12 kernel: Lustre: Skipped 4 previous similar messages Aug 21 01:35:21 dac-e-12 kernel: Lustre: fs1-OST0084: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:35:21 dac-e-12 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:35:21 dac-e-12 kernel: Lustre: fs1-OST0084: Connection restored to 9a3f8291-8d90-e819-b6ab-9b2c8e66825a (at 10.47.20.69@o2ib1) Aug 21 01:35:21 dac-e-12 kernel: Lustre: Skipped 3 previous similar messages Aug 21 01:41:30 dac-e-12 kernel: Lustre: fs1-OST008b: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:41:30 dac-e-12 kernel: Lustre: fs1-OST008b: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:41:30 dac-e-12 kernel: Lustre: Skipped 1 previous similar message HOSTS ------------------------------------------------------------------------- dac-e-11 ------------------------------------------------------------------------------- -- Logs begin at Thu 2019-03-21 15:42:22 GMT, end at Wed 2019-08-21 11:51:58 BST. -- Aug 21 01:22:41 dac-e-11 kernel: Lustre: fs1-MDT000a: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:22:41 dac-e-11 kernel: Lustre: Skipped 598 previous similar messages Aug 21 01:23:42 dac-e-11 kernel: Lustre: fs1-OST0078: Connection restored to efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) Aug 21 01:23:42 dac-e-11 kernel: Lustre: Skipped 125 previous similar messages Aug 21 01:29:14 dac-e-11 kernel: Lustre: fs1-OST0082: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 21 01:29:14 dac-e-11 kernel: Lustre: Skipped 70 previous similar messages Aug 21 01:29:23 dac-e-11 kernel: Lustre: 395247:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347362/real 1566347363] req@ffff983005e8c800 x1642422111954720/t0(0) o105->fs1-OST0083@10.47.21.37@o2ib1:15/16 lens 360/224 e 0 to 1 dl 1566347373 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 21 01:29:23 dac-e-11 kernel: Lustre: 395247:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Aug 21 01:29:23 dac-e-11 kernel: LustreError: 50846:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff98199e3c2e00 Aug 21 01:29:23 dac-e-11 kernel: LustreError: 50787:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff98199e3c2e00 Aug 21 01:29:23 dac-e-11 kernel: LustreError: 50789:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff98199e3c2e00 Aug 21 01:29:23 dac-e-11 kernel: LustreError: 50788:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff98199e3c2e00 Aug 21 01:29:23 dac-e-11 kernel: Lustre: fs1-OST007f: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:29:23 dac-e-11 kernel: LustreError: 29421:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff983030af0050 x1642422187202816/t0(0) o3->4e14a970-7ca0-d5d3-629d-52842291c983@10.47.21.37@o2ib1:244/0 lens 488/440 e 0 to 0 dl 1566347404 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:29:23 dac-e-11 kernel: Lustre: fs1-OST007f: Bulk IO read error with 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1), client will retry: rc -110 Aug 21 01:29:24 dac-e-11 kernel: LustreError: 50788:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff982fb30b4e00 Aug 21 01:29:24 dac-e-11 kernel: LustreError: 50793:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff984701b19600 Aug 21 01:29:24 dac-e-11 kernel: LustreError: 50793:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff984701b19600 Aug 21 01:29:24 dac-e-11 kernel: LustreError: 50793:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff984701b19600 Aug 21 01:29:24 dac-e-11 kernel: LustreError: 50793:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff984701b19600 Aug 21 01:29:24 dac-e-11 kernel: LustreError: 50793:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9848d0c61400 Aug 21 01:29:24 dac-e-11 kernel: LustreError: 50793:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9848d0c61400 Aug 21 01:29:24 dac-e-11 kernel: LustreError: 50793:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9848d0c61400 Aug 21 01:29:24 dac-e-11 kernel: LustreError: 50793:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9848d0c61400 Aug 21 01:29:24 dac-e-11 kernel: LustreError: 50789:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff982fb30b4e00 Aug 21 01:29:24 dac-e-11 kernel: LustreError: 50786:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff982fb30b4e00 Aug 21 01:29:24 dac-e-11 kernel: LustreError: 50787:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff982fb30b4e00 Aug 21 01:29:24 dac-e-11 kernel: LustreError: 50789:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff982fb30b4e00 Aug 21 01:29:24 dac-e-11 kernel: LustreError: 50846:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff982fb30b4e00 Aug 21 01:29:24 dac-e-11 kernel: LustreError: 50788:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff982fb30b4e00 Aug 21 01:29:24 dac-e-11 kernel: LustreError: 50786:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff982fb30b4e00 Aug 21 01:29:24 dac-e-11 kernel: LustreError: 19854:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff98474da5b050 x1642422187212272/t0(0) o3->77aabb7c-29d1-0881-b291-653fb977b8da@10.47.21.36@o2ib1:232/0 lens 488/440 e 1 to 0 dl 1566347392 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:29:24 dac-e-11 kernel: Lustre: fs1-OST0080: Bulk IO read error with 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1), client will retry: rc -110 Aug 21 01:29:25 dac-e-11 kernel: Lustre: 395435:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347365/real 1566347365] req@ffff98310a1c0480 x1642422111955760/t0(0) o104->fs1-OST0079@10.47.20.69@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1566347376 ref 1 fl Rpc:eX/2/ffffffff rc -11/-1 Aug 21 01:29:25 dac-e-11 kernel: Lustre: 395435:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Aug 21 01:29:26 dac-e-11 kernel: LustreError: 21941:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff982fbe1bb850 x1642422187201984/t0(0) o3->ca24e700-cef1-0f8f-792b-2c007834cfd0@10.47.20.69@o2ib1:244/0 lens 488/440 e 0 to 0 dl 1566347404 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:29:26 dac-e-11 kernel: Lustre: fs1-OST007f: Bulk IO read error with ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1), client will retry: rc -110 Aug 21 01:29:29 dac-e-11 kernel: Lustre: fs1-OST007a: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:29:29 dac-e-11 kernel: Lustre: Skipped 9 previous similar messages Aug 21 01:29:34 dac-e-11 kernel: Lustre: 395436:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566347363/real 1566347364] req@ffff9831191a4380 x1642422111955776/t0(0) o104->fs1-OST007c@10.47.20.69@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1566347374 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Aug 21 01:29:34 dac-e-11 kernel: Lustre: 395436:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Aug 21 01:29:37 dac-e-11 kernel: Lustre: fs1-OST0079: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:29:37 dac-e-11 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:29:54 dac-e-11 kernel: Lustre: fs1-OST0083: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:29:54 dac-e-11 kernel: Lustre: Skipped 41 previous similar messages Aug 21 01:30:02 dac-e-11 kernel: Lustre: fs1-MDT000a: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:30:34 dac-e-11 kernel: LustreError: 19947:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 15728640(16777216) req@ffff983030afb850 x1642422187201616/t0(0) o3->4e14a970-7ca0-d5d3-629d-52842291c983@10.47.21.37@o2ib1:294/0 lens 488/440 e 2 to 0 dl 1566347454 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:30:34 dac-e-11 kernel: LustreError: 20153:0:(ldlm_lib.c:3259:target_bulk_io()) @@@ network error on bulk READ req@ffff984750e28850 x1642422187212624/t0(0) o3->77aabb7c-29d1-0881-b291-653fb977b8da@10.47.21.36@o2ib1:287/0 lens 488/440 e 3 to 0 dl 1566347447 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:30:34 dac-e-11 kernel: Lustre: fs1-OST007e: Bulk IO read error with 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1), client will retry: rc -110 Aug 21 01:33:38 dac-e-11 kernel: Lustre: fs1-OST007f: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:33:38 dac-e-11 kernel: Lustre: Skipped 8 previous similar messages Aug 21 01:34:17 dac-e-11 kernel: Lustre: fs1-OST0080: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:34:17 dac-e-11 kernel: Lustre: Skipped 4 previous similar messages Aug 21 01:39:35 dac-e-11 kernel: Lustre: fs1-OST007b: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:39:35 dac-e-11 kernel: Lustre: Skipped 4 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-15 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:46 BST, end at Wed 2019-08-21 11:51:58 BST. -- Aug 21 01:22:41 dac-e-15 kernel: Lustre: fs1-MDT000e: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:22:41 dac-e-15 kernel: Lustre: Skipped 598 previous similar messages Aug 21 01:23:42 dac-e-15 kernel: Lustre: fs1-OST00a8: Connection restored to d51aba7f-3e9b-b409-76a8-ec3ceb1b1ca5 (at 10.47.21.35@o2ib1) Aug 21 01:23:42 dac-e-15 kernel: Lustre: fs1-OST00ac: Connection restored to a9c07af9-d96f-6905-bdea-228af9a88046 (at 10.47.21.32@o2ib1) Aug 21 01:23:42 dac-e-15 kernel: Lustre: Skipped 128 previous similar messages Aug 21 01:29:21 dac-e-15 kernel: Lustre: fs1-OST00ab: Connection restored to 3f28bd14-132c-e629-0b42-8ea0fdd5d2a4 (at 10.47.21.31@o2ib1) Aug 21 01:29:21 dac-e-15 kernel: Lustre: fs1-OST00ac: Connection restored to 3f28bd14-132c-e629-0b42-8ea0fdd5d2a4 (at 10.47.21.31@o2ib1) Aug 21 01:29:21 dac-e-15 kernel: Lustre: Skipped 68 previous similar messages Aug 21 01:29:24 dac-e-15 kernel: Lustre: 177424:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347364/real 1566347364] req@ffff995370962880 x1642422111957920/t0(0) o105->fs1-OST00b0@10.47.21.36@o2ib1:15/16 lens 360/224 e 0 to 1 dl 1566347371 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 21 01:29:24 dac-e-15 kernel: Lustre: 177424:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Aug 21 01:29:24 dac-e-15 kernel: Lustre: fs1-OST00aa: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:29:24 dac-e-15 kernel: LustreError: 347144:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff996a63758800 Aug 21 01:29:24 dac-e-15 kernel: LustreError: 347192:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff996a63758800 Aug 21 01:29:24 dac-e-15 kernel: LustreError: 347143:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff996a63758800 Aug 21 01:29:24 dac-e-15 kernel: LustreError: 347142:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff996a63758800 Aug 21 01:30:34 dac-e-15 kernel: LustreError: 258651:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 15728640(16777216) req@ffff9952467f7050 x1642422187208464/t0(0) o3->4e14a970-7ca0-d5d3-629d-52842291c983@10.47.21.37@o2ib1:289/0 lens 488/440 e 3 to 0 dl 1566347449 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:30:34 dac-e-15 kernel: Lustre: fs1-OST00b3: Bulk IO read error with 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1), client will retry: rc -110 Aug 21 01:30:34 dac-e-15 kernel: LustreError: 179334:0:(ldlm_lib.c:3259:target_bulk_io()) @@@ network error on bulk READ req@ffff996a507d7050 x1642422187215760/t0(0) o3->77aabb7c-29d1-0881-b291-653fb977b8da@10.47.21.36@o2ib1:295/0 lens 488/440 e 2 to 0 dl 1566347455 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:30:34 dac-e-15 kernel: LustreError: 258651:0:(ldlm_lib.c:3268:target_bulk_io()) Skipped 2 previous similar messages Aug 21 01:33:47 dac-e-15 kernel: Lustre: fs1-OST00ad: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:33:47 dac-e-15 kernel: Lustre: Skipped 34 previous similar messages Aug 21 01:36:37 dac-e-15 kernel: Lustre: fs1-OST00a8: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:36:37 dac-e-15 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:38:14 dac-e-15 kernel: Lustre: fs1-OST00b3: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:38:14 dac-e-15 kernel: Lustre: fs1-OST00b3: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:38:14 dac-e-15 kernel: Lustre: Skipped 3 previous similar messages Aug 21 01:38:38 dac-e-15 kernel: Lustre: fs1-OST00af: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting HOSTS ------------------------------------------------------------------------- dac-e-16 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:46 BST, end at Wed 2019-08-21 11:51:58 BST. -- Aug 21 01:22:41 dac-e-16 kernel: Lustre: fs1-MDT000f: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:22:41 dac-e-16 kernel: Lustre: Skipped 598 previous similar messages Aug 21 01:23:42 dac-e-16 kernel: Lustre: fs1-OST00bf: Connection restored to d51aba7f-3e9b-b409-76a8-ec3ceb1b1ca5 (at 10.47.21.35@o2ib1) Aug 21 01:23:42 dac-e-16 kernel: Lustre: fs1-OST00bd: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:23:42 dac-e-16 kernel: Lustre: Skipped 127 previous similar messages Aug 21 01:29:21 dac-e-16 kernel: Lustre: fs1-OST00bb: Connection restored to 9a3f8291-8d90-e819-b6ab-9b2c8e66825a (at 10.47.20.69@o2ib1) Aug 21 01:29:21 dac-e-16 kernel: Lustre: Skipped 71 previous similar messages Aug 21 01:29:23 dac-e-16 kernel: Lustre: 121375:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347362/real 1566347363] req@ffff8f91581bec00 x1642422111954288/t0(0) o105->fs1-OST00be@10.47.21.37@o2ib1:15/16 lens 360/224 e 0 to 1 dl 1566347369 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 21 01:29:23 dac-e-16 kernel: Lustre: 121375:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Aug 21 01:29:23 dac-e-16 kernel: LustreError: 65252:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8f90e12bf800 Aug 21 01:29:23 dac-e-16 kernel: LustreError: 65242:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8f90e12bf800 Aug 21 01:29:23 dac-e-16 kernel: LustreError: 65244:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8f90e12bf800 Aug 21 01:29:23 dac-e-16 kernel: Lustre: fs1-OST00b9: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:29:23 dac-e-16 kernel: LustreError: 42170:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8f901c6be050 x1642422187201920/t0(0) o3->4e14a970-7ca0-d5d3-629d-52842291c983@10.47.21.37@o2ib1:244/0 lens 488/440 e 0 to 0 dl 1566347404 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:29:23 dac-e-16 kernel: Lustre: fs1-OST00b9: Bulk IO read error with 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1), client will retry: rc -110 Aug 21 01:29:24 dac-e-16 kernel: LustreError: 65243:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8f9025a3d400 Aug 21 01:29:24 dac-e-16 kernel: LustreError: 65244:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8f9025a3d400 Aug 21 01:29:24 dac-e-16 kernel: LustreError: 65244:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8f9025a3d400 Aug 21 01:29:24 dac-e-16 kernel: LustreError: 65242:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8f9025a3d400 Aug 21 01:29:24 dac-e-16 kernel: LustreError: 65252:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8f9025a3d400 Aug 21 01:29:24 dac-e-16 kernel: LustreError: 65243:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8f9025a3d400 Aug 21 01:29:24 dac-e-16 kernel: LustreError: 65241:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8f9025a3d400 Aug 21 01:29:24 dac-e-16 kernel: Lustre: 26399:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347361/real 1566347364] req@ffff8f90540e1b00 x1642422111952192/t0(0) o104->fs1-OST00b7@10.47.20.69@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1566347372 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 21 01:29:24 dac-e-16 kernel: Lustre: 26399:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 1 previous similar message Aug 21 01:29:24 dac-e-16 kernel: LustreError: 65244:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8f9025a3d400 Aug 21 01:29:24 dac-e-16 kernel: LustreError: 65242:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8f9025a3d400 Aug 21 01:29:24 dac-e-16 kernel: LustreError: 65252:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8f9025a3d400 Aug 21 01:29:24 dac-e-16 kernel: LustreError: 65241:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8f9025a3d400 Aug 21 01:29:25 dac-e-16 kernel: LustreError: 42178:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8f901c6b7850 x1642422187202576/t0(0) o3->ca24e700-cef1-0f8f-792b-2c007834cfd0@10.47.20.69@o2ib1:212/0 lens 488/440 e 0 to 0 dl 1566347372 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:29:25 dac-e-16 kernel: Lustre: fs1-OST00bb: Bulk IO read error with ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1), client will retry: rc -110 Aug 21 01:29:35 dac-e-16 kernel: Lustre: 122066:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566347364/real 1566347364] req@ffff8f9052b1a400 x1642422111955536/t0(0) o104->fs1-OST00b9@10.47.20.69@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1566347375 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Aug 21 01:29:35 dac-e-16 kernel: Lustre: 122066:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Aug 21 01:29:54 dac-e-16 kernel: Lustre: fs1-OST00b7: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:29:54 dac-e-16 kernel: Lustre: Skipped 3 previous similar messages Aug 21 01:30:34 dac-e-16 kernel: LustreError: 42157:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 15728640(16777216) req@ffff8f901c6bb050 x1642422187203696/t0(0) o3->4e14a970-7ca0-d5d3-629d-52842291c983@10.47.21.37@o2ib1:295/0 lens 488/440 e 2 to 0 dl 1566347455 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:30:34 dac-e-16 kernel: Lustre: fs1-OST00b8: Bulk IO read error with ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1), client will retry: rc -110 Aug 21 01:30:34 dac-e-16 kernel: LustreError: 42157:0:(ldlm_lib.c:3268:target_bulk_io()) Skipped 2 previous similar messages Aug 21 01:33:47 dac-e-16 kernel: Lustre: fs1-OST00b5: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:33:47 dac-e-16 kernel: Lustre: fs1-OST00b5: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:33:47 dac-e-16 kernel: Lustre: Skipped 34 previous similar messages Aug 21 01:35:21 dac-e-16 kernel: Lustre: fs1-MDT000f: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:36:42 dac-e-16 kernel: Lustre: fs1-OST00b7: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:36:42 dac-e-16 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:36:42 dac-e-16 kernel: Lustre: fs1-OST00b7: Connection restored to 9a3f8291-8d90-e819-b6ab-9b2c8e66825a (at 10.47.20.69@o2ib1) Aug 21 01:36:42 dac-e-16 kernel: Lustre: Skipped 3 previous similar messages Aug 21 01:39:05 dac-e-16 kernel: Lustre: fs1-OST00bc: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:41:30 dac-e-16 kernel: Lustre: fs1-OST00bb: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:41:30 dac-e-16 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:41:30 dac-e-16 kernel: Lustre: fs1-OST00bb: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:41:30 dac-e-16 kernel: Lustre: Skipped 4 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-19 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:47 BST, end at Wed 2019-08-21 11:51:58 BST. -- Aug 21 01:22:41 dac-e-19 kernel: Lustre: fs1-MDT0012: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:22:41 dac-e-19 kernel: Lustre: Skipped 598 previous similar messages Aug 21 01:23:42 dac-e-19 kernel: Lustre: fs1-OST00dd: Connection restored to d51aba7f-3e9b-b409-76a8-ec3ceb1b1ca5 (at 10.47.21.35@o2ib1) Aug 21 01:23:42 dac-e-19 kernel: Lustre: Skipped 131 previous similar messages Aug 21 01:29:21 dac-e-19 kernel: Lustre: fs1-OST00e3: Connection restored to 9a3f8291-8d90-e819-b6ab-9b2c8e66825a (at 10.47.20.69@o2ib1) Aug 21 01:29:21 dac-e-19 kernel: Lustre: Skipped 70 previous similar messages Aug 21 01:29:23 dac-e-19 kernel: Lustre: fs1-OST00da: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:29:23 dac-e-19 kernel: Lustre: 383578:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347363/real 1566347363] req@ffff9c9492680d80 x1642422111951008/t0(0) o104->fs1-OST00d9@10.47.20.69@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1566347374 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 21 01:29:23 dac-e-19 kernel: Lustre: 383578:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Aug 21 01:29:23 dac-e-19 kernel: LustreError: 113384:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9c949064ac00 Aug 21 01:29:23 dac-e-19 kernel: LustreError: 113383:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9ca903b52000 Aug 21 01:29:23 dac-e-19 kernel: LustreError: 113384:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9ca903b52000 Aug 21 01:29:23 dac-e-19 kernel: LustreError: 113381:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9ca903b52000 Aug 21 01:29:23 dac-e-19 kernel: LustreError: 113392:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9ca903b52000 Aug 21 01:29:23 dac-e-19 kernel: LustreError: 113383:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9ca95a6bfa00 Aug 21 01:29:23 dac-e-19 kernel: LustreError: 113382:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9ca95a6bfa00 Aug 21 01:29:23 dac-e-19 kernel: LustreError: 392677:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff9ca8c92d6050 x1642422187202528/t0(0) o3->ca24e700-cef1-0f8f-792b-2c007834cfd0@10.47.20.69@o2ib1:244/0 lens 488/440 e 0 to 0 dl 1566347404 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:29:23 dac-e-19 kernel: LustreError: 392677:0:(ldlm_lib.c:3253:target_bulk_io()) Skipped 2 previous similar messages Aug 21 01:29:23 dac-e-19 kernel: LustreError: 113384:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9ca95a6bfa00 Aug 21 01:29:23 dac-e-19 kernel: Lustre: fs1-OST00e0: Bulk IO read error with ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1), client will retry: rc -110 Aug 21 01:29:25 dac-e-19 kernel: Lustre: 383599:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347365/real 1566347365] req@ffff9c94933b6c00 x1642422111955600/t0(0) o104->fs1-OST00db@10.47.20.69@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1566347372 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 21 01:29:26 dac-e-19 kernel: Lustre: fs1-OST00e3: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:29:26 dac-e-19 kernel: Lustre: Skipped 5 previous similar messages Aug 21 01:29:26 dac-e-19 kernel: LustreError: 392576:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff9ca8c230f850 x1642422187202448/t0(0) o3->ca24e700-cef1-0f8f-792b-2c007834cfd0@10.47.20.69@o2ib1:212/0 lens 488/440 e 0 to 0 dl 1566347372 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:29:26 dac-e-19 kernel: Lustre: fs1-OST00e3: Bulk IO read error with ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1), client will retry: rc -110 Aug 21 01:29:37 dac-e-19 kernel: Lustre: fs1-OST00d9: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:29:54 dac-e-19 kernel: Lustre: fs1-OST00dc: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:29:54 dac-e-19 kernel: Lustre: Skipped 26 previous similar messages Aug 21 01:30:02 dac-e-19 kernel: Lustre: fs1-OST00e0: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:30:02 dac-e-19 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:30:34 dac-e-19 kernel: LustreError: 392630:0:(ldlm_lib.c:3259:target_bulk_io()) @@@ network error on bulk READ req@ffff9ca8c230b050 x1642422187202864/t0(0) o3->4e14a970-7ca0-d5d3-629d-52842291c983@10.47.21.37@o2ib1:287/0 lens 488/440 e 3 to 0 dl 1566347447 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:30:34 dac-e-19 kernel: LustreError: 431996:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 15728640(16777216) req@ffff9ca8c57c3050 x1642422187208464/t0(0) o3->ca24e700-cef1-0f8f-792b-2c007834cfd0@10.47.20.69@o2ib1:298/0 lens 488/440 e 2 to 0 dl 1566347458 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:30:34 dac-e-19 kernel: Lustre: fs1-OST00db: Bulk IO read error with ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1), client will retry: rc -110 Aug 21 01:33:47 dac-e-19 kernel: Lustre: fs1-OST00d9: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:33:47 dac-e-19 kernel: Lustre: Skipped 8 previous similar messages Aug 21 01:35:21 dac-e-19 kernel: Lustre: fs1-OST00db: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:36:23 dac-e-19 kernel: Lustre: fs1-OST00db: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:36:23 dac-e-19 kernel: Lustre: fs1-OST00db: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:36:23 dac-e-19 kernel: Lustre: Skipped 2 previous similar messages Aug 21 01:44:03 dac-e-19 kernel: Lustre: fs1-OST00e3: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 21 01:44:03 dac-e-19 kernel: Lustre: Skipped 5 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-17 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:47 BST, end at Wed 2019-08-21 11:51:58 BST. -- Aug 21 01:22:41 dac-e-17 kernel: Lustre: fs1-MDT0010: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:22:41 dac-e-17 kernel: Lustre: Skipped 598 previous similar messages Aug 21 01:23:42 dac-e-17 kernel: Lustre: fs1-OST00c6: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:23:42 dac-e-17 kernel: Lustre: Skipped 129 previous similar messages Aug 21 01:29:21 dac-e-17 kernel: Lustre: fs1-OST00c6: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:29:21 dac-e-17 kernel: Lustre: fs1-OST00c2: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:29:21 dac-e-17 kernel: Lustre: Skipped 69 previous similar messages Aug 21 01:29:23 dac-e-17 kernel: Lustre: fs1-OST00c8: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:29:23 dac-e-17 kernel: LustreError: 126116:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9f7cafb95a00 Aug 21 01:29:23 dac-e-17 kernel: LustreError: 126061:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9f7cafb95a00 Aug 21 01:29:23 dac-e-17 kernel: LustreError: 126060:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9f7cafb95a00 Aug 21 01:29:23 dac-e-17 kernel: LustreError: 126062:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9f7cafb95a00 Aug 21 01:29:23 dac-e-17 kernel: LustreError: 126059:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9f65cf200200 Aug 21 01:29:23 dac-e-17 kernel: LustreError: 126116:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9f65cf200200 Aug 21 01:29:23 dac-e-17 kernel: LustreError: 126060:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9f65cf200200 Aug 21 01:29:23 dac-e-17 kernel: LustreError: 126062:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9f65cf200200 Aug 21 01:29:23 dac-e-17 kernel: LustreError: 126061:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9f656f629c00 Aug 21 01:29:23 dac-e-17 kernel: LustreError: 126059:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9f656f629c00 Aug 21 01:29:23 dac-e-17 kernel: LustreError: 126061:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9f656f629c00 Aug 21 01:29:23 dac-e-17 kernel: LustreError: 126062:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9f656f629c00 Aug 21 01:29:23 dac-e-17 kernel: LustreError: 126059:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9f7cbb072400 Aug 21 01:29:23 dac-e-17 kernel: LustreError: 126061:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9f7cbb072400 Aug 21 01:29:24 dac-e-17 kernel: LustreError: 126061:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9f7cbb072400 Aug 21 01:29:24 dac-e-17 kernel: LustreError: 124666:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff9f64d44c5050 x1642422187203648/t0(0) o3->4e14a970-7ca0-d5d3-629d-52842291c983@10.47.21.37@o2ib1:245/0 lens 488/440 e 0 to 0 dl 1566347405 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:29:24 dac-e-17 kernel: LustreError: 124666:0:(ldlm_lib.c:3253:target_bulk_io()) Skipped 1 previous similar message Aug 21 01:29:24 dac-e-17 kernel: Lustre: fs1-OST00c2: Bulk IO read error with 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1), client will retry: rc -110 Aug 21 01:29:24 dac-e-17 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:29:24 dac-e-17 kernel: LustreError: 124672:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff9f651319e050 x1642422187202512/t0(0) o3->ca24e700-cef1-0f8f-792b-2c007834cfd0@10.47.20.69@o2ib1:212/0 lens 488/440 e 0 to 0 dl 1566347372 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:29:25 dac-e-17 kernel: LustreError: 126059:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff9f7cafb95a00 Aug 21 01:29:25 dac-e-17 kernel: Lustre: fs1-OST00c9: Bulk IO read error with ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1), client will retry: rc -110 Aug 21 01:29:25 dac-e-17 kernel: Lustre: fs1-OST00c5: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:29:25 dac-e-17 kernel: Lustre: Skipped 3 previous similar messages Aug 21 01:29:25 dac-e-17 kernel: LustreError: 124673:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff9f65bc54d850 x1642422187202560/t0(0) o3->ca24e700-cef1-0f8f-792b-2c007834cfd0@10.47.20.69@o2ib1:244/0 lens 488/440 e 0 to 0 dl 1566347404 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:29:26 dac-e-17 kernel: LNetError: 126058:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-103, 0) Aug 21 01:29:26 dac-e-17 kernel: LustreError: 126058:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9f64bfdc7800 Aug 21 01:29:26 dac-e-17 kernel: LustreError: 126058:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9f64bfdc7800 Aug 21 01:29:26 dac-e-17 kernel: LustreError: 126058:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9f64bfdc7800 Aug 21 01:29:26 dac-e-17 kernel: LustreError: 126058:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9f64bfdc7800 Aug 21 01:29:26 dac-e-17 kernel: LustreError: 126058:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9f64bfdc7800 Aug 21 01:29:26 dac-e-17 kernel: LustreError: 126058:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9f64bfdc7800 Aug 21 01:29:26 dac-e-17 kernel: LustreError: 126058:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9f64bfdc7800 Aug 21 01:29:26 dac-e-17 kernel: LustreError: 126058:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff9f64bfdc7800 Aug 21 01:29:26 dac-e-17 kernel: Lustre: fs1-OST00c9: Bulk IO read error with ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1), client will retry: rc -110 Aug 21 01:29:26 dac-e-17 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:29:31 dac-e-17 kernel: Lustre: 204772:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566347364/real 1566347365] req@ffff9f64fc24ba80 x1642422111958448/t0(0) o104->fs1-OST00c1@10.47.20.69@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1566347371 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Aug 21 01:29:31 dac-e-17 kernel: Lustre: 204772:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Aug 21 01:30:02 dac-e-17 kernel: Lustre: fs1-MDT0010: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:30:02 dac-e-17 kernel: Lustre: Skipped 4 previous similar messages Aug 21 01:30:34 dac-e-17 kernel: LustreError: 124671:0:(ldlm_lib.c:3259:target_bulk_io()) @@@ network error on bulk READ req@ffff9f6512078050 x1642422187203712/t0(0) o3->4e14a970-7ca0-d5d3-629d-52842291c983@10.47.21.37@o2ib1:295/0 lens 488/440 e 2 to 0 dl 1566347455 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:30:34 dac-e-17 kernel: Lustre: fs1-OST00c6: Bulk IO read error with 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1), client will retry: rc -110 Aug 21 01:33:47 dac-e-17 kernel: Lustre: fs1-OST00c1: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:33:47 dac-e-17 kernel: Lustre: Skipped 37 previous similar messages Aug 21 01:34:30 dac-e-17 kernel: Lustre: fs1-OST00c6: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:34:30 dac-e-17 kernel: Lustre: Skipped 3 previous similar messages Aug 21 01:37:30 dac-e-17 kernel: Lustre: fs1-OST00ca: Connection restored to 9a3f8291-8d90-e819-b6ab-9b2c8e66825a (at 10.47.20.69@o2ib1) Aug 21 01:37:30 dac-e-17 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:44:03 dac-e-17 kernel: Lustre: fs1-OST00c6: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 21 01:44:03 dac-e-17 kernel: Lustre: Skipped 3 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-2 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:46 BST, end at Wed 2019-08-21 11:51:58 BST. -- Aug 21 01:22:41 dac-e-2 kernel: Lustre: fs1-MDT0001: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:22:41 dac-e-2 kernel: Lustre: Skipped 598 previous similar messages Aug 21 01:23:42 dac-e-2 kernel: Lustre: fs1-OST0017: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 21 01:23:42 dac-e-2 kernel: Lustre: Skipped 127 previous similar messages Aug 21 01:29:21 dac-e-2 kernel: Lustre: fs1-OST0012: Connection restored to 1c9f1f8c-ae8f-76b3-441d-4d74e9e3a7df (at 10.47.21.38@o2ib1) Aug 21 01:29:21 dac-e-2 kernel: Lustre: Skipped 75 previous similar messages Aug 21 01:29:23 dac-e-2 kernel: Lustre: fs1-OST000e: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:29:23 dac-e-2 kernel: Lustre: 15862:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347361/real 1566347363] req@ffff8e288b78bf00 x1642422111951456/t0(0) o105->fs1-OST000f@10.47.20.69@o2ib1:15/16 lens 360/224 e 0 to 1 dl 1566347372 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 21 01:29:23 dac-e-2 kernel: Lustre: 15862:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Aug 21 01:29:24 dac-e-2 kernel: Lustre: 15857:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347364/real 1566347364] req@ffff8e104dc22400 x1642422111957168/t0(0) o105->fs1-OST0011@10.47.21.36@o2ib1:15/16 lens 360/224 e 0 to 1 dl 1566347371 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 21 01:29:24 dac-e-2 kernel: Lustre: 15857:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 1 previous similar message Aug 21 01:29:28 dac-e-2 kernel: Lustre: fs1-OST000f: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:29:28 dac-e-2 kernel: Lustre: Skipped 2 previous similar messages Aug 21 01:29:29 dac-e-2 kernel: Lustre: 16073:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566347362/real 1566347364] req@ffff8e0fedb50000 x1642422111954272/t0(0) o104->fs1-OST0013@10.47.20.69@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1566347369 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Aug 21 01:29:54 dac-e-2 kernel: Lustre: fs1-OST0017: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:29:54 dac-e-2 kernel: Lustre: Skipped 27 previous similar messages Aug 21 01:30:34 dac-e-2 kernel: LustreError: 79304:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 14680064(16777216) req@ffff8e2767f46050 x1642422187218640/t0(0) o3->77aabb7c-29d1-0881-b291-653fb977b8da@10.47.21.36@o2ib1:296/0 lens 488/440 e 2 to 0 dl 1566347456 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:30:34 dac-e-2 kernel: Lustre: fs1-OST000e: Bulk IO read error with 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1), client will retry: rc -110 Aug 21 01:36:18 dac-e-2 kernel: Lustre: fs1-OST0010: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:36:18 dac-e-2 kernel: Lustre: fs1-OST0010: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:36:18 dac-e-2 kernel: Lustre: Skipped 4 previous similar messages Aug 21 01:38:39 dac-e-2 kernel: Lustre: fs1-OST000e: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:38:39 dac-e-2 kernel: Lustre: fs1-OST000e: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:38:39 dac-e-2 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:44:03 dac-e-2 kernel: Lustre: fs1-OST0012: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 21 01:44:03 dac-e-2 kernel: Lustre: Skipped 1 previous similar message HOSTS ------------------------------------------------------------------------- dac-e-18 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:46 BST, end at Wed 2019-08-21 11:51:58 BST. -- Aug 21 01:22:41 dac-e-18 kernel: Lustre: fs1-MDT0011: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:22:41 dac-e-18 kernel: Lustre: Skipped 598 previous similar messages Aug 21 01:23:42 dac-e-18 kernel: Lustre: fs1-OST00cc: Connection restored to d51aba7f-3e9b-b409-76a8-ec3ceb1b1ca5 (at 10.47.21.35@o2ib1) Aug 21 01:23:42 dac-e-18 kernel: Lustre: Skipped 129 previous similar messages Aug 21 01:29:21 dac-e-18 kernel: Lustre: fs1-OST00d5: Connection restored to ba6d10d8-29ae-af16-bb29-fe0009074454 (at 10.47.21.33@o2ib1) Aug 21 01:29:21 dac-e-18 kernel: Lustre: Skipped 70 previous similar messages Aug 21 01:29:23 dac-e-18 kernel: Lustre: 72735:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347361/real 1566347363] req@ffff96d48b659f80 x1642422111953280/t0(0) o105->fs1-OST00d4@10.47.20.69@o2ib1:15/16 lens 360/224 e 0 to 1 dl 1566347368 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 21 01:29:23 dac-e-18 kernel: Lustre: 72735:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Aug 21 01:29:25 dac-e-18 kernel: Lustre: fs1-OST00cd: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:29:54 dac-e-18 kernel: Lustre: fs1-OST00d7: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:29:54 dac-e-18 kernel: Lustre: Skipped 2 previous similar messages Aug 21 01:30:02 dac-e-18 kernel: Lustre: fs1-OST00d4: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:30:02 dac-e-18 kernel: Lustre: fs1-OST00d4: Connection restored to 9a3f8291-8d90-e819-b6ab-9b2c8e66825a (at 10.47.20.69@o2ib1) Aug 21 01:30:02 dac-e-18 kernel: Lustre: Skipped 31 previous similar messages Aug 21 01:30:35 dac-e-18 kernel: LustreError: 157349:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 15728640(16777216) req@ffff96bb5e1f8850 x1642422187208288/t0(0) o3->ca24e700-cef1-0f8f-792b-2c007834cfd0@10.47.20.69@o2ib1:291/0 lens 488/440 e 3 to 0 dl 1566347451 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:30:35 dac-e-18 kernel: Lustre: fs1-OST00d6: Bulk IO read error with ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1), client will retry: rc -110 Aug 21 01:30:35 dac-e-18 kernel: Lustre: Skipped 2 previous similar messages Aug 21 01:33:38 dac-e-18 kernel: Lustre: fs1-OST00d3: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:33:38 dac-e-18 kernel: Lustre: Skipped 4 previous similar messages Aug 21 01:35:21 dac-e-18 kernel: Lustre: fs1-OST00d1: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:35:21 dac-e-18 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:37:30 dac-e-18 kernel: Lustre: fs1-OST00d6: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:37:30 dac-e-18 kernel: Lustre: fs1-OST00d6: Connection restored to 9a3f8291-8d90-e819-b6ab-9b2c8e66825a (at 10.47.20.69@o2ib1) Aug 21 01:37:30 dac-e-18 kernel: Lustre: Skipped 3 previous similar messages Aug 21 01:44:03 dac-e-18 kernel: Lustre: fs1-OST00d2: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 21 01:44:03 dac-e-18 kernel: Lustre: Skipped 1 previous similar message HOSTS ------------------------------------------------------------------------- dac-e-20 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:47 BST, end at Wed 2019-08-21 11:51:58 BST. -- Aug 21 01:22:41 dac-e-20 kernel: Lustre: fs1-MDT0013: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:22:41 dac-e-20 kernel: Lustre: Skipped 598 previous similar messages Aug 21 01:23:42 dac-e-20 kernel: Lustre: fs1-OST00e8: Connection restored to 3f28bd14-132c-e629-0b42-8ea0fdd5d2a4 (at 10.47.21.31@o2ib1) Aug 21 01:23:42 dac-e-20 kernel: Lustre: Skipped 128 previous similar messages Aug 21 01:29:21 dac-e-20 kernel: Lustre: fs1-OST00e9: Connection restored to efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) Aug 21 01:29:21 dac-e-20 kernel: Lustre: Skipped 69 previous similar messages Aug 21 01:29:23 dac-e-20 kernel: Lustre: fs1-OST00ee: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:29:23 dac-e-20 kernel: LustreError: 226999:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff91c0db43ae00 Aug 21 01:29:23 dac-e-20 kernel: LustreError: 226998:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff91c0db43ae00 Aug 21 01:29:23 dac-e-20 kernel: LustreError: 227053:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff91c0db43ae00 Aug 21 01:29:23 dac-e-20 kernel: LustreError: 226996:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff91c0db43ae00 Aug 21 01:29:23 dac-e-20 kernel: LustreError: 226997:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff91c0db43ae00 Aug 21 01:29:23 dac-e-20 kernel: LustreError: 226999:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff91c0db43ae00 Aug 21 01:29:23 dac-e-20 kernel: LustreError: 226998:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff91c0db43ae00 Aug 21 01:29:23 dac-e-20 kernel: LustreError: 226997:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff91c0db43ae00 Aug 21 01:29:23 dac-e-20 kernel: LustreError: 226998:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff91c0db43ae00 Aug 21 01:29:23 dac-e-20 kernel: LustreError: 226996:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff91c0db43ae00 Aug 21 01:29:23 dac-e-20 kernel: LustreError: 227053:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff91c0ccfd1c00 Aug 21 01:29:23 dac-e-20 kernel: LustreError: 226999:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff91c0db43ae00 Aug 21 01:29:23 dac-e-20 kernel: LustreError: 226997:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff91c0d3c14c00 Aug 21 01:29:23 dac-e-20 kernel: Lustre: 15020:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347362/real 1566347363] req@ffff91d849850000 x1642422111951616/t0(0) o105->fs1-OST00eb@10.47.21.37@o2ib1:15/16 lens 360/224 e 0 to 1 dl 1566347369 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 21 01:29:23 dac-e-20 kernel: Lustre: 15020:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Aug 21 01:29:23 dac-e-20 kernel: LustreError: 227053:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff91c0d3c14c00 Aug 21 01:29:23 dac-e-20 kernel: LustreError: 226998:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff91c0d3c14c00 Aug 21 01:29:23 dac-e-20 kernel: LustreError: 226996:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff91c0d3c14c00 Aug 21 01:29:24 dac-e-20 kernel: LustreError: 16927:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff91c11ca7f050 x1642422187203376/t0(0) o3->4e14a970-7ca0-d5d3-629d-52842291c983@10.47.21.37@o2ib1:213/0 lens 488/440 e 0 to 0 dl 1566347373 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:29:24 dac-e-20 kernel: Lustre: fs1-OST00e5: Bulk IO read error with 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1), client will retry: rc -110 Aug 21 01:29:24 dac-e-20 kernel: LustreError: 226999:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff91d8b36d4200 Aug 21 01:29:24 dac-e-20 kernel: LustreError: 226997:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff91d8b36d4200 Aug 21 01:29:24 dac-e-20 kernel: LustreError: 226999:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff91d8b36d4200 Aug 21 01:29:24 dac-e-20 kernel: LustreError: 226996:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff91d8b36d4200 Aug 21 01:29:24 dac-e-20 kernel: LustreError: 226998:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff91d8b36d4200 Aug 21 01:29:24 dac-e-20 kernel: LustreError: 227053:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff91d8b36d4200 Aug 21 01:29:24 dac-e-20 kernel: LustreError: 87281:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff91c11ca7e850 x1642422187201824/t0(0) o3->4e14a970-7ca0-d5d3-629d-52842291c983@10.47.21.37@o2ib1:244/0 lens 488/440 e 0 to 0 dl 1566347404 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:29:24 dac-e-20 kernel: Lustre: fs1-OST00e6: Bulk IO read error with 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1), client will retry: rc -110 Aug 21 01:29:25 dac-e-20 kernel: Lustre: fs1-OST00e4: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:29:25 dac-e-20 kernel: Lustre: Skipped 2 previous similar messages Aug 21 01:29:26 dac-e-20 kernel: LustreError: 87289:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff91c0fa8b5050 x1642422187202464/t0(0) o3->ca24e700-cef1-0f8f-792b-2c007834cfd0@10.47.20.69@o2ib1:212/0 lens 488/440 e 0 to 0 dl 1566347372 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:29:26 dac-e-20 kernel: Lustre: fs1-OST00e4: Bulk IO read error with ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1), client will retry: rc -110 Aug 21 01:29:28 dac-e-20 kernel: Lustre: fs1-OST00e8: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:29:54 dac-e-20 kernel: Lustre: fs1-OST00e8: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:29:54 dac-e-20 kernel: Lustre: Skipped 25 previous similar messages Aug 21 01:30:02 dac-e-20 kernel: Lustre: fs1-MDT0013: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:30:03 dac-e-20 kernel: LustreError: 87265:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff91c0b8525850 x1642422187202656/t0(0) o3->ca24e700-cef1-0f8f-792b-2c007834cfd0@10.47.20.69@o2ib1:262/0 lens 488/440 e 2 to 0 dl 1566347422 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:30:03 dac-e-20 kernel: Lustre: fs1-OST00e8: Bulk IO read error with ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1), client will retry: rc -110 Aug 21 01:30:35 dac-e-20 kernel: LustreError: 87919:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 15728640(16777216) req@ffff91c2d9cef050 x1642422187215456/t0(0) o3->77aabb7c-29d1-0881-b291-653fb977b8da@10.47.21.36@o2ib1:288/0 lens 488/440 e 3 to 0 dl 1566347448 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:30:35 dac-e-20 kernel: Lustre: fs1-OST00eb: Bulk IO read error with 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1), client will retry: rc -110 Aug 21 01:34:30 dac-e-20 kernel: Lustre: fs1-OST00eb: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:34:30 dac-e-20 kernel: Lustre: Skipped 5 previous similar messages Aug 21 01:34:30 dac-e-20 kernel: Lustre: fs1-OST00eb: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:34:30 dac-e-20 kernel: Lustre: Skipped 11 previous similar messages Aug 21 01:35:21 dac-e-20 kernel: Lustre: fs1-OST00ec: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:37:30 dac-e-20 kernel: Lustre: fs1-OST00ee: Connection restored to 9a3f8291-8d90-e819-b6ab-9b2c8e66825a (at 10.47.20.69@o2ib1) Aug 21 01:37:30 dac-e-20 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:39:56 dac-e-20 kernel: Lustre: fs1-OST00eb: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:44:03 dac-e-20 kernel: Lustre: fs1-OST00ea: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 21 01:44:03 dac-e-20 kernel: Lustre: Skipped 3 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-21 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:47 BST, end at Wed 2019-08-21 11:51:58 BST. -- Aug 21 01:22:41 dac-e-21 kernel: Lustre: fs1-MDT0014: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:22:41 dac-e-21 kernel: Lustre: Skipped 598 previous similar messages Aug 21 01:23:42 dac-e-21 kernel: Lustre: fs1-OST00f9: Connection restored to ba6d10d8-29ae-af16-bb29-fe0009074454 (at 10.47.21.33@o2ib1) Aug 21 01:23:42 dac-e-21 kernel: Lustre: Skipped 128 previous similar messages Aug 21 01:29:21 dac-e-21 kernel: Lustre: fs1-OST00f0: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:29:21 dac-e-21 kernel: Lustre: fs1-OST00f9: Connection restored to a9c07af9-d96f-6905-bdea-228af9a88046 (at 10.47.21.32@o2ib1) Aug 21 01:29:21 dac-e-21 kernel: Lustre: Skipped 73 previous similar messages Aug 21 01:29:23 dac-e-21 kernel: Lustre: 15170:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347361/real 1566347363] req@ffff957c52123600 x1642422111955024/t0(0) o105->fs1-OST00fb@10.47.21.37@o2ib1:15/16 lens 360/224 e 0 to 1 dl 1566347372 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 21 01:29:23 dac-e-21 kernel: Lustre: 15170:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 1 previous similar message Aug 21 01:29:23 dac-e-21 kernel: Lustre: fs1-MDT0014: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:29:25 dac-e-21 kernel: Lustre: fs1-OST00f5: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:29:25 dac-e-21 kernel: LNetError: 33394:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-103, 0) Aug 21 01:29:25 dac-e-21 kernel: LustreError: 33394:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff957c026bc800 Aug 21 01:29:25 dac-e-21 kernel: LNet: 33396:0:(o2iblnd_cb.c:413:kiblnd_handle_rx()) PUT_NACK from 10.47.20.69@o2ib1 Aug 21 01:29:25 dac-e-21 kernel: LustreError: 33394:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff957c026bc800 Aug 21 01:29:25 dac-e-21 kernel: LustreError: 33394:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff957c026bc800 Aug 21 01:29:25 dac-e-21 kernel: LustreError: 392738:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff957c17fd3050 x1642422187205536/t0(0) o3->ca24e700-cef1-0f8f-792b-2c007834cfd0@10.47.20.69@o2ib1:246/0 lens 488/440 e 0 to 0 dl 1566347406 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:29:25 dac-e-21 kernel: LustreError: 392738:0:(ldlm_lib.c:3253:target_bulk_io()) Skipped 1 previous similar message Aug 21 01:29:25 dac-e-21 kernel: LustreError: 33394:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff957c026bc800 Aug 21 01:29:25 dac-e-21 kernel: LustreError: 33394:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff957b333a2c00 Aug 21 01:29:25 dac-e-21 kernel: LustreError: 33394:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff957b333a2c00 Aug 21 01:29:25 dac-e-21 kernel: LustreError: 33394:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff957b333a2c00 Aug 21 01:29:25 dac-e-21 kernel: LustreError: 33394:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff957b333a2c00 Aug 21 01:29:25 dac-e-21 kernel: Lustre: 16391:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347365/real 1566347365] req@ffff9593d890e780 x1642422111958416/t0(0) o104->fs1-OST00f5@10.47.20.69@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1566347376 ref 1 fl Rpc:eXS/0/ffffffff rc -11/-1 Aug 21 01:29:25 dac-e-21 kernel: Lustre: 16391:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Aug 21 01:29:25 dac-e-21 kernel: LustreError: 33394:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff957c026bc800 Aug 21 01:29:25 dac-e-21 kernel: LustreError: 33397:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff957b7b7d2000 Aug 21 01:29:25 dac-e-21 kernel: LustreError: 33406:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff957b7b7d2000 Aug 21 01:29:25 dac-e-21 kernel: LustreError: 33394:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff957c026bc800 Aug 21 01:29:25 dac-e-21 kernel: LustreError: 33394:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff957c026bc800 Aug 21 01:29:25 dac-e-21 kernel: LustreError: 33394:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff957c026bc800 Aug 21 01:29:25 dac-e-21 kernel: LustreError: 33394:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff957b333a2c00 Aug 21 01:29:25 dac-e-21 kernel: Lustre: fs1-OST00f9: Bulk IO read error with ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1), client will retry: rc -110 Aug 21 01:29:25 dac-e-21 kernel: Lustre: Skipped 2 previous similar messages Aug 21 01:29:25 dac-e-21 kernel: LustreError: 33394:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff957b333a2c00 Aug 21 01:29:25 dac-e-21 kernel: LustreError: 33394:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff957b333a2c00 Aug 21 01:29:25 dac-e-21 kernel: LustreError: 33394:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff957b333a2c00 Aug 21 01:29:26 dac-e-21 kernel: LustreError: 33396:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff959431381e00 Aug 21 01:29:26 dac-e-21 kernel: LustreError: 33406:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff959431381e00 Aug 21 01:29:26 dac-e-21 kernel: LustreError: 33395:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff957b7b7d2000 Aug 21 01:29:26 dac-e-21 kernel: LustreError: 33398:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff957b7b7d2000 Aug 21 01:29:26 dac-e-21 kernel: LustreError: 33397:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff959431381e00 Aug 21 01:29:26 dac-e-21 kernel: LustreError: 33396:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff959431381e00 Aug 21 01:29:26 dac-e-21 kernel: LustreError: 33397:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff957b64e92e00 Aug 21 01:29:26 dac-e-21 kernel: LustreError: 33395:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff957b64e92e00 Aug 21 01:29:26 dac-e-21 kernel: LustreError: 33398:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff957b64e92e00 Aug 21 01:29:29 dac-e-21 kernel: Lustre: fs1-OST00f1: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:29:29 dac-e-21 kernel: Lustre: Skipped 6 previous similar messages Aug 21 01:30:02 dac-e-21 kernel: Lustre: fs1-OST00f8: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:30:02 dac-e-21 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:30:34 dac-e-21 kernel: LustreError: 392744:0:(ldlm_lib.c:3259:target_bulk_io()) @@@ network error on bulk READ req@ffff957b22e2b850 x1642422187205536/t0(0) o3->ca24e700-cef1-0f8f-792b-2c007834cfd0@10.47.20.69@o2ib1:298/0 lens 488/440 e 2 to 0 dl 1566347458 ref 1 fl Interpret:/2/0 rc 0/0 Aug 21 01:30:34 dac-e-21 kernel: Lustre: fs1-OST00f2: Bulk IO read error with ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1), client will retry: rc -110 Aug 21 01:30:34 dac-e-21 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:30:34 dac-e-21 kernel: LustreError: 392744:0:(ldlm_lib.c:3259:target_bulk_io()) Skipped 2 previous similar messages Aug 21 01:33:38 dac-e-21 kernel: Lustre: fs1-OST00f3: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:33:38 dac-e-21 kernel: Lustre: Skipped 42 previous similar messages Aug 21 01:37:09 dac-e-21 kernel: Lustre: fs1-OST00f1: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:37:09 dac-e-21 kernel: Lustre: fs1-OST00f1: Connection restored to 9a3f8291-8d90-e819-b6ab-9b2c8e66825a (at 10.47.20.69@o2ib1) Aug 21 01:37:09 dac-e-21 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:37:30 dac-e-21 kernel: Lustre: fs1-OST00f9: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:39:04 dac-e-21 kernel: Lustre: fs1-OST00f2: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:44:03 dac-e-21 kernel: Lustre: fs1-OST00fb: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 21 01:44:03 dac-e-21 kernel: Lustre: Skipped 5 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-22 ------------------------------------------------------------------------------- -- Logs begin at Thu 2019-05-09 10:24:44 BST, end at Wed 2019-08-21 11:51:58 BST. -- Aug 21 01:22:41 dac-e-22 kernel: Lustre: fs1-MDT0015: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:22:41 dac-e-22 kernel: Lustre: Skipped 598 previous similar messages Aug 21 01:23:42 dac-e-22 kernel: Lustre: fs1-OST0100: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:23:42 dac-e-22 kernel: Lustre: Skipped 128 previous similar messages Aug 21 01:29:21 dac-e-22 kernel: Lustre: fs1-OST0102: Connection restored to ba6d10d8-29ae-af16-bb29-fe0009074454 (at 10.47.21.33@o2ib1) Aug 21 01:29:21 dac-e-22 kernel: Lustre: fs1-OST0101: Connection restored to ba6d10d8-29ae-af16-bb29-fe0009074454 (at 10.47.21.33@o2ib1) Aug 21 01:29:21 dac-e-22 kernel: Lustre: Skipped 76 previous similar messages Aug 21 01:29:23 dac-e-22 kernel: LustreError: 78663:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff901fe9687c00 Aug 21 01:29:23 dac-e-22 kernel: LustreError: 78662:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff901fe9687c00 Aug 21 01:29:23 dac-e-22 kernel: LustreError: 78660:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff901fe9687c00 Aug 21 01:29:23 dac-e-22 kernel: LustreError: 78661:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff901fe9687c00 Aug 21 01:29:23 dac-e-22 kernel: Lustre: fs1-OST0103: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:29:23 dac-e-22 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:29:23 dac-e-22 kernel: LustreError: 55313:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff90214aaa4850 x1642422187213104/t0(0) o3->77aabb7c-29d1-0881-b291-653fb977b8da@10.47.21.36@o2ib1:212/0 lens 488/440 e 0 to 0 dl 1566347372 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:29:23 dac-e-22 kernel: Lustre: fs1-OST0103: Bulk IO read error with 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1), client will retry: rc -110 Aug 21 01:29:54 dac-e-22 kernel: Lustre: fs1-OST0100: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:29:54 dac-e-22 kernel: Lustre: Skipped 21 previous similar messages Aug 21 01:30:34 dac-e-22 kernel: LustreError: 63344:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 15728640(16777216) req@ffff9007baf1d050 x1642422187204656/t0(0) o3->4e14a970-7ca0-d5d3-629d-52842291c983@10.47.21.37@o2ib1:295/0 lens 488/440 e 2 to 0 dl 1566347455 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:30:34 dac-e-22 kernel: Lustre: fs1-OST0105: Bulk IO read error with 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1), client will retry: rc -110 Aug 21 01:34:05 dac-e-22 kernel: Lustre: fs1-OST0102: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:34:05 dac-e-22 kernel: Lustre: fs1-OST0102: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:34:05 dac-e-22 kernel: Lustre: Skipped 6 previous similar messages Aug 21 01:36:37 dac-e-22 kernel: Lustre: fs1-OST0105: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:36:37 dac-e-22 kernel: Lustre: fs1-OST0105: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:36:37 dac-e-22 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:41:30 dac-e-22 kernel: Lustre: fs1-OST0103: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:41:30 dac-e-22 kernel: Lustre: Skipped 2 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-23 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:47 BST, end at Wed 2019-08-21 11:51:58 BST. -- Aug 21 01:22:41 dac-e-23 kernel: Lustre: fs1-MDT0016: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:22:41 dac-e-23 kernel: Lustre: Skipped 598 previous similar messages Aug 21 01:23:42 dac-e-23 kernel: Lustre: fs1-OST0112: Connection restored to 9a3f8291-8d90-e819-b6ab-9b2c8e66825a (at 10.47.20.69@o2ib1) Aug 21 01:23:42 dac-e-23 kernel: Lustre: Skipped 129 previous similar messages Aug 21 01:29:21 dac-e-23 kernel: Lustre: fs1-OST0109: Connection restored to a9c07af9-d96f-6905-bdea-228af9a88046 (at 10.47.21.32@o2ib1) Aug 21 01:29:21 dac-e-23 kernel: Lustre: Skipped 71 previous similar messages Aug 21 01:29:23 dac-e-23 kernel: Lustre: fs1-OST0110: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:29:23 dac-e-23 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:29:23 dac-e-23 kernel: Lustre: 1668:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347361/real 1566347363] req@ffffa162b8fd1f80 x1642422111951712/t0(0) o104->fs1-OST010c@10.47.20.69@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1566347372 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 21 01:29:23 dac-e-23 kernel: Lustre: 1668:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Aug 21 01:29:23 dac-e-23 kernel: LustreError: 64096:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffffa163872fd600 Aug 21 01:29:23 dac-e-23 kernel: LustreError: 64030:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffffa163872fd600 Aug 21 01:29:23 dac-e-23 kernel: LustreError: 64097:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffffa17bcff56800 Aug 21 01:29:23 dac-e-23 kernel: LustreError: 64035:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffffa17bcff56800 Aug 21 01:29:23 dac-e-23 kernel: LustreError: 64036:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffffa17bcff56800 Aug 21 01:29:23 dac-e-23 kernel: LustreError: 64033:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffffa17bcff56800 Aug 21 01:29:23 dac-e-23 kernel: LustreError: 64035:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffffa17ad4790400 Aug 21 01:29:23 dac-e-23 kernel: LustreError: 64097:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffffa17ad4790400 Aug 21 01:29:23 dac-e-23 kernel: LustreError: 64036:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffffa17ad4790400 Aug 21 01:29:23 dac-e-23 kernel: LustreError: 64033:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffffa17ad4790400 Aug 21 01:29:23 dac-e-23 kernel: LustreError: 13160:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffffa17b3f557850 x1642422187213536/t0(0) o3->77aabb7c-29d1-0881-b291-653fb977b8da@10.47.21.36@o2ib1:244/0 lens 488/440 e 0 to 0 dl 1566347404 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:29:23 dac-e-23 kernel: Lustre: fs1-OST0110: Bulk IO read error with 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1), client will retry: rc -110 Aug 21 01:29:25 dac-e-23 kernel: Lustre: 1668:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347365/real 1566347365] req@ffffa162b8fd1f80 x1642422111951712/t0(0) o104->fs1-OST010c@10.47.20.69@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1566347377 ref 1 fl Rpc:eX/2/ffffffff rc -11/-1 Aug 21 01:29:25 dac-e-23 kernel: Lustre: 1668:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Aug 21 01:29:28 dac-e-23 kernel: Lustre: 90504:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566347361/real 1566347363] req@ffffa14cddfee300 x1642422111951536/t0(0) o104->fs1-OST0112@10.47.20.69@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1566347368 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Aug 21 01:29:30 dac-e-23 kernel: Lustre: 88721:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566347363/real 1566347364] req@ffffa162b75dd100 x1642422111951552/t0(0) o104->fs1-OST0111@10.47.20.69@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1566347370 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Aug 21 01:30:02 dac-e-23 kernel: Lustre: fs1-MDT0016: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:30:02 dac-e-23 kernel: Lustre: Skipped 8 previous similar messages Aug 21 01:30:02 dac-e-23 kernel: Lustre: fs1-MDT0016: Connection restored to 9a3f8291-8d90-e819-b6ab-9b2c8e66825a (at 10.47.20.69@o2ib1) Aug 21 01:30:02 dac-e-23 kernel: Lustre: Skipped 37 previous similar messages Aug 21 01:30:03 dac-e-23 kernel: LustreError: 13154:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffffa16232efa050 x1642422187202592/t0(0) o3->ca24e700-cef1-0f8f-792b-2c007834cfd0@10.47.20.69@o2ib1:269/0 lens 488/440 e 1 to 0 dl 1566347429 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:30:03 dac-e-23 kernel: Lustre: fs1-OST0111: Bulk IO read error with ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1), client will retry: rc -110 Aug 21 01:30:34 dac-e-23 kernel: LustreError: 26629:0:(ldlm_lib.c:3259:target_bulk_io()) @@@ network error on bulk READ req@ffffa17b41ef5850 x1642422187213536/t0(0) o3->77aabb7c-29d1-0881-b291-653fb977b8da@10.47.21.36@o2ib1:296/0 lens 488/440 e 2 to 0 dl 1566347456 ref 1 fl Interpret:/2/0 rc 0/0 Aug 21 01:30:34 dac-e-23 kernel: Lustre: fs1-OST0110: Bulk IO read error with 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1), client will retry: rc -110 Aug 21 01:34:22 dac-e-23 kernel: Lustre: fs1-OST0110: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:34:22 dac-e-23 kernel: Lustre: Skipped 6 previous similar messages Aug 21 01:34:22 dac-e-23 kernel: Lustre: fs1-OST0110: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:34:22 dac-e-23 kernel: Lustre: Skipped 9 previous similar messages Aug 21 01:37:10 dac-e-23 kernel: Lustre: fs1-OST0109: Connection restored to 9a3f8291-8d90-e819-b6ab-9b2c8e66825a (at 10.47.20.69@o2ib1) Aug 21 01:37:10 dac-e-23 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:44:03 dac-e-23 kernel: Lustre: fs1-OST0110: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 21 01:44:03 dac-e-23 kernel: Lustre: Skipped 3 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-3 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:47 BST, end at Wed 2019-08-21 11:51:58 BST. -- Aug 21 01:22:41 dac-e-3 kernel: Lustre: fs1-MDT0002: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:23:42 dac-e-3 kernel: Lustre: fs1-OST0019: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 21 01:23:42 dac-e-3 kernel: Lustre: fs1-OST0020: Connection restored to d51aba7f-3e9b-b409-76a8-ec3ceb1b1ca5 (at 10.47.21.35@o2ib1) Aug 21 01:23:42 dac-e-3 kernel: Lustre: Skipped 128 previous similar messages Aug 21 01:29:21 dac-e-3 kernel: Lustre: fs1-OST001a: Connection restored to d51aba7f-3e9b-b409-76a8-ec3ceb1b1ca5 (at 10.47.21.35@o2ib1) Aug 21 01:29:21 dac-e-3 kernel: Lustre: Skipped 76 previous similar messages Aug 21 01:29:23 dac-e-3 kernel: Lustre: 388567:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347362/real 1566347363] req@ffff9940c3030900 x1642422111953008/t0(0) o104->fs1-OST001f@10.47.20.69@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1566347373 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 21 01:29:23 dac-e-3 kernel: Lustre: 388567:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 276 previous similar messages Aug 21 01:29:24 dac-e-3 kernel: Lustre: fs1-OST0020: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:29:28 dac-e-3 kernel: Lustre: fs1-OST0019: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:29:28 dac-e-3 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:35:21 dac-e-3 kernel: Lustre: fs1-OST0021: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:35:21 dac-e-3 kernel: Lustre: fs1-OST0021: Connection restored to 9a3f8291-8d90-e819-b6ab-9b2c8e66825a (at 10.47.20.69@o2ib1) Aug 21 01:35:21 dac-e-3 kernel: Lustre: Skipped 33 previous similar messages Aug 21 01:37:30 dac-e-3 kernel: Lustre: fs1-OST0021: Connection restored to 9a3f8291-8d90-e819-b6ab-9b2c8e66825a (at 10.47.20.69@o2ib1) Aug 21 01:37:30 dac-e-3 kernel: Lustre: Skipped 2 previous similar messages Aug 21 01:44:03 dac-e-3 kernel: Lustre: fs1-OST0020: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 21 01:44:03 dac-e-3 kernel: Lustre: Skipped 2 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-24 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:47 BST, end at Wed 2019-08-21 11:51:58 BST. -- Aug 21 01:22:41 dac-e-24 kernel: Lustre: fs1-MDT0017: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:22:41 dac-e-24 kernel: Lustre: Skipped 598 previous similar messages Aug 21 01:23:42 dac-e-24 kernel: Lustre: fs1-OST0116: Connection restored to efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) Aug 21 01:23:42 dac-e-24 kernel: Lustre: fs1-OST011b: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:23:42 dac-e-24 kernel: Lustre: Skipped 128 previous similar messages Aug 21 01:29:21 dac-e-24 kernel: Lustre: fs1-OST011e: Connection restored to efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) Aug 21 01:29:21 dac-e-24 kernel: Lustre: Skipped 68 previous similar messages Aug 21 01:29:24 dac-e-24 kernel: Lustre: 277147:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347362/real 1566347364] req@ffff887c20853180 x1642422111951440/t0(0) o105->fs1-OST011c@10.47.21.37@o2ib1:15/16 lens 360/224 e 0 to 1 dl 1566347373 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 21 01:29:24 dac-e-24 kernel: Lustre: 277147:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Aug 21 01:29:24 dac-e-24 kernel: LustreError: 429611:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff887c25604e00 Aug 21 01:29:24 dac-e-24 kernel: LustreError: 429608:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff887c25604e00 Aug 21 01:29:24 dac-e-24 kernel: LustreError: 429609:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff887c25604e00 Aug 21 01:29:24 dac-e-24 kernel: LustreError: 429610:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff887c25604e00 Aug 21 01:29:24 dac-e-24 kernel: Lustre: fs1-OST011c: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:29:24 dac-e-24 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:29:24 dac-e-24 kernel: LustreError: 204624:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff88934c2a2050 x1642422187204496/t0(0) o3->4e14a970-7ca0-d5d3-629d-52842291c983@10.47.21.37@o2ib1:213/0 lens 488/440 e 0 to 0 dl 1566347373 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:29:24 dac-e-24 kernel: Lustre: fs1-OST011c: Bulk IO read error with 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1), client will retry: rc -110 Aug 21 01:29:54 dac-e-24 kernel: Lustre: fs1-OST0117: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:29:54 dac-e-24 kernel: Lustre: Skipped 6 previous similar messages Aug 21 01:29:54 dac-e-24 kernel: Lustre: fs1-OST0117: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:29:54 dac-e-24 kernel: Lustre: Skipped 35 previous similar messages Aug 21 01:30:02 dac-e-24 kernel: Lustre: fs1-OST0115: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:30:02 dac-e-24 kernel: Lustre: fs1-OST0117: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:30:34 dac-e-24 kernel: LustreError: 230297:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 15728640(16777216) req@ffff88933e5df850 x1642422187208016/t0(0) o3->4e14a970-7ca0-d5d3-629d-52842291c983@10.47.21.37@o2ib1:297/0 lens 488/440 e 2 to 0 dl 1566347457 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:30:34 dac-e-24 kernel: Lustre: fs1-OST011a: Bulk IO read error with 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1), client will retry: rc -110 Aug 21 01:30:34 dac-e-24 kernel: LustreError: 230297:0:(ldlm_lib.c:3268:target_bulk_io()) Skipped 1 previous similar message Aug 21 01:34:07 dac-e-24 kernel: Lustre: fs1-OST0119: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:34:07 dac-e-24 kernel: Lustre: Skipped 2 previous similar messages Aug 21 01:34:07 dac-e-24 kernel: Lustre: fs1-OST0119: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:34:07 dac-e-24 kernel: Lustre: Skipped 9 previous similar messages Aug 21 01:38:39 dac-e-24 kernel: Lustre: fs1-OST0116: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:38:39 dac-e-24 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:39:35 dac-e-24 kernel: Lustre: fs1-OST011b: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:39:35 dac-e-24 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:44:03 dac-e-24 kernel: Lustre: fs1-OST011a: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 21 01:44:03 dac-e-24 kernel: Lustre: Skipped 2 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-5 ------------------------------------------------------------------------------- -- Logs begin at Thu 2019-05-09 08:54:36 BST, end at Wed 2019-08-21 11:51:58 BST. -- Aug 21 01:22:41 dac-e-5 kernel: Lustre: fs1-MDT0004: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:22:41 dac-e-5 kernel: Lustre: Skipped 598 previous similar messages Aug 21 01:23:42 dac-e-5 kernel: Lustre: fs1-OST0032: Connection restored to 1c9f1f8c-ae8f-76b3-441d-4d74e9e3a7df (at 10.47.21.38@o2ib1) Aug 21 01:23:42 dac-e-5 kernel: Lustre: fs1-OST0038: Connection restored to d51aba7f-3e9b-b409-76a8-ec3ceb1b1ca5 (at 10.47.21.35@o2ib1) Aug 21 01:23:42 dac-e-5 kernel: Lustre: Skipped 129 previous similar messages Aug 21 01:23:42 dac-e-5 kernel: Lustre: Skipped 2 previous similar messages Aug 21 01:29:21 dac-e-5 kernel: Lustre: fs1-OST0036: Connection restored to 3f28bd14-132c-e629-0b42-8ea0fdd5d2a4 (at 10.47.21.31@o2ib1) Aug 21 01:29:21 dac-e-5 kernel: Lustre: Skipped 65 previous similar messages Aug 21 01:29:23 dac-e-5 kernel: Lustre: 37179:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347362/real 1566347363] req@ffff88248c710000 x1642422111961008/t0(0) o104->fs1-OST003b@10.47.21.36@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1566347373 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 21 01:29:23 dac-e-5 kernel: Lustre: 37179:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 7 previous similar messages Aug 21 01:29:23 dac-e-5 kernel: LustreError: 189640:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883c6da54200 Aug 21 01:29:23 dac-e-5 kernel: LustreError: 189643:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883c6da54200 Aug 21 01:29:23 dac-e-5 kernel: LustreError: 189642:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883c6da54200 Aug 21 01:29:23 dac-e-5 kernel: LustreError: 189637:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88257cc4d000 Aug 21 01:29:23 dac-e-5 kernel: LustreError: 189647:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88257cc4d000 Aug 21 01:29:23 dac-e-5 kernel: LustreError: 189641:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883c6da54200 Aug 21 01:29:23 dac-e-5 kernel: LustreError: 189647:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88257cc4d000 Aug 21 01:29:23 dac-e-5 kernel: LustreError: 189637:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88257cc4d000 Aug 21 01:29:23 dac-e-5 kernel: LustreError: 189639:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff88257cc4d000 Aug 21 01:29:23 dac-e-5 kernel: Lustre: fs1-OST0030: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:29:24 dac-e-5 kernel: Lustre: fs1-OST003b: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:29:24 dac-e-5 kernel: LustreError: 450515:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8824ed617050 x1642422187202320/t0(0) o3->ca24e700-cef1-0f8f-792b-2c007834cfd0@10.47.20.69@o2ib1:244/0 lens 488/440 e 0 to 0 dl 1566347404 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:29:24 dac-e-5 kernel: Lustre: fs1-OST003b: Bulk IO read error with ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1), client will retry: rc -110 Aug 21 01:30:02 dac-e-5 kernel: Lustre: fs1-OST0031: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:30:02 dac-e-5 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:30:02 dac-e-5 kernel: Lustre: fs1-OST0031: Connection restored to 9a3f8291-8d90-e819-b6ab-9b2c8e66825a (at 10.47.20.69@o2ib1) Aug 21 01:30:02 dac-e-5 kernel: Lustre: Skipped 32 previous similar messages Aug 21 01:30:34 dac-e-5 kernel: LustreError: 53363:0:(ldlm_lib.c:3259:target_bulk_io()) @@@ network error on bulk READ req@ffff883d7ddba850 x1642422187213664/t0(0) o3->77aabb7c-29d1-0881-b291-653fb977b8da@10.47.21.36@o2ib1:294/0 lens 488/440 e 2 to 0 dl 1566347454 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:30:34 dac-e-5 kernel: Lustre: fs1-OST0039: Bulk IO read error with 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1), client will retry: rc -110 Aug 21 01:37:10 dac-e-5 kernel: Lustre: fs1-OST0031: Connection restored to 9a3f8291-8d90-e819-b6ab-9b2c8e66825a (at 10.47.20.69@o2ib1) Aug 21 01:37:10 dac-e-5 kernel: Lustre: Skipped 4 previous similar messages Aug 21 01:38:04 dac-e-5 kernel: Lustre: fs1-OST0039: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:39:05 dac-e-5 kernel: Lustre: fs1-MDT0004: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:39:35 dac-e-5 kernel: Lustre: fs1-OST0031: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:39:35 dac-e-5 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:39:35 dac-e-5 kernel: Lustre: fs1-OST0031: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:39:35 dac-e-5 kernel: Lustre: Skipped 5 previous similar messages Aug 21 01:44:03 dac-e-5 kernel: Lustre: fs1-OST0033: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 21 01:44:03 dac-e-5 kernel: Lustre: Skipped 1 previous similar message HOSTS ------------------------------------------------------------------------- dac-e-9 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-05-07 20:43:31 BST, end at Wed 2019-08-21 11:51:58 BST. -- Aug 21 01:22:41 dac-e-9 kernel: Lustre: fs1-MDT0008: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:22:41 dac-e-9 kernel: Lustre: Skipped 598 previous similar messages Aug 21 01:23:42 dac-e-9 kernel: Lustre: fs1-OST0061: Connection restored to ba6d10d8-29ae-af16-bb29-fe0009074454 (at 10.47.21.33@o2ib1) Aug 21 01:23:42 dac-e-9 kernel: Lustre: Skipped 127 previous similar messages Aug 21 01:29:21 dac-e-9 kernel: Lustre: fs1-OST006b: Connection restored to 9a3f8291-8d90-e819-b6ab-9b2c8e66825a (at 10.47.20.69@o2ib1) Aug 21 01:29:21 dac-e-9 kernel: Lustre: Skipped 77 previous similar messages Aug 21 01:29:23 dac-e-9 kernel: LustreError: 48509:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8cf73e7a6e00 Aug 21 01:29:23 dac-e-9 kernel: LustreError: 48507:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8cf73e7a6e00 Aug 21 01:29:23 dac-e-9 kernel: LustreError: 48507:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8d0e8cbc4c00 Aug 21 01:29:23 dac-e-9 kernel: LustreError: 48509:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8d0e8cbc4c00 Aug 21 01:29:23 dac-e-9 kernel: LustreError: 48518:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8d0e8cbc4c00 Aug 21 01:29:23 dac-e-9 kernel: LustreError: 48511:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8d0f0bb2ce00 Aug 21 01:29:23 dac-e-9 kernel: LustreError: 48519:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8d0f0bb2ce00 Aug 21 01:29:23 dac-e-9 kernel: LustreError: 48513:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8d0f3d24d800 Aug 21 01:29:23 dac-e-9 kernel: LustreError: 48512:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8d0f3d24d800 Aug 21 01:29:23 dac-e-9 kernel: LustreError: 48514:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8d0f3d24d800 Aug 21 01:29:23 dac-e-9 kernel: LustreError: 48511:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8d0f3d24d800 Aug 21 01:29:23 dac-e-9 kernel: Lustre: fs1-OST0064: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:29:23 dac-e-9 kernel: Lustre: 44581:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347362/real 1566347363] req@ffff8cf63364a400 x1642422111960896/t0(0) o104->fs1-OST0062@10.47.21.36@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1566347369 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 21 01:29:23 dac-e-9 kernel: Lustre: 44581:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 8 previous similar messages Aug 21 01:29:24 dac-e-9 kernel: LustreError: 48518:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8cf739baaa00 Aug 21 01:29:24 dac-e-9 kernel: LustreError: 48509:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8d0e8cbc4c00 Aug 21 01:29:24 dac-e-9 kernel: LustreError: 48507:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8d0e8cbc4c00 Aug 21 01:29:24 dac-e-9 kernel: LustreError: 48508:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8d0e8cbc4c00 Aug 21 01:29:24 dac-e-9 kernel: LustreError: 48510:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8d0e8cbc4c00 Aug 21 01:29:24 dac-e-9 kernel: LustreError: 48507:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8d0e8cbc4c00 Aug 21 01:29:24 dac-e-9 kernel: LustreError: 48518:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8cf739baaa00 Aug 21 01:29:24 dac-e-9 kernel: LustreError: 48508:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8cf739baaa00 Aug 21 01:29:24 dac-e-9 kernel: LustreError: 48509:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8d0e8cbc4c00 Aug 21 01:29:24 dac-e-9 kernel: LustreError: 48510:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8d0fee31fe00 Aug 21 01:29:24 dac-e-9 kernel: LustreError: 48507:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8d0fee31fe00 Aug 21 01:29:24 dac-e-9 kernel: LustreError: 48518:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8d0e8cbc4c00 Aug 21 01:29:24 dac-e-9 kernel: LustreError: 48510:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8cf739baaa00 Aug 21 01:29:24 dac-e-9 kernel: LustreError: 48507:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8d0fee31fe00 Aug 21 01:29:24 dac-e-9 kernel: LustreError: 48510:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8d0e8cbc4c00 Aug 21 01:29:24 dac-e-9 kernel: LustreError: 48518:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8d0fee31fe00 Aug 21 01:29:24 dac-e-9 kernel: LustreError: 48508:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8d0e8cbc4c00 Aug 21 01:29:24 dac-e-9 kernel: LustreError: 48509:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8d0e8cbc4c00 Aug 21 01:29:24 dac-e-9 kernel: LustreError: 48508:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8d0e8cbc4c00 Aug 21 01:29:24 dac-e-9 kernel: LustreError: 48518:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8d0e8cbc4c00 Aug 21 01:29:24 dac-e-9 kernel: LustreError: 48509:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8d0e8cbc4c00 Aug 21 01:29:24 dac-e-9 kernel: LustreError: 441848:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8d0fd2dec050 x1642422187212256/t0(0) o3->77aabb7c-29d1-0881-b291-653fb977b8da@10.47.21.36@o2ib1:244/0 lens 488/440 e 0 to 0 dl 1566347404 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:29:24 dac-e-9 kernel: Lustre: fs1-OST006b: Bulk IO read error with ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1), client will retry: rc -110 Aug 21 01:29:24 dac-e-9 kernel: Lustre: Skipped 5 previous similar messages Aug 21 01:29:24 dac-e-9 kernel: LustreError: 441848:0:(ldlm_lib.c:3253:target_bulk_io()) Skipped 1 previous similar message Aug 21 01:29:25 dac-e-9 kernel: LustreError: 441844:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8cf70851f850 x1642422187202480/t0(0) o3->ca24e700-cef1-0f8f-792b-2c007834cfd0@10.47.20.69@o2ib1:212/0 lens 488/440 e 0 to 0 dl 1566347372 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:29:25 dac-e-9 kernel: Lustre: fs1-OST0064: Bulk IO read error with ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1), client will retry: rc -110 Aug 21 01:29:25 dac-e-9 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:29:27 dac-e-9 kernel: LNet: 48506:0:(o2iblnd_cb.c:3381:kiblnd_check_conns()) Timed out tx for 10.47.20.69@o2ib1: 0 seconds Aug 21 01:29:27 dac-e-9 kernel: Lustre: 64563:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347364/real 1566347367] req@ffff8cf818c5ad00 x1642422111956688/t0(0) o104->fs1-OST0067@10.47.20.69@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1566347375 ref 1 fl Rpc:eX/2/ffffffff rc 0/-1 Aug 21 01:29:27 dac-e-9 kernel: Lustre: 64563:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Aug 21 01:29:32 dac-e-9 kernel: Lustre: 60554:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566347361/real 1566347362] req@ffff8cf7fefe2880 x1642422111956800/t0(0) o104->fs1-OST0066@10.47.21.36@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1566347372 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 Aug 21 01:30:02 dac-e-9 kernel: Lustre: fs1-MDT0008: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:30:02 dac-e-9 kernel: Lustre: Skipped 3 previous similar messages Aug 21 01:30:02 dac-e-9 kernel: Lustre: fs1-MDT0008: Connection restored to 9a3f8291-8d90-e819-b6ab-9b2c8e66825a (at 10.47.20.69@o2ib1) Aug 21 01:30:02 dac-e-9 kernel: Lustre: Skipped 31 previous similar messages Aug 21 01:30:03 dac-e-9 kernel: LustreError: 442311:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8cf6a7ae6850 x1642422187202192/t0(0) o3->ca24e700-cef1-0f8f-792b-2c007834cfd0@10.47.20.69@o2ib1:270/0 lens 488/440 e 1 to 0 dl 1566347430 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:30:03 dac-e-9 kernel: Lustre: fs1-OST006a: Bulk IO read error with ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1), client will retry: rc -110 Aug 21 01:30:34 dac-e-9 kernel: LustreError: 441847:0:(ldlm_lib.c:3259:target_bulk_io()) @@@ network error on bulk READ req@ffff8cf70851f050 x1642422187203536/t0(0) o3->4e14a970-7ca0-d5d3-629d-52842291c983@10.47.21.37@o2ib1:295/0 lens 488/440 e 2 to 0 dl 1566347455 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:30:34 dac-e-9 kernel: LustreError: 441857:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 15728640(16777216) req@ffff8d0e5d769050 x1642422187215168/t0(0) o3->77aabb7c-29d1-0881-b291-653fb977b8da@10.47.21.36@o2ib1:288/0 lens 488/440 e 3 to 0 dl 1566347448 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:30:34 dac-e-9 kernel: Lustre: fs1-OST0069: Bulk IO read error with 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1), client will retry: rc -110 Aug 21 01:30:34 dac-e-9 kernel: LustreError: 441847:0:(ldlm_lib.c:3259:target_bulk_io()) Skipped 1 previous similar message Aug 21 01:33:38 dac-e-9 kernel: Lustre: fs1-OST0065: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:33:38 dac-e-9 kernel: Lustre: Skipped 9 previous similar messages Aug 21 01:34:17 dac-e-9 kernel: Lustre: fs1-OST006a: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:34:17 dac-e-9 kernel: Lustre: Skipped 5 previous similar messages Aug 21 01:36:18 dac-e-9 kernel: Lustre: fs1-OST0069: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:36:18 dac-e-9 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:39:04 dac-e-9 kernel: Lustre: fs1-OST0067: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:39:04 dac-e-9 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:44:03 dac-e-9 kernel: Lustre: fs1-OST0061: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 21 01:44:03 dac-e-9 kernel: Lustre: Skipped 7 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-4 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-05-07 14:07:28 BST, end at Wed 2019-08-21 11:51:58 BST. -- Aug 21 01:22:41 dac-e-4 kernel: Lustre: fs1-MDT0003: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:23:42 dac-e-4 kernel: Lustre: fs1-OST0025: Connection restored to d51aba7f-3e9b-b409-76a8-ec3ceb1b1ca5 (at 10.47.21.35@o2ib1) Aug 21 01:23:42 dac-e-4 kernel: Lustre: Skipped 129 previous similar messages Aug 21 01:29:21 dac-e-4 kernel: Lustre: fs1-OST002a: Connection restored to d51aba7f-3e9b-b409-76a8-ec3ceb1b1ca5 (at 10.47.21.35@o2ib1) Aug 21 01:29:21 dac-e-4 kernel: Lustre: Skipped 60 previous similar messages Aug 21 01:29:23 dac-e-4 kernel: Lustre: fs1-OST0026: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:29:23 dac-e-4 kernel: Lustre: 15843:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347362/real 1566347363] req@ffff97fd4bb2bf00 x1642422111956048/t0(0) o104->fs1-OST0026@10.47.20.69@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1566347373 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 21 01:29:23 dac-e-4 kernel: Lustre: 15843:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 273 previous similar messages Aug 21 01:29:25 dac-e-4 kernel: Lustre: fs1-OST002f: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:29:25 dac-e-4 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:29:26 dac-e-4 kernel: Lustre: 95244:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347365/real 1566347366] req@ffff97fc593bf980 x1642422111961552/t0(0) o104->fs1-OST002b@10.47.20.69@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1566347372 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 21 01:29:26 dac-e-4 kernel: Lustre: 95244:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 1 previous similar message Aug 21 01:29:31 dac-e-4 kernel: Lustre: fs1-OST0026: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:29:31 dac-e-4 kernel: Lustre: Skipped 3 previous similar messages Aug 21 01:29:35 dac-e-4 kernel: Lustre: 458564:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566347364/real 1566347365] req@ffff97fd9ad58480 x1642422111959152/t0(0) o104->fs1-OST002a@10.47.20.69@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1566347375 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 Aug 21 01:30:02 dac-e-4 kernel: Lustre: fs1-MDT0003: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:30:02 dac-e-4 kernel: Lustre: fs1-MDT0003: Connection restored to 9a3f8291-8d90-e819-b6ab-9b2c8e66825a (at 10.47.20.69@o2ib1) Aug 21 01:30:02 dac-e-4 kernel: Lustre: Skipped 36 previous similar messages Aug 21 01:34:17 dac-e-4 kernel: Lustre: fs1-OST002e: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:34:17 dac-e-4 kernel: Lustre: Skipped 6 previous similar messages Aug 21 01:35:21 dac-e-4 kernel: Lustre: fs1-OST0024: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:35:21 dac-e-4 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:38:04 dac-e-4 kernel: Lustre: fs1-OST0024: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:38:04 dac-e-4 kernel: Lustre: Skipped 2 previous similar messages Aug 21 01:38:38 dac-e-4 kernel: Lustre: fs1-OST0025: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:38:38 dac-e-4 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:44:03 dac-e-4 kernel: Lustre: fs1-OST002c: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 21 01:44:03 dac-e-4 kernel: Lustre: Skipped 1 previous similar message HOSTS ------------------------------------------------------------------------- dac-e-8 ------------------------------------------------------------------------------- -- Logs begin at Thu 2019-05-09 08:54:37 BST, end at Wed 2019-08-21 11:51:58 BST. -- Aug 21 01:22:41 dac-e-8 kernel: Lustre: fs1-MDT0007: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:22:41 dac-e-8 kernel: Lustre: Skipped 598 previous similar messages Aug 21 01:23:42 dac-e-8 kernel: Lustre: fs1-OST0058: Connection restored to d51aba7f-3e9b-b409-76a8-ec3ceb1b1ca5 (at 10.47.21.35@o2ib1) Aug 21 01:23:42 dac-e-8 kernel: Lustre: fs1-OST005f: Connection restored to a9c07af9-d96f-6905-bdea-228af9a88046 (at 10.47.21.32@o2ib1) Aug 21 01:23:42 dac-e-8 kernel: Lustre: Skipped 129 previous similar messages Aug 21 01:29:21 dac-e-8 kernel: Lustre: fs1-OST0058: Connection restored to a9c07af9-d96f-6905-bdea-228af9a88046 (at 10.47.21.32@o2ib1) Aug 21 01:29:21 dac-e-8 kernel: Lustre: Skipped 77 previous similar messages Aug 21 01:29:23 dac-e-8 kernel: Lustre: fs1-OST0058: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:29:23 dac-e-8 kernel: Lustre: 7869:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347361/real 1566347363] req@ffff8faadb156c00 x1642422111958320/t0(0) o104->fs1-OST0058@10.47.21.36@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1566347372 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 21 01:29:23 dac-e-8 kernel: Lustre: 7869:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 278 previous similar messages Aug 21 01:29:23 dac-e-8 kernel: LustreError: 49819:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fa9824b4200 Aug 21 01:29:23 dac-e-8 kernel: LustreError: 49830:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fa9824b4200 Aug 21 01:29:23 dac-e-8 kernel: LustreError: 49821:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fa9824b4200 Aug 21 01:29:23 dac-e-8 kernel: LustreError: 49822:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fa9824b4200 Aug 21 01:29:24 dac-e-8 kernel: LustreError: 49821:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fa9824b4200 Aug 21 01:29:24 dac-e-8 kernel: LustreError: 49822:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fa9824b4200 Aug 21 01:29:24 dac-e-8 kernel: LustreError: 49830:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fa9824b4200 Aug 21 01:29:24 dac-e-8 kernel: LustreError: 49820:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fa9824b4200 Aug 21 01:29:24 dac-e-8 kernel: LustreError: 49821:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fa9824b4200 Aug 21 01:30:02 dac-e-8 kernel: Lustre: fs1-OST0056: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:30:02 dac-e-8 kernel: Lustre: Skipped 2 previous similar messages Aug 21 01:30:03 dac-e-8 kernel: LustreError: 25910:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8faa363e0050 x1642422187201936/t0(0) o3->ca24e700-cef1-0f8f-792b-2c007834cfd0@10.47.20.69@o2ib1:262/0 lens 488/440 e 2 to 0 dl 1566347422 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:30:03 dac-e-8 kernel: Lustre: fs1-OST0057: Bulk IO read error with ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1), client will retry: rc -110 Aug 21 01:34:05 dac-e-8 kernel: Lustre: fs1-OST005e: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:34:05 dac-e-8 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:34:05 dac-e-8 kernel: Lustre: fs1-OST005e: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:34:05 dac-e-8 kernel: Lustre: Skipped 36 previous similar messages Aug 21 01:35:21 dac-e-8 kernel: Lustre: fs1-MDT0007: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:36:37 dac-e-8 kernel: Lustre: fs1-OST005d: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:36:37 dac-e-8 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:44:03 dac-e-8 kernel: Lustre: fs1-OST0058: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 21 01:44:03 dac-e-8 kernel: Lustre: Skipped 5 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-7 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-05-07 14:07:28 BST, end at Wed 2019-08-21 11:51:58 BST. -- Aug 21 01:22:41 dac-e-7 kernel: Lustre: fs1-MDT0006: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:22:41 dac-e-7 kernel: Lustre: Skipped 598 previous similar messages Aug 21 01:23:42 dac-e-7 kernel: Lustre: fs1-OST0053: Connection restored to a9c07af9-d96f-6905-bdea-228af9a88046 (at 10.47.21.32@o2ib1) Aug 21 01:23:42 dac-e-7 kernel: Lustre: Skipped 128 previous similar messages Aug 21 01:29:21 dac-e-7 kernel: Lustre: fs1-OST004f: Connection restored to a9c07af9-d96f-6905-bdea-228af9a88046 (at 10.47.21.32@o2ib1) Aug 21 01:29:21 dac-e-7 kernel: Lustre: Skipped 77 previous similar messages Aug 21 01:29:24 dac-e-7 kernel: Lustre: fs1-OST0048: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:29:24 dac-e-7 kernel: LustreError: 55112:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff968752e75a00 Aug 21 01:29:24 dac-e-7 kernel: LustreError: 55111:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff968752e75a00 Aug 21 01:29:24 dac-e-7 kernel: LustreError: 55114:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff968752e75a00 Aug 21 01:29:24 dac-e-7 kernel: LustreError: 55113:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff968752e75a00 Aug 21 01:29:25 dac-e-7 kernel: Lustre: fs1-OST004a: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:29:25 dac-e-7 kernel: Lustre: Skipped 3 previous similar messages Aug 21 01:30:02 dac-e-7 kernel: Lustre: fs1-OST004a: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:30:02 dac-e-7 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:30:02 dac-e-7 kernel: Lustre: fs1-OST004a: Connection restored to 9a3f8291-8d90-e819-b6ab-9b2c8e66825a (at 10.47.20.69@o2ib1) Aug 21 01:30:02 dac-e-7 kernel: Lustre: Skipped 31 previous similar messages Aug 21 01:30:34 dac-e-7 kernel: LustreError: 405193:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 15728640(16777216) req@ffff9686c7737850 x1642422187219392/t0(0) o3->77aabb7c-29d1-0881-b291-653fb977b8da@10.47.21.36@o2ib1:284/0 lens 488/440 e 3 to 0 dl 1566347444 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:30:34 dac-e-7 kernel: LustreError: 29804:0:(ldlm_lib.c:3259:target_bulk_io()) @@@ network error on bulk READ req@ffff96861aeb5050 x1642422187221152/t0(0) o3->77aabb7c-29d1-0881-b291-653fb977b8da@10.47.21.36@o2ib1:285/0 lens 488/440 e 3 to 0 dl 1566347445 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:30:34 dac-e-7 kernel: Lustre: fs1-OST004c: Bulk IO read error with 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1), client will retry: rc -110 Aug 21 01:36:18 dac-e-7 kernel: Lustre: fs1-OST0051: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:36:18 dac-e-7 kernel: Lustre: Skipped 6 previous similar messages Aug 21 01:36:42 dac-e-7 kernel: Lustre: fs1-OST004c: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:36:42 dac-e-7 kernel: Lustre: Skipped 2 previous similar messages Aug 21 01:38:04 dac-e-7 kernel: Lustre: fs1-OST0048: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:38:39 dac-e-7 kernel: Lustre: fs1-OST004c: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:38:39 dac-e-7 kernel: Lustre: fs1-OST004c: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:38:39 dac-e-7 kernel: Lustre: Skipped 3 previous similar messages Aug 21 01:44:03 dac-e-7 kernel: Lustre: fs1-OST0050: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 21 01:44:03 dac-e-7 kernel: Lustre: Skipped 4 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-6 ------------------------------------------------------------------------------- -- Logs begin at Thu 2019-05-09 08:54:36 BST, end at Wed 2019-08-21 11:51:58 BST. -- Aug 21 01:22:41 dac-e-6 kernel: Lustre: fs1-MDT0005: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 21 01:22:41 dac-e-6 kernel: Lustre: Skipped 598 previous similar messages Aug 21 01:23:42 dac-e-6 kernel: Lustre: fs1-OST0042: Connection restored to 9a3f8291-8d90-e819-b6ab-9b2c8e66825a (at 10.47.20.69@o2ib1) Aug 21 01:23:42 dac-e-6 kernel: Lustre: Skipped 131 previous similar messages Aug 21 01:29:21 dac-e-6 kernel: Lustre: fs1-OST0042: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 21 01:29:21 dac-e-6 kernel: Lustre: Skipped 70 previous similar messages Aug 21 01:29:23 dac-e-6 kernel: LustreError: 58242:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fd9a3cad600 Aug 21 01:29:23 dac-e-6 kernel: Lustre: fs1-OST003c: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:29:23 dac-e-6 kernel: LustreError: 58291:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fd9a3cad600 Aug 21 01:29:23 dac-e-6 kernel: LustreError: 58243:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fd9a3cad600 Aug 21 01:29:23 dac-e-6 kernel: Lustre: 17610:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566347362/real 1566347363] req@ffff8fd9ee735a00 x1642422111955296/t0(0) o104->fs1-OST0042@10.47.21.36@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1566347373 ref 1 fl Rpc:eX/0/ffffffff rc 0/-1 Aug 21 01:29:23 dac-e-6 kernel: Lustre: 17610:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Aug 21 01:29:24 dac-e-6 kernel: LustreError: 33243:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8fda62bd1050 x1642422187202416/t0(0) o3->ca24e700-cef1-0f8f-792b-2c007834cfd0@10.47.20.69@o2ib1:244/0 lens 488/440 e 0 to 0 dl 1566347404 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:29:24 dac-e-6 kernel: Lustre: fs1-OST0041: Bulk IO read error with ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1), client will retry: rc -110 Aug 21 01:29:26 dac-e-6 kernel: LustreError: 33248:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8fda627d7050 x1642422187208544/t0(0) o3->4e14a970-7ca0-d5d3-629d-52842291c983@10.47.21.37@o2ib1:246/0 lens 488/440 e 0 to 0 dl 1566347406 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:29:26 dac-e-6 kernel: Lustre: fs1-OST003c: Bulk IO read error with 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1), client will retry: rc -110 Aug 21 01:29:31 dac-e-6 kernel: Lustre: 22584:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1566347364/real 1566347364] req@ffff8fdb56f1de80 x1642422111956192/t0(0) o104->fs1-OST0041@10.47.20.69@o2ib1:15/16 lens 296/224 e 0 to 1 dl 1566347371 ref 1 fl Rpc:X/2/ffffffff rc -11/-1 Aug 21 01:29:31 dac-e-6 kernel: Lustre: 22584:0:(client.c:2134:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Aug 21 01:29:55 dac-e-6 kernel: Lustre: fs1-OST0041: Connection restored to d51aba7f-3e9b-b409-76a8-ec3ceb1b1ca5 (at 10.47.21.35@o2ib1) Aug 21 01:29:55 dac-e-6 kernel: Lustre: Skipped 34 previous similar messages Aug 21 01:30:02 dac-e-6 kernel: Lustre: fs1-OST003d: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:30:02 dac-e-6 kernel: Lustre: Skipped 4 previous similar messages Aug 21 01:30:35 dac-e-6 kernel: LustreError: 33245:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 15728640(16777216) req@ffff8fda627d5850 x1642422187212896/t0(0) o3->4e14a970-7ca0-d5d3-629d-52842291c983@10.47.21.37@o2ib1:290/0 lens 488/440 e 3 to 0 dl 1566347450 ref 1 fl Interpret:/0/0 rc 0/0 Aug 21 01:30:35 dac-e-6 kernel: Lustre: fs1-OST0042: Bulk IO read error with 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1), client will retry: rc -110 Aug 21 01:33:43 dac-e-6 kernel: Lustre: fs1-OST003d: Client 77aabb7c-29d1-0881-b291-653fb977b8da (at 10.47.21.36@o2ib1) reconnecting Aug 21 01:33:43 dac-e-6 kernel: Lustre: Skipped 2 previous similar messages Aug 21 01:33:43 dac-e-6 kernel: Lustre: fs1-OST003d: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:33:43 dac-e-6 kernel: Lustre: Skipped 8 previous similar messages Aug 21 01:34:07 dac-e-6 kernel: Lustre: fs1-OST0045: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:35:21 dac-e-6 kernel: Lustre: fs1-MDT0005: Client ca24e700-cef1-0f8f-792b-2c007834cfd0 (at 10.47.20.69@o2ib1) reconnecting Aug 21 01:36:18 dac-e-6 kernel: Lustre: fs1-OST0045: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 21 01:36:18 dac-e-6 kernel: Lustre: Skipped 3 previous similar messages Aug 21 01:36:37 dac-e-6 kernel: Lustre: fs1-OST0042: Client 4e14a970-7ca0-d5d3-629d-52842291c983 (at 10.47.21.37@o2ib1) reconnecting Aug 21 01:36:37 dac-e-6 kernel: Lustre: Skipped 1 previous similar message Aug 21 01:44:03 dac-e-6 kernel: Lustre: fs1-OST0044: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 21 01:44:03 dac-e-6 kernel: Lustre: Skipped 4 previous similar messages