HOSTS ------------------------------------------------------------------------- dac-e-1 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-04-16 21:34:08 BST, end at Wed 2019-08-21 11:30:28 BST. -- Aug 20 22:18:27 dac-e-1 kernel: Lustre: fs1-OST0000: Connection restored to 1dd0df5c-469f-6643-3e7e-a97b4a721aca (at 10.47.20.68@o2ib1) Aug 20 22:18:27 dac-e-1 kernel: Lustre: Skipped 138 previous similar messages Aug 20 22:27:02 dac-e-1 kernel: Lustre: fs1-OST0000: Connection restored to bfa3419e-5f60-2112-6155-b28d2a4d1c73 (at 10.47.21.38@o2ib1) Aug 20 22:27:02 dac-e-1 kernel: Lustre: Skipped 13 previous similar messages Aug 20 22:27:04 dac-e-1 kernel: Lustre: fs1-OST0006: Client fs1-MDT0001-mdtlov_UUID (at 10.47.18.2@o2ib1) reconnecting Aug 20 22:45:01 dac-e-1 kernel: Lustre: fs1-OST0009: Connection restored to 1dd0df5c-469f-6643-3e7e-a97b4a721aca (at 10.47.20.68@o2ib1) Aug 20 22:45:01 dac-e-1 kernel: Lustre: Skipped 14 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-10 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-05-07 14:07:27 BST, end at Wed 2019-08-21 11:30:28 BST. -- Aug 20 22:18:27 dac-e-10 kernel: Lustre: fs1-OST0077: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 20 22:18:27 dac-e-10 kernel: Lustre: Skipped 130 previous similar messages Aug 20 22:27:02 dac-e-10 kernel: Lustre: fs1-OST0076: Connection restored to ba6d10d8-29ae-af16-bb29-fe0009074454 (at 10.47.21.33@o2ib1) Aug 20 22:27:02 dac-e-10 kernel: Lustre: fs1-OST0071: Connection restored to efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) Aug 20 22:27:02 dac-e-10 kernel: Lustre: Skipped 12 previous similar messages Aug 20 22:45:01 dac-e-10 kernel: Lustre: fs1-OST0070: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 20 22:45:01 dac-e-10 kernel: Lustre: Skipped 9 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-12 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:45 BST, end at Wed 2019-08-21 11:30:28 BST. -- Aug 20 22:18:27 dac-e-12 kernel: Lustre: fs1-OST0084: Connection restored to d51aba7f-3e9b-b409-76a8-ec3ceb1b1ca5 (at 10.47.21.35@o2ib1) Aug 20 22:18:27 dac-e-12 kernel: Lustre: Skipped 130 previous similar messages Aug 20 22:27:02 dac-e-12 kernel: Lustre: fs1-OST0085: Connection restored to 1c9f1f8c-ae8f-76b3-441d-4d74e9e3a7df (at 10.47.21.38@o2ib1) Aug 20 22:27:02 dac-e-12 kernel: Lustre: Skipped 15 previous similar messages Aug 20 22:27:04 dac-e-12 kernel: Lustre: fs1-OST008f: Client efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) reconnecting Aug 20 22:27:04 dac-e-12 kernel: LNet: 50968:0:(o2iblnd_cb.c:413:kiblnd_handle_rx()) PUT_NACK from 10.47.21.34@o2ib1 Aug 20 22:27:04 dac-e-12 kernel: LNet: 50968:0:(o2iblnd_cb.c:413:kiblnd_handle_rx()) Skipped 1 previous similar message Aug 20 22:27:04 dac-e-12 kernel: LustreError: 300291:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff9d2f245b8850 x1642422159069520/t0(0) o3->efd44c75-059d-5b3e-f4e6-657896b473fc@10.47.21.34@o2ib1:630/0 lens 488/440 e 0 to 0 dl 1566336465 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 22:27:04 dac-e-12 kernel: Lustre: fs1-OST008f: Bulk IO read error with efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1), client will retry: rc -110 Aug 20 22:45:01 dac-e-12 kernel: Lustre: fs1-OST0086: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 20 22:45:01 dac-e-12 kernel: Lustre: Skipped 10 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-15 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:46 BST, end at Wed 2019-08-21 11:30:28 BST. -- Aug 20 22:18:27 dac-e-15 kernel: Lustre: fs1-OST00b3: Connection restored to 1c9f1f8c-ae8f-76b3-441d-4d74e9e3a7df (at 10.47.21.38@o2ib1) Aug 20 22:18:27 dac-e-15 kernel: Lustre: Skipped 129 previous similar messages Aug 20 22:27:02 dac-e-15 kernel: Lustre: fs1-OST00a8: Connection restored to 9a3f8291-8d90-e819-b6ab-9b2c8e66825a (at 10.47.20.69@o2ib1) Aug 20 22:27:02 dac-e-15 kernel: Lustre: fs1-OST00ac: Connection restored to a9c07af9-d96f-6905-bdea-228af9a88046 (at 10.47.21.32@o2ib1) Aug 20 22:27:02 dac-e-15 kernel: Lustre: Skipped 13 previous similar messages Aug 20 22:27:04 dac-e-15 kernel: LustreError: 347143:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff996a89b61a00 Aug 20 22:27:04 dac-e-15 kernel: LustreError: 347144:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff996a89b61a00 Aug 20 22:27:04 dac-e-15 kernel: LustreError: 347142:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff996a89b61a00 Aug 20 22:27:04 dac-e-15 kernel: LustreError: 347145:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff996a89b61a00 Aug 20 22:27:04 dac-e-15 kernel: LustreError: 347192:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff996a89b61a00 Aug 20 22:27:04 dac-e-15 kernel: LustreError: 347143:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff996a89b61a00 Aug 20 22:27:04 dac-e-15 kernel: LustreError: 347144:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff996a89b61a00 Aug 20 22:27:04 dac-e-15 kernel: Lustre: fs1-OST00ad: Client efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) reconnecting Aug 20 22:27:04 dac-e-15 kernel: LustreError: 347192:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff996a89b61a00 Aug 20 22:27:04 dac-e-15 kernel: LustreError: 347143:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff996a89b61a00 Aug 20 22:27:04 dac-e-15 kernel: LustreError: 347144:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff996a89b61a00 Aug 20 22:27:04 dac-e-15 kernel: LustreError: 436886:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff996b98289850 x1642422159069600/t0(0) o3->efd44c75-059d-5b3e-f4e6-657896b473fc@10.47.21.34@o2ib1:618/0 lens 488/440 e 1 to 0 dl 1566336453 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 22:27:04 dac-e-15 kernel: LustreError: 347142:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff996a89b61a00 Aug 20 22:27:04 dac-e-15 kernel: LustreError: 347145:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff996a89b61a00 Aug 20 22:27:04 dac-e-15 kernel: LustreError: 347143:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff996a89b61a00 Aug 20 22:27:04 dac-e-15 kernel: LustreError: 347143:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff996a89b61a00 Aug 20 22:27:04 dac-e-15 kernel: LustreError: 347192:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff996a89b61a00 Aug 20 22:27:04 dac-e-15 kernel: LustreError: 347144:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff996a89b61a00 Aug 20 22:27:04 dac-e-15 kernel: Lustre: fs1-OST00ad: Bulk IO read error with efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1), client will retry: rc -110 Aug 20 22:45:01 dac-e-15 kernel: Lustre: fs1-OST00af: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 20 22:45:01 dac-e-15 kernel: Lustre: Skipped 11 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-11 ------------------------------------------------------------------------------- -- Logs begin at Thu 2019-03-21 15:42:22 GMT, end at Wed 2019-08-21 11:30:28 BST. -- Aug 20 22:18:27 dac-e-11 kernel: Lustre: fs1-OST0082: Connection restored to 1c9f1f8c-ae8f-76b3-441d-4d74e9e3a7df (at 10.47.21.38@o2ib1) Aug 20 22:18:27 dac-e-11 kernel: Lustre: Skipped 130 previous similar messages Aug 20 22:27:02 dac-e-11 kernel: Lustre: fs1-OST007d: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 20 22:27:02 dac-e-11 kernel: Lustre: Skipped 11 previous similar messages Aug 20 22:27:04 dac-e-11 kernel: LustreError: 50792:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff98474f863400 Aug 20 22:27:04 dac-e-11 kernel: LustreError: 50793:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff98474f863400 Aug 20 22:27:04 dac-e-11 kernel: LustreError: 50847:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff98474f863400 Aug 20 22:27:04 dac-e-11 kernel: LustreError: 50790:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff98474f863400 Aug 20 22:27:04 dac-e-11 kernel: LustreError: 50791:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff98474f863400 Aug 20 22:27:04 dac-e-11 kernel: LustreError: 50792:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff98474f863400 Aug 20 22:27:04 dac-e-11 kernel: LustreError: 50793:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff98474f863400 Aug 20 22:27:04 dac-e-11 kernel: LustreError: 50790:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff98474f863400 Aug 20 22:27:04 dac-e-11 kernel: LustreError: 50792:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff98474f863400 Aug 20 22:27:04 dac-e-11 kernel: LustreError: 50847:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff98474f863400 Aug 20 22:27:04 dac-e-11 kernel: LustreError: 50791:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff98474f863400 Aug 20 22:27:04 dac-e-11 kernel: LustreError: 50793:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff98474f863400 Aug 20 22:28:04 dac-e-11 kernel: LustreError: 180721:0:(ldlm_lib.c:3259:target_bulk_io()) @@@ network error on bulk READ req@ffff98483726a850 x1642422159069408/t0(0) o3->efd44c75-059d-5b3e-f4e6-657896b473fc@10.47.21.34@o2ib1:655/0 lens 488/440 e 1 to 0 dl 1566336490 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 22:28:04 dac-e-11 kernel: Lustre: fs1-OST007d: Bulk IO read error with efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1), client will retry: rc -110 Aug 20 22:36:51 dac-e-11 kernel: Lustre: fs1-OST007d: Client efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) reconnecting Aug 20 22:36:51 dac-e-11 kernel: Lustre: fs1-OST007d: Connection restored to efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) Aug 20 22:36:51 dac-e-11 kernel: Lustre: Skipped 12 previous similar messages Aug 20 22:45:01 dac-e-11 kernel: Lustre: fs1-OST007d: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) HOSTS ------------------------------------------------------------------------- dac-e-13 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:45 BST, end at Wed 2019-08-21 11:30:28 BST. -- Aug 20 22:18:27 dac-e-13 kernel: Lustre: fs1-OST0092: Connection restored to (at 10.47.21.31@o2ib1) Aug 20 22:18:27 dac-e-13 kernel: Lustre: Skipped 123 previous similar messages Aug 20 22:27:02 dac-e-13 kernel: Lustre: fs1-OST0098: Connection restored to (at 10.47.21.38@o2ib1) Aug 20 22:27:02 dac-e-13 kernel: Lustre: Skipped 13 previous similar messages Aug 20 22:45:01 dac-e-13 kernel: Lustre: fs1-OST0095: Connection restored to (at 10.47.20.68@o2ib1) Aug 20 22:45:01 dac-e-13 kernel: Lustre: Skipped 11 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-14 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:46 BST, end at Wed 2019-08-21 11:30:28 BST. -- Aug 20 22:18:27 dac-e-14 kernel: Lustre: fs1-OST00a5: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 20 22:18:27 dac-e-14 kernel: Lustre: Skipped 131 previous similar messages Aug 20 22:27:02 dac-e-14 kernel: Lustre: fs1-OST00a0: Connection restored to ba6d10d8-29ae-af16-bb29-fe0009074454 (at 10.47.21.33@o2ib1) Aug 20 22:27:02 dac-e-14 kernel: Lustre: fs1-OST00a4: Connection restored to efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) Aug 20 22:27:02 dac-e-14 kernel: Lustre: Skipped 11 previous similar messages Aug 20 22:27:02 dac-e-14 kernel: Lustre: Skipped 1 previous similar message Aug 20 22:27:04 dac-e-14 kernel: LustreError: 409822:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff977600fc9600 Aug 20 22:27:04 dac-e-14 kernel: LustreError: 409820:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff977600fc9600 Aug 20 22:27:04 dac-e-14 kernel: LustreError: 409821:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff977600fc9600 Aug 20 22:27:04 dac-e-14 kernel: Lustre: fs1-OST00a4: Client efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) reconnecting Aug 20 22:27:04 dac-e-14 kernel: LustreError: 409823:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff977600fc9600 Aug 20 22:27:04 dac-e-14 kernel: LustreError: 409821:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff977600fc9600 Aug 20 22:27:04 dac-e-14 kernel: LustreError: 409878:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff977600fc9600 Aug 20 22:27:04 dac-e-14 kernel: LustreError: 409820:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff977600fc9600 Aug 20 22:27:04 dac-e-14 kernel: LustreError: 409822:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff977600fc9600 Aug 20 22:27:04 dac-e-14 kernel: LustreError: 409823:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff977600fc9600 Aug 20 22:27:04 dac-e-14 kernel: LustreError: 409823:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff977600fc9600 Aug 20 22:27:04 dac-e-14 kernel: LustreError: 409820:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff977600fc9600 Aug 20 22:27:04 dac-e-14 kernel: LustreError: 409822:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff977600fc9600 Aug 20 22:27:04 dac-e-14 kernel: LustreError: 409821:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff977600fc9600 Aug 20 22:27:04 dac-e-14 kernel: LustreError: 409823:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff977600fc9600 Aug 20 22:27:04 dac-e-14 kernel: LustreError: 40841:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff977546bd4050 x1642422159069536/t0(0) o3->efd44c75-059d-5b3e-f4e6-657896b473fc@10.47.21.34@o2ib1:618/0 lens 488/440 e 1 to 0 dl 1566336453 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 22:27:04 dac-e-14 kernel: LustreError: 409823:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff977600fc9600 Aug 20 22:27:04 dac-e-14 kernel: LustreError: 409822:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff977600fc9600 Aug 20 22:27:04 dac-e-14 kernel: Lustre: fs1-OST00a4: Bulk IO read error with efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1), client will retry: rc -110 Aug 20 22:45:01 dac-e-14 kernel: Lustre: fs1-OST009f: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 20 22:45:01 dac-e-14 kernel: Lustre: Skipped 9 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-23 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:47 BST, end at Wed 2019-08-21 11:30:28 BST. -- Aug 20 22:18:27 dac-e-23 kernel: Lustre: fs1-OST010a: Connection restored to efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) Aug 20 22:18:27 dac-e-23 kernel: Lustre: Skipped 127 previous similar messages Aug 20 22:27:02 dac-e-23 kernel: Lustre: fs1-OST010f: Connection restored to efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) Aug 20 22:27:02 dac-e-23 kernel: Lustre: Skipped 12 previous similar messages Aug 20 22:27:04 dac-e-23 kernel: LustreError: 64035:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffffa17bb4eb3200 Aug 20 22:27:04 dac-e-23 kernel: LustreError: 64034:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffffa17bb4eb3200 Aug 20 22:27:04 dac-e-23 kernel: LustreError: 64097:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffffa17bb4eb3200 Aug 20 22:27:04 dac-e-23 kernel: LNetError: 64028:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-103, 0) Aug 20 22:27:04 dac-e-23 kernel: LustreError: 64036:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffffa17bb4eb3200 Aug 20 22:27:04 dac-e-23 kernel: LustreError: 64033:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffffa17c114d8600 Aug 20 22:27:04 dac-e-23 kernel: LustreError: 64028:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffffa17c114d8600 Aug 20 22:27:04 dac-e-23 kernel: LNet: 64033:0:(o2iblnd_cb.c:413:kiblnd_handle_rx()) PUT_NACK from 10.47.21.34@o2ib1 Aug 20 22:27:04 dac-e-23 kernel: Lustre: fs1-OST010b: Client efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) reconnecting Aug 20 22:27:04 dac-e-23 kernel: LustreError: 64028:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffffa17c114d8600 Aug 20 22:27:04 dac-e-23 kernel: LustreError: 64034:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffffa17c114d8600 Aug 20 22:27:04 dac-e-23 kernel: LustreError: 64036:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffffa17c114d8600 Aug 20 22:27:04 dac-e-23 kernel: LustreError: 64097:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffffa17c114d8600 Aug 20 22:27:04 dac-e-23 kernel: LustreError: 156052:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffffa17bba76e850 x1642422159069632/t0(0) o3->efd44c75-059d-5b3e-f4e6-657896b473fc@10.47.21.34@o2ib1:630/0 lens 488/440 e 0 to 0 dl 1566336465 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 22:27:04 dac-e-23 kernel: LustreError: 64097:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffffa17bb4eb3200 Aug 20 22:27:04 dac-e-23 kernel: Lustre: fs1-OST010b: Bulk IO read error with efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1), client will retry: rc -110 Aug 20 22:27:04 dac-e-23 kernel: LustreError: 64034:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffffa17bb4eb3200 Aug 20 22:27:04 dac-e-23 kernel: LustreError: 64033:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffffa17bb4eb3200 Aug 20 22:27:04 dac-e-23 kernel: LustreError: 64036:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffffa17bb4eb3200 Aug 20 22:27:04 dac-e-23 kernel: LustreError: 64035:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffffa17bb4eb3200 Aug 20 22:27:04 dac-e-23 kernel: LustreError: 64097:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffffa17bb4eb3200 Aug 20 22:27:05 dac-e-23 kernel: LustreError: 128512:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffffa17bf137d850 x1642422159068832/t0(0) o3->efd44c75-059d-5b3e-f4e6-657896b473fc@10.47.21.34@o2ib1:630/0 lens 488/440 e 0 to 0 dl 1566336465 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 22:27:05 dac-e-23 kernel: Lustre: fs1-OST0110: Bulk IO read error with efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1), client will retry: rc -110 Aug 20 22:28:04 dac-e-23 kernel: LustreError: 156062:0:(ldlm_lib.c:3268:target_bulk_io()) @@@ truncated bulk READ 14680064(16777216) req@ffffa17c37a22850 x1642422159069632/t0(0) o3->efd44c75-059d-5b3e-f4e6-657896b473fc@10.47.21.34@o2ib1:657/0 lens 488/440 e 1 to 0 dl 1566336492 ref 1 fl Interpret:/2/0 rc 0/0 Aug 20 22:28:04 dac-e-23 kernel: Lustre: fs1-OST010b: Bulk IO read error with efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1), client will retry: rc -110 Aug 20 22:38:12 dac-e-23 kernel: Lustre: fs1-MDT0016: Client efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) reconnecting Aug 20 22:38:12 dac-e-23 kernel: Lustre: Skipped 1 previous similar message Aug 20 22:38:12 dac-e-23 kernel: Lustre: fs1-MDT0016: Connection restored to efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) Aug 20 22:38:12 dac-e-23 kernel: Lustre: Skipped 13 previous similar messages Aug 20 22:40:37 dac-e-23 kernel: Lustre: fs1-OST010b: Client efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) reconnecting Aug 20 22:45:01 dac-e-23 kernel: Lustre: fs1-OST010b: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 20 22:45:01 dac-e-23 kernel: Lustre: Skipped 2 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-17 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:47 BST, end at Wed 2019-08-21 11:30:28 BST. -- Aug 20 22:18:27 dac-e-17 kernel: Lustre: fs1-OST00c3: Connection restored to d51aba7f-3e9b-b409-76a8-ec3ceb1b1ca5 (at 10.47.21.35@o2ib1) Aug 20 22:18:27 dac-e-17 kernel: Lustre: Skipped 129 previous similar messages Aug 20 22:27:02 dac-e-17 kernel: Lustre: fs1-OST00c0: Connection restored to d51aba7f-3e9b-b409-76a8-ec3ceb1b1ca5 (at 10.47.21.35@o2ib1) Aug 20 22:27:02 dac-e-17 kernel: Lustre: Skipped 13 previous similar messages Aug 20 22:45:01 dac-e-17 kernel: Lustre: fs1-OST00c8: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 20 22:45:01 dac-e-17 kernel: Lustre: Skipped 10 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-16 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:46 BST, end at Wed 2019-08-21 11:30:28 BST. -- Aug 20 22:18:27 dac-e-16 kernel: Lustre: fs1-OST00bd: Connection restored to 9a3f8291-8d90-e819-b6ab-9b2c8e66825a (at 10.47.20.69@o2ib1) Aug 20 22:18:27 dac-e-16 kernel: Lustre: fs1-OST00bc: Connection restored to 1c9f1f8c-ae8f-76b3-441d-4d74e9e3a7df (at 10.47.21.38@o2ib1) Aug 20 22:18:27 dac-e-16 kernel: Lustre: Skipped 128 previous similar messages Aug 20 22:18:27 dac-e-16 kernel: Lustre: Skipped 1 previous similar message Aug 20 22:27:02 dac-e-16 kernel: Lustre: fs1-OST00b6: Connection restored to 3f28bd14-132c-e629-0b42-8ea0fdd5d2a4 (at 10.47.21.31@o2ib1) Aug 20 22:27:02 dac-e-16 kernel: Lustre: Skipped 10 previous similar messages Aug 20 22:45:01 dac-e-16 kernel: Lustre: fs1-OST00bb: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 20 22:45:01 dac-e-16 kernel: Lustre: fs1-OST00b5: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 20 22:45:01 dac-e-16 kernel: Lustre: Skipped 12 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-18 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:46 BST, end at Wed 2019-08-21 11:30:28 BST. -- Aug 20 22:18:27 dac-e-18 kernel: Lustre: fs1-OST00d5: Connection restored to d51aba7f-3e9b-b409-76a8-ec3ceb1b1ca5 (at 10.47.21.35@o2ib1) Aug 20 22:18:27 dac-e-18 kernel: Lustre: Skipped 129 previous similar messages Aug 20 22:26:54 dac-e-18 kernel: Lustre: fs1-OST00d1: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 20 22:26:54 dac-e-18 kernel: Lustre: Skipped 11 previous similar messages Aug 20 22:27:04 dac-e-18 kernel: LustreError: 157128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff96bb6caf0a00 Aug 20 22:27:04 dac-e-18 kernel: LustreError: 157129:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff96bb6caf0a00 Aug 20 22:27:04 dac-e-18 kernel: LustreError: 157127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff96bb6caf0a00 Aug 20 22:27:04 dac-e-18 kernel: LustreError: 157181:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff96bb6caf0a00 Aug 20 22:27:04 dac-e-18 kernel: LustreError: 157126:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff96bb6caf0a00 Aug 20 22:27:04 dac-e-18 kernel: LustreError: 157128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff96bb6caf0a00 Aug 20 22:27:04 dac-e-18 kernel: LustreError: 157129:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff96bb6caf0a00 Aug 20 22:27:04 dac-e-18 kernel: LustreError: 157127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff96bb6caf0a00 Aug 20 22:27:04 dac-e-18 kernel: LustreError: 157128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff96bb6caf0a00 Aug 20 22:27:04 dac-e-18 kernel: LustreError: 157128:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff96bb6caf0a00 Aug 20 22:27:04 dac-e-18 kernel: LustreError: 157127:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff96bb6caf0a00 Aug 20 22:27:04 dac-e-18 kernel: LustreError: 157181:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff96bb6caf0a00 Aug 20 22:27:04 dac-e-18 kernel: LustreError: 157126:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff96bb6caf0a00 Aug 20 22:27:04 dac-e-18 kernel: LustreError: 157129:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff96bb6caf0a00 Aug 20 22:27:04 dac-e-18 kernel: LustreError: 157181:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff96bb6caf0a00 Aug 20 22:27:04 dac-e-18 kernel: LustreError: 157126:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff96bb6caf0a00 Aug 20 22:28:04 dac-e-18 kernel: LustreError: 286347:0:(ldlm_lib.c:3259:target_bulk_io()) @@@ network error on bulk READ req@ffff96d3edfd7850 x1642422159069440/t0(0) o3->efd44c75-059d-5b3e-f4e6-657896b473fc@10.47.21.34@o2ib1:655/0 lens 488/440 e 1 to 0 dl 1566336490 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 22:28:04 dac-e-18 kernel: Lustre: fs1-OST00d5: Bulk IO read error with efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1), client will retry: rc -110 Aug 20 22:36:51 dac-e-18 kernel: Lustre: fs1-OST00d5: Client efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) reconnecting Aug 20 22:36:51 dac-e-18 kernel: Lustre: fs1-OST00d5: Connection restored to efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) Aug 20 22:36:51 dac-e-18 kernel: Lustre: Skipped 12 previous similar messages Aug 20 22:45:01 dac-e-18 kernel: Lustre: fs1-OST00d0: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) HOSTS ------------------------------------------------------------------------- dac-e-19 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:47 BST, end at Wed 2019-08-21 11:30:28 BST. -- Aug 20 22:18:27 dac-e-19 kernel: Lustre: fs1-OST00e0: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 20 22:18:27 dac-e-19 kernel: Lustre: Skipped 128 previous similar messages Aug 20 22:27:02 dac-e-19 kernel: Lustre: fs1-OST00da: Connection restored to efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) Aug 20 22:27:02 dac-e-19 kernel: Lustre: Skipped 12 previous similar messages Aug 20 22:45:01 dac-e-19 kernel: Lustre: fs1-OST00e0: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 20 22:45:01 dac-e-19 kernel: Lustre: Skipped 11 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-20 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:47 BST, end at Wed 2019-08-21 11:30:28 BST. -- Aug 20 22:18:27 dac-e-20 kernel: Lustre: fs1-OST00e8: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 20 22:18:27 dac-e-20 kernel: Lustre: fs1-OST00ee: Connection restored to efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) Aug 20 22:18:27 dac-e-20 kernel: Lustre: Skipped 128 previous similar messages Aug 20 22:23:56 dac-e-20 kernel: perf: interrupt took too long (3935 > 3932), lowering kernel.perf_event_max_sample_rate to 50000 Aug 20 22:26:55 dac-e-20 kernel: Lustre: fs1-OST00e9: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 20 22:26:55 dac-e-20 kernel: Lustre: Skipped 10 previous similar messages Aug 20 22:45:01 dac-e-20 kernel: Lustre: fs1-OST00e8: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 20 22:45:01 dac-e-20 kernel: Lustre: Skipped 12 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-2 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:46 BST, end at Wed 2019-08-21 11:30:28 BST. -- Aug 20 22:18:27 dac-e-2 kernel: Lustre: fs1-OST0011: Connection restored to 1c9f1f8c-ae8f-76b3-441d-4d74e9e3a7df (at 10.47.21.38@o2ib1) Aug 20 22:18:27 dac-e-2 kernel: Lustre: Skipped 128 previous similar messages Aug 20 22:22:03 dac-e-2 kernel: perf: interrupt took too long (4949 > 4903), lowering kernel.perf_event_max_sample_rate to 40000 Aug 20 22:27:02 dac-e-2 kernel: Lustre: fs1-OST0017: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 20 22:27:02 dac-e-2 kernel: Lustre: Skipped 16 previous similar messages Aug 20 22:27:04 dac-e-2 kernel: Lustre: 302667:0:(client.c:2134:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1566336422/real 1566336424] req@ffff8e107e57e300 x1642422105543568/t0(0) o13->fs1-OST0006-osc-MDT0001@10.47.18.1@o2ib1:7/4 lens 224/368 e 0 to 1 dl 1566336429 ref 1 fl Rpc:ReX/0/ffffffff rc 0/-1 Aug 20 22:27:04 dac-e-2 kernel: Lustre: fs1-OST0006-osc-MDT0001: Connection to fs1-OST0006 (at 10.47.18.1@o2ib1) was lost; in progress operations using this service will wait for recovery to complete Aug 20 22:45:01 dac-e-2 kernel: Lustre: fs1-OST0012: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 20 22:45:01 dac-e-2 kernel: Lustre: Skipped 8 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-3 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:47 BST, end at Wed 2019-08-21 11:30:28 BST. -- Aug 20 22:18:27 dac-e-3 kernel: Lustre: fs1-OST001e: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 20 22:18:27 dac-e-3 kernel: Lustre: Skipped 129 previous similar messages Aug 20 22:27:02 dac-e-3 kernel: Lustre: fs1-OST0023: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 20 22:27:02 dac-e-3 kernel: Lustre: Skipped 12 previous similar messages Aug 20 22:45:01 dac-e-3 kernel: Lustre: fs1-OST0018: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 20 22:45:01 dac-e-3 kernel: Lustre: Skipped 11 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-4 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-05-07 14:07:28 BST, end at Wed 2019-08-21 11:30:28 BST. -- Aug 20 22:18:27 dac-e-4 kernel: Lustre: fs1-OST002f: Connection restored to efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) Aug 20 22:18:27 dac-e-4 kernel: Lustre: Skipped 127 previous similar messages Aug 20 22:27:02 dac-e-4 kernel: Lustre: fs1-OST002c: Connection restored to 1c9f1f8c-ae8f-76b3-441d-4d74e9e3a7df (at 10.47.21.38@o2ib1) Aug 20 22:27:02 dac-e-4 kernel: Lustre: Skipped 12 previous similar messages Aug 20 22:45:01 dac-e-4 kernel: Lustre: fs1-OST0026: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 20 22:45:01 dac-e-4 kernel: Lustre: Skipped 12 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-21 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:47 BST, end at Wed 2019-08-21 11:30:28 BST. -- Aug 20 22:18:27 dac-e-21 kernel: Lustre: fs1-OST00f7: Connection restored to ba6d10d8-29ae-af16-bb29-fe0009074454 (at 10.47.21.33@o2ib1) Aug 20 22:18:27 dac-e-21 kernel: Lustre: Skipped 127 previous similar messages Aug 20 22:27:02 dac-e-21 kernel: Lustre: fs1-OST00fb: Connection restored to 3f28bd14-132c-e629-0b42-8ea0fdd5d2a4 (at 10.47.21.31@o2ib1) Aug 20 22:27:02 dac-e-21 kernel: Lustre: Skipped 12 previous similar messages Aug 20 22:45:01 dac-e-21 kernel: Lustre: fs1-OST00f2: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 20 22:45:01 dac-e-21 kernel: Lustre: Skipped 11 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-5 ------------------------------------------------------------------------------- -- Logs begin at Thu 2019-05-09 08:54:36 BST, end at Wed 2019-08-21 11:30:28 BST. -- Aug 20 22:18:27 dac-e-5 kernel: Lustre: fs1-OST0033: Connection restored to b70bd8d1-bef1-cafa-1d50-bfa93684ff22 (at 10.47.21.37@o2ib1) Aug 20 22:18:27 dac-e-5 kernel: Lustre: Skipped 132 previous similar messages Aug 20 22:27:02 dac-e-5 kernel: Lustre: fs1-OST0030: Connection restored to 3f28bd14-132c-e629-0b42-8ea0fdd5d2a4 (at 10.47.21.31@o2ib1) Aug 20 22:27:02 dac-e-5 kernel: Lustre: Skipped 7 previous similar messages Aug 20 22:27:04 dac-e-5 kernel: LustreError: 189643:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883c707f6400 Aug 20 22:27:04 dac-e-5 kernel: LustreError: 189642:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883c707f6400 Aug 20 22:27:04 dac-e-5 kernel: LustreError: 189641:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883c707f6400 Aug 20 22:27:04 dac-e-5 kernel: LustreError: 189640:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883c707f6400 Aug 20 22:27:04 dac-e-5 kernel: LustreError: 189643:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883c707f6400 Aug 20 22:27:04 dac-e-5 kernel: LustreError: 189641:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883c707f6400 Aug 20 22:27:04 dac-e-5 kernel: LustreError: 189640:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883c707f6400 Aug 20 22:27:04 dac-e-5 kernel: LustreError: 189642:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883c707f6400 Aug 20 22:27:04 dac-e-5 kernel: LustreError: 189648:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883c707f6400 Aug 20 22:27:04 dac-e-5 kernel: LustreError: 189643:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883c707f6400 Aug 20 22:27:04 dac-e-5 kernel: LustreError: 189641:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883c707f6400 Aug 20 22:27:04 dac-e-5 kernel: LustreError: 189640:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883c707f6400 Aug 20 22:27:04 dac-e-5 kernel: LustreError: 189648:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883c707f6400 Aug 20 22:27:04 dac-e-5 kernel: LustreError: 189642:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883c707f6400 Aug 20 22:27:04 dac-e-5 kernel: LustreError: 189640:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883c707f6400 Aug 20 22:27:04 dac-e-5 kernel: LustreError: 189641:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff883c707f6400 Aug 20 22:28:04 dac-e-5 kernel: LustreError: 313799:0:(ldlm_lib.c:3259:target_bulk_io()) @@@ network error on bulk READ req@ffff883c7707b850 x1642422159069504/t0(0) o3->efd44c75-059d-5b3e-f4e6-657896b473fc@10.47.21.34@o2ib1:655/0 lens 488/440 e 1 to 0 dl 1566336490 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 22:28:04 dac-e-5 kernel: Lustre: fs1-OST0038: Bulk IO read error with efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1), client will retry: rc -110 Aug 20 22:40:37 dac-e-5 kernel: Lustre: fs1-OST0038: Client efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) reconnecting Aug 20 22:40:37 dac-e-5 kernel: Lustre: fs1-OST0038: Connection restored to efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) Aug 20 22:40:37 dac-e-5 kernel: Lustre: Skipped 11 previous similar messages Aug 20 22:45:01 dac-e-5 kernel: Lustre: fs1-OST0035: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) HOSTS ------------------------------------------------------------------------- dac-e-24 ------------------------------------------------------------------------------- -- Logs begin at Wed 2019-05-08 14:07:47 BST, end at Wed 2019-08-21 11:30:28 BST. -- Aug 20 22:18:27 dac-e-24 kernel: Lustre: fs1-OST0114: Connection restored to d51aba7f-3e9b-b409-76a8-ec3ceb1b1ca5 (at 10.47.21.35@o2ib1) Aug 20 22:18:27 dac-e-24 kernel: Lustre: Skipped 131 previous similar messages Aug 20 22:27:02 dac-e-24 kernel: Lustre: fs1-OST011f: Connection restored to a9c07af9-d96f-6905-bdea-228af9a88046 (at 10.47.21.32@o2ib1) Aug 20 22:27:02 dac-e-24 kernel: Lustre: Skipped 10 previous similar messages Aug 20 22:45:01 dac-e-24 kernel: Lustre: fs1-OST011a: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 20 22:45:01 dac-e-24 kernel: Lustre: Skipped 10 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-9 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-05-07 20:43:31 BST, end at Wed 2019-08-21 11:30:28 BST. -- Aug 20 22:18:27 dac-e-9 kernel: Lustre: fs1-OST0060: Connection restored to ba6d10d8-29ae-af16-bb29-fe0009074454 (at 10.47.21.33@o2ib1) Aug 20 22:18:27 dac-e-9 kernel: Lustre: fs1-OST0068: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 20 22:18:27 dac-e-9 kernel: Lustre: Skipped 130 previous similar messages Aug 20 22:27:02 dac-e-9 kernel: Lustre: fs1-OST0068: Connection restored to d51aba7f-3e9b-b409-76a8-ec3ceb1b1ca5 (at 10.47.21.35@o2ib1) Aug 20 22:27:02 dac-e-9 kernel: Lustre: Skipped 14 previous similar messages Aug 20 22:45:01 dac-e-9 kernel: Lustre: fs1-OST0069: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 20 22:45:01 dac-e-9 kernel: Lustre: Skipped 5 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-8 ------------------------------------------------------------------------------- -- Logs begin at Thu 2019-05-09 08:54:37 BST, end at Wed 2019-08-21 11:30:28 BST. -- Aug 20 22:18:27 dac-e-8 kernel: Lustre: fs1-OST0056: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 20 22:18:27 dac-e-8 kernel: Lustre: Skipped 133 previous similar messages Aug 20 22:27:02 dac-e-8 kernel: Lustre: fs1-OST005a: Connection restored to ba6d10d8-29ae-af16-bb29-fe0009074454 (at 10.47.21.33@o2ib1) Aug 20 22:27:02 dac-e-8 kernel: Lustre: Skipped 12 previous similar messages Aug 20 22:27:04 dac-e-8 kernel: LustreError: 49825:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fc19e1de000 Aug 20 22:27:04 dac-e-8 kernel: LustreError: 49824:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fc251760800 Aug 20 22:27:04 dac-e-8 kernel: LustreError: 49823:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fc19e1de000 Aug 20 22:27:04 dac-e-8 kernel: LustreError: 49831:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fc19e1de000 Aug 20 22:27:04 dac-e-8 kernel: LustreError: 49826:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fc19e1de000 Aug 20 22:27:04 dac-e-8 kernel: LustreError: 49824:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fc251760800 Aug 20 22:27:04 dac-e-8 kernel: LustreError: 49823:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fc251760800 Aug 20 22:27:04 dac-e-8 kernel: LustreError: 49825:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fc19e1de000 Aug 20 22:27:04 dac-e-8 kernel: LustreError: 49826:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fc19e1de000 Aug 20 22:27:04 dac-e-8 kernel: LustreError: 49831:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fc19e1de000 Aug 20 22:27:04 dac-e-8 kernel: LustreError: 49824:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fc19e1de000 Aug 20 22:27:04 dac-e-8 kernel: LustreError: 49825:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fc19e1de000 Aug 20 22:27:04 dac-e-8 kernel: LustreError: 49826:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fc19e1de000 Aug 20 22:27:04 dac-e-8 kernel: LustreError: 49831:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fc19e1de000 Aug 20 22:27:04 dac-e-8 kernel: LustreError: 49823:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff8fc19e1de000 Aug 20 22:27:04 dac-e-8 kernel: Lustre: fs1-OST005d: Client efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) reconnecting Aug 20 22:27:05 dac-e-8 kernel: LustreError: 140676:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff8fc1e9a2a850 x1642422159068912/t0(0) o3->efd44c75-059d-5b3e-f4e6-657896b473fc@10.47.21.34@o2ib1:630/0 lens 488/440 e 0 to 0 dl 1566336465 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 22:27:05 dac-e-8 kernel: Lustre: fs1-OST005d: Bulk IO read error with efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1), client will retry: rc -110 Aug 20 22:28:04 dac-e-8 kernel: LustreError: 180110:0:(ldlm_lib.c:3259:target_bulk_io()) @@@ network error on bulk READ req@ffff8fc1e9a28850 x1642422159070288/t0(0) o3->efd44c75-059d-5b3e-f4e6-657896b473fc@10.47.21.34@o2ib1:657/0 lens 488/440 e 1 to 0 dl 1566336492 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 22:28:04 dac-e-8 kernel: Lustre: fs1-OST0055: Bulk IO read error with efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1), client will retry: rc -110 Aug 20 22:37:04 dac-e-8 kernel: Lustre: fs1-OST0055: Client efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) reconnecting Aug 20 22:37:04 dac-e-8 kernel: Lustre: fs1-OST0055: Connection restored to efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) Aug 20 22:37:04 dac-e-8 kernel: Lustre: Skipped 8 previous similar messages Aug 20 22:45:01 dac-e-8 kernel: Lustre: fs1-OST005d: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) HOSTS ------------------------------------------------------------------------- dac-e-22 ------------------------------------------------------------------------------- -- Logs begin at Thu 2019-05-09 10:24:44 BST, end at Wed 2019-08-21 11:30:28 BST. -- Aug 20 22:18:27 dac-e-22 kernel: Lustre: fs1-OST00fd: Connection restored to a9c07af9-d96f-6905-bdea-228af9a88046 (at 10.47.21.32@o2ib1) Aug 20 22:18:27 dac-e-22 kernel: Lustre: fs1-OST0107: Connection restored to d51aba7f-3e9b-b409-76a8-ec3ceb1b1ca5 (at 10.47.21.35@o2ib1) Aug 20 22:18:27 dac-e-22 kernel: Lustre: Skipped 129 previous similar messages Aug 20 22:27:02 dac-e-22 kernel: Lustre: fs1-OST0107: Connection restored to efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) Aug 20 22:27:02 dac-e-22 kernel: Lustre: fs1-OST00fe: Connection restored to 3f28bd14-132c-e629-0b42-8ea0fdd5d2a4 (at 10.47.21.31@o2ib1) Aug 20 22:27:02 dac-e-22 kernel: Lustre: Skipped 12 previous similar messages Aug 20 22:45:01 dac-e-22 kernel: Lustre: fs1-OST0100: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 20 22:45:01 dac-e-22 kernel: Lustre: Skipped 10 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-7 ------------------------------------------------------------------------------- -- Logs begin at Tue 2019-05-07 14:07:28 BST, end at Wed 2019-08-21 11:30:28 BST. -- Aug 20 22:18:27 dac-e-7 kernel: Lustre: fs1-OST0050: Connection restored to 3f28bd14-132c-e629-0b42-8ea0fdd5d2a4 (at 10.47.21.31@o2ib1) Aug 20 22:18:27 dac-e-7 kernel: Lustre: Skipped 128 previous similar messages Aug 20 22:27:02 dac-e-7 kernel: Lustre: fs1-OST0051: Connection restored to 4f2faf4f-1754-0923-7bb5-26c935576df5 (at 10.47.21.36@o2ib1) Aug 20 22:27:02 dac-e-7 kernel: Lustre: fs1-OST004a: Connection restored to efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) Aug 20 22:27:02 dac-e-7 kernel: Lustre: fs1-OST0052: Connection restored to 3f28bd14-132c-e629-0b42-8ea0fdd5d2a4 (at 10.47.21.31@o2ib1) Aug 20 22:27:02 dac-e-7 kernel: Lustre: Skipped 13 previous similar messages Aug 20 22:27:02 dac-e-7 kernel: Lustre: Skipped 13 previous similar messages Aug 20 22:27:04 dac-e-7 kernel: Lustre: fs1-OST004a: Client efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) reconnecting Aug 20 22:27:04 dac-e-7 kernel: LNetError: 55106:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-103, 0) Aug 20 22:27:04 dac-e-7 kernel: LustreError: 55106:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff966e0a4c2400 Aug 20 22:27:04 dac-e-7 kernel: LustreError: 55106:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff96865e966e00 Aug 20 22:27:04 dac-e-7 kernel: LustreError: 55106:0:(events.c:450:server_bulk_callback()) event type 5, status -103, desc ffff96865e966e00 Aug 20 22:27:04 dac-e-7 kernel: LustreError: 304431:0:(ldlm_lib.c:3253:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff968771470850 x1642422159069136/t0(0) o3->efd44c75-059d-5b3e-f4e6-657896b473fc@10.47.21.34@o2ib1:630/0 lens 488/440 e 0 to 0 dl 1566336465 ref 1 fl Interpret:/0/0 rc 0/0 Aug 20 22:27:04 dac-e-7 kernel: LustreError: 55114:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff966e0a4c2400 Aug 20 22:27:04 dac-e-7 kernel: LustreError: 55113:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff966e0a4c2400 Aug 20 22:27:04 dac-e-7 kernel: LustreError: 55119:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff96865e966e00 Aug 20 22:27:04 dac-e-7 kernel: LustreError: 55111:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff96865e966e00 Aug 20 22:27:04 dac-e-7 kernel: LustreError: 55112:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff96865e966e00 Aug 20 22:27:04 dac-e-7 kernel: LustreError: 55114:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff96865e966e00 Aug 20 22:27:04 dac-e-7 kernel: LustreError: 55113:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff96865e966e00 Aug 20 22:27:04 dac-e-7 kernel: LustreError: 55119:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff96865e966e00 Aug 20 22:27:04 dac-e-7 kernel: LustreError: 55111:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff96865e966e00 Aug 20 22:27:04 dac-e-7 kernel: LustreError: 55112:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff96865e966e00 Aug 20 22:27:04 dac-e-7 kernel: LustreError: 55114:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff96865e966e00 Aug 20 22:27:04 dac-e-7 kernel: LNet: 55114:0:(o2iblnd_cb.c:413:kiblnd_handle_rx()) PUT_NACK from 10.47.21.34@o2ib1 Aug 20 22:27:04 dac-e-7 kernel: LustreError: 55113:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff966e0a4c2400 Aug 20 22:27:04 dac-e-7 kernel: LustreError: 55114:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff966e0a4c2400 Aug 20 22:27:04 dac-e-7 kernel: LustreError: 55112:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff96865e966e00 Aug 20 22:27:04 dac-e-7 kernel: Lustre: fs1-OST0048: Bulk IO read error with efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1), client will retry: rc -110 Aug 20 22:27:04 dac-e-7 kernel: LustreError: 55111:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff96865e966e00 Aug 20 22:27:04 dac-e-7 kernel: LustreError: 55119:0:(events.c:450:server_bulk_callback()) event type 5, status -5, desc ffff96865e966e00 Aug 20 22:27:29 dac-e-7 kernel: Lustre: fs1-OST0048: Client efd44c75-059d-5b3e-f4e6-657896b473fc (at 10.47.21.34@o2ib1) reconnecting Aug 20 22:27:29 dac-e-7 kernel: Lustre: Skipped 1 previous similar message Aug 20 22:45:01 dac-e-7 kernel: Lustre: fs1-OST0052: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 20 22:45:01 dac-e-7 kernel: Lustre: Skipped 12 previous similar messages HOSTS ------------------------------------------------------------------------- dac-e-6 ------------------------------------------------------------------------------- -- Logs begin at Thu 2019-05-09 08:54:36 BST, end at Wed 2019-08-21 11:30:28 BST. -- Aug 20 22:18:27 dac-e-6 kernel: Lustre: fs1-OST0047: Connection restored to 9a3f8291-8d90-e819-b6ab-9b2c8e66825a (at 10.47.20.69@o2ib1) Aug 20 22:18:27 dac-e-6 kernel: Lustre: Skipped 129 previous similar messages Aug 20 22:27:02 dac-e-6 kernel: Lustre: fs1-OST0045: Connection restored to 3f28bd14-132c-e629-0b42-8ea0fdd5d2a4 (at 10.47.21.31@o2ib1) Aug 20 22:27:02 dac-e-6 kernel: Lustre: Skipped 13 previous similar messages Aug 20 22:45:01 dac-e-6 kernel: Lustre: fs1-OST0046: Connection restored to 60f2ae35-b178-1c80-ba50-e7c416f9922a (at 10.47.20.68@o2ib1) Aug 20 22:45:01 dac-e-6 kernel: Lustre: Skipped 10 previous similar messages