[Fri Apr 19 16:08:01 2019][6063343.959643] Lustre: fir-OST0000: Connection restored to 217aa13d-68cf-b5e7-ea61-382bfbba5454 (at 10.8.17.24@o2ib6) [Fri Apr 19 16:08:01 2019][6063343.970168] Lustre: Skipped 5 previous similar messages [Fri Apr 19 16:08:53 2019][6063395.344125] Lustre: fir-OST000a: haven't heard from client d323bb56-0650-7a93-2b53-f1cdec522901 (at 10.8.15.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983867253800, cur 1555715333 expire 1555715183 last 1555715106 [Fri Apr 19 16:08:53 2019][6063395.365926] Lustre: Skipped 35 previous similar messages [Fri Apr 19 16:09:08 2019][6063410.862774] Lustre: fir-OST0000: Connection restored to e3025880-856b-b6ea-a1a7-a7e183e1dd60 (at 10.8.8.22@o2ib6) [Fri Apr 19 16:09:08 2019][6063410.873211] Lustre: Skipped 11 previous similar messages [Fri Apr 19 16:09:25 2019][6063427.743445] Lustre: fir-OST0000: Connection restored to 9633991e-ce4f-d92c-b6aa-ec983a0f2b80 (at 10.8.8.23@o2ib6) [Fri Apr 19 16:09:25 2019][6063427.753891] Lustre: Skipped 5 previous similar messages [Fri Apr 19 16:09:48 2019][6063451.176750] Lustre: fir-OST0000: Connection restored to dd0510e7-9ffb-771c-249b-7b72018d8d01 (at 10.8.8.17@o2ib6) [Fri Apr 19 16:09:48 2019][6063451.187188] Lustre: Skipped 4 previous similar messages [Fri Apr 19 16:10:43 2019][6063505.461270] Lustre: fir-OST0000: Connection restored to c685ce6c-10e4-7444-bf47-f6501a7232f0 (at 10.8.8.20@o2ib6) [Fri Apr 19 16:10:43 2019][6063505.471728] Lustre: Skipped 24 previous similar messages [Fri Apr 19 16:13:32 2019][6063675.091592] Lustre: fir-OST0000: Connection restored to 155bf5a6-c4ef-a410-9a3a-b316d9df2b69 (at 10.8.8.19@o2ib6) [Fri Apr 19 16:13:32 2019][6063675.102029] Lustre: Skipped 16 previous similar messages [Fri Apr 19 16:16:22 2019][6063844.474186] Lustre: fir-OST0000: Connection restored to 78f38211-beb8-aca3-b985-9281f7d5f62c (at 10.8.29.8@o2ib6) [Fri Apr 19 16:16:22 2019][6063844.484657] Lustre: Skipped 11 previous similar messages [Fri Apr 19 16:18:14 2019][6063957.140884] Lustre: fir-OST0006: Client 4ba8af50-87b8-6d1e-b475-3c7f5b9b9690 (at 10.8.28.3@o2ib6) reconnecting [Fri Apr 19 16:18:14 2019][6063957.151067] Lustre: Skipped 2 previous similar messages [Fri Apr 19 16:18:17 2019][6063959.668426] Lustre: 96573:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1555715890/real 0] req@ffff98567e383900 x1625552284628720/t0(0) o104->fir-OST0000@10.8.27.3@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555715897 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [Fri Apr 19 16:18:17 2019][6063959.694992] Lustre: 96573:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [Fri Apr 19 16:18:33 2019][6063975.972765] Lustre: fir-OST0008: Client 333cc210-b4e9-a76e-7a59-d3765f12cc19 (at 10.8.18.29@o2ib6) reconnecting [Fri Apr 19 16:18:33 2019][6063975.983027] Lustre: Skipped 518 previous similar messages [Fri Apr 19 16:18:35 2019][6063977.964069] LustreError: 96417:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff98576f352c50 x1629186891142352/t0(0) o4->39b21bb1-06bc-8c69-c9c3-5986c42b070c@10.8.0.65@o2ib6:709/0 lens 488/448 e 0 to 0 dl 1555715959 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 19 16:18:35 2019][6063977.988248] LustreError: 96417:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 20 previous similar messages [Fri Apr 19 16:18:41 2019][6063983.542268] Lustre: 110640:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1555715914/real 0] req@ffff983e7ae68600 x1625552284903792/t0(0) o104->fir-OST0004@10.8.18.18@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555715921 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [Fri Apr 19 16:18:41 2019][6063983.569038] Lustre: 110640:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 41 previous similar messages [Fri Apr 19 16:18:44 2019][6063986.859014] LNetError: 91378:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) [Fri Apr 19 16:18:44 2019][6063986.859015] LNetError: 91379:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) [Fri Apr 19 16:18:44 2019][6063986.859020] LNetError: 91379:0:(lib-msg.c:811:lnet_is_health_check()) Skipped 3 previous similar messages [Fri Apr 19 16:18:44 2019][6063986.859023] LustreError: 91379:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff985e457b0600 [Fri Apr 19 16:18:44 2019][6063986.905124] LustreError: 91378:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff985e457b0600 [Fri Apr 19 16:18:44 2019][6063986.916267] Lustre: fir-OST0008: Bulk IO write error with 39b21bb1-06bc-8c69-c9c3-5986c42b070c (at 10.8.0.65@o2ib6), client will retry: rc = -110 [Fri Apr 19 16:18:44 2019][6063986.929500] Lustre: Skipped 18 previous similar messages [Fri Apr 19 16:18:52 2019][6063994.772912] LustreError: 137-5: fir-OST0003_UUID: not available for connect from 10.8.11.25@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Fri Apr 19 16:18:52 2019][6063994.790427] LustreError: Skipped 48 previous similar messages [Fri Apr 19 16:19:11 2019][6064013.629889] Lustre: fir-OST0004: Client 200cd2c0-a998-2321-a861-785b0654f724 (at 10.8.25.4@o2ib6) reconnecting [Fri Apr 19 16:19:11 2019][6064013.640064] Lustre: Skipped 175 previous similar messages [Fri Apr 19 16:19:25 2019][6064027.814945] LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.0.66@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Fri Apr 19 16:19:25 2019][6064027.832316] LustreError: Skipped 20 previous similar messages [Fri Apr 19 16:21:40 2019][6064162.806276] Lustre: fir-OST0000: Connection restored to fbe187bd-3c7e-1c1e-2397-90b673b213a7 (at 10.9.115.3@o2ib4) [Fri Apr 19 16:21:40 2019][6064162.816798] Lustre: Skipped 1276 previous similar messages [Fri Apr 19 16:22:16 2019][6064198.717787] Lustre: fir-OST0004: Client fa7691ef-d46e-650c-947f-cea897c9625f (at 10.8.17.18@o2ib6) reconnecting [Fri Apr 19 16:22:16 2019][6064198.728071] Lustre: Skipped 565 previous similar messages [Fri Apr 19 16:22:18 2019][6064201.104979] Lustre: 109961:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1555716130/real 0] req@ffff986d32cd2d00 x1625552287470464/t0(0) o104->fir-OST0002@10.8.8.7@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555716138 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [Fri Apr 19 16:22:18 2019][6064201.131566] Lustre: 109961:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [Fri Apr 19 16:22:45 2019][6064228.283830] LustreError: 96661:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff98575c85c050 x1628869945600048/t0(0) o4->65c38a59-13b6-ad9d-d264-242871bd2192@10.8.27.18@o2ib6:189/0 lens 536/456 e 1 to 0 dl 1555716194 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 19 16:22:45 2019][6064228.308087] Lustre: fir-OST0002: Bulk IO write error with 65c38a59-13b6-ad9d-d264-242871bd2192 (at 10.8.27.18@o2ib6), client will retry: rc = -110 [Fri Apr 19 16:23:32 2019][6064274.764581] Lustre: 96761:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1555716190/real 0] req@ffff98384da8d700 x1625552287993312/t0(0) o104->fir-OST000a@10.8.8.7@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555716212 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [Fri Apr 19 16:23:32 2019][6064274.791162] Lustre: 96761:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 65 previous similar messages [Fri Apr 19 16:23:43 2019][6064285.634737] LustreError: 137-5: fir-OST0009_UUID: not available for connect from 10.8.27.35@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Fri Apr 19 16:23:52 2019][6064294.561435] LNetError: 91389:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) [Fri Apr 19 16:23:52 2019][6064294.574129] LustreError: 91389:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff98680a641a00 [Fri Apr 19 16:23:52 2019][6064294.585116] Lustre: fir-OST0000: Bulk IO read error with 0900d00b-62db-e11f-db87-53952587e14c (at 10.8.27.35@o2ib6), client will retry: rc -110 [Fri Apr 19 16:23:52 2019][6064294.598185] Lustre: Skipped 18 previous similar messages [Fri Apr 19 16:26:42 2019][6064464.707274] LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 163s: evicting client at 10.8.0.66@o2ib6 ns: filter-fir-OST0006_UUID lock: ffff9857a5b02f40/0x49e1863bbdd3b362 lrc: 3/0,0 mode: PR/PR res: [0x29a919:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.0.66@o2ib6 remote: 0x19dee8bff17baa90 expref: 408925 pid: 96939 timeout: 6064231 lvb_type: 1 [Fri Apr 19 16:26:42 2019][6064464.759684] LustreError: 117039:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.0.66@o2ib6 arrived at 1555716402 with bad export cookie 5323683825047308892 [Fri Apr 19 16:26:42 2019][6064464.775347] LustreError: 117039:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 98 previous similar messages [Fri Apr 19 16:26:46 2019][6064468.639181] LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.27.23@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Fri Apr 19 16:26:46 2019][6064468.656665] LustreError: Skipped 10 previous similar messages [Fri Apr 19 16:26:47 2019][6064469.913456] Lustre: 96366:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1555716400/real 0] req@ffff985561cb5d00 x1625552290575632/t0(0) o104->fir-OST0004@10.8.17.22@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555716407 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [Fri Apr 19 16:26:47 2019][6064469.940101] Lustre: 96366:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 8 previous similar messages [Fri Apr 19 16:26:58 2019][6064480.760789] LustreError: 96402:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.0.66@o2ib6 arrived at 1555716418 with bad export cookie 5323683825047308892 [Fri Apr 19 16:26:58 2019][6064480.776341] LustreError: 96402:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 2573 previous similar messages [Fri Apr 19 16:27:14 2019][6064496.708411] LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 107s: evicting client at 10.8.0.66@o2ib6 ns: filter-fir-OST0008_UUID lock: ffff985060d84140/0x49e1863bbdd1ab8d lrc: 3/0,0 mode: PR/PR res: [0x29a9fa:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 131072->16777215) flags: 0x60000400000020 nid: 10.8.0.66@o2ib6 remote: 0x19dee8bff16ed368 expref: 405566 pid: 96941 timeout: 6064263 lvb_type: 1 [Fri Apr 19 16:27:30 2019][6064512.768063] LustreError: 117028:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.0.66@o2ib6 arrived at 1555716450 with bad export cookie 5323683825047308899 [Fri Apr 19 16:27:30 2019][6064512.783701] LustreError: 117028:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 7720 previous similar messages [Fri Apr 19 16:28:34 2019][6064576.774073] LustreError: 114630:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.0.66@o2ib6 arrived at 1555716514 with bad export cookie 5323683825047308892 [Fri Apr 19 16:28:34 2019][6064576.789709] LustreError: 114630:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 14346 previous similar messages [Fri Apr 19 16:28:44 2019][6064586.731067] Lustre: fir-OST0008: Client 91177ecb-3af9-b598-0df4-bd163eca9e44 (at 10.8.26.3@o2ib6) reconnecting [Fri Apr 19 16:28:44 2019][6064586.741237] Lustre: Skipped 94 previous similar messages [Fri Apr 19 16:28:51 2019][6064594.137309] LustreError: 74747:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff98653efae300 x1625552291973792/t0(0) o104->fir-OST0006@10.8.0.66@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Fri Apr 19 16:29:19 2019][6064621.637933] LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.23.36@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Fri Apr 19 16:29:19 2019][6064621.655389] LustreError: Skipped 74 previous similar messages [Fri Apr 19 16:29:50 2019][6064652.666947] LustreError: 96577:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk READ req@ffff985839c3a850 x1629302920782512/t0(0) o3->5f6bf182-f013-c30f-dbd1-66c29eaf8cf9@10.8.7.29@o2ib6:610/0 lens 488/440 e 0 to 0 dl 1555716615 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 19 16:29:50 2019][6064652.690989] LustreError: 96577:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 1 previous similar message [Fri Apr 19 16:29:58 2019][6064660.646776] LNetError: 91382:0:(lib-msg.c:811:lnet_is_health_check()) Msg is in inconsistent state, don't perform health checking (-125, 0) [Fri Apr 19 16:29:58 2019][6064660.659475] LustreError: 91382:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9851e93f1e00 [Fri Apr 19 16:29:58 2019][6064660.670456] Lustre: fir-OST0006: Bulk IO read error with 5f6bf182-f013-c30f-dbd1-66c29eaf8cf9 (at 10.8.7.29@o2ib6), client will retry: rc -110 [Fri Apr 19 16:30:01 2019][6064664.359295] LustreError: 91382:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff984b2cb54200 [Fri Apr 19 16:30:01 2019][6064664.370280] LustreError: 96622:0:(ldlm_lib.c:3264:target_bulk_io()) @@@ network error on bulk READ req@ffff98583b2f0450 x1630440299927472/t0(0) o3->1fed4b9d-ca5d-165d-887c-2f1234134d0c@10.8.9.10@o2ib6:626/0 lens 488/440 e 2 to 0 dl 1555716631 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 19 16:30:02 2019][6064664.394670] LustreError: 96622:0:(ldlm_lib.c:3264:target_bulk_io()) Skipped 3 previous similar messages [Fri Apr 19 16:30:05 2019][6064668.066453] LustreError: 91385:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff985e3376ba00 [Fri Apr 19 16:30:09 2019][6064671.782204] LustreError: 91383:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9867fdb61e00 [Fri Apr 19 16:30:09 2019][6064671.793175] LustreError: 91383:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9867fdb61e00 [Fri Apr 19 16:30:09 2019][6064671.804147] LustreError: 91382:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9867fdb61e00 [Fri Apr 19 16:30:09 2019][6064671.815099] LustreError: 91383:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9867fdb61e00 [Fri Apr 19 16:30:09 2019][6064671.826374] Lustre: fir-OST0002: Bulk IO write error with 65c38a59-13b6-ad9d-d264-242871bd2192 (at 10.8.27.18@o2ib6), client will retry: rc = -110 [Fri Apr 19 16:30:10 2019][6064673.194406] LustreError: 91384:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff983d46244c00 [Fri Apr 19 16:30:10 2019][6064673.205379] LustreError: 91384:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff983d46244c00 [Fri Apr 19 16:30:10 2019][6064673.216352] LustreError: 91384:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff983d46244c00 [Fri Apr 19 16:30:10 2019][6064673.227332] LustreError: 91384:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff983d46244c00 [Fri Apr 19 16:30:13 2019][6064676.069817] Lustre: fir-OST0004: Connection restored to c8e6faa5-8e52-627a-b276-9b1da9fb48ae (at 10.8.7.23@o2ib6) [Fri Apr 19 16:30:13 2019][6064676.080250] Lustre: Skipped 133 previous similar messages [Fri Apr 19 16:30:14 2019][6064676.621044] LustreError: 91381:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff984b98b85400 [Fri Apr 19 16:30:16 2019][6064679.038902] LustreError: 91382:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff98663e006400 [Fri Apr 19 16:30:20 2019][6064682.685655] LustreError: 91378:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff985d939ba800 [Fri Apr 19 16:30:20 2019][6064683.245611] LustreError: 91384:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff985d69197600 [Fri Apr 19 16:30:21 2019][6064683.718437] LustreError: 91380:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9851f5ce6800 [Fri Apr 19 16:30:26 2019][6064689.093406] LustreError: 91383:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9869776ac800 [Fri Apr 19 16:30:27 2019][6064690.151434] LustreError: 91384:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff986054504a00 [Fri Apr 19 16:30:27 2019][6064690.162408] LustreError: 91384:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff986054504a00 [Fri Apr 19 16:30:27 2019][6064690.173384] LustreError: 91384:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff986054504a00 [Fri Apr 19 16:30:27 2019][6064690.184335] LustreError: 91385:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff986054504a00 [Fri Apr 19 16:30:28 2019][6064690.666676] LustreError: 91386:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff983802a52000 [Fri Apr 19 16:30:29 2019][6064691.407369] LustreError: 91381:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff98386456ac00 [Fri Apr 19 16:30:30 2019][6064692.828361] LustreError: 91383:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff98549f46a400 [Fri Apr 19 16:30:32 2019][6064694.671202] LustreError: 91384:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff985129f12800 [Fri Apr 19 16:30:32 2019][6064694.977716] LustreError: 91382:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9876f132f600 [Fri Apr 19 16:30:34 2019][6064696.761965] LustreError: 91382:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff98499af18a00 [Fri Apr 19 16:30:35 2019][6064697.822600]ts.c:450:server_bulk_callback()) event type 5, status -125, desc ffff985700370600 [Fri Apr 19 16:30:35 2019][6064697.822602] LustreError: 91383:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff986116b07200 [Fri Apr 19 16:30:36 2019][6064698.706293] LustreError: 91381:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff984691127600 [Fri Apr 19 16:30:37 2019][6064699.573134] LustreError: 91378:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9854d98c4600 [Fri Apr 19 16:30:37 2019][6064700.091816] LustreError: 91385:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9860be424400 [Fri Apr 19 16:30:40 2019][6064702.371225] LustreError: 91386:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9849f0cb7800 [Fri Apr 19 16:30:42 2019][6064704.400485] LustreError: 91389:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff983815da6400 [Fri Apr 19 16:30:42 2019][6064704.484192] LustreError: 91381:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff986625062e00 [Fri Apr 19 16:30:42 2019][6064704.599955] LustreError: 91380:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff985143b00a00 [Fri Apr 19 16:30:42 2019][6064704.788885] LustreError: 116993:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) ldlm_cancel from 10.8.0.66@o2ib6 arrived at 1555716642 with bad export cookie 5323683825047308899 [Fri Apr 19 16:30:42 2019][6064704.804529] LustreError: 116993:0:(ldlm_lockd.c:2322:ldlm_cancel_handler()) Skipped 17811 previous similar messages [Fri Apr 19 16:30:43 2019][6064706.222055] LustreError: 91381:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9850fad8be00 [Fri Apr 19 16:30:43 2019][6064706.233017] LustreError: 91381:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff986116b07200 [Fri Apr 19 16:30:44 2019][6064706.739118] LustreError: 91380:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff986065b80c00 [Fri Apr 19 16:30:44 2019][6064706.745835] LustreError: 91378:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff983fb5dadc00 [Fri Apr 19 16:30:45 2019][6064707.430602] LustreError: 91387:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9861fb031e00 [Fri Apr 19 16:30:46 2019][6064708.911790] LustreError: 91381:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9842db33ba00 [Fri Apr 19 16:30:47 2019][6064709.941276] LustreError: 91384:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff984ac0eddc00 [Fri Apr 19 16:30:48 2019][6064710.472736] LustreError: 91379:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff984b79054400 [Fri Apr 19 16:30:49 2019][6064711.830430] LustreError: 91379:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff985df060b600 [Fri Apr 19 16:30:50 2019][6064712.982505] LustreError: 91381:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff98442c397000 [Fri Apr 19 16:30:50 2019][6064713.129162] LustreError: 91385:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff985d8f17fc00 [Fri Apr 19 16:30:50 2019][6064713.175831] LustreError: 91389:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff984e48ddac00 [Fri Apr 19 16:30:52 2019][6064714.388528] LustreError: 91386:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff985d69193a00 [Fri Apr 19 16:30:53 2019][6064715.756179] LustreError: 91383:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff98566fb09c00 [Fri Apr 19 16:30:53 2019][6064716.124831] LustreError: 91384:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff984c7008a800 [Fri Apr 19 16:30:54 2019][6064716.891840] LustreError: 91378:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff985799d6e800 [Fri Apr 19 16:30:55 2019][6064718.306768] LustreError: 91388:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff986065bc2e00 [Fri Apr 19 16:30:57 2019][6064719.919643] LustreError: 91385:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff984aa1408000 [Fri Apr 19 16:31:00 2019][6064722.794578] LustreError: 91384:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff986c361e0a00 [Fri Apr 19 16:31:00 2019][6064722.867099] LustreError: 91386:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9861fb7f6e00 [Fri Apr 19 16:31:05 2019][6064727.853783] LustreError: 91389:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff98717305bc00 [Fri Apr 19 16:31:05 2019][6064728.073204] LustreError: 91380:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9844e3545200 [Fri Apr 19 16:31:05 2019][6064728.084181] LustreError: 91379:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9844e3545200 [Fri Apr 19 16:31:05 2019][6064728.095154] LustreError: 91380:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9844e3545200 [Fri Apr 19 16:31:05 2019][6064728.106124] LustreError: 91379:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9844e3545200 [Fri Apr 19 16:31:13 2019][6064736.236906] Lustre: 74748:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1555716666/real 0] req@ffff9867fbb7f200 x1625552293286480/t0(0) o104->fir-OST0004@10.8.27.15@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555716673 ref 2 fl Rpc:X/0/ffffffff rc 0/-1 [Fri Apr 19 16:31:13 2019][6064736.263579] Lustre: 74748:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 61 previous similar messages [Fri Apr 19 16:31:18 2019][6064740.508933] LustreError: 91389:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9844795ba400 [Fri Apr 19 16:31:18 2019][6064741.289163] LustreError: 91385:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff985564776c00 [Fri Apr 19 16:31:18 2019][6064741.300139] LustreError: 96422:0:(ldlm_lib.c:3264:target_bulk_io()) @@@ network error on bulk WRITE req@ffff98576f354450 x1628600641536480/t0(0) o4->b385686e-2979-95e2-2cff-881997c248f8@10.8.18.21@o2ib6:701/0 lens 488/448 e 0 to 0 dl 1555716706 ref 1 fl Interpret:/0/0 rc 0/0 [Fri Apr 19 16:31:18 2019][6064741.324714] LustreError: 96422:0:(ldlm_lib.c:3264:target_bulk_io()) Skipped 15 previous similar messages [Fri Apr 19 16:31:21 2019][6064743.806642] LustreError: 91389:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9843845b8600 [Fri Apr 19 16:32:16 2019][6064798.397359] Lustre: fir-OST000a: haven't heard from client 5b697899-e91c-7cc6-c44a-da5d20d26539 (at 10.9.101.58@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98732958fc00, cur 1555716736 expire 1555716586 last 1555716509 [Fri Apr 19 16:32:16 2019][6064798.419320] Lustre: Skipped 11 previous similar messages [Fri Apr 19 16:33:58 2019][6064901.241745] Lustre: fir-OST0000: Client 9e6c31ba-18c6-5055-4d79-460d2380c61d (at 10.8.0.66@o2ib6) reconnecting [Fri Apr 19 16:33:58 2019][6064901.251919] Lustre: Skipped 105 previous similar messages [Fri Apr 19 16:34:41 2019][6064943.398780] Lustre: fir-OST0006: haven't heard from client 9e6c31ba-18c6-5055-4d79-460d2380c61d (at 10.8.0.66@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9863e663ec00, cur 1555716881 expire 1555716731 last 1555716654 [Fri Apr 19 16:34:41 2019][6064943.420608] Lustre: Skipped 5 previous similar messages [Fri Apr 19 16:36:26 2019][6065048.654997] LustreError: 137-5: fir-OST0001_UUID: not available for connect from 10.8.16.4@o2ib6 (no target). If you are running an HA pair check that the target is mounted on the other server. [Fri Apr 19 16:36:26 2019][6065048.672370] LustreError: Skipped 169 previous similar messages [Fri Apr 19 16:38:32 2019][6065174.575368] LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 0 seconds [Fri Apr 19 16:38:32 2019][6065174.585540] LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.202@o2ib7 (0): c: 0, oc: 3, rc: 8 [Fri Apr 19 16:38:32 2019][6065174.601605] LNet: 91376:0:(o2iblnd_cb.c:1484:kiblnd_reconnect_peer()) Abort reconnection of 10.0.10.204@o2ib7: accepting [Fri Apr 19 16:39:52 2019][6065254.994219] Lustre: 74743:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555717185/real 1555717185] req@ffff9863d9d0ad00 x1625552297812704/t0(0) o104->fir-OST0002@10.8.0.66@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555717192 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Fri Apr 19 16:39:52 2019][6065255.021670] Lustre: 74743:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 204 previous similar messages [Fri Apr 19 16:40:17 2019][6065279.419291] Lustre: fir-OST0000: haven't heard from client 9e6c31ba-18c6-5055-4d79-460d2380c61d (at 10.8.0.66@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986e320c5800, cur 1555717217 expire 1555717067 last 1555716990 [Fri Apr 19 16:40:17 2019][6065279.441104] Lustre: Skipped 1 previous similar message [Fri Apr 19 16:41:09 2019][6065332.033953] LustreError: 74743:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.0.66@o2ib6) failed to reply to blocking AST (req@ffff9863d9d0ad00 x1625552297812704 status 0 rc -110), evict it ns: filter-fir-OST0002_UUID lock: ffff9863b1191d40/0x49e1863bbdea2d07 lrc: 4/0,0 mode: PR/PR res: [0x29b6d4:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8388608->536870911) flags: 0x60000400010020 nid: 10.8.0.66@o2ib6 remote: 0x19dee8bff1fe2524 expref: 313467 pid: 74799 timeout: 6065241 lvb_type: 1 [Fri Apr 19 16:41:09 2019][6065332.080017] LustreError: 74743:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) Skipped 2 previous similar messages [Fri Apr 19 16:41:09 2019][6065332.090275] LustreError: 138-a: fir-OST0002: A client on nid 10.8.0.66@o2ib6 was evicted due to a lock blocking callback time out: rc -110 [Fri Apr 19 16:41:09 2019][6065332.102874] LustreError: Skipped 2 previous similar messages [Fri Apr 19 16:41:09 2019][6065332.108735] LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 154s: evicting client at 10.8.0.66@o2ib6 ns: filter-fir-OST0002_UUID lock: ffff9863b1191d40/0x49e1863bbdea2d07 lrc: 3/0,0 mode: PR/PR res: [0x29b6d4:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8388608->536870911) flags: 0x60000400010020 nid: 10.8.0.66@o2ib6 remote: 0x19dee8bff1fe2524 expref: 313468 pid: 74799 timeout: 0 lvb_type: 1 [Fri Apr 19 16:42:21 2019][6065403.449329] Lustre: fir-OST0004: Connection restored to d851fba9-b115-cdc7-e280-01f5ac21500a (at 10.8.13.5@o2ib6) [Fri Apr 19 16:42:21 2019][6065403.459763] Lustre: Skipped 79 previous similar messages [Fri Apr 19 16:43:35 2019][6065477.991691] LustreError: 96524:0:(client.c:1175:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff984c9b073000 x1625552300462480/t0(0) o104->fir-OST0004@10.8.0.66@o2ib6:15/16 lens 296/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1 [Fri Apr 19 16:48:08 2019][6065750.428248] Lustre: fir-OST0006: haven't heard from client d93aae95-4c84-3cee-19a1-32679456e895 (at 10.8.28.1@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9876b0154c00, cur 1555717688 expire 1555717538 last 1555717461 [Fri Apr 19 16:48:08 2019][6065750.450041] Lustre: Skipped 2 previous similar messages [Fri Apr 19 16:58:56 2019][6066399.188083] Lustre: fir-OST0000: Connection restored to 894037b6-9521-3739-9321-f4e3bd742c13 (at 10.8.10.12@o2ib6) [Fri Apr 19 16:58:56 2019][6066399.198629] Lustre: Skipped 19 previous similar messages [Fri Apr 19 17:01:59 2019][6066581.592119] Lustre: 110634:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555718512/real 1555718512] req@ffff9838691c5a00 x1625552312480176/t0(0) o106->fir-OST0002@10.8.27.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1555718519 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Fri Apr 19 17:01:59 2019][6066581.619678] Lustre: 110634:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 14 previous similar messages [Fri Apr 19 17:02:32 2019][6066614.458223] Lustre: fir-OST0008: haven't heard from client 3d638426-8b6f-9f50-0054-935d9f75892d (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9865b2f59c00, cur 1555718552 expire 1555718402 last 1555718325 [Fri Apr 19 17:02:32 2019][6066614.480099] Lustre: Skipped 5 previous similar messages [Fri Apr 19 17:10:04 2019][6067066.613622] Lustre: fir-OST0000: Connection restored to f92f9622-3835-3057-15b3-90b2bfd416b2 (at 10.9.114.5@o2ib4) [Fri Apr 19 17:10:04 2019][6067066.624173] Lustre: Skipped 23 previous similar messages [Fri Apr 19 17:13:21 2019][6067263.480909] Lustre: fir-OST0002: haven't heard from client 7b135a2d-31cd-4608-106e-618d3f26436c (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575ef14800, cur 1555719201 expire 1555719051 last 1555718974 [Fri Apr 19 17:13:21 2019][6067263.502806] Lustre: Skipped 17 previous similar messages [Fri Apr 19 17:14:04 2019][6067306.998737] Lustre: 74818:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555719237/real 1555719237] req@ffff98634ed36000 x1625552330847648/t0(0) o106->fir-OST000a@10.9.112.14@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1555719244 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Fri Apr 19 17:14:04 2019][6067307.026277] Lustre: 74818:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 65 previous similar messages [Fri Apr 19 17:14:18 2019][6067320.999236] Lustre: 110634:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555719251/real 1555719251] req@ffff98600b462100 x1625552330847680/t0(0) o106->fir-OST0000@10.9.112.14@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1555719258 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Fri Apr 19 17:14:18 2019][6067321.026855] Lustre: 110634:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [Fri Apr 19 17:14:39 2019][6067341.999987] Lustre: 96574:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555719272/real 1555719272] req@ffff98653efaad00 x1625552330847664/t0(0) o106->fir-OST0002@10.9.112.14@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1555719279 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Fri Apr 19 17:14:39 2019][6067342.027527] Lustre: 96574:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 9 previous similar messages [Fri Apr 19 17:15:21 2019][6067384.002473] Lustre: 96370:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555719314/real 1555719314] req@ffff9861f1b3fb00 x1625552330847632/t0(0) o106->fir-OST0008@10.9.112.14@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1555719321 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Fri Apr 19 17:15:21 2019][6067384.030029] Lustre: 96370:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 19 previous similar messages [Fri Apr 19 17:16:38 2019][6067461.041200] Lustre: 74818:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555719391/real 1555719391] req@ffff98634ed36000 x1625552330847648/t0(0) o106->fir-OST000a@10.9.112.14@o2ib4:15/16 lens 296/280 e 0 to 1 dl 1555719398 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Fri Apr 19 17:16:38 2019][6067461.068738] Lustre: 74818:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 41 previous similar messages [Fri Apr 19 17:17:17 2019][6067500.341571] LNet: Service thread pid 110634 was inactive for 200.33s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Fri Apr 19 17:17:17 2019][6067500.358775] LNet: Skipped 2 previous similar messages [Fri Apr 19 17:17:17 2019][6067500.364051] Pid: 110634, comm: ll_ost02_100 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 19 17:17:17 2019][6067500.374312] Call Trace: [Fri Apr 19 17:17:17 2019][6067500.376957] [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] [Fri Apr 19 17:17:17 2019][6067500.383767] [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] [Fri Apr 19 17:17:17 2019][6067500.390646] [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] [Fri Apr 19 17:17:17 2019][6067500.397615] [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] [Fri Apr 19 17:17:17 2019][6067500.404296] [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 19 17:17:17 2019][6067500.411266] [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 19 17:17:17 2019][6067500.418577] [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 19 17:17:17 2019][6067500.424942] [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 19 17:17:17 2019][6067500.432091] [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 19 17:17:17 2019][6067500.439998] [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 19 17:17:17 2019][6067500.446513] [<ffffffff850c1c31>] kthread+0xd1/0xe0 [Fri Apr 19 17:17:17 2019][6067500.451607] [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 19 17:17:18 2019][6067500.458295] [<ffffffffffffffff>] 0xffffffffffffffff [Fri Apr 19 17:17:18 2019][6067500.463503] LustreError: dumping log to /tmp/lustre-log.1555719437.110634 [Fri Apr 19 17:17:19 2019][6067501.833875] LNet: Service thread pid 96370 was inactive for 201.82s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Fri Apr 19 17:17:19 2019][6067501.851003] Pid: 96370, comm: ll_ost02_025 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 19 17:17:19 2019][6067501.861217] Call Trace: [Fri Apr 19 17:17:19 2019][6067501.863866] [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] [Fri Apr 19 17:17:19 2019][6067501.870654] [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] [Fri Apr 19 17:17:19 2019][6067501.877512] [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] [Fri Apr 19 17:17:19 2019][6067501.884473] [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] [Fri Apr 19 17:17:19 2019][6067501.891135] [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 19 17:17:19 2019][6067501.898100] [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 19 17:17:19 2019][6067501.905401] [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 19 17:17:19 2019][6067501.911803] [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 19 17:17:19 2019][6067501.918931] [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 19 17:17:19 2019][6067501.926832] [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 19 17:17:19 2019][6067501.933372] [<ffffffff850c1c31>] kthread+0xd1/0xe0 [Fri Apr 19 17:17:19 2019][6067501.938464] [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 19 17:17:19 2019][6067501.945156] [<ffffffffffffffff>] 0xffffffffffffffff [Fri Apr 19 17:17:19 2019][6067501.950359] Pid: 96574, comm: ll_ost02_040 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 19 17:17:19 2019][6067501.960551] Call Trace: [Fri Apr 19 17:17:19 2019][6067501.963189] [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] [Fri Apr 19 17:17:19 2019][6067501.969969] [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] [Fri Apr 19 17:17:19 2019][6067501.976834] [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] [Fri Apr 19 17:17:19 2019][6067501.983778] [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] [Fri Apr 19 17:17:19 2019][6067501.990439] [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 19 17:17:19 2019][6067501.997398] [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 19 17:17:19 2019][6067502.004716] [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 19 17:17:19 2019][6067502.011070] [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 19 17:17:19 2019][6067502.018193] [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 19 17:17:19 2019][6067502.026112] [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 19 17:17:19 2019][6067502.032626] [<ffffffff850c1c31>] kthread+0xd1/0xe0 [Fri Apr 19 17:17:19 2019][6067502.037719] [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 19 17:17:19 2019][6067502.044375] [<ffffffffffffffff>] 0xffffffffffffffff [Fri Apr 19 17:17:19 2019][6067502.049580] Pid: 74818, comm: ll_ost02_091 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 19 17:17:19 2019][6067502.059753] Call Trace: [Fri Apr 19 17:17:19 2019][6067502.062393] [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] [Fri Apr 19 17:17:19 2019][6067502.069181] [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] [Fri Apr 19 17:17:19 2019][6067502.076035] [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] [Fri Apr 19 17:17:19 2019][6067502.082968] [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] [Fri Apr 19 17:17:19 2019][6067502.089637] [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 19 17:17:19 2019][6067502.096653] [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 19 17:17:19 2019][6067502.103954] [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 19 17:17:19 2019][6067502.110348] [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 19 17:17:19 2019][6067502.117486] [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 19 17:17:19 2019][6067502.125415] [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 19 17:17:19 2019][6067502.131941] [<ffffffff850c1c31>] kthread+0xd1/0xe0 [Fri Apr 19 17:17:19 2019][6067502.137069] [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 19 17:17:19 2019][6067502.143725] [<ffffffffffffffff>] 0xffffffffffffffff [Fri Apr 19 17:17:29 2019][6067511.489178] LNet: Service thread pid 96574 completed after 211.48s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Fri Apr 19 17:17:29 2019][6067511.505508] LNet: Skipped 3 previous similar messages [Fri Apr 19 17:32:10 2019][6068392.520602] Lustre: fir-OST000a: haven't heard from client 0d21a5ef-85f7-0841-9caf-55fb6be88fbf (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983fb2650800, cur 1555720330 expire 1555720180 last 1555720103 [Fri Apr 19 17:32:10 2019][6068392.542506] Lustre: Skipped 11 previous similar messages [Fri Apr 19 17:35:11 2019][6068574.260267] Lustre: fir-OST0000: Connection restored to a5728575-a4f7-6b0a-f0a9-44d0ea52ed96 (at 10.9.101.59@o2ib4) [Fri Apr 19 17:35:11 2019][6068574.270880] Lustre: Skipped 16 previous similar messages [Fri Apr 19 17:38:36 2019][6068779.121109] Lustre: 49830:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555720709/real 1555720709] req@ffff9847d023bc00 x1625552359574784/t0(0) o106->fir-OST0002@10.8.1.1@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1555720716 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Fri Apr 19 17:38:36 2019][6068779.148414] Lustre: 49830:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 31 previous similar messages [Fri Apr 19 17:38:57 2019][6068800.121858] Lustre: 96911:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555720730/real 1555720730] req@ffff986526fd2100 x1625552359574768/t0(0) o106->fir-OST000a@10.8.1.1@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1555720737 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Fri Apr 19 17:38:57 2019][6068800.149127] Lustre: 96911:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages [Fri Apr 19 17:39:18 2019][6068820.676409] Lustre: fir-OST0000: Connection restored to bef16258-699d-0e14-bdeb-b454fac00d89 (at 10.9.112.15@o2ib4) [Fri Apr 19 17:39:18 2019][6068820.687025] Lustre: Skipped 11 previous similar messages [Fri Apr 19 17:39:39 2019][6068842.124360] Lustre: 96919:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555720772/real 1555720772] req@ffff98384c1e1200 x1625552359574816/t0(0) o106->fir-OST0004@10.8.1.1@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1555720779 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Fri Apr 19 17:39:39 2019][6068842.151776] Lustre: 96919:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 46 previous similar messages [Fri Apr 19 17:40:56 2019][6068919.153108] Lustre: 110621:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555720849/real 1555720849] req@ffff983e7e209b00 x1625552359574800/t0(0) o106->fir-OST0000@10.8.1.1@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1555720856 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Fri Apr 19 17:40:56 2019][6068919.180573] Lustre: 110621:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 82 previous similar messages [Fri Apr 19 17:43:26 2019][6069069.472482] Lustre: 109985:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555720999/real 1555720999] req@ffff9871208b9800 x1625552362615568/t0(0) o106->fir-OST0002@10.8.15.8@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1555721006 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Fri Apr 19 17:43:26 2019][6069069.499961] Lustre: 109985:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 199 previous similar messages [Fri Apr 19 17:43:52 2019][6069094.545522] Lustre: fir-OST0004: haven't heard from client 8fb6c491-b5b4-1524-6dad-8ba5b0398f17 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986780860800, cur 1555721032 expire 1555720882 last 1555720805 [Fri Apr 19 17:43:52 2019][6069094.567398] Lustre: Skipped 137 previous similar messages [Fri Apr 19 17:55:53 2019][6069816.249451] Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) [Fri Apr 19 17:55:53 2019][6069816.259973] Lustre: Skipped 11 previous similar messages [Fri Apr 19 18:00:05 2019][6070067.582420] Lustre: fir-OST0000: haven't heard from client be7769d0-8b5a-bd60-92d4-cef4bb21b592 (at 10.8.24.3@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9866d0a7bc00, cur 1555722005 expire 1555721855 last 1555721778 [Fri Apr 19 18:00:05 2019][6070067.604248] Lustre: Skipped 23 previous similar messages [Fri Apr 19 18:04:38 2019][6070340.729483] Lustre: fir-OST0000: Connection restored to 7751f34c-206a-edf3-4c97-12b3ac7dcb0e (at 10.8.23.14@o2ib6) [Fri Apr 19 18:04:38 2019][6070340.740009] Lustre: Skipped 5 previous similar messages [Fri Apr 19 18:06:34 2019][6070456.695679] Lustre: fir-OST0000: Connection restored to 894037b6-9521-3739-9321-f4e3bd742c13 (at 10.8.10.12@o2ib6) [Fri Apr 19 18:06:34 2019][6070456.706232] Lustre: Skipped 11 previous similar messages [Fri Apr 19 18:09:25 2019][6070628.585113] Lustre: fir-OST0000: Connection restored to 7408574f-6b07-66ec-bd6d-34d66cefdfdc (at 10.8.11.28@o2ib6) [Fri Apr 19 18:09:26 2019][6070628.595637] Lustre: Skipped 11 previous similar messages [Fri Apr 19 18:12:06 2019][6070788.662916] Lustre: fir-OST0000: Connection restored to c22e763b-2712-8624-a4bb-1c3145d32fd9 (at 10.8.1.14@o2ib6) [Fri Apr 19 18:12:06 2019][6070788.673358] Lustre: Skipped 17 previous similar messages [Fri Apr 19 18:18:13 2019][6071156.023608] Lustre: fir-OST0000: Connection restored to 4d299f05-b526-db6f-54d0-b5a94e0b6d6e (at 10.8.25.33@o2ib6) [Fri Apr 19 18:18:13 2019][6071156.034140] Lustre: Skipped 118 previous similar messages [Fri Apr 19 18:20:50 2019][6071312.626073] Lustre: fir-OST0004: haven't heard from client 176b6467-807b-8109-e9b3-a254586b0b25 (at 10.8.23.25@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985839e52800, cur 1555723250 expire 1555723100 last 1555723023 [Fri Apr 19 18:20:50 2019][6071312.648004] Lustre: Skipped 17 previous similar messages [Fri Apr 19 18:25:31 2019][6071593.640950] Lustre: fir-OST0008: haven't heard from client acac6f70-8d89-9fc3-7694-baeb7f584e24 (at 10.8.24.30@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857e3995c00, cur 1555723531 expire 1555723381 last 1555723304 [Fri Apr 19 18:25:31 2019][6071593.662846] Lustre: Skipped 5 previous similar messages [Fri Apr 19 18:28:16 2019][6071758.643892] Lustre: fir-OST0006: haven't heard from client fb870f29-87b5-3d76-2f08-47467ebddd1f (at 10.8.26.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9858b60b9c00, cur 1555723696 expire 1555723546 last 1555723469 [Fri Apr 19 18:28:16 2019][6071758.665808] Lustre: Skipped 5 previous similar messages [Fri Apr 19 18:30:13 2019][6071875.955132] Lustre: fir-OST0000: Connection restored to fb72c5b2-e25a-6223-1e9a-b5e9009fe8ba (at 10.8.22.19@o2ib6) [Fri Apr 19 18:30:13 2019][6071875.965655] Lustre: Skipped 11 previous similar messages [Fri Apr 19 18:33:57 2019][6072099.656281] Lustre: fir-OST0006: haven't heard from client dd6faf8c-dfaf-a808-02be-6dd27bd582dc (at 10.8.23.22@o2ib6) in 226 seconds. I think it's dead, and I am evicting it. exp ffff985839c3c000, cur 1555724037 expire 1555723887 last 1555723811 [Fri Apr 19 18:33:57 2019][6072099.678216] Lustre: Skipped 66 previous similar messages [Fri Apr 19 18:48:28 2019][6072971.051351] Lustre: fir-OST0000: Connection restored to 986554b5-b5b8-a10d-3b95-d370e13472ba (at 10.8.23.25@o2ib6) [Fri Apr 19 18:48:28 2019][6072971.061870] Lustre: Skipped 5 previous similar messages [Fri Apr 19 18:48:41 2019][6072984.312872] Lustre: 75603:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555724914/real 1555724914] req@ffff9867fbb7d700 x1625552469227952/t0(0) o104->fir-OST0000@10.8.27.23@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555724921 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Fri Apr 19 18:48:41 2019][6072984.340320] Lustre: 75603:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 61 previous similar messages [Fri Apr 19 18:49:23 2019][6073026.351438] Lustre: 75603:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555724956/real 1555724956] req@ffff9867fbb7d700 x1625552469227952/t0(0) o104->fir-OST0000@10.8.27.23@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555724963 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Fri Apr 19 18:49:23 2019][6073026.378869] Lustre: 75603:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [Fri Apr 19 18:49:33 2019][6073035.697610] Lustre: fir-OST0002: haven't heard from client 74b81c99-be13-e368-a07e-ef6de254c5e3 (at 10.8.25.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867811fac00, cur 1555724973 expire 1555724823 last 1555724746 [Fri Apr 19 18:49:33 2019][6073035.719517] Lustre: Skipped 76 previous similar messages [Fri Apr 19 18:50:40 2019][6073103.392296] Lustre: 75603:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555725033/real 1555725033] req@ffff9867fbb7d700 x1625552469227952/t0(0) o104->fir-OST0000@10.8.27.23@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555725040 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Fri Apr 19 18:50:40 2019][6073103.419768] Lustre: 75603:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 10 previous similar messages [Fri Apr 19 18:51:08 2019][6073131.432355] LustreError: 75603:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.27.23@o2ib6) failed to reply to blocking AST (req@ffff9867fbb7d700 x1625552469227952 status 0 rc -110), evict it ns: filter-fir-OST0000_UUID lock: ffff984ca5efba80/0x49e1863bc2dc8a94 lrc: 4/0,0 mode: PW/PW res: [0x6c0000400:0x54c3ba:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8388608->8392703) flags: 0x60000400010020 nid: 10.8.27.23@o2ib6 remote: 0x37ab9c588fb8e393 expref: 155 pid: 96399 timeout: 6073040 lvb_type: 0 [Fri Apr 19 18:51:08 2019][6073131.478855] LustreError: 138-a: fir-OST0000: A client on nid 10.8.27.23@o2ib6 was evicted due to a lock blocking callback time out: rc -110 [Fri Apr 19 18:51:08 2019][6073131.491570] LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 154s: evicting client at 10.8.27.23@o2ib6 ns: filter-fir-OST0000_UUID lock: ffff984ca5efba80/0x49e1863bc2dc8a94 lrc: 3/0,0 mode: PW/PW res: [0x6c0000400:0x54c3ba:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8388608->8392703) flags: 0x60000400010020 nid: 10.8.27.23@o2ib6 remote: 0x37ab9c588fb8e393 expref: 156 pid: 96399 timeout: 0 lvb_type: 0 [Fri Apr 19 19:00:29 2019][6073692.424552] Lustre: fir-OST0000: Connection restored to 85810729-2f82-b4f2-1241-3806d86f03d3 (at 10.8.30.6@o2ib6) [Fri Apr 19 19:00:29 2019][6073692.434993] Lustre: Skipped 17 previous similar messages [Fri Apr 19 19:00:39 2019][6073701.715362] Lustre: fir-OST0006: haven't heard from client 7d180e70-2cd0-f62e-555f-8087c456d881 (at 10.8.12.20@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985758d2ec00, cur 1555725639 expire 1555725489 last 1555725412 [Fri Apr 19 19:00:39 2019][6073701.737399] Lustre: Skipped 22 previous similar messages [Fri Apr 19 19:02:47 2019][6073830.269341] Lustre: 96898:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555725760/real 1555725760] req@ffff986cf57f6000 x1625552513921040/t0(0) o106->fir-OST0008@10.8.15.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1555725767 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Fri Apr 19 19:02:47 2019][6073830.296728] Lustre: 96898:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [Fri Apr 19 19:06:00 2019][6074023.459594] LNet: Service thread pid 96942 was inactive for 200.18s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Fri Apr 19 19:06:00 2019][6074023.476707] LNet: Skipped 2 previous similar messages [Fri Apr 19 19:06:00 2019][6074023.481942] Pid: 96942, comm: ll_ost01_110 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 19 19:06:00 2019][6074023.492147] Call Trace: [Fri Apr 19 19:06:00 2019][6074023.494785] [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] [Fri Apr 19 19:06:00 2019][6074023.501568] [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] [Fri Apr 19 19:06:00 2019][6074023.508429] [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] [Fri Apr 19 19:06:00 2019][6074023.515394] [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] [Fri Apr 19 19:06:00 2019][6074023.522069] [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 19 19:06:00 2019][6074023.529045] [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 19 19:06:00 2019][6074023.536345] [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 19 19:06:00 2019][6074023.542697] [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 19 19:06:00 2019][6074023.549852] [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 19 19:06:00 2019][6074023.557759] [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 19 19:06:00 2019][6074023.564278] [<ffffffff850c1c31>] kthread+0xd1/0xe0 [Fri Apr 19 19:06:00 2019][6074023.569391] [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 19 19:06:00 2019][6074023.576048] [<ffffffffffffffff>] 0xffffffffffffffff [Fri Apr 19 19:06:00 2019][6074023.581278] LustreError: dumping log to /tmp/lustre-log.1555725960.96942 [Fri Apr 19 19:06:01 2019][6074023.965723] LNet: Service thread pid 96274 was inactive for 200.68s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes: [Fri Apr 19 19:06:01 2019][6074023.982860] Pid: 96274, comm: ll_ost01_021 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 19 19:06:01 2019][6074023.993038] Call Trace: [Fri Apr 19 19:06:01 2019][6074023.995693] [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] [Fri Apr 19 19:06:01 2019][6074024.002488] [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] [Fri Apr 19 19:06:01 2019][6074024.009387] [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] [Fri Apr 19 19:06:01 2019][6074024.016333] [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] [Fri Apr 19 19:06:01 2019][6074024.023013] [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 19 19:06:01 2019][6074024.029972] [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 19 19:06:01 2019][6074024.037293] [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 19 19:06:01 2019][6074024.043655] [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 19 19:06:01 2019][6074024.050810] [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 19 19:06:01 2019][6074024.058728] [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 19 19:06:01 2019][6074024.065255] [<ffffffff850c1c31>] kthread+0xd1/0xe0 [Fri Apr 19 19:06:01 2019][6074024.070382] [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 19 19:06:01 2019][6074024.077083] [<ffffffffffffffff>] 0xffffffffffffffff [Fri Apr 19 19:06:01 2019][6074024.082305] Pid: 96898, comm: ll_ost01_086 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 19 19:06:01 2019][6074024.092520] Call Trace: [Fri Apr 19 19:06:01 2019][6074024.095169] [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] [Fri Apr 19 19:06:01 2019][6074024.101953] [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] [Fri Apr 19 19:06:01 2019][6074024.108854] [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] [Fri Apr 19 19:06:01 2019][6074024.115818] [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] [Fri Apr 19 19:06:01 2019][6074024.122491] [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 19 19:06:01 2019][6074024.129483] [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 19 19:06:01 2019][6074024.136769] [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 19 19:06:01 2019][6074024.143154] [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 19 19:06:01 2019][6074024.150281] [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 19 19:06:01 2019][6074024.158209] [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 19 19:06:01 2019][6074024.164731] [<ffffffff850c1c31>] kthread+0xd1/0xe0 [Fri Apr 19 19:06:01 2019][6074024.169860] [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 19 19:06:01 2019][6074024.176511] [<ffffffffffffffff>] 0xffffffffffffffff [Fri Apr 19 19:06:01 2019][6074024.181724] Pid: 96370, comm: ll_ost02_025 3.10.0-957.1.3.el7_lustre.x86_64 #1 SMP Fri Dec 7 14:50:35 PST 2018 [Fri Apr 19 19:06:01 2019][6074024.191934] Call Trace: [Fri Apr 19 19:06:01 2019][6074024.194581] [<ffffffffc0dcd890>] ptlrpc_set_wait+0x500/0x8d0 [ptlrpc] [Fri Apr 19 19:06:01 2019][6074024.201345] [<ffffffffc0d8b185>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc] [Fri Apr 19 19:06:01 2019][6074024.208217] [<ffffffffc0dac86b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc] [Fri Apr 19 19:06:01 2019][6074024.215194] [<ffffffffc166b10b>] ofd_intent_policy+0x69b/0x920 [ofd] [Fri Apr 19 19:06:01 2019][6074024.221865] [<ffffffffc0d8bec6>] ldlm_lock_enqueue+0x366/0xa60 [ptlrpc] [Fri Apr 19 19:06:01 2019][6074024.228794] [<ffffffffc0db48a7>] ldlm_handle_enqueue0+0xa47/0x15a0 [ptlrpc] [Fri Apr 19 19:06:01 2019][6074024.236101] [<ffffffffc0e3b302>] tgt_enqueue+0x62/0x210 [ptlrpc] [Fri Apr 19 19:06:01 2019][6074024.242443] [<ffffffffc0e4235a>] tgt_request_handle+0xaea/0x1580 [ptlrpc] [Fri Apr 19 19:06:01 2019][6074024.249588] [<ffffffffc0de692b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc] [Fri Apr 19 19:06:01 2019][6074024.257476] [<ffffffffc0dea25c>] ptlrpc_main+0xafc/0x1fc0 [ptlrpc] [Fri Apr 19 19:06:01 2019][6074024.263976] [<ffffffff850c1c31>] kthread+0xd1/0xe0 [Fri Apr 19 19:06:01 2019][6074024.269063] [<ffffffff85774c24>] ret_from_fork_nospec_begin+0xe/0x21 [Fri Apr 19 19:06:01 2019][6074024.275711] [<ffffffffffffffff>] 0xffffffffffffffff [Fri Apr 19 19:06:23 2019][6074045.727515] LNet: Service thread pid 96274 completed after 222.44s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources). [Fri Apr 19 19:06:23 2019][6074045.743915] LNet: Skipped 3 previous similar messages [Fri Apr 19 19:10:47 2019][6074310.479823] Lustre: fir-OST0000: Connection restored to 80a4d388-dfdd-5a6f-35e8-374e461c44ba (at 10.8.12.35@o2ib6) [Fri Apr 19 19:10:47 2019][6074310.490348] Lustre: Skipped 125 previous similar messages [Fri Apr 19 19:12:51 2019][6074433.742214] Lustre: fir-OST0004: haven't heard from client 6ccfe46c-719c-a0a8-6fb7-cbaf124cf613 (at 10.8.23.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98583c214400, cur 1555726371 expire 1555726221 last 1555726144 [Fri Apr 19 19:12:51 2019][6074433.764012] Lustre: Skipped 29 previous similar messages [Fri Apr 19 19:22:15 2019][6074997.860260] Lustre: 96261:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555726928/real 1555726928] req@ffff984c9b070c00 x1625552576008720/t0(0) o106->fir-OST0000@10.8.29.7@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1555726935 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Fri Apr 19 19:22:15 2019][6074997.887631] Lustre: 96261:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 111 previous similar messages [Fri Apr 19 19:22:57 2019][6075039.861858] Lustre: 96583:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555726970/real 1555726970] req@ffff985561cb1200 x1625552576008736/t0(0) o106->fir-OST0004@10.8.29.7@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1555726977 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Fri Apr 19 19:22:57 2019][6075039.889232] Lustre: 96583:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 43 previous similar messages [Fri Apr 19 19:23:00 2019][6075043.509541] Lustre: fir-OST0000: Connection restored to dba9c610-260e-8121-cf3c-59fccbb7189a (at 10.8.26.20@o2ib6) [Fri Apr 19 19:23:00 2019][6075043.520060] Lustre: Skipped 17 previous similar messages [Fri Apr 19 19:24:14 2019][6075116.864785] Lustre: 96928:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555727047/real 1555727047] req@ffff984fec62f800 x1625552576008688/t0(0) o106->fir-OST000a@10.8.29.7@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1555727054 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Fri Apr 19 19:24:14 2019][6075116.892170] Lustre: 96928:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 82 previous similar messages [Fri Apr 19 19:25:17 2019][6075179.770247] Lustre: fir-OST0004: haven't heard from client 6e808e83-ef58-b985-baab-1d8b8a2f4595 (at 10.8.29.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984bf6001400, cur 1555727117 expire 1555726967 last 1555726890 [Fri Apr 19 19:25:17 2019][6075179.792066] Lustre: Skipped 23 previous similar messages [Fri Apr 19 19:35:46 2019][6075808.794993] Lustre: fir-OST0006: haven't heard from client 79e06168-17ea-fdf9-6364-d3b5ba57ce2c (at 10.8.20.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983f32ad9c00, cur 1555727746 expire 1555727596 last 1555727519 [Fri Apr 19 19:35:46 2019][6075808.816811] Lustre: Skipped 11 previous similar messages [Fri Apr 19 19:39:15 2019][6076018.252123] Lustre: 96896:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555727948/real 1555727948] req@ffff98381dc11b00 x1625552630050720/t0(0) o106->fir-OST0004@10.8.29.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1555727955 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Fri Apr 19 19:39:15 2019][6076018.279544] Lustre: 96896:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 71 previous similar messages [Fri Apr 19 19:39:36 2019][6076039.252926] Lustre: 96899:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555727969/real 1555727969] req@ffff98386845ec00 x1625552630050752/t0(0) o106->fir-OST0008@10.8.29.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1555727976 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Fri Apr 19 19:39:36 2019][6076039.252928] Lustre: 96919:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555727969/real 1555727969] req@ffff98383d56c500 x1625552630050736/t0(0) o106->fir-OST0006@10.8.29.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1555727976 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Fri Apr 19 19:39:36 2019][6076039.252933] Lustre: 96919:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 15 previous similar messages [Fri Apr 19 19:39:36 2019][6076039.317726] Lustre: 96899:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message [Fri Apr 19 19:40:18 2019][6076081.255534] Lustre: 96919:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555728011/real 1555728011] req@ffff98383d56c500 x1625552630050736/t0(0) o106->fir-OST0006@10.8.29.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1555728018 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Fri Apr 19 19:40:18 2019][6076081.282942] Lustre: 96919:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 35 previous similar messages [Fri Apr 19 19:41:13 2019][6076136.659688] Lustre: fir-OST0000: Connection restored to d2fd779c-157e-52b0-c4f2-2f7aaa06aac8 (at 10.8.23.7@o2ib6) [Fri Apr 19 19:41:13 2019][6076136.670128] Lustre: Skipped 35 previous similar messages [Fri Apr 19 19:41:35 2019][6076158.296464] Lustre: 96896:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555728088/real 1555728088] req@ffff98381dc11b00 x1625552630050720/t0(0) o106->fir-OST0004@10.8.29.4@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1555728095 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Fri Apr 19 19:41:35 2019][6076158.323837] Lustre: 96896:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 57 previous similar messages [Fri Apr 19 19:52:47 2019][6076829.837410] Lustre: fir-OST0006: haven't heard from client f2cf7f2d-60e2-3488-6d60-8a977e65e61d (at 10.8.24.18@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98677d36f400, cur 1555728767 expire 1555728617 last 1555728540 [Fri Apr 19 19:52:47 2019][6076829.859310] Lustre: Skipped 29 previous similar messages [Fri Apr 19 19:54:47 2019][6076950.230883] Lustre: fir-OST0000: Connection restored to bd4172ee-b21a-171f-590d-07ab0a2614bb (at 10.8.12.25@o2ib6) [Fri Apr 19 19:54:47 2019][6076950.241420] Lustre: Skipped 29 previous similar messages [Fri Apr 19 20:04:50 2019][6077553.307167] Lustre: fir-OST0000: Connection restored to e265f84a-d19d-6fce-343c-d86c6eba2d5b (at 10.8.29.3@o2ib6) [Fri Apr 19 20:04:50 2019][6077553.317606] Lustre: Skipped 23 previous similar messages [Fri Apr 19 20:14:52 2019][6078154.884047] Lustre: fir-OST0002: haven't heard from client 459f3177-56aa-5c40-032f-a19e27a39841 (at 10.8.21.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785d7a000, cur 1555730092 expire 1555729942 last 1555729865 [Fri Apr 19 20:14:52 2019][6078154.906002] Lustre: Skipped 41 previous similar messages [Fri Apr 19 20:15:24 2019][6078187.483501] Lustre: fir-OST0000: Connection restored to e69a58ac-0a54-448a-34ab-47e71ec425db (at 10.8.21.21@o2ib6) [Fri Apr 19 20:15:24 2019][6078187.494060] Lustre: Skipped 23 previous similar messages [Fri Apr 19 20:20:38 2019][6078500.897007] Lustre: fir-OST0004: haven't heard from client e8380e67-138b-dee1-0890-cc93755bdc9a (at 10.8.23.12@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98499d64fc00, cur 1555730438 expire 1555730288 last 1555730211 [Fri Apr 19 20:20:38 2019][6078500.918910] Lustre: Skipped 5 previous similar messages [Fri Apr 19 20:26:36 2019][6078858.910104] Lustre: fir-OST0006: haven't heard from client 4ecfd70d-be85-c58a-0eab-204290306236 (at 10.8.13.7@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857820d0400, cur 1555730796 expire 1555730646 last 1555730569 [Fri Apr 19 20:26:36 2019][6078858.931952] Lustre: Skipped 5 previous similar messages [Fri Apr 19 20:26:56 2019][6078879.324421] Lustre: fir-OST0000: Connection restored to 23f9a8de-c6a0-76ea-9378-57e47966c02c (at 10.8.21.23@o2ib6) [Fri Apr 19 20:26:56 2019][6078879.334959] Lustre: Skipped 29 previous similar messages [Fri Apr 19 20:34:43 2019][6079345.935212] Lustre: fir-OST0004: haven't heard from client e458857b-15c5-7752-4466-6b97eda69ac1 (at 10.8.12.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983858615c00, cur 1555731283 expire 1555731133 last 1555731056 [Fri Apr 19 20:34:43 2019][6079345.957122] Lustre: Skipped 11 previous similar messages [Fri Apr 19 20:48:47 2019][6080190.566091] Lustre: fir-OST0000: Connection restored to de71e293-4bab-6a9f-f864-40636c6dd616 (at 10.8.23.12@o2ib6) [Fri Apr 19 20:48:47 2019][6080190.576611] Lustre: Skipped 16 previous similar messages [Fri Apr 19 20:49:44 2019][6080246.962143] Lustre: fir-OST0008: haven't heard from client a83cf22a-5ce8-9491-6c49-fb09144e3d47 (at 10.8.24.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9852b0f4dc00, cur 1555732184 expire 1555732034 last 1555731957 [Fri Apr 19 20:49:44 2019][6080246.984051] Lustre: Skipped 17 previous similar messages [Fri Apr 19 20:57:28 2019][6080711.785047] Lustre: fir-OST0000: Connection restored to e69a58ac-0a54-448a-34ab-47e71ec425db (at 10.8.21.21@o2ib6) [Fri Apr 19 20:57:28 2019][6080711.795579] Lustre: Skipped 6 previous similar messages [Fri Apr 19 21:00:00 2019][6080863.720392] Lustre: fir-OST0000: Connection restored to 3e279d24-e2fb-196e-d7ac-e1a73db143bd (at 10.9.112.16@o2ib4) [Fri Apr 19 21:00:00 2019][6080863.731033] Lustre: Skipped 11 previous similar messages [Fri Apr 19 21:00:23 2019][6080885.989408] Lustre: fir-OST0004: haven't heard from client 84fe3bc2-1419-4dcc-680d-7bd64bcbca1c (at 10.8.12.5@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984990a0c800, cur 1555732823 expire 1555732673 last 1555732596 [Fri Apr 19 21:00:23 2019][6080886.011229] Lustre: Skipped 53 previous similar messages [Fri Apr 19 21:05:54 2019][6081217.644494] Lustre: fir-OST0000: Connection restored to ee4f43b6-eeda-2bf1-5676-b3cc4b94f3db (at 10.8.12.24@o2ib6) [Fri Apr 19 21:05:54 2019][6081217.655020] Lustre: Skipped 5 previous similar messages [Fri Apr 19 21:14:20 2019][6081723.019483] Lustre: fir-OST0006: haven't heard from client c8ab0605-75c5-711e-f6bb-8c20681b865a (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9859720c8800, cur 1555733660 expire 1555733510 last 1555733433 [Fri Apr 19 21:14:20 2019][6081723.041364] Lustre: Skipped 23 previous similar messages [Fri Apr 19 21:17:45 2019][6081928.741880] Lustre: fir-OST0000: Connection restored to 1bf63035-2382-2247-57ec-f4958613068d (at 10.8.24.11@o2ib6) [Fri Apr 19 21:17:45 2019][6081928.752409] Lustre: Skipped 17 previous similar messages [Fri Apr 19 21:25:57 2019][6082420.044336] Lustre: fir-OST0006: haven't heard from client 57e8037b-670f-9dea-abd2-7c376f7ce772 (at 10.8.21.21@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98575a287800, cur 1555734357 expire 1555734207 last 1555734130 [Fri Apr 19 21:25:57 2019][6082420.066220] Lustre: Skipped 5 previous similar messages [Fri Apr 19 21:28:32 2019][6082575.241256] Lustre: fir-OST0000: Connection restored to d5be9cd1-c8c8-17b9-5190-2e4feedff1dc (at 10.8.23.20@o2ib6) [Fri Apr 19 21:28:32 2019][6082575.251820] Lustre: Skipped 47 previous similar messages [Fri Apr 19 21:45:54 2019][6083617.091062] Lustre: fir-OST0002: haven't heard from client b43714ae-7e8e-ef73-c130-5b85e75e0597 (at 10.8.13.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984834453000, cur 1555735554 expire 1555735404 last 1555735327 [Fri Apr 19 21:45:54 2019][6083617.112877] Lustre: Skipped 11 previous similar messages [Fri Apr 19 21:58:46 2019][6084389.120281] Lustre: fir-OST0002: haven't heard from client 8ce287e0-e24b-238e-10ae-f33a52597ac5 (at 10.8.21.17@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984b283df400, cur 1555736326 expire 1555736176 last 1555736099 [Fri Apr 19 21:58:46 2019][6084389.142164] Lustre: Skipped 5 previous similar messages [Fri Apr 19 22:06:04 2019][6084827.372906] Lustre: fir-OST0000: Connection restored to ddd1b310-42a4-1c93-bc8a-ea4ff10c3b50 (at 10.8.10.11@o2ib6) [Fri Apr 19 22:06:04 2019][6084827.383434] Lustre: Skipped 27 previous similar messages [Fri Apr 19 22:16:41 2019][6085464.819354] Lustre: fir-OST0000: Connection restored to 5ad95c58-a291-a852-e763-1230aae68b67 (at 10.8.13.6@o2ib6) [Fri Apr 19 22:16:41 2019][6085464.829790] Lustre: Skipped 5 previous similar messages [Fri Apr 19 22:25:15 2019][6085978.179780] Lustre: fir-OST0008: haven't heard from client 517209f9-b8b4-8857-8aea-a0fef7fa7201 (at 10.8.24.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98381d6ba400, cur 1555737915 expire 1555737765 last 1555737688 [Fri Apr 19 22:25:15 2019][6085978.201609] Lustre: Skipped 5 previous similar messages [Fri Apr 19 22:26:28 2019][6086051.253239] Lustre: fir-OST0000: Connection restored to 25fddf26-1140-8712-6eb9-db80a7fa48a5 (at 10.8.21.17@o2ib6) [Fri Apr 19 22:26:28 2019][6086051.263765] Lustre: Skipped 5 previous similar messages [Fri Apr 19 22:26:31 2019][6086054.186350] Lustre: fir-OST0002: haven't heard from client d41ef3ba-8148-01b1-a29c-f5b1eccaae57 (at 10.8.11.14@o2ib6) in 185 seconds. I think it's dead, and I am evicting it. exp ffff98651fcdc000, cur 1555737991 expire 1555737841 last 1555737806 [Fri Apr 19 22:26:31 2019][6086054.208254] Lustre: Skipped 5 previous similar messages [Fri Apr 19 22:30:56 2019][6086319.731506] LustreError: 96246:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.27.23@o2ib6) returned error from blocking AST (req@ffff9864ebbece00 x1625553183138608 status -107 rc -107), evict it ns: filter-fir-OST0002_UUID lock: ffff9838578be780/0x49e1863bd2b03b58 lrc: 4/0,0 mode: PR/PR res: [0x29dae6:0x0:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 8388608->536870911) flags: 0x60000400010020 nid: 10.8.27.23@o2ib6 remote: 0x4c35e67c1aad6c03 expref: 189 pid: 96265 timeout: 6086235 lvb_type: 1 [Fri Apr 19 22:30:56 2019][6086319.777865] LustreError: 138-a: fir-OST0002: A client on nid 10.8.27.23@o2ib6 was evicted due to a lock blocking callback time out: rc -107 [Fri Apr 19 22:30:56 2019][6086319.790602] LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 10.8.27.23@o2ib6 ns: filter-fir-OST0002_UUID lock: ffff9838578be780/0x49e1863bd2b03b58 lrc: 3/0,0 mode: PR/PR res: [0x29dae6:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8388608->536870911) flags: 0x60000400010020 nid: 10.8.27.23@o2ib6 remote: 0x4c35e67c1aad6c03 expref: 190 pid: 96265 timeout: 0 lvb_type: 1 [Fri Apr 19 22:31:32 2019][6086355.194245] Lustre: fir-OST000a: haven't heard from client 5d513311-9f48-87da-7dd8-40b4c88ea9be (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986785e1b400, cur 1555738292 expire 1555738142 last 1555738065 [Fri Apr 19 22:31:32 2019][6086355.216129] Lustre: Skipped 5 previous similar messages [Fri Apr 19 22:31:33 2019][6086356.728705] Lustre: 96752:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555738286/real 1555738286] req@ffff9864ebbec500 x1625553184569152/t0(0) o104->fir-OST0000@10.8.27.23@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555738293 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Fri Apr 19 22:31:33 2019][6086356.756158] Lustre: 96752:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 27 previous similar messages [Fri Apr 19 22:34:39 2019][6086543.081132] Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) [Fri Apr 19 22:34:39 2019][6086543.091655] Lustre: Skipped 3 previous similar messages [Fri Apr 19 22:47:51 2019][6087334.399829] LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Timed out tx: active_txs, 0 seconds [Fri Apr 19 22:47:51 2019][6087334.410182] LNetError: 91376:0:(o2iblnd_cb.c:3324:kiblnd_check_txs_locked()) Skipped 1 previous similar message [Fri Apr 19 22:47:51 2019][6087334.420442] LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Timed out RDMA with 10.0.10.3@o2ib7 (107): c: 6, oc: 0, rc: 8 [Fri Apr 19 22:47:51 2019][6087334.432604] LNetError: 91376:0:(o2iblnd_cb.c:3399:kiblnd_check_conns()) Skipped 1 previous similar message [Fri Apr 19 22:49:47 2019][6087450.236219] Lustre: fir-OST0002: haven't heard from client 261ee164-624b-aba3-f37d-69ea5bb1f14f (at 10.9.112.17@o2ib4) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986a896e1400, cur 1555739387 expire 1555739237 last 1555739160 [Fri Apr 19 22:49:47 2019][6087450.258188] Lustre: Skipped 4 previous similar messages [Fri Apr 19 22:51:03 2019][6087526.240239] Lustre: fir-OST0002: haven't heard from client 92df64b7-2bb2-b8b2-042a-8cfb3d05ce87 (at 10.8.24.12@o2ib6) in 179 seconds. I think it's dead, and I am evicting it. exp ffff98683bb58000, cur 1555739463 expire 1555739313 last 1555739284 [Fri Apr 19 22:51:03 2019][6087526.262127] Lustre: Skipped 17 previous similar messages [Fri Apr 19 22:52:19 2019][6087602.242246] Lustre: fir-OST0002: haven't heard from client e1d6b5bb-5593-a024-66f9-9b59d64e862b (at 10.8.23.23@o2ib6) in 208 seconds. I think it's dead, and I am evicting it. exp ffff9877a15a3c00, cur 1555739539 expire 1555739389 last 1555739331 [Fri Apr 19 22:52:19 2019][6087602.264198] Lustre: Skipped 11 previous similar messages [Fri Apr 19 22:52:24 2019][6087607.270190] Lustre: 96302:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555739537/real 1555739537] req@ffff9872d92ead00 x1625553250419104/t0(0) o106->fir-OST0008@10.8.23.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1555739544 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Fri Apr 19 22:52:31 2019][6087614.297452] Lustre: 96302:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555739544/real 1555739544] req@ffff9872d92ead00 x1625553250419104/t0(0) o106->fir-OST0008@10.8.23.23@o2ib6:15/16 lens 296/280 e 0 to 1 dl 1555739551 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Fri Apr 19 22:53:14 2019][6087657.371560] Lustre: fir-OST0000: Connection restored to 110f398c-860f-9c94-fae8-88dd42a6445a (at 10.8.24.4@o2ib6) [Fri Apr 19 22:53:14 2019][6087657.381999] Lustre: Skipped 5 previous similar messages [Fri Apr 19 22:56:29 2019][6087852.253581] Lustre: fir-OST0006: haven't heard from client 6b7d18c3-9d7f-f1ff-ee41-d64828865ce4 (at 10.8.30.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98667546cc00, cur 1555739789 expire 1555739639 last 1555739562 [Fri Apr 19 22:56:29 2019][6087852.275377] Lustre: Skipped 11 previous similar messages [Fri Apr 19 23:02:51 2019][6088234.264863] Lustre: fir-OST0006: haven't heard from client d52a0b4e-0c33-9946-39fb-6e0aeab83c80 (at 10.8.20.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984d22d7f400, cur 1555740171 expire 1555740021 last 1555739944 [Fri Apr 19 23:02:51 2019][6088234.286777] Lustre: Skipped 35 previous similar messages [Fri Apr 19 23:12:57 2019][6088840.839776] Lustre: fir-OST0000: Connection restored to (at 10.9.112.17@o2ib4) [Fri Apr 19 23:12:57 2019][6088840.847293] Lustre: Skipped 22 previous similar messages [Fri Apr 19 23:13:30 2019][6088873.849927] Lustre: 96936:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555740803/real 1555740803] req@ffff985e0af40300 x1625553317022816/t0(0) o104->fir-OST0004@10.8.26.19@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555740810 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Fri Apr 19 23:13:37 2019][6088880.877184] Lustre: 96936:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555740810/real 1555740810] req@ffff985e0af40300 x1625553317022816/t0(0) o104->fir-OST0004@10.8.26.19@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555740817 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Fri Apr 19 23:13:44 2019][6088887.904443] Lustre: 96936:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555740817/real 1555740817] req@ffff985e0af40300 x1625553317022816/t0(0) o104->fir-OST0004@10.8.26.19@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555740824 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Fri Apr 19 23:13:51 2019][6088894.931704] Lustre: 96936:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555740824/real 1555740824] req@ffff985e0af40300 x1625553317022816/t0(0) o104->fir-OST0004@10.8.26.19@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555740831 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Fri Apr 19 23:13:58 2019][6088901.958965] Lustre: 96936:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555740831/real 1555740831] req@ffff985e0af40300 x1625553317022816/t0(0) o104->fir-OST0004@10.8.26.19@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555740838 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Fri Apr 19 23:14:12 2019][6088915.986485] Lustre: 96936:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555740845/real 1555740845] req@ffff985e0af40300 x1625553317022816/t0(0) o104->fir-OST0004@10.8.26.19@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555740852 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Fri Apr 19 23:14:12 2019][6088916.013921] Lustre: 96936:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 1 previous similar message [Fri Apr 19 23:14:33 2019][6088937.024264] Lustre: 96936:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555740866/real 1555740866] req@ffff985e0af40300 x1625553317022816/t0(0) o104->fir-OST0004@10.8.26.19@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555740873 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Fri Apr 19 23:14:33 2019][6088937.051729] Lustre: 96936:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [Fri Apr 19 23:15:08 2019][6088972.062586] LustreError: 96936:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.26.19@o2ib6) failed to reply to blocking AST (req@ffff985e0af40300 x1625553317022816 status 0 rc -110), evict it ns: filter-fir-OST0004_UUID lock: ffff9860337d2ac0/0x49e1863bd67b7faa lrc: 4/0,0 mode: PR/PR res: [0x8c0000402:0x51bd31:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8388608->536870911) flags: 0x60000400010020 nid: 10.8.26.19@o2ib6 remote: 0x84c362ea50ed4e01 expref: 7 pid: 96272 timeout: 6088830 lvb_type: 1 [Fri Apr 19 23:15:08 2019][6088972.109138] LustreError: 138-a: fir-OST0004: A client on nid 10.8.26.19@o2ib6 was evicted due to a lock blocking callback time out: rc -110 [Fri Apr 19 23:15:08 2019][6088972.121875] LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 105s: evicting client at 10.8.26.19@o2ib6 ns: filter-fir-OST0004_UUID lock: ffff9860337d2ac0/0x49e1863bd67b7faa lrc: 3/0,0 mode: PR/PR res: [0x8c0000402:0x51bd31:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8388608->536870911) flags: 0x60000400010020 nid: 10.8.26.19@o2ib6 remote: 0x84c362ea50ed4e01 expref: 8 pid: 96272 timeout: 0 lvb_type: 1 [Fri Apr 19 23:15:41 2019][6089004.293770] Lustre: fir-OST0000: haven't heard from client 9b377500-33b2-0196-8ee5-8d318e383f88 (at 10.8.13.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9857844e6800, cur 1555740941 expire 1555740791 last 1555740714 [Fri Apr 19 23:15:41 2019][6089004.315677] Lustre: Skipped 17 previous similar messages [Fri Apr 19 23:24:33 2019][6089536.681143] Lustre: fir-OST0000: Connection restored to 01bf31bc-51a2-4b95-74f5-d8893ea0150c (at 10.8.30.4@o2ib6) [Fri Apr 19 23:24:33 2019][6089536.691587] Lustre: Skipped 29 previous similar messages [Fri Apr 19 23:35:38 2019][6090201.339985] Lustre: fir-OST0002: haven't heard from client f9709d24-98d8-0cac-f2ff-a99b8ba62700 (at 10.8.24.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867811fb000, cur 1555742138 expire 1555741988 last 1555741911 [Fri Apr 19 23:35:38 2019][6090201.361920] Lustre: Skipped 142 previous similar messages [Fri Apr 19 23:36:54 2019][6090277.576390] Lustre: fir-OST0000: Connection restored to f6059855-8157-992b-c39d-d4839583d841 (at 10.8.20.3@o2ib6) [Fri Apr 19 23:36:54 2019][6090277.586824] Lustre: Skipped 47 previous similar messages [Fri Apr 19 23:47:53 2019][6090936.568290] Lustre: fir-OST0000: Connection restored to 86944b63-e282-0425-5312-520ee6361734 (at 10.8.22.10@o2ib6) [Fri Apr 19 23:47:53 2019][6090936.578809] Lustre: Skipped 23 previous similar messages [Fri Apr 19 23:54:58 2019][6091361.380696] Lustre: fir-OST0008: haven't heard from client 0ce7d9f5-f365-1847-fc8d-359a87f4e48f (at 10.8.23.27@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786a30400, cur 1555743298 expire 1555743148 last 1555743071 [Fri Apr 19 23:54:58 2019][6091361.402600] Lustre: Skipped 23 previous similar messages [Sat Apr 20 00:03:57 2019][6091900.415421] Lustre: fir-OST0000: Connection restored to 5291d87c-c332-89fe-37a2-5aad94038a93 (at 10.8.24.16@o2ib6) [Sat Apr 20 00:03:57 2019][6091900.425939] Lustre: Skipped 131 previous similar messages [Sat Apr 20 00:13:15 2019][6092458.421584] Lustre: fir-OST0000: haven't heard from client 03821268-326d-0652-c6af-5c4c6d8580ca (at 10.8.23.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480ee8c400, cur 1555744395 expire 1555744245 last 1555744168 [Sat Apr 20 00:13:15 2019][6092458.443486] Lustre: Skipped 71 previous similar messages [Sat Apr 20 00:23:16 2019][6093059.632806] Lustre: fir-OST0000: Connection restored to e256f342-dd4a-0733-a0fa-e78c997bbd1d (at 10.8.23.27@o2ib6) [Sat Apr 20 00:23:16 2019][6093059.643481] Lustre: Skipped 17 previous similar messages [Sat Apr 20 00:23:29 2019][6093072.442994] Lustre: fir-OST0000: haven't heard from client 4991eca4-e3cf-94dc-efe6-7b7ccbc5f5d3 (at 10.8.10.15@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98481f7ef800, cur 1555745009 expire 1555744859 last 1555744782 [Sat Apr 20 00:23:29 2019][6093072.464878] Lustre: Skipped 29 previous similar messages [Sat Apr 20 00:34:09 2019][6093712.465850] Lustre: fir-OST000a: haven't heard from client 265fca7c-4427-e6ef-651e-7d88a0f94061 (at 10.8.24.32@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9851fa7e8400, cur 1555745649 expire 1555745499 last 1555745422 [Sat Apr 20 00:34:09 2019][6093712.487795] Lustre: Skipped 17 previous similar messages [Sat Apr 20 00:41:03 2019][6094126.681607] Lustre: fir-OST0002: Connection restored to 6f23ee85-f1bd-65e1-fb10-c877f871546c (at 10.8.23.16@o2ib6) [Sat Apr 20 00:41:03 2019][6094126.692132] Lustre: Skipped 75 previous similar messages [Sat Apr 20 00:48:54 2019][6094597.499537] Lustre: fir-OST0008: haven't heard from client 73a26743-df3f-8473-2002-0e98f2ae5d11 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576f3f1800, cur 1555746534 expire 1555746384 last 1555746307 [Sat Apr 20 00:48:54 2019][6094597.521443] Lustre: Skipped 59 previous similar messages [Sat Apr 20 00:55:07 2019][6094970.810529] Lustre: fir-OST0000: Connection restored to 2354d7af-653b-aaa3-2c33-b296ff69d0d2 (at 10.8.10.15@o2ib6) [Sat Apr 20 00:55:07 2019][6094970.821054] Lustre: Skipped 36 previous similar messages [Sat Apr 20 00:58:57 2019][6095200.527825] Lustre: fir-OST0002: haven't heard from client 28ff7641-9fbf-1a1d-5a0d-ef36edc620a5 (at 10.8.10.13@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98663589c800, cur 1555747137 expire 1555746987 last 1555746910 [Sat Apr 20 00:58:57 2019][6095200.549714] Lustre: Skipped 127 previous similar messages [Sat Apr 20 01:06:31 2019][6095655.225254] Lustre: fir-OST0000: Connection restored to c3345055-45dc-9ec0-dd3d-94a5d108b2f2 (at 10.8.26.30@o2ib6) [Sat Apr 20 01:06:31 2019][6095655.235817] Lustre: Skipped 35 previous similar messages [Sat Apr 20 01:12:15 2019][6095998.550341] Lustre: fir-OST0004: haven't heard from client e1333e54-0302-60fe-187f-d6adbfcbd5a6 (at 10.8.11.36@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985815680000, cur 1555747935 expire 1555747785 last 1555747708 [Sat Apr 20 01:12:15 2019][6095998.572223] Lustre: Skipped 3 previous similar messages [Sat Apr 20 01:21:04 2019][6096528.132619] Lustre: fir-OST0000: Connection restored to 7a5d42a3-072b-ada7-8bd9-6b223c35b055 (at 10.8.20.9@o2ib6) [Sat Apr 20 01:21:04 2019][6096528.143061] Lustre: Skipped 35 previous similar messages [Sat Apr 20 01:24:33 2019][6096736.580146] Lustre: fir-OST0006: haven't heard from client 294da5e6-40f1-5404-91b7-88b7c5f91121 (at 10.8.25.25@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98481a6c6c00, cur 1555748673 expire 1555748523 last 1555748446 [Sat Apr 20 01:24:33 2019][6096736.602032] Lustre: Skipped 35 previous similar messages [Sat Apr 20 01:32:20 2019][6097203.606906] Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) [Sat Apr 20 01:32:20 2019][6097203.617429] Lustre: Skipped 130 previous similar messages [Sat Apr 20 01:42:38 2019][6097822.450615] Lustre: fir-OST0000: Connection restored to 179e0b48-b58d-d9b1-7e3f-f996ca06f525 (at 10.8.10.6@o2ib6) [Sat Apr 20 01:42:38 2019][6097822.461077] Lustre: Skipped 11 previous similar messages [Sat Apr 20 01:50:21 2019][6098284.635453] Lustre: fir-OST0002: haven't heard from client b2c50030-59f8-ea05-757d-91e4d29c95da (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff987464256400, cur 1555750221 expire 1555750071 last 1555749994 [Sat Apr 20 01:50:21 2019][6098284.657336] Lustre: Skipped 35 previous similar messages [Sat Apr 20 01:52:45 2019][6098429.485302] Lustre: fir-OST0000: Connection restored to 96dc3f28-35a7-d1a0-d554-ac4259066293 (at 10.8.25.15@o2ib6) [Sat Apr 20 01:52:45 2019][6098429.495830] Lustre: Skipped 29 previous similar messages [Sat Apr 20 01:54:46 2019][6098549.646332] Lustre: fir-OST0000: haven't heard from client 769080f3-7851-354b-38b0-f532cef68479 (at 10.8.25.18@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867f49cd400, cur 1555750486 expire 1555750336 last 1555750259 [Sat Apr 20 01:54:46 2019][6098549.668214] Lustre: Skipped 5 previous similar messages [Sat Apr 20 02:09:26 2019][6099429.679545] Lustre: fir-OST000a: haven't heard from client d4c31c5a-1f80-4ec4-f306-bf8a7a12dfe0 (at 10.8.23.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98576f351400, cur 1555751366 expire 1555751216 last 1555751139 [Sat Apr 20 02:09:26 2019][6099429.701362] Lustre: Skipped 17 previous similar messages [Sat Apr 20 02:18:42 2019][6099985.698627] Lustre: fir-OST0006: haven't heard from client 180d69dd-ef9b-1134-9fe6-1e5c20fe7a82 (at 10.8.22.33@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9867f8209000, cur 1555751922 expire 1555751772 last 1555751695 [Sat Apr 20 02:18:42 2019][6099985.720516] Lustre: Skipped 17 previous similar messages [Sat Apr 20 02:22:22 2019][6100205.708812] Lustre: fir-OST0004: haven't heard from client 1025e8d0-6b9b-6bcb-8e84-00c0a0391a67 (at 10.8.30.16@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98384b242800, cur 1555752142 expire 1555751992 last 1555751915 [Sat Apr 20 02:22:22 2019][6100205.730689] Lustre: Skipped 11 previous similar messages [Sat Apr 20 02:22:40 2019][6100224.366831] Lustre: fir-OST0000: Connection restored to 8998da7f-5cfc-61e2-ded0-d86c6c9d5c32 (at 10.8.25.18@o2ib6) [Sat Apr 20 02:22:40 2019][6100224.377364] Lustre: Skipped 29 previous similar messages [Sat Apr 20 02:26:00 2019][6100424.250358] Lustre: fir-OST0000: Connection restored to c4f458ee-079e-1f6b-715d-4cc60d32c4b8 (at 10.8.11.4@o2ib6) [Sat Apr 20 02:26:00 2019][6100424.260792] Lustre: Skipped 5 previous similar messages [Sat Apr 20 02:37:14 2019][6101098.415495] Lustre: fir-OST0000: Connection restored to da5f786b-e264-8f88-0ed4-a578a7fe601a (at 10.8.23.4@o2ib6) [Sat Apr 20 02:37:14 2019][6101098.425937] Lustre: Skipped 11 previous similar messages [Sat Apr 20 02:37:49 2019][6101132.741923] Lustre: fir-OST0008: haven't heard from client 77547ca9-61ad-9494-778f-71a890774056 (at 10.8.23.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984800e49400, cur 1555753069 expire 1555752919 last 1555752842 [Sat Apr 20 02:37:49 2019][6101132.763834] Lustre: Skipped 5 previous similar messages [Sat Apr 20 02:38:09 2019][6101152.742597] Lustre: fir-OST0006: haven't heard from client 77547ca9-61ad-9494-778f-71a890774056 (at 10.8.23.11@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985838048800, cur 1555753089 expire 1555752939 last 1555752862 [Sat Apr 20 02:38:09 2019][6101152.764506] Lustre: Skipped 1 previous similar message [Sat Apr 20 02:39:05 2019][6101208.751541] Lustre: fir-OST000a: haven't heard from client 4d72186f-8acf-1060-26f0-39cc64b527e6 (at 10.8.23.10@o2ib6) in 187 seconds. I think it's dead, and I am evicting it. exp ffff9862a43fe400, cur 1555753145 expire 1555752995 last 1555752958 [Sat Apr 20 02:39:05 2019][6101208.773416] Lustre: Skipped 3 previous similar messages [Sat Apr 20 02:39:45 2019][6101248.746524] Lustre: fir-OST0008: haven't heard from client 4d72186f-8acf-1060-26f0-39cc64b527e6 (at 10.8.23.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984b5d683800, cur 1555753185 expire 1555753035 last 1555752958 [Sat Apr 20 02:39:45 2019][6101248.768428] Lustre: Skipped 4 previous similar messages [Sat Apr 20 02:41:01 2019][6101324.749258] Lustre: fir-OST0008: haven't heard from client 7c0d3c3e-2e2e-e323-4e8e-8e4e22e2123b (at 10.8.27.23@o2ib6) in 176 seconds. I think it's dead, and I am evicting it. exp ffff984801bfa400, cur 1555753261 expire 1555753111 last 1555753085 [Sat Apr 20 02:42:19 2019][6101402.768344] Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) [Sat Apr 20 02:42:19 2019][6101402.778872] Lustre: Skipped 17 previous similar messages [Sat Apr 20 02:51:23 2019][6101946.772688] Lustre: fir-OST0000: haven't heard from client c53fa2b0-ce23-6613-a54b-c16628309f4c (at 10.8.30.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98546cff6400, cur 1555753883 expire 1555753733 last 1555753656 [Sat Apr 20 02:51:23 2019][6101946.794576] Lustre: Skipped 5 previous similar messages [Sat Apr 20 02:56:29 2019][6102252.785267] Lustre: fir-OST000a: haven't heard from client c3e7914b-0c88-6741-bbc5-e92895266a81 (at 10.8.12.34@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98480c6db800, cur 1555754189 expire 1555754039 last 1555753962 [Sat Apr 20 02:56:29 2019][6102252.807192] Lustre: Skipped 11 previous similar messages [Sat Apr 20 02:59:48 2019][6102452.644277] Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) [Sat Apr 20 02:59:48 2019][6102452.654801] Lustre: Skipped 23 previous similar messages [Sat Apr 20 03:10:21 2019][6103085.178223] Lustre: fir-OST0000: Connection restored to a908dcd0-9db1-9f70-5f22-f8c81c7a1077 (at 10.8.23.10@o2ib6) [Sat Apr 20 03:10:21 2019][6103085.188751] Lustre: Skipped 11 previous similar messages [Sat Apr 20 03:19:45 2019][6103648.855178] Lustre: fir-OST0006: haven't heard from client 956291af-e3b4-a7f2-9bf3-8d6fdab27a4b (at 10.8.26.10@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9854318ed400, cur 1555755585 expire 1555755435 last 1555755358 [Sat Apr 20 03:19:45 2019][6103648.877090] Lustre: Skipped 47 previous similar messages [Sat Apr 20 03:24:30 2019][6103934.679372] Lustre: fir-OST0000: Connection restored to 801c5583-df50-ef54-ebf8-d76e7be7922a (at 10.8.21.18@o2ib6) [Sat Apr 20 03:24:30 2019][6103934.689896] Lustre: Skipped 11 previous similar messages [Sat Apr 20 03:30:40 2019][6104303.860642] Lustre: fir-OST0002: haven't heard from client cf6c504b-8f9f-8ba6-9e0e-926af7aae5da (at 10.8.19.8@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984d7dae8400, cur 1555756240 expire 1555756090 last 1555756013 [Sat Apr 20 03:30:40 2019][6104303.882462] Lustre: Skipped 5 previous similar messages [Sat Apr 20 03:34:19 2019][6104522.870103] Lustre: fir-OST0004: haven't heard from client 26e99557-f312-50eb-b233-bf6defc79319 (at 10.8.11.30@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff983801f1ac00, cur 1555756459 expire 1555756309 last 1555756232 [Sat Apr 20 03:34:19 2019][6104522.892012] Lustre: Skipped 5 previous similar messages [Sat Apr 20 03:40:43 2019][6104907.699023] Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) [Sat Apr 20 03:40:43 2019][6104907.709554] Lustre: Skipped 47 previous similar messages [Sat Apr 20 03:46:12 2019][6105235.896096] Lustre: fir-OST0002: haven't heard from client eed8230d-040b-cbbf-4894-4a729844f5ca (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986d0a84c400, cur 1555757172 expire 1555757022 last 1555756945 [Sat Apr 20 03:46:12 2019][6105235.917984] Lustre: Skipped 11 previous similar messages [Sat Apr 20 03:51:16 2019][6105540.220013] Lustre: fir-OST0002: Connection restored to 926fa24d-f3ab-7ad6-dbc7-f8a15bdf8c5a (at 10.8.19.8@o2ib6) [Sat Apr 20 03:51:16 2019][6105540.230481] Lustre: Skipped 17 previous similar messages [Sat Apr 20 04:02:58 2019][6106241.932275] Lustre: fir-OST0008: haven't heard from client 193ba445-ef1d-8bc3-faf8-504298710007 (at 10.8.27.23@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9863c0fd9800, cur 1555758178 expire 1555758028 last 1555757951 [Sat Apr 20 04:02:58 2019][6106241.954186] Lustre: Skipped 29 previous similar messages [Sat Apr 20 04:03:31 2019][6106275.297063] Lustre: fir-OST0000: Connection restored to 3d648bf8-1d55-290d-eec0-122a40706ad8 (at 10.8.27.23@o2ib6) [Sat Apr 20 04:03:31 2019][6106275.307590] Lustre: Skipped 5 previous similar messages [Sat Apr 20 04:14:44 2019][6106947.966233] Lustre: 96936:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555758877/real 1555758877] req@ffff9867fbb7a400 x1625554271914112/t0(0) o104->fir-OST0004@10.8.23.28@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555758884 ref 1 fl Rpc:X/0/ffffffff rc 0/-1 [Sat Apr 20 04:14:44 2019][6106947.993680] Lustre: 96936:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 6 previous similar messages [Sat Apr 20 04:14:51 2019][6106954.966495] Lustre: 63944:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555758884/real 1555758884] req@ffff985d3ad32100 x1625554271914032/t0(0) o104->fir-OST0000@10.8.23.28@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555758891 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Sat Apr 20 04:15:05 2019][6106968.995015] Lustre: 63944:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555758898/real 1555758898] req@ffff985d3ad32100 x1625554271914032/t0(0) o104->fir-OST0000@10.8.23.28@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555758905 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Sat Apr 20 04:15:05 2019][6106969.022488] Lustre: 63944:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [Sat Apr 20 04:15:26 2019][6106989.960679] Lustre: fir-OST000a: haven't heard from client 2e278f9e-5185-ec02-f40d-75886f396fbf (at 10.8.14.6@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986804b9bc00, cur 1555758926 expire 1555758776 last 1555758699 [Sat Apr 20 04:15:26 2019][6106989.982493] Lustre: Skipped 5 previous similar messages [Sat Apr 20 04:15:26 2019][6106990.004794] Lustre: 96936:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555758919/real 1555758919] req@ffff9867fbb7a400 x1625554271914112/t0(0) o104->fir-OST0004@10.8.23.28@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555758926 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Sat Apr 20 04:15:26 2019][6106990.032254] Lustre: 96936:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 4 previous similar messages [Sat Apr 20 04:16:08 2019][6107032.034362] Lustre: 63944:0:(client.c:2132:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1555758961/real 1555758961] req@ffff985d3ad32100 x1625554271914032/t0(0) o104->fir-OST0000@10.8.23.28@o2ib6:15/16 lens 296/224 e 0 to 1 dl 1555758968 ref 1 fl Rpc:X/2/ffffffff rc 0/-1 [Sat Apr 20 04:16:08 2019][6107032.061826] Lustre: 63944:0:(client.c:2132:ptlrpc_expire_one_request()) Skipped 11 previous similar messages [Sat Apr 20 04:16:22 2019][6107046.043891] LustreError: 96936:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.23.28@o2ib6) failed to reply to blocking AST (req@ffff9867fbb7a400 x1625554271914112 status 0 rc -110), evict it ns: filter-fir-OST0004_UUID lock: ffff985dd758ba80/0x49e1863bf2b95806 lrc: 4/0,0 mode: PR/PR res: [0x8c0000402:0x43525d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x60000400010020 nid: 10.8.23.28@o2ib6 remote: 0x1223ebb78c559f26 expref: 8 pid: 96936 timeout: 6106903 lvb_type: 1 [Sat Apr 20 04:16:22 2019][6107046.090695] LustreError: 138-a: fir-OST0004: A client on nid 10.8.23.28@o2ib6 was evicted due to a lock blocking callback time out: rc -110 [Sat Apr 20 04:16:22 2019][6107046.103422] LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 105s: evicting client at 10.8.23.28@o2ib6 ns: filter-fir-OST0004_UUID lock: ffff985dd758ba80/0x49e1863bf2b95806 lrc: 3/0,0 mode: PR/PR res: [0x8c0000402:0x43525d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x60000400010020 nid: 10.8.23.28@o2ib6 remote: 0x1223ebb78c559f26 expref: 9 pid: 96936 timeout: 0 lvb_type: 1 [Sat Apr 20 04:17:08 2019][6107092.010295] Lustre: fir-OST0000: Connection restored to 5c93f7b4-bdb4-332b-9d7b-c8f1dda3f8c6 (at 10.8.12.11@o2ib6) [Sat Apr 20 04:17:08 2019][6107092.020821] Lustre: Skipped 11 previous similar messages [Sat Apr 20 04:17:11 2019][6107095.074708] LustreError: 63944:0:(ldlm_lockd.c:682:ldlm_handle_ast_error()) ### client (nid 10.8.23.28@o2ib6) failed to reply to blocking AST (req@ffff985d3ad32100 x1625554271914032 status 0 rc -110), evict it ns: filter-fir-OST0000_UUID lock: ffff9850e2a7a880/0x49e1863bf2b959fe lrc: 4/0,0 mode: PR/PR res: [0x6c0000400:0x4362e9:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x60000400010020 nid: 10.8.23.28@o2ib6 remote: 0x1223ebb78c559fce expref: 6 pid: 110656 timeout: 6107002 lvb_type: 1 [Sat Apr 20 04:17:11 2019][6107095.122090] LustreError: 138-a: fir-OST0000: A client on nid 10.8.23.28@o2ib6 was evicted due to a lock blocking callback time out: rc -110 [Sat Apr 20 04:17:11 2019][6107095.153392] LustreError: 94224:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 154s: evicting client at 10.8.23.28@o2ib6 ns: filter-fir-OST0000_UUID lock: ffff9850e2a7a880/0x49e1863bf2b959fe lrc: 3/0,0 mode: PR/PR res: [0x6c0000400:0x4362e9:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 268435456->8589934591) flags: 0x60000400010020 nid: 10.8.23.28@o2ib6 remote: 0x1223ebb78c559fce expref: 7 pid: 110656 timeout: 0 lvb_type: 1 [Sat Apr 20 04:34:32 2019][6108136.004060] Lustre: fir-OST0002: haven't heard from client ea5cb1f7-3600-fc70-71ce-140151be7d71 (at 10.8.30.24@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98663589e400, cur 1555760072 expire 1555759922 last 1555759845 [Sat Apr 20 04:34:32 2019][6108136.025968] Lustre: Skipped 9 previous similar messages [Sat Apr 20 04:41:59 2019][6108583.296032] Lustre: fir-OST0000: Connection restored to 3dbbb061-d93c-c7e8-88c8-e262ff513397 (at 10.8.14.6@o2ib6) [Sat Apr 20 04:41:59 2019][6108583.306477] Lustre: Skipped 23 previous similar messages [Sat Apr 20 04:46:41 2019][6108865.933944] Lustre: fir-OST0000: Connection restored to 92f34919-614d-c857-b0e6-2bbe68fc85f2 (at 10.8.23.28@o2ib6) [Sat Apr 20 04:46:41 2019][6108865.944460] Lustre: Skipped 5 previous similar messages [Sat Apr 20 04:50:38 2019][6109102.044656] Lustre: fir-OST0002: haven't heard from client 20d48284-92b3-15ed-2797-240aab9e01ce (at 10.8.23.31@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff985838d89400, cur 1555761038 expire 1555760888 last 1555760811 [Sat Apr 20 04:50:38 2019][6109102.066579] Lustre: Skipped 5 previous similar messages [Sat Apr 20 05:03:48 2019][6109892.225205] Lustre: fir-OST0000: Connection restored to 009d37bf-e032-6ee3-aaab-db2215e82532 (at 10.8.30.24@o2ib6) [Sat Apr 20 05:03:48 2019][6109892.235728] Lustre: Skipped 5 previous similar messages [Sat Apr 20 05:04:08 2019][6109912.070481] Lustre: fir-OST000a: haven't heard from client 6d89aa7c-d176-1d38-3440-ab43c9e279b6 (at 10.8.23.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984de97ca400, cur 1555761848 expire 1555761698 last 1555761621 [Sat Apr 20 05:04:08 2019][6109912.092386] Lustre: Skipped 5 previous similar messages [Sat Apr 20 05:16:17 2019][6110641.327547] Lustre: fir-OST0004: Client fb2c209a-a3d8-c7bf-5b1f-4aa6b81917ea (at 10.9.108.2@o2ib4) reconnecting [Sat Apr 20 05:16:17 2019][6110641.337812] Lustre: Skipped 12 previous similar messages [Sat Apr 20 05:16:17 2019][6110641.343336] Lustre: fir-OST0004: Connection restored to 20609818-b83c-bf65-0dd2-090d3c6e2314 (at 10.9.108.2@o2ib4) [Sat Apr 20 05:16:18 2019][6110642.297971] LustreError: 96630:0:(ldlm_lib.c:3258:target_bulk_io()) @@@ Reconnect on bulk WRITE req@ffff98746d96e450 x1628586517874208/t0(0) o4->fb2c209a-a3d8-c7bf-5b1f-4aa6b81917ea@10.9.108.2@o2ib4:559/0 lens 488/448 e 0 to 0 dl 1555762619 ref 1 fl Interpret:/0/0 rc 0/0 [Sat Apr 20 05:16:18 2019][6110642.322223] LustreError: 96630:0:(ldlm_lib.c:3258:target_bulk_io()) Skipped 33 previous similar messages [Sat Apr 20 05:16:18 2019][6110642.331933] Lustre: fir-OST0004: Bulk IO write error with fb2c209a-a3d8-c7bf-5b1f-4aa6b81917ea (at 10.9.108.2@o2ib4), client will retry: rc = -110 [Sat Apr 20 05:16:18 2019][6110642.345256] Lustre: Skipped 9 previous similar messages [Sat Apr 20 05:16:51 2019][6110675.802144] LustreError: 137-5: fir-OST0009_UUID: not available for connect from 10.0.10.51@o2ib7 (no target). If you are running an HA pair check that the target is mounted on the other server. [Sat Apr 20 05:16:51 2019][6110675.819608] LustreError: Skipped 5 previous similar messages [Sat Apr 20 05:20:02 2019][6110866.964792] Lustre: fir-OST0000: Connection restored to 14aae05e-d3ff-54ad-8b93-c5dd42954ce5 (at 10.8.23.31@o2ib6) [Sat Apr 20 05:20:02 2019][6110866.975361] Lustre: Skipped 5 previous similar messages [Sat Apr 20 05:24:45 2019][6111149.119317] Lustre: fir-OST0002: haven't heard from client a2ec0a42-55bf-8d43-b23b-0dc608c7116f (at 10.8.25.19@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff98483e4ee400, cur 1555763085 expire 1555762935 last 1555762858 [Sat Apr 20 05:24:45 2019][6111149.141203] Lustre: Skipped 5 previous similar messages [Sat Apr 20 05:31:28 2019][6111552.131422] Lustre: fir-OST0006: haven't heard from client c94f5f49-8f9a-fcff-e3f6-eb27235cadfe (at 10.8.30.36@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848845fbc00, cur 1555763488 expire 1555763338 last 1555763261 [Sat Apr 20 05:31:28 2019][6111552.153343] Lustre: Skipped 5 previous similar messages [Sat Apr 20 05:32:48 2019][6111632.143689] Lustre: fir-OST0000: Connection restored to e8e9feb6-5cc2-22ca-f250-c7608b5324bd (at 10.8.23.19@o2ib6) [Sat Apr 20 05:32:48 2019][6111632.154217] Lustre: Skipped 5 previous similar messages [Sat Apr 20 05:53:19 2019][6112863.764439] Lustre: fir-OST0000: Connection restored to f4b03aa2-d5b7-4f9a-0875-baaa698d022e (at 10.8.25.19@o2ib6) [Sat Apr 20 05:53:19 2019][6112863.775027] Lustre: Skipped 5 previous similar messages [Sat Apr 20 05:53:38 2019][6112882.178508] Lustre: fir-OST0004: haven't heard from client c1fdf221-412b-4f6b-54b5-a81e6c1274d7 (at 10.8.25.28@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9848009fb400, cur 1555764818 expire 1555764668 last 1555764591 [Sat Apr 20 05:53:38 2019][6112882.200423] Lustre: Skipped 5 previous similar messages [Sat Apr 20 05:59:54 2019][6113258.711784] Lustre: fir-OST0000: Connection restored to 71464f83-f435-3a33-e9d6-ef54166e95b7 (at 10.8.30.36@o2ib6) [Sat Apr 20 05:59:54 2019][6113258.722310] Lustre: Skipped 5 previous similar messages [Sat Apr 20 06:17:22 2019][6114306.230069] Lustre: fir-OST000a: haven't heard from client 23f8cc01-28b5-21a8-13fc-0bcdff466db1 (at 10.8.22.26@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9868013f5000, cur 1555766242 expire 1555766092 last 1555766015 [Sat Apr 20 06:17:22 2019][6114306.251970] Lustre: Skipped 5 previous similar messages [Sat Apr 20 06:22:27 2019][6114611.711727] Lustre: fir-OST0000: Connection restored to 275e2730-4d3f-dc89-2b66-e7a8cc62e3d6 (at 10.8.25.28@o2ib6) [Sat Apr 20 06:22:27 2019][6114611.722247] Lustre: Skipped 5 previous similar messages [Sat Apr 20 06:23:10 2019][6114654.242857] Lustre: fir-OST0002: haven't heard from client 6e8d1f7b-93ec-9da0-9a6d-dc0fe6985a2e (at 10.8.13.4@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff984b283d9800, cur 1555766590 expire 1555766440 last 1555766363 [Sat Apr 20 06:23:10 2019][6114654.264706] Lustre: Skipped 5 previous similar messages [Sat Apr 20 06:26:27 2019][6114851.250643] Lustre: fir-OST0002: haven't heard from client 771175fc-d01e-36d7-0f7b-60c455ede36d (at 10.8.19.2@o2ib6) in 227 seconds. I think it's dead, and I am evicting it. exp ffff986786808800, cur 1555766787 expire 1555766637 last 1555766560 [Sat Apr 20 06:26:27 2019][6114851.272455] Lustre: Skipped 5 previous similar messages [Sat Apr 20 06:27:43 2019][6114927.255853] Lustre: fir-OST0004: haven't heard from client ec9c8464-eb47-cb23-3b42-bb47517ad6ae (at 10.8.23.18@o2ib6) in 198 seconds. I think it's dead, and I am evicting it. exp ffff986817880c00, cur 1555766863 expire 1555766713 last 1555766665