[13743975.538118] LustreError: 160952:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 2 previous similar messages
[13744024.911307] LustreError: 228831:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff91be9b077050 x1716222288023616/t0(0) o3->05558b19-5945-639d-fbd6-6ee5d8106fc5@10.210.12.55@tcp1:655/0 lens 488/440 e 0 to 0 dl 1644969740 ref 1 fl Interpret:/0/0 rc 0/0
[13744024.911479] Lustre: oak-OST012d: Bulk IO read error with 05558b19-5945-639d-fbd6-6ee5d8106fc5 (at 10.210.12.55@tcp1), client will retry: rc -110
[13744024.911480] Lustre: Skipped 1 previous similar message
[13744024.955398] LustreError: 228831:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 1 previous similar message
[13744055.120160] LustreError: 137-5: oak-OST014a_UUID: not available for connect from 10.210.12.63@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13744055.137754] LustreError: Skipped 1 previous similar message
[13744125.047426] LustreError: 160936:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[13744260.719088] Lustre: oak-OST0139: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[13744260.729504] Lustre: Skipped 249 previous similar messages
[13744468.567837] Lustre: oak-OST0147: Connection restored to 1b2d4a6c-5db6-e628-1755-247dc3737260 (at 10.51.2.37@o2ib3)
[13744468.578433] Lustre: Skipped 1297 previous similar messages
[13744863.330981] Lustre: oak-OST013d: Client 9fc64b77-e33e-204d-02bc-73983c39db00 (at 10.50.17.21@o2ib2) reconnecting
[13744863.341978] Lustre: Skipped 34 previous similar messages
[13745067.152459] Lustre: oak-OST012f: Connection restored to 1e1f187a-28cb-d390-d52c-e3db41797544 (at 10.51.15.21@o2ib3)
[13745067.163136] Lustre: Skipped 815 previous similar messages
[13745338.004387] LustreError: 160932:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91cf23354050 x1715384617008192/t0(0) o4->233f7470-b0ae-e43d-b32e-99c85e344dbd@10.210.12.17@tcp1:512/0 lens 488/448 e 0 to 0 dl 1644971107 ref 1 fl Interpret:/0/0 rc 0/0
[13745338.028970] LustreError: 160932:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 2 previous similar messages
[13745338.039120] Lustre: oak-OST012d: Bulk IO write error with 233f7470-b0ae-e43d-b32e-99c85e344dbd (at 10.210.12.17@tcp1), client will retry: rc = -110
[13745338.052590] Lustre: Skipped 3 previous similar messages
[13745339.035890] LustreError: 127356:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91a613b8b050 x1715384617008192/t0(0) o4->233f7470-b0ae-e43d-b32e-99c85e344dbd@10.210.12.17@tcp1:516/0 lens 488/448 e 0 to 0 dl 1644971111 ref 1 fl Interpret:/2/0 rc 0/0
[13745339.060619] Lustre: oak-OST012d: Bulk IO write error with 233f7470-b0ae-e43d-b32e-99c85e344dbd (at 10.210.12.17@tcp1), client will retry: rc = -110
[13745503.349458] Lustre: oak-OST013f: Client ae865b82-51e6-6ef5-51e8-057f7a99f1a1 (at 10.210.12.63@tcp1) reconnecting
[13745503.359878] Lustre: Skipped 25 previous similar messages
[13745666.057471] Lustre: oak-OST014b: Connection restored to 1b67ac58-7108-cf3f-afa6-c007e56ee1b6 (at 10.50.8.67@o2ib2)
[13745666.068398] Lustre: Skipped 1007 previous similar messages
[13746105.862708] Lustre: oak-OST0123: Client 9fc64b77-e33e-204d-02bc-73983c39db00 (at 10.50.17.21@o2ib2) reconnecting
[13746105.873213] Lustre: Skipped 94 previous similar messages
[13746264.615213] Lustre: oak-OST0143: Connection restored to d79b3874-bbf2-64aa-6b1f-102bce0d0020 (at 10.51.6.63@o2ib3)
[13746264.625800] Lustre: Skipped 1231 previous similar messages
[13746796.215146] Lustre: oak-OST0119: Client 9fc64b77-e33e-204d-02bc-73983c39db00 (at 10.50.17.21@o2ib2) reconnecting
[13746796.225638] Lustre: Skipped 25 previous similar messages
[13746863.549440] Lustre: oak-OST0113: Connection restored to ec6cafa7-2c96-71e9-0dad-24d0eee2b247 (at 10.0.3.37@o2ib5)
[13746863.559933] Lustre: Skipped 850 previous similar messages
[13747395.153543] Lustre: oak-OST0139: Client 9fc64b77-e33e-204d-02bc-73983c39db00 (at 10.50.17.21@o2ib2) reconnecting
[13747395.163993] Lustre: Skipped 53 previous similar messages
[13747462.591146] Lustre: oak-OST011f: Connection restored to ee36833d-7a97-4f8f-3fad-12018bb3bd79 (at 10.210.12.9@tcp1)
[13747462.601733] Lustre: Skipped 814 previous similar messages
[13748066.347275] Lustre: oak-OST013b: Connection restored to 8c7cb508-c6b9-04a0-a468-4f245a92d1d2 (at 10.50.1.69@o2ib2)
[13748066.358004] Lustre: Skipped 801 previous similar messages
[13748076.446895] Lustre: oak-OST014b: Client 9fc64b77-e33e-204d-02bc-73983c39db00 (at 10.50.17.21@o2ib2) reconnecting
[13748076.457315] Lustre: Skipped 60 previous similar messages
[13748666.574392] Lustre: oak-OST0129: Connection restored to 7b218846-8296-5a90-f251-8d8c57ad58ac (at 10.51.6.3@o2ib3)
[13748666.585556] Lustre: Skipped 658 previous similar messages
[13748804.546606] Lustre: oak-OST0143: Client 9fc64b77-e33e-204d-02bc-73983c39db00 (at 10.50.17.21@o2ib2) reconnecting
[13748804.557031] Lustre: Skipped 46 previous similar messages
[13749266.014671] Lustre: oak-OST0127: Connection restored to f7e431f7-0c56-b864-d233-0e31b62c302b (at 10.50.2.37@o2ib2)
[13749266.025265] Lustre: Skipped 646 previous similar messages
[13749458.147551] Lustre: oak-OST013b: Client 9fc64b77-e33e-204d-02bc-73983c39db00 (at 10.50.17.21@o2ib2) reconnecting
[13749458.157976] Lustre: Skipped 15 previous similar messages
[13749866.639441] Lustre: oak-OST012b: Connection restored to 2e8e2ea1-d309-871f-47c3-fbf87733e2ac (at 10.50.5.45@o2ib2)
[13749866.650038] Lustre: Skipped 1018 previous similar messages
[13750463.554287] Lustre: oak-OST0133: Client 9fc64b77-e33e-204d-02bc-73983c39db00 (at 10.50.17.21@o2ib2) reconnecting
[13750463.564712] Lustre: Skipped 18 previous similar messages
[13750469.899025] Lustre: oak-OST0147: Connection restored to  (at 10.50.7.2@o2ib2)
[13750469.906407] Lustre: Skipped 1284 previous similar messages
[13751069.623226] Lustre: oak-OST0139: Connection restored to 8b3b187c-ea57-a5ef-fd2b-c6848207d167 (at 10.50.17.11@o2ib2)
[13751069.634198] Lustre: Skipped 1227 previous similar messages
[13751152.762181] Lustre: oak-OST0111: Client 9fc64b77-e33e-204d-02bc-73983c39db00 (at 10.50.17.21@o2ib2) reconnecting
[13751152.772600] Lustre: Skipped 66 previous similar messages
[13751267.374391] LustreError: 137-5: oak-OST0110_UUID: not available for connect from 10.50.17.21@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13751267.374392] LustreError: 137-5: oak-OST0112_UUID: not available for connect from 10.50.17.21@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13751267.374394] LustreError: Skipped 1 previous similar message
[13751267.415417] LustreError: Skipped 10 previous similar messages
[13751509.789930] LustreError: 137-5: oak-OST0112_UUID: not available for connect from 10.50.17.21@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13751509.807589] LustreError: Skipped 11 previous similar messages
[13751542.420125] LustreError: 21619:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(2433024)  req@ffff91580aaa6850 x1723223360782272/t0(0) o4->ad79db1a-2c23-373b-fe86-4ffbc81fc761@10.50.5.67@o2ib2:616/0 lens 488/448 e 0 to 0 dl 1644977251 ref 1 fl Interpret:/0/0 rc 0/0
[13751542.420225] Lustre: oak-OST0149: Bulk IO write error with ad79db1a-2c23-373b-fe86-4ffbc81fc761 (at 10.50.5.67@o2ib2), client will retry: rc = -110
[13751542.458566] LustreError: 21619:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 4 previous similar messages
[13751668.218228] Lustre: oak-OST0131: Connection restored to  (at 10.50.10.49@o2ib2)
[13751668.225798] Lustre: Skipped 1162 previous similar messages
[13751757.938327] LustreError: 243543:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91908cfe1050 x1722699300513088/t0(0) o4->82f1e960-b4a0-dc80-a7df-0645c17704f4@10.50.5.64@o2ib2:93/0 lens 488/448 e 0 to 0 dl 1644977483 ref 1 fl Interpret:/0/0 rc 0/0
[13751757.938329] LustreError: 21619:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 1048576(4194304)  req@ffff91611830a850 x1722699300515264/t0(0) o4->82f1e960-b4a0-dc80-a7df-0645c17704f4@10.50.5.64@o2ib2:93/0 lens 488/448 e 0 to 0 dl 1644977483 ref 1 fl Interpret:/0/0 rc 0/0
[13751757.938331] LustreError: 21619:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[13751757.938587] Lustre: oak-OST014d: Bulk IO write error with 82f1e960-b4a0-dc80-a7df-0645c17704f4 (at 10.50.5.64@o2ib2), client will retry: rc = -110
[13751757.938588] Lustre: Skipped 4 previous similar messages
[13751758.018288] LustreError: 243543:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[13751791.211739] Lustre: oak-OST0111: Client 9fc64b77-e33e-204d-02bc-73983c39db00 (at 10.50.17.21@o2ib2) reconnecting
[13751791.222463] Lustre: Skipped 92 previous similar messages
[13751805.831443] LustreError: 162688:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff9183cacdc050 x1723223362230528/t0(0) o4->ad79db1a-2c23-373b-fe86-4ffbc81fc761@10.50.5.67@o2ib2:125/0 lens 488/448 e 0 to 0 dl 1644977515 ref 1 fl Interpret:/0/0 rc 0/0
[13751805.831590] Lustre: oak-OST014b: Bulk IO write error with ad79db1a-2c23-373b-fe86-4ffbc81fc761 (at 10.50.5.67@o2ib2), client will retry: rc = -110
[13751805.831591] Lustre: Skipped 3 previous similar messages
[13751805.876104] LustreError: 162688:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 33 previous similar messages
[13751847.282177] LustreError: 162697:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91357c3b6050 x1722699301297856/t0(0) o4->82f1e960-b4a0-dc80-a7df-0645c17704f4@10.50.5.64@o2ib2:273/0 lens 488/448 e 0 to 0 dl 1644977663 ref 1 fl Interpret:/0/0 rc 0/0
[13751847.283283] Lustre: oak-OST014d: Bulk IO write error with 82f1e960-b4a0-dc80-a7df-0645c17704f4 (at 10.50.5.64@o2ib2), client will retry: rc = -110
[13751847.283284] Lustre: Skipped 33 previous similar messages
[13751847.325415] LustreError: 162697:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 3 previous similar messages
[13751885.118217] LustreError: 160917:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91b45ce2b050 x1723223362687680/t0(0) o4->ad79db1a-2c23-373b-fe86-4ffbc81fc761@10.50.5.67@o2ib2:219/0 lens 488/448 e 0 to 0 dl 1644977609 ref 1 fl Interpret:/0/0 rc 0/0
[13751885.118417] Lustre: oak-OST014b: Bulk IO write error with ad79db1a-2c23-373b-fe86-4ffbc81fc761 (at 10.50.5.67@o2ib2), client will retry: rc = -110
[13751885.118418] Lustre: Skipped 3 previous similar messages
[13751885.161409] LustreError: 160917:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[13751887.424216] Lustre: oak-OST0127: haven't heard from client 71579924-73d3-b0f1-1196-b1b741006e2a (at 10.51.12.4@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91c327c05400, cur 1644977579 expire 1644977429 last 1644977352
[13751898.408599] Lustre: oak-OST0133: haven't heard from client 71579924-73d3-b0f1-1196-b1b741006e2a (at 10.51.12.4@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91c36baea000, cur 1644977590 expire 1644977440 last 1644977363
[13751898.430555] Lustre: Skipped 28 previous similar messages
[13752211.061544] LustreError: 137-5: oak-OST011e_UUID: not available for connect from 10.50.17.21@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13752211.079132] LustreError: Skipped 1 previous similar message
[13752272.276471] Lustre: oak-OST0143: Connection restored to ee36833d-7a97-4f8f-3fad-12018bb3bd79 (at 10.210.12.9@tcp1)
[13752272.287080] Lustre: Skipped 1051 previous similar messages
[13752403.578290] LustreError: 137-5: oak-OST0138_UUID: not available for connect from 10.50.17.21@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13752407.888032] Lustre: oak-OST0111: Client 9fc64b77-e33e-204d-02bc-73983c39db00 (at 10.50.17.21@o2ib2) reconnecting
[13752407.898482] Lustre: Skipped 90 previous similar messages
[13752407.940888] LustreError: 137-5: oak-OST0112_UUID: not available for connect from 10.50.17.21@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13752407.958537] LustreError: Skipped 11 previous similar messages
[13752422.124860] Lustre: oak-OST0123: haven't heard from client 9fc64b77-e33e-204d-02bc-73983c39db00 (at 10.50.17.21@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff919c56af0800, cur 1644978115 expire 1644977965 last 1644977888
[13752605.682913] Lustre: oak-OST0113: haven't heard from client 9fc64b77-e33e-204d-02bc-73983c39db00 (at 10.50.17.21@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91de418be000, cur 1644978299 expire 1644978149 last 1644978072
[13752605.705027] Lustre: Skipped 1 previous similar message
[13752739.746843] LustreError: 244100:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91bee37a6850 x1722687219795456/t0(0) o4->bfaba9be-d132-07bf-7e04-96d9eca8997a@10.50.5.44@o2ib2:315/0 lens 488/448 e 0 to 0 dl 1644978460 ref 1 fl Interpret:/0/0 rc 0/0
[13752739.747170] Lustre: oak-OST0145: Bulk IO write error with bfaba9be-d132-07bf-7e04-96d9eca8997a (at 10.50.5.44@o2ib2), client will retry: rc = -110
[13752739.747171] Lustre: Skipped 1 previous similar message
[13752739.791300] LustreError: 244100:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 4 previous similar messages
[13752803.941698] LustreError: 137-5: oak-OST0122_UUID: not available for connect from 10.50.17.21@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13752873.652301] Lustre: oak-OST012b: Connection restored to 488c76a6-efdb-cc65-ba6d-76a1ff5eefb6 (at 10.50.7.38@o2ib2)
[13752873.662881] Lustre: Skipped 950 previous similar messages
[13752931.326597] LustreError: 21594:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff914ea0e43050 x1723223367160384/t0(0) o4->ad79db1a-2c23-373b-fe86-4ffbc81fc761@10.50.5.67@o2ib2:487/0 lens 488/448 e 0 to 0 dl 1644978632 ref 1 fl Interpret:/0/0 rc 0/0
[13752931.327256] Lustre: oak-OST0141: Bulk IO write error with ad79db1a-2c23-373b-fe86-4ffbc81fc761 (at 10.50.5.67@o2ib2), client will retry: rc = -110
[13752931.327258] Lustre: Skipped 4 previous similar messages
[13752931.371042] LustreError: 21594:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 21 previous similar messages
[13753007.125366] Lustre: oak-OST011d: Client 9fc64b77-e33e-204d-02bc-73983c39db00 (at 10.50.17.21@o2ib2) reconnecting
[13753007.135778] Lustre: Skipped 42 previous similar messages
[13753051.070814] LustreError: 160919:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(4194304)  req@ffff91ee118a7850 x1723223367173056/t0(0) o4->ad79db1a-2c23-373b-fe86-4ffbc81fc761@10.50.5.67@o2ib2:619/0 lens 488/448 e 0 to 0 dl 1644978764 ref 1 fl Interpret:/2/0 rc 0/0
[13753051.070987] Lustre: oak-OST0147: Bulk IO write error with ad79db1a-2c23-373b-fe86-4ffbc81fc761 (at 10.50.5.67@o2ib2), client will retry: rc = -110
[13753051.070988] Lustre: Skipped 21 previous similar messages
[13753051.114975] LustreError: 160919:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[13753098.964657] LustreError: 160932:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(3918810)  req@ffff91c81f26f050 x1723223368666368/t0(0) o4->ad79db1a-2c23-373b-fe86-4ffbc81fc761@10.50.5.67@o2ib2:666/0 lens 488/448 e 0 to 0 dl 1644978811 ref 1 fl Interpret:/0/0 rc 0/0
[13753098.964774] Lustre: oak-OST013b: Bulk IO write error with ad79db1a-2c23-373b-fe86-4ffbc81fc761 (at 10.50.5.67@o2ib2), client will retry: rc = -110
[13753098.964775] Lustre: Skipped 2 previous similar messages
[13753099.009328] LustreError: 160932:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 8 previous similar messages
[13753107.463302] Lustre: oak-OST0113: haven't heard from client 9fc64b77-e33e-204d-02bc-73983c39db00 (at 10.50.17.21@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9169bddb1800, cur 1644978802 expire 1644978652 last 1644978575
[13753219.668170] LustreError: 137-5: oak-OST0114_UUID: not available for connect from 10.50.17.21@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13753422.908074] LustreError: 137-5: oak-OST0132_UUID: not available for connect from 10.50.17.21@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13753434.231352] LustreError: 21587:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(87869)  req@ffff918daaf92850 x1723176267691072/t0(0) o4->5e06a9de-9967-5de3-4f7a-c9b0be6077d3@10.50.7.29@o2ib2:252/0 lens 488/448 e 0 to 0 dl 1644979152 ref 1 fl Interpret:/0/0 rc 0/0
[13753434.256521] Lustre: oak-OST0147: Bulk IO write error with 5e06a9de-9967-5de3-4f7a-c9b0be6077d3 (at 10.50.7.29@o2ib2), client will retry: rc = -110
[13753434.270030] Lustre: Skipped 8 previous similar messages
[13753445.687158] Lustre: oak-OST0143: haven't heard from client 9fc64b77-e33e-204d-02bc-73983c39db00 (at 10.50.17.21@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff914eb22f5800, cur 1644979141 expire 1644978991 last 1644978914
[13753445.709175] Lustre: Skipped 1 previous similar message
[13753458.178344] LustreError: 243537:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(4194304)  req@ffff9191e7303850 x1723223370635008/t0(0) o4->ad79db1a-2c23-373b-fe86-4ffbc81fc761@10.50.5.67@o2ib2:267/0 lens 488/448 e 0 to 0 dl 1644979167 ref 1 fl Interpret:/0/0 rc 0/0
[13753458.178547] Lustre: oak-OST0147: Bulk IO write error with ad79db1a-2c23-373b-fe86-4ffbc81fc761 (at 10.50.5.67@o2ib2), client will retry: rc = -110
[13753458.216902] LustreError: 243537:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 18 previous similar messages
[13753472.235070] Lustre: oak-OST014d: Connection restored to a8af91d8-50f2-dfba-de80-bdf0e55b323f (at 10.210.12.66@tcp1)
[13753472.245747] Lustre: Skipped 888 previous similar messages
[13753576.330497] Lustre: oak-OST013f: haven't heard from client 9fc64b77-e33e-204d-02bc-73983c39db00 (at 10.50.17.21@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91f072d93800, cur 1644979272 expire 1644979122 last 1644979045
[13753600.432921] LustreError: 137-5: oak-OST011c_UUID: not available for connect from 10.50.5.67@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13753608.944023] Lustre: oak-OST0149: Client ad79db1a-2c23-373b-fe86-4ffbc81fc761 (at 10.50.5.67@o2ib2) reconnecting
[13753608.954339] Lustre: Skipped 104 previous similar messages
[13753745.414469] LustreError: 137-5: oak-OST0110_UUID: not available for connect from 10.50.17.21@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13753745.432049] LustreError: Skipped 9 previous similar messages
[13753769.509598] LustreError: 160938:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91c10451e050 x1723223372731264/t0(0) o4->ad79db1a-2c23-373b-fe86-4ffbc81fc761@10.50.5.67@o2ib2:583/0 lens 488/448 e 0 to 0 dl 1644979483 ref 1 fl Interpret:/0/0 rc 0/0
[13753769.509823] Lustre: oak-OST0141: Bulk IO write error with ad79db1a-2c23-373b-fe86-4ffbc81fc761 (at 10.50.5.67@o2ib2), client will retry: rc = -110
[13753769.509824] Lustre: Skipped 18 previous similar messages
[13753769.554264] LustreError: 160938:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 25 previous similar messages
[13753842.374328] LustreError: 21588:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff915d937b0050 x1723223372814080/t0(0) o4->ad79db1a-2c23-373b-fe86-4ffbc81fc761@10.50.5.67@o2ib2:694/0 lens 488/448 e 0 to 0 dl 1644979594 ref 1 fl Interpret:/0/0 rc 0/0
[13753842.375432] Lustre: oak-OST0133: Bulk IO write error with ad79db1a-2c23-373b-fe86-4ffbc81fc761 (at 10.50.5.67@o2ib2), client will retry: rc = -110
[13753842.375433] Lustre: Skipped 24 previous similar messages
[13753842.417622] LustreError: 21588:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 5 previous similar messages
[13753842.831363] Lustre: oak-OST0149: Bulk IO read error with ad79db1a-2c23-373b-fe86-4ffbc81fc761 (at 10.50.5.67@o2ib2), client will retry: rc -110
[13753842.844455] Lustre: Skipped 1 previous similar message
[13753860.764628] LustreError: 21588:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff917014548050 x1723223373712640/t0(0) o4->ad79db1a-2c23-373b-fe86-4ffbc81fc761@10.50.5.67@o2ib2:725/0 lens 488/448 e 0 to 0 dl 1644979625 ref 1 fl Interpret:/0/0 rc 0/0
[13753860.788884] LustreError: 21588:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 41 previous similar messages
[13753865.298626] LustreError: 228829:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff9196217f4850 x1723983878775616/t0(0) o4->6accc209-5bb7-cb1e-a16e-6f072cfef396@10.50.10.11@o2ib2:668/0 lens 488/448 e 0 to 0 dl 1644979568 ref 1 fl Interpret:/0/0 rc 0/0
[13753865.324447] LustreError: 228829:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[13753930.655758] LustreError: 160911:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91e8b8b73850 x1723983879216448/t0(0) o4->6accc209-5bb7-cb1e-a16e-6f072cfef396@10.50.10.11@o2ib2:38/0 lens 488/448 e 0 to 0 dl 1644979693 ref 1 fl Interpret:/0/0 rc 0/0
[13753930.680346] LustreError: 160911:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 9 previous similar messages
[13754007.509697] LustreError: 162701:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff914aa0f84050 x1723223374693376/t0(0) o4->ad79db1a-2c23-373b-fe86-4ffbc81fc761@10.50.5.67@o2ib2:117/0 lens 488/448 e 0 to 0 dl 1644979772 ref 1 fl Interpret:/0/0 rc 0/0
[13754007.510860] Lustre: oak-OST0133: Bulk IO write error with ad79db1a-2c23-373b-fe86-4ffbc81fc761 (at 10.50.5.67@o2ib2), client will retry: rc = -110
[13754007.510861] Lustre: Skipped 82 previous similar messages
[13754007.552923] LustreError: 162701:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 9 previous similar messages
[13754072.519585] Lustre: oak-OST012b: Connection restored to af6fafe1-b1db-e45a-426f-8119715defd2 (at 10.50.10.2@o2ib2)
[13754072.530167] Lustre: Skipped 787 previous similar messages
[13754224.497697] LustreError: 21592:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(4194304)  req@ffff9153c2abb850 x1723223375622080/t0(0) o4->ad79db1a-2c23-373b-fe86-4ffbc81fc761@10.50.5.67@o2ib2:286/0 lens 488/448 e 0 to 0 dl 1644979941 ref 1 fl Interpret:/0/0 rc 0/0
[13754224.497938] LustreError: 243448:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(106496)  req@ffff91d903752050 x1722694539115456/t0(0) o3->47a4ff8a-7fd4-d502-f994-37767c17ed29@10.50.5.56@o2ib2:278/0 lens 488/440 e 0 to 0 dl 1644979933 ref 1 fl Interpret:/0/0 rc 0/0
[13754224.497953] Lustre: oak-OST011b: Bulk IO read error with 47a4ff8a-7fd4-d502-f994-37767c17ed29 (at 10.50.5.56@o2ib2), client will retry: rc -110
[13754224.560796] LustreError: 21592:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 37 previous similar messages
[13754299.678671] Lustre: oak-OST014d: Client eb94fb81-a07b-2695-a9a5-cd74903a3134 (at 10.50.5.33@o2ib2) reconnecting
[13754299.689023] Lustre: Skipped 85 previous similar messages
[13754299.696771] LustreError: 160910:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91bdbfd78850 x1722696596717184/t0(0) o4->eb94fb81-a07b-2695-a9a5-cd74903a3134@10.50.5.33@o2ib2:371/0 lens 488/448 e 0 to 0 dl 1644980026 ref 1 fl Interpret:/0/0 rc 0/0
[13754299.696773] LustreError: 243448:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91c54e3f2850 x1722696596717760/t0(0) o4->eb94fb81-a07b-2695-a9a5-cd74903a3134@10.50.5.33@o2ib2:371/0 lens 488/448 e 0 to 0 dl 1644980026 ref 1 fl Interpret:/0/0 rc 0/0
[13754299.696957] Lustre: oak-OST014d: Bulk IO write error with eb94fb81-a07b-2695-a9a5-cd74903a3134 (at 10.50.5.33@o2ib2), client will retry: rc = -110
[13754299.696958] Lustre: Skipped 28 previous similar messages
[13754323.185947] LustreError: 137-5: oak-OST011c_UUID: not available for connect from 10.50.5.67@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13754368.173982] LustreError: 160923:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91c21856b850 x1723223375624000/t0(0) o4->ad79db1a-2c23-373b-fe86-4ffbc81fc761@10.50.5.67@o2ib2:421/0 lens 488/448 e 0 to 0 dl 1644980076 ref 1 fl Interpret:/2/0 rc 0/0
[13754368.199730] LustreError: 160923:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[13754385.988721] LustreError: 243446:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91b15027a850 x1722694539723904/t0(0) o4->47a4ff8a-7fd4-d502-f994-37767c17ed29@10.50.5.56@o2ib2:460/0 lens 488/448 e 0 to 0 dl 1644980115 ref 1 fl Interpret:/0/0 rc 0/0
[13754386.013094] LustreError: 243446:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 5 previous similar messages
[13754456.218354] LustreError: 160905:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91c618202850 x1723223379025856/t0(0) o4->ad79db1a-2c23-373b-fe86-4ffbc81fc761@10.50.5.67@o2ib2:575/0 lens 488/448 e 0 to 0 dl 1644980230 ref 1 fl Interpret:/0/0 rc 0/0
[13754456.218355] LustreError: 160925:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91d27b108050 x1723223379025792/t0(0) o4->ad79db1a-2c23-373b-fe86-4ffbc81fc761@10.50.5.67@o2ib2:575/0 lens 488/448 e 0 to 0 dl 1644980230 ref 1 fl Interpret:/0/0 rc 0/0
[13754456.218358] LustreError: 160925:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 2 previous similar messages
[13754456.276957] LustreError: 160905:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[13754655.521118] LustreError: 162704:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff915ded439050 x1723913528273536/t0(0) o4->227acc10-fcf1-c357-f1da-048706fcd648@10.50.5.63@o2ib2:8/0 lens 488/448 e 0 to 0 dl 1644980418 ref 1 fl Interpret:/0/0 rc 0/0
[13754655.546670] LustreError: 162704:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 8 previous similar messages
[13754671.620556] Lustre: oak-OST011d: Connection restored to d34cc33f-555c-1a6e-a9d6-bc8c27f9c288 (at 10.50.6.67@o2ib2)
[13754671.631222] Lustre: Skipped 751 previous similar messages
[13754699.838277] LustreError: 127353:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91c0d17d6850 x1723223380009472/t0(0) o4->ad79db1a-2c23-373b-fe86-4ffbc81fc761@10.50.5.67@o2ib2:52/0 lens 488/448 e 0 to 0 dl 1644980462 ref 1 fl Interpret:/0/0 rc 0/0
[13754699.862517] LustreError: 127353:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[13754710.642275] LustreError: 137-5: oak-OST011e_UUID: not available for connect from 10.50.5.67@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13754710.660158] LustreError: Skipped 8 previous similar messages
[13754871.046272] Lustre: oak-OST013b: Bulk IO write error with ad79db1a-2c23-373b-fe86-4ffbc81fc761 (at 10.50.5.67@o2ib2), client will retry: rc = -110
[13754871.059620] Lustre: Skipped 35 previous similar messages
[13754884.154097] Lustre: oak-OST0133: haven't heard from client ad79db1a-2c23-373b-fe86-4ffbc81fc761 (at 10.50.5.67@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91a8dd9d0400, cur 1644980583 expire 1644980433 last 1644980356
[13754884.176081] Lustre: Skipped 1 previous similar message
[13754957.297809] Lustre: oak-OST013b: Client ad79db1a-2c23-373b-fe86-4ffbc81fc761 (at 10.50.5.67@o2ib2) reconnecting
[13754957.308131] Lustre: Skipped 82 previous similar messages
[13755270.634150] Lustre: oak-OST0127: Connection restored to  (at 10.51.14.17@o2ib3)
[13755270.641707] Lustre: Skipped 993 previous similar messages
[13755278.145031] LustreError: 21592:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff9164cdaeb850 x1723223382738496/t0(0) o4->ad79db1a-2c23-373b-fe86-4ffbc81fc761@10.50.5.67@o2ib2:581/0 lens 488/448 e 0 to 0 dl 1644980991 ref 1 fl Interpret:/0/0 rc 0/0
[13755278.170659] LustreError: 21592:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 25 previous similar messages
[13755424.066123] LustreError: 243499:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9182ec831050 x1723223383200192/t0(0) o4->ad79db1a-2c23-373b-fe86-4ffbc81fc761@10.50.5.67@o2ib2:16/0 lens 504/448 e 0 to 0 dl 1644981181 ref 1 fl Interpret:/0/0 rc 0/0
[13755516.643187] Lustre: oak-OST0123: Bulk IO write error with 47a4ff8a-7fd4-d502-f994-37767c17ed29 (at 10.50.5.56@o2ib2), client will retry: rc = -110
[13755516.656554] Lustre: Skipped 29 previous similar messages
[13755584.313539] Lustre: oak-OST0147: Client ad79db1a-2c23-373b-fe86-4ffbc81fc761 (at 10.50.5.67@o2ib2) reconnecting
[13755584.324654] Lustre: Skipped 38 previous similar messages
[13755588.504663] LustreError: 228404:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 3145728(4194304)  req@ffff91c2dc031850 x1723913533273408/t0(0) o3->227acc10-fcf1-c357-f1da-048706fcd648@10.50.5.63@o2ib2:143/0 lens 488/440 e 0 to 0 dl 1644981308 ref 1 fl Interpret:/0/0 rc 0/0
[13755588.530262] Lustre: oak-OST0143: Bulk IO read error with 227acc10-fcf1-c357-f1da-048706fcd648 (at 10.50.5.63@o2ib2), client will retry: rc -110
[13755608.693566] LustreError: 137-5: oak-OST0128_UUID: not available for connect from 10.50.5.63@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13755608.711117] LustreError: Skipped 1 previous similar message
[13755871.285355] Lustre: oak-OST0117: Connection restored to bbba5ad9-9372-2297-5b66-f72cfc361471 (at 10.0.3.25@o2ib5)
[13755871.296427] Lustre: Skipped 909 previous similar messages
[13756330.917852] LustreError: 243536:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 1814528(2863104)  req@ffff917617ffa850 x1723223388181184/t0(0) o4->ad79db1a-2c23-373b-fe86-4ffbc81fc761@10.50.5.67@o2ib2:144/0 lens 488/448 e 0 to 0 dl 1644982064 ref 1 fl Interpret:/0/0 rc 0/0
[13756330.918000] Lustre: oak-OST0143: Bulk IO write error with ad79db1a-2c23-373b-fe86-4ffbc81fc761 (at 10.50.5.67@o2ib2), client will retry: rc = -110
[13756330.918001] Lustre: Skipped 34 previous similar messages
[13756330.962670] LustreError: 243536:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 45 previous similar messages
[13756385.048688] Lustre: oak-OST0141: Client ad79db1a-2c23-373b-fe86-4ffbc81fc761 (at 10.50.5.67@o2ib2) reconnecting
[13756385.059025] Lustre: Skipped 19 previous similar messages
[13756470.071293] Lustre: oak-OST013f: Connection restored to 96736516-d8e7-70f3-8c44-2c43fd1826b8 (at 10.51.13.3@o2ib3)
[13756470.081912] Lustre: Skipped 855 previous similar messages
[13756785.917917] LustreError: 243445:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(1048576)  req@ffff9189bc931050 x1715080755023168/t0(0) o3->65e9b8a8-b461-24e0-da12-5b6939f0b07c@10.210.12.56@tcp1:605/0 lens 488/440 e 0 to 0 dl 1644982525 ref 1 fl Interpret:/0/0 rc 0/0
[13756785.942980] Lustre: oak-OST014d: Bulk IO read error with 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1), client will retry: rc -110
[13756808.526371] LustreError: 137-5: oak-OST0144_UUID: not available for connect from 10.210.12.56@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13756808.543958] LustreError: Skipped 3 previous similar messages
[13756984.907853] Lustre: oak-OST012d: Client 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1) reconnecting
[13756984.918305] Lustre: Skipped 36 previous similar messages
[13757049.348673] LustreError: 243541:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(847872)  req@ffff914c54709050 x1723922015746432/t0(0) o3->65d0a477-eecb-43d2-9ff9-d741bd19bc7f@10.50.5.65@o2ib2:93/0 lens 488/440 e 0 to 0 dl 1644982768 ref 1 fl Interpret:/0/0 rc 0/0
[13757049.376401] Lustre: oak-OST0125: Bulk IO read error with 65d0a477-eecb-43d2-9ff9-d741bd19bc7f (at 10.50.5.65@o2ib2), client will retry: rc -110
[13757071.187299] Lustre: oak-OST013d: Connection restored to 894b9462-4ef1-1bb0-6365-60255fe14a6d (at 10.50.17.24@o2ib2)
[13757071.197962] Lustre: Skipped 1205 previous similar messages
[13757456.448726] LustreError: 160929:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91e755d61050 x1723970549674816/t0(0) o4->b4dcab20-58eb-df05-dbac-816cd3f289e2@10.50.12.15@o2ib2:495/0 lens 488/448 e 0 to 0 dl 1644983170 ref 1 fl Interpret:/0/0 rc 0/0
[13757456.448998] Lustre: oak-OST013f: Bulk IO write error with b4dcab20-58eb-df05-dbac-816cd3f289e2 (at 10.50.12.15@o2ib2), client will retry: rc = -110
[13757456.448999] Lustre: Skipped 10 previous similar messages
[13757456.494444] LustreError: 160929:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 3 previous similar messages
[13757671.362093] Lustre: oak-OST014d: Connection restored to fc6538a6-64f5-1c92-d38a-4c03c9b82dd0 (at 10.210.12.123@tcp1)
[13757671.372913] Lustre: Skipped 867 previous similar messages
[13757839.608069] LustreError: 160929:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(262144)  req@ffff91c0c3ec8050 x1723810565911808/t0(0) o3->36e14f36-3af5-1c46-4849-6c6ac5ccffed@10.50.13.4@o2ib2:138/0 lens 488/440 e 0 to 0 dl 1644983568 ref 1 fl Interpret:/0/0 rc 0/0
[13757839.632980] Lustre: oak-OST014b: Bulk IO read error with 36e14f36-3af5-1c46-4849-6c6ac5ccffed (at 10.50.13.4@o2ib2), client will retry: rc -110
[13757948.189108] Lustre: oak-OST014b: Client 36e14f36-3af5-1c46-4849-6c6ac5ccffed (at 10.50.13.4@o2ib2) reconnecting
[13757948.199426] Lustre: Skipped 12 previous similar messages
[13758193.210517] LustreError: 199272:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST0131: cli ad79db1a-2c23-373b-fe86-4ffbc81fc761 claims 4218880 GRANT, real grant 3665920
[13758195.785714] LustreError: 127349:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST0131: cli ad79db1a-2c23-373b-fe86-4ffbc81fc761 claims 2457600 GRANT, real grant 0
[13758201.896996] LustreError: 243454:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST0131: cli ad79db1a-2c23-373b-fe86-4ffbc81fc761 claims 4218880 GRANT, real grant 0
[13758201.911395] LustreError: 243454:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 1 previous similar message
[13758208.921011] LustreError: 160953:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST0131: cli ad79db1a-2c23-373b-fe86-4ffbc81fc761 claims 4218880 GRANT, real grant 0
[13758219.273540] LustreError: 160922:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST0125: cli ad79db1a-2c23-373b-fe86-4ffbc81fc761 claims 4218880 GRANT, real grant 2351104
[13758219.288459] LustreError: 160922:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 1 previous similar message
[13758235.822568] LustreError: 243451:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST0125: cli ad79db1a-2c23-373b-fe86-4ffbc81fc761 claims 4055040 GRANT, real grant 0
[13758235.837043] LustreError: 243451:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 2 previous similar messages
[13758260.068457] LustreError: 253954:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST0125: cli ad79db1a-2c23-373b-fe86-4ffbc81fc761 claims 4218880 GRANT, real grant 0
[13758260.082854] LustreError: 253954:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 4 previous similar messages
[13758272.682275] Lustre: oak-OST012f: Connection restored to b8aee960-6cca-b8d7-bb46-d1dcde837605 (at 10.50.2.62@o2ib2)
[13758272.692888] Lustre: Skipped 894 previous similar messages
[13758294.684035] LustreError: 253938:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST0125: cli ad79db1a-2c23-373b-fe86-4ffbc81fc761 claims 4218880 GRANT, real grant 0
[13758294.698513] LustreError: 253938:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 10 previous similar messages
[13758504.449049] LustreError: 199272:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST0125: cli ad79db1a-2c23-373b-fe86-4ffbc81fc761 claims 4218880 GRANT, real grant 0
[13758504.463449] LustreError: 199272:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 17 previous similar messages
[13758645.121598] LustreError: 160955:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST0125: cli ad79db1a-2c23-373b-fe86-4ffbc81fc761 claims 4218880 GRANT, real grant 0
[13758645.135989] LustreError: 160955:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 34 previous similar messages
[13758653.806229] LustreError: 162704:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 1048576(4194304)  req@ffff9182ccb34850 x1723223394624576/t0(0) o4->ad79db1a-2c23-373b-fe86-4ffbc81fc761@10.50.5.67@o2ib2:204/0 lens 488/448 e 0 to 0 dl 1644984389 ref 1 fl Interpret:/0/0 rc 0/0
[13758653.806464] Lustre: oak-OST012d: Bulk IO write error with ad79db1a-2c23-373b-fe86-4ffbc81fc761 (at 10.50.5.67@o2ib2), client will retry: rc = -110
[13758653.806465] Lustre: Skipped 1 previous similar message
[13758653.850736] LustreError: 162704:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 12 previous similar messages
[13758733.750163] Lustre: oak-OST013f: Client ad79db1a-2c23-373b-fe86-4ffbc81fc761 (at 10.50.5.67@o2ib2) reconnecting
[13758877.282337] Lustre: oak-OST011d: Connection restored to 99a1409b-11ff-d63b-e3e5-6044909060f0 (at 10.210.12.129@tcp1)
[13758877.293101] Lustre: Skipped 1691 previous similar messages
[13758900.671894] LustreError: 160947:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST0125: cli ad79db1a-2c23-373b-fe86-4ffbc81fc761 claims 1773568 GRANT, real grant 0
[13758900.686453] LustreError: 160947:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 23 previous similar messages
[13759300.361076] LustreError: 127358:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(4194304)  req@ffff91befae83050 x1723970575323904/t0(0) o3->b4dcab20-58eb-df05-dbac-816cd3f289e2@10.50.12.15@o2ib2:93/0 lens 488/440 e 0 to 0 dl 1644985033 ref 1 fl Interpret:/0/0 rc 0/0
[13759300.386159] Lustre: oak-OST0117: Bulk IO read error with b4dcab20-58eb-df05-dbac-816cd3f289e2 (at 10.50.12.15@o2ib2), client will retry: rc -110
[13759348.253341] LustreError: 243442:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(970752)  req@ffff9180c346b850 x1723223397233408/t0(0) o3->ad79db1a-2c23-373b-fe86-4ffbc81fc761@10.50.5.67@o2ib2:136/0 lens 488/440 e 0 to 0 dl 1644985076 ref 1 fl Interpret:/0/0 rc 0/0
[13759348.278291] Lustre: oak-OST0137: Bulk IO read error with ad79db1a-2c23-373b-fe86-4ffbc81fc761 (at 10.50.5.67@o2ib2), client will retry: rc -110
[13759364.721677] Lustre: oak-OST0145: Client ad79db1a-2c23-373b-fe86-4ffbc81fc761 (at 10.50.5.67@o2ib2) reconnecting
[13759364.732116] Lustre: Skipped 8 previous similar messages
[13759378.731260] LustreError: 137-5: oak-OST012c_UUID: not available for connect from 10.50.5.67@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13759378.985631] LustreError: 21605:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff917768e15850 x1723223396660480/t0(0) o4->ad79db1a-2c23-373b-fe86-4ffbc81fc761@10.50.5.67@o2ib2:199/0 lens 504/448 e 0 to 0 dl 1644985139 ref 1 fl Interpret:/2/0 rc 0/0
[13759378.986769] Lustre: oak-OST013f: Bulk IO write error with ad79db1a-2c23-373b-fe86-4ffbc81fc761 (at 10.50.5.67@o2ib2), client will retry: rc = -110
[13759378.986770] Lustre: Skipped 22 previous similar messages
[13759379.028827] LustreError: 21605:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 15 previous similar messages
[13759420.094875] LustreError: 160920:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(3245596)  req@ffff91f0e0dd0050 x1723970576405824/t0(0) o4->b4dcab20-58eb-df05-dbac-816cd3f289e2@10.50.12.15@o2ib2:204/0 lens 488/448 e 0 to 0 dl 1644985144 ref 1 fl Interpret:/0/0 rc 0/0
[13759420.120781] LustreError: 160920:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 11 previous similar messages
[13759456.561391] LustreError: 137-5: oak-OST0132_UUID: not available for connect from 10.50.5.67@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13759465.254644] LustreError: 137-5: oak-OST013e_UUID: not available for connect from 10.50.5.67@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13759476.583067] Lustre: oak-OST0111: Connection restored to 37c9099c-1a38-a699-2712-432f4cb01e44 (at 10.210.12.19@tcp1)
[13759476.593743] Lustre: Skipped 1134 previous similar messages
[13759477.481133] LustreError: 137-5: oak-OST0118_UUID: not available for connect from 10.50.5.67@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13759477.498635] LustreError: Skipped 6 previous similar messages
[13759972.033607] Lustre: oak-OST0147: Client ad79db1a-2c23-373b-fe86-4ffbc81fc761 (at 10.50.5.67@o2ib2) reconnecting
[13759972.043937] Lustre: Skipped 69 previous similar messages
[13760076.282422] Lustre: oak-OST0111: Connection restored to bce82964-b032-43f4-daa4-4feb0c63cf2c (at 10.50.7.3@o2ib2)
[13760076.292918] Lustre: Skipped 1291 previous similar messages
[13760113.535365] LustreError: 243542:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(851529)  req@ffff9143fecec850 x1723223402788032/t0(0) o4->ad79db1a-2c23-373b-fe86-4ffbc81fc761@10.50.5.67@o2ib2:150/0 lens 488/448 e 0 to 0 dl 1644985845 ref 1 fl Interpret:/0/0 rc 0/0
[13760113.535367] LustreError: 162690:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4075520)  req@ffff914c4ac48850 x1723223402787968/t0(0) o4->ad79db1a-2c23-373b-fe86-4ffbc81fc761@10.50.5.67@o2ib2:150/0 lens 488/448 e 0 to 0 dl 1644985845 ref 1 fl Interpret:/0/0 rc 0/0
[13760113.535369] LustreError: 162690:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 35 previous similar messages
[13760113.535506] Lustre: oak-OST014b: Bulk IO write error with ad79db1a-2c23-373b-fe86-4ffbc81fc761 (at 10.50.5.67@o2ib2), client will retry: rc = -110
[13760113.535507] Lustre: Skipped 38 previous similar messages
[13760123.233024] LustreError: 137-5: oak-OST0156_UUID: not available for connect from 10.50.5.67@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13760123.250535] LustreError: Skipped 9 previous similar messages
[13760142.628880] LustreError: 137-5: oak-OST0146_UUID: not available for connect from 10.50.5.67@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13760212.087584] LustreError: 137-5: oak-OST0144_UUID: not available for connect from 10.50.5.67@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13760246.503338] LustreError: 137-5: oak-OST013c_UUID: not available for connect from 10.50.5.67@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13760387.669897] LustreError: 137-5: oak-OST0156_UUID: not available for connect from 10.50.5.67@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13760409.513142] LustreError: 137-5: oak-OST0146_UUID: not available for connect from 10.50.5.67@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13760409.530700] LustreError: Skipped 3 previous similar messages
[13760675.000188] Lustre: oak-OST0119: Connection restored to  (at 10.50.12.14@o2ib2)
[13760675.007759] Lustre: Skipped 1163 previous similar messages
[13760993.772755] LNet: 210834:0:(o2iblnd_cb.c:2629:kiblnd_passive_connect()) Conn stale 10.0.3.4@o2ib5 version 12/12 incarnation 1617295049595010/1644986698638524
[13761276.977869] Lustre: oak-OST0149: Connection restored to 1ab4b7ea-cfcf-7e9e-f0c5-fe7f76dd9f9c (at 10.210.12.115@tcp1)
[13761276.988645] Lustre: Skipped 1217 previous similar messages
[13761743.675052] Lustre: oak-OST011b: Client 265911d7-3757-8f9a-04c9-f973fd6cce47 (at 10.210.12.82@tcp1) reconnecting
[13761743.685471] Lustre: Skipped 69 previous similar messages
[13761875.558012] Lustre: oak-OST0145: Connection restored to 8abb03e7-db16-0b1b-6b1b-9594e6bd4e8e (at 10.50.2.64@o2ib2)
[13761875.568605] Lustre: Skipped 1056 previous similar messages
[13762220.849194] LustreError: 160898:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91e8ed1a7850 x1723913558822720/t0(0) o4->227acc10-fcf1-c357-f1da-048706fcd648@10.50.5.63@o2ib2:3/0 lens 488/448 e 1 to 0 dl 1644987963 ref 1 fl Interpret:/0/0 rc 0/0
[13762220.849406] Lustre: oak-OST013f: Bulk IO write error with 227acc10-fcf1-c357-f1da-048706fcd648 (at 10.50.5.63@o2ib2), client will retry: rc = -110
[13762220.849407] Lustre: Skipped 3 previous similar messages
[13762220.893551] LustreError: 160898:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 4 previous similar messages
[13762263.521228] Lustre: oak-OST013f: Client 227acc10-fcf1-c357-f1da-048706fcd648 (at 10.50.5.63@o2ib2) reconnecting
[13762263.531563] Lustre: Skipped 40 previous similar messages
[13762316.634821] LustreError: 162696:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff918f8ec46050 x1723913558824320/t0(0) o4->227acc10-fcf1-c357-f1da-048706fcd648@10.50.5.63@o2ib2:118/0 lens 488/448 e 0 to 0 dl 1644988078 ref 1 fl Interpret:/2/0 rc 0/0
[13762316.635081] Lustre: oak-OST013f: Bulk IO write error with 227acc10-fcf1-c357-f1da-048706fcd648 (at 10.50.5.63@o2ib2), client will retry: rc = -110
[13762316.635082] Lustre: Skipped 4 previous similar messages
[13762316.679579] LustreError: 162696:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[13762421.558634] Lustre: oak-OST013f: Client 227acc10-fcf1-c357-f1da-048706fcd648 (at 10.50.5.63@o2ib2) reconnecting
[13762421.568955] Lustre: Skipped 2 previous similar messages
[13762421.590675] LustreError: 243443:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91850c02e850 x1723913560347328/t0(0) o4->227acc10-fcf1-c357-f1da-048706fcd648@10.50.5.63@o2ib2:247/0 lens 488/448 e 0 to 0 dl 1644988207 ref 1 fl Interpret:/0/0 rc 0/0
[13762478.082805] Lustre: oak-OST014d: Connection restored to 5212eebf-1b23-9d18-5820-9c58afe0b7ba (at 10.210.12.133@tcp1)
[13762478.093563] Lustre: Skipped 961 previous similar messages
[13762637.908126] LustreError: 137-5: oak-OST011a_UUID: not available for connect from 10.50.5.61@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13762722.700634] Lustre: oak-OST014b: Client b2583baf-fcfe-cf54-b945-b20905221dcd (at 10.50.10.7@o2ib2) reconnecting
[13762722.710975] Lustre: Skipped 8 previous similar messages
[13763078.992978] Lustre: oak-OST0143: Connection restored to  (at 10.50.10.57@o2ib2)
[13763079.000544] Lustre: Skipped 952 previous similar messages
[13763679.627392] Lustre: oak-OST012d: Connection restored to 7c96f719-603e-506a-2dd8-6cf9c88fbcce (at 10.51.2.14@o2ib3)
[13763679.637979] Lustre: Skipped 959 previous similar messages
[13763968.963969] LustreError: 162699:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff914246a99850 x1722699337827776/t0(0) o4->82f1e960-b4a0-dc80-a7df-0645c17704f4@10.50.5.64@o2ib2:238/0 lens 488/448 e 0 to 0 dl 1644989708 ref 1 fl Interpret:/0/0 rc 0/0
[13763968.964552] Lustre: oak-OST0147: Bulk IO write error with eb94fb81-a07b-2695-a9a5-cd74903a3134 (at 10.50.5.33@o2ib2), client will retry: rc = -110
[13763968.964553] Lustre: Skipped 2 previous similar messages
[13763969.009350] LustreError: 162699:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[13764075.702570] Lustre: oak-OST013f: Client 82f1e960-b4a0-dc80-a7df-0645c17704f4 (at 10.50.5.64@o2ib2) reconnecting
[13764075.712897] Lustre: Skipped 2 previous similar messages
[13764112.644855] LustreError: 21613:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(1327104)  req@ffff91919d768050 x1722696638587200/t0(0) o3->eb94fb81-a07b-2695-a9a5-cd74903a3134@10.50.5.33@o2ib2:383/0 lens 488/440 e 0 to 0 dl 1644989853 ref 1 fl Interpret:/0/0 rc 0/0
[13764112.644870] LustreError: 243535:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff917f73ec9050 x1722687283681024/t0(0) o4->bfaba9be-d132-07bf-7e04-96d9eca8997a@10.50.5.44@o2ib2:383/0 lens 488/448 e 0 to 0 dl 1644989853 ref 1 fl Interpret:/0/0 rc 0/0
[13764112.644906] Lustre: oak-OST0133: Bulk IO read error with eb94fb81-a07b-2695-a9a5-cd74903a3134 (at 10.50.5.33@o2ib2), client will retry: rc -110
[13764112.645274] Lustre: oak-OST0141: Bulk IO write error with bfaba9be-d132-07bf-7e04-96d9eca8997a (at 10.50.5.44@o2ib2), client will retry: rc = -110
[13764112.645276] Lustre: Skipped 1 previous similar message
[13764112.735843] LustreError: 21613:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 1 previous similar message
[13764177.170304] Lustre: oak-OST0141: Client bfaba9be-d132-07bf-7e04-96d9eca8997a (at 10.50.5.44@o2ib2) reconnecting
[13764177.180683] Lustre: Skipped 1 previous similar message
[13764184.486964] LustreError: 160945:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(933888)  req@ffff91bdc697f850 x1723983919493632/t0(0) o3->6accc209-5bb7-cb1e-a16e-6f072cfef396@10.50.10.11@o2ib2:453/0 lens 488/440 e 0 to 0 dl 1644989923 ref 1 fl Interpret:/0/0 rc 0/0
[13764184.486975] LustreError: 160928:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91ac95bec050 x1724102985387520/t0(0) o4->2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3@10.50.5.68@o2ib2:453/0 lens 488/448 e 0 to 0 dl 1644989923 ref 1 fl Interpret:/0/0 rc 0/0
[13764184.486977] LustreError: 160928:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 5 previous similar messages
[13764184.487191] Lustre: oak-OST0135: Bulk IO write error with 2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3 (at 10.50.5.68@o2ib2), client will retry: rc = -110
[13764184.487193] Lustre: Skipped 4 previous similar messages
[13764184.512328] Lustre: oak-OST013f: Bulk IO read error with 2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3 (at 10.50.5.68@o2ib2), client will retry: rc -110
[13764184.512329] Lustre: Skipped 1 previous similar message
[13764184.585041] LustreError: 160945:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 1 previous similar message
[13764224.561766] LustreError: 162675:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91709d758850 x1723983919592128/t0(0) o4->6accc209-5bb7-cb1e-a16e-6f072cfef396@10.50.10.11@o2ib2:527/0 lens 488/448 e 0 to 0 dl 1644989997 ref 1 fl Interpret:/0/0 rc 0/0
[13764224.586335] LustreError: 162675:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[13764256.328599] LustreError: 229136:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(929792)  req@ffff91713d10a850 x1723983919592512/t0(0) o3->6accc209-5bb7-cb1e-a16e-6f072cfef396@10.50.10.11@o2ib2:527/0 lens 488/440 e 0 to 0 dl 1644989997 ref 1 fl Interpret:/0/0 rc 0/0
[13764256.353618] Lustre: oak-OST0147: Bulk IO read error with 6accc209-5bb7-cb1e-a16e-6f072cfef396 (at 10.50.10.11@o2ib2), client will retry: rc -110
[13764256.366803] Lustre: Skipped 1 previous similar message
[13764284.157942] Lustre: oak-OST0111: Connection restored to ded9a7c5-4772-1464-d085-edb54ff675c6 (at 10.50.1.29@o2ib2)
[13764284.168523] Lustre: Skipped 953 previous similar messages
[13764328.139798] Lustre: oak-OST0135: Client b4dcab20-58eb-df05-dbac-816cd3f289e2 (at 10.50.12.15@o2ib2) reconnecting
[13764328.150238] Lustre: Skipped 6 previous similar messages
[13764368.155884] Lustre: oak-OST0147: haven't heard from client eb94fb81-a07b-2695-a9a5-cd74903a3134 (at 10.50.5.33@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff916d63eaf400, cur 1644990090 expire 1644989940 last 1644989863
[13764390.893590] LustreError: 137-5: oak-OST0142_UUID: not available for connect from 10.50.5.33@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13764390.911095] LustreError: Skipped 1 previous similar message
[13764401.812396] LustreError: 137-5: oak-OST011a_UUID: not available for connect from 10.50.5.61@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13764403.886523] LustreError: 137-5: oak-OST0122_UUID: not available for connect from 10.50.10.11@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13764408.218746] LustreError: 137-5: oak-OST0110_UUID: not available for connect from 10.50.5.33@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13764408.236396] LustreError: Skipped 1 previous similar message
[13764416.863748] LustreError: 137-5: oak-OST014c_UUID: not available for connect from 10.50.5.61@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13764429.835890] LustreError: 137-5: oak-OST0114_UUID: not available for connect from 10.50.5.33@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13764447.181394] LustreError: 137-5: oak-OST013a_UUID: not available for connect from 10.50.5.33@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13764447.181395] LustreError: 137-5: oak-OST0120_UUID: not available for connect from 10.50.5.33@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13764447.181397] LustreError: Skipped 4 previous similar messages
[13764447.222462] LustreError: Skipped 2 previous similar messages
[13764488.392166] LustreError: 137-5: oak-OST014e_UUID: not available for connect from 10.50.5.33@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13764488.409673] LustreError: Skipped 1 previous similar message
[13764512.667519] LNet: 210834:0:(o2iblnd_cb.c:2629:kiblnd_passive_connect()) Conn stale 10.0.3.4@o2ib5 version 12/12 incarnation 1644988888926534/1644990227333051
[13764572.268741] LustreError: 137-5: oak-OST0110_UUID: not available for connect from 10.50.10.11@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13764572.286337] LustreError: Skipped 23 previous similar messages
[13764601.525319] Lustre: oak-OST013b: haven't heard from client eb94fb81-a07b-2695-a9a5-cd74903a3134 (at 10.50.5.33@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9136f2b94c00, cur 1644990324 expire 1644990174 last 1644990097
[13764672.310286] Lustre: oak-OST0113: Client eb94fb81-a07b-2695-a9a5-cd74903a3134 (at 10.50.5.33@o2ib2) reconnecting
[13764672.320634] Lustre: Skipped 108 previous similar messages
[13764731.206892] Lustre: oak-OST0135: haven't heard from client 6accc209-5bb7-cb1e-a16e-6f072cfef396 (at 10.50.10.11@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9141ae416000, cur 1644990454 expire 1644990304 last 1644990227
[13764731.228916] Lustre: Skipped 6 previous similar messages
[13764742.203831] LustreError: 137-5: oak-OST0142_UUID: not available for connect from 10.50.10.11@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13764742.221507] LustreError: Skipped 16 previous similar messages
[13764830.040246] LustreError: 243439:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(536831)  req@ffff91621b241050 x1723983924824320/t0(0) o4->6accc209-5bb7-cb1e-a16e-6f072cfef396@10.50.10.11@o2ib2:353/0 lens 488/448 e 0 to 0 dl 1644990578 ref 1 fl Interpret:/0/0 rc 0/0
[13764830.065449] LustreError: 243439:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 13 previous similar messages
[13764830.075650] Lustre: oak-OST013d: Bulk IO write error with 6accc209-5bb7-cb1e-a16e-6f072cfef396 (at 10.50.10.11@o2ib2), client will retry: rc = -110
[13764830.089098] Lustre: Skipped 16 previous similar messages
[13764882.846445] Lustre: oak-OST0137: Connection restored to ea7323ea-d4bb-2678-29a1-bc8d87c76269 (at 10.210.12.6@tcp1)
[13764882.857069] Lustre: Skipped 1489 previous similar messages
[13765045.556466] LustreError: 162699:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff9163665bd850 x1722687289836992/t0(0) o4->bfaba9be-d132-07bf-7e04-96d9eca8997a@10.50.5.44@o2ib2:559/0 lens 488/448 e 0 to 0 dl 1644990784 ref 1 fl Interpret:/0/0 rc 0/0
[13765045.556758] Lustre: oak-OST0141: Bulk IO write error with bfaba9be-d132-07bf-7e04-96d9eca8997a (at 10.50.5.44@o2ib2), client will retry: rc = -110
[13765045.595547] LustreError: 162699:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[13765053.042165] LNet: 90577:0:(o2iblnd_cb.c:2629:kiblnd_passive_connect()) Conn stale 10.0.3.4@o2ib5 version 12/12 incarnation 1644990227333051/1644990773664173
[13765481.451971] Lustre: oak-OST0131: Connection restored to c870b475-34f8-d1ac-3897-051a7f74d2e1 (at 10.50.5.29@o2ib2)
[13765481.462547] Lustre: Skipped 1450 previous similar messages
[13765883.672956] LustreError: 160951:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91d6fedca850 x1723970704750720/t0(0) o4->b4dcab20-58eb-df05-dbac-816cd3f289e2@10.50.12.15@o2ib2:640/0 lens 488/448 e 0 to 0 dl 1644991620 ref 1 fl Interpret:/0/0 rc 0/0
[13765883.699091] Lustre: oak-OST0135: Bulk IO write error with b4dcab20-58eb-df05-dbac-816cd3f289e2 (at 10.50.12.15@o2ib2), client will retry: rc = -110
[13765883.712560] Lustre: Skipped 2 previous similar messages
[13765970.723740] Lustre: oak-OST0135: Client b4dcab20-58eb-df05-dbac-816cd3f289e2 (at 10.50.12.15@o2ib2) reconnecting
[13765970.734216] Lustre: Skipped 15 previous similar messages
[13766080.368774] Lustre: oak-OST0123: Connection restored to a1771d7a-63d0-8cd8-9e63-2128188e0783 (at 10.50.9.42@o2ib2)
[13766080.379353] Lustre: Skipped 1541 previous similar messages
[13766503.140463] Lustre: oak-OST0111: Client d7a4c1f3-87e3-d107-56e4-3cbbb5e91c07 (at 10.50.12.3@o2ib2) reconnecting
[13766503.150812] Lustre: Skipped 17 previous similar messages
[13766556.421473] LNet: 64750:0:(o2iblnd_cb.c:2629:kiblnd_passive_connect()) Conn stale 10.0.3.4@o2ib5 version 12/12 incarnation 1644990773664173/1644992277729326
[13766679.361892] Lustre: oak-OST011f: Connection restored to 5e0db087-5559-6ddb-af76-b91bad1c5eb7 (at 10.50.2.41@o2ib2)
[13766679.372521] Lustre: Skipped 1095 previous similar messages
[13766865.497224] LustreError: 21587:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff9153bbd2b050 x1723922070231488/t0(0) o4->65d0a477-eecb-43d2-9ff9-d741bd19bc7f@10.50.5.65@o2ib2:119/0 lens 488/448 e 0 to 0 dl 1644992609 ref 1 fl Interpret:/0/0 rc 0/0
[13766865.497398] Lustre: oak-OST013f: Bulk IO write error with 65d0a477-eecb-43d2-9ff9-d741bd19bc7f (at 10.50.5.65@o2ib2), client will retry: rc = -110
[13766865.536389] LustreError: 21587:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 5 previous similar messages
[13766953.275210] Lustre: oak-OST0135: Client 2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3 (at 10.50.5.68@o2ib2) reconnecting
[13766961.291404] LustreError: 162699:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(282624)  req@ffff918a61c1b050 x1722694601661888/t0(0) o3->47a4ff8a-7fd4-d502-f994-37767c17ed29@10.50.5.56@o2ib2:205/0 lens 488/440 e 0 to 0 dl 1644992695 ref 1 fl Interpret:/0/0 rc 0/0
[13766961.316282] Lustre: oak-OST0141: Bulk IO read error with 47a4ff8a-7fd4-d502-f994-37767c17ed29 (at 10.50.5.56@o2ib2), client will retry: rc -110
[13767176.810865] LustreError: 244100:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff91bf91c6b050 x1724022545229632/t0(0) o3->7caa6746-008a-4c85-9dca-b006031ecd37@10.50.5.30@o2ib2:490/0 lens 488/440 e 0 to 0 dl 1644992980 ref 1 fl Interpret:/0/0 rc 0/0
[13767176.811024] Lustre: oak-OST0141: Bulk IO read error with 7caa6746-008a-4c85-9dca-b006031ecd37 (at 10.50.5.30@o2ib2), client will retry: rc -110
[13767176.849489] LustreError: 244100:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 1 previous similar message
[13767278.832309] Lustre: oak-OST0111: Connection restored to fb4c352d-d5d8-df67-aa5f-fdf2fea140a1 (at 10.50.7.54@o2ib2)
[13767278.842905] Lustre: Skipped 1119 previous similar messages
[13767310.350451] Lustre: oak-OST0117: Client 47a4ff8a-7fd4-d502-f994-37767c17ed29 (at 10.50.5.56@o2ib2) reconnecting
[13767310.360815] Lustre: Skipped 100 previous similar messages
[13767347.800987] LustreError: 253936:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91f18ea30050 x1723922075400832/t0(0) o4->65d0a477-eecb-43d2-9ff9-d741bd19bc7f@10.50.5.65@o2ib2:701/0 lens 488/448 e 0 to 0 dl 1644993191 ref 1 fl Interpret:/0/0 rc 0/0
[13767347.825321] LustreError: 253936:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 4 previous similar messages
[13767390.600967] LustreError: 229136:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff915948250850 x1722699349897600/t0(0) o4->82f1e960-b4a0-dc80-a7df-0645c17704f4@10.50.5.64@o2ib2:719/0 lens 488/448 e 0 to 0 dl 1644993209 ref 1 fl Interpret:/0/0 rc 0/0
[13767390.625362] LustreError: 229136:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[13767502.981061] LustreError: 137-5: oak-OST0114_UUID: not available for connect from 10.50.12.3@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13767502.998611] LustreError: Skipped 16 previous similar messages
[13767512.063981] LustreError: 21610:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(4194304)  req@ffff916420ce8050 x1724327824298816/t0(0) o3->18a389ad-f2fe-5ff3-1696-54295097e8af@10.50.7.39@o2ib2:26/0 lens 488/440 e 0 to 0 dl 1644993271 ref 1 fl Interpret:/0/0 rc 0/0
[13767512.064193] LustreError: 21589:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 1048576(3352357)  req@ffff913d8940b050 x1723913581971520/t0(0) o4->227acc10-fcf1-c357-f1da-048706fcd648@10.50.5.63@o2ib2:34/0 lens 504/448 e 0 to 0 dl 1644993279 ref 1 fl Interpret:/0/0 rc 0/0
[13767512.064195] LustreError: 21589:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 38 previous similar messages
[13767512.064293] Lustre: oak-OST0117: Bulk IO write error with 227acc10-fcf1-c357-f1da-048706fcd648 (at 10.50.5.63@o2ib2), client will retry: rc = -110
[13767512.064294] Lustre: Skipped 49 previous similar messages
[13767512.143924] Lustre: oak-OST011f: Bulk IO read error with 18a389ad-f2fe-5ff3-1696-54295097e8af (at 10.50.7.39@o2ib2), client will retry: rc -110
[13767512.157223] Lustre: Skipped 1 previous similar message
[13767536.008709] LustreError: 243537:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff915094e57050 x1722696654758080/t0(0) o3->eb94fb81-a07b-2695-a9a5-cd74903a3134@10.50.5.33@o2ib2:57/0 lens 488/440 e 0 to 0 dl 1644993302 ref 1 fl Interpret:/0/0 rc 0/0
[13767536.033458] Lustre: oak-OST0137: Bulk IO read error with eb94fb81-a07b-2695-a9a5-cd74903a3134 (at 10.50.5.33@o2ib2), client will retry: rc -110
[13767583.900365] LustreError: 199271:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 3145728(4194304)  req@ffff9144564e8850 x1722689027531072/t0(0) o3->6ed73abe-802e-4651-607f-85efb7af0ce5@10.50.5.43@o2ib2:102/0 lens 488/440 e 0 to 0 dl 1644993347 ref 1 fl Interpret:/0/0 rc 0/0
[13767583.900563] Lustre: oak-OST0149: Bulk IO read error with 6ed73abe-802e-4651-607f-85efb7af0ce5 (at 10.50.5.43@o2ib2), client will retry: rc -110
[13767583.938927] LustreError: 199271:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 1 previous similar message
[13767607.846016] LustreError: 160932:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff91ac95bee850 x1722696654810688/t0(0) o3->eb94fb81-a07b-2695-a9a5-cd74903a3134@10.50.5.33@o2ib2:112/0 lens 488/440 e 0 to 0 dl 1644993357 ref 1 fl Interpret:/0/0 rc 0/0
[13767607.870790] Lustre: oak-OST0143: Bulk IO read error with eb94fb81-a07b-2695-a9a5-cd74903a3134 (at 10.50.5.33@o2ib2), client will retry: rc -110
[13767607.883874] Lustre: Skipped 1 previous similar message
[13767696.613225] LustreError: 162680:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9150a1512850 x1722688218744192/t0(0) o4->4922053b-976c-cd78-fb08-b5907ad19b3e@10.50.5.62@o2ib2:270/0 lens 488/448 e 0 to 0 dl 1644993515 ref 1 fl Interpret:/0/0 rc 0/0
[13767721.546356] LustreError: 137-5: oak-OST014c_UUID: not available for connect from 10.50.12.7@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13767803.213151] LustreError: 199244:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 147s: evicting client at 10.50.2.28@o2ib2  ns: filter-oak-OST013d_UUID lock: ffff916c7e90cec0/0xed112d3016474581 lrc: 3/0,0 mode: PW/PW res: [0x1f417a:0x0:0x0].0x0 rrc: 80 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000000020020 nid: 10.50.2.28@o2ib2 remote: 0xf3d6d8faa3eea554 expref: 9 pid: 168271 timeout: 13801266 lvb_type: 0
[13767823.331057] LustreError: 137-5: oak-OST011c_UUID: not available for connect from 10.50.14.14@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13767877.411165] Lustre: oak-OST014b: Connection restored to  (at 10.51.6.6@o2ib3)
[13767877.418550] Lustre: Skipped 1425 previous similar messages
[13767909.819765] Lustre: oak-OST0143: Client 3afe4eca-43a2-0d7b-612b-7aaa975485b1 (at 10.50.2.28@o2ib2) reconnecting
[13767909.830225] Lustre: Skipped 340 previous similar messages
[13768015.888541] LustreError: 137-5: oak-OST014c_UUID: not available for connect from 10.50.2.28@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13768015.906180] LustreError: Skipped 2 previous similar messages
[13768071.096062] Lustre: oak-OST011f: haven't heard from client 3afe4eca-43a2-0d7b-612b-7aaa975485b1 (at 10.50.2.28@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91eb819cd000, cur 1644993802 expire 1644993652 last 1644993575
[13768071.117969] Lustre: Skipped 5 previous similar messages
[13768110.724793] LustreError: 21593:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(4145152)  req@ffff916c0e52c050 x1723913584164928/t0(0) o4->227acc10-fcf1-c357-f1da-048706fcd648@10.50.5.63@o2ib2:631/0 lens 488/448 e 0 to 0 dl 1644993876 ref 1 fl Interpret:/0/0 rc 0/0
[13768110.724826] Lustre: oak-OST0117: Bulk IO write error with 227acc10-fcf1-c357-f1da-048706fcd648 (at 10.50.5.63@o2ib2), client will retry: rc = -110
[13768110.724828] Lustre: Skipped 24 previous similar messages
[13768110.768977] LustreError: 21593:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 33 previous similar messages
[13768298.551360] Lustre: oak-OST0137: haven't heard from client e773bf1a-125c-5078-7524-2bafe07fda90 (at 10.50.7.32@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91a8e0d2c000, cur 1644994030 expire 1644993880 last 1644993803
[13768306.545478] Lustre: oak-OST0125: haven't heard from client 46b488f8-0a49-1549-ed15-012d35da2038 (at 10.50.7.30@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9161d3579400, cur 1644994038 expire 1644993888 last 1644993811
[13768311.521225] Lustre: oak-OST0131: haven't heard from client 3afe4eca-43a2-0d7b-612b-7aaa975485b1 (at 10.50.2.28@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91b8a1d72800, cur 1644994043 expire 1644993893 last 1644993816
[13768311.543155] Lustre: Skipped 3 previous similar messages
[13768337.526899] Lustre: oak-OST0121: haven't heard from client e773bf1a-125c-5078-7524-2bafe07fda90 (at 10.50.7.32@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91d9eb2f5000, cur 1644994069 expire 1644993919 last 1644993842
[13768340.562880] LustreError: 137-5: oak-OST0118_UUID: not available for connect from 10.50.5.56@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13768340.580375] LustreError: Skipped 36 previous similar messages
[13768376.348733] Lustre: oak-OST0115: haven't heard from client b16d27f3-e553-3a31-1277-009573ebd16d (at 10.50.2.32@o2ib2) in 215 seconds. I think it's dead, and I am evicting it. exp ffff91524a596c00, cur 1644994108 expire 1644993958 last 1644993893
[13768376.370679] Lustre: Skipped 1 previous similar message
[13768477.068707] Lustre: oak-OST0139: Connection restored to 280f4c06-7e23-bdcb-8260-28081a746a51 (at 10.51.16.17@o2ib3)
[13768477.079422] Lustre: Skipped 1435 previous similar messages
[13768900.958085] LustreError: 162716:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(1331200)  req@ffff918f788e1850 x1722699354855936/t0(0) o3->82f1e960-b4a0-dc80-a7df-0645c17704f4@10.50.5.64@o2ib2:661/0 lens 488/440 e 1 to 0 dl 1644994661 ref 1 fl Interpret:/0/0 rc 0/0
[13768900.983123] Lustre: oak-OST0133: Bulk IO read error with 82f1e960-b4a0-dc80-a7df-0645c17704f4 (at 10.50.5.64@o2ib2), client will retry: rc -110
[13768931.315974] LustreError: 137-5: oak-OST013a_UUID: not available for connect from 10.50.5.64@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13768931.333489] LustreError: Skipped 46 previous similar messages
[13768970.317745] Lustre: oak-OST0115: Client 82f1e960-b4a0-dc80-a7df-0645c17704f4 (at 10.50.5.64@o2ib2) reconnecting
[13768970.328087] Lustre: Skipped 345 previous similar messages
[13769076.335552] Lustre: oak-OST014b: Connection restored to  (at 10.51.6.16@o2ib3)
[13769076.343026] Lustre: Skipped 1147 previous similar messages
[13769212.257458] LustreError: 21608:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 1916561(4013713)  req@ffff9153bbd2b050 x1723913588993792/t0(0) o4->227acc10-fcf1-c357-f1da-048706fcd648@10.50.5.63@o2ib2:261/0 lens 488/448 e 0 to 0 dl 1644995016 ref 1 fl Interpret:/0/0 rc 0/0
[13769212.257578] Lustre: oak-OST0117: Bulk IO write error with 227acc10-fcf1-c357-f1da-048706fcd648 (at 10.50.5.63@o2ib2), client will retry: rc = -110
[13769212.257579] Lustre: Skipped 11 previous similar messages
[13769212.302059] LustreError: 21608:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 3 previous similar messages
[13769674.902150] Lustre: oak-OST011b: Connection restored to af282abc-f991-f641-6b7f-7ad234257b60 (at 10.210.13.37@tcp1)
[13769674.912818] Lustre: Skipped 1099 previous similar messages
[13770169.136235] LustreError: 160924:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91be45308850 x1724022558201024/t0(0) o4->7caa6746-008a-4c85-9dca-b006031ecd37@10.50.5.30@o2ib2:429/0 lens 488/448 e 0 to 0 dl 1644995939 ref 1 fl Interpret:/0/0 rc 0/0
[13770169.136490] Lustre: oak-OST014d: Bulk IO write error with 7caa6746-008a-4c85-9dca-b006031ecd37 (at 10.50.5.30@o2ib2), client will retry: rc = -110
[13770169.136491] Lustre: Skipped 8 previous similar messages
[13770169.180919] LustreError: 160924:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 6 previous similar messages
[13770258.363683] Lustre: oak-OST014d: Client 7caa6746-008a-4c85-9dca-b006031ecd37 (at 10.50.5.30@o2ib2) reconnecting
[13770258.374008] Lustre: Skipped 7 previous similar messages
[13770274.808830] Lustre: oak-OST0141: Connection restored to 1e369504-a05d-47b6-dff1-1c29be4dae5e (at 10.51.13.18@o2ib3)
[13770274.819516] Lustre: Skipped 861 previous similar messages
[13770398.642105] Lustre: oak-OST0111: Client 47a4ff8a-7fd4-d502-f994-37767c17ed29 (at 10.50.5.56@o2ib2) reconnecting
[13770398.652482] Lustre: Skipped 1 previous similar message
[13770599.911440] Lustre: oak-OST0135: Client 2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3 (at 10.50.5.68@o2ib2) reconnecting
[13770600.321119] LustreError: 160947:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91a26e2e3850 x1724103008278016/t0(0) o4->2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3@10.50.5.68@o2ib2:177/0 lens 504/448 e 0 to 0 dl 1644996442 ref 1 fl Interpret:/0/0 rc 0/0
[13770873.520266] Lustre: oak-OST0111: Connection restored to 9eb8189a-3985-f41c-7d7b-51af46681d5f (at 10.0.3.29@o2ib5)
[13770873.530763] Lustre: Skipped 1418 previous similar messages
[13770937.747698] Lustre: oak-OST0111: Client 9fc64b77-e33e-204d-02bc-73983c39db00 (at 10.50.17.21@o2ib2) reconnecting
[13770937.758102] Lustre: Skipped 17 previous similar messages
[13771472.238360] Lustre: oak-OST014b: Connection restored to 06e08a68-dbe3-82c7-8c68-f836d5ec2c7f (at 10.210.12.135@tcp1)
[13771472.249139] Lustre: Skipped 1033 previous similar messages
[13771536.968979] Lustre: oak-OST0111: Client 9fc64b77-e33e-204d-02bc-73983c39db00 (at 10.50.17.21@o2ib2) reconnecting
[13771536.979389] Lustre: Skipped 39 previous similar messages
[13771597.620453] LustreError: 137-5: oak-OST011a_UUID: not available for connect from 10.50.17.21@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13771597.638071] LustreError: Skipped 1 previous similar message
[13771937.408914] LustreError: 137-5: oak-OST0110_UUID: not available for connect from 10.50.17.21@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13771937.426495] LustreError: Skipped 19 previous similar messages
[13772071.624328] Lustre: oak-OST0143: Connection restored to d34cc33f-555c-1a6e-a9d6-bc8c27f9c288 (at 10.50.6.67@o2ib2)
[13772071.634953] Lustre: Skipped 1067 previous similar messages
[13772333.446490] Lustre: oak-OST011d: Client 9fc64b77-e33e-204d-02bc-73983c39db00 (at 10.50.17.21@o2ib2) reconnecting
[13772333.457000] Lustre: Skipped 58 previous similar messages
[13772352.893289] LustreError: 137-5: oak-OST0112_UUID: not available for connect from 10.50.17.21@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13772352.893290] LustreError: 137-5: oak-OST011e_UUID: not available for connect from 10.50.17.21@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13772352.928550] LustreError: Skipped 2 previous similar messages
[13772670.372991] Lustre: oak-OST0141: Connection restored to 157a6e65-a216-2e5c-9c3d-99ca2219218c (at 10.50.2.3@o2ib2)
[13772670.383487] Lustre: Skipped 1062 previous similar messages
[13772950.245220] Lustre: oak-OST0131: Client 9fc64b77-e33e-204d-02bc-73983c39db00 (at 10.50.17.21@o2ib2) reconnecting
[13772950.255658] Lustre: Skipped 109 previous similar messages
[13773269.601107] Lustre: oak-OST0149: Connection restored to bd5e7790-2ad0-0cd2-da31-fc8f4845d895 (at 10.51.5.45@o2ib3)
[13773269.611694] Lustre: Skipped 1222 previous similar messages
[13773377.964351] LustreError: 162675:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(4194304)  req@ffff913b05fdd850 x1722689050287488/t0(0) o4->6ed73abe-802e-4651-607f-85efb7af0ce5@10.50.5.43@o2ib2:600/0 lens 488/448 e 0 to 0 dl 1644999130 ref 1 fl Interpret:/0/0 rc 0/0
[13773377.964472] Lustre: oak-OST0137: Bulk IO write error with 6ed73abe-802e-4651-607f-85efb7af0ce5 (at 10.50.5.43@o2ib2), client will retry: rc = -110
[13773377.964473] Lustre: Skipped 10 previous similar messages
[13773378.010782] LustreError: 162675:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 12 previous similar messages
[13773417.655170] LustreError: 137-5: oak-OST0114_UUID: not available for connect from 10.50.17.21@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13773417.655171] LustreError: 137-5: oak-OST0116_UUID: not available for connect from 10.50.17.21@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13773417.655173] LustreError: Skipped 33 previous similar messages
[13773417.696377] LustreError: Skipped 9 previous similar messages
[13773576.469012] Lustre: oak-OST0123: Client 9fc64b77-e33e-204d-02bc-73983c39db00 (at 10.50.17.21@o2ib2) reconnecting
[13773576.479454] Lustre: Skipped 83 previous similar messages
[13773870.470773] Lustre: oak-OST0147: Connection restored to 5282df0d-70af-37db-ccf3-a06241b9319e (at 10.51.4.38@o2ib3)
[13773870.481358] Lustre: Skipped 1410 previous similar messages
[13774376.164346] Lustre: oak-OST011d: Client 9fc64b77-e33e-204d-02bc-73983c39db00 (at 10.50.17.21@o2ib2) reconnecting
[13774376.174970] Lustre: Skipped 33 previous similar messages
[13774471.266899] Lustre: oak-OST0149: Connection restored to 2f07595a-cdc5-96c4-b625-6d62bb84e611 (at 10.50.1.33@o2ib2)
[13774471.277483] Lustre: Skipped 872 previous similar messages
[13774503.442807] LustreError: 21593:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91663b062050 x1724103022443200/t0(0) o4->2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3@10.50.5.68@o2ib2:233/0 lens 488/448 e 0 to 0 dl 1645000273 ref 1 fl Interpret:/0/0 rc 0/0
[13774503.468648] Lustre: oak-OST0147: Bulk IO write error with 2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3 (at 10.50.5.68@o2ib2), client will retry: rc = -110
[13774503.482011] Lustre: Skipped 4 previous similar messages
[13775070.829491] Lustre: oak-OST011d: Connection restored to 471d8473-ce4f-aec7-199c-b5ad6376849b (at 10.50.12.15@o2ib2)
[13775070.840166] Lustre: Skipped 1083 previous similar messages
[13775125.087234] LustreError: 160934:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(122880)  req@ffff91b6b25e3850 x1722688258807680/t0(0) o3->4922053b-976c-cd78-fb08-b5907ad19b3e@10.50.5.62@o2ib2:101/0 lens 488/440 e 0 to 0 dl 1645000896 ref 1 fl Interpret:/0/0 rc 0/0
[13775125.112104] Lustre: oak-OST0131: Bulk IO read error with 4922053b-976c-cd78-fb08-b5907ad19b3e (at 10.50.5.62@o2ib2), client will retry: rc -110
[13775196.576418] Lustre: oak-OST0131: Client 4922053b-976c-cd78-fb08-b5907ad19b3e (at 10.50.5.62@o2ib2) reconnecting
[13775196.586752] Lustre: Skipped 72 previous similar messages
[13775436.392991] LustreError: 243457:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff919a37ff1050 x1722686424526592/t0(0) o4->1fd469cf-8787-5b10-3534-d21dd2e02487@10.50.5.61@o2ib2:414/0 lens 488/448 e 0 to 0 dl 1645001209 ref 1 fl Interpret:/0/0 rc 0/0
[13775436.419313] Lustre: oak-OST0139: Bulk IO write error with 1fd469cf-8787-5b10-3534-d21dd2e02487 (at 10.50.5.61@o2ib2), client will retry: rc = -110
[13775484.287810] LustreError: 160898:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(942080)  req@ffff91d036be7850 x1722688259957120/t0(0) o3->4922053b-976c-cd78-fb08-b5907ad19b3e@10.50.5.62@o2ib2:459/0 lens 488/440 e 0 to 0 dl 1645001254 ref 1 fl Interpret:/0/0 rc 0/0
[13775484.312715] Lustre: oak-OST0129: Bulk IO read error with 4922053b-976c-cd78-fb08-b5907ad19b3e (at 10.50.5.62@o2ib2), client will retry: rc -110
[13775651.910050] LustreError: 243549:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff918ed6ed7850 x1722694644514432/t0(0) o4->47a4ff8a-7fd4-d502-f994-37767c17ed29@10.50.5.56@o2ib2:628/0 lens 488/448 e 0 to 0 dl 1645001423 ref 1 fl Interpret:/0/0 rc 0/0
[13775651.936068] Lustre: oak-OST0147: Bulk IO write error with 47a4ff8a-7fd4-d502-f994-37767c17ed29 (at 10.50.5.56@o2ib2), client will retry: rc = -110
[13775667.307120] LustreError: 162710:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff915ded439850 x1723913613675456/t0(0) o4->227acc10-fcf1-c357-f1da-048706fcd648@10.50.5.63@o2ib2:633/0 lens 488/448 e 0 to 0 dl 1645001428 ref 1 fl Interpret:/0/0 rc 0/0
[13775667.315268] Lustre: oak-OST0145: Bulk IO write error with 227acc10-fcf1-c357-f1da-048706fcd648 (at 10.50.5.63@o2ib2), client will retry: rc = -110
[13775667.344802] LustreError: 162710:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[13775672.645251] Lustre: oak-OST0117: Connection restored to 9e55d772-67df-9cd1-1540-cceaeb7907dd (at 10.210.12.127@tcp1)
[13775672.656080] Lustre: Skipped 1190 previous similar messages
[13775673.151965] LustreError: 21601:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff914b15c89850 x1723913613747776/t0(0) o4->227acc10-fcf1-c357-f1da-048706fcd648@10.50.5.63@o2ib2:647/0 lens 488/448 e 0 to 0 dl 1645001442 ref 1 fl Interpret:/0/0 rc 0/0
[13775673.176586] Lustre: oak-OST0121: Bulk IO write error with 227acc10-fcf1-c357-f1da-048706fcd648 (at 10.50.5.63@o2ib2), client will retry: rc = -110
[13775673.189949] Lustre: Skipped 1 previous similar message
[13775675.853982] LustreError: 21593:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff915ded43a050 x1722694644592896/t0(0) o4->47a4ff8a-7fd4-d502-f994-37767c17ed29@10.50.5.56@o2ib2:645/0 lens 488/448 e 0 to 0 dl 1645001440 ref 1 fl Interpret:/0/0 rc 0/0
[13775675.853989] LustreError: 199269:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(139264)  req@ffff914b15c88050 x1722694644619200/t0(0) o3->47a4ff8a-7fd4-d502-f994-37767c17ed29@10.50.5.56@o2ib2:645/0 lens 488/440 e 0 to 0 dl 1645001440 ref 1 fl Interpret:/0/0 rc 0/0
[13775675.854007] Lustre: oak-OST0131: Bulk IO read error with 47a4ff8a-7fd4-d502-f994-37767c17ed29 (at 10.50.5.56@o2ib2), client will retry: rc -110
[13775675.917618] LustreError: 21593:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[13775691.637086] LustreError: 253937:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91b67470a050 x1722686424865216/t0(0) o4->1fd469cf-8787-5b10-3534-d21dd2e02487@10.50.5.61@o2ib2:692/0 lens 504/448 e 0 to 0 dl 1645001487 ref 1 fl Interpret:/0/0 rc 0/0
[13775691.661466] LustreError: 253937:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 2 previous similar messages
[13775691.671350] Lustre: oak-OST0139: Bulk IO write error with 1fd469cf-8787-5b10-3534-d21dd2e02487 (at 10.50.5.61@o2ib2), client will retry: rc = -110
[13775691.684727] Lustre: Skipped 4 previous similar messages
[13775810.022421] Lustre: oak-OST0117: Client 62de813e-c2c7-3b0a-3fb9-7ad67368c8f7 (at 10.50.10.3@o2ib2) reconnecting
[13775810.032766] Lustre: Skipped 49 previous similar messages
[13775813.732387] LustreError: 137-5: oak-OST0122_UUID: not available for connect from 10.50.10.3@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13775813.749919] LustreError: Skipped 3 previous similar messages
[13775915.100923] LustreError: 137-5: oak-OST0128_UUID: not available for connect from 10.50.10.3@o2ib2 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13775915.118460] LustreError: Skipped 2 previous similar messages
[13776271.394569] Lustre: oak-OST0129: Connection restored to e24bed2f-8211-5d1f-86ba-ea59cd68a302 (at 10.50.9.32@o2ib2)
[13776271.405159] Lustre: Skipped 1326 previous similar messages
[13776870.313380] Lustre: oak-OST0115: Connection restored to 1ab4b7ea-cfcf-7e9e-f0c5-fe7f76dd9f9c (at 10.210.12.115@tcp1)
[13776870.324152] Lustre: Skipped 717 previous similar messages
[13777472.828170] Lustre: oak-OST0111: Connection restored to 4a1f3214-3e75-9581-6103-101b733dde38 (at 10.0.3.14@o2ib5)
[13777472.838662] Lustre: Skipped 913 previous similar messages
[13778071.991988] Lustre: oak-OST0121: Connection restored to  (at 10.51.15.9@o2ib3)
[13778071.999478] Lustre: Skipped 1045 previous similar messages
[13778672.414531] Lustre: oak-OST0111: Connection restored to eea359e6-6b89-c326-60af-8cb99076065b (at 10.210.12.8@tcp1)
[13778672.425109] Lustre: Skipped 1308 previous similar messages
[13779066.543268] Lustre: oak-OST0111: Client 6bab8d10-30ff-c1d2-bbf5-1fd0884ac647 (at 10.210.12.57@tcp1) reconnecting
[13779066.553674] Lustre: Skipped 21 previous similar messages
[13779271.574318] Lustre: oak-OST012b: Connection restored to 767b7c94-5f6a-643b-d3fc-ff266d4e288d (at 10.50.4.60@o2ib2)
[13779271.584904] Lustre: Skipped 833 previous similar messages
[13779870.356482] Lustre: oak-OST0139: Connection restored to d963ee83-3fcf-bfee-1cc5-212ce4916b8e (at 10.51.1.56@o2ib3)
[13779870.367130] Lustre: Skipped 1143 previous similar messages
[13780472.595161] Lustre: oak-OST013d: Connection restored to 17df2ba7-0fc1-f30d-a343-f3481405b870 (at 10.50.2.71@o2ib2)
[13780472.605776] Lustre: Skipped 1268 previous similar messages
[13780861.166261] ses 1:0:0:0: attempting task abort! scmd(ffff91686beef700)
[13780861.173043] ses 1:0:0:0: [sg1] tag#2 CDB: Receive Diagnostic 1c 01 02 ff ff 00
[13780861.180512] scsi target1:0:0: _scsih_tm_display_info: handle(0x0012), sas_address(0x50012be000090bbd), phy(48)
[13780861.190739] scsi target1:0:0: enclosurelogical id(0x50012be000090bbf), slot(48) 
[13780861.198375] scsi target1:0:0: enclosure level(0x0000), connector name(     )
[13780861.209928] ses 1:0:0:0: task abort: SUCCESS scmd(ffff91686beef700)
[13780895.256223] LustreError: 160903:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(20480)  req@ffff91ef23ab0050 x1715530406802560/t0(0) o3->5a5caa86-68df-f806-7e6f-05440e95a4ce@10.210.12.43@tcp1:613/0 lens 488/440 e 0 to 0 dl 1645006693 ref 1 fl Interpret:/0/0 rc 0/0
[13780895.256909] Lustre: oak-OST0123: Bulk IO read error with c4b979fd-3b98-af30-5ea5-32f00f4d8750 (at 10.210.12.64@tcp1), client will retry: rc -110
[13780895.294274] LustreError: 160903:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 3 previous similar messages
[13780918.686844] LustreError: 137-5: oak-OST0138_UUID: not available for connect from 10.210.12.64@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13780918.704471] LustreError: Skipped 1 previous similar message
[13781003.049529] Lustre: oak-OST012b: Client 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1) reconnecting
[13781003.059951] Lustre: Skipped 31 previous similar messages
[13781012.716393] Lustre: oak-OST0119: Client 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1) reconnecting
[13781012.726802] Lustre: Skipped 16 previous similar messages
[13781019.318079] LustreError: 160944:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff91ec11187050 x1715081043030976/t0(0) o3->65e9b8a8-b461-24e0-da12-5b6939f0b07c@10.210.12.56@tcp1:44/0 lens 488/440 e 0 to 0 dl 1645006879 ref 1 fl Interpret:/0/0 rc 0/0
[13781019.342386] Lustre: oak-OST0131: Bulk IO read error with 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1), client will retry: rc -110
[13781019.355564] Lustre: Skipped 3 previous similar messages
[13781035.487501] Lustre: oak-OST011b: Client 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1) reconnecting
[13781035.497915] Lustre: Skipped 42 previous similar messages
[13781071.212450] Lustre: oak-OST0111: Connection restored to 9eb8189a-3985-f41c-7d7b-51af46681d5f (at 10.0.3.29@o2ib5)
[13781071.222972] Lustre: Skipped 1253 previous similar messages
[13781670.158206] Lustre: oak-OST0121: Connection restored to 1ab4b7ea-cfcf-7e9e-f0c5-fe7f76dd9f9c (at 10.210.12.115@tcp1)
[13781670.168966] Lustre: Skipped 1340 previous similar messages
[13782269.246417] Lustre: oak-OST0113: Connection restored to 205b97ce-caed-12f0-b38c-0726091178ee (at 10.50.9.14@o2ib2)
[13782269.256995] Lustre: Skipped 1460 previous similar messages
[13782576.392380] LustreError: 137-5: oak-OST0118_UUID: not available for connect from 10.210.12.61@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13782660.180785] Lustre: oak-OST0111: Client 6d01c866-9ec9-8f84-4f86-ee1db83afc97 (at 10.210.12.61@tcp1) reconnecting
[13782660.191374] Lustre: Skipped 1 previous similar message
[13782667.567624] Lustre: oak-OST0119: Client 6d01c866-9ec9-8f84-4f86-ee1db83afc97 (at 10.210.12.61@tcp1) reconnecting
[13782667.578041] Lustre: Skipped 9 previous similar messages
[13782679.021824] Lustre: oak-OST012b: Client 6d01c866-9ec9-8f84-4f86-ee1db83afc97 (at 10.210.12.61@tcp1) reconnecting
[13782679.032230] Lustre: Skipped 22 previous similar messages
[13782834.893686] LustreError: 160924:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91983aa70050 x1722694704525056/t0(0) o4->c8035945-2d4b-8cc0-e678-4f1615763b9b@10.50.5.52@o2ib2:274/0 lens 488/448 e 0 to 0 dl 1645008619 ref 1 fl Interpret:/0/0 rc 0/0
[13782834.919605] Lustre: oak-OST013f: Bulk IO write error with c8035945-2d4b-8cc0-e678-4f1615763b9b (at 10.50.5.52@o2ib2), client will retry: rc = -110
[13782869.979217] Lustre: oak-OST0121: Connection restored to 50699c5e-2222-8bb2-4ca7-3d8de5c8a69d (at 10.51.14.9@o2ib3)
[13782869.989808] Lustre: Skipped 1164 previous similar messages
[13782907.513852] Lustre: oak-OST013f: Client c8035945-2d4b-8cc0-e678-4f1615763b9b (at 10.50.5.52@o2ib2) reconnecting
[13782907.524168] Lustre: Skipped 3 previous similar messages
[13783394.462185] LustreError: 137-5: oak-OST0138_UUID: not available for connect from 10.210.12.64@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13783469.321025] Lustre: oak-OST0135: Connection restored to  (at 10.51.14.15@o2ib3)
[13783469.328596] Lustre: Skipped 1107 previous similar messages
[13783478.662215] Lustre: oak-OST0121: Client c4b979fd-3b98-af30-5ea5-32f00f4d8750 (at 10.210.12.64@tcp1) reconnecting
[13783478.672647] Lustre: Skipped 1 previous similar message
[13783984.288057] LustreError: 127350:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(610304)  req@ffff9196f0d3a850 x1723913651710528/t0(0) o3->227acc10-fcf1-c357-f1da-048706fcd648@10.50.5.63@o2ib2:671/0 lens 488/440 e 0 to 0 dl 1645009771 ref 1 fl Interpret:/0/0 rc 0/0
[13783984.288146] Lustre: oak-OST0127: Bulk IO read error with 227acc10-fcf1-c357-f1da-048706fcd648 (at 10.50.5.63@o2ib2), client will retry: rc -110
[13783984.326062] LustreError: 127350:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 1 previous similar message
[13784026.624619] Lustre: oak-OST0117: Client 227acc10-fcf1-c357-f1da-048706fcd648 (at 10.50.5.63@o2ib2) reconnecting
[13784026.634942] Lustre: Skipped 25 previous similar messages
[13784067.870063] Lustre: oak-OST013d: Connection restored to a256e27b-c1dc-81ef-4609-ebeba7c6bf96 (at 10.51.4.43@o2ib3)
[13784067.880647] Lustre: Skipped 1270 previous similar messages
[13784104.018177] LustreError: 160917:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff9196f73ce850 x1723913652252224/t0(0) o4->227acc10-fcf1-c357-f1da-048706fcd648@10.50.5.63@o2ib2:50/0 lens 488/448 e 0 to 0 dl 1645009905 ref 1 fl Interpret:/0/0 rc 0/0
[13784104.018366] Lustre: oak-OST0131: Bulk IO write error with 227acc10-fcf1-c357-f1da-048706fcd648 (at 10.50.5.63@o2ib2), client will retry: rc = -110
[13784104.057147] LustreError: 160917:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[13784185.262406] Lustre: oak-OST0131: Client 227acc10-fcf1-c357-f1da-048706fcd648 (at 10.50.5.63@o2ib2) reconnecting
[13784185.272730] Lustre: Skipped 1 previous similar message
[13784367.418476] LustreError: 243448:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(393216)  req@ffff91cf23356850 x1723913653247104/t0(0) o3->227acc10-fcf1-c357-f1da-048706fcd648@10.50.5.63@o2ib2:312/0 lens 488/440 e 0 to 0 dl 1645010167 ref 1 fl Interpret:/0/0 rc 0/0
[13784367.418515] Lustre: oak-OST0129: Bulk IO read error with 227acc10-fcf1-c357-f1da-048706fcd648 (at 10.50.5.63@o2ib2), client will retry: rc -110
[13784367.418516] Lustre: Skipped 1 previous similar message
[13784367.461788] LustreError: 243448:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 1 previous similar message
[13784493.918578] Lustre: oak-OST013d: Client 227acc10-fcf1-c357-f1da-048706fcd648 (at 10.50.5.63@o2ib2) reconnecting
[13784493.928950] Lustre: Skipped 1 previous similar message
[13784667.023293] Lustre: oak-OST012b: Connection restored to  (at 10.51.4.58@o2ib3)
[13784667.030767] Lustre: Skipped 1553 previous similar messages
[13784989.008541] LustreError: 160919:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(4194304)  req@ffff91b87ea95050 x1724327871392256/t0(0) o3->18a389ad-f2fe-5ff3-1696-54295097e8af@10.50.7.39@o2ib2:170/0 lens 488/440 e 0 to 0 dl 1645010780 ref 1 fl Interpret:/0/0 rc 0/0
[13784989.033740] Lustre: oak-OST0133: Bulk IO read error with 18a389ad-f2fe-5ff3-1696-54295097e8af (at 10.50.7.39@o2ib2), client will retry: rc -110
[13784989.046841] Lustre: Skipped 1 previous similar message
[13785204.516834] LustreError: 160919:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91a4dac4a050 x1724103067349888/t0(0) o4->2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3@10.50.5.68@o2ib2:386/0 lens 488/448 e 0 to 0 dl 1645010996 ref 1 fl Interpret:/0/0 rc 0/0
[13785204.517127] Lustre: oak-OST0147: Bulk IO write error with 2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3 (at 10.50.5.68@o2ib2), client will retry: rc = -110
[13785204.517128] Lustre: Skipped 2 previous similar messages
[13785204.561702] LustreError: 160919:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 9 previous similar messages
[13785266.518199] Lustre: oak-OST0141: Connection restored to 7219babe-deb8-a72f-46cf-1134043e94ba (at 10.50.10.28@o2ib2)
[13785266.528864] Lustre: Skipped 1145 previous similar messages
[13785266.558757] Lustre: oak-OST0143: Client 2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3 (at 10.50.5.68@o2ib2) reconnecting
[13785266.569077] Lustre: Skipped 2 previous similar messages
[13785868.292020] Lustre: oak-OST014b: Connection restored to 2b65d4a4-af29-0b5e-1125-568e6f000e6d (at 10.50.12.9@o2ib2)
[13785868.302640] Lustre: Skipped 1040 previous similar messages
[13786449.686988] LustreError: 160906:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91d68988f050 x1722694727129088/t0(0) o4->c8035945-2d4b-8cc0-e678-4f1615763b9b@10.50.5.52@o2ib2:113/0 lens 488/448 e 0 to 0 dl 1645012233 ref 1 fl Interpret:/0/0 rc 0/0
[13786449.687071] Lustre: oak-OST0145: Bulk IO write error with c8035945-2d4b-8cc0-e678-4f1615763b9b (at 10.50.5.52@o2ib2), client will retry: rc = -110
[13786449.687072] Lustre: Skipped 8 previous similar messages
[13786449.731595] LustreError: 160906:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 4 previous similar messages
[13786467.293911] Lustre: oak-OST013f: Connection restored to 0243438e-369f-37cf-d78f-f51dc4026efb (at 10.50.9.22@o2ib2)
[13786467.304496] Lustre: Skipped 1158 previous similar messages
[13786494.001070] Lustre: oak-OST013f: Client c8035945-2d4b-8cc0-e678-4f1615763b9b (at 10.50.5.52@o2ib2) reconnecting
[13786494.011443] Lustre: Skipped 2 previous similar messages
[13787067.060381] Lustre: oak-OST0143: Connection restored to 1d15a02d-9b01-a082-cced-5d7728d12d67 (at 10.50.10.38@o2ib2)
[13787067.071090] Lustre: Skipped 1273 previous similar messages
[13787287.789227] LustreError: 160952:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(921600)  req@ffff919b87b62850 x1722694732525760/t0(0) o3->c8035945-2d4b-8cc0-e678-4f1615763b9b@10.50.5.52@o2ib2:200/0 lens 488/440 e 0 to 0 dl 1645013075 ref 1 fl Interpret:/0/0 rc 0/0
[13787287.814110] Lustre: oak-OST0125: Bulk IO read error with c8035945-2d4b-8cc0-e678-4f1615763b9b (at 10.50.5.52@o2ib2), client will retry: rc -110
[13787306.461317] LustreError: 137-5: oak-OST0140_UUID: not available for connect from 10.210.12.60@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13787400.949695] Lustre: oak-OST0111: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[13787400.960106] Lustre: Skipped 3 previous similar messages
[13787428.475495] Lustre: oak-OST0125: Client c8035945-2d4b-8cc0-e678-4f1615763b9b (at 10.50.5.52@o2ib2) reconnecting
[13787428.485816] Lustre: Skipped 17 previous similar messages
[13787668.724274] Lustre: oak-OST014b: Connection restored to 12262ab2-b081-58be-126e-52006b297891 (at 10.50.2.39@o2ib2)
[13787668.734927] Lustre: Skipped 1844 previous similar messages
[13788267.587991] Lustre: oak-OST0123: Connection restored to 97a324d4-c834-549a-4b22-6ab961bb5783 (at 10.0.3.86@o2ib5)
[13788267.598497] Lustre: Skipped 1537 previous similar messages
[13788868.161235] Lustre: oak-OST014b: Connection restored to 2d7b80e6-0f0d-a2ea-c50b-4f791ee979a6 (at 10.51.16.21@o2ib3)
[13788868.171905] Lustre: Skipped 1188 previous similar messages
[13789251.380387] LustreError: 160919:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff91bbfcc15050 x1715530475373888/t0(0) o3->5a5caa86-68df-f806-7e6f-05440e95a4ce@10.210.12.43@tcp1:680/0 lens 488/440 e 0 to 0 dl 1645015065 ref 1 fl Interpret:/0/0 rc 0/0
[13789251.380599] Lustre: oak-OST012d: Bulk IO read error with 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1), client will retry: rc -110
[13789251.419115] LustreError: 160919:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 1 previous similar message
[13789354.512237] Lustre: oak-OST0147: Client 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1) reconnecting
[13789354.522650] Lustre: Skipped 1 previous similar message
[13789376.475362] Lustre: oak-OST0127: Client 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1) reconnecting
[13789383.676098] Lustre: oak-OST011f: Client 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1) reconnecting
[13789383.686605] Lustre: Skipped 2 previous similar messages
[13789395.407591] Lustre: oak-OST012d: Client 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1) reconnecting
[13789418.470932] Lustre: oak-OST0121: Client 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1) reconnecting
[13789418.481343] Lustre: Skipped 10 previous similar messages
[13789470.594056] Lustre: oak-OST0127: Connection restored to 080fbf36-50e2-e037-d579-97f7ea4dd6a6 (at 10.210.12.51@tcp1)
[13789470.604745] Lustre: Skipped 1749 previous similar messages
[13790072.926047] Lustre: oak-OST012f: Connection restored to 12262ab2-b081-58be-126e-52006b297891 (at 10.50.2.39@o2ib2)
[13790072.936629] Lustre: Skipped 1284 previous similar messages
[13790672.642528] Lustre: oak-OST011d: Connection restored to 97a324d4-c834-549a-4b22-6ab961bb5783 (at 10.0.3.86@o2ib5)
[13790672.653061] Lustre: Skipped 1061 previous similar messages
[13790711.074406] LustreError: 243539:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91663b061850 x1722694755245504/t0(0) o4->c8035945-2d4b-8cc0-e678-4f1615763b9b@10.50.5.52@o2ib2:607/0 lens 488/448 e 0 to 0 dl 1645016502 ref 1 fl Interpret:/0/0 rc 0/0
[13790711.101236] Lustre: oak-OST013f: Bulk IO write error with c8035945-2d4b-8cc0-e678-4f1615763b9b (at 10.50.5.52@o2ib2), client will retry: rc = -110
[13790711.114592] Lustre: Skipped 4 previous similar messages
[13790778.655394] Lustre: oak-OST013f: Client c8035945-2d4b-8cc0-e678-4f1615763b9b (at 10.50.5.52@o2ib2) reconnecting
[13791046.319629] LustreError: 162683:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4111354)  req@ffff91489f027850 x1722694757727040/t0(0) o4->c8035945-2d4b-8cc0-e678-4f1615763b9b@10.50.5.52@o2ib2:201/0 lens 488/448 e 0 to 0 dl 1645016851 ref 1 fl Interpret:/0/0 rc 0/0
[13791046.319892] Lustre: oak-OST0147: Bulk IO write error with 2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3 (at 10.50.5.68@o2ib2), client will retry: rc = -110
[13791046.358697] LustreError: 162683:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 3 previous similar messages
[13791131.394386] Lustre: oak-OST0143: Client 2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3 (at 10.50.5.68@o2ib2) reconnecting
[13791131.404732] Lustre: Skipped 1 previous similar message
[13791132.018324] LustreError: 243538:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9163665bc850 x1724103090382720/t0(0) o4->2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3@10.50.5.68@o2ib2:336/0 lens 488/448 e 0 to 0 dl 1645016986 ref 1 fl Interpret:/0/0 rc 0/0
[13791132.034302] Lustre: oak-OST0143: Bulk IO read error with 2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3 (at 10.50.5.68@o2ib2), client will retry: rc -110
[13791132.034304] Lustre: Skipped 1 previous similar message
[13791132.036492] Lustre: oak-OST0147: Bulk IO write error with 2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3 (at 10.50.5.68@o2ib2), client will retry: rc = -110
[13791132.036493] Lustre: Skipped 3 previous similar messages
[13791132.080069] LustreError: 243538:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 3 previous similar messages
[13791141.255856] Lustre: oak-OST013f: Client c8035945-2d4b-8cc0-e678-4f1615763b9b (at 10.50.5.52@o2ib2) reconnecting
[13791271.882497] Lustre: oak-OST0135: Connection restored to 84f16600-a74a-8a3b-d476-2a157473f8c5 (at 10.50.1.22@o2ib2)
[13791271.893076] Lustre: Skipped 1246 previous similar messages
[13791390.550197] Lustre: oak-OST0141: Client 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1) reconnecting
[13791390.796370] LustreError: 199274:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91a0956af850 x1715756876122304/t0(0) o4->4b39aac6-2dc7-e48f-9434-e3bfde492453@10.210.12.29@tcp1:620/0 lens 488/448 e 0 to 0 dl 1645017270 ref 1 fl Interpret:/0/0 rc 0/0
[13791390.820977] Lustre: oak-OST0141: Bulk IO write error with 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1), client will retry: rc = -110
[13791390.834421] Lustre: Skipped 2 previous similar messages
[13791393.813037] LustreError: 243449:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91f18e3e1050 x1715756876122304/t0(0) o4->4b39aac6-2dc7-e48f-9434-e3bfde492453@10.210.12.29@tcp1:626/0 lens 488/448 e 0 to 0 dl 1645017276 ref 1 fl Interpret:/2/0 rc 0/0
[13791453.400812] LustreError: 160954:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(19857)  req@ffff91d88da85050 x1715184609023744/t0(0) o4->7189afb8-8fe5-7418-d14a-4a60d24330d0@10.210.12.48@tcp1:622/0 lens 488/448 e 0 to 0 dl 1645017272 ref 1 fl Interpret:/0/0 rc 0/0
[13791453.425935] Lustre: oak-OST0143: Bulk IO write error with 7189afb8-8fe5-7418-d14a-4a60d24330d0 (at 10.210.12.48@tcp1), client will retry: rc = -110
[13791453.439372] Lustre: Skipped 1 previous similar message
[13791471.691065] Lustre: oak-OST0143: Client 7189afb8-8fe5-7418-d14a-4a60d24330d0 (at 10.210.12.48@tcp1) reconnecting
[13791471.701473] Lustre: Skipped 1 previous similar message
[13791551.026155] Lustre: oak-OST014b: haven't heard from client bfaba9be-d132-07bf-7e04-96d9eca8997a (at 10.50.5.44@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91bc8ed95800, cur 1645017339 expire 1645017189 last 1645017112
[13791551.048061] Lustre: Skipped 14 previous similar messages
[13791553.025954] Lustre: oak-OST014d: haven't heard from client bfaba9be-d132-07bf-7e04-96d9eca8997a (at 10.50.5.44@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff916dc5f5f800, cur 1645017341 expire 1645017191 last 1645017114
[13791553.047925] Lustre: Skipped 22 previous similar messages
[13791553.692581] Lustre: oak-OST0115: Client 7189afb8-8fe5-7418-d14a-4a60d24330d0 (at 10.210.12.48@tcp1) reconnecting
[13791553.702989] Lustre: Skipped 1 previous similar message
[13791870.929581] Lustre: oak-OST0125: Connection restored to 62738b45-1a5c-dfeb-7614-6040117ce6fe (at 10.50.5.6@o2ib2)
[13791870.940076] Lustre: Skipped 1127 previous similar messages
[13792339.391115] LustreError: 160925:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 1365922(2414498)  req@ffff91bfb1986850 x1724103095213440/t0(0) o4->2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3@10.50.5.68@o2ib2:738/0 lens 488/448 e 0 to 0 dl 1645018143 ref 1 fl Interpret:/0/0 rc 0/0
[13792339.391162] Lustre: oak-OST0137: Bulk IO write error with 2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3 (at 10.50.5.68@o2ib2), client will retry: rc = -110
[13792339.430195] LustreError: 160925:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 8 previous similar messages
[13792410.717126] Lustre: oak-OST012f: Client 2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3 (at 10.50.5.68@o2ib2) reconnecting
[13792410.727464] Lustre: Skipped 40 previous similar messages
[13792470.000598] Lustre: oak-OST0145: Connection restored to 2a3ca72c-b1c9-605d-617d-52de1fac6027 (at 10.51.14.5@o2ib3)
[13792470.011183] Lustre: Skipped 1017 previous similar messages
[13792650.675421] LustreError: 162682:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(1011712)  req@ffff9180c387e850 x1724103096744128/t0(0) o3->2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3@10.50.5.68@o2ib2:293/0 lens 488/440 e 0 to 0 dl 1645018453 ref 1 fl Interpret:/0/0 rc 0/0
[13792650.700399] Lustre: oak-OST0131: Bulk IO read error with 2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3 (at 10.50.5.68@o2ib2), client will retry: rc -110
[13792674.619383] LustreError: 243499:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(814982)  req@ffff9189f0a9a050 x1724103096920256/t0(0) o4->2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3@10.50.5.68@o2ib2:319/0 lens 488/448 e 0 to 0 dl 1645018479 ref 1 fl Interpret:/0/0 rc 0/0
[13792674.619416] Lustre: oak-OST0147: Bulk IO write error with 2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3 (at 10.50.5.68@o2ib2), client will retry: rc = -110
[13792674.619417] Lustre: Skipped 8 previous similar messages
[13792674.663303] LustreError: 243499:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 15 previous similar messages
[13792713.367425] Lustre: oak-OST0131: Client 2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3 (at 10.50.5.68@o2ib2) reconnecting
[13792713.377783] Lustre: Skipped 3 previous similar messages
[13792763.172386] Lustre: oak-OST0143: Client 2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3 (at 10.50.5.68@o2ib2) reconnecting
[13792763.182705] Lustre: Skipped 4 previous similar messages
[13793068.730964] Lustre: oak-OST013d: Connection restored to e9ec1c43-6e89-086b-9840-3cb159bbec79 (at 10.50.5.65@o2ib2)
[13793068.741550] Lustre: Skipped 1204 previous similar messages
[13793667.686191] Lustre: oak-OST0111: Connection restored to eea359e6-6b89-c326-60af-8cb99076065b (at 10.210.12.8@tcp1)
[13793667.696774] Lustre: Skipped 978 previous similar messages
[13794266.917697] Lustre: oak-OST0139: Connection restored to b1d9b878-ebb6-1c35-53c8-a58050157bbe (at 10.50.6.64@o2ib2)
[13794266.928293] Lustre: Skipped 1098 previous similar messages
[13794781.906711] LustreError: 243440:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff915df8930850 x1724103108696064/t0(0) o4->2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3@10.50.5.68@o2ib2:174/0 lens 488/448 e 0 to 0 dl 1645020599 ref 1 fl Interpret:/0/0 rc 0/0
[13794781.906747] LustreError: 229136:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(57344)  req@ffff916611442850 x1724103108731904/t0(0) o3->2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3@10.50.5.68@o2ib2:174/0 lens 488/440 e 0 to 0 dl 1645020599 ref 1 fl Interpret:/0/0 rc 0/0
[13794781.906760] Lustre: oak-OST012d: Bulk IO read error with 2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3 (at 10.50.5.68@o2ib2), client will retry: rc -110
[13794781.907319] Lustre: oak-OST0143: Bulk IO write error with 2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3 (at 10.50.5.68@o2ib2), client will retry: rc = -110
[13794781.907320] Lustre: Skipped 15 previous similar messages
[13794781.989283] LustreError: 243440:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 8 previous similar messages
[13794850.189862] Lustre: oak-OST012d: Client 2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3 (at 10.50.5.68@o2ib2) reconnecting
[13794850.200176] Lustre: Skipped 1 previous similar message
[13794866.199434] Lustre: oak-OST0133: Connection restored to dd5ea95c-ee13-13f4-fe1e-689d46c56db7 (at 10.50.1.67@o2ib2)
[13794866.210021] Lustre: Skipped 1211 previous similar messages
[13795464.860473] Lustre: oak-OST0115: Connection restored to 834d8ea1-98bb-d04d-e595-6857c1f41a64 (at 10.50.14.11@o2ib2)
[13795464.871144] Lustre: Skipped 897 previous similar messages
[13796063.944644] Lustre: oak-OST0149: Connection restored to 48e92354-7d5f-8751-64ca-0c3985569306 (at 10.50.8.68@o2ib2)
[13796063.955233] Lustre: Skipped 1669 previous similar messages
[13796663.995542] Lustre: oak-OST012b: Connection restored to d34cc33f-555c-1a6e-a9d6-bc8c27f9c288 (at 10.50.6.67@o2ib2)
[13796664.006138] Lustre: Skipped 1739 previous similar messages
[13797263.400482] Lustre: oak-OST0125: Connection restored to b96ecd72-6c65-48df-36c1-3e0f835dea63 (at 10.51.14.13@o2ib3)
[13797263.411203] Lustre: Skipped 1373 previous similar messages
[13797364.400493] LustreError: 137-5: oak-OST011c_UUID: not available for connect from 10.210.12.59@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13797459.213233] Lustre: oak-OST011b: Client 5d4d5527-a52f-bc68-278b-aa990b8609d6 (at 10.210.12.59@tcp1) reconnecting
[13797459.223651] Lustre: Skipped 1 previous similar message
[13797461.617278] Lustre: oak-OST0117: Client 5d4d5527-a52f-bc68-278b-aa990b8609d6 (at 10.210.12.59@tcp1) reconnecting
[13797465.615052] Lustre: oak-OST0137: Client 5d4d5527-a52f-bc68-278b-aa990b8609d6 (at 10.210.12.59@tcp1) reconnecting
[13797862.238726] Lustre: oak-OST0145: Connection restored to 97a324d4-c834-549a-4b22-6ab961bb5783 (at 10.0.3.86@o2ib5)
[13797862.249217] Lustre: Skipped 1314 previous similar messages
[13798061.607658] LustreError: 243441:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(1002929)  req@ffff91acdb614850 x1714941875492480/t0(0) o4->5704685e-b0a9-b5c3-be3f-b278a4395bec@10.210.12.72@tcp1:466/0 lens 488/448 e 0 to 0 dl 1645023911 ref 1 fl Interpret:/0/0 rc 0/0
[13798061.633071] Lustre: oak-OST0121: Bulk IO write error with 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1), client will retry: rc = -110
[13798061.646508] Lustre: Skipped 8 previous similar messages
[13798085.552474] LustreError: 21595:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(2806566)  req@ffff915bc311e850 x1714941875504064/t0(0) o4->5704685e-b0a9-b5c3-be3f-b278a4395bec@10.210.12.72@tcp1:471/0 lens 504/448 e 0 to 0 dl 1645023916 ref 1 fl Interpret:/0/0 rc 0/0
[13798085.552476] LustreError: 21593:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(181706)  req@ffff9143fecef850 x1714941875503872/t0(0) o4->5704685e-b0a9-b5c3-be3f-b278a4395bec@10.210.12.72@tcp1:471/0 lens 488/448 e 0 to 0 dl 1645023916 ref 1 fl Interpret:/0/0 rc 0/0
[13798085.552483] LustreError: 162690:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff91663b063850 x1714941875544704/t0(0) o3->5704685e-b0a9-b5c3-be3f-b278a4395bec@10.210.12.72@tcp1:472/0 lens 488/440 e 0 to 0 dl 1645023917 ref 1 fl Interpret:/0/0 rc 0/0
[13798085.552485] LustreError: 162690:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 1 previous similar message
[13798085.552514] Lustre: oak-OST014d: Bulk IO write error with 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1), client will retry: rc = -110
[13798085.552637] Lustre: oak-OST0139: Bulk IO read error with 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1), client will retry: rc -110
[13798085.552638] Lustre: Skipped 1 previous similar message
[13798094.796133] LustreError: 137-5: oak-OST011a_UUID: not available for connect from 10.210.12.72@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13798094.802008] Lustre: oak-OST014d: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[13798094.802009] Lustre: Skipped 21 previous similar messages
[13798094.829785] LustreError: Skipped 1 previous similar message
[13798187.149167] Lustre: oak-OST011b: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[13798187.159594] Lustre: Skipped 5 previous similar messages
[13798189.330844] Lustre: oak-OST0139: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[13798196.328495] Lustre: oak-OST012d: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[13798196.338900] Lustre: Skipped 1 previous similar message
[13798213.581621] Lustre: oak-OST0137: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[13798213.592043] Lustre: Skipped 15 previous similar messages
[13798230.469436] Lustre: oak-OST0111: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[13798230.479845] Lustre: Skipped 3 previous similar messages
[13798263.552174] Lustre: oak-OST011b: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[13798263.562582] Lustre: Skipped 9 previous similar messages
[13798324.457312] Lustre: oak-OST0145: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[13798324.467714] Lustre: Skipped 9 previous similar messages
[13798462.939841] Lustre: oak-OST014b: Connection restored to 99b5b5a5-0765-c613-dfd8-cfcc7cda569c (at 10.50.4.4@o2ib2)
[13798462.950353] Lustre: Skipped 1353 previous similar messages
[13798779.993936] LustreError: 228831:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(263335)  req@ffff91d8fec4e850 x1724103129850752/t0(0) o4->2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3@10.50.5.68@o2ib2:404/0 lens 488/448 e 0 to 0 dl 1645024604 ref 1 fl Interpret:/0/0 rc 0/0
[13798780.019076] Lustre: oak-OST0113: Bulk IO write error with 2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3 (at 10.50.5.68@o2ib2), client will retry: rc = -110
[13798780.032425] Lustre: Skipped 1 previous similar message
[13798818.616515] Lustre: oak-OST0113: Client 2ce9b759-f537-ed4a-f35e-dc1f8e48f8e3 (at 10.50.5.68@o2ib2) reconnecting
[13798818.626879] Lustre: Skipped 4 previous similar messages
[13799061.535559] Lustre: oak-OST0145: Connection restored to  (at 10.50.10.51@o2ib2)
[13799061.543129] Lustre: Skipped 1293 previous similar messages
[13799661.826144] Lustre: oak-OST0135: Connection restored to 2dc703cd-5cca-815b-6dd4-cd3e7d5a0392 (at 10.0.3.49@o2ib5)
[13799661.836695] Lustre: Skipped 1031 previous similar messages
[13800262.952923] Lustre: oak-OST012b: Connection restored to  (at 10.50.3.69@o2ib2)
[13800262.960420] Lustre: Skipped 778 previous similar messages
[13800864.252439] Lustre: oak-OST0133: Connection restored to abf91aa8-29cb-106f-f702-19731db8361c (at 10.50.5.12@o2ib2)
[13800864.263125] Lustre: Skipped 505 previous similar messages
[13800967.178102] Lustre: oak-OST0137: Client 1b588188-9255-1b9f-1308-154ba9e467a3 (at 10.210.12.7@tcp1) reconnecting
[13801293.375957] LustreError: 160955:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91c193af3850 x1724027176036544/t0(0) o4->7caa6746-008a-4c85-9dca-b006031ecd37@10.50.5.30@o2ib2:659/0 lens 488/448 e 0 to 0 dl 1645027124 ref 1 fl Interpret:/0/0 rc 0/0
[13801293.375974] LustreError: 248297:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(1298432)  req@ffff91e4f33b5050 x1724027176038080/t0(0) o3->7caa6746-008a-4c85-9dca-b006031ecd37@10.50.5.30@o2ib2:659/0 lens 488/440 e 0 to 0 dl 1645027124 ref 1 fl Interpret:/0/0 rc 0/0
[13801293.376019] Lustre: oak-OST013b: Bulk IO read error with 7caa6746-008a-4c85-9dca-b006031ecd37 (at 10.50.5.30@o2ib2), client will retry: rc -110
[13801293.376244] Lustre: oak-OST0141: Bulk IO write error with 7caa6746-008a-4c85-9dca-b006031ecd37 (at 10.50.5.30@o2ib2), client will retry: rc = -110
[13801293.453162] LustreError: 160955:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 5 previous similar messages
[13801367.997615] Lustre: oak-OST013b: Client 7caa6746-008a-4c85-9dca-b006031ecd37 (at 10.50.5.30@o2ib2) reconnecting
[13801368.007946] Lustre: Skipped 25 previous similar messages
[13801465.375505] Lustre: oak-OST0149: Connection restored to 7069b57c-6d76-196c-bdfd-7eeec28ffa4d (at 10.50.2.44@o2ib2)
[13801465.386883] Lustre: Skipped 870 previous similar messages
[13802064.295794] Lustre: oak-OST0137: Connection restored to 46d0ef9e-fc15-28ca-cf22-7659399b7268 (at 10.50.8.25@o2ib2)
[13802064.306383] Lustre: Skipped 520 previous similar messages
[13802664.190645] Lustre: oak-OST0129: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[13802664.201316] Lustre: Skipped 1050 previous similar messages
[13803263.988728] Lustre: oak-OST0133: Connection restored to 49bc28ad-d0f8-4839-b977-ca940f844247 (at 10.50.4.17@o2ib2)
[13803263.999309] Lustre: Skipped 1283 previous similar messages
[13803274.589303] Lustre: oak-OST011b: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[13803274.599739] Lustre: Skipped 12 previous similar messages
[13803279.583128] Lustre: oak-OST0113: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[13803279.593547] Lustre: Skipped 17 previous similar messages
[13803289.662414] Lustre: oak-OST0137: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[13803289.672826] Lustre: Skipped 14 previous similar messages
[13803309.158704] Lustre: oak-OST0133: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[13803309.169110] Lustre: Skipped 4 previous similar messages
[13803441.360251] Lustre: oak-OST0121: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[13803441.370662] Lustre: Skipped 1 previous similar message
[13803863.532837] Lustre: oak-OST012f: Connection restored to 75c1d43b-dfae-99c8-7312-dd62033906d8 (at 10.50.8.41@o2ib2)
[13803863.543677] Lustre: Skipped 2174 previous similar messages
[13804182.757112] Lustre: oak-OST0119: Client 5d8bb8d9-e5c7-334f-ec8d-f10452c309b5 (at 10.210.12.66@tcp1) reconnecting
[13804183.211461] LustreError: 160903:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91b8119db050 x1715088710270528/t0(0) o4->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:613/0 lens 488/448 e 0 to 0 dl 1645030098 ref 1 fl Interpret:/0/0 rc 0/0
[13804183.236027] Lustre: oak-OST0119: Bulk IO write error with 5d8bb8d9-e5c7-334f-ec8d-f10452c309b5 (at 10.210.12.66@tcp1), client will retry: rc = -110
[13804183.249467] Lustre: Skipped 5 previous similar messages
[13804349.980327] Lustre: oak-OST0113: Client 5d8bb8d9-e5c7-334f-ec8d-f10452c309b5 (at 10.210.12.66@tcp1) reconnecting
[13804349.990844] Lustre: Skipped 1 previous similar message
[13804371.147560] Lustre: oak-OST0131: Client 5d8bb8d9-e5c7-334f-ec8d-f10452c309b5 (at 10.210.12.66@tcp1) reconnecting
[13804371.158005] Lustre: Skipped 22 previous similar messages
[13804462.626366] Lustre: oak-OST013f: Connection restored to 767b7c94-5f6a-643b-d3fc-ff266d4e288d (at 10.50.4.60@o2ib2)
[13804462.636956] Lustre: Skipped 1511 previous similar messages
[13804508.006045] LustreError: 199244:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 99s: evicting client at 10.210.12.63@tcp1  ns: filter-oak-OST0147_UUID lock: ffff913aa14d1440/0xed112d3019ad74bd lrc: 3/0,0 mode: PR/PR res: [0x5580000402:0x37d57a:0x0].0x0 rrc: 253 type: EXT [0->21224767487] (req 21222662144->21222678527) flags: 0x60000400020020 nid: 10.210.12.63@tcp1 remote: 0x58d476fa2c76dc expref: 429 pid: 228745 timeout: 13838060 lvb_type: 0
[13804508.093096] LustreError: 131093:0:(ldlm_lockd.c:2366:ldlm_cancel_handler()) ldlm_cancel from 10.210.12.63@tcp1 arrived at 1645030327 with bad export cookie 17082484546085592182
[13804994.366197] Lustre: oak-OST012d: Client e44e675f-11ba-536d-fbc7-173bafd860e2 (at 10.210.12.46@tcp1) reconnecting
[13804994.376622] Lustre: Skipped 4 previous similar messages
[13805063.197321] Lustre: oak-OST0125: Connection restored to  (at 10.51.15.22@o2ib3)
[13805063.204982] Lustre: Skipped 1183 previous similar messages
[13805270.995123] Lustre: oak-OST013b: Client 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1) reconnecting
[13805271.005631] Lustre: Skipped 24 previous similar messages
[13805271.018695] LustreError: 253953:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91b5e274d850 x1714947747075072/t0(0) o4->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:190/0 lens 488/448 e 0 to 0 dl 1645031185 ref 1 fl Interpret:/0/0 rc 0/0
[13805271.043215] Lustre: oak-OST013b: Bulk IO write error with 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1), client will retry: rc = -110
[13805293.708514] LustreError: 162670:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91381860a850 x1715239030593536/t0(0) o4->e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda@10.210.12.79@tcp1:212/0 lens 488/448 e 0 to 0 dl 1645031207 ref 1 fl Interpret:/0/0 rc 0/0
[13805293.733027] Lustre: oak-OST0145: Bulk IO write error with e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1), client will retry: rc = -110
[13805436.144524] LustreError: 162678:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(46666)  req@ffff9191a7b13050 x1722686150244288/t0(0) o4->ca3e534b-7250-7951-d417-244820eb007a@10.50.5.2@o2ib2:286/0 lens 488/448 e 0 to 0 dl 1645031281 ref 1 fl Interpret:/0/0 rc 0/0
[13805436.171611] Lustre: oak-OST013f: Bulk IO write error with ca3e534b-7250-7951-d417-244820eb007a (at 10.50.5.2@o2ib2), client will retry: rc = -110
[13805438.407035] Lustre: oak-OST0121: Client 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1) reconnecting
[13805438.417462] Lustre: Skipped 1 previous similar message
[13805661.858227] Lustre: oak-OST012b: Connection restored to 5baa1c8e-b648-2f61-3664-815e43d5004e (at 10.51.2.55@o2ib3)
[13805661.868954] Lustre: Skipped 954 previous similar messages
[13805994.358974] Lustre: 203287:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1645031689/real 1645031689]  req@ffff9151be02b180 x1710530325194304/t0(0) o106->oak-OST0133@10.51.16.2@o2ib3:15/16 lens 296/280 e 0 to 1 dl 1645031817 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
[13806066.330089] LNet: Service thread pid 203287 was inactive for 200.15s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
[13806066.347359] Pid: 203287, comm: ll_ost00_008 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[13806066.358215] Call Trace:
[13806066.361069]  [<ffffffffc1072310>] ptlrpc_set_wait+0x480/0x790 [ptlrpc]
[13806066.368001]  [<ffffffffc102f145>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc]
[13806066.374952]  [<ffffffffc105054b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc]
[13806066.381949]  [<ffffffffc168762b>] ofd_intent_policy+0x69b/0x920 [ofd]
[13806066.388779]  [<ffffffffc10304e6>] ldlm_lock_enqueue+0x376/0x9b0 [ptlrpc]
[13806066.395805]  [<ffffffffc1057a16>] ldlm_handle_enqueue0+0xa86/0x1620 [ptlrpc]
[13806066.403246]  [<ffffffffc10e0a42>] tgt_enqueue+0x62/0x210 [ptlrpc]
[13806066.409663]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[13806066.416902]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[13806066.424898]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[13806066.431457]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[13806066.436612]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[13806066.443318]  [<ffffffffffffffff>] 0xffffffffffffffff
[13806066.448588] LustreError: dumping log to /tmp/lustre-log.1645031889.203287
[13806072.732498] Lustre: oak-OST0149: haven't heard from client 29743f67-a230-24d5-35bc-0a994d743054 (at 10.51.16.2@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91c2f513b800, cur 1645031896 expire 1645031746 last 1645031669
[13806072.787481] LNet: Service thread pid 203287 completed after 206.62s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
[13806083.717446] Lustre: oak-OST0127: haven't heard from client cf6a0896-286b-e9cb-444b-9afffbaff458 (at 10.51.14.15@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91b351170c00, cur 1645031907 expire 1645031757 last 1645031680
[13806083.739582] Lustre: Skipped 17 previous similar messages
[13806084.755359] Lustre: oak-OST013b: haven't heard from client 29743f67-a230-24d5-35bc-0a994d743054 (at 10.51.16.2@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91dc942d7000, cur 1645031908 expire 1645031758 last 1645031681
[13806094.676450] Lustre: oak-OST013f: haven't heard from client cf6a0896-286b-e9cb-444b-9afffbaff458 (at 10.51.14.15@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91df24531000, cur 1645031918 expire 1645031768 last 1645031691
[13806094.698486] Lustre: Skipped 4 previous similar messages
[13806222.504886] Lustre: oak-OST0113: Client 1b0e0de8-4056-d02c-bb09-33adedaa9f96 (at 10.210.12.45@tcp1) reconnecting
[13806222.515300] Lustre: Skipped 73 previous similar messages
[13806260.840410] Lustre: oak-OST0139: Connection restored to 8a175aee-31b6-e168-94c2-4812b701af81 (at 10.51.2.60@o2ib3)
[13806260.851003] Lustre: Skipped 1203 previous similar messages
[13806369.036485] LustreError: 243535:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff918ed857c850 x1722773176101248/t0(0) o4->92c18d6d-606b-b81c-1d4b-8b0d45a7564c@10.50.8.18@o2ib2:449/0 lens 488/448 e 0 to 0 dl 1645032199 ref 1 fl Interpret:/0/0 rc 0/0
[13806369.036695] Lustre: oak-OST0141: Bulk IO write error with 92c18d6d-606b-b81c-1d4b-8b0d45a7564c (at 10.50.8.18@o2ib2), client will retry: rc = -110
[13806369.075558] LustreError: 243535:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 3 previous similar messages
[13806859.901567] Lustre: oak-OST013d: Connection restored to cfe5de08-fee7-14fe-a353-854c585fdc0a (at 10.51.2.45@o2ib3)
[13806859.912165] Lustre: Skipped 881 previous similar messages
[13807207.192562] LustreError: 162711:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 818633(2915785)  req@ffff91759474e850 x1722771297923648/t0(0) o4->b2583baf-fcfe-cf54-b945-b20905221dcd@10.50.10.7@o2ib2:547/0 lens 488/448 e 0 to 0 dl 1645033052 ref 1 fl Interpret:/0/0 rc 0/0
[13807207.193519] Lustre: oak-OST0139: Bulk IO write error with b2583baf-fcfe-cf54-b945-b20905221dcd (at 10.50.10.7@o2ib2), client will retry: rc = -110
[13807207.193521] Lustre: Skipped 3 previous similar messages
[13807207.237065] LustreError: 162711:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 4 previous similar messages
[13807256.044406] Lustre: oak-OST0149: Client 4dc75b54-bcbd-c34d-fadc-fa8df93dcbee (at 10.50.5.45@o2ib2) reconnecting
[13807256.054737] Lustre: Skipped 92 previous similar messages
[13807458.448614] Lustre: oak-OST0147: Connection restored to b721998c-1a52-1c1c-cbd9-f9819458c553 (at 10.210.12.131@tcp1)
[13807458.459374] Lustre: Skipped 1109 previous similar messages
[13808057.365997] Lustre: oak-OST0125: Connection restored to 83e2592a-45e0-f338-d13b-f2cbdfe13bd7 (at 10.50.8.27@o2ib2)
[13808057.376580] Lustre: Skipped 858 previous similar messages
[13808081.225379] Lustre: oak-OST0131: Client 6d01c866-9ec9-8f84-4f86-ee1db83afc97 (at 10.210.12.61@tcp1) reconnecting
[13808081.235784] Lustre: Skipped 46 previous similar messages
[13808656.352610] Lustre: oak-OST0147: Connection restored to ea89e84f-b1fc-a3e1-8980-4ff1b7f131ae (at 10.50.12.10@o2ib2)
[13808656.363326] Lustre: Skipped 836 previous similar messages
[13808673.410423] Lustre: oak-OST014b: haven't heard from client f03a4f44-4fbb-e833-8ac5-ad8fc4cf881d (at 10.50.3.25@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff918a970b5400, cur 1645034503 expire 1645034353 last 1645034276
[13808673.432371] Lustre: Skipped 6 previous similar messages
[13808684.437067] Lustre: oak-OST0117: haven't heard from client f03a4f44-4fbb-e833-8ac5-ad8fc4cf881d (at 10.50.3.25@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91e89754e800, cur 1645034514 expire 1645034364 last 1645034287
[13808684.459001] Lustre: Skipped 3 previous similar messages
[13808787.689490] LustreError: 21611:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4038656)  req@ffff9177087b0050 x1715756978947840/t0(0) o4->4b39aac6-2dc7-e48f-9434-e3bfde492453@10.210.12.29@tcp1:636/0 lens 488/448 e 0 to 0 dl 1645034651 ref 1 fl Interpret:/0/0 rc 0/0
[13808787.689744] Lustre: oak-OST0145: Bulk IO write error with 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1), client will retry: rc = -110
[13808787.689745] Lustre: Skipped 4 previous similar messages
[13808787.734120] LustreError: 21611:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[13808807.217441] LustreError: 137-5: oak-OST0134_UUID: not available for connect from 10.210.12.54@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13808809.061822] Lustre: oak-OST0145: Client 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1) reconnecting
[13808809.072233] Lustre: Skipped 47 previous similar messages
[13809127.605471] LustreError: 243555:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff915a327ae850 x1722574414379968/t0(0) o4->a8a647a5-bb36-3966-6637-e8ee81d6d548@10.210.12.6@tcp1:281/0 lens 488/448 e 0 to 0 dl 1645035051 ref 1 fl Interpret:/0/0 rc 0/0
[13809127.630072] Lustre: oak-OST013b: Bulk IO write error with a8a647a5-bb36-3966-6637-e8ee81d6d548 (at 10.210.12.6@tcp1), client will retry: rc = -110
[13809127.643421] Lustre: Skipped 1 previous similar message
[13809128.626983] LustreError: 21596:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff916be233b050 x1722574414379968/t0(0) o4->a8a647a5-bb36-3966-6637-e8ee81d6d548@10.210.12.6@tcp1:285/0 lens 488/448 e 0 to 0 dl 1645035055 ref 1 fl Interpret:/2/0 rc 0/0
[13809128.651268] LustreError: 21596:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[13809128.661056] Lustre: oak-OST013b: Bulk IO write error with a8a647a5-bb36-3966-6637-e8ee81d6d548 (at 10.210.12.6@tcp1), client will retry: rc = -110
[13809128.674411] Lustre: Skipped 1 previous similar message
[13809256.591080] Lustre: oak-OST014b: Connection restored to d16cb253-11dc-02a5-01bc-e4ca96f3f9b7 (at 10.50.12.8@o2ib2)
[13809256.601663] Lustre: Skipped 797 previous similar messages
[13809855.334785] Lustre: oak-OST014b: Connection restored to 1e9fcc5c-d5a4-fea5-4827-43b86927ab1e (at 10.50.2.36@o2ib2)
[13809855.345362] Lustre: Skipped 831 previous similar messages
[13810069.084750] Lustre: oak-OST011f: Client 80374d70-7d99-f68a-b58b-86c0b7514d6a (at 10.210.12.65@tcp1) reconnecting
[13810069.095173] Lustre: Skipped 69 previous similar messages
[13810125.649740] LustreError: 137-5: oak-OST0140_UUID: not available for connect from 10.210.12.38@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13810176.034499] Lustre: oak-OST0111: Client a8a647a5-bb36-3966-6637-e8ee81d6d548 (at 10.210.12.6@tcp1) reconnecting
[13810176.044820] Lustre: Skipped 45 previous similar messages
[13810176.750747] LustreError: 244101:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91d8ff9ba850 x1722574619368704/t0(0) o4->a8a647a5-bb36-3966-6637-e8ee81d6d548@10.210.12.6@tcp1:576/0 lens 504/448 e 0 to 0 dl 1645036101 ref 1 fl Interpret:/0/0 rc 0/0
[13810176.751939] Lustre: oak-OST0111: Bulk IO write error with a8a647a5-bb36-3966-6637-e8ee81d6d548 (at 10.210.12.6@tcp1), client will retry: rc = -110
[13810176.751940] Lustre: Skipped 1 previous similar message
[13810176.793805] LustreError: 244101:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 2 previous similar messages
[13810342.843134] Lustre: oak-OST011d: Client a8a647a5-bb36-3966-6637-e8ee81d6d548 (at 10.210.12.6@tcp1) reconnecting
[13810342.853457] Lustre: Skipped 79 previous similar messages
[13810400.715174] LustreError: 162680:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk READ failed: rc -107  req@ffff914246a9f050 x1722574655278400/t0(0) o3->a8a647a5-bb36-3966-6637-e8ee81d6d548@10.210.12.6@tcp1:52/0 lens 488/440 e 0 to 0 dl 1645036332 ref 1 fl Interpret:/0/0 rc 0/0
[13810400.739616] Lustre: oak-OST014b: Bulk IO read error with a8a647a5-bb36-3966-6637-e8ee81d6d548 (at 10.210.12.6@tcp1), client will retry: rc -107
[13810454.273198] Lustre: oak-OST0135: Connection restored to e64c41d9-97dc-d5e1-1051-e238ba99ebd4 (at 10.50.2.28@o2ib2)
[13810454.283782] Lustre: Skipped 1420 previous similar messages
[13810662.686153] Lustre: oak-OST0127: Client 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1) reconnecting
[13810662.696555] Lustre: Skipped 24 previous similar messages
[13811054.711043] Lustre: oak-OST0113: Connection restored to ec6cafa7-2c96-71e9-0dad-24d0eee2b247 (at 10.0.3.37@o2ib5)
[13811054.721612] Lustre: Skipped 700 previous similar messages
[13811653.378728] Lustre: oak-OST011d: Connection restored to 97a324d4-c834-549a-4b22-6ab961bb5783 (at 10.0.3.86@o2ib5)
[13811653.389254] Lustre: Skipped 673 previous similar messages
[13812252.452843] Lustre: oak-OST013b: Connection restored to  (at 10.51.15.9@o2ib3)
[13812252.460315] Lustre: Skipped 704 previous similar messages
[13812570.701545] Lustre: oak-OST0121: Client 9fc64b77-e33e-204d-02bc-73983c39db00 (at 10.50.17.21@o2ib2) reconnecting
[13812570.711960] Lustre: Skipped 51 previous similar messages
[13812778.463229] Lustre: oak-OST0117: Client 6d01c866-9ec9-8f84-4f86-ee1db83afc97 (at 10.210.12.61@tcp1) reconnecting
[13812778.473664] Lustre: Skipped 4 previous similar messages
[13812858.250060] Lustre: oak-OST0129: Connection restored to 74b01e71-a95e-02b5-4cdd-9715608dcb26 (at 10.50.5.52@o2ib2)
[13812858.260664] Lustre: Skipped 750 previous similar messages
[13813461.491652] Lustre: oak-OST013f: Connection restored to a24dd7a0-5fa8-502a-258a-91fb9f94b4c0 (at 10.50.5.43@o2ib2)
[13813461.502234] Lustre: Skipped 888 previous similar messages
[13813993.659045] Lustre: oak-OST0125: Client 9fc64b77-e33e-204d-02bc-73983c39db00 (at 10.50.17.21@o2ib2) reconnecting
[13813993.669449] Lustre: Skipped 20 previous similar messages
[13814060.199326] Lustre: oak-OST012d: Connection restored to 13782cfa-f64e-3b30-9b19-eac54c74cba1 (at 10.51.15.10@o2ib3)
[13814060.210095] Lustre: Skipped 1065 previous similar messages
[13814317.017094] Lustre: oak-OST0113: Client 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1) reconnecting
[13814317.027536] Lustre: Skipped 15 previous similar messages
[13814318.352597] LustreError: 243445:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(22843)  req@ffff9138b2392050 x1714998948205376/t0(0) o4->6f45695d-f173-ee1d-e6cb-d38dad7e0879@10.210.12.74@tcp1:134/0 lens 488/448 e 0 to 0 dl 1645040189 ref 1 fl Interpret:/0/0 rc 0/0
[13814318.377763] Lustre: oak-OST012d: Bulk IO write error with 6f45695d-f173-ee1d-e6cb-d38dad7e0879 (at 10.210.12.74@tcp1), client will retry: rc = -110
[13814318.391296] Lustre: Skipped 1 previous similar message
[13814396.589272] Lustre: oak-OST0113: Client 94d559ad-9ea8-2fc9-41c4-60f721062c9a (at 10.210.12.52@tcp1) reconnecting
[13814396.599682] Lustre: Skipped 20 previous similar messages
[13814660.368283] Lustre: oak-OST0123: Connection restored to 1762a9b8-4da9-4e57-339b-f4fe97cc50d0 (at 10.51.1.52@o2ib3)
[13814660.378862] Lustre: Skipped 1057 previous similar messages
[13815259.247543] Lustre: oak-OST0117: Connection restored to 8513533f-cfcb-802b-01a1-cfb62494368c (at 10.50.9.25@o2ib2)
[13815259.258210] Lustre: Skipped 941 previous similar messages
[13815858.875848] Lustre: oak-OST0137: Connection restored to de334834-ada5-7101-0af8-a85e255c7a80 (at 10.50.7.31@o2ib2)
[13815858.886443] Lustre: Skipped 902 previous similar messages
[13816459.628474] Lustre: oak-OST0131: Connection restored to e64c41d9-97dc-d5e1-1051-e238ba99ebd4 (at 10.50.2.28@o2ib2)
[13816459.639123] Lustre: Skipped 753 previous similar messages
[13817058.908791] Lustre: oak-OST0123: Connection restored to d2f49117-87f4-d939-d915-51fa6430aa6e (at 10.51.12.14@o2ib3)
[13817058.919461] Lustre: Skipped 816 previous similar messages
[13817657.776007] Lustre: oak-OST0129: Connection restored to 49044c4e-6a2f-ee70-77e0-627b9daaf804 (at 10.50.2.68@o2ib2)
[13817657.786748] Lustre: Skipped 1296 previous similar messages
[13818257.105037] Lustre: oak-OST013f: Connection restored to 15431f40-18fe-801a-ce02-67f7cb5f3e18 (at 10.210.12.60@tcp1)
[13818257.115763] Lustre: Skipped 969 previous similar messages
[13818858.329257] Lustre: oak-OST0123: Connection restored to  (at 10.51.5.16@o2ib3)
[13818858.336772] Lustre: Skipped 1536 previous similar messages
[13819457.452273] Lustre: oak-OST011f: Connection restored to dfba191f-99e8-9503-e2b2-904454040fbb (at 10.50.5.34@o2ib2)
[13819457.463101] Lustre: Skipped 945 previous similar messages
[13820040.512843] LustreError: 243540:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff918f7a141050 x1715105184326336/t0(0) o4->b4fca75a-2d54-9bf2-331f-67d0c52e5ff2@10.210.12.73@tcp1:603/0 lens 488/448 e 0 to 0 dl 1645045943 ref 1 fl Interpret:/0/0 rc 0/0
[13820040.514847] Lustre: oak-OST013f: Bulk IO write error with b4fca75a-2d54-9bf2-331f-67d0c52e5ff2 (at 10.210.12.73@tcp1), client will retry: rc = -110
[13820040.552110] LustreError: 243540:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 6 previous similar messages
[13820056.193116] Lustre: oak-OST0143: Connection restored to c5155f02-0119-bfc6-a25e-a47ff9754d2f (at 10.50.5.39@o2ib2)
[13820056.203695] Lustre: Skipped 910 previous similar messages
[13820069.784536] Lustre: oak-OST011d: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[13820069.794942] Lustre: Skipped 32 previous similar messages
[13820071.792677] LustreError: 137-5: oak-OST0148_UUID: not available for connect from 10.210.12.73@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13820071.810260] LustreError: Skipped 1 previous similar message
[13820158.540331] Lustre: oak-OST011b: Client 6b6cd352-15e2-b5e6-2cf2-86cbf0e9b626 (at 10.210.12.145@tcp1) reconnecting
[13820158.551066] Lustre: Skipped 13 previous similar messages
[13820177.446323] Lustre: oak-OST0113: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[13820177.456731] Lustre: Skipped 55 previous similar messages
[13820655.055137] Lustre: oak-OST012b: Connection restored to  (at 10.51.5.30@o2ib3)
[13820655.062607] Lustre: Skipped 1031 previous similar messages
[13821253.932372] Lustre: oak-OST0145: Connection restored to dc6310a8-f0c0-fe0b-d354-acfd9448e8fa (at 10.51.4.51@o2ib3)
[13821253.942969] Lustre: Skipped 835 previous similar messages
[13821854.000535] Lustre: oak-OST014b: Connection restored to 2d7b80e6-0f0d-a2ea-c50b-4f791ee979a6 (at 10.51.16.21@o2ib3)
[13821854.011283] Lustre: Skipped 1596 previous similar messages
[13822454.544063] Lustre: oak-OST0125: Connection restored to  (at 10.51.15.12@o2ib3)
[13822454.551697] Lustre: Skipped 968 previous similar messages
[13822938.173274] LustreError: 228401:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[13823053.436376] Lustre: oak-OST014b: Connection restored to  (at 10.51.15.6@o2ib3)
[13823053.443851] Lustre: Skipped 842 previous similar messages
[13823652.152707] Lustre: oak-OST014b: Connection restored to 13782cfa-f64e-3b30-9b19-eac54c74cba1 (at 10.51.15.10@o2ib3)
[13823652.163414] Lustre: Skipped 1542 previous similar messages
[13824250.761573] Lustre: oak-OST0145: Connection restored to cadd1e4c-e079-2260-6d2d-29034d61d7c1 (at 10.50.5.72@o2ib2)
[13824250.772150] Lustre: Skipped 913 previous similar messages
[13824600.512989] Lustre: oak-OST0139: Client 274e2965-2f8d-f91e-8ba9-3f4bdaf24c8b (at 10.210.12.119@tcp1) reconnecting
[13824600.523490] Lustre: Skipped 20 previous similar messages
[13824600.550888] LustreError: 160936:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91d42b776050 x1714909753151232/t0(0) o4->274e2965-2f8d-f91e-8ba9-3f4bdaf24c8b@10.210.12.119@tcp1:691/0 lens 488/448 e 0 to 0 dl 1645050561 ref 1 fl Interpret:/0/0 rc 0/0
[13824600.568993] Lustre: oak-OST0149: Bulk IO write error with a166a60d-30d5-1e72-b799-745de1d3b307 (at 10.210.12.125@tcp1), client will retry: rc = -110
[13824600.568994] Lustre: Skipped 6 previous similar messages
[13824600.594439] LustreError: 160936:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[13824601.067314] LustreError: 160891:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff91ebb70ed850 x1714909753160832/t0(0) o4->274e2965-2f8d-f91e-8ba9-3f4bdaf24c8b@10.210.12.119@tcp1:696/0 lens 488/448 e 0 to 0 dl 1645050566 ref 1 fl Interpret:/2/0 rc 0/0
[13824601.092282] Lustre: oak-OST0139: Bulk IO write error with 274e2965-2f8d-f91e-8ba9-3f4bdaf24c8b (at 10.210.12.119@tcp1), client will retry: rc = -107
[13824601.105967] Lustre: Skipped 3 previous similar messages
[13824601.206309] LustreError: 21621:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91d92a283050 x1714906365159232/t0(0) o4->a166a60d-30d5-1e72-b799-745de1d3b307@10.210.12.125@tcp1:692/0 lens 488/448 e 0 to 0 dl 1645050562 ref 1 fl Interpret:/0/0 rc 0/0
[13824601.230753] LustreError: 21621:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 2 previous similar messages
[13824679.818780] LustreError: 137-5: oak-OST0136_UUID: not available for connect from 10.210.12.119@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13824679.836456] LustreError: Skipped 1 previous similar message
[13824767.960811] Lustre: oak-OST0111: Client a166a60d-30d5-1e72-b799-745de1d3b307 (at 10.210.12.125@tcp1) reconnecting
[13824767.971426] Lustre: Skipped 17 previous similar messages
[13824778.626055] Lustre: oak-OST0121: Client 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1) reconnecting
[13824778.636480] Lustre: Skipped 6 previous similar messages
[13824798.197842] Lustre: oak-OST0131: Client 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1) reconnecting
[13824798.208254] Lustre: Skipped 16 previous similar messages
[13824851.343718] Lustre: oak-OST0135: Connection restored to 7bd5e25d-92e5-9f8c-22fa-8f4c98d09593 (at 10.50.10.6@o2ib2)
[13824851.354311] Lustre: Skipped 826 previous similar messages
[13824861.379130] Lustre: oak-OST0139: Client 274e2965-2f8d-f91e-8ba9-3f4bdaf24c8b (at 10.210.12.119@tcp1) reconnecting
[13824861.389620] Lustre: Skipped 44 previous similar messages
[13824861.453785] LustreError: 160911:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91958238d050 x1714909755156288/t0(0) o4->274e2965-2f8d-f91e-8ba9-3f4bdaf24c8b@10.210.12.119@tcp1:198/0 lens 488/448 e 0 to 0 dl 1645050823 ref 1 fl Interpret:/0/0 rc 0/0
[13824861.478489] LustreError: 160911:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 2 previous similar messages
[13824861.489234] Lustre: oak-OST0139: Bulk IO write error with 274e2965-2f8d-f91e-8ba9-3f4bdaf24c8b (at 10.210.12.119@tcp1), client will retry: rc = -110
[13824861.502817] Lustre: Skipped 4 previous similar messages
[13824915.926387] LustreError: 228404:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9195ac7a1050 x1715041962361408/t0(0) o4->80374d70-7d99-f68a-b58b-86c0b7514d6a@10.210.12.65@tcp1:252/0 lens 488/448 e 0 to 0 dl 1645050877 ref 1 fl Interpret:/0/0 rc 0/0
[13824915.950852] LustreError: 228404:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[13824915.960666] Lustre: oak-OST012b: Bulk IO write error with 80374d70-7d99-f68a-b58b-86c0b7514d6a (at 10.210.12.65@tcp1), client will retry: rc = -110
[13824915.974127] Lustre: Skipped 1 previous similar message
[13824928.569655] LustreError: 243496:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff915c3051b050 x1714909756128576/t0(0) o4->274e2965-2f8d-f91e-8ba9-3f4bdaf24c8b@10.210.12.119@tcp1:264/0 lens 488/448 e 0 to 0 dl 1645050889 ref 1 fl Interpret:/0/0 rc 0/0
[13824928.594310] LustreError: 243496:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 7 previous similar messages
[13824928.604323] Lustre: oak-OST013b: Bulk IO write error with 274e2965-2f8d-f91e-8ba9-3f4bdaf24c8b (at 10.210.12.119@tcp1), client will retry: rc = -110
[13824928.617848] Lustre: Skipped 7 previous similar messages
[13824934.161663] LustreError: 132-0: oak-OST011d: BAD READ CHECKSUM: should have changed on the client or in transit: from 10.210.12.72@tcp1 inode [0x32c0011970:0x15cc6:0x0] object 0x4a40000402:4195123 extent [6190792704-6194987007], client returned csum 4680eaa4 (type 20), server csum c47be3a7 (type 20)
[13824972.703484] LustreError: 162702:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91480d371050 x1715105215182528/t0(0) o4->b4fca75a-2d54-9bf2-331f-67d0c52e5ff2@10.210.12.73@tcp1:264/0 lens 488/448 e 0 to 0 dl 1645050889 ref 1 fl Interpret:/0/0 rc 0/0
[13824972.704315] Lustre: oak-OST013d: Bulk IO write error with e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1), client will retry: rc = -110
[13824972.704317] Lustre: Skipped 14 previous similar messages
[13824972.750445] LustreError: 162702:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 5 previous similar messages
[13824996.651616] LustreError: 162687:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 3145728(4194304)  req@ffff9178cefdc850 x1714907160745856/t0(0) o3->0d4d52ae-6d60-d275-1404-c9b4d99e0974@10.210.12.109@tcp1:266/0 lens 488/440 e 0 to 0 dl 1645050891 ref 1 fl Interpret:/0/0 rc 0/0
[13824996.651736] Lustre: oak-OST012b: Bulk IO read error with 0d4d52ae-6d60-d275-1404-c9b4d99e0974 (at 10.210.12.109@tcp1), client will retry: rc -110
[13824996.690711] LustreError: 162687:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 2 previous similar messages
[13824997.833283] Lustre: oak-OST013d: Client e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1) reconnecting
[13824997.843695] Lustre: Skipped 25 previous similar messages
[13825007.764224] LustreError: 137-5: oak-OST012a_UUID: not available for connect from 10.210.12.73@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13825073.611130] LustreError: 21608:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff916daa3b7050 x1715008648145408/t0(0) o4->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:412/0 lens 488/448 e 0 to 0 dl 1645051037 ref 1 fl Interpret:/0/0 rc 0/0
[13825073.633295] Lustre: oak-OST014d: Bulk IO write error with c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1), client will retry: rc = -110
[13825073.633296] Lustre: Skipped 5 previous similar messages
[13825073.654527] LustreError: 21608:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 15 previous similar messages
[13825109.837118] LustreError: 162710:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff914a0417b050 x1715041969351104/t0(0) o4->80374d70-7d99-f68a-b58b-86c0b7514d6a@10.210.12.65@tcp1:449/0 lens 488/448 e 0 to 0 dl 1645051074 ref 1 fl Interpret:/0/0 rc 0/0
[13825109.861539] LustreError: 162710:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 8 previous similar messages
[13825109.871357] Lustre: oak-OST014b: Bulk IO write error with 80374d70-7d99-f68a-b58b-86c0b7514d6a (at 10.210.12.65@tcp1), client will retry: rc = -110
[13825109.884785] Lustre: Skipped 9 previous similar messages
[13825140.358951] LustreError: 162716:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff914083c80850 x1715008648140416/t0(0) o3->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:410/0 lens 488/440 e 0 to 0 dl 1645051035 ref 1 fl Interpret:/0/0 rc 0/0
[13825140.359173] Lustre: oak-OST0141: Bulk IO read error with c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1), client will retry: rc -110
[13825140.359175] Lustre: Skipped 2 previous similar messages
[13825140.403173] LustreError: 162716:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 2 previous similar messages
[13825147.789663] Lustre: oak-OST0115: Client 80374d70-7d99-f68a-b58b-86c0b7514d6a (at 10.210.12.65@tcp1) reconnecting
[13825147.800097] Lustre: Skipped 148 previous similar messages
[13825220.318565] LustreError: 21608:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff918ed6ed2050 x1714942006699200/t0(0) o4->5704685e-b0a9-b5c3-be3f-b278a4395bec@10.210.12.72@tcp1:561/0 lens 488/448 e 0 to 0 dl 1645051186 ref 1 fl Interpret:/0/0 rc 0/0
[13825220.343541] Lustre: oak-OST0141: Bulk IO write error with 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1), client will retry: rc = -110
[13825220.561045] Lustre: oak-OST012b: Bulk IO read error with 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1), client will retry: rc -110
[13825220.574228] Lustre: Skipped 2 previous similar messages
[13825283.082026] LustreError: 21593:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 3145728(4194304)  req@ffff918f71ec2850 x1714942006612608/t0(0) o3->5704685e-b0a9-b5c3-be3f-b278a4395bec@10.210.12.72@tcp1:557/0 lens 488/440 e 0 to 0 dl 1645051182 ref 1 fl Interpret:/0/0 rc 0/0
[13825283.107553] Lustre: oak-OST0149: Bulk IO read error with 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1), client will retry: rc -110
[13825283.120729] Lustre: Skipped 3 previous similar messages
[13825415.433338] LustreError: 160931:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91b6d893d850 x1714942019039808/t0(0) o4->5704685e-b0a9-b5c3-be3f-b278a4395bec@10.210.12.72@tcp1:1/0 lens 488/448 e 0 to 0 dl 1645051381 ref 1 fl Interpret:/0/0 rc 0/0
[13825415.457667] LustreError: 160931:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 13 previous similar messages
[13825415.467685] Lustre: oak-OST0119: Bulk IO write error with 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1), client will retry: rc = -110
[13825415.481122] Lustre: Skipped 7 previous similar messages
[13825421.548083] Lustre: oak-OST0149: Bulk IO read error with 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1), client will retry: rc -110
[13825421.561265] Lustre: Skipped 1 previous similar message
[13825449.989382] Lustre: oak-OST0115: Connection restored to 2b2b5b47-3d88-dfa6-3173-9352bce15dc5 (at 10.50.5.53@o2ib2)
[13825449.999964] Lustre: Skipped 1412 previous similar messages
[13825460.250469] Lustre: oak-OST0123: Client a2f24d85-30b4-0808-a59e-1ce0f2193fa6 (at 10.210.12.121@tcp1) reconnecting
[13825460.261123] Lustre: Skipped 90 previous similar messages
[13825474.724252] LustreError: 21585:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff918eef5de050 x1714942019033792/t0(0) o3->5704685e-b0a9-b5c3-be3f-b278a4395bec@10.210.12.72@tcp1:1/0 lens 488/440 e 0 to 0 dl 1645051381 ref 1 fl Interpret:/0/0 rc 0/0
[13825474.724442] Lustre: oak-OST011d: Bulk IO read error with 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1), client will retry: rc -110
[13825474.724443] Lustre: Skipped 1 previous similar message
[13825474.768094] LustreError: 21585:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 7 previous similar messages
[13825579.616288] LustreError: 21619:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff918c23e3f850 x1715134497267968/t0(0) o4->cc381d20-202e-f264-43c3-938610d60653@10.210.12.58@tcp1:164/0 lens 488/448 e 0 to 0 dl 1645051544 ref 1 fl Interpret:/0/0 rc 0/0
[13825579.640630] LustreError: 21619:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 14 previous similar messages
[13825642.388869] LustreError: 243440:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(24576)  req@ffff9182ec836850 x1715089042615360/t0(0) o3->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:163/0 lens 488/440 e 0 to 0 dl 1645051543 ref 1 fl Interpret:/0/0 rc 0/0
[13825642.389042] Lustre: oak-OST014b: Bulk IO read error with e44e675f-11ba-536d-fbc7-173bafd860e2 (at 10.210.12.46@tcp1), client will retry: rc -110
[13825642.389044] Lustre: Skipped 7 previous similar messages
[13825642.432494] LustreError: 243440:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 3 previous similar messages
[13825660.234836] LustreError: 137-5: oak-OST0156_UUID: not available for connect from 10.210.12.121@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13826030.776695] LustreError: 228403:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff913d8940a050 x1715081823108480/t0(0) o4->65e9b8a8-b461-24e0-da12-5b6939f0b07c@10.210.12.56@tcp1:615/0 lens 488/448 e 0 to 0 dl 1645051995 ref 1 fl Interpret:/0/0 rc 0/0
[13826030.779541] Lustre: oak-OST0131: Bulk IO write error with 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1), client will retry: rc = -110
[13826030.779542] Lustre: Skipped 45 previous similar messages
[13826030.820627] LustreError: 228403:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 35 previous similar messages
[13826049.396021] Lustre: oak-OST0147: Connection restored to 2e8e2ea1-d309-871f-47c3-fbf87733e2ac (at 10.50.5.45@o2ib2)
[13826049.407014] Lustre: Skipped 1066 previous similar messages
[13826097.531961] LustreError: 253935:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 3145728(4194304)  req@ffff91a292dbf850 x1715081823090688/t0(0) o3->65e9b8a8-b461-24e0-da12-5b6939f0b07c@10.210.12.56@tcp1:614/0 lens 488/440 e 0 to 0 dl 1645051994 ref 1 fl Interpret:/0/0 rc 0/0
[13826097.532133] Lustre: oak-OST013f: Bulk IO read error with 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1), client will retry: rc -110
[13826097.532134] Lustre: Skipped 3 previous similar messages
[13826097.576232] LustreError: 253935:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 1 previous similar message
[13826199.181233] Lustre: oak-OST013f: Client 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1) reconnecting
[13826199.191648] Lustre: Skipped 199 previous similar messages
[13826617.708314] LustreError: 137-5: oak-OST0134_UUID: not available for connect from 10.210.12.129@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13826648.002127] Lustre: oak-OST0149: Connection restored to f7cd9733-9970-c1e4-4322-af5220631562 (at 10.50.5.68@o2ib2)
[13826648.012714] Lustre: Skipped 900 previous similar messages
[13826744.377224] LustreError: 21620:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(24576)  req@ffff91770d4b4850 x1715530944669376/t0(0) o3->5a5caa86-68df-f806-7e6f-05440e95a4ce@10.210.12.43@tcp1:523/0 lens 488/440 e 0 to 0 dl 1645052658 ref 1 fl Interpret:/0/0 rc 0/0
[13826744.377234] Lustre: oak-OST0143: Bulk IO read error with 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1), client will retry: rc -110
[13826744.377235] Lustre: Skipped 1 previous similar message
[13826744.420607] LustreError: 21620:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 3 previous similar messages
[13826800.427477] Lustre: oak-OST011b: Client b4fca75a-2d54-9bf2-331f-67d0c52e5ff2 (at 10.210.12.73@tcp1) reconnecting
[13826800.437964] Lustre: Skipped 47 previous similar messages
[13826817.106430] LustreError: 162670:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9180c3833850 x1715239127319744/t0(0) o4->e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda@10.210.12.79@tcp1:648/0 lens 488/448 e 0 to 0 dl 1645052783 ref 1 fl Interpret:/0/0 rc 0/0
[13826817.130907] LustreError: 162670:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 23 previous similar messages
[13826817.140900] Lustre: oak-OST013d: Bulk IO write error with e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1), client will retry: rc = -110
[13826817.154376] Lustre: Skipped 25 previous similar messages
[13826958.996559] LustreError: 21597:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(20480)  req@ffff91663b064850 x1714967366471296/t0(0) o3->5afa23c2-1fd5-9030-38ce-b03a359082fe@10.210.12.75@tcp1:725/0 lens 488/440 e 0 to 0 dl 1645052860 ref 1 fl Interpret:/0/0 rc 0/0
[13826959.008785] Lustre: oak-OST014b: Bulk IO read error with e44e675f-11ba-536d-fbc7-173bafd860e2 (at 10.210.12.46@tcp1), client will retry: rc -110
[13826959.008786] Lustre: Skipped 3 previous similar messages
[13826959.040010] LustreError: 21597:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 8 previous similar messages
[13827198.572234] LustreError: 21603:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(24576)  req@ffff91649a3d8050 x1721911736190080/t0(0) o3->e44e675f-11ba-536d-fbc7-173bafd860e2@10.210.12.46@tcp1:221/0 lens 488/440 e 0 to 0 dl 1645053111 ref 1 fl Interpret:/0/0 rc 0/0
[13827198.572671] LustreError: 243440:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff9170effb2050 x1721911736191936/t0(0) o4->e44e675f-11ba-536d-fbc7-173bafd860e2@10.210.12.46@tcp1:223/0 lens 488/448 e 0 to 0 dl 1645053113 ref 1 fl Interpret:/0/0 rc 0/0
[13827198.622863] LustreError: 21603:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 4 previous similar messages
[13827225.170132] LustreError: 137-5: oak-OST0140_UUID: not available for connect from 10.210.12.46@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13827247.060840] Lustre: oak-OST0123: Connection restored to a24dd7a0-5fa8-502a-258a-91fb9f94b4c0 (at 10.50.5.43@o2ib2)
[13827247.071487] Lustre: Skipped 1360 previous similar messages
[13827399.032180] Lustre: oak-OST0113: Client e44e675f-11ba-536d-fbc7-173bafd860e2 (at 10.210.12.46@tcp1) reconnecting
[13827399.042592] Lustre: Skipped 207 previous similar messages
[13827605.867257] LustreError: 243541:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff916a6367b850 x1715239137396928/t0(0) o4->e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda@10.210.12.79@tcp1:639/0 lens 488/448 e 0 to 0 dl 1645053529 ref 1 fl Interpret:/0/0 rc 0/0
[13827605.893583] LustreError: 243541:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 3 previous similar messages
[13827605.903803] Lustre: oak-OST0147: Bulk IO write error with e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1), client will retry: rc = -110
[13827605.917256] Lustre: Skipped 30 previous similar messages
[13827629.825002] LustreError: 162694:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91376f78a850 x1715239137419584/t0(0) o4->e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda@10.210.12.79@tcp1:642/0 lens 488/448 e 0 to 0 dl 1645053532 ref 1 fl Interpret:/0/0 rc 0/0
[13827640.661043] LustreError: 137-5: oak-OST0132_UUID: not available for connect from 10.210.12.79@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13827845.675578] Lustre: oak-OST014b: Connection restored to a24dd7a0-5fa8-502a-258a-91fb9f94b4c0 (at 10.50.5.43@o2ib2)
[13827845.686170] Lustre: Skipped 1415 previous similar messages
[13828252.502563] Lustre: oak-OST012d: Client b4fca75a-2d54-9bf2-331f-67d0c52e5ff2 (at 10.210.12.73@tcp1) reconnecting
[13828252.513669] Lustre: Skipped 44 previous similar messages
[13828252.544392] LustreError: 162704:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff916be2339050 x1715105555023680/t0(0) o4->b4fca75a-2d54-9bf2-331f-67d0c52e5ff2@10.210.12.73@tcp1:579/0 lens 488/448 e 0 to 0 dl 1645054224 ref 1 fl Interpret:/0/0 rc 0/0
[13828252.552578] Lustre: oak-OST012d: Bulk IO write error with b4fca75a-2d54-9bf2-331f-67d0c52e5ff2 (at 10.210.12.73@tcp1), client will retry: rc = -110
[13828252.552579] Lustre: Skipped 1 previous similar message
[13828252.588820] LustreError: 162704:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 30 previous similar messages
[13828441.832307] LustreError: 243454:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91ea3bb83850 x1715531343654912/t0(0) o4->5a5caa86-68df-f806-7e6f-05440e95a4ce@10.210.12.43@tcp1:12/0 lens 488/448 e 0 to 0 dl 1645054412 ref 1 fl Interpret:/0/0 rc 0/0
[13828441.856732] LustreError: 243454:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 5 previous similar messages
[13828442.151548] Lustre: oak-OST014b: Bulk IO read error with 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1), client will retry: rc -110
[13828442.164727] Lustre: Skipped 14 previous similar messages
[13828444.429181] Lustre: oak-OST0145: Connection restored to a92605b2-b836-1e0b-805a-7816039df150 (at 10.50.10.11@o2ib2)
[13828444.439853] Lustre: Skipped 748 previous similar messages
[13828491.346967] LustreError: 21618:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff9185a7ae8850 x1715089451289024/t0(0) o3->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:10/0 lens 504/440 e 0 to 0 dl 1645054410 ref 1 fl Interpret:/0/0 rc 0/0
[13828491.347778] Lustre: oak-OST0111: Bulk IO read error with b4fca75a-2d54-9bf2-331f-67d0c52e5ff2 (at 10.210.12.73@tcp1), client will retry: rc -110
[13828491.356847] LustreError: 160934:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4057172)  req@ffff91ef9147b850 x1715089451310592/t0(0) o4->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:11/0 lens 504/448 e 0 to 0 dl 1645054411 ref 1 fl Interpret:/0/0 rc 0/0
[13828491.410742] LustreError: 21618:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 7 previous similar messages
[13828515.308696] LustreError: 160926:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 1748407(3845559)  req@ffff91d91489b050 x1715089451534208/t0(0) o4->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:21/0 lens 552/464 e 0 to 0 dl 1645054421 ref 1 fl Interpret:/0/0 rc 0/0
[13828515.334430] LustreError: 160926:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 3 previous similar messages
[13828522.251131] LustreError: 137-5: oak-OST0124_UUID: not available for connect from 10.210.12.46@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13828522.268793] LustreError: Skipped 2 previous similar messages
[13828635.116509] LustreError: 243535:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff915044aa9850 x1721911908632384/t0(0) o3->e44e675f-11ba-536d-fbc7-173bafd860e2@10.210.12.46@tcp1:209/0 lens 488/440 e 0 to 0 dl 1645054609 ref 1 fl Interpret:/0/0 rc 0/0
[13828635.140883] LustreError: 243535:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 18 previous similar messages
[13828635.150719] Lustre: oak-OST0123: Bulk IO read error with e44e675f-11ba-536d-fbc7-173bafd860e2 (at 10.210.12.46@tcp1), client will retry: rc -110
[13828635.163913] Lustre: Skipped 7 previous similar messages
[13828662.033185] LustreError: 137-5: oak-OST0142_UUID: not available for connect from 10.210.12.58@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13828683.026060] LustreError: 21588:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(20480)  req@ffff9182bb239050 x1715042218691264/t0(0) o3->80374d70-7d99-f68a-b58b-86c0b7514d6a@10.210.12.65@tcp1:206/0 lens 488/440 e 0 to 0 dl 1645054606 ref 1 fl Interpret:/0/0 rc 0/0
[13828683.026670] LustreError: 243539:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff914583285850 x1714948553440000/t0(0) o4->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:209/0 lens 488/448 e 0 to 0 dl 1645054609 ref 1 fl Interpret:/0/0 rc 0/0
[13828683.076645] LustreError: 21588:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 1 previous similar message
[13828716.261212] LustreError: 137-5: oak-OST0118_UUID: not available for connect from 10.210.12.58@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13828716.278822] LustreError: Skipped 1 previous similar message
[13828754.886381] LustreError: 162701:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 1048576(4194304)  req@ffff9151c3ecd850 x1715531353426304/t0(0) o3->5a5caa86-68df-f806-7e6f-05440e95a4ce@10.210.12.43@tcp1:269/0 lens 488/440 e 0 to 0 dl 1645054669 ref 1 fl Interpret:/0/0 rc 0/0
[13828754.912498] LustreError: 162701:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 7 previous similar messages
[13828866.405356] Lustre: oak-OST0137: Client 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1) reconnecting
[13828866.415939] Lustre: Skipped 297 previous similar messages
[13828884.058635] Lustre: oak-OST011b: Bulk IO write error with c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1), client will retry: rc = -110
[13828884.072159] Lustre: Skipped 45 previous similar messages
[13828887.016372] Lustre: oak-OST014b: Bulk IO read error with c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1), client will retry: rc -110
[13828887.029557] Lustre: Skipped 11 previous similar messages
[13828946.571439] LustreError: 243537:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(20480)  req@ffff91363c393050 x1715531368221056/t0(0) o3->5a5caa86-68df-f806-7e6f-05440e95a4ce@10.210.12.43@tcp1:455/0 lens 488/440 e 0 to 0 dl 1645054855 ref 1 fl Interpret:/0/0 rc 0/0
[13828946.576923] LustreError: 21621:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91978a1ce050 x1714905922265728/t0(0) o4->6ca637d5-9e95-16bf-f08c-446745633d32@10.210.12.129@tcp1:456/0 lens 488/448 e 0 to 0 dl 1645054856 ref 1 fl Interpret:/0/0 rc 0/0
[13828946.622084] LustreError: 243537:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 6 previous similar messages
[13828963.248530] LustreError: 137-5: oak-OST0148_UUID: not available for connect from 10.210.12.43@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13828963.248531] LustreError: 137-5: oak-OST0146_UUID: not available for connect from 10.210.12.43@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13829039.911862] Lustre: oak-OST0143: haven't heard from client 53a49d3c-d3c0-23d0-0cfc-52a26b90cb00 (at 10.51.13.4@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9195a6bd1000, cur 1645054919 expire 1645054769 last 1645054692
[13829043.619748] Lustre: oak-OST0123: Connection restored to a6fe3701-42c7-46f0-b462-758ca9032454 (at 10.210.12.29@tcp1)
[13829043.630431] Lustre: Skipped 1247 previous similar messages
[13829048.919545] Lustre: oak-OST012d: haven't heard from client 53a49d3c-d3c0-23d0-0cfc-52a26b90cb00 (at 10.51.13.4@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91b9e00e8800, cur 1645054928 expire 1645054778 last 1645054701
[13829048.941459] Lustre: Skipped 18 previous similar messages
[13829052.883224] Lustre: oak-OST013d: haven't heard from client 53a49d3c-d3c0-23d0-0cfc-52a26b90cb00 (at 10.51.13.4@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91a445b14800, cur 1645054932 expire 1645054782 last 1645054705
[13829066.391653] LustreError: 160901:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff91b6c9e0a850 x1714909379572416/t0(0) o3->f4048b3b-d9e4-178d-cd58-3b66faaca4ee@10.210.12.117@tcp1:587/0 lens 488/440 e 0 to 0 dl 1645054987 ref 1 fl Interpret:/0/0 rc 0/0
[13829066.417990] LustreError: 160901:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 1 previous similar message
[13829084.960045] LustreError: 21610:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9189bc932050 x1715531382703616/t0(0) o4->5a5caa86-68df-f806-7e6f-05440e95a4ce@10.210.12.43@tcp1:657/0 lens 488/448 e 0 to 0 dl 1645055057 ref 1 fl Interpret:/0/0 rc 0/0
[13829084.984435] LustreError: 21610:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 20 previous similar messages
[13829138.252515] LustreError: 228366:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(24576)  req@ffff9191e6133850 x1715531382701248/t0(0) o3->5a5caa86-68df-f806-7e6f-05440e95a4ce@10.210.12.43@tcp1:657/0 lens 488/440 e 0 to 0 dl 1645055057 ref 1 fl Interpret:/0/0 rc 0/0
[13829138.277374] LustreError: 228366:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 2 previous similar messages
[13829163.659488] LustreError: 137-5: oak-OST0144_UUID: not available for connect from 10.210.12.43@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13829566.027094] Lustre: oak-OST0133: Client 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1) reconnecting
[13829566.037505] Lustre: Skipped 232 previous similar messages
[13829566.245359] Lustre: oak-OST0137: Bulk IO write error with 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1), client will retry: rc = -110
[13829566.258928] Lustre: Skipped 8 previous similar messages
[13829569.041404] LustreError: 162704:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk READ failed: rc -107  req@ffff917c883af850 x1715008928239488/t0(0) o3->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:392/0 lens 488/440 e 0 to 0 dl 1645055547 ref 1 fl Interpret:/0/0 rc 0/0
[13829569.066340] Lustre: oak-OST0149: Bulk IO read error with c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1), client will retry: rc -107
[13829569.079522] Lustre: Skipped 15 previous similar messages
[13829617.455808] LustreError: 160926:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2004560(3053136)  req@ffff91e2c1880850 x1715008928114944/t0(0) o3->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:386/0 lens 488/440 e 0 to 0 dl 1645055541 ref 1 fl Interpret:/0/0 rc 0/0
[13829617.481355] LustreError: 160926:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 1 previous similar message
[13829642.829763] Lustre: oak-OST012b: Connection restored to ed62306b-c979-ade9-eb31-8485f9927386 (at 10.50.10.65@o2ib2)
[13829642.840434] Lustre: Skipped 1290 previous similar messages
[13829689.324996] LustreError: 21584:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff91708b768050 x1721912086405568/t0(0) o3->e44e675f-11ba-536d-fbc7-173bafd860e2@10.210.12.46@tcp1:450/0 lens 488/440 e 0 to 0 dl 1645055605 ref 1 fl Interpret:/0/0 rc 0/0
[13829689.349757] LustreError: 21584:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 1 previous similar message
[13829712.309694] LustreError: 137-5: oak-OST014a_UUID: not available for connect from 10.210.12.46@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13829823.911362] LustreError: 21583:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff918f7b1c8050 x1714909805967296/t0(0) o3->274e2965-2f8d-f91e-8ba9-3f4bdaf24c8b@10.210.12.119@tcp1:647/0 lens 488/440 e 0 to 0 dl 1645055802 ref 1 fl Interpret:/0/0 rc 0/0
[13829823.935700] LustreError: 21583:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 10 previous similar messages
[13830119.592808] LustreError: 160897:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 3145728(4194304)  req@ffff91b13cadd050 x1714948666281984/t0(0) o3->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:133/0 lens 488/440 e 0 to 0 dl 1645056043 ref 1 fl Interpret:/0/0 rc 0/0
[13830119.593149] LustreError: 160935:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(1228800)  req@ffff91a159c0e850 x1714942334561600/t0(0) o4->5704685e-b0a9-b5c3-be3f-b278a4395bec@10.210.12.72@tcp1:134/0 lens 488/448 e 0 to 0 dl 1645056044 ref 1 fl Interpret:/0/0 rc 0/0
[13830119.593151] LustreError: 243456:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 1831730(2880306)  req@ffff91a01a002050 x1714942334561472/t0(0) o4->5704685e-b0a9-b5c3-be3f-b278a4395bec@10.210.12.72@tcp1:134/0 lens 504/448 e 0 to 0 dl 1645056044 ref 1 fl Interpret:/0/0 rc 0/0
[13830119.593152] LustreError: 160935:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[13830119.593153] LustreError: 243456:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[13830119.594211] Lustre: oak-OST014b: Bulk IO read error with 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1), client will retry: rc -110
[13830119.594211] Lustre: Skipped 8 previous similar messages
[13830119.708370] LustreError: 160897:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 1 previous similar message
[13830236.112537] Lustre: oak-OST0119: Client c4b979fd-3b98-af30-5ea5-32f00f4d8750 (at 10.210.12.64@tcp1) reconnecting
[13830236.122945] Lustre: Skipped 66 previous similar messages
[13830241.399550] Lustre: oak-OST0149: Connection restored to a92605b2-b836-1e0b-805a-7816039df150 (at 10.50.10.11@o2ib2)
[13830241.410215] Lustre: Skipped 1788 previous similar messages
[13830438.167721] LustreError: 243332:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff915d937b1850 x1714971854838144/t0(0) o4->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:502/0 lens 504/448 e 0 to 0 dl 1645056412 ref 1 fl Interpret:/0/0 rc 0/0
[13830438.192171] LustreError: 243332:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 13 previous similar messages
[13830438.202107] Lustre: oak-OST0133: Bulk IO write error with 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1), client will retry: rc = -110
[13830438.215574] Lustre: Skipped 22 previous similar messages
[13830766.497703] LustreError: 243555:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff917d9bb36850 x1721912235261760/t0(0) o4->e44e675f-11ba-536d-fbc7-173bafd860e2@10.210.12.46@tcp1:25/0 lens 488/448 e 0 to 0 dl 1645056690 ref 1 fl Interpret:/0/0 rc 0/0
[13830766.497709] LustreError: 243536:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff915c7e2b2850 x1721912235271936/t0(0) o3->e44e675f-11ba-536d-fbc7-173bafd860e2@10.210.12.46@tcp1:25/0 lens 488/440 e 0 to 0 dl 1645056690 ref 1 fl Interpret:/0/0 rc 0/0
[13830766.497723] Lustre: oak-OST0123: Bulk IO read error with e44e675f-11ba-536d-fbc7-173bafd860e2 (at 10.210.12.46@tcp1), client will retry: rc -110
[13830766.497724] Lustre: Skipped 1 previous similar message
[13830766.566726] LustreError: 243555:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 5 previous similar messages
[13830797.888458] LustreError: 137-5: oak-OST012e_UUID: not available for connect from 10.210.12.44@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13830843.576450] Lustre: oak-OST012f: Connection restored to bc7396f6-fbb0-5e6f-3bd8-bf31cb05de7c (at 10.50.5.55@o2ib2)
[13830843.587050] Lustre: Skipped 926 previous similar messages
[13830847.056037] Lustre: oak-OST0123: Client 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1) reconnecting
[13830847.066497] Lustre: Skipped 69 previous similar messages
[13830858.962911] LustreError: 199244:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 149s: evicting client at 10.210.12.8@tcp1  ns: filter-oak-OST0145_UUID lock: ffff91eed7ec0480/0xed112d301c2507b4 lrc: 3/0,0 mode: PW/PW res: [0x5540000401:0x3a460d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->36863) flags: 0x60000400030020 nid: 10.210.12.8@tcp1 remote: 0x9058bd9f4d931bd7 expref: 447 pid: 229558 timeout: 13864475 lvb_type: 0
[13830859.004119] LustreError: 199244:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 131 previous similar messages
[13830910.249713] LustreError: 243542:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91850c029050 x1721912236069504/t0(0) o4->e44e675f-11ba-536d-fbc7-173bafd860e2@10.210.12.46@tcp1:159/0 lens 488/448 e 0 to 0 dl 1645056824 ref 1 fl Interpret:/0/0 rc 0/0
[13831110.040361] LustreError: 137-5: oak-OST0124_UUID: not available for connect from 10.210.12.46@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13831162.573846] LustreError: 137-5: oak-OST0142_UUID: not available for connect from 10.210.12.31@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13831176.916991] LustreError: 21588:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91401752d850 x1724766187952128/t0(0) o4->479db7cf-e752-7ee3-6b36-d1f694b751cf@10.210.12.47@tcp1:488/0 lens 488/448 e 0 to 0 dl 1645057153 ref 1 fl Interpret:/0/0 rc 0/0
[13831176.941333] LustreError: 21588:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 23 previous similar messages
[13831176.951194] Lustre: oak-OST013f: Bulk IO write error with 479db7cf-e752-7ee3-6b36-d1f694b751cf (at 10.210.12.47@tcp1), client will retry: rc = -110
[13831176.964636] Lustre: Skipped 26 previous similar messages
[13831255.669308] LustreError: 137-5: oak-OST0148_UUID: not available for connect from 10.210.12.31@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13831268.636078] LustreError: 21596:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff9178cefde050 x1715106040249920/t0(0) o4->b4fca75a-2d54-9bf2-331f-67d0c52e5ff2@10.210.12.73@tcp1:519/0 lens 488/448 e 0 to 0 dl 1645057184 ref 1 fl Interpret:/0/0 rc 0/0
[13831268.661934] LustreError: 21596:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[13831289.667854] LustreError: 137-5: oak-OST0120_UUID: not available for connect from 10.210.12.23@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13831289.685459] LustreError: Skipped 9 previous similar messages
[13831316.552717] LustreError: 253934:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff91c72ee5b850 x1715009149339072/t0(0) o3->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:566/0 lens 488/440 e 0 to 0 dl 1645057231 ref 1 fl Interpret:/0/0 rc 0/0
[13831316.577592] LustreError: 253934:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 30 previous similar messages
[13831364.458353] LustreError: 199272:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91c92fad3050 x1714948749094848/t0(0) o4->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:615/0 lens 488/448 e 0 to 0 dl 1645057280 ref 1 fl Interpret:/0/0 rc 0/0
[13831364.484203] LustreError: 199272:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[13831384.475485] LustreError: 137-5: oak-OST013e_UUID: not available for connect from 10.210.12.125@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13831436.315552] Lustre: oak-OST0147: Bulk IO read error with 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1), client will retry: rc -110
[13831436.328759] Lustre: Skipped 49 previous similar messages
[13831442.367667] Lustre: oak-OST0147: Connection restored to 2e8e2ea1-d309-871f-47c3-fbf87733e2ac (at 10.50.5.45@o2ib2)
[13831442.378362] Lustre: Skipped 1690 previous similar messages
[13831451.405852] Lustre: 199263:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1645057183/real 1645057183]  req@ffff91e45ca57500 x1710530505122944/t0(0) o104->oak-OST0141@10.210.12.60@tcp1:15/16 lens 296/224 e 0 to 1 dl 1645057336 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
[13831451.433538] Lustre: 199263:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[13831453.941758] Lustre: oak-OST014b: Client c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1) reconnecting
[13831453.952168] Lustre: Skipped 518 previous similar messages
[13831685.795597] LustreError: 137-5: oak-OST0132_UUID: not available for connect from 10.210.12.66@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13831685.795598] LustreError: 137-5: oak-OST013e_UUID: not available for connect from 10.210.12.66@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13831685.795600] LustreError: Skipped 7 previous similar messages
[13831825.272132] LustreError: 243525:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9140904bb050 x1715106092024000/t0(0) o4->b4fca75a-2d54-9bf2-331f-67d0c52e5ff2@10.210.12.73@tcp1:384/0 lens 488/448 e 0 to 0 dl 1645057804 ref 1 fl Interpret:/0/0 rc 0/0
[13831825.296598] LustreError: 243525:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 46 previous similar messages
[13831825.306699] Lustre: oak-OST0133: Bulk IO write error with b4fca75a-2d54-9bf2-331f-67d0c52e5ff2 (at 10.210.12.73@tcp1), client will retry: rc = -110
[13831825.320137] Lustre: Skipped 47 previous similar messages
[13831891.557724] LustreError: 21619:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff9140f442c850 x1714971330667968/t0(0) o4->5d4d5527-a52f-bc68-278b-aa990b8609d6@10.210.12.59@tcp1:390/0 lens 488/448 e 0 to 0 dl 1645057810 ref 1 fl Interpret:/0/0 rc 0/0
[13831939.471557] LustreError: 162669:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff91561dadc050 x1715239282046656/t0(0) o3->e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda@10.210.12.79@tcp1:451/0 lens 488/440 e 0 to 0 dl 1645057871 ref 1 fl Interpret:/0/0 rc 0/0
[13831939.471608] LustreError: 243544:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91422afcf850 x1715089825816768/t0(0) o4->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:451/0 lens 488/448 e 0 to 0 dl 1645057871 ref 1 fl Interpret:/0/0 rc 0/0
[13831939.524222] LustreError: 162669:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 26 previous similar messages
[13831963.426292] LustreError: 243496:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff918e2da6d850 x1714971901143424/t0(0) o4->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:454/0 lens 488/448 e 0 to 0 dl 1645057874 ref 1 fl Interpret:/0/0 rc 0/0
[13831963.452090] LustreError: 243496:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 4 previous similar messages
[13832042.701972] Lustre: oak-OST013d: Connection restored to 313e46a1-96cb-c54a-bb11-60568f017fc9 (at 10.50.5.47@o2ib2)
[13832042.714227] Lustre: Skipped 1697 previous similar messages
[13832059.593924] Lustre: oak-OST013d: Client e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1) reconnecting
[13832059.604351] Lustre: Skipped 411 previous similar messages
[13832334.279101] LustreError: 137-5: oak-OST0146_UUID: not available for connect from 10.210.12.43@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13832334.296745] LustreError: Skipped 4 previous similar messages
[13832447.910638] LustreError: 21604:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff915ba8928050 x1715531893401152/t0(0) o4->5a5caa86-68df-f806-7e6f-05440e95a4ce@10.210.12.43@tcp1:257/0 lens 488/448 e 0 to 0 dl 1645058432 ref 1 fl Interpret:/0/0 rc 0/0
[13832447.935002] LustreError: 21604:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 7 previous similar messages
[13832447.944779] Lustre: oak-OST0143: Bulk IO write error with 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1), client will retry: rc = -110
[13832447.958340] Lustre: Skipped 14 previous similar messages
[13832594.235290] LustreError: 160952:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk READ failed: rc -107  req@ffff91c292b6e050 x1715106145316608/t0(0) o3->b4fca75a-2d54-9bf2-331f-67d0c52e5ff2@10.210.12.73@tcp1:404/0 lens 488/440 e 0 to 0 dl 1645058579 ref 1 fl Interpret:/0/0 rc 0/0
[13832594.240845] Lustre: oak-OST013f: Bulk IO read error with b4fca75a-2d54-9bf2-331f-67d0c52e5ff2 (at 10.210.12.73@tcp1), client will retry: rc -107
[13832594.240847] Lustre: Skipped 18 previous similar messages
[13832594.278806] LustreError: 160952:0:(ldlm_lib.c:3305:target_bulk_io()) Skipped 4 previous similar messages
[13832641.321948] Lustre: oak-OST0131: Connection restored to 74b01e71-a95e-02b5-4cdd-9715608dcb26 (at 10.50.5.52@o2ib2)
[13832641.332575] Lustre: Skipped 906 previous similar messages
[13832657.273143] LustreError: 160937:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(24576)  req@ffff91be4530e050 x1715042537928704/t0(0) o3->80374d70-7d99-f68a-b58b-86c0b7514d6a@10.210.12.65@tcp1:397/0 lens 488/440 e 0 to 0 dl 1645058572 ref 1 fl Interpret:/0/0 rc 0/0
[13832657.273205] LustreError: 127352:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91c417f7a850 x1715082661050752/t0(0) o4->65e9b8a8-b461-24e0-da12-5b6939f0b07c@10.210.12.56@tcp1:397/0 lens 488/448 e 0 to 0 dl 1645058572 ref 1 fl Interpret:/0/0 rc 0/0
[13832657.324100] LustreError: 160937:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 9 previous similar messages
[13832668.242910] Lustre: oak-OST013d: Client bed08025-01aa-dea1-aa90-bead26dab2fb (at 10.50.17.21@o2ib2) reconnecting
[13832668.253324] Lustre: Skipped 187 previous similar messages
[13833121.177442] LustreError: 228423:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff915c4f89f850 x1724766311287808/t0(0) o4->479db7cf-e752-7ee3-6b36-d1f694b751cf@10.210.12.47@tcp1:178/0 lens 488/448 e 0 to 0 dl 1645059108 ref 1 fl Interpret:/0/0 rc 0/0
[13833121.201877] LustreError: 228423:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 9 previous similar messages
[13833121.211974] Lustre: oak-OST0143: Bulk IO write error with 479db7cf-e752-7ee3-6b36-d1f694b751cf (at 10.210.12.47@tcp1), client will retry: rc = -110
[13833121.225455] Lustre: Skipped 11 previous similar messages
[13833121.364974] LustreError: 162705:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff914c76be7050 x1724766311188288/t0(0) o4->479db7cf-e752-7ee3-6b36-d1f694b751cf@10.210.12.47@tcp1:177/0 lens 488/448 e 0 to 0 dl 1645059107 ref 1 fl Interpret:/2/0 rc 0/0
[13833140.962382] LustreError: 243498:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff914602d41050 x1724766311501952/t0(0) o4->479db7cf-e752-7ee3-6b36-d1f694b751cf@10.210.12.47@tcp1:197/0 lens 488/448 e 0 to 0 dl 1645059127 ref 1 fl Interpret:/2/0 rc 0/0
[13833140.987134] LustreError: 243498:0:(ldlm_lib.c:3305:target_bulk_io()) Skipped 2 previous similar messages
[13833184.369603] LustreError: 162696:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff918ed6f0e850 x1714980721672000/t0(0) o4->43c70e25-3329-f639-f581-cb1ed49d9949@10.210.12.42@tcp1:173/0 lens 488/448 e 0 to 0 dl 1645059103 ref 1 fl Interpret:/0/0 rc 0/0
[13833184.395403] LustreError: 162696:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 7 previous similar messages
[13833208.332382] Lustre: oak-OST014b: Bulk IO read error with 6bab8d10-30ff-c1d2-bbf5-1fd0884ac647 (at 10.210.12.57@tcp1), client will retry: rc -110
[13833208.332498] LustreError: 21596:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91713d10a050 x1714971946733376/t0(0) o4->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:193/0 lens 488/448 e 0 to 0 dl 1645059123 ref 1 fl Interpret:/0/0 rc 0/0
[13833208.371737] Lustre: Skipped 23 previous similar messages
[13833219.216571] LustreError: 137-5: oak-OST011c_UUID: not available for connect from 10.210.12.75@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13833219.234155] LustreError: Skipped 1 previous similar message
[13833239.920606] Lustre: oak-OST0121: Connection restored to 313e46a1-96cb-c54a-bb11-60568f017fc9 (at 10.50.5.47@o2ib2)
[13833239.931186] Lustre: Skipped 829 previous similar messages
[13833273.229642] Lustre: oak-OST013f: Client 479db7cf-e752-7ee3-6b36-d1f694b751cf (at 10.210.12.47@tcp1) reconnecting
[13833273.240052] Lustre: Skipped 173 previous similar messages
[13833328.115198] LustreError: 162712:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff914279a1e850 x1715009436290368/t0(0) o3->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:322/0 lens 488/440 e 0 to 0 dl 1645059252 ref 1 fl Interpret:/0/0 rc 0/0
[13833328.126861] LustreError: 243456:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 1048576(2620908)  req@ffff91b1f8644850 x1715009436402240/t0(0) o4->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:330/0 lens 488/448 e 0 to 0 dl 1645059260 ref 1 fl Interpret:/0/0 rc 0/0
[13833328.165886] LustreError: 162712:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 16 previous similar messages
[13833346.704867] LustreError: 243498:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk READ failed: rc -107  req@ffff918b7abc8050 x1714948881641408/t0(0) o3->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:404/0 lens 488/440 e 0 to 0 dl 1645059334 ref 1 fl Interpret:/0/0 rc 0/0
[13833400.004680] LustreError: 243525:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff913e79a98850 x1721912349893568/t0(0) o4->e44e675f-11ba-536d-fbc7-173bafd860e2@10.210.12.46@tcp1:405/0 lens 488/448 e 0 to 0 dl 1645059335 ref 1 fl Interpret:/2/0 rc 0/0
[13833400.030976] LustreError: 243525:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 3 previous similar messages
[13833842.439725] Lustre: oak-OST011d: Connection restored to a24dd7a0-5fa8-502a-258a-91fb9f94b4c0 (at 10.50.5.43@o2ib2)
[13833842.450371] Lustre: Skipped 942 previous similar messages
[13833978.402790] Lustre: oak-OST0111: Client bed08025-01aa-dea1-aa90-bead26dab2fb (at 10.50.17.21@o2ib2) reconnecting
[13833978.413394] Lustre: Skipped 327 previous similar messages
[13834090.254755] LustreError: 228403:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9190fdce8850 x1715532190103232/t0(0) o4->5a5caa86-68df-f806-7e6f-05440e95a4ce@10.210.12.43@tcp1:388/0 lens 488/448 e 0 to 0 dl 1645060073 ref 1 fl Interpret:/0/0 rc 0/0
[13834090.279243] LustreError: 228403:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 34 previous similar messages
[13834090.289277] Lustre: oak-OST0143: Bulk IO write error with 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1), client will retry: rc = -110
[13834090.302737] Lustre: Skipped 45 previous similar messages
[13834141.663731] LustreError: 162716:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(24576)  req@ffff914b91e9c050 x1715532190355392/t0(0) o3->5a5caa86-68df-f806-7e6f-05440e95a4ce@10.210.12.43@tcp1:393/0 lens 488/440 e 0 to 0 dl 1645060078 ref 1 fl Interpret:/0/0 rc 0/0
[13834141.688593] LustreError: 162716:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 10 previous similar messages
[13834141.698417] Lustre: oak-OST0121: Bulk IO read error with 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1), client will retry: rc -110
[13834141.711617] Lustre: Skipped 19 previous similar messages
[13834434.130306] LustreError: 137-5: oak-OST0132_UUID: not available for connect from 10.210.12.64@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13834434.147932] LustreError: Skipped 7 previous similar messages
[13834442.934266] Lustre: oak-OST011b: Connection restored to ba401614-5a08-e8d7-0632-52ba9a030f78 (at 10.51.12.15@o2ib3)
[13834442.944937] Lustre: Skipped 1453 previous similar messages
[13834596.846729] LustreError: 162692:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff918c201a3050 x1714909858735744/t0(0) o4->274e2965-2f8d-f91e-8ba9-3f4bdaf24c8b@10.210.12.119@tcp1:77/0 lens 488/448 e 0 to 0 dl 1645060517 ref 1 fl Interpret:/0/0 rc 0/0
[13834596.872627] LustreError: 162692:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[13834612.024486] Lustre: oak-OST012d: Client 274e2965-2f8d-f91e-8ba9-3f4bdaf24c8b (at 10.210.12.119@tcp1) reconnecting
[13834612.035071] Lustre: Skipped 95 previous similar messages
[13834692.659384] LustreError: 21584:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(2195456)  req@ffff917eea147050 x1724766329266816/t0(0) o4->479db7cf-e752-7ee3-6b36-d1f694b751cf@10.210.12.47@tcp1:191/0 lens 488/448 e 0 to 0 dl 1645060631 ref 1 fl Interpret:/0/0 rc 0/0
[13834692.685224] Lustre: oak-OST0129: Bulk IO write error with 479db7cf-e752-7ee3-6b36-d1f694b751cf (at 10.210.12.47@tcp1), client will retry: rc = -110
[13834692.698691] Lustre: Skipped 14 previous similar messages
[13834789.783564] Lustre: 229345:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1645060529/real 1645060529]  req@ffff9160b0750000 x1710530519039680/t0(0) o104->oak-OST0139@10.210.12.29@tcp1:15/16 lens 296/224 e 0 to 1 dl 1645060682 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
[13834812.429823] LustreError: 160891:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff91f18e285850 x1715042939807232/t0(0) o3->80374d70-7d99-f68a-b58b-86c0b7514d6a@10.210.12.65@tcp1:307/0 lens 488/440 e 0 to 0 dl 1645060747 ref 1 fl Interpret:/0/0 rc 0/0
[13834812.429825] LustreError: 244099:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff91ea66653050 x1715042939807360/t0(0) o3->80374d70-7d99-f68a-b58b-86c0b7514d6a@10.210.12.65@tcp1:307/0 lens 488/440 e 0 to 0 dl 1645060747 ref 1 fl Interpret:/0/0 rc 0/0
[13834812.429828] LustreError: 244099:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 8 previous similar messages
[13834812.429841] Lustre: oak-OST012b: Bulk IO read error with 80374d70-7d99-f68a-b58b-86c0b7514d6a (at 10.210.12.65@tcp1), client will retry: rc -110
[13834812.429842] Lustre: Skipped 10 previous similar messages
[13834814.374786] LustreError: 21588:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff916daa3b0850 x1715239752345984/t0(0) o3->e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda@10.210.12.79@tcp1:364/0 lens 488/440 e 0 to 0 dl 1645060804 ref 1 fl Interpret:/0/0 rc 0/0
[13834814.399046] LustreError: 21588:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 14 previous similar messages
[13835041.615685] Lustre: oak-OST0131: Connection restored to 2080b6ff-3ff3-dcca-b09d-676ef5b02916 (at 10.51.6.26@o2ib3)
[13835041.626682] Lustre: Skipped 1988 previous similar messages
[13835378.895136] Lustre: oak-OST013d: Client c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1) reconnecting
[13835378.905554] Lustre: Skipped 172 previous similar messages
[13835379.603108] Lustre: oak-OST0141: Bulk IO write error with c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1), client will retry: rc = -110
[13835379.616632] Lustre: Skipped 4 previous similar messages
[13835435.225922] LustreError: 160938:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff91b5c11a7850 x1715009650792064/t0(0) o3->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:169/0 lens 488/440 e 0 to 0 dl 1645061364 ref 1 fl Interpret:/0/0 rc 0/0
[13835435.226332] LustreError: 160940:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(2104448)  req@ffff91b1cd10f050 x1715427065176128/t0(0) o4->0d1c8726-ce4c-fcc8-6c1b-7179039cabd2@10.210.12.8@tcp1:171/0 lens 488/448 e 0 to 0 dl 1645061366 ref 1 fl Interpret:/0/0 rc 0/0
[13835435.276701] Lustre: oak-OST011d: Bulk IO read error with c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1), client will retry: rc -110
[13835435.289948] Lustre: Skipped 7 previous similar messages
[13835458.368958] LustreError: 137-5: oak-OST013a_UUID: not available for connect from 10.210.12.8@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13835550.906699] LustreError: 162710:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9158ea66e050 x1715009655308928/t0(0) o4->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:345/0 lens 488/448 e 0 to 0 dl 1645061540 ref 1 fl Interpret:/0/0 rc 0/0
[13835550.931132] LustreError: 162710:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 10 previous similar messages
[13835642.682710] Lustre: oak-OST0141: Connection restored to 55ade167-ee14-7aa8-e90b-b61cc094ea5c (at 10.50.6.56@o2ib2)
[13835642.693391] Lustre: Skipped 824 previous similar messages
[13835745.635254] LustreError: 160922:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(17748)  req@ffff91befae87050 x1722879286613056/t0(0) o4->8e0ec672-3964-d47f-f8e6-cd23bd41e731@10.51.14.23@o2ib3:469/0 lens 488/448 e 0 to 0 dl 1645061664 ref 1 fl Interpret:/0/0 rc 0/0
[13835745.660374] LustreError: 160922:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 9 previous similar messages
[13836032.411480] Lustre: oak-OST0143: Client bed08025-01aa-dea1-aa90-bead26dab2fb (at 10.50.17.21@o2ib2) reconnecting
[13836032.421883] Lustre: Skipped 83 previous similar messages
[13836243.923233] Lustre: oak-OST011f: Connection restored to f9505472-7ef8-fe9c-57f9-b043774a6cbb (at 10.50.4.61@o2ib2)
[13836243.935380] Lustre: Skipped 983 previous similar messages
[13836320.525076] LustreError: 162686:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff917cd518c850 x1715043047193344/t0(0) o3->80374d70-7d99-f68a-b58b-86c0b7514d6a@10.210.12.65@tcp1:314/0 lens 488/440 e 0 to 0 dl 1645062264 ref 1 fl Interpret:/0/0 rc 0/0
[13836320.525094] Lustre: oak-OST012b: Bulk IO read error with 80374d70-7d99-f68a-b58b-86c0b7514d6a (at 10.210.12.65@tcp1), client will retry: rc -110
[13836320.525095] Lustre: Skipped 2 previous similar messages
[13836320.568849] LustreError: 162686:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 3 previous similar messages
[13836321.225362] LustreError: 162673:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff918eef58e050 x1714949107502656/t0(0) o4->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:365/0 lens 488/448 e 0 to 0 dl 1645062315 ref 1 fl Interpret:/0/0 rc 0/0
[13836321.251683] Lustre: oak-OST014b: Bulk IO write error with 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1), client will retry: rc = -110
[13836321.265129] Lustre: Skipped 13 previous similar messages
[13836368.429651] LustreError: 162675:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91735b422850 x1715009730277632/t0(0) o4->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:360/0 lens 488/448 e 0 to 0 dl 1645062310 ref 1 fl Interpret:/0/0 rc 0/0
[13836368.455787] LustreError: 162675:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 3 previous similar messages
[13836401.896794] LustreError: 137-5: oak-OST013e_UUID: not available for connect from 10.210.12.79@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13836403.629370] LustreError: 137-5: oak-OST0112_UUID: not available for connect from 10.210.12.7@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13836403.646904] LustreError: Skipped 1 previous similar message
[13836699.910748] Lustre: oak-OST0113: Client 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1) reconnecting
[13836699.921158] Lustre: Skipped 283 previous similar messages
[13836842.484257] Lustre: oak-OST012d: Connection restored to 916093a3-51ee-71cf-ab61-9d8121d32b7c (at 10.51.15.11@o2ib3)
[13836842.494929] Lustre: Skipped 1927 previous similar messages
[13836874.923911] LustreError: 162677:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff91663b061050 x1724766377495872/t0(0) o4->479db7cf-e752-7ee3-6b36-d1f694b751cf@10.210.12.47@tcp1:165/0 lens 488/448 e 0 to 0 dl 1645062870 ref 1 fl Interpret:/2/0 rc 0/0
[13837441.514742] Lustre: oak-OST0131: Connection restored to aab151b2-b669-9bb0-5460-87842297680c (at 10.50.9.57@o2ib2)
[13837441.525783] Lustre: Skipped 845 previous similar messages
[13837446.306991] LustreError: 21592:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff9159641ed050 x1714949173061632/t0(0) o4->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:668/0 lens 488/448 e 0 to 0 dl 1645063373 ref 1 fl Interpret:/0/0 rc 0/0
[13837446.332921] Lustre: oak-OST0133: Bulk IO write error with 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1), client will retry: rc = -110
[13837446.346363] Lustre: Skipped 16 previous similar messages
[13837464.056794] Lustre: oak-OST0133: Client 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1) reconnecting
[13837464.067754] Lustre: Skipped 61 previous similar messages
[13838040.211260] Lustre: oak-OST0125: Connection restored to e2e0a5b4-fb83-0b58-6d5a-dfe8d29c6078 (at 10.50.1.39@o2ib2)
[13838040.221858] Lustre: Skipped 2128 previous similar messages
[13838359.575980] Lustre: oak-OST0125: Client 1b588188-9255-1b9f-1308-154ba9e467a3 (at 10.210.12.7@tcp1) reconnecting
[13838359.586300] Lustre: Skipped 31 previous similar messages
[13838360.072205] LustreError: 253937:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91d92d430050 x1715350181149312/t0(0) o4->1b588188-9255-1b9f-1308-154ba9e467a3@10.210.12.7@tcp1:139/0 lens 488/448 e 0 to 0 dl 1645064354 ref 1 fl Interpret:/0/0 rc 0/0
[13838360.096598] LustreError: 253937:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 17 previous similar messages
[13838360.106697] Lustre: oak-OST0125: Bulk IO write error with 1b588188-9255-1b9f-1308-154ba9e467a3 (at 10.210.12.7@tcp1), client will retry: rc = -110
[13838427.442479] LustreError: 21599:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff9146109af050 x1715106493838272/t0(0) o3->b4fca75a-2d54-9bf2-331f-67d0c52e5ff2@10.210.12.73@tcp1:142/0 lens 488/440 e 0 to 0 dl 1645064357 ref 1 fl Interpret:/0/0 rc 0/0
[13838427.442481] LustreError: 162715:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(3031040)  req@ffff9145c5fd9050 x1715106493839040/t0(0) o4->b4fca75a-2d54-9bf2-331f-67d0c52e5ff2@10.210.12.73@tcp1:142/0 lens 504/448 e 0 to 0 dl 1645064357 ref 1 fl Interpret:/0/0 rc 0/0
[13838427.442490] Lustre: oak-OST0145: Bulk IO read error with b4fca75a-2d54-9bf2-331f-67d0c52e5ff2 (at 10.210.12.73@tcp1), client will retry: rc -110
[13838427.442491] Lustre: Skipped 8 previous similar messages
[13838427.511897] LustreError: 21599:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 2 previous similar messages
[13838439.675683] LustreError: 137-5: oak-OST0132_UUID: not available for connect from 10.210.12.7@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13838442.047487] LustreError: 137-5: oak-OST0114_UUID: not available for connect from 10.210.12.31@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13838639.947962] Lustre: oak-OST013d: Connection restored to  (at 10.51.4.57@o2ib3)
[13838639.955453] Lustre: Skipped 1365 previous similar messages
[13839238.525227] Lustre: oak-OST012d: Connection restored to  (at 10.50.3.65@o2ib2)
[13839238.532713] Lustre: Skipped 1011 previous similar messages
[13839289.725808] LustreError: 243544:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff916339db1050 x1714972009026176/t0(0) o3->5d4d5527-a52f-bc68-278b-aa990b8609d6@10.210.12.59@tcp1:258/0 lens 488/440 e 0 to 0 dl 1645065228 ref 1 fl Interpret:/0/0 rc 0/0
[13839289.732799] Lustre: oak-OST0129: Bulk IO read error with 5d4d5527-a52f-bc68-278b-aa990b8609d6 (at 10.210.12.59@tcp1), client will retry: rc -110
[13839289.732801] LustreError: 160895:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91ef242e2850 x1714949406047872/t0(0) o4->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:258/0 lens 488/448 e 0 to 0 dl 1645065228 ref 1 fl Interpret:/0/0 rc 0/0
[13839289.732801] Lustre: Skipped 1 previous similar message
[13839289.732802] LustreError: 160895:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[13839289.732995] Lustre: oak-OST0145: Bulk IO write error with 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1), client will retry: rc = -110
[13839289.732996] Lustre: Skipped 3 previous similar messages
[13839289.824507] LustreError: 243544:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 1 previous similar message
[13839311.884941] LustreError: 137-5: oak-OST0114_UUID: not available for connect from 10.210.12.31@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13839312.277905] Lustre: oak-OST0145: Client 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1) reconnecting
[13839312.288376] Lustre: Skipped 85 previous similar messages
[13839312.505279] LustreError: 137-5: oak-OST0134_UUID: not available for connect from 10.210.12.44@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13839347.034648] LustreError: 160910:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91cfd8d26050 x1715090781251840/t0(0) o4->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:378/0 lens 488/448 e 0 to 0 dl 1645065348 ref 1 fl Interpret:/0/0 rc 0/0
[13839347.059062] LustreError: 160910:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[13839409.480767] LustreError: 162696:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff9160a5a15850 x1715090781178944/t0(0) o3->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:369/0 lens 488/440 e 0 to 0 dl 1645065339 ref 1 fl Interpret:/0/0 rc 0/0
[13839409.486345] Lustre: oak-OST0141: Bulk IO read error with 5d8bb8d9-e5c7-334f-ec8d-f10452c309b5 (at 10.210.12.66@tcp1), client will retry: rc -110
[13839409.486346] Lustre: Skipped 1 previous similar message
[13839409.487302] LustreError: 228401:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91c2dc033850 x1715090781208000/t0(0) o4->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:374/0 lens 488/448 e 0 to 0 dl 1645065344 ref 1 fl Interpret:/0/0 rc 0/0
[13839409.487303] LustreError: 228401:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[13839409.559930] LustreError: 162696:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 1 previous similar message
[13839425.223512] LustreError: 137-5: oak-OST011a_UUID: not available for connect from 10.210.12.66@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13839425.241170] LustreError: Skipped 2 previous similar messages
[13839753.870788] LustreError: 160919:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91e472305850 x1714972022763520/t0(0) o4->5d4d5527-a52f-bc68-278b-aa990b8609d6@10.210.12.59@tcp1:27/0 lens 488/448 e 0 to 0 dl 1645065752 ref 1 fl Interpret:/0/0 rc 0/0
[13839808.037165] LustreError: 162694:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91853854e850 x1715350224400128/t0(0) o4->1b588188-9255-1b9f-1308-154ba9e467a3@10.210.12.7@tcp1:84/0 lens 488/448 e 0 to 0 dl 1645065809 ref 1 fl Interpret:/0/0 rc 0/0
[13839808.061429] LustreError: 162694:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[13839816.658710] LustreError: 160955:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91dc17ee9850 x1724766390194496/t0(0) o4->479db7cf-e752-7ee3-6b36-d1f694b751cf@10.210.12.47@tcp1:24/0 lens 488/448 e 0 to 0 dl 1645065749 ref 1 fl Interpret:/0/0 rc 0/0
[13839816.659699] LustreError: 244098:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff91f13773f850 x1724766390242112/t0(0) o3->479db7cf-e752-7ee3-6b36-d1f694b751cf@10.210.12.47@tcp1:30/0 lens 488/440 e 0 to 0 dl 1645065755 ref 1 fl Interpret:/0/0 rc 0/0
[13839816.659878] Lustre: oak-OST0143: Bulk IO read error with 479db7cf-e752-7ee3-6b36-d1f694b751cf (at 10.210.12.47@tcp1), client will retry: rc -110
[13839816.659879] Lustre: Skipped 1 previous similar message
[13839831.563487] LustreError: 137-5: oak-OST0126_UUID: not available for connect from 10.210.12.47@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13839831.581081] LustreError: Skipped 1 previous similar message
[13839839.672064] Lustre: oak-OST0149: Connection restored to 3a57957d-0251-b0e1-832e-0703d2b195aa (at 10.50.1.40@o2ib2)
[13839839.682665] Lustre: Skipped 657 previous similar messages
[13839864.565939] LustreError: 243541:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff917ede7dd850 x1715350224474432/t0(0) o4->1b588188-9255-1b9f-1308-154ba9e467a3@10.210.12.7@tcp1:86/0 lens 488/448 e 0 to 0 dl 1645065811 ref 1 fl Interpret:/0/0 rc 0/0
[13839887.049938] LustreError: 137-5: oak-OST013a_UUID: not available for connect from 10.210.12.7@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13839918.772394] Lustre: oak-OST0125: Client 479db7cf-e752-7ee3-6b36-d1f694b751cf (at 10.210.12.47@tcp1) reconnecting
[13839918.782805] Lustre: Skipped 144 previous similar messages
[13840414.503742] LustreError: 160933:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff91ebe3a3d050 x1715106787713280/t0(0) o3->b4fca75a-2d54-9bf2-331f-67d0c52e5ff2@10.210.12.73@tcp1:643/0 lens 488/440 e 0 to 0 dl 1645066368 ref 1 fl Interpret:/0/0 rc 0/0
[13840414.503958] LustreError: 160911:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91b8327d9050 x1714968617751744/t0(0) o4->5afa23c2-1fd5-9030-38ce-b03a359082fe@10.210.12.75@tcp1:644/0 lens 488/448 e 0 to 0 dl 1645066369 ref 1 fl Interpret:/0/0 rc 0/0
[13840414.504247] Lustre: oak-OST0131: Bulk IO write error with 5afa23c2-1fd5-9030-38ce-b03a359082fe (at 10.210.12.75@tcp1), client will retry: rc = -110
[13840414.504248] Lustre: Skipped 9 previous similar messages
[13840414.530202] Lustre: oak-OST013b: Bulk IO read error with 5afa23c2-1fd5-9030-38ce-b03a359082fe (at 10.210.12.75@tcp1), client will retry: rc -110
[13840414.530204] Lustre: Skipped 2 previous similar messages
[13840414.592119] LustreError: 160933:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 4 previous similar messages
[13840438.306043] Lustre: oak-OST0133: Connection restored to c9bb8838-a93c-f321-c22f-51d67c2ac5da (at 10.51.1.69@o2ib3)
[13840438.316619] Lustre: Skipped 589 previous similar messages
[13840522.831316] Lustre: oak-OST0147: Client 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1) reconnecting
[13840522.841725] Lustre: Skipped 82 previous similar messages
[13840523.468446] LustreError: 228403:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91850c02c050 x1715532527742272/t0(0) o4->5a5caa86-68df-f806-7e6f-05440e95a4ce@10.210.12.43@tcp1:47/0 lens 488/448 e 0 to 0 dl 1645066527 ref 1 fl Interpret:/0/0 rc 0/0
[13840523.492780] LustreError: 228403:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[13840582.168778] LustreError: 162672:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff918f29361850 x1715010070828800/t0(0) o3->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:41/0 lens 488/440 e 0 to 0 dl 1645066521 ref 1 fl Interpret:/0/0 rc 0/0
[13840582.168790] Lustre: oak-OST0113: Bulk IO read error with c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1), client will retry: rc -110
[13840582.168791] Lustre: Skipped 2 previous similar messages
[13840582.168912] LustreError: 162714:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91610b853050 x1715010070829568/t0(0) o4->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:42/0 lens 488/448 e 0 to 0 dl 1645066522 ref 1 fl Interpret:/0/0 rc 0/0
[13840582.168913] LustreError: 162714:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 3 previous similar messages
[13840582.247984] LustreError: 162672:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 2 previous similar messages
[13840602.136377] LustreError: 137-5: oak-OST012e_UUID: not available for connect from 10.210.12.25@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13840602.153998] LustreError: Skipped 1 previous similar message
[13840669.118452] LustreError: 243444:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff9146e1af2850 x1715427169574848/t0(0) o3->0d1c8726-ce4c-fcc8-6c1b-7179039cabd2@10.210.12.8@tcp1:155/0 lens 488/440 e 0 to 0 dl 1645066635 ref 1 fl Interpret:/0/0 rc 0/0
[13840669.142700] LustreError: 243444:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[13840743.065750] LustreError: 160899:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9194d6b8e850 x1715350238769216/t0(0) o4->1b588188-9255-1b9f-1308-154ba9e467a3@10.210.12.7@tcp1:264/0 lens 488/448 e 0 to 0 dl 1645066744 ref 1 fl Interpret:/0/0 rc 0/0
[13840797.751997] LustreError: 244098:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff91ea49b0a850 x1715106796948608/t0(0) o3->b4fca75a-2d54-9bf2-331f-67d0c52e5ff2@10.210.12.73@tcp1:262/0 lens 488/440 e 0 to 0 dl 1645066742 ref 1 fl Interpret:/0/0 rc 0/0
[13840797.752090] LustreError: 160944:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff919e7b25a050 x1715106796949504/t0(0) o4->b4fca75a-2d54-9bf2-331f-67d0c52e5ff2@10.210.12.73@tcp1:262/0 lens 488/448 e 0 to 0 dl 1645066742 ref 1 fl Interpret:/0/0 rc 0/0
[13840797.752188] Lustre: oak-OST0115: Bulk IO read error with b4fca75a-2d54-9bf2-331f-67d0c52e5ff2 (at 10.210.12.73@tcp1), client will retry: rc -110
[13840797.752189] Lustre: Skipped 3 previous similar messages
[13840797.821488] LustreError: 244098:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 4 previous similar messages
[13840822.995115] LustreError: 137-5: oak-OST014e_UUID: not available for connect from 10.210.12.73@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13840949.963073] Lustre: oak-OST012f: haven't heard from client 1b588188-9255-1b9f-1308-154ba9e467a3 (at 10.210.12.7@tcp1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff918a970b3800, cur 1645066858 expire 1645066708 last 1645066631
[13840949.985050] Lustre: Skipped 3 previous similar messages
[13841036.989744] Lustre: oak-OST0149: Connection restored to  (at 10.50.5.21@o2ib2)
[13841036.997214] Lustre: Skipped 835 previous similar messages
[13841169.487405] Lustre: oak-OST0113: Client b69058ea-844c-325f-f63b-7347539374f7 (at 10.210.12.51@tcp1) reconnecting
[13841169.497812] Lustre: Skipped 228 previous similar messages
[13841612.127624] LustreError: 21588:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91407d1ab850 x1715106825868288/t0(0) o4->b4fca75a-2d54-9bf2-331f-67d0c52e5ff2@10.210.12.73@tcp1:327/0 lens 488/448 e 0 to 0 dl 1645067562 ref 1 fl Interpret:/0/0 rc 0/0
[13841612.128185] Lustre: oak-OST013b: Bulk IO write error with b4fca75a-2d54-9bf2-331f-67d0c52e5ff2 (at 10.210.12.73@tcp1), client will retry: rc = -110
[13841612.128187] Lustre: Skipped 8 previous similar messages
[13841612.172433] LustreError: 21588:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[13841636.390756] Lustre: oak-OST0111: Connection restored to 5e4bce85-4d6e-90c6-523c-4bdf35e8bc4a (at 10.210.12.7@tcp1)
[13841636.401337] Lustre: Skipped 604 previous similar messages
[13841638.618671] LustreError: 137-5: oak-OST014e_UUID: not available for connect from 10.210.12.73@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13841735.313237] LustreError: 160901:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91c81f268050 x1714968657100544/t0(0) o4->5afa23c2-1fd5-9030-38ce-b03a359082fe@10.210.12.75@tcp1:503/0 lens 488/448 e 0 to 0 dl 1645067738 ref 1 fl Interpret:/0/0 rc 0/0
[13841735.337661] LustreError: 160901:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[13841739.178848] LustreError: 160914:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff919879763850 x1715106829447552/t0(0) o3->b4fca75a-2d54-9bf2-331f-67d0c52e5ff2@10.210.12.73@tcp1:504/0 lens 488/440 e 0 to 0 dl 1645067739 ref 1 fl Interpret:/0/0 rc 0/0
[13841739.203177] LustreError: 160914:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 5 previous similar messages
[13841739.212966] Lustre: oak-OST0115: Bulk IO read error with b4fca75a-2d54-9bf2-331f-67d0c52e5ff2 (at 10.210.12.73@tcp1), client will retry: rc -110
[13841739.226144] Lustre: Skipped 4 previous similar messages
[13841803.733564] LustreError: 243453:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff91d91aa2b050 x1715106829469760/t0(0) o3->b4fca75a-2d54-9bf2-331f-67d0c52e5ff2@10.210.12.73@tcp1:504/0 lens 488/440 e 0 to 0 dl 1645067739 ref 1 fl Interpret:/0/0 rc 0/0
[13841803.733590] LustreError: 160906:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91d91aa28850 x1715106829469888/t0(0) o4->b4fca75a-2d54-9bf2-331f-67d0c52e5ff2@10.210.12.73@tcp1:504/0 lens 488/448 e 0 to 0 dl 1645067739 ref 1 fl Interpret:/0/0 rc 0/0
[13841803.734006] Lustre: oak-OST0111: Bulk IO read error with b4fca75a-2d54-9bf2-331f-67d0c52e5ff2 (at 10.210.12.73@tcp1), client will retry: rc -110
[13841803.797486] LustreError: 243453:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 2 previous similar messages
[13841814.946145] Lustre: oak-OST014b: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[13841814.956717] Lustre: Skipped 55 previous similar messages
[13842235.068246] Lustre: oak-OST0145: Connection restored to  (at 10.50.4.48@o2ib2)
[13842235.075784] Lustre: Skipped 796 previous similar messages
[13842442.445551] Lustre: oak-OST011f: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[13842442.455961] Lustre: Skipped 98 previous similar messages
[13842454.648066] LustreError: 162702:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff917fc5373850 x1714968666136960/t0(0) o4->5afa23c2-1fd5-9030-38ce-b03a359082fe@10.210.12.75@tcp1:473/0 lens 488/448 e 0 to 0 dl 1645068463 ref 1 fl Interpret:/0/0 rc 0/0
[13842454.672758] Lustre: oak-OST013b: Bulk IO write error with 5afa23c2-1fd5-9030-38ce-b03a359082fe (at 10.210.12.75@tcp1), client will retry: rc = -110
[13842454.686192] Lustre: Skipped 9 previous similar messages
[13842455.968799] LustreError: 162716:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff915287334850 x1715061463127360/t0(0) o4->bb55a01e-02a3-cbba-5b76-e478fd40967c@10.210.12.54@tcp1:468/0 lens 488/448 e 0 to 0 dl 1645068458 ref 1 fl Interpret:/0/0 rc 0/0
[13842455.994423] LustreError: 162716:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[13842457.159906] LustreError: 127358:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff919a7b893850 x1715083659125504/t0(0) o4->65e9b8a8-b461-24e0-da12-5b6939f0b07c@10.210.12.56@tcp1:468/0 lens 488/448 e 0 to 0 dl 1645068458 ref 1 fl Interpret:/0/0 rc 0/0
[13842457.184320] LustreError: 127358:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 10 previous similar messages
[13842521.246206] LustreError: 21595:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(28672)  req@ffff91867b739850 x1715061463140416/t0(0) o3->bb55a01e-02a3-cbba-5b76-e478fd40967c@10.210.12.54@tcp1:468/0 lens 488/440 e 0 to 0 dl 1645068458 ref 1 fl Interpret:/0/0 rc 0/0
[13842521.246355] Lustre: oak-OST0113: Bulk IO read error with c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1), client will retry: rc -110
[13842521.246356] Lustre: Skipped 2 previous similar messages
[13842521.290034] LustreError: 21595:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 4 previous similar messages
[13842540.274906] LustreError: 243542:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9185a8029850 x1715091035449344/t0(0) o4->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:554/0 lens 488/448 e 0 to 0 dl 1645068544 ref 1 fl Interpret:/0/0 rc 0/0
[13842540.299325] LustreError: 243542:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 8 previous similar messages
[13842540.372435] LustreError: 162704:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk READ failed: rc -107  req@ffff91489f026050 x1714906456768448/t0(0) o3->a166a60d-30d5-1e72-b799-745de1d3b307@10.210.12.125@tcp1:559/0 lens 488/440 e 0 to 0 dl 1645068549 ref 1 fl Interpret:/0/0 rc 0/0
[13842540.397194] Lustre: oak-OST014b: Bulk IO read error with a166a60d-30d5-1e72-b799-745de1d3b307 (at 10.210.12.125@tcp1), client will retry: rc -107
[13842540.410506] Lustre: Skipped 4 previous similar messages
[13842593.111506] LustreError: 162679:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 3145728(4194304)  req@ffff916ad9e22050 x1715091035429952/t0(0) o3->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:553/0 lens 1016/440 e 0 to 0 dl 1645068543 ref 1 fl Interpret:/0/0 rc 0/0
[13842593.111603] Lustre: oak-OST0141: Bulk IO read error with 5d8bb8d9-e5c7-334f-ec8d-f10452c309b5 (at 10.210.12.66@tcp1), client will retry: rc -110
[13842593.150296] LustreError: 162679:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 7 previous similar messages
[13842631.180978] LustreError: 160946:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff91b102160850 x1714968671768896/t0(0) o3->5afa23c2-1fd5-9030-38ce-b03a359082fe@10.210.12.75@tcp1:645/0 lens 488/440 e 0 to 0 dl 1645068635 ref 1 fl Interpret:/0/0 rc 0/0
[13842631.205343] LustreError: 160946:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 18 previous similar messages
[13842631.215294] Lustre: oak-OST011b: Bulk IO read error with 5afa23c2-1fd5-9030-38ce-b03a359082fe (at 10.210.12.75@tcp1), client will retry: rc -110
[13842631.228729] Lustre: Skipped 7 previous similar messages
[13842653.515693] LustreError: 160929:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91b102162050 x1715532576149824/t0(0) o4->5a5caa86-68df-f806-7e6f-05440e95a4ce@10.210.12.43@tcp1:668/0 lens 488/448 e 0 to 0 dl 1645068658 ref 1 fl Interpret:/0/0 rc 0/0
[13842653.540131] LustreError: 160929:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 4 previous similar messages
[13842785.005128] LustreError: 243526:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff915044aaa050 x1715083673134912/t0(0) o4->65e9b8a8-b461-24e0-da12-5b6939f0b07c@10.210.12.56@tcp1:48/0 lens 488/448 e 0 to 0 dl 1645068793 ref 1 fl Interpret:/0/0 rc 0/0
[13842787.206875] Lustre: oak-OST012f: Bulk IO read error with 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1), client will retry: rc -110
[13842787.220066] Lustre: Skipped 6 previous similar messages
[13842787.582931] LustreError: 26434:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff9136d1fbb800
[13842787.593995] LustreError: 26436:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff9136d1fbb800
[13842787.605009] LustreError: 26435:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff9136d1fbb800
[13842787.616037] LustreError: 26433:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff9136d1fbb800
[13842787.663212] LustreError: 26433:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff913d84d45c00
[13842787.674239] LustreError: 26433:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff913d84d45c00
[13842787.687478] LustreError: 26435:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff913673bc2c00
[13842787.698515] LustreError: 26435:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff913673bc2c00
[13842834.832127] Lustre: oak-OST0135: Connection restored to 43a7bf00-2ec1-3fdd-5cf6-946289157379 (at 10.50.10.69@o2ib2)
[13842834.843155] Lustre: Skipped 788 previous similar messages
[13842868.978050] LustreError: 21600:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff915c4f89e850 x1714978459253696/t0(0) o4->d4faaef8-44e2-b05f-6116-d9c30b2e4e11@10.210.12.60@tcp1:133/0 lens 488/448 e 0 to 0 dl 1645068878 ref 1 fl Interpret:/0/0 rc 0/0
[13842869.002385] LustreError: 21600:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 7 previous similar messages
[13842928.432566] LustreError: 21588:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(24576)  req@ffff918d42fa3850 x1715010151026816/t0(0) o3->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:128/0 lens 488/440 e 0 to 0 dl 1645068873 ref 1 fl Interpret:/0/0 rc 0/0
[13842928.434571] LustreError: 21609:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(4194304)  req@ffff917d20343050 x1714999305549120/t0(0) o4->6f45695d-f173-ee1d-e6cb-d38dad7e0879@10.210.12.74@tcp1:143/0 lens 488/448 e 0 to 0 dl 1645068888 ref 1 fl Interpret:/0/0 rc 0/0
[13842928.434572] LustreError: 21609:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[13842928.492523] LustreError: 21588:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 12 previous similar messages
[13842947.592781] LustreError: 137-5: oak-OST014a_UUID: not available for connect from 10.210.12.60@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13842947.610387] LustreError: Skipped 1 previous similar message
[13842957.325145] LustreError: 137-5: oak-OST0120_UUID: not available for connect from 10.210.12.74@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13842961.043351] LustreError: 137-5: oak-OST0146_UUID: not available for connect from 10.210.12.74@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13843001.814214] LustreError: 243456:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91a7363b1850 x1714911376255936/t0(0) o4->e5a735ca-191d-07c9-fcef-626c0b3a28a8@10.210.12.113@tcp1:264/0 lens 488/448 e 0 to 0 dl 1645069009 ref 1 fl Interpret:/0/0 rc 0/0
[13843001.838761] LustreError: 243456:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 65 previous similar messages
[13843001.945060] LustreError: 26441:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff9196c9739800
[13843001.956266] LustreError: 26441:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff9196c9739800
[13843001.967277] LustreError: 26437:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff9196c9739800
[13843001.978296] LustreError: 26441:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff9196c9739800
[13843001.989325] LustreError: 26441:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff9136ba7c1c00
[13843002.000359] LustreError: 26441:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff9136ba7c1c00
[13843002.011397] LustreError: 26441:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff9136ba7c1c00
[13843002.022425] LustreError: 26441:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff9136ba7c1c00
[13843041.292648] Lustre: oak-OST0141: Client 5d4d5527-a52f-bc68-278b-aa990b8609d6 (at 10.210.12.59@tcp1) reconnecting
[13843041.303136] Lustre: Skipped 156 previous similar messages
[13843048.195020] LustreError: 228831:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 3145728(4194304)  req@ffff91c928724050 x1714914749233536/t0(0) o3->5865c071-3198-0848-a51e-ecc3ec62e180@10.210.12.127@tcp1:264/0 lens 488/440 e 0 to 0 dl 1645069009 ref 1 fl Interpret:/0/0 rc 0/0
[13843048.195147] Lustre: oak-OST0133: Bulk IO read error with cc381d20-202e-f264-43c3-938610d60653 (at 10.210.12.58@tcp1), client will retry: rc -110
[13843048.195148] Lustre: Skipped 22 previous similar messages
[13843048.239445] LustreError: 228831:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 2 previous similar messages
[13843104.625227] Lustre: oak-OST0139: Bulk IO write error with 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1), client will retry: rc = -110
[13843104.638744] Lustre: Skipped 123 previous similar messages
[13843154.210849] LustreError: 162707:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff914e291d6850 x1715083703934336/t0(0) o4->65e9b8a8-b461-24e0-da12-5b6939f0b07c@10.210.12.56@tcp1:418/0 lens 488/448 e 0 to 0 dl 1645069163 ref 1 fl Interpret:/0/0 rc 0/0
[13843154.235269] LustreError: 162707:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 129 previous similar messages
[13843167.965571] LustreError: 199271:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff91701454c050 x1715372171760832/t0(0) o3->5a4881be-a0cb-e632-509b-2e15c534b21a@10.210.12.9@tcp1:382/0 lens 488/440 e 0 to 0 dl 1645069127 ref 1 fl Interpret:/0/0 rc 0/0
[13843167.967134] LustreError: 253952:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91e5a56b2850 x1716549866369408/t0(0) o4->6bab8d10-30ff-c1d2-bbf5-1fd0884ac647@10.210.12.57@tcp1:378/0 lens 488/448 e 0 to 0 dl 1645069123 ref 1 fl Interpret:/0/0 rc 0/0
[13843167.967135] LustreError: 253952:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 3 previous similar messages
[13843168.026943] LustreError: 199271:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 23 previous similar messages
[13843199.819484] LustreError: 137-5: oak-OST0128_UUID: not available for connect from 10.210.12.57@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13843433.988457] Lustre: oak-OST0133: Connection restored to 72ea4c6f-962e-8dd1-1e8b-23b8f5f6f051 (at 10.50.1.25@o2ib2)
[13843433.999043] Lustre: Skipped 1801 previous similar messages
[13843478.923417] LustreError: 199272:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff91e23534a850 x1715083728541248/t0(0) o3->65e9b8a8-b461-24e0-da12-5b6939f0b07c@10.210.12.56@tcp1:746/0 lens 488/440 e 0 to 0 dl 1645069491 ref 1 fl Interpret:/0/0 rc 0/0
[13843478.947770] LustreError: 199272:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 8 previous similar messages
[13843478.957994] Lustre: oak-OST012f: Bulk IO read error with 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1), client will retry: rc -110
[13843478.971276] Lustre: Skipped 34 previous similar messages
[13843647.531398] Lustre: oak-OST0113: Client 6ca637d5-9e95-16bf-f08c-446745633d32 (at 10.210.12.129@tcp1) reconnecting
[13843647.541889] Lustre: Skipped 377 previous similar messages
[13843767.784239] Lustre: oak-OST0139: Bulk IO write error with 0d4d52ae-6d60-d275-1404-c9b4d99e0974 (at 10.210.12.109@tcp1), client will retry: rc = -110
[13843767.797843] Lustre: Skipped 124 previous similar messages
[13843794.136003] LustreError: 21603:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff918f8ec41850 x1715023233586880/t0(0) o4->6d01c866-9ec9-8f84-4f86-ee1db83afc97@10.210.12.61@tcp1:307/0 lens 488/448 e 0 to 0 dl 1645069807 ref 1 fl Interpret:/2/0 rc 0/0
[13843838.693330] LustreError: 243332:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 3145728(4194304)  req@ffff9162c8654050 x1715106893397376/t0(0) o3->b4fca75a-2d54-9bf2-331f-67d0c52e5ff2@10.210.12.73@tcp1:297/0 lens 488/440 e 0 to 0 dl 1645069797 ref 1 fl Interpret:/0/0 rc 0/0
[13843838.718891] LustreError: 243332:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 5 previous similar messages
[13844033.086959] Lustre: oak-OST012d: Connection restored to 7bd5e25d-92e5-9f8c-22fa-8f4c98d09593 (at 10.50.10.6@o2ib2)
[13844033.097555] Lustre: Skipped 1188 previous similar messages
[13844197.474342] LustreError: 160898:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff91d4f456e050 x1714909917371712/t0(0) o3->274e2965-2f8d-f91e-8ba9-3f4bdaf24c8b@10.210.12.119@tcp1:710/0 lens 488/440 e 0 to 0 dl 1645070210 ref 1 fl Interpret:/0/0 rc 0/0
[13844197.498768] LustreError: 160898:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 59 previous similar messages
[13844197.508588] Lustre: oak-OST012f: Bulk IO read error with 274e2965-2f8d-f91e-8ba9-3f4bdaf24c8b (at 10.210.12.119@tcp1), client will retry: rc -110
[13844197.521853] Lustre: Skipped 7 previous similar messages
[13844289.233902] Lustre: oak-OST0119: Client a166a60d-30d5-1e72-b799-745de1d3b307 (at 10.210.12.125@tcp1) reconnecting
[13844289.244396] Lustre: Skipped 156 previous similar messages
[13844412.292515] Lustre: oak-OST0125: Bulk IO write error with 8cf35edb-55d3-f387-ccc5-dac8256d202e (at 10.210.12.133@tcp1), client will retry: rc = -110
[13844412.306793] Lustre: Skipped 51 previous similar messages
[13844460.498233] LustreError: 21620:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(24576)  req@ffff91376efb4850 x1715023250669696/t0(0) o3->6d01c866-9ec9-8f84-4f86-ee1db83afc97@10.210.12.61@tcp1:167/0 lens 488/440 e 0 to 0 dl 1645070422 ref 1 fl Interpret:/0/0 rc 0/0
[13844460.498426] LustreError: 243495:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff913a4dae5050 x1715023250723968/t0(0) o4->6d01c866-9ec9-8f84-4f86-ee1db83afc97@10.210.12.61@tcp1:168/0 lens 488/448 e 0 to 0 dl 1645070423 ref 1 fl Interpret:/0/0 rc 0/0
[13844460.548955] LustreError: 21620:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 3 previous similar messages
[13844484.459808] LustreError: 21612:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff916ad9e27850 x1715023250738752/t0(0) o4->6d01c866-9ec9-8f84-4f86-ee1db83afc97@10.210.12.61@tcp1:170/0 lens 488/448 e 0 to 0 dl 1645070425 ref 1 fl Interpret:/0/0 rc 0/0
[13844484.485525] LustreError: 21612:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 4 previous similar messages
[13844540.954738] LustreError: 243446:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff91e8f6787850 x1715091129664000/t0(0) o4->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:299/0 lens 488/448 e 0 to 0 dl 1645070554 ref 1 fl Interpret:/2/0 rc 0/0
[13844620.495498] LustreError: 137-5: oak-OST0138_UUID: not available for connect from 10.210.12.64@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13844632.835933] Lustre: oak-OST0137: Connection restored to 250775a1-e4a7-9e5e-ff34-34a009ab359c (at 10.50.8.18@o2ib2)
[13844632.846668] Lustre: Skipped 1425 previous similar messages
[13844996.269689] Lustre: oak-OST0135: Client 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1) reconnecting
[13844996.280100] Lustre: Skipped 155 previous similar messages
[13844996.303230] LustreError: 21610:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9191a7b10050 x1714949579722240/t0(0) o4->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:2/0 lens 488/448 e 0 to 0 dl 1645071012 ref 1 fl Interpret:/0/0 rc 0/0
[13844996.328552] LustreError: 21610:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 49 previous similar messages
[13845013.168873] LustreError: 26437:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff915c1abd4c00
[13845013.179900] LustreError: 26440:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff915c1abd4c00
[13845013.190947] LustreError: 26437:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff915c1abd4c00
[13845013.201966] LustreError: 26441:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff915c1abd4c00
[13845013.213317] Lustre: oak-OST0131: Bulk IO write error with 5865c071-3198-0848-a51e-ecc3ec62e180 (at 10.210.12.127@tcp1), client will retry: rc = -110
[13845013.226866] Lustre: Skipped 61 previous similar messages
[13845059.373170] LustreError: 229135:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 1048576(4194304)  req@ffff915a411a4850 x1714914802240960/t0(0) o3->5865c071-3198-0848-a51e-ecc3ec62e180@10.210.12.127@tcp1:5/0 lens 488/440 e 0 to 0 dl 1645071015 ref 1 fl Interpret:/0/0 rc 0/0
[13845059.373386] Lustre: oak-OST0111: Bulk IO read error with 5865c071-3198-0848-a51e-ecc3ec62e180 (at 10.210.12.127@tcp1), client will retry: rc -110
[13845059.373387] Lustre: Skipped 10 previous similar messages
[13845059.418309] LustreError: 229135:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 12 previous similar messages
[13845080.440976] LustreError: 137-5: oak-OST0116_UUID: not available for connect from 10.210.12.127@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13845080.458872] LustreError: Skipped 1 previous similar message
[13845083.340048] LustreError: 243442:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(2620041)  req@ffff916a1ae1f850 x1714914802494848/t0(0) o4->5865c071-3198-0848-a51e-ecc3ec62e180@10.210.12.127@tcp1:21/0 lens 504/448 e 0 to 0 dl 1645071031 ref 1 fl Interpret:/0/0 rc 0/0
[13845083.365328] LustreError: 243442:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[13845107.297823] LustreError: 21596:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(3699567)  req@ffff917045173050 x1714914802514304/t0(0) o4->5865c071-3198-0848-a51e-ecc3ec62e180@10.210.12.127@tcp1:41/0 lens 488/448 e 0 to 0 dl 1645071051 ref 1 fl Interpret:/0/0 rc 0/0
[13845107.323028] LustreError: 21596:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[13845231.465782] Lustre: oak-OST012f: Connection restored to 49ffde34-fdda-ae5f-2b02-f1e084a0de30 (at 10.50.0.63@o2ib2)
[13845231.476402] Lustre: Skipped 1173 previous similar messages
[13845596.120165] Lustre: oak-OST0131: Client c9a6c6f3-0875-9cfd-dcbf-edc86f3433a7 (at 10.210.12.131@tcp1) reconnecting
[13845596.130675] Lustre: Skipped 99 previous similar messages
[13845831.375204] Lustre: oak-OST0141: Connection restored to  (at 10.50.16.2@o2ib2)
[13845831.382689] Lustre: Skipped 873 previous similar messages
[13845950.521294] LustreError: 162689:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff918c1c3d8850 x1714909456505984/t0(0) o4->f4048b3b-d9e4-178d-cd58-3b66faaca4ee@10.210.12.117@tcp1:198/0 lens 488/448 e 0 to 0 dl 1645071963 ref 1 fl Interpret:/0/0 rc 0/0
[13845950.545939] LustreError: 162689:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 46 previous similar messages
[13845950.556417] Lustre: oak-OST013f: Bulk IO write error with f4048b3b-d9e4-178d-cd58-3b66faaca4ee (at 10.210.12.117@tcp1), client will retry: rc = -110
[13845950.569987] Lustre: Skipped 38 previous similar messages
[13845993.640493] LustreError: 162699:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff91366d7b0850 x1715023296276160/t0(0) o3->6d01c866-9ec9-8f84-4f86-ee1db83afc97@10.210.12.61@tcp1:197/0 lens 488/440 e 0 to 0 dl 1645071962 ref 1 fl Interpret:/0/0 rc 0/0
[13845993.640517] Lustre: oak-OST0115: Bulk IO read error with 6d01c866-9ec9-8f84-4f86-ee1db83afc97 (at 10.210.12.61@tcp1), client will retry: rc -110
[13845993.640519] Lustre: Skipped 14 previous similar messages
[13845993.684074] LustreError: 162699:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 5 previous similar messages
[13846017.600260] LustreError: 21593:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff9160a5a11850 x1715023296280512/t0(0) o4->6d01c866-9ec9-8f84-4f86-ee1db83afc97@10.210.12.61@tcp1:197/0 lens 488/448 e 0 to 0 dl 1645071962 ref 1 fl Interpret:/0/0 rc 0/0
[13846017.625978] LustreError: 21593:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[13846215.414887] Lustre: oak-OST0121: Client a166a60d-30d5-1e72-b799-745de1d3b307 (at 10.210.12.125@tcp1) reconnecting
[13846215.425523] Lustre: Skipped 77 previous similar messages
[13846232.232052] LustreError: 253933:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91bbad858050 x1714906516136128/t0(0) o4->a166a60d-30d5-1e72-b799-745de1d3b307@10.210.12.125@tcp1:435/0 lens 488/448 e 0 to 0 dl 1645072200 ref 1 fl Interpret:/0/0 rc 0/0
[13846232.257941] LustreError: 253933:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[13846256.194201] LustreError: 243447:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 1048576(2916352)  req@ffff91e5c832c850 x1714906516159040/t0(0) o4->a166a60d-30d5-1e72-b799-745de1d3b307@10.210.12.125@tcp1:438/0 lens 488/448 e 0 to 0 dl 1645072203 ref 1 fl Interpret:/0/0 rc 0/0
[13846256.221203] LustreError: 243447:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[13846266.713231] LustreError: 137-5: oak-OST0140_UUID: not available for connect from 10.210.12.125@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13846431.137147] Lustre: oak-OST0129: Connection restored to 25a784fe-8b34-217f-75b0-8238570b4b09 (at 10.50.5.63@o2ib2)
[13846431.147742] Lustre: Skipped 927 previous similar messages
[13846578.390166] LustreError: 243440:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9143d7617050 x1714907098131520/t0(0) o4->f2914e3c-b512-2e60-27c3-dea532333a2b@10.210.12.135@tcp1:73/0 lens 488/448 e 0 to 0 dl 1645072593 ref 1 fl Interpret:/0/0 rc 0/0
[13846578.414585] LustreError: 243440:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 30 previous similar messages
[13846578.424662] Lustre: oak-OST0133: Bulk IO write error with f2914e3c-b512-2e60-27c3-dea532333a2b (at 10.210.12.135@tcp1), client will retry: rc = -110
[13846578.438202] Lustre: Skipped 38 previous similar messages
[13846639.547518] LustreError: 162684:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(20480)  req@ffff917389cd9050 x1714949677355712/t0(0) o3->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:73/0 lens 488/440 e 0 to 0 dl 1645072593 ref 1 fl Interpret:/0/0 rc 0/0
[13846639.549589] LustreError: 162672:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(3457024)  req@ffff9140904bd850 x1715532788089664/t0(0) o4->5a5caa86-68df-f806-7e6f-05440e95a4ce@10.210.12.43@tcp1:79/0 lens 504/448 e 0 to 0 dl 1645072599 ref 1 fl Interpret:/0/0 rc 0/0
[13846639.565656] Lustre: oak-OST0145: Bulk IO read error with 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1), client will retry: rc -110
[13846639.565657] Lustre: Skipped 18 previous similar messages
[13846639.618249] LustreError: 162684:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 17 previous similar messages
[13846958.781561] Lustre: oak-OST0121: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[13846958.792072] Lustre: Skipped 214 previous similar messages
[13847029.756407] Lustre: oak-OST0143: Connection restored to 25a784fe-8b34-217f-75b0-8238570b4b09 (at 10.50.5.63@o2ib2)
[13847029.767098] Lustre: Skipped 1632 previous similar messages
[13847382.181389] LustreError: 162717:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff914933525850 x1715091375929984/t0(0) o4->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:125/0 lens 488/448 e 0 to 0 dl 1645073400 ref 1 fl Interpret:/0/0 rc 0/0
[13847382.205883] LustreError: 162717:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 26 previous similar messages
[13847382.215893] Lustre: oak-OST012d: Bulk IO write error with 5d8bb8d9-e5c7-334f-ec8d-f10452c309b5 (at 10.210.12.66@tcp1), client will retry: rc = -110
[13847382.229332] Lustre: Skipped 27 previous similar messages
[13847399.038200] LustreError: 244100:0:(tgt_grant.c:758:tgt_grant_check()) oak-OST0119: cli bb55a01e-02a3-cbba-5b76-e478fd40967c claims 4218880 GRANT, real grant 2744320
[13847399.053126] LustreError: 244100:0:(tgt_grant.c:758:tgt_grant_check()) Skipped 14 previous similar messages
[13847429.266981] LustreError: 21585:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff9174d0795050 x1714949756807168/t0(0) o4->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:124/0 lens 488/448 e 0 to 0 dl 1645073399 ref 1 fl Interpret:/0/0 rc 0/0
[13847429.267167] LustreError: 21583:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff91407d1a8850 x1714949756810432/t0(0) o3->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:124/0 lens 488/440 e 0 to 0 dl 1645073399 ref 1 fl Interpret:/0/0 rc 0/0
[13847429.267169] LustreError: 21583:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 2 previous similar messages
[13847429.267306] Lustre: oak-OST0123: Bulk IO read error with 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1), client will retry: rc -110
[13847429.267307] Lustre: Skipped 6 previous similar messages
[13847470.501968] LustreError: 21613:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff918c1c3dc850 x1714907693310400/t0(0) o4->a2f24d85-30b4-0808-a59e-1ce0f2193fa6@10.210.12.121@tcp1:216/0 lens 488/448 e 0 to 0 dl 1645073491 ref 1 fl Interpret:/2/0 rc 0/0
[13847477.184980] LustreError: 162687:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4172018)  req@ffff9190fdcec050 x1715061927553664/t0(0) o4->bb55a01e-02a3-cbba-5b76-e478fd40967c@10.210.12.54@tcp1:155/0 lens 600/472 e 0 to 0 dl 1645073430 ref 1 fl Interpret:/0/0 rc 0/0
[13847477.210822] LustreError: 162687:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 4 previous similar messages
[13847494.855634] LustreError: 137-5: oak-OST0134_UUID: not available for connect from 10.210.12.111@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13847497.865536] LustreError: 137-5: oak-OST011c_UUID: not available for connect from 10.210.12.59@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13847525.102982] LustreError: 21620:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff918a65e92850 x1714906957812544/t0(0) o4->8cf35edb-55d3-f387-ccc5-dac8256d202e@10.210.12.133@tcp1:211/0 lens 488/448 e 0 to 0 dl 1645073486 ref 1 fl Interpret:/0/0 rc 0/0
[13847525.128807] LustreError: 21620:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[13847548.089701] LustreError: 137-5: oak-OST014e_UUID: not available for connect from 10.210.12.73@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13847557.723859] Lustre: oak-OST0121: Client c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1) reconnecting
[13847557.734276] Lustre: Skipped 98 previous similar messages
[13847629.247932] Lustre: oak-OST0141: Connection restored to f1dd735d-6c66-d8b4-50d4-60d2b73caa84 (at 10.51.16.3@o2ib3)
[13847629.258584] Lustre: Skipped 1030 previous similar messages
[13847668.059476] LustreError: 162671:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk READ failed: rc -107  req@ffff9158dc064050 x1714906958251840/t0(0) o3->8cf35edb-55d3-f387-ccc5-dac8256d202e@10.210.12.133@tcp1:415/0 lens 920/440 e 0 to 0 dl 1645073690 ref 1 fl Interpret:/0/0 rc 0/0
[13847692.806599] LustreError: 199271:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff918c2599e050 x1715083987136896/t0(0) o4->65e9b8a8-b461-24e0-da12-5b6939f0b07c@10.210.12.56@tcp1:381/0 lens 488/448 e 0 to 0 dl 1645073656 ref 1 fl Interpret:/0/0 rc 0/0
[13847692.832558] LustreError: 199271:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 3 previous similar messages
[13847716.580877] LustreError: 137-5: oak-OST0126_UUID: not available for connect from 10.210.12.54@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13847720.674833] LustreError: 137-5: oak-OST0146_UUID: not available for connect from 10.210.12.54@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13847720.692499] LustreError: Skipped 3 previous similar messages
[13847764.668855] LustreError: 21597:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff916daa3b1850 x1714999379981568/t0(0) o4->6f45695d-f173-ee1d-e6cb-d38dad7e0879@10.210.12.74@tcp1:439/0 lens 488/448 e 0 to 0 dl 1645073714 ref 1 fl Interpret:/0/0 rc 0/0
[13847764.694606] LustreError: 21597:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 3 previous similar messages
[13848046.570110] LustreError: 137-5: oak-OST014a_UUID: not available for connect from 10.210.12.43@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13848046.587702] LustreError: Skipped 1 previous similar message
[13848106.246648] LustreError: 162713:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff914be568f050 x1715062001244608/t0(0) o3->bb55a01e-02a3-cbba-5b76-e478fd40967c@10.210.12.54@tcp1:96/0 lens 488/440 e 0 to 0 dl 1645074126 ref 1 fl Interpret:/0/0 rc 0/0
[13848106.270905] LustreError: 162713:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 84 previous similar messages
[13848106.280725] Lustre: oak-OST0147: Bulk IO read error with bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1), client will retry: rc -110
[13848106.293904] Lustre: Skipped 33 previous similar messages
[13848160.943691] Lustre: oak-OST0115: Client 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1) reconnecting
[13848160.954112] Lustre: Skipped 379 previous similar messages
[13848161.525308] Lustre: oak-OST014d: Bulk IO write error with 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1), client will retry: rc = -110
[13848161.538824] Lustre: Skipped 94 previous similar messages
[13848227.816567] Lustre: oak-OST0125: Connection restored to 7747ea82-5827-20c1-bbc2-85f13c467aad (at 10.50.8.24@o2ib2)
[13848227.827178] Lustre: Skipped 1060 previous similar messages
[13848531.368382] LustreError: 162669:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(72113)  req@ffff91376eff5850 x1722622065844544/t0(0) o4->6b1e44d2-87c6-d03d-d92f-f9319d531e93@10.50.7.33@o2ib2:442/0 lens 488/448 e 0 to 0 dl 1645074472 ref 1 fl Interpret:/0/0 rc 0/0
[13848531.393470] LustreError: 162669:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 10 previous similar messages
[13848651.146993] LustreError: 243555:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff9149690d8050 x1715758776227200/t0(0) o3->4b39aac6-2dc7-e48f-9434-e3bfde492453@10.210.12.29@tcp1:585/0 lens 488/440 e 0 to 0 dl 1645074615 ref 1 fl Interpret:/0/0 rc 0/0
[13848651.147125] LustreError: 243536:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff9191eed4c050 x1715023533501440/t0(0) o4->6d01c866-9ec9-8f84-4f86-ee1db83afc97@10.210.12.61@tcp1:585/0 lens 488/448 e 0 to 0 dl 1645074615 ref 1 fl Interpret:/0/0 rc 0/0
[13848651.198349] LustreError: 243555:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 31 previous similar messages
[13848673.810952] LustreError: 137-5: oak-OST0120_UUID: not available for connect from 10.210.12.54@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13848673.828673] LustreError: Skipped 1 previous similar message
[13848676.083725] LustreError: 137-5: oak-OST012c_UUID: not available for connect from 10.210.12.75@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13848681.564249] LustreError: 137-5: oak-OST014c_UUID: not available for connect from 10.210.12.65@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13848681.581977] LustreError: Skipped 1 previous similar message
[13848699.074936] LustreError: 243535:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff9164e564d050 x1715091464129152/t0(0) o4->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:626/0 lens 488/448 e 0 to 0 dl 1645074656 ref 1 fl Interpret:/0/0 rc 0/0
[13848699.104291] LustreError: 243535:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 4 previous similar messages
[13848717.345650] LustreError: 137-5: oak-OST0120_UUID: not available for connect from 10.210.12.66@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13848746.990301] LustreError: 243526:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff9191f0f01050 x1715044030351680/t0(0) o4->80374d70-7d99-f68a-b58b-86c0b7514d6a@10.210.12.65@tcp1:681/0 lens 488/448 e 0 to 0 dl 1645074711 ref 1 fl Interpret:/0/0 rc 0/0
[13848747.016267] LustreError: 243526:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[13848761.018918] Lustre: oak-OST012d: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[13848761.029344] Lustre: Skipped 57 previous similar messages
[13848765.222980] LustreError: 243445:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff916be2339050 x1715758804139136/t0(0) o3->4b39aac6-2dc7-e48f-9434-e3bfde492453@10.210.12.29@tcp1:3/0 lens 488/440 e 0 to 0 dl 1645074788 ref 1 fl Interpret:/0/0 rc 0/0
[13848765.247230] LustreError: 243445:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 34 previous similar messages
[13848765.257101] Lustre: oak-OST0137: Bulk IO read error with 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1), client will retry: rc -110
[13848765.270327] Lustre: Skipped 17 previous similar messages
[13848813.407106] Lustre: oak-OST0131: Bulk IO write error with c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1), client will retry: rc = -110
[13848813.420564] Lustre: Skipped 50 previous similar messages
[13848827.372752] Lustre: oak-OST0143: Connection restored to ab61d733-6278-aecc-ffdb-baa33579a940 (at 10.51.4.33@o2ib3)
[13848827.383368] Lustre: Skipped 1561 previous similar messages
[13848836.952603] LustreError: 160933:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk READ failed: rc -107  req@ffff91ec11183050 x1714949920800128/t0(0) o3->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:77/0 lens 488/440 e 0 to 0 dl 1645074862 ref 1 fl Interpret:/0/0 rc 0/0
[13848836.977125] LustreError: 160933:0:(ldlm_lib.c:3305:target_bulk_io()) Skipped 3 previous similar messages
[13848897.674712] Lustre: oak-OST012b: haven't heard from client fd2c4c0f-a4cd-0a60-cb33-5e9a47204c46 (at 10.210.12.19@tcp1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9136f6b89000, cur 1645074825 expire 1645074675 last 1645074598
[13848897.696783] Lustre: Skipped 1 previous similar message
[13848961.606279] LustreError: 160908:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91ef90ec9050 x1714969036956736/t0(0) o4->5afa23c2-1fd5-9030-38ce-b03a359082fe@10.210.12.75@tcp1:151/0 lens 488/448 e 0 to 0 dl 1645074936 ref 1 fl Interpret:/0/0 rc 0/0
[13848999.050616] LustreError: 137-5: oak-OST012c_UUID: not available for connect from 10.210.12.75@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13848999.068253] LustreError: Skipped 1 previous similar message
[13849009.504309] LustreError: 162715:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff918ec3fce050 x1715758809161600/t0(0) o3->4b39aac6-2dc7-e48f-9434-e3bfde492453@10.210.12.29@tcp1:179/0 lens 488/440 e 0 to 0 dl 1645074964 ref 1 fl Interpret:/0/0 rc 0/0
[13849009.504337] LustreError: 21617:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91817e206850 x1716550366037696/t0(0) o4->6bab8d10-30ff-c1d2-bbf5-1fd0884ac647@10.210.12.57@tcp1:179/0 lens 488/448 e 0 to 0 dl 1645074964 ref 1 fl Interpret:/0/0 rc 0/0
[13849009.554961] LustreError: 162715:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 9 previous similar messages
[13849225.147926] LustreError: 21622:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff91ad04276850 x1714978721311936/t0(0) o3->d4faaef8-44e2-b05f-6116-d9c30b2e4e11@10.210.12.60@tcp1:414/0 lens 504/440 e 0 to 0 dl 1645075199 ref 1 fl Interpret:/0/0 rc 0/0
[13849225.148281] LustreError: 199272:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91d4f456c050 x1715010584481152/t0(0) o4->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:415/0 lens 488/448 e 0 to 0 dl 1645075200 ref 1 fl Interpret:/0/0 rc 0/0
[13849225.148282] LustreError: 199272:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 3 previous similar messages
[13849225.210318] LustreError: 21622:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 2 previous similar messages
[13849258.447482] LustreError: 137-5: oak-OST014c_UUID: not available for connect from 10.210.12.65@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13849258.465168] LustreError: Skipped 4 previous similar messages
[13849359.728961] Lustre: oak-OST0145: Client f2914e3c-b512-2e60-27c3-dea532333a2b (at 10.210.12.135@tcp1) reconnecting
[13849359.739461] Lustre: Skipped 692 previous similar messages
[13849416.773390] LustreError: 21607:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91519071f050 x1715084076841600/t0(0) o4->65e9b8a8-b461-24e0-da12-5b6939f0b07c@10.210.12.56@tcp1:593/0 lens 488/448 e 0 to 0 dl 1645075378 ref 1 fl Interpret:/0/0 rc 0/0
[13849416.785923] Lustre: oak-OST0113: Bulk IO read error with c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1), client will retry: rc -110
[13849416.785924] Lustre: Skipped 24 previous similar messages
[13849416.817998] LustreError: 21607:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[13849416.828182] Lustre: oak-OST011d: Bulk IO write error with 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1), client will retry: rc = -110
[13849416.841871] Lustre: Skipped 37 previous similar messages
[13849426.291856] LustreError: 137-5: oak-OST0146_UUID: not available for connect from 10.210.12.7@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13849426.553334] Lustre: oak-OST0141: Connection restored to f4255b36-c9bf-9a6f-b06a-73d0d2f391ef (at 10.50.8.47@o2ib2)
[13849426.564015] Lustre: Skipped 1430 previous similar messages
[13849495.202357] Lustre: oak-OST0115: haven't heard from client 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91ee2d5d6000, cur 1645075424 expire 1645075274 last 1645075197
[13849497.854311] LustreError: 244099:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91f18e3e2850 x1714978741974976/t0(0) o4->d4faaef8-44e2-b05f-6116-d9c30b2e4e11@10.210.12.60@tcp1:735/0 lens 488/448 e 0 to 0 dl 1645075520 ref 1 fl Interpret:/0/0 rc 0/0
[13849497.879402] LustreError: 244099:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 39 previous similar messages
[13849560.510052] LustreError: 243440:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 1572864(3670016)  req@ffff915287330850 x1714978741955136/t0(0) o3->d4faaef8-44e2-b05f-6116-d9c30b2e4e11@10.210.12.60@tcp1:735/0 lens 952/440 e 0 to 0 dl 1645075520 ref 1 fl Interpret:/0/0 rc 0/0
[13849560.520470] LustreError: 160934:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91d835196050 x1714978741974208/t0(0) o4->d4faaef8-44e2-b05f-6116-d9c30b2e4e11@10.210.12.60@tcp1:735/0 lens 488/448 e 0 to 0 dl 1645075520 ref 1 fl Interpret:/0/0 rc 0/0
[13849560.561411] LustreError: 243440:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 12 previous similar messages
[13849628.822479] LustreError: 137-5: oak-OST012c_UUID: not available for connect from 10.210.12.75@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13849628.840136] LustreError: Skipped 10 previous similar messages
[13849848.007481] LustreError: 160893:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91978ba8a050 x1715427196329984/t0(0) o4->0d1c8726-ce4c-fcc8-6c1b-7179039cabd2@10.210.12.8@tcp1:267/0 lens 488/448 e 0 to 0 dl 1645075807 ref 1 fl Interpret:/0/0 rc 0/0
[13849848.033194] LustreError: 160893:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[13849951.255012] LustreError: 21605:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk READ failed: rc -107  req@ffff91458a43a850 x1715427196957888/t0(0) o3->0d1c8726-ce4c-fcc8-6c1b-7179039cabd2@10.210.12.8@tcp1:439/0 lens 488/440 e 0 to 0 dl 1645075979 ref 1 fl Interpret:/0/0 rc 0/0
[13849958.472763] Lustre: oak-OST0115: Client 5bc2a4b7-b189-1f48-2b19-13f38311d9d9 (at 10.210.12.107@tcp1) reconnecting
[13849958.483255] Lustre: Skipped 313 previous similar messages
[13850026.508250] Lustre: oak-OST0111: Connection restored to f1db0cb0-5cee-ccf9-6484-5189f751ad99 (at 10.51.0.63@o2ib3)
[13850026.518833] Lustre: Skipped 2152 previous similar messages
[13850035.414218] LustreError: 137-5: oak-OST012a_UUID: not available for connect from 10.210.12.131@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13850035.431903] LustreError: Skipped 1 previous similar message
[13850062.306590] Lustre: oak-OST0119: Bulk IO write error with 5afa23c2-1fd5-9030-38ce-b03a359082fe (at 10.210.12.75@tcp1), client will retry: rc = -110
[13850062.320179] Lustre: Skipped 20 previous similar messages
[13850111.514770] Lustre: oak-OST0119: Bulk IO read error with 5d8bb8d9-e5c7-334f-ec8d-f10452c309b5 (at 10.210.12.66@tcp1), client will retry: rc -110
[13850111.527970] Lustre: Skipped 11 previous similar messages
[13850234.573677] LustreError: 160954:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91dd2240f050 x1715091574181120/t0(0) o4->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:721/0 lens 488/448 e 0 to 0 dl 1645076261 ref 1 fl Interpret:/0/0 rc 0/0
[13850234.598189] LustreError: 160954:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 18 previous similar messages
[13850255.249446] LustreError: 243445:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff91919d768050 x1714969061957184/t0(0) o3->5afa23c2-1fd5-9030-38ce-b03a359082fe@10.210.12.75@tcp1:678/0 lens 488/440 e 0 to 0 dl 1645076218 ref 1 fl Interpret:/0/0 rc 0/0
[13850255.274993] LustreError: 243445:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 8 previous similar messages
[13850626.620583] Lustre: oak-OST0131: Connection restored to 99627209-92e6-01ab-afb3-3c05a7969f40 (at 10.50.5.56@o2ib2)
[13850626.631178] Lustre: Skipped 1745 previous similar messages
[13850740.046229] Lustre: oak-OST012f: Client c9a6c6f3-0875-9cfd-dcbf-edc86f3433a7 (at 10.210.12.131@tcp1) reconnecting
[13850740.056725] Lustre: Skipped 225 previous similar messages
[13851020.782873] LustreError: 162690:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff917c883ae050 x1715060546971584/t0(0) o4->dc851ef6-3898-c01c-adc6-dc548a499d03@10.210.12.53@tcp1:686/0 lens 488/448 e 0 to 0 dl 1645076981 ref 1 fl Interpret:/0/0 rc 0/0
[13851020.783217] LustreError: 162685:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(65536)  req@ffff9149690da850 x1715060547004928/t0(0) o3->dc851ef6-3898-c01c-adc6-dc548a499d03@10.210.12.53@tcp1:688/0 lens 488/440 e 0 to 0 dl 1645076983 ref 1 fl Interpret:/0/0 rc 0/0
[13851020.783219] LustreError: 162685:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 4 previous similar messages
[13851020.783230] Lustre: oak-OST011b: Bulk IO read error with dc851ef6-3898-c01c-adc6-dc548a499d03 (at 10.210.12.53@tcp1), client will retry: rc -110
[13851020.783231] Lustre: Skipped 9 previous similar messages
[13851020.861907] LustreError: 162690:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 6 previous similar messages
[13851020.872164] Lustre: oak-OST012f: Bulk IO write error with dc851ef6-3898-c01c-adc6-dc548a499d03 (at 10.210.12.53@tcp1), client will retry: rc = -110
[13851020.885605] Lustre: Skipped 15 previous similar messages
[13851035.381236] LustreError: 137-5: oak-OST0114_UUID: not available for connect from 10.210.12.7@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13851035.398836] LustreError: Skipped 7 previous similar messages
[13851089.020814] LustreError: 243536:0:(tgt_handler.c:651:process_req_last_xid()) @@@ Unexpected xid 617d7157e8700 vs. last_xid 617d7157e91ff
                    req@ffff91376d782050 x1715062406285056/t0(0) o3->bb55a01e-02a3-cbba-5b76-e478fd40967c@10.210.12.54@tcp1:69/0 lens 488/0 e 0 to 0 dl 1645077119 ref 1 fl Interpret:/2/ffffffff rc 0/-1
[13851133.340332] LustreError: 160948:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91cb245d5850 x1715134857806016/t0(0) o4->cc381d20-202e-f264-43c3-938610d60653@10.210.12.58@tcp1:110/0 lens 488/448 e 0 to 0 dl 1645077160 ref 1 fl Interpret:/0/0 rc 0/0
[13851133.364925] LustreError: 160948:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 8 previous similar messages
[13851188.450411] LustreError: 162700:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91376e799850 x1714914974760256/t0(0) o4->5865c071-3198-0848-a51e-ecc3ec62e180@10.210.12.127@tcp1:107/0 lens 488/448 e 0 to 0 dl 1645077157 ref 1 fl Interpret:/0/0 rc 0/0
[13851188.476445] LustreError: 162700:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[13851225.239875] Lustre: oak-OST0111: Connection restored to 99627209-92e6-01ab-afb3-3c05a7969f40 (at 10.50.5.56@o2ib2)
[13851225.250499] Lustre: Skipped 822 previous similar messages
[13851352.726186] Lustre: oak-OST0119: Client e44e675f-11ba-536d-fbc7-173bafd860e2 (at 10.210.12.46@tcp1) reconnecting
[13851352.736599] Lustre: Skipped 139 previous similar messages
[13851380.086836] LustreError: 162686:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff9176668bc050 x1715062435996544/t0(0) o4->bb55a01e-02a3-cbba-5b76-e478fd40967c@10.210.12.54@tcp1:310/0 lens 488/448 e 0 to 0 dl 1645077360 ref 1 fl Interpret:/0/0 rc 0/0
[13851380.112672] LustreError: 162686:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[13851619.619437] LustreError: 243456:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff91a786b57050 x1714950123395008/t0(0) o3->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:530/0 lens 488/440 e 0 to 0 dl 1645077580 ref 1 fl Interpret:/0/0 rc 0/0
[13851619.619583] Lustre: oak-OST0121: Bulk IO read error with 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1), client will retry: rc -110
[13851619.619584] Lustre: Skipped 18 previous similar messages
[13851619.663938] LustreError: 243456:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 14 previous similar messages
[13851738.199274] LustreError: 160948:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91f1def51050 x1714950133868544/t0(0) o4->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:718/0 lens 488/448 e 0 to 0 dl 1645077768 ref 1 fl Interpret:/0/0 rc 0/0
[13851738.223695] LustreError: 160948:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 16 previous similar messages
[13851738.233713] Lustre: oak-OST012b: Bulk IO write error with 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1), client will retry: rc = -110
[13851738.247156] Lustre: Skipped 18 previous similar messages
[13851826.797984] Lustre: oak-OST0143: Connection restored to bc7396f6-fbb0-5e6f-3bd8-bf31cb05de7c (at 10.50.5.55@o2ib2)
[13851826.808573] Lustre: Skipped 900 previous similar messages
[13851859.147422] LustreError: 21603:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff9178bc9f5050 x1715060572051968/t0(0) o4->dc851ef6-3898-c01c-adc6-dc548a499d03@10.210.12.53@tcp1:20/0 lens 488/448 e 0 to 0 dl 1645077825 ref 1 fl Interpret:/0/0 rc 0/0
[13851859.173894] LustreError: 21603:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[13851875.661980] LustreError: 137-5: oak-OST0136_UUID: not available for connect from 10.210.12.135@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13851875.679692] LustreError: Skipped 5 previous similar messages
[13851964.364738] Lustre: oak-OST012d: Client 5d8bb8d9-e5c7-334f-ec8d-f10452c309b5 (at 10.210.12.66@tcp1) reconnecting
[13851964.375157] Lustre: Skipped 252 previous similar messages
[13852290.299562] LustreError: 160902:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff91dd2240b050 x1714979097183232/t0(0) o3->d4faaef8-44e2-b05f-6116-d9c30b2e4e11@10.210.12.60@tcp1:453/0 lens 488/440 e 0 to 0 dl 1645078258 ref 1 fl Interpret:/0/0 rc 0/0
[13852290.324410] LustreError: 160902:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 8 previous similar messages
[13852290.334136] Lustre: oak-OST0137: Bulk IO read error with d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1), client will retry: rc -110
[13852290.347309] Lustre: Skipped 9 previous similar messages
[13852367.129774] LustreError: 253934:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91d835190050 x1715062531376832/t0(0) o4->bb55a01e-02a3-cbba-5b76-e478fd40967c@10.210.12.54@tcp1:589/0 lens 488/448 e 0 to 0 dl 1645078394 ref 1 fl Interpret:/0/0 rc 0/0
[13852367.154414] LustreError: 253934:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 6 previous similar messages
[13852367.164325] Lustre: oak-OST011d: Bulk IO write error with bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1), client will retry: rc = -110
[13852367.177757] Lustre: Skipped 9 previous similar messages
[13852426.162016] Lustre: oak-OST0129: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[13852426.172680] Lustre: Skipped 1398 previous similar messages
[13852548.518909] LustreError: 137-5: oak-OST0156_UUID: not available for connect from 10.210.12.19@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13852548.536752] LustreError: Skipped 9 previous similar messages
[13852572.159467] Lustre: oak-OST0117: Client e5a735ca-191d-07c9-fcef-626c0b3a28a8 (at 10.210.12.113@tcp1) reconnecting
[13852572.169971] Lustre: Skipped 341 previous similar messages
[13852576.725478] LustreError: 162673:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91376f78a850 x1714909622504640/t0(0) o4->f4048b3b-d9e4-178d-cd58-3b66faaca4ee@10.210.12.117@tcp1:747/0 lens 488/448 e 0 to 0 dl 1645078552 ref 1 fl Interpret:/0/0 rc 0/0
[13852576.751402] LustreError: 162673:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[13852616.620431] Lustre: 204914:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1645078400/real 1645078400]  req@ffff9196dcfa6300 x1710530580579200/t0(0) o104->oak-OST0111@10.210.12.53@tcp1:15/16 lens 296/224 e 0 to 1 dl 1645078553 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
[13852616.648192] Lustre: 204914:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[13852691.820989] LustreError: 243536:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff91713d10a050 x1714907480600448/t0(0) o4->0d4d52ae-6d60-d275-1404-c9b4d99e0974@10.210.12.109@tcp1:165/0 lens 488/448 e 0 to 0 dl 1645078725 ref 1 fl Interpret:/2/0 rc 0/0
[13852943.969891] Lustre: oak-OST0143: Bulk IO read error with bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1), client will retry: rc -110
[13852943.983075] Lustre: Skipped 31 previous similar messages
[13852983.924918] LustreError: 162681:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff917a0a440050 x1714909987376704/t0(0) o3->c9a6c6f3-0875-9cfd-dcbf-edc86f3433a7@10.210.12.131@tcp1:388/0 lens 488/440 e 0 to 0 dl 1645078948 ref 1 fl Interpret:/0/0 rc 0/0
[13852983.924925] Lustre: oak-OST0121: Bulk IO write error with dc851ef6-3898-c01c-adc6-dc548a499d03 (at 10.210.12.53@tcp1), client will retry: rc = -110
[13852983.924926] Lustre: Skipped 46 previous similar messages
[13852983.969145] LustreError: 162681:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 29 previous similar messages
[13853025.613681] Lustre: oak-OST014b: Connection restored to b9694c0e-9fc2-f14e-33c8-9018d9fc9806 (at 10.210.12.65@tcp1)
[13853025.624391] Lustre: Skipped 1687 previous similar messages
[13853186.431233] Lustre: oak-OST011b: Client 5865c071-3198-0848-a51e-ecc3ec62e180 (at 10.210.12.127@tcp1) reconnecting
[13853186.441762] Lustre: Skipped 774 previous similar messages
[13853582.789438] LustreError: 162677:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 3145728(4194304)  req@ffff916f4c5f0850 x1714915020927808/t0(0) o3->5865c071-3198-0848-a51e-ecc3ec62e180@10.210.12.127@tcp1:243/0 lens 488/440 e 0 to 0 dl 1645079558 ref 1 fl Interpret:/0/0 rc 0/0
[13853582.789701] Lustre: oak-OST0131: Bulk IO read error with 5865c071-3198-0848-a51e-ecc3ec62e180 (at 10.210.12.127@tcp1), client will retry: rc -110
[13853582.789702] Lustre: Skipped 3 previous similar messages
[13853582.834165] LustreError: 162677:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 3 previous similar messages
[13853607.521223] LustreError: 137-5: oak-OST012c_UUID: not available for connect from 10.210.12.127@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13853607.538917] LustreError: Skipped 18 previous similar messages
[13853624.548676] Lustre: oak-OST0113: Connection restored to 3c8be638-4b95-3b95-0b17-8ea4836f9204 (at 10.50.12.7@o2ib2)
[13853624.559279] Lustre: Skipped 1356 previous similar messages
[13853702.420679] LustreError: 162715:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff918e2da6b850 x1715062709126656/t0(0) o4->bb55a01e-02a3-cbba-5b76-e478fd40967c@10.210.12.54@tcp1:419/0 lens 488/448 e 0 to 0 dl 1645079734 ref 1 fl Interpret:/0/0 rc 0/0
[13853702.445124] LustreError: 162715:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 45 previous similar messages
[13853702.455185] Lustre: oak-OST013b: Bulk IO write error with bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1), client will retry: rc = -110
[13853702.468676] Lustre: Skipped 1 previous similar message
[13853702.691325] LustreError: 243540:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff917fc537f850 x1715062709130368/t0(0) o4->bb55a01e-02a3-cbba-5b76-e478fd40967c@10.210.12.54@tcp1:424/0 lens 488/448 e 0 to 0 dl 1645079739 ref 1 fl Interpret:/2/0 rc 0/0
[13853846.279125] LustreError: 21593:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff9144564e8050 x1715060634167936/t0(0) o4->dc851ef6-3898-c01c-adc6-dc548a499d03@10.210.12.53@tcp1:516/0 lens 488/448 e 0 to 0 dl 1645079831 ref 1 fl Interpret:/0/0 rc 0/0
[13853846.304836] LustreError: 21593:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 6 previous similar messages
[13853858.628527] Lustre: oak-OST0115: Client fd2c4c0f-a4cd-0a60-cb33-5e9a47204c46 (at 10.210.12.19@tcp1) reconnecting
[13853858.638962] Lustre: Skipped 67 previous similar messages
[13853977.556033] LustreError: 243453:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff91d66ad65050 x1715060636988608/t0(0) o3->dc851ef6-3898-c01c-adc6-dc548a499d03@10.210.12.53@tcp1:697/0 lens 488/440 e 0 to 0 dl 1645080012 ref 1 fl Interpret:/0/0 rc 0/0
[13853977.580363] LustreError: 243453:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 6 previous similar messages
[13854232.561647] Lustre: oak-OST013d: Connection restored to 4332b79e-8df2-3478-8cf5-1f58b7df862b (at 10.50.5.64@o2ib2)
[13854232.572557] Lustre: Skipped 916 previous similar messages
[13854516.977136] LustreError: 162686:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff914a15a03050 x1714907799674496/t0(0) o3->d5eb89fe-3ce5-a15e-e193-f0030ed4f8e3@10.210.12.123@tcp1:428/0 lens 488/440 e 0 to 0 dl 1645080498 ref 1 fl Interpret:/0/0 rc 0/0
[13854516.985347] Lustre: oak-OST014d: Bulk IO read error with d5eb89fe-3ce5-a15e-e193-f0030ed4f8e3 (at 10.210.12.123@tcp1), client will retry: rc -110
[13854516.985348] Lustre: Skipped 12 previous similar messages
[13854517.021078] LustreError: 162686:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 8 previous similar messages
[13854631.987925] Lustre: oak-OST0111: Client d5eb89fe-3ce5-a15e-e193-f0030ed4f8e3 (at 10.210.12.123@tcp1) reconnecting
[13854631.998478] Lustre: Skipped 119 previous similar messages
[13854640.915958] LustreError: 243496:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9156706fd850 x1714973084662592/t0(0) o4->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:604/0 lens 488/448 e 0 to 0 dl 1645080674 ref 1 fl Interpret:/0/0 rc 0/0
[13854640.940455] LustreError: 243496:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[13854640.950327] Lustre: oak-OST013b: Bulk IO write error with 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1), client will retry: rc = -110
[13854640.964051] Lustre: Skipped 8 previous similar messages
[13854645.650784] LustreError: 228423:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff914a15a01850 x1722576449755392/t0(0) o4->a8a647a5-bb36-3966-6637-e8ee81d6d548@10.210.12.6@tcp1:614/0 lens 488/448 e 0 to 0 dl 1645080684 ref 1 fl Interpret:/2/0 rc 0/0
[13854707.616185] LustreError: 243495:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2093646(4190798)  req@ffff91572697d050 x1714969348128896/t0(0) o4->5afa23c2-1fd5-9030-38ce-b03a359082fe@10.210.12.75@tcp1:605/0 lens 488/448 e 0 to 0 dl 1645080675 ref 1 fl Interpret:/0/0 rc 0/0
[13854718.525372] LustreError: 137-5: oak-OST0110_UUID: not available for connect from 10.210.12.31@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13854718.543040] LustreError: Skipped 1 previous similar message
[13854831.159422] Lustre: oak-OST0117: Connection restored to bc7396f6-fbb0-5e6f-3bd8-bf31cb05de7c (at 10.50.5.55@o2ib2)
[13854831.170054] Lustre: Skipped 966 previous similar messages
[13855432.400468] Lustre: oak-OST0139: Connection restored to 1cae4158-4c34-2f7a-eb0b-da9101bf8a24 (at 10.50.5.54@o2ib2)
[13855432.411174] Lustre: Skipped 697 previous similar messages
[13856032.849401] Lustre: oak-OST014b: Connection restored to 25a784fe-8b34-217f-75b0-8238570b4b09 (at 10.50.5.63@o2ib2)
[13856032.860293] Lustre: Skipped 1370 previous similar messages
[13856176.630070] Lustre: oak-OST0149: Client a2f24d85-30b4-0808-a59e-1ce0f2193fa6 (at 10.210.12.121@tcp1) reconnecting
[13856176.640606] Lustre: Skipped 263 previous similar messages
[13856177.547212] LustreError: 162688:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff918daaf95850 x1714908078969408/t0(0) o4->a2f24d85-30b4-0808-a59e-1ce0f2193fa6@10.210.12.121@tcp1:639/0 lens 488/448 e 0 to 0 dl 1645082219 ref 1 fl Interpret:/0/0 rc 0/0
[13856177.571800] LustreError: 162688:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 16 previous similar messages
[13856177.581939] Lustre: oak-OST0149: Bulk IO write error with a2f24d85-30b4-0808-a59e-1ce0f2193fa6 (at 10.210.12.121@tcp1), client will retry: rc = -110
[13856177.595638] Lustre: Skipped 17 previous similar messages
[13856240.643870] LustreError: 243542:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91919c985050 x1714908078948608/t0(0) o4->a2f24d85-30b4-0808-a59e-1ce0f2193fa6@10.210.12.121@tcp1:634/0 lens 488/448 e 0 to 0 dl 1645082214 ref 1 fl Interpret:/0/0 rc 0/0
[13856240.670028] LustreError: 243542:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[13856256.434788] Lustre: oak-OST012f: Client a2f24d85-30b4-0808-a59e-1ce0f2193fa6 (at 10.210.12.121@tcp1) reconnecting
[13856631.713025] Lustre: oak-OST0123: Connection restored to 3c8be638-4b95-3b95-0b17-8ea4836f9204 (at 10.50.12.7@o2ib2)
[13856631.723606] Lustre: Skipped 995 previous similar messages
[13857227.223402] Lustre: oak-OST0113: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[13857227.233804] Lustre: Skipped 66 previous similar messages
[13857227.638086] LustreError: 199271:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff918f722f2050 x1714973260712704/t0(0) o4->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:179/0 lens 488/448 e 0 to 0 dl 1645083269 ref 1 fl Interpret:/0/0 rc 0/0
[13857227.662608] Lustre: oak-OST0113: Bulk IO write error with 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1), client will retry: rc = -110
[13857227.676056] Lustre: Skipped 2 previous similar messages
[13857230.320014] Lustre: oak-OST0127: Connection restored to 86110018-10f1-582f-228f-1aa8a96fbb31 (at 10.51.6.38@o2ib3)
[13857230.330960] Lustre: Skipped 638 previous similar messages
[13857293.591084] LustreError: 162683:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff9161dff33050 x1715372267393472/t0(0) o4->5a4881be-a0cb-e632-509b-2e15c534b21a@10.210.12.9@tcp1:178/0 lens 488/448 e 0 to 0 dl 1645083268 ref 1 fl Interpret:/0/0 rc 0/0
[13857293.617044] Lustre: oak-OST0149: Bulk IO write error with 5a4881be-a0cb-e632-509b-2e15c534b21a (at 10.210.12.9@tcp1), client will retry: rc = -110
[13857293.630426] Lustre: Skipped 1 previous similar message
[13857305.926365] Lustre: oak-OST0149: Client 5a4881be-a0cb-e632-509b-2e15c534b21a (at 10.210.12.9@tcp1) reconnecting
[13857305.936682] Lustre: Skipped 1 previous similar message
[13857325.627953] LustreError: 162701:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9165bd836050 x1715063406489664/t0(0) o4->bb55a01e-02a3-cbba-5b76-e478fd40967c@10.210.12.54@tcp1:276/0 lens 504/448 e 0 to 0 dl 1645083366 ref 1 fl Interpret:/0/0 rc 0/0
[13857325.652388] LustreError: 162701:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[13857325.662181] Lustre: oak-OST011d: Bulk IO write error with bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1), client will retry: rc = -110
[13857389.394011] LustreError: 162690:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff916ab206b050 x1714915114495872/t0(0) o4->5865c071-3198-0848-a51e-ecc3ec62e180@10.210.12.127@tcp1:276/0 lens 488/448 e 0 to 0 dl 1645083366 ref 1 fl Interpret:/0/0 rc 0/0
[13857389.394166] LustreError: 229137:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff9182bfa4e050 x1715011380408384/t0(0) o3->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:277/0 lens 488/440 e 0 to 0 dl 1645083367 ref 1 fl Interpret:/0/0 rc 0/0
[13857389.394168] LustreError: 229137:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 10 previous similar messages
[13857389.394350] Lustre: oak-OST0131: Bulk IO read error with c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1), client will retry: rc -110
[13857389.394351] Lustre: Skipped 12 previous similar messages
[13857389.394669] Lustre: oak-OST0133: Bulk IO write error with 5a4881be-a0cb-e632-509b-2e15c534b21a (at 10.210.12.9@tcp1), client will retry: rc = -110
[13857389.394670] Lustre: Skipped 20 previous similar messages
[13857389.492898] LustreError: 162690:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[13857398.075426] Lustre: oak-OST0117: Client 5a4881be-a0cb-e632-509b-2e15c534b21a (at 10.210.12.9@tcp1) reconnecting
[13857398.085759] Lustre: Skipped 21 previous similar messages
[13857404.119903] LustreError: 137-5: oak-OST0144_UUID: not available for connect from 10.210.12.23@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13857404.137905] LustreError: Skipped 1 previous similar message
[13857409.002331] LustreError: 243450:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff91d835190850 x1714973264113280/t0(0) o3->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:363/0 lens 488/440 e 0 to 0 dl 1645083453 ref 1 fl Interpret:/0/0 rc 0/0
[13857409.026763] LustreError: 243450:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 21 previous similar messages
[13857492.957566] Lustre: oak-OST0149: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[13857492.968001] Lustre: Skipped 32 previous similar messages
[13857541.611051] LustreError: 160907:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff91cd94520050 x1714950910823808/t0(0) o3->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:496/0 lens 488/440 e 0 to 0 dl 1645083586 ref 1 fl Interpret:/0/0 rc 0/0
[13857541.613056] Lustre: oak-OST0135: Bulk IO read error with 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1), client will retry: rc -110
[13857541.613056] Lustre: Skipped 7 previous similar messages
[13857541.654206] LustreError: 160907:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[13857835.795848] Lustre: oak-OST0131: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[13857835.806518] Lustre: Skipped 796 previous similar messages
[13858005.161273] Lustre: oak-OST0141: Client f4048b3b-d9e4-178d-cd58-3b66faaca4ee (at 10.210.12.117@tcp1) reconnecting
[13858005.171783] Lustre: Skipped 133 previous similar messages
[13858005.232366] LustreError: 21589:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9183cacdc050 x1714908102012928/t0(0) o4->a2f24d85-30b4-0808-a59e-1ce0f2193fa6@10.210.12.121@tcp1:204/0 lens 488/448 e 0 to 0 dl 1645084049 ref 1 fl Interpret:/0/0 rc 0/0
[13858005.257103] Lustre: oak-OST012f: Bulk IO write error with a2f24d85-30b4-0808-a59e-1ce0f2193fa6 (at 10.210.12.121@tcp1), client will retry: rc = -110
[13858005.270652] Lustre: Skipped 1 previous similar message
[13858060.078079] LustreError: 162701:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff9182d78ff850 x1714908102006528/t0(0) o4->a2f24d85-30b4-0808-a59e-1ce0f2193fa6@10.210.12.121@tcp1:202/0 lens 488/448 e 0 to 0 dl 1645084047 ref 1 fl Interpret:/0/0 rc 0/0
[13858060.078106] LustreError: 21605:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff918ce4efd850 x1714909713061248/t0(0) o3->f4048b3b-d9e4-178d-cd58-3b66faaca4ee@10.210.12.117@tcp1:202/0 lens 488/440 e 0 to 0 dl 1645084047 ref 1 fl Interpret:/0/0 rc 0/0
[13858060.078108] LustreError: 21605:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 4 previous similar messages
[13858060.078277] Lustre: oak-OST013f: Bulk IO read error with f4048b3b-d9e4-178d-cd58-3b66faaca4ee (at 10.210.12.117@tcp1), client will retry: rc -110
[13858060.078278] Lustre: Skipped 1 previous similar message
[13858060.157941] LustreError: 162701:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[13858084.902240] LustreError: 137-5: oak-OST014c_UUID: not available for connect from 10.210.12.54@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13858087.369738] LustreError: 243544:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9150f952a850 x1715372268675840/t0(0) o4->5a4881be-a0cb-e632-509b-2e15c534b21a@10.210.12.9@tcp1:284/0 lens 488/448 e 0 to 0 dl 1645084129 ref 1 fl Interpret:/0/0 rc 0/0
[13858087.395557] LustreError: 243544:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 14 previous similar messages
[13858181.663016] Lustre: oak-OST0143: Bulk IO write error with 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1), client will retry: rc = -110
[13858181.676457] Lustre: Skipped 20 previous similar messages
[13858190.300147] LustreError: 160936:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk READ failed: rc -107  req@ffff91e5c832e850 x1714973292100224/t0(0) o3->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:393/0 lens 488/440 e 0 to 0 dl 1645084238 ref 1 fl Interpret:/0/0 rc 0/0
[13858203.771864] LustreError: 21589:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff9180c513b050 x1716243084448128/t0(0) o4->6b341d38-3674-15b6-7e5e-137d0b4498c0@10.210.12.40@tcp1:357/0 lens 488/448 e 0 to 0 dl 1645084202 ref 1 fl Interpret:/0/0 rc 0/0
[13858227.716663] LustreError: 243496:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff918f80bb9050 x1714951024028032/t0(0) o3->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:378/0 lens 488/440 e 0 to 0 dl 1645084223 ref 1 fl Interpret:/0/0 rc 0/0
[13858227.717104] LustreError: 162671:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff918aa973d850 x1716210720679424/t0(0) o4->c4b979fd-3b98-af30-5ea5-32f00f4d8750@10.210.12.64@tcp1:381/0 lens 488/448 e 0 to 0 dl 1645084226 ref 1 fl Interpret:/0/0 rc 0/0
[13858227.767597] LustreError: 243496:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 9 previous similar messages
[13858235.630755] LustreError: 137-5: oak-OST0136_UUID: not available for connect from 10.210.12.40@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13858235.648730] LustreError: Skipped 2 previous similar messages
[13858297.228721] LustreError: 21588:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9153e2f17050 x1715372269412416/t0(0) o4->5a4881be-a0cb-e632-509b-2e15c534b21a@10.210.12.9@tcp1:496/0 lens 488/448 e 0 to 0 dl 1645084341 ref 1 fl Interpret:/0/0 rc 0/0
[13858297.253004] LustreError: 21588:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 15 previous similar messages
[13858305.621265] Lustre: oak-OST0111: Client a2f24d85-30b4-0808-a59e-1ce0f2193fa6 (at 10.210.12.121@tcp1) reconnecting
[13858305.631777] Lustre: Skipped 87 previous similar messages
[13858371.419489] LustreError: 199269:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff915287333850 x1714908105018944/t0(0) o4->a2f24d85-30b4-0808-a59e-1ce0f2193fa6@10.210.12.121@tcp1:503/0 lens 488/448 e 0 to 0 dl 1645084348 ref 1 fl Interpret:/0/0 rc 0/0
[13858371.445895] LustreError: 199269:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[13858434.896878] Lustre: oak-OST0111: Connection restored to  (at 10.50.0.62@o2ib2)
[13858434.904373] Lustre: Skipped 826 previous similar messages
[13858903.642929] LustreError: 160923:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91f09e3b7850 x1714906792186048/t0(0) o4->a166a60d-30d5-1e72-b799-745de1d3b307@10.210.12.125@tcp1:349/0 lens 488/448 e 0 to 0 dl 1645084949 ref 1 fl Interpret:/0/0 rc 0/0
[13858903.667473] LustreError: 160923:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 13 previous similar messages
[13858903.677508] Lustre: oak-OST0145: Bulk IO write error with a166a60d-30d5-1e72-b799-745de1d3b307 (at 10.210.12.125@tcp1), client will retry: rc = -110
[13858903.691028] Lustre: Skipped 29 previous similar messages
[13858905.611902] Lustre: oak-OST0145: Client a166a60d-30d5-1e72-b799-745de1d3b307 (at 10.210.12.125@tcp1) reconnecting
[13858905.622410] Lustre: Skipped 311 previous similar messages
[13858969.201580] LustreError: 160900:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91d399371050 x1714973319782464/t0(0) o4->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:348/0 lens 488/448 e 0 to 0 dl 1645084948 ref 1 fl Interpret:/0/0 rc 0/0
[13858984.433859] LustreError: 137-5: oak-OST0144_UUID: not available for connect from 10.210.12.23@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13858984.451585] LustreError: Skipped 12 previous similar messages
[13859034.112079] Lustre: oak-OST014d: Connection restored to 8249fe9a-ed9a-7765-c7e5-3a0317d2fdde (at 10.210.12.121@tcp1)
[13859034.122868] Lustre: Skipped 612 previous similar messages
[13859063.274973] Lustre: oak-OST0143: Bulk IO read error with c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1), client will retry: rc -110
[13859063.288193] Lustre: Skipped 17 previous similar messages
[13859100.767866] Lustre: oak-OST0131: Bulk IO read error with c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1), client will retry: rc -110
[13859112.917592] LustreError: 199271:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 3145728(4194304)  req@ffff9154b9330850 x1715063656173952/t0(0) o3->bb55a01e-02a3-cbba-5b76-e478fd40967c@10.210.12.54@tcp1:508/0 lens 488/440 e 0 to 0 dl 1645085108 ref 1 fl Interpret:/0/0 rc 0/0
[13859112.943163] LustreError: 199271:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 5 previous similar messages
[13859160.815998] LustreError: 21587:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff913b2cfba050 x1714910120130816/t0(0) o4->c9a6c6f3-0875-9cfd-dcbf-edc86f3433a7@10.210.12.131@tcp1:545/0 lens 488/448 e 0 to 0 dl 1645085145 ref 1 fl Interpret:/0/0 rc 0/0
[13859160.841810] LustreError: 21587:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[13859184.765992] Lustre: oak-OST0149: Bulk IO read error with d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1), client will retry: rc -110
[13859184.775182] LustreError: 228401:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91a19cf69850 x1714973322366848/t0(0) o4->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:575/0 lens 488/448 e 0 to 0 dl 1645085175 ref 1 fl Interpret:/0/0 rc 0/0
[13859184.804965] Lustre: Skipped 5 previous similar messages
[13859367.844827] Lustre: oak-OST0125: Bulk IO read error with bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1), client will retry: rc -110
[13859376.385199] LustreError: 243498:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff918c2203f850 x1714951191560640/t0(0) o4->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:7/0 lens 488/448 e 0 to 0 dl 1645085362 ref 1 fl Interpret:/0/0 rc 0/0
[13859508.564831] Lustre: oak-OST0147: Client 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1) reconnecting
[13859508.575294] Lustre: Skipped 237 previous similar messages
[13859509.335937] LustreError: 199271:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91376dbbb050 x1715759576935424/t0(0) o4->4b39aac6-2dc7-e48f-9434-e3bfde492453@10.210.12.29@tcp1:199/0 lens 488/448 e 0 to 0 dl 1645085554 ref 1 fl Interpret:/0/0 rc 0/0
[13859509.360363] LustreError: 199271:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 36 previous similar messages
[13859509.370308] Lustre: oak-OST0147: Bulk IO write error with 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1), client will retry: rc = -110
[13859509.383753] Lustre: Skipped 37 previous similar messages
[13859524.729542] Lustre: 206916:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1645085324/real 1645085324]  req@ffff918f4edbf500 x1710530599197056/t0(0) o104->oak-OST013d@10.210.12.7@tcp1:15/16 lens 296/224 e 0 to 1 dl 1645085477 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
[13859524.757132] Lustre: 206916:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 1 previous similar message
[13859567.989773] LustreError: 21588:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff916a1ae18050 x1715372271115840/t0(0) o4->5a4881be-a0cb-e632-509b-2e15c534b21a@10.210.12.9@tcp1:201/0 lens 488/448 e 0 to 0 dl 1645085556 ref 1 fl Interpret:/0/0 rc 0/0
[13859568.015400] LustreError: 21588:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[13859587.285536] LustreError: 137-5: oak-OST014e_UUID: not available for connect from 10.210.12.73@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13859587.303126] LustreError: Skipped 6 previous similar messages
[13859637.472944] Lustre: oak-OST013f: Connection restored to 1762a9b8-4da9-4e57-339b-f4fe97cc50d0 (at 10.51.1.52@o2ib3)
[13859637.483765] Lustre: Skipped 681 previous similar messages
[13859682.531114] Lustre: oak-OST0135: Bulk IO read error with 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1), client will retry: rc -110
[13859682.544336] Lustre: Skipped 8 previous similar messages
[13860175.143050] Lustre: oak-OST014d: Client c9a6c6f3-0875-9cfd-dcbf-edc86f3433a7 (at 10.210.12.131@tcp1) reconnecting
[13860175.153561] Lustre: Skipped 263 previous similar messages
[13860176.127494] LustreError: 228401:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff919505bba850 x1714910123795904/t0(0) o4->c9a6c6f3-0875-9cfd-dcbf-edc86f3433a7@10.210.12.131@tcp1:115/0 lens 488/448 e 0 to 0 dl 1645086225 ref 1 fl Interpret:/0/0 rc 0/0
[13860176.152008] LustreError: 228401:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 10 previous similar messages
[13860176.162130] Lustre: oak-OST014d: Bulk IO write error with c9a6c6f3-0875-9cfd-dcbf-edc86f3433a7 (at 10.210.12.131@tcp1), client will retry: rc = -110
[13860176.175652] Lustre: Skipped 13 previous similar messages
[13860238.585728] LustreError: 243457:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91bc04244050 x1714910123795328/t0(0) o4->c9a6c6f3-0875-9cfd-dcbf-edc86f3433a7@10.210.12.131@tcp1:112/0 lens 488/448 e 0 to 0 dl 1645086222 ref 1 fl Interpret:/0/0 rc 0/0
[13860238.611640] LustreError: 243457:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 3 previous similar messages
[13860241.715141] Lustre: oak-OST0131: Connection restored to 313e46a1-96cb-c54a-bb11-60568f017fc9 (at 10.50.5.47@o2ib2)
[13860241.725735] Lustre: Skipped 1105 previous similar messages
[13860568.757596] LustreError: 162673:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff917d0e8d0050 x1714973361584640/t0(0) o4->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:512/0 lens 488/448 e 0 to 0 dl 1645086622 ref 1 fl Interpret:/2/0 rc 0/0
[13860568.782295] LustreError: 162673:0:(ldlm_lib.c:3305:target_bulk_io()) Skipped 1 previous similar message
[13860621.758736] LustreError: 162690:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(1048576)  req@ffff914fa17b7050 x1715000455753216/t0(0) o3->6f45695d-f173-ee1d-e6cb-d38dad7e0879@10.210.12.74@tcp1:511/0 lens 488/440 e 0 to 0 dl 1645086621 ref 1 fl Interpret:/0/0 rc 0/0
[13860621.761820] Lustre: oak-OST0131: Bulk IO read error with c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1), client will retry: rc -110
[13860621.797029] LustreError: 162690:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 10 previous similar messages
[13860647.505447] LustreError: 137-5: oak-OST0156_UUID: not available for connect from 10.210.12.60@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13860647.523092] LustreError: Skipped 4 previous similar messages
[13860840.311567] Lustre: oak-OST011b: Connection restored to 1cae4158-4c34-2f7a-eb0b-da9101bf8a24 (at 10.50.5.54@o2ib2)
[13860840.322152] Lustre: Skipped 636 previous similar messages
[13860842.946829] Lustre: oak-OST0123: Client c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1) reconnecting
[13860842.957245] Lustre: Skipped 128 previous similar messages
[13860843.378984] LustreError: 127347:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91b416606850 x1715011814869056/t0(0) o4->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:29/0 lens 488/448 e 0 to 0 dl 1645086894 ref 1 fl Interpret:/0/0 rc 0/0
[13860843.403322] LustreError: 127347:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 7 previous similar messages
[13860843.413277] Lustre: oak-OST0123: Bulk IO write error with c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1), client will retry: rc = -110
[13860843.426729] Lustre: Skipped 10 previous similar messages
[13860885.181027] LustreError: 253952:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff91f137739850 x1715011814787392/t0(0) o3->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:25/0 lens 488/440 e 0 to 0 dl 1645086890 ref 1 fl Interpret:/0/0 rc 0/0
[13860885.205815] LustreError: 253952:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 1 previous similar message
[13861196.494555] LustreError: 160921:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91a99d748050 x1714951300108032/t0(0) o4->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:321/0 lens 488/448 e 0 to 0 dl 1645087186 ref 1 fl Interpret:/0/0 rc 0/0
[13861196.495336] LustreError: 199274:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff91f1a58a8850 x1714951300185664/t0(0) o3->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:326/0 lens 488/440 e 0 to 0 dl 1645087191 ref 1 fl Interpret:/0/0 rc 0/0
[13861196.545417] LustreError: 160921:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 5 previous similar messages
[13861268.344237] LustreError: 243526:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff915f89d00850 x1715759623839168/t0(0) o4->4b39aac6-2dc7-e48f-9434-e3bfde492453@10.210.12.29@tcp1:398/0 lens 488/448 e 0 to 0 dl 1645087263 ref 1 fl Interpret:/0/0 rc 0/0
[13861268.370044] LustreError: 243526:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[13861334.231279] LustreError: 137-5: oak-OST014c_UUID: not available for connect from 10.210.12.65@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13861334.248942] LustreError: Skipped 2 previous similar messages
[13861388.083182] LustreError: 160902:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91ad04274050 x1714951309616192/t0(0) o4->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:506/0 lens 488/448 e 0 to 0 dl 1645087371 ref 1 fl Interpret:/0/0 rc 0/0
[13861388.109041] LustreError: 160902:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 3 previous similar messages
[13861440.857027] Lustre: oak-OST012f: Connection restored to 313e46a1-96cb-c54a-bb11-60568f017fc9 (at 10.50.5.47@o2ib2)
[13861440.867622] Lustre: Skipped 705 previous similar messages
[13861441.589497] Lustre: oak-OST0127: Client 80374d70-7d99-f68a-b58b-86c0b7514d6a (at 10.210.12.65@tcp1) reconnecting
[13861441.599910] Lustre: Skipped 287 previous similar messages
[13861451.356455] LustreError: 162704:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9167ff1c8050 x1715350448814720/t0(0) o4->1b588188-9255-1b9f-1308-154ba9e467a3@10.210.12.7@tcp1:637/0 lens 488/448 e 0 to 0 dl 1645087502 ref 1 fl Interpret:/0/0 rc 0/0
[13861451.380856] LustreError: 162704:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 5 previous similar messages
[13861451.390840] Lustre: oak-OST014b: Bulk IO write error with 1b588188-9255-1b9f-1308-154ba9e467a3 (at 10.210.12.7@tcp1), client will retry: rc = -110
[13861451.404285] Lustre: Skipped 18 previous similar messages
[13861454.178865] LustreError: 243535:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff916c0e52f050 x1714951310715328/t0(0) o4->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:644/0 lens 488/448 e 0 to 0 dl 1645087509 ref 1 fl Interpret:/2/0 rc 0/0
[13861454.203574] LustreError: 243535:0:(ldlm_lib.c:3305:target_bulk_io()) Skipped 1 previous similar message
[13861507.834289] LustreError: 162683:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff91586dc8a850 x1714979622317696/t0(0) o3->d4faaef8-44e2-b05f-6116-d9c30b2e4e11@10.210.12.60@tcp1:638/0 lens 488/440 e 0 to 0 dl 1645087503 ref 1 fl Interpret:/0/0 rc 0/0
[13861507.834291] LustreError: 162709:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff91709d75e050 x1714979622317504/t0(0) o3->d4faaef8-44e2-b05f-6116-d9c30b2e4e11@10.210.12.60@tcp1:638/0 lens 488/440 e 0 to 0 dl 1645087503 ref 1 fl Interpret:/0/0 rc 0/0
[13861507.834293] LustreError: 162709:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 1 previous similar message
[13861507.834438] Lustre: oak-OST013f: Bulk IO read error with d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1), client will retry: rc -110
[13861507.834439] Lustre: Skipped 5 previous similar messages
[13861507.913834] LustreError: 162683:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 1 previous similar message
[13861597.255891] LustreError: 199244:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 149s: evicting client at 10.210.12.31@tcp1  ns: filter-oak-OST0113_UUID lock: ffff913ae6e79b00/0xed112d301e6f79c5 lrc: 3/0,0 mode: PW/PW res: [0x47c0000400:0x14c8200:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x60000400010020 nid: 10.210.12.31@tcp1 remote: 0x60404e1af54a2089 expref: 6 pid: 203089 timeout: 13895288 lvb_type: 0
[13861609.183330] LustreError: 228366:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk READ failed: rc -107  req@ffff91586dc8a850 x1715063811729152/t0(0) o3->bb55a01e-02a3-cbba-5b76-e478fd40967c@10.210.12.54@tcp1:45/0 lens 488/440 e 0 to 0 dl 1645087665 ref 1 fl Interpret:/0/0 rc 0/0
[13861675.465844] LustreError: 199270:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff918f8ec41050 x1715063811582336/t0(0) o4->bb55a01e-02a3-cbba-5b76-e478fd40967c@10.210.12.54@tcp1:38/0 lens 488/448 e 0 to 0 dl 1645087658 ref 1 fl Interpret:/0/0 rc 0/0
[13861675.491568] LustreError: 199270:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 6 previous similar messages
[13861775.386988] Lustre: 203719:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1645087581/real 1645087581]  req@ffff915e218b2400 x1710530606005376/t0(0) o104->oak-OST0127@10.210.12.31@tcp1:15/16 lens 296/224 e 0 to 1 dl 1645087734 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
[13861947.225249] LustreError: 137-5: oak-OST014c_UUID: not available for connect from 10.210.12.65@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13861947.242925] LustreError: Skipped 8 previous similar messages
[13862039.547368] Lustre: oak-OST012b: Connection restored to 080fbf36-50e2-e037-d579-97f7ea4dd6a6 (at 10.210.12.51@tcp1)
[13862039.558051] Lustre: Skipped 963 previous similar messages
[13862040.752501] Lustre: oak-OST0137: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[13862040.762925] Lustre: Skipped 426 previous similar messages
[13862226.242037] LustreError: 243453:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91c54e3f1050 x1715759632513920/t0(0) o4->4b39aac6-2dc7-e48f-9434-e3bfde492453@10.210.12.29@tcp1:597/0 lens 488/448 e 0 to 0 dl 1645088217 ref 1 fl Interpret:/0/0 rc 0/0
[13862226.242274] Lustre: oak-OST0115: Bulk IO write error with 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1), client will retry: rc = -110
[13862226.242276] Lustre: Skipped 34 previous similar messages
[13862226.286827] LustreError: 243453:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 11 previous similar messages
[13862250.190694] LustreError: 160937:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff91dc0ebbd850 x1714979634966272/t0(0) o3->d4faaef8-44e2-b05f-6116-d9c30b2e4e11@10.210.12.60@tcp1:632/0 lens 488/440 e 0 to 0 dl 1645088252 ref 1 fl Interpret:/0/0 rc 0/0
[13862250.215576] LustreError: 160937:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 7 previous similar messages
[13862250.225301] Lustre: oak-OST0149: Bulk IO read error with d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1), client will retry: rc -110
[13862250.238485] Lustre: Skipped 10 previous similar messages
[13862390.147950] LustreError: 127348:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91b8a4b29050 x1714951379154624/t0(0) o4->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:68/0 lens 488/448 e 0 to 0 dl 1645088443 ref 1 fl Interpret:/0/0 rc 0/0
[13862390.172289] LustreError: 127348:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 19 previous similar messages
[13862642.096686] Lustre: oak-OST014d: Connection restored to 020cf75a-ba01-c405-3917-2f85d05c8d52 (at 10.51.6.30@o2ib3)
[13862642.107293] Lustre: Skipped 1009 previous similar messages
[13862664.727512] Lustre: oak-OST0145: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[13862664.737931] Lustre: Skipped 239 previous similar messages
[13862700.796173] LustreError: 199273:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff91c1dad84050 x1714951390870336/t0(0) o4->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:384/0 lens 488/448 e 0 to 0 dl 1645088759 ref 1 fl Interpret:/2/0 rc 0/0
[13862757.678179] LustreError: 137-5: oak-OST013a_UUID: not available for connect from 10.210.12.73@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13862757.695775] LustreError: Skipped 2 previous similar messages
[13862799.987596] LustreError: 162683:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 1409024(2457600)  req@ffff9148b3a1e050 x1714979643230272/t0(0) o4->d4faaef8-44e2-b05f-6116-d9c30b2e4e11@10.210.12.60@tcp1:433/0 lens 504/448 e 0 to 0 dl 1645088808 ref 1 fl Interpret:/2/0 rc 0/0
[13862800.013421] LustreError: 162683:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 9 previous similar messages
[13862884.480827] Lustre: oak-OST0145: Bulk IO write error with 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1), client will retry: rc = -110
[13862884.494277] Lustre: Skipped 30 previous similar messages
[13862943.669697] LustreError: 248296:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(1048576)  req@ffff91dc17eee850 x1715024350370176/t0(0) o3->6d01c866-9ec9-8f84-4f86-ee1db83afc97@10.210.12.61@tcp1:562/0 lens 488/440 e 0 to 0 dl 1645088937 ref 1 fl Interpret:/0/0 rc 0/0
[13862943.669768] Lustre: oak-OST0131: Bulk IO read error with 6d01c866-9ec9-8f84-4f86-ee1db83afc97 (at 10.210.12.61@tcp1), client will retry: rc -110
[13862943.669769] Lustre: Skipped 2 previous similar messages
[13862943.713378] LustreError: 248296:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 3 previous similar messages
[13863057.489731] LustreError: 199273:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91d92d1d3850 x1714951423241152/t0(0) o4->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:740/0 lens 488/448 e 0 to 0 dl 1645089115 ref 1 fl Interpret:/0/0 rc 0/0
[13863057.514173] LustreError: 199273:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 16 previous similar messages
[13863243.614497] Lustre: oak-OST014b: Connection restored to 2e3e1f3e-fdf2-1561-a345-edf5d2e908ad (at 10.50.16.4@o2ib2)
[13863243.625101] Lustre: Skipped 769 previous similar messages
[13863411.574642] Lustre: oak-OST0111: Client 6d01c866-9ec9-8f84-4f86-ee1db83afc97 (at 10.210.12.61@tcp1) reconnecting
[13863411.585075] Lustre: Skipped 249 previous similar messages
[13863847.308257] Lustre: oak-OST0147: Connection restored to 72e9c356-8685-a676-ede7-2f53484b502b (at 10.50.10.7@o2ib2)
[13863847.318836] Lustre: Skipped 510 previous similar messages
[13863915.666569] LustreError: 160893:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91ca314a4850 x1714951439538752/t0(0) o4->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:87/0 lens 488/448 e 0 to 0 dl 1645089972 ref 1 fl Interpret:/0/0 rc 0/0
[13863915.690978] LustreError: 160893:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 2 previous similar messages
[13863915.701122] Lustre: oak-OST0121: Bulk IO write error with 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1), client will retry: rc = -110
[13863915.714635] Lustre: Skipped 4 previous similar messages
[13863973.416710] LustreError: 160945:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91957afb6850 x1714951439736512/t0(0) o4->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:91/0 lens 488/448 e 0 to 0 dl 1645089976 ref 1 fl Interpret:/0/0 rc 0/0
[13863973.416840] LustreError: 127347:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff91a4dd2d1050 x1714951439812224/t0(0) o3->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:92/0 lens 488/440 e 0 to 0 dl 1645089977 ref 1 fl Interpret:/0/0 rc 0/0
[13863973.416842] LustreError: 127347:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 3 previous similar messages
[13863973.416852] Lustre: oak-OST0135: Bulk IO read error with 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1), client will retry: rc -110
[13863973.416852] Lustre: Skipped 4 previous similar messages
[13863973.495663] LustreError: 160945:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[13864082.838309] Lustre: oak-OST013f: Client 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1) reconnecting
[13864082.848743] Lustre: Skipped 54 previous similar messages
[13864178.529006] LustreError: 137-5: oak-OST0156_UUID: not available for connect from 10.210.12.60@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13864178.546663] LustreError: Skipped 4 previous similar messages
[13864449.804258] Lustre: oak-OST012b: Connection restored to 566692d9-3760-eb93-5cbc-1337c48c4638 (at 10.50.16.17@o2ib2)
[13864449.814988] Lustre: Skipped 562 previous similar messages
[13864510.158884] LustreError: 137-5: oak-OST0156_UUID: not available for connect from 10.210.12.60@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13864510.176498] LustreError: Skipped 1 previous similar message
[13864667.882054] LustreError: 127348:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 1455105(2503681)  req@ffff91afb9569850 x1715092557506880/t0(0) o4->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:29/0 lens 488/448 e 0 to 0 dl 1645090669 ref 1 fl Interpret:/0/0 rc 0/0
[13864667.907832] LustreError: 127348:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 3 previous similar messages
[13864667.918046] Lustre: oak-OST012b: Bulk IO write error with 5d8bb8d9-e5c7-334f-ec8d-f10452c309b5 (at 10.210.12.66@tcp1), client will retry: rc = -110
[13864667.931999] Lustre: Skipped 8 previous similar messages
[13864687.120547] Lustre: oak-OST012b: Client 5d8bb8d9-e5c7-334f-ec8d-f10452c309b5 (at 10.210.12.66@tcp1) reconnecting
[13864687.130997] Lustre: Skipped 230 previous similar messages
[13864706.699128] LustreError: 199244:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 99s: evicting client at 10.210.12.66@tcp1  ns: filter-oak-OST0141_UUID lock: ffff91aa9a016c00/0xed112d301ea5205f lrc: 3/0,0 mode: PW/PW res: [0x5440000403:0x3810c4:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x60000400030020 nid: 10.210.12.66@tcp1 remote: 0xd91c1fbc6883b360 expref: 187 pid: 203476 timeout: 13898405 lvb_type: 0
[13864795.771659] LustreError: 199271:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91535f964050 x1715063985384768/t0(0) o4->bb55a01e-02a3-cbba-5b76-e478fd40967c@10.210.12.54@tcp1:213/0 lens 488/448 e 0 to 0 dl 1645090853 ref 1 fl Interpret:/0/0 rc 0/0
[13864795.796097] LustreError: 199271:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 3 previous similar messages
[13864859.460314] LustreError: 243445:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(1048576)  req@ffff915880a32850 x1715092558346560/t0(0) o3->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:219/0 lens 504/440 e 0 to 0 dl 1645090859 ref 1 fl Interpret:/0/0 rc 0/0
[13864859.485861] LustreError: 243445:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 4 previous similar messages
[13864859.495650] Lustre: oak-OST014d: Bulk IO read error with 5d8bb8d9-e5c7-334f-ec8d-f10452c309b5 (at 10.210.12.66@tcp1), client will retry: rc -110
[13864859.508819] Lustre: Skipped 4 previous similar messages
[13865049.796176] Lustre: oak-OST011b: Connection restored to 7aa982dc-eea3-6a5b-f096-3007f863c95c (at 10.210.12.72@tcp1)
[13865049.806842] Lustre: Skipped 516 previous similar messages
[13865050.559671] LustreError: 137-5: oak-OST011c_UUID: not available for connect from 10.210.12.59@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13865655.794287] Lustre: oak-OST0145: Client 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1) reconnecting
[13865655.804701] Lustre: Skipped 293 previous similar messages
[13865655.808298] Lustre: oak-OST0139: Connection restored to 15431f40-18fe-801a-ce02-67f7cb5f3e18 (at 10.210.12.60@tcp1)
[13865655.808299] Lustre: Skipped 655 previous similar messages
[13865655.920294] LustreError: 243496:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91770d4b0050 x1714951470301312/t0(0) o4->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:322/0 lens 488/448 e 0 to 0 dl 1645091717 ref 1 fl Interpret:/0/0 rc 0/0
[13865655.944817] LustreError: 243496:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 3 previous similar messages
[13865655.954739] Lustre: oak-OST0145: Bulk IO write error with 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1), client will retry: rc = -110
[13865655.968187] Lustre: Skipped 14 previous similar messages
[13865721.558322] LustreError: 162669:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff916d7f19b050 x1714951470300608/t0(0) o4->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:326/0 lens 488/448 e 0 to 0 dl 1645091721 ref 1 fl Interpret:/2/0 rc 0/0
[13865721.584181] LustreError: 162669:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 10 previous similar messages
[13866254.568228] Lustre: oak-OST0125: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[13866254.578935] Lustre: Skipped 547 previous similar messages
[13866272.646243] Lustre: oak-OST014b: Client c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1) reconnecting
[13866272.656776] Lustre: Skipped 30 previous similar messages
[13866272.770446] LustreError: 199272:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91e8e8601850 x1715012431911232/t0(0) o4->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:186/0 lens 488/448 e 0 to 0 dl 1645092336 ref 1 fl Interpret:/0/0 rc 0/0
[13866272.794877] LustreError: 199272:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 4 previous similar messages
[13866272.804840] Lustre: oak-OST014b: Bulk IO write error with c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1), client will retry: rc = -110
[13866272.818292] Lustre: Skipped 5 previous similar messages
[13866273.551608] LustreError: 160924:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff91d8ffc22050 x1715012431923904/t0(0) o4->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:190/0 lens 504/448 e 0 to 0 dl 1645092340 ref 1 fl Interpret:/2/0 rc 0/0
[13866352.271097] LustreError: 137-5: oak-OST014c_UUID: not available for connect from 10.210.12.25@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13866352.288694] LustreError: Skipped 1 previous similar message
[13866853.510168] Lustre: oak-OST011d: Connection restored to b80be752-f851-25c6-d1f2-d3fd377310cf (at 10.210.12.46@tcp1)
[13866853.520858] Lustre: Skipped 560 previous similar messages
[13866870.013356] LustreError: 160928:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2076384(3124960)  req@ffff91943b3c4050 x1716551343466752/t0(0) o4->6bab8d10-30ff-c1d2-bbf5-1fd0884ac647@10.210.12.57@tcp1:738/0 lens 488/448 e 0 to 0 dl 1645092888 ref 1 fl Interpret:/0/0 rc 0/0
[13866903.379414] LustreError: 137-5: oak-OST0138_UUID: not available for connect from 10.210.12.64@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13866904.495828] Lustre: oak-OST013b: Client 6bab8d10-30ff-c1d2-bbf5-1fd0884ac647 (at 10.210.12.57@tcp1) reconnecting
[13866904.506373] Lustre: Skipped 64 previous similar messages
[13867328.877701] LustreError: 160939:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91b9c732f050 x1715759711653824/t0(0) o4->4b39aac6-2dc7-e48f-9434-e3bfde492453@10.210.12.29@tcp1:489/0 lens 488/448 e 0 to 0 dl 1645093394 ref 1 fl Interpret:/0/0 rc 0/0
[13867328.902117] LustreError: 160939:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 3 previous similar messages
[13867328.912322] Lustre: oak-OST0145: Bulk IO write error with 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1), client will retry: rc = -110
[13867328.925756] Lustre: Skipped 5 previous similar messages
[13867391.051849] LustreError: 137-5: oak-OST0118_UUID: not available for connect from 10.210.12.56@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13867391.069667] LustreError: Skipped 1 previous similar message
[13867457.989436] Lustre: oak-OST012b: Connection restored to  (at 10.50.3.62@o2ib2)
[13867457.996902] Lustre: Skipped 536 previous similar messages
[13867496.535781] Lustre: oak-OST0131: Bulk IO read error with 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1), client will retry: rc -110
[13867496.548964] Lustre: Skipped 3 previous similar messages
[13867503.085840] Lustre: oak-OST012f: Client 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1) reconnecting
[13867503.096250] Lustre: Skipped 149 previous similar messages
[13867629.502990] LustreError: 137-5: oak-OST0118_UUID: not available for connect from 10.210.12.56@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13867629.520572] LustreError: Skipped 1 previous similar message
[13867779.985438] LustreError: 243542:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff914a15a06050 x1714951512066368/t0(0) o4->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:135/0 lens 488/448 e 0 to 0 dl 1645093795 ref 1 fl Interpret:/0/0 rc 0/0
[13867780.011248] LustreError: 243542:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[13868064.438491] Lustre: oak-OST013d: Connection restored to ef0ca35e-4d7b-24ea-8737-c48b55c0297f (at 10.51.13.24@o2ib3)
[13868064.449156] Lustre: Skipped 644 previous similar messages
[13868485.204337] LustreError: 137-5: oak-OST0118_UUID: not available for connect from 10.210.12.56@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13868578.975037] Lustre: oak-OST0111: Client 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1) reconnecting
[13868578.985479] Lustre: Skipped 120 previous similar messages
[13868618.102580] LustreError: 21617:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(1001681)  req@ffff91853854c050 x1714980840182976/t0(0) o4->3b1423ea-f77c-9301-8ee2-822b91cad2d9@10.210.12.10@tcp1:216/0 lens 488/448 e 0 to 0 dl 1645094631 ref 1 fl Interpret:/0/0 rc 0/0
[13868618.102767] Lustre: oak-OST0125: Bulk IO write error with a8a647a5-bb36-3966-6637-e8ee81d6d548 (at 10.210.12.6@tcp1), client will retry: rc = -110
[13868618.102768] Lustre: Skipped 11 previous similar messages
[13868618.146779] LustreError: 21617:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 4 previous similar messages
[13868666.146114] Lustre: oak-OST0141: Connection restored to a24dd7a0-5fa8-502a-258a-91fb9f94b4c0 (at 10.50.5.43@o2ib2)
[13868666.156724] Lustre: Skipped 433 previous similar messages
[13869240.684514] LustreError: 162676:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff917133905050 x1715350566595008/t0(0) o4->1b588188-9255-1b9f-1308-154ba9e467a3@10.210.12.7@tcp1:74/0 lens 488/448 e 0 to 0 dl 1645095244 ref 1 fl Interpret:/0/0 rc 0/0
[13869240.684630] Lustre: oak-OST0137: Bulk IO write error with 1b588188-9255-1b9f-1308-154ba9e467a3 (at 10.210.12.7@tcp1), client will retry: rc = -110
[13869240.684632] Lustre: Skipped 4 previous similar messages
[13869240.728966] LustreError: 162676:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[13869251.920679] Lustre: oak-OST014b: Client 1b588188-9255-1b9f-1308-154ba9e467a3 (at 10.210.12.7@tcp1) reconnecting
[13869251.931007] Lustre: Skipped 153 previous similar messages
[13869269.753997] Lustre: oak-OST0123: Connection restored to  (at 10.51.15.8@o2ib3)
[13869269.761471] Lustre: Skipped 661 previous similar messages
[13869398.228610] LustreError: 137-5: oak-OST0148_UUID: not available for connect from 10.210.12.60@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13869398.246233] LustreError: Skipped 5 previous similar messages
[13869619.731416] LustreError: 127350:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff919f7828b850 x1714980850687616/t0(0) o4->3b1423ea-f77c-9301-8ee2-822b91cad2d9@10.210.12.10@tcp1:524/0 lens 488/448 e 0 to 0 dl 1645095694 ref 1 fl Interpret:/0/0 rc 0/0
[13869619.755885] LustreError: 127350:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 9 previous similar messages
[13869619.765705] Lustre: oak-OST013f: Bulk IO write error with 3b1423ea-f77c-9301-8ee2-822b91cad2d9 (at 10.210.12.10@tcp1), client will retry: rc = -110
[13869619.779148] Lustre: Skipped 5 previous similar messages
[13869719.583483] LustreError: 244099:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(1044480)  req@ffff9195c73ae050 x1715084856992896/t0(0) o3->65e9b8a8-b461-24e0-da12-5b6939f0b07c@10.210.12.56@tcp1:553/0 lens 488/440 e 0 to 0 dl 1645095723 ref 1 fl Interpret:/0/0 rc 0/0
[13869719.583550] Lustre: oak-OST0131: Bulk IO read error with 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1), client will retry: rc -110
[13869719.621711] LustreError: 244099:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 4 previous similar messages
[13869823.184918] LustreError: 243334:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91cd2b2be850 x1715350574020288/t0(0) o4->1b588188-9255-1b9f-1308-154ba9e467a3@10.210.12.7@tcp1:721/0 lens 488/448 e 0 to 0 dl 1645095891 ref 1 fl Interpret:/0/0 rc 0/0
[13869823.209331] LustreError: 243334:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 2 previous similar messages
[13869868.703949] Lustre: oak-OST0113: Connection restored to  (at 10.51.3.33@o2ib3)
[13869868.711591] Lustre: Skipped 939 previous similar messages
[13869901.382920] Lustre: oak-OST0123: Client 1b588188-9255-1b9f-1308-154ba9e467a3 (at 10.210.12.7@tcp1) reconnecting
[13869901.393260] Lustre: Skipped 410 previous similar messages
[13870294.245122] LustreError: 229135:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(24435)  req@ffff9191fc8fa050 x1715240557019136/t0(0) o4->e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda@10.210.12.79@tcp1:375/0 lens 488/448 e 0 to 0 dl 1645096300 ref 1 fl Interpret:/0/0 rc 0/0
[13870294.270229] LustreError: 229135:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 9 previous similar messages
[13870294.280314] Lustre: oak-OST013d: Bulk IO write error with e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1), client will retry: rc = -110
[13870294.293753] Lustre: Skipped 10 previous similar messages
[13870309.619239] LustreError: 137-5: oak-OST013c_UUID: not available for connect from 10.210.12.56@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13870309.636908] LustreError: Skipped 12 previous similar messages
[13870468.233950] Lustre: oak-OST014d: Connection restored to d16cb253-11dc-02a5-01bc-e4ca96f3f9b7 (at 10.50.12.8@o2ib2)
[13870468.244552] Lustre: Skipped 804 previous similar messages
[13870537.085247] Lustre: oak-OST0129: Client 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1) reconnecting
[13870537.095670] Lustre: Skipped 233 previous similar messages
[13870557.640994] LustreError: 243444:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(8192)  req@ffff91456593c050 x1715064267351936/t0(0) o3->bb55a01e-02a3-cbba-5b76-e478fd40967c@10.210.12.54@tcp1:650/0 lens 504/440 e 0 to 0 dl 1645096575 ref 1 fl Interpret:/0/0 rc 0/0
[13870557.665763] Lustre: oak-OST014d: Bulk IO read error with bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1), client will retry: rc -110
[13870557.678934] Lustre: Skipped 1 previous similar message
[13870728.346130] LustreError: 160920:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91d43e347850 x1715372291629312/t0(0) o4->5a4881be-a0cb-e632-509b-2e15c534b21a@10.210.12.9@tcp1:122/0 lens 488/448 e 0 to 0 dl 1645096802 ref 1 fl Interpret:/0/0 rc 0/0
[13870728.370476] LustreError: 160920:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 3 previous similar messages
[13870855.033251] LustreError: 199270:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9171419b3050 x1714980877705792/t0(0) o4->3b1423ea-f77c-9301-8ee2-822b91cad2d9@10.210.12.10@tcp1:247/0 lens 488/448 e 0 to 0 dl 1645096927 ref 1 fl Interpret:/0/0 rc 0/0
[13870855.057702] LustreError: 199270:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[13870933.502394] LustreError: 137-5: oak-OST013a_UUID: not available for connect from 10.210.12.10@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13870933.520048] LustreError: Skipped 2 previous similar messages
[13871066.779994] Lustre: oak-OST0147: Connection restored to f1dd735d-6c66-d8b4-50d4-60d2b73caa84 (at 10.51.16.3@o2ib3)
[13871066.790574] Lustre: Skipped 1262 previous similar messages
[13871666.835519] Lustre: oak-OST0113: Connection restored to 5e4bce85-4d6e-90c6-523c-4bdf35e8bc4a (at 10.210.12.7@tcp1)
[13871666.846113] Lustre: Skipped 763 previous similar messages
[13872269.369395] Lustre: oak-OST013f: Connection restored to  (at 10.50.9.18@o2ib2)
[13872269.376890] Lustre: Skipped 847 previous similar messages
[13872873.096963] Lustre: oak-OST014b: Connection restored to 5df0bdab-b30e-504d-aa3f-1cb04e46cc0f (at 10.51.14.1@o2ib3)
[13872873.107615] Lustre: Skipped 609 previous similar messages
[13873229.655578] Lustre: oak-OST012d: Client 5d8bb8d9-e5c7-334f-ec8d-f10452c309b5 (at 10.210.12.66@tcp1) reconnecting
[13873229.665992] Lustre: Skipped 278 previous similar messages
[13873265.893621] LustreError: 137-5: oak-OST014a_UUID: not available for connect from 10.210.12.7@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13873358.281110] Lustre: oak-OST014d: Client 1b588188-9255-1b9f-1308-154ba9e467a3 (at 10.210.12.7@tcp1) reconnecting
[13873358.291902] Lustre: Skipped 15 previous similar messages
[13873473.416513] Lustre: oak-OST0129: Connection restored to 348a13e9-7855-7c45-c824-7175c68035a2 (at 10.50.5.51@o2ib2)
[13873473.427131] Lustre: Skipped 615 previous similar messages
[13874074.119821] Lustre: oak-OST013d: Connection restored to f1dd735d-6c66-d8b4-50d4-60d2b73caa84 (at 10.51.16.3@o2ib3)
[13874074.130460] Lustre: Skipped 664 previous similar messages
[13874672.905566] Lustre: oak-OST0129: Connection restored to dfba191f-99e8-9503-e2b2-904454040fbb (at 10.50.5.34@o2ib2)
[13874672.916186] Lustre: Skipped 747 previous similar messages
[13875272.247547] Lustre: oak-OST0125: Connection restored to a7228c75-c53b-e2c9-6e64-f881b06e64fc (at 10.50.13.13@o2ib2)
[13875272.258227] Lustre: Skipped 656 previous similar messages
[13875881.574403] Lustre: oak-OST014d: Connection restored to 834d8ea1-98bb-d04d-e595-6857c1f41a64 (at 10.50.14.11@o2ib2)
[13875881.585185] Lustre: Skipped 656 previous similar messages
[13876480.815170] Lustre: oak-OST0149: Connection restored to 3bd5bbc1-b37d-8e15-f28d-8cffa53b2377 (at 10.50.7.56@o2ib2)
[13876480.825829] Lustre: Skipped 495 previous similar messages
[13877081.335662] Lustre: oak-OST0135: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[13877081.346337] Lustre: Skipped 485 previous similar messages
[13877684.471520] Lustre: oak-OST0111: Connection restored to cfc81cd1-c362-4794-1543-9902e3f47ae0 (at 10.50.1.48@o2ib2)
[13877684.482124] Lustre: Skipped 645 previous similar messages
[13878289.703209] Lustre: oak-OST0125: Connection restored to 87339822-dadb-a201-18dc-2977e845c0c2 (at 10.50.9.51@o2ib2)
[13878289.713828] Lustre: Skipped 495 previous similar messages
[13878391.474091] Lustre: oak-OST0111: Client c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1) reconnecting
[13878391.484499] Lustre: Skipped 38 previous similar messages
[13878889.067199] Lustre: oak-OST014d: Connection restored to  (at 10.50.15.6@o2ib2)
[13878889.074673] Lustre: Skipped 518 previous similar messages
[13879487.866571] Lustre: oak-OST011b: Connection restored to 7c7914fd-c95a-edf9-2fc3-9b8dacc86ea7 (at 10.51.2.10@o2ib3)
[13879487.877157] Lustre: Skipped 1651 previous similar messages
[13880087.181816] Lustre: oak-OST013f: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[13880087.192502] Lustre: Skipped 670 previous similar messages
[13880687.172334] Lustre: oak-OST0145: Connection restored to  (at 10.51.13.4@o2ib3)
[13880687.179804] Lustre: Skipped 841 previous similar messages
[13881287.917457] Lustre: oak-OST0137: Connection restored to ec90b2dd-3b9e-e42f-4c1b-048b18586a99 (at 10.51.13.10@o2ib3)
[13881287.928125] Lustre: Skipped 571 previous similar messages
[13881886.639527] Lustre: oak-OST0125: Connection restored to 87cb0d7a-691a-f49e-d75a-6a4f665d73e8 (at 10.51.5.51@o2ib3)
[13881886.650129] Lustre: Skipped 652 previous similar messages
[13882486.327116] Lustre: oak-OST013f: Connection restored to ea7323ea-d4bb-2678-29a1-bc8d87c76269 (at 10.210.12.6@tcp1)
[13882486.337736] Lustre: Skipped 785 previous similar messages
[13883085.124769] Lustre: oak-OST011f: Connection restored to 26b56047-6e10-44fc-530a-21d82261e5c5 (at 10.50.2.10@o2ib2)
[13883085.135346] Lustre: Skipped 669 previous similar messages
[13883684.634313] Lustre: oak-OST0135: Connection restored to f1dd735d-6c66-d8b4-50d4-60d2b73caa84 (at 10.51.16.3@o2ib3)
[13883684.644901] Lustre: Skipped 1578 previous similar messages
[13884283.634817] Lustre: oak-OST011d: Connection restored to b5ea0a18-73a7-66cc-dfe7-e0feb03bf096 (at 10.50.5.33@o2ib2)
[13884283.645407] Lustre: Skipped 918 previous similar messages
[13884882.199549] Lustre: oak-OST012f: Connection restored to 978cda10-455c-b12a-7f7e-9260335b9b21 (at 10.50.14.6@o2ib2)
[13884882.210179] Lustre: Skipped 840 previous similar messages
[13885329.415066] LustreError: 21591:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(1048576)  req@ffff91867b73f050 x1715085259590272/t0(0) o3->65e9b8a8-b461-24e0-da12-5b6939f0b07c@10.210.12.56@tcp1:363/0 lens 488/440 e 0 to 0 dl 1645111388 ref 1 fl Interpret:/0/0 rc 0/0
[13885329.440146] Lustre: oak-OST014d: Bulk IO read error with 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1), client will retry: rc -110
[13885452.843269] Lustre: oak-OST0121: Client 5afa23c2-1fd5-9030-38ce-b03a359082fe (at 10.210.12.75@tcp1) reconnecting
[13885452.853674] Lustre: Skipped 2 previous similar messages
[13885456.839864] Lustre: oak-OST0111: Client 5afa23c2-1fd5-9030-38ce-b03a359082fe (at 10.210.12.75@tcp1) reconnecting
[13885456.850286] Lustre: Skipped 16 previous similar messages
[13885480.825141] Lustre: oak-OST0147: Connection restored to cccd1b41-e9ad-6ea9-1227-d89af4fce67b (at 10.50.16.1@o2ib2)
[13885480.835739] Lustre: Skipped 520 previous similar messages
[13885491.090828] Lustre: oak-OST012f: Client 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1) reconnecting
[13885491.101236] Lustre: Skipped 5 previous similar messages
[13885500.693910] Lustre: oak-OST0141: Client 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1) reconnecting
[13885500.704313] Lustre: Skipped 16 previous similar messages
[13886080.681715] Lustre: oak-OST0135: Connection restored to c476e010-0b79-ff80-57df-a0bb35969f90 (at 10.51.12.1@o2ib3)
[13886080.692298] Lustre: Skipped 669 previous similar messages
[13886686.063657] Lustre: oak-OST011d: Connection restored to ef0ca35e-4d7b-24ea-8737-c48b55c0297f (at 10.51.13.24@o2ib3)
[13886686.074330] Lustre: Skipped 1093 previous similar messages
[13887291.975282] Lustre: oak-OST0139: Connection restored to 8a175aee-31b6-e168-94c2-4812b701af81 (at 10.51.2.60@o2ib3)
[13887291.985864] Lustre: Skipped 839 previous similar messages
[13887892.611245] Lustre: oak-OST0123: Connection restored to 080fbf36-50e2-e037-d579-97f7ea4dd6a6 (at 10.210.12.51@tcp1)
[13887892.621919] Lustre: Skipped 998 previous similar messages
[13888495.818331] Lustre: oak-OST012d: Connection restored to  (at 10.50.9.27@o2ib2)
[13888495.825822] Lustre: Skipped 749 previous similar messages
[13888672.954873] Lustre: oak-OST013b: Client 1b588188-9255-1b9f-1308-154ba9e467a3 (at 10.210.12.7@tcp1) reconnecting
[13888672.965197] Lustre: Skipped 11 previous similar messages
[13888673.296401] LustreError: 243555:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff918ed6f01850 x1715350857728192/t0(0) o4->1b588188-9255-1b9f-1308-154ba9e467a3@10.210.12.7@tcp1:745/0 lens 488/448 e 0 to 0 dl 1645114790 ref 1 fl Interpret:/0/0 rc 0/0
[13888673.321034] Lustre: oak-OST013b: Bulk IO write error with 1b588188-9255-1b9f-1308-154ba9e467a3 (at 10.210.12.7@tcp1), client will retry: rc = -110
[13888673.334405] Lustre: Skipped 10 previous similar messages
[13888729.655443] LustreError: 162694:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(81428)  req@ffff918f72b84850 x1715350857728768/t0(0) o4->1b588188-9255-1b9f-1308-154ba9e467a3@10.210.12.7@tcp1:745/0 lens 488/448 e 0 to 0 dl 1645114790 ref 1 fl Interpret:/0/0 rc 0/0
[13888729.680550] LustreError: 162694:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 6 previous similar messages
[13888751.855420] Lustre: oak-OST0123: Client 1b588188-9255-1b9f-1308-154ba9e467a3 (at 10.210.12.7@tcp1) reconnecting
[13888751.865743] Lustre: Skipped 1 previous similar message
[13888839.951178] Lustre: oak-OST013d: Client 1b588188-9255-1b9f-1308-154ba9e467a3 (at 10.210.12.7@tcp1) reconnecting
[13888859.586545] Lustre: oak-OST0113: Client 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1) reconnecting
[13888859.596949] Lustre: Skipped 41 previous similar messages
[13889095.656491] Lustre: oak-OST014b: Connection restored to  (at 10.50.3.46@o2ib2)
[13889095.664203] Lustre: Skipped 671 previous similar messages
[13889698.032146] Lustre: oak-OST0115: Connection restored to 2dc703cd-5cca-815b-6dd4-cd3e7d5a0392 (at 10.0.3.49@o2ib5)
[13889698.042729] Lustre: Skipped 919 previous similar messages
[13890261.306912] Lustre: oak-OST0111: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[13890261.317325] Lustre: Skipped 4 previous similar messages
[13890261.325502] LustreError: 162714:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91548a261850 x1714979875382336/t0(0) o4->d4faaef8-44e2-b05f-6116-d9c30b2e4e11@10.210.12.60@tcp1:73/0 lens 488/448 e 0 to 0 dl 1645116383 ref 1 fl Interpret:/0/0 rc 0/0
[13890261.328615] Lustre: oak-OST0111: Bulk IO write error with d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1), client will retry: rc = -110
[13890261.328616] Lustre: Skipped 2 previous similar messages
[13890261.368935] LustreError: 162714:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 2 previous similar messages
[13890261.881152] Lustre: oak-OST013f: Bulk IO read error with dc851ef6-3898-c01c-adc6-dc548a499d03 (at 10.210.12.53@tcp1), client will retry: rc -110
[13890261.942989] LustreError: 21598:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9158a8fde050 x1715061084652032/t0(0) o4->dc851ef6-3898-c01c-adc6-dc548a499d03@10.210.12.53@tcp1:72/0 lens 488/448 e 0 to 0 dl 1645116382 ref 1 fl Interpret:/0/0 rc 0/0
[13890261.967241] LustreError: 21598:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[13890263.121130] LustreError: 21615:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9191a7b11850 x1715045729359680/t0(0) o4->80374d70-7d99-f68a-b58b-86c0b7514d6a@10.210.12.65@tcp1:73/0 lens 488/448 e 0 to 0 dl 1645116383 ref 1 fl Interpret:/0/0 rc 0/0
[13890263.145390] LustreError: 21615:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 31 previous similar messages
[13890296.605261] Lustre: oak-OST012d: Connection restored to fc6538a6-64f5-1c92-d38a-4c03c9b82dd0 (at 10.210.12.123@tcp1)
[13890296.616297] Lustre: Skipped 1283 previous similar messages
[13890309.170672] LustreError: 199273:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(20480)  req@ffff91b0c1f8d850 x1721913838698944/t0(0) o3->e44e675f-11ba-536d-fbc7-173bafd860e2@10.210.12.46@tcp1:71/0 lens 488/440 e 0 to 0 dl 1645116381 ref 1 fl Interpret:/0/0 rc 0/0
[13890309.170691] Lustre: oak-OST0147: Bulk IO read error with e44e675f-11ba-536d-fbc7-173bafd860e2 (at 10.210.12.46@tcp1), client will retry: rc -110
[13890309.208608] LustreError: 199273:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 10 previous similar messages
[13890428.603119] Lustre: oak-OST012f: Client 1b588188-9255-1b9f-1308-154ba9e467a3 (at 10.210.12.7@tcp1) reconnecting
[13890428.613448] Lustre: Skipped 15 previous similar messages
[13890430.460624] LustreError: 127350:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff91ef27ca8050 x1715045734296576/t0(0) o3->80374d70-7d99-f68a-b58b-86c0b7514d6a@10.210.12.65@tcp1:245/0 lens 488/440 e 0 to 0 dl 1645116555 ref 1 fl Interpret:/0/0 rc 0/0
[13890430.485144] LustreError: 127350:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 15 previous similar messages
[13890430.495056] Lustre: oak-OST013f: Bulk IO read error with 80374d70-7d99-f68a-b58b-86c0b7514d6a (at 10.210.12.65@tcp1), client will retry: rc -110
[13890430.508229] Lustre: Skipped 10 previous similar messages
[13890434.600231] Lustre: oak-OST0127: Client 1b588188-9255-1b9f-1308-154ba9e467a3 (at 10.210.12.7@tcp1) reconnecting
[13890434.611107] Lustre: Skipped 12 previous similar messages
[13890447.151031] Lustre: oak-OST0145: Client 80374d70-7d99-f68a-b58b-86c0b7514d6a (at 10.210.12.65@tcp1) reconnecting
[13890447.161936] Lustre: Skipped 11 previous similar messages
[13890458.532258] LustreError: 243456:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91d90b3dc050 x1714973075332864/t0(0) o4->5d4d5527-a52f-bc68-278b-aa990b8609d6@10.210.12.59@tcp1:272/0 lens 488/448 e 0 to 0 dl 1645116582 ref 1 fl Interpret:/0/0 rc 0/0
[13890458.556683] LustreError: 243456:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 2 previous similar messages
[13890458.589015] Lustre: oak-OST0129: Bulk IO write error with 5d4d5527-a52f-bc68-278b-aa990b8609d6 (at 10.210.12.59@tcp1), client will retry: rc = -110
[13890458.602466] Lustre: Skipped 49 previous similar messages
[13890459.657001] LustreError: 26436:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff919ee50e2800
[13890459.668018] LustreError: 26433:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff919ee50e2800
[13890459.679032] LustreError: 26434:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff919ee50e2800
[13890459.690046] LustreError: 26435:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff919ee50e2800
[13890460.039523] LustreError: 26436:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91ebe6345000
[13890460.050558] LustreError: 26436:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91ebe6345000
[13890460.061579] LustreError: 26436:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91ebe6345000
[13890460.072596] LustreError: 26433:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91ebe6345000
[13890464.260309] Lustre: oak-OST011b: Bulk IO read error with dc851ef6-3898-c01c-adc6-dc548a499d03 (at 10.210.12.53@tcp1), client will retry: rc -110
[13890464.273510] Lustre: Skipped 2 previous similar messages
[13890466.425164] Lustre: oak-OST0113: Client 5d4d5527-a52f-bc68-278b-aa990b8609d6 (at 10.210.12.59@tcp1) reconnecting
[13890466.435573] Lustre: Skipped 51 previous similar messages
[13890515.773562] Lustre: oak-OST013b: Client e44e675f-11ba-536d-fbc7-173bafd860e2 (at 10.210.12.46@tcp1) reconnecting
[13890515.783971] Lustre: Skipped 2 previous similar messages
[13890524.747459] LustreError: 162717:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 3145728(4194304)  req@ffff91728c608050 x1721913843960512/t0(0) o3->e44e675f-11ba-536d-fbc7-173bafd860e2@10.210.12.46@tcp1:273/0 lens 488/440 e 0 to 0 dl 1645116583 ref 1 fl Interpret:/0/0 rc 0/0
[13890524.747532] Lustre: oak-OST011f: Bulk IO read error with dc851ef6-3898-c01c-adc6-dc548a499d03 (at 10.210.12.53@tcp1), client will retry: rc -110
[13890524.747533] Lustre: Skipped 1 previous similar message
[13890524.791814] LustreError: 162717:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 10 previous similar messages
[13890537.488441] LustreError: 137-5: oak-OST0130_UUID: not available for connect from 10.210.12.46@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13890633.192725] Lustre: oak-OST011d: Client 479db7cf-e752-7ee3-6b36-d1f694b751cf (at 10.210.12.47@tcp1) reconnecting
[13890634.151457] LustreError: 243544:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff91456593d050 x1724768204955968/t0(0) o3->479db7cf-e752-7ee3-6b36-d1f694b751cf@10.210.12.47@tcp1:448/0 lens 488/440 e 0 to 0 dl 1645116758 ref 1 fl Interpret:/0/0 rc 0/0
[13890634.175988] LustreError: 243544:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 70 previous similar messages
[13890634.185818] Lustre: oak-OST011d: Bulk IO read error with 479db7cf-e752-7ee3-6b36-d1f694b751cf (at 10.210.12.47@tcp1), client will retry: rc -110
[13890634.199002] Lustre: Skipped 10 previous similar messages
[13890784.608616] Lustre: oak-OST0123: Client e44e675f-11ba-536d-fbc7-173bafd860e2 (at 10.210.12.46@tcp1) reconnecting
[13890784.619049] Lustre: Skipped 46 previous similar messages
[13890802.932187] LustreError: 162714:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff917731fae850 x1715012999516032/t0(0) o4->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:615/0 lens 488/448 e 0 to 0 dl 1645116925 ref 1 fl Interpret:/0/0 rc 0/0
[13890802.956803] Lustre: oak-OST0129: Bulk IO write error with c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1), client will retry: rc = -110
[13890802.970241] Lustre: Skipped 68 previous similar messages
[13890806.675399] LustreError: 160952:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff91dea7715050 x1714907394766400/t0(0) o4->f2914e3c-b512-2e60-27c3-dea532333a2b@10.210.12.135@tcp1:623/0 lens 488/448 e 0 to 0 dl 1645116933 ref 1 fl Interpret:/2/0 rc 0/0
[13890814.928020] Lustre: oak-OST0133: Bulk IO read error with d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1), client will retry: rc -110
[13890860.109219] LustreError: 243538:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(24576)  req@ffff916d7f19a850 x1715012999530368/t0(0) o3->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:615/0 lens 488/440 e 0 to 0 dl 1645116925 ref 1 fl Interpret:/0/0 rc 0/0
[13890860.109398] Lustre: oak-OST0133: Bulk IO read error with 6d01c866-9ec9-8f84-4f86-ee1db83afc97 (at 10.210.12.61@tcp1), client will retry: rc -110
[13890860.147252] LustreError: 243538:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 6 previous similar messages
[13890861.647466] LustreError: 160939:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff919984b9e050 x1715242116199360/t0(0) o4->e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda@10.210.12.79@tcp1:672/0 lens 488/448 e 0 to 0 dl 1645116982 ref 1 fl Interpret:/0/0 rc 0/0
[13890861.671951] LustreError: 160939:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 82 previous similar messages
[13890861.681953] Lustre: oak-OST013b: Bulk IO write error with e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1), client will retry: rc = -110
[13890861.695482] Lustre: Skipped 84 previous similar messages
[13890884.065903] LustreError: 160931:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff91c6bd089050 x1721913858373824/t0(0) o3->e44e675f-11ba-536d-fbc7-173bafd860e2@10.210.12.46@tcp1:628/0 lens 488/440 e 0 to 0 dl 1645116938 ref 1 fl Interpret:/0/0 rc 0/0
[13890884.091493] LustreError: 160931:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 2 previous similar messages
[13890898.558115] Lustre: oak-OST0149: Connection restored to ddd6e198-f36f-14d9-41a2-19f9ebbc987c (at 10.51.6.2@o2ib3)
[13890898.568690] Lustre: Skipped 2099 previous similar messages
[13890931.974720] LustreError: 160948:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(24576)  req@ffff91e081c92850 x1715242116244608/t0(0) o3->e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda@10.210.12.79@tcp1:677/0 lens 488/440 e 0 to 0 dl 1645116987 ref 1 fl Interpret:/0/0 rc 0/0
[13890931.974733] Lustre: oak-OST0121: Bulk IO read error with e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1), client will retry: rc -110
[13890931.974734] Lustre: Skipped 10 previous similar messages
[13890932.018764] LustreError: 160948:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 6 previous similar messages
[13891085.920292] Lustre: oak-OST0145: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[13891085.930710] Lustre: Skipped 145 previous similar messages
[13891187.589999] LustreError: 253954:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff9197c5a28850 x1721913868031936/t0(0) o3->e44e675f-11ba-536d-fbc7-173bafd860e2@10.210.12.46@tcp1:250/0 lens 488/440 e 0 to 0 dl 1645117315 ref 1 fl Interpret:/0/0 rc 0/0
[13891187.615182] LustreError: 253954:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 14 previous similar messages
[13891187.625017] Lustre: oak-OST0121: Bulk IO read error with e44e675f-11ba-536d-fbc7-173bafd860e2 (at 10.210.12.46@tcp1), client will retry: rc -110
[13891187.639012] Lustre: Skipped 6 previous similar messages
[13891497.201355] Lustre: oak-OST0123: Connection restored to 40a64382-781d-f7f3-5d47-9762a30c7d73 (at 10.51.7.8@o2ib3)
[13891497.211847] Lustre: Skipped 1333 previous similar messages
[13891729.306691] Lustre: oak-OST0115: Client 6d01c866-9ec9-8f84-4f86-ee1db83afc97 (at 10.210.12.61@tcp1) reconnecting
[13891729.317109] Lustre: Skipped 52 previous similar messages
[13891760.276505] LustreError: 21584:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff918f2d63c050 x1715065190996800/t0(0) o4->bb55a01e-02a3-cbba-5b76-e478fd40967c@10.210.12.54@tcp1:67/0 lens 488/448 e 0 to 0 dl 1645117887 ref 1 fl Interpret:/0/0 rc 0/0
[13891760.303513] Lustre: oak-OST0131: Bulk IO write error with bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1), client will retry: rc = -110
[13891760.316938] Lustre: Skipped 13 previous similar messages
[13891892.084283] Lustre: oak-OST0143: Bulk IO write error with 5d4d5527-a52f-bc68-278b-aa990b8609d6 (at 10.210.12.59@tcp1), client will retry: rc = -110
[13891938.002656] LustreError: 243537:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(4194304)  req@ffff9191e7304050 x1715061145987136/t0(0) o3->dc851ef6-3898-c01c-adc6-dc548a499d03@10.210.12.53@tcp1:197/0 lens 488/440 e 0 to 0 dl 1645118017 ref 1 fl Interpret:/0/0 rc 0/0
[13891938.002741] Lustre: oak-OST011b: Bulk IO read error with dc851ef6-3898-c01c-adc6-dc548a499d03 (at 10.210.12.53@tcp1), client will retry: rc -110
[13891938.040862] LustreError: 243537:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 3 previous similar messages
[13891954.777782] Lustre: 206886:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1645117858/real 1645117858]  req@ffff917feff41b00 x1710530798965248/t0(0) o106->oak-OST014d@10.51.7.8@o2ib3:15/16 lens 296/280 e 0 to 1 dl 1645117986 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
[13892013.918186] Lustre: oak-OST0149: Bulk IO write error with c4b979fd-3b98-af30-5ea5-32f00f4d8750 (at 10.210.12.64@tcp1), client will retry: rc = -110
[13892013.931623] Lustre: Skipped 45 previous similar messages
[13892020.846206] Lustre: oak-OST0139: haven't heard from client 40a64382-781d-f7f3-5d47-9762a30c7d73 (at 10.51.7.8@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91c876f3c000, cur 1645118053 expire 1645117903 last 1645117826
[13892029.833955] Lustre: oak-OST0135: haven't heard from client 40a64382-781d-f7f3-5d47-9762a30c7d73 (at 10.51.7.8@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91d21cb4ac00, cur 1645118062 expire 1645117912 last 1645117835
[13892029.855789] Lustre: Skipped 12 previous similar messages
[13892053.278575] Lustre: oak-OST011b: Bulk IO write error with bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1), client will retry: rc = -110
[13892053.292012] Lustre: Skipped 36 previous similar messages
[13892056.778904] LustreError: 243497:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(24576)  req@ffff916b3ea26850 x1716211209803776/t0(0) o3->c4b979fd-3b98-af30-5ea5-32f00f4d8750@10.210.12.64@tcp1:317/0 lens 488/440 e 0 to 0 dl 1645118137 ref 1 fl Interpret:/0/0 rc 0/0
[13892056.803903] LustreError: 243497:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 2 previous similar messages
[13892080.725076] LustreError: 243499:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(24576)  req@ffff91919c981850 x1715085447455360/t0(0) o3->65e9b8a8-b461-24e0-da12-5b6939f0b07c@10.210.12.56@tcp1:320/0 lens 488/440 e 0 to 0 dl 1645118140 ref 1 fl Interpret:/0/0 rc 0/0
[13892080.725778] LustreError: 162669:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91381860b050 x1715242146978880/t0(0) o4->e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda@10.210.12.79@tcp1:325/0 lens 488/448 e 0 to 0 dl 1645118145 ref 1 fl Interpret:/0/0 rc 0/0
[13892080.778419] LustreError: 243499:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 11 previous similar messages
[13892091.771844] LustreError: 160896:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff91efa944a850 x1724768281493120/t0(0) o3->479db7cf-e752-7ee3-6b36-d1f694b751cf@10.210.12.47@tcp1:399/0 lens 488/440 e 0 to 0 dl 1645118219 ref 1 fl Interpret:/0/0 rc 0/0
[13892091.796274] LustreError: 160896:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 88 previous similar messages
[13892096.133327] Lustre: oak-OST0125: Connection restored to 7aa982dc-eea3-6a5b-f096-3007f863c95c (at 10.210.12.72@tcp1)
[13892096.144041] Lustre: Skipped 1961 previous similar messages
[13892098.244733] LustreError: 137-5: oak-OST0120_UUID: not available for connect from 10.210.12.79@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13892098.262337] LustreError: Skipped 1 previous similar message
[13892332.129924] Lustre: oak-OST0127: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[13892332.140356] Lustre: Skipped 97 previous similar messages
[13892402.501579] Lustre: oak-OST0145: Bulk IO write error with e01e4c82-9fd9-5cc5-61dd-b09c7945df2b (at 10.210.12.11@tcp1), client will retry: rc = -110
[13892402.515029] Lustre: Skipped 6 previous similar messages
[13892463.989251] LustreError: 162692:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 3145728(4194304)  req@ffff91820a855050 x1714952069344256/t0(0) o3->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:708/0 lens 488/440 e 0 to 0 dl 1645118528 ref 1 fl Interpret:/0/0 rc 0/0
[13892463.989322] Lustre: oak-OST0127: Bulk IO read error with 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1), client will retry: rc -110
[13892463.989323] Lustre: Skipped 22 previous similar messages
[13892464.033532] LustreError: 162692:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 8 previous similar messages
[13892577.828427] Lustre: oak-OST0113: Bulk IO write error with 6ca637d5-9e95-16bf-f08c-446745633d32 (at 10.210.12.129@tcp1), client will retry: rc = -110
[13892577.841982] Lustre: Skipped 40 previous similar messages
[13892695.458494] Lustre: oak-OST0149: Connection restored to b9694c0e-9fc2-f14e-33c8-9018d9fc9806 (at 10.210.12.65@tcp1)
[13892695.469276] Lustre: Skipped 1197 previous similar messages
[13892758.374691] LustreError: 21594:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff916611446850 x1714952077485120/t0(0) o3->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:314/0 lens 488/440 e 0 to 0 dl 1645118889 ref 1 fl Interpret:/0/0 rc 0/0
[13892758.398948] LustreError: 21594:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 45 previous similar messages
[13893031.623223] Lustre: oak-OST0121: Client 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1) reconnecting
[13893031.633688] Lustre: Skipped 144 previous similar messages
[13893294.083364] Lustre: oak-OST0125: Connection restored to 7bd5e25d-92e5-9f8c-22fa-8f4c98d09593 (at 10.50.10.6@o2ib2)
[13893294.093974] Lustre: Skipped 1757 previous similar messages
[13893547.663418] LustreError: 160930:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91cfe99e4850 x1714911702695488/t0(0) o4->e5a735ca-191d-07c9-fcef-626c0b3a28a8@10.210.12.113@tcp1:345/0 lens 488/448 e 0 to 0 dl 1645119675 ref 1 fl Interpret:/0/0 rc 0/0
[13893547.688102] Lustre: oak-OST013d: Bulk IO write error with e5a735ca-191d-07c9-fcef-626c0b3a28a8 (at 10.210.12.113@tcp1), client will retry: rc = -110
[13893549.364660] LustreError: 244101:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff91d8b3772850 x1714915132522560/t0(0) o4->5865c071-3198-0848-a51e-ecc3ec62e180@10.210.12.127@tcp1:352/0 lens 488/448 e 0 to 0 dl 1645119682 ref 1 fl Interpret:/2/0 rc 0/0
[13893549.389491] LustreError: 244101:0:(ldlm_lib.c:3305:target_bulk_io()) Skipped 2 previous similar messages
[13893589.861877] LustreError: 229137:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 3145728(4194304)  req@ffff9182ccad4850 x1724768349538496/t0(0) o3->479db7cf-e752-7ee3-6b36-d1f694b751cf@10.210.12.47@tcp1:344/0 lens 488/440 e 0 to 0 dl 1645119674 ref 1 fl Interpret:/0/0 rc 0/0
[13893589.861961] Lustre: oak-OST0119: Bulk IO read error with 479db7cf-e752-7ee3-6b36-d1f694b751cf (at 10.210.12.47@tcp1), client will retry: rc -110
[13893589.861962] Lustre: Skipped 9 previous similar messages
[13893589.907903] LustreError: 229137:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 4 previous similar messages
[13893613.804698] LustreError: 243536:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(24576)  req@ffff9191f0f07850 x1715242184130048/t0(0) o3->e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda@10.210.12.79@tcp1:344/0 lens 488/440 e 0 to 0 dl 1645119674 ref 1 fl Interpret:/0/0 rc 0/0
[13893613.829584] LustreError: 243536:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 18 previous similar messages
[13893644.372391] LustreError: 199244:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 99s: evicting client at 10.210.12.74@tcp1  ns: filter-oak-OST014b_UUID lock: ffff9180fb61f500/0xed112d3020d0a00d lrc: 4/0,0 mode: PW/PW res: [0x5680000401:0x361d03:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x60000400030020 nid: 10.210.12.74@tcp1 remote: 0x1a73bb524c290656 expref: 195 pid: 199259 timeout: 13927413 lvb_type: 0
[13893713.152977] Lustre: oak-OST0119: Client 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1) reconnecting
[13893713.163386] Lustre: Skipped 27 previous similar messages
[13893893.017490] Lustre: oak-OST0119: Connection restored to  (at 10.50.6.47@o2ib2)
[13893893.024976] Lustre: Skipped 2056 previous similar messages
[13894067.929501] LustreError: 253956:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 3145728(4194304)  req@ffff919efb944850 x1715001478562496/t0(0) o3->6f45695d-f173-ee1d-e6cb-d38dad7e0879@10.210.12.74@tcp1:46/0 lens 1016/440 e 0 to 0 dl 1645120131 ref 1 fl Interpret:/0/0 rc 0/0
[13894080.501116] LustreError: 137-5: oak-OST0140_UUID: not available for connect from 10.210.12.74@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13894347.156921] Lustre: oak-OST0129: Client 6f45695d-f173-ee1d-e6cb-d38dad7e0879 (at 10.210.12.74@tcp1) reconnecting
[13894347.167580] Lustre: Skipped 101 previous similar messages
[13894491.639932] Lustre: oak-OST0113: Connection restored to  (at 10.51.14.15@o2ib3)
[13894491.647573] Lustre: Skipped 1482 previous similar messages
[13894583.223605] LustreError: 160942:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91f0ae645050 x1715024713561984/t0(0) o4->6d01c866-9ec9-8f84-4f86-ee1db83afc97@10.210.12.61@tcp1:629/0 lens 488/448 e 0 to 0 dl 1645120714 ref 1 fl Interpret:/0/0 rc 0/0
[13894583.249065] LustreError: 160942:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 38 previous similar messages
[13894583.258850] Lustre: oak-OST0113: Bulk IO write error with 6d01c866-9ec9-8f84-4f86-ee1db83afc97 (at 10.210.12.61@tcp1), client will retry: rc = -110
[13894583.258851] Lustre: Skipped 36 previous similar messages
[13894584.072889] LustreError: 160911:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff91c18e3e8850 x1715024713555904/t0(0) o4->6d01c866-9ec9-8f84-4f86-ee1db83afc97@10.210.12.61@tcp1:634/0 lens 488/448 e 0 to 0 dl 1645120719 ref 1 fl Interpret:/2/0 rc 0/0
[13894642.838725] LustreError: 21596:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 3145728(4194304)  req@ffff9185a7aed050 x1715024713524096/t0(0) o3->6d01c866-9ec9-8f84-4f86-ee1db83afc97@10.210.12.61@tcp1:628/0 lens 488/440 e 0 to 0 dl 1645120713 ref 1 fl Interpret:/0/0 rc 0/0
[13894642.838818] Lustre: oak-OST0149: Bulk IO read error with 6d01c866-9ec9-8f84-4f86-ee1db83afc97 (at 10.210.12.61@tcp1), client will retry: rc -110
[13894642.838820] Lustre: Skipped 27 previous similar messages
[13894642.882973] LustreError: 21596:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 5 previous similar messages
[13895004.233781] Lustre: oak-OST012b: Client 6f45695d-f173-ee1d-e6cb-d38dad7e0879 (at 10.210.12.74@tcp1) reconnecting
[13895004.244385] Lustre: Skipped 16 previous similar messages
[13895091.032319] Lustre: oak-OST0147: Connection restored to  (at 10.50.6.11@o2ib2)
[13895091.039783] Lustre: Skipped 1529 previous similar messages
[13895098.016690] Lustre: oak-OST012b: Bulk IO write error with e01e4c82-9fd9-5cc5-61dd-b09c7945df2b (at 10.210.12.11@tcp1), client will retry: rc = -110
[13895098.030126] Lustre: Skipped 9 previous similar messages
[13895145.910140] LustreError: 160923:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 3145728(4194304)  req@ffff9195f7387050 x1721913965092992/t0(0) o3->e44e675f-11ba-536d-fbc7-173bafd860e2@10.210.12.46@tcp1:390/0 lens 1016/440 e 0 to 0 dl 1645121230 ref 1 fl Interpret:/0/0 rc 0/0
[13895145.935780] LustreError: 160923:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 2 previous similar messages
[13895277.529440] LustreError: 160901:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk READ failed: rc -107  req@ffff91be9b074850 x1714974583720832/t0(0) o3->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:575/0 lens 488/440 e 0 to 0 dl 1645121415 ref 1 fl Interpret:/0/0 rc 0/0
[13895277.554513] Lustre: oak-OST0135: Bulk IO read error with 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1), client will retry: rc -107
[13895277.567713] Lustre: Skipped 11 previous similar messages
[13895581.624155] LustreError: 228403:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff914ad1098050 x1715013308992000/t0(0) o4->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:120/0 lens 488/448 e 0 to 0 dl 1645121715 ref 1 fl Interpret:/0/0 rc 0/0
[13895581.648596] LustreError: 228403:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 25 previous similar messages
[13895581.658666] Lustre: oak-OST012f: Bulk IO write error with c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1), client will retry: rc = -110
[13895581.672101] Lustre: Skipped 17 previous similar messages
[13895648.956773] LustreError: 243535:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff916cc622e050 x1715093102228352/t0(0) o3->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:121/0 lens 488/440 e 0 to 0 dl 1645121716 ref 1 fl Interpret:/0/0 rc 0/0
[13895689.615152] Lustre: oak-OST013d: Connection restored to 4025d53e-c01c-f5a6-a7de-c3a1c13db1f4 (at 10.50.10.5@o2ib2)
[13895689.625771] Lustre: Skipped 995 previous similar messages
[13895743.765548] Lustre: oak-OST0143: Client e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1) reconnecting
[13895743.775975] Lustre: Skipped 81 previous similar messages
[13895744.171240] Lustre: oak-OST011d: Bulk IO write error with 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1), client will retry: rc = -110
[13895744.184680] Lustre: Skipped 3 previous similar messages
[13895791.689652] LustreError: 162704:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 3145728(4194304)  req@ffff9165bd831050 x1715085600308544/t0(0) o3->65e9b8a8-b461-24e0-da12-5b6939f0b07c@10.210.12.56@tcp1:283/0 lens 488/440 e 0 to 0 dl 1645121878 ref 1 fl Interpret:/0/0 rc 0/0
[13895791.696456] LustreError: 160891:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 1523712(2572288)  req@ffff91a22936a850 x1715093104512832/t0(0) o4->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:283/0 lens 504/448 e 0 to 0 dl 1645121878 ref 1 fl Interpret:/0/0 rc 0/0
[13895791.696458] LustreError: 160891:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[13895791.696584] Lustre: oak-OST0141: Bulk IO write error with 5d8bb8d9-e5c7-334f-ec8d-f10452c309b5 (at 10.210.12.66@tcp1), client will retry: rc = -110
[13895791.696584] Lustre: Skipped 7 previous similar messages
[13895974.510202] Lustre: oak-OST0143: Bulk IO write error with e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1), client will retry: rc = -110
[13896137.990176] Lustre: oak-OST0141: Bulk IO write error with e44e675f-11ba-536d-fbc7-173bafd860e2 (at 10.210.12.46@tcp1), client will retry: rc = -110
[13896138.003652] Lustre: Skipped 19 previous similar messages
[13896150.982670] LustreError: 160907:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91d05a47c850 x1715024738790784/t0(0) o4->6d01c866-9ec9-8f84-4f86-ee1db83afc97@10.210.12.61@tcp1:640/0 lens 488/448 e 0 to 0 dl 1645122235 ref 1 fl Interpret:/0/0 rc 0/0
[13896150.982675] LustreError: 160914:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff91bd25ad4050 x1724768423022848/t0(0) o3->479db7cf-e752-7ee3-6b36-d1f694b751cf@10.210.12.47@tcp1:640/0 lens 488/440 e 0 to 0 dl 1645122235 ref 1 fl Interpret:/0/0 rc 0/0
[13896150.982688] Lustre: oak-OST012b: Bulk IO read error with 479db7cf-e752-7ee3-6b36-d1f694b751cf (at 10.210.12.47@tcp1), client will retry: rc -110
[13896150.982689] Lustre: Skipped 3 previous similar messages
[13896151.052318] LustreError: 160907:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 3 previous similar messages
[13896178.871312] LustreError: 137-5: oak-OST0144_UUID: not available for connect from 10.210.12.31@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13896180.038012] LustreError: 137-5: oak-OST012e_UUID: not available for connect from 10.210.12.56@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13896180.055644] LustreError: Skipped 2 previous similar messages
[13896198.888871] LustreError: 253955:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91f18dcde850 x1715013325540096/t0(0) o4->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:678/0 lens 488/448 e 0 to 0 dl 1645122273 ref 1 fl Interpret:/0/0 rc 0/0
[13896198.914673] LustreError: 253955:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 4 previous similar messages
[13896216.809495] LustreError: 137-5: oak-OST012c_UUID: not available for connect from 10.210.12.25@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13896216.827092] LustreError: Skipped 2 previous similar messages
[13896284.804605] LustreError: 162700:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff916e2cbf1050 x1715093118941888/t0(0) o4->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:72/0 lens 488/448 e 0 to 0 dl 1645122422 ref 1 fl Interpret:/0/0 rc 0/0
[13896284.828966] LustreError: 162700:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 35 previous similar messages
[13896288.595244] Lustre: oak-OST013f: Connection restored to b9694c0e-9fc2-f14e-33c8-9018d9fc9806 (at 10.210.12.65@tcp1)
[13896288.605918] Lustre: Skipped 1385 previous similar messages
[13896334.084254] LustreError: 21613:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk READ failed: rc -107  req@ffff915ae7231850 x1715065424657472/t0(0) o3->bb55a01e-02a3-cbba-5b76-e478fd40967c@10.210.12.54@tcp1:124/0 lens 488/440 e 0 to 0 dl 1645122474 ref 1 fl Interpret:/0/0 rc 0/0
[13896334.108778] LustreError: 21613:0:(ldlm_lib.c:3305:target_bulk_io()) Skipped 3 previous similar messages
[13896355.710956] Lustre: oak-OST0111: Client 3b1423ea-f77c-9301-8ee2-822b91cad2d9 (at 10.210.12.10@tcp1) reconnecting
[13896355.721426] Lustre: Skipped 351 previous similar messages
[13896390.524300] LustreError: 21587:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff914ad9f26850 x1715093119940736/t0(0) o3->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:115/0 lens 488/440 e 0 to 0 dl 1645122465 ref 1 fl Interpret:/0/0 rc 0/0
[13896390.534844] LustreError: 253953:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91e8f0653050 x1715093119980416/t0(0) o4->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:119/0 lens 488/448 e 0 to 0 dl 1645122469 ref 1 fl Interpret:/0/0 rc 0/0
[13896390.576051] LustreError: 21587:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 16 previous similar messages
[13896410.248563] LustreError: 137-5: oak-OST012a_UUID: not available for connect from 10.210.12.54@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13896450.042325] LustreError: 137-5: oak-OST0120_UUID: not available for connect from 10.210.12.79@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13896450.059925] LustreError: Skipped 3 previous similar messages
[13896563.926149] LustreError: 253956:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk READ failed: rc -107  req@ffff919bf5e29850 x1714974667621760/t0(0) o3->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:355/0 lens 488/440 e 0 to 0 dl 1645122705 ref 1 fl Interpret:/0/0 rc 0/0
[13896563.951099] LustreError: 253956:0:(ldlm_lib.c:3305:target_bulk_io()) Skipped 3 previous similar messages
[13896564.656312] Lustre: oak-OST0127: Bulk IO write error with 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1), client will retry: rc = -110
[13896564.670983] Lustre: Skipped 24 previous similar messages
[13896586.748873] Lustre: oak-OST0139: haven't heard from client 48ce2b21-1791-42f3-0c14-26eb4ce59fda (at 10.51.12.2@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91d8d9130800, cur 1645122630 expire 1645122480 last 1645122403
[13896654.033992] LustreError: 228423:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 1048576(4194304)  req@ffff914ea0e43850 x1714952185562880/t0(0) o3->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:374/0 lens 488/440 e 0 to 0 dl 1645122724 ref 1 fl Interpret:/0/0 rc 0/0
[13896654.034011] LustreError: 199271:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91832ca86050 x1721914002194880/t0(0) o4->e44e675f-11ba-536d-fbc7-173bafd860e2@10.210.12.46@tcp1:375/0 lens 488/448 e 0 to 0 dl 1645122725 ref 1 fl Interpret:/0/0 rc 0/0
[13896654.034013] LustreError: 199271:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[13896654.095603] LustreError: 228423:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 16 previous similar messages
[13896662.567211] Lustre: oak-OST014b: haven't heard from client e26085b3-2577-1890-b02f-ddc87be9255b (at 10.51.16.2@o2ib3) in 222 seconds. I think it's dead, and I am evicting it. exp ffff91bc5304ec00, cur 1645122706 expire 1645122556 last 1645122484
[13896662.589119] Lustre: Skipped 202 previous similar messages
[13896664.579973] Lustre: oak-OST0149: haven't heard from client a4ef6053-5de8-7d8b-4226-c263b9ab791f (at 10.50.15.7@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91c029202c00, cur 1645122708 expire 1645122558 last 1645122481
[13896664.601953] Lustre: Skipped 2 previous similar messages
[13896667.565346] Lustre: oak-OST0119: haven't heard from client e26085b3-2577-1890-b02f-ddc87be9255b (at 10.51.16.2@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91f072d95c00, cur 1645122711 expire 1645122561 last 1645122484
[13896667.587289] Lustre: Skipped 1 previous similar message
[13896667.647807] LustreError: 243351:0:(client.c:1210:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff91e36a981f80 x1710530826184448/t0(0) o106->oak-OST014d@10.51.16.2@o2ib3:15/16 lens 296/280 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1
[13896667.999777] LustreError: 137-5: oak-OST0148_UUID: not available for connect from 10.210.12.42@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13896756.477578] Lustre: oak-OST013d: Bulk IO read error with 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1), client will retry: rc -110
[13896756.490792] Lustre: Skipped 52 previous similar messages
[13896889.376593] Lustre: oak-OST0127: Connection restored to aef304b4-8679-e0d5-9fa5-c78542ec1535 (at 10.50.2.25@o2ib2)
[13896889.387201] Lustre: Skipped 1709 previous similar messages
[13896965.539589] Lustre: oak-OST0141: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[13896965.550004] Lustre: Skipped 501 previous similar messages
[13897033.664197] Lustre: oak-OST0125: haven't heard from client f8506977-bc95-9fbf-2758-79bfd3006844 (at 10.51.14.1@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91b61375bc00, cur 1645123078 expire 1645122928 last 1645122851
[13897033.686111] Lustre: Skipped 6 previous similar messages
[13897402.102824] LustreError: 21587:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9146e1af3850 x1724768526066560/t0(0) o4->479db7cf-e752-7ee3-6b36-d1f694b751cf@10.210.12.47@tcp1:435/0 lens 488/448 e 0 to 0 dl 1645123540 ref 1 fl Interpret:/0/0 rc 0/0
[13897402.127401] LustreError: 21587:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 40 previous similar messages
[13897402.137408] Lustre: oak-OST0135: Bulk IO write error with 479db7cf-e752-7ee3-6b36-d1f694b751cf (at 10.210.12.47@tcp1), client will retry: rc = -110
[13897402.151413] Lustre: Skipped 14 previous similar messages
[13897459.165322] Lustre: oak-OST014d: Bulk IO read error with c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1), client will retry: rc -110
[13897459.178895] Lustre: Skipped 4 previous similar messages
[13897468.430662] LustreError: 162672:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff9148799e1050 x1716244805353024/t0(0) o4->6b341d38-3674-15b6-7e5e-137d0b4498c0@10.210.12.40@tcp1:435/0 lens 488/448 e 0 to 0 dl 1645123540 ref 1 fl Interpret:/0/0 rc 0/0
[13897468.456485] LustreError: 162672:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 4 previous similar messages
[13897489.790491] Lustre: oak-OST0139: Connection restored to 0465062f-c53d-e09b-c11a-86f450e750ba (at 10.50.1.72@o2ib2)
[13897489.801177] Lustre: Skipped 1042 previous similar messages
[13897516.339123] LustreError: 160894:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91bcce4d7050 x1715093150810752/t0(0) o4->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:487/0 lens 488/448 e 0 to 0 dl 1645123592 ref 1 fl Interpret:/0/0 rc 0/0
[13897516.340317] LustreError: 160910:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(1048576)  req@ffff91b262352050 x1716211351091520/t0(0) o3->c4b979fd-3b98-af30-5ea5-32f00f4d8750@10.210.12.64@tcp1:493/0 lens 488/440 e 0 to 0 dl 1645123598 ref 1 fl Interpret:/0/0 rc 0/0
[13897580.568997] Lustre: oak-OST014b: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[13897580.579636] Lustre: Skipped 39 previous similar messages
[13897731.915293] LustreError: 243541:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff916514c15850 x1715093165816640/t0(0) o4->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:704/0 lens 488/448 e 0 to 0 dl 1645123809 ref 1 fl Interpret:/0/0 rc 0/0
[13897731.941094] LustreError: 243541:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 13 previous similar messages
[13897743.990968] LustreError: 137-5: oak-OST012a_UUID: not available for connect from 10.210.12.66@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13897744.008587] LustreError: Skipped 6 previous similar messages
[13897750.465316] LustreError: 137-5: oak-OST0134_UUID: not available for connect from 10.210.12.43@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13897858.687650] LustreError: 162694:0:(tgt_handler.c:651:process_req_last_xid()) @@@ Unexpected xid 617c2abf25480 vs. last_xid 617c2abf2573f
                    req@ffff9161dff35850 x1714974736143488/t0(0) o3->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:142/0 lens 488/0 e 0 to 0 dl 1645124002 ref 1 fl Interpret:/2/ffffffff rc 0/-1
[13897875.132612] LustreError: 160955:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk READ failed: rc -107  req@ffff91c54e3f1850 x1714970691660032/t0(0) o3->5afa23c2-1fd5-9030-38ce-b03a359082fe@10.210.12.75@tcp1:159/0 lens 488/440 e 0 to 0 dl 1645124019 ref 1 fl Interpret:/0/0 rc 0/0
[13897892.457641] LustreError: 162685:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk READ failed: rc -107  req@ffff915e42683050 x1714970693834752/t0(0) o3->5afa23c2-1fd5-9030-38ce-b03a359082fe@10.210.12.75@tcp1:176/0 lens 488/440 e 0 to 0 dl 1645124036 ref 1 fl Interpret:/0/0 rc 0/0
[13898006.520908] LustreError: 160943:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff919e80ec6050 x1715760370379264/t0(0) o3->4b39aac6-2dc7-e48f-9434-e3bfde492453@10.210.12.29@tcp1:290/0 lens 488/440 e 0 to 0 dl 1645124150 ref 1 fl Interpret:/0/0 rc 0/0
[13898006.545276] LustreError: 160943:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 41 previous similar messages
[13898088.943205] Lustre: oak-OST0129: Connection restored to 6c8f221e-4d8d-3853-7ab4-a3202bff4f37 (at 10.50.6.8@o2ib2)
[13898088.953738] Lustre: Skipped 2511 previous similar messages
[13898195.838666] Lustre: oak-OST014d: haven't heard from client f91cba6e-11a0-da92-4774-0869b9b9886f (at 10.51.14.15@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91b00feb8400, cur 1645124243 expire 1645124093 last 1645124016
[13898195.862883] Lustre: Skipped 30 previous similar messages
[13898207.146626] Lustre: oak-OST0135: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[13898207.157066] Lustre: Skipped 667 previous similar messages
[13898271.663421] Lustre: oak-OST0147: haven't heard from client 636494ee-6101-0308-f332-fcee8da41d2c (at 10.50.14.7@o2ib2) in 163 seconds. I think it's dead, and I am evicting it. exp ffff91aab55a1c00, cur 1645124319 expire 1645124169 last 1645124156
[13898271.685411] Lustre: Skipped 61 previous similar messages
[13898321.536649] Lustre: oak-OST013b: Bulk IO write error with e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1), client will retry: rc = -110
[13898321.550089] Lustre: Skipped 52 previous similar messages
[13898377.676510] LustreError: 199273:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff919b47091050 x1716244890427712/t0(0) o4->6b341d38-3674-15b6-7e5e-137d0b4498c0@10.210.12.40@tcp1:602/0 lens 488/448 e 0 to 0 dl 1645124462 ref 1 fl Interpret:/0/0 rc 0/0
[13898377.676975] LustreError: 127348:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff91f13773f850 x1714944939381952/t0(0) o3->5704685e-b0a9-b5c3-be3f-b278a4395bec@10.210.12.72@tcp1:604/0 lens 488/440 e 0 to 0 dl 1645124464 ref 1 fl Interpret:/0/0 rc 0/0
[13898377.676976] LustreError: 127348:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 16 previous similar messages
[13898377.676994] Lustre: oak-OST0133: Bulk IO read error with 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1), client will retry: rc -110
[13898377.676994] Lustre: Skipped 26 previous similar messages
[13898377.756861] LustreError: 199273:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[13898400.681171] LustreError: 137-5: oak-OST0134_UUID: not available for connect from 10.210.12.43@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13898400.698865] LustreError: Skipped 11 previous similar messages
[13898401.291160] LustreError: 137-5: oak-OST0142_UUID: not available for connect from 10.210.12.23@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13898401.308746] LustreError: Skipped 2 previous similar messages
[13898688.046792] Lustre: oak-OST0135: Connection restored to  (at 10.50.17.9@o2ib2)
[13898688.054364] Lustre: Skipped 1581 previous similar messages
[13898859.742146] Lustre: oak-OST0115: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[13898859.752583] Lustre: Skipped 192 previous similar messages
[13898950.262289] LustreError: 162696:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91909a5fb850 x1714944945445952/t0(0) o4->5704685e-b0a9-b5c3-be3f-b278a4395bec@10.210.12.72@tcp1:480/0 lens 488/448 e 0 to 0 dl 1645125095 ref 1 fl Interpret:/0/0 rc 0/0
[13898950.278471] Lustre: oak-OST012f: Bulk IO write error with 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1), client will retry: rc = -110
[13898950.278472] Lustre: Skipped 13 previous similar messages
[13898950.305815] LustreError: 162696:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 15 previous similar messages
[13899000.404431] LustreError: 162701:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff9169d3026850 x1715046016970176/t0(0) o4->80374d70-7d99-f68a-b58b-86c0b7514d6a@10.210.12.65@tcp1:474/0 lens 488/448 e 0 to 0 dl 1645125089 ref 1 fl Interpret:/0/0 rc 0/0
[13899000.430227] LustreError: 162701:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 3 previous similar messages
[13899124.300342] Lustre: oak-OST0147: Bulk IO read error with 5d8bb8d9-e5c7-334f-ec8d-f10452c309b5 (at 10.210.12.66@tcp1), client will retry: rc -110
[13899124.313519] Lustre: Skipped 5 previous similar messages
[13899192.008777] LustreError: 162679:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91622dbd8050 x1721914026674944/t0(0) o4->e44e675f-11ba-536d-fbc7-173bafd860e2@10.210.12.46@tcp1:670/0 lens 488/448 e 0 to 0 dl 1645125285 ref 1 fl Interpret:/0/0 rc 0/0
[13899220.208039] LustreError: 137-5: oak-OST0140_UUID: not available for connect from 10.210.12.42@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13899220.225741] LustreError: Skipped 3 previous similar messages
[13899286.897733] Lustre: oak-OST0133: Connection restored to 6da567f0-c49b-4d35-fa48-7222226d9e96 (at 10.50.10.46@o2ib2)
[13899286.908405] Lustre: Skipped 1112 previous similar messages
[13899287.806713] LustreError: 160948:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff91f09e3b7050 x1715013532087104/t0(0) o3->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:0/0 lens 488/440 e 0 to 0 dl 1645125370 ref 1 fl Interpret:/0/0 rc 0/0
[13899287.806920] LustreError: 160938:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91ba01606050 x1714980142706560/t0(0) o4->d4faaef8-44e2-b05f-6116-d9c30b2e4e11@10.210.12.60@tcp1:1/0 lens 488/448 e 0 to 0 dl 1645125371 ref 1 fl Interpret:/0/0 rc 0/0
[13899287.857710] LustreError: 160948:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 6 previous similar messages
[13899308.875994] LustreError: 137-5: oak-OST011a_UUID: not available for connect from 10.210.12.60@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13899475.638854] Lustre: oak-OST0123: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[13899475.649270] Lustre: Skipped 412 previous similar messages
[13899479.415331] LustreError: 21615:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(2850816)  req@ffff918a61c18850 x1714970738137920/t0(0) o4->5afa23c2-1fd5-9030-38ce-b03a359082fe@10.210.12.75@tcp1:202/0 lens 504/448 e 0 to 0 dl 1645125572 ref 1 fl Interpret:/0/0 rc 0/0
[13899479.441102] LustreError: 21615:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 3 previous similar messages
[13899508.587645] LustreError: 137-5: oak-OST0132_UUID: not available for connect from 10.210.12.65@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13899508.605311] LustreError: Skipped 1 previous similar message
[13899613.891533] LustreError: 160906:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91b262357050 x1716552340885824/t0(0) o4->6bab8d10-30ff-c1d2-bbf5-1fd0884ac647@10.210.12.57@tcp1:390/0 lens 488/448 e 0 to 0 dl 1645125760 ref 1 fl Interpret:/0/0 rc 0/0
[13899613.915948] LustreError: 160906:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 28 previous similar messages
[13899613.925933] Lustre: oak-OST0147: Bulk IO write error with 6bab8d10-30ff-c1d2-bbf5-1fd0884ac647 (at 10.210.12.57@tcp1), client will retry: rc = -110
[13899613.939364] Lustre: Skipped 36 previous similar messages
[13899662.786375] LustreError: 137-5: oak-OST0130_UUID: not available for connect from 10.210.12.65@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13899662.803961] LustreError: Skipped 2 previous similar messages
[13899886.257861] Lustre: oak-OST011b: Connection restored to 37c9099c-1a38-a699-2712-432f4cb01e44 (at 10.210.12.19@tcp1)
[13899886.268634] Lustre: Skipped 1222 previous similar messages
[13899939.792664] LustreError: 21597:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff915c90ba4850 x1714906824315840/t0(0) o4->a166a60d-30d5-1e72-b799-745de1d3b307@10.210.12.125@tcp1:718/0 lens 488/448 e 0 to 0 dl 1645126088 ref 1 fl Interpret:/2/0 rc 0/0
[13899984.235417] LustreError: 137-5: oak-OST012a_UUID: not available for connect from 10.210.12.43@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13899984.253058] LustreError: Skipped 1 previous similar message
[13900006.326936] LustreError: 162707:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91828a36c050 x1715024822859904/t0(0) o4->6d01c866-9ec9-8f84-4f86-ee1db83afc97@10.210.12.61@tcp1:714/0 lens 488/448 e 0 to 0 dl 1645126084 ref 1 fl Interpret:/0/0 rc 0/0
[13900006.326967] LustreError: 162691:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff915ba8928050 x1715024822862400/t0(0) o3->6d01c866-9ec9-8f84-4f86-ee1db83afc97@10.210.12.61@tcp1:714/0 lens 488/440 e 0 to 0 dl 1645126084 ref 1 fl Interpret:/0/0 rc 0/0
[13900006.326969] LustreError: 162691:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 10 previous similar messages
[13900006.327226] Lustre: oak-OST0113: Bulk IO read error with 6d01c866-9ec9-8f84-4f86-ee1db83afc97 (at 10.210.12.61@tcp1), client will retry: rc -110
[13900006.327228] Lustre: Skipped 17 previous similar messages
[13900076.960948] Lustre: oak-OST0141: Client 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1) reconnecting
[13900076.971378] Lustre: Skipped 179 previous similar messages
[13900214.070849] LustreError: 160921:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91a784f4d850 x1716245031967808/t0(0) o4->6b341d38-3674-15b6-7e5e-137d0b4498c0@10.210.12.40@tcp1:234/0 lens 488/448 e 0 to 0 dl 1645126359 ref 1 fl Interpret:/0/0 rc 0/0
[13900214.085950] Lustre: oak-OST0149: Bulk IO write error with 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1), client will retry: rc = -110
[13900214.085951] Lustre: Skipped 14 previous similar messages
[13900214.114545] LustreError: 160921:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 13 previous similar messages
[13900292.744863] LustreError: 21624:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91cd94522050 x1714944989750848/t0(0) o4->5704685e-b0a9-b5c3-be3f-b278a4395bec@10.210.12.72@tcp1:258/0 lens 488/448 e 0 to 0 dl 1645126383 ref 1 fl Interpret:/0/0 rc 0/0
[13900292.770634] LustreError: 21624:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 7 previous similar messages
[13900319.149880] LustreError: 137-5: oak-OST0128_UUID: not available for connect from 10.210.12.31@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13900436.396702] Lustre: oak-OST0119: haven't heard from client 6ca637d5-9e95-16bf-f08c-446745633d32 (at 10.210.12.129@tcp1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9148d3353400, cur 1645126489 expire 1645126339 last 1645126262
[13900436.418806] Lustre: Skipped 8 previous similar messages
[13900484.333710] LustreError: 160954:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 951176(1999752)  req@ffff91ef0d6bf850 x1716245031968064/t0(0) o4->6b341d38-3674-15b6-7e5e-137d0b4498c0@10.210.12.40@tcp1:451/0 lens 488/448 e 0 to 0 dl 1645126576 ref 1 fl Interpret:/2/0 rc 0/0
[13900484.359501] LustreError: 160954:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[13900485.355917] Lustre: oak-OST014d: Connection restored to 2d14d64f-d9c6-774d-30c4-3498295aae53 (at 10.50.9.34@o2ib2)
[13900485.366508] Lustre: Skipped 1230 previous similar messages
[13900580.122487] LustreError: 160921:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91a063042050 x1714952340940992/t0(0) o4->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:543/0 lens 488/448 e 0 to 0 dl 1645126668 ref 1 fl Interpret:/0/0 rc 0/0
[13900580.148300] LustreError: 160921:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 6 previous similar messages
[13900602.459197] LustreError: 137-5: oak-OST0128_UUID: not available for connect from 10.210.12.31@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13900602.476822] LustreError: Skipped 8 previous similar messages
[13900691.174805] Lustre: oak-OST0125: Client 6d01c866-9ec9-8f84-4f86-ee1db83afc97 (at 10.210.12.61@tcp1) reconnecting
[13900691.185253] Lustre: Skipped 561 previous similar messages
[13900706.902192] Lustre: oak-OST0149: Bulk IO read error with e44e675f-11ba-536d-fbc7-173bafd860e2 (at 10.210.12.46@tcp1), client will retry: rc -110
[13900706.915392] Lustre: Skipped 20 previous similar messages
[13900771.722913] LustreError: 127347:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91cd817d0850 x1716211443036224/t0(0) o4->c4b979fd-3b98-af30-5ea5-32f00f4d8750@10.210.12.64@tcp1:729/0 lens 488/448 e 0 to 0 dl 1645126854 ref 1 fl Interpret:/0/0 rc 0/0
[13900771.723589] LustreError: 160954:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(1048576)  req@ffff91a2a1ec0050 x1721914049300928/t0(0) o3->e44e675f-11ba-536d-fbc7-173bafd860e2@10.210.12.46@tcp1:734/0 lens 504/440 e 0 to 0 dl 1645126859 ref 1 fl Interpret:/0/0 rc 0/0
[13900771.723590] LustreError: 160954:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 18 previous similar messages
[13900771.783682] LustreError: 127347:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 5 previous similar messages
[13900789.393361] LustreError: 137-5: oak-OST0134_UUID: not available for connect from 10.210.12.46@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13900789.410956] LustreError: Skipped 8 previous similar messages
[13900883.302377] LustreError: 199271:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff915566374050 x1714952365100416/t0(0) o4->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:153/0 lens 488/448 e 0 to 0 dl 1645127033 ref 1 fl Interpret:/0/0 rc 0/0
[13900883.326836] LustreError: 199271:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 32 previous similar messages
[13900883.336730] Lustre: oak-OST011d: Bulk IO write error with 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1), client will retry: rc = -110
[13900883.350162] Lustre: Skipped 54 previous similar messages
[13901084.321314] Lustre: oak-OST013f: Connection restored to a6fe3701-42c7-46f0-b462-758ca9032454 (at 10.210.12.29@tcp1)
[13901084.331983] Lustre: Skipped 1225 previous similar messages
[13901130.956747] LustreError: 160902:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff919879760050 x1716552363609984/t0(0) o4->6bab8d10-30ff-c1d2-bbf5-1fd0884ac647@10.210.12.57@tcp1:351/0 lens 488/448 e 0 to 0 dl 1645127231 ref 1 fl Interpret:/0/0 rc 0/0
[13901130.983177] LustreError: 160902:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 3 previous similar messages
[13901164.066574] LustreError: 137-5: oak-OST014a_UUID: not available for connect from 10.210.12.72@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13901306.558771] Lustre: oak-OST0141: Client e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1) reconnecting
[13901306.569178] Lustre: Skipped 541 previous similar messages
[13901466.277280] LustreError: 243449:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(32768)  req@ffff9196f73c8850 x1715760492569152/t0(0) o3->4b39aac6-2dc7-e48f-9434-e3bfde492453@10.210.12.29@tcp1:688/0 lens 488/440 e 0 to 0 dl 1645127568 ref 1 fl Interpret:/0/0 rc 0/0
[13901466.302235] LustreError: 243449:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 6 previous similar messages
[13901466.311964] Lustre: oak-OST0135: Bulk IO read error with 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1), client will retry: rc -110
[13901466.325148] Lustre: Skipped 9 previous similar messages
[13901489.468204] LustreError: 21599:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9145c5fdf050 x1715242443453632/t0(0) o4->e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda@10.210.12.79@tcp1:1/0 lens 488/448 e 0 to 0 dl 1645127636 ref 1 fl Interpret:/0/0 rc 0/0
[13901489.492371] LustreError: 21599:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 30 previous similar messages
[13901489.502169] Lustre: oak-OST0143: Bulk IO write error with e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1), client will retry: rc = -110
[13901489.515608] Lustre: Skipped 32 previous similar messages
[13901538.123965] LustreError: 162702:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(1000973)  req@ffff915880a34050 x1715242443453760/t0(0) o4->e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda@10.210.12.79@tcp1:1/0 lens 488/448 e 0 to 0 dl 1645127636 ref 1 fl Interpret:/0/0 rc 0/0
[13901618.938817] LustreError: 244098:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff919eb717c050 x1716552409531840/t0(0) o4->6bab8d10-30ff-c1d2-bbf5-1fd0884ac647@10.210.12.57@tcp1:136/0 lens 488/448 e 0 to 0 dl 1645127771 ref 1 fl Interpret:/2/0 rc 0/0
[13901683.847130] Lustre: oak-OST013d: Connection restored to c8c804bb-e728-6f8c-527d-295ebfa95786 (at 10.50.4.35@o2ib2)
[13901683.857722] Lustre: Skipped 1294 previous similar messages
[13901695.533970] LustreError: 137-5: oak-OST014c_UUID: not available for connect from 10.210.12.8@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13901695.551465] LustreError: Skipped 1 previous similar message
[13901726.889404] LustreError: 127349:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff9198a1487050 x1714945033595776/t0(0) o4->5704685e-b0a9-b5c3-be3f-b278a4395bec@10.210.12.72@tcp1:245/0 lens 488/448 e 0 to 0 dl 1645127880 ref 1 fl Interpret:/2/0 rc 0/0
[13901909.871212] Lustre: oak-OST0111: Client 6f45695d-f173-ee1d-e6cb-d38dad7e0879 (at 10.210.12.74@tcp1) reconnecting
[13901909.881615] Lustre: Skipped 482 previous similar messages
[13901951.215801] LustreError: 137-5: oak-OST013a_UUID: not available for connect from 10.210.12.11@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13901951.233600] LustreError: Skipped 8 previous similar messages
[13902261.616696] LustreError: 228872:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91958f392050 x1715093299448896/t0(0) o4->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:25/0 lens 488/448 e 0 to 0 dl 1645128415 ref 1 fl Interpret:/0/0 rc 0/0
[13902261.618020] Lustre: oak-OST013f: Bulk IO write error with 5d8bb8d9-e5c7-334f-ec8d-f10452c309b5 (at 10.210.12.66@tcp1), client will retry: rc = -110
[13902261.618021] Lustre: Skipped 53 previous similar messages
[13902261.660034] LustreError: 228872:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 31 previous similar messages
[13902262.725039] Lustre: oak-OST0135: Bulk IO read error with 6bab8d10-30ff-c1d2-bbf5-1fd0884ac647 (at 10.210.12.57@tcp1), client will retry: rc -110
[13902262.738221] Lustre: Skipped 69 previous similar messages
[13902282.863000] Lustre: oak-OST0111: Connection restored to 11277db1-252d-53a3-037c-e2be03eeb9ee (at 10.51.6.40@o2ib3)
[13902282.874168] Lustre: Skipped 2145 previous similar messages
[13902328.485511] LustreError: 127357:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff919ecba0d850 x1716552421774656/t0(0) o4->6bab8d10-30ff-c1d2-bbf5-1fd0884ac647@10.210.12.57@tcp1:21/0 lens 488/448 e 0 to 0 dl 1645128411 ref 1 fl Interpret:/0/0 rc 0/0
[13902328.486121] LustreError: 228872:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff91b4f01af850 x1716552421774336/t0(0) o3->6bab8d10-30ff-c1d2-bbf5-1fd0884ac647@10.210.12.57@tcp1:26/0 lens 488/440 e 0 to 0 dl 1645128416 ref 1 fl Interpret:/2/0 rc 0/0
[13902328.486122] LustreError: 228872:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 65 previous similar messages
[13902328.547327] LustreError: 127357:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 24 previous similar messages
[13902518.388512] Lustre: oak-OST0143: Client e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1) reconnecting
[13902518.398917] Lustre: Skipped 505 previous similar messages
[13902597.877329] LustreError: 137-5: oak-OST0144_UUID: not available for connect from 10.210.12.42@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13902597.894922] LustreError: Skipped 2 previous similar messages
[13902881.657083] Lustre: oak-OST011f: Connection restored to bb719fde-6549-fed7-314d-539d729bbbca (at 10.51.0.72@o2ib3)
[13902881.667679] Lustre: Skipped 1310 previous similar messages
[13902903.352200] Lustre: oak-OST0143: Bulk IO write error with e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1), client will retry: rc = -110
[13902903.352345] Lustre: oak-OST0141: Bulk IO read error with e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1), client will retry: rc -110
[13902903.352346] Lustre: Skipped 4 previous similar messages
[13902903.384313] Lustre: Skipped 35 previous similar messages
[13903031.845625] LustreError: 160952:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91bdc697f850 x1715093325221504/t0(0) o4->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:42/0 lens 488/448 e 0 to 0 dl 1645129187 ref 1 fl Interpret:/0/0 rc 0/0
[13903031.870001] LustreError: 160952:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 26 previous similar messages
[13903160.791823] Lustre: oak-OST0137: Client e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1) reconnecting
[13903160.802262] Lustre: Skipped 190 previous similar messages
[13903213.751082] LustreError: 160955:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(1048576)  req@ffff91cd817d3850 x1716245186262208/t0(0) o3->6b341d38-3674-15b6-7e5e-137d0b4498c0@10.210.12.40@tcp1:166/0 lens 488/440 e 0 to 0 dl 1645129311 ref 1 fl Interpret:/0/0 rc 0/0
[13903213.751238] LustreError: 127352:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91f119d93850 x1721914088310208/t0(0) o4->e44e675f-11ba-536d-fbc7-173bafd860e2@10.210.12.46@tcp1:167/0 lens 488/448 e 0 to 0 dl 1645129312 ref 1 fl Interpret:/0/0 rc 0/0
[13903213.751240] LustreError: 127352:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 7 previous similar messages
[13903213.811943] LustreError: 160955:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 6 previous similar messages
[13903239.535073] LustreError: 137-5: oak-OST0120_UUID: not available for connect from 10.210.12.29@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13903239.552661] LustreError: Skipped 1 previous similar message
[13903481.025454] Lustre: oak-OST0133: Connection restored to  (at 10.51.6.5@o2ib3)
[13903481.032898] Lustre: Skipped 1048 previous similar messages
[13903837.756582] Lustre: oak-OST0149: Client 3b1423ea-f77c-9301-8ee2-822b91cad2d9 (at 10.210.12.10@tcp1) reconnecting
[13903837.768454] Lustre: Skipped 99 previous similar messages
[13903837.936733] LustreError: 162676:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff917320b08850 x1714981269874048/t0(0) o4->3b1423ea-f77c-9301-8ee2-822b91cad2d9@10.210.12.10@tcp1:91/0 lens 488/448 e 0 to 0 dl 1645129991 ref 1 fl Interpret:/0/0 rc 0/0
[13903837.961057] LustreError: 162676:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 11 previous similar messages
[13903837.971137] Lustre: oak-OST0149: Bulk IO write error with 3b1423ea-f77c-9301-8ee2-822b91cad2d9 (at 10.210.12.10@tcp1), client will retry: rc = -110
[13903837.984577] Lustre: Skipped 13 previous similar messages
[13904079.825415] Lustre: oak-OST0113: Connection restored to  (at 10.50.13.8@o2ib2)
[13904079.833062] Lustre: Skipped 1000 previous similar messages
[13904678.970534] Lustre: oak-OST012d: Connection restored to 55fa4ecb-ff66-16c3-914e-01f9698a4d93 (at 10.50.12.16@o2ib2)
[13904678.981199] Lustre: Skipped 917 previous similar messages
[13905278.337955] Lustre: oak-OST0141: Connection restored to 55fa4ecb-ff66-16c3-914e-01f9698a4d93 (at 10.50.12.16@o2ib2)
[13905278.348631] Lustre: Skipped 797 previous similar messages
[13905877.258843] Lustre: oak-OST014d: Connection restored to 1184ccb7-20db-b237-2e46-4625d1beb1ac (at 10.50.10.37@o2ib2)
[13905877.270051] Lustre: Skipped 850 previous similar messages
[13906405.531148] Lustre: oak-OST0135: Client e5a735ca-191d-07c9-fcef-626c0b3a28a8 (at 10.210.12.113@tcp1) reconnecting
[13906405.541745] Lustre: Skipped 72 previous similar messages
[13906478.635758] Lustre: oak-OST0145: Connection restored to 2e8e2ea1-d309-871f-47c3-fbf87733e2ac (at 10.50.5.45@o2ib2)
[13906478.646357] Lustre: Skipped 1067 previous similar messages
[13906539.343681] Lustre: oak-OST0111: Client 5afa23c2-1fd5-9030-38ce-b03a359082fe (at 10.210.12.75@tcp1) reconnecting
[13906539.354346] Lustre: Skipped 15 previous similar messages
[13906539.938998] LustreError: 162708:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91709d75d850 x1714970950475072/t0(0) o4->5afa23c2-1fd5-9030-38ce-b03a359082fe@10.210.12.75@tcp1:537/0 lens 488/448 e 0 to 0 dl 1645132702 ref 1 fl Interpret:/0/0 rc 0/0
[13906539.947233] Lustre: oak-OST0111: Bulk IO write error with 5afa23c2-1fd5-9030-38ce-b03a359082fe (at 10.210.12.75@tcp1), client will retry: rc = -110
[13906539.947235] Lustre: Skipped 5 previous similar messages
[13906539.982306] LustreError: 162708:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 6 previous similar messages
[13906708.796595] Lustre: oak-OST0127: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[13906708.807106] Lustre: Skipped 5 previous similar messages
[13906751.857964] LustreError: 162676:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9177d7031050 x1715093371167936/t0(0) o4->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:746/0 lens 488/448 e 0 to 0 dl 1645132911 ref 1 fl Interpret:/0/0 rc 0/0
[13906751.882394] LustreError: 162676:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 3 previous similar messages
[13906751.892439] Lustre: oak-OST0145: Bulk IO write error with 5d8bb8d9-e5c7-334f-ec8d-f10452c309b5 (at 10.210.12.66@tcp1), client will retry: rc = -110
[13906751.905870] Lustre: Skipped 4 previous similar messages
[13906796.885641] Lustre: oak-OST0147: Bulk IO read error with 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1), client will retry: rc -110
[13906796.898867] Lustre: Skipped 6 previous similar messages
[13906805.507714] LustreError: 162688:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff914ac3d0c850 x1715242569183040/t0(0) o4->e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda@10.210.12.79@tcp1:744/0 lens 488/448 e 0 to 0 dl 1645132909 ref 1 fl Interpret:/0/0 rc 0/0
[13906805.516046] LustreError: 243446:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(4194304)  req@ffff91f0a2ca0050 x1715093371144256/t0(0) o3->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:746/0 lens 488/440 e 0 to 0 dl 1645132911 ref 1 fl Interpret:/0/0 rc 0/0
[13906805.558607] LustreError: 162688:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 8 previous similar messages
[13906875.312198] LustreError: 137-5: oak-OST0138_UUID: not available for connect from 10.210.12.43@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13906875.329803] LustreError: Skipped 1 previous similar message
[13906973.318735] LustreError: 160924:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff91b5757df050 x1715534525416576/t0(0) o3->5a5caa86-68df-f806-7e6f-05440e95a4ce@10.210.12.43@tcp1:217/0 lens 488/440 e 0 to 0 dl 1645133137 ref 1 fl Interpret:/0/0 rc 0/0
[13906973.343077] LustreError: 160924:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 29 previous similar messages
[13906973.352895] Lustre: oak-OST0145: Bulk IO read error with 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1), client will retry: rc -110
[13906973.366084] Lustre: Skipped 5 previous similar messages
[13906974.434827] Lustre: oak-OST013f: Bulk IO write error with 3b1423ea-f77c-9301-8ee2-822b91cad2d9 (at 10.210.12.10@tcp1), client will retry: rc = -110
[13906974.448299] Lustre: Skipped 40 previous similar messages
[13907008.498472] Lustre: oak-OST0149: Client 0f13ac10-e5de-07d4-1915-b61e5ad35dce (at 10.210.12.115@tcp1) reconnecting
[13907008.509004] Lustre: Skipped 310 previous similar messages
[13907079.921472] Lustre: oak-OST013f: Connection restored to 4580346c-6415-5ca3-7a8d-c0a9e9615250 (at 10.51.12.18@o2ib3)
[13907079.932497] Lustre: Skipped 1164 previous similar messages
[13907286.715307] LustreError: 137-5: oak-OST0136_UUID: not available for connect from 10.210.12.40@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13907286.733034] LustreError: Skipped 4 previous similar messages
[13907413.634708] LustreError: 162685:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk READ failed: rc -107  req@ffff916d7f19a050 x1715046267625152/t0(0) o3->80374d70-7d99-f68a-b58b-86c0b7514d6a@10.210.12.65@tcp1:661/0 lens 488/440 e 0 to 0 dl 1645133581 ref 1 fl Interpret:/0/0 rc 0/0
[13907413.659508] LustreError: 162685:0:(ldlm_lib.c:3305:target_bulk_io()) Skipped 1 previous similar message
[13907413.669224] Lustre: oak-OST012d: Bulk IO read error with 80374d70-7d99-f68a-b58b-86c0b7514d6a (at 10.210.12.65@tcp1), client will retry: rc -107
[13907414.622199] LustreError: 21583:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff9177087b0850 x1715046267625024/t0(0) o3->80374d70-7d99-f68a-b58b-86c0b7514d6a@10.210.12.65@tcp1:661/0 lens 488/440 e 0 to 0 dl 1645133581 ref 1 fl Interpret:/0/0 rc 0/0
[13907414.646471] LustreError: 21583:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 6 previous similar messages
[13907422.317618] Lustre: oak-OST011d: Bulk IO write error with 6bab8d10-30ff-c1d2-bbf5-1fd0884ac647 (at 10.210.12.57@tcp1), client will retry: rc = -110
[13907422.331074] Lustre: Skipped 5 previous similar messages
[13907476.216976] LustreError: 243542:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(1048576)  req@ffff914ac3d0d050 x1715046267764096/t0(0) o3->80374d70-7d99-f68a-b58b-86c0b7514d6a@10.210.12.65@tcp1:663/0 lens 504/440 e 0 to 0 dl 1645133583 ref 1 fl Interpret:/0/0 rc 0/0
[13907476.217062] LustreError: 21590:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff918e94c37850 x1714907540803968/t0(0) o4->0d4d52ae-6d60-d275-1404-c9b4d99e0974@10.210.12.109@tcp1:664/0 lens 488/448 e 0 to 0 dl 1645133584 ref 1 fl Interpret:/0/0 rc 0/0
[13907476.217064] LustreError: 21590:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 7 previous similar messages
[13907476.277892] LustreError: 243542:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 4 previous similar messages
[13907477.502403] LustreError: 244099:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk READ failed: rc -107  req@ffff91c72ee58050 x1716245327877504/t0(0) o3->6b341d38-3674-15b6-7e5e-137d0b4498c0@10.210.12.40@tcp1:723/0 lens 488/440 e 0 to 0 dl 1645133643 ref 1 fl Interpret:/0/0 rc 0/0
[13907502.247617] LustreError: 137-5: oak-OST0136_UUID: not available for connect from 10.210.12.109@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13907607.492920] Lustre: oak-OST0111: Client 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1) reconnecting
[13907607.503372] Lustre: Skipped 204 previous similar messages
[13907679.106097] Lustre: oak-OST011f: Connection restored to 1620716c-1942-3555-76e1-e88f2b7e8c43 (at 10.51.6.33@o2ib3)
[13907679.116681] Lustre: Skipped 1842 previous similar messages
[13907738.711621] LustreError: 162704:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff918c201a2050 x1714947531006592/t0(0) o4->e01e4c82-9fd9-5cc5-61dd-b09c7945df2b@10.210.12.11@tcp1:168/0 lens 488/448 e 0 to 0 dl 1645133843 ref 1 fl Interpret:/0/0 rc 0/0
[13907738.737430] LustreError: 162704:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 4 previous similar messages
[13907786.610379] LustreError: 160901:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff91ef2486f850 x1715534533116992/t0(0) o3->5a5caa86-68df-f806-7e6f-05440e95a4ce@10.210.12.43@tcp1:213/0 lens 488/440 e 0 to 0 dl 1645133888 ref 1 fl Interpret:/0/0 rc 0/0
[13907786.610519] Lustre: oak-OST0145: Bulk IO read error with 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1), client will retry: rc -110
[13907786.610520] Lustre: Skipped 6 previous similar messages
[13907786.654690] LustreError: 160901:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 4 previous similar messages
[13907804.971418] LustreError: 137-5: oak-OST0122_UUID: not available for connect from 10.210.12.117@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13907804.989088] LustreError: Skipped 6 previous similar messages
[13908092.948649] LustreError: 228677:0:(tgt_handler.c:651:process_req_last_xid()) @@@ Unexpected xid 618ea823eabc0 vs. last_xid 618ea823ec2ff
                    req@ffff91ab67ca0850 x1716245346823104/t0(0) o3->6b341d38-3674-15b6-7e5e-137d0b4498c0@10.210.12.40@tcp1:586/0 lens 488/0 e 0 to 0 dl 1645134261 ref 1 fl Interpret:/2/ffffffff rc 0/-1
[13908253.887647] Lustre: oak-OST0123: Client b4fca75a-2d54-9bf2-331f-67d0c52e5ff2 (at 10.210.12.73@tcp1) reconnecting
[13908253.898062] Lustre: Skipped 395 previous similar messages
[13908277.792050] Lustre: oak-OST0139: Connection restored to e10c3e22-d196-9db7-8dfa-d975b5ef1879 (at 10.50.8.30@o2ib2)
[13908277.802634] Lustre: Skipped 1249 previous similar messages
[13908369.478533] LustreError: 162700:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9172d677b050 x1714908321832000/t0(0) o4->0f13ac10-e5de-07d4-1915-b61e5ad35dce@10.210.12.115@tcp1:104/0 lens 488/448 e 0 to 0 dl 1645134534 ref 1 fl Interpret:/0/0 rc 0/0
[13908369.503042] LustreError: 162700:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 24 previous similar messages
[13908369.503713] Lustre: oak-OST013f: Bulk IO write error with 0f13ac10-e5de-07d4-1915-b61e5ad35dce (at 10.210.12.115@tcp1), client will retry: rc = -110
[13908369.503714] Lustre: Skipped 39 previous similar messages
[13908448.768178] LustreError: 137-5: oak-OST0112_UUID: not available for connect from 10.210.12.115@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13908448.785891] LustreError: Skipped 2 previous similar messages
[13908863.849227] Lustre: oak-OST0129: Client 5d8bb8d9-e5c7-334f-ec8d-f10452c309b5 (at 10.210.12.66@tcp1) reconnecting
[13908863.859637] Lustre: Skipped 162 previous similar messages
[13908879.538889] Lustre: oak-OST014d: Connection restored to 4881b494-d9fe-4c3e-0129-4f23331562aa (at 10.50.7.29@o2ib2)
[13908879.549467] Lustre: Skipped 671 previous similar messages
[13908912.355200] LustreError: 21615:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff9164cdaeb850 x1715024893840576/t0(0) o3->6d01c866-9ec9-8f84-4f86-ee1db83afc97@10.210.12.61@tcp1:599/0 lens 488/440 e 0 to 0 dl 1645135029 ref 1 fl Interpret:/0/0 rc 0/0
[13908912.355269] Lustre: oak-OST0133: Bulk IO read error with 6d01c866-9ec9-8f84-4f86-ee1db83afc97 (at 10.210.12.61@tcp1), client will retry: rc -110
[13908912.355271] Lustre: Skipped 4 previous similar messages
[13908912.359802] LustreError: 160948:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91b150278850 x1715093413593152/t0(0) o4->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:599/0 lens 488/448 e 0 to 0 dl 1645135029 ref 1 fl Interpret:/0/0 rc 0/0
[13908912.359804] LustreError: 160948:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 10 previous similar messages
[13908912.435262] LustreError: 21615:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 4 previous similar messages
[13909113.322353] Lustre: oak-OST011b: haven't heard from client fdef5f27-0b97-defc-3712-3663d5a395ff (at 10.51.14.19@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91c029206800, cur 1645135187 expire 1645135037 last 1645134960
[13909113.344345] Lustre: Skipped 9 previous similar messages
[13909179.915938] LustreError: 21617:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff913d5c919850 x1715013813641984/t0(0) o4->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:160/0 lens 488/448 e 0 to 0 dl 1645135345 ref 1 fl Interpret:/0/0 rc 0/0
[13909179.940276] LustreError: 21617:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 6 previous similar messages
[13909179.950146] Lustre: oak-OST0133: Bulk IO write error with c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1), client will retry: rc = -110
[13909179.963655] Lustre: Skipped 9 previous similar messages
[13909223.729465] LustreError: 162675:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(20049)  req@ffff91867b73a850 x1715426664297344/t0(0) o4->94d559ad-9ea8-2fc9-41c4-60f721062c9a@10.210.12.52@tcp1:159/0 lens 488/448 e 0 to 0 dl 1645135344 ref 1 fl Interpret:/0/0 rc 0/0
[13909223.729801] LustreError: 228401:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(20480)  req@ffff91e1a3e5a050 x1715013813626944/t0(0) o3->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:160/0 lens 488/440 e 0 to 0 dl 1645135345 ref 1 fl Interpret:/0/0 rc 0/0
[13909223.729803] LustreError: 228401:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 2 previous similar messages
[13909223.789233] LustreError: 162675:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 4 previous similar messages
[13909257.106545] LustreError: 137-5: oak-OST012e_UUID: not available for connect from 10.210.12.73@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13909257.124145] LustreError: Skipped 3 previous similar messages
[13909478.644482] Lustre: oak-OST0111: Connection restored to 962451b3-9815-1647-4d58-74345634d1e2 (at 10.50.6.38@o2ib2)
[13909478.655143] Lustre: Skipped 1657 previous similar messages
[13909726.741588] LustreError: 127347:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff91eecf172850 x1714973404485824/t0(0) o3->5d4d5527-a52f-bc68-278b-aa990b8609d6@10.210.12.59@tcp1:664/0 lens 488/440 e 0 to 0 dl 1645135849 ref 1 fl Interpret:/0/0 rc 0/0
[13909726.741749] Lustre: oak-OST0129: Bulk IO read error with 5d4d5527-a52f-bc68-278b-aa990b8609d6 (at 10.210.12.59@tcp1), client will retry: rc -110
[13909726.741750] Lustre: Skipped 9 previous similar messages
[13909726.785786] LustreError: 127347:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 3 previous similar messages
[13909848.994938] Lustre: oak-OST0137: Client 5d4d5527-a52f-bc68-278b-aa990b8609d6 (at 10.210.12.59@tcp1) reconnecting
[13909849.005380] Lustre: Skipped 504 previous similar messages
[13910081.531950] Lustre: oak-OST0111: Connection restored to eea359e6-6b89-c326-60af-8cb99076065b (at 10.210.12.8@tcp1)
[13910081.542540] Lustre: Skipped 1246 previous similar messages
[13910659.913070] LustreError: 160927:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff91db0bc90050 x1715085872871168/t0(0) o3->65e9b8a8-b461-24e0-da12-5b6939f0b07c@10.210.12.56@tcp1:73/0 lens 488/440 e 0 to 0 dl 1645136768 ref 1 fl Interpret:/0/0 rc 0/0
[13910659.913183] LustreError: 160923:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff919df8951050 x1714945219928256/t0(0) o4->5704685e-b0a9-b5c3-be3f-b278a4395bec@10.210.12.72@tcp1:74/0 lens 488/448 e 0 to 0 dl 1645136769 ref 1 fl Interpret:/0/0 rc 0/0
[13910659.913415] Lustre: oak-OST0115: Bulk IO write error with 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1), client will retry: rc = -110
[13910659.913416] Lustre: Skipped 17 previous similar messages
[13910659.983319] LustreError: 160927:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 2 previous similar messages
[13910659.993193] Lustre: oak-OST013b: Bulk IO read error with 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1), client will retry: rc -110
[13910660.006373] Lustre: Skipped 5 previous similar messages
[13910678.638515] Lustre: oak-OST0131: Client 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1) reconnecting
[13910678.648920] Lustre: Skipped 119 previous similar messages
[13910680.723850] Lustre: oak-OST011d: Connection restored to 938e4fcb-40fa-8576-5696-2871684d71bd (at 10.50.5.8@o2ib2)
[13910680.734407] Lustre: Skipped 941 previous similar messages
[13910803.883089] LustreError: 243448:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91cd94522050 x1715085905343872/t0(0) o4->65e9b8a8-b461-24e0-da12-5b6939f0b07c@10.210.12.56@tcp1:283/0 lens 488/448 e 0 to 0 dl 1645136978 ref 1 fl Interpret:/0/0 rc 0/0
[13910803.907526] LustreError: 243448:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 15 previous similar messages
[13910803.917520] Lustre: oak-OST014d: Bulk IO write error with 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1), client will retry: rc = -110
[13910803.931055] Lustre: Skipped 1 previous similar message
[13911282.743598] Lustre: oak-OST013f: Connection restored to bfd7c29e-205a-a45e-5add-4648092cd4c2 (at 10.50.6.39@o2ib2)
[13911282.754214] Lustre: Skipped 957 previous similar messages
[13911881.293462] Lustre: oak-OST0125: Connection restored to 1e1f187a-28cb-d390-d52c-e3db41797544 (at 10.51.15.21@o2ib3)
[13911881.304127] Lustre: Skipped 875 previous similar messages
[13912201.891437] Lustre: oak-OST0133: Client 1b588188-9255-1b9f-1308-154ba9e467a3 (at 10.210.12.7@tcp1) reconnecting
[13912201.901766] Lustre: Skipped 39 previous similar messages
[13912303.907872] LustreError: 160899:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[13912323.697236] LustreError: 160897:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[13912476.140583] Lustre: oak-OST0129: haven't heard from client 90111784-329b-3248-9f72-ee2373514578 (at 10.51.15.3@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91a8e0d2a400, cur 1645138558 expire 1645138408 last 1645138331
[13912476.162515] Lustre: Skipped 8 previous similar messages
[13912481.974133] Lustre: oak-OST0145: Connection restored to 7db617e0-a322-35ce-992d-730fbb97f163 (at 10.51.1.29@o2ib3)
[13912481.984825] Lustre: Skipped 1017 previous similar messages
[13912485.131925] Lustre: oak-OST014d: haven't heard from client 90111784-329b-3248-9f72-ee2373514578 (at 10.51.15.3@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91d0d54b0000, cur 1645138567 expire 1645138417 last 1645138340
[13912485.153835] Lustre: Skipped 148 previous similar messages
[13913032.921166] Lustre: oak-OST013f: Client 80374d70-7d99-f68a-b58b-86c0b7514d6a (at 10.210.12.65@tcp1) reconnecting
[13913032.931595] Lustre: Skipped 26 previous similar messages
[13913033.270706] LustreError: 228831:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91d206510050 x1715046409609152/t0(0) o4->80374d70-7d99-f68a-b58b-86c0b7514d6a@10.210.12.65@tcp1:249/0 lens 488/448 e 0 to 0 dl 1645139209 ref 1 fl Interpret:/0/0 rc 0/0
[13913033.295279] Lustre: oak-OST0145: Bulk IO write error with 80374d70-7d99-f68a-b58b-86c0b7514d6a (at 10.210.12.65@tcp1), client will retry: rc = -110
[13913081.011469] Lustre: oak-OST012b: Connection restored to 916093a3-51ee-71cf-ab61-9d8121d32b7c (at 10.51.15.11@o2ib3)
[13913081.022153] Lustre: Skipped 815 previous similar messages
[13913206.806267] Lustre: oak-OST014b: Client 80374d70-7d99-f68a-b58b-86c0b7514d6a (at 10.210.12.65@tcp1) reconnecting
[13913206.816678] Lustre: Skipped 2 previous similar messages
[13913214.371336] Lustre: oak-OST0137: haven't heard from client 9ae6d8b2-45e2-7032-40ca-c019885f430e (at 10.50.13.3@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91ce3f64c800, cur 1645139298 expire 1645139148 last 1645139071
[13913214.393303] Lustre: Skipped 4 previous similar messages
[13913218.328270] Lustre: oak-OST0127: haven't heard from client 9ae6d8b2-45e2-7032-40ca-c019885f430e (at 10.50.13.3@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91d43b448800, cur 1645139302 expire 1645139152 last 1645139075
[13913238.218716] Lustre: oak-OST0117: Client 80374d70-7d99-f68a-b58b-86c0b7514d6a (at 10.210.12.65@tcp1) reconnecting
[13913238.229124] Lustre: Skipped 6 previous similar messages
[13913279.231566] Lustre: oak-OST0133: Client 80374d70-7d99-f68a-b58b-86c0b7514d6a (at 10.210.12.65@tcp1) reconnecting
[13913279.241973] Lustre: Skipped 4 previous similar messages
[13913681.438782] Lustre: oak-OST0125: Connection restored to 776400dd-7447-858b-13ed-29f579e7c804 (at 10.50.7.32@o2ib2)
[13913681.449469] Lustre: Skipped 948 previous similar messages
[13914084.744451] Lustre: oak-OST0147: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[13914084.754882] Lustre: Skipped 9 previous similar messages
[13914084.921841] LustreError: 21584:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff915953101050 x1714980382024320/t0(0) o4->d4faaef8-44e2-b05f-6116-d9c30b2e4e11@10.210.12.60@tcp1:550/0 lens 488/448 e 0 to 0 dl 1645140265 ref 1 fl Interpret:/0/0 rc 0/0
[13914084.946187] LustreError: 21584:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 5 previous similar messages
[13914084.955995] Lustre: oak-OST0147: Bulk IO write error with d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1), client will retry: rc = -110
[13914084.969432] Lustre: Skipped 5 previous similar messages
[13914085.612407] LustreError: 162694:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff9136dcbab850 x1714980382016064/t0(0) o4->d4faaef8-44e2-b05f-6116-d9c30b2e4e11@10.210.12.60@tcp1:553/0 lens 488/448 e 0 to 0 dl 1645140268 ref 1 fl Interpret:/2/0 rc 0/0
[13914085.637112] LustreError: 162694:0:(ldlm_lib.c:3305:target_bulk_io()) Skipped 3 previous similar messages
[13914131.681149] LustreError: 21606:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff918f7b1cd050 x1715046438917376/t0(0) o4->80374d70-7d99-f68a-b58b-86c0b7514d6a@10.210.12.65@tcp1:546/0 lens 488/448 e 0 to 0 dl 1645140261 ref 1 fl Interpret:/0/0 rc 0/0
[13914131.681272] Lustre: oak-OST0145: Bulk IO write error with 80374d70-7d99-f68a-b58b-86c0b7514d6a (at 10.210.12.65@tcp1), client will retry: rc = -110
[13914131.681273] Lustre: Skipped 6 previous similar messages
[13914131.725819] LustreError: 21606:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 3 previous similar messages
[13914162.218736] Lustre: oak-OST011d: Client 5a4881be-a0cb-e632-509b-2e15c534b21a (at 10.210.12.9@tcp1) reconnecting
[13914162.229068] Lustre: Skipped 2 previous similar messages
[13914163.046600] LustreError: 137-5: oak-OST0148_UUID: not available for connect from 10.210.12.64@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13914163.064188] LustreError: Skipped 4 previous similar messages
[13914241.911200] Lustre: oak-OST0111: Client c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1) reconnecting
[13914241.921627] Lustre: Skipped 15 previous similar messages
[13914282.453883] Lustre: oak-OST0149: Connection restored to e10ff487-e8ae-c597-6928-490cf86bc28a (at 10.50.4.28@o2ib2)
[13914282.464472] Lustre: Skipped 926 previous similar messages
[13914285.705288] Lustre: oak-OST011b: Client c4b979fd-3b98-af30-5ea5-32f00f4d8750 (at 10.210.12.64@tcp1) reconnecting
[13914285.715710] Lustre: Skipped 141 previous similar messages
[13914363.076009] Lustre: oak-OST0115: Client c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1) reconnecting
[13914363.086422] Lustre: Skipped 28 previous similar messages
[13914881.108787] Lustre: oak-OST013f: Connection restored to b721998c-1a52-1c1c-cbd9-f9819458c553 (at 10.210.12.131@tcp1)
[13914881.119542] Lustre: Skipped 1061 previous similar messages
[13915482.218339] Lustre: oak-OST0129: Connection restored to 9723b0e8-146d-3b75-c37a-7497ce06f21c (at 10.50.1.44@o2ib2)
[13915482.228928] Lustre: Skipped 707 previous similar messages
[13915569.618394] Lustre: oak-OST013f: haven't heard from client 4831dd49-5d2b-49d6-af49-bb71a3d9949c (at 10.51.15.13@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91ef2ccab000, cur 1645141659 expire 1645141509 last 1645141432
[13915569.640533] Lustre: Skipped 1 previous similar message
[13916082.344883] Lustre: oak-OST0147: Connection restored to 08e827f3-caa3-2cb0-507e-5bd8243ac158 (at 10.51.12.2@o2ib3)
[13916082.355474] Lustre: Skipped 680 previous similar messages
[13916682.562502] Lustre: oak-OST0145: Connection restored to d68efeb9-ff66-610a-8f04-303d4aa2ac0a (at 10.51.6.21@o2ib3)
[13916682.573221] Lustre: Skipped 1501 previous similar messages
[13917282.653104] Lustre: oak-OST0137: Connection restored to  (at 10.51.15.12@o2ib3)
[13917282.660661] Lustre: Skipped 792 previous similar messages
[13917881.204855] Lustre: oak-OST0139: Connection restored to  (at 10.51.5.16@o2ib3)
[13917881.212351] Lustre: Skipped 1704 previous similar messages
[13917995.260677] Lustre: oak-OST013f: Client f4048b3b-d9e4-178d-cd58-3b66faaca4ee (at 10.210.12.117@tcp1) reconnecting
[13917995.271170] Lustre: Skipped 8 previous similar messages
[13917995.276884] LustreError: 160948:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91b87e35f850 x1714909803461120/t0(0) o4->f4048b3b-d9e4-178d-cd58-3b66faaca4ee@10.210.12.117@tcp1:690/0 lens 488/448 e 0 to 0 dl 1645144180 ref 1 fl Interpret:/0/0 rc 0/0
[13917995.301461] LustreError: 160948:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 5 previous similar messages
[13917995.311506] Lustre: oak-OST013f: Bulk IO write error with f4048b3b-d9e4-178d-cd58-3b66faaca4ee (at 10.210.12.117@tcp1), client will retry: rc = -110
[13917995.325066] Lustre: Skipped 2 previous similar messages
[13917995.892385] LustreError: 160937:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91c715ed7850 x1714909803460992/t0(0) o4->f4048b3b-d9e4-178d-cd58-3b66faaca4ee@10.210.12.117@tcp1:689/0 lens 488/448 e 0 to 0 dl 1645144179 ref 1 fl Interpret:/0/0 rc 0/0
[13917995.916923] LustreError: 160937:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[13917995.926952] Lustre: oak-OST013f: Bulk IO write error with f4048b3b-d9e4-178d-cd58-3b66faaca4ee (at 10.210.12.117@tcp1), client will retry: rc = -110
[13917995.940521] Lustre: Skipped 1 previous similar message
[13918049.158574] LustreError: 137-5: oak-OST0138_UUID: not available for connect from 10.210.12.54@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13918049.176698] LustreError: Skipped 2 previous similar messages
[13918068.032593] LustreError: 137-5: oak-OST013c_UUID: not available for connect from 10.210.12.75@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13918068.050192] LustreError: Skipped 1 previous similar message
[13918135.798371] Lustre: oak-OST012b: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[13918174.555358] Lustre: oak-OST0143: Client 6d01c866-9ec9-8f84-4f86-ee1db83afc97 (at 10.210.12.61@tcp1) reconnecting
[13918174.565845] Lustre: Skipped 176 previous similar messages
[13918479.762387] Lustre: oak-OST0125: Connection restored to 00f73bb3-245d-df73-6e5b-6349096626b0 (at 10.50.2.67@o2ib2)
[13918479.773001] Lustre: Skipped 1603 previous similar messages
[13918538.886134] Lustre: oak-OST0117: Client a166a60d-30d5-1e72-b799-745de1d3b307 (at 10.210.12.125@tcp1) reconnecting
[13918538.896630] Lustre: Skipped 16 previous similar messages
[13919078.315106] Lustre: oak-OST0129: Connection restored to 10a8ef29-efa9-94e6-5353-edbfb5fae3ff (at 10.51.5.25@o2ib3)
[13919078.325690] Lustre: Skipped 1009 previous similar messages
[13919679.285981] Lustre: oak-OST0131: Connection restored to 916093a3-51ee-71cf-ab61-9d8121d32b7c (at 10.51.15.11@o2ib3)
[13919679.296668] Lustre: Skipped 1246 previous similar messages
[13920277.872469] Lustre: oak-OST0121: Connection restored to  (at 10.51.15.12@o2ib3)
[13920277.880060] Lustre: Skipped 1208 previous similar messages
[13920434.358134] Lustre: oak-OST0129: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[13920434.368563] Lustre: Skipped 22 previous similar messages
[13920434.481466] LustreError: 160927:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91964b3e6850 x1714945452206720/t0(0) o4->5704685e-b0a9-b5c3-be3f-b278a4395bec@10.210.12.72@tcp1:123/0 lens 488/448 e 0 to 0 dl 1645146633 ref 1 fl Interpret:/2/0 rc 0/0
[13920434.490085] Lustre: oak-OST013b: Bulk IO write error with 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1), client will retry: rc = -110
[13920434.519517] LustreError: 160927:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 3 previous similar messages
[13920434.955834] LustreError: 160897:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff91da4c18b850 x1714945452211520/t0(0) o4->5704685e-b0a9-b5c3-be3f-b278a4395bec@10.210.12.72@tcp1:123/0 lens 488/448 e 0 to 0 dl 1645146633 ref 1 fl Interpret:/2/0 rc 0/0
[13920435.328563] LustreError: 243450:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91adc09d3850 x1714945452218752/t0(0) o4->5704685e-b0a9-b5c3-be3f-b278a4395bec@10.210.12.72@tcp1:123/0 lens 488/448 e 0 to 0 dl 1645146633 ref 1 fl Interpret:/0/0 rc 0/0
[13920435.353125] LustreError: 243450:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[13920435.357775] Lustre: oak-OST013b: Bulk IO write error with 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1), client will retry: rc = -110
[13920435.357776] Lustre: Skipped 5 previous similar messages
[13920501.302382] LustreError: 160951:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91d43b4de050 x1714945452213312/t0(0) o4->5704685e-b0a9-b5c3-be3f-b278a4395bec@10.210.12.72@tcp1:123/0 lens 488/448 e 0 to 0 dl 1645146633 ref 1 fl Interpret:/0/0 rc 0/0
[13920501.328604] Lustre: oak-OST014d: Bulk IO write error with 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1), client will retry: rc = -110
[13920501.342135] Lustre: Skipped 4 previous similar messages
[13920514.285011] LustreError: 137-5: oak-OST013c_UUID: not available for connect from 10.210.12.72@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13920514.303098] LustreError: Skipped 1 previous similar message
[13920516.417981] Lustre: oak-OST014d: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[13920516.428428] Lustre: Skipped 1 previous similar message
[13920602.894092] Lustre: oak-OST0123: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[13920709.982413] Lustre: oak-OST0133: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[13920709.992890] Lustre: Skipped 25 previous similar messages
[13920876.826600] Lustre: oak-OST011f: Connection restored to e10ff487-e8ae-c597-6928-490cf86bc28a (at 10.50.4.28@o2ib2)
[13920876.837188] Lustre: Skipped 1407 previous similar messages
[13921476.150241] Lustre: oak-OST0113: Connection restored to 08e827f3-caa3-2cb0-507e-5bd8243ac158 (at 10.51.12.2@o2ib3)
[13921476.160828] Lustre: Skipped 855 previous similar messages
[13921911.215640] Lustre: oak-OST013f: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[13921911.226046] Lustre: Skipped 2 previous similar messages
[13921911.281000] LustreError: 21620:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff918bb1fc7850 x1714980495324736/t0(0) o4->d4faaef8-44e2-b05f-6116-d9c30b2e4e11@10.210.12.60@tcp1:93/0 lens 488/448 e 0 to 0 dl 1645148113 ref 1 fl Interpret:/0/0 rc 0/0
[13921911.305467] Lustre: oak-OST0145: Bulk IO write error with d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1), client will retry: rc = -110
[13921911.899913] LustreError: 21612:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff916ab206b850 x1714980495261312/t0(0) o4->d4faaef8-44e2-b05f-6116-d9c30b2e4e11@10.210.12.60@tcp1:89/0 lens 488/448 e 0 to 0 dl 1645148109 ref 1 fl Interpret:/0/0 rc 0/0
[13921911.926572] LustreError: 21612:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 7 previous similar messages
[13921911.937472] Lustre: oak-OST013f: Bulk IO write error with d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1), client will retry: rc = -110
[13921911.950907] Lustre: Skipped 7 previous similar messages
[13921913.895055] LustreError: 21600:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9174d0797850 x1714980495365760/t0(0) o4->d4faaef8-44e2-b05f-6116-d9c30b2e4e11@10.210.12.60@tcp1:95/0 lens 488/448 e 0 to 0 dl 1645148115 ref 1 fl Interpret:/0/0 rc 0/0
[13921913.895250] Lustre: oak-OST0145: Bulk IO write error with d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1), client will retry: rc = -110
[13921913.895251] Lustre: Skipped 1 previous similar message
[13921913.939635] LustreError: 21600:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 7 previous similar messages
[13922075.313489] Lustre: oak-OST014b: Connection restored to b072e58c-a9ad-befd-5eb6-8daa35f00cce (at 10.50.2.47@o2ib2)
[13922075.324094] Lustre: Skipped 780 previous similar messages
[13922078.820538] Lustre: oak-OST011b: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[13922078.830951] Lustre: Skipped 5 previous similar messages
[13922117.298510] Lustre: oak-OST0115: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[13922117.308929] Lustre: Skipped 18 previous similar messages
[13922179.548634] Lustre: oak-OST011d: haven't heard from client 9e0bec42-2516-cb4f-fa3a-d5b2b27d85db (at 10.51.13.23@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91aa64b40800, cur 1645148285 expire 1645148135 last 1645148058
[13922179.570783] Lustre: Skipped 2 previous similar messages
[13922182.583967] Lustre: oak-OST013b: haven't heard from client 9e0bec42-2516-cb4f-fa3a-d5b2b27d85db (at 10.51.13.23@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff919690b86000, cur 1645148288 expire 1645148138 last 1645148061
[13922190.527114] Lustre: oak-OST0131: haven't heard from client 9e0bec42-2516-cb4f-fa3a-d5b2b27d85db (at 10.51.13.23@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91e3c1c46c00, cur 1645148296 expire 1645148146 last 1645148069
[13922677.624310] Lustre: oak-OST011f: Connection restored to cf2f9e97-f24f-326b-2645-71fbadcee615 (at 10.50.9.67@o2ib2)
[13922677.634890] Lustre: Skipped 866 previous similar messages
[13923276.813219] Lustre: oak-OST0131: Connection restored to 916093a3-51ee-71cf-ab61-9d8121d32b7c (at 10.51.15.11@o2ib3)
[13923276.823896] Lustre: Skipped 892 previous similar messages
[13923396.642148] Lustre: oak-OST0115: haven't heard from client 0a464c45-e5dc-c630-2fde-08a162e2ed6e (at 10.51.15.14@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91bc37ae3c00, cur 1645149505 expire 1645149355 last 1645149278
[13923558.201990] Lustre: oak-OST014d: haven't heard from client 35bee231-4504-a89a-02e5-7f6285d0268a (at 10.50.13.2@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff914e5f618800, cur 1645149667 expire 1645149517 last 1645149440
[13923558.223907] Lustre: Skipped 3 previous similar messages
[13923559.200619] Lustre: oak-OST0145: haven't heard from client 35bee231-4504-a89a-02e5-7f6285d0268a (at 10.50.13.2@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91dfd9875000, cur 1645149668 expire 1645149518 last 1645149441
[13923559.222611] Lustre: Skipped 3 previous similar messages
[13923751.889060] Lustre: oak-OST0145: Client 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1) reconnecting
[13923751.899541] Lustre: Skipped 6 previous similar messages
[13923751.906050] LustreError: 243497:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff916cc6229850 x1714952921780032/t0(0) o4->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:425/0 lens 488/448 e 0 to 0 dl 1645149955 ref 1 fl Interpret:/0/0 rc 0/0
[13923751.930575] LustreError: 243497:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 7 previous similar messages
[13923751.940921] Lustre: oak-OST0145: Bulk IO write error with 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1), client will retry: rc = -110
[13923751.954452] Lustre: Skipped 13 previous similar messages
[13923752.411808] LustreError: 162678:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9177d7034050 x1715534717909632/t0(0) o4->5a5caa86-68df-f806-7e6f-05440e95a4ce@10.210.12.43@tcp1:424/0 lens 488/448 e 0 to 0 dl 1645149954 ref 1 fl Interpret:/0/0 rc 0/0
[13923752.436289] LustreError: 162678:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[13923752.446176] Lustre: oak-OST011f: Bulk IO write error with 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1), client will retry: rc = -110
[13923752.459610] Lustre: Skipped 1 previous similar message
[13923753.469253] LustreError: 243495:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff916cc622f050 x1715534717934976/t0(0) o4->5a5caa86-68df-f806-7e6f-05440e95a4ce@10.210.12.43@tcp1:429/0 lens 488/448 e 0 to 0 dl 1645149959 ref 1 fl Interpret:/0/0 rc 0/0
[13923753.493757] LustreError: 243495:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[13923753.503709] Lustre: oak-OST013b: Bulk IO write error with 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1), client will retry: rc = -110
[13923753.517146] Lustre: Skipped 1 previous similar message
[13923832.251366] LustreError: 137-5: oak-OST0142_UUID: not available for connect from 10.210.12.73@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13923875.670631] Lustre: oak-OST013d: Connection restored to 91d4cde6-c786-55a5-5ed0-39d2fe9e24c2 (at 10.51.12.16@o2ib3)
[13923875.681442] Lustre: Skipped 938 previous similar messages
[13923919.695140] Lustre: oak-OST0121: Client 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1) reconnecting
[13923919.705647] Lustre: Skipped 8 previous similar messages
[13923942.092852] Lustre: oak-OST0115: Client 7189afb8-8fe5-7418-d14a-4a60d24330d0 (at 10.210.12.48@tcp1) reconnecting
[13923942.103287] Lustre: Skipped 69 previous similar messages
[13924474.581127] Lustre: oak-OST011f: Connection restored to 86bf896b-a2aa-b4a1-25b7-5ef5f50ac871 (at 10.50.5.13@o2ib2)
[13924474.591942] Lustre: Skipped 1049 previous similar messages
[13925073.408334] Lustre: oak-OST0143: Connection restored to  (at 10.51.15.6@o2ib3)
[13925073.416106] Lustre: Skipped 802 previous similar messages
[13925242.324833] Lustre: oak-OST013b: Client 7189afb8-8fe5-7418-d14a-4a60d24330d0 (at 10.210.12.48@tcp1) reconnecting
[13925242.335237] Lustre: Skipped 42 previous similar messages
[13925242.902609] LustreError: 243442:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff914d6d5b2850 x1715185176031872/t0(0) o4->7189afb8-8fe5-7418-d14a-4a60d24330d0@10.210.12.48@tcp1:410/0 lens 488/448 e 0 to 0 dl 1645151450 ref 1 fl Interpret:/0/0 rc 0/0
[13925242.927179] LustreError: 243442:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 6 previous similar messages
[13925242.937173] Lustre: oak-OST013b: Bulk IO write error with 7189afb8-8fe5-7418-d14a-4a60d24330d0 (at 10.210.12.48@tcp1), client will retry: rc = -110
[13925242.950614] Lustre: Skipped 6 previous similar messages
[13925289.740805] LustreError: 21613:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff9154b9331850 x1715061744672064/t0(0) o4->dc851ef6-3898-c01c-adc6-dc548a499d03@10.210.12.53@tcp1:406/0 lens 488/448 e 0 to 0 dl 1645151446 ref 1 fl Interpret:/0/0 rc 0/0
[13925289.740842] LustreError: 162678:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(1048576)  req@ffff91914fbc4050 x1714971392839680/t0(0) o3->5afa23c2-1fd5-9030-38ce-b03a359082fe@10.210.12.75@tcp1:407/0 lens 504/440 e 0 to 0 dl 1645151447 ref 1 fl Interpret:/0/0 rc 0/0
[13925289.740876] Lustre: oak-OST0143: Bulk IO write error with 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1), client will retry: rc = -110
[13925289.740906] Lustre: oak-OST0133: Bulk IO read error with 5afa23c2-1fd5-9030-38ce-b03a359082fe (at 10.210.12.75@tcp1), client will retry: rc -110
[13925289.818247] LustreError: 21613:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 5 previous similar messages
[13925320.161400] Lustre: oak-OST0143: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[13925321.314675] LustreError: 137-5: oak-OST0130_UUID: not available for connect from 10.210.12.59@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13925409.100716] Lustre: oak-OST0123: Client 7189afb8-8fe5-7418-d14a-4a60d24330d0 (at 10.210.12.48@tcp1) reconnecting
[13925409.111191] Lustre: Skipped 5 previous similar messages
[13925428.277569] Lustre: oak-OST0113: Client 479db7cf-e752-7ee3-6b36-d1f694b751cf (at 10.210.12.47@tcp1) reconnecting
[13925428.288015] Lustre: Skipped 188 previous similar messages
[13925473.658508] Lustre: oak-OST0131: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[13925473.668935] Lustre: Skipped 12 previous similar messages
[13925563.799668] Lustre: oak-OST0123: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[13925563.810105] Lustre: Skipped 12 previous similar messages
[13925673.159060] Lustre: oak-OST0131: Connection restored to a9cebbbf-4465-65a4-bf00-99bae7741269 (at 10.51.13.2@o2ib3)
[13925673.169666] Lustre: Skipped 1039 previous similar messages
[13926272.403728] Lustre: oak-OST0139: Connection restored to 3dca6c4f-deb0-65ec-37e1-a5a3805ebf53 (at 10.50.2.56@o2ib2)
[13926272.414355] Lustre: Skipped 861 previous similar messages
[13926872.133508] Lustre: oak-OST013f: Connection restored to e263d36e-2927-0254-d7ef-4e4a2eac5a5f (at 10.50.1.30@o2ib2)
[13926872.144131] Lustre: Skipped 913 previous similar messages
[13927470.687021] Lustre: oak-OST0121: Connection restored to 7bd5e25d-92e5-9f8c-22fa-8f4c98d09593 (at 10.50.10.6@o2ib2)
[13927470.697619] Lustre: Skipped 1173 previous similar messages
[13927623.034996] Lustre: oak-OST011b: Client e01e4c82-9fd9-5cc5-61dd-b09c7945df2b (at 10.210.12.11@tcp1) reconnecting
[13927623.045406] Lustre: Skipped 4 previous similar messages
[13927623.455231] LustreError: 253938:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91a99d74f050 x1714947727136000/t0(0) o4->e01e4c82-9fd9-5cc5-61dd-b09c7945df2b@10.210.12.11@tcp1:531/0 lens 488/448 e 0 to 0 dl 1645153836 ref 1 fl Interpret:/0/0 rc 0/0
[13927623.481012] Lustre: oak-OST011b: Bulk IO write error with e01e4c82-9fd9-5cc5-61dd-b09c7945df2b (at 10.210.12.11@tcp1), client will retry: rc = -110
[13927623.495068] Lustre: Skipped 5 previous similar messages
[13927623.990929] LustreError: 160917:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9196fc268850 x1714947727136640/t0(0) o4->e01e4c82-9fd9-5cc5-61dd-b09c7945df2b@10.210.12.11@tcp1:532/0 lens 488/448 e 0 to 0 dl 1645153837 ref 1 fl Interpret:/0/0 rc 0/0
[13927624.015360] LustreError: 160917:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[13927624.025385] Lustre: oak-OST011b: Bulk IO write error with e01e4c82-9fd9-5cc5-61dd-b09c7945df2b (at 10.210.12.11@tcp1), client will retry: rc = -110
[13927624.025520] LustreError: 160948:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff91e081c90050 x1714947727136640/t0(0) o4->e01e4c82-9fd9-5cc5-61dd-b09c7945df2b@10.210.12.11@tcp1:534/0 lens 488/448 e 0 to 0 dl 1645153839 ref 1 fl Interpret:/2/0 rc 0/0
[13927624.063610] Lustre: Skipped 3 previous similar messages
[13927625.024417] LustreError: 127357:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91974353a850 x1714947727136640/t0(0) o4->e01e4c82-9fd9-5cc5-61dd-b09c7945df2b@10.210.12.11@tcp1:535/0 lens 488/448 e 0 to 0 dl 1645153840 ref 1 fl Interpret:/2/0 rc 0/0
[13927625.025762] Lustre: oak-OST011b: Bulk IO write error with e01e4c82-9fd9-5cc5-61dd-b09c7945df2b (at 10.210.12.11@tcp1), client will retry: rc = -110
[13927625.025763] Lustre: Skipped 1 previous similar message
[13927625.067651] LustreError: 127357:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 3 previous similar messages
[13927793.928952] Lustre: oak-OST0149: Client e01e4c82-9fd9-5cc5-61dd-b09c7945df2b (at 10.210.12.11@tcp1) reconnecting
[13927793.939445] Lustre: Skipped 2 previous similar messages
[13927803.244296] LustreError: 160936:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff91cfd3064850 x1714947727795712/t0(0) o3->e01e4c82-9fd9-5cc5-61dd-b09c7945df2b@10.210.12.11@tcp1:712/0 lens 488/440 e 0 to 0 dl 1645154017 ref 1 fl Interpret:/0/0 rc 0/0
[13927803.268706] Lustre: oak-OST0131: Bulk IO read error with e01e4c82-9fd9-5cc5-61dd-b09c7945df2b (at 10.210.12.11@tcp1), client will retry: rc -110
[13928070.177627] Lustre: oak-OST0119: Connection restored to e3150863-7432-096b-85ab-79ff68231330 (at 10.51.1.24@o2ib3)
[13928070.188206] Lustre: Skipped 1598 previous similar messages
[13928668.800629] Lustre: oak-OST012b: Connection restored to  (at 10.50.7.2@o2ib2)
[13928668.808022] Lustre: Skipped 1195 previous similar messages
[13929268.413539] Lustre: oak-OST0149: Connection restored to 5dbda705-a67c-ce47-2e21-0baa3337235a (at 10.50.3.28@o2ib2)
[13929268.424115] Lustre: Skipped 1342 previous similar messages
[13929867.069017] Lustre: oak-OST0111: Connection restored to 097ece3c-9f18-7c64-d1a4-979765e8510b (at 10.0.3.24@o2ib5)
[13929867.079972] Lustre: Skipped 1094 previous similar messages
[13930467.309229] Lustre: oak-OST0143: Connection restored to 07e17a08-ae81-f8f4-7758-1592083eea88 (at 10.50.3.20@o2ib2)
[13930467.319821] Lustre: Skipped 1286 previous similar messages
[13930724.406302] LustreError: 21591:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(1048576)  req@ffff915a327ac850 x1714947749135808/t0(0) o3->e01e4c82-9fd9-5cc5-61dd-b09c7945df2b@10.210.12.11@tcp1:566/0 lens 488/440 e 0 to 0 dl 1645156891 ref 1 fl Interpret:/0/0 rc 0/0
[13930724.406391] Lustre: oak-OST014d: Bulk IO read error with e01e4c82-9fd9-5cc5-61dd-b09c7945df2b (at 10.210.12.11@tcp1), client will retry: rc -110
[13930724.444471] LustreError: 21591:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 2 previous similar messages
[13930848.985124] Lustre: oak-OST0131: Client e01e4c82-9fd9-5cc5-61dd-b09c7945df2b (at 10.210.12.11@tcp1) reconnecting
[13930848.995550] Lustre: Skipped 21 previous similar messages
[13930853.913941] Lustre: oak-OST012b: Client 1b0e0de8-4056-d02c-bb09-33adedaa9f96 (at 10.210.12.45@tcp1) reconnecting
[13930853.924432] Lustre: Skipped 28 previous similar messages
[13930904.604194] Lustre: oak-OST0123: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[13930904.614641] Lustre: Skipped 9 previous similar messages
[13930973.283542] Lustre: oak-OST011d: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[13930973.293952] Lustre: Skipped 10 previous similar messages
[13931029.109615] Lustre: oak-OST0113: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[13931029.120044] Lustre: Skipped 10 previous similar messages
[13931066.010460] Lustre: oak-OST011f: Connection restored to ef8da979-3077-9708-0c8c-a646245f23fe (at 10.50.13.15@o2ib2)
[13931066.021145] Lustre: Skipped 1263 previous similar messages
[13931127.471727] Lustre: oak-OST0111: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[13931287.850234] Lustre: oak-OST013d: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[13931287.860688] Lustre: Skipped 4 previous similar messages
[13931672.811777] Lustre: oak-OST0119: Connection restored to  (at 10.51.0.67@o2ib3)
[13931672.819264] Lustre: Skipped 879 previous similar messages
[13932272.075588] Lustre: oak-OST0115: Connection restored to f1db0cb0-5cee-ccf9-6484-5189f751ad99 (at 10.51.0.63@o2ib3)
[13932272.086206] Lustre: Skipped 971 previous similar messages
[13932352.002824] LustreError: 160910:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 1048576(2572288)  req@ffff91c887f92850 x1714971509814656/t0(0) o4->5afa23c2-1fd5-9030-38ce-b03a359082fe@10.210.12.75@tcp1:690/0 lens 488/448 e 0 to 0 dl 1645158525 ref 1 fl Interpret:/0/0 rc 0/0
[13932352.028866] Lustre: oak-OST011f: Bulk IO write error with 5afa23c2-1fd5-9030-38ce-b03a359082fe (at 10.210.12.75@tcp1), client will retry: rc = -110
[13932352.042339] Lustre: Skipped 1 previous similar message
[13932382.002245] LustreError: 137-5: oak-OST014c_UUID: not available for connect from 10.210.12.75@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13932382.028483] Lustre: oak-OST011f: Client 5afa23c2-1fd5-9030-38ce-b03a359082fe (at 10.210.12.75@tcp1) reconnecting
[13932382.038891] Lustre: Skipped 1 previous similar message
[13932459.195405] Lustre: oak-OST0115: Client 1b0e0de8-4056-d02c-bb09-33adedaa9f96 (at 10.210.12.45@tcp1) reconnecting
[13932459.206330] Lustre: Skipped 2 previous similar messages
[13932871.550691] Lustre: oak-OST011f: Connection restored to b9818893-1835-c766-e85c-4a4a4926273e (at 10.50.2.53@o2ib2)
[13932871.561285] Lustre: Skipped 1046 previous similar messages
[13933327.471169] Lustre: oak-OST013b: haven't heard from client d03b6970-81df-750c-1bbc-b9cb89d0d1c1 (at 10.50.13.1@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9196d5d0c800, cur 1645159460 expire 1645159310 last 1645159233
[13933327.493090] Lustre: Skipped 2 previous similar messages
[13933333.447696] Lustre: oak-OST0139: haven't heard from client d03b6970-81df-750c-1bbc-b9cb89d0d1c1 (at 10.50.13.1@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9180c5238800, cur 1645159466 expire 1645159316 last 1645159239
[13933333.469669] Lustre: Skipped 1 previous similar message
[13933473.921328] Lustre: oak-OST0129: Connection restored to b9694c0e-9fc2-f14e-33c8-9018d9fc9806 (at 10.210.12.65@tcp1)
[13933473.932005] Lustre: Skipped 895 previous similar messages
[13934072.850074] Lustre: oak-OST011d: Connection restored to 5070a18e-9288-0635-4199-1a1f5fbb081a (at 10.50.4.64@o2ib2)
[13934072.860672] Lustre: Skipped 944 previous similar messages
[13934672.193772] Lustre: oak-OST0111: Connection restored to eea359e6-6b89-c326-60af-8cb99076065b (at 10.210.12.8@tcp1)
[13934672.204390] Lustre: Skipped 929 previous similar messages
[13935275.804705] Lustre: oak-OST0117: Connection restored to fb4c352d-d5d8-df67-aa5f-fdf2fea140a1 (at 10.50.7.54@o2ib2)
[13935275.815291] Lustre: Skipped 963 previous similar messages
[13935875.019268] Lustre: oak-OST0117: Connection restored to  (at 10.51.14.21@o2ib3)
[13935875.026837] Lustre: Skipped 1122 previous similar messages
[13936473.646388] Lustre: oak-OST0119: Connection restored to  (at 10.50.17.9@o2ib2)
[13936473.653876] Lustre: Skipped 1422 previous similar messages
[13937074.399734] Lustre: oak-OST0111: Connection restored to eea359e6-6b89-c326-60af-8cb99076065b (at 10.210.12.8@tcp1)
[13937074.410312] Lustre: Skipped 1368 previous similar messages
[13937288.912112] Lustre: oak-OST0115: haven't heard from client 79b7e747-509a-4359-63d2-1e30858e6dbd (at 10.51.15.8@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91c2efc39000, cur 1645163431 expire 1645163281 last 1645163204
[13937329.780597] Lustre: oak-OST011b: haven't heard from client 79b7e747-509a-4359-63d2-1e30858e6dbd (at 10.51.15.8@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91d804a2fc00, cur 1645163472 expire 1645163322 last 1645163245
[13937676.796431] Lustre: oak-OST0111: Connection restored to eea359e6-6b89-c326-60af-8cb99076065b (at 10.210.12.8@tcp1)
[13937676.807030] Lustre: Skipped 1244 previous similar messages
[13938277.526812] Lustre: oak-OST0111: Connection restored to a4bb6e92-53d3-62cd-283e-9cd0fb55adf8 (at 10.50.17.38@o2ib2)
[13938277.537490] Lustre: Skipped 1221 previous similar messages
[13938876.110339] Lustre: oak-OST0143: Connection restored to 566692d9-3760-eb93-5cbc-1337c48c4638 (at 10.50.16.17@o2ib2)
[13938876.121073] Lustre: Skipped 1153 previous similar messages
[13939476.817167] Lustre: oak-OST0111: Connection restored to 9eb8189a-3985-f41c-7d7b-51af46681d5f (at 10.0.3.29@o2ib5)
[13939476.827716] Lustre: Skipped 1697 previous similar messages
[13940076.060571] Lustre: oak-OST0135: Connection restored to b072e58c-a9ad-befd-5eb6-8daa35f00cce (at 10.50.2.47@o2ib2)
[13940076.071171] Lustre: Skipped 1364 previous similar messages
[13940655.694614] Lustre: oak-OST0111: haven't heard from client 17f2e6b2-2423-a7c6-e8db-6f83830552f2 (at 10.51.6.2@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91a14c666400, cur 1645166806 expire 1645166656 last 1645166579
[13940675.156898] Lustre: oak-OST012f: Connection restored to 1e852f06-c801-0c5b-d0d3-f3e45937e428 (at 10.51.5.27@o2ib3)
[13940675.167481] Lustre: Skipped 1242 previous similar messages
[13941274.537159] Lustre: oak-OST0123: Connection restored to  (at 10.51.6.17@o2ib3)
[13941274.544624] Lustre: Skipped 1381 previous similar messages
[13941874.268861] Lustre: oak-OST0149: Connection restored to 37ba0792-be00-fd72-c115-6a10b5883dc6 (at 10.51.13.1@o2ib3)
[13941874.279450] Lustre: Skipped 1148 previous similar messages
[13942473.862573] Lustre: oak-OST0139: Connection restored to d4739e2d-918f-145e-2abc-95d78ff9e7e5 (at 10.50.4.49@o2ib2)
[13942473.873182] Lustre: Skipped 1446 previous similar messages
[13943074.376614] Lustre: oak-OST0137: Connection restored to bc7396f6-fbb0-5e6f-3bd8-bf31cb05de7c (at 10.50.5.55@o2ib2)
[13943074.388302] Lustre: Skipped 1143 previous similar messages
[13943679.150005] Lustre: oak-OST0125: Connection restored to 17df2ba7-0fc1-f30d-a343-f3481405b870 (at 10.50.2.71@o2ib2)
[13943679.161845] Lustre: Skipped 972 previous similar messages
[13944278.908279] Lustre: oak-OST0123: Connection restored to  (at 10.50.13.8@o2ib2)
[13944278.915770] Lustre: Skipped 1087 previous similar messages
[13944880.433642] Lustre: oak-OST011d: Connection restored to e6950155-2c5c-997a-576a-8841a6399e6e (at 10.50.1.35@o2ib2)
[13944880.444231] Lustre: Skipped 733 previous similar messages
[13945057.953369] Lustre: oak-OST012f: haven't heard from client 1f5496a9-eb1a-a833-91d8-57893d7b34a3 (at 10.51.12.20@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff914ebce23400, cur 1645171219 expire 1645171069 last 1645170992
[13945061.946120] Lustre: oak-OST013d: haven't heard from client 1f5496a9-eb1a-a833-91d8-57893d7b34a3 (at 10.51.12.20@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91ebe6341400, cur 1645171223 expire 1645171073 last 1645170996
[13945237.515843] Lustre: oak-OST0139: haven't heard from client 1f5496a9-eb1a-a833-91d8-57893d7b34a3 (at 10.51.12.20@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91c67f36a800, cur 1645171399 expire 1645171249 last 1645171172
[13945480.152248] Lustre: oak-OST011d: Connection restored to  (at 10.50.0.64@o2ib2)
[13945480.159743] Lustre: Skipped 724 previous similar messages
[13945717.345328] Lustre: oak-OST012f: haven't heard from client 1f5496a9-eb1a-a833-91d8-57893d7b34a3 (at 10.51.12.20@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91ef2ccae400, cur 1645171880 expire 1645171730 last 1645171653
[13946068.533088] Lustre: oak-OST012f: haven't heard from client 1f5496a9-eb1a-a833-91d8-57893d7b34a3 (at 10.51.12.20@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91c327c01000, cur 1645172232 expire 1645172082 last 1645172005
[13946080.933918] Lustre: oak-OST0111: Connection restored to 9e481091-2c29-0032-be39-272874617a20 (at 10.50.9.52@o2ib2)
[13946080.944503] Lustre: Skipped 745 previous similar messages
[13946254.042114] Lustre: oak-OST0139: haven't heard from client 1f5496a9-eb1a-a833-91d8-57893d7b34a3 (at 10.51.12.20@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91b36cd32c00, cur 1645172418 expire 1645172268 last 1645172191
[13946679.954666] Lustre: oak-OST0125: Connection restored to a1771d7a-63d0-8cd8-9e63-2128188e0783 (at 10.50.9.42@o2ib2)
[13946679.965271] Lustre: Skipped 585 previous similar messages
[13947278.566047] Lustre: oak-OST011b: Connection restored to b2007b39-9285-7818-253f-56a583205455 (at 10.51.6.1@o2ib3)
[13947278.576571] Lustre: Skipped 523 previous similar messages
[13947880.447201] Lustre: oak-OST0111: Connection restored to eea359e6-6b89-c326-60af-8cb99076065b (at 10.210.12.8@tcp1)
[13947880.457818] Lustre: Skipped 646 previous similar messages
[13948479.252469] Lustre: oak-OST014d: Connection restored to 6f889484-d12f-e0ed-ab81-999d8df527db (at 10.51.16.18@o2ib3)
[13948479.263141] Lustre: Skipped 813 previous similar messages
[13949079.093085] Lustre: oak-OST0139: Connection restored to 7c2cd166-8a6a-c25d-706d-42283e0eb91d (at 10.51.2.50@o2ib3)
[13949079.103681] Lustre: Skipped 803 previous similar messages
[13949085.183290] Lustre: oak-OST0113: haven't heard from client 0c31559a-a934-17d6-9101-a92566c818e8 (at 10.51.7.15@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91d91dfb6400, cur 1645175256 expire 1645175106 last 1645175029
[13949085.205252] Lustre: Skipped 1 previous similar message
[13949677.826724] Lustre: oak-OST014d: Connection restored to b2007b39-9285-7818-253f-56a583205455 (at 10.51.6.1@o2ib3)
[13949677.837223] Lustre: Skipped 895 previous similar messages
[13950037.868913] Lustre: oak-OST0113: haven't heard from client 0c31559a-a934-17d6-9101-a92566c818e8 (at 10.51.7.15@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff913d84d41400, cur 1645176211 expire 1645176061 last 1645175984
[13950281.059690] Lustre: oak-OST012b: Connection restored to cd354002-287c-320d-4786-ef84b940faf7 (at 10.50.9.50@o2ib2)
[13950281.070270] Lustre: Skipped 1004 previous similar messages
[13950879.758250] Lustre: oak-OST011d: Connection restored to f26da41f-d7e6-58ad-6ba0-1dc28c5d9f57 (at 10.50.6.70@o2ib2)
[13950879.768840] Lustre: Skipped 715 previous similar messages
[13951234.955275] Lustre: oak-OST0113: haven't heard from client 0c31559a-a934-17d6-9101-a92566c818e8 (at 10.51.7.15@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91d5c0f04c00, cur 1645177411 expire 1645177261 last 1645177184
[13951480.909249] Lustre: oak-OST013b: Connection restored to eb3e302a-b357-21b5-7d72-6890b8fc150f (at 10.50.2.72@o2ib2)
[13951480.919832] Lustre: Skipped 780 previous similar messages
[13952080.825517] Lustre: oak-OST0121: Connection restored to f836dd20-d3cf-0429-a256-7007d47ef375 (at 10.50.9.56@o2ib2)
[13952080.836162] Lustre: Skipped 1153 previous similar messages
[13952432.029395] Lustre: oak-OST0113: haven't heard from client 0c31559a-a934-17d6-9101-a92566c818e8 (at 10.51.7.15@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91ea3fd41800, cur 1645178611 expire 1645178461 last 1645178384
[13952680.375016] Lustre: oak-OST013f: Connection restored to f720ccf2-9c93-97b1-7979-3042158d6592 (at 10.50.17.39@o2ib2)
[13952680.385688] Lustre: Skipped 1278 previous similar messages
[13953281.627562] Lustre: oak-OST014b: Connection restored to  (at 10.51.15.13@o2ib3)
[13953281.635127] Lustre: Skipped 1164 previous similar messages
[13953881.575461] Lustre: oak-OST0135: Connection restored to b9818893-1835-c766-e85c-4a4a4926273e (at 10.50.2.53@o2ib2)
[13953881.586039] Lustre: Skipped 1125 previous similar messages
[13954481.554168] Lustre: oak-OST0121: Connection restored to 938e4fcb-40fa-8576-5696-2871684d71bd (at 10.50.5.8@o2ib2)
[13954481.564679] Lustre: Skipped 926 previous similar messages
[13955081.833465] Lustre: oak-OST011b: Connection restored to f71d1062-f20c-92ad-d3d7-66389b8ec719 (at 10.51.12.13@o2ib3)
[13955081.844138] Lustre: Skipped 695 previous similar messages
[13955685.441159] Lustre: oak-OST0127: Connection restored to  (at 10.50.10.57@o2ib2)
[13955685.448716] Lustre: Skipped 691 previous similar messages
[13956284.082158] Lustre: oak-OST0143: Connection restored to 71bc0f4c-3bf2-f6e2-a555-cc123604d10e (at 10.50.16.12@o2ib2)
[13956284.092828] Lustre: Skipped 672 previous similar messages
[13956883.376224] Lustre: oak-OST014d: Connection restored to a5a2e0f5-0864-9d95-c2c5-f87c49df0a6e (at 10.51.1.59@o2ib3)
[13956883.387181] Lustre: Skipped 1305 previous similar messages
[13957482.895859] Lustre: oak-OST0141: Connection restored to  (at 10.51.1.17@o2ib3)
[13957482.903389] Lustre: Skipped 1257 previous similar messages
[13958082.723968] Lustre: oak-OST0129: Connection restored to 1e354a56-47c1-6143-541f-2368817925be (at 10.50.17.40@o2ib2)
[13958082.734648] Lustre: Skipped 1265 previous similar messages
[13958682.354305] Lustre: oak-OST0119: Connection restored to 06c1c22b-e4ac-81cb-ef54-4744df1b23e1 (at 10.51.7.12@o2ib3)
[13958682.364912] Lustre: Skipped 1109 previous similar messages
[13959284.554578] Lustre: oak-OST013d: Connection restored to  (at 10.51.16.20@o2ib3)
[13959284.562135] Lustre: Skipped 832 previous similar messages
[13959873.772622] Lustre: 205025:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1645185942/real 1645185942]  req@ffff9151a00e6780 x1710530999064896/t0(0) o106->oak-OST0129@10.51.13.24@o2ib3:15/16 lens 296/280 e 0 to 1 dl 1645186070 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
[13959878.928518] Lustre: oak-OST0129: haven't heard from client 45a3cd79-3161-f171-3496-b7ae23c7a6b0 (at 10.51.13.24@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91ce3f64a800, cur 1645186076 expire 1645185926 last 1645185849
[13959884.391757] Lustre: oak-OST0131: Connection restored to 080fbf36-50e2-e037-d579-97f7ea4dd6a6 (at 10.210.12.51@tcp1)
[13959884.402423] Lustre: Skipped 764 previous similar messages
[13960483.305881] Lustre: oak-OST0111: Connection restored to eea359e6-6b89-c326-60af-8cb99076065b (at 10.210.12.8@tcp1)
[13960483.316496] Lustre: Skipped 746 previous similar messages
[13961083.704100] Lustre: oak-OST013d: Connection restored to 2cfbbad6-c7e0-3657-a51c-256f64a6bd0a (at 10.51.13.5@o2ib3)
[13961083.714682] Lustre: Skipped 800 previous similar messages
[13961687.418290] Lustre: oak-OST012d: Connection restored to de334834-ada5-7101-0af8-a85e255c7a80 (at 10.50.7.31@o2ib2)
[13961687.428907] Lustre: Skipped 806 previous similar messages
[13962287.418656] Lustre: oak-OST0123: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[13962287.429410] Lustre: Skipped 714 previous similar messages
[13962887.166007] Lustre: oak-OST012f: Connection restored to  (at 10.50.9.37@o2ib2)
[13962887.173473] Lustre: Skipped 663 previous similar messages
[13963486.913924] Lustre: oak-OST012d: Connection restored to 894b9462-4ef1-1bb0-6365-60255fe14a6d (at 10.50.17.24@o2ib2)
[13963486.924611] Lustre: Skipped 788 previous similar messages
[13964087.330802] Lustre: oak-OST0145: Connection restored to 83f61e92-4c9d-b1fe-42f0-577e2587d888 (at 10.50.7.39@o2ib2)
[13964087.341400] Lustre: Skipped 624 previous similar messages
[13964699.764878] Lustre: oak-OST012d: Connection restored to 776400dd-7447-858b-13ed-29f579e7c804 (at 10.50.7.32@o2ib2)
[13964699.775555] Lustre: Skipped 900 previous similar messages
[13965298.500028] Lustre: oak-OST014d: Connection restored to 5c387f56-f0fa-847c-11e4-cfd988587244 (at 10.50.8.56@o2ib2)
[13965298.510659] Lustre: Skipped 621 previous similar messages
[13965899.459547] Lustre: oak-OST0111: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[13965899.470228] Lustre: Skipped 640 previous similar messages
[13966507.194225] Lustre: oak-OST0133: Connection restored to  (at 10.50.6.11@o2ib2)
[13966507.201710] Lustre: Skipped 620 previous similar messages
[13967107.311773] Lustre: oak-OST0117: Connection restored to 72eaf2df-907e-10db-7344-e8188bb48d82 (at 10.50.9.12@o2ib2)
[13967107.322589] Lustre: Skipped 719 previous similar messages
[13967707.699476] Lustre: oak-OST0141: Connection restored to caf0a0c8-099f-6745-58da-10c59ae04aaf (at 10.50.2.60@o2ib2)
[13967707.710075] Lustre: Skipped 784 previous similar messages
[13968306.466047] Lustre: oak-OST0137: Connection restored to a256e27b-c1dc-81ef-4609-ebeba7c6bf96 (at 10.51.4.43@o2ib3)
[13968306.476668] Lustre: Skipped 858 previous similar messages
[13968913.035934] Lustre: oak-OST0133: Connection restored to 65e390b1-4e5c-22f3-a78e-a3edf0132ddb (at 10.50.2.63@o2ib2)
[13968913.046537] Lustre: Skipped 798 previous similar messages
[13969513.185796] Lustre: oak-OST0127: Connection restored to a9cebbbf-4465-65a4-bf00-99bae7741269 (at 10.51.13.2@o2ib3)
[13969513.196414] Lustre: Skipped 756 previous similar messages
[13970111.961387] Lustre: oak-OST0139: Connection restored to ca2e46d5-bfa6-73b9-3298-0d35b5b278ca (at 10.50.16.9@o2ib2)
[13970111.971971] Lustre: Skipped 732 previous similar messages
[13970718.262283] Lustre: oak-OST0143: Connection restored to f720ccf2-9c93-97b1-7979-3042158d6592 (at 10.50.17.39@o2ib2)
[13970718.273025] Lustre: Skipped 686 previous similar messages
[13971319.835480] Lustre: oak-OST0143: Connection restored to  (at 10.50.4.8@o2ib2)
[13971319.842882] Lustre: Skipped 713 previous similar messages
[13971918.737346] Lustre: oak-OST014b: Connection restored to 4881b494-d9fe-4c3e-0129-4f23331562aa (at 10.50.7.29@o2ib2)
[13971918.747929] Lustre: Skipped 668 previous similar messages
[13972519.270827] Lustre: oak-OST013f: Connection restored to 7d80b8da-c6df-787b-485b-160cbf76be2d (at 10.51.2.16@o2ib3)
[13972519.281418] Lustre: Skipped 689 previous similar messages
[13973119.439087] Lustre: oak-OST0115: Connection restored to 3a57957d-0251-b0e1-832e-0703d2b195aa (at 10.50.1.40@o2ib2)
[13973119.449679] Lustre: Skipped 609 previous similar messages
[13973603.989604] LustreError: 160893:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91eacd625850 x1715068884242048/t0(0) o4->bb55a01e-02a3-cbba-5b76-e478fd40967c@10.210.12.54@tcp1:506/0 lens 488/448 e 0 to 0 dl 1645199866 ref 1 fl Interpret:/0/0 rc 0/0
[13973604.015609] Lustre: oak-OST0139: Bulk IO write error with bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1), client will retry: rc = -110
[13973623.644209] LustreError: 137-5: oak-OST0132_UUID: not available for connect from 10.210.12.54@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13973625.556948] Lustre: oak-OST0139: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[13973625.567364] Lustre: Skipped 58 previous similar messages
[13973716.059756] Lustre: oak-OST0111: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[13973716.070173] Lustre: Skipped 1 previous similar message
[13973720.166918] Lustre: oak-OST0133: Connection restored to 83034250-f855-6c12-471e-38d9aeacf85d (at 10.210.12.54@tcp1)
[13973720.177595] Lustre: Skipped 888 previous similar messages
[13973746.413507] Lustre: oak-OST014b: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[13973746.423929] Lustre: Skipped 10 previous similar messages
[13973806.566836] Lustre: oak-OST0137: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[13973806.577329] Lustre: Skipped 9 previous similar messages
[13973897.190960] Lustre: oak-OST013d: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[13973897.201366] Lustre: Skipped 3 previous similar messages
[13974320.372201] Lustre: oak-OST011b: Connection restored to 8fbf6eea-4385-fb2a-e1ba-f6078d1935b3 (at 10.50.2.16@o2ib2)
[13974320.382780] Lustre: Skipped 597 previous similar messages
[13974920.241669] Lustre: oak-OST0139: Connection restored to 86eee6bc-6017-fb1a-3873-1027e48995f7 (at 10.50.12.3@o2ib2)
[13974920.252257] Lustre: Skipped 619 previous similar messages
[13975519.509929] Lustre: oak-OST0131: Connection restored to 127c1687-4990-e583-6728-a72d15d0a29e (at 10.50.7.47@o2ib2)
[13975519.520522] Lustre: Skipped 593 previous similar messages
[13976119.341925] Lustre: oak-OST013f: Connection restored to ebf178f6-1771-c187-ac93-409752d2ac2a (at 10.50.9.38@o2ib2)
[13976119.352507] Lustre: Skipped 537 previous similar messages
[13976717.924419] Lustre: oak-OST014d: Connection restored to 98bd9aab-2cbd-95a4-4e29-43c4c3e1c3df (at 10.51.7.11@o2ib3)
[13976717.935000] Lustre: Skipped 557 previous similar messages
[13977319.008538] Lustre: oak-OST013f: Connection restored to 74a465f8-66d2-a778-f17d-812bb5379207 (at 10.51.0.64@o2ib3)
[13977319.019154] Lustre: Skipped 729 previous similar messages
[13977920.266007] Lustre: oak-OST0133: Connection restored to 0c1a4fb6-e070-71be-72de-9304d6a54c74 (at 10.0.3.51@o2ib5)
[13977920.276844] Lustre: Skipped 579 previous similar messages
[13978519.454030] Lustre: oak-OST012b: Connection restored to 7546b7b6-8c46-0689-6dde-6d667c01fcfa (at 10.51.5.56@o2ib3)
[13978519.464637] Lustre: Skipped 626 previous similar messages
[13978613.486111] Lustre: oak-OST0145: Client 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1) reconnecting
[13978613.496528] Lustre: Skipped 3 previous similar messages
[13978613.648871] LustreError: 244098:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91a1bfd87850 x1715761200025728/t0(0) o4->4b39aac6-2dc7-e48f-9434-e3bfde492453@10.210.12.29@tcp1:303/0 lens 488/448 e 0 to 0 dl 1645204948 ref 1 fl Interpret:/0/0 rc 0/0
[13978613.675825] Lustre: oak-OST0145: Bulk IO write error with 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1), client will retry: rc = -110
[13978614.359153] LustreError: 243534:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff918f71db4050 x1714973978014848/t0(0) o4->5d4d5527-a52f-bc68-278b-aa990b8609d6@10.210.12.59@tcp1:303/0 lens 488/448 e 0 to 0 dl 1645204948 ref 1 fl Interpret:/0/0 rc 0/0
[13978614.383868] Lustre: oak-OST012b: Bulk IO write error with 5d4d5527-a52f-bc68-278b-aa990b8609d6 (at 10.210.12.59@tcp1), client will retry: rc = -110
[13978615.146280] LustreError: 26433:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff9136f53ac000
[13978615.157299] LustreError: 26433:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff9136f53ac000
[13978615.168335] LustreError: 26433:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff9136f53ac000
[13978615.179354] LustreError: 26435:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff9136f53ac000
[13978615.379661] LustreError: 199274:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91c193af2850 x1715185542372992/t0(0) o4->7189afb8-8fe5-7418-d14a-4a60d24330d0@10.210.12.48@tcp1:306/0 lens 488/448 e 0 to 0 dl 1645204951 ref 1 fl Interpret:/0/0 rc 0/0
[13978615.404144] LustreError: 199274:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 8 previous similar messages
[13978615.414237] Lustre: oak-OST0113: Bulk IO write error with 7189afb8-8fe5-7418-d14a-4a60d24330d0 (at 10.210.12.48@tcp1), client will retry: rc = -110
[13978615.427668] Lustre: Skipped 8 previous similar messages
[13978680.004649] LustreError: 162712:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 3145728(4194304)  req@ffff9136383bf050 x1715761200026240/t0(0) o3->4b39aac6-2dc7-e48f-9434-e3bfde492453@10.210.12.29@tcp1:303/0 lens 488/440 e 0 to 0 dl 1645204948 ref 1 fl Interpret:/0/0 rc 0/0
[13978680.004873] Lustre: oak-OST0121: Bulk IO read error with 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1), client will retry: rc -110
[13978680.004874] Lustre: Skipped 2 previous similar messages
[13978680.048997] LustreError: 162712:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 4 previous similar messages
[13978780.359422] Lustre: oak-OST0131: Client 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1) reconnecting
[13978780.369847] Lustre: Skipped 9 previous similar messages
[13978850.939435] Lustre: oak-OST0125: Client 5d4d5527-a52f-bc68-278b-aa990b8609d6 (at 10.210.12.59@tcp1) reconnecting
[13978850.949945] Lustre: Skipped 20 previous similar messages
[13979063.789663] Lustre: oak-OST014d: Client 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1) reconnecting
[13979063.800099] Lustre: Skipped 8 previous similar messages
[13979125.579542] Lustre: oak-OST0125: Connection restored to  (at 10.51.4.1@o2ib3)
[13979125.586947] Lustre: Skipped 736 previous similar messages
[13979725.477791] Lustre: oak-OST0149: Connection restored to 7e7e9ac5-0ff6-2601-4894-f85117745486 (at 10.50.9.49@o2ib2)
[13979725.488376] Lustre: Skipped 575 previous similar messages
[13979810.559593] Lustre: oak-OST0133: Client dc851ef6-3898-c01c-adc6-dc548a499d03 (at 10.210.12.53@tcp1) reconnecting
[13979810.570024] Lustre: Skipped 1 previous similar message
[13979810.704677] LustreError: 228404:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91bc763cd850 x1715062138127680/t0(0) o4->dc851ef6-3898-c01c-adc6-dc548a499d03@10.210.12.53@tcp1:752/0 lens 488/448 e 0 to 0 dl 1645206152 ref 1 fl Interpret:/0/0 rc 0/0
[13979810.729096] LustreError: 228404:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 6 previous similar messages
[13979810.739052] Lustre: oak-OST0133: Bulk IO write error with dc851ef6-3898-c01c-adc6-dc548a499d03 (at 10.210.12.53@tcp1), client will retry: rc = -110
[13979810.752489] Lustre: Skipped 6 previous similar messages
[13979876.531611] LustreError: 21623:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91ef0d6bd850 x1715062138109056/t0(0) o4->dc851ef6-3898-c01c-adc6-dc548a499d03@10.210.12.53@tcp1:751/0 lens 488/448 e 0 to 0 dl 1645206151 ref 1 fl Interpret:/0/0 rc 0/0
[13979876.557474] Lustre: oak-OST0113: Bulk IO write error with dc851ef6-3898-c01c-adc6-dc548a499d03 (at 10.210.12.53@tcp1), client will retry: rc = -110
[13979876.570917] Lustre: Skipped 1 previous similar message
[13980051.381266] LustreError: 21623:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[13980133.575811] Lustre: oak-OST0119: Client dc851ef6-3898-c01c-adc6-dc548a499d03 (at 10.210.12.53@tcp1) reconnecting
[13980133.586250] Lustre: Skipped 52 previous similar messages
[13980324.112801] Lustre: oak-OST012f: Connection restored to f71d1062-f20c-92ad-d3d7-66389b8ec719 (at 10.51.12.13@o2ib3)
[13980324.123596] Lustre: Skipped 751 previous similar messages
[13980477.876567] Lustre: oak-OST0141: haven't heard from client 0affa990-eba6-ec14-90ec-ba1760399674 (at 10.0.3.20@o2ib5) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9156e4849c00, cur 1645206725 expire 1645206575 last 1645206498
[13980690.791938] LustreError: 243451:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(68872)  req@ffff91a8bdd14050 x1723074973413824/t0(0) o4->13c35dd5-38f8-1ff0-e52d-3e3e4fcf74b0@10.50.13.15@o2ib2:33/0 lens 488/448 e 0 to 0 dl 1645206943 ref 1 fl Interpret:/0/0 rc 0/0
[13980690.791960] Lustre: oak-OST014d: Bulk IO write error with 13c35dd5-38f8-1ff0-e52d-3e3e4fcf74b0 (at 10.50.13.15@o2ib2), client will retry: rc = -110
[13980690.830433] LustreError: 243451:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[13980834.995980] Lustre: oak-OST014b: haven't heard from client 25e5ac36-319c-0bc3-104c-5873f2c998e3 (at 10.50.12.4@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9173ddd84400, cur 1645207083 expire 1645206933 last 1645206856
[13980835.017942] Lustre: Skipped 4 previous similar messages
[13980840.006154] Lustre: oak-OST011f: haven't heard from client 13c35dd5-38f8-1ff0-e52d-3e3e4fcf74b0 (at 10.50.13.15@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff917501e3c800, cur 1645207088 expire 1645206938 last 1645206861
[13980840.028208] Lustre: Skipped 45 previous similar messages
[13980924.306053] Lustre: oak-OST0111: Connection restored to 2a3ee281-af9e-ac0d-65d5-3e31f92ae00d (at 10.51.6.60@o2ib3)
[13980924.316631] Lustre: Skipped 1530 previous similar messages
[13981523.154774] Lustre: oak-OST013d: Connection restored to  (at 10.0.3.12@o2ib5)
[13981523.162157] Lustre: Skipped 1205 previous similar messages
[13982124.396843] Lustre: oak-OST014b: Connection restored to 81ce8b63-456a-872d-0643-7b6a62e875c2 (at 10.51.6.70@o2ib3)
[13982124.407440] Lustre: Skipped 990 previous similar messages
[13982726.791706] Lustre: oak-OST014b: Connection restored to b4873b44-db69-85c9-576f-945bfd0ad384 (at 10.0.3.26@o2ib5)
[13982726.802196] Lustre: Skipped 988 previous similar messages
[13983329.274018] Lustre: 243372:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1645209455/real 1645209455]  req@ffff91834f5d2880 x1710531042088704/t0(0) o106->oak-OST0139@10.51.7.14@o2ib3:15/16 lens 296/280 e 0 to 1 dl 1645209583 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
[13983329.525106] Lustre: oak-OST0123: Connection restored to 42228356-6056-ae58-809d-d255d66b2a5d (at 10.50.9.55@o2ib2)
[13983329.535817] Lustre: Skipped 1239 previous similar messages
[13983329.936135] Lustre: oak-OST0115: haven't heard from client 82662b4b-4947-e599-1ee8-e5ea7ec445d3 (at 10.51.15.21@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91e8ed8e2000, cur 1645209584 expire 1645209434 last 1645209357
[13983329.958178] Lustre: Skipped 1 previous similar message
[13983334.941749] Lustre: oak-OST0113: haven't heard from client 0c31559a-a934-17d6-9101-a92566c818e8 (at 10.51.7.15@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91d33b316000, cur 1645209589 expire 1645209439 last 1645209362
[13983334.963665] Lustre: Skipped 8 previous similar messages
[13983335.951588] Lustre: oak-OST0137: haven't heard from client 4b3cc144-1d72-dbc3-25e2-a2b4c3a22f7c (at 10.51.12.22@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91d7be697400, cur 1645209590 expire 1645209440 last 1645209363
[13983335.973617] Lustre: Skipped 1 previous similar message
[13983339.009920] Lustre: oak-OST0139: haven't heard from client e2bbe8eb-0c3e-5c92-fb40-986b1939f973 (at 10.51.7.14@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91d82afcd000, cur 1645209593 expire 1645209443 last 1645209366
[13983339.031857] Lustre: Skipped 3 previous similar messages
[13983411.758621] Lustre: oak-OST013b: haven't heard from client 0732115c-3965-6531-f21f-c76b6e6dbcea (at 10.51.12.24@o2ib3) in 220 seconds. I think it's dead, and I am evicting it. exp ffff91d7be696800, cur 1645209666 expire 1645209516 last 1645209446
[13983488.550261] Lustre: oak-OST014d: haven't heard from client bf733f5b-785f-579e-4173-34f1c4054a2c (at 10.51.15.22@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91b351173800, cur 1645209743 expire 1645209593 last 1645209516
[13983488.572255] Lustre: Skipped 8 previous similar messages
[13983713.999480] Lustre: oak-OST014b: haven't heard from client 1f5496a9-eb1a-a833-91d8-57893d7b34a3 (at 10.51.12.20@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91a445b11800, cur 1645209969 expire 1645209819 last 1645209742
[13983714.021477] Lustre: Skipped 3 previous similar messages
[13983930.661195] Lustre: oak-OST0145: Connection restored to 2d7b80e6-0f0d-a2ea-c50b-4f791ee979a6 (at 10.51.16.21@o2ib3)
[13983930.671872] Lustre: Skipped 1044 previous similar messages
[13984529.740567] Lustre: oak-OST0143: Connection restored to e6950155-2c5c-997a-576a-8841a6399e6e (at 10.50.1.35@o2ib2)
[13984529.751151] Lustre: Skipped 935 previous similar messages
[13985129.514727] Lustre: oak-OST013f: Connection restored to 121d79c7-b34f-dcf4-58e9-962c4afc9126 (at 10.50.8.36@o2ib2)
[13985129.525314] Lustre: Skipped 792 previous similar messages
[13985733.963255] Lustre: oak-OST013b: Connection restored to b2007b39-9285-7818-253f-56a583205455 (at 10.51.6.1@o2ib3)
[13985733.973769] Lustre: Skipped 1064 previous similar messages
[13985814.898063] Lustre: oak-OST012f: haven't heard from client 382c7b19-31d5-4112-975b-7583e2b55d17 (at 10.51.14.15@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91b63a1f3000, cur 1645212075 expire 1645211925 last 1645211848
[13985814.920057] Lustre: Skipped 6 previous similar messages
[13986332.936058] Lustre: oak-OST0141: Connection restored to  (at 10.51.4.6@o2ib3)
[13986332.943438] Lustre: Skipped 796 previous similar messages
[13986931.496143] Lustre: oak-OST011b: Connection restored to 8024f37e-9c14-411d-604f-49331bcb7e7a (at 10.50.9.72@o2ib2)
[13986931.506739] Lustre: Skipped 598 previous similar messages
[13987541.309896] Lustre: oak-OST014d: Connection restored to 8a175aee-31b6-e168-94c2-4812b701af81 (at 10.51.2.60@o2ib3)
[13987541.320479] Lustre: Skipped 713 previous similar messages
[13988140.461188] Lustre: oak-OST013f: Connection restored to 43b45cbd-14a3-0e54-7755-76513da7a096 (at 10.51.1.42@o2ib3)
[13988140.472533] Lustre: Skipped 2241 previous similar messages
[13988739.256126] Lustre: oak-OST0111: Connection restored to 080fbf36-50e2-e037-d579-97f7ea4dd6a6 (at 10.210.12.51@tcp1)
[13988739.266813] Lustre: Skipped 2414 previous similar messages
[13989338.746507] Lustre: oak-OST012f: Connection restored to dbb5e0a2-7f70-2066-fadd-3a0ac943f796 (at 10.51.5.61@o2ib3)
[13989338.757090] Lustre: Skipped 1071 previous similar messages
[13989938.246971] Lustre: oak-OST014d: Connection restored to 9dd85d36-fd39-cffc-4cf0-91d71c907eef (at 10.50.1.56@o2ib2)
[13989938.257606] Lustre: Skipped 960 previous similar messages
[13990537.524804] Lustre: oak-OST0147: Connection restored to f9253fe3-4f1d-7d5c-4c23-a08286c20d87 (at 10.50.6.15@o2ib2)
[13990537.535382] Lustre: Skipped 1443 previous similar messages
[13991138.516795] Lustre: oak-OST012b: Connection restored to 44205dca-68c8-d64e-c952-7cce1db00805 (at 10.50.17.13@o2ib2)
[13991138.527900] Lustre: Skipped 1103 previous similar messages
[13991737.069158] Lustre: oak-OST014b: Connection restored to 980ce7e0-df3a-7912-c3c6-485e181ecc98 (at 10.50.15.10@o2ib2)
[13991737.079821] Lustre: Skipped 875 previous similar messages
[13992335.615391] Lustre: oak-OST0147: Connection restored to bfd7c29e-205a-a45e-5add-4648092cd4c2 (at 10.50.6.39@o2ib2)
[13992335.625974] Lustre: Skipped 1795 previous similar messages
[13992940.263477] Lustre: oak-OST013d: Connection restored to aef304b4-8679-e0d5-9fa5-c78542ec1535 (at 10.50.2.25@o2ib2)
[13992940.274420] Lustre: Skipped 947 previous similar messages
[13993538.831024] Lustre: oak-OST0139: Connection restored to 3059db8e-b628-6326-b0b9-f61b3ef5610e (at 10.51.5.46@o2ib3)
[13993538.841622] Lustre: Skipped 1415 previous similar messages
[13993881.277496] Lustre: oak-OST012d: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[13993881.341354] LustreError: 160952:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91d1609f8050 x1714980874002624/t0(0) o4->d4faaef8-44e2-b05f-6116-d9c30b2e4e11@10.210.12.60@tcp1:509/0 lens 488/448 e 0 to 0 dl 1645220254 ref 1 fl Interpret:/0/0 rc 0/0
[13993881.365865] LustreError: 160952:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[13993881.376320] Lustre: oak-OST012d: Bulk IO write error with d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1), client will retry: rc = -110
[13993881.389903] Lustre: Skipped 1 previous similar message
[13993882.193287] LustreError: 127352:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91d1609fc050 x1714980874002048/t0(0) o4->d4faaef8-44e2-b05f-6116-d9c30b2e4e11@10.210.12.60@tcp1:508/0 lens 488/448 e 0 to 0 dl 1645220253 ref 1 fl Interpret:/0/0 rc 0/0
[13993882.217803] LustreError: 127352:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 2 previous similar messages
[13993882.227789] Lustre: oak-OST012d: Bulk IO write error with d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1), client will retry: rc = -110
[13993882.241227] Lustre: Skipped 2 previous similar messages
[13993883.874048] LustreError: 160936:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91ef239f7850 x1715014637262528/t0(0) o4->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:516/0 lens 488/448 e 0 to 0 dl 1645220261 ref 1 fl Interpret:/0/0 rc 0/0
[13993883.898823] Lustre: oak-OST013f: Bulk IO write error with c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1), client will retry: rc = -110
[13993930.955840] LustreError: 243497:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 3145728(4194304)  req@ffff9180c76f1050 x1714980874000832/t0(0) o3->d4faaef8-44e2-b05f-6116-d9c30b2e4e11@10.210.12.60@tcp1:508/0 lens 488/440 e 0 to 0 dl 1645220253 ref 1 fl Interpret:/0/0 rc 0/0
[13993930.981590] Lustre: oak-OST0137: Bulk IO read error with d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1), client will retry: rc -110
[13993930.994816] Lustre: Skipped 4 previous similar messages
[13993962.946027] LustreError: 137-5: oak-OST0140_UUID: not available for connect from 10.210.12.66@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13994056.478932] Lustre: oak-OST0131: Client 6f45695d-f173-ee1d-e6cb-d38dad7e0879 (at 10.210.12.74@tcp1) reconnecting
[13994056.489344] Lustre: Skipped 6 previous similar messages
[13994123.708282] LustreError: 228366:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91489f026850 x1715094151473280/t0(0) o4->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:754/0 lens 488/448 e 0 to 0 dl 1645220499 ref 1 fl Interpret:/0/0 rc 0/0
[13994123.713542] Lustre: oak-OST012b: Bulk IO write error with d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1), client will retry: rc = -110
[13994123.713543] Lustre: Skipped 8 previous similar messages
[13994123.762052] LustreError: 228366:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 9 previous similar messages
[13994128.182486] LustreError: 162674:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91907aafd850 x1715761398043776/t0(0) o4->4b39aac6-2dc7-e48f-9434-e3bfde492453@10.210.12.29@tcp1:0/0 lens 488/448 e 0 to 0 dl 1645220500 ref 1 fl Interpret:/0/0 rc 0/0
[13994128.206742] LustreError: 162674:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 13 previous similar messages
[13994128.217545] Lustre: oak-OST0111: Bulk IO write error with 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1), client will retry: rc = -110
[13994128.230978] Lustre: Skipped 15 previous similar messages
[13994128.415149] LustreError: 21594:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff918f1fd4f850 x1715761398045312/t0(0) o4->4b39aac6-2dc7-e48f-9434-e3bfde492453@10.210.12.29@tcp1:6/0 lens 488/448 e 0 to 0 dl 1645220506 ref 1 fl Interpret:/2/0 rc 0/0
[13994138.355122] Lustre: oak-OST0117: Connection restored to  (at 10.51.15.13@o2ib3)
[13994138.362676] Lustre: Skipped 1176 previous similar messages
[13994170.476754] LustreError: 162706:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff918ec3fc9050 x1715062282803584/t0(0) o4->dc851ef6-3898-c01c-adc6-dc548a499d03@10.210.12.53@tcp1:752/0 lens 488/448 e 0 to 0 dl 1645220497 ref 1 fl Interpret:/0/0 rc 0/0
[13994170.502871] Lustre: oak-OST0113: Bulk IO write error with dc851ef6-3898-c01c-adc6-dc548a499d03 (at 10.210.12.53@tcp1), client will retry: rc = -110
[13994170.516399] Lustre: Skipped 13 previous similar messages
[13994172.878774] LustreError: 162673:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91817e204850 x1715014641592512/t0(0) o4->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:48/0 lens 488/448 e 0 to 0 dl 1645220548 ref 1 fl Interpret:/0/0 rc 0/0
[13994172.903151] LustreError: 162673:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 12 previous similar messages
[13994173.492552] LustreError: 21610:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff9185aa8cf850 x1715014641580864/t0(0) o4->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:51/0 lens 488/448 e 0 to 0 dl 1645220551 ref 1 fl Interpret:/2/0 rc 0/0
[13994173.517908] LustreError: 21610:0:(ldlm_lib.c:3305:target_bulk_io()) Skipped 1 previous similar message
[13994202.756335] LustreError: 137-5: oak-OST014a_UUID: not available for connect from 10.210.12.43@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13994204.800590] LustreError: 137-5: oak-OST013c_UUID: not available for connect from 10.210.12.48@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13994204.818269] LustreError: Skipped 1 previous similar message
[13994207.937650] LustreError: 137-5: oak-OST013c_UUID: not available for connect from 10.210.12.8@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13994218.371342] LustreError: 162716:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff917eea146050 x1714974183805056/t0(0) o4->5d4d5527-a52f-bc68-278b-aa990b8609d6@10.210.12.59@tcp1:46/0 lens 488/448 e 0 to 0 dl 1645220546 ref 1 fl Interpret:/0/0 rc 0/0
[13994218.397278] Lustre: oak-OST0113: Bulk IO write error with 5d4d5527-a52f-bc68-278b-aa990b8609d6 (at 10.210.12.59@tcp1), client will retry: rc = -110
[13994218.410707] Lustre: Skipped 7 previous similar messages
[13994252.909066] LustreError: 137-5: oak-OST0140_UUID: not available for connect from 10.210.12.59@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13994252.929214] Lustre: oak-OST0113: Client 5d4d5527-a52f-bc68-278b-aa990b8609d6 (at 10.210.12.59@tcp1) reconnecting
[13994252.939648] Lustre: Skipped 78 previous similar messages
[13994457.865141] LustreError: 162681:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff916118309850 x1723945670060544/t0(0) o4->ebb89704-8dcb-738e-08f3-59645d9d1d58@10.50.6.46@o2ib2:252/0 lens 488/448 e 0 to 0 dl 1645220752 ref 1 fl Interpret:/0/0 rc 0/0
[13994457.891241] Lustre: oak-OST0133: Bulk IO write error with ebb89704-8dcb-738e-08f3-59645d9d1d58 (at 10.50.6.46@o2ib2), client will retry: rc = -110
[13994737.272911] Lustre: oak-OST0119: Connection restored to 1325a26d-75c0-dda8-47f6-89b207afd477 (at 10.51.2.68@o2ib3)
[13994737.283495] Lustre: Skipped 1625 previous similar messages
[13995288.223498] Lustre: oak-OST0133: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[13995288.233977] Lustre: Skipped 202 previous similar messages
[13995288.319519] LustreError: 162701:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9166aca72050 x1715069335751168/t0(0) o4->bb55a01e-02a3-cbba-5b76-e478fd40967c@10.210.12.54@tcp1:414/0 lens 488/448 e 0 to 0 dl 1645221669 ref 1 fl Interpret:/0/0 rc 0/0
[13995288.344035] LustreError: 162701:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 5 previous similar messages
[13995288.354068] Lustre: oak-OST0147: Bulk IO write error with bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1), client will retry: rc = -110
[13995336.101309] Lustre: oak-OST011b: Connection restored to 7d03c8de-fa73-e098-9e5b-f12d98485186 (at 10.50.1.55@o2ib2)
[13995336.111901] Lustre: Skipped 827 previous similar messages
[13995343.026960] LustreError: 21588:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff916c0e52d850 x1715062289012096/t0(0) o4->dc851ef6-3898-c01c-adc6-dc548a499d03@10.210.12.53@tcp1:406/0 lens 488/448 e 0 to 0 dl 1645221661 ref 1 fl Interpret:/0/0 rc 0/0
[13995343.026962] LustreError: 162698:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 3145728(4194304)  req@ffff916c0e52f050 x1715062289011904/t0(0) o3->dc851ef6-3898-c01c-adc6-dc548a499d03@10.210.12.53@tcp1:406/0 lens 488/440 e 0 to 0 dl 1645221661 ref 1 fl Interpret:/0/0 rc 0/0
[13995343.027159] Lustre: oak-OST0145: Bulk IO read error with dc851ef6-3898-c01c-adc6-dc548a499d03 (at 10.210.12.53@tcp1), client will retry: rc -110
[13995343.027268] Lustre: oak-OST0145: Bulk IO write error with b4fca75a-2d54-9bf2-331f-67d0c52e5ff2 (at 10.210.12.73@tcp1), client will retry: rc = -110
[13995343.027269] Lustre: Skipped 6 previous similar messages
[13995343.110321] LustreError: 21588:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[13995364.036034] LustreError: 137-5: oak-OST0142_UUID: not available for connect from 10.210.12.59@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13995365.038809] LustreError: 137-5: oak-OST0148_UUID: not available for connect from 10.210.12.53@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13995365.060651] Lustre: oak-OST0113: Client dc851ef6-3898-c01c-adc6-dc548a499d03 (at 10.210.12.53@tcp1) reconnecting
[13995454.577155] Lustre: oak-OST0121: Client b4fca75a-2d54-9bf2-331f-67d0c52e5ff2 (at 10.210.12.73@tcp1) reconnecting
[13995454.587574] Lustre: Skipped 2 previous similar messages
[13995936.421548] Lustre: oak-OST0145: Connection restored to  (at 10.51.3.22@o2ib3)
[13995936.429018] Lustre: Skipped 912 previous similar messages
[13996537.491477] Lustre: oak-OST0113: Connection restored to 612334a0-616c-86f1-bbbc-1f12050c0dbf (at 10.50.7.33@o2ib2)
[13996537.502076] Lustre: Skipped 860 previous similar messages
[13997139.168081] Lustre: oak-OST011b: Connection restored to 07c826f5-cd86-48e2-c47b-9ebe60acdc95 (at 10.50.14.15@o2ib2)
[13997139.178824] Lustre: Skipped 1337 previous similar messages
[13997728.648612] Lustre: oak-OST0149: Client 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1) reconnecting
[13997728.659035] Lustre: Skipped 104 previous similar messages
[13997728.744503] LustreError: 243453:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91a214c1d850 x1714954254762304/t0(0) o4->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:590/0 lens 488/448 e 0 to 0 dl 1645224110 ref 1 fl Interpret:/0/0 rc 0/0
[13997728.768934] LustreError: 243453:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 6 previous similar messages
[13997728.778933] Lustre: oak-OST0149: Bulk IO write error with 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1), client will retry: rc = -110
[13997728.793233] Lustre: Skipped 2 previous similar messages
[13997729.362010] LustreError: 244099:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91a9eee5f850 x1715062318195328/t0(0) o4->dc851ef6-3898-c01c-adc6-dc548a499d03@10.210.12.53@tcp1:591/0 lens 488/448 e 0 to 0 dl 1645224111 ref 1 fl Interpret:/0/0 rc 0/0
[13997734.373816] LustreError: 160896:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91bbf82b6850 x1715062318195328/t0(0) o4->dc851ef6-3898-c01c-adc6-dc548a499d03@10.210.12.53@tcp1:596/0 lens 488/448 e 0 to 0 dl 1645224116 ref 1 fl Interpret:/2/0 rc 0/0
[13997734.398546] Lustre: oak-OST0113: Bulk IO write error with dc851ef6-3898-c01c-adc6-dc548a499d03 (at 10.210.12.53@tcp1), client will retry: rc = -110
[13997734.412037] Lustre: Skipped 2 previous similar messages
[13997739.389955] Lustre: oak-OST0131: Connection restored to 0465062f-c53d-e09b-c11a-86f450e750ba (at 10.50.1.72@o2ib2)
[13997739.400585] Lustre: Skipped 874 previous similar messages
[13997784.861693] LustreError: 243457:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff919879767850 x1715094228695040/t0(0) o4->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:590/0 lens 488/448 e 0 to 0 dl 1645224110 ref 1 fl Interpret:/0/0 rc 0/0
[13997784.861712] LustreError: 160910:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(20480)  req@ffff91a8845f9050 x1714954254759488/t0(0) o3->11a5f679-c52d-ef2d-3d8a-d81b71720c61@10.210.12.31@tcp1:590/0 lens 488/440 e 0 to 0 dl 1645224110 ref 1 fl Interpret:/0/0 rc 0/0
[13997784.861714] LustreError: 160910:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 1 previous similar message
[13997784.861731] Lustre: oak-OST013f: Bulk IO read error with 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1), client will retry: rc -110
[13997784.861732] Lustre: Skipped 1 previous similar message
[13997784.861949] Lustre: oak-OST0139: Bulk IO write error with 5d8bb8d9-e5c7-334f-ec8d-f10452c309b5 (at 10.210.12.66@tcp1), client will retry: rc = -110
[13997784.861949] Lustre: Skipped 3 previous similar messages
[13997784.959557] LustreError: 243457:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 7 previous similar messages
[13997808.379087] LustreError: 137-5: oak-OST014c_UUID: not available for connect from 10.210.12.79@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13997808.396703] LustreError: Skipped 1 previous similar message
[13997808.401218] Lustre: oak-OST013f: Client e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1) reconnecting
[13997808.401220] Lustre: Skipped 6 previous similar messages
[13997809.094821] LustreError: 137-5: oak-OST012a_UUID: not available for connect from 10.210.12.56@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13997814.309989] LustreError: 137-5: oak-OST0120_UUID: not available for connect from 10.210.12.31@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[13997814.327585] LustreError: Skipped 1 previous similar message
[13997895.697512] Lustre: oak-OST0145: Client 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1) reconnecting
[13997895.707923] Lustre: Skipped 4 previous similar messages
[13997975.793889] Lustre: oak-OST0143: Client e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1) reconnecting
[13997975.804340] Lustre: Skipped 160 previous similar messages
[13998172.962830] Lustre: 253950:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1645224335/real 1645224335]  req@ffff919bc9821680 x1710531107492480/t0(0) o106->oak-OST011b@10.51.14.15@o2ib3:15/16 lens 296/280 e 0 to 1 dl 1645224463 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
[13998245.410722] LNet: Service thread pid 253950 was inactive for 200.63s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
[13998245.428742] Pid: 253950, comm: ll_ost00_039 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[13998245.439601] Call Trace:
[13998245.442846]  [<ffffffffc1072310>] ptlrpc_set_wait+0x480/0x790 [ptlrpc]
[13998245.450037]  [<ffffffffc102f145>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc]
[13998245.457325]  [<ffffffffc105054b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc]
[13998245.464702]  [<ffffffffc168762b>] ofd_intent_policy+0x69b/0x920 [ofd]
[13998245.471673]  [<ffffffffc10304e6>] ldlm_lock_enqueue+0x376/0x9b0 [ptlrpc]
[13998245.478929]  [<ffffffffc1057a16>] ldlm_handle_enqueue0+0xa86/0x1620 [ptlrpc]
[13998245.486846]  [<ffffffffc10e0a42>] tgt_enqueue+0x62/0x210 [ptlrpc]
[13998245.493617]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[13998245.501107]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[13998245.509426]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[13998245.516272]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[13998245.521878]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[13998245.528880]  [<ffffffffffffffff>] 0xffffffffffffffff
[13998245.534229] LustreError: dumping log to /tmp/lustre-log.1645224535.253950
[13998252.667736] Lustre: oak-OST0139: haven't heard from client b3489547-4492-aeea-00a1-1e981eefc200 (at 10.51.14.15@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91963e7c9400, cur 1645224543 expire 1645224393 last 1645224316
[13998252.690527] Lustre: Skipped 18 previous similar messages
[13998252.702027] LNet: Service thread pid 253950 completed after 207.94s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
[13998340.729049] Lustre: oak-OST013b: Connection restored to e10c3e22-d196-9db7-8dfa-d975b5ef1879 (at 10.50.8.30@o2ib2)
[13998340.739646] Lustre: Skipped 1430 previous similar messages
[13998940.279213] Lustre: oak-OST0131: Connection restored to 1e369504-a05d-47b6-dff1-1c29be4dae5e (at 10.51.13.18@o2ib3)
[13998940.290012] Lustre: Skipped 872 previous similar messages
[13999540.363885] Lustre: oak-OST011b: Connection restored to 080fbf36-50e2-e037-d579-97f7ea4dd6a6 (at 10.210.12.51@tcp1)
[13999540.374558] Lustre: Skipped 1215 previous similar messages
[14000139.506809] Lustre: oak-OST0147: Connection restored to bbafc8fb-b3eb-ebc6-fdd7-737354a89048 (at 10.51.1.51@o2ib3)
[14000139.517392] Lustre: Skipped 1367 previous similar messages
[14000527.300182] Lustre: oak-OST012b: Client 1b0e0de8-4056-d02c-bb09-33adedaa9f96 (at 10.210.12.45@tcp1) reconnecting
[14000527.310615] Lustre: Skipped 15 previous similar messages
[14000527.736364] LustreError: 162690:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff916a99773850 x1715094149498112/t0(0) o4->1b0e0de8-4056-d02c-bb09-33adedaa9f96@10.210.12.45@tcp1:376/0 lens 488/448 e 0 to 0 dl 1645226916 ref 1 fl Interpret:/0/0 rc 0/0
[14000527.760777] LustreError: 162690:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 4 previous similar messages
[14000527.770798] Lustre: oak-OST012b: Bulk IO write error with 1b0e0de8-4056-d02c-bb09-33adedaa9f96 (at 10.210.12.45@tcp1), client will retry: rc = -110
[14000527.784232] Lustre: Skipped 7 previous similar messages
[14000529.046183] LustreError: 162701:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff918f78995850 x1715062336383360/t0(0) o4->dc851ef6-3898-c01c-adc6-dc548a499d03@10.210.12.53@tcp1:377/0 lens 488/448 e 0 to 0 dl 1645226917 ref 1 fl Interpret:/0/0 rc 0/0
[14000529.070777] Lustre: oak-OST0113: Bulk IO write error with dc851ef6-3898-c01c-adc6-dc548a499d03 (at 10.210.12.53@tcp1), client will retry: rc = -110
[14000530.312104] LustreError: 243332:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff918f788e1850 x1715094149522048/t0(0) o4->1b0e0de8-4056-d02c-bb09-33adedaa9f96@10.210.12.45@tcp1:381/0 lens 488/448 e 0 to 0 dl 1645226921 ref 1 fl Interpret:/0/0 rc 0/0
[14000530.336525] LustreError: 243332:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 2 previous similar messages
[14000699.785175] Lustre: oak-OST0117: Client 1b0e0de8-4056-d02c-bb09-33adedaa9f96 (at 10.210.12.45@tcp1) reconnecting
[14000699.795587] Lustre: Skipped 3 previous similar messages
[14000738.079799] Lustre: oak-OST0143: Connection restored to 55ade167-ee14-7aa8-e90b-b61cc094ea5c (at 10.50.6.56@o2ib2)
[14000738.090649] Lustre: Skipped 1843 previous similar messages
[14000776.772452] Lustre: oak-OST0115: Client 5a4881be-a0cb-e632-509b-2e15c534b21a (at 10.210.12.9@tcp1) reconnecting
[14000776.782772] Lustre: Skipped 35 previous similar messages
[14001337.069753] Lustre: oak-OST0147: Connection restored to 93258f08-cea9-f886-84cb-997737b28b94 (at 10.51.2.62@o2ib3)
[14001337.080347] Lustre: Skipped 1136 previous similar messages
[14001893.817325] Lustre: oak-OST0129: haven't heard from client 43cda6b0-c8d2-ecce-5d4f-5faa35bc5d00 (at 10.51.14.9@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9136617b7000, cur 1645228193 expire 1645228043 last 1645227966
[14001893.839532] Lustre: Skipped 11 previous similar messages
[14001936.677294] Lustre: oak-OST0123: Connection restored to  (at 10.50.6.41@o2ib2)
[14001936.684768] Lustre: Skipped 1163 previous similar messages
[14002535.344542] Lustre: oak-OST0117: Connection restored to 3362c77a-2096-c98e-90ba-a15ed550df3d (at 10.50.2.31@o2ib2)
[14002535.355138] Lustre: Skipped 1151 previous similar messages
[14003134.102183] Lustre: oak-OST013f: Connection restored to cd018073-e38f-fbd3-4753-aae732183dca (at 10.50.5.59@o2ib2)
[14003134.112763] Lustre: Skipped 1175 previous similar messages
[14003733.075452] Lustre: oak-OST0145: Connection restored to e9ec1c43-6e89-086b-9840-3cb159bbec79 (at 10.50.5.65@o2ib2)
[14003733.086031] Lustre: Skipped 847 previous similar messages
[14004332.301554] Lustre: oak-OST0111: Connection restored to 2dc703cd-5cca-815b-6dd4-cd3e7d5a0392 (at 10.0.3.49@o2ib5)
[14004332.312050] Lustre: Skipped 1128 previous similar messages
[14004931.402763] Lustre: oak-OST0139: Connection restored to  (at 10.51.4.8@o2ib3)
[14004931.410449] Lustre: Skipped 992 previous similar messages
[14005087.600413] LustreError: 160932:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91ef238db050 x1715243442835648/t0(0) o4->e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda@10.210.12.79@tcp1:357/0 lens 488/448 e 0 to 0 dl 1645231427 ref 1 fl Interpret:/0/0 rc 0/0
[14005087.600775] Lustre: oak-OST012f: Bulk IO write error with e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1), client will retry: rc = -110
[14005087.600776] Lustre: Skipped 3 previous similar messages
[14005087.645133] LustreError: 160932:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[14005109.199506] LustreError: 137-5: oak-OST0148_UUID: not available for connect from 10.210.12.79@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14005109.233855] Lustre: oak-OST012f: Client e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1) reconnecting
[14005109.244266] Lustre: Skipped 17 previous similar messages
[14005200.444208] Lustre: oak-OST0127: Client e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1) reconnecting
[14005221.299804] Lustre: oak-OST014b: Client e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1) reconnecting
[14005221.310252] Lustre: Skipped 6 previous similar messages
[14005441.489325] Lustre: oak-OST011f: Client 7189afb8-8fe5-7418-d14a-4a60d24330d0 (at 10.210.12.48@tcp1) reconnecting
[14005441.499732] Lustre: Skipped 10 previous similar messages
[14005441.768965] LustreError: 162710:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff917320b0c050 x1715185662606912/t0(0) o4->7189afb8-8fe5-7418-d14a-4a60d24330d0@10.210.12.48@tcp1:17/0 lens 488/448 e 0 to 0 dl 1645231842 ref 1 fl Interpret:/0/0 rc 0/0
[14005441.793575] Lustre: oak-OST011f: Bulk IO write error with 7189afb8-8fe5-7418-d14a-4a60d24330d0 (at 10.210.12.48@tcp1), client will retry: rc = -110
[14005441.807036] Lustre: Skipped 2 previous similar messages
[14005441.829088] LustreError: 162717:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff9134cbfdd850 x1715185662620160/t0(0) o4->7189afb8-8fe5-7418-d14a-4a60d24330d0@10.210.12.48@tcp1:22/0 lens 488/448 e 0 to 0 dl 1645231847 ref 1 fl Interpret:/2/0 rc 0/0
[14005442.300670] LustreError: 162677:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff915fd9fa6850 x1715185662607040/t0(0) o4->7189afb8-8fe5-7418-d14a-4a60d24330d0@10.210.12.48@tcp1:18/0 lens 488/448 e 0 to 0 dl 1645231843 ref 1 fl Interpret:/0/0 rc 0/0
[14005442.325003] LustreError: 162677:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 2 previous similar messages
[14005442.335078] LustreError: 21600:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff9164be394850 x1715185662607040/t0(0) o4->7189afb8-8fe5-7418-d14a-4a60d24330d0@10.210.12.48@tcp1:22/0 lens 488/448 e 0 to 0 dl 1645231847 ref 1 fl Interpret:/2/0 rc 0/0
[14005530.284751] Lustre: oak-OST0113: Connection restored to  (at 10.0.3.12@o2ib5)
[14005530.292349] Lustre: Skipped 917 previous similar messages
[14006128.984013] Lustre: oak-OST012f: Connection restored to  (at 10.51.3.14@o2ib3)
[14006128.991481] Lustre: Skipped 2772 previous similar messages
[14006731.334818] Lustre: oak-OST0129: Connection restored to c10b6301-1800-4510-94bd-883f949efde1 (at 10.50.6.68@o2ib2)
[14006731.345437] Lustre: Skipped 1506 previous similar messages
[14007331.662034] Lustre: oak-OST0131: Connection restored to 13782cfa-f64e-3b30-9b19-eac54c74cba1 (at 10.51.15.10@o2ib3)
[14007331.672709] Lustre: Skipped 1077 previous similar messages
[14007671.772130] Lustre: oak-OST0135: haven't heard from client accc66e1-5d9a-c1c3-e686-fdcdcca4d204 (at 10.51.14.9@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91f072d91c00, cur 1645233985 expire 1645233835 last 1645233758
[14007671.794051] Lustre: Skipped 2 previous similar messages
[14007932.638385] Lustre: oak-OST012d: Connection restored to 9faab410-931d-224c-0590-b37be1578f00 (at 10.51.12.12@o2ib3)
[14007932.649097] Lustre: Skipped 1134 previous similar messages
[14008532.613533] Lustre: oak-OST013f: Connection restored to 004ec874-506a-fff0-05e0-d4bb5fbc74bf (at 10.50.5.15@o2ib2)
[14008532.624115] Lustre: Skipped 1037 previous similar messages
[14008726.834873] LustreError: 243542:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2867200(3915776)  req@ffff913a4dae2050 x1715761714063104/t0(0) o4->4b39aac6-2dc7-e48f-9434-e3bfde492453@10.210.12.29@tcp1:238/0 lens 504/448 e 0 to 0 dl 1645235083 ref 1 fl Interpret:/0/0 rc 0/0
[14008726.835107] Lustre: oak-OST012f: Bulk IO write error with e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1), client will retry: rc = -110
[14008726.835108] Lustre: Skipped 6 previous similar messages
[14008726.879566] LustreError: 243542:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[14008751.789115] Lustre: oak-OST012f: Client e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1) reconnecting
[14008751.799549] Lustre: Skipped 1 previous similar message
[14008840.152269] Lustre: oak-OST0121: Client e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1) reconnecting
[14008840.162673] Lustre: Skipped 4 previous similar messages
[14008859.978622] Lustre: oak-OST0117: Client e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1) reconnecting
[14008859.989040] Lustre: Skipped 12 previous similar messages
[14008897.594696] Lustre: oak-OST014b: Client 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1) reconnecting
[14008897.605136] Lustre: Skipped 3 previous similar messages
[14009133.902125] Lustre: oak-OST012f: Connection restored to ae907b4f-c5e6-3faf-34df-6c94b3fa3fee (at 10.50.5.62@o2ib2)
[14009133.912737] Lustre: Skipped 884 previous similar messages
[14009733.171156] Lustre: oak-OST013b: Connection restored to 5b2e8db7-a7d8-f3f0-2a21-c4a37ce6855f (at 10.50.9.15@o2ib2)
[14009733.181741] Lustre: Skipped 645 previous similar messages
[14009947.258795] Lustre: oak-OST014d: haven't heard from client 1c4af898-bf4d-1791-1e3b-347619b83e52 (at 10.51.15.18@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91d439049000, cur 1645236266 expire 1645236116 last 1645236039
[14009947.280787] Lustre: Skipped 2 previous similar messages
[14010333.415510] Lustre: oak-OST013f: Connection restored to 6fe93d71-a5c2-7c88-391f-d59607fec978 (at 10.51.6.55@o2ib3)
[14010333.426102] Lustre: Skipped 466 previous similar messages
[14010932.066858] Lustre: oak-OST011f: Connection restored to 2d7b80e6-0f0d-a2ea-c50b-4f791ee979a6 (at 10.51.16.21@o2ib3)
[14010932.077529] Lustre: Skipped 876 previous similar messages
[14011531.358518] Lustre: oak-OST013b: Connection restored to eee699b2-cb8e-0a91-6752-be0f592e4945 (at 10.50.13.2@o2ib2)
[14011531.369095] Lustre: Skipped 756 previous similar messages
[14012132.540681] Lustre: oak-OST0147: Connection restored to  (at 10.50.4.9@o2ib2)
[14012132.548064] Lustre: Skipped 632 previous similar messages
[14012731.106670] Lustre: oak-OST013f: Connection restored to 5f5174f5-55fa-8ffb-fdda-16c253f4b547 (at 10.51.1.41@o2ib3)
[14012731.117276] Lustre: Skipped 824 previous similar messages
[14012834.533104] Lustre: 168283:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1645239032/real 1645239032]  req@ffff91446c442400 x1710531184801792/t0(0) o106->oak-OST014d@10.51.12.11@o2ib3:15/16 lens 296/280 e 0 to 1 dl 1645239160 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
[14012898.197468] Lustre: oak-OST012f: haven't heard from client 4dc47145-b212-d350-51c8-ee272db525dd (at 10.51.12.11@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91d0d5491400, cur 1645239224 expire 1645239074 last 1645238997
[14012899.069175] Lustre: oak-OST013b: haven't heard from client 4dc47145-b212-d350-51c8-ee272db525dd (at 10.51.12.11@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91d0d5491c00, cur 1645239225 expire 1645239075 last 1645238998
[14012900.109836] Lustre: oak-OST0133: haven't heard from client 4dc47145-b212-d350-51c8-ee272db525dd (at 10.51.12.11@o2ib3) in 228 seconds. I think it's dead, and I am evicting it. exp ffff9196c5150800, cur 1645239226 expire 1645239076 last 1645238998
[14012900.131871] Lustre: Skipped 17 previous similar messages
[14013239.441710] md: md1: data-check done.
[14013330.461020] Lustre: oak-OST0127: Connection restored to  (at 10.50.6.52@o2ib2)
[14013330.468492] Lustre: Skipped 754 previous similar messages
[14013930.845804] Lustre: oak-OST013b: Connection restored to cf346ddd-2ee1-e262-27d0-94e0a104ae94 (at 10.51.6.14@o2ib3)
[14013930.856393] Lustre: Skipped 714 previous similar messages
[14014530.999889] Lustre: oak-OST0121: Connection restored to 4025d53e-c01c-f5a6-a7de-c3a1c13db1f4 (at 10.50.10.5@o2ib2)
[14014531.010474] Lustre: Skipped 674 previous similar messages
[14015132.021763] Lustre: oak-OST0149: Connection restored to 83034250-f855-6c12-471e-38d9aeacf85d (at 10.210.12.54@tcp1)
[14015132.032443] Lustre: Skipped 760 previous similar messages
[14015731.148008] Lustre: oak-OST0131: Connection restored to d963ee83-3fcf-bfee-1cc5-212ce4916b8e (at 10.51.1.56@o2ib3)
[14015731.158587] Lustre: Skipped 967 previous similar messages
[14016331.199774] Lustre: oak-OST0143: Connection restored to 3851b2a4-e5ac-9541-93fc-7fd9ef65d8f5 (at 10.50.7.20@o2ib2)
[14016331.210355] Lustre: Skipped 775 previous similar messages
[14016604.065606] Lustre: oak-OST013d: haven't heard from client 6699749a-f12a-de81-ba60-7090c771f61c (at 10.51.14.9@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9146a457bc00, cur 1645242939 expire 1645242789 last 1645242712
[14016604.087667] Lustre: Skipped 6 previous similar messages
[14016934.629192] Lustre: oak-OST0137: Connection restored to 72eaf2df-907e-10db-7344-e8188bb48d82 (at 10.50.9.12@o2ib2)
[14016934.639765] Lustre: Skipped 686 previous similar messages
[14017533.811811] Lustre: oak-OST0141: Connection restored to bbba5ad9-9372-2297-5b66-f72cfc361471 (at 10.0.3.25@o2ib5)
[14017533.822326] Lustre: Skipped 735 previous similar messages
[14018132.956104] Lustre: oak-OST0127: Connection restored to 3195013f-4cca-32ee-d13b-67b22cd84ad2 (at 10.50.1.16@o2ib2)
[14018132.966689] Lustre: Skipped 612 previous similar messages
[14018732.899430] Lustre: oak-OST014b: Connection restored to  (at 10.50.7.2@o2ib2)
[14018732.906810] Lustre: Skipped 695 previous similar messages
[14019332.019303] Lustre: oak-OST0121: Connection restored to 3ab806c9-09e9-291d-925c-0c353edff7d6 (at 10.210.12.48@tcp1)
[14019332.029983] Lustre: Skipped 738 previous similar messages
[14019933.612013] Lustre: oak-OST0131: Connection restored to f737e776-c753-2b23-866e-4368b7fcef83 (at 10.51.1.22@o2ib3)
[14019933.622636] Lustre: Skipped 592 previous similar messages
[14020533.721948] Lustre: oak-OST011d: Connection restored to 6bf3a453-32ad-fae9-7830-7939bb27cf82 (at 10.50.10.20@o2ib2)
[14020533.732622] Lustre: Skipped 648 previous similar messages
[14021133.532535] Lustre: oak-OST012d: Connection restored to 86eee6bc-6017-fb1a-3873-1027e48995f7 (at 10.50.12.3@o2ib2)
[14021133.543111] Lustre: Skipped 663 previous similar messages
[14021732.456758] Lustre: oak-OST0133: Connection restored to 2dc703cd-5cca-815b-6dd4-cd3e7d5a0392 (at 10.0.3.49@o2ib5)
[14021732.467246] Lustre: Skipped 725 previous similar messages
[14022332.608943] Lustre: oak-OST0131: Connection restored to 2dc703cd-5cca-815b-6dd4-cd3e7d5a0392 (at 10.0.3.49@o2ib5)
[14022332.619453] Lustre: Skipped 655 previous similar messages
[14022932.422835] Lustre: oak-OST012d: Connection restored to 2dc703cd-5cca-815b-6dd4-cd3e7d5a0392 (at 10.0.3.49@o2ib5)
[14022932.433377] Lustre: Skipped 728 previous similar messages
[14023531.796134] Lustre: oak-OST0133: Connection restored to cc107881-9f34-e151-f59d-58eb60eb5c8e (at 10.51.4.46@o2ib3)
[14023531.806726] Lustre: Skipped 689 previous similar messages
[14024131.933448] Lustre: oak-OST0137: Connection restored to  (at 10.50.3.62@o2ib2)
[14024131.940918] Lustre: Skipped 743 previous similar messages
[14024731.221949] Lustre: oak-OST0113: Connection restored to 5e4bce85-4d6e-90c6-523c-4bdf35e8bc4a (at 10.210.12.7@tcp1)
[14024731.232561] Lustre: Skipped 745 previous similar messages
[14025330.114361] Lustre: oak-OST0133: Connection restored to bbba5ad9-9372-2297-5b66-f72cfc361471 (at 10.0.3.25@o2ib5)
[14025330.124958] Lustre: Skipped 952 previous similar messages
[14025928.919882] Lustre: oak-OST013b: Connection restored to a50d9b1d-84ff-729c-a17d-78f25cb23247 (at 10.51.7.15@o2ib3)
[14025928.930492] Lustre: Skipped 683 previous similar messages
[14026528.992198] Lustre: oak-OST0141: Connection restored to  (at 10.50.7.2@o2ib2)
[14026528.999610] Lustre: Skipped 650 previous similar messages
[14027130.952977] Lustre: oak-OST011f: Connection restored to 5df0bdab-b30e-504d-aa3f-1cb04e46cc0f (at 10.51.14.1@o2ib3)
[14027130.963589] Lustre: Skipped 773 previous similar messages
[14027729.926995] Lustre: oak-OST0141: Connection restored to bbba5ad9-9372-2297-5b66-f72cfc361471 (at 10.0.3.25@o2ib5)
[14027729.937509] Lustre: Skipped 753 previous similar messages
[14028329.554541] Lustre: oak-OST013b: Connection restored to 92714055-bc95-a3ff-653f-640af784f9bf (at 10.51.6.58@o2ib3)
[14028329.565161] Lustre: Skipped 981 previous similar messages
[14028928.135140] Lustre: oak-OST0139: Connection restored to 3e72271e-d8a9-bff9-a052-ac4ffc30f7d6 (at 10.210.12.40@tcp1)
[14028928.145818] Lustre: Skipped 768 previous similar messages
[14029476.360369] md: md19: data-check done.
[14029527.228235] Lustre: oak-OST011b: Connection restored to 010c0fa9-cabf-a6b8-0616-88e4b85f309a (at 10.50.2.30@o2ib2)
[14029527.238838] Lustre: Skipped 943 previous similar messages
[14030128.881610] Lustre: oak-OST011b: Connection restored to 4412eb0d-dd51-9863-f399-13f67dd089c7 (at 10.50.7.18@o2ib2)
[14030128.892192] Lustre: Skipped 868 previous similar messages
[14030728.424114] Lustre: oak-OST0143: Connection restored to 0923ba4e-f4d3-9b0e-c80c-4d51653819ef (at 10.50.8.61@o2ib2)
[14030728.434761] Lustre: Skipped 786 previous similar messages
[14031327.960828] Lustre: oak-OST0127: Connection restored to 90511c1a-0572-899c-3ffd-f74d738b1a31 (at 10.50.1.38@o2ib2)
[14031327.971412] Lustre: Skipped 882 previous similar messages
[14031926.607940] Lustre: oak-OST0125: Connection restored to 341d238a-9858-e96d-d91e-09be3a2661d6 (at 10.51.4.18@o2ib3)
[14031926.618518] Lustre: Skipped 859 previous similar messages
[14032526.377154] Lustre: oak-OST013d: Connection restored to  (at 10.50.6.48@o2ib2)
[14032526.384673] Lustre: Skipped 1418 previous similar messages
[14033124.938570] Lustre: oak-OST0147: Connection restored to 820a1ce2-6182-e058-f1e9-e3948266ca32 (at 10.51.4.21@o2ib3)
[14033124.949152] Lustre: Skipped 2350 previous similar messages
[14033723.597454] Lustre: oak-OST011b: Connection restored to 35c9ea2a-285e-d3a0-91c6-7b9f7263033d (at 10.51.6.34@o2ib3)
[14033723.608095] Lustre: Skipped 1590 previous similar messages
[14034324.007169] Lustre: oak-OST014b: Connection restored to  (at 10.50.7.2@o2ib2)
[14034324.014566] Lustre: Skipped 1353 previous similar messages
[14034923.005663] Lustre: oak-OST0145: Connection restored to 6c2fbd18-a47d-94af-ad13-db2c057feb17 (at 10.50.9.36@o2ib2)
[14034923.016391] Lustre: Skipped 824 previous similar messages
[14035522.577012] Lustre: oak-OST011b: Connection restored to 2dc703cd-5cca-815b-6dd4-cd3e7d5a0392 (at 10.0.3.49@o2ib5)
[14035522.587502] Lustre: Skipped 818 previous similar messages
[14036121.912801] Lustre: oak-OST0125: Connection restored to 25f5d5dd-3ec2-9fe1-ee3e-b19f017aeed7 (at 10.51.6.50@o2ib3)
[14036121.923394] Lustre: Skipped 852 previous similar messages
[14036722.138003] Lustre: oak-OST0111: Connection restored to 2dc703cd-5cca-815b-6dd4-cd3e7d5a0392 (at 10.0.3.49@o2ib5)
[14036722.148650] Lustre: Skipped 869 previous similar messages
[14037320.991993] Lustre: oak-OST0129: Connection restored to fa202a18-aa95-b01f-fab7-7e4269883f98 (at 10.50.16.10@o2ib2)
[14037321.002676] Lustre: Skipped 971 previous similar messages
[14037919.579091] Lustre: oak-OST011d: Connection restored to 48813602-6143-bd10-868d-89547dd3a8e5 (at 10.50.8.43@o2ib2)
[14037919.589676] Lustre: Skipped 922 previous similar messages
[14038519.661387] Lustre: oak-OST012d: Connection restored to ca2f65af-d147-d7ba-0d14-5b6792f912fe (at 10.50.9.44@o2ib2)
[14038519.671980] Lustre: Skipped 782 previous similar messages
[14039118.480181] Lustre: oak-OST0131: Connection restored to 767b7c94-5f6a-643b-d3fc-ff266d4e288d (at 10.50.4.60@o2ib2)
[14039118.490812] Lustre: Skipped 1030 previous similar messages
[14039718.650193] Lustre: oak-OST0129: Connection restored to 080fbf36-50e2-e037-d579-97f7ea4dd6a6 (at 10.210.12.51@tcp1)
[14039718.660859] Lustre: Skipped 712 previous similar messages
[14040317.385848] Lustre: oak-OST012d: Connection restored to 916093a3-51ee-71cf-ab61-9d8121d32b7c (at 10.51.15.11@o2ib3)
[14040317.396521] Lustre: Skipped 646 previous similar messages
[14040920.325001] Lustre: oak-OST011d: Connection restored to b0fb8f7d-f365-c78f-b271-58d0a3707652 (at 10.51.2.25@o2ib3)
[14040920.335662] Lustre: Skipped 709 previous similar messages
[14041519.133994] Lustre: oak-OST0133: Connection restored to a7228c75-c53b-e2c9-6e64-f881b06e64fc (at 10.50.13.13@o2ib2)
[14041519.144702] Lustre: Skipped 645 previous similar messages
[14042122.545314] Lustre: oak-OST0141: Connection restored to  (at 10.50.7.2@o2ib2)
[14042122.552699] Lustre: Skipped 772 previous similar messages
[14042725.050703] Lustre: oak-OST0113: Connection restored to 491df614-5ff0-2c9f-0a9d-38e20f90da12 (at 10.50.6.69@o2ib2)
[14042725.061283] Lustre: Skipped 774 previous similar messages
[14043327.688050] Lustre: oak-OST013f: Connection restored to 033b48ab-9633-d6d6-a05b-11e7e6f562ae (at 10.51.12.17@o2ib3)
[14043327.698752] Lustre: Skipped 718 previous similar messages
[14043929.262295] Lustre: oak-OST0147: Connection restored to 7b218846-8296-5a90-f251-8d8c57ad58ac (at 10.51.6.3@o2ib3)
[14043929.272794] Lustre: Skipped 662 previous similar messages
[14044536.185223] Lustre: oak-OST012d: Connection restored to 49044c4e-6a2f-ee70-77e0-627b9daaf804 (at 10.50.2.68@o2ib2)
[14044536.195801] Lustre: Skipped 633 previous similar messages
[14045135.469443] Lustre: oak-OST0145: Connection restored to 5b44aa98-63f7-68a9-3a7c-55fa5232c4f2 (at 10.51.2.18@o2ib3)
[14045135.480036] Lustre: Skipped 874 previous similar messages
[14045734.138838] Lustre: oak-OST011b: Connection restored to 19cddcc8-ef13-c2fa-cbaa-5f2f8a1894f6 (at 10.51.1.57@o2ib3)
[14045734.149453] Lustre: Skipped 783 previous similar messages
[14046335.357093] Lustre: oak-OST0131: Connection restored to 080fbf36-50e2-e037-d579-97f7ea4dd6a6 (at 10.210.12.51@tcp1)
[14046335.367831] Lustre: Skipped 870 previous similar messages
[14046935.827398] Lustre: oak-OST0137: Connection restored to 5df0bdab-b30e-504d-aa3f-1cb04e46cc0f (at 10.51.14.1@o2ib3)
[14046935.838002] Lustre: Skipped 789 previous similar messages
[14047536.956993] Lustre: oak-OST0119: Connection restored to 53f7c8fb-ba13-9226-8f66-6ca61c432359 (at 10.51.12.11@o2ib3)
[14047536.967754] Lustre: Skipped 674 previous similar messages
[14048137.230545] Lustre: oak-OST0145: Connection restored to 9faab410-931d-224c-0590-b37be1578f00 (at 10.51.12.12@o2ib3)
[14048137.241229] Lustre: Skipped 814 previous similar messages
[14048736.320301] Lustre: oak-OST0141: Connection restored to b9694c0e-9fc2-f14e-33c8-9018d9fc9806 (at 10.210.12.65@tcp1)
[14048736.331022] Lustre: Skipped 849 previous similar messages
[14049336.183562] Lustre: oak-OST0145: Connection restored to  (at 10.51.13.12@o2ib3)
[14049336.191285] Lustre: Skipped 750 previous similar messages
[14049935.030968] Lustre: oak-OST0147: Connection restored to da38bf5f-d67e-0540-79b1-ed3fbf50ea94 (at 10.50.6.59@o2ib2)
[14049935.041559] Lustre: Skipped 773 previous similar messages
[14050533.912607] Lustre: oak-OST012f: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[14050533.923674] Lustre: Skipped 499 previous similar messages
[14051132.788762] Lustre: oak-OST014d: Connection restored to 65adbb58-c422-2bfc-8c6a-177e1a0a6a1e (at 10.210.12.73@tcp1)
[14051132.799442] Lustre: Skipped 580 previous similar messages
[14051526.243912] Lustre: oak-OST013f: haven't heard from client 9d61d533-9657-8b18-d48d-c15a11e8c4ac (at 10.50.3.4@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff916722642c00, cur 1645277946 expire 1645277796 last 1645277719
[14051526.265729] Lustre: Skipped 15 previous similar messages
[14051527.221047] Lustre: oak-OST0129: haven't heard from client 84242e94-1811-d567-dd53-2a3a9b096ad5 (at 10.50.3.19@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff919bcb45c400, cur 1645277947 expire 1645277797 last 1645277720
[14051527.242973] Lustre: Skipped 1 previous similar message
[14051529.187427] Lustre: oak-OST0137: haven't heard from client 729a5272-9010-d0bc-0021-9476498d8942 (at 10.50.3.1@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff914d2f240000, cur 1645277949 expire 1645277799 last 1645277722
[14051529.209262] Lustre: Skipped 2 previous similar messages
[14051541.154869] Lustre: oak-OST012b: haven't heard from client f83eedc6-f113-33c3-e98a-e2972ad5ab9b (at 10.50.3.21@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9173fc131000, cur 1645277961 expire 1645277811 last 1645277734
[14051541.176778] Lustre: Skipped 1 previous similar message
[14051548.230673] Lustre: oak-OST0145: haven't heard from client 8265642c-59d5-90a2-d61f-56dda2964590 (at 10.50.3.12@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91df8bde3400, cur 1645277968 expire 1645277818 last 1645277741
[14051732.473801] Lustre: oak-OST014d: Connection restored to 83034250-f855-6c12-471e-38d9aeacf85d (at 10.210.12.54@tcp1)
[14051732.484465] Lustre: Skipped 553 previous similar messages
[14052255.416555] Lustre: oak-OST0143: haven't heard from client bb1094de-8241-7791-a621-6038846a539b (at 10.210.12.62@tcp1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91699e525400, cur 1645278677 expire 1645278527 last 1645278450
[14052255.438555] Lustre: Skipped 1 previous similar message
[14052332.668452] Lustre: oak-OST0133: Connection restored to aab151b2-b669-9bb0-5460-87842297680c (at 10.50.9.57@o2ib2)
[14052332.679067] Lustre: Skipped 647 previous similar messages
[14052931.660205] Lustre: oak-OST012d: Connection restored to  (at 10.51.13.14@o2ib3)
[14052931.667759] Lustre: Skipped 696 previous similar messages
[14052985.479977] md: md17: data-check done.
[14053534.949686] Lustre: oak-OST0149: Connection restored to  (at 10.50.3.50@o2ib2)
[14053534.957156] Lustre: Skipped 702 previous similar messages
[14054135.510634] Lustre: oak-OST0139: Connection restored to 1e1f187a-28cb-d390-d52c-e3db41797544 (at 10.51.15.21@o2ib3)
[14054135.521305] Lustre: Skipped 772 previous similar messages
[14054734.260828] Lustre: oak-OST0111: Connection restored to  (at 10.0.3.12@o2ib5)
[14054734.268249] Lustre: Skipped 530 previous similar messages
[14055336.938260] Lustre: oak-OST0147: Connection restored to  (at 10.50.7.2@o2ib2)
[14055336.945644] Lustre: Skipped 595 previous similar messages
[14055937.238064] Lustre: oak-OST013d: Connection restored to b28a6b2f-d0b4-e8a9-0775-0dab6a037a94 (at 10.51.1.37@o2ib3)
[14055937.248705] Lustre: Skipped 818 previous similar messages
[14056536.332444] Lustre: oak-OST0139: Connection restored to f018ad68-7ac6-ea1c-7424-cc5dc83bee46 (at 10.51.6.9@o2ib3)
[14056536.342934] Lustre: Skipped 643 previous similar messages
[14057135.809848] Lustre: oak-OST0135: Connection restored to 55c1e9f1-1a4d-1f72-179c-f4bf6ddb18c7 (at 10.50.14.7@o2ib2)
[14057135.820431] Lustre: Skipped 695 previous similar messages
[14057734.849756] Lustre: oak-OST0117: Connection restored to 1f6365ca-e337-61b3-c1a8-3cb02b421469 (at 10.51.6.51@o2ib3)
[14057734.860344] Lustre: Skipped 818 previous similar messages
[14058338.120551] Lustre: oak-OST0145: Connection restored to 7349d20f-0127-a226-9837-c301f17bacec (at 10.50.7.30@o2ib2)
[14058338.131139] Lustre: Skipped 688 previous similar messages
[14058938.301840] Lustre: oak-OST0149: Connection restored to 37b5220b-dd12-456d-d6dd-f37c3691dae9 (at 10.50.7.25@o2ib2)
[14058938.312431] Lustre: Skipped 657 previous similar messages
[14059536.880786] Lustre: oak-OST0145: Connection restored to 1e354a56-47c1-6143-541f-2368817925be (at 10.50.17.40@o2ib2)
[14059536.891504] Lustre: Skipped 665 previous similar messages
[14060135.577914] Lustre: oak-OST0117: Connection restored to eea359e6-6b89-c326-60af-8cb99076065b (at 10.210.12.8@tcp1)
[14060135.588500] Lustre: Skipped 986 previous similar messages
[14060734.898341] Lustre: oak-OST012b: Connection restored to  (at 10.50.15.11@o2ib2)
[14060734.905903] Lustre: Skipped 812 previous similar messages
[14061335.303724] Lustre: oak-OST0135: Connection restored to 7069b57c-6d76-196c-bdfd-7eeec28ffa4d (at 10.50.2.44@o2ib2)
[14061335.314306] Lustre: Skipped 859 previous similar messages
[14061941.242675] Lustre: oak-OST0147: Connection restored to f54f42c3-70db-03cc-633b-9f59c27ebab2 (at 10.50.16.18@o2ib2)
[14061941.253341] Lustre: Skipped 889 previous similar messages
[14062541.239266] Lustre: oak-OST0125: Connection restored to 97a324d4-c834-549a-4b22-6ab961bb5783 (at 10.0.3.86@o2ib5)
[14062541.249757] Lustre: Skipped 871 previous similar messages
[14063146.876772] Lustre: oak-OST012f: Connection restored to b64d58fc-68cc-de10-0f1c-7d31660c5f3f (at 10.50.7.49@o2ib2)
[14063146.887355] Lustre: Skipped 913 previous similar messages
[14063746.264605] Lustre: oak-OST0137: Connection restored to 4d4d1720-982f-7514-b35e-269a3b9c9cc0 (at 10.50.2.59@o2ib2)
[14063746.275191] Lustre: Skipped 831 previous similar messages
[14064345.516109] Lustre: oak-OST0117: Connection restored to 44205dca-68c8-d64e-c952-7cce1db00805 (at 10.50.17.13@o2ib2)
[14064345.526866] Lustre: Skipped 895 previous similar messages
[14064944.307520] Lustre: oak-OST0141: Connection restored to eec30a01-36c9-964b-d8a2-6c6b21942773 (at 10.51.15.2@o2ib3)
[14064944.318120] Lustre: Skipped 768 previous similar messages
[14065548.143592] Lustre: oak-OST0143: Connection restored to 609a6db9-8551-97d6-b621-d79e589b5dae (at 10.0.3.34@o2ib5)
[14065548.154092] Lustre: Skipped 628 previous similar messages
[14065891.133880] Lustre: oak-OST0131: Client 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1) reconnecting
[14065891.144288] Lustre: Skipped 16 previous similar messages
[14065904.423801] Lustre: oak-OST0111: Client 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1) reconnecting
[14065904.434213] Lustre: Skipped 6 previous similar messages
[14066147.942359] Lustre: oak-OST0115: Connection restored to b4873b44-db69-85c9-576f-945bfd0ad384 (at 10.0.3.26@o2ib5)
[14066147.952853] Lustre: Skipped 573 previous similar messages
[14066746.661724] Lustre: oak-OST0149: Connection restored to 1135e55c-0bfb-485c-1d0c-23617bcae1bd (at 10.50.7.37@o2ib2)
[14066746.672317] Lustre: Skipped 908 previous similar messages
[14067345.879268] Lustre: oak-OST0131: Connection restored to 6da567f0-c49b-4d35-fa48-7222226d9e96 (at 10.50.10.46@o2ib2)
[14067345.889934] Lustre: Skipped 906 previous similar messages
[14067944.685209] Lustre: oak-OST0137: Connection restored to  (at 10.50.3.52@o2ib2)
[14067944.692753] Lustre: Skipped 907 previous similar messages
[14068139.064130] Lustre: oak-OST013b: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[14068139.954347] LustreError: 21585:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff915287334050 x1714981389145472/t0(0) o4->d4faaef8-44e2-b05f-6116-d9c30b2e4e11@10.210.12.60@tcp1:202/0 lens 488/448 e 0 to 0 dl 1645294692 ref 1 fl Interpret:/0/0 rc 0/0
[14068139.979964] LustreError: 21585:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[14068139.989699] Lustre: oak-OST013b: Bulk IO write error with d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1), client will retry: rc = -110
[14068140.003135] Lustre: Skipped 2 previous similar messages
[14068140.986842] LustreError: 162685:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff917ede7da850 x1714981389145472/t0(0) o4->d4faaef8-44e2-b05f-6116-d9c30b2e4e11@10.210.12.60@tcp1:207/0 lens 488/448 e 0 to 0 dl 1645294697 ref 1 fl Interpret:/2/0 rc 0/0
[14068141.011258] LustreError: 162685:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[14068141.021064] Lustre: oak-OST013b: Bulk IO write error with d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1), client will retry: rc = -110
[14068198.681012] LustreError: 199271:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff91918a49b050 x1714981389186368/t0(0) o3->d4faaef8-44e2-b05f-6116-d9c30b2e4e11@10.210.12.60@tcp1:207/0 lens 552/440 e 0 to 0 dl 1645294697 ref 1 fl Interpret:/0/0 rc 0/0
[14068198.681171] Lustre: oak-OST0117: Bulk IO read error with d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1), client will retry: rc -110
[14068198.681173] Lustre: Skipped 5 previous similar messages
[14068198.725203] LustreError: 199271:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 6 previous similar messages
[14068308.356840] Lustre: oak-OST0143: Client 5a4881be-a0cb-e632-509b-2e15c534b21a (at 10.210.12.9@tcp1) reconnecting
[14068308.367156] Lustre: Skipped 2 previous similar messages
[14068313.894218] Lustre: oak-OST0137: Client 5a4881be-a0cb-e632-509b-2e15c534b21a (at 10.210.12.9@tcp1) reconnecting
[14068313.904537] Lustre: Skipped 22 previous similar messages
[14068480.764887] md: md13: data-check done.
[14068543.239504] Lustre: oak-OST013d: Connection restored to 13782cfa-f64e-3b30-9b19-eac54c74cba1 (at 10.51.15.10@o2ib3)
[14068543.250192] Lustre: Skipped 835 previous similar messages
[14069147.494013] Lustre: oak-OST014b: Connection restored to 72e9c356-8685-a676-ede7-2f53484b502b (at 10.50.10.7@o2ib2)
[14069147.504621] Lustre: Skipped 878 previous similar messages
[14069746.987903] Lustre: oak-OST0141: Connection restored to 98cf0fb1-e0ae-9e7a-9e3c-4c60b6ffce6f (at 10.50.3.32@o2ib2)
[14069746.998509] Lustre: Skipped 855 previous similar messages
[14070345.536218] Lustre: oak-OST0137: Connection restored to de1950ad-3443-5935-1178-b398aae04d6e (at 10.50.6.18@o2ib2)
[14070345.546797] Lustre: Skipped 712 previous similar messages
[14070944.217678] Lustre: oak-OST013d: Connection restored to b9d027e5-37a4-66dc-b22d-59694d3b5404 (at 10.50.10.31@o2ib2)
[14070944.228374] Lustre: Skipped 797 previous similar messages
[14071543.214066] Lustre: oak-OST012f: Connection restored to  (at 10.51.13.12@o2ib3)
[14071543.221618] Lustre: Skipped 3865 previous similar messages
[14072141.875572] Lustre: oak-OST0147: Connection restored to d2f49117-87f4-d939-d915-51fa6430aa6e (at 10.51.12.14@o2ib3)
[14072141.886257] Lustre: Skipped 900 previous similar messages
[14072741.751265] Lustre: oak-OST0125: Connection restored to e64c41d9-97dc-d5e1-1051-e238ba99ebd4 (at 10.50.2.28@o2ib2)
[14072741.761876] Lustre: Skipped 944 previous similar messages
[14073340.924184] Lustre: oak-OST0133: Connection restored to 632d6174-105f-f820-efb3-94e214802e62 (at 10.51.16.24@o2ib3)
[14073340.934904] Lustre: Skipped 612 previous similar messages
[14073940.602953] Lustre: oak-OST011d: Connection restored to 36214d22-20db-1d78-3f7c-5c85d7b7a82d (at 10.51.5.54@o2ib3)
[14073940.613553] Lustre: Skipped 741 previous similar messages
[14074541.577329] Lustre: oak-OST013f: Connection restored to 2ea60ed1-68c7-bf0d-b805-f29015466655 (at 10.50.6.16@o2ib2)
[14074541.587909] Lustre: Skipped 1118 previous similar messages
[14075142.321992] Lustre: oak-OST012d: Connection restored to  (at 10.51.13.14@o2ib3)
[14075142.329549] Lustre: Skipped 1454 previous similar messages
[14075741.160316] Lustre: oak-OST0121: Connection restored to dcd6dcbd-b586-2f53-584c-9ffd51a5c60f (at 10.51.4.50@o2ib3)
[14075741.170896] Lustre: Skipped 1373 previous similar messages
[14076340.667949] Lustre: oak-OST0113: Connection restored to 1135e55c-0bfb-485c-1d0c-23617bcae1bd (at 10.50.7.37@o2ib2)
[14076340.678528] Lustre: Skipped 1301 previous similar messages
[14076939.316272] Lustre: oak-OST0111: Connection restored to edd06616-023a-51fb-d0e1-f960c04746e6 (at 10.0.3.32@o2ib5)
[14076939.326803] Lustre: Skipped 2023 previous similar messages
[14077538.685037] Lustre: oak-OST012f: Connection restored to 9332ec98-8441-e173-4e6c-bb71123298eb (at 10.50.10.3@o2ib2)
[14077538.695623] Lustre: Skipped 1620 previous similar messages
[14078137.828344] Lustre: oak-OST0121: Connection restored to f328dd72-9435-5f0a-0703-7cce3de1cc1e (at 10.50.2.49@o2ib2)
[14078137.838923] Lustre: Skipped 750 previous similar messages
[14078736.668240] Lustre: oak-OST0123: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[14078736.678927] Lustre: Skipped 737 previous similar messages
[14079335.258949] Lustre: oak-OST0149: Connection restored to 73eec9b8-5cb6-07fd-8976-f7320b3ab08e (at 10.50.3.34@o2ib2)
[14079335.269553] Lustre: Skipped 940 previous similar messages
[14079933.927981] Lustre: oak-OST0125: Connection restored to 4025d53e-c01c-f5a6-a7de-c3a1c13db1f4 (at 10.50.10.5@o2ib2)
[14079933.938666] Lustre: Skipped 966 previous similar messages
[14080533.186977] Lustre: oak-OST0149: Connection restored to 5a9c80de-d628-bd5c-f163-6bcf54540cb9 (at 10.51.4.22@o2ib3)
[14080533.197560] Lustre: Skipped 858 previous similar messages
[14081131.778019] Lustre: oak-OST0141: Connection restored to 83f61e92-4c9d-b1fe-42f0-577e2587d888 (at 10.50.7.39@o2ib2)
[14081131.788757] Lustre: Skipped 1032 previous similar messages
[14081730.694886] Lustre: oak-OST0141: Connection restored to 204a8d45-9a5b-7fc5-81b8-08c3dd1c7b49 (at 10.50.13.3@o2ib2)
[14081730.706382] Lustre: Skipped 1148 previous similar messages
[14082335.724107] Lustre: oak-OST0141: Connection restored to  (at 10.50.14.13@o2ib2)
[14082335.731670] Lustre: Skipped 993 previous similar messages
[14082934.331745] Lustre: oak-OST0115: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[14082934.342449] Lustre: Skipped 920 previous similar messages
[14083533.522427] Lustre: oak-OST011f: Connection restored to  (at 10.51.13.14@o2ib3)
[14083533.529982] Lustre: Skipped 955 previous similar messages
[14084133.540682] Lustre: oak-OST0147: Connection restored to 080fbf36-50e2-e037-d579-97f7ea4dd6a6 (at 10.210.12.51@tcp1)
[14084133.551352] Lustre: Skipped 730 previous similar messages
[14084732.596282] Lustre: oak-OST014b: Connection restored to 83034250-f855-6c12-471e-38d9aeacf85d (at 10.210.12.54@tcp1)
[14084732.606962] Lustre: Skipped 817 previous similar messages
[14085333.106160] Lustre: oak-OST0119: Connection restored to 1f6365ca-e337-61b3-c1a8-3cb02b421469 (at 10.51.6.51@o2ib3)
[14085333.116778] Lustre: Skipped 942 previous similar messages
[14085933.270806] Lustre: oak-OST014b: Connection restored to 7353b318-3799-6026-fc0c-4e502f211f86 (at 10.50.2.18@o2ib2)
[14085933.281414] Lustre: Skipped 697 previous similar messages
[14086532.145568] Lustre: oak-OST0137: Connection restored to b5d9daf4-05bf-3542-cb62-fc2b018bd2f0 (at 10.50.14.12@o2ib2)
[14086532.156252] Lustre: Skipped 776 previous similar messages
[14087131.732340] Lustre: oak-OST0141: Connection restored to bbba5ad9-9372-2297-5b66-f72cfc361471 (at 10.0.3.25@o2ib5)
[14087131.742865] Lustre: Skipped 851 previous similar messages
[14087735.638201] Lustre: oak-OST0125: Connection restored to 15431f40-18fe-801a-ce02-67f7cb5f3e18 (at 10.210.12.60@tcp1)
[14087735.648868] Lustre: Skipped 663 previous similar messages
[14088334.217354] Lustre: oak-OST0135: Connection restored to b4873b44-db69-85c9-576f-945bfd0ad384 (at 10.0.3.26@o2ib5)
[14088334.227888] Lustre: Skipped 731 previous similar messages
[14088934.082076] Lustre: oak-OST0123: Connection restored to 116a4f85-56a0-7651-2af5-17ae11376aba (at 10.50.4.70@o2ib2)
[14088934.092750] Lustre: Skipped 931 previous similar messages
[14089532.695798] Lustre: oak-OST0147: Connection restored to 3195013f-4cca-32ee-d13b-67b22cd84ad2 (at 10.50.1.16@o2ib2)
[14089532.706392] Lustre: Skipped 918 previous similar messages
[14090132.100645] Lustre: oak-OST0137: Connection restored to 7bd5e25d-92e5-9f8c-22fa-8f4c98d09593 (at 10.50.10.6@o2ib2)
[14090132.111238] Lustre: Skipped 939 previous similar messages
[14090731.413920] Lustre: oak-OST0121: Connection restored to 5e59651d-b6dc-49b5-4df4-271a7c1a1a6f (at 10.50.12.5@o2ib2)
[14090731.424567] Lustre: Skipped 979 previous similar messages
[14091331.911558] Lustre: oak-OST0123: Connection restored to d58381cd-eaed-70a2-681b-6663c8d28df5 (at 10.210.12.56@tcp1)
[14091331.922222] Lustre: Skipped 792 previous similar messages
[14091930.705383] Lustre: oak-OST012d: Connection restored to b9694c0e-9fc2-f14e-33c8-9018d9fc9806 (at 10.210.12.65@tcp1)
[14091930.716046] Lustre: Skipped 800 previous similar messages
[14092529.295048] Lustre: oak-OST0129: Connection restored to ef8da979-3077-9708-0c8c-a646245f23fe (at 10.50.13.15@o2ib2)
[14092529.305714] Lustre: Skipped 706 previous similar messages
[14093128.527663] Lustre: oak-OST0135: Connection restored to 8dde2243-2229-7a80-3629-22aed9dc1df3 (at 10.50.15.2@o2ib2)
[14093128.538272] Lustre: Skipped 898 previous similar messages
[14093727.938008] Lustre: oak-OST0141: Connection restored to 121d79c7-b34f-dcf4-58e9-962c4afc9126 (at 10.50.8.36@o2ib2)
[14093727.948596] Lustre: Skipped 685 previous similar messages
[14094329.866077] Lustre: oak-OST013f: Connection restored to 080fbf36-50e2-e037-d579-97f7ea4dd6a6 (at 10.210.12.51@tcp1)
[14094329.876767] Lustre: Skipped 655 previous similar messages
[14094928.621182] Lustre: oak-OST0147: Connection restored to ed62306b-c979-ade9-eb31-8485f9927386 (at 10.50.10.65@o2ib2)
[14094928.631846] Lustre: Skipped 743 previous similar messages
[14095527.175297] Lustre: oak-OST0121: Connection restored to af282abc-f991-f641-6b7f-7ad234257b60 (at 10.210.13.37@tcp1)
[14095527.185978] Lustre: Skipped 757 previous similar messages
[14096126.600927] Lustre: oak-OST0139: Connection restored to b4873b44-db69-85c9-576f-945bfd0ad384 (at 10.0.3.26@o2ib5)
[14096126.611423] Lustre: Skipped 681 previous similar messages
[14096725.341642] Lustre: oak-OST013b: Connection restored to  (at 10.51.15.1@o2ib3)
[14096725.349107] Lustre: Skipped 914 previous similar messages
[14097325.193259] Lustre: oak-OST0139: Connection restored to 8fd6aa2c-7a73-8519-5ed8-19de1eb4afc1 (at 10.50.10.1@o2ib2)
[14097325.203842] Lustre: Skipped 904 previous similar messages
[14097925.186778] Lustre: oak-OST0149: Connection restored to  (at 10.51.16.20@o2ib3)
[14097925.194368] Lustre: Skipped 948 previous similar messages
[14098526.743213] Lustre: oak-OST0127: Connection restored to d58381cd-eaed-70a2-681b-6663c8d28df5 (at 10.210.12.56@tcp1)
[14098526.753904] Lustre: Skipped 625 previous similar messages
[14098987.985706] LustreError: 137-5: oak-OST011a_UUID: not available for connect from 10.210.12.40@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14099092.250220] Lustre: oak-OST012b: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[14099092.260638] Lustre: Skipped 19 previous similar messages
[14099118.643732] Lustre: oak-OST013d: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[14099125.642991] Lustre: oak-OST0145: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[14099125.653434] Lustre: oak-OST0145: Connection restored to 3e72271e-d8a9-bff9-a052-ac4ffc30f7d6 (at 10.210.12.40@tcp1)
[14099125.664155] Lustre: Skipped 857 previous similar messages
[14099164.315019] Lustre: oak-OST0119: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[14099164.325423] Lustre: Skipped 1 previous similar message
[14099183.784903] Lustre: oak-OST0125: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[14099220.787731] Lustre: oak-OST011d: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[14099220.798154] Lustre: Skipped 1 previous similar message
[14099264.143809] Lustre: oak-OST0133: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[14099264.154217] Lustre: Skipped 1 previous similar message
[14099365.048260] Lustre: oak-OST0121: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[14099365.058666] Lustre: Skipped 6 previous similar messages
[14099559.229187] Lustre: oak-OST0113: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[14099559.239601] Lustre: Skipped 1 previous similar message
[14099571.612587] md: md39: data-check done.
[14099724.816525] Lustre: oak-OST0117: Connection restored to 5c07de10-ad61-1c19-1e52-cabbbc709d3c (at 10.210.12.10@tcp1)
[14099724.827189] Lustre: Skipped 694 previous similar messages
[14100324.538113] Lustre: oak-OST0129: Connection restored to 352ae6c4-109a-5cfb-ea2d-c62bae5b3a2a (at 10.50.1.8@o2ib2)
[14100324.548623] Lustre: Skipped 664 previous similar messages
[14100924.281952] Lustre: oak-OST0145: Connection restored to 25577177-9587-f924-1703-399fc75a2d0a (at 10.50.9.3@o2ib2)
[14100924.292457] Lustre: Skipped 625 previous similar messages
[14101523.581187] Lustre: oak-OST0127: Connection restored to 471d8473-ce4f-aec7-199c-b5ad6376849b (at 10.50.12.15@o2ib2)
[14101523.591852] Lustre: Skipped 643 previous similar messages
[14102123.523636] Lustre: oak-OST012b: Connection restored to 30b24583-c402-7f66-c166-247bb7ab092f (at 10.51.14.2@o2ib3)
[14102123.534219] Lustre: Skipped 774 previous similar messages
[14102723.227577] Lustre: oak-OST0135: Connection restored to 938e4fcb-40fa-8576-5696-2871684d71bd (at 10.50.5.8@o2ib2)
[14102723.238075] Lustre: Skipped 885 previous similar messages
[14103323.706739] Lustre: oak-OST013d: Connection restored to 5dbda705-a67c-ce47-2e21-0baa3337235a (at 10.50.3.28@o2ib2)
[14103323.717319] Lustre: Skipped 844 previous similar messages
[14103922.856715] Lustre: oak-OST0123: Connection restored to b64d58fc-68cc-de10-0f1c-7d31660c5f3f (at 10.50.7.49@o2ib2)
[14103922.867306] Lustre: Skipped 849 previous similar messages
[14104521.517956] Lustre: oak-OST0133: Connection restored to  (at 10.50.0.64@o2ib2)
[14104521.525430] Lustre: Skipped 855 previous similar messages
[14105120.755049] Lustre: oak-OST012f: Connection restored to 07c826f5-cd86-48e2-c47b-9ebe60acdc95 (at 10.50.14.15@o2ib2)
[14105120.765711] Lustre: Skipped 972 previous similar messages
[14105725.612958] Lustre: oak-OST0149: Connection restored to 566692d9-3760-eb93-5cbc-1337c48c4638 (at 10.50.16.17@o2ib2)
[14105725.623631] Lustre: Skipped 699 previous similar messages
[14106326.567342] Lustre: oak-OST0137: Connection restored to 3ca3d83a-0b3a-619d-a202-22b3aac74f78 (at 10.50.1.53@o2ib2)
[14106326.577960] Lustre: Skipped 617 previous similar messages
[14106927.450331] Lustre: oak-OST013d: Connection restored to fba8005f-f107-35f2-a145-5414b74ab7bf (at 10.51.12.3@o2ib3)
[14106927.460943] Lustre: Skipped 556 previous similar messages
[14107526.421375] Lustre: oak-OST0131: Connection restored to d61a1999-1860-4450-6f4c-b19b832004d7 (at 10.51.14.23@o2ib3)
[14107526.432093] Lustre: Skipped 1503 previous similar messages
[14108125.149513] Lustre: oak-OST0133: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[14108125.160196] Lustre: Skipped 1576 previous similar messages
[14108725.220128] Lustre: oak-OST013f: Connection restored to 2b2b5b47-3d88-dfa6-3173-9352bce15dc5 (at 10.50.5.53@o2ib2)
[14108725.230707] Lustre: Skipped 981 previous similar messages
[14109324.294244] Lustre: oak-OST0149: Connection restored to f363e962-fb98-a66e-131f-da63e1fa0c8f (at 10.50.12.4@o2ib2)
[14109324.304867] Lustre: Skipped 697 previous similar messages
[14109923.857663] Lustre: oak-OST0117: Connection restored to 83034250-f855-6c12-471e-38d9aeacf85d (at 10.210.12.54@tcp1)
[14109923.868339] Lustre: Skipped 722 previous similar messages
[14110525.585852] Lustre: oak-OST0141: Connection restored to dcf50f0e-bb42-f2a0-6cab-85fbec7a53f3 (at 10.51.4.16@o2ib3)
[14110525.596431] Lustre: Skipped 674 previous similar messages
[14111124.732631] Lustre: oak-OST012d: Connection restored to 27d57912-6776-078b-59ad-39eae43109d2 (at 10.50.7.34@o2ib2)
[14111124.743206] Lustre: Skipped 742 previous similar messages
[14111725.097368] Lustre: oak-OST012b: Connection restored to 080fbf36-50e2-e037-d579-97f7ea4dd6a6 (at 10.210.12.51@tcp1)
[14111725.108091] Lustre: Skipped 761 previous similar messages
[14112325.875443] Lustre: oak-OST0111: Connection restored to  (at 10.51.0.65@o2ib3)
[14112325.882929] Lustre: Skipped 728 previous similar messages
[14112927.007823] Lustre: oak-OST012d: Connection restored to 05714c79-1e84-c278-6358-6ecaaa743e1a (at 10.51.7.14@o2ib3)
[14112927.018416] Lustre: Skipped 1204 previous similar messages
[14113527.879596] Lustre: oak-OST014d: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[14113527.890266] Lustre: Skipped 690 previous similar messages
[14114127.150177] Lustre: oak-OST013d: Connection restored to 86260d7e-9477-e551-bf77-7320641a5d5a (at 10.50.5.16@o2ib2)
[14114127.160771] Lustre: Skipped 627 previous similar messages
[14114725.981390] Lustre: oak-OST0133: Connection restored to bce82964-b032-43f4-daa4-4feb0c63cf2c (at 10.50.7.3@o2ib2)
[14114725.991887] Lustre: Skipped 613 previous similar messages
[14115324.558642] Lustre: oak-OST0149: Connection restored to 97cbb879-24a6-a81f-ba52-733dfacac123 (at 10.51.12.9@o2ib3)
[14115324.569270] Lustre: Skipped 583 previous similar messages
[14115923.104207] Lustre: oak-OST0131: Connection restored to  (at 10.51.3.31@o2ib3)
[14115923.111674] Lustre: Skipped 838 previous similar messages
[14116522.781359] Lustre: oak-OST0139: Connection restored to bce82964-b032-43f4-daa4-4feb0c63cf2c (at 10.50.7.3@o2ib2)
[14116522.791877] Lustre: Skipped 2658 previous similar messages
[14117121.567524] Lustre: oak-OST0137: Connection restored to cf02ae25-68b0-47db-54b2-2f0309227169 (at 10.50.2.70@o2ib2)
[14117121.578109] Lustre: Skipped 1432 previous similar messages
[14117720.756608] Lustre: oak-OST0131: Connection restored to 2c092d36-5923-59af-aac5-141371affe6a (at 10.51.4.35@o2ib3)
[14117720.767191] Lustre: Skipped 1424 previous similar messages
[14118320.800013] Lustre: oak-OST013d: Connection restored to 5282df0d-70af-37db-ccf3-a06241b9319e (at 10.51.4.38@o2ib3)
[14118320.810589] Lustre: Skipped 1074 previous similar messages
[14118919.563030] Lustre: oak-OST0129: Connection restored to 116a4f85-56a0-7651-2af5-17ae11376aba (at 10.50.4.70@o2ib2)
[14118919.573610] Lustre: Skipped 1399 previous similar messages
[14119518.234249] Lustre: oak-OST0149: Connection restored to f018ad68-7ac6-ea1c-7424-cc5dc83bee46 (at 10.51.6.9@o2ib3)
[14119518.244746] Lustre: Skipped 734 previous similar messages
[14120116.810569] Lustre: oak-OST011b: Connection restored to 9e481091-2c29-0032-be39-272874617a20 (at 10.50.9.52@o2ib2)
[14120116.821193] Lustre: Skipped 2303 previous similar messages
[14120715.623906] Lustre: oak-OST012d: Connection restored to 5fba3f7f-c9d1-e723-6511-02f291b56eac (at 10.50.17.5@o2ib2)
[14120715.634488] Lustre: Skipped 1776 previous similar messages
[14121013.295326] md: data-check of RAID array md45
[14121019.443906] md: data-check of RAID array md43
[14121025.561645] md: data-check of RAID array md13
[14121031.691527] md: data-check of RAID array md17
[14121037.816003] md: data-check of RAID array md19
[14121043.939827] md: data-check of RAID array md1
[14121050.082176] md: data-check of RAID array md39
[14121315.348303] Lustre: oak-OST0111: Connection restored to efd27206-c371-7156-ea1f-cd9acbf9e1d5 (at 10.50.1.70@o2ib2)
[14121315.358884] Lustre: Skipped 1435 previous similar messages
[14121914.546210] Lustre: oak-OST011f: Connection restored to 2b6419cd-35c8-26e0-6b8b-1e121125f381 (at 10.50.2.42@o2ib2)
[14121914.556798] Lustre: Skipped 1165 previous similar messages
[14122513.316657] Lustre: oak-OST0125: Connection restored to db205550-c1d1-04fd-22e5-8fd53a0a0a3a (at 10.50.6.31@o2ib2)
[14122513.327259] Lustre: Skipped 1196 previous similar messages
[14123113.292641] Lustre: oak-OST014b: Connection restored to  (at 10.50.9.18@o2ib2)
[14123113.300118] Lustre: Skipped 1492 previous similar messages
[14123712.596071] Lustre: oak-OST011b: Connection restored to f94deebd-8de5-a544-3246-a787f9570e29 (at 10.210.12.15@tcp1)
[14123712.606741] Lustre: Skipped 1312 previous similar messages
[14124311.368094] Lustre: oak-OST0121: Connection restored to 3f016e1f-54ab-68d5-4642-4d55843078e2 (at 10.210.11.23@tcp1)
[14124311.378759] Lustre: Skipped 1153 previous similar messages
[14124910.359053] Lustre: oak-OST0113: Connection restored to c77af7fc-49aa-2585-7816-df90a6917753 (at 10.50.17.3@o2ib2)
[14124910.369644] Lustre: Skipped 1000 previous similar messages
[14125509.884067] Lustre: oak-OST0121: Connection restored to 080fbf36-50e2-e037-d579-97f7ea4dd6a6 (at 10.210.12.51@tcp1)
[14125509.894761] Lustre: Skipped 1094 previous similar messages
[14126109.272482] Lustre: oak-OST012b: Connection restored to  (at 10.50.4.3@o2ib2)
[14126109.279865] Lustre: Skipped 907 previous similar messages
[14126709.209560] Lustre: oak-OST0145: Connection restored to 3f016e1f-54ab-68d5-4642-4d55843078e2 (at 10.210.11.23@tcp1)
[14126709.220238] Lustre: Skipped 945 previous similar messages
[14127308.007460] Lustre: oak-OST0129: Connection restored to 3f016e1f-54ab-68d5-4642-4d55843078e2 (at 10.210.11.23@tcp1)
[14127308.018141] Lustre: Skipped 1015 previous similar messages
[14127660.101162] md: md35: data-check done.
[14127906.667318] Lustre: oak-OST0129: Connection restored to  (at 10.50.10.53@o2ib2)
[14127906.674871] Lustre: Skipped 735 previous similar messages
[14128505.870584] Lustre: oak-OST014d: Connection restored to 9f2485c7-01e9-df4d-61cd-9249c4955d05 (at 10.51.2.26@o2ib3)
[14128505.881241] Lustre: Skipped 688 previous similar messages
[14129109.183505] Lustre: oak-OST0135: Connection restored to a0709baf-3ddc-7828-c781-219eb94c1809 (at 10.50.7.69@o2ib2)
[14129109.194091] Lustre: Skipped 626 previous similar messages
[14129708.944475] Lustre: oak-OST0129: Connection restored to  (at 10.50.12.17@o2ib2)
[14129708.952121] Lustre: Skipped 561 previous similar messages
[14130313.374486] Lustre: oak-OST0113: Connection restored to 820a1ce2-6182-e058-f1e9-e3948266ca32 (at 10.51.4.21@o2ib3)
[14130313.385075] Lustre: Skipped 766 previous similar messages
[14130914.277646] Lustre: oak-OST0117: Connection restored to  (at 10.0.2.3@o2ib5)
[14130914.284938] Lustre: Skipped 698 previous similar messages
[14131513.647804] Lustre: oak-OST011f: Connection restored to 2812a9b3-b6f5-4918-ae96-1f2250fc2778 (at 10.50.10.41@o2ib2)
[14131513.658504] Lustre: Skipped 662 previous similar messages
[14132113.619154] Lustre: oak-OST0131: Connection restored to 72e9c356-8685-a676-ede7-2f53484b502b (at 10.50.10.7@o2ib2)
[14132113.629756] Lustre: Skipped 788 previous similar messages
[14132714.246794] Lustre: oak-OST0115: Connection restored to  (at 10.0.2.3@o2ib5)
[14132714.254109] Lustre: Skipped 691 previous similar messages
[14133313.231517] Lustre: oak-OST0133: Connection restored to 80374d94-41e6-6f5b-384d-68674ee94b69 (at 10.50.4.34@o2ib2)
[14133313.242120] Lustre: Skipped 746 previous similar messages
[14133913.593132] Lustre: oak-OST014d: Connection restored to  (at 10.0.2.3@o2ib5)
[14133913.600484] Lustre: Skipped 971 previous similar messages
[14134512.219869] Lustre: oak-OST011f: Connection restored to  (at 10.0.2.3@o2ib5)
[14134512.227260] Lustre: Skipped 791 previous similar messages
[14135111.486897] Lustre: oak-OST0119: Connection restored to 820a1ce2-6182-e058-f1e9-e3948266ca32 (at 10.51.4.21@o2ib3)
[14135111.497490] Lustre: Skipped 696 previous similar messages
[14135710.475069] Lustre: oak-OST011b: Connection restored to b80be752-f851-25c6-d1f2-d3fd377310cf (at 10.210.12.46@tcp1)
[14135710.485736] Lustre: Skipped 776 previous similar messages
[14136309.324162] Lustre: oak-OST0117: Connection restored to  (at 10.0.2.3@o2ib5)
[14136309.331476] Lustre: Skipped 988 previous similar messages
[14136907.875221] Lustre: oak-OST0123: Connection restored to  (at 10.0.2.3@o2ib5)
[14136907.882566] Lustre: Skipped 715 previous similar messages
[14137506.561471] Lustre: oak-OST0135: Connection restored to 5c7bde3e-ffd7-fac1-1b11-c372109835b5 (at 10.51.6.4@o2ib3)
[14137506.571996] Lustre: Skipped 873 previous similar messages
[14138105.174359] Lustre: oak-OST0131: Connection restored to 40a64382-781d-f7f3-5d47-9762a30c7d73 (at 10.51.7.8@o2ib3)
[14138105.184891] Lustre: Skipped 802 previous similar messages
[14138703.817688] Lustre: oak-OST014b: Connection restored to 8a542161-c6da-012a-5f15-3267427cffa2 (at 10.51.5.58@o2ib3)
[14138703.828277] Lustre: Skipped 688 previous similar messages
[14139302.944877] Lustre: oak-OST0137: Connection restored to 82b139ed-5c1e-7b52-4068-9e0eefc516eb (at 10.51.3.48@o2ib3)
[14139302.955470] Lustre: Skipped 1008 previous similar messages
[14139901.841522] Lustre: oak-OST0135: Connection restored to  (at 10.51.15.4@o2ib3)
[14139901.849010] Lustre: Skipped 683 previous similar messages
[14140502.727969] Lustre: oak-OST0129: Connection restored to bbafc8fb-b3eb-ebc6-fdd7-737354a89048 (at 10.51.1.51@o2ib3)
[14140502.738549] Lustre: Skipped 614 previous similar messages
[14141103.390907] Lustre: oak-OST012b: Connection restored to 97a324d4-c834-549a-4b22-6ab961bb5783 (at 10.0.3.86@o2ib5)
[14141103.401461] Lustre: Skipped 702 previous similar messages
[14141703.397531] Lustre: oak-OST0125: Connection restored to 471d8473-ce4f-aec7-199c-b5ad6376849b (at 10.50.12.15@o2ib2)
[14141703.408200] Lustre: Skipped 658 previous similar messages
[14142302.610790] Lustre: oak-OST013b: Connection restored to a6e7d2b7-fc8f-3f38-afc4-932ac9918b00 (at 10.51.15.7@o2ib3)
[14142302.621413] Lustre: Skipped 654 previous similar messages
[14142412.103298] Lustre: oak-OST011d: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[14142412.113708] Lustre: Skipped 25 previous similar messages
[14142412.564961] LustreError: 160930:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91d41e8c8050 x1716249751848192/t0(0) o4->6b341d38-3674-15b6-7e5e-137d0b4498c0@10.210.12.40@tcp1:614/0 lens 488/448 e 0 to 0 dl 1645369094 ref 1 fl Interpret:/0/0 rc 0/0
[14142412.589414] LustreError: 160930:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[14142412.599448] Lustre: oak-OST0129: Bulk IO write error with 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1), client will retry: rc = -110
[14142412.612916] Lustre: Skipped 1 previous similar message
[14142903.401112] Lustre: oak-OST012d: Connection restored to 3e72271e-d8a9-bff9-a052-ac4ffc30f7d6 (at 10.210.12.40@tcp1)
[14142903.411819] Lustre: Skipped 737 previous similar messages
[14143502.112196] Lustre: oak-OST0115: Connection restored to f720ccf2-9c93-97b1-7979-3042158d6592 (at 10.50.17.39@o2ib2)
[14143502.122874] Lustre: Skipped 701 previous similar messages
[14144103.472078] Lustre: oak-OST011f: Connection restored to d2f49117-87f4-d939-d915-51fa6430aa6e (at 10.51.12.14@o2ib3)
[14144103.482759] Lustre: Skipped 1513 previous similar messages
[14144703.524424] Lustre: oak-OST011b: Connection restored to  (at 10.51.5.38@o2ib3)
[14144703.531892] Lustre: Skipped 1154 previous similar messages
[14145305.120033] Lustre: oak-OST0129: Connection restored to 080fbf36-50e2-e037-d579-97f7ea4dd6a6 (at 10.210.12.51@tcp1)
[14145305.130696] Lustre: Skipped 671 previous similar messages
[14145904.759981] Lustre: oak-OST013d: Connection restored to 7b218846-8296-5a90-f251-8d8c57ad58ac (at 10.51.6.3@o2ib3)
[14145904.770490] Lustre: Skipped 753 previous similar messages
[14146505.460926] Lustre: oak-OST0147: Connection restored to  (at 10.51.0.67@o2ib3)
[14146505.468402] Lustre: Skipped 830 previous similar messages
[14147104.313868] Lustre: oak-OST0121: Connection restored to 820a1ce2-6182-e058-f1e9-e3948266ca32 (at 10.51.4.21@o2ib3)
[14147104.324448] Lustre: Skipped 1041 previous similar messages
[14147706.840572] Lustre: oak-OST0111: Connection restored to 6273d9a2-1357-be01-fa1a-e59246cee0f4 (at 10.210.12.105@tcp1)
[14147706.851327] Lustre: Skipped 739 previous similar messages
[14148305.951589] Lustre: oak-OST013d: Connection restored to 09c203ff-6325-0abb-eaa6-bf24067880ae (at 10.50.7.41@o2ib2)
[14148305.962168] Lustre: Skipped 771 previous similar messages
[14148905.319846] Lustre: oak-OST011d: Connection restored to 53f7c8fb-ba13-9226-8f66-6ca61c432359 (at 10.51.12.11@o2ib3)
[14148905.330521] Lustre: Skipped 824 previous similar messages
[14149504.903620] Lustre: oak-OST013d: Connection restored to  (at 10.50.13.8@o2ib2)
[14149504.911092] Lustre: Skipped 828 previous similar messages
[14149892.369210] md: md37: data-check done.
[14150104.228845] Lustre: oak-OST0149: Connection restored to b0c6db9f-4fad-09bf-e7a1-61ea810bc0cf (at 10.50.3.33@o2ib2)
[14150104.239487] Lustre: Skipped 849 previous similar messages
[14150703.514338] Lustre: oak-OST0115: Connection restored to 582484fd-3cd6-e7cb-180b-ae6af9fe1e87 (at 10.50.10.8@o2ib2)
[14150703.524932] Lustre: Skipped 779 previous similar messages
[14151302.658113] Lustre: oak-OST012b: Connection restored to 180617bd-5ecb-4dc6-7ea5-3f6ca4b81e51 (at 10.50.2.57@o2ib2)
[14151302.668755] Lustre: Skipped 814 previous similar messages
[14151903.978824] Lustre: oak-OST011b: Connection restored to a9ab975c-a0d0-5f8a-8d1c-fbe50b233482 (at 10.210.12.71@tcp1)
[14151903.989516] Lustre: Skipped 756 previous similar messages
[14152503.665462] Lustre: oak-OST012b: Connection restored to 32f48011-f5b6-e96a-54bf-48203e3c5ca1 (at 10.51.6.37@o2ib3)
[14152503.676063] Lustre: Skipped 895 previous similar messages
[14153102.503618] Lustre: oak-OST011b: Connection restored to  (at 10.0.2.3@o2ib5)
[14153102.510934] Lustre: Skipped 929 previous similar messages
[14153707.053523] Lustre: oak-OST012d: Connection restored to 313b6b90-f0a1-74dd-0d07-4089e510bdb1 (at 10.50.3.39@o2ib2)
[14153707.064099] Lustre: Skipped 756 previous similar messages
[14154305.601999] Lustre: oak-OST011f: Connection restored to 90511c1a-0572-899c-3ffd-f74d738b1a31 (at 10.50.1.38@o2ib2)
[14154305.612666] Lustre: Skipped 815 previous similar messages
[14154906.230450] Lustre: oak-OST011d: Connection restored to  (at 10.0.2.3@o2ib5)
[14154906.237782] Lustre: Skipped 745 previous similar messages
[14155505.797059] Lustre: oak-OST0117: Connection restored to 632203a5-d0d9-0bd4-bad2-d07d25f9efe1 (at 10.50.1.52@o2ib2)
[14155505.807634] Lustre: Skipped 633 previous similar messages
[14156106.111939] Lustre: oak-OST0135: Connection restored to a80f6d08-4459-2bd2-7cca-724b51375a1b (at 10.50.15.13@o2ib2)
[14156106.122600] Lustre: Skipped 960 previous similar messages
[14156704.981819] Lustre: oak-OST0129: Connection restored to 9e605e4a-e095-4bfd-4f3b-794a85f35288 (at 10.50.10.18@o2ib2)
[14156704.992494] Lustre: Skipped 1057 previous similar messages
[14157304.818109] Lustre: oak-OST0125: Connection restored to  (at 10.50.7.6@o2ib2)
[14157304.825495] Lustre: Skipped 1148 previous similar messages
[14157548.851847] LustreError: 160933:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[14157903.700036] Lustre: oak-OST0139: Connection restored to aaf7e268-3d17-c9bd-43a5-ee3393d496aa (at 10.50.8.49@o2ib2)
[14157903.710620] Lustre: Skipped 1145 previous similar messages
[14158503.509958] Lustre: oak-OST0129: Connection restored to  (at 10.51.16.1@o2ib3)
[14158503.517424] Lustre: Skipped 990 previous similar messages
[14159103.009791] Lustre: oak-OST0141: Connection restored to  (at 10.50.7.6@o2ib2)
[14159103.017172] Lustre: Skipped 879 previous similar messages
[14159703.891162] Lustre: oak-OST0129: Connection restored to 4025d53e-c01c-f5a6-a7de-c3a1c13db1f4 (at 10.50.10.5@o2ib2)
[14159703.901749] Lustre: Skipped 727 previous similar messages
[14160310.857850] Lustre: oak-OST0135: Connection restored to  (at 10.51.15.4@o2ib3)
[14160310.865317] Lustre: Skipped 535 previous similar messages
[14160909.703923] Lustre: oak-OST0115: Connection restored to  (at 10.0.2.3@o2ib5)
[14160909.711295] Lustre: Skipped 634 previous similar messages
[14161509.675576] Lustre: oak-OST011f: Connection restored to 5e4bce85-4d6e-90c6-523c-4bdf35e8bc4a (at 10.210.12.7@tcp1)
[14161509.686156] Lustre: Skipped 887 previous similar messages
[14161763.758520] LustreError: 253934:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91a9060d8850 x1716249839386176/t0(0) o4->6b341d38-3674-15b6-7e5e-137d0b4498c0@10.210.12.40@tcp1:384/0 lens 488/448 e 0 to 0 dl 1645388494 ref 1 fl Interpret:/0/0 rc 0/0
[14161763.758764] Lustre: oak-OST013b: Bulk IO write error with 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1), client will retry: rc = -110
[14161763.797766] LustreError: 253934:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 10 previous similar messages
[14161764.909717] Lustre: oak-OST0111: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[14161764.920130] Lustre: Skipped 24 previous similar messages
[14161794.843245] LustreError: 137-5: oak-OST013a_UUID: not available for connect from 10.210.12.40@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14161794.862683] Lustre: oak-OST0131: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[14162109.407137] Lustre: oak-OST0141: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[14162109.417820] Lustre: Skipped 921 previous similar messages
[14162709.699840] Lustre: oak-OST013b: Connection restored to 2812a9b3-b6f5-4918-ae96-1f2250fc2778 (at 10.50.10.41@o2ib2)
[14162709.710513] Lustre: Skipped 1303 previous similar messages
[14163308.885914] Lustre: oak-OST013d: Connection restored to 278af922-4d46-e32c-1ac8-45d085af6af6 (at 10.50.6.27@o2ib2)
[14163308.896558] Lustre: Skipped 1092 previous similar messages
[14163344.221547] LustreError: 162713:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff915c7e2b6050 x1716249866597824/t0(0) o4->6b341d38-3674-15b6-7e5e-137d0b4498c0@10.210.12.40@tcp1:461/0 lens 488/448 e 0 to 0 dl 1645390081 ref 1 fl Interpret:/0/0 rc 0/0
[14163344.221748] Lustre: oak-OST0131: Bulk IO write error with 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1), client will retry: rc = -110
[14163344.221749] Lustre: Skipped 10 previous similar messages
[14163344.266354] LustreError: 162713:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[14163368.164359] LustreError: 162702:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 0(1130496)  req@ffff91914f942850 x1716249866600768/t0(0) o4->6b341d38-3674-15b6-7e5e-137d0b4498c0@10.210.12.40@tcp1:465/0 lens 488/448 e 0 to 0 dl 1645390085 ref 1 fl Interpret:/0/0 rc 0/0
[14163368.164519] Lustre: oak-OST013b: Bulk IO write error with 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1), client will retry: rc = -110
[14163368.164520] Lustre: Skipped 2 previous similar messages
[14163368.208776] LustreError: 162702:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[14163377.110925] LustreError: 137-5: oak-OST014a_UUID: not available for connect from 10.210.12.29@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14163378.147323] Lustre: oak-OST013b: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[14163378.371397] LustreError: 137-5: oak-OST0140_UUID: not available for connect from 10.210.12.121@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14163464.418154] Lustre: oak-OST0113: Client a2f24d85-30b4-0808-a59e-1ce0f2193fa6 (at 10.210.12.121@tcp1) reconnecting
[14163464.428685] Lustre: Skipped 13 previous similar messages
[14163466.815148] Lustre: oak-OST011b: Client a2f24d85-30b4-0808-a59e-1ce0f2193fa6 (at 10.210.12.121@tcp1) reconnecting
[14163466.825739] Lustre: Skipped 2 previous similar messages
[14163471.515196] Lustre: oak-OST0117: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[14163471.525681] Lustre: Skipped 6 previous similar messages
[14163482.312247] Lustre: oak-OST0117: Client 5865c071-3198-0848-a51e-ecc3ec62e180 (at 10.210.12.127@tcp1) reconnecting
[14163482.322919] Lustre: Skipped 55 previous similar messages
[14163508.495559] Lustre: oak-OST0129: Client d5eb89fe-3ce5-a15e-e193-f0030ed4f8e3 (at 10.210.12.123@tcp1) reconnecting
[14163508.506344] Lustre: Skipped 27 previous similar messages
[14163912.405222] Lustre: oak-OST013b: Connection restored to f1dd735d-6c66-d8b4-50d4-60d2b73caa84 (at 10.51.16.3@o2ib3)
[14163912.415876] Lustre: Skipped 727 previous similar messages
[14164517.488115] Lustre: oak-OST0111: Connection restored to ad12f851-216b-c412-f51f-518aba8ea609 (at 10.210.9.195@tcp1)
[14164517.498815] Lustre: Skipped 458 previous similar messages
[14165117.647973] Lustre: oak-OST011b: Connection restored to ad12f851-216b-c412-f51f-518aba8ea609 (at 10.210.9.195@tcp1)
[14165117.658721] Lustre: Skipped 546 previous similar messages
[14165723.701477] Lustre: oak-OST0111: Connection restored to e10ff487-e8ae-c597-6928-490cf86bc28a (at 10.50.4.28@o2ib2)
[14165723.712057] Lustre: Skipped 453 previous similar messages
[14166322.254196] Lustre: oak-OST012d: Connection restored to 5e4bce85-4d6e-90c6-523c-4bdf35e8bc4a (at 10.210.12.7@tcp1)
[14166322.264786] Lustre: Skipped 592 previous similar messages
[14166923.530632] Lustre: oak-OST0123: Connection restored to ad12f851-216b-c412-f51f-518aba8ea609 (at 10.210.9.195@tcp1)
[14166923.541300] Lustre: Skipped 570 previous similar messages
[14167522.158956] Lustre: oak-OST013d: Connection restored to b4873b44-db69-85c9-576f-945bfd0ad384 (at 10.0.3.26@o2ib5)
[14167522.169445] Lustre: Skipped 695 previous similar messages
[14168123.847349] Lustre: oak-OST0131: Connection restored to 97cbb879-24a6-a81f-ba52-733dfacac123 (at 10.51.12.9@o2ib3)
[14168123.857933] Lustre: Skipped 647 previous similar messages
[14168726.385630] Lustre: oak-OST011f: Connection restored to 2dc703cd-5cca-815b-6dd4-cd3e7d5a0392 (at 10.0.3.49@o2ib5)
[14168726.396129] Lustre: Skipped 561 previous similar messages
[14169326.283102] Lustre: oak-OST0133: Connection restored to aa7f7b58-f62e-bf3c-3911-726ba7f5204c (at 10.0.3.52@o2ib5)
[14169326.293896] Lustre: Skipped 617 previous similar messages
[14169929.953601] Lustre: oak-OST0149: Connection restored to  (at 10.51.16.1@o2ib3)
[14169929.961068] Lustre: Skipped 849 previous similar messages
[14170528.650791] Lustre: oak-OST0143: Connection restored to 2dc703cd-5cca-815b-6dd4-cd3e7d5a0392 (at 10.0.3.49@o2ib5)
[14170528.661321] Lustre: Skipped 539 previous similar messages
[14171130.354707] Lustre: oak-OST012d: Connection restored to cd354002-287c-320d-4786-ef84b940faf7 (at 10.50.9.50@o2ib2)
[14171130.365322] Lustre: Skipped 558 previous similar messages
[14171730.865504] Lustre: oak-OST0143: Connection restored to b06d09e3-c7f9-ed9d-5de1-180a8f296088 (at 10.0.3.5@o2ib5)
[14171730.876493] Lustre: Skipped 712 previous similar messages
[14172332.724286] Lustre: oak-OST0125: Connection restored to 72e9c356-8685-a676-ede7-2f53484b502b (at 10.50.10.7@o2ib2)
[14172332.734958] Lustre: Skipped 932 previous similar messages
[14172937.501733] Lustre: oak-OST0119: Connection restored to  (at 10.0.2.3@o2ib5)
[14172937.509025] Lustre: Skipped 661 previous similar messages
[14173539.134971] Lustre: oak-OST0139: Connection restored to 978cda10-455c-b12a-7f7e-9260335b9b21 (at 10.50.14.6@o2ib2)
[14173539.145578] Lustre: Skipped 583 previous similar messages
[14173826.462349] Lustre: oak-OST0143: Client 274e2965-2f8d-f91e-8ba9-3f4bdaf24c8b (at 10.210.12.119@tcp1) reconnecting
[14173826.472838] Lustre: Skipped 1 previous similar message
[14173827.078168] LustreError: 162712:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9147b351e850 x1714910170055488/t0(0) o4->274e2965-2f8d-f91e-8ba9-3f4bdaf24c8b@10.210.12.119@tcp1:450/0 lens 488/448 e 0 to 0 dl 1645400640 ref 1 fl Interpret:/0/0 rc 0/0
[14173827.102978] Lustre: oak-OST0143: Bulk IO write error with 274e2965-2f8d-f91e-8ba9-3f4bdaf24c8b (at 10.210.12.119@tcp1), client will retry: rc = -110
[14173827.103012] LustreError: 21591:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff91574722d850 x1714910170055488/t0(0) o4->274e2965-2f8d-f91e-8ba9-3f4bdaf24c8b@10.210.12.119@tcp1:451/0 lens 488/448 e 0 to 0 dl 1645400641 ref 1 fl Interpret:/2/0 rc 0/0
[14173827.141201] Lustre: Skipped 2 previous similar messages
[14173994.709990] Lustre: oak-OST0115: Client 274e2965-2f8d-f91e-8ba9-3f4bdaf24c8b (at 10.210.12.119@tcp1) reconnecting
[14173994.720501] Lustre: Skipped 4 previous similar messages
[14174004.769903] Lustre: oak-OST0133: Client 274e2965-2f8d-f91e-8ba9-3f4bdaf24c8b (at 10.210.12.119@tcp1) reconnecting
[14174004.780456] Lustre: Skipped 15 previous similar messages
[14174140.250789] Lustre: oak-OST013d: Connection restored to 22da1096-0ec4-8ed5-8b19-0a2094d2f36e (at 10.50.6.71@o2ib2)
[14174140.261370] Lustre: Skipped 655 previous similar messages
[14174739.509771] Lustre: oak-OST011f: Connection restored to f9253fe3-4f1d-7d5c-4c23-a08286c20d87 (at 10.50.6.15@o2ib2)
[14174739.520357] Lustre: Skipped 768 previous similar messages
[14175342.015857] Lustre: oak-OST013d: Connection restored to  (at 10.0.3.17@o2ib5)
[14175342.023312] Lustre: Skipped 609 previous similar messages
[14175940.860718] Lustre: oak-OST014b: Connection restored to 9e481091-2c29-0032-be39-272874617a20 (at 10.50.9.52@o2ib2)
[14175940.871374] Lustre: Skipped 894 previous similar messages
[14176539.897788] Lustre: oak-OST014d: Connection restored to  (at 10.51.3.47@o2ib3)
[14176539.905258] Lustre: Skipped 2678 previous similar messages
[14177138.649164] Lustre: oak-OST013d: Connection restored to  (at 10.51.4.5@o2ib3)
[14177138.656559] Lustre: Skipped 2335 previous similar messages
[14177737.339989] Lustre: oak-OST0127: Connection restored to 27d57912-6776-078b-59ad-39eae43109d2 (at 10.50.7.34@o2ib2)
[14177737.350576] Lustre: Skipped 3075 previous similar messages
[14178069.738494] Lustre: oak-OST011b: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[14178069.748928] Lustre: Skipped 4 previous similar messages
[14178072.737908] Lustre: oak-OST0135: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[14178072.748322] Lustre: Skipped 5 previous similar messages
[14178082.022439] Lustre: oak-OST0149: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[14178082.032868] Lustre: Skipped 1 previous similar message
[14178095.142955] Lustre: oak-OST0131: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[14178095.153411] Lustre: Skipped 1 previous similar message
[14178336.058505] Lustre: oak-OST012d: Connection restored to ad12f851-216b-c412-f51f-518aba8ea609 (at 10.210.9.195@tcp1)
[14178336.069218] Lustre: Skipped 2060 previous similar messages
[14178935.314139] Lustre: oak-OST013d: Connection restored to eee699b2-cb8e-0a91-6752-be0f592e4945 (at 10.50.13.2@o2ib2)
[14178935.324846] Lustre: Skipped 1207 previous similar messages
[14179534.172413] Lustre: oak-OST011b: Connection restored to 00f73bb3-245d-df73-6e5b-6349096626b0 (at 10.50.2.67@o2ib2)
[14179534.183133] Lustre: Skipped 1306 previous similar messages
[14180133.457754] Lustre: oak-OST0125: Connection restored to 9aa7ef2f-4a2b-0e11-ec94-763eadc83857 (at 10.50.2.69@o2ib2)
[14180133.468337] Lustre: Skipped 665 previous similar messages
[14180732.613299] Lustre: oak-OST0141: Connection restored to ad12f851-216b-c412-f51f-518aba8ea609 (at 10.210.9.195@tcp1)
[14180732.623980] Lustre: Skipped 808 previous similar messages
[14181333.452607] Lustre: oak-OST011b: Connection restored to 1564128a-b82c-a8c0-90e4-b65ea238812e (at 10.210.12.23@tcp1)
[14181333.463277] Lustre: Skipped 1119 previous similar messages
[14181943.756069] Lustre: oak-OST0127: Connection restored to bd2401ee-d808-067a-35c7-f3561e2a7ea5 (at 10.50.6.12@o2ib2)
[14181943.766653] Lustre: Skipped 1092 previous similar messages
[14182542.463791] Lustre: oak-OST0117: Connection restored to a600c6bc-796e-25e6-f7a1-3a35be4d8dbc (at 10.50.10.43@o2ib2)
[14182542.474455] Lustre: Skipped 958 previous similar messages
[14182968.803343] md: md55: data-check done.
[14183142.067373] Lustre: oak-OST014b: Connection restored to aa7f7b58-f62e-bf3c-3911-726ba7f5204c (at 10.0.3.52@o2ib5)
[14183142.077917] Lustre: Skipped 1823 previous similar messages
[14183740.976511] Lustre: oak-OST0147: Connection restored to f1dd735d-6c66-d8b4-50d4-60d2b73caa84 (at 10.51.16.3@o2ib3)
[14183740.987127] Lustre: Skipped 1114 previous similar messages
[14184340.421701] Lustre: oak-OST011b: Connection restored to  (at 10.51.15.9@o2ib3)
[14184340.429165] Lustre: Skipped 913 previous similar messages
[14184938.976615] Lustre: oak-OST013f: Connection restored to 080fbf36-50e2-e037-d579-97f7ea4dd6a6 (at 10.210.12.51@tcp1)
[14184938.987282] Lustre: Skipped 792 previous similar messages
[14185538.072570] Lustre: oak-OST0121: Connection restored to 5cb30d20-25e9-7fca-19eb-f05e49009c40 (at 10.50.6.6@o2ib2)
[14185538.083091] Lustre: Skipped 747 previous similar messages
[14186139.578505] Lustre: oak-OST013d: Connection restored to  (at 10.51.6.31@o2ib3)
[14186139.585970] Lustre: Skipped 1061 previous similar messages
[14186740.462135] Lustre: oak-OST011f: Connection restored to  (at 10.51.6.27@o2ib3)
[14186740.469624] Lustre: Skipped 1088 previous similar messages
[14187339.324571] Lustre: oak-OST0143: Connection restored to aa7f7b58-f62e-bf3c-3911-726ba7f5204c (at 10.0.3.52@o2ib5)
[14187339.335061] Lustre: Skipped 1156 previous similar messages
[14187937.962580] Lustre: oak-OST0113: Connection restored to 79af0850-e83d-5600-b554-ac16b67404e5 (at 10.50.17.27@o2ib2)
[14187937.973246] Lustre: Skipped 1061 previous similar messages
[14188536.993322] Lustre: oak-OST0141: Connection restored to ebf178f6-1771-c187-ac93-409752d2ac2a (at 10.50.9.38@o2ib2)
[14188537.003902] Lustre: Skipped 1209 previous similar messages
[14188579.042857] LustreError: 21608:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff9140f4428050 x1715070864708544/t0(0) o4->bb55a01e-02a3-cbba-5b76-e478fd40967c@10.210.12.54@tcp1:79/0 lens 488/448 e 0 to 0 dl 1645415369 ref 1 fl Interpret:/0/0 rc 0/0
[14188579.048905] Lustre: oak-OST014d: Bulk IO write error with bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1), client will retry: rc = -110
[14188579.048907] Lustre: Skipped 2 previous similar messages
[14188579.087457] LustreError: 21608:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 4 previous similar messages
[14188603.516833] Lustre: oak-OST012b: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[14188603.527278] Lustre: Skipped 6 previous similar messages
[14188696.045798] Lustre: oak-OST0121: Client 5a4881be-a0cb-e632-509b-2e15c534b21a (at 10.210.12.9@tcp1) reconnecting
[14188696.056146] Lustre: Skipped 4 previous similar messages
[14188700.822622] Lustre: oak-OST0115: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[14188700.833087] Lustre: Skipped 18 previous similar messages
[14188711.592225] Lustre: oak-OST013b: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[14188711.602637] Lustre: Skipped 14 previous similar messages
[14188735.493974] Lustre: oak-OST0143: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[14188735.504387] Lustre: Skipped 16 previous similar messages
[14188760.036914] LustreError: 127353:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91a5cc05a050 x1716553770720576/t0(0) o4->6bab8d10-30ff-c1d2-bbf5-1fd0884ac647@10.210.12.57@tcp1:316/0 lens 488/448 e 0 to 0 dl 1645415606 ref 1 fl Interpret:/0/0 rc 0/0
[14188760.061328] LustreError: 127353:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 3 previous similar messages
[14188760.071356] Lustre: oak-OST013d: Bulk IO write error with 6bab8d10-30ff-c1d2-bbf5-1fd0884ac647 (at 10.210.12.57@tcp1), client will retry: rc = -110
[14188760.084819] Lustre: Skipped 4 previous similar messages
[14188761.086437] LustreError: 243449:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91a16c4e9050 x1714982552577728/t0(0) o4->d4faaef8-44e2-b05f-6116-d9c30b2e4e11@10.210.12.60@tcp1:316/0 lens 488/448 e 0 to 0 dl 1645415606 ref 1 fl Interpret:/0/0 rc 0/0
[14188761.110845] LustreError: 243449:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 3 previous similar messages
[14188761.120755] Lustre: oak-OST0145: Bulk IO write error with d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1), client will retry: rc = -110
[14188761.134254] Lustre: Skipped 3 previous similar messages
[14188762.251535] LustreError: 160904:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91dc78d95850 x1716553770819200/t0(0) o4->6bab8d10-30ff-c1d2-bbf5-1fd0884ac647@10.210.12.57@tcp1:320/0 lens 504/448 e 0 to 0 dl 1645415610 ref 1 fl Interpret:/0/0 rc 0/0
[14188931.247319] Lustre: oak-OST013f: Client 6bab8d10-30ff-c1d2-bbf5-1fd0884ac647 (at 10.210.12.57@tcp1) reconnecting
[14188931.257736] Lustre: Skipped 16 previous similar messages
[14188933.250091] LustreError: 243449:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff91d43931d050 x1716553773699200/t0(0) o3->6bab8d10-30ff-c1d2-bbf5-1fd0884ac647@10.210.12.57@tcp1:494/0 lens 488/440 e 0 to 0 dl 1645415784 ref 1 fl Interpret:/0/0 rc 0/0
[14188933.274596] LustreError: 243449:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 2 previous similar messages
[14188933.275204] LustreError: 21608:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk READ failed: rc -107  req@ffff91850b741850 x1716553773699392/t0(0) o3->6bab8d10-30ff-c1d2-bbf5-1fd0884ac647@10.210.12.57@tcp1:494/0 lens 488/440 e 0 to 0 dl 1645415784 ref 1 fl Interpret:/0/0 rc 0/0
[14188933.275206] LustreError: 21608:0:(ldlm_lib.c:3305:target_bulk_io()) Skipped 1 previous similar message
[14188933.275326] Lustre: oak-OST014d: Bulk IO read error with 6bab8d10-30ff-c1d2-bbf5-1fd0884ac647 (at 10.210.12.57@tcp1), client will retry: rc -107
[14188933.275326] Lustre: Skipped 1 previous similar message
[14189135.662368] Lustre: oak-OST0129: Connection restored to 14ad98e5-c9ac-881a-c0f6-1773bc3a28b6 (at 10.50.9.8@o2ib2)
[14189135.672892] Lustre: Skipped 1425 previous similar messages
[14189735.616253] Lustre: oak-OST0145: Connection restored to 9302e624-458d-9703-c596-989a9cd136ca (at 10.50.5.61@o2ib2)
[14189735.626855] Lustre: Skipped 938 previous similar messages
[14190334.769445] Lustre: oak-OST014d: Connection restored to 8dde2243-2229-7a80-3629-22aed9dc1df3 (at 10.50.15.2@o2ib2)
[14190334.780022] Lustre: Skipped 1008 previous similar messages
[14190680.675986] Lustre: oak-OST012f: Client 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1) reconnecting
[14190680.686479] Lustre: Skipped 44 previous similar messages
[14190681.086704] LustreError: 228831:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91c232071050 x1715088110176320/t0(0) o4->65e9b8a8-b461-24e0-da12-5b6939f0b07c@10.210.12.56@tcp1:732/0 lens 488/448 e 0 to 0 dl 1645417532 ref 1 fl Interpret:/0/0 rc 0/0
[14190681.111300] Lustre: oak-OST012f: Bulk IO write error with 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1), client will retry: rc = -110
[14190681.124738] Lustre: Skipped 3 previous similar messages
[14190733.354173] LustreError: 160940:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91b3d3836850 x1715088110176448/t0(0) o4->65e9b8a8-b461-24e0-da12-5b6939f0b07c@10.210.12.56@tcp1:732/0 lens 488/448 e 0 to 0 dl 1645417532 ref 1 fl Interpret:/0/0 rc 0/0
[14190733.354492] Lustre: oak-OST012d: Bulk IO write error with 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1), client will retry: rc = -110
[14190733.393446] LustreError: 160940:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[14190760.503497] Lustre: oak-OST012d: Client 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1) reconnecting
[14190848.619576] Lustre: oak-OST0115: Client 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1) reconnecting
[14190848.629983] Lustre: Skipped 9 previous similar messages
[14190933.800181] Lustre: oak-OST0117: Connection restored to ea7323ea-d4bb-2678-29a1-bc8d87c76269 (at 10.210.12.6@tcp1)
[14190933.810759] Lustre: Skipped 1186 previous similar messages
[14191532.767470] Lustre: oak-OST0139: Connection restored to 43b45cbd-14a3-0e54-7755-76513da7a096 (at 10.51.1.42@o2ib3)
[14191532.778045] Lustre: Skipped 718 previous similar messages
[14192132.776803] Lustre: oak-OST0123: Connection restored to  (at 10.51.6.18@o2ib3)
[14192132.784277] Lustre: Skipped 759 previous similar messages
[14192733.228738] Lustre: oak-OST011d: Connection restored to 7bd5e25d-92e5-9f8c-22fa-8f4c98d09593 (at 10.50.10.6@o2ib2)
[14192733.239402] Lustre: Skipped 616 previous similar messages
[14193331.838012] Lustre: oak-OST0147: Connection restored to ad12f851-216b-c412-f51f-518aba8ea609 (at 10.210.9.195@tcp1)
[14193331.848695] Lustre: Skipped 805 previous similar messages
[14193930.383225] Lustre: oak-OST0119: Connection restored to ee36833d-7a97-4f8f-3fad-12018bb3bd79 (at 10.210.12.9@tcp1)
[14193930.393801] Lustre: Skipped 712 previous similar messages
[14194532.202302] Lustre: oak-OST0133: Connection restored to 8dde2243-2229-7a80-3629-22aed9dc1df3 (at 10.50.15.2@o2ib2)
[14194532.212887] Lustre: Skipped 521 previous similar messages
[14194971.095024] LustreError: 160901:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91a78e13e850 x1715070976131072/t0(0) o4->bb55a01e-02a3-cbba-5b76-e478fd40967c@10.210.12.54@tcp1:444/0 lens 488/448 e 0 to 0 dl 1645421774 ref 1 fl Interpret:/0/0 rc 0/0
[14194971.095263] Lustre: oak-OST014d: Bulk IO write error with bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1), client will retry: rc = -110
[14194971.095264] Lustre: Skipped 1 previous similar message
[14194971.139766] LustreError: 160901:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[14194992.875868] LustreError: 137-5: oak-OST0146_UUID: not available for connect from 10.210.12.54@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14194992.893465] LustreError: Skipped 1 previous similar message
[14194992.896742] Lustre: oak-OST014d: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[14194992.896743] Lustre: Skipped 11 previous similar messages
[14195086.875512] Lustre: oak-OST0129: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[14195086.885927] Lustre: Skipped 2 previous similar messages
[14195102.155948] Lustre: oak-OST013f: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[14195102.166367] Lustre: Skipped 16 previous similar messages
[14195131.255948] Lustre: oak-OST0115: Connection restored to  (at 10.51.15.5@o2ib3)
[14195131.263426] Lustre: Skipped 628 previous similar messages
[14195730.480908] Lustre: oak-OST012f: Connection restored to ad12f851-216b-c412-f51f-518aba8ea609 (at 10.210.9.195@tcp1)
[14195730.491594] Lustre: Skipped 621 previous similar messages
[14196329.949083] Lustre: oak-OST013f: Connection restored to ad12f851-216b-c412-f51f-518aba8ea609 (at 10.210.9.195@tcp1)
[14196329.959744] Lustre: Skipped 509 previous similar messages
[14196661.335403] Lustre: oak-OST0133: Client 47e61d14-8684-47f7-2fe1-8f9f293344d7 (at 10.210.12.6@tcp1) reconnecting
[14196661.345722] Lustre: Skipped 7 previous similar messages
[14196661.681829] LustreError: 243549:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff917522db7050 x1725112687527680/t0(0) o4->47e61d14-8684-47f7-2fe1-8f9f293344d7@10.210.12.6@tcp1:690/0 lens 488/448 e 0 to 0 dl 1645423530 ref 1 fl Interpret:/0/0 rc 0/0
[14196661.706561] Lustre: oak-OST0135: Bulk IO write error with 47e61d14-8684-47f7-2fe1-8f9f293344d7 (at 10.210.12.6@tcp1), client will retry: rc = -110
[14196661.719977] Lustre: Skipped 1 previous similar message
[14196662.300326] LustreError: 21616:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff915c3051a050 x1725112687295360/t0(0) o4->47e61d14-8684-47f7-2fe1-8f9f293344d7@10.210.12.6@tcp1:687/0 lens 488/448 e 0 to 0 dl 1645423527 ref 1 fl Interpret:/0/0 rc 0/0
[14196662.324849] Lustre: oak-OST0133: Bulk IO write error with 47e61d14-8684-47f7-2fe1-8f9f293344d7 (at 10.210.12.6@tcp1), client will retry: rc = -110
[14196928.727240] Lustre: oak-OST0139: Connection restored to f552989a-f6f4-b6d9-c227-dace3bb26afc (at 10.51.1.47@o2ib3)
[14196928.737825] Lustre: Skipped 472 previous similar messages
[14197529.802015] Lustre: oak-OST012f: Connection restored to 228ae6fa-8d3e-0580-91af-ad0d80aefa70 (at 10.51.13.7@o2ib3)
[14197529.812849] Lustre: Skipped 799 previous similar messages
[14197700.183188] LustreError: 160900:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91b8119d9050 x1714908143723648/t0(0) o4->d5eb89fe-3ce5-a15e-e193-f0030ed4f8e3@10.210.12.123@tcp1:159/0 lens 488/448 e 0 to 0 dl 1645424509 ref 1 fl Interpret:/0/0 rc 0/0
[14197700.209471] Lustre: oak-OST012d: Bulk IO write error with d5eb89fe-3ce5-a15e-e193-f0030ed4f8e3 (at 10.210.12.123@tcp1), client will retry: rc = -110
[14197720.005120] Lustre: oak-OST012d: Client d5eb89fe-3ce5-a15e-e193-f0030ed4f8e3 (at 10.210.12.123@tcp1) reconnecting
[14197720.015613] Lustre: Skipped 2 previous similar messages
[14197720.537411] LustreError: 137-5: oak-OST013a_UUID: not available for connect from 10.210.12.119@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14197807.689447] Lustre: oak-OST011d: Client d5eb89fe-3ce5-a15e-e193-f0030ed4f8e3 (at 10.210.12.123@tcp1) reconnecting
[14197807.699960] Lustre: Skipped 8 previous similar messages
[14197809.686102] Lustre: oak-OST011b: Client d5eb89fe-3ce5-a15e-e193-f0030ed4f8e3 (at 10.210.12.123@tcp1) reconnecting
[14197809.696603] Lustre: Skipped 3 previous similar messages
[14197812.738343] Lustre: oak-OST0111: Client 274e2965-2f8d-f91e-8ba9-3f4bdaf24c8b (at 10.210.12.119@tcp1) reconnecting
[14197812.748878] Lustre: Skipped 4 previous similar messages
[14197816.739661] Lustre: oak-OST0121: Client 274e2965-2f8d-f91e-8ba9-3f4bdaf24c8b (at 10.210.12.119@tcp1) reconnecting
[14197816.750168] Lustre: Skipped 29 previous similar messages
[14198128.766186] Lustre: oak-OST0117: Connection restored to  (at 10.0.2.3@o2ib5)
[14198128.773496] Lustre: Skipped 780 previous similar messages
[14198727.673562] Lustre: oak-OST0117: Connection restored to  (at 10.0.2.3@o2ib5)
[14198727.680857] Lustre: Skipped 434 previous similar messages
[14199326.579756] Lustre: oak-OST014d: Connection restored to  (at 10.51.15.9@o2ib3)
[14199326.587227] Lustre: Skipped 567 previous similar messages
[14199926.830810] Lustre: oak-OST0113: Connection restored to  (at 10.0.2.3@o2ib5)
[14199926.838129] Lustre: Skipped 905 previous similar messages
[14200525.783118] Lustre: oak-OST0115: Connection restored to 08a06951-8edb-bdbd-cbdf-3d3b47d52f9e (at 10.0.3.8@o2ib5)
[14200525.793525] Lustre: Skipped 821 previous similar messages
[14201124.654026] Lustre: oak-OST013b: Connection restored to ad12f851-216b-c412-f51f-518aba8ea609 (at 10.210.9.195@tcp1)
[14201124.664747] Lustre: Skipped 2928 previous similar messages
[14201728.937814] Lustre: oak-OST012d: Connection restored to ae16efa7-ec39-2da5-fd55-ba1a1d6bd027 (at 10.50.7.22@o2ib2)
[14201728.948399] Lustre: Skipped 1739 previous similar messages
[14202331.928818] Lustre: oak-OST012b: Connection restored to ea7323ea-d4bb-2678-29a1-bc8d87c76269 (at 10.210.12.6@tcp1)
[14202331.939781] Lustre: Skipped 834 previous similar messages
[14202934.695831] Lustre: oak-OST012d: Connection restored to d602e4c7-90af-04b0-464e-da09170b4eb5 (at 10.51.2.40@o2ib3)
[14202934.706407] Lustre: Skipped 527 previous similar messages
[14203536.452010] Lustre: oak-OST0121: Connection restored to 7bd5e25d-92e5-9f8c-22fa-8f4c98d09593 (at 10.50.10.6@o2ib2)
[14203536.462694] Lustre: Skipped 508 previous similar messages
[14204138.252181] Lustre: oak-OST0127: Connection restored to 27d57912-6776-078b-59ad-39eae43109d2 (at 10.50.7.34@o2ib2)
[14204138.262800] Lustre: Skipped 1050 previous similar messages
[14204736.951957] Lustre: oak-OST0135: Connection restored to  (at 10.50.0.62@o2ib2)
[14204736.959430] Lustre: Skipped 704 previous similar messages
[14205339.811797] Lustre: oak-OST0123: Connection restored to c193930f-3dab-28da-0eac-55422998d272 (at 10.51.13.13@o2ib3)
[14205339.822485] Lustre: Skipped 612 previous similar messages
[14205398.967478] md: md51: data-check done.
[14205941.169277] Lustre: oak-OST011b: Connection restored to ef8da979-3077-9708-0c8c-a646245f23fe (at 10.50.13.15@o2ib2)
[14205941.180039] Lustre: Skipped 574 previous similar messages
[14206539.721325] Lustre: oak-OST011b: Connection restored to b3525ce3-e2b1-9e88-98b3-e4c6720e1d4c (at 10.51.6.19@o2ib3)
[14206539.731907] Lustre: Skipped 669 previous similar messages
[14207138.331559] Lustre: oak-OST0135: Connection restored to b856a4d0-26d2-c794-7253-5babe5526678 (at 10.50.17.42@o2ib2)
[14207138.342231] Lustre: Skipped 634 previous similar messages
[14207737.314107] Lustre: oak-OST0143: Connection restored to ea7323ea-d4bb-2678-29a1-bc8d87c76269 (at 10.210.12.6@tcp1)
[14207737.324680] Lustre: Skipped 571 previous similar messages
[14208336.695923] Lustre: oak-OST0117: Connection restored to  (at 10.50.13.11@o2ib2)
[14208336.703479] Lustre: Skipped 511 previous similar messages
[14208936.899106] Lustre: oak-OST013d: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[14208936.909871] Lustre: Skipped 657 previous similar messages
[14209536.173226] Lustre: oak-OST0125: Connection restored to  (at 10.0.3.17@o2ib5)
[14209536.180656] Lustre: Skipped 605 previous similar messages
[14210136.360358] Lustre: oak-OST0115: Connection restored to  (at 10.50.13.11@o2ib2)
[14210136.367915] Lustre: Skipped 620 previous similar messages
[14210737.873372] Lustre: oak-OST0141: Connection restored to b7a090fc-c548-c007-1623-5f25b59df54e (at 10.51.4.27@o2ib3)
[14210737.883969] Lustre: Skipped 655 previous similar messages
[14211339.060282] Lustre: oak-OST014d: Connection restored to 45c2e5cc-2c98-6e71-2527-5e163cca9c95 (at 10.50.15.1@o2ib2)
[14211339.070864] Lustre: Skipped 627 previous similar messages
[14211939.644126] Lustre: oak-OST011f: Connection restored to 582484fd-3cd6-e7cb-180b-ae6af9fe1e87 (at 10.50.10.8@o2ib2)
[14211939.654711] Lustre: Skipped 581 previous similar messages
[14212538.629216] Lustre: oak-OST014b: Connection restored to 71bc0f4c-3bf2-f6e2-a555-cc123604d10e (at 10.50.16.12@o2ib2)
[14212538.639892] Lustre: Skipped 594 previous similar messages
[14213137.618443] Lustre: oak-OST013f: Connection restored to 097ece3c-9f18-7c64-d1a4-979765e8510b (at 10.0.3.24@o2ib5)
[14213137.628935] Lustre: Skipped 541 previous similar messages
[14213736.409514] Lustre: oak-OST0113: Connection restored to 097ece3c-9f18-7c64-d1a4-979765e8510b (at 10.0.3.24@o2ib5)
[14213736.420056] Lustre: Skipped 547 previous similar messages
[14214335.076019] Lustre: oak-OST0113: Connection restored to 097ece3c-9f18-7c64-d1a4-979765e8510b (at 10.0.3.24@o2ib5)
[14214335.086516] Lustre: Skipped 487 previous similar messages
[14214935.078635] Lustre: oak-OST011d: Connection restored to ad12f851-216b-c412-f51f-518aba8ea609 (at 10.210.9.195@tcp1)
[14214935.089302] Lustre: Skipped 545 previous similar messages
[14215537.116132] Lustre: oak-OST0111: Connection restored to edd06616-023a-51fb-d0e1-f960c04746e6 (at 10.0.3.32@o2ib5)
[14215537.126625] Lustre: Skipped 1290 previous similar messages
[14216136.474532] Lustre: oak-OST012f: Connection restored to e5d4151d-94bb-31d0-aed5-7e54367726dc (at 10.51.4.23@o2ib3)
[14216136.485212] Lustre: Skipped 755 previous similar messages
[14216738.373222] Lustre: oak-OST0125: Connection restored to 471d8473-ce4f-aec7-199c-b5ad6376849b (at 10.50.12.15@o2ib2)
[14216738.383890] Lustre: Skipped 800 previous similar messages
[14217338.087070] Lustre: oak-OST0127: Connection restored to 27d57912-6776-078b-59ad-39eae43109d2 (at 10.50.7.34@o2ib2)
[14217338.097675] Lustre: Skipped 828 previous similar messages
[14217937.353637] Lustre: oak-OST0119: Connection restored to  (at 10.51.15.5@o2ib3)
[14217937.361109] Lustre: Skipped 829 previous similar messages
[14218537.969013] Lustre: oak-OST0143: Connection restored to 43b45cbd-14a3-0e54-7755-76513da7a096 (at 10.51.1.42@o2ib3)
[14218537.979593] Lustre: Skipped 1026 previous similar messages
[14219139.811267] Lustre: oak-OST011b: Connection restored to 9faab410-931d-224c-0590-b37be1578f00 (at 10.51.12.12@o2ib3)
[14219139.821939] Lustre: Skipped 994 previous similar messages
[14219744.807076] Lustre: oak-OST011b: Connection restored to 16c643ca-5cb3-5b84-ccfa-e2c6d3b077dc (at 10.0.3.129@o2ib5)
[14219744.817658] Lustre: Skipped 1914 previous similar messages
[14220033.080863] md: md57: data-check done.
[14220343.442928] Lustre: oak-OST012d: Connection restored to  (at 10.51.2.31@o2ib3)
[14220343.450391] Lustre: Skipped 966 previous similar messages
[14220942.499345] Lustre: oak-OST0119: Connection restored to  (at 10.50.13.11@o2ib2)
[14220942.506898] Lustre: Skipped 670 previous similar messages
[14221544.850776] Lustre: oak-OST0145: Connection restored to 9faab410-931d-224c-0590-b37be1578f00 (at 10.51.12.12@o2ib3)
[14221544.861444] Lustre: Skipped 676 previous similar messages
[14222144.148245] Lustre: oak-OST0119: Connection restored to ad12f851-216b-c412-f51f-518aba8ea609 (at 10.210.9.195@tcp1)
[14222144.158928] Lustre: Skipped 538 previous similar messages
[14222752.879073] Lustre: oak-OST013d: Connection restored to c1347675-c12d-ceec-2964-2e84829a2b24 (at 10.51.2.13@o2ib3)
[14222752.889770] Lustre: Skipped 526 previous similar messages
[14223352.108179] Lustre: oak-OST0123: Connection restored to 882ccd32-3ae0-f471-1b18-684949c470eb (at 10.50.5.44@o2ib2)
[14223352.118767] Lustre: Skipped 566 previous similar messages
[14223954.617908] Lustre: oak-OST012b: Connection restored to 449b91c7-68a2-5c80-cb9f-59caae5bc713 (at 10.51.6.57@o2ib3)
[14223954.628500] Lustre: Skipped 679 previous similar messages
[14224553.405279] Lustre: oak-OST0111: Connection restored to 08a06951-8edb-bdbd-cbdf-3d3b47d52f9e (at 10.0.3.8@o2ib5)
[14224553.415690] Lustre: Skipped 777 previous similar messages
[14225154.453625] Lustre: oak-OST0145: Connection restored to ad12f851-216b-c412-f51f-518aba8ea609 (at 10.210.9.195@tcp1)
[14225154.464329] Lustre: Skipped 760 previous similar messages
[14225757.054895] Lustre: oak-OST0135: Connection restored to ee36833d-7a97-4f8f-3fad-12018bb3bd79 (at 10.210.12.9@tcp1)
[14225757.065480] Lustre: Skipped 648 previous similar messages
[14226358.781207] Lustre: oak-OST013d: Connection restored to 080fbf36-50e2-e037-d579-97f7ea4dd6a6 (at 10.210.12.51@tcp1)
[14226358.791877] Lustre: Skipped 739 previous similar messages
[14226674.129815] md: md49: data-check done.
[14226958.305536] Lustre: oak-OST0117: Connection restored to 7bd5e25d-92e5-9f8c-22fa-8f4c98d09593 (at 10.50.10.6@o2ib2)
[14226958.316111] Lustre: Skipped 793 previous similar messages
[14227558.509231] Lustre: oak-OST013b: Connection restored to 90511c1a-0572-899c-3ffd-f74d738b1a31 (at 10.50.1.38@o2ib2)
[14227558.519811] Lustre: Skipped 769 previous similar messages
[14228157.125360] Lustre: oak-OST0149: Connection restored to  (at 10.50.15.11@o2ib2)
[14228157.132926] Lustre: Skipped 949 previous similar messages
[14228757.651384] Lustre: oak-OST0143: Connection restored to  (at 10.51.15.12@o2ib3)
[14228757.658941] Lustre: Skipped 704 previous similar messages
[14229357.582508] Lustre: oak-OST0133: Connection restored to 7bd5e25d-92e5-9f8c-22fa-8f4c98d09593 (at 10.50.10.6@o2ib2)
[14229357.593102] Lustre: Skipped 548 previous similar messages
[14229958.424037] Lustre: oak-OST012f: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[14229958.434747] Lustre: Skipped 688 previous similar messages
[14230557.489636] Lustre: oak-OST0139: Connection restored to 08e827f3-caa3-2cb0-507e-5bd8243ac158 (at 10.51.12.2@o2ib3)
[14230557.500224] Lustre: Skipped 1018 previous similar messages
[14231156.150754] Lustre: oak-OST011f: Connection restored to 963f25d3-89fe-9827-147e-b4bc4770d3be (at 10.0.3.127@o2ib5)
[14231156.161331] Lustre: Skipped 700 previous similar messages
[14231755.015000] Lustre: oak-OST011d: Connection restored to 11c447ed-34d1-2f48-2b83-a19364bb5a74 (at 10.210.12.107@tcp1)
[14231755.025801] Lustre: Skipped 652 previous similar messages
[14232057.820505] Lustre: 229331:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1645458763/real 1645458763]  req@ffff91b4f19d9f80 x1710533674612480/t0(0) o106->oak-OST013f@10.210.12.7@tcp1:15/16 lens 296/280 e 0 to 1 dl 1645458916 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
[14232105.442721] LNet: Service thread pid 229331 was inactive for 200.73s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
[14232105.459972] Pid: 229331, comm: ll_ost00_025 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14232105.470969] Call Trace:
[14232105.473704]  [<ffffffffc1072310>] ptlrpc_set_wait+0x480/0x790 [ptlrpc]
[14232105.480568]  [<ffffffffc102f145>] ldlm_run_ast_work+0xd5/0x3a0 [ptlrpc]
[14232105.487483]  [<ffffffffc105054b>] ldlm_glimpse_locks+0x3b/0x100 [ptlrpc]
[14232105.494542]  [<ffffffffc168762b>] ofd_intent_policy+0x69b/0x920 [ofd]
[14232105.501275]  [<ffffffffc10304e6>] ldlm_lock_enqueue+0x376/0x9b0 [ptlrpc]
[14232105.508300]  [<ffffffffc1057a16>] ldlm_handle_enqueue0+0xa86/0x1620 [ptlrpc]
[14232105.515643]  [<ffffffffc10e0a42>] tgt_enqueue+0x62/0x210 [ptlrpc]
[14232105.522038]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14232105.529206]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14232105.537161]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14232105.543721]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14232105.548895]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14232105.555619]  [<ffffffffffffffff>] 0xffffffffffffffff
[14232105.560884] LustreError: dumping log to /tmp/lustre-log.1645458964.229331
[14232130.238513] Lustre: oak-OST014d: haven't heard from client 1b588188-9255-1b9f-1308-154ba9e467a3 (at 10.210.12.7@tcp1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff913c8a7f0800, cur 1645458989 expire 1645458839 last 1645458762
[14232130.260510] Lustre: Skipped 29 previous similar messages
[14232130.360483] LNet: Service thread pid 229331 completed after 225.71s. This indicates the system was overloaded (too many service threads, or there were not enough hardware resources).
[14232130.364887] LustreError: 230165:0:(client.c:1210:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff91ca8e511f80 x1710533683097600/t0(0) o106->oak-OST013f@10.210.12.7@tcp1:15/16 lens 296/280 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1
[14232356.515346] Lustre: oak-OST011b: Connection restored to 1564128a-b82c-a8c0-90e4-b65ea238812e (at 10.210.12.23@tcp1)
[14232356.526061] Lustre: Skipped 3415 previous similar messages
[14232956.232545] Lustre: oak-OST014d: Connection restored to 0daf0491-344e-4941-a1c9-9dd34a1df9a9 (at 10.50.5.67@o2ib2)
[14232956.243167] Lustre: Skipped 1007 previous similar messages
[14233555.234344] Lustre: oak-OST013f: Connection restored to  (at 10.51.2.34@o2ib3)
[14233555.241813] Lustre: Skipped 1262 previous similar messages
[14233675.401062] Lustre: oak-OST0131: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[14233675.411502] Lustre: Skipped 7 previous similar messages
[14233675.848765] LustreError: 127352:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91d4f0f5e050 x1716250438976896/t0(0) o4->6b341d38-3674-15b6-7e5e-137d0b4498c0@10.210.12.40@tcp1:41/0 lens 488/448 e 0 to 0 dl 1645460631 ref 1 fl Interpret:/0/0 rc 0/0
[14233675.873320] Lustre: oak-OST0131: Bulk IO write error with 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1), client will retry: rc = -110
[14233676.740108] Lustre: oak-OST0131: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[14233677.413010] LustreError: 160931:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91b4da719850 x1716250438976768/t0(0) o4->6b341d38-3674-15b6-7e5e-137d0b4498c0@10.210.12.40@tcp1:45/0 lens 488/448 e 0 to 0 dl 1645460635 ref 1 fl Interpret:/2/0 rc 0/0
[14233677.437357] LustreError: 160931:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[14233677.447238] Lustre: oak-OST0131: Bulk IO write error with 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1), client will retry: rc = -110
[14233677.460668] Lustre: Skipped 1 previous similar message
[14233732.608647] LustreError: 127349:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91b4da71a050 x1715071320539008/t0(0) o4->bb55a01e-02a3-cbba-5b76-e478fd40967c@10.210.12.54@tcp1:41/0 lens 488/448 e 0 to 0 dl 1645460631 ref 1 fl Interpret:/0/0 rc 0/0
[14233732.608902] Lustre: oak-OST014d: Bulk IO write error with bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1), client will retry: rc = -110
[14233732.608903] Lustre: Skipped 2 previous similar messages
[14233732.653508] LustreError: 127349:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[14233754.551082] LustreError: 137-5: oak-OST0134_UUID: not available for connect from 10.210.12.40@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14233755.665613] Lustre: oak-OST014d: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[14233842.522199] Lustre: oak-OST0129: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[14233851.170724] Lustre: oak-OST0119: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[14233851.181132] Lustre: Skipped 29 previous similar messages
[14233896.872993] md: md47: data-check done.
[14234139.720194] LustreError: 162698:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff9178bc9f2850 x1715053919931712/t0(0) o4->09198db0-ac48-c42e-4de3-f0cdbdb971ff@10.210.12.36@tcp1:454/0 lens 488/448 e 0 to 0 dl 1645461044 ref 1 fl Interpret:/0/0 rc 0/0
[14234139.728344] Lustre: oak-OST0145: Bulk IO write error with 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1), client will retry: rc = -110
[14234139.728345] Lustre: Skipped 2 previous similar messages
[14234139.764987] LustreError: 162698:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 5 previous similar messages
[14234154.515349] Lustre: oak-OST0117: Connection restored to c6e5ec6f-5fa9-82f2-d1c1-5ff9382e19ed (at 10.51.6.22@o2ib3)
[14234154.525934] Lustre: Skipped 940 previous similar messages
[14234169.065858] Lustre: oak-OST0145: Client 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1) reconnecting
[14234169.076311] Lustre: Skipped 14 previous similar messages
[14234261.585192] Lustre: oak-OST0123: Client 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1) reconnecting
[14234326.509781] Lustre: oak-OST0117: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[14234326.520195] Lustre: Skipped 28 previous similar messages
[14234754.623552] Lustre: oak-OST013b: Connection restored to e61413af-4a2b-dfce-a1d1-57c66fcf951f (at 10.50.7.14@o2ib2)
[14234754.634135] Lustre: Skipped 871 previous similar messages
[14235354.467165] Lustre: oak-OST011d: Connection restored to c0e7578b-1fe2-9f68-a510-ed7d565338f0 (at 10.50.10.10@o2ib2)
[14235354.477849] Lustre: Skipped 752 previous similar messages
[14235954.038309] Lustre: oak-OST0125: Connection restored to  (at 10.51.12.23@o2ib3)
[14235954.045882] Lustre: Skipped 787 previous similar messages
[14236553.070518] Lustre: oak-OST0145: Connection restored to 028558ee-1ea8-1c9e-54d2-65c3ca73525f (at 10.50.9.41@o2ib2)
[14236553.081195] Lustre: Skipped 911 previous similar messages
[14237151.652069] Lustre: oak-OST0113: Connection restored to 4c9e0256-e075-dfd1-6ae9-4ef369f624bd (at 10.51.7.6@o2ib3)
[14237151.662567] Lustre: Skipped 1329 previous similar messages
[14237750.704079] Lustre: oak-OST0129: Connection restored to 963f25d3-89fe-9827-147e-b4bc4770d3be (at 10.0.3.127@o2ib5)
[14237750.714668] Lustre: Skipped 1052 previous similar messages
[14238350.500240] Lustre: oak-OST0115: Connection restored to 097ece3c-9f18-7c64-d1a4-979765e8510b (at 10.0.3.24@o2ib5)
[14238350.510747] Lustre: Skipped 1162 previous similar messages
[14238950.257238] Lustre: oak-OST0123: Connection restored to 0b9c22f9-64db-6776-985a-6f5501eb15d2 (at 10.51.6.7@o2ib3)
[14238950.267836] Lustre: Skipped 1069 previous similar messages
[14239549.560798] Lustre: oak-OST014b: Connection restored to 95445bc6-31c8-d381-4d7e-81960375af0e (at 10.50.4.25@o2ib2)
[14239549.571385] Lustre: Skipped 1114 previous similar messages
[14240149.728776] Lustre: oak-OST0129: Connection restored to 8d9f00a2-9995-03f1-abe9-54d67a777fc3 (at 10.50.4.42@o2ib2)
[14240149.739383] Lustre: Skipped 1124 previous similar messages
[14240750.510084] Lustre: oak-OST012d: Connection restored to 963f25d3-89fe-9827-147e-b4bc4770d3be (at 10.0.3.127@o2ib5)
[14240750.521708] Lustre: Skipped 1089 previous similar messages
[14241352.576796] Lustre: oak-OST0127: Connection restored to 27d57912-6776-078b-59ad-39eae43109d2 (at 10.50.7.34@o2ib2)
[14241352.587426] Lustre: Skipped 960 previous similar messages
[14241954.084947] Lustre: oak-OST0115: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[14241954.095617] Lustre: Skipped 959 previous similar messages
[14242553.676016] Lustre: oak-OST0119: Connection restored to 1d43f61d-d839-b43a-c616-54b32079590c (at 10.51.6.66@o2ib3)
[14242553.686630] Lustre: Skipped 1746 previous similar messages
[14243152.252163] Lustre: oak-OST0141: Connection restored to 42228356-6056-ae58-809d-d255d66b2a5d (at 10.50.9.55@o2ib2)
[14243152.262758] Lustre: Skipped 1249 previous similar messages
[14243750.968027] Lustre: oak-OST0137: Connection restored to 963f25d3-89fe-9827-147e-b4bc4770d3be (at 10.0.3.127@o2ib5)
[14243750.978610] Lustre: Skipped 1147 previous similar messages
[14244350.200248] Lustre: oak-OST0133: Connection restored to ecca0655-c675-594a-d893-1abfbf4afcb2 (at 10.51.6.71@o2ib3)
[14244350.210836] Lustre: Skipped 1788 previous similar messages
[14244948.896167] Lustre: oak-OST0129: Connection restored to d90e169d-4372-25c4-60a4-1558ef84b94e (at 10.50.4.44@o2ib2)
[14244948.906769] Lustre: Skipped 1388 previous similar messages
[14245548.518960] Lustre: oak-OST0119: Connection restored to bd65b2a3-48a0-d6f2-6e15-a5dd696dd3c7 (at 10.51.6.8@o2ib3)
[14245548.529454] Lustre: Skipped 1450 previous similar messages
[14246148.345752] Lustre: oak-OST013f: Connection restored to 4f0d1b43-9180-ff69-8774-a1ec250b2851 (at 10.51.6.56@o2ib3)
[14246148.356351] Lustre: Skipped 2226 previous similar messages
[14246747.236931] Lustre: oak-OST014b: Connection restored to  (at 10.51.15.15@o2ib3)
[14246747.244500] Lustre: Skipped 1011 previous similar messages
[14246971.172230] Lustre: oak-OST0141: haven't heard from client 3419dd96-5265-4302-2810-26af6caeb5e7 (at 10.51.14.2@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91de407b5c00, cur 1645473866 expire 1645473716 last 1645473639
[14246971.194161] Lustre: Skipped 30 previous similar messages
[14247346.391736] Lustre: oak-OST012b: Connection restored to  (at 10.50.0.61@o2ib2)
[14247346.399206] Lustre: Skipped 985 previous similar messages
[14247945.241009] Lustre: oak-OST014b: Connection restored to a76603f3-4525-1b87-8e4d-96d9aba3f722 (at 10.51.12.19@o2ib3)
[14247945.251676] Lustre: Skipped 1239 previous similar messages
[14247979.156160] LustreError: 160931:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91d90b3dd850 x1715071523813824/t0(0) o4->bb55a01e-02a3-cbba-5b76-e478fd40967c@10.210.12.54@tcp1:737/0 lens 488/448 e 0 to 0 dl 1645474917 ref 1 fl Interpret:/0/0 rc 0/0
[14247979.156384] Lustre: oak-OST0143: Bulk IO write error with bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1), client will retry: rc = -110
[14247979.156385] Lustre: Skipped 5 previous similar messages
[14247979.200956] LustreError: 160931:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 4 previous similar messages
[14248013.012198] Lustre: oak-OST014d: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[14248013.022623] Lustre: Skipped 13 previous similar messages
[14248092.471879] LustreError: 137-5: oak-OST012e_UUID: not available for connect from 10.210.12.79@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14248094.470000] Lustre: oak-OST0125: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[14248179.505040] Lustre: oak-OST0137: Client e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1) reconnecting
[14248179.515449] Lustre: Skipped 77 previous similar messages
[14248543.990614] Lustre: oak-OST0139: Connection restored to  (at 10.51.6.27@o2ib3)
[14248543.998087] Lustre: Skipped 1353 previous similar messages
[14249142.859550] Lustre: oak-OST0113: Connection restored to f88b6f15-4e50-3090-d8f4-09e1cf7961f2 (at 10.50.1.1@o2ib2)
[14249142.870040] Lustre: Skipped 1405 previous similar messages
[14249741.424559] Lustre: oak-OST011f: Connection restored to f1529a1b-c253-04fd-9843-12715a2efac6 (at 10.50.10.61@o2ib2)
[14249741.435224] Lustre: Skipped 1166 previous similar messages
[14250340.376100] Lustre: oak-OST0141: Connection restored to 45af99cf-0929-72df-1da2-a7dc0bc2a5cf (at 10.51.2.17@o2ib3)
[14250340.387178] Lustre: Skipped 1365 previous similar messages
[14250939.921896] Lustre: oak-OST0137: Connection restored to 963f25d3-89fe-9827-147e-b4bc4770d3be (at 10.0.3.127@o2ib5)
[14250939.932514] Lustre: Skipped 1271 previous similar messages
[14251539.768122] Lustre: oak-OST0131: Connection restored to 7d4a1d93-5a4b-b2e6-ba3e-364f757657c1 (at 10.51.6.45@o2ib3)
[14251539.778715] Lustre: Skipped 1214 previous similar messages
[14251956.996595] LustreError: 162701:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[14252138.323921] Lustre: oak-OST0127: Connection restored to de706e46-5416-1ac2-3b77-d981bced830c (at 10.51.6.52@o2ib3)
[14252138.334573] Lustre: Skipped 1365 previous similar messages
[14252736.936912] Lustre: oak-OST0147: Connection restored to e6999c1b-d689-7657-00e4-f9b8d18f6b38 (at 10.50.7.51@o2ib2)
[14252736.947492] Lustre: Skipped 1073 previous similar messages
[14253336.392425] Lustre: oak-OST0127: Connection restored to 34e56232-180b-3d78-2332-b3cac39d3580 (at 10.50.1.23@o2ib2)
[14253336.403028] Lustre: Skipped 1138 previous similar messages
[14253935.984626] Lustre: oak-OST0147: Connection restored to b452e192-b21d-9910-e9e2-904895baaeee (at 10.51.5.6@o2ib3)
[14253935.995142] Lustre: Skipped 1250 previous similar messages
[14254534.668594] Lustre: oak-OST0135: Connection restored to 26501873-8a2d-bbe3-98a8-1eec4caca93e (at 10.51.1.46@o2ib3)
[14254534.679178] Lustre: Skipped 1344 previous similar messages
[14255133.772345] Lustre: oak-OST0131: Connection restored to 0fe742b7-a217-ae9d-9fcb-f22d8a6d13b0 (at 10.50.9.29@o2ib2)
[14255133.782929] Lustre: Skipped 1192 previous similar messages
[14255734.176687] Lustre: oak-OST0115: Connection restored to cd354002-287c-320d-4786-ef84b940faf7 (at 10.50.9.50@o2ib2)
[14255734.187274] Lustre: Skipped 1310 previous similar messages
[14256332.887744] Lustre: oak-OST013d: Connection restored to dd48517b-1edb-5244-25f4-bdf75572f026 (at 10.50.1.19@o2ib2)
[14256332.898322] Lustre: Skipped 1578 previous similar messages
[14256931.731590] Lustre: oak-OST0133: Connection restored to ae16efa7-ec39-2da5-fd55-ba1a1d6bd027 (at 10.50.7.22@o2ib2)
[14256931.742170] Lustre: Skipped 1037 previous similar messages
[14257530.931365] Lustre: oak-OST0133: Connection restored to d2f49117-87f4-d939-d915-51fa6430aa6e (at 10.51.12.14@o2ib3)
[14257530.942035] Lustre: Skipped 1179 previous similar messages
[14258130.495550] Lustre: oak-OST0117: Connection restored to 684a9b09-ef4a-553f-fc09-c29c5b4ead46 (at 10.210.12.49@tcp1)
[14258130.506221] Lustre: Skipped 1146 previous similar messages
[14258729.717938] Lustre: oak-OST012f: Connection restored to f1db0cb0-5cee-ccf9-6484-5189f751ad99 (at 10.51.0.63@o2ib3)
[14258729.728572] Lustre: Skipped 939 previous similar messages
[14259330.472995] Lustre: oak-OST013d: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[14259330.483663] Lustre: Skipped 938 previous similar messages
[14259929.693372] Lustre: oak-OST011d: Connection restored to b8ee58d3-1671-02fb-863e-61099182d3fc (at 10.50.14.9@o2ib2)
[14259929.703962] Lustre: Skipped 1059 previous similar messages
[14260528.857927] Lustre: oak-OST0141: Connection restored to 73559a48-c9ef-b253-40bd-55a01f5f935d (at 10.50.2.20@o2ib2)
[14260528.868537] Lustre: Skipped 1156 previous similar messages
[14261127.458536] Lustre: oak-OST0145: Connection restored to  (at 10.50.14.14@o2ib2)
[14261127.466138] Lustre: Skipped 1107 previous similar messages
[14261726.295903] Lustre: oak-OST0115: Connection restored to  (at 10.51.15.13@o2ib3)
[14261726.303537] Lustre: Skipped 843 previous similar messages
[14262306.009913] Lustre: oak-OST0119: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[14262306.020352] Lustre: Skipped 30 previous similar messages
[14262325.983797] Lustre: oak-OST0139: Connection restored to 981a916c-30aa-1526-5783-d49f11ad590c (at 10.50.7.55@o2ib2)
[14262325.994421] Lustre: Skipped 1577 previous similar messages
[14262925.905321] Lustre: oak-OST0123: Connection restored to b2007b39-9285-7818-253f-56a583205455 (at 10.51.6.1@o2ib3)
[14262925.915817] Lustre: Skipped 2252 previous similar messages
[14263524.569553] Lustre: oak-OST013b: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[14263524.580230] Lustre: Skipped 1570 previous similar messages
[14264123.172905] Lustre: oak-OST0137: Connection restored to 7bd5e25d-92e5-9f8c-22fa-8f4c98d09593 (at 10.50.10.6@o2ib2)
[14264123.183483] Lustre: Skipped 978 previous similar messages
[14264722.747423] Lustre: oak-OST0127: Connection restored to b633a03e-e51b-9874-c6e6-799b3eb86948 (at 10.50.7.60@o2ib2)
[14264722.758001] Lustre: Skipped 1560 previous similar messages
[14265321.344684] Lustre: oak-OST0143: Connection restored to 9e481091-2c29-0032-be39-272874617a20 (at 10.50.9.52@o2ib2)
[14265321.355264] Lustre: Skipped 1116 previous similar messages
[14265919.993373] Lustre: oak-OST014b: Connection restored to 963f25d3-89fe-9827-147e-b4bc4770d3be (at 10.0.3.127@o2ib5)
[14265920.003974] Lustre: Skipped 953 previous similar messages
[14266518.648060] Lustre: oak-OST0111: Connection restored to  (at 10.51.6.17@o2ib3)
[14266518.655523] Lustre: Skipped 929 previous similar messages
[14267118.283614] Lustre: oak-OST0139: Connection restored to bbafc8fb-b3eb-ebc6-fdd7-737354a89048 (at 10.51.1.51@o2ib3)
[14267118.294193] Lustre: Skipped 917 previous similar messages
[14267159.214231] Lustre: oak-OST0111: Client 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1) reconnecting
[14267159.224659] Lustre: Skipped 63 previous similar messages
[14267209.098552] LustreError: 137-5: oak-OST0140_UUID: not available for connect from 10.210.12.36@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14267720.433731] Lustre: oak-OST0119: Connection restored to  (at 10.50.12.17@o2ib2)
[14267720.441294] Lustre: Skipped 1372 previous similar messages
[14268319.668522] Lustre: oak-OST0141: Connection restored to 963f25d3-89fe-9827-147e-b4bc4770d3be (at 10.0.3.127@o2ib5)
[14268319.679125] Lustre: Skipped 1047 previous similar messages
[14268869.583332] Lustre: oak-OST014d: Client b4fca75a-2d54-9bf2-331f-67d0c52e5ff2 (at 10.210.12.73@tcp1) reconnecting
[14268870.546626] LustreError: 243534:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff917d0453e850 x1715110814897344/t0(0) o4->b4fca75a-2d54-9bf2-331f-67d0c52e5ff2@10.210.12.73@tcp1:590/0 lens 488/448 e 0 to 0 dl 1645495910 ref 1 fl Interpret:/0/0 rc 0/0
[14268870.571041] LustreError: 243534:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 2 previous similar messages
[14268870.580967] Lustre: oak-OST014d: Bulk IO write error with b4fca75a-2d54-9bf2-331f-67d0c52e5ff2 (at 10.210.12.73@tcp1), client will retry: rc = -110
[14268870.594405] Lustre: Skipped 4 previous similar messages
[14268918.407471] Lustre: oak-OST0145: Connection restored to  (at 10.51.15.9@o2ib3)
[14268918.414938] Lustre: Skipped 1283 previous similar messages
[14268928.754391] LustreError: 243498:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff917768e10850 x1715110814900928/t0(0) o3->b4fca75a-2d54-9bf2-331f-67d0c52e5ff2@10.210.12.73@tcp1:593/0 lens 488/440 e 0 to 0 dl 1645495913 ref 1 fl Interpret:/0/0 rc 0/0
[14268928.780209] Lustre: oak-OST012f: Bulk IO read error with b4fca75a-2d54-9bf2-331f-67d0c52e5ff2 (at 10.210.12.73@tcp1), client will retry: rc -110
[14268928.793388] Lustre: Skipped 3 previous similar messages
[14269040.215464] Lustre: oak-OST012f: Client b4fca75a-2d54-9bf2-331f-67d0c52e5ff2 (at 10.210.12.73@tcp1) reconnecting
[14269042.287793] Lustre: oak-OST014d: Client b4fca75a-2d54-9bf2-331f-67d0c52e5ff2 (at 10.210.12.73@tcp1) reconnecting
[14269048.882599] Lustre: oak-OST0145: Client b4fca75a-2d54-9bf2-331f-67d0c52e5ff2 (at 10.210.12.73@tcp1) reconnecting
[14269048.893090] Lustre: Skipped 2 previous similar messages
[14269517.574697] Lustre: oak-OST0141: Connection restored to 963f25d3-89fe-9827-147e-b4bc4770d3be (at 10.0.3.127@o2ib5)
[14269517.585277] Lustre: Skipped 851 previous similar messages
[14270117.257652] Lustre: oak-OST0149: Connection restored to 1b67ac58-7108-cf3f-afa6-c007e56ee1b6 (at 10.50.8.67@o2ib2)
[14270117.268231] Lustre: Skipped 914 previous similar messages
[14270717.461514] Lustre: oak-OST0119: Connection restored to eea359e6-6b89-c326-60af-8cb99076065b (at 10.210.12.8@tcp1)
[14270717.472103] Lustre: Skipped 834 previous similar messages
[14271316.593451] Lustre: oak-OST012b: Connection restored to  (at 10.50.7.5@o2ib2)
[14271316.600851] Lustre: Skipped 735 previous similar messages
[14271915.711737] Lustre: oak-OST0149: Connection restored to ea7323ea-d4bb-2678-29a1-bc8d87c76269 (at 10.210.12.6@tcp1)
[14271915.722328] Lustre: Skipped 656 previous similar messages
[14272514.646488] Lustre: oak-OST014b: Connection restored to c52e7b60-2531-a009-c0b9-74b47eb51b5b (at 10.50.5.35@o2ib2)
[14272514.657113] Lustre: Skipped 1868 previous similar messages
[14273113.863289] Lustre: oak-OST014b: Connection restored to  (at 10.50.9.20@o2ib2)
[14273113.870757] Lustre: Skipped 2254 previous similar messages
[14273712.824518] Lustre: oak-OST013b: Connection restored to  (at 10.51.2.34@o2ib3)
[14273712.831984] Lustre: Skipped 2196 previous similar messages
[14274311.791050] Lustre: oak-OST0123: Connection restored to  (at 10.51.4.24@o2ib3)
[14274311.798536] Lustre: Skipped 1390 previous similar messages
[14274912.404557] Lustre: oak-OST0111: Connection restored to 9eb8189a-3985-f41c-7d7b-51af46681d5f (at 10.0.3.29@o2ib5)
[14274912.415083] Lustre: Skipped 1055 previous similar messages
[14275511.214255] Lustre: oak-OST013f: Connection restored to a7b62c41-7553-148f-73ab-3228571efb3f (at 10.51.13.11@o2ib3)
[14275511.224961] Lustre: Skipped 1034 previous similar messages
[14276109.892805] Lustre: oak-OST011b: Connection restored to d51bd149-d9d3-89a0-6eaa-4a54de16625a (at 10.50.8.60@o2ib2)
[14276109.903413] Lustre: Skipped 2045 previous similar messages
[14276710.289341] LustreError: 162683:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(20480)  req@ffff91456593e850 x1715253626390080/t0(0) o3->fc2f7135-b26d-de80-6d70-eb54f2b3d7bd@10.210.13.35@tcp1:100/0 lens 488/440 e 0 to 0 dl 1645503725 ref 1 fl Interpret:/0/0 rc 0/0
[14276710.314292] Lustre: oak-OST0147: Bulk IO read error with fc2f7135-b26d-de80-6d70-eb54f2b3d7bd (at 10.210.13.35@tcp1), client will retry: rc -110
[14276712.201743] Lustre: oak-OST0141: Connection restored to 963f25d3-89fe-9827-147e-b4bc4770d3be (at 10.0.3.127@o2ib5)
[14276712.212335] Lustre: Skipped 815 previous similar messages
[14276745.793459] LustreError: 137-5: oak-OST0128_UUID: not available for connect from 10.210.13.35@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14276810.115692] LustreError: 199244:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 150s: evicting client at 10.210.13.35@tcp1  ns: filter-oak-OST0147_UUID lock: ffff91785a3b3a80/0xed112d30451d4ea5 lrc: 4/0,0 mode: PW/PW res: [0x501cc5:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) flags: 0x60000400010020 nid: 10.210.13.35@tcp1 remote: 0x3f3c17c51b99e8a8 expref: 11 pid: 259132 timeout: 14311510 lvb_type: 0
[14276810.156115] LustreError: 199244:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 1 previous similar message
[14276834.617332] Lustre: oak-OST011f: Client 5a4881be-a0cb-e632-509b-2e15c534b21a (at 10.210.12.9@tcp1) reconnecting
[14276834.627684] Lustre: Skipped 7 previous similar messages
[14276835.620093] Lustre: oak-OST011b: Client 5a4881be-a0cb-e632-509b-2e15c534b21a (at 10.210.12.9@tcp1) reconnecting
[14276835.630412] Lustre: Skipped 4 previous similar messages
[14276836.620649] Lustre: oak-OST0119: Client 5a4881be-a0cb-e632-509b-2e15c534b21a (at 10.210.12.9@tcp1) reconnecting
[14276836.631053] Lustre: Skipped 2 previous similar messages
[14276838.970832] Lustre: oak-OST013f: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[14276838.981264] Lustre: Skipped 8 previous similar messages
[14276843.384307] Lustre: oak-OST0129: Client fc2f7135-b26d-de80-6d70-eb54f2b3d7bd (at 10.210.13.35@tcp1) reconnecting
[14276843.394729] Lustre: Skipped 19 previous similar messages
[14276855.110560] Lustre: oak-OST0131: Client fc2f7135-b26d-de80-6d70-eb54f2b3d7bd (at 10.210.13.35@tcp1) reconnecting
[14276855.120966] Lustre: Skipped 11 previous similar messages
[14277301.975108] LustreError: 243539:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[14277312.439511] Lustre: oak-OST0145: Connection restored to 963f25d3-89fe-9827-147e-b4bc4770d3be (at 10.0.3.127@o2ib5)
[14277312.450096] Lustre: Skipped 614 previous similar messages
[14277912.411991] Lustre: oak-OST014b: Connection restored to efa3260b-85f8-753d-2bfc-fbd16b6c6f94 (at 10.50.14.5@o2ib2)
[14277912.422881] Lustre: Skipped 961 previous similar messages
[14278514.701252] Lustre: oak-OST011f: Connection restored to fc6538a6-64f5-1c92-d38a-4c03c9b82dd0 (at 10.210.12.123@tcp1)
[14278514.712012] Lustre: Skipped 671 previous similar messages
[14279113.351553] Lustre: oak-OST012b: Connection restored to 89507400-42d7-a037-6f53-cceb900296af (at 10.50.9.43@o2ib2)
[14279113.362132] Lustre: Skipped 1330 previous similar messages
[14279712.225007] Lustre: oak-OST0115: Connection restored to 612334a0-616c-86f1-bbbc-1f12050c0dbf (at 10.50.7.33@o2ib2)
[14279712.235584] Lustre: Skipped 1099 previous similar messages
[14280310.779492] Lustre: oak-OST012b: Connection restored to 89507400-42d7-a037-6f53-cceb900296af (at 10.50.9.43@o2ib2)
[14280310.790486] Lustre: Skipped 1566 previous similar messages
[14280909.574224] Lustre: oak-OST012d: Connection restored to  (at 10.50.6.46@o2ib2)
[14280909.581710] Lustre: Skipped 1535 previous similar messages
[14281081.277888] Lustre: oak-OST0137: Client 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1) reconnecting
[14281083.274255] Lustre: oak-OST0111: Client 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1) reconnecting
[14281083.284680] Lustre: Skipped 16 previous similar messages
[14281089.268898] Lustre: oak-OST0119: Client 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1) reconnecting
[14281089.279415] Lustre: Skipped 2 previous similar messages
[14281512.071032] Lustre: oak-OST0145: Connection restored to  (at 10.51.0.67@o2ib3)
[14281512.078517] Lustre: Skipped 2167 previous similar messages
[14282111.238789] Lustre: oak-OST0127: Connection restored to edd06616-023a-51fb-d0e1-f960c04746e6 (at 10.0.3.32@o2ib5)
[14282111.249299] Lustre: Skipped 1290 previous similar messages
[14282709.871802] Lustre: oak-OST0133: Connection restored to 7bd5e25d-92e5-9f8c-22fa-8f4c98d09593 (at 10.50.10.6@o2ib2)
[14282709.882388] Lustre: Skipped 888 previous similar messages
[14283308.743863] Lustre: oak-OST0113: Connection restored to ffc5d26c-5660-1121-8b09-c25bc9167b01 (at 10.210.12.13@tcp1)
[14283308.754531] Lustre: Skipped 838 previous similar messages
[14283907.406385] Lustre: oak-OST012d: Connection restored to 963f25d3-89fe-9827-147e-b4bc4770d3be (at 10.0.3.127@o2ib5)
[14283907.416966] Lustre: Skipped 956 previous similar messages
[14284510.398408] Lustre: oak-OST0141: Connection restored to  (at 10.50.15.6@o2ib2)
[14284510.405888] Lustre: Skipped 864 previous similar messages
[14285110.105288] Lustre: oak-OST0147: Connection restored to 612334a0-616c-86f1-bbbc-1f12050c0dbf (at 10.50.7.33@o2ib2)
[14285110.115884] Lustre: Skipped 914 previous similar messages
[14285709.146844] Lustre: oak-OST012d: Connection restored to cfc9c6ea-1edd-991c-bbee-2e1d53db371f (at 10.210.12.117@tcp1)
[14285709.157609] Lustre: Skipped 1034 previous similar messages
[14286308.437971] Lustre: oak-OST0121: Connection restored to 80374d94-41e6-6f5b-384d-68674ee94b69 (at 10.50.4.34@o2ib2)
[14286308.448582] Lustre: Skipped 1251 previous similar messages
[14286907.510381] Lustre: oak-OST0117: Connection restored to 8a542161-c6da-012a-5f15-3267427cffa2 (at 10.51.5.58@o2ib3)
[14286907.520986] Lustre: Skipped 900 previous similar messages
[14287506.575543] Lustre: oak-OST011f: Connection restored to 0daf0491-344e-4941-a1c9-9dd34a1df9a9 (at 10.50.5.67@o2ib2)
[14287506.586209] Lustre: Skipped 1421 previous similar messages
[14288108.083682] Lustre: oak-OST0131: Connection restored to 080fbf36-50e2-e037-d579-97f7ea4dd6a6 (at 10.210.12.51@tcp1)
[14288108.094359] Lustre: Skipped 1144 previous similar messages
[14288667.790840] Lustre: oak-OST0145: Client e6faf4e3-d6e0-d000-bf7b-6ceec524d5b6 (at 10.210.12.111@tcp1) reconnecting
[14288668.082969] LustreError: 228873:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91c6eb4a4050 x1714906442011456/t0(0) o4->e6faf4e3-d6e0-d000-bf7b-6ceec524d5b6@10.210.12.111@tcp1:52/0 lens 488/448 e 0 to 0 dl 1645515757 ref 1 fl Interpret:/0/0 rc 0/0
[14288668.107661] Lustre: oak-OST0145: Bulk IO write error with e6faf4e3-d6e0-d000-bf7b-6ceec524d5b6 (at 10.210.12.111@tcp1), client will retry: rc = -110
[14288668.652562] LustreError: 160927:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91ddedf5f850 x1714906442014528/t0(0) o4->e6faf4e3-d6e0-d000-bf7b-6ceec524d5b6@10.210.12.111@tcp1:52/0 lens 488/448 e 0 to 0 dl 1645515757 ref 1 fl Interpret:/0/0 rc 0/0
[14288668.676979] LustreError: 160927:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[14288668.686805] Lustre: oak-OST0145: Bulk IO write error with e6faf4e3-d6e0-d000-bf7b-6ceec524d5b6 (at 10.210.12.111@tcp1), client will retry: rc = -110
[14288668.700329] Lustre: Skipped 1 previous similar message
[14288669.531743] Lustre: oak-OST0145: Client e6faf4e3-d6e0-d000-bf7b-6ceec524d5b6 (at 10.210.12.111@tcp1) reconnecting
[14288669.984321] LustreError: 160934:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91ae0361a050 x1714906442043712/t0(0) o4->e6faf4e3-d6e0-d000-bf7b-6ceec524d5b6@10.210.12.111@tcp1:58/0 lens 488/448 e 0 to 0 dl 1645515763 ref 1 fl Interpret:/0/0 rc 0/0
[14288670.009101] Lustre: oak-OST0145: Bulk IO write error with e6faf4e3-d6e0-d000-bf7b-6ceec524d5b6 (at 10.210.12.111@tcp1), client will retry: rc = -110
[14288710.937705] Lustre: oak-OST0137: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[14288710.948380] Lustre: Skipped 1025 previous similar messages
[14288729.272514] LustreError: 127356:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145172(4193748)  req@ffff91c9cf7ab850 x1714947407981632/t0(0) o4->5704685e-b0a9-b5c3-be3f-b278a4395bec@10.210.12.72@tcp1:52/0 lens 488/448 e 0 to 0 dl 1645515757 ref 1 fl Interpret:/0/0 rc 0/0
[14288729.298448] Lustre: oak-OST0149: Bulk IO write error with 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1), client will retry: rc = -110
[14288729.311884] Lustre: Skipped 2 previous similar messages
[14288748.361470] Lustre: oak-OST0149: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[14288835.102147] Lustre: oak-OST0145: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[14288843.593117] Lustre: oak-OST0125: Client e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1) reconnecting
[14288843.603562] Lustre: Skipped 45 previous similar messages
[14288860.648760] Lustre: oak-OST0127: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[14288860.659174] Lustre: Skipped 34 previous similar messages
[14289309.727161] Lustre: oak-OST0131: Connection restored to b5d14e2b-86e2-1895-f5d0-166f45b97496 (at 10.51.1.44@o2ib3)
[14289309.737741] Lustre: Skipped 1275 previous similar messages
[14289908.734977] Lustre: oak-OST014b: Connection restored to 582484fd-3cd6-e7cb-180b-ae6af9fe1e87 (at 10.50.10.8@o2ib2)
[14289908.745601] Lustre: Skipped 1647 previous similar messages
[14290507.803601] Lustre: oak-OST0111: Connection restored to 097ece3c-9f18-7c64-d1a4-979765e8510b (at 10.0.3.24@o2ib5)
[14290507.814093] Lustre: Skipped 1579 previous similar messages
[14291107.327905] Lustre: oak-OST0147: Connection restored to f8296b31-7721-4ee1-a24a-b8740b04be5b (at 10.50.7.63@o2ib2)
[14291107.338586] Lustre: Skipped 1370 previous similar messages
[14291707.027947] Lustre: oak-OST0149: Connection restored to 80374d94-41e6-6f5b-384d-68674ee94b69 (at 10.50.4.34@o2ib2)
[14291707.038538] Lustre: Skipped 1380 previous similar messages
[14292305.627592] Lustre: oak-OST0145: Connection restored to 42ca853b-b0d8-df5a-3f39-ea9e3deaa37a (at 10.50.7.45@o2ib2)
[14292305.638204] Lustre: Skipped 1368 previous similar messages
[14292904.268779] Lustre: oak-OST0119: Connection restored to b28a6b2f-d0b4-e8a9-0775-0dab6a037a94 (at 10.51.1.37@o2ib3)
[14292904.279379] Lustre: Skipped 1672 previous similar messages
[14293503.779008] Lustre: oak-OST0115: Connection restored to 80374d94-41e6-6f5b-384d-68674ee94b69 (at 10.50.4.34@o2ib2)
[14293503.789605] Lustre: Skipped 1487 previous similar messages
[14294104.894248] Lustre: oak-OST0119: Connection restored to 41772207-4afc-879d-b0dc-864a5fddd764 (at 10.50.4.33@o2ib2)
[14294104.904918] Lustre: Skipped 1131 previous similar messages
[14294706.650491] Lustre: oak-OST014d: Connection restored to bb719fde-6549-fed7-314d-539d729bbbca (at 10.51.0.72@o2ib3)
[14294706.661083] Lustre: Skipped 698 previous similar messages
[14295306.000457] Lustre: oak-OST013b: Connection restored to  (at 10.51.6.5@o2ib3)
[14295306.007841] Lustre: Skipped 550 previous similar messages
[14295906.231561] Lustre: oak-OST0147: Connection restored to a80f6d08-4459-2bd2-7cca-724b51375a1b (at 10.50.15.13@o2ib2)
[14295906.242222] Lustre: Skipped 565 previous similar messages
[14296505.512280] Lustre: oak-OST012d: Connection restored to 1541d13a-3e02-38af-ab9e-9e12a6e9c4d5 (at 10.51.15.14@o2ib3)
[14296505.522950] Lustre: Skipped 996 previous similar messages
[14296773.755575] LustreError: 127349:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(1048576)  req@ffff919a7b4f9850 x1714997860556736/t0(0) o3->1be164da-11a7-a3a8-bd17-8ca7a5aab4e8@10.210.12.69@tcp1:564/0 lens 488/440 e 0 to 0 dl 1645523819 ref 1 fl Interpret:/0/0 rc 0/0
[14296773.755636] Lustre: oak-OST0131: Bulk IO read error with 1be164da-11a7-a3a8-bd17-8ca7a5aab4e8 (at 10.210.12.69@tcp1), client will retry: rc -110
[14296773.793823] LustreError: 127349:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 10 previous similar messages
[14296871.886582] Lustre: oak-OST014d: Client 1be164da-11a7-a3a8-bd17-8ca7a5aab4e8 (at 10.210.12.69@tcp1) reconnecting
[14296871.897162] Lustre: Skipped 1 previous similar message
[14296879.456651] Lustre: oak-OST011d: Client 1be164da-11a7-a3a8-bd17-8ca7a5aab4e8 (at 10.210.12.69@tcp1) reconnecting
[14296879.467076] Lustre: Skipped 18 previous similar messages
[14296894.460258] Lustre: oak-OST0139: Client 1be164da-11a7-a3a8-bd17-8ca7a5aab4e8 (at 10.210.12.69@tcp1) reconnecting
[14296894.470726] Lustre: Skipped 13 previous similar messages
[14296919.896506] Lustre: oak-OST0145: Client 1be164da-11a7-a3a8-bd17-8ca7a5aab4e8 (at 10.210.12.69@tcp1) reconnecting
[14296919.906936] Lustre: Skipped 1 previous similar message
[14297045.946987] Lustre: oak-OST0127: Client 1be164da-11a7-a3a8-bd17-8ca7a5aab4e8 (at 10.210.12.69@tcp1) reconnecting
[14297045.957393] Lustre: Skipped 1 previous similar message
[14297104.669828] Lustre: oak-OST0117: Connection restored to 684a9b09-ef4a-553f-fc09-c29c5b4ead46 (at 10.210.12.49@tcp1)
[14297104.680594] Lustre: Skipped 914 previous similar messages
[14297706.154167] Lustre: oak-OST0139: Connection restored to e1051d4f-8b35-40f4-95bb-1d5125c6f189 (at 10.51.1.18@o2ib3)
[14297706.164750] Lustre: Skipped 748 previous similar messages
[14298305.264271] Lustre: oak-OST0137: Connection restored to  (at 10.50.10.72@o2ib2)
[14298305.271844] Lustre: Skipped 1084 previous similar messages
[14298904.294308] Lustre: oak-OST011d: Connection restored to  (at 10.50.15.6@o2ib2)
[14298904.301778] Lustre: Skipped 952 previous similar messages
[14299359.091060] LustreError: 160894:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff919c2639e050 x1714906480115968/t0(0) o4->e6faf4e3-d6e0-d000-bf7b-6ceec524d5b6@10.210.12.111@tcp1:150/0 lens 488/448 e 0 to 0 dl 1645526425 ref 1 fl Interpret:/0/0 rc 0/0
[14299359.091368] Lustre: oak-OST0145: Bulk IO write error with e6faf4e3-d6e0-d000-bf7b-6ceec524d5b6 (at 10.210.12.111@tcp1), client will retry: rc = -110
[14299359.130474] LustreError: 160894:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[14299389.761933] Lustre: oak-OST0145: Client e6faf4e3-d6e0-d000-bf7b-6ceec524d5b6 (at 10.210.12.111@tcp1) reconnecting
[14299481.943564] Lustre: oak-OST014d: Client e6faf4e3-d6e0-d000-bf7b-6ceec524d5b6 (at 10.210.12.111@tcp1) reconnecting
[14299504.340296] Lustre: oak-OST013d: Connection restored to eee699b2-cb8e-0a91-6752-be0f592e4945 (at 10.50.13.2@o2ib2)
[14299504.350889] Lustre: Skipped 985 previous similar messages
[14300103.854543] Lustre: oak-OST013d: Connection restored to ad12f851-216b-c412-f51f-518aba8ea609 (at 10.210.9.195@tcp1)
[14300103.865391] Lustre: Skipped 825 previous similar messages
[14300705.380178] Lustre: oak-OST012d: Connection restored to 797bb197-13a6-1620-dd0f-17321ceff735 (at 10.51.1.53@o2ib3)
[14300705.390828] Lustre: Skipped 1104 previous similar messages
[14301305.924138] Lustre: oak-OST0145: Connection restored to f0baa3b1-f3a1-90de-e3da-c913c1605f85 (at 10.50.13.6@o2ib2)
[14301305.934730] Lustre: Skipped 756 previous similar messages
[14301907.143560] Lustre: oak-OST0133: Connection restored to 4025d53e-c01c-f5a6-a7de-c3a1c13db1f4 (at 10.50.10.5@o2ib2)
[14301907.154209] Lustre: Skipped 1052 previous similar messages
[14302507.549730] Lustre: oak-OST0129: Connection restored to bb719fde-6549-fed7-314d-539d729bbbca (at 10.51.0.72@o2ib3)
[14302507.560954] Lustre: Skipped 855 previous similar messages
[14303106.443881] Lustre: oak-OST0127: Connection restored to 27d57912-6776-078b-59ad-39eae43109d2 (at 10.50.7.34@o2ib2)
[14303106.454518] Lustre: Skipped 961 previous similar messages
[14303705.710462] Lustre: oak-OST011b: Connection restored to  (at 10.0.2.3@o2ib5)
[14303705.717761] Lustre: Skipped 717 previous similar messages
[14304305.812044] Lustre: oak-OST012f: Connection restored to a80f6d08-4459-2bd2-7cca-724b51375a1b (at 10.50.15.13@o2ib2)
[14304305.822721] Lustre: Skipped 775 previous similar messages
[14304562.894492] LustreError: 137-5: oak-OST0146_UUID: not available for connect from 10.210.12.73@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14304658.041286] Lustre: oak-OST014d: Client b4fca75a-2d54-9bf2-331f-67d0c52e5ff2 (at 10.210.12.73@tcp1) reconnecting
[14304658.051699] Lustre: Skipped 13 previous similar messages
[14304663.482141] Lustre: oak-OST0145: Client b4fca75a-2d54-9bf2-331f-67d0c52e5ff2 (at 10.210.12.73@tcp1) reconnecting
[14304907.802342] Lustre: oak-OST0135: Connection restored to 080fbf36-50e2-e037-d579-97f7ea4dd6a6 (at 10.210.12.51@tcp1)
[14304907.813026] Lustre: Skipped 1021 previous similar messages
[14305512.007619] Lustre: oak-OST013f: Connection restored to a7b62c41-7553-148f-73ab-3228571efb3f (at 10.51.13.11@o2ib3)
[14305512.018288] Lustre: Skipped 896 previous similar messages
[14306110.963841] Lustre: oak-OST0127: Connection restored to 97a324d4-c834-549a-4b22-6ab961bb5783 (at 10.0.3.86@o2ib5)
[14306110.974407] Lustre: Skipped 1045 previous similar messages
[14306602.652331] Lustre: oak-OST0133: Client 66682e34-b129-b800-fbb3-f41d4d13f0db (at 10.210.9.195@tcp1) reconnecting
[14306602.662776] Lustre: Skipped 1 previous similar message
[14306603.655657] LustreError: 160937:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91bbad85f850 x1712441808885952/t0(0) o4->66682e34-b129-b800-fbb3-f41d4d13f0db@10.210.9.195@tcp1:668/0 lens 488/448 e 0 to 0 dl 1645533738 ref 1 fl Interpret:/0/0 rc 0/0
[14306603.680091] LustreError: 160937:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 2 previous similar messages
[14306603.690144] Lustre: oak-OST0133: Bulk IO write error with 66682e34-b129-b800-fbb3-f41d4d13f0db (at 10.210.9.195@tcp1), client will retry: rc = -110
[14306603.703586] Lustre: Skipped 2 previous similar messages
[14306675.764665] LustreError: 137-5: oak-OST013c_UUID: not available for connect from 10.210.12.60@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14306679.694548] LustreError: 137-5: oak-OST013c_UUID: not available for connect from 10.210.9.195@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14306709.575765] Lustre: oak-OST014b: Connection restored to b5d39b73-c9ff-daa4-7beb-4f9c4da0a03e (at 10.51.1.20@o2ib3)
[14306709.586347] Lustre: Skipped 1572 previous similar messages
[14306764.042540] Lustre: oak-OST0129: Client e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1) reconnecting
[14306764.052950] Lustre: Skipped 6 previous similar messages
[14306765.926101] Lustre: oak-OST0131: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[14306765.936537] Lustre: Skipped 14 previous similar messages
[14306769.386527] Lustre: oak-OST0135: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[14306769.396965] Lustre: Skipped 4 previous similar messages
[14306773.618735] Lustre: oak-OST0145: Client b4fca75a-2d54-9bf2-331f-67d0c52e5ff2 (at 10.210.12.73@tcp1) reconnecting
[14306773.629147] Lustre: Skipped 8 previous similar messages
[14306816.615716] Lustre: oak-OST0115: Client 66682e34-b129-b800-fbb3-f41d4d13f0db (at 10.210.9.195@tcp1) reconnecting
[14306816.626143] Lustre: Skipped 4 previous similar messages
[14306872.620007] Lustre: oak-OST0125: Client 66682e34-b129-b800-fbb3-f41d4d13f0db (at 10.210.9.195@tcp1) reconnecting
[14306872.630417] Lustre: Skipped 1 previous similar message
[14307078.848434] Lustre: oak-OST0119: Client 66682e34-b129-b800-fbb3-f41d4d13f0db (at 10.210.9.195@tcp1) reconnecting
[14307308.203386] Lustre: oak-OST0121: Connection restored to e1051d4f-8b35-40f4-95bb-1d5125c6f189 (at 10.51.1.18@o2ib3)
[14307308.214296] Lustre: Skipped 1200 previous similar messages
[14307908.888399] Lustre: oak-OST0143: Connection restored to  (at 10.50.13.8@o2ib2)
[14307908.895877] Lustre: Skipped 919 previous similar messages
[14308508.492413] Lustre: oak-OST0123: Connection restored to 46bd8889-1b94-f38a-f202-7268f2e23290 (at 10.50.13.10@o2ib2)
[14308508.503081] Lustre: Skipped 865 previous similar messages
[14308860.584654] LustreError: 137-5: oak-OST0116_UUID: not available for connect from 10.210.12.23@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14308948.150941] Lustre: oak-OST012b: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14308948.161430] Lustre: Skipped 19 previous similar messages
[14309107.898864] Lustre: oak-OST014b: Connection restored to e1051d4f-8b35-40f4-95bb-1d5125c6f189 (at 10.51.1.18@o2ib3)
[14309107.909481] Lustre: Skipped 865 previous similar messages
[14309343.358179] LustreError: 229136:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff915917d8e850 x1715537683118592/t0(0) o4->5a5caa86-68df-f806-7e6f-05440e95a4ce@10.210.12.43@tcp1:332/0 lens 488/448 e 0 to 0 dl 1645536422 ref 1 fl Interpret:/0/0 rc 0/0
[14309343.358482] Lustre: oak-OST0149: Bulk IO write error with 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1), client will retry: rc = -110
[14309343.397409] LustreError: 229136:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[14309363.238084] LustreError: 137-5: oak-OST0146_UUID: not available for connect from 10.210.12.73@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14309367.986780] Lustre: oak-OST0149: Client 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1) reconnecting
[14309367.997199] Lustre: Skipped 22 previous similar messages
[14309457.237821] Lustre: oak-OST0119: Client 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1) reconnecting
[14309457.248375] Lustre: Skipped 3 previous similar messages
[14309497.697313] LustreError: 11-0: oak-MDT0000-lwp-OST012d: operation ldlm_enqueue to node 10.0.2.52@o2ib5 failed: rc = -107
[14309497.697347] Lustre: oak-MDT0000-lwp-OST013b: Connection to oak-MDT0000 (at 10.0.2.52@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
[14309497.697349] Lustre: Skipped 30 previous similar messages
[14309497.700897] LustreError: 198216:0:(import.c:706:ptlrpc_connect_import_locked()) already connecting
[14309497.739322] LustreError: Skipped 145 previous similar messages
[14310138.659086] LNetError: 26416:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 11 seconds
[14310138.669400] LNetError: 26416:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with 10.0.2.52@o2ib5 (61): c: 0, oc: 0, rc: 8
[14310188.537863] LNetError: 26416:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 1 seconds
[14310188.548094] LNetError: 26416:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with 10.0.2.51@o2ib5 (51): c: 0, oc: 0, rc: 8
[14310188.560655] Lustre: 198233:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1645537187/real 1645537237]  req@ffff91772c2aa880 x1710536945114816/t0(0) o400->MGC10.0.2.51@o2ib5@10.0.2.51@o2ib5:26/25 lens 224/224 e 0 to 1 dl 1645537340 ref 1 fl Rpc:eXN/0/ffffffff rc 0/-1
[14310188.560813] LustreError: 166-1: MGC10.0.2.51@o2ib5: Connection to MGS (at 10.0.2.51@o2ib5) was lost; in progress operations using this service will fail
[14310188.602990] Lustre: 198233:0:(client.c:2169:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[14310435.171356] Lustre: oak-OST0111: Connection restored to oak-MDT0003-mdtlov_UUID (at 10.0.2.52@o2ib5)
[14310435.180726] Lustre: Skipped 560 previous similar messages
[14310451.126273] LustreError: 167-0: oak-MDT0001-lwp-OST012f: This client was evicted by oak-MDT0001; in progress operations using this service will fail.
[14310451.139906] LustreError: Skipped 61 previous similar messages
[14310451.146113] Lustre: Evicted from MGS (at 10.0.2.51@o2ib5) after server handle changed from 0x7c52ee8d6181faa1 to 0x5cee0d1fc48bf647
[14310451.163930] LustreError: 198217:0:(client.c:1233:ptlrpc_import_delay_req()) @@@ invalidate in flight  req@ffff914cad83bf00 x1710536910312832/t0(0) o103->oak-MDT0000-lwp-OST0135@10.0.2.52@o2ib5:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:W/0/ffffffff rc 0/-1
[14310451.187046] LustreError: 198217:0:(client.c:1233:ptlrpc_import_delay_req()) Skipped 2 previous similar messages
[14310451.249272] LustreError: 11-0: oak-MDT0000-lwp-OST013f: operation quota_acquire to node 10.0.2.52@o2ib5 failed: rc = -11
[14310451.260371] LustreError: Skipped 21 previous similar messages
[14310453.247376] LustreError: 11-0: oak-MDT0000-lwp-OST013f: operation quota_acquire to node 10.0.2.52@o2ib5 failed: rc = -11
[14310453.258505] LustreError: Skipped 1022 previous similar messages
[14310457.239657] LustreError: 11-0: oak-MDT0000-lwp-OST013f: operation quota_acquire to node 10.0.2.52@o2ib5 failed: rc = -11
[14310457.250762] LustreError: Skipped 2127 previous similar messages
[14310465.222967] LustreError: 11-0: oak-MDT0000-lwp-OST013f: operation quota_acquire to node 10.0.2.52@o2ib5 failed: rc = -11
[14310465.234083] LustreError: Skipped 3488 previous similar messages
[14310481.184752] LustreError: 11-0: oak-MDT0000-lwp-OST013f: operation quota_acquire to node 10.0.2.52@o2ib5 failed: rc = -11
[14310481.195851] LustreError: Skipped 7232 previous similar messages
[14310501.080669] LustreError: 167-0: oak-MDT0005-lwp-OST012f: This client was evicted by oak-MDT0005; in progress operations using this service will fail.
[14310501.094269] LustreError: Skipped 123 previous similar messages
[14310513.496078] LustreError: 11-0: oak-MDT0000-lwp-OST0111: operation quota_acquire to node 10.0.2.52@o2ib5 failed: rc = -11
[14310513.507198] LustreError: Skipped 5806 previous similar messages
[14310521.729441] Lustre: oak-OST0129: Connection restored to 41208e16-c47d-1ba0-c7bb-5898819a538d (at 10.210.12.125@tcp1)
[14310521.740215] Lustre: Skipped 372 previous similar messages
[14310521.770566] Lustre: oak-OST011b: deleting orphan objects from 0x49c0000401:21808359 to 0x49c0000401:21808417
[14310521.770601] Lustre: oak-OST0115: deleting orphan objects from 0x4840000402:21705880 to 0x4840000402:21705921
[14310521.770682] Lustre: oak-OST012f: deleting orphan objects from 0x4ec0000400:7135983 to 0x4ec0000400:7136001
[14310521.770915] Lustre: oak-OST0123: deleting orphan objects from 0x4bc0000402:22852891 to 0x4bc0000402:22852961
[14310521.770916] Lustre: oak-OST0127: deleting orphan objects from 0x4cc0000400:6983966 to 0x4cc0000400:6984001
[14310521.771019] Lustre: oak-OST0121: deleting orphan objects from 0x4b40000402:23231797 to 0x4b40000402:23231841
[14310521.783583] Lustre: oak-OST0129: deleting orphan objects from 0x4d40000400:7089420 to 0x4d40000400:7089473
[14310521.783584] Lustre: oak-OST0139: deleting orphan objects from 0x51c0000402:6898580 to 0x51c0000402:6898657
[14310521.784931] Lustre: oak-OST012b: deleting orphan objects from 0x4dc0000400:6925153 to 0x4dc0000400:6925217
[14310521.784934] Lustre: oak-OST0111: deleting orphan objects from 0x4740000401:21657371 to 0x4740000401:21657409
[14310521.787150] Lustre: oak-OST0113: deleting orphan objects from 0x47c0000400:21805219 to 0x47c0000400:21805281
[14310521.787157] Lustre: oak-OST0125: deleting orphan objects from 0x4c40000400:7114207 to 0x4c40000400:7114273
[14310521.788779] Lustre: oak-OST013f: deleting orphan objects from 0x5400000403:3803050 to 0x5400000403:3803233
[14310521.788783] Lustre: oak-OST0135: deleting orphan objects from 0x5040000400:6739131 to 0x5040000400:6739201
[14310521.802833] Lustre: oak-OST0133: deleting orphan objects from 0x4fc0000400:7072857 to 0x4fc0000400:7072897
[14310521.802833] Lustre: oak-OST0131: deleting orphan objects from 0x4f40000400:7044141 to 0x4f40000400:7044225
[14310521.823915] Lustre: oak-OST0147: deleting orphan objects from 0x5580000402:3712827 to 0x5580000402:3712897
[14310521.824031] Lustre: oak-OST0137: deleting orphan objects from 0x50c0000400:6536820 to 0x50c0000400:6536865
[14310521.824033] Lustre: oak-OST0143: deleting orphan objects from 0x5500000401:3905100 to 0x5500000401:3905185
[14310521.826351] Lustre: oak-OST0141: deleting orphan objects from 0x5440000403:3719667 to 0x5440000403:3719745
[14310521.826419] Lustre: oak-OST013b: deleting orphan objects from 0x5380000400:3795112 to 0x5380000400:3795169
[14310521.826421] Lustre: oak-OST014b: deleting orphan objects from 0x5680000401:3593105 to 0x5680000401:3593185
[14310521.827233] Lustre: oak-OST0119: deleting orphan objects from 0x4940000401:21771964 to 0x4940000401:21772001
[14310521.827234] Lustre: oak-OST0149: deleting orphan objects from 0x55c0000402:3842834 to 0x55c0000402:3842881
[14310521.839024] Lustre: oak-OST011d: deleting orphan objects from 0x4a40000401:22199737 to 0x4a40000401:22199777
[14310521.839099] Lustre: oak-OST013d: deleting orphan objects from 0x53c0000400:3603094 to 0x53c0000400:3603169
[14310521.839101] Lustre: oak-OST0117: deleting orphan objects from 0x48c0000401:22046497 to 0x48c0000401:22046561
[14310521.879477] Lustre: oak-OST011f: deleting orphan objects from 0x4ac0000401:22439701 to 0x4ac0000401:22439745
[14310521.879578] Lustre: oak-OST012d: deleting orphan objects from 0x4e40000400:6377548 to 0x4e40000400:6377633
[14310521.879580] Lustre: oak-OST0145: deleting orphan objects from 0x5540000401:3875314 to 0x5540000401:3875393
[14310521.894161] Lustre: oak-OST014d: deleting orphan objects from 0x56c0000401:3602033 to 0x56c0000401:3602177
[14310522.382010] Lustre: oak-OST0113: deleting orphan objects from 0x47c0000401:3873102 to 0x47c0000401:3873121
[14310522.382023] Lustre: oak-OST0115: deleting orphan objects from 0x4840000400:3792483 to 0x4840000400:3792577
[14310522.382024] Lustre: oak-OST011d: deleting orphan objects from 0x4a40000402:4202569 to 0x4a40000402:4202625
[14310522.382151] Lustre: oak-OST011f: deleting orphan objects from 0x4ac0000402:4262447 to 0x4ac0000402:4262529
[14310522.382254] Lustre: oak-OST0117: deleting orphan objects from 0x48c0000402:4061340 to 0x48c0000402:4061377
[14310522.383507] Lustre: oak-OST011b: deleting orphan objects from 0x49c0000402:3965840 to 0x49c0000402:3965921
[14310522.383512] Lustre: oak-OST0111: deleting orphan objects from 0x4740000402:3802150 to 0x4740000402:3802241
[14310522.392251] Lustre: oak-OST0123: deleting orphan objects from 0x4bc0000400:4352566 to 0x4bc0000400:4352641
[14310522.392252] Lustre: oak-OST0129: deleting orphan objects from 0x4d40000402:2467942 to 0x4d40000402:2468033
[14310522.392253] Lustre: oak-OST0127: deleting orphan objects from 0x4cc0000402:2356772 to 0x4cc0000402:2356961
[14310522.392391] Lustre: oak-OST0121: deleting orphan objects from 0x4b40000400:4440447 to 0x4b40000400:4440481
[14310522.392492] Lustre: oak-OST012b: deleting orphan objects from 0x4dc0000402:2409994 to 0x4dc0000402:2410145
[14310522.392747] Lustre: oak-OST012d: deleting orphan objects from 0x4e40000402:2257863 to 0x4e40000402:2258017
[14310522.401992] Lustre: oak-OST0135: deleting orphan objects from 0x5040000402:1974775 to 0x5040000402:1975009
[14310522.407643] Lustre: oak-OST012f: deleting orphan objects from 0x4ec0000402:2469709 to 0x4ec0000402:2469889
[14310522.434903] Lustre: oak-OST0133: deleting orphan objects from 0x4fc0000402:2448712 to 0x4fc0000402:2448833
[14310522.435008] Lustre: oak-OST0131: deleting orphan objects from 0x4f40000402:2437914 to 0x4f40000402:2437985
[14310522.435014] Lustre: oak-OST0137: deleting orphan objects from 0x50c0000402:2305846 to 0x50c0000402:2305953
[14310522.439025] Lustre: oak-OST013f: deleting orphan objects from 0x5400000404:1432661 to 0x5400000404:1432993
[14310522.441672] Lustre: oak-OST013d: deleting orphan objects from 0x53c0000402:1385070 to 0x53c0000402:1385153
[14310522.506205] Lustre: oak-OST0141: deleting orphan objects from 0x5440000404:1409158 to 0x5440000404:1409537
[14310522.538472] Lustre: oak-OST0139: deleting orphan objects from 0x51c0000400:2115498 to 0x51c0000400:2115841
[14310522.538628] Lustre: oak-OST0143: deleting orphan objects from 0x5500000400:1468636 to 0x5500000400:1468833
[14310522.538629] Lustre: oak-OST0145: deleting orphan objects from 0x5540000400:1447852 to 0x5540000400:1448097
[14310522.539632] Lustre: oak-OST0147: deleting orphan objects from 0x5580000401:1395303 to 0x5580000401:1395649
[14310522.539636] Lustre: oak-OST014d: deleting orphan objects from 0x56c0000402:1326859 to 0x56c0000402:1327201
[14310522.578134] Lustre: oak-OST014b: deleting orphan objects from 0x5680000402:1326827 to 0x5680000402:1326945
[14310522.578138] Lustre: oak-OST0149: deleting orphan objects from 0x55c0000401:1444531 to 0x55c0000401:1444897
[14310522.604909] Lustre: oak-OST0125: deleting orphan objects from 0x4c40000402:2458006 to 0x4c40000402:2458177
[14310522.608344] Lustre: oak-OST0119: deleting orphan objects from 0x4940000402:3943091 to 0x4940000402:3943169
[14310522.608353] Lustre: oak-OST013b: deleting orphan objects from 0x5380000402:1430571 to 0x5380000402:1430881
[14310525.176140] Lustre: oak-OST0111: deleting orphan objects from 0x4740000400:5524946 to 0x4740000400:5524961
[14310525.176389] Lustre: oak-OST0113: deleting orphan objects from 0x47c0000402:5571920 to 0x47c0000402:5571937
[14310525.176400] Lustre: oak-OST0115: deleting orphan objects from 0x4840000401:5547092 to 0x4840000401:5547137
[14310525.176400] Lustre: oak-OST011b: deleting orphan objects from 0x49c0000400:5405977 to 0x49c0000400:5406017
[14310525.176505] Lustre: oak-OST0119: deleting orphan objects from 0x4940000400:5556595 to 0x4940000400:5556641
[14310525.177030] Lustre: oak-OST011f: deleting orphan objects from 0x4ac0000400:5835250 to 0x4ac0000400:5835297
[14310525.177037] Lustre: oak-OST0125: deleting orphan objects from 0x4c40000401:3578406 to 0x4c40000401:3578433
[14310525.177414] Lustre: oak-OST0121: deleting orphan objects from 0x4b40000401:6077300 to 0x4b40000401:6077345
[14310525.177416] Lustre: oak-OST0123: deleting orphan objects from 0x4bc0000401:5935339 to 0x4bc0000401:5935361
[14310525.177417] Lustre: oak-OST0129: deleting orphan objects from 0x4d40000401:3551294 to 0x4d40000401:3551329
[14310525.177520] Lustre: oak-OST0127: deleting orphan objects from 0x4cc0000401:3514557 to 0x4cc0000401:3514593
[14310525.198228] Lustre: oak-OST012d: deleting orphan objects from 0x4e40000401:3271723 to 0x4e40000401:3271745
[14310525.198229] Lustre: oak-OST0131: deleting orphan objects from 0x4f40000401:3573231 to 0x4f40000401:3573249
[14310525.198230] Lustre: oak-OST012f: deleting orphan objects from 0x4ec0000401:3583844 to 0x4ec0000401:3583873
[14310525.198345] Lustre: oak-OST0135: deleting orphan objects from 0x5040000401:3420664 to 0x5040000401:3420705
[14310525.198487] Lustre: oak-OST0137: deleting orphan objects from 0x50c0000401:3328059 to 0x50c0000401:3328097
[14310525.198633] Lustre: oak-OST0139: deleting orphan objects from 0x51c0000401:1820529 to 0x51c0000401:1820545
[14310525.198747] Lustre: oak-OST013b: deleting orphan objects from 0x5380000401:1684931 to 0x5380000401:1685025
[14310525.198851] Lustre: oak-OST013f: deleting orphan objects from 0x5400000400:1694695 to 0x5400000400:1694785
[14310525.198953] Lustre: oak-OST013d: deleting orphan objects from 0x53c0000401:1620993 to 0x53c0000401:1621057
[14310525.199057] Lustre: oak-OST0143: deleting orphan objects from 0x5500000402:1739124 to 0x5500000402:1739201
[14310525.199160] Lustre: oak-OST0145: deleting orphan objects from 0x5540000402:1725729 to 0x5540000402:1725793
[14310525.199263] Lustre: oak-OST0149: deleting orphan objects from 0x55c0000400:1716096 to 0x55c0000400:1716129
[14310525.199373] Lustre: oak-OST014b: deleting orphan objects from 0x5680000bd0:1415991 to 0x5680000bd0:1416033
[14310525.199472] Lustre: oak-OST014d: deleting orphan objects from 0x56c0000400:1425399 to 0x56c0000400:1425473
[14310525.199576] Lustre: oak-OST012b: deleting orphan objects from 0x4dc0000401:3498604 to 0x4dc0000401:3498689
[14310525.199685] Lustre: oak-OST0117: deleting orphan objects from 0x48c0000400:5680316 to 0x48c0000400:5680353
[14310525.199786] Lustre: oak-OST0133: deleting orphan objects from 0x4fc0000401:3556105 to 0x4fc0000401:3556129
[14310525.199889] Lustre: oak-OST0141: deleting orphan objects from 0x5440000400:1662248 to 0x5440000400:1662337
[14310525.199992] Lustre: oak-OST011d: deleting orphan objects from 0x4a40000400:5773124 to 0x4a40000400:5773153
[14310525.200095] Lustre: oak-OST0147: deleting orphan objects from 0x5580000400:1664395 to 0x5580000400:1664449
[14310526.774452] Lustre: oak-OST0115: deleting orphan objects from 0x0:21563908 to 0x0:21563937
[14310526.774454] Lustre: oak-OST0117: deleting orphan objects from 0x0:22051336 to 0x0:22051361
[14310526.774732] Lustre: oak-OST011d: deleting orphan objects from 0x0:20566637 to 0x0:20566657
[14310526.774734] Lustre: oak-OST0111: deleting orphan objects from 0x0:21413483 to 0x0:21413505
[14310526.774735] Lustre: oak-OST0125: deleting orphan objects from 0x0:9305720 to 0x0:9305825
[14310526.774838] Lustre: oak-OST0119: deleting orphan objects from 0x0:21669028 to 0x0:21669089
[14310526.774944] Lustre: oak-OST012d: deleting orphan objects from 0x0:8407420 to 0x0:8407457
[14310526.775474] Lustre: oak-OST012f: deleting orphan objects from 0x0:9329178 to 0x0:9329249
[14310526.775502] Lustre: oak-OST0129: deleting orphan objects from 0x0:9261379 to 0x0:9261473
[14310526.775601] Lustre: oak-OST012b: deleting orphan objects from 0x0:9087909 to 0x0:9087969
[14310526.775959] Lustre: oak-OST0133: deleting orphan objects from 0x0:9300299 to 0x0:9300385
[14310526.777980] Lustre: oak-OST0135: deleting orphan objects from 0x0:7722697 to 0x0:7722753
[14310526.777983] Lustre: oak-OST013d: deleting orphan objects from 0x0:5140939 to 0x0:5141089
[14310526.778052] Lustre: oak-OST013f: deleting orphan objects from 0x0:5378194 to 0x0:5378369
[14310526.778159] Lustre: oak-OST0149: deleting orphan objects from 0x0:5458478 to 0x0:5458593
[14310526.778261] Lustre: oak-OST0141: deleting orphan objects from 0x0:5278540 to 0x0:5278625
[14310526.780058] Lustre: oak-OST0131: deleting orphan objects from 0x0:9260250 to 0x0:9260321
[14310526.780061] Lustre: oak-OST0139: deleting orphan objects from 0x0:8111202 to 0x0:8111265
[14310526.780101] Lustre: oak-OST0143: deleting orphan objects from 0x0:5542646 to 0x0:5542817
[14310526.780184] Lustre: oak-OST0113: deleting orphan objects from 0x0:21590451 to 0x0:21590497
[14310526.780546] Lustre: oak-OST011f: deleting orphan objects from 0x0:20909504 to 0x0:20909537
[14310526.780727] Lustre: oak-OST0145: deleting orphan objects from 0x0:5503785 to 0x0:5503905
[14310526.780729] Lustre: oak-OST014b: deleting orphan objects from 0x0:4952479 to 0x0:4952609
[14310526.781130] Lustre: oak-OST0121: deleting orphan objects from 0x0:21767460 to 0x0:21767489
[14310526.781208] Lustre: oak-OST0123: deleting orphan objects from 0x0:21394015 to 0x0:21394081
[14310526.781211] Lustre: oak-OST0147: deleting orphan objects from 0x0:5258036 to 0x0:5258209
[14310526.781646] Lustre: oak-OST0127: deleting orphan objects from 0x0:9129379 to 0x0:9129473
[14310526.783457] Lustre: oak-OST011b: deleting orphan objects from 0x0:21818082 to 0x0:21818113
[14310526.783746] Lustre: oak-OST0137: deleting orphan objects from 0x0:8632172 to 0x0:8632225
[14310526.783747] Lustre: oak-OST013b: deleting orphan objects from 0x0:5346556 to 0x0:5346657
[14310526.783750] Lustre: oak-OST014d: deleting orphan objects from 0x0:4954461 to 0x0:4954785
[14310533.415473] Lustre: oak-OST0127: deleting orphan objects from 0x4cc0000403:48566 to 0x4cc0000403:48609
[14310533.415474] Lustre: oak-OST012f: deleting orphan objects from 0x4ec0000403:52438 to 0x4ec0000403:52481
[14310533.415475] Lustre: oak-OST011b: deleting orphan objects from 0x49c0000bd0:25566 to 0x49c0000bd0:25601
[14310533.415574] Lustre: oak-OST0119: deleting orphan objects from 0x4940000bd0:26188 to 0x4940000bd0:26209
[14310533.415779] Lustre: oak-OST014d: deleting orphan objects from 0x56c0000404:85761 to 0x56c0000404:85825
[14310533.415780] Lustre: oak-OST0149: deleting orphan objects from 0x55c0000403:79374 to 0x55c0000403:79521
[14310533.415887] Lustre: oak-OST0117: deleting orphan objects from 0x48c0000bd0:25709 to 0x48c0000bd0:25729
[14310533.416008] Lustre: oak-OST0133: deleting orphan objects from 0x4fc0000403:53009 to 0x4fc0000403:53057
[14310533.416108] Lustre: oak-OST0111: deleting orphan objects from 0x4740000bd0:24925 to 0x4740000bd0:24961
[14310533.416215] Lustre: oak-OST013b: deleting orphan objects from 0x5380000404:77504 to 0x5380000404:77537
[14310533.420237] Lustre: oak-OST0113: deleting orphan objects from 0x47c0000bd0:24011 to 0x47c0000bd0:24033
[14310533.420238] Lustre: oak-OST0129: deleting orphan objects from 0x4d40000403:49049 to 0x4d40000403:49121
[14310533.451709] Lustre: oak-OST012d: deleting orphan objects from 0x4e40000403:44164 to 0x4e40000403:44257
[14310533.451712] Lustre: oak-OST0131: deleting orphan objects from 0x4f40000403:50888 to 0x4f40000403:50913
[14310533.467400] Lustre: oak-OST0137: deleting orphan objects from 0x50c0000403:46026 to 0x50c0000403:46049
[14310533.467402] Lustre: oak-OST0121: deleting orphan objects from 0x4b40000bd0:31065 to 0x4b40000bd0:31105
[14310533.509973] Lustre: oak-OST013d: deleting orphan objects from 0x53c0000404:70548 to 0x53c0000404:70689
[14310533.509974] Lustre: oak-OST0143: deleting orphan objects from 0x5500000403:81164 to 0x5500000403:81345
[14310533.600690] Lustre: oak-OST0135: deleting orphan objects from 0x5040000403:50816 to 0x5040000403:50849
[14310533.600696] Lustre: oak-OST0123: deleting orphan objects from 0x4bc0000bd0:30043 to 0x4bc0000bd0:30081
[14310533.623355] Lustre: oak-OST0141: deleting orphan objects from 0x5440000401:73808 to 0x5440000401:73889
[14310533.623482] Lustre: oak-OST012b: deleting orphan objects from 0x4dc0000403:48476 to 0x4dc0000403:48545
[14310533.623485] Lustre: oak-OST0125: deleting orphan objects from 0x4c40000403:50225 to 0x4c40000403:50241
[14310533.692929] Lustre: oak-OST011f: deleting orphan objects from 0x4ac0000bd0:29658 to 0x4ac0000bd0:29697
[14310533.693108] Lustre: oak-OST013f: deleting orphan objects from 0x5400000401:73046 to 0x5400000401:73153
[14310533.693111] Lustre: oak-OST0145: deleting orphan objects from 0x5540000403:81253 to 0x5540000403:81345
[14310533.702995] Lustre: oak-OST011d: deleting orphan objects from 0x4a40000bd0:30348 to 0x4a40000bd0:30369
[14310533.703087] Lustre: oak-OST0147: deleting orphan objects from 0x5580000403:76661 to 0x5580000403:76737
[14310533.703092] Lustre: oak-OST014b: deleting orphan objects from 0x5680000403:84380 to 0x5680000403:84449
[14310533.730843] Lustre: oak-OST0115: deleting orphan objects from 0x4840000bd0:22870 to 0x4840000bd0:22913
[14310541.967233] Lustre: oak-OST0117: deleting orphan objects from 0x48c0000bd1:232949 to 0x48c0000bd1:232993
[14310541.967348] Lustre: oak-OST0123: deleting orphan objects from 0x4bc0000bd1:273668 to 0x4bc0000bd1:273697
[14310541.967548] Lustre: oak-OST0125: deleting orphan objects from 0x4c40000404:522964 to 0x4c40000404:523009
[14310541.967549] Lustre: oak-OST0139: deleting orphan objects from 0x51c0000404:751723 to 0x51c0000404:751745
[14310541.967661] Lustre: oak-OST0113: deleting orphan objects from 0x47c0000bd1:203142 to 0x47c0000bd1:203169
[14310541.967661] Lustre: oak-OST012b: deleting orphan objects from 0x4dc0000404:501354 to 0x4dc0000404:501377
[14310541.967768] Lustre: oak-OST0129: deleting orphan objects from 0x4d40000404:518262 to 0x4d40000404:518305
[14310541.967972] Lustre: oak-OST0119: deleting orphan objects from 0x4940000bd1:226806 to 0x4940000bd1:226849
[14310541.967973] Lustre: oak-OST0137: deleting orphan objects from 0x50c0000404:470029 to 0x50c0000404:470049
[14310541.968078] Lustre: oak-OST013f: deleting orphan objects from 0x5400000402:910230 to 0x5400000402:910273
[14310541.968192] Lustre: oak-OST012d: deleting orphan objects from 0x4e40000404:451030 to 0x4e40000404:451073
[14310541.968294] Lustre: oak-OST014b: deleting orphan objects from 0x5680000400:956147 to 0x5680000400:956193
[14310541.969384] Lustre: oak-OST013d: deleting orphan objects from 0x53c0000403:841186 to 0x53c0000403:841217
[14310541.969388] Lustre: oak-OST0149: deleting orphan objects from 0x55c0000404:915692 to 0x55c0000404:915713
[14310541.978746] Lustre: oak-OST0135: deleting orphan objects from 0x5040000404:568262 to 0x5040000404:568289
[14310541.978752] Lustre: oak-OST013b: deleting orphan objects from 0x5380000403:912138 to 0x5380000403:912161
[14310541.981501] Lustre: oak-OST011f: deleting orphan objects from 0x4ac0000bd1:257588 to 0x4ac0000bd1:257633
[14310541.981502] Lustre: oak-OST0141: deleting orphan objects from 0x5440000402:882155 to 0x5440000402:882177
[14310541.981515] Lustre: oak-OST011d: deleting orphan objects from 0x4a40000bd1:273864 to 0x4a40000bd1:273889
[14310541.981622] Lustre: oak-OST0111: deleting orphan objects from 0x4740000bd1:218056 to 0x4740000bd1:218081
[14310541.981730] Lustre: oak-OST014d: deleting orphan objects from 0x56c0000403:958536 to 0x56c0000403:958561
[14310541.987634] Lustre: oak-OST0131: deleting orphan objects from 0x4f40000404:522044 to 0x4f40000404:522081
[14310541.987637] Lustre: oak-OST0121: deleting orphan objects from 0x4b40000bd1:303116 to 0x4b40000bd1:303137
[14310542.009050] Lustre: oak-OST0145: deleting orphan objects from 0x5540000404:928335 to 0x5540000404:928353
[14310542.009051] Lustre: oak-OST0115: deleting orphan objects from 0x4840000bd1:206832 to 0x4840000bd1:206849
[14310542.009065] Lustre: oak-OST0133: deleting orphan objects from 0x4fc0000404:520953 to 0x4fc0000404:520993
[14310542.009167] Lustre: oak-OST011b: deleting orphan objects from 0x49c0000bd1:221166 to 0x49c0000bd1:221185
[14310542.009270] Lustre: oak-OST0143: deleting orphan objects from 0x5500000404:936000 to 0x5500000404:936033
[14310542.009370] Lustre: oak-OST012f: deleting orphan objects from 0x4ec0000404:530797 to 0x4ec0000404:530817
[14310542.018353] Lustre: oak-OST0147: deleting orphan objects from 0x5580000404:879351 to 0x5580000404:879393
[14310542.018358] Lustre: oak-OST0127: deleting orphan objects from 0x4cc0000404:507282 to 0x4cc0000404:507297
[14310676.073703] Lustre: oak-OST0113: Connection restored to 797bb197-13a6-1620-dd0f-17321ceff735 (at 10.51.1.53@o2ib3)
[14310676.084347] Lustre: Skipped 465 previous similar messages
[14310740.887572] blk_update_request: I/O error, dev dm-205, sector 8
[14310744.264037] md: md63 stopped.
[14310744.295212] md/raid:md63: device dm-589 operational as raid disk 0
[14310744.301660] md/raid:md63: device dm-554 operational as raid disk 9
[14310744.308094] md/raid:md63: device dm-537 operational as raid disk 8
[14310744.314540] md/raid:md63: device dm-551 operational as raid disk 7
[14310744.320977] md/raid:md63: device dm-536 operational as raid disk 6
[14310744.327408] md/raid:md63: device dm-535 operational as raid disk 5
[14310744.333836] md/raid:md63: device dm-591 operational as raid disk 4
[14310744.340277] md/raid:md63: device dm-617 operational as raid disk 3
[14310744.346710] md/raid:md63: device dm-610 operational as raid disk 2
[14310744.353137] md/raid:md63: device dm-597 operational as raid disk 1
[14310744.360514] md/raid:md63: raid level 6 active with 10 out of 10 devices, algorithm 2
[14310744.381810] md63: detected capacity change from 0 to 112003075014656
[14310745.307842] LDISKFS-fs (md63): file extents enabled, maximum tree depth=5
[14310745.688625] LDISKFS-fs (md63): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[14310746.121862] LDISKFS-fs (md63): file extents enabled, maximum tree depth=5
[14310746.486276] LDISKFS-fs (md63): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[14310805.173041] ses 16:0:7:0: attempting task abort! scmd(ffff91ae347b6e40)
[14310805.179911] ses 16:0:7:0: [sg833] tag#9 CDB: Inquiry 12 00 00 00 24 00
[14310805.186704] scsi target16:0:7: _scsih_tm_display_info: handle(0x0027), sas_address(0x5000ccab0506b23c), phy(48)
[14310805.197015] scsi target16:0:7: enclosurelogical id(0x5000ccab0506b200), slot(102) 
[14310805.204822] scsi target16:0:7: enclosure level(0x0001), connector name(     )
[14310805.217873] ses 16:0:7:0: task abort: SUCCESS scmd(ffff91ae347b6e40)
[14310805.224519] ses 16:0:7:0: attempting task abort! scmd(ffff917050da2d80)
[14310805.231393] ses 16:0:7:0: [sg833] tag#28 CDB: Inquiry 12 00 00 00 24 00
[14310805.238253] scsi target16:0:7: _scsih_tm_display_info: handle(0x0027), sas_address(0x5000ccab0506b23c), phy(48)
[14310805.248569] scsi target16:0:7: enclosurelogical id(0x5000ccab0506b200), slot(102) 
[14310805.256376] scsi target16:0:7: enclosure level(0x0001), connector name(     )
[14310805.269624] ses 16:0:7:0: task abort: SUCCESS scmd(ffff917050da2d80)
[14310806.114735] ses 16:0:7:0: attempting task abort! scmd(ffff91a6c408b640)
[14310806.121638] ses 16:0:7:0: [sg833] tag#22 CDB: Inquiry 12 00 00 00 24 00
[14310806.128520] scsi target16:0:7: _scsih_tm_display_info: handle(0x0027), sas_address(0x5000ccab0506b23c), phy(48)
[14310806.138836] scsi target16:0:7: enclosurelogical id(0x5000ccab0506b200), slot(102) 
[14310806.146642] scsi target16:0:7: enclosure level(0x0001), connector name(     )
[14310806.160631] ses 16:0:7:0: task abort: SUCCESS scmd(ffff91a6c408b640)
[14310816.987909] Lustre: oak-OST0149: haven't heard from client 29297750-6369-9ebf-87b8-5e6b9cd784ca (at 10.210.12.105@tcp1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91a3c7781c00, cur 1645537867 expire 1645537717 last 1645537640
[14310817.009351] LustreError: 137-5: oak-OST014f_UUID: not available for connect from 10.51.0.61@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14310817.026069] Lustre: oak-OST014f: Not available for connect from 10.51.3.51@o2ib3 (not set up)
[14310817.026071] Lustre: Skipped 1 previous similar message
[14310817.028475] Lustre: oak-OST014f: new disk, initializing
[14310817.031926] Lustre: srv-oak-OST014f: No data found on store. Initialize space
[14310817.056250] Lustre: Skipped 1 previous similar message
[14310817.098497] Lustre: oak-OST014f: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900
[14310817.980091] Lustre: oak-OST013f: haven't heard from client 29297750-6369-9ebf-87b8-5e6b9cd784ca (at 10.210.12.105@tcp1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91d53212b800, cur 1645537868 expire 1645537718 last 1645537641
[14310818.002177] Lustre: Skipped 12 previous similar messages
[14310823.674378] Lustre: cli-oak-OST014f-super: Allocated super-sequence [0x0000005780000400-0x00000057c0000400]:14f:ost]
[14310974.617895] blk_update_request: I/O error, dev dm-205, sector 8
[14310976.658840] md: md65 stopped.
[14310976.670472] md/raid:md65: device dm-608 operational as raid disk 0
[14310976.676906] md/raid:md65: device dm-545 operational as raid disk 9
[14310976.683338] md/raid:md65: device dm-571 operational as raid disk 8
[14310976.689793] md/raid:md65: device dm-547 operational as raid disk 7
[14310976.696230] md/raid:md65: device dm-566 operational as raid disk 6
[14310976.702802] md/raid:md65: device dm-559 operational as raid disk 5
[14310976.709298] md/raid:md65: device dm-619 operational as raid disk 4
[14310976.715803] md/raid:md65: device dm-624 operational as raid disk 3
[14310976.722304] md/raid:md65: device dm-613 operational as raid disk 2
[14310976.728855] md/raid:md65: device dm-620 operational as raid disk 1
[14310976.736269] md/raid:md65: raid level 6 active with 10 out of 10 devices, algorithm 2
[14310976.757493] md65: detected capacity change from 0 to 112003075014656
[14310977.492474] LDISKFS-fs (md65): file extents enabled, maximum tree depth=5
[14310977.672351] Lustre: oak-OST012f: Connection restored to 916093a3-51ee-71cf-ab61-9d8121d32b7c (at 10.51.15.11@o2ib3)
[14310977.683055] Lustre: Skipped 2001 previous similar messages
[14310977.896230] LDISKFS-fs (md65): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[14310978.410242] LDISKFS-fs (md65): file extents enabled, maximum tree depth=5
[14310978.778298] LDISKFS-fs (md65): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[14310981.289176] Lustre: oak-OST0151: new disk, initializing
[14310981.322231] Lustre: srv-oak-OST0151: No data found on store. Initialize space
[14310981.487304] Lustre: oak-OST0151: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900
[14310986.487185] Lustre: cli-oak-OST0151-super: Allocated super-sequence [0x0000005800000400-0x0000005840000400]:151:ost]
[14311061.898577] blk_update_request: I/O error, dev dm-205, sector 8
[14311064.836349] md: md67 stopped.
[14311064.847031] md/raid:md67: device dm-628 operational as raid disk 0
[14311064.853465] md/raid:md67: device dm-577 operational as raid disk 9
[14311064.859895] md/raid:md67: device dm-568 operational as raid disk 8
[14311064.866344] md/raid:md67: device dm-561 operational as raid disk 7
[14311064.872797] md/raid:md67: device dm-564 operational as raid disk 6
[14311064.879269] md/raid:md67: device dm-569 operational as raid disk 5
[14311064.885701] md/raid:md67: device dm-615 operational as raid disk 4
[14311064.892126] md/raid:md67: device dm-635 operational as raid disk 3
[14311064.898557] md/raid:md67: device dm-631 operational as raid disk 2
[14311064.905012] md/raid:md67: device dm-630 operational as raid disk 1
[14311064.912302] md/raid:md67: raid level 6 active with 10 out of 10 devices, algorithm 2
[14311064.933451] md67: detected capacity change from 0 to 112003075014656
[14311065.591352] LDISKFS-fs (md67): file extents enabled, maximum tree depth=5
[14311065.979615] LDISKFS-fs (md67): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[14311066.381402] LDISKFS-fs (md67): file extents enabled, maximum tree depth=5
[14311066.747435] LDISKFS-fs (md67): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[14311068.985526] Lustre: oak-OST0153: new disk, initializing
[14311069.003646] Lustre: srv-oak-OST0153: No data found on store. Initialize space
[14311069.125108] Lustre: oak-OST0153: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900
[14311074.007907] Lustre: cli-oak-OST0153-super: Allocated super-sequence [0x0000005880000400-0x00000058c0000400]:153:ost]
[14311164.906859] blk_update_request: I/O error, dev dm-205, sector 8
[14311166.699570] md: md69 stopped.
[14311166.711448] md/raid:md69: device dm-655 operational as raid disk 0
[14311166.717885] md/raid:md69: device dm-593 operational as raid disk 9
[14311166.724311] md/raid:md69: device dm-572 operational as raid disk 8
[14311166.730735] md/raid:md69: device dm-574 operational as raid disk 7
[14311166.737159] md/raid:md69: device dm-575 operational as raid disk 6
[14311166.743585] md/raid:md69: device dm-588 operational as raid disk 5
[14311166.750015] md/raid:md69: device dm-627 operational as raid disk 4
[14311166.756444] md/raid:md69: device dm-662 operational as raid disk 3
[14311166.762874] md/raid:md69: device dm-639 operational as raid disk 2
[14311166.769303] md/raid:md69: device dm-623 operational as raid disk 1
[14311166.776790] md/raid:md69: raid level 6 active with 10 out of 10 devices, algorithm 2
[14311166.797881] md69: detected capacity change from 0 to 112003075014656
[14311167.505610] LDISKFS-fs (md69): file extents enabled, maximum tree depth=5
[14311167.904601] LDISKFS-fs (md69): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[14311168.389471] LDISKFS-fs (md69): file extents enabled, maximum tree depth=5
[14311168.761935] LDISKFS-fs (md69): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[14311171.199832] Lustre: oak-OST0155: new disk, initializing
[14311171.219360] Lustre: srv-oak-OST0155: No data found on store. Initialize space
[14311171.337567] Lustre: oak-OST0155: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900
[14311176.588316] Lustre: cli-oak-OST0155-super: Allocated super-sequence [0x0000005900000400-0x0000005940000400]:155:ost]
[14311217.727727] blk_update_request: I/O error, dev dm-205, sector 8
[14311220.982892] md: md71 stopped.
[14311220.994566] md/raid:md71: device dm-760 operational as raid disk 0
[14311221.000994] md/raid:md71: device dm-717 operational as raid disk 9
[14311221.007417] md/raid:md71: device dm-719 operational as raid disk 8
[14311221.013844] md/raid:md71: device dm-712 operational as raid disk 7
[14311221.020282] md/raid:md71: device dm-705 operational as raid disk 6
[14311221.026708] md/raid:md71: device dm-707 operational as raid disk 5
[14311221.033140] md/raid:md71: device dm-762 operational as raid disk 4
[14311221.039583] md/raid:md71: device dm-755 operational as raid disk 3
[14311221.046016] md/raid:md71: device dm-748 operational as raid disk 2
[14311221.052454] md/raid:md71: device dm-764 operational as raid disk 1
[14311221.061212] md/raid:md71: raid level 6 active with 10 out of 10 devices, algorithm 2
[14311221.082228] md71: detected capacity change from 0 to 112003075014656
[14311221.828585] LDISKFS-fs (md71): file extents enabled, maximum tree depth=5
[14311222.262630] LDISKFS-fs (md71): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[14311222.737412] LDISKFS-fs (md71): file extents enabled, maximum tree depth=5
[14311223.108077] LDISKFS-fs (md71): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[14311225.329708] Lustre: oak-OST0157: new disk, initializing
[14311225.365364] Lustre: srv-oak-OST0157: No data found on store. Initialize space
[14311225.488353] Lustre: oak-OST0157: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900
[14311231.339651] Lustre: cli-oak-OST0157-super: Allocated super-sequence [0x0000005940000400-0x0000005980000400]:157:ost]
[14311272.505431] blk_update_request: I/O error, dev dm-205, sector 8
[14311274.539708] md: md73 stopped.
[14311274.549963] md/raid:md73: device dm-706 operational as raid disk 0
[14311274.556423] md/raid:md73: device dm-669 operational as raid disk 9
[14311274.562868] md/raid:md73: device dm-663 operational as raid disk 8
[14311274.569303] md/raid:md73: device dm-664 operational as raid disk 7
[14311274.575742] md/raid:md73: device dm-652 operational as raid disk 6
[14311274.582196] md/raid:md73: device dm-661 operational as raid disk 5
[14311274.588647] md/raid:md73: device dm-729 operational as raid disk 4
[14311274.595086] md/raid:md73: device dm-731 operational as raid disk 3
[14311274.601534] md/raid:md73: device dm-711 operational as raid disk 2
[14311274.607971] md/raid:md73: device dm-714 operational as raid disk 1
[14311274.615508] md/raid:md73: raid level 6 active with 10 out of 10 devices, algorithm 2
[14311274.637162] md73: detected capacity change from 0 to 112003075014656
[14311275.297537] LDISKFS-fs (md73): file extents enabled, maximum tree depth=5
[14311275.781558] LDISKFS-fs (md73): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[14311276.212389] LDISKFS-fs (md73): file extents enabled, maximum tree depth=5
[14311276.571992] LDISKFS-fs (md73): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[14311278.959182] Lustre: oak-OST0159: new disk, initializing
[14311278.987421] Lustre: srv-oak-OST0159: No data found on store. Initialize space
[14311279.140991] Lustre: oak-OST0159: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900
[14311284.416673] Lustre: cli-oak-OST0159-super: Allocated super-sequence [0x00000059c0000400-0x0000005a00000400]:159:ost]
[14311366.185482] blk_update_request: I/O error, dev dm-205, sector 8
[14311368.098348] md: md75 stopped.
[14311368.108536] md/raid:md75: device dm-744 operational as raid disk 0
[14311368.114964] md/raid:md75: device dm-681 operational as raid disk 9
[14311368.121388] md/raid:md75: device dm-683 operational as raid disk 8
[14311368.127810] md/raid:md75: device dm-689 operational as raid disk 7
[14311368.134233] md/raid:md75: device dm-686 operational as raid disk 6
[14311368.140659] md/raid:md75: device dm-678 operational as raid disk 5
[14311368.147082] md/raid:md75: device dm-732 operational as raid disk 4
[14311368.153504] md/raid:md75: device dm-743 operational as raid disk 3
[14311368.159928] md/raid:md75: device dm-724 operational as raid disk 2
[14311368.166353] md/raid:md75: device dm-742 operational as raid disk 1
[14311368.173577] md/raid:md75: raid level 6 active with 10 out of 10 devices, algorithm 2
[14311368.194896] md75: detected capacity change from 0 to 112003075014656
[14311368.938962] LDISKFS-fs (md75): file extents enabled, maximum tree depth=5
[14311369.328644] LDISKFS-fs (md75): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[14311369.746040] LDISKFS-fs (md75): file extents enabled, maximum tree depth=5
[14311370.102631] LDISKFS-fs (md75): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[14311372.382435] Lustre: oak-OST015b: new disk, initializing
[14311372.416455] Lustre: srv-oak-OST015b: No data found on store. Initialize space
[14311372.592715] Lustre: oak-OST015b: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900
[14311379.829910] Lustre: cli-oak-OST015b-super: Allocated super-sequence [0x0000005a40000400-0x0000005a80000400]:15b:ost]
[14311442.807188] blk_update_request: I/O error, dev dm-205, sector 8
[14311445.086131] md: md77 stopped.
[14311445.096244] md/raid:md77: device dm-728 operational as raid disk 0
[14311445.102673] md/raid:md77: device dm-682 operational as raid disk 9
[14311445.109098] md/raid:md77: device dm-693 operational as raid disk 8
[14311445.115519] md/raid:md77: device dm-700 operational as raid disk 7
[14311445.121942] md/raid:md77: device dm-687 operational as raid disk 6
[14311445.128364] md/raid:md77: device dm-697 operational as raid disk 5
[14311445.134803] md/raid:md77: device dm-745 operational as raid disk 4
[14311445.141230] md/raid:md77: device dm-757 operational as raid disk 3
[14311445.147653] md/raid:md77: device dm-741 operational as raid disk 2
[14311445.154079] md/raid:md77: device dm-753 operational as raid disk 1
[14311445.161313] md/raid:md77: raid level 6 active with 10 out of 10 devices, algorithm 2
[14311445.182382] md77: detected capacity change from 0 to 112003075014656
[14311445.964761] LDISKFS-fs (md77): file extents enabled, maximum tree depth=5
[14311446.351222] LDISKFS-fs (md77): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[14311446.788776] LDISKFS-fs (md77): file extents enabled, maximum tree depth=5
[14311447.274250] LDISKFS-fs (md77): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[14311450.693273] Lustre: oak-OST015d: new disk, initializing
[14311450.720461] Lustre: srv-oak-OST015d: No data found on store. Initialize space
[14311450.870692] Lustre: oak-OST015d: Imperative Recovery enabled, recovery window shrunk from 300-900 down to 150-900
[14311456.630842] Lustre: cli-oak-OST015d-super: Allocated super-sequence [0x0000005ac0000400-0x0000005b00000400]:15d:ost]
[14311496.181591] blk_update_request: I/O error, dev dm-205, sector 8
[14311498.350156] md: md79 stopped.
[14311498.360843] md/raid:md79: device dm-754 operational as raid disk 0
[14311498.367293] md/raid:md79: device dm-695 operational as raid disk 9
[14311498.373719] md/raid:md79: device dm-699 operational as raid disk 8
[14311498.380157] md/raid:md79: device dm-680 operational as raid disk 7
[14311498.386587] md/raid:md79: device dm-690 operational as raid disk 6
[14311498.393040] md/raid:md79: device dm-691 operational as raid disk 5
[14311498.399513] md/raid:md79: device dm-763 operational as raid disk 4
[14311498.405953] md/raid:md79: device dm-761 operational as raid disk 3
[14311498.412379] md/raid:md79: device dm-758 operational as raid disk 2
[14311498.418801] md/raid:md79: device dm-740 operational as raid disk 1
[14311498.427374] md/raid:md79: raid level 6 active with 10 out of 10 devices, algorithm 2
[14311498.448797] md79: detected capacity change from 0 to 112003075014656
[14311499.172490] LDISKFS-fs (md79): file extents enabled, maximum tree depth=5
[14311499.589662] LDISKFS-fs (md79): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[14311500.128170] LDISKFS-fs (md79): file extents enabled, maximum tree depth=5
[14311500.494506] LDISKFS-fs (md79): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[14311576.535902] Lustre: oak-OST0113: Connection restored to 72e9c356-8685-a676-ede7-2f53484b502b (at 10.50.10.7@o2ib2)
[14311576.546515] Lustre: Skipped 13533 previous similar messages
[14312177.306920] Lustre: oak-OST014d: Connection restored to ad12f851-216b-c412-f51f-518aba8ea609 (at 10.210.9.195@tcp1)
[14312177.317721] Lustre: Skipped 957 previous similar messages
[14312612.619466] Lustre: oak-OST013d: haven't heard from client 0d1c8726-ce4c-fcc8-6c1b-7179039cabd2 (at 10.210.12.8@tcp1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9178ee933400, cur 1645539667 expire 1645539517 last 1645539440
[14312612.641589] Lustre: Skipped 15 previous similar messages
[14312617.645241] Lustre: oak-OST014b: haven't heard from client 0d1c8726-ce4c-fcc8-6c1b-7179039cabd2 (at 10.210.12.8@tcp1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff913f66367000, cur 1645539672 expire 1645539522 last 1645539445
[14312617.667276] Lustre: Skipped 18 previous similar messages
[14312776.566473] Lustre: oak-OST0143: Connection restored to dd24f53c-f593-3bf4-b3ba-3f9a7289f787 (at 10.210.12.36@tcp1)
[14312776.577148] Lustre: Skipped 862 previous similar messages
[14313377.148017] Lustre: oak-OST0153: Connection restored to e57d0197-94f4-5c3b-6159-b845a19bf5d8 (at 10.50.16.5@o2ib2)
[14313377.158601] Lustre: Skipped 1031 previous similar messages
[14313980.570721] Lustre: oak-OST0125: Connection restored to c8d15be5-5238-2528-45c3-8c06cacdd0d3 (at 10.50.1.45@o2ib2)
[14313980.581323] Lustre: Skipped 640 previous similar messages
[14314589.067138] Lustre: oak-OST012d: Connection restored to 27d57912-6776-078b-59ad-39eae43109d2 (at 10.50.7.34@o2ib2)
[14314589.077825] Lustre: Skipped 742 previous similar messages
[14315190.987427] Lustre: oak-OST0151: Connection restored to 39d0f2bf-c4d0-c58b-e434-699489ac41f3 (at 10.50.1.10@o2ib2)
[14315190.998073] Lustre: Skipped 814 previous similar messages
[14315790.915637] Lustre: oak-OST0153: Connection restored to 5290337e-35bf-6154-3d5a-b01e32eaf7c5 (at 10.50.8.44@o2ib2)
[14315790.926219] Lustre: Skipped 949 previous similar messages
[14316389.763543] Lustre: oak-OST015b: Connection restored to 19564d1b-52d9-bf0b-75d7-d97ffab3e50d (at 10.0.3.57@o2ib5)
[14316389.774040] Lustre: Skipped 750 previous similar messages
[14316988.806794] Lustre: oak-OST014f: Connection restored to 309dc1e4-930a-6053-8940-903da12f7d9c (at 10.210.12.76@tcp1)
[14316988.817509] Lustre: Skipped 626 previous similar messages
[14317588.289792] Lustre: oak-OST0111: Connection restored to 2812a9b3-b6f5-4918-ae96-1f2250fc2778 (at 10.50.10.41@o2ib2)
[14317588.300479] Lustre: Skipped 814 previous similar messages
[14318188.692390] Lustre: oak-OST014f: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[14318188.703057] Lustre: Skipped 997 previous similar messages
[14318787.316914] Lustre: oak-OST013b: Connection restored to  (at 10.50.6.49@o2ib2)
[14318787.324386] Lustre: Skipped 6485 previous similar messages
[14319099.945282] Lustre: oak-OST0119: Client 6b6cd352-15e2-b5e6-2cf2-86cbf0e9b626 (at 10.210.12.145@tcp1) reconnecting
[14319099.955782] Lustre: Skipped 17 previous similar messages
[14319106.007180] Lustre: oak-OST011b: Client 6b6cd352-15e2-b5e6-2cf2-86cbf0e9b626 (at 10.210.12.145@tcp1) reconnecting
[14319387.003441] Lustre: oak-OST0159: Connection restored to e57d0197-94f4-5c3b-6159-b845a19bf5d8 (at 10.50.16.5@o2ib2)
[14319387.014014] Lustre: Skipped 3316 previous similar messages
[14319757.659397] LustreError: 21606:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 3145728(4194304)  req@ffff917eea141050 x1715054491019392/t0(0) o3->09198db0-ac48-c42e-4de3-f0cdbdb971ff@10.210.12.36@tcp1:208/0 lens 488/440 e 0 to 0 dl 1645546868 ref 1 fl Interpret:/0/0 rc 0/0
[14319757.685091] Lustre: oak-OST012f: Bulk IO read error with 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1), client will retry: rc -110
[14319757.698303] Lustre: Skipped 10 previous similar messages
[14319865.268547] Lustre: oak-OST0145: Client 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1) reconnecting
[14319865.278992] Lustre: Skipped 14 previous similar messages
[14319870.019444] Lustre: oak-OST0125: Client 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1) reconnecting
[14319870.029882] Lustre: Skipped 1 previous similar message
[14319872.016142] Lustre: oak-OST0127: Client 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1) reconnecting
[14319872.026759] Lustre: Skipped 3 previous similar messages
[14319909.907101] Lustre: oak-OST015d: haven't heard from client 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9180c523e000, cur 1645546982 expire 1645546832 last 1645546755
[14319909.929120] Lustre: Skipped 1 previous similar message
[14319949.243668] LustreError: 243456:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff91d4f456f850 x1716225638543936/t0(0) o3->6b6cd352-15e2-b5e6-2cf2-86cbf0e9b626@10.210.12.145@tcp1:397/0 lens 488/440 e 0 to 0 dl 1645547057 ref 1 fl Interpret:/0/0 rc 0/0
[14319949.243824] Lustre: oak-OST014d: Bulk IO read error with 6b6cd352-15e2-b5e6-2cf2-86cbf0e9b626 (at 10.210.12.145@tcp1), client will retry: rc -110
[14319949.282556] LustreError: 243456:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 1 previous similar message
[14319972.288716] LustreError: 137-5: oak-OST013c_UUID: not available for connect from 10.210.12.145@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14319972.306419] LustreError: Skipped 233 previous similar messages
[14319973.195443] LustreError: 243498:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 1048576(2445312)  req@ffff9146fbd5b050 x1715055706132928/t0(0) o4->92828426-e9f0-508a-3226-9184d3612426@10.210.12.38@tcp1:428/0 lens 488/448 e 0 to 0 dl 1645547088 ref 1 fl Interpret:/0/0 rc 0/0
[14319973.221426] Lustre: oak-OST0155: Bulk IO write error with 92828426-e9f0-508a-3226-9184d3612426 (at 10.210.12.38@tcp1), client will retry: rc = -110
[14319973.234857] Lustre: Skipped 2 previous similar messages
[14319974.858533] LustreError: 137-5: oak-OST0158_UUID: not available for connect from 10.210.12.39@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14319985.914999] Lustre: oak-OST0151: Connection restored to d2f49117-87f4-d939-d915-51fa6430aa6e (at 10.51.12.14@o2ib3)
[14319985.925670] Lustre: Skipped 2211 previous similar messages
[14320001.946530] Lustre: oak-OST0155: Client 92828426-e9f0-508a-3226-9184d3612426 (at 10.210.12.38@tcp1) reconnecting
[14320001.956979] Lustre: Skipped 26 previous similar messages
[14320059.766030] Lustre: oak-OST014d: Client 6b6cd352-15e2-b5e6-2cf2-86cbf0e9b626 (at 10.210.12.145@tcp1) reconnecting
[14320076.416447] Lustre: oak-OST0141: Client 6b6cd352-15e2-b5e6-2cf2-86cbf0e9b626 (at 10.210.12.145@tcp1) reconnecting
[14320076.426948] Lustre: Skipped 52 previous similar messages
[14320096.549254] Lustre: oak-OST0153: haven't heard from client 1398f2c9-c8d8-aea5-1ce6-ef88e0cc6405 (at 10.210.12.39@tcp1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff9162badd9800, cur 1645547169 expire 1645547019 last 1645546942
[14320188.733333] LustreError: 248297:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91be9b071850 x1715071818901120/t0(0) o4->bb55a01e-02a3-cbba-5b76-e478fd40967c@10.210.12.54@tcp1:631/0 lens 488/448 e 0 to 0 dl 1645547291 ref 1 fl Interpret:/0/0 rc 0/0
[14320188.733626] Lustre: oak-OST0143: Bulk IO write error with bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1), client will retry: rc = -110
[14320188.772579] LustreError: 248297:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[14320212.335349] Lustre: oak-OST0143: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[14320212.345786] Lustre: Skipped 53 previous similar messages
[14320260.581844] LustreError: 162713:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 1048576(2428928)  req@ffff9183cacda050 x1715054494027456/t0(0) o4->09198db0-ac48-c42e-4de3-f0cdbdb971ff@10.210.12.36@tcp1:716/0 lens 488/448 e 0 to 0 dl 1645547376 ref 1 fl Interpret:/0/0 rc 0/0
[14320260.607809] Lustre: oak-OST0145: Bulk IO write error with 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1), client will retry: rc = -110
[14320260.621290] Lustre: Skipped 2 previous similar messages
[14320289.153432] Lustre: oak-OST0145: Client 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1) reconnecting
[14320585.200523] Lustre: oak-OST011d: Connection restored to 2d7b80e6-0f0d-a2ea-c50b-4f791ee979a6 (at 10.51.16.21@o2ib3)
[14320585.211189] Lustre: Skipped 1161 previous similar messages
[14321184.042180] Lustre: oak-OST0149: Connection restored to 963f25d3-89fe-9827-147e-b4bc4770d3be (at 10.0.3.127@o2ib5)
[14321184.052759] Lustre: Skipped 1567 previous similar messages
[14321782.767947] Lustre: oak-OST011f: Connection restored to 9dd85d36-fd39-cffc-4cf0-91d71c907eef (at 10.50.1.56@o2ib2)
[14321782.778522] Lustre: Skipped 1517 previous similar messages
[14322041.785208] Lustre: oak-OST0123: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[14322041.795747] Lustre: Skipped 62 previous similar messages
[14322058.255830] Lustre: oak-OST0115: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[14322058.266258] Lustre: Skipped 22 previous similar messages
[14322387.382326] Lustre: oak-OST012b: Connection restored to 348a13e9-7855-7c45-c824-7175c68035a2 (at 10.50.5.51@o2ib2)
[14322387.392904] Lustre: Skipped 1396 previous similar messages
[14322726.688041] Lustre: oak-OST0141: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[14322726.698455] Lustre: Skipped 8 previous similar messages
[14322727.192285] LustreError: 127351:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91c98ece6050 x1715071829152128/t0(0) o4->bb55a01e-02a3-cbba-5b76-e478fd40967c@10.210.12.54@tcp1:219/0 lens 488/448 e 0 to 0 dl 1645549899 ref 1 fl Interpret:/0/0 rc 0/0
[14322727.216854] Lustre: oak-OST0155: Bulk IO write error with bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1), client will retry: rc = -110
[14322728.625798] LustreError: 21623:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91dae6661850 x1715071829152256/t0(0) o4->bb55a01e-02a3-cbba-5b76-e478fd40967c@10.210.12.54@tcp1:223/0 lens 488/448 e 0 to 0 dl 1645549903 ref 1 fl Interpret:/2/0 rc 0/0
[14322728.650132] LustreError: 21623:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[14322728.660104] Lustre: oak-OST0141: Bulk IO write error with bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1), client will retry: rc = -110
[14322728.673541] Lustre: Skipped 1 previous similar message
[14322894.766130] Lustre: oak-OST013b: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[14322894.776545] Lustre: Skipped 1 previous similar message
[14322903.760874] Lustre: oak-OST0127: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[14322903.771278] Lustre: Skipped 21 previous similar messages
[14322986.050365] Lustre: oak-OST013b: Connection restored to bd65b2a3-48a0-d6f2-6e15-a5dd696dd3c7 (at 10.51.6.8@o2ib3)
[14322986.060909] Lustre: Skipped 1472 previous similar messages
[14323585.193214] Lustre: oak-OST014b: Connection restored to 7ff60668-8016-5d05-b0eb-edc799b40f84 (at 10.51.12.10@o2ib3)
[14323585.203896] Lustre: Skipped 2004 previous similar messages
[14324000.631764] LustreError: 127358:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[14324184.755172] Lustre: oak-OST0153: Connection restored to 5e4bce85-4d6e-90c6-523c-4bdf35e8bc4a (at 10.210.12.7@tcp1)
[14324184.765754] Lustre: Skipped 2409 previous similar messages
[14324701.401016] Lustre: oak-OST013b: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[14324701.411426] Lustre: Skipped 5 previous similar messages
[14324702.188075] LustreError: 243526:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91728f751850 x1714983411566400/t0(0) o4->d4faaef8-44e2-b05f-6116-d9c30b2e4e11@10.210.12.60@tcp1:687/0 lens 488/448 e 0 to 0 dl 1645551877 ref 1 fl Interpret:/0/0 rc 0/0
[14324702.212632] Lustre: oak-OST013b: Bulk IO write error with d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1), client will retry: rc = -110
[14324703.209590] LustreError: 160920:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9198f0784850 x1714983411566400/t0(0) o4->d4faaef8-44e2-b05f-6116-d9c30b2e4e11@10.210.12.60@tcp1:693/0 lens 488/448 e 0 to 0 dl 1645551883 ref 1 fl Interpret:/2/0 rc 0/0
[14324703.234211] Lustre: oak-OST013b: Bulk IO write error with d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1), client will retry: rc = -110
[14324782.534569] LustreError: 137-5: oak-OST014a_UUID: not available for connect from 10.210.12.40@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14324784.597203] Lustre: oak-OST0145: Connection restored to 72e9c356-8685-a676-ede7-2f53484b502b (at 10.50.10.7@o2ib2)
[14324784.607799] Lustre: Skipped 2039 previous similar messages
[14324867.891120] Lustre: oak-OST011b: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[14324867.901606] Lustre: Skipped 3 previous similar messages
[14324872.209460] Lustre: oak-OST0145: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[14324872.219908] Lustre: Skipped 20 previous similar messages
[14324880.226297] LustreError: 21592:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff918fc7976850 x1716250920017280/t0(0) o3->6b341d38-3674-15b6-7e5e-137d0b4498c0@10.210.12.40@tcp1:114/0 lens 488/440 e 0 to 0 dl 1645552059 ref 1 fl Interpret:/0/0 rc 0/0
[14324880.250648] Lustre: oak-OST0123: Bulk IO read error with 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1), client will retry: rc -110
[14324880.263829] Lustre: Skipped 1 previous similar message
[14324880.344748] Lustre: oak-OST0125: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[14324880.355294] Lustre: Skipped 36 previous similar messages
[14325384.197060] Lustre: oak-OST0113: Connection restored to  (at 10.51.15.3@o2ib3)
[14325384.204598] Lustre: Skipped 1489 previous similar messages
[14325948.451507] Lustre: oak-OST0121: Client 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1) reconnecting
[14325948.461919] Lustre: Skipped 8 previous similar messages
[14325951.146725] Lustre: oak-OST011f: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[14325951.157174] Lustre: Skipped 2 previous similar messages
[14325956.970538] Lustre: oak-OST0111: Client 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1) reconnecting
[14325956.980950] Lustre: Skipped 7 previous similar messages
[14325966.397705] Lustre: oak-OST014d: Client b4fca75a-2d54-9bf2-331f-67d0c52e5ff2 (at 10.210.12.73@tcp1) reconnecting
[14325966.408221] Lustre: Skipped 10 previous similar messages
[14325982.745587] Lustre: oak-OST0129: Connection restored to e5abbf7d-e5e6-332b-7725-53dff65e566e (at 10.50.4.16@o2ib2)
[14325982.756204] Lustre: Skipped 2070 previous similar messages
[14326048.412126] Lustre: oak-OST0139: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[14326048.422562] Lustre: Skipped 5 previous similar messages
[14326581.510557] Lustre: oak-OST0117: Connection restored to  (at 10.51.5.72@o2ib3)
[14326581.518058] Lustre: Skipped 3229 previous similar messages
[14327072.643702] Lustre: oak-OST013d: Client 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1) reconnecting
[14327077.513787] Lustre: oak-OST0121: Client 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1) reconnecting
[14327077.524198] Lustre: Skipped 6 previous similar messages
[14327088.728358] Lustre: oak-OST011f: Client 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1) reconnecting
[14327088.738792] Lustre: Skipped 18 previous similar messages
[14327181.049040] Lustre: oak-OST011b: Connection restored to ec2bf979-59d1-9bff-9f32-205ef2acc482 (at 10.50.17.25@o2ib2)
[14327181.059709] Lustre: Skipped 2548 previous similar messages
[14327780.828064] Lustre: oak-OST0155: Connection restored to 010c0fa9-cabf-a6b8-0616-88e4b85f309a (at 10.50.2.30@o2ib2)
[14327780.838644] Lustre: Skipped 2195 previous similar messages
[14328379.638912] Lustre: oak-OST015b: Connection restored to  (at 10.51.0.66@o2ib3)
[14328379.646398] Lustre: Skipped 1985 previous similar messages
[14328978.614656] Lustre: oak-OST0123: Connection restored to e2ecf35d-5356-16a9-a2e1-a931a78b229b (at 10.51.2.47@o2ib3)
[14328978.625266] Lustre: Skipped 1804 previous similar messages
[14329579.935168] Lustre: oak-OST011b: Connection restored to  (at 10.51.15.3@o2ib3)
[14329579.942641] Lustre: Skipped 2754 previous similar messages
[14330178.548301] Lustre: oak-OST0121: Connection restored to b8b68d6b-45f6-9491-ce0d-e3607eec1df7 (at 10.51.4.63@o2ib3)
[14330178.558886] Lustre: Skipped 2380 previous similar messages
[14330777.874513] Lustre: oak-OST014f: Connection restored to 074fb900-b211-45e9-d5d5-3a79c52acd75 (at 10.210.12.17@tcp1)
[14330777.885287] Lustre: Skipped 2098 previous similar messages
[14331376.717748] Lustre: oak-OST0149: Connection restored to 4025d53e-c01c-f5a6-a7de-c3a1c13db1f4 (at 10.50.10.5@o2ib2)
[14331376.728369] Lustre: Skipped 2348 previous similar messages
[14331975.626381] Lustre: oak-OST015d: Connection restored to ad7d9419-d1db-2c8c-185a-26a2198bdccd (at 10.51.3.39@o2ib3)
[14331975.636976] Lustre: Skipped 2087 previous similar messages
[14332345.165740] Lustre: oak-OST0153: Client 3f016e1f-54ab-68d5-4642-4d55843078e2 (at 10.210.11.23@tcp1) reconnecting
[14332345.176196] Lustre: Skipped 4 previous similar messages
[14332360.344343] Lustre: oak-OST0151: Client 3f016e1f-54ab-68d5-4642-4d55843078e2 (at 10.210.11.23@tcp1) reconnecting
[14332369.114048] Lustre: oak-OST014b: Client 3f016e1f-54ab-68d5-4642-4d55843078e2 (at 10.210.11.23@tcp1) reconnecting
[14332400.389366] Lustre: oak-OST0149: Client 3f016e1f-54ab-68d5-4642-4d55843078e2 (at 10.210.11.23@tcp1) reconnecting
[14332400.399977] Lustre: Skipped 1 previous similar message
[14332439.055302] Lustre: oak-OST013d: Client 3f016e1f-54ab-68d5-4642-4d55843078e2 (at 10.210.11.23@tcp1) reconnecting
[14332574.245324] Lustre: oak-OST015b: Connection restored to 2812a9b3-b6f5-4918-ae96-1f2250fc2778 (at 10.50.10.41@o2ib2)
[14332574.257589] Lustre: Skipped 2130 previous similar messages
[14333172.866007] Lustre: oak-OST0159: Connection restored to c977f6f4-35de-1e9a-97fb-439d33a2737d (at 10.0.3.6@o2ib5)
[14333172.876435] Lustre: Skipped 1679 previous similar messages
[14333771.564621] Lustre: oak-OST012f: Connection restored to b0ece11f-4f3a-392e-b625-14fcf38a3d92 (at 10.50.14.8@o2ib2)
[14333771.575473] Lustre: Skipped 1988 previous similar messages
[14334370.556383] Lustre: oak-OST0125: Connection restored to  (at 10.51.0.67@o2ib3)
[14334370.563905] Lustre: Skipped 1960 previous similar messages
[14334971.210119] Lustre: oak-OST014f: Connection restored to c8c804bb-e728-6f8c-527d-295ebfa95786 (at 10.50.4.35@o2ib2)
[14334971.220714] Lustre: Skipped 1710 previous similar messages
[14335571.337711] Lustre: oak-OST014d: Connection restored to 55fa4ecb-ff66-16c3-914e-01f9698a4d93 (at 10.50.12.16@o2ib2)
[14335571.348415] Lustre: Skipped 1554 previous similar messages
[14336171.408258] Lustre: oak-OST013f: Connection restored to 2dc703cd-5cca-815b-6dd4-cd3e7d5a0392 (at 10.0.3.49@o2ib5)
[14336171.418800] Lustre: Skipped 1406 previous similar messages
[14336769.966492] Lustre: oak-OST0121: Connection restored to 46321288-503a-e4fe-72c0-e0f80ca6ae60 (at 10.51.6.44@o2ib3)
[14336769.977107] Lustre: Skipped 3176 previous similar messages
[14337368.968154] Lustre: oak-OST0137: Connection restored to 080fbf36-50e2-e037-d579-97f7ea4dd6a6 (at 10.210.12.51@tcp1)
[14337368.978826] Lustre: Skipped 1753 previous similar messages
[14337967.531264] Lustre: oak-OST0159: Connection restored to 95ceee19-b1cb-9489-eb13-6462d6c898cd (at 10.51.12.21@o2ib3)
[14337967.541942] Lustre: Skipped 2770 previous similar messages
[14338567.764133] Lustre: oak-OST0157: Connection restored to 1e354a56-47c1-6143-541f-2368817925be (at 10.50.17.40@o2ib2)
[14338567.774817] Lustre: Skipped 2093 previous similar messages
[14339167.056373] Lustre: oak-OST0155: Connection restored to 5716f6d5-1ff3-352d-6ffc-7dd360d957ba (at 10.51.4.69@o2ib3)
[14339167.066959] Lustre: Skipped 1874 previous similar messages
[14339765.637745] Lustre: oak-OST015d: Connection restored to 99627209-92e6-01ab-afb3-3c05a7969f40 (at 10.50.5.56@o2ib2)
[14339765.648415] Lustre: Skipped 1558 previous similar messages
[14339889.894384] Lustre: oak-OST0111: Client bed08025-01aa-dea1-aa90-bead26dab2fb (at 10.50.17.21@o2ib2) reconnecting
[14339889.904805] Lustre: Skipped 6 previous similar messages
[14340365.234418] Lustre: oak-OST0151: Connection restored to  (at 10.50.15.11@o2ib2)
[14340365.242037] Lustre: Skipped 1502 previous similar messages
[14340964.130206] Lustre: oak-OST015b: Connection restored to d61a1999-1860-4450-6f4c-b19b832004d7 (at 10.51.14.23@o2ib3)
[14340964.140880] Lustre: Skipped 1747 previous similar messages
[14341018.576714] Lustre: oak-OST0133: haven't heard from client cece835f-7db0-6bcd-3288-c86d9688e4ba (at 10.51.15.7@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91963e7cb800, cur 1645568142 expire 1645567992 last 1645567915
[14341020.589233] Lustre: oak-OST0129: haven't heard from client cece835f-7db0-6bcd-3288-c86d9688e4ba (at 10.51.15.7@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff919d97ee8c00, cur 1645568144 expire 1645567994 last 1645567917
[14341020.611193] Lustre: Skipped 24 previous similar messages
[14341539.221890] Lustre: oak-OST014f: Client e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1) reconnecting
[14341539.232302] Lustre: Skipped 1 previous similar message
[14341539.241246] LustreError: 243455:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91d421078850 x1715245262770432/t0(0) o4->e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda@10.210.12.79@tcp1:199/0 lens 488/448 e 0 to 0 dl 1645568754 ref 1 fl Interpret:/0/0 rc 0/0
[14341539.256004] Lustre: oak-OST011d: Bulk IO write error with e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1), client will retry: rc = -110
[14341539.279112] LustreError: 243455:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 6 previous similar messages
[14341563.142956] Lustre: oak-OST0159: Connection restored to 0453ccd0-e0b0-0e88-c206-cfc123f6a6e1 (at 10.50.8.64@o2ib2)
[14341563.153551] Lustre: Skipped 1699 previous similar messages
[14341705.375860] Lustre: oak-OST0147: Client 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1) reconnecting
[14341710.521241] Lustre: oak-OST0135: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[14341710.531651] Lustre: Skipped 1 previous similar message
[14341712.891359] Lustre: oak-OST0115: Client 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1) reconnecting
[14341712.901887] Lustre: Skipped 11 previous similar messages
[14341719.220851] Lustre: oak-OST0117: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[14341719.231272] Lustre: Skipped 1 previous similar message
[14341728.335869] Lustre: oak-OST0119: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[14341728.346301] Lustre: Skipped 4 previous similar messages
[14341755.322042] Lustre: oak-OST011b: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[14341755.332449] Lustre: Skipped 5 previous similar messages
[14342161.989557] Lustre: oak-OST013d: Connection restored to 2a2bdf9e-cf37-ebcf-4507-0de7e8bde666 (at 10.50.9.9@o2ib2)
[14342162.000059] Lustre: Skipped 1903 previous similar messages
[14342760.846961] Lustre: oak-OST011d: Connection restored to 10195af4-6a2b-655f-bb92-6ca79e043b31 (at 10.50.5.4@o2ib2)
[14342760.857512] Lustre: Skipped 1779 previous similar messages
[14343359.486356] Lustre: oak-OST0115: Connection restored to c193930f-3dab-28da-0eac-55422998d272 (at 10.51.13.13@o2ib3)
[14343359.497040] Lustre: Skipped 1703 previous similar messages
[14343958.188017] Lustre: oak-OST014f: Connection restored to eee699b2-cb8e-0a91-6752-be0f592e4945 (at 10.50.13.2@o2ib2)
[14343958.198676] Lustre: Skipped 1667 previous similar messages
[14344557.361850] Lustre: oak-OST0113: Connection restored to aef304b4-8679-e0d5-9fa5-c78542ec1535 (at 10.50.2.25@o2ib2)
[14344557.372429] Lustre: Skipped 2518 previous similar messages
[14345158.663158] Lustre: oak-OST0131: Connection restored to  (at 10.50.16.2@o2ib2)
[14345158.670669] Lustre: Skipped 1636 previous similar messages
[14345757.618939] Lustre: oak-OST0139: Connection restored to  (at 10.51.15.12@o2ib3)
[14345757.626511] Lustre: Skipped 1786 previous similar messages
[14346356.281462] Lustre: oak-OST0133: Connection restored to 95ceee19-b1cb-9489-eb13-6462d6c898cd (at 10.51.12.21@o2ib3)
[14346356.292252] Lustre: Skipped 2337 previous similar messages
[14346956.981081] Lustre: oak-OST0157: Connection restored to 24f4fd3c-862d-7711-d262-7168de5300fc (at 10.51.12.5@o2ib3)
[14346956.991683] Lustre: Skipped 1940 previous similar messages
[14347556.389109] Lustre: oak-OST014b: Connection restored to 7e7e9ac5-0ff6-2601-4894-f85117745486 (at 10.50.9.49@o2ib2)
[14347556.399692] Lustre: Skipped 2542 previous similar messages
[14347845.851201] LustreError: 162698:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[14348154.948212] Lustre: oak-OST0125: Connection restored to  (at 10.50.16.8@o2ib2)
[14348154.955678] Lustre: Skipped 1718 previous similar messages
[14348753.665697] Lustre: oak-OST0155: Connection restored to d58381cd-eaed-70a2-681b-6663c8d28df5 (at 10.210.12.56@tcp1)
[14348753.676362] Lustre: Skipped 1908 previous similar messages
[14349353.195289] Lustre: oak-OST0157: Connection restored to 471d8473-ce4f-aec7-199c-b5ad6376849b (at 10.50.12.15@o2ib2)
[14349353.205987] Lustre: Skipped 2218 previous similar messages
[14349700.775057] Lustre: oak-OST0147: Client 1be164da-11a7-a3a8-bd17-8ca7a5aab4e8 (at 10.210.12.69@tcp1) reconnecting
[14349700.785525] Lustre: Skipped 4 previous similar messages
[14349701.016863] LustreError: 162678:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9179f20e0050 x1714998310112960/t0(0) o4->1be164da-11a7-a3a8-bd17-8ca7a5aab4e8@10.210.12.69@tcp1:78/0 lens 488/448 e 0 to 0 dl 1645576938 ref 1 fl Interpret:/0/0 rc 0/0
[14349701.017894] Lustre: oak-OST0147: Bulk IO write error with 1be164da-11a7-a3a8-bd17-8ca7a5aab4e8 (at 10.210.12.69@tcp1), client will retry: rc = -110
[14349701.017895] Lustre: Skipped 10 previous similar messages
[14349701.060332] LustreError: 162678:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 3 previous similar messages
[14349780.529650] LustreError: 137-5: oak-OST015e_UUID: not available for connect from 10.210.12.69@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14349782.547187] LustreError: 137-5: oak-OST0136_UUID: not available for connect from 10.210.12.69@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14349869.362588] Lustre: oak-OST012d: Client 1be164da-11a7-a3a8-bd17-8ca7a5aab4e8 (at 10.210.12.69@tcp1) reconnecting
[14349881.312751] Lustre: oak-OST0117: Client 1be164da-11a7-a3a8-bd17-8ca7a5aab4e8 (at 10.210.12.69@tcp1) reconnecting
[14349881.323189] Lustre: Skipped 26 previous similar messages
[14349951.802186] Lustre: oak-OST014b: Connection restored to 0e45fe5d-752d-d535-332d-fa13e8c4fef7 (at 10.50.12.6@o2ib2)
[14349951.812769] Lustre: Skipped 3109 previous similar messages
[14350551.971014] Lustre: oak-OST0111: Connection restored to  (at 10.51.2.31@o2ib3)
[14350551.978483] Lustre: Skipped 2356 previous similar messages
[14351150.955539] Lustre: oak-OST0151: Connection restored to 43a7bf00-2ec1-3fdd-5cf6-946289157379 (at 10.50.10.69@o2ib2)
[14351150.966267] Lustre: Skipped 2320 previous similar messages
[14351750.131545] Lustre: oak-OST0141: Connection restored to 51ab7e05-78ff-bf41-8734-92d0767bb7ee (at 10.50.7.21@o2ib2)
[14351750.142126] Lustre: Skipped 1223 previous similar messages
[14352350.314279] Lustre: oak-OST0127: Connection restored to 27d57912-6776-078b-59ad-39eae43109d2 (at 10.50.7.34@o2ib2)
[14352350.324929] Lustre: Skipped 1646 previous similar messages
[14352949.087242] Lustre: oak-OST0111: Connection restored to 2923db79-f47c-6a89-eb88-9777651f3826 (at 10.51.1.3@o2ib3)
[14352949.097733] Lustre: Skipped 1704 previous similar messages
[14353549.412106] Lustre: oak-OST0145: Connection restored to  (at 10.50.7.2@o2ib2)
[14353549.419953] Lustre: Skipped 2059 previous similar messages
[14354148.411408] Lustre: oak-OST015b: Connection restored to ab61d733-6278-aecc-ffdb-baa33579a940 (at 10.51.4.33@o2ib3)
[14354148.422001] Lustre: Skipped 1807 previous similar messages
[14354521.295792] Lustre: oak-OST015d: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[14354521.958557] LustreError: 243555:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9182b9e04050 x1714983608068224/t0(0) o4->d4faaef8-44e2-b05f-6116-d9c30b2e4e11@10.210.12.60@tcp1:385/0 lens 504/448 e 0 to 0 dl 1645581775 ref 1 fl Interpret:/0/0 rc 0/0
[14354521.983336] Lustre: oak-OST015d: Bulk IO write error with d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1), client will retry: rc = -110
[14354521.996812] Lustre: Skipped 1 previous similar message
[14354691.551241] Lustre: oak-OST011b: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[14354691.561658] Lustre: Skipped 2 previous similar messages
[14354695.544210] Lustre: oak-OST0111: Client d4faaef8-44e2-b05f-6116-d9c30b2e4e11 (at 10.210.12.60@tcp1) reconnecting
[14354695.554693] Lustre: Skipped 21 previous similar messages
[14354747.124170] Lustre: oak-OST0153: Connection restored to 963f25d3-89fe-9827-147e-b4bc4770d3be (at 10.0.3.127@o2ib5)
[14354747.134754] Lustre: Skipped 2092 previous similar messages
[14355050.116256] LustreError: 21585:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[14355345.688968] Lustre: oak-OST0155: Connection restored to 6273d9a2-1357-be01-fa1a-e59246cee0f4 (at 10.210.12.105@tcp1)
[14355345.699726] Lustre: Skipped 2162 previous similar messages
[14355944.336542] Lustre: oak-OST015f: Connection restored to e596798d-98e3-2570-92e9-a92e6aea98bf (at 10.210.12.64@tcp1)
[14355944.347216] Lustre: Skipped 2096 previous similar messages
[14356063.629009] LustreError: 137-5: oak-OST0122_UUID: not available for connect from 10.210.12.71@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14356150.212411] Lustre: oak-OST012f: Client 140cbe03-4bf8-8306-6b3f-3f3adb8eb8d5 (at 10.210.12.71@tcp1) reconnecting
[14356150.222843] Lustre: Skipped 13 previous similar messages
[14356151.683297] Lustre: oak-OST011b: Client 140cbe03-4bf8-8306-6b3f-3f3adb8eb8d5 (at 10.210.12.71@tcp1) reconnecting
[14356151.693735] Lustre: Skipped 10 previous similar messages
[14356153.685319] Lustre: oak-OST0147: Client 140cbe03-4bf8-8306-6b3f-3f3adb8eb8d5 (at 10.210.12.71@tcp1) reconnecting
[14356153.695733] Lustre: Skipped 3 previous similar messages
[14356157.854882] Lustre: oak-OST0115: Client 140cbe03-4bf8-8306-6b3f-3f3adb8eb8d5 (at 10.210.12.71@tcp1) reconnecting
[14356157.865334] Lustre: Skipped 2 previous similar messages
[14356166.855989] Lustre: oak-OST012b: Client 140cbe03-4bf8-8306-6b3f-3f3adb8eb8d5 (at 10.210.12.71@tcp1) reconnecting
[14356166.866407] Lustre: Skipped 7 previous similar messages
[14356191.619896] Lustre: oak-OST011f: Client 140cbe03-4bf8-8306-6b3f-3f3adb8eb8d5 (at 10.210.12.71@tcp1) reconnecting
[14356191.630328] Lustre: Skipped 5 previous similar messages
[14356543.324859] Lustre: oak-OST0139: Connection restored to d1b48e85-1de4-7df7-4ce4-43c2dbac9631 (at 10.50.5.11@o2ib2)
[14356543.335465] Lustre: Skipped 2743 previous similar messages
[14357141.894177] Lustre: oak-OST0137: Connection restored to 820a1ce2-6182-e058-f1e9-e3948266ca32 (at 10.51.4.21@o2ib3)
[14357141.904756] Lustre: Skipped 2217 previous similar messages
[14357741.344846] Lustre: oak-OST0157: Connection restored to 3e72271e-d8a9-bff9-a052-ac4ffc30f7d6 (at 10.210.12.40@tcp1)
[14357741.355515] Lustre: Skipped 1996 previous similar messages
[14358340.314814] Lustre: oak-OST0159: Connection restored to b9b011f8-2c4b-d3d3-04e0-682dba50cd67 (at 10.210.12.42@tcp1)
[14358340.325484] Lustre: Skipped 2965 previous similar messages
[14358939.508498] Lustre: oak-OST011d: Connection restored to 46bd8889-1b94-f38a-f202-7268f2e23290 (at 10.50.13.10@o2ib2)
[14358939.519176] Lustre: Skipped 1771 previous similar messages
[14359540.177757] Lustre: oak-OST0145: Connection restored to fa202a18-aa95-b01f-fab7-7e4269883f98 (at 10.50.16.10@o2ib2)
[14359540.188550] Lustre: Skipped 1541 previous similar messages
[14360138.791881] Lustre: oak-OST0157: Connection restored to 15431f40-18fe-801a-ce02-67f7cb5f3e18 (at 10.210.12.60@tcp1)
[14360138.802591] Lustre: Skipped 1759 previous similar messages
[14360738.049437] Lustre: oak-OST0133: Connection restored to 080fbf36-50e2-e037-d579-97f7ea4dd6a6 (at 10.210.12.51@tcp1)
[14360738.060134] Lustre: Skipped 1961 previous similar messages
[14361147.732775] Lustre: oak-OST0117: Client 4786d009-fea5-0503-4a05-5dec2d29d509 (at 10.210.12.15@tcp1) reconnecting
[14361147.743202] Lustre: Skipped 7 previous similar messages
[14361151.728802] Lustre: oak-OST0119: Client 4786d009-fea5-0503-4a05-5dec2d29d509 (at 10.210.12.15@tcp1) reconnecting
[14361151.739218] Lustre: Skipped 5 previous similar messages
[14361336.686582] Lustre: oak-OST0129: Connection restored to b9268101-92c3-27cd-2000-c6a30a0cdb21 (at 10.51.2.38@o2ib3)
[14361336.697165] Lustre: Skipped 1693 previous similar messages
[14361935.235255] Lustre: oak-OST015f: Connection restored to 55ade167-ee14-7aa8-e90b-b61cc094ea5c (at 10.50.6.56@o2ib2)
[14361935.245837] Lustre: Skipped 1785 previous similar messages
[14362534.298650] Lustre: oak-OST0159: Connection restored to a9e8098e-525e-4ce6-042f-991d371b8d2d (at 10.51.6.39@o2ib3)
[14362534.309293] Lustre: Skipped 2265 previous similar messages
[14363133.267048] Lustre: oak-OST0155: Connection restored to 4025d53e-c01c-f5a6-a7de-c3a1c13db1f4 (at 10.50.10.5@o2ib2)
[14363133.277638] Lustre: Skipped 1897 previous similar messages
[14363733.863194] Lustre: oak-OST011d: Connection restored to 46bd8889-1b94-f38a-f202-7268f2e23290 (at 10.50.13.10@o2ib2)
[14363733.873867] Lustre: Skipped 1342 previous similar messages
[14364332.903071] Lustre: oak-OST0145: Connection restored to 4025d53e-c01c-f5a6-a7de-c3a1c13db1f4 (at 10.50.10.5@o2ib2)
[14364332.913681] Lustre: Skipped 1915 previous similar messages
[14364934.699384] Lustre: oak-OST015d: Connection restored to c01cc11f-1e0c-378c-4781-ae7e62a23ee0 (at 10.51.15.16@o2ib3)
[14364934.710050] Lustre: Skipped 1360 previous similar messages
[14365533.906468] Lustre: oak-OST0149: Connection restored to dcf50f0e-bb42-f2a0-6cab-85fbec7a53f3 (at 10.51.4.16@o2ib3)
[14365533.917068] Lustre: Skipped 1927 previous similar messages
[14366133.136476] Lustre: oak-OST0143: Connection restored to 38c93889-36e6-2464-4486-1e36f6eebee7 (at 10.51.5.40@o2ib3)
[14366133.147068] Lustre: Skipped 2357 previous similar messages
[14366731.885996] Lustre: oak-OST015b: Connection restored to c77af7fc-49aa-2585-7816-df90a6917753 (at 10.50.17.3@o2ib2)
[14366731.896578] Lustre: Skipped 1282 previous similar messages
[14367330.461037] Lustre: oak-OST0133: Connection restored to 61c6b9fe-2b90-a9ce-3651-82e83a219a08 (at 10.50.8.62@o2ib2)
[14367330.471677] Lustre: Skipped 2099 previous similar messages
[14367930.147274] Lustre: oak-OST015f: Connection restored to c9bb8838-a93c-f321-c22f-51d67c2ac5da (at 10.51.1.69@o2ib3)
[14367930.157870] Lustre: Skipped 1690 previous similar messages
[14368529.320763] Lustre: oak-OST011f: Connection restored to 582484fd-3cd6-e7cb-180b-ae6af9fe1e87 (at 10.50.10.8@o2ib2)
[14368529.331380] Lustre: Skipped 2178 previous similar messages
[14369128.791044] Lustre: oak-OST0121: Connection restored to 5290337e-35bf-6154-3d5a-b01e32eaf7c5 (at 10.50.8.44@o2ib2)
[14369128.801653] Lustre: Skipped 1224 previous similar messages
[14369727.597960] Lustre: oak-OST0155: Connection restored to ee36833d-7a97-4f8f-3fad-12018bb3bd79 (at 10.210.12.9@tcp1)
[14369727.608569] Lustre: Skipped 1036 previous similar messages
[14370326.361685] Lustre: oak-OST0159: Connection restored to 7bfbf310-f064-043a-78fe-2b6da6038484 (at 10.50.10.17@o2ib2)
[14370326.372365] Lustre: Skipped 1263 previous similar messages
[14370925.123324] Lustre: oak-OST014b: Connection restored to  (at 10.51.13.14@o2ib3)
[14370925.130908] Lustre: Skipped 927 previous similar messages
[14371312.385253] LustreError: 160920:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[14371523.962814] Lustre: oak-OST0141: Connection restored to 3abc1bac-6e1a-08b0-893b-1e34e457b6fb (at 10.50.5.32@o2ib2)
[14371523.973423] Lustre: Skipped 2975 previous similar messages
[14371584.714503] Lustre: oak-OST015b: Client 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1) reconnecting
[14371584.934135] LustreError: 243452:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9196aa3d5050 x1715767145301824/t0(0) o4->4b39aac6-2dc7-e48f-9434-e3bfde492453@10.210.12.29@tcp1:120/0 lens 488/448 e 0 to 0 dl 1645598875 ref 1 fl Interpret:/0/0 rc 0/0
[14371584.959217] Lustre: oak-OST015b: Bulk IO write error with 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1), client will retry: rc = -110
[14371587.844543] Lustre: oak-OST014f: Client e01e4c82-9fd9-5cc5-61dd-b09c7945df2b (at 10.210.12.11@tcp1) reconnecting
[14371587.854962] Lustre: Skipped 1 previous similar message
[14371587.975791] LustreError: 253938:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91a06cd4c050 x1714949500614592/t0(0) o4->e01e4c82-9fd9-5cc5-61dd-b09c7945df2b@10.210.12.11@tcp1:122/0 lens 488/448 e 0 to 0 dl 1645598877 ref 1 fl Interpret:/0/0 rc 0/0
[14371588.000225] LustreError: 253938:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[14371588.000883] Lustre: oak-OST014f: Bulk IO write error with e01e4c82-9fd9-5cc5-61dd-b09c7945df2b (at 10.210.12.11@tcp1), client will retry: rc = -110
[14371588.000884] Lustre: Skipped 1 previous similar message
[14371639.568093] LustreError: 127349:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff9193c3791050 x1715767145321600/t0(0) o3->4b39aac6-2dc7-e48f-9434-e3bfde492453@10.210.12.29@tcp1:123/0 lens 488/440 e 0 to 0 dl 1645598878 ref 1 fl Interpret:/0/0 rc 0/0
[14371639.568322] Lustre: oak-OST0129: Bulk IO read error with 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1), client will retry: rc -110
[14371639.568334] LustreError: 212522:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91adcfd13050 x1715767145328192/t0(0) o4->4b39aac6-2dc7-e48f-9434-e3bfde492453@10.210.12.29@tcp1:125/0 lens 488/448 e 0 to 0 dl 1645598880 ref 1 fl Interpret:/2/0 rc 0/0
[14371639.568510] Lustre: oak-OST015b: Bulk IO write error with 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1), client will retry: rc = -110
[14371639.568511] Lustre: Skipped 1 previous similar message
[14371639.651437] LustreError: 127349:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 3 previous similar messages
[14371667.112479] LustreError: 137-5: oak-OST015a_UUID: not available for connect from 10.210.12.29@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14371667.117731] Lustre: oak-OST0147: Client 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1) reconnecting
[14371667.140799] LustreError: Skipped 1 previous similar message
[14371754.488312] Lustre: oak-OST011b: Client e01e4c82-9fd9-5cc5-61dd-b09c7945df2b (at 10.210.12.11@tcp1) reconnecting
[14371754.498717] Lustre: Skipped 4 previous similar messages
[14371763.095995] Lustre: oak-OST0127: Client 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1) reconnecting
[14371763.106402] Lustre: Skipped 80 previous similar messages
[14371798.982032] LustreError: 21618:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[14372122.524342] Lustre: oak-OST013f: Connection restored to  (at 10.51.3.70@o2ib3)
[14372122.531870] Lustre: Skipped 4353 previous similar messages
[14372721.120833] Lustre: oak-OST012d: Connection restored to 153a6c97-da30-ff2a-b36b-e687db59475d (at 10.50.17.45@o2ib2)
[14372721.131555] Lustre: Skipped 3004 previous similar messages
[14373321.070575] Lustre: oak-OST0133: Connection restored to  (at 10.50.0.61@o2ib2)
[14373321.078055] Lustre: Skipped 1962 previous similar messages
[14373920.929501] Lustre: oak-OST0125: Connection restored to e5d4151d-94bb-31d0-aed5-7e54367726dc (at 10.51.4.23@o2ib3)
[14373920.940102] Lustre: Skipped 1044 previous similar messages
[14374240.892727] Lustre: oak-OST0159: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14374240.903476] Lustre: Skipped 14 previous similar messages
[14374243.344811] Lustre: oak-OST012b: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14374243.355227] Lustre: Skipped 2 previous similar messages
[14374246.699506] Lustre: 203496:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1645601277/real 1645601277]  req@ffff917b37bb7080 x1710539738952704/t0(0) o106->oak-OST0149@10.210.12.23@tcp1:15/16 lens 296/280 e 0 to 1 dl 1645601450 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
[14374247.342258] Lustre: oak-OST015f: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14374348.402005] Lustre: oak-OST0149: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14374348.412710] Lustre: Skipped 1 previous similar message
[14374519.973814] Lustre: oak-OST0155: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[14374519.984495] Lustre: Skipped 914 previous similar messages
[14375121.147808] Lustre: oak-OST0149: Connection restored to 9af80435-8fd4-9674-5e82-364c6d4344d5 (at 10.50.1.46@o2ib2)
[14375121.158384] Lustre: Skipped 789 previous similar messages
[14375342.766965] Lustre: oak-OST0113: Client 1398f2c9-c8d8-aea5-1ce6-ef88e0cc6405 (at 10.210.12.39@tcp1) reconnecting
[14375342.777378] Lustre: Skipped 1 previous similar message
[14375343.692183] LustreError: 160942:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91deeb563050 x1716235075772032/t0(0) o4->1398f2c9-c8d8-aea5-1ce6-ef88e0cc6405@10.210.12.39@tcp1:112/0 lens 488/448 e 0 to 0 dl 1645602642 ref 1 fl Interpret:/0/0 rc 0/0
[14375343.717025] Lustre: oak-OST0113: Bulk IO write error with 1398f2c9-c8d8-aea5-1ce6-ef88e0cc6405 (at 10.210.12.39@tcp1), client will retry: rc = -110
[14375343.730611] Lustre: Skipped 1 previous similar message
[14375346.212096] Lustre: oak-OST013b: Client 1398f2c9-c8d8-aea5-1ce6-ef88e0cc6405 (at 10.210.12.39@tcp1) reconnecting
[14375346.982209] LustreError: 244098:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91c25f086050 x1716235075906880/t0(0) o4->1398f2c9-c8d8-aea5-1ce6-ef88e0cc6405@10.210.12.39@tcp1:116/0 lens 488/448 e 0 to 0 dl 1645602646 ref 1 fl Interpret:/0/0 rc 0/0
[14375347.007218] Lustre: oak-OST0159: Bulk IO write error with 1398f2c9-c8d8-aea5-1ce6-ef88e0cc6405 (at 10.210.12.39@tcp1), client will retry: rc = -110
[14375347.226832] Lustre: oak-OST014d: Bulk IO read error with 1398f2c9-c8d8-aea5-1ce6-ef88e0cc6405 (at 10.210.12.39@tcp1), client will retry: rc -110
[14375347.240013] Lustre: Skipped 4 previous similar messages
[14375398.685721] LustreError: 243448:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(3383296)  req@ffff91cb000b9050 x1716235075787136/t0(0) o4->1398f2c9-c8d8-aea5-1ce6-ef88e0cc6405@10.210.12.39@tcp1:113/0 lens 488/448 e 0 to 0 dl 1645602643 ref 1 fl Interpret:/0/0 rc 0/0
[14375398.685936] Lustre: oak-OST014f: Bulk IO write error with 1398f2c9-c8d8-aea5-1ce6-ef88e0cc6405 (at 10.210.12.39@tcp1), client will retry: rc = -110
[14375398.685937] Lustre: Skipped 4 previous similar messages
[14375398.730596] LustreError: 243448:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 9 previous similar messages
[14375511.342859] Lustre: oak-OST014f: Client 1398f2c9-c8d8-aea5-1ce6-ef88e0cc6405 (at 10.210.12.39@tcp1) reconnecting
[14375511.353266] Lustre: Skipped 2 previous similar messages
[14375719.870471] Lustre: oak-OST0113: Connection restored to 1cd8c3ed-3a7a-1906-2c50-8828ca0d536f (at 10.50.17.33@o2ib2)
[14375719.881153] Lustre: Skipped 1042 previous similar messages
[14375830.715287] Lustre: oak-OST0127: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14375830.725706] Lustre: Skipped 10 previous similar messages
[14376038.934692] Lustre: oak-OST0121: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14376038.945169] Lustre: Skipped 15 previous similar messages
[14376191.341108] Lustre: oak-OST0155: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14376323.710950] Lustre: oak-OST011f: Connection restored to  (at 10.0.2.3@o2ib5)
[14376323.718272] Lustre: Skipped 920 previous similar messages
[14376338.376972] Lustre: oak-OST0135: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14376338.387404] Lustre: Skipped 5 previous similar messages
[14376926.322161] Lustre: oak-OST0157: Connection restored to  (at 10.50.7.2@o2ib2)
[14376926.329622] Lustre: Skipped 601 previous similar messages
[14377524.983417] Lustre: oak-OST0129: Connection restored to  (at 10.0.3.12@o2ib5)
[14377524.990795] Lustre: Skipped 763 previous similar messages
[14378127.324344] Lustre: oak-OST0111: Connection restored to 8e0d26dd-3f09-64a3-41d2-b7bb925cf363 (at 10.51.2.15@o2ib3)
[14378127.334946] Lustre: Skipped 628 previous similar messages
[14378725.928982] Lustre: oak-OST015b: Connection restored to c10b6301-1800-4510-94bd-883f949efde1 (at 10.50.6.68@o2ib2)
[14378725.939576] Lustre: Skipped 962 previous similar messages
[14379326.538721] Lustre: oak-OST014f: Connection restored to  (at 10.50.7.2@o2ib2)
[14379326.546101] Lustre: Skipped 930 previous similar messages
[14379927.514282] Lustre: oak-OST011b: Connection restored to d579f8e1-497a-3771-f6d4-ffe96dea772a (at 10.210.12.62@tcp1)
[14379927.524950] Lustre: Skipped 837 previous similar messages
[14380526.649248] Lustre: oak-OST011b: Connection restored to 46bd8889-1b94-f38a-f202-7268f2e23290 (at 10.50.13.10@o2ib2)
[14380526.659924] Lustre: Skipped 783 previous similar messages
[14381126.327289] Lustre: oak-OST014f: Connection restored to 9eb8189a-3985-f41c-7d7b-51af46681d5f (at 10.0.3.29@o2ib5)
[14381126.337815] Lustre: Skipped 810 previous similar messages
[14381725.141573] Lustre: oak-OST015f: Connection restored to 080fbf36-50e2-e037-d579-97f7ea4dd6a6 (at 10.210.12.51@tcp1)
[14381725.152242] Lustre: Skipped 831 previous similar messages
[14382324.058857] Lustre: oak-OST0159: Connection restored to ddd6e198-f36f-14d9-41a2-19f9ebbc987c (at 10.51.6.2@o2ib3)
[14382324.069345] Lustre: Skipped 1202 previous similar messages
[14382924.407904] Lustre: oak-OST0157: Connection restored to f71d1062-f20c-92ad-d3d7-66389b8ec719 (at 10.51.12.13@o2ib3)
[14382924.418571] Lustre: Skipped 1024 previous similar messages
[14383079.612055] Lustre: oak-OST0115: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14383079.621444] LustreError: 253937:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff91ef256ae050 x1714989207753600/t0(0) o3->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:319/0 lens 488/440 e 0 to 0 dl 1645610399 ref 1 fl Interpret:/0/0 rc 0/0
[14383079.621445] LustreError: 253937:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 6 previous similar messages
[14383079.621511] Lustre: oak-OST0133: Bulk IO read error with 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1), client will retry: rc -110
[14383079.659152] Lustre: oak-OST015f: Bulk IO write error with 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1), client will retry: rc = -110
[14383079.659153] Lustre: Skipped 8 previous similar messages
[14383079.689045] Lustre: Skipped 22 previous similar messages
[14383080.332680] LustreError: 160955:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91b9730ff850 x1714989207742528/t0(0) o4->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:319/0 lens 488/448 e 0 to 0 dl 1645610399 ref 1 fl Interpret:/0/0 rc 0/0
[14383080.357165] LustreError: 160955:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 6 previous similar messages
[14383080.367267] Lustre: oak-OST015b: Bulk IO write error with 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1), client will retry: rc = -110
[14383080.380714] Lustre: Skipped 2 previous similar messages
[14383080.393607] Lustre: oak-OST0133: Bulk IO read error with 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1), client will retry: rc -110
[14383080.406778] Lustre: Skipped 4 previous similar messages
[14383084.688707] LustreError: 26436:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91ee2d5d4c00
[14383084.699764] LustreError: 26433:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91ee2d5d4c00
[14383084.710789] LustreError: 26436:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91ee2d5d4c00
[14383084.721808] LustreError: 26435:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91ee2d5d4c00
[14383084.733118] Lustre: oak-OST015b: Bulk IO write error with 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1), client will retry: rc = -110
[14383085.480172] LustreError: 160950:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91b9730fd050 x1714989207749312/t0(0) o4->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:319/0 lens 488/448 e 0 to 0 dl 1645610399 ref 1 fl Interpret:/0/0 rc 0/0
[14383085.504595] LustreError: 160950:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 4 previous similar messages
[14383132.114987] LustreError: 228677:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(1048576)  req@ffff91b9730fd850 x1714989207707776/t0(0) o3->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:318/0 lens 488/440 e 0 to 0 dl 1645610398 ref 1 fl Interpret:/0/0 rc 0/0
[14383132.115075] Lustre: oak-OST014d: Bulk IO read error with 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1), client will retry: rc -110
[14383132.153178] LustreError: 228677:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 2 previous similar messages
[14383246.942730] Lustre: oak-OST013b: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14383246.953132] Lustre: Skipped 1 previous similar message
[14383425.121253] Lustre: oak-OST014d: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14383425.131682] Lustre: Skipped 5 previous similar messages
[14383523.886252] Lustre: oak-OST015f: Connection restored to 097ece3c-9f18-7c64-d1a4-979765e8510b (at 10.0.3.24@o2ib5)
[14383523.896754] Lustre: Skipped 1251 previous similar messages
[14384034.837298] Lustre: oak-OST0111: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14384034.844469] LustreError: 137-5: oak-OST013e_UUID: not available for connect from 10.210.12.23@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14384034.865344] Lustre: Skipped 13 previous similar messages
[14384035.074943] LustreError: 162692:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff9175f8c3e850 x1714989233038656/t0(0) o3->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:521/0 lens 504/440 e 0 to 0 dl 1645611356 ref 1 fl Interpret:/0/0 rc 0/0
[14384035.099309] Lustre: oak-OST0135: Bulk IO read error with 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1), client will retry: rc -110
[14384035.112482] Lustre: Skipped 2 previous similar messages
[14384035.591684] LustreError: 127358:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff91be9f58d850 x1714989233009472/t0(0) o3->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:520/0 lens 488/440 e 0 to 0 dl 1645611355 ref 1 fl Interpret:/0/0 rc 0/0
[14384035.616181] Lustre: oak-OST013f: Bulk IO read error with 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1), client will retry: rc -110
[14384040.847910] LustreError: 162685:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9154e7874850 x1714989233032192/t0(0) o4->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:520/0 lens 488/448 e 0 to 0 dl 1645611355 ref 1 fl Interpret:/0/0 rc 0/0
[14384040.872570] Lustre: oak-OST013d: Bulk IO write error with 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1), client will retry: rc = -110
[14384040.886017] Lustre: Skipped 1 previous similar message
[14384054.979560] LustreError: 228829:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91bd5c4ed850 x1714989233246720/t0(0) o4->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:528/0 lens 488/448 e 0 to 0 dl 1645611363 ref 1 fl Interpret:/0/0 rc 0/0
[14384055.446792] Lustre: oak-OST015f: Bulk IO write error with 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1), client will retry: rc = -110
[14384055.591589] LustreError: 26433:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91e8908d9c00
[14384055.602652] LustreError: 26436:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91e8908d9c00
[14384055.613670] LustreError: 26435:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91e8908d9c00
[14384055.624684] LustreError: 26433:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91e8908d9c00
[14384055.635693] LustreError: 26434:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff913b7fc82c00
[14384055.646705] LustreError: 26436:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff913b7fc82c00
[14384055.657723] LustreError: 26435:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff913b7fc82c00
[14384055.668742] LustreError: 26433:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff913b7fc82c00
[14384086.911938] LNetError: 26416:0:(o2iblnd_cb.c:3383:kiblnd_check_txs_locked()) Timed out tx: tx_queue, 2 seconds
[14384086.922165] LNetError: 26416:0:(o2iblnd_cb.c:3458:kiblnd_check_conns()) Timed out RDMA with 10.0.2.206@o2ib5 (0): c: 0, oc: 1, rc: 2
[14384086.934643] LustreError: 26416:0:(events.c:455:server_bulk_callback()) event type 5, status -103, desc ffff91d83a596000
[14384086.945805] LustreError: 26416:0:(events.c:455:server_bulk_callback()) event type 5, status -103, desc ffff91a289958000
[14384086.956888] LustreError: 26416:0:(events.c:455:server_bulk_callback()) event type 5, status -103, desc ffff91a289958000
[14384086.967917] LustreError: 26416:0:(events.c:455:server_bulk_callback()) event type 5, status -103, desc ffff915b59eddc00
[14384086.978948] LustreError: 26416:0:(events.c:455:server_bulk_callback()) event type 5, status -103, desc ffff91ccd75fd000
[14384086.989989] LustreError: 26416:0:(events.c:455:server_bulk_callback()) event type 5, status -103, desc ffff91ccd75fd000
[14384089.030189] LustreError: 21599:0:(ldlm_lib.c:3362:target_bulk_io()) @@@ network error on bulk READ  req@ffff918f7807a050 x1714939659429952/t0(0) o3->f0e330e4-5665-2d15-58f0-48fc23564e27@10.210.12.21@tcp1:526/0 lens 488/440 e 0 to 0 dl 1645611361 ref 1 fl Interpret:/0/0 rc 0/0
[14384089.030823] LustreError: 160944:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff91bf6dc53050 x1714989233009472/t0(0) o3->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:525/0 lens 488/440 e 0 to 0 dl 1645611360 ref 1 fl Interpret:/2/0 rc 0/0
[14384089.030832] LustreError: 243453:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91bf6dc50850 x1715427936728896/t0(0) o4->94d559ad-9ea8-2fc9-41c4-60f721062c9a@10.210.12.52@tcp1:526/0 lens 488/448 e 0 to 0 dl 1645611361 ref 1 fl Interpret:/0/0 rc 0/0
[14384089.030835] Lustre: oak-OST0135: Bulk IO read error with 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1), client will retry: rc -110
[14384089.031010] Lustre: oak-OST0155: Bulk IO write error with 94d559ad-9ea8-2fc9-41c4-60f721062c9a (at 10.210.12.52@tcp1), client will retry: rc = -110
[14384089.031011] Lustre: Skipped 5 previous similar messages
[14384089.138253] LustreError: 21599:0:(ldlm_lib.c:3362:target_bulk_io()) Skipped 2 previous similar messages
[14384098.724248] LustreError: 253935:0:(ldlm_lib.c:3362:target_bulk_io()) @@@ network error on bulk READ  req@ffff9195c73ad850 x1714939659430144/t0(0) o3->f0e330e4-5665-2d15-58f0-48fc23564e27@10.210.12.21@tcp1:526/0 lens 488/440 e 0 to 0 dl 1645611361 ref 1 fl Interpret:/0/0 rc 0/0
[14384098.724433] LustreError: 162674:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff9154e7874050 x1714939659430080/t0(0) o3->f0e330e4-5665-2d15-58f0-48fc23564e27@10.210.12.21@tcp1:526/0 lens 488/440 e 0 to 0 dl 1645611361 ref 1 fl Interpret:/0/0 rc 0/0
[14384098.724435] LustreError: 162674:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 1 previous similar message
[14384098.724638] Lustre: oak-OST0127: Bulk IO read error with f0e330e4-5665-2d15-58f0-48fc23564e27 (at 10.210.12.21@tcp1), client will retry: rc -110
[14384098.724638] Lustre: Skipped 4 previous similar messages
[14384098.732647] LustreError: 26437:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff917bc4794c00
[14384098.732651] LustreError: 26441:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff917bc4794c00
[14384098.732832] Lustre: oak-OST0141: Bulk IO write error with 94d559ad-9ea8-2fc9-41c4-60f721062c9a (at 10.210.12.52@tcp1), client will retry: rc = -110
[14384098.732833] Lustre: Skipped 1 previous similar message
[14384098.732979] LustreError: 26439:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff918a86df2c00
[14384098.732981] LustreError: 26441:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff9156e484dc00
[14384098.732984] LustreError: 26440:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff918a86df2c00
[14384098.732985] LustreError: 26441:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91b8f324bc00
[14384098.732988] LustreError: 26439:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff9156e484dc00
[14384098.732992] LustreError: 26440:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff915b59edfc00
[14384098.732994] LustreError: 26441:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff915b59edfc00
[14384098.733002] LustreError: 26437:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff915b59ed8400
[14384098.733004] LustreError: 26440:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff915b59ed8400
[14384098.733006] LustreError: 26437:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff915b59ed8400
[14384098.733008] LustreError: 26440:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff915b59ed8400
[14384098.734214] LustreError: 26439:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91ccd75ff800
[14384098.734218] LustreError: 26439:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91ccd75ff800
[14384098.734220] LustreError: 26439:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91ccd75ff800
[14384098.734223] LustreError: 26439:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91ccd75ff800
[14384098.734350] LustreError: 26440:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91d83a597000
[14384098.734353] LustreError: 26441:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91d83a597000
[14384098.734357] LustreError: 26440:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff9157794cf400
[14384098.734360] LustreError: 26441:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff913de0941000
[14384098.734362] LustreError: 26440:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff913de0941000
[14384098.734364] LustreError: 26441:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91a18e68c800
[14384098.734367] LustreError: 26441:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91a18e68c800
[14384098.734369] LustreError: 26440:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff9156e484e800
[14384098.734372] LustreError: 26441:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff9156e484e800
[14384098.736750] LustreError: 26437:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91ea3fd40000
[14384098.736760] LustreError: 26441:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff916798be8800
[14384098.736764] LustreError: 26441:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff916798be8800
[14384098.736765] LustreError: 26440:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91b5ba118c00
[14384098.736769] LustreError: 26441:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91b5ba118c00
[14384098.736772] LustreError: 26440:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91e968318400
[14384098.736772] LustreError: 26437:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff9156e484a000
[14384098.736774] LustreError: 26439:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91e968318400
[14384098.736774] LustreError: 26441:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff9156e484a000
[14384098.736777] LustreError: 26440:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91e96831ac00
[14384098.736779] LustreError: 26437:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91e96831ac00
[14384098.736781] LustreError: 26440:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff917bc4790400
[14384098.736784] LustreError: 26437:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff917bc4790400
[14384098.736786] LustreError: 26440:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff917bc4790400
[14384098.736788] LustreError: 26441:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff917bc4790400
[14384098.736789] LustreError: 26437:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff919abc025000
[14384098.736790] LustreError: 26440:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff919abc025000
[14384098.736792] LustreError: 26437:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff919abc025000
[14384098.736794] LustreError: 26441:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff919abc025000
[14384098.737126] LustreError: 26440:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff919e6b37b800
[14384098.737129] LustreError: 26437:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91baae23f000
[14384098.737132] LustreError: 26440:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91baae23f000
[14384098.737134] LustreError: 26437:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff916798beb800
[14384098.737136] LustreError: 26439:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff916798beb800
[14384098.737138] LustreError: 26440:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91ccd75f8c00
[14384098.737140] LustreError: 26437:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91ccd75f8c00
[14384098.737140] LustreError: 26439:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff9156e484d000
[14384098.737142] LustreError: 26440:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff9156e484d000
[14384098.737144] LustreError: 26439:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91baae23ac00
[14384098.737146] LustreError: 26437:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91baae23ac00
[14384098.737147] LustreError: 26439:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff918d5b7cbc00
[14384098.737150] LustreError: 26440:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff918d5b7cbc00
[14384098.737150] LustreError: 26439:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91b61375d800
[14384098.737153] LustreError: 26437:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91b61375d800
[14384098.737155] LustreError: 26439:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff917bc4795000
[14384098.737157] LustreError: 26440:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff917bc4795000
[14384098.737159] LustreError: 26439:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91d5c0f06400
[14384098.737161] LustreError: 26437:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91d5c0f06400
[14384098.737163] LustreError: 26439:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff9156e484c400
[14384098.737166] LustreError: 26440:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff9156e484c400
[14384098.737167] LustreError: 26437:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91baae23b000
[14384098.737168] LustreError: 26439:0:(events.c:455:server_bulk_callback()) event type 5, status -125, desc ffff91baae23b000
[14384099.572169] LustreError: 253935:0:(ldlm_lib.c:3362:target_bulk_io()) Skipped 31 previous similar messages
[14384122.736658] Lustre: oak-OST014d: Connection restored to f6aa34b8-0e07-56c4-2406-0e49a3e4bebd (at 10.50.9.68@o2ib2)
[14384122.747254] Lustre: Skipped 1050 previous similar messages
[14384134.809170] LustreError: 137-5: oak-OST0112_UUID: not available for connect from 10.210.12.23@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14384201.607698] Lustre: oak-OST0121: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14384201.618106] Lustre: Skipped 3 previous similar messages
[14384468.177395] Lustre: oak-OST0135: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14384468.187804] Lustre: Skipped 13 previous similar messages
[14384721.809228] Lustre: oak-OST0143: Connection restored to cd354002-287c-320d-4786-ef84b940faf7 (at 10.50.9.50@o2ib2)
[14384721.819836] Lustre: Skipped 1003 previous similar messages
[14385320.613074] Lustre: oak-OST0147: Connection restored to  (at 10.50.16.2@o2ib2)
[14385320.620557] Lustre: Skipped 930 previous similar messages
[14385919.180193] Lustre: oak-OST0149: Connection restored to  (at 10.51.12.23@o2ib3)
[14385919.187790] Lustre: Skipped 1195 previous similar messages
[14386519.231403] Lustre: oak-OST015b: Connection restored to 6e78836d-0375-5f95-c654-33eca074cabb (at 10.51.16.23@o2ib3)
[14386519.242085] Lustre: Skipped 1233 previous similar messages
[14387118.914500] Lustre: oak-OST0151: Connection restored to 6e78836d-0375-5f95-c654-33eca074cabb (at 10.51.16.23@o2ib3)
[14387118.925182] Lustre: Skipped 1466 previous similar messages
[14387718.944677] Lustre: oak-OST0125: Connection restored to 9464f0c7-3309-c3fa-0301-59c3a84ae794 (at 10.51.14.20@o2ib3)
[14387718.955342] Lustre: Skipped 1245 previous similar messages
[14388318.389611] Lustre: oak-OST0153: Connection restored to b06d09e3-c7f9-ed9d-5de1-180a8f296088 (at 10.0.3.5@o2ib5)
[14388318.400016] Lustre: Skipped 1162 previous similar messages
[14388876.989794] md: md27: data-check done.
[14388917.853486] Lustre: oak-OST0119: Connection restored to  (at 10.0.2.3@o2ib5)
[14388917.860796] Lustre: Skipped 1333 previous similar messages
[14389516.600169] Lustre: oak-OST0137: Connection restored to  (at 10.51.2.6@o2ib3)
[14389516.607590] Lustre: Skipped 1373 previous similar messages
[14389988.984168] Lustre: oak-OST0111: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14389988.994607] Lustre: Skipped 39 previous similar messages
[14389989.419336] LustreError: 253937:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9196f5269850 x1714989360514176/t0(0) o4->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:438/0 lens 488/448 e 0 to 0 dl 1645617313 ref 1 fl Interpret:/0/0 rc 0/0
[14389989.443810] LustreError: 253937:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 6 previous similar messages
[14389989.453843] Lustre: oak-OST015f: Bulk IO write error with 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1), client will retry: rc = -110
[14389989.467298] Lustre: Skipped 8 previous similar messages
[14389989.777476] Lustre: oak-OST0135: Bulk IO read error with 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1), client will retry: rc -110
[14389989.790649] Lustre: Skipped 27 previous similar messages
[14390116.542226] Lustre: oak-OST0113: Connection restored to eca846f2-e0f8-cd26-6673-27d0794785c5 (at 10.51.4.42@o2ib3)
[14390116.552807] Lustre: Skipped 1339 previous similar messages
[14390716.977071] Lustre: oak-OST0157: Connection restored to 86eee6bc-6017-fb1a-3873-1027e48995f7 (at 10.50.12.3@o2ib2)
[14390716.987667] Lustre: Skipped 1206 previous similar messages
[14391316.669402] Lustre: oak-OST013f: Connection restored to 8a542161-c6da-012a-5f15-3267427cffa2 (at 10.51.5.58@o2ib3)
[14391316.679987] Lustre: Skipped 913 previous similar messages
[14391916.947739] Lustre: oak-OST0151: Connection restored to 4b2143bf-d529-0088-2e06-e914b3416be8 (at 10.50.17.34@o2ib2)
[14391916.958475] Lustre: Skipped 1096 previous similar messages
[14392021.004438] Lustre: oak-OST0157: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14392021.560044] LustreError: 243452:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91cfd1b89050 x1714989413955648/t0(0) o4->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:218/0 lens 504/448 e 0 to 0 dl 1645619358 ref 1 fl Interpret:/0/0 rc 0/0
[14392021.561351] Lustre: oak-OST0157: Bulk IO write error with 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1), client will retry: rc = -110
[14392021.598051] LustreError: 243452:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 2 previous similar messages
[14392515.872465] Lustre: oak-OST014d: Connection restored to 3fe3a0cf-c788-1e16-b3e8-485e676d54f4 (at 10.51.7.9@o2ib3)
[14392515.883023] Lustre: Skipped 1009 previous similar messages
[14392529.885438] Lustre: oak-OST0111: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14392529.895848] Lustre: Skipped 37 previous similar messages
[14392530.648709] LustreError: 162709:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff9163303b8050 x1714989423568384/t0(0) o3->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:675/0 lens 488/440 e 0 to 0 dl 1645619815 ref 1 fl Interpret:/0/0 rc 0/0
[14392530.673180] Lustre: oak-OST013b: Bulk IO read error with 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1), client will retry: rc -110
[14393114.800697] Lustre: oak-OST0151: Connection restored to 5de3621d-407c-a4ab-5438-35c12a4afb01 (at 10.50.2.22@o2ib2)
[14393114.811273] Lustre: Skipped 630 previous similar messages
[14393714.468945] Lustre: oak-OST0111: Connection restored to edd06616-023a-51fb-d0e1-f960c04746e6 (at 10.0.3.32@o2ib5)
[14393714.479449] Lustre: Skipped 660 previous similar messages
[14394052.643889] Lustre: oak-OST012d: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14394052.654300] Lustre: Skipped 38 previous similar messages
[14394316.965741] Lustre: oak-OST0111: Connection restored to edd06616-023a-51fb-d0e1-f960c04746e6 (at 10.0.3.32@o2ib5)
[14394316.976230] Lustre: Skipped 690 previous similar messages
[14394915.935715] Lustre: oak-OST014f: Connection restored to ddd6e198-f36f-14d9-41a2-19f9ebbc987c (at 10.51.6.2@o2ib3)
[14394915.946250] Lustre: Skipped 1094 previous similar messages
[14395514.623429] Lustre: oak-OST0115: Connection restored to 46bd8889-1b94-f38a-f202-7268f2e23290 (at 10.50.13.10@o2ib2)
[14395514.634100] Lustre: Skipped 957 previous similar messages
[14396113.215420] Lustre: oak-OST0111: Connection restored to ebca1434-df9c-4ff4-e7ff-f002e459523f (at 10.50.6.50@o2ib2)
[14396113.225999] Lustre: Skipped 962 previous similar messages
[14396712.286147] Lustre: oak-OST014f: Connection restored to 9eb8189a-3985-f41c-7d7b-51af46681d5f (at 10.0.3.29@o2ib5)
[14396712.296639] Lustre: Skipped 1139 previous similar messages
[14397312.081050] Lustre: oak-OST0115: Connection restored to ad12f851-216b-c412-f51f-518aba8ea609 (at 10.210.9.195@tcp1)
[14397312.091730] Lustre: Skipped 1023 previous similar messages
[14397430.768680] Lustre: oak-OST0111: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14397430.779087] Lustre: Skipped 38 previous similar messages
[14397431.394155] LustreError: 243457:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff919dad27c850 x1714989614243840/t0(0) o3->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:348/0 lens 504/440 e 0 to 0 dl 1645624773 ref 1 fl Interpret:/0/0 rc 0/0
[14397431.418737] Lustre: oak-OST0121: Bulk IO read error with 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1), client will retry: rc -110
[14397910.755925] Lustre: oak-OST0123: Connection restored to c8c804bb-e728-6f8c-527d-295ebfa95786 (at 10.50.4.35@o2ib2)
[14397910.766506] Lustre: Skipped 1104 previous similar messages
[14398509.382301] Lustre: oak-OST0113: Connection restored to ad12f851-216b-c412-f51f-518aba8ea609 (at 10.210.9.195@tcp1)
[14398509.392977] Lustre: Skipped 858 previous similar messages
[14399108.239297] Lustre: oak-OST015f: Connection restored to b4873b44-db69-85c9-576f-945bfd0ad384 (at 10.0.3.26@o2ib5)
[14399108.250059] Lustre: Skipped 1055 previous similar messages
[14399149.046398] LustreError: 228423:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(8192)  req@ffff9135f538d850 x1714989693349504/t0(0) o3->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:513/0 lens 504/440 e 0 to 0 dl 1645626448 ref 1 fl Interpret:/0/0 rc 0/0
[14399149.071180] LustreError: 228423:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 3 previous similar messages
[14399149.080899] Lustre: oak-OST0135: Bulk IO read error with 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1), client will retry: rc -110
[14399258.080024] Lustre: oak-OST012f: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14399258.090449] Lustre: Skipped 1 previous similar message
[14399258.694278] Lustre: oak-OST014f: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14399258.704722] Lustre: Skipped 1 previous similar message
[14399260.075749] Lustre: oak-OST0143: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14399260.086158] Lustre: Skipped 1 previous similar message
[14399269.683167] Lustre: oak-OST015d: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14399269.693604] Lustre: Skipped 5 previous similar messages
[14399289.411970] Lustre: oak-OST013b: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14399289.422413] Lustre: Skipped 17 previous similar messages
[14399656.042407] Lustre: oak-OST0137: haven't heard from client 5a4881be-a0cb-e632-509b-2e15c534b21a (at 10.210.12.9@tcp1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91512cf23000, cur 1645626922 expire 1645626772 last 1645626695
[14399656.064322] Lustre: Skipped 1 previous similar message
[14399660.035256] Lustre: oak-OST0155: haven't heard from client 5a4881be-a0cb-e632-509b-2e15c534b21a (at 10.210.12.9@tcp1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff917bc4793c00, cur 1645626926 expire 1645626776 last 1645626699
[14399660.057185] Lustre: Skipped 23 previous similar messages
[14399661.033979] Lustre: oak-OST0145: haven't heard from client 5a4881be-a0cb-e632-509b-2e15c534b21a (at 10.210.12.9@tcp1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91ac9a442000, cur 1645626927 expire 1645626777 last 1645626700
[14399661.055946] Lustre: Skipped 3 previous similar messages
[14399707.048202] Lustre: oak-OST0159: Connection restored to a9ab975c-a0d0-5f8a-8d1c-fbe50b233482 (at 10.210.12.71@tcp1)
[14399707.058893] Lustre: Skipped 1258 previous similar messages
[14399990.602077] Lustre: oak-OST0115: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14399990.612487] Lustre: Skipped 1 previous similar message
[14400305.689721] Lustre: oak-OST014b: Connection restored to b33708f0-39c1-6ae9-a6b0-7e626787b698 (at 10.50.10.29@o2ib2)
[14400305.700406] Lustre: Skipped 894 previous similar messages
[14400704.624959] LustreError: 243456:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(8192)  req@ffff919a920a6850 x1714989756056448/t0(0) o3->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:572/0 lens 504/440 e 0 to 0 dl 1645628017 ref 1 fl Interpret:/0/0 rc 0/0
[14400704.649742] Lustre: oak-OST0135: Bulk IO read error with 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1), client will retry: rc -110
[14400742.182407] LustreError: 137-5: oak-OST0130_UUID: not available for connect from 10.210.12.23@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14400824.240625] Lustre: oak-OST0131: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14400833.400198] Lustre: oak-OST0125: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14400833.410618] Lustre: Skipped 1 previous similar message
[14400840.678760] Lustre: oak-OST0127: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14400840.689168] Lustre: Skipped 6 previous similar messages
[14400844.643252] Lustre: oak-OST0133: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14400844.653676] Lustre: Skipped 10 previous similar messages
[14400904.246001] Lustre: oak-OST0131: Connection restored to 97a324d4-c834-549a-4b22-6ab961bb5783 (at 10.0.3.86@o2ib5)
[14400904.256504] Lustre: Skipped 1228 previous similar messages
[14401507.649819] Lustre: oak-OST0129: Connection restored to efa3260b-85f8-753d-2bfc-fbd16b6c6f94 (at 10.50.14.5@o2ib2)
[14401507.660653] Lustre: Skipped 652 previous similar messages
[14401672.798056] Lustre: oak-OST0141: Client 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1) reconnecting
[14401672.808478] Lustre: Skipped 16 previous similar messages
[14401760.063240] Lustre: oak-OST0119: Client 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1) reconnecting
[14401760.073755] Lustre: Skipped 6 previous similar messages
[14401761.062455] Lustre: oak-OST0111: Client 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1) reconnecting
[14401761.072864] Lustre: Skipped 1 previous similar message
[14401765.066635] Lustre: oak-OST0117: Client 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1) reconnecting
[14401765.077045] Lustre: Skipped 22 previous similar messages
[14402106.999597] Lustre: oak-OST0155: Connection restored to f71d1062-f20c-92ad-d3d7-66389b8ec719 (at 10.51.12.13@o2ib3)
[14402107.010289] Lustre: Skipped 927 previous similar messages
[14402705.807306] Lustre: oak-OST012b: Connection restored to 97a324d4-c834-549a-4b22-6ab961bb5783 (at 10.0.3.86@o2ib5)
[14402705.817805] Lustre: Skipped 796 previous similar messages
[14403305.505994] Lustre: oak-OST0143: Connection restored to 1cc3b6e6-945c-3742-e968-c8b047c536df (at 10.51.13.16@o2ib3)
[14403305.516679] Lustre: Skipped 1181 previous similar messages
[14403907.966784] Lustre: oak-OST0151: Connection restored to 1e1f187a-28cb-d390-d52c-e3db41797544 (at 10.51.15.21@o2ib3)
[14403907.977468] Lustre: Skipped 1511 previous similar messages
[14404338.680348] Lustre: oak-OST0149: Client 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1) reconnecting
[14404343.847103] Lustre: oak-OST0127: Client 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1) reconnecting
[14404350.563532] Lustre: oak-OST0133: Client 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1) reconnecting
[14404384.581625] Lustre: oak-OST0119: Client 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1) reconnecting
[14404424.609691] Lustre: oak-OST0151: Client 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1) reconnecting
[14404433.897412] Lustre: oak-OST0147: Client 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1) reconnecting
[14404433.907818] Lustre: Skipped 1 previous similar message
[14404452.158286] Lustre: oak-OST0121: Client 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1) reconnecting
[14404452.168702] Lustre: Skipped 1 previous similar message
[14404506.584942] Lustre: oak-OST014b: Connection restored to 60dde226-735c-8ef7-fadc-3531367e2bf8 (at 10.50.9.16@o2ib2)
[14404506.595521] Lustre: Skipped 1845 previous similar messages
[14405105.967932] Lustre: oak-OST0127: Connection restored to 8f53e468-63e2-04a8-79ec-4df3796b3b9d (at 10.51.2.72@o2ib3)
[14405105.978542] Lustre: Skipped 2085 previous similar messages
[14405680.757673] LustreError: 228829:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[14405704.607713] Lustre: oak-OST015f: Connection restored to 686419f6-1add-c297-337b-36a567dc474e (at 10.51.2.41@o2ib3)
[14405704.618299] Lustre: Skipped 3504 previous similar messages
[14405799.260285] LustreError: 243453:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[14406304.135939] Lustre: oak-OST0125: Connection restored to 46bd8889-1b94-f38a-f202-7268f2e23290 (at 10.50.13.10@o2ib2)
[14406304.146616] Lustre: Skipped 2216 previous similar messages
[14406786.337451] LustreError: 162708:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(1048576)  req@ffff91918a49e050 x1714990031430016/t0(0) o3->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:612/0 lens 488/440 e 0 to 0 dl 1645634097 ref 1 fl Interpret:/0/0 rc 0/0
[14406786.362526] Lustre: oak-OST0127: Bulk IO read error with 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1), client will retry: rc -110
[14406883.770938] Lustre: oak-OST0115: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14406883.781394] Lustre: Skipped 6 previous similar messages
[14406887.881354] Lustre: oak-OST0113: Client 97e0b978-47b3-d181-6e4a-d3ffe3c29d05 (at 10.210.12.9@tcp1) reconnecting
[14406887.891692] Lustre: Skipped 2 previous similar messages
[14406902.895604] Lustre: oak-OST011d: Connection restored to  (at 10.51.4.30@o2ib3)
[14406902.903110] Lustre: Skipped 1794 previous similar messages
[14407503.031772] Lustre: oak-OST014f: Connection restored to  (at 10.50.7.2@o2ib2)
[14407503.039164] Lustre: Skipped 2229 previous similar messages
[14408101.630601] Lustre: oak-OST011b: Connection restored to ef8da979-3077-9708-0c8c-a646245f23fe (at 10.50.13.15@o2ib2)
[14408101.641284] Lustre: Skipped 2419 previous similar messages
[14408700.615708] Lustre: oak-OST0155: Connection restored to 64dca258-5640-cfbf-bae1-94657ac7def0 (at 10.50.5.49@o2ib2)
[14408700.626288] Lustre: Skipped 2819 previous similar messages
[14409300.290817] Lustre: oak-OST014f: Connection restored to 3362c77a-2096-c98e-90ba-a15ed550df3d (at 10.50.2.31@o2ib2)
[14409300.301402] Lustre: Skipped 1297 previous similar messages
[14409898.872430] Lustre: oak-OST015f: Connection restored to d68efeb9-ff66-610a-8f04-303d4aa2ac0a (at 10.51.6.21@o2ib3)
[14409898.883751] Lustre: Skipped 2899 previous similar messages
[14410497.648909] Lustre: oak-OST015f: Connection restored to  (at 10.50.13.11@o2ib2)
[14410497.656466] Lustre: Skipped 1488 previous similar messages
[14411096.291226] Lustre: oak-OST015f: Connection restored to dfc64fb4-351a-e881-0ef3-347f2dd85bbc (at 10.51.5.70@o2ib3)
[14411096.301816] Lustre: Skipped 1915 previous similar messages
[14411695.255336] Lustre: oak-OST015d: Connection restored to b80be752-f851-25c6-d1f2-d3fd377310cf (at 10.210.12.46@tcp1)
[14411695.266006] Lustre: Skipped 1371 previous similar messages
[14412294.064377] Lustre: oak-OST0153: Connection restored to eed6b76e-529d-c2fc-2ead-90ca747a7a35 (at 10.50.8.22@o2ib2)
[14412294.074978] Lustre: Skipped 1394 previous similar messages
[14412892.638294] Lustre: oak-OST0123: Connection restored to 28235747-a8e9-41e8-01b4-cb2c625ecf44 (at 10.50.6.61@o2ib2)
[14412892.648961] Lustre: Skipped 1513 previous similar messages
[14413491.697056] Lustre: oak-OST0155: Connection restored to 1cc3b6e6-945c-3742-e968-c8b047c536df (at 10.51.13.16@o2ib3)
[14413491.707728] Lustre: Skipped 1299 previous similar messages
[14414090.695312] Lustre: oak-OST011f: Connection restored to 5c07de10-ad61-1c19-1e52-cabbbc709d3c (at 10.210.12.10@tcp1)
[14414090.705983] Lustre: Skipped 2250 previous similar messages
[14414172.571689] Lustre: oak-OST0149: Client fd2c4c0f-a4cd-0a60-cb33-5e9a47204c46 (at 10.210.12.19@tcp1) reconnecting
[14414172.582136] Lustre: Skipped 35 previous similar messages
[14414172.813556] LustreError: 162708:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff913b5b329050 x1714964408017152/t0(0) o4->fd2c4c0f-a4cd-0a60-cb33-5e9a47204c46@10.210.12.19@tcp1:532/0 lens 488/448 e 0 to 0 dl 1645641567 ref 1 fl Interpret:/0/0 rc 0/0
[14414172.838339] Lustre: oak-OST0149: Bulk IO write error with fd2c4c0f-a4cd-0a60-cb33-5e9a47204c46 (at 10.210.12.19@tcp1), client will retry: rc = -110
[14414172.851781] Lustre: Skipped 1 previous similar message
[14414173.521829] LustreError: 21594:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff917ca1388050 x1714964408016896/t0(0) o4->fd2c4c0f-a4cd-0a60-cb33-5e9a47204c46@10.210.12.19@tcp1:531/0 lens 488/448 e 0 to 0 dl 1645641566 ref 1 fl Interpret:/0/0 rc 0/0
[14414173.523040] Lustre: oak-OST015f: Bulk IO write error with e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1), client will retry: rc = -110
[14414173.559711] LustreError: 21594:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[14414173.753074] Lustre: oak-OST015f: Client e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1) reconnecting
[14414173.763476] Lustre: Skipped 1 previous similar message
[14414231.983774] LustreError: 229135:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(1048576)  req@ffff91510ada2850 x1714964408013504/t0(0) o3->fd2c4c0f-a4cd-0a60-cb33-5e9a47204c46@10.210.12.19@tcp1:531/0 lens 488/440 e 0 to 0 dl 1645641566 ref 1 fl Interpret:/0/0 rc 0/0
[14414231.983837] Lustre: oak-OST0131: Bulk IO read error with fd2c4c0f-a4cd-0a60-cb33-5e9a47204c46 (at 10.210.12.19@tcp1), client will retry: rc -110
[14414231.983989] LustreError: 162691:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff917a74d0a050 x1715245805713216/t0(0) o4->e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda@10.210.12.79@tcp1:533/0 lens 488/448 e 0 to 0 dl 1645641568 ref 1 fl Interpret:/0/0 rc 0/0
[14414231.983990] LustreError: 162691:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[14414231.984212] Lustre: oak-OST0153: Bulk IO write error with e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1), client will retry: rc = -110
[14414231.984212] Lustre: Skipped 4 previous similar messages
[14414232.076764] LustreError: 229135:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 5 previous similar messages
[14414251.583346] LustreError: 137-5: oak-OST0126_UUID: not available for connect from 10.210.12.19@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14414252.531446] LustreError: 137-5: oak-OST0156_UUID: not available for connect from 10.210.12.36@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14414252.626268] Lustre: oak-OST0149: Client fd2c4c0f-a4cd-0a60-cb33-5e9a47204c46 (at 10.210.12.19@tcp1) reconnecting
[14414339.373542] Lustre: oak-OST0121: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14414339.384025] Lustre: Skipped 2 previous similar messages
[14414347.405563] Lustre: oak-OST0111: Client 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1) reconnecting
[14414347.415996] Lustre: Skipped 48 previous similar messages
[14414365.653116] Lustre: oak-OST0157: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14414365.663677] Lustre: Skipped 61 previous similar messages
[14414365.944189] LustreError: 162687:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff919199870850 x1714990407476352/t0(0) o4->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:729/0 lens 488/448 e 0 to 0 dl 1645641764 ref 1 fl Interpret:/0/0 rc 0/0
[14414365.968604] LustreError: 162687:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 3 previous similar messages
[14414365.978465] Lustre: oak-OST0157: Bulk IO write error with 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1), client will retry: rc = -110
[14414365.991902] Lustre: Skipped 4 previous similar messages
[14414401.019690] Lustre: oak-OST012d: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14414401.030189] Lustre: Skipped 11 previous similar messages
[14414591.253935] LustreError: 160925:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff9196aa3d4850 x1724771066649728/t0(0) o4->479db7cf-e752-7ee3-6b36-d1f694b751cf@10.210.12.47@tcp1:134/0 lens 488/448 e 0 to 0 dl 1645641924 ref 1 fl Interpret:/0/0 rc 0/0
[14414591.254185] Lustre: oak-OST0159: Bulk IO write error with 479db7cf-e752-7ee3-6b36-d1f694b751cf (at 10.210.12.47@tcp1), client will retry: rc = -110
[14414591.293188] LustreError: 160925:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 5 previous similar messages
[14414609.118505] LustreError: 137-5: oak-OST0146_UUID: not available for connect from 10.210.12.47@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14414609.141313] Lustre: oak-OST0159: Client 479db7cf-e752-7ee3-6b36-d1f694b751cf (at 10.210.12.47@tcp1) reconnecting
[14414689.338283] Lustre: oak-OST0153: Connection restored to c01cc11f-1e0c-378c-4781-ae7e62a23ee0 (at 10.51.15.16@o2ib3)
[14414689.348950] Lustre: Skipped 1377 previous similar messages
[14414777.289274] Lustre: oak-OST0157: haven't heard from client 1b327e00-c8a6-87c6-5b99-3c817bee2e3c (at 10.50.6.29@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff917e093b4c00, cur 1645642080 expire 1645641930 last 1645641853
[14414777.311191] Lustre: Skipped 1 previous similar message
[14415287.928945] Lustre: oak-OST014d: Connection restored to 38fe6cc7-46f5-426d-3e80-9104164c5e7e (at 10.50.7.57@o2ib2)
[14415287.939553] Lustre: Skipped 1487 previous similar messages
[14415681.344982] Lustre: oak-OST015d: Client 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1) reconnecting
[14415681.355431] Lustre: Skipped 37 previous similar messages
[14415681.578504] LustreError: 160910:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91ef238d8850 x1715538651742208/t0(0) o4->5a5caa86-68df-f806-7e6f-05440e95a4ce@10.210.12.43@tcp1:534/0 lens 488/448 e 0 to 0 dl 1645643079 ref 1 fl Interpret:/0/0 rc 0/0
[14415681.603170] Lustre: oak-OST015d: Bulk IO write error with 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1), client will retry: rc = -110
[14415681.616641] Lustre: Skipped 1 previous similar message
[14415683.101804] LustreError: 253951:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91dae6666850 x1715245816201664/t0(0) o4->e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda@10.210.12.79@tcp1:536/0 lens 488/448 e 0 to 0 dl 1645643081 ref 1 fl Interpret:/0/0 rc 0/0
[14415683.126232] LustreError: 253951:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 2 previous similar messages
[14415683.136230] Lustre: oak-OST0153: Bulk IO write error with e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1), client will retry: rc = -110
[14415683.149717] Lustre: Skipped 2 previous similar messages
[14415684.297897] LustreError: 127353:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91d1196d0050 x1715245816229696/t0(0) o4->e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda@10.210.12.79@tcp1:541/0 lens 488/448 e 0 to 0 dl 1645643086 ref 1 fl Interpret:/0/0 rc 0/0
[14415684.322381] LustreError: 127353:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 2 previous similar messages
[14415685.131248] Lustre: oak-OST0153: Bulk IO write error with e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda (at 10.210.12.79@tcp1), client will retry: rc = -110
[14415685.144702] Lustre: Skipped 9 previous similar messages
[14415740.919125] LustreError: 160956:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(1048576)  req@ffff91b7fd4e7850 x1724771087283648/t0(0) o3->479db7cf-e752-7ee3-6b36-d1f694b751cf@10.210.12.47@tcp1:534/0 lens 488/440 e 0 to 0 dl 1645643079 ref 1 fl Interpret:/0/0 rc 0/0
[14415740.919187] Lustre: oak-OST014d: Bulk IO read error with a52b170e-1c40-8c67-003d-ccc0fed95599 (at 10.210.12.67@tcp1), client will retry: rc -110
[14415740.919188] Lustre: Skipped 5 previous similar messages
[14415740.962814] LustreError: 160956:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 3 previous similar messages
[14415760.922545] Lustre: oak-OST015f: Client 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1) reconnecting
[14415760.925461] LustreError: 137-5: oak-OST0138_UUID: not available for connect from 10.210.12.43@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14415760.950688] Lustre: Skipped 9 previous similar messages
[14415848.812371] Lustre: oak-OST0115: Client a52b170e-1c40-8c67-003d-ccc0fed95599 (at 10.210.12.67@tcp1) reconnecting
[14415857.795036] LustreError: 229137:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff91909170a050 x1715089976606144/t0(0) o3->65e9b8a8-b461-24e0-da12-5b6939f0b07c@10.210.12.56@tcp1:716/0 lens 488/440 e 0 to 0 dl 1645643261 ref 1 fl Interpret:/0/0 rc 0/0
[14415857.819382] LustreError: 229137:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 7 previous similar messages
[14415857.829350] Lustre: oak-OST0125: Bulk IO read error with 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1), client will retry: rc -110
[14415857.842557] Lustre: Skipped 2 previous similar messages
[14415886.921613] Lustre: oak-OST0151: Connection restored to f87dbe9a-d359-04ee-95be-d98819411346 (at 10.50.10.34@o2ib2)
[14415886.932288] Lustre: Skipped 2894 previous similar messages
[14415931.528040] LustreError: 162673:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff917f7eaa2050 x1715016472982528/t0(0) o4->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:724/0 lens 488/448 e 0 to 0 dl 1645643269 ref 1 fl Interpret:/0/0 rc 0/0
[14415931.528238] Lustre: oak-OST0113: Bulk IO write error with c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1), client will retry: rc = -110
[14415931.567276] LustreError: 162673:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 5 previous similar messages
[14415949.607694] Lustre: oak-OST0149: Client fd2c4c0f-a4cd-0a60-cb33-5e9a47204c46 (at 10.210.12.19@tcp1) reconnecting
[14415949.618101] Lustre: Skipped 252 previous similar messages
[14415954.011751] LustreError: 137-5: oak-OST011e_UUID: not available for connect from 10.210.12.38@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14415954.029723] LustreError: Skipped 4 previous similar messages
[14416014.753753] LustreError: 199244:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 149s: evicting client at 10.210.12.19@tcp1  ns: filter-oak-OST0157_UUID lock: ffff91439f91ad00/0xed112d3063eab94a lrc: 3/0,0 mode: PW/PW res: [0x5940000401:0x240e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->8191) flags: 0x60000400020020 nid: 10.210.12.19@tcp1 remote: 0x51235c9d21ad2027 expref: 7 pid: 205374 timeout: 14451053 lvb_type: 0
[14416014.794761] LustreError: 226873:0:(client.c:1210:ptlrpc_import_delay_req()) @@@ IMP_CLOSED   req@ffff913cfb2f0480 x1710542325222400/t0(0) o105->oak-OST0157@10.210.12.19@tcp1:15/16 lens 360/224 e 0 to 0 dl 0 ref 1 fl Rpc:/0/ffffffff rc 0/-1
[14416056.483439] LustreError: 21616:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff914e22eef850 x1714964471359168/t0(0) o3->fd2c4c0f-a4cd-0a60-cb33-5e9a47204c46@10.210.12.19@tcp1:157/0 lens 488/440 e 0 to 0 dl 1645643457 ref 1 fl Interpret:/0/0 rc 0/0
[14416056.507741] Lustre: oak-OST0131: Bulk IO read error with fd2c4c0f-a4cd-0a60-cb33-5e9a47204c46 (at 10.210.12.19@tcp1), client will retry: rc -110
[14416093.204993] Lustre: oak-OST013d: Client c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1) reconnecting
[14416093.215404] Lustre: Skipped 172 previous similar messages
[14416486.269267] Lustre: oak-OST0117: Connection restored to 9aa7ef2f-4a2b-0e11-ec94-763eadc83857 (at 10.50.2.69@o2ib2)
[14416486.279878] Lustre: Skipped 1722 previous similar messages
[14416642.126952] LustreError: 160953:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[14416737.701238] LustreError: 160908:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[14416803.362328] Lustre: oak-OST013f: Client a52b170e-1c40-8c67-003d-ccc0fed95599 (at 10.210.12.67@tcp1) reconnecting
[14416803.372729] Lustre: Skipped 1 previous similar message
[14416803.528700] LustreError: 162669:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff917fc5378850 x1715095690383296/t0(0) o4->a52b170e-1c40-8c67-003d-ccc0fed95599@10.210.12.67@tcp1:153/0 lens 488/448 e 0 to 0 dl 1645644208 ref 1 fl Interpret:/0/0 rc 0/0
[14416803.553703] Lustre: oak-OST013f: Bulk IO write error with a52b170e-1c40-8c67-003d-ccc0fed95599 (at 10.210.12.67@tcp1), client will retry: rc = -110
[14416803.567202] Lustre: Skipped 5 previous similar messages
[14416865.661705] LustreError: 21604:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff917911fa2050 x1715538661192640/t0(0) o4->5a5caa86-68df-f806-7e6f-05440e95a4ce@10.210.12.43@tcp1:149/0 lens 488/448 e 0 to 0 dl 1645644204 ref 1 fl Interpret:/0/0 rc 0/0
[14416865.662024] Lustre: oak-OST014f: Bulk IO write error with 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1), client will retry: rc = -110
[14416865.700881] LustreError: 21604:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 9 previous similar messages
[14416883.197304] LustreError: 137-5: oak-OST0138_UUID: not available for connect from 10.210.12.43@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14416884.306897] LustreError: 137-5: oak-OST015c_UUID: not available for connect from 10.210.12.29@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14416884.325309] LustreError: Skipped 2 previous similar messages
[14416967.721426] LustreError: 160897:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[14416996.925756] LustreError: 127358:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[14417086.412842] Lustre: oak-OST0151: Connection restored to e9b2525d-3d3e-fd5d-fe20-21370390750f (at 10.210.12.50@tcp1)
[14417086.423507] Lustre: Skipped 2191 previous similar messages
[14417685.332746] Lustre: oak-OST012d: Connection restored to  (at 10.0.2.3@o2ib5)
[14417685.340046] Lustre: Skipped 1466 previous similar messages
[14417760.676007] LustreError: 160905:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[14417779.746545] LustreError: 253956:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[14417799.136755] LustreError: 199274:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[14417809.710840] LustreError: 243441:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[14417876.182614] LustreError: 248296:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[14417884.497447] LustreError: 160931:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[14418285.979602] Lustre: oak-OST0119: Connection restored to  (at 10.51.15.22@o2ib3)
[14418285.987348] Lustre: Skipped 952 previous similar messages
[14418886.316769] Lustre: oak-OST012b: Connection restored to  (at 10.50.7.2@o2ib2)
[14418886.324227] Lustre: Skipped 785 previous similar messages
[14418996.597626] LustreError: 162695:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff916188557850 x1715428090802688/t0(0) o4->94d559ad-9ea8-2fc9-41c4-60f721062c9a@10.210.12.52@tcp1:16/0 lens 488/448 e 0 to 0 dl 1645646336 ref 1 fl Interpret:/0/0 rc 0/0
[14418996.598693] Lustre: oak-OST015d: Bulk IO write error with 94d559ad-9ea8-2fc9-41c4-60f721062c9a (at 10.210.12.52@tcp1), client will retry: rc = -110
[14418996.598694] Lustre: Skipped 9 previous similar messages
[14418996.642277] LustreError: 162695:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[14419011.630895] Lustre: oak-OST015d: Client 94d559ad-9ea8-2fc9-41c4-60f721062c9a (at 10.210.12.52@tcp1) reconnecting
[14419011.641325] Lustre: Skipped 282 previous similar messages
[14419097.862019] Lustre: oak-OST0111: Client 94d559ad-9ea8-2fc9-41c4-60f721062c9a (at 10.210.12.52@tcp1) reconnecting
[14419097.872450] Lustre: Skipped 7 previous similar messages
[14419399.048848] LustreError: 160897:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[14419401.778021] LustreError: 160904:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[14419439.739094] Lustre: oak-OST0155: Client b880b230-b2d3-6e4c-a85d-fbc038732418 (at 10.210.12.44@tcp1) reconnecting
[14419439.749543] Lustre: Skipped 29 previous similar messages
[14419440.020846] LustreError: 160908:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91b5ad333850 x1714948863966208/t0(0) o4->b880b230-b2d3-6e4c-a85d-fbc038732418@10.210.12.44@tcp1:528/0 lens 488/448 e 0 to 0 dl 1645646848 ref 1 fl Interpret:/0/0 rc 0/0
[14419440.045705] Lustre: oak-OST0155: Bulk IO write error with b880b230-b2d3-6e4c-a85d-fbc038732418 (at 10.210.12.44@tcp1), client will retry: rc = -110
[14419440.059315] Lustre: Skipped 2 previous similar messages
[14419440.745081] LustreError: 127356:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91b4e2a1b050 x1714948863965696/t0(0) o4->b880b230-b2d3-6e4c-a85d-fbc038732418@10.210.12.44@tcp1:527/0 lens 488/448 e 0 to 0 dl 1645646847 ref 1 fl Interpret:/0/0 rc 0/0
[14419440.769579] LustreError: 127356:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[14419441.118352] Lustre: oak-OST0135: Bulk IO write error with 5d8bb8d9-e5c7-334f-ec8d-f10452c309b5 (at 10.210.12.66@tcp1), client will retry: rc = -110
[14419441.131797] Lustre: Skipped 4 previous similar messages
[14419442.827029] LustreError: 21624:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91cfd313f850 x1724771148718848/t0(0) o4->479db7cf-e752-7ee3-6b36-d1f694b751cf@10.210.12.47@tcp1:527/0 lens 488/448 e 0 to 0 dl 1645646847 ref 1 fl Interpret:/0/0 rc 0/0
[14419442.851413] LustreError: 21624:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 5 previous similar messages
[14419443.873659] Lustre: oak-OST015d: Bulk IO write error with 1be164da-11a7-a3a8-bd17-8ca7a5aab4e8 (at 10.210.12.69@tcp1), client will retry: rc = -110
[14419443.887097] Lustre: Skipped 3 previous similar messages
[14419444.927919] LustreError: 21622:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91dc78d92050 x1715245860746688/t0(0) o4->e44f890c-cbe4-2a8e-31d1-2c4dc7e8adda@10.210.12.79@tcp1:535/0 lens 488/448 e 0 to 0 dl 1645646855 ref 1 fl Interpret:/0/0 rc 0/0
[14419444.952784] LustreError: 21622:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 16 previous similar messages
[14419475.613532] LustreError: 248297:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[14419484.954591] Lustre: oak-OST0157: Connection restored to 1cc3b6e6-945c-3742-e968-c8b047c536df (at 10.51.13.16@o2ib3)
[14419484.965684] Lustre: Skipped 948 previous similar messages
[14419490.478355] LustreError: 244099:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[14419499.636049] LustreError: 127351:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91c5c05c6850 x1714964538154048/t0(0) o4->fd2c4c0f-a4cd-0a60-cb33-5e9a47204c46@10.210.12.19@tcp1:527/0 lens 488/448 e 0 to 0 dl 1645646847 ref 1 fl Interpret:/0/0 rc 0/0
[14419499.636055] LustreError: 243448:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 0(1048576)  req@ffff91c1feb1e050 x1724771148718592/t0(0) o3->479db7cf-e752-7ee3-6b36-d1f694b751cf@10.210.12.47@tcp1:527/0 lens 488/440 e 0 to 0 dl 1645646847 ref 1 fl Interpret:/0/0 rc 0/0
[14419499.636097] Lustre: oak-OST0131: Bulk IO read error with 479db7cf-e752-7ee3-6b36-d1f694b751cf (at 10.210.12.47@tcp1), client will retry: rc -110
[14419499.636398] Lustre: oak-OST015d: Bulk IO write error with 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1), client will retry: rc = -110
[14419499.636399] Lustre: Skipped 16 previous similar messages
[14419499.719216] LustreError: 127351:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[14419503.508574] LustreError: 212522:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91b09db3b050 x1716225993610496/t0(0) o4->6b6cd352-15e2-b5e6-2cf2-86cbf0e9b626@10.210.12.145@tcp1:591/0 lens 488/448 e 0 to 0 dl 1645646911 ref 1 fl Interpret:/0/0 rc 0/0
[14419516.210485] LustreError: 248295:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[14419520.665414] LustreError: 137-5: oak-OST0132_UUID: not available for connect from 10.210.12.69@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14419521.997598] LustreError: 137-5: oak-OST0126_UUID: not available for connect from 10.210.12.19@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14419522.015185] LustreError: Skipped 2 previous similar messages
[14419703.319955] Lustre: oak-OST0123: haven't heard from client a79a8a88-9508-da4f-f13f-77b61fd4de38 (at 10.51.15.6@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91cfe29d6c00, cur 1645647018 expire 1645646868 last 1645646791
[14419703.342014] Lustre: Skipped 8 previous similar messages
[14419707.331933] Lustre: oak-OST011d: haven't heard from client a79a8a88-9508-da4f-f13f-77b61fd4de38 (at 10.51.15.6@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91b36cd35c00, cur 1645647022 expire 1645646872 last 1645646795
[14419707.353943] Lustre: Skipped 8 previous similar messages
[14420088.442989] Lustre: oak-OST0113: Connection restored to a8316a6f-eaa1-c7fb-d6c0-c9a00a0df52e (at 10.51.16.22@o2ib3)
[14420088.453657] Lustre: Skipped 1369 previous similar messages
[14420687.827085] Lustre: oak-OST0143: Connection restored to 6ddab9a9-b0cf-55e2-2cd0-3d3d737a2aa4 (at 10.210.12.145@tcp1)
[14420687.837860] Lustre: Skipped 968 previous similar messages
[14421287.379634] Lustre: oak-OST0143: Connection restored to 51ab7e05-78ff-bf41-8734-92d0767bb7ee (at 10.50.7.21@o2ib2)
[14421287.390232] Lustre: Skipped 1647 previous similar messages
[14421886.006269] Lustre: oak-OST015d: Connection restored to fba8005f-f107-35f2-a145-5414b74ab7bf (at 10.51.12.3@o2ib3)
[14421886.016956] Lustre: Skipped 1375 previous similar messages
[14422486.056849] Lustre: oak-OST0125: Connection restored to  (at 10.51.13.15@o2ib3)
[14422486.064446] Lustre: Skipped 975 previous similar messages
[14423085.281493] Lustre: oak-OST0113: Connection restored to c10b6301-1800-4510-94bd-883f949efde1 (at 10.50.6.68@o2ib2)
[14423085.292076] Lustre: Skipped 1029 previous similar messages
[14423684.270799] Lustre: oak-OST0147: Connection restored to  (at 10.51.13.14@o2ib3)
[14423684.278669] Lustre: Skipped 953 previous similar messages
[14424283.525287] Lustre: oak-OST011f: Connection restored to 55fa4ecb-ff66-16c3-914e-01f9698a4d93 (at 10.50.12.16@o2ib2)
[14424283.536005] Lustre: Skipped 941 previous similar messages
[14424882.299586] Lustre: oak-OST0151: Connection restored to 53f7c8fb-ba13-9226-8f66-6ca61c432359 (at 10.51.12.11@o2ib3)
[14424882.310299] Lustre: Skipped 522 previous similar messages
[14425483.807905] Lustre: oak-OST0139: Connection restored to  (at 10.50.4.13@o2ib2)
[14425483.815676] Lustre: Skipped 623 previous similar messages
[14425844.697975] Lustre: oak-OST011d: Client b4fca75a-2d54-9bf2-331f-67d0c52e5ff2 (at 10.210.12.73@tcp1) reconnecting
[14425844.708441] Lustre: Skipped 278 previous similar messages
[14425844.985493] LustreError: 160902:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91c8b41c4050 x1715111328551488/t0(0) o4->b4fca75a-2d54-9bf2-331f-67d0c52e5ff2@10.210.12.73@tcp1:152/0 lens 488/448 e 0 to 0 dl 1645653267 ref 1 fl Interpret:/0/0 rc 0/0
[14425845.013036] Lustre: oak-OST011d: Bulk IO write error with b4fca75a-2d54-9bf2-331f-67d0c52e5ff2 (at 10.210.12.73@tcp1), client will retry: rc = -110
[14425845.026467] Lustre: Skipped 2 previous similar messages
[14425845.608298] LustreError: 160907:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff91b9c2d25850 x1715111328552512/t0(0) o4->b4fca75a-2d54-9bf2-331f-67d0c52e5ff2@10.210.12.73@tcp1:157/0 lens 488/448 e 0 to 0 dl 1645653272 ref 1 fl Interpret:/2/0 rc 0/0
[14425845.633037] LustreError: 160907:0:(ldlm_lib.c:3305:target_bulk_io()) Skipped 2 previous similar messages
[14426011.995779] Lustre: oak-OST0125: Client b4fca75a-2d54-9bf2-331f-67d0c52e5ff2 (at 10.210.12.73@tcp1) reconnecting
[14426012.006236] Lustre: Skipped 1 previous similar message
[14426082.911230] Lustre: oak-OST014f: Connection restored to 097ece3c-9f18-7c64-d1a4-979765e8510b (at 10.0.3.24@o2ib5)
[14426082.921759] Lustre: Skipped 1448 previous similar messages
[14426367.991983] Lustre: oak-OST012b: Client 347afad8-483e-1e01-1248-6758c9189641 (at 10.51.7.14@o2ib3) reconnecting
[14426368.002333] Lustre: Skipped 27 previous similar messages
[14426682.804218] Lustre: oak-OST0153: Connection restored to a7ecaa46-7a8e-2567-1fe5-a4e141b7b7d2 (at 10.51.1.15@o2ib3)
[14426682.814809] Lustre: Skipped 2837 previous similar messages
[14427282.376607] Lustre: oak-OST0151: Connection restored to  (at 10.51.13.15@o2ib3)
[14427282.384239] Lustre: Skipped 1237 previous similar messages
[14427596.410947] LustreError: 228677:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[14427882.921310] Lustre: oak-OST011d: Connection restored to b3bea626-05bb-685f-cccb-b314e73f06c0 (at 10.51.2.48@o2ib3)
[14427882.931895] Lustre: Skipped 810 previous similar messages
[14428482.203375] Lustre: oak-OST015f: Connection restored to 6d5f6930-7f7b-3f69-4f72-88220d8e4d1e (at 10.51.1.58@o2ib3)
[14428482.213968] Lustre: Skipped 1664 previous similar messages
[14428607.570618] Lustre: oak-OST015f: Client 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1) reconnecting
[14428607.581063] Lustre: Skipped 46 previous similar messages
[14428608.308753] LustreError: 162685:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9143feced850 x1715096257888128/t0(0) o4->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:656/0 lens 488/448 e 0 to 0 dl 1645656036 ref 1 fl Interpret:/0/0 rc 0/0
[14428608.323936] Lustre: oak-OST015f: Bulk IO write error with 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1), client will retry: rc = -110
[14428608.323937] Lustre: Skipped 5 previous similar messages
[14428608.352403] LustreError: 162685:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 6 previous similar messages
[14428609.046960] LustreError: 21618:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91622dbdf050 x1715096257945984/t0(0) o4->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:662/0 lens 488/448 e 0 to 0 dl 1645656042 ref 1 fl Interpret:/0/0 rc 0/0
[14428609.071384] LustreError: 21618:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 3 previous similar messages
[14428609.081337] Lustre: oak-OST0153: Bulk IO write error with 5d8bb8d9-e5c7-334f-ec8d-f10452c309b5 (at 10.210.12.66@tcp1), client will retry: rc = -110
[14428609.094776] Lustre: Skipped 5 previous similar messages
[14428668.875614] LustreError: 162700:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff916a7c6e3850 x1714964704777600/t0(0) o4->fd2c4c0f-a4cd-0a60-cb33-5e9a47204c46@10.210.12.19@tcp1:656/0 lens 488/448 e 0 to 0 dl 1645656036 ref 1 fl Interpret:/0/0 rc 0/0
[14428668.875919] Lustre: oak-OST0149: Bulk IO write error with fd2c4c0f-a4cd-0a60-cb33-5e9a47204c46 (at 10.210.12.19@tcp1), client will retry: rc = -110
[14428668.875920] Lustre: Skipped 1 previous similar message
[14428668.920405] LustreError: 162700:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 5 previous similar messages
[14428687.375873] LustreError: 137-5: oak-OST0126_UUID: not available for connect from 10.210.12.19@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14428687.398451] Lustre: oak-OST0149: Client fd2c4c0f-a4cd-0a60-cb33-5e9a47204c46 (at 10.210.12.19@tcp1) reconnecting
[14428687.408865] Lustre: Skipped 3 previous similar messages
[14428780.427439] Lustre: oak-OST011f: Client 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1) reconnecting
[14428780.437870] Lustre: Skipped 4 previous similar messages
[14428821.331808] Lustre: oak-OST015b: haven't heard from client fd2c4c0f-a4cd-0a60-cb33-5e9a47204c46 (at 10.210.12.19@tcp1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91d302b76400, cur 1645656158 expire 1645656008 last 1645655931
[14428840.888482] LustreError: 160940:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91cc08376050 x1715096263603072/t0(0) o4->5d8bb8d9-e5c7-334f-ec8d-f10452c309b5@10.210.12.66@tcp1:138/0 lens 488/448 e 0 to 0 dl 1645656273 ref 1 fl Interpret:/0/0 rc 0/0
[14428840.912983] LustreError: 160940:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[14428840.922862] Lustre: oak-OST0153: Bulk IO write error with 5d8bb8d9-e5c7-334f-ec8d-f10452c309b5 (at 10.210.12.66@tcp1), client will retry: rc = -110
[14428840.936314] Lustre: Skipped 5 previous similar messages
[14428917.768718] LustreError: 137-5: oak-OST015a_UUID: not available for connect from 10.210.12.66@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14428917.786307] LustreError: Skipped 1 previous similar message
[14429005.739262] Lustre: oak-OST0143: Client 5d8bb8d9-e5c7-334f-ec8d-f10452c309b5 (at 10.210.12.66@tcp1) reconnecting
[14429005.749667] Lustre: Skipped 111 previous similar messages
[14429082.053201] Lustre: oak-OST013b: Connection restored to 6f394792-8c67-d7b4-787d-58080fb7a9fd (at 10.50.1.41@o2ib2)
[14429082.063865] Lustre: Skipped 1742 previous similar messages
[14429604.379866] Lustre: oak-OST0145: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14429604.390303] Lustre: Skipped 29 previous similar messages
[14429680.609025] Lustre: oak-OST0131: Connection restored to dc6310a8-f0c0-fe0b-d354-acfd9448e8fa (at 10.51.4.51@o2ib3)
[14429680.619612] Lustre: Skipped 1780 previous similar messages
[14429926.036129] Lustre: oak-OST0135: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14429926.046548] Lustre: Skipped 9 previous similar messages
[14430279.693074] Lustre: oak-OST0153: Connection restored to 1564128a-b82c-a8c0-90e4-b65ea238812e (at 10.210.12.23@tcp1)
[14430279.703756] Lustre: Skipped 1134 previous similar messages
[14430882.291053] Lustre: oak-OST011b: Connection restored to  (at 10.0.2.3@o2ib5)
[14430882.298349] Lustre: Skipped 1107 previous similar messages
[14430946.357126] Lustre: oak-OST0115: Client 0f13ac10-e5de-07d4-1915-b61e5ad35dce (at 10.210.12.115@tcp1) reconnecting
[14430946.367673] Lustre: Skipped 4 previous similar messages
[14431482.114189] Lustre: oak-OST011f: Connection restored to 46bd8889-1b94-f38a-f202-7268f2e23290 (at 10.50.13.10@o2ib2)
[14431482.124874] Lustre: Skipped 927 previous similar messages
[14432082.242114] Lustre: oak-OST015b: Connection restored to  (at 10.50.7.2@o2ib2)
[14432082.249667] Lustre: Skipped 1536 previous similar messages
[14432681.889430] Lustre: oak-OST013b: Connection restored to 566314ac-13e3-cfa0-4429-150bdba07a39 (at 10.210.12.59@tcp1)
[14432681.900351] Lustre: Skipped 1135 previous similar messages
[14433281.101161] Lustre: oak-OST015b: Connection restored to 684a9b09-ef4a-553f-fc09-c29c5b4ead46 (at 10.210.12.49@tcp1)
[14433281.111825] Lustre: Skipped 1739 previous similar messages
[14433423.230673] LustreError: 162692:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[14433880.807146] Lustre: oak-OST0147: Connection restored to eef96efb-0b41-3c4e-f978-d6b32cfef0f1 (at 10.51.4.64@o2ib3)
[14433880.817729] Lustre: Skipped 1625 previous similar messages
[14434479.367795] Lustre: oak-OST0113: Connection restored to b80be752-f851-25c6-d1f2-d3fd377310cf (at 10.210.12.46@tcp1)
[14434479.378475] Lustre: Skipped 1737 previous similar messages
[14435081.018147] Lustre: oak-OST0115: Connection restored to 3e72271e-d8a9-bff9-a052-ac4ffc30f7d6 (at 10.210.12.40@tcp1)
[14435081.028836] Lustre: Skipped 1497 previous similar messages
[14435515.014423] Lustre: oak-OST0123: Client 7189afb8-8fe5-7418-d14a-4a60d24330d0 (at 10.210.12.48@tcp1) reconnecting
[14435515.024844] Lustre: Skipped 30 previous similar messages
[14435679.915989] Lustre: oak-OST0115: Connection restored to c10b6301-1800-4510-94bd-883f949efde1 (at 10.50.6.68@o2ib2)
[14435679.926618] Lustre: Skipped 1157 previous similar messages
[14436278.588662] Lustre: oak-OST0159: Connection restored to d078775c-2410-ddaa-f93c-e12005aae541 (at 10.50.1.57@o2ib2)
[14436278.599294] Lustre: Skipped 985 previous similar messages
[14436877.283954] Lustre: oak-OST011d: Connection restored to ee36833d-7a97-4f8f-3fad-12018bb3bd79 (at 10.210.12.9@tcp1)
[14436877.294541] Lustre: Skipped 1342 previous similar messages
[14437479.371828] Lustre: oak-OST0137: Connection restored to e26dd6e0-4807-800a-d479-e0310ffdafea (at 10.51.16.2@o2ib3)
[14437479.382440] Lustre: Skipped 1017 previous similar messages
[14438078.597178] Lustre: oak-OST0135: Connection restored to d34cc33f-555c-1a6e-a9d6-bc8c27f9c288 (at 10.50.6.67@o2ib2)
[14438078.607758] Lustre: Skipped 1357 previous similar messages
[14438677.286833] Lustre: oak-OST0113: Connection restored to  (at 10.51.13.14@o2ib3)
[14438677.294390] Lustre: Skipped 1385 previous similar messages
[14439277.982500] Lustre: oak-OST0113: Connection restored to 53f7c8fb-ba13-9226-8f66-6ca61c432359 (at 10.51.12.11@o2ib3)
[14439277.993190] Lustre: Skipped 1478 previous similar messages
[14439704.801517] Lustre: oak-OST0111: Client e44a250f-303f-8c71-9469-648478fb4b8e (at 10.51.7.10@o2ib3) reconnecting
[14439704.811846] Lustre: Skipped 59 previous similar messages
[14439716.132078] LustreError: 137-5: oak-OST014c_UUID: not available for connect from 10.51.7.13@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14439716.132079] LustreError: 137-5: oak-OST0130_UUID: not available for connect from 10.51.7.13@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14439716.167099] LustreError: Skipped 1 previous similar message
[14439722.432831] Lustre: oak-OST0111: Client 20b36988-ad2e-3db2-ffc1-28aca3348c17 (at 10.51.7.13@o2ib3) reconnecting
[14439722.443455] Lustre: Skipped 2 previous similar messages
[14439762.598439] Lustre: oak-OST0145: Client 685231a1-c235-c135-c161-29a0ee359b98 (at 10.51.7.5@o2ib3) reconnecting
[14439877.251416] Lustre: oak-OST0113: Connection restored to 4a082840-902b-d346-3572-f33652a8517f (at 10.50.6.72@o2ib2)
[14439877.262011] Lustre: Skipped 1191 previous similar messages
[14440475.869863] Lustre: oak-OST0131: Connection restored to 566314ac-13e3-cfa0-4429-150bdba07a39 (at 10.210.12.59@tcp1)
[14440475.880559] Lustre: Skipped 1457 previous similar messages
[14441076.474833] Lustre: oak-OST013d: Connection restored to 84f16600-a74a-8a3b-d476-2a157473f8c5 (at 10.50.1.22@o2ib2)
[14441076.485409] Lustre: Skipped 1552 previous similar messages
[14441675.797337] Lustre: oak-OST0135: Connection restored to 51ab7e05-78ff-bf41-8734-92d0767bb7ee (at 10.50.7.21@o2ib2)
[14441675.807921] Lustre: Skipped 1405 previous similar messages
[14441991.168619] Lustre: oak-OST0149: Client 6bab8d10-30ff-c1d2-bbf5-1fd0884ac647 (at 10.210.12.57@tcp1) reconnecting
[14441991.179035] Lustre: Skipped 2 previous similar messages
[14441995.989015] Lustre: oak-OST0145: Client 6bab8d10-30ff-c1d2-bbf5-1fd0884ac647 (at 10.210.12.57@tcp1) reconnecting
[14441995.999459] Lustre: Skipped 9 previous similar messages
[14442013.572269] Lustre: oak-OST0129: Client 6bab8d10-30ff-c1d2-bbf5-1fd0884ac647 (at 10.210.12.57@tcp1) reconnecting
[14442013.582684] Lustre: Skipped 9 previous similar messages
[14442033.689676] Lustre: oak-OST011b: Client 6bab8d10-30ff-c1d2-bbf5-1fd0884ac647 (at 10.210.12.57@tcp1) reconnecting
[14442033.700114] Lustre: Skipped 2 previous similar messages
[14442040.824094] LustreError: 21584:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff918c21e74050 x1716555445154816/t0(0) o4->6bab8d10-30ff-c1d2-bbf5-1fd0884ac647@10.210.12.57@tcp1:537/0 lens 488/448 e 0 to 0 dl 1645669507 ref 1 fl Interpret:/0/0 rc 0/0
[14442040.848694] Lustre: oak-OST014f: Bulk IO write error with 6bab8d10-30ff-c1d2-bbf5-1fd0884ac647 (at 10.210.12.57@tcp1), client will retry: rc = -110
[14442277.629943] Lustre: oak-OST011d: Connection restored to 37c9099c-1a38-a699-2712-432f4cb01e44 (at 10.210.12.19@tcp1)
[14442277.640931] Lustre: Skipped 1234 previous similar messages
[14442876.467964] Lustre: oak-OST015b: Connection restored to aab151b2-b669-9bb0-5460-87842297680c (at 10.50.9.57@o2ib2)
[14442876.478552] Lustre: Skipped 1070 previous similar messages
[14443475.784290] Lustre: oak-OST0157: Connection restored to c476e010-0b79-ff80-57df-a0bb35969f90 (at 10.51.12.1@o2ib3)
[14443475.794913] Lustre: Skipped 1090 previous similar messages
[14444074.391793] Lustre: oak-OST013d: Connection restored to 5424d457-bcc9-00ef-8c94-d87e1d99cc55 (at 10.50.7.59@o2ib2)
[14444074.402402] Lustre: Skipped 1149 previous similar messages
[14444548.990756] Lustre: oak-OST0111: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14444548.991519] LustreError: 137-5: oak-OST0156_UUID: not available for connect from 10.210.12.23@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14444549.018820] Lustre: Skipped 29 previous similar messages
[14444549.600531] LustreError: 160935:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff919e80ec5050 x1714992939412096/t0(0) o4->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:26/0 lens 488/448 e 0 to 0 dl 1645672016 ref 1 fl Interpret:H/0/0 rc 0/0
[14444549.625115] Lustre: oak-OST012b: Bulk IO write error with 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1), client will retry: rc = -110
[14444553.994080] Lustre: oak-OST0135: Client 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1) reconnecting
[14444554.004612] Lustre: Skipped 1 previous similar message
[14444554.580431] LustreError: 243537:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9190f599b850 x1714992939412160/t0(0) o4->876b8936-6ca1-2492-ad54-001855c0b3b0@10.210.12.23@tcp1:26/0 lens 488/448 e 0 to 0 dl 1645672016 ref 1 fl Interpret:/0/0 rc 0/0
[14444554.604766] LustreError: 243537:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[14444554.604933] Lustre: oak-OST015b: Bulk IO write error with 876b8936-6ca1-2492-ad54-001855c0b3b0 (at 10.210.12.23@tcp1), client will retry: rc = -110
[14444554.604933] Lustre: Skipped 1 previous similar message
[14444673.939202] Lustre: oak-OST0153: Connection restored to 923ea94f-6662-1bf9-1a6a-c36a4e4ebfe1 (at 10.50.1.28@o2ib2)
[14444673.949788] Lustre: Skipped 1900 previous similar messages
[14445274.822581] Lustre: oak-OST013b: Connection restored to  (at 10.51.13.12@o2ib3)
[14445274.830135] Lustre: Skipped 966 previous similar messages
[14445880.527484] Lustre: oak-OST015b: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[14445880.538149] Lustre: Skipped 1254 previous similar messages
[14446481.570543] Lustre: oak-OST014f: Connection restored to  (at 10.50.7.2@o2ib2)
[14446481.578010] Lustre: Skipped 1691 previous similar messages
[14447080.145130] Lustre: oak-OST0143: Connection restored to da9c6119-02bc-f370-6fe9-64d4fa45b93d (at 10.50.7.66@o2ib2)
[14447080.155930] Lustre: Skipped 1226 previous similar messages
[14447679.182401] Lustre: oak-OST0127: Connection restored to ce5acce5-7180-2f74-7227-711d0cd68b44 (at 10.51.15.20@o2ib3)
[14447679.193069] Lustre: Skipped 2692 previous similar messages
[14448278.442706] Lustre: oak-OST013f: Connection restored to aa7f7b58-f62e-bf3c-3911-726ba7f5204c (at 10.0.3.52@o2ib5)
[14448278.453227] Lustre: Skipped 1191 previous similar messages
[14448877.253741] Lustre: oak-OST0157: Connection restored to 08e827f3-caa3-2cb0-507e-5bd8243ac158 (at 10.51.12.2@o2ib3)
[14448877.264352] Lustre: Skipped 1008 previous similar messages
[14449475.891671] Lustre: oak-OST0115: Connection restored to  (at 10.50.3.59@o2ib2)
[14449475.899309] Lustre: Skipped 1531 previous similar messages
[14450075.787181] Lustre: oak-OST0157: Connection restored to 32f48011-f5b6-e96a-54bf-48203e3c5ca1 (at 10.51.6.37@o2ib3)
[14450075.797908] Lustre: Skipped 1238 previous similar messages
[14450675.252612] Lustre: oak-OST0147: Connection restored to 8fbf6eea-4385-fb2a-e1ba-f6078d1935b3 (at 10.50.2.16@o2ib2)
[14450675.263204] Lustre: Skipped 1328 previous similar messages
[14451276.036697] Lustre: oak-OST0133: Connection restored to e2e0a5b4-fb83-0b58-6d5a-dfe8d29c6078 (at 10.50.1.39@o2ib2)
[14451276.047349] Lustre: Skipped 1137 previous similar messages
[14451736.668379] Lustre: oak-OST0151: Client 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1) reconnecting
[14451737.586389] LustreError: 253933:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91e05dcce850 x1715767805144384/t0(0) o4->4b39aac6-2dc7-e48f-9434-e3bfde492453@10.210.12.29@tcp1:438/0 lens 488/448 e 0 to 0 dl 1645679223 ref 1 fl Interpret:/0/0 rc 0/0
[14451737.611165] Lustre: oak-OST0151: Bulk IO write error with 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1), client will retry: rc = -110
[14451737.624781] Lustre: Skipped 1 previous similar message
[14451796.382229] LustreError: 243457:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91b416607850 x1715767805144640/t0(0) o4->4b39aac6-2dc7-e48f-9434-e3bfde492453@10.210.12.29@tcp1:438/0 lens 488/448 e 0 to 0 dl 1645679223 ref 1 fl Interpret:/0/0 rc 0/0
[14451796.408695] Lustre: oak-OST0155: Bulk IO write error with 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1), client will retry: rc = -110
[14451816.499597] Lustre: oak-OST0155: Client 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1) reconnecting
[14451874.984920] Lustre: oak-OST013b: Connection restored to  (at 10.50.1.61@o2ib2)
[14451874.992392] Lustre: Skipped 1716 previous similar messages
[14451910.998531] Lustre: oak-OST0147: Client 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1) reconnecting
[14451951.361512] Lustre: oak-OST0151: Client 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1) reconnecting
[14451951.371957] Lustre: Skipped 1 previous similar message
[14452476.798022] Lustre: oak-OST014d: Connection restored to 777822bb-d83f-06a9-ed31-c3219430692d (at 10.50.1.64@o2ib2)
[14452476.808677] Lustre: Skipped 1636 previous similar messages
[14453075.737188] Lustre: oak-OST0151: Connection restored to 0a011db6-92dc-3c19-d585-772b8405b9de (at 10.51.5.14@o2ib3)
[14453075.747780] Lustre: Skipped 1052 previous similar messages
[14453675.171077] Lustre: oak-OST0111: Connection restored to ef0ca35e-4d7b-24ea-8737-c48b55c0297f (at 10.51.13.24@o2ib3)
[14453675.181775] Lustre: Skipped 1530 previous similar messages
[14454273.784516] Lustre: oak-OST0147: Connection restored to b7e3925e-6d6c-b265-4707-4074d142abbe (at 10.50.7.17@o2ib2)
[14454273.795333] Lustre: Skipped 1349 previous similar messages
[14454873.700325] Lustre: oak-OST0127: Connection restored to f99b7a8b-5882-53ac-1091-b46e939ebe50 (at 10.51.2.53@o2ib3)
[14454873.710903] Lustre: Skipped 1889 previous similar messages
[14455472.308233] Lustre: oak-OST013f: Connection restored to  (at 10.50.9.21@o2ib2)
[14455472.315702] Lustre: Skipped 1483 previous similar messages
[14456072.431430] Lustre: oak-OST011f: Connection restored to  (at 10.51.5.52@o2ib3)
[14456072.438942] Lustre: Skipped 1537 previous similar messages
[14456671.374673] Lustre: oak-OST0111: Connection restored to edd06616-023a-51fb-d0e1-f960c04746e6 (at 10.0.3.32@o2ib5)
[14456671.385199] Lustre: Skipped 1336 previous similar messages
[14457270.480250] Lustre: oak-OST015d: Connection restored to fb75c3ae-00e7-a651-8417-b0467bdf032d (at 10.50.5.38@o2ib2)
[14457270.490864] Lustre: Skipped 1209 previous similar messages
[14457870.398333] Lustre: oak-OST0141: Connection restored to f737e776-c753-2b23-866e-4368b7fcef83 (at 10.51.1.22@o2ib3)
[14457870.408924] Lustre: Skipped 1203 previous similar messages
[14458469.432443] Lustre: oak-OST0127: Connection restored to f737e776-c753-2b23-866e-4368b7fcef83 (at 10.51.1.22@o2ib3)
[14458469.443065] Lustre: Skipped 1038 previous similar messages
[14459069.934736] Lustre: oak-OST0133: Connection restored to 1675443e-d7e9-8f00-7311-925c6fb1e6e2 (at 10.50.9.31@o2ib2)
[14459069.945414] Lustre: Skipped 1161 previous similar messages
[14459671.928688] Lustre: oak-OST0153: Connection restored to 45af99cf-0929-72df-1da2-a7dc0bc2a5cf (at 10.51.2.17@o2ib3)
[14459671.939274] Lustre: Skipped 915 previous similar messages
[14460270.500599] Lustre: oak-OST015b: Connection restored to a8af91d8-50f2-dfba-de80-bdf0e55b323f (at 10.210.12.66@tcp1)
[14460270.511267] Lustre: Skipped 1128 previous similar messages
[14460869.085694] Lustre: oak-OST012f: Connection restored to 4f0d1b43-9180-ff69-8774-a1ec250b2851 (at 10.51.6.56@o2ib3)
[14460869.096296] Lustre: Skipped 1921 previous similar messages
[14461473.185401] Lustre: oak-OST012b: Connection restored to 3059db8e-b628-6326-b0b9-f61b3ef5610e (at 10.51.5.46@o2ib3)
[14461473.196198] Lustre: Skipped 1653 previous similar messages
[14462071.838043] Lustre: oak-OST0157: Connection restored to  (at 10.51.15.12@o2ib3)
[14462071.845619] Lustre: Skipped 2271 previous similar messages
[14462568.156240] Lustre: oak-OST0149: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[14462568.418548] LustreError: 21611:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff915df8937850 x1715072402912320/t0(0) o4->bb55a01e-02a3-cbba-5b76-e478fd40967c@10.210.12.54@tcp1:725/0 lens 488/448 e 0 to 0 dl 1645690080 ref 1 fl Interpret:/0/0 rc 0/0
[14462568.443203] Lustre: oak-OST0149: Bulk IO write error with bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1), client will retry: rc = -110
[14462568.694136] LustreError: 21597:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff9148b3a19850 x1715072402912448/t0(0) o4->bb55a01e-02a3-cbba-5b76-e478fd40967c@10.210.12.54@tcp1:730/0 lens 488/448 e 0 to 0 dl 1645690085 ref 1 fl Interpret:/2/0 rc 0/0
[14462568.967213] LustreError: 243496:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff918d35442850 x1715072402912576/t0(0) o4->bb55a01e-02a3-cbba-5b76-e478fd40967c@10.210.12.54@tcp1:725/0 lens 488/448 e 0 to 0 dl 1645690080 ref 1 fl Interpret:/0/0 rc 0/0
[14462568.991856] LustreError: 243496:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[14462569.001934] Lustre: oak-OST0149: Bulk IO write error with bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1), client will retry: rc = -110
[14462569.015498] Lustre: Skipped 2 previous similar messages
[14462646.549055] LustreError: 137-5: oak-OST0138_UUID: not available for connect from 10.210.12.40@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14462646.566653] LustreError: Skipped 1 previous similar message
[14462671.810573] Lustre: oak-OST0147: Connection restored to c8d15be5-5238-2528-45c3-8c06cacdd0d3 (at 10.50.1.45@o2ib2)
[14462671.821269] Lustre: Skipped 1634 previous similar messages
[14462735.868385] Lustre: oak-OST0117: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[14462735.878812] Lustre: Skipped 15 previous similar messages
[14462738.778592] Lustre: oak-OST0123: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[14462738.789061] Lustre: Skipped 4 previous similar messages
[14462743.972914] Lustre: oak-OST0141: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[14462743.983340] Lustre: Skipped 6 previous similar messages
[14462754.965975] Lustre: oak-OST0143: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[14462754.976410] Lustre: Skipped 20 previous similar messages
[14462775.108608] Lustre: oak-OST012f: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[14462775.119112] Lustre: Skipped 8 previous similar messages
[14462818.432142] Lustre: oak-OST0127: Client bb55a01e-02a3-cbba-5b76-e478fd40967c (at 10.210.12.54@tcp1) reconnecting
[14462818.442674] Lustre: Skipped 1 previous similar message
[14463270.915468] Lustre: oak-OST0127: Connection restored to 080fbf36-50e2-e037-d579-97f7ea4dd6a6 (at 10.210.12.51@tcp1)
[14463270.926192] Lustre: Skipped 1131 previous similar messages
[14463873.645665] Lustre: oak-OST014b: Connection restored to 980ce7e0-df3a-7912-c3c6-485e181ecc98 (at 10.50.15.10@o2ib2)
[14463873.656387] Lustre: Skipped 679 previous similar messages
[14464472.199631] Lustre: oak-OST0151: Connection restored to 49ffde34-fdda-ae5f-2b02-f1e084a0de30 (at 10.50.0.63@o2ib2)
[14464472.210237] Lustre: Skipped 1026 previous similar messages
[14465071.083069] Lustre: oak-OST0159: Connection restored to fec83e69-a194-5e68-7ce2-8b9d21d08b2a (at 10.51.2.64@o2ib3)
[14465071.093664] Lustre: Skipped 1502 previous similar messages
[14465669.843179] Lustre: oak-OST0123: Connection restored to ea7323ea-d4bb-2678-29a1-bc8d87c76269 (at 10.210.12.6@tcp1)
[14465669.853787] Lustre: Skipped 1680 previous similar messages
[14466268.466307] Lustre: oak-OST014f: Connection restored to d7562e3a-a4d2-79f0-2778-2a5b795ec38c (at 10.50.9.10@o2ib2)
[14466268.476972] Lustre: Skipped 1419 previous similar messages
[14466765.104123] Lustre: oak-OST012f: Client 1be164da-11a7-a3a8-bd17-8ca7a5aab4e8 (at 10.210.12.69@tcp1) reconnecting
[14466765.114775] Lustre: Skipped 1 previous similar message
[14466867.454807] Lustre: oak-OST0133: Connection restored to 43b45cbd-14a3-0e54-7755-76513da7a096 (at 10.51.1.42@o2ib3)
[14466867.465557] Lustre: Skipped 1692 previous similar messages
[14467467.245506] Lustre: oak-OST0153: Connection restored to  (at 10.50.7.2@o2ib2)
[14467467.252938] Lustre: Skipped 751 previous similar messages
[14468065.935394] Lustre: oak-OST0129: Connection restored to cadd1e4c-e079-2260-6d2d-29034d61d7c1 (at 10.50.5.72@o2ib2)
[14468065.945997] Lustre: Skipped 817 previous similar messages
[14468477.206291] Lustre: oak-OST013d: Client 65e9b8a8-b461-24e0-da12-5b6939f0b07c (at 10.210.12.56@tcp1) reconnecting
[14468477.216710] Lustre: Skipped 6 previous similar messages
[14468665.694095] Lustre: oak-OST0139: Connection restored to 080fbf36-50e2-e037-d579-97f7ea4dd6a6 (at 10.210.12.51@tcp1)
[14468665.704806] Lustre: Skipped 670 previous similar messages
[14469264.279536] Lustre: oak-OST0127: Connection restored to 9e481091-2c29-0032-be39-272874617a20 (at 10.50.9.52@o2ib2)
[14469264.290192] Lustre: Skipped 2197 previous similar messages
[14469279.714249] Lustre: oak-OST0141: Client 3b1423ea-f77c-9301-8ee2-822b91cad2d9 (at 10.210.12.10@tcp1) reconnecting
[14469279.724893] Lustre: Skipped 1 previous similar message
[14469279.956077] LustreError: 243454:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff919b960c5850 x1714982971821248/t0(0) o4->3b1423ea-f77c-9301-8ee2-822b91cad2d9@10.210.12.10@tcp1:657/0 lens 488/448 e 0 to 0 dl 1645696807 ref 1 fl Interpret:/0/0 rc 0/0
[14469279.980500] LustreError: 243454:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[14469279.990322] Lustre: oak-OST0141: Bulk IO write error with 3b1423ea-f77c-9301-8ee2-822b91cad2d9 (at 10.210.12.10@tcp1), client will retry: rc = -110
[14469280.003824] Lustre: Skipped 1 previous similar message
[14469280.578565] LustreError: 160936:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91b59db93850 x1714982971821120/t0(0) o4->3b1423ea-f77c-9301-8ee2-822b91cad2d9@10.210.12.10@tcp1:656/0 lens 488/448 e 0 to 0 dl 1645696806 ref 1 fl Interpret:/0/0 rc 0/0
[14469280.603148] LustreError: 160936:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[14469280.613091] Lustre: oak-OST0141: Bulk IO write error with 3b1423ea-f77c-9301-8ee2-822b91cad2d9 (at 10.210.12.10@tcp1), client will retry: rc = -110
[14469280.613212] LustreError: 127352:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff91c198e18850 x1714982971821120/t0(0) o4->3b1423ea-f77c-9301-8ee2-822b91cad2d9@10.210.12.10@tcp1:662/0 lens 488/448 e 0 to 0 dl 1645696812 ref 1 fl Interpret:/2/0 rc 0/0
[14469280.651323] Lustre: Skipped 2 previous similar messages
[14469322.011593] LustreError: 199273:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91df2d5be850 x1715767896258560/t0(0) o4->4b39aac6-2dc7-e48f-9434-e3bfde492453@10.210.12.29@tcp1:655/0 lens 488/448 e 0 to 0 dl 1645696805 ref 1 fl Interpret:/0/0 rc 0/0
[14469322.037663] Lustre: oak-OST0137: Bulk IO write error with 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1), client will retry: rc = -110
[14469322.051126] Lustre: Skipped 1 previous similar message
[14469345.961803] LustreError: 160893:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91a7bdbe1050 x1715767896264064/t0(0) o4->4b39aac6-2dc7-e48f-9434-e3bfde492453@10.210.12.29@tcp1:659/0 lens 488/448 e 0 to 0 dl 1645696809 ref 1 fl Interpret:/0/0 rc 0/0
[14469345.988050] Lustre: oak-OST0137: Bulk IO write error with 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1), client will retry: rc = -110
[14469356.889389] LustreError: 137-5: oak-OST0158_UUID: not available for connect from 10.210.12.29@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14469356.907660] Lustre: oak-OST0137: Client 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1) reconnecting
[14469356.918126] Lustre: Skipped 1 previous similar message
[14469449.918026] Lustre: oak-OST014f: Client 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1) reconnecting
[14469452.434480] Lustre: oak-OST0111: Client 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1) reconnecting
[14469452.444899] Lustre: Skipped 1 previous similar message
[14469465.639362] Lustre: oak-OST0117: Client 3b1423ea-f77c-9301-8ee2-822b91cad2d9 (at 10.210.12.10@tcp1) reconnecting
[14469465.649777] Lustre: Skipped 8 previous similar messages
[14469474.473147] Lustre: oak-OST0115: Client 3b1423ea-f77c-9301-8ee2-822b91cad2d9 (at 10.210.12.10@tcp1) reconnecting
[14469474.483582] Lustre: Skipped 16 previous similar messages
[14469863.049773] Lustre: oak-OST014f: Connection restored to ea7323ea-d4bb-2678-29a1-bc8d87c76269 (at 10.210.12.6@tcp1)
[14469863.060358] Lustre: Skipped 1618 previous similar messages
[14470462.988665] Lustre: oak-OST013f: Connection restored to 632d6174-105f-f820-efb3-94e214802e62 (at 10.51.16.24@o2ib3)
[14470462.999349] Lustre: Skipped 1575 previous similar messages
[14471061.585862] Lustre: oak-OST013f: Connection restored to ad12f851-216b-c412-f51f-518aba8ea609 (at 10.210.9.195@tcp1)
[14471061.596534] Lustre: Skipped 1398 previous similar messages
[14471660.280956] Lustre: oak-OST0157: Connection restored to 8fd6aa2c-7a73-8519-5ed8-19de1eb4afc1 (at 10.50.10.1@o2ib2)
[14471660.291615] Lustre: Skipped 1211 previous similar messages
[14472259.340854] Lustre: oak-OST0157: Connection restored to b05afedb-bc44-7b43-1e85-7e1fcc3ed586 (at 10.50.1.12@o2ib2)
[14472259.351432] Lustre: Skipped 1420 previous similar messages
[14472859.110912] Lustre: oak-OST013d: Connection restored to c870b475-34f8-d1ac-3897-051a7f74d2e1 (at 10.50.5.29@o2ib2)
[14472859.121509] Lustre: Skipped 1378 previous similar messages
[14473457.820045] Lustre: oak-OST0127: Connection restored to 5fba2acf-ae02-5644-cada-7da1e686119f (at 10.51.2.35@o2ib3)
[14473457.830653] Lustre: Skipped 1356 previous similar messages
[14474056.713785] Lustre: oak-OST0153: Connection restored to 45af99cf-0929-72df-1da2-a7dc0bc2a5cf (at 10.51.2.17@o2ib3)
[14474056.724375] Lustre: Skipped 1185 previous similar messages
[14474657.611549] Lustre: oak-OST0149: Connection restored to 4a082840-902b-d346-3572-f33652a8517f (at 10.50.6.72@o2ib2)
[14474657.622196] Lustre: Skipped 800 previous similar messages
[14475257.331435] Lustre: oak-OST0135: Connection restored to ea7323ea-d4bb-2678-29a1-bc8d87c76269 (at 10.210.12.6@tcp1)
[14475257.342029] Lustre: Skipped 765 previous similar messages
[14475862.576935] Lustre: oak-OST013b: Connection restored to ee36833d-7a97-4f8f-3fad-12018bb3bd79 (at 10.210.12.9@tcp1)
[14475862.587543] Lustre: Skipped 1010 previous similar messages
[14476462.233653] Lustre: oak-OST015b: Connection restored to f7cd9733-9970-c1e4-4322-af5220631562 (at 10.50.5.68@o2ib2)
[14476462.244825] Lustre: Skipped 888 previous similar messages
[14477062.472536] Lustre: oak-OST0139: Connection restored to f1dd735d-6c66-d8b4-50d4-60d2b73caa84 (at 10.51.16.3@o2ib3)
[14477062.483156] Lustre: Skipped 759 previous similar messages
[14477661.458066] Lustre: oak-OST015d: Connection restored to 797bb197-13a6-1620-dd0f-17321ceff735 (at 10.51.1.53@o2ib3)
[14477661.468681] Lustre: Skipped 1088 previous similar messages
[14478260.108272] Lustre: oak-OST0155: Connection restored to 8a9d0ce0-956a-1298-9a74-26ba8b9b05a8 (at 10.50.1.6@o2ib2)
[14478260.118784] Lustre: Skipped 884 previous similar messages
[14478859.913086] Lustre: oak-OST0141: Connection restored to  (at 10.51.5.65@o2ib3)
[14478859.920562] Lustre: Skipped 879 previous similar messages
[14479459.223068] Lustre: oak-OST0157: Connection restored to 55c1e9f1-1a4d-1f72-179c-f4bf6ddb18c7 (at 10.50.14.7@o2ib2)
[14479459.233684] Lustre: Skipped 992 previous similar messages
[14480057.957386] Lustre: oak-OST0131: Connection restored to 97a324d4-c834-549a-4b22-6ab961bb5783 (at 10.0.3.86@o2ib5)
[14480057.967881] Lustre: Skipped 956 previous similar messages
[14480657.322159] Lustre: oak-OST0123: Connection restored to  (at 10.51.15.12@o2ib3)
[14480657.329757] Lustre: Skipped 1040 previous similar messages
[14481256.259133] Lustre: oak-OST0157: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[14481256.269806] Lustre: Skipped 932 previous similar messages
[14481855.023228] Lustre: oak-OST0127: Connection restored to 86eee6bc-6017-fb1a-3873-1027e48995f7 (at 10.50.12.3@o2ib2)
[14481855.033814] Lustre: Skipped 1063 previous similar messages
[14482453.823390] Lustre: oak-OST0117: Connection restored to cc1635db-749d-a61f-d041-27ccf13a08db (at 10.50.5.36@o2ib2)
[14482453.833970] Lustre: Skipped 3209 previous similar messages
[14483052.601049] Lustre: oak-OST0129: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[14483052.611891] Lustre: Skipped 2882 previous similar messages
[14483651.788709] Lustre: oak-OST015b: Connection restored to  (at 10.51.15.6@o2ib3)
[14483651.796256] Lustre: Skipped 2054 previous similar messages
[14484250.759830] Lustre: oak-OST015f: Connection restored to 0855a048-397f-06aa-fa67-fd361082546d (at 10.50.17.2@o2ib2)
[14484250.770429] Lustre: Skipped 1730 previous similar messages
[14484849.359019] Lustre: oak-OST015f: Connection restored to 0e5ce9df-b1b8-ec40-c3b3-5e1b77368ebc (at 10.51.1.55@o2ib3)
[14484849.369608] Lustre: Skipped 1314 previous similar messages
[14485448.426012] Lustre: oak-OST0123: Connection restored to f328dd72-9435-5f0a-0703-7cce3de1cc1e (at 10.50.2.49@o2ib2)
[14485448.436606] Lustre: Skipped 1029 previous similar messages
[14486047.751831] Lustre: oak-OST0123: Connection restored to 97a324d4-c834-549a-4b22-6ab961bb5783 (at 10.0.3.86@o2ib5)
[14486047.762333] Lustre: Skipped 1389 previous similar messages
[14486649.637432] Lustre: oak-OST0151: Connection restored to a80f6d08-4459-2bd2-7cca-724b51375a1b (at 10.50.15.13@o2ib2)
[14486649.648209] Lustre: Skipped 1054 previous similar messages
[14487248.276421] Lustre: oak-OST0155: Connection restored to 8817ba1c-727b-5352-25f7-4d78dc602696 (at 10.210.12.68@tcp1)
[14487248.287097] Lustre: Skipped 654 previous similar messages
[14487458.816546] Lustre: oak-OST011d: Client 92828426-e9f0-508a-3226-9184d3612426 (at 10.210.12.38@tcp1) reconnecting
[14487458.816547] Lustre: oak-OST011f: Client 92828426-e9f0-508a-3226-9184d3612426 (at 10.210.12.38@tcp1) reconnecting
[14487458.837366] Lustre: Skipped 25 previous similar messages
[14487458.877553] LustreError: 127356:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91afb221d850 x1715057001986688/t0(0) o4->92828426-e9f0-508a-3226-9184d3612426@10.210.12.38@tcp1:706/0 lens 488/448 e 0 to 0 dl 1645714976 ref 1 fl Interpret:/0/0 rc 0/0
[14487458.902001] LustreError: 127356:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[14487458.912111] Lustre: oak-OST0133: Bulk IO write error with 92828426-e9f0-508a-3226-9184d3612426 (at 10.210.12.38@tcp1), client will retry: rc = -110
[14487847.497427] Lustre: oak-OST014b: Connection restored to eee699b2-cb8e-0a91-6752-be0f592e4945 (at 10.50.13.2@o2ib2)
[14487847.508014] Lustre: Skipped 774 previous similar messages
[14488446.284611] Lustre: oak-OST012f: Connection restored to  (at 10.50.4.30@o2ib2)
[14488446.292095] Lustre: Skipped 1161 previous similar messages
[14489045.840390] Lustre: oak-OST0131: Connection restored to a1f03f47-1d53-e240-1865-67d24a3567e9 (at 10.51.5.18@o2ib3)
[14489045.850971] Lustre: Skipped 2850 previous similar messages
[14489644.400681] Lustre: oak-OST0155: Connection restored to 4025d53e-c01c-f5a6-a7de-c3a1c13db1f4 (at 10.50.10.5@o2ib2)
[14489644.411497] Lustre: Skipped 1390 previous similar messages
[14490243.074005] Lustre: oak-OST0147: Connection restored to f94deebd-8de5-a544-3246-a787f9570e29 (at 10.210.12.15@tcp1)
[14490243.084670] Lustre: Skipped 954 previous similar messages
[14490841.780377] Lustre: oak-OST0157: Connection restored to c193930f-3dab-28da-0eac-55422998d272 (at 10.51.13.13@o2ib3)
[14490841.791106] Lustre: Skipped 1001 previous similar messages
[14491441.316044] Lustre: oak-OST014f: Connection restored to c476e010-0b79-ff80-57df-a0bb35969f90 (at 10.51.12.1@o2ib3)
[14491441.326723] Lustre: Skipped 4699 previous similar messages
[14492040.217174] Lustre: oak-OST015f: Connection restored to  (at 10.51.6.11@o2ib3)
[14492040.224643] Lustre: Skipped 1525 previous similar messages
[14492639.719061] Lustre: oak-OST0121: Connection restored to d579f8e1-497a-3771-f6d4-ffe96dea772a (at 10.210.12.62@tcp1)
[14492639.729915] Lustre: Skipped 965 previous similar messages
[14493238.903582] Lustre: oak-OST014f: Connection restored to  (at 10.51.0.65@o2ib3)
[14493238.911237] Lustre: Skipped 1232 previous similar messages
[14493838.479680] Lustre: oak-OST0157: Connection restored to  (at 10.50.10.53@o2ib2)
[14493838.487237] Lustre: Skipped 1078 previous similar messages
[14494437.424117] Lustre: oak-OST0153: Connection restored to 080fbf36-50e2-e037-d579-97f7ea4dd6a6 (at 10.210.12.51@tcp1)
[14494437.434791] Lustre: Skipped 1077 previous similar messages
[14495039.009771] Lustre: oak-OST015d: Connection restored to  (at 10.50.3.50@o2ib2)
[14495039.017297] Lustre: Skipped 989 previous similar messages
[14495637.676271] Lustre: oak-OST013d: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[14495637.686944] Lustre: Skipped 2500 previous similar messages
[14496237.893938] Lustre: oak-OST015d: Connection restored to 080fbf36-50e2-e037-d579-97f7ea4dd6a6 (at 10.210.12.51@tcp1)
[14496237.904684] Lustre: Skipped 1103 previous similar messages
[14496302.678223] LustreError: 137-5: oak-OST014a_UUID: not available for connect from 10.210.12.49@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14496399.889327] Lustre: oak-OST014d: Client 0d74ac3e-379b-ab1b-da57-9cd708a46466 (at 10.210.12.49@tcp1) reconnecting
[14496413.060881] Lustre: oak-OST0157: Client 0d74ac3e-379b-ab1b-da57-9cd708a46466 (at 10.210.12.49@tcp1) reconnecting
[14496424.417607] Lustre: oak-OST015f: Client 0d74ac3e-379b-ab1b-da57-9cd708a46466 (at 10.210.12.49@tcp1) reconnecting
[14496448.573469] Lustre: oak-OST014b: Client 0d74ac3e-379b-ab1b-da57-9cd708a46466 (at 10.210.12.49@tcp1) reconnecting
[14496543.705991] LustreError: 244101:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91bbf82b3850 x1725566649124608/t0(0) o4->97e0b978-47b3-d181-6e4a-d3ffe3c29d05@10.210.12.9@tcp1:5/0 lens 488/448 e 0 to 0 dl 1645724090 ref 1 fl Interpret:/0/0 rc 0/0
[14496543.706239] Lustre: oak-OST0159: Bulk IO write error with 97e0b978-47b3-d181-6e4a-d3ffe3c29d05 (at 10.210.12.9@tcp1), client will retry: rc = -110
[14496543.744974] LustreError: 244101:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[14496575.772586] Lustre: oak-OST0159: Client 97e0b978-47b3-d181-6e4a-d3ffe3c29d05 (at 10.210.12.9@tcp1) reconnecting
[14496674.014740] Lustre: oak-OST0125: Client 97e0b978-47b3-d181-6e4a-d3ffe3c29d05 (at 10.210.12.9@tcp1) reconnecting
[14496674.025057] Lustre: Skipped 2 previous similar messages
[14496719.724603] Lustre: oak-OST0127: Client 97e0b978-47b3-d181-6e4a-d3ffe3c29d05 (at 10.210.12.9@tcp1) reconnecting
[14496719.734959] Lustre: Skipped 17 previous similar messages
[14496833.732983] Lustre: oak-OST0133: Client 92828426-e9f0-508a-3226-9184d3612426 (at 10.210.12.38@tcp1) reconnecting
[14496833.743404] Lustre: Skipped 1 previous similar message
[14496834.232723] LustreError: 160902:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91ef256db850 x1715057048217088/t0(0) o4->92828426-e9f0-508a-3226-9184d3612426@10.210.12.38@tcp1:344/0 lens 488/448 e 0 to 0 dl 1645724429 ref 1 fl Interpret:/0/0 rc 0/0
[14496834.259440] Lustre: oak-OST0147: Bulk IO write error with 92828426-e9f0-508a-3226-9184d3612426 (at 10.210.12.38@tcp1), client will retry: rc = -110
[14496834.272873] Lustre: Skipped 2 previous similar messages
[14496835.254236] LustreError: 243450:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff919712862850 x1715057048217088/t0(0) o4->92828426-e9f0-508a-3226-9184d3612426@10.210.12.38@tcp1:348/0 lens 488/448 e 0 to 0 dl 1645724433 ref 1 fl Interpret:/2/0 rc 0/0
[14496835.278648] LustreError: 243450:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[14496835.288482] Lustre: oak-OST0147: Bulk IO write error with 92828426-e9f0-508a-3226-9184d3612426 (at 10.210.12.38@tcp1), client will retry: rc = -110
[14496835.301948] Lustre: Skipped 1 previous similar message
[14496836.526632] Lustre: oak-OST0155: Connection restored to 86110018-10f1-582f-228f-1aa8a96fbb31 (at 10.51.6.38@o2ib3)
[14496836.537314] Lustre: Skipped 1103 previous similar messages
[14497005.442153] Lustre: oak-OST0123: Client 92828426-e9f0-508a-3226-9184d3612426 (at 10.210.12.38@tcp1) reconnecting
[14497005.452967] Lustre: Skipped 2 previous similar messages
[14497436.532416] Lustre: oak-OST0121: Connection restored to 72e9c356-8685-a676-ede7-2f53484b502b (at 10.50.10.7@o2ib2)
[14497436.543077] Lustre: Skipped 1100 previous similar messages
[14498044.952719] Lustre: oak-OST011f: Connection restored to  (at 10.50.16.2@o2ib2)
[14498044.960366] Lustre: Skipped 890 previous similar messages
[14498643.998523] Lustre: oak-OST0155: Connection restored to  (at 10.51.15.4@o2ib3)
[14498644.005992] Lustre: Skipped 929 previous similar messages
[14499242.726712] Lustre: oak-OST0147: Connection restored to e95a0af8-32e9-9a68-a82a-f9b1c65f93f0 (at 10.50.7.1@o2ib2)
[14499242.737521] Lustre: Skipped 787 previous similar messages
[14499841.811821] Lustre: oak-OST0131: Connection restored to 74a465f8-66d2-a778-f17d-812bb5379207 (at 10.51.0.64@o2ib3)
[14499841.822942] Lustre: Skipped 760 previous similar messages
[14499881.805392] Lustre: oak-OST011b: Client f82ac34f-4b3a-8ef0-8e11-38edade2effc (at 10.51.7.2@o2ib3) reconnecting
[14499881.815658] Lustre: Skipped 34 previous similar messages
[14499885.307849] LustreError: 137-5: oak-OST0122_UUID: not available for connect from 10.51.7.4@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14499885.307850] LustreError: 137-5: oak-OST0126_UUID: not available for connect from 10.51.7.4@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14499885.307851] LustreError: 137-5: oak-OST0124_UUID: not available for connect from 10.51.7.4@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14499885.360096] LustreError: Skipped 8 previous similar messages
[14499889.339336] LustreError: 137-5: oak-OST013e_UUID: not available for connect from 10.51.7.1@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14499892.220598] LustreError: 137-5: oak-OST0130_UUID: not available for connect from 10.51.7.3@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14499892.220599] LustreError: 137-5: oak-OST0114_UUID: not available for connect from 10.51.7.3@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14499892.220600] LustreError: 137-5: oak-OST0136_UUID: not available for connect from 10.51.7.3@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14499892.220601] LustreError: Skipped 2 previous similar messages
[14499892.220603] LustreError: Skipped 2 previous similar messages
[14499892.284706] LustreError: Skipped 5 previous similar messages
[14499946.625772] Lustre: oak-OST0127: Client 3b042b0e-f945-b8d0-cedd-f51878ec7fe8 (at 10.51.7.8@o2ib3) reconnecting
[14499946.636173] Lustre: Skipped 54 previous similar messages
[14500035.804577] Lustre: oak-OST013b: Client 7806fadc-0a91-29ad-5e69-cb2a183f3972 (at 10.51.7.7@o2ib3) reconnecting
[14500035.815049] Lustre: Skipped 8 previous similar messages
[14500050.485930] LustreError: 137-5: oak-OST0138_UUID: not available for connect from 10.51.7.8@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14500050.503365] LustreError: Skipped 1 previous similar message
[14500084.231927] LustreError: 137-5: oak-OST011e_UUID: not available for connect from 10.51.7.1@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14500084.249343] LustreError: Skipped 6 previous similar messages
[14500442.091151] Lustre: oak-OST014f: Connection restored to  (at 10.50.7.2@o2ib2)
[14500442.098529] Lustre: Skipped 1927 previous similar messages
[14501040.877989] Lustre: oak-OST011f: Connection restored to  (at 10.50.16.2@o2ib2)
[14501040.885464] Lustre: Skipped 1222 previous similar messages
[14501639.522951] Lustre: oak-OST0127: Connection restored to 080fbf36-50e2-e037-d579-97f7ea4dd6a6 (at 10.210.12.51@tcp1)
[14501639.533620] Lustre: Skipped 819 previous similar messages
[14502240.181366] Lustre: oak-OST0155: Connection restored to  (at 10.50.15.11@o2ib2)
[14502240.188975] Lustre: Skipped 841 previous similar messages
[14502840.902161] Lustre: oak-OST013d: Connection restored to ca2e46d5-bfa6-73b9-3298-0d35b5b278ca (at 10.50.16.9@o2ib2)
[14502840.912977] Lustre: Skipped 1067 previous similar messages
[14503439.465437] Lustre: oak-OST0111: Connection restored to  (at 10.50.3.53@o2ib2)
[14503439.472903] Lustre: Skipped 1371 previous similar messages
[14504038.844575] Lustre: oak-OST0159: Connection restored to 74a465f8-66d2-a778-f17d-812bb5379207 (at 10.51.0.64@o2ib3)
[14504038.855326] Lustre: Skipped 1137 previous similar messages
[14504639.627927] Lustre: oak-OST0159: Connection restored to 97cbb879-24a6-a81f-ba52-733dfacac123 (at 10.51.12.9@o2ib3)
[14504639.638594] Lustre: Skipped 1118 previous similar messages
[14505239.059031] Lustre: oak-OST0133: Connection restored to 916093a3-51ee-71cf-ab61-9d8121d32b7c (at 10.51.15.11@o2ib3)
[14505239.069697] Lustre: Skipped 1060 previous similar messages
[14505841.040497] Lustre: oak-OST0153: Connection restored to 97cbb879-24a6-a81f-ba52-733dfacac123 (at 10.51.12.9@o2ib3)
[14505841.051133] Lustre: Skipped 865 previous similar messages
[14506439.801070] Lustre: oak-OST012b: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[14506439.811748] Lustre: Skipped 831 previous similar messages
[14506920.918308] LustreError: 160911:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[14507038.357248] Lustre: oak-OST0143: Connection restored to 2a199385-e7e7-8728-51af-ca75fe3a692e (at 10.50.17.29@o2ib2)
[14507038.367918] Lustre: Skipped 1005 previous similar messages
[14507640.613146] Lustre: oak-OST015b: Connection restored to  (at 10.50.7.2@o2ib2)
[14507640.620536] Lustre: Skipped 1342 previous similar messages
[14507653.133812] LustreError: 21619:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff913d583b5050 x1715539315836992/t0(0) o4->5a5caa86-68df-f806-7e6f-05440e95a4ce@10.210.12.43@tcp1:566/0 lens 488/448 e 0 to 0 dl 1645735221 ref 1 fl Interpret:/0/0 rc 0/0
[14507653.133992] Lustre: oak-OST0153: Bulk IO write error with 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1), client will retry: rc = -110
[14507653.172990] LustreError: 21619:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[14507679.114131] LustreError: 137-5: oak-OST014a_UUID: not available for connect from 10.210.12.49@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14507679.131901] LustreError: Skipped 6 previous similar messages
[14507679.917576] Lustre: oak-OST0153: Client 5a5caa86-68df-f806-7e6f-05440e95a4ce (at 10.210.12.43@tcp1) reconnecting
[14507679.927986] Lustre: Skipped 64 previous similar messages
[14507771.722192] Lustre: oak-OST0111: Client 0d74ac3e-379b-ab1b-da57-9cd708a46466 (at 10.210.12.49@tcp1) reconnecting
[14507771.732607] Lustre: Skipped 6 previous similar messages
[14507860.582383] LustreError: 137-5: oak-OST015e_UUID: not available for connect from 10.210.12.74@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14507953.122279] Lustre: oak-OST015f: Client 6f45695d-f173-ee1d-e6cb-d38dad7e0879 (at 10.210.12.74@tcp1) reconnecting
[14507953.132691] Lustre: Skipped 69 previous similar messages
[14507989.260768] Lustre: oak-OST0127: Client 6f45695d-f173-ee1d-e6cb-d38dad7e0879 (at 10.210.12.74@tcp1) reconnecting
[14507989.271180] Lustre: Skipped 28 previous similar messages
[14508239.470931] Lustre: oak-OST014f: Connection restored to 9eb8189a-3985-f41c-7d7b-51af46681d5f (at 10.0.3.29@o2ib5)
[14508239.481460] Lustre: Skipped 1527 previous similar messages
[14508713.275344] Lustre: oak-OST0111: Client 1be164da-11a7-a3a8-bd17-8ca7a5aab4e8 (at 10.210.12.69@tcp1) reconnecting
[14508713.285821] Lustre: Skipped 6 previous similar messages
[14508721.271290] Lustre: oak-OST0115: Client 1be164da-11a7-a3a8-bd17-8ca7a5aab4e8 (at 10.210.12.69@tcp1) reconnecting
[14508721.281694] Lustre: Skipped 14 previous similar messages
[14508738.815664] Lustre: oak-OST0137: Client 1be164da-11a7-a3a8-bd17-8ca7a5aab4e8 (at 10.210.12.69@tcp1) reconnecting
[14508738.826230] Lustre: Skipped 2 previous similar messages
[14508838.963271] Lustre: oak-OST0143: Connection restored to e596798d-98e3-2570-92e9-a92e6aea98bf (at 10.210.12.64@tcp1)
[14508838.973939] Lustre: Skipped 1192 previous similar messages
[14509281.116613] LustreError: 243453:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91cd36c10050 x1714974335983552/t0(0) o4->5afa23c2-1fd5-9030-38ce-b03a359082fe@10.210.12.75@tcp1:682/0 lens 488/448 e 0 to 0 dl 1645736847 ref 1 fl Interpret:/0/0 rc 0/0
[14509281.116841] Lustre: oak-OST015b: Bulk IO write error with 5afa23c2-1fd5-9030-38ce-b03a359082fe (at 10.210.12.75@tcp1), client will retry: rc = -110
[14509281.116842] Lustre: Skipped 2 previous similar messages
[14509281.161330] LustreError: 243453:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[14509302.618133] LustreError: 137-5: oak-OST014a_UUID: not available for connect from 10.210.12.49@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14509302.803276] Lustre: oak-OST015b: Client 5afa23c2-1fd5-9030-38ce-b03a359082fe (at 10.210.12.75@tcp1) reconnecting
[14509389.402138] Lustre: oak-OST0157: Client 5afa23c2-1fd5-9030-38ce-b03a359082fe (at 10.210.12.75@tcp1) reconnecting
[14509437.814998] Lustre: oak-OST015d: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[14509437.825710] Lustre: Skipped 1107 previous similar messages
[14510040.547847] Lustre: oak-OST0155: Connection restored to 582484fd-3cd6-e7cb-180b-ae6af9fe1e87 (at 10.50.10.8@o2ib2)
[14510040.558431] Lustre: Skipped 1002 previous similar messages
[14510640.021609] Lustre: oak-OST012b: Connection restored to efa3260b-85f8-753d-2bfc-fbd16b6c6f94 (at 10.50.14.5@o2ib2)
[14510640.032234] Lustre: Skipped 1048 previous similar messages
[14511239.468494] Lustre: oak-OST0119: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[14511239.479199] Lustre: Skipped 1557 previous similar messages
[14511841.922435] Lustre: oak-OST0153: Connection restored to  (at 10.50.3.58@o2ib2)
[14511841.929907] Lustre: Skipped 1362 previous similar messages
[14512441.282499] Lustre: oak-OST0149: Connection restored to 55c1e9f1-1a4d-1f72-179c-f4bf6ddb18c7 (at 10.50.14.7@o2ib2)
[14512441.293087] Lustre: Skipped 1008 previous similar messages
[14513039.943225] Lustre: oak-OST013d: Connection restored to  (at 10.50.7.2@o2ib2)
[14513039.950671] Lustre: Skipped 1150 previous similar messages
[14513638.990777] Lustre: oak-OST014f: Connection restored to 8817ba1c-727b-5352-25f7-4d78dc602696 (at 10.210.12.68@tcp1)
[14513639.001499] Lustre: Skipped 1403 previous similar messages
[14514242.017250] Lustre: oak-OST0159: Connection restored to 06c1c22b-e4ac-81cb-ef54-4744df1b23e1 (at 10.51.7.12@o2ib3)
[14514242.027849] Lustre: Skipped 862 previous similar messages
[14514841.579611] Lustre: oak-OST0149: Connection restored to  (at 10.51.15.4@o2ib3)
[14514841.587077] Lustre: Skipped 1117 previous similar messages
[14515442.056230] Lustre: oak-OST0153: Connection restored to fcb658a0-e3f9-6034-2fe7-0c07f976347c (at 10.50.0.71@o2ib2)
[14515442.066805] Lustre: Skipped 1133 previous similar messages
[14516041.478539] Lustre: oak-OST011d: Connection restored to 99b5b5a5-0765-c613-dfd8-cfcc7cda569c (at 10.50.4.4@o2ib2)
[14516041.489046] Lustre: Skipped 1168 previous similar messages
[14516641.008834] Lustre: oak-OST0159: Connection restored to 2e65563e-3083-57c7-a9c3-432051f83abc (at 10.51.0.68@o2ib3)
[14516641.019457] Lustre: Skipped 1211 previous similar messages
[14517239.671497] Lustre: oak-OST0137: Connection restored to c977f6f4-35de-1e9a-97fb-439d33a2737d (at 10.0.3.6@o2ib5)
[14517239.681912] Lustre: Skipped 1316 previous similar messages
[14517739.678686] Lustre: oak-OST0111: Client 971db391-3243-93db-9d97-af535db702bb (at 10.51.7.11@o2ib3) reconnecting
[14517739.689040] Lustre: Skipped 71 previous similar messages
[14517744.041063] LustreError: 137-5: oak-OST0146_UUID: not available for connect from 10.51.7.15@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14517775.787807] Lustre: oak-OST0111: Client e44a250f-303f-8c71-9469-648478fb4b8e (at 10.51.7.10@o2ib3) reconnecting
[14517775.798130] Lustre: Skipped 17 previous similar messages
[14517838.870438] Lustre: oak-OST0149: Connection restored to ab61d733-6278-aecc-ffdb-baa33579a940 (at 10.51.4.33@o2ib3)
[14517838.881118] Lustre: Skipped 978 previous similar messages
[14518378.972096] LustreError: 212522:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91aba96ec850 x1715051718235648/t0(0) o4->80374d70-7d99-f68a-b58b-86c0b7514d6a@10.210.12.65@tcp1:742/0 lens 488/448 e 0 to 0 dl 1645745967 ref 1 fl Interpret:/0/0 rc 0/0
[14518378.972436] Lustre: oak-OST011f: Bulk IO write error with 80374d70-7d99-f68a-b58b-86c0b7514d6a (at 10.210.12.65@tcp1), client will retry: rc = -110
[14518378.972437] Lustre: Skipped 1 previous similar message
[14518379.016792] LustreError: 212522:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 3 previous similar messages
[14518400.872456] Lustre: oak-OST011f: Client 80374d70-7d99-f68a-b58b-86c0b7514d6a (at 10.210.12.65@tcp1) reconnecting
[14518400.882921] Lustre: Skipped 4 previous similar messages
[14518437.428985] Lustre: oak-OST011f: Connection restored to d761ff1f-fadf-d10a-f00e-74a20580e81b (at 10.210.12.11@tcp1)
[14518437.440119] Lustre: Skipped 1479 previous similar messages
[14518488.653048] Lustre: oak-OST011d: Client 80374d70-7d99-f68a-b58b-86c0b7514d6a (at 10.210.12.65@tcp1) reconnecting
[14518488.663460] Lustre: Skipped 2 previous similar messages
[14519037.547434] Lustre: oak-OST0151: Connection restored to b721998c-1a52-1c1c-cbd9-f9819458c553 (at 10.210.12.131@tcp1)
[14519037.558196] Lustre: Skipped 1774 previous similar messages
[14519259.836897] Lustre: oak-OST0149: Client b69058ea-844c-325f-f63b-7347539374f7 (at 10.210.12.51@tcp1) reconnecting
[14519259.847327] Lustre: Skipped 111 previous similar messages
[14519260.535234] LustreError: 160946:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91b45ce55050 x1714941707006720/t0(0) o4->b69058ea-844c-325f-f63b-7347539374f7@10.210.12.51@tcp1:179/0 lens 488/448 e 0 to 0 dl 1645746914 ref 1 fl Interpret:/0/0 rc 0/0
[14519260.564167] Lustre: oak-OST0149: Bulk IO write error with b69058ea-844c-325f-f63b-7347539374f7 (at 10.210.12.51@tcp1), client will retry: rc = -110
[14519260.577666] Lustre: Skipped 3 previous similar messages
[14519264.367047] LustreError: 160893:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff91f16f671050 x1715768053607424/t0(0) o4->4b39aac6-2dc7-e48f-9434-e3bfde492453@10.210.12.29@tcp1:119/0 lens 488/448 e 0 to 0 dl 1645746854 ref 1 fl Interpret:/0/0 rc 0/0
[14519264.367274] Lustre: oak-OST0129: Bulk IO write error with 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1), client will retry: rc = -110
[14519264.406559] LustreError: 160893:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[14519283.658712] LustreError: 137-5: oak-OST0150_UUID: not available for connect from 10.210.12.29@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14519283.661394] Lustre: oak-OST015f: Client 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1) reconnecting
[14519283.686699] LustreError: Skipped 1 previous similar message
[14519372.045050] Lustre: oak-OST012f: Client 4b39aac6-2dc7-e48f-9434-e3bfde492453 (at 10.210.12.29@tcp1) reconnecting
[14519372.055490] Lustre: Skipped 3 previous similar messages
[14519445.656948] Lustre: oak-OST012f: Client b69058ea-844c-325f-f63b-7347539374f7 (at 10.210.12.51@tcp1) reconnecting
[14519445.667356] Lustre: Skipped 31 previous similar messages
[14519637.927127] Lustre: oak-OST0135: Connection restored to 72e9c356-8685-a676-ede7-2f53484b502b (at 10.50.10.7@o2ib2)
[14519637.937726] Lustre: Skipped 1342 previous similar messages
[14519722.278317] Lustre: oak-OST012f: Client b69058ea-844c-325f-f63b-7347539374f7 (at 10.210.12.51@tcp1) reconnecting
[14520236.843007] Lustre: oak-OST015d: Connection restored to 52cd7f8b-e0d9-26a0-cb6a-5a14c8ceacce (at 10.51.13.9@o2ib3)
[14520236.853607] Lustre: Skipped 1345 previous similar messages
[14520835.515222] Lustre: oak-OST012f: Connection restored to 7ce52af9-b9cc-6a1a-4f45-1c63ca6f6364 (at 10.51.2.49@o2ib3)
[14520835.525808] Lustre: Skipped 1550 previous similar messages
[14521434.505078] Lustre: oak-OST015f: Connection restored to a6a6aa2b-42c4-aeb4-b8d5-c31980f11905 (at 10.51.1.16@o2ib3)
[14521434.515832] Lustre: Skipped 2189 previous similar messages
[14522033.678112] Lustre: oak-OST014f: Connection restored to 92e4555f-45a5-bddb-4e52-f34f9782782a (at 10.50.2.29@o2ib2)
[14522033.688847] Lustre: Skipped 1733 previous similar messages
[14522632.255125] Lustre: oak-OST0111: Connection restored to ded9a7c5-4772-1464-d085-edb54ff675c6 (at 10.50.1.29@o2ib2)
[14522632.265710] Lustre: Skipped 1385 previous similar messages
[14523230.943366] Lustre: oak-OST014f: Connection restored to eee699b2-cb8e-0a91-6752-be0f592e4945 (at 10.50.13.2@o2ib2)
[14523230.953971] Lustre: Skipped 2567 previous similar messages
[14523248.598939] Lustre: oak-OST0113: Client b17226f5-e55e-40a8-722a-0ee72011f267 (at 10.210.12.13@tcp1) reconnecting
[14523248.631396] LustreError: 162715:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91681d98c850 x1714924149180864/t0(0) o4->b17226f5-e55e-40a8-722a-0ee72011f267@10.210.12.13@tcp1:397/0 lens 488/448 e 0 to 0 dl 1645750907 ref 1 fl Interpret:/0/0 rc 0/0
[14523248.656109] Lustre: oak-OST0113: Bulk IO write error with b17226f5-e55e-40a8-722a-0ee72011f267 (at 10.210.12.13@tcp1), client will retry: rc = -110
[14523248.669609] Lustre: Skipped 2 previous similar messages
[14523249.378596] LustreError: 243440:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff914b78354850 x1714924149180736/t0(0) o4->b17226f5-e55e-40a8-722a-0ee72011f267@10.210.12.13@tcp1:397/0 lens 488/448 e 0 to 0 dl 1645750907 ref 1 fl Interpret:/0/0 rc 0/0
[14523249.403704] LustreError: 243440:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[14523249.414359] Lustre: oak-OST0113: Bulk IO write error with b17226f5-e55e-40a8-722a-0ee72011f267 (at 10.210.12.13@tcp1), client will retry: rc = -110
[14523249.429073] Lustre: Skipped 1 previous similar message
[14523415.657747] Lustre: oak-OST0155: Client 6c4f6cc2-bd5d-4837-3cd2-cf2eacb9f2ef (at 10.210.12.68@tcp1) reconnecting
[14523415.668209] Lustre: Skipped 2 previous similar messages
[14523830.078565] Lustre: oak-OST014b: Connection restored to  (at 10.51.5.30@o2ib3)
[14523830.086037] Lustre: Skipped 1847 previous similar messages
[14524222.626424] LustreError: 160899:0:(lprocfs_jobstats.c:283:lprocfs_job_stats_log()) Invalid jobid size (37), expect(32)
[14524429.696807] Lustre: oak-OST0155: Connection restored to  (at 10.50.16.11@o2ib2)
[14524429.704571] Lustre: Skipped 1606 previous similar messages
[14525028.678208] Lustre: oak-OST0131: Connection restored to 513c26fb-6b04-7685-eea3-05debb535338 (at 10.51.5.31@o2ib3)
[14525028.688897] Lustre: Skipped 1518 previous similar messages
[14525628.515698] Lustre: oak-OST0133: Connection restored to dd8d0d48-27c0-6a3d-f3e4-c2a17d82a3cb (at 10.51.5.57@o2ib3)
[14525628.526305] Lustre: Skipped 2239 previous similar messages
[14526228.985502] Lustre: oak-OST0141: Connection restored to  (at 10.50.4.30@o2ib2)
[14526228.992986] Lustre: Skipped 1143 previous similar messages
[14526830.155512] Lustre: oak-OST0157: Connection restored to 6d63cbc4-6143-910c-ba58-90558ade6da5 (at 10.50.12.13@o2ib2)
[14526830.166183] Lustre: Skipped 1218 previous similar messages
[14527429.105029] Lustre: oak-OST0149: Connection restored to 3abc1bac-6e1a-08b0-893b-1e34e457b6fb (at 10.50.5.32@o2ib2)
[14527429.115672] Lustre: Skipped 1148 previous similar messages
[14527548.247910] LustreError: 127358:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91c98ece2050 x1714924180826624/t0(0) o4->b17226f5-e55e-40a8-722a-0ee72011f267@10.210.12.13@tcp1:123/0 lens 488/448 e 0 to 0 dl 1645755163 ref 1 fl Interpret:/0/0 rc 0/0
[14527548.248189] Lustre: oak-OST0113: Bulk IO write error with b17226f5-e55e-40a8-722a-0ee72011f267 (at 10.210.12.13@tcp1), client will retry: rc = -110
[14527548.248191] Lustre: Skipped 3 previous similar messages
[14527548.292915] LustreError: 127358:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 3 previous similar messages
[14527573.617867] Lustre: oak-OST0113: Client b17226f5-e55e-40a8-722a-0ee72011f267 (at 10.210.12.13@tcp1) reconnecting
[14527573.628300] Lustre: Skipped 44 previous similar messages
[14527666.496705] Lustre: oak-OST0117: Client b17226f5-e55e-40a8-722a-0ee72011f267 (at 10.210.12.13@tcp1) reconnecting
[14527666.507550] Lustre: Skipped 9 previous similar messages
[14527697.274038] Lustre: oak-OST0111: Client b17226f5-e55e-40a8-722a-0ee72011f267 (at 10.210.12.13@tcp1) reconnecting
[14527697.284448] Lustre: Skipped 5 previous similar messages
[14528028.537250] Lustre: oak-OST0157: Connection restored to  (at 10.51.15.9@o2ib3)
[14528028.544715] Lustre: Skipped 1439 previous similar messages
[14528628.015958] Lustre: oak-OST014f: Connection restored to b05afedb-bc44-7b43-1e85-7e1fcc3ed586 (at 10.50.1.12@o2ib2)
[14528628.026586] Lustre: Skipped 1101 previous similar messages
[14529226.983143] Lustre: oak-OST012b: Connection restored to  (at 10.51.6.20@o2ib3)
[14529226.990632] Lustre: Skipped 1368 previous similar messages
[14529828.359802] Lustre: oak-OST0153: Connection restored to 1cc3b6e6-945c-3742-e968-c8b047c536df (at 10.51.13.16@o2ib3)
[14529828.370487] Lustre: Skipped 1203 previous similar messages
[14530428.107309] Lustre: oak-OST015b: Connection restored to 777010eb-04be-6190-58ce-47411b2d8929 (at 10.50.7.24@o2ib2)
[14530428.118251] Lustre: Skipped 1112 previous similar messages
[14531026.728720] Lustre: oak-OST0135: Connection restored to ddd6e198-f36f-14d9-41a2-19f9ebbc987c (at 10.51.6.2@o2ib3)
[14531026.739214] Lustre: Skipped 1070 previous similar messages
[14531625.916977] Lustre: oak-OST0159: Connection restored to ccf03fa0-a8d9-39d7-4290-e2d1f60add74 (at 10.50.10.40@o2ib2)
[14531625.927695] Lustre: Skipped 1123 previous similar messages
[14532228.454722] Lustre: oak-OST015f: Connection restored to 5424d457-bcc9-00ef-8c94-d87e1d99cc55 (at 10.50.7.59@o2ib2)
[14532228.465304] Lustre: Skipped 1053 previous similar messages
[14532827.595072] Lustre: oak-OST0123: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[14532827.605758] Lustre: Skipped 1092 previous similar messages
[14533426.618118] Lustre: oak-OST015d: Connection restored to  (at 10.51.13.15@o2ib3)
[14533426.625670] Lustre: Skipped 1003 previous similar messages
[14534026.683350] Lustre: oak-OST015b: Connection restored to  (at 10.51.15.13@o2ib3)
[14534026.690921] Lustre: Skipped 1053 previous similar messages
[14534625.251196] Lustre: oak-OST0157: Connection restored to ab61d733-6278-aecc-ffdb-baa33579a940 (at 10.51.4.33@o2ib3)
[14534625.261805] Lustre: Skipped 1143 previous similar messages
[14535224.238438] Lustre: oak-OST0143: Connection restored to  (at 10.51.13.23@o2ib3)
[14535224.246008] Lustre: Skipped 1204 previous similar messages
[14535826.862806] Lustre: oak-OST011d: Connection restored to 19564d1b-52d9-bf0b-75d7-d97ffab3e50d (at 10.0.3.57@o2ib5)
[14535826.873306] Lustre: Skipped 1045 previous similar messages
[14536426.201981] Lustre: oak-OST011f: Connection restored to ab61d733-6278-aecc-ffdb-baa33579a940 (at 10.51.4.33@o2ib3)
[14536426.212578] Lustre: Skipped 991 previous similar messages
[14537024.770736] Lustre: oak-OST014f: Connection restored to b8ee58d3-1671-02fb-863e-61099182d3fc (at 10.50.14.9@o2ib2)
[14537024.781462] Lustre: Skipped 1236 previous similar messages
[14537623.660182] Lustre: oak-OST0139: Connection restored to a370fd15-e267-4e78-3e63-ecb806369b9c (at 10.210.12.53@tcp1)
[14537623.670846] Lustre: Skipped 1150 previous similar messages
[14538222.511414] Lustre: oak-OST0115: Connection restored to ab61d733-6278-aecc-ffdb-baa33579a940 (at 10.51.4.33@o2ib3)
[14538222.522015] Lustre: Skipped 1083 previous similar messages
[14538587.640727] Lustre: oak-OST0145: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[14538587.651132] Lustre: Skipped 3 previous similar messages
[14538588.321832] LustreError: 160900:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91ce04665050 x1714948920924544/t0(0) o4->5704685e-b0a9-b5c3-be3f-b278a4395bec@10.210.12.72@tcp1:676/0 lens 488/448 e 0 to 0 dl 1645766286 ref 1 fl Interpret:/0/0 rc 0/0
[14538588.346375] LustreError: 160900:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 3 previous similar messages
[14538588.356441] Lustre: oak-OST0145: Bulk IO write error with 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1), client will retry: rc = -110
[14538588.369894] Lustre: Skipped 3 previous similar messages
[14538757.464663] Lustre: oak-OST012f: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[14538757.475089] Lustre: Skipped 1 previous similar message
[14538772.004149] Lustre: oak-OST0141: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[14538772.014553] Lustre: Skipped 11 previous similar messages
[14538789.294542] Lustre: oak-OST0147: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[14538789.304944] Lustre: Skipped 12 previous similar messages
[14538823.662331] Lustre: oak-OST014f: Connection restored to ef0ca35e-4d7b-24ea-8737-c48b55c0297f (at 10.51.13.24@o2ib3)
[14538823.673021] Lustre: Skipped 1093 previous similar messages
[14539423.243896] Lustre: oak-OST013f: Connection restored to c8c804bb-e728-6f8c-527d-295ebfa95786 (at 10.50.4.35@o2ib2)
[14539423.254499] Lustre: Skipped 1025 previous similar messages
[14540021.801684] Lustre: oak-OST013f: Connection restored to f0baa3b1-f3a1-90de-e3da-c913c1605f85 (at 10.50.13.6@o2ib2)
[14540021.812265] Lustre: Skipped 1644 previous similar messages
[14540621.587249] Lustre: oak-OST011f: Connection restored to b4873b44-db69-85c9-576f-945bfd0ad384 (at 10.0.3.26@o2ib5)
[14540621.597754] Lustre: Skipped 1347 previous similar messages
[14541168.518839] Lustre: oak-OST0131: Client 6d01c866-9ec9-8f84-4f86-ee1db83afc97 (at 10.210.12.61@tcp1) reconnecting
[14541168.529265] Lustre: Skipped 6 previous similar messages
[14541169.189909] LustreError: 243441:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91baed423850 x1715027336325312/t0(0) o4->6d01c866-9ec9-8f84-4f86-ee1db83afc97@10.210.12.61@tcp1:244/0 lens 488/448 e 0 to 0 dl 1645768874 ref 1 fl Interpret:/0/0 rc 0/0
[14541169.195030] Lustre: oak-OST0139: Bulk IO write error with 6d01c866-9ec9-8f84-4f86-ee1db83afc97 (at 10.210.12.61@tcp1), client will retry: rc = -110
[14541169.228434] LustreError: 243441:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 2 previous similar messages
[14541220.786909] Lustre: oak-OST0149: Connection restored to b8aee960-6cca-b8d7-bb46-d1dcde837605 (at 10.50.2.62@o2ib2)
[14541220.797491] Lustre: Skipped 1041 previous similar messages
[14541819.526059] Lustre: oak-OST0153: Connection restored to  (at 10.50.7.2@o2ib2)
[14541819.533456] Lustre: Skipped 1222 previous similar messages
[14542419.354624] Lustre: oak-OST0145: Connection restored to 45af99cf-0929-72df-1da2-a7dc0bc2a5cf (at 10.51.2.17@o2ib3)
[14542419.365208] Lustre: Skipped 1106 previous similar messages
[14543019.289321] Lustre: oak-OST0153: Connection restored to 13782cfa-f64e-3b30-9b19-eac54c74cba1 (at 10.51.15.10@o2ib3)
[14543019.300011] Lustre: Skipped 926 previous similar messages
[14543618.917440] Lustre: oak-OST015d: Connection restored to  (at 10.51.16.1@o2ib3)
[14543618.924919] Lustre: Skipped 944 previous similar messages
[14543960.517732] Lustre: oak-OST0111: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[14543960.528234] Lustre: Skipped 1 previous similar message
[14543960.631316] LustreError: 21601:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff914933522050 x1714949056790400/t0(0) o4->5704685e-b0a9-b5c3-be3f-b278a4395bec@10.210.12.72@tcp1:24/0 lens 488/448 e 0 to 0 dl 1645771674 ref 1 fl Interpret:/0/0 rc 0/0
[14543960.655560] LustreError: 21601:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 4 previous similar messages
[14543960.665323] Lustre: oak-OST011d: Bulk IO write error with 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1), client will retry: rc = -110
[14543960.678753] Lustre: Skipped 6 previous similar messages
[14543961.199938] LustreError: 243440:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff914e130ba850 x1714949056785664/t0(0) o4->5704685e-b0a9-b5c3-be3f-b278a4395bec@10.210.12.72@tcp1:22/0 lens 488/448 e 0 to 0 dl 1645771672 ref 1 fl Interpret:/0/0 rc 0/0
[14543961.224581] Lustre: oak-OST011d: Bulk IO write error with 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1), client will retry: rc = -110
[14543961.238044] Lustre: Skipped 1 previous similar message
[14544021.710851] LustreError: 21620:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff9137cc79e850 x1714949056776512/t0(0) o4->5704685e-b0a9-b5c3-be3f-b278a4395bec@10.210.12.72@tcp1:19/0 lens 488/448 e 0 to 0 dl 1645771669 ref 1 fl Interpret:/0/0 rc 0/0
[14544021.710999] Lustre: oak-OST015f: Bulk IO write error with 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1), client will retry: rc = -110
[14544021.711000] Lustre: Skipped 2 previous similar messages
[14544021.755373] LustreError: 21620:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[14544039.743680] Lustre: oak-OST015f: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[14544039.754340] Lustre: Skipped 1 previous similar message
[14544039.885489] LustreError: 137-5: oak-OST0148_UUID: not available for connect from 10.210.12.145@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14544126.066146] Lustre: oak-OST0119: Client 6b6cd352-15e2-b5e6-2cf2-86cbf0e9b626 (at 10.210.12.145@tcp1) reconnecting
[14544126.076642] Lustre: Skipped 5 previous similar messages
[14544128.334923] Lustre: oak-OST0123: Client e44e675f-11ba-536d-fbc7-173bafd860e2 (at 10.210.12.46@tcp1) reconnecting
[14544128.345333] Lustre: Skipped 14 previous similar messages
[14544133.521384] Lustre: oak-OST0129: Client 6b6cd352-15e2-b5e6-2cf2-86cbf0e9b626 (at 10.210.12.145@tcp1) reconnecting
[14544133.531930] Lustre: Skipped 25 previous similar messages
[14544142.069874] Lustre: oak-OST012b: Client e44e675f-11ba-536d-fbc7-173bafd860e2 (at 10.210.12.46@tcp1) reconnecting
[14544142.080305] Lustre: Skipped 20 previous similar messages
[14544158.080081] Lustre: oak-OST0137: Client e44e675f-11ba-536d-fbc7-173bafd860e2 (at 10.210.12.46@tcp1) reconnecting
[14544158.090491] Lustre: Skipped 6 previous similar messages
[14544190.053048] Lustre: oak-OST0145: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[14544190.063506] Lustre: Skipped 9 previous similar messages
[14544219.230031] Lustre: oak-OST015d: Connection restored to 6d63cbc4-6143-910c-ba58-90558ade6da5 (at 10.50.12.13@o2ib2)
[14544219.240701] Lustre: Skipped 1202 previous similar messages
[14544280.301160] Lustre: oak-OST012b: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[14544280.311578] Lustre: Skipped 3 previous similar messages
[14544818.147005] Lustre: oak-OST015d: Connection restored to b8aee960-6cca-b8d7-bb46-d1dcde837605 (at 10.50.2.62@o2ib2)
[14544818.157588] Lustre: Skipped 1068 previous similar messages
[14545418.666315] Lustre: oak-OST0143: Connection restored to 1541d13a-3e02-38af-ab9e-9e12a6e9c4d5 (at 10.51.15.14@o2ib3)
[14545418.677013] Lustre: Skipped 867 previous similar messages
[14546017.912808] Lustre: oak-OST014f: Connection restored to  (at 10.0.3.12@o2ib5)
[14546017.920230] Lustre: Skipped 1026 previous similar messages
[14546617.294035] Lustre: oak-OST0143: Connection restored to a6e7d2b7-fc8f-3f38-afc4-932ac9918b00 (at 10.51.15.7@o2ib3)
[14546617.304651] Lustre: Skipped 750 previous similar messages
[14547217.938119] Lustre: oak-OST0115: Connection restored to 13782cfa-f64e-3b30-9b19-eac54c74cba1 (at 10.51.15.10@o2ib3)
[14547217.948806] Lustre: Skipped 731 previous similar messages
[14547821.021359] Lustre: oak-OST0123: Connection restored to 1135e55c-0bfb-485c-1d0c-23617bcae1bd (at 10.50.7.37@o2ib2)
[14547821.031971] Lustre: Skipped 829 previous similar messages
[14548419.579289] Lustre: oak-OST014d: Connection restored to 97a324d4-c834-549a-4b22-6ab961bb5783 (at 10.0.3.86@o2ib5)
[14548419.589810] Lustre: Skipped 914 previous similar messages
[14549019.342020] Lustre: oak-OST0153: Connection restored to 1cc3b6e6-945c-3742-e968-c8b047c536df (at 10.51.13.16@o2ib3)
[14549019.352736] Lustre: Skipped 887 previous similar messages
[14549059.409037] Lustre: oak-OST014f: Client 6d01c866-9ec9-8f84-4f86-ee1db83afc97 (at 10.210.12.61@tcp1) reconnecting
[14549059.527267] LustreError: 162709:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff917909ffc850 x1715027460077696/t0(0) o4->6d01c866-9ec9-8f84-4f86-ee1db83afc97@10.210.12.61@tcp1:604/0 lens 488/448 e 0 to 0 dl 1645776784 ref 1 fl Interpret:/0/0 rc 0/0
[14549059.551712] LustreError: 162709:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 3 previous similar messages
[14549059.561725] Lustre: oak-OST014f: Bulk IO write error with 6d01c866-9ec9-8f84-4f86-ee1db83afc97 (at 10.210.12.61@tcp1), client will retry: rc = -110
[14549059.575194] Lustre: Skipped 2 previous similar messages
[14549060.411114] LustreError: 21584:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff917595c33050 x1715027460182784/t0(0) o4->6d01c866-9ec9-8f84-4f86-ee1db83afc97@10.210.12.61@tcp1:606/0 lens 488/448 e 0 to 0 dl 1645776786 ref 1 fl Interpret:/0/0 rc 0/0
[14549060.418243] Lustre: oak-OST014f: Bulk IO write error with 6d01c866-9ec9-8f84-4f86-ee1db83afc97 (at 10.210.12.61@tcp1), client will retry: rc = -110
[14549060.448934] LustreError: 21584:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 2 previous similar messages
[14549121.256258] LustreError: 228366:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff9150190ad050 x1714924427478464/t0(0) o3->b17226f5-e55e-40a8-722a-0ee72011f267@10.210.12.13@tcp1:600/0 lens 488/440 e 0 to 0 dl 1645776780 ref 1 fl Interpret:/0/0 rc 0/0
[14549121.256392] LustreError: 21606:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff9162d5ac6850 x1714924427478976/t0(0) o4->b17226f5-e55e-40a8-722a-0ee72011f267@10.210.12.13@tcp1:601/0 lens 488/448 e 0 to 0 dl 1645776781 ref 1 fl Interpret:/0/0 rc 0/0
[14549121.256468] Lustre: oak-OST014d: Bulk IO read error with b17226f5-e55e-40a8-722a-0ee72011f267 (at 10.210.12.13@tcp1), client will retry: rc -110
[14549121.256606] Lustre: oak-OST0133: Bulk IO write error with b17226f5-e55e-40a8-722a-0ee72011f267 (at 10.210.12.13@tcp1), client will retry: rc = -110
[14549121.256607] Lustre: Skipped 2 previous similar messages
[14549121.339640] LustreError: 228366:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 1 previous similar message
[14549139.497978] Lustre: oak-OST0133: Client b17226f5-e55e-40a8-722a-0ee72011f267 (at 10.210.12.13@tcp1) reconnecting
[14549139.508402] Lustre: Skipped 2 previous similar messages
[14549226.685633] Lustre: oak-OST0113: Client 6d01c866-9ec9-8f84-4f86-ee1db83afc97 (at 10.210.12.61@tcp1) reconnecting
[14549245.154608] LustreError: 21624:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91f1a0792850 x1715027461417088/t0(0) o4->6d01c866-9ec9-8f84-4f86-ee1db83afc97@10.210.12.61@tcp1:37/0 lens 488/448 e 0 to 0 dl 1645776972 ref 1 fl Interpret:/0/0 rc 0/0
[14549245.179144] Lustre: oak-OST014f: Bulk IO write error with 6d01c866-9ec9-8f84-4f86-ee1db83afc97 (at 10.210.12.61@tcp1), client will retry: rc = -110
[14549245.192574] Lustre: Skipped 1 previous similar message
[14549245.592399] LustreError: 160931:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff91d4f2019850 x1715027461396992/t0(0) o4->6d01c866-9ec9-8f84-4f86-ee1db83afc97@10.210.12.61@tcp1:37/0 lens 488/448 e 0 to 0 dl 1645776972 ref 1 fl Interpret:/2/0 rc 0/0
[14549320.381552] LustreError: 137-5: oak-OST0144_UUID: not available for connect from 10.210.12.61@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14549320.405033] Lustre: oak-OST015d: Client 6d01c866-9ec9-8f84-4f86-ee1db83afc97 (at 10.210.12.61@tcp1) reconnecting
[14549320.415443] Lustre: Skipped 49 previous similar messages
[14549618.128617] Lustre: oak-OST0115: Connection restored to a370fd15-e267-4e78-3e63-ecb806369b9c (at 10.210.12.53@tcp1)
[14549618.139305] Lustre: Skipped 969 previous similar messages
[14549644.111332] Lustre: oak-OST011d: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[14549644.121780] Lustre: Skipped 56 previous similar messages
[14549644.370803] LustreError: 243538:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff915c7e2b4850 x1714949140559552/t0(0) o4->5704685e-b0a9-b5c3-be3f-b278a4395bec@10.210.12.72@tcp1:432/0 lens 488/448 e 0 to 0 dl 1645777367 ref 1 fl Interpret:/0/0 rc 0/0
[14549644.395262] LustreError: 243538:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 7 previous similar messages
[14549644.405374] Lustre: oak-OST011d: Bulk IO write error with 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1), client will retry: rc = -110
[14549644.418808] Lustre: Skipped 10 previous similar messages
[14549644.627572] LustreError: 21606:0:(ldlm_lib.c:3305:target_bulk_io()) @@@ bulk WRITE failed: rc -107  req@ffff91456593d050 x1714949140560192/t0(0) o4->5704685e-b0a9-b5c3-be3f-b278a4395bec@10.210.12.72@tcp1:437/0 lens 488/448 e 0 to 0 dl 1645777372 ref 1 fl Interpret:/2/0 rc 0/0
[14549644.652175] LustreError: 21606:0:(ldlm_lib.c:3305:target_bulk_io()) Skipped 2 previous similar messages
[14550218.475121] Lustre: oak-OST014f: Connection restored to  (at 10.50.7.2@o2ib2)
[14550218.482583] Lustre: Skipped 870 previous similar messages
[14550818.295624] Lustre: oak-OST015b: Connection restored to c23938c2-385b-1d46-2ca6-6894afb16a15 (at 10.210.12.70@tcp1)
[14550818.306504] Lustre: Skipped 893 previous similar messages
[14551416.847127] Lustre: oak-OST0147: Connection restored to 086f92ba-0975-6436-0aed-ed08eba90a64 (at 10.51.2.2@o2ib3)
[14551416.857636] Lustre: Skipped 848 previous similar messages
[14551573.818041] Lustre: oak-OST012f: Client b880b230-b2d3-6e4c-a85d-fbc038732418 (at 10.210.12.44@tcp1) reconnecting
[14551573.828443] Lustre: Skipped 25 previous similar messages
[14552015.452654] Lustre: oak-OST0137: Connection restored to ef8da979-3077-9708-0c8c-a646245f23fe (at 10.50.13.15@o2ib2)
[14552015.463319] Lustre: Skipped 921 previous similar messages
[14552614.834310] Lustre: oak-OST012b: Connection restored to 2b2b5b47-3d88-dfa6-3173-9352bce15dc5 (at 10.50.5.53@o2ib2)
[14552614.844905] Lustre: Skipped 741 previous similar messages
[14553216.500851] Lustre: oak-OST0153: Connection restored to 5849b69e-832f-c197-b5b4-3c67bdd4b699 (at 10.50.8.55@o2ib2)
[14553216.512900] Lustre: Skipped 587 previous similar messages
[14553815.714492] Lustre: oak-OST012b: Connection restored to e10ff487-e8ae-c597-6928-490cf86bc28a (at 10.50.4.28@o2ib2)
[14553815.725086] Lustre: Skipped 754 previous similar messages
[14554416.798561] Lustre: oak-OST0155: Connection restored to eea359e6-6b89-c326-60af-8cb99076065b (at 10.210.12.8@tcp1)
[14554416.809234] Lustre: Skipped 1443 previous similar messages
[14555015.760710] Lustre: oak-OST011b: Connection restored to 0453ccd0-e0b0-0e88-c206-cfc123f6a6e1 (at 10.50.8.64@o2ib2)
[14555015.771363] Lustre: Skipped 602 previous similar messages
[14555616.127345] Lustre: oak-OST0149: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[14555616.138022] Lustre: Skipped 531 previous similar messages
[14556214.944076] Lustre: oak-OST0111: Connection restored to 08cbd602-0c02-703f-1ce5-fb9d8f463217 (at 10.51.12.7@o2ib3)
[14556214.954745] Lustre: Skipped 570 previous similar messages
[14556814.850846] Lustre: oak-OST012b: Connection restored to e64c41d9-97dc-d5e1-1051-e238ba99ebd4 (at 10.50.2.28@o2ib2)
[14556814.861658] Lustre: Skipped 636 previous similar messages
[14557413.827497] Lustre: oak-OST0159: Connection restored to f2a34e5f-2bc1-93d2-dc2f-9f9388c9aa60 (at 10.51.1.68@o2ib3)
[14557413.838074] Lustre: Skipped 632 previous similar messages
[14557660.107259] Lustre: oak-OST013d: Client 5d8bb8d9-e5c7-334f-ec8d-f10452c309b5 (at 10.210.12.66@tcp1) reconnecting
[14557660.118151] Lustre: Skipped 16 previous similar messages
[14557664.100638] Lustre: oak-OST0147: Client 5d8bb8d9-e5c7-334f-ec8d-f10452c309b5 (at 10.210.12.66@tcp1) reconnecting
[14557664.111070] Lustre: Skipped 14 previous similar messages
[14557674.721958] Lustre: oak-OST0125: Client 5d8bb8d9-e5c7-334f-ec8d-f10452c309b5 (at 10.210.12.66@tcp1) reconnecting
[14557674.732370] Lustre: Skipped 13 previous similar messages
[14558012.423624] Lustre: oak-OST0153: Connection restored to ea7323ea-d4bb-2678-29a1-bc8d87c76269 (at 10.210.12.6@tcp1)
[14558012.434211] Lustre: Skipped 665 previous similar messages
[14558611.409460] Lustre: oak-OST0159: Connection restored to d319a0df-b3f5-7028-0643-cf292b8e79b9 (at 10.51.1.5@o2ib3)
[14558611.419952] Lustre: Skipped 797 previous similar messages
[14559214.688797] Lustre: oak-OST015d: Connection restored to 19cddcc8-ef13-c2fa-cbaa-5f2f8a1894f6 (at 10.51.1.57@o2ib3)
[14559214.699407] Lustre: Skipped 772 previous similar messages
[14559817.872477] Lustre: oak-OST0157: Connection restored to 19cddcc8-ef13-c2fa-cbaa-5f2f8a1894f6 (at 10.51.1.57@o2ib3)
[14559817.883093] Lustre: Skipped 694 previous similar messages
[14560417.752891] Lustre: oak-OST0143: Connection restored to 916093a3-51ee-71cf-ab61-9d8121d32b7c (at 10.51.15.11@o2ib3)
[14560417.763561] Lustre: Skipped 873 previous similar messages
[14561019.023525] Lustre: oak-OST0111: Connection restored to f5619dd4-90a7-9bf6-f914-69fce38d355e (at 10.210.12.63@tcp1)
[14561019.034217] Lustre: Skipped 703 previous similar messages
[14561617.820745] Lustre: oak-OST013f: Connection restored to 19564d1b-52d9-bf0b-75d7-d97ffab3e50d (at 10.0.3.57@o2ib5)
[14561617.831252] Lustre: Skipped 856 previous similar messages
[14562219.281319] Lustre: oak-OST0125: Connection restored to a9ab975c-a0d0-5f8a-8d1c-fbe50b233482 (at 10.210.12.71@tcp1)
[14562219.291994] Lustre: Skipped 823 previous similar messages
[14562818.116826] Lustre: oak-OST0135: Connection restored to 10f920aa-fe61-c10e-8ba5-12cbb156184f (at 10.50.2.45@o2ib2)
[14562818.127499] Lustre: Skipped 715 previous similar messages
[14563423.975023] Lustre: oak-OST014f: Connection restored to 632d6174-105f-f820-efb3-94e214802e62 (at 10.51.16.24@o2ib3)
[14563423.985861] Lustre: Skipped 803 previous similar messages
[14564025.383423] Lustre: oak-OST0153: Connection restored to  (at 10.50.14.13@o2ib2)
[14564025.390972] Lustre: Skipped 1003 previous similar messages
[14564629.812508] Lustre: oak-OST0127: Connection restored to c193930f-3dab-28da-0eac-55422998d272 (at 10.51.13.13@o2ib3)
[14564629.823200] Lustre: Skipped 897 previous similar messages
[14565229.529212] Lustre: oak-OST0159: Connection restored to c193930f-3dab-28da-0eac-55422998d272 (at 10.51.13.13@o2ib3)
[14565229.540490] Lustre: Skipped 789 previous similar messages
[14565830.639719] Lustre: oak-OST0125: Connection restored to  (at 10.50.12.17@o2ib2)
[14565830.647277] Lustre: Skipped 806 previous similar messages
[14566429.875316] Lustre: oak-OST0139: Connection restored to 97a324d4-c834-549a-4b22-6ab961bb5783 (at 10.0.3.86@o2ib5)
[14566429.885822] Lustre: Skipped 1194 previous similar messages
[14567034.470119] Lustre: oak-OST0141: Connection restored to eea359e6-6b89-c326-60af-8cb99076065b (at 10.210.12.8@tcp1)
[14567034.480741] Lustre: Skipped 1005 previous similar messages
[14567170.644994] Lustre: oak-OST014d: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[14567170.655461] Lustre: Skipped 1 previous similar message
[14567170.854976] LustreError: 162714:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff916b6b465850 x1714949252374720/t0(0) o4->5704685e-b0a9-b5c3-be3f-b278a4395bec@10.210.12.72@tcp1:640/0 lens 488/448 e 0 to 0 dl 1645794940 ref 1 fl Interpret:/0/0 rc 0/0
[14567170.879441] LustreError: 162714:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 4 previous similar messages
[14567170.889395] Lustre: oak-OST014d: Bulk IO write error with 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1), client will retry: rc = -110
[14567170.902831] Lustre: Skipped 5 previous similar messages
[14567249.411501] LustreError: 137-5: oak-OST0152_UUID: not available for connect from 10.210.12.44@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14567250.448819] LustreError: 137-5: oak-OST0148_UUID: not available for connect from 10.210.12.43@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14567250.466453] LustreError: Skipped 1 previous similar message
[14567337.620192] Lustre: oak-OST0131: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[14567337.630616] Lustre: Skipped 1 previous similar message
[14567341.618236] Lustre: oak-OST015b: Client ae865b82-51e6-6ef5-51e8-057f7a99f1a1 (at 10.210.12.63@tcp1) reconnecting
[14567341.628764] Lustre: Skipped 12 previous similar messages
[14567367.251914] Lustre: oak-OST0143: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[14567367.262323] Lustre: Skipped 62 previous similar messages
[14567384.386716] Lustre: oak-OST015d: haven't heard from client b880b230-b2d3-6e4c-a85d-fbc038732418 (at 10.210.12.44@tcp1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff913ffe52b800, cur 1645795058 expire 1645794908 last 1645794831
[14567633.505157] Lustre: oak-OST0123: Connection restored to 3c8be638-4b95-3b95-0b17-8ea4836f9204 (at 10.50.12.7@o2ib2)
[14567633.515751] Lustre: Skipped 867 previous similar messages
[14568232.219172] Lustre: oak-OST0135: Connection restored to 97a324d4-c834-549a-4b22-6ab961bb5783 (at 10.0.3.86@o2ib5)
[14568232.229678] Lustre: Skipped 909 previous similar messages
[14568834.824223] Lustre: oak-OST011f: Connection restored to ddd6e198-f36f-14d9-41a2-19f9ebbc987c (at 10.51.6.2@o2ib3)
[14568834.834784] Lustre: Skipped 922 previous similar messages
[14569434.695492] Lustre: oak-OST0153: Connection restored to cccd1b41-e9ad-6ea9-1227-d89af4fce67b (at 10.50.16.1@o2ib2)
[14569434.706072] Lustre: Skipped 925 previous similar messages
[14570034.748186] Lustre: oak-OST0155: Connection restored to 55c1e9f1-1a4d-1f72-179c-f4bf6ddb18c7 (at 10.50.14.7@o2ib2)
[14570034.758902] Lustre: Skipped 989 previous similar messages
[14570635.296171] Lustre: oak-OST0153: Connection restored to  (at 10.51.15.4@o2ib3)
[14570635.303698] Lustre: Skipped 786 previous similar messages
[14571234.637056] Lustre: oak-OST015d: Connection restored to c476e010-0b79-ff80-57df-a0bb35969f90 (at 10.51.12.1@o2ib3)
[14571234.647730] Lustre: Skipped 831 previous similar messages
[14571833.334260] Lustre: oak-OST0123: Connection restored to c1128418-a0a6-64ac-063e-f5c6b0d5c65d (at 10.210.12.58@tcp1)
[14571833.344927] Lustre: Skipped 713 previous similar messages
[14572162.714943] Lustre: 201796:0:(client.c:2169:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1645799674/real 1645799674]  req@ffff913ae1fe6300 x1710546930910208/t0(0) o106->oak-OST015d@10.50.14.11@o2ib2:15/16 lens 296/280 e 0 to 1 dl 1645799847 ref 1 fl Rpc:X/0/ffffffff rc 0/-1
[14572162.745058] LustreError: 201796:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) ### client (nid 10.50.14.11@o2ib2) returned error from glimpse AST (req@ffff913ae1fe6300 x1710546930910208 status -107 rc -107), evict it ns: filter-oak-OST015d_UUID lock: ffff91ad71c54800/0xed112d3080341555 lrc: 3/0,0 mode: PW/PW res: [0x576e6:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1875967) flags: 0x40000000020000 nid: 10.50.14.11@o2ib2 remote: 0x82ef63bfb9be553c expref: 14 pid: 243390 timeout: 0 lvb_type: 0
[14572162.792000] LustreError: 201796:0:(ldlm_lockd.c:681:ldlm_handle_ast_error()) Skipped 1 previous similar message
[14572162.802620] LustreError: 138-a: oak-OST015d: A client on nid 10.50.14.11@o2ib2 was evicted due to a lock glimpse callback time out: rc -107
[14572162.815648] LustreError: Skipped 1 previous similar message
[14572162.821685] LustreError: 199244:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 1645799848s: evicting client at 10.50.14.11@o2ib2  ns: filter-oak-OST015d_UUID lock: ffff91ad71c54800/0xed112d3080341555 lrc: 3/0,0 mode: PW/PW res: [0x576e6:0x0:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->1875967) flags: 0x40000000020000 nid: 10.50.14.11@o2ib2 remote: 0x82ef63bfb9be553c expref: 15 pid: 243390 timeout: 0 lvb_type: 0
[14572193.694274] Lustre: oak-OST0135: haven't heard from client 6467b6ff-0a19-4e0d-f526-5840afc1177a (at 10.50.14.11@o2ib2) in 227 seconds. I think it's dead, and I am evicting it. exp ffff913e68fd0000, cur 1645799879 expire 1645799729 last 1645799652
[14572432.930354] Lustre: oak-OST014f: Connection restored to  (at 10.50.7.2@o2ib2)
[14572432.937755] Lustre: Skipped 911 previous similar messages
[14572464.965330] LustreError: 21598:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 2097152(4194304)  req@ffff913c03d9c850 x1716556376225216/t0(0) o3->6bab8d10-30ff-c1d2-bbf5-1fd0884ac647@10.210.12.57@tcp1:608/0 lens 488/440 e 0 to 0 dl 1645800193 ref 1 fl Interpret:/0/0 rc 0/0
[14572464.965643] Lustre: oak-OST015d: Bulk IO read error with 6bab8d10-30ff-c1d2-bbf5-1fd0884ac647 (at 10.210.12.57@tcp1), client will retry: rc -110
[14572464.965644] Lustre: Skipped 1 previous similar message
[14572465.009338] LustreError: 21598:0:(ldlm_lib.c:3371:target_bulk_io()) Skipped 7 previous similar messages
[14572494.503995] LustreError: 137-5: oak-OST015a_UUID: not available for connect from 10.210.12.46@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14572494.521597] LustreError: Skipped 1 previous similar message
[14572582.686068] Lustre: oak-OST015d: Client 6bab8d10-30ff-c1d2-bbf5-1fd0884ac647 (at 10.210.12.57@tcp1) reconnecting
[14572582.696479] Lustre: Skipped 1 previous similar message
[14572587.757310] Lustre: oak-OST011b: Client 5d8bb8d9-e5c7-334f-ec8d-f10452c309b5 (at 10.210.12.66@tcp1) reconnecting
[14572587.767781] Lustre: Skipped 2 previous similar messages
[14572591.937025] Lustre: oak-OST011b: Client e44e675f-11ba-536d-fbc7-173bafd860e2 (at 10.210.12.46@tcp1) reconnecting
[14572591.947583] Lustre: Skipped 27 previous similar messages
[14572599.935197] Lustre: oak-OST012b: Client e44e675f-11ba-536d-fbc7-173bafd860e2 (at 10.210.12.46@tcp1) reconnecting
[14572599.945604] Lustre: Skipped 9 previous similar messages
[14572741.418542] Lustre: oak-OST0137: Client 5d8bb8d9-e5c7-334f-ec8d-f10452c309b5 (at 10.210.12.66@tcp1) reconnecting
[14572741.429207] Lustre: Skipped 3 previous similar messages
[14573032.139810] Lustre: oak-OST0159: Connection restored to e5d4151d-94bb-31d0-aed5-7e54367726dc (at 10.51.4.23@o2ib3)
[14573032.150452] Lustre: Skipped 890 previous similar messages
[14573632.304595] Lustre: oak-OST012d: Connection restored to  (at 10.0.3.12@o2ib5)
[14573632.312020] Lustre: Skipped 663 previous similar messages
[14573915.177787] Lustre: oak-OST0153: Client 7189afb8-8fe5-7418-d14a-4a60d24330d0 (at 10.210.12.48@tcp1) reconnecting
[14573915.188199] Lustre: Skipped 14 previous similar messages
[14573915.695316] LustreError: 243496:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff914eee3a8850 x1715190463957888/t0(0) o4->7189afb8-8fe5-7418-d14a-4a60d24330d0@10.210.12.48@tcp1:602/0 lens 488/448 e 0 to 0 dl 1645801697 ref 1 fl Interpret:/0/0 rc 0/0
[14573915.720195] Lustre: oak-OST0153: Bulk IO write error with 7189afb8-8fe5-7418-d14a-4a60d24330d0 (at 10.210.12.48@tcp1), client will retry: rc = -110
[14573916.694891] LustreError: 160923:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9196a8bab050 x1716556389042560/t0(0) o4->6bab8d10-30ff-c1d2-bbf5-1fd0884ac647@10.210.12.57@tcp1:603/0 lens 488/448 e 0 to 0 dl 1645801698 ref 1 fl Interpret:/0/0 rc 0/0
[14573916.719660] Lustre: oak-OST0145: Bulk IO write error with 6bab8d10-30ff-c1d2-bbf5-1fd0884ac647 (at 10.210.12.57@tcp1), client will retry: rc = -110
[14573972.965149] LustreError: 162716:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff917432cd1050 x1715190463956608/t0(0) o4->7189afb8-8fe5-7418-d14a-4a60d24330d0@10.210.12.48@tcp1:602/0 lens 488/448 e 0 to 0 dl 1645801697 ref 1 fl Interpret:/0/0 rc 0/0
[14573972.965397] Lustre: oak-OST015d: Bulk IO write error with 6bab8d10-30ff-c1d2-bbf5-1fd0884ac647 (at 10.210.12.57@tcp1), client will retry: rc = -110
[14573972.965398] Lustre: Skipped 2 previous similar messages
[14573973.009837] LustreError: 162716:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 4 previous similar messages
[14573994.582862] Lustre: oak-OST014f: Client 7189afb8-8fe5-7418-d14a-4a60d24330d0 (at 10.210.12.48@tcp1) reconnecting
[14573994.593346] Lustre: Skipped 2 previous similar messages
[14574083.520211] Lustre: oak-OST0143: Client 6bab8d10-30ff-c1d2-bbf5-1fd0884ac647 (at 10.210.12.57@tcp1) reconnecting
[14574083.530627] Lustre: Skipped 1 previous similar message
[14574100.714599] Lustre: oak-OST0131: Client 6bab8d10-30ff-c1d2-bbf5-1fd0884ac647 (at 10.210.12.57@tcp1) reconnecting
[14574100.725011] Lustre: Skipped 32 previous similar messages
[14574143.395529] Lustre: oak-OST0135: Client 7189afb8-8fe5-7418-d14a-4a60d24330d0 (at 10.210.12.48@tcp1) reconnecting
[14574143.405946] Lustre: Skipped 12 previous similar messages
[14574232.612170] Lustre: oak-OST0153: Connection restored to 8817ba1c-727b-5352-25f7-4d78dc602696 (at 10.210.12.68@tcp1)
[14574232.622858] Lustre: Skipped 944 previous similar messages
[14574833.970870] Lustre: oak-OST0141: Connection restored to 5de3621d-407c-a4ab-5438-35c12a4afb01 (at 10.50.2.22@o2ib2)
[14574833.981498] Lustre: Skipped 820 previous similar messages
[14575193.543495] LustreError: 162703:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff917052082850 x1715025023172416/t0(0) o4->c191e9a5-bf3b-065b-ff25-5b40e323dcaa@10.210.12.25@tcp1:329/0 lens 488/448 e 0 to 0 dl 1645802934 ref 1 fl Interpret:/0/0 rc 0/0
[14575193.569611] Lustre: oak-OST0153: Bulk IO write error with c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1), client will retry: rc = -110
[14575193.583127] Lustre: Skipped 3 previous similar messages
[14575228.263477] Lustre: oak-OST0153: Client c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1) reconnecting
[14575228.273882] Lustre: Skipped 22 previous similar messages
[14575315.224961] Lustre: oak-OST0141: Client 45f51dc2-2609-0cee-6a15-5d0f3095c69e (at 10.210.12.7@tcp1) reconnecting
[14575315.235347] Lustre: Skipped 4 previous similar messages
[14575367.218813] Lustre: oak-OST0111: Client c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1) reconnecting
[14575367.229230] Lustre: Skipped 27 previous similar messages
[14575432.930297] Lustre: oak-OST0141: Connection restored to 2a7c3bcd-f685-fa09-489d-ff098ebfe3bc (at 10.50.7.44@o2ib2)
[14575432.940959] Lustre: Skipped 791 previous similar messages
[14575516.511025] Lustre: oak-OST0129: Client c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1) reconnecting
[14575516.521430] Lustre: Skipped 2 previous similar messages
[14576033.192397] Lustre: oak-OST0153: Connection restored to f71d1062-f20c-92ad-d3d7-66389b8ec719 (at 10.51.12.13@o2ib3)
[14576033.203072] Lustre: Skipped 768 previous similar messages
[14576631.995140] Lustre: oak-OST0113: Connection restored to 64318772-01c1-e47d-bbca-5ca564224bca (at 10.0.3.42@o2ib5)
[14576632.005645] Lustre: Skipped 874 previous similar messages
[14577180.597922] LustreError: 162714:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff9182bfa4a050 x1716226436841664/t0(0) o4->6b6cd352-15e2-b5e6-2cf2-86cbf0e9b626@10.210.12.145@tcp1:52/0 lens 488/448 e 0 to 0 dl 1645804922 ref 1 fl Interpret:/0/0 rc 0/0
[14577180.598198] Lustre: oak-OST014f: Bulk IO write error with 6b6cd352-15e2-b5e6-2cf2-86cbf0e9b626 (at 10.210.12.145@tcp1), client will retry: rc = -110
[14577180.637315] LustreError: 162714:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 2 previous similar messages
[14577212.118229] Lustre: oak-OST014f: Client 6b6cd352-15e2-b5e6-2cf2-86cbf0e9b626 (at 10.210.12.145@tcp1) reconnecting
[14577233.044804] Lustre: oak-OST012b: Connection restored to f1db0cb0-5cee-ccf9-6484-5189f751ad99 (at 10.51.0.63@o2ib3)
[14577233.055392] Lustre: Skipped 949 previous similar messages
[14577299.359366] Lustre: oak-OST0113: Client 6b6cd352-15e2-b5e6-2cf2-86cbf0e9b626 (at 10.210.12.145@tcp1) reconnecting
[14577299.369907] Lustre: Skipped 10 previous similar messages
[14577832.547258] Lustre: oak-OST015f: Connection restored to 19564d1b-52d9-bf0b-75d7-d97ffab3e50d (at 10.0.3.57@o2ib5)
[14577832.557750] Lustre: Skipped 891 previous similar messages
[14578432.052624] Lustre: oak-OST0159: Connection restored to c193930f-3dab-28da-0eac-55422998d272 (at 10.51.13.13@o2ib3)
[14578432.063325] Lustre: Skipped 1089 previous similar messages
[14579034.625679] Lustre: oak-OST012d: Connection restored to a8586668-443b-0864-1227-22e9ec80d467 (at 10.50.15.8@o2ib2)
[14579034.636280] Lustre: Skipped 815 previous similar messages
[14579635.991718] Lustre: oak-OST0123: Connection restored to 7bd5e25d-92e5-9f8c-22fa-8f4c98d09593 (at 10.50.10.6@o2ib2)
[14579636.002300] Lustre: Skipped 715 previous similar messages
[14580234.588556] Lustre: oak-OST0153: Connection restored to  (at 10.50.7.2@o2ib2)
[14580234.595940] Lustre: Skipped 4318 previous similar messages
[14580833.409527] Lustre: oak-OST0127: Connection restored to 3362c77a-2096-c98e-90ba-a15ed550df3d (at 10.50.2.31@o2ib2)
[14580833.420113] Lustre: Skipped 3342 previous similar messages
[14581432.003903] Lustre: oak-OST0135: Connection restored to 49bc28ad-d0f8-4839-b977-ca940f844247 (at 10.50.4.17@o2ib2)
[14581432.014584] Lustre: Skipped 1777 previous similar messages
[14582032.750005] Lustre: oak-OST014b: Connection restored to 99a1409b-11ff-d63b-e3e5-6044909060f0 (at 10.210.12.129@tcp1)
[14582032.760762] Lustre: Skipped 2062 previous similar messages
[14582089.455840] LustreError: 21583:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff915cf2227850 x1714976763493440/t0(0) o4->5d4d5527-a52f-bc68-278b-aa990b8609d6@10.210.12.59@tcp1:444/0 lens 488/448 e 0 to 0 dl 1645809844 ref 1 fl Interpret:/0/0 rc 0/0
[14582089.456154] Lustre: oak-OST014f: Bulk IO write error with 6b6cd352-15e2-b5e6-2cf2-86cbf0e9b626 (at 10.210.12.145@tcp1), client will retry: rc = -110
[14582089.456156] Lustre: Skipped 2 previous similar messages
[14582089.500546] LustreError: 21583:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 8 previous similar messages
[14582113.401649] LustreError: 162694:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff9170effb4050 x1715190876035136/t0(0) o4->7189afb8-8fe5-7418-d14a-4a60d24330d0@10.210.12.48@tcp1:450/0 lens 488/448 e 0 to 0 dl 1645809850 ref 1 fl Interpret:/0/0 rc 0/0
[14582113.427650] Lustre: oak-OST015f: Bulk IO write error with 7189afb8-8fe5-7418-d14a-4a60d24330d0 (at 10.210.12.48@tcp1), client will retry: rc = -110
[14582113.441079] Lustre: Skipped 8 previous similar messages
[14582122.211779] Lustre: oak-OST014f: Client 6b6cd352-15e2-b5e6-2cf2-86cbf0e9b626 (at 10.210.12.145@tcp1) reconnecting
[14582122.222281] Lustre: Skipped 14 previous similar messages
[14582211.796325] Lustre: oak-OST0113: Client 7189afb8-8fe5-7418-d14a-4a60d24330d0 (at 10.210.12.48@tcp1) reconnecting
[14582211.806745] Lustre: Skipped 4 previous similar messages
[14582216.366760] Lustre: oak-OST0113: Client e44e675f-11ba-536d-fbc7-173bafd860e2 (at 10.210.12.46@tcp1) reconnecting
[14582216.377176] Lustre: Skipped 41 previous similar messages
[14582228.734398] Lustre: oak-OST0153: Client 7189afb8-8fe5-7418-d14a-4a60d24330d0 (at 10.210.12.48@tcp1) reconnecting
[14582228.744814] Lustre: Skipped 55 previous similar messages
[14582254.672588] Lustre: oak-OST0115: Client 5d4d5527-a52f-bc68-278b-aa990b8609d6 (at 10.210.12.59@tcp1) reconnecting
[14582254.683006] Lustre: Skipped 19 previous similar messages
[14582306.527543] Lustre: oak-OST0117: Client 5d4d5527-a52f-bc68-278b-aa990b8609d6 (at 10.210.12.59@tcp1) reconnecting
[14582306.537956] Lustre: Skipped 17 previous similar messages
[14582631.545024] Lustre: oak-OST015b: Connection restored to ee36833d-7a97-4f8f-3fad-12018bb3bd79 (at 10.210.12.9@tcp1)
[14582631.555605] Lustre: Skipped 1433 previous similar messages
[14583230.657694] Lustre: oak-OST0137: Connection restored to  (at 10.50.14.14@o2ib2)
[14583230.665255] Lustre: Skipped 1468 previous similar messages
[14583829.513294] Lustre: oak-OST0155: Connection restored to efa3260b-85f8-753d-2bfc-fbd16b6c6f94 (at 10.50.14.5@o2ib2)
[14583829.523986] Lustre: Skipped 1781 previous similar messages
[14583877.038906] Lustre: oak-OST014d: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[14583877.382419] LustreError: 243456:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91ecd7c9f850 x1714949343741440/t0(0) o4->5704685e-b0a9-b5c3-be3f-b278a4395bec@10.210.12.72@tcp1:23/0 lens 488/448 e 0 to 0 dl 1645811688 ref 1 fl Interpret:/0/0 rc 0/0
[14583877.406763] LustreError: 243456:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 2 previous similar messages
[14583877.417042] Lustre: oak-OST014d: Bulk IO write error with 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1), client will retry: rc = -110
[14583880.740216] LustreError: 253955:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91ef2486a850 x1716556430499968/t0(0) o4->6bab8d10-30ff-c1d2-bbf5-1fd0884ac647@10.210.12.57@tcp1:26/0 lens 488/448 e 0 to 0 dl 1645811691 ref 1 fl Interpret:/0/0 rc 0/0
[14583880.764795] Lustre: oak-OST015d: Bulk IO write error with 6bab8d10-30ff-c1d2-bbf5-1fd0884ac647 (at 10.210.12.57@tcp1), client will retry: rc = -110
[14583932.649065] LustreError: 21613:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff9189c8c89050 x1716556430484992/t0(0) o4->6bab8d10-30ff-c1d2-bbf5-1fd0884ac647@10.210.12.57@tcp1:22/0 lens 488/448 e 0 to 0 dl 1645811687 ref 1 fl Interpret:/0/0 rc 0/0
[14583932.657615] Lustre: oak-OST0145: Bulk IO write error with 6bab8d10-30ff-c1d2-bbf5-1fd0884ac647 (at 10.210.12.57@tcp1), client will retry: rc = -110
[14583932.688123] LustreError: 21613:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[14583957.904255] Lustre: oak-OST0145: Client 6bab8d10-30ff-c1d2-bbf5-1fd0884ac647 (at 10.210.12.57@tcp1) reconnecting
[14583957.914660] Lustre: Skipped 1 previous similar message
[14584044.959854] Lustre: oak-OST0117: Client 6b341d38-3674-15b6-7e5e-137d0b4498c0 (at 10.210.12.40@tcp1) reconnecting
[14584044.970315] Lustre: Skipped 1 previous similar message
[14584060.951223] LustreError: 21616:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk READ  req@ffff915db4167050 x1721916558924416/t0(0) o3->e44e675f-11ba-536d-fbc7-173bafd860e2@10.210.12.46@tcp1:205/0 lens 488/440 e 0 to 0 dl 1645811870 ref 1 fl Interpret:/0/0 rc 0/0
[14584060.975552] Lustre: oak-OST0131: Bulk IO read error with e44e675f-11ba-536d-fbc7-173bafd860e2 (at 10.210.12.46@tcp1), client will retry: rc -110
[14584060.988742] Lustre: Skipped 7 previous similar messages
[14584081.077167] Lustre: oak-OST012b: Client 5704685e-b0a9-b5c3-be3f-b278a4395bec (at 10.210.12.72@tcp1) reconnecting
[14584081.087698] Lustre: Skipped 78 previous similar messages
[14584172.183042] LustreError: 228831:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91b5aa565850 x1716556430839680/t0(0) o4->6bab8d10-30ff-c1d2-bbf5-1fd0884ac647@10.210.12.57@tcp1:256/0 lens 488/448 e 0 to 0 dl 1645811921 ref 1 fl Interpret:/0/0 rc 0/0
[14584172.183298] Lustre: oak-OST015d: Bulk IO write error with 6bab8d10-30ff-c1d2-bbf5-1fd0884ac647 (at 10.210.12.57@tcp1), client will retry: rc = -110
[14584172.183299] Lustre: Skipped 1 previous similar message
[14584172.227798] LustreError: 228831:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 1 previous similar message
[14584193.308481] Lustre: oak-OST015d: Client 6bab8d10-30ff-c1d2-bbf5-1fd0884ac647 (at 10.210.12.57@tcp1) reconnecting
[14584193.319070] Lustre: Skipped 5 previous similar messages
[14584428.803181] Lustre: oak-OST013d: Connection restored to 7e6be936-f24f-e848-1e9a-42d572a1e93c (at 10.51.1.66@o2ib3)
[14584428.813792] Lustre: Skipped 1585 previous similar messages
[14584716.409931] Lustre: oak-OST0159: Client 6d01c866-9ec9-8f84-4f86-ee1db83afc97 (at 10.210.12.61@tcp1) reconnecting
[14584716.420411] Lustre: Skipped 8 previous similar messages
[14584716.795188] LustreError: 243333:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91a20d877850 x1715028011889920/t0(0) o4->6d01c866-9ec9-8f84-4f86-ee1db83afc97@10.210.12.61@tcp1:105/0 lens 488/448 e 0 to 0 dl 1645812525 ref 1 fl Interpret:/0/0 rc 0/0
[14584716.819958] LustreError: 243333:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 2 previous similar messages
[14584716.830077] Lustre: oak-OST0159: Bulk IO write error with 6d01c866-9ec9-8f84-4f86-ee1db83afc97 (at 10.210.12.61@tcp1), client will retry: rc = -110
[14584716.843597] Lustre: Skipped 1 previous similar message
[14584718.706534] LustreError: 127352:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91ef0cc21850 x1715028011941056/t0(0) o4->6d01c866-9ec9-8f84-4f86-ee1db83afc97@10.210.12.61@tcp1:110/0 lens 488/448 e 0 to 0 dl 1645812530 ref 1 fl Interpret:/0/0 rc 0/0
[14584771.007713] LustreError: 228873:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 3145728(4194304)  req@ffff91d4f6d0f850 x1715028011890048/t0(0) o4->6d01c866-9ec9-8f84-4f86-ee1db83afc97@10.210.12.61@tcp1:105/0 lens 488/448 e 0 to 0 dl 1645812525 ref 1 fl Interpret:/0/0 rc 0/0
[14584771.007946] Lustre: oak-OST015d: Bulk IO write error with 6d01c866-9ec9-8f84-4f86-ee1db83afc97 (at 10.210.12.61@tcp1), client will retry: rc = -110
[14584771.007947] Lustre: Skipped 1 previous similar message
[14584771.052510] LustreError: 228873:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 3 previous similar messages
[14584796.520647] LustreError: 137-5: oak-OST0148_UUID: not available for connect from 10.210.12.61@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14584796.538475] LustreError: Skipped 1 previous similar message
[14585030.817008] Lustre: oak-OST0127: Connection restored to 2d7b80e6-0f0d-a2ea-c50b-4f791ee979a6 (at 10.51.16.21@o2ib3)
[14585030.827685] Lustre: Skipped 1007 previous similar messages
[14585630.413590] Lustre: oak-OST0157: Connection restored to 187c5146-6fef-3342-70d1-8f594265901f (at 10.50.10.39@o2ib2)
[14585630.424315] Lustre: Skipped 1491 previous similar messages
[14586228.962323] Lustre: oak-OST0135: Connection restored to 3195013f-4cca-32ee-d13b-67b22cd84ad2 (at 10.50.1.16@o2ib2)
[14586228.972926] Lustre: Skipped 1338 previous similar messages
[14586827.959706] Lustre: oak-OST015d: Connection restored to  (at 10.51.0.66@o2ib3)
[14586827.967187] Lustre: Skipped 1324 previous similar messages
[14587426.680264] Lustre: oak-OST014b: Connection restored to ade84b92-3785-6c92-32b5-43930c0ed191 (at 10.51.6.28@o2ib3)
[14587426.690864] Lustre: Skipped 1260 previous similar messages
[14588025.902508] Lustre: oak-OST0115: Connection restored to 413e394f-c3a1-ac3a-927b-e67762c23767 (at 10.51.1.28@o2ib3)
[14588025.913086] Lustre: Skipped 1727 previous similar messages
[14588604.883444] Lustre: oak-OST0147: Client 6bd9ec22-2a0f-08d2-ad90-fde76028ef12 (at 10.51.0.66@o2ib3) reconnecting
[14588604.893776] Lustre: Skipped 31 previous similar messages
[14588624.833947] Lustre: oak-OST0151: Connection restored to 19564d1b-52d9-bf0b-75d7-d97ffab3e50d (at 10.0.3.57@o2ib5)
[14588624.844523] Lustre: Skipped 2154 previous similar messages
[14588646.100173] Lustre: oak-OST0141: Client 6bd9ec22-2a0f-08d2-ad90-fde76028ef12 (at 10.51.0.66@o2ib3) reconnecting
[14588646.110582] Lustre: Skipped 12 previous similar messages
[14589223.484014] Lustre: oak-OST0121: Connection restored to  (at 10.51.15.1@o2ib3)
[14589223.491492] Lustre: Skipped 1438 previous similar messages
[14589703.495969] Lustre: oak-OST012f: Client c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1) reconnecting
[14589712.077298] Lustre: oak-OST0157: Client c191e9a5-bf3b-065b-ff25-5b40e323dcaa (at 10.210.12.25@tcp1) reconnecting
[14589822.035159] Lustre: oak-OST0123: Connection restored to 4c7fd0e0-6dd1-64d9-b0fe-37379be18245 (at 10.50.3.5@o2ib2)
[14589822.045738] Lustre: Skipped 1994 previous similar messages
[14589851.105454] Lustre: oak-OST0111: Client 97442201-a515-dc4c-10bb-c0f2abdb2d28 (at 10.51.7.15@o2ib3) reconnecting
[14589851.116075] Lustre: Skipped 3 previous similar messages
[14589883.058671] Lustre: oak-OST0111: Client 20b36988-ad2e-3db2-ffc1-28aca3348c17 (at 10.51.7.13@o2ib3) reconnecting
[14589883.069025] Lustre: Skipped 41 previous similar messages
[14589910.952097] LustreError: 137-5: oak-OST012c_UUID: not available for connect from 10.51.7.15@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14589910.969794] LustreError: Skipped 2 previous similar messages
[14589955.542867] Lustre: oak-OST0145: haven't heard from client f82ac34f-4b3a-8ef0-8e11-38edade2effc (at 10.51.7.2@o2ib3) in 217 seconds. I think it's dead, and I am evicting it. exp ffff91d43904dc00, cur 1645817684 expire 1645817534 last 1645817467
[14589955.564690] Lustre: Skipped 26 previous similar messages
[14589965.600919] Lustre: oak-OST011b: haven't heard from client f82ac34f-4b3a-8ef0-8e11-38edade2effc (at 10.51.7.2@o2ib3) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91da74e40c00, cur 1645817694 expire 1645817544 last 1645817467
[14589965.622778] Lustre: Skipped 1 previous similar message
[14589998.305968] Lustre: oak-OST015d: Client 1b0e0de8-4056-d02c-bb09-33adedaa9f96 (at 10.210.12.45@tcp1) reconnecting
[14589998.305969] Lustre: oak-OST0155: Client 1b0e0de8-4056-d02c-bb09-33adedaa9f96 (at 10.210.12.45@tcp1) reconnecting
[14589998.305971] Lustre: Skipped 15 previous similar messages
[14589998.612069] LustreError: 243438:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91535f961850 x1715096766334848/t0(0) o4->1b0e0de8-4056-d02c-bb09-33adedaa9f96@10.210.12.45@tcp1:117/0 lens 488/448 e 0 to 0 dl 1645817822 ref 1 fl Interpret:/0/0 rc 0/0
[14589998.636662] Lustre: oak-OST015d: Bulk IO write error with 1b0e0de8-4056-d02c-bb09-33adedaa9f96 (at 10.210.12.45@tcp1), client will retry: rc = -110
[14589998.650091] Lustre: Skipped 3 previous similar messages
[14589999.143781] LustreError: 21596:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff91384e512850 x1715096766334720/t0(0) o4->1b0e0de8-4056-d02c-bb09-33adedaa9f96@10.210.12.45@tcp1:116/0 lens 488/448 e 0 to 0 dl 1645817821 ref 1 fl Interpret:/0/0 rc 0/0
[14589999.168112] LustreError: 21596:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 3 previous similar messages
[14590062.102764] LustreError: 21604:0:(ldlm_lib.c:3371:target_bulk_io()) @@@ truncated bulk READ 3145728(4194304)  req@ffff9137d79e4850 x1715096766313088/t0(0) o3->1b0e0de8-4056-d02c-bb09-33adedaa9f96@10.210.12.45@tcp1:114/0 lens 488/440 e 0 to 0 dl 1645817819 ref 1 fl Interpret:/0/0 rc 0/0
[14590062.128493] Lustre: oak-OST0123: Bulk IO read error with 1b0e0de8-4056-d02c-bb09-33adedaa9f96 (at 10.210.12.45@tcp1), client will retry: rc -110
[14590062.141844] Lustre: Skipped 3 previous similar messages
[14590078.798066] LustreError: 137-5: oak-OST015a_UUID: not available for connect from 10.210.12.46@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14590084.653462] LustreError: 137-5: oak-OST013c_UUID: not available for connect from 10.51.7.1@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14590084.653463] LustreError: 137-5: oak-OST013e_UUID: not available for connect from 10.51.7.1@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14590084.688317] LustreError: Skipped 5 previous similar messages
[14590097.226319] LustreError: 137-5: oak-OST0136_UUID: not available for connect from 10.51.7.15@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14590097.226320] LustreError: 137-5: oak-OST012c_UUID: not available for connect from 10.51.7.15@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14590097.226321] LustreError: 137-5: oak-OST0126_UUID: not available for connect from 10.51.7.15@o2ib3 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14590097.279574] LustreError: Skipped 1 previous similar message
[14590423.851482] Lustre: oak-OST0151: Connection restored to  (at 10.51.13.15@o2ib3)
[14590423.860203] Lustre: Skipped 1981 previous similar messages
[14590464.839544] Lustre: oak-OST014b: Client 43c70e25-3329-f639-f581-cb1ed49d9949 (at 10.210.12.42@tcp1) reconnecting
[14590464.849951] Lustre: Skipped 170 previous similar messages
[14590465.627134] LustreError: 229136:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff916f846af850 x1714985241021568/t0(0) o4->43c70e25-3329-f639-f581-cb1ed49d9949@10.210.12.42@tcp1:582/0 lens 504/448 e 0 to 0 dl 1645818287 ref 1 fl Interpret:/0/0 rc 0/0
[14590465.651867] Lustre: oak-OST014b: Bulk IO write error with 43c70e25-3329-f639-f581-cb1ed49d9949 (at 10.210.12.42@tcp1), client will retry: rc = -110
[14590465.665403] Lustre: Skipped 4 previous similar messages
[14590467.844747] LustreError: 162712:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff916611446850 x1714985241090752/t0(0) o4->43c70e25-3329-f639-f581-cb1ed49d9949@10.210.12.42@tcp1:587/0 lens 488/448 e 0 to 0 dl 1645818292 ref 1 fl Interpret:/0/0 rc 0/0
[14590470.357832] Lustre: oak-OST0143: Bulk IO write error with 233f7470-b0ae-e43d-b32e-99c85e344dbd (at 10.210.12.17@tcp1), client will retry: rc = -110
[14590470.371917] Lustre: Skipped 6 previous similar messages
[14590544.636184] LustreError: 137-5: oak-OST0154_UUID: not available for connect from 10.210.12.42@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14591023.226561] Lustre: oak-OST0137: Connection restored to e65f39f4-de96-417e-e82a-947503a05dd5 (at 10.51.1.40@o2ib3)
[14591023.237145] Lustre: Skipped 1694 previous similar messages
[14591622.130613] Lustre: oak-OST0117: Connection restored to  (at 10.50.13.8@o2ib2)
[14591622.138101] Lustre: Skipped 1370 previous similar messages
[14592221.114622] Lustre: oak-OST014d: Connection restored to f0baa3b1-f3a1-90de-e3da-c913c1605f85 (at 10.50.13.6@o2ib2)
[14592221.125247] Lustre: Skipped 1659 previous similar messages
[14592819.844828] Lustre: oak-OST0149: Connection restored to 8c7cb508-c6b9-04a0-a468-4f245a92d1d2 (at 10.50.1.69@o2ib2)
[14592819.855514] Lustre: Skipped 1433 previous similar messages
[14593247.102370] Lustre: oak-OST0153: Client 97e0b978-47b3-d181-6e4a-d3ffe3c29d05 (at 10.210.12.9@tcp1) reconnecting
[14593247.112728] Lustre: Skipped 65 previous similar messages
[14593247.318812] LustreError: 162690:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff914cad977050 x1725567416593472/t0(0) o4->97e0b978-47b3-d181-6e4a-d3ffe3c29d05@10.210.12.9@tcp1:351/0 lens 488/448 e 0 to 0 dl 1645821076 ref 1 fl Interpret:/0/0 rc 0/0
[14593247.343210] LustreError: 162690:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 7 previous similar messages
[14593247.353168] Lustre: oak-OST0153: Bulk IO write error with 97e0b978-47b3-d181-6e4a-d3ffe3c29d05 (at 10.210.12.9@tcp1), client will retry: rc = -110
[14593248.065999] LustreError: 162705:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9149d7fe7050 x1725567416593088/t0(0) o4->97e0b978-47b3-d181-6e4a-d3ffe3c29d05@10.210.12.9@tcp1:351/0 lens 488/448 e 0 to 0 dl 1645821076 ref 1 fl Interpret:/0/0 rc 0/0
[14593248.090349] LustreError: 162705:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[14593249.097487] LustreError: 21606:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff916514c11050 x1725567416593088/t0(0) o4->97e0b978-47b3-d181-6e4a-d3ffe3c29d05@10.210.12.9@tcp1:356/0 lens 488/448 e 0 to 0 dl 1645821081 ref 1 fl Interpret:/2/0 rc 0/0
[14593249.113503] Lustre: oak-OST0153: Bulk IO write error with 97e0b978-47b3-d181-6e4a-d3ffe3c29d05 (at 10.210.12.9@tcp1), client will retry: rc = -110
[14593249.113504] Lustre: Skipped 2 previous similar messages
[14593249.140944] LustreError: 21606:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[14593317.844420] LustreError: 162709:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2853619(3902195)  req@ffff917b27de4850 x1725567416647552/t0(0) o4->97e0b978-47b3-d181-6e4a-d3ffe3c29d05@10.210.12.9@tcp1:355/0 lens 488/448 e 0 to 0 dl 1645821080 ref 1 fl Interpret:/0/0 rc 0/0
[14593317.870564] Lustre: oak-OST015f: Bulk IO write error with 97e0b978-47b3-d181-6e4a-d3ffe3c29d05 (at 10.210.12.9@tcp1), client will retry: rc = -110
[14593317.884132] Lustre: Skipped 2 previous similar messages
[14593326.869566] Lustre: oak-OST015f: Client 97e0b978-47b3-d181-6e4a-d3ffe3c29d05 (at 10.210.12.9@tcp1) reconnecting
[14593326.879900] Lustre: Skipped 1 previous similar message
[14593419.184990] Lustre: oak-OST0129: Connection restored to  (at 10.51.12.23@o2ib3)
[14593419.192578] Lustre: Skipped 1285 previous similar messages
[14593443.048130] Lustre: oak-OST0117: haven't heard from client 97e0b978-47b3-d181-6e4a-d3ffe3c29d05 (at 10.210.12.9@tcp1) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91b46cdfe800, cur 1645821180 expire 1645821030 last 1645820953
[14593443.070073] Lustre: Skipped 2 previous similar messages
[14593460.135952] Lustre: oak-OST0111: Client 97e0b978-47b3-d181-6e4a-d3ffe3c29d05 (at 10.210.12.9@tcp1) reconnecting
[14593460.146332] Lustre: Skipped 28 previous similar messages
[14594018.066109] Lustre: oak-OST0147: Connection restored to bfd7c29e-205a-a45e-5add-4648092cd4c2 (at 10.50.6.39@o2ib2)
[14594018.076696] Lustre: Skipped 1369 previous similar messages
[14594616.963486] Lustre: oak-OST011f: Connection restored to ee36833d-7a97-4f8f-3fad-12018bb3bd79 (at 10.210.12.9@tcp1)
[14594616.974153] Lustre: Skipped 1328 previous similar messages
[14595062.027294] Lustre: oak-OST015b: Client 7189afb8-8fe5-7418-d14a-4a60d24330d0 (at 10.210.12.48@tcp1) reconnecting
[14595062.081406] LustreError: 229136:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff9137f74ac050 x1715191415944000/t0(0) o4->7189afb8-8fe5-7418-d14a-4a60d24330d0@10.210.12.48@tcp1:660/0 lens 488/448 e 0 to 0 dl 1645822895 ref 1 fl Interpret:/0/0 rc 0/0
[14595062.105866] LustreError: 229136:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[14595062.115780] Lustre: oak-OST015b: Bulk IO write error with 7189afb8-8fe5-7418-d14a-4a60d24330d0 (at 10.210.12.48@tcp1), client will retry: rc = -110
[14595064.095517] LustreError: 243499:0:(ldlm_lib.c:3356:target_bulk_io()) @@@ Reconnect on bulk WRITE  req@ffff918f78991850 x1715191415959680/t0(0) o4->7189afb8-8fe5-7418-d14a-4a60d24330d0@10.210.12.48@tcp1:665/0 lens 488/448 e 0 to 0 dl 1645822900 ref 1 fl Interpret:/0/0 rc 0/0
[14595064.119958] LustreError: 243499:0:(ldlm_lib.c:3356:target_bulk_io()) Skipped 1 previous similar message
[14595064.120197] Lustre: oak-OST015b: Bulk IO write error with 7189afb8-8fe5-7418-d14a-4a60d24330d0 (at 10.210.12.48@tcp1), client will retry: rc = -110
[14595113.384744] LustreError: 21596:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) @@@ truncated bulk GET 2097152(4194304)  req@ffff9146b3449050 x1715191415943744/t0(0) o4->7189afb8-8fe5-7418-d14a-4a60d24330d0@10.210.12.48@tcp1:660/0 lens 488/448 e 0 to 0 dl 1645822895 ref 1 fl Interpret:/0/0 rc 0/0
[14595113.385040] Lustre: oak-OST013d: Bulk IO write error with 140cbe03-4bf8-8306-6b3f-3f3adb8eb8d5 (at 10.210.12.71@tcp1), client will retry: rc = -110
[14595113.385042] Lustre: Skipped 2 previous similar messages
[14595113.429681] LustreError: 21596:0:(sec.c:2511:sptlrpc_svc_unwrap_bulk()) Skipped 5 previous similar messages
[14595141.253675] LustreError: 137-5: oak-OST014c_UUID: not available for connect from 10.210.12.48@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14595141.274187] Lustre: oak-OST015f: Client 7189afb8-8fe5-7418-d14a-4a60d24330d0 (at 10.210.12.48@tcp1) reconnecting
[14595141.284610] Lustre: Skipped 1 previous similar message
[14595215.596551] Lustre: oak-OST0119: Connection restored to 6ae520d0-72d9-8bd8-648e-50f1afb84bfd (at 10.50.10.4@o2ib2)
[14595215.607128] Lustre: Skipped 1299 previous similar messages
[14595228.819915] Lustre: oak-OST014b: Client 43c70e25-3329-f639-f581-cb1ed49d9949 (at 10.210.12.42@tcp1) reconnecting
[14595228.830329] Lustre: Skipped 3 previous similar messages
[14595814.384092] Lustre: oak-OST011f: Connection restored to 1675443e-d7e9-8f00-7311-925c6fb1e6e2 (at 10.50.9.31@o2ib2)
[14595814.394725] Lustre: Skipped 1825 previous similar messages
[14596413.562624] Lustre: oak-OST0143: Connection restored to  (at 10.51.1.32@o2ib3)
[14596413.570368] Lustre: Skipped 1438 previous similar messages
[14597012.871192] Lustre: oak-OST015d: Connection restored to b06d09e3-c7f9-ed9d-5de1-180a8f296088 (at 10.0.3.5@o2ib5)
[14597012.881603] Lustre: Skipped 1300 previous similar messages
[14597611.451207] Lustre: oak-OST015b: Connection restored to d44b3311-056b-8e67-c2c4-43e28929a299 (at 10.210.12.39@tcp1)
[14597611.461976] Lustre: Skipped 1395 previous similar messages
[14597964.827468] Lustre: oak-OST0111: Client 11a5f679-c52d-ef2d-3d8a-d81b71720c61 (at 10.210.12.31@tcp1) reconnecting
[14597964.837886] Lustre: Skipped 92 previous similar messages
[14598210.989936] Lustre: oak-OST0147: Connection restored to f328dd72-9435-5f0a-0703-7cce3de1cc1e (at 10.50.2.49@o2ib2)
[14598211.000566] Lustre: Skipped 2687 previous similar messages
[14598809.698763] Lustre: oak-OST015d: Connection restored to c977f6f4-35de-1e9a-97fb-439d33a2737d (at 10.0.3.6@o2ib5)
[14598809.709175] Lustre: Skipped 1619 previous similar messages
[14599410.020125] Lustre: oak-OST0141: Connection restored to 0a533dfe-225b-7e43-d25a-448dd2509c55 (at 10.50.8.7@o2ib2)
[14599410.030619] Lustre: Skipped 1622 previous similar messages
[14600008.693659] Lustre: oak-OST014b: Connection restored to c23938c2-385b-1d46-2ca6-6894afb16a15 (at 10.210.12.70@tcp1)
[14600008.704416] Lustre: Skipped 1545 previous similar messages
[14600607.271956] Lustre: oak-OST014f: Connection restored to 2dc703cd-5cca-815b-6dd4-cd3e7d5a0392 (at 10.0.3.49@o2ib5)
[14600607.282558] Lustre: Skipped 1590 previous similar messages
[14600695.481608] Lustre: oak-OST0117: Client 47e61d14-8684-47f7-2fe1-8f9f293344d7 (at 10.210.12.6@tcp1) reconnecting
[14600695.491997] Lustre: Skipped 6 previous similar messages
[14600699.933166] Lustre: oak-OST013f: Client a52b170e-1c40-8c67-003d-ccc0fed95599 (at 10.210.12.67@tcp1) reconnecting
[14600699.943615] Lustre: Skipped 12 previous similar messages
[14601205.869138] Lustre: oak-OST0149: Connection restored to 612334a0-616c-86f1-bbbc-1f12050c0dbf (at 10.50.7.33@o2ib2)
[14601205.879896] Lustre: Skipped 1664 previous similar messages
[14601804.864381] Lustre: oak-OST012f: Connection restored to  (at 10.0.3.17@o2ib5)
[14601804.871769] Lustre: Skipped 1178 previous similar messages
[14602404.536365] Lustre: oak-OST0113: Connection restored to cccd1b41-e9ad-6ea9-1227-d89af4fce67b (at 10.50.16.1@o2ib2)
[14602404.547014] Lustre: Skipped 1042 previous similar messages
[14603003.566862] Lustre: oak-OST012d: Connection restored to fa202a18-aa95-b01f-fab7-7e4269883f98 (at 10.50.16.10@o2ib2)
[14603003.577540] Lustre: Skipped 1163 previous similar messages
[14603602.333513] Lustre: oak-OST0111: Connection restored to 930fa9cb-a616-c251-f068-fc37f44d4ee3 (at 10.0.3.36@o2ib5)
[14603602.344142] Lustre: Skipped 1923 previous similar messages
[14604203.015530] Lustre: oak-OST012b: Connection restored to 9e55d772-67df-9cd1-1540-cceaeb7907dd (at 10.210.12.127@tcp1)
[14604203.026293] Lustre: Skipped 1955 previous similar messages
[14604802.603304] Lustre: oak-OST011b: Connection restored to 2a199385-e7e7-8728-51af-ca75fe3a692e (at 10.50.17.29@o2ib2)
[14604802.614005] Lustre: Skipped 1812 previous similar messages
[14605401.290990] Lustre: oak-OST0113: Connection restored to 98bd9aab-2cbd-95a4-4e29-43c4c3e1c3df (at 10.51.7.11@o2ib3)
[14605401.301573] Lustre: Skipped 1213 previous similar messages
[14606000.046179] Lustre: oak-OST014f: Connection restored to  (at 10.50.6.40@o2ib2)
[14606000.053657] Lustre: Skipped 1339 previous similar messages
[14606598.680925] Lustre: oak-OST0111: Connection restored to  (at 10.50.0.64@o2ib2)
[14606598.688484] Lustre: Skipped 2591 previous similar messages
[14607197.817368] Lustre: oak-OST013f: Connection restored to 89507400-42d7-a037-6f53-cceb900296af (at 10.50.9.43@o2ib2)
[14607197.828092] Lustre: Skipped 1702 previous similar messages
[14607268.062232] LustreError: 137-5: oak-OST0150_UUID: not available for connect from 10.210.12.36@tcp1 (no target). If you are running an HA pair check that the target is mounted on the other server.
[14607336.754277] LustreError: 199244:0:(ldlm_lockd.c:256:expired_lock_main()) ### lock callback timer expired after 149s: evicting client at 10.210.12.36@tcp1  ns: filter-oak-OST0149_UUID lock: ffff91af507f9680/0xed112d30841d046f lrc: 3/0,0 mode: PW/PW res: [0x55c0000402:0x3ace0d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) flags: 0x60000480020020 nid: 10.210.12.36@tcp1 remote: 0xd3054c5a2e4e92b expref: 184 pid: 243356 timeout: 14642840 lvb_type: 0
[14607336.796894] LustreError: 199244:0:(ldlm_lockd.c:256:expired_lock_main()) Skipped 5 previous similar messages
[14607356.277984] Lustre: oak-OST0129: Client 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1) reconnecting
[14607356.288411] Lustre: Skipped 10 previous similar messages
[14607358.274100] Lustre: oak-OST0123: Client 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1) reconnecting
[14607359.273169] Lustre: oak-OST011b: Client 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1) reconnecting
[14607359.283676] Lustre: Skipped 9 previous similar messages
[14607361.274207] Lustre: oak-OST011d: Client 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1) reconnecting
[14607361.284722] Lustre: Skipped 6 previous similar messages
[14607366.793439] Lustre: oak-OST0111: Client 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1) reconnecting
[14607366.803866] Lustre: Skipped 5 previous similar messages
[14607375.791409] Lustre: oak-OST0115: Client 09198db0-ac48-c42e-4de3-f0cdbdb971ff (at 10.210.12.36@tcp1) reconnecting
[14607375.801849] Lustre: Skipped 3 previous similar messages
[14607796.665967] Lustre: oak-OST013b: Connection restored to 028558ee-1ea8-1c9e-54d2-65c3ca73525f (at 10.50.9.41@o2ib2)
[14607796.676553] Lustre: Skipped 1501 previous similar messages
[14608396.171924] Lustre: oak-OST0155: Connection restored to 51ab7e05-78ff-bf41-8734-92d0767bb7ee (at 10.50.7.21@o2ib2)
[14608396.182507] Lustre: Skipped 1227 previous similar messages
[14608997.093842] Lustre: oak-OST0133: Connection restored to 080fbf36-50e2-e037-d579-97f7ea4dd6a6 (at 10.210.12.51@tcp1)
[14608997.104511] Lustre: Skipped 1042 previous similar messages
[14609596.851413] Lustre: oak-OST0141: Connection restored to a06ac670-6be7-72df-b6f8-2efe92eb96fe (at 10.210.12.67@tcp1)
[14609596.862256] Lustre: Skipped 1416 previous similar messages
[14610195.399844] Lustre: oak-OST0157: Connection restored to f5619dd4-90a7-9bf6-f914-69fce38d355e (at 10.210.12.63@tcp1)
[14610195.410760] Lustre: Skipped 987 previous similar messages
[14610794.985016] Lustre: oak-OST014b: Connection restored to 292f620a-d471-57d6-d1f1-eb0b0867c0bd (at 10.210.12.43@tcp1)
[14610794.995734] Lustre: Skipped 1200 previous similar messages
[14611247.937092] Lustre: oak-OST015b: Export ffff91cafe528400 already connecting from 10.51.15.3@o2ib3
[14611252.924179] Lustre: oak-OST015b: Export ffff91cafe528400 already connecting from 10.51.15.3@o2ib3
[14611257.938143] Lustre: oak-OST015b: Export ffff91cafe528400 already connecting from 10.51.15.3@o2ib3
[14611260.771825] Lustre: oak-OST015b: Export ffff91962a7d7c00 already connecting from 10.50.16.16@o2ib2
[14611265.763819] Lustre: oak-OST015b: Export ffff91962a7d7c00 already connecting from 10.50.16.16@o2ib2
[14611275.771057] Lustre: oak-OST015b: Export ffff91962a7d7c00 already connecting from 10.50.16.16@o2ib2
[14611275.780266] Lustre: Skipped 2 previous similar messages
[14611295.247505] Lustre: oak-OST015b: Export ffff91edac0a8400 already connecting from 10.50.15.13@o2ib2
[14611295.256786] Lustre: Skipped 8 previous similar messages
[14611328.605517] Lustre: oak-OST015b: Export ffff91b2ed917000 already connecting from 10.210.13.37@tcp1
[14611328.614724] Lustre: Skipped 42 previous similar messages
[14611341.653001] INFO: task ll_ost_io00_001:199270 blocked for more than 120 seconds.
[14611341.660748] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[14611341.668909] ll_ost_io00_001 D ffff919623b94200     0 199270      2 0x00000080
[14611341.676492] Call Trace:
[14611341.679234]  [<ffffffffa2f87229>] schedule+0x29/0x70
[14611341.684547]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14611341.691769]  [<ffffffffa28c6d10>] ? wake_up_atomic_t+0x30/0x30
[14611341.697864]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14611341.705166]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14611341.712094]  [<ffffffffc1512eb1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
[14611341.719973]  [<ffffffffa2a28802>] ? kmem_cache_alloc+0x1c2/0x1f0
[14611341.726278]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14611341.733155]  [<ffffffffc1516e8e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611341.740557]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14611341.748206]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611341.755429]  [<ffffffffc1679425>] ofd_trans_start+0x75/0xf0 [ofd]
[14611341.761780]  [<ffffffffc1680701>] ofd_commitrw_write+0xa31/0x1db0 [ofd]
[14611341.768860]  [<ffffffffc1684c8c>] ofd_commitrw+0x47c/0xa50 [ofd]
[14611341.775372]  [<ffffffffc10e159c>] obd_commitrw+0x9c/0x370 [ptlrpc]
[14611341.781848]  [<ffffffffc10e5a42>] tgt_brw_write+0xf02/0x1ae0 [ptlrpc]
[14611341.788540]  [<ffffffffa28e65ff>] ? enqueue_entity+0x2ef/0xbe0
[14611341.794654]  [<ffffffffc107d090>] ? lustre_msg_buf_v2+0x1e0/0x1e0 [ptlrpc]
[14611341.801796]  [<ffffffffc096be79>] ? cfs_binheap_bubble+0x29/0x140 [libcfs]
[14611341.808918]  [<ffffffffa2f85482>] ? mutex_lock+0x12/0x2f
[14611341.814527]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14611341.821687]  [<ffffffffc10c0fa1>] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc]
[14611341.829512]  [<ffffffffc0953bde>] ? ktime_get_real_seconds+0xe/0x10 [libcfs]
[14611341.837668]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14611341.845614]  [<ffffffffc1089555>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
[14611341.852648]  [<ffffffffa28daf40>] ? wake_up_state+0x20/0x20
[14611341.858505]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14611341.865046]  [<ffffffffc108f570>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
[14611341.872686]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14611341.877825]  [<ffffffffa28c5b50>] ? insert_kthread_work+0x40/0x40
[14611341.884181]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14611341.890878]  [<ffffffffa28c5b50>] ? insert_kthread_work+0x40/0x40
[14611341.897233] INFO: task ll_ost_io00_002:199271 blocked for more than 120 seconds.
[14611341.904874] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[14611341.912937] ll_ost_io00_002 D ffff919647bf2100     0 199271      2 0x00000080
[14611341.920375] Call Trace:
[14611341.923085]  [<ffffffffa2f87229>] schedule+0x29/0x70
[14611341.928349]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14611341.935581]  [<ffffffffa28c6d10>] ? wake_up_atomic_t+0x30/0x30
[14611341.941685]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14611341.949046]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14611341.955908]  [<ffffffffc1512eb1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
[14611341.963782]  [<ffffffffa2a28802>] ? kmem_cache_alloc+0x1c2/0x1f0
[14611341.970052]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14611341.976940]  [<ffffffffc1516e8e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611341.984366]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14611341.992022]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611341.999235]  [<ffffffffc1679425>] ofd_trans_start+0x75/0xf0 [ofd]
[14611342.005743]  [<ffffffffc1680701>] ofd_commitrw_write+0xa31/0x1db0 [ofd]
[14611342.012686]  [<ffffffffc1684c8c>] ofd_commitrw+0x47c/0xa50 [ofd]
[14611342.019023]  [<ffffffffc10e159c>] obd_commitrw+0x9c/0x370 [ptlrpc]
[14611342.025507]  [<ffffffffc10e5a42>] tgt_brw_write+0xf02/0x1ae0 [ptlrpc]
[14611342.032195]  [<ffffffffa28e65ff>] ? enqueue_entity+0x2ef/0xbe0
[14611342.038304]  [<ffffffffc107d090>] ? lustre_msg_buf_v2+0x1e0/0x1e0 [ptlrpc]
[14611342.045452]  [<ffffffffc096be79>] ? cfs_binheap_bubble+0x29/0x140 [libcfs]
[14611342.052591]  [<ffffffffc096c036>] ? cfs_binheap_relocate+0xa6/0x1e0 [libcfs]
[14611342.059922]  [<ffffffffc1037770>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
[14611342.067423]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14611342.074668]  [<ffffffffc10c0fa1>] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc]
[14611342.082667]  [<ffffffffc0953bde>] ? ktime_get_real_seconds+0xe/0x10 [libcfs]
[14611342.090429]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14611342.098724]  [<ffffffffc1089555>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
[14611342.105788]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14611342.112331]  [<ffffffffc108f570>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
[14611342.120000]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14611342.125536]  [<ffffffffa28c5b50>] ? insert_kthread_work+0x40/0x40
[14611342.132048]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14611342.138859]  [<ffffffffa28c5b50>] ? insert_kthread_work+0x40/0x40
[14611342.145227] INFO: task ll_ost_io01_000:199272 blocked for more than 120 seconds.
[14611342.152869] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[14611342.160945] ll_ost_io01_000 D ffff919647bf0000     0 199272      2 0x00000080
[14611342.168492] Call Trace:
[14611342.171208]  [<ffffffffa2f87229>] schedule+0x29/0x70
[14611342.176458]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14611342.183589]  [<ffffffffa28c6d10>] ? wake_up_atomic_t+0x30/0x30
[14611342.189677]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14611342.197004]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14611342.203866]  [<ffffffffc1512eb1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
[14611342.211687]  [<ffffffffa2a28802>] ? kmem_cache_alloc+0x1c2/0x1f0
[14611342.217946]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14611342.224819]  [<ffffffffc1516e8e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611342.232214]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14611342.239868]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611342.247110]  [<ffffffffc1679425>] ofd_trans_start+0x75/0xf0 [ofd]
[14611342.253466]  [<ffffffffc1680701>] ofd_commitrw_write+0xa31/0x1db0 [ofd]
[14611342.260331]  [<ffffffffc1684c8c>] ofd_commitrw+0x47c/0xa50 [ofd]
[14611342.266644]  [<ffffffffc10e159c>] obd_commitrw+0x9c/0x370 [ptlrpc]
[14611342.273099]  [<ffffffffc10e5a42>] tgt_brw_write+0xf02/0x1ae0 [ptlrpc]
[14611342.279786]  [<ffffffffa28e65ff>] ? enqueue_entity+0x2ef/0xbe0
[14611342.285878]  [<ffffffffc096be79>] ? cfs_binheap_bubble+0x29/0x140 [libcfs]
[14611342.293001]  [<ffffffffc096c036>] ? cfs_binheap_relocate+0xa6/0x1e0 [libcfs]
[14611342.300322]  [<ffffffffc1037770>] ? target_send_reply_msg+0x170/0x170 [ptlrpc]
[14611342.307822]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14611342.314969]  [<ffffffffc10c0fa1>] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc]
[14611342.322791]  [<ffffffffc0953bde>] ? ktime_get_real_seconds+0xe/0x10 [libcfs]
[14611342.330106]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14611342.338032]  [<ffffffffc1089555>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
[14611342.345060]  [<ffffffffa28daf40>] ? wake_up_state+0x20/0x20
[14611342.350909]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14611342.357458]  [<ffffffffc108f570>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
[14611342.365104]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14611342.370240]  [<ffffffffa28c5b50>] ? insert_kthread_work+0x40/0x40
[14611342.376586]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14611342.383284]  [<ffffffffa28c5b50>] ? insert_kthread_work+0x40/0x40
[14611342.389634] INFO: task ll_ost_io01_002:199274 blocked for more than 120 seconds.
[14611342.397279] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[14611342.405346] ll_ost_io01_002 D ffff919647bf1080     0 199274      2 0x00000080
[14611342.412771] Call Trace:
[14611342.415482]  [<ffffffffa2f87229>] schedule+0x29/0x70
[14611342.420712]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14611342.427827]  [<ffffffffa28c6d10>] ? wake_up_atomic_t+0x30/0x30
[14611342.433908]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14611342.441203]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14611342.447994]  [<ffffffffc1512eb1>] ? osd_declare_xattr_set+0xf1/0x3a0 [osd_ldiskfs]
[14611342.455803]  [<ffffffffa2a28802>] ? kmem_cache_alloc+0x1c2/0x1f0
[14611342.462058]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14611342.468932]  [<ffffffffc1516e8e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611342.476370]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14611342.484063]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611342.491278]  [<ffffffffc1679425>] ofd_trans_start+0x75/0xf0 [ofd]
[14611342.497622]  [<ffffffffc1680701>] ofd_commitrw_write+0xa31/0x1db0 [ofd]
[14611342.504531]  [<ffffffffc1684c8c>] ofd_commitrw+0x47c/0xa50 [ofd]
[14611342.510841]  [<ffffffffc10e159c>] obd_commitrw+0x9c/0x370 [ptlrpc]
[14611342.517299]  [<ffffffffc10e5a42>] tgt_brw_write+0xf02/0x1ae0 [ptlrpc]
[14611342.523990]  [<ffffffffa28e65ff>] ? enqueue_entity+0x2ef/0xbe0
[14611342.530104]  [<ffffffffc107d090>] ? lustre_msg_buf_v2+0x1e0/0x1e0 [ptlrpc]
[14611342.537239]  [<ffffffffc096be79>] ? cfs_binheap_bubble+0x29/0x140 [libcfs]
[14611342.544355]  [<ffffffffa2f85482>] ? mutex_lock+0x12/0x2f
[14611342.549962]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14611342.557119]  [<ffffffffc10c0fa1>] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc]
[14611342.564938]  [<ffffffffc0953bde>] ? ktime_get_real_seconds+0xe/0x10 [libcfs]
[14611342.572255]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14611342.580183]  [<ffffffffc1089555>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
[14611342.587218]  [<ffffffffa28d3143>] ? __wake_up+0x13/0x20
[14611342.592717]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14611342.599256]  [<ffffffffc108f570>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
[14611342.606895]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14611342.612025]  [<ffffffffa28c5b50>] ? insert_kthread_work+0x40/0x40
[14611342.618370]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14611342.625055]  [<ffffffffa28c5b50>] ? insert_kthread_work+0x40/0x40
[14611342.631419] INFO: task ll_ost01_003:200581 blocked for more than 120 seconds.
[14611342.638800] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[14611342.646866] ll_ost01_003    D ffff919688fa9080     0 200581      2 0x00000080
[14611342.654289] Call Trace:
[14611342.657000]  [<ffffffffa29063b2>] ? ktime_get_ts64+0x52/0xf0
[14611342.662915]  [<ffffffffa2f87229>] schedule+0x29/0x70
[14611342.668136]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14611342.675251]  [<ffffffffa28c6d10>] ? wake_up_atomic_t+0x30/0x30
[14611342.681330]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14611342.688627]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14611342.695421]  [<ffffffffc154c2c0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
[14611342.702795]  [<ffffffffa2a28802>] ? kmem_cache_alloc+0x1c2/0x1f0
[14611342.709048]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14611342.715918]  [<ffffffffc1516e8e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611342.723306]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14611342.730950]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611342.738198]  [<ffffffffc10d9383>] tgt_client_data_update+0x303/0x5e0 [ptlrpc]
[14611342.745605]  [<ffffffffc10d9a7b>] tgt_client_new+0x41b/0x610 [ptlrpc]
[14611342.752300]  [<ffffffffc1675453>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
[14611342.758833]  [<ffffffffc103e486>] target_handle_connect+0xec6/0x2bf0 [ptlrpc]
[14611342.766213]  [<ffffffffa296b0c5>] ? tracing_is_on+0x15/0x30
[14611342.772038]  [<ffffffffa296ed9d>] ? tracing_record_cmdline+0x1d/0x120
[14611342.778725]  [<ffffffffa29775ab>] ? probe_sched_wakeup+0x2b/0xa0
[14611342.784980]  [<ffffffffa28d7200>] ? check_preempt_curr+0x90/0xa0
[14611342.791262]  [<ffffffffc10e72ba>] tgt_request_handle+0x4fa/0x1570 [ptlrpc]
[14611342.798408]  [<ffffffffc10c0fa1>] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc]
[14611342.806214]  [<ffffffffa290792f>] ? __getnstimeofday64+0x3f/0xd0
[14611342.812490]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14611342.820421]  [<ffffffffc1089555>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
[14611342.827448]  [<ffffffffa28d3143>] ? __wake_up+0x13/0x20
[14611342.832947]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14611342.839482]  [<ffffffffc108f570>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
[14611342.847120]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14611342.852246]  [<ffffffffa28c5b50>] ? insert_kthread_work+0x40/0x40
[14611342.858583]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14611342.865265]  [<ffffffffa28c5b50>] ? insert_kthread_work+0x40/0x40
[14611342.871610] INFO: task ll_ost00_005:201796 blocked for more than 120 seconds.
[14611342.878984] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[14611342.887052] ll_ost00_005    D ffff91f18a30b180     0 201796      2 0x00000080
[14611342.894476] Call Trace:
[14611342.897188]  [<ffffffffa2f87229>] schedule+0x29/0x70
[14611342.902413]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14611342.909532]  [<ffffffffa28c6d10>] ? wake_up_atomic_t+0x30/0x30
[14611342.915613]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14611342.922909]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14611342.929698]  [<ffffffffc154c2c0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
[14611342.937076]  [<ffffffffa2a28802>] ? kmem_cache_alloc+0x1c2/0x1f0
[14611342.943333]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14611342.950199]  [<ffffffffc1516e8e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611342.957585]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14611342.965225]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611342.972460]  [<ffffffffc10d8bc0>] tgt_server_data_update+0x3c0/0x510 [ptlrpc]
[14611342.979869]  [<ffffffffc10d9f0d>] tgt_client_del+0x29d/0x6b0 [ptlrpc]
[14611342.986562]  [<ffffffffc16715ec>] ofd_obd_disconnect+0x1ac/0x220 [ofd]
[14611342.993354]  [<ffffffffc1040355>] target_handle_disconnect+0x1a5/0x470 [ptlrpc]
[14611343.000935]  [<ffffffffc10e0738>] tgt_disconnect+0x58/0x170 [ptlrpc]
[14611343.007556]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14611343.014697]  [<ffffffffc10c0fa1>] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc]
[14611343.022516]  [<ffffffffc0953bde>] ? ktime_get_real_seconds+0xe/0x10 [libcfs]
[14611343.029833]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14611343.037753]  [<ffffffffc1089555>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
[14611343.044784]  [<ffffffffa28d3143>] ? __wake_up+0x13/0x20
[14611343.050282]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14611343.056819]  [<ffffffffc108f570>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
[14611343.064453]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14611343.069582]  [<ffffffffa28c5b50>] ? insert_kthread_work+0x40/0x40
[14611343.075920]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14611343.082602]  [<ffffffffa28c5b50>] ? insert_kthread_work+0x40/0x40
[14611343.088941] INFO: task ll_ost00_008:203287 blocked for more than 120 seconds.
[14611343.096321] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[14611343.104387] ll_ost00_008    D ffff91f15eff5280     0 203287      2 0x00000080
[14611343.111812] Call Trace:
[14611343.114522]  [<ffffffffa2f87229>] schedule+0x29/0x70
[14611343.119739]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14611343.126861]  [<ffffffffa28c6d10>] ? wake_up_atomic_t+0x30/0x30
[14611343.132940]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14611343.140233]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14611343.147011]  [<ffffffffc1538530>] ? osd_declare_write+0x350/0x490 [osd_ldiskfs]
[14611343.154559]  [<ffffffffa2a28802>] ? kmem_cache_alloc+0x1c2/0x1f0
[14611343.160812]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14611343.167682]  [<ffffffffc1516e8e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611343.175066]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14611343.182705]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611343.189916]  [<ffffffffc1679425>] ofd_trans_start+0x75/0xf0 [ofd]
[14611343.196262]  [<ffffffffc167ca73>] ofd_attr_set+0x4b3/0xb90 [ofd]
[14611343.202530]  [<ffffffffc1668dcd>] ofd_setattr_hdl+0x31d/0x940 [ofd]
[14611343.209106]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14611343.216292]  [<ffffffffc10c0fa1>] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc]
[14611343.224147]  [<ffffffffc0953bde>] ? ktime_get_real_seconds+0xe/0x10 [libcfs]
[14611343.231505]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14611343.239463]  [<ffffffffc1089555>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
[14611343.246523]  [<ffffffffa28daf40>] ? wake_up_state+0x20/0x20
[14611343.252401]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14611343.258972]  [<ffffffffc108f570>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
[14611343.266646]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14611343.271815]  [<ffffffffa28c5b50>] ? insert_kthread_work+0x40/0x40
[14611343.278188]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14611343.284902]  [<ffffffffa28c5b50>] ? insert_kthread_work+0x40/0x40
[14611343.291274] INFO: task ll_ost01_010:203475 blocked for more than 120 seconds.
[14611343.298682] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[14611343.306783] ll_ost01_010    D ffff9195e87bd280     0 203475      2 0x00000080
[14611343.314309] Call Trace:
[14611343.317034]  [<ffffffffa2a27479>] ? ___slab_alloc+0x229/0x520
[14611343.323033]  [<ffffffffa2f87229>] schedule+0x29/0x70
[14611343.328263]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14611343.335375]  [<ffffffffa28c6d10>] ? wake_up_atomic_t+0x30/0x30
[14611343.341456]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14611343.348752]  [<ffffffffc096055f>] ? cfs_hash_buckets_realloc+0x1bf/0x690 [libcfs]
[14611343.356476]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14611343.363267]  [<ffffffffc154c2c0>] ? osd_declare_qid+0x200/0x4a0 [osd_ldiskfs]
[14611343.370644]  [<ffffffffa2a28802>] ? kmem_cache_alloc+0x1c2/0x1f0
[14611343.376897]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14611343.383767]  [<ffffffffc1516e8e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611343.391152]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14611343.398798]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611343.406043]  [<ffffffffc10d9383>] tgt_client_data_update+0x303/0x5e0 [ptlrpc]
[14611343.413452]  [<ffffffffc10d9a7b>] tgt_client_new+0x41b/0x610 [ptlrpc]
[14611343.420145]  [<ffffffffc1675453>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
[14611343.426695]  [<ffffffffc103e486>] target_handle_connect+0xec6/0x2bf0 [ptlrpc]
[14611343.434079]  [<ffffffffa296b0c5>] ? tracing_is_on+0x15/0x30
[14611343.439903]  [<ffffffffa296ed9d>] ? tracing_record_cmdline+0x1d/0x120
[14611343.446588]  [<ffffffffa29775ab>] ? probe_sched_wakeup+0x2b/0xa0
[14611343.452839]  [<ffffffffa28d7200>] ? check_preempt_curr+0x90/0xa0
[14611343.459119]  [<ffffffffc10e72ba>] tgt_request_handle+0x4fa/0x1570 [ptlrpc]
[14611343.466267]  [<ffffffffc10c0fa1>] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc]
[14611343.474079]  [<ffffffffa290792f>] ? __getnstimeofday64+0x3f/0xd0
[14611343.480358]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14611343.488278]  [<ffffffffc1089555>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
[14611343.495313]  [<ffffffffa28d3143>] ? __wake_up+0x13/0x20
[14611343.500813]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14611343.507347]  [<ffffffffc108f570>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
[14611343.514982]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14611343.520109]  [<ffffffffa28c5b50>] ? insert_kthread_work+0x40/0x40
[14611343.526449]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14611343.533130]  [<ffffffffa28c5b50>] ? insert_kthread_work+0x40/0x40
[14611343.539471] INFO: task ll_ost01_012:203498 blocked for more than 120 seconds.
[14611343.546848] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[14611343.554917] ll_ost01_012    D ffff91f0dca3a100     0 203498      2 0x00000080
[14611343.562340] Call Trace:
[14611343.565050]  [<ffffffffa2f87229>] schedule+0x29/0x70
[14611343.570270]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14611343.577386]  [<ffffffffa28c6d10>] ? wake_up_atomic_t+0x30/0x30
[14611343.583470]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14611343.590768]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14611343.597556]  [<ffffffffc1538530>] ? osd_declare_write+0x350/0x490 [osd_ldiskfs]
[14611343.605107]  [<ffffffffa2a28802>] ? kmem_cache_alloc+0x1c2/0x1f0
[14611343.611361]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14611343.618232]  [<ffffffffc1516e8e>] ? osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611343.625616]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14611343.633257]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611343.640468]  [<ffffffffc1679425>] ofd_trans_start+0x75/0xf0 [ofd]
[14611343.646807]  [<ffffffffc167ca73>] ofd_attr_set+0x4b3/0xb90 [ofd]
[14611343.653065]  [<ffffffffc1668dcd>] ofd_setattr_hdl+0x31d/0x940 [ofd]
[14611343.659612]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14611343.666761]  [<ffffffffc10c0fa1>] ? ptlrpc_nrs_req_get_nolock0+0xd1/0x170 [ptlrpc]
[14611343.674589]  [<ffffffffc0953bde>] ? ktime_get_real_seconds+0xe/0x10 [libcfs]
[14611343.681905]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14611343.689825]  [<ffffffffc1089555>] ? ptlrpc_wait_event+0xa5/0x360 [ptlrpc]
[14611343.696854]  [<ffffffffa28d3143>] ? __wake_up+0x13/0x20
[14611343.702351]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14611343.708890]  [<ffffffffc108f570>] ? ptlrpc_register_service+0xf80/0xf80 [ptlrpc]
[14611343.716522]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14611343.721648]  [<ffffffffa28c5b50>] ? insert_kthread_work+0x40/0x40
[14611343.727988]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14611343.734673]  [<ffffffffa28c5b50>] ? insert_kthread_work+0x40/0x40
[14611346.240402] LNet: Service thread pid 160896 was inactive for 200.28s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
[14611346.257859] Pid: 160896, comm: ll_ost_io01_051 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14611346.268958] Call Trace:
[14611346.271917]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14611346.279244]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14611346.286559]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14611346.293360]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14611346.300614]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14611346.308301]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611346.315550]  [<ffffffffc1679425>] ofd_trans_start+0x75/0xf0 [ofd]
[14611346.321924]  [<ffffffffc1680701>] ofd_commitrw_write+0xa31/0x1db0 [ofd]
[14611346.329375]  [<ffffffffc1684c8c>] ofd_commitrw+0x47c/0xa50 [ofd]
[14611346.335760]  [<ffffffffc10e159c>] obd_commitrw+0x9c/0x370 [ptlrpc]
[14611346.342274]  [<ffffffffc10e5a42>] tgt_brw_write+0xf02/0x1ae0 [ptlrpc]
[14611346.349219]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14611346.356656]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14611346.364689]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14611346.371430]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14611346.376624]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14611346.383817]  [<ffffffffffffffff>] 0xffffffffffffffff
[14611346.389101] LustreError: dumping log to /tmp/lustre-log.1645839126.160896
[14611346.733379] Pid: 243538, comm: ll_ost_io00_028 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14611346.744531] Call Trace:
[14611346.747272]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14611346.754591]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14611346.761908]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14611346.768713]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14611346.775617]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14611346.783769]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611346.791024]  [<ffffffffc1679425>] ofd_trans_start+0x75/0xf0 [ofd]
[14611346.797580]  [<ffffffffc1680701>] ofd_commitrw_write+0xa31/0x1db0 [ofd]
[14611346.804474]  [<ffffffffc1684c8c>] ofd_commitrw+0x47c/0xa50 [ofd]
[14611346.810761]  [<ffffffffc10e159c>] obd_commitrw+0x9c/0x370 [ptlrpc]
[14611346.817754]  [<ffffffffc10e5a42>] tgt_brw_write+0xf02/0x1ae0 [ptlrpc]
[14611346.825487]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14611346.832691]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14611346.840644]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14611346.847208]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14611346.852359]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14611346.859071]  [<ffffffffffffffff>] 0xffffffffffffffff
[14611346.864331] LNet: Service thread pid 160941 was inactive for 200.29s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
[14611346.881565] LNet: Skipped 1 previous similar message
[14611346.886824] Pid: 160941, comm: ll_ost_io01_092 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14611346.897922] Call Trace:
[14611346.900643]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14611346.907780]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14611346.915093]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14611346.921896]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14611346.928778]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14611346.936448]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611346.943694]  [<ffffffffc1679425>] ofd_trans_start+0x75/0xf0 [ofd]
[14611346.950068]  [<ffffffffc1680701>] ofd_commitrw_write+0xa31/0x1db0 [ofd]
[14611346.956956]  [<ffffffffc1684c8c>] ofd_commitrw+0x47c/0xa50 [ofd]
[14611346.963233]  [<ffffffffc10e159c>] obd_commitrw+0x9c/0x370 [ptlrpc]
[14611346.969739]  [<ffffffffc10e5a42>] tgt_brw_write+0xf02/0x1ae0 [ptlrpc]
[14611346.976488]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14611346.983730]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14611346.991772]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14611346.998421]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14611347.003814]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14611347.010539]  [<ffffffffffffffff>] 0xffffffffffffffff
[14611347.015865] Pid: 21598, comm: ll_ost_io00_101 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14611347.026888] Call Trace:
[14611347.029691]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14611347.036884]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14611347.044201]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14611347.051010]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14611347.057901]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14611347.065565]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611347.072812]  [<ffffffffc1679425>] ofd_trans_start+0x75/0xf0 [ofd]
[14611347.079194]  [<ffffffffc1680701>] ofd_commitrw_write+0xa31/0x1db0 [ofd]
[14611347.086085]  [<ffffffffc1684c8c>] ofd_commitrw+0x47c/0xa50 [ofd]
[14611347.092458]  [<ffffffffc10e159c>] obd_commitrw+0x9c/0x370 [ptlrpc]
[14611347.098971]  [<ffffffffc10e5a42>] tgt_brw_write+0xf02/0x1ae0 [ptlrpc]
[14611347.105721]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14611347.113086]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14611347.121055]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14611347.127661]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14611347.132837]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14611347.139557]  [<ffffffffffffffff>] 0xffffffffffffffff
[14611347.144817] Pid: 160952, comm: ll_ost_io01_103 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14611347.155913] Call Trace:
[14611347.158628]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14611347.165767]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14611347.173089]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14611347.179901]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14611347.186870]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14611347.194555]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611347.201804]  [<ffffffffc1679425>] ofd_trans_start+0x75/0xf0 [ofd]
[14611347.208176]  [<ffffffffc1680701>] ofd_commitrw_write+0xa31/0x1db0 [ofd]
[14611347.215066]  [<ffffffffc1684c8c>] ofd_commitrw+0x47c/0xa50 [ofd]
[14611347.221353]  [<ffffffffc10e159c>] obd_commitrw+0x9c/0x370 [ptlrpc]
[14611347.227865]  [<ffffffffc10e5a42>] tgt_brw_write+0xf02/0x1ae0 [ptlrpc]
[14611347.234605]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14611347.241773]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14611347.249724]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14611347.256395]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14611347.261555]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14611347.268322]  [<ffffffffffffffff>] 0xffffffffffffffff
[14611347.273584] LNet: Service thread pid 203498 was inactive for 200.49s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
[14611347.772695] LNet: Service thread pid 160953 was inactive for 200.67s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
[14611347.785978] LNet: Skipped 18 previous similar messages
[14611347.791431] LustreError: dumping log to /tmp/lustre-log.1645839128.160953
[14611348.795196] LNet: Service thread pid 162705 was inactive for 200.55s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
[14611348.808389] LNet: Skipped 30 previous similar messages
[14611348.813782] LustreError: dumping log to /tmp/lustre-log.1645839129.162705
[14611349.815721] LustreError: dumping log to /tmp/lustre-log.1645839130.21616
[14611351.858747] LNet: Service thread pid 160924 was inactive for 200.53s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
[14611351.871926] LNet: Skipped 12 previous similar messages
[14611351.877319] LustreError: dumping log to /tmp/lustre-log.1645839132.160924
[14611353.901792] LustreError: dumping log to /tmp/lustre-log.1645839134.259129
[14611354.923302] LustreError: dumping log to /tmp/lustre-log.1645839135.243368
[14611356.455579] LNet: Service thread pid 199270 was inactive for 200.30s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
[14611356.468798] LNet: Skipped 14 previous similar messages
[14611356.474194] LustreError: dumping log to /tmp/lustre-log.1645839136.199270
[14611356.966340] LustreError: dumping log to /tmp/lustre-log.1645839137.243449
[14611357.987853] LustreError: dumping log to /tmp/lustre-log.1645839138.160951
[14611359.009387] LustreError: dumping log to /tmp/lustre-log.1645839139.160956
[14611361.052403] LustreError: dumping log to /tmp/lustre-log.1645839141.228522
[14611361.563178] LustreError: dumping log to /tmp/lustre-log.1645839142.259127
[14611362.584684] LustreError: dumping log to /tmp/lustre-log.1645839143.21595
[14611364.116965] LustreError: dumping log to /tmp/lustre-log.1645839144.162700
[14611366.670749] LNet: Service thread pid 162673 was inactive for 200.27s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
[14611366.683934] LNet: Skipped 19 previous similar messages
[14611366.689344] LustreError: dumping log to /tmp/lustre-log.1645839147.162673
[14611368.203146] LustreError: dumping log to /tmp/lustre-log.1645839148.201796
[14611369.224545] LustreError: dumping log to /tmp/lustre-log.1645839149.160915
[14611370.246132] LustreError: dumping log to /tmp/lustre-log.1645839150.162696
[14611370.756820] LustreError: dumping log to /tmp/lustre-log.1645839151.21613
[14611372.289093] LustreError: dumping log to /tmp/lustre-log.1645839152.243525
[14611373.310612] LustreError: dumping log to /tmp/lustre-log.1645839153.228401
[14611374.332135] LustreError: dumping log to /tmp/lustre-log.1645839154.160920
[14611375.353652] LustreError: dumping log to /tmp/lustre-log.1645839155.243339
[14611376.886919] LustreError: dumping log to /tmp/lustre-log.1645839157.243334
[14611377.908436] LustreError: dumping log to /tmp/lustre-log.1645839158.229328
[14611383.525782] LNet: Service thread pid 167638 was inactive for 200.76s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
[14611383.539046] LNet: Skipped 18 previous similar messages
[14611383.544440] LustreError: dumping log to /tmp/lustre-log.1645839164.167638
[14611386.590349] LustreError: dumping log to /tmp/lustre-log.1645839167.243398
[14611387.611856] LustreError: dumping log to /tmp/lustre-log.1645839168.228526
[14611388.633370] LustreError: dumping log to /tmp/lustre-log.1645839169.199272
[14611391.697939] LustreError: dumping log to /tmp/lustre-log.1645839172.21590
[14611392.719441] LustreError: dumping log to /tmp/lustre-log.1645839173.243495
[14611392.999102] Lustre: oak-OST015b: Export ffff91ada4804c00 already connecting from 10.50.9.9@o2ib2
[14611393.008131] Lustre: Skipped 145 previous similar messages
[14611393.740960] LustreError: dumping log to /tmp/lustre-log.1645839174.243542
[14611393.863873] Lustre: oak-OST015f: Connection restored to 3f4eb3b0-1b11-09af-4476-d96bd10ba10a (at 10.51.6.67@o2ib3)
[14611393.874690] Lustre: Skipped 1263 previous similar messages
[14611395.784003] LustreError: dumping log to /tmp/lustre-log.1645839176.243526
[14611397.827030] LustreError: dumping log to /tmp/lustre-log.1645839178.160909
[14611398.848599] LustreError: dumping log to /tmp/lustre-log.1645839179.243496
[14611400.891592] LustreError: dumping log to /tmp/lustre-log.1645839181.127349
[14611403.956137] LustreError: dumping log to /tmp/lustre-log.1645839184.21591
[14611410.085235] LustreError: dumping log to /tmp/lustre-log.1645839190.228521
[14611412.128262] LustreError: dumping log to /tmp/lustre-log.1645839192.243387
[14611413.374216] Lustre: oak-OST015b: haven't heard from client 9537e9b7-a8e1-24d3-92db-6e4c22e74409 (at <unknown>) in 227 seconds. I think it's dead, and I am evicting it. exp ffff91cafe528400, cur 1645839194 expire 1645839044 last 1645838967
[14611416.214334] LNet: Service thread pid 160933 was inactive for 200.53s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
[14611416.227518] LNet: Skipped 30 previous similar messages
[14611416.232909] LustreError: dumping log to /tmp/lustre-log.1645839196.160933
[14611417.746611] LustreError: dumping log to /tmp/lustre-log.1645839198.168272
[14611419.789644] LustreError: dumping log to /tmp/lustre-log.1645839200.200581
[14611421.321929] LustreError: dumping log to /tmp/lustre-log.1645839201.203475
[14611422.343439] LustreError: dumping log to /tmp/lustre-log.1645839202.168273
[14611423.364959] LustreError: dumping log to /tmp/lustre-log.1645839204.243446
[14611426.429509] LustreError: dumping log to /tmp/lustre-log.1645839207.243448
[14611427.451034] LustreError: dumping log to /tmp/lustre-log.1645839208.212522
[14611428.472557] LustreError: dumping log to /tmp/lustre-log.1645839209.21602
[14611430.515577] LustreError: dumping log to /tmp/lustre-log.1645839211.160907
[14611488.742054] LNet: Service thread pid 243386 was inactive for 246.34s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
[14611488.755243] LNet: Skipped 19 previous similar messages
[14611488.760675] LustreError: dumping log to /tmp/lustre-log.1645839269.243386
[14611489.763582] LustreError: dumping log to /tmp/lustre-log.1645839270.248296
[14611493.849641] LustreError: dumping log to /tmp/lustre-log.1645839274.127351
[14611495.892686] LustreError: dumping log to /tmp/lustre-log.1645839276.168274
[14611498.958230] LustreError: dumping log to /tmp/lustre-log.1645839279.160935
[14611499.978768] LustreError: dumping log to /tmp/lustre-log.1645839280.243360
[14611501.000265] LustreError: dumping log to /tmp/lustre-log.1645839281.200584
[14611503.043298] LustreError: dumping log to /tmp/lustre-log.1645839283.228673
[14611509.172399] LustreError: dumping log to /tmp/lustre-log.1645839290.162711
[14611510.193926] LustreError: dumping log to /tmp/lustre-log.1645839291.203476
[14611520.901234] Lustre: oak-OST015b: Export ffff917ff6b19800 already connecting from 10.50.7.33@o2ib2
[14611520.910359] Lustre: Skipped 410 previous similar messages
[14611566.377373] LustreError: dumping log to /tmp/lustre-log.1645839347.160938
[14611568.420412] LustreError: dumping log to /tmp/lustre-log.1645839349.228525
[14611572.507471] LustreError: dumping log to /tmp/lustre-log.1645839353.204913
[14611575.571037] LustreError: dumping log to /tmp/lustre-log.1645839356.21592
[14611582.721652] LustreError: dumping log to /tmp/lustre-log.1645839363.160927
[14611583.743165] LustreError: dumping log to /tmp/lustre-log.1645839364.243383
[14611584.764684] LustreError: dumping log to /tmp/lustre-log.1645839365.228747
[14611635.840549] LNet: Service thread pid 199257 was inactive for 346.07s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
[14611635.853730] LNet: Skipped 28 previous similar messages
[14611635.859118] LustreError: dumping log to /tmp/lustre-log.1645839417.199257
[14611637.884585] LustreError: dumping log to /tmp/lustre-log.1645839419.244100
[14611650.141789] LNet: Service thread pid 162701 was inactive for 346.12s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
[14611650.159071] LNet: Skipped 2 previous similar messages
[14611650.164381] Pid: 162701, comm: ll_ost_io00_069 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14611650.175580] Call Trace:
[14611650.178386]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14611650.185619]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14611650.192947]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14611650.199803]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14611650.206776]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14611650.214473]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611650.221739]  [<ffffffffc1679425>] ofd_trans_start+0x75/0xf0 [ofd]
[14611650.228237]  [<ffffffffc167d8e8>] ofd_object_punch+0x798/0xd90 [ofd]
[14611650.234995]  [<ffffffffc166a7f3>] ofd_punch_hdl+0x4f3/0xa80 [ofd]
[14611650.241442]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14611650.248642]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14611650.256648]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14611650.263220]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14611650.268405]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14611650.275115]  [<ffffffffffffffff>] 0xffffffffffffffff
[14611650.280388] LustreError: dumping log to /tmp/lustre-log.1645839431.162701
[14611650.288064] Pid: 243352, comm: ll_ost01_070 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14611650.298934] Call Trace:
[14611650.301668]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14611650.308832]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14611650.316168]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14611650.322978]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14611650.329873]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14611650.337554]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611650.344809]  [<ffffffffc10d8bc0>] tgt_server_data_update+0x3c0/0x510 [ptlrpc]
[14611650.352268]  [<ffffffffc10d9f0d>] tgt_client_del+0x29d/0x6b0 [ptlrpc]
[14611650.359015]  [<ffffffffc16715ec>] ofd_obd_disconnect+0x1ac/0x220 [ofd]
[14611650.365824]  [<ffffffffc1040355>] target_handle_disconnect+0x1a5/0x470 [ptlrpc]
[14611650.373454]  [<ffffffffc10e0738>] tgt_disconnect+0x58/0x170 [ptlrpc]
[14611650.380141]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14611650.387324]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14611650.395298]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14611650.401880]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14611650.407039]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14611650.413751]  [<ffffffffffffffff>] 0xffffffffffffffff
[14611654.227863] LNet: Service thread pid 243353 was inactive for 346.74s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
[14611654.245117] LNet: Skipped 1 previous similar message
[14611654.250341] Pid: 243353, comm: ll_ost01_071 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14611654.261177] Call Trace:
[14611654.263906]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14611654.271053]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14611654.278371]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14611654.285174]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14611654.292069]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14611654.299761]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611654.307025]  [<ffffffffc10d9383>] tgt_client_data_update+0x303/0x5e0 [ptlrpc]
[14611654.314501]  [<ffffffffc10d9a7b>] tgt_client_new+0x41b/0x610 [ptlrpc]
[14611654.321282]  [<ffffffffc1675453>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
[14611654.327845]  [<ffffffffc103e486>] target_handle_connect+0xec6/0x2bf0 [ptlrpc]
[14611654.335300]  [<ffffffffc10e72ba>] tgt_request_handle+0x4fa/0x1570 [ptlrpc]
[14611654.342516]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14611654.350474]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14611654.357050]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14611654.362209]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14611654.368926]  [<ffffffffffffffff>] 0xffffffffffffffff
[14611654.374212] LustreError: dumping log to /tmp/lustre-log.1645839435.243353
[14611713.475879] LNet: Service thread pid 162680 was inactive for 397.44s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
[14611713.493330] Pid: 162680, comm: ll_ost_io00_048 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14611713.504441] Call Trace:
[14611713.507239]  [<ffffffffa2b97738>] call_rwsem_down_read_failed+0x18/0x30
[14611713.514146]  [<ffffffffc150accc>] osd_read_lock+0x5c/0xe0 [osd_ldiskfs]
[14611713.521188]  [<ffffffffc1682803>] ofd_preprw_write.isra.30+0xd3/0xea0 [ofd]
[14611713.528535]  [<ffffffffc16839ef>] ofd_preprw+0x41f/0x1240 [ofd]
[14611713.534738]  [<ffffffffc10e5809>] tgt_brw_write+0xcc9/0x1ae0 [ptlrpc]
[14611713.541649]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14611713.548841]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14611713.556879]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14611713.563476]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14611713.568672]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14611713.575383]  [<ffffffffffffffff>] 0xffffffffffffffff
[14611713.580695] LustreError: dumping log to /tmp/lustre-log.1645839494.162680
[14611718.583471] Pid: 160893, comm: ll_ost_io01_048 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14611718.594571] Call Trace:
[14611718.597306]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14611718.604487]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14611718.611830]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14611718.618648]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14611718.625530]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14611718.633211]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611718.640463]  [<ffffffffc1680bee>] ofd_commitrw_write+0xf1e/0x1db0 [ofd]
[14611718.647361]  [<ffffffffc1684c8c>] ofd_commitrw+0x47c/0xa50 [ofd]
[14611718.653644]  [<ffffffffc10e159c>] obd_commitrw+0x9c/0x370 [ptlrpc]
[14611718.660168]  [<ffffffffc10e5a42>] tgt_brw_write+0xf02/0x1ae0 [ptlrpc]
[14611718.666934]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14611718.674109]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14611718.682056]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14611718.688627]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14611718.693786]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14611718.700497]  [<ffffffffffffffff>] 0xffffffffffffffff
[14611718.705764] LustreError: dumping log to /tmp/lustre-log.1645839500.160893
[14611727.778120] LustreError: dumping log to /tmp/lustre-log.1645839509.127357
[14611729.820158] LustreError: dumping log to /tmp/lustre-log.1645839511.228748
[14611735.949265] LustreError: dumping log to /tmp/lustre-log.1645839517.243343
[14611740.137094] Lustre: 243555:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply
                    req@ffff9146fbd59850 x1725258682904640/t0(0) o4->5f03f22e-b20a-aa0b-2ceb-1d936803c018@10.0.3.64@o2ib5:681/0 lens 488/448 e 21 to 0 dl 1645839526 ref 2 fl Interpret:/0/0 rc 0/0
[14611740.166186] Lustre: 243555:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 10 previous similar messages
[14611741.056853] Lustre: 27480:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply
                    req@ffff91f197652850 x1725486029679040/t0(0) o1000->oak-MDT0000-mdtlov_UUID@10.0.2.52@o2ib5:682/0 lens 408/4320 e 20 to 0 dl 1645839527 ref 2 fl Interpret:/0/0 rc 0/0
[14611741.085079] Lustre: 27480:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2 previous similar messages
[14611742.134243] Lustre: 243370:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply
                    req@ffff91a613b8b850 x1725486029733440/t0(0) o2->oak-MDT0000-mdtlov_UUID@10.0.2.52@o2ib5:683/0 lens 440/432 e 21 to 0 dl 1645839528 ref 2 fl Interpret:/0/0 rc 0/0
[14611742.162218] Lustre: 243370:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 36 previous similar messages
[14611744.135376] Lustre: 162677:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply
                    req@ffff918a65e93850 x1715035921784896/t0(0) o4->ae865b82-51e6-6ef5-51e8-057f7a99f1a1@10.210.12.63@tcp1:685/0 lens 12680/448 e 21 to 0 dl 1645839530 ref 2 fl Interpret:/0/0 rc 0/0
[14611744.165016] Lustre: 162677:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 15 previous similar messages
[14611748.131660] Lustre: 243374:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply
                    req@ffff91a613b8c850 x1725486030003008/t0(0) o2->oak-MDT0000-mdtlov_UUID@10.0.2.52@o2ib5:689/0 lens 440/432 e 21 to 0 dl 1645839534 ref 2 fl Interpret:/0/0 rc 0/0
[14611748.159684] Lustre: 243374:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 7 previous similar messages
[14611756.134216] Lustre: 162672:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply
                    req@ffff917f779c6850 x1715136308799168/t0(0) o4->cc381d20-202e-f264-43c3-938610d60653@10.210.12.58@tcp1:697/0 lens 3560/448 e 21 to 0 dl 1645839542 ref 2 fl Interpret:/0/0 rc 0/0
[14611756.163558] Lustre: 162672:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 19 previous similar messages
[14611772.135321] Lustre: 160922:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply
                    req@ffff919c01b0b050 x1716226620607552/t0(0) o4->6b6cd352-15e2-b5e6-2cf2-86cbf0e9b626@10.210.12.145@tcp1:713/0 lens 488/448 e 11 to 0 dl 1645839558 ref 2 fl Interpret:/0/0 rc 0/0
[14611772.164676] Lustre: 160922:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 22 previous similar messages
[14611776.502891] Lustre: oak-OST015b: Export ffff914f6e4ce400 already connecting from 10.0.3.30@o2ib5
[14611776.511973] Lustre: Skipped 1646 previous similar messages
[14611787.025128] LustreError: dumping log to /tmp/lustre-log.1645839568.21596
[14611799.969129] Lustre: oak-OST015b: Client db208f47-4bcb-231d-93b0-4917d80e5830 (at 10.50.7.38@o2ib2) reconnecting
[14611799.979472] Lustre: Skipped 6 previous similar messages
[14611801.326370] LustreError: dumping log to /tmp/lustre-log.1645839582.243345
[14611802.171997] Lustre: oak-OST015b: Client 86ee597a-55bd-0d41-5194-7703d84027e9 (at 10.50.6.42@o2ib2) reconnecting
[14611802.182370] Lustre: Skipped 1 previous similar message
[14611804.135564] Lustre: 199263:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply
                    req@ffff91cba8f9a050 x1725474939362816/t0(0) o6->oak-MDT0001-mdtlov_UUID@10.0.2.51@o2ib5:745/0 lens 544/432 e 7 to 0 dl 1645839590 ref 2 fl Interpret:/0/0 rc 0/0
[14611804.163437] Lustre: 199263:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 27 previous similar messages
[14611805.412440] LustreError: dumping log to /tmp/lustre-log.1645839586.243405
[14611807.455658] LustreError: dumping log to /tmp/lustre-log.1645839589.203090
[14611809.392045] Lustre: oak-OST015b: Client 12e39cbb-70bc-e610-53c0-5be581af8812 (at 10.50.5.29@o2ib2) reconnecting
[14611809.402417] Lustre: Skipped 5 previous similar messages
[14611809.498505] LustreError: dumping log to /tmp/lustre-log.1645839591.229565
[14611819.487600] Lustre: oak-OST015b: Client 59b2bb76-d493-5ecd-5109-ae86fc122457 (at 10.0.3.60@o2ib5) reconnecting
[14611819.497836] Lustre: Skipped 4 previous similar messages
[14611836.064107] Lustre: oak-OST015b: Client cc381d20-202e-f264-43c3-938610d60653 (at 10.210.12.58@tcp1) reconnecting
[14611836.074529] Lustre: Skipped 16 previous similar messages
[14611866.703480] LustreError: dumping log to /tmp/lustre-log.1645839648.204455
[14611870.159932] Lustre: oak-OST015b: Client 49d48f1f-d954-f6f5-eece-6b3d4f612d04 (at 10.51.1.32@o2ib3) reconnecting
[14611870.170253] Lustre: Skipped 7 previous similar messages
[14611872.832625] LustreError: dumping log to /tmp/lustre-log.1645839654.160931
[14611873.087983] Lustre: 243543:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply
                    req@ffff917f944ba850 x1723802543273984/t0(0) o4->e107257b-9e9e-9940-59a9-02040f31e2cc@10.50.9.42@o2ib2:59/0 lens 4584/448 e 4 to 0 dl 1645839659 ref 2 fl Interpret:/0/0 rc 0/0
[14611873.117076] Lustre: 243543:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 31 previous similar messages
[14611874.875620] LustreError: dumping log to /tmp/lustre-log.1645839656.201431
[14611876.918655] LustreError: dumping log to /tmp/lustre-log.1645839658.243389
[14611881.005715] LustreError: dumping log to /tmp/lustre-log.1645839662.160895
[14611883.047803] LustreError: dumping log to /tmp/lustre-log.1645839664.229342
[14611948.424890] LNet: Service thread pid 229329 was inactive for 548.68s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
[14611948.438063] LNet: Skipped 20 previous similar messages
[14611948.443453] LustreError: dumping log to /tmp/lustre-log.1645839730.229329
[14611950.467926] LNet: Service thread pid 162678 was inactive for 547.82s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
[14611950.485166] LNet: Skipped 1 previous similar message
[14611950.490378] Pid: 162678, comm: ll_ost_io00_046 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14611950.501470] Call Trace:
[14611950.504200]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14611950.511363]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14611950.518674]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14611950.525481]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14611950.532362]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14611950.540051]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611950.547315]  [<ffffffffc1679425>] ofd_trans_start+0x75/0xf0 [ofd]
[14611950.553701]  [<ffffffffc1680701>] ofd_commitrw_write+0xa31/0x1db0 [ofd]
[14611950.560595]  [<ffffffffc1684c8c>] ofd_commitrw+0x47c/0xa50 [ofd]
[14611950.566896]  [<ffffffffc10e159c>] obd_commitrw+0x9c/0x370 [ptlrpc]
[14611950.573424]  [<ffffffffc10e5a42>] tgt_brw_write+0xf02/0x1ae0 [ptlrpc]
[14611950.580169]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14611950.587338]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14611950.595282]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14611950.601842]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14611950.606997]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14611950.613707]  [<ffffffffffffffff>] 0xffffffffffffffff
[14611950.618956] LustreError: dumping log to /tmp/lustre-log.1645839732.162678
[14611950.626521] Pid: 162687, comm: ll_ost_io00_055 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14611950.637614] Call Trace:
[14611950.640335]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14611950.647482]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14611950.654798]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14611950.661595]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14611950.668475]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14611950.676154]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611950.683397]  [<ffffffffc1680bee>] ofd_commitrw_write+0xf1e/0x1db0 [ofd]
[14611950.690315]  [<ffffffffc1684c8c>] ofd_commitrw+0x47c/0xa50 [ofd]
[14611950.696610]  [<ffffffffc10e159c>] obd_commitrw+0x9c/0x370 [ptlrpc]
[14611950.703154]  [<ffffffffc10e5a42>] tgt_brw_write+0xf02/0x1ae0 [ptlrpc]
[14611950.709904]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14611950.717097]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14611950.725060]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14611950.731642]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14611950.736811]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14611950.743547]  [<ffffffffffffffff>] 0xffffffffffffffff
[14611952.510937] Pid: 168280, comm: ll_ost00_074 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14611952.521953] Call Trace:
[14611952.524728]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14611952.532016]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14611952.539454]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14611952.546363]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14611952.553343]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14611952.561116]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611952.568477]  [<ffffffffc10d9383>] tgt_client_data_update+0x303/0x5e0 [ptlrpc]
[14611952.576077]  [<ffffffffc10d9a7b>] tgt_client_new+0x41b/0x610 [ptlrpc]
[14611952.582965]  [<ffffffffc1675453>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
[14611952.589604]  [<ffffffffc103e486>] target_handle_connect+0xec6/0x2bf0 [ptlrpc]
[14611952.597170]  [<ffffffffc10e72ba>] tgt_request_handle+0x4fa/0x1570 [ptlrpc]
[14611952.604493]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14611952.612575]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14611952.619272]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14611952.624525]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14611952.631331]  [<ffffffffffffffff>] 0xffffffffffffffff
[14611952.636688] LustreError: dumping log to /tmp/lustre-log.1645839734.168280
[14611958.640034] Pid: 127355, comm: ll_ost_io01_120 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14611958.651132] Call Trace:
[14611958.653847]  [<ffffffffa2b97738>] call_rwsem_down_read_failed+0x18/0x30
[14611958.660735]  [<ffffffffc150accc>] osd_read_lock+0x5c/0xe0 [osd_ldiskfs]
[14611958.667634]  [<ffffffffc1682803>] ofd_preprw_write.isra.30+0xd3/0xea0 [ofd]
[14611958.674871]  [<ffffffffc16839ef>] ofd_preprw+0x41f/0x1240 [ofd]
[14611958.681070]  [<ffffffffc10e5809>] tgt_brw_write+0xcc9/0x1ae0 [ptlrpc]
[14611958.687857]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14611958.695032]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14611958.702985]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14611958.709546]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14611958.714702]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14611958.721420]  [<ffffffffffffffff>] 0xffffffffffffffff
[14611958.726702] LustreError: dumping log to /tmp/lustre-log.1645839740.127355
[14611960.683318] Pid: 243346, comm: ll_ost01_064 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14611960.694161] Call Trace:
[14611960.696894]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14611960.704040]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14611960.711353]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14611960.718149]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14611960.725032]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14611960.732708]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14611960.739953]  [<ffffffffc10d9383>] tgt_client_data_update+0x303/0x5e0 [ptlrpc]
[14611960.747431]  [<ffffffffc10d9a7b>] tgt_client_new+0x41b/0x610 [ptlrpc]
[14611960.754184]  [<ffffffffc1675453>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
[14611960.760729]  [<ffffffffc103e486>] target_handle_connect+0xec6/0x2bf0 [ptlrpc]
[14611960.768173]  [<ffffffffc10e72ba>] tgt_request_handle+0x4fa/0x1570 [ptlrpc]
[14611960.775349]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14611960.783304]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14611960.789904]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14611960.795062]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14611960.801781]  [<ffffffffffffffff>] 0xffffffffffffffff
[14611960.807039] LustreError: dumping log to /tmp/lustre-log.1645839742.243346
[14611961.022791] Lustre: oak-OST015b: Client 3b1423ea-f77c-9301-8ee2-822b91cad2d9 (at 10.210.12.10@tcp1) reconnecting
[14611961.033199] Lustre: Skipped 7 previous similar messages
[14611994.672653] Lustre: oak-OST0111: Connection restored to ec6cafa7-2c96-71e9-0dad-24d0eee2b247 (at 10.0.3.37@o2ib5)
[14611994.683146] Lustre: Skipped 1525 previous similar messages
[14612006.012945] Lustre: 160922:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/5), not sending early reply
                    req@ffff91a7dc48b050 x1715136309317760/t0(0) o4->cc381d20-202e-f264-43c3-938610d60653@10.210.12.58@tcp1:193/0 lens 568/448 e 1 to 0 dl 1645839793 ref 2 fl Interpret:/0/0 rc 0/0
[14612006.042130] Lustre: 160922:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 18 previous similar messages
[14612013.801976] LustreError: dumping log to /tmp/lustre-log.1645839795.206858
[14612019.932075] LustreError: dumping log to /tmp/lustre-log.1645839802.228524
[14612024.018149] LustreError: dumping log to /tmp/lustre-log.1645839806.160904
[14612028.103220] LustreError: dumping log to /tmp/lustre-log.1645839810.243376
[14612030.146292] LustreError: dumping log to /tmp/lustre-log.1645839812.204187
[14612032.189290] LustreError: dumping log to /tmp/lustre-log.1645839814.206914
[14612091.437396] LustreError: dumping log to /tmp/lustre-log.1645839873.228752
[14612095.523387] LustreError: dumping log to /tmp/lustre-log.1645839877.253934
[14612099.609502] LustreError: dumping log to /tmp/lustre-log.1645839881.228869
[14612101.652493] LustreError: dumping log to /tmp/lustre-log.1645839883.201742
[14612103.695519] LustreError: dumping log to /tmp/lustre-log.1645839886.253953
[14612105.738544] LustreError: dumping log to /tmp/lustre-log.1645839888.228677
[14612107.781579] LustreError: dumping log to /tmp/lustre-log.1645839890.21623
[14612109.824651] LustreError: dumping log to /tmp/lustre-log.1645839892.243454
[14612111.867659] LustreError: dumping log to /tmp/lustre-log.1645839894.230670
[14612113.186917] Lustre: oak-OST015b: Client c0eb3889-bf82-52e1-04e8-ea1b36a52789 (at 10.51.12.2@o2ib3) reconnecting
[14612113.197234] Lustre: Skipped 6 previous similar messages
[14612164.986556] LustreError: dumping log to /tmp/lustre-log.1645839947.203496
[14612169.072624] LustreError: dumping log to /tmp/lustre-log.1645839951.243335
[14612171.115740] LustreError: dumping log to /tmp/lustre-log.1645839953.243371
[14612175.201730] LustreError: dumping log to /tmp/lustre-log.1645839957.203720
[14612179.287803] LustreError: dumping log to /tmp/lustre-log.1645839961.203777
[14612238.535817] LustreError: dumping log to /tmp/lustre-log.1645840021.243348
[14612240.578865] LustreError: dumping log to /tmp/lustre-log.1645840023.228750
[14612242.622887] LustreError: dumping log to /tmp/lustre-log.1645840025.230167
[14612246.707971] LustreError: dumping log to /tmp/lustre-log.1645840029.243379
[14612250.794028] LNet: Service thread pid 206920 was inactive for 749.72s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
[14612250.811314] LNet: Skipped 4 previous similar messages
[14612250.816629] Pid: 206920, comm: ll_ost00_121 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14612250.827504] Call Trace:
[14612250.830273]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14612250.837507]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14612250.844914]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14612250.851797]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14612250.858747]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14612250.866472]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14612250.873770]  [<ffffffffc10d9383>] tgt_client_data_update+0x303/0x5e0 [ptlrpc]
[14612250.881308]  [<ffffffffc10d9a7b>] tgt_client_new+0x41b/0x610 [ptlrpc]
[14612250.888105]  [<ffffffffc1675453>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
[14612250.894709]  [<ffffffffc103e486>] target_handle_connect+0xec6/0x2bf0 [ptlrpc]
[14612250.902190]  [<ffffffffc10e72ba>] tgt_request_handle+0x4fa/0x1570 [ptlrpc]
[14612250.909439]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14612250.917461]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14612250.924029]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14612250.929198]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14612250.935997]  [<ffffffffffffffff>] 0xffffffffffffffff
[14612250.941359] LustreError: dumping log to /tmp/lustre-log.1645840033.206920
[14612258.966235] Pid: 160943, comm: ll_ost_io01_094 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14612258.977336] Call Trace:
[14612258.980075]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14612258.987223]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14612258.994538]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14612259.001335]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14612259.008217]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14612259.015893]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14612259.023137]  [<ffffffffc1680bee>] ofd_commitrw_write+0xf1e/0x1db0 [ofd]
[14612259.030030]  [<ffffffffc1684c8c>] ofd_commitrw+0x47c/0xa50 [ofd]
[14612259.036304]  [<ffffffffc10e159c>] obd_commitrw+0x9c/0x370 [ptlrpc]
[14612259.042850]  [<ffffffffc10e5a42>] tgt_brw_write+0xf02/0x1ae0 [ptlrpc]
[14612259.049612]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14612259.056797]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14612259.064751]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14612259.071321]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14612259.076473]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14612259.083193]  [<ffffffffffffffff>] 0xffffffffffffffff
[14612259.088450] LustreError: dumping log to /tmp/lustre-log.1645840041.160943
[14612282.205720] Lustre: 206892:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-23), not sending early reply
                    req@ffff91664cfaa850 x1719243128350720/t0(0) o9->7057b2d1-9e7b-948a-1431-015967a7d250@10.0.3.17@o2ib5:469/0 lens 224/224 e 0 to 0 dl 1645840069 ref 2 fl Interpret:/0/0 rc 0/0
[14612282.234899] Lustre: 206892:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 30 previous similar messages
[14612287.285458] Lustre: oak-OST015b: Export ffff91edac0a8400 already connecting from 10.50.15.13@o2ib2
[14612287.294653] Lustre: Skipped 4404 previous similar messages
[14612320.257224] LNet: Service thread pid 204454 was inactive for 797.46s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
[14612320.274511] LNet: Skipped 1 previous similar message
[14612320.279731] Pid: 204454, comm: ll_ost01_022 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14612320.290568] Call Trace:
[14612320.293321]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14612320.300521]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14612320.307843]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14612320.314640]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14612320.321560]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14612320.329295]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14612320.336588]  [<ffffffffc10d9383>] tgt_client_data_update+0x303/0x5e0 [ptlrpc]
[14612320.344125]  [<ffffffffc10d9a7b>] tgt_client_new+0x41b/0x610 [ptlrpc]
[14612320.350893]  [<ffffffffc1675453>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
[14612320.357438]  [<ffffffffc103e486>] target_handle_connect+0xec6/0x2bf0 [ptlrpc]
[14612320.364907]  [<ffffffffc10e72ba>] tgt_request_handle+0x4fa/0x1570 [ptlrpc]
[14612320.372082]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14612320.380026]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14612320.386588]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14612320.391741]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14612320.398458]  [<ffffffffffffffff>] 0xffffffffffffffff
[14612320.403716] LustreError: dumping log to /tmp/lustre-log.1645840103.204454
[14612322.300263] Pid: 229509, comm: ll_ost00_092 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14612322.311101] Call Trace:
[14612322.313838]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14612322.320994]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14612322.328371]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14612322.335200]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14612322.342183]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14612322.349964]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14612322.357286]  [<ffffffffc10d9383>] tgt_client_data_update+0x303/0x5e0 [ptlrpc]
[14612322.364853]  [<ffffffffc10d9a7b>] tgt_client_new+0x41b/0x610 [ptlrpc]
[14612322.371643]  [<ffffffffc1675453>] ofd_obd_connect+0x3a3/0x4c0 [ofd]
[14612322.378191]  [<ffffffffc103e486>] target_handle_connect+0xec6/0x2bf0 [ptlrpc]
[14612322.385623]  [<ffffffffc10e72ba>] tgt_request_handle+0x4fa/0x1570 [ptlrpc]
[14612322.392845]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14612322.400874]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14612322.407571]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14612322.412860]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14612322.419594]  [<ffffffffffffffff>] 0xffffffffffffffff
[14612322.424951] LustreError: dumping log to /tmp/lustre-log.1645840105.229509
[14612322.432610] Pid: 199273, comm: ll_ost_io01_001 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14612322.443705] Call Trace:
[14612322.446444]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14612322.453594]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14612322.460939]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14612322.467756]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14612322.474652]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14612322.482329]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14612322.489576]  [<ffffffffc1680bee>] ofd_commitrw_write+0xf1e/0x1db0 [ofd]
[14612322.496475]  [<ffffffffc1684c8c>] ofd_commitrw+0x47c/0xa50 [ofd]
[14612322.502750]  [<ffffffffc10e159c>] obd_commitrw+0x9c/0x370 [ptlrpc]
[14612322.509260]  [<ffffffffc10e5a42>] tgt_brw_write+0xf02/0x1ae0 [ptlrpc]
[14612322.516004]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14612322.523179]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14612322.531126]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14612322.537690]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14612322.542842]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14612322.549550]  [<ffffffffffffffff>] 0xffffffffffffffff
[14612330.472407] LustreError: dumping log to /tmp/lustre-log.1645840113.160906
[14612334.559467] LustreError: dumping log to /tmp/lustre-log.1645840117.243372
[14612342.730622] LustreError: dumping log to /tmp/lustre-log.1645840125.21767
[14612344.773659] LustreError: dumping log to /tmp/lustre-log.1645840127.21778
[14612363.160983] LustreError: dumping log to /tmp/lustre-log.1645840146.21768
[14612375.419186] LustreError: dumping log to /tmp/lustre-log.1645840158.218088
[14612397.225307] Lustre: oak-OST015b: Client 1185a5d2-4bff-b58c-00ed-87ed88d988df (at 10.50.9.35@o2ib2) reconnecting
[14612397.235654] Lustre: Skipped 5 previous similar messages
[14612408.107748] LustreError: dumping log to /tmp/lustre-log.1645840191.259128
[14612410.150779] LustreError: dumping log to /tmp/lustre-log.1645840193.203507
[14612421.041317] md: md1: data-check interrupted.
[14612421.505291] md: md11: data-check interrupted.
[14612422.003987] md: md13: data-check interrupted.
[14612422.038884] md: md15: data-check interrupted.
[14612422.074835] md: md17: data-check interrupted.
[14612422.289280] md: md19: data-check interrupted.
[14612422.674401] md: md21: data-check interrupted.
[14612423.032489] md: md23: data-check interrupted.
[14612423.239984] md: md25: data-check interrupted.
[14612423.574176] md: md29: data-check interrupted.
[14612423.854562] md: md3: data-check interrupted.
[14612424.189654] md: md31: data-check interrupted.
[14612424.218097] md: md33: data-check interrupted.
[14612424.649538] md: md39: data-check interrupted.
[14612424.716101] md: md41: data-check interrupted.
[14612425.059606] md: md43: data-check interrupted.
[14612425.546370] md: md45: data-check interrupted.
[14612425.651133] md: md5: data-check interrupted.
[14612426.092034] md: md53: data-check interrupted.
[14612426.293607] md: md59: data-check interrupted.
[14612426.705593] md: md61: data-check interrupted.
[14612427.092626] md: md7: data-check interrupted.
[14612427.340017] md: md9: data-check interrupted.
[14612461.226661] LNet: Service thread pid 160940 was inactive for 896.15s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
[14612461.239839] LNet: Skipped 48 previous similar messages
[14612461.245231] LustreError: dumping log to /tmp/lustre-log.1645840244.160940
[14612465.312809] LustreError: dumping log to /tmp/lustre-log.1645840248.259136
[14612479.195454] md: data-check of RAID array md21
[14612485.318687] md: data-check of RAID array md35
[14612491.462461] md: data-check of RAID array md13
[14612497.605935] md: data-check of RAID array md17
[14612503.749625] md: data-check of RAID array md19
[14612509.901141] md: data-check of RAID array md9
[14612516.046450] md: data-check of RAID array md25
[14612522.214118] md: data-check of RAID array md11
[14612528.334807] md: data-check of RAID array md1
[14612534.483685] md: data-check of RAID array md23
[14612540.624811] md: data-check of RAID array md33
[14612544.991126] LustreError: dumping log to /tmp/lustre-log.1645840328.127348
[14612546.779662] md: data-check of RAID array md39
[14612552.906950] md: data-check of RAID array md41
[14612553.163247] LNet: Service thread pid 160919 was inactive for 947.70s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
[14612553.180483] LNet: Skipped 2 previous similar messages
[14612553.185790] Pid: 160919, comm: ll_ost_io01_070 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14612553.196881] Call Trace:
[14612553.199616]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14612553.206768]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14612553.214084]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14612553.220878]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14612553.227760]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14612553.235440]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14612553.242682]  [<ffffffffc1679425>] ofd_trans_start+0x75/0xf0 [ofd]
[14612553.249056]  [<ffffffffc1680701>] ofd_commitrw_write+0xa31/0x1db0 [ofd]
[14612553.255936]  [<ffffffffc1684c8c>] ofd_commitrw+0x47c/0xa50 [ofd]
[14612553.262215]  [<ffffffffc10e159c>] obd_commitrw+0x9c/0x370 [ptlrpc]
[14612553.268740]  [<ffffffffc10e5a42>] tgt_brw_write+0xf02/0x1ae0 [ptlrpc]
[14612553.275477]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14612553.282644]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14612553.290590]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14612553.297151]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14612553.302304]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14612553.309014]  [<ffffffffffffffff>] 0xffffffffffffffff
[14612553.314263] LustreError: dumping log to /tmp/lustre-log.1645840336.160919
[14612555.206830] Pid: 199260, comm: ll_ost01_000 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14612555.217717] Call Trace:
[14612555.220451]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14612555.227596]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14612555.234921]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14612555.241715]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14612555.248598]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14612555.256272]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14612555.263510]  [<ffffffffc10d8bc0>] tgt_server_data_update+0x3c0/0x510 [ptlrpc]
[14612555.270987]  [<ffffffffc10d9f0d>] tgt_client_del+0x29d/0x6b0 [ptlrpc]
[14612555.277732]  [<ffffffffc16715ec>] ofd_obd_disconnect+0x1ac/0x220 [ofd]
[14612555.284536]  [<ffffffffc1040355>] target_handle_disconnect+0x1a5/0x470 [ptlrpc]
[14612555.292137]  [<ffffffffc10e0738>] tgt_disconnect+0x58/0x170 [ptlrpc]
[14612555.298784]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14612555.305954]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14612555.313898]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14612555.320460]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14612555.325620]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14612555.332338]  [<ffffffffffffffff>] 0xffffffffffffffff
[14612555.337587] LustreError: dumping log to /tmp/lustre-log.1645840338.199260
[14612559.292359] Pid: 162712, comm: ll_ost_io00_080 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14612559.303468] Call Trace:
[14612559.306223]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14612559.313377]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14612559.320691]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14612559.327496]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14612559.334377]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14612559.342057]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14612559.349301]  [<ffffffffc1679425>] ofd_trans_start+0x75/0xf0 [ofd]
[14612559.355673]  [<ffffffffc167d8e8>] ofd_object_punch+0x798/0xd90 [ofd]
[14612559.362295]  [<ffffffffc166a7f3>] ofd_punch_hdl+0x4f3/0xa80 [ofd]
[14612559.368660]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14612559.375877]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14612559.383831]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14612559.390395]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14612559.395546]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14612559.402264]  [<ffffffffffffffff>] 0xffffffffffffffff
[14612559.407514] LustreError: dumping log to /tmp/lustre-log.1645840342.162712
[14612594.052260] Lustre: oak-OST012b: Connection restored to 667f9a3a-6547-a698-3478-abc7fa63037b (at 10.50.2.26@o2ib2)
[14612594.062844] Lustre: Skipped 821 previous similar messages
[14612608.325249] Pid: 162717, comm: ll_ost_io00_085 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14612608.336357] Call Trace:
[14612608.339089]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14612608.346236]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14612608.353550]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14612608.360355]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14612608.367245]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14612608.374916]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14612608.382160]  [<ffffffffc1680bee>] ofd_commitrw_write+0xf1e/0x1db0 [ofd]
[14612608.389051]  [<ffffffffc1684c8c>] ofd_commitrw+0x47c/0xa50 [ofd]
[14612608.395327]  [<ffffffffc10e159c>] obd_commitrw+0x9c/0x370 [ptlrpc]
[14612608.401836]  [<ffffffffc10e5a42>] tgt_brw_write+0xf02/0x1ae0 [ptlrpc]
[14612608.408573]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14612608.415740]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14612608.423685]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14612608.430246]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14612608.435400]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14612608.442110]  [<ffffffffffffffff>] 0xffffffffffffffff
[14612608.447357] LustreError: dumping log to /tmp/lustre-log.1645840391.162717
[14612616.497438] Pid: 160903, comm: ll_ost_io01_058 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14612616.508554] Call Trace:
[14612616.511298]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14612616.518503]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14612616.525897]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14612616.532772]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14612616.539727]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14612616.547472]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14612616.554759]  [<ffffffffc1680bee>] ofd_commitrw_write+0xf1e/0x1db0 [ofd]
[14612616.561656]  [<ffffffffc1684c8c>] ofd_commitrw+0x47c/0xa50 [ofd]
[14612616.567933]  [<ffffffffc10e159c>] obd_commitrw+0x9c/0x370 [ptlrpc]
[14612616.574463]  [<ffffffffc10e5a42>] tgt_brw_write+0xf02/0x1ae0 [ptlrpc]
[14612616.581196]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14612616.588363]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14612616.596309]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14612616.602873]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14612616.608024]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14612616.614733]  [<ffffffffffffffff>] 0xffffffffffffffff
[14612616.619982] LustreError: dumping log to /tmp/lustre-log.1645840400.160903
[14612620.583501] LustreError: dumping log to /tmp/lustre-log.1645840404.229341
[14612624.669562] LustreError: dumping log to /tmp/lustre-log.1645840408.259141
[14612648.862919] Lustre: 160922:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:87s); client may timeout.  req@ffff91a230274850 x1710968848761472/t0(0) o4->39e89f1c-cb27-eddb-826b-2405d7884fbb@10.0.3.5@o2ib5:745/0 lens 488/0 e 0 to 0 dl 1645840345 ref 1 fl Interpret:H/0/ffffffff rc 0/-1
[14612698.218989] LustreError: dumping log to /tmp/lustre-log.1645840481.243450
[14612702.304880] LustreError: dumping log to /tmp/lustre-log.1645840486.228675
[14612706.390945] LustreError: dumping log to /tmp/lustre-log.1645840490.167641
[14612771.768080] LustreError: dumping log to /tmp/lustre-log.1645840555.228745
[14612779.940249] LustreError: dumping log to /tmp/lustre-log.1645840563.199258
[14612784.026599] LustreError: dumping log to /tmp/lustre-log.1645840567.229333
[14612795.740043] Lustre: 21607:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply
                    req@ffff917f944bb050 x1715028380271552/t0(0) o4->6d01c866-9ec9-8f84-4f86-ee1db83afc97@10.210.12.61@tcp1:229/0 lens 488/0 e 0 to 0 dl 1645840584 ref 2 fl New:/0/ffffffff rc 0/-1
[14612795.769379] Lustre: 21607:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2208 previous similar messages
[14612835.708733] LustreError: 243543:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.0.3.6@o2ib5: deadline 755:18s ago
                    req@ffff914f8a5d6050 x1710971471812928/t0(0) o4->b2a077d9-161c-eda8-66bb-0efe482f9fda@10.0.3.6@o2ib5:246/0 lens 488/0 e 0 to 0 dl 1645840601 ref 1 fl Interpret:H/0/ffffffff rc 0/-1
[14612835.740762] Lustre: 243543:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (755:18s); client may timeout.  req@ffff914f8a5d6050 x1710971471812928/t0(0) o4->b2a077d9-161c-eda8-66bb-0efe482f9fda@10.0.3.6@o2ib5:246/0 lens 488/0 e 0 to 0 dl 1645840601 ref 1 fl Interpret:H/0/ffffffff rc 0/-1
[14612837.842253] LustreError: 243543:0:(service.c:2128:ptlrpc_server_handle_request()) @@@ Dropping timed-out request from 12345-10.0.3.25@o2ib5: deadline 600:6s ago
                    req@ffff9182ba0d4850 x1710959849163456/t0(0) o4->bbba5ad9-9372-2297-5b66-f72cfc361471@10.0.3.25@o2ib5:260/0 lens 488/0 e 1 to 0 dl 1645840615 ref 1 fl Interpret:H/0/ffffffff rc 0/-1
[14612837.874369] LustreError: 243543:0:(service.c:2128:ptlrpc_server_handle_request()) Skipped 9 previous similar messages
[14612837.885213] Lustre: 243543:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:6s); client may timeout.  req@ffff9182ba0d4850 x1710959849163456/t0(0) o4->bbba5ad9-9372-2297-5b66-f72cfc361471@10.0.3.25@o2ib5:260/0 lens 488/0 e 1 to 0 dl 1645840615 ref 1 fl Interpret:H/0/ffffffff rc 0/-1
[14612837.914291] Lustre: 243543:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 9 previous similar messages
[14612841.232277] LustreError: dumping log to /tmp/lustre-log.1645840625.259135
[14612849.404417] LustreError: dumping log to /tmp/lustre-log.1645840633.203088
[14612857.575575] LNet: Service thread pid 243373 was inactive for 1150.06s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
[14612857.592909] LNet: Skipped 4 previous similar messages
[14612857.598207] Pid: 243373, comm: ll_ost01_091 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14612857.609043] Call Trace:
[14612857.611778]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14612857.618937]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14612857.626252]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14612857.633048]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14612857.639931]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14612857.647607]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14612857.654846]  [<ffffffffc10d8bc0>] tgt_server_data_update+0x3c0/0x510 [ptlrpc]
[14612857.662313]  [<ffffffffc10d9f0d>] tgt_client_del+0x29d/0x6b0 [ptlrpc]
[14612857.669057]  [<ffffffffc16715ec>] ofd_obd_disconnect+0x1ac/0x220 [ofd]
[14612857.675862]  [<ffffffffc1040355>] target_handle_disconnect+0x1a5/0x470 [ptlrpc]
[14612857.683461]  [<ffffffffc10e0738>] tgt_disconnect+0x58/0x170 [ptlrpc]
[14612857.690109]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14612857.697277]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14612857.705223]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14612857.711785]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14612857.716939]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14612857.723654]  [<ffffffffffffffff>] 0xffffffffffffffff
[14612857.728906] LustreError: dumping log to /tmp/lustre-log.1645840641.243373
[14612857.736475] Pid: 243396, comm: ll_ost01_114 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14612857.747312] Call Trace:
[14612857.750030]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14612857.757182]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14612857.764497]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14612857.771293]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14612857.778173]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14612857.785843]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14612857.793079]  [<ffffffffc10d8bc0>] tgt_server_data_update+0x3c0/0x510 [ptlrpc]
[14612857.800542]  [<ffffffffc10d9f0d>] tgt_client_del+0x29d/0x6b0 [ptlrpc]
[14612857.807285]  [<ffffffffc16715ec>] ofd_obd_disconnect+0x1ac/0x220 [ofd]
[14612857.814091]  [<ffffffffc1040355>] target_handle_disconnect+0x1a5/0x470 [ptlrpc]
[14612857.821697]  [<ffffffffc10e0738>] tgt_disconnect+0x58/0x170 [ptlrpc]
[14612857.828356]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14612857.835537]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14612857.843494]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14612857.850073]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14612857.855252]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14612857.861977]  [<ffffffffffffffff>] 0xffffffffffffffff
[14612861.661944] Pid: 205374, comm: ll_ost00_020 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14612861.672783] Call Trace:
[14612861.675519]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14612861.682673]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14612861.689984]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14612861.696782]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14612861.703689]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14612861.711382]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14612861.718667]  [<ffffffffc10d8bc0>] tgt_server_data_update+0x3c0/0x510 [ptlrpc]
[14612861.726190]  [<ffffffffc10d9f0d>] tgt_client_del+0x29d/0x6b0 [ptlrpc]
[14612861.732975]  [<ffffffffc16715ec>] ofd_obd_disconnect+0x1ac/0x220 [ofd]
[14612861.739811]  [<ffffffffc1040355>] target_handle_disconnect+0x1a5/0x470 [ptlrpc]
[14612861.747448]  [<ffffffffc10e0738>] tgt_disconnect+0x58/0x170 [ptlrpc]
[14612861.754141]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14612861.761348]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14612861.769335]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14612861.775949]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14612861.781164]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14612861.787933]  [<ffffffffffffffff>] 0xffffffffffffffff
[14612861.793247] LustreError: dumping log to /tmp/lustre-log.1645840645.205374
[14612885.983438] Lustre: oak-OST015b: Export ffff915035b3c000 already connecting from 10.0.3.25@o2ib5
[14612885.992464] Lustre: Skipped 5320 previous similar messages
[14612914.283292] Lustre: oak-OST015b: Client 3315d2c0-91e7-5c31-baba-74ee60d7ae8f (at 10.51.2.11@o2ib3) reconnecting
[14612914.293612] Lustre: Skipped 944 previous similar messages
[14612931.125005] Pid: 206916, comm: ll_ost00_120 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14612931.135839] Call Trace:
[14612931.138572]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14612931.145722]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14612931.153035]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14612931.159831]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14612931.166713]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14612931.174391]  [<ffffffffc1439873>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
[14612931.181628]  [<ffffffffa2abd7fa>] dqget+0x41a/0x470
[14612931.186779]  [<ffffffffa2abe604>] dquot_get_dqblk+0x14/0x1f0
[14612931.192710]  [<ffffffffc154ae15>] osd_acct_index_lookup+0x235/0x480 [osd_ldiskfs]
[14612931.200475]  [<ffffffffc14b086d>] lquotactl_slv+0x27d/0xa00 [lquota]
[14612931.207107]  [<ffffffffc166571c>] ofd_quotactl+0x13c/0x380 [ofd]
[14612931.213389]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14612931.220593]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14612931.228538]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14612931.235100]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14612931.240253]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14612931.246961]  [<ffffffffffffffff>] 0xffffffffffffffff
[14612931.252212] LustreError: dumping log to /tmp/lustre-log.1645840715.206916
[14612931.260354] Pid: 167639, comm: ll_ost00_061 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14612931.271189] Call Trace:
[14612931.273904]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14612931.281052]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14612931.288365]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14612931.295160]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14612931.302043]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14612931.309712]  [<ffffffffc1439873>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
[14612931.316948]  [<ffffffffa2abd7fa>] dqget+0x41a/0x470
[14612931.322101]  [<ffffffffa2abe604>] dquot_get_dqblk+0x14/0x1f0
[14612931.328033]  [<ffffffffc154ae15>] osd_acct_index_lookup+0x235/0x480 [osd_ldiskfs]
[14612931.335797]  [<ffffffffc14b086d>] lquotactl_slv+0x27d/0xa00 [lquota]
[14612931.342428]  [<ffffffffc166571c>] ofd_quotactl+0x13c/0x380 [ofd]
[14612931.348713]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14612931.355898]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14612931.363842]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14612931.370405]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14612931.375559]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14612931.382267]  [<ffffffffffffffff>] 0xffffffffffffffff
[14612935.210908] LustreError: dumping log to /tmp/lustre-log.1645840719.243358
[14612955.641679] LustreError: dumping log to /tmp/lustre-log.1645840740.160946
[14612973.087868] md: md1: data-check interrupted.
[14612973.114804] md: md11: data-check interrupted.
[14612973.391129] md: md13: data-check interrupted.
[14612973.410939] md: md17: data-check interrupted.
[14612973.657543] md: md19: data-check interrupted.
[14612973.921871] md: md21: data-check interrupted.
[14612974.156269] md: md23: data-check interrupted.
[14612974.557397] md: md25: data-check interrupted.
[14612974.700951] md: md33: data-check interrupted.
[14612974.905449] md: md35: data-check interrupted.
[14612975.360357] md: md39: data-check interrupted.
[14612975.516984] md: md41: data-check interrupted.
[14612975.948913] md: md9: data-check interrupted.
[14612976.071629] LustreError: dumping log to /tmp/lustre-log.1645840760.253936
[14612978.210667] md: data-check of RAID array md7
[14612984.243767] LustreError: dumping log to /tmp/lustre-log.1645840768.243350
[14612984.354879] md: data-check of RAID array md29
[14612990.502942] md: data-check of RAID array md31
[14613000.588049] LustreError: dumping log to /tmp/lustre-log.1645840785.162672
[14613004.674123] LustreError: dumping log to /tmp/lustre-log.1645840789.162671
[14613008.760192] LustreError: dumping log to /tmp/lustre-log.1645840793.162683
[14613012.846265] LustreError: dumping log to /tmp/lustre-log.1645840797.21589
[14613021.018412] LustreError: dumping log to /tmp/lustre-log.1645840805.21605
[14613029.190547] LustreError: dumping log to /tmp/lustre-log.1645840813.21600
[14613033.276696] LustreError: dumping log to /tmp/lustre-log.1645840817.21587
[14613037.362740] LustreError: dumping log to /tmp/lustre-log.1645840821.162709
[14613041.448761] LustreError: dumping log to /tmp/lustre-log.1645840826.162710
[14613065.965222] LNet: Service thread pid 21603 was inactive for 1202.88s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
[14613065.978395] LNet: Skipped 87 previous similar messages
[14613065.983784] LustreError: dumping log to /tmp/lustre-log.1645840850.21603
[14613072.061445] md: md29: data-check interrupted.
[14613072.389596] md: md31: data-check interrupted.
[14613072.705807] md: md7: data-check interrupted.
[14613082.309734] LustreError: dumping log to /tmp/lustre-log.1645840866.243367
[14613086.395578] LustreError: dumping log to /tmp/lustre-log.1645840871.243366
[14613102.740353] LustreError: dumping log to /tmp/lustre-log.1645840887.229332
[14613123.170329] LustreError: dumping log to /tmp/lustre-log.1645840907.243364
[14613139.514594] LustreError: dumping log to /tmp/lustre-log.1645840924.243341
[14613143.600635] LustreError: dumping log to /tmp/lustre-log.1645840928.203089
[14613192.661536] Lustre: oak-OST011b: Connection restored to  (at 10.51.15.22@o2ib3)
[14613192.669094] Lustre: Skipped 1716 previous similar messages
[14613233.494196] LNet: Service thread pid 243374 was inactive for 1203.84s. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
[14613233.511527] LNet: Skipped 4 previous similar messages
[14613233.516831] Pid: 243374, comm: ll_ost01_092 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14613233.527662] Call Trace:
[14613233.530378]  [<ffffffffc166dbc3>] ofd_create_hdl+0xcb3/0x20e0 [ofd]
[14613233.536921]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14613233.544139]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14613233.552105]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14613233.558665]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14613233.563815]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14613233.570537]  [<ffffffffffffffff>] 0xffffffffffffffff
[14613233.575802] LustreError: dumping log to /tmp/lustre-log.1645841018.243374
[14613241.666402] Pid: 228871, comm: ll_ost01_049 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14613241.677240] Call Trace:
[14613241.679955]  [<ffffffffc166dbc3>] ofd_create_hdl+0xcb3/0x20e0 [ofd]
[14613241.686496]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14613241.693714]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14613241.701687]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14613241.708248]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14613241.713400]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14613241.720110]  [<ffffffffffffffff>] 0xffffffffffffffff
[14613241.725368] LustreError: dumping log to /tmp/lustre-log.1645841026.228871
[14613253.924690] Pid: 243381, comm: ll_ost01_099 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14613253.935532] Call Trace:
[14613253.938248]  [<ffffffffa2a6b320>] __wait_on_freeing_inode+0xb0/0xf0
[14613253.944787]  [<ffffffffa2a6b543>] find_inode_fast+0x73/0xc0
[14613253.950633]  [<ffffffffa2a6c8e9>] iget_locked+0x79/0x240
[14613253.956218]  [<ffffffffc1463272>] ldiskfs_iget+0x42/0xc40 [ldiskfs]
[14613253.962771]  [<ffffffffc1515776>] osd_iget+0x26/0x2d0 [osd_ldiskfs]
[14613253.969326]  [<ffffffffc153d5a0>] osd_obj_map_lookup+0x260/0x510 [osd_ldiskfs]
[14613253.976819]  [<ffffffffc1529e39>] osd_oi_lookup+0x139/0x1e0 [osd_ldiskfs]
[14613253.983874]  [<ffffffffc1525145>] osd_fid_lookup+0x445/0x1d90 [osd_ldiskfs]
[14613253.991112]  [<ffffffffc1526af1>] osd_object_init+0x61/0x110 [osd_ldiskfs]
[14613253.998253]  [<ffffffffc0d9f83b>] lu_object_start.isra.35+0x8b/0x120 [obdclass]
[14613254.005868]  [<ffffffffc0da3754>] lu_object_find_at+0x234/0xab0 [obdclass]
[14613254.013038]  [<ffffffffc0da3fe6>] lu_object_find+0x16/0x20 [obdclass]
[14613254.019763]  [<ffffffffc16799d5>] ofd_object_find+0x35/0x100 [ofd]
[14613254.026222]  [<ffffffffc1688c2b>] ofd_lvbo_init+0x35b/0x84d [ofd]
[14613254.032595]  [<ffffffffc105791c>] ldlm_handle_enqueue0+0x98c/0x1620 [ptlrpc]
[14613254.039979]  [<ffffffffc10e0a42>] tgt_enqueue+0x62/0x210 [ptlrpc]
[14613254.046394]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14613254.053574]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14613254.061522]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14613254.068084]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14613254.073236]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14613254.079946]  [<ffffffffffffffff>] 0xffffffffffffffff
[14613254.085206] LustreError: dumping log to /tmp/lustre-log.1645841039.243381
[14613281.709993] LNet: 205093:0:(o2iblnd_cb.c:2629:kiblnd_passive_connect()) Conn stale 10.0.3.59@o2ib5 version 12/12 incarnation 1645323953904132/1645841064079163
[14613284.494051] LNet: 205093:0:(o2iblnd_cb.c:2629:kiblnd_passive_connect()) Conn stale 10.0.3.61@o2ib5 version 12/12 incarnation 1645323966647056/1645841067099473
[14613287.788023] LNet: 205093:0:(o2iblnd_cb.c:2629:kiblnd_passive_connect()) Conn stale 10.0.3.66@o2ib5 version 12/12 incarnation 1645323933012644/1645841067196407
[14613294.785777] Pid: 243370, comm: ll_ost01_088 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14613294.796620] Call Trace:
[14613294.799349]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14613294.806502]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14613294.813816]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14613294.820612]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14613294.827501]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14613294.835181]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14613294.842415]  [<ffffffffc10d8bc0>] tgt_server_data_update+0x3c0/0x510 [ptlrpc]
[14613294.849902]  [<ffffffffc10d9f0d>] tgt_client_del+0x29d/0x6b0 [ptlrpc]
[14613294.856655]  [<ffffffffc16715ec>] ofd_obd_disconnect+0x1ac/0x220 [ofd]
[14613294.863460]  [<ffffffffc1040355>] target_handle_disconnect+0x1a5/0x470 [ptlrpc]
[14613294.871059]  [<ffffffffc10e0738>] tgt_disconnect+0x58/0x170 [ptlrpc]
[14613294.877717]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14613294.884882]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14613294.892828]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14613294.899399]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14613294.904563]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14613294.911279]  [<ffffffffffffffff>] 0xffffffffffffffff
[14613294.916548] LustreError: dumping log to /tmp/lustre-log.1645841080.243370
[14613319.301705] Pid: 259132, comm: ll_ost00_047 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14613319.312539] Call Trace:
[14613319.315269]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14613319.322420]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14613319.329733]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14613319.336528]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14613319.343411]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14613319.351091]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14613319.358333]  [<ffffffffc10d8bc0>] tgt_server_data_update+0x3c0/0x510 [ptlrpc]
[14613319.365814]  [<ffffffffc10d9f0d>] tgt_client_del+0x29d/0x6b0 [ptlrpc]
[14613319.372556]  [<ffffffffc16715ec>] ofd_obd_disconnect+0x1ac/0x220 [ofd]
[14613319.379361]  [<ffffffffc1040355>] target_handle_disconnect+0x1a5/0x470 [ptlrpc]
[14613319.386959]  [<ffffffffc10e0738>] tgt_disconnect+0x58/0x170 [ptlrpc]
[14613319.393608]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14613319.400776]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14613319.408721]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14613319.415293]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14613319.420445]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14613319.427152]  [<ffffffffffffffff>] 0xffffffffffffffff
[14613319.432404] LustreError: dumping log to /tmp/lustre-log.1645841104.259132
[14613335.645896] LustreError: dumping log to /tmp/lustre-log.1645841120.201743
[14613339.732062] LustreError: dumping log to /tmp/lustre-log.1645841125.243354
[14613384.678909] LustreError: dumping log to /tmp/lustre-log.1645841170.168275
[14613386.543127] Lustre: 160922:0:(service.c:2165:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600:136s); client may timeout.  req@ffff91aa225f4050 x1710968855922944/t0(0) o10->39e89f1c-cb27-eddb-826b-2405d7884fbb@10.0.3.5@o2ib5:680/0 lens 440/0 e 1 to 0 dl 1645841035 ref 1 fl Interpret:H/0/ffffffff rc 0/-1
[14613386.572383] Lustre: 160922:0:(service.c:2165:ptlrpc_server_handle_request()) Skipped 12 previous similar messages
[14613394.485128] Lustre: 21607:0:(service.c:1372:ptlrpc_at_send_early_reply()) @@@ Couldn't add any time (5/-150), not sending early reply
                    req@ffff9134d5bb9850 x1715192016157184/t0(0) o4->7189afb8-8fe5-7418-d14a-4a60d24330d0@10.210.12.48@tcp1:74/0 lens 488/0 e 0 to 0 dl 1645841184 ref 2 fl New:/2/ffffffff rc 0/-1
[14613394.514383] Lustre: 21607:0:(service.c:1372:ptlrpc_at_send_early_reply()) Skipped 2248 previous similar messages
[14613401.023055] LustreError: dumping log to /tmp/lustre-log.1645841186.199263
[14613429.625549] LustreError: dumping log to /tmp/lustre-log.1645841215.243351
[14613433.711692] LustreError: dumping log to /tmp/lustre-log.1645841219.243384
[14613485.046441] Lustre: oak-OST015b: Export ffff913d4f718c00 already connecting from 10.0.3.6@o2ib5
[14613485.055390] Lustre: Skipped 2526 previous similar messages
[14613486.830590] LustreError: dumping log to /tmp/lustre-log.1645841272.259133
[14613503.174965] LustreError: dumping log to /tmp/lustre-log.1645841288.229334
[14613513.481527] Lustre: oak-OST0159: Client ae865b82-51e6-6ef5-51e8-057f7a99f1a1 (at 10.210.12.63@tcp1) reconnecting
[14613513.491933] Lustre: Skipped 1108 previous similar messages
[14613531.777350] LustreError: dumping log to /tmp/lustre-log.1645841317.168283
[14613544.035702] Pid: 206886, comm: ll_ost00_116 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14613544.046544] Call Trace:
[14613544.049275]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14613544.056431]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14613544.063752]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14613544.070545]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14613544.077427]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14613544.085097]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14613544.092341]  [<ffffffffc10d8bc0>] tgt_server_data_update+0x3c0/0x510 [ptlrpc]
[14613544.099805]  [<ffffffffc10d9f0d>] tgt_client_del+0x29d/0x6b0 [ptlrpc]
[14613544.106547]  [<ffffffffc16715ec>] ofd_obd_disconnect+0x1ac/0x220 [ofd]
[14613544.113350]  [<ffffffffc1040355>] target_handle_disconnect+0x1a5/0x470 [ptlrpc]
[14613544.120951]  [<ffffffffc10e0738>] tgt_disconnect+0x58/0x170 [ptlrpc]
[14613544.127599]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14613544.134767]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14613544.142712]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14613544.149274]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14613544.154428]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14613544.161135]  [<ffffffffffffffff>] 0xffffffffffffffff
[14613544.166388] LustreError: dumping log to /tmp/lustre-log.1645841329.206886
[14613576.724207] Pid: 243397, comm: ll_ost01_115 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14613576.735046] Call Trace:
[14613576.737773]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14613576.744923]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14613576.752237]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14613576.759034]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14613576.765916]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14613576.773594]  [<ffffffffc1439873>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
[14613576.780829]  [<ffffffffa2abd7fa>] dqget+0x41a/0x470
[14613576.785981]  [<ffffffffa2abe604>] dquot_get_dqblk+0x14/0x1f0
[14613576.791913]  [<ffffffffc154ae15>] osd_acct_index_lookup+0x235/0x480 [osd_ldiskfs]
[14613576.799687]  [<ffffffffc14b086d>] lquotactl_slv+0x27d/0xa00 [lquota]
[14613576.806318]  [<ffffffffc166571c>] ofd_quotactl+0x13c/0x380 [ofd]
[14613576.812601]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14613576.819823]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14613576.827794]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14613576.834355]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14613576.839516]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14613576.846225]  [<ffffffffffffffff>] 0xffffffffffffffff
[14613576.851492] LustreError: dumping log to /tmp/lustre-log.1645841362.243397
[14613658.445751] Pid: 243361, comm: ll_ost01_079 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14613658.456591] Call Trace:
[14613658.459322]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14613658.466462]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14613658.473779]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14613658.480574]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14613658.487456]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14613658.495133]  [<ffffffffc1439873>] ldiskfs_acquire_dquot+0x53/0xb0 [ldiskfs]
[14613658.502368]  [<ffffffffa2abd7fa>] dqget+0x41a/0x470
[14613658.507531]  [<ffffffffa2abe604>] dquot_get_dqblk+0x14/0x1f0
[14613658.513460]  [<ffffffffc154ae15>] osd_acct_index_lookup+0x235/0x480 [osd_ldiskfs]
[14613658.521225]  [<ffffffffc14b086d>] lquotactl_slv+0x27d/0xa00 [lquota]
[14613658.527855]  [<ffffffffc166571c>] ofd_quotactl+0x13c/0x380 [ofd]
[14613658.534135]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14613658.541353]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14613658.549325]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14613658.555884]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14613658.561038]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14613658.567747]  [<ffffffffffffffff>] 0xffffffffffffffff
[14613658.572999] LustreError: dumping log to /tmp/lustre-log.1645841444.243361
[14613662.531701] Pid: 205179, comm: ll_ost00_013 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14613662.542536] Call Trace:
[14613662.545269]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14613662.552416]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14613662.559730]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14613662.566527]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14613662.573408]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14613662.581085]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14613662.588330]  [<ffffffffc10d8bc0>] tgt_server_data_update+0x3c0/0x510 [ptlrpc]
[14613662.595809]  [<ffffffffc10d9f0d>] tgt_client_del+0x29d/0x6b0 [ptlrpc]
[14613662.602553]  [<ffffffffc16715ec>] ofd_obd_disconnect+0x1ac/0x220 [ofd]
[14613662.609357]  [<ffffffffc1040355>] target_handle_disconnect+0x1a5/0x470 [ptlrpc]
[14613662.616954]  [<ffffffffc10e0738>] tgt_disconnect+0x58/0x170 [ptlrpc]
[14613662.623607]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14613662.630772]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14613662.638718]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14613662.645280]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14613662.650434]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14613662.657141]  [<ffffffffffffffff>] 0xffffffffffffffff
[14613662.662395] LustreError: dumping log to /tmp/lustre-log.1645841448.205179
[14613687.048193] Pid: 230671, comm: ll_ost00_108 3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1 SMP Mon Dec 14 21:25:04 PST 2020
[14613687.059034] Call Trace:
[14613687.061763]  [<ffffffffc0275085>] wait_transaction_locked+0x85/0xd0 [jbd2]
[14613687.068914]  [<ffffffffc0275378>] add_transaction_credits+0x278/0x310 [jbd2]
[14613687.076227]  [<ffffffffc0275601>] start_this_handle+0x1a1/0x430 [jbd2]
[14613687.083024]  [<ffffffffc0275ab3>] jbd2__journal_start+0xf3/0x1f0 [jbd2]
[14613687.089904]  [<ffffffffc14425b9>] __ldiskfs_journal_start_sb+0x69/0xe0 [ldiskfs]
[14613687.097584]  [<ffffffffc1516e8e>] osd_trans_start+0x20e/0x4e0 [osd_ldiskfs]
[14613687.104828]  [<ffffffffc10d8bc0>] tgt_server_data_update+0x3c0/0x510 [ptlrpc]
[14613687.112290]  [<ffffffffc10d9f0d>] tgt_client_del+0x29d/0x6b0 [ptlrpc]
[14613687.119025]  [<ffffffffc16715ec>] ofd_obd_disconnect+0x1ac/0x220 [ofd]
[14613687.125828]  [<ffffffffc1040355>] target_handle_disconnect+0x1a5/0x470 [ptlrpc]
[14613687.133420]  [<ffffffffc10e0738>] tgt_disconnect+0x58/0x170 [ptlrpc]
[14613687.140069]  [<ffffffffc10e789a>] tgt_request_handle+0xada/0x1570 [ptlrpc]
[14613687.147237]  [<ffffffffc108c73b>] ptlrpc_server_handle_request+0x24b/0xab0 [ptlrpc]
[14613687.155180]  [<ffffffffc10900a4>] ptlrpc_main+0xb34/0x1470 [ptlrpc]
[14613687.161744]  [<ffffffffa28c5c21>] kthread+0xd1/0xe0
[14613687.166897]  [<ffffffffa2f94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[14613687.173612]  [<ffffffffffffffff>] 0xffffffffffffffff
[14613687.178865] LustreError: dumping log to /tmp/lustre-log.1645841473.230671
[14613695.220271] LNet: Service thread pid 201994 was inactive for 1201.46s. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
[14613695.233535] LNet: Skipped 18 previous similar messages
[14613695.238922] LustreError: dumping log to /tmp/lustre-log.1645841481.201994
[14613699.306536] LustreError: dumping log to /tmp/lustre-log.1645841485.243388
[14613707.478471] LustreError: dumping log to /tmp/lustre-log.1645841493.203778
[14613736.080936] LustreError: dumping log to /tmp/lustre-log.1645841522.168271
[14613792.207582] Lustre: oak-OST013f: Connection restored to 325d0120-4965-a109-c966-639221c6aff5 (at 10.210.12.27@tcp1)
[14613792.218247] Lustre: Skipped 1720 previous similar messages
[14613805.544157] LustreError: dumping log to /tmp/lustre-log.1645841591.243377
[14613823.236185] SysRq : Trigger a crash
[14613823.239987] BUG: unable to handle kernel NULL pointer dereference at           (null)
[14613823.248093] IP: [<ffffffffa2c74a76>] sysrq_handle_crash+0x16/0x20
[14613823.254458] PGD 7036dc0067 PUD 880fdac067 PMD 0 
[14613823.259401] Oops: 0002 [#1] SMP 
[14613823.262921] Modules linked in: lustre(OE) mdc(OE) osp(OE) ofd(OE) lfsck(OE) ost(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) raid456 async_raid6_recov async_memcpy async_pq raid6_pq libcrc32c async_xor xor async_tx ldiskfs(OE) lmv(OE) osc(OE) lov(OE) fid(OE) fld(OE) ptlrpc(OE) obdclass(OE) iTCO_wdt dell_smbios iTCO_vendor_support dell_wmi_descriptor dcdbas skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr ko2iblnd(OE) lnet(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace libcfs(OE) fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) ib_uverbs(OE) mlx4_ib(OE) ib_core(OE) mlx4_en(OE) mlx4_core(OE) dm_service_time ses
[14613823.335016]  enclosure sg i2c_i801 lpc_ich mei_me mei wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad dm_multipath dm_mod sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops mlx5_core(OE) ahci ttm mlxfw(OE) devlink libahci crct10dif_pclmul mpt3sas(OE) crct10dif_common tg3 mlx_compat(OE) drm crc32c_intel libata raid_class ptp megaraid_sas scsi_transport_sas pps_core drm_panel_orientation_quirks nfit libnvdimm [last unloaded: mdc]
[14613823.382338] CPU: 10 PID: 252757 Comm: bash Kdump: loaded Tainted: G           OE  ------------   3.10.0-1160.6.1.el7_lustre.pl1.x86_64 #1
[14613823.394898] Hardware name: Dell Inc. PowerEdge R640/06NR82, BIOS 2.10.0 11/12/2020
[14613823.402704] task: ffff916a681db180 ti: ffff9176da7a4000 task.ti: ffff9176da7a4000
[14613823.410417] RIP: 0010:[<ffffffffa2c74a76>]  [<ffffffffa2c74a76>] sysrq_handle_crash+0x16/0x20
[14613823.419200] RSP: 0018:ffff9176da7a7e58  EFLAGS: 00010246
[14613823.424751] RAX: ffffffffa2c74a60 RBX: ffffffffa34e74a0 RCX: 0000000000000000
[14613823.432127] RDX: 0000000000000000 RSI: ffff91923f1538d8 RDI: 0000000000000063
[14613823.439501] RBP: ffff9176da7a7e58 R08: ffffffffa380487c R09: ffffffffa38d1467
[14613823.446875] R10: 000000000002b297 R11: 000000000002b296 R12: 0000000000000063
[14613823.454250] R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
[14613823.461617] FS:  00007f9a6cc1b740(0000) GS:ffff91923f140000(0000) knlGS:0000000000000000
[14613823.469944] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[14613823.475934] CR2: 0000000000000000 CR3: 0000009cb3748000 CR4: 00000000007607e0
[14613823.483308] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[14613823.490685] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[14613823.498058] PKRU: 55555554
[14613823.501023] Call Trace:
[14613823.503731]  [<ffffffffa2c7529d>] __handle_sysrq+0x10d/0x170
[14613823.509636]  [<ffffffffa2c75708>] write_sysrq_trigger+0x28/0x40
[14613823.515801]  [<ffffffffa2ac6f30>] proc_reg_write+0x40/0x80
[14613823.521533]  [<ffffffffa2a4db90>] vfs_write+0xc0/0x1f0
[14613823.526917]  [<ffffffffa2a4e96f>] SyS_write+0x7f/0xf0
[14613823.532220]  [<ffffffffa2f94f92>] system_call_fastpath+0x25/0x2a
[14613823.538470] Code: eb 9b 45 01 f4 45 39 65 34 75 e5 4c 89 ef e8 e2 f7 ff ff eb db 0f 1f 44 00 00 55 48 89 e5 c7 05 61 34 7d 00 01 00 00 00 0f ae f8 <c6> 04 25 00 00 00 00 01 5d c3 0f 1f 44 00 00 55 31 c0 c7 05 de 
[14613823.559019] RIP  [<ffffffffa2c74a76>] sysrq_handle_crash+0x16/0x20
[14613823.565462]  RSP <ffff9176da7a7e58>
[14613823.569205] CR2: 0000000000000000