Details
-
Bug
-
Resolution: Unresolved
-
Minor
-
None
-
Lustre 2.8.0
-
None
-
Server: 2.7.1, b2_7_fe/34
Client: master, build# 3303, RHEL 7
-
3
-
9223372036854775807
Description
This issue was created by maloo for Saurabh Tandan <saurabh.tandan@intel.com>
This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/2b045008-bb03-11e5-9137-5254006e85c2.
The sub-test test_78 failed with the following error:
test failed to respond and timed out
dmesg log:
[10439.680759] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 78: Enable policy and specify tunings right away ===================================== 17:01:07 \(1452704467\) [10439.938935] Lustre: DEBUG MARKER: == sanityn test 78: Enable policy and specify tunings right away ===================================== 17:01:07 (1452704467) [10449.341259] LNet: Host 10.1.6.67 reset our connection while we were sending data; it may have rebooted. [10449.343091] Lustre: 7118:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1452704477/real 1452704477] req@ffff88001b58a100 x1523260436268040/t0(0) o400->lustre-OST0000-osc-ffff8800368af800@10.1.6.67@tcp:28/4 lens 224/224 e 0 to 1 dl 1452704484 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [10449.345275] Lustre: lustre-OST0000-osc-ffff8800368af800: Connection to lustre-OST0000 (at 10.1.6.67@tcp) was lost; in progress operations using this service will wait for recovery to complete [10449.346597] Lustre: Skipped 1 previous similar message [10451.341056] Lustre: 7117:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1452704472/real 1452704472] req@ffff88001b589b00 x1523260436267976/t0(0) o400->lustre-OST0001-osc-ffff8800368af800@10.1.6.67@tcp:28/4 lens 224/224 e 0 to 1 dl 1452704479 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [10451.343500] Lustre: 7117:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 15 previous similar messages [10451.344376] Lustre: lustre-OST0001-osc-ffff8800368af800: Connection to lustre-OST0001 (at 10.1.6.67@tcp) was lost; in progress operations using this service will wait for recovery to complete [10451.345642] Lustre: Skipped 7 previous similar messages [10456.346044] Lustre: 7118:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1452704477/real 1452704477] req@ffff88001b589200 x1523260436268044/t0(0) o400->lustre-OST0001-osc-ffff8800368af800@10.1.6.67@tcp:28/4 lens 224/224 e 0 to 1 dl 1452704484 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [10456.348407] Lustre: 7118:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 13 previous similar messages [10467.346042] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1452704484/real 1452704484] req@ffff88002a88e700 x1523260436268168/t0(0) o8->lustre-OST0000-osc-ffff8800368af800@10.1.6.67@tcp:28/4 lens 520/544 e 0 to 1 dl 1452704495 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [10467.348313] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 11 previous similar messages [10472.346045] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1452704489/real 0] req@ffff88002a88f000 x1523260436268212/t0(0) o8->lustre-OST0001-osc-ffff8800368af800@10.1.6.67@tcp:28/4 lens 520/544 e 0 to 1 dl 1452704500 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [10472.348250] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [10487.348063] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1452704499/real 0] req@ffff88002a88d500 x1523260436268260/t0(0) o8->lustre-OST0000-osc-ffff8800368af800@10.1.6.67@tcp:28/4 lens 520/544 e 0 to 1 dl 1452704515 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1 [10487.350107] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 5 previous similar messages [10524.514309] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1452704549/real 1452704552] req@ffff88002a88cf00 x1523260436268412/t0(0) o8->lustre-OST0001-osc-ffff8800368af800@10.1.6.67@tcp:28/4 lens 520/544 e 0 to 1 dl 1452704565 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [10524.520275] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 7 previous similar messages [10561.346330] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1452704564/real 1452704564] req@ffff88002a88d200 x1523260436268584/t0(0) o8->lustre-OST0000-osc-ffff8800368af800@10.1.6.67@tcp:28/4 lens 520/544 e 0 to 1 dl 1452704589 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [10561.351317] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 57 previous similar messages [10626.346595] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1452704654/real 1452704654] req@ffff88002a88ea00 x1523260436269368/t0(0) o8->lustre-OST0000-osc-ffff8800368af800@10.1.6.67@tcp:28/4 lens 520/544 e 0 to 1 dl 1452704679 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [10626.352846] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 117 previous similar messages [10756.346646] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1452704784/real 1452704784] req@ffff88002a88c300 x1523260436270876/t0(0) o8->lustre-OST0000-osc-ffff8800368af800@10.1.6.67@tcp:28/4 lens 520/544 e 0 to 1 dl 1452704809 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [10756.351678] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 298 previous similar messages [11016.346320] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1452705044/real 1452705044] req@ffff88002a88db00 x1523260436273760/t0(0) o8->lustre-OST0000-osc-ffff8800368af800@10.1.6.67@tcp:28/4 lens 520/544 e 0 to 1 dl 1452705069 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [11016.351233] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 564 previous similar messages [11531.346322] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1452705559/real 1452705559] req@ffff88002a88de00 x1523260436279432/t0(0) o8->lustre-OST0000-osc-ffff8800368af800@10.1.6.67@tcp:28/4 lens 520/544 e 0 to 1 dl 1452705584 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [11531.352554] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 1108 previous similar messages [12136.347472] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1452706164/real 1452706164] req@ffff88002a88e100 x1523260436286396/t0(0) o8->lustre-OST0001-osc-ffff880038a97000@10.1.6.67@tcp:28/4 lens 520/544 e 0 to 1 dl 1452706189 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [12136.351570] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 1368 previous similar messages [12741.347621] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1452706769/real 1452706769] req@ffff88002a88c000 x1523260436293316/t0(0) o8->lustre-OST0000-osc-ffff8800368af800@10.1.6.67@tcp:28/4 lens 520/544 e 0 to 1 dl 1452706794 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [12741.351700] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 1375 previous similar messages [13341.348089] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1452707369/real 1452707369] req@ffff88002a88de00 x1523260436300420/t0(0) o8->lustre-OST0005-osc-ffff880038a97000@10.1.6.67@tcp:28/4 lens 520/544 e 0 to 1 dl 1452707394 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [13341.351725] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 1413 previous similar messages [13946.347428] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1452707974/real 1452707974] req@ffff88002a88c000 x1523260436307284/t0(0) o8->lustre-OST0000-osc-ffff8800368af800@10.1.6.67@tcp:28/4 lens 520/544 e 0 to 1 dl 1452707999 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1 [13946.351913] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 1354 previous similar messages