[LU-7684] Intern master<->2.7.1 - sanityn test_78: test failed to respond and timed out

Related — Thu, 16 Mar 2017 19:07:55 +0000

This issue was created by maloo for Saurabh Tandan <saurabh.tandan@intel.com>

This issue relates to the following test suite run: https://testing.hpdd.intel.com/test_sets/2b045008-bb03-11e5-9137-5254006e85c2.

The sub-test test_78 failed with the following error:

test failed to respond and timed out

dmesg log:

[10439.680759] Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 78: Enable policy and specify tunings right away ===================================== 17:01:07 \(1452704467\)
[10439.938935] Lustre: DEBUG MARKER: == sanityn test 78: Enable policy and specify tunings right away ===================================== 17:01:07 (1452704467)
[10449.341259] LNet: Host 10.1.6.67 reset our connection while we were sending data; it may have rebooted.
[10449.343091] Lustre: 7118:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1452704477/real 1452704477]  req@ffff88001b58a100 x1523260436268040/t0(0) o400->lustre-OST0000-osc-ffff8800368af800@10.1.6.67@tcp:28/4 lens 224/224 e 0 to 1 dl 1452704484 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
[10449.345275] Lustre: lustre-OST0000-osc-ffff8800368af800: Connection to lustre-OST0000 (at 10.1.6.67@tcp) was lost; in progress operations using this service will wait for recovery to complete
[10449.346597] Lustre: Skipped 1 previous similar message
[10451.341056] Lustre: 7117:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1452704472/real 1452704472]  req@ffff88001b589b00 x1523260436267976/t0(0) o400->lustre-OST0001-osc-ffff8800368af800@10.1.6.67@tcp:28/4 lens 224/224 e 0 to 1 dl 1452704479 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
[10451.343500] Lustre: 7117:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 15 previous similar messages
[10451.344376] Lustre: lustre-OST0001-osc-ffff8800368af800: Connection to lustre-OST0001 (at 10.1.6.67@tcp) was lost; in progress operations using this service will wait for recovery to complete
[10451.345642] Lustre: Skipped 7 previous similar messages
[10456.346044] Lustre: 7118:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1452704477/real 1452704477]  req@ffff88001b589200 x1523260436268044/t0(0) o400->lustre-OST0001-osc-ffff8800368af800@10.1.6.67@tcp:28/4 lens 224/224 e 0 to 1 dl 1452704484 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
[10456.348407] Lustre: 7118:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 13 previous similar messages
[10467.346042] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1452704484/real 1452704484]  req@ffff88002a88e700 x1523260436268168/t0(0) o8->lustre-OST0000-osc-ffff8800368af800@10.1.6.67@tcp:28/4 lens 520/544 e 0 to 1 dl 1452704495 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
[10467.348313] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 11 previous similar messages
[10472.346045] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1452704489/real 0]  req@ffff88002a88f000 x1523260436268212/t0(0) o8->lustre-OST0001-osc-ffff8800368af800@10.1.6.67@tcp:28/4 lens 520/544 e 0 to 1 dl 1452704500 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
[10472.348250] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[10487.348063] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1452704499/real 0]  req@ffff88002a88d500 x1523260436268260/t0(0) o8->lustre-OST0000-osc-ffff8800368af800@10.1.6.67@tcp:28/4 lens 520/544 e 0 to 1 dl 1452704515 ref 2 fl Rpc:XN/0/ffffffff rc 0/-1
[10487.350107] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[10524.514309] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1452704549/real 1452704552]  req@ffff88002a88cf00 x1523260436268412/t0(0) o8->lustre-OST0001-osc-ffff8800368af800@10.1.6.67@tcp:28/4 lens 520/544 e 0 to 1 dl 1452704565 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
[10524.520275] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
[10561.346330] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1452704564/real 1452704564]  req@ffff88002a88d200 x1523260436268584/t0(0) o8->lustre-OST0000-osc-ffff8800368af800@10.1.6.67@tcp:28/4 lens 520/544 e 0 to 1 dl 1452704589 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
[10561.351317] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 57 previous similar messages
[10626.346595] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1452704654/real 1452704654]  req@ffff88002a88ea00 x1523260436269368/t0(0) o8->lustre-OST0000-osc-ffff8800368af800@10.1.6.67@tcp:28/4 lens 520/544 e 0 to 1 dl 1452704679 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
[10626.352846] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 117 previous similar messages
[10756.346646] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1452704784/real 1452704784]  req@ffff88002a88c300 x1523260436270876/t0(0) o8->lustre-OST0000-osc-ffff8800368af800@10.1.6.67@tcp:28/4 lens 520/544 e 0 to 1 dl 1452704809 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
[10756.351678] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 298 previous similar messages
[11016.346320] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1452705044/real 1452705044]  req@ffff88002a88db00 x1523260436273760/t0(0) o8->lustre-OST0000-osc-ffff8800368af800@10.1.6.67@tcp:28/4 lens 520/544 e 0 to 1 dl 1452705069 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
[11016.351233] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 564 previous similar messages
[11531.346322] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1452705559/real 1452705559]  req@ffff88002a88de00 x1523260436279432/t0(0) o8->lustre-OST0000-osc-ffff8800368af800@10.1.6.67@tcp:28/4 lens 520/544 e 0 to 1 dl 1452705584 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
[11531.352554] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 1108 previous similar messages
[12136.347472] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1452706164/real 1452706164]  req@ffff88002a88e100 x1523260436286396/t0(0) o8->lustre-OST0001-osc-ffff880038a97000@10.1.6.67@tcp:28/4 lens 520/544 e 0 to 1 dl 1452706189 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
[12136.351570] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 1368 previous similar messages
[12741.347621] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1452706769/real 1452706769]  req@ffff88002a88c000 x1523260436293316/t0(0) o8->lustre-OST0000-osc-ffff8800368af800@10.1.6.67@tcp:28/4 lens 520/544 e 0 to 1 dl 1452706794 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
[12741.351700] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 1375 previous similar messages
[13341.348089] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1452707369/real 1452707369]  req@ffff88002a88de00 x1523260436300420/t0(0) o8->lustre-OST0005-osc-ffff880038a97000@10.1.6.67@tcp:28/4 lens 520/544 e 0 to 1 dl 1452707394 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
[13341.351725] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 1413 previous similar messages
[13946.347428] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1452707974/real 1452707974]  req@ffff88002a88c000 x1523260436307284/t0(0) o8->lustre-OST0000-osc-ffff8800368af800@10.1.6.67@tcp:28/4 lens 520/544 e 0 to 1 dl 1452707999 ref 1 fl Rpc:XN/0/ffffffff rc 0/-1
[13946.351913] Lustre: 7116:0:(client.c:2047:ptlrpc_expire_one_request()) Skipped 1354 previous similar messages

Whamcloud Community JIRA

[LU-7684] Intern master<->2.7.1 - sanityn test_78: test failed to respond and timed out