[LU-6919] replay-single test_70b: "Cannot send after transport endpoint shutdown" running dbench

• Type: Bug
• Resolution: Cannot Reproduce
• Priority: Minor
• Affects Version/s: Lustre 2.7.0
• Severity: 3
• Assignee: WC Triage
• Reporter: Aditya Pandit (Inactive)

Description

The test was executed for 50 iterations out of that it failed for 4.

fre0107: 1 6274 0.00 MB/sec execute 143 sec latency 198693.612 ms
fre0107: [6274] open ./clients/client0/~dmtmp/WORDPRO/BENCHS.LWP failed for handle 11182 (Cannot send after transport endpoint shutdown)
fre0107: [6277] open ./clients/client0/~dmtmp/WORDPRO/BENCHS.LWP failed for handle 11183 (Cannot send after transport endpoint shutdown)
fre0107: (6278) ERROR: handle 11183 was not found
fre0107: Child failed with status 1
fre0107: status script Total(sec) E(xcluded) S(low)
fre0107: ------------------------------------------------------------------------------------
fre0107:
fre0107: touch: missing file operand
fre0107: Try `touch --help' for more information.
pdsh@fre0107: fre0107: ssh exited with exit code 1
fre0108: [6481] unlink ./clients/client0/~dmtmp/WORDPRO/BENCHS1.LWP failed (Cannot send after transport endpoint shutdown) - expected NT_STATUS_OK
fre0108: ERROR: child 0 failed at line 6481
fre0108: Child failed with status 1
fre0108: status script Total(sec) E(xcluded) S(low)
fre0108: ------------------------------------------------------------------------------------
fre0108:
fre0108: touch: missing file operand
fre0108: Try `touch --help' for more information.
pdsh@fre0107: fre0108: ssh exited with exit code 1
fre0108: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 182 sec
fre0107: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 182 sec
fre0108: dbench: no process killed
fre0107: dbench: no process killed
pdsh@fre0107: fre0108: ssh exited with exit code 1
pdsh@fre0107: fre0107: ssh exited with exit code 1
replay-single test_70b: @@@@@@ FAIL: dbench stopped on some of fre0107,fre0108!
Trace dump:
= /usr/lib64/lustre/tests/test-framework.sh:4732:error_noexit()
= /usr/lib64/lustre/tests/replay-single.sh:2080:test_70b()
= /usr/lib64/lustre/tests/test-framework.sh:5010:run_one()
= /usr/lib64/lustre/tests/test-framework.sh:5047:run_one_logged()
= /usr/lib64/lustre/tests/test-framework.sh:4864:run_test()
= /usr/lib64/lustre/tests/replay-single.sh:2101:main()
Dumping lctl log to /tmp/test_logs/1437990212/replay-single.test_70b.*.1437990422.log
fre0106: Warning: Permanently added 'fre0107,192.168.101.7' (RSA) to the list of known hosts.

fre0108: Warning: Permanently added 'fre0107,192.168.101.7' (RSA) to the list of known hosts.

fre0105: Warning: Permanently added 'fre0107,192.168.101.7' (RSA) to the list of known hosts.

fre0107: dbench: no process killed
fre0108: dbench: no process killed
pdsh@fre0107: fre0107: ssh exited with exit code 1
pdsh@fre0107: fre0108: ssh exited with exit code 1
replay-single test_70b: @@@@@@ FAIL: rundbench load on fre0107,fre0108 failed!
Trace dump:
= /usr/lib64/lustre/tests/test-framework.sh:4732:error_noexit()
= /usr/lib64/lustre/tests/test-framework.sh:4763:error()
= /usr/lib64/lustre/tests/replay-single.sh:2099:test_70b()
= /usr/lib64/lustre/tests/test-framework.sh:5010:run_one()
= /usr/lib64/lustre/tests/test-framework.sh:5047:run_one_logged()
= /usr/lib64/lustre/tests/test-framework.sh:4864:run_test()
= /usr/lib64/lustre/tests/replay-single.sh:2101:main()
Dumping lctl log to /tmp/test_logs/1437990212/replay-single.test_70b.*.1437990424.log
FAIL 70b (208s)

Issue Links

• is related to: LU-7265 replay-single test_70b timeout: NULL pointer dereference in __mutex_lock_slowpath
• is related to: LU-6844 replay-single test 70b failure: 'rundbench load on * failed!'
• is related to: LU-6935 replay-single test_70b FAIL: import is not in FULL state

Comments

[Aditya Pandit (Inactive)]

console output of all the machines.

[Andreas Dilger]

Link to LU-6844 because of similar failure, but it may have a different cause.

[Andreas Dilger]

Aditya, do you have the console logs from this test? It looks like the client has been evicted for some reason.

Also, it would be useful for you to comment about which role each of the fre0105-0107 nodes is playing (client, MDS, OSS) so that we don't have to guess what is happening.