[LU-15561] sanity-benchmark test iozone fails with “write: Cannot send after transport endpoint shutdown”

• Status: Open
• Project: Lustre
• Affects Version/s: Lustre 2.15.0, Lustre 2.15.3
• Type: Bug
• Priority: Minor
• Reporter: James Nunez (Inactive)
• Assignee: WC Triage
• Resolution: Unresolved
• Severity: 3

Description

sanity-benchmark test_iozone fails with the following seen in the suite_log for Lustre 2.14.57.73 at https://testing.whamcloud.com/test_sets/416541cf-8753-464e-bd88-e3b28a7a51ec

	Run began: Wed Feb  2 19:38:38 2022

	Include fsync in write timing
	>>> I/O Diagnostic mode enabled. <<<
	Performance measurements are invalid in this mode.
	Record Size 512 kB
	File size set to 5785592 kB
	Command line used: iozone -i 0 -i 1 -i 2 -e -+d -r 512 -s 5785592 -f /mnt/lustre/d0.iozone/iozone
	Output is in kBytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 kBytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride                                    
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
         5785592     512
Error writing block 4617, fd= 3
write: Cannot send after transport endpoint shutdown

iozone: interrupted

exiting iozone

 sanity-benchmark test_iozone: @@@@@@ FAIL: iozone (1) failed 
  Trace dump:
  = /usr/lib64/lustre/tests/test-framework.sh:6386:error()
  = /usr/lib64/lustre/tests/sanity-benchmark.sh:129:test_iozone()
 

The first time I can find this test failing with the ‘transport endpoint’ error in the logs is on December 3, 2021 for Lustre 2.14.55.169 with logs at https://testing.whamcloud.com/test_sets/f1801fa9-12c2-4fcd-89b5-64278457c303 .

We see this for interop testing. For example for Lustre 2.12.7 servers and 2.15.0.RC2 clients at https://testing.whamcloud.com/test_sets/02aea16e-fddb-43ce-bde8-0a2041dd8b85