Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-15561

sanity-benchmark test iozone fails with “write: Cannot send after transport endpoint shutdown”

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Minor
    • None
    • Lustre 2.15.0, Lustre 2.15.3
    • None
    • 3
    • 9223372036854775807

    Description

      sanity-benchmark test_iozone fails with the following seen in the suite_log for Lustre 2.14.57.73 at https://testing.whamcloud.com/test_sets/416541cf-8753-464e-bd88-e3b28a7a51ec

      	Run began: Wed Feb  2 19:38:38 2022
      
      	Include fsync in write timing
      	>>> I/O Diagnostic mode enabled. <<<
      	Performance measurements are invalid in this mode.
      	Record Size 512 kB
      	File size set to 5785592 kB
      	Command line used: iozone -i 0 -i 1 -i 2 -e -+d -r 512 -s 5785592 -f /mnt/lustre/d0.iozone/iozone
      	Output is in kBytes/sec
      	Time Resolution = 0.000001 seconds.
      	Processor cache size set to 1024 kBytes.
      	Processor cache line size set to 32 bytes.
      	File stride size set to 17 * record size.
                                                                    random    random     bkwd    record    stride                                    
                    kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
               5785592     512
      Error writing block 4617, fd= 3
      write: Cannot send after transport endpoint shutdown
      
      iozone: interrupted
      
      exiting iozone
      
       sanity-benchmark test_iozone: @@@@@@ FAIL: iozone (1) failed 
        Trace dump:
        = /usr/lib64/lustre/tests/test-framework.sh:6386:error()
        = /usr/lib64/lustre/tests/sanity-benchmark.sh:129:test_iozone()
       

      The first time I can find this test failing with the ‘transport endpoint’ error in the logs is on December 3, 2021 for Lustre 2.14.55.169 with logs at https://testing.whamcloud.com/test_sets/f1801fa9-12c2-4fcd-89b5-64278457c303 .

      We see this for interop testing. For example for Lustre 2.12.7 servers and 2.15.0.RC2 clients at https://testing.whamcloud.com/test_sets/02aea16e-fddb-43ce-bde8-0a2041dd8b85

      Attachments

        Issue Links

          Activity

            People

              wc-triage WC Triage
              jamesanunez James Nunez (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: