Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-2816

sanity-benchmark test_bonnie slow after 4MB BRW RPC patch

Details

    • Bug
    • Resolution: Fixed
    • Blocker
    • Lustre 2.4.0
    • Lustre 2.4.0
    • 3
    • 6820

    Description

      This issue was created by maloo for sarah <sarah@whamcloud.com>

      This issue relates to the following test suite run: https://maloo.whamcloud.com/test_sets/91d8ae78-755b-11e2-bf59-52540035b04c.

      The sub-test test_bonnie failed with the following error:

      test failed to respond and timed out

      From the stdout log, I found the test ran extremely slow compare to the success run:

      It took almost 45 minutes for the failed test:

      16:44:39:== sanity-benchmark test bonnie: bonnie++ == 16:44:30 (1360629870)
16:44:39:min OST has 1809000kB available, using 3845408kB file size
16:44:39:debug=0
16:44:39:running as uid/gid/euid/egid 500/500/500/500, groups:
16:44:39: [touch] [/mnt/lustre/d0_runas_test/f16388]
16:44:39:running as uid/gid/euid/egid 500/500/500/500, groups:
16:44:39: [bonnie++] [-f] [-r] [0] [-s3755] [-n] [10] [-u500:500] [-d/mnt/lustre/d0.bonnie]
16:44:39:Using uid:500, gid:500.
16:47:56:Writing intelligently...done
17:29:18:Rewriting...done

      Only 9 minutes for the success test:

      01:31:59:== sanity-benchmark test bonnie: bonnie++ == 01:31:52 (1359624712)
01:31:59:min OST has 1783256kB available, using 3845408kB file size
01:31:59:debug=0
01:31:59:running as uid/gid/euid/egid 500/500/500/500, groups:
01:31:59: [touch] [/mnt/lustre/d0_runas_test/f22196]
01:31:59:running as uid/gid/euid/egid 500/500/500/500, groups:
01:31:59: [bonnie++] [-f] [-r] [0] [-s3755] [-n] [10] [-u500:500] [-d/mnt/lustre/d0.bonnie]
01:31:59:Using uid:500, gid:500.
01:35:21:Writing intelligently...done
01:40:56:Rewriting...done

      Attachments

        Issue Links

          is related to

          New Feature - A new feature of the product, which has yet to be developed. LU-1431 Support for larger than 1MB sequential I/O RPCs

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. LU-2957 Proc tnable to control RAS_INCREASE_STEP

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            [LU-2816] sanity-benchmark test_bonnie slow after 4MB BRW RPC patch
            pjones Peter Jones made changes -
            Fix Version/s New: Lustre 2.4.0 [ 10154 ]
            jay Jinshan Xiong (Inactive) made changes -
            Resolution New: Fixed [ 1 ]
            Status Original: In Progress [ 3 ] New: Resolved [ 5 ]
            green Oleg Drokin made changes -
            Link New: This issue is related to LU-2957 [ LU-2957 ]
            jay Jinshan Xiong (Inactive) made changes -
            Attachment New: Untitled.png [ 12286 ]
            jay Jinshan Xiong (Inactive) made changes -
            Attachment Original: Untitled.png [ 12283 ]
            jay Jinshan Xiong (Inactive) made changes -
            Comment [ I performed test benchmark on rosso. I think the 4MB patch can actually boost the write performance.

            |Threads|Write-1MB|ReWrite-1MB|Write-4MB|ReWrite-4MB|
            |1 |704|724|701|732|
            |4 |2235|2263|2452|2334|
            |16 |2482|2736|3354|3351|
            |32 |3021|2970|3297|3322|

            I performed the test with checksum turned off and writethrough cache disabled on the OSTs, which can lower the performance number 10%~20% so I think we should disable it by default. Also, I applied patch 5164 in my test.

            !http://jira.whamcloud.com/secure/attachment/12283/Untitled.png! ]
            jay Jinshan Xiong (Inactive) made changes -
            Attachment New: Untitled.png [ 12283 ]
            jay Jinshan Xiong (Inactive) made changes -
            Status Original: Open [ 1 ] New: In Progress [ 3 ]
            doug Doug Oucharek (Inactive) made changes -
            Assignee Original: WC Triage [ wc-triage ] New: Jinshan Xiong [ jay ]
            adilger Andreas Dilger made changes -
            Link New: This issue is related to LU-1431 [ LU-1431 ]

            People

              jay Jinshan Xiong (Inactive)
              maloo Maloo
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: