Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-3321

2.x single thread/process throughput degraded from 1.8

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Lustre 2.4.0
    • Fix Version/s: Lustre 2.6.0
    • Labels:
    • Environment:
      Tested on 2.3.64 and 1.8.9 clients with 4 OSS x 3 - 32 GB OST ramdisks
    • Epic/Theme:
    • Severity:
      3
    • Rank (Obsolete):
      8259

      Description

      Single thread/process throughput on tag 2.3.64 is degraded from 1.8.9 and significantly degraded when the client hits its caching limit (llite.*.max_cached_mb). Attached graph shows lnet stats sampled every second for a single dd writing 2 - 64 GB files followed by a dropping cache and reading the same two files. The tests were not done simultaenously but the graph has them starting from the same point. It also takes a significant amount of time to drop the cache on 2.3.64.

      Lustre 2.3.64
      Write (dd if=/dev/zero of=testfile bs=1M)
      68719476736 bytes (69 GB) copied, 110.459 s, 622 MB/s
      68719476736 bytes (69 GB) copied, 147.935 s, 465 MB/s

      Drop caches (echo 1 > /proc/sys/vm/drop_caches)
      real 0m43.075s

      Read (dd if=testfile of=/dev/null bs=1M)
      68719476736 bytes (69 GB) copied, 99.2963 s, 692 MB/s
      68719476736 bytes (69 GB) copied, 142.611 s, 482 MB/s

      Lustre 1.8.9
      Write (dd if=/dev/zero of=testfile bs=1M)
      68719476736 bytes (69 GB) copied, 63.3077 s, 1.1 GB/s
      68719476736 bytes (69 GB) copied, 67.4487 s, 1.0 GB/s

      Drop caches (echo 1 > /proc/sys/vm/drop_caches)
      real 0m9.189s

      Read (dd if=testfile of=/dev/null bs=1M)
      68719476736 bytes (69 GB) copied, 46.4591 s, 1.5 GB/s
      68719476736 bytes (69 GB) copied, 52.3635 s, 1.3 GB/s

        Attachments

        1. dd_throughput_comparison.png
          dd_throughput_comparison.png
          6 kB
        2. dd_throughput_comparison_with_change_5446.png
          dd_throughput_comparison_with_change_5446.png
          7 kB
        3. mcm8_wcd.png
          mcm8_wcd.png
          9 kB
        4. perf3.png
          perf3.png
          103 kB
        5. lu-3321-singlethreadperf.tgz
          391 kB
        6. lu-3321-singlethreadperf2.tgz
          564 kB
        7. cpustat.scr
          0.5 kB

          Issue Links

            Activity

              People

              Assignee:
              jay Jinshan Xiong (Inactive)
              Reporter:
              jfilizetti Jeremy Filizetti
              Votes:
              0 Vote for this issue
              Watchers:
              23 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: