Uploaded image for project: 'Lustre'
  1. Lustre
  2. LU-66

obdfilter-survey performance issue on NUMA system

Details

    • Improvement
    • Resolution: Fixed
    • Minor
    • Lustre 2.1.0
    • Lustre 2.1.0
    • None
    • 22,980
    • 8541

    Description

      this is just copy of bug 22980, but I think it's better to track & discuss it at here:

      Hello,

      Testing our new IO servers we have an issue with obdfilter-survey. Our OSSs are based on 4
      Nehalem-EX processors, connected to a Boxboro chipset. Every socket has 6 cores. On every OST we
      have several FC channels connected to our storage bay.

      When we perform raw tests with sgpdd-survey, over 24 luns we get ~4400 MB/s on write and more than
      5500 MB/s on read.

      Then if we start a Lustre filesystem and we test these 24 osts with obdfilter-survey (size=24192
      rszlo=1024 rszhi=1024 nobjlo=1 nobjhi=2 thrlo=1 thrhi=16 case=disk tests_str="write read" sh
      obdfilter-survey) we always have a performance limit on 1200 MB/s for write and read.

      If we perform IOzone tests from five clients (2 threads per client, connected to the server with
      Infiniband) we get more than 2500 MB/s.

      Then we disconnected two sockets using command "echo 0 > /sys/devices/system/cpu/cpu5/online" on
      every cpu belonging to these two sockets and we get expected results on obdfilter-survey (4600 MB/s
      on write and 5500 MB/s on read). If we only disconnect one socket then obdfilter-survey gives us a
      max of 1600 MB/s. Using only one socket results are slightly worse than with two sockets.

      We also made these tests with Lustre 1.6, with other storage bays and with similar platforms (4
      sockets and 8 cpus per socket) having always the same kind of problem. If we activate the
      hyper-threading functionality on every socket then performances are even worse.

      It's like if obdfilter-survey has any kind of saturation when there are many sockets. What do you
      think? Thanks,

      Attachments

        1. affinity_map
          0.2 kB
        2. affinity_results.tgz
          465 kB
        3. bull_obdfilter_survey_chart_110309.pdf
          65 kB
        4. bull_obdfilter_survey_chart_110319.pdf
          66 kB
        5. full_results_kmalloc.tgz
          346 kB
        6. full_results.tgz
          727 kB
        7. lctl_setaffinity_v2.patch
          4 kB
        8. new_results_kmalloc.tgz
          78 kB
        9. obdfilter-survey_results.txt
          17 kB
        10. remove_vmalloc.patch
          3 kB

        Activity

          People

            niu Niu Yawei (Inactive)
            liang Liang Zhen (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: