[LU-1408] single client's performance regression test

• Type: Bug
• Resolution: Fixed
• Priority: Blocker
• Fix Version/s: Lustre 2.3.0, Lustre 2.6.0
• Environment: lustre-2.2, b2_1 and lustre-1.8.7; CentOS6.2 on both servers and clients
• Severity: 3
• Assignee: Hongchao Zhang
• Reporter: Shuichi Ihara (Inactive)

Description

This is a single client performance regression on 2.2 compared to 2.1.2 or 1.8.x.

I filed LU-744 before for another single client's performance regression, but that regression also happened on 2.1.x as well as 2.2 when the amount of file size is larger than client's memory size.

So, this regression might not be related to LU-744, but there is an another regression on 2.2 even if amount of file size is smaller than client's memory size.

Issue Links

• is related to: LU-969 2.1 client stack overruns
• is related to: LU-1413 difference of single client's performance between b2_1 and 2.1.2RC0
• is related to: LU-744 Single client's performance degradation on 2.1

Comments

[Hongchao Zhang]

Hi Ihara,

Have you tested the patch set 5 at http://review.whamcloud.com/#change,2889, which is a little different from patch set 2, thanks!

[Hongchao Zhang]

Hi Ihara,

the patch is updated, and could you please help to test patch set 5 on http://review.whamcloud.com/#change,2889? Thanks

[Hongchao Zhang]

Hi Ihara

could you please help to test the patch set 3 on http://review.whamcloud.com/#change,2889, the patch set 2 disables
the stack check, which is enabled for non x86-64 architecture, thanks!

[Shuichi Ihara (Inactive)]

I did test patch set 2 on http://review.whamcloud.com/#change,2889
After my testing is done and verified, I wondered if I could set "verified" flag as one of manual tester, but I couldn't find how.
Any advises how we can do that?

[Oleg Drokin]

Thanks for confirming the results.

Can you please tell me which patches did you tests? Just the one in gerrit?

[Shuichi Ihara (Inactive)]

ok, our system is back and tested LU-1408 patches.
Tested on an client (X5675, 3.07GHz, 48GB memory, QDR Infiniband), RHEL6.2, lustre-2.1.2-RC1 (both servers and client).

Confirmed the patches fix the performance regression problem. Here is test results.

===== without patch (original 2.1.2-RC1) =====
# mpirun -np 4 /root/IOR -b 8g -t 1m -F -C -w -e -k -vv -o /lustre/file

... snip ...

Max Write: 1401.19 MiB/sec (1469.26 MB/sec)

# mpirun -np 4 /root/IOR -b 8g -t 1m -F -C -r -e -vv -o /lustre/file

... snip ...

Max Read:  1510.69 MiB/sec (1584.07 MB/sec)

===== with LU-1408 patch =====
# mpirun -np 4 /root/IOR -b 8g -t 1m -F -C -w -e -k -vv -o /lustre/file

... snip ...

Max Write: 2578.54 MiB/sec (2703.80 MB/sec)

# pdsh -a "sync; echo 3 > /proc/sys/vm/drop_caches"

# mpirun -np 4 /root/IOR -b 8g -t 1m -F -C -r -e -vv -o /lustre/file

... snip ...

Max Read:  2663.67 MiB/sec (2793.06 MB/sec)

[James A Simmons]

Yes the fiber is QDR. Each node has 8 Intel Xeon CPU core (mode E5520) at 2.27GHz. Actually for this set of test I had checksums on. I just realized that. No matter; with checksum on or off the results are pretty close to each other.

[Shuichi Ihara (Inactive)]

hmm.. it seems to be pretty lower than I had even with patch. Server and clients are connected with QDR Infiniband? and how many CPU cores on the client? and disabled checksum just in case?

[Shuichi Ihara (Inactive)]

hmm.. it seems to be pretty lower than I had even with patch. Server and clients are connected with QDR Infiniband? and how many CPU cores on the client? and disabled checksum just in case?

[James A Simmons]

Finished a run against master (2.2.53). Cliff can you try running on a single client as well. The results I got were:

*************************************************************************************
No LU-1408 patch
*************************************************************************************

16 Threads one single node
Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)
--------- --------- --------- ---------- ------- --------- --------- ---------- ------- --------
write 607.24 568.29 596.03 14.05 151.81 142.07 149.01 3.51 110.01801
read 723.09 700.82 714.84 8.14 180.77 175.20 178.71 2.04 91.69101

Max Write: 607.24 MiB/sec (636.74 MB/sec)
Max Read: 723.09 MiB/sec (758.21 MB/sec)

*************************************************************************************
With LU-1408 patch
*************************************************************************************
16 Threads one single node
Operation Max (MiB) Min (MiB) Mean (MiB) Std Dev Max (OPs) Min (OPs) Mean (OPs) Std Dev Mean (s)
--------- --------- --------- ---------- ------- --------- --------- ---------- ------- --------
write 611.51 445.17 569.68 62.68 152.88 111.29 142.42 15.67 116.72391
read 771.25 746.29 757.22 9.74 192.81 186.57 189.30 2.44 86.56250

Max Write: 611.51 MiB/sec (641.21 MB/sec)
Max Read: 771.25 MiB/sec (808.71 MB/sec)

[James A Simmons]

Oh I see. Just like Cliff I was not seeing really big difference between with and without LU-1408 patch. Also like Cliff I was testing with multiple nodes of various thread counts per node. Thank for you for clarifying this. Will do another round of testing to see if this makes a difference.