[LU-1408] single client's performance regression test

• Type: Bug
• Resolution: Fixed
• Priority: Blocker
• Fix Version/s: Lustre 2.3.0, Lustre 2.6.0
• Environment: lustre-2.2, b2_1 and lustre-1.8.7; CentOS6.2 on both servers and clients
• Severity: 3
• Assignee: Hongchao Zhang
• Reporter: Shuichi Ihara (Inactive)

Description

This is a single client performance regression on 2.2 compared to 2.1.2 or 1.8.x.

I filed LU-744 before for another single client's performance regression, but that regression also happened on 2.1.x as well as 2.2 when the amount of file size is larger than client's memory size.

So, this regression might not be related to LU-744, but there is an another regression on 2.2 even if amount of file size is smaller than client's memory size.

Issue Links

• is related to: LU-969 2.1 client stack overruns
• is related to: LU-1413 difference of single client's performance between b2_1 and 2.1.2RC0
• is related to: LU-744 Single client's performance degradation on 2.1

Comments

[Shuichi Ihara (Inactive)]

Hi Hongchao,

Just tested the patch with 2.1.2-RC1. (since LU-969 patches were droped on 2.1.2-RC2), the latest patch seems to be OK.

2.1.2-RC0 without any patches
# mpirun -np 4 /work/tools/bin/IOR -b 8g -t 1m -F -C -w -e -k -vv -o /lustre/file

Max Write: 2537.56 MiB/sec (2660.83 MB/sec)

# pdsh -a "sync; echo 3 > /proc/sys/vm/drop_caches"

# mpirun -np 4 /work/tools/bin/IOR -b 8g -t 1m -F -C -r -k -vv -o /lustre/file

Max Read:  2848.48 MiB/sec (2986.85 MB/sec)

2.1.2-RC1 + patches
# mpirun -np 4 /work/tools/bin/IOR -b 8g -t 1m -F -C -w -e -k -vv -o /lustre/file

Max Write: 2468.03 MiB/sec (2587.92 MB/sec)

# pdsh -a "sync; echo 3 > /proc/sys/vm/drop_caches"

# mpirun -np 4 /work/tools/bin/IOR -b 8g -t 1m -F -C -r -k -vv -o /lustre/file

Max Read:  2881.06 MiB/sec (3021.01 MB/sec)

[Hongchao Zhang]

Hi Ihara, I have add you as one of the reviewers of the patch, you can add review feedback according to the test result, thanks!

[Shuichi Ihara (Inactive)]

Hongchao, sorry for delay... I will test patch soon. keep you updates here once my testing is done.
any advices I can add "manual test" flag on http://review.whamcloud.com/#change,2889, after test is done and if result is OK?

[Hongchao Zhang]

Hi Ihara,

Have you tested the patch set 5 at http://review.whamcloud.com/#change,2889, which is a little different from patch set 2, thanks!

[Hongchao Zhang]

Hi Ihara,

the patch is updated, and could you please help to test patch set 5 on http://review.whamcloud.com/#change,2889? Thanks

[Hongchao Zhang]

Hi Ihara

could you please help to test the patch set 3 on http://review.whamcloud.com/#change,2889, the patch set 2 disables
the stack check, which is enabled for non x86-64 architecture, thanks!

[Shuichi Ihara (Inactive)]

I did test patch set 2 on http://review.whamcloud.com/#change,2889
After my testing is done and verified, I wondered if I could set "verified" flag as one of manual tester, but I couldn't find how.
Any advises how we can do that?

[Oleg Drokin]

Thanks for confirming the results.

Can you please tell me which patches did you tests? Just the one in gerrit?

[Shuichi Ihara (Inactive)]

ok, our system is back and tested LU-1408 patches.
Tested on an client (X5675, 3.07GHz, 48GB memory, QDR Infiniband), RHEL6.2, lustre-2.1.2-RC1 (both servers and client).

Confirmed the patches fix the performance regression problem. Here is test results.

===== without patch (original 2.1.2-RC1) =====
# mpirun -np 4 /root/IOR -b 8g -t 1m -F -C -w -e -k -vv -o /lustre/file

... snip ...

Max Write: 1401.19 MiB/sec (1469.26 MB/sec)

# mpirun -np 4 /root/IOR -b 8g -t 1m -F -C -r -e -vv -o /lustre/file

... snip ...

Max Read:  1510.69 MiB/sec (1584.07 MB/sec)

===== with LU-1408 patch =====
# mpirun -np 4 /root/IOR -b 8g -t 1m -F -C -w -e -k -vv -o /lustre/file

... snip ...

Max Write: 2578.54 MiB/sec (2703.80 MB/sec)

# pdsh -a "sync; echo 3 > /proc/sys/vm/drop_caches"

# mpirun -np 4 /root/IOR -b 8g -t 1m -F -C -r -e -vv -o /lustre/file

... snip ...

Max Read:  2663.67 MiB/sec (2793.06 MB/sec)

[James A Simmons]

Yes the fiber is QDR. Each node has 8 Intel Xeon CPU core (mode E5520) at 2.27GHz. Actually for this set of test I had checksums on. I just realized that. No matter; with checksum on or off the results are pretty close to each other.

[Shuichi Ihara (Inactive)]

hmm.. it seems to be pretty lower than I had even with patch. Server and clients are connected with QDR Infiniband? and how many CPU cores on the client? and disabled checksum just in case?