[LU-744] Single client's performance degradation on 2.1

• Type: Bug
• Resolution: Duplicate
• Priority: Critical
• Affects Version/s: Lustre 2.2.0, Lustre 2.3.0
• Severity: 3
• Assignee: Jinshan Xiong (Inactive)
• Reporter: Shuichi Ihara (Inactive)

Description

During the performance testing on lustre-2.1, I saw the single client's performance degradation on it.
Here is IOR results on the single cleints with 2.1 and also lustre-1.8.6.80 for comparing.
I ran IOR (IOR -t 1m -b 32g -w -r -vv -F -o /lustre/ior.out/file) on the single client with 1, 2, 4 and 8 processes.

Write(MiB/sec)
v1.8.6.80 v2.1
446.25 411.43
808.53 761.30
1484.18 1151.41
1967.42 1172.06

Read(MiB/sec)
v1.8.6.80 v2.1
823.90 595.71
1449.49 1071.76
2502.49 1517.79
3133.43 1746.30

Tested on same infrastracture(hardware and network). The client just turned off the checksum on both testing.

Issue Links

• is related to: LU-1408 single client's performance regression test
• is related to: LU-1413 difference of single client's performance between b2_1 and 2.1.2RC0
• is related to: LU-3321 2.x single thread/process throughput degraded from 1.8
• is related to: LU-141 port lustre client page cache shrinker back to clio
• is related to: LU-1201 Lustre crypto hash cleanup

Comments

[Peter Jones]

This should have been addressed by LU-3321

[Cliff White (Inactive)]

Servers are Intel Xeon, 64GB RAM. IOR options were taken from this bug, -t 1m -b 32g
Client had same/similar CPU, 64GB RAM

[Shuichi Ihara (Inactive)]

Cliff, what are server's CPU type, memory size? IOR options, file size. The performance depends client's specs, network and storage.
We are getting much better perforamnce on the current master and 1.8 client is still fast on some of numbers.
I will file the latest numbers here.

[Cliff White (Inactive)]

Test single client performance against 2.3.64 servers, versions tested: 1.8.8, 2.1.5,2.3.0,2.3.64

[Jinshan Xiong (Inactive)]

All patches have been landed. More work is also needed.

[Shuichi Ihara (Inactive)]

Andreas,

As far as I tested, 4943 helped perforamnce improveemnts, but even that patches applied, perforamnce is still lower than b1_8.

[Andreas Dilger]

Jinshan,
with http://review.whamcloud.com/4943 landed to master, are there any patches left to land under this bug, or can it be closed?

[Prakash Surya (Inactive)]

Gregoire, that's interesting. I wouldn't immediately expect #2929 to make much of a performance impact. How many iterations did you run? I'm curious if those numbers are within the natural variance of the test, or if they're actually because of the changes in #2929. Jinshan, would you expect performance to increase because of that patch?

[Gregoire Pichon]

Jinshan,

What is the status of the patch http://review.whamcloud.com/#change,2929 you posted several months ago for b2_1 release ?
Why has it never been landed ?

I have made some measurements and results are significant: from 4% to 50% improvement depending on the platform I tested on.

Here are the results.

Hardware configuration:
30 OSTs
2 OSS : 4 sockets, 32 cores, 64GB memory, 2xIB, 4xFC8-2port
ClientA : 4 sockets Nehalem-EX, 32 cores, 64GB memory, 1xIB
ClientB : 2 sockets SandyBridge-EP, 16 cores, 64GB memory, 1xIB
Interconnect is QDR Infiniband

Software configuration:
kernel 2.6.32-220
lustre 2.1.3 + ORNL-22 + a few other patches

IOR file per process, blockSize=4GiB, xfersize=1MiB, fsync=1.
This gives an aggregate filesize of 120 GiB.

          #tasks    write   read   configuration
ClientA       30     1121   1079   lustre 2.1.3
ClientA       30     1782   1413   lustre 2.1.3 + #2929

ClientB       16     2482   2149   lustre 2.1.3
ClientB       16     2616   2244   lustre 2.1.3 + #2929

[Shuichi Ihara (Inactive)]

new test results includes b1_8, master and master+patch.