[LU-744] Single client's performance degradation on 2.1

• Type: Bug
• Resolution: Duplicate
• Priority: Critical
• Affects Version/s: Lustre 2.2.0, Lustre 2.3.0
• Severity: 3
• Assignee: Jinshan Xiong (Inactive)
• Reporter: Shuichi Ihara (Inactive)

Description

During the performance testing on lustre-2.1, I saw the single client's performance degradation on it.
Here is IOR results on the single cleints with 2.1 and also lustre-1.8.6.80 for comparing.
I ran IOR (IOR -t 1m -b 32g -w -r -vv -F -o /lustre/ior.out/file) on the single client with 1, 2, 4 and 8 processes.

Write(MiB/sec)
v1.8.6.80 v2.1
446.25 411.43
808.53 761.30
1484.18 1151.41
1967.42 1172.06

Read(MiB/sec)
v1.8.6.80 v2.1
823.90 595.71
1449.49 1071.76
2502.49 1517.79
3133.43 1746.30

Tested on same infrastracture(hardware and network). The client just turned off the checksum on both testing.

Issue Links

• is related to: LU-1408 single client's performance regression test
• is related to: LU-1413 difference of single client's performance between b2_1 and 2.1.2RC0
• is related to: LU-3321 2.x single thread/process throughput degraded from 1.8
• is related to: LU-141 port lustre client page cache shrinker back to clio
• is related to: LU-1201 Lustre crypto hash cleanup

Comments

[Cliff White (Inactive)]

Test single client performance against 2.3.64 servers, versions tested: 1.8.8, 2.1.5,2.3.0,2.3.64

[Jinshan Xiong (Inactive)]

All patches have been landed. More work is also needed.

[Shuichi Ihara (Inactive)]

Andreas,

As far as I tested, 4943 helped perforamnce improveemnts, but even that patches applied, perforamnce is still lower than b1_8.

[Andreas Dilger]

Jinshan,
with http://review.whamcloud.com/4943 landed to master, are there any patches left to land under this bug, or can it be closed?

[Prakash Surya (Inactive)]

Gregoire, that's interesting. I wouldn't immediately expect #2929 to make much of a performance impact. How many iterations did you run? I'm curious if those numbers are within the natural variance of the test, or if they're actually because of the changes in #2929. Jinshan, would you expect performance to increase because of that patch?

[Gregoire Pichon]

Jinshan,

What is the status of the patch http://review.whamcloud.com/#change,2929 you posted several months ago for b2_1 release ?
Why has it never been landed ?

I have made some measurements and results are significant: from 4% to 50% improvement depending on the platform I tested on.

Here are the results.

Hardware configuration:
30 OSTs
2 OSS : 4 sockets, 32 cores, 64GB memory, 2xIB, 4xFC8-2port
ClientA : 4 sockets Nehalem-EX, 32 cores, 64GB memory, 1xIB
ClientB : 2 sockets SandyBridge-EP, 16 cores, 64GB memory, 1xIB
Interconnect is QDR Infiniband

Software configuration:
kernel 2.6.32-220
lustre 2.1.3 + ORNL-22 + a few other patches

IOR file per process, blockSize=4GiB, xfersize=1MiB, fsync=1.
This gives an aggregate filesize of 120 GiB.

          #tasks    write   read   configuration
ClientA       30     1121   1079   lustre 2.1.3
ClientA       30     1782   1413   lustre 2.1.3 + #2929

ClientB       16     2482   2149   lustre 2.1.3
ClientB       16     2616   2244   lustre 2.1.3 + #2929

[Shuichi Ihara (Inactive)]

new test results includes b1_8, master and master+patch.

[Shuichi Ihara (Inactive)]

OK, tested again on client with b1_8, master mater+4943 patches, and from this test, I ran multiple iterations of IOR.

Configuration
8 x OSS : 2 x E5-2670 (2.6GHz), 64GB memory, Centos6.3+master(2.3.58)/w FDR, total 32 OSTs
1 x Client : 2 x E5-2680 (2.7GHz), 64GB memory, Centos6.3/w FDR (tested with b1_8, master and master+patch as patchless client)

nproc=12
                   iteration=1   iteration=2   iteration=3
master(2.3.58)     3547 MiB/s    2754 MiB/s    2633 MiB/s
master+patch(4943) 3775 MiB/s    3407 MiB/s    2841 MiB/s  
b1_8               4212 MiB/s    4012 MiB/s    3750 MiB/s

nproc=16
                   iteration=1   iteration=2   iteration=3
master(2.3.58)     3617 MiB/s    3286 MiB/s    3149 MiB/s
master+patch(4943) 4077 MiB/s    3269 MiB/s    3511 MiB/s  
b1_8               4851 MiB/s    4255 MiB/s    4277 MiB/s

[Andreas Dilger]

Ihara, could you please extract out the performance numbers for this patch and the previous ones in a small table like was done for the previous tests?

[Jinshan Xiong (Inactive)]

Hi Ihara, what's the performance of b1_8 again on the same platform?

[Jinshan Xiong (Inactive)]

CPU is still a bottleneck. The write speed dropped after OSC LRU cache stepped in and immediately drove the CPU usage to 100%. Let me see if I can optimize it.