[LU-744] Single client's performance degradation on 2.1

• Type: Bug
• Resolution: Duplicate
• Priority: Critical
• Affects Version/s: Lustre 2.2.0, Lustre 2.3.0
• Severity: 3
• Assignee: Jinshan Xiong (Inactive)
• Reporter: Shuichi Ihara (Inactive)

Description

During the performance testing on lustre-2.1, I saw the single client's performance degradation on it.
Here is IOR results on the single cleints with 2.1 and also lustre-1.8.6.80 for comparing.
I ran IOR (IOR -t 1m -b 32g -w -r -vv -F -o /lustre/ior.out/file) on the single client with 1, 2, 4 and 8 processes.

Write(MiB/sec)
v1.8.6.80 v2.1
446.25 411.43
808.53 761.30
1484.18 1151.41
1967.42 1172.06

Read(MiB/sec)
v1.8.6.80 v2.1
823.90 595.71
1449.49 1071.76
2502.49 1517.79
3133.43 1746.30

Tested on same infrastracture(hardware and network). The client just turned off the checksum on both testing.

Issue Links

• is related to: LU-1408 single client's performance regression test
• is related to: LU-1413 difference of single client's performance between b2_1 and 2.1.2RC0
• is related to: LU-3321 2.x single thread/process throughput degraded from 1.8
• is related to: LU-141 port lustre client page cache shrinker back to clio
• is related to: LU-1201 Lustre crypto hash cleanup

Comments

[Jinshan Xiong (Inactive)]

Hi Ihara, I saw there is significant CPU usage for library mca_btl_sm.so(11.7%) and libopen-pal.so.0.0.0(4.7%) but the performance data shown on Sep 5 they only consumed 0.13% and 0.05%. They are openmpi libraries. Did you do any upgrade on these libraries?

Anyway, I revised patch 4519 and restored 4472 to remove memory stalls, please apply them in your next benchmark. However we have to figure out why openmpi libraries consumed so much cpu before seeing the performance improvement.

[Jinshan Xiong (Inactive)]

Frederik for delay response. From the test results, it looks there may be some issues with LU-2139 patches. You can see it in the collectl stats:

46 45 72551 10187 2G 10M 410M 399M 188M 162M 0 0 379904 371
47 47 74169 7109 2G 11M 786M 775M 269M 162M 0 0 385024 376
25 25 40289 4190 2G 11M 982M 971M 312M 162M 0 0 200704 196
6 6 8440 229 2G 11M 982M 971M 313M 162M 0 0 0 0
7 7 10545 249 2G 11M 982M 971M 313M 164M 0 0 0 0

.....

Unknown macro: {20 seconds later}

7 7 10639 241 2G 11M 983M 973M 311M 163M 0 0 0 0
9 8 12408 236 2G 11M 983M 973M 311M 163M 0 0 0 0
34 34 53022 4218 1G 11M 1G 1G 357M 163M 0 0 258048 252
50 50 77645 7414 1G 11M 1G 1G 447M 163M 0 0 422912 413

There were some IO activities for 2 or 3 seconds and then stay quiet for around 20 seconds and then do IO again. It seems like LRU budget was running out so OSC had to wait for commit on OST to be finished.

I will work on this. Thanks for testing.

[Shuichi Ihara (Inactive)]

Jinshan, tested master + patch 4519 on both servers and client, but it seems to be still same results.

[Frederik Ferner (Inactive)]

So far all these tests have been done with 2.3.0 on the servers. I've not tried 2.3.54 on any of my test servers yet. I'll try to find some time over the next few days.

[Andreas Dilger]

Frederik, I'm assuming for your test results that you are running the same version on both the client and aerver? Would it also be possible for you to test 2.3.0 clients with 2.3.54 servers and vice versa? That would allow us to isolate if the slowdown seen with 2.3.54 is due to changes in the client or server.

[Frederik Ferner (Inactive)]

Jinshan,

apologies for not providing the information from the start, I've also now realised that this might be better suited in a new ticket. So let me know if you prefer me to open a new ticket.

My current test setup is a small file system with all servers on Lustre 2.3, 2 OSSes, 6 OSTs in total (3 per OSS). All servers and test clients are attached via 10GigE. Network throughput has been tested and the test client can send with 1100MB/s to each server in turn using netperf. LNET selftest throughput also reaches 1100MB/s sending from one client to both servers at the same time.

I've now repeated a small test with ior and different version on the clients. The test client only has 4GB RAM, in my tests on 2.3.54 (master up to commit 8229702 with patches 4245,4374,4375,4471,4472) I can write small files relatively fast but 4GB files are slow. I've not tested reading as this is not my main concern at the moment. (I'm hoping to achieve 900MB/s sustained write speed over 10GigE from a single process to accomodate a new detector we will commission early next year, my hope was on 2.X clients to provide higher single thread performance than 1.8.)

Ior command used:
ior -o /mnt/play01/tmp/stripe-all/ior_dat -w -k -t1m -b 4g -i 1 -e

client details	write speed [MiB/s]
1.8.8, checksums on	487.61
1.8.8, checksums off	592.90
2.3.0, checksums on	440.36
2.3.0, checksums off	441.63
2.3.54+patches, checksums on	30.21
2.3.54+patches, checksums off	34.12
2.3.54+patches, checksums on, 1GB file	313.47

opreport and collectl output for all the tests with 4GB files are attached in lu744-dls-20121113.tar.gz

Let me know if you need anything else or if I need to run oprofile differently as I wasn't familiar with oprofile before.

[Jinshan Xiong (Inactive)]

Hi Ihara, I still saw high contention in cl_page_put and stats. Can you please try this patch 4519 where I disabled stats completely. For the cl_page_put() part, I will think about a way to solve it.

[Shuichi Ihara (Inactive)]

Hi Jinshan, just ran same testing after applied two patches (4471 and 4472) to the master. Please check all results and statistics.

[Jinshan Xiong (Inactive)]

Yes, only those two on master.

[Shuichi Ihara (Inactive)]

Jinshan, so, just two patches (4471 and 4472) to master is fine? then, collect stats during the IOR. no need to apply any patches to the master for this debuging, right?

[Jinshan Xiong (Inactive)]

Can you please describe the test env in detail and tell me the specific performance number before and after applying patch? Also, please collect performance data with oprofile and collectl as what Ihara did.

There are two new patches(4471 and 4472) I submitted yesterday, can you please also give it a try?