[LU-744] Single client's performance degradation on 2.1

• Type: Bug
• Resolution: Duplicate
• Priority: Critical
• Affects Version/s: Lustre 2.2.0, Lustre 2.3.0
• Severity: 3
• Assignee: Jinshan Xiong (Inactive)
• Reporter: Shuichi Ihara (Inactive)

Description

During the performance testing on lustre-2.1, I saw the single client's performance degradation on it.
Here is IOR results on the single cleints with 2.1 and also lustre-1.8.6.80 for comparing.
I ran IOR (IOR -t 1m -b 32g -w -r -vv -F -o /lustre/ior.out/file) on the single client with 1, 2, 4 and 8 processes.

Write(MiB/sec)
v1.8.6.80 v2.1
446.25 411.43
808.53 761.30
1484.18 1151.41
1967.42 1172.06

Read(MiB/sec)
v1.8.6.80 v2.1
823.90 595.71
1449.49 1071.76
2502.49 1517.79
3133.43 1746.30

Tested on same infrastracture(hardware and network). The client just turned off the checksum on both testing.

Issue Links

• is related to: LU-1408 single client's performance regression test
• is related to: LU-1413 difference of single client's performance between b2_1 and 2.1.2RC0
• is related to: LU-3321 2.x single thread/process throughput degraded from 1.8
• is related to: LU-141 port lustre client page cache shrinker back to clio
• is related to: LU-1201 Lustre crypto hash cleanup

Comments

[Jinshan Xiong (Inactive)]

Hi Ihara, this is because CPU is still under contention so the performance dropped when the hosekeeping work started. Can you please run the benchmark one more time with patches 4519, 4472 and 4617. This should help a little bit.

[Prakash Surya (Inactive)]

Jinshan, Frederik, When using the LU-2139 patches on the client but not on the server, it is normal to see the IO pause/stall as you are seeing. I'm not sure if this is happening for this this, but what can happen is:

1. Client performs IO
2. Client receives completion callback for bulk RPC
3. Bulk pages now clean but "unstable" (uncommitted on OST)
4. NR_UNSTABLE_NFS incremented for each unstable page (due to http://review.whamcloud.com/4245)
5. NR_UNSTABLE_NFS grows larger than (background_thresh + dirty_thresh)/2
6. Kernel stalls IO waiting for NR_UNSTABLE_NFS to decrease (via kernel function: balance_dirty_pages)
7. Client receives Lustre ping sometime in future (around 20 seconds later?), updating last_committed
8. Bulk pages now "stable" on client and can be reclaimed, lowering NR_UNSTABLE_NFS
9. Go back to step 1.

Reading the above comments, it looks like the LU-2139 patches are working as intended (avoiding OOMs at the cost of performance). Although I admit, the performance is terrible when you hit the NR_UNSTABLE_NFS limit and the kernel halts all IO (put is better than OOM, IMO). To improve on this, http://review.whamcloud.com/4375 needs to be applied to both clients and servers. This will allow the server to proactively commit bulk pages as they come in, hopefully preventing the client from exhausting its memory with unstable pages and avoiding the "stall" in balance_dirty_pages. With it applied to the server, I'd expect NR_UNSTABLE_NFS to remain "low", and the 4GB file speeds to reflect the 1GB speeds.

Please keep in mind, the LU-2139 patches are all experimental and subject to change.

On the client, with the LU-2139 patches applied, you might find it interesting to watch lctl get_param llite.*.unstable_stats and cat /proc/meminfo | grep NFS_Unstable as the test is running.

For example:

$ watch -n0.1 'lctl get_param llite.*.unstable_stats'
$ watch -n0.1 'cat /proc/meminfo | grep NFS_Unstable'

Those will give you an idea for the amount of unstable pages the client has at a given time. If that value gets "high" (exact value depends on your dirty limits, but probably around 1/4 of RAM) then what I detailed above is most likely the cause for the bad performance.

[Shuichi Ihara (Inactive)]

Jinshan,

Yes, I upgraded MPI libbary a couple of weeks ago. I found a hardware problem and fixed it. Now mca_btl_sm_component_progress less consuming. it's still high compared to previous library though...

This attachment includes three test results

1. master without any patches
2. master + 4519 (2nd patch) + 4472 (2nd patch)
3. master + 4519 (2nd patch) + 4472 (2nd patch) and run mpi with pthread, instead of shared memory.

patches help less CPU consuming and improve the performance, but still drop the performance when the client is no free memory.

[Jinshan Xiong (Inactive)]

Hi Ihara, I saw there is significant CPU usage for library mca_btl_sm.so(11.7%) and libopen-pal.so.0.0.0(4.7%) but the performance data shown on Sep 5 they only consumed 0.13% and 0.05%. They are openmpi libraries. Did you do any upgrade on these libraries?

Anyway, I revised patch 4519 and restored 4472 to remove memory stalls, please apply them in your next benchmark. However we have to figure out why openmpi libraries consumed so much cpu before seeing the performance improvement.

[Jinshan Xiong (Inactive)]

Frederik for delay response. From the test results, it looks there may be some issues with LU-2139 patches. You can see it in the collectl stats:

46 45 72551 10187 2G 10M 410M 399M 188M 162M 0 0 379904 371
47 47 74169 7109 2G 11M 786M 775M 269M 162M 0 0 385024 376
25 25 40289 4190 2G 11M 982M 971M 312M 162M 0 0 200704 196
6 6 8440 229 2G 11M 982M 971M 313M 162M 0 0 0 0
7 7 10545 249 2G 11M 982M 971M 313M 164M 0 0 0 0

.....

Unknown macro: {20 seconds later}

7 7 10639 241 2G 11M 983M 973M 311M 163M 0 0 0 0
9 8 12408 236 2G 11M 983M 973M 311M 163M 0 0 0 0
34 34 53022 4218 1G 11M 1G 1G 357M 163M 0 0 258048 252
50 50 77645 7414 1G 11M 1G 1G 447M 163M 0 0 422912 413

There were some IO activities for 2 or 3 seconds and then stay quiet for around 20 seconds and then do IO again. It seems like LRU budget was running out so OSC had to wait for commit on OST to be finished.

I will work on this. Thanks for testing.

[Shuichi Ihara (Inactive)]

Jinshan, tested master + patch 4519 on both servers and client, but it seems to be still same results.

[Frederik Ferner (Inactive)]

So far all these tests have been done with 2.3.0 on the servers. I've not tried 2.3.54 on any of my test servers yet. I'll try to find some time over the next few days.

[Andreas Dilger]

Frederik, I'm assuming for your test results that you are running the same version on both the client and aerver? Would it also be possible for you to test 2.3.0 clients with 2.3.54 servers and vice versa? That would allow us to isolate if the slowdown seen with 2.3.54 is due to changes in the client or server.

[Frederik Ferner (Inactive)]

Jinshan,

apologies for not providing the information from the start, I've also now realised that this might be better suited in a new ticket. So let me know if you prefer me to open a new ticket.

My current test setup is a small file system with all servers on Lustre 2.3, 2 OSSes, 6 OSTs in total (3 per OSS). All servers and test clients are attached via 10GigE. Network throughput has been tested and the test client can send with 1100MB/s to each server in turn using netperf. LNET selftest throughput also reaches 1100MB/s sending from one client to both servers at the same time.

I've now repeated a small test with ior and different version on the clients. The test client only has 4GB RAM, in my tests on 2.3.54 (master up to commit 8229702 with patches 4245,4374,4375,4471,4472) I can write small files relatively fast but 4GB files are slow. I've not tested reading as this is not my main concern at the moment. (I'm hoping to achieve 900MB/s sustained write speed over 10GigE from a single process to accomodate a new detector we will commission early next year, my hope was on 2.X clients to provide higher single thread performance than 1.8.)

Ior command used:
ior -o /mnt/play01/tmp/stripe-all/ior_dat -w -k -t1m -b 4g -i 1 -e

client details	write speed [MiB/s]
1.8.8, checksums on	487.61
1.8.8, checksums off	592.90
2.3.0, checksums on	440.36
2.3.0, checksums off	441.63
2.3.54+patches, checksums on	30.21
2.3.54+patches, checksums off	34.12
2.3.54+patches, checksums on, 1GB file	313.47

opreport and collectl output for all the tests with 4GB files are attached in lu744-dls-20121113.tar.gz

Let me know if you need anything else or if I need to run oprofile differently as I wasn't familiar with oprofile before.

[Jinshan Xiong (Inactive)]

Hi Ihara, I still saw high contention in cl_page_put and stats. Can you please try this patch 4519 where I disabled stats completely. For the cl_page_put() part, I will think about a way to solve it.

[Shuichi Ihara (Inactive)]

Hi Jinshan, just ran same testing after applied two patches (4471 and 4472) to the master. Please check all results and statistics.