|
during testing, I saw high CPU usages with ptlrpcd-brw and kswapd process on 2.1. small kswapd is showed up on 1.8's testing, but it's not frequency. Howerver, with 2.1, kswapd is always high CPU usages.
note, I turned off the lustre checksum in this testing, so this is not cuased by checksum overhead.
|
|
A key difference between 2.1 and 1.8 is that there is no caching memory(max_dirty_mb) limitation in 2.1, this will cause high CPU usage of kswapd, but I'm not sure if this is the root cause of performance degradation for this IO intensive program. For read case, the first thing we need to know is the RPC size.
Can you please collect the following information on both read and write case for 1.8 and 2.1 specifically:
1. `vmstat 1' while running IOR;
2. `lctl get_param osc.lustre-OST*-osc-ffff*.rpc_stats' after the IOR is finished, make sure it's cleared before starting the test;
3. it will be perfect to have oprofile running during the test.
Thanks.
|
|
I wonder what's the raw speed capability of the link?
We have a caching bug in 1.8 that manifests itself as too fast reads if you just did the writes even if more writes than RAM.
Often read come out faster than the link speed - that's how we noticed.
|
|
The network between server and client is QDR Infiniband. So, numbers should be reasonable. Also, an client only has 12GB memory and I'm writing the larger data (256GB) than memory size - no cache effect on this.
btw, the servers (4 OSSs) have 16GB memory, but turned off the read cache. Even turn on it, the total file size (256GB) is still larger then amount of server's memory.
I'm going to collect the data Jay requested.
|
|
I understand your test size is bigger than the client RAM.
In our Oracle testing we found that there is a caching bug in 1.8 that leads to old write data not being discarded. So later on when you read the data, some significant part of it comes from client cache.
This may or may not contribute to the problem you are seeing of course and I see writes are also somewhat slower which could not be explained by that caching problem we saw.
Also just to confirm, this is 4x QDR, right that can have up to 4 gigabytes/sec of useful bandwidth.
Getting the data Jinshan requested is a good start indeed.
|
|
attached the all stats which I got on 2.1.
I did write and read separately on IOR benchmark and got vmstat, oprofile and rpc stats during the benchmark.
will run same benchamrk on 1.8.
|
|
Got vmstat and rpc_stats during IOR bencharmk with lsutre-1.8.
oprofile didn't work on this kernel.. due to the following error messages when I ran opreport.
opreport error: basic_string::_S_construct NULL not valid
|
|
plesae have a look at log files and oprofile, and let me know if you need more information.
|
|
Reassign to Jinshan
|
|
Tested again on the current master branch, the write number are a littile bit improved, but the read number is same and still big gap compared to 1.8.x. I think the current master code should have multiple ptlrpc threads, right? but it doesn't seem to affect yet for the single performance improvement.
|
|
From more testing and monitoring storage IO statistics, it looks like the performance is good if amount of file size < client's memory.
As total file size is close to client memory size and go beyond, client performance goes down.
|
|
we'll address this in 2.3 due to the io engine work taking place under LU-1030.
|
|
Can we confirm this is a client-side issue - e.g. by measuring 1.8 and 2.x clients v. 2.x servers?
|
|
To eeb: from what Ihara had seen, if the writing file size exceeded memory size, the performance dropped a lot. So I think this may be a client side. However, I didn't see any other report for this kind of issue, one reason may be this is not noticed or they can't generate this fast IO.
For LU-1030, I'm clarifying that it won't help fix this issue. It will change OSC behavior a lot so any attempt to address performance issue before that is no use.
Actually I have done performance benchmark many times but I didn't see this issue. I guess one reason would be I can't generate such high speed IO with the hardwares in our lab.
|
|
Here is what I demonstrated. Tested IOR on single client (12 therads) and collected memory size and Lustre throughput on an client during the IO. This is tested on Lustre-2.2RC2 for servers and client.
Once the client's memory usage reaches the memory size, lustre IO performance goes down.
Write Performance
Read Performance
|
|
This is another test results when servers are running with 2.2, but client is 1.8.7. (The checksum is diabled.)
write test
Read test
|
|
I guess this is because there is no LRU for async_pages in 2.x clients. The LRU mechanism is way too complex in 1.8 clients so I have an idea to limit # of caching pages at OSC layer.
|
|
I'm working on a workaround patch to limit the max caching pages per OSC.
|
|
Hi Ihara,
Can you please try patch at: http://review.whamcloud.com/2514 to see if it can help. Please notice that this patch is for debug purpose only and shouldn't be applied to production system. Also please collect memory usage statistic and oprofile results while you're running the test, thanks.
|
|
Here is the data I ran IOR file-per-process on hyperion.
Server: lustre 2.1.0/rhel5/x86_64
Client: lustre 2.2.0/rhel5/x86_64
lustre 1.8.7/rhel5/x86_64
number of client: 1
Transfer size: 1M
blocksize: 192G/#thread
filesize: 192G
Write:
Thread 1.8.7 2.2.0
1 178 245
2 296 328
4 568 434
8 608 492
12 778 498
16 677 489
Read:
Thread 1.8.7 2.2.0
1 154 250
2 272 329
4 393 426
8 402 447
12 400 454
16 391 455
|
|
Thank you, Minh, I guess fast IO is a necessity to reproduce this problem. How many OSS nodes are there on Hyperion and what's their peak IO speed?
|
|
I don't know what they are currently using, but we have more than enough hardware available to swamp one client's QDR IB link.
I know there are at least 18 (soon to be 20) NetApp 60-bay enclosures with dual controllers, so the hyperion folks can set up more Lustre servers if needed.
|
|
Jay,
Thank you for posting patches for testing/debugging.
I'm now traveling, but will test this patches quickly when I'm back.
|
|
On chaos4, we have 4 OSS, each have 2 LUNS connected to DDN 9550
Here is obdfilter-survey from 1 of the oss
Tue Apr 10 11:42:35 PDT 2012 Obdfilter-survey for case=disk from hyperion1155
ost 2 sz 33554432K rsz 1024K obj 2 thr 2 write 293.83 [ 133.86, 157.84] rewrite 284.56 [ 127.87, 149.85] read 307.96 [ 137.86, 165.83]
ost 2 sz 33554432K rsz 1024K obj 2 thr 4 write 414.07 [ 156.84, 271.72] rewrite 387.44 [ 131.86, 239.76] read 384.67 [ 155.84, 263.73]
ost 2 sz 33554432K rsz 1024K obj 2 thr 8 write 513.81 [ 142.85, 364.63] rewrite 480.19 [ 189.97, 272.73] read 437.79 [ 181.90, 249.75]
ost 2 sz 33554432K rsz 1024K obj 2 thr 16 write 545.77 [ 158.84, 363.63] rewrite 491.09 [ 151.84, 319.67] read 454.01 [ 205.79, 247.75]
ost 2 sz 33554432K rsz 1024K obj 2 thr 32 write 554.57 [ 126.87, 403.59] rewrite 469.79 [ 153.84, 306.69] read 427.92 [ 194.80, 233.76]
ost 2 sz 33554432K rsz 1024K obj 4 thr 4 write 359.17 [ 120.88, 252.74] rewrite 410.65 [ 151.85, 239.76] read 417.68 [ 169.83, 227.77]
ost 2 sz 33554432K rsz 1024K obj 4 thr 8 write 506.46 [ 160.84, 411.58] rewrite 506.73 [ 173.82, 324.67] read 459.12 [ 191.80, 255.74]
ost 2 sz 33554432K rsz 1024K obj 4 thr 16 write 587.72 [ 225.84, 362.63] rewrite 528.92 [ 178.82, 318.68] read 443.22 [ 197.80, 249.75]
ost 2 sz 33554432K rsz 1024K obj 4 thr 32 write 572.57 [ 190.81, 394.60] rewrite 497.53 [ 185.81, 316.68] read 400.52 [ 144.85, 331.66]
ost 2 sz 33554432K rsz 1024K obj 8 thr 8 write 476.39 [ 191.80, 268.75] rewrite 531.77 [ 202.79, 314.68] read 449.59 [ 180.82, 260.73]
ost 2 sz 33554432K rsz 1024K obj 8 thr 16 write 530.49 [ 187.81, 313.68] rewrite 529.15 [ 188.81, 323.67] read 463.28 [ 191.80, 253.74]
ost 2 sz 33554432K rsz 1024K obj 8 thr 32 write 572.40 [ 203.79, 335.66] rewrite 486.10 [ 162.83, 309.69] read 431.22 [ 189.81, 254.74]
ost 2 sz 33554432K rsz 1024K obj 16 thr 16 write 520.57 [ 194.80, 306.69] rewrite 469.09 [ 134.86, 291.70] read 460.47 [ 199.80, 254.74]
ost 2 sz 33554432K rsz 1024K obj 16 thr 32 write 600.63 [ 199.80, 355.64] rewrite 492.24 [ 127.87, 323.68] read 428.18 [ 103.89, 249.75]
ost 2 sz 33554432K rsz 1024K obj 32 thr 32 write 567.20 [ 225.77, 313.68] rewrite 457.29 [ 147.85, 296.70] read 428.57 [ 183.81, 236.76]
done!
|
|
attached are IOR resutls, memory usages and oprofile output when I ran IOR benchmark on the original 2.2 and patched 2.2.
tested on the single client (QDR Infiniband, 48GB memory) with 12 IOR threads. The amount of file size is 384GB (32GB x 12 thread).
|
|
Thanks for the test, Ihara.
Now each OSC uses 128M memory at maximum and this looks too much small in your case, especially for the read case.
Please try patch set 2(http://review.whamcloud.com/2514) where you can set how much memory will be used for cache, for example:
lctl set_param osc.<osc1>.max_cache_mb=256
to set osc1 will use 256M memory for caching.
Also, you forgot to tell oprofile where to find lustre object files so it couldn't interpret address to symbol name.
|
|
Hi Jay,
I just tested second patchset. It was tested with 256m, 512m, 1024m and 2048m for cache_mb.
It looks like still lower than the case when filesize < client's memory size. Attached includes ior results, memory usage and oprofile results on each testing.
Sorry, previous oprofile results, I was not pointing kernel modules. This one contains what you want?
|
|
Hi Ihara, thanks. Can you please refresh my memory for what was the performance of writing/reading a file less than memory size?
It looks like the performance data varied a lot from time to time, is this because different clients were used? In that case, it may make more sense to run patched/unpatched/b1_8, along with the case that file size is less than memory on the same kind of client to make comparison a bit easier.
Yes, this oprofile result is better, but it can be even better to print instruction address the cpu was busy on(I forgot the opreport option). However, I'm suppose contention on lock client_obd_list_lock would be significant in this case and I'm fixing this at LU-1030.
|
|
The previous 2.2 numbers, I did test 2.2 without patch on the same client and confirmed the performance is better if filesize < client's memory.
anyway, I just tested again various versions on the same hardware.
Server : lustre-2.2
Client : lustre-1.8.7.80, lustre-2.1.1 (w/patch and wo/patch), 2.2.0 (w/patch and wo/patch)
checksum is disabled.
lctl set_param osc.<osc1>.max_cache_mb=1024m (if we use patched client)
Here is results.
The test is simple. it runs IOR on single client with 12 Thread.
IOR.bin -o /lustre/ior.out/file -b 96g -t 1m -F -C -w -e -vv -k (write)
sync; echo 3 > /proc/sys/vm/drop_caches (all servers and client)
IOR.bin -o /lustre/ior.out/file -b ${fsize}g -t 1m -F -C -r -e -vv (read)
The attachment includes all IOR results, oprofile output and memory usages.
You can see the following results on 2.2.0/collectl.out.
If there is enough memory, Write speed is around 3GB/sec, but once it reaches the memory size, write speed goes down to 2GB/sec.
With patches, free memory is keeping, but write speed is around 2.3GB/sec.
|
|
In bug LU-1201 there is also http://jira.whamcloud.com/secure/attachment/11303/lustre-singleclient-comparison.xlsx with a good comparison of single-client performance with and without checksums enabled.
|
|
The problem I see with this patch is that it is moving in the wrong direction. Administrators want to be able to specify the cache limit for all Lustre filesystems on a node, while adding a cache limit per OSC doesn't really improve anything for them.
At a site like LLNL, they have over 3000 OSCs on the client, so any per-OSC limit will either have to be so small that it hurts performance, or it will be so large that it is much larger than the total RAM, and effectively no limit at all and only adding extra overhead to do useless LRU management.
I'd rather see more effort put into understanding why Lustre IO pages do not work well with the Linux VM page cache management, and fix that. This will provide global cache management, and avoid memory pressure for all users, and will also improve more as the Linux page cache management improves also.
|
|
From the oprofile and other stats, it was obvious that the CPU was busy evicting pages(kswapd used 100% CPU). Based on this situation, I think the problem is that the kswapd couldn't free caching pages as fast as the RPC engine wrote dirty pages back(otherwise, the writing process will be blocked at waiting for obd_dirty_pages). At last, there is no free pages in the system and writing processes are choked up freeing pages themselves and this degraded the write performance a lot. An obvious way to fix this problem is to free pages while the processes are producing them, this way we can distribute the overhead of freeing pages to every write syscall and also limit the total memory consumed by lustre. This is why I worked out this patch and I agree with the problem you mentioned so I didn't start the landing process. Anyway after this patch is used, the performance improved ~20% and I can no longer see 100% CPU time on kswapd.
However, this is based on my educated guess I'm not sure this is correct. Can you please elaborate on your idea and I will be pretty happy to verify and implement it? Thanks.
|
|
In the oprofile results during page eviction, are there any functions that show up as being very expensive that might be optimized? In the past there were code paths in CLIO that did too many expensive locking operations, and there may still be some paths that can be improved. My gut feeling is that we are keeping pages "active" somehow (references, pgcache_balance, etc) that makes it harder for the kernel to clear Lustre pages. I tried looking into this a bit, but there aren't very good statistics for seeing how many pages are in use (pgcache_balance is missing from 2.x clients, and dump_page_cache is too verbose).
Also, in the normal kernel code paths, I believe that kswapd is rarely doing page cleaning. Instead, this is normally handled on a per-block-device basis, so that it can be done in parallel on both the block devices and the CPUs. Is there some way that we could get ptlrpcd to do page cleaning itself, or re-introduce the per-CPU LRU as was done in 1.8?
|
|
From the oprofile result, the busiest part was osc_teardown_async_page() which should be called to destroy a caching page. I did find some env staff ate lots of CPU time as well, but it didn't help replace them by using journal_info to cache cl_env so I believe that is not a problem.
I don't think the # of dirty pages is the problem because they're limited by obd_dirty_pages and cl_dirty_max per osc, which is 384M at maximum on Ihara's node. So I guess the problem is that kswapd was evicting the caching pages too slow - remember that kswapd is per numa node(correct me if I'm wrong); in other words, if there existed per-cpu kswapd daemon, we couldn't see this problem at all.
Per_CPU LRU code was complex in 1.8, sorry about that because it was me who implemented it. So I just wanted to work out some thing simpler to address the problem - but all in all I have to know the patch does fix the problem(actually it did because from Ihara's test, the performance didn't drop if the file size was over memory size); based on this result, the next step will be to address the problem when there are many OSTs.
|
|
We have also identified this degradation of the single client's performance at Customer sites (Tera100 for instance) and in Bull's R&D lab, and would be interested in having a fix provided for b2_1.
Please, note that we can help testing new versions of a patch when available.
|
|
Hi Pichon, I'm working on this issue to make it be production ready.
As usual, before we're working on a new hardware, we have to understand what performance is with the current code, this way we can know if our fix is really working in the future.
Can you please do the performance benchmark with the following branches/patches:
1. 1.8
2. 2.1
3. 2.1 + LU-744 (patch 2929)
4. master
5. master + LU-744 (patch 2514)
Please be sure that the file size is far bigger than the memory size, and do the test with optimal block size(stripe_size * OST #) and different thread #.
|
|
Jinshan,
I will not be able to do the performance benchmark in the 1.8 release since this release has never been integrated in the Bull distribution.
However, we can still compare the single client's performance in lustre 2.1 without and with the patch, when application workload is small (application memory + application pagecache is less than client's memory) and large (twice client's memory).
When we will have integrated lustre master (in few weeks) we could do the same kind of tests.
By the way, I think Shuichi Ihara has provided comprehensive results in various branches. Would you need additional information or results to help your current developments ?
|
|
I think performance benchmark on master is necessary because new RPC engine is not landed in 2.1. This patch affects performance a lot so I think we should do performance improvment based on that patch.
The reason I also want the performance result on 1.8 is that people always compare the performance between 1.8 and 2.x so I guess it would be helpful to have that number.
Just in case, when I'm talking about different versions, I only mean the version on the client side. You can use the same version of server all the time, and you will need only one client to run those tests.
|
|
Sorry for long suspending on this work, but I will resume these benchmarks on this week.
Keep updates on here.
Thanks!
|
|
Hi Jay,
This is new test results. I did test on a Sandybrige Server, PCIgen3 and FDR which means we have
more bandwidth. But, we are seeing more big performance gap between b1_8 and master (and also b2_1).
This is still single client testing. Just ran 12 IOR threads on the single client
and each of thread write/read 84GB files. (total file size > 1TB.)
Please see attached. Here is quick summary.
1. b1_8 client regardless server is running with 2.1.2 or master, we can get mostly same performance
of b1_8 (server and client)
2. With 2.1.2 and master client, we got 42-55% performance regression
3. LU-744 patch for 2.1.2 didn't help
4. LU-744 patch for master helps a little bit, but still have 34% regressions
|
|
Ihara,
Thank you very much for the test results - this is helpful.
From the test result, the new RPC engine improved the performance significantly on your node; the purpose of LU-744 is to not make you frustrated when you're writing a file with size greater than the memory size  .
The next step for me is to generalize the LU-744 patch to make it work for large number of OSCs. I will check with senior engineers at whamcloud for the solution. After that, I will focus on the performance issue again.
|
|
I've pushed patch set 13 to http://review.whamcloud.com/2514, this patch should address the problem of having too many OSCs. Please give it a try.
|
|
So is this working going to go into 2.3?
|
|
James
We'd love to have this in 2.3 (and even 2.1.3) but we'll have to see when it is ready. At the moment we are still iterating to find a fix that is suitable for production.
Peter
|
|
Hi James,
I just pushed patch set 15 which is pretty close to production use. Please give it a try if you're interested.
|
|
Integrated into our image. Will do regression testing then a performance evaluation after.
|
|
Tested again with 1.8.8 and the master with latest LU-744 patches (patch set 16), but it was not big performance improvements compared to previous results.
|
|
Hi Ihara,
LLNL has seen a huge performance improvement with patch http://review.whamcloud.com/3627, can you please apply the patch along with LU-744 and see if it can help on your machine?
|
|
will test and update results soon. Thanks!
|
|
Hi Jay,
Tested the latest two LU-744 patches and LU-1666 on the single client (16CPUs and 64GB memory, FDR).
We saw some improvements with patches, but the overall behavior was same. The amount of file size (32GB) < client's memory size, we got 5.1GB/sec, but once no free memory on the client, the performance goes down. (2.7GB/sec)
I collected oprofile and collectl (memory usages and client's throughput) during the both IOR. (amount of file size is 32GB and 256GB)
|
|
from the opreport of 256gb, the aggregate of obdclass.ko consumes 37.3% CPU. However, it lacked function names I guess you missed some switches for opreport.
Usually I run opreport as follows:
opreport -alwdg -p /lib/modules/`uname -r`/updates/kernel/fs/lustre -s sample -o out.txt
|
|
We've noticed client memory swapping with 2.x causes significant performance loss. Attaching a graph of some "dd" operations against lustre, with and without sysctl vm.drop_caches=1 in between. Scales are memory bytes versus time.
|
|
Hi Nathan, there is LRU for llite pages in 1.x. So it would be interesting in figuring out whether it is lustre LRU or kernel page swapping process to free pages significantly in 1.8. Also it would be helpful to see what it will be with the patch in this ticket.
|
|
attached is re-tested results (the latest LU-744 and LU-1666 patches). write total 4 times larger files (256GB) of client's memory size. and it included fixed opreport outputs.
|
|
Hi Ihara, I pushed a combined patch at http://review.whamcloud.com/3924 with some changes to remove contention at cs_pages stats. Please benchmark it and collect stats of collectl and oprofile with switches -alwdgp. Thanks.
|
|
Hi Jay, Sure, I will test with latest RPMs and feedback you sooner.
|
|
above if client free memory size > 4GB, this is also really improved. 5.9GB/sec per client that is actually I got bandwidth between server and client on the RDMA bandwidth testing and lnet selftesting on FDR!
But, after the client exceeds the memory size, it goes down to 4GB/sec. This is also big improvements from previous results, but still slower than b1.8. I'm attaching all collected information. (collectl, ior results and opreport output)
|
5.9GB/sec per client that is actually I got bandwidth between server and client on the RDMA bandwidth testing and lnet selftesting on FDR!
But, after the client exceeds the memory size, it goes down to 4GB/sec.
Ihara, I think you & Jinshan just set a new record for single-client IO performance with Lustre. 
Looking at the memory usage, it does seem that most of the memory is in inactive, but doesn't even start to get cleaned up in the 7s it takes to fill the memory, let alone being cleaned up at the rate that Lustre is writing it. I'm assuming that the collectl output above for "Lustre Client" is real data RPCs sent over the network, since it definitely shouldn't be caching nearly so much data, so it isn't a case of "data going directly into cache, then getting slower when it starts writing out cache".
Also of interest is that the "Slab" usage is growing by 1GB/s. That is 1/6 of the memory used by the cache, and a sign of high memory overhead from Lustre for each page of dirty data and/or RPCs. While not directly related to this bug, if Lustre used less memory for itself it would delay the time before the memory ran out...
|
|
it looks good. Can you please add this patch and run the benchmark again: http://review.whamcloud.com/4001? Thanks.
|
|
Ihara, can you please tell me the configuration of llite.*.max_cached_mb?
|
Also of interest is that the "Slab" usage is growing by 1GB/s. That is 1/6 of the memory used by the cache, and a sign of high memory overhead from Lustre for each page of dirty data and/or RPCs. While not directly related to this bug, if Lustre used less memory for itself it would delay the time before the memory ran out...
I suspect this is because those lustre pages are still in the kernel's LRU cache even after OSCs try to discard them. So my recent patch tries to remove them out of kernel's LRU and free them voluntarily.
|
|
Jinshan, here is current max_cached_mb on the client. I will try your new patches.
- lctl get_param llite.*.max_cached_mb
llite.lustre-ffff88106a250c00.max_cached_mb=
users: 24
max_cached_mb: 48398
used_mb: 48398
unused_mb: 0
reclaim_count: 0
|
|
I see, please try the patch anyway though it may not help your case.
|
|
Jinshan,
I just applied new patches as well, but it didn't help very much. Attached test results after patches applied.
|
|
the above seems to be mostly about big streaming i/o. should I open a new bug for random i/o problems, or does it fit into this discussion?
I've been doing some 2.1.3 pre-rollout testing, and there seems to be a client problem with small random reads. performance is considerably worse on 2.1.3 clients than 1.8.8 clients. it's about a 35x slowdown for 4k random read i/o.
tests use the same files on a rhel6 x86_64 2.1.3 server (stock 2.1.3 rpm is used), QDR IB fabric, single disk or md8+2 lun for an OST, all client & server VFS caches dropped between trials.
checksums on or off and client rpc's 8 or 32 makes little difference. I've also tried umount'ing the fs from the 1.8.8 client between tests to make sure there's no hidden caching, but that didn't change anything.
random read ->
IOR -a POSIX -C -r -F -k -e -t $i -z -b 1024m -o /mnt/yo96/rjh/blah
| i/o size |
client version |
single disk lun |
md8+2 lun |
| i=4k |
1.8.8 |
20.70 MB/s |
22.0 MB/s |
| i=4k |
2.1.3 |
0.55 MB/s |
0.6 MB/s |
| i/o size |
client version |
single disk lun |
md8+2 lun |
| i=1M |
1.8.8 |
87 MB/s |
137 MB/s |
| i=1M |
2.1.3 |
63 MB/s |
83 MB/s |
although these numbers are for a single process, the same trend applies when the IOR is scaled up to 8 processes/node and to multiple nodes.
|
|
Robin, it would be better to file the random IO issue as a separate bug. This one is already very long and complex, and it is likely that the solution to the random IO performance will be different than what is being implemented here.
|
|
10-4. I've created LU-2032 for it.
|
|
Hi,
Here are the measurements I made with different Lustre versions.
I don't see any improvement with the patch in lustre 2.1.
In lustre 2.3, results are good. The patch gives more than 40% improvement.
By the way, I don't understand why the read performance is lower than write (although obdfilter performance is better in read than in write).
Hardware configuration:
30 OSTs
2 OSS (2 sockets, 16 cores, 32GB memory, 2xIB FDR, 4xFC8-2port)
1 Client (2 sockets, 16 cores, 32 GB memory, 1xIB FDR)
Software configuration:
OSS
lustre2.1.3 : lustre 2.1.3 + ORNL-22 + a few other patches
Client
lustre1.8.8-wc1 : standard lustre1.8.8-wc1 client rpms
lustre2.1.3 : lustre2.1.3 + ORNL-22 + a few other patches
lustre2.1.3+lu-744 : same as lustre2.1.3 plus patch #2929
lustre2.2.93 : lustre 2.2.93
lustre2.2.93+lu-744 : lustre 2.2.93 plus patches #3924 and #4001
In the lustre1.8.8-wc1, lustre2.1.3+lu-744 and lustre2.2.93+lu-744 configurations, I have left default value for max_cached_mb (24GiB).
IOR file per process, 16 processes, blockSize=4GiB, xfersize=1MiB, fsync=1.
This gives an aggregate filesize of 64 GiB.
With the last configuration I have results with several max_cached_mb tuning.
|
|
Gregoire, if it isn't too much to ask, could you please also try the current master client (2.3.53+). It already has the LU-744 patch landed, so it should be at least as fast as 2.2.93 + LU-744.
|
|
Here are the results with lustre 2.3.53+ (master up to patch a9444bc)
Results are not as good as version lustre2.2.93+lu-744 described above.
In fact, master does not include all the fixes nested in patches #3924 and #4001:
fix related to coh_page_guard (LU-1666 obdclass: reduce lock contention on coh_page_guard)
fix that removes some stats overhead (LU-744 osc: remove stats)
and fix that removes LRU pages voluntarily (LU-744 llite: remove LRU pages voluntarily)
|
|
For b2_1, probably we need new IO engine to boost performance. I will work out a production patch for removing stats so that we can get better performance on 2.3 and master.
|
|
Hi Jinshan, any updates on this or we can do something to see how much performance improvements?
|
|
There is one more patch needs productizing, I will finish it soon
|
|
I'm quite interest in these patches, as I'm currently trying to implement a file system where all traffic is via Ethernet, OSS attached with (dual bonded) 10GigE. A small number of clients connected via 10GigE should each be able to write with 900MB/s from a single stream. Currently with a 1.8.8 client writing to 2.3.0 OSSes and network checksums turned off, I get about 700MB/s. Upgrading the client to 2.3.0 I don't seem to get above 450MB/s, checksums don't make much difference here. (IOR 1M block size)
I've so far not had much luck trying the patches attached to this ticket without OOM on my client.
|
|
Hi Ihara, I pushed two patches to address the stats problem:
http://review.whamcloud.com/
{4471,4472}
Can you please give them a try? Please collect stats when you're running the patch, thanks.
Hi Frederik, can you please try patches http://review.whamcloud.com/
{4245,4374,4375}
, it may solve your problem if you're hitting the same one by LLNL.
|
|
Using those patches, I managed to compile a client from the git master branch and run my ior benchmark. It didn't improve performance but my client didn't suffer OOM either. I've not added any other patches on top of master (as off Monday evening: commit 82297027514416985a5557cfe154e174014804ba), as none of them seemed to apply cleanly. Were you expecting me to see higher performance? Are there any other patches I should test?
Frederik
|
|
Can you please describe the test env in detail and tell me the specific performance number before and after applying patch? Also, please collect performance data with oprofile and collectl as what Ihara did.
There are two new patches(4471 and 4472) I submitted yesterday, can you please also give it a try?
|
|
Jinshan, so, just two patches (4471 and 4472) to master is fine? then, collect stats during the IOR. no need to apply any patches to the master for this debuging, right?
|
|
Yes, only those two on master.
|
|
Hi Jinshan, just ran same testing after applied two patches (4471 and 4472) to the master. Please check all results and statistics.
|
|
Hi Ihara, I still saw high contention in cl_page_put and stats. Can you please try this patch 4519 where I disabled stats completely. For the cl_page_put() part, I will think about a way to solve it.
|
|
Jinshan,
apologies for not providing the information from the start, I've also now realised that this might be better suited in a new ticket. So let me know if you prefer me to open a new ticket.
My current test setup is a small file system with all servers on Lustre 2.3, 2 OSSes, 6 OSTs in total (3 per OSS). All servers and test clients are attached via 10GigE. Network throughput has been tested and the test client can send with 1100MB/s to each server in turn using netperf. LNET selftest throughput also reaches 1100MB/s sending from one client to both servers at the same time.
I've now repeated a small test with ior and different version on the clients. The test client only has 4GB RAM, in my tests on 2.3.54 (master up to commit 8229702 with patches 4245,4374,4375,4471,4472) I can write small files relatively fast but 4GB files are slow. I've not tested reading as this is not my main concern at the moment. (I'm hoping to achieve 900MB/s sustained write speed over 10GigE from a single process to accomodate a new detector we will commission early next year, my hope was on 2.X clients to provide higher single thread performance than 1.8.)
Ior command used:
ior -o /mnt/play01/tmp/stripe-all/ior_dat -w -k -t1m -b 4g -i 1 -e
| client details |
write speed [MiB/s] |
| 1.8.8, checksums on |
487.61 |
| 1.8.8, checksums off |
592.90 |
| 2.3.0, checksums on |
440.36 |
| 2.3.0, checksums off |
441.63 |
| 2.3.54+patches, checksums on |
30.21 |
| 2.3.54+patches, checksums off |
34.12 |
| 2.3.54+patches, checksums on, 1GB file |
313.47 |
opreport and collectl output for all the tests with 4GB files are attached in lu744-dls-20121113.tar.gz
Let me know if you need anything else or if I need to run oprofile differently as I wasn't familiar with oprofile before.
|
|
Frederik, I'm assuming for your test results that you are running the same version on both the client and aerver? Would it also be possible for you to test 2.3.0 clients with 2.3.54 servers and vice versa? That would allow us to isolate if the slowdown seen with 2.3.54 is due to changes in the client or server.
|
|
So far all these tests have been done with 2.3.0 on the servers. I've not tried 2.3.54 on any of my test servers yet. I'll try to find some time over the next few days.
|
|
Jinshan, tested master + patch 4519 on both servers and client, but it seems to be still same results.
|
|
Frederik for delay response. From the test results, it looks there may be some issues with LU-2139 patches. You can see it in the collectl stats:
46 45 72551 10187 2G 10M 410M 399M 188M 162M 0 0 379904 371
47 47 74169 7109 2G 11M 786M 775M 269M 162M 0 0 385024 376
25 25 40289 4190 2G 11M 982M 971M 312M 162M 0 0 200704 196
6 6 8440 229 2G 11M 982M 971M 313M 162M 0 0 0 0
7 7 10545 249 2G 11M 982M 971M 313M 164M 0 0 0 0
.....
Unknown macro: {20 seconds later}
7 7 10639 241 2G 11M 983M 973M 311M 163M 0 0 0 0
9 8 12408 236 2G 11M 983M 973M 311M 163M 0 0 0 0
34 34 53022 4218 1G 11M 1G 1G 357M 163M 0 0 258048 252
50 50 77645 7414 1G 11M 1G 1G 447M 163M 0 0 422912 413
There were some IO activities for 2 or 3 seconds and then stay quiet for around 20 seconds and then do IO again. It seems like LRU budget was running out so OSC had to wait for commit on OST to be finished.
I will work on this. Thanks for testing.
|
|
Hi Ihara, I saw there is significant CPU usage for library mca_btl_sm.so(11.7%) and libopen-pal.so.0.0.0(4.7%) but the performance data shown on Sep 5 they only consumed 0.13% and 0.05%. They are openmpi libraries. Did you do any upgrade on these libraries?
Anyway, I revised patch 4519 and restored 4472 to remove memory stalls, please apply them in your next benchmark. However we have to figure out why openmpi libraries consumed so much cpu before seeing the performance improvement.
|
|
Jinshan,
Yes, I upgraded MPI libbary a couple of weeks ago. I found a hardware problem and fixed it. Now mca_btl_sm_component_progress less consuming. it's still high compared to previous library though...
This attachment includes three test results
patches help less CPU consuming and improve the performance, but still drop the performance when the client is no free memory.
|
|
Jinshan, Frederik, When using the LU-2139 patches on the client but not on the server, it is normal to see the IO pause/stall as you are seeing. I'm not sure if this is happening for this this, but what can happen is:
1. Client performs IO
2. Client receives completion callback for bulk RPC
3. Bulk pages now clean but "unstable" (uncommitted on OST)
4. NR_UNSTABLE_NFS incremented for each unstable page (due to http://review.whamcloud.com/4245)
5. NR_UNSTABLE_NFS grows larger than (background_thresh + dirty_thresh)/2
6. Kernel stalls IO waiting for NR_UNSTABLE_NFS to decrease (via kernel function: balance_dirty_pages)
7. Client receives Lustre ping sometime in future (around 20 seconds later?), updating last_committed
8. Bulk pages now "stable" on client and can be reclaimed, lowering NR_UNSTABLE_NFS
9. Go back to step 1.
Reading the above comments, it looks like the LU-2139 patches are working as intended (avoiding OOMs at the cost of performance). Although I admit, the performance is terrible when you hit the NR_UNSTABLE_NFS limit and the kernel halts all IO (put is better than OOM, IMO). To improve on this, http://review.whamcloud.com/4375 needs to be applied to both clients and servers. This will allow the server to proactively commit bulk pages as they come in, hopefully preventing the client from exhausting its memory with unstable pages and avoiding the "stall" in balance_dirty_pages. With it applied to the server, I'd expect NR_UNSTABLE_NFS to remain "low", and the 4GB file speeds to reflect the 1GB speeds.
Please keep in mind, the LU-2139 patches are all experimental and subject to change.
On the client, with the LU-2139 patches applied, you might find it interesting to watch lctl get_param llite.*.unstable_stats and cat /proc/meminfo | grep NFS_Unstable as the test is running.
For example:
Those will give you an idea for the amount of unstable pages the client has at a given time. If that value gets "high" (exact value depends on your dirty limits, but probably around 1/4 of RAM) then what I detailed above is most likely the cause for the bad performance.
|
|
Hi Ihara, this is because CPU is still under contention so the performance dropped when the hosekeeping work started. Can you please run the benchmark one more time with patches 4519, 4472 and 4617. This should help a little bit.
|
|
There is a new patch for performance tune at: http://review.whamcloud.com/4943. Please give it a try.
|
|
My next patch will be to remove top cache of cl_page.
|
|
Jinshan,
I just tested http://review.whamcloud.com/4943
attached includes all results and oprofile output.
it looks obviously better than previous numbers. but I wonder if we could get more better performance since we are getting 5.6GB/sec sometimes. (see collectl.out) want to keep these around these numbers
|
|
It might help with interpreting the opreport data if the -p option is used. According the the opreport man page:
Without it, external module symbols don't get resolved:
Although the opreport-alwdg-p_lustre.out file seems to have all the useful bits.
|
|
CPU is still a bottleneck. The write speed dropped after OSC LRU cache stepped in and immediately drove the CPU usage to 100%. Let me see if I can optimize it.
|
|
Hi Ihara, what's the performance of b1_8 again on the same platform?
|
|
Ihara, could you please extract out the performance numbers for this patch and the previous ones in a small table like was done for the previous tests?
|
|
OK, tested again on client with b1_8, master mater+4943 patches, and from this test, I ran multiple iterations of IOR.
|
|
new test results includes b1_8, master and master+patch.
|
|
Jinshan,
What is the status of the patch http://review.whamcloud.com/#change,2929 you posted several months ago for b2_1 release ?
Why has it never been landed ?
I have made some measurements and results are significant: from 4% to 50% improvement depending on the platform I tested on.
Here are the results.
Hardware configuration:
30 OSTs
2 OSS : 4 sockets, 32 cores, 64GB memory, 2xIB, 4xFC8-2port
ClientA : 4 sockets Nehalem-EX, 32 cores, 64GB memory, 1xIB
ClientB : 2 sockets SandyBridge-EP, 16 cores, 64GB memory, 1xIB
Interconnect is QDR Infiniband
Software configuration:
kernel 2.6.32-220
lustre 2.1.3 + ORNL-22 + a few other patches
IOR file per process, blockSize=4GiB, xfersize=1MiB, fsync=1.
This gives an aggregate filesize of 120 GiB.
|
|
Gregoire, that's interesting. I wouldn't immediately expect #2929 to make much of a performance impact. How many iterations did you run? I'm curious if those numbers are within the natural variance of the test, or if they're actually because of the changes in #2929. Jinshan, would you expect performance to increase because of that patch?
|
|
Jinshan,
with http://review.whamcloud.com/4943 landed to master, are there any patches left to land under this bug, or can it be closed?
|
|
Andreas,
As far as I tested, 4943 helped perforamnce improveemnts, but even that patches applied, perforamnce is still lower than b1_8.
|
|
All patches have been landed. More work is also needed.
|
|
Test single client performance against 2.3.64 servers, versions tested: 1.8.8, 2.1.5,2.3.0,2.3.64
|
|
Cliff, what are server's CPU type, memory size? IOR options, file size. The performance depends client's specs, network and storage.
We are getting much better perforamnce on the current master and 1.8 client is still fast on some of numbers.
I will file the latest numbers here.
|
|
Servers are Intel Xeon, 64GB RAM. IOR options were taken from this bug, -t 1m -b 32g
Client had same/similar CPU, 64GB RAM
|
|
This should have been addressed by LU-3321
|
Generated at Sat Feb 10 01:10:00 UTC 2024 using Jira 9.4.14#940014-sha1:734e6822bbf0d45eff9af51f82432957f73aa32c.
2.4 Single Client 3May2013.xlsx
574.1.pdf
ior-256gb.tar.gz
stats-1.8.zip