https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4720 Lustre <p>Hi,</p> <p>With the above environment, need suggestion on performance at the clients, since i'm struck. Appreciate your help.</p> <p>I'm getting the block device write performance of 9.6GB/s with 16 LUNs which is measured through XDD. I ran obdfilter survey at the OSS machines and getting around 8.4GB/s as write performance. I've measured the LNET performance and getting 9.6GB/s between OSS machines and 8 clients. which But when I run IOR in the clients, i'm getting around 2.6GB/s for write performance with one client. While I run it across two nodes, getting 4.4GB/s of write throughput. But while I scale beyond 2nodes, getting the same performance of 4GB/s only. Could you please help to find the root cause for this performance availability issue.</p> MDS Servers(2nos) - 12cores, 64GB Memory, QDR Single port<br/> OSS Servers (2nos) - 16cores, 64GB Memory, FDR multi rail(2ports from each OSS)<br/> Clients(8nos) - 12cores, 64GB memory, QDR single port<br/> Lustre Version - 2.5<br/> CentOS version - 6.4<br/> No. of OSTs - 16(Configured in RAID-6(8+2)) and load balanced between OSS servers with 8 OSTs on each<br/> LU-4720

Help on performance scalability

Improvement Minor Open Unresolved WC Triage Chakravarthy Nagarajan Thu, 6 Mar 2014 03:58:43 +0000 Wed, 12 Mar 2014 17:23:14 +0000 Lustre 2.5.0 0 7 <p>How many numbers for threads are you using per client? </p> <p>I've set the no. of threads in OSS as 256. Means allocated 32 threads per client. Please advise.</p> <p>Can you try 6 threads per client? </p> <p>It has reduced the performance with 2 clients by 50% and the scalability issue still remains while I run with 4 clients. I've initially set the no.of threads according to the total no. of spindles. But still no luck. Do you think Metadata may an issue, since i've 2 MDts configured in RAID-1 only instead of RAID-1+0. </p> <p>What is your single thread performance when ran on a single client? If you have 6 threads and 1GB/s that seems a little odd. Are you running IOR with one file per process or with a single file? </p> <p>Are you using DNE to have 2 metadata targets? IOR is not metadata intensive so it should not be a serious factor. </p> <p>On a single client i'm getting 2.2GB/s. Yes i'm using IOR with N-N only and also using DNE with 2 MDTs.</p> <p>The MDT doesn't have anything to do with IO performance. The metadata on the MDS has nothing to do with block allocation.</p> <p>Thanks and even i've realized the same by monitoring the MDS utilization. I've tried the following, but no luck. Please advise.<br/> 1. Disable checksum at the clients<br/> 2. Increase RPCs in flight to 32 at the cleints<br/> 3. Disable LRu re-sizing at the clients<br/> 4. Set readahead_max_file_size to 1M at the oss machines<br/> 5. Tested with multiple thread counts upto 512 at oss machines.</p> <p>The issues is obdfilter ran at oss machines resulting 8.4GB/s, but clients are unable to get the same.</p> Development Rank 1|hzwgw7: Rank (Obsolete) 12974