https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1447 Lustre <p>Dear support, <br/> we are running Lustre 2.2 on our system and we noticed that also with a low usage of the file system, the MDS is always with a load avarage around 10 and comparing with Lustre 1.8.4 with a high I/O the MDS is basically always around load avarage of 1.80. Is this the normal behavior of Lustre 2.2?</p> MDS HW <br/> ----------------------------------------------------------------------------------------------------<br/> Linux XXXX.admin.cscs.ch 2.6.32-220.7.1.el6_lustre.g9c8f747.x86_64 <br/> Architecture: x86_64<br/> CPU op-mode(s): 32-bit, 64-bit<br/> Byte Order: Little Endian<br/> CPU(s): 16<br/> Vendor ID: AuthenticAMD<br/> CPU family: 16<br/> 64Gb RAM<br/> Interconnect IB 40Gb/s<br/> <br/> MDT LSI 5480 Pikes Peak <br/> SSDs SLC <br/> ----------------------------------------------------------------------------------------------------<br/> <br/> OSS HW<br/> ----------------------------------------------------------------------------------------------------<br/> Architecture: x86_64<br/> CPU op-mode(s): 32-bit, 64-bit<br/> Byte Order: Little Endian<br/> CPU(s): 32<br/> Vendor ID: GenuineIntel<br/> CPU family: 6<br/> 64Gb RAM<br/> Interconnect IB 40Gb/s<br/> <br/> OST LSI 7900<br/> ----------------------------------------------------------------------------------------------------<br/> <br/> 1 MDS + 1 fail over<br/> 12 OSS - 6 OST per OSS<br/> LU-1447

MDS Load avarage

Task Minor Resolved Cannot Reproduce Oleg Drokin Fabio Verzelloni Wed, 30 May 2012 04:48:32 +0000 Mon, 29 May 2017 03:52:48 +0000 Mon, 29 May 2017 03:52:48 +0000 Lustre 2.2.0 0 2 <p>I see that there is not an issue with high CPU usage, rather some of the threads are sleeping in D state.<br/> The top snapshot is not long enough to show them.<br/> Can you please do ps ax and filter out only threads in the D state and then post the output here?<br/> (e.g. ps ax | grep D)</p> <p>Yesterday evening we had a hang of the file system ( ticket <a href="http://jira.whamcloud.com/browse/LU-1451" class="external-link" rel="nofollow">http://jira.whamcloud.com/browse/LU-1451</a> ) and now the load avarage is back to 'normal' (load average: 0.26, 0.24, 0.30 ) also during heavy I/O but we are having drop down of performance, ( <a href="http://jira.whamcloud.com/browse/LU-1455" class="external-link" rel="nofollow">http://jira.whamcloud.com/browse/LU-1455</a> ).</p> <p>This is the all cluster ps -ax | grep D after a while of working.</p> <p>Based on these it seems that weisshorn07, 09, 13 ... are having serious overload issues (likely induced by the disk subsystems there). Any chance you can survey your disk subsystem to see what's going on? I suspect it's not really happy to have a lot of parallel IO ongoing. Also since some other OSSes are less busy, it appears the IO is not distributed all that evenly.</p> <p>In a lot of cases with "weak" (parallel-io wise) disk subsystems limiting number of ost io threads possible should help the situation I think, by reducing the overload.<br/> Ongoing work on NRS should help the issue even more by allowing to limit number of in-progress RPCs per target.</p> <p>The MDS does not have any processes in D state, and I assume at the time this snapshot was taken MDS Load Average was pretty small?</p> <p>The disk HW is:<br/> LSI 7900 <br/> 6 controllers<br/> 8 enclosure x controller<br/> RAID 6 - SATA 7.2RPM</p> <p>Our 'max_rpmcs_in_flight' on the MDS is:<br/> <span class="error">&#91;root@weisshonr01 lustre&#93;</span># cat ./osc/scratch-OST0047-osc-MDT0000/max_rpcs_in_flight<br/> 8</p> <p>and the threads_<span class="error">&#91;max,min,started&#93;</span> are:</p> <p><span class="error">&#91;root@weisshorn01 lustre&#93;</span># cat ./mgs/MGS/mgs/threads_max<br/> 32<br/> <span class="error">&#91;root@weisshorn01 lustre&#93;</span># cat ./mgs/MGS/mgs/threads_started<br/> 32<br/> <span class="error">&#91;root@weisshorn01 lustre&#93;</span># cat ./mgs/MGS/mgs/threads_min<br/> 3</p> <p>&#8211;</p> <p><span class="error">&#91;root@weisshorn01 scratch-MDT0000&#93;</span># cat ./mdt_mds/threads_max<br/> 512<br/> <span class="error">&#91;root@weisshorn01 scratch-MDT0000&#93;</span># pwd<br/> /proc/fs/lustre/mdt/scratch-MDT0000<br/> <span class="error">&#91;root@weisshorn01 scratch-MDT0000&#93;</span># cat ./mdt_mds/threads_min <br/> 2<br/> <span class="error">&#91;root@weisshorn01 scratch-MDT0000&#93;</span># cat ./mdt_mds/threads_started <br/> 2</p> <p>on the OSS:</p> <p><span class="error">&#91;root@weisshorn03 lustre&#93;</span># cat ./ost/OSS/ost/threads_max<br/> 512<br/> <span class="error">&#91;root@weisshorn03 lustre&#93;</span># cat ./ost/OSS/ost/threads_min <br/> 128<br/> <span class="error">&#91;root@weisshorn03 lustre&#93;</span># cat ./ost/OSS/ost/threads_started <br/> 512</p> <p>on the client/MDS side the max_rpcs_in_flight is:</p> <p><span class="error">&#91;root@weisshorn01 lustre&#93;</span># cat ./osc/scratch-OST0005-osc-MDT0000/max_rpcs_in_flight<br/> 8</p> <p>So far we didn't see anymore the high load average on the MDS instead the load average on the OSS when we run benchmark with block size of 4096k OSS load goes to 200-300 and with 'top' we see "x.x%wa" in some cases increasing.</p> <p>Do you have any suggestion about the right tuning based on our hardware/configuration? Also on the client side? ( Cray XE6 1500 nodes~ )<br/> Thanks</p> <p>Fabio</p> <p>Well, it's somewhat expected that as you increase the write activity, load average on OSTs goes up.</p> <p>Essentially what's going on is every write RPC (in 1M chunks) from every client is going to consume 1 OSS io thread.<br/> If the thread blocks doing IO, you get +1 to LA.<br/> The more threads in this state, the higher is the LA. Additionally a lot of the disk controllers hate a lot of parallel IO (because it's in effect highly random IO from their perspective).</p> <p>So solutions for you are multiple:<br/> 1. If it's just the high LA that bothers you and the rest of the system performs well and the speed is acceptable - just ignore the LA.<br/> 2. If you think your system could perform better under load (I am not really familiar with that particular LSI controller, you might want to speak with LSI guys to see what's the optimal IO pattern for it) - you might try decreasing OSS max thread number )to 256, 128 and so on) and see what impact would it have.<br/> If you gather some more stats from your disk controllers to see the IO pattern as seen by it we might see if something looks out of place too (e.g. if you have a lot of small IO, that would not be good). What is the load-generating application doing?</p> <p>There is mostly no client-specific tuning on the client side that you can do that would relieve the situation without additionally ruining e.g. single client performance, so I suggest you to concentrate on servers here. (the possible exception is some read-ahead settings when you expect to have a lot of small read traffic).</p> <p>Close old ticket.</p> Development Rank 1|hzvzzb: Rank (Obsolete) 10075