https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4768 Lustre <p>ost-survey on this environment</p> <p> mds, oss: lustre 2.4.2 (CentOS 6.5 with lustre patched kernel RPM)<br/> client: lustre 2.4.2 (CentOS 6.5 patchless client, lustre iokit 1.4.0.1)</p> <p> command line: ost-survey -s 100 /lustre</p> <p>will hang, chewing all the CPU in the attempt to<br/> echo 0 in /proc/fs/lustre/llite/*/max_cached_mb . This is due to the<br/> subroutine cache_off in the ost-survey script.</p> <p>In a lustre client 2.1.6 the max_cached_mb appears as<br/> a single number</p> <ol> <li>cat /proc/fs/lustre/llite/*/max_cached_mb<br/> 18114</li> </ol> <p>but on a 2.4.2 client is different</p> <ol> <li>cat /proc/fs/lustre/llite/*/max_cached_mb<br/> users: 2<br/> max_cached_mb: 96766<br/> used_mb: 21806<br/> unused_mb: 74960<br/> reclaim_count: 0</li> </ol> <p>So probably the script has not been fixed to work with the new output.</p> mds, oss: lustre 2.4.2 (CentOS 6.5 with lustre patched kernel RPM)<br/> client: lustre 2.4.2 (CentOS 6.5 patchless client, lustre iokit 1.4.0.1)<br/> LU-4768

ost-survey hangs on client 2.4

Bug Minor Resolved Fixed James Nunez francesco de giorgi Fri, 14 Mar 2014 11:52:49 +0000 Tue, 9 Dec 2014 18:28:03 +0000 Sun, 5 Oct 2014 12:59:17 +0000 Lustre 2.4.2 Lustre 2.7.0 0 10 <p>Two things are wrong here:</p> <ul class="alternate" type="square"> <li>it doesn't make sense to allow the client to have no pages cached at all. ll_max_cached_mb_seq_write() should always allow at least PTLRPC_MAX_BRW_PAGES of cached data, so tha the client can make well-formed RPCs. It probably should just silently limit the input value to max(pages_number, PTLRPC_MAX_BRW_PAGES) if the requested value is smaller.</li> <li>the ost-survey script should be changed to ask for a minimum of 256 pages</li> </ul> <p>I'm pretty sure this affects 2.5 client also, but don't have one up right now to examine.</p> <p>Anyway I'd just add while it's a minor problem for things like initial set up testing, it can become a big problem if trying to debug possible bad filesystems. If you forget to remove ost-survey or modify it, and start testing from a client, that client hangs. If you're not someone aware of the problem, it can cause a bit of a sysadmin freak-out.</p> <p>ost-survey has other problems too, for example:</p> <p>It looks to me like it assumes you only have one filesystem mounted so when it does things like count ost's, it looks at:<br/> /proc/fs/lustre/lov/&#42;&#45;clilov&#45;&#42;/numobd -where it seems like it should look at /proc/fs/lustre/lov/$MNT&#45;clilov&#45;&#42;/numobd.</p> <p>It uses the long deprecated positional parameters for setstripe, so spews a lot of errors</p> <p>I think this tool really does needs to be fixed or just removed. If it's too low a priority to get a proper fix and review, then just not distributing it would be preferable.</p> <p>Scott</p> <p>Proposed master patch at: <a href="http://review.whamcloud.com/11971" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/11971</a></p> <p>This patch only allows ost-survey to run and does not modify ll_max_cached_mb_seq_write().</p> <p>The comment was made to, in ost-survey, set max_cached_mb to 256 pages. The patch sets max_cached_mb to 256 pages (in MB), 256 * pagesize / 1024 * 1024, but this allows the read values to be, possibly, generous due to reading from cache. For example, on the system I was testing this patch on, the page size is 2621440. So, max_cached_mb is set to 640. Here are the results for max_cached_mb = 2 and max_cached_mb = 640:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>With max_cached_mb = 2: Ost# Read(MB/s) Write(MB/s) Read-time Write-time ---------------------------------------------------- 0 258.701 125.005 3.865 8.000 1 341.453 107.399 2.929 9.311 2 340.931 117.782 2.933 8.490 3 277.857 105.958 3.599 9.438 With max_cached_mb = 640: Ost# Read(MB/s) Write(MB/s) Read-time Write-time ---------------------------------------------------- 0 1227.975 128.612 0.814 7.775 1 1109.066 108.828 0.902 9.189 2 1069.722 127.521 0.935 7.842 3 1050.668 107.104 0.952 9.337 </pre> </div></div> <p>Write performance is about the same in each case, but read performance is much larger with max_cached_mb = 640. Do we really want max_cached_mb to be 256 pages?</p> <p>The patch for <a href="https://jira.whamcloud.com/browse/LU-5030" title="&quot;lctl {get,set}_param&quot; should also check in /sys/fs/{lnet,lustre}" class="issue-link" data-issue-key="LU-5030"><del>LU-5030</del></a> made significant changes to ost-survey; <a href="http://review.whamcloud.com/#/c/10534/8" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/10534/8</a> .</p> <p>Yes the patch for <a href="https://jira.whamcloud.com/browse/LU-5030" title="&quot;lctl {get,set}_param&quot; should also check in /sys/fs/{lnet,lustre}" class="issue-link" data-issue-key="LU-5030"><del>LU-5030</del></a> removed the direct access to the proc filesystem. Instead it uses lctl <span class="error">&#91;s|g&#93;</span>et_param as it should. Let see if the patch does the wrong thing</p> <p>I've updated the patch to correctly get the page size and, as Andreas asked for, modified ll_max_cached_mb_seq_write() to set the requested pages to be the max of the requested pages and PTLRPC_MAX_BRW_PAGES.</p> <p>Landed for 2.7</p> Related LU-5030 LU-5773 Development Epic/Theme test Rank 1|hzwhn3: Rank (Obsolete) 13113 Severity