https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12216 Lustre <p>lctl set_param debug_mb was always in MB. But in 2.10.x its wrong.</p> <p> lctl get_param debug_mb<br/> debug_mb=4801</p> <p>Results in log file that is 474MB<br/> And you can set values like this.</p> <ol> <li>lctl set_param debug_mb=50000<br/> debug_mb=50000</li> <li>lctl get_param debug_mb<br/> debug_mb=49919<br/> I think in the past 4GB was the limit.</li> </ol> LU-12216

lctl set_param debug_mb=xxx set debug buffer to the wrong size

Bug Minor Resolved Not a Bug Patrick Farrell Mahmoud Hanafi Tue, 23 Apr 2019 03:01:11 +0000 Tue, 23 Apr 2019 19:30:00 +0000 Tue, 23 Apr 2019 19:30:00 +0000 Lustre 2.10.6 0 3 <p>Mahmoud,</p> <p>There has never been a fixed upper limit on how large debug_mb can be set.  The behavior you're seeing is intended.</p> <p>Additionally, the debug_mb value is the <b>total</b> buffer size across all CPUs ('CPU' here is defined as "thing a process can be scheduled on, visible in cat /proc/cpuinfo"), the actual buffer space is evenly divided among CPUs.</p> <p>So if you have a 4801 MiB debug buffer and 16 CPUs, each CPU has ~300 MiB of debug buffer.  This means that if you have a single threaded workload (like a single process reading or writing a file), it will fill the 300 MiB and then begin to wrap around.  Because Lustre has various network daemons, you will see some activity on other CPUs, so you will never get debug messages from just one CPU.</p> <p>So this means that you will never get <span class="error">&#91;with these settings and 16 CPU cores&#93;</span> more than 300 MiB of log (at one time) from a given CPU.</p> <p>I am assuming when you say you have a 474 MiB debug log, you mean that's the largest log you get when recording a particular activity, and you expect more.  (it is also possible that you are recording everything, if you have a time limited process).</p> <p>If you look at a Lustre debug log line:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000100:00000001:3.0:1555092720.263653:0:6377:0:(client.c:701:ptlrpc_request_bufs_pack()) Process entered </pre> </div></div> <p>The third field (3.0 here) reflects the CPU in use.  (3 is the CPU, 0 reflects whether or not you're in an interrupt context - it will mostly be 0).</p> <p>But, assuming you are running out of debug buffer:<br/> If you look at your 474 MiB debug log, you will find the majority of your messages are from just a few CPUs, maybe just one.</p> <p>All of this is expected.  If you need a larger per CPU buffer, the solution is simply to set an extremely large debug_mb.  debug_mb is not allocated until it is used, and anything that was used is freed when you set it back to a lower value, so setting it to enormous values for a short time to capture a specific event is OK.  (Do not leave it at those large values or you will be wasting memory.  This is purely a temporary thing for gathering logs.)</p> <p>In the past the upper limit was like 4GB. At least in &lt;2.7.x</p> <p>Mahmoud,</p> <p>Perhaps you had nodes with a small memory footprint.  The limit for debug_mb has been 80% of memory since some time before Lustre 2.0 was released.</p> <p>In any case, the behavior here is all as intended - Do you have any further questions?</p> <p>no, thank you</p> <p>Thanks, Mahmoud!</p> Development Rank 1|i00f87: Rank (Obsolete) 9223372036854775807 Severity