https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-5043 Lustre <p>We are seeing a problem with our Lustre clients on our BG/Q I/O Nodes (ION). It appears that all of the ptlrpcd are stuck running on "CPU" number zero. This is especially problematic for this CPU because all of the hardware interrupts for the networking card are wired only to CPU 0. This results in a client that is unresponsive under write loads (and maybe others) from the BG/Q compute nodes.</p> <p>Some current details:</p> <ul> <li>PPC64</li> <li>17 cores with 4-way threading, so Linux sees 68 CPUs.</li> <li>Hardware interrupts routed to CPU0 only</li> <li>CONFIG_NUMA is not set</li> <li>Lustre 2.4.0-28chaos, which contains the "<a href="https://jira.whamcloud.com/browse/LU-4509" title="clio can be stuck in osc_extent_wait" class="issue-link" data-issue-key="LU-4509"><del>LU-4509</del></a> ptlrpc: re-enqueue ptlrpcd worker" patch (see github.com/chaos/lustre)</li> </ul> <p>When I watch CPU usage on a node using <tt>top</tt>, I see 100% CPU usage on CPU 0, and very little happening on the rest of the CPUs. Sometimes the flush-lustre thread uses a fair bit of CPU time CPU0 as well.</p> <p>100+ sysiod threads (the user-space process that handles forwarded IO from the compute nodes) are all running, but using very little CPU time in aggregate. At least as far as I can tell. I have a feeling that I'm not seeing everything in <tt>top</tt> for some reason. Maybe I'm wrong.</p> <p>I hacked the ptlrpcd() function to set the allowed CPUs according to my own mask, which is cpu_active_mask but with CPU 0 removed using cpumask_clear_cpu(). Sure enough, now CPU 1 is where 100% is spent insead of CPU 0. I understand why CPU 0 is avoided, but not why CPU 1 is the only CPU used out of all that are available.</p> <p>The cpu mask handling code in Lustre is none too easy to follow... Is there some setting that Lustre is using that would bind it to a single CPU? I can't see it yet...</p> LU-5043

ptlrpcd threads only run on one CPU

Bug Critical Closed Fixed WC Triage Christopher Morrone Sat, 10 May 2014 00:38:00 +0000 Mon, 12 May 2014 12:56:30 +0000 Mon, 12 May 2014 12:56:30 +0000 0 4 <p>I am not sure that I understand the logic here in ptlrpcd_select_pc():</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">case</span> PDL_POLICY_SAME: idx = cfs_smp_processor_id() % ptlrpcds-&gt;pd_nthreads; <span class="code-keyword">break</span>; </pre> </div></div> <p>What makes us think that this will result in a ptlrpcd thread that runs on the same CPU as the caller?</p> <p>OK, you guys (and Lustre in general) are off the hook for this issue, I think. This was either self- or vendor-inflicted months ago.</p> <p>It looks like someone got the bright idea to set "isolcpu=0,1,2,3" on the kernel command line. This "feature" isolates those cpus from normal scheduler activities. It does not, apparently, stop new kernel threads from starting up on those isolated CPUs. So many of the Lustre kernel threads got started on CPU 0 and because that CPU is isolated, they never get moved anywhere else.</p> <p>There is a possibility that Lustre supposed to do something to play nice with that option...but if I find that out I'll just start a new ticket.</p> <p>Closing ticket.</p> <p>thanks Chris.</p> Development Rank 1|hzwm9b: Rank (Obsolete) 13936 Severity