https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9441 Lustre <p>During benchmarking at large scale on KNL+Omnipath system (8k nodes) we saw periodic OS noise due to lustre kernel threads that greatly affects the performance of small-message MPI collectives (MPI_Barrier, MPI_Allreduce with small datasize, etc).</p> <p>The request is for lustre to use kernel threads in a deterministic manner when the load is low. In this case the ideal usage would have been to use only the thread(s) in CPT 0 which can be setup to run only on KNL tile 0 (cores 0 and 1). Then when no significant I/O is going on, a benchmark thread can be bound to a tile other than 0 and not see any lustre noise.</p> <p>This is more than just a benchmarking setting, it is common for allreduce to be a bottleneck especially at scale, and usually HPC applications have long phases with no I/O.</p> LU-9441

Use kernel threads in predictable fashion to confine OS noise

Improvement Major Resolved Fixed James A Simmons Larry Meadows Wed, 3 May 2017 14:16:11 +0000 Wed, 17 Feb 2021 23:20:39 +0000 Wed, 17 Feb 2021 23:20:38 +0000 Lustre 2.14.0 0 3 <p>Hi Dmitry,</p> <p>Can you please investigate this report?</p> <p>Thanks.<br/> Joe</p> <p>Looking at the lustre code for ptlrpdc it appears that every ptlrpcd thread wakes up at least 1x/second regardless of activity.<br/> I have profiles from the KNL machine at TACC (Stampede) showing activity on 60 different lustre threads even with no I/O going on. Kernel tracing also confirms this.<br/> I need a resolution for this, it seriously affects performance of MPI collectives, especially as the node count increases.<br/> Please respond with your plans to resolve this issue.</p> <p>Hi Larry,</p> <p>This is on Dmitry's plate to investigate and propose potential solutions. We will post to the ticket as soon as possible.</p> <p>Thanks.<br/> Joe</p> <p>Amir Shehata (amir.shehata@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28496" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28496</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9441" title="Use kernel threads in predictable fashion to confine OS noise" class="issue-link" data-issue-key="LU-9441"><del>LU-9441</del></a> pltrpc: don't wakeup on 1 second intervals<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 6ddeed98c9f65486743b2a30b7ea0468fb4cf952</p> <p>The <a href="https://jira.whamcloud.com/browse/LU-9660" title="reduce ptlrpcd wakeups on idle system" class="issue-link" data-issue-key="LU-9660"><del>LU-9660</del></a> patch was landed, but I'm wondering if there is more we can do here. For example, batching inodes by NID ranges (i.e. submask) so that they ping at the same time. For example, if we scheduled pings on clients so that <tt>seconds % interval == (NID &gt;&gt; 9) % interval</tt> was true, then groups of 2^9 = 512 nodes would ping in the same second. For jobs that are typically scheduled with "nearby" NIDs, or for small clusters, this would put the ping overhead in one timestep, instead of introducing jitter across all timesteps.</p> <p>Landing <a href="https://jira.whamcloud.com/browse/LU-7236" title="connections on demand" class="issue-link" data-issue-key="LU-7236"><del>LU-7236</del></a> would help this further, as we don't need pinging if the client has disconnected from the server(s) (at the expense of a small latency hit when it next sends an RPC to the server again).</p> <p>Similarly, if a single application was batching up operations for the servers in a coordinated manner between clients, this should be staggered to avoid contention between jobs, and would avoid adding random jitter to all timesteps of a computation. Only one timestep per period would be impacted by the background work. Something like ping on <tt>time % (hash(jobid) &amp; 15)</tt> or similar could be used to coordinate autonomously between clients running the same job.</p> <p><a href="https://review.whamcloud.com/#/c/38091/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/38091/</a> should limit the pinger noise. Any other ideas?</p> <p>Your patch isolates the pinger <em>thread</em> on the client, but it doesn't do anything to isolate the network traffic to avoid jitter in the communication.</p> <p>I think one of the things that has been lost with the timer changes for the ping (and other) wakeups is coordination between the threads and clients. On the one hand, you don't want <b>all</b> clients and threads to wake at the same time, but some amount of coordination (i.e. a few bad timesteps across all threads at one time and then many good timesteps in a row) is better than each thread having a <b>different</b> bad timestep on a continual basis.</p> <p>As I described in my previous comments here, aligning the client pings by NID or JobID (e.g. "<tt>(seconds &amp; interval) == (hash(jobid) &amp; interval)</tt>", where <tt>interval</tt> is a power-of-two value close to the ping interval) would minimize the number of bad timesteps and maximize good ones. The synchronicity would depend on how well-sync'd the client clocks are, but it could be isolated to a few msec per tens of seconds.</p> <p>The same is true of some of the other timeouts - they could be aligned to happen on a coarse-grained interval (every second or few seconds) rather than randomly, so that when some work needs to be done, there is a bunch, and then it is quiet again as long as possible.</p> <p>Some of this was discussed in the context of <a href="https://review.whamcloud.com/36701" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36701</a> but I don't think it was implemented.</p> <p>James Simmons (jsimmons@infradead.org) uploaded a new patch: <a href="https://review.whamcloud.com/38730" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38730</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9441" title="Use kernel threads in predictable fashion to confine OS noise" class="issue-link" data-issue-key="LU-9441"><del>LU-9441</del></a> llite: bind kthread thread to accepted node set<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3b0e24c39a994463b6eaaa8c424c33b3815280ad</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/38730/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/38730/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9441" title="Use kernel threads in predictable fashion to confine OS noise" class="issue-link" data-issue-key="LU-9441"><del>LU-9441</del></a> llite: bind kthread thread to accepted node set<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: d6e103e6950d99b88141d4b26982889258c774c5</p> <p>Landed for 2.14</p> <p>Andreas wants an offset to running the threads based on the jobid. I think that is one more patch.</p> <p>James, that could be moved to a separate ticket. </p> <p>2.14 is closing so let's track anything else under a new ticket</p> Related LU-9660 LU-4423 LU-7236 LU-13258 Development Epic/Theme Performance jitter Rank 1|hzzbtr: Rank (Obsolete) 9223372036854775807