https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7245 Lustre <p>While working on enhancing the lnetctl utility it was discovered that more SMP scaling improvements can be done to the currently supported LND driver.</p> Any TCP, infiniband or Gemini network system LU-7245

Improve SMP scaling support for LND drivers

Improvement Minor Resolved Fixed James A Simmons James A Simmons patch Fri, 2 Oct 2015 17:56:41 +0000 Thu, 13 May 2021 18:56:06 +0000 Sat, 22 Apr 2017 18:22:40 +0000 Lustre 2.9.0 Lustre 2.9.0 0 8 <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="http://review.whamcloud.com/16710" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16710</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7245" title="Improve SMP scaling support for LND drivers" class="issue-link" data-issue-key="LU-7245"><del>LU-7245</del></a> socklnd: Bind peers to a specific CPT<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 05f09aec7aecbd1d57eb321da2f4a65056d9a483</p> <p>James</p> <p>Are you thinking of targeting this work for 2.9?</p> <p>Peter</p> <p>James, since you asked, a few notes about NUMA.</p> <p>Reasoning about NUMA is a lot like thinking about a cluster, except you spend a lot of time worrying about cache lines instead of files. (Much of the terminology is also similar, which becomes confusing when discussing NUMA issues for systems that are part of a cluster.) It is worth noting that NUMA considerations already apply once a system has more than one socket. Ideally, the process driving I/O, the memory involved, and the interface involved all live on the same socket.</p> <p>In practice, the memory placement may have been done by some user space process outside our control, and the same goes for the process that initiates the I/O. Selection of an interface that is close in the topology of the system is useful, but that already assumes a multi-rail type configuration. Much of the time there will be no choice because there is only one interface. (LNet routers are an exception: there we do have full control over the location of all buffers and threads relative to the interfaces used.)</p> <p>So the main concern becomes doing the best we can in areas we do control, in particular avoiding cache line bouncing. Placing a datastructure like <tt>ksock_peer</tt> in the same CPT as the interface helps a little bit here, but only a little. The layout of the <tt>ksock_peer</tt> structure is actually a good example of what not to do. Take a look at the first few members, which will likely all be in the same cache line:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">typedef struct ksock_peer { struct list_head ksnp_list; <span class="code-comment">/* stash on global peer list */</span> cfs_time_t ksnp_last_alive; <span class="code-comment">/* when (in jiffies) I was last alive */</span> lnet_process_id_t ksnp_id; <span class="code-comment">/* who's on the other end(s) */</span> atomic_t ksnp_refcount; <span class="code-comment">/* # users */</span> <span class="code-object">int</span> ksnp_sharecount; <span class="code-comment">/* lconf usage counter */</span> </pre> </div></div> <p><tt>ksnp_list</tt> and <tt>ksnp_id</tt> are semi-constant, and read by any thread that does a lookup of some peer in the hash table (shared/read access). In contrast <tt>ksnp_refcount</tt> and <tt>ksnp_last_alive</tt> are updated by threads doing work for this particular peer (exclusive/write access). So a lookup of some unrelated peer causes a cache line bounce between the CPU doing the lookup and the CPU managing the I/O. This particular case can be mitigated by being very careful with the layout of a datastructure, and by making sure that threads that do modify the structure run on the same socket, even if that socket is not where the datastructure lives.</p> <p>The patch for the Gemini SMP work under <a href="https://jira.whamcloud.com/browse/LU-2544" title="Add SMP scaling to Cray Gemini driver" class="issue-link" data-issue-key="LU-2544">LU-2544</a> reworked a lot of the data structures to deal with what you described. Never looked closely at the other LND drivers but I can see the problem there. Will require a lot of data structure reworking :-/</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/16710/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16710/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7245" title="Improve SMP scaling support for LND drivers" class="issue-link" data-issue-key="LU-7245"><del>LU-7245</del></a> socklnd: Bind peers to a specific CPT<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 68eb6e841f49d41a289bd0b3f559973b6cb31738</p> <p>I suspect more work will coming from this ticket for 2.9.</p> <p>Patches for this work already landed and the multi-rail work filled in the rest of the gaps.</p> Related LU-2544 LU-5960 LU-14676 Development End date Wed, 28 Oct 2015 17:56:41 +0000 Epic/Theme lnet Rank 1|hzxpfj: Rank (Obsolete) 9223372036854775807 Start date Fri, 2 Oct 2015 17:56:41 +0000