https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LUDOC-286 Lustre Documentation <p>The solution to <a href="https://jira.whamcloud.com/browse/LU-3322" title="ko2iblnd support for different map_on_demand and peer_credits between systems" class="issue-link" data-issue-key="LU-3322"><del>LU-3322</del></a> has made changes to how we can configure map_on_demand and peer_credits for o2iblnd-based connections. In that ticket, Jeremy has outlined some good information which should be captured in our documentation to help users better understand how to work with three tuning parameters: map_on_demand, peer_credits, and concurrent_sends.</p> <p>We should also be documenting what good pre-sets to use for different node types: MDS/OSS, LNet routers, and clients.</p> LUDOC-286

Document the effects of LU-3322

Improvement Critical Open Unresolved Lustre Manual Triage Doug Oucharek Wed, 27 May 2015 23:19:53 +0000 Tue, 10 Apr 2018 17:20:29 +0000 0 8 <p>This patch has landed to master, but as yet there is no documentation patch for the manual.</p> <p>Comments from Jeremy in <a href="https://jira.whamcloud.com/browse/LU-3322" title="ko2iblnd support for different map_on_demand and peer_credits between systems" class="issue-link" data-issue-key="LU-3322"><del>LU-3322</del></a>:</p> <blockquote> <p>Summary:<br/> Normally the Lustre ko2iblnd can only operate with identical peer_credits and map_on_demand between systems. The patch affects the active (initiator) connection in IB for clients/routers and the passive (responder) connections for IB with servers/routers. Passive connections will automatically negotiate down if the parameters permit and reject if they are to high for the remote request. Active connections will send their defaults initially and if rejected attempt to use the lower values from the reject message. If the values are higher the active connection won't retry because its not supported.</p> <p>There are 3 parameters in the ko2iblnd of interest here: peer_credits, map_on_demand, and concurrent_sends. The default settings for ko2iblnd are <tt>peer_credits=8</tt>, <tt>concurrent_sends=8</tt>, and <tt>map_on_demand=0</tt> (disabled)</p> <p>peer_credits determines how many messages you can receive from a single connection (queue pair).<br/> map_on_demand determines the number of DMA segments per credit that are sent. Each segment is usually (always?) a page and is sent as a separate work request so these are typically (256 &#42; 4k pages) going across the wire.<br/> concurrent_sends determines how many send messages you can queue at a time to a single connection, concurrent_sends can't be less than half of peer_credits but concurrent_sends needs to be &lt;= 62 to not exceed the maximum number of work requests per queue pair in the standard Mellanox ConnectX[123] HCAs.</p> <p>The relation between those values is: work_requests_allocated = (map_on_demand + 1) &#42; concurrent_sends</p> <p>You can see your max work requests per queue pair with:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> ibv_devinfo -v | grep max_qp_wr max_qp_wr: 16384 </pre> </div></div> <p>My recommendation is the following for maximum compatibility.<br/> Lustre MDS/OSS Severs and Routers, use the patch and run with:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>peer_credits=124 concurrent_sends=62 map_on_demand=256 </pre> </div></div> <p>Existing Lustre clients:<br/> Need to apply the patch or lower the current values for peer_credits and concurrent_sends to match the OSS/MDS setting.</p></blockquote> <p>Do we need to include some or all of Amir's compatibility matrix (<a href="https://jira.hpdd.intel.com/secure/attachment/18893/compatibility%20matrix.xlsx" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/secure/attachment/18893/compatibility%20matrix.xlsx</a>) in the manual to tell users what options exist for configuring with mixed-Lustre-version networks?</p> <p>The manual section related to this should be marked conditional for Lustre 2.8 using the <tt>&lt;para condition="l28"&gt;</tt> tag.</p> <p>The attached spreadsheet lists 4th case "OPN = Map-on-demand-&gt;on" but likely should read "OPN = Map-on-demand-&gt; off"</p> Related LU-3322 LUDOC-310 Development Rank 1|hzxedr: Rank (Obsolete) 9223372036854775807