https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10129 Lustre <p>With patch <a href="https://review.whamcloud.com/29290/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29290/</a>, map-on-demand set to 32 doesn't work. map-on-demand needs to be set to 256 in order for large transfers to complete. </p> LU-10129

map-on-demand set to 32 doesn't work on OPA

Bug Critical Resolved Fixed Amir Shehata Amir Shehata llnl Tue, 17 Oct 2017 16:44:42 +0000 Wed, 2 Jan 2019 20:45:37 +0000 Fri, 22 Dec 2017 12:41:10 +0000 Lustre 2.11.0 0 12 <p>We do have OPA fabrics, so I've added the llnl and topllnl labels to this ticket based on the following comment in <a href="https://jira.whamcloud.com/browse/LU-10089" title="kiblnd_fmr_pool_map() Failed to map mr 10/11 elements" class="issue-link" data-issue-key="LU-10089"><del>LU-10089</del></a>:</p> <blockquote> <p>One thing to note, if you're using OPA you should use map-on-demand set to 256. I'm still analyzing this issue and hopefully will have a patch soon. This issue is tracked under <a href="https://jira.whamcloud.com/browse/LU-10129" title="map-on-demand set to 32 doesn&#39;t work on OPA" class="issue-link" data-issue-key="LU-10129"><del>LU-10129</del></a></p></blockquote> <p>I haven't seen unexplained symptoms on our OPA-connected nodes.</p> <p>What would I likely see?  disconnects/reconnects?  Failed BRW operations?</p> <p>At the LND layer RDMA writes fail. So that could translate to RPC failures/bulk write failures, and other FS issues.</p> <p>Here is a summary of my investigation on map-on-demand, and a proposal for a change to resolve the issues around this. I'd like feedback to see if I missed something:<br/> <a href="https://wiki.hpdd.intel.com/display/LNet/o2iblnd+map_on_demand" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/display/LNet/o2iblnd+map_on_demand</a></p> <p>While are looking to fix this maybe its time to revist <a href="https://jira.whamcloud.com/browse/LU-7650" title="ko2iblnd map_on_demand can&#39;t negotitate when page sizes are different between nodes." class="issue-link" data-issue-key="LU-7650"><del>LU-7650</del></a>.</p> <p>Amir Shehata (amir.shehata@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29995" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29995</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10129" title="map-on-demand set to 32 doesn&#39;t work on OPA" class="issue-link" data-issue-key="LU-10129"><del>LU-10129</del></a> lnd: rework map_on_demand behavior<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d6fb47b0e3cc1521f8f33dcc7dd78a9abcb27154</p> <p>Amir Shehata (amir.shehata@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30309" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30309</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10129" title="map-on-demand set to 32 doesn&#39;t work on OPA" class="issue-link" data-issue-key="LU-10129"><del>LU-10129</del></a> lnd: set device capabilities<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 200fb9081989add33457554a221ecfe087da1176</p> <p>Does hfi1 module parameter "num_user_contexts" have to match the LND map_on_demand setting ?</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/30309/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30309/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10129" title="map-on-demand set to 32 doesn&#39;t work on OPA" class="issue-link" data-issue-key="LU-10129"><del>LU-10129</del></a> lnd: set device capabilities<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 70a73c52d5a780e580e49ccfa778c4beab340c9c</p> <p>Do this patch and 29290 need to be backported to 2.10?  I'm not requesting it, just asking the question.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/29995/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29995/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10129" title="map-on-demand set to 32 doesn&#39;t work on OPA" class="issue-link" data-issue-key="LU-10129"><del>LU-10129</del></a> lnd: rework map_on_demand behavior<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 2c38e8f5b08e7475ffc37915fe0300505a5759db</p> <p>Landed for 2.11. Flagged for consideration for 2.10.x until Amir is back in the office and able to comment</p> <p>These list of patches don't need to be ported to 2.10.x. They were instigated by:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LU-9983 ko2iblnd: allow <span class="code-keyword">for</span> discontiguous fragments </pre> </div></div> <p>which is not in 2.10</p> <p>Di committed his changed</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">LU-9983 osp: align the OSP request size by 4k </pre> </div></div> <p>which should hide the initial problem of discontigous fragments.</p> <p><a href="https://jira.whamcloud.com/browse/LU-9500" title="MOFED 4/mlx5: Aligning non-aligned page addresses trigger dump_cqe" class="issue-link" data-issue-key="LU-9500"><del>LU-9500</del></a> proposes similar fix for fragment alignment.</p> Related LU-7650 LU-10157 Development Rank 1|hzzm0n: Rank (Obsolete) 9223372036854775807 Severity