Whamcloud Community JIRA https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 [LU-10157] LNET_MAX_IOV hard coded to 256 https://jira.whamcloud.com/browse/LU-10157 Lustre <p>LNET_MAX_IOV is hard coded to 256 which works well for architectures with 4096 PAGE_SIZE. However on systems with page sizes bigger or smaller than 4K it will result in long MD count calculations, causing write errors.</p> LU-10157 LNET_MAX_IOV hard coded to 256 Bug Blocker Resolved Fixed Alexey Lyashkov Amir Shehata Tue, 24 Oct 2017 23:40:05 +0000 Wed, 24 Jan 2024 16:08:37 +0000 Tue, 16 Jun 2020 03:35:59 +0000 Lustre 2.12.0 Lustre 2.10.6 Lustre 2.14.0 Lustre 2.12.7 0 12 <p>This issue can be worked around by setting the max_pages_per_rpc to 16. The power8 page size is 65536, 64 value would yield 4MB RPC write. When we reduced that to 1MB RPC write (max_pages_per_rpc == 16) then the RDMA write will work properly. </p> <p>Looking at the code the problem seems to be the way we calculate the number of MDs to describe the data to RDMA.</p> <p>in ptlrpc_register_bulk()</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">total_md = (desc-&gt;bd_iov_count + LNET_MAX_IOV - 1) / LNET_MAX_IOV; </pre> </div></div> <p>total_md is the number of MDs to use to transfer all the data that needs to be transferred.</p> <p>Unfortunately, LNET_MAX_IOV is hard-coded to 256. 256 is used because there is an underlying assumption that the page size is 4K, thus 256 * 4K = 1M.</p> <p>On the power8 machine the page size is 64K. And they set the max_pages_per_rpc to 64, intending for a 4MB RPC sizes.</p> <p>As an example let's take the case where we max out the number of pages to 64. If you plugin the numbers in the above equation, you get:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">64 + 256 - 1 / 256 = 1 </pre> </div></div> <p>So you essentially try to shove 4MB worth of data into 1 MD. LNet expects that each MD will only describe 1MB of buffers.</p> <p>If we define LNET_MAX_IOV to </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">#define LNET_MAX_IOV (1024*1024) / PAGE_SIZE </pre> </div></div> <p>then on a power8 machine LNET_MAX_IOV will be 16, resulting in:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">64 + 16 - 1 / 16 = 4 </pre> </div></div> <p>4 MDs each describing 1 MB of data, which should work.</p> <p>This analysis also holds considering the results of our testing, when we reduced max_pages_per_rpc to 16, since no matter what we're always going to have a maximum of 1MB in 1 MD.</p> <p>James Simmons (uja.ornl@yahoo.com) uploaded a new patch: <a href="https://review.whamcloud.com/31559" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31559</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10157" title="LNET_MAX_IOV hard coded to 256" class="issue-link" data-issue-key="LU-10157"><del>LU-10157</del></a> lnet: make LNET_MAX_IOV dependent on page size<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 1237508cdb9dc8e748c375dafaaea3a286ae936a</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/31559/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31559/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10157" title="LNET_MAX_IOV hard coded to 256" class="issue-link" data-issue-key="LU-10157"><del>LU-10157</del></a> lnet: make LNET_MAX_IOV dependent on page size<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 272e49ce2d5d6883e6ca1b00a9322b3a23b2e55a</p> <p>Landed for 2.12</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33148" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33148</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10157" title="LNET_MAX_IOV hard coded to 256" class="issue-link" data-issue-key="LU-10157"><del>LU-10157</del></a> lnet: make LNET_MAX_IOV dependent on page size<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: ef932917ad3a77d1e9fdeaeb39774108bf25fc19</p> <p>Landed patch don't help with ARM64 client with 64k page size to have work with x86 servers in case 4mb bulk is allowed.<br/> for other cases it's limits a number pages for random IO which can attached to the bulk.<br/> It looks we need much complex solution with both LNET_MTU and number fragments limits and don't rely to the MAX_IOV to calculate an number MD's for bulk.</p> <p>This is ptlrpc bug, not a lnet. And should be fixed with bulk preparation time.<br/> Patch will be send in short time, until it - 64k pages is not usable at all.</p> <p>Alexey Lyashkov (c17817@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/37385" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37385</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10157" title="LNET_MAX_IOV hard coded to 256" class="issue-link" data-issue-key="LU-10157"><del>LU-10157</del></a> lnet: restore an maximal fragments count<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a4ad80c4ecb584e951426cd90d6f084886d50903</p> <p>Alexey Lyashkov (c17817@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/37386" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37386</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10157" title="LNET_MAX_IOV hard coded to 256" class="issue-link" data-issue-key="LU-10157"><del>LU-10157</del></a> ptlrpc: separate number MD and refrences for bulk<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: bf78709e14c6b774cd62a9ed9a93cf1b5138d3bb</p> <p>Alexey Lyashkov (c17817@cray.com) uploaded a new patch: <a href="https://review.whamcloud.com/37387" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37387</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10157" title="LNET_MAX_IOV hard coded to 256" class="issue-link" data-issue-key="LU-10157"><del>LU-10157</del></a> ptlrpc: fill md correctly.<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 3d34163fa8aa9834c13f03b4f471b0ccd40784d6</p> <p>Resetting LNET_MAX_IOV to 256 will make the Power8 nodes send 16MB instead of 1 MB. This will put memory pressure on the compute nodes that will impact user applications. What I have been thinking about is breaking the relationship of LNet fragments and the page size like what is done with the page cache. This is done by using xarrays. The Xarray API has the ability to map a range of indexes to the same object. This was done for huge table support. So in the case of LNet we could allocation an Xarray 256 in size and then on 64K page systems access the 7 object will give you the first page at the proper offset. On an x86 system this would give you the 7th page. I will work on adding the xarray infrastructure to lustre today.</p> <p>James - it looks you read a patch incorrectly. You say about it in slack, and i reply to it already.Lets remember a discussion in slack. xarray don't needs for it, completely don't needs.</p> <p>I adds a code to limit a MD with transfer size in additional to the fragments count, as it before.<br/> So currently MD can be 1mb with 16 fragments or 256 fragments each 4k and less, which limit is hit early.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> + <span class="code-keyword">if</span> (((desc-&gt;bd_iov_count % LNET_MAX_IOV) == 0) || + (desc-&gt;bd_nob_last == LNET_MTU)) { + desc-&gt;bd_mds_off[desc-&gt;bd_md_count] = desc-&gt;bd_iov_count; + desc-&gt;bd_md_count ++; + desc-&gt;bd_nob_last = 0; + LASSERT(desc-&gt;bd_md_count &lt;= PTLRPC_BULK_OPS_COUNT); + } </pre> </div></div> <p>So client don't able to send 16mb transfer as you point.</p> <p>Whole patch series tested on soft AARH64 with RHEL8 with 64k page size (ConnectIB VF), and on real HW AARCH64 with RHEL8 (Mellanox CX-4 card) with same page size, server side is Sonexion L300 (2.12 based + Mellanox CX-4) and Lustre 2.5 + Connect CX3 card.</p> <p>I'm looking at struct lnet_libmd which has </p> <p>lnet_kiov_t      kiov<span class="error">&#91;LNET_MAX_IOV&#93;</span> which ends up being 256 pages no?</p> <p>No. up 256 fragments, depend of overall transfer size.<br/> random io workload with &lt;= 4k size on 64k pages - will fill a 256 entries in single transfer - but overall size is 1Mb or less.</p> <p>Hmm, somehow there are two comments here from Gerrit Updater about <a href="https://review.whamcloud.com/#/c/37386" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/37386</a>, but none about <a href="https://review.whamcloud.com/#/c/37387" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/37387</a>. Please note that three pending patches (37385, 37386, and 37387) are tied to this ticket.</p> <p>I need to really think about this. I just see potential issues with confusion with fragment count and number of pages. I see it in the patch for lnet selftest user land tool for example. I think a better way can be done. Especially since this change touches all of the LNet layer yet its a IB specific issue.</p> <p>I think LNet should check an MD region size before send or attach. Other possibility is it export an fragments / MTU size tunable into ptlrpc instead of defines. </p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37386/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37386/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10157" title="LNET_MAX_IOV hard coded to 256" class="issue-link" data-issue-key="LU-10157"><del>LU-10157</del></a> ptlrpc: separate number MD and refrences for bulk<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 8a7f2d4b11801eae4c91904da9f9750a012a6b11</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37387/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37387/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10157" title="LNET_MAX_IOV hard coded to 256" class="issue-link" data-issue-key="LU-10157"><del>LU-10157</del></a> ptlrpc: fill md correctly.<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e1ac9e74844dc75d77ef740b3a44fad2efde30c5</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/37385/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37385/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10157" title="LNET_MAX_IOV hard coded to 256" class="issue-link" data-issue-key="LU-10157"><del>LU-10157</del></a> lnet: restore an maximal fragments count<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 4072d863c240fa5466f0f616f7e9b1cfcdf0aa0e</p> <p>Landed for 2.14</p> <p>Serguei Smirnov (ssmirnov@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/41275" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41275</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10157" title="LNET_MAX_IOV hard coded to 256" class="issue-link" data-issue-key="LU-10157"><del>LU-10157</del></a> ptlrpc: separate number MD and refrences for bulk<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 0d8b5acaf0cf3e741652e93e9894c43a5bf21f2e</p> <p>Serguei Smirnov (ssmirnov@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/41276" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41276</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10157" title="LNET_MAX_IOV hard coded to 256" class="issue-link" data-issue-key="LU-10157"><del>LU-10157</del></a> ptlrpc: fill md correctly.<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 1b3f8e4ad1adeb52505aaa34be2a9246997658bf</p> <p>Serguei Smirnov (ssmirnov@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/41277" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41277</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10157" title="LNET_MAX_IOV hard coded to 256" class="issue-link" data-issue-key="LU-10157"><del>LU-10157</del></a> lnet: restore an maximal fragments count<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: 1<br/> Commit: 385819a69da21e8caf1b4b5ef80beb1b87f2c560</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/41275/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41275/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10157" title="LNET_MAX_IOV hard coded to 256" class="issue-link" data-issue-key="LU-10157"><del>LU-10157</del></a> ptlrpc: separate number MD and refrences for bulk<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 39cc8bd8d3747583e5029e4cd9520115ab0c6ff1</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/41276/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41276/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10157" title="LNET_MAX_IOV hard coded to 256" class="issue-link" data-issue-key="LU-10157"><del>LU-10157</del></a> ptlrpc: fill md correctly.<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: 8ef7c5c45567fc9cdedfe4242a6c5b73193ab9fe</p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/41277/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/41277/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10157" title="LNET_MAX_IOV hard coded to 256" class="issue-link" data-issue-key="LU-10157"><del>LU-10157</del></a> lnet: restore an maximal fragments count<br/> Project: fs/lustre-release<br/> Branch: b2_12<br/> Current Patch Set: <br/> Commit: e39ba78737b63128cfc7df9d49c8dd49a30ce590</p> Duplicate LU-10775 Related LU-10129 LU-7650 LU-13181 LU-10073 LU-6387 LU-10300 Development Rank 1|hzzmdz: Rank (Obsolete) 9223372036854775807 Severity