https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7990 Lustre <p> Add large bulk IO support, e.g. make the ptlrpc be able to size 16MB IO, to improve the performance.</p> LU-7990

Large bulk IO support

New Feature Minor Resolved Fixed Jinshan Xiong Gu Zheng patch Wed, 6 Apr 2016 10:34:59 +0000 Tue, 8 Dec 2020 05:53:20 +0000 Thu, 5 May 2016 19:09:29 +0000 Lustre 2.9.0 Lustre 2.11.0 Lustre 2.10.2 0 15 <p>Gu Zheng (gzheng@ddn.com) uploaded a new patch: <a href="http://review.whamcloud.com/19366" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19366</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7990" title=" Large bulk IO support" class="issue-link" data-issue-key="LU-7990"><del>LU-7990</del></a> rpc: increase bulk size<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 17aaa0120997e1b6428c449454e45be9c11ef62f</p> <p>Gu Zheng (gzheng@ddn.com) uploaded a new patch: <a href="http://review.whamcloud.com/19367" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19367</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7990" title=" Large bulk IO support" class="issue-link" data-issue-key="LU-7990"><del>LU-7990</del></a> ofd: add a parameter to adjust preferred brw size<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: bf1ebfadacd9f77c0e46103c1c76b8da9e1cb1b0</p> <p>Gu Zheng (gzheng@ddn.com) uploaded a new patch: <a href="http://review.whamcloud.com/19368" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19368</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7990" title=" Large bulk IO support" class="issue-link" data-issue-key="LU-7990"><del>LU-7990</del></a> clio: revise readahead to support 16MB IO<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: ed70b211cbed6d74487707746f098be98787b777</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/19366/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19366/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7990" title=" Large bulk IO support" class="issue-link" data-issue-key="LU-7990"><del>LU-7990</del></a> rpc: increase bulk size<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 51b32ac2b9b86a600e92ab03f387717e526153d3</p> <p>I like to see DDN work to decrease a overall Lustre performance. Last merged patch replace a kmalloc allocations with vmalloc so dramatically increase vmalloc lock contention for kernels without per CPU vmalloc code. </p> <p>Ouch that needs to be fixed.</p> <p>I guess you are referring to the memory allocation in ptlrpc_new_bulk(). With 16M RPC support, the maximum value of nfrags is 4096, and this will lead to a memory allocation of 64KB in maximum. This is less than the maximum memory size kmalloc() can allocate in modern systems. If the memory allocation can;t be fulfilled by kmalloc(), it will then fall down to vmalloc(). I didn't see any problem with it.</p> <p>Is it? What is that maximum size? I thought it was page size times two?</p> <p>Hi Alexey, did you actually see a performance degression? Performance is really important to us, so please open a ticket if so. <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/> I do agree with Jinshan on the change of ptlrpc_new_bulk().</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/19367/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19367/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7990" title=" Large bulk IO support" class="issue-link" data-issue-key="LU-7990"><del>LU-7990</del></a> ofd: add a parameter to adjust preferred brw size<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 60032c840c3764755861a61a5ffd6d2ccd5446c9</p> <p>Li Xi,</p> <p>we have lots problems with vmalloc on RHEL6 kernels. That kernel have a single spinlock to protect a vmalloc allocations.<br/> So it's very high loaded spinlock on server side when we need an allocate a bulk for each BRW request.<br/> Typical performance down is 20% in that case. Probably you run own tests on less loaded system or at system with kernels 3.8 and up, where some vmalloc improvements merged into kernel, in that case you may don't hit it problem.</p> <p>Hi Alexey, thanks for sharing this information. We will run benchmarks to check this too.</p> <p>From my point view, you should look to very different side if you want to increase a size of transfer.<br/> Currently we use a single page per element in transfer, it increase overhead on each parts. Size of array, number WR / SGE requests on IB transfer, ... but two main points exist.<br/> 1) one object &lt;= page size limitation exist only for socklnd, due bug / feauture to use an tcp_sendpage, but if we kill it and return to use kernel_sockmsg or tcp_sendpages directly we able to send more data via single call. IB lnd, GNI lnd, OmniPath .. all able to send up 2Gb per single SGE / WR entry.<br/> 2) OSC always send a data via continues region. Yes, region is continues in virtual memory, but we need anyway remap pages to DMA area when we need a prepare a SGE list, from other side it have 16 continues regions.</p> <p>So from my point of view - bulk code should be changed to access an OSC view as input and have a region to SGE page conversion inside of LNet as SGE list don't have a limitation to number of segments and may better controlled by low level drivers.<br/> It's decrease a memory usage on OST side also.</p> <p>You make valid points Alexey. I think I like to bring up is that it has been recommend that we move from kernel_sock** to using the netlink api. I currently don't have the cycles for that but its something we should look into.</p> <p>In a related question - does the large BRW RPC size also increase the initial ocd_grant request, so that the client can at least form one full-size RPC to the OST after a connect? The default is currently 2MB (i.e. 2x 1MB RPCs) but this initial grant should probably be increased to 8MB or more when the RPC size is increased.</p> <p>Alexey, would you please give us speicfic IO pattern you think performance drops? we had a lot of benchmark and didn't see any perforamnce regresions so far and singinicant improved performance, instead. </p> <p>The <tt>ptlrpc_new_bulk()</tt> code is changed from calling <tt>OBD_ALLOC()</tt> to use <tt>OBD_ALLOC_LARGE()</tt>, which in Lustre 2.7.55 and later is:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">#define OBD_ALLOC_LARGE(ptr, size) \ <span class="code-keyword">do</span> { \ OBD_ALLOC_GFP(ptr, size, GFP_NOFS | __GFP_NOWARN); \ <span class="code-keyword">if</span> (ptr == NULL) \ OBD_VMALLOC(ptr, size); \ } <span class="code-keyword">while</span> (0) </pre> </div></div> <p>so it will try <tt>kmalloc()</tt> first at whatever size is requested, and only if this fails will it fall back to <tt>vmalloc()</tt> if there isn't a large-order allocation available to fulfill this request. For a fragmented 16MB RPC this will be 64KB at most, so it is likely that kmalloc() will succeed to get an order-4 allocation in most cases. Also, the size of this allocation is driven by the actual RPC size, so it won't do a larger allocation if large RPCs are not enabled. I don't see any significant problem in the current implementation.</p> <p>Alexey, there is work being done on the upstream kernel to allow higher-order page allocations for IO, which would reduce the number of fragments in the IB SG list and could optimize both Lustre RPCs and IB-connected storage like SRP. It would be welcome if you would investigate your proposal to use higher-order allocations or virtual mappings to reduce the SG list and to update the socklnd code to be more efficient. Patches should probably be submitted under a different LU ticket.</p> <p>It sounds like there will be significant changes from VFS level for the support of large page order allocation. Right now kernel dirties pages one by one therefore the order of page allocation is always 1.</p> <p>Alexey, It's good to see this problem from a higher level. Please document your proposal in detail so that we can revisit this later after we have sufficient support from kernel.</p> <p>Andreas - I'm reluctant to increase the initial grant because I would like to be conservative before 16M RPC is widely used. Without this change, the first RPC would be small in size, but after that the RPC size should be 16MB, not a big deal.</p> <p>Andreas,</p> <p>64k allocations failed easly after some time. If you look into several LNet OOM tickets you may see it. So any aged system including servers will switch to use vmalloc for this code.</p> <p>As about SG lists - server side R/O cache kills an IB OFED cache as pool code can't find older map. I worked on other possibilities time to time. when looked to rework an o2ib LND.</p> <p>Over time these larger allocations are going to fail due to the increasing memory fragmentation so the vmalloc penalty will show up.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/19368/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19368/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7990" title=" Large bulk IO support" class="issue-link" data-issue-key="LU-7990"><del>LU-7990</del></a> clio: revise readahead to support 16MB IO<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: d8467ab8a2ca15fbbd5be3429c9cf9ceb0fa78b8</p> <p>Patches have landed to master for 2.9.0</p> <p>Add <a href="https://jira.whamcloud.com/browse/LUDOC-356" title="Lustre Manual - Large Bulk IO" class="issue-link" data-issue-key="LUDOC-356"><del>LUDOC-356</del></a> into lustre manual tracking for Large bulk IO feature.</p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/26955" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26955</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7990" title=" Large bulk IO support" class="issue-link" data-issue-key="LU-7990"><del>LU-7990</del></a> llite: increase whole-file readahead to RPC size<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 25f80fc94b7988e9d356de200573027bc5178a32</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/26955/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26955/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7990" title=" Large bulk IO support" class="issue-link" data-issue-key="LU-7990"><del>LU-7990</del></a> llite: increase whole-file readahead to RPC size<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 627d0133d9d7bef908544537a8a2b2093719fc72</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29738" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29738</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7990" title=" Large bulk IO support" class="issue-link" data-issue-key="LU-7990"><del>LU-7990</del></a> llite: increase whole-file readahead to RPC size<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 9482cd464b354cd1a3edddb158d42eea2d733419</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/29738/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29738/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7990" title=" Large bulk IO support" class="issue-link" data-issue-key="LU-7990"><del>LU-7990</del></a> llite: increase whole-file readahead to RPC size<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 5c1877c4143bddcd3c5c4f2397db5622c2116fce</p> Related LUDOC-356 LU-9356 LU-4533 LU-8193 LU-7140 Development Rank 1|hzy6wn: Rank (Obsolete) 9223372036854775807