https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6770 Lustre <p> With many OSCs, the osc will pre-alloc memory at start.<br/> That will occupy the memory of application, especially when the<br/> client need to interact with hundreds of OSTs.</p> <p> We can solve it by using a golbal per_cpu pool 'osc_rq_pools' rather than<br/> local pool for per osc to change this situation. The upper limit<br/> size of requests in pools is about 1 percent of the total memory.</p> <p> Also, administrator can use a module parameter to limit the momory<br/> usage by:<br/> options osc osc_reqpool_mem_max=num<br/> The unit of num is MB, and the upper limit will be:<br/> MIN(num, 1% total memory)</p> LU-6770

use per_cpu request pool osc_rq_pools

Improvement Minor Resolved Fixed Yang Sheng Wang Shilong patch Sat, 27 Jun 2015 06:45:20 +0000 Fri, 4 Aug 2017 19:41:27 +0000 Thu, 13 Aug 2015 02:46:26 +0000 Lustre 2.8.0 0 9 <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="http://review.whamcloud.com/15422" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15422</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6770" title="use per_cpu request pool osc_rq_pools" class="issue-link" data-issue-key="LU-6770"><del>LU-6770</del></a> osc: use per_cpu request pool osc_rq_pools<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8fec66ecc9871d9b0c52f7f5ce65bda3a130cc38</p> <p>Yang Sheng</p> <p>Could you please take care of this patch?</p> <p>Thanks</p> <p>Peter</p> <p>Hello Wang,</p> <p>I have reviewed the patch you provided and have started to run Lustre, with this patch, on my local system to verify that it works well.</p> <p>Once I have completed my testing I will work with you to get the patch reviewed by others, so that it can be landed.</p> <p>Thanks,<br/> Yang Sheng.</p> <p>The patch has passed test &amp; review process. But Oleg has some comment as below:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[CST下午1时53分29秒] yang sheng: could you please give me some point about http://review.whamcloud.com/#/c/15422/ [CST下午1时54分01秒] yang sheng: can it be landed or still need waiting a while? [CST下午1时54分07秒] Oleg Drokin: why do we need percpu pool there? [CST下午1时54分28秒] Oleg Drokin: I mean it's still an improvement, but what if I have 260 CPUs? [CST下午1时54分40秒] Oleg Drokin: I would imagine havign a static pool of a fixed size is probably best of all [CST下午1时57分45秒] Oleg Drokin: I think the pool does not need to be super big. Just a fixed number of reqests, something like 50 (or 100, need to see how big they are) should be enough. we only expect to use them during severe OOM anyway [CST下午1时58分00秒] Oleg Drokin: with perhaps a module parameter if somebody wants an override [CST下午1时59分05秒] yang sheng: Yes, it is reasonable. [CST下午1时59分44秒] yang sheng: as this patch given the limit is %1 of total memory. [CST下午2时00分16秒] yang sheng: seem big than you are point. [CST下午2时01分19秒] Oleg Drokin: Yes. I feel it's really excessive. But the initial reasoning was that every OSC could have up to 32M of dirty pages and can send up to 8 (default) RPCs in flight. [CST下午2时01分40秒] Oleg Drokin: so every OSC had this pool in order to send the many RPCs even in OOM [CST下午2时02分02秒] Oleg Drokin: in reality if you have 2000 OSTs, it's unlikely you'd have dirty pages in all of them at the same time [CST下午2时02分11秒] Oleg Drokin: so we need to be reasonable here [CST下午2时02分38秒] Oleg Drokin: 1% of 1T of memory is still a cool 10G [CST下午2时03分34秒] yang sheng: so a fixed size is enough to handle such stiuation. [CST下午2时03分46秒] Oleg Drokin: finding a proper number is going to be tricky, but I feel it should be on the lower side somewhere in tens or low hundreds for most cases except perhaps the most extreme ones [CST下午2时04分10秒] Oleg Drokin: that's why having an override is important of course, with a good documentation about it like I explained above [CST下午2时05分02秒] yang sheng: I see. got it. Thank you very much. Oleg. </pre> </div></div> <p>Hi Yang Sheng,</p> <p>So maybe we can set @osc_reqpool_mem_max=100MB or so in default, and it will try to allocate memory by checking<br/> min(100M， 1% of memory), dose this make sense for you?</p> <p>Best Regards,<br/> Shilong</p> <p>Hi, Shilong,</p> <p>I think maybe we still need consider OST number when we decide the mem_max parameter. Unless it big than a fixed number or override by module paramter. How do you think about it?</p> <p>Thanks,<br/> Yang Sheng</p> <p>Li Xi (lixi@ddn.com) uploaded a new patch: <a href="http://review.whamcloud.com/15585" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15585</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6770" title="use per_cpu request pool osc_rq_pools" class="issue-link" data-issue-key="LU-6770"><del>LU-6770</del></a> osc: use global osc_rq_pool to reduce memory usage<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: c22aa4d8c553e974214a27e516728d88df73663c</p> <p>patch <a href="http://review.whamcloud.com/15585" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15585</a> Abandoned. new patch is <a href="http://review.whamcloud.com/#/c/15422/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/15422/</a></p> <p>Here is quick benchmark results on master with/without <a href="http://review.whamcloud.com/#/c/15422" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/15422</a><br/> 4 x OSS and an client(2 x E5-2660v3, 20 CPU cores, 128GB memory and 1 x FDR Infiniband)</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'>&nbsp;</td> <td class='confluenceTd'>master(w/o stress)</td> <td class='confluenceTd'>master(w/ stress)</td> <td class='confluenceTd'>master+15422(w/o stress)</td> <td class='confluenceTd'>master+15422(w/ stress)</td> </tr> <tr> <td class='confluenceTd'>Write(M/B/sec) </td> <td class='confluenceTd'>5604</td> <td class='confluenceTd'>4838</td> <td class='confluenceTd'>5702</td> <td class='confluenceTd'>4846</td> </tr> <tr> <td class='confluenceTd'>Read(M/B/sec) </td> <td class='confluenceTd'>4218</td> <td class='confluenceTd'>3703</td> <td class='confluenceTd'>4261</td> <td class='confluenceTd'>3939</td> </tr> </tbody></table> </div> <p>Here is IOR syntax on this test.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># mpirun -np 10 /work/ihara/IOR -w -e -t 1m -b 26g -k -F -o /scratch1/file # pdsh -g oss,client "sync; echo 3 &gt; /proc/sys/vm/drop_caches" # mpirun -np 10 /work/ihara/IOR -r -e -t 1m -b 26g -F -o /scratch1/file </pre> </div></div> <p>For IOR with stress testing, I generated memory pressure with "stress" command and ran IOR under it.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># stress --vm 10 --vm-bytes 10G </pre> </div></div> <p>No perforamnce regression with patch 15422 so far.</p> <p>This is memory usage on client with 200 OSTs configuation.</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'>&nbsp;</td> <td class='confluenceTd'>Mem usage</td> <td class='confluenceTd'>Slab allocation</td> </tr> <tr> <td class='confluenceTd'>master</td> <td class='confluenceTd'> 444616</td> <td class='confluenceTd'> 136412</td> </tr> <tr> <td class='confluenceTd'>master+15422</td> <td class='confluenceTd'> 91324 </td> <td class='confluenceTd'>64412 </td> </tr> </tbody></table> </div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Mem usage=MemFree(Before mount) - MemFree(after mount) Slab allocation=Slab(After mount) - Slab(Before mount) </pre> </div></div> <p>Patch 15422 helps to reduce significant memory usages.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/15422/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/15422/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6770" title="use per_cpu request pool osc_rq_pools" class="issue-link" data-issue-key="LU-6770"><del>LU-6770</del></a> osc: use global osc_rq_pool to reduce memory usage<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 44c4f47c4d1f185831d4629cc9ca5ae5f50a8e07</p> <p>Patch landed, Close this ticket.</p> <p>Wrong ticket number.</p> Related Development Rank 1|hzxgqn: Rank (Obsolete) 9223372036854775807