https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-7689 Lustre <p>lu_site_init() will allocate a hash table during the setup of both client and osd,<br/> which use the same default formula: we assume the lu_site cache can take up to 20% of total memory.</p> <p>It makes sense for osd but on client, we are allocating a ~128M hash table on a 32G box per mount. To make it worse we are mounting multiple lustre fs on a box so it can take ~128M * mounts of memory.</p> <p>We have lu_cache_percent as a module param but we should limit the hash table size by default on clients.</p> LU-7689

limit lu_site hash table size on clients

Improvement Minor Resolved Fixed WC Triage Li Dongyang patch Wed, 20 Jan 2016 01:16:11 +0000 Mon, 14 Mar 2016 03:19:03 +0000 Mon, 14 Mar 2016 03:19:03 +0000 Lustre 2.9.0 0 4 <p>Li Dongyang (dongyang.li@anu.edu.au) uploaded a new patch: <a href="http://review.whamcloud.com/18048" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18048</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7689" title="limit lu_site hash table size on clients" class="issue-link" data-issue-key="LU-7689"><del>LU-7689</del></a> obdclass: limit lu_site hash table size on clients<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: d715c3ad1be054f52c9787472fae9d16b5e6f94e</p> <p>I've done mdtest with 16 processes, 262144 objects each, 3 iterations.<br/> That gives us 4 million objects each iteration.</p> <p>without the patch:</p> <p>mdtest-1.9.3 was launched with 16 total task(s) on 1 node(s)<br/> Command line used: /system/Benchmarks/mdtest/1.9.3/mdtest -d /mnt/testfs/z00/dyl900/mdtest -n 262144 -i 3<br/> Path: /mnt/testfs/z00/dyl900<br/> FS: 854.4 TiB Used FS: 0.6% Inodes: 854.5 Mi Used Inodes: 0.0%</p> <p>16 tasks, 4194304 files/directories</p> <p>SUMMARY: (of 3 iterations)<br/> Operation Max Min Mean Std Dev<br/> --------- &#8212; &#8212; ---- -------<br/> Directory creation: 4169.451 4049.008 4113.665 49.569<br/> Directory stat : 8553.647 7678.810 8243.482 399.929<br/> Directory removal : 2312.643 1684.909 1941.188 268.901<br/> File creation : 3714.316 3666.937 3686.580 20.171<br/> File stat : 7936.584 7511.260 7787.747 195.697<br/> File read : 8772.322 7397.213 8055.384 562.922<br/> File removal : 2814.176 2177.893 2579.797 285.497<br/> Tree creation : 3472.106 2173.215 2956.409 562.996<br/> Tree removal : 7.550 6.808 7.093 0.326</p> <p>with the patch applied:<br/> mdtest-1.9.3 was launched with 16 total task(s) on 1 node(s)<br/> Command line used: /system/Benchmarks/mdtest/1.9.3/mdtest -d /mnt/testfs/z00/dyl900/mdtest -n 262144 -i 3<br/> Path: /mnt/testfs/z00/dyl900<br/> FS: 854.4 TiB Used FS: 0.6% Inodes: 854.5 Mi Used Inodes: 0.0%</p> <p>16 tasks, 4194304 files/directories</p> <p>SUMMARY: (of 3 iterations)<br/> Operation Max Min Mean Std Dev<br/> --------- &#8212; &#8212; ---- -------<br/> Directory creation: 4258.673 4128.302 4178.702 57.184<br/> Directory stat : 8203.775 8060.866 8147.118 61.982<br/> Directory removal : 2356.021 1868.107 2173.262 217.180<br/> File creation : 3683.373 3608.999 3655.445 33.067<br/> File stat : 8058.992 7698.923 7898.330 149.529<br/> File read : 8838.700 8575.945 8680.637 113.714<br/> File removal : 2857.897 2262.444 2657.039 279.036<br/> Tree creation : 4132.319 2755.784 3630.101 620.513<br/> Tree removal : 7.673 6.568 7.151 0.453</p> <p>and the hash_bd_depmax is 130 without the patch, 221 while applied.</p> <p>Is there any other particular benchmark I should run?<br/> Thanks </p> <p>mdtest cannot handle the client-side cache properly since it is general test tool, not Lustre special. I would suggest to do the following test for the performance of traversing directory:</p> <p>1) Assume the test directory is TDIR, under the TDIR, generate 10M regular files, you can do that via 'touch' (slow), or via Lustre test tool 'createmany' (faster than touch).<br/> 2) After generating the test data set, remount the client to drop the client side cache. Then "ls -l TDIR" to measure the performance without cache.<br/> 3) And then "ls -l TDIR" again to measure the performance with cache.</p> <p>You can measure the performance with and without your patch, then compare them. Usually, we use multiple clients to generate the test data set in parallel to increase the create speed. 0-stripe file is the most fast case. Depends on you test environment and time, you also can consider to measure 1-stripe and 4-stirpes cases.</p> <p>Hi all,<br/> I followed the comment from nash, created 10M files with createmany and here are the results from 0-stripe file:<br/> without the patch:<br/> <span class="error">&#91;12:04:58 root@r3:z00&#93;</span> # time ls -l dyl900 &gt; /dev/null</p> <p>real 26m57.756s<br/> user 1m7.876s<br/> sys 15m39.102s<br/> <span class="error">&#91;12:32:54 root@r3:z00&#93;</span> # time ls -l dyl900 &gt; /dev/null</p> <p>real 27m19.028s<br/> user 1m6.301s<br/> sys 15m55.731s</p> <p>with the patch:<br/> <span class="error">&#91;14:15:29 root@r3:z00&#93;</span> # time ls -l dyl900 &gt; /dev/null</p> <p>real 25m10.092s<br/> user 1m6.833s<br/> sys 13m53.716s<br/> <span class="error">&#91;14:40:51 root@r3:z00&#93;</span> # time ls -l dyl900 &gt; /dev/null</p> <p>real 28m52.916s<br/> user 1m8.438s<br/> sys 16m9.032s</p> <p>Looking into testing with 1-stripe and 4-stripes files, but I do have a question: how can I create 1-stripe and 4-stripes files using 'createmany'?</p> <blockquote> <p>Looking into testing with 1-stripe and 4-stripes files, but I do have a question: how can I create 1-stripe and 4-stripes files using 'createmany'?</p></blockquote> <p>"lfs setstripe -c 1 $TDIR", then all the regular files created under $TDIR (after the setstripe) will be 1-striped unless you specify the file layout manually. Similarly for 4-striped case, specify "-c 4".</p> <p>Here are the results for 1 and 4 stripes:</p> <p>1-stripe<br/> baseline:<br/> bash-4.1# time ls -l testdir &gt; /dev/null</p> <p>real 25m22.670s<br/> user 1m4.831s<br/> sys 13m41.840s<br/> bash-4.1# time ls -l testdir &gt; /dev/null</p> <p>real 27m5.056s<br/> user 1m5.908s<br/> sys 15m2.135s</p> <p>patched:<br/> bash-4.1# time ls -l testdir &gt; /dev/null </p> <p>real 25m42.354s<br/> user 1m9.786s<br/> sys 13m57.352s<br/> bash-4.1# time ls -l testdir &gt; /dev/null </p> <p>real 27m3.739s<br/> user 1m10.196s<br/> sys 14m59.850s</p> <p>4-stripes:</p> <p>baseline:<br/> bash-4.1# time ls -l testdir &gt; /dev/null </p> <p>real 41m45.328s<br/> user 1m14.652s<br/> sys 32m12.239s<br/> bash-4.1# time ls -l testdir &gt; /dev/null </p> <p>real 46m57.302s<br/> user 1m15.295s<br/> sys 36m12.470s</p> <p>patched:<br/> bash-4.1# time ls -l testdir &gt; /dev/null </p> <p>real 39m48.663s<br/> user 1m15.712s<br/> sys 29m54.248s<br/> bash-4.1# time ls -l testdir &gt; /dev/null </p> <p>real 42m28.805s<br/> user 1m13.754s<br/> sys 33m8.527s</p> <p>It beats me that with 4 stripes the patched client is actually faster, I did multiple runs to verify that. </p> <p>Dongyang, thanks for sharing the test results.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/18048/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/18048/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-7689" title="limit lu_site hash table size on clients" class="issue-link" data-issue-key="LU-7689"><del>LU-7689</del></a> obdclass: limit lu_site hash table size on clients<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 522c1eb4d2f5faf1fa87be07d9617df1439fc0d6</p> Development Rank 1|hzxylr: Rank (Obsolete) 9223372036854775807