https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8641 Lustre <p>1<br/> on real HW clusters we run metabench with parameters :</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">mbench_NFILES=50000 mbench_THREADS=16 + /test-tools/mpich2/metabench/src/metabench -w /mnt/fs1/d0.metabench -c 50000 -C -S -k -p /test-tools/mpich2/metabench/dictionary + chmod 0777 /mnt/fs1 drwxrwxrwx 5 root root 270336 Apr 25 10:02 /mnt/fs1 + su mpiuser sh -c <span class="code-quote">"/usr/bin/mpirun -np 112 /test-tools/mpich2/metabench/src/metabench -w /mnt/fs1/d0.metabench -c 50000 -C -S -k -p /test-tools/mpich2/metabench/dictionary "</span> </pre> </div></div> <p>test itself comleted in 20 minutes with creation rate :</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">------------- ------------ ---------- ---------- ---------- Total 5600000 1112.045 5035.769 Elapsed 5600000 1112.047 5035.758 ------------- ------------ ---------- ---------- ---------- Average 50000 1097.669 45.556 Std Dev 11.592 0.489 ( 1.06%) ( 1.07%) [2016-04-0025 10:21:30] Leaving par_create_multidir </pre> </div></div> <p>then test runs rm -rf &lt;dir&gt; with 5 millions file and can not be completed in 1.5 hours.</p> <p>2. On real cluster it makes sense to leave files not deleted, so total number of files would grow with time.</p> LU-8641

speedup run_metabech () : make cleanup optional

Improvement Minor Resolved Fixed WC Triage Elena Gryaznova patch Mon, 26 Sep 2016 18:42:02 +0000 Sun, 23 Apr 2017 03:52:26 +0000 Sun, 23 Apr 2017 03:52:26 +0000 Lustre 2.8.0 Lustre 2.10.0 0 4 <p>Elena Gryaznova (elena.gryaznova@seagate.com) uploaded a new patch: <a href="http://review.whamcloud.com/22852" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/22852</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8641" title="speedup run_metabech () : make cleanup optional" class="issue-link" data-issue-key="LU-8641"><del>LU-8641</del></a> tests: tests: speedup metabench<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8e33d6aca1411f524e40493f0e39e44b02054e48</p> <p>Instead of using "rm -r $dir" to empty directories, it is likely much faster to use "find $dir -print0 -type f | xargs -0 unlink; rm -r $dir" to delete the files first, then the remaining directories (if any).</p> <p>Ideally it would be better to actually fix the "rm -r" performance for large directories, rather than just fixing the test script, since this is a performance problem that also affects users. This is discussed in <a href="https://jira.whamcloud.com/browse/LU-3308" title="large readdir chunk size slows unlink/&quot;rm -r&quot; performance" class="issue-link" data-issue-key="LU-3308">LU-3308</a>, including options to reduce readdir() RPC chunksize (possibly dynamically when contention is present), and alternately keeping the directory contents cached on the client after the DLM lock has been cancelled (as allowed by POSIX) to avoid cache ping-pong.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/22852/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/22852/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8641" title="speedup run_metabech () : make cleanup optional" class="issue-link" data-issue-key="LU-8641"><del>LU-8641</del></a> tests: tests: speedup metabench<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a9339b851789348348e77a02bff16f8d3af69091</p> <p>Landed for 2.10</p> Duplicate Related LU-3308 Development Rank 1|hzypmv: Rank (Obsolete) 9223372036854775807