https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LUDOC-36 Lustre Documentation <p>We need to add details of how to use this new tool in the manual for 2.2</p> LUDOC-36

Document mds-survey

New Feature Blocker Closed Fixed Minh Diep Peter Jones releases Wed, 18 Jan 2012 18:05:14 +0000 Tue, 27 Nov 2012 07:19:05 +0000 Tue, 27 Mar 2012 13:27:15 +0000 0 1 <p>If the manual is only about how to use mdt_survey script like obdfilter_survey. Minh might be the right person to write it, because he wrote the script. As I knew, he already wrote a README for this script. Thanks.</p> <p>Ah yes. Good idea - Thanks Di!</p> <p>Hi Minh,</p> <p>After you wrote the initial content, I can help to test your instructions. It would be a good opportunity for me to learn how mds-survey works.</p> <p>Thanks,<br/> Zhiqi</p> <p>24.5 Testing MDS Performance (mds-survey)</p> <p>The mds-survey script tests the local metadata performance using the echo_client to drive different layers of the MDS stack: mdd, mdt, osd.</p> <p>It can be used with the following classes of operations</p> <p> 1. Open-create/mkdir/create<br/> 2. Lookup/getattr/setxattr<br/> 3. Delete/destroy<br/> 4. Unlink/rmdir</p> <p> These operations will be run by a variable number of concurrent threads and will test with the number of <br/> directories specified by the user. The run can be executed such that all threads operate in a single directory <br/> (dir_count=1) or in private/unique directory (dir_count=x thrlo=x thrhi=x).</p> <p> The mdd, mdt, or osd instance is driven directly. The script automatically loads the obdecho module if required <br/> and creates instance of echo_client.</p> <p> This script can also create OST objects by providing stripe_count greater than zero.</p> <p>To perform a run:<br/> 1. Start the Lustre MDT.<br/> The Lustre MDT should be mounted on the MDS node to be tested.<br/> 2. Start the Lustre OSTs (optional, only required when test with OST objects)<br/> The Lustre OSTs should be mounted on the OSS node(s).<br/> 3. Run the mds-survey script as explain below</p> <p> The script must be customized according to the components under test and<br/> where it should keep its working files. Customization variables are<br/> described as followed:</p> <p> thrlo threads to start testing. skipped if less than dir_count<br/> thrhi maximum number of threads to test<br/> targets MDT instance<br/> file_count total number of files to test<br/> dir_count total number of directories to test<br/> stripe_count number stripe on OST objects<br/> tests_str test operations. Must have at least "create" and "destroy"<br/> start_number base number for each thread to prevent name collisions<br/> layer MDS stack's layer to be tested</p> <ul class="alternate" type="square"> <li>Create a Lustre configuration using your normal methods</li> </ul> <p> a. Run without OST objects creation:<br/> Setup the Lustre MDS without OST mounted. Then invoke the mds-survey script<br/> $ thrhi=64 file_count=200000 sh mds-survey</p> <p> b. Run with OST objects creation:<br/> Setup the Lustre MDS with at least one OST mounted. Then invoke the mds-survey script with stripe_count <br/> parameter<br/> $ thrhi=64 file_count=200000 stripe_count=2 sh mds-survey</p> <p> Note: a specific mdt instance can be specified using targets variable.<br/> $ targets=lustre-MDT0000 thrhi=64 file_count=200000 stripe_count=2 sh mds-survey</p> <p>Output files:</p> <p>When the script runs, it creates a number of working files and a pair of<br/> result files. All files start with the prefix given by ${rslt}.</p> <p>${rslt}.summary same as stdout<br/> ${rslt}.script_* per-host test script files<br/> ${rslt}.detail_tmp* per-mdt result files<br/> ${rslt}.detail collected result files for post-mortem</p> <p>The script iterates over the given numbers of threads performing<br/> all the specified tests and checking that all test processes<br/> completed successfully.</p> <p>Note that the script may not clean up properly if it is aborted or if it<br/> encounters an unrecoverable error. In this case, manual cleanup may be<br/> required, possibly including killing any running instances of 'lctl' (local<br/> or remote), removing echo_client instances created by the script and<br/> unloading obdecho.</p> <p>Script output:</p> <p>The summary file and stdout contain lines like...</p> <p>mdt 1 file 100000 dir 4 thr 4 create 5652.05 [ 999.01,46940.48] destroy 5797.79 [ 0.00,52951.55]</p> <p>mdt 1 is the total number of MDTs under test.<br/> file 100000 is the total number of files to operate<br/> dir 4 is the total number of directories to operate<br/> thr 4 is the total number of threads operate over all directories<br/> create<br/> destroy are the test name. More tests will be displayed on the same line.<br/> 565.05 is the aggregate operations over all MDTs measured by<br/> dividing the total number of operations by the elapsed time.<br/> <span class="error">&#91;999.01,46940.48&#93;</span> are the minimum and maximum instantaneous operation seen on<br/> any individual MDT.</p> <p>I often see <tt>SHORT</tt> and <tt>ERROR</tt> in the output of my mds-survey run. This needs to be explained in the doc too.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"># mds-survey Tue Feb 7 15:20:45 EST 2012 /usr/bin/mds-survey from wc0008 mdt 1 file 100000 dir 4 thr 4 create 7039.07 ERROR lookup 191596.67 SHORT md_getattr 106189.70 SHORT setxattr 18632.91 ERROR destroy 10423.02 ERROR mdt 1 file 100000 dir 4 thr 8 create 6911.21 ERROR lookup 218940.93 SHORT md_getattr 130979.97 SHORT setxattr 4427.82 ERROR destroy 6362.05 ERROR mdt 1 file 100000 dir 4 thr 16 create 5213.21 ERROR lookup 189467.26 SHORT md_getattr 113743.37 ERROR setxattr 1720.42 ERROR destroy 706.99 ERROR mdt 1 file 100000 dir 4 thr 32 create </pre> </div></div> <p>Script is here: </p> <p><a href="http://review.whamcloud.com/#change,1969" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,1969</a></p> <p>Related LU is here:</p> <p><a href="http://jira.whamcloud.com/browse/LU-633" class="external-link" rel="nofollow">http://jira.whamcloud.com/browse/LU-633</a></p> <p>Minh, can you explain the SHORT messages?</p> <p>From Minh:<br/> ERROR often caused by out of inode on the mdt<br/> SHORT often means there isn't enough files to generate enough data to collect between 1 second</p> <p>Minh did this work under <a href="http://review.whamcloud.com/#change,2072" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,2072</a></p> <p>merged with master</p> Duplicate LUDOC-38 Development Rank 1|hzvl2v: Rank (Obsolete) 7176