https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-17037 Lustre <p>As a long term effort to improve the overall stability of lustre, the test suite should be evaluated and modified to allow for testing against OST and MDT sets which contain index numbers that are both high and sparse.</p> <p>What I mean by this is that in many cases we're seeing more sites choosing to deploy flash OSTs within the first 100 slots, and then moving all HDD OSTs to index slots &gt; 100 or vice-versa.</p> <p>This has been shown to introduce issues which some features (most recently a memory corruption issue within pool quotas: <a href="https://jira.whamcloud.com/browse/LU-17034" title="memory corruption caused by bug in qmt_seed_glbe_all" class="issue-link" data-issue-key="LU-17034"><del>LU-17034</del></a>)</p> <p>Right now the existing test suite assumes that OSTs will exist on certain low-index numbers which are hard coded into it.</p> <p>This is sub-optimal to catch cases such as <a href="https://jira.whamcloud.com/browse/LU-17034" title="memory corruption caused by bug in qmt_seed_glbe_all" class="issue-link" data-issue-key="LU-17034"><del>LU-17034</del></a>, as a result we will need to modify the test suite to allow for testing these cases for all features.</p> LU-17037

Tests should run with high and sparse index numbers for OSTs and MDTs

Task Minor In Progress Unresolved Jian Yu Colin Faber tests Thu, 17 Aug 2023 15:04:34 +0000 Thu, 26 Oct 2023 23:43:07 +0000 Lustre 2.15.3 Lustre 2.17.0 0 4 <p>Note that there are some test cases which are already testing sparse OST indexes, for example <tt>conf-sanity.sh</tt> <tt>test_81</tt>, <tt>test_82a</tt>.</p> <p>There is already <tt>test-framework.sh</tt> support for non-sequential OST numbers by using <tt>OST_INDEX_LIST</tt> to specify the index values, but it might make sense to improve this support as needed. This is "documented" in <tt>lustre/tests/cfg/local.sh</tt>:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># OST indices can be specified as follows: # OSTINDEX1="1" # OSTINDEX2="2" # OSTINDEX3="4" # ...... # or # OST_INDEX_LIST="[1,2,4-6,8]" # [n-m,l-k,...], where n &lt; m and l &lt; k, etc. # # The default index value of an individual OST is its facet number minus 1. # More specific ones override more general ones. See facet_index(). </pre> </div></div> <p>What needs to be done here is to fix the many, many subtests that assume <tt>ost1 == OST0000</tt>, <tt>ost2 == OST0001</tt>, etc. (often using the facet number - 1 as the index, or the index number + 1 as the facet name), and instead use helpers that map the facet name/number to the OST number in <tt>OST_INDEX_LIST</tt> (which is mapped internally to an associative array <tt>$OST_INDICES</tt>.</p> <p>There are some helper functions that exist, but might need to be updated, and definitely need to be used more widely:</p> <ul class="alternate" type="square"> <li><tt>facet_number()</tt> converts a facet name like <tt>ostN</tt> to the facet number <tt>N</tt>, looks OK</li> <li><tt>facet_type()</tt> converts a facet name like <tt>ostN</tt> to the type <tt>OST</tt>, looks OK</li> <li><tt>facet_svc()</tt> converts a facet name like <tt>ostN</tt> to the service name via <tt>${facet}_svc</tt> variables, likely <tt>$fsname-$typeXXXX</tt> (not sure)</li> <li><tt>facet_index()</tt> converts a facet name like <tt>ostN</tt> to the OST index number <tt>n</tt> (whatever it is), via <tt>OSTINDEXN</tt> or <tt>OST_INDICES<span class="error">&#91;N&#93;</span></tt> variables, though I'm not sure why we have both). This <em>should</em> be used <b>lots</b> of places but is not.</li> </ul> <p>It might be useful to add some more helper functions to simplify the remapping, like:</p> <ul class="alternate" type="square"> <li><tt>facet_ost_name()</tt> converts an index number like "<tt>n</tt>" to the OST facet name <tt>ostN</tt></li> <li><tt>facet_mdt_name()</tt> converts an index number like "<tt>n</tt>" to the MDT facet name <tt>mdsN</tt></li> </ul> <p>There is currently no support for non-contiguous MDT index numbers in <tt>test-framework.sh</tt>, and I don't think this has been tested anywhere. Until we get MDT pools, I'm not sure if there is much motivation to configure discontiguous index numbers, but I'm sure it will happen somewhere eventually. However, I don't think implementing support for testing this and fixing the many resulting bugs is a priority compared to fixing discontiguous OST support.</p> <p>Probably due to the many subtests that need to be fixed, it would make sense to split patches into separate files (or maybe multiple patches for large scripts like <tt>sanity.sh</tt>) so that they can land independently, unless there are only a few changes in a single file.</p> <p>It might be possible to test which subtests are having obvious problems by running "<tt>env=OST_INDEX_LIST=<span class="error">&#91;0,10,20,40,55,60,80&#93;</span></tt>" (for <tt>OSTCOUNT=8</tt>) or similar in autotest (or just set <tt>OST_INDEX_LIST</tt> in your local test environment) and run through the test scripts multiple times to fix failures as they are hit. Probably a huge number of test failures would be hit if there is no <tt>OST0000</tt>, so that might be last to test after other subtests are fixed.</p> <p>In conf-sanity test_82a(), the random sparse indices for OSTs are generated as follows:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> # Format OSTs with random sparse indices. local i local index local ost_indices local LOV_V1_INSANE_STRIPE_COUNT=65532 <span class="code-keyword">for</span> i in $(seq $OSTCOUNT); <span class="code-keyword">do</span> index=$(((RANDOM * 2) % LOV_V1_INSANE_STRIPE_COUNT)) ost_indices+=<span class="code-quote">" $index"</span> done ost_indices=$(comma_list $ost_indices) stack_trap <span class="code-quote">"restore_ostindex"</span> EXIT echo -e <span class="code-quote">"\nFormat $OSTCOUNT OSTs with sparse indices $ost_indices"</span> OST_INDEX_LIST=[$ost_indices] formatall </pre> </div></div> <p>To make a quick experiment, I used the above way in <tt>cfg/local.sh</tt> to set <tt>OST_INDEX_LIST</tt> with random sparse indices, and then ran runtests. It passed:<br/> <a href="https://testing.whamcloud.com/test_sets/b4443be0-b163-4ec9-9271-f36464ac41c4" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/b4443be0-b163-4ec9-9271-f36464ac41c4</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 95248 4340 82252 6% /mnt/lustre[MDT:0] lustre-OST090c_UUID 142216 7288 120928 6% /mnt/lustre[OST:2316] lustre-OST4b24_UUID 142216 9088 119128 8% /mnt/lustre[OST:19236] lustre-OST9234_UUID 142216 9416 118800 8% /mnt/lustre[OST:37428] lustre-OST986c_UUID 142216 14568 113648 12% /mnt/lustre[OST:39020] filesystem_summary: 568864 40360 472504 8% /mnt/lustre </pre> </div></div> <p>I'm going to push a fortestonly patch to run the full test group by autotest with the above change to see which subtests are failing.</p> <p>"Jian Yu &lt;yujian@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/52158" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/52158</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-17037" title="Tests should run with high and sparse index numbers for OSTs and MDTs" class="issue-link" data-issue-key="LU-17037">LU-17037</a> tests: full group testing with sparse OST indexes<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: a5a1a2346e87afe5914b8ab098583ae8920e4598</p> <p>I'm vetting the test results in <a href="https://review.whamcloud.com/52158" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/52158</a> on master branch.<br/> With sparse OST indexes "OST_INDEX_LIST=<span class="error">&#91;0,10,20,40,55,60,80&#93;</span>" (for OSTCOUNT=7) and "ENABLE_QUOTA=yes" specified, at least performance-sanity test 2 and sanity-benchmark test dbench crashed on master branch. I just updated <a href="https://jira.whamcloud.com/browse/LU-17034" title="memory corruption caused by bug in qmt_seed_glbe_all" class="issue-link" data-issue-key="LU-17034"><del>LU-17034</del></a> with the detailed LBUG info.<br/>  </p> <p>I think of particular interest is also sanity-quota and ost-pools, since pools + quota + sparse OST index was the source of the problem.</p> <p>Here are the full-dne-part-{1,2,3} test results with "OST_INDEX_LIST=<span class="error">&#91;0,10,20,40,55,60,80&#93;</span>" and "ENABLE_QUOTA=yes":<br/> <a href="https://testing.whamcloud.com/test_sessions/551b2e1f-2493-411f-9cbd-bb28dd0b1607" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/551b2e1f-2493-411f-9cbd-bb28dd0b1607</a><br/> <a href="https://testing.whamcloud.com/test_sessions/f7c3d574-e349-42ee-99f3-35c761205148" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/f7c3d574-e349-42ee-99f3-35c761205148</a><br/> <a href="https://testing.whamcloud.com/test_sessions/a1fffb9a-56b2-439a-ab48-20f9c5fcd5eb" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/a1fffb9a-56b2-439a-ab48-20f9c5fcd5eb</a><br/> sanity-quota hit the LBUG at test 0. ost-pools didn't crash and it passed with 38 subtests out of 56.</p> <p>I just removed the "ENABLE_QUOTA=yes" test parameter and triggered the full group testing again to make LBUG not block other test suites. After <a href="https://jira.whamcloud.com/browse/LU-17034" title="memory corruption caused by bug in qmt_seed_glbe_all" class="issue-link" data-issue-key="LU-17034"><del>LU-17034</del></a> is fixed, I'll add the parameter and test again.<br/> Now I'm looking into the non-LBUG failures and trying to fix them.</p> Related LU-17034 LU-16331 Development Rank 1|i03t87: Rank (Obsolete) 9223372036854775807 Severity