https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9887 Lustre <p>This issue was created by maloo for Bob Glossman &lt;bob.glossman@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/742f02b4-837a-11e7-b90b-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/742f02b4-837a-11e7-b90b-5254006e85c2</a>.</p> <p>The sub-test test_9a failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(4) Got speed 952, expected less than 144 </pre> </div></div> <p>This might be a dup of <a href="https://jira.whamcloud.com/browse/LU-8877" title="sanity-lfsck test_9a error" class="issue-link" data-issue-key="LU-8877"><del>LU-8877</del></a>, but those haven't been reported for quite a while.<br/> Creating a new Jira ticket for recent instances. Will let somebody else decide if they are dups.</p> <p>Info required for matching: sanity-lfsck 9a<br/> Info required for matching: sanity-lfsck 9b</p> LU-9887

sanity-lfsck test_9a: FAIL: (4) Got speed 952, expected less than 144

Bug Critical Resolved Fixed nasf Maloo Thu, 17 Aug 2017 19:45:30 +0000 Mon, 7 Jan 2019 18:52:56 +0000 Fri, 1 Dec 2017 05:28:52 +0000 Lustre 2.11.0 Lustre 2.10.6 0 7 <p>think this fail may be showing up now due to recent landing of "<a href="https://jira.whamcloud.com/browse/LU-9295" title="sanity-lfsck tests 9a and 9b skipped due to uninitialized server version" class="issue-link" data-issue-key="LU-9295"><del>LU-9295</del></a> test: skip sanity-lfsck 9 less than 2.8"</p> <p>Before that tests 9a &amp; 9b were probably always skipped.</p> <p>James Nunez (james.a.nunez@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28588" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28588</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9887" title="sanity-lfsck test_9a: FAIL: (4) Got speed 952, expected less than 144" class="issue-link" data-issue-key="LU-9887"><del>LU-9887</del></a> tests: stop running sanity-lfsck tests 9a,9b<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 584667e902fda338b0f0a7fdbdb85edbfe320441</p> <p>more on master:<br/> <a href="https://testing.hpdd.intel.com/test_sets/66b2ad28-839f-11e7-b33c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/66b2ad28-839f-11e7-b33c-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/cb972bee-83a7-11e7-b33c-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/cb972bee-83a7-11e7-b33c-5254006e85c2</a></p> <p>One minor nit to fix when this test is repaired - it is adding 20% margin to both the time and the calculated speed, which is why it expects "<tt>less than 144"</tt> instead of "<tt>less than 120</tt>" for the actual speed:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> local RUN_TIME1=10 local TIME_DIFF=2 local MAX_SPEED=$((BASE_SPEED1 * (RUN_TIME1 + TIME_DIFF) / \ RUN_TIME1 * 12 / 10)) </pre> </div></div> <p>Either the <tt>TIME_DIFF</tt> or the <tt>12 / 10</tt> should be removed.</p> <p>Also, to improve the calculations, rather than using the <tt>RUN_TIME1</tt> and <tt>RUN_TIME2</tt> values to calculate the elapsed time, it would be better to record the start and end times for each step:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> $START_LAYOUT -r -s $BASE_SPEED1 || error <span class="code-quote">"(2) Fail to start LFSCK!"</span> local START_TIME1=$SECONDS sleep 10 STATUS=$($SHOW_LAYOUT | awk <span class="code-quote">'/^status/ { print $2 }'</span>) local RUN_TIME1=$((SECONDS - START_TIME)) </pre> </div></div> <p>I put the start/end times both after the commands for consistency, since SSH can sometimes take a noticeable time on the VMs.</p> <p>The issue is related with div_u64() and do_div(). Originally, we use do_div() in LFSCK for calculating the LFSCK scanning speed. Recently, it is replaced by div_u64() via the patch <a href="https://review.whamcloud.com/26466" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/26466</a>. But the main difference between the two functions are:</p> <p>The do_div(a, b) returns the remainder, the quotient is put in the dividend(@a).<br/> The div_u64(a, b) returns the quotient, both the dividend(@a) and divisor(@b) will NOT be changed.</p> <p>Both of them use 64-bits dividend(@a) and 32-bits divisor(@b). For LFSCK case, do_div() is enough. I will make patch to fix it.</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28617" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28617</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9887" title="sanity-lfsck test_9a: FAIL: (4) Got speed 952, expected less than 144" class="issue-link" data-issue-key="LU-9887"><del>LU-9887</del></a> lfsck: repalce div_u64 with do_div<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: cc198dafc29269a06fc3d72aebb8e73357e32144</p> <p>That was done to avoid <a href="https://jira.hpdd.intel.com/browse/LU-6174" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-6174</a>. I guess we need to look at math64.h to see what the proper function is.</p> <p>But div_u64() also uses 32-bits input parameter as the divisor, the return value of div_u64() is 64 bits.</p> <p>On the other hand, for LFSCK, the divisor is the run time with second unit. It is impossible that the LFSCK run time exceeds 32-bit seconds. So do_div() is enough for LFSCK.</p> <p>True but if you want to do that you need to add a comment saying its okay to truncate the result. What will happen in the future is some one will look at the code and assume do_div() is wrong due to the truncate issue. So I ask you place a comment stating this. Personally I like to see things done is proper way i.e use a correct div64 function instead of reverting to do_div() but its not a hard requirement.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=simmonsja" class="user-hover" rel="simmonsja">simmonsja</a>, <a href="https://review.whamcloud.com/28617" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28617</a> is refreshed, the new version uses dir_u64(), but gets the result from the function return value. Please check. Thanks!</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/28588/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28588/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9887" title="sanity-lfsck test_9a: FAIL: (4) Got speed 952, expected less than 144" class="issue-link" data-issue-key="LU-9887"><del>LU-9887</del></a> tests: ignore error sanity-lfsck test 9a,b<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 6754b09335508ca4d977d10d1d05b5befd1a8aad</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/28617/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28617/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9887" title="sanity-lfsck test_9a: FAIL: (4) Got speed 952, expected less than 144" class="issue-link" data-issue-key="LU-9887"><del>LU-9887</del></a> lfsck: calculate LFSCK speed properly<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: cf800c062c8c6424c442509139297095f8a708db</p> <p>Landed for 2.11</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29293" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29293</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9887" title="sanity-lfsck test_9a: FAIL: (4) Got speed 952, expected less than 144" class="issue-link" data-issue-key="LU-9887"><del>LU-9887</del></a> tests: ignore error sanity-lfsck test 9a,b<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 917d9a4021fc0ccb911fbb2b0e261a781b91f2be</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/29294" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29294</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9887" title="sanity-lfsck test_9a: FAIL: (4) Got speed 952, expected less than 144" class="issue-link" data-issue-key="LU-9887"><del>LU-9887</del></a> lfsck: calculate LFSCK speed properly<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 8eb4db6c1125511d870cd848fd8cd5eba9e944eb</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/29293/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29293/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9887" title="sanity-lfsck test_9a: FAIL: (4) Got speed 952, expected less than 144" class="issue-link" data-issue-key="LU-9887"><del>LU-9887</del></a> tests: ignore error sanity-lfsck test 9a,b<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 2efab9c82e806dc53b98fcb157aaed60af79a799</p> <p>still seeing fails on master after the landing of <a href="https://review.whamcloud.com/28588" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28588</a> and <a href="https://review.whamcloud.com/28617:" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28617:</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/8ab76200-afbd-11e7-8d8d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/8ab76200-afbd-11e7-8d8d-5254006e85c2</a></p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/29294/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/29294/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9887" title="sanity-lfsck test_9a: FAIL: (4) Got speed 952, expected less than 144" class="issue-link" data-issue-key="LU-9887"><del>LU-9887</del></a> lfsck: calculate LFSCK speed properly<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 0f14db83ab0fe0b505e3eabb7b51619cd42e5155</p> <blockquote> <p>still seeing fails on master after the landing of <a href="https://review.whamcloud.com/28588" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28588</a> and <a href="https://review.whamcloud.com/28617:" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28617:</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/8ab76200-afbd-11e7-8d8d-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/8ab76200-afbd-11e7-8d8d-5254006e85c2</a></p></blockquote> <p>This is a different issue that is caused by calculation error. As you can, the diff is (145 - 144) / 144, it can be ignored in our VM test environment.</p> <blockquote> <p>sanity-lfsck test_9b: @@@@@@ FAIL: (10) Speed 145, expected &lt; 144</p></blockquote> <p>So we need a new ticket to track making this test more robust?</p> <p>Currently, we allow some test error range for lfsck speed. If we want to make the test more robust, then either enlarge such error range or test more large data set. But there is no absolute solution for that.</p> <p>So is the current test actually telling us anything useful? It sounds like you are saying that the failures for this test are because the failure threshold is too low. If that is the case, we should either raise the threshold to reduce these failures or else remove the test. As things stand it is failing quite often but just being assumed to be fine.</p> <p>Raising the threshold will be the most simple solution. I will push a patch soon with this ticket number.</p> <p>Fan Yong (fan.yong@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30247" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30247</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9887" title="sanity-lfsck test_9a: FAIL: (4) Got speed 952, expected less than 144" class="issue-link" data-issue-key="LU-9887"><del>LU-9887</del></a> tests: adjust lfsck speek test error range<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 644ae8cc3930cc504b7cfd24c745c0b78b6481d4</p> <p>More patch for this ticket.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/30247/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30247/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9887" title="sanity-lfsck test_9a: FAIL: (4) Got speed 952, expected less than 144" class="issue-link" data-issue-key="LU-9887"><del>LU-9887</del></a> tests: adjust lfsck speek test error range<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 0457798e95e3628d7b4f0894fdc2dd13c2dd23f6</p> Duplicate LU-9894 Related LU-6174 LU-9295 Development Rank 1|hzzimn: Rank (Obsolete) 9223372036854775807 Severity