https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-6206 Lustre <p>This issue was created by maloo for Bob Glossman &lt;bob.glossman@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/29448bbe-ac56-11e4-b832-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/29448bbe-ac56-11e4-b832-5254006e85c2</a>.</p> <p>The sub-test test_53a failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$'Assertion 27 failed: (($tmax2 &lt; $tmin)) (expanded: ((4 &lt; 3))) Insane OST thread counts' </pre> </div></div> <p>Suspect this may be due to the very recent landing of <a href="https://jira.whamcloud.com/browse/LU-1214" title="PTLRPC related modules cleanup" class="issue-link" data-issue-key="LU-1214"><del>LU-1214</del></a> ptlrpc: start minimum service threads.<br/> It made changes in the close neighborhood of the failed tests.</p> <p>Info required for matching: conf-sanity 53a</p> LU-6206

conf-sanity test_53a: Insane OST thread counts

Bug Minor Resolved Incomplete Andreas Dilger Maloo Wed, 4 Feb 2015 13:54:36 +0000 Fri, 3 Jun 2016 18:09:19 +0000 Fri, 3 Jun 2016 18:09:19 +0000 0 7 <p>more, all reported in the last day:<br/> <a href="https://testing.hpdd.intel.com/test_sets/c3cdcb1c-ac30-11e4-b832-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/c3cdcb1c-ac30-11e4-b832-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/e5314afc-ac3c-11e4-bc6f-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/e5314afc-ac3c-11e4-bc6f-5254006e85c2</a><br/> <a href="https://testing.hpdd.intel.com/test_sets/b1220374-ac3b-11e4-b832-5254006e85c2" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/b1220374-ac3b-11e4-b832-5254006e85c2</a></p> <p>Patch landed yesterday in <a href="https://jira.whamcloud.com/browse/LU-1214" title="PTLRPC related modules cleanup" class="issue-link" data-issue-key="LU-1214"><del>LU-1214</del></a>: <a href="http://review.whamcloud.com/#/c/2876/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/2876/</a></p> <p>looking at the event history of Patch Set 17 in <a href="http://review.whamcloud.com/#/c/2876" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/2876</a> I see FAILs in conf-sanity. don't think it should have gotten +maloo. suspect some TEI fault there. shouldn't have landed.</p> <p>Looking into why it fails now.</p> <p>I suspect this line in conf-sanity.sh:</p> <p>lassert 27 "$msg" '(($tmax2 &lt; $tmin))' || return $?</p> <p>It looks entirely logic reversed to me. think it should be:</p> <p>lassert 27 "$msg" '(($tmin &lt; $tmax2))' || return $?</p> <p>Of course there could be additional flaws introduced by the bad commit covered up by always hitting this one first.</p> <p>Yep, just confirmed it is the conf-sanity test changes that break things. Let see if your above change is all that is needed. BTW what is the string I should add to the patch to only run this test?</p> <p>I think you are looking for something like:</p> <p>Test-Parameters: fortestonly testlist=conf-sanity,conf-sanity,conf-sanity envdefintions=ONLY=53a</p> <p>in the commit header.<br/> see <a href="https://wiki.hpdd.intel.com/display/PUB/Changing+Test+Parameters+with+Gerrit+Commit+Messages" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.hpdd.intel.com/display/PUB/Changing+Test+Parameters+with+Gerrit+Commit+Messages</a></p> <p>When ready to land probably need to edit the commit header &amp; remove Test-parameters.</p> <p>We have reverted <a href="http://review.whamcloud.com/#/c/2876" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/2876</a></p> <p>We've reverted the original patch from master. </p> <p>Patches that are currently based on a tree with that patch in it should rebase to after the reverted patch. We are also looking to disable these tests temporarily so that any patches that haven't hit this failure yet can pass. </p> <p>Yes Bob it was that one line in the test script that was breaking everything. I will push a new version of <a href="https://jira.whamcloud.com/browse/LU-1214" title="PTLRPC related modules cleanup" class="issue-link" data-issue-key="LU-1214"><del>LU-1214</del></a> with that fix. The lesson to learn here is never trust Maloo claiming to pass but to always look at the logs.</p> <p>Thanks James.</p> <p>PS: we are also investigating how/why Maloo marked this test failure with Verified +1 when it clearly failed the review-dne-part-1 test.</p> <p>We know that review-zfs is currently optional so the presence of a failure in that test wouldn't itself cause Maloo to mark the overall result -1, but the other failure should have. It should be noted that (excluding the hit from this issue) we have resolved a major blocker for ZFS testing (<a href="https://jira.whamcloud.com/browse/LU-5242" title="Test hang sanity test_132, test_133: umount ost" class="issue-link" data-issue-key="LU-5242"><del>LU-5242</del></a>) and are expecting to change review-zfs test results to be enforced in the near future now that we expect it to pass regularly.</p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/13823" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13823</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6206" title="conf-sanity test_53a: Insane OST thread counts" class="issue-link" data-issue-key="LU-6206"><del>LU-6206</del></a> ptlrpc: start minimum service threads<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 34ec2c93de940ecca8dcf2aa6c760cf8c1b133cc</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/13823/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13823/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-6206" title="conf-sanity test_53a: Insane OST thread counts" class="issue-link" data-issue-key="LU-6206"><del>LU-6206</del></a> ptlrpc: start minimum service threads<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: cafa669a43062c5097d40803b9ba14e66edbae25</p> <p>Andreas Dilger (andreas.dilger@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/14682" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14682</a><br/> Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-6206" title="conf-sanity test_53a: Insane OST thread counts" class="issue-link" data-issue-key="LU-6206"><del>LU-6206</del></a> ptlrpc: start minimum service threads"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 241fd8ca40271665c399d79ded71d0a8d28a247a</p> <p>Andreas Dilger (andreas.dilger@intel.com) merged in patch <a href="http://review.whamcloud.com/14682/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/14682/</a><br/> Subject: Revert "<a href="https://jira.whamcloud.com/browse/LU-6206" title="conf-sanity test_53a: Insane OST thread counts" class="issue-link" data-issue-key="LU-6206"><del>LU-6206</del></a> ptlrpc: start minimum service threads"<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e15535ff63bf90b93fa33393ed5d7f7f85813895</p> <p>This test is no longer failing. Will leave cleanups to some future date.</p> Related LU-1214 LU-6563 LU-1214 Development Rank 1|hzx5mf: Rank (Obsolete) 17360 Severity