https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-15089 Lustre <p>This issue was created by maloo for Elena &lt;elena.gryaznova@hpe.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/9d97b8d9-9da1-4f76-a595-f827627ca5b9" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/9d97b8d9-9da1-4f76-a595-f827627ca5b9</a></p> <p>test_230 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>expected number of tcp connections 18 </pre> </div></div> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity-lnet test_230 - expected number of tcp connections 18</p> LU-15089

sanity-lnet test_230: expected number of tcp connections 18

Bug Critical Resolved Fixed Serguei Smirnov Maloo Tue, 12 Oct 2021 16:21:25 +0000 Wed, 3 Nov 2021 03:30:15 +0000 Wed, 3 Nov 2021 03:30:15 +0000 Lustre 2.15.0 Lustre 2.15.0 0 5 <p>Serguei, this looks like it is related to the <tt>conns_per_peer</tt> patch?</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=jamesanunez" class="user-hover" rel="jamesanunez">jamesanunez</a>, something is broken with <tt>sanity-lnet</tt>. This subtest is continually failing:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>test_230 Error: 'expected number of tcp connections 18' Failure Rate: 35.00% of most recent 100 runs, 0 skipped (all branches) </pre> </div></div> <p>(70/207 in the past 3 days) but the <tt>sanity-lnet</tt> test run is always being marked as "PASS":<br/> <a href="https://testing.whamcloud.com/test_sets/64366b2e-89d8-4fea-a98d-cd5a8af7ef21" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/64366b2e-89d8-4fea-a98d-cd5a8af7ef21</a></p> <p>I suspect there is something missing at the end of <tt>sanity-lnet.sh</tt> that is supposed to check for the failure status?</p> <p>The test failure was clearly introduced by patch <a href="https://review.whamcloud.com/44417" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44417</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-12815" title="Create multiple TCP sockets per SockLND" class="issue-link" data-issue-key="LU-12815"><del>LU-12815</del></a> socklnd: set conns_per_peer based on link speed</tt>" because it was first hit when testing that patch, then immediately started failing regularly on master on 2021-08-17 after it landed.</p> <p>"Serguei Smirnov &lt;ssmirnov@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/45331" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45331</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15089" title="sanity-lnet test_230: expected number of tcp connections 18" class="issue-link" data-issue-key="LU-15089"><del>LU-15089</del></a> tests: allow enough time to create tcp connections<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7a432a1048bb45b0265a3308520c0ff5b7671287</p> <p>It isn't obvious why <tt>sanity-lnet.sh</tt> is not reporting a failure when <tt>test_230()</tt> is failing. The script has mostly the same bits at the end as <tt>sanity.sh</tt>:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>complete $SECONDS cleanup_testsuite exit_status </pre> </div></div> <p>and the test logs clearly show that <tt>complete()</tt> is finding the <tt>FAIL</tt> in <tt>$TESTSUITELOG</tt>, and <tt>cleanup_testsuite()</tt> is run as seen by "<tt>Cleaning up LNet</tt>":</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity-lnet test complete, duration 824 sec =========== 15:31:11 (1634830271) sanity-lnet: FAIL: test_230 expected number of tcp connections 26 Cleaning up LNet </pre> </div></div> <p>It then proceeds to format and mount a new filesystem (<tt>setupall()</tt>), and the last line in the log is from <tt>init_param_vars()</tt>, and then <tt>exit_status()</tt> returns "0" (<tt>PASS</tt>) instead of finding any error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: trevis-5vm9 /usr/sbin/lctl set_param lod.*.mdt_hash=crush lod.lustre-MDT0000-mdtlov.mdt_hash=crush CMD: trevis-5vm9 /usr/sbin/lctl set_param -P *.*.lbug_on_grant_miscount=1 </pre> </div></div> <p>The only thing that I can think of is that <tt>$TESTSUITELOG</tt> is somehow unset/removed/truncated and the <tt>FAIL</tt> line is missed? However, I only see <tt>$TESTSUITELOG</tt> being set/removed in <tt>init_test_env()</tt> and that is only called at the start of the script.</p> <p>It might be worthwhile to print out a line in <tt>exit_status()</tt> to help debug this:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> exit_status () { local status=0 local logs=<span class="code-quote">"$TESTSUITELOG $1"</span> + echo <span class="code-quote">"== checking <span class="code-keyword">for</span> failures in logs: <span class="code-quote">'$logs'</span> =="</span> <span class="code-keyword">for</span> log in $logs; <span class="code-keyword">do</span> <span class="code-keyword">if</span> [ -f <span class="code-quote">"$log"</span> ]; then grep -qw FAIL $log &amp;&amp; status=1 fi done + echo <span class="code-quote">"== exiting with status=$status =="</span> exit $status } </pre> </div></div> <p>I think we are deleting $TESTSUITELOG in sanity-lnet.sh:cleanup_testsuite():</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>cleanup_testsuite() { trap "" EXIT # Cleanup any tmp files created by the sub tests rm -f $TMP/sanity-lnet* cleanup_netns cleanup_lnet if $restore_mounts; then setupall || error "Failed to setup Lustre after test execution" elif $restore_modules; then load_modules || error "Couldn't load modules after test execution" fi return 0 } </pre> </div></div> <p>That <tt>rm &#45;f $TMP/sanity-lnet*</tt> should probably be <tt>rm &#45;f $TMP/sanity-lnet-*.yaml</tt></p> <p>Opened <a href="https://jira.whamcloud.com/browse/LU-15150" class="external-link" rel="nofollow">https://jira.whamcloud.com/browse/LU-15150</a> for ^</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/45331/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45331/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-15089" title="sanity-lnet test_230: expected number of tcp connections 18" class="issue-link" data-issue-key="LU-15089"><del>LU-15089</del></a> tests: allow enough time to create tcp connections<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 5c766b005bf3e0bca0efa9d87ccf230e7cba97cc</p> <p>Landed for 2.15</p> Related LU-15150 LU-15152 Development Rank 1|i026yv: Rank (Obsolete) 9223372036854775807 Severity