https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-14936 Lustre <p>This issue was created by maloo for S Buisson &lt;sbuisson@ddn.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/ce3dea6f-13cd-4e3d-9615-b25380e9e3ed" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/ce3dea6f-13cd-4e3d-9615-b25380e9e3ed</a></p> <p>test_140 failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>test_140 returned 1 </pre> </div></div> <p>It seems to be a bug with sanity test_140 itself, as the output does not differ from a successful run:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanity test 140: Check reasonable stack depth (shouldn't LBUG) ==================================== 07:30:38 (1628839838) The symlink depth = 40 open symlink_self returns 40 Resetting fail_loc on all nodes...CMD: trevis-21vm1,trevis-33vm1,trevis-79vm11.trevis.whamcloud.com,trevis-79vm12 lctl set_param -n fail_loc=0 fail_val=0 2&gt;/dev/null done. 07:30:49 (1628839849) waited for trevis-21vm1 network 5s </pre> </div></div> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanity test_140 - test_140 returned 1</p> LU-14936

sanity test_140 returned 1

Bug Minor Open Unresolved WC Triage Maloo Fri, 13 Aug 2021 11:40:54 +0000 Thu, 19 Aug 2021 18:04:37 +0000 0 4 <p>I wonder if this may happen if a <tt>stack_trap</tt> registered by this subtest returns a non-zero value, since those are run after the test sub shell exits, but before the test is "finished"?</p> <p>I also saw this same failure on sanity test_398b - finished with "<tt>waited for trevis-39vm11 network 5s</tt>" then failed with "<tt>returned 1</tt>". </p> <p>sanity test 4, the same failure w/o obvious reason<br/> <a href="https://testing.whamcloud.com/test_sets/cb7fa77c-c447-4af8-8294-6846c61a7019" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/cb7fa77c-c447-4af8-8294-6846c61a7019</a> .</p> <p>Maybe this issue is related to the recent commit 67752f6d at <a href="https://review.whamcloud.com/44033" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/44033</a>, and seems it only happens to group review-ldiskfs-arm, especially on MDS1.</p> <p>The t-f change shows:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> check_node_health() { local nodes=${1:-$(comma_list $(nodes_list))} - - <span class="code-keyword">for</span> node in ${nodes<span class="code-comment">//,/ }; <span class="code-keyword">do</span> </span>- check_network <span class="code-quote">"$node"</span> 5 - <span class="code-keyword">if</span> [ $? -eq 0 ]; then - do_node $node <span class="code-quote">"$LCTL get_param catastrophe 2&gt;&amp;1"</span> | - grep -q <span class="code-quote">"catastrophe=1"</span> &amp;&amp; - error <span class="code-quote">"$node:LBUG/LASSERT detected"</span> || <span class="code-keyword">true</span> - fi - done + local health=$TMP/node_health.$$ + + do_nodes $nodes <span class="code-quote">"$LCTL get_param catastrophe 2&gt;&amp;1"</span> | tee $health | + grep <span class="code-quote">"catastrophe=1"</span> &amp;&amp; error <span class="code-quote">"LBUG/LASSERT detected"</span> + # Only check/report network health <span class="code-keyword">if</span> get_param isn't reported, since + # *clearly* the network is working <span class="code-keyword">if</span> get_param returned something. + <span class="code-keyword">if</span> (( $(grep -c catastro $health) != $(wc -w &lt;&lt;&lt; ${nodes<span class="code-comment">//,/ }) )); then </span>+ <span class="code-keyword">for</span> node in ${nodes<span class="code-comment">//,/}; <span class="code-keyword">do</span> </span>+ check_network $node 5 + done + fi + rm -f $health } check_network() { local host=$1 local max=$2 local sleep=${3:-5} [ <span class="code-quote">"$host"</span> = <span class="code-quote">"$HOSTNAME"</span> ] &amp;&amp; <span class="code-keyword">return</span> 0 <span class="code-keyword">if</span> ! wait_for_function --quiet <span class="code-quote">"ping -c 1 -w 3 $host"</span> $max $sleep; then echo <span class="code-quote">"$(date +<span class="code-quote">'%H:%M:%S (%s)'</span>) waited <span class="code-keyword">for</span> $host network ${max}s"</span> exit 1 fi } </pre> </div></div> <p>I was wondering the same about 44033, but that patch only landed 2 days ago and failures on master have been happening for at least 6 days when this ticket was opened. </p> Duplicate LU-14857 Related LU-14773 Development Rank 1|i021o7: Rank (Obsolete) 9223372036854775807 Severity