https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9602 Lustre <p><a href="https://testing.hpdd.intel.com/test_sessions/2df73ff0-8703-4bf1-9f76-2ea9f98949fb" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sessions/2df73ff0-8703-4bf1-9f76-2ea9f98949fb</a></p> <p>From suite_log:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>PASS fail_client_mds (85783s) Starting client: onyx-36vm1.onyx.hpdd.intel.com: -o user_xattr,flock onyx-36vm7:onyx-36vm3:/lustre /mnt/lustre CMD: onyx-36vm1.onyx.hpdd.intel.com mkdir -p /mnt/lustre CMD: onyx-36vm1.onyx.hpdd.intel.com mount -t lustre -o user_xattr,flock onyx-36vm7:onyx-36vm3:/lustre /mnt/lustre == recovery-random-scale test complete, duration 85822 sec =========================================== 17:10:21 (1495991421) rm: cannot remove '/mnt/lustre/d0.tar-onyx-36vm6.onyx.hpdd.intel.com/etc': Directory not empty recovery-random-scale : @@@@@@ FAIL: remove sub-test dirs failed Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:4952:error() = /usr/lib64/lustre/tests/test-framework.sh:4471:check_and_cleanup_lustre() = /usr/lib64/lustre/tests/recovery-random-scale.sh:267:main() </pre> </div></div> <p>and</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Status: PASS: rc=0 CMD: onyx-36vm5,onyx-36vm6 test -f /tmp/client-load.pid &amp;&amp; { kill -s TERM \$(cat /tmp/client-load.pid); rm -f /tmp/client-load.pid; } SKIP pairwise_fail (3867s) Starting client: onyx-36vm1.onyx.hpdd.intel.com: -o user_xattr,flock onyx-36vm7:onyx-36vm3:/lustre /mnt/lustre CMD: onyx-36vm1.onyx.hpdd.intel.com mkdir -p /mnt/lustre CMD: onyx-36vm1.onyx.hpdd.intel.com mount -t lustre -o user_xattr,flock onyx-36vm7:onyx-36vm3:/lustre /mnt/lustre == recovery-double-scale test complete, duration 3904 sec ============================================ 18:20:55 (1495995655) rm: cannot remove '/mnt/lustre/d0.tar-onyx-36vm6.onyx.hpdd.intel.com/etc/fonts/conf.d': Directory not empty recovery-double-scale : @@@@@@ FAIL: remove sub-test dirs failed Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:4952:error() = /usr/lib64/lustre/tests/test-framework.sh:4471:check_and_cleanup_lustre() = /usr/lib64/lustre/tests/recovery-double-scale.sh:309:main() </pre> </div></div> onyx, failover<br/> &nbsp;&nbsp;clients: EL7, master branch, v2.9.58, b3591<br/> &nbsp;&nbsp;servers: EL7, zfs, master branch, v2.9.58, b3591<br/> LU-9602

recovery-random-scale test_fail_client_mds: PASS but marked as FAIL

Bug Major In Progress Unresolved Alex Deiter James Casper Mon, 5 Jun 2017 21:44:57 +0000 Fri, 14 Apr 2023 23:00:50 +0000 Lustre 2.10.0 Lustre 2.11.0 Lustre 2.12.0 Lustre 2.10.4 Lustre 2.10.5 Lustre 2.13.0 Lustre 2.10.7 Lustre 2.12.1 Lustre 2.12.3 Lustre 2.14.0 Lustre 2.12.5 0 5 <p>Also seen in this config with recovery-double-scale: test pairwise_fail: SKIP but marked as FAIL</p> <p>James Nunez (james.a.nunez@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/28264" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28264</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9602" title="recovery-random-scale test_fail_client_mds: PASS but marked as FAIL" class="issue-link" data-issue-key="LU-9602">LU-9602</a> test: kill all tar and dd processes<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 51a95be4f89237a7e2dd16d8af99793b0126c5b9</p> <p>We see several of the recovery-*-scale tests fail in this way; we are not able to clean up the Lustre directory.</p> <p>For example, a recent failure in recovery-mds-scale test_failover_mds provides some new information; <a href="https://testing.hpdd.intel.com/test_sets/4929a310-fded-11e7-bd00-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/4929a310-fded-11e7-bd00-52540065bddc</a>. Looking at the suite_log, we see the test fail due to rm failing</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>rm: cannot remove '/mnt/lustre/d0.tar-onyx-41vm4.onyx.hpdd.intel.com/etc': Directory not empty recovery-mds-scale : @@@@@@ FAIL: remove sub-test dirs failed Trace dump: = /usr/lib64/lustre/tests/test-framework.sh:5335:error() = /usr/lib64/lustre/tests/test-framework.sh:4829:check_and_cleanup_lustre() </pre> </div></div> <p>Later in the log, we can see that run_tar.sh is still running:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Stopping client onyx-41vm1.onyx.hpdd.intel.com /mnt/lustre opts: Stopping client onyx-41vm4.onyx.hpdd.intel.com /mnt/lustre opts: COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME sleep 3581 root cwd DIR 1273,181606 4096 144115205289292362 /mnt/lustre/d0.tar-onyx-41vm4.onyx.hpdd.intel.com run_tar.s 13671 root cwd DIR 1273,181606 4096 144115205289292362 /mnt/lustre/d0.tar-onyx-41vm4.onyx.hpdd.intel.com /mnt/lustre is still busy, wait one second /mnt/lustre is still busy, wait one second /mnt/lustre is still busy, wait one second </pre> </div></div> <p>The run_tar, run_dd, etc. processes are started and the PID of each is stored in a file called client-load.pid in /tmp on the client node. In this case vm4 started run_tar as seen here </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Started client load: tar on onyx-41vm4 CMD: onyx-41vm4 PATH=/ […] LCTL=/usr/sbin/lctl FSNAME=lustre run_tar.sh client loads pids: CMD: onyx-41vm3,onyx-41vm4 cat /tmp/client-load.pid onyx-41vm3: 12666 onyx-41vm4: 13671 </pre> </div></div> <p>In stop_process(), we kill the PID saved in /tmp/client-load.pid</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">2416 # Stop the process which had its PID saved in a file. 2417 stop_process() { 2418 local nodes=$1 2419 local pid_file=$2 2420 2421 [ -z <span class="code-quote">"$nodes"</span> -o -z <span class="code-quote">"$pid_file"</span> ] &amp;&amp; <span class="code-keyword">return</span> 0 2422 2423 do_nodes $nodes "test -f $pid_file &amp;&amp; 2424 { kill -s TERM \\\$(cat $pid_file); rm -f $pid_file; }" || <span class="code-keyword">true</span> 2425 } </pre> </div></div> <p>We see that vm3 tries to clean up it's run_dd PID and fails. What about vm4:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>CMD: onyx-41vm3,onyx-41vm4 test -f /tmp/client-load.pid &amp;&amp; { kill -s TERM \$(cat /tmp/client-load.pid); rm -f /tmp/client-load.pid; } onyx-41vm3: sh: line 1: kill: (12666) - No such process /usr/lib64/lustre/tests/recovery-mds-scale.sh: line 104: 9503 Killed do_node $client "PATH=$PATH MOUNT=$MOUNT ERRORS_OK=$ERRORS_OK BREAK_ON_ERROR=$BREAK_ON_ERROR END_RUN_FILE=$END_RUN_FILE LOAD_PID_FILE=$LOAD_PID_FILE TESTLOG_PREFIX=$TESTLOG_PREFIX TESTNAME=$TESTNAME DBENCH_LIB=$DBENCH_LIB DBENCH_SRC=$DBENCH_SRC CLIENT_COUNT=$((CLIENTCOUNT - 1)) LFS=$LFS LCTL=$LCTL FSNAME=$FSNAME run_${load}.sh" /usr/lib64/lustre/tests/recovery-mds-scale.sh: line 104: 9697 Killed do_node $client "PATH=$PATH MOUNT=$MOUNT ERRORS_OK=$ERRORS_OK BREAK_ON_ERROR=$BREAK_ON_ERROR END_RUN_FILE=$END_RUN_FILE LOAD_PID_FILE=$LOAD_PID_FILE TESTLOG_PREFIX=$TESTLOG_PREFIX TESTNAME=$TESTNAME DBENCH_LIB=$DBENCH_LIB DBENCH_SRC=$DBENCH_SRC CLIENT_COUNT=$((CLIENTCOUNT - 1)) LFS=$LFS LCTL=$LCTL FSNAME=$FSNAME run_${load}.sh" </pre> </div></div> <p>Looking at the results at <a href="https://testing.whamcloud.com/test_sets/3bb67f2c-156c-42e3-8b23-9050d1d799a4" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/3bb67f2c-156c-42e3-8b23-9050d1d799a4</a>, the test passes, but we can see in the suite_log that not all processes were killed</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Stopping client onyx-42vm3.onyx.whamcloud.com /mnt/lustre opts: COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME run_dd.sh 16846 root cwd unknown 1273,181606 /mnt/lustre/d0.dd-onyx-42vm3.onyx.whamcloud.com dd 16909 root cwd unknown 1273,181606 /mnt/lustre/d0.dd-onyx-42vm3.onyx.whamcloud.com dd 16909 root 1w REG 1273,181606 26873016320 144115205272502274 /mnt/lustre/d0.dd-onyx-42vm3.onyx.whamcloud.com/dd-file /mnt/lustre is still busy, wait one second </pre> </div></div> <p>Looking at the run_dd_debug log on onyx-42vm3, we tried to kill the process but</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2021-05-14 00:35:09: client load was signaled to terminate ++++ ps -eo '%c %p %r' ++++ awk '/ 16845 / {print $3}' +++ local PGID= +++ kill -TERM - /usr/lib64/lustre/tests/functions.sh: line 166: kill: -: arguments must be process or job IDs +++ sleep 5 +++ kill -KILL - /usr/lib64/lustre/tests/functions.sh: line 168: kill: -: arguments must be process or job IDs ++ sleep 5 ++ kill -KILL -16804 </pre> </div></div> <p>one more with "remove sub-test dirs failed" marked as PASS<br/> <a href="https://testing.whamcloud.com/test_logs/966cb14d-a670-47ba-8141-f5f54088c23c/show_text" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_logs/966cb14d-a670-47ba-8141-f5f54088c23c/show_text</a></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> == recovery-<span class="code-object">double</span>-scale test complete, duration 10761 sec ========================================================== 07:46:23 (1637307983) rm: cannot remove <span class="code-quote">'/mnt/lustre/d0.tar-onyx-78vm3.onyx.whamcloud.com'</span>: Directory not empty rm: cannot remove <span class="code-quote">'/mnt/lustre/d0.tar-onyx-78vm7.onyx.whamcloud.com/etc'</span>: Directory not empty recovery-<span class="code-object">double</span>-scale : @@@@@@ FAIL: remove sub-test dirs failed </pre> </div></div> Related LU-15140 LU-14738 Development Rank 1|hzze87: Rank (Obsolete) 9223372036854775807 Severity