https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10543 Lustre <p>This issue was created by maloo for Bob Glossman &lt;bob.glossman@intel.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.hpdd.intel.com/test_sets/9df61cce-fe48-11e7-bd00-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.hpdd.intel.com/test_sets/9df61cce-fe48-11e7-bd00-52540065bddc</a></p> <p>This failure occurs after racer is complete, while test script is exiting.<br/> Can't see any evidence of panics or oops in logs.<br/> ConMan just disconnects from all the nodes and then all get rebooted.</p> LU-10543

racer: /mnt/lustre2 is still busy, wait one second

Bug Minor Open Unresolved WC Triage Maloo Sun, 21 Jan 2018 15:54:44 +0000 Mon, 30 Oct 2023 20:13:45 +0000 Lustre 2.11.0 Lustre 2.12.0 Lustre 2.10.4 Lustre 2.13.0 Lustre 2.12.1 Lustre 2.14.0 Lustre 2.12.4 Lustre 2.12.6 0 8 <p>Recent racer hangs:<br/> <a href="https://testing.whamcloud.com/test_sets/7f83ac30-9b18-11e8-b0aa-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/7f83ac30-9b18-11e8-b0aa-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/2ff5cab8-9ae6-11e8-8ee3-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/2ff5cab8-9ae6-11e8-8ee3-52540065bddc</a><br/> <a href="https://testing.whamcloud.com/test_sets/83e235b6-9a74-11e8-b0aa-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/83e235b6-9a74-11e8-b0aa-52540065bddc</a></p> <p>racer test 1 looks like it completes successfully, but we aren't able to umount the second Lustre mount</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>while umount /mnt/lustre2 2&gt;&amp;1 | grep -q busy; do echo /mnt/lustre2 is still busy, wait one second &amp;&amp; sleep 1; done; fi Stopping client trevis-33vm6.trevis.whamcloud.com /mnt/lustre2 opts: Stopping client trevis-33vm7.trevis.whamcloud.com /mnt/lustre2 opts: /mnt/lustre2 is still busy, wait one second /mnt/lustre2 is still busy, wait one second /mnt/lustre2 is still busy, wait one second /mnt/lustre2 is still busy, wait one second /mnt/lustre2 is still busy, wait one second </pre> </div></div> <p>It looks like there are no client process still running</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>there should be NO racer processes: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND Filesystem 1K-blocks Used Available Use% Mounted on 10.9.4.180@tcp:/lustre 13532932 202516 12463840 2% /mnt/lustre We survived /usr/lib64/lustre/tests/racer/racer.sh for 900 seconds. there should be NO racer processes: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND Filesystem 1K-blocks Used Available Use% Mounted on 10.9.4.180@tcp:/lustre 13532932 198284 12471324 2% /mnt/lustre We survived /usr/lib64/lustre/tests/racer/racer.sh for 900 seconds. pid=17959 rc=0 pid=17960 rc=0 </pre> </div></div> <p>I checked the stack traces, and <tt>unmount</tt> does not appear. That is not surprising, given that the unmount command is returning an error instead of hanging during cleanup. This implies some kind of leak in the vfsmnt or inode refcount. That is likely due to some kind of change in the dcache or inode hanfling in llite. </p> <p><del>James Nunez (jnunez@whamcloud.com) uploaded a new patch:</del> <a href="https://review.whamcloud.com/32986" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32986</a><br/> <del>Subject: <a href="https://jira.whamcloud.com/browse/LU-10543" title="racer: /mnt/lustre2 is still busy, wait one second" class="issue-link" data-issue-key="LU-10543">LU-10543</a> tests: determine racer hang</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: master</del><br/> <del>Current Patch Set: 1</del><br/> <del>Commit: c6b7177dd126812f0f4a0535b65f6b42b282f797</del></p> <p>In some cases, we see that racer completes and it is marked as PASS, but the next test suite is not able to start and timeout due to mount2 busy. </p> <p>See <a href="https://testing.whamcloud.com/test_sessions/91c9307e-4e4c-421f-a197-5e9827f00438" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sessions/91c9307e-4e4c-421f-a197-5e9827f00438</a>; racer is marked as PASS, but replay-single hangs with, looking at the suite_log</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>-----============= acceptance-small: replay-single ============----- Fri Aug 10 23:26:13 UTC 2018 Running: bash /usr/lib64/lustre/tests/replay-single.sh == replay-single test complete, duration -o sec ====================================================== 23:26:13 (1533943573) excepting tests: Stopping clients: trevis-6vm10,trevis-6vm9.trevis.whamcloud.com /mnt/lustre2 (opts:) CMD: trevis-6vm10,trevis-6vm9.trevis.whamcloud.com running=\$(grep -c /mnt/lustre2' ' /proc/mounts); if [ \$running -ne 0 ] ; then echo Stopping client \$(hostname) /mnt/lustre2 opts:; lsof /mnt/lustre2 || need_kill=no; if [ x != x -a x\$need_kill != xno ]; then pids=\$(lsof -t /mnt/lustre2 | sort -u); if [ -n \"\$pids\" ]; then kill -9 \$pids; fi fi; while umount /mnt/lustre2 2&gt;&amp;1 | grep -q busy; do echo /mnt/lustre2 is still busy, wait one second &amp;&amp; sleep 1; done; fi Stopping client trevis-6vm9.trevis.whamcloud.com /mnt/lustre2 opts: Stopping client trevis-6vm10.trevis.whamcloud.com /mnt/lustre2 opts: /mnt/lustre2 is still busy, wait one second /mnt/lustre2 is still busy, wait one second /mnt/lustre2 is still busy, wait one second /mnt/lustre2 is still busy, wait one second ... </pre> </div></div> <p>Looking at the stack traces from several recent racer timeouts (run with <tt>env=DURATION=3600</tt>) shows that there is a lingering "<tt>cp</tt>" process running on the stuck client:<br/> <a href="https://testing.whamcloud.com/test_sets/ad623a1d-dd28-4b4f-a323-f3893798c7d7" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/ad623a1d-dd28-4b4f-a323-f3893798c7d7</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 4688.804205] cp S ffff9ca9258aa0e0 0 6454 1 0x00000080 [ 4688.805787] Call Trace: [ 4688.806328] [&lt;ffffffff83585da9&gt;] schedule+0x29/0x70 [ 4688.807351] [&lt;ffffffffc0e0f867&gt;] ? vvp_io_init+0x347/0x480 [lustre] [ 4688.808579] [&lt;ffffffffc08ec15b&gt;] ? cl_io_init0.isra.13+0x8b/0x160 [obdclass] [ 4688.809924] [&lt;ffffffffc0e04fd1&gt;] ? cl_glimpse_lock+0x311/0x370 [lustre] [ 4688.811325] [&lt;ffffffffc0e0537d&gt;] ? cl_glimpse_size0+0x22d/0x260 [lustre] [ 4688.812576] [&lt;ffffffffc0dc607e&gt;] ? ll_getattr_dentry+0x53e/0x8e0 [lustre] [ 4688.813933] [&lt;ffffffff83060392&gt;] ? user_path_at_empty+0x72/0xc0 [ 4688.815083] [&lt;ffffffffc0dc643b&gt;] ? ll_getattr+0x1b/0x20 [lustre] [ 4688.816266] [&lt;ffffffff83052b49&gt;] ? vfs_getattr+0x49/0x80 [ 4688.817200] [&lt;ffffffff83052c75&gt;] ? vfs_fstatat+0x75/0xc0 [ 4688.818213] [&lt;ffffffff8305301e&gt;] ? SYSC_newstat+0x2e/0x60 [ 4688.819180] [&lt;ffffffff83592e15&gt;] ? system_call_after_swapgs+0xa2/0x13a [ 4688.820398] [&lt;ffffffff83592e09&gt;] ? system_call_after_swapgs+0x96/0x13a [ 4688.821538] [&lt;ffffffff83592e15&gt;] ? system_call_after_swapgs+0xa2/0x13a [ 4688.822755] [&lt;ffffffff83592e09&gt;] ? system_call_after_swapgs+0x96/0x13a [ 4688.823886] [&lt;ffffffff83592e15&gt;] ? system_call_after_swapgs+0xa2/0x13a [ 4688.825089] [&lt;ffffffff83592e09&gt;] ? system_call_after_swapgs+0x96/0x13a [ 4688.826220] [&lt;ffffffff83592e15&gt;] ? system_call_after_swapgs+0xa2/0x13a [ 4688.827464] [&lt;ffffffff83592e09&gt;] ? system_call_after_swapgs+0x96/0x13a [ 4688.828684] [&lt;ffffffff83592e15&gt;] ? system_call_after_swapgs+0xa2/0x13a [ 4688.830021] [&lt;ffffffff83592e09&gt;] ? system_call_after_swapgs+0x96/0x13a [ 4688.831226] [&lt;ffffffff83592e15&gt;] ? system_call_after_swapgs+0xa2/0x13a [ 4688.832570] [&lt;ffffffff830534de&gt;] ? SyS_newstat+0xe/0x10 [ 4688.833591] [&lt;ffffffff83592ed2&gt;] ? system_call_fastpath+0x25/0x2a [ 4688.834768] [&lt;ffffffff83592e15&gt;] ? system_call_after_swapgs+0xa2/0x13a : [ 4748.927655] Lustre: 6454:0:(client.c:2282:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1612251797/real 1612251797] req@ffff9ca9157cd680 x1690564340618304/t0(0) o49-&gt;lustre-MDT0000-mdc-ffff9ca93876b800@10.9.6.224@tcp:12/10 lens 472/1040 e 0 to 1 dl 1612251804 ref 2 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'cp.0' </pre> </div></div> <p>A second better-formed stack is on the other client:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 4684.466221] cp S ffffa0ea19d48000 0 16037 1 0x00000080 [ 4684.467522] Call Trace: [ 4684.467975] [&lt;ffffffff9b785da9&gt;] schedule+0x29/0x70 [ 4684.468962] [&lt;ffffffffc0bc9c85&gt;] ldlm_completion_ast+0x1e5/0x9d0 [ptlrpc] [ 4684.471257] [&lt;ffffffffc0bb85ed&gt;] ldlm_lock_match_with_skip+0x1ad/0x830 [ptlrpc] [ 4684.474583] [&lt;ffffffffc0dbbd70&gt;] osc_enqueue_base+0x100/0x530 [osc] [ 4684.477063] [&lt;ffffffffc0dc64b9&gt;] osc_lock_enqueue+0x359/0x830 [osc] [ 4684.478339] [&lt;ffffffffc09755a5&gt;] cl_lock_enqueue+0x65/0x120 [obdclass] [ 4684.479542] [&lt;ffffffffc0e21a45&gt;] lov_lock_enqueue+0x95/0x150 [lov] [ 4684.480727] [&lt;ffffffffc09755a5&gt;] cl_lock_enqueue+0x65/0x120 [obdclass] [ 4684.481887] [&lt;ffffffffc0975b37&gt;] cl_lock_request+0x67/0x1f0 [obdclass] [ 4684.483127] [&lt;ffffffffc097961b&gt;] cl_io_lock+0xfb/0x280 [obdclass] [ 4684.484222] [&lt;ffffffffc097a06d&gt;] cl_io_loop+0x8d/0x200 [obdclass] [ 4684.485523] [&lt;ffffffffc0ec563f&gt;] cl_setattr_ost+0x28f/0x3e0 [lustre] [ 4684.486645] [&lt;ffffffffc0e9da5a&gt;] ll_setattr_raw+0x102a/0x1140 [lustre] [ 4684.487880] [&lt;ffffffffc0e9dbd3&gt;] ll_setattr+0x63/0xc0 [lustre] [ 4684.488931] [&lt;ffffffff9b26cefc&gt;] notify_change+0x30c/0x4d0 [ 4684.489978] [&lt;ffffffff9b24af35&gt;] do_truncate+0x75/0xc0 [ 4684.490898] [&lt;ffffffff9b25d487&gt;] do_last+0x627/0x1340 [ 4684.493054] [&lt;ffffffff9b25e26d&gt;] path_openat+0xcd/0x5a0 [ 4684.496260] [&lt;ffffffff9b2604bd&gt;] do_filp_open+0x4d/0xb0 [ 4684.498210] [&lt;ffffffff9b24c094&gt;] do_sys_open+0x124/0x220 </pre> </div></div> <p>A different test run also showed the same stack:<br/> <a href="https://testing.whamcloud.com/test_sets/eb16868e-613f-40fb-a820-18e0e627f694" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/eb16868e-613f-40fb-a820-18e0e627f694</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[ 4022.294274] cp S ffff97d81fe99900 0 22278 1 0x00000080 [ 4022.295585] Call Trace: [ 4022.296011] [&lt;ffffffff9dd85da9&gt;] schedule+0x29/0x70 [ 4022.296868] [&lt;ffffffffc0a9dc85&gt;] ldlm_completion_ast+0x1e5/0x9d0 [ptlrpc] [ 4022.299128] [&lt;ffffffffc0a8c5ed&gt;] ldlm_lock_match_with_skip+0x1ad/0x830 [ptlrpc] [ 4022.302394] [&lt;ffffffffc0c8fd70&gt;] osc_enqueue_base+0x100/0x530 [osc] [ 4022.304799] [&lt;ffffffffc0c9a4b9&gt;] osc_lock_enqueue+0x359/0x830 [osc] [ 4022.305932] [&lt;ffffffffc08495a5&gt;] cl_lock_enqueue+0x65/0x120 [obdclass] [ 4022.307161] [&lt;ffffffffc0cf5a45&gt;] lov_lock_enqueue+0x95/0x150 [lov] [ 4022.308241] [&lt;ffffffffc08495a5&gt;] cl_lock_enqueue+0x65/0x120 [obdclass] [ 4022.309428] [&lt;ffffffffc0849b37&gt;] cl_lock_request+0x67/0x1f0 [obdclass] [ 4022.310562] [&lt;ffffffffc084d61b&gt;] cl_io_lock+0xfb/0x280 [obdclass] [ 4022.311687] [&lt;ffffffffc084e06d&gt;] cl_io_loop+0x8d/0x200 [obdclass] [ 4022.312868] [&lt;ffffffffc0d9963f&gt;] cl_setattr_ost+0x28f/0x3e0 [lustre] [ 4022.314033] [&lt;ffffffffc0d71a5a&gt;] ll_setattr_raw+0x102a/0x1140 [lustre] [ 4022.315162] [&lt;ffffffffc0d71bd3&gt;] ll_setattr+0x63/0xc0 [lustre] [ 4022.316231] [&lt;ffffffff9d86cefc&gt;] notify_change+0x30c/0x4d0 [ 4022.317174] [&lt;ffffffff9d84af35&gt;] do_truncate+0x75/0xc0 [ 4022.318122] [&lt;ffffffff9d85d487&gt;] do_last+0x627/0x1340 [ 4022.320212] [&lt;ffffffff9d85e26d&gt;] path_openat+0xcd/0x5a0 [ 4022.322229] [&lt;ffffffff9d8604bd&gt;] do_filp_open+0x4d/0xb0 [ 4022.324109] [&lt;ffffffff9d84c094&gt;] do_sys_open+0x124/0x220 [ 4022.326209] [&lt;ffffffff9d84c1ae&gt;] SyS_open+0x1e/0x20 </pre> </div></div> <p>Can easily hit this issue on the Arm cluster by running the racer test suite several times.</p> <p>With similar cp stack:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [jenkins@lustre-rec5k7yz-01 ~]$ ps -efH|grep lustre2/ jenkins   873969  873837  0 09:36 pts/1    00:00:00           grep --color=auto lustre2/ root      426463       1  0 Sep13 ?        00:00:00   cp -p /bin/sleep /mnt/lustre2/racer/4 [jenkins@lustre-rec5k7yz-01 ~]$ sudo cat /proc/426463/stack [&lt;0&gt;] __switch_to+0x7c/0xbc [&lt;0&gt;] ldlm_completion_ast+0x7a4/0xdbc [ptlrpc] [&lt;0&gt;] ldlm_lock_match_with_skip+0x1c8/0x854 [ptlrpc] [&lt;0&gt;] osc_enqueue_base+0x100/0x550 [osc] [&lt;0&gt;] osc_lock_enqueue+0x434/0xc20 [osc] [&lt;0&gt;] cl_lock_enqueue+0x98/0x1e0 [obdclass] [&lt;0&gt;] lov_lock_enqueue+0x90/0x22c [lov] [&lt;0&gt;] cl_lock_enqueue+0x98/0x1e0 [obdclass] [&lt;0&gt;] cl_lock_request+0xa0/0x270 [obdclass] [&lt;0&gt;] cl_lockset_lock+0x12c/0x240 [obdclass] [&lt;0&gt;] cl_io_lock+0xd0/0x204 [obdclass] [&lt;0&gt;] cl_io_loop+0xb8/0x280 [obdclass] [&lt;0&gt;] cl_setattr_ost+0x290/0x364 [lustre] [&lt;0&gt;] ll_setattr_raw+0x908/0x1160 [lustre] [&lt;0&gt;] ll_setattr+0x8c/0x130 [lustre] [&lt;0&gt;] notify_change+0x25c/0x43c [&lt;0&gt;] do_truncate+0x84/0xe0 [&lt;0&gt;] handle_truncate+0xd4/0x124 [&lt;0&gt;] do_open+0xc4/0x30c [&lt;0&gt;] path_openat+0x10c/0x1d0 [&lt;0&gt;] do_filp_open+0x84/0x134 [&lt;0&gt;] do_sys_openat2+0x208/0x2f0 [&lt;0&gt;] __arm64_sys_openat+0x6c/0xb4 [&lt;0&gt;] invoke_syscall+0x50/0x11c [&lt;0&gt;] el0_svc_common.constprop.0+0x158/0x164 [&lt;0&gt;] do_el0_svc+0x2c/0x9c [&lt;0&gt;] el0_svc+0x20/0x30 [&lt;0&gt;] el0_sync_handler+0xb0/0xb4 [&lt;0&gt;] el0_sync+0x160/0x180 [jenkins@lustre-rec5k7yz-01 ~]$ </pre> </div></div> <p>And the related test script should be:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> lustre-release$ cat lustre/tests/racer/file_exec.sh #!/bin/bash trap <span class="code-quote">'kill $(jobs -p)'</span> EXIT org_LANG=$LANG export LANG=C DIR=$1 MAX=$2 PROG=/bin/sleep <span class="code-keyword">while</span> /bin/<span class="code-keyword">true</span> ; <span class="code-keyword">do</span>         file=$((RANDOM % MAX))         cp -p $PROG $DIR/$file &gt; /dev/<span class="code-keyword">null</span> 2&gt;&amp;1         $DIR/$file 0.$((RANDOM % 5 + 1)) 2&gt; /dev/<span class="code-keyword">null</span>         sleep $((RANDOM % 3)) done 2&gt;&amp;1 | egrep -v <span class="code-quote">"Segmentation fault|Bus error"</span> export LANG=$org_LANG </pre> </div></div> <p> </p> <p>It seems that the problem is here:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> cat lustre/tests/racer/racer.sh ... 15 RACER_PROGS=${RACER_PROGS:-"file_create dir_create file_rm file_rename \ 16 file_link file_symlink file_list file_concat file_exec file_chown \ 17 file_chmod file_mknod file_truncate file_delxattr file_getxattr \ 18 file_setxattr"} ... 37 racer_cleanup() 38 { 39         echo <span class="code-quote">"racer cleanup"</span> 40         <span class="code-keyword">for</span> P in $RACER_PROGS; <span class="code-keyword">do</span> 41                 killall -q $P.sh 42         done ...</pre> </div></div> <p>It should also kill all the child processes of the sh scripts. Because if you force to kill all the file test sh scripts, you can't expect their child processes in a normal state, these child processes should be killed/cleaned up as well, because they could be in an abnormal state.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=xinliang" class="user-hover" rel="xinliang">xinliang</a> please try <a href="https://review.whamcloud.com/46384" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/46384</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-15248" title="Improve racer cleanup" class="issue-link" data-issue-key="LU-15248">LU-15248</a> tests: kill all fs users at the end of racer</tt>" to see if this fixes the problem for you. </p> <p>Hi <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="user-hover" rel="adilger">adilger</a>, with patch 46384 it still gets stuck. These cp processes seem can't be killed by fuser cmd.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> stuck logs ... /mnt/lustre2 is still busy, wait one second /mnt/lustre2 is still busy, wait one second /mnt/lustre2 is still busy, wait one second /mnt/lustre2 is still busy, wait one second /mnt/lustre2 is still busy, wait one second /mnt/lustre2 is still busy, wait one second /mnt/lustre2 is still busy, wait one second /mnt/lustre2 is still busy, wait one second ... [root@lustre-pdbirbyf-02 ~]# ps -efH|grep <span class="code-quote">"cp -p"</span> root     3362672 3360694  0 08:01 pts/0    00:00:00           grep --color=auto cp -p root     2975599       1  0 05:54 ?        00:00:00   cp -p /bin/sleep /mnt/lustre/racer/1 root     2982999       1  0 05:54 ?        00:00:00   cp -p /bin/sleep /mnt/lustre/racer/1 root     1553902       1  0 06:37 ?        00:00:00   cp -p /bin/sleep /mnt/lustre2/racer3/11 [root@lustre-pdbirbyf-02 ~]# fuser -k -m  /mnt/lustre2/racer3/ [root@lustre-pdbirbyf-02 ~]# ps -efH|grep <span class="code-quote">"cp -p"</span> root     3362795 3360694  0 08:01 pts/0    00:00:00           grep --color=auto cp -p root     2975599       1  0 05:54 ?        00:00:00   cp -p /bin/sleep /mnt/lustre/racer/1 root     2982999       1  0 05:54 ?        00:00:00   cp -p /bin/sleep /mnt/lustre/racer/1 root     1553902       1  0 06:37 ?        00:00:00   cp -p /bin/sleep /mnt/lustre2/racer3/11 </pre> </div></div> <p> </p> <p>And some cp processes in D(disk sleep) state are unkillable:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [root@lustre-pdbirbyf-02 ~]# kill -9 1553902                                                                                                                      [root@lustre-pdbirbyf-02 ~]# ps --ppid 1 -fH|grep cp                                                                                                              root     1553902       1  0 06:37 ?        00:00:00 cp -p /bin/sleep /mnt/lustre2/racer3/11                                                                       root     2982999       1  0 05:54 ?        00:00:00 cp -p /bin/sleep /mnt/lustre/racer/1                                                                          root     2975599       1  0 05:54 ?        00:00:00 cp -p /bin/sleep /mnt/lustre/racer/1 [root@lustre-pdbirbyf-02 ~]# killall -v cp                                                                                                                        Killed cp(1553902) with signal 15                                                                                                                                 Killed cp(2975599) with signal 15                                                                                                                                 Killed cp(2982999) with signal 15                                                                                                                                 [root@lustre-pdbirbyf-02 ~]# ps --ppid 1 -fH|grep cp                                        root     1553902       1  0 06:37 ?        00:00:00 cp -p /bin/sleep /mnt/lustre2/racer3/11</pre> </div></div> <p>There is a second patch <a href="https://review.whamcloud.com/45605" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/45605</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-15248" title="Improve racer cleanup" class="issue-link" data-issue-key="LU-15248">LU-15248</a> tests: improve racer cleanup</tt>" for that same ticket. However, it isn't clear to me if that patch is just hiding the test failure or if it could potentially be close to a real fix for the problem? </p> <p>It might be enough to add the hunks from that patch in <tt>obd_get_request_slot()</tt>:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (rc == -EINTR) rc = -ERESTARTSYS; </pre> </div></div> <p>and then add a check in {{ptlrpc_interrupted_set()} to see if the request is a lock enqueue and don't interrupt it in this case. Something like the following (which I haven't tested at all and is just a guess at might work):</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> list_for_each_entry(req, &amp;set-&gt;set_requests, rq_set_chain) { <span class="code-keyword">if</span> (req-&gt;rq_intr) <span class="code-keyword">continue</span>; <span class="code-keyword">if</span> (req-&gt;rq_phase != RQ_PHASE_RPC &amp;&amp; req-&gt;rq_phase != RQ_PHASE_UNREG_RPC &amp;&amp; !req-&gt;rq_allow_intr) <span class="code-keyword">continue</span>; <span class="code-keyword">if</span> (req-&gt;rq_reqmsg &amp;&amp; lustre_msg_get_opc(req-&gt;rq_reqmsg) == LDLM_ENQUEUE &amp;&amp; !__fatal_signal_pending(current)) <span class="code-keyword">continue</span>; </pre> </div></div> <p>This would have the drawback that <tt>LDLM_ENQUEUE</tt> RPCs could not normally be interrupted, but at least the threads wouldn't be stuck permanently...</p> <p>The alternative might be a check in <tt>ldlm_completion_ast()</tt> if the task can be interrupted? It <em>looks</em> like this function is already using <tt>l_wait_event_abortable()</tt>, but there have been a number of changes in this area and it may be that the interrupt handling is not working properly (or it is interrupted but restarts immediately, the lustre client debug logs should show one way or the other.</p> <p>Thanks for the explanation, Andreas. I'm trying to understand the problem. Will try patch 45605 later.</p> <p>Hmm, it looks like the problem is a little complicated. The racer test suite runs racing test scripts for a DURATION of time and then kills all the test scripts.</p> <p>It seems Lustre threads should be killed gracefully by a signal like SIGTERM, which doesn't yet, right?  Lustre should handle such a signal to exit properly.</p> Related LU-11219 LU-15248 LU-7073 LU-10904 Development Rank 1|hzzrg7: Rank (Obsolete) 9223372036854775807 Severity