https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4520 Lustre <p>When executing the following simple script, the in-kernel client fails with error "Text file busy", while the out-of-kernel 2.4.2 client compiled against 2.6.32 vanilla works.</p> <p>-----------<br/> #!/bin/bash</p> <p>rm -f ./test.sh<br/> touch ./test.sh &amp;&amp; chmod a+x ./test.sh<br/> echo "echo foo" &gt; ./test.sh<br/> ./test.sh<br/> echo "echo foo" &gt; ./test.sh<br/> ------------</p> <p>The following script works in all cases:</p> <p>-----------<br/> #!/bin/bash</p> <p>rm -f ./test.sh<br/> touch ./test.sh &amp;&amp; chmod a+x ./test.sh<br/> echo "echo foo" &gt; ./test.sh<br/> bash ./test.sh<br/> echo "echo foo" &gt; ./test.sh<br/> ------------</p> - Client mainline kernel 3.12.5 with patches (mentioned in <a href="https://jira.whamcloud.com/browse/LU-4400" title="Another LBUG with NFS reexport mainline 3.12 client" class="issue-link" data-issue-key="LU-4400"><strike>LU-4400</strike></a>) / lustre-utils 2.4.2 <br/> - Servers lustre 2.4.2/ZFS OSDs and 2.4.2/ldiskfs OSDs<br/> - ko2iblnd an ksocklnd LU-4520

Text file busy error -- mainline 3.12 client

Bug Major Resolved Fixed WC Triage Roland Fehrenbacher Tue, 21 Jan 2014 18:09:37 +0000 Sun, 22 Sep 2019 16:33:40 +0000 Sun, 22 Sep 2019 16:33:40 +0000 Lustre 2.4.2 1 8 <p>Update: The issue is also present in current 3.14 kernel.org git</p> <p>Hello,</p> <p>I've been running tcpdump on that issue (see attached files). Shortly put:</p> <ul class="alternate" type="square"> <li>erroneous (ETXTBSY) case shows two consecutive read of the 'test.sh' file, one "Protected read" followed by a "Concurrent read"</li> <li>while the working case shows only a single "Concurrent read"</li> </ul> <p>Could that be the reason why the MDT gets confused ?</p> <p>(hope it helps)</p> <p>Best,</p> <p>Cédric</p> <p>Might be interesting to geta tcpdump of the working 2.6.32/2.4.2 client.</p> <p>Hello,</p> <p>Attached the packets captured from the working 2.6.32/2.4.2 client.</p> <p>As you can see, the read pattern corresponding to the "./test.sh" is quite different, with consecutive "Concurrent read" separated by "MDS_CLOSE".</p> <p>(hope it helps),</p> <p>Cédric</p> <p>Ha, yes, maybe I should have said that attached packets captures corresponds only to the "./test.sh; echo 'echo foo' &gt; ./test.sh" part of the test case.</p> <p>Hello,</p> <p>I've attached the 'strace' and corresponding 'lctl dk ...' for the failing "./test.sh; echo 'echo foo' &gt; ./test.sh" part of the test case.</p> <p>Can someone make sense of it (iow. why we bump into that "(file.c:436:ll_intent_file_open()) lock enqueue: err: -26" error) ?</p> <p>Thanks in advance,</p> <p>Cédric</p> <p>Hello John,</p> <p>Thank you for for links.<br/> I have backported your patch from <a href="https://jira.whamcloud.com/browse/LU-4429" title="clients leaking open handles/bad lock matching in ll_md_blocking_ast" class="issue-link" data-issue-key="LU-4429"><del>LU-4429</del></a> to the mainline kernel (see attachment).<br/> It helps, but a ETXTBSY is still spawned (at a different step), though it can now be recovered without 'rm'-ing the file:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ LUSTRE_DIR=/idiap/temp/cdufour/bug9615-LU4520 $ rm -f "${LUSTRE_DIR}"/test.sh $ touch "${LUSTRE_DIR}"/test.sh $ chmod a+x "${LUSTRE_DIR}"/test.sh $ echo 'echo foo' &gt; "${LUSTRE_DIR}"/test.sh $ "${LUSTRE_DIR}"/test.sh foo $ echo 'echo foo' &gt; "${LUSTRE_DIR}"/test.sh $ "${LUSTRE_DIR}"/test.sh bash: /idiap/temp/cdufour/bug9615-LU4520/test.sh: Text file busy $ "${LUSTRE_DIR}"/test.sh foo </pre> </div></div> <p>Is server-side patch mentioned in <a href="https://jira.whamcloud.com/browse/LU-4152" title=" layout locks can cause deadlock" class="issue-link" data-issue-key="LU-4152"><del>LU-4152</del></a> also required to entirely fix this issue ?</p> <p>@Roland: I checked GIT 2.4.2: for-mentioned patch is not applied</p> <p>Best,</p> <p>Cédric</p> <p>ADDENDUM: if two write operations are performed before the script execution, no more ETXTBSY:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ touch "${LUSTRE_DIR}"/test.sh $ "${LUSTRE_DIR}"/test.sh bash: /idiap/temp/cdufour/bug9615-LU4520/test.sh: Text file busy $ "${LUSTRE_DIR}"/test.sh foo $ touch "${LUSTRE_DIR}"/test.sh $ touch "${LUSTRE_DIR}"/test.sh $ "${LUSTRE_DIR}"/test.sh foo $ "${LUSTRE_DIR}"/test.sh foo </pre> </div></div> <p>Hello Again,</p> <p>I've picked up test case from <a href="https://jira.whamcloud.com/browse/LU-4398" title="mdt_object_open_lock() may not flush conflicting handles" class="issue-link" data-issue-key="LU-4398"><del>LU-4398</del></a> and 'lctl dk'-ed it (<b>without</b> patch from <a href="https://jira.whamcloud.com/browse/LU-4429" title="clients leaking open handles/bad lock matching in ll_md_blocking_ast" class="issue-link" data-issue-key="LU-4429"><del>LU-4429</del></a>).</p> <p>Patch from <a href="https://jira.whamcloud.com/browse/LU-4429" title="clients leaking open handles/bad lock matching in ll_md_blocking_ast" class="issue-link" data-issue-key="LU-4429"><del>LU-4429</del></a> does not help in any way (it just allows to get rid of the additional 'rm' in the test case).</p> <p>I'm no LDLM guru but I'm surprised that the 'touch' steps leaves a CR lock behind, which is "picked up" by the 'cp' step afterwards. Shouldn't all locks concerning a given file be cleared after a process is done with it (e.g. 'touch') ?</p> <p>Cédric</p> <p>Hello Cédric,</p> <p>You might try the patch from <a href="https://jira.whamcloud.com/browse/LU-4398" title="mdt_object_open_lock() may not flush conflicting handles" class="issue-link" data-issue-key="LU-4398"><del>LU-4398</del></a>, see <a href="http://review.whamcloud.com/#/c/9063/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9063/</a>. If you have the time then please do report any findings.</p> <p>LDLM locks are intended to be cached so the fact that these locks are still held after close does not indicate an error. Can you print the CR lock left behind and describe the 'cp' step.</p> <p>Thanks, John</p> <p>Hi John,</p> <p>your patch is for the MDT. Is it plausible that the problem originates from there, given the fact<br/> that the stock 2.4.2 client works flawlessly in this regard (see the first message in this thread)?</p> <p>Roland</p> <p>Indeed. There are some changes to lookup (with the introduction of atomic_open) between 2.6.32 and 3.12 which may account for the difference. But I have not checked that this is in fact the case here.</p> <p>(to answer John's question above)</p> <p>'error' test case (the 'success' test case is identical, save for the 'touch' step which is ommitted)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>rm -f "${LUSTRE_DIR}"/echo rm -f "${LUSTRE_DIR}"/echo # Needed to make sure everything is cleaned-up rm -f "${LUSTRE_DIR}"/echo # Needed to make really sure everything is cleaned-up touch "${LUSTRE_DIR}"/echo # 'touch' step cp -p /bin/echo "${LUSTRE_DIR}"/echo # 'cp' step "${LUSTRE_DIR}"/echo 'Hello World!' # 'echo' step -&gt; ETXTBSY </pre> </div></div> <p>As for the "left-behind" CR lock, your explanation about caching makes sense; it is picked up during the 'cp' step:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(ldlm_resource.c:1406:ldlm_resource_dump()) ### ### ns: lustre-2-MDT0000-mdc-ffff880138d22800 lock: ffff880034f11e00/0x251fdce27c789bc6 lrc: 1/0,0 mode: CR/CR res: [0x200000e36:0x4:0x0].0 bits 0x9 rrc: 2 type: IBT flags: 0x20000000000 nid: local remote: 0x3ce47269da805bef expref: -99 pid: 14005 timeout: 0 lvb_type: 0 </pre> </div></div> <p>and - if I get correctly - "re"-used for futher CR locking.</p> <p>Now, the other notorious difference between the 'error' and the 'success' test case is regarding the CR lock requested for the initial 'getattr' operation in the 'cp' step;</p> <p>In the 'error' case:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(lmv_intent.c:304:lmv_intent_lock()) INTENT LOCK 'getattr' for 'echo' on [0x200000d85:0xf:0x0] (lmv_intent.c:263:lmv_intent_lookup()) LOOKUP_INTENT with fid1=[0x200000d85:0xf:0x0], fid2=[0x0:0x0:0x0], name='echo' -&gt; mds #0 (ldlm_lock.c:758:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-2-MDT0000-mdc-ffff880138d22800 lock: ffff88018cf1cc00/0x251fdce27c789bcd lrc: 3/1,0 mode: --/CR res: [0x200000d85:0xf:0x0].0 bits 0x0 rrc: 2 type: IBT flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 14007 timeout: 0 lvb_type: 0 (ldlm_request.c:898:ldlm_cli_enqueue()) ### client-side enqueue START, flags 1000 (ldlm_request.c:960:ldlm_cli_enqueue()) ### sending request ns: lustre-2-MDT0000-mdc-ffff880138d22800 lock: ffff88018cf1cc00/0x251fdce27c789bcd lrc: 3/1,0 mode: --/CR res: [0x200000d85:0xf:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 14007 timeout: 0 lvb_type: 0 (ldlm_request.c:606:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-2-MDT0000-mdc-ffff880138d22800 lock: ffff88018cf1cc00/0x251fdce27c789bcd lrc: 4/1,0 mode: --/CR res: [0x200000d85:0xf:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x0 nid: local remote: 0x3ce47269da805cdd expref: -99 pid: 14007 timeout: 0 lvb_type: 0 </pre> </div></div> <p>While in the 'success' case:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>(lmv_intent.c:304:lmv_intent_lock()) INTENT LOCK 'getattr' for 'echo' on [0x200000d85:0xf:0x0] (lmv_intent.c:263:lmv_intent_lookup()) LOOKUP_INTENT with fid1=[0x200000d85:0xf:0x0], fid2=[0x0:0x0:0x0], name='echo' -&gt; mds #0 (ldlm_lock.c:758:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-2-MDT0000-mdc-ffff880138d22800 lock: ffff880034f11200/0x251fdce27c789c21 lrc: 3/1,0 mode: --/CR res: [0x200000d85:0xf:0x0].0 bits 0x0 rrc: 2 type: IBT flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 14015 timeout: 0 lvb_type: 0 (ldlm_request.c:898:ldlm_cli_enqueue()) ### client-side enqueue START, flags 1000 (ldlm_request.c:960:ldlm_cli_enqueue()) ### sending request ns: lustre-2-MDT0000-mdc-ffff880138d22800 lock: ffff880034f11200/0x251fdce27c789c21 lrc: 3/1,0 mode: --/CR res: [0x200000d85:0xf:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 14015 timeout: 0 lvb_type: 0 (ldlm_request.c:535:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-2-MDT0000-mdc-ffff880138d22800 lock: ffff880034f11200/0x251fdce27c789c21 lrc: 4/1,0 mode: --/CR res: [0x200000d85:0xf:0x0].0 bits 0x2 rrc: 2 type: IBT flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 14015 timeout: 0 lvb_type: 0 </pre> </div></div> <p>(this leads to no ETXTBSY; but is the "ABORTED" lock attempt normal?)</p> <p>NOTE: the attached <a href="https://jira.whamcloud.com/browse/LU-4398" title="mdt_object_open_lock() may not flush conflicting handles" class="issue-link" data-issue-key="LU-4398"><del>LU-4398</del></a>.tar.bz2 file contains the details of the test cases and the corresponding 'lctl dk' outputs (for each step)</p> <p>If Roland is to apply the <a href="https://jira.whamcloud.com/browse/LU-4398" title="mdt_object_open_lock() may not flush conflicting handles" class="issue-link" data-issue-key="LU-4398"><del>LU-4398</del></a> patch to the server, must it be done jointly with <a href="https://jira.whamcloud.com/browse/LU-4429" title="clients leaking open handles/bad lock matching in ll_md_blocking_ast" class="issue-link" data-issue-key="LU-4429"><del>LU-4429</del></a> patch on the client ?</p> <p>What about the patch for <a href="https://jira.whamcloud.com/browse/LU-4430" title="mdt_mfd_open() tests for FMODE_EXEC in error path" class="issue-link" data-issue-key="LU-4430"><del>LU-4430</del></a> "mdt: check for MDS_FMODE_EXEC in mdt_mfd_open()" (<a href="http://review.whamcloud.com/8719)?" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/8719)?</a> It could be the error handling of the kernel trying to open a file for execute is leaking the reference on the file?</p> <p>Hi John,</p> <p>your patch (<a href="https://jira.whamcloud.com/browse/LU-4398" title="mdt_object_open_lock() may not flush conflicting handles" class="issue-link" data-issue-key="LU-4398"><del>LU-4398</del></a>, <a href="http://review.whamcloud.com/#/c/9063/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#/c/9063/</a>) applied to 2.5.1 indeed fixes the issue. The problem still exists in 2.5.1 without the patch. Great job.</p> <p>Roland</p> <p>I should have added, that it is indeed also necessary to include <a href="https://jira.whamcloud.com/browse/LU-4429" title="clients leaking open handles/bad lock matching in ll_md_blocking_ast" class="issue-link" data-issue-key="LU-4429"><del>LU-4429</del></a> on the client.</p> <p>Roland</p> <p>Wasn't the <a href="https://jira.whamcloud.com/browse/LU-4429" title="clients leaking open handles/bad lock matching in ll_md_blocking_ast" class="issue-link" data-issue-key="LU-4429"><del>LU-4429</del></a> fix already in 2.5.1?</p> <p>Peter</p> <p>Peter,</p> <p>yes, it is. But we're working with the in-kernel client based on 3.12 and for that we needed to make a backport. The <a href="https://jira.whamcloud.com/browse/LU-4429" title="clients leaking open handles/bad lock matching in ll_md_blocking_ast" class="issue-link" data-issue-key="LU-4429"><del>LU-4429</del></a> fix is also included in linux-next, so will probably be in 3.15.</p> <p>Roland</p> <p>Ah yes - thanks for clarifying!</p> <p>Fixed a long time ago</p> Related LU-4398 LU-4429 Development Rank 1|hzwdbj: Rank (Obsolete) 12363 Severity