https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8441 Lustre <p> Here's our reproducer:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">sh -c <span class="code-quote">'cd /p/lscratchd/$USER &amp;&amp; (f=toss-3321; rm -f $f; cp /bin/ls $f; od -N1 $f; ./$f; echo &gt; $f; rm -f $f)'</span> </pre> </div></div> <p>This looks similiar to <a href="https://jira.whamcloud.com/browse/LU-6232" title="Text file busy error -- lustre 2.6.0 client" class="issue-link" data-issue-key="LU-6232"><del>LU-6232</del></a>. This affects emacs which is impacting our users. This behaviour is a difference in how xemacs vs. vi deal with files that they already have open. With vi, it always writes to a temporary new file which it then moves over top of the file being edited. With xemacs, the original file is moved to &lt;file&gt;~ and a new file written on the first write. After that it overwrites the new file. One can see this by running them, saving a file, checking the inode number with "ls -i &lt;file&gt;", and then repeating the save and check operations. With xemacs the inode number won't change for each save. With vi, it will.</p> lustre-2.5.5-6chaos_2.6.32_573.26.1.1chaos.ch5.4.x86_64.x86_64 LU-8441

Text file busy error after overwriting file

Bug Minor Resolved Duplicate Jian Yu Teresa Kamakea llnl llnlfixready Tue, 26 Jul 2016 23:13:27 +0000 Wed, 22 Nov 2017 01:55:50 +0000 Wed, 22 Nov 2017 01:55:50 +0000 Lustre 2.8.0 Lustre 2.5.5 0 7 <p>I tried running the reproducer (slightly enhanced to show what it is doing) several times in a row and didn't see any problems (on 2.5.3.90 and 2.8.0 clients, 2.5.41.1 server), but I'm not sure what it is supposed to show:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[mythtv@twoshoes ~]$ sh -c 'cd /myth/tmp/mythtv &amp;&amp; (f=toss-3321; rm -f $f; cp /bin/ls $f; od -N1 $f; ./$f -li; echo &gt; $f; ls -li $f; rm -f $f)' 0000000 000177 0000001 total 1 144116715221943228 -rwxr-xr-x 1 mythtv mythtv 114024 2016-08-05 11:41 toss-3321 144116715221943228 -rwxr-xr-x 1 mythtv mythtv 1 2016-08-05 11:41 toss-3321 [mythtv@twoshoes ~]$ sh -c 'cd /myth/tmp/mythtv &amp;&amp; (f=toss-3321; rm -f $f; cp /bin/ls $f; od -N1 $f; ./$f -li; echo &gt; $f; ls -li $f; rm -f $f)' 0000000 000177 0000001 total 1 144116715221943230 -rwxr-xr-x 1 mythtv mythtv 114024 2016-08-05 11:41 toss-3321 144116715221943230 -rwxr-xr-x 1 mythtv mythtv 1 2016-08-05 11:41 toss-3321 </pre> </div></div> <p>Should this be reporting "text flie busy" at the "<tt>echo &gt; $f</tt>" stage?</p> <p>Sounds like <a href="https://jira.whamcloud.com/browse/LU-4367" title="unlink performance regression on lustre-2.5.52 client" class="issue-link" data-issue-key="LU-4367"><del>LU-4367</del></a> that had a patch adopted ages ago, but that was not 100% correct and was recently correctly refixed by <a href="https://jira.whamcloud.com/browse/LU-8019" title="Openlock breakage" class="issue-link" data-issue-key="LU-8019"><del>LU-8019</del></a></p> <p>Andreas, yes the echo is what fails with ETXTBSY.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">$ bass6@cab690 ~ /dev/pts/49 Fri Aug 05 14:10:59 &gt; sh -x -c <span class="code-quote">'cd /p/lscratchd/$USER &amp;&amp; (f=toss-3321; rm -f $f; cp /bin/ls $f; od -N1 $f; ./$f; echo &gt; $f; rm -f $f)'</span> + cd /p/lscratchd/bass6 + f=toss-3321 + rm -f toss-3321 + cp /bin/ls toss-3321 + od -N1 toss-3321 0000000 000177 0000001 + ./toss-3321 bin tmp toss-3321 y z + echo sh: toss-3321: Text file busy + rm -f toss-3321 </pre> </div></div> <p>Ned</p> <p>So it seems that this issue will not be present when your clients move to 2.8.x. Is this issue serious enough that you would consider upgrading your 2.5.x clients to a newer version or will you just wait to eradicate this when you upgrade to 2.8.x?</p> <p>Peter</p> <p>Peter, we do not plan to upgrade our clients to 2.8 until sometime in 2017. This bug impacts the workflow of at least one of our users. While not serious, bugs like this force our users to adopt workarounds that tend to propagate and long outlive the original problem. So it would be nice to not have to wait for 2.8 for a fix, and I would rank it as low priority.</p> <p>Sure Ned.</p> <p>Jian</p> <p>Can you please port the fix for <a href="https://jira.whamcloud.com/browse/LU-4367" title="unlink performance regression on lustre-2.5.52 client" class="issue-link" data-issue-key="LU-4367"><del>LU-4367</del></a> to the 2.5 FE branch</p> <p>Thanks</p> <p>Peter</p> <blockquote><p>Peter, we do not plan to upgrade our clients to 2.8 until sometime in 2017. </p></blockquote> <p>To clarify, that comment applies only to our existing client clusters. We will be bringing some new client and server clusters online running Lustre 2.8 over the next few weeks and months.</p> <p>Peter, this issue is still affecting our Lustre 2.8 clients.</p> <p>Hi Ned,</p> <p>I set up a test cluster with the latest 2.8 FE build on both clients and servers, tried to run the reproducer but didn't see the "Text file busy" issue:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[bass6@onyx-21vm5 ~]$ lctl get_param -n version lustre: 2.8.0.51 kernel: patchless_client build: 2.8.0.51-g9bda4c1-CHANGED-2.6.32-642.15.1.el6.x86_64 </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[bass6@onyx-21vm5 ~]$ sh -x -c 'cd /mnt/lustre/$USER &amp;&amp; (f=toss-3321; rm -f $f; cp /bin/ls $f; od -N1 $f; ./$f; echo &gt; $f; rm -f $f)' + cd /mnt/lustre/bass6 + f=toss-3321 + rm -f toss-3321 + cp /bin/ls toss-3321 + od -N1 toss-3321 0000000 000177 0000001 + ./toss-3321 toss-3321 + echo + rm -f toss-3321 </pre> </div></div> <p>Hi Ned,</p> <p>Could you please reproduce the failure and gather Lustre debug logs as follows?<br/> 1. On Client and MDS nodes, run the following commands to set full debug level:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lctl set_param debug=-1 debug_mb=1024 # lctl clear </pre> </div></div> <p>2. On Client node, run the following command to reproduce the failure:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># sh -x -c 'cd /p/lscratchd/$USER &amp;&amp; (f=toss-3321; rm -f $f; cp /bin/ls $f; od -N1 $f; ./$f; echo &gt; $f; rm -f $f)' </pre> </div></div> <p>3. On Client and MDS nodes, run the following command to gather debug logs:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lctl dk &gt; /tmp/debug-$(hostname -s)-$(date +%s) </pre> </div></div> <p>4. Upload the logs to this ticket for investigation.</p> <p>Thank you.</p> <p>Jian,</p> <p>I tar'd up the log files and attached them. Let me know if you need anything else.</p> <p>Thanks</p> <p>Thank you for the logs, Teresa. Let me look into them.</p> <p>Hi Teresa,</p> <p>"Text file busy" is error code "-ETXTBSY" (-26). In Client debug log, the error code was returned as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000080:00000001:0.0:1502922949.651746:0:179938:0:(namei.c:506:ll_lookup_it()) Process entered 00000080:00200000:0.0:1502922949.651746:0:179938:0:(namei.c:513:ll_lookup_it()) VFS Op:name=toss-3321,dir=144115205255725787/33554436(ffff8802c5555638),intent=open|creat ------8&lt;------ 00000002:00100000:0.0:1502922949.652793:0:179938:0:(mdc_locks.c:643:mdc_finish_enqueue()) @@@ op: 3 disposition: b, status: -26 req@ffff88031c0cf800 x1570675218372588/t0(0) o101-&gt;lsd-MDT0000-mdc-ffff88041e231000@172.19.2.102@o2ib100:12/10 lens 2464/600 e 0 to 0 dl 1502923055 ref 1 fl Complete:R/0/0 rc 301/301 </pre> </div></div> <p>The error status -26 was returned from MDS, while it seems the MDS debug log is inadequate because there is no -26 error code in the log. By looking into Lustre codes, I found "-ETXTBSY" was returned from mdt_write_get() and mdt_write_deny() in lustre/mdt/mdt_open.c. And both of the two functions were called from mdt_mfd_open(). It seems there is a race between the execution and write of the file toss-3321.</p> <p>Could you please try to add a "sync" between "./$f;" and "echo &gt; $f;" in the command line to see if the error still occurs? If it's still reproducible, could you please gather the Lustre debug log on MDS again? Right before running the "sh -x -c ..." command, please run "lctl dk &gt;/dev/null" on both Client and MDS to clear the debug buffer so that we can get the exact debug logs.</p> <p>Thank you.</p> <p>Hi Teresa,</p> <p>The similar issue was reported in <a href="https://jira.whamcloud.com/browse/LU-7727" title="open with FMODE_EXEC fails with ETXTBSY after a failed FMODE_WRITE open attempt on a read only client" class="issue-link" data-issue-key="LU-7727"><del>LU-7727</del></a>, which was fixed in Lustre 2.9.0. I back-ported the patch to Lustre b2_8_fe branch in <a href="https://review.whamcloud.com/28606" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28606</a>. Could you please try it? Thank you.</p> <p>Thank you Jian. I'll check in with Ned on trying the back-ported patch.</p> <p>You're welcome, Teresa.</p> <p>Hi Jian Yu, while we haven't tested your backported patch, <a href="https://jira.whamcloud.com/browse/LU-7727" title="open with FMODE_EXEC fails with ETXTBSY after a failed FMODE_WRITE open attempt on a read only client" class="issue-link" data-issue-key="LU-7727"><del>LU-7727</del></a> and patch 28606 relate specifically to read-only clients. This issue does not involve read-only clients, so I suspect it is not actually fixed.</p> <p>Hm, it's weird that the ticket is marked as resolved while apparently the problem is still present?</p> <p>Anyway I was looking into the client log provided to see if potentially <a href="https://jira.whamcloud.com/browse/LU-8371" title="Optimize open of known negative dentry" class="issue-link" data-issue-key="LU-8371"><del>LU-8371</del></a> might be useful here and noticed that the version you use does not match anything I have.<br/> I know LLNL used to host their Lustre repo on github, but that seems to ended at 2.5 and I cannot find the new location.</p> <p>Oleg</p> <p>We host the LLNL tree now - fs/lustre-release-fe-llnl.git</p> <p>Peter</p> <p>so I pulled that repo and tested 2.8.0_12.chaos tag and llnl/2.8.0-llnl branch (both appear to point at the same point) and still things don't match.</p> <p>In particular in file.c:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000080:00000001:20.0:1502922949.651650:0:179942:0:(file.c:331:ll_md_close()) Process leaving (rc=0 : 0 : 0) 00000080:00000001:20.0:1502922949.651651:0:179942:0:(file.c:392:ll_file_release()) Process leaving (rc=0 : 0 : 0) </pre> </div></div> <p>But:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">#ifdef CONFIG_FS_POSIX_ACL <span class="code-keyword">if</span> (sbi-&gt;ll_flags &amp; LL_SBI_RMT_CLIENT &amp;&amp; inode == inode-&gt;i_sb-&gt;s_root-&gt;d_inode) { struct ll_file_data *fd = LUSTRE_FPRIVATE(file); LASSERT(fd != NULL); 331&gt;&gt;&gt; <span class="code-keyword">if</span> (unlikely(fd-&gt;fd_flags &amp; LL_FILE_RMTACL)) { fd-&gt;fd_flags &amp;= ~LL_FILE_RMTACL; rct_del(&amp;sbi-&gt;ll_rct, current_pid()); et_search_free(&amp;sbi-&gt;ll_et, current_pid()); } } #endif </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> } 392&gt;&gt;&gt;&gt; op_data = ll_prep_md_op_data(NULL, parent-&gt;d_inode, de-&gt;d_inode, name, len, 0, LUSTRE_OPC_ANY, NULL); <span class="code-keyword">if</span> (IS_ERR(op_data)) RETURN(PTR_ERR(op_data)); </pre> </div></div> <p>Hi Oleg,</p> <blockquote><p>I pulled that repo and tested 2.8.0_12.chaos tag and llnl/2.8.0-llnl branch (both appear to point at the same point) and still things don't match.</p></blockquote> <p>What two things you are comparing - what doesn't match what?<br/> Thanks.</p> <p>The version on the client when the debug log was generated: Lustre: Lustre: Build Version: -13chaos-CHANGED-2.6.32-696.3.1.1chaos.ch5.6.x86_64</p> <p>The version on the MDS when the debug log was generated: Lustre: Lustre: Build Version: -13chaos-CHANGED-2.6.32-696.3.1.1chaos.ch5.6.x86_64</p> <p> </p> <p>Oleg,</p> <p>Sorry for the confusion.  Those look like Lustre 2.5 builds (I believe all our 2.8 builds are based on weak modules).  I'll huddle with Teresa and get back to you.</p> <p> </p> <p>Olaf: the code does not match the logs.<br/> Basically in those two lines of log above we can see that file.c lines 331 and 392 should be either GOTO, EXIT or RETURN. Additionally the function name should be as specified.</p> <p>I think the whole idea or reopening this ticket was due to 2.8 still exhibiting the ETXTBUSY issue? If that is so, please provide 2.8 logs with the problem.</p> <p>Oleg,</p> <p>Teresa will test with 2.8 and post here.</p> <p>I have attached the debug logs. There are 3 different scenarios: lustre 2.8 client/lustre2.8 server, lustre 2.5 client/lustre2.5 server, and lustre2.8 client/lustre2.5 server. The text file busy error shows up in the 2.5/2.5 scenario but not the others. The specific version information is included in the log files.</p> <p>Ned was mistaken when he stated we still see the problem on Lustre 2.8.  The cluster he tested on, and that the end-user reported the issue on, had been accidentally switched to the Lustre 2.5 client.</p> <p>We see the problem only in the 2.5 client.</p> <p>Olaf</p> <p>So am I correct in thinking that this means that the original theory that this is a duplicate of <a href="https://jira.whamcloud.com/browse/LU-7727" title="open with FMODE_EXEC fails with ETXTBSY after a failed FMODE_WRITE open attempt on a read only client" class="issue-link" data-issue-key="LU-7727"><del>LU-7727</del></a> holds true and that you would no longer expect to see this either when you apply the fix to your 2.5.x distribution or else upgrade to your 2.8.x distribution?</p> <p>Peter</p> <blockquote> <p>So am I correct in thinking that this means that the original theory that this is a duplicate of <del><a href="https://jira.whamcloud.com/browse/LU-7727" title="open with FMODE_EXEC fails with ETXTBSY after a failed FMODE_WRITE open attempt on a read only client" class="issue-link" data-issue-key="LU-7727"><del>LU-7727</del></a></del> holds true and that you would no longer expect to see this either when you apply the fix to your 2.5.x distribution</p></blockquote> <p>No, the patch for <del><a href="https://jira.whamcloud.com/browse/LU-7727" title="open with FMODE_EXEC fails with ETXTBSY after a failed FMODE_WRITE open attempt on a read only client" class="issue-link" data-issue-key="LU-7727"><del>LU-7727</del></a></del> is very specific to a read-only mount.  The patch says, in effect, "If the mount is read-only, and the file is being opened with the O_WRITE flag, then fail immediately instead of sending a request to the MDT".</p> <p>Without that patch, a request is sent to the MDT which then causes side-affects on other clients.</p> <p>It has a similar change on the MDT, for the case where an MDT receives an open request from a client which has mounted read-only.</p> <p>In our case, no clients are mounted read-only, so neither of those changes would have any effect.</p> <blockquote> <p>or else upgrade to your 2.8.x distribution?</p></blockquote> <p>Correct, when all our systems are at 2.8.x we do not expect to see this.</p> <p>Since the issue does not occur in 2.8.x, it may well be that there is an existing patch that could be backported, but <a href="https://jira.whamcloud.com/browse/LU-7727" title="open with FMODE_EXEC fails with ETXTBSY after a failed FMODE_WRITE open attempt on a read only client" class="issue-link" data-issue-key="LU-7727"><del>LU-7727</del></a> is not it.</p> <p>Ah yes. I did not read far enough back in the comments. So, <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=yujian" class="user-hover" rel="yujian">yujian</a> I think that the situation is now that this is believed to be a duplicate of <a href="https://jira.whamcloud.com/browse/LU-4367" title="unlink performance regression on lustre-2.5.52 client" class="issue-link" data-issue-key="LU-4367"><del>LU-4367</del></a> but that change is not a simple back port to a 2.5.x branch and so the question is - is this an impactful enough issue to warrant taking on the risk of introducing a largish change that has not been proven in production environments elsewhere?</p> <p>It looks like <a href="https://jira.whamcloud.com/browse/LU-8109" title="Separated MGS/MDS: conf-sanity test_32a: Unable to mount (-114)" class="issue-link" data-issue-key="LU-8109"><del>LU-8109</del></a> and all the preceding patches including <a href="https://jira.whamcloud.com/browse/LU-3544" title="Writing to new files under NFS export from Lustre will result in ENOENT (SLES11SP2)" class="issue-link" data-issue-key="LU-3544"><del>LU-3544</del></a> is necessary to fix this, but the exact serie is quite long and and not exactly known at this time. what is known is there were multiple problems introduced and fixed along the way until finally ironing out all wrinkles here.</p> <p>Hi Olaf,<br/> The fix is <a href="https://jira.whamcloud.com/browse/LU-8019" title="Openlock breakage" class="issue-link" data-issue-key="LU-8019"><del>LU-8019</del></a> and the prior patches. While trying to back-port the patch, I found it had a long dependency chain, including those for <a href="https://jira.whamcloud.com/browse/LU-3544" title="Writing to new files under NFS export from Lustre will result in ENOENT (SLES11SP2)" class="issue-link" data-issue-key="LU-3544"><del>LU-3544</del></a>, which contains more patches.</p> <p>Hi Jian and Oleg,</p> <p>Thank you for investigating. Given the complexity and risk we can close this notfix and we will do the same in our local ticket.</p> <p>For my education, can you tell me where the EBUSY is coming from in our broken case, and describe of some of the factors that lead to this? It need not be a complete and perfect description, just some hints that help understand the relevant code paths.</p> <p>Hi Olaf,<br/> According to debug logs, there were some analyses in the previous comment <a href="https://jira.hpdd.intel.com/browse/LU-8441?focusedCommentId=205806&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-205806" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-8441?focusedCommentId=205806&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-205806</a> about where the ETXTBSY is coming from.</p> <p>Hi Oleg,<br/> About the factors that lead to the issue, could you please give some hints? Thank you. </p> <p>This is mostly due to lingering file opens for write that got cached on the client. So when the exec comes it sees the file is opened for write and bails out (server side). We tried to just obtain a necessary ldlm lock before opening, but that proved to be very expensive.</p> <p>The cached open on the other hand is the real problem here, originally mostly aimed at nfs opened files, it managed to be enabled for other types of opens at times leading to such problems.</p> <p>Thanks, Jian and Oleg.</p> <p>That's all LLNL needs for this. You can close notfix (or whatever your normal process is).</p> <p>Thank you, Olaf. I'm closing this ticket as a duplicate of <a href="https://jira.whamcloud.com/browse/LU-8019" title="Openlock breakage" class="issue-link" data-issue-key="LU-8019"><del>LU-8019</del></a>.</p> Duplicate LU-8019 LU-7727 Development Rank 1|hzyiof: Rank (Obsolete) 9223372036854775807 Severity