https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10457 Lustre <p>If open_by_handle_at() is called in O_WRONLY or O_RDWR mode and then the file descriptor is closed, other lustre clients will still report ETXTBSY.</p> <p>Example:</p> <p>On cn16<br/> =======<br/> bschubert@cn16 ~&gt;sudo ~/src/test/open-test /mnt/lustre_client-ES24/bschubert/ime/test7 1<br/> Opened /mnt/lustre_client-ES24/bschubert/ime/test7/test7, fd: 4<br/> Closed d: 4</p> <p>Now on cn41<br/> =========<br/> bschubert@cn41 ~&gt;/mnt/lustre_client-ES24/bschubert/ime//test7<br/> -bash: /mnt/lustre_client-ES24/bschubert/ime//test7: Text file busy</p> <p>test7 is just any file which has the the execution bit set.</p> LU-10457

open_by_handle_at() in write mode triggers ETXTBSY

Bug Major Resolved Duplicate Oleg Drokin Diego Moreno Thu, 4 Jan 2018 19:48:23 +0000 Thu, 9 Jan 2020 19:48:50 +0000 Thu, 9 Jan 2020 19:47:59 +0000 0 13 <p>I reproduced the problem easily. Interesting, the first run would return ETXTBUSY, but the next run succeeds.</p> <p><span class="error">&#91;root@server17-el7-vm1 LU-10457&#93;</span># ./open-test /mnt/lustre/file 1<br/> Opened /mnt/lustre/file/file, fd: 4<br/> Closed d: 4</p> <p><span class="error">&#91;root@server17-el7-vm3 lustre&#93;</span># ./file <br/> -bash: ./file: /bin/bash: bad interpreter: Text file busy<br/> <span class="error">&#91;root@server17-el7-vm3 lustre&#93;</span># ./file <br/> hello!</p> <p>As expected, mdt_write_deny() returns -ETXTBUSY when the failure happens</p> <p> <br/> 00000004:00000001:0.0:1515116333.246172:0:12336:0:(mdt_open.c:342:mdt_mfd_open()) Process entered<br/> 00000004:00000001:0.0:1515116333.246173:0:12336:0:(mdt_open.c:174:mdt_write_deny()) Process entered<br/> 00000004:00000001:0.0:1515116333.246174:0:12336:0:(mdt_open.c:181:mdt_write_deny()) Process leaving (rc=18446744073709551590 : -26 : ffffffffffffffe6)<br/> 00000004:00000001:0.0:1515116333.246175:0:12336:0:(mdt_open.c:391:mdt_mfd_open()) Process leaving (rc=18446744073709551590 : -26 : ffffffffffffffe6)<br/> 00000004:00000001:0.0:1515116333.246176:0:12336:0:(mdt_open.c:615:mdt_finish_open()) Process leaving (rc=18446744073709551590 : -26 : ffffffffffffffe6)<br/> 00000004:00000001:0.0:1515116333.246178:0:12336:0:(mdt_open.c:1589:mdt_reint_open()) Process leaving</p> <p>When open-test closing the file, no mdt_mfd_close() is called on MDS side. And the trace on client side confirmed that  ll_md_close() dosn't call ll_md_real_close() because md_lock_match() returns 8. So, this program holds some LDLM lock when closing?</p> <p>00000002:00000001:0.0:1515116985.549766:0:22146:0:(mdc_locks.c:149:mdc_lock_match()) Process leaving (rc=8 : 8 : 8)<br/> 00800000:00000001:0.0:1515116985.549766:0:22146:0:(obd_class.h:1740:md_lock_match()) Process leaving (rc=8 : 8 : 8)<br/> 00800000:00000001:0.0:1515116985.549767:0:22146:0:(lmv_obd.c:2965:lmv_lock_match()) Process leaving (rc=8 : 8 : 8)<br/> 00000080:00000001:0.0:1515116985.549768:0:22146:0:(obd_class.h:1740:md_lock_match()) Process leaving (rc=8 : 8 : 8)<br/> 00000080:00000001:0.0:1515116985.549771:0:22146:0:(file.c:314:ll_md_close()) Process leaving (rc=0 : 0 : 0)</p> <p>So following things happend in sequence:</p> <ol> <li>open-test got LDLM lock of the file ./test</li> <li>open-test closed the file, yet do not call ll_md_real_close() because holding the LDLM lock</li> <li>./test program tried to open the file with EXEC yet got -ETXTBUSY in mdt_write_deny()</li> <li>The mdt_blocking_ast() was triggered, it cancels of the LDLM lock got by open-test program, and mdt_mfd_close()/mdt_write_put() is called to release the reference</li> <li>The next run of open-test can succeed because of step 4.</li> </ol> <p>The open-test get the LDLM lock when opening file because ldd-&gt;lld_nfs_dentry is set to 1 in ll_iget_for_nfs().</p> <p> </p> <p>Hm, I tested this on the latest master on rhel7.2 (disregard the centos6 in the hostname) and don't see any problems, what version are you testing on, what kernel:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@centos6-16 tests]# cat /tmp/test.sh #!/bin/bash cp /bin/ls /mnt/lustre /mnt/lustre/ls -ld . /tmp/open-test /mnt/lustre/ls 2 TIME=0 while ! /mnt/lustre2/ls -ld . ; do echo nope ; TIME=$((TIME + 1)) ; sleep 1 ; done echo Waited $TIME seconds for the open to clear [root@centos6-16 tests]# bash /tmp/test.sh drwxrwxr-x 12 green green 12288 Jan 4 01:37 . Opened /mnt/lustre/ls/ls, fd: 4 Closed d: 4 /tmp/test.sh: line 8: /mnt/lustre2/ls: Text file busy nope drwxrwxr-x 12 green green 12288 Jan 4 01:37 . Waited 1 seconds for the open to clear </pre> </div></div> <p>I guess I did not read it far enough, yes there's one ETXTBUSY report due to the open lock.</p> <p>it appears that the name_to_handle_at/open_by_handle_at use nfs-encoded export operation leading to the nfs detecting logic to trigger so the system sort of operates as designed.</p> <p>It's going to be tricky to separate real nfs from these users I guess and we don't want the extra lock hit when opening the file. I guess the new downgrade logic might help us here to get a bigger lock and then just drop the unneeded bit.</p> <p>Hmm, I can't imagine how that this works as it is supposed to, even in the NFS case. Maybe I should have pointed this out in more detail, but in my initial example I used <b>two different nodes</b>. <br/> For NFS or any other overlay file system, one can expect that there are multiple nodes involved. For NFS users typically would create/modify files on their desktop to later on execute them natively on Lustre.<br/> For the IME use case, the file is opened for multiple reasons in RW mode on the ime server, but users also later on want to use the files natively on Lustre. </p> <p>Ah, actually Li was also using two different nodes. Sorry, I only saw server17-el7 and didn't notice the differentiation between -vm1 and -vm3.</p> <p>Which Lustre version is this with? On the systems I tested with, it would never succeed to execute on the other node, until either </p> <ul class="alternate" type="square"> <li>the node the had opened the file would execute it itself</li> <li>the node the had opened the file would unmount lustre</li> <li>I would be patient and wait for a very long time (&gt; 30min)</li> </ul> <p>Please note that /mnt/lustre and /mnt/lustre 2 are different mountpoints, so they act the same as two different nodes, just more convenient to test.</p> <p>This is resolved by <a href="https://review.whamcloud.com/#/c/9063/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/9063/</a> (<a href="https://jira.whamcloud.com/browse/LU-4398" title="mdt_object_open_lock() may not flush conflicting handles" class="issue-link" data-issue-key="LU-4398"><del>LU-4398</del></a> mdt: acquire an open lock for write or execute). But that change was reverted from master due to the metadata performance impact described by DDN in <a href="https://jira.whamcloud.com/browse/LU-5197" title="A performance regression of &quot;FileRead&quot; metadata operation" class="issue-link" data-issue-key="LU-5197"><del>LU-5197</del></a>.</p> <p>Perhaps the exiting functionality of open leases could be used in the open by handle path to address this issue without incurring the performance drop.</p> <p>&gt; Which Lustre version is this with? </p> <p>I was testing on master branch. I guess you are using IEEL3 (2.7)? Something might have been changed between them.</p> <p>Seems <a href="https://review.whamcloud.com/#/c/9063/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/9063/</a> (<a href="https://jira.whamcloud.com/browse/LU-4398" title="mdt_object_open_lock() may not flush conflicting handles" class="issue-link" data-issue-key="LU-4398"><del>LU-4398</del></a> mdt: acquire an open lock for write or execute) can resolve the problem, after applied it back on latest master, never reproduced the issue.<br/> <span class="error">&#91;root@vm3 ~&#93;</span># ./open_test /mnt/lustre/file 1<br/> Opened /mnt/lustre/file/file, fd: 4<br/> Closed d: 4</p> <p><span class="error">&#91;root@vm6 ~&#93;</span># /mnt/lustre/file<br/> hello lustre<br/> <span class="error">&#91;root@vm6 ~&#93;</span># /mnt/lustre/file<br/> hello lustre<br/> <span class="error">&#91;root@vm6 ~&#93;</span># /mnt/lustre/file<br/> hello lustre<br/> <span class="error">&#91;root@vm6 ~&#93;</span># /mnt/lustre/file<br/> hello lustre<br/> <span class="error">&#91;root@vm6 ~&#93;</span># /mnt/lustre/file<br/> hello lustre</p> <p>Oleg pointed me at this, I reported a duplicate and contributed a patch and test case:<br/> <a href="https://review.whamcloud.com/#/c/31304/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/31304/</a></p> <p>If we limited my patch to executable files as Oleg suggested, that might fit the bill. Curious what others think. I'll refresh tomorrow.</p> <p>Hi all, I just resubmit <a href="https://jira.whamcloud.com/browse/LU-4398" title="mdt_object_open_lock() may not flush conflicting handles" class="issue-link" data-issue-key="LU-4398"><del>LU-4398</del></a> (<a href="https://review.whamcloud.com/32020" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32020</a>) as Jinshan suggested, with it applied, the problem is gone, and with some simple tests, no significant regression found, but still, please feel free to try and test it more, thanks.</p> <p>Hi all, I think there another implication of this issue. Our customer is complaining that quotas are not correctly released. We have basically mostly worked around the ETXTBSY issue, but I don't think we can do anything about quotas on our side.<br/> Looking at the patches, I <b>think</b> this patch <a href="https://review.whamcloud.com/32020" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32020</a> will not help, as it will try to release conflicting locks on an O_EXEC attempt. The alternative patch from Pattrick  <a href="https://review.whamcloud.com/#/c/31304/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/31304/</a> should work, as it always sends an mds close from the client, if the file was opened in write mode. Is there any side effect? It should just remove an NFS optimization?</p> <p>It's a pretty significant optimization, but that aside, yes, I think it should be OK.</p> <p>We have (at least for now) decided to live with the problem.</p> <p>Why do you think this affects quotas, though?  I'm a little puzzled about the connection?</p> <p>Without any prove if the root cause of <a href="https://jira.whamcloud.com/browse/LU-10457" title="open_by_handle_at() in write mode triggers ETXTBSY" class="issue-link" data-issue-key="LU-10457"><del>LU-10457</del></a> is also the root of our quota issue - I would assume that a file that gets unlinked, but that is still open on the MDS will not release the data and so also will not release quotas.</p> <p>Ah.  Interesting, I see your point.  Yes, that seems possible.</p> <p>similar fix in <a href="https://review.whamcloud.com/32265" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/32265</a> for <a href="https://jira.whamcloud.com/browse/LU-4398" title="mdt_object_open_lock() may not flush conflicting handles" class="issue-link" data-issue-key="LU-4398"><del>LU-4398</del></a> and I've already confimred patch solves ETXTBSY issue without performance regressions.</p> <p>Here is test resutls</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@c01 ~]# cat /scratch1/test echo hello [root@c02 ~]# /scratch1/a.out /scratch1/test 1 Opened /scratch1/test/test, fd: 4 Closed d: 4 [root@c01 ~]# /scratch1/test hello [root@c01 ~]# /scratch1/test hello [root@c01 ~]# /scratch1/test hello </pre> </div></div> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>16 client, 256 process (mdtest -n 2000 -u -vv -F -d /scratch1/mdtest.out/ -i 3 -p 10) master SUMMARY: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 89389.726 78571.859 83605.731 4448.114 File stat : 263650.433 221026.947 238222.946 18348.631 File read : 113141.781 111882.782 112494.749 514.582 File removal : 121785.424 109912.532 114749.674 5090.305 Tree creation : 204.674 27.510 140.893 80.383 Tree removal : 30.096 28.858 29.363 0.531 V-1: Entering timestamp... master + patch https://review.whamcloud.com/32265 SUMMARY: (of 3 iterations) Operation Max Min Mean Std Dev --------- --- --- ---- ------- File creation : 84851.987 82996.117 83819.233 772.021 File stat : 262610.064 215623.544 244595.023 20687.611 File read : 115494.747 112069.322 113774.145 1398.468 File removal : 121395.003 115276.620 118372.989 2498.373 Tree creation : 223.484 65.453 156.498 66.722 Tree removal : 28.673 17.037 22.827 4.751 V-1: Entering timestamp... </pre> </div></div> <p>I was missing test of quota issue. As Bernd mentioned above, at the end, we might need Patrick's patch to solve both problems...</p> <p>the unlink issue is somewhat trivially fixable by just ensuring that unlink also revokes open bit (we already revoke the lookup bit). The slowdown would only happen for files that actually have openlock cached on a client.</p> <p>This might have been fixed by patch <a href="https://review.whamcloud.com/36641" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/36641</a> "<tt><a href="https://jira.whamcloud.com/browse/LU-8585" title="All Lustre test suites should pass with subdirectory mount" class="issue-link" data-issue-key="LU-8585">LU-8585</a> llite: don't cache MDS_OPEN_LOCK for volatile files</tt>".</p> Duplicate LU-8585 LU-10667 Related LU-4398 LU-12661 Development Rank 1|hzzqiv: Rank (Obsolete) 9223372036854775807 Severity