https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12997 Lustre <p>We have been seeing getcwd() return ENOENT on directories that are, in<br/> fact, always there. We can reliably reproduce this problem with the<br/> attached test-getcwd.c code on Lustre Server 2.12.2 and Lustre Client<br/> 2.12.3 on RHEL7.7 as well as many other Lustre version and RHEL7<br/> version combinations.</p> <p>We see reports in <a href="https://jira.whamcloud.com/browse/LU-9735" title="Sles12Sp2 and 2.9 getcwd() sometimes fails" class="issue-link" data-issue-key="LU-9735"><del>LU-9735</del></a> about RHEL7 clients getting an ENOENT return<br/> from getcwd(), but I don't understand if a solution is in the works or<br/> not. We are also not sure if this is a Lustre problem, an RHEL kernel<br/> problem, or both.</p> <p>The LD_PRELOAD workaround from <a href="https://jira.whamcloud.com/browse/LU-9735" title="Sles12Sp2 and 2.9 getcwd() sometimes fails" class="issue-link" data-issue-key="LU-9735"><del>LU-9735</del></a> is working for us, but I am<br/> wondering if there is a proper solution pending. Is there anything we<br/> can do to help?</p> RHEL7.7 as well as other minor versions of RHEL7 on x86_64. LU-12997

getcwd() returns ENOENT on RHEL7

Bug Minor Resolved Not a Bug WC Triage K. Scott Rowe Thu, 21 Nov 2019 22:55:17 +0000 Mon, 13 Jul 2020 20:47:28 +0000 Mon, 13 Jul 2020 20:47:28 +0000 Lustre 2.12.2 0 9 <p>This is a race in ll_splice_alias() due to the use of d_move() when the inode is for a directories. Their are fixes for this but it corrects the way Lustre handles it dcache which causes other types of breakage. I tried some ideas to fix this but its a work in progress. Basically the bug is $MOUNT/.lustre/fid/$fid_for_mount means that dentry $fid_for_mount == dentry $MOUNT which causes a circular loop that crashes the node. I might be able to handle this special case using d_real() that is in newer kernels.</p> <p>This sounds like the bug fixed upstream by</p> <p>Commit 61647823aa92 ("VFS: close race between getcwd() and d_move()")</p> <p>Fixed in v4.16</p> <p>Probably the best approach for lustre supporting older kernels is to copy d_drop() from a newer kernel into libcfs, and use that instead of the exported d_drop.</p> <p> </p> <p>I submitted a bugzilla:</p> <p><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1811124" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.redhat.com/show_bug.cgi?id=1811124</a></p> <p>Thanks for continuing to work on this.  I have some good news.  We upgraded our servers from Lustre-2.5.5 to Lustre-2.10.8 on Feb. 20, 2020 and the problem has been greatly reduced.</p> <p>The test program (test-getcwd.c) still fails when run on our Lustre filesystem but the number of times a user has run into this getcwd() problem with other code since we upgraded our Lustre servers has dropped to almost zero.</p> <p> </p> <p>I was told by RedHat that a fix was landed to RHEL7.8</p> <p>W installed a machine with RHEL-7.8 using kernel 3.10.0-1127.el7.x86_64 and while I see a bug fix for a getcwd problem</p> <p> </p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ rpm -qi --changelog kernel|grep getcwd - [fs] vfs: close race between getcwd() and d_move() (Miklos Szeredi) [1631631]</pre> </div></div> <p> </p> <p> </p> <p><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1631631" class="external-link" target="_blank" rel="nofollow noopener">1631631</a> is a different bug id than <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1811124" class="external-link" target="_blank" rel="nofollow noopener">1811124</a> that James A Simmons reported.  And, I can still reproduce the problem on our Lustre-2.10.8 filesystem using the 2.12.4 client on RHEL-7.8.</p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ ./test-getcwd /lustre/aoc/sciops/krowe/tmp test-getcwd: test-getcwd.c:44: main: Assertion `rc == 0' failed. Aborted </pre> </div></div> <p> </p> <p> </p> <p>RHEL7.8 contains a fix so this can be closed. If people encounter this issue please move to RHEL7.8</p> <p>Perhaps I wasn't clear enough.</p> <p>This is still a problem.</p> <p>I can still reproduce this error with RHEL7.8 using the test-getcwd.c program above.</p> <p> </p> <p>Sigh. RedHat claimed this was fixed. Its going to take some push to get them to resolve this. I don't have the power to resolve this. Some one with greater influence with RedHat will have to discuss a fix.</p> <p>Peter can you take over this issue since you seem to have better relations with RedHat to resolve this.</p> <p>Do you have the ability to test this on an RHEL7.8 host?  It would be good to have a second data point.  I suppose it is possible I am seeing this issue with our RHEL7.8 host for some other reason that I can't think of.</p> <p>The kernel was just upgraded on my test RHEL-7.8 machine.  It is now running (3.10.0-1127.8.2.el7.x86_64) and I no longer get getcwd() failures</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>$ ./test-getcwd /lustre/aoc/sciops/krowe/tmp getcwd succeeded</pre> </div></div> <p>I don't understand why this failed with kernel 3.10.0-1127.el7.x86_64 and works now but assuming it continues to work after more kernel updates I would say this problem may be fixed.  Again, if you have the ability to check this yourself, please do.  My environment may be customized in strange ways.</p> <p>I have since tested a later kernel, 3.10.0-1127.13.1.el7.x86_64, and it also works.  So I think the solution is to upgrade to at least kernel 3.10.0-1127.el7.x86_64.</p> <p> </p> <p>This ticket can be closed.  Thanks for your help.</p> <p> </p> Related LU-9868 Development Rank 1|i00psf: Rank (Obsolete) 9223372036854775807 Severity