https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-9735 Lustre <p>This is a duplicate of <a href="https://jira.whamcloud.com/browse/LU-9208" title="getcwd() sometimes fails" class="issue-link" data-issue-key="LU-9208"><del>LU-9208</del></a>. Opening this case for tracking for nasa. We start to see this once we updated the clients to Sles12SP2 and lustre2.9</p> <p>Using the test code provide <a href="https://jira.whamcloud.com/browse/LU-9208" title="getcwd() sometimes fails" class="issue-link" data-issue-key="LU-9208"><del>LU-9208</del></a> (miranda) I was able to reproduce the bug on a single node.</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Iteration = 868, Run Time = 0.9614 sec., Transfer Rate = 120.7790 10e+06 Bytes/sec/proc Iteration = 869, Run Time = 1.5308 sec., Transfer Rate = 75.8561 10e+06 Bytes/sec/proc forrtl: severe (121): Cannot access current working directory <span class="code-keyword">for</span> unit 10012, file <span class="code-quote">"Unknown"</span> Image PC Routine Line Source miranda 0000000000409F29 Unknown Unknown Unknown miranda 00000000004169D2 Unknown Unknown Unknown miranda 0000000000404045 Unknown Unknown Unknown miranda 0000000000402FDE Unknown Unknown Unknown libc.so.6 00002AAAAB5B96E5 Unknown Unknown Unknown miranda 0000000000402EE9 Unknown Unknown Unknown MPT ERROR: MPI_COMM_WORLD rank 12 has terminated without calling MPI_Finalize() aborting job </pre> </div></div> <p> I was able to capture some debug logs I have attached to the case. I was unable to reproduce it using "+trace". But will continue to try.</p> LU-9735

Sles12Sp2 and 2.9 getcwd() sometimes fails

Bug Critical Resolved Fixed James A Simmons Mahmoud Hanafi ORNL Wed, 5 Jul 2017 06:57:59 +0000 Wed, 24 Jul 2019 17:36:25 +0000 Wed, 24 Jul 2019 17:14:47 +0000 Lustre 2.9.0 Lustre 2.11.0 Lustre 2.10.4 1 24 <p>Bobijam</p> <p>Could you please assist with this issue?</p> <p>Thanks</p> <p>Peter</p> <p>Is it NFS involved? I haven't found -2 (ENOENT) error is the log.</p> <p>NFS is not involved. I'll try to get additional debugging to catch the ENOENT.</p> <p> </p> <p> I am unable to reproduce with -1 debugging. What is the min debugging level need to diag the issue?</p> <p>How about start with "trace inode info other dentry rpctrace vfstrace console" ? And if possible add "dlmtrace" as well.</p> <p>Uploaded miranda.debug.1499341246.gz but I still didn't see any ENOENT Error. This was the error on the client</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Iteration = 9649, Run Time = 2.5712 sec., Transfer Rate = 45.1620 10e+06 Bytes/sec/proc forrtl: severe (121): Cannot access current working directory <span class="code-keyword">for</span> unit 10013, file <span class="code-quote">"Unknown"</span> Image PC Routine Line Source miranda 0000000000409F29 Unknown Unknown Unknown miranda 00000000004169D2 Unknown Unknown Unknown miranda 0000000000404045 Unknown Unknown Unknown miranda 0000000000402FDE Unknown Unknown Unknown libc.so.6 00002AAAAB5B96E5 Unknown Unknown Unknown miranda 0000000000402EE9 Unknown Unknown Unknown MPT ERROR: MPI_COMM_WORLD rank 13 has terminated without calling MPI_Finalize() aborting job </pre> </div></div> <p>Is it possible to locate where does "miranda 0000000000409F29 Unknown Unknown Unknown" call in miranda_io.F ?</p> <p>And what files does miranda_io.F created/write/read ? I don't know fortran well.</p> <p>attached object dump of miranda.</p> <p> </p> <p>The calling path should be like this</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>402fd9 callq 402ff0 &lt;MAIN__&gt; 404040 callq 415ea0 &lt;for_open&gt; 416a27 callq 44f410 &lt;for__reopen_file&gt; 44f48b callq 44f550 &lt;for__compute_filename&gt; 4509cd callq 402cb0 &lt;getcwd@plt&gt; </pre> </div></div> <p>it seems that the getcwd is called from the open() for write, and the open() is trying to get the full path name of the file.</p> <p>That is correct. forran open always call getcwd to get full path before doing the actual open.</p> <p>I uploaded new debug log to ftp site: /uploads/<a href="https://jira.whamcloud.com/browse/LU-9735" title="Sles12Sp2 and 2.9 getcwd() sometimes fails" class="issue-link" data-issue-key="LU-9735"><del>LU-9735</del></a>/miranda.debug.1499351104.gz</p> <p>What about a debug patch to help diag the issue.</p> <p> </p> <p> The files written are all called 'miranda_io.out.100xx' where xx is the rank number.</p> <p>In what path did you start the test program? Is it possible that during the test run, your working path got some change temporary?</p> <p>The log shows that miranda_io.out.10013 has been going through the read phase while no write happens on it. And other miranda_io.out.xxxx has the write phase happened. </p> <p>The thing is getcwd only determine the current working directory, is it under Lustre file system? If not, Lustre log cannot know the reason why it fails. </p> <p>yes the working directory is on the lustre file system.</p> <p> </p> <p>This is similar issue as <del><a href="https://jira.whamcloud.com/browse/LU-645" title="getcwd fails" class="issue-link" data-issue-key="LU-645"><del>LU-645</del></a></del>. Our LD_PRELOAD workaround works in this case as well.<br/> I wrote a system tap script to trace <br/> kernel.function("SyS_getcwd@../fs/dcache.c:3254")</p> <p>Here you can see at the failure it returns <br/> 0xfffffffffffffffe</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">return</span>=<span class="code-keyword">return</span>=0x1c call=buf=0x7fffffff5a67 size=0x1000 <span class="code-keyword">return</span>=<span class="code-keyword">return</span>=0x1c call=buf=0x7fffffff5a67 size=0x1000 <span class="code-keyword">return</span>=<span class="code-keyword">return</span>=0x1c call=buf=0x7fffffff5a67 size=0x1000 <span class="code-keyword">return</span>=<span class="code-keyword">return</span>=0xfffffffffffffffe call=buf=0x7fffffff5a67 size=0x1000 <span class="code-keyword">return</span>=<span class="code-keyword">return</span>=0x1c call=buf=0x7fffffff5a67 size=0x1000 <span class="code-keyword">return</span>=<span class="code-keyword">return</span>=0x1c call=buf=0x7fffffff5a67 size=0x1000 <span class="code-keyword">return</span>=<span class="code-keyword">return</span>=0x1c call=buf=0x7fffffff5a67 size=0x1000 <span class="code-keyword">return</span>=<span class="code-keyword">return</span>=0x1c call=buf=0x8bc040 size=0x1000 <span class="code-keyword">return</span>=<span class="code-keyword">return</span>=0xb </pre> </div></div> <p>error of 0x1c is ENOSPC, error of 0xb is EAGAIN<br/> #define ENOSPC 28 /* No space left on device */<br/> #define EAGAIN 11 /* Try again */</p> <p>return=0xfffffffffffffffe is -2</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">SyS_getcwd 0 miranda.debug(26848): -&gt; buf=0x7fffffff5a67 size=0x1000 0x2aaaab676d92 : getcwd+0x72/0x6d0 [/lib64/libc-2.22.so] 0x452762 : for__compute_filename+0x1482/0x16d0 [/home7/mhanafi/LUSTRE/BUGS/GETPWD/miranda.debug] 0x453a89 : for__open_proc+0xc9/0x34b0 [/home7/mhanafi/LUSTRE/BUGS/GETPWD/miranda.debug] Returning from: 0xffffffff81225fa0 : sys_getcwd+0x0/0x190 [kernel] Returning to : 0xffffffff81618e1e : tracesys_phase2+0x84/0x89 [kernel] 0xffffffff8101ae0c : try_stack_unwind+0x17c/0x190 [kernel] 0xffffffff81019ce4 : dump_trace+0x64/0x380 [kernel] 0xffffffffa1ae1664 [stap_971bda5852e0877040783dde29bd60d_26789+0xe664/0x0] 0xffffffffa1ae2315 [stap_971bda5852e0877040783dde29bd60d_26789+0xf315/0x0] 0xffffffffa1ae24d0 [stap_971bda5852e0877040783dde29bd60d_26789+0xf4d0/0x0] 0xffffffff8105c18b : trampoline_handler+0x11b/0x1d0 [kernel] 0xffffffff8105bcae : kretprobe_trampoline+0x25/0x57 [kernel] 0xffffffff81618e1e : tracesys_phase2+0x84/0x89 [kernel] (inexact) SyS_getcwd 135 miranda.debug(26848): &lt;- <span class="code-keyword">return</span>=-2 </pre> </div></div> <p>I think this is what is is happing.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">SYSCALL_DEFINE2(getcwd, <span class="code-object">char</span> __user *, buf, unsigned <span class="code-object">long</span>, size) { <span class="code-object">int</span> error; struct path pwd, root; <span class="code-object">char</span> *page = __getname(); <span class="code-keyword">if</span> (!page) <span class="code-keyword">return</span> -ENOMEM; rcu_read_lock(); get_fs_root_and_pwd_rcu(current-&gt;fs, &amp;root, &amp;pwd); error = -ENOENT; <span class="code-keyword">if</span> (!d_unlinked(pwd.dentry)) { </pre> </div></div> <p>In the getcwd syscall between</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> get_fs_root_and_pwd_rcu(current-&gt;fs, &amp;root, &amp;pwd); </pre> </div></div> <p>and</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">if</span> (!d_unlinked(pwd.dentry)) { </pre> </div></div> <p>The dentry has is getting dropped. From systemtap trace you can see here that .d_iname="1stripe" is dropped and getpwd errors out.</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">__d_drop 363 dentry={.d_flags=2621646, .d_seq={.sequence=2787}, .d_hash={.next=0x0, .pprev=0xffffc90002553280}, .d_parent=0xffff88105c8bc918, .d_name={&lt;union&gt;={&lt;<span class="code-keyword">class&</span>gt;={.hash=1222078897, .len=7}, .hash_len=31286849969}, .name=<span class="code-quote">"1stripe"</span>}, .d_inode=0xffff880dd6b2c550, .d_iname=<span class="code-quote">"1stripe"</span>, .d_lockref={&lt;union&gt;={.lock_count=120259084289, &lt;<span class="code-keyword">class&</span>gt;={.lock={&lt;union&gt;={.rlock={.raw_lock={.val={.counter=1}}}}}, .count=28}}}, .d_op=0xffffffffa0ef8240, .d_sb=0xffff880816ab6800, .d_time=0, .d_fsdata=0xffff880e403b3740, .d_lru={.next=0x __d_drop 363 dentry={.d_flags=142, .d_seq={.sequence=1}, .d_hash={.next=0x0, .pprev=0x0}, .d_parent=0xffff88105c8bc918, .d_name={&lt;union&gt;={&lt;<span class="code-keyword">class&</span>gt;={.hash=1222078897, .len=7}, .hash_len=31286849969}, .name=<span class="code-quote">"1stripe"</span>}, .d_inode=0x0, .d_iname=<span class="code-quote">"1stripe"</span>, .d_lockref={&lt;union&gt;={.lock_count=4294967297, &lt;<span class="code-keyword">class&</span>gt;={.lock={&lt;union&gt;={.rlock={.raw_lock={.val={.counter=1}}}}}, .count=1}}}, .d_op=0xffffffffa0ef8240, .d_sb=0xffff880816ab6800, .d_time=0, .d_fsdata=0x0, .d_lru={.next=0xffff88105c85ebd8, .prev=0xffff88105c85ebd8}, .d_child={ __d_drop 363 dentry={.d_flags=142, .d_seq={.sequence=2}, .d_hash={.next=0x0, .pprev=0x0}, .d_parent=0xffff88105c8bc918, .d_name={&lt;union&gt;={&lt;<span class="code-keyword">class&</span>gt;={.hash=1222078897, .len=7}, .hash_len=31286849969}, .name=<span class="code-quote">"1stripe"</span>}, .d_inode=0x0, .d_iname=<span class="code-quote">"1stripe"</span>, .d_lockref={&lt;union&gt;={.lock_count=18446743523953737729, &lt;<span class="code-keyword">class&</span>gt;={.lock={&lt;union&gt;={.rlock={.raw_lock={.val={.counter=1}}}}}, .count=-128}}}, .d_op=0xffffffffa0ef8240, .d_sb=0xffff880816ab6800, .d_time=0, .d_fsdata=0x0, .d_lru={.next=0xffff88105c85ebd8, .prev=0xffff88105c85ebd8 SyS_getcwd 0 miranda.debug(354): &lt;- <span class="code-keyword">return</span>=-2 /nobackupp8/mhanafi/1stripe 0x2aaaab676d92 [/lib64/libc-2.22.so+0xddd92/0x39f000] Returning from: 0xffffffff81225fa0 : sys_getcwd+0x0/0x190 [kernel] Returning to : 0xffffffff81618e1e : tracesys_phase2+0x84/0x89 [kernel] 0xffffffff8101ae0c : try_stack_unwind+0x17c/0x190 [kernel] 0xffffffff81019ce4 : dump_trace+0x64/0x380 [kernel] 0xffffffffa1155b09 [stap_257b7a801b68bce2a60161ccda5d2586_4_302+0x12b09/0x0] 0xffffffffa1156865 [stap_257b7a801b68bce2a60161ccda5d2586_4_302+0x13865/0x0] 0xffffffffa1156a20 [stap_257b7a801b68bce2a60161ccda5d2586_4_302+0x13a20/0x0] 0xffffffff8105c18b : trampoline_handler+0x11b/0x1d0 [kernel] 0xffffffff8105bcae : kretprobe_trampoline+0x25/0x57 [kernel] 0xffffffff81618e1e : tracesys_phase2+0x84/0x89 [kernel] (inexact) Pass 5: run completed in 0usr/180sys/11238real ms. </pre> </div></div> <p>Would you mind trying this patch?</p> <p>No patch was attached.</p> <p>Can you see <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/27536/27536_unoptimize-atomic_open-of-negative-dentry.patch" title="unoptimize-atomic_open-of-negative-dentry.patch attached to LU-9735">unoptimize-atomic_open-of-negative-dentry.patch^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> </p> <p>I can see the attached path now. But is it not in Gerrit?</p> <p>We tried the patch and the code failed they same way.</p> <p> Could this be related to <a href="https://jira.hpdd.intel.com/browse/LU-8891?" class="external-link" target="_blank" rel="nofollow noopener">https://jira.hpdd.intel.com/browse/LU-8891?</a></p> <p> </p> <p>So it looks like the sequence of events unfolds like this:<br/> Process1: open_create("1stripe/somefilename"); -&gt; this in turn goes to cancel the update lock we have for "1stripe". But the lock is shared with lookup lock, so cancelling it means the dentry is dropped.<br/> Process2: does getcwd - revalidates the "1stripe" first and then checks validity.</p> <p>In the bigger log it's all happening at 1499341246.216525<br/> Process 1 is 34167 and process 2 is 34165:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000080:00200000:12.0:1499341246.216524:0:34165:0:(dcache.c:357:ll_revalidate_nd()) VFS Op:name=1stripe, flags=17 00000080:00002000:14.0:1499341246.216525:0:34167:0:(llite_internal.h:1388:d_lustre_invalidate()) invalidate dentry 1stripe (ffff8808fcbfdd98) parent ffff8808fcbfdf18 inode ffff8810414180d0 refc 265 00000080:00000001:12.0:1499341246.216525:0:34165:0:(dcache.c:360:ll_revalidate_nd()) Process leaving (rc=1 : 1 : 1) 00000080:00000001:12.0:1499341246.216525:0:34165:0:(file.c:3695:ll_inode_permission()) Process entered </pre> </div></div> <p>This sounds kind of a normal race - I don't see why dentry could not be invalidated between lookup and check and then revalidated later (also explains why NFS would trip this at times).<br/> I see no easy way to quash this without a kernel patch and even that likely would not be too easy and we'd need to get Al Viro behind it I imagine.</p> <p>Any updates?</p> <p>we‘re still investigating it</p> <p>Btw, I did get it to fail with sles12sp2 and lustre2.10.</p> <p>The problem is clear as Oleg mentioned. It doesn't revalidate dentry in the sys call of getcwd() so we need to patch the kernel. I can't think of any workaround solution at this moment.</p> <p>For reference, uploaded the code for the getcwd() retry hack we are using with LD_PRELOAD to work around this problem.  We have not noticed any significant performance impacts with this approach.  Perhaps similar code can be added to Lustre itself.</p> <p>Built with something like:</p> <p><tt>gcc -Wall -c -fPIC -DPIC getcwdHack.c -o getcwdHack.lo</tt><br/> <tt>gcc -Wall -shared getcwdHack.lo -ldl -Wl,-soname -Wl,libgetcwdHack.so -o libgetcwdHack.so</tt></p> <p>Run with something like:</p> <p><tt>export LD_PRELOAD=`pwd`/libgetcwdHack.so</tt></p> <p>The retry mechanism has to be applied to Intel/SGI MPI code (libmpifort.so). The string "forrtl" is included in it and it is the only runtime library which calls getcwd(). So only MPI Fortran code is affected.</p> <p>openmpi and mvapich2 are not affected (nothing calls getcwd() there).</p> <p>The kernel maintains a dentry lookup hash list. If getcwd() returns a dentry which is not in that list, then -ENOENT is returned. This is the case at initialization time of <tt>struct hlist_bl_node</tt> and when it has been deleted from that list. This works pretty well with local file systems.</p> <p>Can someone explain how Lustre handles the local dentry lookup hash list?<br/> Is it rebuilt from time to time?<br/> Is it possible that getcwd() can be called before the dentry is added to the list?</p> <p>Since kernel 3.12, getcwd() performance has been improved a lot by the way. Spin locks have been replaced with RCU.</p> <p>Sebastien:<br/> The way Lustre handles the local dentry lookup hash is the same as any other local filesystem. It is not possible for getcwd to be called before teh entry was added to the list because the lookup places it there.</p> <p>The problem at hand is the entry could be removed from the list between lookup and getcwd. On normal filesystems that's possible if the entry was unlinked meanwhile. On Lustre (and to a degree on NFS) additional way to get the entry dropped from the list is because the entry became stale (i.e. somebody attempted a conflicting operation that revoked the lock of that entry so it was invalidated). Since getcwd does not revalidate entries it might return false errors in those cases.</p> <p>I've talked to our NFS developer Neil Brown meanwhile. getcwd() shouldn't need to revalidate the dentry. NFS code is not affected - confirmed by our customer. Also upstream Lustre code doesn't seem to be affected - Neil checked the source.<br/> If you can provide us your affected Lustre source, then he offers to have a look at it.</p> <p>I think I've found the Lustre 2.9.0 bug for SLES12-SP2. ll_set_fs_pwd() only supports the fs-&gt;lock spin lock or the fs-&gt;lock write lock but not the RCU locking done by getcwd() in kernel 4.4.</p> <p>Bobi Jam (bobijam@hotmail.com) uploaded a new patch: <a href="https://review.whamcloud.com/28907" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28907</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9735" title="Sles12Sp2 and 2.9 getcwd() sometimes fails" class="issue-link" data-issue-key="LU-9735"><del>LU-9735</del></a> compat: heed the fs_struct::seq<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 898c57e49743e5bf57b5fb689484c71fc909bb61</p> <p>Would you try this patch?</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=sparschauer" class="user-hover" rel="sparschauer">sparschauer</a>, I also saw the following email on the linux-fsdevel mailing list from Neil Brown:</p> <blockquote> <p>Subject: Re: <span class="error">&#91;PATCH&#93;</span> d_move() vs d_unhashed() race: retry under d_lock</p> <p>On Fri, Sep 08 2017, Goldwyn Rodrigues wrote:<br/> &gt;<br/> &gt; This is a follow-up of Alexey's patch at <a href="https://patchwork.kernel.org/patch/9455345/" class="external-link" target="_blank" rel="nofollow noopener">https://patchwork.kernel.org/patch/9455345/</a><br/> &gt; with suggestions proposed by Al Viro.<br/> &gt; <br/> &gt; d_move() and d_unhashed() may race because there is a small window<br/> &gt; where the dentry is unhashed. This may result in ENOENT (for getcwd).<br/> &gt; This must be checked under d_lock. However, in order to keep the fast<br/> &gt; path, perform the d_unhashed without d_lock first, and in the unlikely<br/> &gt; event that it succeeds, perform the check again under d_lock.</p> <p>For your consideration, here is an alternate patch which - I believe -<br/> achieves the same end. I think this approach is a little more robust,<br/> but there isn't a lot in it - Goldwyn's is arguably simpler so might be<br/> better for that reason.</p> <p>NeilBrown</p></blockquote> <p>The actual patch is at <a href="https://patchwork.kernel.org/patch/9945159/" class="external-link" target="_blank" rel="nofollow noopener">https://patchwork.kernel.org/patch/9945159/</a> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>From dfaa166e2afaed051c388dc9f43d1468020b5e22 Mon Sep 17 00:00:00 2001 From: NeilBrown &lt;neilb@suse.com&gt; Date: Fri, 8 Sep 2017 16:03:42 +1000 Subject: [PATCH] VFS: close race between getcwd() and d_move() d_move() will call __d_drop() and then __d_rehash() on the dentry being moved. This creates a small window when the dentry appears to be unhashed. Many tests of d_unhashed() are made under -&gt;d_lock and so are safe from racing with this window, but some aren't. In particular, getcwd() calls d_unlinked() (which calls d_unhashed()) without d_lock protection, so it can race. This races has been seen in practice with lustre, which uses d_move() as part of name lookup. See: https://jira.hpdd.intel.com/browse/LU-9735 It could race with a regular rename(), and result in ENOENT instead of either the 'before' or 'after' name. We could fix this race by taking d_lock an rechecking when d_unhashed() reports true. Alternately when can remove the window, which is the approach this patch takes. When __d_drop and __d_rehash are used to move a dentry, an extra flag is passed which causes d_hash.pprev to not be cleared, and to not be tested. Signed-off-by: NeilBrown &lt;neilb@suse.com&gt; </pre> </div></div> <p>A fix for this problem is now shipped in the latest kernel version for sles12sp3. The description of the fix is as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>From: NeilBrown &lt;neilb@suse.com&gt; Subject: getcwd: Close race with d_move called by lustre Patch-mainline: Not yet, under development References: bsc#1052593 When lustre invalidates a dentry (e.g. do to a recalled lock) and then revalidates it, ll_splice_alias() will call d_move() to move the old alias to the name of a new one. This will d_drop then d_rehash the old dentry, creating a small window when the dentry in unhashed. If getcwd is run at this time, it might incorrectly think that the dentry really is unhashed, and so return ENOENT. This is a bug in lustre (it shouldn't call d_move()) but we can work around it in getcwd by taking the d_lock to avoid the race. First we test without the lock as the common case does not involve any race. If we find the the dentry appears to be unhashed, we take the lock and check again. Signed-off-by: Neil Brown &lt;neilb@suse.com&gt; </pre> </div></div> <p>So can we safely close this as "not a bug" from a Lustre perspective safe in the knowledge that this will be fixed in current version of SLES?</p> <p>I would vote for 'yes', but it's only fixed for new versions of sles12sp3. not fixed for anything older. Do other commenters in this ticket have opinions?</p> <p>Has there been any feedback on the other proposed fix <a href="https://review.whamcloud.com/28907" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28907</a> ?</p> <p>The patch has been submitted from SLE11-SP3-LTSS to SLE15. New kernels containing the fix will be released soon.</p> <p>What about the above comment </p> <p>"This is a bug in lustre (it shouldn't call d_move())"</p> <p>We carry #28907 in our nas-2.10.x and it fixed our problem. It probably would be 4-6 months before we can upgrade to sles12sp3.</p> <p>Would #28907 conflict with SUSE's workaround in sles12sp3?</p> <p>Neil Brown while proposed that WA thought it was a bug in lustre. I think we should have a valid fix.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/28907/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/28907/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9735" title="Sles12Sp2 and 2.9 getcwd() sometimes fails" class="issue-link" data-issue-key="LU-9735"><del>LU-9735</del></a> compat: heed the fs_struct::seq<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: fff1163fdb41190b59adb8d90919e0adf37f68fb</p> <p>Landed for 2.11</p> <p>Minh Diep (minh.diep@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/31106" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31106</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9735" title="Sles12Sp2 and 2.9 getcwd() sometimes fails" class="issue-link" data-issue-key="LU-9735"><del>LU-9735</del></a> compat: heed the fs_struct::seq<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: 6c21f76ae7ef6cb8004dd60db87583c101b50aa6</p> <p>John L. Hammond (john.hammond@intel.com) merged in patch <a href="https://review.whamcloud.com/31106/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/31106/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-9735" title="Sles12Sp2 and 2.9 getcwd() sometimes fails" class="issue-link" data-issue-key="LU-9735"><del>LU-9735</del></a> compat: heed the fs_struct::seq<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 030b15004d3acf6b98c198263fcca232129568cc</p> <p>Hi!</p> <p>As an additional datapoint, we'd like to report that we've been seeing this exact same behavior with the latest Maintenance Release (2.10.3) and the latest available CentOS 7.4 kernel</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"># uname -a Linux sh-104-39.<span class="code-object">int</span> 3.10.0-693.21.1.el7.x86_64 #1 SMP Wed Mar 7 19:03:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux # cat /sys/fs/lustre/version 2.10.3 </pre> </div></div> <p> </p> <p>Symptoms are the same stack as initially reported, and happened while running VASP jobs:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-quote">"forrtl: severe (121): Cannot access current working directory <span class="code-keyword">for</span> unit 7, file "</span>Unknown" Image PC Routine Line Source vasp_gam 000000000140B496 Unknown Unknown Unknown vasp_gam 000000000142511E Unknown Unknown Unknown vasp_gam 000000000091665F Unknown Unknown Unknown vasp_gam 0000000000CFE655 Unknown Unknown Unknown vasp_gam 00000000012AF330 Unknown Unknown Unknown vasp_gam 0000000000408D1E Unknown Unknown Unknown libc-2.17.so 00007F839E16FC05 __libc_start_main Unknown Unknown vasp_gam 0000000000408C29 Unknown Unknown Unknown</pre> </div></div> <p> </p> <p>The "try-again" workaround provided by @Nathan works great and we're recommending our users to use it for now. With the libgetcwdHack.so library LD_PRELOADed, the application generates this kind of log:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">NF: getcwd: mpi rank -1, host sh-104-39.<span class="code-object">int</span>: [190701]: failed: size 4096, buf 0x7fffd6c0959b, ret (nil): No such file or directory NF: getcwd: mpi rank -1, host sh-104-39.<span class="code-object">int</span>: [190701]: succeeded at <span class="code-keyword">try</span> 2 of 10: size 4096, buf 0x7fffd6c0959b, ret 0x7fffd6c0959b, path /scratch/users/freitas/chemical_reactions/vasp_simulations/C_fixed_V/07_restart_3 NF: getcwd: mpi rank -1, host sh-104-39.<span class="code-object">int</span>: [190695]: failed: size 4096, buf 0x7ffe51196e9b, ret (nil): No such file or directory NF: getcwd: mpi rank -1, host sh-104-39.<span class="code-object">int</span>: [190695]: succeeded at <span class="code-keyword">try</span> 2 of 10: size 4096, buf 0x7ffe51196e9b, ret 0x7ffe51196e9b, path /scratch/users/freitas/chemical_reactions/vasp_simulations/C_fixed_V/07_restart_3 NF: getcwd: mpi rank -1, host sh-104-39.<span class="code-object">int</span>: [190699]: failed: size 4096, buf 0x7ffdb3f1f31b, ret (nil): No such file or directory NF: getcwd: mpi rank -1, host sh-104-39.<span class="code-object">int</span>: [190699]: succeeded at <span class="code-keyword">try</span> 2 of 10: size 4096, buf 0x7ffdb3f1f31b, ret 0x7ffdb3f1f31b, path /scratch/users/freitas/chemical_reactions/vasp_simulations/C_fixed_V/07_restart_3 NF: getcwd: mpi rank -1, host sh-104-39.<span class="code-object">int</span>: [190697]: failed: size 4096, buf 0x7ffe9713df9b, ret (nil): No such file or directory NF: getcwd: mpi rank -1, host sh-104-39.<span class="code-object">int</span>: [190697]: succeeded at <span class="code-keyword">try</span> 2 of 10: size 4096, buf 0x7ffe9713df9b, ret 0x7ffe9713df9b, path /scratch/users/freitas/chemical_reactions/vasp_simulations/C_fixed_V/07_restart_3 NF: getcwd: mpi rank -1, host sh-104-39.<span class="code-object">int</span>: [190695]: failed: size 4096, buf 0x7ffe51196e9b, ret (nil): No such file or directory NF: getcwd: mpi rank -1, host sh-104-39.<span class="code-object">int</span>: [190695]: succeeded at <span class="code-keyword">try</span> 2 of 10: size 4096, buf 0x7ffe51196e9b, ret 0x7ffe51196e9b, path /scratch/users/freitas/chemical_reactions/vasp_simulations/C_fixed_V/07_restart_3 </pre> </div></div> <p>which seem very indicative of the same error.</p> <p>We're looking forward to the fix in 2.10.4.</p> <p>Cheers,<br/> -- <br/> Kilian</p> <p>Hi All,</p> <p> </p> <p>I did not see any reaction to Mahmoud Hanafi 's comment from the 26th October 2017, citing Neil Brown:</p> <p>"This is a bug in lustre (it shouldn't call d_move())"</p> <p> </p> <p>We have installed 2.10.4 at a customer's site which encountered this problem (with RHEL 7 clients), and even though this considerably decreased the number of occurences, we still see the "small window when dentry is unhashed", making the job fail.</p> <p>Is there something that can be done in ll_splice_alias() to close this race ?</p> <p>I understand this is closed by SuSE in their latest kernels, but this is not the case for earlier kernels, nor for RHEL kernels.</p> <p>Or should we push Red Hat to apply the same kind of patch SuSE did (does not seem really fair to me) ?</p> <p>It looks like we hit the same issue on Lustre 2.10.5 client and Centos 7.5 kernel. Should the fix come from Lustre or kernel, as I'm confused by the previous discussion.</p> <p>I wonder if the fixes from <a href="https://jira.whamcloud.com/browse/LU-9868" title="dcache/namei fixes for lustre" class="issue-link" data-issue-key="LU-9868">LU-9868</a> would fix this? Note the patch post has a bug in it. I have a fix but haven't pushed it.</p> <p>Can you give  <a href="https://review.whamcloud.com/#/c/28486" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/28486</a> a try.</p> <p>As reported the earlier patch for this bug didn't completely solve the problem. The work from <a href="https://jira.whamcloud.com/browse/LU-9868" title="dcache/namei fixes for lustre" class="issue-link" data-issue-key="LU-9868">LU-9868</a> has been reported as solving this problem which is now linked to this ticket.</p> <p>So it appears that the patch for <a href="https://jira.whamcloud.com/browse/LU-9868" title="dcache/namei fixes for lustre" class="issue-link" data-issue-key="LU-9868">LU-9868</a> while fixing this bug has exposed another potential bug in lustre. If you run sanity test 233 you see</p> <p><span class="error">&#91;37212.956888&#93;</span> VFS: Lookup of '<span class="error">&#91;0x200000007:0x1:0x0&#93;</span>' in lustre lustre would have caused loop</p> <p><span class="error">&#91;37217.817624&#93;</span> Lustre: DEBUG MARKER: sanity test_233a: @@@@@@ FAIL: cannot access /lustre/lustre using its FID '<span class="error">&#91;0x200000007:0x1:0x0&#93;</span>'</p> <p><span class="error">&#91;37236.855424&#93;</span> Lustre: DEBUG MARKER: == sanity test 233b: checking that OBF of the FS .lustre succeeds ==================================== 03:34:33 (1538379273)</p> <p><span class="error">&#91;37238.362201&#93;</span> VFS: Lookup of '<span class="error">&#91;0x200000002:0x1:0x0&#93;</span>' in lustre lustre would have caused loop</p> <p><span class="error">&#91;37243.442480&#93;</span> Lustre: DEBUG MARKER: sanity test_233b: @@@@@@ FAIL: cannot access /lustre/lustre/.lustre using its FID '<span class="error">&#91;0x200000002:0x1:0x0&#93;</span>'</p> <p>Some how the parent child relationship got inverted. Will investigate.</p> <p>We can close this case.</p> <p>ok so, given that the initial fix seems to satisfy NASA (the original reporter) we can close the ticket. <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=simmonsja" class="user-hover" rel="simmonsja">simmonsja</a> can you track any remaining work under a new ticket?</p> <p>I already have another ticket for this <img class="emoticon" src="https://jira.whamcloud.com/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> Duplicate LU-9208 Related LU-9868 LU-10164 Development Rank 1|hzzg2v: Rank (Obsolete) 9223372036854775807 Severity