https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-16589 Lustre <p>This issue was created by maloo for jianyu &lt;yujian@whamcloud.com&gt;</p> <p>This issue relates to the following test suite run: <a href="https://testing.whamcloud.com/test_sets/5fae0359-520f-440f-9515-09d401c4ae9c" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/5fae0359-520f-440f-9515-09d401c4ae9c</a></p> <p>test_55d failed with the following error:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanityn test 55d: rename file vs link ================= 17:25:02 (1677173102) CMD: onyx-65vm4 /usr/sbin/lctl set_param fail_loc=0x155 fail_loc=0x155 ln: failed to create hard link '/mnt/lustre2/d55d.sanityn/d55d.sanityn/' =&gt; '/mnt/lustre2/d55d.sanityn/f1': No such file or directory sanityn test_55d: @@@@@@ FAIL: (2) mv succeeded </pre> </div></div> <p>Test session details:<br/> clients: <a href="https://build.whamcloud.com/job/lustre-master/4392" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-master/4392</a> - 5.14.21-150400.24.28-default<br/> servers: <a href="https://build.whamcloud.com/job/lustre-master/4392" class="external-link" target="_blank" rel="nofollow noopener">https://build.whamcloud.com/job/lustre-master/4392</a> - 4.18.0-425.10.1.el8_lustre.x86_64</p> <p>&lt;&lt;Please provide additional information about the failure here&gt;&gt;</p> <p>VVVVVVV DO NOT REMOVE LINES BELOW, Added by Maloo for auto-association VVVVVVV<br/> sanityn test_55d - (2) mv succeeded</p> LU-16589

sanityn test_55d: FAIL: (2) mv succeeded

Bug Minor Resolved Fixed Jian Yu Maloo Fri, 24 Feb 2023 00:19:45 +0000 Wed, 19 Apr 2023 03:37:43 +0000 Wed, 19 Apr 2023 03:37:43 +0000 Lustre 2.16.0 Lustre 2.15.3 Lustre 2.16.0 Lustre 2.15.3 0 11 <p>Besides SLES 15 SP4 client, the same failure also occurred on SLES 15 SP3, RHEL 9.0 and RHEL 9.1 clients on master branch.</p> <p>The ln command in coreutils-8.22 passed on RHEL 7.9 client and the strace outputs were:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>1677185837.907658 stat("/mnt/lustre2/d55d.sanityn/d55d.sanityn/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 1677185837.908877 lstat("/mnt/lustre2/d55d.sanityn/f1", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 1677185837.909992 linkat(AT_FDCWD, "/mnt/lustre2/d55d.sanityn/f1", AT_FDCWD, "/mnt/lustre2/d55d.sanityn/d55d.sanityn/f1", 0) = 0 </pre> </div></div> <p>While the ln command in coreutils-8.32 failed on SLES 15 SP4 client with strace outputs as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>1677175127.556014 linkat(AT_FDCWD, "/mnt/lustre2/d55d.sanityn/f1", AT_FDCWD, "/mnt/lustre2/d55d.sanityn/d55d.sanityn/", 0) = -1 ENOENT (No such file or directory) 1677175127.557279 newfstatat(AT_FDCWD, "/mnt/lustre2/d55d.sanityn/f1", {st_mode=S_IFREG|0644, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0 1677175127.558307 write(2, "ln: ", 4) = 4 1677175127.558492 write(2, "failed to create hard link '/mnt"..., 102) = 102 1677175127.558945 write(2, ": No such file or directory", 27) = 27 1677175127.559092 write(2, "\n", 1) = 1 </pre> </div></div> <p>The stat() and lstat() operations were removed from ln by the following commit in coreutils-8.30:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit 571f63f5010b047a8a3250304053f05949faded4 Author: Paul Eggert &lt;eggert@cs.ucla.edu&gt; AuthorDate: Fri Oct 19 12:19:43 2018 -0700 Commit: Paul Eggert &lt;eggert@cs.ucla.edu&gt; CommitDate: Fri Oct 19 12:38:34 2018 -0700 ln: avoid directory hard-link races Previously, 'ln A B' did 'stat("B"), lstat("A"), link("A","B")' where the stat and lstat were necessary to avoid hard-linking directories on systems that can hard-link directories. Now, in situations that prohibit hard links to directories, 'ln A B' merely does 'link("A","B")'. The new behavior avoids some races and should be more efficient. This patch was inspired by Bug#10020, which was about 'ln'. * bootstrap.conf (gnulib_modules): Add unlinkdir. * src/force-link.c (force_linkat, force_symlinkat): New arg for error number of previous try. Return error number, 0, or -1 if error, success, or success after removal. All callers changed. * src/ln.c: Include priv-set.h, unlinkdir.h. (beware_hard_dir_link): New static var. (errnoize, atomic_link): New functions. (target_directory_operand): Use errnoize for simplicity. (do_link): New arg for error number of previous try. All callers changed. Do each link atomically if possible. (main): Do -r check earlier. Remove linkdir privileges so we can use a single linkat/symlinkat instead of a racy substitute for the common case of 'ln A B' and 'ln -s A B'. Set beware_hard_dir_link to disable this optimization. </pre> </div></div> <p>I'm creating a patch to add ls command before running ln in sanityn/55d to trigger stat()/statx().</p> <p>"Jian Yu &lt;yujian@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50127" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50127</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16589" title="sanityn test_55d: FAIL: (2) mv succeeded" class="issue-link" data-issue-key="LU-16589"><del>LU-16589</del></a> tests: fix dir hard-link failure in sanityn/55d<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 7f5977ab6e1a6f0c81ad74121350d2872da813e0</p> <p>I found without adding ls command before ln, just removing the trailing slash from "$tdir/" can also make the ln command succeed:</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>sanityn/55d</b></div><div class="codeContent panelContent"> <pre class="code-java"> # link in reverse locking order - ln $DIR2/$tdir/f1 $DIR2/$tdir/$tdir/ + ln $DIR2/$tdir/f1 $DIR2/$tdir/$tdir </pre> </div></div> <p>$DIR2/$tdir/$tdir/f1 was created as a hard link to file $DIR2/$tdir/f1.</p> <p> </p> <p>The failure can be simply reproduced as follows:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># touch /mnt/lustre/tfile # mkdir /mnt/lustre/tdir # ln /mnt/lustre/tfile /mnt/lustre/tdir/ ln: failed to create hard link '/mnt/lustre/tdir/' =&gt; '/mnt/lustre/tfile': No such file or directory # ln /mnt/lustre/tfile /mnt/lustre/tdir # ls /mnt/lustre/tdir tfile </pre> </div></div> <p>The strace outputs of "ln /mnt/lustre/tfile /mnt/lustre/tdir/" (with a trailing slash) are:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>1677747428.112785 linkat(AT_FDCWD, "/mnt/lustre/tfile", AT_FDCWD, "/mnt/lustre/tdir/", 0) = -1 ENOENT (No such file or directory) 1677747428.113910 newfstatat(AT_FDCWD, "/mnt/lustre/tfile", {st_mode=S_IFREG|0644, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0 &lt;~snip~&gt; 1677747428.116781 write(2, "ln: ", 4) = 4 1677747428.116922 write(2, "failed to create hard link '/mnt"..., 69) = 69 &lt;~snip~&gt; 1677747428.117833 write(2, ": No such file or directory", 27) = 27 1677747428.117941 write(2, "\n", 1) = 1 </pre> </div></div> <p>The strace outputs of "ln /mnt/lustre/tfile /mnt/lustre/tdir" (without a trailing slash) are:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>1677747364.665989 linkat(AT_FDCWD, "/mnt/lustre/tfile", AT_FDCWD, "/mnt/lustre/tdir", 0) = -1 EEXIST (File exists) 1677747364.668271 openat(AT_FDCWD, "/mnt/lustre/tdir", O_RDONLY|O_PATH|O_DIRECTORY) = 3 1677747364.670377 linkat(AT_FDCWD, "/mnt/lustre/tfile", 3, "tfile", 0) = 0 </pre> </div></div> <p>The difference in behavior of "<tt>ln</tt>" and the success/failure of the test appears to boil down to the return code of <tt>linkat()</tt> with/without the trailing "<tt>/</tt>".</p> <p>So this looks like a dcache lookup issue in Lustre, or possibly something slightly with how <tt>linkat()</tt> is being handled by the kernel or Lustre (e.g. returning "<tt>-ENOENT</tt>" instead of "<tt>-EEXIST</tt>")? The presence of the trailing "<tt>/</tt>" <em>shouldn't</em> make any difference to the kernel (AFAIK it should strip out "<tt>/</tt>" during lookup?) but possibly something strange happening during path processing.</p> <p>Jian, could you please capture the Lustre kernel <tt>debug=all</tt> logs for the two test cases, and check how the two <tt>linkat()</tt> calls are different in Lustre. Probably only the time from the start to the end of the <tt>linkat()</tt> syscall is interesting, and where <tt>ENOENT</tt> vs. <tt>EEXIST</tt> are being generated.</p> <p>Sure, Andreas. I have gathered the full debug logs and been looking at them.</p> <p>In Lustre debug logs, with the trailing "<tt>/</tt>", there was no "<tt>-ENOENT</tt>" error. <tt>ll_link()</tt> from client and <tt>mdt_reint_link()</tt> from MDS were not called.<br/> Without the trailing "<tt>/</tt>", <tt>ll_link()</tt> was called from client, and <tt>mdt_reint_link()</tt> on MDS returned "<tt>-EEXIST</tt>":</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000004:00000040:0.0:1677747185.111782:0:22177:0:(mdt_reint.c:1456:mdt_reint_link()) link target tdir existed! 00000004:00000001:0.0:1677747185.111783:0:22177:0:(mdt_reint.c:1457:mdt_reint_link()) Process leaving via unlock_source (rc=18446744073709551599 : -17 : 0xffffffffffffffef) </pre> </div></div> <p>I'm looking into <tt>linkat()</tt> to see where "<tt>ENOENT</tt>" is generated.<br/>  </p> <p>In <tt>linkat()&#45;&gt;__file_name_split_at()-&gt;&#95;_hurd_file_name_split()</tt>:</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>__hurd_file_name_split()</b></div><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">const</span> <span class="code-object">char</span> *lastslash = strrchr (file_name, <span class="code-quote">'/'</span>); <span class="code-keyword">if</span> (lastslash != NULL) { <span class="code-keyword">if</span> (lastslash == file_name) { <span class="code-comment">/* <span class="code-quote">"/foobar"</span> =&gt; crdir + <span class="code-quote">"foobar"</span>. */</span> *name = (<span class="code-object">char</span> *) file_name + 1; <span class="code-keyword">return</span> (*use_init_port) (INIT_PORT_CRDIR, &amp;addref); } <span class="code-keyword">else</span> { <span class="code-comment">/* <span class="code-quote">"/dir1/dir2/.../file"</span>. */</span> <span class="code-object">char</span> dirname[lastslash - file_name + 1]; memcpy (dirname, file_name, lastslash - file_name); dirname[lastslash - file_name] = <span class="code-quote">'\0'</span>; *name = (<span class="code-object">char</span> *) lastslash + 1; <span class="code-keyword">return</span> __hurd_file_name_lookup (use_init_port, get_dtable_port, lookup, dirname, 0, 0, dir); } } <span class="code-keyword">else</span> <span class="code-keyword">if</span> (file_name[0] == <span class="code-quote">'\0'</span>) <span class="code-keyword">return</span> ENOENT; <span class="code-keyword">else</span> { <span class="code-comment">/* <span class="code-quote">"foobar"</span> =&gt; cwdir + <span class="code-quote">"foobar"</span>. */</span> *name = (<span class="code-object">char</span> *) file_name; <span class="code-keyword">return</span> (*use_init_port) (INIT_PORT_CWDIR, &amp;addref); } </pre> </div></div> <p>And in <tt>__hurd_file_name_lookup</tt>:</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>__hurd_file_name_lookup()</b></div><div class="codeContent panelContent"> <pre class="code-java"> /* The caller wants to require that the file we look up is a directory. We can <span class="code-keyword">do</span> <span class="code-keyword">this</span> without an extra RPC by appending a trailing slash to the file name we look up. */ size_t len = strlen (file_name); <span class="code-keyword">if</span> (len == 0) file_name = <span class="code-quote">"/"</span>; <span class="code-keyword">else</span> <span class="code-keyword">if</span> (file_name[len - 1] != <span class="code-quote">'/'</span>) { <span class="code-object">char</span> *n = alloca (len + 2); memcpy (n, file_name, len); n[len] = <span class="code-quote">'/'</span>; n[len + 1] = <span class="code-quote">'\0'</span>; file_name = n; } </pre> </div></div> <p>I haven't found out anything is wrong here.</p> <p>What's the result of this command on local filesystem? e.g. xfs?</p> <blockquote><p>What's the result of this command on local filesystem? e.g. xfs?</p></blockquote> <p>It works on local ext4 filesystem:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># df -T /root/ Filesystem Type 1K-blocks Used Available Use% Mounted on /dev/vda2 ext4 20466256 5675620 13725676 30% / # touch /root/tfile # mkdir /root/tdir # ln /root/tfile /root/tdir/ # ls /root/tdir/ tfile </pre> </div></div> <p>It turned out the <tt>ENOENT</tt> error was returned from kernel <tt>do_linkat()-&gt;filename_create()</tt>:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># rm -rf /mnt/lustre/* # touch /mnt/lustre/tfile # mkdir /mnt/lustre/tdir # trace-cmd record -g do_linkat -p function_graph ln /mnt/lustre/tfile /mnt/lustre/tdir/ plugin 'function_graph' ln: failed to create hard link '/mnt/lustre/tdir/' =&gt; '/mnt/lustre/tfile': No such file or directory CPU0 data recorded at offset=0x177000 0 bytes in size (0 uncompressed) CPU1 data recorded at offset=0x177000 349 bytes in size (4096 uncompressed) # trace-cmd report cpus=2 ln-31499 [001] 2019833.457879: funcgraph_entry: | do_linkat() { ln-31499 [001] 2019833.457888: funcgraph_entry: 0.449 us | irq_enter_rcu(); ln-31499 [001] 2019833.457889: funcgraph_entry: 8.943 us | __sysvec_irq_work(); ln-31499 [001] 2019833.457898: funcgraph_entry: 0.368 us | irq_exit_rcu(); ln-31499 [001] 2019833.457899: funcgraph_entry: # 1459.949 us | user_path_at_empty(); ln-31499 [001] 2019833.459366: funcgraph_entry: 0.340 us | irq_enter_rcu(); ln-31499 [001] 2019833.459366: funcgraph_entry: 0.834 us | __sysvec_irq_work(); ln-31499 [001] 2019833.459367: funcgraph_entry: 0.304 us | irq_exit_rcu(); ln-31499 [001] 2019833.459368: funcgraph_entry: 0.769 us | getname_flags(); ln-31499 [001] 2019833.459369: funcgraph_entry: ! 297.289 us | filename_create(); ln-31499 [001] 2019833.459667: funcgraph_entry: 4.795 us | dput(); ln-31499 [001] 2019833.459672: funcgraph_entry: 0.343 us | mntput(); ln-31499 [001] 2019833.459673: funcgraph_exit: # 1794.900 us | } </pre> </div></div> <p>By comparing the attached Ftrace outputs (with and without the trailing "<tt>/</tt>", <tt>&#45;-max-graph-depth</tt> is 6), and the source code of <tt>filename_create()</tt> in kernel <tt>fs/namei.c</tt>, we can see the <tt>ENOENT</tt> error was returned as follows:</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>fs/namei.c</b></div><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">static</span> struct dentry *filename_create(<span class="code-object">int</span> dfd, struct filename *name, struct path *path, unsigned <span class="code-object">int</span> lookup_flags) { <span class="code-comment">// ...... </span> /* * Special <span class="code-keyword">case</span> - lookup gave negative, but... we had foo/bar/ * From the vfs_mknod() POV we just have a negative dentry - * all is fine. Let<span class="code-quote">'s be bastards - you had / on the end, you'</span>ve * been asking <span class="code-keyword">for</span> (non-existent) directory. -ENOENT <span class="code-keyword">for</span> you. */ <span class="code-keyword">if</span> (unlikely(!is_dir &amp;&amp; last.name[last.len])) { error = -ENOENT; <span class="code-keyword">goto</span> fail; } <span class="code-comment">// ...... </span>} </pre> </div></div> <p> <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/48354/48354_ftrace.with_slash.depth_6.txt" title="ftrace.with_slash.depth_6.txt attached to LU-16589">ftrace.with_slash.depth_6.txt^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/48355/48355_ftrace.without_slash.depth_6.txt" title="ftrace.without_slash.depth_6.txt attached to LU-16589">ftrace.without_slash.depth_6.txt^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> </p> <p>Jian, looking at what test_55d is doing, it is unclear why the <tt>linkat()-&gt;filename_create()</tt> operation is returning <tt>-ENOENT</tt>? The directory <b>does</b> exist, since it was created with <tt>mkdir</tt> on the previous line, so <em>either</em> the <tt>mkdir $DIR/$tdir/$tdir</tt> operation should have instantiated this directory into dcache, or the <tt>linkat()</tt> syscall <b>should</b> have looked up the second <tt>$tdir</tt> component during traversal.</p> <p>It would be interesting to see what the test does with "<tt>ln $DIR2/$tdir/f1 $DIR2/$tdir/$tdir/f1</tt>" (i.e. with the target filename specified)? That is really what the "<tt>ln</tt>" command is trying to do - link <tt>f1</tt> into the second <tt>$tdir</tt> directory so that the delayed <tt>mv $DIR2/$tdir/f1 $DIR2/$tdir/$tdir</tt> that was trying to create a <b>file</b> named <tt>$tdir</tt> cannot succeed when there is later a <b>directory</b> named <tt>$tdir</tt>.</p> <p>Even if that allows the test to pass (which would be slightly better than the "pre-stat"), there is still a dcache bug in there somewhere, because the kernel lookup of the second <tt>$tdir/</tt> should not have failed.</p> <p>What else is interesting is that newer kernels (since v5.18-rc2-188-gb3d4650d82c7) have slightly different code here:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">static</span> struct dentry *filename_create(<span class="code-object">int</span> dfd, struct filename *name, struct path *path, unsigned <span class="code-object">int</span> lookup_flags) { unsigned <span class="code-object">int</span> create_flags = LOOKUP_CREATE | LOOKUP_EXCL; : : /* * Special <span class="code-keyword">case</span> - lookup gave negative, but... we had foo/bar/ * From the vfs_mknod() POV we just have a negative dentry - * all is fine. Let<span class="code-quote">'s be bastards - you had / on the end, you'</span>ve * been asking <span class="code-keyword">for</span> (non-existent) directory. -ENOENT <span class="code-keyword">for</span> you. */ <span class="code-keyword">if</span> (unlikely(!create_flags)) { error = -ENOENT; <span class="code-keyword">goto</span> fail; } </pre> </div></div> <p>so it would be interesting to know what <tt>create_flags</tt> are being passed from Lustre?</p> <p>It just happens that the commit v5.18-rc2-188-gb3d4650d82c7 that changed this code was written by Neil Brown, whom I've CC'd here in case he has some insight into what is going wrong here. It definitely seems from the commit message that this is a similar situation being hit in Lustre as was previously hit by NFS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> VFS: filename_create(): fix incorrect intent. When asked to create a path ending '/', but which is not to be a directory (LOOKUP_DIRECTORY not set), filename_create() will never try to create the file. If it doesn't exist, -ENOENT is reported. However, it still passes LOOKUP_CREATE|LOOKUP_EXCL to the filesystems -&gt;lookup() function, even though there is no intent to create. This is misleading and can cause incorrect behaviour. If you try ln -s foo /path/dir/ where 'dir' is a directory on an NFS filesystem which is not currently known in the dcache, this will fail with ENOENT. But as the name is not in the dcache, nfs_lookup gets called with LOOKUP_CREATE|LOOKUP_EXCL and so it returns NULL without performing any lookup, with the expectation that a subsequent call to create the target will be made, and the lookup can be combined with the creation. In the case with a trailing '/' and no LOOKUP_DIRECTORY, that call is never made. Instead filename_create() sees that the dentry is not (yet) positive and returns -ENOENT - even though the directory actually exists. So only set LOOKUP_CREATE|LOOKUP_EXCL if there really is an intent to create, and use the absence of these flags to decide if -ENOENT should be returned. Note that filename_parentat() is only interested in LOOKUP_REVAL, so we split that out and store it in 'reval_flag'. __lookup_hash() then gets reval_flag combined with whatever create flags were determined to be needed. </pre> </div></div> <p>It isn't clear whether we can add a workaround in the Lustre <tt>-&gt;lookup()</tt> method to handle this case or only fix the test (to add the <tt>/f1</tt> component at the end) and get the client distros to apply Neil's patch. It looks like both SLES15sp4 and RHEL9.x are using a 5.14-based kernel, and are missing the v5.18-rc2-188-gb3d4650d82c7 patch, or we are somehow not handling the <tt>LOOKUP_&#42;</tt> flags properly in Lustre. The changed code may have been affected by v5.14-rc7-68-g0ee50b47532a "<tt>namei: change filename_parentat() calling conventions</tt>", so it is possible we didn't update to those new conventions?</p> <p>Hi Andreas,</p> <p> I think it likely that you have identified the problem.  Especially if it is possible that the "rename" happening in the background might have called d_lustre_invalidate() on the original dentry.</p> <p>I think it unlikely that the filename_parentat() change is relevant.  That is an internal api in namei.c</p> <p>The only way I can think of to work around the problem in lustre is to skip the optimisation in ll_lookup_nd() for a CREATE that isn't an OPEN.  You might be able to detect this particular case by seeing there is an invalid dentry still present.  Maybe.</p> <p>I really should apply that patch to SP4.  We have it in SP3.</p> <p> </p> <p>Thank you Andreas and Neil for the detailed analysis.</p> <blockquote><p>It would be interesting to see what the test does with "ln $DIR2/$tdir/f1 $DIR2/$tdir/$tdir/f1" (i.e. with the target filename specified)?</p></blockquote> <p>Test passed on SLES 15 SP4 client (with kernel 5.14.21-150400.24.28-default):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>== sanityn test 55d: rename file vs link ============================================================= 05:52:19 (1678254739) CMD: trevis-86vm8 /usr/sbin/lctl set_param fail_loc=0x155 fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file Resetting fail_loc on all nodes...CMD: trevis-86vm7.trevis.whamcloud.com,trevis-86vm8,trevis-87vm7 lctl set_param -n fail_loc=0 fail_val=0 2&gt;/dev/null done. CMD: trevis-86vm7.trevis.whamcloud.com /usr/sbin/lctl get_param catastrophe 2&gt;&amp;1 CMD: trevis-86vm8 /usr/sbin/lctl get_param catastrophe 2&gt;&amp;1 CMD: trevis-87vm7 /usr/sbin/lctl get_param catastrophe 2&gt;&amp;1 CMD: trevis-86vm7.trevis.whamcloud.com,trevis-86vm8,trevis-87vm7 dmesg PASS 55d (9s) </pre> </div></div> <blockquote><p>The only way I can think of to work around the problem in lustre is to skip the optimization in ll_lookup_nd() for a CREATE that isn't an OPEN. </p></blockquote> <p>The optimization was added by <a href="https://review.whamcloud.com/8257" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/8257</a> ("<a href="https://jira.whamcloud.com/browse/LU-4185" title="Incorrect permission handling when creating existing directories at ICHEC" class="issue-link" data-issue-key="LU-4185"><del>LU-4185</del></a> llite: Revise create with no open optimization"):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>commit a2d5b2e83c0a512a3ea59698e8481621ab5856c2 Author: Bobi Jam &lt;bobijam@whamcloud.com&gt; AuthorDate: Wed Nov 13 15:56:25 2013 +0800 Commit: Oleg Drokin &lt;green@whamcloud.com&gt; CommitDate: Wed Oct 5 03:51:18 2016 +0000 LU-4185 llite: Revise create with no open optimization Currently ll_lookup_nd just returns a negative when we are trying to create something with no open (read mkdir). This is all fine most of the cases, except if the directory where we are trying to do is not writeable by us. In that case vfs_create would return EPERM seeing as how a negative dentry means the create cannot proceed. But in reality if there is an existing name there that we just did not have cached, the proper return is EEXIST that could only happen if we did do the lookup. So amend the optimization to only take place if the directory is writeable by us, otherwise do the full lookup. </pre> </div></div> <p>Would it work to change ll_lookup_nd() to skip the optimization if flags contains LOOKUP_CREATE | LOOKUP_EXCL and LOOKUP_CREATE + !LOOKUP_OPEN + !LOOKUP_EXCL still use the optimization?</p> <p>This workaround should go under an #ifdef so that the optimization is only disabled for kernels that are affected by it. It definitely is not needed for kernels newer than 5.18, but I'm not totally sure what is the oldest version affected, or if there is a #define we can check that was introduced in the original patch. This is also complicated by vendor backports, so there may need to be both a vanilla kernel version check and a vendor version check. </p> <p>&gt; Would it work to change ll_lookup_nd() to skip the optimization if flags contains LOOKUP_CREATE | LOOKUP_EXCL and LOOKUP_CREATE + !LOOKUP_OPEN + !LOOKUP_EXCL still use the optimization?</p> <p>I don't think -&gt;lookup is ever called with LOOKUP_CREATE but not LOOKUP_EXCL.  And if it is, it is every likely to include LOOKUP_OPEN.  Most syscalls that create names give an error (EEXIST) if the name exists.  The only exception is open(O_CREATE).</p> <p>Do we really need to fix this is lustre?  It is clearly not a lustre bug.  If we want test_55d to succeed more reliably we can just change the ln command to "<tt>ln $DIR2/$tdir/f1 $DIR2/$tdir/$tdir/f1</tt>" as Andreas suggested.  That performs exactly the same effective test, but avoids the kernel bug.</p> <p> </p> <blockquote><p>I'm not totally sure what is the oldest version affected</p></blockquote> <p>Even on RHEL 7.9 (with kernel 3.10.0-1160), after changing <tt>ln</tt> from version 8.22 to 8.32, the "<tt>ln /mnt/lustre/tfile /mnt/lustre/tdir/</tt>" command also failed with <tt>-ENOENT</tt>.<br/> This means the kernel issue exists in all of the vendor kernels we support now.</p> <p>Jian, then it seems this bug has existed for so long and could even be considered an issue in the new coreutils? At least it is fixed in newer kernels, and it seems like a very uncommon use case, so it isn't clear that putting a ton of effort into fixing it in Lustre is worthwhile. Neil can work on backporting the fix to SLES15 (if needed), and James or someone can file a ticket to fix it in RHEL8/9.</p> <p>It seems the right thing for now is to fix the test (either add "/f1" at the end or remove the trailing "/") so that it is executing the original intent of the patch that added it.</p> <p>A separate patch should be made with a new subtest for only this "ln" case. This new subtest should be skipped for new coreutils and old kernels:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> local ln_ver=$(ln --version | awk <span class="code-quote">'/coreutils/ { print $4 }'</span>) (( $(version_code $ln_ver) &lt; $(version code 8.32) )) || (( $(version_code $(uname -r)) &gt;= $(version_code 5.18) )) || skip <span class="code-quote">"need coreutils &lt; 8.32 or kernel &gt;= 5.18 <span class="code-keyword">for</span> ln"</span> touch $DIR/$tfile || error <span class="code-quote">"create failed"</span> mkdir $DIR/$tdir || error <span class="code-quote">"mkdir failed"</span> ln $DIR/$tfile $DIR/tdir/ || error <span class="code-quote">"ln to <span class="code-quote">'$tdir/'</span> failed"</span> </pre> </div></div> <p>This will at least add test coverage for newer kernels and keep it for old coreutils, so that it doesn't regress in the future.  Additional checks can be added to allow running the test when we know particular distro kernels are fixed. </p> <p>Sure, Andreas. I just updated <a href="https://review.whamcloud.com/50127" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/50127</a> to fix sanityn/55d. I'm working on a separate patch to add a new subtest for "<tt>ln</tt>". BTW, I just found the actual version of coreutils that started containing the "<tt>ln</tt>" change is 8.31.</p> <p>"Jian Yu &lt;yujian@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50265" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50265</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16589" title="sanityn test_55d: FAIL: (2) mv succeeded" class="issue-link" data-issue-key="LU-16589"><del>LU-16589</del></a> tests: add sanity/31l to test ln command<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 8b6e20e63be8edc4fda22f2318fa8df350c7297b</p> <p>"Jian Yu &lt;yujian@whamcloud.com&gt;" uploaded a new patch: <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50348" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50348</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16589" title="sanityn test_55d: FAIL: (2) mv succeeded" class="issue-link" data-issue-key="LU-16589"><del>LU-16589</del></a> tests: fix hard-link failure in sanityn/55d<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: 1<br/> Commit: 5b3526d01808b0dd270ae0655918fa3b7fc4f941</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50127/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50127/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16589" title="sanityn test_55d: FAIL: (2) mv succeeded" class="issue-link" data-issue-key="LU-16589"><del>LU-16589</del></a> tests: fix hard-link failure in sanityn/55d<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: 25c6b7ad2859729197c3cc6e6dcf0621e4bda6fa</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50265/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50265/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16589" title="sanityn test_55d: FAIL: (2) mv succeeded" class="issue-link" data-issue-key="LU-16589"><del>LU-16589</del></a> tests: add sanity/31l to test ln command<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e998d21caf99e32495950219e88dd9e7f981363e</p> <p>"Oleg Drokin &lt;green@whamcloud.com&gt;" merged in patch <a href="https://review.whamcloud.com/c/fs/lustre-release/+/50348/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/c/fs/lustre-release/+/50348/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-16589" title="sanityn test_55d: FAIL: (2) mv succeeded" class="issue-link" data-issue-key="LU-16589"><del>LU-16589</del></a> tests: fix hard-link failure in sanityn/55d<br/> Project: fs/lustre-release<br/> Branch: b2_15<br/> Current Patch Set: <br/> Commit: b01209416cb73e06d50a2bf00855e56fcc37ed02</p> <p>Landed for 2.16</p> Duplicate Related LU-4725 Development Rank 1|i03ewn: Rank (Obsolete) 9223372036854775807 Severity