https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-8015 Lustre <p>I had a directory that contained two files (one is a hardlink of another):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># ls -il /hardlink/ total 120 163865 -rwxr-xr-x 2 root root 53400 Nov 11 2010 ed 163865 -rwxr-xr-x 2 root root 53400 Nov 11 2010 red </pre> </div></div> <p>After copying the directory to a Lustre filesystem:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>cp -a /hardlink /mnt/lustre/ </pre> </div></div> <p>Running lustre_rsync hit the following errors:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre># lustre_rsync --source=/mnt/lustre/ --target=/mnt/backup/ --mdt=lustre-MDT0000 --user=cl1 --statuslog lustre_rsync.log --verbose Lustre filesystem: lustre MDT device: lustre-MDT0000 Source: /mnt/lustre/ Target: /mnt/backup/ Statuslog: lustre_rsync.log Changelog registration: cl1 Starting changelog record: 0 Replication of operation failed(-22): 5 HLINK (3) [0x200000401:0x6e2:0x0] [0x200000401:0x6e1:0x0] ed Replication of operation failed(-21): 6 SATTR (14) [0x200000401:0x6e1:0x0] [0x200000401:0x6e1:0x0] Errors: 2 lustre_rsync took 0 seconds Changelog records consumed: 7 </pre> </div></div> <p>The type of the target filesystem /mnt/backup/ is ext4. </p> Lustre Build: <a href="https://build.hpdd.intel.com/job/lustre-master/3343/">https://build.hpdd.intel.com/job/lustre-master/3343/</a> LU-8015

Replication of operation failed(-22): 2885 HLINK (3)

Bug Minor Resolved Fixed Jian Yu Jian Yu Wed, 13 Apr 2016 01:21:44 +0000 Mon, 26 Sep 2016 21:03:29 +0000 Wed, 27 Apr 2016 15:56:19 +0000 Lustre 2.9.0 Lustre 2.9.0 1 13 <p>The output (with "--debug 2" option) of lustre_rsync command, and the Lustre debug logs (with "debug=-1" level) for Client and MDS are attached.</p> <p>The detailed output of lustre_rsync command:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>***** Start 5 HLINK (3) [0x200000401:0x6e2:0x0] [0x200000401:0x6e1:0x0] ed ***** fid2path hardlink/red, ed, 0 rc=0 fid2path hardlink/ed, ed, 1 rc=0 ##### End 5 HLINK (3) [0x200000401:0x6e2:0x0] [0x200000401:0x6e1:0x0] ed rc=-22 ##### ***** Start 6 SATTR (14) [0x200000401:0x6e1:0x0] [0x200000401:0x6e1:0x0] ***** setattr: /mnt/lustre//.lustre/fid/[0x200000401:0x6e1:0x0] /mnt/backup//hardlink [0x200000401:0x6e1:0x0] ##### End 6 SATTR (14) [0x200000401:0x6e1:0x0] [0x200000401:0x6e1:0x0] rc=-21 ##### </pre> </div></div> <p>Debug log on MDS:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000004:20000000:0.0:1460512029.531552:0:31484:0:(mdt_hsm.c:345:mdt_hsm_state_set()) Incompatible flag change on [0x200000401:0x6e2:0x0]flags=0x2 00000004:00000001:0.0:1460512029.531554:0:31484:0:(mdt_hsm.c:346:mdt_hsm_state_set()) Process leaving via out_unlock (rc=18446744073709551594 : -22 : 0xffffffffffffffea) </pre> </div></div> <p>Debug log on Client:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000002:00000001:0.0:1460512029.686738:0:17272:0:(mdc_request.c:1703:mdc_ioc_hsm_state_set()) Process leaving via out (rc=18446744073709551594 : -22 : 0xffffffffffffffea) &lt;~snip~&gt; 00000002:00000001:0.0:1460512029.686761:0:17272:0:(mdc_request.c:2075:mdc_iocontrol()) Process leaving via out (rc=18446744073709551594 : -22 : 0xffffffffffffffea) 00800000:00000001:0.0:1460512029.686763:0:17272:0:(obd_class.h:1143:obd_iocontrol()) Process leaving (rc=18446744073709551594 : -22 : ffffffffffffffea) 00800000:00000001:0.0:1460512029.686765:0:17272:0:(lmv_obd.c:1175:lmv_iocontrol()) Process leaving (rc=18446744073709551594 : -22 : ffffffffffffffea) 00000080:00000001:0.0:1460512029.686767:0:17272:0:(obd_class.h:1143:obd_iocontrol()) Process leaving (rc=18446744073709551594 : -22 : ffffffffffffffea) 00000080:00000010:0.0:1460512029.686768:0:17272:0:(llite_lib.c:2514:ll_finish_md_op_data()) kfreed 'op_data': 312 at ffff88004e3c0000. 00000080:00000001:0.0:1460512029.686770:0:17272:0:(file.c:2102:ll_hsm_state_set()) Process leaving (rc=18446744073709551594 : -22 : ffffffffffffffea) 00000080:00000040:0.0:1460512029.686772:0:17272:0:(llite_lib.c:1712:ll_setattr_raw()) [0x200000401:0x6e2:0x0]HSM set dirty failed: rc2 = -22 00000080:00000001:0.0:1460512029.686774:0:17272:0:(llite_lib.c:1715:ll_setattr_raw()) Process leaving </pre> </div></div> <p>Hi Jian,</p> <p>Is this something you can have a look at?</p> <p>Thanks.<br/> Joe</p> <p>Can you track down what the "Incompatible flag change" is? Is lustre_rsync trying to copy some HSM xattr from the source to the target?</p> <p>On Client side, the code path is ll_setattr()-&gt;ll_setattr_raw():</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>ll_setattr_raw()</b></div><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (ll_file_test_and_clear_flag(lli, LLIF_DATA_MODIFIED)) { struct hsm_state_set hss = { .hss_valid = HSS_SETMASK, .hss_setmask = HS_DIRTY, }; <span class="code-object">int</span> rc2; rc2 = ll_hsm_state_set(inode, &amp;hss); ------------------------------------- &gt; ll_hsm_state_set() <span class="code-keyword">finally</span> called mdc_ioc_hsm_state_set() and got -EINVAL from MDS side /* truncate and write can happen at the same time, so that * the file can be set modified even though the file is not * restored from released state, and ll_hsm_state_set() is * not applicable <span class="code-keyword">for</span> the file, and rc2 &lt; 0 is normal in <span class="code-keyword">this</span> * <span class="code-keyword">case</span>. */ <span class="code-keyword">if</span> (rc2 &lt; 0) CDEBUG(D_INFO, DFID <span class="code-quote">"HSM set dirty failed: rc2 = %d\n"</span>, PFID(ll_inode2fid(inode)), rc2); } </pre> </div></div> <p>On MDS side, mdt_hsm_state_set() was called to handle the MDS_HSM_STATE_SET rpc from Client:</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>mdt_hsm_state_set()</b></div><div class="codeContent panelContent"> <pre class="code-java"> /* Check <span class="code-keyword">for</span> inconsistant HSM flagset. * DIRTY without EXISTS: no dirty <span class="code-keyword">if</span> no archive was created. * DIRTY and RELEASED: a dirty file could not be released. * RELEASED without ARCHIVED: <span class="code-keyword">do</span> not release a non-archived file. * LOST without ARCHIVED: cannot lost a non-archived file. */ flags = ma-&gt;ma_hsm.mh_flags; <span class="code-keyword">if</span> ((flags &amp; HS_DIRTY &amp;&amp; !(flags &amp; HS_EXISTS)) || (flags &amp; HS_RELEASED &amp;&amp; flags &amp; HS_DIRTY) || (flags &amp; HS_RELEASED &amp;&amp; !(flags &amp; HS_ARCHIVED)) || (flags &amp; HS_LOST &amp;&amp; !(flags &amp; HS_ARCHIVED))) { CDEBUG(D_HSM, <span class="code-quote">"Incompatible flag change on "</span>DFID <span class="code-quote">"flags="</span>LPX64<span class="code-quote">"\n"</span>, PFID(&amp;info-&gt;mti_body-&gt;mbo_fid1), flags); GOTO(out_unlock, rc = -EINVAL); -------------------------------------------&gt; -EINVAL was returned to mdc_ioc_hsm_state_set() on Client side } </pre> </div></div> <p>There are two places where LLIF_DATA_MODIFIED is likely to be set. One is in vvp_io_write_start(), and the other is in vvp_io_setattr_fini().<br/> From the Client debug log, we can see:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000080:00000001:0.0:1460512029.678658:0:17272:0:(vvp_io.c:1060:vvp_io_write_start()) Process entered 00000080:00200000:0.0:1460512029.678660:0:17272:0:(vvp_io.c:1080:vvp_io_write_start()) write: [0, 53400) &lt;~snip~&gt; 00000080:00200000:0.0:1460512029.681032:0:17272:0:(vvp_io.c:1131:vvp_io_write_start()) write: nob 53400, result: 53400 00000080:00000001:0.0:1460512029.681033:0:17272:0:(vvp_io.c:1144:vvp_io_write_start()) Process leaving (rc=0 : 0 : 0) </pre> </div></div> <p>And also:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>00000080:00200000:0.0:1460512029.681212:0:17272:0:(vvp_io.c:312:vvp_io_fini()) [0x200000401:0x6e2:0x0] ignore/verify layout 0/0, layout version 0 restore needed 0 </pre> </div></div> <p>The value after all of the "restore needed" messages is 0 in the debug log. So, LLIF_DATA_MODIFIED was only set in vvp_io_write_start(), not in vvp_io_setattr_fini(), and the file is not a restored file.</p> <p>According to the analysis in the above comments, we can see just checking LLIF_DATA_MODIFIED flag is not enough to make sure a file is a restored file. So, the following codes in ll_setattr_raw() need to be fixed:</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>ll_setattr_raw()</b></div><div class="codeContent panelContent"> <pre class="code-java"> /* If the file was restored, it needs to set dirty flag. * * We've already sent MDS_DATA_MODIFIED flag in * ll_md_setattr() <span class="code-keyword">for</span> truncate. However, the MDT refuses to * set the HS_DIRTY flag on released files, so we have to set * it again <span class="code-keyword">if</span> the file has been restored. Please check how * LLIF_DATA_MODIFIED is set in vvp_io_setattr_fini(). * * Please notice that <span class="code-keyword">if</span> the file is not released, the previous * MDS_DATA_MODIFIED has taken effect and usually * LLIF_DATA_MODIFIED is not set(see vvp_io_setattr_fini()). * This way we can save an RPC <span class="code-keyword">for</span> common open + trunc * operation. */ <span class="code-keyword">if</span> (ll_file_test_and_clear_flag(lli, LLIF_DATA_MODIFIED)) { </pre> </div></div> <p>Hi Jinshan,</p> <p>Could you please take a look at the above analysis to see whether it's in a correct direction? Thank you.</p> <p>the race in ll_setattr_raw() should be normal and that error shouldn't be seen by user space applications.</p> <p>Jinshan is right. In the implementation of ll_setattr_raw(), the value of rc2 returned by ll_hsm_state_set() doesn't return to ll_setattr_raw(). However, in the experiment of ignoring the return value of ll_hsm_state_set(), the HLINK operation passed. Need investigate more. </p> <p>It turns out the "-EINVAL" error is another one returned from the following codes in lustre/utils/lustre_rsync.c:</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>lr_link()</b></div><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (info-&gt;src[0] == 0 || info-&gt;dest[0] == 0) /* Could not find the source or destination. This can happen when some links don't exist anymore. */ <span class="code-keyword">return</span> -EINVAL; </pre> </div></div> <p>And the defect is in the following codes:</p> <div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader" style="border-bottom-width: 1px;"><b>lr_link()</b></div><div class="codeContent panelContent"> <pre class="code-java"> len = strlen(info-&gt;path) - strlen(info-&gt;name); <span class="code-keyword">if</span> (len &gt;= 0 &amp;&amp; strcmp(info-&gt;path + len, info-&gt;name) == 0) ...... <span class="code-keyword">else</span> <span class="code-keyword">if</span> (info-&gt;src[0] == 0) ...... </pre> </div></div> <p>The above condition used to find out info-&gt;src and info-&gt;dest has defect in the following situations:<br/> 1) the source name ends with the destination name<br/> 2) the file has two links with the same name but in different directories</p> <p>With the guidance from John and Jinshan, I'm creating a patch to fix the above issue.</p> <p>Jian Yu (jian.yu@intel.com) uploaded a new patch: <a href="http://review.whamcloud.com/19599" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19599</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8015" title="Replication of operation failed(-22): 2885 HLINK (3)" class="issue-link" data-issue-key="LU-8015"><del>LU-8015</del></a> utils: fix lr_link() to get correct src and dest<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 878a003b75d49606f124e16ef61ae4d620cd49c3</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/19599/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/19599/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-8015" title="Replication of operation failed(-22): 2885 HLINK (3)" class="issue-link" data-issue-key="LU-8015"><del>LU-8015</del></a> utils: fix lr_link() to get correct src and dest<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: a72e6ee375248636cac1ea212d517254c8850b88</p> <p>Landed to master for 2.9.0</p> Related Development Rank 1|hzy7tb: Rank (Obsolete) 9223372036854775807 Severity