https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4876 Lustre <p>When migrate directory MDT-object_A's metadata from MDT_x to MDT_y, all its children name entries needs to be moved to MDT_y. If some name entry originally referenced a remote MDT-object_B which was linked under "/REMOTE_PARENT_DIR" with a dummy name entry on MDT_y, but after the migration, the MDT-object_B and its new name entry resides on the same MDT_y, then the dummy entry for the MDT-object_B should be removed from "<tt>REMOTE_PARENT_DIR/</tt>" <b>iff</b> there are no links remaining on that MDT.</p> LU-4876

LFSCK remove entry from /REMOTE_PARENT_DIR if MDT-object name reside on the same MDT after migration

Bug Major Open Unresolved Lai Siyao nasf dne2 lfsck Wed, 9 Apr 2014 21:43:28 +0000 Mon, 31 Jan 2022 23:14:25 +0000 0 6 <p>Hmm, we can remove the object from "REMOTE_PARENT_DIR" when we update the linkEA, but this will add extra complication into osd_xattr_set. But even we leave dummy inode in REMOTE_PARENT_DIR, it did not effect the consistency of the FS, though it will leave some orphans there and I also wonder if LFSCK will do sth special for REMOTE_PARENT_DIR for DNE check. </p> <p>From the LFSCK view, if two name entries reference the same MDT-object, but its nlink count is 1, that is inconsistent status.</p> <p>It seems to me that there are two separate issues here:</p> <ul class="alternate" type="square"> <li>DNE should not leave the filesystem in an inconsistent state under normal operation, and should not depend on LFSCK.</li> <li>LFSCK3 should repair the filesystem correctly in this case (regardless of how it got into this state)</li> </ul> <p>I have created another ticket for LFSCK: <a href="https://jira.whamcloud.com/browse/LU-5985" title="LFSCK 5: remove dummy name entry from /REMOTE_PARENT_DIR if the MDT-object and its name entry reside on the same MDT" class="issue-link" data-issue-key="LU-5985"><del>LU-5985</del></a>.</p> <p>Lai, is this something you can look at while fixing the striped directory migration in <a href="https://jira.whamcloud.com/browse/LU-4684" title="DNE3: allow migrating DNE striped directory" class="issue-link" data-issue-key="LU-4684"><del>LU-4684</del></a>?</p> <p>Hello,</p> <p>I have migrate related question. I see probable LFSCK-related inconsistency possible. I catched ldiskfs inconsistency after directory migration. Started this command</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> SLOW=yes RUNAS_ID=1000 CLEANUP=cleanupall SETUP=setupall OSTCOUNT=4 MDSCOUNT=4 OSTSIZE=600000 MDTSIZE=300000 ONLY=230h sh /usr/lib/lustre/tests/sanity.sh </pre> </div></div> <p>and run fsck just after test has been finished.</p> <p>Fsck shows:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Inode 25043 ref count is 5, should be 4.  Fix? no Pass 5: Checking group summary information Free blocks count wrong (32862, counted=32853). Fix? no Free inodes count wrong (99721, counted=99718). Fix? no lustre-MDT0000: ********** WARNING: Filesystem still has errors ********** </pre> </div></div> <p>Inode with broken count is "/ROOT"</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> # debugfs lustre-mdt1 debugfs 1.42.13.wc6 (05-Feb-2017) debugfs:  ncheck 25043 Inode    Pathname 25043    <span class="code-comment">//ROOT</span> </pre> </div></div> <p>Inode has 5 links</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> # debugfs -R <span class="code-quote">"stat &lt;25043&gt;"</span> lustre-mdt1 debugfs 1.42.13.wc6 (05-Feb-2017) Inode: 25043   Type: directory    Mode:  0755   Flags: 0x0 Generation: 3666376863    Version: 0x00000001:00000008 User:     0   Group:     0   Project:     0   Size: 4096 File ACL: 0    Directory ACL: 0 Links: 5   Blockcount: 8 Fragment:  Address: 0    <span class="code-object">Number</span>: 0    Size: 0 ctime: 0x5ba2dcf7:00000000 -- Wed Sep 19 19:34:15 2018 atime: 0x5ba2dcbc:00000000 -- Wed Sep 19 19:33:16 2018 mtime: 0x5ba2dcf7:00000000 -- Wed Sep 19 19:34:15 2018 crtime: 0x5ba2dcbc:b0d6b4b8 -- Wed Sep 19 19:33:16 2018 Size of extra inode fields: 32 Extended attributes stored in inode body:   lma = <span class="code-quote">"00 00 00 00 00 00 00 00 07 00 00 00 02 00 00 00 01 00 00 00 00 00 00 00 "</span> (24)   lma: fid=[0x200000007:0x1:0x0] compat=0 incompat=0 BLOCKS: (0):12623 TOTAL: 1 </pre> </div></div> <p>But should be 4</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> debugfs(mdt1):  ls -l /ROOT   25043   40755 (2)      0      0    4096 19-Sep-2018 19:34 .       2   40755 (2)      0      0    4096 19-Sep-2018 19:33 ..   25044   40755 (18)      0      0    4096 19-Sep-2018 19:33 .lustre   25049   40000 (18)      0      0    4096 19-Sep-2018 19:34 d230h.sanity </pre> </div></div> <p>sanity 230h test is about migrating root's subdirectory to mdt2. The main operation is</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> $LFS migrate -m1 $DIR/$tdir/migrate_dir/.. || error <span class="code-quote">"migrating $tdir fail"</span> </pre> </div></div> <p>After this operation $DIR/$tdi and $DIR/$tdir/migrate_dir/ are moved to mdt2<br/> Checked in image. This directories are actually moved there:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> debugfs(mdt2):  ls -l /REMOTE_PARENT_DIR 25001   40755 (2)      0      0    4096 19-Sep-2018 19:34 .     2   40755 (2)      0      0    4096 19-Sep-2018 19:33 ..   25046   40755 (2)      0      0    4096 19-Sep-2018 19:34 0x240000404:0x1:0x0 # lfs fid2path /mnt/lustre 0x240000404:0x1:0x0 /mnt/lustre/d230h.sanity </pre> </div></div> <p>Do you have idea why anode has reference 5, then it need have reference 4?</p> <p>Thanks.</p> <p>Hi Artem, can you test with latest master code? I think patches for <a href="https://jira.whamcloud.com/browse/LU-4684" title="DNE3: allow migrating DNE striped directory" class="issue-link" data-issue-key="LU-4684"><del>LU-4684</del></a> has fixed this issue.</p> <p>Artem, I also noticed this, but it was fixed (at least for 2.12) by unmounting the MDT and running e2fsck on the unmounted device. While it was mounted, the change was only in the journal and did not show up in the filesystem itself.</p> <p><a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=adilger" class="user-hover" rel="adilger">adilger</a> Thanks for idea. I believe the target was not mounted during fsck. I used CLEANUP=cleanupall option.<br/> <a href="https://jira.whamcloud.com/secure/ViewProfile.jspa?name=laisiyao" class="user-hover" rel="laisiyao">laisiyao</a> will try with patches from <a href="https://jira.whamcloud.com/browse/LU-4684" title="DNE3: allow migrating DNE striped directory" class="issue-link" data-issue-key="LU-4684"><del>LU-4684</del></a>. Thanks.</p> <p>Artem, it just looks that way because of the e2fsck messages:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Pass 5: Checking group summary information Free blocks count wrong (32862, counted=32853). Free inodes count wrong (99721, counted=99718). </pre> </div></div> <p>which usually means that the filesystem is mounted, or at least was not unmounted cleanly.</p> <p>I've also seen a few cases where e2fsck complains about hard-linked directories in <tt>REMOTE_PARENT_DIR</tt>:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Entry '0x200007168:0x4af:0x0' in /REMOTE_PARENT_DIR (1119354881) is a link to directory /ROOT/.lustre/lost+found/MDT0000/[0x200005233:0x89:0x0]-O-0 (8938935). Clear? yes </pre> </div></div> <p>It looks like e2fsck fixes this by removing the entry from REMOTE_PARENT_DIR, but LFSCK shouldn't get into this situation in the first place.</p> Related LU-15383 LU-5985 LU-15388 LU-10329 LU-14168 LU-4684 Development Rank 1|hzwjpz: Rank (Obsolete) 13487 Severity