https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-11481 Lustre <p>A directory has an entry for subdirectory "2fe", but the object ID stored for that entry does not exist:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>alias ll="ls -l" [root@catalyst101:~]# ll /p/lustre3/videousr/YLI/mmcommons/data/images_v1 ls: cannot access /p/lustre3/videousr/YLI/mmcommons/data/images_v1/2fe: No such file or directory total 0 d????????? ? ? ? ? ? 2fe </pre> </div></div> <p>And when using zdb on the MDT to examine images_v1, one sees that 2fe refers to an object ID that is invalid:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@porter81:snap]# zdb -ddddd porter81/mdt0 533741247 Dataset porter81/mdt0 [ZPL], ID 148, cr_txg 98, 910G, 61852198 objects, rootbp DVA[0]=&lt;4:88d9c400:200&gt; DVA[1]=&lt;5:25ca03c200:200&gt; [L0 DMU objset] fletcher4 lz4 LE contiguous unique double size=800L/200P birth=1214040L/1214040P fill=61852198 cksum=139cf672b7:5dc8d6146f6:f8e6add4f57c:1e27e38477f5c0 Object lvl iblk dblk dsize dnsize lsize %full type 533741247 2 128K 16K 231K 512 528K 100.00 ZFS directory 192 bonus System attributes dnode flags: USED_BYTES USERUSED_ACCOUNTED USEROBJUSED_ACCOUNTED SPILL_BLKPTR dnode maxblkid: 32 path ???&lt;object#533741247&gt; uid 0 gid 2093 atime Mon Oct 8 11:01:28 2018 mtime Wed Oct 3 15:53:08 2018 ctime Wed Oct 3 15:53:08 2018 crtime Mon Oct 1 20:53:54 2018 gen 1090081 mode 42700 size 2 parent 533740502 links 3 pflags 0 rdev 0x0000000000000000 SA xattrs: 204 bytes, 3 entries trusted.lma = \000\000\000\000\000\000\000\0002@\000\000\002\000\000\000\245\037\001\000\000\000\000\000 trusted.link = \337\361\352\021\001\000\000\0003\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\033\000\000\000\002\000\000@F\000\0001\213\000\000\000\000images_v1 trusted.version = \022\231\236+\011\000\000\000 Fat ZAP stats: Pointer table: 1024 elements zt_blk: 0 zt_numblks: 0 zt_shift: 10 zt_blks_copied: 0 zt_nextblk: 0 ZAP entries: 1 Leaf blocks: 32 Total blocks: 33 zap_block_type: 0x8000000000000001 zap_magic: 0x2f52ab2ab zap_salt: 0x3e3cbee7f Leafs with 2^n pointers: 5: 32 ******************************** Blocks with n*5 entries: 0: 32 ******************************** Blocks n/10 full: 1: 32 ******************************** Entries with n chunks: 4: 1 * Buckets with n entries: 0: 16383 **************************************** 1: 1 * 2fe = 533742980 (type: Directory) Indirect blocks: 0 L1 6:1a0095d000:a00 20000L/a00P F=33 B=1133009/1133009 0 L0 4:d99372200:200 4000L/200P F=1 B=1133009/1133009 4000 L0 4:2b78affa00:e00 4000L/e00P F=1 B=1132989/1132989 8000 L0 4:1a409fa00:e00 4000L/e00P F=1 B=1133008/1133008 c000 L0 4:dbecc8800:e00 4000L/e00P F=1 B=1133003/1133003 10000 L0 4:2d07544a00:e00 4000L/e00P F=1 B=1132997/1132997 14000 L0 5:11130c9600:e00 4000L/e00P F=1 B=1133005/1133005 18000 L0 5:1053a11c00:e00 4000L/e00P F=1 B=1132991/1132991 1c000 L0 4:2d07545800:e00 4000L/e00P F=1 B=1132997/1132997 20000 L0 6:1a41dd7c00:e00 4000L/e00P F=1 B=1133002/1133002 24000 L0 5:112ca4cc00:e00 4000L/e00P F=1 B=1133007/1133007 28000 L0 5:559e31000:e00 4000L/e00P F=1 B=1133000/1133000 2c000 L0 4:d91a7e000:e00 4000L/e00P F=1 B=1133004/1133004 30000 L0 4:d99372400:e00 4000L/e00P F=1 B=1133009/1133009 34000 L0 4:265bf62800:e00 4000L/e00P F=1 B=1132993/1132993 38000 L0 6:134c5fcc00:e00 4000L/e00P F=1 B=1132992/1132992 3c000 L0 5:559e31e00:e00 4000L/e00P F=1 B=1133000/1133000 40000 L0 5:11130ca400:e00 4000L/e00P F=1 B=1133005/1133005 44000 L0 4:dbeccac00:e00 4000L/e00P F=1 B=1133003/1133003 48000 L0 4:2b78b02200:e00 4000L/e00P F=1 B=1132989/1132989 4c000 L0 6:134c5ff400:e00 4000L/e00P F=1 B=1132992/1132992 50000 L0 4:1a40a2400:e00 4000L/e00P F=1 B=1133008/1133008 54000 L0 5:11130cb200:e00 4000L/e00P F=1 B=1133005/1133005 58000 L0 6:19f0f10c00:e00 4000L/e00P F=1 B=1132991/1132991 5c000 L0 4:1a40a3200:e00 4000L/e00P F=1 B=1133008/1133008 60000 L0 7:b97b6aa00:e00 4000L/e00P F=1 B=1133004/1133004 64000 L0 5:112ca4f400:e00 4000L/e00P F=1 B=1133007/1133007 68000 L0 4:17f825800:e00 4000L/e00P F=1 B=1132999/1132999 6c000 L0 6:1a2429de00:e00 4000L/e00P F=1 B=1132995/1132995 70000 L0 6:1a41dd9a00:e00 4000L/e00P F=1 B=1133002/1133002 74000 L0 7:129d29e800:e00 4000L/e00P F=1 B=1133007/1133007 78000 L0 4:dbeccca00:e00 4000L/e00P F=1 B=1133003/1133003 7c000 L0 4:17f826600:e00 4000L/e00P F=1 B=1132999/1132999 80000 L0 5:569fa5000:e00 4000L/e00P F=1 B=1132994/1132994 segment [0000000000000000, 0000000000084000) size 528K [root@porter81:snap]# zdb -ddddd porter81/mdt0 533742980 Dataset porter81/mdt0 [ZPL], ID 148, cr_txg 98, 910G, 61852198 objects, rootbp DVA[0]=&lt;4:88d9c400:200&gt; DVA[1]=&lt;5:25ca03c200:200&gt; [L0 DMU objset] fletcher4 lz4 LE contiguous unique double size=800L/200P birth=1214040L/1214040P fill=61852198 cksum=139cf672b7:5dc8d6146f6:f8e6add4f57c:1e27e38477f5c0 Object lvl iblk dblk dsize dnsize lsize %full type zdb: dmu_bonus_hold(533742980) failed, errno 2 </pre> </div></div> <p>This is on a new file system that has not been used by end-users yet, but which we attempted to copy data to. More specifically:<br/> 1, We copied about 500 million files/dirs to it<br/> 2. We tried to use lfs migrate -M to move some large subtrees from one MDT to another, but that failed due to a Lustre 2.8 bug with lfs migrate<br/> 3. We deleted most of the files/dirs</p> <ul> <li>The servers did not crash, as far as I can recall, while we were performing all the copy and delete operations. But I cannot be certain of that.</li> <li>We inspected the console logs on the servers and clients but found nothing that sounded like it indicated object creation or destruction failing.</li> </ul> server: RHEL 7.4 derivative, zfs-0.7.11-4llnl.ch6.x86_64, lustre-2.10.4_1.chaos <br/> client: RHEL 7.4 derivative, lustre-2.8.2_4.chaos-1 <br/> We are using DNE 1 with two MDTs on two servers, porter81 and porter82 <br/> for the zfs tag, see <a href="https://github.com/LLNL/zfs/releases">https://github.com/LLNL/zfs/releases</a> <br/> for our 2.10 tag, see <a href="https://github.com/LLNL/lustre/">https://github.com/LLNL/lustre/</a> <br/> for our 2.8 tag, see lustre-release-fe-llnl on gerritt LU-11481

corrupt directory

Bug Critical Resolved Fixed Lai Siyao Olaf Faaland llnl Mon, 8 Oct 2018 19:30:40 +0000 Mon, 25 Feb 2019 21:59:47 +0000 Fri, 15 Feb 2019 13:26:26 +0000 Lustre 2.10.4 Lustre 2.10.7 0 5 <p>We are uncertain whether <b>lfs migrate</b> was involved. If there is anything I could look for within either of the MDTs to determine whether <b>lfs migrate</b> was attempting to migrate this directory, or its parent, to help corroborate or rule that out, let me know.</p> <p>I've attached the console logs for the two MDSs. The files are named "console.porter</p> {81,82} <p>.tgz"</p> <p>This corruption was discovered early in the day on 2018-10-04 and the first attempt to copy this directory tree to the new file system, was started on 2018-09-22. So the damage must have occurred during the period covered by these logs.</p> <p>MDT0000 is in pool porter81.<br/> MDT0001 is in pool porter82.</p> <p>both pools report state "ONLINE", which means no errors.</p> <p>We do not have debug logs for MDT0000; it was stopped and re-started on 2018-10-04.</p> <p>Lai</p> <p>Could you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>The only clue I see is from porter81:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2018-10-04 04:29:58 [125075.971428] LustreError: 166475:0:(mdt_open.c:1515:mdt_reint_open()) lustre3-MDT0000: name '2fe' present, but FID [0x200004031:0x1216b:0x0] is invalid </pre> </div></div> <p>But it only tells that the FID of '2fe' doesn't exist. Can you use LFSCK to fix this inconsistency?</p> <p>Yes, I can run lfsck. Is there anything else I can look for, that might give a clue as to how this happened?</p> <p>It may be related with dir migration, because in 2.10, dir migration will migrate dirent of all sub files to target first, and then migrate inodes of sub files, if it fails in the middle, it may leave some dirents point to nowhere.</p> <p>I started lfsck about 40 hours ago, via </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>pdsh -w e81 lctl lfsck_start --all --create-ostobj on --create-mdtobj on --delay-create-ostobj on --orphan </pre> </div></div> <p>The invalid directory entry has been removed. Ignoring the "0" valued results, it now reports:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@porteri:toss-4318]# pdsh -w e81 lctl lfsck_query | awk "\$NF &gt; 0 {print}" e81: layout_mdts_completed: 2 e81: layout_osts_completed: 79 e81: layout_osts_unknown: 1 e81: namespace_mdts_completed: 2 e81: namespace_repaired: 1 </pre> </div></div> <p>1. Is there any way for me to know in more detail, what it changed?<br/> 2. What does layout_osts_unknown mean?</p> <p>I found that many (possibly all) of the OSTs had BUG: reports in their console logs, and that one OST had been failed over to its partner. I will create a ticket for that issue if there isn't one already, and link to it here.</p> <p>After I powered on the OSS that was off, and the OST moved back, I re-ran lfsck_query got this instead:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[root@porteri:toss-4318]# pdsh -w e81 lctl lfsck_query | awk "\$NF &gt; 0 {print}" e81: layout_mdts_completed: 2 e81: layout_osts_completed: 80 e81: namespace_mdts_completed: 2 e81: namespace_repaired: 1 </pre> </div></div> <p>This file system has 80 OSTs and 2 MDTs.</p> <p>Does that mean that:<br/> 1. layout_osts_unknown meant it could not fetch the status of 1 OST's lfsck<br/> 2. It is now telling me that all 2 MDTs finished and all 80 OSTs finished the lfsck?</p> <p>thanks</p> <p>layout_osts_unknown should be a bug, maybe it's because that OST crashed, or server replied a status not known.</p> <p>The second run should have finished lfsck on all MDTs and OSTs.</p> <p>I need to check the BUG info to better understand it, and see if I can inject error in the code to get that output.</p> <p>Can you provide the BUG info?</p> <p>Hello Lai,</p> <p>Sorry for the long delay.  Please see <a href="https://jira.whamcloud.com/browse/LU-11620" class="external-link" rel="nofollow">https://jira.whamcloud.com/browse/LU-11620</a> for one stack related to lfsck.</p> <p>Hi.  What's the plan for this issue (the creation of the inconsistency, not lfsck)?  Thanks.</p> <p>Directory migration is rewritten, which fixed many issues in migration, but it's in 2.12.</p> <blockquote><p>Directory migration is rewritten, which fixed many issues in migration, but it's in 2.12.</p></blockquote> <p>OK, then you're saying:<br/> (a) directory migration in 2.10 is unsafe - risks data loss - and should not be used,<br/> and<br/> (b) there is nothing more to do on this issue and no additional debug code is necessary</p> <p>Is that correct?<br/> Thanks</p> <p>Olaf Faaland-LLNL (faaland1@llnl.gov) uploaded a new patch: <a href="https://review.whamcloud.com/33960" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33960</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11481" title="corrupt directory" class="issue-link" data-issue-key="LU-11481"><del>LU-11481</del></a> utils: disable lfs migrate -m<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: d08d4a3b232c0e1a6a1fb9d2ee6f315fd26ae498</p> <p>In case (a) is "yes", I've uploaded a patch for b2_10.</p> <p>Yes, Olaf.</p> <p>Hello Lai,</p> <p>I've added you as a reviewer on my patch, which at last update passed tests except sanity-scrub.test_9 which seems to me like it's unrelated to my patch - but maybe I'm mistaken.  Can you kick it so that the review-dne-part-2, which includes sanity-scrub, is re-tested?</p> <p>thanks</p> <p>I re-triggered it</p> <p>Pushed against Master by mistake.  This one will be abandoned.</p> <p><del>Olaf Faaland-LLNL (faaland1@llnl.gov) uploaded a new patch: <a href="https://review.whamcloud.com/34130" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/34130</a></del><br/> <del>Subject: <a href="https://jira.whamcloud.com/browse/LU-11481" title="corrupt directory" class="issue-link" data-issue-key="LU-11481"><del>LU-11481</del></a> utils: disable lfs migrate -m</del><br/> <del>Project: fs/lustre-release</del><br/> <del>Branch: master</del><br/> <del>Current Patch Set: 1</del><br/> <del>Commit: 459ba774583997e616a04715709fc2f671dbe0bb</del></p> <p>Oleg Drokin (green@whamcloud.com) merged in patch <a href="https://review.whamcloud.com/33960/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33960/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-11481" title="corrupt directory" class="issue-link" data-issue-key="LU-11481"><del>LU-11481</del></a> utils: disable lfs migrate -m<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: <br/> Commit: 3b7e4ac3bb896d66613e9a6bafbcf6c01a1ac63d</p> <p>Landed for 2.10.7. Not needed on master</p> Related Development Rank 1|i003sf: Rank (Obsolete) 9223372036854775807 Severity