https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-2627 Lustre <p>Doing an ls gives the following error<br/> ls: reading directory d4_stats/: Input/output error</p> <p>client error:<br/> <span class="error">&#91;5237686.818045&#93;</span> LustreError: 77522:0:(dir.c:648:ll_readdir()) error reading dir <span class="error">&#91;0x4488b6ced74:0x1edb5:0x0&#93;</span> at 0: rc -5<br/> <span class="error">&#91;5237686.849844&#93;</span> LustreError: 77522:0:(dir.c:648:ll_readdir()) Skipped 51 previous similar messages</p> <p>MDT Error:<br/> Jan 16 11:18:37 nbp1-mds kernel: Lustre: 15390:0:(mdd_object.c:2412:__mdd_readpage()) build page failed: -5!</p> <p>Please advise on debug flags to use to gather logs.</p> LU-2627

/bin/ls gets Input/output error

Bug Critical Resolved Fixed Cliff White Mahmoud Hanafi Wed, 16 Jan 2013 14:24:11 +0000 Wed, 3 Jul 2013 21:02:54 +0000 Thu, 21 Mar 2013 20:23:50 +0000 Lustre 2.1.3 0 12 <p>MDS has logged these messages<br/> an 15 15:23:11 nbp1-mds kernel: LDISKFS-fs warning (device dm-2): dx_probe: dx entry: limit != root limit<br/> Jan 15 15:23:11 nbp1-mds kernel: LDISKFS-fs warning (device dm-2): dx_probe: Corrupt dir inode 158428053, running e2fsck is recommended.<br/> Jan 15 15:23:11 nbp1-mds kernel: LDISKFS-fs warning (device dm-2): dx_probe: dx entry: limit != root limit<br/> Jan 15 15:23:11 nbp1-mds kernel: LDISKFS-fs warning (device dm-2): dx_probe: Corrupt dir inode 158428056, running e2fsck is recommended.<br/> Jan 15 15:23:45 nbp1-mds kernel: LDISKFS-fs warning (device dm-2): dx_probe: dx entry: limit != root limit<br/> Jan 15 15:23:45 nbp1-mds kernel: LDISKFS-fs warning (device dm-2): dx_probe: Corrupt dir inode 158427968, running e2fsck is recommended.<br/> Jan 15 15:23:45 nbp1-mds kernel: LDISKFS-fs warning (device dm-2): dx_probe: dx entry: limit != root limit<br/> Jan 15 15:23:45 nbp1-mds kernel: LDISKFS-fs warning (device dm-2): dx_probe: Corrupt dir inode 158428043, running e2fsck is recommended.<br/> Jan 16 03:28:55 nbp1-mds kernel: LDISKFS-fs warning (device dm-2): dx_probe: Unrecognised inode hash code 5 for directory #37273881<br/> Jan 16 03:28:55 nbp1-mds kernel: LDISKFS-fs warning (device dm-2): dx_probe: Corrupt dir inode 37273881, running e2fsck is recommended.<br/> Jan 16 07:24:28 nbp1-mds kernel: LDISKFS-fs warning (device dm-2): dx_probe: Unrecognised inode hash code 4 for directory #39331752<br/> Jan 16 07:24:28 nbp1-mds kernel: LDISKFS-fs warning (device dm-2): dx_probe: Corrupt dir inode 39331752, running e2fsck is recommended.<br/> Jan 16 07:25:11 nbp1-mds kernel: LDISKFS-fs warning (device dm-2): dx_probe: Unrecognised inode hash code 5 for directory #37273881<br/> Jan 16 07:25:11 nbp1-mds kernel: LDISKFS-fs warning (device dm-2): dx_probe: Corrupt dir inode 37273881, running e2fsck is recommended.<br/> Jan 16 09:59:07 nbp1-mds kernel: LDISKFS-fs warning (device dm-2): dx_probe: Unrecognised inode hash code 107 for directory #15731753<br/> Jan 16 09:59:07 nbp1-mds kernel: LDISKFS-fs warning (device dm-2): dx_probe: Corrupt dir inode 15731753, running e2fsck is recommended.<br/> Jan 16 11:13:11 nbp1-mds kernel: LDISKFS-fs warning (device dm-2): dx_probe: Unrecognised inode hash code 10 for directory #37272081<br/> Jan 16 11:13:11 nbp1-mds kernel: LDISKFS-fs warning (device dm-2): dx_probe: Corrupt dir inode 37272081, running e2fsck is recommended.</p> <p>Can unmount and just run e2fsck on the mdt device?</p> <p>Yes, you should umount and fsck the MDT. You do not have to umount clients, however clients may block while the MDT is down.</p> <p>Can you please provide the exact options to used for the fsck command</p> <p>First, check all your logs and see if you are having hardware failures. Is there any error logging in your disk hardware? The device under dm-2 may have an issue.<br/> For fsck, first run</p> <ol> <li>fsck -fn &lt;yourMDTdevice&gt;</li> </ol> <p>This is a read-only pass, and should give you an idea of what is going on. <br/> Then, you can run </p> <ol> <li>fsck -fy &lt;yourMDTdevice&gt;<br/> to repair. </li> </ol> <p>Read only pass has lots of errors like this<br/> Error while reading over extent tree in inode 14726136: Corrupt extent header</p> <p>Can you post the full output?</p> <p>And are you using e2fsprogs from Whamcloud? Please indicate the version of e2fsprogs you have installed.</p> <p>Also, is there any indicate of hardware issue with the disk?</p> <p>long list of this </p> <p>nbp1-MDT0000 has been mounted 110 times without being checked, check forced.<br/> Pass 1: Checking inodes, blocks, and sizes<br/> Error while reading over extent tree in inode 8502968: Corrupt extent header<br/> Clear inode? no</p> <p>Error while reading over extent tree in inode 8503011: Corrupt extent header<br/> Clear inode? no</p> <p>Error while reading over extent tree in inode 8503034: Corrupt extent header<br/> Clear inode? no</p> <p>Error while reading over extent tree in inode 8503327: Corrupt extent header<br/> Clear inode? no</p> <p>Error while reading over extent tree in inode 8503340: Corrupt extent header<br/> Clear inode? no</p> <p>Error while reading over extent tree in inode 8503345: Corrupt extent header<br/> Clear inode? no</p> <p>Error while reading over extent tree in inode 8503781: Corrupt extent header<br/> Clear inode? no</p> <p>Error while reading over extent tree in inode 8503785: Corrupt extent header<br/> Clear inode? no</p> <p>Error while reading over extent tree in inode 8503787: Corrupt extent header<br/> Clear inode? no</p> <p>Error while reading over extent tree in inode 8503801: Corrupt extent header<br/> Clear inode? no</p> <p>Error while reading over extent tree in inode 8503805: Corrupt extent header<br/> Clear inode? no</p> <p>Error while reading over extent tree in inode 8503808: Corrupt extent header<br/> Clear inode? no</p> <p>Error while reading over extent tree in inode 8503810: Corrupt extent header<br/> Clear inode? no</p> <p>Error while reading over extent tree in inode 8503956: Corrupt extent header<br/> Clear inode? no</p> <p>Error while reading over extent tree in inode 8503961: Corrupt extent header<br/> Clear inode? no</p> <p>Error while reading over extent tree in inode 8504005: Corrupt extent header<br/> Clear inode? no</p> <p>Error while reading over extent tree in inode 8510949: Corrupt extent header<br/> Clear inode? no</p> <p>Error while reading over extent tree in inode 8541695: Corrupt extent header<br/> Clear inode? no</p> <p>I just stopped it for now.</p> <p>FYI- this is 1.8 upgraded to 2.x filesystem.<br/> we have the following</p> <p>e2fsprogs-1.41.90.wc4-7.el6.x86_64<br/> lustre-ldiskfs-3.3.0-1nasS_2.6.32_279.2.1.el6.20120824.x86_64.lustre213.x86_64</p> <p>That is rather bad. You need to verify that your disk hardware is healthy, you may be seeing a disk failure. Do you have a backup? </p> <p>Can you give us your kernel version, and the version on all Lustre RPMS? You compile your own Lustre?</p> <p>Hardware is healthy. We don't have backups. But I am able to remount the mdt. <br/> but got these errors<br/> Jan 16 14:24:10 nbp1-mds kernel: Lustre: nbp1-MDT0000: disconnecting 1 stale clients<br/> Jan 16 14:24:10 nbp1-mds kernel: LustreError: 86971:0:(mdt_handler.c:2792:mdt_recovery()) operation 35 on unconnected MDS from 12345-10.151.53.248@o2ib<br/> Jan 16 14:24:10 nbp1-mds kernel: LustreError: 86971:0:(mdt_handler.c:2792:mdt_recovery()) Skipped 2802 previous similar messages<br/> Jan 16 14:24:10 nbp1-mds kernel: Lustre: 87267:0:(ldlm_lib.c:946:target_handle_connect()) nbp1-MDT0000: connection from 14c5fa4f-a16b-c855-4b79-5ba10c67a331@10.151.53.248@o2ib recovering/t491111894205 exp (null) cur 1358375050 last 0<br/> Jan 16 14:24:10 nbp1-mds kernel: Lustre: 87267:0:(ldlm_lib.c:946:target_handle_connect()) Skipped 4169 previous similar messages<br/> Jan 16 14:24:10 nbp1-mds kernel: Lustre: nbp1-MDT0000: Client f0dbfa3e-028e-c2ce-22e3-c3f171e89ebf (at 10.151.26.25@o2ib) reconnecting, waiting for 12090 clients in recovery for 3:12<br/> Jan 16 14:24:10 nbp1-mds kernel: Lustre: nbp1-MDT0000: Denying connection for new client 10.151.53.248@o2ib (at 14c5fa4f-a16b-c855-4b79-5ba10c67a331), waiting for 745 clients in recovery for 3:12<br/> Jan 16 14:24:10 nbp1-mds kernel: Lustre: Skipped 47 previous similar messages<br/> Jan 16 14:24:12 nbp1-mds kernel: Lustre: nbp1-MDT0000: Client dd76159a-cc95-ee10-7333-4317d790b9fb (at 10.151.27.24@o2ib) reconnecting, waiting for 12090 clients in recovery for 3:12<br/> Jan 16 14:24:13 nbp1-mds kernel: Lustre: nbp1-MDT0000: sending delayed replies to recovered clients<br/> Jan 16 14:24:15 nbp1-mds kernel: Lustre: MDS mdd_obd-nbp1-MDT0000: nbp1-OST000a_UUID now active, resetting orphans<br/> Jan 16 14:24:15 nbp1-mds kernel: Lustre: Skipped 91 previous similar messages<br/> Jan 16 14:24:15 nbp1-mds kernel: Lustre: MDS mdd_obd-nbp1-MDT0000: nbp1-OST0002_UUID now active, resetting orphans<br/> Jan 16 14:24:15 nbp1-mds kernel: Lustre: Skipped 1 previous similar message<br/> Jan 16 14:24:15 nbp1-mds kernel: Lustre: 86698:0:(mdd_orphans.c:371:orph_key_test_and_del()) Found orphan! Delete it<br/> Jan 16 14:24:15 nbp1-mds kernel: Lustre: 86698:0:(mdd_orphans.c:371:orph_key_test_and_del()) Skipped 2579 previous similar messages<br/> Jan 16 14:24:16 nbp1-mds kernel: Lustre: MDS mdd_obd-nbp1-MDT0000: nbp1-OST0025_UUID now active, resetting orphans<br/> Jan 16 14:24:16 nbp1-mds kernel: Lustre: Skipped 22 previous similar messages<br/> Jan 16 14:24:18 nbp1-mds kernel: Lustre: 86698:0:(mdd_orphans.c:371:orph_key_test_and_del()) Found orphan! Delete it<br/> Jan 16 14:24:18 nbp1-mds kernel: Lustre: 86698:0:(mdd_orphans.c:371:orph_key_test_and_del()) Skipped 1 previous similar message<br/> Jan 16 14:24:22 nbp1-mds kernel: Lustre: 86698:0:(mdd_orphans.c:371:orph_key_test_and_del()) Found orphan! Delete it<br/> Jan 16 14:24:22 nbp1-mds kernel: Lustre: 86698:0:(mdd_orphans.c:371:orph_key_test_and_del()) Skipped 18 previous similar messages<br/> Jan 16 14:24:30 nbp1-mds kernel: Lustre: 86698:0:(mdd_orphans.c:371:orph_key_test_and_del()) Found orphan! Delete it<br/> Jan 16 14:24:30 nbp1-mds kernel: Lustre: 86698:0:(mdd_orphans.c:371:orph_key_test_and_del()) Skipped 156 previous similar messages<br/> Jan 16 14:24:45 nbp1-mds kernel: LDISKFS-fs warning (device dm-2): dx_probe: Unrecognised inode hash code 10 for directory #37272081<br/> Jan 16 14:24:45 nbp1-mds kernel: LDISKFS-fs warning (device dm-2): dx_probe: Corrupt dir inode 37272081, running e2fsck is recommended.<br/> Jan 16 14:24:45 nbp1-mds kernel: Lustre: 87669:0:(mdd_object.c:2412:__mdd_readpage()) build page failed: -5!<br/> Jan 16 14:24:45 nbp1-mds kernel: Lustre: 87669:0:(mdd_object.c:2412:__mdd_readpage()) Skipped 1 previous similar message<br/> Jan 16 14:24:46 nbp1-mds kernel: Lustre: 86698:0:(mdd_orphans.c:371:orph_key_test_and_del()) Found orphan! Delete it<br/> Jan 16 14:24:46 nbp1-mds kernel: Lustre: 86698:0:(mdd_orphans.c:371:orph_key_test_and_del()) Skipped 1476 previous similar messages</p> <p>Here is list of server lustre rpms<br/> lustre-debuginfo-2.1.3-1nasS_2.6.32_279.2.1.el6.20120824.x86_64.lustre213.x86_64<br/> lustre-2.1.3-1nasS_2.6.32_279.2.1.el6.20120824.x86_64.lustre213.x86_64<br/> lustre-systemtap-0.9.3-2.noarch<br/> lustre-modules-2.1.3-1nasS_2.6.32_279.2.1.el6.20120824.x86_64.lustre213.x86_64<br/> lustre-ldiskfs-debuginfo-3.3.0-1nasS_2.6.32_279.2.1.el6.20120824.x86_64.lustre213.x86_64<br/> lustre-ldiskfs-3.3.0-1nasS_2.6.32_279.2.1.el6.20120824.x86_64.lustre213.x86_64<br/> lustre-source-2.1.3-1nasS_2.6.32_279.2.1.el6.20120824.x86_64.lustre213.x86_64<br/> lustre-tools-0.7.10-2.noarch<br/> lustre-tests-2.1.3-1nasS_2.6.32_279.2.1.el6.20120824.x86_64.lustre213.x86_64</p> <p>Linux nbp1-mds 2.6.32-279.2.1.el6.20120824.x86_64.lustre213 #1 SMP Mon Aug 27 15:02:12 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux</p> <p>our source tree is out on github</p> <p>should I remount the mdt for now so the clients can recover or hold off.</p> <p>The only actual errors are the LDISKFS-fs warnings. The rest are mostly standard restart. (errors should start with LustreError) <br/> It would have been better to have run fsck -fy prior to the remount.<br/> We are concerned that the issues reported by fsck -fn will like result in lost files if you run fsck -fy (the inodes with errors may be removed). You need to verify the health of the hardware prior to attempting any filesystem repair. <br/> If the hardware is healthy, fsck -fy will likely make the filesystem operable, but you will probably have to identify and restore the affected files.</p> <p>Should we upgrade our e2fsprogs and try. The hardware is differently healthy.</p> <p>The git source for the server can be found at<br/> <a href="https://github.com/jlan/lustre-nas" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/jlan/lustre-nas</a><br/> with branch nas-2.1.3, tag 2.1.3-1nasS.</p> <p>Do you have any documentation of identifying and restoring the affected files?</p> <p>Yes, you should upgrade to the latest e2fsprogs from <a href="http://downloads.whamcloud.com/public/e2fsprogs/" class="external-link" target="_blank" rel="nofollow noopener">http://downloads.whamcloud.com/public/e2fsprogs/</a></p> <p>after the new e2fsck most of there errors are</p> <p>Fast symlink 21070423 has EXTENT_FL set. Clear? no</p> <p>and </p> <p>Inode 11172447 symlink missing NUL terminator. Fix? no</p> <p>is that good or bad?</p> <p>Not horrible. We think you may lose some symlinks. I would go ahead and say 'y'</p> <p>hmmm e2fsck -vn SIGSEGV one it started checking directory structures</p> <p>nbp1-mds ~/newrpms # e2fsck <del>vn /dev/mapper/nbp1</del>-vg-mdt1 &gt; initcheck.out<br/> e2fsck 1.42.3.wc3 (15-Aug-2012)<br/> Signal (11) SIGSEGV si_code=SEGV_MAPERR fault addr=0x8<br/> e2fsck<span class="error">&#91;0x42d1dd&#93;</span><br/> /lib64/libc.so.6<span class="error">&#91;0x3a42432900&#93;</span><br/> e2fsck<span class="error">&#91;0x41160a&#93;</span><br/> e2fsck(e2fsck_process_bad_inode+0x693)<span class="error">&#91;0x419b13&#93;</span><br/> e2fsck<span class="error">&#91;0x419cfb&#93;</span><br/> /lib64/libext2fs.so.2(ext2fs_dblist_iterate2+0x87)<span class="error">&#91;0x7fffed8c9447&#93;</span><br/> e2fsck(e2fsck_pass2+0x10b)<span class="error">&#91;0x418ceb&#93;</span><br/> e2fsck(e2fsck_run+0x4f)<span class="error">&#91;0x40eb6f&#93;</span><br/> e2fsck(main+0xbd2)<span class="error">&#91;0x40cbf2&#93;</span><br/> /lib64/libc.so.6(__libc_start_main+0xfd)<span class="error">&#91;0x3a4241ecdd&#93;</span><br/> e2fsck<span class="error">&#91;0x409c49&#93;</span></p> <p>How did the extents feature get enabled on the MDT filesystem? This is not a standard formatting option, and not something that we test locally. It is likely the root cause of the problems that you are seeing.</p> <p>here is what we have. Should we remove "extent" Option?<br/> nbp1-mds ~/newrpms # tune2fs <del>l /dev/mapper/nbp1</del>-vg-mdt1<br/> tune2fs 1.42.3.wc3 (15-Aug-2012)<br/> Filesystem volume name: nbp1-MDT0000<br/> Last mounted on: /<br/> Filesystem UUID: aa5a51d9-5858-4bad-b6b6-668298ae0a7e<br/> Filesystem magic number: 0xEF53<br/> Filesystem revision #: 1 (dynamic)<br/> Filesystem features: has_journal ext_attr resize_inode dir_index filetype extent flex_bg dirdata sparse_super large_file huge_file uninit_bg dir_nlink extra_isize<br/> Filesystem flags: signed_directory_hash <br/> Default mount options: (none)<br/> Filesystem state: clean<br/> Errors behavior: Continue<br/> Filesystem OS type: Linux<br/> Inode count: 268435456<br/> Block count: 268435456<br/> Reserved block count: 0<br/> Free blocks: 224938984<br/> Free inodes: 204578188<br/> First block: 0<br/> Block size: 4096<br/> Fragment size: 4096<br/> Reserved GDT blocks: 960<br/> Blocks per group: 32768<br/> Fragments per group: 32768<br/> Inodes per group: 32768<br/> Inode blocks per group: 4096<br/> Flex block group size: 16<br/> Filesystem created: Wed Jun 8 19:54:48 2011<br/> Last mount time: Wed Jan 16 14:17:10 2013<br/> Last write time: Wed Jan 16 15:30:31 2013<br/> Mount count: 112<br/> Maximum mount count: 20<br/> Last checked: Wed Jun 8 19:54:48 2011<br/> Check interval: 15552000 (6 months)<br/> Next check after: Mon Dec 5 18:54:48 2011<br/> Lifetime writes: 7121 GB<br/> Reserved blocks uid: 0 (user root)<br/> Reserved blocks gid: 0 (group root)<br/> First inode: 11<br/> Inode size: 512<br/> Required extra isize: 28<br/> Desired extra isize: 28<br/> Journal inode: 8<br/> Default directory hash: half_md4<br/> Directory Hash Seed: 9bb09704-2b6c-4030-afe1-fcb7935216aa<br/> Journal backup: inode blocks</p> <p>Can you give us the output of 'tune2fs -l &lt;device&gt;' and tunefs.lustre -print &lt;device&gt; ?</p> <p>nbp1-mds ~/newrpms # tune2fs <del>l /dev/mapper/nbp1</del>-vg-mdt1<br/> tune2fs 1.42.3.wc3 (15-Aug-2012)<br/> Filesystem volume name: nbp1-MDT0000<br/> Last mounted on: /<br/> Filesystem UUID: aa5a51d9-5858-4bad-b6b6-668298ae0a7e<br/> Filesystem magic number: 0xEF53<br/> Filesystem revision #: 1 (dynamic)<br/> Filesystem features: has_journal ext_attr resize_inode dir_index filetype extent flex_bg dirdata sparse_super large_file huge_file uninit_bg dir_nlink extra_isize<br/> Filesystem flags: signed_directory_hash <br/> Default mount options: (none)<br/> Filesystem state: clean<br/> Errors behavior: Continue<br/> Filesystem OS type: Linux<br/> Inode count: 268435456<br/> Block count: 268435456<br/> Reserved block count: 0<br/> Free blocks: 224938984<br/> Free inodes: 204578188<br/> First block: 0<br/> Block size: 4096<br/> Fragment size: 4096<br/> Reserved GDT blocks: 960<br/> Blocks per group: 32768<br/> Fragments per group: 32768<br/> Inodes per group: 32768<br/> Inode blocks per group: 4096<br/> Flex block group size: 16<br/> Filesystem created: Wed Jun 8 19:54:48 2011<br/> Last mount time: Wed Jan 16 14:17:10 2013<br/> Last write time: Wed Jan 16 15:30:31 2013<br/> Mount count: 112<br/> Maximum mount count: 20<br/> Last checked: Wed Jun 8 19:54:48 2011<br/> Check interval: 15552000 (6 months)<br/> Next check after: Mon Dec 5 18:54:48 2011<br/> Lifetime writes: 7121 GB<br/> Reserved blocks uid: 0 (user root)<br/> Reserved blocks gid: 0 (group root)<br/> First inode: 11<br/> Inode size: 512<br/> Required extra isize: 28<br/> Desired extra isize: 28<br/> Journal inode: 8<br/> Default directory hash: half_md4<br/> Directory Hash Seed: 9bb09704-2b6c-4030-afe1-fcb7935216aa<br/> Journal backup: inode blocks<br/> nbp1-mds ~/newrpms # tunefs.lustre <del>print /dev/mapper/nbp1</del>-vg-mdt1<br/> checking for existing Lustre data: found CONFIGS/mountdata<br/> Reading CONFIGS/mountdata</p> <p> Read previous values:<br/> Target: nbp1-MDT0000<br/> Index: 0<br/> Lustre FS: nbp1<br/> Mount type: ldiskfs<br/> Flags: 0x401<br/> (MDT )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters: mgsnode=10.151.26.26@o2ib lov.stripesize=1048576 lov.stripecount=4 mdd.quota_type=u mdt.identity_info=/usr/sbin/l_getidentity</p> <p> Permanent disk data:<br/> Target: nbp1-MDT0000<br/> Index: 0<br/> Lustre FS: nbp1<br/> Mount type: ldiskfs<br/> Flags: 0x441<br/> (MDT update )<br/> Persistent mount opts: iopen_nopriv,user_xattr,errors=remount-ro<br/> Parameters: mgsnode=10.151.26.26@o2ib lov.stripesize=1048576 lov.stripecount=4 mdd.quota_type=u mdt.identity_info=/usr/sbin/l_getidentity rint</p> <p>Writing CONFIGS/mountdata<br/> nbp1-mds ~/newrpms # </p> <p>sorry, looks like our replies crossed. I see you already supplied the tune2fs</p> <p>Can you please run e2fsck under gdb with "-n" option and paste the resulting stack trace here? I can't see enough of where the problem is above.</p> <p>If you have a spare SATA disk I would recommend making a full backup of the MDT device with "dd", since this would go relatively quickly (maybe at 100MB/s, so a few hours for the full backup). This may be important in case running the real e2fsck doesn't go well (depending on what corruption is being seen).</p> <p>rogram received signal SIGSEGV, Segmentation fault.<br/> check_symlink (ctx=0x645060, pctx=0x0, ino=8503956, inode=0x7fffffffe180, buf=&lt;value optimized out&gt;) at pass1.c:190<br/> 190 if (ext2fs_extent_open2(ctx-&gt;fs, pctx-&gt;ino, inode, &amp;handle))<br/> Missing separate debuginfos, use: debuginfo-install db4-4.7.25-16.el6.x86_64 glibc-2.12-1.47.el6.x86_64 libblkid-2.17.2-12.4.el6.x86_64 libuuid-2.17.2-12.4.el6.x86_64<br/> (gdb) bt<br/> #0 check_symlink (ctx=0x645060, pctx=0x0, ino=8503956, inode=0x7fffffffe180, buf=&lt;value optimized out&gt;) at pass1.c:190<br/> #1 0x0000000000419b13 in e2fsck_process_bad_inode (ctx=0x645060, dir=&lt;value optimized out&gt;, ino=8503956, buf=0x66eda0 "") at pass2.c:1402<br/> #2 0x0000000000419cfb in check_dir_block (fs=0x645510, db=0x7fffe11d2a68, priv_data=0x7fffffffe570) at pass2.c:1044<br/> #3 0x00007fffed8c9447 in ext2fs_dblist_iterate2 (dblist=0x649100, func=0x419b70 &lt;check_dir_block&gt;, priv_data=0x7fffffffe570) at dblist.c:239<br/> #4 0x0000000000418ceb in e2fsck_pass2 (ctx=0x645060) at pass2.c:148<br/> #5 0x000000000040eb6f in e2fsck_run (ctx=0x645060) at e2fsck.c:226<br/> #6 0x000000000040cbf2 in main (argc=&lt;value optimized out&gt;, argv=&lt;value optimized out&gt;) at unix.c:1852</p> <p>dd is going to take 20hours! I have created a snapshot of the volume so we can run the fsck on it</p> <p>the latest e2fsck has a glitch, and I uploaded a patch for it (<a href="http://review.whamcloud.com/5045" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5045</a>)</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedHeader panelHeader" style="border-bottom-width: 1px;"><b>commit message</b></div><div class="preformattedContent panelContent"> <pre> LU-2627 e2fsck: check_symlink() SIGSEGV Since e2fsck_pass1_check_symlink()-&gt; check_symlink(ctx, NULL, ino, inode, buf), we should use 'ino' instead of 'pctx-&gt;ino' in check_symlink(). </pre> </div></div> <p>this is just for the SIGSEGV issue.</p> <p>As Andreas suggested, run e2fsck with run the patched e2fsck with "-n" under gdb and paste the resulting stack trace so that we can diagnose what the problem could be.</p> <p>Running e2fsck with '-n' won't change the disk device.</p> <p>Looks like the patch got us past the SIGSEGV. But looks like the fsck will remove all symlinks!</p> <p>It is calling out what looks like all symlinks as invalid.</p> <p>This is why we urge a backup before you fsck -y. Will the snapshot allow you to restore the symlinks?</p> <p>It is not clear to me why it is removing all the symlunks. Is it because of the extent option? How would we restore the symlinks from the dd backup?</p> <p>here is the summary of the test fsck.</p> <p>nbp1-MDT0000: ********** WARNING: Filesystem still has errors **********</p> <p> 63829418 inodes used (23.78%, out of 268435456)<br/> 3242 non-contiguous files (0.0%)<br/> 35652 non-contiguous directories (0.1%)</p> <ol> <li>of inodes with ind/dind/tind blocks: 0/0/0<br/> Extent depth histogram: 63096846/16544/13<br/> 43499835 blocks used (16.20%, out of 268435456)<br/> 0 bad blocks<br/> 21886 large files</li> </ol> <p> 62047854 regular files<br/> 773172 directories<br/> 0 character device files<br/> 0 block device files<br/> 20 fifos<br/> 6652 links<br/> 612271 symbolic links (349798 fast symbolic links)<br/> 65 sockets<br/> ------------<br/> 63836054 files</p> <p>I can upload the full upload of the output.</p> <p><span class="error">&#91;root@pladmin4:~/mhanafi&#93;</span>$ grep invalid fck.out | wc -l<br/> 396020</p> <p>We are not certain that the symlinks would be deleted, in a case such as this it is always desirable to have a backup, if possible.</p> <p>please compress and upload fck.out.</p> <p>I want to check whether those invalid symlink file are those long symlink which miss NUL terminator. Something like</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedHeader panelHeader" style="border-bottom-width: 1px;"><b>an example</b></div><div class="preformattedContent panelContent"> <pre>Pass 1: Checking inodes, blocks, and sizes Inode 121351 symlink missing NUL terminator. Fix? no ... ... Pass 2: Checking directory structure Symlink /path/to/long/symlink/file (inode #121351) is invalid. Clear? no ... </pre> </div></div> <p>If it's this case, latest e2fsck should be capable of fixing them. (like <a href="https://jira.whamcloud.com/browse/LU-1540" title="e2fsck remove too many symlinks" class="issue-link" data-issue-key="LU-1540"><del>LU-1540</del></a> indicates)</p> <p>file is uploaded</p> <p>Filed <a href="https://jira.whamcloud.com/browse/LU-2634" title="short symlinks on MDT with &quot;extents&quot; have EXT4_EXTENTS_FL set" class="issue-link" data-issue-key="LU-2634"><del>LU-2634</del></a> for tracking issue with EXT4_EXTENTS_FL set on symlinks for MDT with "extents" feature enabled.</p> <p>Bobijam, I think that the problem is with e2fsck rejecting short symlinks with the EXT4_EXTENTS_FL set. The <a href="https://jira.whamcloud.com/browse/LU-1540" title="e2fsck remove too many symlinks" class="issue-link" data-issue-key="LU-1540"><del>LU-1540</del></a> NUL termination problem appears that it would be fixed correctly with the current e2fsck. This EXT4_EXTENTS_FL appears to be a bug in the osd-ldiskfs code, if "extents" is enabled, for which I've filed <a href="https://jira.whamcloud.com/browse/LU-2634" title="short symlinks on MDT with &quot;extents&quot; have EXT4_EXTENTS_FL set" class="issue-link" data-issue-key="LU-2634"><del>LU-2634</del></a>. Since we never format the MDT with "extents", we have never seen such a problem in our testing.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Inode 9482890 symlink missing NUL terminator. Fix? no Inode 9482897 symlink missing NUL terminator. Fix? no Fast symlink 9482914 has EXTENT_FL set. Clear? no Fast symlink 9482917 has EXTENT_FL set. Clear? no Fast symlink 9482921 has EXTENT_FL set. Clear? no </pre> </div></div> <p>It makes sense to change e2fsck to accept such inodes and just clear the EXT4_EXTENTS_FL instead of considering it corrupted. That will allow recovering the filesystem without the need to restore the symlinks (which would just get EXT4_EXTENTS_FL set again, until <a href="https://jira.whamcloud.com/browse/LU-2634" title="short symlinks on MDT with &quot;extents&quot; have EXT4_EXTENTS_FL set" class="issue-link" data-issue-key="LU-2634"><del>LU-2634</del></a> is fixed).</p> <p>This was a 1.8.x filesystem that was upgraded. So I think the extent option is leftover from the 1.8.x format.</p> <p>Looking at the e2fsck code, it appears that it will correctly remove just the EXTENT_FL flag, rather than clear the whole inode:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">if</span> (extent_fs &amp;&amp; (inode-&gt;i_flags &amp; EXT4_EXTENTS_FL) &amp;&amp; LINUX_S_ISLNK(inode-&gt;i_mode) &amp;&amp; !ext2fs_inode_has_valid_blocks2(fs, inode) &amp;&amp; fix_problem(ctx, PR_1_FAST_SYMLINK_EXTENT_FL, &amp;pctx)) { inode-&gt;i_flags &amp;= ~EXT4_EXTENTS_FL; e2fsck_write_inode(ctx, ino, inode, <span class="code-quote">"pass1"</span>); } </pre> </div></div> <p>so the only confusion is that the PR_1_FAST_SYMLINK_EXTENT_FL problem code is asking "Clear", which might be confusing to some (including myself) as asking whether the <em>inode</em> should be cleared instead of the <em>flag</em> being cleared. I will submit a patch to fix this.</p> <p>The later errors:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Symlink /ROOT/pheimbac/ecco/2013-01-seaice-adjoint/MITgcm_latest/mysetups/arctic210x192x50/build_forw/timeave_cumulate.F (inode #68169598) is invalid. Clear? no Symlink /ROOT/pheimbac/ecco/2013-01-seaice-adjoint/MITgcm_latest/mysetups/arctic210x192x50/build_forw/cal_compdates.F (inode #68169136) is invalid. Clear? no </pre> </div></div> <p>should not be hit if the earlier checks to clear EXT4_EXTENT_FL had been allowed to clear this flag from the short symlinks.</p> <p>There are some further errors, much later in the log. There are ~20 of the following errors in Pass 2:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Pass 2: Checking directory structure Second entry 'IE_t040101_000000.log' (inode=18364943) in directory inode 1837308 5 should be '..' Fix? no Entry '..' in /ROOT/xjia/Saturn/run_IdealizedSW_notilt_1e275_newgrid2_highorder/ RESULTS/run_all/IE (18373085) is duplicate '..' entry. Fix? no Entry '..' in /ROOT/xjia/Saturn/run_IdealizedSW_notilt_1e275_newgrid2_highorder/ RESULTS/run_all/IE (18373085) is duplicate '..' entry. Fix? no Entry '..' in /ROOT/xjia/Saturn/run_IdealizedSW_notilt_1e275_newgrid2_highorder/ RESULTS/run_all/IE (18373085) is a link to directory /ROOT/xjia/Saturn/run_Ideal izedSW_notilt_1e275_newgrid2_highorder/RESULTS/run_all (13221653). Clear? no </pre> </div></div> <p>that appear a bit unusual, but are not fatally broken. There are ~20 matching errors for the unfixed ".." entries later in Pass 3:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Pass 3: Checking directory connectivity '..' in /ROOT/xjia/Saturn/run_IdealizedSW_notilt_1e275_newgrid2_highorder/RESULTS/run_all/IE (18373085) is &lt;The NULL inode&gt; (0), should be /ROOT/xjia/Saturn/run_IdealizedSW_notilt_1e275_newgrid2_highorder/RESULTS/run_all (13221653). Fix? no </pre> </div></div> <p>and a few minor errors in Pass 3A:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Pass 3A: Optimizing directories Duplicate entry 'c_t_f.x' in /ROOT/aiannett/NCC/Testing/Back-Face-Step (77623393) found. Clear? no Entry 'c_t_f.x' in /ROOT/aiannett/NCC/Testing/Back-Face-Step (77623393) has a non-unique filename. Rename to c_t_f.~0? no Duplicate entry 'b1b2b3.x' in /ROOT/aiannett/NCC/Testing/Back-Face-Step (77623393) found. Clear? no Entry 'b1b2b3.x' in /ROOT/aiannett/NCC/Testing/Back-Face-Step (77623393) has a non-unique filename. Rename to b1b2b3~0? no </pre> </div></div> <p>It appears that the entries that would be "fixed" in Pass 2 will likely appear in lost+found once they are fixed, and if you want to recover those files you could mount the MDT locally with <tt>mount -t ldiskfs</tt> and rename them from <tt>.../lost+found/#inode</tt> to the path given for each inode number.</p> <p>I think you could go ahead with running <tt>e2fsck -fy</tt> on the snapshot, mount the snapshot MDT filesystem locally as ldiskfs to verify a handful of the symlinks are still intact, and check lost+found for the ~20 or so inodes that would need to be fixed (you could even write a short script to rename them if downtime is critical). If that works OK, then when you take the real MDT filesystem offline for repair, please make another snapshot at that time, run the <tt>e2fsck -fy</tt> on the real MDT, mount as ldiskfs and repair the files in lost+found before unmounting and remounting it again as lustre.</p> <p>In order to get the number of messages in the e2fsck log to a manageable number, I filtered out all of the duplicate messages:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>egrep -v "^$|^Fast symlink .* EXTENT_FL|^Inode .* missing NUL terminator|^Clear" e2fsck.log &gt; e2fsck-filtered.log </pre> </div></div> <p>I had also filtered out "<tt>^Symlink.*is invalid</tt>" messages, but I don't think you should hit them during the repairing e2fsck run.</p> <p>I also see in your MDT feature list that there is the "dirdata" feature enabled, but this is definitely NOT a feature that would have been enabled with a filesystem formatted with 1.8. Also, the ".." corruption is definitely not random.</p> <p>Did you perhaps run the Xyratex "upgrade" tool on the MDT filesystem?</p> <p>I believe that this would be the root cause of the ".." corruption. My understanding is that it was deleting the ".." entry to add the FID, and then re-inserting it into the directory, but ext4/e2fsck require that the ".." entry immediately follow the "." entry at the start.</p> <p>We did not use the xyratex upgrade tool. But we added that dirdata option at some point. Should we remove that option?</p> <p>Uploading the fsck ran on the snap. Please review before we run on the real mdt device.</p> <p>Looking at the test e2fsck log, one new directory is getting yet a different error related to the "." entry:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Directory entry for '.' in /ROOT/msekula/fun/camrad (13208388) is big. Split? yes Missing '..' in directory inode 13208388. Fix? yes Setting filetype for entry '..' in /ROOT/msekula/fun/camrad (13208388) to 2. Entry '..' in /ROOT/msekula/fun/camrad (13208388) is duplicate '..' entry. Fix? yes </pre> </div></div> <p>I suspect that there is some code in e2fsck or in ldiskfs that is not handling the dirdata field correctly. It likely relates to <a href="https://jira.whamcloud.com/browse/LU-2638" title="corruption of MDT &quot;..&quot; entry in some ldiskfs directories" class="issue-link" data-issue-key="LU-2638"><del>LU-2638</del></a>. There are several files moved to <tt>lost+found</tt> as I suspected, but it looks like the majority of symlinks are fine.</p> <p>It doesn't seem that a large number of directories will be repaired, so I think it makes sense to go ahead and fix the real MDT at this point. The only other thing you might want to check before doing the final is if you run "<tt>e2fsck -fy</tt>" on the snapshot a second time that it passes cleanly without any repairs. About 30 directories will be moved to lost+found, but they can be moved back to their correct location, and nothing should be lost.</p> <p>The next question to figure out what has caused this problem. When did you upgrade to 2.1? Were these directories existing before the upgrade from 1.8, or were they created afterward? How large are the directories (number of entries = "find ${directory} -print | wc -l", size of directory = "ls -ld ${directory}")? Do you know if the directories where renamed after they were created? How long has it been since you last ran e2fsck? Have you run it since the upgrade?</p> <p>It has been a very long time since we have ran e2fsck and that was during the 1.8.x code. We have never ran e2fsck since moving to 2.1. </p> <p>Should we remove the dirdata options? </p> <p>I will check the date and size of the directories. We may want to just archive these and restore them after the fsck or tar/delete/untar them.</p> <p>The "dirdata" option is enabled by default for 2.x filesystems, but I don't think it is necessarily advisable to disable it at this time. It does appear at first glance that running e2fsck after removing the dirdata feature would handle this correctly and clear the extra dirdata flag in each dirent, but we haven't tested this at all, and it would also cause the MDS to become considerably slower.</p> <p>So far I don't see any indication besides the mixup with ".." entries that there is anything seriously wrong with these directories. The bytes at the start of the directory are used for ".", "..", and the htree index on directories over 4kB in size, and not user data. e2fsck should regenerate all of the needed information from redundant information elsewhere, except being able to move the entry from lost+found back to the proper place in the tree.</p> <p>What is your current state? What help can we give you?</p> <p>At this point we have been able to run fsck on the mdt and have recovered from the errors.</p> <p>Is the issue closed, or is there some other help we can give you?</p> <p>We seem to have hit this issue again on the same filesystem.</p> <p>pfe1 ~ # ls -l /nobackupp1/xmeng/run_sc_anisopi/run06_dipole_semiimpl_nohyp_taug<br/> r_60000ss/SC<br/> ls: reading directory /nobackupp1/xmeng/run_sc_anisopi/run06_dipole_semiimpl_noh<br/> yp_taugr_60000ss/SC: Input/output error<br/> total 0</p> <p>from the mdt<br/> Feb 8 06:50:58 nbp1-mds kernel: LDISKFS-fs warning (device dm-4): dx_probe: Unrecognised inode hash code 18 for directory #17309149<br/> Feb 8 06:50:58 nbp1-mds kernel: LDISKFS-fs warning (device dm-4): dx_probe: Corrupt dir inode 17309149, running e2fsck is recommended.<br/> Feb 8 06:51:57 nbp1-mds kernel: LDISKFS-fs warning (device dm-4): dx_probe: Unrecognised inode hash code 8 for directory #17309159<br/> Feb 8 06:51:57 nbp1-mds kernel: LDISKFS-fs warning (device dm-4): dx_probe: Corrupt dir inode 17309159, running e2fsck is recommended.<br/> Feb 8 08:35:12 nbp1-mds kernel: LDISKFS-fs warning (device dm-4): dx_probe: Unrecognised inode hash code 15 for directory #130557236<br/> Feb 8 08:35:12 nbp1-mds kernel: LDISKFS-fs warning (device dm-4): dx_probe: Corrupt dir inode 130557236, running e2fsck is recommended.<br/> Feb 8 11:45:38 nbp1-mds kernel: LDISKFS-fs warning (device dm-4): dx_probe: Unrecognised inode hash code 3 for directory #157287952<br/> Feb 8 11:45:39 nbp1-mds kernel: LDISKFS-fs warning (device dm-4): dx_probe: Corrupt dir inode 157287952, running e2fsck is recommended.<br/> Feb 8 11:46:07 nbp1-mds kernel: LDISKFS-fs warning (device dm-4): dx_probe: Unrecognised inode hash code 4 for directory #157331367<br/> Feb 8 11:46:07 nbp1-mds kernel: LDISKFS-fs warning (device dm-4): dx_probe: Corrupt dir inode 157331367, running e2fsck is recommended.</p> <p>This problem will persist for large 1.8 directories that are renamed until a version of the <a href="https://jira.whamcloud.com/browse/LU-2638" title="corruption of MDT &quot;..&quot; entry in some ldiskfs directories" class="issue-link" data-issue-key="LU-2638"><del>LU-2638</del></a> patch <a href="http://review.whamcloud.com/5179" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/5179</a> is applied. For the short term, until this patch is applied, it is possible to disable the dirdata feature on the unmounted MDT filesystem:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>tune2fs -O dirdata /dev/mdtdev </pre> </div></div> <p>though this will have some negative performance impact for all newly-created files when doing name lookups and "ls -l".</p> <p>uploading fsck output for review before we run it for real.</p> <p>There is nothing new in the fsck output compared to last time. I think you should go ahead and run fsck.</p> <p>As per NASA ok to close ticket</p> Duplicate LU-3519 Related LU-2634 LU-2638 Development Rank 1|hzvfgf: Rank (Obsolete) 6149 Severity