https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4402 Lustre <p>Starting with an otherwise operating filesystem, we had a nfs issue on the management server that does nfsroot to the nodes. This caused the nodes to hang on shell probes, ssh, etc, but lustre appeared to work okay, until the mount came back and there was a spew of I/O errors. We had -o errors=panic, so the nodes rebooted and we have a crash dump as well. A few of the interesting/disturbing messages are below and a complete log capture of the interval is attached. We rebooted every single one of our lustre systems that mounted this nfsroot and started up lustre. At this point, an e2fsck seems prudent given the messages? Please advise.</p> <p>And for clarity's sake, this is from a completely separate system than <a href="https://jira.whamcloud.com/browse/LU-486" title="ldiskfs_valid_block_bitmap: Invalid block bitmap" class="issue-link" data-issue-key="LU-486"><del>LU-486</del></a> that I just updated.</p> <p>Dec 19 14:07:28 atlas-oss3b4 kernel: <span class="error">&#91;1987655.565953&#93;</span> end_request: I/O error, dev dm-0, sector 2641342080<br/> ... more of those ...<br/> Dec 19 14:07:34 atlas-oss2f4 kernel: <span class="error">&#91;1987662.210829&#93;</span> LDISKFS-fs error (device dm-8): ldiskfs_ext_find_extent: bad header/extent in inode #395571: invalid magic - magic 5fa6, entries 39658, max 42407(0), depth 37176(0)<br/> Dec 19 14:09:02 atlas-linkfarm kernel: <span class="error">&#91;1987621.160868&#93;</span> LustreError: 13071:0:(tgt_lastrcvd.c:577:tgt_client_new()) linkfarm-MDT0000: Failed to write client lcd at idx 18888, rc -30<br/> Dec 19 14:09:11 atlas-mds1 kernel: <span class="error">&#91;1355985.425764&#93;</span> LustreError: 15139:0:(osp_precreate.c:484:osp_precreate_send()) atlas1-OST0043-osc-MDT0000: can\'t precreate: rc = -30<br/> Dec 19 14:09:11 atlas-mds1 kernel: <span class="error">&#91;1355985.439222&#93;</span> LustreError: 15139:0:(osp_precreate.c:484:osp_precreate_send()) Skipped 990 previous similar messages<br/> Dec 19 14:09:11 atlas-mds1 kernel: <span class="error">&#91;1355985.451122&#93;</span> LustreError: 15139:0:(osp_precreate.c:989:osp_precreate_thread()) atlas1-OST0043-osc-MDT0000: cannot precreate objects: rc = -30<br/> Dec 19 14:09:11 atlas-mds1 kernel: <span class="error">&#91;1355985.465640&#93;</span> LustreError: 15139:0:(osp_precreate.c:989:osp_precreate_thread()) Skipped 990 previous similar messages<br/> Dec 19 14:09:55 atlas-mds1 kernel: <span class="error">&#91;1356029.474926&#93;</span> INFO: task mdt00_027:12952 blocked for more than 120 seconds.<br/> Dec 19 14:10:46 atlas-oss1d1 kernel: <span class="error">&#91;691120.297183&#93;</span> LDISKFS-fs error (device dm-1): ldiskfs_ext_find_extent: bad header/extent in inode #395600: invalid magic - magic a7bc, entries 21131, max 744(0), depth 0(0)<br/> Dec 19 14:12:42 atlas-oss2d6 kernel: <span class="error">&#91;1987967.199569&#93;</span> Buffer I/O error on device dm-9, logical block 5638<br/> Dec 19 14:12:42 atlas-oss2d6 kernel: <span class="error">&#91;1987967.199571&#93;</span> lost page write due to I/O error on dm-9<br/> Dec 19 14:12:42 atlas-oss2d6 kernel: <span class="error">&#91;1987967.199578&#93;</span> LDISKFS-fs error (device dm-9): kmmpd:<br/> Dec 19 14:12:42 atlas-oss2d6 kernel: <span class="error">&#91;1987967.199582&#93;</span> Aborting journal on device dm-9-8.<br/> Dec 19 14:12:42 atlas-oss2d6 kernel: <span class="error">&#91;1987967.199586&#93;</span> Buffer I/O error on device dm-9, logical block 137<br/> Dec 19 14:12:42 atlas-oss2d6 kernel: <span class="error">&#91;1987967.199588&#93;</span> Error writing to MMP block<br/> Dec 19 14:12:42 atlas-oss2d6 kernel: <span class="error">&#91;1987967.199589&#93;</span> lost page write due to I/O error on dm-9<br/> Dec 19 14:12:42 atlas-oss2d6 kernel: <span class="error">&#91;1987967.199590&#93;</span><br/> Dec 19 14:12:42 atlas-oss2d6 kernel: <span class="error">&#91;1987967.199592&#93;</span> LDISKFS-fs (dm-9): Remounting filesystem read-only<br/> Dec 19 14:13:25 atlas-oss2f5 kernel: <span class="error">&#91;1988014.344558&#93;</span> LDISKFS-fs error (device dm-6): ldiskfs_ext_find_extent: bad header/extent in inode #268955: invalid magic - magic e79d, entries 37634, max 32686(0), depth 47774(0)</p> <p>Dec 19 14:14:50 atlas-oss1a5 kernel: <span class="error">&#91;691359.246757&#93;</span> LDISKFS-fs error (device dm-9): file system corruption: inode #591204 logical block 447 mapped to 137004702958273 (size 1)<br/> Dec 19 14:14:53 atlas-oss1a5 kernel: <span class="error">&#91;691362.375749&#93;</span> LDISKFS-fs error (device dm-9): file system corruption: inode #591204 logical block 447 mapped to 137004702958273 (size 1)<br/> Dec 19 14:14:57 atlas-oss1a5 kernel: <span class="error">&#91;691366.367144&#93;</span> LDISKFS-fs error (device dm-9): file system corruption: inode #591204 logical block 447 mapped to 137004702958273 (size 1)<br/> Dec 19 14:15:02 atlas-oss1a5 kernel: <span class="error">&#91;691371.356427&#93;</span> LDISKFS-fs error (device dm-9): file system corruption: inode #591204 logical block 447 mapped to 137004702958273 (size 1)<br/> Dec 19 14:15:08 atlas-oss1a5 kernel: <span class="error">&#91;691377.343445&#93;</span> LDISKFS-fs error (device dm-9): file system corruption: inode #591204 logical block 447 mapped to 137004702958273 (size 1)<br/> Dec 19 14:15:15 atlas-oss1a5 kernel: <span class="error">&#91;691384.328232&#93;</span> LDISKFS-fs error (device dm-9): file system corruption: inode #591204 logical block 447 mapped to 137004702958273 (size 1)<br/> Dec 19 14:15:16 atlas-oss2e1 kernel: <span class="error">&#91;1988136.371625&#93;</span> LDISKFS-fs error (device dm-3): ldiskfs_ext_find_extent: bad header/extent in inode #330052: invalid magic - magic 53be, entries 21067, max 517(0),<br/> depth 0(0)<br/> Dec 19 14:15:23 atlas-oss1a5 kernel: <span class="error">&#91;691392.310962&#93;</span> LDISKFS-fs error (device dm-9): file system corruption: inode #591204 logical block 447 mapped to 137004702958273 (size 1)<br/> Dec 19 14:15:32 atlas-oss1a5 kernel: <span class="error">&#91;691401.291535&#93;</span> LDISKFS-fs error (device dm-9): file system corruption: inode #591204 logical block 447 mapped to 137004702958273 (size 1)<br/> Dec 19 14:15:42 atlas-oss1a5 kernel: <span class="error">&#91;691411.269872&#93;</span> LDISKFS-fs error (device dm-9): file system corruption: inode #591204 logical block 447 mapped to 137004702958273 (size 1)</p> <p>Dec 19 14:21:50 atlas-oss1d3 kernel: <span class="error">&#91;691780.266369&#93;</span> LDISKFS-fs error (device dm-2): ldiskfs_ext_find_extent: bad header/extent in inode #199528: invalid magic - magic 0, entries 0, max 0(0), depth 0(0<br/> )</p> <p>Dec 19 14:22:41 atlas-oss1d3 kernel: <span class="error">&#91;691831.895277&#93;</span> LDISKFS-fs error (device dm-2): ldiskfs_ext_get_blocks: inode #199528: (comm ll_ost_io02_007) bad extent address iblock: 447, depth: 1 pblock 0</p> RHEL6.4/distro IB/2.6.32-358.18.1.el6<br/> LU-4402

Ldiskfs errors ldiskfs_ext_find_extent, ldiskfs_ext_get_blocks, corruption

Bug Major Resolved Fixed Alex Zhuravlev Blake Caldwell Fri, 20 Dec 2013 04:29:37 +0000 Sat, 8 Feb 2014 05:35:02 +0000 Sat, 1 Feb 2014 01:55:31 +0000 Lustre 2.4.1 0 6 <p>&gt;Dec 19 14:07:25 atlas-mgs2 kernel: <span class="error">&#91;1988887.590310&#93;</span> sd 6:0:8:1: <span class="error">&#91;sdw&#93;</span> Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.</p> <p>sounds like at some point underlying device changed?</p> <p>Well I'm going to say that it's not directly correlated, but there's more to it. That host is not actually part of the filesystem. It connects to, but does not have any LUNs mounted on the same SAS attached array that the MGS and MDS share, so perhaps it indicates an event on one of the other systems. We've grown accustomed to seeing those messages with RHEL hosts when something happens on the storage array (object storage arrays from a different brand do this too) and it has not indicated an actual LUN assignment change. It could be caused from an event on one of the other systems atlas-mds1, atlas-mds3, atlas-mgs1 such as an IO error after the nfsroot returned.</p> <p>thanks for clarification.<br/> can it be that some configuration changes on the storage system caused following I/O errors:<br/> Dec 19 14:07:28 atlas-oss3b4 kernel: <span class="error">&#91;1987655.605866&#93;</span> end_request: I/O error, dev dm-0, sector 2641344352<br/> ?</p> <p>the confusing thing is that it's number of the systems started to observe corruptions:<br/> Dec 19 14:10:46 atlas-oss1d1 kernel: <span class="error">&#91;691120.297183&#93;</span> LDISKFS-fs error (device dm-1): ldiskfs_ext_find_extent: bad header/extent in inode #395600: invalid magic - magic a7bc, entries 21131, max 744(0), depth 0(0)<br/> Dec 19 14:11:34 atlas-oss1b2 kernel: <span class="error">&#91;691163.534948&#93;</span> LDISKFS-fs error (device dm-18): ldiskfs_ext_find_extent: bad header/extent in inode #268868: invalid magic - magic 222a, entries 21150, max 9(0), depth 0(0)<br/> Dec 19 14:11:44 atlas-oss1a5 kernel: <span class="error">&#91;691173.077090&#93;</span> LDISKFS-fs error (device dm-9): ldiskfs_ext_find_extent: bad header/extent in inode #591204: invalid magic - magic a9bf, entries 21131, max 868(0), depth 0(0)<br/> Dec 19 14:11:55 atlas-oss1d4 kernel: <span class="error">&#91;691184.758850&#93;</span> LDISKFS-fs error (device dm-4): ldiskfs_ext_find_extent: bad header/extent in inode #269305: invalid magic - magic b84, entries 44395, max 21257(0), depth 62507(0)</p> <p>I agree its strange the number of errors. atlas-oss1d1 and atlas-oss1b2 use physically separate storage arrays. Of those listed only atlas-oss1d1 and atlas-oss1d4 share the same storage device. I checked the logs on the OST storage controllers and all that they saw was the hosts log out and then back in when they were rebooted. </p> <p>The common piece that sticks out to me is that all systems had their nfsroot filesystems disrupted. They have recovered from this transparently a hundred times before.</p> <p>Are there any next steps on this ticket given the information that has been posted? ie. should this ticket be closed?</p> <p>What would be the best way of validating what it is saying about this message... assuming the inode still exists? Since this is the ldiskfs layer, how do we correlate to inode #395571? Debugfs? Anything live without a downtime for e2fsck?</p> <p>Dec 19 14:07:34 atlas-oss2f4 kernel: <span class="error">&#91;1987662.210829&#93;</span> LDISKFS-fs error (device dm-8): ldiskfs_ext_find_extent: bad header/extent in inode #395571: invalid magic - magic 5fa6, entries 39658, max 42407(0), depth 37176(0)</p> <p>you can try on the mounted filesystem: debugfs -R "stat &lt;395571&gt;"</p> <p>Thanks. It turns out that since just had a downtime and I was able to run e2fsck across all OSTs. There were only 2 problems found and they did not correlate to the issues in the log messages. 1 problem was with inode 3, which I gather is a user quota file. The 2nd inode had a fid, but could not be found with fid2path. If the i_size for inode 3 difference is tolerable, then I believe we have arrived at the end of the road with this case.</p> <p><span class="error">&#91;root@atlas-oss2e4 ~&#93;</span># e2fsck -f /dev/mapper/atlas-ddn2e-l22<br/> e2fsck 1.42.7.wc1 (12-Apr-2013)<br/> Pass 1: Checking inodes, blocks, and sizes<br/> Inode 3, i_size is 92160, should be 106496. Fix&lt;y&gt;? yes<br/> Pass 2: Checking directory structure<br/> Pass 3: Checking directory connectivity<br/> Pass 4: Checking reference counts<br/> Pass 5: Checking group summary information<br/> <span class="error">&#91;root@atlas-oss2i1 ~&#93;</span># e2fsck -f /dev/mapper/atlas-ddn2i-l2<br/> e2fsck 1.42.7.wc1 (12-Apr-2013)<br/> Pass 1: Checking inodes, blocks, and sizes<br/> Inode 209, i_size is 25165824, should be 26214400. Fix&lt;y&gt;? yes<br/> Pass 2: Checking directory structure<br/> Pass 3: Checking directory connectivity<br/> Pass 4: Checking reference counts<br/> Pass 5: Checking group summary information</p> <p><span class="error">&#91;root@atlas-oss2i1 ~&#93;</span># debugfs -R "stat &lt;209&gt;" /dev/mapper/atlas-ddn2i-l2<br/> debugfs 1.42.7.wc1 (12-Apr-2013)<br/> Inode: 209 Type: regular Mode: 0666 Flags: 0x80000<br/> Generation: 3823026230 Version: 0x00000003:00036bbd<br/> User: 9032 Group: 18319 Size: 25165824<br/> File ACL: 0 Directory ACL: 0<br/> Links: 1 Blockcount: 51208<br/> Fragment: Address: 0 Number: 0 Size: 0<br/> ctime: 0x52ceeba5:00000000 &#8211; Thu Jan 9 13:34:13 2014<br/> atime: 0x52ceeba5:00000000 &#8211; Thu Jan 9 13:34:13 2014<br/> mtime: 0x52ceeba5:00000000 &#8211; Thu Jan 9 13:34:13 2014<br/> crtime: 0x524b4897:d033afac &#8211; Tue Oct 1 18:11:35 2013<br/> Size of extra inode fields: 28<br/> Extended attributes stored in inode body: <br/> lma = "00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 7c 00 00 00 00 00 00 00 " (24)<br/> lma: fid=<span class="error">&#91;0x1000000:0x7c000000:0x0&#93;</span> compat=0 incompat=0<br/> fid = "88 14 00 00 02 00 00 00 38 02 00 00 03 00 00 00 " (16)<br/> fid: parent=<span class="error">&#91;0x200001488:0x238:0x0&#93;</span> stripe=3<br/> EXTENTS:<br/> (ETB0):60304384, (0-255):60295936-60296191, (256-511):60296960-60297215, (512-1023):60298240-60298751, (1024-2047):60302336-60303359, (2048-4095):60305408-60307455, (4096-6143):60317696-60319743, (6144-6399):60327936-60328191</p> <p><span class="error">&#91;root@atlas-oss2i1 ~&#93;</span># debugfs -R "ncheck 209" /dev/mapper/atlas-ddn2i-l2<br/> debugfs 1.42.7.wc1 (12-Apr-2013)<br/> Inode Pathname<br/> 209 /O/0/d28/124<br/> <span class="error">&#91;root@client1 etc&#93;</span># lfs fid2path /lustre/atlas1 <span class="error">&#91;0x1000000:0x7c000000:0x0&#93;</span><br/> fid2path error: No such file or directory</p> <p>Blake, </p> <p>Should we close this ticket or is there something else you need to be resolved?</p> <p>Thanks, <br/> James</p> <p>This can be closed. Thanks James.</p> Development Rank 1|hzwbpb: Rank (Obsolete) 12084 Severity