https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-4523 Lustre <p>The customer, Yale, encountered file system corruption on one of their OST devices, dm-20 which is "scratch-OST0028". Customer fan e2fsck on that device, which fixed the corruption, but now they would like to have a RCA to prevent it from happening in future.</p> <p>The corruption was first reported on Jan-11, but there aren't any irregular events on the storage side that would have caused such corruption, which could indicate the corruption happened sometime before and was only reported on the 11th.</p> <p>Jan 11 11:54:33 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Conn stale 10.191.133.6@o2ib <span class="error">&#91;old ver: 12, new ver: 12&#93;</span><br/> Jan 11 11:54:33 oss7 kernel: Lustre: 2916:0:(o2iblnd_cb.c:2249:kiblnd_passive_connect()) Skipped 101 previous similar messages<br/> Jan 11 11:59:52 oss7 kernel: LDISKFS-fs error (device dm-20): ldiskfs_mb_free_metadata: Double free of blocks 30208 (30208 148)<br/> Jan 11 11:59:52 oss7 kernel: Aborting journal on device dm-20-8.<br/> Jan 11 11:59:52 oss7 kernel: LDISKFS-fs (dm-20): Remounting filesystem read-only<br/> Jan 11 11:59:52 oss7 kernel: LDISKFS-fs error (device dm-20) in ldiskfs_reserve_inode_write: Journal has aborted<br/> Jan 11 11:59:52 oss7 kernel: LDISKFS-fs error (device dm-20) in ldiskfs_ext_remove_space: Journal has aborted<br/> Jan 11 11:59:52 oss7 kernel: LDISKFS-fs error (device dm-20) in ldiskfs_reserve_inode_write: Journal has aborted<br/> Jan 11 11:59:52 oss7 kernel: LDISKFS-fs error (device dm-20) in ldiskfs_orphan_del: Journal has aborted<br/> Jan 11 11:59:52 oss7 kernel: LDISKFS-fs error (device dm-20) in ldiskfs_reserve_inode_write: Journal has aborted<br/> Jan 11 11:59:52 oss7 kernel: LDISKFS-fs error (device dm-20) in ldiskfs_ext_truncate: Journal has aborted<br/> Jan 11 11:59:52 oss7 kernel: LustreError: 22850:0:(fsfilt-ldiskfs.c:369:fsfilt_ldiskfs_start()) error starting handle for op 8 (106 credits): rc -30</p> <p>FSCK output:<br/> e2fsck 1.42.3.wc3 (15-Aug-2012)<br/> device /dev/mapper/ost_scratch_40 mounted by lustre per /proc/fs/lustre/obdfilter/scratch-OST0028/mntdev<br/> Warning! /dev/mapper/ost_scratch_40 is mounted.<br/> MMP interval is 10 seconds and total wait time is 42 seconds. Please wait...<br/> Warning: skipping journal recovery because doing a read-only filesystem check.<br/> scratch-OST0028 contains a file system with errors, check forced.<br/> Pass 1: Checking inodes, blocks, and sizes</p> <p>Running additional passes to resolve blocks claimed by more than one inode...<br/> Pass 1B: Rescanning for multiply-claimed blocks</p> <p>What other debugging or data can be pulled to explain the problem?</p> <p>Thanks,<br/> Oz</p> LU-4523

Need explanation for FS corruption - ldiskfs_mb_free_metadata: Double free of blocks

Bug Minor Resolved Duplicate Hongchao Zhang Oz Rentas Tue, 21 Jan 2014 19:32:19 +0000 Tue, 11 Feb 2014 15:54:28 +0000 Tue, 11 Feb 2014 15:54:28 +0000 0 4 <p>Oz</p> <p>We'll certainly need details about which Lustre version is in use and some logs - dmesg from this node (or syslog) as a start, say 24 hours into the past.</p> <p>Thanks</p> <p>Peter</p> <p>Ah, yes, of course. Sorry about that. I've attached the missing files.</p> <p>Lustre: 1.8.9<br/> Kernel: 2.6.18-348.1.1.el5</p> <p>Hongchao has been looking at this information</p> <p>Hongchao</p> <p>As per our recent discussion on this topic I understand that you believe this issue to be a duplication of <a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a> which periodically affected our internal testing on older Lustre releases but has not been seen on 2.4.x and newer releases and that you believe this is due to an issue in the underlying ext4 code that has been addressed with newer kernel versions.</p> <p>Do I have this right? Is there anything to add/correct?</p> <p>Thanks</p> <p>Peter</p> <p>Hi Peter,</p> <p>Yes, it could be problem related to the ext4 (patched a little by Lustre and renamed to ldiskfs), there are some similar ticket (<a href="https://jira.whamcloud.com/browse/LU-482" title="Test failure on test suite replay-dual, subtest test_0a" class="issue-link" data-issue-key="LU-482"><del>LU-482</del></a>, <a href="https://jira.whamcloud.com/browse/LU-699" title="replay-dual test_1 fails to remount mdt" class="issue-link" data-issue-key="LU-699"><del>LU-699</del></a>, etc)</p> <p>btw, there is a similar issues reported on Redhat,</p> <p><a href="https://access.redhat.com/site/solutions/157393" class="external-link" target="_blank" rel="nofollow noopener">https://access.redhat.com/site/solutions/157393</a></p> <p>Thanks</p> <p>Thank you for this very useful information. It has been passed on to the customer.<br/> We have since upgraded the OS / Lustre build on the servers. This ticket can be closed.</p> <p>ok thanks Oz</p> Development Rank 1|hzwdcf: Rank (Obsolete) 12367 Severity