https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-1026 Lustre <p>The last week, one of our customer got the corrupted messages in the ldiskfs, then OSS remounted that OST with readonly.<br/> Here is when we got the error messages. the situation is very similar to <a href="https://jira.whamcloud.com/browse/LU-501" title="ldiskfs on disk corruption" class="issue-link" data-issue-key="LU-501"><del>LU-501</del></a>, but don't know this is exactly same problem. Please check on this log files.</p> <p>Jan 19 18:25:46 lustre-oss-0-0 kernel: <span class="error">&#91;8145936.472484&#93;</span> LDISKFS-fs error (device dm-5): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 23828corrupted: 4190 blocks free in bitmap, 4189 - in gd<br/> Jan 19 18:25:46 lustre-oss-0-0 kernel: <span class="error">&#91;8145936.472677&#93;</span> <br/> Jan 19 18:25:46 lustre-oss-0-0 kernel: <span class="error">&#91;8145936.472727&#93;</span> Aborting journal on device dm-5-8.<br/> Jan 19 18:25:46 lustre-oss-0-0 kernel: <span class="error">&#91;8145936.473961&#93;</span> LDISKFS-fs (dm-5): Remounting filesystem read-only<br/> Jan 19 18:25:46 lustre-oss-0-0 kernel: <span class="error">&#91;8145936.510914&#93;</span> LDISKFS-fs error (device dm-5): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 23828corrupted: 4190 blocks free in bitmap, 4189 - in gd<br/> Jan 19 18:25:46 lustre-oss-0-0 kernel: <span class="error">&#91;8145936.511103&#93;</span> <br/> Jan 19 18:25:46 lustre-oss-0-0 kernel: <span class="error">&#91;8145936.513219&#93;</span> LDISKFS-fs error (device dm-5): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 23828corrupted: 4190 blocks free in bitmap, 4189 - in gd<br/> Jan 19 18:25:46 lustre-oss-0-0 kernel: <span class="error">&#91;8145936.513396&#93;</span> <br/> Jan 19 18:25:46 lustre-oss-0-0 kernel: <span class="error">&#91;8145936.515388&#93;</span> LDISKFS-fs error (device dm-5): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 23828corrupted: 4190 blocks free in bitmap, 4189 - in gd<br/> Jan 19 18:25:46 lustre-oss-0-0 kernel: <span class="error">&#91;8145936.515562&#93;</span> <br/> Jan 19 18:25:46 lustre-oss-0-0 kernel: <span class="error">&#91;8145936.517502&#93;</span> LDISKFS-fs error (device dm-5): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 23828corrupted: 4190 blocks free in bitmap, 4189 - in gd<br/> Jan 19 18:25:46 lustre-oss-0-0 kernel: <span class="error">&#91;8145936.518511&#93;</span> <br/> Jan 19 18:25:46 lustre-oss-0-0 kernel: <span class="error">&#91;8145936.520586&#93;</span> LDISKFS-fs error (device dm-5): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 23828corrupted: 4190 blocks free in bitmap, 4189 - in gd<br/> Jan 19 18:25:46 lustre-oss-0-0 kernel: <span class="error">&#91;8145936.520753&#93;</span> </p> lustre-1.8.4 LU-1026

ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 23828 corrupted

Bug Minor Resolved Fixed Hongchao Zhang Shuichi Ihara p4hc p4j Tue, 24 Jan 2012 09:39:51 +0000 Thu, 14 Jun 2018 21:41:16 +0000 Fri, 4 Dec 2015 18:21:56 +0000 Lustre 1.8.x (1.8.0 - 1.8.5) Lustre 2.8.0 0 13 <p>Were you able to repair this with fsck?</p> <p>yes and fixed, but still don't know why this problem happen.</p> <p>another OSS's log file.</p> <p>In addition, they got same messages on two OSTs. (I will file another /var/log/messages later)<br/> But, when we ran e2fsck, we got the following messages on the one OST, but on the another OST, there were no modification and fixed by e2fsck.</p> <p>e2fsck's log for OST0000</p> <p>the log file of e2fsck to OST0005.</p> <p>Hi Andreas</p> <p>Could you please have a look at this one?</p> <p>Thanks</p> <p>Peter</p> <p>Comparing the error message from messages-another-oss.gz it reports group 41776 was corrupted (blocks 1368915968-1368948735) and the e2fsck output reports block 1368942246, so these are related.</p> <p>I recall seeing bugs like this with Lustre 1.8 that were fixed by adding proper locking for the group descriptors, and fixing the extent cache code, but I'm not sure whether those patches are relevant to your version of ldiskfs or not.</p> <p>What kernel version is being used? Is this ext3-based ldiskfs or ext4-based ldiskfs?</p> <p>The useful error messages from the log, for future reference.</p> <p>messages.gz:<br/> Jan 19 18:25:46 lustre-oss-0-0 kernel: <span class="error">&#91;8145936.472484&#93;</span> LDISKFS-fs error (device dm-5): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 23828 corrupted: 4190 blocks free in bitmap, 4189 - in gd<br/> Jan 19 18:25:46 lustre-oss-0-0 kernel: <span class="error">&#91;8145936.472677&#93;</span> <br/> Jan 19 18:25:46 lustre-oss-0-0 kernel: <span class="error">&#91;8145936.472727&#93;</span> Aborting journal on device dm-5-8.<br/> Jan 19 18:25:46 lustre-oss-0-0 kernel: <span class="error">&#91;8145936.473961&#93;</span> LDISKFS-fs (dm-5): Remounting filesystem read-only<br/> Jan 19 18:25:46 lustre-oss-0-0 kernel: <span class="error">&#91;8145936.936599&#93;</span> LustreError: 18976:0:(obd.h:1394:obd_transno_commit_cb()) lfs-OST0005: transno 47254073325 commit error: 2</p> <p>messages-another-oss.gz:<br/> Jan 20 09:46:54 lustre-oss-0-1 kernel: <span class="error">&#91;8201201.840066&#93;</span> LDISKFS-fs (dm-0): warning: checktime reached, running e2fsck is recommended<br/> Jan 20 09:46:54 lustre-oss-0-1 kernel: <span class="error">&#91;8201201.853653&#93;</span> LDISKFS-fs (dm-0): mounted filesystem with ordered data mode<br/> Jan 20 09:46:54 lustre-oss-0-1 kernel: <span class="error">&#91;8201202.085942&#93;</span> Lustre: lfs-OST0000: Now serving lfs-OST0000 on /dev/mpath/0E3030430100 with recovery enabled<br/> Jan 20 09:53:13 lustre-oss-0-1 kernel: <span class="error">&#91;8201580.664236&#93;</span> LDISKFS-fs error (device dm-0): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 41776 corrupted: 2145 blocks free in bitmap, 2144 - in gd<br/> Jan 20 09:53:13 lustre-oss-0-1 kernel: <span class="error">&#91;8201580.664496&#93;</span> Aborting journal on device dm-0-8.<br/> Jan 20 09:53:13 lustre-oss-0-1 kernel: <span class="error">&#91;8201580.665637&#93;</span> LDISKFS-fs (dm-0): Remounting filesystem read-only<br/> Jan 20 09:53:13 lustre-oss-0-1 kernel: <span class="error">&#91;8201580.732042&#93;</span> LustreError: 25964:0:(obd.h:1394:obd_transno_commit_cb()) lfs-OST0000: transno 55834575965 commit error: 2</p> <p>Hi Andreas, </p> <p>They are using lustre-1.8.4 with ext4 based ldiskfs.<br/> I also remember we saw similar problem on 1.8.x and 1.6.x.<br/> <a href="https://bugzilla.lustre.org/show_bug.cgi?id=18810" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.lustre.org/show_bug.cgi?id=18810</a></p> <p>The patches were landed in 1.8.1.1.</p> <p>The another similar problem is also filed on bugzilla as #22091, but this is also fixed in 1.8.2.</p> <p>If <a href="https://jira.whamcloud.com/browse/LU-501" title="ldiskfs on disk corruption" class="issue-link" data-issue-key="LU-501"><del>LU-501</del></a> is also related to this problem, the root cause of this problem still remains even in 2.1 codes?</p> <p>We hit this today on a lustre-1.8.4, ext3-based ldiskfs OST:<br/> &gt; Jan 30 11:45:30 kernel: LDISKFS-fs error (device sdc1): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 5791corrupted: 6127 blocks free in bitmap, 3087 - in gd</p> <p>Running e2fsck 1.41.10.sun2 on that OST shreddered the file system, though. Perhaps specifying '-yf' on the cmd line was not a good idea, but I wouldn't know otherwise. The OST now doesn't mount (also as '-t ldiskfs') with<br/> &gt; LDISKFS-fs: ldiskfs_check_descriptors: Checksum for group 0 failed (19912!=56814)<br/> Re-enabling the uninit_bg feature and re-running e2fsck did not help.</p> <p>Kernel log messages</p> <p>e2fsck output</p> <p>We just experienced a similar issue with lustre 1.8.8 on RHEL5.9 (2.6.18-308.4.1.el5). Could you comment whether this is related this issue <a href="https://jira.whamcloud.com/browse/LU-1026" title="ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 23828 corrupted" class="issue-link" data-issue-key="LU-1026"><del>LU-1026</del></a> or if a new bug should be opened?</p> <p>The message in the kernel log:<br/> Feb 14 22:09:22 sns-oss4 kernel: <span class="error">&#91;132425.563619&#93;</span> LDISKFS-fs error (device dm-1): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 19519corrupted: 3547 blocks free in bitmap, 14336 - in gd</p> <p>On running e2fsck (e2fsprogs-1.42.3.wc1-0redhat) we had the following:<br/> Block bitmap differences: <del>(639602688</del>-639604480) <br/> <del>(639606784</del><del>639608831) -(639610880</del><del>639612927) -(639614976</del>-639617023) <br/> <del>(639619072</del><del>639621119) -(639623168</del>-639623971)<br/> Fix? no</p> <p>Free blocks count wrong for group #19519 (14336, counted=3547).<br/> Fix? no</p> <p>Free blocks count wrong (2687836218, counted=2687825429).<br/> Fix? no</p> <p>So ldiskfs just tripped up on the bitmap from group #19519, which is exactly what e2fsck said. It looks like e2fsck fixed the free block accounting and we are back to a consistent state. I don't see any relation to the duplicate messages reported by the reporter, so this was unrelated.</p> <p>Shilong Wang (wshilong@ddn.com) uploaded a new patch: <a href="http://review.whamcloud.com/13043" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13043</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-1026" title="ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 23828 corrupted" class="issue-link" data-issue-key="LU-1026"><del>LU-1026</del></a> ldiskfs: fix race when setting bitmap_uptodate flag<br/> Project: fs/lustre-release<br/> Branch: b2_1<br/> Current Patch Set: 1<br/> Commit: 20a657ea773fe8e9175329cbbcbf063fa4dda494</p> <p>We hit several bugs like these under rhel5:</p> <p>Nov 10 10:44:37 t2s007011 kernel: LDISKFS-fs error (device dm-10): ldiskfs_valid_block_bitmap: Invalid block bitmap - block_group = 100095, block = 3279912962<br/> Nov 10 10:44:37 t2s007011 kernel: Aborting journal on device dm-10-8.<br/> Nov 10 10:44:37 t2s007011 kernel: LustreError: 32112:0:(filter_io_26.c:789:filter_commitrw_write()) Failure to commit OST transaction (-5)?</p> <p>I think Following commit from upstream might be related:</p> <p>ext4: fix race when setting bitmap_uptodate flag</p> <p>In ext4_read_</p> {inode,block} <p>_bitmap() we were setting bitmap_uptodate()<br/> before submitting the buffer for read. The is bad, since we check<br/> bitmap_uptodate() without locking the buffer, and so if another<br/> process is racing with us, it's possible that they will think the<br/> bitmap is uptodate even though the read has not completed yet,<br/> resulting in inodes and blocks potentially getting allocated more than<br/> once if we get really unlucky.</p> <p>Notice, if we don't totally load bitmap from disk, we might get a garbage block bitmap,<br/> and we may failed check here, we have never met this problem for rhel6 which<br/> might because flex_bg skip sanity checks.</p> <p>Could someone comment this, same issue reported on: <a href="https://jira.whamcloud.com/browse/LU-486" title="ldiskfs_valid_block_bitmap: Invalid block bitmap" class="issue-link" data-issue-key="LU-486"><del>LU-486</del></a></p> <p>Best Regards,<br/> Wang Shilong</p> <p>Bobijam/Lai</p> <p>Could you please review this proposed fix?</p> <p>Thanks</p> <p>Peter</p> <p>Shilong Wang (wshilong@ddn.com) uploaded a new patch: <a href="http://review.whamcloud.com/13080" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13080</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-1026" title="ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 23828 corrupted" class="issue-link" data-issue-key="LU-1026"><del>LU-1026</del></a> ldiskfs: fix race when setting bitmap_uptodate flag<br/> Project: fs/lustre-release<br/> Branch: b1_8<br/> Current Patch Set: 1<br/> Commit: ca4d034e6ddb6a851b6764019ba6ea16e76dd11f</p> <p>we hit same issue again, this time, debug patch (<a href="http://review.whamcloud.com/#change,1107" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/#change,1107</a>) was enabled and got following addtinal messages below.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Feb 14 18:33:51 t2s007001 kernel: ffffffffffffffff &lt;2&gt;ffffffffffffffff &lt;2&gt;ffffffffffffffff &lt;2&gt;LDISKFS-fs error (device dm-0): ldiskfs_valid_block_bitmap: Invalid block bitmap - group_first_block = 3412328448, block_bitmap = 3412328448, inode_bitmap = 3412328449 inode_table_bitmap = 3412328450, inode_table_block_per_group =512, next_zero_bit = 512, block_group = 104136, block = 3412328450 Feb 14 18:33:51 t2s007001 kernel: Aborting journal on device dm-0-8. Feb 14 18:33:51 t2s007001 kernel: LDISKFS-fs (dm-0): Remounting filesystem read-only </pre> </div></div> <p>we also applied patch <a href="http://review.whamcloud.com/13080" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13080</a>, but it didn't help.</p> <p>Hongchao is looking into this issue</p> <p>Hi Shuichi,</p> <p>as per the additional debug messages, it's the bitmap occupied by the "inode tables" that corrupted, the 512th bit (from zero) in the block bitmap is zero!<br/> and it should be larger than or equal to 514 (1 block for bitmap + 1 block for inode bimtap + 512 blocks for inode table).</p> <p>the content of the block bitmap is also printed in the debug patch, could you please put the content of it here? Thanks!</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">+ printk(KERN_CRIT<span class="code-quote">"block bitmap of block_group %d : \n"</span>, block_group); + <span class="code-keyword">for</span> (i = 0; i &lt; (sb-&gt;s_blocksize &gt;&gt; 3); i++) { + printk(KERN_CRIT<span class="code-quote">"%016lx "</span>, *(((<span class="code-object">long</span> <span class="code-object">int</span>*)bh-&gt;b_data) + i)); + <span class="code-keyword">if</span> (i &amp;&amp; ((i % 4) == 0)) + printk(KERN_CRIT<span class="code-quote">"\n"</span>); + } </pre> </div></div> <p>Yes, I found following messages in the log. full log is attachd.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Feb 14 18:33:51 t2s007001 kernel: block bitmap of block_group 104136 </pre> </div></div> <p>It looks like the on-disk bitmap has the right bit set:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Feb 14 18:33:51 t2s007001 kernel: block bitmap of block_group 104136 : Feb 14 18:33:51 t2s007001 kernel: ffffffffffffffff &lt;2&gt;ffffffffffffffff &lt;2&gt;ffffffffffffffff &lt;2&gt;ffffffffffffffff &lt;2&gt;ffffffffffffffff &lt;2&gt; Feb 14 18:33:51 t2s007001 kernel: ffffffffffffffff &lt;2&gt;ffffffffffffffff &lt;2&gt;ffffffffffffffff &lt;2&gt;ffffffffffffffff &lt;2&gt; Feb 14 18:33:51 t2s007001 kernel: ffffffffffffffff &lt;2&gt;ffffffffffffffff &lt;2&gt;00007fffffffffff &lt;2&gt;ffffffffffffffff &lt;2&gt; Feb 14 18:33:51 t2s007001 kernel: LDISKFS-fs error (device dm-0): ldiskfs_valid_block_bitmap: Invalid block bitmap - group_first_block = 3412328448, block_bitmap = 3412328448, inode_bitmap = 3412328449 inode_table_bitmap = 3412328450, inode_table_block_per_group = 512, next_zero_bit = 512, block_group = 104136, block = 3412328450 </pre> </div></div> <p>Each block of ffff is 64 bits. While the output is a badly formatted, since it appears there are 5x 64 bits on the first line, but the code matches this so it looks correct. </p> <p>I guess one option to mitigate this bug if the root cause cannot be found is to just stop allocation from this group by setting the free block count to zero and move on to another group. It may even be that this is done in the upstream kernel? The error would need to be changed to a warning so that the filesystem is not change to read-only. </p> <p>Hi Andreas,</p> <p>Looking at upstream kernel, following commit:</p> <p>commit 163a203ddb36c36d4a1c942aececda0cc8d06aa7<br/> Author: Darrick J. Wong &lt;darrick.wong@oracle.com&gt;<br/> Date: Wed Aug 28 17:35:51 2013 -0400</p> <p> ext4: mark block group as corrupt on block bitmap error</p> <p>This commit will make us to avoid further allocating/deallocation with corrupt block group.</p> <p>We had following messages outputing when fsck:</p> <p>e2fsck 1.42.9.wc1 (24-Feb-2014)<br/> work0-OST0000: recovering journal<br/> Pass 1: Checking inodes, blocks, and sizes<br/> Pass 2: Checking directory structure<br/> Pass 3: Checking directory connectivity<br/> Pass 4: Checking reference counts<br/> Pass 5: Checking group summary information<br/> Free blocks count wrong (579778260, counted=464050298).<br/> Fix? yes</p> <p>Free inodes count wrong (937631607, counted=937484879).<br/> Fix? yes</p> <p>work0-OST0000: ***** FILE SYSTEM WAS MODIFIED *****<br/> work0-OST0000: 18292145/955777024 files (2.1% non-contiguous), 3359057798/3823108096 blocks</p> <p>Notice Free blocks count are some differences here, and it seems fsck could not fix bitmap errors(no?), it seems<br/> fsck only reset free count here, next time, we might still encounter same errors.</p> <p>there is a ticket to track the problem of wrong count of "Free blocks" and "Free inodes" in <a href="https://jira.whamcloud.com/browse/LU-4557" title="Negative used block number of OST after OSS crashes and reboots" class="issue-link" data-issue-key="LU-4557"><del>LU-4557</del></a>, and that problem should be different from this one.</p> <p>the upstream kernel patch <a href="https://git.kernel.org/cgit/linux/kernel/git/tytso/ext4.git/commit/?id=163a203ddb36c36d4a1c942aececda0cc8d06aa7" class="external-link" target="_blank" rel="nofollow noopener">https://git.kernel.org/cgit/linux/kernel/git/tytso/ext4.git/commit/?id=163a203ddb36c36d4a1c942aececda0cc8d06aa7</a><br/> only marked the group to be corrupted and the current mount will still be marked as READONLY if "ERRORS_CONT" isn't set.</p> <p>I wonder whether this issue is caused by some temporal failure somewhere or some race(for the buffer isn't locked) for the following content printed<br/> by the debug patch is correct, how about locking the buffer before calling "ext4_valid_block_bitmap" and also re-test again if bitmap error is detected?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">--- linux-2.6.18-348.1.1.el5.x86_64.orig/fs/ext4/balloc.c 2014-11-24 05:32:30.894982225 +0800 +++ linux-2.6.18-348.1.1.el5.x86_64/fs/ext4/balloc.c 2014-11-24 05:50:11.287001297 +0800 @@ -275,6 +275,13 @@ <span class="code-keyword">static</span> <span class="code-object">int</span> ext4_valid_block_bitmap(struc <span class="code-comment">/* good bitmap <span class="code-keyword">for</span> inode tables */</span> <span class="code-keyword">return</span> 1; + smp_mb(); + next_zero_bit = ext4_find_next_zero_bit(bh-&gt;b_data, + offset + EXT4_SB(sb)-&gt;s_itb_per_group, + offset); + <span class="code-keyword">if</span> (next_zero_bit &gt;= offset + EXT4_SB(sb)-&gt;s_itb_per_group) + <span class="code-keyword">return</span> 1; + err_out: ext4_error(sb, <span class="code-quote">"Invalid block bitmap - block_group = %d, block = %llu"</span>, block_group, bitmap_blk); @@ -350,7 +357,11 @@ ext4_read_block_bitmap(struct super_bloc block_group, bitmap_blk); <span class="code-keyword">return</span> NULL; } + + lock_buffer(bh); ext4_valid_block_bitmap(sb, desc, block_group, bh); + unlock_buffer(bh); + /* * file system mounted not to panic on error, * <span class="code-keyword">continue</span> with corrupt bitmap </pre> </div></div> <p>Note that the block and inode summary in the superblock is only updated during a clean unmount. It is not updated during normal usage since the per-group totals are used instead. </p> <p>Hi Hong Chao,</p> <p>Could you give a formal patch(with your suggestion)?</p> <p>Shilong Wang (wshilong@ddn.com) uploaded a new patch: <a href="http://review.whamcloud.com/13991" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13991</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-1026" title="ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 23828 corrupted" class="issue-link" data-issue-key="LU-1026"><del>LU-1026</del></a> ldiskfs: try one more time with locked buffer<br/> Project: fs/lustre-release<br/> Branch: b2_1<br/> Current Patch Set: 1<br/> Commit: 25a9e3904313190714b4e22ad1b3f50e1cef62d5</p> <p>Shilong Wang (wshilong@ddn.com) uploaded a new patch: <a href="http://review.whamcloud.com/13992" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/13992</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-1026" title="ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 23828 corrupted" class="issue-link" data-issue-key="LU-1026"><del>LU-1026</del></a> ldiskfs: make bg RO if bitmaps validations fail<br/> Project: fs/lustre-release<br/> Branch: b2_1<br/> Current Patch Set: 1<br/> Commit: 635f7aa3a562d6af4887a463900768ec872a60a9</p> <p>Hi Shilong, </p> <p>sorry for delayed response, and thanks you very much for creating the corresponding patch!</p> <p>Hi Shilong,</p> <p>Do you manage to test with the new patch? and what is the result? Thanks!</p> <p>Hi HongChao,</p> <p>This bug was hard to reproduce, it happend in customers' machine several months, so we are going to apply the patch,<br/> and will let you know results.</p> <p>Wang Shilong (wshilong@ddn.com) uploaded a new patch: <a href="http://review.whamcloud.com/16679" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16679</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-1026" title="ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 23828 corrupted" class="issue-link" data-issue-key="LU-1026"><del>LU-1026</del></a> ldiskfs: more bitmaps corruption handling<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 5378282c5f0f9bbd12e00cde80e5fc757a091548</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="http://review.whamcloud.com/16679/" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16679/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-1026" title="ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 23828 corrupted" class="issue-link" data-issue-key="LU-1026"><del>LU-1026</del></a> ldiskfs: make bitmaps corruption not fatal<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e727c383db8b2485d9e6137895136699d57ea047</p> <p>Landed for 2.8</p> <p>Shilong,<br/> it would be great if you submitted this patch <a href="http://review.whamcloud.com/16679" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16679</a> (and any parts of <a href="http://review.whamcloud.com/16312" class="external-link" target="_blank" rel="nofollow noopener">http://review.whamcloud.com/16312</a> needed) to upstream ext4, so that we don't have to maintain this forever in the future. The commit comment should be updated to match the newer kernel and reference ext4 instead of ldiskfs, like:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>ext4: make bitmap corruption not fatal There can be occasional reasons for bitmap problems, which are detected by ext4_mb_check_ondisk_bitmap() and cause the filesystem to be remounted read-only due to ext4_error(): EXT4-fs error (device /dev/dm-6-8): ext4_mb_generate_buddy:755: group 294, block 0: block bitmap and bg descriptor inconsistent: 20180 vs 20181 free clusters Aborting journal on device dm-6-8. EXT4-fs (dm-6): Remounting filesystem read-only This might be caused by some ext4 internal bugs, which are addressed separately. This patch makes ext4 more robust by the following changes: - ext4_read_block_bitmap() printed error, so do not call ext4_error() again - mark all bits in bitmap used so that it will not be used for allocation - mark block group corrupt, use ext4_warning() instead of ext4_error() Tested by following script: TEST_DEV="/dev/sdb" TEST_MNT="/mnt/ext4" mkdir -p $TEST_MNT mkfs.ext4 -F $TEST_DEV mount -t ext4 $TEST_DEV $TEST_MNT dd if=/dev/zero of=$TEST_MNT/largefile oflag=direct bs=10485760 count=200 umount $TEST_MNT dd if=/dev/zero of=$TEST_DEV oflag=direct bs=4096 seek=641 count=10 mount -t ext4 $TEST_DEV $TEST_MNT rm -f $TEST_MNT/largefile dd if=/dev/zero of=$TEST_MNT/largefile oflag=direct bs=10485760 count=200 &amp;&amp; echo "FILESYSTEM still usable after bitmaps corrupts happen" umount $TEST_MNT e2fsck $TEST_DEV -y Signed-off-by: Wang Shilong &lt;wshilong@ddn.com&gt; Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-1026 Reviewed-on: http://review.whamcloud.com/16679 Reviewed-by: Bob Glossman &lt;bob.glossman@intel.com&gt; Reviewed-by: Yang Sheng &lt;yang.sheng@intel.com&gt; Reviewed-by: Oleg Drokin &lt;oleg.drokin@intel.com&gt; </pre> </div></div> <p>We may have hit this bug after large amount of data was deleted, from a nearly full filesystem, and then data was being written again. We are going to see if we can reproduce it on our test filesystem.</p> <p> Can we get a backport to 2.7.2fe.</p> <p>Thanks,<br/> Mahmoud</p> Duplicate LU-9410 Related LU-7114 LU-8252 Development Rank 1|hzw08n: Rank (Obsolete) 10118 Severity