[#LU-1026] ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 23828 corrupted

Feb 14 18:33:51 t2s007001 kernel: ffffffffffffffff <2>ffffffffffffffff <2>ffffffffffffffff <2>LDISKFS-fs error (device dm-0): ldiskfs_valid_block_bitmap: Invalid block bitmap - group_first_block = 3412328448, block_bitmap = 3412328448, inode_bitmap = 3412328449 inode_table_bitmap = 3412328450, inode_table_block_per_group =512, next_zero_bit = 512, block_group = 104136, block = 3412328450 Feb 14 18:33:51 t2s007001 kernel: Aborting journal on device dm-0-8. Feb 14 18:33:51 t2s007001 kernel: LDISKFS-fs (dm-0): Remounting filesystem read-only

+ printk(KERN_CRIT"block bitmap of block_group %d : \n", block_group); + for (i = 0; i < (sb->s_blocksize >> 3); i++) { + printk(KERN_CRIT"%016lx ", *(((long int*)bh->b_data) + i)); + if (i && ((i % 4) == 0)) + printk(KERN_CRIT"\n"); + }

Feb 14 18:33:51 t2s007001 kernel: block bitmap of block_group 104136 : Feb 14 18:33:51 t2s007001 kernel: ffffffffffffffff <2>ffffffffffffffff <2>ffffffffffffffff <2>ffffffffffffffff <2>ffffffffffffffff <2> Feb 14 18:33:51 t2s007001 kernel: ffffffffffffffff <2>ffffffffffffffff <2>ffffffffffffffff <2>ffffffffffffffff <2> Feb 14 18:33:51 t2s007001 kernel: ffffffffffffffff <2>ffffffffffffffff <2>00007fffffffffff <2>ffffffffffffffff <2> Feb 14 18:33:51 t2s007001 kernel: LDISKFS-fs error (device dm-0): ldiskfs_valid_block_bitmap: Invalid block bitmap - group_first_block = 3412328448, block_bitmap = 3412328448, inode_bitmap = 3412328449 inode_table_bitmap = 3412328450, inode_table_block_per_group = 512, next_zero_bit = 512, block_group = 104136, block = 3412328450

--- linux-2.6.18-348.1.1.el5.x86_64.orig/fs/ext4/balloc.c 2014-11-24 05:32:30.894982225 +0800 +++ linux-2.6.18-348.1.1.el5.x86_64/fs/ext4/balloc.c 2014-11-24 05:50:11.287001297 +0800 @@ -275,6 +275,13 @@ static int ext4_valid_block_bitmap(struc /* good bitmap for inode tables */ return 1; + smp_mb(); + next_zero_bit = ext4_find_next_zero_bit(bh->b_data, + offset + EXT4_SB(sb)->s_itb_per_group, + offset); + if (next_zero_bit >= offset + EXT4_SB(sb)->s_itb_per_group) + return 1; + err_out: ext4_error(sb, "Invalid block bitmap - block_group = %d, block = %llu", block_group, bitmap_blk); @@ -350,7 +357,11 @@ ext4_read_block_bitmap(struct super_bloc block_group, bitmap_blk); return NULL; } + + lock_buffer(bh); ext4_valid_block_bitmap(sb, desc, block_group, bh); + unlock_buffer(bh); + /* * file system mounted not to panic on error, * continue with corrupt bitmap

Shilong,
it would be great if you submitted this patch http://review.whamcloud.com/16679 (and any parts of http://review.whamcloud.com/16312 needed) to upstream ext4, so that we don't have to maintain this forever in the future. The commit comment should be updated to match the newer kernel and reference ext4 instead of ldiskfs, like:

ext4: make bitmap corruption not fatal

There can be occasional reasons for bitmap problems, which are
detected by ext4_mb_check_ondisk_bitmap() and cause the
filesystem to be remounted read-only due to ext4_error():

 EXT4-fs error (device /dev/dm-6-8): ext4_mb_generate_buddy:755:
    group 294, block 0: block bitmap and bg descriptor inconsistent:
    20180 vs 20181 free clusters
 Aborting journal on device dm-6-8.
 EXT4-fs (dm-6): Remounting filesystem read-only

This might be caused by some ext4 internal bugs, which are addressed
separately.  This patch makes ext4 more robust by the following changes:

- ext4_read_block_bitmap() printed error, so do not call ext4_error() again
- mark all bits in bitmap used so that it will not be used for allocation
- mark block group corrupt, use ext4_warning() instead of ext4_error()

Tested by following script:

TEST_DEV="/dev/sdb"
TEST_MNT="/mnt/ext4"

mkdir -p $TEST_MNT
mkfs.ext4 -F $TEST_DEV

mount -t ext4 $TEST_DEV $TEST_MNT
dd if=/dev/zero of=$TEST_MNT/largefile oflag=direct bs=10485760 count=200
umount $TEST_MNT
dd if=/dev/zero of=$TEST_DEV oflag=direct bs=4096 seek=641 count=10
mount -t ext4 $TEST_DEV $TEST_MNT
rm -f $TEST_MNT/largefile
dd if=/dev/zero of=$TEST_MNT/largefile oflag=direct bs=10485760 count=200 &&
      echo "FILESYSTEM still usable after bitmaps corrupts happen"
umount $TEST_MNT
e2fsck $TEST_DEV -y

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Intel-bug-id: https://jira.hpdd.intel.com/browse/LU-1026 
Reviewed-on: http://review.whamcloud.com/16679
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

[LU-1026] ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 23828 corrupted Created: 24/Jan/12 Updated: 14/Jun/18 Resolved: 04/Dec/15
Status:	Resolved
Project:	Lustre
Component/s:	None
Affects Version/s:	Lustre 1.8.x (1.8.0 - 1.8.5)
Fix Version/s:	Lustre 2.8.0

[LU-1026] ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 23828 corrupted Created: 24/Jan/12 Updated: 14/Jun/18 Resolved: 04/Dec/15