[LU-9410] on-disk bitmap corrupted

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.10.1, Lustre 2.11.0
• Affects Version/s: Lustre 2.7.0
• Severity: 3
• Assignee: nasf (Inactive)
• Reporter: Mahmoud Hanafi

Description

We had 2 OSS and 3 different OST crash with bitmap corrupted messages.

Apr  3 18:38:16 nbp1-oss6 kernel: LDISKFS-fs error (device dm-42): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 245659corrupted: 32768 blocks free in bitmap, 0 - in gd
Apr  3 18:38:16 nbp1-oss6 kernel: 
Apr  3 18:38:16 nbp1-oss6 kernel: Aborting journal on device dm-3.
Apr  3 18:38:16 nbp1-oss6 kernel: LDISKFS-fs (dm-42): Remounting filesystem read-only
Apr  3 18:38:16 nbp1-oss6 kernel: LDISKFS-fs error (device dm-42): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 245660corrupted: 32768 blocks free in bitmap, 0 - in gd

These errors were on 2 different backend RAID devices. Note worthy  items:
1 .The filesystem was +90% full and 1/2 of the data was deleted.
2. OSTs are formatted with " -E packed_meta_blocks=1 "

Issue Links

• duplicates: LU-1026 ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 23828 corrupted
• duplicates: LU-7114 ldiskfs: corrupted bitmaps handling patches
• is related to: LU-10837 no bitmap check if block bitmap is uninitialized

Comments

[nasf (Inactive)]

Yes, master also needs the patch 28566.

[Jay Lan (Inactive)]

Do I need this patch for 2.10.0?

[nasf (Inactive)]

I think that there may be something can be improved for mke2fs, not e2fsck.

[Mahmoud Hanafi]

Does this patch require any changes to e2fsck?

[nasf (Inactive)]

mhanafi Thanks for the update.

[Mahmoud Hanafi]

updated: we have applied https://review.whamcloud.com/28566 Friday and the filesystem has been stable.

[nasf (Inactive)]

Sorry I typed the patch number. I wanted to say it is stable with 28550.

Then it is reasonable. As I explained above, 28550 may do more than the necessary fixes. But since it runs stable, you can keep it until next 'corruption'.

[Mahmoud Hanafi]

Sorry I typed the patch number. I wanted to say it is stable with 28550.

 

[nasf (Inactive)]

The patch 28550 will take effect before 28566, so if 28550 is applied, then 28566 is meaningless. But 28550 may do more things than the necessary fixes. I am afraid of some penitential side-effect.

The filesystem is stable with the workaround patch (/28489/). Can we run with this patch for sometime without any underlining filesystem issues? Or should we replace it with 28566 ASAP.

It is interesting to know that. Because 28489 is just a debug patch, I cannot imagine how it can resolve your issue. It may because your system has jumped over the groups with "BLOCK_UNINIT" flag and zero free blocks in GDP. If it is true, then applying 28566 will not show you more benefit. Since your system is stable running, you can replace the patches with 28566 when it 'corrupted' next time.

[Mahmoud Hanafi]

The filesystem is stable with the workaround patch (/28489/). Can we run with this patch for sometime without any underlining filesystem issues? Or should we replace it with 28566 ASAP.

[Jay Lan (Inactive)]

I did a build with #28566 and #28550 yesterday. For testing purpose, do these two conflict?
I will undo #28550, but if these two do not collide, we can do testing with the builds I did yesterday.

Never mind. I just did another build with #28550 pulled out.