[LU-9410] on-disk bitmap corrupted

• Type: Bug
• Resolution: Fixed
• Priority: Critical
• Fix Version/s: Lustre 2.10.1, Lustre 2.11.0
• Affects Version/s: Lustre 2.7.0
• Severity: 3
• Assignee: nasf (Inactive)
• Reporter: Mahmoud Hanafi

Description

We had 2 OSS and 3 different OST crash with bitmap corrupted messages.

Apr  3 18:38:16 nbp1-oss6 kernel: LDISKFS-fs error (device dm-42): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 245659corrupted: 32768 blocks free in bitmap, 0 - in gd
Apr  3 18:38:16 nbp1-oss6 kernel: 
Apr  3 18:38:16 nbp1-oss6 kernel: Aborting journal on device dm-3.
Apr  3 18:38:16 nbp1-oss6 kernel: LDISKFS-fs (dm-42): Remounting filesystem read-only
Apr  3 18:38:16 nbp1-oss6 kernel: LDISKFS-fs error (device dm-42): ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 245660corrupted: 32768 blocks free in bitmap, 0 - in gd

These errors were on 2 different backend RAID devices. Note worthy  items:
1 .The filesystem was +90% full and 1/2 of the data was deleted.
2. OSTs are formatted with " -E packed_meta_blocks=1 "

Issue Links

• duplicates: LU-1026 ldiskfs_mb_check_ondisk_bitmap: on-disk bitmap for group 23828 corrupted
• duplicates: LU-7114 ldiskfs: corrupted bitmaps handling patches
• is related to: LU-10837 no bitmap check if block bitmap is uninitialized

Comments

[Gerrit Updater]

John L. Hammond ([e-mail]) merged in patch https://review.whamcloud.com/28765/
Subject: LU-9410 ldiskfs: no check mb bitmap if flex_bg enabled
Project: fs/lustre-release
Branch: b2_10
Current Patch Set:
Commit: 27f5b8b16416b04a561d0b0121860e2a5188be4a

[Gerrit Updater]

Minh Diep ([e-mail]) uploaded a new patch: https://review.whamcloud.com/28765
Subject: LU-9410 ldiskfs: no check mb bitmap if flex_bg enabled
Project: fs/lustre-release
Branch: b2_10
Current Patch Set: 1
Commit: 53d836b1e5d255558639fe8e4eae78a87a176d04

[Gerrit Updater]

Oleg Drokin ([e-mail]) merged in patch https://review.whamcloud.com/28566/
Subject: LU-9410 ldiskfs: no check mb bitmap if flex_bg enabled
Project: fs/lustre-release
Branch: master
Current Patch Set:
Commit: 5506c15a65b3eebb9f15000105e6eb7c02742a10

[nasf (Inactive)]

Yes, master also needs the patch 28566.

[Jay Lan (Inactive)]

Do I need this patch for 2.10.0?

[nasf (Inactive)]

I think that there may be something can be improved for mke2fs, not e2fsck.

[Mahmoud Hanafi]

Does this patch require any changes to e2fsck?

[nasf (Inactive)]

mhanafi Thanks for the update.

[Mahmoud Hanafi]

updated: we have applied https://review.whamcloud.com/28566 Friday and the filesystem has been stable.

[nasf (Inactive)]

Sorry I typed the patch number. I wanted to say it is stable with 28550.

Then it is reasonable. As I explained above, 28550 may do more than the necessary fixes. But since it runs stable, you can keep it until next 'corruption'.