[LU-1998] Can't fsck/mount mdadm device with external journal after journal was deleted and re-added. LDISKFS-fs: journal UUID does not match

• Status: Closed
• Project: Lustre
• Affects Version/s: Lustre 1.8.x (1.8.0 - 1.8.5)
• Type: Epic
• Priority: Major
• Reporter: Joe Mervini
• Assignee: WC Triage
• Resolution: Fixed

Environment

Sun Blade (redsky) hardware running TOSS-4.4.4-1 (RHEL 5.5) and Sun J4400 JBODs

Description

We encountered a problem with one of our OSTs that would not start because of "[ 5437.653340] md2: bitmap superblock UUID mismatch". Also while we were able to run fsck.ldiskfs (livermore's renaming) on the other md device on the OSS, on the problem target fsck was refusing to run because of the following error: "external journal does not support filesystem". So following directions posted by Andreas in an earlier lustre discuss thread, we removed the journal, ran fsck and re-added the external journal.

After running those steps and trying to mount -t lustre, it fails asking whether the ldiskfs module is available with "LDISKFS-fs: journal UUID does not match" in dmesg. In addition, fsck still reports the journal does not support filesystem.

I need some help fixing this problem ASAP because this is a production system.

dmesg output:
[ 2414.128187] LDISKFS-fs: journal UUID does not match
[ 2415.232953] LustreError: 7358:0:(obd_mount.c:1292:server_kernel_mount()) premount /dev/md2:0x0 ldiskfs failed: -22, ldiskfs2 failed: -19. Is the ldiskfs module available?
[ 2415.232963] LustreError: 7358:0:(obd_mount.c:1618:server_fill_super()) Unable to mount device /dev/md2: -22
[ 2415.232973] LustreError: 7358:0:(obd_mount.c:2050:lustre_fill_super()) Unable to mount (-22)

Comments

[Johann Lombardi (Inactive)]

The problem might be related to the major/minor of the journal block device which have changed (those numbers are hardcoded in the ldiskfs filesystem). Could you please provide us with:
1. the exact commands you ran to add/remove the journal
2. the output of dumpe2fs -h against the ldiskfs & jbd filesystems
3. the output of ls -l /dev

Thanks

[Joe Mervini]

Never mind. I decided to remake the journal file system (mke2fs -b 4096 -O journal_dev /dev/md12) and after doing that everything came back fine.

[Johann Lombardi (Inactive)]

ok