[LU-588] IO hangs from MMP

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.1.5, Lustre 1.8.9
• Affects Version/s: Lustre 2.1.0, Lustre 1.8.6
• Environment: RHEL 5.6 (2.6.18-238.19.1.el5) with one SCSI device handler patch from RHEL 5.7 kernels
• Severity: 3
• Assignee: Jian Yu
• Reporter: Jeremy Filizetti

Description

I've had my mkfs.lustre commands hang from time to time while formatting all of our OSTs on an OSS simultaneously (29-30 OSTs). The problem shows up with all of the mke2fs completed but the mkfs.lustre being stuck in a TASK_UNINTERRUPTIBLE state. The system starts reporting hung tasks for mkfs.lustre, kmmpd kernel threads, and a few other system resources that are stuck waiting on mutexs from the MMP issue.

I see the following message in the dmesg/syslog.
Aug 4 10:56:12 s01ns030 kernel: LDISKFS-fs warning (device dm-78): kmmpd: kmmpd being stopped since MMP feature has been disabled.
Aug 4 10:56:16 s01ns030 kernel: LDISKFS-fs warning (device dm-70): kmmpd: kmmpd being stopped since MMP feature has been disabled.
Aug 4 10:56:17 s01ns030 kernel: LDISKFS-fs warning (device dm-66): kmmpd: kmmpd being stopped since MMP feature has been disabled.

After adding some printks to kmmpd and forcing a panic, it looks like the issue is that the buffer_head being used by the kmmpd kthread is zeroed. The problem seems to be in ldiskfs_put_super that the buffer_head for the super block is being released prior to the kmmpd kthread being stopped.

Moving the release of the super block buffer head to after the MMP thread has stopped appears to have fixed the issue for me.

— ext4-mmp-rhel5.patch.orig 2011-08-11 12:01:59.000000000 +0000
+++ ext4-mmp-rhel5.patch 2011-08-11 12:06:42.000000000 +0000
@@ -522,12 +522,21 @@

#include "ext4.h"
#include "ext4_jbd2.h"
-@@ -698,6 +700,8 @@ static void ext4_put_super(struct super_
+@@ -682,7 +682,6 @@
+ percpu_counter_destroy(&sbi->s_freeinodes_counter);
+ percpu_counter_destroy(&sbi->s_dirs_counter);
+ percpu_counter_destroy(&sbi->s_dirtyblocks_counter);
+- brelse(sbi->s_sbh);
+ #ifdef CONFIG_QUOTA
+ for (i = 0; i < MAXQUOTAS; i++)
+ kfree(sbi->s_qf_names[i]);
+@@ -698,6 +700,9 @@ static void ext4_put_super(struct super_
invalidate_bdev(sbi->journal_bdev, 0);
ext4_blkdev_remove(sbi);
}
+ if (sbi->s_mmp_tsk)
+ kthread_stop(sbi->s_mmp_tsk);
++ brelse(sbi->s_sbh);
sb->s_fs_info = NULL;
/*

• Now that we are completely done shutting down the

Issue Links

• Trackbacks: Lustre 1.8.x known issues tracker While testing against Lustre b18 branch, we would hit known bugs which were already reported in Lustre Bugzilla https://bugzilla.lustre.org/. In order to move away from relying on Bugzilla, we would create a JIRA

Comments

[Jian Yu]

Patches were landed to Lustre b1_8 and b2_1 branches.

[Jian Yu]

Oleg, could you please cherry-pick the patch of http://review.whamcloud.com/3172 to Lustre b2_1 branch?

Since we can not cherry pick the patch from Lustre b1_8 branch to b2_1 branch, I ported the patch for b2_1 in http://review.whamcloud.com/5745.

[Jian Yu]

Does an equivalent patch need to land on b2_1?

Yes, Peter.

Oleg, could you please cherry-pick the patch of http://review.whamcloud.com/3172 to Lustre b2_1 branch? Thanks.

[Peter Jones]

Does an equivalent patch need to land on b2_1?

[James A Simmons]

For SLES 2.6.32+ kernels yes that upstream commit is there.

[Jeremy Filizetti]

Its been a while since I've had a chance to revisit this but I figured I'd provide an update.

From the looks of things all versions ext4/ldiskfs will need this to be fixed to prevent the possible buffer_head reuse although RHEL5 is the only one I think affected by the hang caused from the kthread_stop issue.

The upstream kernel has commit 63706172f332fd3f6e7458ebfb35fa6de9c21dc5 which I believe prevents the issue from calling kthread_stop on a kthread that has already returned. RHEL 6.2+ has that commit but I'm not sure about SLES.