[#LU-206] Memory corruption due to MMP (1.8.5)

[LU-206] Memory corruption due to MMP (1.8.5) Created: 10/Apr/11 Updated: 20/Apr/11 Resolved: 20/Apr/11
Status:	Resolved
Project:	Lustre
Component/s:	None
Affects Version/s:	Lustre 1.8.6
Fix Version/s:	Lustre 2.1.0, Lustre 1.8.6

Type:

Bug

Priority:

Blocker

Reporter:

Johann Lombardi (Inactive)

Assignee:

Zhenyu Xu

Resolution:

Fixed

Votes:

Labels:

None

Severity:

Rank (Obsolete):

5072

Description

I have been chasing a weird bug with MMP on 2.6.39-rc1 (i want to push MMP upstream) and it turns out to be an horrible memory corruption which can impact any kernels.

+       mmpd_data = kmalloc(sizeof(struct mmpd_data *), GFP_KERNEL);
                                   ^^^^^^^^^^^^^^^^^^^
                                OUCH!!! should be sizeof(struct mmpd_data)
+       if (!mmpd_data) {
+               ext3_warning(sb, KERN_ERR, "not enough memory for mmpd_data");
+               goto failed;
+       }   
+       mmpd_data->sb = sb; 
+       mmpd_data->bh = bh;

The memory corruption was introduced by bugzilla ticket 22117 in 1.8.5.
This issue can cause all sorts of oops everywhere in the kernel.

Comments

Comment by Johann Lombardi (Inactive) [ 10/Apr/11 ]

BTW, 2.1 is impacted too.

Comment by Alex Zhuravlev [ 10/Apr/11 ]

On Mon, Apr 11, 2011 at 8:15 AM, Johann Lombardi (JIRA)

struct mmpd_data {
struct buffer_head bh; / bh from initial read_mmp_block() */
struct super_block sb; / super block of the fs */
};

so it's 16 bytes on 64bit arch ? iirc, the minimal allocation is 32 bytes in
slab/slub due to cpu cacheline.

Comment by Zhenyu Xu [ 11/Apr/11 ]

patch at http://review.whamcloud.com/420

Comment by Brian Murrell (Inactive) [ 11/Apr/11 ]

Is this serious enough that we need to consider a 1.8.5.1 (and/or 2.0.0.x) release?

Comment by Johann Lombardi (Inactive) [ 11/Apr/11 ]

I use x86_64 with SLUB. In slabinfo, i have kmalloc-8/16/32/../8192 slabs, so it seems that we can have allocations smaller than 16 bytes.

I have just checked on RHEL5 and the smallest kmalloc slab is indeed size-32.

Comment by Build Master (Inactive) [ 18/Apr/11 ]

Integrated in lustre-master-centos5 #200
~~LU-206~~ Memory corruption due to MMP (1.8.5)

Oleg Drokin : 19de0afd629a3ffe714486cc75264ea14cea7346
Files :

ldiskfs/kernel_patches/patches/ext4-mmp-rhel6.patch
ldiskfs/kernel_patches/patches/ext3-mmp-2.6.18-vanilla.patch
ldiskfs/kernel_patches/patches/ext4-mmp-rhel5.patch

Comment by Build Master (Inactive) [ 18/Apr/11 ]

Integrated in lustre-master » client,ubuntu-x86_64 #28
~~LU-206~~ Memory corruption due to MMP (1.8.5)

Oleg Drokin : 19de0afd629a3ffe714486cc75264ea14cea7346
Files :

ldiskfs/kernel_patches/patches/ext4-mmp-rhel6.patch
ldiskfs/kernel_patches/patches/ext4-mmp-rhel5.patch
ldiskfs/kernel_patches/patches/ext3-mmp-2.6.18-vanilla.patch

Comment by Build Master (Inactive) [ 18/Apr/11 ]

Integrated in lustre-master » client,el6-i686 #28
~~LU-206~~ Memory corruption due to MMP (1.8.5)

Oleg Drokin : 19de0afd629a3ffe714486cc75264ea14cea7346
Files :

ldiskfs/kernel_patches/patches/ext4-mmp-rhel5.patch
ldiskfs/kernel_patches/patches/ext4-mmp-rhel6.patch
ldiskfs/kernel_patches/patches/ext3-mmp-2.6.18-vanilla.patch

Comment by Build Master (Inactive) [ 18/Apr/11 ]

Integrated in lustre-master » client,el5-i686 #29
~~LU-206~~ Memory corruption due to MMP (1.8.5)

Oleg Drokin : 19de0afd629a3ffe714486cc75264ea14cea7346
Files :

ldiskfs/kernel_patches/patches/ext4-mmp-rhel6.patch
ldiskfs/kernel_patches/patches/ext4-mmp-rhel5.patch
ldiskfs/kernel_patches/patches/ext3-mmp-2.6.18-vanilla.patch

Comment by Build Master (Inactive) [ 18/Apr/11 ]

Integrated in lustre-master » server,el5-x86_64 #29
~~LU-206~~ Memory corruption due to MMP (1.8.5)

Oleg Drokin : 19de0afd629a3ffe714486cc75264ea14cea7346
Files :

ldiskfs/kernel_patches/patches/ext3-mmp-2.6.18-vanilla.patch
ldiskfs/kernel_patches/patches/ext4-mmp-rhel6.patch
ldiskfs/kernel_patches/patches/ext4-mmp-rhel5.patch

Comment by Build Master (Inactive) [ 18/Apr/11 ]

Integrated in lustre-master » client,el6-x86_64 #29
~~LU-206~~ Memory corruption due to MMP (1.8.5)

Oleg Drokin : 19de0afd629a3ffe714486cc75264ea14cea7346
Files :

ldiskfs/kernel_patches/patches/ext4-mmp-rhel6.patch
ldiskfs/kernel_patches/patches/ext4-mmp-rhel5.patch
ldiskfs/kernel_patches/patches/ext3-mmp-2.6.18-vanilla.patch

Comment by Build Master (Inactive) [ 18/Apr/11 ]

Integrated in lustre-master » server,el5-i686 #29
~~LU-206~~ Memory corruption due to MMP (1.8.5)

Oleg Drokin : 19de0afd629a3ffe714486cc75264ea14cea7346
Files :

ldiskfs/kernel_patches/patches/ext4-mmp-rhel5.patch
ldiskfs/kernel_patches/patches/ext3-mmp-2.6.18-vanilla.patch
ldiskfs/kernel_patches/patches/ext4-mmp-rhel6.patch