[LU-457] MDS crash under DDN file stress tests

• Status: Resolved
• Project: Lustre
• Affects Version/s: Lustre 1.8.6
• Type: Bug
• Priority: Major
• Reporter: Cliff White (Inactive)
• Assignee: Hongchao Zhang
• Resolution: Cannot Reproduce

Environment

Hyperion - LLNL Running chaos-4 , 2.6.18-238.12.1.el5_lustre.g266a955
1.8.6-RC3 build

• Severity: 3

Description

Running the DDN file stress tests with tests spread across 60 clients, MDS panic'd,
panic output attached.

Comments

[Johann Lombardi (Inactive)]

2011-06-23 14:55:52 Kernel BUG at mm/slab.c:3482
2011-06-23 14:55:52 Pid: 9565, comm: ldlm_cn_02 Tainted: G 2.6.18-238.12.1.el5_lustre.g266a955 #1
2011-06-23 14:55:52 RIP: 0010:[<ffffffff8000751c>] [<ffffffff8000751c>] kmem_cache_free+0x89/0x1dd
2011-06-23 14:55:52 RSP: 0018:ffff814026f15c80 EFLAGS: 00010207
2011-06-23 14:55:52 RAX: 0000000000000080 RBX: ffff81100104a840 RCX: 0000000000000200
2011-06-23 14:55:52 RDX: ffff810138039030 RSI: 0000000038039030 RDI: 0000000000000001
2011-06-23 14:55:52 RBP: ffff81102a4a9a00 R08: 0000000000000000 R09: 0000000000000000
2011-06-23 14:55:52 R10: ffff81100104a840 R11: 0000000000000158 R12: ffff811028ee5040
2011-06-23 14:55:52 R13: ffff81100104a840 R14: ffff81302530a0c0 R15: 00000000000000f6
2011-06-23 14:55:52 FS: 00002aaaab01f6e0(0000) GS:ffff811038155e40(0000) knlGS:0000000000000000
2011-06-23 14:55:52 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
2011-06-23 14:55:52 CR2: 00002aaaaad626e0 CR3: 0000002f43886000 CR4: 00000000000006e0
2011-06-23 14:55:52 Process ldlm_cn_02 (pid: 9565, threadinfo ffff814026f14000, task ffff814037bab040)
2011-06-23 14:55:52 Call Trace:
2011-06-23 14:55:52 [<ffffffff886ca3eb>] :ptlrpc:ldlm_resource_putref_internal+0x3ab/0x460
2011-06-23 14:55:52 LustreError: 3297:0:(mds_open.c:1693:mds_close()) @@@ no handle for file close ino 81431268: cookie 0x0 req@ffff813fd3b6b000 x1372347904393553/t0 o35->c339861d-0c0a-b9ee-a39d-9cfe591452c1@NET_0x50000c0a87223_UUID:0/0 lens 408/864 e 0 to 0 dl 1308866169 ref 1 fl Interpret:/0/0 rc 0/0
2011-06-23 14:55:52 LustreError: 3297:0:(mds_open.c:1693:mds_close()) Skipped 3269 previous similar messages
2011-06-23 14:55:52 [<ffffffff886c58d2>] :ptlrpc:ldlm_lock_put+0x372/0x3d0
2011-06-23 14:55:52 [<ffffffff886e58a2>] :ptlrpc:ldlm_request_cancel+0x222/0x330
2011-06-23 14:55:52 [<ffffffff8870bd69>] :ptlrpc:lustre_pack_reply+0x29/0xb0
2011-06-23 14:55:52 [<ffffffff886e5bcd>] :ptlrpc:ldlm_handle_cancel+0x21d/0x2e0
2011-06-23 14:55:52 [<ffffffff88707ef5>] :ptlrpc:lustre_msg_get_opc+0x35/0xf0
2011-06-23 14:55:52 [<ffffffff886e5f98>] :ptlrpc:ldlm_cancel_handler+0x308/0x690
2011-06-23 14:55:52 [<ffffffff88707ef5>] :ptlrpc:lustre_msg_get_opc+0x35/0xf0
2011-06-23 14:55:52 [<ffffffff887176d9>] :ptlrpc:ptlrpc_server_handle_request+0x989/0xe00
2011-06-23 14:55:52 [<ffffffff88717e35>] :ptlrpc:ptlrpc_wait_event+0x2e5/0x310
2011-06-23 14:55:52 [<ffffffff8008c85d>] __wake_up_common+0x3e/0x68
2011-06-23 14:55:52 [<ffffffff88718dc6>] :ptlrpc:ptlrpc_main+0xf66/0x1120
2011-06-23 14:55:52 [<ffffffff8005dfb1>] child_rip+0xa/0x11
2011-06-23 14:55:52 [<ffffffff88717e60>] :ptlrpc:ptlrpc_main+0x0/0x1120
2011-06-23 14:55:52 [<ffffffff8005dfa7>] child_rip+0x0/0x11

3478 void kmem_cache_free(struct kmem_cache *cachep, void *objp)
3479 {
3480 unsigned long flags;
3481
3482 BUG_ON(virt_to_cache(objp) != cachep);

It looks like a slab corruption. Cliff, is this easy to reproduce?

[Cliff White (Inactive)]

And we have the second crash, but in a different location.
Data attached, will reboot Hyperion servers onto 1.8.5-56 at 9 or so PDT unless otherwise advised

[Cliff White (Inactive)]

It seems I can make it crash again, but not the same location.

[Peter Jones]

HongChao

Please can you look into this bug as your top priority today and make sure to provide a status update as to your findings (or just what you have looked into) before you leave for the day.

Thanks

Peter

[Andreas Dilger]

A second crash in a diiferent part of the code makes me think that there is memory corruption happening.

Do we have any ability to enable slab debug, list debug, etc to try and isolate this problem earlier?

On very tricky cases, Ricardo had also written a patch that made all allocations "use on once" and any thread that accessed old memory would Oops and dump a stack.

[Johann Lombardi (Inactive)]

> A second crash in a diiferent part of the code makes me think that there is memory corruption happening.

Yes, that's what i initially suspected.

> Do we have any ability to enable slab debug, list debug, etc to try and isolate this problem earlier?

I think the chaos kernel has list debug but no slab debug.
Also, it would be great to get a crash dump.

[Cliff White (Inactive)]

Reverted to 1.8.5 GA, ext4 version lustre-modules-1.8.5-2.6.18_194.17.1.el5_lustre.1.8.5 lustre-tests-1.8.5-2.6.18_194.17.1.el5_lustre.1.8.5 kernel-2.6.18-194.17.1.el5_lustre.1.8.5 lustre-tools-llnl-1.2-6.ch4.4 lustre-1.8.5-2.6.18_194.17.1.el5_lustre.1.8.5 lustre-ldiskfs-3.1.4-2.6.18_194.17.1.el5_lustre.1.8.5 Have run fsstress for > 45 minutes, no crash, few errors - RAID is still rebuiding. I am attaching the full console log from the last crash, from start of MDT to crash.

[Cliff White (Inactive)]

Spoke (literally) about two minutes too soon, MDT just panic'd - Log attached, previous mds log not attached

[Cliff White (Inactive)]

Crash running 1.8.5 GA -ext4

[Johann Lombardi (Inactive)]

All crashes are in different area of the code. Those kind of problem are hard to diagnose.
We really need to recompile the kernel with slab debugging and collect a crash dump.
If not possible on hyperion, can we try on toro?
Cliff, can you reproduce it at a smaller scale (small = something that could fit in toro)?

[Hongchao Zhang]

it's hard to diagnose the problem. in 1.8.5 GA+ext4, the NULL pointer reference in RCU callback context should be a very rare
case, what kind of problems can cause it?
can we do this stress test with ext3 (<8TB) under 1.8.6/1.8.5 to check whether it still panics, then isolate the problem to be
an ext4-related issue?

[Hongchao Zhang]

btw, I have discussed with Yujian, and she said the fsstress was only tested against 2.X before, and the scale was about
120 clients and no problem, but she was not clear whether the ldiskfs in those tests was against ext4.

[Cliff White (Inactive)]

This is the MDS console log from the second 1.8.6, showing everything from mount to crash

[Cliff White (Inactive)]

Second 1.8.6 crash

[Cliff White (Inactive)]

Replicate 1.8.5 GA crash w/10 clients

[Jian Yu]

btw, I have discussed with Yujian, and she said the fsstress was only tested against 2.X before, and the scale was about
120 clients and no problem, but she was not clear whether the ldiskfs in those tests was against ext4.

After doing a search on Bugzilla, I found the latest run of this test on Lustre master branch was against 2.0.0 Build 41 (v1_10_0_41):

Branch: v1_10_0_41(client and servers) - ext3
Distro/Arch: RHEL5/x86_64(client and servers)
Cluster: Hyperion
MDS Node: hyperion-mds7
OSS Node: hyperion-ost[5,6,7]
Client Nodes: 224 (patchless)
Network Type: o2ib
Test: client mount 4 mounts but run fs_test only on 1
Result: bug 22802

And it was also run on Lustre b1_8 branch several times, the latest run was against 1.8.2 and passed, please refer to bug 21292.

[Cliff White (Inactive)]

System crashed after 2 hours, but immediately rebooted - no crash dump. Log attached

[Cliff White (Inactive)]

Friday night fsstress crash

[Hongchao Zhang]

in the corrupted slab cache, the block size is 128, and both the previous and next block of the corrupted one are used
by end IO routine of RAID10, maybe this issue be related to RAID10? but nothing found at the related code in 2.6.18-238.12.1.

[Build Master (Inactive)]

Integrated in lustre-b1_8 » x86_64,client,el6,inkernel #93
LU-457 Enable slab debug for RHEL5/x86_64

Johann Lombardi : 529529a3dfbb8849f479c15f75e8f875ba5427b4
Files :

• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config
• lustre/mds/mds_open.c
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config
• lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch

[Build Master (Inactive)]

Integrated in lustre-b1_8 » i686,client,el6,inkernel #93
LU-457 Enable slab debug for RHEL5/x86_64

Johann Lombardi : 529529a3dfbb8849f479c15f75e8f875ba5427b4
Files :

• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config
• lustre/mds/mds_open.c
• lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch

[Build Master (Inactive)]

Integrated in lustre-b1_8 » i686,client,el5,ofa #93
LU-457 Enable slab debug for RHEL5/x86_64

Johann Lombardi : 529529a3dfbb8849f479c15f75e8f875ba5427b4
Files :

• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config
• lustre/mds/mds_open.c
• lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch

[Build Master (Inactive)]

Integrated in lustre-b1_8 » x86_64,client,ubuntu1004,inkernel #93
LU-457 Enable slab debug for RHEL5/x86_64

Johann Lombardi : 529529a3dfbb8849f479c15f75e8f875ba5427b4
Files :

• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config
• lustre/mds/mds_open.c
• lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch

[Build Master (Inactive)]

Integrated in lustre-b1_8 » i686,client,el5,inkernel #93
LU-457 Enable slab debug for RHEL5/x86_64

Johann Lombardi : 529529a3dfbb8849f479c15f75e8f875ba5427b4
Files :

• lustre/mds/mds_open.c
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config
• lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config

[Build Master (Inactive)]

Integrated in lustre-b1_8 » i686,server,el5,ofa #93
LU-457 Enable slab debug for RHEL5/x86_64

Johann Lombardi : 529529a3dfbb8849f479c15f75e8f875ba5427b4
Files :

• lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch
• lustre/mds/mds_open.c
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config

[Build Master (Inactive)]

Integrated in lustre-b1_8 » i686,server,el5,inkernel #93
LU-457 Enable slab debug for RHEL5/x86_64

Johann Lombardi : 529529a3dfbb8849f479c15f75e8f875ba5427b4
Files :

• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config
• lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch
• lustre/mds/mds_open.c

[Build Master (Inactive)]

Integrated in lustre-b1_8 » x86_64,server,el5,ofa #93
LU-457 Enable slab debug for RHEL5/x86_64

Johann Lombardi : 529529a3dfbb8849f479c15f75e8f875ba5427b4
Files :

• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config
• lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch
• lustre/mds/mds_open.c
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config

[Build Master (Inactive)]

Integrated in lustre-b1_8 » x86_64,client,el5,inkernel #93
LU-457 Enable slab debug for RHEL5/x86_64

Johann Lombardi : 529529a3dfbb8849f479c15f75e8f875ba5427b4
Files :

• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config
• lustre/mds/mds_open.c
• lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config

[Build Master (Inactive)]

Integrated in lustre-b1_8 » x86_64,client,el5,ofa #93
LU-457 Enable slab debug for RHEL5/x86_64

Johann Lombardi : 529529a3dfbb8849f479c15f75e8f875ba5427b4
Files :

• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config
• lustre/mds/mds_open.c
• lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config

[Build Master (Inactive)]

Integrated in lustre-b1_8 » x86_64,server,el5,inkernel #93
LU-457 Enable slab debug for RHEL5/x86_64

Johann Lombardi : 529529a3dfbb8849f479c15f75e8f875ba5427b4
Files :

• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config
• lustre/mds/mds_open.c
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config
• lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch

[Peter Jones]

This is suspected to be unrelated to Lustre. The Hyperion configuration has been fixed and Cliff will re-test next week to confirm

[Build Master (Inactive)]

Integrated in lustre-b1_8 » x86_64,client,el5,inkernel #95
Revert "LU-457 Enable slab debug for RHEL5/x86_64"

Johann Lombardi : a0375b393ef34c457bec68835ed6ac2f37b5a537
Files :

• lustre/mds/mds_open.c
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config
• lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config

[Build Master (Inactive)]

Integrated in lustre-b1_8 » x86_64,client,el5,ofa #95
Revert "LU-457 Enable slab debug for RHEL5/x86_64"

Johann Lombardi : a0375b393ef34c457bec68835ed6ac2f37b5a537
Files :

• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config
• lustre/mds/mds_open.c
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config
• lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch

[Build Master (Inactive)]

Integrated in lustre-b1_8 » x86_64,server,el5,ofa #95
Revert "LU-457 Enable slab debug for RHEL5/x86_64"

Johann Lombardi : a0375b393ef34c457bec68835ed6ac2f37b5a537
Files :

• lustre/mds/mds_open.c
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config
• lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config

[Build Master (Inactive)]

Integrated in lustre-b1_8 » i686,client,el6,inkernel #95
Revert "LU-457 Enable slab debug for RHEL5/x86_64"

Johann Lombardi : a0375b393ef34c457bec68835ed6ac2f37b5a537
Files :

• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config
• lustre/mds/mds_open.c
• lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config

[Build Master (Inactive)]

Integrated in lustre-b1_8 » x86_64,client,el6,inkernel #95
Revert "LU-457 Enable slab debug for RHEL5/x86_64"

Johann Lombardi : a0375b393ef34c457bec68835ed6ac2f37b5a537
Files :

• lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config
• lustre/mds/mds_open.c

[Build Master (Inactive)]

Integrated in lustre-b1_8 » i686,client,el5,inkernel #95
Revert "LU-457 Enable slab debug for RHEL5/x86_64"

Johann Lombardi : a0375b393ef34c457bec68835ed6ac2f37b5a537
Files :

• lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch
• lustre/mds/mds_open.c
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config

[Build Master (Inactive)]

Integrated in lustre-b1_8 » x86_64,server,el5,inkernel #95
Revert "LU-457 Enable slab debug for RHEL5/x86_64"

Johann Lombardi : a0375b393ef34c457bec68835ed6ac2f37b5a537
Files :

• lustre/mds/mds_open.c
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config
• lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch

[Build Master (Inactive)]

Integrated in lustre-b1_8 » x86_64,client,ubuntu1004,inkernel #95
Revert "LU-457 Enable slab debug for RHEL5/x86_64"

Johann Lombardi : a0375b393ef34c457bec68835ed6ac2f37b5a537
Files :

• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config
• lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch
• lustre/mds/mds_open.c

[Build Master (Inactive)]

Integrated in lustre-b1_8 » i686,client,el5,ofa #95
Revert "LU-457 Enable slab debug for RHEL5/x86_64"

Johann Lombardi : a0375b393ef34c457bec68835ed6ac2f37b5a537
Files :

• lustre/mds/mds_open.c
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config
• lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config

[Build Master (Inactive)]

Integrated in lustre-b1_8 » i686,server,el5,ofa #95
Revert "LU-457 Enable slab debug for RHEL5/x86_64"

Johann Lombardi : a0375b393ef34c457bec68835ed6ac2f37b5a537
Files :

• lustre/mds/mds_open.c
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config
• lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config

[Build Master (Inactive)]

Integrated in lustre-b1_8 » i686,server,el5,inkernel #95
Revert "LU-457 Enable slab debug for RHEL5/x86_64"

Johann Lombardi : a0375b393ef34c457bec68835ed6ac2f37b5a537
Files :

• lustre/kernel_patches/patches/md-avoid-bug_on-when-bmc-overflow.patch
• lustre/mds/mds_open.c
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64-smp.config
• lustre/kernel_patches/kernel_configs/kernel-2.6.18-2.6-rhel5-x86_64.config

[Peter Jones]

Has not happened for a long time so let's close and reopen if it reoccurs