[LU-5764] Crash of MDS on "apparent buffer overflow"

Related — Wed, 20 May 2015 13:20:10 +0000
Hi,
On a file system with 300 OSTs, messages about "apparent buffer overflow" can be seen in the syslog of the MDS, and after some time (between 5 and 30 minutes), the MDS crashes.
Here is the console output:
<3>proc_file_read: Apparent buffer overflow!
<3>proc_file_read: Apparent buffer overflow!
<3>proc_file_read: Apparent buffer overflow!
<3>proc_file_read: Apparent buffer overflow!
<3>proc_file_read: Apparent buffer overflow!
<3>proc_file_read: Apparent buffer overflow!
<4>------------[ cut here ]------------
<4>WARNING: at lib/list_debug.c:48 list_del+0x6e/0xa0() (Not tainted)
<4>Hardware name: bullx
<4>list_del corruption. prev->next should be ffff880335386000, but was 4d2d3030332d7366
<4>Modules linked in: osp(U) mdd(U) lfsck(U) lod(U) mdt(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lustre(U) lov(U) osc(U) mdc(U) lquota(U) fid(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc ipmi_devintf acpi_cpufreq freq_table mperf rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U) ib_cm(U) ipv6 ib_uverbs(U) ib_umad(U) mlx4_ib(U) ib_sa(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_round_robin scsi_dh_emc dm_multipath mic(U) uinput ses enclosure serio_raw compat(U) cxgb3 mdio lpfc scsi_transport_fc scsi_tgt igb i2c_algo_bit i2c_core ptp pps_core sg lpc_ich mfd_core ioatdma dca shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom aacraid ata_generic pata_jmicron usb_storage ahci dm_mirror dm_region_hash dm_log dm_mod megaraid_sas [last unloaded: scsi_wait_scan]
<4>Pid: 28, comm: events/1 Not tainted 2.6.32-431.29.2.el6.Bull.58.x86_64 #1
<4>Call Trace:
<4> [<ffffffff81070e77>] ? warn_slowpath_common+0x87/0xc0
<4> [<ffffffff81070f66>] ? warn_slowpath_fmt+0x46/0x50
<4> [<ffffffff8129593e>] ? list_del+0x6e/0xa0
<4> [<ffffffff81171008>] ? free_block+0xc8/0x180
<4> [<ffffffff811712f1>] ? drain_array+0xc1/0x100
<4> [<ffffffff811721de>] ? cache_reap+0x8e/0x250
<4> [<ffffffff81172150>] ? cache_reap+0x0/0x250
<4> [<ffffffff81093d80>] ? worker_thread+0x170/0x2a0
<4> [<ffffffff8109a300>] ? autoremove_wake_function+0x0/0x40
<4> [<ffffffff81093c10>] ? worker_thread+0x0/0x2a0
<4> [<ffffffff81099f56>] ? kthread+0x96/0xa0
<4> [<ffffffff8100c20a>] ? child_rip+0xa/0x20
<4> [<ffffffff81099ec0>] ? kthread+0x0/0xa0
<4> [<ffffffff8100c200>] ? child_rip+0x0/0x20
<4>---[ end trace cc0bf07e83b7a669 ]---
<4>general protection fault: 0000 [#1] SMP
<4>last sysfs file: /sys/devices/pci0000:80/0000:80:07.0/0000:85:00.0/host12/rport-12:0-0/target12:0:0/12:0:0:19/state
<4>CPU 1
<4>Modules linked in: osp(U) mdd(U) lfsck(U) lod(U) mdt(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lustre(U) lov(U) osc(U) mdc(U) lquota(U) fid(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc ipmi_devintf acpi_cpufreq freq_table mperf rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U) ib_cm(U) ipv6 ib_uverbs(U) ib_umad(U) mlx4_ib(U) ib_sa(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_round_robin scsi_dh_emc dm_multipath mic(U) uinput ses enclosure serio_raw compat(U) cxgb3 mdio lpfc scsi_transport_fc scsi_tgt igb i2c_algo_bit i2c_core ptp pps_core sg lpc_ich mfd_core ioatdma dca shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom aacraid ata_generic pata_jmicron usb_storage ahci dm_mirror dm_region_hash dm_log dm_mod megaraid_sas [last unloaded: scsi_wait_scan]
<4>
<4>Pid: 28, comm: events/1 Tainted: G        W  ---------------    2.6.32-431.29.2.el6.Bull.58.x86_64 #1 Bull SAS bullx/X8DAH
<4>RIP: 0010:[<ffffffff812958e0>]  [<ffffffff812958e0>] list_del+0x10/0xa0
<4>RSP: 0018:ffff88033acf5d10  EFLAGS: 00010082
<4>RAX: 6c2d303030305444 RBX: ffff88032d169000 RCX: 000000000000100c
<4>RDX: ffff88033fee0340 RSI: ffff88032d174000 RDI: ffff88032d169000
<4>RBP: ffff88033acf5d20 R08: ffff88033fee0340 R09: 0000000000000006
<4>R10: 0000000000000001 R11: 0000000000000000 R12: 000000000000000b
<4>R13: ffff88033ac11e58 R14: 0000000000000008 R15: ffffea0000000000
<4>FS:  0000000000000000(0000) GS:ffff880028220000(0000) knlGS:0000000000000000
<4>CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
<4>CR2: 00007f5482ad7000 CR3: 000000062e1c4000 CR4: 00000000000007e0
<4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<4>Process events/1 (pid: 28, threadinfo ffff88033acf4000, task ffff88033acccb40)
<4>Stack:
<4> 000000000000000b ffff88063cd30400 ffff88033acf5d80 ffffffff81171008
<4><d> ffff88033fee0340 ffff88032d169000 000000000000100c ffff88032d169b40
<4><d> 0000000000016cc0 ffff88033ac11e00 ffff88063cd30400 000000000000000b
<4>Call Trace:
<4> [<ffffffff81171008>] free_block+0xc8/0x180
<4> [<ffffffff811712f1>] drain_array+0xc1/0x100
<4> [<ffffffff811721de>] cache_reap+0x8e/0x250
<4> [<ffffffff81172150>] ? cache_reap+0x0/0x250
<4> [<ffffffff81093d80>] worker_thread+0x170/0x2a0
<4> [<ffffffff8109a300>] ? autoremove_wake_function+0x0/0x40
<4> [<ffffffff81093c10>] ? worker_thread+0x0/0x2a0
<4> [<ffffffff81099f56>] kthread+0x96/0xa0
<4> [<ffffffff8100c20a>] child_rip+0xa/0x20
<4> [<ffffffff81099ec0>] ? kthread+0x0/0xa0
<4> [<ffffffff8100c200>] ? child_rip+0x0/0x20
<4>Code: 89 95 fc fe ff ff e9 ab fd ff ff 4c 8b ad e8 fe ff ff e9 db fd ff ff 90 90 90 90 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 8b 47 08 <4c> 8b 00 4c 39 c7 75 39 48 8b 03 4c 8b 40 08 4c 39 c3 75 4c 48
<1>RIP  [<ffffffff812958e0>] list_del+0x10/0xa0
<4> RSP <ffff88033acf5d10>
This issue seems to be related to ~~LU-4483~~, but unfortunately there is no fix yet.
Thanks,
Sebastien.
Whamcloud Community JIRA

[LU-5764] Crash of MDS on "apparent buffer overflow"
