Details
-
Bug
-
Resolution: Fixed
-
Major
-
None
-
Lustre 2.5.3
-
None
-
3
-
16172
Description
Hi,
On a file system with 300 OSTs, messages about "apparent buffer overflow" can be seen in the syslog of the MDS, and after some time (between 5 and 30 minutes), the MDS crashes.
Here is the console output:
<3>proc_file_read: Apparent buffer overflow! <3>proc_file_read: Apparent buffer overflow! <3>proc_file_read: Apparent buffer overflow! <3>proc_file_read: Apparent buffer overflow! <3>proc_file_read: Apparent buffer overflow! <3>proc_file_read: Apparent buffer overflow! <4>------------[ cut here ]------------ <4>WARNING: at lib/list_debug.c:48 list_del+0x6e/0xa0() (Not tainted) <4>Hardware name: bullx <4>list_del corruption. prev->next should be ffff880335386000, but was 4d2d3030332d7366 <4>Modules linked in: osp(U) mdd(U) lfsck(U) lod(U) mdt(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lustre(U) lov(U) osc(U) mdc(U) lquota(U) fid(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc ipmi_devintf acpi_cpufreq freq_table mperf rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U) ib_cm(U) ipv6 ib_uverbs(U) ib_umad(U) mlx4_ib(U) ib_sa(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_round_robin scsi_dh_emc dm_multipath mic(U) uinput ses enclosure serio_raw compat(U) cxgb3 mdio lpfc scsi_transport_fc scsi_tgt igb i2c_algo_bit i2c_core ptp pps_core sg lpc_ich mfd_core ioatdma dca shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom aacraid ata_generic pata_jmicron usb_storage ahci dm_mirror dm_region_hash dm_log dm_mod megaraid_sas [last unloaded: scsi_wait_scan] <4>Pid: 28, comm: events/1 Not tainted 2.6.32-431.29.2.el6.Bull.58.x86_64 #1 <4>Call Trace: <4> [<ffffffff81070e77>] ? warn_slowpath_common+0x87/0xc0 <4> [<ffffffff81070f66>] ? warn_slowpath_fmt+0x46/0x50 <4> [<ffffffff8129593e>] ? list_del+0x6e/0xa0 <4> [<ffffffff81171008>] ? free_block+0xc8/0x180 <4> [<ffffffff811712f1>] ? drain_array+0xc1/0x100 <4> [<ffffffff811721de>] ? cache_reap+0x8e/0x250 <4> [<ffffffff81172150>] ? cache_reap+0x0/0x250 <4> [<ffffffff81093d80>] ? worker_thread+0x170/0x2a0 <4> [<ffffffff8109a300>] ? autoremove_wake_function+0x0/0x40 <4> [<ffffffff81093c10>] ? worker_thread+0x0/0x2a0 <4> [<ffffffff81099f56>] ? kthread+0x96/0xa0 <4> [<ffffffff8100c20a>] ? child_rip+0xa/0x20 <4> [<ffffffff81099ec0>] ? kthread+0x0/0xa0 <4> [<ffffffff8100c200>] ? child_rip+0x0/0x20 <4>---[ end trace cc0bf07e83b7a669 ]--- <4>general protection fault: 0000 [#1] SMP <4>last sysfs file: /sys/devices/pci0000:80/0000:80:07.0/0000:85:00.0/host12/rport-12:0-0/target12:0:0/12:0:0:19/state <4>CPU 1 <4>Modules linked in: osp(U) mdd(U) lfsck(U) lod(U) mdt(U) mgc(U) fsfilt_ldiskfs(U) osd_ldiskfs(U) ldiskfs(U) lustre(U) lov(U) osc(U) mdc(U) lquota(U) fid(U) fld(U) ko2iblnd(U) ptlrpc(U) obdclass(U) lnet(U) lvfs(U) sha512_generic sha256_generic crc32c_intel libcfs(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc ipmi_devintf acpi_cpufreq freq_table mperf rdma_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U) ib_cm(U) ipv6 ib_uverbs(U) ib_umad(U) mlx4_ib(U) ib_sa(U) mlx4_core(U) ib_mthca(U) ib_mad(U) ib_core(U) dm_round_robin scsi_dh_emc dm_multipath mic(U) uinput ses enclosure serio_raw compat(U) cxgb3 mdio lpfc scsi_transport_fc scsi_tgt igb i2c_algo_bit i2c_core ptp pps_core sg lpc_ich mfd_core ioatdma dca shpchp ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom aacraid ata_generic pata_jmicron usb_storage ahci dm_mirror dm_region_hash dm_log dm_mod megaraid_sas [last unloaded: scsi_wait_scan] <4> <4>Pid: 28, comm: events/1 Tainted: G W --------------- 2.6.32-431.29.2.el6.Bull.58.x86_64 #1 Bull SAS bullx/X8DAH <4>RIP: 0010:[<ffffffff812958e0>] [<ffffffff812958e0>] list_del+0x10/0xa0 <4>RSP: 0018:ffff88033acf5d10 EFLAGS: 00010082 <4>RAX: 6c2d303030305444 RBX: ffff88032d169000 RCX: 000000000000100c <4>RDX: ffff88033fee0340 RSI: ffff88032d174000 RDI: ffff88032d169000 <4>RBP: ffff88033acf5d20 R08: ffff88033fee0340 R09: 0000000000000006 <4>R10: 0000000000000001 R11: 0000000000000000 R12: 000000000000000b <4>R13: ffff88033ac11e58 R14: 0000000000000008 R15: ffffea0000000000 <4>FS: 0000000000000000(0000) GS:ffff880028220000(0000) knlGS:0000000000000000 <4>CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b <4>CR2: 00007f5482ad7000 CR3: 000000062e1c4000 CR4: 00000000000007e0 <4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 <4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 <4>Process events/1 (pid: 28, threadinfo ffff88033acf4000, task ffff88033acccb40) <4>Stack: <4> 000000000000000b ffff88063cd30400 ffff88033acf5d80 ffffffff81171008 <4><d> ffff88033fee0340 ffff88032d169000 000000000000100c ffff88032d169b40 <4><d> 0000000000016cc0 ffff88033ac11e00 ffff88063cd30400 000000000000000b <4>Call Trace: <4> [<ffffffff81171008>] free_block+0xc8/0x180 <4> [<ffffffff811712f1>] drain_array+0xc1/0x100 <4> [<ffffffff811721de>] cache_reap+0x8e/0x250 <4> [<ffffffff81172150>] ? cache_reap+0x0/0x250 <4> [<ffffffff81093d80>] worker_thread+0x170/0x2a0 <4> [<ffffffff8109a300>] ? autoremove_wake_function+0x0/0x40 <4> [<ffffffff81093c10>] ? worker_thread+0x0/0x2a0 <4> [<ffffffff81099f56>] kthread+0x96/0xa0 <4> [<ffffffff8100c20a>] child_rip+0xa/0x20 <4> [<ffffffff81099ec0>] ? kthread+0x0/0xa0 <4> [<ffffffff8100c200>] ? child_rip+0x0/0x20 <4>Code: 89 95 fc fe ff ff e9 ab fd ff ff 4c 8b ad e8 fe ff ff e9 db fd ff ff 90 90 90 90 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 8b 47 08 <4c> 8b 00 4c 39 c7 75 39 48 8b 03 4c 8b 40 08 4c 39 c3 75 4c 48 <1>RIP [<ffffffff812958e0>] list_del+0x10/0xa0 <4> RSP <ffff88033acf5d10>
This issue seems to be related to LU-4483, but unfortunately there is no fix yet.
Thanks,
Sebastien.
