[LU-12581] Slab object "Freechain corrupt" on client

• Type: Bug
• Resolution: Fixed
• Priority: Minor
• Fix Version/s: Lustre 2.13.0, Lustre 2.12.3
• Severity: 3
• Assignee: Zhenyu Xu
• Reporter: Zhenyu Xu

Description

From a backtrace of a crashed node.

[167960.759257] INFO: Slab 0xffffd8487699eb80 objects=512 used=442 fp=0xffff91cd667ae428 flags=0x6fffff00000081
[167960.759260] INFO: Object 0xffff91cd667ae1a0 @offset=416 fp=0xffff000000000000
 
[167960.759266] Bytes b4 ffff91cd667ae190: f0 13 fe 80 cd 91 ff ff 98 14 fe 80 cd 91 ff ff ................
[167960.759277] Object ffff91cd667ae1a0: 00 00 00 00 00 00 ff ff ........
[167960.759283] CPU: 24 PID: 25390 Comm: pool Kdump: loaded Tainted: G B OE ------------ 3.10.0-957.21.3.el7.x86_64 #1
[167960.759285] Hardware name: HP ProLiant BL460c Gen9, BIOS I36 12/29/2018
[167960.759288] Call Trace:
[167960.759301] [<ffffffff99363107>] dump_stack+0x19/0x1b
[167960.759310] [<ffffffff98e18c51>] print_trailer+0x161/0x280
[167960.759315] [<ffffffff98e1987f>] on_freelist+0xff/0x270
[167960.759321] [<ffffffff9935f98f>] free_debug_processing+0xa7/0x253
[167960.759357] [<ffffffffc16a8965>] ? ll_xattr_set_3_11+0x605/0xa90 [lustre]
[167960.759362] [<ffffffff98dd64f5>] ? kvfree+0x35/0x40
[167960.759366] [<ffffffff98dd64f5>] ? kvfree+0x35/0x40
[167960.759371] [<ffffffff98e1bb70>] __slab_free+0x250/0x2f0
[167960.759377] [<ffffffff98e69828>] ? generic_setxattr+0x68/0x80
[167960.759381] [<ffffffff98e6a005>] ? __vfs_setxattr_noperm+0x65/0x1b0
[167960.759387] [<ffffffff98f2013e>] ? evm_inode_setxattr+0xe/0x10
[167960.759390] [<ffffffff98dd64f5>] ? kvfree+0x35/0x40
[167960.759395] [<ffffffff98e1c2d6>] kfree+0x106/0x140
[167960.759399] [<ffffffff98dd64f5>] kvfree+0x35/0x40
[167960.759403] [<ffffffff98e6a36b>] setxattr+0x15b/0x1e0
[167960.759408] [<ffffffff98e5327d>] ? putname+0x3d/0x60
[167960.759412] [<ffffffff98e54492>] ? user_path_at_empty+0x72/0xc0
[167960.759418] [<ffffffff98e448a8>] ? __sb_start_write+0x58/0x110
[167960.759423] [<ffffffff98e76c71>] ? do_utimes+0xf1/0x180
[167960.759428] [<ffffffff98e6a657>] SyS_setxattr+0xb7/0x100
[167960.759434] [<ffffffff99375ddb>] system_call_fastpath+0x22/0x27
[167964.710134] =============================================================================
[167964.710141] BUG kmalloc-8 (Tainted: G B OE ------------ ): Freechain corrupt
[167964.710143] -----------------------------------------------------------------------------
 
[167964.710148] INFO: Slab 0xffffd847f71be440 objects=512 used=510 fp=0xffff91ad86f910a0 flags=0x2fffff00000081
[167964.710151] INFO: Object 0xffff91ad86f910a8 @offset=168 fp=0xffff000000000000
 
[167964.710156] Bytes b4 ffff91ad86f91098: 30 29 78 f6 a2 91 ff ff a8 10 f9 86 ad 91 ff ff 0)x.............
[167964.710159] Object ffff91ad86f910a8: 00 00 00 00 00 00 ff ff ........
[167964.710164] CPU: 20 PID: 11480 Comm: ptlrpcd_00_00 Kdump: loaded Tainted: G B OE ------------ 3.10.0-957.21.3.el7.x86_64 #1
[167964.710167] Hardware name: HP ProLiant BL460c Gen9, BIOS I36 12/29/2018
[167964.710169] Call Trace:
[167964.710180] [<ffffffff99363107>] dump_stack+0x19/0x1b
[167964.710188] [<ffffffff98e18c51>] print_trailer+0x161/0x280
[167964.710193] [<ffffffff98e1987f>] on_freelist+0xff/0x270
[167964.710198] [<ffffffff9935f98f>] free_debug_processing+0xa7/0x253
[167964.710451] [<ffffffffc172e558>] ? brw_interpret+0x428/0xe70 [osc]
[167964.710460] [<ffffffffc172e558>] ? brw_interpret+0x428/0xe70 [osc]
[167964.710465] [<ffffffff98e1bb70>] __slab_free+0x250/0x2f0
[167964.710482] [<ffffffffc10bfc4e>] ? ktime_get_real_seconds+0xe/0x10 [libcfs]
[167964.710491] [<ffffffffc172e558>] ? brw_interpret+0x428/0xe70 [osc]
[167964.710496] [<ffffffff98e1c2d6>] kfree+0x106/0x140
[167964.710505] [<ffffffffc172e558>] brw_interpret+0x428/0xe70 [osc]
[167964.710554] [<ffffffffc148d2ab>] ptlrpc_check_set.part.23+0x44b/0x1db0 [ptlrpc]
[167964.710585] [<ffffffffc148ec6b>] ptlrpc_check_set+0x5b/0xe0 [ptlrpc]
[167964.710620] [<ffffffffc14baf8b>] ptlrpcd_check+0x4cb/0x5e0 [ptlrpc]
[167964.710651] [<ffffffffc14bb34b>] ptlrpcd+0x2ab/0x550 [ptlrpc]
[167964.710657] [<ffffffff98cd6b60>] ? wake_up_state+0x20/0x20
[167964.710687] [<ffffffffc14bb0a0>] ? ptlrpcd_check+0x5e0/0x5e0 [ptlrpc]
[167964.710692] [<ffffffff98cc1da1>] kthread+0xd1/0xe0
[167964.710696] [<ffffffff98cc1cd0>] ? insert_kthread_work+0x40/0x40
[167964.710702] [<ffffffff99375c37>] ret_from_fork_nospec_begin+0x21/0x21
[167964.710705] [<ffffffff98cc1cd0>] ? insert_kthread_work+0x40/0x40
[167966.447183] general protection fault: 0000 [#1] SMP 
[167966.470825] Modules linked in: nfsd osc(OE) mgc(OE) lustre(OE) lmv(OE) fld(OE) mdc(OE) fid(OE) lov(OE) cts ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) rpcsec_gss_krb5 nfsv4 dns_resolver libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx5_ib(OE) ib_uverbs(OE) mlx5_core(OE) mlxfw(OE) mlx4_en(OE) ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_recent xt_conntrack nf_conntrack iptable_filter dm_mirror dm_region_hash dm_log dm_mod mlx4_ib(OE) ib_core(OE) sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul crc32c_intel iTCO_wdt ghash_clmulni_intel iTCO_vendor_support aesni_intel lrw gf128mul glue_helper ablk_helper cryptd mlx4_core(OE) mgag200
[167966.818207] ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm mlx_compat(OE) drm_panel_orientation_quirks devlink ses enclosure hpwdt ioatdma pcspkr i2c_i801 lpc_ich ipmi_si pcc_cpufreq ipmi_devintf ipmi_msghandler acpi_power_meter wmi binfmt_misc knem(OE) auth_rpcgss ip_tables bridge stp llc xfs isci libsas qla3xxx e1000e igb i2c_algo_bit megaraid_sas aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase arcmsr ahci libahci sata_nv sata_svw bnx2x libcrc32c bnx2 ext4 mbcache jbd2 sata_sil libata tg3 e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg ixgbe crct10dif_pclmul crct10dif_common hpsa dca mdio ptp hpilo scsi_transport_sas pps_core [last unloaded: ipmi_msghandler]
[167967.161875] CPU: 16 PID: 25390 Comm: pool Kdump: loaded Tainted: G B OE ------------ 3.10.0-957.21.3.el7.x86_64 #1
[167967.219950] Hardware name: HP ProLiant BL460c Gen9, BIOS I36 12/29/2018
[167967.253323] task: ffff91a4c9b61040 ti: ffff91ab9d60c000 task.ti: ffff91ab9d60c000
[167967.291322] RIP: 0010:[<ffffffff98e1f093>] [<ffffffff98e1f093>] __kmalloc_node+0x193/0x2b0
[167967.332872] RSP: 0018:ffff91ab9d60fd30 EFLAGS: 00010246
[167967.360609] RAX: 0000000000000000 RBX: 00000000000000d0 RCX: 0000000000005f04
[167967.397223] RDX: 0000000000005f03 RSI: 0000000000000000 RDI: 000000000001f040
[167967.433284] RBP: ffff91ab9d60fd70 R08: ffff91afbfc5f040 R09: ffff91913fc03e00
[167967.468880] R10: ffff91913fc03e00 R11: ffffffff98dd646d R12: 00000000000000d0
[167967.505507] R13: 0000000000000006 R14: ffff000000000000 R15: 00000000ffffffff
[167967.541869] FS: 00002aaad2030700(0000) GS:ffff91afbfc40000(0000) knlGS:0000000000000000
[167967.582857] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[167967.612082] CR2: 00007ffff7ff7000 CR3: 000000143b61e000 CR4: 00000000003607e0
[167967.649899] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[167967.689542] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[167967.729248] Call Trace:
[167967.743250] [<ffffffff98dd646d>] ? kvmalloc_node+0x8d/0xe0
[167967.773337] [<ffffffff98dd646d>] kvmalloc_node+0x8d/0xe0
[167967.801855] [<ffffffff98e6a2c0>] setxattr+0xb0/0x1e0
[167967.829201] [<ffffffff98e5327d>] ? putname+0x3d/0x60
[167967.858680] [<ffffffff98e54492>] ? user_path_at_empty+0x72/0xc0
[167967.890204] [<ffffffff98e448a8>] ? __sb_start_write+0x58/0x110
[167967.921097] [<ffffffff98e76c71>] ? do_utimes+0xf1/0x180
[167967.949259] [<ffffffff98e6a657>] SyS_setxattr+0xb7/0x100
[167967.979511] [<ffffffff99375ddb>] system_call_fastpath+0x22/0x27
[167968.012007] Code: 4c 89 4d d0 e8 df 1d 01 00 4c 8b 4d d0 49 89 c2 4c 8b 5d c8 e9 eb fe ff ff 0f 1f 80 00 00 00 00 49 63 42 20 48 8d 4a 01 49 8b 3a <49> 8b 1c 06 4c 89 f0 65 48 0f c7 0f 0f 94 c0 84 c0 0f 84 c2 fe 
[167968.112107] RIP [<ffffffff98e1f093>] __kmalloc_node+0x193/0x2b0
[167968.148871] RSP <ffff91ab9d60fd30>