[LU-12618] List corruption in lnet

Related — Tue, 7 Apr 2020 14:21:54 +0000

[602080.689569] ------------[ cut here ]------------
[602080.694306] WARNING: CPU: 23 PID: 3716 at lib/list_debug.c:29 __list_add+0x65/0xc0
[602080.701989] list_add corruption. next->prev should be prev (ffff886bae1f0a98), but was dead000000000200. (next=ffff888f64636000).
[602080.713769] Modules linked in: osc(OE) mgc(OE) lustre(OE) lmv(OE) fld(OE) mdc(OE) fid(OE) lov(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) sha512_ssse3 sha512_generic crypto_null libcfs(OE) vtsspp(OE) sep4_1(OE) socperf2_0(OE) ebtable_filter ebtables ip6table_filter ip6_tables pax(OE) rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache bridge 8021q garp mrp stp llc rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx5_ib(OE) mlx4_en(OE) mlx4_ib(OE) ib_core(OE) mlx4_core(OE) ipt_REJECT nf_reject_ipv4 xt_conntrack iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle skx_edac edac_core vfat fat intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel
[602080.786461]  hpilo hpwdt aesni_intel lrw gf128mul glue_helper ablk_helper ses enclosure mei_me cryptd mei ipmi_si pcspkr shpchp joydev ipmi_devintf wmi sg ipmi_msghandler lpc_ich acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sunrpc knem(OE) ip_tables xfs sr_mod cdrom sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect mlx5_core(OE) sysimgblt mlxfw(OE) fb_sys_fops ahci ttm libahci mlx_compat(OE) bnx2x(OE) uas tg3(OE) devlink mdio drm smartpqi(OE) crct10dif_pclmul crct10dif_common ptp scsi_transport_sas libata usb_storage libcrc32c crc32c_intel i2c_core pps_core dm_mirror dm_region_hash dm_log dm_mod
[602080.843594] CPU: 23 PID: 3716 Comm: kiblnd_sd_11_01 Tainted: G        W  OE  ------------   3.10.0-693.el7.x86_64 #1
[602080.854240] Hardware name: HPE ProLiant DL360 Gen10/ProLiant DL360 Gen10, BIOS U32 06/20/2018
[602080.862880]  ffff88238d297c48 0000000029e8e2dd ffff88238d297bf8 ffffffff816a3d91
[602080.870425]  ffff88238d297c38 ffffffff810879c8 0000001dae25b800 ffff886b81b5ac00
[602080.877971]  ffff888f64636000 ffff886bae1f0a98 ffff886b81b5ac00 ffff886ba43f8030
[602080.885517] Call Trace:
[602080.888061]  [<ffffffff816a3d91>] dump_stack+0x19/0x1b
[602080.893305]  [<ffffffff810879c8>] __warn+0xd8/0x100
[602080.898286]  [<ffffffff81087a4f>] warn_slowpath_fmt+0x5f/0x80
[602080.904140]  [<ffffffff8133d8a5>] __list_add+0x65/0xc0
[602080.909388]  [<ffffffffc0d8b676>] lnet_msg_commit+0x66/0x180 [lnet]
[602080.915767]  [<ffffffffc0d981bd>] lnet_parse+0x3ed/0xcf0 [lnet]
[602080.921800]  [<ffffffffc0fedd9b>] kiblnd_handle_rx+0x1eb/0x640 [ko2iblnd]
[602080.928701]  [<ffffffffc0ff4336>] kiblnd_scheduler+0xe66/0x10a0 [ko2iblnd]
[602080.935687]  [<ffffffff810ce54e>] ? dequeue_task_fair+0x41e/0x660
[602080.941889]  [<ffffffff810c4810>] ? wake_up_state+0x20/0x20
[602080.947568]  [<ffffffffc0ff34d0>] ? kiblnd_cq_event+0x80/0x80 [ko2iblnd]
[602080.954379]  [<ffffffff810b098f>] kthread+0xcf/0xe0
[602080.959359]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[602080.965562]  [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[602080.971064]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40

After investigating, looks into piece of code:

.....
        if (msg->msg_rx_committed) {
                /* forwarding msg committed for both receiving and sending */
                if (cpt != msg->msg_rx_cpt) {
                        lnet_net_unlock(cpt);
                        cpt2 = msg->msg_rx_cpt;
                        lnet_net_lock(cpt2);
                }
                lnet_msg_decommit_rx(msg, status);
        }
......

The cpt2 store msg_rx_cpt and then get the cpt_lock, but the msg_rx_cpt could be changed in this period. Then we may mess up the msc_active list below.

Whamcloud Community JIRA

[LU-12618] List corruption in lnet