[LU-11702] ibc_rxs is created and freed with different size

Mon, 20 Dec 2021 21:03:33 +0000

[ 8125.253125] LNet: Using FastReg for registration
[ 8125.831072] LNet: Added LNI 172.18.70.140@o2ib1 [128/2048/0/180]
[ 8125.840305] LNet: 8866:0:(o2iblnd.c:943:kiblnd_create_conn()) peer 10.14.252.1@o2ib1 - queue depth reduced from 128 to 21  to allow for qp creation
[ 8125.856172] LNetError: 4:0:(o2iblnd_cb.c:2840:kiblnd_rejected()) 10.14.252.1@o2ib1 rejected: consumer defined fatal error
[ 8125.870235] LNet: 4:0:(o2iblnd_cb.c:2207:kiblnd_peer_connect_failed()) Deleting messages for 10.14.252.1@o2ib1: connection failed
[ 8125.870273] LNet: 9852:0:(o2iblnd_cb.c:497:kiblnd_rx_complete()) Rx from 10.14.252.1@o2ib1 failed: 5
[ 8125.871400] BUG: unable to handle kernel paging request at ffffeb04004b9a40
[ 8125.871405] IP: [<ffffffff811dee05>] kfree+0x55/0x140
[ 8125.871406] PGD 0 
[ 8125.871407] Oops: 0000 [#1] SMP 
[ 8125.871432] Modules linked in: ko2iblnd(OE) lnet(OE) libcfs(OE) rpcrdma ib_isert iscsi_target_mod target_core_mod ib_iser libiscsi scsi_transport_iscsi ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx5_ib edac_core intel_powerclamp coretemp intel_rapl iosf_mbi ttm iTCO_wdt drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops iTCO_vendor_support drm kvm irqbypass joydev crc32_pclmul ghash_clmulni_intel ipmi_si aesni_intel ipmi_devintf pcspkr lrw ipmi_msghandler gf128mul wmi acpi_pad glue_helper ablk_helper cryptd ioatdma acpi_power_meter sg i2c_i801 mei_me mei lpc_ich shpchp acpi_cpufreq nfsd binfmt_misc auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic hfi1(OE) rdmavt(OE) mlx5_core igb ib_core ahci devlink crct10dif_pclmul
[ 8125.871436]  libahci crct10dif_common ptp crc32c_intel i2c_algo_bit libata dca i2c_core pps_core dm_mirror dm_region_hash dm_log dm_mod
[ 8125.871438] CPU: 17 PID: 9844 Comm: kiblnd_connd Tainted: G          IOE  ------------   3.10.0-693.el7.x86_64 #1
[ 8125.871439] Hardware name: Intel Corporation S2600KPF/S2600KPF, BIOS SE5C610.86B.01.01.0008.021120151325 02/11/2015
[ 8125.871440] task: ffff880174259fa0 ti: ffff88084df30000 task.ti: ffff88084df30000
[ 8125.871442] RIP: 0010:[<ffffffff811dee05>]  [<ffffffff811dee05>] kfree+0x55/0x140
[ 8125.871442] RSP: 0018:ffff88084df33dc8  EFLAGS: 00010282
[ 8125.871443] RAX: ffffeb04004b9a40 RBX: ffffc90012e69000 RCX: 0000000180080007
[ 8125.871443] RDX: 000077ff80000000 RSI: ffffea0041778400 RDI: ffffc90012e69000
[ 8125.871444] RBP: ffff88084df33de0 R08: ffff88105de10000 R09: 0000000180080007
[ 8125.871444] R10: 0000000000000001 R11: ffffea0041778400 R12: ffff8808f3fca800
[ 8125.871444] R13: ffffffffc0817294 R14: 00000000000011e0 R15: ffff881053ac3000
[ 8125.871445] FS:  0000000000000000(0000) GS:ffff88085ee40000(0000) knlGS:0000000000000000
[ 8125.871446] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 8125.871446] CR2: ffffeb04004b9a40 CR3: 00000000019f2000 CR4: 00000000001407e0
[ 8125.871447] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8125.871447] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8125.871447] Stack:
[ 8125.871448]  ffff881053ac3000 ffff8808f3fca800 ffff8808426e6000 ffff88084df33e18
[ 8125.871449]  ffffffffc0817294 ffff881053ac3018 0000000000000246 00000000000003e8
[ 8125.871450]  000000005b67fd54 ffff881053ac3000 ffff88084df33ec0 ffffffffc082a5e7
[ 8125.871450] Call Trace:
[ 8125.871457]  [<ffffffffc0817294>] kiblnd_destroy_conn+0x184/0x6e0 [ko2iblnd]
[ 8125.871462]  [<ffffffffc082a5e7>] kiblnd_connd+0x6a7/0xa00 [ko2iblnd]
[ 8125.871465]  [<ffffffff810c4810>] ? wake_up_state+0x20/0x20
[ 8125.871468]  [<ffffffffc0829f40>] ? kiblnd_check_conns+0x860/0x860 [ko2iblnd]
[ 8125.871470]  [<ffffffff810b098f>] kthread+0xcf/0xe0
[ 8125.871471]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[ 8125.871474]  [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[ 8125.871475]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[ 8125.871484] Code: 8b 15 30 a2 81 00 48 01 d8 72 0e 48 c7 c2 00 00 00 80 48 2b 15 05 4e 84 00 48 01 d0 48 c1 e8 0c 48 c1 e0 06 48 03 05 e3 4d 84 00 <48> 8b 10 80 e6 80 0f 85 c5 00 00 00 49 89 c3 49 8b 03 a8 80 0f 
[ 8125.871486] RIP  [<ffffffff811dee05>] kfree+0x55/0x140
[ 8125.871486]  RSP <ffff88084df33dc8>
[ 8125.871486] CR2: ffffeb04004b9a40

00000800:00000010:3.0:1533566495.528345:0:5501:0:(o2iblnd.c:861:kiblnd_create_conn()) alloc '(conn->ibc_rxs)': 26832 at ffffc90012e69000 (tot 75387476).
00000800:00000010:17.0:1533566495.592663:0:5553:0:(o2iblnd.c:1045:kiblnd_destroy_conn()) kfreed 'conn->ibc_rxs': 4576 at ffffc90012e69000 (tot 75470220).

Size was changed by:

00000800:00000400:17.0:1533566495.530176:0:5501:0:(o2iblnd.c:943:kiblnd_create_conn()) peer 172.18.2.3@o2ib - queue depth reduced from 128 to 21  to allow for qp creation

it shouldn't matter but LIBCFS_FREE decides whether to use kfree or vfree depending on a size.

Whamcloud Community JIRA

[LU-11702] ibc_rxs is created and freed with different size