Details
-
Bug
-
Resolution: Duplicate
-
Major
-
None
-
Lustre 2.12.3
-
None
-
CentOS 7.6
-
2
-
9223372036854775807
Description
We have been upgrading our Lnet routers recently to 2.12.3 and all of them crashed simultaneously tonight with the following assertion:
[39140.467535] LNetError: 14141:0:(lib-msg.c:894:lnet_finalize()) ASSERTION( !(((current_thread_info()->preempt_count) & ((((1UL << (10))-1) << ((0 + 8) + 8)) | (((1UL << (8))-1) << (0 + 8)) | (((1UL << (1))-1) << (((0 + 8) + 8) + 10))))) [39140.491917] general protection fault: 0000 [#1] SMP [39140.491969] Modules linked in: ko2iblnd(OE) lnet(OE) libcfs(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx5_ib(OE) mlx5_core(OE) mlxfw(OE) mlx4_en(OE) mlx4_ib(OE) ib_uverbsm [39140.491977] crct10dif_pclmul crct10dif_common tg3 libahci megaraid_sas ptp libata crc32c_intel pps_core [last unloaded: mlx_compat] [39140.491982] CPU: 0 PID: 14141 Comm: kiblnd_connd Kdump: loaded Tainted: G OE ------------ 3.10.0-957.27.2.el7.x86_64 #1 [39140.491983] Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.10.5 07/25/2019 [39140.491985] task: ffff90b918b1a080 ti: ffff90b8fa518000 task.ti: ffff90b8fa518000 [39140.491995] RIP: 0010:[<ffffffff886f3875>] [<ffffffff886f3875>] cpuacct_charge+0x35/0x50 [39140.491997] RSP: 0018:ffff90b91c603dd0 EFLAGS: 00010006 [39140.491998] RAX: 18244c8948c18cb8 RBX: ffff90b918b1a0e8 RCX: 000000000000ffff [39140.492000] RDX: ffffffff8925b640 RSI: 0000000001743e28 RDI: ffff90b918b1a080 [39140.492002] RBP: ffff90b91c603dd0 R08: ffffffffffffb820 R09: 000000000000040f [39140.492003] R10: 0000000000000004 R11: 0000000000000005 R12: 0000000001743e28 [39140.492005] R13: ffff90b91c61ac00 R14: ffff90b918b1a080 R15: 0000000000000000 [39140.492008] FS: 0000000000000000(0000) GS:ffff90b91c600000(0000) knlGS:0000000000000000 [39140.492010] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [39140.492011] CR2: 00007fd46cd96248 CR3: 0000000154c10000 CR4: 00000000003607f0 [39140.492013] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [39140.492015] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [39140.492016] Call Trace: [39140.492025] <IRQ> [39140.492025] [<ffffffff886e143c>] update_curr+0x14c/0x1e0 [39140.492029] [<ffffffff886e295d>] task_tick_fair+0x2bd/0x660 [39140.492034] [<ffffffff88634919>] ? sched_clock+0x9/0x10 [39140.492038] [<ffffffff886db1f5>] ? sched_clock_cpu+0x85/0xc0 [39140.492041] [<ffffffff886d60ad>] scheduler_tick+0xcd/0x150 [39140.492046] [<ffffffff8870c160>] ? tick_sched_do_timer+0x50/0x50 [39140.492051] [<ffffffff886ac3a5>] update_process_times+0x65/0x80 [39140.492055] [<ffffffff8870bed0>] tick_sched_handle+0x30/0x70 [39140.492058] [<ffffffff8870c199>] tick_sched_timer+0x39/0x80 [39140.492065] [<ffffffff886c71e3>] __hrtimer_run_queues+0xf3/0x270 [39140.492069] [<ffffffff886c776f>] hrtimer_interrupt+0xaf/0x1d0 [39140.492076] [<ffffffff8865a61b>] local_apic_timer_interrupt+0x3b/0x60 [39140.492081] [<ffffffff88d7b6e3>] smp_apic_timer_interrupt+0x43/0x60 [39140.492087] [<ffffffff88d77df2>] apic_timer_interrupt+0x162/0x170 [39140.492111] <EOI> [39140.492111] [<ffffffffc0ac3f9d>] ? lnet_finalize+0x98d/0x9a0 [lnet] [39140.492127] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.492156] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.492171] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.492184] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.492196] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.492206] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.492219] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.492232] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.492243] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.492254] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.492264] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.492276] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.492288] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.492299] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.492309] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.492319] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.492330] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.492341] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.492353] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.492363] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.492372] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.492383] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.492395] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.492406] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.492416] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.492425] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.492436] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.492447] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.492457] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.492468] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.492476] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.492487] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.492501] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.492512] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.492522] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.492530] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.492541] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.492552] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.492563] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.492573] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.492582] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.492592] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.492603] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.492614] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.492624] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.492632] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.492642] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.492653] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.492664] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.492674] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.492682] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.492693] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.492704] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.492714] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.492724] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.492732] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.492743] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.492753] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.492764] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.492774] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.492782] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.492792] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.492803] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.492813] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.492823] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.492831] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.492842] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.492853] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.492863] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.492873] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.492881] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.492892] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.492902] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.492913] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.492923] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.492931] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.492941] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.492952] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.492962] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.492972] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.492980] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.492991] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.493002] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.493012] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.493022] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.493030] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.493040] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.493051] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.493061] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.493071] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.493079] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.493089] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.493100] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.493111] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.493121] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.493128] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.493139] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.493150] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.493160] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.493170] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.493178] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.493188] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.493199] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.493209] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.493219] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.493227] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.493237] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.493248] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.493258] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.493268] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.493276] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.493287] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.493297] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.493308] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.493318] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.493325] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.493336] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.493347] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.493357] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.493367] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.493375] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.493385] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.493396] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.493406] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.493416] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.493424] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.493434] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.493445] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.493455] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.493466] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.493473] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.493484] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.493496] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.493508] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.493518] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.493525] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.493536] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.493547] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.493557] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.493567] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.493575] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.493585] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.493596] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.493606] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.493616] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.493624] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.493634] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.493645] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.493655] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.493665] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.493673] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.493684] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.493694] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.493704] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.493714] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.493722] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.493733] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.493743] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.493753] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.493763] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.493771] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.493782] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.493792] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.493802] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.493813] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.493820] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.493831] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.493842] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.493852] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.493862] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.493869] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.493880] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.493891] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.493901] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.493911] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.493918] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.493929] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.493940] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.493950] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.493960] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.493967] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.493978] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.493989] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.493999] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.494009] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.494017] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.494027] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.494038] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.494048] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.494058] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.494065] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.494076] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.494087] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.494097] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.494107] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.494114] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.494125] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.494136] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.494146] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.494156] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.494163] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.494174] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.494185] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.494195] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.494205] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.494212] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.494223] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.494233] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.494243] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.494253] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.494261] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.494271] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.494282] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.494292] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.494302] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.494310] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.494320] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.494332] [<ffffffffc0ac0082>] ? libcfs_nid2str_r+0xe2/0x130 [lnet] [39140.494343] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.494353] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.494363] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.494372] [<ffffffffc090ff97>] ? libcfs_debug_msg+0x57/0x80 [libcfs] [39140.494382] [<ffffffffc0acd71a>] ? lnet_post_send_locked+0x41a/0x9c0 [lnet] [39140.494391] [<ffffffffc090fae8>] ? libcfs_debug_vmsg2+0x6d8/0xb30 [libcfs] [39140.494402] [<ffffffffc0acf9a8>] ? lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [39140.494412] [<ffffffffc0ac3401>] ? lnet_health_check+0x6a1/0x8b0 [lnet] [39140.494423] [<ffffffffc0ac377f>] ? lnet_finalize+0x16f/0x9a0 [lnet] [39140.494432] [<ffffffffc0babd22>] ? kiblnd_pool_free_node+0x82/0x170 [ko2iblnd] [39140.494440] [<ffffffffc0bb561d>] ? kiblnd_tx_done+0x10d/0x3e0 [ko2iblnd] [39140.494447] [<ffffffffc0bb593b>] ? kiblnd_txlist_done+0x4b/0x60 [ko2iblnd] [39140.494454] [<ffffffffc0bbab83>] ? kiblnd_check_conns+0x553/0x880 [ko2iblnd] [39140.494465] [<ffffffffc09213ba>] ? cfs_percpt_unlock+0x1a/0xb0 [libcfs] [39140.494473] [<ffffffffc0bbfc1b>] ? kiblnd_connd+0x83b/0xa00 [ko2iblnd] [39140.494476] [<ffffffff886d7c40>] ? wake_up_state+0x20/0x20 [39140.494484] [<ffffffffc0bbf3e0>] ? kiblnd_cm_callback+0x2380/0x2380 [ko2iblnd] [39140.494487] [<ffffffff886c2e81>] ? kthread+0xd1/0xe0 [39140.494490] [<ffffffff886c2db0>] ? insert_kthread_work+0x40/0x40 [39140.494495] [<ffffffff88d76c37>] ? ret_from_fork_nospec_begin+0x21/0x21 [39140.494499] [<ffffffff886c2db0>] ? insert_kthread_work+0x40/0x40 [39140.494536] Code: 48 89 e5 48 63 48 18 48 8b 87 40 09 00 00 48 8b 50 48 eb 0b 66 90 48 8b 50 68 48 85 d2 74 1b 48 8b 42 40 48 03 04 cd a0 bf 34 89 <48> 01 30 48 8b 02 48 8b 40 40 48 85 c0 75 dc 5d c3 66 2e 0f 1f [39140.494541] RIP [<ffffffff886f3875>] cpuacct_charge+0x35/0x50 [39140.494541] RSP <ffff90b91c603dd0>
[root@sh-rtr-fir-1-1 127.0.0.1-2019-10-25-21:01:51]# rpm -qa | grep lustre lustre-client-2.12.3-1.el7.x86_64 lustre-client-dkms-2.12.3-1.el7.noarch [root@sh-rtr-fir-1-1 127.0.0.1-2019-10-25-21:01:51]# uname -a Linux sh-rtr-fir-1-1.int 3.10.0-957.27.2.el7.x86_64 #1 SMP Mon Jul 29 17:46:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
