https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-12906 Lustre <p>Using 2.12.3 servers and clients, we hit this bug once on an OSS, once on a client:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[197803.220678] LNetError: 34981:0:(lib-move.c:2729:lnet_detach_rsp_tracker()) ASSERTION( rspt-&gt;rspt_cpt == cpt ) failed: [197803.220682] LNetError: 34981:0:(lib-move.c:2729:lnet_detach_rsp_tracker()) LBUG [197803.220684] Pid: 34981, comm: kiblnd_sd_01_00 3.10.0-957.27.2.el7.x86_64 #1 SMP Mon Jul 29 17:46:05 UTC 2019 [197803.220684] Call Trace: [197803.220706] [&lt;ffffffffc0e777cc&gt;] libcfs_call_trace+0x8c/0xc0 [libcfs] [197803.220712] [&lt;ffffffffc0e7787c&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] [197803.220730] [&lt;ffffffffc0f1849b&gt;] lnet_detach_rsp_tracker+0x5b/0x60 [lnet] [197803.220739] [&lt;ffffffffc0f08d3a&gt;] lnet_finalize+0x72a/0x9a0 [lnet] [197803.220748] [&lt;ffffffffc0f12a51&gt;] lnet_post_send_locked+0x751/0x9c0 [lnet] [197803.220757] [&lt;ffffffffc0f149a8&gt;] lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [197803.220765] [&lt;ffffffffc0f075ec&gt;] lnet_msg_decommit+0xec/0x700 [lnet] [197803.220772] [&lt;ffffffffc0f089b7&gt;] lnet_finalize+0x3a7/0x9a0 [lnet] [197803.220783] [&lt;ffffffffc14c861d&gt;] kiblnd_tx_done+0x10d/0x3e0 [ko2iblnd] [197803.220789] [&lt;ffffffffc14d3b0d&gt;] kiblnd_scheduler+0x89d/0x1180 [ko2iblnd] [197803.220793] [&lt;ffffffff8d4c2e81&gt;] kthread+0xd1/0xe0 [197803.220797] [&lt;ffffffff8db76c37&gt;] ret_from_fork_nospec_end+0x0/0x39 [197803.220822] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff [197803.220823] Kernel panic - not syncing: LBUG [197803.220826] CPU: 37 PID: 34981 Comm: kiblnd_sd_01_00 Kdump: loaded Tainted: G OE ------------ 3.10.0-957.27.2.el7.x86_64 #1 [197803.220827] Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 2.9.1 12/04/2018 [197803.220828] Call Trace: [197803.220833] [&lt;ffffffff8db64147&gt;] dump_stack+0x19/0x1b [197803.220835] [&lt;ffffffff8db5d850&gt;] panic+0xe8/0x21f [197803.220842] [&lt;ffffffffc0e778cb&gt;] lbug_with_loc+0x9b/0xa0 [libcfs] [197803.220851] [&lt;ffffffffc0f1849b&gt;] lnet_detach_rsp_tracker+0x5b/0x60 [lnet] [197803.220860] [&lt;ffffffffc0f08d3a&gt;] lnet_finalize+0x72a/0x9a0 [lnet] [197803.220863] [&lt;ffffffff8d502372&gt;] ? ktime_get_ts64+0x52/0xf0 [197803.220872] [&lt;ffffffffc0f12a51&gt;] lnet_post_send_locked+0x751/0x9c0 [lnet] [197803.220880] [&lt;ffffffffc0f149a8&gt;] lnet_return_tx_credits_locked+0x2a8/0x490 [lnet] [197803.220888] [&lt;ffffffffc0f075ec&gt;] lnet_msg_decommit+0xec/0x700 [lnet] [197803.220896] [&lt;ffffffffc0f089b7&gt;] lnet_finalize+0x3a7/0x9a0 [lnet] [197803.220901] [&lt;ffffffffc14c861d&gt;] kiblnd_tx_done+0x10d/0x3e0 [ko2iblnd] [197803.220906] [&lt;ffffffffc14d3b0d&gt;] kiblnd_scheduler+0x89d/0x1180 [ko2iblnd] [197803.220908] [&lt;ffffffff8d4e220e&gt;] ? dequeue_task_fair+0x41e/0x660 [197803.220911] [&lt;ffffffff8d42a59e&gt;] ? __switch_to+0xce/0x580 [197803.220913] [&lt;ffffffff8d4d7c40&gt;] ? wake_up_state+0x20/0x20 [197803.220919] [&lt;ffffffffc14d3270&gt;] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd] [197803.220920] [&lt;ffffffff8d4c2e81&gt;] kthread+0xd1/0xe0 [197803.220923] [&lt;ffffffff8d4c2db0&gt;] ? insert_kthread_work+0x40/0x40 [197803.220925] [&lt;ffffffff8db76c37&gt;] ret_from_fork_nospec_begin+0x21/0x21 [197803.220927] [&lt;ffffffff8d4c2db0&gt;] ? insert_kthread_work+0x40/0x40 </pre> </div></div> <p>We're using Mellanox OFED 4.7 on servers / routers / this client (not all clients have been upgraded yet).</p> <p>We can provide crash dumps upon request. This seems to be a new problem either with 2.12.3 vs 2.12.0 or with Mellanox OFED 4.7 vs 4.5.</p> <p>Thanks!<br/> Stephane</p> CentOS 7.6 LU-12906

LBUG ASSERTION( rspt->rspt_cpt == cpt ) failed

Bug Major Resolved Duplicate Amir Shehata Stephane Thiell Sat, 26 Oct 2019 02:45:23 +0000 Fri, 31 Jul 2020 15:10:39 +0000 Thu, 12 Dec 2019 19:49:28 +0000 Lustre 2.12.3 0 6 <p>On a server (no crash dump found, I'll investigate why...):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[Fri Oct 25 10:06:46 2019][772788.297828] LustreError: 49526:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff9960e18c5200^M [Fri Oct 25 10:06:46 2019][772788.412180] LustreError: 49526:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff993f267bfa00^M [Fri Oct 25 10:06:46 2019][772788.414316] LustreError: 49525:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff993cb80dd400^M [Fri Oct 25 10:06:46 2019][772788.414321] LustreError: 49525:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff99504a97e800^M [Fri Oct 25 10:06:46 2019][772788.414328] LustreError: 49525:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff99417c564a00^M [Fri Oct 25 10:06:46 2019][772788.414334] LustreError: 49525:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff994561e9f600^M [Fri Oct 25 10:06:46 2019][772788.523551] LustreError: 49526:0:(events.c:450:server_bulk_callback()) event type 5, status -125, desc ffff996515360200^M [Fri Oct 25 10:06:46 2019][772788.534421] LNetError: 49526:0:(lib-move.c:2729:lnet_detach_rsp_tracker()) ASSERTION( rspt-&gt;rspt_cpt == cpt ) failed: ^M [Fri Oct 25 10:06:46 2019][772788.545237] LNetError: 49526:0:(lib-move.c:2729:lnet_detach_rsp_tracker()) LBUG^M [Fri Oct 25 10:06:46 2019][772788.552642] Pid: 49526, comm: kiblnd_sd_01_01 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1 SMP Mon Aug 5 15:28:37 PDT 2019^M [Fri Oct 25 10:06:46 2019][772788.563420] Call Trace:^M [Fri Oct 25 10:06:46 2019][772788.565978] [&lt;ffffffffc0c467cc&gt;] libcfs_call_trace+0x8c/0xc0 [libcfs]^M [Fri Oct 25 10:06:46 2019][772788.572640] [&lt;ffffffffc0c4687c&gt;] lbug_with_loc+0x4c/0xa0 [libcfs]^M [Fri Oct 25 10:06:46 2019][772788.578940] [&lt;ffffffffc0d4e49b&gt;] lnet_detach_rsp_tracker+0x5b/0x60 [lnet]^M [Fri Oct 25 10:06:46 2019][772788.585943] [&lt;ffffffffc0d3ed3a&gt;] lnet_finalize+0x72a/0x9a0 [lnet]^M [Fri Oct 25 10:06:46 2019][772788.592251] [&lt;ffffffffc0d48a51&gt;] lnet_post_send_locked+0x751/0x9c0 [lnet]^M [Fri Oct 25 10:06:46 2019][772788.599255] [&lt;ffffffffc0d4a9a8&gt;] lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]^M [Fri Oct 25 10:06:46 2019][772788.606942] [&lt;ffffffffc0d3d5ec&gt;] lnet_msg_decommit+0xec/0x700 [lnet]^M [Fri Oct 25 10:06:46 2019][772788.613514] [&lt;ffffffffc0d3e9b7&gt;] lnet_finalize+0x3a7/0x9a0 [lnet]^M [Fri Oct 25 10:06:46 2019][772788.619814] [&lt;ffffffffc0cbb61d&gt;] kiblnd_tx_done+0x10d/0x3e0 [ko2iblnd]^M [Fri Oct 25 10:06:46 2019][772788.626556] [&lt;ffffffffc0cc6b0d&gt;] kiblnd_scheduler+0x89d/0x1180 [ko2iblnd]^M [Fri Oct 25 10:06:46 2019][772788.633550] [&lt;ffffffff976c2e81&gt;] kthread+0xd1/0xe0^M [Fri Oct 25 10:06:46 2019][772788.638551] [&lt;ffffffff97d77c24&gt;] ret_from_fork_nospec_begin+0xe/0x21^M [Fri Oct 25 10:06:46 2019][772788.645112] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff^M [Fri Oct 25 10:06:46 2019][772788.650230] Kernel panic - not syncing: LBUG^M [Fri Oct 25 10:06:46 2019][772788.654593] CPU: 1 PID: 49526 Comm: kiblnd_sd_01_01 Kdump: loaded Tainted: G OE ------------ 3.10.0-957.27.2.el7_lustre.pl1.x86_64 #1^M [Fri Oct 25 10:06:46 2019][772788.667789] Hardware name: Dell Inc. PowerEdge R6415/065PKD, BIOS 1.10.6 08/15/2019^M [Fri Oct 25 10:06:46 2019][772788.675529] Call Trace:^M [Fri Oct 25 10:06:46 2019][772788.678071] [&lt;ffffffff97d65147&gt;] dump_stack+0x19/0x1b^M [Fri Oct 25 10:06:47 2019][772788.683296] [&lt;ffffffff97d5e850&gt;] panic+0xe8/0x21f^M [Fri Oct 25 10:06:47 2019][772788.688179] [&lt;ffffffffc0c468cb&gt;] lbug_with_loc+0x9b/0xa0 [libcfs]^M [Fri Oct 25 10:06:47 2019][772788.694458] [&lt;ffffffffc0d4e49b&gt;] lnet_detach_rsp_tracker+0x5b/0x60 [lnet]^M [Fri Oct 25 10:06:47 2019][772788.701424] [&lt;ffffffffc0d3ed3a&gt;] lnet_finalize+0x72a/0x9a0 [lnet]^M [Fri Oct 25 10:06:47 2019][772788.707695] [&lt;ffffffff97702372&gt;] ? ktime_get_ts64+0x52/0xf0^M [Fri Oct 25 10:06:47 2019][772788.713447] [&lt;ffffffffc0d48a51&gt;] lnet_post_send_locked+0x751/0x9c0 [lnet]^M [Fri Oct 25 10:06:47 2019][772788.720414] [&lt;ffffffffc0d4a9a8&gt;] lnet_return_tx_credits_locked+0x2a8/0x490 [lnet]^M [Fri Oct 25 10:06:47 2019][772788.728075] [&lt;ffffffffc0d3d5ec&gt;] lnet_msg_decommit+0xec/0x700 [lnet]^M [Fri Oct 25 10:06:47 2019][772788.734610] [&lt;ffffffffc0d3e9b7&gt;] lnet_finalize+0x3a7/0x9a0 [lnet]^M [Fri Oct 25 10:06:47 2019][772788.740881] [&lt;ffffffffc0cbb61d&gt;] kiblnd_tx_done+0x10d/0x3e0 [ko2iblnd]^M [Fri Oct 25 10:06:47 2019][772788.747581] [&lt;ffffffffc0cc6b0d&gt;] kiblnd_scheduler+0x89d/0x1180 [ko2iblnd]^M [Fri Oct 25 10:06:47 2019][772788.754546] [&lt;ffffffff976e220e&gt;] ? dequeue_task_fair+0x41e/0x660^M [Fri Oct 25 10:06:47 2019][772788.760728] [&lt;ffffffff9762a59e&gt;] ? __switch_to+0xce/0x580^M [Fri Oct 25 10:06:47 2019][772788.766299] [&lt;ffffffff976d7c40&gt;] ? wake_up_state+0x20/0x20^M [Fri Oct 25 10:06:47 2019][772788.771959] [&lt;ffffffffc0cc6270&gt;] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd]^M [Fri Oct 25 10:06:47 2019][772788.778743] [&lt;ffffffff976c2e81&gt;] kthread+0xd1/0xe0^M [Fri Oct 25 10:06:47 2019][772788.783709] [&lt;ffffffff976c2db0&gt;] ? insert_kthread_work+0x40/0x40^M [Fri Oct 25 10:06:47 2019][772788.789889] [&lt;ffffffff97d77c24&gt;] ret_from_fork_nospec_begin+0xe/0x21^M [Fri Oct 25 10:06:47 2019][772788.796414] [&lt;ffffffff976c2db0&gt;] ? insert_kthread_work+0x40/0x40^M </pre> </div></div> <p>Amir</p> <p>Could you please advise?</p> <p>Thanks</p> <p>Peter</p> <p>There are two patches which I think should resolve this issue:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> LU-12568 lnet: Defer rspt cleanup when MD queued <span class="code-keyword">for</span> unlink LU-12441 lnet: Detach rspt when md_threshold is infinite</pre> </div></div> <p><a href="https://jira.whamcloud.com/browse/LU-12568" title="LNetError: 28086:0:(lib-move.c:2862:lnet_detach_rsp_tracker()) ASSERTION( rspt-&gt;rspt_cpt == cpt ) failed" class="issue-link" data-issue-key="LU-12568"><del>LU-12568</del></a> fixes a memory corruption which could be leading to both of <a href="https://jira.whamcloud.com/browse/LU-12906" title="LBUG ASSERTION( rspt-&gt;rspt_cpt == cpt ) failed" class="issue-link" data-issue-key="LU-12906"><del>LU-12906</del></a> and <a href="https://jira.whamcloud.com/browse/LU-12907" title="LNet routers: LNetError: 14141:0:(lib-msg.c:894:lnet_finalize()) ASSERTION( !(((current_thread_info()-&gt;preempt_count) &amp; ((((1UL &lt;&lt; (10))-1) &lt;&lt; ((0 + 8) + 8)) | (((1UL &lt;&lt; (8))-1) &lt;&lt; (0 + 8)) | (((1UL &lt;&lt; (1))-1) &lt;&lt; (((0 + 8) + 8) + 10)))))" class="issue-link" data-issue-key="LU-12907"><del>LU-12907</del></a>.</p> <p>I can port these back to b2.12. Would you be able to test if they resolve the issue?</p> <p>the patch for <a href="https://jira.whamcloud.com/browse/LU-12441" title="Response tracker is not detached on router ping reply" class="issue-link" data-issue-key="LU-12441"><del>LU-12441</del></a> could be applied as is on b2_12</p> <p>Attached is an updated patch for <span class="nobr"><a href="https://jira.whamcloud.com/secure/attachment/33727/33727_LU-12568.patch" title="LU-12568.patch attached to LU-12906">LU-12568.patch^{<img class="rendericon" src="https://jira.whamcloud.com/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> which should work on b2_12</p> <p>Would we be able to try these two patches out and see if they resolve the problems you're seeing on <a href="https://jira.whamcloud.com/browse/LU-12906" title="LBUG ASSERTION( rspt-&gt;rspt_cpt == cpt ) failed" class="issue-link" data-issue-key="LU-12906"><del>LU-12906</del></a> and <a href="https://jira.whamcloud.com/browse/LU-12907" title="LNet routers: LNetError: 14141:0:(lib-msg.c:894:lnet_finalize()) ASSERTION( !(((current_thread_info()-&gt;preempt_count) &amp; ((((1UL &lt;&lt; (10))-1) &lt;&lt; ((0 + 8) + 8)) | (((1UL &lt;&lt; (8))-1) &lt;&lt; (0 + 8)) | (((1UL &lt;&lt; (1))-1) &lt;&lt; (((0 + 8) + 8) + 10)))))" class="issue-link" data-issue-key="LU-12907"><del>LU-12907</del></a>? I think if the router problem is happening continuously it would be quicker to test these two patches on the routers.</p> <p>Thanks Amir! I'll see what we can do to test this patch, at least on the LNet routers first. I'll keep you posted.</p> <p>As both the mentioned patches are landed for 2.12.4 can we close out this ticket as a duplicate?</p> <p>Peter, that sounds good to me. We haven't hit this issue since we added the two patches above. Thanks!</p> <p>Thanks Stephane!</p> Duplicate Related LU-12568 LU-12441 Development Rank 1|i00om7: Rank (Obsolete) 9223372036854775807 Severity