https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-10459 Lustre <p>LBUG occurs immediately when we try to do any IO on clients. Multiple clients impacted</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Jan 4 21:57:38 soak-17 kernel: LNetError: 12570:0:(o2iblnd_cb.c:991:kiblnd_check_sends_locked()) ASSERTION( conn-&gt;ibc_nsends_posted &lt;= conn-&gt;ibc_queue_depth ) failed: Jan 4 21:57:38 soak-17 kernel: LNetError: 12570:0:(o2iblnd_cb.c:991:kiblnd_check_sends_locked()) LBUG Jan 4 21:57:38 soak-17 kernel: Pid: 12570, comm: kiblnd_sd_00_00 Jan 4 21:57:38 soak-17 kernel: #012Call Trace: Jan 4 21:57:38 soak-17 kernel: [&lt;ffffffffc097c7ae&gt;] libcfs_call_trace+0x4e/0x60 [libcfs] Jan 4 21:57:38 soak-17 kernel: [&lt;ffffffffc097c83c&gt;] lbug_with_loc+0x4c/0xb0 [libcfs] Jan 4 21:57:38 soak-17 kernel: [&lt;ffffffffc0c2666b&gt;] kiblnd_check_sends_locked+0xd8b/0xd90 [ko2iblnd] Jan 4 21:57:38 soak-17 kernel: [&lt;ffffffffc0538b5c&gt;] ? mlx4_ib_post_recv+0x1dc/0x310 [mlx4_ib] Jan 4 21:57:38 soak-17 kernel: [&lt;ffffffffc0c27f50&gt;] kiblnd_post_rx+0x160/0x520 [ko2iblnd] Jan 4 21:57:38 soak-17 kernel: [&lt;ffffffffc0c284ea&gt;] kiblnd_recv+0x1da/0x7b0 [ko2iblnd] Jan 4 21:57:38 soak-17 kernel: [&lt;ffffffffc0a00573&gt;] lnet_ni_recv+0xc3/0x320 [lnet] Jan 4 21:57:38 soak-17 kernel: [&lt;ffffffffc0a02e06&gt;] lnet_parse_local+0x4c6/0xd40 [lnet] Jan 4 21:57:38 soak-17 kernel: [&lt;ffffffff810c7705&gt;] ? sched_clock_cpu+0x85/0xc0 Jan 4 21:57:38 soak-17 kernel: [&lt;ffffffffc0a03f4a&gt;] lnet_parse+0x8ca/0xfc0 [lnet] Jan 4 21:57:38 soak-17 kernel: [&lt;ffffffffc0c261ac&gt;] ? kiblnd_check_sends_locked+0x8cc/0xd90 [ko2iblnd] Jan 4 21:57:38 soak-17 kernel: [&lt;ffffffff81029557&gt;] ? __switch_to+0xd7/0x510 Jan 4 21:57:38 soak-17 kernel: [&lt;ffffffffc0c28e63&gt;] kiblnd_handle_rx+0x213/0x6b0 [ko2iblnd] Jan 4 21:57:38 soak-17 kernel: [&lt;ffffffffc0c2facf&gt;] kiblnd_scheduler+0xf0f/0x1150 [ko2iblnd] Jan 4 21:57:38 soak-17 kernel: [&lt;ffffffff810ce55e&gt;] ? dequeue_task_fair+0x41e/0x660 Jan 4 21:57:38 soak-17 kernel: [&lt;ffffffff810c7705&gt;] ? sched_clock_cpu+0x85/0xc0 Jan 4 21:57:38 soak-17 kernel: [&lt;ffffffff810c4820&gt;] ? default_wake_function+0x0/0x20 Jan 4 21:57:38 soak-17 kernel: [&lt;ffffffffc0c2ebc0&gt;] ? kiblnd_scheduler+0x0/0x1150 [ko2iblnd] Jan 4 21:57:38 soak-17 kernel: [&lt;ffffffff810b099f&gt;] kthread+0xcf/0xe0 Jan 4 21:57:38 soak-17 kernel: [&lt;ffffffff810b08d0&gt;] ? kthread+0x0/0xe0 Jan 4 21:57:38 soak-17 kernel: [&lt;ffffffff816b4fd8&gt;] ret_from_fork+0x58/0x90 Jan 4 21:57:38 soak-17 kernel: [&lt;ffffffff810b08d0&gt;] ? kthread+0x0/0xe0 Jan 4 21:57:38 soak-17 kernel: </pre> </div></div> <p>Multiple crash dumps available on Spirit</p> Soak performance cluster, version=2.10.56_84_gd645c72, RHEL 7.4 kernel LU-10459

LBUG o2iblnd_cb.c:991:kiblnd_check_sends_locked()) ASSERTION( conn->ibc_nsends_posted <= conn->ibc_queue_depth ) failed:

Bug Critical Resolved Fixed Amir Shehata Cliff White soak Thu, 4 Jan 2018 22:09:00 +0000 Mon, 7 Jan 2019 19:09:52 +0000 Fri, 19 Jan 2018 20:48:37 +0000 Lustre 2.11.0 Lustre 2.11.0 0 8 <p>This is most likely related to: <a href="https://jira.whamcloud.com/browse/LU-10291" title="remove concurrent_sends tunable" class="issue-link" data-issue-key="LU-10291"><del>LU-10291</del></a> lnd: remove concurrent_sends tunable</p> <p>This only affects master. I'm investigating.</p> <p>Amir Shehata (amir.shehata@intel.com) uploaded a new patch: <a href="https://review.whamcloud.com/30751" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30751</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10459" title="LBUG o2iblnd_cb.c:991:kiblnd_check_sends_locked()) ASSERTION( conn-&gt;ibc_nsends_posted &lt;= conn-&gt;ibc_queue_depth ) failed:" class="issue-link" data-issue-key="LU-10459"><del>LU-10459</del></a> lnd: throttle tx based on queue depth<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: 1<br/> Commit: 86d289fed3c0eeccc3a0650d7e5a842391d11c3e</p> <p>Testing the above patch on soak, appears to fix the immediate LBUG.<br/> Soak has been running now for about 30 minutes, will see how we do.</p> <p>I checked b2_10, it doesn't look like <a href="https://jira.whamcloud.com/browse/LU-10291" title="remove concurrent_sends tunable" class="issue-link" data-issue-key="LU-10291"><del>LU-10291</del></a> lnd: remove concurrent_sends tunable was ported over, so I'm wondering if this is the same issue. That assert was hit due to the above patch. </p> <p>Ah, sorry wrong bug - my bad</p> <p>sorry, commit was added against the wrong ticket.</p> <p>I am currently seeing this on a lustre-review-ib build version=2.10.56_86_gd8827a8</p> <p>The patch which fixes the issue hasn't landed yet.</p> <p>Oleg Drokin (oleg.drokin@intel.com) merged in patch <a href="https://review.whamcloud.com/30751/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/30751/</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10459" title="LBUG o2iblnd_cb.c:991:kiblnd_check_sends_locked()) ASSERTION( conn-&gt;ibc_nsends_posted &lt;= conn-&gt;ibc_queue_depth ) failed:" class="issue-link" data-issue-key="LU-10459"><del>LU-10459</del></a> lnd: throttle tx based on queue depth<br/> Project: fs/lustre-release<br/> Branch: master<br/> Current Patch Set: <br/> Commit: e86f55798ca7bc8f7fe22dd48c9d9f52c1bb029a</p> <p>Landed to master for 2.11.0</p> <p>Minh Diep (mdiep@whamcloud.com) uploaded a new patch: <a href="https://review.whamcloud.com/33150" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/33150</a><br/> Subject: <a href="https://jira.whamcloud.com/browse/LU-10459" title="LBUG o2iblnd_cb.c:991:kiblnd_check_sends_locked()) ASSERTION( conn-&gt;ibc_nsends_posted &lt;= conn-&gt;ibc_queue_depth ) failed:" class="issue-link" data-issue-key="LU-10459"><del>LU-10459</del></a> lnd: throttle tx based on queue depth<br/> Project: fs/lustre-release<br/> Branch: b2_10<br/> Current Patch Set: 1<br/> Commit: dce2da916afe3fa474e2199b4993c91ced4e45cf</p> Related LU-10291 Development Rank 1|hzzqjb: Rank (Obsolete) 9223372036854775807 Severity