https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13218 Lustre <p>conf-sanity test_98 hangs for review-dne-zfs-part-3 for the patch <a href="https://review.whamcloud.com/37445/" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/37445/</a> for <a href="https://jira.whamcloud.com/browse/LU-12593" title="update_log corruption" class="issue-link" data-issue-key="LU-12593"><del>LU-12593</del></a>. We've seen this test hang with the errors below once only so far.</p> <p>Looking at the hang at <a href="https://testing.whamcloud.com/test_sets/469de552-4869-11ea-b58e-52540065bddc" class="external-link" target="_blank" rel="nofollow noopener">https://testing.whamcloud.com/test_sets/469de552-4869-11ea-b58e-52540065bddc</a>, in the client1 console log we see LNET issues</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[19629.856282] Lustre: Unmounted lustre-client [19659.244701] Lustre: DEBUG MARKER: lsmod | grep lnet &gt; /dev/null &amp;&amp; lctl dl | grep ' ST ' || true [19659.384715] Key type lgssc unregistered [19661.272912] LNet: 5951:0:(socklnd.c:2550:ksocknal_shutdown()) waiting for 1 peers to disconnect [19665.275764] LNet: 5951:0:(socklnd.c:2550:ksocknal_shutdown()) waiting for 1 peers to disconnect [19673.280446] LNet: 5951:0:(socklnd.c:2550:ksocknal_shutdown()) waiting for 1 peers to disconnect [19689.288850] LNet: 5951:0:(socklnd.c:2550:ksocknal_shutdown()) waiting for 1 peers to disconnect [19721.304636] LNet: 5951:0:(socklnd.c:2550:ksocknal_shutdown()) waiting for 1 peers to disconnect [19785.335214] LNet: 5951:0:(socklnd.c:2550:ksocknal_shutdown()) waiting for 1 peers to disconnect [19809.073214] INFO: task socknal_sd00_01:4386 blocked for more than 120 seconds. [19809.074558] "echo 0 &gt; /proc/sys/kernel/hung_task_timeout_secs" disables this message. [19809.075866] socknal_sd00_01 D ffff8e17ffd1ac80 0 4386 2 0x00000080 [19809.077194] Call Trace: [19809.077665] [&lt;ffffffffb0581929&gt;] schedule_preempt_disabled+0x29/0x70 [19809.078735] [&lt;ffffffffb057f8b7&gt;] __mutex_lock_slowpath+0xc7/0x1d0 [19809.079762] [&lt;ffffffffb057ec8f&gt;] mutex_lock+0x1f/0x2f [19809.080741] [&lt;ffffffffc0891e31&gt;] lnet_nid2peerni_locked+0x71/0x150 [lnet] [19809.082031] [&lt;ffffffffc087ed01&gt;] lnet_parse+0x791/0x11f0 [lnet] [19809.083044] [&lt;ffffffffc0916838&gt;] ksocknal_process_receive+0x498/0xde0 [ksocklnd] [19809.084277] [&lt;ffffffffc0917626&gt;] ksocknal_scheduler+0x206/0xd50 [ksocklnd] [19809.085499] [&lt;ffffffffafec72e0&gt;] ? wake_up_atomic_t+0x30/0x30 [19809.086567] [&lt;ffffffffc0917420&gt;] ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd] [19809.087734] [&lt;ffffffffafec61f1&gt;] kthread+0xd1/0xe0 [19809.088612] [&lt;ffffffffafec6120&gt;] ? insert_kthread_work+0x40/0x40 [19809.089668] [&lt;ffffffffb058dd37&gt;] ret_from_fork_nospec_begin+0x21/0x21 [19809.090850] [&lt;ffffffffafec6120&gt;] ? insert_kthread_work+0x40/0x40 [19913.395403] LNet: 5951:0:(socklnd.c:2550:ksocknal_shutdown()) waiting for 1 peers to disconnect [19929.146713] INFO: task socknal_sd00_01:4386 blocked for more than 120 seconds. </pre> </div></div> LU-13218

conf-sanity test 98 hangs in socknal_sd00_01: lnet_nid2peerni_locked

Bug Minor Closed Duplicate Amir Shehata James Nunez Fri, 7 Feb 2020 18:58:01 +0000 Thu, 24 Mar 2022 16:22:41 +0000 Thu, 24 Mar 2022 16:22:41 +0000 Lustre 2.12.4 0 3 <p>This looks like a shutdown race condition. We call shutdown while a message from a peer is arriving. We lock the api mutex during shutdown. Then the receiving thread tries to lock it to create a peer NI, leading to this deadlock. the Shudown is waiting for the lnd thread to finish.</p> <p>We shouldn't be trying to create any new NIs while we're shutting down.</p> <p>I've seen this same thing with ko2iblnd on 2.12.5 yesterday as well.</p> <p>Looks like <a href="https://jira.whamcloud.com/browse/LU-15616" title="sanity-lnet test_226: Timeout occurred after 112 minutes, last suite running was sanity-lnet" class="issue-link" data-issue-key="LU-15616"><del>LU-15616</del></a> or <a href="https://jira.whamcloud.com/browse/LU-15618" title="ksock_conn ref leak on shutdown" class="issue-link" data-issue-key="LU-15618"><del>LU-15618</del></a> (or both) were hit.</p> Duplicate LU-15616 LU-15618 Development Rank 1|i00tef: Rank (Obsolete) 9223372036854775807 Severity