https://jira.whamcloud.com This file is an XML representation of an issue en-us 9.4.14 940014 05-12-2023 https://jira.whamcloud.com/browse/LU-13652 Lustre <p>OSS LBUG. First time we have seen this.</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> [1574769.939126] LNetError: 7420:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds [1574769.972906] LNetError: 7420:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.151.11.102@o2ib (293): c: 32, oc: 0, rc: 32 [1574968.944839] LNetError: 7420:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Timed out tx: active_txs, 1 seconds [1574968.978608] LNetError: 7420:0:(o2iblnd_cb.c:3351:kiblnd_check_txs_locked()) Skipped 3 previous similar messages [1574969.012379] LNetError: 7420:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Timed out RDMA with 10.151.24.203@o2ib (247): c: 32, oc: 0, rc: 32 [1574969.053585] LNetError: 7420:0:(o2iblnd_cb.c:3426:kiblnd_check_conns()) Skipped 3 previous similar messages [1575256.183968] Lustre: nbp8-OST0103: Connection restored to 10dfb7c6-2481-1ba8-d8c9-5458677b6b29 (at 10.151.31.52@o2ib) [1575256.183973] Lustre: Skipped 15281 previous similar messages [1575337.223394] LNetError: 8719:0:(peer.c:280:lnet_destroy_peer_locked()) ASSERTION( list_empty(&amp;lp-&gt;lp_peer_nets) ) failed: [1575337.260035] LNetError: 8719:0:(peer.c:280:lnet_destroy_peer_locked()) LBUG [1575337.283229] Pid: 8719, comm: lnet_discovery 3.10.0-1062.12.1.el7_lustre2124.x86_64 #1 SMP Tue Mar 17 13:32:19 PDT 2020 [1575337.283233] Call Trace: [1575337.283243] [&lt;ffffffffc0cbd7cc&gt;] libcfs_call_trace+0x8c/0xc0 [libcfs] [1575337.305316] [&lt;ffffffffc0cbd87c&gt;] lbug_with_loc+0x4c/0xa0 [libcfs] [1575337.305340] [&lt;ffffffffc0d56a8a&gt;] lnet_destroy_peer_locked+0x24a/0x350 [lnet] [1575337.305351] [&lt;ffffffffc0d570c5&gt;] lnet_peer_discovery_complete+0x2a5/0x350 [lnet] [1575337.305361] [&lt;ffffffffc0d5bd20&gt;] lnet_peer_discovery+0x6c0/0x1150 [lnet] [1575337.305365] [&lt;ffffffffb20c61f1&gt;] kthread+0xd1/0xe0 [1575337.305368] [&lt;ffffffffb278dd37&gt;] ret_from_fork_nospec_end+0x0/0x39 [1575337.305389] [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff [1575337.305391] Kernel panic - not syncing: LBUG [1575337.305393] CPU: 11 PID: 8719 Comm: lnet_discovery Kdump: loaded Tainted: G OE ------------ 3.10.0-1062.12.1.el7_lustre2124.x86_64 #1 [1575337.305394] Hardware name: SGI.COM SUMMIT/S2600GZ, BIOS SE5C600.86B.02.01.0002.082220131453 08/22/2013 [1575337.305395] Call Trace: [1575337.305399] [&lt;ffffffffb277ac43&gt;] dump_stack+0x19/0x1b [1575337.305402] [&lt;ffffffffb2774987&gt;] panic+0xe8/0x21f [1575337.305408] [&lt;ffffffffc0cbd8cb&gt;] lbug_with_loc+0x9b/0xa0 [libcfs] [1575337.305417] [&lt;ffffffffc0d56a8a&gt;] lnet_destroy_peer_locked+0x24a/0x350 [lnet] [1575337.305425] [&lt;ffffffffc0d570c5&gt;] lnet_peer_discovery_complete+0x2a5/0x350 [lnet] [1575337.305434] [&lt;ffffffffc0d5bd20&gt;] lnet_peer_discovery+0x6c0/0x1150 [lnet] [1575337.305436] [&lt;ffffffffb20c72e0&gt;] ? wake_up_atomic_t+0x30/0x30 [1575337.305444] [&lt;ffffffffc0d5b660&gt;] ? lnet_peer_merge_data+0xde0/0xde0 [lnet] [1575337.305446] [&lt;ffffffffb20c61f1&gt;] kthread+0xd1/0xe0 [1575337.305448] [&lt;ffffffffb20c6120&gt;] ? insert_kthread_work+0x40/0x40 [1575337.305450] [&lt;ffffffffb278dd37&gt;] ret_from_fork_nospec_begin+0x21/0x21 [1575337.305452] [&lt;ffffffffb20c6120&gt;] ? insert_kthread_work+0x40/0x40 </pre> </div></div> LU-13652

[1575337.260035] LNetError: 8719:0:(peer.c:280:lnet_destroy_peer_locked()) LBUG

Bug Major Resolved Duplicate Amir Shehata Mahmoud Hanafi Tue, 9 Jun 2020 02:25:09 +0000 Sat, 19 Sep 2020 12:53:20 +0000 Sat, 19 Sep 2020 12:53:14 +0000 Lustre 2.12.4 0 4 <p>Hi Mahmoud,</p> <p>This looks like it could be a duplicate of this: <a href="https://jira.whamcloud.com/browse/LU-9971" class="external-link" rel="nofollow">https://jira.whamcloud.com/browse/LU-9971</a></p> <p>Do you have the patches which are outlined on this ticket?</p> <p>Hi Amir,</p> <p><a href="https://jira.whamcloud.com/browse/LU-9971" title="MR: ABA problem in lnet_discover_peer_locked" class="issue-link" data-issue-key="LU-9971"><del>LU-9971</del></a> commit did not land to b2_12.</p> <p>Thanks, Jay</p> <p>Amir</p> <p>Could you please port these fixes to b2_12 so that they can be considered for 2.12.6 (and NASA can carry the patches in the meantime)</p> <p>Thanks</p> <p>Peter</p> <p>I uploaded the relevant patches to b2_12.</p> <p>Thank you, Amir. I will cherry-pick the two patches you uploaded once they land to b2_12.</p> <p>Hey Jay, There was a problem with one of the patches which I just fixed.</p> <p>You'll need three patches:</p> <p><a href="https://review.whamcloud.com/#/c/38890/2" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/38890/2</a></p> <p><a href="https://review.whamcloud.com/#/c/38891/2" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/38891/2</a></p> <p><a href="https://review.whamcloud.com/#/c/38892/2" class="external-link" target="_blank" rel="nofollow noopener">https://review.whamcloud.com/#/c/38892/2</a></p> <p> </p> <p>It ended up being two patches in the end, but everything is in b2_12 for 2.12.6 now</p> Related LU-9971 Development Rank 1|i0129r: Rank (Obsolete) 9223372036854775807 Severity